[
  {
    "path": ".gitignore",
    "content": ".DS_Store\n\n"
  },
  {
    "path": "COPYING",
    "content": "                    GNU AFFERO GENERAL PUBLIC LICENSE\n                       Version 3, 19 November 2007\n\n Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>\n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n\n                            Preamble\n\n  The GNU Affero General Public License is a free, copyleft license for\nsoftware and other kinds of works, specifically designed to ensure\ncooperation with the community in the case of network server software.\n\n  The licenses for most software and other practical works are designed\nto take away your freedom to share and change the works.  By contrast,\nour General Public Licenses are intended to guarantee your freedom to\nshare and change all versions of a program--to make sure it remains free\nsoftware for all its users.\n\n  When we speak of free software, we are referring to freedom, not\nprice.  Our General Public Licenses are designed to make sure that you\nhave the freedom to distribute copies of free software (and charge for\nthem if you wish), that you receive source code or can get it if you\nwant it, that you can change the software or use pieces of it in new\nfree programs, and that you know you can do these things.\n\n  Developers that use our General Public Licenses protect your rights\nwith two steps: (1) assert copyright on the software, and (2) offer\nyou this License which gives you legal permission to copy, distribute\nand/or modify the software.\n\n  A secondary benefit of defending all users' freedom is that\nimprovements made in alternate versions of the program, if they\nreceive widespread use, become available for other developers to\nincorporate.  Many developers of free software are heartened and\nencouraged by the resulting cooperation.  However, in the case of\nsoftware used on network servers, this result may fail to come about.\nThe GNU General Public License permits making a modified version and\nletting the public access it on a server without ever releasing its\nsource code to the public.\n\n  The GNU Affero General Public License is designed specifically to\nensure that, in such cases, the modified source code becomes available\nto the community.  It requires the operator of a network server to\nprovide the source code of the modified version running there to the\nusers of that server.  Therefore, public use of a modified version, on\na publicly accessible server, gives the public access to the source\ncode of the modified version.\n\n  An older license, called the Affero General Public License and\npublished by Affero, was designed to accomplish similar goals.  This is\na different license, not a version of the Affero GPL, but Affero has\nreleased a new version of the Affero GPL which permits relicensing under\nthis license.\n\n  The precise terms and conditions for copying, distribution and\nmodification follow.\n\n                       TERMS AND CONDITIONS\n\n  0. Definitions.\n\n  \"This License\" refers to version 3 of the GNU Affero General Public License.\n\n  \"Copyright\" also means copyright-like laws that apply to other kinds of\nworks, such as semiconductor masks.\n\n  \"The Program\" refers to any copyrightable work licensed under this\nLicense.  Each licensee is addressed as \"you\".  \"Licensees\" and\n\"recipients\" may be individuals or organizations.\n\n  To \"modify\" a work means to copy from or adapt all or part of the work\nin a fashion requiring copyright permission, other than the making of an\nexact copy.  The resulting work is called a \"modified version\" of the\nearlier work or a work \"based on\" the earlier work.\n\n  A \"covered work\" means either the unmodified Program or a work based\non the Program.\n\n  To \"propagate\" a work means to do anything with it that, without\npermission, would make you directly or secondarily liable for\ninfringement under applicable copyright law, except executing it on a\ncomputer or modifying a private copy.  Propagation includes copying,\ndistribution (with or without modification), making available to the\npublic, and in some countries other activities as well.\n\n  To \"convey\" a work means any kind of propagation that enables other\nparties to make or receive copies.  Mere interaction with a user through\na computer network, with no transfer of a copy, is not conveying.\n\n  An interactive user interface displays \"Appropriate Legal Notices\"\nto the extent that it includes a convenient and prominently visible\nfeature that (1) displays an appropriate copyright notice, and (2)\ntells the user that there is no warranty for the work (except to the\nextent that warranties are provided), that licensees may convey the\nwork under this License, and how to view a copy of this License.  If\nthe interface presents a list of user commands or options, such as a\nmenu, a prominent item in the list meets this criterion.\n\n  1. Source Code.\n\n  The \"source code\" for a work means the preferred form of the work\nfor making modifications to it.  \"Object code\" means any non-source\nform of a work.\n\n  A \"Standard Interface\" means an interface that either is an official\nstandard defined by a recognized standards body, or, in the case of\ninterfaces specified for a particular programming language, one that\nis widely used among developers working in that language.\n\n  The \"System Libraries\" of an executable work include anything, other\nthan the work as a whole, that (a) is included in the normal form of\npackaging a Major Component, but which is not part of that Major\nComponent, and (b) serves only to enable use of the work with that\nMajor Component, or to implement a Standard Interface for which an\nimplementation is available to the public in source code form.  A\n\"Major Component\", in this context, means a major essential component\n(kernel, window system, and so on) of the specific operating system\n(if any) on which the executable work runs, or a compiler used to\nproduce the work, or an object code interpreter used to run it.\n\n  The \"Corresponding Source\" for a work in object code form means all\nthe source code needed to generate, install, and (for an executable\nwork) run the object code and to modify the work, including scripts to\ncontrol those activities.  However, it does not include the work's\nSystem Libraries, or general-purpose tools or generally available free\nprograms which are used unmodified in performing those activities but\nwhich are not part of the work.  For example, Corresponding Source\nincludes interface definition files associated with source files for\nthe work, and the source code for shared libraries and dynamically\nlinked subprograms that the work is specifically designed to require,\nsuch as by intimate data communication or control flow between those\nsubprograms and other parts of the work.\n\n  The Corresponding Source need not include anything that users\ncan regenerate automatically from other parts of the Corresponding\nSource.\n\n  The Corresponding Source for a work in source code form is that\nsame work.\n\n  2. Basic Permissions.\n\n  All rights granted under this License are granted for the term of\ncopyright on the Program, and are irrevocable provided the stated\nconditions are met.  This License explicitly affirms your unlimited\npermission to run the unmodified Program.  The output from running a\ncovered work is covered by this License only if the output, given its\ncontent, constitutes a covered work.  This License acknowledges your\nrights of fair use or other equivalent, as provided by copyright law.\n\n  You may make, run and propagate covered works that you do not\nconvey, without conditions so long as your license otherwise remains\nin force.  You may convey covered works to others for the sole purpose\nof having them make modifications exclusively for you, or provide you\nwith facilities for running those works, provided that you comply with\nthe terms of this License in conveying all material for which you do\nnot control copyright.  Those thus making or running the covered works\nfor you must do so exclusively on your behalf, under your direction\nand control, on terms that prohibit them from making any copies of\nyour copyrighted material outside their relationship with you.\n\n  Conveying under any other circumstances is permitted solely under\nthe conditions stated below.  Sublicensing is not allowed; section 10\nmakes it unnecessary.\n\n  3. Protecting Users' Legal Rights From Anti-Circumvention Law.\n\n  No covered work shall be deemed part of an effective technological\nmeasure under any applicable law fulfilling obligations under article\n11 of the WIPO copyright treaty adopted on 20 December 1996, or\nsimilar laws prohibiting or restricting circumvention of such\nmeasures.\n\n  When you convey a covered work, you waive any legal power to forbid\ncircumvention of technological measures to the extent such circumvention\nis effected by exercising rights under this License with respect to\nthe covered work, and you disclaim any intention to limit operation or\nmodification of the work as a means of enforcing, against the work's\nusers, your or third parties' legal rights to forbid circumvention of\ntechnological measures.\n\n  4. Conveying Verbatim Copies.\n\n  You may convey verbatim copies of the Program's source code as you\nreceive it, in any medium, provided that you conspicuously and\nappropriately publish on each copy an appropriate copyright notice;\nkeep intact all notices stating that this License and any\nnon-permissive terms added in accord with section 7 apply to the code;\nkeep intact all notices of the absence of any warranty; and give all\nrecipients a copy of this License along with the Program.\n\n  You may charge any price or no price for each copy that you convey,\nand you may offer support or warranty protection for a fee.\n\n  5. Conveying Modified Source Versions.\n\n  You may convey a work based on the Program, or the modifications to\nproduce it from the Program, in the form of source code under the\nterms of section 4, provided that you also meet all of these conditions:\n\n    a) The work must carry prominent notices stating that you modified\n    it, and giving a relevant date.\n\n    b) The work must carry prominent notices stating that it is\n    released under this License and any conditions added under section\n    7.  This requirement modifies the requirement in section 4 to\n    \"keep intact all notices\".\n\n    c) You must license the entire work, as a whole, under this\n    License to anyone who comes into possession of a copy.  This\n    License will therefore apply, along with any applicable section 7\n    additional terms, to the whole of the work, and all its parts,\n    regardless of how they are packaged.  This License gives no\n    permission to license the work in any other way, but it does not\n    invalidate such permission if you have separately received it.\n\n    d) If the work has interactive user interfaces, each must display\n    Appropriate Legal Notices; however, if the Program has interactive\n    interfaces that do not display Appropriate Legal Notices, your\n    work need not make them do so.\n\n  A compilation of a covered work with other separate and independent\nworks, which are not by their nature extensions of the covered work,\nand which are not combined with it such as to form a larger program,\nin or on a volume of a storage or distribution medium, is called an\n\"aggregate\" if the compilation and its resulting copyright are not\nused to limit the access or legal rights of the compilation's users\nbeyond what the individual works permit.  Inclusion of a covered work\nin an aggregate does not cause this License to apply to the other\nparts of the aggregate.\n\n  6. Conveying Non-Source Forms.\n\n  You may convey a covered work in object code form under the terms\nof sections 4 and 5, provided that you also convey the\nmachine-readable Corresponding Source under the terms of this License,\nin one of these ways:\n\n    a) Convey the object code in, or embodied in, a physical product\n    (including a physical distribution medium), accompanied by the\n    Corresponding Source fixed on a durable physical medium\n    customarily used for software interchange.\n\n    b) Convey the object code in, or embodied in, a physical product\n    (including a physical distribution medium), accompanied by a\n    written offer, valid for at least three years and valid for as\n    long as you offer spare parts or customer support for that product\n    model, to give anyone who possesses the object code either (1) a\n    copy of the Corresponding Source for all the software in the\n    product that is covered by this License, on a durable physical\n    medium customarily used for software interchange, for a price no\n    more than your reasonable cost of physically performing this\n    conveying of source, or (2) access to copy the\n    Corresponding Source from a network server at no charge.\n\n    c) Convey individual copies of the object code with a copy of the\n    written offer to provide the Corresponding Source.  This\n    alternative is allowed only occasionally and noncommercially, and\n    only if you received the object code with such an offer, in accord\n    with subsection 6b.\n\n    d) Convey the object code by offering access from a designated\n    place (gratis or for a charge), and offer equivalent access to the\n    Corresponding Source in the same way through the same place at no\n    further charge.  You need not require recipients to copy the\n    Corresponding Source along with the object code.  If the place to\n    copy the object code is a network server, the Corresponding Source\n    may be on a different server (operated by you or a third party)\n    that supports equivalent copying facilities, provided you maintain\n    clear directions next to the object code saying where to find the\n    Corresponding Source.  Regardless of what server hosts the\n    Corresponding Source, you remain obligated to ensure that it is\n    available for as long as needed to satisfy these requirements.\n\n    e) Convey the object code using peer-to-peer transmission, provided\n    you inform other peers where the object code and Corresponding\n    Source of the work are being offered to the general public at no\n    charge under subsection 6d.\n\n  A separable portion of the object code, whose source code is excluded\nfrom the Corresponding Source as a System Library, need not be\nincluded in conveying the object code work.\n\n  A \"User Product\" is either (1) a \"consumer product\", which means any\ntangible personal property which is normally used for personal, family,\nor household purposes, or (2) anything designed or sold for incorporation\ninto a dwelling.  In determining whether a product is a consumer product,\ndoubtful cases shall be resolved in favor of coverage.  For a particular\nproduct received by a particular user, \"normally used\" refers to a\ntypical or common use of that class of product, regardless of the status\nof the particular user or of the way in which the particular user\nactually uses, or expects or is expected to use, the product.  A product\nis a consumer product regardless of whether the product has substantial\ncommercial, industrial or non-consumer uses, unless such uses represent\nthe only significant mode of use of the product.\n\n  \"Installation Information\" for a User Product means any methods,\nprocedures, authorization keys, or other information required to install\nand execute modified versions of a covered work in that User Product from\na modified version of its Corresponding Source.  The information must\nsuffice to ensure that the continued functioning of the modified object\ncode is in no case prevented or interfered with solely because\nmodification has been made.\n\n  If you convey an object code work under this section in, or with, or\nspecifically for use in, a User Product, and the conveying occurs as\npart of a transaction in which the right of possession and use of the\nUser Product is transferred to the recipient in perpetuity or for a\nfixed term (regardless of how the transaction is characterized), the\nCorresponding Source conveyed under this section must be accompanied\nby the Installation Information.  But this requirement does not apply\nif neither you nor any third party retains the ability to install\nmodified object code on the User Product (for example, the work has\nbeen installed in ROM).\n\n  The requirement to provide Installation Information does not include a\nrequirement to continue to provide support service, warranty, or updates\nfor a work that has been modified or installed by the recipient, or for\nthe User Product in which it has been modified or installed.  Access to a\nnetwork may be denied when the modification itself materially and\nadversely affects the operation of the network or violates the rules and\nprotocols for communication across the network.\n\n  Corresponding Source conveyed, and Installation Information provided,\nin accord with this section must be in a format that is publicly\ndocumented (and with an implementation available to the public in\nsource code form), and must require no special password or key for\nunpacking, reading or copying.\n\n  7. Additional Terms.\n\n  \"Additional permissions\" are terms that supplement the terms of this\nLicense by making exceptions from one or more of its conditions.\nAdditional permissions that are applicable to the entire Program shall\nbe treated as though they were included in this License, to the extent\nthat they are valid under applicable law.  If additional permissions\napply only to part of the Program, that part may be used separately\nunder those permissions, but the entire Program remains governed by\nthis License without regard to the additional permissions.\n\n  When you convey a copy of a covered work, you may at your option\nremove any additional permissions from that copy, or from any part of\nit.  (Additional permissions may be written to require their own\nremoval in certain cases when you modify the work.)  You may place\nadditional permissions on material, added by you to a covered work,\nfor which you have or can give appropriate copyright permission.\n\n  Notwithstanding any other provision of this License, for material you\nadd to a covered work, you may (if authorized by the copyright holders of\nthat material) supplement the terms of this License with terms:\n\n    a) Disclaiming warranty or limiting liability differently from the\n    terms of sections 15 and 16 of this License; or\n\n    b) Requiring preservation of specified reasonable legal notices or\n    author attributions in that material or in the Appropriate Legal\n    Notices displayed by works containing it; or\n\n    c) Prohibiting misrepresentation of the origin of that material, or\n    requiring that modified versions of such material be marked in\n    reasonable ways as different from the original version; or\n\n    d) Limiting the use for publicity purposes of names of licensors or\n    authors of the material; or\n\n    e) Declining to grant rights under trademark law for use of some\n    trade names, trademarks, or service marks; or\n\n    f) Requiring indemnification of licensors and authors of that\n    material by anyone who conveys the material (or modified versions of\n    it) with contractual assumptions of liability to the recipient, for\n    any liability that these contractual assumptions directly impose on\n    those licensors and authors.\n\n  All other non-permissive additional terms are considered \"further\nrestrictions\" within the meaning of section 10.  If the Program as you\nreceived it, or any part of it, contains a notice stating that it is\ngoverned by this License along with a term that is a further\nrestriction, you may remove that term.  If a license document contains\na further restriction but permits relicensing or conveying under this\nLicense, you may add to a covered work material governed by the terms\nof that license document, provided that the further restriction does\nnot survive such relicensing or conveying.\n\n  If you add terms to a covered work in accord with this section, you\nmust place, in the relevant source files, a statement of the\nadditional terms that apply to those files, or a notice indicating\nwhere to find the applicable terms.\n\n  Additional terms, permissive or non-permissive, may be stated in the\nform of a separately written license, or stated as exceptions;\nthe above requirements apply either way.\n\n  8. Termination.\n\n  You may not propagate or modify a covered work except as expressly\nprovided under this License.  Any attempt otherwise to propagate or\nmodify it is void, and will automatically terminate your rights under\nthis License (including any patent licenses granted under the third\nparagraph of section 11).\n\n  However, if you cease all violation of this License, then your\nlicense from a particular copyright holder is reinstated (a)\nprovisionally, unless and until the copyright holder explicitly and\nfinally terminates your license, and (b) permanently, if the copyright\nholder fails to notify you of the violation by some reasonable means\nprior to 60 days after the cessation.\n\n  Moreover, your license from a particular copyright holder is\nreinstated permanently if the copyright holder notifies you of the\nviolation by some reasonable means, this is the first time you have\nreceived notice of violation of this License (for any work) from that\ncopyright holder, and you cure the violation prior to 30 days after\nyour receipt of the notice.\n\n  Termination of your rights under this section does not terminate the\nlicenses of parties who have received copies or rights from you under\nthis License.  If your rights have been terminated and not permanently\nreinstated, you do not qualify to receive new licenses for the same\nmaterial under section 10.\n\n  9. Acceptance Not Required for Having Copies.\n\n  You are not required to accept this License in order to receive or\nrun a copy of the Program.  Ancillary propagation of a covered work\noccurring solely as a consequence of using peer-to-peer transmission\nto receive a copy likewise does not require acceptance.  However,\nnothing other than this License grants you permission to propagate or\nmodify any covered work.  These actions infringe copyright if you do\nnot accept this License.  Therefore, by modifying or propagating a\ncovered work, you indicate your acceptance of this License to do so.\n\n  10. Automatic Licensing of Downstream Recipients.\n\n  Each time you convey a covered work, the recipient automatically\nreceives a license from the original licensors, to run, modify and\npropagate that work, subject to this License.  You are not responsible\nfor enforcing compliance by third parties with this License.\n\n  An \"entity transaction\" is a transaction transferring control of an\norganization, or substantially all assets of one, or subdividing an\norganization, or merging organizations.  If propagation of a covered\nwork results from an entity transaction, each party to that\ntransaction who receives a copy of the work also receives whatever\nlicenses to the work the party's predecessor in interest had or could\ngive under the previous paragraph, plus a right to possession of the\nCorresponding Source of the work from the predecessor in interest, if\nthe predecessor has it or can get it with reasonable efforts.\n\n  You may not impose any further restrictions on the exercise of the\nrights granted or affirmed under this License.  For example, you may\nnot impose a license fee, royalty, or other charge for exercise of\nrights granted under this License, and you may not initiate litigation\n(including a cross-claim or counterclaim in a lawsuit) alleging that\nany patent claim is infringed by making, using, selling, offering for\nsale, or importing the Program or any portion of it.\n\n  11. Patents.\n\n  A \"contributor\" is a copyright holder who authorizes use under this\nLicense of the Program or a work on which the Program is based.  The\nwork thus licensed is called the contributor's \"contributor version\".\n\n  A contributor's \"essential patent claims\" are all patent claims\nowned or controlled by the contributor, whether already acquired or\nhereafter acquired, that would be infringed by some manner, permitted\nby this License, of making, using, or selling its contributor version,\nbut do not include claims that would be infringed only as a\nconsequence of further modification of the contributor version.  For\npurposes of this definition, \"control\" includes the right to grant\npatent sublicenses in a manner consistent with the requirements of\nthis License.\n\n  Each contributor grants you a non-exclusive, worldwide, royalty-free\npatent license under the contributor's essential patent claims, to\nmake, use, sell, offer for sale, import and otherwise run, modify and\npropagate the contents of its contributor version.\n\n  In the following three paragraphs, a \"patent license\" is any express\nagreement or commitment, however denominated, not to enforce a patent\n(such as an express permission to practice a patent or covenant not to\nsue for patent infringement).  To \"grant\" such a patent license to a\nparty means to make such an agreement or commitment not to enforce a\npatent against the party.\n\n  If you convey a covered work, knowingly relying on a patent license,\nand the Corresponding Source of the work is not available for anyone\nto copy, free of charge and under the terms of this License, through a\npublicly available network server or other readily accessible means,\nthen you must either (1) cause the Corresponding Source to be so\navailable, or (2) arrange to deprive yourself of the benefit of the\npatent license for this particular work, or (3) arrange, in a manner\nconsistent with the requirements of this License, to extend the patent\nlicense to downstream recipients.  \"Knowingly relying\" means you have\nactual knowledge that, but for the patent license, your conveying the\ncovered work in a country, or your recipient's use of the covered work\nin a country, would infringe one or more identifiable patents in that\ncountry that you have reason to believe are valid.\n\n  If, pursuant to or in connection with a single transaction or\narrangement, you convey, or propagate by procuring conveyance of, a\ncovered work, and grant a patent license to some of the parties\nreceiving the covered work authorizing them to use, propagate, modify\nor convey a specific copy of the covered work, then the patent license\nyou grant is automatically extended to all recipients of the covered\nwork and works based on it.\n\n  A patent license is \"discriminatory\" if it does not include within\nthe scope of its coverage, prohibits the exercise of, or is\nconditioned on the non-exercise of one or more of the rights that are\nspecifically granted under this License.  You may not convey a covered\nwork if you are a party to an arrangement with a third party that is\nin the business of distributing software, under which you make payment\nto the third party based on the extent of your activity of conveying\nthe work, and under which the third party grants, to any of the\nparties who would receive the covered work from you, a discriminatory\npatent license (a) in connection with copies of the covered work\nconveyed by you (or copies made from those copies), or (b) primarily\nfor and in connection with specific products or compilations that\ncontain the covered work, unless you entered into that arrangement,\nor that patent license was granted, prior to 28 March 2007.\n\n  Nothing in this License shall be construed as excluding or limiting\nany implied license or other defenses to infringement that may\notherwise be available to you under applicable patent law.\n\n  12. No Surrender of Others' Freedom.\n\n  If conditions are imposed on you (whether by court order, agreement or\notherwise) that contradict the conditions of this License, they do not\nexcuse you from the conditions of this License.  If you cannot convey a\ncovered work so as to satisfy simultaneously your obligations under this\nLicense and any other pertinent obligations, then as a consequence you may\nnot convey it at all.  For example, if you agree to terms that obligate you\nto collect a royalty for further conveying from those to whom you convey\nthe Program, the only way you could satisfy both those terms and this\nLicense would be to refrain entirely from conveying the Program.\n\n  13. Remote Network Interaction; Use with the GNU General Public License.\n\n  Notwithstanding any other provision of this License, if you modify the\nProgram, your modified version must prominently offer all users\ninteracting with it remotely through a computer network (if your version\nsupports such interaction) an opportunity to receive the Corresponding\nSource of your version by providing access to the Corresponding Source\nfrom a network server at no charge, through some standard or customary\nmeans of facilitating copying of software.  This Corresponding Source\nshall include the Corresponding Source for any work covered by version 3\nof the GNU General Public License that is incorporated pursuant to the\nfollowing paragraph.\n\n  Notwithstanding any other provision of this License, you have\npermission to link or combine any covered work with a work licensed\nunder version 3 of the GNU General Public License into a single\ncombined work, and to convey the resulting work.  The terms of this\nLicense will continue to apply to the part which is the covered work,\nbut the work with which it is combined will remain governed by version\n3 of the GNU General Public License.\n\n  14. Revised Versions of this License.\n\n  The Free Software Foundation may publish revised and/or new versions of\nthe GNU Affero General Public License from time to time.  Such new versions\nwill be similar in spirit to the present version, but may differ in detail to\naddress new problems or concerns.\n\n  Each version is given a distinguishing version number.  If the\nProgram specifies that a certain numbered version of the GNU Affero General\nPublic License \"or any later version\" applies to it, you have the\noption of following the terms and conditions either of that numbered\nversion or of any later version published by the Free Software\nFoundation.  If the Program does not specify a version number of the\nGNU Affero General Public License, you may choose any version ever published\nby the Free Software Foundation.\n\n  If the Program specifies that a proxy can decide which future\nversions of the GNU Affero General Public License can be used, that proxy's\npublic statement of acceptance of a version permanently authorizes you\nto choose that version for the Program.\n\n  Later license versions may give you additional or different\npermissions.  However, no additional obligations are imposed on any\nauthor or copyright holder as a result of your choosing to follow a\nlater version.\n\n  15. Disclaimer of Warranty.\n\n  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY\nAPPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT\nHOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM \"AS IS\" WITHOUT WARRANTY\nOF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,\nTHE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\nPURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM\nIS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF\nALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n\n  16. Limitation of Liability.\n\n  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING\nWILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS\nTHE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY\nGENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE\nUSE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF\nDATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD\nPARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),\nEVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF\nSUCH DAMAGES.\n\n  17. Interpretation of Sections 15 and 16.\n\n  If the disclaimer of warranty and limitation of liability provided\nabove cannot be given local legal effect according to their terms,\nreviewing courts shall apply local law that most closely approximates\nan absolute waiver of all civil liability in connection with the\nProgram, unless a warranty or assumption of liability accompanies a\ncopy of the Program in return for a fee.\n\n                     END OF TERMS AND CONDITIONS\n\n            How to Apply These Terms to Your New Programs\n\n  If you develop a new program, and you want it to be of the greatest\npossible use to the public, the best way to achieve this is to make it\nfree software which everyone can redistribute and change under these terms.\n\n  To do so, attach the following notices to the program.  It is safest\nto attach them to the start of each source file to most effectively\nstate the exclusion of warranty; and each file should have at least\nthe \"copyright\" line and a pointer to where the full notice is found.\n\n    <one line to give the program's name and a brief idea of what it does.>\n    Copyright (C) <year>  <name of author>\n\n    This program is free software: you can redistribute it and/or modify\n    it under the terms of the GNU Affero General Public License as published by\n    the Free Software Foundation, either version 3 of the License, or\n    (at your option) any later version.\n\n    This program is distributed in the hope that it will be useful,\n    but WITHOUT ANY WARRANTY; without even the implied warranty of\n    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n    GNU Affero General Public License for more details.\n\n    You should have received a copy of the GNU Affero General Public License\n    along with this program.  If not, see <https://www.gnu.org/licenses/>.\n\nAlso add information on how to contact you by electronic and paper mail.\n\n  If your software can interact with users remotely through a computer\nnetwork, you should also make sure that it provides a way for users to\nget its source.  For example, if your program is a web application, its\ninterface could display a \"Source\" link that leads users to an archive\nof the code.  There are many ways you could offer source, and different\nsolutions will be better for different programs; see section 13 for the\nspecific requirements.\n\n  You should also get your employer (if you work as a programmer) or school,\nif any, to sign a \"copyright disclaimer\" for the program, if necessary.\nFor more information on this, and how to apply and follow the GNU AGPL, see\n<https://www.gnu.org/licenses/>.\n"
  },
  {
    "path": "README.md",
    "content": "# X's Recommendation Algorithm\n\nX's Recommendation Algorithm is a set of services and jobs that are responsible for serving feeds of posts and other content across all X product surfaces (e.g. For You Timeline, Search, Explore, Notifications). For an introduction to how the algorithm works, please refer to our [engineering blog](https://blog.x.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm).\n\n## Architecture\n\nProduct surfaces at X are built on a shared set of data, models, and software frameworks. The shared components included in this repository are listed below:\n\n| Type | Component | Description |\n|------------|------------|------------|\n| Data | [tweetypie](tweetypie/server/README.md) | Core service that handles the reading and writing of post data. |\n|      | [unified-user-actions](unified_user_actions/README.md) | Real-time stream of user actions on X. |\n|      | [user-signal-service](user-signal-service/README.md) | Centralized platform to retrieve explicit (e.g. likes, replies) and implicit (e.g. profile visits, tweet clicks) user signals. |\n| Model | [SimClusters](src/scala/com/twitter/simclusters_v2/README.md) | Community detection and sparse embeddings into those communities. |\n|       | [TwHIN](https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/README.md) | Dense knowledge graph embeddings for Users and Posts. |\n|       | [trust-and-safety-models](trust_and_safety_models/README.md) | Models for detecting NSFW or abusive content. |\n|       | [real-graph](src/scala/com/twitter/interaction_graph/README.md) | Model to predict the likelihood of an X User interacting with another User. |\n|       | [tweepcred](src/scala/com/twitter/graph/batch/job/tweepcred/README) | Page-Rank algorithm for calculating X User reputation. |\n|       | [recos-injector](recos-injector/README.md) | Streaming event processor for building input streams for [GraphJet](https://github.com/twitter/GraphJet) based services. |\n|       | [graph-feature-service](graph-feature-service/README.md) | Serves graph features for a directed pair of users (e.g. how many of User A's following liked posts from User B). |\n|       | [topic-social-proof](topic-social-proof/README.md) | Identifies topics related to individual posts. |\n|       | [representation-scorer](representation-scorer/README.md) | Compute scores between pairs of entities (Users, Posts, etc.) using embedding similarity. |\n| Software framework | [navi](navi/README.md) | High performance, machine learning model serving written in Rust. |\n|                    | [product-mixer](product-mixer/README.md) | Software framework for building feeds of content. |\n|                    | [timelines-aggregation-framework](timelines/data_processing/ml_util/aggregation_framework/README.md) | Framework for generating aggregate features in batch or real time. |\n|                    | [representation-manager](representation-manager/README.md) | Service to retrieve embeddings (i.e. SimClusers and TwHIN). |\n|                    | [twml](twml/README.md) | Legacy machine learning framework built on TensorFlow v1. |\n\nThe product surfaces currently included in this repository are the For You Timeline and Recommended Notifications.\n\n### For You Timeline\n\nThe diagram below illustrates how major services and jobs interconnect to construct a For You Timeline.\n\n![](docs/system-diagram.png)\n\nThe core components of the For You Timeline included in this repository are listed below:\n\n| Type | Component | Description |\n|------------|------------|------------|\n| Candidate Source | [search-index](src/java/com/twitter/search/README.md) | Find and rank In-Network posts. ~50% of posts come from this candidate source. |\n|                  | [tweet-mixer](tweet-mixer) | Coordination layer for fetching Out-of-Network tweet candidates from underlying compute services. |\n|                  | [user-tweet-entity-graph](src/scala/com/twitter/recos/user_tweet_entity_graph/README.md) (UTEG)| Maintains an in memory User to Post interaction graph, and finds candidates based on traversals of this graph. This is built on the [GraphJet](https://github.com/twitter/GraphJet) framework. Several other GraphJet based features and candidate sources are located [here](src/scala/com/twitter/recos). |\n|                  | [follow-recommendation-service](follow-recommendations-service/README.md) (FRS)| Provides Users with recommendations for accounts to follow, and posts from those accounts. |\n| Ranking | [light-ranker](src/python/twitter/deepbird/projects/timelines/scripts/models/earlybird/README.md) | Light Ranker model used by search index (Earlybird) to rank posts. |\n|         | [heavy-ranker](https://github.com/twitter/the-algorithm-ml/blob/main/projects/home/recap/README.md) | Neural network for ranking candidate posts. One of the main signals used to select timeline posts post candidate sourcing. |\n| Post mixing & filtering | [home-mixer](home-mixer/README.md) | Main service used to construct and serve the Home Timeline. Built on [product-mixer](product-mixer/README.md). |\n|                          | [visibility-filters](visibilitylib/README.md) | Responsible for filtering X content to support legal compliance, improve product quality, increase user trust, protect revenue through the use of hard-filtering, visible product treatments, and coarse-grained downranking. |\n|                          | [timelineranker](timelineranker/README.md) | Legacy service which provides relevance-scored posts from the Earlybird Search Index and UTEG service. |\n\n### Recommended Notifications\n\nThe core components of Recommended Notifications included in this repository are listed below:\n\n| Type | Component | Description |\n|------------|------------|------------|\n| Service | [pushservice](pushservice/README.md) | Main recommendation service at X used to surface recommendations to our users via notifications.\n| Ranking | [pushservice-light-ranker](pushservice/src/main/python/models/light_ranking/README.md) | Light Ranker model used by pushservice to rank posts. Bridges candidate generation and heavy ranking by pre-selecting highly-relevant candidates from the initial huge candidate pool. |\n|         | [pushservice-heavy-ranker](pushservice/src/main/python/models/heavy_ranking/README.md) | Multi-task learning model to predict the probabilities that the target users will open and engage with the sent notifications. |\n\n## Build and test code\n\nWe include Bazel BUILD files for most components, but not a top-level BUILD or WORKSPACE file. We plan to add a more complete build and test system in the future.\n\n## Contributing\n\nWe invite the community to submit GitHub issues and pull requests for suggestions on improving the recommendation algorithm. We are working on tools to manage these suggestions and sync changes to our internal repository. Any security concerns or issues should be routed to our official [bug bounty program](https://hackerone.com/x) through HackerOne. We hope to benefit from the collective intelligence and expertise of the global community in helping us identify issues and suggest improvements, ultimately leading to a better X.\n\nRead our blog on the open source initiative [here](https://blog.x.com/en_us/topics/company/2023/a-new-era-of-transparency-for-twitter).\n"
  },
  {
    "path": "RETREIVAL_SIGNALS.md",
    "content": "# Signals for Candidate Sources\n\n## Overview\n\nThe candidate sourcing stage within the Twitter Recommendation algorithm serves to significantly narrow down the item size from approximately 1 billion to just a few thousand. This process utilizes Twitter user behavior as the primary input for the algorithm. This document comprehensively enumerates all the signals during the candidate sourcing phase.\n\n| Signals               |  Description                                                          |\n| :-------------------- | :-------------------------------------------------------------------- |\n| Author Follow         | The accounts which user explicit follows.                             |\n| Author Unfollow       | The accounts which user recently unfollows.                           |\n| Author Mute           | The accounts which user have muted.                                   |\n| Author Block          | The accounts which user have blocked                                  |\n| Tweet Favorite        | The tweets which user clicked the like botton.                        | \n| Tweet Unfavorite      | The tweets which user clicked the unlike botton.                      |       \n| Retweet               | The tweets which user retweeted                                       |\n| Quote Tweet           | The tweets which user retweeted with comments.                        |\n| Tweet Reply           | The tweets which user replied.                                        |\n| Tweet Share           | The tweets which user clicked the share botton.                       |\n| Tweet Bookmark        | The tweets which user clicked the bookmark botton.                    |\n| Tweet Click           | The tweets which user clicked and viewed the tweet detail page.       |\n| Tweet Video Watch     | The video tweets which user watched certain seconds or percentage.    |\n| Tweet Don't like      | The tweets which user clicked \"Not interested in this tweet\" botton.  |\n| Tweet Report          | The tweets which user clicked \"Report Tweet\" botton.                  |\n| Notification Open     | The push notification tweets which user opened.                       |\n| Ntab click            | The tweets which user click on the Notifications page.                |               \n| User AddressBook      | The author accounts identifiers of the user's addressbook.            | \n\n## Usage Details\n\nTwitter uses these user signals as training labels and/or ML features in the each candidate sourcing algorithms. The following tables shows how they are used in the each components.\n\n| Signals               | USS                | SimClusters        |  TwHin             |   UTEG             | FRS                |  Light Ranking     |\n| :-------------------- | :----------------- | :----------------- | :----------------- | :----------------- | :----------------- | :----------------- | \n| Author Follow         | Features           | Features / Labels  | Features / Labels  | Features           | Features / Labels  | N/A                |\n| Author Unfollow       | Features           | N/A                | N/A                | N/A                | N/A                | N/A                |\n| Author Mute           | Features           | N/A                | N/A                | N/A                | Features           | N/A                |\n| Author Block          | Features           | N/A                | N/A                | N/A                | Features           | N/A                |\n| Tweet Favorite        | Features           | Features           | Features / Labels  | Features           | Features / Labels  | Features / Labels  |\n| Tweet Unfavorite      | Features           | Features           | N/A                | N/A                | N/A                | N/A                |       \n| Retweet               | Features           | N/A                | Features / Labels  | Features           | Features / Labels  | Features / Labels  |\n| Quote Tweet           | Features           | N/A                | Features / Labels  | Features           | Features / Labels  | Features / Labels  |\n| Tweet Reply           | Features           | N/A                | Features           | Features           | Features / Labels  | Features           |\n| Tweet Share           | Features           | N/A                | N/A                | N/A                | Features           | N/A                |\n| Tweet Bookmark        | Features           | N/A                | N/A                | N/A                | N/A                | N/A                |\n| Tweet Click           | Features           | N/A                | N/A                | N/A                | Features           | Labels             |\n| Tweet Video Watch     | Features           | Features           | N/A                | N/A                | N/A                | Labels             |\n| Tweet Don't like      | Features           | N/A                | N/A                | N/A                | N/A                | N/A                |\n| Tweet Report          | Features           | N/A                | N/A                | N/A                | N/A                | N/A                |\n| Notification Open     | Features           | Features           | Features           | N/A                | Features           | N/A                |                       \n| Ntab click            | Features           | Features           | Features           | N/A                | Features           | N/A                |\n| User AddressBook      | N/A                | N/A                | N/A                | N/A                | Features           | N/A                |"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/BUILD",
    "content": "target(\n    name = \"faiss\",\n    dependencies = [\n        \"ann/src/main/java/com/twitter/ann/faiss/swig:swig-artifactory\",\n    ],\n)\n\njava_library(\n    name = \"swig-native-utils\",\n    sources = [\"*.java\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [],\n)\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/NativeUtils.java",
    "content": "package com.twitter.ann.faiss;\n\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.nio.file.Files;\nimport java.nio.file.StandardCopyOption;\nimport java.util.Locale;\n\npublic final class NativeUtils {\n\n  private static final int MIN_PREFIX_LENGTH = 3;\n  public static final String NATIVE_FOLDER_PATH_PREFIX = \"nativeutils\";\n\n  public static File temporaryDir;\n\n  private NativeUtils() {\n  }\n\n  private static File unpackLibraryFromJarInternal(String path) throws IOException {\n    if (null == path || !path.startsWith(\"/\")) {\n      throw new IllegalArgumentException(\"The path has to be absolute (start with '/').\");\n    }\n\n    String[] parts = path.split(\"/\");\n    String filename = (parts.length > 1) ? parts[parts.length - 1] : null;\n\n    if (filename == null || filename.length() < MIN_PREFIX_LENGTH) {\n      throw new IllegalArgumentException(\"The filename has to be at least 3 characters long.\");\n    }\n\n    if (temporaryDir == null) {\n      temporaryDir = createTempDirectory(NATIVE_FOLDER_PATH_PREFIX);\n      temporaryDir.deleteOnExit();\n    }\n\n    File temp = new File(temporaryDir, filename);\n\n    try (InputStream is = NativeUtils.class.getResourceAsStream(path)) {\n      Files.copy(is, temp.toPath(), StandardCopyOption.REPLACE_EXISTING);\n    } catch (IOException e) {\n      temp.delete();\n      throw e;\n    } catch (NullPointerException e) {\n      temp.delete();\n      throw new FileNotFoundException(\"File \" + path + \" was not found inside JAR.\");\n    }\n\n    return temp;\n  }\n\n  /**\n   * Unpack library from JAR into temporary path\n   *\n   * @param path The path of file inside JAR as absolute path (beginning with\n   *             '/'), e.g. /package/File.ext\n   * @throws IOException              If temporary file creation or read/write\n   *                                  operation fails\n   * @throws IllegalArgumentException If source file (param path) does not exist\n   * @throws IllegalArgumentException If the path is not absolute or if the\n   *                                  filename is shorter than three characters\n   *                                  (restriction of\n   *                                  {@link File#createTempFile(java.lang.String, java.lang.String)}).\n   * @throws FileNotFoundException    If the file could not be found inside the\n   *                                  JAR.\n   */\n  public static void unpackLibraryFromJar(String path) throws IOException {\n    unpackLibraryFromJarInternal(path);\n  }\n\n  /**\n   * Loads library from current JAR archive\n   * <p>\n   * The file from JAR is copied into system temporary directory and then loaded.\n   * The temporary file is deleted after\n   * exiting.\n   * Method uses String as filename because the pathname is \"abstract\", not\n   * system-dependent.\n   *\n   * @param path The path of file inside JAR as absolute path (beginning with\n   *             '/'), e.g. /package/File.ext\n   * @throws IOException              If temporary file creation or read/write\n   *                                  operation fails\n   * @throws IllegalArgumentException If source file (param path) does not exist\n   * @throws IllegalArgumentException If the path is not absolute or if the\n   *                                  filename is shorter than three characters\n   *                                  (restriction of\n   *                                  {@link File#createTempFile(java.lang.String, java.lang.String)}).\n   * @throws FileNotFoundException    If the file could not be found inside the\n   *                                  JAR.\n   */\n  public static void loadLibraryFromJar(String path) throws IOException {\n    File temp = unpackLibraryFromJarInternal(path);\n\n    try (InputStream is = NativeUtils.class.getResourceAsStream(path)) {\n      Files.copy(is, temp.toPath(), StandardCopyOption.REPLACE_EXISTING);\n    } catch (IOException e) {\n      temp.delete();\n      throw e;\n    } catch (NullPointerException e) {\n      temp.delete();\n      throw new FileNotFoundException(\"File \" + path + \" was not found inside JAR.\");\n    }\n\n    try {\n      System.load(temp.getAbsolutePath());\n    } finally {\n      temp.deleteOnExit();\n    }\n  }\n\n  private static File createTempDirectory(String prefix) throws IOException {\n    String tempDir = System.getProperty(\"java.io.tmpdir\");\n    File generatedDir = new File(tempDir, prefix + System.nanoTime());\n\n    if (!generatedDir.mkdir()) {\n      throw new IOException(\"Failed to create temp directory \" + generatedDir.getName());\n    }\n\n    return generatedDir;\n  }\n\n  public enum OSType {\n    Windows, MacOS, Linux, Other\n  }\n\n  protected static OSType detectedOS;\n\n  /**\n   * detect the operating system from the os.name System property and cache\n   * the result\n   *\n   * @returns - the operating system detected\n   */\n  public static OSType getOperatingSystemType() {\n    if (detectedOS == null) {\n      String osname = System.getProperty(\"os.name\", \"generic\").toLowerCase(Locale.ENGLISH);\n      if ((osname.contains(\"mac\")) || (osname.contains(\"darwin\"))) {\n        detectedOS = OSType.MacOS;\n      } else if (osname.contains(\"win\")) {\n        detectedOS = OSType.Windows;\n      } else if (osname.contains(\"nux\")) {\n        detectedOS = OSType.Linux;\n      } else {\n        detectedOS = OSType.Other;\n      }\n    }\n    return detectedOS;\n  }\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/AlignedTableFloat32.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class AlignedTableFloat32 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected AlignedTableFloat32(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(AlignedTableFloat32 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_AlignedTableFloat32(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setTab(SWIGTYPE_p_faiss__AlignedTableTightAllocT_float_32_t value) {\n    swigfaissJNI.AlignedTableFloat32_tab_set(swigCPtr, this, SWIGTYPE_p_faiss__AlignedTableTightAllocT_float_32_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_faiss__AlignedTableTightAllocT_float_32_t getTab() {\n    long cPtr = swigfaissJNI.AlignedTableFloat32_tab_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__AlignedTableTightAllocT_float_32_t(cPtr, false);\n  }\n\n  public void setNumel(long value) {\n    swigfaissJNI.AlignedTableFloat32_numel_set(swigCPtr, this, value);\n  }\n\n  public long getNumel() {\n    return swigfaissJNI.AlignedTableFloat32_numel_get(swigCPtr, this);\n  }\n\n  public static long round_capacity(long n) {\n    return swigfaissJNI.AlignedTableFloat32_round_capacity(n);\n  }\n\n  public AlignedTableFloat32() {\n    this(swigfaissJNI.new_AlignedTableFloat32__SWIG_0(), true);\n  }\n\n  public AlignedTableFloat32(long n) {\n    this(swigfaissJNI.new_AlignedTableFloat32__SWIG_1(n), true);\n  }\n\n  public long itemsize() {\n    return swigfaissJNI.AlignedTableFloat32_itemsize(swigCPtr, this);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.AlignedTableFloat32_resize(swigCPtr, this, n);\n  }\n\n  public void clear() {\n    swigfaissJNI.AlignedTableFloat32_clear(swigCPtr, this);\n  }\n\n  public long size() {\n    return swigfaissJNI.AlignedTableFloat32_size(swigCPtr, this);\n  }\n\n  public long nbytes() {\n    return swigfaissJNI.AlignedTableFloat32_nbytes(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_float get() {\n    long cPtr = swigfaissJNI.AlignedTableFloat32_get__SWIG_0(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public SWIGTYPE_p_float data() {\n    long cPtr = swigfaissJNI.AlignedTableFloat32_data__SWIG_0(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/AlignedTableUint16.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class AlignedTableUint16 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected AlignedTableUint16(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(AlignedTableUint16 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_AlignedTableUint16(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setTab(SWIGTYPE_p_faiss__AlignedTableTightAllocT_uint16_t_32_t value) {\n    swigfaissJNI.AlignedTableUint16_tab_set(swigCPtr, this, SWIGTYPE_p_faiss__AlignedTableTightAllocT_uint16_t_32_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_faiss__AlignedTableTightAllocT_uint16_t_32_t getTab() {\n    long cPtr = swigfaissJNI.AlignedTableUint16_tab_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__AlignedTableTightAllocT_uint16_t_32_t(cPtr, false);\n  }\n\n  public void setNumel(long value) {\n    swigfaissJNI.AlignedTableUint16_numel_set(swigCPtr, this, value);\n  }\n\n  public long getNumel() {\n    return swigfaissJNI.AlignedTableUint16_numel_get(swigCPtr, this);\n  }\n\n  public static long round_capacity(long n) {\n    return swigfaissJNI.AlignedTableUint16_round_capacity(n);\n  }\n\n  public AlignedTableUint16() {\n    this(swigfaissJNI.new_AlignedTableUint16__SWIG_0(), true);\n  }\n\n  public AlignedTableUint16(long n) {\n    this(swigfaissJNI.new_AlignedTableUint16__SWIG_1(n), true);\n  }\n\n  public long itemsize() {\n    return swigfaissJNI.AlignedTableUint16_itemsize(swigCPtr, this);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.AlignedTableUint16_resize(swigCPtr, this, n);\n  }\n\n  public void clear() {\n    swigfaissJNI.AlignedTableUint16_clear(swigCPtr, this);\n  }\n\n  public long size() {\n    return swigfaissJNI.AlignedTableUint16_size(swigCPtr, this);\n  }\n\n  public long nbytes() {\n    return swigfaissJNI.AlignedTableUint16_nbytes(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_uint16_t get() {\n    long cPtr = swigfaissJNI.AlignedTableUint16_get__SWIG_0(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_uint16_t(cPtr, false);\n  }\n\n  public SWIGTYPE_p_uint16_t data() {\n    long cPtr = swigfaissJNI.AlignedTableUint16_data__SWIG_0(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_uint16_t(cPtr, false);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/AlignedTableUint8.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class AlignedTableUint8 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected AlignedTableUint8(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(AlignedTableUint8 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_AlignedTableUint8(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setTab(SWIGTYPE_p_faiss__AlignedTableTightAllocT_unsigned_char_32_t value) {\n    swigfaissJNI.AlignedTableUint8_tab_set(swigCPtr, this, SWIGTYPE_p_faiss__AlignedTableTightAllocT_unsigned_char_32_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_faiss__AlignedTableTightAllocT_unsigned_char_32_t getTab() {\n    long cPtr = swigfaissJNI.AlignedTableUint8_tab_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__AlignedTableTightAllocT_unsigned_char_32_t(cPtr, false);\n  }\n\n  public void setNumel(long value) {\n    swigfaissJNI.AlignedTableUint8_numel_set(swigCPtr, this, value);\n  }\n\n  public long getNumel() {\n    return swigfaissJNI.AlignedTableUint8_numel_get(swigCPtr, this);\n  }\n\n  public static long round_capacity(long n) {\n    return swigfaissJNI.AlignedTableUint8_round_capacity(n);\n  }\n\n  public AlignedTableUint8() {\n    this(swigfaissJNI.new_AlignedTableUint8__SWIG_0(), true);\n  }\n\n  public AlignedTableUint8(long n) {\n    this(swigfaissJNI.new_AlignedTableUint8__SWIG_1(n), true);\n  }\n\n  public long itemsize() {\n    return swigfaissJNI.AlignedTableUint8_itemsize(swigCPtr, this);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.AlignedTableUint8_resize(swigCPtr, this, n);\n  }\n\n  public void clear() {\n    swigfaissJNI.AlignedTableUint8_clear(swigCPtr, this);\n  }\n\n  public long size() {\n    return swigfaissJNI.AlignedTableUint8_size(swigCPtr, this);\n  }\n\n  public long nbytes() {\n    return swigfaissJNI.AlignedTableUint8_nbytes(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_unsigned_char get() {\n    long cPtr = swigfaissJNI.AlignedTableUint8_get__SWIG_0(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public SWIGTYPE_p_unsigned_char data() {\n    long cPtr = swigfaissJNI.AlignedTableUint8_data__SWIG_0(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ArrayInvertedLists.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ArrayInvertedLists extends InvertedLists {\n  private transient long swigCPtr;\n\n  protected ArrayInvertedLists(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.ArrayInvertedLists_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ArrayInvertedLists obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ArrayInvertedLists(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setCodes(ByteVectorVector value) {\n    swigfaissJNI.ArrayInvertedLists_codes_set(swigCPtr, this, ByteVectorVector.getCPtr(value), value);\n  }\n\n  public ByteVectorVector getCodes() {\n    long cPtr = swigfaissJNI.ArrayInvertedLists_codes_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ByteVectorVector(cPtr, false);\n  }\n\n  public void setIds(SWIGTYPE_p_std__vectorT_std__vectorT_int64_t_t_t value) {\n    swigfaissJNI.ArrayInvertedLists_ids_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_std__vectorT_int64_t_t_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_std__vectorT_int64_t_t_t getIds() {\n    long cPtr = swigfaissJNI.ArrayInvertedLists_ids_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_std__vectorT_int64_t_t_t(cPtr, false);\n  }\n\n  public ArrayInvertedLists(long nlist, long code_size) {\n    this(swigfaissJNI.new_ArrayInvertedLists(nlist, code_size), true);\n  }\n\n  public long list_size(long list_no) {\n    return swigfaissJNI.ArrayInvertedLists_list_size(swigCPtr, this, list_no);\n  }\n\n  public SWIGTYPE_p_unsigned_char get_codes(long list_no) {\n    long cPtr = swigfaissJNI.ArrayInvertedLists_get_codes(swigCPtr, this, list_no);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public LongVector get_ids(long list_no) {\n    return new LongVector(swigfaissJNI.ArrayInvertedLists_get_ids(swigCPtr, this, list_no), false);\n}\n\n  public long add_entries(long list_no, long n_entry, LongVector ids, SWIGTYPE_p_unsigned_char code) {\n    return swigfaissJNI.ArrayInvertedLists_add_entries(swigCPtr, this, list_no, n_entry, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void update_entries(long list_no, long offset, long n_entry, LongVector ids, SWIGTYPE_p_unsigned_char code) {\n    swigfaissJNI.ArrayInvertedLists_update_entries(swigCPtr, this, list_no, offset, n_entry, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void resize(long list_no, long new_size) {\n    swigfaissJNI.ArrayInvertedLists_resize(swigCPtr, this, list_no, new_size);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/AutoTuneCriterion.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class AutoTuneCriterion {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected AutoTuneCriterion(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(AutoTuneCriterion obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_AutoTuneCriterion(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setNq(long value) {\n    swigfaissJNI.AutoTuneCriterion_nq_set(swigCPtr, this, value);\n  }\n\n  public long getNq() {\n    return swigfaissJNI.AutoTuneCriterion_nq_get(swigCPtr, this);\n}\n\n  public void setNnn(long value) {\n    swigfaissJNI.AutoTuneCriterion_nnn_set(swigCPtr, this, value);\n  }\n\n  public long getNnn() {\n    return swigfaissJNI.AutoTuneCriterion_nnn_get(swigCPtr, this);\n}\n\n  public void setGt_nnn(long value) {\n    swigfaissJNI.AutoTuneCriterion_gt_nnn_set(swigCPtr, this, value);\n  }\n\n  public long getGt_nnn() {\n    return swigfaissJNI.AutoTuneCriterion_gt_nnn_get(swigCPtr, this);\n}\n\n  public void setGt_D(FloatVector value) {\n    swigfaissJNI.AutoTuneCriterion_gt_D_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getGt_D() {\n    long cPtr = swigfaissJNI.AutoTuneCriterion_gt_D_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public void setGt_I(SWIGTYPE_p_std__vectorT_int64_t_t value) {\n    swigfaissJNI.AutoTuneCriterion_gt_I_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_int64_t_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_int64_t_t getGt_I() {\n    long cPtr = swigfaissJNI.AutoTuneCriterion_gt_I_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_int64_t_t(cPtr, false);\n  }\n\n  public void set_groundtruth(int gt_nnn, SWIGTYPE_p_float gt_D_in, LongVector gt_I_in) {\n    swigfaissJNI.AutoTuneCriterion_set_groundtruth(swigCPtr, this, gt_nnn, SWIGTYPE_p_float.getCPtr(gt_D_in), SWIGTYPE_p_long_long.getCPtr(gt_I_in.data()), gt_I_in);\n  }\n\n  public double evaluate(SWIGTYPE_p_float D, LongVector I) {\n    return swigfaissJNI.AutoTuneCriterion_evaluate(swigCPtr, this, SWIGTYPE_p_float.getCPtr(D), SWIGTYPE_p_long_long.getCPtr(I.data()), I);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/BUILD",
    "content": "java_library(\n    name = \"swig-local\",\n    sources = [\"*.java\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \"ann/src/main/java/com/twitter/ann/faiss:swig-native-utils\",\n        \"ann/src/main/java/com/twitter/ann/faiss/swig/resources\",\n    ],\n)\n\njava_library(\n    name = \"swig-artifactory\",\n    sources = [\"*.java\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/ann/faiss/swig:resources\",\n        \"ann/src/main/java/com/twitter/ann/faiss:swig-native-utils\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/BitstringReader.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class BitstringReader {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected BitstringReader(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(BitstringReader obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_BitstringReader(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setCode(SWIGTYPE_p_unsigned_char value) {\n    swigfaissJNI.BitstringReader_code_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_unsigned_char getCode() {\n    long cPtr = swigfaissJNI.BitstringReader_code_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public void setCode_size(long value) {\n    swigfaissJNI.BitstringReader_code_size_set(swigCPtr, this, value);\n  }\n\n  public long getCode_size() {\n    return swigfaissJNI.BitstringReader_code_size_get(swigCPtr, this);\n  }\n\n  public void setI(long value) {\n    swigfaissJNI.BitstringReader_i_set(swigCPtr, this, value);\n  }\n\n  public long getI() {\n    return swigfaissJNI.BitstringReader_i_get(swigCPtr, this);\n  }\n\n  public BitstringReader(SWIGTYPE_p_unsigned_char code, long code_size) {\n    this(swigfaissJNI.new_BitstringReader(SWIGTYPE_p_unsigned_char.getCPtr(code), code_size), true);\n  }\n\n  public long read(int nbit) {\n    return swigfaissJNI.BitstringReader_read(swigCPtr, this, nbit);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/BitstringWriter.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class BitstringWriter {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected BitstringWriter(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(BitstringWriter obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_BitstringWriter(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setCode(SWIGTYPE_p_unsigned_char value) {\n    swigfaissJNI.BitstringWriter_code_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_unsigned_char getCode() {\n    long cPtr = swigfaissJNI.BitstringWriter_code_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public void setCode_size(long value) {\n    swigfaissJNI.BitstringWriter_code_size_set(swigCPtr, this, value);\n  }\n\n  public long getCode_size() {\n    return swigfaissJNI.BitstringWriter_code_size_get(swigCPtr, this);\n  }\n\n  public void setI(long value) {\n    swigfaissJNI.BitstringWriter_i_set(swigCPtr, this, value);\n  }\n\n  public long getI() {\n    return swigfaissJNI.BitstringWriter_i_get(swigCPtr, this);\n  }\n\n  public BitstringWriter(SWIGTYPE_p_unsigned_char code, long code_size) {\n    this(swigfaissJNI.new_BitstringWriter(SWIGTYPE_p_unsigned_char.getCPtr(code), code_size), true);\n  }\n\n  public void write(long x, int nbit) {\n    swigfaissJNI.BitstringWriter_write(swigCPtr, this, x, nbit);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/BufferList.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class BufferList {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected BufferList(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(BufferList obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_BufferList(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setBuffer_size(long value) {\n    swigfaissJNI.BufferList_buffer_size_set(swigCPtr, this, value);\n  }\n\n  public long getBuffer_size() {\n    return swigfaissJNI.BufferList_buffer_size_get(swigCPtr, this);\n  }\n\n  public void setBuffers(SWIGTYPE_p_std__vectorT_faiss__BufferList__Buffer_t value) {\n    swigfaissJNI.BufferList_buffers_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_faiss__BufferList__Buffer_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_faiss__BufferList__Buffer_t getBuffers() {\n    long cPtr = swigfaissJNI.BufferList_buffers_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_faiss__BufferList__Buffer_t(cPtr, false);\n  }\n\n  public void setWp(long value) {\n    swigfaissJNI.BufferList_wp_set(swigCPtr, this, value);\n  }\n\n  public long getWp() {\n    return swigfaissJNI.BufferList_wp_get(swigCPtr, this);\n  }\n\n  public BufferList(long buffer_size) {\n    this(swigfaissJNI.new_BufferList(buffer_size), true);\n  }\n\n  public void append_buffer() {\n    swigfaissJNI.BufferList_append_buffer(swigCPtr, this);\n  }\n\n  public void add(long id, float dis) {\n    swigfaissJNI.BufferList_add(swigCPtr, this, id, dis);\n  }\n\n  public void copy_range(long ofs, long n, LongVector dest_ids, SWIGTYPE_p_float dest_dis) {\n    swigfaissJNI.BufferList_copy_range(swigCPtr, this, ofs, n, SWIGTYPE_p_long_long.getCPtr(dest_ids.data()), dest_ids, SWIGTYPE_p_float.getCPtr(dest_dis));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ByteVector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ByteVector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected ByteVector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ByteVector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ByteVector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public ByteVector() {\n    this(swigfaissJNI.new_ByteVector(), true);\n  }\n\n  public void push_back(short arg0) {\n    swigfaissJNI.ByteVector_push_back(swigCPtr, this, arg0);\n  }\n\n  public void clear() {\n    swigfaissJNI.ByteVector_clear(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_unsigned_char data() {\n    long cPtr = swigfaissJNI.ByteVector_data(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public long size() {\n    return swigfaissJNI.ByteVector_size(swigCPtr, this);\n  }\n\n  public short at(long n) {\n    return swigfaissJNI.ByteVector_at(swigCPtr, this, n);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.ByteVector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.ByteVector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(ByteVector other) {\n    swigfaissJNI.ByteVector_swap(swigCPtr, this, ByteVector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ByteVectorVector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ByteVectorVector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected ByteVectorVector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ByteVectorVector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ByteVectorVector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public ByteVectorVector() {\n    this(swigfaissJNI.new_ByteVectorVector(), true);\n  }\n\n  public void push_back(ByteVector arg0) {\n    swigfaissJNI.ByteVectorVector_push_back(swigCPtr, this, ByteVector.getCPtr(arg0), arg0);\n  }\n\n  public void clear() {\n    swigfaissJNI.ByteVectorVector_clear(swigCPtr, this);\n  }\n\n  public ByteVector data() {\n    long cPtr = swigfaissJNI.ByteVectorVector_data(swigCPtr, this);\n    return (cPtr == 0) ? null : new ByteVector(cPtr, false);\n  }\n\n  public long size() {\n    return swigfaissJNI.ByteVectorVector_size(swigCPtr, this);\n  }\n\n  public ByteVector at(long n) {\n    return new ByteVector(swigfaissJNI.ByteVectorVector_at(swigCPtr, this, n), true);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.ByteVectorVector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.ByteVectorVector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(ByteVectorVector other) {\n    swigfaissJNI.ByteVectorVector_swap(swigCPtr, this, ByteVectorVector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/CenteringTransform.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class CenteringTransform extends VectorTransform {\n  private transient long swigCPtr;\n\n  protected CenteringTransform(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.CenteringTransform_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(CenteringTransform obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_CenteringTransform(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setMean(FloatVector value) {\n    swigfaissJNI.CenteringTransform_mean_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getMean() {\n    long cPtr = swigfaissJNI.CenteringTransform_mean_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public CenteringTransform(int d) {\n    this(swigfaissJNI.new_CenteringTransform__SWIG_0(d), true);\n  }\n\n  public CenteringTransform() {\n    this(swigfaissJNI.new_CenteringTransform__SWIG_1(), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.CenteringTransform_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void apply_noalloc(long n, SWIGTYPE_p_float x, SWIGTYPE_p_float xt) {\n    swigfaissJNI.CenteringTransform_apply_noalloc(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(xt));\n  }\n\n  public void reverse_transform(long n, SWIGTYPE_p_float xt, SWIGTYPE_p_float x) {\n    swigfaissJNI.CenteringTransform_reverse_transform(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(xt), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/CharVector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class CharVector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected CharVector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(CharVector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_CharVector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public CharVector() {\n    this(swigfaissJNI.new_CharVector(), true);\n  }\n\n  public void push_back(char arg0) {\n    swigfaissJNI.CharVector_push_back(swigCPtr, this, arg0);\n  }\n\n  public void clear() {\n    swigfaissJNI.CharVector_clear(swigCPtr, this);\n  }\n\n  public String data() {\n    return swigfaissJNI.CharVector_data(swigCPtr, this);\n  }\n\n  public long size() {\n    return swigfaissJNI.CharVector_size(swigCPtr, this);\n  }\n\n  public char at(long n) {\n    return swigfaissJNI.CharVector_at(swigCPtr, this, n);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.CharVector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.CharVector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(CharVector other) {\n    swigfaissJNI.CharVector_swap(swigCPtr, this, CharVector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/Clustering.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class Clustering extends ClusteringParameters {\n  private transient long swigCPtr;\n\n  protected Clustering(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.Clustering_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(Clustering obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_Clustering(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setD(long value) {\n    swigfaissJNI.Clustering_d_set(swigCPtr, this, value);\n  }\n\n  public long getD() {\n    return swigfaissJNI.Clustering_d_get(swigCPtr, this);\n  }\n\n  public void setK(long value) {\n    swigfaissJNI.Clustering_k_set(swigCPtr, this, value);\n  }\n\n  public long getK() {\n    return swigfaissJNI.Clustering_k_get(swigCPtr, this);\n  }\n\n  public void setCentroids(FloatVector value) {\n    swigfaissJNI.Clustering_centroids_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getCentroids() {\n    long cPtr = swigfaissJNI.Clustering_centroids_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public void setIteration_stats(SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t value) {\n    swigfaissJNI.Clustering_iteration_stats_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t getIteration_stats() {\n    long cPtr = swigfaissJNI.Clustering_iteration_stats_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t(cPtr, false);\n  }\n\n  public Clustering(int d, int k) {\n    this(swigfaissJNI.new_Clustering__SWIG_0(d, k), true);\n  }\n\n  public Clustering(int d, int k, ClusteringParameters cp) {\n    this(swigfaissJNI.new_Clustering__SWIG_1(d, k, ClusteringParameters.getCPtr(cp), cp), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x, Index index, SWIGTYPE_p_float x_weights) {\n    swigfaissJNI.Clustering_train__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), Index.getCPtr(index), index, SWIGTYPE_p_float.getCPtr(x_weights));\n  }\n\n  public void train(long n, SWIGTYPE_p_float x, Index index) {\n    swigfaissJNI.Clustering_train__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), Index.getCPtr(index), index);\n  }\n\n  public void train_encoded(long nx, SWIGTYPE_p_unsigned_char x_in, Index codec, Index index, SWIGTYPE_p_float weights) {\n    swigfaissJNI.Clustering_train_encoded__SWIG_0(swigCPtr, this, nx, SWIGTYPE_p_unsigned_char.getCPtr(x_in), Index.getCPtr(codec), codec, Index.getCPtr(index), index, SWIGTYPE_p_float.getCPtr(weights));\n  }\n\n  public void train_encoded(long nx, SWIGTYPE_p_unsigned_char x_in, Index codec, Index index) {\n    swigfaissJNI.Clustering_train_encoded__SWIG_1(swigCPtr, this, nx, SWIGTYPE_p_unsigned_char.getCPtr(x_in), Index.getCPtr(codec), codec, Index.getCPtr(index), index);\n  }\n\n  public void post_process_centroids() {\n    swigfaissJNI.Clustering_post_process_centroids(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/Clustering1D.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class Clustering1D extends Clustering {\n  private transient long swigCPtr;\n\n  protected Clustering1D(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.Clustering1D_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(Clustering1D obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_Clustering1D(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public Clustering1D(int k) {\n    this(swigfaissJNI.new_Clustering1D__SWIG_0(k), true);\n  }\n\n  public Clustering1D(int k, ClusteringParameters cp) {\n    this(swigfaissJNI.new_Clustering1D__SWIG_1(k, ClusteringParameters.getCPtr(cp), cp), true);\n  }\n\n  public void train_exact(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.Clustering1D_train_exact(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ClusteringIterationStats.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ClusteringIterationStats {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected ClusteringIterationStats(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ClusteringIterationStats obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ClusteringIterationStats(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setObj(float value) {\n    swigfaissJNI.ClusteringIterationStats_obj_set(swigCPtr, this, value);\n  }\n\n  public float getObj() {\n    return swigfaissJNI.ClusteringIterationStats_obj_get(swigCPtr, this);\n  }\n\n  public void setTime(double value) {\n    swigfaissJNI.ClusteringIterationStats_time_set(swigCPtr, this, value);\n  }\n\n  public double getTime() {\n    return swigfaissJNI.ClusteringIterationStats_time_get(swigCPtr, this);\n  }\n\n  public void setTime_search(double value) {\n    swigfaissJNI.ClusteringIterationStats_time_search_set(swigCPtr, this, value);\n  }\n\n  public double getTime_search() {\n    return swigfaissJNI.ClusteringIterationStats_time_search_get(swigCPtr, this);\n  }\n\n  public void setImbalance_factor(double value) {\n    swigfaissJNI.ClusteringIterationStats_imbalance_factor_set(swigCPtr, this, value);\n  }\n\n  public double getImbalance_factor() {\n    return swigfaissJNI.ClusteringIterationStats_imbalance_factor_get(swigCPtr, this);\n  }\n\n  public void setNsplit(int value) {\n    swigfaissJNI.ClusteringIterationStats_nsplit_set(swigCPtr, this, value);\n  }\n\n  public int getNsplit() {\n    return swigfaissJNI.ClusteringIterationStats_nsplit_get(swigCPtr, this);\n  }\n\n  public ClusteringIterationStats() {\n    this(swigfaissJNI.new_ClusteringIterationStats(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ClusteringParameters.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ClusteringParameters {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected ClusteringParameters(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ClusteringParameters obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ClusteringParameters(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setNiter(int value) {\n    swigfaissJNI.ClusteringParameters_niter_set(swigCPtr, this, value);\n  }\n\n  public int getNiter() {\n    return swigfaissJNI.ClusteringParameters_niter_get(swigCPtr, this);\n  }\n\n  public void setNredo(int value) {\n    swigfaissJNI.ClusteringParameters_nredo_set(swigCPtr, this, value);\n  }\n\n  public int getNredo() {\n    return swigfaissJNI.ClusteringParameters_nredo_get(swigCPtr, this);\n  }\n\n  public void setVerbose(boolean value) {\n    swigfaissJNI.ClusteringParameters_verbose_set(swigCPtr, this, value);\n  }\n\n  public boolean getVerbose() {\n    return swigfaissJNI.ClusteringParameters_verbose_get(swigCPtr, this);\n  }\n\n  public void setSpherical(boolean value) {\n    swigfaissJNI.ClusteringParameters_spherical_set(swigCPtr, this, value);\n  }\n\n  public boolean getSpherical() {\n    return swigfaissJNI.ClusteringParameters_spherical_get(swigCPtr, this);\n  }\n\n  public void setInt_centroids(boolean value) {\n    swigfaissJNI.ClusteringParameters_int_centroids_set(swigCPtr, this, value);\n  }\n\n  public boolean getInt_centroids() {\n    return swigfaissJNI.ClusteringParameters_int_centroids_get(swigCPtr, this);\n  }\n\n  public void setUpdate_index(boolean value) {\n    swigfaissJNI.ClusteringParameters_update_index_set(swigCPtr, this, value);\n  }\n\n  public boolean getUpdate_index() {\n    return swigfaissJNI.ClusteringParameters_update_index_get(swigCPtr, this);\n  }\n\n  public void setFrozen_centroids(boolean value) {\n    swigfaissJNI.ClusteringParameters_frozen_centroids_set(swigCPtr, this, value);\n  }\n\n  public boolean getFrozen_centroids() {\n    return swigfaissJNI.ClusteringParameters_frozen_centroids_get(swigCPtr, this);\n  }\n\n  public void setMin_points_per_centroid(int value) {\n    swigfaissJNI.ClusteringParameters_min_points_per_centroid_set(swigCPtr, this, value);\n  }\n\n  public int getMin_points_per_centroid() {\n    return swigfaissJNI.ClusteringParameters_min_points_per_centroid_get(swigCPtr, this);\n  }\n\n  public void setMax_points_per_centroid(int value) {\n    swigfaissJNI.ClusteringParameters_max_points_per_centroid_set(swigCPtr, this, value);\n  }\n\n  public int getMax_points_per_centroid() {\n    return swigfaissJNI.ClusteringParameters_max_points_per_centroid_get(swigCPtr, this);\n  }\n\n  public void setSeed(int value) {\n    swigfaissJNI.ClusteringParameters_seed_set(swigCPtr, this, value);\n  }\n\n  public int getSeed() {\n    return swigfaissJNI.ClusteringParameters_seed_get(swigCPtr, this);\n  }\n\n  public void setDecode_block_size(long value) {\n    swigfaissJNI.ClusteringParameters_decode_block_size_set(swigCPtr, this, value);\n  }\n\n  public long getDecode_block_size() {\n    return swigfaissJNI.ClusteringParameters_decode_block_size_get(swigCPtr, this);\n  }\n\n  public ClusteringParameters() {\n    this(swigfaissJNI.new_ClusteringParameters(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/DistanceComputer.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class DistanceComputer {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected DistanceComputer(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(DistanceComputer obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_DistanceComputer(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void set_query(SWIGTYPE_p_float x) {\n    swigfaissJNI.DistanceComputer_set_query(swigCPtr, this, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public float symmetric_dis(long i, long j) {\n    return swigfaissJNI.DistanceComputer_symmetric_dis(swigCPtr, this, i, j);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/DoubleVector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class DoubleVector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected DoubleVector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(DoubleVector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_DoubleVector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public DoubleVector() {\n    this(swigfaissJNI.new_DoubleVector(), true);\n  }\n\n  public void push_back(double arg0) {\n    swigfaissJNI.DoubleVector_push_back(swigCPtr, this, arg0);\n  }\n\n  public void clear() {\n    swigfaissJNI.DoubleVector_clear(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_double data() {\n    long cPtr = swigfaissJNI.DoubleVector_data(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_double(cPtr, false);\n  }\n\n  public long size() {\n    return swigfaissJNI.DoubleVector_size(swigCPtr, this);\n  }\n\n  public double at(long n) {\n    return swigfaissJNI.DoubleVector_at(swigCPtr, this, n);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.DoubleVector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.DoubleVector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(DoubleVector other) {\n    swigfaissJNI.DoubleVector_swap(swigCPtr, this, DoubleVector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/FloatVector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class FloatVector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected FloatVector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(FloatVector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_FloatVector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public FloatVector() {\n    this(swigfaissJNI.new_FloatVector(), true);\n  }\n\n  public void push_back(float arg0) {\n    swigfaissJNI.FloatVector_push_back(swigCPtr, this, arg0);\n  }\n\n  public void clear() {\n    swigfaissJNI.FloatVector_clear(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_float data() {\n    long cPtr = swigfaissJNI.FloatVector_data(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public long size() {\n    return swigfaissJNI.FloatVector_size(swigCPtr, this);\n  }\n\n  public float at(long n) {\n    return swigfaissJNI.FloatVector_at(swigCPtr, this, n);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.FloatVector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.FloatVector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(FloatVector other) {\n    swigfaissJNI.FloatVector_swap(swigCPtr, this, FloatVector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/FloatVectorVector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class FloatVectorVector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected FloatVectorVector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(FloatVectorVector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_FloatVectorVector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public FloatVectorVector() {\n    this(swigfaissJNI.new_FloatVectorVector(), true);\n  }\n\n  public void push_back(FloatVector arg0) {\n    swigfaissJNI.FloatVectorVector_push_back(swigCPtr, this, FloatVector.getCPtr(arg0), arg0);\n  }\n\n  public void clear() {\n    swigfaissJNI.FloatVectorVector_clear(swigCPtr, this);\n  }\n\n  public FloatVector data() {\n    long cPtr = swigfaissJNI.FloatVectorVector_data(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public long size() {\n    return swigfaissJNI.FloatVectorVector_size(swigCPtr, this);\n  }\n\n  public FloatVector at(long n) {\n    return new FloatVector(swigfaissJNI.FloatVectorVector_at(swigCPtr, this, n), true);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.FloatVectorVector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.FloatVectorVector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(FloatVectorVector other) {\n    swigfaissJNI.FloatVectorVector_swap(swigCPtr, this, FloatVectorVector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/GenHammingComputer16.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class GenHammingComputer16 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected GenHammingComputer16(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(GenHammingComputer16 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_GenHammingComputer16(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA0(long value) {\n    swigfaissJNI.GenHammingComputer16_a0_set(swigCPtr, this, value);\n  }\n\n  public long getA0() {\n    return swigfaissJNI.GenHammingComputer16_a0_get(swigCPtr, this);\n  }\n\n  public void setA1(long value) {\n    swigfaissJNI.GenHammingComputer16_a1_set(swigCPtr, this, value);\n  }\n\n  public long getA1() {\n    return swigfaissJNI.GenHammingComputer16_a1_get(swigCPtr, this);\n  }\n\n  public GenHammingComputer16(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    this(swigfaissJNI.new_GenHammingComputer16(SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size), true);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b8) {\n    return swigfaissJNI.GenHammingComputer16_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b8));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/GenHammingComputer32.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class GenHammingComputer32 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected GenHammingComputer32(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(GenHammingComputer32 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_GenHammingComputer32(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA0(long value) {\n    swigfaissJNI.GenHammingComputer32_a0_set(swigCPtr, this, value);\n  }\n\n  public long getA0() {\n    return swigfaissJNI.GenHammingComputer32_a0_get(swigCPtr, this);\n  }\n\n  public void setA1(long value) {\n    swigfaissJNI.GenHammingComputer32_a1_set(swigCPtr, this, value);\n  }\n\n  public long getA1() {\n    return swigfaissJNI.GenHammingComputer32_a1_get(swigCPtr, this);\n  }\n\n  public void setA2(long value) {\n    swigfaissJNI.GenHammingComputer32_a2_set(swigCPtr, this, value);\n  }\n\n  public long getA2() {\n    return swigfaissJNI.GenHammingComputer32_a2_get(swigCPtr, this);\n  }\n\n  public void setA3(long value) {\n    swigfaissJNI.GenHammingComputer32_a3_set(swigCPtr, this, value);\n  }\n\n  public long getA3() {\n    return swigfaissJNI.GenHammingComputer32_a3_get(swigCPtr, this);\n  }\n\n  public GenHammingComputer32(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    this(swigfaissJNI.new_GenHammingComputer32(SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size), true);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b8) {\n    return swigfaissJNI.GenHammingComputer32_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b8));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/GenHammingComputer8.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class GenHammingComputer8 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected GenHammingComputer8(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(GenHammingComputer8 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_GenHammingComputer8(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA0(long value) {\n    swigfaissJNI.GenHammingComputer8_a0_set(swigCPtr, this, value);\n  }\n\n  public long getA0() {\n    return swigfaissJNI.GenHammingComputer8_a0_get(swigCPtr, this);\n  }\n\n  public GenHammingComputer8(SWIGTYPE_p_unsigned_char a, int code_size) {\n    this(swigfaissJNI.new_GenHammingComputer8(SWIGTYPE_p_unsigned_char.getCPtr(a), code_size), true);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b) {\n    return swigfaissJNI.GenHammingComputer8_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/GenHammingComputerM8.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class GenHammingComputerM8 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected GenHammingComputerM8(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(GenHammingComputerM8 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_GenHammingComputerM8(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA(SWIGTYPE_p_unsigned_long value) {\n    swigfaissJNI.GenHammingComputerM8_a_set(swigCPtr, this, SWIGTYPE_p_unsigned_long.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_unsigned_long getA() {\n    long cPtr = swigfaissJNI.GenHammingComputerM8_a_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_long(cPtr, false);\n  }\n\n  public void setN(int value) {\n    swigfaissJNI.GenHammingComputerM8_n_set(swigCPtr, this, value);\n  }\n\n  public int getN() {\n    return swigfaissJNI.GenHammingComputerM8_n_get(swigCPtr, this);\n  }\n\n  public GenHammingComputerM8(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    this(swigfaissJNI.new_GenHammingComputerM8(SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size), true);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b8) {\n    return swigfaissJNI.GenHammingComputerM8_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b8));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/HNSW.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class HNSW {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected HNSW(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(HNSW obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_HNSW(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  static public class MinimaxHeap {\n    private transient long swigCPtr;\n    protected transient boolean swigCMemOwn;\n  \n    protected MinimaxHeap(long cPtr, boolean cMemoryOwn) {\n      swigCMemOwn = cMemoryOwn;\n      swigCPtr = cPtr;\n    }\n  \n    protected static long getCPtr(MinimaxHeap obj) {\n      return (obj == null) ? 0 : obj.swigCPtr;\n    }\n  \n    @SuppressWarnings(\"deprecation\")\n    protected void finalize() {\n      delete();\n    }\n  \n    public synchronized void delete() {\n      if (swigCPtr != 0) {\n        if (swigCMemOwn) {\n          swigCMemOwn = false;\n          swigfaissJNI.delete_HNSW_MinimaxHeap(swigCPtr);\n        }\n        swigCPtr = 0;\n      }\n    }\n  \n    public void setN(int value) {\n      swigfaissJNI.HNSW_MinimaxHeap_n_set(swigCPtr, this, value);\n    }\n  \n    public int getN() {\n      return swigfaissJNI.HNSW_MinimaxHeap_n_get(swigCPtr, this);\n    }\n  \n    public void setK(int value) {\n      swigfaissJNI.HNSW_MinimaxHeap_k_set(swigCPtr, this, value);\n    }\n  \n    public int getK() {\n      return swigfaissJNI.HNSW_MinimaxHeap_k_get(swigCPtr, this);\n    }\n  \n    public void setNvalid(int value) {\n      swigfaissJNI.HNSW_MinimaxHeap_nvalid_set(swigCPtr, this, value);\n    }\n  \n    public int getNvalid() {\n      return swigfaissJNI.HNSW_MinimaxHeap_nvalid_get(swigCPtr, this);\n    }\n  \n    public void setIds(IntVector value) {\n      swigfaissJNI.HNSW_MinimaxHeap_ids_set(swigCPtr, this, IntVector.getCPtr(value), value);\n    }\n  \n    public IntVector getIds() {\n      long cPtr = swigfaissJNI.HNSW_MinimaxHeap_ids_get(swigCPtr, this);\n      return (cPtr == 0) ? null : new IntVector(cPtr, false);\n    }\n  \n    public void setDis(FloatVector value) {\n      swigfaissJNI.HNSW_MinimaxHeap_dis_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n    }\n  \n    public FloatVector getDis() {\n      long cPtr = swigfaissJNI.HNSW_MinimaxHeap_dis_get(swigCPtr, this);\n      return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n    }\n  \n    public MinimaxHeap(int n) {\n      this(swigfaissJNI.new_HNSW_MinimaxHeap(n), true);\n    }\n  \n    public void push(int i, float v) {\n      swigfaissJNI.HNSW_MinimaxHeap_push(swigCPtr, this, i, v);\n    }\n  \n    public float max() {\n      return swigfaissJNI.HNSW_MinimaxHeap_max(swigCPtr, this);\n    }\n  \n    public int size() {\n      return swigfaissJNI.HNSW_MinimaxHeap_size(swigCPtr, this);\n    }\n  \n    public void clear() {\n      swigfaissJNI.HNSW_MinimaxHeap_clear(swigCPtr, this);\n    }\n  \n    public int pop_min(SWIGTYPE_p_float vmin_out) {\n      return swigfaissJNI.HNSW_MinimaxHeap_pop_min__SWIG_0(swigCPtr, this, SWIGTYPE_p_float.getCPtr(vmin_out));\n    }\n  \n    public int pop_min() {\n      return swigfaissJNI.HNSW_MinimaxHeap_pop_min__SWIG_1(swigCPtr, this);\n    }\n  \n    public int count_below(float thresh) {\n      return swigfaissJNI.HNSW_MinimaxHeap_count_below(swigCPtr, this, thresh);\n    }\n  \n  }\n\n  static public class NodeDistCloser {\n    private transient long swigCPtr;\n    protected transient boolean swigCMemOwn;\n  \n    protected NodeDistCloser(long cPtr, boolean cMemoryOwn) {\n      swigCMemOwn = cMemoryOwn;\n      swigCPtr = cPtr;\n    }\n  \n    protected static long getCPtr(NodeDistCloser obj) {\n      return (obj == null) ? 0 : obj.swigCPtr;\n    }\n  \n    @SuppressWarnings(\"deprecation\")\n    protected void finalize() {\n      delete();\n    }\n  \n    public synchronized void delete() {\n      if (swigCPtr != 0) {\n        if (swigCMemOwn) {\n          swigCMemOwn = false;\n          swigfaissJNI.delete_HNSW_NodeDistCloser(swigCPtr);\n        }\n        swigCPtr = 0;\n      }\n    }\n  \n    public void setD(float value) {\n      swigfaissJNI.HNSW_NodeDistCloser_d_set(swigCPtr, this, value);\n    }\n  \n    public float getD() {\n      return swigfaissJNI.HNSW_NodeDistCloser_d_get(swigCPtr, this);\n    }\n  \n    public void setId(int value) {\n      swigfaissJNI.HNSW_NodeDistCloser_id_set(swigCPtr, this, value);\n    }\n  \n    public int getId() {\n      return swigfaissJNI.HNSW_NodeDistCloser_id_get(swigCPtr, this);\n    }\n  \n    public NodeDistCloser(float d, int id) {\n      this(swigfaissJNI.new_HNSW_NodeDistCloser(d, id), true);\n    }\n  \n  }\n\n  static public class NodeDistFarther {\n    private transient long swigCPtr;\n    protected transient boolean swigCMemOwn;\n  \n    protected NodeDistFarther(long cPtr, boolean cMemoryOwn) {\n      swigCMemOwn = cMemoryOwn;\n      swigCPtr = cPtr;\n    }\n  \n    protected static long getCPtr(NodeDistFarther obj) {\n      return (obj == null) ? 0 : obj.swigCPtr;\n    }\n  \n    @SuppressWarnings(\"deprecation\")\n    protected void finalize() {\n      delete();\n    }\n  \n    public synchronized void delete() {\n      if (swigCPtr != 0) {\n        if (swigCMemOwn) {\n          swigCMemOwn = false;\n          swigfaissJNI.delete_HNSW_NodeDistFarther(swigCPtr);\n        }\n        swigCPtr = 0;\n      }\n    }\n  \n    public void setD(float value) {\n      swigfaissJNI.HNSW_NodeDistFarther_d_set(swigCPtr, this, value);\n    }\n  \n    public float getD() {\n      return swigfaissJNI.HNSW_NodeDistFarther_d_get(swigCPtr, this);\n    }\n  \n    public void setId(int value) {\n      swigfaissJNI.HNSW_NodeDistFarther_id_set(swigCPtr, this, value);\n    }\n  \n    public int getId() {\n      return swigfaissJNI.HNSW_NodeDistFarther_id_get(swigCPtr, this);\n    }\n  \n    public NodeDistFarther(float d, int id) {\n      this(swigfaissJNI.new_HNSW_NodeDistFarther(d, id), true);\n    }\n  \n  }\n\n  public void setAssign_probas(DoubleVector value) {\n    swigfaissJNI.HNSW_assign_probas_set(swigCPtr, this, DoubleVector.getCPtr(value), value);\n  }\n\n  public DoubleVector getAssign_probas() {\n    long cPtr = swigfaissJNI.HNSW_assign_probas_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new DoubleVector(cPtr, false);\n  }\n\n  public void setCum_nneighbor_per_level(IntVector value) {\n    swigfaissJNI.HNSW_cum_nneighbor_per_level_set(swigCPtr, this, IntVector.getCPtr(value), value);\n  }\n\n  public IntVector getCum_nneighbor_per_level() {\n    long cPtr = swigfaissJNI.HNSW_cum_nneighbor_per_level_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new IntVector(cPtr, false);\n  }\n\n  public void setLevels(IntVector value) {\n    swigfaissJNI.HNSW_levels_set(swigCPtr, this, IntVector.getCPtr(value), value);\n  }\n\n  public IntVector getLevels() {\n    long cPtr = swigfaissJNI.HNSW_levels_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new IntVector(cPtr, false);\n  }\n\n  public void setOffsets(Uint64Vector value) {\n    swigfaissJNI.HNSW_offsets_set(swigCPtr, this, Uint64Vector.getCPtr(value), value);\n  }\n\n  public Uint64Vector getOffsets() {\n    long cPtr = swigfaissJNI.HNSW_offsets_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new Uint64Vector(cPtr, false);\n  }\n\n  public void setNeighbors(IntVector value) {\n    swigfaissJNI.HNSW_neighbors_set(swigCPtr, this, IntVector.getCPtr(value), value);\n  }\n\n  public IntVector getNeighbors() {\n    long cPtr = swigfaissJNI.HNSW_neighbors_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new IntVector(cPtr, false);\n  }\n\n  public void setEntry_point(int value) {\n    swigfaissJNI.HNSW_entry_point_set(swigCPtr, this, value);\n  }\n\n  public int getEntry_point() {\n    return swigfaissJNI.HNSW_entry_point_get(swigCPtr, this);\n  }\n\n  public void setRng(SWIGTYPE_p_faiss__RandomGenerator value) {\n    swigfaissJNI.HNSW_rng_set(swigCPtr, this, SWIGTYPE_p_faiss__RandomGenerator.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_faiss__RandomGenerator getRng() {\n    long cPtr = swigfaissJNI.HNSW_rng_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__RandomGenerator(cPtr, false);\n  }\n\n  public void setMax_level(int value) {\n    swigfaissJNI.HNSW_max_level_set(swigCPtr, this, value);\n  }\n\n  public int getMax_level() {\n    return swigfaissJNI.HNSW_max_level_get(swigCPtr, this);\n  }\n\n  public void setEfConstruction(int value) {\n    swigfaissJNI.HNSW_efConstruction_set(swigCPtr, this, value);\n  }\n\n  public int getEfConstruction() {\n    return swigfaissJNI.HNSW_efConstruction_get(swigCPtr, this);\n  }\n\n  public void setEfSearch(int value) {\n    swigfaissJNI.HNSW_efSearch_set(swigCPtr, this, value);\n  }\n\n  public int getEfSearch() {\n    return swigfaissJNI.HNSW_efSearch_get(swigCPtr, this);\n  }\n\n  public void setCheck_relative_distance(boolean value) {\n    swigfaissJNI.HNSW_check_relative_distance_set(swigCPtr, this, value);\n  }\n\n  public boolean getCheck_relative_distance() {\n    return swigfaissJNI.HNSW_check_relative_distance_get(swigCPtr, this);\n  }\n\n  public void setUpper_beam(int value) {\n    swigfaissJNI.HNSW_upper_beam_set(swigCPtr, this, value);\n  }\n\n  public int getUpper_beam() {\n    return swigfaissJNI.HNSW_upper_beam_get(swigCPtr, this);\n  }\n\n  public void setSearch_bounded_queue(boolean value) {\n    swigfaissJNI.HNSW_search_bounded_queue_set(swigCPtr, this, value);\n  }\n\n  public boolean getSearch_bounded_queue() {\n    return swigfaissJNI.HNSW_search_bounded_queue_get(swigCPtr, this);\n  }\n\n  public void set_default_probas(int M, float levelMult) {\n    swigfaissJNI.HNSW_set_default_probas(swigCPtr, this, M, levelMult);\n  }\n\n  public void set_nb_neighbors(int level_no, int n) {\n    swigfaissJNI.HNSW_set_nb_neighbors(swigCPtr, this, level_no, n);\n  }\n\n  public int nb_neighbors(int layer_no) {\n    return swigfaissJNI.HNSW_nb_neighbors(swigCPtr, this, layer_no);\n  }\n\n  public int cum_nb_neighbors(int layer_no) {\n    return swigfaissJNI.HNSW_cum_nb_neighbors(swigCPtr, this, layer_no);\n  }\n\n  public void neighbor_range(long no, int layer_no, SWIGTYPE_p_unsigned_long begin, SWIGTYPE_p_unsigned_long end) {\n    swigfaissJNI.HNSW_neighbor_range(swigCPtr, this, no, layer_no, SWIGTYPE_p_unsigned_long.getCPtr(begin), SWIGTYPE_p_unsigned_long.getCPtr(end));\n  }\n\n  public HNSW(int M) {\n    this(swigfaissJNI.new_HNSW__SWIG_0(M), true);\n  }\n\n  public HNSW() {\n    this(swigfaissJNI.new_HNSW__SWIG_1(), true);\n  }\n\n  public int random_level() {\n    return swigfaissJNI.HNSW_random_level(swigCPtr, this);\n  }\n\n  public void fill_with_random_links(long n) {\n    swigfaissJNI.HNSW_fill_with_random_links(swigCPtr, this, n);\n  }\n\n  public void add_links_starting_from(DistanceComputer ptdis, int pt_id, int nearest, float d_nearest, int level, SWIGTYPE_p_omp_lock_t locks, VisitedTable vt) {\n    swigfaissJNI.HNSW_add_links_starting_from(swigCPtr, this, DistanceComputer.getCPtr(ptdis), ptdis, pt_id, nearest, d_nearest, level, SWIGTYPE_p_omp_lock_t.getCPtr(locks), VisitedTable.getCPtr(vt), vt);\n  }\n\n  public void add_with_locks(DistanceComputer ptdis, int pt_level, int pt_id, SWIGTYPE_p_std__vectorT_omp_lock_t_t locks, VisitedTable vt) {\n    swigfaissJNI.HNSW_add_with_locks(swigCPtr, this, DistanceComputer.getCPtr(ptdis), ptdis, pt_level, pt_id, SWIGTYPE_p_std__vectorT_omp_lock_t_t.getCPtr(locks), VisitedTable.getCPtr(vt), vt);\n  }\n\n  public int search_from_candidates(DistanceComputer qdis, int k, LongVector I, SWIGTYPE_p_float D, HNSW.MinimaxHeap candidates, VisitedTable vt, HNSWStats stats, int level, int nres_in) {\n    return swigfaissJNI.HNSW_search_from_candidates__SWIG_0(swigCPtr, this, DistanceComputer.getCPtr(qdis), qdis, k, SWIGTYPE_p_long_long.getCPtr(I.data()), I, SWIGTYPE_p_float.getCPtr(D), HNSW.MinimaxHeap.getCPtr(candidates), candidates, VisitedTable.getCPtr(vt), vt, HNSWStats.getCPtr(stats), stats, level, nres_in);\n  }\n\n  public int search_from_candidates(DistanceComputer qdis, int k, LongVector I, SWIGTYPE_p_float D, HNSW.MinimaxHeap candidates, VisitedTable vt, HNSWStats stats, int level) {\n    return swigfaissJNI.HNSW_search_from_candidates__SWIG_1(swigCPtr, this, DistanceComputer.getCPtr(qdis), qdis, k, SWIGTYPE_p_long_long.getCPtr(I.data()), I, SWIGTYPE_p_float.getCPtr(D), HNSW.MinimaxHeap.getCPtr(candidates), candidates, VisitedTable.getCPtr(vt), vt, HNSWStats.getCPtr(stats), stats, level);\n  }\n\n  public SWIGTYPE_p_std__priority_queueT_std__pairT_float_int_t_t search_from_candidate_unbounded(SWIGTYPE_p_std__pairT_float_int_t node, DistanceComputer qdis, int ef, VisitedTable vt, HNSWStats stats) {\n    return new SWIGTYPE_p_std__priority_queueT_std__pairT_float_int_t_t(swigfaissJNI.HNSW_search_from_candidate_unbounded(swigCPtr, this, SWIGTYPE_p_std__pairT_float_int_t.getCPtr(node), DistanceComputer.getCPtr(qdis), qdis, ef, VisitedTable.getCPtr(vt), vt, HNSWStats.getCPtr(stats), stats), true);\n  }\n\n  public HNSWStats search(DistanceComputer qdis, int k, LongVector I, SWIGTYPE_p_float D, VisitedTable vt) {\n    return new HNSWStats(swigfaissJNI.HNSW_search(swigCPtr, this, DistanceComputer.getCPtr(qdis), qdis, k, SWIGTYPE_p_long_long.getCPtr(I.data()), I, SWIGTYPE_p_float.getCPtr(D), VisitedTable.getCPtr(vt), vt), true);\n  }\n\n  public void reset() {\n    swigfaissJNI.HNSW_reset(swigCPtr, this);\n  }\n\n  public void clear_neighbor_tables(int level) {\n    swigfaissJNI.HNSW_clear_neighbor_tables(swigCPtr, this, level);\n  }\n\n  public void print_neighbor_stats(int level) {\n    swigfaissJNI.HNSW_print_neighbor_stats(swigCPtr, this, level);\n  }\n\n  public int prepare_level_tab(long n, boolean preset_levels) {\n    return swigfaissJNI.HNSW_prepare_level_tab__SWIG_0(swigCPtr, this, n, preset_levels);\n  }\n\n  public int prepare_level_tab(long n) {\n    return swigfaissJNI.HNSW_prepare_level_tab__SWIG_1(swigCPtr, this, n);\n  }\n\n  public static void shrink_neighbor_list(DistanceComputer qdis, SWIGTYPE_p_std__priority_queueT_faiss__HNSW__NodeDistFarther_t input, SWIGTYPE_p_std__vectorT_faiss__HNSW__NodeDistFarther_t output, int max_size) {\n    swigfaissJNI.HNSW_shrink_neighbor_list(DistanceComputer.getCPtr(qdis), qdis, SWIGTYPE_p_std__priority_queueT_faiss__HNSW__NodeDistFarther_t.getCPtr(input), SWIGTYPE_p_std__vectorT_faiss__HNSW__NodeDistFarther_t.getCPtr(output), max_size);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/HNSWStats.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class HNSWStats {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected HNSWStats(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(HNSWStats obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_HNSWStats(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setN1(long value) {\n    swigfaissJNI.HNSWStats_n1_set(swigCPtr, this, value);\n  }\n\n  public long getN1() {\n    return swigfaissJNI.HNSWStats_n1_get(swigCPtr, this);\n  }\n\n  public void setN2(long value) {\n    swigfaissJNI.HNSWStats_n2_set(swigCPtr, this, value);\n  }\n\n  public long getN2() {\n    return swigfaissJNI.HNSWStats_n2_get(swigCPtr, this);\n  }\n\n  public void setN3(long value) {\n    swigfaissJNI.HNSWStats_n3_set(swigCPtr, this, value);\n  }\n\n  public long getN3() {\n    return swigfaissJNI.HNSWStats_n3_get(swigCPtr, this);\n  }\n\n  public void setNdis(long value) {\n    swigfaissJNI.HNSWStats_ndis_set(swigCPtr, this, value);\n  }\n\n  public long getNdis() {\n    return swigfaissJNI.HNSWStats_ndis_get(swigCPtr, this);\n  }\n\n  public void setNreorder(long value) {\n    swigfaissJNI.HNSWStats_nreorder_set(swigCPtr, this, value);\n  }\n\n  public long getNreorder() {\n    return swigfaissJNI.HNSWStats_nreorder_get(swigCPtr, this);\n  }\n\n  public HNSWStats(long n1, long n2, long n3, long ndis, long nreorder) {\n    this(swigfaissJNI.new_HNSWStats__SWIG_0(n1, n2, n3, ndis, nreorder), true);\n  }\n\n  public HNSWStats(long n1, long n2, long n3, long ndis) {\n    this(swigfaissJNI.new_HNSWStats__SWIG_1(n1, n2, n3, ndis), true);\n  }\n\n  public HNSWStats(long n1, long n2, long n3) {\n    this(swigfaissJNI.new_HNSWStats__SWIG_2(n1, n2, n3), true);\n  }\n\n  public HNSWStats(long n1, long n2) {\n    this(swigfaissJNI.new_HNSWStats__SWIG_3(n1, n2), true);\n  }\n\n  public HNSWStats(long n1) {\n    this(swigfaissJNI.new_HNSWStats__SWIG_4(n1), true);\n  }\n\n  public HNSWStats() {\n    this(swigfaissJNI.new_HNSWStats__SWIG_5(), true);\n  }\n\n  public void reset() {\n    swigfaissJNI.HNSWStats_reset(swigCPtr, this);\n  }\n\n  public void combine(HNSWStats other) {\n    swigfaissJNI.HNSWStats_combine(swigCPtr, this, HNSWStats.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/HStackInvertedLists.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class HStackInvertedLists extends ReadOnlyInvertedLists {\n  private transient long swigCPtr;\n\n  protected HStackInvertedLists(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.HStackInvertedLists_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(HStackInvertedLists obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_HStackInvertedLists(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setIls(SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t value) {\n    swigfaissJNI.HStackInvertedLists_ils_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t getIls() {\n    long cPtr = swigfaissJNI.HStackInvertedLists_ils_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t(cPtr, false);\n  }\n\n  public HStackInvertedLists(int nil, SWIGTYPE_p_p_faiss__InvertedLists ils) {\n    this(swigfaissJNI.new_HStackInvertedLists(nil, SWIGTYPE_p_p_faiss__InvertedLists.getCPtr(ils)), true);\n  }\n\n  public long list_size(long list_no) {\n    return swigfaissJNI.HStackInvertedLists_list_size(swigCPtr, this, list_no);\n  }\n\n  public SWIGTYPE_p_unsigned_char get_codes(long list_no) {\n    long cPtr = swigfaissJNI.HStackInvertedLists_get_codes(swigCPtr, this, list_no);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public LongVector get_ids(long list_no) {\n    return new LongVector(swigfaissJNI.HStackInvertedLists_get_ids(swigCPtr, this, list_no), false);\n}\n\n  public void prefetch_lists(LongVector list_nos, int nlist) {\n    swigfaissJNI.HStackInvertedLists_prefetch_lists(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, nlist);\n  }\n\n  public void release_codes(long list_no, SWIGTYPE_p_unsigned_char codes) {\n    swigfaissJNI.HStackInvertedLists_release_codes(swigCPtr, this, list_no, SWIGTYPE_p_unsigned_char.getCPtr(codes));\n  }\n\n  public void release_ids(long list_no, LongVector ids) {\n    swigfaissJNI.HStackInvertedLists_release_ids(swigCPtr, this, list_no, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids);\n  }\n\n  public long get_single_id(long list_no, long offset) {\n    return swigfaissJNI.HStackInvertedLists_get_single_id(swigCPtr, this, list_no, offset);\n}\n\n  public SWIGTYPE_p_unsigned_char get_single_code(long list_no, long offset) {\n    long cPtr = swigfaissJNI.HStackInvertedLists_get_single_code(swigCPtr, this, list_no, offset);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/HammingComputer16.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class HammingComputer16 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected HammingComputer16(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(HammingComputer16 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_HammingComputer16(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA0(long value) {\n    swigfaissJNI.HammingComputer16_a0_set(swigCPtr, this, value);\n  }\n\n  public long getA0() {\n    return swigfaissJNI.HammingComputer16_a0_get(swigCPtr, this);\n  }\n\n  public void setA1(long value) {\n    swigfaissJNI.HammingComputer16_a1_set(swigCPtr, this, value);\n  }\n\n  public long getA1() {\n    return swigfaissJNI.HammingComputer16_a1_get(swigCPtr, this);\n  }\n\n  public HammingComputer16() {\n    this(swigfaissJNI.new_HammingComputer16__SWIG_0(), true);\n  }\n\n  public HammingComputer16(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    this(swigfaissJNI.new_HammingComputer16__SWIG_1(SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size), true);\n  }\n\n  public void set(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    swigfaissJNI.HammingComputer16_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b8) {\n    return swigfaissJNI.HammingComputer16_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b8));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/HammingComputer20.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class HammingComputer20 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected HammingComputer20(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(HammingComputer20 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_HammingComputer20(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA0(long value) {\n    swigfaissJNI.HammingComputer20_a0_set(swigCPtr, this, value);\n  }\n\n  public long getA0() {\n    return swigfaissJNI.HammingComputer20_a0_get(swigCPtr, this);\n  }\n\n  public void setA1(long value) {\n    swigfaissJNI.HammingComputer20_a1_set(swigCPtr, this, value);\n  }\n\n  public long getA1() {\n    return swigfaissJNI.HammingComputer20_a1_get(swigCPtr, this);\n  }\n\n  public void setA2(SWIGTYPE_p_uint32_t value) {\n    swigfaissJNI.HammingComputer20_a2_set(swigCPtr, this, SWIGTYPE_p_uint32_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_uint32_t getA2() {\n    return new SWIGTYPE_p_uint32_t(swigfaissJNI.HammingComputer20_a2_get(swigCPtr, this), true);\n  }\n\n  public HammingComputer20() {\n    this(swigfaissJNI.new_HammingComputer20__SWIG_0(), true);\n  }\n\n  public HammingComputer20(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    this(swigfaissJNI.new_HammingComputer20__SWIG_1(SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size), true);\n  }\n\n  public void set(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    swigfaissJNI.HammingComputer20_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b8) {\n    return swigfaissJNI.HammingComputer20_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b8));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/HammingComputer32.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class HammingComputer32 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected HammingComputer32(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(HammingComputer32 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_HammingComputer32(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA0(long value) {\n    swigfaissJNI.HammingComputer32_a0_set(swigCPtr, this, value);\n  }\n\n  public long getA0() {\n    return swigfaissJNI.HammingComputer32_a0_get(swigCPtr, this);\n  }\n\n  public void setA1(long value) {\n    swigfaissJNI.HammingComputer32_a1_set(swigCPtr, this, value);\n  }\n\n  public long getA1() {\n    return swigfaissJNI.HammingComputer32_a1_get(swigCPtr, this);\n  }\n\n  public void setA2(long value) {\n    swigfaissJNI.HammingComputer32_a2_set(swigCPtr, this, value);\n  }\n\n  public long getA2() {\n    return swigfaissJNI.HammingComputer32_a2_get(swigCPtr, this);\n  }\n\n  public void setA3(long value) {\n    swigfaissJNI.HammingComputer32_a3_set(swigCPtr, this, value);\n  }\n\n  public long getA3() {\n    return swigfaissJNI.HammingComputer32_a3_get(swigCPtr, this);\n  }\n\n  public HammingComputer32() {\n    this(swigfaissJNI.new_HammingComputer32__SWIG_0(), true);\n  }\n\n  public HammingComputer32(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    this(swigfaissJNI.new_HammingComputer32__SWIG_1(SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size), true);\n  }\n\n  public void set(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    swigfaissJNI.HammingComputer32_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b8) {\n    return swigfaissJNI.HammingComputer32_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b8));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/HammingComputer4.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class HammingComputer4 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected HammingComputer4(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(HammingComputer4 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_HammingComputer4(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA0(SWIGTYPE_p_uint32_t value) {\n    swigfaissJNI.HammingComputer4_a0_set(swigCPtr, this, SWIGTYPE_p_uint32_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_uint32_t getA0() {\n    return new SWIGTYPE_p_uint32_t(swigfaissJNI.HammingComputer4_a0_get(swigCPtr, this), true);\n  }\n\n  public HammingComputer4() {\n    this(swigfaissJNI.new_HammingComputer4__SWIG_0(), true);\n  }\n\n  public HammingComputer4(SWIGTYPE_p_unsigned_char a, int code_size) {\n    this(swigfaissJNI.new_HammingComputer4__SWIG_1(SWIGTYPE_p_unsigned_char.getCPtr(a), code_size), true);\n  }\n\n  public void set(SWIGTYPE_p_unsigned_char a, int code_size) {\n    swigfaissJNI.HammingComputer4_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(a), code_size);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b) {\n    return swigfaissJNI.HammingComputer4_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/HammingComputer64.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class HammingComputer64 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected HammingComputer64(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(HammingComputer64 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_HammingComputer64(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA0(long value) {\n    swigfaissJNI.HammingComputer64_a0_set(swigCPtr, this, value);\n  }\n\n  public long getA0() {\n    return swigfaissJNI.HammingComputer64_a0_get(swigCPtr, this);\n  }\n\n  public void setA1(long value) {\n    swigfaissJNI.HammingComputer64_a1_set(swigCPtr, this, value);\n  }\n\n  public long getA1() {\n    return swigfaissJNI.HammingComputer64_a1_get(swigCPtr, this);\n  }\n\n  public void setA2(long value) {\n    swigfaissJNI.HammingComputer64_a2_set(swigCPtr, this, value);\n  }\n\n  public long getA2() {\n    return swigfaissJNI.HammingComputer64_a2_get(swigCPtr, this);\n  }\n\n  public void setA3(long value) {\n    swigfaissJNI.HammingComputer64_a3_set(swigCPtr, this, value);\n  }\n\n  public long getA3() {\n    return swigfaissJNI.HammingComputer64_a3_get(swigCPtr, this);\n  }\n\n  public void setA4(long value) {\n    swigfaissJNI.HammingComputer64_a4_set(swigCPtr, this, value);\n  }\n\n  public long getA4() {\n    return swigfaissJNI.HammingComputer64_a4_get(swigCPtr, this);\n  }\n\n  public void setA5(long value) {\n    swigfaissJNI.HammingComputer64_a5_set(swigCPtr, this, value);\n  }\n\n  public long getA5() {\n    return swigfaissJNI.HammingComputer64_a5_get(swigCPtr, this);\n  }\n\n  public void setA6(long value) {\n    swigfaissJNI.HammingComputer64_a6_set(swigCPtr, this, value);\n  }\n\n  public long getA6() {\n    return swigfaissJNI.HammingComputer64_a6_get(swigCPtr, this);\n  }\n\n  public void setA7(long value) {\n    swigfaissJNI.HammingComputer64_a7_set(swigCPtr, this, value);\n  }\n\n  public long getA7() {\n    return swigfaissJNI.HammingComputer64_a7_get(swigCPtr, this);\n  }\n\n  public HammingComputer64() {\n    this(swigfaissJNI.new_HammingComputer64__SWIG_0(), true);\n  }\n\n  public HammingComputer64(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    this(swigfaissJNI.new_HammingComputer64__SWIG_1(SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size), true);\n  }\n\n  public void set(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    swigfaissJNI.HammingComputer64_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b8) {\n    return swigfaissJNI.HammingComputer64_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b8));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/HammingComputer8.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class HammingComputer8 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected HammingComputer8(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(HammingComputer8 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_HammingComputer8(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA0(long value) {\n    swigfaissJNI.HammingComputer8_a0_set(swigCPtr, this, value);\n  }\n\n  public long getA0() {\n    return swigfaissJNI.HammingComputer8_a0_get(swigCPtr, this);\n  }\n\n  public HammingComputer8() {\n    this(swigfaissJNI.new_HammingComputer8__SWIG_0(), true);\n  }\n\n  public HammingComputer8(SWIGTYPE_p_unsigned_char a, int code_size) {\n    this(swigfaissJNI.new_HammingComputer8__SWIG_1(SWIGTYPE_p_unsigned_char.getCPtr(a), code_size), true);\n  }\n\n  public void set(SWIGTYPE_p_unsigned_char a, int code_size) {\n    swigfaissJNI.HammingComputer8_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(a), code_size);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b) {\n    return swigfaissJNI.HammingComputer8_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/HammingComputerDefault.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class HammingComputerDefault {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected HammingComputerDefault(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(HammingComputerDefault obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_HammingComputerDefault(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA8(SWIGTYPE_p_unsigned_char value) {\n    swigfaissJNI.HammingComputerDefault_a8_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_unsigned_char getA8() {\n    long cPtr = swigfaissJNI.HammingComputerDefault_a8_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public void setQuotient8(int value) {\n    swigfaissJNI.HammingComputerDefault_quotient8_set(swigCPtr, this, value);\n  }\n\n  public int getQuotient8() {\n    return swigfaissJNI.HammingComputerDefault_quotient8_get(swigCPtr, this);\n  }\n\n  public void setRemainder8(int value) {\n    swigfaissJNI.HammingComputerDefault_remainder8_set(swigCPtr, this, value);\n  }\n\n  public int getRemainder8() {\n    return swigfaissJNI.HammingComputerDefault_remainder8_get(swigCPtr, this);\n  }\n\n  public HammingComputerDefault() {\n    this(swigfaissJNI.new_HammingComputerDefault__SWIG_0(), true);\n  }\n\n  public HammingComputerDefault(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    this(swigfaissJNI.new_HammingComputerDefault__SWIG_1(SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size), true);\n  }\n\n  public void set(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    swigfaissJNI.HammingComputerDefault_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b8) {\n    return swigfaissJNI.HammingComputerDefault_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b8));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/HammingComputerM4.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class HammingComputerM4 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected HammingComputerM4(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(HammingComputerM4 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_HammingComputerM4(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA(SWIGTYPE_p_uint32_t value) {\n    swigfaissJNI.HammingComputerM4_a_set(swigCPtr, this, SWIGTYPE_p_uint32_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_uint32_t getA() {\n    long cPtr = swigfaissJNI.HammingComputerM4_a_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_uint32_t(cPtr, false);\n  }\n\n  public void setN(int value) {\n    swigfaissJNI.HammingComputerM4_n_set(swigCPtr, this, value);\n  }\n\n  public int getN() {\n    return swigfaissJNI.HammingComputerM4_n_get(swigCPtr, this);\n  }\n\n  public HammingComputerM4() {\n    this(swigfaissJNI.new_HammingComputerM4__SWIG_0(), true);\n  }\n\n  public HammingComputerM4(SWIGTYPE_p_unsigned_char a4, int code_size) {\n    this(swigfaissJNI.new_HammingComputerM4__SWIG_1(SWIGTYPE_p_unsigned_char.getCPtr(a4), code_size), true);\n  }\n\n  public void set(SWIGTYPE_p_unsigned_char a4, int code_size) {\n    swigfaissJNI.HammingComputerM4_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(a4), code_size);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b8) {\n    return swigfaissJNI.HammingComputerM4_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b8));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/HammingComputerM8.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class HammingComputerM8 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected HammingComputerM8(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(HammingComputerM8 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_HammingComputerM8(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setA(SWIGTYPE_p_unsigned_long value) {\n    swigfaissJNI.HammingComputerM8_a_set(swigCPtr, this, SWIGTYPE_p_unsigned_long.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_unsigned_long getA() {\n    long cPtr = swigfaissJNI.HammingComputerM8_a_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_long(cPtr, false);\n  }\n\n  public void setN(int value) {\n    swigfaissJNI.HammingComputerM8_n_set(swigCPtr, this, value);\n  }\n\n  public int getN() {\n    return swigfaissJNI.HammingComputerM8_n_get(swigCPtr, this);\n  }\n\n  public HammingComputerM8() {\n    this(swigfaissJNI.new_HammingComputerM8__SWIG_0(), true);\n  }\n\n  public HammingComputerM8(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    this(swigfaissJNI.new_HammingComputerM8__SWIG_1(SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size), true);\n  }\n\n  public void set(SWIGTYPE_p_unsigned_char a8, int code_size) {\n    swigfaissJNI.HammingComputerM8_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(a8), code_size);\n  }\n\n  public int hamming(SWIGTYPE_p_unsigned_char b8) {\n    return swigfaissJNI.HammingComputerM8_hamming(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(b8));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IDSelector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IDSelector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected IDSelector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IDSelector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IDSelector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public boolean is_member(long id) {\n    return swigfaissJNI.IDSelector_is_member(swigCPtr, this, id);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IDSelectorArray.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IDSelectorArray extends IDSelector {\n  private transient long swigCPtr;\n\n  protected IDSelectorArray(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IDSelectorArray_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IDSelectorArray obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IDSelectorArray(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setN(long value) {\n    swigfaissJNI.IDSelectorArray_n_set(swigCPtr, this, value);\n  }\n\n  public long getN() {\n    return swigfaissJNI.IDSelectorArray_n_get(swigCPtr, this);\n  }\n\n  public void setIds(LongVector value) {\n    swigfaissJNI.IDSelectorArray_ids_set(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(value.data()), value);\n  }\n\n  public LongVector getIds() {\n    return new LongVector(swigfaissJNI.IDSelectorArray_ids_get(swigCPtr, this), false);\n}\n\n  public IDSelectorArray(long n, LongVector ids) {\n    this(swigfaissJNI.new_IDSelectorArray(n, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids), true);\n  }\n\n  public boolean is_member(long id) {\n    return swigfaissJNI.IDSelectorArray_is_member(swigCPtr, this, id);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IDSelectorBatch.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IDSelectorBatch extends IDSelector {\n  private transient long swigCPtr;\n\n  protected IDSelectorBatch(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IDSelectorBatch_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IDSelectorBatch obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IDSelectorBatch(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setNbits(int value) {\n    swigfaissJNI.IDSelectorBatch_nbits_set(swigCPtr, this, value);\n  }\n\n  public int getNbits() {\n    return swigfaissJNI.IDSelectorBatch_nbits_get(swigCPtr, this);\n  }\n\n  public void setMask(long value) {\n    swigfaissJNI.IDSelectorBatch_mask_set(swigCPtr, this, value);\n  }\n\n  public long getMask() {\n    return swigfaissJNI.IDSelectorBatch_mask_get(swigCPtr, this);\n}\n\n  public IDSelectorBatch(long n, LongVector indices) {\n    this(swigfaissJNI.new_IDSelectorBatch(n, SWIGTYPE_p_long_long.getCPtr(indices.data()), indices), true);\n  }\n\n  public boolean is_member(long id) {\n    return swigfaissJNI.IDSelectorBatch_is_member(swigCPtr, this, id);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IDSelectorRange.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IDSelectorRange extends IDSelector {\n  private transient long swigCPtr;\n\n  protected IDSelectorRange(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IDSelectorRange_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IDSelectorRange obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IDSelectorRange(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setImin(long value) {\n    swigfaissJNI.IDSelectorRange_imin_set(swigCPtr, this, value);\n  }\n\n  public long getImin() {\n    return swigfaissJNI.IDSelectorRange_imin_get(swigCPtr, this);\n}\n\n  public void setImax(long value) {\n    swigfaissJNI.IDSelectorRange_imax_set(swigCPtr, this, value);\n  }\n\n  public long getImax() {\n    return swigfaissJNI.IDSelectorRange_imax_get(swigCPtr, this);\n}\n\n  public IDSelectorRange(long imin, long imax) {\n    this(swigfaissJNI.new_IDSelectorRange(imin, imax), true);\n  }\n\n  public boolean is_member(long id) {\n    return swigfaissJNI.IDSelectorRange_is_member(swigCPtr, this, id);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ITQMatrix.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ITQMatrix extends LinearTransform {\n  private transient long swigCPtr;\n\n  protected ITQMatrix(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.ITQMatrix_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ITQMatrix obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ITQMatrix(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setMax_iter(int value) {\n    swigfaissJNI.ITQMatrix_max_iter_set(swigCPtr, this, value);\n  }\n\n  public int getMax_iter() {\n    return swigfaissJNI.ITQMatrix_max_iter_get(swigCPtr, this);\n  }\n\n  public void setSeed(int value) {\n    swigfaissJNI.ITQMatrix_seed_set(swigCPtr, this, value);\n  }\n\n  public int getSeed() {\n    return swigfaissJNI.ITQMatrix_seed_get(swigCPtr, this);\n  }\n\n  public void setInit_rotation(DoubleVector value) {\n    swigfaissJNI.ITQMatrix_init_rotation_set(swigCPtr, this, DoubleVector.getCPtr(value), value);\n  }\n\n  public DoubleVector getInit_rotation() {\n    long cPtr = swigfaissJNI.ITQMatrix_init_rotation_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new DoubleVector(cPtr, false);\n  }\n\n  public ITQMatrix(int d) {\n    this(swigfaissJNI.new_ITQMatrix__SWIG_0(d), true);\n  }\n\n  public ITQMatrix() {\n    this(swigfaissJNI.new_ITQMatrix__SWIG_1(), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.ITQMatrix_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ITQTransform.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ITQTransform extends VectorTransform {\n  private transient long swigCPtr;\n\n  protected ITQTransform(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.ITQTransform_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ITQTransform obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ITQTransform(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setMean(FloatVector value) {\n    swigfaissJNI.ITQTransform_mean_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getMean() {\n    long cPtr = swigfaissJNI.ITQTransform_mean_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public void setDo_pca(boolean value) {\n    swigfaissJNI.ITQTransform_do_pca_set(swigCPtr, this, value);\n  }\n\n  public boolean getDo_pca() {\n    return swigfaissJNI.ITQTransform_do_pca_get(swigCPtr, this);\n  }\n\n  public void setItq(ITQMatrix value) {\n    swigfaissJNI.ITQTransform_itq_set(swigCPtr, this, ITQMatrix.getCPtr(value), value);\n  }\n\n  public ITQMatrix getItq() {\n    long cPtr = swigfaissJNI.ITQTransform_itq_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ITQMatrix(cPtr, false);\n  }\n\n  public void setMax_train_per_dim(int value) {\n    swigfaissJNI.ITQTransform_max_train_per_dim_set(swigCPtr, this, value);\n  }\n\n  public int getMax_train_per_dim() {\n    return swigfaissJNI.ITQTransform_max_train_per_dim_get(swigCPtr, this);\n  }\n\n  public void setPca_then_itq(LinearTransform value) {\n    swigfaissJNI.ITQTransform_pca_then_itq_set(swigCPtr, this, LinearTransform.getCPtr(value), value);\n  }\n\n  public LinearTransform getPca_then_itq() {\n    long cPtr = swigfaissJNI.ITQTransform_pca_then_itq_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new LinearTransform(cPtr, false);\n  }\n\n  public ITQTransform(int d_in, int d_out, boolean do_pca) {\n    this(swigfaissJNI.new_ITQTransform__SWIG_0(d_in, d_out, do_pca), true);\n  }\n\n  public ITQTransform(int d_in, int d_out) {\n    this(swigfaissJNI.new_ITQTransform__SWIG_1(d_in, d_out), true);\n  }\n\n  public ITQTransform(int d_in) {\n    this(swigfaissJNI.new_ITQTransform__SWIG_2(d_in), true);\n  }\n\n  public ITQTransform() {\n    this(swigfaissJNI.new_ITQTransform__SWIG_3(), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.ITQTransform_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void apply_noalloc(long n, SWIGTYPE_p_float x, SWIGTYPE_p_float xt) {\n    swigfaissJNI.ITQTransform_apply_noalloc(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(xt));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IVFPQSearchParameters.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IVFPQSearchParameters extends IVFSearchParameters {\n  private transient long swigCPtr;\n\n  protected IVFPQSearchParameters(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IVFPQSearchParameters_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IVFPQSearchParameters obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IVFPQSearchParameters(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setScan_table_threshold(long value) {\n    swigfaissJNI.IVFPQSearchParameters_scan_table_threshold_set(swigCPtr, this, value);\n  }\n\n  public long getScan_table_threshold() {\n    return swigfaissJNI.IVFPQSearchParameters_scan_table_threshold_get(swigCPtr, this);\n  }\n\n  public void setPolysemous_ht(int value) {\n    swigfaissJNI.IVFPQSearchParameters_polysemous_ht_set(swigCPtr, this, value);\n  }\n\n  public int getPolysemous_ht() {\n    return swigfaissJNI.IVFPQSearchParameters_polysemous_ht_get(swigCPtr, this);\n  }\n\n  public IVFPQSearchParameters() {\n    this(swigfaissJNI.new_IVFPQSearchParameters(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IVFSearchParameters.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IVFSearchParameters {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected IVFSearchParameters(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IVFSearchParameters obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IVFSearchParameters(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setNprobe(long value) {\n    swigfaissJNI.IVFSearchParameters_nprobe_set(swigCPtr, this, value);\n  }\n\n  public long getNprobe() {\n    return swigfaissJNI.IVFSearchParameters_nprobe_get(swigCPtr, this);\n  }\n\n  public void setMax_codes(long value) {\n    swigfaissJNI.IVFSearchParameters_max_codes_set(swigCPtr, this, value);\n  }\n\n  public long getMax_codes() {\n    return swigfaissJNI.IVFSearchParameters_max_codes_get(swigCPtr, this);\n  }\n\n  public IVFSearchParameters() {\n    this(swigfaissJNI.new_IVFSearchParameters(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/Index.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class Index {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected Index(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(Index obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_Index(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setD(int value) {\n    swigfaissJNI.Index_d_set(swigCPtr, this, value);\n  }\n\n  public int getD() {\n    return swigfaissJNI.Index_d_get(swigCPtr, this);\n  }\n\n  public void setNtotal(long value) {\n    swigfaissJNI.Index_ntotal_set(swigCPtr, this, value);\n  }\n\n  public long getNtotal() {\n    return swigfaissJNI.Index_ntotal_get(swigCPtr, this);\n}\n\n  public void setVerbose(boolean value) {\n    swigfaissJNI.Index_verbose_set(swigCPtr, this, value);\n  }\n\n  public boolean getVerbose() {\n    return swigfaissJNI.Index_verbose_get(swigCPtr, this);\n  }\n\n  public void setIs_trained(boolean value) {\n    swigfaissJNI.Index_is_trained_set(swigCPtr, this, value);\n  }\n\n  public boolean getIs_trained() {\n    return swigfaissJNI.Index_is_trained_get(swigCPtr, this);\n  }\n\n  public void setMetric_type(MetricType value) {\n    swigfaissJNI.Index_metric_type_set(swigCPtr, this, value.swigValue());\n  }\n\n  public MetricType getMetric_type() {\n    return MetricType.swigToEnum(swigfaissJNI.Index_metric_type_get(swigCPtr, this));\n  }\n\n  public void setMetric_arg(float value) {\n    swigfaissJNI.Index_metric_arg_set(swigCPtr, this, value);\n  }\n\n  public float getMetric_arg() {\n    return swigfaissJNI.Index_metric_arg_get(swigCPtr, this);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.Index_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void add(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.Index_add(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void add_with_ids(long n, SWIGTYPE_p_float x, LongVector xids) {\n    swigfaissJNI.Index_add_with_ids(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids);\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.Index_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void range_search(long n, SWIGTYPE_p_float x, float radius, RangeSearchResult result) {\n    swigfaissJNI.Index_range_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), radius, RangeSearchResult.getCPtr(result), result);\n  }\n\n  public void assign(long n, SWIGTYPE_p_float x, LongVector labels, long k) {\n    swigfaissJNI.Index_assign__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, k);\n  }\n\n  public void assign(long n, SWIGTYPE_p_float x, LongVector labels) {\n    swigfaissJNI.Index_assign__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void reset() {\n    swigfaissJNI.Index_reset(swigCPtr, this);\n  }\n\n  public long remove_ids(IDSelector sel) {\n    return swigfaissJNI.Index_remove_ids(swigCPtr, this, IDSelector.getCPtr(sel), sel);\n  }\n\n  public void reconstruct(long key, SWIGTYPE_p_float recons) {\n    swigfaissJNI.Index_reconstruct(swigCPtr, this, key, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public void reconstruct_n(long i0, long ni, SWIGTYPE_p_float recons) {\n    swigfaissJNI.Index_reconstruct_n(swigCPtr, this, i0, ni, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public void search_and_reconstruct(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels, SWIGTYPE_p_float recons) {\n    swigfaissJNI.Index_search_and_reconstruct(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public void compute_residual(SWIGTYPE_p_float x, SWIGTYPE_p_float residual, long key) {\n    swigfaissJNI.Index_compute_residual(swigCPtr, this, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(residual), key);\n  }\n\n  public void compute_residual_n(long n, SWIGTYPE_p_float xs, SWIGTYPE_p_float residuals, LongVector keys) {\n    swigfaissJNI.Index_compute_residual_n(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(xs), SWIGTYPE_p_float.getCPtr(residuals), SWIGTYPE_p_long_long.getCPtr(keys.data()), keys);\n  }\n\n  public DistanceComputer get_distance_computer() {\n    long cPtr = swigfaissJNI.Index_get_distance_computer(swigCPtr, this);\n    return (cPtr == 0) ? null : new DistanceComputer(cPtr, false);\n  }\n\n  public long sa_code_size() {\n    return swigfaissJNI.Index_sa_code_size(swigCPtr, this);\n  }\n\n  public void sa_encode(long n, SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char bytes) {\n    swigfaissJNI.Index_sa_encode(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(bytes));\n  }\n\n  public void sa_decode(long n, SWIGTYPE_p_unsigned_char bytes, SWIGTYPE_p_float x) {\n    swigfaissJNI.Index_sa_decode(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(bytes), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public IndexIVF toIVF() {\n    long cPtr = swigfaissJNI.Index_toIVF(swigCPtr, this);\n    return (cPtr == 0) ? null : new IndexIVF(cPtr, false);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/Index2Layer.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class Index2Layer extends IndexFlatCodes {\n  private transient long swigCPtr;\n\n  protected Index2Layer(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.Index2Layer_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(Index2Layer obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_Index2Layer(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setQ1(Level1Quantizer value) {\n    swigfaissJNI.Index2Layer_q1_set(swigCPtr, this, Level1Quantizer.getCPtr(value), value);\n  }\n\n  public Level1Quantizer getQ1() {\n    long cPtr = swigfaissJNI.Index2Layer_q1_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new Level1Quantizer(cPtr, false);\n  }\n\n  public void setPq(ProductQuantizer value) {\n    swigfaissJNI.Index2Layer_pq_set(swigCPtr, this, ProductQuantizer.getCPtr(value), value);\n  }\n\n  public ProductQuantizer getPq() {\n    long cPtr = swigfaissJNI.Index2Layer_pq_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ProductQuantizer(cPtr, false);\n  }\n\n  public void setCode_size_1(long value) {\n    swigfaissJNI.Index2Layer_code_size_1_set(swigCPtr, this, value);\n  }\n\n  public long getCode_size_1() {\n    return swigfaissJNI.Index2Layer_code_size_1_get(swigCPtr, this);\n  }\n\n  public void setCode_size_2(long value) {\n    swigfaissJNI.Index2Layer_code_size_2_set(swigCPtr, this, value);\n  }\n\n  public long getCode_size_2() {\n    return swigfaissJNI.Index2Layer_code_size_2_get(swigCPtr, this);\n  }\n\n  public Index2Layer(Index quantizer, long nlist, int M, int nbit, MetricType metric) {\n    this(swigfaissJNI.new_Index2Layer__SWIG_0(Index.getCPtr(quantizer), quantizer, nlist, M, nbit, metric.swigValue()), true);\n  }\n\n  public Index2Layer(Index quantizer, long nlist, int M, int nbit) {\n    this(swigfaissJNI.new_Index2Layer__SWIG_1(Index.getCPtr(quantizer), quantizer, nlist, M, nbit), true);\n  }\n\n  public Index2Layer(Index quantizer, long nlist, int M) {\n    this(swigfaissJNI.new_Index2Layer__SWIG_2(Index.getCPtr(quantizer), quantizer, nlist, M), true);\n  }\n\n  public Index2Layer() {\n    this(swigfaissJNI.new_Index2Layer__SWIG_3(), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.Index2Layer_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.Index2Layer_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public DistanceComputer get_distance_computer() {\n    long cPtr = swigfaissJNI.Index2Layer_get_distance_computer(swigCPtr, this);\n    return (cPtr == 0) ? null : new DistanceComputer(cPtr, false);\n  }\n\n  public void transfer_to_IVFPQ(IndexIVFPQ other) {\n    swigfaissJNI.Index2Layer_transfer_to_IVFPQ(swigCPtr, this, IndexIVFPQ.getCPtr(other), other);\n  }\n\n  public void sa_encode(long n, SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char bytes) {\n    swigfaissJNI.Index2Layer_sa_encode(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(bytes));\n  }\n\n  public void sa_decode(long n, SWIGTYPE_p_unsigned_char bytes, SWIGTYPE_p_float x) {\n    swigfaissJNI.Index2Layer_sa_decode(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(bytes), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexBinary.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexBinary {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected IndexBinary(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexBinary obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexBinary(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setD(int value) {\n    swigfaissJNI.IndexBinary_d_set(swigCPtr, this, value);\n  }\n\n  public int getD() {\n    return swigfaissJNI.IndexBinary_d_get(swigCPtr, this);\n  }\n\n  public void setCode_size(int value) {\n    swigfaissJNI.IndexBinary_code_size_set(swigCPtr, this, value);\n  }\n\n  public int getCode_size() {\n    return swigfaissJNI.IndexBinary_code_size_get(swigCPtr, this);\n  }\n\n  public void setNtotal(long value) {\n    swigfaissJNI.IndexBinary_ntotal_set(swigCPtr, this, value);\n  }\n\n  public long getNtotal() {\n    return swigfaissJNI.IndexBinary_ntotal_get(swigCPtr, this);\n}\n\n  public void setVerbose(boolean value) {\n    swigfaissJNI.IndexBinary_verbose_set(swigCPtr, this, value);\n  }\n\n  public boolean getVerbose() {\n    return swigfaissJNI.IndexBinary_verbose_get(swigCPtr, this);\n  }\n\n  public void setIs_trained(boolean value) {\n    swigfaissJNI.IndexBinary_is_trained_set(swigCPtr, this, value);\n  }\n\n  public boolean getIs_trained() {\n    return swigfaissJNI.IndexBinary_is_trained_get(swigCPtr, this);\n  }\n\n  public void setMetric_type(MetricType value) {\n    swigfaissJNI.IndexBinary_metric_type_set(swigCPtr, this, value.swigValue());\n  }\n\n  public MetricType getMetric_type() {\n    return MetricType.swigToEnum(swigfaissJNI.IndexBinary_metric_type_get(swigCPtr, this));\n  }\n\n  public void train(long n, SWIGTYPE_p_unsigned_char x) {\n    swigfaissJNI.IndexBinary_train(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x));\n  }\n\n  public void add(long n, SWIGTYPE_p_unsigned_char x) {\n    swigfaissJNI.IndexBinary_add(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x));\n  }\n\n  public void add_with_ids(long n, SWIGTYPE_p_unsigned_char x, LongVector xids) {\n    swigfaissJNI.IndexBinary_add_with_ids(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids);\n  }\n\n  public void search(long n, SWIGTYPE_p_unsigned_char x, long k, SWIGTYPE_p_int distances, LongVector labels) {\n    swigfaissJNI.IndexBinary_search(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), k, SWIGTYPE_p_int.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void range_search(long n, SWIGTYPE_p_unsigned_char x, int radius, RangeSearchResult result) {\n    swigfaissJNI.IndexBinary_range_search(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), radius, RangeSearchResult.getCPtr(result), result);\n  }\n\n  public void assign(long n, SWIGTYPE_p_unsigned_char x, LongVector labels, long k) {\n    swigfaissJNI.IndexBinary_assign__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, k);\n  }\n\n  public void assign(long n, SWIGTYPE_p_unsigned_char x, LongVector labels) {\n    swigfaissJNI.IndexBinary_assign__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexBinary_reset(swigCPtr, this);\n  }\n\n  public long remove_ids(IDSelector sel) {\n    return swigfaissJNI.IndexBinary_remove_ids(swigCPtr, this, IDSelector.getCPtr(sel), sel);\n  }\n\n  public void reconstruct(long key, SWIGTYPE_p_unsigned_char recons) {\n    swigfaissJNI.IndexBinary_reconstruct(swigCPtr, this, key, SWIGTYPE_p_unsigned_char.getCPtr(recons));\n  }\n\n  public void reconstruct_n(long i0, long ni, SWIGTYPE_p_unsigned_char recons) {\n    swigfaissJNI.IndexBinary_reconstruct_n(swigCPtr, this, i0, ni, SWIGTYPE_p_unsigned_char.getCPtr(recons));\n  }\n\n  public void search_and_reconstruct(long n, SWIGTYPE_p_unsigned_char x, long k, SWIGTYPE_p_int distances, LongVector labels, SWIGTYPE_p_unsigned_char recons) {\n    swigfaissJNI.IndexBinary_search_and_reconstruct(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), k, SWIGTYPE_p_int.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, SWIGTYPE_p_unsigned_char.getCPtr(recons));\n  }\n\n  public void display() {\n    swigfaissJNI.IndexBinary_display(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexBinaryFlat.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexBinaryFlat extends IndexBinary {\n  private transient long swigCPtr;\n\n  protected IndexBinaryFlat(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexBinaryFlat_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexBinaryFlat obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexBinaryFlat(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setXb(ByteVector value) {\n    swigfaissJNI.IndexBinaryFlat_xb_set(swigCPtr, this, ByteVector.getCPtr(value), value);\n  }\n\n  public ByteVector getXb() {\n    long cPtr = swigfaissJNI.IndexBinaryFlat_xb_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ByteVector(cPtr, false);\n  }\n\n  public void setUse_heap(boolean value) {\n    swigfaissJNI.IndexBinaryFlat_use_heap_set(swigCPtr, this, value);\n  }\n\n  public boolean getUse_heap() {\n    return swigfaissJNI.IndexBinaryFlat_use_heap_get(swigCPtr, this);\n  }\n\n  public void setQuery_batch_size(long value) {\n    swigfaissJNI.IndexBinaryFlat_query_batch_size_set(swigCPtr, this, value);\n  }\n\n  public long getQuery_batch_size() {\n    return swigfaissJNI.IndexBinaryFlat_query_batch_size_get(swigCPtr, this);\n  }\n\n  public IndexBinaryFlat(long d) {\n    this(swigfaissJNI.new_IndexBinaryFlat__SWIG_0(d), true);\n  }\n\n  public void add(long n, SWIGTYPE_p_unsigned_char x) {\n    swigfaissJNI.IndexBinaryFlat_add(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x));\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexBinaryFlat_reset(swigCPtr, this);\n  }\n\n  public void search(long n, SWIGTYPE_p_unsigned_char x, long k, SWIGTYPE_p_int distances, LongVector labels) {\n    swigfaissJNI.IndexBinaryFlat_search(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), k, SWIGTYPE_p_int.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void range_search(long n, SWIGTYPE_p_unsigned_char x, int radius, RangeSearchResult result) {\n    swigfaissJNI.IndexBinaryFlat_range_search(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), radius, RangeSearchResult.getCPtr(result), result);\n  }\n\n  public void reconstruct(long key, SWIGTYPE_p_unsigned_char recons) {\n    swigfaissJNI.IndexBinaryFlat_reconstruct(swigCPtr, this, key, SWIGTYPE_p_unsigned_char.getCPtr(recons));\n  }\n\n  public long remove_ids(IDSelector sel) {\n    return swigfaissJNI.IndexBinaryFlat_remove_ids(swigCPtr, this, IDSelector.getCPtr(sel), sel);\n  }\n\n  public IndexBinaryFlat() {\n    this(swigfaissJNI.new_IndexBinaryFlat__SWIG_1(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexBinaryFromFloat.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexBinaryFromFloat extends IndexBinary {\n  private transient long swigCPtr;\n\n  protected IndexBinaryFromFloat(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexBinaryFromFloat_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexBinaryFromFloat obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexBinaryFromFloat(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setIndex(Index value) {\n    swigfaissJNI.IndexBinaryFromFloat_index_set(swigCPtr, this, Index.getCPtr(value), value);\n  }\n\n  public Index getIndex() {\n    long cPtr = swigfaissJNI.IndexBinaryFromFloat_index_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new Index(cPtr, false);\n  }\n\n  public void setOwn_fields(boolean value) {\n    swigfaissJNI.IndexBinaryFromFloat_own_fields_set(swigCPtr, this, value);\n  }\n\n  public boolean getOwn_fields() {\n    return swigfaissJNI.IndexBinaryFromFloat_own_fields_get(swigCPtr, this);\n  }\n\n  public IndexBinaryFromFloat() {\n    this(swigfaissJNI.new_IndexBinaryFromFloat__SWIG_0(), true);\n  }\n\n  public IndexBinaryFromFloat(Index index) {\n    this(swigfaissJNI.new_IndexBinaryFromFloat__SWIG_1(Index.getCPtr(index), index), true);\n  }\n\n  public void add(long n, SWIGTYPE_p_unsigned_char x) {\n    swigfaissJNI.IndexBinaryFromFloat_add(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x));\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexBinaryFromFloat_reset(swigCPtr, this);\n  }\n\n  public void search(long n, SWIGTYPE_p_unsigned_char x, long k, SWIGTYPE_p_int distances, LongVector labels) {\n    swigfaissJNI.IndexBinaryFromFloat_search(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), k, SWIGTYPE_p_int.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void train(long n, SWIGTYPE_p_unsigned_char x) {\n    swigfaissJNI.IndexBinaryFromFloat_train(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexBinaryHNSW.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexBinaryHNSW extends IndexBinary {\n  private transient long swigCPtr;\n\n  protected IndexBinaryHNSW(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexBinaryHNSW_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexBinaryHNSW obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexBinaryHNSW(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setHnsw(HNSW value) {\n    swigfaissJNI.IndexBinaryHNSW_hnsw_set(swigCPtr, this, HNSW.getCPtr(value), value);\n  }\n\n  public HNSW getHnsw() {\n    long cPtr = swigfaissJNI.IndexBinaryHNSW_hnsw_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new HNSW(cPtr, false);\n  }\n\n  public void setOwn_fields(boolean value) {\n    swigfaissJNI.IndexBinaryHNSW_own_fields_set(swigCPtr, this, value);\n  }\n\n  public boolean getOwn_fields() {\n    return swigfaissJNI.IndexBinaryHNSW_own_fields_get(swigCPtr, this);\n  }\n\n  public void setStorage(IndexBinary value) {\n    swigfaissJNI.IndexBinaryHNSW_storage_set(swigCPtr, this, IndexBinary.getCPtr(value), value);\n  }\n\n  public IndexBinary getStorage() {\n    long cPtr = swigfaissJNI.IndexBinaryHNSW_storage_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new IndexBinary(cPtr, false);\n  }\n\n  public IndexBinaryHNSW() {\n    this(swigfaissJNI.new_IndexBinaryHNSW__SWIG_0(), true);\n  }\n\n  public IndexBinaryHNSW(int d, int M) {\n    this(swigfaissJNI.new_IndexBinaryHNSW__SWIG_1(d, M), true);\n  }\n\n  public IndexBinaryHNSW(int d) {\n    this(swigfaissJNI.new_IndexBinaryHNSW__SWIG_2(d), true);\n  }\n\n  public IndexBinaryHNSW(IndexBinary storage, int M) {\n    this(swigfaissJNI.new_IndexBinaryHNSW__SWIG_3(IndexBinary.getCPtr(storage), storage, M), true);\n  }\n\n  public IndexBinaryHNSW(IndexBinary storage) {\n    this(swigfaissJNI.new_IndexBinaryHNSW__SWIG_4(IndexBinary.getCPtr(storage), storage), true);\n  }\n\n  public DistanceComputer get_distance_computer() {\n    long cPtr = swigfaissJNI.IndexBinaryHNSW_get_distance_computer(swigCPtr, this);\n    return (cPtr == 0) ? null : new DistanceComputer(cPtr, false);\n  }\n\n  public void add(long n, SWIGTYPE_p_unsigned_char x) {\n    swigfaissJNI.IndexBinaryHNSW_add(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x));\n  }\n\n  public void train(long n, SWIGTYPE_p_unsigned_char x) {\n    swigfaissJNI.IndexBinaryHNSW_train(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x));\n  }\n\n  public void search(long n, SWIGTYPE_p_unsigned_char x, long k, SWIGTYPE_p_int distances, LongVector labels) {\n    swigfaissJNI.IndexBinaryHNSW_search(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), k, SWIGTYPE_p_int.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void reconstruct(long key, SWIGTYPE_p_unsigned_char recons) {\n    swigfaissJNI.IndexBinaryHNSW_reconstruct(swigCPtr, this, key, SWIGTYPE_p_unsigned_char.getCPtr(recons));\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexBinaryHNSW_reset(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexBinaryIVF.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexBinaryIVF extends IndexBinary {\n  private transient long swigCPtr;\n\n  protected IndexBinaryIVF(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexBinaryIVF_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexBinaryIVF obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexBinaryIVF(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setInvlists(InvertedLists value) {\n    swigfaissJNI.IndexBinaryIVF_invlists_set(swigCPtr, this, InvertedLists.getCPtr(value), value);\n  }\n\n  public InvertedLists getInvlists() {\n    long cPtr = swigfaissJNI.IndexBinaryIVF_invlists_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n  }\n\n  public void setOwn_invlists(boolean value) {\n    swigfaissJNI.IndexBinaryIVF_own_invlists_set(swigCPtr, this, value);\n  }\n\n  public boolean getOwn_invlists() {\n    return swigfaissJNI.IndexBinaryIVF_own_invlists_get(swigCPtr, this);\n  }\n\n  public void setNprobe(long value) {\n    swigfaissJNI.IndexBinaryIVF_nprobe_set(swigCPtr, this, value);\n  }\n\n  public long getNprobe() {\n    return swigfaissJNI.IndexBinaryIVF_nprobe_get(swigCPtr, this);\n  }\n\n  public void setMax_codes(long value) {\n    swigfaissJNI.IndexBinaryIVF_max_codes_set(swigCPtr, this, value);\n  }\n\n  public long getMax_codes() {\n    return swigfaissJNI.IndexBinaryIVF_max_codes_get(swigCPtr, this);\n  }\n\n  public void setUse_heap(boolean value) {\n    swigfaissJNI.IndexBinaryIVF_use_heap_set(swigCPtr, this, value);\n  }\n\n  public boolean getUse_heap() {\n    return swigfaissJNI.IndexBinaryIVF_use_heap_get(swigCPtr, this);\n  }\n\n  public void setDirect_map(SWIGTYPE_p_DirectMap value) {\n    swigfaissJNI.IndexBinaryIVF_direct_map_set(swigCPtr, this, SWIGTYPE_p_DirectMap.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_DirectMap getDirect_map() {\n    return new SWIGTYPE_p_DirectMap(swigfaissJNI.IndexBinaryIVF_direct_map_get(swigCPtr, this), true);\n  }\n\n  public void setQuantizer(IndexBinary value) {\n    swigfaissJNI.IndexBinaryIVF_quantizer_set(swigCPtr, this, IndexBinary.getCPtr(value), value);\n  }\n\n  public IndexBinary getQuantizer() {\n    long cPtr = swigfaissJNI.IndexBinaryIVF_quantizer_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new IndexBinary(cPtr, false);\n  }\n\n  public void setNlist(long value) {\n    swigfaissJNI.IndexBinaryIVF_nlist_set(swigCPtr, this, value);\n  }\n\n  public long getNlist() {\n    return swigfaissJNI.IndexBinaryIVF_nlist_get(swigCPtr, this);\n  }\n\n  public void setOwn_fields(boolean value) {\n    swigfaissJNI.IndexBinaryIVF_own_fields_set(swigCPtr, this, value);\n  }\n\n  public boolean getOwn_fields() {\n    return swigfaissJNI.IndexBinaryIVF_own_fields_get(swigCPtr, this);\n  }\n\n  public void setCp(ClusteringParameters value) {\n    swigfaissJNI.IndexBinaryIVF_cp_set(swigCPtr, this, ClusteringParameters.getCPtr(value), value);\n  }\n\n  public ClusteringParameters getCp() {\n    long cPtr = swigfaissJNI.IndexBinaryIVF_cp_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ClusteringParameters(cPtr, false);\n  }\n\n  public void setClustering_index(Index value) {\n    swigfaissJNI.IndexBinaryIVF_clustering_index_set(swigCPtr, this, Index.getCPtr(value), value);\n  }\n\n  public Index getClustering_index() {\n    long cPtr = swigfaissJNI.IndexBinaryIVF_clustering_index_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new Index(cPtr, false);\n  }\n\n  public IndexBinaryIVF(IndexBinary quantizer, long d, long nlist) {\n    this(swigfaissJNI.new_IndexBinaryIVF__SWIG_0(IndexBinary.getCPtr(quantizer), quantizer, d, nlist), true);\n  }\n\n  public IndexBinaryIVF() {\n    this(swigfaissJNI.new_IndexBinaryIVF__SWIG_1(), true);\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexBinaryIVF_reset(swigCPtr, this);\n  }\n\n  public void train(long n, SWIGTYPE_p_unsigned_char x) {\n    swigfaissJNI.IndexBinaryIVF_train(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x));\n  }\n\n  public void add(long n, SWIGTYPE_p_unsigned_char x) {\n    swigfaissJNI.IndexBinaryIVF_add(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x));\n  }\n\n  public void add_with_ids(long n, SWIGTYPE_p_unsigned_char x, LongVector xids) {\n    swigfaissJNI.IndexBinaryIVF_add_with_ids(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids);\n  }\n\n  public void add_core(long n, SWIGTYPE_p_unsigned_char x, LongVector xids, LongVector precomputed_idx) {\n    swigfaissJNI.IndexBinaryIVF_add_core(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids, SWIGTYPE_p_long_long.getCPtr(precomputed_idx.data()), precomputed_idx);\n  }\n\n  public void search_preassigned(long n, SWIGTYPE_p_unsigned_char x, long k, LongVector assign, SWIGTYPE_p_int centroid_dis, SWIGTYPE_p_int distances, LongVector labels, boolean store_pairs, IVFSearchParameters params) {\n    swigfaissJNI.IndexBinaryIVF_search_preassigned__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), k, SWIGTYPE_p_long_long.getCPtr(assign.data()), assign, SWIGTYPE_p_int.getCPtr(centroid_dis), SWIGTYPE_p_int.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, store_pairs, IVFSearchParameters.getCPtr(params), params);\n  }\n\n  public void search_preassigned(long n, SWIGTYPE_p_unsigned_char x, long k, LongVector assign, SWIGTYPE_p_int centroid_dis, SWIGTYPE_p_int distances, LongVector labels, boolean store_pairs) {\n    swigfaissJNI.IndexBinaryIVF_search_preassigned__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), k, SWIGTYPE_p_long_long.getCPtr(assign.data()), assign, SWIGTYPE_p_int.getCPtr(centroid_dis), SWIGTYPE_p_int.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, store_pairs);\n  }\n\n  public SWIGTYPE_p_faiss__BinaryInvertedListScanner get_InvertedListScanner(boolean store_pairs) {\n    long cPtr = swigfaissJNI.IndexBinaryIVF_get_InvertedListScanner__SWIG_0(swigCPtr, this, store_pairs);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__BinaryInvertedListScanner(cPtr, false);\n  }\n\n  public SWIGTYPE_p_faiss__BinaryInvertedListScanner get_InvertedListScanner() {\n    long cPtr = swigfaissJNI.IndexBinaryIVF_get_InvertedListScanner__SWIG_1(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__BinaryInvertedListScanner(cPtr, false);\n  }\n\n  public void search(long n, SWIGTYPE_p_unsigned_char x, long k, SWIGTYPE_p_int distances, LongVector labels) {\n    swigfaissJNI.IndexBinaryIVF_search(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), k, SWIGTYPE_p_int.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void range_search(long n, SWIGTYPE_p_unsigned_char x, int radius, RangeSearchResult result) {\n    swigfaissJNI.IndexBinaryIVF_range_search(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), radius, RangeSearchResult.getCPtr(result), result);\n  }\n\n  public void range_search_preassigned(long n, SWIGTYPE_p_unsigned_char x, int radius, LongVector assign, SWIGTYPE_p_int centroid_dis, RangeSearchResult result) {\n    swigfaissJNI.IndexBinaryIVF_range_search_preassigned(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), radius, SWIGTYPE_p_long_long.getCPtr(assign.data()), assign, SWIGTYPE_p_int.getCPtr(centroid_dis), RangeSearchResult.getCPtr(result), result);\n  }\n\n  public void reconstruct(long key, SWIGTYPE_p_unsigned_char recons) {\n    swigfaissJNI.IndexBinaryIVF_reconstruct(swigCPtr, this, key, SWIGTYPE_p_unsigned_char.getCPtr(recons));\n  }\n\n  public void reconstruct_n(long i0, long ni, SWIGTYPE_p_unsigned_char recons) {\n    swigfaissJNI.IndexBinaryIVF_reconstruct_n(swigCPtr, this, i0, ni, SWIGTYPE_p_unsigned_char.getCPtr(recons));\n  }\n\n  public void search_and_reconstruct(long n, SWIGTYPE_p_unsigned_char x, long k, SWIGTYPE_p_int distances, LongVector labels, SWIGTYPE_p_unsigned_char recons) {\n    swigfaissJNI.IndexBinaryIVF_search_and_reconstruct(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(x), k, SWIGTYPE_p_int.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, SWIGTYPE_p_unsigned_char.getCPtr(recons));\n  }\n\n  public void reconstruct_from_offset(long list_no, long offset, SWIGTYPE_p_unsigned_char recons) {\n    swigfaissJNI.IndexBinaryIVF_reconstruct_from_offset(swigCPtr, this, list_no, offset, SWIGTYPE_p_unsigned_char.getCPtr(recons));\n  }\n\n  public long remove_ids(IDSelector sel) {\n    return swigfaissJNI.IndexBinaryIVF_remove_ids(swigCPtr, this, IDSelector.getCPtr(sel), sel);\n  }\n\n  public void merge_from(IndexBinaryIVF other, long add_id) {\n    swigfaissJNI.IndexBinaryIVF_merge_from(swigCPtr, this, IndexBinaryIVF.getCPtr(other), other, add_id);\n  }\n\n  public long get_list_size(long list_no) {\n    return swigfaissJNI.IndexBinaryIVF_get_list_size(swigCPtr, this, list_no);\n  }\n\n  public void make_direct_map(boolean new_maintain_direct_map) {\n    swigfaissJNI.IndexBinaryIVF_make_direct_map__SWIG_0(swigCPtr, this, new_maintain_direct_map);\n  }\n\n  public void make_direct_map() {\n    swigfaissJNI.IndexBinaryIVF_make_direct_map__SWIG_1(swigCPtr, this);\n  }\n\n  public void set_direct_map_type(SWIGTYPE_p_DirectMap__Type type) {\n    swigfaissJNI.IndexBinaryIVF_set_direct_map_type(swigCPtr, this, SWIGTYPE_p_DirectMap__Type.getCPtr(type));\n  }\n\n  public void replace_invlists(InvertedLists il, boolean own) {\n    swigfaissJNI.IndexBinaryIVF_replace_invlists__SWIG_0(swigCPtr, this, InvertedLists.getCPtr(il), il, own);\n  }\n\n  public void replace_invlists(InvertedLists il) {\n    swigfaissJNI.IndexBinaryIVF_replace_invlists__SWIG_1(swigCPtr, this, InvertedLists.getCPtr(il), il);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexFlat.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexFlat extends IndexFlatCodes {\n  private transient long swigCPtr;\n\n  protected IndexFlat(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexFlat_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexFlat obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexFlat(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public IndexFlat(long d, MetricType metric) {\n    this(swigfaissJNI.new_IndexFlat__SWIG_0(d, metric.swigValue()), true);\n  }\n\n  public IndexFlat(long d) {\n    this(swigfaissJNI.new_IndexFlat__SWIG_1(d), true);\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexFlat_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void range_search(long n, SWIGTYPE_p_float x, float radius, RangeSearchResult result) {\n    swigfaissJNI.IndexFlat_range_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), radius, RangeSearchResult.getCPtr(result), result);\n  }\n\n  public void reconstruct(long key, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexFlat_reconstruct(swigCPtr, this, key, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public void compute_distance_subset(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexFlat_compute_distance_subset(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public SWIGTYPE_p_float get_xb() {\n    long cPtr = swigfaissJNI.IndexFlat_get_xb__SWIG_0(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public IndexFlat() {\n    this(swigfaissJNI.new_IndexFlat__SWIG_2(), true);\n  }\n\n  public DistanceComputer get_distance_computer() {\n    long cPtr = swigfaissJNI.IndexFlat_get_distance_computer(swigCPtr, this);\n    return (cPtr == 0) ? null : new DistanceComputer(cPtr, false);\n  }\n\n  public void sa_encode(long n, SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char bytes) {\n    swigfaissJNI.IndexFlat_sa_encode(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(bytes));\n  }\n\n  public void sa_decode(long n, SWIGTYPE_p_unsigned_char bytes, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexFlat_sa_decode(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(bytes), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexFlat1D.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexFlat1D extends IndexFlatL2 {\n  private transient long swigCPtr;\n\n  protected IndexFlat1D(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexFlat1D_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexFlat1D obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexFlat1D(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setContinuous_update(boolean value) {\n    swigfaissJNI.IndexFlat1D_continuous_update_set(swigCPtr, this, value);\n  }\n\n  public boolean getContinuous_update() {\n    return swigfaissJNI.IndexFlat1D_continuous_update_get(swigCPtr, this);\n  }\n\n  public void setPerm(SWIGTYPE_p_std__vectorT_int64_t_t value) {\n    swigfaissJNI.IndexFlat1D_perm_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_int64_t_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_int64_t_t getPerm() {\n    long cPtr = swigfaissJNI.IndexFlat1D_perm_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_int64_t_t(cPtr, false);\n  }\n\n  public IndexFlat1D(boolean continuous_update) {\n    this(swigfaissJNI.new_IndexFlat1D__SWIG_0(continuous_update), true);\n  }\n\n  public IndexFlat1D() {\n    this(swigfaissJNI.new_IndexFlat1D__SWIG_1(), true);\n  }\n\n  public void update_permutation() {\n    swigfaissJNI.IndexFlat1D_update_permutation(swigCPtr, this);\n  }\n\n  public void add(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexFlat1D_add(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexFlat1D_reset(swigCPtr, this);\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexFlat1D_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexFlatCodes.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexFlatCodes extends Index {\n  private transient long swigCPtr;\n\n  protected IndexFlatCodes(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexFlatCodes_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexFlatCodes obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexFlatCodes(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setCode_size(long value) {\n    swigfaissJNI.IndexFlatCodes_code_size_set(swigCPtr, this, value);\n  }\n\n  public long getCode_size() {\n    return swigfaissJNI.IndexFlatCodes_code_size_get(swigCPtr, this);\n  }\n\n  public void setCodes(ByteVector value) {\n    swigfaissJNI.IndexFlatCodes_codes_set(swigCPtr, this, ByteVector.getCPtr(value), value);\n  }\n\n  public ByteVector getCodes() {\n    long cPtr = swigfaissJNI.IndexFlatCodes_codes_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ByteVector(cPtr, false);\n  }\n\n  public void add(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexFlatCodes_add(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexFlatCodes_reset(swigCPtr, this);\n  }\n\n  public void reconstruct_n(long i0, long ni, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexFlatCodes_reconstruct_n(swigCPtr, this, i0, ni, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public void reconstruct(long key, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexFlatCodes_reconstruct(swigCPtr, this, key, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public long sa_code_size() {\n    return swigfaissJNI.IndexFlatCodes_sa_code_size(swigCPtr, this);\n  }\n\n  public long remove_ids(IDSelector sel) {\n    return swigfaissJNI.IndexFlatCodes_remove_ids(swigCPtr, this, IDSelector.getCPtr(sel), sel);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexFlatIP.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexFlatIP extends IndexFlat {\n  private transient long swigCPtr;\n\n  protected IndexFlatIP(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexFlatIP_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexFlatIP obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexFlatIP(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public IndexFlatIP(long d) {\n    this(swigfaissJNI.new_IndexFlatIP__SWIG_0(d), true);\n  }\n\n  public IndexFlatIP() {\n    this(swigfaissJNI.new_IndexFlatIP__SWIG_1(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexFlatL2.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexFlatL2 extends IndexFlat {\n  private transient long swigCPtr;\n\n  protected IndexFlatL2(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexFlatL2_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexFlatL2 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexFlatL2(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public IndexFlatL2(long d) {\n    this(swigfaissJNI.new_IndexFlatL2__SWIG_0(d), true);\n  }\n\n  public IndexFlatL2() {\n    this(swigfaissJNI.new_IndexFlatL2__SWIG_1(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexHNSW.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexHNSW extends Index {\n  private transient long swigCPtr;\n\n  protected IndexHNSW(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexHNSW_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexHNSW obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexHNSW(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setHnsw(HNSW value) {\n    swigfaissJNI.IndexHNSW_hnsw_set(swigCPtr, this, HNSW.getCPtr(value), value);\n  }\n\n  public HNSW getHnsw() {\n    long cPtr = swigfaissJNI.IndexHNSW_hnsw_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new HNSW(cPtr, false);\n  }\n\n  public void setOwn_fields(boolean value) {\n    swigfaissJNI.IndexHNSW_own_fields_set(swigCPtr, this, value);\n  }\n\n  public boolean getOwn_fields() {\n    return swigfaissJNI.IndexHNSW_own_fields_get(swigCPtr, this);\n  }\n\n  public void setStorage(Index value) {\n    swigfaissJNI.IndexHNSW_storage_set(swigCPtr, this, Index.getCPtr(value), value);\n  }\n\n  public Index getStorage() {\n    long cPtr = swigfaissJNI.IndexHNSW_storage_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new Index(cPtr, false);\n  }\n\n  public void setReconstruct_from_neighbors(ReconstructFromNeighbors value) {\n    swigfaissJNI.IndexHNSW_reconstruct_from_neighbors_set(swigCPtr, this, ReconstructFromNeighbors.getCPtr(value), value);\n  }\n\n  public ReconstructFromNeighbors getReconstruct_from_neighbors() {\n    long cPtr = swigfaissJNI.IndexHNSW_reconstruct_from_neighbors_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ReconstructFromNeighbors(cPtr, false);\n  }\n\n  public IndexHNSW(int d, int M, MetricType metric) {\n    this(swigfaissJNI.new_IndexHNSW__SWIG_0(d, M, metric.swigValue()), true);\n  }\n\n  public IndexHNSW(int d, int M) {\n    this(swigfaissJNI.new_IndexHNSW__SWIG_1(d, M), true);\n  }\n\n  public IndexHNSW(int d) {\n    this(swigfaissJNI.new_IndexHNSW__SWIG_2(d), true);\n  }\n\n  public IndexHNSW() {\n    this(swigfaissJNI.new_IndexHNSW__SWIG_3(), true);\n  }\n\n  public IndexHNSW(Index storage, int M) {\n    this(swigfaissJNI.new_IndexHNSW__SWIG_4(Index.getCPtr(storage), storage, M), true);\n  }\n\n  public IndexHNSW(Index storage) {\n    this(swigfaissJNI.new_IndexHNSW__SWIG_5(Index.getCPtr(storage), storage), true);\n  }\n\n  public void add(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexHNSW_add(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexHNSW_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexHNSW_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void reconstruct(long key, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexHNSW_reconstruct(swigCPtr, this, key, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexHNSW_reset(swigCPtr, this);\n  }\n\n  public void shrink_level_0_neighbors(int size) {\n    swigfaissJNI.IndexHNSW_shrink_level_0_neighbors(swigCPtr, this, size);\n  }\n\n  public void search_level_0(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_int nearest, SWIGTYPE_p_float nearest_d, SWIGTYPE_p_float distances, LongVector labels, int nprobe, int search_type) {\n    swigfaissJNI.IndexHNSW_search_level_0__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_int.getCPtr(nearest), SWIGTYPE_p_float.getCPtr(nearest_d), SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, nprobe, search_type);\n  }\n\n  public void search_level_0(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_int nearest, SWIGTYPE_p_float nearest_d, SWIGTYPE_p_float distances, LongVector labels, int nprobe) {\n    swigfaissJNI.IndexHNSW_search_level_0__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_int.getCPtr(nearest), SWIGTYPE_p_float.getCPtr(nearest_d), SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, nprobe);\n  }\n\n  public void search_level_0(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_int nearest, SWIGTYPE_p_float nearest_d, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexHNSW_search_level_0__SWIG_2(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_int.getCPtr(nearest), SWIGTYPE_p_float.getCPtr(nearest_d), SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void init_level_0_from_knngraph(int k, SWIGTYPE_p_float D, LongVector I) {\n    swigfaissJNI.IndexHNSW_init_level_0_from_knngraph(swigCPtr, this, k, SWIGTYPE_p_float.getCPtr(D), SWIGTYPE_p_long_long.getCPtr(I.data()), I);\n  }\n\n  public void init_level_0_from_entry_points(int npt, SWIGTYPE_p_int points, SWIGTYPE_p_int nearests) {\n    swigfaissJNI.IndexHNSW_init_level_0_from_entry_points(swigCPtr, this, npt, SWIGTYPE_p_int.getCPtr(points), SWIGTYPE_p_int.getCPtr(nearests));\n  }\n\n  public void reorder_links() {\n    swigfaissJNI.IndexHNSW_reorder_links(swigCPtr, this);\n  }\n\n  public void link_singletons() {\n    swigfaissJNI.IndexHNSW_link_singletons(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexHNSW2Level.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexHNSW2Level extends IndexHNSW {\n  private transient long swigCPtr;\n\n  protected IndexHNSW2Level(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexHNSW2Level_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexHNSW2Level obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexHNSW2Level(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public IndexHNSW2Level() {\n    this(swigfaissJNI.new_IndexHNSW2Level__SWIG_0(), true);\n  }\n\n  public IndexHNSW2Level(Index quantizer, long nlist, int m_pq, int M) {\n    this(swigfaissJNI.new_IndexHNSW2Level__SWIG_1(Index.getCPtr(quantizer), quantizer, nlist, m_pq, M), true);\n  }\n\n  public void flip_to_ivf() {\n    swigfaissJNI.IndexHNSW2Level_flip_to_ivf(swigCPtr, this);\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexHNSW2Level_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexHNSWFlat.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexHNSWFlat extends IndexHNSW {\n  private transient long swigCPtr;\n\n  protected IndexHNSWFlat(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexHNSWFlat_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexHNSWFlat obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexHNSWFlat(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public IndexHNSWFlat() {\n    this(swigfaissJNI.new_IndexHNSWFlat__SWIG_0(), true);\n  }\n\n  public IndexHNSWFlat(int d, int M, MetricType metric) {\n    this(swigfaissJNI.new_IndexHNSWFlat__SWIG_1(d, M, metric.swigValue()), true);\n  }\n\n  public IndexHNSWFlat(int d, int M) {\n    this(swigfaissJNI.new_IndexHNSWFlat__SWIG_2(d, M), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexHNSWPQ.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexHNSWPQ extends IndexHNSW {\n  private transient long swigCPtr;\n\n  protected IndexHNSWPQ(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexHNSWPQ_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexHNSWPQ obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexHNSWPQ(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public IndexHNSWPQ() {\n    this(swigfaissJNI.new_IndexHNSWPQ__SWIG_0(), true);\n  }\n\n  public IndexHNSWPQ(int d, int pq_m, int M) {\n    this(swigfaissJNI.new_IndexHNSWPQ__SWIG_1(d, pq_m, M), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexHNSWPQ_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexHNSWSQ.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexHNSWSQ extends IndexHNSW {\n  private transient long swigCPtr;\n\n  protected IndexHNSWSQ(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexHNSWSQ_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexHNSWSQ obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexHNSWSQ(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public IndexHNSWSQ() {\n    this(swigfaissJNI.new_IndexHNSWSQ__SWIG_0(), true);\n  }\n\n  public IndexHNSWSQ(int d, SWIGTYPE_p_ScalarQuantizer__QuantizerType qtype, int M, MetricType metric) {\n    this(swigfaissJNI.new_IndexHNSWSQ__SWIG_1(d, SWIGTYPE_p_ScalarQuantizer__QuantizerType.getCPtr(qtype), M, metric.swigValue()), true);\n  }\n\n  public IndexHNSWSQ(int d, SWIGTYPE_p_ScalarQuantizer__QuantizerType qtype, int M) {\n    this(swigfaissJNI.new_IndexHNSWSQ__SWIG_2(d, SWIGTYPE_p_ScalarQuantizer__QuantizerType.getCPtr(qtype), M), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexIDMap.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexIDMap extends Index {\n  private transient long swigCPtr;\n\n  protected IndexIDMap(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexIDMap_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexIDMap obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexIDMap(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setIndex(Index value) {\n    swigfaissJNI.IndexIDMap_index_set(swigCPtr, this, Index.getCPtr(value), value);\n  }\n\n  public Index getIndex() {\n    long cPtr = swigfaissJNI.IndexIDMap_index_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new Index(cPtr, false);\n  }\n\n  public void setOwn_fields(boolean value) {\n    swigfaissJNI.IndexIDMap_own_fields_set(swigCPtr, this, value);\n  }\n\n  public boolean getOwn_fields() {\n    return swigfaissJNI.IndexIDMap_own_fields_get(swigCPtr, this);\n  }\n\n  public void setId_map(SWIGTYPE_p_std__vectorT_int64_t_t value) {\n    swigfaissJNI.IndexIDMap_id_map_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_int64_t_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_int64_t_t getId_map() {\n    long cPtr = swigfaissJNI.IndexIDMap_id_map_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_int64_t_t(cPtr, false);\n  }\n\n  public IndexIDMap(Index index) {\n    this(swigfaissJNI.new_IndexIDMap__SWIG_0(Index.getCPtr(index), index), true);\n  }\n\n  public void add_with_ids(long n, SWIGTYPE_p_float x, LongVector xids) {\n    swigfaissJNI.IndexIDMap_add_with_ids(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids);\n  }\n\n  public void add(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexIDMap_add(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexIDMap_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexIDMap_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexIDMap_reset(swigCPtr, this);\n  }\n\n  public long remove_ids(IDSelector sel) {\n    return swigfaissJNI.IndexIDMap_remove_ids(swigCPtr, this, IDSelector.getCPtr(sel), sel);\n  }\n\n  public void range_search(long n, SWIGTYPE_p_float x, float radius, RangeSearchResult result) {\n    swigfaissJNI.IndexIDMap_range_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), radius, RangeSearchResult.getCPtr(result), result);\n  }\n\n  public IndexIDMap() {\n    this(swigfaissJNI.new_IndexIDMap__SWIG_1(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexIVF.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexIVF extends Index {\n  private transient long swigCPtr;\n\n  protected IndexIVF(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexIVF_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexIVF obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexIVF(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setInvlists(InvertedLists value) {\n    swigfaissJNI.IndexIVF_invlists_set(swigCPtr, this, InvertedLists.getCPtr(value), value);\n  }\n\n  public InvertedLists getInvlists() {\n    long cPtr = swigfaissJNI.IndexIVF_invlists_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n  }\n\n  public void setOwn_invlists(boolean value) {\n    swigfaissJNI.IndexIVF_own_invlists_set(swigCPtr, this, value);\n  }\n\n  public boolean getOwn_invlists() {\n    return swigfaissJNI.IndexIVF_own_invlists_get(swigCPtr, this);\n  }\n\n  public void setCode_size(long value) {\n    swigfaissJNI.IndexIVF_code_size_set(swigCPtr, this, value);\n  }\n\n  public long getCode_size() {\n    return swigfaissJNI.IndexIVF_code_size_get(swigCPtr, this);\n  }\n\n  public void setNprobe(long value) {\n    swigfaissJNI.IndexIVF_nprobe_set(swigCPtr, this, value);\n  }\n\n  public long getNprobe() {\n    return swigfaissJNI.IndexIVF_nprobe_get(swigCPtr, this);\n  }\n\n  public void setMax_codes(long value) {\n    swigfaissJNI.IndexIVF_max_codes_set(swigCPtr, this, value);\n  }\n\n  public long getMax_codes() {\n    return swigfaissJNI.IndexIVF_max_codes_get(swigCPtr, this);\n  }\n\n  public void setParallel_mode(int value) {\n    swigfaissJNI.IndexIVF_parallel_mode_set(swigCPtr, this, value);\n  }\n\n  public int getParallel_mode() {\n    return swigfaissJNI.IndexIVF_parallel_mode_get(swigCPtr, this);\n  }\n\n  public int getPARALLEL_MODE_NO_HEAP_INIT() {\n    return swigfaissJNI.IndexIVF_PARALLEL_MODE_NO_HEAP_INIT_get(swigCPtr, this);\n  }\n\n  public void setDirect_map(SWIGTYPE_p_DirectMap value) {\n    swigfaissJNI.IndexIVF_direct_map_set(swigCPtr, this, SWIGTYPE_p_DirectMap.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_DirectMap getDirect_map() {\n    return new SWIGTYPE_p_DirectMap(swigfaissJNI.IndexIVF_direct_map_get(swigCPtr, this), true);\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexIVF_reset(swigCPtr, this);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexIVF_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void add(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexIVF_add(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void add_with_ids(long n, SWIGTYPE_p_float x, LongVector xids) {\n    swigfaissJNI.IndexIVF_add_with_ids(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids);\n  }\n\n  public void add_core(long n, SWIGTYPE_p_float x, LongVector xids, LongVector precomputed_idx) {\n    swigfaissJNI.IndexIVF_add_core(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids, SWIGTYPE_p_long_long.getCPtr(precomputed_idx.data()), precomputed_idx);\n  }\n\n  public void encode_vectors(long n, SWIGTYPE_p_float x, LongVector list_nos, SWIGTYPE_p_unsigned_char codes, boolean include_listno) {\n    swigfaissJNI.IndexIVF_encode_vectors__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, SWIGTYPE_p_unsigned_char.getCPtr(codes), include_listno);\n  }\n\n  public void encode_vectors(long n, SWIGTYPE_p_float x, LongVector list_nos, SWIGTYPE_p_unsigned_char codes) {\n    swigfaissJNI.IndexIVF_encode_vectors__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, SWIGTYPE_p_unsigned_char.getCPtr(codes));\n  }\n\n  public void add_sa_codes(long n, SWIGTYPE_p_unsigned_char codes, LongVector xids) {\n    swigfaissJNI.IndexIVF_add_sa_codes(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(codes), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids);\n  }\n\n  public void train_residual(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexIVF_train_residual(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void search_preassigned(long n, SWIGTYPE_p_float x, long k, LongVector assign, SWIGTYPE_p_float centroid_dis, SWIGTYPE_p_float distances, LongVector labels, boolean store_pairs, IVFSearchParameters params, IndexIVFStats stats) {\n    swigfaissJNI.IndexIVF_search_preassigned__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_long_long.getCPtr(assign.data()), assign, SWIGTYPE_p_float.getCPtr(centroid_dis), SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, store_pairs, IVFSearchParameters.getCPtr(params), params, IndexIVFStats.getCPtr(stats), stats);\n  }\n\n  public void search_preassigned(long n, SWIGTYPE_p_float x, long k, LongVector assign, SWIGTYPE_p_float centroid_dis, SWIGTYPE_p_float distances, LongVector labels, boolean store_pairs, IVFSearchParameters params) {\n    swigfaissJNI.IndexIVF_search_preassigned__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_long_long.getCPtr(assign.data()), assign, SWIGTYPE_p_float.getCPtr(centroid_dis), SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, store_pairs, IVFSearchParameters.getCPtr(params), params);\n  }\n\n  public void search_preassigned(long n, SWIGTYPE_p_float x, long k, LongVector assign, SWIGTYPE_p_float centroid_dis, SWIGTYPE_p_float distances, LongVector labels, boolean store_pairs) {\n    swigfaissJNI.IndexIVF_search_preassigned__SWIG_2(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_long_long.getCPtr(assign.data()), assign, SWIGTYPE_p_float.getCPtr(centroid_dis), SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, store_pairs);\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexIVF_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void range_search(long n, SWIGTYPE_p_float x, float radius, RangeSearchResult result) {\n    swigfaissJNI.IndexIVF_range_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), radius, RangeSearchResult.getCPtr(result), result);\n  }\n\n  public void range_search_preassigned(long nx, SWIGTYPE_p_float x, float radius, LongVector keys, SWIGTYPE_p_float coarse_dis, RangeSearchResult result, boolean store_pairs, IVFSearchParameters params, IndexIVFStats stats) {\n    swigfaissJNI.IndexIVF_range_search_preassigned__SWIG_0(swigCPtr, this, nx, SWIGTYPE_p_float.getCPtr(x), radius, SWIGTYPE_p_long_long.getCPtr(keys.data()), keys, SWIGTYPE_p_float.getCPtr(coarse_dis), RangeSearchResult.getCPtr(result), result, store_pairs, IVFSearchParameters.getCPtr(params), params, IndexIVFStats.getCPtr(stats), stats);\n  }\n\n  public void range_search_preassigned(long nx, SWIGTYPE_p_float x, float radius, LongVector keys, SWIGTYPE_p_float coarse_dis, RangeSearchResult result, boolean store_pairs, IVFSearchParameters params) {\n    swigfaissJNI.IndexIVF_range_search_preassigned__SWIG_1(swigCPtr, this, nx, SWIGTYPE_p_float.getCPtr(x), radius, SWIGTYPE_p_long_long.getCPtr(keys.data()), keys, SWIGTYPE_p_float.getCPtr(coarse_dis), RangeSearchResult.getCPtr(result), result, store_pairs, IVFSearchParameters.getCPtr(params), params);\n  }\n\n  public void range_search_preassigned(long nx, SWIGTYPE_p_float x, float radius, LongVector keys, SWIGTYPE_p_float coarse_dis, RangeSearchResult result, boolean store_pairs) {\n    swigfaissJNI.IndexIVF_range_search_preassigned__SWIG_2(swigCPtr, this, nx, SWIGTYPE_p_float.getCPtr(x), radius, SWIGTYPE_p_long_long.getCPtr(keys.data()), keys, SWIGTYPE_p_float.getCPtr(coarse_dis), RangeSearchResult.getCPtr(result), result, store_pairs);\n  }\n\n  public void range_search_preassigned(long nx, SWIGTYPE_p_float x, float radius, LongVector keys, SWIGTYPE_p_float coarse_dis, RangeSearchResult result) {\n    swigfaissJNI.IndexIVF_range_search_preassigned__SWIG_3(swigCPtr, this, nx, SWIGTYPE_p_float.getCPtr(x), radius, SWIGTYPE_p_long_long.getCPtr(keys.data()), keys, SWIGTYPE_p_float.getCPtr(coarse_dis), RangeSearchResult.getCPtr(result), result);\n  }\n\n  public SWIGTYPE_p_faiss__InvertedListScanner get_InvertedListScanner(boolean store_pairs) {\n    long cPtr = swigfaissJNI.IndexIVF_get_InvertedListScanner__SWIG_0(swigCPtr, this, store_pairs);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__InvertedListScanner(cPtr, false);\n  }\n\n  public SWIGTYPE_p_faiss__InvertedListScanner get_InvertedListScanner() {\n    long cPtr = swigfaissJNI.IndexIVF_get_InvertedListScanner__SWIG_1(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__InvertedListScanner(cPtr, false);\n  }\n\n  public void reconstruct(long key, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexIVF_reconstruct(swigCPtr, this, key, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public void update_vectors(int nv, LongVector idx, SWIGTYPE_p_float v) {\n    swigfaissJNI.IndexIVF_update_vectors(swigCPtr, this, nv, SWIGTYPE_p_long_long.getCPtr(idx.data()), idx, SWIGTYPE_p_float.getCPtr(v));\n  }\n\n  public void reconstruct_n(long i0, long ni, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexIVF_reconstruct_n(swigCPtr, this, i0, ni, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public void search_and_reconstruct(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexIVF_search_and_reconstruct(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public void reconstruct_from_offset(long list_no, long offset, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexIVF_reconstruct_from_offset(swigCPtr, this, list_no, offset, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public long remove_ids(IDSelector sel) {\n    return swigfaissJNI.IndexIVF_remove_ids(swigCPtr, this, IDSelector.getCPtr(sel), sel);\n  }\n\n  public void check_compatible_for_merge(IndexIVF other) {\n    swigfaissJNI.IndexIVF_check_compatible_for_merge(swigCPtr, this, IndexIVF.getCPtr(other), other);\n  }\n\n  public void merge_from(IndexIVF other, long add_id) {\n    swigfaissJNI.IndexIVF_merge_from(swigCPtr, this, IndexIVF.getCPtr(other), other, add_id);\n  }\n\n  public void copy_subset_to(IndexIVF other, int subset_type, long a1, long a2) {\n    swigfaissJNI.IndexIVF_copy_subset_to(swigCPtr, this, IndexIVF.getCPtr(other), other, subset_type, a1, a2);\n  }\n\n  public long get_list_size(long list_no) {\n    return swigfaissJNI.IndexIVF_get_list_size(swigCPtr, this, list_no);\n  }\n\n  public void make_direct_map(boolean new_maintain_direct_map) {\n    swigfaissJNI.IndexIVF_make_direct_map__SWIG_0(swigCPtr, this, new_maintain_direct_map);\n  }\n\n  public void make_direct_map() {\n    swigfaissJNI.IndexIVF_make_direct_map__SWIG_1(swigCPtr, this);\n  }\n\n  public void set_direct_map_type(SWIGTYPE_p_DirectMap__Type type) {\n    swigfaissJNI.IndexIVF_set_direct_map_type(swigCPtr, this, SWIGTYPE_p_DirectMap__Type.getCPtr(type));\n  }\n\n  public void replace_invlists(InvertedLists il, boolean own) {\n    swigfaissJNI.IndexIVF_replace_invlists__SWIG_0(swigCPtr, this, InvertedLists.getCPtr(il), il, own);\n  }\n\n  public void replace_invlists(InvertedLists il) {\n    swigfaissJNI.IndexIVF_replace_invlists__SWIG_1(swigCPtr, this, InvertedLists.getCPtr(il), il);\n  }\n\n  public long sa_code_size() {\n    return swigfaissJNI.IndexIVF_sa_code_size(swigCPtr, this);\n  }\n\n  public void sa_encode(long n, SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char bytes) {\n    swigfaissJNI.IndexIVF_sa_encode(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(bytes));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexIVFFlat.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexIVFFlat extends IndexIVF {\n  private transient long swigCPtr;\n\n  protected IndexIVFFlat(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexIVFFlat_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexIVFFlat obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexIVFFlat(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public IndexIVFFlat(Index quantizer, long d, long nlist_, MetricType arg3) {\n    this(swigfaissJNI.new_IndexIVFFlat__SWIG_0(Index.getCPtr(quantizer), quantizer, d, nlist_, arg3.swigValue()), true);\n  }\n\n  public IndexIVFFlat(Index quantizer, long d, long nlist_) {\n    this(swigfaissJNI.new_IndexIVFFlat__SWIG_1(Index.getCPtr(quantizer), quantizer, d, nlist_), true);\n  }\n\n  public void add_core(long n, SWIGTYPE_p_float x, LongVector xids, LongVector precomputed_idx) {\n    swigfaissJNI.IndexIVFFlat_add_core(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids, SWIGTYPE_p_long_long.getCPtr(precomputed_idx.data()), precomputed_idx);\n  }\n\n  public void encode_vectors(long n, SWIGTYPE_p_float x, LongVector list_nos, SWIGTYPE_p_unsigned_char codes, boolean include_listnos) {\n    swigfaissJNI.IndexIVFFlat_encode_vectors__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, SWIGTYPE_p_unsigned_char.getCPtr(codes), include_listnos);\n  }\n\n  public void encode_vectors(long n, SWIGTYPE_p_float x, LongVector list_nos, SWIGTYPE_p_unsigned_char codes) {\n    swigfaissJNI.IndexIVFFlat_encode_vectors__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, SWIGTYPE_p_unsigned_char.getCPtr(codes));\n  }\n\n  public SWIGTYPE_p_faiss__InvertedListScanner get_InvertedListScanner(boolean store_pairs) {\n    long cPtr = swigfaissJNI.IndexIVFFlat_get_InvertedListScanner(swigCPtr, this, store_pairs);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__InvertedListScanner(cPtr, false);\n  }\n\n  public void reconstruct_from_offset(long list_no, long offset, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexIVFFlat_reconstruct_from_offset(swigCPtr, this, list_no, offset, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public void sa_decode(long n, SWIGTYPE_p_unsigned_char bytes, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexIVFFlat_sa_decode(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(bytes), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public IndexIVFFlat() {\n    this(swigfaissJNI.new_IndexIVFFlat__SWIG_2(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexIVFFlatDedup.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexIVFFlatDedup extends IndexIVFFlat {\n  private transient long swigCPtr;\n\n  protected IndexIVFFlatDedup(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexIVFFlatDedup_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexIVFFlatDedup obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexIVFFlatDedup(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setInstances(SWIGTYPE_p_std__unordered_multimapT_int64_t_int64_t_t value) {\n    swigfaissJNI.IndexIVFFlatDedup_instances_set(swigCPtr, this, SWIGTYPE_p_std__unordered_multimapT_int64_t_int64_t_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__unordered_multimapT_int64_t_int64_t_t getInstances() {\n    return new SWIGTYPE_p_std__unordered_multimapT_int64_t_int64_t_t(swigfaissJNI.IndexIVFFlatDedup_instances_get(swigCPtr, this), true);\n  }\n\n  public IndexIVFFlatDedup(Index quantizer, long d, long nlist_, MetricType arg3) {\n    this(swigfaissJNI.new_IndexIVFFlatDedup__SWIG_0(Index.getCPtr(quantizer), quantizer, d, nlist_, arg3.swigValue()), true);\n  }\n\n  public IndexIVFFlatDedup(Index quantizer, long d, long nlist_) {\n    this(swigfaissJNI.new_IndexIVFFlatDedup__SWIG_1(Index.getCPtr(quantizer), quantizer, d, nlist_), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexIVFFlatDedup_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void add_with_ids(long n, SWIGTYPE_p_float x, LongVector xids) {\n    swigfaissJNI.IndexIVFFlatDedup_add_with_ids(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids);\n  }\n\n  public void search_preassigned(long n, SWIGTYPE_p_float x, long k, LongVector assign, SWIGTYPE_p_float centroid_dis, SWIGTYPE_p_float distances, LongVector labels, boolean store_pairs, IVFSearchParameters params, IndexIVFStats stats) {\n    swigfaissJNI.IndexIVFFlatDedup_search_preassigned__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_long_long.getCPtr(assign.data()), assign, SWIGTYPE_p_float.getCPtr(centroid_dis), SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, store_pairs, IVFSearchParameters.getCPtr(params), params, IndexIVFStats.getCPtr(stats), stats);\n  }\n\n  public void search_preassigned(long n, SWIGTYPE_p_float x, long k, LongVector assign, SWIGTYPE_p_float centroid_dis, SWIGTYPE_p_float distances, LongVector labels, boolean store_pairs, IVFSearchParameters params) {\n    swigfaissJNI.IndexIVFFlatDedup_search_preassigned__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_long_long.getCPtr(assign.data()), assign, SWIGTYPE_p_float.getCPtr(centroid_dis), SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, store_pairs, IVFSearchParameters.getCPtr(params), params);\n  }\n\n  public void search_preassigned(long n, SWIGTYPE_p_float x, long k, LongVector assign, SWIGTYPE_p_float centroid_dis, SWIGTYPE_p_float distances, LongVector labels, boolean store_pairs) {\n    swigfaissJNI.IndexIVFFlatDedup_search_preassigned__SWIG_2(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_long_long.getCPtr(assign.data()), assign, SWIGTYPE_p_float.getCPtr(centroid_dis), SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, store_pairs);\n  }\n\n  public long remove_ids(IDSelector sel) {\n    return swigfaissJNI.IndexIVFFlatDedup_remove_ids(swigCPtr, this, IDSelector.getCPtr(sel), sel);\n  }\n\n  public void range_search(long n, SWIGTYPE_p_float x, float radius, RangeSearchResult result) {\n    swigfaissJNI.IndexIVFFlatDedup_range_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), radius, RangeSearchResult.getCPtr(result), result);\n  }\n\n  public void update_vectors(int nv, LongVector idx, SWIGTYPE_p_float v) {\n    swigfaissJNI.IndexIVFFlatDedup_update_vectors(swigCPtr, this, nv, SWIGTYPE_p_long_long.getCPtr(idx.data()), idx, SWIGTYPE_p_float.getCPtr(v));\n  }\n\n  public void reconstruct_from_offset(long list_no, long offset, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexIVFFlatDedup_reconstruct_from_offset(swigCPtr, this, list_no, offset, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public IndexIVFFlatDedup() {\n    this(swigfaissJNI.new_IndexIVFFlatDedup__SWIG_2(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexIVFPQ.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexIVFPQ extends IndexIVF {\n  private transient long swigCPtr;\n\n  protected IndexIVFPQ(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexIVFPQ_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexIVFPQ obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexIVFPQ(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setBy_residual(boolean value) {\n    swigfaissJNI.IndexIVFPQ_by_residual_set(swigCPtr, this, value);\n  }\n\n  public boolean getBy_residual() {\n    return swigfaissJNI.IndexIVFPQ_by_residual_get(swigCPtr, this);\n  }\n\n  public void setPq(ProductQuantizer value) {\n    swigfaissJNI.IndexIVFPQ_pq_set(swigCPtr, this, ProductQuantizer.getCPtr(value), value);\n  }\n\n  public ProductQuantizer getPq() {\n    long cPtr = swigfaissJNI.IndexIVFPQ_pq_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ProductQuantizer(cPtr, false);\n  }\n\n  public void setDo_polysemous_training(boolean value) {\n    swigfaissJNI.IndexIVFPQ_do_polysemous_training_set(swigCPtr, this, value);\n  }\n\n  public boolean getDo_polysemous_training() {\n    return swigfaissJNI.IndexIVFPQ_do_polysemous_training_get(swigCPtr, this);\n  }\n\n  public void setPolysemous_training(PolysemousTraining value) {\n    swigfaissJNI.IndexIVFPQ_polysemous_training_set(swigCPtr, this, PolysemousTraining.getCPtr(value), value);\n  }\n\n  public PolysemousTraining getPolysemous_training() {\n    long cPtr = swigfaissJNI.IndexIVFPQ_polysemous_training_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new PolysemousTraining(cPtr, false);\n  }\n\n  public void setScan_table_threshold(long value) {\n    swigfaissJNI.IndexIVFPQ_scan_table_threshold_set(swigCPtr, this, value);\n  }\n\n  public long getScan_table_threshold() {\n    return swigfaissJNI.IndexIVFPQ_scan_table_threshold_get(swigCPtr, this);\n  }\n\n  public void setPolysemous_ht(int value) {\n    swigfaissJNI.IndexIVFPQ_polysemous_ht_set(swigCPtr, this, value);\n  }\n\n  public int getPolysemous_ht() {\n    return swigfaissJNI.IndexIVFPQ_polysemous_ht_get(swigCPtr, this);\n  }\n\n  public void setUse_precomputed_table(int value) {\n    swigfaissJNI.IndexIVFPQ_use_precomputed_table_set(swigCPtr, this, value);\n  }\n\n  public int getUse_precomputed_table() {\n    return swigfaissJNI.IndexIVFPQ_use_precomputed_table_get(swigCPtr, this);\n  }\n\n  public void setPrecomputed_table(SWIGTYPE_p_AlignedTableT_float_t value) {\n    swigfaissJNI.IndexIVFPQ_precomputed_table_set(swigCPtr, this, SWIGTYPE_p_AlignedTableT_float_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_AlignedTableT_float_t getPrecomputed_table() {\n    return new SWIGTYPE_p_AlignedTableT_float_t(swigfaissJNI.IndexIVFPQ_precomputed_table_get(swigCPtr, this), true);\n  }\n\n  public IndexIVFPQ(Index quantizer, long d, long nlist, long M, long nbits_per_idx, MetricType metric) {\n    this(swigfaissJNI.new_IndexIVFPQ__SWIG_0(Index.getCPtr(quantizer), quantizer, d, nlist, M, nbits_per_idx, metric.swigValue()), true);\n  }\n\n  public IndexIVFPQ(Index quantizer, long d, long nlist, long M, long nbits_per_idx) {\n    this(swigfaissJNI.new_IndexIVFPQ__SWIG_1(Index.getCPtr(quantizer), quantizer, d, nlist, M, nbits_per_idx), true);\n  }\n\n  public void encode_vectors(long n, SWIGTYPE_p_float x, LongVector list_nos, SWIGTYPE_p_unsigned_char codes, boolean include_listnos) {\n    swigfaissJNI.IndexIVFPQ_encode_vectors__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, SWIGTYPE_p_unsigned_char.getCPtr(codes), include_listnos);\n  }\n\n  public void encode_vectors(long n, SWIGTYPE_p_float x, LongVector list_nos, SWIGTYPE_p_unsigned_char codes) {\n    swigfaissJNI.IndexIVFPQ_encode_vectors__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, SWIGTYPE_p_unsigned_char.getCPtr(codes));\n  }\n\n  public void sa_decode(long n, SWIGTYPE_p_unsigned_char bytes, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexIVFPQ_sa_decode(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(bytes), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void add_core(long n, SWIGTYPE_p_float x, LongVector xids, LongVector precomputed_idx) {\n    swigfaissJNI.IndexIVFPQ_add_core(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids, SWIGTYPE_p_long_long.getCPtr(precomputed_idx.data()), precomputed_idx);\n  }\n\n  public void add_core_o(long n, SWIGTYPE_p_float x, LongVector xids, SWIGTYPE_p_float residuals_2, LongVector precomputed_idx) {\n    swigfaissJNI.IndexIVFPQ_add_core_o__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids, SWIGTYPE_p_float.getCPtr(residuals_2), SWIGTYPE_p_long_long.getCPtr(precomputed_idx.data()), precomputed_idx);\n  }\n\n  public void add_core_o(long n, SWIGTYPE_p_float x, LongVector xids, SWIGTYPE_p_float residuals_2) {\n    swigfaissJNI.IndexIVFPQ_add_core_o__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids, SWIGTYPE_p_float.getCPtr(residuals_2));\n  }\n\n  public void train_residual(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexIVFPQ_train_residual(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void train_residual_o(long n, SWIGTYPE_p_float x, SWIGTYPE_p_float residuals_2) {\n    swigfaissJNI.IndexIVFPQ_train_residual_o(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(residuals_2));\n  }\n\n  public void reconstruct_from_offset(long list_no, long offset, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexIVFPQ_reconstruct_from_offset(swigCPtr, this, list_no, offset, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public long find_duplicates(LongVector ids, SWIGTYPE_p_unsigned_long lims) {\n    return swigfaissJNI.IndexIVFPQ_find_duplicates(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, SWIGTYPE_p_unsigned_long.getCPtr(lims));\n  }\n\n  public void encode(long key, SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char code) {\n    swigfaissJNI.IndexIVFPQ_encode(swigCPtr, this, key, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void encode_multiple(long n, LongVector keys, SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char codes, boolean compute_keys) {\n    swigfaissJNI.IndexIVFPQ_encode_multiple__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_long_long.getCPtr(keys.data()), keys, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(codes), compute_keys);\n  }\n\n  public void encode_multiple(long n, LongVector keys, SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char codes) {\n    swigfaissJNI.IndexIVFPQ_encode_multiple__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_long_long.getCPtr(keys.data()), keys, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(codes));\n  }\n\n  public void decode_multiple(long n, LongVector keys, SWIGTYPE_p_unsigned_char xcodes, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexIVFPQ_decode_multiple(swigCPtr, this, n, SWIGTYPE_p_long_long.getCPtr(keys.data()), keys, SWIGTYPE_p_unsigned_char.getCPtr(xcodes), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public SWIGTYPE_p_faiss__InvertedListScanner get_InvertedListScanner(boolean store_pairs) {\n    long cPtr = swigfaissJNI.IndexIVFPQ_get_InvertedListScanner(swigCPtr, this, store_pairs);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__InvertedListScanner(cPtr, false);\n  }\n\n  public void precompute_table() {\n    swigfaissJNI.IndexIVFPQ_precompute_table(swigCPtr, this);\n  }\n\n  public IndexIVFPQ() {\n    this(swigfaissJNI.new_IndexIVFPQ__SWIG_2(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexIVFPQStats.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexIVFPQStats {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected IndexIVFPQStats(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexIVFPQStats obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexIVFPQStats(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setNrefine(long value) {\n    swigfaissJNI.IndexIVFPQStats_nrefine_set(swigCPtr, this, value);\n  }\n\n  public long getNrefine() {\n    return swigfaissJNI.IndexIVFPQStats_nrefine_get(swigCPtr, this);\n  }\n\n  public void setN_hamming_pass(long value) {\n    swigfaissJNI.IndexIVFPQStats_n_hamming_pass_set(swigCPtr, this, value);\n  }\n\n  public long getN_hamming_pass() {\n    return swigfaissJNI.IndexIVFPQStats_n_hamming_pass_get(swigCPtr, this);\n  }\n\n  public void setSearch_cycles(long value) {\n    swigfaissJNI.IndexIVFPQStats_search_cycles_set(swigCPtr, this, value);\n  }\n\n  public long getSearch_cycles() {\n    return swigfaissJNI.IndexIVFPQStats_search_cycles_get(swigCPtr, this);\n  }\n\n  public void setRefine_cycles(long value) {\n    swigfaissJNI.IndexIVFPQStats_refine_cycles_set(swigCPtr, this, value);\n  }\n\n  public long getRefine_cycles() {\n    return swigfaissJNI.IndexIVFPQStats_refine_cycles_get(swigCPtr, this);\n  }\n\n  public IndexIVFPQStats() {\n    this(swigfaissJNI.new_IndexIVFPQStats(), true);\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexIVFPQStats_reset(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexIVFScalarQuantizer.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexIVFScalarQuantizer extends IndexIVF {\n  private transient long swigCPtr;\n\n  protected IndexIVFScalarQuantizer(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexIVFScalarQuantizer_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexIVFScalarQuantizer obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexIVFScalarQuantizer(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setSq(SWIGTYPE_p_ScalarQuantizer value) {\n    swigfaissJNI.IndexIVFScalarQuantizer_sq_set(swigCPtr, this, SWIGTYPE_p_ScalarQuantizer.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_ScalarQuantizer getSq() {\n    return new SWIGTYPE_p_ScalarQuantizer(swigfaissJNI.IndexIVFScalarQuantizer_sq_get(swigCPtr, this), true);\n  }\n\n  public void setBy_residual(boolean value) {\n    swigfaissJNI.IndexIVFScalarQuantizer_by_residual_set(swigCPtr, this, value);\n  }\n\n  public boolean getBy_residual() {\n    return swigfaissJNI.IndexIVFScalarQuantizer_by_residual_get(swigCPtr, this);\n  }\n\n  public IndexIVFScalarQuantizer(Index quantizer, long d, long nlist, SWIGTYPE_p_ScalarQuantizer__QuantizerType qtype, MetricType metric, boolean encode_residual) {\n    this(swigfaissJNI.new_IndexIVFScalarQuantizer__SWIG_0(Index.getCPtr(quantizer), quantizer, d, nlist, SWIGTYPE_p_ScalarQuantizer__QuantizerType.getCPtr(qtype), metric.swigValue(), encode_residual), true);\n  }\n\n  public IndexIVFScalarQuantizer(Index quantizer, long d, long nlist, SWIGTYPE_p_ScalarQuantizer__QuantizerType qtype, MetricType metric) {\n    this(swigfaissJNI.new_IndexIVFScalarQuantizer__SWIG_1(Index.getCPtr(quantizer), quantizer, d, nlist, SWIGTYPE_p_ScalarQuantizer__QuantizerType.getCPtr(qtype), metric.swigValue()), true);\n  }\n\n  public IndexIVFScalarQuantizer(Index quantizer, long d, long nlist, SWIGTYPE_p_ScalarQuantizer__QuantizerType qtype) {\n    this(swigfaissJNI.new_IndexIVFScalarQuantizer__SWIG_2(Index.getCPtr(quantizer), quantizer, d, nlist, SWIGTYPE_p_ScalarQuantizer__QuantizerType.getCPtr(qtype)), true);\n  }\n\n  public IndexIVFScalarQuantizer() {\n    this(swigfaissJNI.new_IndexIVFScalarQuantizer__SWIG_3(), true);\n  }\n\n  public void train_residual(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexIVFScalarQuantizer_train_residual(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void encode_vectors(long n, SWIGTYPE_p_float x, LongVector list_nos, SWIGTYPE_p_unsigned_char codes, boolean include_listnos) {\n    swigfaissJNI.IndexIVFScalarQuantizer_encode_vectors__SWIG_0(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, SWIGTYPE_p_unsigned_char.getCPtr(codes), include_listnos);\n  }\n\n  public void encode_vectors(long n, SWIGTYPE_p_float x, LongVector list_nos, SWIGTYPE_p_unsigned_char codes) {\n    swigfaissJNI.IndexIVFScalarQuantizer_encode_vectors__SWIG_1(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, SWIGTYPE_p_unsigned_char.getCPtr(codes));\n  }\n\n  public void add_core(long n, SWIGTYPE_p_float x, LongVector xids, LongVector precomputed_idx) {\n    swigfaissJNI.IndexIVFScalarQuantizer_add_core(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids, SWIGTYPE_p_long_long.getCPtr(precomputed_idx.data()), precomputed_idx);\n  }\n\n  public SWIGTYPE_p_faiss__InvertedListScanner get_InvertedListScanner(boolean store_pairs) {\n    long cPtr = swigfaissJNI.IndexIVFScalarQuantizer_get_InvertedListScanner(swigCPtr, this, store_pairs);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__InvertedListScanner(cPtr, false);\n  }\n\n  public void reconstruct_from_offset(long list_no, long offset, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexIVFScalarQuantizer_reconstruct_from_offset(swigCPtr, this, list_no, offset, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public void sa_decode(long n, SWIGTYPE_p_unsigned_char bytes, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexIVFScalarQuantizer_sa_decode(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(bytes), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexIVFStats.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexIVFStats {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected IndexIVFStats(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexIVFStats obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexIVFStats(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setNq(long value) {\n    swigfaissJNI.IndexIVFStats_nq_set(swigCPtr, this, value);\n  }\n\n  public long getNq() {\n    return swigfaissJNI.IndexIVFStats_nq_get(swigCPtr, this);\n  }\n\n  public void setNlist(long value) {\n    swigfaissJNI.IndexIVFStats_nlist_set(swigCPtr, this, value);\n  }\n\n  public long getNlist() {\n    return swigfaissJNI.IndexIVFStats_nlist_get(swigCPtr, this);\n  }\n\n  public void setNdis(long value) {\n    swigfaissJNI.IndexIVFStats_ndis_set(swigCPtr, this, value);\n  }\n\n  public long getNdis() {\n    return swigfaissJNI.IndexIVFStats_ndis_get(swigCPtr, this);\n  }\n\n  public void setNheap_updates(long value) {\n    swigfaissJNI.IndexIVFStats_nheap_updates_set(swigCPtr, this, value);\n  }\n\n  public long getNheap_updates() {\n    return swigfaissJNI.IndexIVFStats_nheap_updates_get(swigCPtr, this);\n  }\n\n  public void setQuantization_time(double value) {\n    swigfaissJNI.IndexIVFStats_quantization_time_set(swigCPtr, this, value);\n  }\n\n  public double getQuantization_time() {\n    return swigfaissJNI.IndexIVFStats_quantization_time_get(swigCPtr, this);\n  }\n\n  public void setSearch_time(double value) {\n    swigfaissJNI.IndexIVFStats_search_time_set(swigCPtr, this, value);\n  }\n\n  public double getSearch_time() {\n    return swigfaissJNI.IndexIVFStats_search_time_get(swigCPtr, this);\n  }\n\n  public IndexIVFStats() {\n    this(swigfaissJNI.new_IndexIVFStats(), true);\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexIVFStats_reset(swigCPtr, this);\n  }\n\n  public void add(IndexIVFStats other) {\n    swigfaissJNI.IndexIVFStats_add(swigCPtr, this, IndexIVFStats.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexLSH.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexLSH extends IndexFlatCodes {\n  private transient long swigCPtr;\n\n  protected IndexLSH(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexLSH_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexLSH obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexLSH(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setNbits(int value) {\n    swigfaissJNI.IndexLSH_nbits_set(swigCPtr, this, value);\n  }\n\n  public int getNbits() {\n    return swigfaissJNI.IndexLSH_nbits_get(swigCPtr, this);\n  }\n\n  public void setRotate_data(boolean value) {\n    swigfaissJNI.IndexLSH_rotate_data_set(swigCPtr, this, value);\n  }\n\n  public boolean getRotate_data() {\n    return swigfaissJNI.IndexLSH_rotate_data_get(swigCPtr, this);\n  }\n\n  public void setTrain_thresholds(boolean value) {\n    swigfaissJNI.IndexLSH_train_thresholds_set(swigCPtr, this, value);\n  }\n\n  public boolean getTrain_thresholds() {\n    return swigfaissJNI.IndexLSH_train_thresholds_get(swigCPtr, this);\n  }\n\n  public void setRrot(RandomRotationMatrix value) {\n    swigfaissJNI.IndexLSH_rrot_set(swigCPtr, this, RandomRotationMatrix.getCPtr(value), value);\n  }\n\n  public RandomRotationMatrix getRrot() {\n    long cPtr = swigfaissJNI.IndexLSH_rrot_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new RandomRotationMatrix(cPtr, false);\n  }\n\n  public void setThresholds(FloatVector value) {\n    swigfaissJNI.IndexLSH_thresholds_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getThresholds() {\n    long cPtr = swigfaissJNI.IndexLSH_thresholds_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public IndexLSH(long d, int nbits, boolean rotate_data, boolean train_thresholds) {\n    this(swigfaissJNI.new_IndexLSH__SWIG_0(d, nbits, rotate_data, train_thresholds), true);\n  }\n\n  public IndexLSH(long d, int nbits, boolean rotate_data) {\n    this(swigfaissJNI.new_IndexLSH__SWIG_1(d, nbits, rotate_data), true);\n  }\n\n  public IndexLSH(long d, int nbits) {\n    this(swigfaissJNI.new_IndexLSH__SWIG_2(d, nbits), true);\n  }\n\n  public SWIGTYPE_p_float apply_preprocess(long n, SWIGTYPE_p_float x) {\n    long cPtr = swigfaissJNI.IndexLSH_apply_preprocess(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexLSH_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexLSH_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void transfer_thresholds(LinearTransform vt) {\n    swigfaissJNI.IndexLSH_transfer_thresholds(swigCPtr, this, LinearTransform.getCPtr(vt), vt);\n  }\n\n  public IndexLSH() {\n    this(swigfaissJNI.new_IndexLSH__SWIG_3(), true);\n  }\n\n  public void sa_encode(long n, SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char bytes) {\n    swigfaissJNI.IndexLSH_sa_encode(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(bytes));\n  }\n\n  public void sa_decode(long n, SWIGTYPE_p_unsigned_char bytes, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexLSH_sa_decode(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(bytes), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexPQ.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexPQ extends IndexFlatCodes {\n  private transient long swigCPtr;\n\n  protected IndexPQ(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexPQ_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexPQ obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexPQ(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setPq(ProductQuantizer value) {\n    swigfaissJNI.IndexPQ_pq_set(swigCPtr, this, ProductQuantizer.getCPtr(value), value);\n  }\n\n  public ProductQuantizer getPq() {\n    long cPtr = swigfaissJNI.IndexPQ_pq_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ProductQuantizer(cPtr, false);\n  }\n\n  public IndexPQ(int d, long M, long nbits, MetricType metric) {\n    this(swigfaissJNI.new_IndexPQ__SWIG_0(d, M, nbits, metric.swigValue()), true);\n  }\n\n  public IndexPQ(int d, long M, long nbits) {\n    this(swigfaissJNI.new_IndexPQ__SWIG_1(d, M, nbits), true);\n  }\n\n  public IndexPQ() {\n    this(swigfaissJNI.new_IndexPQ__SWIG_2(), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexPQ_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexPQ_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void sa_encode(long n, SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char bytes) {\n    swigfaissJNI.IndexPQ_sa_encode(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(bytes));\n  }\n\n  public void sa_decode(long n, SWIGTYPE_p_unsigned_char bytes, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexPQ_sa_decode(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(bytes), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public DistanceComputer get_distance_computer() {\n    long cPtr = swigfaissJNI.IndexPQ_get_distance_computer(swigCPtr, this);\n    return (cPtr == 0) ? null : new DistanceComputer(cPtr, false);\n  }\n\n  public void setDo_polysemous_training(boolean value) {\n    swigfaissJNI.IndexPQ_do_polysemous_training_set(swigCPtr, this, value);\n  }\n\n  public boolean getDo_polysemous_training() {\n    return swigfaissJNI.IndexPQ_do_polysemous_training_get(swigCPtr, this);\n  }\n\n  public void setPolysemous_training(PolysemousTraining value) {\n    swigfaissJNI.IndexPQ_polysemous_training_set(swigCPtr, this, PolysemousTraining.getCPtr(value), value);\n  }\n\n  public PolysemousTraining getPolysemous_training() {\n    long cPtr = swigfaissJNI.IndexPQ_polysemous_training_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new PolysemousTraining(cPtr, false);\n  }\n\n  public void setSearch_type(IndexPQ.Search_type_t value) {\n    swigfaissJNI.IndexPQ_search_type_set(swigCPtr, this, value.swigValue());\n  }\n\n  public IndexPQ.Search_type_t getSearch_type() {\n    return IndexPQ.Search_type_t.swigToEnum(swigfaissJNI.IndexPQ_search_type_get(swigCPtr, this));\n  }\n\n  public void setEncode_signs(boolean value) {\n    swigfaissJNI.IndexPQ_encode_signs_set(swigCPtr, this, value);\n  }\n\n  public boolean getEncode_signs() {\n    return swigfaissJNI.IndexPQ_encode_signs_get(swigCPtr, this);\n  }\n\n  public void setPolysemous_ht(int value) {\n    swigfaissJNI.IndexPQ_polysemous_ht_set(swigCPtr, this, value);\n  }\n\n  public int getPolysemous_ht() {\n    return swigfaissJNI.IndexPQ_polysemous_ht_get(swigCPtr, this);\n  }\n\n  public void search_core_polysemous(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexPQ_search_core_polysemous(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void hamming_distance_histogram(long n, SWIGTYPE_p_float x, long nb, SWIGTYPE_p_float xb, LongVector dist_histogram) {\n    swigfaissJNI.IndexPQ_hamming_distance_histogram(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), nb, SWIGTYPE_p_float.getCPtr(xb), SWIGTYPE_p_long_long.getCPtr(dist_histogram.data()), dist_histogram);\n  }\n\n  public void hamming_distance_table(long n, SWIGTYPE_p_float x, SWIGTYPE_p_int dis) {\n    swigfaissJNI.IndexPQ_hamming_distance_table(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_int.getCPtr(dis));\n  }\n\n  public final static class Search_type_t {\n    public final static IndexPQ.Search_type_t ST_PQ = new IndexPQ.Search_type_t(\"ST_PQ\");\n    public final static IndexPQ.Search_type_t ST_HE = new IndexPQ.Search_type_t(\"ST_HE\");\n    public final static IndexPQ.Search_type_t ST_generalized_HE = new IndexPQ.Search_type_t(\"ST_generalized_HE\");\n    public final static IndexPQ.Search_type_t ST_SDC = new IndexPQ.Search_type_t(\"ST_SDC\");\n    public final static IndexPQ.Search_type_t ST_polysemous = new IndexPQ.Search_type_t(\"ST_polysemous\");\n    public final static IndexPQ.Search_type_t ST_polysemous_generalize = new IndexPQ.Search_type_t(\"ST_polysemous_generalize\");\n\n    public final int swigValue() {\n      return swigValue;\n    }\n\n    public String toString() {\n      return swigName;\n    }\n\n    public static Search_type_t swigToEnum(int swigValue) {\n      if (swigValue < swigValues.length && swigValue >= 0 && swigValues[swigValue].swigValue == swigValue)\n        return swigValues[swigValue];\n      for (int i = 0; i < swigValues.length; i++)\n        if (swigValues[i].swigValue == swigValue)\n          return swigValues[i];\n      throw new IllegalArgumentException(\"No enum \" + Search_type_t.class + \" with value \" + swigValue);\n    }\n\n    private Search_type_t(String swigName) {\n      this.swigName = swigName;\n      this.swigValue = swigNext++;\n    }\n\n    private Search_type_t(String swigName, int swigValue) {\n      this.swigName = swigName;\n      this.swigValue = swigValue;\n      swigNext = swigValue+1;\n    }\n\n    private Search_type_t(String swigName, Search_type_t swigEnum) {\n      this.swigName = swigName;\n      this.swigValue = swigEnum.swigValue;\n      swigNext = this.swigValue+1;\n    }\n\n    private static Search_type_t[] swigValues = { ST_PQ, ST_HE, ST_generalized_HE, ST_SDC, ST_polysemous, ST_polysemous_generalize };\n    private static int swigNext = 0;\n    private final int swigValue;\n    private final String swigName;\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexPQStats.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexPQStats {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected IndexPQStats(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexPQStats obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexPQStats(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setNq(long value) {\n    swigfaissJNI.IndexPQStats_nq_set(swigCPtr, this, value);\n  }\n\n  public long getNq() {\n    return swigfaissJNI.IndexPQStats_nq_get(swigCPtr, this);\n  }\n\n  public void setNcode(long value) {\n    swigfaissJNI.IndexPQStats_ncode_set(swigCPtr, this, value);\n  }\n\n  public long getNcode() {\n    return swigfaissJNI.IndexPQStats_ncode_get(swigCPtr, this);\n  }\n\n  public void setN_hamming_pass(long value) {\n    swigfaissJNI.IndexPQStats_n_hamming_pass_set(swigCPtr, this, value);\n  }\n\n  public long getN_hamming_pass() {\n    return swigfaissJNI.IndexPQStats_n_hamming_pass_get(swigCPtr, this);\n  }\n\n  public IndexPQStats() {\n    this(swigfaissJNI.new_IndexPQStats(), true);\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexPQStats_reset(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexRefine.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexRefine extends Index {\n  private transient long swigCPtr;\n\n  protected IndexRefine(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexRefine_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexRefine obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexRefine(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setBase_index(Index value) {\n    swigfaissJNI.IndexRefine_base_index_set(swigCPtr, this, Index.getCPtr(value), value);\n  }\n\n  public Index getBase_index() {\n    long cPtr = swigfaissJNI.IndexRefine_base_index_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new Index(cPtr, false);\n  }\n\n  public void setRefine_index(Index value) {\n    swigfaissJNI.IndexRefine_refine_index_set(swigCPtr, this, Index.getCPtr(value), value);\n  }\n\n  public Index getRefine_index() {\n    long cPtr = swigfaissJNI.IndexRefine_refine_index_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new Index(cPtr, false);\n  }\n\n  public void setOwn_fields(boolean value) {\n    swigfaissJNI.IndexRefine_own_fields_set(swigCPtr, this, value);\n  }\n\n  public boolean getOwn_fields() {\n    return swigfaissJNI.IndexRefine_own_fields_get(swigCPtr, this);\n  }\n\n  public void setOwn_refine_index(boolean value) {\n    swigfaissJNI.IndexRefine_own_refine_index_set(swigCPtr, this, value);\n  }\n\n  public boolean getOwn_refine_index() {\n    return swigfaissJNI.IndexRefine_own_refine_index_get(swigCPtr, this);\n  }\n\n  public void setK_factor(float value) {\n    swigfaissJNI.IndexRefine_k_factor_set(swigCPtr, this, value);\n  }\n\n  public float getK_factor() {\n    return swigfaissJNI.IndexRefine_k_factor_get(swigCPtr, this);\n  }\n\n  public IndexRefine(Index base_index, Index refine_index) {\n    this(swigfaissJNI.new_IndexRefine__SWIG_0(Index.getCPtr(base_index), base_index, Index.getCPtr(refine_index), refine_index), true);\n  }\n\n  public IndexRefine() {\n    this(swigfaissJNI.new_IndexRefine__SWIG_1(), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexRefine_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void add(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexRefine_add(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexRefine_reset(swigCPtr, this);\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexRefine_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void reconstruct(long key, SWIGTYPE_p_float recons) {\n    swigfaissJNI.IndexRefine_reconstruct(swigCPtr, this, key, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n  public long sa_code_size() {\n    return swigfaissJNI.IndexRefine_sa_code_size(swigCPtr, this);\n  }\n\n  public void sa_encode(long n, SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char bytes) {\n    swigfaissJNI.IndexRefine_sa_encode(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(bytes));\n  }\n\n  public void sa_decode(long n, SWIGTYPE_p_unsigned_char bytes, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexRefine_sa_decode(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(bytes), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexRefineFlat.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexRefineFlat extends IndexRefine {\n  private transient long swigCPtr;\n\n  protected IndexRefineFlat(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexRefineFlat_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexRefineFlat obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexRefineFlat(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public IndexRefineFlat(Index base_index) {\n    this(swigfaissJNI.new_IndexRefineFlat__SWIG_0(Index.getCPtr(base_index), base_index), true);\n  }\n\n  public IndexRefineFlat(Index base_index, SWIGTYPE_p_float xb) {\n    this(swigfaissJNI.new_IndexRefineFlat__SWIG_1(Index.getCPtr(base_index), base_index, SWIGTYPE_p_float.getCPtr(xb)), true);\n  }\n\n  public IndexRefineFlat() {\n    this(swigfaissJNI.new_IndexRefineFlat__SWIG_2(), true);\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexRefineFlat_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexScalarQuantizer.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexScalarQuantizer extends IndexFlatCodes {\n  private transient long swigCPtr;\n\n  protected IndexScalarQuantizer(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexScalarQuantizer_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexScalarQuantizer obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexScalarQuantizer(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setSq(SWIGTYPE_p_ScalarQuantizer value) {\n    swigfaissJNI.IndexScalarQuantizer_sq_set(swigCPtr, this, SWIGTYPE_p_ScalarQuantizer.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_ScalarQuantizer getSq() {\n    return new SWIGTYPE_p_ScalarQuantizer(swigfaissJNI.IndexScalarQuantizer_sq_get(swigCPtr, this), true);\n  }\n\n  public IndexScalarQuantizer(int d, SWIGTYPE_p_ScalarQuantizer__QuantizerType qtype, MetricType metric) {\n    this(swigfaissJNI.new_IndexScalarQuantizer__SWIG_0(d, SWIGTYPE_p_ScalarQuantizer__QuantizerType.getCPtr(qtype), metric.swigValue()), true);\n  }\n\n  public IndexScalarQuantizer(int d, SWIGTYPE_p_ScalarQuantizer__QuantizerType qtype) {\n    this(swigfaissJNI.new_IndexScalarQuantizer__SWIG_1(d, SWIGTYPE_p_ScalarQuantizer__QuantizerType.getCPtr(qtype)), true);\n  }\n\n  public IndexScalarQuantizer() {\n    this(swigfaissJNI.new_IndexScalarQuantizer__SWIG_2(), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexScalarQuantizer_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexScalarQuantizer_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public DistanceComputer get_distance_computer() {\n    long cPtr = swigfaissJNI.IndexScalarQuantizer_get_distance_computer(swigCPtr, this);\n    return (cPtr == 0) ? null : new DistanceComputer(cPtr, false);\n  }\n\n  public void sa_encode(long n, SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char bytes) {\n    swigfaissJNI.IndexScalarQuantizer_sa_encode(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(bytes));\n  }\n\n  public void sa_decode(long n, SWIGTYPE_p_unsigned_char bytes, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexScalarQuantizer_sa_decode(swigCPtr, this, n, SWIGTYPE_p_unsigned_char.getCPtr(bytes), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexShards.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexShards {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected IndexShards(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexShards obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexShards(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public IndexShards(boolean threaded, boolean successive_ids) {\n    this(swigfaissJNI.new_IndexShards__SWIG_0(threaded, successive_ids), true);\n  }\n\n  public IndexShards(boolean threaded) {\n    this(swigfaissJNI.new_IndexShards__SWIG_1(threaded), true);\n  }\n\n  public IndexShards() {\n    this(swigfaissJNI.new_IndexShards__SWIG_2(), true);\n  }\n\n  public IndexShards(int d, boolean threaded, boolean successive_ids) {\n    this(swigfaissJNI.new_IndexShards__SWIG_3(d, threaded, successive_ids), true);\n  }\n\n  public IndexShards(int d, boolean threaded) {\n    this(swigfaissJNI.new_IndexShards__SWIG_4(d, threaded), true);\n  }\n\n  public IndexShards(int d) {\n    this(swigfaissJNI.new_IndexShards__SWIG_5(d), true);\n  }\n\n  public void add_shard(Index index) {\n    swigfaissJNI.IndexShards_add_shard(swigCPtr, this, Index.getCPtr(index), index);\n  }\n\n  public void remove_shard(Index index) {\n    swigfaissJNI.IndexShards_remove_shard(swigCPtr, this, Index.getCPtr(index), index);\n  }\n\n  public void add(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexShards_add(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void add_with_ids(long n, SWIGTYPE_p_float x, LongVector xids) {\n    swigfaissJNI.IndexShards_add_with_ids(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_long_long.getCPtr(xids.data()), xids);\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexShards_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexShards_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void setSuccessive_ids(boolean value) {\n    swigfaissJNI.IndexShards_successive_ids_set(swigCPtr, this, value);\n  }\n\n  public boolean getSuccessive_ids() {\n    return swigfaissJNI.IndexShards_successive_ids_get(swigCPtr, this);\n  }\n\n  public void syncWithSubIndexes() {\n    swigfaissJNI.IndexShards_syncWithSubIndexes(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IndexSplitVectors.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IndexSplitVectors extends Index {\n  private transient long swigCPtr;\n\n  protected IndexSplitVectors(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IndexSplitVectors_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IndexSplitVectors obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IndexSplitVectors(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setOwn_fields(boolean value) {\n    swigfaissJNI.IndexSplitVectors_own_fields_set(swigCPtr, this, value);\n  }\n\n  public boolean getOwn_fields() {\n    return swigfaissJNI.IndexSplitVectors_own_fields_get(swigCPtr, this);\n  }\n\n  public void setThreaded(boolean value) {\n    swigfaissJNI.IndexSplitVectors_threaded_set(swigCPtr, this, value);\n  }\n\n  public boolean getThreaded() {\n    return swigfaissJNI.IndexSplitVectors_threaded_get(swigCPtr, this);\n  }\n\n  public void setSub_indexes(SWIGTYPE_p_std__vectorT_faiss__Index_p_t value) {\n    swigfaissJNI.IndexSplitVectors_sub_indexes_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_faiss__Index_p_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_faiss__Index_p_t getSub_indexes() {\n    long cPtr = swigfaissJNI.IndexSplitVectors_sub_indexes_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_faiss__Index_p_t(cPtr, false);\n  }\n\n  public void setSum_d(long value) {\n    swigfaissJNI.IndexSplitVectors_sum_d_set(swigCPtr, this, value);\n  }\n\n  public long getSum_d() {\n    return swigfaissJNI.IndexSplitVectors_sum_d_get(swigCPtr, this);\n}\n\n  public void add_sub_index(Index arg0) {\n    swigfaissJNI.IndexSplitVectors_add_sub_index(swigCPtr, this, Index.getCPtr(arg0), arg0);\n  }\n\n  public void sync_with_sub_indexes() {\n    swigfaissJNI.IndexSplitVectors_sync_with_sub_indexes(swigCPtr, this);\n  }\n\n  public void add(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexSplitVectors_add(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.IndexSplitVectors_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.IndexSplitVectors_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void reset() {\n    swigfaissJNI.IndexSplitVectors_reset(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IntVector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IntVector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected IntVector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IntVector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IntVector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public IntVector() {\n    this(swigfaissJNI.new_IntVector(), true);\n  }\n\n  public void push_back(int arg0) {\n    swigfaissJNI.IntVector_push_back(swigCPtr, this, arg0);\n  }\n\n  public void clear() {\n    swigfaissJNI.IntVector_clear(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_int data() {\n    long cPtr = swigfaissJNI.IntVector_data(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_int(cPtr, false);\n  }\n\n  public long size() {\n    return swigfaissJNI.IntVector_size(swigCPtr, this);\n  }\n\n  public int at(long n) {\n    return swigfaissJNI.IntVector_at(swigCPtr, this, n);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.IntVector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.IntVector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(IntVector other) {\n    swigfaissJNI.IntVector_swap(swigCPtr, this, IntVector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/InterruptCallback.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class InterruptCallback {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected InterruptCallback(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(InterruptCallback obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_InterruptCallback(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public boolean want_interrupt() {\n    return swigfaissJNI.InterruptCallback_want_interrupt(swigCPtr, this);\n  }\n\n  public static void clear_instance() {\n    swigfaissJNI.InterruptCallback_clear_instance();\n  }\n\n  public static void check() {\n    swigfaissJNI.InterruptCallback_check();\n  }\n\n  public static boolean is_interrupted() {\n    return swigfaissJNI.InterruptCallback_is_interrupted();\n  }\n\n  public static long get_period_hint(long flops) {\n    return swigfaissJNI.InterruptCallback_get_period_hint(flops);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/IntersectionCriterion.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class IntersectionCriterion extends AutoTuneCriterion {\n  private transient long swigCPtr;\n\n  protected IntersectionCriterion(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.IntersectionCriterion_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(IntersectionCriterion obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_IntersectionCriterion(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setR(long value) {\n    swigfaissJNI.IntersectionCriterion_R_set(swigCPtr, this, value);\n  }\n\n  public long getR() {\n    return swigfaissJNI.IntersectionCriterion_R_get(swigCPtr, this);\n}\n\n  public IntersectionCriterion(long nq, long R) {\n    this(swigfaissJNI.new_IntersectionCriterion(nq, R), true);\n  }\n\n  public double evaluate(SWIGTYPE_p_float D, LongVector I) {\n    return swigfaissJNI.IntersectionCriterion_evaluate(swigCPtr, this, SWIGTYPE_p_float.getCPtr(D), SWIGTYPE_p_long_long.getCPtr(I.data()), I);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/InvertedLists.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class InvertedLists {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected InvertedLists(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(InvertedLists obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_InvertedLists(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setNlist(long value) {\n    swigfaissJNI.InvertedLists_nlist_set(swigCPtr, this, value);\n  }\n\n  public long getNlist() {\n    return swigfaissJNI.InvertedLists_nlist_get(swigCPtr, this);\n  }\n\n  public void setCode_size(long value) {\n    swigfaissJNI.InvertedLists_code_size_set(swigCPtr, this, value);\n  }\n\n  public long getCode_size() {\n    return swigfaissJNI.InvertedLists_code_size_get(swigCPtr, this);\n  }\n\n  public long list_size(long list_no) {\n    return swigfaissJNI.InvertedLists_list_size(swigCPtr, this, list_no);\n  }\n\n  public SWIGTYPE_p_unsigned_char get_codes(long list_no) {\n    long cPtr = swigfaissJNI.InvertedLists_get_codes(swigCPtr, this, list_no);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public LongVector get_ids(long list_no) {\n    return new LongVector(swigfaissJNI.InvertedLists_get_ids(swigCPtr, this, list_no), false);\n}\n\n  public void release_codes(long list_no, SWIGTYPE_p_unsigned_char codes) {\n    swigfaissJNI.InvertedLists_release_codes(swigCPtr, this, list_no, SWIGTYPE_p_unsigned_char.getCPtr(codes));\n  }\n\n  public void release_ids(long list_no, LongVector ids) {\n    swigfaissJNI.InvertedLists_release_ids(swigCPtr, this, list_no, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids);\n  }\n\n  public long get_single_id(long list_no, long offset) {\n    return swigfaissJNI.InvertedLists_get_single_id(swigCPtr, this, list_no, offset);\n}\n\n  public SWIGTYPE_p_unsigned_char get_single_code(long list_no, long offset) {\n    long cPtr = swigfaissJNI.InvertedLists_get_single_code(swigCPtr, this, list_no, offset);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public void prefetch_lists(LongVector list_nos, int nlist) {\n    swigfaissJNI.InvertedLists_prefetch_lists(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, nlist);\n  }\n\n  public long add_entry(long list_no, long theid, SWIGTYPE_p_unsigned_char code) {\n    return swigfaissJNI.InvertedLists_add_entry(swigCPtr, this, list_no, theid, SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public long add_entries(long list_no, long n_entry, LongVector ids, SWIGTYPE_p_unsigned_char code) {\n    return swigfaissJNI.InvertedLists_add_entries(swigCPtr, this, list_no, n_entry, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void update_entry(long list_no, long offset, long id, SWIGTYPE_p_unsigned_char code) {\n    swigfaissJNI.InvertedLists_update_entry(swigCPtr, this, list_no, offset, id, SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void update_entries(long list_no, long offset, long n_entry, LongVector ids, SWIGTYPE_p_unsigned_char code) {\n    swigfaissJNI.InvertedLists_update_entries(swigCPtr, this, list_no, offset, n_entry, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void resize(long list_no, long new_size) {\n    swigfaissJNI.InvertedLists_resize(swigCPtr, this, list_no, new_size);\n  }\n\n  public void reset() {\n    swigfaissJNI.InvertedLists_reset(swigCPtr, this);\n  }\n\n  public void merge_from(InvertedLists oivf, long add_id) {\n    swigfaissJNI.InvertedLists_merge_from(swigCPtr, this, InvertedLists.getCPtr(oivf), oivf, add_id);\n  }\n\n  public double imbalance_factor() {\n    return swigfaissJNI.InvertedLists_imbalance_factor(swigCPtr, this);\n  }\n\n  public void print_stats() {\n    swigfaissJNI.InvertedLists_print_stats(swigCPtr, this);\n  }\n\n  public long compute_ntotal() {\n    return swigfaissJNI.InvertedLists_compute_ntotal(swigCPtr, this);\n  }\n\n  static public class ScopedIds {\n    private transient long swigCPtr;\n    protected transient boolean swigCMemOwn;\n  \n    protected ScopedIds(long cPtr, boolean cMemoryOwn) {\n      swigCMemOwn = cMemoryOwn;\n      swigCPtr = cPtr;\n    }\n  \n    protected static long getCPtr(ScopedIds obj) {\n      return (obj == null) ? 0 : obj.swigCPtr;\n    }\n  \n    @SuppressWarnings(\"deprecation\")\n    protected void finalize() {\n      delete();\n    }\n  \n    public synchronized void delete() {\n      if (swigCPtr != 0) {\n        if (swigCMemOwn) {\n          swigCMemOwn = false;\n          swigfaissJNI.delete_InvertedLists_ScopedIds(swigCPtr);\n        }\n        swigCPtr = 0;\n      }\n    }\n  \n    public void setIl(InvertedLists value) {\n      swigfaissJNI.InvertedLists_ScopedIds_il_set(swigCPtr, this, InvertedLists.getCPtr(value), value);\n    }\n  \n    public InvertedLists getIl() {\n      long cPtr = swigfaissJNI.InvertedLists_ScopedIds_il_get(swigCPtr, this);\n      return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n    }\n  \n    public void setIds(LongVector value) {\n      swigfaissJNI.InvertedLists_ScopedIds_ids_set(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(value.data()), value);\n    }\n  \n    public LongVector getIds() {\n      return new LongVector(swigfaissJNI.InvertedLists_ScopedIds_ids_get(swigCPtr, this), false);\n  }\n  \n    public void setList_no(long value) {\n      swigfaissJNI.InvertedLists_ScopedIds_list_no_set(swigCPtr, this, value);\n    }\n  \n    public long getList_no() {\n      return swigfaissJNI.InvertedLists_ScopedIds_list_no_get(swigCPtr, this);\n    }\n  \n    public ScopedIds(InvertedLists il, long list_no) {\n      this(swigfaissJNI.new_InvertedLists_ScopedIds(InvertedLists.getCPtr(il), il, list_no), true);\n    }\n  \n    public LongVector get() {\n      return new LongVector(swigfaissJNI.InvertedLists_ScopedIds_get(swigCPtr, this), false);\n  }\n  \n  }\n\n  static public class ScopedCodes {\n    private transient long swigCPtr;\n    protected transient boolean swigCMemOwn;\n  \n    protected ScopedCodes(long cPtr, boolean cMemoryOwn) {\n      swigCMemOwn = cMemoryOwn;\n      swigCPtr = cPtr;\n    }\n  \n    protected static long getCPtr(ScopedCodes obj) {\n      return (obj == null) ? 0 : obj.swigCPtr;\n    }\n  \n    @SuppressWarnings(\"deprecation\")\n    protected void finalize() {\n      delete();\n    }\n  \n    public synchronized void delete() {\n      if (swigCPtr != 0) {\n        if (swigCMemOwn) {\n          swigCMemOwn = false;\n          swigfaissJNI.delete_InvertedLists_ScopedCodes(swigCPtr);\n        }\n        swigCPtr = 0;\n      }\n    }\n  \n    public void setIl(InvertedLists value) {\n      swigfaissJNI.InvertedLists_ScopedCodes_il_set(swigCPtr, this, InvertedLists.getCPtr(value), value);\n    }\n  \n    public InvertedLists getIl() {\n      long cPtr = swigfaissJNI.InvertedLists_ScopedCodes_il_get(swigCPtr, this);\n      return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n    }\n  \n    public void setCodes(SWIGTYPE_p_unsigned_char value) {\n      swigfaissJNI.InvertedLists_ScopedCodes_codes_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(value));\n    }\n  \n    public SWIGTYPE_p_unsigned_char getCodes() {\n      long cPtr = swigfaissJNI.InvertedLists_ScopedCodes_codes_get(swigCPtr, this);\n      return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n    }\n  \n    public void setList_no(long value) {\n      swigfaissJNI.InvertedLists_ScopedCodes_list_no_set(swigCPtr, this, value);\n    }\n  \n    public long getList_no() {\n      return swigfaissJNI.InvertedLists_ScopedCodes_list_no_get(swigCPtr, this);\n    }\n  \n    public ScopedCodes(InvertedLists il, long list_no) {\n      this(swigfaissJNI.new_InvertedLists_ScopedCodes__SWIG_0(InvertedLists.getCPtr(il), il, list_no), true);\n    }\n  \n    public ScopedCodes(InvertedLists il, long list_no, long offset) {\n      this(swigfaissJNI.new_InvertedLists_ScopedCodes__SWIG_1(InvertedLists.getCPtr(il), il, list_no, offset), true);\n    }\n  \n    public SWIGTYPE_p_unsigned_char get() {\n      long cPtr = swigfaissJNI.InvertedLists_ScopedCodes_get(swigCPtr, this);\n      return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n    }\n  \n  }\n\n  public final static long INVALID_CODE_SIZE = swigfaissJNI.InvertedLists_INVALID_CODE_SIZE_get();\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/InvertedListsPtrVector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class InvertedListsPtrVector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected InvertedListsPtrVector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(InvertedListsPtrVector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_InvertedListsPtrVector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public InvertedListsPtrVector() {\n    this(swigfaissJNI.new_InvertedListsPtrVector(), true);\n  }\n\n  public void push_back(InvertedLists arg0) {\n    swigfaissJNI.InvertedListsPtrVector_push_back(swigCPtr, this, InvertedLists.getCPtr(arg0), arg0);\n  }\n\n  public void clear() {\n    swigfaissJNI.InvertedListsPtrVector_clear(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_p_faiss__InvertedLists data() {\n    long cPtr = swigfaissJNI.InvertedListsPtrVector_data(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_p_faiss__InvertedLists(cPtr, false);\n  }\n\n  public long size() {\n    return swigfaissJNI.InvertedListsPtrVector_size(swigCPtr, this);\n  }\n\n  public InvertedLists at(long n) {\n    long cPtr = swigfaissJNI.InvertedListsPtrVector_at(swigCPtr, this, n);\n    return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.InvertedListsPtrVector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.InvertedListsPtrVector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(InvertedListsPtrVector other) {\n    swigfaissJNI.InvertedListsPtrVector_swap(swigCPtr, this, InvertedListsPtrVector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/Level1Quantizer.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class Level1Quantizer {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected Level1Quantizer(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(Level1Quantizer obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_Level1Quantizer(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setQuantizer(Index value) {\n    swigfaissJNI.Level1Quantizer_quantizer_set(swigCPtr, this, Index.getCPtr(value), value);\n  }\n\n  public Index getQuantizer() {\n    long cPtr = swigfaissJNI.Level1Quantizer_quantizer_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new Index(cPtr, false);\n  }\n\n  public void setNlist(long value) {\n    swigfaissJNI.Level1Quantizer_nlist_set(swigCPtr, this, value);\n  }\n\n  public long getNlist() {\n    return swigfaissJNI.Level1Quantizer_nlist_get(swigCPtr, this);\n  }\n\n  public void setQuantizer_trains_alone(char value) {\n    swigfaissJNI.Level1Quantizer_quantizer_trains_alone_set(swigCPtr, this, value);\n  }\n\n  public char getQuantizer_trains_alone() {\n    return swigfaissJNI.Level1Quantizer_quantizer_trains_alone_get(swigCPtr, this);\n  }\n\n  public void setOwn_fields(boolean value) {\n    swigfaissJNI.Level1Quantizer_own_fields_set(swigCPtr, this, value);\n  }\n\n  public boolean getOwn_fields() {\n    return swigfaissJNI.Level1Quantizer_own_fields_get(swigCPtr, this);\n  }\n\n  public void setCp(ClusteringParameters value) {\n    swigfaissJNI.Level1Quantizer_cp_set(swigCPtr, this, ClusteringParameters.getCPtr(value), value);\n  }\n\n  public ClusteringParameters getCp() {\n    long cPtr = swigfaissJNI.Level1Quantizer_cp_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ClusteringParameters(cPtr, false);\n  }\n\n  public void setClustering_index(Index value) {\n    swigfaissJNI.Level1Quantizer_clustering_index_set(swigCPtr, this, Index.getCPtr(value), value);\n  }\n\n  public Index getClustering_index() {\n    long cPtr = swigfaissJNI.Level1Quantizer_clustering_index_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new Index(cPtr, false);\n  }\n\n  public void train_q1(long n, SWIGTYPE_p_float x, boolean verbose, MetricType metric_type) {\n    swigfaissJNI.Level1Quantizer_train_q1(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), verbose, metric_type.swigValue());\n  }\n\n  public long coarse_code_size() {\n    return swigfaissJNI.Level1Quantizer_coarse_code_size(swigCPtr, this);\n  }\n\n  public void encode_listno(long list_no, SWIGTYPE_p_unsigned_char code) {\n    swigfaissJNI.Level1Quantizer_encode_listno(swigCPtr, this, list_no, SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public long decode_listno(SWIGTYPE_p_unsigned_char code) {\n    return swigfaissJNI.Level1Quantizer_decode_listno(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(code));\n}\n\n  public Level1Quantizer(Index quantizer, long nlist) {\n    this(swigfaissJNI.new_Level1Quantizer__SWIG_0(Index.getCPtr(quantizer), quantizer, nlist), true);\n  }\n\n  public Level1Quantizer() {\n    this(swigfaissJNI.new_Level1Quantizer__SWIG_1(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/LinearTransform.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class LinearTransform extends VectorTransform {\n  private transient long swigCPtr;\n\n  protected LinearTransform(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.LinearTransform_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(LinearTransform obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_LinearTransform(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setHave_bias(boolean value) {\n    swigfaissJNI.LinearTransform_have_bias_set(swigCPtr, this, value);\n  }\n\n  public boolean getHave_bias() {\n    return swigfaissJNI.LinearTransform_have_bias_get(swigCPtr, this);\n  }\n\n  public void setIs_orthonormal(boolean value) {\n    swigfaissJNI.LinearTransform_is_orthonormal_set(swigCPtr, this, value);\n  }\n\n  public boolean getIs_orthonormal() {\n    return swigfaissJNI.LinearTransform_is_orthonormal_get(swigCPtr, this);\n  }\n\n  public void setA(FloatVector value) {\n    swigfaissJNI.LinearTransform_A_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getA() {\n    long cPtr = swigfaissJNI.LinearTransform_A_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public void setB(FloatVector value) {\n    swigfaissJNI.LinearTransform_b_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getB() {\n    long cPtr = swigfaissJNI.LinearTransform_b_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public LinearTransform(int d_in, int d_out, boolean have_bias) {\n    this(swigfaissJNI.new_LinearTransform__SWIG_0(d_in, d_out, have_bias), true);\n  }\n\n  public LinearTransform(int d_in, int d_out) {\n    this(swigfaissJNI.new_LinearTransform__SWIG_1(d_in, d_out), true);\n  }\n\n  public LinearTransform(int d_in) {\n    this(swigfaissJNI.new_LinearTransform__SWIG_2(d_in), true);\n  }\n\n  public LinearTransform() {\n    this(swigfaissJNI.new_LinearTransform__SWIG_3(), true);\n  }\n\n  public void apply_noalloc(long n, SWIGTYPE_p_float x, SWIGTYPE_p_float xt) {\n    swigfaissJNI.LinearTransform_apply_noalloc(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(xt));\n  }\n\n  public void transform_transpose(long n, SWIGTYPE_p_float y, SWIGTYPE_p_float x) {\n    swigfaissJNI.LinearTransform_transform_transpose(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(y), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void reverse_transform(long n, SWIGTYPE_p_float xt, SWIGTYPE_p_float x) {\n    swigfaissJNI.LinearTransform_reverse_transform(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(xt), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void set_is_orthonormal() {\n    swigfaissJNI.LinearTransform_set_is_orthonormal(swigCPtr, this);\n  }\n\n  public void setVerbose(boolean value) {\n    swigfaissJNI.LinearTransform_verbose_set(swigCPtr, this, value);\n  }\n\n  public boolean getVerbose() {\n    return swigfaissJNI.LinearTransform_verbose_get(swigCPtr, this);\n  }\n\n  public void print_if_verbose(String name, DoubleVector mat, int n, int d) {\n    swigfaissJNI.LinearTransform_print_if_verbose(swigCPtr, this, name, DoubleVector.getCPtr(mat), mat, n, d);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/LongVector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class LongVector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected LongVector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(LongVector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_LongVector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public LongVector() {\n    this(swigfaissJNI.new_LongVector(), true);\n  }\n\n  public void push_back(long arg0) {\n    swigfaissJNI.LongVector_push_back(swigCPtr, this, arg0);\n  }\n\n  public void clear() {\n    swigfaissJNI.LongVector_clear(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_long_long data() {\n    long cPtr = swigfaissJNI.LongVector_data(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_long_long(cPtr, false);\n  }\n\n  public long size() {\n    return swigfaissJNI.LongVector_size(swigCPtr, this);\n  }\n\n  public long at(long n) {\n    return swigfaissJNI.LongVector_at(swigCPtr, this, n);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.LongVector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.LongVector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(LongVector other) {\n    swigfaissJNI.LongVector_swap(swigCPtr, this, LongVector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/LongVectorVector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class LongVectorVector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected LongVectorVector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(LongVectorVector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_LongVectorVector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public LongVectorVector() {\n    this(swigfaissJNI.new_LongVectorVector(), true);\n  }\n\n  public void push_back(SWIGTYPE_p_std__vectorT_long_t arg0) {\n    swigfaissJNI.LongVectorVector_push_back(swigCPtr, this, SWIGTYPE_p_std__vectorT_long_t.getCPtr(arg0));\n  }\n\n  public void clear() {\n    swigfaissJNI.LongVectorVector_clear(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_std__vectorT_long_t data() {\n    long cPtr = swigfaissJNI.LongVectorVector_data(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_long_t(cPtr, false);\n  }\n\n  public long size() {\n    return swigfaissJNI.LongVectorVector_size(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_std__vectorT_long_t at(long n) {\n    return new SWIGTYPE_p_std__vectorT_long_t(swigfaissJNI.LongVectorVector_at(swigCPtr, this, n), true);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.LongVectorVector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.LongVectorVector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(LongVectorVector other) {\n    swigfaissJNI.LongVectorVector_swap(swigCPtr, this, LongVectorVector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/MapLong2Long.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class MapLong2Long {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected MapLong2Long(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(MapLong2Long obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_MapLong2Long(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setMap(SWIGTYPE_p_std__unordered_mapT_long_long_t value) {\n    swigfaissJNI.MapLong2Long_map_set(swigCPtr, this, SWIGTYPE_p_std__unordered_mapT_long_long_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__unordered_mapT_long_long_t getMap() {\n    return new SWIGTYPE_p_std__unordered_mapT_long_long_t(swigfaissJNI.MapLong2Long_map_get(swigCPtr, this), true);\n  }\n\n  public void add(long n, SWIGTYPE_p_long keys, SWIGTYPE_p_long vals) {\n    swigfaissJNI.MapLong2Long_add(swigCPtr, this, n, SWIGTYPE_p_long.getCPtr(keys), SWIGTYPE_p_long.getCPtr(vals));\n  }\n\n  public int search(int key) {\n    return swigfaissJNI.MapLong2Long_search(swigCPtr, this, key);\n  }\n\n  public void search_multiple(long n, SWIGTYPE_p_long keys, SWIGTYPE_p_long vals) {\n    swigfaissJNI.MapLong2Long_search_multiple(swigCPtr, this, n, SWIGTYPE_p_long.getCPtr(keys), SWIGTYPE_p_long.getCPtr(vals));\n  }\n\n  public MapLong2Long() {\n    this(swigfaissJNI.new_MapLong2Long(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/MaskedInvertedLists.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class MaskedInvertedLists extends ReadOnlyInvertedLists {\n  private transient long swigCPtr;\n\n  protected MaskedInvertedLists(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.MaskedInvertedLists_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(MaskedInvertedLists obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_MaskedInvertedLists(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setIl0(InvertedLists value) {\n    swigfaissJNI.MaskedInvertedLists_il0_set(swigCPtr, this, InvertedLists.getCPtr(value), value);\n  }\n\n  public InvertedLists getIl0() {\n    long cPtr = swigfaissJNI.MaskedInvertedLists_il0_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n  }\n\n  public void setIl1(InvertedLists value) {\n    swigfaissJNI.MaskedInvertedLists_il1_set(swigCPtr, this, InvertedLists.getCPtr(value), value);\n  }\n\n  public InvertedLists getIl1() {\n    long cPtr = swigfaissJNI.MaskedInvertedLists_il1_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n  }\n\n  public MaskedInvertedLists(InvertedLists il0, InvertedLists il1) {\n    this(swigfaissJNI.new_MaskedInvertedLists(InvertedLists.getCPtr(il0), il0, InvertedLists.getCPtr(il1), il1), true);\n  }\n\n  public long list_size(long list_no) {\n    return swigfaissJNI.MaskedInvertedLists_list_size(swigCPtr, this, list_no);\n  }\n\n  public SWIGTYPE_p_unsigned_char get_codes(long list_no) {\n    long cPtr = swigfaissJNI.MaskedInvertedLists_get_codes(swigCPtr, this, list_no);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public LongVector get_ids(long list_no) {\n    return new LongVector(swigfaissJNI.MaskedInvertedLists_get_ids(swigCPtr, this, list_no), false);\n}\n\n  public void release_codes(long list_no, SWIGTYPE_p_unsigned_char codes) {\n    swigfaissJNI.MaskedInvertedLists_release_codes(swigCPtr, this, list_no, SWIGTYPE_p_unsigned_char.getCPtr(codes));\n  }\n\n  public void release_ids(long list_no, LongVector ids) {\n    swigfaissJNI.MaskedInvertedLists_release_ids(swigCPtr, this, list_no, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids);\n  }\n\n  public long get_single_id(long list_no, long offset) {\n    return swigfaissJNI.MaskedInvertedLists_get_single_id(swigCPtr, this, list_no, offset);\n}\n\n  public SWIGTYPE_p_unsigned_char get_single_code(long list_no, long offset) {\n    long cPtr = swigfaissJNI.MaskedInvertedLists_get_single_code(swigCPtr, this, list_no, offset);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public void prefetch_lists(LongVector list_nos, int nlist) {\n    swigfaissJNI.MaskedInvertedLists_prefetch_lists(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, nlist);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/MetricType.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic final class MetricType {\n  public final static MetricType METRIC_INNER_PRODUCT = new MetricType(\"METRIC_INNER_PRODUCT\", swigfaissJNI.METRIC_INNER_PRODUCT_get());\n  public final static MetricType METRIC_L2 = new MetricType(\"METRIC_L2\", swigfaissJNI.METRIC_L2_get());\n  public final static MetricType METRIC_L1 = new MetricType(\"METRIC_L1\");\n  public final static MetricType METRIC_Linf = new MetricType(\"METRIC_Linf\");\n  public final static MetricType METRIC_Lp = new MetricType(\"METRIC_Lp\");\n  public final static MetricType METRIC_Canberra = new MetricType(\"METRIC_Canberra\", swigfaissJNI.METRIC_Canberra_get());\n  public final static MetricType METRIC_BrayCurtis = new MetricType(\"METRIC_BrayCurtis\");\n  public final static MetricType METRIC_JensenShannon = new MetricType(\"METRIC_JensenShannon\");\n\n  public final int swigValue() {\n    return swigValue;\n  }\n\n  public String toString() {\n    return swigName;\n  }\n\n  public static MetricType swigToEnum(int swigValue) {\n    if (swigValue < swigValues.length && swigValue >= 0 && swigValues[swigValue].swigValue == swigValue)\n      return swigValues[swigValue];\n    for (int i = 0; i < swigValues.length; i++)\n      if (swigValues[i].swigValue == swigValue)\n        return swigValues[i];\n    throw new IllegalArgumentException(\"No enum \" + MetricType.class + \" with value \" + swigValue);\n  }\n\n  private MetricType(String swigName) {\n    this.swigName = swigName;\n    this.swigValue = swigNext++;\n  }\n\n  private MetricType(String swigName, int swigValue) {\n    this.swigName = swigName;\n    this.swigValue = swigValue;\n    swigNext = swigValue+1;\n  }\n\n  private MetricType(String swigName, MetricType swigEnum) {\n    this.swigName = swigName;\n    this.swigValue = swigEnum.swigValue;\n    swigNext = this.swigValue+1;\n  }\n\n  private static MetricType[] swigValues = { METRIC_INNER_PRODUCT, METRIC_L2, METRIC_L1, METRIC_Linf, METRIC_Lp, METRIC_Canberra, METRIC_BrayCurtis, METRIC_JensenShannon };\n  private static int swigNext = 0;\n  private final int swigValue;\n  private final String swigName;\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/MultiIndexQuantizer.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class MultiIndexQuantizer extends Index {\n  private transient long swigCPtr;\n\n  protected MultiIndexQuantizer(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.MultiIndexQuantizer_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(MultiIndexQuantizer obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_MultiIndexQuantizer(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setPq(ProductQuantizer value) {\n    swigfaissJNI.MultiIndexQuantizer_pq_set(swigCPtr, this, ProductQuantizer.getCPtr(value), value);\n  }\n\n  public ProductQuantizer getPq() {\n    long cPtr = swigfaissJNI.MultiIndexQuantizer_pq_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ProductQuantizer(cPtr, false);\n  }\n\n  public MultiIndexQuantizer(int d, long M, long nbits) {\n    this(swigfaissJNI.new_MultiIndexQuantizer__SWIG_0(d, M, nbits), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.MultiIndexQuantizer_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.MultiIndexQuantizer_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public void add(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.MultiIndexQuantizer_add(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void reset() {\n    swigfaissJNI.MultiIndexQuantizer_reset(swigCPtr, this);\n  }\n\n  public MultiIndexQuantizer() {\n    this(swigfaissJNI.new_MultiIndexQuantizer__SWIG_1(), true);\n  }\n\n  public void reconstruct(long key, SWIGTYPE_p_float recons) {\n    swigfaissJNI.MultiIndexQuantizer_reconstruct(swigCPtr, this, key, SWIGTYPE_p_float.getCPtr(recons));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/MultiIndexQuantizer2.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class MultiIndexQuantizer2 extends MultiIndexQuantizer {\n  private transient long swigCPtr;\n\n  protected MultiIndexQuantizer2(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.MultiIndexQuantizer2_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(MultiIndexQuantizer2 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_MultiIndexQuantizer2(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setAssign_indexes(SWIGTYPE_p_std__vectorT_faiss__Index_p_t value) {\n    swigfaissJNI.MultiIndexQuantizer2_assign_indexes_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_faiss__Index_p_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_faiss__Index_p_t getAssign_indexes() {\n    long cPtr = swigfaissJNI.MultiIndexQuantizer2_assign_indexes_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_faiss__Index_p_t(cPtr, false);\n  }\n\n  public void setOwn_fields(boolean value) {\n    swigfaissJNI.MultiIndexQuantizer2_own_fields_set(swigCPtr, this, value);\n  }\n\n  public boolean getOwn_fields() {\n    return swigfaissJNI.MultiIndexQuantizer2_own_fields_get(swigCPtr, this);\n  }\n\n  public MultiIndexQuantizer2(int d, long M, long nbits, SWIGTYPE_p_p_faiss__Index indexes) {\n    this(swigfaissJNI.new_MultiIndexQuantizer2__SWIG_0(d, M, nbits, SWIGTYPE_p_p_faiss__Index.getCPtr(indexes)), true);\n  }\n\n  public MultiIndexQuantizer2(int d, long nbits, Index assign_index_0, Index assign_index_1) {\n    this(swigfaissJNI.new_MultiIndexQuantizer2__SWIG_1(d, nbits, Index.getCPtr(assign_index_0), assign_index_0, Index.getCPtr(assign_index_1), assign_index_1), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.MultiIndexQuantizer2_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void search(long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels) {\n    swigfaissJNI.MultiIndexQuantizer2_search(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/NormalizationTransform.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class NormalizationTransform extends VectorTransform {\n  private transient long swigCPtr;\n\n  protected NormalizationTransform(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.NormalizationTransform_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(NormalizationTransform obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_NormalizationTransform(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setNorm(float value) {\n    swigfaissJNI.NormalizationTransform_norm_set(swigCPtr, this, value);\n  }\n\n  public float getNorm() {\n    return swigfaissJNI.NormalizationTransform_norm_get(swigCPtr, this);\n  }\n\n  public NormalizationTransform(int d, float norm) {\n    this(swigfaissJNI.new_NormalizationTransform__SWIG_0(d, norm), true);\n  }\n\n  public NormalizationTransform(int d) {\n    this(swigfaissJNI.new_NormalizationTransform__SWIG_1(d), true);\n  }\n\n  public NormalizationTransform() {\n    this(swigfaissJNI.new_NormalizationTransform__SWIG_2(), true);\n  }\n\n  public void apply_noalloc(long n, SWIGTYPE_p_float x, SWIGTYPE_p_float xt) {\n    swigfaissJNI.NormalizationTransform_apply_noalloc(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(xt));\n  }\n\n  public void reverse_transform(long n, SWIGTYPE_p_float xt, SWIGTYPE_p_float x) {\n    swigfaissJNI.NormalizationTransform_reverse_transform(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(xt), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/OPQMatrix.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class OPQMatrix extends LinearTransform {\n  private transient long swigCPtr;\n\n  protected OPQMatrix(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.OPQMatrix_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(OPQMatrix obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_OPQMatrix(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setM(int value) {\n    swigfaissJNI.OPQMatrix_M_set(swigCPtr, this, value);\n  }\n\n  public int getM() {\n    return swigfaissJNI.OPQMatrix_M_get(swigCPtr, this);\n  }\n\n  public void setNiter(int value) {\n    swigfaissJNI.OPQMatrix_niter_set(swigCPtr, this, value);\n  }\n\n  public int getNiter() {\n    return swigfaissJNI.OPQMatrix_niter_get(swigCPtr, this);\n  }\n\n  public void setNiter_pq(int value) {\n    swigfaissJNI.OPQMatrix_niter_pq_set(swigCPtr, this, value);\n  }\n\n  public int getNiter_pq() {\n    return swigfaissJNI.OPQMatrix_niter_pq_get(swigCPtr, this);\n  }\n\n  public void setNiter_pq_0(int value) {\n    swigfaissJNI.OPQMatrix_niter_pq_0_set(swigCPtr, this, value);\n  }\n\n  public int getNiter_pq_0() {\n    return swigfaissJNI.OPQMatrix_niter_pq_0_get(swigCPtr, this);\n  }\n\n  public void setMax_train_points(long value) {\n    swigfaissJNI.OPQMatrix_max_train_points_set(swigCPtr, this, value);\n  }\n\n  public long getMax_train_points() {\n    return swigfaissJNI.OPQMatrix_max_train_points_get(swigCPtr, this);\n  }\n\n  public void setVerbose(boolean value) {\n    swigfaissJNI.OPQMatrix_verbose_set(swigCPtr, this, value);\n  }\n\n  public boolean getVerbose() {\n    return swigfaissJNI.OPQMatrix_verbose_get(swigCPtr, this);\n  }\n\n  public void setPq(ProductQuantizer value) {\n    swigfaissJNI.OPQMatrix_pq_set(swigCPtr, this, ProductQuantizer.getCPtr(value), value);\n  }\n\n  public ProductQuantizer getPq() {\n    long cPtr = swigfaissJNI.OPQMatrix_pq_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ProductQuantizer(cPtr, false);\n  }\n\n  public OPQMatrix(int d, int M, int d2) {\n    this(swigfaissJNI.new_OPQMatrix__SWIG_0(d, M, d2), true);\n  }\n\n  public OPQMatrix(int d, int M) {\n    this(swigfaissJNI.new_OPQMatrix__SWIG_1(d, M), true);\n  }\n\n  public OPQMatrix(int d) {\n    this(swigfaissJNI.new_OPQMatrix__SWIG_2(d), true);\n  }\n\n  public OPQMatrix() {\n    this(swigfaissJNI.new_OPQMatrix__SWIG_3(), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.OPQMatrix_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/OnDiskInvertedLists.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class OnDiskInvertedLists extends InvertedLists {\n  private transient long swigCPtr;\n\n  protected OnDiskInvertedLists(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.OnDiskInvertedLists_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(OnDiskInvertedLists obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_OnDiskInvertedLists(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setLists(SWIGTYPE_p_std__vectorT_faiss__OnDiskOneList_t value) {\n    swigfaissJNI.OnDiskInvertedLists_lists_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_faiss__OnDiskOneList_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_faiss__OnDiskOneList_t getLists() {\n    long cPtr = swigfaissJNI.OnDiskInvertedLists_lists_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_faiss__OnDiskOneList_t(cPtr, false);\n  }\n\n  static public class Slot {\n    private transient long swigCPtr;\n    protected transient boolean swigCMemOwn;\n  \n    protected Slot(long cPtr, boolean cMemoryOwn) {\n      swigCMemOwn = cMemoryOwn;\n      swigCPtr = cPtr;\n    }\n  \n    protected static long getCPtr(Slot obj) {\n      return (obj == null) ? 0 : obj.swigCPtr;\n    }\n  \n    @SuppressWarnings(\"deprecation\")\n    protected void finalize() {\n      delete();\n    }\n  \n    public synchronized void delete() {\n      if (swigCPtr != 0) {\n        if (swigCMemOwn) {\n          swigCMemOwn = false;\n          swigfaissJNI.delete_OnDiskInvertedLists_Slot(swigCPtr);\n        }\n        swigCPtr = 0;\n      }\n    }\n  \n    public void setOffset(long value) {\n      swigfaissJNI.OnDiskInvertedLists_Slot_offset_set(swigCPtr, this, value);\n    }\n  \n    public long getOffset() {\n      return swigfaissJNI.OnDiskInvertedLists_Slot_offset_get(swigCPtr, this);\n    }\n  \n    public void setCapacity(long value) {\n      swigfaissJNI.OnDiskInvertedLists_Slot_capacity_set(swigCPtr, this, value);\n    }\n  \n    public long getCapacity() {\n      return swigfaissJNI.OnDiskInvertedLists_Slot_capacity_get(swigCPtr, this);\n    }\n  \n    public Slot(long offset, long capacity) {\n      this(swigfaissJNI.new_OnDiskInvertedLists_Slot__SWIG_0(offset, capacity), true);\n    }\n  \n    public Slot() {\n      this(swigfaissJNI.new_OnDiskInvertedLists_Slot__SWIG_1(), true);\n    }\n  \n  }\n\n  public void setSlots(SWIGTYPE_p_std__listT_faiss__OnDiskInvertedLists__Slot_t value) {\n    swigfaissJNI.OnDiskInvertedLists_slots_set(swigCPtr, this, SWIGTYPE_p_std__listT_faiss__OnDiskInvertedLists__Slot_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__listT_faiss__OnDiskInvertedLists__Slot_t getSlots() {\n    long cPtr = swigfaissJNI.OnDiskInvertedLists_slots_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__listT_faiss__OnDiskInvertedLists__Slot_t(cPtr, false);\n  }\n\n  public void setFilename(String value) {\n    swigfaissJNI.OnDiskInvertedLists_filename_set(swigCPtr, this, value);\n  }\n\n  public String getFilename() {\n    return swigfaissJNI.OnDiskInvertedLists_filename_get(swigCPtr, this);\n  }\n\n  public void setTotsize(long value) {\n    swigfaissJNI.OnDiskInvertedLists_totsize_set(swigCPtr, this, value);\n  }\n\n  public long getTotsize() {\n    return swigfaissJNI.OnDiskInvertedLists_totsize_get(swigCPtr, this);\n  }\n\n  public void setPtr(SWIGTYPE_p_unsigned_char value) {\n    swigfaissJNI.OnDiskInvertedLists_ptr_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_unsigned_char getPtr() {\n    long cPtr = swigfaissJNI.OnDiskInvertedLists_ptr_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public void setRead_only(boolean value) {\n    swigfaissJNI.OnDiskInvertedLists_read_only_set(swigCPtr, this, value);\n  }\n\n  public boolean getRead_only() {\n    return swigfaissJNI.OnDiskInvertedLists_read_only_get(swigCPtr, this);\n  }\n\n  public OnDiskInvertedLists(long nlist, long code_size, String filename) {\n    this(swigfaissJNI.new_OnDiskInvertedLists__SWIG_0(nlist, code_size, filename), true);\n  }\n\n  public long list_size(long list_no) {\n    return swigfaissJNI.OnDiskInvertedLists_list_size(swigCPtr, this, list_no);\n  }\n\n  public SWIGTYPE_p_unsigned_char get_codes(long list_no) {\n    long cPtr = swigfaissJNI.OnDiskInvertedLists_get_codes(swigCPtr, this, list_no);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public LongVector get_ids(long list_no) {\n    return new LongVector(swigfaissJNI.OnDiskInvertedLists_get_ids(swigCPtr, this, list_no), false);\n}\n\n  public long add_entries(long list_no, long n_entry, LongVector ids, SWIGTYPE_p_unsigned_char code) {\n    return swigfaissJNI.OnDiskInvertedLists_add_entries(swigCPtr, this, list_no, n_entry, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void update_entries(long list_no, long offset, long n_entry, LongVector ids, SWIGTYPE_p_unsigned_char code) {\n    swigfaissJNI.OnDiskInvertedLists_update_entries(swigCPtr, this, list_no, offset, n_entry, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void resize(long list_no, long new_size) {\n    swigfaissJNI.OnDiskInvertedLists_resize(swigCPtr, this, list_no, new_size);\n  }\n\n  public long merge_from(SWIGTYPE_p_p_faiss__InvertedLists ils, int n_il, boolean verbose) {\n    return swigfaissJNI.OnDiskInvertedLists_merge_from__SWIG_0(swigCPtr, this, SWIGTYPE_p_p_faiss__InvertedLists.getCPtr(ils), n_il, verbose);\n  }\n\n  public long merge_from(SWIGTYPE_p_p_faiss__InvertedLists ils, int n_il) {\n    return swigfaissJNI.OnDiskInvertedLists_merge_from__SWIG_1(swigCPtr, this, SWIGTYPE_p_p_faiss__InvertedLists.getCPtr(ils), n_il);\n  }\n\n  public long merge_from_1(InvertedLists il, boolean verbose) {\n    return swigfaissJNI.OnDiskInvertedLists_merge_from_1__SWIG_0(swigCPtr, this, InvertedLists.getCPtr(il), il, verbose);\n  }\n\n  public long merge_from_1(InvertedLists il) {\n    return swigfaissJNI.OnDiskInvertedLists_merge_from_1__SWIG_1(swigCPtr, this, InvertedLists.getCPtr(il), il);\n  }\n\n  public void crop_invlists(long l0, long l1) {\n    swigfaissJNI.OnDiskInvertedLists_crop_invlists(swigCPtr, this, l0, l1);\n  }\n\n  public void prefetch_lists(LongVector list_nos, int nlist) {\n    swigfaissJNI.OnDiskInvertedLists_prefetch_lists(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, nlist);\n  }\n\n  public void setLocks(SWIGTYPE_p_faiss__LockLevels value) {\n    swigfaissJNI.OnDiskInvertedLists_locks_set(swigCPtr, this, SWIGTYPE_p_faiss__LockLevels.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_faiss__LockLevels getLocks() {\n    long cPtr = swigfaissJNI.OnDiskInvertedLists_locks_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__LockLevels(cPtr, false);\n  }\n\n  public void setPf(SWIGTYPE_p_faiss__OnDiskInvertedLists__OngoingPrefetch value) {\n    swigfaissJNI.OnDiskInvertedLists_pf_set(swigCPtr, this, SWIGTYPE_p_faiss__OnDiskInvertedLists__OngoingPrefetch.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_faiss__OnDiskInvertedLists__OngoingPrefetch getPf() {\n    long cPtr = swigfaissJNI.OnDiskInvertedLists_pf_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__OnDiskInvertedLists__OngoingPrefetch(cPtr, false);\n  }\n\n  public void setPrefetch_nthread(int value) {\n    swigfaissJNI.OnDiskInvertedLists_prefetch_nthread_set(swigCPtr, this, value);\n  }\n\n  public int getPrefetch_nthread() {\n    return swigfaissJNI.OnDiskInvertedLists_prefetch_nthread_get(swigCPtr, this);\n  }\n\n  public void do_mmap() {\n    swigfaissJNI.OnDiskInvertedLists_do_mmap(swigCPtr, this);\n  }\n\n  public void update_totsize(long new_totsize) {\n    swigfaissJNI.OnDiskInvertedLists_update_totsize(swigCPtr, this, new_totsize);\n  }\n\n  public void resize_locked(long list_no, long new_size) {\n    swigfaissJNI.OnDiskInvertedLists_resize_locked(swigCPtr, this, list_no, new_size);\n  }\n\n  public long allocate_slot(long capacity) {\n    return swigfaissJNI.OnDiskInvertedLists_allocate_slot(swigCPtr, this, capacity);\n  }\n\n  public void free_slot(long offset, long capacity) {\n    swigfaissJNI.OnDiskInvertedLists_free_slot(swigCPtr, this, offset, capacity);\n  }\n\n  public void set_all_lists_sizes(SWIGTYPE_p_unsigned_long sizes) {\n    swigfaissJNI.OnDiskInvertedLists_set_all_lists_sizes(swigCPtr, this, SWIGTYPE_p_unsigned_long.getCPtr(sizes));\n  }\n\n  public OnDiskInvertedLists() {\n    this(swigfaissJNI.new_OnDiskInvertedLists__SWIG_1(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/OnDiskInvertedListsIOHook.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class OnDiskInvertedListsIOHook {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected OnDiskInvertedListsIOHook(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(OnDiskInvertedListsIOHook obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_OnDiskInvertedListsIOHook(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public OnDiskInvertedListsIOHook() {\n    this(swigfaissJNI.new_OnDiskInvertedListsIOHook(), true);\n  }\n\n  public void write(InvertedLists ils, SWIGTYPE_p_IOWriter f) {\n    swigfaissJNI.OnDiskInvertedListsIOHook_write(swigCPtr, this, InvertedLists.getCPtr(ils), ils, SWIGTYPE_p_IOWriter.getCPtr(f));\n  }\n\n  public InvertedLists read(SWIGTYPE_p_IOReader f, int io_flags) {\n    long cPtr = swigfaissJNI.OnDiskInvertedListsIOHook_read(swigCPtr, this, SWIGTYPE_p_IOReader.getCPtr(f), io_flags);\n    return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n  }\n\n  public InvertedLists read_ArrayInvertedLists(SWIGTYPE_p_IOReader f, int io_flags, long nlist, long code_size, Uint64Vector sizes) {\n    long cPtr = swigfaissJNI.OnDiskInvertedListsIOHook_read_ArrayInvertedLists(swigCPtr, this, SWIGTYPE_p_IOReader.getCPtr(f), io_flags, nlist, code_size, Uint64Vector.getCPtr(sizes), sizes);\n    return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/OnDiskOneList.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class OnDiskOneList {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected OnDiskOneList(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(OnDiskOneList obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_OnDiskOneList(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setSize(long value) {\n    swigfaissJNI.OnDiskOneList_size_set(swigCPtr, this, value);\n  }\n\n  public long getSize() {\n    return swigfaissJNI.OnDiskOneList_size_get(swigCPtr, this);\n  }\n\n  public void setCapacity(long value) {\n    swigfaissJNI.OnDiskOneList_capacity_set(swigCPtr, this, value);\n  }\n\n  public long getCapacity() {\n    return swigfaissJNI.OnDiskOneList_capacity_get(swigCPtr, this);\n  }\n\n  public void setOffset(long value) {\n    swigfaissJNI.OnDiskOneList_offset_set(swigCPtr, this, value);\n  }\n\n  public long getOffset() {\n    return swigfaissJNI.OnDiskOneList_offset_get(swigCPtr, this);\n  }\n\n  public OnDiskOneList() {\n    this(swigfaissJNI.new_OnDiskOneList(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/OneRecallAtRCriterion.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class OneRecallAtRCriterion extends AutoTuneCriterion {\n  private transient long swigCPtr;\n\n  protected OneRecallAtRCriterion(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.OneRecallAtRCriterion_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(OneRecallAtRCriterion obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_OneRecallAtRCriterion(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setR(long value) {\n    swigfaissJNI.OneRecallAtRCriterion_R_set(swigCPtr, this, value);\n  }\n\n  public long getR() {\n    return swigfaissJNI.OneRecallAtRCriterion_R_get(swigCPtr, this);\n}\n\n  public OneRecallAtRCriterion(long nq, long R) {\n    this(swigfaissJNI.new_OneRecallAtRCriterion(nq, R), true);\n  }\n\n  public double evaluate(SWIGTYPE_p_float D, LongVector I) {\n    return swigfaissJNI.OneRecallAtRCriterion_evaluate(swigCPtr, this, SWIGTYPE_p_float.getCPtr(D), SWIGTYPE_p_long_long.getCPtr(I.data()), I);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/OperatingPoint.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class OperatingPoint {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected OperatingPoint(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(OperatingPoint obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_OperatingPoint(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setPerf(double value) {\n    swigfaissJNI.OperatingPoint_perf_set(swigCPtr, this, value);\n  }\n\n  public double getPerf() {\n    return swigfaissJNI.OperatingPoint_perf_get(swigCPtr, this);\n  }\n\n  public void setT(double value) {\n    swigfaissJNI.OperatingPoint_t_set(swigCPtr, this, value);\n  }\n\n  public double getT() {\n    return swigfaissJNI.OperatingPoint_t_get(swigCPtr, this);\n  }\n\n  public void setKey(String value) {\n    swigfaissJNI.OperatingPoint_key_set(swigCPtr, this, value);\n  }\n\n  public String getKey() {\n    return swigfaissJNI.OperatingPoint_key_get(swigCPtr, this);\n  }\n\n  public void setCno(long value) {\n    swigfaissJNI.OperatingPoint_cno_set(swigCPtr, this, value);\n  }\n\n  public long getCno() {\n    return swigfaissJNI.OperatingPoint_cno_get(swigCPtr, this);\n}\n\n  public OperatingPoint() {\n    this(swigfaissJNI.new_OperatingPoint(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/OperatingPointVector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class OperatingPointVector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected OperatingPointVector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(OperatingPointVector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_OperatingPointVector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public OperatingPointVector() {\n    this(swigfaissJNI.new_OperatingPointVector(), true);\n  }\n\n  public void push_back(OperatingPoint arg0) {\n    swigfaissJNI.OperatingPointVector_push_back(swigCPtr, this, OperatingPoint.getCPtr(arg0), arg0);\n  }\n\n  public void clear() {\n    swigfaissJNI.OperatingPointVector_clear(swigCPtr, this);\n  }\n\n  public OperatingPoint data() {\n    long cPtr = swigfaissJNI.OperatingPointVector_data(swigCPtr, this);\n    return (cPtr == 0) ? null : new OperatingPoint(cPtr, false);\n  }\n\n  public long size() {\n    return swigfaissJNI.OperatingPointVector_size(swigCPtr, this);\n  }\n\n  public OperatingPoint at(long n) {\n    return new OperatingPoint(swigfaissJNI.OperatingPointVector_at(swigCPtr, this, n), true);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.OperatingPointVector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.OperatingPointVector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(OperatingPointVector other) {\n    swigfaissJNI.OperatingPointVector_swap(swigCPtr, this, OperatingPointVector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/OperatingPoints.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class OperatingPoints {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected OperatingPoints(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(OperatingPoints obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_OperatingPoints(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setAll_pts(OperatingPointVector value) {\n    swigfaissJNI.OperatingPoints_all_pts_set(swigCPtr, this, OperatingPointVector.getCPtr(value), value);\n  }\n\n  public OperatingPointVector getAll_pts() {\n    long cPtr = swigfaissJNI.OperatingPoints_all_pts_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new OperatingPointVector(cPtr, false);\n  }\n\n  public void setOptimal_pts(OperatingPointVector value) {\n    swigfaissJNI.OperatingPoints_optimal_pts_set(swigCPtr, this, OperatingPointVector.getCPtr(value), value);\n  }\n\n  public OperatingPointVector getOptimal_pts() {\n    long cPtr = swigfaissJNI.OperatingPoints_optimal_pts_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new OperatingPointVector(cPtr, false);\n  }\n\n  public OperatingPoints() {\n    this(swigfaissJNI.new_OperatingPoints(), true);\n  }\n\n  public int merge_with(OperatingPoints other, String prefix) {\n    return swigfaissJNI.OperatingPoints_merge_with__SWIG_0(swigCPtr, this, OperatingPoints.getCPtr(other), other, prefix);\n  }\n\n  public int merge_with(OperatingPoints other) {\n    return swigfaissJNI.OperatingPoints_merge_with__SWIG_1(swigCPtr, this, OperatingPoints.getCPtr(other), other);\n  }\n\n  public void clear() {\n    swigfaissJNI.OperatingPoints_clear(swigCPtr, this);\n  }\n\n  public boolean add(double perf, double t, String key, long cno) {\n    return swigfaissJNI.OperatingPoints_add__SWIG_0(swigCPtr, this, perf, t, key, cno);\n  }\n\n  public boolean add(double perf, double t, String key) {\n    return swigfaissJNI.OperatingPoints_add__SWIG_1(swigCPtr, this, perf, t, key);\n  }\n\n  public double t_for_perf(double perf) {\n    return swigfaissJNI.OperatingPoints_t_for_perf(swigCPtr, this, perf);\n  }\n\n  public void display(boolean only_optimal) {\n    swigfaissJNI.OperatingPoints_display__SWIG_0(swigCPtr, this, only_optimal);\n  }\n\n  public void display() {\n    swigfaissJNI.OperatingPoints_display__SWIG_1(swigCPtr, this);\n  }\n\n  public void all_to_gnuplot(String fname) {\n    swigfaissJNI.OperatingPoints_all_to_gnuplot(swigCPtr, this, fname);\n  }\n\n  public void optimal_to_gnuplot(String fname) {\n    swigfaissJNI.OperatingPoints_optimal_to_gnuplot(swigCPtr, this, fname);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/PCAMatrix.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class PCAMatrix extends LinearTransform {\n  private transient long swigCPtr;\n\n  protected PCAMatrix(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.PCAMatrix_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(PCAMatrix obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_PCAMatrix(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setEigen_power(float value) {\n    swigfaissJNI.PCAMatrix_eigen_power_set(swigCPtr, this, value);\n  }\n\n  public float getEigen_power() {\n    return swigfaissJNI.PCAMatrix_eigen_power_get(swigCPtr, this);\n  }\n\n  public void setEpsilon(float value) {\n    swigfaissJNI.PCAMatrix_epsilon_set(swigCPtr, this, value);\n  }\n\n  public float getEpsilon() {\n    return swigfaissJNI.PCAMatrix_epsilon_get(swigCPtr, this);\n  }\n\n  public void setRandom_rotation(boolean value) {\n    swigfaissJNI.PCAMatrix_random_rotation_set(swigCPtr, this, value);\n  }\n\n  public boolean getRandom_rotation() {\n    return swigfaissJNI.PCAMatrix_random_rotation_get(swigCPtr, this);\n  }\n\n  public void setMax_points_per_d(long value) {\n    swigfaissJNI.PCAMatrix_max_points_per_d_set(swigCPtr, this, value);\n  }\n\n  public long getMax_points_per_d() {\n    return swigfaissJNI.PCAMatrix_max_points_per_d_get(swigCPtr, this);\n  }\n\n  public void setBalanced_bins(int value) {\n    swigfaissJNI.PCAMatrix_balanced_bins_set(swigCPtr, this, value);\n  }\n\n  public int getBalanced_bins() {\n    return swigfaissJNI.PCAMatrix_balanced_bins_get(swigCPtr, this);\n  }\n\n  public void setMean(FloatVector value) {\n    swigfaissJNI.PCAMatrix_mean_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getMean() {\n    long cPtr = swigfaissJNI.PCAMatrix_mean_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public void setEigenvalues(FloatVector value) {\n    swigfaissJNI.PCAMatrix_eigenvalues_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getEigenvalues() {\n    long cPtr = swigfaissJNI.PCAMatrix_eigenvalues_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public void setPCAMat(FloatVector value) {\n    swigfaissJNI.PCAMatrix_PCAMat_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getPCAMat() {\n    long cPtr = swigfaissJNI.PCAMatrix_PCAMat_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public PCAMatrix(int d_in, int d_out, float eigen_power, boolean random_rotation) {\n    this(swigfaissJNI.new_PCAMatrix__SWIG_0(d_in, d_out, eigen_power, random_rotation), true);\n  }\n\n  public PCAMatrix(int d_in, int d_out, float eigen_power) {\n    this(swigfaissJNI.new_PCAMatrix__SWIG_1(d_in, d_out, eigen_power), true);\n  }\n\n  public PCAMatrix(int d_in, int d_out) {\n    this(swigfaissJNI.new_PCAMatrix__SWIG_2(d_in, d_out), true);\n  }\n\n  public PCAMatrix(int d_in) {\n    this(swigfaissJNI.new_PCAMatrix__SWIG_3(d_in), true);\n  }\n\n  public PCAMatrix() {\n    this(swigfaissJNI.new_PCAMatrix__SWIG_4(), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.PCAMatrix_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void copy_from(PCAMatrix other) {\n    swigfaissJNI.PCAMatrix_copy_from(swigCPtr, this, PCAMatrix.getCPtr(other), other);\n  }\n\n  public void prepare_Ab() {\n    swigfaissJNI.PCAMatrix_prepare_Ab(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/PQDecoder16.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class PQDecoder16 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected PQDecoder16(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(PQDecoder16 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_PQDecoder16(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setCode(SWIGTYPE_p_uint16_t value) {\n    swigfaissJNI.PQDecoder16_code_set(swigCPtr, this, SWIGTYPE_p_uint16_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_uint16_t getCode() {\n    long cPtr = swigfaissJNI.PQDecoder16_code_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_uint16_t(cPtr, false);\n  }\n\n  public PQDecoder16(SWIGTYPE_p_unsigned_char code, int nbits) {\n    this(swigfaissJNI.new_PQDecoder16(SWIGTYPE_p_unsigned_char.getCPtr(code), nbits), true);\n  }\n\n  public long decode() {\n    return swigfaissJNI.PQDecoder16_decode(swigCPtr, this);\n  }\n\n  public final static int nbits = swigfaissJNI.PQDecoder16_nbits_get();\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/PQDecoder8.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class PQDecoder8 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected PQDecoder8(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(PQDecoder8 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_PQDecoder8(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setCode(SWIGTYPE_p_unsigned_char value) {\n    swigfaissJNI.PQDecoder8_code_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_unsigned_char getCode() {\n    long cPtr = swigfaissJNI.PQDecoder8_code_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public PQDecoder8(SWIGTYPE_p_unsigned_char code, int nbits) {\n    this(swigfaissJNI.new_PQDecoder8(SWIGTYPE_p_unsigned_char.getCPtr(code), nbits), true);\n  }\n\n  public long decode() {\n    return swigfaissJNI.PQDecoder8_decode(swigCPtr, this);\n  }\n\n  public final static int nbits = swigfaissJNI.PQDecoder8_nbits_get();\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/PQDecoderGeneric.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class PQDecoderGeneric {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected PQDecoderGeneric(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(PQDecoderGeneric obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_PQDecoderGeneric(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setCode(SWIGTYPE_p_unsigned_char value) {\n    swigfaissJNI.PQDecoderGeneric_code_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_unsigned_char getCode() {\n    long cPtr = swigfaissJNI.PQDecoderGeneric_code_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public void setOffset(short value) {\n    swigfaissJNI.PQDecoderGeneric_offset_set(swigCPtr, this, value);\n  }\n\n  public short getOffset() {\n    return swigfaissJNI.PQDecoderGeneric_offset_get(swigCPtr, this);\n  }\n\n  public int getNbits() {\n    return swigfaissJNI.PQDecoderGeneric_nbits_get(swigCPtr, this);\n  }\n\n  public long getMask() {\n    return swigfaissJNI.PQDecoderGeneric_mask_get(swigCPtr, this);\n  }\n\n  public void setReg(short value) {\n    swigfaissJNI.PQDecoderGeneric_reg_set(swigCPtr, this, value);\n  }\n\n  public short getReg() {\n    return swigfaissJNI.PQDecoderGeneric_reg_get(swigCPtr, this);\n  }\n\n  public PQDecoderGeneric(SWIGTYPE_p_unsigned_char code, int nbits) {\n    this(swigfaissJNI.new_PQDecoderGeneric(SWIGTYPE_p_unsigned_char.getCPtr(code), nbits), true);\n  }\n\n  public long decode() {\n    return swigfaissJNI.PQDecoderGeneric_decode(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/PQEncoder16.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class PQEncoder16 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected PQEncoder16(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(PQEncoder16 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_PQEncoder16(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setCode(SWIGTYPE_p_uint16_t value) {\n    swigfaissJNI.PQEncoder16_code_set(swigCPtr, this, SWIGTYPE_p_uint16_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_uint16_t getCode() {\n    long cPtr = swigfaissJNI.PQEncoder16_code_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_uint16_t(cPtr, false);\n  }\n\n  public PQEncoder16(SWIGTYPE_p_unsigned_char code, int nbits) {\n    this(swigfaissJNI.new_PQEncoder16(SWIGTYPE_p_unsigned_char.getCPtr(code), nbits), true);\n  }\n\n  public void encode(long x) {\n    swigfaissJNI.PQEncoder16_encode(swigCPtr, this, x);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/PQEncoder8.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class PQEncoder8 {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected PQEncoder8(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(PQEncoder8 obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_PQEncoder8(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setCode(SWIGTYPE_p_unsigned_char value) {\n    swigfaissJNI.PQEncoder8_code_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_unsigned_char getCode() {\n    long cPtr = swigfaissJNI.PQEncoder8_code_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public PQEncoder8(SWIGTYPE_p_unsigned_char code, int nbits) {\n    this(swigfaissJNI.new_PQEncoder8(SWIGTYPE_p_unsigned_char.getCPtr(code), nbits), true);\n  }\n\n  public void encode(long x) {\n    swigfaissJNI.PQEncoder8_encode(swigCPtr, this, x);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/PQEncoderGeneric.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class PQEncoderGeneric {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected PQEncoderGeneric(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(PQEncoderGeneric obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_PQEncoderGeneric(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setCode(SWIGTYPE_p_unsigned_char value) {\n    swigfaissJNI.PQEncoderGeneric_code_set(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_unsigned_char getCode() {\n    long cPtr = swigfaissJNI.PQEncoderGeneric_code_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public void setOffset(short value) {\n    swigfaissJNI.PQEncoderGeneric_offset_set(swigCPtr, this, value);\n  }\n\n  public short getOffset() {\n    return swigfaissJNI.PQEncoderGeneric_offset_get(swigCPtr, this);\n  }\n\n  public int getNbits() {\n    return swigfaissJNI.PQEncoderGeneric_nbits_get(swigCPtr, this);\n  }\n\n  public void setReg(short value) {\n    swigfaissJNI.PQEncoderGeneric_reg_set(swigCPtr, this, value);\n  }\n\n  public short getReg() {\n    return swigfaissJNI.PQEncoderGeneric_reg_get(swigCPtr, this);\n  }\n\n  public PQEncoderGeneric(SWIGTYPE_p_unsigned_char code, int nbits, short offset) {\n    this(swigfaissJNI.new_PQEncoderGeneric__SWIG_0(SWIGTYPE_p_unsigned_char.getCPtr(code), nbits, offset), true);\n  }\n\n  public PQEncoderGeneric(SWIGTYPE_p_unsigned_char code, int nbits) {\n    this(swigfaissJNI.new_PQEncoderGeneric__SWIG_1(SWIGTYPE_p_unsigned_char.getCPtr(code), nbits), true);\n  }\n\n  public void encode(long x) {\n    swigfaissJNI.PQEncoderGeneric_encode(swigCPtr, this, x);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ParameterRange.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ParameterRange {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected ParameterRange(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ParameterRange obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ParameterRange(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setName(String value) {\n    swigfaissJNI.ParameterRange_name_set(swigCPtr, this, value);\n  }\n\n  public String getName() {\n    return swigfaissJNI.ParameterRange_name_get(swigCPtr, this);\n  }\n\n  public void setValues(DoubleVector value) {\n    swigfaissJNI.ParameterRange_values_set(swigCPtr, this, DoubleVector.getCPtr(value), value);\n  }\n\n  public DoubleVector getValues() {\n    long cPtr = swigfaissJNI.ParameterRange_values_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new DoubleVector(cPtr, false);\n  }\n\n  public ParameterRange() {\n    this(swigfaissJNI.new_ParameterRange(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ParameterSpace.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ParameterSpace {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected ParameterSpace(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ParameterSpace obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ParameterSpace(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setParameter_ranges(SWIGTYPE_p_std__vectorT_faiss__ParameterRange_t value) {\n    swigfaissJNI.ParameterSpace_parameter_ranges_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_faiss__ParameterRange_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_faiss__ParameterRange_t getParameter_ranges() {\n    long cPtr = swigfaissJNI.ParameterSpace_parameter_ranges_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_faiss__ParameterRange_t(cPtr, false);\n  }\n\n  public void setVerbose(int value) {\n    swigfaissJNI.ParameterSpace_verbose_set(swigCPtr, this, value);\n  }\n\n  public int getVerbose() {\n    return swigfaissJNI.ParameterSpace_verbose_get(swigCPtr, this);\n  }\n\n  public void setN_experiments(int value) {\n    swigfaissJNI.ParameterSpace_n_experiments_set(swigCPtr, this, value);\n  }\n\n  public int getN_experiments() {\n    return swigfaissJNI.ParameterSpace_n_experiments_get(swigCPtr, this);\n  }\n\n  public void setBatchsize(long value) {\n    swigfaissJNI.ParameterSpace_batchsize_set(swigCPtr, this, value);\n  }\n\n  public long getBatchsize() {\n    return swigfaissJNI.ParameterSpace_batchsize_get(swigCPtr, this);\n  }\n\n  public void setThread_over_batches(boolean value) {\n    swigfaissJNI.ParameterSpace_thread_over_batches_set(swigCPtr, this, value);\n  }\n\n  public boolean getThread_over_batches() {\n    return swigfaissJNI.ParameterSpace_thread_over_batches_get(swigCPtr, this);\n  }\n\n  public void setMin_test_duration(double value) {\n    swigfaissJNI.ParameterSpace_min_test_duration_set(swigCPtr, this, value);\n  }\n\n  public double getMin_test_duration() {\n    return swigfaissJNI.ParameterSpace_min_test_duration_get(swigCPtr, this);\n  }\n\n  public ParameterSpace() {\n    this(swigfaissJNI.new_ParameterSpace(), true);\n  }\n\n  public long n_combinations() {\n    return swigfaissJNI.ParameterSpace_n_combinations(swigCPtr, this);\n  }\n\n  public boolean combination_ge(long c1, long c2) {\n    return swigfaissJNI.ParameterSpace_combination_ge(swigCPtr, this, c1, c2);\n  }\n\n  public String combination_name(long cno) {\n    return swigfaissJNI.ParameterSpace_combination_name(swigCPtr, this, cno);\n  }\n\n  public void display() {\n    swigfaissJNI.ParameterSpace_display(swigCPtr, this);\n  }\n\n  public ParameterRange add_range(String name) {\n    return new ParameterRange(swigfaissJNI.ParameterSpace_add_range(swigCPtr, this, name), false);\n  }\n\n  public void initialize(Index index) {\n    swigfaissJNI.ParameterSpace_initialize(swigCPtr, this, Index.getCPtr(index), index);\n  }\n\n  public void set_index_parameters(Index index, long cno) {\n    swigfaissJNI.ParameterSpace_set_index_parameters__SWIG_0(swigCPtr, this, Index.getCPtr(index), index, cno);\n  }\n\n  public void set_index_parameters(Index index, String param_string) {\n    swigfaissJNI.ParameterSpace_set_index_parameters__SWIG_1(swigCPtr, this, Index.getCPtr(index), index, param_string);\n  }\n\n  public void set_index_parameter(Index index, String name, double val) {\n    swigfaissJNI.ParameterSpace_set_index_parameter(swigCPtr, this, Index.getCPtr(index), index, name, val);\n  }\n\n  public void update_bounds(long cno, OperatingPoint op, SWIGTYPE_p_double upper_bound_perf, SWIGTYPE_p_double lower_bound_t) {\n    swigfaissJNI.ParameterSpace_update_bounds(swigCPtr, this, cno, OperatingPoint.getCPtr(op), op, SWIGTYPE_p_double.getCPtr(upper_bound_perf), SWIGTYPE_p_double.getCPtr(lower_bound_t));\n  }\n\n  public void explore(Index index, long nq, SWIGTYPE_p_float xq, AutoTuneCriterion crit, OperatingPoints ops) {\n    swigfaissJNI.ParameterSpace_explore(swigCPtr, this, Index.getCPtr(index), index, nq, SWIGTYPE_p_float.getCPtr(xq), AutoTuneCriterion.getCPtr(crit), crit, OperatingPoints.getCPtr(ops), ops);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/PartitionStats.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class PartitionStats {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected PartitionStats(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(PartitionStats obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_PartitionStats(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setBissect_cycles(long value) {\n    swigfaissJNI.PartitionStats_bissect_cycles_set(swigCPtr, this, value);\n  }\n\n  public long getBissect_cycles() {\n    return swigfaissJNI.PartitionStats_bissect_cycles_get(swigCPtr, this);\n  }\n\n  public void setCompress_cycles(long value) {\n    swigfaissJNI.PartitionStats_compress_cycles_set(swigCPtr, this, value);\n  }\n\n  public long getCompress_cycles() {\n    return swigfaissJNI.PartitionStats_compress_cycles_get(swigCPtr, this);\n  }\n\n  public PartitionStats() {\n    this(swigfaissJNI.new_PartitionStats(), true);\n  }\n\n  public void reset() {\n    swigfaissJNI.PartitionStats_reset(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/PermutationObjective.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class PermutationObjective {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected PermutationObjective(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(PermutationObjective obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_PermutationObjective(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setN(int value) {\n    swigfaissJNI.PermutationObjective_n_set(swigCPtr, this, value);\n  }\n\n  public int getN() {\n    return swigfaissJNI.PermutationObjective_n_get(swigCPtr, this);\n  }\n\n  public double compute_cost(SWIGTYPE_p_int perm) {\n    return swigfaissJNI.PermutationObjective_compute_cost(swigCPtr, this, SWIGTYPE_p_int.getCPtr(perm));\n  }\n\n  public double cost_update(SWIGTYPE_p_int perm, int iw, int jw) {\n    return swigfaissJNI.PermutationObjective_cost_update(swigCPtr, this, SWIGTYPE_p_int.getCPtr(perm), iw, jw);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/PolysemousTraining.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class PolysemousTraining extends SimulatedAnnealingParameters {\n  private transient long swigCPtr;\n\n  protected PolysemousTraining(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.PolysemousTraining_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(PolysemousTraining obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_PolysemousTraining(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setOptimization_type(PolysemousTraining.Optimization_type_t value) {\n    swigfaissJNI.PolysemousTraining_optimization_type_set(swigCPtr, this, value.swigValue());\n  }\n\n  public PolysemousTraining.Optimization_type_t getOptimization_type() {\n    return PolysemousTraining.Optimization_type_t.swigToEnum(swigfaissJNI.PolysemousTraining_optimization_type_get(swigCPtr, this));\n  }\n\n  public void setNtrain_permutation(int value) {\n    swigfaissJNI.PolysemousTraining_ntrain_permutation_set(swigCPtr, this, value);\n  }\n\n  public int getNtrain_permutation() {\n    return swigfaissJNI.PolysemousTraining_ntrain_permutation_get(swigCPtr, this);\n  }\n\n  public void setDis_weight_factor(double value) {\n    swigfaissJNI.PolysemousTraining_dis_weight_factor_set(swigCPtr, this, value);\n  }\n\n  public double getDis_weight_factor() {\n    return swigfaissJNI.PolysemousTraining_dis_weight_factor_get(swigCPtr, this);\n  }\n\n  public void setMax_memory(long value) {\n    swigfaissJNI.PolysemousTraining_max_memory_set(swigCPtr, this, value);\n  }\n\n  public long getMax_memory() {\n    return swigfaissJNI.PolysemousTraining_max_memory_get(swigCPtr, this);\n  }\n\n  public void setLog_pattern(String value) {\n    swigfaissJNI.PolysemousTraining_log_pattern_set(swigCPtr, this, value);\n  }\n\n  public String getLog_pattern() {\n    return swigfaissJNI.PolysemousTraining_log_pattern_get(swigCPtr, this);\n  }\n\n  public PolysemousTraining() {\n    this(swigfaissJNI.new_PolysemousTraining(), true);\n  }\n\n  public void optimize_pq_for_hamming(ProductQuantizer pq, long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.PolysemousTraining_optimize_pq_for_hamming(swigCPtr, this, ProductQuantizer.getCPtr(pq), pq, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void optimize_ranking(ProductQuantizer pq, long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.PolysemousTraining_optimize_ranking(swigCPtr, this, ProductQuantizer.getCPtr(pq), pq, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void optimize_reproduce_distances(ProductQuantizer pq) {\n    swigfaissJNI.PolysemousTraining_optimize_reproduce_distances(swigCPtr, this, ProductQuantizer.getCPtr(pq), pq);\n  }\n\n  public long memory_usage_per_thread(ProductQuantizer pq) {\n    return swigfaissJNI.PolysemousTraining_memory_usage_per_thread(swigCPtr, this, ProductQuantizer.getCPtr(pq), pq);\n  }\n\n  public final static class Optimization_type_t {\n    public final static PolysemousTraining.Optimization_type_t OT_None = new PolysemousTraining.Optimization_type_t(\"OT_None\");\n    public final static PolysemousTraining.Optimization_type_t OT_ReproduceDistances_affine = new PolysemousTraining.Optimization_type_t(\"OT_ReproduceDistances_affine\");\n    public final static PolysemousTraining.Optimization_type_t OT_Ranking_weighted_diff = new PolysemousTraining.Optimization_type_t(\"OT_Ranking_weighted_diff\");\n\n    public final int swigValue() {\n      return swigValue;\n    }\n\n    public String toString() {\n      return swigName;\n    }\n\n    public static Optimization_type_t swigToEnum(int swigValue) {\n      if (swigValue < swigValues.length && swigValue >= 0 && swigValues[swigValue].swigValue == swigValue)\n        return swigValues[swigValue];\n      for (int i = 0; i < swigValues.length; i++)\n        if (swigValues[i].swigValue == swigValue)\n          return swigValues[i];\n      throw new IllegalArgumentException(\"No enum \" + Optimization_type_t.class + \" with value \" + swigValue);\n    }\n\n    private Optimization_type_t(String swigName) {\n      this.swigName = swigName;\n      this.swigValue = swigNext++;\n    }\n\n    private Optimization_type_t(String swigName, int swigValue) {\n      this.swigName = swigName;\n      this.swigValue = swigValue;\n      swigNext = swigValue+1;\n    }\n\n    private Optimization_type_t(String swigName, Optimization_type_t swigEnum) {\n      this.swigName = swigName;\n      this.swigValue = swigEnum.swigValue;\n      swigNext = this.swigValue+1;\n    }\n\n    private static Optimization_type_t[] swigValues = { OT_None, OT_ReproduceDistances_affine, OT_Ranking_weighted_diff };\n    private static int swigNext = 0;\n    private final int swigValue;\n    private final String swigName;\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ProductQuantizer.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ProductQuantizer {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected ProductQuantizer(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ProductQuantizer obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ProductQuantizer(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setD(long value) {\n    swigfaissJNI.ProductQuantizer_d_set(swigCPtr, this, value);\n  }\n\n  public long getD() {\n    return swigfaissJNI.ProductQuantizer_d_get(swigCPtr, this);\n  }\n\n  public void setM(long value) {\n    swigfaissJNI.ProductQuantizer_M_set(swigCPtr, this, value);\n  }\n\n  public long getM() {\n    return swigfaissJNI.ProductQuantizer_M_get(swigCPtr, this);\n  }\n\n  public void setNbits(long value) {\n    swigfaissJNI.ProductQuantizer_nbits_set(swigCPtr, this, value);\n  }\n\n  public long getNbits() {\n    return swigfaissJNI.ProductQuantizer_nbits_get(swigCPtr, this);\n  }\n\n  public void setDsub(long value) {\n    swigfaissJNI.ProductQuantizer_dsub_set(swigCPtr, this, value);\n  }\n\n  public long getDsub() {\n    return swigfaissJNI.ProductQuantizer_dsub_get(swigCPtr, this);\n  }\n\n  public void setCode_size(long value) {\n    swigfaissJNI.ProductQuantizer_code_size_set(swigCPtr, this, value);\n  }\n\n  public long getCode_size() {\n    return swigfaissJNI.ProductQuantizer_code_size_get(swigCPtr, this);\n  }\n\n  public void setKsub(long value) {\n    swigfaissJNI.ProductQuantizer_ksub_set(swigCPtr, this, value);\n  }\n\n  public long getKsub() {\n    return swigfaissJNI.ProductQuantizer_ksub_get(swigCPtr, this);\n  }\n\n  public void setVerbose(boolean value) {\n    swigfaissJNI.ProductQuantizer_verbose_set(swigCPtr, this, value);\n  }\n\n  public boolean getVerbose() {\n    return swigfaissJNI.ProductQuantizer_verbose_get(swigCPtr, this);\n  }\n\n  public void setTrain_type(ProductQuantizer.train_type_t value) {\n    swigfaissJNI.ProductQuantizer_train_type_set(swigCPtr, this, value.swigValue());\n  }\n\n  public ProductQuantizer.train_type_t getTrain_type() {\n    return ProductQuantizer.train_type_t.swigToEnum(swigfaissJNI.ProductQuantizer_train_type_get(swigCPtr, this));\n  }\n\n  public void setCp(ClusteringParameters value) {\n    swigfaissJNI.ProductQuantizer_cp_set(swigCPtr, this, ClusteringParameters.getCPtr(value), value);\n  }\n\n  public ClusteringParameters getCp() {\n    long cPtr = swigfaissJNI.ProductQuantizer_cp_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ClusteringParameters(cPtr, false);\n  }\n\n  public void setAssign_index(Index value) {\n    swigfaissJNI.ProductQuantizer_assign_index_set(swigCPtr, this, Index.getCPtr(value), value);\n  }\n\n  public Index getAssign_index() {\n    long cPtr = swigfaissJNI.ProductQuantizer_assign_index_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new Index(cPtr, false);\n  }\n\n  public void setCentroids(FloatVector value) {\n    swigfaissJNI.ProductQuantizer_centroids_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getCentroids() {\n    long cPtr = swigfaissJNI.ProductQuantizer_centroids_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public SWIGTYPE_p_float get_centroids(long m, long i) {\n    long cPtr = swigfaissJNI.ProductQuantizer_get_centroids(swigCPtr, this, m, i);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public void train(int n, SWIGTYPE_p_float x) {\n    swigfaissJNI.ProductQuantizer_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public ProductQuantizer(long d, long M, long nbits) {\n    this(swigfaissJNI.new_ProductQuantizer__SWIG_0(d, M, nbits), true);\n  }\n\n  public ProductQuantizer() {\n    this(swigfaissJNI.new_ProductQuantizer__SWIG_1(), true);\n  }\n\n  public void set_derived_values() {\n    swigfaissJNI.ProductQuantizer_set_derived_values(swigCPtr, this);\n  }\n\n  public void set_params(SWIGTYPE_p_float centroids, int m) {\n    swigfaissJNI.ProductQuantizer_set_params(swigCPtr, this, SWIGTYPE_p_float.getCPtr(centroids), m);\n  }\n\n  public void compute_code(SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char code) {\n    swigfaissJNI.ProductQuantizer_compute_code(swigCPtr, this, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void compute_codes(SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char codes, long n) {\n    swigfaissJNI.ProductQuantizer_compute_codes(swigCPtr, this, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(codes), n);\n  }\n\n  public void compute_codes_with_assign_index(SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char codes, long n) {\n    swigfaissJNI.ProductQuantizer_compute_codes_with_assign_index(swigCPtr, this, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(codes), n);\n  }\n\n  public void decode(SWIGTYPE_p_unsigned_char code, SWIGTYPE_p_float x) {\n    swigfaissJNI.ProductQuantizer_decode__SWIG_0(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(code), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void decode(SWIGTYPE_p_unsigned_char code, SWIGTYPE_p_float x, long n) {\n    swigfaissJNI.ProductQuantizer_decode__SWIG_1(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(code), SWIGTYPE_p_float.getCPtr(x), n);\n  }\n\n  public void compute_code_from_distance_table(SWIGTYPE_p_float tab, SWIGTYPE_p_unsigned_char code) {\n    swigfaissJNI.ProductQuantizer_compute_code_from_distance_table(swigCPtr, this, SWIGTYPE_p_float.getCPtr(tab), SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void compute_distance_table(SWIGTYPE_p_float x, SWIGTYPE_p_float dis_table) {\n    swigfaissJNI.ProductQuantizer_compute_distance_table(swigCPtr, this, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(dis_table));\n  }\n\n  public void compute_inner_prod_table(SWIGTYPE_p_float x, SWIGTYPE_p_float dis_table) {\n    swigfaissJNI.ProductQuantizer_compute_inner_prod_table(swigCPtr, this, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(dis_table));\n  }\n\n  public void compute_distance_tables(long nx, SWIGTYPE_p_float x, SWIGTYPE_p_float dis_tables) {\n    swigfaissJNI.ProductQuantizer_compute_distance_tables(swigCPtr, this, nx, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(dis_tables));\n  }\n\n  public void compute_inner_prod_tables(long nx, SWIGTYPE_p_float x, SWIGTYPE_p_float dis_tables) {\n    swigfaissJNI.ProductQuantizer_compute_inner_prod_tables(swigCPtr, this, nx, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(dis_tables));\n  }\n\n  public void search(SWIGTYPE_p_float x, long nx, SWIGTYPE_p_unsigned_char codes, long ncodes, SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t res, boolean init_finalize_heap) {\n    swigfaissJNI.ProductQuantizer_search__SWIG_0(swigCPtr, this, SWIGTYPE_p_float.getCPtr(x), nx, SWIGTYPE_p_unsigned_char.getCPtr(codes), ncodes, SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t.getCPtr(res), init_finalize_heap);\n  }\n\n  public void search(SWIGTYPE_p_float x, long nx, SWIGTYPE_p_unsigned_char codes, long ncodes, SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t res) {\n    swigfaissJNI.ProductQuantizer_search__SWIG_1(swigCPtr, this, SWIGTYPE_p_float.getCPtr(x), nx, SWIGTYPE_p_unsigned_char.getCPtr(codes), ncodes, SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t.getCPtr(res));\n  }\n\n  public void search_ip(SWIGTYPE_p_float x, long nx, SWIGTYPE_p_unsigned_char codes, long ncodes, SWIGTYPE_p_faiss__HeapArrayT_faiss__CMinT_float_int64_t_t_t res, boolean init_finalize_heap) {\n    swigfaissJNI.ProductQuantizer_search_ip__SWIG_0(swigCPtr, this, SWIGTYPE_p_float.getCPtr(x), nx, SWIGTYPE_p_unsigned_char.getCPtr(codes), ncodes, SWIGTYPE_p_faiss__HeapArrayT_faiss__CMinT_float_int64_t_t_t.getCPtr(res), init_finalize_heap);\n  }\n\n  public void search_ip(SWIGTYPE_p_float x, long nx, SWIGTYPE_p_unsigned_char codes, long ncodes, SWIGTYPE_p_faiss__HeapArrayT_faiss__CMinT_float_int64_t_t_t res) {\n    swigfaissJNI.ProductQuantizer_search_ip__SWIG_1(swigCPtr, this, SWIGTYPE_p_float.getCPtr(x), nx, SWIGTYPE_p_unsigned_char.getCPtr(codes), ncodes, SWIGTYPE_p_faiss__HeapArrayT_faiss__CMinT_float_int64_t_t_t.getCPtr(res));\n  }\n\n  public void setSdc_table(FloatVector value) {\n    swigfaissJNI.ProductQuantizer_sdc_table_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getSdc_table() {\n    long cPtr = swigfaissJNI.ProductQuantizer_sdc_table_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public void compute_sdc_table() {\n    swigfaissJNI.ProductQuantizer_compute_sdc_table(swigCPtr, this);\n  }\n\n  public void search_sdc(SWIGTYPE_p_unsigned_char qcodes, long nq, SWIGTYPE_p_unsigned_char bcodes, long ncodes, SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t res, boolean init_finalize_heap) {\n    swigfaissJNI.ProductQuantizer_search_sdc__SWIG_0(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(qcodes), nq, SWIGTYPE_p_unsigned_char.getCPtr(bcodes), ncodes, SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t.getCPtr(res), init_finalize_heap);\n  }\n\n  public void search_sdc(SWIGTYPE_p_unsigned_char qcodes, long nq, SWIGTYPE_p_unsigned_char bcodes, long ncodes, SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t res) {\n    swigfaissJNI.ProductQuantizer_search_sdc__SWIG_1(swigCPtr, this, SWIGTYPE_p_unsigned_char.getCPtr(qcodes), nq, SWIGTYPE_p_unsigned_char.getCPtr(bcodes), ncodes, SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t.getCPtr(res));\n  }\n\n  public final static class train_type_t {\n    public final static ProductQuantizer.train_type_t Train_default = new ProductQuantizer.train_type_t(\"Train_default\");\n    public final static ProductQuantizer.train_type_t Train_hot_start = new ProductQuantizer.train_type_t(\"Train_hot_start\");\n    public final static ProductQuantizer.train_type_t Train_shared = new ProductQuantizer.train_type_t(\"Train_shared\");\n    public final static ProductQuantizer.train_type_t Train_hypercube = new ProductQuantizer.train_type_t(\"Train_hypercube\");\n    public final static ProductQuantizer.train_type_t Train_hypercube_pca = new ProductQuantizer.train_type_t(\"Train_hypercube_pca\");\n\n    public final int swigValue() {\n      return swigValue;\n    }\n\n    public String toString() {\n      return swigName;\n    }\n\n    public static train_type_t swigToEnum(int swigValue) {\n      if (swigValue < swigValues.length && swigValue >= 0 && swigValues[swigValue].swigValue == swigValue)\n        return swigValues[swigValue];\n      for (int i = 0; i < swigValues.length; i++)\n        if (swigValues[i].swigValue == swigValue)\n          return swigValues[i];\n      throw new IllegalArgumentException(\"No enum \" + train_type_t.class + \" with value \" + swigValue);\n    }\n\n    private train_type_t(String swigName) {\n      this.swigName = swigName;\n      this.swigValue = swigNext++;\n    }\n\n    private train_type_t(String swigName, int swigValue) {\n      this.swigName = swigName;\n      this.swigValue = swigValue;\n      swigNext = swigValue+1;\n    }\n\n    private train_type_t(String swigName, train_type_t swigEnum) {\n      this.swigName = swigName;\n      this.swigValue = swigEnum.swigValue;\n      swigNext = this.swigValue+1;\n    }\n\n    private static train_type_t[] swigValues = { Train_default, Train_hot_start, Train_shared, Train_hypercube, Train_hypercube_pca };\n    private static int swigNext = 0;\n    private final int swigValue;\n    private final String swigName;\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ProgressiveDimClustering.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ProgressiveDimClustering extends ProgressiveDimClusteringParameters {\n  private transient long swigCPtr;\n\n  protected ProgressiveDimClustering(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.ProgressiveDimClustering_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ProgressiveDimClustering obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ProgressiveDimClustering(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setD(long value) {\n    swigfaissJNI.ProgressiveDimClustering_d_set(swigCPtr, this, value);\n  }\n\n  public long getD() {\n    return swigfaissJNI.ProgressiveDimClustering_d_get(swigCPtr, this);\n  }\n\n  public void setK(long value) {\n    swigfaissJNI.ProgressiveDimClustering_k_set(swigCPtr, this, value);\n  }\n\n  public long getK() {\n    return swigfaissJNI.ProgressiveDimClustering_k_get(swigCPtr, this);\n  }\n\n  public void setCentroids(FloatVector value) {\n    swigfaissJNI.ProgressiveDimClustering_centroids_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getCentroids() {\n    long cPtr = swigfaissJNI.ProgressiveDimClustering_centroids_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public void setIteration_stats(SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t value) {\n    swigfaissJNI.ProgressiveDimClustering_iteration_stats_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t getIteration_stats() {\n    long cPtr = swigfaissJNI.ProgressiveDimClustering_iteration_stats_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t(cPtr, false);\n  }\n\n  public ProgressiveDimClustering(int d, int k) {\n    this(swigfaissJNI.new_ProgressiveDimClustering__SWIG_0(d, k), true);\n  }\n\n  public ProgressiveDimClustering(int d, int k, ProgressiveDimClusteringParameters cp) {\n    this(swigfaissJNI.new_ProgressiveDimClustering__SWIG_1(d, k, ProgressiveDimClusteringParameters.getCPtr(cp), cp), true);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x, ProgressiveDimIndexFactory factory) {\n    swigfaissJNI.ProgressiveDimClustering_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), ProgressiveDimIndexFactory.getCPtr(factory), factory);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ProgressiveDimClusteringParameters.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ProgressiveDimClusteringParameters extends ClusteringParameters {\n  private transient long swigCPtr;\n\n  protected ProgressiveDimClusteringParameters(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.ProgressiveDimClusteringParameters_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ProgressiveDimClusteringParameters obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ProgressiveDimClusteringParameters(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setProgressive_dim_steps(int value) {\n    swigfaissJNI.ProgressiveDimClusteringParameters_progressive_dim_steps_set(swigCPtr, this, value);\n  }\n\n  public int getProgressive_dim_steps() {\n    return swigfaissJNI.ProgressiveDimClusteringParameters_progressive_dim_steps_get(swigCPtr, this);\n  }\n\n  public void setApply_pca(boolean value) {\n    swigfaissJNI.ProgressiveDimClusteringParameters_apply_pca_set(swigCPtr, this, value);\n  }\n\n  public boolean getApply_pca() {\n    return swigfaissJNI.ProgressiveDimClusteringParameters_apply_pca_get(swigCPtr, this);\n  }\n\n  public ProgressiveDimClusteringParameters() {\n    this(swigfaissJNI.new_ProgressiveDimClusteringParameters(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ProgressiveDimIndexFactory.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ProgressiveDimIndexFactory {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected ProgressiveDimIndexFactory(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ProgressiveDimIndexFactory obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ProgressiveDimIndexFactory(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public ProgressiveDimIndexFactory() {\n    this(swigfaissJNI.new_ProgressiveDimIndexFactory(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/RandomRotationMatrix.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class RandomRotationMatrix extends LinearTransform {\n  private transient long swigCPtr;\n\n  protected RandomRotationMatrix(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.RandomRotationMatrix_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(RandomRotationMatrix obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_RandomRotationMatrix(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public RandomRotationMatrix(int d_in, int d_out) {\n    this(swigfaissJNI.new_RandomRotationMatrix__SWIG_0(d_in, d_out), true);\n  }\n\n  public void init(int seed) {\n    swigfaissJNI.RandomRotationMatrix_init(swigCPtr, this, seed);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.RandomRotationMatrix_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public RandomRotationMatrix() {\n    this(swigfaissJNI.new_RandomRotationMatrix__SWIG_1(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/RangeQueryResult.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class RangeQueryResult {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected RangeQueryResult(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(RangeQueryResult obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_RangeQueryResult(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setQno(long value) {\n    swigfaissJNI.RangeQueryResult_qno_set(swigCPtr, this, value);\n  }\n\n  public long getQno() {\n    return swigfaissJNI.RangeQueryResult_qno_get(swigCPtr, this);\n}\n\n  public void setNres(long value) {\n    swigfaissJNI.RangeQueryResult_nres_set(swigCPtr, this, value);\n  }\n\n  public long getNres() {\n    return swigfaissJNI.RangeQueryResult_nres_get(swigCPtr, this);\n  }\n\n  public void setPres(RangeSearchPartialResult value) {\n    swigfaissJNI.RangeQueryResult_pres_set(swigCPtr, this, RangeSearchPartialResult.getCPtr(value), value);\n  }\n\n  public RangeSearchPartialResult getPres() {\n    long cPtr = swigfaissJNI.RangeQueryResult_pres_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new RangeSearchPartialResult(cPtr, false);\n  }\n\n  public void add(float dis, long id) {\n    swigfaissJNI.RangeQueryResult_add(swigCPtr, this, dis, id);\n  }\n\n  public RangeQueryResult() {\n    this(swigfaissJNI.new_RangeQueryResult(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/RangeSearchPartialResult.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class RangeSearchPartialResult extends BufferList {\n  private transient long swigCPtr;\n\n  protected RangeSearchPartialResult(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.RangeSearchPartialResult_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(RangeSearchPartialResult obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_RangeSearchPartialResult(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setRes(RangeSearchResult value) {\n    swigfaissJNI.RangeSearchPartialResult_res_set(swigCPtr, this, RangeSearchResult.getCPtr(value), value);\n  }\n\n  public RangeSearchResult getRes() {\n    long cPtr = swigfaissJNI.RangeSearchPartialResult_res_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new RangeSearchResult(cPtr, false);\n  }\n\n  public void setQueries(SWIGTYPE_p_std__vectorT_faiss__RangeQueryResult_t value) {\n    swigfaissJNI.RangeSearchPartialResult_queries_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_faiss__RangeQueryResult_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_faiss__RangeQueryResult_t getQueries() {\n    long cPtr = swigfaissJNI.RangeSearchPartialResult_queries_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_faiss__RangeQueryResult_t(cPtr, false);\n  }\n\n  public RangeQueryResult new_result(long qno) {\n    return new RangeQueryResult(swigfaissJNI.RangeSearchPartialResult_new_result(swigCPtr, this, qno), false);\n  }\n\n  public void set_lims() {\n    swigfaissJNI.RangeSearchPartialResult_set_lims(swigCPtr, this);\n  }\n\n  public void copy_result(boolean incremental) {\n    swigfaissJNI.RangeSearchPartialResult_copy_result__SWIG_0(swigCPtr, this, incremental);\n  }\n\n  public void copy_result() {\n    swigfaissJNI.RangeSearchPartialResult_copy_result__SWIG_1(swigCPtr, this);\n  }\n\n  public static void merge(SWIGTYPE_p_std__vectorT_faiss__RangeSearchPartialResult_p_t partial_results, boolean do_delete) {\n    swigfaissJNI.RangeSearchPartialResult_merge__SWIG_0(SWIGTYPE_p_std__vectorT_faiss__RangeSearchPartialResult_p_t.getCPtr(partial_results), do_delete);\n  }\n\n  public static void merge(SWIGTYPE_p_std__vectorT_faiss__RangeSearchPartialResult_p_t partial_results) {\n    swigfaissJNI.RangeSearchPartialResult_merge__SWIG_1(SWIGTYPE_p_std__vectorT_faiss__RangeSearchPartialResult_p_t.getCPtr(partial_results));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/RangeSearchResult.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class RangeSearchResult {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected RangeSearchResult(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(RangeSearchResult obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_RangeSearchResult(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setNq(long value) {\n    swigfaissJNI.RangeSearchResult_nq_set(swigCPtr, this, value);\n  }\n\n  public long getNq() {\n    return swigfaissJNI.RangeSearchResult_nq_get(swigCPtr, this);\n  }\n\n  public void setLims(SWIGTYPE_p_unsigned_long value) {\n    swigfaissJNI.RangeSearchResult_lims_set(swigCPtr, this, SWIGTYPE_p_unsigned_long.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_unsigned_long getLims() {\n    long cPtr = swigfaissJNI.RangeSearchResult_lims_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_long(cPtr, false);\n  }\n\n  public void setLabels(LongVector value) {\n    swigfaissJNI.RangeSearchResult_labels_set(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(value.data()), value);\n  }\n\n  public LongVector getLabels() {\n    return new LongVector(swigfaissJNI.RangeSearchResult_labels_get(swigCPtr, this), false);\n}\n\n  public void setDistances(SWIGTYPE_p_float value) {\n    swigfaissJNI.RangeSearchResult_distances_set(swigCPtr, this, SWIGTYPE_p_float.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_float getDistances() {\n    long cPtr = swigfaissJNI.RangeSearchResult_distances_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public void setBuffer_size(long value) {\n    swigfaissJNI.RangeSearchResult_buffer_size_set(swigCPtr, this, value);\n  }\n\n  public long getBuffer_size() {\n    return swigfaissJNI.RangeSearchResult_buffer_size_get(swigCPtr, this);\n  }\n\n  public void do_allocation() {\n    swigfaissJNI.RangeSearchResult_do_allocation(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ReadOnlyInvertedLists.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ReadOnlyInvertedLists extends InvertedLists {\n  private transient long swigCPtr;\n\n  protected ReadOnlyInvertedLists(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.ReadOnlyInvertedLists_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ReadOnlyInvertedLists obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ReadOnlyInvertedLists(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public long add_entries(long list_no, long n_entry, LongVector ids, SWIGTYPE_p_unsigned_char code) {\n    return swigfaissJNI.ReadOnlyInvertedLists_add_entries(swigCPtr, this, list_no, n_entry, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void update_entries(long list_no, long offset, long n_entry, LongVector ids, SWIGTYPE_p_unsigned_char code) {\n    swigfaissJNI.ReadOnlyInvertedLists_update_entries(swigCPtr, this, list_no, offset, n_entry, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void resize(long list_no, long new_size) {\n    swigfaissJNI.ReadOnlyInvertedLists_resize(swigCPtr, this, list_no, new_size);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ReconstructFromNeighbors.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ReconstructFromNeighbors {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected ReconstructFromNeighbors(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ReconstructFromNeighbors obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ReconstructFromNeighbors(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public IndexHNSW getIndex() {\n    return new IndexHNSW(swigfaissJNI.ReconstructFromNeighbors_index_get(swigCPtr, this), false);\n  }\n\n  public void setM(long value) {\n    swigfaissJNI.ReconstructFromNeighbors_M_set(swigCPtr, this, value);\n  }\n\n  public long getM() {\n    return swigfaissJNI.ReconstructFromNeighbors_M_get(swigCPtr, this);\n  }\n\n  public void setK(long value) {\n    swigfaissJNI.ReconstructFromNeighbors_k_set(swigCPtr, this, value);\n  }\n\n  public long getK() {\n    return swigfaissJNI.ReconstructFromNeighbors_k_get(swigCPtr, this);\n  }\n\n  public void setNsq(long value) {\n    swigfaissJNI.ReconstructFromNeighbors_nsq_set(swigCPtr, this, value);\n  }\n\n  public long getNsq() {\n    return swigfaissJNI.ReconstructFromNeighbors_nsq_get(swigCPtr, this);\n  }\n\n  public void setCode_size(long value) {\n    swigfaissJNI.ReconstructFromNeighbors_code_size_set(swigCPtr, this, value);\n  }\n\n  public long getCode_size() {\n    return swigfaissJNI.ReconstructFromNeighbors_code_size_get(swigCPtr, this);\n  }\n\n  public void setK_reorder(int value) {\n    swigfaissJNI.ReconstructFromNeighbors_k_reorder_set(swigCPtr, this, value);\n  }\n\n  public int getK_reorder() {\n    return swigfaissJNI.ReconstructFromNeighbors_k_reorder_get(swigCPtr, this);\n  }\n\n  public void setCodebook(FloatVector value) {\n    swigfaissJNI.ReconstructFromNeighbors_codebook_set(swigCPtr, this, FloatVector.getCPtr(value), value);\n  }\n\n  public FloatVector getCodebook() {\n    long cPtr = swigfaissJNI.ReconstructFromNeighbors_codebook_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new FloatVector(cPtr, false);\n  }\n\n  public void setCodes(ByteVector value) {\n    swigfaissJNI.ReconstructFromNeighbors_codes_set(swigCPtr, this, ByteVector.getCPtr(value), value);\n  }\n\n  public ByteVector getCodes() {\n    long cPtr = swigfaissJNI.ReconstructFromNeighbors_codes_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ByteVector(cPtr, false);\n  }\n\n  public void setNtotal(long value) {\n    swigfaissJNI.ReconstructFromNeighbors_ntotal_set(swigCPtr, this, value);\n  }\n\n  public long getNtotal() {\n    return swigfaissJNI.ReconstructFromNeighbors_ntotal_get(swigCPtr, this);\n  }\n\n  public void setD(long value) {\n    swigfaissJNI.ReconstructFromNeighbors_d_set(swigCPtr, this, value);\n  }\n\n  public long getD() {\n    return swigfaissJNI.ReconstructFromNeighbors_d_get(swigCPtr, this);\n  }\n\n  public void setDsub(long value) {\n    swigfaissJNI.ReconstructFromNeighbors_dsub_set(swigCPtr, this, value);\n  }\n\n  public long getDsub() {\n    return swigfaissJNI.ReconstructFromNeighbors_dsub_get(swigCPtr, this);\n  }\n\n  public ReconstructFromNeighbors(IndexHNSW index, long k, long nsq) {\n    this(swigfaissJNI.new_ReconstructFromNeighbors__SWIG_0(IndexHNSW.getCPtr(index), index, k, nsq), true);\n  }\n\n  public ReconstructFromNeighbors(IndexHNSW index, long k) {\n    this(swigfaissJNI.new_ReconstructFromNeighbors__SWIG_1(IndexHNSW.getCPtr(index), index, k), true);\n  }\n\n  public ReconstructFromNeighbors(IndexHNSW index) {\n    this(swigfaissJNI.new_ReconstructFromNeighbors__SWIG_2(IndexHNSW.getCPtr(index), index), true);\n  }\n\n  public void add_codes(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.ReconstructFromNeighbors_add_codes(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public long compute_distances(long n, LongVector shortlist, SWIGTYPE_p_float query, SWIGTYPE_p_float distances) {\n    return swigfaissJNI.ReconstructFromNeighbors_compute_distances(swigCPtr, this, n, SWIGTYPE_p_long_long.getCPtr(shortlist.data()), shortlist, SWIGTYPE_p_float.getCPtr(query), SWIGTYPE_p_float.getCPtr(distances));\n  }\n\n  public void estimate_code(SWIGTYPE_p_float x, int i, SWIGTYPE_p_unsigned_char code) {\n    swigfaissJNI.ReconstructFromNeighbors_estimate_code(swigCPtr, this, SWIGTYPE_p_float.getCPtr(x), i, SWIGTYPE_p_unsigned_char.getCPtr(code));\n  }\n\n  public void reconstruct(int i, SWIGTYPE_p_float x, SWIGTYPE_p_float tmp) {\n    swigfaissJNI.ReconstructFromNeighbors_reconstruct(swigCPtr, this, i, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(tmp));\n  }\n\n  public void reconstruct_n(int n0, int ni, SWIGTYPE_p_float x) {\n    swigfaissJNI.ReconstructFromNeighbors_reconstruct_n(swigCPtr, this, n0, ni, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public void get_neighbor_table(int i, SWIGTYPE_p_float out) {\n    swigfaissJNI.ReconstructFromNeighbors_get_neighbor_table(swigCPtr, this, i, SWIGTYPE_p_float.getCPtr(out));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/RemapDimensionsTransform.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class RemapDimensionsTransform extends VectorTransform {\n  private transient long swigCPtr;\n\n  protected RemapDimensionsTransform(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.RemapDimensionsTransform_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(RemapDimensionsTransform obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_RemapDimensionsTransform(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setMap(IntVector value) {\n    swigfaissJNI.RemapDimensionsTransform_map_set(swigCPtr, this, IntVector.getCPtr(value), value);\n  }\n\n  public IntVector getMap() {\n    long cPtr = swigfaissJNI.RemapDimensionsTransform_map_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new IntVector(cPtr, false);\n  }\n\n  public RemapDimensionsTransform(int d_in, int d_out, SWIGTYPE_p_int map) {\n    this(swigfaissJNI.new_RemapDimensionsTransform__SWIG_0(d_in, d_out, SWIGTYPE_p_int.getCPtr(map)), true);\n  }\n\n  public RemapDimensionsTransform(int d_in, int d_out, boolean uniform) {\n    this(swigfaissJNI.new_RemapDimensionsTransform__SWIG_1(d_in, d_out, uniform), true);\n  }\n\n  public RemapDimensionsTransform(int d_in, int d_out) {\n    this(swigfaissJNI.new_RemapDimensionsTransform__SWIG_2(d_in, d_out), true);\n  }\n\n  public void apply_noalloc(long n, SWIGTYPE_p_float x, SWIGTYPE_p_float xt) {\n    swigfaissJNI.RemapDimensionsTransform_apply_noalloc(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(xt));\n  }\n\n  public void reverse_transform(long n, SWIGTYPE_p_float xt, SWIGTYPE_p_float x) {\n    swigfaissJNI.RemapDimensionsTransform_reverse_transform(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(xt), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public RemapDimensionsTransform() {\n    this(swigfaissJNI.new_RemapDimensionsTransform__SWIG_3(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/ReproduceDistancesObjective.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class ReproduceDistancesObjective extends PermutationObjective {\n  private transient long swigCPtr;\n\n  protected ReproduceDistancesObjective(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.ReproduceDistancesObjective_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(ReproduceDistancesObjective obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_ReproduceDistancesObjective(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setDis_weight_factor(double value) {\n    swigfaissJNI.ReproduceDistancesObjective_dis_weight_factor_set(swigCPtr, this, value);\n  }\n\n  public double getDis_weight_factor() {\n    return swigfaissJNI.ReproduceDistancesObjective_dis_weight_factor_get(swigCPtr, this);\n  }\n\n  public static double sqr(double x) {\n    return swigfaissJNI.ReproduceDistancesObjective_sqr(x);\n  }\n\n  public double dis_weight(double x) {\n    return swigfaissJNI.ReproduceDistancesObjective_dis_weight(swigCPtr, this, x);\n  }\n\n  public void setSource_dis(DoubleVector value) {\n    swigfaissJNI.ReproduceDistancesObjective_source_dis_set(swigCPtr, this, DoubleVector.getCPtr(value), value);\n  }\n\n  public DoubleVector getSource_dis() {\n    long cPtr = swigfaissJNI.ReproduceDistancesObjective_source_dis_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new DoubleVector(cPtr, false);\n  }\n\n  public void setTarget_dis(SWIGTYPE_p_double value) {\n    swigfaissJNI.ReproduceDistancesObjective_target_dis_set(swigCPtr, this, SWIGTYPE_p_double.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_double getTarget_dis() {\n    long cPtr = swigfaissJNI.ReproduceDistancesObjective_target_dis_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_double(cPtr, false);\n  }\n\n  public void setWeights(DoubleVector value) {\n    swigfaissJNI.ReproduceDistancesObjective_weights_set(swigCPtr, this, DoubleVector.getCPtr(value), value);\n  }\n\n  public DoubleVector getWeights() {\n    long cPtr = swigfaissJNI.ReproduceDistancesObjective_weights_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new DoubleVector(cPtr, false);\n  }\n\n  public double get_source_dis(int i, int j) {\n    return swigfaissJNI.ReproduceDistancesObjective_get_source_dis(swigCPtr, this, i, j);\n  }\n\n  public double compute_cost(SWIGTYPE_p_int perm) {\n    return swigfaissJNI.ReproduceDistancesObjective_compute_cost(swigCPtr, this, SWIGTYPE_p_int.getCPtr(perm));\n  }\n\n  public double cost_update(SWIGTYPE_p_int perm, int iw, int jw) {\n    return swigfaissJNI.ReproduceDistancesObjective_cost_update(swigCPtr, this, SWIGTYPE_p_int.getCPtr(perm), iw, jw);\n  }\n\n  public ReproduceDistancesObjective(int n, SWIGTYPE_p_double source_dis_in, SWIGTYPE_p_double target_dis_in, double dis_weight_factor) {\n    this(swigfaissJNI.new_ReproduceDistancesObjective(n, SWIGTYPE_p_double.getCPtr(source_dis_in), SWIGTYPE_p_double.getCPtr(target_dis_in), dis_weight_factor), true);\n  }\n\n  public static void compute_mean_stdev(SWIGTYPE_p_double tab, long n2, SWIGTYPE_p_double mean_out, SWIGTYPE_p_double stddev_out) {\n    swigfaissJNI.ReproduceDistancesObjective_compute_mean_stdev(SWIGTYPE_p_double.getCPtr(tab), n2, SWIGTYPE_p_double.getCPtr(mean_out), SWIGTYPE_p_double.getCPtr(stddev_out));\n  }\n\n  public void set_affine_target_dis(SWIGTYPE_p_double source_dis_in) {\n    swigfaissJNI.ReproduceDistancesObjective_set_affine_target_dis(swigCPtr, this, SWIGTYPE_p_double.getCPtr(source_dis_in));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_AlignedTableT_float_32_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_AlignedTableT_float_32_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_AlignedTableT_float_32_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_AlignedTableT_float_32_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_AlignedTableT_float_32_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_AlignedTableT_float_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_AlignedTableT_float_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_AlignedTableT_float_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_AlignedTableT_float_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_AlignedTableT_float_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_DirectMap.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_DirectMap {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_DirectMap(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_DirectMap() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_DirectMap obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_DirectMap__Type.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_DirectMap__Type {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_DirectMap__Type(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_DirectMap__Type() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_DirectMap__Type obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_FILE.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_FILE {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_FILE(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_FILE() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_FILE obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_IOReader.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_IOReader {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_IOReader(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_IOReader() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_IOReader obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_IOWriter.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_IOWriter {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_IOWriter(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_IOWriter() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_IOWriter obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_ScalarQuantizer.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_ScalarQuantizer {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_ScalarQuantizer(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_ScalarQuantizer() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_ScalarQuantizer obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_ScalarQuantizer__QuantizerType.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_ScalarQuantizer__QuantizerType {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_ScalarQuantizer__QuantizerType(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_ScalarQuantizer__QuantizerType() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_ScalarQuantizer__QuantizerType obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_double.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_double {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_double(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_double() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_double obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__AlignedTableTightAllocT_float_32_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__AlignedTableTightAllocT_float_32_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__AlignedTableTightAllocT_float_32_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__AlignedTableTightAllocT_float_32_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__AlignedTableTightAllocT_float_32_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__AlignedTableTightAllocT_uint16_t_32_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__AlignedTableTightAllocT_uint16_t_32_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__AlignedTableTightAllocT_uint16_t_32_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__AlignedTableTightAllocT_uint16_t_32_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__AlignedTableTightAllocT_uint16_t_32_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__AlignedTableTightAllocT_unsigned_char_32_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__AlignedTableTightAllocT_unsigned_char_32_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__AlignedTableTightAllocT_unsigned_char_32_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__AlignedTableTightAllocT_unsigned_char_32_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__AlignedTableTightAllocT_unsigned_char_32_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__BinaryInvertedListScanner.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__BinaryInvertedListScanner {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__BinaryInvertedListScanner(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__BinaryInvertedListScanner() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__BinaryInvertedListScanner obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_float_int64_t_t_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__HeapArrayT_faiss__CMinT_float_int64_t_t_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__HeapArrayT_faiss__CMinT_float_int64_t_t_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__HeapArrayT_faiss__CMinT_float_int64_t_t_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__HeapArrayT_faiss__CMinT_float_int64_t_t_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__HeapArrayT_faiss__CMinT_float_int64_t_t_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__IOReader.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__IOReader {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__IOReader(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__IOReader() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__IOReader obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__IOWriter.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__IOWriter {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__IOWriter(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__IOWriter() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__IOWriter obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__InvertedListScanner.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__InvertedListScanner {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__InvertedListScanner(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__InvertedListScanner() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__InvertedListScanner obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__LockLevels.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__LockLevels {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__LockLevels(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__LockLevels() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__LockLevels obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__OnDiskInvertedLists__OngoingPrefetch.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__OnDiskInvertedLists__OngoingPrefetch {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__OnDiskInvertedLists__OngoingPrefetch(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__OnDiskInvertedLists__OngoingPrefetch() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__OnDiskInvertedLists__OngoingPrefetch obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_faiss__RandomGenerator.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_faiss__RandomGenerator {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_faiss__RandomGenerator(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_faiss__RandomGenerator() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_faiss__RandomGenerator obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_float.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_float {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_float(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_float() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_float obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_int.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_int {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_int(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_int() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_int obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_long.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_long {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_long(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_long() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_long obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_long_long.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_long_long {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_long_long(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_long_long() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_long_long obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_omp_lock_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_omp_lock_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_omp_lock_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_omp_lock_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_omp_lock_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_p_faiss__Index.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_p_faiss__Index {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_p_faiss__Index(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_p_faiss__Index() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_p_faiss__Index obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_p_faiss__InvertedLists.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_p_faiss__InvertedLists {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_p_faiss__InvertedLists(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_p_faiss__InvertedLists() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_p_faiss__InvertedLists obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_p_faiss__VectorTransform.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_p_faiss__VectorTransform {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_p_faiss__VectorTransform(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_p_faiss__VectorTransform() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_p_faiss__VectorTransform obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__listT_faiss__OnDiskInvertedLists__Slot_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__listT_faiss__OnDiskInvertedLists__Slot_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__listT_faiss__OnDiskInvertedLists__Slot_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__listT_faiss__OnDiskInvertedLists__Slot_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__listT_faiss__OnDiskInvertedLists__Slot_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__pairT_float_int_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__pairT_float_int_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__pairT_float_int_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__pairT_float_int_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__pairT_float_int_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__priority_queueT_faiss__HNSW__NodeDistFarther_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__priority_queueT_faiss__HNSW__NodeDistFarther_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__priority_queueT_faiss__HNSW__NodeDistFarther_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__priority_queueT_faiss__HNSW__NodeDistFarther_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__priority_queueT_faiss__HNSW__NodeDistFarther_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__priority_queueT_std__pairT_float_int_t_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__priority_queueT_std__pairT_float_int_t_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__priority_queueT_std__pairT_float_int_t_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__priority_queueT_std__pairT_float_int_t_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__priority_queueT_std__pairT_float_int_t_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__unordered_mapT_long_long_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__unordered_mapT_long_long_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__unordered_mapT_long_long_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__unordered_mapT_long_long_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__unordered_mapT_long_long_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__unordered_multimapT_int64_t_int64_t_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__unordered_multimapT_int64_t_int64_t_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__unordered_multimapT_int64_t_int64_t_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__unordered_multimapT_int64_t_int64_t_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__unordered_multimapT_int64_t_int64_t_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_faiss__BufferList__Buffer_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_faiss__BufferList__Buffer_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_faiss__BufferList__Buffer_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_faiss__BufferList__Buffer_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_faiss__BufferList__Buffer_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_faiss__ClusteringIterationStats_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_faiss__HNSW__NodeDistFarther_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_faiss__HNSW__NodeDistFarther_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_faiss__HNSW__NodeDistFarther_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_faiss__HNSW__NodeDistFarther_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_faiss__HNSW__NodeDistFarther_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_faiss__Index_p_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_faiss__Index_p_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_faiss__Index_p_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_faiss__Index_p_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_faiss__Index_p_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_faiss__OnDiskOneList_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_faiss__OnDiskOneList_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_faiss__OnDiskOneList_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_faiss__OnDiskOneList_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_faiss__OnDiskOneList_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_faiss__ParameterRange_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_faiss__ParameterRange_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_faiss__ParameterRange_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_faiss__ParameterRange_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_faiss__ParameterRange_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_faiss__RangeQueryResult_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_faiss__RangeQueryResult_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_faiss__RangeQueryResult_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_faiss__RangeQueryResult_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_faiss__RangeQueryResult_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_faiss__RangeSearchPartialResult_p_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_faiss__RangeSearchPartialResult_p_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_faiss__RangeSearchPartialResult_p_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_faiss__RangeSearchPartialResult_p_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_faiss__RangeSearchPartialResult_p_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_int64_t_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_int64_t_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_int64_t_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_int64_t_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_int64_t_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_long_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_long_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_long_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_long_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_long_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_omp_lock_t_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_omp_lock_t_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_omp_lock_t_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_omp_lock_t_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_omp_lock_t_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_std__vectorT_int64_t_t_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_std__vectorT_int64_t_t_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_std__vectorT_int64_t_t_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_std__vectorT_int64_t_t_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_std__vectorT_int64_t_t_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_std__vectorT_std__vectorT_unsigned_long_t_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_std__vectorT_std__vectorT_unsigned_long_t_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_std__vectorT_std__vectorT_unsigned_long_t_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_std__vectorT_std__vectorT_unsigned_long_t_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_std__vectorT_std__vectorT_unsigned_long_t_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_uint16_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_uint16_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_uint16_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_uint16_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_uint16_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_uint32_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_uint32_t {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_uint32_t(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_uint32_t() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_uint32_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_unsigned_char.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_unsigned_char {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_unsigned_char(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_unsigned_char() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_unsigned_char obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_unsigned_long.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_unsigned_long {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_unsigned_long(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_unsigned_long() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_unsigned_long obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SWIGTYPE_p_void.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SWIGTYPE_p_void {\n  private transient long swigCPtr;\n\n  protected SWIGTYPE_p_void(long cPtr, @SuppressWarnings(\"unused\") boolean futureUse) {\n    swigCPtr = cPtr;\n  }\n\n  protected SWIGTYPE_p_void() {\n    swigCPtr = 0;\n  }\n\n  protected static long getCPtr(SWIGTYPE_p_void obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n}\n\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SimulatedAnnealingOptimizer.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SimulatedAnnealingOptimizer extends SimulatedAnnealingParameters {\n  private transient long swigCPtr;\n\n  protected SimulatedAnnealingOptimizer(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.SimulatedAnnealingOptimizer_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(SimulatedAnnealingOptimizer obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_SimulatedAnnealingOptimizer(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setObj(PermutationObjective value) {\n    swigfaissJNI.SimulatedAnnealingOptimizer_obj_set(swigCPtr, this, PermutationObjective.getCPtr(value), value);\n  }\n\n  public PermutationObjective getObj() {\n    long cPtr = swigfaissJNI.SimulatedAnnealingOptimizer_obj_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new PermutationObjective(cPtr, false);\n  }\n\n  public void setN(int value) {\n    swigfaissJNI.SimulatedAnnealingOptimizer_n_set(swigCPtr, this, value);\n  }\n\n  public int getN() {\n    return swigfaissJNI.SimulatedAnnealingOptimizer_n_get(swigCPtr, this);\n  }\n\n  public void setLogfile(SWIGTYPE_p_FILE value) {\n    swigfaissJNI.SimulatedAnnealingOptimizer_logfile_set(swigCPtr, this, SWIGTYPE_p_FILE.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_FILE getLogfile() {\n    long cPtr = swigfaissJNI.SimulatedAnnealingOptimizer_logfile_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_FILE(cPtr, false);\n  }\n\n  public SimulatedAnnealingOptimizer(PermutationObjective obj, SimulatedAnnealingParameters p) {\n    this(swigfaissJNI.new_SimulatedAnnealingOptimizer(PermutationObjective.getCPtr(obj), obj, SimulatedAnnealingParameters.getCPtr(p), p), true);\n  }\n\n  public void setRnd(SWIGTYPE_p_faiss__RandomGenerator value) {\n    swigfaissJNI.SimulatedAnnealingOptimizer_rnd_set(swigCPtr, this, SWIGTYPE_p_faiss__RandomGenerator.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_faiss__RandomGenerator getRnd() {\n    long cPtr = swigfaissJNI.SimulatedAnnealingOptimizer_rnd_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_faiss__RandomGenerator(cPtr, false);\n  }\n\n  public void setInit_cost(double value) {\n    swigfaissJNI.SimulatedAnnealingOptimizer_init_cost_set(swigCPtr, this, value);\n  }\n\n  public double getInit_cost() {\n    return swigfaissJNI.SimulatedAnnealingOptimizer_init_cost_get(swigCPtr, this);\n  }\n\n  public double optimize(SWIGTYPE_p_int perm) {\n    return swigfaissJNI.SimulatedAnnealingOptimizer_optimize(swigCPtr, this, SWIGTYPE_p_int.getCPtr(perm));\n  }\n\n  public double run_optimization(SWIGTYPE_p_int best_perm) {\n    return swigfaissJNI.SimulatedAnnealingOptimizer_run_optimization(swigCPtr, this, SWIGTYPE_p_int.getCPtr(best_perm));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SimulatedAnnealingParameters.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SimulatedAnnealingParameters {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected SimulatedAnnealingParameters(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(SimulatedAnnealingParameters obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_SimulatedAnnealingParameters(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setInit_temperature(double value) {\n    swigfaissJNI.SimulatedAnnealingParameters_init_temperature_set(swigCPtr, this, value);\n  }\n\n  public double getInit_temperature() {\n    return swigfaissJNI.SimulatedAnnealingParameters_init_temperature_get(swigCPtr, this);\n  }\n\n  public void setTemperature_decay(double value) {\n    swigfaissJNI.SimulatedAnnealingParameters_temperature_decay_set(swigCPtr, this, value);\n  }\n\n  public double getTemperature_decay() {\n    return swigfaissJNI.SimulatedAnnealingParameters_temperature_decay_get(swigCPtr, this);\n  }\n\n  public void setN_iter(int value) {\n    swigfaissJNI.SimulatedAnnealingParameters_n_iter_set(swigCPtr, this, value);\n  }\n\n  public int getN_iter() {\n    return swigfaissJNI.SimulatedAnnealingParameters_n_iter_get(swigCPtr, this);\n  }\n\n  public void setN_redo(int value) {\n    swigfaissJNI.SimulatedAnnealingParameters_n_redo_set(swigCPtr, this, value);\n  }\n\n  public int getN_redo() {\n    return swigfaissJNI.SimulatedAnnealingParameters_n_redo_get(swigCPtr, this);\n  }\n\n  public void setSeed(int value) {\n    swigfaissJNI.SimulatedAnnealingParameters_seed_set(swigCPtr, this, value);\n  }\n\n  public int getSeed() {\n    return swigfaissJNI.SimulatedAnnealingParameters_seed_get(swigCPtr, this);\n  }\n\n  public void setVerbose(int value) {\n    swigfaissJNI.SimulatedAnnealingParameters_verbose_set(swigCPtr, this, value);\n  }\n\n  public int getVerbose() {\n    return swigfaissJNI.SimulatedAnnealingParameters_verbose_get(swigCPtr, this);\n  }\n\n  public void setOnly_bit_flips(boolean value) {\n    swigfaissJNI.SimulatedAnnealingParameters_only_bit_flips_set(swigCPtr, this, value);\n  }\n\n  public boolean getOnly_bit_flips() {\n    return swigfaissJNI.SimulatedAnnealingParameters_only_bit_flips_get(swigCPtr, this);\n  }\n\n  public void setInit_random(boolean value) {\n    swigfaissJNI.SimulatedAnnealingParameters_init_random_set(swigCPtr, this, value);\n  }\n\n  public boolean getInit_random() {\n    return swigfaissJNI.SimulatedAnnealingParameters_init_random_get(swigCPtr, this);\n  }\n\n  public SimulatedAnnealingParameters() {\n    this(swigfaissJNI.new_SimulatedAnnealingParameters(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SliceInvertedLists.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SliceInvertedLists extends ReadOnlyInvertedLists {\n  private transient long swigCPtr;\n\n  protected SliceInvertedLists(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.SliceInvertedLists_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(SliceInvertedLists obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_SliceInvertedLists(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setIl(InvertedLists value) {\n    swigfaissJNI.SliceInvertedLists_il_set(swigCPtr, this, InvertedLists.getCPtr(value), value);\n  }\n\n  public InvertedLists getIl() {\n    long cPtr = swigfaissJNI.SliceInvertedLists_il_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n  }\n\n  public void setI0(long value) {\n    swigfaissJNI.SliceInvertedLists_i0_set(swigCPtr, this, value);\n  }\n\n  public long getI0() {\n    return swigfaissJNI.SliceInvertedLists_i0_get(swigCPtr, this);\n}\n\n  public void setI1(long value) {\n    swigfaissJNI.SliceInvertedLists_i1_set(swigCPtr, this, value);\n  }\n\n  public long getI1() {\n    return swigfaissJNI.SliceInvertedLists_i1_get(swigCPtr, this);\n}\n\n  public SliceInvertedLists(InvertedLists il, long i0, long i1) {\n    this(swigfaissJNI.new_SliceInvertedLists(InvertedLists.getCPtr(il), il, i0, i1), true);\n  }\n\n  public long list_size(long list_no) {\n    return swigfaissJNI.SliceInvertedLists_list_size(swigCPtr, this, list_no);\n  }\n\n  public SWIGTYPE_p_unsigned_char get_codes(long list_no) {\n    long cPtr = swigfaissJNI.SliceInvertedLists_get_codes(swigCPtr, this, list_no);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public LongVector get_ids(long list_no) {\n    return new LongVector(swigfaissJNI.SliceInvertedLists_get_ids(swigCPtr, this, list_no), false);\n}\n\n  public void release_codes(long list_no, SWIGTYPE_p_unsigned_char codes) {\n    swigfaissJNI.SliceInvertedLists_release_codes(swigCPtr, this, list_no, SWIGTYPE_p_unsigned_char.getCPtr(codes));\n  }\n\n  public void release_ids(long list_no, LongVector ids) {\n    swigfaissJNI.SliceInvertedLists_release_ids(swigCPtr, this, list_no, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids);\n  }\n\n  public long get_single_id(long list_no, long offset) {\n    return swigfaissJNI.SliceInvertedLists_get_single_id(swigCPtr, this, list_no, offset);\n}\n\n  public SWIGTYPE_p_unsigned_char get_single_code(long list_no, long offset) {\n    long cPtr = swigfaissJNI.SliceInvertedLists_get_single_code(swigCPtr, this, list_no, offset);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public void prefetch_lists(LongVector list_nos, int nlist) {\n    swigfaissJNI.SliceInvertedLists_prefetch_lists(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, nlist);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/SlidingIndexWindow.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class SlidingIndexWindow {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected SlidingIndexWindow(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(SlidingIndexWindow obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_SlidingIndexWindow(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setIndex(Index value) {\n    swigfaissJNI.SlidingIndexWindow_index_set(swigCPtr, this, Index.getCPtr(value), value);\n  }\n\n  public Index getIndex() {\n    long cPtr = swigfaissJNI.SlidingIndexWindow_index_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new Index(cPtr, false);\n  }\n\n  public void setIls(ArrayInvertedLists value) {\n    swigfaissJNI.SlidingIndexWindow_ils_set(swigCPtr, this, ArrayInvertedLists.getCPtr(value), value);\n  }\n\n  public ArrayInvertedLists getIls() {\n    long cPtr = swigfaissJNI.SlidingIndexWindow_ils_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ArrayInvertedLists(cPtr, false);\n  }\n\n  public void setN_slice(int value) {\n    swigfaissJNI.SlidingIndexWindow_n_slice_set(swigCPtr, this, value);\n  }\n\n  public int getN_slice() {\n    return swigfaissJNI.SlidingIndexWindow_n_slice_get(swigCPtr, this);\n  }\n\n  public void setNlist(long value) {\n    swigfaissJNI.SlidingIndexWindow_nlist_set(swigCPtr, this, value);\n  }\n\n  public long getNlist() {\n    return swigfaissJNI.SlidingIndexWindow_nlist_get(swigCPtr, this);\n  }\n\n  public void setSizes(SWIGTYPE_p_std__vectorT_std__vectorT_unsigned_long_t_t value) {\n    swigfaissJNI.SlidingIndexWindow_sizes_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_std__vectorT_unsigned_long_t_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_std__vectorT_unsigned_long_t_t getSizes() {\n    long cPtr = swigfaissJNI.SlidingIndexWindow_sizes_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_std__vectorT_unsigned_long_t_t(cPtr, false);\n  }\n\n  public SlidingIndexWindow(Index index) {\n    this(swigfaissJNI.new_SlidingIndexWindow(Index.getCPtr(index), index), true);\n  }\n\n  public void step(Index sub_index, boolean remove_oldest) {\n    swigfaissJNI.SlidingIndexWindow_step(swigCPtr, this, Index.getCPtr(sub_index), sub_index, remove_oldest);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/StopWordsInvertedLists.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class StopWordsInvertedLists extends ReadOnlyInvertedLists {\n  private transient long swigCPtr;\n\n  protected StopWordsInvertedLists(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.StopWordsInvertedLists_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(StopWordsInvertedLists obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_StopWordsInvertedLists(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setIl0(InvertedLists value) {\n    swigfaissJNI.StopWordsInvertedLists_il0_set(swigCPtr, this, InvertedLists.getCPtr(value), value);\n  }\n\n  public InvertedLists getIl0() {\n    long cPtr = swigfaissJNI.StopWordsInvertedLists_il0_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n  }\n\n  public void setMaxsize(long value) {\n    swigfaissJNI.StopWordsInvertedLists_maxsize_set(swigCPtr, this, value);\n  }\n\n  public long getMaxsize() {\n    return swigfaissJNI.StopWordsInvertedLists_maxsize_get(swigCPtr, this);\n  }\n\n  public StopWordsInvertedLists(InvertedLists il, long maxsize) {\n    this(swigfaissJNI.new_StopWordsInvertedLists(InvertedLists.getCPtr(il), il, maxsize), true);\n  }\n\n  public long list_size(long list_no) {\n    return swigfaissJNI.StopWordsInvertedLists_list_size(swigCPtr, this, list_no);\n  }\n\n  public SWIGTYPE_p_unsigned_char get_codes(long list_no) {\n    long cPtr = swigfaissJNI.StopWordsInvertedLists_get_codes(swigCPtr, this, list_no);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public LongVector get_ids(long list_no) {\n    return new LongVector(swigfaissJNI.StopWordsInvertedLists_get_ids(swigCPtr, this, list_no), false);\n}\n\n  public void release_codes(long list_no, SWIGTYPE_p_unsigned_char codes) {\n    swigfaissJNI.StopWordsInvertedLists_release_codes(swigCPtr, this, list_no, SWIGTYPE_p_unsigned_char.getCPtr(codes));\n  }\n\n  public void release_ids(long list_no, LongVector ids) {\n    swigfaissJNI.StopWordsInvertedLists_release_ids(swigCPtr, this, list_no, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids);\n  }\n\n  public long get_single_id(long list_no, long offset) {\n    return swigfaissJNI.StopWordsInvertedLists_get_single_id(swigCPtr, this, list_no, offset);\n}\n\n  public SWIGTYPE_p_unsigned_char get_single_code(long list_no, long offset) {\n    long cPtr = swigfaissJNI.StopWordsInvertedLists_get_single_code(swigCPtr, this, list_no, offset);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public void prefetch_lists(LongVector list_nos, int nlist) {\n    swigfaissJNI.StopWordsInvertedLists_prefetch_lists(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, nlist);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/Uint64Vector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class Uint64Vector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected Uint64Vector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(Uint64Vector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_Uint64Vector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public Uint64Vector() {\n    this(swigfaissJNI.new_Uint64Vector(), true);\n  }\n\n  public void push_back(long arg0) {\n    swigfaissJNI.Uint64Vector_push_back(swigCPtr, this, arg0);\n  }\n\n  public void clear() {\n    swigfaissJNI.Uint64Vector_clear(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_unsigned_long data() {\n    long cPtr = swigfaissJNI.Uint64Vector_data(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_long(cPtr, false);\n  }\n\n  public long size() {\n    return swigfaissJNI.Uint64Vector_size(swigCPtr, this);\n  }\n\n  public long at(long n) {\n    return swigfaissJNI.Uint64Vector_at(swigCPtr, this, n);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.Uint64Vector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.Uint64Vector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(Uint64Vector other) {\n    swigfaissJNI.Uint64Vector_swap(swigCPtr, this, Uint64Vector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/VStackInvertedLists.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class VStackInvertedLists extends ReadOnlyInvertedLists {\n  private transient long swigCPtr;\n\n  protected VStackInvertedLists(long cPtr, boolean cMemoryOwn) {\n    super(swigfaissJNI.VStackInvertedLists_SWIGUpcast(cPtr), cMemoryOwn);\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(VStackInvertedLists obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_VStackInvertedLists(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n    super.delete();\n  }\n\n  public void setIls(SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t value) {\n    swigfaissJNI.VStackInvertedLists_ils_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t getIls() {\n    long cPtr = swigfaissJNI.VStackInvertedLists_ils_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_faiss__InvertedLists_const_p_t(cPtr, false);\n  }\n\n  public void setCumsz(SWIGTYPE_p_std__vectorT_int64_t_t value) {\n    swigfaissJNI.VStackInvertedLists_cumsz_set(swigCPtr, this, SWIGTYPE_p_std__vectorT_int64_t_t.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_std__vectorT_int64_t_t getCumsz() {\n    long cPtr = swigfaissJNI.VStackInvertedLists_cumsz_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_std__vectorT_int64_t_t(cPtr, false);\n  }\n\n  public VStackInvertedLists(int nil, SWIGTYPE_p_p_faiss__InvertedLists ils) {\n    this(swigfaissJNI.new_VStackInvertedLists(nil, SWIGTYPE_p_p_faiss__InvertedLists.getCPtr(ils)), true);\n  }\n\n  public long list_size(long list_no) {\n    return swigfaissJNI.VStackInvertedLists_list_size(swigCPtr, this, list_no);\n  }\n\n  public SWIGTYPE_p_unsigned_char get_codes(long list_no) {\n    long cPtr = swigfaissJNI.VStackInvertedLists_get_codes(swigCPtr, this, list_no);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public LongVector get_ids(long list_no) {\n    return new LongVector(swigfaissJNI.VStackInvertedLists_get_ids(swigCPtr, this, list_no), false);\n}\n\n  public void release_codes(long list_no, SWIGTYPE_p_unsigned_char codes) {\n    swigfaissJNI.VStackInvertedLists_release_codes(swigCPtr, this, list_no, SWIGTYPE_p_unsigned_char.getCPtr(codes));\n  }\n\n  public void release_ids(long list_no, LongVector ids) {\n    swigfaissJNI.VStackInvertedLists_release_ids(swigCPtr, this, list_no, SWIGTYPE_p_long_long.getCPtr(ids.data()), ids);\n  }\n\n  public long get_single_id(long list_no, long offset) {\n    return swigfaissJNI.VStackInvertedLists_get_single_id(swigCPtr, this, list_no, offset);\n}\n\n  public SWIGTYPE_p_unsigned_char get_single_code(long list_no, long offset) {\n    long cPtr = swigfaissJNI.VStackInvertedLists_get_single_code(swigCPtr, this, list_no, offset);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_unsigned_char(cPtr, false);\n  }\n\n  public void prefetch_lists(LongVector list_nos, int nlist) {\n    swigfaissJNI.VStackInvertedLists_prefetch_lists(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(list_nos.data()), list_nos, nlist);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/VectorTransform.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class VectorTransform {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected VectorTransform(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(VectorTransform obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_VectorTransform(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setD_in(int value) {\n    swigfaissJNI.VectorTransform_d_in_set(swigCPtr, this, value);\n  }\n\n  public int getD_in() {\n    return swigfaissJNI.VectorTransform_d_in_get(swigCPtr, this);\n  }\n\n  public void setD_out(int value) {\n    swigfaissJNI.VectorTransform_d_out_set(swigCPtr, this, value);\n  }\n\n  public int getD_out() {\n    return swigfaissJNI.VectorTransform_d_out_get(swigCPtr, this);\n  }\n\n  public void setIs_trained(boolean value) {\n    swigfaissJNI.VectorTransform_is_trained_set(swigCPtr, this, value);\n  }\n\n  public boolean getIs_trained() {\n    return swigfaissJNI.VectorTransform_is_trained_get(swigCPtr, this);\n  }\n\n  public void train(long n, SWIGTYPE_p_float x) {\n    swigfaissJNI.VectorTransform_train(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n  }\n\n  public SWIGTYPE_p_float apply(long n, SWIGTYPE_p_float x) {\n    long cPtr = swigfaissJNI.VectorTransform_apply(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x));\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public void apply_noalloc(long n, SWIGTYPE_p_float x, SWIGTYPE_p_float xt) {\n    swigfaissJNI.VectorTransform_apply_noalloc(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(xt));\n  }\n\n  public void reverse_transform(long n, SWIGTYPE_p_float xt, SWIGTYPE_p_float x) {\n    swigfaissJNI.VectorTransform_reverse_transform(swigCPtr, this, n, SWIGTYPE_p_float.getCPtr(xt), SWIGTYPE_p_float.getCPtr(x));\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/VectorTransformVector.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class VectorTransformVector {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected VectorTransformVector(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(VectorTransformVector obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_VectorTransformVector(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public VectorTransformVector() {\n    this(swigfaissJNI.new_VectorTransformVector(), true);\n  }\n\n  public void push_back(VectorTransform arg0) {\n    swigfaissJNI.VectorTransformVector_push_back(swigCPtr, this, VectorTransform.getCPtr(arg0), arg0);\n  }\n\n  public void clear() {\n    swigfaissJNI.VectorTransformVector_clear(swigCPtr, this);\n  }\n\n  public SWIGTYPE_p_p_faiss__VectorTransform data() {\n    long cPtr = swigfaissJNI.VectorTransformVector_data(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_p_faiss__VectorTransform(cPtr, false);\n  }\n\n  public long size() {\n    return swigfaissJNI.VectorTransformVector_size(swigCPtr, this);\n  }\n\n  public VectorTransform at(long n) {\n    long cPtr = swigfaissJNI.VectorTransformVector_at(swigCPtr, this, n);\n    return (cPtr == 0) ? null : new VectorTransform(cPtr, false);\n  }\n\n  public void resize(long n) {\n    swigfaissJNI.VectorTransformVector_resize(swigCPtr, this, n);\n  }\n\n  public void reserve(long n) {\n    swigfaissJNI.VectorTransformVector_reserve(swigCPtr, this, n);\n  }\n\n  public void swap(VectorTransformVector other) {\n    swigfaissJNI.VectorTransformVector_swap(swigCPtr, this, VectorTransformVector.getCPtr(other), other);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/VisitedTable.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class VisitedTable {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected VisitedTable(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(VisitedTable obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_VisitedTable(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setVisited(ByteVector value) {\n    swigfaissJNI.VisitedTable_visited_set(swigCPtr, this, ByteVector.getCPtr(value), value);\n  }\n\n  public ByteVector getVisited() {\n    long cPtr = swigfaissJNI.VisitedTable_visited_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new ByteVector(cPtr, false);\n  }\n\n  public void setVisno(int value) {\n    swigfaissJNI.VisitedTable_visno_set(swigCPtr, this, value);\n  }\n\n  public int getVisno() {\n    return swigfaissJNI.VisitedTable_visno_get(swigCPtr, this);\n  }\n\n  public VisitedTable(int size) {\n    this(swigfaissJNI.new_VisitedTable(size), true);\n  }\n\n  public void set(int no) {\n    swigfaissJNI.VisitedTable_set(swigCPtr, this, no);\n  }\n\n  public boolean get(int no) {\n    return swigfaissJNI.VisitedTable_get(swigCPtr, this, no);\n  }\n\n  public void advance() {\n    swigfaissJNI.VisitedTable_advance(swigCPtr, this);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/doubleArray.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class doubleArray {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected doubleArray(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(doubleArray obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_doubleArray(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public doubleArray(int nelements) {\n    this(swigfaissJNI.new_doubleArray(nelements), true);\n  }\n\n  public double getitem(int index) {\n    return swigfaissJNI.doubleArray_getitem(swigCPtr, this, index);\n  }\n\n  public void setitem(int index, double value) {\n    swigfaissJNI.doubleArray_setitem(swigCPtr, this, index, value);\n  }\n\n  public SWIGTYPE_p_double cast() {\n    long cPtr = swigfaissJNI.doubleArray_cast(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_double(cPtr, false);\n  }\n\n  public static doubleArray frompointer(SWIGTYPE_p_double t) {\n    long cPtr = swigfaissJNI.doubleArray_frompointer(SWIGTYPE_p_double.getCPtr(t));\n    return (cPtr == 0) ? null : new doubleArray(cPtr, false);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/floatArray.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class floatArray {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected floatArray(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(floatArray obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_floatArray(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public floatArray(int nelements) {\n    this(swigfaissJNI.new_floatArray(nelements), true);\n  }\n\n  public float getitem(int index) {\n    return swigfaissJNI.floatArray_getitem(swigCPtr, this, index);\n  }\n\n  public void setitem(int index, float value) {\n    swigfaissJNI.floatArray_setitem(swigCPtr, this, index, value);\n  }\n\n  public SWIGTYPE_p_float cast() {\n    long cPtr = swigfaissJNI.floatArray_cast(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public static floatArray frompointer(SWIGTYPE_p_float t) {\n    long cPtr = swigfaissJNI.floatArray_frompointer(SWIGTYPE_p_float.getCPtr(t));\n    return (cPtr == 0) ? null : new floatArray(cPtr, false);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/float_maxheap_array_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class float_maxheap_array_t {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected float_maxheap_array_t(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(float_maxheap_array_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_float_maxheap_array_t(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setNh(long value) {\n    swigfaissJNI.float_maxheap_array_t_nh_set(swigCPtr, this, value);\n  }\n\n  public long getNh() {\n    return swigfaissJNI.float_maxheap_array_t_nh_get(swigCPtr, this);\n  }\n\n  public void setK(long value) {\n    swigfaissJNI.float_maxheap_array_t_k_set(swigCPtr, this, value);\n  }\n\n  public long getK() {\n    return swigfaissJNI.float_maxheap_array_t_k_get(swigCPtr, this);\n  }\n\n  public void setIds(LongVector value) {\n    swigfaissJNI.float_maxheap_array_t_ids_set(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(value.data()), value);\n  }\n\n  public LongVector getIds() {\n    return new LongVector(swigfaissJNI.float_maxheap_array_t_ids_get(swigCPtr, this), false);\n}\n\n  public void setVal(SWIGTYPE_p_float value) {\n    swigfaissJNI.float_maxheap_array_t_val_set(swigCPtr, this, SWIGTYPE_p_float.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_float getVal() {\n    long cPtr = swigfaissJNI.float_maxheap_array_t_val_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public SWIGTYPE_p_float get_val(long key) {\n    long cPtr = swigfaissJNI.float_maxheap_array_t_get_val(swigCPtr, this, key);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public LongVector get_ids(long key) {\n    return new LongVector(swigfaissJNI.float_maxheap_array_t_get_ids(swigCPtr, this, key), false);\n}\n\n  public void heapify() {\n    swigfaissJNI.float_maxheap_array_t_heapify(swigCPtr, this);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_float vin, long j0, long i0, long ni) {\n    swigfaissJNI.float_maxheap_array_t_addn__SWIG_0(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), j0, i0, ni);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_float vin, long j0, long i0) {\n    swigfaissJNI.float_maxheap_array_t_addn__SWIG_1(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), j0, i0);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_float vin, long j0) {\n    swigfaissJNI.float_maxheap_array_t_addn__SWIG_2(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), j0);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_float vin) {\n    swigfaissJNI.float_maxheap_array_t_addn__SWIG_3(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin));\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_float vin, LongVector id_in, long id_stride, long i0, long ni) {\n    swigfaissJNI.float_maxheap_array_t_addn_with_ids__SWIG_0(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in, id_stride, i0, ni);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_float vin, LongVector id_in, long id_stride, long i0) {\n    swigfaissJNI.float_maxheap_array_t_addn_with_ids__SWIG_1(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in, id_stride, i0);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_float vin, LongVector id_in, long id_stride) {\n    swigfaissJNI.float_maxheap_array_t_addn_with_ids__SWIG_2(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in, id_stride);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_float vin, LongVector id_in) {\n    swigfaissJNI.float_maxheap_array_t_addn_with_ids__SWIG_3(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_float vin) {\n    swigfaissJNI.float_maxheap_array_t_addn_with_ids__SWIG_4(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin));\n  }\n\n  public void reorder() {\n    swigfaissJNI.float_maxheap_array_t_reorder(swigCPtr, this);\n  }\n\n  public void per_line_extrema(SWIGTYPE_p_float vals_out, LongVector idx_out) {\n    swigfaissJNI.float_maxheap_array_t_per_line_extrema(swigCPtr, this, SWIGTYPE_p_float.getCPtr(vals_out), SWIGTYPE_p_long_long.getCPtr(idx_out.data()), idx_out);\n  }\n\n  public float_maxheap_array_t() {\n    this(swigfaissJNI.new_float_maxheap_array_t(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/float_minheap_array_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class float_minheap_array_t {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected float_minheap_array_t(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(float_minheap_array_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_float_minheap_array_t(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setNh(long value) {\n    swigfaissJNI.float_minheap_array_t_nh_set(swigCPtr, this, value);\n  }\n\n  public long getNh() {\n    return swigfaissJNI.float_minheap_array_t_nh_get(swigCPtr, this);\n  }\n\n  public void setK(long value) {\n    swigfaissJNI.float_minheap_array_t_k_set(swigCPtr, this, value);\n  }\n\n  public long getK() {\n    return swigfaissJNI.float_minheap_array_t_k_get(swigCPtr, this);\n  }\n\n  public void setIds(LongVector value) {\n    swigfaissJNI.float_minheap_array_t_ids_set(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(value.data()), value);\n  }\n\n  public LongVector getIds() {\n    return new LongVector(swigfaissJNI.float_minheap_array_t_ids_get(swigCPtr, this), false);\n}\n\n  public void setVal(SWIGTYPE_p_float value) {\n    swigfaissJNI.float_minheap_array_t_val_set(swigCPtr, this, SWIGTYPE_p_float.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_float getVal() {\n    long cPtr = swigfaissJNI.float_minheap_array_t_val_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public SWIGTYPE_p_float get_val(long key) {\n    long cPtr = swigfaissJNI.float_minheap_array_t_get_val(swigCPtr, this, key);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public LongVector get_ids(long key) {\n    return new LongVector(swigfaissJNI.float_minheap_array_t_get_ids(swigCPtr, this, key), false);\n}\n\n  public void heapify() {\n    swigfaissJNI.float_minheap_array_t_heapify(swigCPtr, this);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_float vin, long j0, long i0, long ni) {\n    swigfaissJNI.float_minheap_array_t_addn__SWIG_0(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), j0, i0, ni);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_float vin, long j0, long i0) {\n    swigfaissJNI.float_minheap_array_t_addn__SWIG_1(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), j0, i0);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_float vin, long j0) {\n    swigfaissJNI.float_minheap_array_t_addn__SWIG_2(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), j0);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_float vin) {\n    swigfaissJNI.float_minheap_array_t_addn__SWIG_3(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin));\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_float vin, LongVector id_in, long id_stride, long i0, long ni) {\n    swigfaissJNI.float_minheap_array_t_addn_with_ids__SWIG_0(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in, id_stride, i0, ni);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_float vin, LongVector id_in, long id_stride, long i0) {\n    swigfaissJNI.float_minheap_array_t_addn_with_ids__SWIG_1(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in, id_stride, i0);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_float vin, LongVector id_in, long id_stride) {\n    swigfaissJNI.float_minheap_array_t_addn_with_ids__SWIG_2(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in, id_stride);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_float vin, LongVector id_in) {\n    swigfaissJNI.float_minheap_array_t_addn_with_ids__SWIG_3(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_float vin) {\n    swigfaissJNI.float_minheap_array_t_addn_with_ids__SWIG_4(swigCPtr, this, nj, SWIGTYPE_p_float.getCPtr(vin));\n  }\n\n  public void reorder() {\n    swigfaissJNI.float_minheap_array_t_reorder(swigCPtr, this);\n  }\n\n  public void per_line_extrema(SWIGTYPE_p_float vals_out, LongVector idx_out) {\n    swigfaissJNI.float_minheap_array_t_per_line_extrema(swigCPtr, this, SWIGTYPE_p_float.getCPtr(vals_out), SWIGTYPE_p_long_long.getCPtr(idx_out.data()), idx_out);\n  }\n\n  public float_minheap_array_t() {\n    this(swigfaissJNI.new_float_minheap_array_t(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/intArray.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class intArray {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected intArray(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(intArray obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_intArray(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public intArray(int nelements) {\n    this(swigfaissJNI.new_intArray(nelements), true);\n  }\n\n  public int getitem(int index) {\n    return swigfaissJNI.intArray_getitem(swigCPtr, this, index);\n  }\n\n  public void setitem(int index, int value) {\n    swigfaissJNI.intArray_setitem(swigCPtr, this, index, value);\n  }\n\n  public SWIGTYPE_p_int cast() {\n    long cPtr = swigfaissJNI.intArray_cast(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_int(cPtr, false);\n  }\n\n  public static intArray frompointer(SWIGTYPE_p_int t) {\n    long cPtr = swigfaissJNI.intArray_frompointer(SWIGTYPE_p_int.getCPtr(t));\n    return (cPtr == 0) ? null : new intArray(cPtr, false);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/int_maxheap_array_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class int_maxheap_array_t {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected int_maxheap_array_t(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(int_maxheap_array_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_int_maxheap_array_t(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setNh(long value) {\n    swigfaissJNI.int_maxheap_array_t_nh_set(swigCPtr, this, value);\n  }\n\n  public long getNh() {\n    return swigfaissJNI.int_maxheap_array_t_nh_get(swigCPtr, this);\n  }\n\n  public void setK(long value) {\n    swigfaissJNI.int_maxheap_array_t_k_set(swigCPtr, this, value);\n  }\n\n  public long getK() {\n    return swigfaissJNI.int_maxheap_array_t_k_get(swigCPtr, this);\n  }\n\n  public void setIds(LongVector value) {\n    swigfaissJNI.int_maxheap_array_t_ids_set(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(value.data()), value);\n  }\n\n  public LongVector getIds() {\n    return new LongVector(swigfaissJNI.int_maxheap_array_t_ids_get(swigCPtr, this), false);\n}\n\n  public void setVal(SWIGTYPE_p_int value) {\n    swigfaissJNI.int_maxheap_array_t_val_set(swigCPtr, this, SWIGTYPE_p_int.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_int getVal() {\n    long cPtr = swigfaissJNI.int_maxheap_array_t_val_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_int(cPtr, false);\n  }\n\n  public SWIGTYPE_p_int get_val(long key) {\n    long cPtr = swigfaissJNI.int_maxheap_array_t_get_val(swigCPtr, this, key);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_int(cPtr, false);\n  }\n\n  public LongVector get_ids(long key) {\n    return new LongVector(swigfaissJNI.int_maxheap_array_t_get_ids(swigCPtr, this, key), false);\n}\n\n  public void heapify() {\n    swigfaissJNI.int_maxheap_array_t_heapify(swigCPtr, this);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_int vin, long j0, long i0, long ni) {\n    swigfaissJNI.int_maxheap_array_t_addn__SWIG_0(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), j0, i0, ni);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_int vin, long j0, long i0) {\n    swigfaissJNI.int_maxheap_array_t_addn__SWIG_1(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), j0, i0);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_int vin, long j0) {\n    swigfaissJNI.int_maxheap_array_t_addn__SWIG_2(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), j0);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_int vin) {\n    swigfaissJNI.int_maxheap_array_t_addn__SWIG_3(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin));\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_int vin, LongVector id_in, long id_stride, long i0, long ni) {\n    swigfaissJNI.int_maxheap_array_t_addn_with_ids__SWIG_0(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in, id_stride, i0, ni);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_int vin, LongVector id_in, long id_stride, long i0) {\n    swigfaissJNI.int_maxheap_array_t_addn_with_ids__SWIG_1(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in, id_stride, i0);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_int vin, LongVector id_in, long id_stride) {\n    swigfaissJNI.int_maxheap_array_t_addn_with_ids__SWIG_2(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in, id_stride);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_int vin, LongVector id_in) {\n    swigfaissJNI.int_maxheap_array_t_addn_with_ids__SWIG_3(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_int vin) {\n    swigfaissJNI.int_maxheap_array_t_addn_with_ids__SWIG_4(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin));\n  }\n\n  public void reorder() {\n    swigfaissJNI.int_maxheap_array_t_reorder(swigCPtr, this);\n  }\n\n  public void per_line_extrema(SWIGTYPE_p_int vals_out, LongVector idx_out) {\n    swigfaissJNI.int_maxheap_array_t_per_line_extrema(swigCPtr, this, SWIGTYPE_p_int.getCPtr(vals_out), SWIGTYPE_p_long_long.getCPtr(idx_out.data()), idx_out);\n  }\n\n  public int_maxheap_array_t() {\n    this(swigfaissJNI.new_int_maxheap_array_t(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/int_minheap_array_t.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class int_minheap_array_t {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected int_minheap_array_t(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(int_minheap_array_t obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_int_minheap_array_t(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public void setNh(long value) {\n    swigfaissJNI.int_minheap_array_t_nh_set(swigCPtr, this, value);\n  }\n\n  public long getNh() {\n    return swigfaissJNI.int_minheap_array_t_nh_get(swigCPtr, this);\n  }\n\n  public void setK(long value) {\n    swigfaissJNI.int_minheap_array_t_k_set(swigCPtr, this, value);\n  }\n\n  public long getK() {\n    return swigfaissJNI.int_minheap_array_t_k_get(swigCPtr, this);\n  }\n\n  public void setIds(LongVector value) {\n    swigfaissJNI.int_minheap_array_t_ids_set(swigCPtr, this, SWIGTYPE_p_long_long.getCPtr(value.data()), value);\n  }\n\n  public LongVector getIds() {\n    return new LongVector(swigfaissJNI.int_minheap_array_t_ids_get(swigCPtr, this), false);\n}\n\n  public void setVal(SWIGTYPE_p_int value) {\n    swigfaissJNI.int_minheap_array_t_val_set(swigCPtr, this, SWIGTYPE_p_int.getCPtr(value));\n  }\n\n  public SWIGTYPE_p_int getVal() {\n    long cPtr = swigfaissJNI.int_minheap_array_t_val_get(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_int(cPtr, false);\n  }\n\n  public SWIGTYPE_p_int get_val(long key) {\n    long cPtr = swigfaissJNI.int_minheap_array_t_get_val(swigCPtr, this, key);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_int(cPtr, false);\n  }\n\n  public LongVector get_ids(long key) {\n    return new LongVector(swigfaissJNI.int_minheap_array_t_get_ids(swigCPtr, this, key), false);\n}\n\n  public void heapify() {\n    swigfaissJNI.int_minheap_array_t_heapify(swigCPtr, this);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_int vin, long j0, long i0, long ni) {\n    swigfaissJNI.int_minheap_array_t_addn__SWIG_0(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), j0, i0, ni);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_int vin, long j0, long i0) {\n    swigfaissJNI.int_minheap_array_t_addn__SWIG_1(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), j0, i0);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_int vin, long j0) {\n    swigfaissJNI.int_minheap_array_t_addn__SWIG_2(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), j0);\n  }\n\n  public void addn(long nj, SWIGTYPE_p_int vin) {\n    swigfaissJNI.int_minheap_array_t_addn__SWIG_3(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin));\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_int vin, LongVector id_in, long id_stride, long i0, long ni) {\n    swigfaissJNI.int_minheap_array_t_addn_with_ids__SWIG_0(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in, id_stride, i0, ni);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_int vin, LongVector id_in, long id_stride, long i0) {\n    swigfaissJNI.int_minheap_array_t_addn_with_ids__SWIG_1(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in, id_stride, i0);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_int vin, LongVector id_in, long id_stride) {\n    swigfaissJNI.int_minheap_array_t_addn_with_ids__SWIG_2(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in, id_stride);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_int vin, LongVector id_in) {\n    swigfaissJNI.int_minheap_array_t_addn_with_ids__SWIG_3(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin), SWIGTYPE_p_long_long.getCPtr(id_in.data()), id_in);\n  }\n\n  public void addn_with_ids(long nj, SWIGTYPE_p_int vin) {\n    swigfaissJNI.int_minheap_array_t_addn_with_ids__SWIG_4(swigCPtr, this, nj, SWIGTYPE_p_int.getCPtr(vin));\n  }\n\n  public void reorder() {\n    swigfaissJNI.int_minheap_array_t_reorder(swigCPtr, this);\n  }\n\n  public void per_line_extrema(SWIGTYPE_p_int vals_out, LongVector idx_out) {\n    swigfaissJNI.int_minheap_array_t_per_line_extrema(swigCPtr, this, SWIGTYPE_p_int.getCPtr(vals_out), SWIGTYPE_p_long_long.getCPtr(idx_out.data()), idx_out);\n  }\n\n  public int_minheap_array_t() {\n    this(swigfaissJNI.new_int_minheap_array_t(), true);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/longArray.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class longArray {\n  private transient long swigCPtr;\n  protected transient boolean swigCMemOwn;\n\n  protected longArray(long cPtr, boolean cMemoryOwn) {\n    swigCMemOwn = cMemoryOwn;\n    swigCPtr = cPtr;\n  }\n\n  protected static long getCPtr(longArray obj) {\n    return (obj == null) ? 0 : obj.swigCPtr;\n  }\n\n  @SuppressWarnings(\"deprecation\")\n  protected void finalize() {\n    delete();\n  }\n\n  public synchronized void delete() {\n    if (swigCPtr != 0) {\n      if (swigCMemOwn) {\n        swigCMemOwn = false;\n        swigfaissJNI.delete_longArray(swigCPtr);\n      }\n      swigCPtr = 0;\n    }\n  }\n\n  public longArray(int nelements) {\n    this(swigfaissJNI.new_longArray(nelements), true);\n  }\n\n  public long getitem(int index) {\n    return swigfaissJNI.longArray_getitem(swigCPtr, this, index);\n  }\n\n  public void setitem(int index, long value) {\n    swigfaissJNI.longArray_setitem(swigCPtr, this, index, value);\n  }\n\n  public SWIGTYPE_p_long_long cast() {\n    long cPtr = swigfaissJNI.longArray_cast(swigCPtr, this);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_long_long(cPtr, false);\n  }\n\n  public static longArray frompointer(SWIGTYPE_p_long_long t) {\n    long cPtr = swigfaissJNI.longArray_frompointer(SWIGTYPE_p_long_long.getCPtr(t));\n    return (cPtr == 0) ? null : new longArray(cPtr, false);\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/resources/.gitignore",
    "content": "*.so\n*.so.0\n*.so.1\n*.so.3\n*.so.5\n*.so.6\n*.dylib\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/resources/.gitkeep",
    "content": ""
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/resources/BUILD",
    "content": "resources(\n    name = \"resources\",\n    sources = [\n        \"*.dylib\",\n        \"*.so\",\n        \"*.so.0\",\n        \"*.so.1\",\n        \"*.so.3\",\n        \"*.so.5\",\n        \"*.so.6\",\n    ],\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n        \"visibility://visibility:private\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/swigfaiss.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic class swigfaiss implements swigfaissConstants {\n  public static void bitvec_print(SWIGTYPE_p_unsigned_char b, long d) {\n    swigfaissJNI.bitvec_print(SWIGTYPE_p_unsigned_char.getCPtr(b), d);\n  }\n\n  public static void fvecs2bitvecs(SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char b, long d, long n) {\n    swigfaissJNI.fvecs2bitvecs(SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(b), d, n);\n  }\n\n  public static void bitvecs2fvecs(SWIGTYPE_p_unsigned_char b, SWIGTYPE_p_float x, long d, long n) {\n    swigfaissJNI.bitvecs2fvecs(SWIGTYPE_p_unsigned_char.getCPtr(b), SWIGTYPE_p_float.getCPtr(x), d, n);\n  }\n\n  public static void fvec2bitvec(SWIGTYPE_p_float x, SWIGTYPE_p_unsigned_char b, long d) {\n    swigfaissJNI.fvec2bitvec(SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_unsigned_char.getCPtr(b), d);\n  }\n\n  public static void bitvec_shuffle(long n, long da, long db, SWIGTYPE_p_int order, SWIGTYPE_p_unsigned_char a, SWIGTYPE_p_unsigned_char b) {\n    swigfaissJNI.bitvec_shuffle(n, da, db, SWIGTYPE_p_int.getCPtr(order), SWIGTYPE_p_unsigned_char.getCPtr(a), SWIGTYPE_p_unsigned_char.getCPtr(b));\n  }\n\n  public static void setHamming_batch_size(long value) {\n    swigfaissJNI.hamming_batch_size_set(value);\n  }\n\n  public static long getHamming_batch_size() {\n    return swigfaissJNI.hamming_batch_size_get();\n  }\n\n  public static int popcount64(long x) {\n    return swigfaissJNI.popcount64(x);\n  }\n\n  public static void hammings(SWIGTYPE_p_unsigned_char a, SWIGTYPE_p_unsigned_char b, long na, long nb, long nbytespercode, SWIGTYPE_p_int dis) {\n    swigfaissJNI.hammings(SWIGTYPE_p_unsigned_char.getCPtr(a), SWIGTYPE_p_unsigned_char.getCPtr(b), na, nb, nbytespercode, SWIGTYPE_p_int.getCPtr(dis));\n  }\n\n  public static void hammings_knn_hc(SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t ha, SWIGTYPE_p_unsigned_char a, SWIGTYPE_p_unsigned_char b, long nb, long ncodes, int ordered) {\n    swigfaissJNI.hammings_knn_hc(SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t.getCPtr(ha), SWIGTYPE_p_unsigned_char.getCPtr(a), SWIGTYPE_p_unsigned_char.getCPtr(b), nb, ncodes, ordered);\n  }\n\n  public static void hammings_knn(SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t ha, SWIGTYPE_p_unsigned_char a, SWIGTYPE_p_unsigned_char b, long nb, long ncodes, int ordered) {\n    swigfaissJNI.hammings_knn(SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t.getCPtr(ha), SWIGTYPE_p_unsigned_char.getCPtr(a), SWIGTYPE_p_unsigned_char.getCPtr(b), nb, ncodes, ordered);\n  }\n\n  public static void hammings_knn_mc(SWIGTYPE_p_unsigned_char a, SWIGTYPE_p_unsigned_char b, long na, long nb, long k, long ncodes, SWIGTYPE_p_int distances, LongVector labels) {\n    swigfaissJNI.hammings_knn_mc(SWIGTYPE_p_unsigned_char.getCPtr(a), SWIGTYPE_p_unsigned_char.getCPtr(b), na, nb, k, ncodes, SWIGTYPE_p_int.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels);\n  }\n\n  public static void hamming_range_search(SWIGTYPE_p_unsigned_char a, SWIGTYPE_p_unsigned_char b, long na, long nb, int radius, long ncodes, RangeSearchResult result) {\n    swigfaissJNI.hamming_range_search(SWIGTYPE_p_unsigned_char.getCPtr(a), SWIGTYPE_p_unsigned_char.getCPtr(b), na, nb, radius, ncodes, RangeSearchResult.getCPtr(result), result);\n  }\n\n  public static void hamming_count_thres(SWIGTYPE_p_unsigned_char bs1, SWIGTYPE_p_unsigned_char bs2, long n1, long n2, int ht, long ncodes, SWIGTYPE_p_unsigned_long nptr) {\n    swigfaissJNI.hamming_count_thres(SWIGTYPE_p_unsigned_char.getCPtr(bs1), SWIGTYPE_p_unsigned_char.getCPtr(bs2), n1, n2, ht, ncodes, SWIGTYPE_p_unsigned_long.getCPtr(nptr));\n  }\n\n  public static long match_hamming_thres(SWIGTYPE_p_unsigned_char bs1, SWIGTYPE_p_unsigned_char bs2, long n1, long n2, int ht, long ncodes, LongVector idx, SWIGTYPE_p_int dis) {\n    return swigfaissJNI.match_hamming_thres(SWIGTYPE_p_unsigned_char.getCPtr(bs1), SWIGTYPE_p_unsigned_char.getCPtr(bs2), n1, n2, ht, ncodes, SWIGTYPE_p_long_long.getCPtr(idx.data()), idx, SWIGTYPE_p_int.getCPtr(dis));\n  }\n\n  public static void crosshamming_count_thres(SWIGTYPE_p_unsigned_char dbs, long n, int ht, long ncodes, SWIGTYPE_p_unsigned_long nptr) {\n    swigfaissJNI.crosshamming_count_thres(SWIGTYPE_p_unsigned_char.getCPtr(dbs), n, ht, ncodes, SWIGTYPE_p_unsigned_long.getCPtr(nptr));\n  }\n\n  public static int get_num_gpus() {\n    return swigfaissJNI.get_num_gpus();\n  }\n\n  public static String get_compile_options() {\n    return swigfaissJNI.get_compile_options();\n  }\n\n  public static double getmillisecs() {\n    return swigfaissJNI.getmillisecs();\n  }\n\n  public static long get_mem_usage_kb() {\n    return swigfaissJNI.get_mem_usage_kb();\n  }\n\n  public static long get_cycles() {\n    return swigfaissJNI.get_cycles();\n  }\n\n  public static void fvec_madd(long n, SWIGTYPE_p_float a, float bf, SWIGTYPE_p_float b, SWIGTYPE_p_float c) {\n    swigfaissJNI.fvec_madd(n, SWIGTYPE_p_float.getCPtr(a), bf, SWIGTYPE_p_float.getCPtr(b), SWIGTYPE_p_float.getCPtr(c));\n  }\n\n  public static int fvec_madd_and_argmin(long n, SWIGTYPE_p_float a, float bf, SWIGTYPE_p_float b, SWIGTYPE_p_float c) {\n    return swigfaissJNI.fvec_madd_and_argmin(n, SWIGTYPE_p_float.getCPtr(a), bf, SWIGTYPE_p_float.getCPtr(b), SWIGTYPE_p_float.getCPtr(c));\n  }\n\n  public static void reflection(SWIGTYPE_p_float u, SWIGTYPE_p_float x, long n, long d, long nu) {\n    swigfaissJNI.reflection(SWIGTYPE_p_float.getCPtr(u), SWIGTYPE_p_float.getCPtr(x), n, d, nu);\n  }\n\n  public static void matrix_qr(int m, int n, SWIGTYPE_p_float a) {\n    swigfaissJNI.matrix_qr(m, n, SWIGTYPE_p_float.getCPtr(a));\n  }\n\n  public static void ranklist_handle_ties(int k, LongVector idx, SWIGTYPE_p_float dis) {\n    swigfaissJNI.ranklist_handle_ties(k, SWIGTYPE_p_long_long.getCPtr(idx.data()), idx, SWIGTYPE_p_float.getCPtr(dis));\n  }\n\n  public static long ranklist_intersection_size(long k1, LongVector v1, long k2, LongVector v2) {\n    return swigfaissJNI.ranklist_intersection_size(k1, SWIGTYPE_p_long_long.getCPtr(v1.data()), v1, k2, SWIGTYPE_p_long_long.getCPtr(v2.data()), v2);\n  }\n\n  public static long merge_result_table_with(long n, long k, LongVector I0, SWIGTYPE_p_float D0, LongVector I1, SWIGTYPE_p_float D1, boolean keep_min, long translation) {\n    return swigfaissJNI.merge_result_table_with__SWIG_0(n, k, SWIGTYPE_p_long_long.getCPtr(I0.data()), I0, SWIGTYPE_p_float.getCPtr(D0), SWIGTYPE_p_long_long.getCPtr(I1.data()), I1, SWIGTYPE_p_float.getCPtr(D1), keep_min, translation);\n  }\n\n  public static long merge_result_table_with(long n, long k, LongVector I0, SWIGTYPE_p_float D0, LongVector I1, SWIGTYPE_p_float D1, boolean keep_min) {\n    return swigfaissJNI.merge_result_table_with__SWIG_1(n, k, SWIGTYPE_p_long_long.getCPtr(I0.data()), I0, SWIGTYPE_p_float.getCPtr(D0), SWIGTYPE_p_long_long.getCPtr(I1.data()), I1, SWIGTYPE_p_float.getCPtr(D1), keep_min);\n  }\n\n  public static long merge_result_table_with(long n, long k, LongVector I0, SWIGTYPE_p_float D0, LongVector I1, SWIGTYPE_p_float D1) {\n    return swigfaissJNI.merge_result_table_with__SWIG_2(n, k, SWIGTYPE_p_long_long.getCPtr(I0.data()), I0, SWIGTYPE_p_float.getCPtr(D0), SWIGTYPE_p_long_long.getCPtr(I1.data()), I1, SWIGTYPE_p_float.getCPtr(D1));\n  }\n\n  public static double imbalance_factor(int n, int k, LongVector assign) {\n    return swigfaissJNI.imbalance_factor__SWIG_0(n, k, SWIGTYPE_p_long_long.getCPtr(assign.data()), assign);\n  }\n\n  public static double imbalance_factor(int k, SWIGTYPE_p_int hist) {\n    return swigfaissJNI.imbalance_factor__SWIG_1(k, SWIGTYPE_p_int.getCPtr(hist));\n  }\n\n  public static void fvec_argsort(long n, SWIGTYPE_p_float vals, SWIGTYPE_p_unsigned_long perm) {\n    swigfaissJNI.fvec_argsort(n, SWIGTYPE_p_float.getCPtr(vals), SWIGTYPE_p_unsigned_long.getCPtr(perm));\n  }\n\n  public static void fvec_argsort_parallel(long n, SWIGTYPE_p_float vals, SWIGTYPE_p_unsigned_long perm) {\n    swigfaissJNI.fvec_argsort_parallel(n, SWIGTYPE_p_float.getCPtr(vals), SWIGTYPE_p_unsigned_long.getCPtr(perm));\n  }\n\n  public static int ivec_hist(long n, SWIGTYPE_p_int v, int vmax, SWIGTYPE_p_int hist) {\n    return swigfaissJNI.ivec_hist(n, SWIGTYPE_p_int.getCPtr(v), vmax, SWIGTYPE_p_int.getCPtr(hist));\n  }\n\n  public static void bincode_hist(long n, long nbits, SWIGTYPE_p_unsigned_char codes, SWIGTYPE_p_int hist) {\n    swigfaissJNI.bincode_hist(n, nbits, SWIGTYPE_p_unsigned_char.getCPtr(codes), SWIGTYPE_p_int.getCPtr(hist));\n  }\n\n  public static long ivec_checksum(long n, SWIGTYPE_p_int a) {\n    return swigfaissJNI.ivec_checksum(n, SWIGTYPE_p_int.getCPtr(a));\n  }\n\n  public static SWIGTYPE_p_float fvecs_maybe_subsample(long d, SWIGTYPE_p_unsigned_long n, long nmax, SWIGTYPE_p_float x, boolean verbose, long seed) {\n    long cPtr = swigfaissJNI.fvecs_maybe_subsample__SWIG_0(d, SWIGTYPE_p_unsigned_long.getCPtr(n), nmax, SWIGTYPE_p_float.getCPtr(x), verbose, seed);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public static SWIGTYPE_p_float fvecs_maybe_subsample(long d, SWIGTYPE_p_unsigned_long n, long nmax, SWIGTYPE_p_float x, boolean verbose) {\n    long cPtr = swigfaissJNI.fvecs_maybe_subsample__SWIG_1(d, SWIGTYPE_p_unsigned_long.getCPtr(n), nmax, SWIGTYPE_p_float.getCPtr(x), verbose);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public static SWIGTYPE_p_float fvecs_maybe_subsample(long d, SWIGTYPE_p_unsigned_long n, long nmax, SWIGTYPE_p_float x) {\n    long cPtr = swigfaissJNI.fvecs_maybe_subsample__SWIG_2(d, SWIGTYPE_p_unsigned_long.getCPtr(n), nmax, SWIGTYPE_p_float.getCPtr(x));\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public static void binary_to_real(long d, SWIGTYPE_p_unsigned_char x_in, SWIGTYPE_p_float x_out) {\n    swigfaissJNI.binary_to_real(d, SWIGTYPE_p_unsigned_char.getCPtr(x_in), SWIGTYPE_p_float.getCPtr(x_out));\n  }\n\n  public static void real_to_binary(long d, SWIGTYPE_p_float x_in, SWIGTYPE_p_unsigned_char x_out) {\n    swigfaissJNI.real_to_binary(d, SWIGTYPE_p_float.getCPtr(x_in), SWIGTYPE_p_unsigned_char.getCPtr(x_out));\n  }\n\n  public static long hash_bytes(SWIGTYPE_p_unsigned_char bytes, long n) {\n    return swigfaissJNI.hash_bytes(SWIGTYPE_p_unsigned_char.getCPtr(bytes), n);\n  }\n\n  public static boolean check_openmp() {\n    return swigfaissJNI.check_openmp();\n  }\n\n  public static float kmeans_clustering(long d, long n, long k, SWIGTYPE_p_float x, SWIGTYPE_p_float centroids) {\n    return swigfaissJNI.kmeans_clustering(d, n, k, SWIGTYPE_p_float.getCPtr(x), SWIGTYPE_p_float.getCPtr(centroids));\n  }\n\n  public static void setIndexPQ_stats(IndexPQStats value) {\n    swigfaissJNI.indexPQ_stats_set(IndexPQStats.getCPtr(value), value);\n  }\n\n  public static IndexPQStats getIndexPQ_stats() {\n    long cPtr = swigfaissJNI.indexPQ_stats_get();\n    return (cPtr == 0) ? null : new IndexPQStats(cPtr, false);\n  }\n\n  public static void setIndexIVF_stats(IndexIVFStats value) {\n    swigfaissJNI.indexIVF_stats_set(IndexIVFStats.getCPtr(value), value);\n  }\n\n  public static IndexIVFStats getIndexIVF_stats() {\n    long cPtr = swigfaissJNI.indexIVF_stats_get();\n    return (cPtr == 0) ? null : new IndexIVFStats(cPtr, false);\n  }\n\n  public static short[] getHamdis_tab_ham_bytes() {\n    return swigfaissJNI.hamdis_tab_ham_bytes_get();\n  }\n\n  public static int generalized_hamming_64(long a) {\n    return swigfaissJNI.generalized_hamming_64(a);\n  }\n\n  public static void generalized_hammings_knn_hc(SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t ha, SWIGTYPE_p_unsigned_char a, SWIGTYPE_p_unsigned_char b, long nb, long code_size, int ordered) {\n    swigfaissJNI.generalized_hammings_knn_hc__SWIG_0(SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t.getCPtr(ha), SWIGTYPE_p_unsigned_char.getCPtr(a), SWIGTYPE_p_unsigned_char.getCPtr(b), nb, code_size, ordered);\n  }\n\n  public static void generalized_hammings_knn_hc(SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t ha, SWIGTYPE_p_unsigned_char a, SWIGTYPE_p_unsigned_char b, long nb, long code_size) {\n    swigfaissJNI.generalized_hammings_knn_hc__SWIG_1(SWIGTYPE_p_faiss__HeapArrayT_faiss__CMaxT_int_int64_t_t_t.getCPtr(ha), SWIGTYPE_p_unsigned_char.getCPtr(a), SWIGTYPE_p_unsigned_char.getCPtr(b), nb, code_size);\n  }\n\n  public static void check_compatible_for_merge(Index index1, Index index2) {\n    swigfaissJNI.check_compatible_for_merge(Index.getCPtr(index1), index1, Index.getCPtr(index2), index2);\n  }\n\n  public static IndexIVF extract_index_ivf(Index index) {\n    long cPtr = swigfaissJNI.extract_index_ivf__SWIG_0(Index.getCPtr(index), index);\n    return (cPtr == 0) ? null : new IndexIVF(cPtr, false);\n  }\n\n  public static IndexIVF try_extract_index_ivf(Index index) {\n    long cPtr = swigfaissJNI.try_extract_index_ivf__SWIG_0(Index.getCPtr(index), index);\n    return (cPtr == 0) ? null : new IndexIVF(cPtr, false);\n  }\n\n  public static void merge_into(Index index0, Index index1, boolean shift_ids) {\n    swigfaissJNI.merge_into(Index.getCPtr(index0), index0, Index.getCPtr(index1), index1, shift_ids);\n  }\n\n  public static void search_centroid(Index index, SWIGTYPE_p_float x, int n, LongVector centroid_ids) {\n    swigfaissJNI.search_centroid(Index.getCPtr(index), index, SWIGTYPE_p_float.getCPtr(x), n, SWIGTYPE_p_long_long.getCPtr(centroid_ids.data()), centroid_ids);\n  }\n\n  public static void search_and_return_centroids(Index index, long n, SWIGTYPE_p_float xin, int k, SWIGTYPE_p_float distances, LongVector labels, LongVector query_centroid_ids, LongVector result_centroid_ids) {\n    swigfaissJNI.search_and_return_centroids(Index.getCPtr(index), index, n, SWIGTYPE_p_float.getCPtr(xin), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, SWIGTYPE_p_long_long.getCPtr(query_centroid_ids.data()), query_centroid_ids, SWIGTYPE_p_long_long.getCPtr(result_centroid_ids.data()), result_centroid_ids);\n  }\n\n  public static ArrayInvertedLists get_invlist_range(Index index, int i0, int i1) {\n    long cPtr = swigfaissJNI.get_invlist_range(Index.getCPtr(index), index, i0, i1);\n    return (cPtr == 0) ? null : new ArrayInvertedLists(cPtr, false);\n  }\n\n  public static void set_invlist_range(Index index, int i0, int i1, ArrayInvertedLists src) {\n    swigfaissJNI.set_invlist_range(Index.getCPtr(index), index, i0, i1, ArrayInvertedLists.getCPtr(src), src);\n  }\n\n  public static void search_with_parameters(Index index, long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels, IVFSearchParameters params, SWIGTYPE_p_unsigned_long nb_dis, SWIGTYPE_p_double ms_per_stage) {\n    swigfaissJNI.search_with_parameters__SWIG_0(Index.getCPtr(index), index, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, IVFSearchParameters.getCPtr(params), params, SWIGTYPE_p_unsigned_long.getCPtr(nb_dis), SWIGTYPE_p_double.getCPtr(ms_per_stage));\n  }\n\n  public static void search_with_parameters(Index index, long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels, IVFSearchParameters params, SWIGTYPE_p_unsigned_long nb_dis) {\n    swigfaissJNI.search_with_parameters__SWIG_1(Index.getCPtr(index), index, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, IVFSearchParameters.getCPtr(params), params, SWIGTYPE_p_unsigned_long.getCPtr(nb_dis));\n  }\n\n  public static void search_with_parameters(Index index, long n, SWIGTYPE_p_float x, long k, SWIGTYPE_p_float distances, LongVector labels, IVFSearchParameters params) {\n    swigfaissJNI.search_with_parameters__SWIG_2(Index.getCPtr(index), index, n, SWIGTYPE_p_float.getCPtr(x), k, SWIGTYPE_p_float.getCPtr(distances), SWIGTYPE_p_long_long.getCPtr(labels.data()), labels, IVFSearchParameters.getCPtr(params), params);\n  }\n\n  public static void range_search_with_parameters(Index index, long n, SWIGTYPE_p_float x, float radius, RangeSearchResult result, IVFSearchParameters params, SWIGTYPE_p_unsigned_long nb_dis, SWIGTYPE_p_double ms_per_stage) {\n    swigfaissJNI.range_search_with_parameters__SWIG_0(Index.getCPtr(index), index, n, SWIGTYPE_p_float.getCPtr(x), radius, RangeSearchResult.getCPtr(result), result, IVFSearchParameters.getCPtr(params), params, SWIGTYPE_p_unsigned_long.getCPtr(nb_dis), SWIGTYPE_p_double.getCPtr(ms_per_stage));\n  }\n\n  public static void range_search_with_parameters(Index index, long n, SWIGTYPE_p_float x, float radius, RangeSearchResult result, IVFSearchParameters params, SWIGTYPE_p_unsigned_long nb_dis) {\n    swigfaissJNI.range_search_with_parameters__SWIG_1(Index.getCPtr(index), index, n, SWIGTYPE_p_float.getCPtr(x), radius, RangeSearchResult.getCPtr(result), result, IVFSearchParameters.getCPtr(params), params, SWIGTYPE_p_unsigned_long.getCPtr(nb_dis));\n  }\n\n  public static void range_search_with_parameters(Index index, long n, SWIGTYPE_p_float x, float radius, RangeSearchResult result, IVFSearchParameters params) {\n    swigfaissJNI.range_search_with_parameters__SWIG_2(Index.getCPtr(index), index, n, SWIGTYPE_p_float.getCPtr(x), radius, RangeSearchResult.getCPtr(result), result, IVFSearchParameters.getCPtr(params), params);\n  }\n\n  public static void setHnsw_stats(HNSWStats value) {\n    swigfaissJNI.hnsw_stats_set(HNSWStats.getCPtr(value), value);\n  }\n\n  public static HNSWStats getHnsw_stats() {\n    long cPtr = swigfaissJNI.hnsw_stats_get();\n    return (cPtr == 0) ? null : new HNSWStats(cPtr, false);\n  }\n\n  public static void setPrecomputed_table_max_bytes(long value) {\n    swigfaissJNI.precomputed_table_max_bytes_set(value);\n  }\n\n  public static long getPrecomputed_table_max_bytes() {\n    return swigfaissJNI.precomputed_table_max_bytes_get();\n  }\n\n  public static void initialize_IVFPQ_precomputed_table(SWIGTYPE_p_int use_precomputed_table, Index quantizer, ProductQuantizer pq, SWIGTYPE_p_AlignedTableT_float_32_t precomputed_table, boolean verbose) {\n    swigfaissJNI.initialize_IVFPQ_precomputed_table(SWIGTYPE_p_int.getCPtr(use_precomputed_table), Index.getCPtr(quantizer), quantizer, ProductQuantizer.getCPtr(pq), pq, SWIGTYPE_p_AlignedTableT_float_32_t.getCPtr(precomputed_table), verbose);\n  }\n\n  public static void setIndexIVFPQ_stats(IndexIVFPQStats value) {\n    swigfaissJNI.indexIVFPQ_stats_set(IndexIVFPQStats.getCPtr(value), value);\n  }\n\n  public static IndexIVFPQStats getIndexIVFPQ_stats() {\n    long cPtr = swigfaissJNI.indexIVFPQ_stats_get();\n    return (cPtr == 0) ? null : new IndexIVFPQStats(cPtr, false);\n  }\n\n  public static Index downcast_index(Index index) {\n    long cPtr = swigfaissJNI.downcast_index(Index.getCPtr(index), index);\n    return (cPtr == 0) ? null : new Index(cPtr, false);\n  }\n\n  public static VectorTransform downcast_VectorTransform(VectorTransform vt) {\n    long cPtr = swigfaissJNI.downcast_VectorTransform(VectorTransform.getCPtr(vt), vt);\n    return (cPtr == 0) ? null : new VectorTransform(cPtr, false);\n  }\n\n  public static IndexBinary downcast_IndexBinary(IndexBinary index) {\n    long cPtr = swigfaissJNI.downcast_IndexBinary(IndexBinary.getCPtr(index), index);\n    return (cPtr == 0) ? null : new IndexBinary(cPtr, false);\n  }\n\n  public static Index upcast_IndexShards(IndexShards index) {\n    long cPtr = swigfaissJNI.upcast_IndexShards(IndexShards.getCPtr(index), index);\n    return (cPtr == 0) ? null : new Index(cPtr, false);\n  }\n\n  public static void write_index(Index idx, String fname) {\n    swigfaissJNI.write_index__SWIG_0(Index.getCPtr(idx), idx, fname);\n  }\n\n  public static void write_index(Index idx, SWIGTYPE_p_FILE f) {\n    swigfaissJNI.write_index__SWIG_1(Index.getCPtr(idx), idx, SWIGTYPE_p_FILE.getCPtr(f));\n  }\n\n  public static void write_index(Index idx, SWIGTYPE_p_faiss__IOWriter writer) {\n    swigfaissJNI.write_index__SWIG_2(Index.getCPtr(idx), idx, SWIGTYPE_p_faiss__IOWriter.getCPtr(writer));\n  }\n\n  public static void write_index_binary(IndexBinary idx, String fname) {\n    swigfaissJNI.write_index_binary__SWIG_0(IndexBinary.getCPtr(idx), idx, fname);\n  }\n\n  public static void write_index_binary(IndexBinary idx, SWIGTYPE_p_FILE f) {\n    swigfaissJNI.write_index_binary__SWIG_1(IndexBinary.getCPtr(idx), idx, SWIGTYPE_p_FILE.getCPtr(f));\n  }\n\n  public static void write_index_binary(IndexBinary idx, SWIGTYPE_p_faiss__IOWriter writer) {\n    swigfaissJNI.write_index_binary__SWIG_2(IndexBinary.getCPtr(idx), idx, SWIGTYPE_p_faiss__IOWriter.getCPtr(writer));\n  }\n\n  public static int getIO_FLAG_READ_ONLY() {\n    return swigfaissJNI.IO_FLAG_READ_ONLY_get();\n  }\n\n  public static int getIO_FLAG_ONDISK_SAME_DIR() {\n    return swigfaissJNI.IO_FLAG_ONDISK_SAME_DIR_get();\n  }\n\n  public static int getIO_FLAG_SKIP_IVF_DATA() {\n    return swigfaissJNI.IO_FLAG_SKIP_IVF_DATA_get();\n  }\n\n  public static int getIO_FLAG_MMAP() {\n    return swigfaissJNI.IO_FLAG_MMAP_get();\n  }\n\n  public static Index read_index(String fname, int io_flags) {\n    long cPtr = swigfaissJNI.read_index__SWIG_0(fname, io_flags);\n    return (cPtr == 0) ? null : new Index(cPtr, true);\n  }\n\n  public static Index read_index(String fname) {\n    long cPtr = swigfaissJNI.read_index__SWIG_1(fname);\n    return (cPtr == 0) ? null : new Index(cPtr, true);\n  }\n\n  public static Index read_index(SWIGTYPE_p_FILE f, int io_flags) {\n    long cPtr = swigfaissJNI.read_index__SWIG_2(SWIGTYPE_p_FILE.getCPtr(f), io_flags);\n    return (cPtr == 0) ? null : new Index(cPtr, true);\n  }\n\n  public static Index read_index(SWIGTYPE_p_FILE f) {\n    long cPtr = swigfaissJNI.read_index__SWIG_3(SWIGTYPE_p_FILE.getCPtr(f));\n    return (cPtr == 0) ? null : new Index(cPtr, true);\n  }\n\n  public static Index read_index(SWIGTYPE_p_faiss__IOReader reader, int io_flags) {\n    long cPtr = swigfaissJNI.read_index__SWIG_4(SWIGTYPE_p_faiss__IOReader.getCPtr(reader), io_flags);\n    return (cPtr == 0) ? null : new Index(cPtr, true);\n  }\n\n  public static Index read_index(SWIGTYPE_p_faiss__IOReader reader) {\n    long cPtr = swigfaissJNI.read_index__SWIG_5(SWIGTYPE_p_faiss__IOReader.getCPtr(reader));\n    return (cPtr == 0) ? null : new Index(cPtr, true);\n  }\n\n  public static IndexBinary read_index_binary(String fname, int io_flags) {\n    long cPtr = swigfaissJNI.read_index_binary__SWIG_0(fname, io_flags);\n    return (cPtr == 0) ? null : new IndexBinary(cPtr, true);\n  }\n\n  public static IndexBinary read_index_binary(String fname) {\n    long cPtr = swigfaissJNI.read_index_binary__SWIG_1(fname);\n    return (cPtr == 0) ? null : new IndexBinary(cPtr, true);\n  }\n\n  public static IndexBinary read_index_binary(SWIGTYPE_p_FILE f, int io_flags) {\n    long cPtr = swigfaissJNI.read_index_binary__SWIG_2(SWIGTYPE_p_FILE.getCPtr(f), io_flags);\n    return (cPtr == 0) ? null : new IndexBinary(cPtr, true);\n  }\n\n  public static IndexBinary read_index_binary(SWIGTYPE_p_FILE f) {\n    long cPtr = swigfaissJNI.read_index_binary__SWIG_3(SWIGTYPE_p_FILE.getCPtr(f));\n    return (cPtr == 0) ? null : new IndexBinary(cPtr, true);\n  }\n\n  public static IndexBinary read_index_binary(SWIGTYPE_p_faiss__IOReader reader, int io_flags) {\n    long cPtr = swigfaissJNI.read_index_binary__SWIG_4(SWIGTYPE_p_faiss__IOReader.getCPtr(reader), io_flags);\n    return (cPtr == 0) ? null : new IndexBinary(cPtr, true);\n  }\n\n  public static IndexBinary read_index_binary(SWIGTYPE_p_faiss__IOReader reader) {\n    long cPtr = swigfaissJNI.read_index_binary__SWIG_5(SWIGTYPE_p_faiss__IOReader.getCPtr(reader));\n    return (cPtr == 0) ? null : new IndexBinary(cPtr, true);\n  }\n\n  public static void write_VectorTransform(VectorTransform vt, String fname) {\n    swigfaissJNI.write_VectorTransform(VectorTransform.getCPtr(vt), vt, fname);\n  }\n\n  public static VectorTransform read_VectorTransform(String fname) {\n    long cPtr = swigfaissJNI.read_VectorTransform(fname);\n    return (cPtr == 0) ? null : new VectorTransform(cPtr, true);\n  }\n\n  public static ProductQuantizer read_ProductQuantizer(String fname) {\n    long cPtr = swigfaissJNI.read_ProductQuantizer__SWIG_0(fname);\n    return (cPtr == 0) ? null : new ProductQuantizer(cPtr, true);\n  }\n\n  public static ProductQuantizer read_ProductQuantizer(SWIGTYPE_p_faiss__IOReader reader) {\n    long cPtr = swigfaissJNI.read_ProductQuantizer__SWIG_1(SWIGTYPE_p_faiss__IOReader.getCPtr(reader));\n    return (cPtr == 0) ? null : new ProductQuantizer(cPtr, true);\n  }\n\n  public static void write_ProductQuantizer(ProductQuantizer pq, String fname) {\n    swigfaissJNI.write_ProductQuantizer__SWIG_0(ProductQuantizer.getCPtr(pq), pq, fname);\n  }\n\n  public static void write_ProductQuantizer(ProductQuantizer pq, SWIGTYPE_p_faiss__IOWriter f) {\n    swigfaissJNI.write_ProductQuantizer__SWIG_1(ProductQuantizer.getCPtr(pq), pq, SWIGTYPE_p_faiss__IOWriter.getCPtr(f));\n  }\n\n  public static void write_InvertedLists(InvertedLists ils, SWIGTYPE_p_faiss__IOWriter f) {\n    swigfaissJNI.write_InvertedLists(InvertedLists.getCPtr(ils), ils, SWIGTYPE_p_faiss__IOWriter.getCPtr(f));\n  }\n\n  public static InvertedLists read_InvertedLists(SWIGTYPE_p_faiss__IOReader reader, int io_flags) {\n    long cPtr = swigfaissJNI.read_InvertedLists__SWIG_0(SWIGTYPE_p_faiss__IOReader.getCPtr(reader), io_flags);\n    return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n  }\n\n  public static InvertedLists read_InvertedLists(SWIGTYPE_p_faiss__IOReader reader) {\n    long cPtr = swigfaissJNI.read_InvertedLists__SWIG_1(SWIGTYPE_p_faiss__IOReader.getCPtr(reader));\n    return (cPtr == 0) ? null : new InvertedLists(cPtr, false);\n  }\n\n  public static Index index_factory(int d, String description, MetricType metric) {\n    long cPtr = swigfaissJNI.index_factory__SWIG_0(d, description, metric.swigValue());\n    return (cPtr == 0) ? null : new Index(cPtr, true);\n  }\n\n  public static Index index_factory(int d, String description) {\n    long cPtr = swigfaissJNI.index_factory__SWIG_1(d, description);\n    return (cPtr == 0) ? null : new Index(cPtr, true);\n  }\n\n  public static void setIndex_factory_verbose(int value) {\n    swigfaissJNI.index_factory_verbose_set(value);\n  }\n\n  public static int getIndex_factory_verbose() {\n    return swigfaissJNI.index_factory_verbose_get();\n  }\n\n  public static IndexBinary index_binary_factory(int d, String description) {\n    long cPtr = swigfaissJNI.index_binary_factory(d, description);\n    return (cPtr == 0) ? null : new IndexBinary(cPtr, true);\n  }\n\n  public static void simd_histogram_8(SWIGTYPE_p_uint16_t data, int n, SWIGTYPE_p_uint16_t min, int shift, SWIGTYPE_p_int hist) {\n    swigfaissJNI.simd_histogram_8(SWIGTYPE_p_uint16_t.getCPtr(data), n, SWIGTYPE_p_uint16_t.getCPtr(min), shift, SWIGTYPE_p_int.getCPtr(hist));\n  }\n\n  public static void simd_histogram_16(SWIGTYPE_p_uint16_t data, int n, SWIGTYPE_p_uint16_t min, int shift, SWIGTYPE_p_int hist) {\n    swigfaissJNI.simd_histogram_16(SWIGTYPE_p_uint16_t.getCPtr(data), n, SWIGTYPE_p_uint16_t.getCPtr(min), shift, SWIGTYPE_p_int.getCPtr(hist));\n  }\n\n  public static void setPartition_stats(PartitionStats value) {\n    swigfaissJNI.partition_stats_set(PartitionStats.getCPtr(value), value);\n  }\n\n  public static PartitionStats getPartition_stats() {\n    long cPtr = swigfaissJNI.partition_stats_get();\n    return (cPtr == 0) ? null : new PartitionStats(cPtr, false);\n  }\n\n  public static float CMin_float_partition_fuzzy(SWIGTYPE_p_float vals, LongVector ids, long n, long q_min, long q_max, SWIGTYPE_p_unsigned_long q_out) {\n    return swigfaissJNI.CMin_float_partition_fuzzy(SWIGTYPE_p_float.getCPtr(vals), SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, n, q_min, q_max, SWIGTYPE_p_unsigned_long.getCPtr(q_out));\n  }\n\n  public static float CMax_float_partition_fuzzy(SWIGTYPE_p_float vals, LongVector ids, long n, long q_min, long q_max, SWIGTYPE_p_unsigned_long q_out) {\n    return swigfaissJNI.CMax_float_partition_fuzzy(SWIGTYPE_p_float.getCPtr(vals), SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, n, q_min, q_max, SWIGTYPE_p_unsigned_long.getCPtr(q_out));\n  }\n\n  public static SWIGTYPE_p_uint16_t CMax_uint16_partition_fuzzy(SWIGTYPE_p_uint16_t vals, LongVector ids, long n, long q_min, long q_max, SWIGTYPE_p_unsigned_long q_out) {\n    return new SWIGTYPE_p_uint16_t(swigfaissJNI.CMax_uint16_partition_fuzzy__SWIG_0(SWIGTYPE_p_uint16_t.getCPtr(vals), SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, n, q_min, q_max, SWIGTYPE_p_unsigned_long.getCPtr(q_out)), true);\n  }\n\n  public static SWIGTYPE_p_uint16_t CMin_uint16_partition_fuzzy(SWIGTYPE_p_uint16_t vals, LongVector ids, long n, long q_min, long q_max, SWIGTYPE_p_unsigned_long q_out) {\n    return new SWIGTYPE_p_uint16_t(swigfaissJNI.CMin_uint16_partition_fuzzy__SWIG_0(SWIGTYPE_p_uint16_t.getCPtr(vals), SWIGTYPE_p_long_long.getCPtr(ids.data()), ids, n, q_min, q_max, SWIGTYPE_p_unsigned_long.getCPtr(q_out)), true);\n  }\n\n  public static SWIGTYPE_p_uint16_t CMax_uint16_partition_fuzzy(SWIGTYPE_p_uint16_t vals, SWIGTYPE_p_int ids, long n, long q_min, long q_max, SWIGTYPE_p_unsigned_long q_out) {\n    return new SWIGTYPE_p_uint16_t(swigfaissJNI.CMax_uint16_partition_fuzzy__SWIG_1(SWIGTYPE_p_uint16_t.getCPtr(vals), SWIGTYPE_p_int.getCPtr(ids), n, q_min, q_max, SWIGTYPE_p_unsigned_long.getCPtr(q_out)), true);\n  }\n\n  public static SWIGTYPE_p_uint16_t CMin_uint16_partition_fuzzy(SWIGTYPE_p_uint16_t vals, SWIGTYPE_p_int ids, long n, long q_min, long q_max, SWIGTYPE_p_unsigned_long q_out) {\n    return new SWIGTYPE_p_uint16_t(swigfaissJNI.CMin_uint16_partition_fuzzy__SWIG_1(SWIGTYPE_p_uint16_t.getCPtr(vals), SWIGTYPE_p_int.getCPtr(ids), n, q_min, q_max, SWIGTYPE_p_unsigned_long.getCPtr(q_out)), true);\n  }\n\n  public static void omp_set_num_threads(int num_threads) {\n    swigfaissJNI.omp_set_num_threads(num_threads);\n  }\n\n  public static int omp_get_max_threads() {\n    return swigfaissJNI.omp_get_max_threads();\n  }\n\n  public static SWIGTYPE_p_void memcpy(SWIGTYPE_p_void dest, SWIGTYPE_p_void src, long n) {\n    long cPtr = swigfaissJNI.memcpy(SWIGTYPE_p_void.getCPtr(dest), SWIGTYPE_p_void.getCPtr(src), n);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_void(cPtr, false);\n  }\n\n  public static SWIGTYPE_p_float cast_integer_to_float_ptr(int x) {\n    long cPtr = swigfaissJNI.cast_integer_to_float_ptr(x);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_float(cPtr, false);\n  }\n\n  public static SWIGTYPE_p_long cast_integer_to_long_ptr(int x) {\n    long cPtr = swigfaissJNI.cast_integer_to_long_ptr(x);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_long(cPtr, false);\n  }\n\n  public static SWIGTYPE_p_int cast_integer_to_int_ptr(int x) {\n    long cPtr = swigfaissJNI.cast_integer_to_int_ptr(x);\n    return (cPtr == 0) ? null : new SWIGTYPE_p_int(cPtr, false);\n  }\n\n  public static void ignore_SIGTTIN() {\n    swigfaissJNI.ignore_SIGTTIN();\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/swigfaissConstants.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\n\npublic interface swigfaissConstants {\n  public final static int FAISS_VERSION_MAJOR = swigfaissJNI.FAISS_VERSION_MAJOR_get();\n  public final static int FAISS_VERSION_MINOR = swigfaissJNI.FAISS_VERSION_MINOR_get();\n  public final static int FAISS_VERSION_PATCH = swigfaissJNI.FAISS_VERSION_PATCH_get();\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/faiss/swig/swigfaissJNI.java",
    "content": "/* ----------------------------------------------------------------------------\n * This file was automatically generated by SWIG (http://www.swig.org).\n * Version 4.0.2\n *\n * Do not make changes to this file unless you know what you are doing--modify\n * the SWIG interface file instead.\n * ----------------------------------------------------------------------------- */\n\npackage com.twitter.ann.faiss;\nimport com.twitter.ann.faiss.NativeUtils;\npublic class swigfaissJNI {\n\n  static {\n    try {\n      if (NativeUtils.getOperatingSystemType() == NativeUtils.OSType.MacOS) {\n        NativeUtils.loadLibraryFromJar(\"/com/twitter/ann/faiss/swig/resources/swigfaiss.dylib\");\n      } else {\n        NativeUtils.loadLibraryFromJar(\"/com/twitter/ann/faiss/swig/resources/libstdc++.so.6\");\n        NativeUtils.loadLibraryFromJar(\"/com/twitter/ann/faiss/swig/resources/libgcc_s.so.1\");\n        NativeUtils.loadLibraryFromJar(\"/com/twitter/ann/faiss/swig/resources/libgomp.so.1\");\n        NativeUtils.loadLibraryFromJar(\"/com/twitter/ann/faiss/swig/resources/libquadmath.so.0\");\n        NativeUtils.loadLibraryFromJar(\"/com/twitter/ann/faiss/swig/resources/libgfortran.so.5\");\n        NativeUtils.loadLibraryFromJar(\"/com/twitter/ann/faiss/swig/resources/swigfaiss.so\");\n      }\n    } catch (Exception e) {\n      System.err.println(\"Native code library failed to load. \\n\" + e);\n      System.exit(1);\n    }\n  }\n\n  public final static native long new_intArray(int jarg1);\n  public final static native void delete_intArray(long jarg1);\n  public final static native int intArray_getitem(long jarg1, intArray jarg1_, int jarg2);\n  public final static native void intArray_setitem(long jarg1, intArray jarg1_, int jarg2, int jarg3);\n  public final static native long intArray_cast(long jarg1, intArray jarg1_);\n  public final static native long intArray_frompointer(long jarg1);\n  public final static native long new_floatArray(int jarg1);\n  public final static native void delete_floatArray(long jarg1);\n  public final static native float floatArray_getitem(long jarg1, floatArray jarg1_, int jarg2);\n  public final static native void floatArray_setitem(long jarg1, floatArray jarg1_, int jarg2, float jarg3);\n  public final static native long floatArray_cast(long jarg1, floatArray jarg1_);\n  public final static native long floatArray_frompointer(long jarg1);\n  public final static native long new_longArray(int jarg1);\n  public final static native void delete_longArray(long jarg1);\n  public final static native long longArray_getitem(long jarg1, longArray jarg1_, int jarg2);\n  public final static native void longArray_setitem(long jarg1, longArray jarg1_, int jarg2, long jarg3);\n  public final static native long longArray_cast(long jarg1, longArray jarg1_);\n  public final static native long longArray_frompointer(long jarg1);\n  public final static native long new_doubleArray(int jarg1);\n  public final static native void delete_doubleArray(long jarg1);\n  public final static native double doubleArray_getitem(long jarg1, doubleArray jarg1_, int jarg2);\n  public final static native void doubleArray_setitem(long jarg1, doubleArray jarg1_, int jarg2, double jarg3);\n  public final static native long doubleArray_cast(long jarg1, doubleArray jarg1_);\n  public final static native long doubleArray_frompointer(long jarg1);\n  public final static native long new_FloatVector();\n  public final static native void FloatVector_push_back(long jarg1, FloatVector jarg1_, float jarg2);\n  public final static native void FloatVector_clear(long jarg1, FloatVector jarg1_);\n  public final static native long FloatVector_data(long jarg1, FloatVector jarg1_);\n  public final static native long FloatVector_size(long jarg1, FloatVector jarg1_);\n  public final static native float FloatVector_at(long jarg1, FloatVector jarg1_, long jarg2);\n  public final static native void FloatVector_resize(long jarg1, FloatVector jarg1_, long jarg2);\n  public final static native void FloatVector_reserve(long jarg1, FloatVector jarg1_, long jarg2);\n  public final static native void FloatVector_swap(long jarg1, FloatVector jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native void delete_FloatVector(long jarg1);\n  public final static native long new_DoubleVector();\n  public final static native void DoubleVector_push_back(long jarg1, DoubleVector jarg1_, double jarg2);\n  public final static native void DoubleVector_clear(long jarg1, DoubleVector jarg1_);\n  public final static native long DoubleVector_data(long jarg1, DoubleVector jarg1_);\n  public final static native long DoubleVector_size(long jarg1, DoubleVector jarg1_);\n  public final static native double DoubleVector_at(long jarg1, DoubleVector jarg1_, long jarg2);\n  public final static native void DoubleVector_resize(long jarg1, DoubleVector jarg1_, long jarg2);\n  public final static native void DoubleVector_reserve(long jarg1, DoubleVector jarg1_, long jarg2);\n  public final static native void DoubleVector_swap(long jarg1, DoubleVector jarg1_, long jarg2, DoubleVector jarg2_);\n  public final static native void delete_DoubleVector(long jarg1);\n  public final static native long new_ByteVector();\n  public final static native void ByteVector_push_back(long jarg1, ByteVector jarg1_, short jarg2);\n  public final static native void ByteVector_clear(long jarg1, ByteVector jarg1_);\n  public final static native long ByteVector_data(long jarg1, ByteVector jarg1_);\n  public final static native long ByteVector_size(long jarg1, ByteVector jarg1_);\n  public final static native short ByteVector_at(long jarg1, ByteVector jarg1_, long jarg2);\n  public final static native void ByteVector_resize(long jarg1, ByteVector jarg1_, long jarg2);\n  public final static native void ByteVector_reserve(long jarg1, ByteVector jarg1_, long jarg2);\n  public final static native void ByteVector_swap(long jarg1, ByteVector jarg1_, long jarg2, ByteVector jarg2_);\n  public final static native void delete_ByteVector(long jarg1);\n  public final static native long new_CharVector();\n  public final static native void CharVector_push_back(long jarg1, CharVector jarg1_, char jarg2);\n  public final static native void CharVector_clear(long jarg1, CharVector jarg1_);\n  public final static native String CharVector_data(long jarg1, CharVector jarg1_);\n  public final static native long CharVector_size(long jarg1, CharVector jarg1_);\n  public final static native char CharVector_at(long jarg1, CharVector jarg1_, long jarg2);\n  public final static native void CharVector_resize(long jarg1, CharVector jarg1_, long jarg2);\n  public final static native void CharVector_reserve(long jarg1, CharVector jarg1_, long jarg2);\n  public final static native void CharVector_swap(long jarg1, CharVector jarg1_, long jarg2, CharVector jarg2_);\n  public final static native void delete_CharVector(long jarg1);\n  public final static native long new_Uint64Vector();\n  public final static native void Uint64Vector_push_back(long jarg1, Uint64Vector jarg1_, long jarg2);\n  public final static native void Uint64Vector_clear(long jarg1, Uint64Vector jarg1_);\n  public final static native long Uint64Vector_data(long jarg1, Uint64Vector jarg1_);\n  public final static native long Uint64Vector_size(long jarg1, Uint64Vector jarg1_);\n  public final static native long Uint64Vector_at(long jarg1, Uint64Vector jarg1_, long jarg2);\n  public final static native void Uint64Vector_resize(long jarg1, Uint64Vector jarg1_, long jarg2);\n  public final static native void Uint64Vector_reserve(long jarg1, Uint64Vector jarg1_, long jarg2);\n  public final static native void Uint64Vector_swap(long jarg1, Uint64Vector jarg1_, long jarg2, Uint64Vector jarg2_);\n  public final static native void delete_Uint64Vector(long jarg1);\n  public final static native long new_LongVector();\n  public final static native void LongVector_push_back(long jarg1, LongVector jarg1_, long jarg2);\n  public final static native void LongVector_clear(long jarg1, LongVector jarg1_);\n  public final static native long LongVector_data(long jarg1, LongVector jarg1_);\n  public final static native long LongVector_size(long jarg1, LongVector jarg1_);\n  public final static native long LongVector_at(long jarg1, LongVector jarg1_, long jarg2);\n  public final static native void LongVector_resize(long jarg1, LongVector jarg1_, long jarg2);\n  public final static native void LongVector_reserve(long jarg1, LongVector jarg1_, long jarg2);\n  public final static native void LongVector_swap(long jarg1, LongVector jarg1_, long jarg2, LongVector jarg2_);\n  public final static native void delete_LongVector(long jarg1);\n  public final static native long new_IntVector();\n  public final static native void IntVector_push_back(long jarg1, IntVector jarg1_, int jarg2);\n  public final static native void IntVector_clear(long jarg1, IntVector jarg1_);\n  public final static native long IntVector_data(long jarg1, IntVector jarg1_);\n  public final static native long IntVector_size(long jarg1, IntVector jarg1_);\n  public final static native int IntVector_at(long jarg1, IntVector jarg1_, long jarg2);\n  public final static native void IntVector_resize(long jarg1, IntVector jarg1_, long jarg2);\n  public final static native void IntVector_reserve(long jarg1, IntVector jarg1_, long jarg2);\n  public final static native void IntVector_swap(long jarg1, IntVector jarg1_, long jarg2, IntVector jarg2_);\n  public final static native void delete_IntVector(long jarg1);\n  public final static native long new_VectorTransformVector();\n  public final static native void VectorTransformVector_push_back(long jarg1, VectorTransformVector jarg1_, long jarg2, VectorTransform jarg2_);\n  public final static native void VectorTransformVector_clear(long jarg1, VectorTransformVector jarg1_);\n  public final static native long VectorTransformVector_data(long jarg1, VectorTransformVector jarg1_);\n  public final static native long VectorTransformVector_size(long jarg1, VectorTransformVector jarg1_);\n  public final static native long VectorTransformVector_at(long jarg1, VectorTransformVector jarg1_, long jarg2);\n  public final static native void VectorTransformVector_resize(long jarg1, VectorTransformVector jarg1_, long jarg2);\n  public final static native void VectorTransformVector_reserve(long jarg1, VectorTransformVector jarg1_, long jarg2);\n  public final static native void VectorTransformVector_swap(long jarg1, VectorTransformVector jarg1_, long jarg2, VectorTransformVector jarg2_);\n  public final static native void delete_VectorTransformVector(long jarg1);\n  public final static native long new_OperatingPointVector();\n  public final static native void OperatingPointVector_push_back(long jarg1, OperatingPointVector jarg1_, long jarg2, OperatingPoint jarg2_);\n  public final static native void OperatingPointVector_clear(long jarg1, OperatingPointVector jarg1_);\n  public final static native long OperatingPointVector_data(long jarg1, OperatingPointVector jarg1_);\n  public final static native long OperatingPointVector_size(long jarg1, OperatingPointVector jarg1_);\n  public final static native long OperatingPointVector_at(long jarg1, OperatingPointVector jarg1_, long jarg2);\n  public final static native void OperatingPointVector_resize(long jarg1, OperatingPointVector jarg1_, long jarg2);\n  public final static native void OperatingPointVector_reserve(long jarg1, OperatingPointVector jarg1_, long jarg2);\n  public final static native void OperatingPointVector_swap(long jarg1, OperatingPointVector jarg1_, long jarg2, OperatingPointVector jarg2_);\n  public final static native void delete_OperatingPointVector(long jarg1);\n  public final static native long new_InvertedListsPtrVector();\n  public final static native void InvertedListsPtrVector_push_back(long jarg1, InvertedListsPtrVector jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native void InvertedListsPtrVector_clear(long jarg1, InvertedListsPtrVector jarg1_);\n  public final static native long InvertedListsPtrVector_data(long jarg1, InvertedListsPtrVector jarg1_);\n  public final static native long InvertedListsPtrVector_size(long jarg1, InvertedListsPtrVector jarg1_);\n  public final static native long InvertedListsPtrVector_at(long jarg1, InvertedListsPtrVector jarg1_, long jarg2);\n  public final static native void InvertedListsPtrVector_resize(long jarg1, InvertedListsPtrVector jarg1_, long jarg2);\n  public final static native void InvertedListsPtrVector_reserve(long jarg1, InvertedListsPtrVector jarg1_, long jarg2);\n  public final static native void InvertedListsPtrVector_swap(long jarg1, InvertedListsPtrVector jarg1_, long jarg2, InvertedListsPtrVector jarg2_);\n  public final static native void delete_InvertedListsPtrVector(long jarg1);\n  public final static native long new_FloatVectorVector();\n  public final static native void FloatVectorVector_push_back(long jarg1, FloatVectorVector jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native void FloatVectorVector_clear(long jarg1, FloatVectorVector jarg1_);\n  public final static native long FloatVectorVector_data(long jarg1, FloatVectorVector jarg1_);\n  public final static native long FloatVectorVector_size(long jarg1, FloatVectorVector jarg1_);\n  public final static native long FloatVectorVector_at(long jarg1, FloatVectorVector jarg1_, long jarg2);\n  public final static native void FloatVectorVector_resize(long jarg1, FloatVectorVector jarg1_, long jarg2);\n  public final static native void FloatVectorVector_reserve(long jarg1, FloatVectorVector jarg1_, long jarg2);\n  public final static native void FloatVectorVector_swap(long jarg1, FloatVectorVector jarg1_, long jarg2, FloatVectorVector jarg2_);\n  public final static native void delete_FloatVectorVector(long jarg1);\n  public final static native long new_ByteVectorVector();\n  public final static native void ByteVectorVector_push_back(long jarg1, ByteVectorVector jarg1_, long jarg2, ByteVector jarg2_);\n  public final static native void ByteVectorVector_clear(long jarg1, ByteVectorVector jarg1_);\n  public final static native long ByteVectorVector_data(long jarg1, ByteVectorVector jarg1_);\n  public final static native long ByteVectorVector_size(long jarg1, ByteVectorVector jarg1_);\n  public final static native long ByteVectorVector_at(long jarg1, ByteVectorVector jarg1_, long jarg2);\n  public final static native void ByteVectorVector_resize(long jarg1, ByteVectorVector jarg1_, long jarg2);\n  public final static native void ByteVectorVector_reserve(long jarg1, ByteVectorVector jarg1_, long jarg2);\n  public final static native void ByteVectorVector_swap(long jarg1, ByteVectorVector jarg1_, long jarg2, ByteVectorVector jarg2_);\n  public final static native void delete_ByteVectorVector(long jarg1);\n  public final static native long new_LongVectorVector();\n  public final static native void LongVectorVector_push_back(long jarg1, LongVectorVector jarg1_, long jarg2);\n  public final static native void LongVectorVector_clear(long jarg1, LongVectorVector jarg1_);\n  public final static native long LongVectorVector_data(long jarg1, LongVectorVector jarg1_);\n  public final static native long LongVectorVector_size(long jarg1, LongVectorVector jarg1_);\n  public final static native long LongVectorVector_at(long jarg1, LongVectorVector jarg1_, long jarg2);\n  public final static native void LongVectorVector_resize(long jarg1, LongVectorVector jarg1_, long jarg2);\n  public final static native void LongVectorVector_reserve(long jarg1, LongVectorVector jarg1_, long jarg2);\n  public final static native void LongVectorVector_swap(long jarg1, LongVectorVector jarg1_, long jarg2, LongVectorVector jarg2_);\n  public final static native void delete_LongVectorVector(long jarg1);\n  public final static native void bitvec_print(long jarg1, long jarg2);\n  public final static native void fvecs2bitvecs(long jarg1, long jarg2, long jarg3, long jarg4);\n  public final static native void bitvecs2fvecs(long jarg1, long jarg2, long jarg3, long jarg4);\n  public final static native void fvec2bitvec(long jarg1, long jarg2, long jarg3);\n  public final static native void bitvec_shuffle(long jarg1, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native void BitstringWriter_code_set(long jarg1, BitstringWriter jarg1_, long jarg2);\n  public final static native long BitstringWriter_code_get(long jarg1, BitstringWriter jarg1_);\n  public final static native void BitstringWriter_code_size_set(long jarg1, BitstringWriter jarg1_, long jarg2);\n  public final static native long BitstringWriter_code_size_get(long jarg1, BitstringWriter jarg1_);\n  public final static native void BitstringWriter_i_set(long jarg1, BitstringWriter jarg1_, long jarg2);\n  public final static native long BitstringWriter_i_get(long jarg1, BitstringWriter jarg1_);\n  public final static native long new_BitstringWriter(long jarg1, long jarg2);\n  public final static native void BitstringWriter_write(long jarg1, BitstringWriter jarg1_, long jarg2, int jarg3);\n  public final static native void delete_BitstringWriter(long jarg1);\n  public final static native void BitstringReader_code_set(long jarg1, BitstringReader jarg1_, long jarg2);\n  public final static native long BitstringReader_code_get(long jarg1, BitstringReader jarg1_);\n  public final static native void BitstringReader_code_size_set(long jarg1, BitstringReader jarg1_, long jarg2);\n  public final static native long BitstringReader_code_size_get(long jarg1, BitstringReader jarg1_);\n  public final static native void BitstringReader_i_set(long jarg1, BitstringReader jarg1_, long jarg2);\n  public final static native long BitstringReader_i_get(long jarg1, BitstringReader jarg1_);\n  public final static native long new_BitstringReader(long jarg1, long jarg2);\n  public final static native long BitstringReader_read(long jarg1, BitstringReader jarg1_, int jarg2);\n  public final static native void delete_BitstringReader(long jarg1);\n  public final static native void hamming_batch_size_set(long jarg1);\n  public final static native long hamming_batch_size_get();\n  public final static native int popcount64(long jarg1);\n  public final static native void hammings(long jarg1, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native void hammings_knn_hc(long jarg1, long jarg2, long jarg3, long jarg4, long jarg5, int jarg6);\n  public final static native void hammings_knn(long jarg1, long jarg2, long jarg3, long jarg4, long jarg5, int jarg6);\n  public final static native void hammings_knn_mc(long jarg1, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, long jarg7, long jarg8, LongVector jarg8_);\n  public final static native void hamming_range_search(long jarg1, long jarg2, long jarg3, long jarg4, int jarg5, long jarg6, long jarg7, RangeSearchResult jarg7_);\n  public final static native void hamming_count_thres(long jarg1, long jarg2, long jarg3, long jarg4, int jarg5, long jarg6, long jarg7);\n  public final static native long match_hamming_thres(long jarg1, long jarg2, long jarg3, long jarg4, int jarg5, long jarg6, long jarg7, LongVector jarg7_, long jarg8);\n  public final static native void crosshamming_count_thres(long jarg1, long jarg2, int jarg3, long jarg4, long jarg5);\n  public final static native int get_num_gpus();\n  public final static native int METRIC_INNER_PRODUCT_get();\n  public final static native int METRIC_L2_get();\n  public final static native int METRIC_Canberra_get();\n  public final static native String get_compile_options();\n  public final static native double getmillisecs();\n  public final static native long get_mem_usage_kb();\n  public final static native long get_cycles();\n  public final static native void fvec_madd(long jarg1, long jarg2, float jarg3, long jarg4, long jarg5);\n  public final static native int fvec_madd_and_argmin(long jarg1, long jarg2, float jarg3, long jarg4, long jarg5);\n  public final static native void reflection(long jarg1, long jarg2, long jarg3, long jarg4, long jarg5);\n  public final static native void matrix_qr(int jarg1, int jarg2, long jarg3);\n  public final static native void ranklist_handle_ties(int jarg1, long jarg2, LongVector jarg2_, long jarg3);\n  public final static native long ranklist_intersection_size(long jarg1, long jarg2, LongVector jarg2_, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native long merge_result_table_with__SWIG_0(long jarg1, long jarg2, long jarg3, LongVector jarg3_, long jarg4, long jarg5, LongVector jarg5_, long jarg6, boolean jarg7, long jarg8);\n  public final static native long merge_result_table_with__SWIG_1(long jarg1, long jarg2, long jarg3, LongVector jarg3_, long jarg4, long jarg5, LongVector jarg5_, long jarg6, boolean jarg7);\n  public final static native long merge_result_table_with__SWIG_2(long jarg1, long jarg2, long jarg3, LongVector jarg3_, long jarg4, long jarg5, LongVector jarg5_, long jarg6);\n  public final static native double imbalance_factor__SWIG_0(int jarg1, int jarg2, long jarg3, LongVector jarg3_);\n  public final static native double imbalance_factor__SWIG_1(int jarg1, long jarg2);\n  public final static native void fvec_argsort(long jarg1, long jarg2, long jarg3);\n  public final static native void fvec_argsort_parallel(long jarg1, long jarg2, long jarg3);\n  public final static native int ivec_hist(long jarg1, long jarg2, int jarg3, long jarg4);\n  public final static native void bincode_hist(long jarg1, long jarg2, long jarg3, long jarg4);\n  public final static native long ivec_checksum(long jarg1, long jarg2);\n  public final static native long fvecs_maybe_subsample__SWIG_0(long jarg1, long jarg2, long jarg3, long jarg4, boolean jarg5, long jarg6);\n  public final static native long fvecs_maybe_subsample__SWIG_1(long jarg1, long jarg2, long jarg3, long jarg4, boolean jarg5);\n  public final static native long fvecs_maybe_subsample__SWIG_2(long jarg1, long jarg2, long jarg3, long jarg4);\n  public final static native void binary_to_real(long jarg1, long jarg2, long jarg3);\n  public final static native void real_to_binary(long jarg1, long jarg2, long jarg3);\n  public final static native long hash_bytes(long jarg1, long jarg2);\n  public final static native boolean check_openmp();\n  public final static native int FAISS_VERSION_MAJOR_get();\n  public final static native int FAISS_VERSION_MINOR_get();\n  public final static native int FAISS_VERSION_PATCH_get();\n  public final static native void Index_d_set(long jarg1, Index jarg1_, int jarg2);\n  public final static native int Index_d_get(long jarg1, Index jarg1_);\n  public final static native void Index_ntotal_set(long jarg1, Index jarg1_, long jarg2);\n  public final static native long Index_ntotal_get(long jarg1, Index jarg1_);\n  public final static native void Index_verbose_set(long jarg1, Index jarg1_, boolean jarg2);\n  public final static native boolean Index_verbose_get(long jarg1, Index jarg1_);\n  public final static native void Index_is_trained_set(long jarg1, Index jarg1_, boolean jarg2);\n  public final static native boolean Index_is_trained_get(long jarg1, Index jarg1_);\n  public final static native void Index_metric_type_set(long jarg1, Index jarg1_, int jarg2);\n  public final static native int Index_metric_type_get(long jarg1, Index jarg1_);\n  public final static native void Index_metric_arg_set(long jarg1, Index jarg1_, float jarg2);\n  public final static native float Index_metric_arg_get(long jarg1, Index jarg1_);\n  public final static native void delete_Index(long jarg1);\n  public final static native void Index_train(long jarg1, Index jarg1_, long jarg2, long jarg3);\n  public final static native void Index_add(long jarg1, Index jarg1_, long jarg2, long jarg3);\n  public final static native void Index_add_with_ids(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void Index_search(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void Index_range_search(long jarg1, Index jarg1_, long jarg2, long jarg3, float jarg4, long jarg5, RangeSearchResult jarg5_);\n  public final static native void Index_assign__SWIG_0(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void Index_assign__SWIG_1(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void Index_reset(long jarg1, Index jarg1_);\n  public final static native long Index_remove_ids(long jarg1, Index jarg1_, long jarg2, IDSelector jarg2_);\n  public final static native void Index_reconstruct(long jarg1, Index jarg1_, long jarg2, long jarg3);\n  public final static native void Index_reconstruct_n(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void Index_search_and_reconstruct(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_, long jarg7);\n  public final static native void Index_compute_residual(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void Index_compute_residual_n(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_);\n  public final static native long Index_get_distance_computer(long jarg1, Index jarg1_);\n  public final static native long Index_sa_code_size(long jarg1, Index jarg1_);\n  public final static native void Index_sa_encode(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void Index_sa_decode(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native long Index_toIVF(long jarg1, Index jarg1_);\n  public final static native void ClusteringParameters_niter_set(long jarg1, ClusteringParameters jarg1_, int jarg2);\n  public final static native int ClusteringParameters_niter_get(long jarg1, ClusteringParameters jarg1_);\n  public final static native void ClusteringParameters_nredo_set(long jarg1, ClusteringParameters jarg1_, int jarg2);\n  public final static native int ClusteringParameters_nredo_get(long jarg1, ClusteringParameters jarg1_);\n  public final static native void ClusteringParameters_verbose_set(long jarg1, ClusteringParameters jarg1_, boolean jarg2);\n  public final static native boolean ClusteringParameters_verbose_get(long jarg1, ClusteringParameters jarg1_);\n  public final static native void ClusteringParameters_spherical_set(long jarg1, ClusteringParameters jarg1_, boolean jarg2);\n  public final static native boolean ClusteringParameters_spherical_get(long jarg1, ClusteringParameters jarg1_);\n  public final static native void ClusteringParameters_int_centroids_set(long jarg1, ClusteringParameters jarg1_, boolean jarg2);\n  public final static native boolean ClusteringParameters_int_centroids_get(long jarg1, ClusteringParameters jarg1_);\n  public final static native void ClusteringParameters_update_index_set(long jarg1, ClusteringParameters jarg1_, boolean jarg2);\n  public final static native boolean ClusteringParameters_update_index_get(long jarg1, ClusteringParameters jarg1_);\n  public final static native void ClusteringParameters_frozen_centroids_set(long jarg1, ClusteringParameters jarg1_, boolean jarg2);\n  public final static native boolean ClusteringParameters_frozen_centroids_get(long jarg1, ClusteringParameters jarg1_);\n  public final static native void ClusteringParameters_min_points_per_centroid_set(long jarg1, ClusteringParameters jarg1_, int jarg2);\n  public final static native int ClusteringParameters_min_points_per_centroid_get(long jarg1, ClusteringParameters jarg1_);\n  public final static native void ClusteringParameters_max_points_per_centroid_set(long jarg1, ClusteringParameters jarg1_, int jarg2);\n  public final static native int ClusteringParameters_max_points_per_centroid_get(long jarg1, ClusteringParameters jarg1_);\n  public final static native void ClusteringParameters_seed_set(long jarg1, ClusteringParameters jarg1_, int jarg2);\n  public final static native int ClusteringParameters_seed_get(long jarg1, ClusteringParameters jarg1_);\n  public final static native void ClusteringParameters_decode_block_size_set(long jarg1, ClusteringParameters jarg1_, long jarg2);\n  public final static native long ClusteringParameters_decode_block_size_get(long jarg1, ClusteringParameters jarg1_);\n  public final static native long new_ClusteringParameters();\n  public final static native void delete_ClusteringParameters(long jarg1);\n  public final static native void ClusteringIterationStats_obj_set(long jarg1, ClusteringIterationStats jarg1_, float jarg2);\n  public final static native float ClusteringIterationStats_obj_get(long jarg1, ClusteringIterationStats jarg1_);\n  public final static native void ClusteringIterationStats_time_set(long jarg1, ClusteringIterationStats jarg1_, double jarg2);\n  public final static native double ClusteringIterationStats_time_get(long jarg1, ClusteringIterationStats jarg1_);\n  public final static native void ClusteringIterationStats_time_search_set(long jarg1, ClusteringIterationStats jarg1_, double jarg2);\n  public final static native double ClusteringIterationStats_time_search_get(long jarg1, ClusteringIterationStats jarg1_);\n  public final static native void ClusteringIterationStats_imbalance_factor_set(long jarg1, ClusteringIterationStats jarg1_, double jarg2);\n  public final static native double ClusteringIterationStats_imbalance_factor_get(long jarg1, ClusteringIterationStats jarg1_);\n  public final static native void ClusteringIterationStats_nsplit_set(long jarg1, ClusteringIterationStats jarg1_, int jarg2);\n  public final static native int ClusteringIterationStats_nsplit_get(long jarg1, ClusteringIterationStats jarg1_);\n  public final static native long new_ClusteringIterationStats();\n  public final static native void delete_ClusteringIterationStats(long jarg1);\n  public final static native void Clustering_d_set(long jarg1, Clustering jarg1_, long jarg2);\n  public final static native long Clustering_d_get(long jarg1, Clustering jarg1_);\n  public final static native void Clustering_k_set(long jarg1, Clustering jarg1_, long jarg2);\n  public final static native long Clustering_k_get(long jarg1, Clustering jarg1_);\n  public final static native void Clustering_centroids_set(long jarg1, Clustering jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long Clustering_centroids_get(long jarg1, Clustering jarg1_);\n  public final static native void Clustering_iteration_stats_set(long jarg1, Clustering jarg1_, long jarg2);\n  public final static native long Clustering_iteration_stats_get(long jarg1, Clustering jarg1_);\n  public final static native long new_Clustering__SWIG_0(int jarg1, int jarg2);\n  public final static native long new_Clustering__SWIG_1(int jarg1, int jarg2, long jarg3, ClusteringParameters jarg3_);\n  public final static native void Clustering_train__SWIG_0(long jarg1, Clustering jarg1_, long jarg2, long jarg3, long jarg4, Index jarg4_, long jarg5);\n  public final static native void Clustering_train__SWIG_1(long jarg1, Clustering jarg1_, long jarg2, long jarg3, long jarg4, Index jarg4_);\n  public final static native void Clustering_train_encoded__SWIG_0(long jarg1, Clustering jarg1_, long jarg2, long jarg3, long jarg4, Index jarg4_, long jarg5, Index jarg5_, long jarg6);\n  public final static native void Clustering_train_encoded__SWIG_1(long jarg1, Clustering jarg1_, long jarg2, long jarg3, long jarg4, Index jarg4_, long jarg5, Index jarg5_);\n  public final static native void Clustering_post_process_centroids(long jarg1, Clustering jarg1_);\n  public final static native void delete_Clustering(long jarg1);\n  public final static native long new_Clustering1D__SWIG_0(int jarg1);\n  public final static native long new_Clustering1D__SWIG_1(int jarg1, long jarg2, ClusteringParameters jarg2_);\n  public final static native void Clustering1D_train_exact(long jarg1, Clustering1D jarg1_, long jarg2, long jarg3);\n  public final static native void delete_Clustering1D(long jarg1);\n  public final static native void ProgressiveDimClusteringParameters_progressive_dim_steps_set(long jarg1, ProgressiveDimClusteringParameters jarg1_, int jarg2);\n  public final static native int ProgressiveDimClusteringParameters_progressive_dim_steps_get(long jarg1, ProgressiveDimClusteringParameters jarg1_);\n  public final static native void ProgressiveDimClusteringParameters_apply_pca_set(long jarg1, ProgressiveDimClusteringParameters jarg1_, boolean jarg2);\n  public final static native boolean ProgressiveDimClusteringParameters_apply_pca_get(long jarg1, ProgressiveDimClusteringParameters jarg1_);\n  public final static native long new_ProgressiveDimClusteringParameters();\n  public final static native void delete_ProgressiveDimClusteringParameters(long jarg1);\n  public final static native void delete_ProgressiveDimIndexFactory(long jarg1);\n  public final static native long new_ProgressiveDimIndexFactory();\n  public final static native void ProgressiveDimClustering_d_set(long jarg1, ProgressiveDimClustering jarg1_, long jarg2);\n  public final static native long ProgressiveDimClustering_d_get(long jarg1, ProgressiveDimClustering jarg1_);\n  public final static native void ProgressiveDimClustering_k_set(long jarg1, ProgressiveDimClustering jarg1_, long jarg2);\n  public final static native long ProgressiveDimClustering_k_get(long jarg1, ProgressiveDimClustering jarg1_);\n  public final static native void ProgressiveDimClustering_centroids_set(long jarg1, ProgressiveDimClustering jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long ProgressiveDimClustering_centroids_get(long jarg1, ProgressiveDimClustering jarg1_);\n  public final static native void ProgressiveDimClustering_iteration_stats_set(long jarg1, ProgressiveDimClustering jarg1_, long jarg2);\n  public final static native long ProgressiveDimClustering_iteration_stats_get(long jarg1, ProgressiveDimClustering jarg1_);\n  public final static native long new_ProgressiveDimClustering__SWIG_0(int jarg1, int jarg2);\n  public final static native long new_ProgressiveDimClustering__SWIG_1(int jarg1, int jarg2, long jarg3, ProgressiveDimClusteringParameters jarg3_);\n  public final static native void ProgressiveDimClustering_train(long jarg1, ProgressiveDimClustering jarg1_, long jarg2, long jarg3, long jarg4, ProgressiveDimIndexFactory jarg4_);\n  public final static native void delete_ProgressiveDimClustering(long jarg1);\n  public final static native float kmeans_clustering(long jarg1, long jarg2, long jarg3, long jarg4, long jarg5);\n  public final static native void ProductQuantizer_d_set(long jarg1, ProductQuantizer jarg1_, long jarg2);\n  public final static native long ProductQuantizer_d_get(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_M_set(long jarg1, ProductQuantizer jarg1_, long jarg2);\n  public final static native long ProductQuantizer_M_get(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_nbits_set(long jarg1, ProductQuantizer jarg1_, long jarg2);\n  public final static native long ProductQuantizer_nbits_get(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_dsub_set(long jarg1, ProductQuantizer jarg1_, long jarg2);\n  public final static native long ProductQuantizer_dsub_get(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_code_size_set(long jarg1, ProductQuantizer jarg1_, long jarg2);\n  public final static native long ProductQuantizer_code_size_get(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_ksub_set(long jarg1, ProductQuantizer jarg1_, long jarg2);\n  public final static native long ProductQuantizer_ksub_get(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_verbose_set(long jarg1, ProductQuantizer jarg1_, boolean jarg2);\n  public final static native boolean ProductQuantizer_verbose_get(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_train_type_set(long jarg1, ProductQuantizer jarg1_, int jarg2);\n  public final static native int ProductQuantizer_train_type_get(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_cp_set(long jarg1, ProductQuantizer jarg1_, long jarg2, ClusteringParameters jarg2_);\n  public final static native long ProductQuantizer_cp_get(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_assign_index_set(long jarg1, ProductQuantizer jarg1_, long jarg2, Index jarg2_);\n  public final static native long ProductQuantizer_assign_index_get(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_centroids_set(long jarg1, ProductQuantizer jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long ProductQuantizer_centroids_get(long jarg1, ProductQuantizer jarg1_);\n  public final static native long ProductQuantizer_get_centroids(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3);\n  public final static native void ProductQuantizer_train(long jarg1, ProductQuantizer jarg1_, int jarg2, long jarg3);\n  public final static native long new_ProductQuantizer__SWIG_0(long jarg1, long jarg2, long jarg3);\n  public final static native long new_ProductQuantizer__SWIG_1();\n  public final static native void ProductQuantizer_set_derived_values(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_set_params(long jarg1, ProductQuantizer jarg1_, long jarg2, int jarg3);\n  public final static native void ProductQuantizer_compute_code(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3);\n  public final static native void ProductQuantizer_compute_codes(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void ProductQuantizer_compute_codes_with_assign_index(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void ProductQuantizer_decode__SWIG_0(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3);\n  public final static native void ProductQuantizer_decode__SWIG_1(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void ProductQuantizer_compute_code_from_distance_table(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3);\n  public final static native void ProductQuantizer_compute_distance_table(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3);\n  public final static native void ProductQuantizer_compute_inner_prod_table(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3);\n  public final static native void ProductQuantizer_compute_distance_tables(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void ProductQuantizer_compute_inner_prod_tables(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void ProductQuantizer_search__SWIG_0(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, boolean jarg7);\n  public final static native void ProductQuantizer_search__SWIG_1(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native void ProductQuantizer_search_ip__SWIG_0(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, boolean jarg7);\n  public final static native void ProductQuantizer_search_ip__SWIG_1(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native void ProductQuantizer_sdc_table_set(long jarg1, ProductQuantizer jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long ProductQuantizer_sdc_table_get(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_compute_sdc_table(long jarg1, ProductQuantizer jarg1_);\n  public final static native void ProductQuantizer_search_sdc__SWIG_0(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, boolean jarg7);\n  public final static native void ProductQuantizer_search_sdc__SWIG_1(long jarg1, ProductQuantizer jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native void delete_ProductQuantizer(long jarg1);\n  public final static native void PQEncoderGeneric_code_set(long jarg1, PQEncoderGeneric jarg1_, long jarg2);\n  public final static native long PQEncoderGeneric_code_get(long jarg1, PQEncoderGeneric jarg1_);\n  public final static native void PQEncoderGeneric_offset_set(long jarg1, PQEncoderGeneric jarg1_, short jarg2);\n  public final static native short PQEncoderGeneric_offset_get(long jarg1, PQEncoderGeneric jarg1_);\n  public final static native int PQEncoderGeneric_nbits_get(long jarg1, PQEncoderGeneric jarg1_);\n  public final static native void PQEncoderGeneric_reg_set(long jarg1, PQEncoderGeneric jarg1_, short jarg2);\n  public final static native short PQEncoderGeneric_reg_get(long jarg1, PQEncoderGeneric jarg1_);\n  public final static native long new_PQEncoderGeneric__SWIG_0(long jarg1, int jarg2, short jarg3);\n  public final static native long new_PQEncoderGeneric__SWIG_1(long jarg1, int jarg2);\n  public final static native void PQEncoderGeneric_encode(long jarg1, PQEncoderGeneric jarg1_, long jarg2);\n  public final static native void delete_PQEncoderGeneric(long jarg1);\n  public final static native void PQEncoder8_code_set(long jarg1, PQEncoder8 jarg1_, long jarg2);\n  public final static native long PQEncoder8_code_get(long jarg1, PQEncoder8 jarg1_);\n  public final static native long new_PQEncoder8(long jarg1, int jarg2);\n  public final static native void PQEncoder8_encode(long jarg1, PQEncoder8 jarg1_, long jarg2);\n  public final static native void delete_PQEncoder8(long jarg1);\n  public final static native void PQEncoder16_code_set(long jarg1, PQEncoder16 jarg1_, long jarg2);\n  public final static native long PQEncoder16_code_get(long jarg1, PQEncoder16 jarg1_);\n  public final static native long new_PQEncoder16(long jarg1, int jarg2);\n  public final static native void PQEncoder16_encode(long jarg1, PQEncoder16 jarg1_, long jarg2);\n  public final static native void delete_PQEncoder16(long jarg1);\n  public final static native void PQDecoderGeneric_code_set(long jarg1, PQDecoderGeneric jarg1_, long jarg2);\n  public final static native long PQDecoderGeneric_code_get(long jarg1, PQDecoderGeneric jarg1_);\n  public final static native void PQDecoderGeneric_offset_set(long jarg1, PQDecoderGeneric jarg1_, short jarg2);\n  public final static native short PQDecoderGeneric_offset_get(long jarg1, PQDecoderGeneric jarg1_);\n  public final static native int PQDecoderGeneric_nbits_get(long jarg1, PQDecoderGeneric jarg1_);\n  public final static native long PQDecoderGeneric_mask_get(long jarg1, PQDecoderGeneric jarg1_);\n  public final static native void PQDecoderGeneric_reg_set(long jarg1, PQDecoderGeneric jarg1_, short jarg2);\n  public final static native short PQDecoderGeneric_reg_get(long jarg1, PQDecoderGeneric jarg1_);\n  public final static native long new_PQDecoderGeneric(long jarg1, int jarg2);\n  public final static native long PQDecoderGeneric_decode(long jarg1, PQDecoderGeneric jarg1_);\n  public final static native void delete_PQDecoderGeneric(long jarg1);\n  public final static native int PQDecoder8_nbits_get();\n  public final static native void PQDecoder8_code_set(long jarg1, PQDecoder8 jarg1_, long jarg2);\n  public final static native long PQDecoder8_code_get(long jarg1, PQDecoder8 jarg1_);\n  public final static native long new_PQDecoder8(long jarg1, int jarg2);\n  public final static native long PQDecoder8_decode(long jarg1, PQDecoder8 jarg1_);\n  public final static native void delete_PQDecoder8(long jarg1);\n  public final static native int PQDecoder16_nbits_get();\n  public final static native void PQDecoder16_code_set(long jarg1, PQDecoder16 jarg1_, long jarg2);\n  public final static native long PQDecoder16_code_get(long jarg1, PQDecoder16 jarg1_);\n  public final static native long new_PQDecoder16(long jarg1, int jarg2);\n  public final static native long PQDecoder16_decode(long jarg1, PQDecoder16 jarg1_);\n  public final static native void delete_PQDecoder16(long jarg1);\n  public final static native void VectorTransform_d_in_set(long jarg1, VectorTransform jarg1_, int jarg2);\n  public final static native int VectorTransform_d_in_get(long jarg1, VectorTransform jarg1_);\n  public final static native void VectorTransform_d_out_set(long jarg1, VectorTransform jarg1_, int jarg2);\n  public final static native int VectorTransform_d_out_get(long jarg1, VectorTransform jarg1_);\n  public final static native void VectorTransform_is_trained_set(long jarg1, VectorTransform jarg1_, boolean jarg2);\n  public final static native boolean VectorTransform_is_trained_get(long jarg1, VectorTransform jarg1_);\n  public final static native void VectorTransform_train(long jarg1, VectorTransform jarg1_, long jarg2, long jarg3);\n  public final static native long VectorTransform_apply(long jarg1, VectorTransform jarg1_, long jarg2, long jarg3);\n  public final static native void VectorTransform_apply_noalloc(long jarg1, VectorTransform jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void VectorTransform_reverse_transform(long jarg1, VectorTransform jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void delete_VectorTransform(long jarg1);\n  public final static native void LinearTransform_have_bias_set(long jarg1, LinearTransform jarg1_, boolean jarg2);\n  public final static native boolean LinearTransform_have_bias_get(long jarg1, LinearTransform jarg1_);\n  public final static native void LinearTransform_is_orthonormal_set(long jarg1, LinearTransform jarg1_, boolean jarg2);\n  public final static native boolean LinearTransform_is_orthonormal_get(long jarg1, LinearTransform jarg1_);\n  public final static native void LinearTransform_A_set(long jarg1, LinearTransform jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long LinearTransform_A_get(long jarg1, LinearTransform jarg1_);\n  public final static native void LinearTransform_b_set(long jarg1, LinearTransform jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long LinearTransform_b_get(long jarg1, LinearTransform jarg1_);\n  public final static native long new_LinearTransform__SWIG_0(int jarg1, int jarg2, boolean jarg3);\n  public final static native long new_LinearTransform__SWIG_1(int jarg1, int jarg2);\n  public final static native long new_LinearTransform__SWIG_2(int jarg1);\n  public final static native long new_LinearTransform__SWIG_3();\n  public final static native void LinearTransform_apply_noalloc(long jarg1, LinearTransform jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void LinearTransform_transform_transpose(long jarg1, LinearTransform jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void LinearTransform_reverse_transform(long jarg1, LinearTransform jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void LinearTransform_set_is_orthonormal(long jarg1, LinearTransform jarg1_);\n  public final static native void LinearTransform_verbose_set(long jarg1, LinearTransform jarg1_, boolean jarg2);\n  public final static native boolean LinearTransform_verbose_get(long jarg1, LinearTransform jarg1_);\n  public final static native void LinearTransform_print_if_verbose(long jarg1, LinearTransform jarg1_, String jarg2, long jarg3, DoubleVector jarg3_, int jarg4, int jarg5);\n  public final static native void delete_LinearTransform(long jarg1);\n  public final static native long new_RandomRotationMatrix__SWIG_0(int jarg1, int jarg2);\n  public final static native void RandomRotationMatrix_init(long jarg1, RandomRotationMatrix jarg1_, int jarg2);\n  public final static native void RandomRotationMatrix_train(long jarg1, RandomRotationMatrix jarg1_, long jarg2, long jarg3);\n  public final static native long new_RandomRotationMatrix__SWIG_1();\n  public final static native void delete_RandomRotationMatrix(long jarg1);\n  public final static native void PCAMatrix_eigen_power_set(long jarg1, PCAMatrix jarg1_, float jarg2);\n  public final static native float PCAMatrix_eigen_power_get(long jarg1, PCAMatrix jarg1_);\n  public final static native void PCAMatrix_epsilon_set(long jarg1, PCAMatrix jarg1_, float jarg2);\n  public final static native float PCAMatrix_epsilon_get(long jarg1, PCAMatrix jarg1_);\n  public final static native void PCAMatrix_random_rotation_set(long jarg1, PCAMatrix jarg1_, boolean jarg2);\n  public final static native boolean PCAMatrix_random_rotation_get(long jarg1, PCAMatrix jarg1_);\n  public final static native void PCAMatrix_max_points_per_d_set(long jarg1, PCAMatrix jarg1_, long jarg2);\n  public final static native long PCAMatrix_max_points_per_d_get(long jarg1, PCAMatrix jarg1_);\n  public final static native void PCAMatrix_balanced_bins_set(long jarg1, PCAMatrix jarg1_, int jarg2);\n  public final static native int PCAMatrix_balanced_bins_get(long jarg1, PCAMatrix jarg1_);\n  public final static native void PCAMatrix_mean_set(long jarg1, PCAMatrix jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long PCAMatrix_mean_get(long jarg1, PCAMatrix jarg1_);\n  public final static native void PCAMatrix_eigenvalues_set(long jarg1, PCAMatrix jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long PCAMatrix_eigenvalues_get(long jarg1, PCAMatrix jarg1_);\n  public final static native void PCAMatrix_PCAMat_set(long jarg1, PCAMatrix jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long PCAMatrix_PCAMat_get(long jarg1, PCAMatrix jarg1_);\n  public final static native long new_PCAMatrix__SWIG_0(int jarg1, int jarg2, float jarg3, boolean jarg4);\n  public final static native long new_PCAMatrix__SWIG_1(int jarg1, int jarg2, float jarg3);\n  public final static native long new_PCAMatrix__SWIG_2(int jarg1, int jarg2);\n  public final static native long new_PCAMatrix__SWIG_3(int jarg1);\n  public final static native long new_PCAMatrix__SWIG_4();\n  public final static native void PCAMatrix_train(long jarg1, PCAMatrix jarg1_, long jarg2, long jarg3);\n  public final static native void PCAMatrix_copy_from(long jarg1, PCAMatrix jarg1_, long jarg2, PCAMatrix jarg2_);\n  public final static native void PCAMatrix_prepare_Ab(long jarg1, PCAMatrix jarg1_);\n  public final static native void delete_PCAMatrix(long jarg1);\n  public final static native void ITQMatrix_max_iter_set(long jarg1, ITQMatrix jarg1_, int jarg2);\n  public final static native int ITQMatrix_max_iter_get(long jarg1, ITQMatrix jarg1_);\n  public final static native void ITQMatrix_seed_set(long jarg1, ITQMatrix jarg1_, int jarg2);\n  public final static native int ITQMatrix_seed_get(long jarg1, ITQMatrix jarg1_);\n  public final static native void ITQMatrix_init_rotation_set(long jarg1, ITQMatrix jarg1_, long jarg2, DoubleVector jarg2_);\n  public final static native long ITQMatrix_init_rotation_get(long jarg1, ITQMatrix jarg1_);\n  public final static native long new_ITQMatrix__SWIG_0(int jarg1);\n  public final static native long new_ITQMatrix__SWIG_1();\n  public final static native void ITQMatrix_train(long jarg1, ITQMatrix jarg1_, long jarg2, long jarg3);\n  public final static native void delete_ITQMatrix(long jarg1);\n  public final static native void ITQTransform_mean_set(long jarg1, ITQTransform jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long ITQTransform_mean_get(long jarg1, ITQTransform jarg1_);\n  public final static native void ITQTransform_do_pca_set(long jarg1, ITQTransform jarg1_, boolean jarg2);\n  public final static native boolean ITQTransform_do_pca_get(long jarg1, ITQTransform jarg1_);\n  public final static native void ITQTransform_itq_set(long jarg1, ITQTransform jarg1_, long jarg2, ITQMatrix jarg2_);\n  public final static native long ITQTransform_itq_get(long jarg1, ITQTransform jarg1_);\n  public final static native void ITQTransform_max_train_per_dim_set(long jarg1, ITQTransform jarg1_, int jarg2);\n  public final static native int ITQTransform_max_train_per_dim_get(long jarg1, ITQTransform jarg1_);\n  public final static native void ITQTransform_pca_then_itq_set(long jarg1, ITQTransform jarg1_, long jarg2, LinearTransform jarg2_);\n  public final static native long ITQTransform_pca_then_itq_get(long jarg1, ITQTransform jarg1_);\n  public final static native long new_ITQTransform__SWIG_0(int jarg1, int jarg2, boolean jarg3);\n  public final static native long new_ITQTransform__SWIG_1(int jarg1, int jarg2);\n  public final static native long new_ITQTransform__SWIG_2(int jarg1);\n  public final static native long new_ITQTransform__SWIG_3();\n  public final static native void ITQTransform_train(long jarg1, ITQTransform jarg1_, long jarg2, long jarg3);\n  public final static native void ITQTransform_apply_noalloc(long jarg1, ITQTransform jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void delete_ITQTransform(long jarg1);\n  public final static native void OPQMatrix_M_set(long jarg1, OPQMatrix jarg1_, int jarg2);\n  public final static native int OPQMatrix_M_get(long jarg1, OPQMatrix jarg1_);\n  public final static native void OPQMatrix_niter_set(long jarg1, OPQMatrix jarg1_, int jarg2);\n  public final static native int OPQMatrix_niter_get(long jarg1, OPQMatrix jarg1_);\n  public final static native void OPQMatrix_niter_pq_set(long jarg1, OPQMatrix jarg1_, int jarg2);\n  public final static native int OPQMatrix_niter_pq_get(long jarg1, OPQMatrix jarg1_);\n  public final static native void OPQMatrix_niter_pq_0_set(long jarg1, OPQMatrix jarg1_, int jarg2);\n  public final static native int OPQMatrix_niter_pq_0_get(long jarg1, OPQMatrix jarg1_);\n  public final static native void OPQMatrix_max_train_points_set(long jarg1, OPQMatrix jarg1_, long jarg2);\n  public final static native long OPQMatrix_max_train_points_get(long jarg1, OPQMatrix jarg1_);\n  public final static native void OPQMatrix_verbose_set(long jarg1, OPQMatrix jarg1_, boolean jarg2);\n  public final static native boolean OPQMatrix_verbose_get(long jarg1, OPQMatrix jarg1_);\n  public final static native void OPQMatrix_pq_set(long jarg1, OPQMatrix jarg1_, long jarg2, ProductQuantizer jarg2_);\n  public final static native long OPQMatrix_pq_get(long jarg1, OPQMatrix jarg1_);\n  public final static native long new_OPQMatrix__SWIG_0(int jarg1, int jarg2, int jarg3);\n  public final static native long new_OPQMatrix__SWIG_1(int jarg1, int jarg2);\n  public final static native long new_OPQMatrix__SWIG_2(int jarg1);\n  public final static native long new_OPQMatrix__SWIG_3();\n  public final static native void OPQMatrix_train(long jarg1, OPQMatrix jarg1_, long jarg2, long jarg3);\n  public final static native void delete_OPQMatrix(long jarg1);\n  public final static native void RemapDimensionsTransform_map_set(long jarg1, RemapDimensionsTransform jarg1_, long jarg2, IntVector jarg2_);\n  public final static native long RemapDimensionsTransform_map_get(long jarg1, RemapDimensionsTransform jarg1_);\n  public final static native long new_RemapDimensionsTransform__SWIG_0(int jarg1, int jarg2, long jarg3);\n  public final static native long new_RemapDimensionsTransform__SWIG_1(int jarg1, int jarg2, boolean jarg3);\n  public final static native long new_RemapDimensionsTransform__SWIG_2(int jarg1, int jarg2);\n  public final static native void RemapDimensionsTransform_apply_noalloc(long jarg1, RemapDimensionsTransform jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void RemapDimensionsTransform_reverse_transform(long jarg1, RemapDimensionsTransform jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native long new_RemapDimensionsTransform__SWIG_3();\n  public final static native void delete_RemapDimensionsTransform(long jarg1);\n  public final static native void NormalizationTransform_norm_set(long jarg1, NormalizationTransform jarg1_, float jarg2);\n  public final static native float NormalizationTransform_norm_get(long jarg1, NormalizationTransform jarg1_);\n  public final static native long new_NormalizationTransform__SWIG_0(int jarg1, float jarg2);\n  public final static native long new_NormalizationTransform__SWIG_1(int jarg1);\n  public final static native long new_NormalizationTransform__SWIG_2();\n  public final static native void NormalizationTransform_apply_noalloc(long jarg1, NormalizationTransform jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void NormalizationTransform_reverse_transform(long jarg1, NormalizationTransform jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void delete_NormalizationTransform(long jarg1);\n  public final static native void CenteringTransform_mean_set(long jarg1, CenteringTransform jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long CenteringTransform_mean_get(long jarg1, CenteringTransform jarg1_);\n  public final static native long new_CenteringTransform__SWIG_0(int jarg1);\n  public final static native long new_CenteringTransform__SWIG_1();\n  public final static native void CenteringTransform_train(long jarg1, CenteringTransform jarg1_, long jarg2, long jarg3);\n  public final static native void CenteringTransform_apply_noalloc(long jarg1, CenteringTransform jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void CenteringTransform_reverse_transform(long jarg1, CenteringTransform jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void delete_CenteringTransform(long jarg1);\n  public final static native void IndexFlatCodes_code_size_set(long jarg1, IndexFlatCodes jarg1_, long jarg2);\n  public final static native long IndexFlatCodes_code_size_get(long jarg1, IndexFlatCodes jarg1_);\n  public final static native void IndexFlatCodes_codes_set(long jarg1, IndexFlatCodes jarg1_, long jarg2, ByteVector jarg2_);\n  public final static native long IndexFlatCodes_codes_get(long jarg1, IndexFlatCodes jarg1_);\n  public final static native void IndexFlatCodes_add(long jarg1, IndexFlatCodes jarg1_, long jarg2, long jarg3);\n  public final static native void IndexFlatCodes_reset(long jarg1, IndexFlatCodes jarg1_);\n  public final static native void IndexFlatCodes_reconstruct_n(long jarg1, IndexFlatCodes jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexFlatCodes_reconstruct(long jarg1, IndexFlatCodes jarg1_, long jarg2, long jarg3);\n  public final static native long IndexFlatCodes_sa_code_size(long jarg1, IndexFlatCodes jarg1_);\n  public final static native long IndexFlatCodes_remove_ids(long jarg1, IndexFlatCodes jarg1_, long jarg2, IDSelector jarg2_);\n  public final static native void delete_IndexFlatCodes(long jarg1);\n  public final static native long new_IndexFlat__SWIG_0(long jarg1, int jarg2);\n  public final static native long new_IndexFlat__SWIG_1(long jarg1);\n  public final static native void IndexFlat_search(long jarg1, IndexFlat jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexFlat_range_search(long jarg1, IndexFlat jarg1_, long jarg2, long jarg3, float jarg4, long jarg5, RangeSearchResult jarg5_);\n  public final static native void IndexFlat_reconstruct(long jarg1, IndexFlat jarg1_, long jarg2, long jarg3);\n  public final static native void IndexFlat_compute_distance_subset(long jarg1, IndexFlat jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native long IndexFlat_get_xb__SWIG_0(long jarg1, IndexFlat jarg1_);\n  public final static native long new_IndexFlat__SWIG_2();\n  public final static native long IndexFlat_get_distance_computer(long jarg1, IndexFlat jarg1_);\n  public final static native void IndexFlat_sa_encode(long jarg1, IndexFlat jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexFlat_sa_decode(long jarg1, IndexFlat jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void delete_IndexFlat(long jarg1);\n  public final static native long new_IndexFlatIP__SWIG_0(long jarg1);\n  public final static native long new_IndexFlatIP__SWIG_1();\n  public final static native void delete_IndexFlatIP(long jarg1);\n  public final static native long new_IndexFlatL2__SWIG_0(long jarg1);\n  public final static native long new_IndexFlatL2__SWIG_1();\n  public final static native void delete_IndexFlatL2(long jarg1);\n  public final static native void IndexFlat1D_continuous_update_set(long jarg1, IndexFlat1D jarg1_, boolean jarg2);\n  public final static native boolean IndexFlat1D_continuous_update_get(long jarg1, IndexFlat1D jarg1_);\n  public final static native void IndexFlat1D_perm_set(long jarg1, IndexFlat1D jarg1_, long jarg2);\n  public final static native long IndexFlat1D_perm_get(long jarg1, IndexFlat1D jarg1_);\n  public final static native long new_IndexFlat1D__SWIG_0(boolean jarg1);\n  public final static native long new_IndexFlat1D__SWIG_1();\n  public final static native void IndexFlat1D_update_permutation(long jarg1, IndexFlat1D jarg1_);\n  public final static native void IndexFlat1D_add(long jarg1, IndexFlat1D jarg1_, long jarg2, long jarg3);\n  public final static native void IndexFlat1D_reset(long jarg1, IndexFlat1D jarg1_);\n  public final static native void IndexFlat1D_search(long jarg1, IndexFlat1D jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void delete_IndexFlat1D(long jarg1);\n  public final static native void IndexLSH_nbits_set(long jarg1, IndexLSH jarg1_, int jarg2);\n  public final static native int IndexLSH_nbits_get(long jarg1, IndexLSH jarg1_);\n  public final static native void IndexLSH_rotate_data_set(long jarg1, IndexLSH jarg1_, boolean jarg2);\n  public final static native boolean IndexLSH_rotate_data_get(long jarg1, IndexLSH jarg1_);\n  public final static native void IndexLSH_train_thresholds_set(long jarg1, IndexLSH jarg1_, boolean jarg2);\n  public final static native boolean IndexLSH_train_thresholds_get(long jarg1, IndexLSH jarg1_);\n  public final static native void IndexLSH_rrot_set(long jarg1, IndexLSH jarg1_, long jarg2, RandomRotationMatrix jarg2_);\n  public final static native long IndexLSH_rrot_get(long jarg1, IndexLSH jarg1_);\n  public final static native void IndexLSH_thresholds_set(long jarg1, IndexLSH jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long IndexLSH_thresholds_get(long jarg1, IndexLSH jarg1_);\n  public final static native long new_IndexLSH__SWIG_0(long jarg1, int jarg2, boolean jarg3, boolean jarg4);\n  public final static native long new_IndexLSH__SWIG_1(long jarg1, int jarg2, boolean jarg3);\n  public final static native long new_IndexLSH__SWIG_2(long jarg1, int jarg2);\n  public final static native long IndexLSH_apply_preprocess(long jarg1, IndexLSH jarg1_, long jarg2, long jarg3);\n  public final static native void IndexLSH_train(long jarg1, IndexLSH jarg1_, long jarg2, long jarg3);\n  public final static native void IndexLSH_search(long jarg1, IndexLSH jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexLSH_transfer_thresholds(long jarg1, IndexLSH jarg1_, long jarg2, LinearTransform jarg2_);\n  public final static native void delete_IndexLSH(long jarg1);\n  public final static native long new_IndexLSH__SWIG_3();\n  public final static native void IndexLSH_sa_encode(long jarg1, IndexLSH jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexLSH_sa_decode(long jarg1, IndexLSH jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void SimulatedAnnealingParameters_init_temperature_set(long jarg1, SimulatedAnnealingParameters jarg1_, double jarg2);\n  public final static native double SimulatedAnnealingParameters_init_temperature_get(long jarg1, SimulatedAnnealingParameters jarg1_);\n  public final static native void SimulatedAnnealingParameters_temperature_decay_set(long jarg1, SimulatedAnnealingParameters jarg1_, double jarg2);\n  public final static native double SimulatedAnnealingParameters_temperature_decay_get(long jarg1, SimulatedAnnealingParameters jarg1_);\n  public final static native void SimulatedAnnealingParameters_n_iter_set(long jarg1, SimulatedAnnealingParameters jarg1_, int jarg2);\n  public final static native int SimulatedAnnealingParameters_n_iter_get(long jarg1, SimulatedAnnealingParameters jarg1_);\n  public final static native void SimulatedAnnealingParameters_n_redo_set(long jarg1, SimulatedAnnealingParameters jarg1_, int jarg2);\n  public final static native int SimulatedAnnealingParameters_n_redo_get(long jarg1, SimulatedAnnealingParameters jarg1_);\n  public final static native void SimulatedAnnealingParameters_seed_set(long jarg1, SimulatedAnnealingParameters jarg1_, int jarg2);\n  public final static native int SimulatedAnnealingParameters_seed_get(long jarg1, SimulatedAnnealingParameters jarg1_);\n  public final static native void SimulatedAnnealingParameters_verbose_set(long jarg1, SimulatedAnnealingParameters jarg1_, int jarg2);\n  public final static native int SimulatedAnnealingParameters_verbose_get(long jarg1, SimulatedAnnealingParameters jarg1_);\n  public final static native void SimulatedAnnealingParameters_only_bit_flips_set(long jarg1, SimulatedAnnealingParameters jarg1_, boolean jarg2);\n  public final static native boolean SimulatedAnnealingParameters_only_bit_flips_get(long jarg1, SimulatedAnnealingParameters jarg1_);\n  public final static native void SimulatedAnnealingParameters_init_random_set(long jarg1, SimulatedAnnealingParameters jarg1_, boolean jarg2);\n  public final static native boolean SimulatedAnnealingParameters_init_random_get(long jarg1, SimulatedAnnealingParameters jarg1_);\n  public final static native long new_SimulatedAnnealingParameters();\n  public final static native void delete_SimulatedAnnealingParameters(long jarg1);\n  public final static native void PermutationObjective_n_set(long jarg1, PermutationObjective jarg1_, int jarg2);\n  public final static native int PermutationObjective_n_get(long jarg1, PermutationObjective jarg1_);\n  public final static native double PermutationObjective_compute_cost(long jarg1, PermutationObjective jarg1_, long jarg2);\n  public final static native double PermutationObjective_cost_update(long jarg1, PermutationObjective jarg1_, long jarg2, int jarg3, int jarg4);\n  public final static native void delete_PermutationObjective(long jarg1);\n  public final static native void ReproduceDistancesObjective_dis_weight_factor_set(long jarg1, ReproduceDistancesObjective jarg1_, double jarg2);\n  public final static native double ReproduceDistancesObjective_dis_weight_factor_get(long jarg1, ReproduceDistancesObjective jarg1_);\n  public final static native double ReproduceDistancesObjective_sqr(double jarg1);\n  public final static native double ReproduceDistancesObjective_dis_weight(long jarg1, ReproduceDistancesObjective jarg1_, double jarg2);\n  public final static native void ReproduceDistancesObjective_source_dis_set(long jarg1, ReproduceDistancesObjective jarg1_, long jarg2, DoubleVector jarg2_);\n  public final static native long ReproduceDistancesObjective_source_dis_get(long jarg1, ReproduceDistancesObjective jarg1_);\n  public final static native void ReproduceDistancesObjective_target_dis_set(long jarg1, ReproduceDistancesObjective jarg1_, long jarg2);\n  public final static native long ReproduceDistancesObjective_target_dis_get(long jarg1, ReproduceDistancesObjective jarg1_);\n  public final static native void ReproduceDistancesObjective_weights_set(long jarg1, ReproduceDistancesObjective jarg1_, long jarg2, DoubleVector jarg2_);\n  public final static native long ReproduceDistancesObjective_weights_get(long jarg1, ReproduceDistancesObjective jarg1_);\n  public final static native double ReproduceDistancesObjective_get_source_dis(long jarg1, ReproduceDistancesObjective jarg1_, int jarg2, int jarg3);\n  public final static native double ReproduceDistancesObjective_compute_cost(long jarg1, ReproduceDistancesObjective jarg1_, long jarg2);\n  public final static native double ReproduceDistancesObjective_cost_update(long jarg1, ReproduceDistancesObjective jarg1_, long jarg2, int jarg3, int jarg4);\n  public final static native long new_ReproduceDistancesObjective(int jarg1, long jarg2, long jarg3, double jarg4);\n  public final static native void ReproduceDistancesObjective_compute_mean_stdev(long jarg1, long jarg2, long jarg3, long jarg4);\n  public final static native void ReproduceDistancesObjective_set_affine_target_dis(long jarg1, ReproduceDistancesObjective jarg1_, long jarg2);\n  public final static native void delete_ReproduceDistancesObjective(long jarg1);\n  public final static native void SimulatedAnnealingOptimizer_obj_set(long jarg1, SimulatedAnnealingOptimizer jarg1_, long jarg2, PermutationObjective jarg2_);\n  public final static native long SimulatedAnnealingOptimizer_obj_get(long jarg1, SimulatedAnnealingOptimizer jarg1_);\n  public final static native void SimulatedAnnealingOptimizer_n_set(long jarg1, SimulatedAnnealingOptimizer jarg1_, int jarg2);\n  public final static native int SimulatedAnnealingOptimizer_n_get(long jarg1, SimulatedAnnealingOptimizer jarg1_);\n  public final static native void SimulatedAnnealingOptimizer_logfile_set(long jarg1, SimulatedAnnealingOptimizer jarg1_, long jarg2);\n  public final static native long SimulatedAnnealingOptimizer_logfile_get(long jarg1, SimulatedAnnealingOptimizer jarg1_);\n  public final static native long new_SimulatedAnnealingOptimizer(long jarg1, PermutationObjective jarg1_, long jarg2, SimulatedAnnealingParameters jarg2_);\n  public final static native void SimulatedAnnealingOptimizer_rnd_set(long jarg1, SimulatedAnnealingOptimizer jarg1_, long jarg2);\n  public final static native long SimulatedAnnealingOptimizer_rnd_get(long jarg1, SimulatedAnnealingOptimizer jarg1_);\n  public final static native void SimulatedAnnealingOptimizer_init_cost_set(long jarg1, SimulatedAnnealingOptimizer jarg1_, double jarg2);\n  public final static native double SimulatedAnnealingOptimizer_init_cost_get(long jarg1, SimulatedAnnealingOptimizer jarg1_);\n  public final static native double SimulatedAnnealingOptimizer_optimize(long jarg1, SimulatedAnnealingOptimizer jarg1_, long jarg2);\n  public final static native double SimulatedAnnealingOptimizer_run_optimization(long jarg1, SimulatedAnnealingOptimizer jarg1_, long jarg2);\n  public final static native void delete_SimulatedAnnealingOptimizer(long jarg1);\n  public final static native void PolysemousTraining_optimization_type_set(long jarg1, PolysemousTraining jarg1_, int jarg2);\n  public final static native int PolysemousTraining_optimization_type_get(long jarg1, PolysemousTraining jarg1_);\n  public final static native void PolysemousTraining_ntrain_permutation_set(long jarg1, PolysemousTraining jarg1_, int jarg2);\n  public final static native int PolysemousTraining_ntrain_permutation_get(long jarg1, PolysemousTraining jarg1_);\n  public final static native void PolysemousTraining_dis_weight_factor_set(long jarg1, PolysemousTraining jarg1_, double jarg2);\n  public final static native double PolysemousTraining_dis_weight_factor_get(long jarg1, PolysemousTraining jarg1_);\n  public final static native void PolysemousTraining_max_memory_set(long jarg1, PolysemousTraining jarg1_, long jarg2);\n  public final static native long PolysemousTraining_max_memory_get(long jarg1, PolysemousTraining jarg1_);\n  public final static native void PolysemousTraining_log_pattern_set(long jarg1, PolysemousTraining jarg1_, String jarg2);\n  public final static native String PolysemousTraining_log_pattern_get(long jarg1, PolysemousTraining jarg1_);\n  public final static native long new_PolysemousTraining();\n  public final static native void PolysemousTraining_optimize_pq_for_hamming(long jarg1, PolysemousTraining jarg1_, long jarg2, ProductQuantizer jarg2_, long jarg3, long jarg4);\n  public final static native void PolysemousTraining_optimize_ranking(long jarg1, PolysemousTraining jarg1_, long jarg2, ProductQuantizer jarg2_, long jarg3, long jarg4);\n  public final static native void PolysemousTraining_optimize_reproduce_distances(long jarg1, PolysemousTraining jarg1_, long jarg2, ProductQuantizer jarg2_);\n  public final static native long PolysemousTraining_memory_usage_per_thread(long jarg1, PolysemousTraining jarg1_, long jarg2, ProductQuantizer jarg2_);\n  public final static native void delete_PolysemousTraining(long jarg1);\n  public final static native void IndexPQ_pq_set(long jarg1, IndexPQ jarg1_, long jarg2, ProductQuantizer jarg2_);\n  public final static native long IndexPQ_pq_get(long jarg1, IndexPQ jarg1_);\n  public final static native long new_IndexPQ__SWIG_0(int jarg1, long jarg2, long jarg3, int jarg4);\n  public final static native long new_IndexPQ__SWIG_1(int jarg1, long jarg2, long jarg3);\n  public final static native long new_IndexPQ__SWIG_2();\n  public final static native void IndexPQ_train(long jarg1, IndexPQ jarg1_, long jarg2, long jarg3);\n  public final static native void IndexPQ_search(long jarg1, IndexPQ jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexPQ_sa_encode(long jarg1, IndexPQ jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexPQ_sa_decode(long jarg1, IndexPQ jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native long IndexPQ_get_distance_computer(long jarg1, IndexPQ jarg1_);\n  public final static native void IndexPQ_do_polysemous_training_set(long jarg1, IndexPQ jarg1_, boolean jarg2);\n  public final static native boolean IndexPQ_do_polysemous_training_get(long jarg1, IndexPQ jarg1_);\n  public final static native void IndexPQ_polysemous_training_set(long jarg1, IndexPQ jarg1_, long jarg2, PolysemousTraining jarg2_);\n  public final static native long IndexPQ_polysemous_training_get(long jarg1, IndexPQ jarg1_);\n  public final static native void IndexPQ_search_type_set(long jarg1, IndexPQ jarg1_, int jarg2);\n  public final static native int IndexPQ_search_type_get(long jarg1, IndexPQ jarg1_);\n  public final static native void IndexPQ_encode_signs_set(long jarg1, IndexPQ jarg1_, boolean jarg2);\n  public final static native boolean IndexPQ_encode_signs_get(long jarg1, IndexPQ jarg1_);\n  public final static native void IndexPQ_polysemous_ht_set(long jarg1, IndexPQ jarg1_, int jarg2);\n  public final static native int IndexPQ_polysemous_ht_get(long jarg1, IndexPQ jarg1_);\n  public final static native void IndexPQ_search_core_polysemous(long jarg1, IndexPQ jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexPQ_hamming_distance_histogram(long jarg1, IndexPQ jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexPQ_hamming_distance_table(long jarg1, IndexPQ jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void delete_IndexPQ(long jarg1);\n  public final static native void IndexPQStats_nq_set(long jarg1, IndexPQStats jarg1_, long jarg2);\n  public final static native long IndexPQStats_nq_get(long jarg1, IndexPQStats jarg1_);\n  public final static native void IndexPQStats_ncode_set(long jarg1, IndexPQStats jarg1_, long jarg2);\n  public final static native long IndexPQStats_ncode_get(long jarg1, IndexPQStats jarg1_);\n  public final static native void IndexPQStats_n_hamming_pass_set(long jarg1, IndexPQStats jarg1_, long jarg2);\n  public final static native long IndexPQStats_n_hamming_pass_get(long jarg1, IndexPQStats jarg1_);\n  public final static native long new_IndexPQStats();\n  public final static native void IndexPQStats_reset(long jarg1, IndexPQStats jarg1_);\n  public final static native void delete_IndexPQStats(long jarg1);\n  public final static native void indexPQ_stats_set(long jarg1, IndexPQStats jarg1_);\n  public final static native long indexPQ_stats_get();\n  public final static native void MultiIndexQuantizer_pq_set(long jarg1, MultiIndexQuantizer jarg1_, long jarg2, ProductQuantizer jarg2_);\n  public final static native long MultiIndexQuantizer_pq_get(long jarg1, MultiIndexQuantizer jarg1_);\n  public final static native long new_MultiIndexQuantizer__SWIG_0(int jarg1, long jarg2, long jarg3);\n  public final static native void MultiIndexQuantizer_train(long jarg1, MultiIndexQuantizer jarg1_, long jarg2, long jarg3);\n  public final static native void MultiIndexQuantizer_search(long jarg1, MultiIndexQuantizer jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void MultiIndexQuantizer_add(long jarg1, MultiIndexQuantizer jarg1_, long jarg2, long jarg3);\n  public final static native void MultiIndexQuantizer_reset(long jarg1, MultiIndexQuantizer jarg1_);\n  public final static native long new_MultiIndexQuantizer__SWIG_1();\n  public final static native void MultiIndexQuantizer_reconstruct(long jarg1, MultiIndexQuantizer jarg1_, long jarg2, long jarg3);\n  public final static native void delete_MultiIndexQuantizer(long jarg1);\n  public final static native void MultiIndexQuantizer2_assign_indexes_set(long jarg1, MultiIndexQuantizer2 jarg1_, long jarg2);\n  public final static native long MultiIndexQuantizer2_assign_indexes_get(long jarg1, MultiIndexQuantizer2 jarg1_);\n  public final static native void MultiIndexQuantizer2_own_fields_set(long jarg1, MultiIndexQuantizer2 jarg1_, boolean jarg2);\n  public final static native boolean MultiIndexQuantizer2_own_fields_get(long jarg1, MultiIndexQuantizer2 jarg1_);\n  public final static native long new_MultiIndexQuantizer2__SWIG_0(int jarg1, long jarg2, long jarg3, long jarg4);\n  public final static native long new_MultiIndexQuantizer2__SWIG_1(int jarg1, long jarg2, long jarg3, Index jarg3_, long jarg4, Index jarg4_);\n  public final static native void MultiIndexQuantizer2_train(long jarg1, MultiIndexQuantizer2 jarg1_, long jarg2, long jarg3);\n  public final static native void MultiIndexQuantizer2_search(long jarg1, MultiIndexQuantizer2 jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void delete_MultiIndexQuantizer2(long jarg1);\n  public final static native void InvertedLists_nlist_set(long jarg1, InvertedLists jarg1_, long jarg2);\n  public final static native long InvertedLists_nlist_get(long jarg1, InvertedLists jarg1_);\n  public final static native void InvertedLists_code_size_set(long jarg1, InvertedLists jarg1_, long jarg2);\n  public final static native long InvertedLists_code_size_get(long jarg1, InvertedLists jarg1_);\n  public final static native long InvertedLists_INVALID_CODE_SIZE_get();\n  public final static native long InvertedLists_list_size(long jarg1, InvertedLists jarg1_, long jarg2);\n  public final static native long InvertedLists_get_codes(long jarg1, InvertedLists jarg1_, long jarg2);\n  public final static native long InvertedLists_get_ids(long jarg1, InvertedLists jarg1_, long jarg2);\n  public final static native void InvertedLists_release_codes(long jarg1, InvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void InvertedLists_release_ids(long jarg1, InvertedLists jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native long InvertedLists_get_single_id(long jarg1, InvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native long InvertedLists_get_single_code(long jarg1, InvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void InvertedLists_prefetch_lists(long jarg1, InvertedLists jarg1_, long jarg2, LongVector jarg2_, int jarg3);\n  public final static native long InvertedLists_add_entry(long jarg1, InvertedLists jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native long InvertedLists_add_entries(long jarg1, InvertedLists jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void InvertedLists_update_entry(long jarg1, InvertedLists jarg1_, long jarg2, long jarg3, long jarg4, long jarg5);\n  public final static native void InvertedLists_update_entries(long jarg1, InvertedLists jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_, long jarg6);\n  public final static native void InvertedLists_resize(long jarg1, InvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void InvertedLists_reset(long jarg1, InvertedLists jarg1_);\n  public final static native void InvertedLists_merge_from(long jarg1, InvertedLists jarg1_, long jarg2, InvertedLists jarg2_, long jarg3);\n  public final static native void delete_InvertedLists(long jarg1);\n  public final static native double InvertedLists_imbalance_factor(long jarg1, InvertedLists jarg1_);\n  public final static native void InvertedLists_print_stats(long jarg1, InvertedLists jarg1_);\n  public final static native long InvertedLists_compute_ntotal(long jarg1, InvertedLists jarg1_);\n  public final static native void InvertedLists_ScopedIds_il_set(long jarg1, InvertedLists.ScopedIds jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native long InvertedLists_ScopedIds_il_get(long jarg1, InvertedLists.ScopedIds jarg1_);\n  public final static native void InvertedLists_ScopedIds_ids_set(long jarg1, InvertedLists.ScopedIds jarg1_, long jarg2, LongVector jarg2_);\n  public final static native long InvertedLists_ScopedIds_ids_get(long jarg1, InvertedLists.ScopedIds jarg1_);\n  public final static native void InvertedLists_ScopedIds_list_no_set(long jarg1, InvertedLists.ScopedIds jarg1_, long jarg2);\n  public final static native long InvertedLists_ScopedIds_list_no_get(long jarg1, InvertedLists.ScopedIds jarg1_);\n  public final static native long new_InvertedLists_ScopedIds(long jarg1, InvertedLists jarg1_, long jarg2);\n  public final static native long InvertedLists_ScopedIds_get(long jarg1, InvertedLists.ScopedIds jarg1_);\n  public final static native void delete_InvertedLists_ScopedIds(long jarg1);\n  public final static native void InvertedLists_ScopedCodes_il_set(long jarg1, InvertedLists.ScopedCodes jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native long InvertedLists_ScopedCodes_il_get(long jarg1, InvertedLists.ScopedCodes jarg1_);\n  public final static native void InvertedLists_ScopedCodes_codes_set(long jarg1, InvertedLists.ScopedCodes jarg1_, long jarg2);\n  public final static native long InvertedLists_ScopedCodes_codes_get(long jarg1, InvertedLists.ScopedCodes jarg1_);\n  public final static native void InvertedLists_ScopedCodes_list_no_set(long jarg1, InvertedLists.ScopedCodes jarg1_, long jarg2);\n  public final static native long InvertedLists_ScopedCodes_list_no_get(long jarg1, InvertedLists.ScopedCodes jarg1_);\n  public final static native long new_InvertedLists_ScopedCodes__SWIG_0(long jarg1, InvertedLists jarg1_, long jarg2);\n  public final static native long new_InvertedLists_ScopedCodes__SWIG_1(long jarg1, InvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native long InvertedLists_ScopedCodes_get(long jarg1, InvertedLists.ScopedCodes jarg1_);\n  public final static native void delete_InvertedLists_ScopedCodes(long jarg1);\n  public final static native void ArrayInvertedLists_codes_set(long jarg1, ArrayInvertedLists jarg1_, long jarg2, ByteVectorVector jarg2_);\n  public final static native long ArrayInvertedLists_codes_get(long jarg1, ArrayInvertedLists jarg1_);\n  public final static native void ArrayInvertedLists_ids_set(long jarg1, ArrayInvertedLists jarg1_, long jarg2);\n  public final static native long ArrayInvertedLists_ids_get(long jarg1, ArrayInvertedLists jarg1_);\n  public final static native long new_ArrayInvertedLists(long jarg1, long jarg2);\n  public final static native long ArrayInvertedLists_list_size(long jarg1, ArrayInvertedLists jarg1_, long jarg2);\n  public final static native long ArrayInvertedLists_get_codes(long jarg1, ArrayInvertedLists jarg1_, long jarg2);\n  public final static native long ArrayInvertedLists_get_ids(long jarg1, ArrayInvertedLists jarg1_, long jarg2);\n  public final static native long ArrayInvertedLists_add_entries(long jarg1, ArrayInvertedLists jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void ArrayInvertedLists_update_entries(long jarg1, ArrayInvertedLists jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_, long jarg6);\n  public final static native void ArrayInvertedLists_resize(long jarg1, ArrayInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void delete_ArrayInvertedLists(long jarg1);\n  public final static native long ReadOnlyInvertedLists_add_entries(long jarg1, ReadOnlyInvertedLists jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void ReadOnlyInvertedLists_update_entries(long jarg1, ReadOnlyInvertedLists jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_, long jarg6);\n  public final static native void ReadOnlyInvertedLists_resize(long jarg1, ReadOnlyInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void delete_ReadOnlyInvertedLists(long jarg1);\n  public final static native void HStackInvertedLists_ils_set(long jarg1, HStackInvertedLists jarg1_, long jarg2);\n  public final static native long HStackInvertedLists_ils_get(long jarg1, HStackInvertedLists jarg1_);\n  public final static native long new_HStackInvertedLists(int jarg1, long jarg2);\n  public final static native long HStackInvertedLists_list_size(long jarg1, HStackInvertedLists jarg1_, long jarg2);\n  public final static native long HStackInvertedLists_get_codes(long jarg1, HStackInvertedLists jarg1_, long jarg2);\n  public final static native long HStackInvertedLists_get_ids(long jarg1, HStackInvertedLists jarg1_, long jarg2);\n  public final static native void HStackInvertedLists_prefetch_lists(long jarg1, HStackInvertedLists jarg1_, long jarg2, LongVector jarg2_, int jarg3);\n  public final static native void HStackInvertedLists_release_codes(long jarg1, HStackInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void HStackInvertedLists_release_ids(long jarg1, HStackInvertedLists jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native long HStackInvertedLists_get_single_id(long jarg1, HStackInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native long HStackInvertedLists_get_single_code(long jarg1, HStackInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void delete_HStackInvertedLists(long jarg1);\n  public final static native void SliceInvertedLists_il_set(long jarg1, SliceInvertedLists jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native long SliceInvertedLists_il_get(long jarg1, SliceInvertedLists jarg1_);\n  public final static native void SliceInvertedLists_i0_set(long jarg1, SliceInvertedLists jarg1_, long jarg2);\n  public final static native long SliceInvertedLists_i0_get(long jarg1, SliceInvertedLists jarg1_);\n  public final static native void SliceInvertedLists_i1_set(long jarg1, SliceInvertedLists jarg1_, long jarg2);\n  public final static native long SliceInvertedLists_i1_get(long jarg1, SliceInvertedLists jarg1_);\n  public final static native long new_SliceInvertedLists(long jarg1, InvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native long SliceInvertedLists_list_size(long jarg1, SliceInvertedLists jarg1_, long jarg2);\n  public final static native long SliceInvertedLists_get_codes(long jarg1, SliceInvertedLists jarg1_, long jarg2);\n  public final static native long SliceInvertedLists_get_ids(long jarg1, SliceInvertedLists jarg1_, long jarg2);\n  public final static native void SliceInvertedLists_release_codes(long jarg1, SliceInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void SliceInvertedLists_release_ids(long jarg1, SliceInvertedLists jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native long SliceInvertedLists_get_single_id(long jarg1, SliceInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native long SliceInvertedLists_get_single_code(long jarg1, SliceInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void SliceInvertedLists_prefetch_lists(long jarg1, SliceInvertedLists jarg1_, long jarg2, LongVector jarg2_, int jarg3);\n  public final static native void delete_SliceInvertedLists(long jarg1);\n  public final static native void VStackInvertedLists_ils_set(long jarg1, VStackInvertedLists jarg1_, long jarg2);\n  public final static native long VStackInvertedLists_ils_get(long jarg1, VStackInvertedLists jarg1_);\n  public final static native void VStackInvertedLists_cumsz_set(long jarg1, VStackInvertedLists jarg1_, long jarg2);\n  public final static native long VStackInvertedLists_cumsz_get(long jarg1, VStackInvertedLists jarg1_);\n  public final static native long new_VStackInvertedLists(int jarg1, long jarg2);\n  public final static native long VStackInvertedLists_list_size(long jarg1, VStackInvertedLists jarg1_, long jarg2);\n  public final static native long VStackInvertedLists_get_codes(long jarg1, VStackInvertedLists jarg1_, long jarg2);\n  public final static native long VStackInvertedLists_get_ids(long jarg1, VStackInvertedLists jarg1_, long jarg2);\n  public final static native void VStackInvertedLists_release_codes(long jarg1, VStackInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void VStackInvertedLists_release_ids(long jarg1, VStackInvertedLists jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native long VStackInvertedLists_get_single_id(long jarg1, VStackInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native long VStackInvertedLists_get_single_code(long jarg1, VStackInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void VStackInvertedLists_prefetch_lists(long jarg1, VStackInvertedLists jarg1_, long jarg2, LongVector jarg2_, int jarg3);\n  public final static native void delete_VStackInvertedLists(long jarg1);\n  public final static native void MaskedInvertedLists_il0_set(long jarg1, MaskedInvertedLists jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native long MaskedInvertedLists_il0_get(long jarg1, MaskedInvertedLists jarg1_);\n  public final static native void MaskedInvertedLists_il1_set(long jarg1, MaskedInvertedLists jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native long MaskedInvertedLists_il1_get(long jarg1, MaskedInvertedLists jarg1_);\n  public final static native long new_MaskedInvertedLists(long jarg1, InvertedLists jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native long MaskedInvertedLists_list_size(long jarg1, MaskedInvertedLists jarg1_, long jarg2);\n  public final static native long MaskedInvertedLists_get_codes(long jarg1, MaskedInvertedLists jarg1_, long jarg2);\n  public final static native long MaskedInvertedLists_get_ids(long jarg1, MaskedInvertedLists jarg1_, long jarg2);\n  public final static native void MaskedInvertedLists_release_codes(long jarg1, MaskedInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void MaskedInvertedLists_release_ids(long jarg1, MaskedInvertedLists jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native long MaskedInvertedLists_get_single_id(long jarg1, MaskedInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native long MaskedInvertedLists_get_single_code(long jarg1, MaskedInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void MaskedInvertedLists_prefetch_lists(long jarg1, MaskedInvertedLists jarg1_, long jarg2, LongVector jarg2_, int jarg3);\n  public final static native void delete_MaskedInvertedLists(long jarg1);\n  public final static native void StopWordsInvertedLists_il0_set(long jarg1, StopWordsInvertedLists jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native long StopWordsInvertedLists_il0_get(long jarg1, StopWordsInvertedLists jarg1_);\n  public final static native void StopWordsInvertedLists_maxsize_set(long jarg1, StopWordsInvertedLists jarg1_, long jarg2);\n  public final static native long StopWordsInvertedLists_maxsize_get(long jarg1, StopWordsInvertedLists jarg1_);\n  public final static native long new_StopWordsInvertedLists(long jarg1, InvertedLists jarg1_, long jarg2);\n  public final static native long StopWordsInvertedLists_list_size(long jarg1, StopWordsInvertedLists jarg1_, long jarg2);\n  public final static native long StopWordsInvertedLists_get_codes(long jarg1, StopWordsInvertedLists jarg1_, long jarg2);\n  public final static native long StopWordsInvertedLists_get_ids(long jarg1, StopWordsInvertedLists jarg1_, long jarg2);\n  public final static native void StopWordsInvertedLists_release_codes(long jarg1, StopWordsInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void StopWordsInvertedLists_release_ids(long jarg1, StopWordsInvertedLists jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native long StopWordsInvertedLists_get_single_id(long jarg1, StopWordsInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native long StopWordsInvertedLists_get_single_code(long jarg1, StopWordsInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void StopWordsInvertedLists_prefetch_lists(long jarg1, StopWordsInvertedLists jarg1_, long jarg2, LongVector jarg2_, int jarg3);\n  public final static native void delete_StopWordsInvertedLists(long jarg1);\n  public final static native void Level1Quantizer_quantizer_set(long jarg1, Level1Quantizer jarg1_, long jarg2, Index jarg2_);\n  public final static native long Level1Quantizer_quantizer_get(long jarg1, Level1Quantizer jarg1_);\n  public final static native void Level1Quantizer_nlist_set(long jarg1, Level1Quantizer jarg1_, long jarg2);\n  public final static native long Level1Quantizer_nlist_get(long jarg1, Level1Quantizer jarg1_);\n  public final static native void Level1Quantizer_quantizer_trains_alone_set(long jarg1, Level1Quantizer jarg1_, char jarg2);\n  public final static native char Level1Quantizer_quantizer_trains_alone_get(long jarg1, Level1Quantizer jarg1_);\n  public final static native void Level1Quantizer_own_fields_set(long jarg1, Level1Quantizer jarg1_, boolean jarg2);\n  public final static native boolean Level1Quantizer_own_fields_get(long jarg1, Level1Quantizer jarg1_);\n  public final static native void Level1Quantizer_cp_set(long jarg1, Level1Quantizer jarg1_, long jarg2, ClusteringParameters jarg2_);\n  public final static native long Level1Quantizer_cp_get(long jarg1, Level1Quantizer jarg1_);\n  public final static native void Level1Quantizer_clustering_index_set(long jarg1, Level1Quantizer jarg1_, long jarg2, Index jarg2_);\n  public final static native long Level1Quantizer_clustering_index_get(long jarg1, Level1Quantizer jarg1_);\n  public final static native void Level1Quantizer_train_q1(long jarg1, Level1Quantizer jarg1_, long jarg2, long jarg3, boolean jarg4, int jarg5);\n  public final static native long Level1Quantizer_coarse_code_size(long jarg1, Level1Quantizer jarg1_);\n  public final static native void Level1Quantizer_encode_listno(long jarg1, Level1Quantizer jarg1_, long jarg2, long jarg3);\n  public final static native long Level1Quantizer_decode_listno(long jarg1, Level1Quantizer jarg1_, long jarg2);\n  public final static native long new_Level1Quantizer__SWIG_0(long jarg1, Index jarg1_, long jarg2);\n  public final static native long new_Level1Quantizer__SWIG_1();\n  public final static native void delete_Level1Quantizer(long jarg1);\n  public final static native void IVFSearchParameters_nprobe_set(long jarg1, IVFSearchParameters jarg1_, long jarg2);\n  public final static native long IVFSearchParameters_nprobe_get(long jarg1, IVFSearchParameters jarg1_);\n  public final static native void IVFSearchParameters_max_codes_set(long jarg1, IVFSearchParameters jarg1_, long jarg2);\n  public final static native long IVFSearchParameters_max_codes_get(long jarg1, IVFSearchParameters jarg1_);\n  public final static native long new_IVFSearchParameters();\n  public final static native void delete_IVFSearchParameters(long jarg1);\n  public final static native void IndexIVF_invlists_set(long jarg1, IndexIVF jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native long IndexIVF_invlists_get(long jarg1, IndexIVF jarg1_);\n  public final static native void IndexIVF_own_invlists_set(long jarg1, IndexIVF jarg1_, boolean jarg2);\n  public final static native boolean IndexIVF_own_invlists_get(long jarg1, IndexIVF jarg1_);\n  public final static native void IndexIVF_code_size_set(long jarg1, IndexIVF jarg1_, long jarg2);\n  public final static native long IndexIVF_code_size_get(long jarg1, IndexIVF jarg1_);\n  public final static native void IndexIVF_nprobe_set(long jarg1, IndexIVF jarg1_, long jarg2);\n  public final static native long IndexIVF_nprobe_get(long jarg1, IndexIVF jarg1_);\n  public final static native void IndexIVF_max_codes_set(long jarg1, IndexIVF jarg1_, long jarg2);\n  public final static native long IndexIVF_max_codes_get(long jarg1, IndexIVF jarg1_);\n  public final static native void IndexIVF_parallel_mode_set(long jarg1, IndexIVF jarg1_, int jarg2);\n  public final static native int IndexIVF_parallel_mode_get(long jarg1, IndexIVF jarg1_);\n  public final static native int IndexIVF_PARALLEL_MODE_NO_HEAP_INIT_get(long jarg1, IndexIVF jarg1_);\n  public final static native void IndexIVF_direct_map_set(long jarg1, IndexIVF jarg1_, long jarg2);\n  public final static native long IndexIVF_direct_map_get(long jarg1, IndexIVF jarg1_);\n  public final static native void IndexIVF_reset(long jarg1, IndexIVF jarg1_);\n  public final static native void IndexIVF_train(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3);\n  public final static native void IndexIVF_add(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3);\n  public final static native void IndexIVF_add_with_ids(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void IndexIVF_add_core(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, LongVector jarg5_);\n  public final static native void IndexIVF_encode_vectors__SWIG_0(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, boolean jarg6);\n  public final static native void IndexIVF_encode_vectors__SWIG_1(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void IndexIVF_add_sa_codes(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void IndexIVF_train_residual(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3);\n  public final static native void IndexIVF_search_preassigned__SWIG_0(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, long jarg8, LongVector jarg8_, boolean jarg9, long jarg10, IVFSearchParameters jarg10_, long jarg11, IndexIVFStats jarg11_);\n  public final static native void IndexIVF_search_preassigned__SWIG_1(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, long jarg8, LongVector jarg8_, boolean jarg9, long jarg10, IVFSearchParameters jarg10_);\n  public final static native void IndexIVF_search_preassigned__SWIG_2(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, long jarg8, LongVector jarg8_, boolean jarg9);\n  public final static native void IndexIVF_search(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexIVF_range_search(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, float jarg4, long jarg5, RangeSearchResult jarg5_);\n  public final static native void IndexIVF_range_search_preassigned__SWIG_0(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, float jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, RangeSearchResult jarg7_, boolean jarg8, long jarg9, IVFSearchParameters jarg9_, long jarg10, IndexIVFStats jarg10_);\n  public final static native void IndexIVF_range_search_preassigned__SWIG_1(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, float jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, RangeSearchResult jarg7_, boolean jarg8, long jarg9, IVFSearchParameters jarg9_);\n  public final static native void IndexIVF_range_search_preassigned__SWIG_2(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, float jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, RangeSearchResult jarg7_, boolean jarg8);\n  public final static native void IndexIVF_range_search_preassigned__SWIG_3(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, float jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, RangeSearchResult jarg7_);\n  public final static native long IndexIVF_get_InvertedListScanner__SWIG_0(long jarg1, IndexIVF jarg1_, boolean jarg2);\n  public final static native long IndexIVF_get_InvertedListScanner__SWIG_1(long jarg1, IndexIVF jarg1_);\n  public final static native void IndexIVF_reconstruct(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3);\n  public final static native void IndexIVF_update_vectors(long jarg1, IndexIVF jarg1_, int jarg2, long jarg3, LongVector jarg3_, long jarg4);\n  public final static native void IndexIVF_reconstruct_n(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexIVF_search_and_reconstruct(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_, long jarg7);\n  public final static native void IndexIVF_reconstruct_from_offset(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native long IndexIVF_remove_ids(long jarg1, IndexIVF jarg1_, long jarg2, IDSelector jarg2_);\n  public final static native void IndexIVF_check_compatible_for_merge(long jarg1, IndexIVF jarg1_, long jarg2, IndexIVF jarg2_);\n  public final static native void IndexIVF_merge_from(long jarg1, IndexIVF jarg1_, long jarg2, IndexIVF jarg2_, long jarg3);\n  public final static native void IndexIVF_copy_subset_to(long jarg1, IndexIVF jarg1_, long jarg2, IndexIVF jarg2_, int jarg3, long jarg4, long jarg5);\n  public final static native void delete_IndexIVF(long jarg1);\n  public final static native long IndexIVF_get_list_size(long jarg1, IndexIVF jarg1_, long jarg2);\n  public final static native void IndexIVF_make_direct_map__SWIG_0(long jarg1, IndexIVF jarg1_, boolean jarg2);\n  public final static native void IndexIVF_make_direct_map__SWIG_1(long jarg1, IndexIVF jarg1_);\n  public final static native void IndexIVF_set_direct_map_type(long jarg1, IndexIVF jarg1_, long jarg2);\n  public final static native void IndexIVF_replace_invlists__SWIG_0(long jarg1, IndexIVF jarg1_, long jarg2, InvertedLists jarg2_, boolean jarg3);\n  public final static native void IndexIVF_replace_invlists__SWIG_1(long jarg1, IndexIVF jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native long IndexIVF_sa_code_size(long jarg1, IndexIVF jarg1_);\n  public final static native void IndexIVF_sa_encode(long jarg1, IndexIVF jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexIVFStats_nq_set(long jarg1, IndexIVFStats jarg1_, long jarg2);\n  public final static native long IndexIVFStats_nq_get(long jarg1, IndexIVFStats jarg1_);\n  public final static native void IndexIVFStats_nlist_set(long jarg1, IndexIVFStats jarg1_, long jarg2);\n  public final static native long IndexIVFStats_nlist_get(long jarg1, IndexIVFStats jarg1_);\n  public final static native void IndexIVFStats_ndis_set(long jarg1, IndexIVFStats jarg1_, long jarg2);\n  public final static native long IndexIVFStats_ndis_get(long jarg1, IndexIVFStats jarg1_);\n  public final static native void IndexIVFStats_nheap_updates_set(long jarg1, IndexIVFStats jarg1_, long jarg2);\n  public final static native long IndexIVFStats_nheap_updates_get(long jarg1, IndexIVFStats jarg1_);\n  public final static native void IndexIVFStats_quantization_time_set(long jarg1, IndexIVFStats jarg1_, double jarg2);\n  public final static native double IndexIVFStats_quantization_time_get(long jarg1, IndexIVFStats jarg1_);\n  public final static native void IndexIVFStats_search_time_set(long jarg1, IndexIVFStats jarg1_, double jarg2);\n  public final static native double IndexIVFStats_search_time_get(long jarg1, IndexIVFStats jarg1_);\n  public final static native long new_IndexIVFStats();\n  public final static native void IndexIVFStats_reset(long jarg1, IndexIVFStats jarg1_);\n  public final static native void IndexIVFStats_add(long jarg1, IndexIVFStats jarg1_, long jarg2, IndexIVFStats jarg2_);\n  public final static native void delete_IndexIVFStats(long jarg1);\n  public final static native void indexIVF_stats_set(long jarg1, IndexIVFStats jarg1_);\n  public final static native long indexIVF_stats_get();\n  public final static native short[] hamdis_tab_ham_bytes_get();\n  public final static native void HammingComputer4_a0_set(long jarg1, HammingComputer4 jarg1_, long jarg2);\n  public final static native long HammingComputer4_a0_get(long jarg1, HammingComputer4 jarg1_);\n  public final static native long new_HammingComputer4__SWIG_0();\n  public final static native long new_HammingComputer4__SWIG_1(long jarg1, int jarg2);\n  public final static native void HammingComputer4_set(long jarg1, HammingComputer4 jarg1_, long jarg2, int jarg3);\n  public final static native int HammingComputer4_hamming(long jarg1, HammingComputer4 jarg1_, long jarg2);\n  public final static native void delete_HammingComputer4(long jarg1);\n  public final static native void HammingComputer8_a0_set(long jarg1, HammingComputer8 jarg1_, long jarg2);\n  public final static native long HammingComputer8_a0_get(long jarg1, HammingComputer8 jarg1_);\n  public final static native long new_HammingComputer8__SWIG_0();\n  public final static native long new_HammingComputer8__SWIG_1(long jarg1, int jarg2);\n  public final static native void HammingComputer8_set(long jarg1, HammingComputer8 jarg1_, long jarg2, int jarg3);\n  public final static native int HammingComputer8_hamming(long jarg1, HammingComputer8 jarg1_, long jarg2);\n  public final static native void delete_HammingComputer8(long jarg1);\n  public final static native void HammingComputer16_a0_set(long jarg1, HammingComputer16 jarg1_, long jarg2);\n  public final static native long HammingComputer16_a0_get(long jarg1, HammingComputer16 jarg1_);\n  public final static native void HammingComputer16_a1_set(long jarg1, HammingComputer16 jarg1_, long jarg2);\n  public final static native long HammingComputer16_a1_get(long jarg1, HammingComputer16 jarg1_);\n  public final static native long new_HammingComputer16__SWIG_0();\n  public final static native long new_HammingComputer16__SWIG_1(long jarg1, int jarg2);\n  public final static native void HammingComputer16_set(long jarg1, HammingComputer16 jarg1_, long jarg2, int jarg3);\n  public final static native int HammingComputer16_hamming(long jarg1, HammingComputer16 jarg1_, long jarg2);\n  public final static native void delete_HammingComputer16(long jarg1);\n  public final static native void HammingComputer20_a0_set(long jarg1, HammingComputer20 jarg1_, long jarg2);\n  public final static native long HammingComputer20_a0_get(long jarg1, HammingComputer20 jarg1_);\n  public final static native void HammingComputer20_a1_set(long jarg1, HammingComputer20 jarg1_, long jarg2);\n  public final static native long HammingComputer20_a1_get(long jarg1, HammingComputer20 jarg1_);\n  public final static native void HammingComputer20_a2_set(long jarg1, HammingComputer20 jarg1_, long jarg2);\n  public final static native long HammingComputer20_a2_get(long jarg1, HammingComputer20 jarg1_);\n  public final static native long new_HammingComputer20__SWIG_0();\n  public final static native long new_HammingComputer20__SWIG_1(long jarg1, int jarg2);\n  public final static native void HammingComputer20_set(long jarg1, HammingComputer20 jarg1_, long jarg2, int jarg3);\n  public final static native int HammingComputer20_hamming(long jarg1, HammingComputer20 jarg1_, long jarg2);\n  public final static native void delete_HammingComputer20(long jarg1);\n  public final static native void HammingComputer32_a0_set(long jarg1, HammingComputer32 jarg1_, long jarg2);\n  public final static native long HammingComputer32_a0_get(long jarg1, HammingComputer32 jarg1_);\n  public final static native void HammingComputer32_a1_set(long jarg1, HammingComputer32 jarg1_, long jarg2);\n  public final static native long HammingComputer32_a1_get(long jarg1, HammingComputer32 jarg1_);\n  public final static native void HammingComputer32_a2_set(long jarg1, HammingComputer32 jarg1_, long jarg2);\n  public final static native long HammingComputer32_a2_get(long jarg1, HammingComputer32 jarg1_);\n  public final static native void HammingComputer32_a3_set(long jarg1, HammingComputer32 jarg1_, long jarg2);\n  public final static native long HammingComputer32_a3_get(long jarg1, HammingComputer32 jarg1_);\n  public final static native long new_HammingComputer32__SWIG_0();\n  public final static native long new_HammingComputer32__SWIG_1(long jarg1, int jarg2);\n  public final static native void HammingComputer32_set(long jarg1, HammingComputer32 jarg1_, long jarg2, int jarg3);\n  public final static native int HammingComputer32_hamming(long jarg1, HammingComputer32 jarg1_, long jarg2);\n  public final static native void delete_HammingComputer32(long jarg1);\n  public final static native void HammingComputer64_a0_set(long jarg1, HammingComputer64 jarg1_, long jarg2);\n  public final static native long HammingComputer64_a0_get(long jarg1, HammingComputer64 jarg1_);\n  public final static native void HammingComputer64_a1_set(long jarg1, HammingComputer64 jarg1_, long jarg2);\n  public final static native long HammingComputer64_a1_get(long jarg1, HammingComputer64 jarg1_);\n  public final static native void HammingComputer64_a2_set(long jarg1, HammingComputer64 jarg1_, long jarg2);\n  public final static native long HammingComputer64_a2_get(long jarg1, HammingComputer64 jarg1_);\n  public final static native void HammingComputer64_a3_set(long jarg1, HammingComputer64 jarg1_, long jarg2);\n  public final static native long HammingComputer64_a3_get(long jarg1, HammingComputer64 jarg1_);\n  public final static native void HammingComputer64_a4_set(long jarg1, HammingComputer64 jarg1_, long jarg2);\n  public final static native long HammingComputer64_a4_get(long jarg1, HammingComputer64 jarg1_);\n  public final static native void HammingComputer64_a5_set(long jarg1, HammingComputer64 jarg1_, long jarg2);\n  public final static native long HammingComputer64_a5_get(long jarg1, HammingComputer64 jarg1_);\n  public final static native void HammingComputer64_a6_set(long jarg1, HammingComputer64 jarg1_, long jarg2);\n  public final static native long HammingComputer64_a6_get(long jarg1, HammingComputer64 jarg1_);\n  public final static native void HammingComputer64_a7_set(long jarg1, HammingComputer64 jarg1_, long jarg2);\n  public final static native long HammingComputer64_a7_get(long jarg1, HammingComputer64 jarg1_);\n  public final static native long new_HammingComputer64__SWIG_0();\n  public final static native long new_HammingComputer64__SWIG_1(long jarg1, int jarg2);\n  public final static native void HammingComputer64_set(long jarg1, HammingComputer64 jarg1_, long jarg2, int jarg3);\n  public final static native int HammingComputer64_hamming(long jarg1, HammingComputer64 jarg1_, long jarg2);\n  public final static native void delete_HammingComputer64(long jarg1);\n  public final static native void HammingComputerDefault_a8_set(long jarg1, HammingComputerDefault jarg1_, long jarg2);\n  public final static native long HammingComputerDefault_a8_get(long jarg1, HammingComputerDefault jarg1_);\n  public final static native void HammingComputerDefault_quotient8_set(long jarg1, HammingComputerDefault jarg1_, int jarg2);\n  public final static native int HammingComputerDefault_quotient8_get(long jarg1, HammingComputerDefault jarg1_);\n  public final static native void HammingComputerDefault_remainder8_set(long jarg1, HammingComputerDefault jarg1_, int jarg2);\n  public final static native int HammingComputerDefault_remainder8_get(long jarg1, HammingComputerDefault jarg1_);\n  public final static native long new_HammingComputerDefault__SWIG_0();\n  public final static native long new_HammingComputerDefault__SWIG_1(long jarg1, int jarg2);\n  public final static native void HammingComputerDefault_set(long jarg1, HammingComputerDefault jarg1_, long jarg2, int jarg3);\n  public final static native int HammingComputerDefault_hamming(long jarg1, HammingComputerDefault jarg1_, long jarg2);\n  public final static native void delete_HammingComputerDefault(long jarg1);\n  public final static native void HammingComputerM8_a_set(long jarg1, HammingComputerM8 jarg1_, long jarg2);\n  public final static native long HammingComputerM8_a_get(long jarg1, HammingComputerM8 jarg1_);\n  public final static native void HammingComputerM8_n_set(long jarg1, HammingComputerM8 jarg1_, int jarg2);\n  public final static native int HammingComputerM8_n_get(long jarg1, HammingComputerM8 jarg1_);\n  public final static native long new_HammingComputerM8__SWIG_0();\n  public final static native long new_HammingComputerM8__SWIG_1(long jarg1, int jarg2);\n  public final static native void HammingComputerM8_set(long jarg1, HammingComputerM8 jarg1_, long jarg2, int jarg3);\n  public final static native int HammingComputerM8_hamming(long jarg1, HammingComputerM8 jarg1_, long jarg2);\n  public final static native void delete_HammingComputerM8(long jarg1);\n  public final static native void HammingComputerM4_a_set(long jarg1, HammingComputerM4 jarg1_, long jarg2);\n  public final static native long HammingComputerM4_a_get(long jarg1, HammingComputerM4 jarg1_);\n  public final static native void HammingComputerM4_n_set(long jarg1, HammingComputerM4 jarg1_, int jarg2);\n  public final static native int HammingComputerM4_n_get(long jarg1, HammingComputerM4 jarg1_);\n  public final static native long new_HammingComputerM4__SWIG_0();\n  public final static native long new_HammingComputerM4__SWIG_1(long jarg1, int jarg2);\n  public final static native void HammingComputerM4_set(long jarg1, HammingComputerM4 jarg1_, long jarg2, int jarg3);\n  public final static native int HammingComputerM4_hamming(long jarg1, HammingComputerM4 jarg1_, long jarg2);\n  public final static native void delete_HammingComputerM4(long jarg1);\n  public final static native int generalized_hamming_64(long jarg1);\n  public final static native void GenHammingComputer8_a0_set(long jarg1, GenHammingComputer8 jarg1_, long jarg2);\n  public final static native long GenHammingComputer8_a0_get(long jarg1, GenHammingComputer8 jarg1_);\n  public final static native long new_GenHammingComputer8(long jarg1, int jarg2);\n  public final static native int GenHammingComputer8_hamming(long jarg1, GenHammingComputer8 jarg1_, long jarg2);\n  public final static native void delete_GenHammingComputer8(long jarg1);\n  public final static native void GenHammingComputer16_a0_set(long jarg1, GenHammingComputer16 jarg1_, long jarg2);\n  public final static native long GenHammingComputer16_a0_get(long jarg1, GenHammingComputer16 jarg1_);\n  public final static native void GenHammingComputer16_a1_set(long jarg1, GenHammingComputer16 jarg1_, long jarg2);\n  public final static native long GenHammingComputer16_a1_get(long jarg1, GenHammingComputer16 jarg1_);\n  public final static native long new_GenHammingComputer16(long jarg1, int jarg2);\n  public final static native int GenHammingComputer16_hamming(long jarg1, GenHammingComputer16 jarg1_, long jarg2);\n  public final static native void delete_GenHammingComputer16(long jarg1);\n  public final static native void GenHammingComputer32_a0_set(long jarg1, GenHammingComputer32 jarg1_, long jarg2);\n  public final static native long GenHammingComputer32_a0_get(long jarg1, GenHammingComputer32 jarg1_);\n  public final static native void GenHammingComputer32_a1_set(long jarg1, GenHammingComputer32 jarg1_, long jarg2);\n  public final static native long GenHammingComputer32_a1_get(long jarg1, GenHammingComputer32 jarg1_);\n  public final static native void GenHammingComputer32_a2_set(long jarg1, GenHammingComputer32 jarg1_, long jarg2);\n  public final static native long GenHammingComputer32_a2_get(long jarg1, GenHammingComputer32 jarg1_);\n  public final static native void GenHammingComputer32_a3_set(long jarg1, GenHammingComputer32 jarg1_, long jarg2);\n  public final static native long GenHammingComputer32_a3_get(long jarg1, GenHammingComputer32 jarg1_);\n  public final static native long new_GenHammingComputer32(long jarg1, int jarg2);\n  public final static native int GenHammingComputer32_hamming(long jarg1, GenHammingComputer32 jarg1_, long jarg2);\n  public final static native void delete_GenHammingComputer32(long jarg1);\n  public final static native void GenHammingComputerM8_a_set(long jarg1, GenHammingComputerM8 jarg1_, long jarg2);\n  public final static native long GenHammingComputerM8_a_get(long jarg1, GenHammingComputerM8 jarg1_);\n  public final static native void GenHammingComputerM8_n_set(long jarg1, GenHammingComputerM8 jarg1_, int jarg2);\n  public final static native int GenHammingComputerM8_n_get(long jarg1, GenHammingComputerM8 jarg1_);\n  public final static native long new_GenHammingComputerM8(long jarg1, int jarg2);\n  public final static native int GenHammingComputerM8_hamming(long jarg1, GenHammingComputerM8 jarg1_, long jarg2);\n  public final static native void delete_GenHammingComputerM8(long jarg1);\n  public final static native void generalized_hammings_knn_hc__SWIG_0(long jarg1, long jarg2, long jarg3, long jarg4, long jarg5, int jarg6);\n  public final static native void generalized_hammings_knn_hc__SWIG_1(long jarg1, long jarg2, long jarg3, long jarg4, long jarg5);\n  public final static native void check_compatible_for_merge(long jarg1, Index jarg1_, long jarg2, Index jarg2_);\n  public final static native long extract_index_ivf__SWIG_0(long jarg1, Index jarg1_);\n  public final static native long try_extract_index_ivf__SWIG_0(long jarg1, Index jarg1_);\n  public final static native void merge_into(long jarg1, Index jarg1_, long jarg2, Index jarg2_, boolean jarg3);\n  public final static native void search_centroid(long jarg1, Index jarg1_, long jarg2, int jarg3, long jarg4, LongVector jarg4_);\n  public final static native void search_and_return_centroids(long jarg1, Index jarg1_, long jarg2, long jarg3, int jarg4, long jarg5, long jarg6, LongVector jarg6_, long jarg7, LongVector jarg7_, long jarg8, LongVector jarg8_);\n  public final static native void SlidingIndexWindow_index_set(long jarg1, SlidingIndexWindow jarg1_, long jarg2, Index jarg2_);\n  public final static native long SlidingIndexWindow_index_get(long jarg1, SlidingIndexWindow jarg1_);\n  public final static native void SlidingIndexWindow_ils_set(long jarg1, SlidingIndexWindow jarg1_, long jarg2, ArrayInvertedLists jarg2_);\n  public final static native long SlidingIndexWindow_ils_get(long jarg1, SlidingIndexWindow jarg1_);\n  public final static native void SlidingIndexWindow_n_slice_set(long jarg1, SlidingIndexWindow jarg1_, int jarg2);\n  public final static native int SlidingIndexWindow_n_slice_get(long jarg1, SlidingIndexWindow jarg1_);\n  public final static native void SlidingIndexWindow_nlist_set(long jarg1, SlidingIndexWindow jarg1_, long jarg2);\n  public final static native long SlidingIndexWindow_nlist_get(long jarg1, SlidingIndexWindow jarg1_);\n  public final static native void SlidingIndexWindow_sizes_set(long jarg1, SlidingIndexWindow jarg1_, long jarg2);\n  public final static native long SlidingIndexWindow_sizes_get(long jarg1, SlidingIndexWindow jarg1_);\n  public final static native long new_SlidingIndexWindow(long jarg1, Index jarg1_);\n  public final static native void SlidingIndexWindow_step(long jarg1, SlidingIndexWindow jarg1_, long jarg2, Index jarg2_, boolean jarg3);\n  public final static native void delete_SlidingIndexWindow(long jarg1);\n  public final static native long get_invlist_range(long jarg1, Index jarg1_, int jarg2, int jarg3);\n  public final static native void set_invlist_range(long jarg1, Index jarg1_, int jarg2, int jarg3, long jarg4, ArrayInvertedLists jarg4_);\n  public final static native void search_with_parameters__SWIG_0(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_, long jarg7, IVFSearchParameters jarg7_, long jarg8, long jarg9);\n  public final static native void search_with_parameters__SWIG_1(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_, long jarg7, IVFSearchParameters jarg7_, long jarg8);\n  public final static native void search_with_parameters__SWIG_2(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_, long jarg7, IVFSearchParameters jarg7_);\n  public final static native void range_search_with_parameters__SWIG_0(long jarg1, Index jarg1_, long jarg2, long jarg3, float jarg4, long jarg5, RangeSearchResult jarg5_, long jarg6, IVFSearchParameters jarg6_, long jarg7, long jarg8);\n  public final static native void range_search_with_parameters__SWIG_1(long jarg1, Index jarg1_, long jarg2, long jarg3, float jarg4, long jarg5, RangeSearchResult jarg5_, long jarg6, IVFSearchParameters jarg6_, long jarg7);\n  public final static native void range_search_with_parameters__SWIG_2(long jarg1, Index jarg1_, long jarg2, long jarg3, float jarg4, long jarg5, RangeSearchResult jarg5_, long jarg6, IVFSearchParameters jarg6_);\n  public final static native void IndexScalarQuantizer_sq_set(long jarg1, IndexScalarQuantizer jarg1_, long jarg2);\n  public final static native long IndexScalarQuantizer_sq_get(long jarg1, IndexScalarQuantizer jarg1_);\n  public final static native long new_IndexScalarQuantizer__SWIG_0(int jarg1, long jarg2, int jarg3);\n  public final static native long new_IndexScalarQuantizer__SWIG_1(int jarg1, long jarg2);\n  public final static native long new_IndexScalarQuantizer__SWIG_2();\n  public final static native void IndexScalarQuantizer_train(long jarg1, IndexScalarQuantizer jarg1_, long jarg2, long jarg3);\n  public final static native void IndexScalarQuantizer_search(long jarg1, IndexScalarQuantizer jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native long IndexScalarQuantizer_get_distance_computer(long jarg1, IndexScalarQuantizer jarg1_);\n  public final static native void IndexScalarQuantizer_sa_encode(long jarg1, IndexScalarQuantizer jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexScalarQuantizer_sa_decode(long jarg1, IndexScalarQuantizer jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void delete_IndexScalarQuantizer(long jarg1);\n  public final static native void IndexIVFScalarQuantizer_sq_set(long jarg1, IndexIVFScalarQuantizer jarg1_, long jarg2);\n  public final static native long IndexIVFScalarQuantizer_sq_get(long jarg1, IndexIVFScalarQuantizer jarg1_);\n  public final static native void IndexIVFScalarQuantizer_by_residual_set(long jarg1, IndexIVFScalarQuantizer jarg1_, boolean jarg2);\n  public final static native boolean IndexIVFScalarQuantizer_by_residual_get(long jarg1, IndexIVFScalarQuantizer jarg1_);\n  public final static native long new_IndexIVFScalarQuantizer__SWIG_0(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, int jarg5, boolean jarg6);\n  public final static native long new_IndexIVFScalarQuantizer__SWIG_1(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, int jarg5);\n  public final static native long new_IndexIVFScalarQuantizer__SWIG_2(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native long new_IndexIVFScalarQuantizer__SWIG_3();\n  public final static native void IndexIVFScalarQuantizer_train_residual(long jarg1, IndexIVFScalarQuantizer jarg1_, long jarg2, long jarg3);\n  public final static native void IndexIVFScalarQuantizer_encode_vectors__SWIG_0(long jarg1, IndexIVFScalarQuantizer jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, boolean jarg6);\n  public final static native void IndexIVFScalarQuantizer_encode_vectors__SWIG_1(long jarg1, IndexIVFScalarQuantizer jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void IndexIVFScalarQuantizer_add_core(long jarg1, IndexIVFScalarQuantizer jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, LongVector jarg5_);\n  public final static native long IndexIVFScalarQuantizer_get_InvertedListScanner(long jarg1, IndexIVFScalarQuantizer jarg1_, boolean jarg2);\n  public final static native void IndexIVFScalarQuantizer_reconstruct_from_offset(long jarg1, IndexIVFScalarQuantizer jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexIVFScalarQuantizer_sa_decode(long jarg1, IndexIVFScalarQuantizer jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void delete_IndexIVFScalarQuantizer(long jarg1);\n  public final static native void HNSW_MinimaxHeap_n_set(long jarg1, HNSW.MinimaxHeap jarg1_, int jarg2);\n  public final static native int HNSW_MinimaxHeap_n_get(long jarg1, HNSW.MinimaxHeap jarg1_);\n  public final static native void HNSW_MinimaxHeap_k_set(long jarg1, HNSW.MinimaxHeap jarg1_, int jarg2);\n  public final static native int HNSW_MinimaxHeap_k_get(long jarg1, HNSW.MinimaxHeap jarg1_);\n  public final static native void HNSW_MinimaxHeap_nvalid_set(long jarg1, HNSW.MinimaxHeap jarg1_, int jarg2);\n  public final static native int HNSW_MinimaxHeap_nvalid_get(long jarg1, HNSW.MinimaxHeap jarg1_);\n  public final static native void HNSW_MinimaxHeap_ids_set(long jarg1, HNSW.MinimaxHeap jarg1_, long jarg2, IntVector jarg2_);\n  public final static native long HNSW_MinimaxHeap_ids_get(long jarg1, HNSW.MinimaxHeap jarg1_);\n  public final static native void HNSW_MinimaxHeap_dis_set(long jarg1, HNSW.MinimaxHeap jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long HNSW_MinimaxHeap_dis_get(long jarg1, HNSW.MinimaxHeap jarg1_);\n  public final static native long new_HNSW_MinimaxHeap(int jarg1);\n  public final static native void HNSW_MinimaxHeap_push(long jarg1, HNSW.MinimaxHeap jarg1_, int jarg2, float jarg3);\n  public final static native float HNSW_MinimaxHeap_max(long jarg1, HNSW.MinimaxHeap jarg1_);\n  public final static native int HNSW_MinimaxHeap_size(long jarg1, HNSW.MinimaxHeap jarg1_);\n  public final static native void HNSW_MinimaxHeap_clear(long jarg1, HNSW.MinimaxHeap jarg1_);\n  public final static native int HNSW_MinimaxHeap_pop_min__SWIG_0(long jarg1, HNSW.MinimaxHeap jarg1_, long jarg2);\n  public final static native int HNSW_MinimaxHeap_pop_min__SWIG_1(long jarg1, HNSW.MinimaxHeap jarg1_);\n  public final static native int HNSW_MinimaxHeap_count_below(long jarg1, HNSW.MinimaxHeap jarg1_, float jarg2);\n  public final static native void delete_HNSW_MinimaxHeap(long jarg1);\n  public final static native void HNSW_NodeDistCloser_d_set(long jarg1, HNSW.NodeDistCloser jarg1_, float jarg2);\n  public final static native float HNSW_NodeDistCloser_d_get(long jarg1, HNSW.NodeDistCloser jarg1_);\n  public final static native void HNSW_NodeDistCloser_id_set(long jarg1, HNSW.NodeDistCloser jarg1_, int jarg2);\n  public final static native int HNSW_NodeDistCloser_id_get(long jarg1, HNSW.NodeDistCloser jarg1_);\n  public final static native long new_HNSW_NodeDistCloser(float jarg1, int jarg2);\n  public final static native void delete_HNSW_NodeDistCloser(long jarg1);\n  public final static native void HNSW_NodeDistFarther_d_set(long jarg1, HNSW.NodeDistFarther jarg1_, float jarg2);\n  public final static native float HNSW_NodeDistFarther_d_get(long jarg1, HNSW.NodeDistFarther jarg1_);\n  public final static native void HNSW_NodeDistFarther_id_set(long jarg1, HNSW.NodeDistFarther jarg1_, int jarg2);\n  public final static native int HNSW_NodeDistFarther_id_get(long jarg1, HNSW.NodeDistFarther jarg1_);\n  public final static native long new_HNSW_NodeDistFarther(float jarg1, int jarg2);\n  public final static native void delete_HNSW_NodeDistFarther(long jarg1);\n  public final static native void HNSW_assign_probas_set(long jarg1, HNSW jarg1_, long jarg2, DoubleVector jarg2_);\n  public final static native long HNSW_assign_probas_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_cum_nneighbor_per_level_set(long jarg1, HNSW jarg1_, long jarg2, IntVector jarg2_);\n  public final static native long HNSW_cum_nneighbor_per_level_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_levels_set(long jarg1, HNSW jarg1_, long jarg2, IntVector jarg2_);\n  public final static native long HNSW_levels_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_offsets_set(long jarg1, HNSW jarg1_, long jarg2, Uint64Vector jarg2_);\n  public final static native long HNSW_offsets_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_neighbors_set(long jarg1, HNSW jarg1_, long jarg2, IntVector jarg2_);\n  public final static native long HNSW_neighbors_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_entry_point_set(long jarg1, HNSW jarg1_, int jarg2);\n  public final static native int HNSW_entry_point_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_rng_set(long jarg1, HNSW jarg1_, long jarg2);\n  public final static native long HNSW_rng_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_max_level_set(long jarg1, HNSW jarg1_, int jarg2);\n  public final static native int HNSW_max_level_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_efConstruction_set(long jarg1, HNSW jarg1_, int jarg2);\n  public final static native int HNSW_efConstruction_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_efSearch_set(long jarg1, HNSW jarg1_, int jarg2);\n  public final static native int HNSW_efSearch_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_check_relative_distance_set(long jarg1, HNSW jarg1_, boolean jarg2);\n  public final static native boolean HNSW_check_relative_distance_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_upper_beam_set(long jarg1, HNSW jarg1_, int jarg2);\n  public final static native int HNSW_upper_beam_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_search_bounded_queue_set(long jarg1, HNSW jarg1_, boolean jarg2);\n  public final static native boolean HNSW_search_bounded_queue_get(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_set_default_probas(long jarg1, HNSW jarg1_, int jarg2, float jarg3);\n  public final static native void HNSW_set_nb_neighbors(long jarg1, HNSW jarg1_, int jarg2, int jarg3);\n  public final static native int HNSW_nb_neighbors(long jarg1, HNSW jarg1_, int jarg2);\n  public final static native int HNSW_cum_nb_neighbors(long jarg1, HNSW jarg1_, int jarg2);\n  public final static native void HNSW_neighbor_range(long jarg1, HNSW jarg1_, long jarg2, int jarg3, long jarg4, long jarg5);\n  public final static native long new_HNSW__SWIG_0(int jarg1);\n  public final static native long new_HNSW__SWIG_1();\n  public final static native int HNSW_random_level(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_fill_with_random_links(long jarg1, HNSW jarg1_, long jarg2);\n  public final static native void HNSW_add_links_starting_from(long jarg1, HNSW jarg1_, long jarg2, DistanceComputer jarg2_, int jarg3, int jarg4, float jarg5, int jarg6, long jarg7, long jarg8, VisitedTable jarg8_);\n  public final static native void HNSW_add_with_locks(long jarg1, HNSW jarg1_, long jarg2, DistanceComputer jarg2_, int jarg3, int jarg4, long jarg5, long jarg6, VisitedTable jarg6_);\n  public final static native int HNSW_search_from_candidates__SWIG_0(long jarg1, HNSW jarg1_, long jarg2, DistanceComputer jarg2_, int jarg3, long jarg4, LongVector jarg4_, long jarg5, long jarg6, HNSW.MinimaxHeap jarg6_, long jarg7, VisitedTable jarg7_, long jarg8, HNSWStats jarg8_, int jarg9, int jarg10);\n  public final static native int HNSW_search_from_candidates__SWIG_1(long jarg1, HNSW jarg1_, long jarg2, DistanceComputer jarg2_, int jarg3, long jarg4, LongVector jarg4_, long jarg5, long jarg6, HNSW.MinimaxHeap jarg6_, long jarg7, VisitedTable jarg7_, long jarg8, HNSWStats jarg8_, int jarg9);\n  public final static native long HNSW_search_from_candidate_unbounded(long jarg1, HNSW jarg1_, long jarg2, long jarg3, DistanceComputer jarg3_, int jarg4, long jarg5, VisitedTable jarg5_, long jarg6, HNSWStats jarg6_);\n  public final static native long HNSW_search(long jarg1, HNSW jarg1_, long jarg2, DistanceComputer jarg2_, int jarg3, long jarg4, LongVector jarg4_, long jarg5, long jarg6, VisitedTable jarg6_);\n  public final static native void HNSW_reset(long jarg1, HNSW jarg1_);\n  public final static native void HNSW_clear_neighbor_tables(long jarg1, HNSW jarg1_, int jarg2);\n  public final static native void HNSW_print_neighbor_stats(long jarg1, HNSW jarg1_, int jarg2);\n  public final static native int HNSW_prepare_level_tab__SWIG_0(long jarg1, HNSW jarg1_, long jarg2, boolean jarg3);\n  public final static native int HNSW_prepare_level_tab__SWIG_1(long jarg1, HNSW jarg1_, long jarg2);\n  public final static native void HNSW_shrink_neighbor_list(long jarg1, DistanceComputer jarg1_, long jarg2, long jarg3, int jarg4);\n  public final static native void delete_HNSW(long jarg1);\n  public final static native void HNSWStats_n1_set(long jarg1, HNSWStats jarg1_, long jarg2);\n  public final static native long HNSWStats_n1_get(long jarg1, HNSWStats jarg1_);\n  public final static native void HNSWStats_n2_set(long jarg1, HNSWStats jarg1_, long jarg2);\n  public final static native long HNSWStats_n2_get(long jarg1, HNSWStats jarg1_);\n  public final static native void HNSWStats_n3_set(long jarg1, HNSWStats jarg1_, long jarg2);\n  public final static native long HNSWStats_n3_get(long jarg1, HNSWStats jarg1_);\n  public final static native void HNSWStats_ndis_set(long jarg1, HNSWStats jarg1_, long jarg2);\n  public final static native long HNSWStats_ndis_get(long jarg1, HNSWStats jarg1_);\n  public final static native void HNSWStats_nreorder_set(long jarg1, HNSWStats jarg1_, long jarg2);\n  public final static native long HNSWStats_nreorder_get(long jarg1, HNSWStats jarg1_);\n  public final static native long new_HNSWStats__SWIG_0(long jarg1, long jarg2, long jarg3, long jarg4, long jarg5);\n  public final static native long new_HNSWStats__SWIG_1(long jarg1, long jarg2, long jarg3, long jarg4);\n  public final static native long new_HNSWStats__SWIG_2(long jarg1, long jarg2, long jarg3);\n  public final static native long new_HNSWStats__SWIG_3(long jarg1, long jarg2);\n  public final static native long new_HNSWStats__SWIG_4(long jarg1);\n  public final static native long new_HNSWStats__SWIG_5();\n  public final static native void HNSWStats_reset(long jarg1, HNSWStats jarg1_);\n  public final static native void HNSWStats_combine(long jarg1, HNSWStats jarg1_, long jarg2, HNSWStats jarg2_);\n  public final static native void delete_HNSWStats(long jarg1);\n  public final static native void hnsw_stats_set(long jarg1, HNSWStats jarg1_);\n  public final static native long hnsw_stats_get();\n  public final static native long ReconstructFromNeighbors_index_get(long jarg1, ReconstructFromNeighbors jarg1_);\n  public final static native void ReconstructFromNeighbors_M_set(long jarg1, ReconstructFromNeighbors jarg1_, long jarg2);\n  public final static native long ReconstructFromNeighbors_M_get(long jarg1, ReconstructFromNeighbors jarg1_);\n  public final static native void ReconstructFromNeighbors_k_set(long jarg1, ReconstructFromNeighbors jarg1_, long jarg2);\n  public final static native long ReconstructFromNeighbors_k_get(long jarg1, ReconstructFromNeighbors jarg1_);\n  public final static native void ReconstructFromNeighbors_nsq_set(long jarg1, ReconstructFromNeighbors jarg1_, long jarg2);\n  public final static native long ReconstructFromNeighbors_nsq_get(long jarg1, ReconstructFromNeighbors jarg1_);\n  public final static native void ReconstructFromNeighbors_code_size_set(long jarg1, ReconstructFromNeighbors jarg1_, long jarg2);\n  public final static native long ReconstructFromNeighbors_code_size_get(long jarg1, ReconstructFromNeighbors jarg1_);\n  public final static native void ReconstructFromNeighbors_k_reorder_set(long jarg1, ReconstructFromNeighbors jarg1_, int jarg2);\n  public final static native int ReconstructFromNeighbors_k_reorder_get(long jarg1, ReconstructFromNeighbors jarg1_);\n  public final static native void ReconstructFromNeighbors_codebook_set(long jarg1, ReconstructFromNeighbors jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long ReconstructFromNeighbors_codebook_get(long jarg1, ReconstructFromNeighbors jarg1_);\n  public final static native void ReconstructFromNeighbors_codes_set(long jarg1, ReconstructFromNeighbors jarg1_, long jarg2, ByteVector jarg2_);\n  public final static native long ReconstructFromNeighbors_codes_get(long jarg1, ReconstructFromNeighbors jarg1_);\n  public final static native void ReconstructFromNeighbors_ntotal_set(long jarg1, ReconstructFromNeighbors jarg1_, long jarg2);\n  public final static native long ReconstructFromNeighbors_ntotal_get(long jarg1, ReconstructFromNeighbors jarg1_);\n  public final static native void ReconstructFromNeighbors_d_set(long jarg1, ReconstructFromNeighbors jarg1_, long jarg2);\n  public final static native long ReconstructFromNeighbors_d_get(long jarg1, ReconstructFromNeighbors jarg1_);\n  public final static native void ReconstructFromNeighbors_dsub_set(long jarg1, ReconstructFromNeighbors jarg1_, long jarg2);\n  public final static native long ReconstructFromNeighbors_dsub_get(long jarg1, ReconstructFromNeighbors jarg1_);\n  public final static native long new_ReconstructFromNeighbors__SWIG_0(long jarg1, IndexHNSW jarg1_, long jarg2, long jarg3);\n  public final static native long new_ReconstructFromNeighbors__SWIG_1(long jarg1, IndexHNSW jarg1_, long jarg2);\n  public final static native long new_ReconstructFromNeighbors__SWIG_2(long jarg1, IndexHNSW jarg1_);\n  public final static native void ReconstructFromNeighbors_add_codes(long jarg1, ReconstructFromNeighbors jarg1_, long jarg2, long jarg3);\n  public final static native long ReconstructFromNeighbors_compute_distances(long jarg1, ReconstructFromNeighbors jarg1_, long jarg2, long jarg3, LongVector jarg3_, long jarg4, long jarg5);\n  public final static native void ReconstructFromNeighbors_estimate_code(long jarg1, ReconstructFromNeighbors jarg1_, long jarg2, int jarg3, long jarg4);\n  public final static native void ReconstructFromNeighbors_reconstruct(long jarg1, ReconstructFromNeighbors jarg1_, int jarg2, long jarg3, long jarg4);\n  public final static native void ReconstructFromNeighbors_reconstruct_n(long jarg1, ReconstructFromNeighbors jarg1_, int jarg2, int jarg3, long jarg4);\n  public final static native void ReconstructFromNeighbors_get_neighbor_table(long jarg1, ReconstructFromNeighbors jarg1_, int jarg2, long jarg3);\n  public final static native void delete_ReconstructFromNeighbors(long jarg1);\n  public final static native void IndexHNSW_hnsw_set(long jarg1, IndexHNSW jarg1_, long jarg2, HNSW jarg2_);\n  public final static native long IndexHNSW_hnsw_get(long jarg1, IndexHNSW jarg1_);\n  public final static native void IndexHNSW_own_fields_set(long jarg1, IndexHNSW jarg1_, boolean jarg2);\n  public final static native boolean IndexHNSW_own_fields_get(long jarg1, IndexHNSW jarg1_);\n  public final static native void IndexHNSW_storage_set(long jarg1, IndexHNSW jarg1_, long jarg2, Index jarg2_);\n  public final static native long IndexHNSW_storage_get(long jarg1, IndexHNSW jarg1_);\n  public final static native void IndexHNSW_reconstruct_from_neighbors_set(long jarg1, IndexHNSW jarg1_, long jarg2, ReconstructFromNeighbors jarg2_);\n  public final static native long IndexHNSW_reconstruct_from_neighbors_get(long jarg1, IndexHNSW jarg1_);\n  public final static native long new_IndexHNSW__SWIG_0(int jarg1, int jarg2, int jarg3);\n  public final static native long new_IndexHNSW__SWIG_1(int jarg1, int jarg2);\n  public final static native long new_IndexHNSW__SWIG_2(int jarg1);\n  public final static native long new_IndexHNSW__SWIG_3();\n  public final static native long new_IndexHNSW__SWIG_4(long jarg1, Index jarg1_, int jarg2);\n  public final static native long new_IndexHNSW__SWIG_5(long jarg1, Index jarg1_);\n  public final static native void delete_IndexHNSW(long jarg1);\n  public final static native void IndexHNSW_add(long jarg1, IndexHNSW jarg1_, long jarg2, long jarg3);\n  public final static native void IndexHNSW_train(long jarg1, IndexHNSW jarg1_, long jarg2, long jarg3);\n  public final static native void IndexHNSW_search(long jarg1, IndexHNSW jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexHNSW_reconstruct(long jarg1, IndexHNSW jarg1_, long jarg2, long jarg3);\n  public final static native void IndexHNSW_reset(long jarg1, IndexHNSW jarg1_);\n  public final static native void IndexHNSW_shrink_level_0_neighbors(long jarg1, IndexHNSW jarg1_, int jarg2);\n  public final static native void IndexHNSW_search_level_0__SWIG_0(long jarg1, IndexHNSW jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, long jarg7, long jarg8, LongVector jarg8_, int jarg9, int jarg10);\n  public final static native void IndexHNSW_search_level_0__SWIG_1(long jarg1, IndexHNSW jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, long jarg7, long jarg8, LongVector jarg8_, int jarg9);\n  public final static native void IndexHNSW_search_level_0__SWIG_2(long jarg1, IndexHNSW jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, long jarg7, long jarg8, LongVector jarg8_);\n  public final static native void IndexHNSW_init_level_0_from_knngraph(long jarg1, IndexHNSW jarg1_, int jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void IndexHNSW_init_level_0_from_entry_points(long jarg1, IndexHNSW jarg1_, int jarg2, long jarg3, long jarg4);\n  public final static native void IndexHNSW_reorder_links(long jarg1, IndexHNSW jarg1_);\n  public final static native void IndexHNSW_link_singletons(long jarg1, IndexHNSW jarg1_);\n  public final static native long new_IndexHNSWFlat__SWIG_0();\n  public final static native long new_IndexHNSWFlat__SWIG_1(int jarg1, int jarg2, int jarg3);\n  public final static native long new_IndexHNSWFlat__SWIG_2(int jarg1, int jarg2);\n  public final static native void delete_IndexHNSWFlat(long jarg1);\n  public final static native long new_IndexHNSWPQ__SWIG_0();\n  public final static native long new_IndexHNSWPQ__SWIG_1(int jarg1, int jarg2, int jarg3);\n  public final static native void IndexHNSWPQ_train(long jarg1, IndexHNSWPQ jarg1_, long jarg2, long jarg3);\n  public final static native void delete_IndexHNSWPQ(long jarg1);\n  public final static native long new_IndexHNSWSQ__SWIG_0();\n  public final static native long new_IndexHNSWSQ__SWIG_1(int jarg1, long jarg2, int jarg3, int jarg4);\n  public final static native long new_IndexHNSWSQ__SWIG_2(int jarg1, long jarg2, int jarg3);\n  public final static native void delete_IndexHNSWSQ(long jarg1);\n  public final static native long new_IndexHNSW2Level__SWIG_0();\n  public final static native long new_IndexHNSW2Level__SWIG_1(long jarg1, Index jarg1_, long jarg2, int jarg3, int jarg4);\n  public final static native void IndexHNSW2Level_flip_to_ivf(long jarg1, IndexHNSW2Level jarg1_);\n  public final static native void IndexHNSW2Level_search(long jarg1, IndexHNSW2Level jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void delete_IndexHNSW2Level(long jarg1);\n  public final static native long new_IndexIVFFlat__SWIG_0(long jarg1, Index jarg1_, long jarg2, long jarg3, int jarg4);\n  public final static native long new_IndexIVFFlat__SWIG_1(long jarg1, Index jarg1_, long jarg2, long jarg3);\n  public final static native void IndexIVFFlat_add_core(long jarg1, IndexIVFFlat jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, LongVector jarg5_);\n  public final static native void IndexIVFFlat_encode_vectors__SWIG_0(long jarg1, IndexIVFFlat jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, boolean jarg6);\n  public final static native void IndexIVFFlat_encode_vectors__SWIG_1(long jarg1, IndexIVFFlat jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native long IndexIVFFlat_get_InvertedListScanner(long jarg1, IndexIVFFlat jarg1_, boolean jarg2);\n  public final static native void IndexIVFFlat_reconstruct_from_offset(long jarg1, IndexIVFFlat jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexIVFFlat_sa_decode(long jarg1, IndexIVFFlat jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native long new_IndexIVFFlat__SWIG_2();\n  public final static native void delete_IndexIVFFlat(long jarg1);\n  public final static native void IndexIVFFlatDedup_instances_set(long jarg1, IndexIVFFlatDedup jarg1_, long jarg2);\n  public final static native long IndexIVFFlatDedup_instances_get(long jarg1, IndexIVFFlatDedup jarg1_);\n  public final static native long new_IndexIVFFlatDedup__SWIG_0(long jarg1, Index jarg1_, long jarg2, long jarg3, int jarg4);\n  public final static native long new_IndexIVFFlatDedup__SWIG_1(long jarg1, Index jarg1_, long jarg2, long jarg3);\n  public final static native void IndexIVFFlatDedup_train(long jarg1, IndexIVFFlatDedup jarg1_, long jarg2, long jarg3);\n  public final static native void IndexIVFFlatDedup_add_with_ids(long jarg1, IndexIVFFlatDedup jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void IndexIVFFlatDedup_search_preassigned__SWIG_0(long jarg1, IndexIVFFlatDedup jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, long jarg8, LongVector jarg8_, boolean jarg9, long jarg10, IVFSearchParameters jarg10_, long jarg11, IndexIVFStats jarg11_);\n  public final static native void IndexIVFFlatDedup_search_preassigned__SWIG_1(long jarg1, IndexIVFFlatDedup jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, long jarg8, LongVector jarg8_, boolean jarg9, long jarg10, IVFSearchParameters jarg10_);\n  public final static native void IndexIVFFlatDedup_search_preassigned__SWIG_2(long jarg1, IndexIVFFlatDedup jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, long jarg8, LongVector jarg8_, boolean jarg9);\n  public final static native long IndexIVFFlatDedup_remove_ids(long jarg1, IndexIVFFlatDedup jarg1_, long jarg2, IDSelector jarg2_);\n  public final static native void IndexIVFFlatDedup_range_search(long jarg1, IndexIVFFlatDedup jarg1_, long jarg2, long jarg3, float jarg4, long jarg5, RangeSearchResult jarg5_);\n  public final static native void IndexIVFFlatDedup_update_vectors(long jarg1, IndexIVFFlatDedup jarg1_, int jarg2, long jarg3, LongVector jarg3_, long jarg4);\n  public final static native void IndexIVFFlatDedup_reconstruct_from_offset(long jarg1, IndexIVFFlatDedup jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native long new_IndexIVFFlatDedup__SWIG_2();\n  public final static native void delete_IndexIVFFlatDedup(long jarg1);\n  public final static native void OnDiskOneList_size_set(long jarg1, OnDiskOneList jarg1_, long jarg2);\n  public final static native long OnDiskOneList_size_get(long jarg1, OnDiskOneList jarg1_);\n  public final static native void OnDiskOneList_capacity_set(long jarg1, OnDiskOneList jarg1_, long jarg2);\n  public final static native long OnDiskOneList_capacity_get(long jarg1, OnDiskOneList jarg1_);\n  public final static native void OnDiskOneList_offset_set(long jarg1, OnDiskOneList jarg1_, long jarg2);\n  public final static native long OnDiskOneList_offset_get(long jarg1, OnDiskOneList jarg1_);\n  public final static native long new_OnDiskOneList();\n  public final static native void delete_OnDiskOneList(long jarg1);\n  public final static native void OnDiskInvertedLists_lists_set(long jarg1, OnDiskInvertedLists jarg1_, long jarg2);\n  public final static native long OnDiskInvertedLists_lists_get(long jarg1, OnDiskInvertedLists jarg1_);\n  public final static native void OnDiskInvertedLists_Slot_offset_set(long jarg1, OnDiskInvertedLists.Slot jarg1_, long jarg2);\n  public final static native long OnDiskInvertedLists_Slot_offset_get(long jarg1, OnDiskInvertedLists.Slot jarg1_);\n  public final static native void OnDiskInvertedLists_Slot_capacity_set(long jarg1, OnDiskInvertedLists.Slot jarg1_, long jarg2);\n  public final static native long OnDiskInvertedLists_Slot_capacity_get(long jarg1, OnDiskInvertedLists.Slot jarg1_);\n  public final static native long new_OnDiskInvertedLists_Slot__SWIG_0(long jarg1, long jarg2);\n  public final static native long new_OnDiskInvertedLists_Slot__SWIG_1();\n  public final static native void delete_OnDiskInvertedLists_Slot(long jarg1);\n  public final static native void OnDiskInvertedLists_slots_set(long jarg1, OnDiskInvertedLists jarg1_, long jarg2);\n  public final static native long OnDiskInvertedLists_slots_get(long jarg1, OnDiskInvertedLists jarg1_);\n  public final static native void OnDiskInvertedLists_filename_set(long jarg1, OnDiskInvertedLists jarg1_, String jarg2);\n  public final static native String OnDiskInvertedLists_filename_get(long jarg1, OnDiskInvertedLists jarg1_);\n  public final static native void OnDiskInvertedLists_totsize_set(long jarg1, OnDiskInvertedLists jarg1_, long jarg2);\n  public final static native long OnDiskInvertedLists_totsize_get(long jarg1, OnDiskInvertedLists jarg1_);\n  public final static native void OnDiskInvertedLists_ptr_set(long jarg1, OnDiskInvertedLists jarg1_, long jarg2);\n  public final static native long OnDiskInvertedLists_ptr_get(long jarg1, OnDiskInvertedLists jarg1_);\n  public final static native void OnDiskInvertedLists_read_only_set(long jarg1, OnDiskInvertedLists jarg1_, boolean jarg2);\n  public final static native boolean OnDiskInvertedLists_read_only_get(long jarg1, OnDiskInvertedLists jarg1_);\n  public final static native long new_OnDiskInvertedLists__SWIG_0(long jarg1, long jarg2, String jarg3);\n  public final static native long OnDiskInvertedLists_list_size(long jarg1, OnDiskInvertedLists jarg1_, long jarg2);\n  public final static native long OnDiskInvertedLists_get_codes(long jarg1, OnDiskInvertedLists jarg1_, long jarg2);\n  public final static native long OnDiskInvertedLists_get_ids(long jarg1, OnDiskInvertedLists jarg1_, long jarg2);\n  public final static native long OnDiskInvertedLists_add_entries(long jarg1, OnDiskInvertedLists jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void OnDiskInvertedLists_update_entries(long jarg1, OnDiskInvertedLists jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_, long jarg6);\n  public final static native void OnDiskInvertedLists_resize(long jarg1, OnDiskInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native long OnDiskInvertedLists_merge_from__SWIG_0(long jarg1, OnDiskInvertedLists jarg1_, long jarg2, int jarg3, boolean jarg4);\n  public final static native long OnDiskInvertedLists_merge_from__SWIG_1(long jarg1, OnDiskInvertedLists jarg1_, long jarg2, int jarg3);\n  public final static native long OnDiskInvertedLists_merge_from_1__SWIG_0(long jarg1, OnDiskInvertedLists jarg1_, long jarg2, InvertedLists jarg2_, boolean jarg3);\n  public final static native long OnDiskInvertedLists_merge_from_1__SWIG_1(long jarg1, OnDiskInvertedLists jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native void OnDiskInvertedLists_crop_invlists(long jarg1, OnDiskInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void OnDiskInvertedLists_prefetch_lists(long jarg1, OnDiskInvertedLists jarg1_, long jarg2, LongVector jarg2_, int jarg3);\n  public final static native void delete_OnDiskInvertedLists(long jarg1);\n  public final static native void OnDiskInvertedLists_locks_set(long jarg1, OnDiskInvertedLists jarg1_, long jarg2);\n  public final static native long OnDiskInvertedLists_locks_get(long jarg1, OnDiskInvertedLists jarg1_);\n  public final static native void OnDiskInvertedLists_pf_set(long jarg1, OnDiskInvertedLists jarg1_, long jarg2);\n  public final static native long OnDiskInvertedLists_pf_get(long jarg1, OnDiskInvertedLists jarg1_);\n  public final static native void OnDiskInvertedLists_prefetch_nthread_set(long jarg1, OnDiskInvertedLists jarg1_, int jarg2);\n  public final static native int OnDiskInvertedLists_prefetch_nthread_get(long jarg1, OnDiskInvertedLists jarg1_);\n  public final static native void OnDiskInvertedLists_do_mmap(long jarg1, OnDiskInvertedLists jarg1_);\n  public final static native void OnDiskInvertedLists_update_totsize(long jarg1, OnDiskInvertedLists jarg1_, long jarg2);\n  public final static native void OnDiskInvertedLists_resize_locked(long jarg1, OnDiskInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native long OnDiskInvertedLists_allocate_slot(long jarg1, OnDiskInvertedLists jarg1_, long jarg2);\n  public final static native void OnDiskInvertedLists_free_slot(long jarg1, OnDiskInvertedLists jarg1_, long jarg2, long jarg3);\n  public final static native void OnDiskInvertedLists_set_all_lists_sizes(long jarg1, OnDiskInvertedLists jarg1_, long jarg2);\n  public final static native long new_OnDiskInvertedLists__SWIG_1();\n  public final static native long new_OnDiskInvertedListsIOHook();\n  public final static native void OnDiskInvertedListsIOHook_write(long jarg1, OnDiskInvertedListsIOHook jarg1_, long jarg2, InvertedLists jarg2_, long jarg3);\n  public final static native long OnDiskInvertedListsIOHook_read(long jarg1, OnDiskInvertedListsIOHook jarg1_, long jarg2, int jarg3);\n  public final static native long OnDiskInvertedListsIOHook_read_ArrayInvertedLists(long jarg1, OnDiskInvertedListsIOHook jarg1_, long jarg2, int jarg3, long jarg4, long jarg5, long jarg6, Uint64Vector jarg6_);\n  public final static native void delete_OnDiskInvertedListsIOHook(long jarg1);\n  public final static native void IVFPQSearchParameters_scan_table_threshold_set(long jarg1, IVFPQSearchParameters jarg1_, long jarg2);\n  public final static native long IVFPQSearchParameters_scan_table_threshold_get(long jarg1, IVFPQSearchParameters jarg1_);\n  public final static native void IVFPQSearchParameters_polysemous_ht_set(long jarg1, IVFPQSearchParameters jarg1_, int jarg2);\n  public final static native int IVFPQSearchParameters_polysemous_ht_get(long jarg1, IVFPQSearchParameters jarg1_);\n  public final static native long new_IVFPQSearchParameters();\n  public final static native void delete_IVFPQSearchParameters(long jarg1);\n  public final static native void precomputed_table_max_bytes_set(long jarg1);\n  public final static native long precomputed_table_max_bytes_get();\n  public final static native void IndexIVFPQ_by_residual_set(long jarg1, IndexIVFPQ jarg1_, boolean jarg2);\n  public final static native boolean IndexIVFPQ_by_residual_get(long jarg1, IndexIVFPQ jarg1_);\n  public final static native void IndexIVFPQ_pq_set(long jarg1, IndexIVFPQ jarg1_, long jarg2, ProductQuantizer jarg2_);\n  public final static native long IndexIVFPQ_pq_get(long jarg1, IndexIVFPQ jarg1_);\n  public final static native void IndexIVFPQ_do_polysemous_training_set(long jarg1, IndexIVFPQ jarg1_, boolean jarg2);\n  public final static native boolean IndexIVFPQ_do_polysemous_training_get(long jarg1, IndexIVFPQ jarg1_);\n  public final static native void IndexIVFPQ_polysemous_training_set(long jarg1, IndexIVFPQ jarg1_, long jarg2, PolysemousTraining jarg2_);\n  public final static native long IndexIVFPQ_polysemous_training_get(long jarg1, IndexIVFPQ jarg1_);\n  public final static native void IndexIVFPQ_scan_table_threshold_set(long jarg1, IndexIVFPQ jarg1_, long jarg2);\n  public final static native long IndexIVFPQ_scan_table_threshold_get(long jarg1, IndexIVFPQ jarg1_);\n  public final static native void IndexIVFPQ_polysemous_ht_set(long jarg1, IndexIVFPQ jarg1_, int jarg2);\n  public final static native int IndexIVFPQ_polysemous_ht_get(long jarg1, IndexIVFPQ jarg1_);\n  public final static native void IndexIVFPQ_use_precomputed_table_set(long jarg1, IndexIVFPQ jarg1_, int jarg2);\n  public final static native int IndexIVFPQ_use_precomputed_table_get(long jarg1, IndexIVFPQ jarg1_);\n  public final static native void IndexIVFPQ_precomputed_table_set(long jarg1, IndexIVFPQ jarg1_, long jarg2);\n  public final static native long IndexIVFPQ_precomputed_table_get(long jarg1, IndexIVFPQ jarg1_);\n  public final static native long new_IndexIVFPQ__SWIG_0(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, int jarg6);\n  public final static native long new_IndexIVFPQ__SWIG_1(long jarg1, Index jarg1_, long jarg2, long jarg3, long jarg4, long jarg5);\n  public final static native void IndexIVFPQ_encode_vectors__SWIG_0(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, boolean jarg6);\n  public final static native void IndexIVFPQ_encode_vectors__SWIG_1(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void IndexIVFPQ_sa_decode(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexIVFPQ_add_core(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, LongVector jarg5_);\n  public final static native void IndexIVFPQ_add_core_o__SWIG_0(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexIVFPQ_add_core_o__SWIG_1(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void IndexIVFPQ_train_residual(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3);\n  public final static native void IndexIVFPQ_train_residual_o(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexIVFPQ_reconstruct_from_offset(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native long IndexIVFPQ_find_duplicates(long jarg1, IndexIVFPQ jarg1_, long jarg2, LongVector jarg2_, long jarg3);\n  public final static native void IndexIVFPQ_encode(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexIVFPQ_encode_multiple__SWIG_0(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3, LongVector jarg3_, long jarg4, long jarg5, boolean jarg6);\n  public final static native void IndexIVFPQ_encode_multiple__SWIG_1(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3, LongVector jarg3_, long jarg4, long jarg5);\n  public final static native void IndexIVFPQ_decode_multiple(long jarg1, IndexIVFPQ jarg1_, long jarg2, long jarg3, LongVector jarg3_, long jarg4, long jarg5);\n  public final static native long IndexIVFPQ_get_InvertedListScanner(long jarg1, IndexIVFPQ jarg1_, boolean jarg2);\n  public final static native void IndexIVFPQ_precompute_table(long jarg1, IndexIVFPQ jarg1_);\n  public final static native long new_IndexIVFPQ__SWIG_2();\n  public final static native void delete_IndexIVFPQ(long jarg1);\n  public final static native void initialize_IVFPQ_precomputed_table(long jarg1, long jarg2, Index jarg2_, long jarg3, ProductQuantizer jarg3_, long jarg4, boolean jarg5);\n  public final static native void IndexIVFPQStats_nrefine_set(long jarg1, IndexIVFPQStats jarg1_, long jarg2);\n  public final static native long IndexIVFPQStats_nrefine_get(long jarg1, IndexIVFPQStats jarg1_);\n  public final static native void IndexIVFPQStats_n_hamming_pass_set(long jarg1, IndexIVFPQStats jarg1_, long jarg2);\n  public final static native long IndexIVFPQStats_n_hamming_pass_get(long jarg1, IndexIVFPQStats jarg1_);\n  public final static native void IndexIVFPQStats_search_cycles_set(long jarg1, IndexIVFPQStats jarg1_, long jarg2);\n  public final static native long IndexIVFPQStats_search_cycles_get(long jarg1, IndexIVFPQStats jarg1_);\n  public final static native void IndexIVFPQStats_refine_cycles_set(long jarg1, IndexIVFPQStats jarg1_, long jarg2);\n  public final static native long IndexIVFPQStats_refine_cycles_get(long jarg1, IndexIVFPQStats jarg1_);\n  public final static native long new_IndexIVFPQStats();\n  public final static native void IndexIVFPQStats_reset(long jarg1, IndexIVFPQStats jarg1_);\n  public final static native void delete_IndexIVFPQStats(long jarg1);\n  public final static native void indexIVFPQ_stats_set(long jarg1, IndexIVFPQStats jarg1_);\n  public final static native long indexIVFPQ_stats_get();\n  public final static native void IndexBinary_d_set(long jarg1, IndexBinary jarg1_, int jarg2);\n  public final static native int IndexBinary_d_get(long jarg1, IndexBinary jarg1_);\n  public final static native void IndexBinary_code_size_set(long jarg1, IndexBinary jarg1_, int jarg2);\n  public final static native int IndexBinary_code_size_get(long jarg1, IndexBinary jarg1_);\n  public final static native void IndexBinary_ntotal_set(long jarg1, IndexBinary jarg1_, long jarg2);\n  public final static native long IndexBinary_ntotal_get(long jarg1, IndexBinary jarg1_);\n  public final static native void IndexBinary_verbose_set(long jarg1, IndexBinary jarg1_, boolean jarg2);\n  public final static native boolean IndexBinary_verbose_get(long jarg1, IndexBinary jarg1_);\n  public final static native void IndexBinary_is_trained_set(long jarg1, IndexBinary jarg1_, boolean jarg2);\n  public final static native boolean IndexBinary_is_trained_get(long jarg1, IndexBinary jarg1_);\n  public final static native void IndexBinary_metric_type_set(long jarg1, IndexBinary jarg1_, int jarg2);\n  public final static native int IndexBinary_metric_type_get(long jarg1, IndexBinary jarg1_);\n  public final static native void delete_IndexBinary(long jarg1);\n  public final static native void IndexBinary_train(long jarg1, IndexBinary jarg1_, long jarg2, long jarg3);\n  public final static native void IndexBinary_add(long jarg1, IndexBinary jarg1_, long jarg2, long jarg3);\n  public final static native void IndexBinary_add_with_ids(long jarg1, IndexBinary jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void IndexBinary_search(long jarg1, IndexBinary jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexBinary_range_search(long jarg1, IndexBinary jarg1_, long jarg2, long jarg3, int jarg4, long jarg5, RangeSearchResult jarg5_);\n  public final static native void IndexBinary_assign__SWIG_0(long jarg1, IndexBinary jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void IndexBinary_assign__SWIG_1(long jarg1, IndexBinary jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void IndexBinary_reset(long jarg1, IndexBinary jarg1_);\n  public final static native long IndexBinary_remove_ids(long jarg1, IndexBinary jarg1_, long jarg2, IDSelector jarg2_);\n  public final static native void IndexBinary_reconstruct(long jarg1, IndexBinary jarg1_, long jarg2, long jarg3);\n  public final static native void IndexBinary_reconstruct_n(long jarg1, IndexBinary jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexBinary_search_and_reconstruct(long jarg1, IndexBinary jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_, long jarg7);\n  public final static native void IndexBinary_display(long jarg1, IndexBinary jarg1_);\n  public final static native void Index2Layer_q1_set(long jarg1, Index2Layer jarg1_, long jarg2, Level1Quantizer jarg2_);\n  public final static native long Index2Layer_q1_get(long jarg1, Index2Layer jarg1_);\n  public final static native void Index2Layer_pq_set(long jarg1, Index2Layer jarg1_, long jarg2, ProductQuantizer jarg2_);\n  public final static native long Index2Layer_pq_get(long jarg1, Index2Layer jarg1_);\n  public final static native void Index2Layer_code_size_1_set(long jarg1, Index2Layer jarg1_, long jarg2);\n  public final static native long Index2Layer_code_size_1_get(long jarg1, Index2Layer jarg1_);\n  public final static native void Index2Layer_code_size_2_set(long jarg1, Index2Layer jarg1_, long jarg2);\n  public final static native long Index2Layer_code_size_2_get(long jarg1, Index2Layer jarg1_);\n  public final static native long new_Index2Layer__SWIG_0(long jarg1, Index jarg1_, long jarg2, int jarg3, int jarg4, int jarg5);\n  public final static native long new_Index2Layer__SWIG_1(long jarg1, Index jarg1_, long jarg2, int jarg3, int jarg4);\n  public final static native long new_Index2Layer__SWIG_2(long jarg1, Index jarg1_, long jarg2, int jarg3);\n  public final static native long new_Index2Layer__SWIG_3();\n  public final static native void delete_Index2Layer(long jarg1);\n  public final static native void Index2Layer_train(long jarg1, Index2Layer jarg1_, long jarg2, long jarg3);\n  public final static native void Index2Layer_search(long jarg1, Index2Layer jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native long Index2Layer_get_distance_computer(long jarg1, Index2Layer jarg1_);\n  public final static native void Index2Layer_transfer_to_IVFPQ(long jarg1, Index2Layer jarg1_, long jarg2, IndexIVFPQ jarg2_);\n  public final static native void Index2Layer_sa_encode(long jarg1, Index2Layer jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void Index2Layer_sa_decode(long jarg1, Index2Layer jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexBinaryFlat_xb_set(long jarg1, IndexBinaryFlat jarg1_, long jarg2, ByteVector jarg2_);\n  public final static native long IndexBinaryFlat_xb_get(long jarg1, IndexBinaryFlat jarg1_);\n  public final static native void IndexBinaryFlat_use_heap_set(long jarg1, IndexBinaryFlat jarg1_, boolean jarg2);\n  public final static native boolean IndexBinaryFlat_use_heap_get(long jarg1, IndexBinaryFlat jarg1_);\n  public final static native void IndexBinaryFlat_query_batch_size_set(long jarg1, IndexBinaryFlat jarg1_, long jarg2);\n  public final static native long IndexBinaryFlat_query_batch_size_get(long jarg1, IndexBinaryFlat jarg1_);\n  public final static native long new_IndexBinaryFlat__SWIG_0(long jarg1);\n  public final static native void IndexBinaryFlat_add(long jarg1, IndexBinaryFlat jarg1_, long jarg2, long jarg3);\n  public final static native void IndexBinaryFlat_reset(long jarg1, IndexBinaryFlat jarg1_);\n  public final static native void IndexBinaryFlat_search(long jarg1, IndexBinaryFlat jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexBinaryFlat_range_search(long jarg1, IndexBinaryFlat jarg1_, long jarg2, long jarg3, int jarg4, long jarg5, RangeSearchResult jarg5_);\n  public final static native void IndexBinaryFlat_reconstruct(long jarg1, IndexBinaryFlat jarg1_, long jarg2, long jarg3);\n  public final static native long IndexBinaryFlat_remove_ids(long jarg1, IndexBinaryFlat jarg1_, long jarg2, IDSelector jarg2_);\n  public final static native long new_IndexBinaryFlat__SWIG_1();\n  public final static native void delete_IndexBinaryFlat(long jarg1);\n  public final static native void IndexBinaryIVF_invlists_set(long jarg1, IndexBinaryIVF jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native long IndexBinaryIVF_invlists_get(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_own_invlists_set(long jarg1, IndexBinaryIVF jarg1_, boolean jarg2);\n  public final static native boolean IndexBinaryIVF_own_invlists_get(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_nprobe_set(long jarg1, IndexBinaryIVF jarg1_, long jarg2);\n  public final static native long IndexBinaryIVF_nprobe_get(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_max_codes_set(long jarg1, IndexBinaryIVF jarg1_, long jarg2);\n  public final static native long IndexBinaryIVF_max_codes_get(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_use_heap_set(long jarg1, IndexBinaryIVF jarg1_, boolean jarg2);\n  public final static native boolean IndexBinaryIVF_use_heap_get(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_direct_map_set(long jarg1, IndexBinaryIVF jarg1_, long jarg2);\n  public final static native long IndexBinaryIVF_direct_map_get(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_quantizer_set(long jarg1, IndexBinaryIVF jarg1_, long jarg2, IndexBinary jarg2_);\n  public final static native long IndexBinaryIVF_quantizer_get(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_nlist_set(long jarg1, IndexBinaryIVF jarg1_, long jarg2);\n  public final static native long IndexBinaryIVF_nlist_get(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_own_fields_set(long jarg1, IndexBinaryIVF jarg1_, boolean jarg2);\n  public final static native boolean IndexBinaryIVF_own_fields_get(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_cp_set(long jarg1, IndexBinaryIVF jarg1_, long jarg2, ClusteringParameters jarg2_);\n  public final static native long IndexBinaryIVF_cp_get(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_clustering_index_set(long jarg1, IndexBinaryIVF jarg1_, long jarg2, Index jarg2_);\n  public final static native long IndexBinaryIVF_clustering_index_get(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native long new_IndexBinaryIVF__SWIG_0(long jarg1, IndexBinary jarg1_, long jarg2, long jarg3);\n  public final static native long new_IndexBinaryIVF__SWIG_1();\n  public final static native void delete_IndexBinaryIVF(long jarg1);\n  public final static native void IndexBinaryIVF_reset(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_train(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3);\n  public final static native void IndexBinaryIVF_add(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3);\n  public final static native void IndexBinaryIVF_add_with_ids(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void IndexBinaryIVF_add_core(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, LongVector jarg5_);\n  public final static native void IndexBinaryIVF_search_preassigned__SWIG_0(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, long jarg8, LongVector jarg8_, boolean jarg9, long jarg10, IVFSearchParameters jarg10_);\n  public final static native void IndexBinaryIVF_search_preassigned__SWIG_1(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, long jarg8, LongVector jarg8_, boolean jarg9);\n  public final static native long IndexBinaryIVF_get_InvertedListScanner__SWIG_0(long jarg1, IndexBinaryIVF jarg1_, boolean jarg2);\n  public final static native long IndexBinaryIVF_get_InvertedListScanner__SWIG_1(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_search(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexBinaryIVF_range_search(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3, int jarg4, long jarg5, RangeSearchResult jarg5_);\n  public final static native void IndexBinaryIVF_range_search_preassigned(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3, int jarg4, long jarg5, LongVector jarg5_, long jarg6, long jarg7, RangeSearchResult jarg7_);\n  public final static native void IndexBinaryIVF_reconstruct(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3);\n  public final static native void IndexBinaryIVF_reconstruct_n(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexBinaryIVF_search_and_reconstruct(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_, long jarg7);\n  public final static native void IndexBinaryIVF_reconstruct_from_offset(long jarg1, IndexBinaryIVF jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native long IndexBinaryIVF_remove_ids(long jarg1, IndexBinaryIVF jarg1_, long jarg2, IDSelector jarg2_);\n  public final static native void IndexBinaryIVF_merge_from(long jarg1, IndexBinaryIVF jarg1_, long jarg2, IndexBinaryIVF jarg2_, long jarg3);\n  public final static native long IndexBinaryIVF_get_list_size(long jarg1, IndexBinaryIVF jarg1_, long jarg2);\n  public final static native void IndexBinaryIVF_make_direct_map__SWIG_0(long jarg1, IndexBinaryIVF jarg1_, boolean jarg2);\n  public final static native void IndexBinaryIVF_make_direct_map__SWIG_1(long jarg1, IndexBinaryIVF jarg1_);\n  public final static native void IndexBinaryIVF_set_direct_map_type(long jarg1, IndexBinaryIVF jarg1_, long jarg2);\n  public final static native void IndexBinaryIVF_replace_invlists__SWIG_0(long jarg1, IndexBinaryIVF jarg1_, long jarg2, InvertedLists jarg2_, boolean jarg3);\n  public final static native void IndexBinaryIVF_replace_invlists__SWIG_1(long jarg1, IndexBinaryIVF jarg1_, long jarg2, InvertedLists jarg2_);\n  public final static native void IndexBinaryFromFloat_index_set(long jarg1, IndexBinaryFromFloat jarg1_, long jarg2, Index jarg2_);\n  public final static native long IndexBinaryFromFloat_index_get(long jarg1, IndexBinaryFromFloat jarg1_);\n  public final static native void IndexBinaryFromFloat_own_fields_set(long jarg1, IndexBinaryFromFloat jarg1_, boolean jarg2);\n  public final static native boolean IndexBinaryFromFloat_own_fields_get(long jarg1, IndexBinaryFromFloat jarg1_);\n  public final static native long new_IndexBinaryFromFloat__SWIG_0();\n  public final static native long new_IndexBinaryFromFloat__SWIG_1(long jarg1, Index jarg1_);\n  public final static native void delete_IndexBinaryFromFloat(long jarg1);\n  public final static native void IndexBinaryFromFloat_add(long jarg1, IndexBinaryFromFloat jarg1_, long jarg2, long jarg3);\n  public final static native void IndexBinaryFromFloat_reset(long jarg1, IndexBinaryFromFloat jarg1_);\n  public final static native void IndexBinaryFromFloat_search(long jarg1, IndexBinaryFromFloat jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexBinaryFromFloat_train(long jarg1, IndexBinaryFromFloat jarg1_, long jarg2, long jarg3);\n  public final static native void IndexBinaryHNSW_hnsw_set(long jarg1, IndexBinaryHNSW jarg1_, long jarg2, HNSW jarg2_);\n  public final static native long IndexBinaryHNSW_hnsw_get(long jarg1, IndexBinaryHNSW jarg1_);\n  public final static native void IndexBinaryHNSW_own_fields_set(long jarg1, IndexBinaryHNSW jarg1_, boolean jarg2);\n  public final static native boolean IndexBinaryHNSW_own_fields_get(long jarg1, IndexBinaryHNSW jarg1_);\n  public final static native void IndexBinaryHNSW_storage_set(long jarg1, IndexBinaryHNSW jarg1_, long jarg2, IndexBinary jarg2_);\n  public final static native long IndexBinaryHNSW_storage_get(long jarg1, IndexBinaryHNSW jarg1_);\n  public final static native long new_IndexBinaryHNSW__SWIG_0();\n  public final static native long new_IndexBinaryHNSW__SWIG_1(int jarg1, int jarg2);\n  public final static native long new_IndexBinaryHNSW__SWIG_2(int jarg1);\n  public final static native long new_IndexBinaryHNSW__SWIG_3(long jarg1, IndexBinary jarg1_, int jarg2);\n  public final static native long new_IndexBinaryHNSW__SWIG_4(long jarg1, IndexBinary jarg1_);\n  public final static native void delete_IndexBinaryHNSW(long jarg1);\n  public final static native long IndexBinaryHNSW_get_distance_computer(long jarg1, IndexBinaryHNSW jarg1_);\n  public final static native void IndexBinaryHNSW_add(long jarg1, IndexBinaryHNSW jarg1_, long jarg2, long jarg3);\n  public final static native void IndexBinaryHNSW_train(long jarg1, IndexBinaryHNSW jarg1_, long jarg2, long jarg3);\n  public final static native void IndexBinaryHNSW_search(long jarg1, IndexBinaryHNSW jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexBinaryHNSW_reconstruct(long jarg1, IndexBinaryHNSW jarg1_, long jarg2, long jarg3);\n  public final static native void IndexBinaryHNSW_reset(long jarg1, IndexBinaryHNSW jarg1_);\n  public final static native void IndexRefine_base_index_set(long jarg1, IndexRefine jarg1_, long jarg2, Index jarg2_);\n  public final static native long IndexRefine_base_index_get(long jarg1, IndexRefine jarg1_);\n  public final static native void IndexRefine_refine_index_set(long jarg1, IndexRefine jarg1_, long jarg2, Index jarg2_);\n  public final static native long IndexRefine_refine_index_get(long jarg1, IndexRefine jarg1_);\n  public final static native void IndexRefine_own_fields_set(long jarg1, IndexRefine jarg1_, boolean jarg2);\n  public final static native boolean IndexRefine_own_fields_get(long jarg1, IndexRefine jarg1_);\n  public final static native void IndexRefine_own_refine_index_set(long jarg1, IndexRefine jarg1_, boolean jarg2);\n  public final static native boolean IndexRefine_own_refine_index_get(long jarg1, IndexRefine jarg1_);\n  public final static native void IndexRefine_k_factor_set(long jarg1, IndexRefine jarg1_, float jarg2);\n  public final static native float IndexRefine_k_factor_get(long jarg1, IndexRefine jarg1_);\n  public final static native long new_IndexRefine__SWIG_0(long jarg1, Index jarg1_, long jarg2, Index jarg2_);\n  public final static native long new_IndexRefine__SWIG_1();\n  public final static native void IndexRefine_train(long jarg1, IndexRefine jarg1_, long jarg2, long jarg3);\n  public final static native void IndexRefine_add(long jarg1, IndexRefine jarg1_, long jarg2, long jarg3);\n  public final static native void IndexRefine_reset(long jarg1, IndexRefine jarg1_);\n  public final static native void IndexRefine_search(long jarg1, IndexRefine jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexRefine_reconstruct(long jarg1, IndexRefine jarg1_, long jarg2, long jarg3);\n  public final static native long IndexRefine_sa_code_size(long jarg1, IndexRefine jarg1_);\n  public final static native void IndexRefine_sa_encode(long jarg1, IndexRefine jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void IndexRefine_sa_decode(long jarg1, IndexRefine jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void delete_IndexRefine(long jarg1);\n  public final static native long new_IndexRefineFlat__SWIG_0(long jarg1, Index jarg1_);\n  public final static native long new_IndexRefineFlat__SWIG_1(long jarg1, Index jarg1_, long jarg2);\n  public final static native long new_IndexRefineFlat__SWIG_2();\n  public final static native void IndexRefineFlat_search(long jarg1, IndexRefineFlat jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void delete_IndexRefineFlat(long jarg1);\n  public final static native void IndexSplitVectors_own_fields_set(long jarg1, IndexSplitVectors jarg1_, boolean jarg2);\n  public final static native boolean IndexSplitVectors_own_fields_get(long jarg1, IndexSplitVectors jarg1_);\n  public final static native void IndexSplitVectors_threaded_set(long jarg1, IndexSplitVectors jarg1_, boolean jarg2);\n  public final static native boolean IndexSplitVectors_threaded_get(long jarg1, IndexSplitVectors jarg1_);\n  public final static native void IndexSplitVectors_sub_indexes_set(long jarg1, IndexSplitVectors jarg1_, long jarg2);\n  public final static native long IndexSplitVectors_sub_indexes_get(long jarg1, IndexSplitVectors jarg1_);\n  public final static native void IndexSplitVectors_sum_d_set(long jarg1, IndexSplitVectors jarg1_, long jarg2);\n  public final static native long IndexSplitVectors_sum_d_get(long jarg1, IndexSplitVectors jarg1_);\n  public final static native void IndexSplitVectors_add_sub_index(long jarg1, IndexSplitVectors jarg1_, long jarg2, Index jarg2_);\n  public final static native void IndexSplitVectors_sync_with_sub_indexes(long jarg1, IndexSplitVectors jarg1_);\n  public final static native void IndexSplitVectors_add(long jarg1, IndexSplitVectors jarg1_, long jarg2, long jarg3);\n  public final static native void IndexSplitVectors_search(long jarg1, IndexSplitVectors jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexSplitVectors_train(long jarg1, IndexSplitVectors jarg1_, long jarg2, long jarg3);\n  public final static native void IndexSplitVectors_reset(long jarg1, IndexSplitVectors jarg1_);\n  public final static native void delete_IndexSplitVectors(long jarg1);\n  public final static native void IndexIDMap_index_set(long jarg1, IndexIDMap jarg1_, long jarg2, Index jarg2_);\n  public final static native long IndexIDMap_index_get(long jarg1, IndexIDMap jarg1_);\n  public final static native void IndexIDMap_own_fields_set(long jarg1, IndexIDMap jarg1_, boolean jarg2);\n  public final static native boolean IndexIDMap_own_fields_get(long jarg1, IndexIDMap jarg1_);\n  public final static native void IndexIDMap_id_map_set(long jarg1, IndexIDMap jarg1_, long jarg2);\n  public final static native long IndexIDMap_id_map_get(long jarg1, IndexIDMap jarg1_);\n  public final static native long new_IndexIDMap__SWIG_0(long jarg1, Index jarg1_);\n  public final static native void IndexIDMap_add_with_ids(long jarg1, IndexIDMap jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void IndexIDMap_add(long jarg1, IndexIDMap jarg1_, long jarg2, long jarg3);\n  public final static native void IndexIDMap_search(long jarg1, IndexIDMap jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexIDMap_train(long jarg1, IndexIDMap jarg1_, long jarg2, long jarg3);\n  public final static native void IndexIDMap_reset(long jarg1, IndexIDMap jarg1_);\n  public final static native long IndexIDMap_remove_ids(long jarg1, IndexIDMap jarg1_, long jarg2, IDSelector jarg2_);\n  public final static native void IndexIDMap_range_search(long jarg1, IndexIDMap jarg1_, long jarg2, long jarg3, float jarg4, long jarg5, RangeSearchResult jarg5_);\n  public final static native void delete_IndexIDMap(long jarg1);\n  public final static native long new_IndexIDMap__SWIG_1();\n  public final static native long new_IndexShards__SWIG_0(boolean jarg1, boolean jarg2);\n  public final static native long new_IndexShards__SWIG_1(boolean jarg1);\n  public final static native long new_IndexShards__SWIG_2();\n  public final static native long new_IndexShards__SWIG_3(int jarg1, boolean jarg2, boolean jarg3);\n  public final static native long new_IndexShards__SWIG_4(int jarg1, boolean jarg2);\n  public final static native long new_IndexShards__SWIG_5(int jarg1);\n  public final static native void IndexShards_add_shard(long jarg1, IndexShards jarg1_, long jarg2, Index jarg2_);\n  public final static native void IndexShards_remove_shard(long jarg1, IndexShards jarg1_, long jarg2, Index jarg2_);\n  public final static native void IndexShards_add(long jarg1, IndexShards jarg1_, long jarg2, long jarg3);\n  public final static native void IndexShards_add_with_ids(long jarg1, IndexShards jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void IndexShards_search(long jarg1, IndexShards jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6, LongVector jarg6_);\n  public final static native void IndexShards_train(long jarg1, IndexShards jarg1_, long jarg2, long jarg3);\n  public final static native void IndexShards_successive_ids_set(long jarg1, IndexShards jarg1_, boolean jarg2);\n  public final static native boolean IndexShards_successive_ids_get(long jarg1, IndexShards jarg1_);\n  public final static native void IndexShards_syncWithSubIndexes(long jarg1, IndexShards jarg1_);\n  public final static native void delete_IndexShards(long jarg1);\n  public final static native long downcast_index(long jarg1, Index jarg1_);\n  public final static native long downcast_VectorTransform(long jarg1, VectorTransform jarg1_);\n  public final static native long downcast_IndexBinary(long jarg1, IndexBinary jarg1_);\n  public final static native long upcast_IndexShards(long jarg1, IndexShards jarg1_);\n  public final static native void write_index__SWIG_0(long jarg1, Index jarg1_, String jarg2);\n  public final static native void write_index__SWIG_1(long jarg1, Index jarg1_, long jarg2);\n  public final static native void write_index__SWIG_2(long jarg1, Index jarg1_, long jarg2);\n  public final static native void write_index_binary__SWIG_0(long jarg1, IndexBinary jarg1_, String jarg2);\n  public final static native void write_index_binary__SWIG_1(long jarg1, IndexBinary jarg1_, long jarg2);\n  public final static native void write_index_binary__SWIG_2(long jarg1, IndexBinary jarg1_, long jarg2);\n  public final static native int IO_FLAG_READ_ONLY_get();\n  public final static native int IO_FLAG_ONDISK_SAME_DIR_get();\n  public final static native int IO_FLAG_SKIP_IVF_DATA_get();\n  public final static native int IO_FLAG_MMAP_get();\n  public final static native long read_index__SWIG_0(String jarg1, int jarg2);\n  public final static native long read_index__SWIG_1(String jarg1);\n  public final static native long read_index__SWIG_2(long jarg1, int jarg2);\n  public final static native long read_index__SWIG_3(long jarg1);\n  public final static native long read_index__SWIG_4(long jarg1, int jarg2);\n  public final static native long read_index__SWIG_5(long jarg1);\n  public final static native long read_index_binary__SWIG_0(String jarg1, int jarg2);\n  public final static native long read_index_binary__SWIG_1(String jarg1);\n  public final static native long read_index_binary__SWIG_2(long jarg1, int jarg2);\n  public final static native long read_index_binary__SWIG_3(long jarg1);\n  public final static native long read_index_binary__SWIG_4(long jarg1, int jarg2);\n  public final static native long read_index_binary__SWIG_5(long jarg1);\n  public final static native void write_VectorTransform(long jarg1, VectorTransform jarg1_, String jarg2);\n  public final static native long read_VectorTransform(String jarg1);\n  public final static native long read_ProductQuantizer__SWIG_0(String jarg1);\n  public final static native long read_ProductQuantizer__SWIG_1(long jarg1);\n  public final static native void write_ProductQuantizer__SWIG_0(long jarg1, ProductQuantizer jarg1_, String jarg2);\n  public final static native void write_ProductQuantizer__SWIG_1(long jarg1, ProductQuantizer jarg1_, long jarg2);\n  public final static native void write_InvertedLists(long jarg1, InvertedLists jarg1_, long jarg2);\n  public final static native long read_InvertedLists__SWIG_0(long jarg1, int jarg2);\n  public final static native long read_InvertedLists__SWIG_1(long jarg1);\n  public final static native void AutoTuneCriterion_nq_set(long jarg1, AutoTuneCriterion jarg1_, long jarg2);\n  public final static native long AutoTuneCriterion_nq_get(long jarg1, AutoTuneCriterion jarg1_);\n  public final static native void AutoTuneCriterion_nnn_set(long jarg1, AutoTuneCriterion jarg1_, long jarg2);\n  public final static native long AutoTuneCriterion_nnn_get(long jarg1, AutoTuneCriterion jarg1_);\n  public final static native void AutoTuneCriterion_gt_nnn_set(long jarg1, AutoTuneCriterion jarg1_, long jarg2);\n  public final static native long AutoTuneCriterion_gt_nnn_get(long jarg1, AutoTuneCriterion jarg1_);\n  public final static native void AutoTuneCriterion_gt_D_set(long jarg1, AutoTuneCriterion jarg1_, long jarg2, FloatVector jarg2_);\n  public final static native long AutoTuneCriterion_gt_D_get(long jarg1, AutoTuneCriterion jarg1_);\n  public final static native void AutoTuneCriterion_gt_I_set(long jarg1, AutoTuneCriterion jarg1_, long jarg2);\n  public final static native long AutoTuneCriterion_gt_I_get(long jarg1, AutoTuneCriterion jarg1_);\n  public final static native void AutoTuneCriterion_set_groundtruth(long jarg1, AutoTuneCriterion jarg1_, int jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native double AutoTuneCriterion_evaluate(long jarg1, AutoTuneCriterion jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native void delete_AutoTuneCriterion(long jarg1);\n  public final static native void OneRecallAtRCriterion_R_set(long jarg1, OneRecallAtRCriterion jarg1_, long jarg2);\n  public final static native long OneRecallAtRCriterion_R_get(long jarg1, OneRecallAtRCriterion jarg1_);\n  public final static native long new_OneRecallAtRCriterion(long jarg1, long jarg2);\n  public final static native double OneRecallAtRCriterion_evaluate(long jarg1, OneRecallAtRCriterion jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native void delete_OneRecallAtRCriterion(long jarg1);\n  public final static native void IntersectionCriterion_R_set(long jarg1, IntersectionCriterion jarg1_, long jarg2);\n  public final static native long IntersectionCriterion_R_get(long jarg1, IntersectionCriterion jarg1_);\n  public final static native long new_IntersectionCriterion(long jarg1, long jarg2);\n  public final static native double IntersectionCriterion_evaluate(long jarg1, IntersectionCriterion jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native void delete_IntersectionCriterion(long jarg1);\n  public final static native void OperatingPoint_perf_set(long jarg1, OperatingPoint jarg1_, double jarg2);\n  public final static native double OperatingPoint_perf_get(long jarg1, OperatingPoint jarg1_);\n  public final static native void OperatingPoint_t_set(long jarg1, OperatingPoint jarg1_, double jarg2);\n  public final static native double OperatingPoint_t_get(long jarg1, OperatingPoint jarg1_);\n  public final static native void OperatingPoint_key_set(long jarg1, OperatingPoint jarg1_, String jarg2);\n  public final static native String OperatingPoint_key_get(long jarg1, OperatingPoint jarg1_);\n  public final static native void OperatingPoint_cno_set(long jarg1, OperatingPoint jarg1_, long jarg2);\n  public final static native long OperatingPoint_cno_get(long jarg1, OperatingPoint jarg1_);\n  public final static native long new_OperatingPoint();\n  public final static native void delete_OperatingPoint(long jarg1);\n  public final static native void OperatingPoints_all_pts_set(long jarg1, OperatingPoints jarg1_, long jarg2, OperatingPointVector jarg2_);\n  public final static native long OperatingPoints_all_pts_get(long jarg1, OperatingPoints jarg1_);\n  public final static native void OperatingPoints_optimal_pts_set(long jarg1, OperatingPoints jarg1_, long jarg2, OperatingPointVector jarg2_);\n  public final static native long OperatingPoints_optimal_pts_get(long jarg1, OperatingPoints jarg1_);\n  public final static native long new_OperatingPoints();\n  public final static native int OperatingPoints_merge_with__SWIG_0(long jarg1, OperatingPoints jarg1_, long jarg2, OperatingPoints jarg2_, String jarg3);\n  public final static native int OperatingPoints_merge_with__SWIG_1(long jarg1, OperatingPoints jarg1_, long jarg2, OperatingPoints jarg2_);\n  public final static native void OperatingPoints_clear(long jarg1, OperatingPoints jarg1_);\n  public final static native boolean OperatingPoints_add__SWIG_0(long jarg1, OperatingPoints jarg1_, double jarg2, double jarg3, String jarg4, long jarg5);\n  public final static native boolean OperatingPoints_add__SWIG_1(long jarg1, OperatingPoints jarg1_, double jarg2, double jarg3, String jarg4);\n  public final static native double OperatingPoints_t_for_perf(long jarg1, OperatingPoints jarg1_, double jarg2);\n  public final static native void OperatingPoints_display__SWIG_0(long jarg1, OperatingPoints jarg1_, boolean jarg2);\n  public final static native void OperatingPoints_display__SWIG_1(long jarg1, OperatingPoints jarg1_);\n  public final static native void OperatingPoints_all_to_gnuplot(long jarg1, OperatingPoints jarg1_, String jarg2);\n  public final static native void OperatingPoints_optimal_to_gnuplot(long jarg1, OperatingPoints jarg1_, String jarg2);\n  public final static native void delete_OperatingPoints(long jarg1);\n  public final static native void ParameterRange_name_set(long jarg1, ParameterRange jarg1_, String jarg2);\n  public final static native String ParameterRange_name_get(long jarg1, ParameterRange jarg1_);\n  public final static native void ParameterRange_values_set(long jarg1, ParameterRange jarg1_, long jarg2, DoubleVector jarg2_);\n  public final static native long ParameterRange_values_get(long jarg1, ParameterRange jarg1_);\n  public final static native long new_ParameterRange();\n  public final static native void delete_ParameterRange(long jarg1);\n  public final static native void ParameterSpace_parameter_ranges_set(long jarg1, ParameterSpace jarg1_, long jarg2);\n  public final static native long ParameterSpace_parameter_ranges_get(long jarg1, ParameterSpace jarg1_);\n  public final static native void ParameterSpace_verbose_set(long jarg1, ParameterSpace jarg1_, int jarg2);\n  public final static native int ParameterSpace_verbose_get(long jarg1, ParameterSpace jarg1_);\n  public final static native void ParameterSpace_n_experiments_set(long jarg1, ParameterSpace jarg1_, int jarg2);\n  public final static native int ParameterSpace_n_experiments_get(long jarg1, ParameterSpace jarg1_);\n  public final static native void ParameterSpace_batchsize_set(long jarg1, ParameterSpace jarg1_, long jarg2);\n  public final static native long ParameterSpace_batchsize_get(long jarg1, ParameterSpace jarg1_);\n  public final static native void ParameterSpace_thread_over_batches_set(long jarg1, ParameterSpace jarg1_, boolean jarg2);\n  public final static native boolean ParameterSpace_thread_over_batches_get(long jarg1, ParameterSpace jarg1_);\n  public final static native void ParameterSpace_min_test_duration_set(long jarg1, ParameterSpace jarg1_, double jarg2);\n  public final static native double ParameterSpace_min_test_duration_get(long jarg1, ParameterSpace jarg1_);\n  public final static native long new_ParameterSpace();\n  public final static native long ParameterSpace_n_combinations(long jarg1, ParameterSpace jarg1_);\n  public final static native boolean ParameterSpace_combination_ge(long jarg1, ParameterSpace jarg1_, long jarg2, long jarg3);\n  public final static native String ParameterSpace_combination_name(long jarg1, ParameterSpace jarg1_, long jarg2);\n  public final static native void ParameterSpace_display(long jarg1, ParameterSpace jarg1_);\n  public final static native long ParameterSpace_add_range(long jarg1, ParameterSpace jarg1_, String jarg2);\n  public final static native void ParameterSpace_initialize(long jarg1, ParameterSpace jarg1_, long jarg2, Index jarg2_);\n  public final static native void ParameterSpace_set_index_parameters__SWIG_0(long jarg1, ParameterSpace jarg1_, long jarg2, Index jarg2_, long jarg3);\n  public final static native void ParameterSpace_set_index_parameters__SWIG_1(long jarg1, ParameterSpace jarg1_, long jarg2, Index jarg2_, String jarg3);\n  public final static native void ParameterSpace_set_index_parameter(long jarg1, ParameterSpace jarg1_, long jarg2, Index jarg2_, String jarg3, double jarg4);\n  public final static native void ParameterSpace_update_bounds(long jarg1, ParameterSpace jarg1_, long jarg2, long jarg3, OperatingPoint jarg3_, long jarg4, long jarg5);\n  public final static native void ParameterSpace_explore(long jarg1, ParameterSpace jarg1_, long jarg2, Index jarg2_, long jarg3, long jarg4, long jarg5, AutoTuneCriterion jarg5_, long jarg6, OperatingPoints jarg6_);\n  public final static native void delete_ParameterSpace(long jarg1);\n  public final static native long index_factory__SWIG_0(int jarg1, String jarg2, int jarg3);\n  public final static native long index_factory__SWIG_1(int jarg1, String jarg2);\n  public final static native void index_factory_verbose_set(int jarg1);\n  public final static native int index_factory_verbose_get();\n  public final static native long index_binary_factory(int jarg1, String jarg2);\n  public final static native void simd_histogram_8(long jarg1, int jarg2, long jarg3, int jarg4, long jarg5);\n  public final static native void simd_histogram_16(long jarg1, int jarg2, long jarg3, int jarg4, long jarg5);\n  public final static native void PartitionStats_bissect_cycles_set(long jarg1, PartitionStats jarg1_, long jarg2);\n  public final static native long PartitionStats_bissect_cycles_get(long jarg1, PartitionStats jarg1_);\n  public final static native void PartitionStats_compress_cycles_set(long jarg1, PartitionStats jarg1_, long jarg2);\n  public final static native long PartitionStats_compress_cycles_get(long jarg1, PartitionStats jarg1_);\n  public final static native long new_PartitionStats();\n  public final static native void PartitionStats_reset(long jarg1, PartitionStats jarg1_);\n  public final static native void delete_PartitionStats(long jarg1);\n  public final static native void partition_stats_set(long jarg1, PartitionStats jarg1_);\n  public final static native long partition_stats_get();\n  public final static native void float_minheap_array_t_nh_set(long jarg1, float_minheap_array_t jarg1_, long jarg2);\n  public final static native long float_minheap_array_t_nh_get(long jarg1, float_minheap_array_t jarg1_);\n  public final static native void float_minheap_array_t_k_set(long jarg1, float_minheap_array_t jarg1_, long jarg2);\n  public final static native long float_minheap_array_t_k_get(long jarg1, float_minheap_array_t jarg1_);\n  public final static native void float_minheap_array_t_ids_set(long jarg1, float_minheap_array_t jarg1_, long jarg2, LongVector jarg2_);\n  public final static native long float_minheap_array_t_ids_get(long jarg1, float_minheap_array_t jarg1_);\n  public final static native void float_minheap_array_t_val_set(long jarg1, float_minheap_array_t jarg1_, long jarg2);\n  public final static native long float_minheap_array_t_val_get(long jarg1, float_minheap_array_t jarg1_);\n  public final static native long float_minheap_array_t_get_val(long jarg1, float_minheap_array_t jarg1_, long jarg2);\n  public final static native long float_minheap_array_t_get_ids(long jarg1, float_minheap_array_t jarg1_, long jarg2);\n  public final static native void float_minheap_array_t_heapify(long jarg1, float_minheap_array_t jarg1_);\n  public final static native void float_minheap_array_t_addn__SWIG_0(long jarg1, float_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native void float_minheap_array_t_addn__SWIG_1(long jarg1, float_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, long jarg5);\n  public final static native void float_minheap_array_t_addn__SWIG_2(long jarg1, float_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void float_minheap_array_t_addn__SWIG_3(long jarg1, float_minheap_array_t jarg1_, long jarg2, long jarg3);\n  public final static native void float_minheap_array_t_addn_with_ids__SWIG_0(long jarg1, float_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, long jarg6, long jarg7);\n  public final static native void float_minheap_array_t_addn_with_ids__SWIG_1(long jarg1, float_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, long jarg6);\n  public final static native void float_minheap_array_t_addn_with_ids__SWIG_2(long jarg1, float_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void float_minheap_array_t_addn_with_ids__SWIG_3(long jarg1, float_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void float_minheap_array_t_addn_with_ids__SWIG_4(long jarg1, float_minheap_array_t jarg1_, long jarg2, long jarg3);\n  public final static native void float_minheap_array_t_reorder(long jarg1, float_minheap_array_t jarg1_);\n  public final static native void float_minheap_array_t_per_line_extrema(long jarg1, float_minheap_array_t jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native long new_float_minheap_array_t();\n  public final static native void delete_float_minheap_array_t(long jarg1);\n  public final static native void int_minheap_array_t_nh_set(long jarg1, int_minheap_array_t jarg1_, long jarg2);\n  public final static native long int_minheap_array_t_nh_get(long jarg1, int_minheap_array_t jarg1_);\n  public final static native void int_minheap_array_t_k_set(long jarg1, int_minheap_array_t jarg1_, long jarg2);\n  public final static native long int_minheap_array_t_k_get(long jarg1, int_minheap_array_t jarg1_);\n  public final static native void int_minheap_array_t_ids_set(long jarg1, int_minheap_array_t jarg1_, long jarg2, LongVector jarg2_);\n  public final static native long int_minheap_array_t_ids_get(long jarg1, int_minheap_array_t jarg1_);\n  public final static native void int_minheap_array_t_val_set(long jarg1, int_minheap_array_t jarg1_, long jarg2);\n  public final static native long int_minheap_array_t_val_get(long jarg1, int_minheap_array_t jarg1_);\n  public final static native long int_minheap_array_t_get_val(long jarg1, int_minheap_array_t jarg1_, long jarg2);\n  public final static native long int_minheap_array_t_get_ids(long jarg1, int_minheap_array_t jarg1_, long jarg2);\n  public final static native void int_minheap_array_t_heapify(long jarg1, int_minheap_array_t jarg1_);\n  public final static native void int_minheap_array_t_addn__SWIG_0(long jarg1, int_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native void int_minheap_array_t_addn__SWIG_1(long jarg1, int_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, long jarg5);\n  public final static native void int_minheap_array_t_addn__SWIG_2(long jarg1, int_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void int_minheap_array_t_addn__SWIG_3(long jarg1, int_minheap_array_t jarg1_, long jarg2, long jarg3);\n  public final static native void int_minheap_array_t_addn_with_ids__SWIG_0(long jarg1, int_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, long jarg6, long jarg7);\n  public final static native void int_minheap_array_t_addn_with_ids__SWIG_1(long jarg1, int_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, long jarg6);\n  public final static native void int_minheap_array_t_addn_with_ids__SWIG_2(long jarg1, int_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void int_minheap_array_t_addn_with_ids__SWIG_3(long jarg1, int_minheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void int_minheap_array_t_addn_with_ids__SWIG_4(long jarg1, int_minheap_array_t jarg1_, long jarg2, long jarg3);\n  public final static native void int_minheap_array_t_reorder(long jarg1, int_minheap_array_t jarg1_);\n  public final static native void int_minheap_array_t_per_line_extrema(long jarg1, int_minheap_array_t jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native long new_int_minheap_array_t();\n  public final static native void delete_int_minheap_array_t(long jarg1);\n  public final static native void float_maxheap_array_t_nh_set(long jarg1, float_maxheap_array_t jarg1_, long jarg2);\n  public final static native long float_maxheap_array_t_nh_get(long jarg1, float_maxheap_array_t jarg1_);\n  public final static native void float_maxheap_array_t_k_set(long jarg1, float_maxheap_array_t jarg1_, long jarg2);\n  public final static native long float_maxheap_array_t_k_get(long jarg1, float_maxheap_array_t jarg1_);\n  public final static native void float_maxheap_array_t_ids_set(long jarg1, float_maxheap_array_t jarg1_, long jarg2, LongVector jarg2_);\n  public final static native long float_maxheap_array_t_ids_get(long jarg1, float_maxheap_array_t jarg1_);\n  public final static native void float_maxheap_array_t_val_set(long jarg1, float_maxheap_array_t jarg1_, long jarg2);\n  public final static native long float_maxheap_array_t_val_get(long jarg1, float_maxheap_array_t jarg1_);\n  public final static native long float_maxheap_array_t_get_val(long jarg1, float_maxheap_array_t jarg1_, long jarg2);\n  public final static native long float_maxheap_array_t_get_ids(long jarg1, float_maxheap_array_t jarg1_, long jarg2);\n  public final static native void float_maxheap_array_t_heapify(long jarg1, float_maxheap_array_t jarg1_);\n  public final static native void float_maxheap_array_t_addn__SWIG_0(long jarg1, float_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native void float_maxheap_array_t_addn__SWIG_1(long jarg1, float_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, long jarg5);\n  public final static native void float_maxheap_array_t_addn__SWIG_2(long jarg1, float_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void float_maxheap_array_t_addn__SWIG_3(long jarg1, float_maxheap_array_t jarg1_, long jarg2, long jarg3);\n  public final static native void float_maxheap_array_t_addn_with_ids__SWIG_0(long jarg1, float_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, long jarg6, long jarg7);\n  public final static native void float_maxheap_array_t_addn_with_ids__SWIG_1(long jarg1, float_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, long jarg6);\n  public final static native void float_maxheap_array_t_addn_with_ids__SWIG_2(long jarg1, float_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void float_maxheap_array_t_addn_with_ids__SWIG_3(long jarg1, float_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void float_maxheap_array_t_addn_with_ids__SWIG_4(long jarg1, float_maxheap_array_t jarg1_, long jarg2, long jarg3);\n  public final static native void float_maxheap_array_t_reorder(long jarg1, float_maxheap_array_t jarg1_);\n  public final static native void float_maxheap_array_t_per_line_extrema(long jarg1, float_maxheap_array_t jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native long new_float_maxheap_array_t();\n  public final static native void delete_float_maxheap_array_t(long jarg1);\n  public final static native void int_maxheap_array_t_nh_set(long jarg1, int_maxheap_array_t jarg1_, long jarg2);\n  public final static native long int_maxheap_array_t_nh_get(long jarg1, int_maxheap_array_t jarg1_);\n  public final static native void int_maxheap_array_t_k_set(long jarg1, int_maxheap_array_t jarg1_, long jarg2);\n  public final static native long int_maxheap_array_t_k_get(long jarg1, int_maxheap_array_t jarg1_);\n  public final static native void int_maxheap_array_t_ids_set(long jarg1, int_maxheap_array_t jarg1_, long jarg2, LongVector jarg2_);\n  public final static native long int_maxheap_array_t_ids_get(long jarg1, int_maxheap_array_t jarg1_);\n  public final static native void int_maxheap_array_t_val_set(long jarg1, int_maxheap_array_t jarg1_, long jarg2);\n  public final static native long int_maxheap_array_t_val_get(long jarg1, int_maxheap_array_t jarg1_);\n  public final static native long int_maxheap_array_t_get_val(long jarg1, int_maxheap_array_t jarg1_, long jarg2);\n  public final static native long int_maxheap_array_t_get_ids(long jarg1, int_maxheap_array_t jarg1_, long jarg2);\n  public final static native void int_maxheap_array_t_heapify(long jarg1, int_maxheap_array_t jarg1_);\n  public final static native void int_maxheap_array_t_addn__SWIG_0(long jarg1, int_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native void int_maxheap_array_t_addn__SWIG_1(long jarg1, int_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, long jarg5);\n  public final static native void int_maxheap_array_t_addn__SWIG_2(long jarg1, int_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native void int_maxheap_array_t_addn__SWIG_3(long jarg1, int_maxheap_array_t jarg1_, long jarg2, long jarg3);\n  public final static native void int_maxheap_array_t_addn_with_ids__SWIG_0(long jarg1, int_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, long jarg6, long jarg7);\n  public final static native void int_maxheap_array_t_addn_with_ids__SWIG_1(long jarg1, int_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5, long jarg6);\n  public final static native void int_maxheap_array_t_addn_with_ids__SWIG_2(long jarg1, int_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void int_maxheap_array_t_addn_with_ids__SWIG_3(long jarg1, int_maxheap_array_t jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_);\n  public final static native void int_maxheap_array_t_addn_with_ids__SWIG_4(long jarg1, int_maxheap_array_t jarg1_, long jarg2, long jarg3);\n  public final static native void int_maxheap_array_t_reorder(long jarg1, int_maxheap_array_t jarg1_);\n  public final static native void int_maxheap_array_t_per_line_extrema(long jarg1, int_maxheap_array_t jarg1_, long jarg2, long jarg3, LongVector jarg3_);\n  public final static native long new_int_maxheap_array_t();\n  public final static native void delete_int_maxheap_array_t(long jarg1);\n  public final static native float CMin_float_partition_fuzzy(long jarg1, long jarg2, LongVector jarg2_, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native float CMax_float_partition_fuzzy(long jarg1, long jarg2, LongVector jarg2_, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native void AlignedTableUint8_tab_set(long jarg1, AlignedTableUint8 jarg1_, long jarg2);\n  public final static native long AlignedTableUint8_tab_get(long jarg1, AlignedTableUint8 jarg1_);\n  public final static native void AlignedTableUint8_numel_set(long jarg1, AlignedTableUint8 jarg1_, long jarg2);\n  public final static native long AlignedTableUint8_numel_get(long jarg1, AlignedTableUint8 jarg1_);\n  public final static native long AlignedTableUint8_round_capacity(long jarg1);\n  public final static native long new_AlignedTableUint8__SWIG_0();\n  public final static native long new_AlignedTableUint8__SWIG_1(long jarg1);\n  public final static native long AlignedTableUint8_itemsize(long jarg1, AlignedTableUint8 jarg1_);\n  public final static native void AlignedTableUint8_resize(long jarg1, AlignedTableUint8 jarg1_, long jarg2);\n  public final static native void AlignedTableUint8_clear(long jarg1, AlignedTableUint8 jarg1_);\n  public final static native long AlignedTableUint8_size(long jarg1, AlignedTableUint8 jarg1_);\n  public final static native long AlignedTableUint8_nbytes(long jarg1, AlignedTableUint8 jarg1_);\n  public final static native long AlignedTableUint8_get__SWIG_0(long jarg1, AlignedTableUint8 jarg1_);\n  public final static native long AlignedTableUint8_data__SWIG_0(long jarg1, AlignedTableUint8 jarg1_);\n  public final static native void delete_AlignedTableUint8(long jarg1);\n  public final static native void AlignedTableUint16_tab_set(long jarg1, AlignedTableUint16 jarg1_, long jarg2);\n  public final static native long AlignedTableUint16_tab_get(long jarg1, AlignedTableUint16 jarg1_);\n  public final static native void AlignedTableUint16_numel_set(long jarg1, AlignedTableUint16 jarg1_, long jarg2);\n  public final static native long AlignedTableUint16_numel_get(long jarg1, AlignedTableUint16 jarg1_);\n  public final static native long AlignedTableUint16_round_capacity(long jarg1);\n  public final static native long new_AlignedTableUint16__SWIG_0();\n  public final static native long new_AlignedTableUint16__SWIG_1(long jarg1);\n  public final static native long AlignedTableUint16_itemsize(long jarg1, AlignedTableUint16 jarg1_);\n  public final static native void AlignedTableUint16_resize(long jarg1, AlignedTableUint16 jarg1_, long jarg2);\n  public final static native void AlignedTableUint16_clear(long jarg1, AlignedTableUint16 jarg1_);\n  public final static native long AlignedTableUint16_size(long jarg1, AlignedTableUint16 jarg1_);\n  public final static native long AlignedTableUint16_nbytes(long jarg1, AlignedTableUint16 jarg1_);\n  public final static native long AlignedTableUint16_get__SWIG_0(long jarg1, AlignedTableUint16 jarg1_);\n  public final static native long AlignedTableUint16_data__SWIG_0(long jarg1, AlignedTableUint16 jarg1_);\n  public final static native void delete_AlignedTableUint16(long jarg1);\n  public final static native void AlignedTableFloat32_tab_set(long jarg1, AlignedTableFloat32 jarg1_, long jarg2);\n  public final static native long AlignedTableFloat32_tab_get(long jarg1, AlignedTableFloat32 jarg1_);\n  public final static native void AlignedTableFloat32_numel_set(long jarg1, AlignedTableFloat32 jarg1_, long jarg2);\n  public final static native long AlignedTableFloat32_numel_get(long jarg1, AlignedTableFloat32 jarg1_);\n  public final static native long AlignedTableFloat32_round_capacity(long jarg1);\n  public final static native long new_AlignedTableFloat32__SWIG_0();\n  public final static native long new_AlignedTableFloat32__SWIG_1(long jarg1);\n  public final static native long AlignedTableFloat32_itemsize(long jarg1, AlignedTableFloat32 jarg1_);\n  public final static native void AlignedTableFloat32_resize(long jarg1, AlignedTableFloat32 jarg1_, long jarg2);\n  public final static native void AlignedTableFloat32_clear(long jarg1, AlignedTableFloat32 jarg1_);\n  public final static native long AlignedTableFloat32_size(long jarg1, AlignedTableFloat32 jarg1_);\n  public final static native long AlignedTableFloat32_nbytes(long jarg1, AlignedTableFloat32 jarg1_);\n  public final static native long AlignedTableFloat32_get__SWIG_0(long jarg1, AlignedTableFloat32 jarg1_);\n  public final static native long AlignedTableFloat32_data__SWIG_0(long jarg1, AlignedTableFloat32 jarg1_);\n  public final static native void delete_AlignedTableFloat32(long jarg1);\n  public final static native long CMax_uint16_partition_fuzzy__SWIG_0(long jarg1, long jarg2, LongVector jarg2_, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native long CMin_uint16_partition_fuzzy__SWIG_0(long jarg1, long jarg2, LongVector jarg2_, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native long CMax_uint16_partition_fuzzy__SWIG_1(long jarg1, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native long CMin_uint16_partition_fuzzy__SWIG_1(long jarg1, long jarg2, long jarg3, long jarg4, long jarg5, long jarg6);\n  public final static native void omp_set_num_threads(int jarg1);\n  public final static native int omp_get_max_threads();\n  public final static native long memcpy(long jarg1, long jarg2, long jarg3);\n  public final static native long cast_integer_to_float_ptr(int jarg1);\n  public final static native long cast_integer_to_long_ptr(int jarg1);\n  public final static native long cast_integer_to_int_ptr(int jarg1);\n  public final static native void RangeSearchResult_nq_set(long jarg1, RangeSearchResult jarg1_, long jarg2);\n  public final static native long RangeSearchResult_nq_get(long jarg1, RangeSearchResult jarg1_);\n  public final static native void RangeSearchResult_lims_set(long jarg1, RangeSearchResult jarg1_, long jarg2);\n  public final static native long RangeSearchResult_lims_get(long jarg1, RangeSearchResult jarg1_);\n  public final static native void RangeSearchResult_labels_set(long jarg1, RangeSearchResult jarg1_, long jarg2, LongVector jarg2_);\n  public final static native long RangeSearchResult_labels_get(long jarg1, RangeSearchResult jarg1_);\n  public final static native void RangeSearchResult_distances_set(long jarg1, RangeSearchResult jarg1_, long jarg2);\n  public final static native long RangeSearchResult_distances_get(long jarg1, RangeSearchResult jarg1_);\n  public final static native void RangeSearchResult_buffer_size_set(long jarg1, RangeSearchResult jarg1_, long jarg2);\n  public final static native long RangeSearchResult_buffer_size_get(long jarg1, RangeSearchResult jarg1_);\n  public final static native void RangeSearchResult_do_allocation(long jarg1, RangeSearchResult jarg1_);\n  public final static native void delete_RangeSearchResult(long jarg1);\n  public final static native boolean IDSelector_is_member(long jarg1, IDSelector jarg1_, long jarg2);\n  public final static native void delete_IDSelector(long jarg1);\n  public final static native void IDSelectorRange_imin_set(long jarg1, IDSelectorRange jarg1_, long jarg2);\n  public final static native long IDSelectorRange_imin_get(long jarg1, IDSelectorRange jarg1_);\n  public final static native void IDSelectorRange_imax_set(long jarg1, IDSelectorRange jarg1_, long jarg2);\n  public final static native long IDSelectorRange_imax_get(long jarg1, IDSelectorRange jarg1_);\n  public final static native long new_IDSelectorRange(long jarg1, long jarg2);\n  public final static native boolean IDSelectorRange_is_member(long jarg1, IDSelectorRange jarg1_, long jarg2);\n  public final static native void delete_IDSelectorRange(long jarg1);\n  public final static native void IDSelectorArray_n_set(long jarg1, IDSelectorArray jarg1_, long jarg2);\n  public final static native long IDSelectorArray_n_get(long jarg1, IDSelectorArray jarg1_);\n  public final static native void IDSelectorArray_ids_set(long jarg1, IDSelectorArray jarg1_, long jarg2, LongVector jarg2_);\n  public final static native long IDSelectorArray_ids_get(long jarg1, IDSelectorArray jarg1_);\n  public final static native long new_IDSelectorArray(long jarg1, long jarg2, LongVector jarg2_);\n  public final static native boolean IDSelectorArray_is_member(long jarg1, IDSelectorArray jarg1_, long jarg2);\n  public final static native void delete_IDSelectorArray(long jarg1);\n  public final static native void IDSelectorBatch_nbits_set(long jarg1, IDSelectorBatch jarg1_, int jarg2);\n  public final static native int IDSelectorBatch_nbits_get(long jarg1, IDSelectorBatch jarg1_);\n  public final static native void IDSelectorBatch_mask_set(long jarg1, IDSelectorBatch jarg1_, long jarg2);\n  public final static native long IDSelectorBatch_mask_get(long jarg1, IDSelectorBatch jarg1_);\n  public final static native long new_IDSelectorBatch(long jarg1, long jarg2, LongVector jarg2_);\n  public final static native boolean IDSelectorBatch_is_member(long jarg1, IDSelectorBatch jarg1_, long jarg2);\n  public final static native void delete_IDSelectorBatch(long jarg1);\n  public final static native void BufferList_buffer_size_set(long jarg1, BufferList jarg1_, long jarg2);\n  public final static native long BufferList_buffer_size_get(long jarg1, BufferList jarg1_);\n  public final static native void BufferList_buffers_set(long jarg1, BufferList jarg1_, long jarg2);\n  public final static native long BufferList_buffers_get(long jarg1, BufferList jarg1_);\n  public final static native void BufferList_wp_set(long jarg1, BufferList jarg1_, long jarg2);\n  public final static native long BufferList_wp_get(long jarg1, BufferList jarg1_);\n  public final static native long new_BufferList(long jarg1);\n  public final static native void delete_BufferList(long jarg1);\n  public final static native void BufferList_append_buffer(long jarg1, BufferList jarg1_);\n  public final static native void BufferList_add(long jarg1, BufferList jarg1_, long jarg2, float jarg3);\n  public final static native void BufferList_copy_range(long jarg1, BufferList jarg1_, long jarg2, long jarg3, long jarg4, LongVector jarg4_, long jarg5);\n  public final static native void RangeQueryResult_qno_set(long jarg1, RangeQueryResult jarg1_, long jarg2);\n  public final static native long RangeQueryResult_qno_get(long jarg1, RangeQueryResult jarg1_);\n  public final static native void RangeQueryResult_nres_set(long jarg1, RangeQueryResult jarg1_, long jarg2);\n  public final static native long RangeQueryResult_nres_get(long jarg1, RangeQueryResult jarg1_);\n  public final static native void RangeQueryResult_pres_set(long jarg1, RangeQueryResult jarg1_, long jarg2, RangeSearchPartialResult jarg2_);\n  public final static native long RangeQueryResult_pres_get(long jarg1, RangeQueryResult jarg1_);\n  public final static native void RangeQueryResult_add(long jarg1, RangeQueryResult jarg1_, float jarg2, long jarg3);\n  public final static native long new_RangeQueryResult();\n  public final static native void delete_RangeQueryResult(long jarg1);\n  public final static native void RangeSearchPartialResult_res_set(long jarg1, RangeSearchPartialResult jarg1_, long jarg2, RangeSearchResult jarg2_);\n  public final static native long RangeSearchPartialResult_res_get(long jarg1, RangeSearchPartialResult jarg1_);\n  public final static native void RangeSearchPartialResult_queries_set(long jarg1, RangeSearchPartialResult jarg1_, long jarg2);\n  public final static native long RangeSearchPartialResult_queries_get(long jarg1, RangeSearchPartialResult jarg1_);\n  public final static native long RangeSearchPartialResult_new_result(long jarg1, RangeSearchPartialResult jarg1_, long jarg2);\n  public final static native void RangeSearchPartialResult_set_lims(long jarg1, RangeSearchPartialResult jarg1_);\n  public final static native void RangeSearchPartialResult_copy_result__SWIG_0(long jarg1, RangeSearchPartialResult jarg1_, boolean jarg2);\n  public final static native void RangeSearchPartialResult_copy_result__SWIG_1(long jarg1, RangeSearchPartialResult jarg1_);\n  public final static native void RangeSearchPartialResult_merge__SWIG_0(long jarg1, boolean jarg2);\n  public final static native void RangeSearchPartialResult_merge__SWIG_1(long jarg1);\n  public final static native void delete_RangeSearchPartialResult(long jarg1);\n  public final static native void DistanceComputer_set_query(long jarg1, DistanceComputer jarg1_, long jarg2);\n  public final static native float DistanceComputer_symmetric_dis(long jarg1, DistanceComputer jarg1_, long jarg2, long jarg3);\n  public final static native void delete_DistanceComputer(long jarg1);\n  public final static native boolean InterruptCallback_want_interrupt(long jarg1, InterruptCallback jarg1_);\n  public final static native void delete_InterruptCallback(long jarg1);\n  public final static native void InterruptCallback_clear_instance();\n  public final static native void InterruptCallback_check();\n  public final static native boolean InterruptCallback_is_interrupted();\n  public final static native long InterruptCallback_get_period_hint(long jarg1);\n  public final static native void VisitedTable_visited_set(long jarg1, VisitedTable jarg1_, long jarg2, ByteVector jarg2_);\n  public final static native long VisitedTable_visited_get(long jarg1, VisitedTable jarg1_);\n  public final static native void VisitedTable_visno_set(long jarg1, VisitedTable jarg1_, int jarg2);\n  public final static native int VisitedTable_visno_get(long jarg1, VisitedTable jarg1_);\n  public final static native long new_VisitedTable(int jarg1);\n  public final static native void VisitedTable_set(long jarg1, VisitedTable jarg1_, int jarg2);\n  public final static native boolean VisitedTable_get(long jarg1, VisitedTable jarg1_, int jarg2);\n  public final static native void VisitedTable_advance(long jarg1, VisitedTable jarg1_);\n  public final static native void delete_VisitedTable(long jarg1);\n  public final static native void ignore_SIGTTIN();\n  public final static native void MapLong2Long_map_set(long jarg1, MapLong2Long jarg1_, long jarg2);\n  public final static native long MapLong2Long_map_get(long jarg1, MapLong2Long jarg1_);\n  public final static native void MapLong2Long_add(long jarg1, MapLong2Long jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native int MapLong2Long_search(long jarg1, MapLong2Long jarg1_, int jarg2);\n  public final static native void MapLong2Long_search_multiple(long jarg1, MapLong2Long jarg1_, long jarg2, long jarg3, long jarg4);\n  public final static native long new_MapLong2Long();\n  public final static native void delete_MapLong2Long(long jarg1);\n  public final static native long Clustering_SWIGUpcast(long jarg1);\n  public final static native long Clustering1D_SWIGUpcast(long jarg1);\n  public final static native long ProgressiveDimClusteringParameters_SWIGUpcast(long jarg1);\n  public final static native long ProgressiveDimClustering_SWIGUpcast(long jarg1);\n  public final static native long LinearTransform_SWIGUpcast(long jarg1);\n  public final static native long RandomRotationMatrix_SWIGUpcast(long jarg1);\n  public final static native long PCAMatrix_SWIGUpcast(long jarg1);\n  public final static native long ITQMatrix_SWIGUpcast(long jarg1);\n  public final static native long ITQTransform_SWIGUpcast(long jarg1);\n  public final static native long OPQMatrix_SWIGUpcast(long jarg1);\n  public final static native long RemapDimensionsTransform_SWIGUpcast(long jarg1);\n  public final static native long NormalizationTransform_SWIGUpcast(long jarg1);\n  public final static native long CenteringTransform_SWIGUpcast(long jarg1);\n  public final static native long IndexFlatCodes_SWIGUpcast(long jarg1);\n  public final static native long IndexFlat_SWIGUpcast(long jarg1);\n  public final static native long IndexFlatIP_SWIGUpcast(long jarg1);\n  public final static native long IndexFlatL2_SWIGUpcast(long jarg1);\n  public final static native long IndexFlat1D_SWIGUpcast(long jarg1);\n  public final static native long IndexLSH_SWIGUpcast(long jarg1);\n  public final static native long ReproduceDistancesObjective_SWIGUpcast(long jarg1);\n  public final static native long SimulatedAnnealingOptimizer_SWIGUpcast(long jarg1);\n  public final static native long PolysemousTraining_SWIGUpcast(long jarg1);\n  public final static native long IndexPQ_SWIGUpcast(long jarg1);\n  public final static native long MultiIndexQuantizer_SWIGUpcast(long jarg1);\n  public final static native long MultiIndexQuantizer2_SWIGUpcast(long jarg1);\n  public final static native long ArrayInvertedLists_SWIGUpcast(long jarg1);\n  public final static native long ReadOnlyInvertedLists_SWIGUpcast(long jarg1);\n  public final static native long HStackInvertedLists_SWIGUpcast(long jarg1);\n  public final static native long SliceInvertedLists_SWIGUpcast(long jarg1);\n  public final static native long VStackInvertedLists_SWIGUpcast(long jarg1);\n  public final static native long MaskedInvertedLists_SWIGUpcast(long jarg1);\n  public final static native long StopWordsInvertedLists_SWIGUpcast(long jarg1);\n  public final static native long IndexIVF_SWIGUpcast(long jarg1);\n  public final static native long IndexScalarQuantizer_SWIGUpcast(long jarg1);\n  public final static native long IndexIVFScalarQuantizer_SWIGUpcast(long jarg1);\n  public final static native long IndexHNSW_SWIGUpcast(long jarg1);\n  public final static native long IndexHNSWFlat_SWIGUpcast(long jarg1);\n  public final static native long IndexHNSWPQ_SWIGUpcast(long jarg1);\n  public final static native long IndexHNSWSQ_SWIGUpcast(long jarg1);\n  public final static native long IndexHNSW2Level_SWIGUpcast(long jarg1);\n  public final static native long IndexIVFFlat_SWIGUpcast(long jarg1);\n  public final static native long IndexIVFFlatDedup_SWIGUpcast(long jarg1);\n  public final static native long OnDiskInvertedLists_SWIGUpcast(long jarg1);\n  public final static native long IVFPQSearchParameters_SWIGUpcast(long jarg1);\n  public final static native long IndexIVFPQ_SWIGUpcast(long jarg1);\n  public final static native long Index2Layer_SWIGUpcast(long jarg1);\n  public final static native long IndexBinaryFlat_SWIGUpcast(long jarg1);\n  public final static native long IndexBinaryIVF_SWIGUpcast(long jarg1);\n  public final static native long IndexBinaryFromFloat_SWIGUpcast(long jarg1);\n  public final static native long IndexBinaryHNSW_SWIGUpcast(long jarg1);\n  public final static native long IndexRefine_SWIGUpcast(long jarg1);\n  public final static native long IndexRefineFlat_SWIGUpcast(long jarg1);\n  public final static native long IndexSplitVectors_SWIGUpcast(long jarg1);\n  public final static native long IndexIDMap_SWIGUpcast(long jarg1);\n  public final static native long OneRecallAtRCriterion_SWIGUpcast(long jarg1);\n  public final static native long IntersectionCriterion_SWIGUpcast(long jarg1);\n  public final static native long IDSelectorRange_SWIGUpcast(long jarg1);\n  public final static native long IDSelectorArray_SWIGUpcast(long jarg1);\n  public final static native long IDSelectorBatch_SWIGUpcast(long jarg1);\n  public final static native long RangeSearchPartialResult_SWIGUpcast(long jarg1);\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/hnsw/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/twitter/bijection:core\",\n        \"3rdparty/jvm/commons-lang\",\n        \"3rdparty/jvm/org/apache/thrift\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-java\",\n        \"mediaservices/commons/src/main/scala:futuretracker\",\n        \"scrooge/scrooge-core\",\n        \"src/java/com/twitter/search/common/file\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/hnsw/DistanceFunction.java",
    "content": "package com.twitter.ann.hnsw;\n\npublic interface DistanceFunction<T, Q> {\n  /**\n   * Distance between two items.\n   */\n  float distance(T t, Q q);\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/hnsw/DistancedItem.java",
    "content": "package com.twitter.ann.hnsw;\n\n/**\n * An item associated with a float distance\n * @param <T> The type of the item.\n */\npublic class DistancedItem<T> {\n  private final T item;\n  private final float distance;\n\n  public DistancedItem(T item, float distance) {\n    this.item = item;\n    this.distance = distance;\n  }\n\n  public T getItem() {\n    return item;\n  }\n\n  public float getDistance() {\n    return distance;\n  }\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/hnsw/DistancedItemQueue.java",
    "content": "package com.twitter.ann.hnsw;\n\nimport java.util.ArrayList;\nimport java.util.Comparator;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.PriorityQueue;\n\n/**\n * Container for items with their distance.\n *\n * @param <U> Type of origin/reference element.\n * @param <T> Type of element that the queue will hold\n */\npublic class DistancedItemQueue<U, T> implements Iterable<DistancedItem<T>> {\n  private final U origin;\n  private final DistanceFunction<U, T> distFn;\n  private final PriorityQueue<DistancedItem<T>> queue;\n  private final boolean minQueue;\n  /**\n   * Creates ontainer for items with their distances.\n   *\n   * @param origin Origin (reference) point\n   * @param initial Initial list of elements to add in the structure\n   * @param minQueue True for min queue, False for max queue\n   * @param distFn Distance function\n   */\n  public DistancedItemQueue(\n      U origin,\n      List<T> initial,\n      boolean minQueue,\n      DistanceFunction<U, T> distFn\n  ) {\n    this.origin = origin;\n    this.distFn = distFn;\n    this.minQueue = minQueue;\n    final Comparator<DistancedItem<T>> cmp;\n    if (minQueue) {\n      cmp = (o1, o2) -> Float.compare(o1.getDistance(), o2.getDistance());\n    } else {\n      cmp = (o1, o2) -> Float.compare(o2.getDistance(), o1.getDistance());\n    }\n    this.queue = new PriorityQueue<>(cmp);\n    enqueueAll(initial);\n    new DistancedItemQueue<>(origin, distFn, queue, minQueue);\n  }\n\n  private DistancedItemQueue(\n      U origin,\n      DistanceFunction<U, T> distFn,\n      PriorityQueue<DistancedItem<T>> queue,\n      boolean minQueue\n  ) {\n    this.origin = origin;\n    this.distFn = distFn;\n    this.queue = queue;\n    this.minQueue = minQueue;\n  }\n\n  /**\n   * Enqueues all the items into the queue.\n   */\n  public void enqueueAll(List<T> list) {\n    for (T t : list) {\n      enqueue(t);\n    }\n  }\n\n  /**\n   * Return if queue is non empty or not\n   *\n   * @return true if queue is not empty else false\n   */\n  public boolean nonEmpty() {\n    return !queue.isEmpty();\n  }\n\n  /**\n   * Return root of the queue\n   *\n   * @return root of the queue i.e min/max element depending upon min-max queue\n   */\n  public DistancedItem<T> peek() {\n    return queue.peek();\n  }\n\n  /**\n   * Dequeue root of the queue.\n   *\n   * @return remove and return root of the queue i.e min/max element depending upon min-max queue\n   */\n  public DistancedItem<T> dequeue() {\n    return queue.poll();\n  }\n\n  /**\n   * Dequeue all the elements from queueu with ordering mantained\n   *\n   * @return remove all the elements in the order of the queue i.e min/max queue.\n   */\n  public List<DistancedItem<T>> dequeueAll() {\n    final List<DistancedItem<T>> list = new ArrayList<>(queue.size());\n    while (!queue.isEmpty()) {\n      list.add(queue.poll());\n    }\n\n    return list;\n  }\n\n  /**\n   * Convert queue to list\n   *\n   * @return list of elements of queue with distance and without any specific ordering\n   */\n  public List<DistancedItem<T>> toList() {\n    return new ArrayList<>(queue);\n  }\n\n  /**\n   * Convert queue to list\n   *\n   * @return list of elements of queue without any specific ordering\n   */\n  List<T> toListWithItem() {\n    List<T> list = new ArrayList<>(queue.size());\n    Iterator<DistancedItem<T>> itr = iterator();\n    while (itr.hasNext()) {\n      list.add(itr.next().getItem());\n    }\n    return list;\n  }\n\n  /**\n   * Enqueue an item into the queue\n   */\n  public void enqueue(T item) {\n    queue.add(new DistancedItem<>(item, distFn.distance(origin, item)));\n  }\n\n  /**\n   * Enqueue an item into the queue with its distance.\n   */\n  public void enqueue(T item, float distance) {\n    queue.add(new DistancedItem<>(item, distance));\n  }\n\n  /**\n   * Size\n   *\n   * @return size of the queue\n   */\n  public int size() {\n    return queue.size();\n  }\n\n  /**\n   * Is Min queue\n   *\n   * @return true if min queue else false\n   */\n  public boolean isMinQueue() {\n    return minQueue;\n  }\n\n  /**\n   * Returns origin (base element) of the queue\n   *\n   * @return origin of the queue\n   */\n  public U getOrigin() {\n    return origin;\n  }\n\n  /**\n   * Return a new queue with ordering reversed.\n   */\n  public DistancedItemQueue<U, T> reverse() {\n    final PriorityQueue<DistancedItem<T>> rqueue =\n        new PriorityQueue<>(queue.comparator().reversed());\n    if (queue.isEmpty()) {\n      return new DistancedItemQueue<>(origin, distFn, rqueue, !isMinQueue());\n    }\n\n    final Iterator<DistancedItem<T>> itr = iterator();\n    while (itr.hasNext()) {\n      rqueue.add(itr.next());\n    }\n\n    return new DistancedItemQueue<>(origin, distFn, rqueue, !isMinQueue());\n  }\n\n  @Override\n  public Iterator<DistancedItem<T>> iterator() {\n    return queue.iterator();\n  }\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/hnsw/HnswIndex.java",
    "content": "package com.twitter.ann.hnsw;\n\nimport java.io.IOException;\nimport java.nio.ByteBuffer;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Objects;\nimport java.util.Optional;\nimport java.util.Random;\nimport java.util.Set;\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.atomic.AtomicReference;\nimport java.util.concurrent.locks.Lock;\nimport java.util.concurrent.locks.ReadWriteLock;\nimport java.util.concurrent.locks.ReentrantLock;\nimport java.util.concurrent.locks.ReentrantReadWriteLock;\nimport java.util.function.Function;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableList;\n\nimport org.apache.thrift.TException;\n\nimport com.twitter.ann.common.IndexOutputFile;\nimport com.twitter.ann.common.thriftjava.HnswInternalIndexMetadata;\nimport com.twitter.bijection.Injection;\nimport com.twitter.logging.Logger;\nimport com.twitter.mediaservices.commons.codec.ArrayByteBufferCodec;\nimport com.twitter.search.common.file.AbstractFile;\n\n/**\n * Typed multithreaded HNSW implementation supporting creation/querying of approximate nearest neighbour\n * Paper: https://arxiv.org/pdf/1603.09320.pdf\n * Multithreading impl based on NMSLIB version : https://github.com/nmslib/hnsw/blob/master/hnswlib/hnswalg.h\n *\n * @param <T> The type of items inserted / searched in the HNSW index.\n * @param <Q> The type of KNN query.\n */\npublic class HnswIndex<T, Q> {\n  private static final Logger LOG = Logger.get(HnswIndex.class);\n  private static final String METADATA_FILE_NAME = \"hnsw_internal_metadata\";\n  private static final String GRAPH_FILE_NAME = \"hnsw_internal_graph\";\n  private static final int MAP_SIZE_FACTOR = 5;\n\n  private final DistanceFunction<T, T> distFnIndex;\n  private final DistanceFunction<Q, T> distFnQuery;\n  private final int efConstruction;\n  private final int maxM;\n  private final int maxM0;\n  private final double levelMultiplier;\n  private final AtomicReference<HnswMeta<T>> graphMeta = new AtomicReference<>();\n  private final Map<HnswNode<T>, ImmutableList<T>> graph;\n  // To take lock on vertex level\n  private final ConcurrentHashMap<T, ReadWriteLock> locks;\n  // To take lock on whole graph only if vertex addition is on layer above the current maxLevel\n  private final ReentrantLock globalLock;\n  private final Function<T, ReadWriteLock> lockProvider;\n\n  private final RandomProvider randomProvider;\n\n  // Probability of reevaluating connections of an element in the neighborhood during an update\n  // Can be used as a knob to adjust update_speed/search_speed tradeoff.\n  private final float updateNeighborProbability;\n\n  /**\n   * Creates instance of hnsw index.\n   *\n   * @param distFnIndex      Any distance metric/non metric that specifies similarity between two items for indexing.\n   * @param distFnQuery      Any distance metric/non metric that specifies similarity between item for which nearest neighbours queried for and already indexed item.\n   * @param efConstruction   Provide speed vs index quality tradeoff, higher the value better the quality and higher the time to create index.\n   *                         Valid range of efConstruction can be anywhere between 1 and tens of thousand. Typically, it should be set so that a search of M\n   *                         neighbors with ef=efConstruction should end in recall>0.95.\n   * @param maxM             Maximum connections per layer except 0th level.\n   *                         Optimal values between 5-48.\n   *                         Smaller M generally produces better result for lower recalls and/ or lower dimensional data,\n   *                         while bigger M is better for high recall and/ or high dimensional, data on the expense of more memory/disk usage\n   * @param expectedElements Approximate number of elements to be indexed\n   */\n  protected HnswIndex(\n      DistanceFunction<T, T> distFnIndex,\n      DistanceFunction<Q, T> distFnQuery,\n      int efConstruction,\n      int maxM,\n      int expectedElements,\n      RandomProvider randomProvider\n  ) {\n    this(distFnIndex,\n        distFnQuery,\n        efConstruction,\n        maxM,\n        expectedElements,\n        new HnswMeta<>(-1, Optional.empty()),\n        new ConcurrentHashMap<>(MAP_SIZE_FACTOR * expectedElements),\n        randomProvider\n    );\n  }\n\n  private HnswIndex(\n      DistanceFunction<T, T> distFnIndex,\n      DistanceFunction<Q, T> distFnQuery,\n      int efConstruction,\n      int maxM,\n      int expectedElements,\n      HnswMeta<T> graphMeta,\n      Map<HnswNode<T>, ImmutableList<T>> graph,\n      RandomProvider randomProvider\n  ) {\n    this.distFnIndex = distFnIndex;\n    this.distFnQuery = distFnQuery;\n    this.efConstruction = efConstruction;\n    this.maxM = maxM;\n    this.maxM0 = 2 * maxM;\n    this.levelMultiplier = 1.0 / Math.log(1.0 * maxM);\n    this.graphMeta.set(graphMeta);\n    this.graph = graph;\n    this.locks = new ConcurrentHashMap<>(MAP_SIZE_FACTOR * expectedElements);\n    this.globalLock = new ReentrantLock();\n    this.lockProvider = key -> new ReentrantReadWriteLock();\n    this.randomProvider = randomProvider;\n    this.updateNeighborProbability = 1.0f;\n  }\n\n  /**\n   * wireConnectionForAllLayers finds connections for a new element and creates bi-direction links.\n   * The method assumes using a reentrant lock to link list reads.\n   *\n   * @param entryPoint the global entry point\n   * @param item       the item for which the connections are found\n   * @param itemLevel  the level of the added item (maximum layer in which we wire the connections)\n   * @param maxLayer   the level of the entry point\n   */\n  private void wireConnectionForAllLayers(final T entryPoint, final T item, final int itemLevel,\n                                          final int maxLayer, final boolean isUpdate) {\n    T curObj = entryPoint;\n    if (itemLevel < maxLayer) {\n      curObj = bestEntryPointUntilLayer(curObj, item, maxLayer, itemLevel, distFnIndex);\n    }\n    for (int level = Math.min(itemLevel, maxLayer); level >= 0; level--) {\n      final DistancedItemQueue<T, T> candidates =\n          searchLayerForCandidates(item, curObj, efConstruction, level, distFnIndex, isUpdate);\n      curObj = mutuallyConnectNewElement(item, candidates, level, isUpdate);\n    }\n  }\n\n  /**\n   * Insert the item into HNSW index.\n   */\n  public void insert(final T item) throws IllegalDuplicateInsertException {\n    final Lock itemLock = locks.computeIfAbsent(item, lockProvider).writeLock();\n    itemLock.lock();\n    try {\n      final HnswMeta<T> metadata = graphMeta.get();\n      // If the graph already have the item, should not re-insert it again\n      // Need to check entry point in case we reinsert first item where is are no graph\n      // but only a entry point\n      if (graph.containsKey(HnswNode.from(0, item))\n          || (metadata.getEntryPoint().isPresent()\n          && Objects.equals(metadata.getEntryPoint().get(), item))) {\n        throw new IllegalDuplicateInsertException(\n            \"Duplicate insertion is not supported: \" + item);\n      }\n      final int curLevel = getRandomLevel();\n      Optional<T> entryPoint = metadata.getEntryPoint();\n      // The global lock prevents two threads from making changes to the entry point. This lock\n      // should get taken very infrequently. Something like log-base-levelMultiplier(num items)\n      // For a full explanation of locking see this document: http://go/hnsw-locking\n      int maxLevelCopy = metadata.getMaxLevel();\n      if (curLevel > maxLevelCopy) {\n        globalLock.lock();\n        // Re initialize the entryPoint and maxLevel in case these are changed by any other thread\n        // No need to check the condition again since,\n        // it is already checked at the end before updating entry point struct\n        // No need to unlock for optimization and keeping as is if condition fails since threads\n        // will not be entering this section a lot.\n        final HnswMeta<T> temp = graphMeta.get();\n        entryPoint = temp.getEntryPoint();\n        maxLevelCopy = temp.getMaxLevel();\n      }\n\n      if (entryPoint.isPresent()) {\n        wireConnectionForAllLayers(entryPoint.get(), item, curLevel, maxLevelCopy, false);\n      }\n\n      if (curLevel > maxLevelCopy) {\n        Preconditions.checkState(globalLock.isHeldByCurrentThread(),\n            \"Global lock not held before updating entry point\");\n        graphMeta.set(new HnswMeta<>(curLevel, Optional.of(item)));\n      }\n    } finally {\n      if (globalLock.isHeldByCurrentThread()) {\n        globalLock.unlock();\n      }\n      itemLock.unlock();\n    }\n  }\n\n  /**\n   * set connections of an element with synchronization\n   * The only other place that should have the lock for writing is during\n   * the element insertion\n   */\n  private void setConnectionList(final T item, int layer, List<T> connections) {\n    final Lock candidateLock = locks.computeIfAbsent(item, lockProvider).writeLock();\n    candidateLock.lock();\n    try {\n      graph.put(\n          HnswNode.from(layer, item),\n          ImmutableList.copyOf(connections)\n      );\n    } finally {\n      candidateLock.unlock();\n    }\n  }\n\n  /**\n   * Reinsert the item into HNSW index.\n   * This method updates the links of an element assuming\n   * the element's distance function is changed externally (e.g. by updating the features)\n   */\n\n  public void reInsert(final T item) {\n    final HnswMeta<T> metadata = graphMeta.get();\n\n    Optional<T> entryPoint = metadata.getEntryPoint();\n\n    Preconditions.checkState(entryPoint.isPresent(),\n        \"Update cannot be performed if entry point is not present\");\n\n    // This is a check for the single element case\n    if (entryPoint.get().equals(item) && graph.isEmpty()) {\n      return;\n    }\n\n    Preconditions.checkState(graph.containsKey(HnswNode.from(0, item)),\n        \"Graph does not contain the item to be updated at level 0\");\n\n    int curLevel = 0;\n\n    int maxLevelCopy = metadata.getMaxLevel();\n\n    for (int layer = maxLevelCopy; layer >= 0; layer--) {\n      if (graph.containsKey(HnswNode.from(layer, item))) {\n        curLevel = layer;\n        break;\n      }\n    }\n\n    // Updating the links of the elements from the 1-hop radius of the updated element\n\n    for (int layer = 0; layer <= curLevel; layer++) {\n\n      // Filling the element sets for candidates and updated elements\n      final HashSet<T> setCand = new HashSet<T>();\n      final HashSet<T> setNeigh = new HashSet<T>();\n      final List<T> listOneHop = getConnectionListForRead(item, layer);\n\n      if (listOneHop.isEmpty()) {\n        LOG.debug(\"No links for the updated element. Empty dataset?\");\n        continue;\n      }\n\n      setCand.add(item);\n\n      for (T elOneHop : listOneHop) {\n        setCand.add(elOneHop);\n        if (randomProvider.get().nextFloat() > updateNeighborProbability) {\n          continue;\n        }\n        setNeigh.add(elOneHop);\n        final List<T> listTwoHop = getConnectionListForRead(elOneHop, layer);\n\n        if (listTwoHop.isEmpty()) {\n          LOG.debug(\"No links for the updated element. Empty dataset?\");\n        }\n\n        for (T oneHopEl : listTwoHop) {\n          setCand.add(oneHopEl);\n        }\n      }\n      // No need to update the item itself, so remove it\n      setNeigh.remove(item);\n\n      // Updating the link lists of elements from setNeigh:\n      for (T neigh : setNeigh) {\n        final HashSet<T> setCopy = new HashSet<T>(setCand);\n        setCopy.remove(neigh);\n        int keepElementsNum = Math.min(efConstruction, setCopy.size());\n        final DistancedItemQueue<T, T> candidates = new DistancedItemQueue<>(\n            neigh,\n            ImmutableList.of(),\n            false,\n            distFnIndex\n        );\n        for (T cand : setCopy) {\n          final float distance = distFnIndex.distance(neigh, cand);\n          if (candidates.size() < keepElementsNum) {\n            candidates.enqueue(cand, distance);\n          } else {\n            if (distance < candidates.peek().getDistance()) {\n              candidates.dequeue();\n              candidates.enqueue(cand, distance);\n            }\n          }\n        }\n        final ImmutableList<T> neighbours = selectNearestNeighboursByHeuristic(\n            candidates,\n            layer == 0 ? maxM0 : maxM\n        );\n\n        final List<T> temp = getConnectionListForRead(neigh, layer);\n        if (temp.isEmpty()) {\n          LOG.debug(\"existing linkslist is empty. Corrupt index\");\n        }\n        if (neighbours.isEmpty()) {\n          LOG.debug(\"predicted linkslist is empty. Corrupt index\");\n        }\n        setConnectionList(neigh, layer, neighbours);\n\n      }\n\n\n    }\n    wireConnectionForAllLayers(metadata.getEntryPoint().get(), item, curLevel, maxLevelCopy, true);\n  }\n\n  /**\n   * This method can be used to get the graph statistics, specifically\n   * it prints the histogram of inbound connections for each element.\n   */\n  private String getStats() {\n    int histogramMaxBins = 50;\n    int[] histogram = new int[histogramMaxBins];\n    HashMap<T, Integer> mmap = new HashMap<T, Integer>();\n    for (HnswNode<T> key : graph.keySet()) {\n      if (key.level == 0) {\n        List<T> linkList = getConnectionListForRead(key.item, key.level);\n        for (T node : linkList) {\n          int a = mmap.computeIfAbsent(node, k -> 0);\n          mmap.put(node, a + 1);\n\n        }\n      }\n    }\n\n    for (T key : mmap.keySet()) {\n      int ind = mmap.get(key) < histogramMaxBins - 1 ? mmap.get(key) : histogramMaxBins - 1;\n      histogram[ind]++;\n    }\n    int minNonZeroIndex;\n    for (minNonZeroIndex = histogramMaxBins - 1; minNonZeroIndex >= 0; minNonZeroIndex--) {\n      if (histogram[minNonZeroIndex] > 0) {\n        break;\n      }\n    }\n\n    String output = \"\";\n    for (int i = 0; i <= minNonZeroIndex; i++) {\n      output += \"\" + i + \"\\t\" + histogram[i] / (0.01f * mmap.keySet().size()) + \"\\n\";\n    }\n\n    return output;\n  }\n\n  private int getRandomLevel() {\n    return (int) (-Math.log(randomProvider.get().nextDouble()) * levelMultiplier);\n  }\n\n  /**\n   * Note that to avoid deadlocks it is important that this method is called after all the searches\n   * of the graph have completed. If you take a lock on any items discovered in the graph after\n   * this, you may get stuck waiting on a thread that is waiting for item to be fully inserted.\n   * <p>\n   * Note: when using concurrent writers we can miss connections that we would otherwise get.\n   * This will reduce the recall.\n   * <p>\n   * For a full explanation of locking see this document: http://go/hnsw-locking\n   * The method returns the closest nearest neighbor (can be used as an enter point)\n   */\n  private T mutuallyConnectNewElement(\n      final T item,\n      final DistancedItemQueue<T, T> candidates, // Max queue\n      final int level,\n      final boolean isUpdate\n  ) {\n\n    // Using maxM here. Its implementation is ambiguous in HNSW paper,\n    // so using the way it is getting used in Hnsw lib.\n    final ImmutableList<T> neighbours = selectNearestNeighboursByHeuristic(candidates, maxM);\n    setConnectionList(item, level, neighbours);\n    final int M = level == 0 ? maxM0 : maxM;\n    for (T nn : neighbours) {\n      if (nn.equals(item)) {\n        continue;\n      }\n      final Lock curLock = locks.computeIfAbsent(nn, lockProvider).writeLock();\n      curLock.lock();\n      try {\n        final HnswNode<T> key = HnswNode.from(level, nn);\n        final ImmutableList<T> connections = graph.getOrDefault(key, ImmutableList.of());\n        final boolean isItemAlreadyPresent =\n            isUpdate && connections.indexOf(item) != -1 ? true : false;\n\n        // If `item` is already present in the neighboring connections,\n        // then no need to modify any connections or run the search heuristics.\n        if (isItemAlreadyPresent) {\n          continue;\n        }\n\n        final ImmutableList<T> updatedConnections;\n        if (connections.size() < M) {\n          final List<T> temp = new ArrayList<>(connections);\n          temp.add(item);\n          updatedConnections = ImmutableList.copyOf(temp.iterator());\n        } else {\n          // Max Queue\n          final DistancedItemQueue<T, T> queue = new DistancedItemQueue<>(\n              nn,\n              connections,\n              false,\n              distFnIndex\n          );\n          queue.enqueue(item);\n          updatedConnections = selectNearestNeighboursByHeuristic(queue, M);\n        }\n        if (updatedConnections.isEmpty()) {\n          LOG.debug(\"Internal error: predicted linkslist is empty\");\n        }\n\n        graph.put(key, updatedConnections);\n      } finally {\n        curLock.unlock();\n      }\n    }\n    return neighbours.get(0);\n  }\n\n  /*\n   *  bestEntryPointUntilLayer starts the graph search for item from the entry point\n   *  until the searches reaches the selectedLayer layer.\n   *  @return a point from selectedLayer layer, was the closest on the (selectedLayer+1) layer\n   */\n  private <K> T bestEntryPointUntilLayer(\n      final T entryPoint,\n      final K item,\n      int maxLayer,\n      int selectedLayer,\n      DistanceFunction<K, T> distFn\n  ) {\n    T curObj = entryPoint;\n    if (selectedLayer < maxLayer) {\n      float curDist = distFn.distance(item, curObj);\n      for (int level = maxLayer; level > selectedLayer; level--) {\n        boolean changed = true;\n        while (changed) {\n          changed = false;\n          final List<T> list = getConnectionListForRead(curObj, level);\n          for (T nn : list) {\n            final float tempDist = distFn.distance(item, nn);\n            if (tempDist < curDist) {\n              curDist = tempDist;\n              curObj = nn;\n              changed = true;\n            }\n          }\n        }\n      }\n    }\n\n    return curObj;\n  }\n\n\n  @VisibleForTesting\n  protected ImmutableList<T> selectNearestNeighboursByHeuristic(\n      final DistancedItemQueue<T, T> candidates, // Max queue\n      final int maxConnections\n  ) {\n    Preconditions.checkState(!candidates.isMinQueue(),\n        \"candidates in selectNearestNeighboursByHeuristic should be a max queue\");\n\n    final T baseElement = candidates.getOrigin();\n    if (candidates.size() <= maxConnections) {\n      List<T> list = candidates.toListWithItem();\n      list.remove(baseElement);\n      return ImmutableList.copyOf(list);\n    } else {\n      final List<T> resSet = new ArrayList<>(maxConnections);\n      // Min queue for closest elements first\n      final DistancedItemQueue<T, T> minQueue = candidates.reverse();\n      while (minQueue.nonEmpty()) {\n        if (resSet.size() >= maxConnections) {\n          break;\n        }\n        final DistancedItem<T> candidate = minQueue.dequeue();\n\n        // We do not want to creates loops:\n        // While heuristic is used only for creating the links\n        if (candidate.getItem().equals(baseElement)) {\n          continue;\n        }\n\n        boolean toInclude = true;\n        for (T e : resSet) {\n          // Do not include candidate if the distance from candidate to any of existing item in\n          // resSet is closer to the distance from the candidate to the item. By doing this, the\n          // connection of graph will be more diverse, and in case of highly clustered data set,\n          // connections will be made between clusters instead of all being in the same cluster.\n          final float dist = distFnIndex.distance(e, candidate.getItem());\n          if (dist < candidate.getDistance()) {\n            toInclude = false;\n            break;\n          }\n        }\n\n        if (toInclude) {\n          resSet.add(candidate.getItem());\n        }\n      }\n      return ImmutableList.copyOf(resSet);\n    }\n  }\n\n  /**\n   * Search the index for the neighbours.\n   *\n   * @param query           Query\n   * @param numOfNeighbours Number of neighbours to search for.\n   * @param ef              This param controls the accuracy of the search.\n   *                        Bigger the ef better the accuracy on the expense of latency.\n   *                        Keep it atleast number of neighbours to find.\n   * @return Neighbours\n   */\n  public List<DistancedItem<T>> searchKnn(final Q query, final int numOfNeighbours, final int ef) {\n    final HnswMeta<T> metadata = graphMeta.get();\n    if (metadata.getEntryPoint().isPresent()) {\n      T entryPoint = bestEntryPointUntilLayer(metadata.getEntryPoint().get(),\n          query, metadata.getMaxLevel(), 0, distFnQuery);\n      // Get the actual neighbours from 0th layer\n      final List<DistancedItem<T>> neighbours =\n          searchLayerForCandidates(query, entryPoint, Math.max(ef, numOfNeighbours),\n              0, distFnQuery, false).dequeueAll();\n      Collections.reverse(neighbours);\n      return neighbours.size() > numOfNeighbours\n          ? neighbours.subList(0, numOfNeighbours) : neighbours;\n    } else {\n      return Collections.emptyList();\n    }\n  }\n\n  // This method is currently not used\n  // It is needed for debugging purposes only\n  private void checkIntegrity(String message) {\n    final HnswMeta<T> metadata = graphMeta.get();\n    for (HnswNode<T> node : graph.keySet()) {\n      List<T> linkList = graph.get(node);\n\n      for (T el : linkList) {\n        if (el.equals(node.item)) {\n          LOG.debug(message);\n          throw new RuntimeException(\"integrity check failed\");\n        }\n      }\n    }\n  }\n\n  private <K> DistancedItemQueue<K, T> searchLayerForCandidates(\n      final K item,\n      final T entryPoint,\n      final int ef,\n      final int level,\n      final DistanceFunction<K, T> distFn,\n      boolean isUpdate\n  ) {\n    // Min queue\n    final DistancedItemQueue<K, T> cQueue = new DistancedItemQueue<>(\n        item,\n        Collections.singletonList(entryPoint),\n        true,\n        distFn\n    );\n    // Max Queue\n    final DistancedItemQueue<K, T> wQueue = cQueue.reverse();\n    final Set<T> visited = new HashSet<>();\n    float lowerBoundDistance = wQueue.peek().getDistance();\n    visited.add(entryPoint);\n\n    while (cQueue.nonEmpty()) {\n      final DistancedItem<T> candidate = cQueue.peek();\n      if (candidate.getDistance() > lowerBoundDistance) {\n        break;\n      }\n\n      cQueue.dequeue();\n      final List<T> list = getConnectionListForRead(candidate.getItem(), level);\n      for (T nn : list) {\n        if (!visited.contains(nn)) {\n          visited.add(nn);\n          final float distance = distFn.distance(item, nn);\n          if (wQueue.size() < ef || distance < wQueue.peek().getDistance()) {\n            cQueue.enqueue(nn, distance);\n\n            if (isUpdate && item.equals(nn)) {\n              continue;\n            }\n\n            wQueue.enqueue(nn, distance);\n            if (wQueue.size() > ef) {\n              wQueue.dequeue();\n            }\n\n            lowerBoundDistance = wQueue.peek().getDistance();\n          }\n        }\n      }\n    }\n\n    return wQueue;\n  }\n\n  /**\n   * Serialize hnsw index\n   */\n  public void toDirectory(IndexOutputFile indexOutputFile, Injection<T, byte[]> injection)\n    throws IOException, TException {\n  final int totalGraphEntries = HnswIndexIOUtil.saveHnswGraphEntries(\n      graph,\n      indexOutputFile.createFile(GRAPH_FILE_NAME).getOutputStream(),\n      injection);\n\n  HnswIndexIOUtil.saveMetadata(\n      graphMeta.get(),\n      efConstruction,\n      maxM,\n      totalGraphEntries,\n      injection,\n      indexOutputFile.createFile(METADATA_FILE_NAME).getOutputStream());\n}\n\n  /**\n   * Load hnsw index\n   */\n  public static <T, Q> HnswIndex<T, Q> loadHnswIndex(\n      DistanceFunction<T, T> distFnIndex,\n      DistanceFunction<Q, T> distFnQuery,\n      AbstractFile directory,\n      Injection<T, byte[]> injection,\n      RandomProvider randomProvider) throws IOException, TException {\n    final AbstractFile graphFile = directory.getChild(GRAPH_FILE_NAME);\n    final AbstractFile metadataFile = directory.getChild(METADATA_FILE_NAME);\n    final HnswInternalIndexMetadata metadata = HnswIndexIOUtil.loadMetadata(metadataFile);\n    final Map<HnswNode<T>, ImmutableList<T>> graph =\n        HnswIndexIOUtil.loadHnswGraph(graphFile, injection, metadata.numElements);\n    final ByteBuffer entryPointBB = metadata.entryPoint;\n    final HnswMeta<T> graphMeta = new HnswMeta<>(\n        metadata.maxLevel,\n        entryPointBB == null ? Optional.empty()\n            : Optional.of(injection.invert(ArrayByteBufferCodec.decode(entryPointBB)).get())\n    );\n    return new HnswIndex<>(\n        distFnIndex,\n        distFnQuery,\n        metadata.efConstruction,\n        metadata.maxM,\n        metadata.numElements,\n        graphMeta,\n        graph,\n        randomProvider\n    );\n  }\n\n  private List<T> getConnectionListForRead(T node, int level) {\n    final Lock curLock = locks.computeIfAbsent(node, lockProvider).readLock();\n    curLock.lock();\n    final List<T> list;\n    try {\n      list = graph\n          .getOrDefault(HnswNode.from(level, node), ImmutableList.of());\n    } finally {\n      curLock.unlock();\n    }\n\n    return list;\n  }\n\n  @VisibleForTesting\n  AtomicReference<HnswMeta<T>> getGraphMeta() {\n    return graphMeta;\n  }\n\n  @VisibleForTesting\n  Map<T, ReadWriteLock> getLocks() {\n    return locks;\n  }\n\n  @VisibleForTesting\n  Map<HnswNode<T>, ImmutableList<T>> getGraph() {\n    return graph;\n  }\n\n  public interface RandomProvider {\n    /**\n     * RandomProvider interface made public for scala 2.12 compat\n     */\n    Random get();\n  }\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/hnsw/HnswIndexIOUtil.java",
    "content": "package com.twitter.ann.hnsw;\n\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.io.OutputStream;\nimport java.nio.ByteBuffer;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.stream.Collectors;\n\nimport com.google.common.collect.ImmutableList;\n\nimport org.apache.thrift.TDeserializer;\nimport org.apache.thrift.TException;\nimport org.apache.thrift.TSerializer;\nimport org.apache.thrift.protocol.TBinaryProtocol;\nimport org.apache.thrift.protocol.TProtocol;\nimport org.apache.thrift.transport.TIOStreamTransport;\nimport org.apache.thrift.transport.TTransportException;\n\nimport com.twitter.ann.common.thriftjava.HnswGraphEntry;\nimport com.twitter.ann.common.thriftjava.HnswInternalIndexMetadata;\nimport com.twitter.bijection.Injection;\nimport com.twitter.mediaservices.commons.codec.ArrayByteBufferCodec;\nimport com.twitter.search.common.file.AbstractFile;\n\npublic final class HnswIndexIOUtil {\n  private HnswIndexIOUtil() {\n  }\n\n  /**\n   * Save thrift object in file\n   */\n  public static <T> void saveMetadata(\n      HnswMeta<T> graphMeta,\n      int efConstruction,\n      int maxM,\n      int numElements,\n      Injection<T, byte[]> injection,\n      OutputStream outputStream\n  ) throws IOException, TException {\n    final int maxLevel = graphMeta.getMaxLevel();\n    final HnswInternalIndexMetadata metadata = new HnswInternalIndexMetadata(\n        maxLevel,\n        efConstruction,\n        maxM,\n        numElements\n    );\n\n    if (graphMeta.getEntryPoint().isPresent()) {\n      metadata.setEntryPoint(injection.apply(graphMeta.getEntryPoint().get()));\n    }\n    final TSerializer serializer = new TSerializer(new TBinaryProtocol.Factory());\n    outputStream.write(serializer.serialize(metadata));\n    outputStream.close();\n  }\n\n  /**\n   * Load Hnsw index metadata\n   */\n  public static HnswInternalIndexMetadata loadMetadata(AbstractFile file)\n      throws IOException, TException {\n    final HnswInternalIndexMetadata obj = new HnswInternalIndexMetadata();\n    final TDeserializer deserializer = new TDeserializer(new TBinaryProtocol.Factory());\n    deserializer.deserialize(obj, file.getByteSource().read());\n    return obj;\n  }\n\n  /**\n   * Load Hnsw graph entries from file\n   */\n  public static <T> Map<HnswNode<T>, ImmutableList<T>> loadHnswGraph(\n      AbstractFile file,\n      Injection<T, byte[]> injection,\n      int numElements\n  ) throws IOException, TException {\n    final InputStream stream = file.getByteSource().openBufferedStream();\n    final TProtocol protocol = new TBinaryProtocol(new TIOStreamTransport(stream));\n    final Map<HnswNode<T>, ImmutableList<T>> graph =\n        new HashMap<>(numElements);\n    while (true) {\n      try {\n        final HnswGraphEntry entry = new HnswGraphEntry();\n        entry.read(protocol);\n        final HnswNode<T> node = HnswNode.from(entry.level,\n            injection.invert(ArrayByteBufferCodec.decode(entry.key)).get());\n        final List<T> list = entry.getNeighbours().stream()\n            .map(bb -> injection.invert(ArrayByteBufferCodec.decode(bb)).get())\n            .collect(Collectors.toList());\n        graph.put(node, ImmutableList.copyOf(list.iterator()));\n      } catch (TException e) {\n        if (e instanceof TTransportException\n            && TTransportException.class.cast(e).getType() == TTransportException.END_OF_FILE) {\n          stream.close();\n          break;\n        }\n        stream.close();\n        throw e;\n      }\n    }\n\n    return graph;\n  }\n\n  /**\n   * Save hnsw graph in file\n   *\n   * @return number of keys in the graph\n   */\n  public static <T> int saveHnswGraphEntries(\n      Map<HnswNode<T>, ImmutableList<T>> graph,\n      OutputStream outputStream,\n      Injection<T, byte[]> injection\n  ) throws IOException, TException {\n    final TProtocol protocol = new TBinaryProtocol(new TIOStreamTransport(outputStream));\n    final Set<HnswNode<T>> nodes = graph.keySet();\n    for (HnswNode<T> node : nodes) {\n      final HnswGraphEntry entry = new HnswGraphEntry();\n      entry.setLevel(node.level);\n      entry.setKey(injection.apply(node.item));\n      final List<ByteBuffer> nn = graph.getOrDefault(node, ImmutableList.of()).stream()\n          .map(t -> ByteBuffer.wrap(injection.apply(t)))\n          .collect(Collectors.toList());\n      entry.setNeighbours(nn);\n      entry.write(protocol);\n    }\n\n    outputStream.close();\n    return nodes.size();\n  }\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/hnsw/HnswMeta.java",
    "content": "package com.twitter.ann.hnsw;\n\nimport java.util.Objects;\nimport java.util.Optional;\n\nclass HnswMeta<T> {\n  private final int maxLevel;\n  private final Optional<T> entryPoint;\n\n  HnswMeta(int maxLevel, Optional<T> entryPoint) {\n    this.maxLevel = maxLevel;\n    this.entryPoint = entryPoint;\n  }\n\n  public int getMaxLevel() {\n    return maxLevel;\n  }\n\n  public Optional<T> getEntryPoint() {\n    return entryPoint;\n  }\n\n  @Override\n  public boolean equals(Object o) {\n    if (this == o) {\n      return true;\n    }\n    if (o == null || getClass() != o.getClass()) {\n      return false;\n    }\n    HnswMeta<?> hnswMeta = (HnswMeta<?>) o;\n    return maxLevel == hnswMeta.maxLevel\n        && Objects.equals(entryPoint, hnswMeta.entryPoint);\n  }\n\n  @Override\n  public int hashCode() {\n    return Objects.hash(maxLevel, entryPoint);\n  }\n\n  @Override\n  public String toString() {\n    return \"HnswMeta{maxLevel=\" + maxLevel + \", entryPoint=\" + entryPoint + '}';\n  }\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/hnsw/HnswNode.java",
    "content": "package com.twitter.ann.hnsw;\n\nimport org.apache.commons.lang.builder.EqualsBuilder;\nimport org.apache.commons.lang.builder.HashCodeBuilder;\n\npublic class HnswNode<T> {\n  public final int level;\n  public final T item;\n\n  public HnswNode(int level, T item) {\n    this.level = level;\n    this.item = item;\n  }\n\n  /**\n   * Create a hnsw node.\n   */\n  public static <T> HnswNode<T> from(int level, T item) {\n    return new HnswNode<>(level, item);\n  }\n\n  @Override\n  public boolean equals(Object o) {\n    if (o == this) {\n      return true;\n    }\n    if (!(o instanceof HnswNode)) {\n      return false;\n    }\n\n    HnswNode<?> that = (HnswNode<?>) o;\n    return new EqualsBuilder()\n        .append(this.item, that.item)\n        .append(this.level, that.level)\n        .isEquals();\n  }\n\n  @Override\n  public int hashCode() {\n    return new HashCodeBuilder()\n        .append(item)\n        .append(level)\n        .toHashCode();\n  }\n}\n"
  },
  {
    "path": "ann/src/main/java/com/twitter/ann/hnsw/IllegalDuplicateInsertException.java",
    "content": "package com.twitter.ann.hnsw;\n\npublic class IllegalDuplicateInsertException extends Exception {\n  public IllegalDuplicateInsertException(String message) {\n    super(message);\n  }\n}\n"
  },
  {
    "path": "ann/src/main/python/dataflow/BUILD.bazel",
    "content": "resources(\n    name = \"sql\",\n    sources = [\"bq.sql\"],\n)\n\npython3_library(\n    name = \"faiss_indexing\",\n    sources = [\"**/*.py\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":sql\",\n        \"3rdparty/python/apache-beam:default\",\n        \"3rdparty/python/faiss-gpu:default\",\n        \"3rdparty/python/gcsfs:default\",\n        \"3rdparty/python/google-cloud-bigquery:default\",\n        \"3rdparty/python/google-cloud-storage\",\n        \"3rdparty/python/numpy:default\",\n        \"3rdparty/python/pandas:default\",\n        \"3rdparty/python/pandas-gbq:default\",\n        \"3rdparty/python/pyarrow:default\",\n        \"src/python/twitter/ml/common/apache_beam\",\n    ],\n)\n\npython37_binary(\n    name = \"faiss_indexing_bin\",\n    sources = [\"faiss_index_bq_dataset.py\"],\n    platforms = [\n        \"current\",\n        \"linux_x86_64\",\n    ],\n    tags = [\"no-mypy\"],\n    zip_safe = False,\n    dependencies = [\n        \":faiss_indexing\",\n        \"3rdparty/python/_closures/ann/src/main/python/dataflow:faiss_indexing_bin\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/python/dataflow/bq.sql",
    "content": "WITH maxts as (SELECT as value MAX(ts) as ts FROM `twttr-recos-ml-prod.ssedhain.twhin_tweet_avg_embedding`)\nSELECT entityId, embedding\nFROM `twttr-recos-ml-prod.ssedhain.twhin_tweet_avg_embedding`\nWHERE ts >= (select max(maxts) from maxts)\nAND DATE(TIMESTAMP_MILLIS(createdAt)) <= (select max(maxts) from maxts)\nAND DATE(TIMESTAMP_MILLIS(createdAt)) >= DATE_SUB((select max(maxts) from maxts), INTERVAL 1 DAY)"
  },
  {
    "path": "ann/src/main/python/dataflow/faiss_index_bq_dataset.py",
    "content": "import argparse\nimport logging\nimport os\nimport pkgutil\nimport sys\nfrom urllib.parse import urlsplit\n\nimport apache_beam as beam\nfrom apache_beam.options.pipeline_options import PipelineOptions\nimport faiss\n\n\ndef parse_d6w_config(argv=None):\n  \"\"\"Parse d6w config.\n  :param argv: d6w config\n  :return: dictionary containing d6w config\n  \"\"\"\n\n  parser = argparse.ArgumentParser(\n    description=\"See https://docbird.twitter.biz/d6w/model.html for any parameters inherited from d6w job config\"\n  )\n  parser.add_argument(\"--job_name\", dest=\"job_name\", required=True, help=\"d6w attribute\")\n  parser.add_argument(\"--project\", dest=\"project\", required=True, help=\"d6w attribute\")\n  parser.add_argument(\n    \"--staging_location\", dest=\"staging_location\", required=True, help=\"d6w attribute\"\n  )\n  parser.add_argument(\"--temp_location\", dest=\"temp_location\", required=True, help=\"d6w attribute\")\n  parser.add_argument(\n    \"--output_location\",\n    dest=\"output_location\",\n    required=True,\n    help=\"GCS bucket and path where resulting artifacts are uploaded\",\n  )\n  parser.add_argument(\n    \"--service_account_email\", dest=\"service_account_email\", required=True, help=\"d6w attribute\"\n  )\n  parser.add_argument(\n    \"--factory_string\",\n    dest=\"factory_string\",\n    required=False,\n    help=\"FAISS factory string describing index to build. See https://github.com/facebookresearch/faiss/wiki/The-index-factory\",\n  )\n  parser.add_argument(\n    \"--metric\",\n    dest=\"metric\",\n    required=True,\n    help=\"Metric used to compute distance between embeddings. Valid values are 'l2', 'ip', 'l1', 'linf'\",\n  )\n  parser.add_argument(\n    \"--use_gpu\",\n    dest=\"gpu\",\n    required=True,\n    help=\"--use_gpu=yes if you want to use GPU during index building\",\n  )\n\n  known_args, unknown_args = parser.parse_known_args(argv)\n  d6w_config = vars(known_args)\n  d6w_config[\"gpu\"] = d6w_config[\"gpu\"].lower() == \"yes\"\n  d6w_config[\"metric\"] = parse_metric(d6w_config)\n\n  \"\"\"\n  WARNING: Currently, d6w (a Twitter tool used to deploy Dataflow jobs to GCP) and\n  PipelineOptions.for_dataflow_runner (a helper method in twitter.ml.common.apache_beam) do not\n  play nicely together. The helper method will overwrite some of the config specified in the d6w\n  file using the defaults in https://sourcegraph.twitter.biz/git.twitter.biz/source/-/blob/src/python/twitter/ml/common/apache_beam/__init__.py?L24.'\n  However, the d6w output message will still report that the config specified in the d6w file was used.\n  \"\"\"\n  logging.warning(\n    f\"The following d6w config parameters will be overwritten by the defaults in \"\n    f\"https://sourcegraph.twitter.biz/git.twitter.biz/source/-/blob/src/python/twitter/ml/common/apache_beam/__init__.py?L24\\n\"\n    f\"{str(unknown_args)}\"\n  )\n  return d6w_config\n\n\ndef get_bq_query():\n  \"\"\"\n  Query is expected to return rows with unique entityId\n  \"\"\"\n  return pkgutil.get_data(__name__, \"bq.sql\").decode(\"utf-8\")\n\n\ndef parse_metric(config):\n  metric_str = config[\"metric\"].lower()\n  if metric_str == \"l2\":\n    return faiss.METRIC_L2\n  elif metric_str == \"ip\":\n    return faiss.METRIC_INNER_PRODUCT\n  elif metric_str == \"l1\":\n    return faiss.METRIC_L1\n  elif metric_str == \"linf\":\n    return faiss.METRIC_Linf\n  else:\n    raise Exception(f\"Unknown metric: {metric_str}\")\n\n\ndef run_pipeline(argv=[]):\n  config = parse_d6w_config(argv)\n  argv_with_extras = argv\n  if config[\"gpu\"]:\n    argv_with_extras.extend([\"--experiments\", \"use_runner_v2\"])\n    argv_with_extras.extend(\n      [\"--experiments\", \"worker_accelerator=type:nvidia-tesla-t4;count:1;install-nvidia-driver\"]\n    )\n    argv_with_extras.extend(\n      [\n        \"--worker_harness_container_image\",\n        \"gcr.io/twttr-recos-ml-prod/dataflow-gpu/beam2_39_0_py3_7\",\n      ]\n    )\n\n  options = PipelineOptions(argv_with_extras)\n  output_bucket_name = urlsplit(config[\"output_location\"]).netloc\n\n  with beam.Pipeline(options=options) as p:\n    input_data = p | \"Read from BigQuery\" >> beam.io.ReadFromBigQuery(\n      method=beam.io.ReadFromBigQuery.Method.DIRECT_READ,\n      query=get_bq_query(),\n      use_standard_sql=True,\n    )\n\n    index_built = input_data | \"Build and upload index\" >> beam.CombineGlobally(\n      MergeAndBuildIndex(\n        output_bucket_name,\n        config[\"output_location\"],\n        config[\"factory_string\"],\n        config[\"metric\"],\n        config[\"gpu\"],\n      )\n    )\n\n    # Make linter happy\n    index_built\n\n\nclass MergeAndBuildIndex(beam.CombineFn):\n  def __init__(self, bucket_name, gcs_output_path, factory_string, metric, gpu):\n    self.bucket_name = bucket_name\n    self.gcs_output_path = gcs_output_path\n    self.factory_string = factory_string\n    self.metric = metric\n    self.gpu = gpu\n\n  def create_accumulator(self):\n    return []\n\n  def add_input(self, accumulator, element):\n    accumulator.append(element)\n    return accumulator\n\n  def merge_accumulators(self, accumulators):\n    merged = []\n    for accum in accumulators:\n      merged.extend(accum)\n    return merged\n\n  def extract_output(self, rows):\n    # Reimports are needed on workers\n    import glob\n    import subprocess\n\n    import faiss\n    from google.cloud import storage\n    import numpy as np\n\n    client = storage.Client()\n    bucket = client.get_bucket(self.bucket_name)\n\n    logging.info(\"Building FAISS index\")\n    logging.info(f\"There are {len(rows)} rows\")\n\n    ids = np.array([x[\"entityId\"] for x in rows]).astype(\"long\")\n    embeds = np.array([x[\"embedding\"] for x in rows]).astype(\"float32\")\n    dimensions = len(embeds[0])\n    N = ids.shape[0]\n    logging.info(f\"There are {dimensions} dimensions\")\n\n    if self.factory_string is None:\n      M = 48\n\n      divideable_dimensions = (dimensions // M) * M\n      if divideable_dimensions != dimensions:\n        opq_prefix = f\"OPQ{M}_{divideable_dimensions}\"\n      else:\n        opq_prefix = f\"OPQ{M}\"\n\n      clusters = N // 20\n      self.factory_string = f\"{opq_prefix},IVF{clusters},PQ{M}\"\n\n    logging.info(f\"Factory string is {self.factory_string}, metric={self.metric}\")\n\n    if self.gpu:\n      logging.info(\"Using GPU\")\n\n      res = faiss.StandardGpuResources()\n      cpu_index = faiss.index_factory(dimensions, self.factory_string, self.metric)\n      cpu_index = faiss.IndexIDMap(cpu_index)\n      gpu_index = faiss.index_cpu_to_gpu(res, 0, cpu_index)\n      gpu_index.train(embeds)\n      gpu_index.add_with_ids(embeds, ids)\n      cpu_index = faiss.index_gpu_to_cpu(gpu_index)\n    else:\n      logging.info(\"Using CPU\")\n\n      cpu_index = faiss.index_factory(dimensions, self.factory_string, self.metric)\n      cpu_index = faiss.IndexIDMap(cpu_index)\n      cpu_index.train(embeds)\n      cpu_index.add_with_ids(embeds, ids)\n\n    logging.info(\"Built faiss index\")\n\n    local_path = \"/indices\"\n    logging.info(f\"Writing indices to local {local_path}\")\n    subprocess.run(f\"mkdir -p {local_path}\".strip().split())\n    local_index_path = os.path.join(local_path, \"result.index\")\n\n    faiss.write_index(cpu_index, local_index_path)\n    logging.info(f\"Done writing indices to local {local_path}\")\n\n    logging.info(f\"Uploading to GCS with path {self.gcs_output_path}\")\n    assert os.path.isdir(local_path)\n    for local_file in glob.glob(local_path + \"/*\"):\n      remote_path = os.path.join(\n        self.gcs_output_path.split(\"/\")[-1], local_file[1 + len(local_path) :]\n      )\n      blob = bucket.blob(remote_path)\n      blob.upload_from_filename(local_file)\n\n\nif __name__ == \"__main__\":\n  logging.getLogger().setLevel(logging.INFO)\n  run_pipeline(sys.argv)\n"
  },
  {
    "path": "ann/src/main/python/dataflow/worker_harness/Dockerfile",
    "content": "FROM --platform=linux/amd64 nvidia/cuda:11.2.2-cudnn8-runtime-ubuntu20.04\n\nRUN \\\n  # Add Deadsnakes repository that has a variety of Python packages for Ubuntu.\n  # See: https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa\n  apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F23C5A6CF475977595C89F51BA6932366A755776 \\\n  && echo \"deb http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal main\" >> /etc/apt/sources.list.d/custom.list \\\n  && echo \"deb-src http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal main\" >> /etc/apt/sources.list.d/custom.list \\\n  && apt-get update \\\n  && apt-get install -y curl \\\n  python3.7 \\\n  # With python3.8 package, distutils need to be installed separately.\n  python3.7-distutils \\\n  python3-dev \\\n  python3.7-dev \\\n  libpython3.7-dev \\\n  python3-apt \\\n  gcc \\\n  g++ \\\n  && rm -rf /var/lib/apt/lists/*\nRUN update-alternatives --install /usr/bin/python python /usr/bin/python3.7 10\nRUN rm -f /usr/bin/python3 && ln -s /usr/bin/python3.7 /usr/bin/python3\nRUN \\\n  curl https://bootstrap.pypa.io/get-pip.py | python \\\n  && pip3 install pip==22.0.3 \\\n  && python3 -m pip install --no-cache-dir apache-beam[gcp]==2.39.0\n# Verify that there are no conflicting dependencies.\nRUN pip3 check\n\n# Copy the Apache Beam worker dependencies from the Beam Python 3.7 SDK image.\nCOPY --from=apache/beam_python3.7_sdk:2.39.0 /opt/apache/beam /opt/apache/beam\n\n# Set the entrypoint to Apache Beam SDK worker launcher.\nENTRYPOINT [ \"/opt/apache/beam/boot\" ]"
  },
  {
    "path": "ann/src/main/python/dataflow/worker_harness/cloudbuild.yml",
    "content": "steps:\n- name: 'gcr.io/cloud-builders/docker'\n  args: ['build', '-t', 'gcr.io/twttr-recos-ml-prod/dataflow-gpu/beam2_39_0_py3_7', '.']\n- name: 'gcr.io/cloud-builders/docker'\n  args: ['push', 'gcr.io/twttr-recos-ml-prod/dataflow-gpu/beam2_39_0_py3_7']\nimages: ['gcr.io/twttr-recos-ml-prod/dataflow-gpu/beam2_39_0_py3_7']"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/annoy/AnnoyCommon.scala",
    "content": "package com.twitter.ann.annoy\n\nimport com.twitter.ann.common.RuntimeParams\nimport com.twitter.ann.common.thriftscala.AnnoyIndexMetadata\nimport com.twitter.bijection.Injection\nimport com.twitter.mediaservices.commons.codec.ThriftByteBufferCodec\nimport com.twitter.ann.common.thriftscala.{AnnoyRuntimeParam, RuntimeParams => ServiceRuntimeParams}\nimport scala.util.{Failure, Success, Try}\n\nobject AnnoyCommon {\n  private[annoy] lazy val MetadataCodec = new ThriftByteBufferCodec(AnnoyIndexMetadata)\n  private[annoy] val IndexFileName = \"annoy_index\"\n  private[annoy] val MetaDataFileName = \"annoy_index_metadata\"\n  private[annoy] val IndexIdMappingFileName = \"annoy_index_id_mapping\"\n\n  val RuntimeParamsInjection: Injection[AnnoyRuntimeParams, ServiceRuntimeParams] =\n    new Injection[AnnoyRuntimeParams, ServiceRuntimeParams] {\n      override def apply(scalaParams: AnnoyRuntimeParams): ServiceRuntimeParams = {\n        ServiceRuntimeParams.AnnoyParam(\n          AnnoyRuntimeParam(\n            scalaParams.nodesToExplore\n          )\n        )\n      }\n\n      override def invert(thriftParams: ServiceRuntimeParams): Try[AnnoyRuntimeParams] =\n        thriftParams match {\n          case ServiceRuntimeParams.AnnoyParam(annoyParam) =>\n            Success(\n              AnnoyRuntimeParams(annoyParam.numOfNodesToExplore)\n            )\n          case p => Failure(new IllegalArgumentException(s\"Expected AnnoyRuntimeParams got $p\"))\n        }\n    }\n}\n\ncase class AnnoyRuntimeParams(\n  /* Number of vectors to evaluate while searching. A larger value will give more accurate results, but will take longer time to return.\n   * Default value would be numberOfTrees*numberOfNeigboursRequested\n   */\n  nodesToExplore: Option[Int])\n    extends RuntimeParams {\n  override def toString: String = s\"AnnoyRuntimeParams( nodesToExplore = $nodesToExplore)\"\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/annoy/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/spotify:annoy-java\",\n        \"3rdparty/jvm/com/spotify:annoy-snapshot\",\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/file_store\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"mediaservices/commons\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/scala/com/twitter/ml/api/embedding\",\n    ],\n    exports = [\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"src/java/com/twitter/common_internal/hadoop\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/scala/com/twitter/ml/api/embedding\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/annoy/RawAnnoyIndexBuilder.scala",
    "content": "package com.twitter.ann.annoy\n\nimport com.spotify.annoy.jni.base.{Annoy => AnnoyLib}\nimport com.twitter.ann.annoy.AnnoyCommon.IndexFileName\nimport com.twitter.ann.annoy.AnnoyCommon.MetaDataFileName\nimport com.twitter.ann.annoy.AnnoyCommon.MetadataCodec\nimport com.twitter.ann.common.EmbeddingType._\nimport com.twitter.ann.common._\nimport com.twitter.ann.common.thriftscala.AnnoyIndexMetadata\nimport com.twitter.concurrent.AsyncSemaphore\nimport com.twitter.mediaservices.commons.codec.ArrayByteBufferCodec\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.search.common.file.LocalFile\nimport com.twitter.util.Future\nimport com.twitter.util.FuturePool\nimport java.io.File\nimport java.nio.file.Files\nimport org.apache.beam.sdk.io.fs.ResourceId\nimport scala.collection.JavaConverters._\n\nprivate[annoy] object RawAnnoyIndexBuilder {\n  private[annoy] def apply[D <: Distance[D]](\n    dimension: Int,\n    numOfTrees: Int,\n    metric: Metric[D],\n    futurePool: FuturePool\n  ): RawAppendable[AnnoyRuntimeParams, D] with Serialization = {\n    val indexBuilder = AnnoyLib.newIndex(dimension, annoyMetric(metric))\n    new RawAnnoyIndexBuilder(dimension, numOfTrees, metric, indexBuilder, futurePool)\n  }\n\n  private[this] def annoyMetric(metric: Metric[_]): AnnoyLib.Metric = {\n    metric match {\n      case L2 => AnnoyLib.Metric.EUCLIDEAN\n      case Cosine => AnnoyLib.Metric.ANGULAR\n      case _ => throw new RuntimeException(\"Not supported: \" + metric)\n    }\n  }\n}\n\nprivate[this] class RawAnnoyIndexBuilder[D <: Distance[D]](\n  dimension: Int,\n  numOfTrees: Int,\n  metric: Metric[D],\n  indexBuilder: AnnoyLib.Builder,\n  futurePool: FuturePool)\n    extends RawAppendable[AnnoyRuntimeParams, D]\n    with Serialization {\n  private[this] var counter = 0\n  // Note: Only one thread can access the underlying index, multithreaded index building not supported\n  private[this] val semaphore = new AsyncSemaphore(1)\n\n  override def append(embedding: EmbeddingVector): Future[Long] =\n    semaphore.acquireAndRun({\n      counter += 1\n      indexBuilder.addItem(\n        counter,\n        embedding.toArray\n          .map(float => float2Float(float))\n          .toList\n          .asJava\n      )\n\n      Future.value(counter)\n    })\n\n  override def toQueryable: Queryable[Long, AnnoyRuntimeParams, D] = {\n    val tempDirParent = Files.createTempDirectory(\"raw_annoy_index\").toFile\n    tempDirParent.deleteOnExit\n    val tempDir = new LocalFile(tempDirParent)\n    this.toDirectory(tempDir)\n    RawAnnoyQueryIndex(\n      dimension,\n      metric,\n      futurePool,\n      tempDir\n    )\n  }\n\n  override def toDirectory(directory: ResourceId): Unit = {\n    toDirectory(new IndexOutputFile(directory))\n  }\n\n  /**\n   * Serialize the annoy index in a directory.\n   * @param directory: Directory to save to.\n   */\n  override def toDirectory(directory: AbstractFile): Unit = {\n    toDirectory(new IndexOutputFile(directory))\n  }\n\n  private def toDirectory(directory: IndexOutputFile): Unit = {\n    val indexFile = directory.createFile(IndexFileName)\n    saveIndex(indexFile)\n\n    val metaDataFile = directory.createFile(MetaDataFileName)\n    saveMetadata(metaDataFile)\n  }\n\n  private[this] def saveIndex(indexFile: IndexOutputFile): Unit = {\n    val index = indexBuilder\n      .build(numOfTrees)\n    val temp = new LocalFile(File.createTempFile(IndexFileName, null))\n    index.save(temp.getPath)\n    indexFile.copyFrom(temp.getByteSource.openStream())\n    temp.delete()\n  }\n\n  private[this] def saveMetadata(metadataFile: IndexOutputFile): Unit = {\n    val numberOfVectorsIndexed = counter\n    val metadata = AnnoyIndexMetadata(\n      dimension,\n      Metric.toThrift(metric),\n      numOfTrees,\n      numberOfVectorsIndexed\n    )\n    val bytes = ArrayByteBufferCodec.decode(MetadataCodec.encode(metadata))\n    val temp = new LocalFile(File.createTempFile(MetaDataFileName, null))\n    temp.getByteSink.write(bytes)\n    metadataFile.copyFrom(temp.getByteSource.openStream())\n    temp.delete()\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/annoy/RawAnnoyQueryIndex.scala",
    "content": "package com.twitter.ann.annoy\n\nimport com.spotify.annoy.{ANNIndex, IndexType}\nimport com.twitter.ann.annoy.AnnoyCommon._\nimport com.twitter.ann.common._\nimport com.twitter.ann.common.EmbeddingType._\nimport com.twitter.mediaservices.commons.codec.ArrayByteBufferCodec\nimport com.twitter.search.common.file.{AbstractFile, LocalFile}\nimport com.twitter.util.{Future, FuturePool}\nimport java.io.File\nimport scala.collection.JavaConverters._\n\nprivate[annoy] object RawAnnoyQueryIndex {\n  private[annoy] def apply[D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    futurePool: FuturePool,\n    directory: AbstractFile\n  ): Queryable[Long, AnnoyRuntimeParams, D] = {\n    val metadataFile = directory.getChild(MetaDataFileName)\n    val indexFile = directory.getChild(IndexFileName)\n    val metadata = MetadataCodec.decode(\n      ArrayByteBufferCodec.encode(metadataFile.getByteSource.read())\n    )\n\n    val existingDimension = metadata.dimension\n    assert(\n      existingDimension == dimension,\n      s\"Dimensions do not match. requested: $dimension existing: $existingDimension\"\n    )\n\n    val existingMetric = Metric.fromThrift(metadata.distanceMetric)\n    assert(\n      existingMetric == metric,\n      s\"DistanceMetric do not match. requested: $metric existing: $existingMetric\"\n    )\n\n    val index = loadIndex(indexFile, dimension, annoyMetric(metric))\n    new RawAnnoyQueryIndex[D](\n      dimension,\n      metric,\n      metadata.numOfTrees,\n      index,\n      futurePool\n    )\n  }\n\n  private[this] def annoyMetric(metric: Metric[_]): IndexType = {\n    metric match {\n      case L2 => IndexType.EUCLIDEAN\n      case Cosine => IndexType.ANGULAR\n      case _ => throw new RuntimeException(\"Not supported: \" + metric)\n    }\n  }\n\n  private[this] def loadIndex(\n    indexFile: AbstractFile,\n    dimension: Int,\n    indexType: IndexType\n  ): ANNIndex = {\n    var localIndexFile = indexFile\n\n    // If not a local file copy to local, so that it can be memory mapped.\n    if (!indexFile.isInstanceOf[LocalFile]) {\n      val tempFile = File.createTempFile(IndexFileName, null)\n      tempFile.deleteOnExit()\n\n      val temp = new LocalFile(tempFile)\n      indexFile.copyTo(temp)\n      localIndexFile = temp\n    }\n\n    new ANNIndex(\n      dimension,\n      localIndexFile.getPath(),\n      indexType\n    )\n  }\n}\n\nprivate[this] class RawAnnoyQueryIndex[D <: Distance[D]](\n  dimension: Int,\n  metric: Metric[D],\n  numOfTrees: Int,\n  index: ANNIndex,\n  futurePool: FuturePool)\n    extends Queryable[Long, AnnoyRuntimeParams, D]\n    with AutoCloseable {\n  override def query(\n    embedding: EmbeddingVector,\n    numOfNeighbours: Int,\n    runtimeParams: AnnoyRuntimeParams\n  ): Future[List[Long]] = {\n    queryWithDistance(embedding, numOfNeighbours, runtimeParams)\n      .map(_.map(_.neighbor))\n  }\n\n  override def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbours: Int,\n    runtimeParams: AnnoyRuntimeParams\n  ): Future[List[NeighborWithDistance[Long, D]]] = {\n    futurePool {\n      val queryVector = embedding.toArray\n      val neigboursToRequest = neighboursToRequest(numOfNeighbours, runtimeParams)\n      val neigbours = index\n        .getNearestWithDistance(queryVector, neigboursToRequest)\n        .asScala\n        .take(numOfNeighbours)\n        .map { nn =>\n          val id = nn.getFirst.toLong\n          val distance = metric.fromAbsoluteDistance(nn.getSecond)\n          NeighborWithDistance(id, distance)\n        }\n        .toList\n\n      neigbours\n    }\n  }\n\n  // Annoy java lib do not expose param for numOfNodesToExplore.\n  // Default number is numOfTrees*numOfNeigbours.\n  // Simple hack is to artificially increase the numOfNeighbours to be requested and then just cap it before returning.\n  private[this] def neighboursToRequest(\n    numOfNeighbours: Int,\n    annoyParams: AnnoyRuntimeParams\n  ): Int = {\n    annoyParams.nodesToExplore match {\n      case Some(nodesToExplore) => {\n        val neigboursToRequest = nodesToExplore / numOfTrees\n        if (neigboursToRequest < numOfNeighbours)\n          numOfNeighbours\n        else\n          neigboursToRequest\n      }\n      case _ => numOfNeighbours\n    }\n  }\n\n  // To close the memory map based file resource.\n  override def close(): Unit = index.close()\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/annoy/TypedAnnoyIndex.scala",
    "content": "package com.twitter.ann.annoy\n\nimport com.twitter.ann.common._\nimport com.twitter.bijection.Injection\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.util.FuturePool\n\n// Class to provide Annoy based ann index.\nobject TypedAnnoyIndex {\n\n  /**\n   * Create Annoy based typed index builder that serializes index to a directory (HDFS/Local file system).\n   * It cannot be used in scalding as it leverage C/C++ jni bindings, whose build conflicts with version of some libs installed on hadoop.\n   * You can use it on aurora or with IndexBuilding job which triggers scalding job but then streams data to aurora machine for building index.\n   * @param dimension dimension of embedding\n   * @param numOfTrees builds a forest of numOfTrees trees.\n   *                   More trees gives higher precision when querying at the cost of increased memory and disk storage requirement at the build time.\n   *                   At runtime the index will be memory mapped, so memory wont be an issue but disk storage would be needed.\n   * @param metric     distance metric for nearest neighbour search\n   * @param injection Injection to convert bytes to Id.\n   * @tparam T Type of Id for embedding\n   * @tparam D Typed Distance\n   * @return Serializable AnnoyIndex\n   */\n  def indexBuilder[T, D <: Distance[D]](\n    dimension: Int,\n    numOfTrees: Int,\n    metric: Metric[D],\n    injection: Injection[T, Array[Byte]],\n    futurePool: FuturePool\n  ): Appendable[T, AnnoyRuntimeParams, D] with Serialization = {\n    TypedAnnoyIndexBuilderWithFile(dimension, numOfTrees, metric, injection, futurePool)\n  }\n\n  /**\n   * Load Annoy based queryable index from a directory\n   * @param dimension dimension of embedding\n   * @param metric distance metric for nearest neighbour search\n   * @param injection Injection to convert bytes to Id.\n   * @param futurePool FuturePool\n   * @param directory Directory (HDFS/Local file system) where serialized index is stored.\n   * @tparam T Type of Id for embedding\n   * @tparam D Typed Distance\n   * @return Typed Queryable AnnoyIndex\n   */\n  def loadQueryableIndex[T, D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    injection: Injection[T, Array[Byte]],\n    futurePool: FuturePool,\n    directory: AbstractFile\n  ): Queryable[T, AnnoyRuntimeParams, D] = {\n    TypedAnnoyQueryIndexWithFile(dimension, metric, injection, futurePool, directory)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/annoy/TypedAnnoyIndexBuilderWithFile.scala",
    "content": "package com.twitter.ann.annoy\n\nimport com.twitter.ann.annoy.AnnoyCommon.IndexIdMappingFileName\nimport com.twitter.ann.common._\nimport com.twitter.ann.file_store.WritableIndexIdFileStore\nimport com.twitter.bijection.Injection\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.util.Future\nimport com.twitter.util.FuturePool\nimport org.apache.beam.sdk.io.fs.ResourceId\n\nprivate[annoy] object TypedAnnoyIndexBuilderWithFile {\n  private[annoy] def apply[T, D <: Distance[D]](\n    dimension: Int,\n    numOfTrees: Int,\n    metric: Metric[D],\n    injection: Injection[T, Array[Byte]],\n    futurePool: FuturePool\n  ): Appendable[T, AnnoyRuntimeParams, D] with Serialization = {\n    val index = RawAnnoyIndexBuilder(dimension, numOfTrees, metric, futurePool)\n    val writableFileStore = WritableIndexIdFileStore(injection)\n    new TypedAnnoyIndexBuilderWithFile[T, D](index, writableFileStore)\n  }\n}\n\nprivate[this] class TypedAnnoyIndexBuilderWithFile[T, D <: Distance[D]](\n  indexBuilder: RawAppendable[AnnoyRuntimeParams, D] with Serialization,\n  store: WritableIndexIdFileStore[T])\n    extends Appendable[T, AnnoyRuntimeParams, D]\n    with Serialization {\n  private[this] val transformedIndex = IndexTransformer.transformAppendable(indexBuilder, store)\n\n  override def append(entity: EntityEmbedding[T]): Future[Unit] = {\n    transformedIndex.append(entity)\n  }\n\n  override def toDirectory(directory: ResourceId): Unit = {\n    indexBuilder.toDirectory(directory)\n    toDirectory(new IndexOutputFile(directory))\n  }\n\n  override def toDirectory(directory: AbstractFile): Unit = {\n    indexBuilder.toDirectory(directory)\n    toDirectory(new IndexOutputFile(directory))\n  }\n\n  private def toDirectory(directory: IndexOutputFile): Unit = {\n    val indexIdFile = directory.createFile(IndexIdMappingFileName)\n    store.save(indexIdFile)\n  }\n\n  override def toQueryable: Queryable[T, AnnoyRuntimeParams, D] = {\n    transformedIndex.toQueryable\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/annoy/TypedAnnoyQueryIndexWithFile.scala",
    "content": "package com.twitter.ann.annoy\n\nimport com.twitter.ann.annoy.AnnoyCommon._\nimport com.twitter.ann.common._\nimport com.twitter.ann.file_store.ReadableIndexIdFileStore\nimport com.twitter.bijection.Injection\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.util.FuturePool\n\nprivate[annoy] object TypedAnnoyQueryIndexWithFile {\n  private[annoy] def apply[T, D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    injection: Injection[T, Array[Byte]],\n    futurePool: FuturePool,\n    directory: AbstractFile\n  ): Queryable[T, AnnoyRuntimeParams, D] = {\n    val deserializer =\n      new TypedAnnoyQueryIndexWithFile(dimension, metric, futurePool, injection)\n    deserializer.fromDirectory(directory)\n  }\n}\n\nprivate[this] class TypedAnnoyQueryIndexWithFile[T, D <: Distance[D]](\n  dimension: Int,\n  metric: Metric[D],\n  futurePool: FuturePool,\n  injection: Injection[T, Array[Byte]])\n    extends QueryableDeserialization[\n      T,\n      AnnoyRuntimeParams,\n      D,\n      Queryable[T, AnnoyRuntimeParams, D]\n    ] {\n  override def fromDirectory(directory: AbstractFile): Queryable[T, AnnoyRuntimeParams, D] = {\n    val index = RawAnnoyQueryIndex(dimension, metric, futurePool, directory)\n\n    val indexIdFile = directory.getChild(IndexIdMappingFileName)\n    val readableFileStore = ReadableIndexIdFileStore(indexIdFile, injection)\n    IndexTransformer.transformQueryable(index, readableFileStore)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/brute_force/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/serialization\",\n        \"ann/src/main/thrift/com/twitter/ann/serialization:serialization-scala\",\n        \"src/java/com/twitter/search/common/file\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/brute_force/BruteForceDeserialization.scala",
    "content": "package com.twitter.ann.brute_force\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.twitter.ann.common.{Distance, EntityEmbedding, Metric, QueryableDeserialization}\nimport com.twitter.ann.serialization.{PersistedEmbeddingInjection, ThriftIteratorIO}\nimport com.twitter.ann.serialization.thriftscala.PersistedEmbedding\nimport com.twitter.search.common.file.{AbstractFile, LocalFile}\nimport com.twitter.util.FuturePool\nimport java.io.File\n\n/**\n * @param factory creates a BruteForceIndex from the arguments. This is only exposed for testing.\n *                If for some reason you pass this arg in make sure that it eagerly consumes the\n *                iterator. If you don't you might close the input stream that the iterator is\n *                using.\n * @tparam T the id of the embeddings\n */\nclass BruteForceDeserialization[T, D <: Distance[D]] @VisibleForTesting private[brute_force] (\n  metric: Metric[D],\n  embeddingInjection: PersistedEmbeddingInjection[T],\n  futurePool: FuturePool,\n  thriftIteratorIO: ThriftIteratorIO[PersistedEmbedding],\n  factory: (Metric[D], FuturePool, Iterator[EntityEmbedding[T]]) => BruteForceIndex[T, D])\n    extends QueryableDeserialization[T, BruteForceRuntimeParams.type, D, BruteForceIndex[T, D]] {\n  import BruteForceIndex._\n\n  def this(\n    metric: Metric[D],\n    embeddingInjection: PersistedEmbeddingInjection[T],\n    futurePool: FuturePool,\n    thriftIteratorIO: ThriftIteratorIO[PersistedEmbedding]\n  ) = {\n    this(\n      metric,\n      embeddingInjection,\n      futurePool,\n      thriftIteratorIO,\n      factory = BruteForceIndex.apply[T, D]\n    )\n  }\n\n  override def fromDirectory(\n    serializationDirectory: AbstractFile\n  ): BruteForceIndex[T, D] = {\n    val file = File.createTempFile(DataFileName, \"tmp\")\n    file.deleteOnExit()\n    val temp = new LocalFile(file)\n    val dataFile = serializationDirectory.getChild(DataFileName)\n    dataFile.copyTo(temp)\n    val inputStream = temp.getByteSource.openBufferedStream()\n    try {\n      val iterator: Iterator[PersistedEmbedding] = thriftIteratorIO.fromInputStream(inputStream)\n\n      val embeddings = iterator.map { thriftEmbedding =>\n        embeddingInjection.invert(thriftEmbedding).get\n      }\n\n      factory(metric, futurePool, embeddings)\n    } finally {\n      inputStream.close()\n      temp.delete()\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/brute_force/BruteForceIndex.scala",
    "content": "package com.twitter.ann.brute_force\n\nimport com.twitter.ann.common.Appendable\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.EmbeddingType._\nimport com.twitter.ann.common.EntityEmbedding\nimport com.twitter.ann.common.IndexOutputFile\nimport com.twitter.ann.common.Metric\nimport com.twitter.ann.common.NeighborWithDistance\nimport com.twitter.ann.common.Queryable\nimport com.twitter.ann.common.RuntimeParams\nimport com.twitter.ann.common.Serialization\nimport com.twitter.ann.serialization.PersistedEmbeddingInjection\nimport com.twitter.ann.serialization.ThriftIteratorIO\nimport com.twitter.ann.serialization.thriftscala.PersistedEmbedding\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.util.Future\nimport com.twitter.util.FuturePool\nimport java.util.concurrent.ConcurrentLinkedQueue\nimport org.apache.beam.sdk.io.fs.ResourceId\nimport scala.collection.JavaConverters._\nimport scala.collection.mutable\n\nobject BruteForceRuntimeParams extends RuntimeParams\n\nobject BruteForceIndex {\n  val DataFileName = \"BruteForceFileData\"\n\n  def apply[T, D <: Distance[D]](\n    metric: Metric[D],\n    futurePool: FuturePool,\n    initialEmbeddings: Iterator[EntityEmbedding[T]] = Iterator()\n  ): BruteForceIndex[T, D] = {\n    val linkedQueue = new ConcurrentLinkedQueue[EntityEmbedding[T]]\n    initialEmbeddings.foreach(embedding => linkedQueue.add(embedding))\n    new BruteForceIndex(metric, futurePool, linkedQueue)\n  }\n}\n\nclass BruteForceIndex[T, D <: Distance[D]] private (\n  metric: Metric[D],\n  futurePool: FuturePool,\n  // visible for serialization\n  private[brute_force] val linkedQueue: ConcurrentLinkedQueue[EntityEmbedding[T]])\n    extends Appendable[T, BruteForceRuntimeParams.type, D]\n    with Queryable[T, BruteForceRuntimeParams.type, D] {\n\n  override def append(embedding: EntityEmbedding[T]): Future[Unit] = {\n    futurePool {\n      linkedQueue.add(embedding)\n    }\n  }\n\n  override def toQueryable: Queryable[T, BruteForceRuntimeParams.type, D] = this\n\n  override def query(\n    embedding: EmbeddingVector,\n    numOfNeighbours: Int,\n    runtimeParams: BruteForceRuntimeParams.type\n  ): Future[List[T]] = {\n    queryWithDistance(embedding, numOfNeighbours, runtimeParams).map { neighborsWithDistance =>\n      neighborsWithDistance.map(_.neighbor)\n    }\n  }\n\n  override def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbours: Int,\n    runtimeParams: BruteForceRuntimeParams.type\n  ): Future[List[NeighborWithDistance[T, D]]] = {\n    futurePool {\n      // Use the reverse ordering so that we can call dequeue to remove the largest element.\n      val ordering = Ordering.by[NeighborWithDistance[T, D], D](_.distance)\n      val priorityQueue =\n        new mutable.PriorityQueue[NeighborWithDistance[T, D]]()(ordering)\n      linkedQueue\n        .iterator()\n        .asScala\n        .foreach { entity =>\n          val neighborWithDistance =\n            NeighborWithDistance(entity.id, metric.distance(entity.embedding, embedding))\n          priorityQueue.+=(neighborWithDistance)\n          if (priorityQueue.size > numOfNeighbours) {\n            priorityQueue.dequeue()\n          }\n        }\n      val reverseList: List[NeighborWithDistance[T, D]] =\n        priorityQueue.dequeueAll\n      reverseList.reverse\n    }\n  }\n}\n\nobject SerializableBruteForceIndex {\n  def apply[T, D <: Distance[D]](\n    metric: Metric[D],\n    futurePool: FuturePool,\n    embeddingInjection: PersistedEmbeddingInjection[T],\n    thriftIteratorIO: ThriftIteratorIO[PersistedEmbedding]\n  ): SerializableBruteForceIndex[T, D] = {\n    val bruteForceIndex = BruteForceIndex[T, D](metric, futurePool)\n\n    new SerializableBruteForceIndex(bruteForceIndex, embeddingInjection, thriftIteratorIO)\n  }\n}\n\n/**\n * This is a class that wrapps a BruteForceIndex and provides a method for serialization.\n *\n  * @param bruteForceIndex all queries and updates are sent to this index.\n * @param embeddingInjection injection that can convert embeddings to thrift embeddings.\n * @param thriftIteratorIO class that provides a way to write PersistedEmbeddings to disk\n */\nclass SerializableBruteForceIndex[T, D <: Distance[D]](\n  bruteForceIndex: BruteForceIndex[T, D],\n  embeddingInjection: PersistedEmbeddingInjection[T],\n  thriftIteratorIO: ThriftIteratorIO[PersistedEmbedding])\n    extends Appendable[T, BruteForceRuntimeParams.type, D]\n    with Queryable[T, BruteForceRuntimeParams.type, D]\n    with Serialization {\n  import BruteForceIndex._\n\n  override def append(entity: EntityEmbedding[T]): Future[Unit] =\n    bruteForceIndex.append(entity)\n\n  override def toQueryable: Queryable[T, BruteForceRuntimeParams.type, D] = this\n\n  override def query(\n    embedding: EmbeddingVector,\n    numOfNeighbours: Int,\n    runtimeParams: BruteForceRuntimeParams.type\n  ): Future[List[T]] =\n    bruteForceIndex.query(embedding, numOfNeighbours, runtimeParams)\n\n  override def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbours: Int,\n    runtimeParams: BruteForceRuntimeParams.type\n  ): Future[List[NeighborWithDistance[T, D]]] =\n    bruteForceIndex.queryWithDistance(embedding, numOfNeighbours, runtimeParams)\n\n  override def toDirectory(serializationDirectory: ResourceId): Unit = {\n    toDirectory(new IndexOutputFile(serializationDirectory))\n  }\n\n  override def toDirectory(serializationDirectory: AbstractFile): Unit = {\n    toDirectory(new IndexOutputFile(serializationDirectory))\n  }\n\n  private def toDirectory(serializationDirectory: IndexOutputFile): Unit = {\n    val outputStream = serializationDirectory.createFile(DataFileName).getOutputStream()\n    val thriftEmbeddings =\n      bruteForceIndex.linkedQueue.iterator().asScala.map { embedding =>\n        embeddingInjection(embedding)\n      }\n    try {\n      thriftIteratorIO.toOutputStream(thriftEmbeddings, outputStream)\n    } finally {\n      outputStream.close()\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/AnnInjections.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.bijection.{Bijection, Injection}\n\n// Class providing commonly used injections that can be used directly with ANN apis.\n// Injection  prefixed with `J` can be used in java directly with ANN apis.\nobject AnnInjections {\n  val LongInjection: Injection[Long, Array[Byte]] = Injection.long2BigEndian\n\n  def StringInjection: Injection[String, Array[Byte]] = Injection.utf8\n\n  def IntInjection: Injection[Int, Array[Byte]] = Injection.int2BigEndian\n\n  val JLongInjection: Injection[java.lang.Long, Array[Byte]] =\n    Bijection.long2Boxed\n      .asInstanceOf[Bijection[Long, java.lang.Long]]\n      .inverse\n      .andThen(LongInjection)\n\n  val JStringInjection: Injection[java.lang.String, Array[Byte]] =\n    StringInjection\n\n  val JIntInjection: Injection[java.lang.Integer, Array[Byte]] =\n    Bijection.int2Boxed\n      .asInstanceOf[Bijection[Int, java.lang.Integer]]\n      .inverse\n      .andThen(IntInjection)\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/Api.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.ml.api.embedding.Embedding\nimport com.twitter.ml.api.embedding.EmbeddingMath\nimport com.twitter.ml.api.embedding.EmbeddingSerDe\nimport com.twitter.util.Future\n\nobject EmbeddingType {\n  type EmbeddingVector = Embedding[Float]\n  val embeddingSerDe = EmbeddingSerDe.apply[Float]\n  private[common] val math = EmbeddingMath.Float\n}\n\n/**\n * Typed entity with an embedding associated with it.\n * @param id : Unique Id for an entity.\n * @param embedding : Embedding/Vector of an entity.\n * @tparam T: Type of id.\n */\ncase class EntityEmbedding[T](id: T, embedding: EmbeddingVector)\n\n// Query interface for ANN\ntrait Queryable[T, P <: RuntimeParams, D <: Distance[D]] {\n\n  /**\n   * ANN query for ids.\n   * @param embedding: Embedding/Vector to be queried with.\n   * @param numOfNeighbors: Number of neighbours to be queried for.\n   * @param runtimeParams: Runtime params associated with index to control accuracy/latency etc.\n   * @return List of approximate nearest neighbour ids.\n   */\n  def query(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Future[List[T]]\n\n  /**\n   * ANN query for ids with distance.\n   * @param embedding: Embedding/Vector to be queried with.\n   * @param numOfNeighbors: Number of neighbours to be queried for.\n   * @param runtimeParams: Runtime params associated with index to control accuracy/latency etc.\n   * @return List of approximate nearest neighbour ids with distance from the query embedding.\n   */\n  def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Future[List[NeighborWithDistance[T, D]]]\n}\n\n// Query interface for ANN over indexes that are grouped\ntrait QueryableGrouped[T, P <: RuntimeParams, D <: Distance[D]] extends Queryable[T, P, D] {\n\n  /**\n   * ANN query for ids.\n   * @param embedding: Embedding/Vector to be queried with.\n   * @param numOfNeighbors: Number of neighbours to be queried for.\n   * @param runtimeParams: Runtime params associated with index to control accuracy/latency etc.\n   * @param key: Optional key to lookup specific ANN index and perform query there\n   * @return List of approximate nearest neighbour ids.\n   */\n  def query(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: P,\n    key: Option[String]\n  ): Future[List[T]]\n\n  /**\n   * ANN query for ids with distance.\n   * @param embedding: Embedding/Vector to be queried with.\n   * @param numOfNeighbors: Number of neighbours to be queried for.\n   * @param runtimeParams: Runtime params associated with index to control accuracy/latency etc.\n   * @param key: Optional key to lookup specific ANN index and perform query there\n   * @return List of approximate nearest neighbour ids with distance from the query embedding.\n   */\n  def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: P,\n    key: Option[String]\n  ): Future[List[NeighborWithDistance[T, D]]]\n}\n\n/**\n * Runtime params associated with index to control accuracy/latency etc while querying.\n */\ntrait RuntimeParams {}\n\n/**\n * ANN query result with distance.\n * @param neighbor : Id of the neighbours\n * @param distance: Distance of neighbour from query ex: D: CosineDistance, L2Distance, InnerProductDistance\n */\ncase class NeighborWithDistance[T, D <: Distance[D]](neighbor: T, distance: D)\n\n/**\n * ANN query result with seed entity for which this neighbor was provided.\n * @param seed: Seed Id for which ann query was called\n * @param neighbor : Id of the neighbours\n */\ncase class NeighborWithSeed[T1, T2](seed: T1, neighbor: T2)\n\n/**\n * ANN query result with distance with seed entity for which this neighbor was provided.\n * @param seed: Seed Id for which ann query was called\n * @param neighbor : Id of the neighbours\n * @param distance: Distance of neighbour from query ex: D: CosineDistance, L2Distance, InnerProductDistance\n */\ncase class NeighborWithDistanceWithSeed[T1, T2, D <: Distance[D]](\n  seed: T1,\n  neighbor: T2,\n  distance: D)\n\ntrait RawAppendable[P <: RuntimeParams, D <: Distance[D]] {\n\n  /**\n   * Append an embedding in an index.\n   * @param embedding: Embedding/Vector\n   * @return Future of long id associated with embedding autogenerated.\n   */\n  def append(embedding: EmbeddingVector): Future[Long]\n\n  /**\n   * Convert an Appendable to Queryable interface to query an index.\n   */\n  def toQueryable: Queryable[Long, P, D]\n}\n\n// Index building interface for ANN.\ntrait Appendable[T, P <: RuntimeParams, D <: Distance[D]] {\n\n  /**\n   *  Append an entity with embedding in an index.\n   * @param entity: Entity with its embedding\n   */\n  def append(entity: EntityEmbedding[T]): Future[Unit]\n\n  /**\n   * Convert an Appendable to Queryable interface to query an index.\n   */\n  def toQueryable: Queryable[T, P, D]\n}\n\n// Updatable index interface for ANN.\ntrait Updatable[T] {\n  def update(entity: EntityEmbedding[T]): Future[Unit]\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/twitter/bijection:core\",\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/org/apache/beam:beam-sdks-java-io-google-cloud-platform\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"finatra/inject/inject-mdc/src/main/scala\",\n        \"mediaservices/commons/src/main/scala:futuretracker\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/scala/com/twitter/ml/api/embedding\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/com/twitter/bijection:core\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/EmbeddingProducer.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.stitch.Stitch\n\ntrait EmbeddingProducer[T] {\n\n  /**\n   * Produce an embedding from type T. Implementations of this could do a lookup from an id to an\n   * embedding. Or they could run a deep model on features that output and embedding.\n   * @return An embedding Stitch. See go/stitch for details on how to use the Stitch API.\n   */\n  def produceEmbedding(input: T): Stitch[Option[EmbeddingType.EmbeddingVector]]\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/IndexOutputFile.scala",
    "content": "package com.twitter.ann.common\n\nimport com.google.common.io.ByteStreams\nimport com.twitter.ann.common.thriftscala.AnnIndexMetadata\nimport com.twitter.mediaservices.commons.codec.ArrayByteBufferCodec\nimport com.twitter.mediaservices.commons.codec.ThriftByteBufferCodec\nimport com.twitter.search.common.file.AbstractFile\nimport java.io.IOException\nimport java.io.InputStream\nimport java.io.OutputStream\nimport java.nio.channels.Channels\nimport org.apache.beam.sdk.io.FileSystems\nimport org.apache.beam.sdk.io.fs.MoveOptions\nimport org.apache.beam.sdk.io.fs.ResolveOptions\nimport org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions\nimport org.apache.beam.sdk.io.fs.ResourceId\nimport org.apache.beam.sdk.util.MimeTypes\nimport org.apache.hadoop.io.IOUtils\nimport scala.collection.JavaConverters._\n\n/**\n * This class creates a wrapper around GCS filesystem and HDFS filesystem for the index\n * generation job. It implements the basic methods required by the index generation job and hides\n * the logic around handling HDFS vs GCS.\n */\nclass IndexOutputFile(val abstractFile: AbstractFile, val resourceId: ResourceId) {\n\n  // Success file name\n  private val SUCCESS_FILE = \"_SUCCESS\"\n  private val INDEX_METADATA_FILE = \"ANN_INDEX_METADATA\"\n  private val MetadataCodec = new ThriftByteBufferCodec[AnnIndexMetadata](AnnIndexMetadata)\n\n  /**\n   * Constructor for ResourceId. This is used for GCS filesystem\n   * @param resourceId\n   */\n  def this(resourceId: ResourceId) = {\n    this(null, resourceId)\n  }\n\n  /**\n   * Constructor for AbstractFile. This is used for HDFS and local filesystem\n   * @param abstractFile\n   */\n  def this(abstractFile: AbstractFile) = {\n    this(abstractFile, null)\n  }\n\n  /**\n   * Returns true if this instance is around an AbstractFile.\n   * @return\n   */\n  def isAbstractFile(): Boolean = {\n    abstractFile != null\n  }\n\n  /**\n   * Creates a _SUCCESS file in the current directory.\n   */\n  def createSuccessFile(): Unit = {\n    if (isAbstractFile()) {\n      abstractFile.createSuccessFile()\n    } else {\n      val successFile =\n        resourceId.resolve(SUCCESS_FILE, ResolveOptions.StandardResolveOptions.RESOLVE_FILE)\n      val successWriterChannel = FileSystems.create(successFile, MimeTypes.BINARY)\n      successWriterChannel.close()\n    }\n  }\n\n  /**\n   * Returns whether the current instance represents a directory\n   * @return True if the current instance is a directory\n   */\n  def isDirectory(): Boolean = {\n    if (isAbstractFile()) {\n      abstractFile.isDirectory\n    } else {\n      resourceId.isDirectory\n    }\n  }\n\n  /**\n   * Return the current path of the file represented by the current instance\n   * @return The path string of the file/directory\n   */\n  def getPath(): String = {\n    if (isAbstractFile()) {\n      abstractFile.getPath.toString\n    } else {\n      if (resourceId.isDirectory) {\n        resourceId.getCurrentDirectory.toString\n      } else {\n        resourceId.getCurrentDirectory.toString + resourceId.getFilename\n      }\n    }\n  }\n\n  /**\n   * Creates a new file @param fileName in the current directory.\n   * @param fileName\n   * @return A new file inside the current directory\n   */\n  def createFile(fileName: String): IndexOutputFile = {\n    if (isAbstractFile()) {\n      // AbstractFile treats files and directories the same way. Hence, not checking for directory\n      // here.\n      new IndexOutputFile(abstractFile.getChild(fileName))\n    } else {\n      if (!resourceId.isDirectory) {\n        // If this is not a directory, throw exception.\n        throw new IllegalArgumentException(getPath() + \" is not a directory.\")\n      }\n      new IndexOutputFile(\n        resourceId.resolve(fileName, ResolveOptions.StandardResolveOptions.RESOLVE_FILE))\n    }\n  }\n\n  /**\n   * Creates a new directory @param directoryName in the current directory.\n   * @param directoryName\n   * @return A new directory inside the current directory\n   */\n  def createDirectory(directoryName: String): IndexOutputFile = {\n    if (isAbstractFile()) {\n      // AbstractFile treats files and directories the same way. Hence, not checking for directory\n      // here.\n      val dir = abstractFile.getChild(directoryName)\n      dir.mkdirs()\n      new IndexOutputFile(dir)\n    } else {\n      if (!resourceId.isDirectory) {\n        // If this is not a directory, throw exception.\n        throw new IllegalArgumentException(getPath() + \" is not a directory.\")\n      }\n      val newResourceId =\n        resourceId.resolve(directoryName, ResolveOptions.StandardResolveOptions.RESOLVE_DIRECTORY)\n\n      // Create a tmp file and delete in order to trigger directory creation\n      val tmpFile =\n        newResourceId.resolve(\"tmp\", ResolveOptions.StandardResolveOptions.RESOLVE_FILE)\n      val tmpWriterChannel = FileSystems.create(tmpFile, MimeTypes.BINARY)\n      tmpWriterChannel.close()\n      FileSystems.delete(List(tmpFile).asJava, MoveOptions.StandardMoveOptions.IGNORE_MISSING_FILES)\n\n      new IndexOutputFile(newResourceId)\n    }\n  }\n\n  def getChild(fileName: String, isDirectory: Boolean = false): IndexOutputFile = {\n    if (isAbstractFile()) {\n      new IndexOutputFile(abstractFile.getChild(fileName))\n    } else {\n      val resolveOption = if (isDirectory) {\n        StandardResolveOptions.RESOLVE_DIRECTORY\n      } else {\n        StandardResolveOptions.RESOLVE_FILE\n      }\n      new IndexOutputFile(resourceId.resolve(fileName, resolveOption))\n    }\n  }\n\n  /**\n   * Returns an OutputStream for the underlying file.\n   * Note: Close the OutputStream after writing\n   * @return\n   */\n  def getOutputStream(): OutputStream = {\n    if (isAbstractFile()) {\n      abstractFile.getByteSink.openStream()\n    } else {\n      if (resourceId.isDirectory) {\n        // If this is a directory, throw exception.\n        throw new IllegalArgumentException(getPath() + \" is a directory.\")\n      }\n      val writerChannel = FileSystems.create(resourceId, MimeTypes.BINARY)\n      Channels.newOutputStream(writerChannel)\n    }\n  }\n\n  /**\n   * Returns an InputStream for the underlying file.\n   * Note: Close the InputStream after reading\n   * @return\n   */\n  def getInputStream(): InputStream = {\n    if (isAbstractFile()) {\n      abstractFile.getByteSource.openStream()\n    } else {\n      if (resourceId.isDirectory) {\n        // If this is a directory, throw exception.\n        throw new IllegalArgumentException(getPath() + \" is a directory.\")\n      }\n      val readChannel = FileSystems.open(resourceId)\n      Channels.newInputStream(readChannel)\n    }\n  }\n\n  /**\n   * Copies content from the srcIn into the current file.\n   * @param srcIn\n   */\n  def copyFrom(srcIn: InputStream): Unit = {\n    val out = getOutputStream()\n    try {\n      IOUtils.copyBytes(srcIn, out, 4096)\n      out.close()\n    } catch {\n      case ex: IOException =>\n        IOUtils.closeStream(out);\n        throw ex;\n    }\n  }\n\n  def writeIndexMetadata(annIndexMetadata: AnnIndexMetadata): Unit = {\n    val out = createFile(INDEX_METADATA_FILE).getOutputStream()\n    val bytes = ArrayByteBufferCodec.decode(MetadataCodec.encode(annIndexMetadata))\n    out.write(bytes)\n    out.close()\n  }\n\n  def loadIndexMetadata(): AnnIndexMetadata = {\n    val in = ByteStreams.toByteArray(getInputStream())\n    MetadataCodec.decode(ArrayByteBufferCodec.encode(in))\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/IndexTransformer.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.storehaus.{ReadableStore, Store}\nimport com.twitter.util.Future\n\n// Utility to transform raw index to typed index using Store\nobject IndexTransformer {\n\n  /**\n   * Transform a long type queryable index to Typed queryable index\n   * @param index: Raw Queryable index\n   * @param store: Readable store to provide mappings between Long and T\n   * @tparam T: Type to transform to\n   * @tparam P: Runtime params\n   * @return Queryable index typed on T\n   */\n  def transformQueryable[T, P <: RuntimeParams, D <: Distance[D]](\n    index: Queryable[Long, P, D],\n    store: ReadableStore[Long, T]\n  ): Queryable[T, P, D] = {\n    new Queryable[T, P, D] {\n      override def query(\n        embedding: EmbeddingVector,\n        numOfNeighbors: Int,\n        runtimeParams: P\n      ): Future[List[T]] = {\n        val neighbors = index.query(embedding, numOfNeighbors, runtimeParams)\n        neighbors\n          .flatMap(nn => {\n            val ids = nn.map(id => store.get(id).map(_.get))\n            Future\n              .collect(ids)\n              .map(_.toList)\n          })\n      }\n\n      override def queryWithDistance(\n        embedding: EmbeddingVector,\n        numOfNeighbors: Int,\n        runtimeParams: P\n      ): Future[List[NeighborWithDistance[T, D]]] = {\n        val neighbors = index.queryWithDistance(embedding, numOfNeighbors, runtimeParams)\n        neighbors\n          .flatMap(nn => {\n            val ids = nn.map(obj =>\n              store.get(obj.neighbor).map(id => NeighborWithDistance(id.get, obj.distance)))\n            Future\n              .collect(ids)\n              .map(_.toList)\n          })\n      }\n    }\n  }\n\n  /**\n   * Transform a long type appendable index to Typed appendable index\n   * @param index: Raw Appendable index\n   * @param store: Writable store to store mappings between Long and T\n   * @tparam T: Type to transform to\n   * @return Appendable index typed on T\n   */\n  def transformAppendable[T, P <: RuntimeParams, D <: Distance[D]](\n    index: RawAppendable[P, D],\n    store: Store[Long, T]\n  ): Appendable[T, P, D] = {\n    new Appendable[T, P, D]() {\n      override def append(entity: EntityEmbedding[T]): Future[Unit] = {\n        index\n          .append(entity.embedding)\n          .flatMap(id => store.put((id, Some(entity.id))))\n      }\n\n      override def toQueryable: Queryable[T, P, D] = {\n        transformQueryable(index.toQueryable, store)\n      }\n    }\n  }\n\n  /**\n   * Transform a long type appendable and queryable index to Typed appendable and queryable index\n   * @param index: Raw Appendable and queryable index\n   * @param store: Store to provide/store mappings between Long and T\n   * @tparam T: Type to transform to\n   * @tparam Index: Index\n   * @return Appendable and queryable index typed on T\n   */\n  def transform1[\n    Index <: RawAppendable[P, D] with Queryable[Long, P, D],\n    T,\n    P <: RuntimeParams,\n    D <: Distance[D]\n  ](\n    index: Index,\n    store: Store[Long, T]\n  ): Queryable[T, P, D] with Appendable[T, P, D] = {\n    val queryable = transformQueryable(index, store)\n    val appendable = transformAppendable(index, store)\n\n    new Queryable[T, P, D] with Appendable[T, P, D] {\n      override def query(\n        embedding: EmbeddingVector,\n        numOfNeighbors: Int,\n        runtimeParams: P\n      ) = queryable.query(embedding, numOfNeighbors, runtimeParams)\n\n      override def queryWithDistance(\n        embedding: EmbeddingVector,\n        numOfNeighbors: Int,\n        runtimeParams: P\n      ) = queryable.queryWithDistance(embedding, numOfNeighbors, runtimeParams)\n\n      override def append(entity: EntityEmbedding[T]) = appendable.append(entity)\n\n      override def toQueryable: Queryable[T, P, D] = appendable.toQueryable\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/MemoizedInEpochs.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\n\n// Memoization with a twist\n// New epoch reuse K:V pairs from previous and recycle everything else\nclass MemoizedInEpochs[K, V](f: K => Try[V]) extends Logging {\n  private var memoizedCalls: Map[K, V] = Map.empty\n\n  def epoch(keys: Seq[K]): Seq[V] = {\n    val newSet = keys.toSet\n    val keysToBeComputed = newSet.diff(memoizedCalls.keySet)\n    val computedKeysAndValues = keysToBeComputed.map { key =>\n      info(s\"Memoize ${key}\")\n      (key, f(key))\n    }\n    val keysAndValuesAfterFilteringFailures = computedKeysAndValues\n      .flatMap {\n        case (key, Return(value)) => Some((key, value))\n        case (key, Throw(e)) =>\n          warn(s\"Calling f for ${key} has failed\", e)\n\n          None\n      }\n    val keysReusedFromLastEpoch = memoizedCalls.filterKeys(newSet.contains)\n    memoizedCalls = keysReusedFromLastEpoch ++ keysAndValuesAfterFilteringFailures\n\n    debug(s\"Final memoization is ${memoizedCalls.keys.mkString(\", \")}\")\n\n    keys.flatMap(memoizedCalls.get)\n  }\n\n  def currentEpochKeys: Set[K] = memoizedCalls.keySet\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/Metric.scala",
    "content": "package com.twitter.ann.common\n\nimport com.google.common.collect.ImmutableBiMap\nimport com.twitter.ann.common.EmbeddingType._\nimport com.twitter.ann.common.thriftscala.DistanceMetric\nimport com.twitter.ann.common.thriftscala.{CosineDistance => ServiceCosineDistance}\nimport com.twitter.ann.common.thriftscala.{Distance => ServiceDistance}\nimport com.twitter.ann.common.thriftscala.{InnerProductDistance => ServiceInnerProductDistance}\nimport com.twitter.ann.common.thriftscala.{EditDistance => ServiceEditDistance}\nimport com.twitter.ann.common.thriftscala.{L2Distance => ServiceL2Distance}\nimport com.twitter.bijection.Injection\nimport scala.util.Failure\nimport scala.util.Success\nimport scala.util.Try\n\n// Ann distance metrics\ntrait Distance[D] extends Any with Ordered[D] {\n  def distance: Float\n}\n\ncase class L2Distance(distance: Float) extends AnyVal with Distance[L2Distance] {\n  override def compare(that: L2Distance): Int =\n    Ordering.Float.compare(this.distance, that.distance)\n}\n\ncase class CosineDistance(distance: Float) extends AnyVal with Distance[CosineDistance] {\n  override def compare(that: CosineDistance): Int =\n    Ordering.Float.compare(this.distance, that.distance)\n}\n\ncase class InnerProductDistance(distance: Float)\n    extends AnyVal\n    with Distance[InnerProductDistance] {\n  override def compare(that: InnerProductDistance): Int =\n    Ordering.Float.compare(this.distance, that.distance)\n}\n\ncase class EditDistance(distance: Float) extends AnyVal with Distance[EditDistance] {\n  override def compare(that: EditDistance): Int =\n    Ordering.Float.compare(this.distance, that.distance)\n}\n\nobject Metric {\n  private[this] val thriftMetricMapping = ImmutableBiMap.of(\n    L2,\n    DistanceMetric.L2,\n    Cosine,\n    DistanceMetric.Cosine,\n    InnerProduct,\n    DistanceMetric.InnerProduct,\n    Edit,\n    DistanceMetric.EditDistance\n  )\n\n  def fromThrift(metric: DistanceMetric): Metric[_ <: Distance[_]] = {\n    thriftMetricMapping.inverse().get(metric)\n  }\n\n  def toThrift(metric: Metric[_ <: Distance[_]]): DistanceMetric = {\n    thriftMetricMapping.get(metric)\n  }\n\n  def fromString(metricName: String): Metric[_ <: Distance[_]]\n    with Injection[_, ServiceDistance] = {\n    metricName match {\n      case \"Cosine\" => Cosine\n      case \"L2\" => L2\n      case \"InnerProduct\" => InnerProduct\n      case \"EditDistance\" => Edit\n      case _ =>\n        throw new IllegalArgumentException(s\"No Metric with the name $metricName\")\n    }\n  }\n}\n\nsealed trait Metric[D <: Distance[D]] {\n  def distance(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): D\n  def absoluteDistance(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): Float\n  def fromAbsoluteDistance(distance: Float): D\n}\n\ncase object L2 extends Metric[L2Distance] with Injection[L2Distance, ServiceDistance] {\n  override def distance(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): L2Distance = {\n    fromAbsoluteDistance(MetricUtil.l2distance(embedding1, embedding2).toFloat)\n  }\n\n  override def fromAbsoluteDistance(distance: Float): L2Distance = {\n    L2Distance(distance)\n  }\n\n  override def absoluteDistance(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): Float = distance(embedding1, embedding2).distance\n\n  override def apply(scalaDistance: L2Distance): ServiceDistance = {\n    ServiceDistance.L2Distance(ServiceL2Distance(scalaDistance.distance))\n  }\n\n  override def invert(serviceDistance: ServiceDistance): Try[L2Distance] = {\n    serviceDistance match {\n      case ServiceDistance.L2Distance(l2Distance) =>\n        Success(L2Distance(l2Distance.distance.toFloat))\n      case distance =>\n        Failure(new IllegalArgumentException(s\"Expected an l2 distance but got $distance\"))\n    }\n  }\n}\n\ncase object Cosine extends Metric[CosineDistance] with Injection[CosineDistance, ServiceDistance] {\n  override def distance(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): CosineDistance = {\n    fromAbsoluteDistance(1 - MetricUtil.cosineSimilarity(embedding1, embedding2))\n  }\n\n  override def fromAbsoluteDistance(distance: Float): CosineDistance = {\n    CosineDistance(distance)\n  }\n\n  override def absoluteDistance(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): Float = distance(embedding1, embedding2).distance\n\n  override def apply(scalaDistance: CosineDistance): ServiceDistance = {\n    ServiceDistance.CosineDistance(ServiceCosineDistance(scalaDistance.distance))\n  }\n\n  override def invert(serviceDistance: ServiceDistance): Try[CosineDistance] = {\n    serviceDistance match {\n      case ServiceDistance.CosineDistance(cosineDistance) =>\n        Success(CosineDistance(cosineDistance.distance.toFloat))\n      case distance =>\n        Failure(new IllegalArgumentException(s\"Expected a cosine distance but got $distance\"))\n    }\n  }\n}\n\ncase object InnerProduct\n    extends Metric[InnerProductDistance]\n    with Injection[InnerProductDistance, ServiceDistance] {\n  override def distance(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): InnerProductDistance = {\n    fromAbsoluteDistance(1 - MetricUtil.dot(embedding1, embedding2))\n  }\n\n  override def fromAbsoluteDistance(distance: Float): InnerProductDistance = {\n    InnerProductDistance(distance)\n  }\n\n  override def absoluteDistance(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): Float = distance(embedding1, embedding2).distance\n\n  override def apply(scalaDistance: InnerProductDistance): ServiceDistance = {\n    ServiceDistance.InnerProductDistance(ServiceInnerProductDistance(scalaDistance.distance))\n  }\n\n  override def invert(\n    serviceDistance: ServiceDistance\n  ): Try[InnerProductDistance] = {\n    serviceDistance match {\n      case ServiceDistance.InnerProductDistance(cosineDistance) =>\n        Success(InnerProductDistance(cosineDistance.distance.toFloat))\n      case distance =>\n        Failure(\n          new IllegalArgumentException(s\"Expected a inner product distance but got $distance\")\n        )\n    }\n  }\n}\n\ncase object Edit extends Metric[EditDistance] with Injection[EditDistance, ServiceDistance] {\n\n  private def intDistance(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector,\n    pos1: Int,\n    pos2: Int,\n    precomputedDistances: scala.collection.mutable.Map[(Int, Int), Int]\n  ): Int = {\n    // return the remaining characters of other String\n    if (pos1 == 0) return pos2\n    if (pos2 == 0) return pos1\n\n    // To check if the recursive tree\n    // for given n & m has already been executed\n    precomputedDistances.getOrElse(\n      (pos1, pos2), {\n        // We might want to change this so that capitals are considered the same.\n        // Also maybe some characters that look similar should also be the same.\n        val computed = if (embedding1(pos1 - 1) == embedding2(pos2 - 1)) {\n          intDistance(embedding1, embedding2, pos1 - 1, pos2 - 1, precomputedDistances)\n        } else { // If characters are nt equal, we need to\n          // find the minimum cost out of all 3 operations.\n          val insert = intDistance(embedding1, embedding2, pos1, pos2 - 1, precomputedDistances)\n          val del = intDistance(embedding1, embedding2, pos1 - 1, pos2, precomputedDistances)\n          val replace =\n            intDistance(embedding1, embedding2, pos1 - 1, pos2 - 1, precomputedDistances)\n          1 + Math.min(insert, Math.min(del, replace))\n        }\n        precomputedDistances.put((pos1, pos2), computed)\n        computed\n      }\n    )\n  }\n\n  override def distance(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): EditDistance = {\n    val editDistance = intDistance(\n      embedding1,\n      embedding2,\n      embedding1.length,\n      embedding2.length,\n      scala.collection.mutable.Map[(Int, Int), Int]()\n    )\n    EditDistance(editDistance)\n  }\n\n  override def fromAbsoluteDistance(distance: Float): EditDistance = {\n    EditDistance(distance.toInt)\n  }\n\n  override def absoluteDistance(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): Float = distance(embedding1, embedding2).distance\n\n  override def apply(scalaDistance: EditDistance): ServiceDistance = {\n    ServiceDistance.EditDistance(ServiceEditDistance(scalaDistance.distance.toInt))\n  }\n\n  override def invert(\n    serviceDistance: ServiceDistance\n  ): Try[EditDistance] = {\n    serviceDistance match {\n      case ServiceDistance.EditDistance(cosineDistance) =>\n        Success(EditDistance(cosineDistance.distance.toFloat))\n      case distance =>\n        Failure(\n          new IllegalArgumentException(s\"Expected a inner product distance but got $distance\")\n        )\n    }\n  }\n}\n\nobject MetricUtil {\n  private[ann] def dot(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): Float = {\n    math.dotProduct(embedding1, embedding2)\n  }\n\n  private[ann] def l2distance(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): Double = {\n    math.l2Distance(embedding1, embedding2)\n  }\n\n  private[ann] def cosineSimilarity(\n    embedding1: EmbeddingVector,\n    embedding2: EmbeddingVector\n  ): Float = {\n    math.cosineSimilarity(embedding1, embedding2).toFloat\n  }\n\n  private[ann] def norm(\n    embedding: EmbeddingVector\n  ): EmbeddingVector = {\n    math.normalize(embedding)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/QueryableById.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.stitch.Stitch\n\n/**\n * This is a trait that allows you to query for nearest neighbors given an arbitrary type T1. This is\n * in contrast to a regular com.twitter.ann.common.Appendable, which takes an embedding as the input\n * argument.\n *\n * This interface uses the Stitch API for batching. See go/stitch for details on how to use it.\n *\n * @tparam T1 type of the query.\n * @tparam T2 type of the result.\n * @tparam P runtime parameters supported by the index.\n * @tparam D distance function used in the index.\n */\ntrait QueryableById[T1, T2, P <: RuntimeParams, D <: Distance[D]] {\n  def queryById(\n    id: T1,\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Stitch[List[T2]]\n\n  def queryByIdWithDistance(\n    id: T1,\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Stitch[List[NeighborWithDistance[T2, D]]]\n\n  def batchQueryById(\n    ids: Seq[T1],\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Stitch[List[NeighborWithSeed[T1, T2]]]\n\n  def batchQueryWithDistanceById(\n    ids: Seq[T1],\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Stitch[List[NeighborWithDistanceWithSeed[T1, T2, D]]]\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/QueryableByIdImplementation.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.stitch.Stitch\n\n/**\n * Implementation of QueryableById that composes an EmbeddingProducer and a Queryable so that we\n * can get nearest neighbors given an id of type T1\n * @param embeddingProducer provides an embedding given an id.\n * @param queryable provides a list of neighbors given an embedding.\n * @tparam T1 type of the query.\n * @tparam T2 type of the result.\n * @tparam P runtime parameters supported by the index.\n * @tparam D distance function used in the index.\n */\nclass QueryableByIdImplementation[T1, T2, P <: RuntimeParams, D <: Distance[D]](\n  embeddingProducer: EmbeddingProducer[T1],\n  queryable: Queryable[T2, P, D])\n    extends QueryableById[T1, T2, P, D] {\n  override def queryById(\n    id: T1,\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Stitch[List[T2]] = {\n    embeddingProducer.produceEmbedding(id).flatMap { embeddingOption =>\n      embeddingOption\n        .map { embedding =>\n          Stitch.callFuture(queryable.query(embedding, numOfNeighbors, runtimeParams))\n        }.getOrElse {\n          Stitch.value(List.empty)\n        }\n    }\n  }\n\n  override def queryByIdWithDistance(\n    id: T1,\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Stitch[List[NeighborWithDistance[T2, D]]] = {\n    embeddingProducer.produceEmbedding(id).flatMap { embeddingOption =>\n      embeddingOption\n        .map { embedding =>\n          Stitch.callFuture(queryable.queryWithDistance(embedding, numOfNeighbors, runtimeParams))\n        }.getOrElse {\n          Stitch.value(List.empty)\n        }\n    }\n  }\n\n  override def batchQueryById(\n    ids: Seq[T1],\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Stitch[List[NeighborWithSeed[T1, T2]]] = {\n    Stitch\n      .traverse(ids) { id =>\n        embeddingProducer.produceEmbedding(id).flatMap { embeddingOption =>\n          embeddingOption\n            .map { embedding =>\n              Stitch\n                .callFuture(queryable.query(embedding, numOfNeighbors, runtimeParams)).map(\n                  _.map(neighbor => NeighborWithSeed(id, neighbor)))\n            }.getOrElse {\n              Stitch.value(List.empty)\n            }.handle { case _ => List.empty }\n        }\n      }.map { _.toList.flatten }\n  }\n\n  override def batchQueryWithDistanceById(\n    ids: Seq[T1],\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Stitch[List[NeighborWithDistanceWithSeed[T1, T2, D]]] = {\n    Stitch\n      .traverse(ids) { id =>\n        embeddingProducer.produceEmbedding(id).flatMap { embeddingOption =>\n          embeddingOption\n            .map { embedding =>\n              Stitch\n                .callFuture(queryable.queryWithDistance(embedding, numOfNeighbors, runtimeParams))\n                .map(_.map(neighbor =>\n                  NeighborWithDistanceWithSeed(id, neighbor.neighbor, neighbor.distance)))\n            }.getOrElse {\n              Stitch.value(List.empty)\n            }.handle { case _ => List.empty }\n        }\n      }.map {\n        _.toList.flatten\n      }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/QueryableOperations.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.util.Future\n\nobject QueryableOperations {\n  implicit class Map[T, P <: RuntimeParams, D <: Distance[D]](\n    val q: Queryable[T, P, D]) {\n    def mapRuntimeParameters(f: P => P): Queryable[T, P, D] = {\n      new Queryable[T, P, D] {\n        def query(\n          embedding: EmbeddingVector,\n          numOfNeighbors: Int,\n          runtimeParams: P\n        ): Future[List[T]] = q.query(embedding, numOfNeighbors, f(runtimeParams))\n\n        def queryWithDistance(\n          embedding: EmbeddingVector,\n          numOfNeighbors: Int,\n          runtimeParams: P\n        ): Future[List[NeighborWithDistance[T, D]]] =\n          q.queryWithDistance(embedding, numOfNeighbors, f(runtimeParams))\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/ReadWriteFuturePool.scala",
    "content": "package com.twitter.ann.common\nimport com.google.common.annotations.VisibleForTesting\nimport com.twitter.util.{Future, FuturePool}\n\ntrait ReadWriteFuturePool {\n  def read[T](f: => T): Future[T]\n  def write[T](f: => T): Future[T]\n}\n\nobject ReadWriteFuturePool {\n  def apply(readPool: FuturePool, writePool: FuturePool): ReadWriteFuturePool = {\n    new ReadWriteFuturePoolANN(readPool, writePool)\n  }\n\n  def apply(commonPool: FuturePool): ReadWriteFuturePool = {\n    new ReadWriteFuturePoolANN(commonPool, commonPool)\n  }\n}\n\n@VisibleForTesting\nprivate[ann] class ReadWriteFuturePoolANN(readPool: FuturePool, writePool: FuturePool)\n    extends ReadWriteFuturePool {\n  def read[T](f: => T): Future[T] = {\n    readPool.apply(f)\n  }\n  def write[T](f: => T): Future[T] = {\n    writePool.apply(f)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/Serialization.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.search.common.file.AbstractFile\nimport org.apache.beam.sdk.io.fs.ResourceId\n\n/**\n * Interface for writing an Appendable to a directory.\n */\ntrait Serialization {\n  def toDirectory(\n    serializationDirectory: AbstractFile\n  ): Unit\n\n  def toDirectory(\n    serializationDirectory: ResourceId\n  ): Unit\n}\n\n/**\n * Interface for reading a Queryable from a directory\n * @tparam T the id of the embeddings\n * @tparam Q type of the Queryable that is deserialized.\n */\ntrait QueryableDeserialization[T, P <: RuntimeParams, D <: Distance[D], Q <: Queryable[T, P, D]] {\n  def fromDirectory(\n    serializationDirectory: AbstractFile\n  ): Q\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/ServiceClientQueryable.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.ann.common.EmbeddingType._\nimport com.twitter.ann.common.thriftscala.{\n  NearestNeighborQuery,\n  NearestNeighborResult,\n  Distance => ServiceDistance,\n  RuntimeParams => ServiceRuntimeParams\n}\nimport com.twitter.bijection.Injection\nimport com.twitter.finagle.Service\nimport com.twitter.mediaservices.commons.codec.ArrayByteBufferCodec\nimport com.twitter.util.Future\n\nclass ServiceClientQueryable[T, P <: RuntimeParams, D <: Distance[D]](\n  service: Service[NearestNeighborQuery, NearestNeighborResult],\n  runtimeParamInjection: Injection[P, ServiceRuntimeParams],\n  distanceInjection: Injection[D, ServiceDistance],\n  idInjection: Injection[T, Array[Byte]])\n    extends Queryable[T, P, D] {\n  override def query(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Future[List[T]] = {\n    service\n      .apply(\n        NearestNeighborQuery(\n          embeddingSerDe.toThrift(embedding),\n          withDistance = false,\n          runtimeParamInjection(runtimeParams),\n          numOfNeighbors\n        )\n      )\n      .map { result =>\n        result.nearestNeighbors.map { nearestNeighbor =>\n          idInjection.invert(ArrayByteBufferCodec.decode(nearestNeighbor.id)).get\n        }.toList\n      }\n  }\n\n  override def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Future[List[NeighborWithDistance[T, D]]] =\n    service\n      .apply(\n        NearestNeighborQuery(\n          embeddingSerDe.toThrift(embedding),\n          withDistance = true,\n          runtimeParamInjection(runtimeParams),\n          numOfNeighbors\n        )\n      )\n      .map { result =>\n        result.nearestNeighbors.map { nearestNeighbor =>\n          NeighborWithDistance(\n            idInjection.invert(ArrayByteBufferCodec.decode(nearestNeighbor.id)).get,\n            distanceInjection.invert(nearestNeighbor.distance.get).get\n          )\n        }.toList\n      }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/ShardApi.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.util.Future\nimport scala.util.Random\n\ntrait ShardFunction[T] {\n\n  /**\n   * Shard function to shard embedding based on total shards and embedding data.\n   * @param shards\n   * @param entity\n   * @return Shard index, from 0(Inclusive) to shards(Exclusive))\n   */\n  def apply(shards: Int, entity: EntityEmbedding[T]): Int\n}\n\n/**\n * Randomly shards the embeddings based on number of total shards.\n */\nclass RandomShardFunction[T] extends ShardFunction[T] {\n  def apply(shards: Int, entity: EntityEmbedding[T]): Int = {\n    Random.nextInt(shards)\n  }\n}\n\n/**\n * Sharded appendable to shard the embedding into different appendable indices\n * @param indices: Sequence of appendable indices\n * @param shardFn: Shard function to shard data into different indices\n * @param shards: Total shards\n * @tparam T: Type of id.\n */\nclass ShardedAppendable[T, P <: RuntimeParams, D <: Distance[D]](\n  indices: Seq[Appendable[T, P, D]],\n  shardFn: ShardFunction[T],\n  shards: Int)\n    extends Appendable[T, P, D] {\n  override def append(entity: EntityEmbedding[T]): Future[Unit] = {\n    val shard = shardFn(shards, entity)\n    val index = indices(shard)\n    index.append(entity)\n  }\n\n  override def toQueryable: Queryable[T, P, D] = {\n    new ComposedQueryable[T, P, D](indices.map(_.toQueryable))\n  }\n}\n\n/**\n * Composition of sequence of queryable indices, it queries all the indices,\n * and merges the result in memory to return the K nearest neighbours\n * @param indices: Sequence of queryable indices\n * @tparam T: Type of id\n * @tparam P: Type of runtime param\n * @tparam D: Type of distance metric\n */\nclass ComposedQueryable[T, P <: RuntimeParams, D <: Distance[D]](\n  indices: Seq[Queryable[T, P, D]])\n    extends Queryable[T, P, D] {\n  private[this] val ordering =\n    Ordering.by[NeighborWithDistance[T, D], D](_.distance)\n  override def query(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Future[List[T]] = {\n    val neighbours = queryWithDistance(embedding, numOfNeighbors, runtimeParams)\n    neighbours.map(list => list.map(nn => nn.neighbor))\n  }\n\n  override def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Future[List[NeighborWithDistance[T, D]]] = {\n    val futures = Future.collect(\n      indices.map(index => index.queryWithDistance(embedding, numOfNeighbors, runtimeParams))\n    )\n    futures.map { list =>\n      list.flatten\n        .sorted(ordering)\n        .take(numOfNeighbors)\n        .toList\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/ShardedSerialization.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.search.common.file.AbstractFile.Filter\nimport com.twitter.util.Future\nimport org.apache.beam.sdk.io.fs.ResourceId\nimport scala.collection.JavaConverters._\n\nobject ShardConstants {\n  val ShardPrefix = \"shard_\"\n}\n\n/**\n * Serialize shards to directory\n * @param shards: List of shards to serialize\n */\nclass ShardedSerialization(\n  shards: Seq[Serialization])\n    extends Serialization {\n  override def toDirectory(directory: AbstractFile): Unit = {\n    toDirectory(new IndexOutputFile(directory))\n  }\n\n  override def toDirectory(directory: ResourceId): Unit = {\n    toDirectory(new IndexOutputFile(directory))\n  }\n\n  private def toDirectory(directory: IndexOutputFile): Unit = {\n    shards.indices.foreach { shardId =>\n      val shardDirectory = directory.createDirectory(ShardConstants.ShardPrefix + shardId)\n      val serialization = shards(shardId)\n      if (shardDirectory.isAbstractFile) {\n        serialization.toDirectory(shardDirectory.abstractFile)\n      } else {\n        serialization.toDirectory(shardDirectory.resourceId)\n      }\n    }\n  }\n}\n\n/**\n * Deserialize directories containing index shards data to a composed queryable\n * @param deserializationFn function to deserialize a shard file to Queryable\n * @tparam T the id of the embeddings\n * @tparam P : Runtime params type\n * @tparam D: Distance metric type\n */\nclass ComposedQueryableDeserialization[T, P <: RuntimeParams, D <: Distance[D]](\n  deserializationFn: (AbstractFile) => Queryable[T, P, D])\n    extends QueryableDeserialization[T, P, D, Queryable[T, P, D]] {\n  override def fromDirectory(directory: AbstractFile): Queryable[T, P, D] = {\n    val shardDirs = directory\n      .listFiles(new Filter {\n        override def accept(file: AbstractFile): Boolean =\n          file.getName.startsWith(ShardConstants.ShardPrefix)\n      })\n      .asScala\n      .toList\n\n    val indices = shardDirs\n      .map { shardDir =>\n        deserializationFn(shardDir)\n      }\n\n    new ComposedQueryable[T, P, D](indices)\n  }\n}\n\nclass ShardedIndexBuilderWithSerialization[T, P <: RuntimeParams, D <: Distance[D]](\n  shardedIndex: ShardedAppendable[T, P, D],\n  shardedSerialization: ShardedSerialization)\n    extends Appendable[T, P, D]\n    with Serialization {\n  override def append(entity: EntityEmbedding[T]): Future[Unit] = {\n    shardedIndex.append(entity)\n  }\n\n  override def toDirectory(directory: AbstractFile): Unit = {\n    shardedSerialization.toDirectory(directory)\n  }\n\n  override def toDirectory(directory: ResourceId): Unit = {\n    shardedSerialization.toDirectory(directory)\n  }\n\n  override def toQueryable: Queryable[T, P, D] = {\n    shardedIndex.toQueryable\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/common/Task.scala",
    "content": "package com.twitter.ann.common\n\nimport com.twitter.finagle.stats.CategorizingExceptionStatsHandler\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.tracing.DefaultTracer\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.finagle.util.Rng\nimport com.twitter.inject.logging.MDCKeys\nimport com.twitter.util.Closable\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport com.twitter.util.Timer\nimport com.twitter.util.logging.Logging\nimport java.util.concurrent.atomic.AtomicInteger\nimport org.slf4j.MDC\n\n/**\n * A Task that will be scheduled to execute periodically on every interval. If a task takes\n * longer than an interval to complete, it will be immediately scheduled to run.\n */\ntrait Task extends Closable { self: Logging =>\n\n  // Exposed if the implementation of `task` need to report failures\n  val exnStatsHandler = new CategorizingExceptionStatsHandler(categorizer = _ => Some(\"failures\"))\n\n  protected val statsReceiver: StatsReceiver\n  private val totalTasks = statsReceiver.counter(\"total\")\n  private val successfulTasks = statsReceiver.counter(\"success\")\n  private val taskLatency = statsReceiver.stat(\"latency_ms\")\n\n  private val activeTasks = new AtomicInteger(0)\n\n  protected[common] val rng: Rng = Rng.threadLocal\n  protected[common] val timer: Timer = DefaultTimer\n\n  @volatile private var taskLoop: Future[Unit] = null\n\n  /** Execute the task wih bookkeeping **/\n  private def run(): Future[Unit] = {\n    totalTasks.incr()\n    activeTasks.getAndIncrement()\n\n    val start = Time.now\n    val runningTask =\n      // Setup a new trace root for this task. We also want logs to contain\n      // the same trace information finatra populates for requests.\n      // See com.twitter.finatra.thrift.filters.TraceIdMDCFilter\n      Trace.letTracerAndNextId(DefaultTracer) {\n        val trace = Trace()\n        MDC.put(MDCKeys.TraceId, trace.id.traceId.toString)\n        MDC.put(MDCKeys.TraceSampled, trace.id._sampled.getOrElse(false).toString)\n        MDC.put(MDCKeys.TraceSpanId, trace.id.spanId.toString)\n\n        info(s\"starting task ${getClass.toString}\")\n        task()\n          .onSuccess({ _ =>\n            info(s\"completed task ${getClass.toString}\")\n            successfulTasks.incr()\n          })\n          .onFailure({ e =>\n            warn(s\"failed task. \", e)\n            exnStatsHandler.record(statsReceiver, e)\n          })\n      }\n\n    runningTask.transform { _ =>\n      val elapsed = Time.now - start\n      activeTasks.getAndDecrement()\n      taskLatency.add(elapsed.inMilliseconds)\n\n      Future\n        .sleep(taskInterval)(timer)\n        .before(run())\n    }\n  }\n\n  // Body of a task to run\n  protected def task(): Future[Unit]\n\n  // Task interval\n  protected def taskInterval: Duration\n\n  /**\n   * Start the task after random jitter\n   */\n  final def jitteredStart(): Unit = synchronized {\n    if (taskLoop != null) {\n      throw new RuntimeException(s\"task already started\")\n    } else {\n      val jitterNs = rng.nextLong(taskInterval.inNanoseconds)\n      val jitter = Duration.fromNanoseconds(jitterNs)\n\n      taskLoop = Future\n        .sleep(jitter)(timer)\n        .before(run())\n    }\n  }\n\n  /**\n   * Start the task without applying any delay\n   */\n  final def startImmediately(): Unit = synchronized {\n    if (taskLoop != null) {\n      throw new RuntimeException(s\"task already started\")\n    } else {\n      taskLoop = run()\n    }\n  }\n\n  /**\n   * Close the task. A closed task cannot be restarted.\n   */\n  override def close(deadline: Time): Future[Unit] = {\n    if (taskLoop != null) {\n      taskLoop.raise(new InterruptedException(\"task closed\"))\n    }\n    Future.Done\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/dataflow/offline/ANNIndexBuilderBeamJob.scala",
    "content": "package com.twitter.ann.dataflow.offline\n\nimport com.spotify.scio.ScioContext\nimport com.spotify.scio.ScioMetrics\nimport com.twitter.ann.annoy.TypedAnnoyIndex\nimport com.twitter.ann.brute_force.SerializableBruteForceIndex\nimport com.twitter.ann.common.thriftscala.AnnIndexMetadata\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.Cosine\nimport com.twitter.ann.common.EntityEmbedding\nimport com.twitter.ann.common.IndexOutputFile\nimport com.twitter.ann.common.Metric\nimport com.twitter.ann.common.ReadWriteFuturePool\nimport com.twitter.ann.faiss.FaissIndexer\nimport com.twitter.ann.hnsw.TypedHnswIndex\nimport com.twitter.ann.serialization.PersistedEmbeddingInjection\nimport com.twitter.ann.serialization.ThriftIteratorIO\nimport com.twitter.ann.serialization.thriftscala.PersistedEmbedding\nimport com.twitter.ann.util.IndexBuilderUtils\nimport com.twitter.beam.io.bigquery.BigQueryIO\nimport com.twitter.beam.io.dal.DalObservedDatasetRegistration\nimport com.twitter.beam.job.DateRange\nimport com.twitter.beam.job.DateRangeOptions\nimport com.twitter.cortex.ml.embeddings.common._\nimport com.twitter.ml.api.embedding.Embedding\nimport com.twitter.ml.api.embedding.EmbeddingMath\nimport com.twitter.ml.api.embedding.EmbeddingSerDe\nimport com.twitter.ml.api.thriftscala.{Embedding => TEmbedding}\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.SemanticCoreId\nimport com.twitter.ml.featurestore.lib.TfwId\nimport com.twitter.ml.featurestore.lib.TweetId\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.scalding.DateOps\nimport com.twitter.scalding.RichDate\nimport com.twitter.scio_internal.job.ScioBeamJob\nimport com.twitter.statebird.v2.thriftscala.{Environment => StatebirdEnvironment}\nimport com.twitter.util.Await\nimport com.twitter.util.FuturePool\nimport com.twitter.wtf.beam.bq_embedding_export.BQQueryUtils\nimport java.time.Instant\nimport java.util.TimeZone\nimport java.util.concurrent.Executors\nimport org.apache.beam.sdk.io.FileSystems\nimport org.apache.beam.sdk.io.fs.ResolveOptions\nimport org.apache.beam.sdk.io.fs.ResourceId\nimport org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead\nimport org.apache.beam.sdk.options.Default\nimport org.apache.beam.sdk.options.Description\nimport org.apache.beam.sdk.transforms.DoFn\nimport org.apache.beam.sdk.transforms.DoFn._\nimport org.apache.beam.sdk.transforms.PTransform\nimport org.apache.beam.sdk.transforms.ParDo\nimport org.apache.beam.sdk.values.KV\nimport org.apache.beam.sdk.values.PCollection\nimport org.apache.beam.sdk.values.PDone\nimport org.slf4j.Logger\nimport org.slf4j.LoggerFactory\n\ntrait ANNOptions extends DateRangeOptions {\n  @Description(\"Output GCS path for the generated index\")\n  def getOutputPath(): String\n  def setOutputPath(value: String): Unit\n\n  @Description(\"If set, the index is grouped\")\n  @Default.Boolean(false)\n  def getGrouped: Boolean\n  def setGrouped(value: Boolean): Unit\n\n  @Description(\n    \"If set, a segment will be registered for the provided DAL dataset module which will trigger \" +\n      \"DAL registration.\")\n  @Default.Boolean(false)\n  def getEnableDalRegistration: Boolean\n  def setEnableDalRegistration(value: Boolean): Unit\n\n  @Description(\n    \"Output GCS path for the generated index. The OutputPath should be of the format \" +\n      \"'gs://user.{{user_name}}.dp.gcp.twttr.net/subDir/outputDir' and OutputDALPath will be \" +\n      \"'subDir/outputDir' for this to work\")\n  def getOutputDALPath: String\n  def setOutputDALPath(value: String): Unit\n\n  @Description(\"Get ANN index dataset name\")\n  def getDatasetModuleName: String\n  def setDatasetModuleName(value: String): Unit\n\n  @Description(\"Get ANN index dataset owner role\")\n  def getDatasetOwnerRole: String\n  def setDatasetOwnerRole(value: String): Unit\n\n  @Description(\"If set, index is written in <output>/<timestamp>\")\n  @Default.Boolean(false)\n  def getOutputWithTimestamp: Boolean\n  def setOutputWithTimestamp(value: Boolean): Unit\n\n  @Description(\"File which contains a SQL query to retrieve embeddings from BQ\")\n  def getDatasetSqlPath: String\n  def setDatasetSqlPath(value: String): Unit\n\n  @Description(\"Dimension of embedding in the input data. See go/ann\")\n  def getDimension: Int\n  def setDimension(value: Int): Unit\n\n  @Description(\"The type of entity ID that is used with the embeddings. See go/ann\")\n  def getEntityKind: String\n  def setEntityKind(value: String): Unit\n\n  @Description(\"The kind of index you want to generate (HNSW/Annoy/Brute Force/faiss). See go/ann\")\n  def getAlgo: String\n  def setAlgo(value: String): Unit\n\n  @Description(\"Distance metric (InnerProduct/Cosine/L2). See go/ann\")\n  def getMetric: String\n  def setMetric(value: String): Unit\n\n  @Description(\"Specifies how many parallel inserts happen to the index. See go/ann\")\n  def getConcurrencyLevel: Int\n  def setConcurrencyLevel(value: Int): Unit\n\n  @Description(\n    \"Used by HNSW algo. Larger value increases build time but will give better recall. See go/ann\")\n  def getEfConstruction: Int\n  def setEfConstruction(value: Int): Unit\n\n  @Description(\n    \"Used by HNSW algo. Larger value increases the index size but will give better recall. \" +\n      \"See go/ann\")\n  def getMaxM: Int\n  def setMaxM(value: Int): Unit\n\n  @Description(\"Used by HNSW algo. Approximate number of elements that will be indexed. See go/ann\")\n  def getExpectedElements: Int\n  def setExpectedElements(value: Int): Unit\n\n  @Description(\n    \"Used by Annoy. num_trees is provided during build time and affects the build time and the \" +\n      \"index size. A larger value will give more accurate results, but larger indexes. See go/ann\")\n  def getAnnoyNumTrees: Int\n  def setAnnoyNumTrees(value: Int): Unit\n\n  @Description(\n    \"FAISS factory string determines the ANN algorithm and compression. \" +\n      \"See https://github.com/facebookresearch/faiss/wiki/The-index-factory\")\n  def getFAISSFactoryString: String\n  def setFAISSFactoryString(value: String): Unit\n\n  @Description(\"Sample rate for training during creation of FAISS index. Default is 0.05f\")\n  @Default.Float(0.05f)\n  def getTrainingSampleRate: Float\n  def setTrainingSampleRate(value: Float): Unit\n}\n\n/**\n * Builds ANN index.\n *\n * The input embeddings are read from BigQuery using the input SQL query. The output from this SQL\n * query needs to have two columns, \"entityID\" [Long] and \"embedding\" [List[Double]]\n *\n * Output directory supported is GCS bucket\n */\nobject ANNIndexBuilderBeamJob extends ScioBeamJob[ANNOptions] {\n  val counterNameSpace = \"ANNIndexBuilderBeamJob\"\n  val LOG: Logger = LoggerFactory.getLogger(this.getClass)\n  implicit val timeZone: TimeZone = DateOps.UTC\n\n  def configurePipeline(sc: ScioContext, opts: ANNOptions): Unit = {\n    val startDate: RichDate = RichDate(opts.interval.getStart.toDate)\n    val endDate: RichDate = RichDate(opts.interval.getEnd.toDate)\n    val instant = Instant.now()\n    val out = {\n      val base = FileSystems.matchNewResource(opts.getOutputPath, /*isDirectory=*/ true)\n      if (opts.getOutputWithTimestamp) {\n        base.resolve(\n          instant.toEpochMilli.toString,\n          ResolveOptions.StandardResolveOptions.RESOLVE_DIRECTORY)\n      } else {\n        base\n      }\n    }\n\n    // Define template variables which we would like to be replaced in the corresponding sql file\n    val templateVariables =\n      Map(\n        \"START_DATE\" -> startDate.toString(DateOps.DATETIME_HMS_WITH_DASH),\n        \"END_DATE\" -> endDate.toString(DateOps.DATETIME_HMS_WITH_DASH)\n      )\n\n    val embeddingFetchQuery =\n      BQQueryUtils.getBQQueryFromSqlFile(opts.getDatasetSqlPath, templateVariables)\n\n    val sCollection = if (opts.getGrouped) {\n      sc.customInput(\n        \"Read grouped data from BQ\",\n        BigQueryIO\n          .readClass[GroupedEmbeddingData]()\n          .fromQuery(embeddingFetchQuery).usingStandardSql()\n          .withMethod(TypedRead.Method.DIRECT_READ)\n      )\n    } else {\n      sc.customInput(\n        \"Read flat data from BQ\",\n        BigQueryIO\n          .readClass[FlatEmbeddingData]().fromQuery(embeddingFetchQuery).usingStandardSql()\n          .withMethod(TypedRead.Method.DIRECT_READ)\n      )\n    }\n\n    val processedCollection =\n      sCollection\n        .flatMap(transformTableRowToKeyVal)\n        .groupBy(_.getKey)\n        .map {\n          case (groupName, groupValue) =>\n            Map(groupName -> groupValue.map(_.getValue))\n        }\n\n    val annIndexMetadata =\n      AnnIndexMetadata(timestamp = Some(instant.getEpochSecond), withGroups = Some(opts.getGrouped))\n\n    // Count the number of groups and output the ANN index metadata\n    processedCollection.count.map(count => {\n      val annGroupedIndexMetadata = annIndexMetadata.copy(\n        numGroups = Some(count.intValue())\n      )\n      val indexOutDir = new IndexOutputFile(out)\n      indexOutDir.writeIndexMetadata(annGroupedIndexMetadata)\n    })\n\n    // Generate Index\n    processedCollection.saveAsCustomOutput(\n      \"Serialise to Disk\",\n      OutputSink(\n        out,\n        opts.getAlgo.equals(\"faiss\"),\n        opts.getOutputDALPath,\n        opts.getEnableDalRegistration,\n        opts.getDatasetModuleName,\n        opts.getDatasetOwnerRole,\n        instant,\n        opts.getDate(),\n        counterNameSpace\n      )\n    )\n  }\n\n  def transformTableRowToKeyVal(\n    data: BaseEmbeddingData\n  ): Option[KV[String, KV[Long, TEmbedding]]] = {\n    val transformTable = ScioMetrics.counter(counterNameSpace, \"transform_table_row_to_kv\")\n    for {\n      id <- data.entityId\n    } yield {\n      transformTable.inc()\n      val groupName: String = if (data.isInstanceOf[GroupedEmbeddingData]) {\n        (data.asInstanceOf[GroupedEmbeddingData]).groupId.get\n      } else {\n        \"\"\n      }\n\n      KV.of[String, KV[Long, TEmbedding]](\n        groupName,\n        KV.of[Long, TEmbedding](\n          id,\n          EmbeddingSerDe.toThrift(Embedding(data.embedding.map(_.toFloat).toArray)))\n      )\n    }\n  }\n\n  case class OutputSink(\n    outDir: ResourceId,\n    isFaiss: Boolean,\n    outputDALPath: String,\n    enableDalRegistration: Boolean,\n    datasetModuleName: String,\n    datasetOwnerRole: String,\n    instant: Instant,\n    date: DateRange,\n    counterNameSpace: String)\n      extends PTransform[PCollection[Map[String, Iterable[KV[Long, TEmbedding]]]], PDone] {\n    override def expand(input: PCollection[Map[String, Iterable[KV[Long, TEmbedding]]]]): PDone = {\n      PDone.in {\n        val dummyOutput = {\n          if (isFaiss) {\n            input\n              .apply(\n                \"Build&WriteFaissANNIndex\",\n                ParDo.of(new BuildFaissANNIndex(outDir, counterNameSpace))\n              )\n          } else {\n            input\n              .apply(\n                \"Build&WriteANNIndex\",\n                ParDo.of(new BuildANNIndex(outDir, counterNameSpace))\n              )\n          }\n        }\n\n        if (enableDalRegistration) {\n          input\n            .apply(\n              \"Register DAL Dataset\",\n              DalObservedDatasetRegistration(\n                datasetModuleName,\n                datasetOwnerRole,\n                outputDALPath,\n                instant,\n                Some(StatebirdEnvironment.Prod),\n                Some(\"ANN Index Data Files\"))\n            )\n            .getPipeline\n        } else {\n          dummyOutput.getPipeline\n        }\n      }\n    }\n  }\n\n  class BuildANNIndex(outDir: ResourceId, counterNameSpace: String)\n      extends DoFn[Map[String, Iterable[KV[Long, TEmbedding]]], Unit] {\n\n    def transformKeyValToEmbeddingWithEntity[T <: EntityId](\n      entityKind: EntityKind[T]\n    )(\n      keyVal: KV[Long, TEmbedding]\n    ): EntityEmbedding[T] = {\n      val entityId = entityKind match {\n        case UserKind => UserId(keyVal.getKey).toThrift\n        case TweetKind => TweetId(keyVal.getKey).toThrift\n        case TfwKind => TfwId(keyVal.getKey).toThrift\n        case SemanticCoreKind => SemanticCoreId(keyVal.getKey).toThrift\n        case _ => throw new IllegalArgumentException(s\"Unsupported embedding kind: $entityKind\")\n      }\n      EntityEmbedding[T](\n        EntityId.fromThrift(entityId).asInstanceOf[T],\n        EmbeddingSerDe.fromThrift(keyVal.getValue))\n    }\n\n    @ProcessElement\n    def processElement[T <: EntityId, D <: Distance[D]](\n      @Element dataGrouped: Map[String, Iterable[KV[Long, TEmbedding]]],\n      context: ProcessContext\n    ): Unit = {\n      val opts = context.getPipelineOptions.as(classOf[ANNOptions])\n      val uncastEntityKind = EntityKind.getEntityKind(opts.getEntityKind)\n      val entityKind = uncastEntityKind.asInstanceOf[EntityKind[T]]\n      val transformKVtoEmbeddings =\n        ScioMetrics.counter(counterNameSpace, \"transform_kv_to_embeddings\")\n\n      val _ = dataGrouped.map {\n        case (groupName, data) =>\n          val annEmbeddings = data.map { kv =>\n            transformKVtoEmbeddings.inc()\n            transformKeyValToEmbeddingWithEntity(entityKind)(kv)\n          }\n\n          val out = {\n            if (opts.getGrouped && groupName != \"\") {\n              outDir.resolve(groupName, ResolveOptions.StandardResolveOptions.RESOLVE_DIRECTORY)\n            } else {\n              outDir\n            }\n          }\n          LOG.info(s\"Writing output to ${out}\")\n\n          val metric = Metric.fromString(opts.getMetric).asInstanceOf[Metric[D]]\n          val concurrencyLevel = opts.getConcurrencyLevel\n          val dimension = opts.getDimension\n          val threadPool = Executors.newFixedThreadPool(concurrencyLevel)\n\n          LOG.info(s\"Building ANN index of type ${opts.getAlgo}\")\n          val serialization = opts.getAlgo match {\n            case \"brute_force\" =>\n              val PersistedEmbeddingIO =\n                new ThriftIteratorIO[PersistedEmbedding](PersistedEmbedding)\n              SerializableBruteForceIndex(\n                metric,\n                FuturePool.apply(threadPool),\n                new PersistedEmbeddingInjection(entityKind.byteInjection),\n                PersistedEmbeddingIO\n              )\n            case \"annoy\" =>\n              TypedAnnoyIndex.indexBuilder(\n                dimension,\n                opts.getAnnoyNumTrees,\n                metric,\n                entityKind.byteInjection,\n                FuturePool.apply(threadPool)\n              )\n            case \"hnsw\" =>\n              val efConstruction = opts.getEfConstruction\n              val maxM = opts.getMaxM\n              val expectedElements = opts.getExpectedElements\n              TypedHnswIndex.serializableIndex(\n                dimension,\n                metric,\n                efConstruction,\n                maxM,\n                expectedElements,\n                entityKind.byteInjection,\n                ReadWriteFuturePool(FuturePool.apply(threadPool))\n              )\n          }\n\n          val future =\n            IndexBuilderUtils.addToIndex(serialization, annEmbeddings.toSeq, concurrencyLevel)\n          Await.result(future.map { _ =>\n            serialization.toDirectory(out)\n          })\n      }\n    }\n  }\n\n  class BuildFaissANNIndex(outDir: ResourceId, counterNameSpace: String)\n      extends DoFn[Map[String, Iterable[KV[Long, TEmbedding]]], Unit] {\n\n    @ProcessElement\n    def processElement[D <: Distance[D]](\n      @Element dataGrouped: Map[String, Iterable[KV[Long, TEmbedding]]],\n      context: ProcessContext\n    ): Unit = {\n      val opts = context.getPipelineOptions.as(classOf[ANNOptions])\n      val transformKVtoEmbeddings =\n        ScioMetrics.counter(counterNameSpace, \"transform_kv_to_embeddings\")\n\n      val _ = dataGrouped.map {\n        case (groupName, data) =>\n          val out = {\n            if (opts.getGrouped && groupName != \"\") {\n              outDir.resolve(groupName, ResolveOptions.StandardResolveOptions.RESOLVE_DIRECTORY)\n            } else {\n              outDir\n            }\n          }\n          LOG.info(s\"Writing output to ${out}\")\n\n          val metric = Metric.fromString(opts.getMetric).asInstanceOf[Metric[D]]\n          val maybeNormalizedPipe = data.map { kv =>\n            transformKVtoEmbeddings.inc()\n            val embedding = EmbeddingSerDe.floatEmbeddingSerDe.fromThrift(kv.getValue)\n            EntityEmbedding[Long](\n              kv.getKey,\n              if (metric == Cosine) {\n                EmbeddingMath.Float.normalize(embedding)\n              } else {\n                embedding\n              }\n            )\n          }\n\n          // Generate Index\n          FaissIndexer.buildAndWriteFaissIndex(\n            maybeNormalizedPipe,\n            opts.getTrainingSampleRate,\n            opts.getFAISSFactoryString,\n            metric,\n            new IndexOutputFile(out))\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/dataflow/offline/BUILD",
    "content": "scala_library(\n    name = \"index_builder_lib\",\n    sources = [\n        \"*.scala\",\n    ],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/spotify:scio-core\",\n        \"3rdparty/jvm/org/apache/beam:beam-sdks-java-core\",\n        \"ann/src/main/java/com/twitter/ann/faiss\",\n        \"ann/src/main/scala/com/twitter/ann/annoy\",\n        \"ann/src/main/scala/com/twitter/ann/brute_force\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/faiss\",\n        \"ann/src/main/scala/com/twitter/ann/hnsw\",\n        \"ann/src/main/scala/com/twitter/ann/serialization\",\n        \"ann/src/main/scala/com/twitter/ann/util\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"beam-internal/src/main/scala/com/twitter/beam/io/bigquery\",\n        \"beam-internal/src/main/scala/com/twitter/beam/io/dal\",\n        \"beam-internal/src/main/scala/com/twitter/beam/job\",\n        \"beam-internal/src/main/scala/com/twitter/scio_internal/runner/dataflow\",\n        \"src/scala/com/twitter/cortex/ml/embeddings/common:Helpers\",\n        \"src/scala/com/twitter/ml/featurestore/lib\",\n        \"src/scala/com/twitter/wtf/beam/bq_embedding_export:bq_embedding_export_lib\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/dataflow/offline/BaseEmbeddingData.scala",
    "content": "package com.twitter.ann.dataflow.offline\n\ntrait BaseEmbeddingData {\n  val entityId: Option[Long]\n  val embedding: Seq[Double]\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/dataflow/offline/FlatEmbeddingData.scala",
    "content": "package com.twitter.ann.dataflow.offline\n\nimport com.twitter.beam.schemas.SchemaFieldName\n\ncase class FlatEmbeddingData(\n  @SchemaFieldName(\"entityId\") entityId: Option[Long],\n  @SchemaFieldName(\"embedding\") embedding: Seq[Double])\n    extends BaseEmbeddingData\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/dataflow/offline/GroupedEmbeddingData.scala",
    "content": "package com.twitter.ann.dataflow.offline\n\nimport com.twitter.beam.schemas.SchemaFieldName\n\ncase class GroupedEmbeddingData(\n  @SchemaFieldName(\"entityId\") entityId: Option[Long],\n  @SchemaFieldName(\"embedding\") embedding: Seq[Double],\n  @SchemaFieldName(\"groupId\") groupId: Option[String],\n) extends BaseEmbeddingData\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/experimental/BUILD.bazel",
    "content": "scala_library(\n    name = \"server\",\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-only\"],\n    dependencies = [\n        \"ann/src/main/scala/com/twitter/ann/annoy\",\n        \"ann/src/main/scala/com/twitter/ann/brute_force\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/hnsw\",\n    ],\n)\n\nhadoop_binary(\n    name = \"benchmarking\",\n    basename = \"benchmarking\",\n    main = \"com.twitter.ann.experimental.Runner\",\n    platform = \"java8\",\n    runtime_platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-compatible:migrated\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \":server\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/experimental/Runner.scala",
    "content": "package com.twitter.ann.experimental\n\nimport com.twitter.ann.annoy.{AnnoyRuntimeParams, TypedAnnoyIndex}\nimport com.twitter.ann.brute_force.{BruteForceIndex, BruteForceRuntimeParams}\nimport com.twitter.ann.common.{Cosine, CosineDistance, EntityEmbedding, ReadWriteFuturePool}\nimport com.twitter.ann.hnsw.{HnswParams, TypedHnswIndex}\nimport com.twitter.bijection.Injection\nimport com.twitter.ml.api.embedding.Embedding\nimport com.twitter.search.common.file.LocalFile\nimport com.twitter.util.{Await, Future, FuturePool}\nimport java.nio.file.Files\nimport java.util\nimport java.util.concurrent.Executors\nimport java.util.{Collections, Random}\nimport scala.collection.JavaConverters._\nimport scala.collection.mutable\n\nobject Runner {\n  def main(args: Array[String]): Unit = {\n    val rng = new Random()\n    val dimen = 300\n    val neighbours = 20\n    val trainDataSetSize = 2000\n    val testDataSetSize = 30\n\n    // Hnsw (ef -> (time, recall))\n    val hnswEfConfig = new mutable.HashMap[Int, (Float, Float)]\n    val efConstruction = 200\n    val maxM = 16\n    val threads = 24\n    val efSearch =\n      Seq(20, 30, 50, 70, 100, 120)\n    efSearch.foreach(hnswEfConfig.put(_, (0.0f, 0.0f)))\n\n    // Annoy (nodes to explore -> (time, recall))\n    val numOfTrees = 80\n    val annoyConfig = new mutable.HashMap[Int, (Float, Float)]\n    val nodesToExplore = Seq(0, 2000, 3000, 5000, 7000, 10000, 15000, 20000,\n      30000, 35000, 40000, 50000)\n    nodesToExplore.foreach(annoyConfig.put(_, (0.0f, 0.0f)))\n    val injection = Injection.int2BigEndian\n    val distance = Cosine\n    val exec = Executors.newFixedThreadPool(threads)\n    val pool = FuturePool.apply(exec)\n    val hnswMultiThread =\n      TypedHnswIndex.index[Int, CosineDistance](\n        dimen,\n        distance,\n        efConstruction = efConstruction,\n        maxM = maxM,\n        trainDataSetSize,\n        ReadWriteFuturePool(pool)\n      )\n\n    val bruteforce = BruteForceIndex[Int, CosineDistance](distance, pool)\n    val annoyBuilder =\n      TypedAnnoyIndex.indexBuilder(dimen, numOfTrees, distance, injection, FuturePool.immediatePool)\n    val temp = new LocalFile(Files.createTempDirectory(\"test\").toFile)\n\n    println(\"Creating bruteforce.........\")\n    val data =\n      Collections.synchronizedList(new util.ArrayList[EntityEmbedding[Int]]())\n    val bruteforceFutures = 1 to trainDataSetSize map { id =>\n      val vec = Array.fill(dimen)(rng.nextFloat() * 50)\n      val emb = EntityEmbedding[Int](id, Embedding(vec))\n      data.add(emb)\n      bruteforce.append(emb)\n    }\n\n    Await.result(Future.collect(bruteforceFutures))\n\n    println(\"Creating hnsw multithread test.........\")\n    val (_, multiThreadInsertion) = time {\n      Await.result(Future.collect(data.asScala.toList.map { emb =>\n        hnswMultiThread.append(emb)\n      }))\n    }\n\n    println(\"Creating annoy.........\")\n    val (_, annoyTime) = time {\n      Await.result(Future.collect(data.asScala.toList.map(emb =>\n        annoyBuilder.append(emb))))\n      annoyBuilder.toDirectory(temp)\n    }\n\n    val annoyQuery = TypedAnnoyIndex.loadQueryableIndex(\n      dimen,\n      Cosine,\n      injection,\n      FuturePool.immediatePool,\n      temp\n    )\n\n    val hnswQueryable = hnswMultiThread.toQueryable\n\n    println(s\"Total train size : $trainDataSetSize\")\n    println(s\"Total querySize : $testDataSetSize\")\n    println(s\"Dimension : $dimen\")\n    println(s\"Distance type : $distance\")\n    println(s\"Annoy index creation time trees: $numOfTrees => $annoyTime ms\")\n    println(\n      s\"Hnsw multi thread creation time : $multiThreadInsertion ms efCons: $efConstruction maxM $maxM thread : $threads\")\n    println(\"Querying.........\")\n    var bruteForceTime = 0.0f\n    1 to testDataSetSize foreach { id =>\n      println(\"Querying id \" + id)\n      val embedding = Embedding(Array.fill(dimen)(rng.nextFloat()))\n\n      val (list, timeTakenB) =\n        time(\n          Await\n            .result(\n              bruteforce.query(embedding, neighbours, BruteForceRuntimeParams))\n            .toSet)\n      bruteForceTime += timeTakenB\n\n      val annoyConfigCopy = annoyConfig.toMap\n      val hnswEfConfigCopy = hnswEfConfig.toMap\n\n      hnswEfConfigCopy.keys.foreach { ef =>\n        val (nn, timeTaken) =\n          time(Await\n            .result(hnswQueryable.query(embedding, neighbours, HnswParams(ef)))\n            .toSet)\n        val recall = (list.intersect(nn).size) * 1.0f / neighbours\n        val (oldTime, oldRecall) = hnswEfConfig(ef)\n        hnswEfConfig.put(ef, (oldTime + timeTaken, oldRecall + recall))\n      }\n\n      annoyConfigCopy.keys.foreach { nodes =>\n        val (nn, timeTaken) =\n          time(\n            Await.result(\n              annoyQuery\n                .query(embedding,\n                  neighbours,\n                  AnnoyRuntimeParams(nodesToExplore = Some(nodes)))\n                .map(_.toSet)))\n        val recall = (list.intersect(nn).size) * 1.0f / neighbours\n        val (oldTime, oldRecall) = annoyConfig(nodes)\n        annoyConfig.put(nodes, (oldTime + timeTaken, oldRecall + recall))\n      }\n    }\n\n    println(\n      s\"Bruteforce avg query time : ${bruteForceTime / testDataSetSize} ms\")\n\n    efSearch.foreach { ef =>\n      val data = hnswEfConfig(ef)\n      println(\n        s\"Hnsw avg recall and time with query ef : $ef => ${data._2 / testDataSetSize} ${data._1 / testDataSetSize} ms\"\n      )\n    }\n\n    nodesToExplore.foreach { n =>\n      val data = annoyConfig(n)\n      println(\n        s\"Annoy avg recall and time with nodes_to_explore :  $n => ${data._2 / testDataSetSize} ${data._1 / testDataSetSize} ms\"\n      )\n    }\n\n    exec.shutdown()\n  }\n\n  def time[T](fn: => T): (T, Long) = {\n    val start = System.currentTimeMillis()\n    val result = fn\n    val end = System.currentTimeMillis()\n    (result, (end - start))\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/faiss/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/org/mapdb\",\n        \"ann/src/main/java/com/twitter/ann/faiss\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/serialization\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"mediaservices/commons/src/main/scala:futuretracker\",\n        \"src/java/com/twitter/common_internal/hadoop\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/scala/com/twitter/ml/api/embedding\",\n    ],\n    exports = [\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"src/java/com/twitter/common_internal/hadoop\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/scala/com/twitter/ml/api/embedding\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/faiss/FaissCommon.scala",
    "content": "package com.twitter.ann.faiss\n\nimport com.twitter.ann.common.thriftscala.FaissRuntimeParam\nimport com.twitter.bijection.Injection\nimport scala.util.Failure\nimport scala.util.Success\nimport scala.util.Try\nimport com.twitter.ann.common.thriftscala.{RuntimeParams => ServiceRuntimeParams}\nimport com.twitter.search.common.file.AbstractFile\n\nobject FaissCommon {\n  val RuntimeParamsInjection: Injection[FaissParams, ServiceRuntimeParams] =\n    new Injection[FaissParams, ServiceRuntimeParams] {\n      override def apply(scalaParams: FaissParams): ServiceRuntimeParams = {\n        ServiceRuntimeParams.FaissParam(\n          FaissRuntimeParam(\n            scalaParams.nprobe,\n            scalaParams.quantizerEf,\n            scalaParams.quantizerKFactorRF,\n            scalaParams.quantizerNprobe,\n            scalaParams.ht)\n        )\n      }\n\n      override def invert(thriftParams: ServiceRuntimeParams): Try[FaissParams] =\n        thriftParams match {\n          case ServiceRuntimeParams.FaissParam(faissParam) =>\n            Success(\n              FaissParams(\n                faissParam.nprobe,\n                faissParam.quantizerEf,\n                faissParam.quantizerKfactorRf,\n                faissParam.quantizerNprobe,\n                faissParam.ht))\n          case p => Failure(new IllegalArgumentException(s\"Expected FaissParams got $p\"))\n        }\n    }\n\n  def isValidFaissIndex(path: AbstractFile): Boolean = {\n    path.isDirectory &&\n    path.hasSuccessFile &&\n    path.getChild(\"faiss.index\").exists()\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/faiss/FaissIndex.scala",
    "content": "package com.twitter.ann.faiss\n\nimport com.twitter.ann.common.Queryable\nimport com.twitter.ann.common._\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.util.logging.Logging\n\ncase class FaissParams(\n  nprobe: Option[Int],\n  quantizerEf: Option[Int],\n  quantizerKFactorRF: Option[Int],\n  quantizerNprobe: Option[Int],\n  ht: Option[Int])\n    extends RuntimeParams {\n  override def toString: String = s\"FaissParams(${toLibraryString})\"\n\n  def toLibraryString: String =\n    Seq(\n      nprobe.map { n => s\"nprobe=${n}\" },\n      quantizerEf.map { ef => s\"quantizer_efSearch=${ef}\" },\n      quantizerKFactorRF.map { k => s\"quantizer_k_factor_rf=${k}\" },\n      quantizerNprobe.map { n => s\"quantizer_nprobe=${n}\" },\n      ht.map { ht => s\"ht=${ht}\" },\n    ).flatten.mkString(\",\")\n}\n\nobject FaissIndex {\n  def loadIndex[T, D <: Distance[D]](\n    outerDimension: Int,\n    outerMetric: Metric[D],\n    directory: AbstractFile\n  ): Queryable[T, FaissParams, D] = {\n    new QueryableIndexAdapter[T, D] with Logging {\n      protected val metric: Metric[D] = outerMetric\n      protected val dimension: Int = outerDimension\n      protected val index: Index = {\n        info(s\"Loading faiss with ${swigfaiss.get_compile_options()}\")\n\n        QueryableIndexAdapter.loadJavaIndex(directory)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/faiss/FaissIndexer.scala",
    "content": "package com.twitter.ann.faiss\n\nimport com.google.common.base.Preconditions\nimport com.twitter.ann.common.Cosine\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.EntityEmbedding\nimport com.twitter.ann.common.IndexOutputFile\nimport com.twitter.ann.common.InnerProduct\nimport com.twitter.ann.common.L2\nimport com.twitter.ann.common.Metric\nimport com.twitter.ml.api.embedding.EmbeddingMath\nimport com.twitter.scalding.Execution\nimport com.twitter.scalding.TypedPipe\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.search.common.file.FileUtils\nimport com.twitter.util.logging.Logging\nimport java.io.File\nimport scala.util.Random\n\ntrait FaissIndexer extends Logging {\n\n  /**\n   * Produce faiss index file specified by factory string\n   *\n   * @param pipe Embeddings to be indexed\n   * @param sampleRate Fraction of embeddings used for training. Regardless of this parameter, all embeddings are present in the output.\n   * @param factoryString Faiss factory string, see https://github.com/facebookresearch/faiss/wiki/The-index-factory\n   * @param metric Metric to use\n   * @param outputDirectory Directory where _SUCCESS and faiss.index will be written.\n   */\n  def build[D <: Distance[D]](\n    pipe: TypedPipe[EntityEmbedding[Long]],\n    sampleRate: Float,\n    factoryString: String,\n    metric: Metric[D],\n    outputDirectory: AbstractFile\n  ): Execution[Unit] = {\n    outputDirectory.mkdirs()\n    Preconditions.checkState(\n      outputDirectory.canRead,\n      \"Failed to create parent directories for %s\",\n      outputDirectory.toString)\n\n    val maybeNormalizedPipe = if (l2Normalize(metric)) {\n      pipe.map { idAndEmbedding =>\n        EntityEmbedding(idAndEmbedding.id, EmbeddingMath.Float.normalize(idAndEmbedding.embedding))\n      }\n    } else {\n      pipe\n    }\n\n    maybeNormalizedPipe.toIterableExecution.flatMap { annEmbeddings =>\n      logger.info(s\"${factoryString}\")\n      val t1 = System.nanoTime\n      buildAndWriteFaissIndex(\n        Random.shuffle(annEmbeddings),\n        sampleRate,\n        factoryString,\n        metric,\n        new IndexOutputFile(outputDirectory))\n      val duration = (System.nanoTime - t1) / 1e9d\n      logger.info(s\"It took ${duration}s to build and index\")\n\n      Execution.unit\n    }\n  }\n\n  def buildAndWriteFaissIndex[D <: Distance[D]](\n    entities: Iterable[EntityEmbedding[Long]],\n    sampleRate: Float,\n    factoryString: String,\n    metricType: Metric[D],\n    outputDirectory: IndexOutputFile\n  ): Unit = {\n    val metric = parseMetric(metricType)\n    val datasetSize = entities.size.toLong\n    val dimensions = entities.head.embedding.length\n    logger.info(s\"There are $datasetSize embeddings\")\n    logger.info(s\"Faiss compile options are ${swigfaiss.get_compile_options()}\")\n    logger.info(s\"OMP threads count is ${swigfaiss.omp_get_max_threads()}\")\n\n    val index = swigfaiss.index_factory(dimensions, factoryString, metric)\n    index.setVerbose(true)\n    val idMap = new IndexIDMap(index)\n\n    val trainingSetSize = Math.min(datasetSize, Math.round(datasetSize * sampleRate))\n    val ids = toIndexVector(entities)\n    val fullDataset = toFloatVector(dimensions, entities)\n    logger.info(\"Finished bridging full dataset\")\n    idMap.train(trainingSetSize, fullDataset.data())\n    logger.info(\"Finished training\")\n    idMap.add_with_ids(datasetSize, fullDataset.data(), ids)\n    logger.info(\"Added data to the index\")\n\n    val tmpFile = File.createTempFile(\"faiss.index\", \".tmp\")\n    swigfaiss.write_index(idMap, tmpFile.toString)\n    logger.info(s\"Wrote to tmp file ${tmpFile.toString}\")\n    copyToOutputAndCreateSuccess(FileUtils.getFileHandle(tmpFile.toString), outputDirectory)\n    logger.info(\"Copied file\")\n  }\n\n  private def copyToOutputAndCreateSuccess(\n    tmpFile: AbstractFile,\n    outputDirectory: IndexOutputFile\n  ) = {\n    val outputFile = outputDirectory.createFile(\"faiss.index\")\n    logger.info(s\"Final output file is ${outputFile.getPath()}\")\n    outputFile.copyFrom(tmpFile.getByteSource.openStream())\n    outputDirectory.createSuccessFile()\n  }\n\n  private def toFloatVector(\n    dimensions: Int,\n    entities: Iterable[EntityEmbedding[Long]]\n  ): FloatVector = {\n    require(entities.nonEmpty)\n\n    val vector = new FloatVector()\n    vector.reserve(dimensions.toLong * entities.size.toLong)\n    for (entity <- entities) {\n      for (value <- entity.embedding) {\n        vector.push_back(value)\n      }\n    }\n\n    vector\n  }\n\n  private def toIndexVector(embeddings: Iterable[EntityEmbedding[Long]]): LongVector = {\n    require(embeddings.nonEmpty)\n\n    val vector = new LongVector()\n    vector.reserve(embeddings.size)\n    for (embedding <- embeddings) {\n      vector.push_back(embedding.id)\n    }\n\n    vector\n  }\n\n  private def parseMetric[D <: Distance[D]](metric: Metric[D]): MetricType = metric match {\n    case L2 => MetricType.METRIC_L2\n    case InnerProduct => MetricType.METRIC_INNER_PRODUCT\n    case Cosine => MetricType.METRIC_INNER_PRODUCT\n    case _ => throw new AbstractMethodError(s\"Not implemented for metric ${metric}\")\n  }\n\n  private def l2Normalize[D <: Distance[D]](metric: Metric[D]): Boolean = metric match {\n    case Cosine => true\n    case _ => false\n  }\n}\n\nobject FaissIndexer extends FaissIndexer {}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/faiss/HourlyDirectoryWithSuccessFileListing.scala",
    "content": "package com.twitter.ann.faiss\n\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.search.common.file.FileUtils\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Time\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\nimport java.util.Locale\n\nobject HourlyDirectoryWithSuccessFileListing extends Logging {\n  private val SUCCESS_FILE_NAME = \"_SUCCESS\"\n\n  def listHourlyIndexDirectories(\n    root: AbstractFile,\n    startingFrom: Time,\n    count: Int,\n    lookbackInterval: Int\n  ): Seq[AbstractFile] = listingStep(root, startingFrom, count, lookbackInterval)\n\n  private def listingStep(\n    root: AbstractFile,\n    startingFrom: Time,\n    remainingDirectoriesToFind: Int,\n    remainingAttempts: Int\n  ): List[AbstractFile] = {\n    if (remainingDirectoriesToFind == 0 || remainingAttempts == 0) {\n      return List.empty\n    }\n\n    val head = getSuccessfulDirectoryForDate(root, startingFrom)\n\n    val previousHour = startingFrom - 1.hour\n\n    head match {\n      case Throw(e) =>\n        listingStep(root, previousHour, remainingDirectoriesToFind, remainingAttempts - 1)\n      case Return(directory) =>\n        directory ::\n          listingStep(root, previousHour, remainingDirectoriesToFind - 1, remainingAttempts - 1)\n    }\n  }\n\n  private def getSuccessfulDirectoryForDate(\n    root: AbstractFile,\n    date: Time\n  ): Try[AbstractFile] = {\n    val folder = root.getPath + \"/\" + date.format(\"yyyy/MM/dd/HH\", Locale.ROOT)\n    val successPath =\n      folder + \"/\" + SUCCESS_FILE_NAME\n\n    debug(s\"Checking ${successPath}\")\n\n    Try(FileUtils.getFileHandle(successPath)).flatMap { file =>\n      if (file.canRead) {\n        Try(FileUtils.getFileHandle(folder))\n      } else {\n        Throw(new IllegalArgumentException(s\"Found ${file.toString} but can't read it\"))\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/faiss/HourlyShardedIndex.scala",
    "content": "package com.twitter.ann.faiss\n\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.MemoizedInEpochs\nimport com.twitter.ann.common.Metric\nimport com.twitter.ann.common.Task\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\nimport java.util.concurrent.atomic.AtomicReference\n\nobject HourlyShardedIndex {\n  def loadIndex[T, D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    directory: AbstractFile,\n    shardsToLoad: Int,\n    shardWatchInterval: Duration,\n    lookbackInterval: Int,\n    statsReceiver: StatsReceiver\n  ): HourlyShardedIndex[T, D] = {\n    new HourlyShardedIndex[T, D](\n      metric,\n      dimension,\n      directory,\n      shardsToLoad,\n      shardWatchInterval,\n      lookbackInterval,\n      statsReceiver)\n  }\n}\n\nclass HourlyShardedIndex[T, D <: Distance[D]](\n  outerMetric: Metric[D],\n  outerDimension: Int,\n  directory: AbstractFile,\n  shardsToLoad: Int,\n  shardWatchInterval: Duration,\n  lookbackInterval: Int,\n  override protected val statsReceiver: StatsReceiver)\n    extends QueryableIndexAdapter[T, D]\n    with Logging\n    with Task {\n  // QueryableIndexAdapter\n  protected val metric: Metric[D] = outerMetric\n  protected val dimension: Int = outerDimension\n  protected def index: Index = {\n    castedIndex.get()\n  }\n\n  // Task trait\n  protected def task(): Future[Unit] = Future.value(reloadShards())\n  protected def taskInterval: Duration = shardWatchInterval\n\n  private def loadIndex(directory: AbstractFile): Try[Index] =\n    Try(QueryableIndexAdapter.loadJavaIndex(directory))\n\n  private val shardsCache = new MemoizedInEpochs[AbstractFile, Index](loadIndex)\n  // Destroying original index invalidate casted index. Keep a reference to both.\n  private val originalIndex = new AtomicReference[IndexShards]()\n  private val castedIndex = new AtomicReference[Index]()\n  private def reloadShards(): Unit = {\n    val freshDirectories =\n      HourlyDirectoryWithSuccessFileListing.listHourlyIndexDirectories(\n        directory,\n        Time.now,\n        shardsToLoad,\n        lookbackInterval)\n\n    if (shardsCache.currentEpochKeys == freshDirectories.toSet) {\n      info(\"Not reloading shards, as they're exactly same\")\n    } else {\n      val shards = shardsCache.epoch(freshDirectories)\n      val indexShards = new IndexShards(dimension, false, false)\n      for (shard <- shards) {\n        indexShards.add_shard(shard)\n      }\n\n      replaceIndex(() => {\n        castedIndex.set(swigfaiss.upcast_IndexShards(indexShards))\n        originalIndex.set(indexShards)\n      })\n\n      // Potentially it's time to drop huge native index from memory, ask for GC\n      System.gc()\n    }\n\n    require(castedIndex.get() != null, \"Failed to find any shards during startup\")\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/faiss/QueryableIndexAdapter.scala",
    "content": "package com.twitter.ann.faiss\n\nimport com.twitter.ann.common.Cosine\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.ann.common.Metric\nimport com.twitter.ann.common.NeighborWithDistance\nimport com.twitter.ann.common.Queryable\nimport com.twitter.ml.api.embedding.EmbeddingMath\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.search.common.file.FileUtils\nimport com.twitter.util.Future\nimport com.twitter.util.logging.Logging\nimport java.io.File\nimport java.util.concurrent.locks.ReentrantReadWriteLock\n\nobject QueryableIndexAdapter extends Logging {\n  // swigfaiss.read_index doesn't support hdfs files, hence a copy to temporary directory\n  def loadJavaIndex(directory: AbstractFile): Index = {\n    val indexFile = directory.getChild(\"faiss.index\")\n    val tmpFile = File.createTempFile(\"faiss.index\", \".tmp\")\n    val tmpAbstractFile = FileUtils.getFileHandle(tmpFile.toString)\n    indexFile.copyTo(tmpAbstractFile)\n    val index = swigfaiss.read_index(tmpAbstractFile.getPath)\n\n    if (!tmpFile.delete()) {\n      error(s\"Failed to delete ${tmpFile.toString}\")\n    }\n\n    index\n  }\n}\n\ntrait QueryableIndexAdapter[T, D <: Distance[D]] extends Queryable[T, FaissParams, D] {\n  this: Logging =>\n\n  private val MAX_COSINE_DISTANCE = 1f\n\n  protected def index: Index\n  protected val metric: Metric[D]\n  protected val dimension: Int\n\n  private def maybeNormalizeEmbedding(embeddingVector: EmbeddingVector): EmbeddingVector = {\n    // There is no direct support for Cosine, but l2norm + ip == Cosine by definition\n    if (metric == Cosine) {\n      EmbeddingMath.Float.normalize(embeddingVector)\n    } else {\n      embeddingVector\n    }\n  }\n\n  private def maybeTranslateToCosineDistanceInplace(array: floatArray, len: Int): Unit = {\n    // Faiss reports Cosine similarity while we need Cosine distance.\n    if (metric == Cosine) {\n      for (index <- 0 until len) {\n        val similarity = array.getitem(index)\n        if (similarity < 0 || similarity > 1) {\n          warn(s\"Expected similarity to be between 0 and 1, got ${similarity} instead\")\n          array.setitem(index, MAX_COSINE_DISTANCE)\n        } else {\n          array.setitem(index, 1 - similarity)\n        }\n      }\n    }\n  }\n\n  private val paramsLock = new ReentrantReadWriteLock()\n  private var currentParams: Option[String] = None\n  // Assume that parameters rarely change and try read lock first\n  private def ensuringParams[R](parameterString: String, f: () => R): R = {\n    paramsLock.readLock().lock()\n    try {\n      if (currentParams.contains(parameterString)) {\n        return f()\n      }\n    } finally {\n      paramsLock.readLock().unlock()\n    }\n\n    paramsLock.writeLock().lock()\n    try {\n      currentParams = Some(parameterString)\n      new ParameterSpace().set_index_parameters(index, parameterString)\n\n      f()\n    } finally {\n      paramsLock.writeLock().unlock()\n    }\n  }\n\n  def replaceIndex(f: () => Unit): Unit = {\n    paramsLock.writeLock().lock()\n    try {\n      currentParams = None\n\n      f()\n    } finally {\n      paramsLock.writeLock().unlock()\n    }\n  }\n\n  def query(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: FaissParams\n  ): Future[List[T]] = {\n    Future.value(\n      ensuringParams(\n        runtimeParams.toLibraryString,\n        () => {\n          val distances = new floatArray(numOfNeighbors)\n          val indexes = new LongVector()\n          indexes.resize(numOfNeighbors)\n\n          val normalizedEmbedding = maybeNormalizeEmbedding(embedding)\n          index.search(\n            // Number of query embeddings\n            1,\n            // Array of query embeddings\n            toFloatArray(normalizedEmbedding).cast(),\n            // Number of neighbours to return\n            numOfNeighbors,\n            // Location to store neighbour distances\n            distances.cast(),\n            // Location to store neighbour identifiers\n            indexes\n          )\n          // This is a shortcoming of current swig bindings\n          // Nothing prevents JVM from freeing distances while inside index.search\n          // This might be removed once we start passing FloatVector\n          // Why java.lang.ref.Reference.reachabilityFence doesn't compile?\n          debug(distances)\n\n          toSeq(indexes, numOfNeighbors).toList.asInstanceOf[List[T]]\n        }\n      ))\n  }\n\n  def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: FaissParams\n  ): Future[List[NeighborWithDistance[T, D]]] = {\n    Future.value(\n      ensuringParams(\n        runtimeParams.toLibraryString,\n        () => {\n          val distances = new floatArray(numOfNeighbors)\n          val indexes = new LongVector()\n          indexes.resize(numOfNeighbors)\n\n          val normalizedEmbedding = maybeNormalizeEmbedding(embedding)\n          index.search(\n            // Number of query embeddings\n            1,\n            // Array of query embeddings\n            toFloatArray(normalizedEmbedding).cast(),\n            // Number of neighbours to return\n            numOfNeighbors,\n            // Location to store neighbour distances\n            distances.cast(),\n            // Location to store neighbour identifiers\n            indexes\n          )\n\n          val ids = toSeq(indexes, numOfNeighbors).toList.asInstanceOf[List[T]]\n\n          maybeTranslateToCosineDistanceInplace(distances, numOfNeighbors)\n\n          val distancesSeq = toSeq(distances, numOfNeighbors)\n\n          ids.zip(distancesSeq).map {\n            case (id, distance) =>\n              NeighborWithDistance(id, metric.fromAbsoluteDistance(distance))\n          }\n        }\n      ))\n  }\n\n  private def toFloatArray(emb: EmbeddingVector): floatArray = {\n    val nativeArray = new floatArray(emb.length)\n    for ((value, aIdx) <- emb.iterator.zipWithIndex) {\n      nativeArray.setitem(aIdx, value)\n    }\n\n    nativeArray\n  }\n\n  private def toSeq(vector: LongVector, len: Long): Seq[Long] = {\n    (0L until len).map(vector.at)\n  }\n\n  private def toSeq(array: floatArray, len: Int): Seq[Float] = {\n    (0 until len).map(array.getitem)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/featurestore/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"src/scala/com/twitter/ml/featurestore/lib\",\n        \"src/scala/com/twitter/ml/featurestore/lib/online\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/featurestore/FeatureStoreEmbeddingProducer.scala",
    "content": "package com.twitter.ann.featurestore\n\nimport com.twitter.ann.common.EmbeddingProducer\nimport com.twitter.finagle.stats.{InMemoryStatsReceiver, StatsReceiver}\nimport com.twitter.ml.api.embedding.{Embedding, EmbeddingSerDe}\nimport com.twitter.ml.api.thriftscala\nimport com.twitter.ml.api.thriftscala.{Embedding => TEmbedding}\nimport com.twitter.ml.featurestore.lib.dataset.online.VersionedOnlineAccessDataset\nimport com.twitter.ml.featurestore.lib.{EntityId, RawFloatTensor}\nimport com.twitter.ml.featurestore.lib.dataset.DatasetParams\nimport com.twitter.ml.featurestore.lib.entity.EntityWithId\nimport com.twitter.ml.featurestore.lib.feature.{BoundFeature, BoundFeatureSet}\nimport com.twitter.ml.featurestore.lib.online.{FeatureStoreClient, FeatureStoreRequest}\nimport com.twitter.ml.featurestore.lib.params.FeatureStoreParams\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.opcontext.Attribution\nimport com.twitter.strato.client.Client\n\nobject FeatureStoreEmbeddingProducer {\n  def apply[T <: EntityId](\n    dataset: VersionedOnlineAccessDataset[T, TEmbedding],\n    version: Long,\n    boundFeature: BoundFeature[T, RawFloatTensor],\n    client: Client,\n    statsReceiver: StatsReceiver = new InMemoryStatsReceiver,\n    featureStoreAttributions: Seq[Attribution] = Seq.empty\n  ): EmbeddingProducer[EntityWithId[T]] = {\n    val featureStoreParams = FeatureStoreParams(\n      perDataset = Map(\n        dataset.id -> DatasetParams(datasetVersion = Some(version))\n      ),\n      global = DatasetParams(attributions = featureStoreAttributions)\n    )\n    val featureStoreClient = FeatureStoreClient(\n      BoundFeatureSet(boundFeature),\n      client,\n      statsReceiver,\n      featureStoreParams\n    )\n    new FeatureStoreEmbeddingProducer(boundFeature, featureStoreClient)\n  }\n}\n\nprivate[featurestore] class FeatureStoreEmbeddingProducer[T <: EntityId](\n  boundFeature: BoundFeature[T, RawFloatTensor],\n  featureStoreClient: FeatureStoreClient)\n    extends EmbeddingProducer[EntityWithId[T]] {\n  // Looks up embedding from online feature store for an entity.\n  override def produceEmbedding(input: EntityWithId[T]): Stitch[Option[Embedding[Float]]] = {\n    val featureStoreRequest = FeatureStoreRequest(\n      entityIds = Seq(input)\n    )\n\n    Stitch.callFuture(featureStoreClient(featureStoreRequest).map { predictionRecord =>\n      predictionRecord.getFeatureValue(boundFeature) match {\n        case Some(featureValue) => {\n          val embedding = EmbeddingSerDe.floatEmbeddingSerDe.fromThrift(\n            thriftscala.Embedding(Some(featureValue.value))\n          )\n          Some(embedding)\n        }\n        case _ => None\n      }\n    })\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/file_store/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"mediaservices/commons/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/file_store/ReadableIndexIdFileStore.scala",
    "content": "package com.twitter.ann.file_store\n\nimport com.twitter.ann.common.thriftscala.FileBasedIndexIdStore\nimport com.twitter.bijection.Injection\nimport com.twitter.mediaservices.commons.codec.{ArrayByteBufferCodec, ThriftByteBufferCodec}\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.storehaus.ReadableStore\nimport java.nio.ByteBuffer\n\nobject ReadableIndexIdFileStore {\n\n  /**\n   * @param file : File path to read serialized long indexId <-> Id mapping from.\n   * @param injection: Injection to convert bytes to Id.\n   * @tparam V: Type of Id\n   * @return File based Readable Store\n   */\n  def apply[V](\n    file: AbstractFile,\n    injection: Injection[V, Array[Byte]]\n  ): ReadableStore[Long, V] = {\n    val codec = new ThriftByteBufferCodec(FileBasedIndexIdStore)\n    val store: Map[Long, V] = codec\n      .decode(loadFile(file))\n      .indexIdMap\n      .getOrElse(Map.empty[Long, ByteBuffer])\n      .toMap\n      .mapValues(value => injection.invert(ArrayByteBufferCodec.decode(value)).get)\n    ReadableStore.fromMap[Long, V](store)\n  }\n\n  private[this] def loadFile(file: AbstractFile): ByteBuffer = {\n    ArrayByteBufferCodec.encode(file.getByteSource.read())\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/file_store/WritableIndexIdFileStore.scala",
    "content": "package com.twitter.ann.file_store\n\nimport com.twitter.ann.common.IndexOutputFile\nimport com.twitter.ann.common.thriftscala.FileBasedIndexIdStore\nimport com.twitter.bijection.Injection\nimport com.twitter.mediaservices.commons.codec.ArrayByteBufferCodec\nimport com.twitter.mediaservices.commons.codec.ThriftByteBufferCodec\nimport com.twitter.storehaus.Store\nimport com.twitter.util.Future\nimport java.util.concurrent.{ConcurrentHashMap => JConcurrentHashMap}\nimport scala.collection.JavaConverters._\n\nobject WritableIndexIdFileStore {\n\n  /**\n   * @param injection: Injection to convert typed Id to bytes.\n   * @tparam V: Type of Id\n   * @return File based Writable Store\n   */\n  def apply[V](\n    injection: Injection[V, Array[Byte]]\n  ): WritableIndexIdFileStore[V] = {\n    new WritableIndexIdFileStore[V](\n      new JConcurrentHashMap[Long, Option[V]],\n      injection\n    )\n  }\n}\n\nclass WritableIndexIdFileStore[V] private (\n  map: JConcurrentHashMap[Long, Option[V]],\n  injection: Injection[V, Array[Byte]])\n    extends Store[Long, V] {\n\n  private[this] val store = Store.fromJMap(map)\n\n  override def get(k: Long): Future[Option[V]] = {\n    store.get(k)\n  }\n\n  override def put(kv: (Long, Option[V])): Future[Unit] = {\n    store.put(kv)\n  }\n\n  /**\n   * Serialize and store the mapping in thrift format\n   * @param file : File path to store serialized long indexId <-> Id mapping\n   */\n  def save(file: IndexOutputFile): Unit = {\n    saveThrift(toThrift(), file)\n  }\n\n  def getInjection: Injection[V, Array[Byte]] = injection\n\n  private[this] def toThrift(): FileBasedIndexIdStore = {\n    val indexIdMap = map.asScala\n      .collect {\n        case (key, Some(value)) => (key, ArrayByteBufferCodec.encode(injection.apply(value)))\n      }\n\n    FileBasedIndexIdStore(Some(indexIdMap))\n  }\n\n  private[this] def saveThrift(thriftObj: FileBasedIndexIdStore, file: IndexOutputFile): Unit = {\n    val codec = new ThriftByteBufferCodec(FileBasedIndexIdStore)\n    val bytes = ArrayByteBufferCodec.decode(codec.encode(thriftObj))\n    val outputStream = file.getOutputStream()\n    outputStream.write(bytes)\n    outputStream.close()\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/hnsw/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/org/mapdb\",\n        \"ann/src/main/java/com/twitter/ann/hnsw\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/serialization\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"mediaservices/commons/src/main/scala:futuretracker\",\n        \"src/java/com/twitter/common_internal/hadoop\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/scala/com/twitter/ml/api/embedding\",\n    ],\n    exports = [\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"src/java/com/twitter/common_internal/hadoop\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/scala/com/twitter/ml/api/embedding\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/hnsw/DistanceFunctionGenerator.scala",
    "content": "package com.twitter.ann.hnsw\n\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.ann.common.{Cosine, Distance, InnerProduct, Metric}\n\nprivate[hnsw] object DistanceFunctionGenerator {\n  def apply[T, D <: Distance[D]](\n    metric: Metric[D],\n    idToEmbeddingFn: (T) => EmbeddingVector\n  ): DistanceFunctionGenerator[T] = {\n    // Use InnerProduct for cosine and normalize the vectors before appending and querying.\n    val updatedMetric = metric match {\n      case Cosine => InnerProduct\n      case _ => metric\n    }\n\n    val distFnIndex = new DistanceFunction[T, T] {\n      override def distance(id1: T, id2: T) =\n        updatedMetric.absoluteDistance(\n          idToEmbeddingFn(id1),\n          idToEmbeddingFn(id2)\n        )\n    }\n\n    val distFnQuery = new DistanceFunction[EmbeddingVector, T] {\n      override def distance(embedding: EmbeddingVector, id: T) =\n        updatedMetric.absoluteDistance(embedding, idToEmbeddingFn(id))\n    }\n\n    DistanceFunctionGenerator(distFnIndex, distFnQuery, metric == Cosine)\n  }\n}\n\nprivate[hnsw] case class DistanceFunctionGenerator[T](\n  index: DistanceFunction[T, T],\n  query: DistanceFunction[EmbeddingVector, T],\n  shouldNormalize: Boolean)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/hnsw/Hnsw.scala",
    "content": "package com.twitter.ann.hnsw\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.twitter.ann.common.EmbeddingType._\nimport com.twitter.ann.common.Metric.toThrift\nimport com.twitter.ann.common._\nimport com.twitter.ann.common.thriftscala.DistanceMetric\nimport com.twitter.ann.hnsw.HnswIndex.RandomProvider\nimport com.twitter.util.Future\nimport java.util.Random\nimport java.util.concurrent.ConcurrentHashMap\nimport java.util.concurrent.ThreadLocalRandom\nimport java.util.concurrent.locks.Lock\nimport java.util.concurrent.locks.ReentrantLock\nimport scala.collection.JavaConverters._\n\nprivate[hnsw] object Hnsw {\n  private[hnsw] def apply[T, D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    efConstruction: Int,\n    maxM: Int,\n    expectedElements: Int,\n    futurePool: ReadWriteFuturePool,\n    idEmbeddingMap: IdEmbeddingMap[T]\n  ): Hnsw[T, D] = {\n    val randomProvider = new RandomProvider {\n      override def get(): Random = ThreadLocalRandom.current()\n    }\n    val distFn =\n      DistanceFunctionGenerator(metric, (key: T) => idEmbeddingMap.get(key))\n    val internalIndex = new HnswIndex[T, EmbeddingVector](\n      distFn.index,\n      distFn.query,\n      efConstruction,\n      maxM,\n      expectedElements,\n      randomProvider\n    )\n    new Hnsw[T, D](\n      dimension,\n      metric,\n      internalIndex,\n      futurePool,\n      idEmbeddingMap,\n      distFn.shouldNormalize,\n      LockedAccess.apply(expectedElements)\n    )\n  }\n}\n\nprivate[hnsw] object LockedAccess {\n  protected[hnsw] def apply[T](expectedElements: Int): LockedAccess[T] =\n    DefaultLockedAccess(new ConcurrentHashMap[T, Lock](expectedElements))\n  protected[hnsw] def apply[T](): LockedAccess[T] =\n    DefaultLockedAccess(new ConcurrentHashMap[T, Lock]())\n}\n\nprivate[hnsw] case class DefaultLockedAccess[T](locks: ConcurrentHashMap[T, Lock])\n    extends LockedAccess[T] {\n  override def lockProvider(item: T) = locks.computeIfAbsent(item, (_: T) => new ReentrantLock())\n}\n\nprivate[hnsw] trait LockedAccess[T] {\n  protected def lockProvider(item: T): Lock\n  def lock[K](item: T)(fn: => K): K = {\n    val lock = lockProvider(item)\n    lock.lock()\n    try {\n      fn\n    } finally {\n      lock.unlock()\n    }\n  }\n}\n\n@VisibleForTesting\nprivate[hnsw] class Hnsw[T, D <: Distance[D]](\n  dimension: Int,\n  metric: Metric[D],\n  hnswIndex: HnswIndex[T, EmbeddingVector],\n  readWriteFuturePool: ReadWriteFuturePool,\n  idEmbeddingMap: IdEmbeddingMap[T],\n  shouldNormalize: Boolean,\n  lockedAccess: LockedAccess[T] = LockedAccess.apply[T]())\n    extends Appendable[T, HnswParams, D]\n    with Queryable[T, HnswParams, D]\n    with Updatable[T] {\n  override def append(entity: EntityEmbedding[T]): Future[Unit] = {\n    readWriteFuturePool.write {\n      val indexDimension = entity.embedding.length\n      assert(\n        toThrift(metric) == DistanceMetric.EditDistance || indexDimension == dimension,\n        s\"Dimension mismatch for index(${indexDimension}) and embedding($dimension)\"\n      )\n\n      lockedAccess.lock(entity.id) {\n        // To make this thread-safe, we are using ConcurrentHashMap#putIfAbsent underneath,\n        // so if there is a pre-existing item, put() will return something that is not null\n        val embedding = idEmbeddingMap.putIfAbsent(entity.id, updatedEmbedding(entity.embedding))\n\n        if (embedding == null) { // New element - insert into the index\n          hnswIndex.insert(entity.id)\n        } else { // Existing element - update the embedding and graph structure\n          throw new IllegalDuplicateInsertException(\n            \"Append method does not permit duplicates (try using update method): \" + entity.id)\n        }\n      }\n    } onFailure { e =>\n      Future.exception(e)\n    }\n  }\n\n  override def toQueryable: Queryable[T, HnswParams, D] = this\n\n  override def query(\n    embedding: EmbeddingVector,\n    numOfNeighbours: Int,\n    runtimeParams: HnswParams\n  ): Future[List[T]] = {\n    queryWithDistance(embedding, numOfNeighbours, runtimeParams)\n      .map(_.map(_.neighbor))\n  }\n\n  override def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbours: Int,\n    runtimeParams: HnswParams\n  ): Future[List[NeighborWithDistance[T, D]]] = {\n    val indexDimension = embedding.length\n    assert(\n      toThrift(metric) == DistanceMetric.EditDistance || indexDimension == dimension,\n      s\"Dimension mismatch for index(${indexDimension}) and embedding($dimension)\"\n    )\n    readWriteFuturePool.read {\n      hnswIndex\n        .searchKnn(updatedEmbedding(embedding), numOfNeighbours, runtimeParams.ef)\n        .asScala\n        .map { nn =>\n          NeighborWithDistance(\n            nn.getItem,\n            metric.fromAbsoluteDistance(nn.getDistance)\n          )\n        }\n        .toList\n    }\n  }\n\n  private[this] def updatedEmbedding(embedding: EmbeddingVector): EmbeddingVector = {\n    if (shouldNormalize) {\n      MetricUtil.norm(embedding)\n    } else {\n      embedding\n    }\n  }\n\n  def getIndex: HnswIndex[T, EmbeddingVector] = hnswIndex\n  def getDimen: Int = dimension\n  def getMetric: Metric[D] = metric\n  def getIdEmbeddingMap: IdEmbeddingMap[T] = idEmbeddingMap\n  override def update(\n    entity: EntityEmbedding[T]\n  ): Future[Unit] = {\n    readWriteFuturePool.write {\n      val indexDimension = entity.embedding.length\n      assert(\n        toThrift(metric) == DistanceMetric.EditDistance || indexDimension == dimension,\n        s\"Dimension mismatch for index(${indexDimension}) and embedding($dimension)\"\n      )\n\n      lockedAccess.lock(entity.id) {\n        val embedding = idEmbeddingMap.put(entity.id, updatedEmbedding(entity.embedding))\n        if (embedding == null) { // New element - insert into the index\n          hnswIndex.insert(entity.id)\n        } else { // Existing element - update the embedding and graph structure\n          hnswIndex.reInsert(entity.id);\n        }\n      }\n    } onFailure { e =>\n      Future.exception(e)\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/hnsw/HnswCommon.scala",
    "content": "package com.twitter.ann.hnsw\n\nimport com.twitter.ann.common.RuntimeParams\nimport com.twitter.ann.common.thriftscala.HnswIndexMetadata\nimport com.twitter.ann.common.thriftscala.HnswRuntimeParam\nimport com.twitter.ann.common.thriftscala.{RuntimeParams => ServiceRuntimeParams}\nimport com.twitter.bijection.Injection\nimport com.twitter.mediaservices.commons.codec.ThriftByteBufferCodec\nimport com.twitter.search.common.file.AbstractFile\nimport scala.util.Failure\nimport scala.util.Success\nimport scala.util.Try\n\nobject HnswCommon {\n  private[hnsw] lazy val MetadataCodec = new ThriftByteBufferCodec(HnswIndexMetadata)\n  private[hnsw] val MetaDataFileName = \"hnsw_index_metadata\"\n  private[hnsw] val EmbeddingMappingFileName = \"hnsw_embedding_mapping\"\n  private[hnsw] val InternalIndexDir = \"hnsw_internal_index\"\n  private[hnsw] val HnswInternalMetadataFileName = \"hnsw_internal_metadata\"\n  private[hnsw] val HnswInternalGraphFileName = \"hnsw_internal_graph\"\n\n  val RuntimeParamsInjection: Injection[HnswParams, ServiceRuntimeParams] =\n    new Injection[HnswParams, ServiceRuntimeParams] {\n      override def apply(scalaParams: HnswParams): ServiceRuntimeParams = {\n        ServiceRuntimeParams.HnswParam(\n          HnswRuntimeParam(\n            scalaParams.ef\n          )\n        )\n      }\n\n      override def invert(thriftParams: ServiceRuntimeParams): Try[HnswParams] =\n        thriftParams match {\n          case ServiceRuntimeParams.HnswParam(hnswParam) =>\n            Success(\n              HnswParams(hnswParam.ef)\n            )\n          case p => Failure(new IllegalArgumentException(s\"Expected HnswRuntimeParam got $p\"))\n        }\n    }\n\n  def isValidHnswIndex(path: AbstractFile): Boolean = {\n    path.isDirectory &&\n    path.hasSuccessFile &&\n    path.getChild(MetaDataFileName).exists() &&\n    path.getChild(EmbeddingMappingFileName).exists() &&\n    path.getChild(InternalIndexDir).exists() &&\n    path.getChild(InternalIndexDir).getChild(HnswInternalMetadataFileName).exists() &&\n    path.getChild(InternalIndexDir).getChild(HnswInternalGraphFileName).exists()\n  }\n}\n\n/**\n * Hnsw runtime params\n * @param ef: The size of the dynamic list for the nearest neighbors (used during the search).\n *          Higher ef leads to more accurate but slower search.\n *          ef cannot be set lower than the number of queried nearest neighbors k.\n *          The value ef of can be anything between k and the size of the dataset.\n */\ncase class HnswParams(ef: Int) extends RuntimeParams {\n  override def toString: String = s\"HnswParams(ef = $ef)\"\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/hnsw/HnswIOUtil.scala",
    "content": "package com.twitter.ann.hnsw\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.ann.common.thriftscala.HnswIndexMetadata\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.EntityEmbedding\nimport com.twitter.ann.common.Metric\nimport com.twitter.ann.hnsw.HnswCommon._\nimport com.twitter.ann.serialization.PersistedEmbeddingInjection\nimport com.twitter.ann.serialization.ThriftIteratorIO\nimport com.twitter.ann.serialization.thriftscala.PersistedEmbedding\nimport com.twitter.bijection.Injection\nimport com.twitter.mediaservices.commons.codec.ArrayByteBufferCodec\nimport com.twitter.search.common.file.AbstractFile\nimport java.io.BufferedInputStream\nimport java.io.BufferedOutputStream\nimport java.io.OutputStream\n\nprivate[hnsw] object HnswIOUtil {\n  private val BufferSize = 64 * 1024 // Default 64Kb\n\n  @VisibleForTesting\n  private[hnsw] def loadEmbeddings[T](\n    embeddingFile: AbstractFile,\n    injection: Injection[T, Array[Byte]],\n    idEmbeddingMap: IdEmbeddingMap[T],\n  ): IdEmbeddingMap[T] = {\n    val inputStream = {\n      val stream = embeddingFile.getByteSource.openStream()\n      if (stream.isInstanceOf[BufferedInputStream]) {\n        stream\n      } else {\n        new BufferedInputStream(stream, BufferSize)\n      }\n    }\n\n    val thriftIteratorIO =\n      new ThriftIteratorIO[PersistedEmbedding](PersistedEmbedding)\n    val iterator = thriftIteratorIO.fromInputStream(inputStream)\n    val embeddingInjection = new PersistedEmbeddingInjection(injection)\n    try {\n      iterator.foreach { persistedEmbedding =>\n        val embedding = embeddingInjection.invert(persistedEmbedding).get\n        idEmbeddingMap.putIfAbsent(embedding.id, embedding.embedding)\n        Unit\n      }\n    } finally {\n      inputStream.close()\n    }\n    idEmbeddingMap\n  }\n\n  @VisibleForTesting\n  private[hnsw] def saveEmbeddings[T](\n    stream: OutputStream,\n    injection: Injection[T, Array[Byte]],\n    iter: Iterator[(T, EmbeddingVector)]\n  ): Unit = {\n    val thriftIteratorIO =\n      new ThriftIteratorIO[PersistedEmbedding](PersistedEmbedding)\n    val embeddingInjection = new PersistedEmbeddingInjection(injection)\n    val iterator = iter.map {\n      case (id, emb) =>\n        embeddingInjection(EntityEmbedding(id, emb))\n    }\n    val outputStream = {\n      if (stream.isInstanceOf[BufferedOutputStream]) {\n        stream\n      } else {\n        new BufferedOutputStream(stream, BufferSize)\n      }\n    }\n    try {\n      thriftIteratorIO.toOutputStream(iterator, outputStream)\n    } finally {\n      outputStream.close()\n    }\n  }\n\n  @VisibleForTesting\n  private[hnsw] def saveIndexMetadata(\n    dimension: Int,\n    metric: Metric[_ <: Distance[_]],\n    numElements: Int,\n    metadataStream: OutputStream\n  ): Unit = {\n    val metadata = HnswIndexMetadata(\n      dimension,\n      Metric.toThrift(metric),\n      numElements\n    )\n    val bytes = ArrayByteBufferCodec.decode(MetadataCodec.encode(metadata))\n    metadataStream.write(bytes)\n    metadataStream.close()\n  }\n\n  @VisibleForTesting\n  private[hnsw] def loadIndexMetadata(\n    metadataFile: AbstractFile\n  ): HnswIndexMetadata = {\n    MetadataCodec.decode(\n      ArrayByteBufferCodec.encode(metadataFile.getByteSource.read())\n    )\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/hnsw/IdEmbeddingMap.scala",
    "content": "package com.twitter.ann.hnsw\n\nimport com.twitter.ann.common.EmbeddingType._\nimport java.io.OutputStream\n\ntrait IdEmbeddingMap[T] {\n  def putIfAbsent(id: T, embedding: EmbeddingVector): EmbeddingVector\n  def put(id: T, embedding: EmbeddingVector): EmbeddingVector\n  def get(id: T): EmbeddingVector\n  def iter(): Iterator[(T, EmbeddingVector)]\n  def size(): Int\n  def toDirectory(embeddingFileOutputStream: OutputStream): Unit\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/hnsw/JMapBasedIdEmbeddingMap.scala",
    "content": "package com.twitter.ann.hnsw\n\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.bijection.Injection\nimport com.twitter.search.common.file.AbstractFile\nimport java.io.OutputStream\nimport java.util.concurrent.ConcurrentHashMap\nimport scala.collection.JavaConverters._\n\nprivate[hnsw] object JMapBasedIdEmbeddingMap {\n\n  /**\n   * Creates in-memory concurrent hashmap based container that for storing id embedding mapping.\n   * @param expectedElements: Expected num of elements for sizing hint, need not be exact.\n   */\n  def applyInMemory[T](expectedElements: Int): IdEmbeddingMap[T] =\n    new JMapBasedIdEmbeddingMap[T](\n      new ConcurrentHashMap[T, EmbeddingVector](expectedElements),\n      Option.empty\n    )\n\n  /**\n   * Creates in-memory concurrent hashmap based container that can be serialized to disk for storing id embedding mapping.\n   * @param expectedElements: Expected num of elements for sizing hint, need not be exact.\n   * @param injection : Injection for typed Id T to Array[Byte]\n   */\n  def applyInMemoryWithSerialization[T](\n    expectedElements: Int,\n    injection: Injection[T, Array[Byte]]\n  ): IdEmbeddingMap[T] =\n    new JMapBasedIdEmbeddingMap[T](\n      new ConcurrentHashMap[T, EmbeddingVector](expectedElements),\n      Some(injection)\n    )\n\n  /**\n   * Loads id embedding mapping in in-memory concurrent hashmap.\n   * @param embeddingFile: Local/Hdfs file path for embeddings\n   * @param injection : Injection for typed Id T to Array[Byte]\n   * @param numElements: Expected num of elements for sizing hint, need not be exact\n   */\n  def loadInMemory[T](\n    embeddingFile: AbstractFile,\n    injection: Injection[T, Array[Byte]],\n    numElements: Option[Int] = Option.empty\n  ): IdEmbeddingMap[T] = {\n    val map = numElements match {\n      case Some(elements) => new ConcurrentHashMap[T, EmbeddingVector](elements)\n      case None => new ConcurrentHashMap[T, EmbeddingVector]()\n    }\n    HnswIOUtil.loadEmbeddings(\n      embeddingFile,\n      injection,\n      new JMapBasedIdEmbeddingMap(map, Some(injection))\n    )\n  }\n}\n\nprivate[this] class JMapBasedIdEmbeddingMap[T](\n  map: java.util.concurrent.ConcurrentHashMap[T, EmbeddingVector],\n  injection: Option[Injection[T, Array[Byte]]])\n    extends IdEmbeddingMap[T] {\n  override def putIfAbsent(id: T, embedding: EmbeddingVector): EmbeddingVector = {\n    map.putIfAbsent(id, embedding)\n  }\n\n  override def put(id: T, embedding: EmbeddingVector): EmbeddingVector = {\n    map.put(id, embedding)\n  }\n\n  override def get(id: T): EmbeddingVector = {\n    map.get(id)\n  }\n\n  override def iter(): Iterator[(T, EmbeddingVector)] =\n    map\n      .entrySet()\n      .iterator()\n      .asScala\n      .map(e => (e.getKey, e.getValue))\n\n  override def size(): Int = map.size()\n\n  override def toDirectory(embeddingFile: OutputStream): Unit = {\n    HnswIOUtil.saveEmbeddings(embeddingFile, injection.get, iter())\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/hnsw/MapDbBasedIdEmbeddingMap.scala",
    "content": "package com.twitter.ann.hnsw\n\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.bijection.Injection\nimport com.twitter.ml.api.embedding.Embedding\nimport com.twitter.search.common.file.AbstractFile\nimport java.io.OutputStream\nimport org.mapdb.DBMaker\nimport org.mapdb.HTreeMap\nimport org.mapdb.Serializer\nimport scala.collection.JavaConverters._\n\n/**\n * This class currently only support querying and creates map db on fly from thrift serialized embedding mapping\n * Implement index creation with this or altogether replace mapdb with some better performing solution as it takes a lot of time to create/query or precreate while serializing thrift embeddings\n */\nprivate[hnsw] object MapDbBasedIdEmbeddingMap {\n\n  /**\n   * Loads id embedding mapping in mapDB based container leveraging memory mapped files.\n   * @param embeddingFile: Local/Hdfs file path for embeddings\n   * @param injection : Injection for typed Id T to Array[Byte]\n   */\n  def loadAsReadonly[T](\n    embeddingFile: AbstractFile,\n    injection: Injection[T, Array[Byte]]\n  ): IdEmbeddingMap[T] = {\n    val diskDb = DBMaker\n      .tempFileDB()\n      .concurrencyScale(32)\n      .fileMmapEnable()\n      .fileMmapEnableIfSupported()\n      .fileMmapPreclearDisable()\n      .cleanerHackEnable()\n      .closeOnJvmShutdown()\n      .make()\n\n    val mapDb = diskDb\n      .hashMap(\"mapdb\", Serializer.BYTE_ARRAY, Serializer.FLOAT_ARRAY)\n      .createOrOpen()\n\n    HnswIOUtil.loadEmbeddings(\n      embeddingFile,\n      injection,\n      new MapDbBasedIdEmbeddingMap(mapDb, injection)\n    )\n  }\n}\n\nprivate[this] class MapDbBasedIdEmbeddingMap[T](\n  mapDb: HTreeMap[Array[Byte], Array[Float]],\n  injection: Injection[T, Array[Byte]])\n    extends IdEmbeddingMap[T] {\n  override def putIfAbsent(id: T, embedding: EmbeddingVector): EmbeddingVector = {\n    val value = mapDb.putIfAbsent(injection.apply(id), embedding.toArray)\n    if (value == null) null else Embedding(value)\n  }\n\n  override def put(id: T, embedding: EmbeddingVector): EmbeddingVector = {\n    val value = mapDb.put(injection.apply(id), embedding.toArray)\n    if (value == null) null else Embedding(value)\n  }\n\n  override def get(id: T): EmbeddingVector = {\n    Embedding(mapDb.get(injection.apply(id)))\n  }\n\n  override def iter(): Iterator[(T, EmbeddingVector)] = {\n    mapDb\n      .entrySet()\n      .iterator()\n      .asScala\n      .map(entry => (injection.invert(entry.getKey).get, Embedding(entry.getValue)))\n  }\n\n  override def size(): Int = mapDb.size()\n\n  override def toDirectory(embeddingFile: OutputStream): Unit = {\n    HnswIOUtil.saveEmbeddings(embeddingFile, injection, iter())\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/hnsw/SerializableHnsw.scala",
    "content": "package com.twitter.ann.hnsw\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.ann.common._\nimport com.twitter.ann.common.thriftscala.HnswIndexMetadata\nimport com.twitter.ann.hnsw.HnswCommon._\nimport com.twitter.ann.hnsw.HnswIndex.RandomProvider\nimport com.twitter.bijection.Injection\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.search.common.file.FileUtils\nimport com.twitter.util.Future\nimport java.io.IOException\nimport java.util.concurrent.ThreadLocalRandom\nimport java.util.Random\nimport org.apache.beam.sdk.io.fs.ResourceId\n\nprivate[hnsw] object SerializableHnsw {\n  private[hnsw] def apply[T, D <: Distance[D]](\n    index: Hnsw[T, D],\n    injection: Injection[T, Array[Byte]]\n  ): SerializableHnsw[T, D] = {\n    new SerializableHnsw[T, D](\n      index,\n      injection\n    )\n  }\n\n  private[hnsw] def loadMapBasedQueryableIndex[T, D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    injection: Injection[T, Array[Byte]],\n    futurePool: ReadWriteFuturePool,\n    directory: AbstractFile\n  ): SerializableHnsw[T, D] = {\n    val metadata = HnswIOUtil.loadIndexMetadata(directory.getChild(MetaDataFileName))\n    validateMetadata(dimension, metric, metadata)\n    val idEmbeddingMap = JMapBasedIdEmbeddingMap.loadInMemory(\n      directory.getChild(EmbeddingMappingFileName),\n      injection,\n      Some(metadata.numElements)\n    )\n    loadIndex(\n      dimension,\n      metric,\n      injection,\n      futurePool,\n      directory,\n      idEmbeddingMap,\n      metadata\n    )\n  }\n\n  private[hnsw] def loadMMappedBasedQueryableIndex[T, D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    injection: Injection[T, Array[Byte]],\n    futurePool: ReadWriteFuturePool,\n    directory: AbstractFile\n  ): SerializableHnsw[T, D] = {\n    val metadata = HnswIOUtil.loadIndexMetadata(directory.getChild(MetaDataFileName))\n    validateMetadata(dimension, metric, metadata)\n    loadIndex(\n      dimension,\n      metric,\n      injection,\n      futurePool,\n      directory,\n      MapDbBasedIdEmbeddingMap\n        .loadAsReadonly(directory.getChild(EmbeddingMappingFileName), injection),\n      metadata\n    )\n  }\n\n  private[hnsw] def loadIndex[T, D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    injection: Injection[T, Array[Byte]],\n    futurePool: ReadWriteFuturePool,\n    directory: AbstractFile,\n    idEmbeddingMap: IdEmbeddingMap[T],\n    metadata: HnswIndexMetadata\n  ): SerializableHnsw[T, D] = {\n    val distFn =\n      DistanceFunctionGenerator(metric, (key: T) => idEmbeddingMap.get(key))\n    val randomProvider = new RandomProvider {\n      override def get(): Random = ThreadLocalRandom.current()\n    }\n    val internalIndex = HnswIndex.loadHnswIndex[T, EmbeddingVector](\n      distFn.index,\n      distFn.query,\n      directory.getChild(InternalIndexDir),\n      injection,\n      randomProvider\n    )\n\n    val index = new Hnsw[T, D](\n      dimension,\n      metric,\n      internalIndex,\n      futurePool,\n      idEmbeddingMap,\n      distFn.shouldNormalize,\n      LockedAccess.apply(metadata.numElements)\n    )\n\n    new SerializableHnsw(index, injection)\n  }\n\n  private[this] def validateMetadata[D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    existingMetadata: HnswIndexMetadata\n  ): Unit = {\n    assert(\n      existingMetadata.dimension == dimension,\n      s\"Dimensions do not match. requested: $dimension existing: ${existingMetadata.dimension}\"\n    )\n\n    val existingMetric = Metric.fromThrift(existingMetadata.distanceMetric)\n    assert(\n      existingMetric == metric,\n      s\"DistanceMetric do not match. requested: $metric existing: $existingMetric\"\n    )\n  }\n}\n\n@VisibleForTesting\nprivate[hnsw] class SerializableHnsw[T, D <: Distance[D]](\n  index: Hnsw[T, D],\n  injection: Injection[T, Array[Byte]])\n    extends Appendable[T, HnswParams, D]\n    with Queryable[T, HnswParams, D]\n    with Serialization\n    with Updatable[T] {\n  override def append(entity: EntityEmbedding[T]) = index.append(entity)\n\n  override def toQueryable: Queryable[T, HnswParams, D] = index.toQueryable\n\n  override def query(\n    embedding: EmbeddingVector,\n    numOfNeighbours: Int,\n    runtimeParams: HnswParams\n  ) = index.query(embedding, numOfNeighbours, runtimeParams)\n\n  override def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbours: Int,\n    runtimeParams: HnswParams\n  ) = index.queryWithDistance(embedding, numOfNeighbours, runtimeParams)\n\n  def toDirectory(directory: ResourceId): Unit = {\n    toDirectory(new IndexOutputFile(directory))\n  }\n\n  def toDirectory(directory: AbstractFile): Unit = {\n    // Create a temp dir with time prefix, and then do a rename after serialization\n    val tmpDir = FileUtils.getTmpFileHandle(directory)\n    if (!tmpDir.exists()) {\n      tmpDir.mkdirs()\n    }\n\n    toDirectory(new IndexOutputFile(tmpDir))\n\n    // Rename tmp dir to original directory supplied\n    if (!tmpDir.rename(directory)) {\n      throw new IOException(s\"Failed to rename ${tmpDir.getPath} to ${directory.getPath}\")\n    }\n  }\n\n  private def toDirectory(indexFile: IndexOutputFile): Unit = {\n    // Save java based hnsw index\n    index.getIndex.toDirectory(indexFile.createDirectory(InternalIndexDir), injection)\n\n    // Save index metadata\n    HnswIOUtil.saveIndexMetadata(\n      index.getDimen,\n      index.getMetric,\n      index.getIdEmbeddingMap.size(),\n      indexFile.createFile(MetaDataFileName).getOutputStream()\n    )\n\n    // Save embedding mapping\n    index.getIdEmbeddingMap.toDirectory(\n      indexFile.createFile(EmbeddingMappingFileName).getOutputStream())\n\n    // Create _SUCCESS file\n    indexFile.createSuccessFile()\n  }\n\n  override def update(\n    entity: EntityEmbedding[T]\n  ): Future[Unit] = {\n    index.update(entity)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/hnsw/TypedHnswIndex.scala",
    "content": "package com.twitter.ann.hnsw\n\nimport com.twitter.ann.common._\nimport com.twitter.bijection.Injection\nimport com.twitter.search.common.file.AbstractFile\n\n// Class to provide HNSW based approximate nearest neighbour index\nobject TypedHnswIndex {\n\n  /**\n   * Creates in-memory HNSW based index which supports querying/addition/updates of the entity embeddings.\n   * See https://docbird.twitter.biz/ann/hnsw.html to check information about arguments.\n   *\n   * @param dimension Dimension of the embedding to be indexed\n   * @param metric Distance metric (InnerProduct/Cosine/L2)\n   * @param efConstruction The parameter has the same meaning as ef, but controls the\n   *                       index_time/index_accuracy ratio. Bigger ef_construction leads to longer\n   *                       construction, but better index quality. At some point, increasing\n   *                       ef_construction does not improve the quality of the index. One way to\n   *                       check if the selection of ef_construction was ok is to measure a recall\n   *                       for M nearest neighbor search when ef = ef_constuction: if the recall is\n   *                       lower than 0.9, than there is room for improvement.\n   * @param maxM The number of bi-directional links created for every new element during construction.\n   *             Reasonable range for M is 2-100. Higher M work better on datasets with high\n   *             intrinsic dimensionality and/or high recall, while low M work better for datasets\n   *             with low intrinsic dimensionality and/or low recalls. The parameter also determines\n   *             the algorithm's memory consumption, bigger the param more the memory requirement.\n   *             For high dimensional datasets (word embeddings, good face descriptors), higher M\n   *             are required (e.g. M=48, 64) for optimal performance at high recall.\n   *             The range M=12-48 is ok for the most of the use cases.\n   * @param expectedElements Approximate number of elements to be indexed\n   * @param readWriteFuturePool Future pool for performing read (query) and write operation (addition/updates).\n   * @tparam T Type of item to index\n   * @tparam D Type of distance\n   */\n  def index[T, D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    efConstruction: Int,\n    maxM: Int,\n    expectedElements: Int,\n    readWriteFuturePool: ReadWriteFuturePool\n  ): Appendable[T, HnswParams, D] with Queryable[T, HnswParams, D] with Updatable[T] = {\n    Hnsw[T, D](\n      dimension,\n      metric,\n      efConstruction,\n      maxM,\n      expectedElements,\n      readWriteFuturePool,\n      JMapBasedIdEmbeddingMap.applyInMemory[T](expectedElements)\n    )\n  }\n\n  /**\n   * Creates in-memory HNSW based index which supports querying/addition/updates of the entity embeddings.\n   * It can be serialized to a directory (HDFS/Local file system)\n   * See https://docbird.twitter.biz/ann/hnsw.html to check information about arguments.\n   *\n   * @param dimension Dimension of the embedding to be indexed\n   * @param metric Distance metric (InnerProduct/Cosine/L2)\n   * @param efConstruction The parameter has the same meaning as ef, but controls the\n   *                       index_time/index_accuracy ratio. Bigger ef_construction leads to longer\n   *                       construction, but better index quality. At some point, increasing\n   *                       ef_construction does not improve the quality of the index. One way to\n   *                       check if the selection of ef_construction was ok is to measure a recall\n   *                       for M nearest neighbor search when ef = ef_constuction: if the recall is\n   *                       lower than 0.9, than there is room for improvement.\n   * @param maxM The number of bi-directional links created for every new element during construction.\n   *             Reasonable range for M is 2-100. Higher M work better on datasets with high\n   *             intrinsic dimensionality and/or high recall, while low M work better for datasets\n   *             with low intrinsic dimensionality and/or low recalls. The parameter also determines\n   *             the algorithm's memory consumption, bigger the param more the memory requirement.\n   *             For high dimensional datasets (word embeddings, good face descriptors), higher M\n   *             are required (e.g. M=48, 64) for optimal performance at high recall.\n   *             The range M=12-48 is ok for the most of the use cases.\n   * @param expectedElements Approximate number of elements to be indexed\n   * @param injection Injection for typed Id T to Array[Byte]\n   * @param readWriteFuturePool Future pool for performing read (query) and write operation (addition/updates).\n   * @tparam T Type of item to index\n   * @tparam D Type of distance\n   */\n  def serializableIndex[T, D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    efConstruction: Int,\n    maxM: Int,\n    expectedElements: Int,\n    injection: Injection[T, Array[Byte]],\n    readWriteFuturePool: ReadWriteFuturePool\n  ): Appendable[T, HnswParams, D]\n    with Queryable[T, HnswParams, D]\n    with Updatable[T]\n    with Serialization = {\n    val index = Hnsw[T, D](\n      dimension,\n      metric,\n      efConstruction,\n      maxM,\n      expectedElements,\n      readWriteFuturePool,\n      JMapBasedIdEmbeddingMap\n        .applyInMemoryWithSerialization[T](expectedElements, injection)\n    )\n\n    SerializableHnsw[T, D](\n      index,\n      injection\n    )\n  }\n\n  /**\n   * Loads HNSW index from a directory to in-memory\n   * @param dimension dimension of the embedding to be indexed\n   * @param metric Distance metric\n   * @param readWriteFuturePool Future pool for performing read (query) and write operation (addition/updates).\n   * @param injection : Injection for typed Id T to Array[Byte]\n   * @param directory : Directory(HDFS/Local file system) where hnsw index is stored\n   * @tparam T : Type of item to index\n   * @tparam D : Type of distance\n   */\n  def loadIndex[T, D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    injection: Injection[T, Array[Byte]],\n    readWriteFuturePool: ReadWriteFuturePool,\n    directory: AbstractFile\n  ): Appendable[T, HnswParams, D]\n    with Queryable[T, HnswParams, D]\n    with Updatable[T]\n    with Serialization = {\n    SerializableHnsw.loadMapBasedQueryableIndex[T, D](\n      dimension,\n      metric,\n      injection,\n      readWriteFuturePool,\n      directory\n    )\n  }\n\n  /**\n   * Loads a HNSW index from a directory and memory map it.\n   * It will take less memory but rely more on disk as it leverages memory mapped file backed by disk.\n   * Latency will go up considerably (Could be by factor of > 10x) if used on instance with low\n   * memory since lot of page faults may occur. Best use case to use would with scalding jobs\n   * where mapper/reducers instance are limited by 8gb memory.\n   * @param dimension dimension of the embedding to be indexed\n   * @param metric Distance metric\n   * @param readWriteFuturePool Future pool for performing read (query) and write operation (addition/updates).\n   * @param injection Injection for typed Id T to Array[Byte]\n   * @param directory Directory(HDFS/Local file system) where hnsw index is stored\n   * @tparam T Type of item to index\n   * @tparam D Type of distance\n   */\n  def loadMMappedIndex[T, D <: Distance[D]](\n    dimension: Int,\n    metric: Metric[D],\n    injection: Injection[T, Array[Byte]],\n    readWriteFuturePool: ReadWriteFuturePool,\n    directory: AbstractFile\n  ): Appendable[T, HnswParams, D]\n    with Queryable[T, HnswParams, D]\n    with Updatable[T]\n    with Serialization = {\n    SerializableHnsw.loadMMappedBasedQueryableIndex[T, D](\n      dimension,\n      metric,\n      injection,\n      readWriteFuturePool,\n      directory\n    )\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/manhattan/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/bijection:core\",\n        \"3rdparty/jvm/com/twitter/bijection:scrooge\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"src/scala/com/twitter/ml/api/embedding\",\n        \"storage/clients/manhattan\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/manhattan/ManhattanEmbeddingProducer.scala",
    "content": "package com.twitter.ann.manhattan\n\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.ann.common.{EmbeddingProducer, EmbeddingType}\nimport com.twitter.bijection.Injection\nimport com.twitter.ml.api.embedding.{EmbeddingBijection, EmbeddingSerDe}\nimport com.twitter.ml.api.{thriftscala => thrift}\nimport com.twitter.stitch.Stitch\nimport com.twitter.storage.client.manhattan.bijections.Bijections\nimport com.twitter.storage.client.manhattan.bijections.Bijections.BinaryScalaInjection\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpoint\nimport com.twitter.storage.client.manhattan.kv.impl.{\n  DescriptorP1L0,\n  ReadOnlyKeyDescriptor,\n  ValueDescriptor\n}\n\nprivate[manhattan] class ManhattanEmbeddingProducer[T](\n  keyDescriptor: DescriptorP1L0.DKey[T],\n  valueDescriptor: ValueDescriptor.EmptyValue[EmbeddingVector],\n  manhattanEndpoint: ManhattanKVEndpoint)\n    extends EmbeddingProducer[T] {\n\n  /**\n   * Lookup an embedding from manhattan given a key of type T.\n   *\n   * @return An embedding stitch.\n   *         An easy way to get a Future from a Stitch is to run Stitch.run(stitch)\n   */\n  override def produceEmbedding(input: T): Stitch[Option[EmbeddingVector]] = {\n    val fullKey = keyDescriptor.withPkey(input)\n    val stitchResult = manhattanEndpoint.get(fullKey, valueDescriptor)\n    stitchResult.map { resultOption =>\n      resultOption.map(_.contents)\n    }\n  }\n}\n\nobject ManhattanEmbeddingProducer {\n  private[manhattan] def keyDescriptor[T](\n    injection: Injection[T, Array[Byte]],\n    dataset: String\n  ): DescriptorP1L0.DKey[T] =\n    ReadOnlyKeyDescriptor(injection.andThen(Bijections.BytesBijection))\n      .withDataset(dataset)\n\n  private[manhattan] val EmbeddingDescriptor: ValueDescriptor.EmptyValue[\n    EmbeddingType.EmbeddingVector\n  ] = {\n    val embeddingBijection = new EmbeddingBijection(EmbeddingSerDe.floatEmbeddingSerDe)\n    val thriftInjection = BinaryScalaInjection[thrift.Embedding](thrift.Embedding)\n    ValueDescriptor(embeddingBijection.andThen(thriftInjection))\n  }\n\n  def apply[T](\n    dataset: String,\n    injection: Injection[T, Array[Byte]],\n    manhattanEndpoint: ManhattanKVEndpoint\n  ): EmbeddingProducer[T] = {\n    val descriptor = keyDescriptor(injection, dataset)\n    new ManhattanEmbeddingProducer(descriptor, EmbeddingDescriptor, manhattanEndpoint)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/manhattan/README",
    "content": "# Description\n\nThe ManhattanEmbeddingProducer is an EmbeddingProducer that is backed by a static manhattan dataset.\n\n# Setting up Data\n\nData needs to be setup correctly in manhattan in order to be able to read the data using the\nManhattanEmbeddingProducer. You can use the EmbeddingSamplingJob to do this. The job can reads\nembedding data from HDFS and re-writes it in the manhattan data format on HDFS."
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/benchmark/BUILD",
    "content": "scala_library(\n    name = \"benchmark\",\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-compatible:migrated\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \":user_item_knn-scala\",\n        \"3rdparty/src/jvm/com/twitter/scalding:args\",\n        \"3rdparty/src/jvm/com/twitter/scalding:core\",\n        \"3rdparty/src/jvm/com/twitter/scalding:date\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/scalding/offline\",\n        \"src/scala/com/twitter/scalding_internal/dalv2\",\n        \"src/scala/com/twitter/scalding_internal/job\",\n        \"src/scala/com/twitter/scalding_internal/job/analytics_batch\",\n        \"src/scala/com/twitter/scalding_internal/multiformat/format\",\n    ],\n)\n\nhadoop_binary(\n    name = \"benchmark-adhoc\",\n    main = \"com.twitter.scalding.Tool\",\n    platform = \"java8\",\n    runtime_platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-compatible:migrated\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \":benchmark\",\n        \"3rdparty/jvm/org/slf4j:slf4j-jdk14\",\n    ],\n)\n\ncreate_datasets(\n    base_name = \"user_item_knn\",\n    description = \"List of the top recommendations per search entity (user)\",\n    java_schema = \"com.twitter.ann.knn.thriftjava.Knn\",\n    platform = \"java8\",\n    role = \"cortex-mlx\",\n    scala_schema = \"com.twitter.ann.knn.thriftscala.Knn\",\n    segment_type = \"partitioned\",\n    tags = [\"bazel-compatible\"],\n    java_dependencies = [\n        \"ann/src/main/thrift/com/twitter/ann/knn:thrift-java\",\n    ],\n    scala_dependencies = [\n        \"ann/src/main/thrift/com/twitter/ann/knn:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/benchmark/Knn.scala",
    "content": "package com.twitter.ann.scalding.offline.com.twitter.ann.scalding.benchmark\n\n/*\nThis job will generate KNN ground truth based user and item embeddings.\n */\n\nimport com.twitter.scalding.typed.TypedPipe\nimport com.twitter.scalding._\nimport com.twitter.scalding_internal.dalv2.DALWrite.D\nimport com.twitter.ann.knn.thriftscala.Knn\nimport com.twitter.ann.knn.thriftscala.Neighbor\nimport com.twitter.ann.scalding.offline.IndexingStrategy\nimport com.twitter.ann.scalding.offline.KnnHelper\nimport com.twitter.ann.common.Distance\nimport com.twitter.ml.featurestore.lib.embedding.EmbeddingWithEntity\nimport com.twitter.cortex.ml.embeddings.common.EmbeddingFormatArgsParser\nimport com.twitter.cortex.ml.embeddings.common.EntityKind\nimport java.util.TimeZone\nimport com.twitter.scalding_internal.dalv2.DALWrite._\nimport com.twitter.ann.scalding.benchmark.UserItemKnnScalaDataset\nimport com.twitter.scalding_internal.job.TwitterExecutionApp\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.UserId\n\n/**\n * This job will take consumer and item embeddings(either url or tweet) and output Knn entities (user id, (distance, item id)).\n *\n * Example command to run this adhoc job:\n *\n * scalding remote run \\\n * --target ann/src/main/scala/com/twitter/ann/scalding/benchmark:benchmark-adhoc \\\n * --hadoop-properties \"mapreduce.map.memory.mb=8192 mapreduce.map.java.opts='-Xmx7618M' mapreduce.reduce.memory.mb=8192 mapreduce.reduce.java.opts='-Xmx7618M' mapred.task.timeout=0\" \\\n * --submitter hadoopnest3.smf1.twitter.com \\\n * --user cortex-mlx \\\n * --submitter-memory 8000.megabyte \\\n * --main-class com.twitter.ann.scalding.offline.com.twitter.ann.scalding.benchmark.KnnJob -- \\\n * --dalEnvironment Prod \\\n * --search_space_entity_type user \\\n * --user.feature_store_embedding ConsumerFollowEmbedding300Dataset \\\n * --user.feature_store_major_version 1569196895 \\\n * --user.date_range 2019-10-23 \\\n * --search_space.feature_store_embedding ConsumerFollowEmbedding300Dataset \\\n * --search_space.feature_store_major_version 1569196895 \\\n * --search_space.date_range 2019-10-23 \\\n * --date 2019-10-25 \\\n * --version \"consumer_follower_test\" \\\n * --reducers 10000 \\\n * --num_of_random_groups 20 \\\n * --num_replicas 1000 \\\n * --indexing_strategy.metric InnerProduct \\\n * --indexing_strategy.type hnsw \\\n * --indexing_strategy.dimension 300 \\\n * --indexing_strategy.ef_construction 30 \\\n * --indexing_strategy.max_m 10 \\\n * --indexing_strategy.ef_query 50 \\\n * --search_space_shards 3000 \\\n * --query_shards 3000 \\\n * --search_space.read_sample_ratio 0.038\n */\ntrait KnnJobBase {\n  val seed: Long = 123\n\n  def getKnnDataset[B <: EntityId, D <: Distance[D]](\n    args: Args\n  )(\n    implicit uniqueID: UniqueID\n  ): TypedPipe[Knn] = {\n\n    val consumerPipe: TypedPipe[EmbeddingWithEntity[UserId]] = EmbeddingFormatArgsParser.User\n      .getEmbeddingFormat(args, \"user\")\n      .getEmbeddings\n\n    val itemPipe = EntityKind\n      .getEntityKind(args(\"search_space_entity_type\"))\n      .parser\n      .getEmbeddingFormat(args, \"search_space\")\n      .getEmbeddings\n\n    KnnHelper\n    // Refer to the documentation of findNearestNeighboursWithIndexingStrategy for more\n    // information about how to set these settings.\n      .findNearestNeighboursWithIndexingStrategy[UserId, B, D](\n        queryEmbeddings = consumerPipe,\n        searchSpaceEmbeddings = itemPipe.asInstanceOf[TypedPipe[EmbeddingWithEntity[B]]],\n        numNeighbors = args.int(\"candidate_per_user\", 20),\n        reducersOption = args.optional(\"reducers\").map(_.toInt),\n        numOfSearchGroups = args.int(\"num_of_random_groups\"),\n        numReplicas = args.int(\"num_replicas\"),\n        indexingStrategy = IndexingStrategy.parse(args).asInstanceOf[IndexingStrategy[D]],\n        queryShards = args.optional(\"query_shards\").map(_.toInt),\n        searchSpaceShards = args.optional(\"search_space_shards\").map(_.toInt)\n      )\n      .map {\n        case (user, items) =>\n          val neighbors = items.map {\n            case (item, distance) =>\n              Neighbor(\n                distance.distance,\n                item.toThrift\n              )\n          }\n          Knn(user.toThrift, neighbors)\n      }\n  }\n}\n\nobject KnnJob extends TwitterExecutionApp with KnnJobBase {\n\n  val KnnPathSuffix: String = \"/user/cortex-mlx/qualatative_analysis/knn_ground_truth/\"\n  val partitionKey: String = \"version\"\n\n  override def job: Execution[Unit] = Execution.withId { implicit uniqueId =>\n    Execution.getArgs.flatMap { args: Args =>\n      implicit val timeZone: TimeZone = TimeZone.getDefault\n      implicit val dateParser: DateParser = DateParser.default\n      implicit val dateRange: DateRange = DateRange.parse(args.list(\"date\"))(timeZone, dateParser)\n\n      getKnnDataset(args).writeDALExecution(\n        UserItemKnnScalaDataset,\n        D.Daily,\n        D.Suffix(KnnPathSuffix),\n        D.Parquet,\n        Set(D.Partition(partitionKey, args(\"version\"), D.PartitionType.String))\n      )\n    }\n  }\n\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/bijection:scrooge\",\n        \"3rdparty/src/jvm/com/twitter/scalding:args\",\n        \"3rdparty/src/jvm/com/twitter/scalding:commons\",\n        \"3rdparty/src/jvm/com/twitter/scalding:core\",\n        \"ann/src/main/scala/com/twitter/ann/brute_force\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/hnsw\",\n        \"ann/src/main/scala/com/twitter/ann/util\",\n        \"cortex-core/entity-embeddings/src/thrift/com/twitter/entityembeddings/neighbors:embeddings-knn-thrift-scala\",\n        \"src/scala/com/twitter/cortex/ml/embeddings/common:Helpers-deploy\",\n        \"src/scala/com/twitter/pluck/source/core_workflows/user_model:condensed_user_state-scala\",\n        \"src/scala/com/twitter/scalding_internal/dalv2\",\n        \"src/scala/com/twitter/scalding_internal/job\",\n        \"src/scala/com/twitter/scalding_internal/multiformat/format\",\n        \"src/scala/com/twitter/scalding_internal/parquet_thrift\",\n        \"usersource/snapshot/src/main/scala/com/twitter/usersource/snapshot/flat:usersource_flat-scala\",\n        \"usersource/snapshot/src/main/thrift/com/twitter/usersource/snapshot/flat:flat-scala\",\n    ],\n)\n\nhadoop_binary(\n    name = \"ann-offline-deploy\",\n    main = \"com.twitter.scalding.Tool\",\n    platform = \"java8\",\n    runtime_platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-compatible:migrated\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \":offline\",\n        \"3rdparty/jvm/org/slf4j:slf4j-jdk14\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/IndexingStrategy.scala",
    "content": "package com.twitter.ann.scalding.offline\n\nimport com.twitter.ann.brute_force.{BruteForceIndex, BruteForceRuntimeParams}\nimport com.twitter.ann.common.{Distance, EntityEmbedding, Metric, ReadWriteFuturePool}\nimport com.twitter.ann.hnsw.{HnswParams, TypedHnswIndex}\nimport com.twitter.ann.util.IndexBuilderUtils\nimport com.twitter.scalding.Args\nimport com.twitter.util.logging.Logger\nimport com.twitter.util.{Await, FuturePool}\n\n/**\n * IndexingStrategy is used for determining how we will build the index when doing a KNN in\n * scalding. Right now there are 2 strategies a BruteForce and HNSW strategy.\n * @tparam D distance that the index uses.\n */\nsealed trait IndexingStrategy[D <: Distance[D]] {\n  private[offline] def buildIndex[T](\n    indexItems: TraversableOnce[EntityEmbedding[T]]\n  ): ParameterlessQueryable[T, _, D]\n}\n\nobject IndexingStrategy {\n\n  /**\n   * Parse an indexing strategy from scalding args.\n   * ${argumentName}.type Is hsnw or brute_force\n   * ${argumentName}.type is the metric to use. See Metric.fromString for options.\n   *\n   * hsnw has these additional parameters:\n   * ${argumentName}.dimension the number of dimension for the embeddings.\n   * ${argumentName}.ef_construction, ${argumentName}.ef_construction and ${argumentName}.ef_query.\n   * See TypedHnswIndex for more details on these parameters.\n   * @param args scalding arguments to parse.\n   * @param argumentName A specifier to use in case you want to parse more than one indexing\n   *                     strategy. indexing_strategy by default.\n   * @return parse indexing strategy\n   */\n  def parse(\n    args: Args,\n    argumentName: String = \"indexing_strategy\"\n  ): IndexingStrategy[_] = {\n    def metricArg[D <: Distance[D]] =\n      Metric.fromString(args(s\"$argumentName.metric\")).asInstanceOf[Metric[D]]\n\n    args(s\"$argumentName.type\") match {\n      case \"brute_force\" =>\n        BruteForceIndexingStrategy(metricArg)\n      case \"hnsw\" =>\n        val dimensionArg = args.int(s\"$argumentName.dimension\")\n        val efConstructionArg = args.int(s\"$argumentName.ef_construction\")\n        val maxMArg = args.int(s\"$argumentName.max_m\")\n        val efQuery = args.int(s\"$argumentName.ef_query\")\n        HnswIndexingStrategy(\n          dimension = dimensionArg,\n          metric = metricArg,\n          efConstruction = efConstructionArg,\n          maxM = maxMArg,\n          hnswParams = HnswParams(efQuery)\n        )\n    }\n  }\n}\n\ncase class BruteForceIndexingStrategy[D <: Distance[D]](metric: Metric[D])\n    extends IndexingStrategy[D] {\n  private[offline] def buildIndex[T](\n    indexItems: TraversableOnce[EntityEmbedding[T]]\n  ): ParameterlessQueryable[T, _, D] = {\n    val appendable = BruteForceIndex[T, D](metric, FuturePool.immediatePool)\n    indexItems.foreach { item =>\n      Await.result(appendable.append(item))\n    }\n    val queryable = appendable.toQueryable\n    ParameterlessQueryable[T, BruteForceRuntimeParams.type, D](\n      queryable,\n      BruteForceRuntimeParams\n    )\n  }\n}\n\ncase class HnswIndexingStrategy[D <: Distance[D]](\n  dimension: Int,\n  metric: Metric[D],\n  efConstruction: Int,\n  maxM: Int,\n  hnswParams: HnswParams,\n  concurrencyLevel: Int = 1)\n    extends IndexingStrategy[D] {\n  private[offline] def buildIndex[T](\n    indexItems: TraversableOnce[EntityEmbedding[T]]\n  ): ParameterlessQueryable[T, _, D] = {\n\n    val log: Logger = Logger(getClass)\n    val appendable = TypedHnswIndex.index[T, D](\n      dimension = dimension,\n      metric = metric,\n      efConstruction = efConstruction,\n      maxM = maxM,\n      // This is not really that important.\n      expectedElements = 1000,\n      readWriteFuturePool = ReadWriteFuturePool(FuturePool.immediatePool)\n    )\n    val future =\n      IndexBuilderUtils\n        .addToIndex(appendable, indexItems.toStream, concurrencyLevel)\n        .map { numberUpdates =>\n          log.info(s\"Performed $numberUpdates updates\")\n        }\n    Await.result(future)\n    val queryable = appendable.toQueryable\n    ParameterlessQueryable(\n      queryable,\n      hnswParams\n    )\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/KnnDebug.scala",
    "content": "package com.twitter.ann.scalding.offline\n\nimport com.twitter.core_workflows.user_model.thriftscala.CondensedUserState\nimport com.twitter.cortex.ml.embeddings.common.{DataSourceManager, GraphEdge, Helpers, UserKind}\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.entityembeddings.neighbors.thriftscala.{EntityKey, NearestNeighbors}\nimport com.twitter.pluck.source.core_workflows.user_model.CondensedUserStateScalaDataset\nimport com.twitter.scalding._\nimport com.twitter.scalding.typed.TypedPipe\nimport com.twitter.scalding_internal.dalv2.DAL\nimport com.twitter.usersource.snapshot.flat.UsersourceFlatScalaDataset\nimport com.twitter.usersource.snapshot.flat.thriftscala.FlatUser\n\ncase class ConsumerAssoc(consumerId: UserId, assoc: List[String])\n\nobject KnnDebug {\n\n  def getConsumerAssociations(\n    graph: TypedPipe[GraphEdge[UserId, UserId]],\n    usernames: TypedPipe[(UserId, String)],\n    reducers: Int\n  ): TypedPipe[ConsumerAssoc] = {\n    graph\n      .groupBy(_.itemId)\n      .join(usernames).withReducers(reducers)\n      .values\n      .map {\n        case (edge: GraphEdge[UserId, UserId], producerScreenName: String) =>\n          ConsumerAssoc(consumerId = edge.consumerId, assoc = List(producerScreenName))\n      }\n      .groupBy(_.consumerId).withReducers(reducers)\n      .reduce[ConsumerAssoc] {\n        case (uFollow1: ConsumerAssoc, uFollow2: ConsumerAssoc) =>\n          ConsumerAssoc(consumerId = uFollow1.consumerId, assoc = uFollow1.assoc ++ uFollow2.assoc)\n      }\n      .values\n  }\n\n  /**\n   * Write the neighbors and a set of follows to a tsv for easier analysis during debugging\n   * We take the set of users with between 25-50 follows and grab only those users\n   *\n   * This returns 4 strings of the form:\n   * consumerId, state, followUserName<f>followUserName<f>followUserName, neighborName<n>neighborName<n>neighborName\n   */\n  def getDebugTable(\n    neighborsPipe: TypedPipe[(EntityKey, NearestNeighbors)],\n    shards: Int,\n    reducers: Int,\n    limit: Int = 10000,\n    userDataset: Option[TypedPipe[FlatUser]] = None,\n    followDataset: Option[TypedPipe[GraphEdge[UserId, UserId]]] = None,\n    consumerStatesDataset: Option[TypedPipe[CondensedUserState]] = None,\n    minFollows: Int = 25,\n    maxFollows: Int = 50\n  )(\n    implicit dateRange: DateRange\n  ): TypedPipe[(String, String, String, String)] = {\n\n    val usersourcePipe: TypedPipe[FlatUser] = userDataset\n      .getOrElse(DAL.readMostRecentSnapshot(UsersourceFlatScalaDataset, dateRange).toTypedPipe)\n\n    val followGraph: TypedPipe[GraphEdge[UserId, UserId]] = followDataset\n      .getOrElse(new DataSourceManager().getFollowGraph())\n\n    val consumerStates: TypedPipe[CondensedUserState] = consumerStatesDataset\n      .getOrElse(DAL.read(CondensedUserStateScalaDataset).toTypedPipe)\n\n    val usernames: TypedPipe[(UserId, String)] = usersourcePipe.flatMap { flatUser =>\n      (flatUser.screenName, flatUser.id) match {\n        case (Some(name: String), Some(userId: Long)) => Some((UserId(userId), name))\n        case _ => None\n      }\n    }.fork\n\n    val consumerFollows: TypedPipe[ConsumerAssoc] =\n      getConsumerAssociations(followGraph, usernames, reducers)\n        .filter { uFollow => (uFollow.assoc.size > minFollows && uFollow.assoc.size < maxFollows) }\n\n    val neighborGraph: TypedPipe[GraphEdge[UserId, UserId]] = neighborsPipe\n      .limit(limit)\n      .flatMap {\n        case (entityKey: EntityKey, neighbors: NearestNeighbors) =>\n          Helpers.optionalToLong(entityKey.id) match {\n            case Some(entityId: Long) =>\n              neighbors.neighbors.flatMap { neighbor =>\n                Helpers\n                  .optionalToLong(neighbor.neighbor.id)\n                  .map { neighborId =>\n                    GraphEdge[UserId, UserId](\n                      consumerId = UserId(entityId),\n                      itemId = UserId(neighborId),\n                      weight = 1.0F)\n                  }\n              }\n            case None => List()\n          }\n      }\n    val consumerNeighbors: TypedPipe[ConsumerAssoc] =\n      getConsumerAssociations(neighborGraph, usernames, reducers)\n\n    consumerFollows\n      .groupBy(_.consumerId)\n      .join(consumerStates.groupBy { consumer => UserId(consumer.uid) }).withReducers(reducers)\n      .join(consumerNeighbors.groupBy(_.consumerId)).withReducers(reducers)\n      .values\n      .map {\n        case ((uFollow: ConsumerAssoc, state: CondensedUserState), uNeighbors: ConsumerAssoc) =>\n          (\n            UserKind.stringInjection(uFollow.consumerId),\n            state.state.toString,\n            uFollow.assoc mkString \"<f>\",\n            uNeighbors.assoc mkString \"<n>\")\n      }\n      .shard(shards)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/KnnEntityRecoDebugJob.scala",
    "content": "package com.twitter.ann.scalding.offline\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.Metric\nimport com.twitter.cortex.ml.embeddings.common.EntityKind\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.scalding.typed.TypedPipe\nimport com.twitter.scalding._\nimport com.twitter.scalding_internal.job.TwitterExecutionApp\n\n/**\n * This job do an exhaustive search for nearest neighbours helpful for debugging recommendations\n * for a given list of sample queryIds and entity embeddings for the recos to be made.\n * Sample job script:\n  ./bazel bundle ann/src/main/scala/com/twitter/ann/scalding/offline:ann-offline-deploy\n\n  oscar hdfs \\\n  --screen --tee log.txt \\\n  --hadoop-client-memory 6000 \\\n  --hadoop-properties \"yarn.app.mapreduce.am.resource.mb=6000;yarn.app.mapreduce.am.command-opts='-Xmx7500m';mapreduce.map.memory.mb=7500;mapreduce.reduce.java.opts='-Xmx6000m';mapreduce.reduce.memory.mb=7500;mapred.task.timeout=36000000;\" \\\n  --bundle ann-offline-deploy \\\n  --min-split-size 284217728 \\\n  --host hadoopnest1.smf1.twitter.com \\\n  --tool com.twitter.ann.scalding.offline.KnnEntityRecoDebugJob -- \\\n  --neighbors 10 \\\n  --metric InnerProduct \\\n  --query_entity_kind user \\\n  --search_space_entity_kind user \\\n  --query.embedding_path /user/apoorvs/sample_embeddings \\\n  --query.embedding_format tab \\\n  --search_space.embedding_path /user/apoorvs/sample_embeddings \\\n  --search_space.embedding_format tab \\\n  --query_ids 974308319300149248 988871266244464640 2719685122 2489777564 \\\n  --output_path /user/apoorvs/adhochadoop/test \\\n  --reducers 100\n */\nobject KnnEntityRecoDebugJob extends TwitterExecutionApp {\n  override def job: Execution[Unit] = Execution.withId { implicit uniqueId =>\n    Execution.getArgs.flatMap { args: Args =>\n      val queryEntityKind = EntityKind.getEntityKind(args(\"query_entity_kind\"))\n      val searchSpaceEntityKind = EntityKind.getEntityKind(args(\"search_space_entity_kind\"))\n      val metric = Metric.fromString(args(\"metric\"))\n      run(queryEntityKind, searchSpaceEntityKind, metric, args)\n    }\n  }\n\n  private[this] def run[A <: EntityId, B <: EntityId, D <: Distance[D]](\n    uncastQueryEntityKind: EntityKind[_],\n    uncastSearchSpaceEntityKind: EntityKind[_],\n    uncastMetric: Metric[_],\n    args: Args\n  )(\n    implicit uniqueID: UniqueID\n  ): Execution[Unit] = {\n    import KnnHelper._\n\n    val numNeighbors = args.int(\"neighbors\")\n    val reducers = args.getOrElse(\"reducers\", \"100\").toInt\n\n    val queryEntityKind = uncastQueryEntityKind.asInstanceOf[EntityKind[A]]\n    val searchSpaceEntityKind = uncastSearchSpaceEntityKind.asInstanceOf[EntityKind[B]]\n    val metric = uncastMetric.asInstanceOf[Metric[D]]\n\n    // Filter the query entity embeddings with the queryIds\n    val queryIds = args.list(\"query_ids\")\n    assert(queryIds.nonEmpty)\n    val filterQueryIds: TypedPipe[A] = TypedPipe\n      .from(queryIds)\n      .map(queryEntityKind.stringInjection.invert(_).get)\n    val queryEmbeddings = queryEntityKind.parser.getEmbeddingFormat(args, \"query\").getEmbeddings\n\n    // Get the neighbour embeddings\n    val searchSpaceEmbeddings =\n      searchSpaceEntityKind.parser.getEmbeddingFormat(args, \"search_space\").getEmbeddings\n\n    val nearestNeighborString = findNearestNeighbours(\n      queryEmbeddings,\n      searchSpaceEmbeddings,\n      metric,\n      numNeighbors,\n      Some(filterQueryIds),\n      reducers\n    )(queryEntityKind.ordering, uniqueID).map(\n      nearestNeighborsToString(_, queryEntityKind, searchSpaceEntityKind)\n    )\n\n    // Write the nearest neighbor string to one part file.\n    nearestNeighborString\n      .shard(1)\n      .writeExecution(TypedTsv(args(\"output_path\")))\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/KnnHelper.scala",
    "content": "package com.twitter.ann.scalding.offline\n\nimport com.twitter.ann.common._\nimport com.twitter.ann.hnsw.{HnswParams, TypedHnswIndex}\nimport com.twitter.bijection.Injection\nimport com.twitter.cortex.ml.embeddings.common.{EntityKind, Helpers, UserKind}\nimport com.twitter.entityembeddings.neighbors.thriftscala.{EntityKey, NearestNeighbors, Neighbor}\nimport com.twitter.ml.api.embedding.Embedding\nimport com.twitter.ml.api.embedding.EmbeddingMath.{Float => math}\nimport com.twitter.ml.featurestore.lib.embedding.EmbeddingWithEntity\nimport com.twitter.ml.featurestore.lib.{EntityId, UserId}\nimport com.twitter.scalding.typed.{TypedPipe, UnsortedGrouped}\nimport com.twitter.scalding.{Args, DateRange, Stat, TextLine, UniqueID}\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.util.{Await, FuturePool}\nimport scala.util.Random\n\ncase class Index[T, D <: Distance[D]](\n  injection: Injection[T, Array[Byte]],\n  metric: Metric[D],\n  dimension: Int,\n  directory: AbstractFile) {\n  lazy val annIndex = TypedHnswIndex.loadIndex[T, D](\n    dimension,\n    metric,\n    injection,\n    ReadWriteFuturePool(FuturePool.immediatePool),\n    directory\n  )\n}\n\nobject KnnHelper {\n  def getFilteredUserEmbeddings(\n    args: Args,\n    filterPath: Option[String],\n    reducers: Int,\n    useHashJoin: Boolean\n  )(\n    implicit dateRange: DateRange\n  ): TypedPipe[EmbeddingWithEntity[UserId]] = {\n    val userEmbeddings: TypedPipe[EmbeddingWithEntity[UserId]] =\n      UserKind.parser.getEmbeddingFormat(args, \"consumer\").getEmbeddings\n    filterPath match {\n      case Some(fileName: String) =>\n        val filterUserIds: TypedPipe[UserId] = TypedPipe\n          .from(TextLine(fileName))\n          .flatMap { idLine =>\n            Helpers.optionalToLong(idLine)\n          }\n          .map { id =>\n            UserId(id)\n          }\n        Helpers\n          .adjustableJoin(\n            left = userEmbeddings.groupBy(_.entityId),\n            right = filterUserIds.asKeys,\n            useHashJoin = useHashJoin,\n            reducers = Some(reducers)\n          ).map {\n            case (_, (embedding, _)) => embedding\n          }\n      case None => userEmbeddings\n    }\n  }\n\n  def getNeighborsPipe[T <: EntityId, D <: Distance[D]](\n    args: Args,\n    uncastEntityKind: EntityKind[_],\n    uncastMetric: Metric[_],\n    ef: Int,\n    consumerEmbeddings: TypedPipe[EmbeddingWithEntity[UserId]],\n    abstractFile: Option[AbstractFile],\n    reducers: Int,\n    numNeighbors: Int,\n    dimension: Int\n  )(\n    implicit dateRange: DateRange\n  ): TypedPipe[(EntityKey, NearestNeighbors)] = {\n    val entityKind = uncastEntityKind.asInstanceOf[EntityKind[T]]\n    val injection = entityKind.byteInjection\n    val metric = uncastMetric.asInstanceOf[Metric[D]]\n    abstractFile match {\n      case Some(directory: AbstractFile) =>\n        val index = Index(injection, metric, dimension, directory)\n        consumerEmbeddings\n          .map { embedding =>\n            val knn = Await.result(\n              index.annIndex.queryWithDistance(\n                Embedding(embedding.embedding.toArray),\n                numNeighbors,\n                HnswParams(ef)\n              )\n            )\n            val neighborList = knn\n              .filter(_.neighbor.toString != embedding.entityId.userId.toString)\n              .map(nn =>\n                Neighbor(\n                  neighbor = EntityKey(nn.neighbor.toString),\n                  similarity = Some(1 - nn.distance.distance)))\n            EntityKey(embedding.entityId.toString) -> NearestNeighbors(neighborList)\n          }\n      case None =>\n        val producerEmbeddings: TypedPipe[EmbeddingWithEntity[UserId]] =\n          UserKind.parser.getEmbeddingFormat(args, \"producer\").getEmbeddings\n\n        bruteForceNearestNeighbors(\n          consumerEmbeddings,\n          producerEmbeddings,\n          numNeighbors,\n          reducers\n        )\n    }\n  }\n\n  def bruteForceNearestNeighbors(\n    consumerEmbeddings: TypedPipe[EmbeddingWithEntity[UserId]],\n    producerEmbeddings: TypedPipe[EmbeddingWithEntity[UserId]],\n    numNeighbors: Int,\n    reducers: Int\n  ): TypedPipe[(EntityKey, NearestNeighbors)] = {\n    consumerEmbeddings\n      .cross(producerEmbeddings)\n      .map {\n        case (cEmbed: EmbeddingWithEntity[UserId], pEmbed: EmbeddingWithEntity[UserId]) =>\n          // Cosine similarity\n          val cEmbedNorm = math.l2Norm(cEmbed.embedding).toFloat\n          val pEmbedNorm = math.l2Norm(pEmbed.embedding).toFloat\n          val distance: Float = -math.dotProduct(\n            (math.scalarProduct(cEmbed.embedding, 1 / cEmbedNorm)),\n            math.scalarProduct(pEmbed.embedding, 1 / pEmbedNorm))\n          (\n            UserKind.stringInjection(cEmbed.entityId),\n            (distance, UserKind.stringInjection(pEmbed.entityId)))\n      }\n      .groupBy(_._1).withReducers(reducers)\n      .sortWithTake(numNeighbors) {\n        case ((_: String, (sim1: Float, _: String)), (_: String, (sim2: Float, _: String))) =>\n          sim1 < sim2\n      }\n      .map {\n        case (consumerId: String, (prodSims: Seq[(String, (Float, String))])) =>\n          EntityKey(consumerId) -> NearestNeighbors(\n            prodSims.map {\n              case (consumerId: String, (sim: Float, prodId: String)) =>\n                Neighbor(neighbor = EntityKey(prodId), similarity = Some(-sim.toDouble))\n            }\n          )\n      }\n  }\n\n  /**\n   * Calculate the nearest neighbors exhaustively between two entity embeddings using one as query and other as the search space.\n   * @param queryEmbeddings entity embeddings for queries\n   * @param searchSpaceEmbeddings entity embeddings for search space\n   * @param metric distance metric\n   * @param numNeighbors number of neighbors\n   * @param queryIdsFilter optional query ids to filter to query entity embeddings\n   * @param reducers number of reducers for grouping\n   * @param isSearchSpaceLarger Used for optimization: Is the search space larger than the query space? Ignored if numOfSearchGroups > 1.\n   * @param numOfSearchGroups we divide the search space into these groups (randomly). Useful when the search space is too large. Overrides isSearchSpaceLarger.\n   * @param numReplicas Each search group will be responsible for 1/numReplicas queryEmebeddings.\n   *                    This might speed up the search when the size of the index embeddings is\n   *                    large.\n   * @tparam A type of query entity\n   * @tparam B type of search space entity\n   * @tparam D type of distance\n   */\n  def findNearestNeighbours[A <: EntityId, B <: EntityId, D <: Distance[D]](\n    queryEmbeddings: TypedPipe[EmbeddingWithEntity[A]],\n    searchSpaceEmbeddings: TypedPipe[EmbeddingWithEntity[B]],\n    metric: Metric[D],\n    numNeighbors: Int = 10,\n    queryIdsFilter: Option[TypedPipe[A]] = Option.empty,\n    reducers: Int = 100,\n    mappers: Int = 100,\n    isSearchSpaceLarger: Boolean = true,\n    numOfSearchGroups: Int = 1,\n    numReplicas: Int = 1,\n    useCounters: Boolean = true\n  )(\n    implicit ordering: Ordering[A],\n    uid: UniqueID\n  ): TypedPipe[(A, Seq[(B, D)])] = {\n    val filteredQueryEmbeddings = queryIdsFilter match {\n      case Some(filter) => {\n        queryEmbeddings.groupBy(_.entityId).hashJoin(filter.asKeys).map {\n          case (x, (embedding, _)) => embedding\n        }\n      }\n      case None => queryEmbeddings\n    }\n\n    if (numOfSearchGroups > 1) {\n      val indexingStrategy = BruteForceIndexingStrategy(metric)\n      findNearestNeighboursWithIndexingStrategy(\n        queryEmbeddings,\n        searchSpaceEmbeddings,\n        numNeighbors,\n        numOfSearchGroups,\n        indexingStrategy,\n        numReplicas,\n        Some(reducers),\n        useCounters = useCounters\n      )\n    } else {\n      findNearestNeighboursViaCross(\n        filteredQueryEmbeddings,\n        searchSpaceEmbeddings,\n        metric,\n        numNeighbors,\n        reducers,\n        mappers,\n        isSearchSpaceLarger)\n    }\n  }\n\n  /**\n   * Calculate the nearest neighbors using the specified indexing strategy between two entity\n   * embeddings using one as query and other as the search space.\n   * @param queryEmbeddings entity embeddings for queries\n   * @param searchSpaceEmbeddings entity embeddings for search space. You should be able to fit\n   *                              searchSpaceEmbeddings.size / numOfSearchGroups into memory.\n   * @param numNeighbors number of neighbors\n   * @param reducersOption number of reducers for the final sortedTake.\n   * @param numOfSearchGroups we divide the search space into these groups (randomly). Useful when\n   *                          the search space is too large. Search groups are shards. Choose this\n   *                          number by ensuring searchSpaceEmbeddings.size / numOfSearchGroups\n   *                          embeddings will fit into memory.\n   * @param numReplicas Each search group will be responsible for 1/numReplicas queryEmebeddings.\n   *                    By increasing this number, we can parallelize the work and reduce end to end\n   *                    running times.\n   * @param indexingStrategy How we will search for nearest neighbors within a search group\n   * @param queryShards one step we have is to fan out the query embeddings. We create one entry\n   *                    per search group. If numOfSearchGroups is large, then this fan out can take\n   *                    a long time. You can shard the query shard first to parallelize this\n   *                    process. One way to estimate what value to use:\n   *                    queryEmbeddings.size * numOfSearchGroups / queryShards should be around 1GB.\n   * @param searchSpaceShards this param is similar to queryShards. Except it shards the search\n   *                          space when numReplicas is too large. One way to estimate what value\n   *                          to use: searchSpaceEmbeddings.size * numReplicas / searchSpaceShards\n   *                          should be around 1GB.\n   * @tparam A type of query entity\n   * @tparam B type of search space entity\n   * @tparam D type of distance\n   * @return a pipe keyed by the index embedding. The values are the list of numNeighbors nearest\n   *         neighbors along with their distances.\n   */\n  def findNearestNeighboursWithIndexingStrategy[A <: EntityId, B <: EntityId, D <: Distance[D]](\n    queryEmbeddings: TypedPipe[EmbeddingWithEntity[A]],\n    searchSpaceEmbeddings: TypedPipe[EmbeddingWithEntity[B]],\n    numNeighbors: Int,\n    numOfSearchGroups: Int,\n    indexingStrategy: IndexingStrategy[D],\n    numReplicas: Int = 1,\n    reducersOption: Option[Int] = None,\n    queryShards: Option[Int] = None,\n    searchSpaceShards: Option[Int] = None,\n    useCounters: Boolean = true\n  )(\n    implicit ordering: Ordering[A],\n    uid: UniqueID\n  ): UnsortedGrouped[A, Seq[(B, D)]] = {\n\n    implicit val ord: Ordering[NNKey] = Ordering.by(NNKey.unapply)\n\n    val entityEmbeddings = searchSpaceEmbeddings.map { embedding: EmbeddingWithEntity[B] =>\n      val entityEmbedding =\n        EntityEmbedding(embedding.entityId, Embedding(embedding.embedding.toArray))\n      entityEmbedding\n    }\n\n    val shardedSearchSpace = shard(entityEmbeddings, searchSpaceShards)\n\n    val groupedSearchSpaceEmbeddings = shardedSearchSpace\n      .flatMap { entityEmbedding =>\n        val searchGroup = Random.nextInt(numOfSearchGroups)\n        (0 until numReplicas).map { replica =>\n          (NNKey(searchGroup, replica, Some(numReplicas)), entityEmbedding)\n        }\n      }\n\n    val shardedQueries = shard(queryEmbeddings, queryShards)\n\n    val groupedQueryEmbeddings = shardedQueries\n      .flatMap { entity =>\n        val replica = Random.nextInt(numReplicas)\n        (0 until numOfSearchGroups).map { searchGroup =>\n          (NNKey(searchGroup, replica, Some(numReplicas)), entity)\n        }\n      }.group\n      .withReducers(reducersOption.getOrElse(numOfSearchGroups * numReplicas))\n\n    val numberAnnIndexQueries = Stat(\"NumberAnnIndexQueries\")\n    val annIndexQueryTotalMs = Stat(\"AnnIndexQueryTotalMs\")\n    val numberIndexBuilds = Stat(\"NumberIndexBuilds\")\n    val annIndexBuildTotalMs = Stat(\"AnnIndexBuildTotalMs\")\n    val groupedKnn = groupedQueryEmbeddings\n      .cogroup(groupedSearchSpaceEmbeddings) {\n        case (_, queryIter, searchSpaceIter) =>\n          // This index build happens numReplicas times. Ideally we could serialize the queryable.\n          // And only build the index once per search group.\n          // The issues with that now are:\n          // - The HNSW queryable is not serializable in scalding\n          // - The way that map reduce works requires that there is a job that write out the search\n          //   space embeddings numReplicas times. In the current setup, we can do that by sharding\n          //   the embeddings first and then fanning out. But if we had a single queryable, we would\n          //   not be able to shard it easily and writing this out would take a long time.\n          val indexBuildStartTime = System.currentTimeMillis()\n          val queryable = indexingStrategy.buildIndex(searchSpaceIter)\n          if (useCounters) {\n            numberIndexBuilds.inc()\n            annIndexBuildTotalMs.incBy(System.currentTimeMillis() - indexBuildStartTime)\n          }\n          queryIter.flatMap { query =>\n            val queryStartTime = System.currentTimeMillis()\n            val embedding = Embedding(query.embedding.toArray)\n            val result = Await.result(\n              queryable.queryWithDistance(embedding, numNeighbors)\n            )\n            val queryToTopNeighbors = result\n              .map { neighbor =>\n                (query.entityId, (neighbor.neighbor, neighbor.distance))\n              }\n            if (useCounters) {\n              numberAnnIndexQueries.inc()\n              annIndexQueryTotalMs.incBy(System.currentTimeMillis() - queryStartTime)\n            }\n            queryToTopNeighbors\n          }\n      }\n      .values\n      .group\n\n    val groupedKnnWithReducers = reducersOption\n      .map { reducers =>\n        groupedKnn\n          .withReducers(reducers)\n      }.getOrElse(groupedKnn)\n\n    groupedKnnWithReducers\n      .sortedTake(numNeighbors) {\n        Ordering\n          .by[(B, D), D] {\n            case (_, distance) => distance\n          }\n      }\n  }\n\n  private[this] def shard[T](\n    pipe: TypedPipe[T],\n    numberOfShards: Option[Int]\n  ): TypedPipe[T] = {\n    numberOfShards\n      .map { shards =>\n        pipe.shard(shards)\n      }.getOrElse(pipe)\n  }\n\n  private[this] def findNearestNeighboursViaCross[A <: EntityId, B <: EntityId, D <: Distance[D]](\n    queryEmbeddings: TypedPipe[EmbeddingWithEntity[A]],\n    searchSpaceEmbeddings: TypedPipe[EmbeddingWithEntity[B]],\n    metric: Metric[D],\n    numNeighbors: Int,\n    reducers: Int,\n    mappers: Int,\n    isSearchSpaceLarger: Boolean\n  )(\n    implicit ordering: Ordering[A]\n  ): TypedPipe[(A, Seq[(B, D)])] = {\n\n    val crossed: TypedPipe[(A, (B, D))] = if (isSearchSpaceLarger) {\n      searchSpaceEmbeddings\n        .shard(mappers)\n        .cross(queryEmbeddings).map {\n          case (searchSpaceEmbedding, queryEmbedding) =>\n            val distance = metric.distance(searchSpaceEmbedding.embedding, queryEmbedding.embedding)\n            (queryEmbedding.entityId, (searchSpaceEmbedding.entityId, distance))\n        }\n    } else {\n      queryEmbeddings\n        .shard(mappers)\n        .cross(searchSpaceEmbeddings).map {\n          case (queryEmbedding, searchSpaceEmbedding) =>\n            val distance = metric.distance(searchSpaceEmbedding.embedding, queryEmbedding.embedding)\n            (queryEmbedding.entityId, (searchSpaceEmbedding.entityId, distance))\n        }\n    }\n\n    crossed\n      .groupBy(_._1)\n      .withReducers(reducers)\n      .sortedTake(numNeighbors) {\n        Ordering\n          .by[(A, (B, D)), D] {\n            case (_, (_, distance)) => distance\n          } // Sort by distance metric in ascending order\n      }.map {\n        case (queryId, neighbors) =>\n          (queryId, neighbors.map(_._2))\n      }\n  }\n\n  /**\n   * Convert nearest neighbors to string format.\n   * By default format would be (queryId  neighbourId:distance  neighbourId:distance .....) in ascending order of distance.\n   * @param nearestNeighbors nearest neighbors tuple in form of (queryId, Seq[(neighborId, distance)]\n   * @param queryEntityKind entity kind of query\n   * @param neighborEntityKind entity kind of search space/neighbors\n   * @param idDistanceSeparator String separator to separate a single neighborId and distance. Default to colon (:)\n   * @param neighborSeparator String operator to separate neighbors. Default to tab\n   * @tparam A type of query entity\n   * @tparam B type of search space entity\n   * @tparam D type of distance\n   */\n  def nearestNeighborsToString[A <: EntityId, B <: EntityId, D <: Distance[D]](\n    nearestNeighbors: (A, Seq[(B, D)]),\n    queryEntityKind: EntityKind[A],\n    neighborEntityKind: EntityKind[B],\n    idDistanceSeparator: String = \":\",\n    neighborSeparator: String = \"\\t\"\n  ): String = {\n    val (queryId, neighbors) = nearestNeighbors\n    val formattedNeighbors = neighbors.map {\n      case (neighbourId, distance) =>\n        s\"${neighborEntityKind.stringInjection.apply(neighbourId)}$idDistanceSeparator${distance.distance}\"\n    }\n    (queryEntityKind.stringInjection.apply(queryId) +: formattedNeighbors)\n      .mkString(neighborSeparator)\n  }\n\n  private[this] case class NNKey(\n    searchGroup: Int,\n    replica: Int,\n    maxReplica: Option[Int] = None) {\n    override def hashCode(): Int =\n      maxReplica.map(_ * searchGroup + replica).getOrElse(super.hashCode())\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/KnnOfflineJob.scala",
    "content": "package com.twitter.ann.scalding.offline\n\nimport com.twitter.ann.common.Metric\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.ml.featurestore.lib.embedding.EmbeddingWithEntity\nimport com.twitter.cortex.ml.embeddings.common.EntityKind\nimport com.twitter.entityembeddings.neighbors.thriftscala.{EntityKey, NearestNeighbors}\nimport com.twitter.scalding.commons.source.VersionedKeyValSource\nimport com.twitter.scalding.typed.TypedPipe\nimport com.twitter.scalding.{Args, DateOps, DateParser, DateRange, Execution, TypedTsv, UniqueID}\nimport com.twitter.scalding_internal.job.TwitterExecutionApp\nimport com.twitter.search.common.file.{AbstractFile, LocalFile}\nimport java.util.TimeZone\n\n/**\n * Generates the nearest neighbour for users and store them in Manhattan format i.e sequence files.\n * See README for oscar usage.\n */\nobject KnnOfflineJob extends TwitterExecutionApp {\n  override def job: Execution[Unit] = Execution.withId { implicit uniqueId =>\n    Execution.getArgs.flatMap { args: Args =>\n      val knnDirectoryOpt: Option[String] = args.optional(\"knn_directory\")\n      knnDirectoryOpt match {\n        case Some(knnDirectory) =>\n          Execution.withCachedFile(knnDirectory) { directory =>\n            execute(args, Some(new LocalFile(directory.file)))\n          }\n        case None =>\n          execute(args, None)\n      }\n    }\n  }\n\n  /**\n   * Execute KnnOfflineJob\n   * @param args: The args object for this job\n   * @param abstractFile: An optional of producer embedding path\n   */\n  def execute(\n    args: Args,\n    abstractFile: Option[AbstractFile]\n  )(\n    implicit uniqueID: UniqueID\n  ): Execution[Unit] = {\n    implicit val tz: TimeZone = TimeZone.getDefault()\n    implicit val dp: DateParser = DateParser.default\n    implicit val dateRange = DateRange.parse(args.list(\"date\"))(DateOps.UTC, DateParser.default)\n    implicit val keyInject = BinaryScalaCodec(EntityKey)\n    implicit val valueInject = BinaryScalaCodec(NearestNeighbors)\n\n    val entityKind = EntityKind.getEntityKind(args(\"producer_entity_kind\"))\n    val metric = Metric.fromString(args(\"metric\"))\n    val outputPath: String = args(\"output_path\")\n    val numNeighbors: Int = args(\"neighbors\").toInt\n    val ef = args.getOrElse(\"ef\", numNeighbors.toString).toInt\n    val reducers: Int = args(\"reducers\").toInt\n    val knnDimension: Int = args(\"dimension\").toInt\n    val debugOutputPath: Option[String] = args.optional(\"debug_output_path\")\n    val filterPath: Option[String] = args.optional(\"users_filter_path\")\n    val shards: Int = args.getOrElse(\"shards\", \"100\").toInt\n    val useHashJoin: Boolean = args.getOrElse(\"use_hash_join\", \"false\").toBoolean\n    val mhOutput = VersionedKeyValSource[EntityKey, NearestNeighbors](\n      path = outputPath,\n      sourceVersion = None,\n      sinkVersion = None,\n      maxFailures = 0,\n      versionsToKeep = 1\n    )\n\n    val consumerEmbeddings: TypedPipe[EmbeddingWithEntity[UserId]] =\n      KnnHelper.getFilteredUserEmbeddings(\n        args,\n        filterPath,\n        reducers,\n        useHashJoin\n      )\n\n    val neighborsPipe: TypedPipe[(EntityKey, NearestNeighbors)] = KnnHelper.getNeighborsPipe(\n      args,\n      entityKind,\n      metric,\n      ef,\n      consumerEmbeddings,\n      abstractFile,\n      reducers,\n      numNeighbors,\n      knnDimension\n    )\n\n    val neighborsExecution: Execution[Unit] = neighborsPipe\n      .writeExecution(mhOutput)\n\n    // Write manual Inspection\n    debugOutputPath match {\n      case Some(path: String) =>\n        val debugExecution: Execution[Unit] = KnnDebug\n          .getDebugTable(\n            neighborsPipe = neighborsPipe,\n            shards = shards,\n            reducers = reducers\n          )\n          .writeExecution(TypedTsv(path))\n        Execution.zip(debugExecution, neighborsExecution).unit\n      case None => neighborsExecution\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/KnnTruthSetGenerator.scala",
    "content": "package com.twitter.ann.scalding.offline\n\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.Metric\nimport com.twitter.ann.scalding.offline.KnnHelper.nearestNeighborsToString\nimport com.twitter.cortex.ml.embeddings.common.EntityKind\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.scalding.source.TypedText\nimport com.twitter.scalding.Args\nimport com.twitter.scalding.Execution\nimport com.twitter.scalding.UniqueID\nimport com.twitter.scalding_internal.job.TwitterExecutionApp\n\n/**\n * This job reads index embedding data, query embeddings data, and split into index set, query set and true nearest neigbor set\n * from query to index.\n */\nobject KnnTruthSetGenerator extends TwitterExecutionApp {\n  override def job: Execution[Unit] = Execution.withId { implicit uniqueId =>\n    Execution.getArgs.flatMap { args: Args =>\n      val queryEntityKind = EntityKind.getEntityKind(args(\"query_entity_kind\"))\n      val indexEntityKind = EntityKind.getEntityKind(args(\"index_entity_kind\"))\n      val metric = Metric.fromString(args(\"metric\"))\n      run(queryEntityKind, indexEntityKind, metric, args)\n    }\n  }\n\n  private[this] def run[A <: EntityId, B <: EntityId, D <: Distance[D]](\n    uncastQueryEntityKind: EntityKind[_],\n    uncastIndexSpaceEntityKind: EntityKind[_],\n    uncastMetric: Metric[_],\n    args: Args\n  )(\n    implicit uniqueID: UniqueID\n  ): Execution[Unit] = {\n    val queryEntityKind = uncastQueryEntityKind.asInstanceOf[EntityKind[A]]\n    val indexEntityKind = uncastIndexSpaceEntityKind.asInstanceOf[EntityKind[B]]\n    val metric = uncastMetric.asInstanceOf[Metric[D]]\n\n    val reducers = args.int(\"reducers\")\n    val mappers = args.int(\"mappers\")\n    val numNeighbors = args.int(\"neighbors\")\n    val knnOutputPath = args(\"truth_set_output_path\")\n    val querySamplePercent = args.double(\"query_sample_percent\", 100) / 100\n    val indexSamplePercent = args.double(\"index_sample_percent\", 100) / 100\n\n    val queryEmbeddings = queryEntityKind.parser\n      .getEmbeddingFormat(args, \"query\")\n      .getEmbeddings\n      .sample(querySamplePercent)\n\n    val indexEmbeddings = indexEntityKind.parser\n      .getEmbeddingFormat(args, \"index\")\n      .getEmbeddings\n      .sample(indexSamplePercent)\n\n    // calculate and write knn\n    val knnExecution = KnnHelper\n      .findNearestNeighbours(\n        queryEmbeddings,\n        indexEmbeddings,\n        metric,\n        numNeighbors,\n        reducers = reducers,\n        mappers = mappers\n      )(queryEntityKind.ordering, uniqueID).map(\n        nearestNeighborsToString(_, queryEntityKind, indexEntityKind)\n      )\n      .shard(1)\n      .writeExecution(TypedText.tsv(knnOutputPath))\n\n    // write query set embeddings\n    val querySetExecution = queryEntityKind.parser\n      .getEmbeddingFormat(args, \"query_set_output\")\n      .writeEmbeddings(queryEmbeddings)\n\n    // write index set embeddings\n    val indexSetExecution = indexEntityKind.parser\n      .getEmbeddingFormat(args, \"index_set_output\")\n      .writeEmbeddings(indexEmbeddings)\n\n    Execution.zip(knnExecution, querySetExecution, indexSetExecution).unit\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/ParameterlessQueryable.scala",
    "content": "package com.twitter.ann.scalding.offline\n\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.ann.common.{Distance, NeighborWithDistance, Queryable, RuntimeParams}\nimport com.twitter.util.Future\n\nprivate[offline] case class ParameterlessQueryable[T, P <: RuntimeParams, D <: Distance[D]](\n  queryable: Queryable[T, P, D],\n  runtimeParamsForAllQueries: P) {\n\n  /**\n   * ANN query for ids with distance.\n   *\n   * @param embedding      : Embedding/Vector to be queried with.\n   * @param numOfNeighbors : Number of neighbours to be queried for.\n   *\n   * @return List of approximate nearest neighbour ids with distance from the query embedding.\n   */\n  def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int\n  ): Future[List[NeighborWithDistance[T, D]]] =\n    queryable.queryWithDistance(embedding, numOfNeighbors, runtimeParamsForAllQueries)\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/README",
    "content": "# Description\n\nThis pipeline uses hnsw and scalding to create an hnsw index based on producers embeddings, which\nit then uses to construct lists of producer suggestions for each user.\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/faissindexbuilder/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java11\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \"3rdparty/src/jvm/com/twitter/scalding:args\",\n        \"3rdparty/src/jvm/com/twitter/scalding:core\",\n        \"ann/src/main/scala/com/twitter/ann/annoy\",\n        \"ann/src/main/scala/com/twitter/ann/brute_force\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/faiss\",\n        \"ann/src/main/scala/com/twitter/ann/serialization\",\n        \"ann/src/main/scala/com/twitter/ann/util\",\n        \"src/scala/com/twitter/cortex/ml/embeddings/common:Helpers\",\n        \"src/scala/com/twitter/scalding_internal/job\",\n    ],\n)\n\nhadoop_binary(\n    name = \"faissindexbuilder-deploy\",\n    main = \"com.twitter.ann.scalding.offline.faissindexbuilder.IndexBuilderApp\",\n    platform = \"java11\",\n    runtime_platform = \"java11\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-compatible:migrated\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \":faissindexbuilder\",\n        \"3rdparty/jvm/org/slf4j:slf4j-jdk14\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/faissindexbuilder/IndexBuilder.scala",
    "content": "package com.twitter.ann.scalding.offline.faissindexbuilder\n\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.EntityEmbedding\nimport com.twitter.ann.common.Metric\nimport com.twitter.ann.faiss.FaissIndexer\nimport com.twitter.cortex.ml.embeddings.common.EmbeddingFormat\nimport com.twitter.ml.api.embedding.Embedding\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.scalding.Execution\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.util.logging.Logging\n\nobject IndexBuilder extends FaissIndexer with Logging {\n  def run[T <: UserId, D <: Distance[D]](\n    embeddingFormat: EmbeddingFormat[T],\n    embeddingLimit: Option[Int],\n    sampleRate: Float,\n    factoryString: String,\n    metric: Metric[D],\n    outputDirectory: AbstractFile,\n    numDimensions: Int\n  ): Execution[Unit] = {\n    val embeddingsPipe = embeddingFormat.getEmbeddings\n    val limitedEmbeddingsPipe = embeddingLimit\n      .map { limit =>\n        embeddingsPipe.limit(limit)\n      }.getOrElse(embeddingsPipe)\n\n    val annEmbeddingPipe = limitedEmbeddingsPipe.map { embedding =>\n      val embeddingSize = embedding.embedding.length\n      assert(\n        embeddingSize == numDimensions,\n        s\"Specified number of dimensions $numDimensions does not match the dimensions of the \" +\n          s\"embedding $embeddingSize\"\n      )\n      EntityEmbedding[Long](embedding.entityId.userId, Embedding(embedding.embedding.toArray))\n    }\n\n    build(annEmbeddingPipe, sampleRate, factoryString, metric, outputDirectory)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/faissindexbuilder/IndexBuilderApp.scala",
    "content": "package com.twitter.ann.scalding.offline.faissindexbuilder\n\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.Metric\nimport com.twitter.cortex.ml.embeddings.common._\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.scalding.Args\nimport com.twitter.scalding.DateOps\nimport com.twitter.scalding.DateParser\nimport com.twitter.scalding.DateRange\nimport com.twitter.scalding.Execution\nimport com.twitter.scalding_internal.job.TwitterExecutionApp\nimport com.twitter.search.common.file.FileUtils\nimport com.twitter.util.logging.Logging\nimport java.util.Calendar\nimport java.util.TimeZone\n\ntrait IndexBuilderExecutable extends Logging {\n  // This method is used to cast the entityKind and the metric to have parameters.\n  def indexBuilderExecution[T <: UserId, D <: Distance[D]](\n    args: Args\n  ): Execution[Unit] = {\n    // parse the arguments for this job\n    val uncastEntityKind = EntityKind.getEntityKind(args(\"entity_kind\"))\n    val uncastMetric = Metric.fromString(args(\"metric\"))\n    val entityKind = uncastEntityKind.asInstanceOf[EntityKind[T]]\n    val metric = uncastMetric.asInstanceOf[Metric[D]]\n    val uncastDateRange = args.list(\"embedding_date_range\")\n    val embeddingDateRange = if (uncastDateRange.nonEmpty) {\n      Some(DateRange.parse(uncastDateRange)(DateOps.UTC, DateParser.default))\n    } else {\n      None\n    }\n    val embeddingFormat =\n      entityKind.parser.getEmbeddingFormat(args, \"input\", providedDateRange = embeddingDateRange)\n    val numDimensions = args.int(\"num_dimensions\")\n    val embeddingLimit = args.optional(\"embedding_limit\").map(_.toInt)\n    val outputDirectory = FileUtils.getFileHandle(args(\"output_dir\"))\n    val factoryString = args.optional(\"factory_string\").get\n    val sampleRate = args.float(\"training_sample_rate\", 0.05f)\n\n    logger.debug(s\"Job args: ${args.toString}\")\n\n    val finalOutputDirectory = embeddingDateRange\n      .map { range =>\n        val cal = Calendar.getInstance(TimeZone.getTimeZone(\"UTC\"))\n        cal.setTime(range.end)\n        outputDirectory\n          .getChild(s\"${cal.get(Calendar.YEAR)}\")\n          .getChild(f\"${cal.get(Calendar.MONTH) + 1}%02d\")\n          .getChild(f\"${cal.get(Calendar.DAY_OF_MONTH)}%02d\")\n      }.getOrElse(outputDirectory)\n\n    logger.info(s\"Final output directory is ${finalOutputDirectory.getPath}\")\n\n    IndexBuilder\n      .run(\n        embeddingFormat,\n        embeddingLimit,\n        sampleRate,\n        factoryString,\n        metric,\n        finalOutputDirectory,\n        numDimensions\n      ).onComplete { _ =>\n        Unit\n      }\n  }\n}\n\nobject IndexBuilderApp extends TwitterExecutionApp with IndexBuilderExecutable {\n  override def job: Execution[Unit] = Execution.getArgs.flatMap { args: Args =>\n    indexBuilderExecution(args)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/indexbuilder/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \"3rdparty/src/jvm/com/twitter/scalding:args\",\n        \"3rdparty/src/jvm/com/twitter/scalding:core\",\n        \"ann/src/main/scala/com/twitter/ann/annoy\",\n        \"ann/src/main/scala/com/twitter/ann/brute_force\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/hnsw\",\n        \"ann/src/main/scala/com/twitter/ann/serialization\",\n        \"ann/src/main/scala/com/twitter/ann/util\",\n        \"src/scala/com/twitter/cortex/ml/embeddings/common:Helpers\",\n        \"src/scala/com/twitter/scalding_internal/job\",\n    ],\n)\n\nhadoop_binary(\n    name = \"indexbuilder-deploy\",\n    main = \"com.twitter.ann.scalding.offline.indexbuilder.IndexBuilderApp\",\n    platform = \"java8\",\n    runtime_platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-compatible:migrated\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \":indexbuilder\",\n        \"3rdparty/jvm/org/slf4j:slf4j-jdk14\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/indexbuilder/IndexBuilder.scala",
    "content": "package com.twitter.ann.scalding.offline.indexbuilder\n\nimport com.twitter.ann.common.Appendable\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.EntityEmbedding\nimport com.twitter.ann.common.Serialization\nimport com.twitter.ann.util.IndexBuilderUtils\nimport com.twitter.cortex.ml.embeddings.common.EmbeddingFormat\nimport com.twitter.ml.api.embedding.Embedding\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.scalding.Execution\nimport com.twitter.scalding_internal.job.FutureHelper\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.util.logging.Logger\n\nobject IndexBuilder {\n  private[this] val Log = Logger.apply[IndexBuilder.type]\n\n  def run[T <: EntityId, _, D <: Distance[D]](\n    embeddingFormat: EmbeddingFormat[T],\n    embeddingLimit: Option[Int],\n    index: Appendable[T, _, D] with Serialization,\n    concurrencyLevel: Int,\n    outputDirectory: AbstractFile,\n    numDimensions: Int\n  ): Execution[Unit] = {\n    val embeddingsPipe = embeddingFormat.getEmbeddings\n    val limitedEmbeddingsPipe = embeddingLimit\n      .map { limit =>\n        embeddingsPipe.limit(limit)\n      }.getOrElse(embeddingsPipe)\n\n    val annEmbeddingPipe = limitedEmbeddingsPipe.map { embedding =>\n      val embeddingSize = embedding.embedding.length\n      assert(\n        embeddingSize == numDimensions,\n        s\"Specified number of dimensions $numDimensions does not match the dimensions of the \" +\n          s\"embedding $embeddingSize\"\n      )\n      EntityEmbedding[T](embedding.entityId, Embedding(embedding.embedding.toArray))\n    }\n\n    annEmbeddingPipe.toIterableExecution.flatMap { annEmbeddings =>\n      val future = IndexBuilderUtils.addToIndex(index, annEmbeddings.toStream, concurrencyLevel)\n      val result = future.map { numberUpdates =>\n        Log.info(s\"Performed $numberUpdates updates\")\n        index.toDirectory(outputDirectory)\n        Log.info(s\"Finished writing to $outputDirectory\")\n      }\n      FutureHelper.executionFrom(result).unit\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/indexbuilder/IndexBuilderApp.scala",
    "content": "package com.twitter.ann.scalding.offline.indexbuilder\n\nimport com.twitter.ann.annoy.TypedAnnoyIndex\nimport com.twitter.ann.brute_force.SerializableBruteForceIndex\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.Metric\nimport com.twitter.ann.common.ReadWriteFuturePool\nimport com.twitter.ann.hnsw.TypedHnswIndex\nimport com.twitter.ann.serialization.thriftscala.PersistedEmbedding\nimport com.twitter.ann.serialization.PersistedEmbeddingInjection\nimport com.twitter.ann.serialization.ThriftIteratorIO\nimport com.twitter.cortex.ml.embeddings.common._\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.scalding.Args\nimport com.twitter.scalding.Execution\nimport com.twitter.scalding_internal.job.TwitterExecutionApp\nimport com.twitter.search.common.file.FileUtils\nimport com.twitter.util.FuturePool\nimport java.util.concurrent.Executors\n\ntrait IndexBuilderExecutable {\n  // This method is used to cast the entityKind and the metric to have parameters.\n  def indexBuilderExecution[T <: EntityId, D <: Distance[D]](\n    args: Args\n  ): Execution[Unit] = {\n    // parse the arguments for this job\n    val uncastEntityKind = EntityKind.getEntityKind(args(\"entity_kind\"))\n    val uncastMetric = Metric.fromString(args(\"metric\"))\n    val entityKind = uncastEntityKind.asInstanceOf[EntityKind[T]]\n    val metric = uncastMetric.asInstanceOf[Metric[D]]\n    val embeddingFormat = entityKind.parser.getEmbeddingFormat(args, \"input\")\n    val injection = entityKind.byteInjection\n    val numDimensions = args.int(\"num_dimensions\")\n    val embeddingLimit = args.optional(\"embedding_limit\").map(_.toInt)\n    val concurrencyLevel = args.int(\"concurrency_level\")\n    val outputDirectory = FileUtils.getFileHandle(args(\"output_dir\"))\n\n    println(s\"Job args: ${args.toString}\")\n    val threadPool = Executors.newFixedThreadPool(concurrencyLevel)\n\n    val serialization = args(\"algo\") match {\n      case \"brute_force\" =>\n        val PersistedEmbeddingIO = new ThriftIteratorIO[PersistedEmbedding](PersistedEmbedding)\n        SerializableBruteForceIndex[T, D](\n          metric,\n          FuturePool.apply(threadPool),\n          new PersistedEmbeddingInjection[T](injection),\n          PersistedEmbeddingIO\n        )\n      case \"annoy\" =>\n        TypedAnnoyIndex.indexBuilder[T, D](\n          numDimensions,\n          args.int(\"annoy_num_trees\"),\n          metric,\n          injection,\n          FuturePool.apply(threadPool)\n        )\n      case \"hnsw\" =>\n        val efConstruction = args.int(\"ef_construction\")\n        val maxM = args.int(\"max_m\")\n        val expectedElements = args.int(\"expected_elements\")\n        TypedHnswIndex.serializableIndex[T, D](\n          numDimensions,\n          metric,\n          efConstruction,\n          maxM,\n          expectedElements,\n          injection,\n          ReadWriteFuturePool(FuturePool.apply(threadPool))\n        )\n    }\n    IndexBuilder\n      .run(\n        embeddingFormat,\n        embeddingLimit,\n        serialization,\n        concurrencyLevel,\n        outputDirectory,\n        numDimensions\n      ).onComplete { _ =>\n        threadPool.shutdown()\n        Unit\n      }\n  }\n}\n\nobject IndexBuilderApp extends TwitterExecutionApp with IndexBuilderExecutable {\n  override def job: Execution[Unit] = Execution.getArgs.flatMap { args: Args =>\n    indexBuilderExecution(args)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/indexbuilder/README.rst",
    "content": "********\nOverview\n********\nThis job reads embedding data from HDFS in the embedding formats supported by the cortex MLX team. It converts that data into the ANN format and adds it to an ANN index. The ANN index is serialized and save to disk.\n\n*****************\nRunning In Aurora\n*****************\n\nSet up example\n==============\nThis job builds an ANN index based on hnsw algorithm using user embeddings available in hdfs.\n\n.. code-block:: bash\n\n  $ export JOB_NAME=ann_index_builder\n  $ export OUTPUT_PATH=hdfs:///user/$USER/${JOB_NAME}_test\n\n  $ CPU=32 RAM_GB=150 DISK_GB=60 aurora job create smf1/$USER/devel/$JOB_NAME ann/src/main/aurora/index_builder/aurora_builder.aurora \\\n    --bind=profile.name=$JOB_NAME \\\n    --bind=profile.role=$USER \\\n    --bind=profile.output_dir=$OUTPUT_PATH \\\n    --bind=profile.entity_kind=user \\\n    --bind=profile.embedding_args='--input.embedding_format tab --input.embedding_path /user/cortex-mlx/official_examples/ann/non_pii_random_user_embeddings_tab_format' \\\n    --bind=profile.num_dimensions=300 \\\n    --bind=profile.algo=hnsw \\\n    --bind=profile.ef_construction=200 \\\n    --bind=profile.max_m=16 \\\n    --bind=profile.expected_elements=10000000 \\\n    --bind=profile.metric=InnerProduct \\\n    --bind=profile.concurrency_level=32 \\\n    --bind=profile.hadoop_cluster=dw2-smf1\n\nThis job builds an ANN index based on hnsw algorithm using producer embeddings (Major version 1546473691) available in feature store.\n\n.. code-block:: bash\n\n  $ export JOB_NAME=ann_index_builder\n  $ export OUTPUT_PATH=hdfs:///user/$USER/${JOB_NAME}_test\n\n  $ CPU=32 RAM_GB=150 DISK_GB=60 aurora job create smf1/$USER/devel/$JOB_NAME ann/src/main/aurora/index_builder/aurora_builder.aurora \\\n    --bind=profile.name=$JOB_NAME \\\n    --bind=profile.role=$USER \\\n    --bind=profile.output_dir=$OUTPUT_PATH \\\n    --bind=profile.entity_kind=user \\\n    --bind=profile.embedding_args='--input.feature_store_embedding ProducerFollowEmbedding300Dataset --input.feature_store_major_version 1546473691 --input.date_range 2019-01-02' \\\n    --bind=profile.num_dimensions=300 \\\n    --bind=profile.algo=hnsw \\\n    --bind=profile.ef_construction=200 \\\n    --bind=profile.max_m=16 \\\n    --bind=profile.expected_elements=10000000 \\\n    --bind=profile.metric=InnerProduct \\\n    --bind=profile.concurrency_level=32 \\\n    --bind=profile.hadoop_cluster=dw2-smf1\n\n\n*************\nJob arguments\n*************\n\nEnviroment variables (resources):\n==============\n- **CPU** Number of cpu cores (default: 32)\n- **RAM_GB** RAM in gigabytes (default: 150)\n- **DISK_GB** Disk in gigabytes (default: 60)\n\nGeneral arguments (specified as **--profile.{options}**):\n==============\n- **name** Aurora job name\n- **role** Aurora role\n- **hadoop_cluster** Hadoop cluster for data. dw2-smf1/proc-atla.\n- **input_dir** Path of saved embeddings in hdfs without prefixing `hdfs://`\n- **entity_kind** The type of entity id that is use with the embeddings. Possible options:\n\n  - word\n  - url\n  - user\n  - tweet\n  - tfwId\n\n- **embedding_args** Embedding format args. See the documentation in `com.twitter.cortex.ml.embeddings.common.EmbeddingFormatArgsParser` for a full explanation of the input options. Possible options:\n\n  1. **input.embedding_format** Format of the serialized embedding.\n\n     - usertensor\n     - usercontinuous\n     - comma\n     - tab\n\n  2. **input.embedding_path** Path of saved embeddings in hdfs without prefixing `hdfs://`\n\n  3. **input.{feature_store_args}** For feature store related args like `feature_store_embedding`, `feature_store_major_version`, `date_range`:\n\n- **output_dir** Where to save the produced serialized ann index. Save to HDFS by specifying the full URI. e.g `hdfs://hadoop-dw2-nn.smf1.twitter.com/user/<user>/index_file` or using the default cluster `hdfs:///user/<user>/index_file`.\n- **num_dimensions** Dimension of embedding in the input data. An exception will be thrown if any entry does not have a number of dimensions equal to this number.\n- **metric** Distance metric (InnerProduct/Cosine/L2)\n- **concurrency_level** Specifies how many parallel inserts happen to the index. This should probably be set to the number of cores on the machine.\n- **algo** The kind of index you want to ouput. The supported options right now are:\n\n  1. **hnsw** (Metric supported: Cosine, L2, InnerProduct)\n\n     .. _hnsw: https://arxiv.org/abs/1603.09320\n\n     - **ef\\_construction** : Larger value increases build time but will give better recall. Good start value : 200\n     - **max\\_m** : Larger value increases will increase the index size but will give better recall. Optimal Range : 6-48. Good starting value 16.\n     - **expected\\_elements** : Approximate number of elements that will be indexed.\n\n  2. **annoy** (Metric supported: Cosine, L2)\n\n     .. _annoy: https://github.com/spotify/annoy\n\n     - **annoy\\_num\\_trees** This parameter is required for annoy. From the annoy documentation: num_trees is provided during build time and affects the build time and the index size. A larger value will give more accurate results, but larger indexes.\n\n  3. **brute_force** (Metric supported: Cosine, L2, InnerProduct)\n\n\nDeveloping locally\n===================\n\nFor building and testing custom ann index builder job,\nYou can create job bundle locally, upload to packer and then it can be used with the job using `profile.packer_package` for name,  `profile.packer_role` for role and `profile.packer_version` for bundle version.\n\n.. code-block:: bash\n\n  ./bazel bundle ann/src/main/scala/com/twitter/ann/scalding/offline/indexbuilder:indexbuilder-deploy \\\n  --bundle-jvm-archive=zip\n\n.. code-block:: bash\n\n  packer add_version --cluster=atla <role> <package_name> dist/indexbuilder-deploy.zip\n\n\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/indexbuilderfrombq/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \"3rdparty/src/jvm/com/twitter/scalding:args\",\n        \"3rdparty/src/jvm/com/twitter/scalding:core\",\n        \"ann/src/main/scala/com/twitter/ann/annoy\",\n        \"ann/src/main/scala/com/twitter/ann/brute_force\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/hnsw\",\n        \"ann/src/main/scala/com/twitter/ann/serialization\",\n        \"ann/src/main/scala/com/twitter/ann/util\",\n        \"src/scala/com/twitter/cortex/ml/embeddings/common:Helpers\",\n        \"src/scala/com/twitter/scalding_internal/bigquery\",\n        \"src/scala/com/twitter/scalding_internal/job\",\n    ],\n)\n\nhadoop_binary(\n    name = \"ann-index-builder\",\n    main = \"com.twitter.ann.scalding.offline.indexbuilderfrombq.IndexBuilderFromBQApp\",\n    platform = \"java8\",\n    runtime_platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-compatible:migrated\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \":indexbuilderfrombq\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/indexbuilderfrombq/IndexBuilderFromBQ.scala",
    "content": "package com.twitter.ann.scalding.offline.indexbuilderfrombq\n\nimport com.twitter.ann.common.Appendable\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.EntityEmbedding\nimport com.twitter.ann.common.Serialization\nimport com.twitter.ann.util.IndexBuilderUtils\nimport com.twitter.ml.api.embedding.Embedding\nimport com.twitter.ml.featurestore.lib.embedding.EmbeddingWithEntity\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.scalding.Execution\nimport com.twitter.scalding.TypedPipe\nimport com.twitter.scalding_internal.job.FutureHelper\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.util.logging.Logger\n\nobject IndexBuilder {\n  private[this] val Log = Logger.apply[IndexBuilder.type]\n\n  def run[T <: EntityId, _, D <: Distance[D]](\n    embeddingsPipe: TypedPipe[EmbeddingWithEntity[T]],\n    embeddingLimit: Option[Int],\n    index: Appendable[T, _, D] with Serialization,\n    concurrencyLevel: Int,\n    outputDirectory: AbstractFile,\n    numDimensions: Int\n  ): Execution[Unit] = {\n    val limitedEmbeddingsPipe = embeddingLimit\n      .map { limit =>\n        embeddingsPipe.limit(limit)\n      }.getOrElse(embeddingsPipe)\n\n    val annEmbeddingPipe = limitedEmbeddingsPipe.map { embedding =>\n      val embeddingSize = embedding.embedding.length\n      assert(\n        embeddingSize == numDimensions,\n        s\"Specified number of dimensions $numDimensions does not match the dimensions of the \" +\n          s\"embedding $embeddingSize\"\n      )\n      EntityEmbedding[T](embedding.entityId, Embedding(embedding.embedding.toArray))\n    }\n\n    annEmbeddingPipe.toIterableExecution.flatMap { annEmbeddings =>\n      val future = IndexBuilderUtils.addToIndex(index, annEmbeddings.toStream, concurrencyLevel)\n      val result = future.map { numberUpdates =>\n        Log.info(s\"Performed $numberUpdates updates\")\n        index.toDirectory(outputDirectory)\n        Log.info(s\"Finished writing to $outputDirectory\")\n      }\n      FutureHelper.executionFrom(result).unit\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/scalding/offline/indexbuilderfrombq/IndexBuilderFromBQApp.scala",
    "content": "package com.twitter.ann.scalding.offline.indexbuilderfrombq\n\nimport com.google.auth.oauth2.ServiceAccountCredentials\nimport com.google.cloud.bigquery.BigQueryOptions\nimport com.google.cloud.bigquery.QueryJobConfiguration\nimport com.twitter.ann.annoy.TypedAnnoyIndex\nimport com.twitter.ann.brute_force.SerializableBruteForceIndex\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.Metric\nimport com.twitter.ann.common.ReadWriteFuturePool\nimport com.twitter.ann.hnsw.TypedHnswIndex\nimport com.twitter.ann.serialization.PersistedEmbeddingInjection\nimport com.twitter.ann.serialization.ThriftIteratorIO\nimport com.twitter.ann.serialization.thriftscala.PersistedEmbedding\nimport com.twitter.cortex.ml.embeddings.common._\nimport com.twitter.ml.api.embedding.Embedding\nimport com.twitter.ml.featurestore.lib._\nimport com.twitter.ml.featurestore.lib.embedding.EmbeddingWithEntity\nimport com.twitter.scalding.Args\nimport com.twitter.scalding.Execution\nimport com.twitter.scalding.typed.TypedPipe\nimport com.twitter.scalding_internal.bigquery.BigQueryConfig\nimport com.twitter.scalding_internal.bigquery.BigQuerySource\nimport com.twitter.scalding_internal.job.TwitterExecutionApp\nimport com.twitter.scalding_internal.multiformat.format.keyval.KeyVal\nimport com.twitter.search.common.file.FileUtils\nimport com.twitter.util.FuturePool\nimport java.io.FileInputStream\nimport java.time.LocalDateTime\nimport java.time.ZoneOffset\nimport java.util.concurrent.Executors\nimport org.apache.avro.generic.GenericRecord\nimport scala.collection.JavaConverters._\n\n/**\n * Scalding execution app for building ANN index from embeddings present in BigQuery table.\n * The output index is written to a GCS file.\n *\n * Note:\n * - Assumes input data has the fields entityId\n * - Assumes input data has the fields embedding\n *\n * Command for running the app (from source repo root):\n * scalding remote run \\\n *   --target ann/src/main/scala/com/twitter/ann/scalding/offline/indexbuilderfrombq:ann-index-builder-binary\n */\ntrait IndexBuilderFromBQExecutable {\n  // This method is used to cast the entityKind and the metric to have parameters.\n  def indexBuilderExecution[T <: EntityId, D <: Distance[D]](\n    args: Args\n  ): Execution[Unit] = {\n    // parse the arguments for this job\n    val uncastEntityKind = EntityKind.getEntityKind(args(\"entity_kind\"))\n    val uncastMetric = Metric.fromString(args(\"metric\"))\n    val entityKind = uncastEntityKind.asInstanceOf[EntityKind[T]]\n    val metric = uncastMetric.asInstanceOf[Metric[D]]\n    val injection = entityKind.byteInjection\n    val numDimensions = args.int(\"num_dimensions\")\n    val embeddingLimit = args.optional(\"embedding_limit\").map(_.toInt)\n    val concurrencyLevel = args.int(\"concurrency_level\")\n\n    val bigQuery =\n      BigQueryOptions\n        .newBuilder().setProjectId(args.required(\"bq_gcp_job_project\")).setCredentials(\n          ServiceAccountCredentials.fromStream(\n            new FileInputStream(args.required(\"gcp_service_account_key_json\")))).build().getService\n\n    // Query to get the latest partition of the BigQuery table.\n    val query =\n      s\"SELECT MAX(ts) AS RecentPartition FROM ${args.required(\"bq_gcp_table_project\")}.${args\n        .required(\"bq_dataset\")}.${args.required(\"bq_table\")}\"\n    val queryConfig = QueryJobConfiguration\n      .newBuilder(query)\n      .setUseLegacySql(false)\n      .build\n    val recentPartition =\n      bigQuery\n        .query(queryConfig).iterateAll().asScala.map(field => {\n          field.get(0).getStringValue\n        }).toArray.apply(0)\n\n    // Query to extract the embeddings from the latest partition of the BigQuery table\n    val bigQueryConfig = BigQueryConfig(\n      args.required(\"bq_gcp_table_project\"),\n      args\n        .required(\"bq_dataset\"),\n      args.required(\"bq_table\"))\n      .withServiceAccountKey(args.required(\"gcp_service_account_key_json\"))\n\n    val bqFilter = Some(\n      s\"ts >= '${recentPartition}' AND DATE(TIMESTAMP_MILLIS(createdAt)) >= DATE_SUB(DATE('${recentPartition}'), INTERVAL 1 DAY) AND DATE(TIMESTAMP_MILLIS(createdAt)) <= DATE('${recentPartition}')\")\n    val withFilterBigQueryConfig = bqFilter\n      .map { filter: String =>\n        bigQueryConfig.withFilter(filter)\n      }.getOrElse(bigQueryConfig)\n    val source = new BigQuerySource(withFilterBigQueryConfig)\n      .andThen(avroMapper)\n\n    val sourcePipe = TypedPipe\n      .from(source)\n      .map(transform[T](entityKind))\n\n    println(s\"Job args: ${args.toString}\")\n    val threadPool = Executors.newFixedThreadPool(concurrencyLevel)\n\n    val serialization = args(\"algo\") match {\n      case \"brute_force\" =>\n        val PersistedEmbeddingIO = new ThriftIteratorIO[PersistedEmbedding](PersistedEmbedding)\n        SerializableBruteForceIndex[T, D](\n          metric,\n          FuturePool.apply(threadPool),\n          new PersistedEmbeddingInjection[T](injection),\n          PersistedEmbeddingIO\n        )\n      case \"annoy\" =>\n        TypedAnnoyIndex.indexBuilder[T, D](\n          numDimensions,\n          args.int(\"annoy_num_trees\"),\n          metric,\n          injection,\n          FuturePool.apply(threadPool)\n        )\n      case \"hnsw\" =>\n        val efConstruction = args.int(\"ef_construction\")\n        val maxM = args.int(\"max_m\")\n        val expectedElements = args.int(\"expected_elements\")\n        TypedHnswIndex.serializableIndex[T, D](\n          numDimensions,\n          metric,\n          efConstruction,\n          maxM,\n          expectedElements,\n          injection,\n          ReadWriteFuturePool(FuturePool.apply(threadPool))\n        )\n    }\n\n    // Output directory for the ANN index. We place the index under a timestamped directory which\n    // will be used by the ANN service to read the latest index\n    val timestamp = LocalDateTime.now().toEpochSecond(ZoneOffset.UTC)\n    val outputDirectory = FileUtils.getFileHandle(args(\"output_dir\") + \"/\" + timestamp)\n    IndexBuilder\n      .run(\n        sourcePipe,\n        embeddingLimit,\n        serialization,\n        concurrencyLevel,\n        outputDirectory,\n        numDimensions\n      ).onComplete { _ =>\n        threadPool.shutdown()\n        Unit\n      }\n\n  }\n\n  def avroMapper(row: GenericRecord): KeyVal[Long, java.util.List[Double]] = {\n    val entityId = row.get(\"entityId\")\n    val embedding = row.get(\"embedding\")\n\n    KeyVal(\n      entityId.toString.toLong,\n      embedding.asInstanceOf[java.util.List[Double]]\n    )\n  }\n\n  def transform[T <: EntityId](\n    entityKind: EntityKind[T]\n  )(\n    bqRecord: KeyVal[Long, java.util.List[Double]]\n  ): EmbeddingWithEntity[T] = {\n    val embeddingArray = bqRecord.value.asScala.map(_.floatValue()).toArray\n    val entity_id = entityKind match {\n      case UserKind => UserId(bqRecord.key).toThrift\n      case TweetKind => TweetId(bqRecord.key).toThrift\n      case TfwKind => TfwId(bqRecord.key).toThrift\n      case SemanticCoreKind => SemanticCoreId(bqRecord.key).toThrift\n      case _ => throw new IllegalArgumentException(s\"Unsupported embedding kind: $entityKind\")\n    }\n    EmbeddingWithEntity[T](\n      EntityId.fromThrift(entity_id).asInstanceOf[T],\n      Embedding(embeddingArray))\n  }\n}\n\n/*\nscalding remote run \\\n--target ann/src/main/scala/com/twitter/ann/scalding/offline/indexbuilderfrombq:ann-index-builder-binary\n */\nobject IndexBuilderFromBQApp extends TwitterExecutionApp with IndexBuilderFromBQExecutable {\n  override def job: Execution[Unit] = Execution.getArgs.flatMap { args: Args =>\n    indexBuilderExecution(args)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/serialization/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/bijection:core\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/thrift/com/twitter/ann/serialization:serialization-scala\",\n        \"mediaservices/commons\",\n        \"scrooge/scrooge-core\",\n        \"src/scala/com/twitter/scalding_internal/multiformat/format\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/serialization/DummyANNIndexInjection.scala",
    "content": "package com.twitter.ann.serialization\n\nimport com.twitter.scalding_internal.multiformat.format.keyval.KeyValInjection\nimport com.twitter.scalding_internal.multiformat.format.keyval.KeyValInjection.Long2BigEndian\n\n/**\nDummy injection required to writeup dummy dal dataset to ANN folder.\n**/\nobject DummyANNIndexInjection {\n  val injection: KeyValInjection[Long, Long] =\n    KeyValInjection[Long, Long](Long2BigEndian, Long2BigEndian)\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/serialization/PersistedEmbeddingInjection.scala",
    "content": "package com.twitter.ann.serialization\n\nimport com.twitter.ann.common.EntityEmbedding\nimport com.twitter.ann.common.EmbeddingType._\nimport com.twitter.ann.serialization.thriftscala.PersistedEmbedding\nimport com.twitter.bijection.Injection\nimport com.twitter.mediaservices.commons.codec.ArrayByteBufferCodec\nimport java.nio.ByteBuffer\nimport scala.util.Try\n\n/**\n * Injection that converts from the ann.common.Embedding to the thrift PersistedEmbedding.\n */\nclass PersistedEmbeddingInjection[T](\n  idByteInjection: Injection[T, Array[Byte]])\n    extends Injection[EntityEmbedding[T], PersistedEmbedding] {\n  override def apply(entity: EntityEmbedding[T]): PersistedEmbedding = {\n    val byteBuffer = ByteBuffer.wrap(idByteInjection(entity.id))\n    PersistedEmbedding(byteBuffer, embeddingSerDe.toThrift(entity.embedding))\n  }\n\n  override def invert(persistedEmbedding: PersistedEmbedding): Try[EntityEmbedding[T]] = {\n    val idTry = idByteInjection.invert(ArrayByteBufferCodec.decode(persistedEmbedding.id))\n    idTry.map { id =>\n      EntityEmbedding(id, embeddingSerDe.fromThrift(persistedEmbedding.embedding))\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/serialization/ThriftIteratorIO.scala",
    "content": "package com.twitter.ann.serialization\n\nimport com.twitter.scrooge.{ThriftStruct, ThriftStructCodec}\nimport java.io.{InputStream, OutputStream}\nimport org.apache.thrift.protocol.TBinaryProtocol\nimport org.apache.thrift.transport.{TIOStreamTransport, TTransportException}\n\n/**\n * Class that can serialize and deserialize an iterator of thrift objects.\n * This class can do things lazily so there is no need to have all the object into memory.\n */\nclass ThriftIteratorIO[T <: ThriftStruct](\n  codec: ThriftStructCodec[T]) {\n  def toOutputStream(\n    iterator: Iterator[T],\n    outputStream: OutputStream\n  ): Unit = {\n    val protocol = (new TBinaryProtocol.Factory).getProtocol(new TIOStreamTransport(outputStream))\n    iterator.foreach { thriftObject =>\n      codec.encode(thriftObject, protocol)\n    }\n  }\n\n  /**\n   * Returns an iterator that lazily reads from an inputStream.\n   * @return\n   */\n  def fromInputStream(\n    inputStream: InputStream\n  ): Iterator[T] = {\n    ThriftIteratorIO.getIterator(codec, inputStream)\n  }\n}\n\nobject ThriftIteratorIO {\n  private def getIterator[T <: ThriftStruct](\n    codec: ThriftStructCodec[T],\n    inputStream: InputStream\n  ): Iterator[T] = {\n    val protocol = (new TBinaryProtocol.Factory).getProtocol(new TIOStreamTransport(inputStream))\n\n    def getNext: Option[T] =\n      try {\n        Some(codec.decode(protocol))\n      } catch {\n        case e: TTransportException if e.getType == TTransportException.END_OF_FILE =>\n          inputStream.close()\n          None\n      }\n\n    Iterator\n      .continually[Option[T]](getNext)\n      .takeWhile(_.isDefined)\n      // It should be safe to call get on here since we are only take the defined ones.\n      .map(_.get)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/loadtest/AnnLoadTest.scala",
    "content": "package com.twitter.ann.service.loadtest\n\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.ann.common.{Appendable, Distance, EntityEmbedding, Queryable, RuntimeParams}\nimport com.twitter.util.logging.Logger\nimport com.twitter.util.{Duration, Future}\n\nclass AnnIndexQueryLoadTest(\n  worker: AnnLoadTestWorker = new AnnLoadTestWorker()) {\n  lazy val logger = Logger(getClass.getName)\n\n  def performQueries[T, P <: RuntimeParams, D <: Distance[D]](\n    queryable: Queryable[T, P, D],\n    qps: Int,\n    duration: Duration,\n    queries: Seq[Query[T]],\n    concurrencyLevel: Int,\n    runtimeConfigurations: Seq[QueryTimeConfiguration[T, P]]\n  ): Future[Unit] = {\n    logger.info(s\"Query set: ${queries.size}\")\n    val res = Future.traverseSequentially(runtimeConfigurations) { config =>\n      logger.info(s\"Run load test with runtime config $config\")\n      worker.runWithQps(\n        queryable,\n        queries,\n        qps,\n        duration,\n        config,\n        concurrencyLevel\n      )\n    }\n    res.onSuccess { _ =>\n      logger.info(s\"Done loadtest with $qps for ${duration.inMilliseconds / 1000} sec\")\n    }\n    res.unit\n  }\n}\n\n/**\n * @param embedding Embedding vector\n * @param trueNeighbours List of true neighbour ids. Empty in case true neighbours dataset not available\n * @tparam T Type of neighbour\n */\ncase class Query[T](embedding: EmbeddingVector, trueNeighbours: Seq[T] = Seq.empty)\n\nclass AnnIndexBuildLoadTest(\n  buildRecorder: LoadTestBuildRecorder,\n  embeddingIndexer: EmbeddingIndexer = new EmbeddingIndexer()) {\n  lazy val logger = Logger(getClass.getName)\n  def indexEmbeddings[T, P <: RuntimeParams, D <: Distance[D]](\n    appendable: Appendable[T, P, D],\n    indexSet: Seq[EntityEmbedding[T]],\n    concurrencyLevel: Int\n  ): Future[Queryable[T, P, D]] = {\n    logger.info(s\"Index set: ${indexSet.size}\")\n    val queryable = embeddingIndexer\n      .indexEmbeddings(\n        appendable,\n        buildRecorder,\n        indexSet,\n        concurrencyLevel\n      ).onSuccess(_ => logger.info(s\"Done indexing..\"))\n\n    queryable\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/loadtest/AnnLoadTestMain.scala",
    "content": "package com.twitter.ann.service.loadtest\n\nimport com.twitter.ann.annoy.AnnoyCommon\nimport com.twitter.ann.annoy.AnnoyRuntimeParams\nimport com.twitter.ann.annoy.TypedAnnoyIndex\nimport com.twitter.ann.common._\nimport com.twitter.ann.common.thriftscala.{Distance => ServiceDistance}\nimport com.twitter.ann.common.thriftscala.{RuntimeParams => ServiceRuntimeParams}\nimport com.twitter.ann.faiss.FaissCommon\nimport com.twitter.ann.faiss.FaissParams\nimport com.twitter.ann.hnsw.HnswCommon\nimport com.twitter.ann.hnsw.HnswParams\nimport com.twitter.ann.hnsw.TypedHnswIndex\nimport com.twitter.bijection.Injection\nimport com.twitter.cortex.ml.embeddings.common.EntityKind\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.finatra.mtls.modules.ServiceIdentifierModule\nimport com.twitter.inject.server.TwitterServer\nimport com.twitter.util._\nimport java.util.concurrent.TimeUnit\n\n/**\n * To build and upload:\n *  $ ./bazel bundle ann/src/main/scala/com/twitter/ann/service/loadtest:bin --bundle-jvm-archive=zip\n *  $ packer add_version --cluster=smf1 $USER ann-loadtest dist/ann-loadtest.zip\n */\nobject AnnLoadTestMain extends TwitterServer {\n  private[this] val algo =\n    flag[String](\"algo\", \"load test server types: [annoy/hnsw]\")\n  private[this] val targetQPS =\n    flag[Int](\"qps\", \"target QPS for load test\")\n  private[this] val queryIdType =\n    flag[String](\n      \"query_id_type\",\n      \"query id type for load test: [long/string/int/user/tweet/word/url/tfwId]\")\n  private[this] val indexIdType =\n    flag[String](\n      \"index_id_type\",\n      \"index id type for load test: [long/string/int/user/tweet/word/url/tfwId]\")\n  private[this] val metric =\n    flag[String](\"metric\", \"metric type for load test: [Cosine/L2/InnerProduct]\")\n  private[this] val durationSec =\n    flag[Int](\"duration_sec\", \"duration for the load test in sec\")\n  private[this] val numberOfNeighbors =\n    flag[Seq[Int]](\"number_of_neighbors\", Seq(), \"number of neighbors\")\n  private[this] val dimension = flag[Int](\"embedding_dimension\", \"dimension of embeddings\")\n  private[this] val querySetDir =\n    flag[String](\"query_set_dir\", \"\", \"Directory containing the queries\")\n  private[this] val indexSetDir =\n    flag[String](\n      \"index_set_dir\",\n      \"\",\n      \"Directory containing the embeddings to be indexed\"\n    )\n  private[this] val truthSetDir =\n    flag[String](\"truth_set_dir\", \"\", \"Directory containing the truth data\")\n  private[this] val loadTestType =\n    flag[String](\"loadtest_type\", \"Load test type [server/local]\")\n  private[this] val serviceDestination =\n    flag[String](\"service_destination\", \"wily address of remote query service\")\n  private[this] val concurrencyLevel =\n    flag[Int](\"concurrency_level\", 8, \"number of concurrent operations on the index\")\n\n  // Queries with random embeddings\n  private[this] val withRandomQueries =\n    flag[Boolean](\"with_random_queries\", false, \"query with random embeddings\")\n  private[this] val randomQueriesCount =\n    flag[Int](\"random_queries_count\", 50000, \"total random queries\")\n  private[this] val randomEmbeddingMinValue =\n    flag[Float](\"random_embedding_min_value\", -1.0f, \"Min value of random embeddings\")\n  private[this] val randomEmbeddingMaxValue =\n    flag[Float](\"random_embedding_max_value\", 1.0f, \"Max value of random embeddings\")\n\n  // parameters for annoy\n  private[this] val numOfNodesToExplore =\n    flag[Seq[Int]](\"annoy_num_of_nodes_to_explore\", Seq(), \"number of nodes to explore\")\n  private[this] val numOfTrees =\n    flag[Int](\"annoy_num_trees\", 0, \"number of trees to build\")\n\n  // parameters for HNSW\n  private[this] val efConstruction = flag[Int](\"hnsw_ef_construction\", \"ef for Hnsw construction\")\n  private[this] val ef = flag[Seq[Int]](\"hnsw_ef\", Seq(), \"ef for Hnsw query\")\n  private[this] val maxM = flag[Int](\"hnsw_max_m\", \"maxM for Hnsw\")\n\n  // FAISS\n  private[this] val nprobe = flag[Seq[Int]](\"faiss_nprobe\", Seq(), \"nprobe for faiss query\")\n  private[this] val quantizerEf =\n    flag[Seq[Int]](\"faiss_quantizerEf\", Seq(0), \"quantizerEf for faiss query\")\n  private[this] val quantizerKfactorRF =\n    flag[Seq[Int]](\"faiss_quantizerKfactorRF\", Seq(0), \"quantizerEf for faiss query\")\n  private[this] val quantizerNprobe =\n    flag[Seq[Int]](\"faiss_quantizerNprobe\", Seq(0), \"quantizerNprobe for faiss query\")\n  private[this] val ht =\n    flag[Seq[Int]](\"faiss_ht\", Seq(0), \"ht for faiss query\")\n\n  implicit val timer: Timer = DefaultTimer\n\n  override def start(): Unit = {\n    logger.info(\"Starting load test..\")\n    logger.info(flag.getAll().mkString(\"\\t\"))\n\n    assert(numberOfNeighbors().nonEmpty, \"number_of_neighbors not defined\")\n    assert(dimension() > 0, s\"Invalid dimension ${dimension()}\")\n\n    val inMemoryBuildRecorder = new InMemoryLoadTestBuildRecorder\n\n    val queryableFuture = buildQueryable(inMemoryBuildRecorder)\n    val queryConfig = getQueryRuntimeConfig\n    val result = queryableFuture.flatMap { queryable =>\n      performQueries(queryable, queryConfig, getQueries)\n    }\n\n    Await.result(result)\n    System.out.println(s\"Target QPS: ${targetQPS()}\")\n    System.out.println(s\"Duration per test: ${durationSec()}\")\n    System.out.println(s\"Concurrency Level: ${concurrencyLevel()}\")\n\n    LoadTestUtils\n      .printResults(inMemoryBuildRecorder, queryConfig)\n      .foreach(System.out.println)\n\n    Await.result(close())\n    System.exit(0)\n  }\n\n  private[this] def getQueries[Q, I]: Seq[Query[I]] = {\n    if (withRandomQueries()) {\n      assert(\n        truthSetDir().isEmpty,\n        \"Cannot use truth set when query with random embeddings enabled\"\n      )\n      val queries = LoadTestUtils.getRandomQuerySet(\n        dimension(),\n        randomQueriesCount(),\n        randomEmbeddingMinValue(),\n        randomEmbeddingMaxValue()\n      )\n\n      queries.map(Query[I](_))\n    } else {\n      assert(querySetDir().nonEmpty, \"Query set path is empty\")\n      assert(queryIdType().nonEmpty, \"Query id type is empty\")\n      val queries = LoadTestUtils.getEmbeddingsSet[Q](querySetDir(), queryIdType())\n\n      if (truthSetDir().nonEmpty) {\n        // Join the queries with truth set data.\n        assert(indexIdType().nonEmpty, \"Index id type is empty\")\n        val truthSetMap =\n          LoadTestUtils.getTruthSetMap[Q, I](truthSetDir(), queryIdType(), indexIdType())\n        queries.map(entity => Query[I](entity.embedding, truthSetMap(entity.id)))\n      } else {\n        queries.map(entity => Query[I](entity.embedding))\n      }\n    }\n  }\n\n  private[this] def getQueryRuntimeConfig[\n    T,\n    P <: RuntimeParams\n  ]: Seq[QueryTimeConfiguration[T, P]] = {\n    val queryTimeConfig = algo() match {\n      case \"annoy\" =>\n        assert(numOfNodesToExplore().nonEmpty, \"Must specify the num_of_nodes_to_explore\")\n        logger.info(s\"Querying annoy index with num_of_nodes_to_explore ${numOfNodesToExplore()}\")\n        for {\n          numNodes <- numOfNodesToExplore()\n          numOfNeighbors <- numberOfNeighbors()\n        } yield {\n          buildQueryTimeConfig[T, AnnoyRuntimeParams](\n            numOfNeighbors,\n            AnnoyRuntimeParams(Some(numNodes)),\n            Map(\n              \"numNodes\" -> numNodes.toString,\n              \"numberOfNeighbors\" -> numOfNeighbors.toString\n            )\n          ).asInstanceOf[QueryTimeConfiguration[T, P]]\n        }\n      case \"hnsw\" =>\n        assert(ef().nonEmpty, \"Must specify ef\")\n        logger.info(s\"Querying hnsw index with ef ${ef()}\")\n        for {\n          ef <- ef()\n          numOfNeighbors <- numberOfNeighbors()\n        } yield {\n          buildQueryTimeConfig[T, HnswParams](\n            numOfNeighbors,\n            HnswParams(ef),\n            Map(\n              \"efConstruction\" -> ef.toString,\n              \"numberOfNeighbors\" -> numOfNeighbors.toString\n            )\n          ).asInstanceOf[QueryTimeConfiguration[T, P]]\n        }\n      case \"faiss\" =>\n        assert(nprobe().nonEmpty, \"Must specify nprobe\")\n        def toNonZeroOptional(x: Int): Option[Int] = if (x != 0) Some(x) else None\n        for {\n          numOfNeighbors <- numberOfNeighbors()\n          runNProbe <- nprobe()\n          runQEF <- quantizerEf()\n          runKFactorEF <- quantizerKfactorRF()\n          runQNProbe <- quantizerNprobe()\n          runHT <- ht()\n        } yield {\n          val params = FaissParams(\n            Some(runNProbe),\n            toNonZeroOptional(runQEF),\n            toNonZeroOptional(runKFactorEF),\n            toNonZeroOptional(runQNProbe),\n            toNonZeroOptional(runHT))\n          buildQueryTimeConfig[T, FaissParams](\n            numOfNeighbors,\n            params,\n            Map(\n              \"nprobe\" -> params.nprobe.toString,\n              \"quantizer_efSearch\" -> params.quantizerEf.toString,\n              \"quantizer_k_factor_rf\" -> params.quantizerKFactorRF.toString,\n              \"quantizer_nprobe\" -> params.quantizerNprobe.toString,\n              \"ht\" -> params.ht.toString,\n              \"numberOfNeighbors\" -> numOfNeighbors.toString,\n            )\n          ).asInstanceOf[QueryTimeConfiguration[T, P]]\n        }\n      case _ => throw new IllegalArgumentException(s\"server type: $algo is not supported yet\")\n    }\n\n    queryTimeConfig\n  }\n\n  private def buildQueryable[T, P <: RuntimeParams, D <: Distance[D]](\n    inMemoryBuildRecorder: InMemoryLoadTestBuildRecorder\n  ): Future[Queryable[T, P, D]] = {\n    val queryable = loadTestType() match {\n      case \"remote\" => {\n        assert(serviceDestination().nonEmpty, \"Service destination not defined\")\n        logger.info(s\"Running load test with remote service ${serviceDestination()}\")\n        LoadTestUtils.buildRemoteServiceQueryClient[T, P, D](\n          serviceDestination(),\n          \"ann-load-test\",\n          statsReceiver,\n          injector.instance[ServiceIdentifier],\n          getRuntimeParamInjection[P],\n          getDistanceInjection[D],\n          getIndexIdInjection[T]\n        )\n      }\n      case \"local\" => {\n        logger.info(\"Running load test locally..\")\n        assert(indexSetDir().nonEmpty, \"Index set path is empty\")\n        val statsLoadTestBuildRecorder = new StatsLoadTestBuildRecorder(statsReceiver)\n        val buildRecorder =\n          new ComposedLoadTestBuildRecorder(Seq(inMemoryBuildRecorder, statsLoadTestBuildRecorder))\n        indexEmbeddingsAndGetQueryable[T, P, D](\n          buildRecorder,\n          LoadTestUtils.getEmbeddingsSet(indexSetDir(), indexIdType())\n        )\n      }\n    }\n    queryable\n  }\n\n  private def indexEmbeddingsAndGetQueryable[T, P <: RuntimeParams, D <: Distance[D]](\n    buildRecorder: LoadTestBuildRecorder,\n    indexSet: Seq[EntityEmbedding[T]]\n  ): Future[Queryable[T, P, D]] = {\n    logger.info(s\"Indexing entity embeddings in index set with size ${indexSet.size}\")\n    val metric = getDistanceMetric[D]\n    val indexIdInjection = getIndexIdInjection[T]\n    val indexBuilder = new AnnIndexBuildLoadTest(buildRecorder)\n    val appendable = algo() match {\n      case \"annoy\" =>\n        assert(numOfTrees() > 0, \"Must specify the number of trees for annoy\")\n        logger.info(\n          s\"Creating annoy index locally with num_of_trees: ${numOfTrees()}\"\n        )\n        TypedAnnoyIndex\n          .indexBuilder(\n            dimension(),\n            numOfTrees(),\n            metric,\n            indexIdInjection,\n            FuturePool.interruptibleUnboundedPool\n          )\n      case \"hnsw\" =>\n        assert(efConstruction() > 0 && maxM() > 0, \"Must specify ef_construction and max_m\")\n        logger.info(\n          s\"Creating hnsw index locally with max_m: ${maxM()} and ef_construction: ${efConstruction()}\"\n        )\n        TypedHnswIndex\n          .index[T, D](\n            dimension(),\n            metric,\n            efConstruction(),\n            maxM(),\n            indexSet.size,\n            ReadWriteFuturePool(FuturePool.interruptibleUnboundedPool)\n          )\n    }\n\n    indexBuilder\n      .indexEmbeddings(appendable, indexSet, concurrencyLevel())\n      .asInstanceOf[Future[Queryable[T, P, D]]]\n  }\n\n  private[this] def performQueries[T, P <: RuntimeParams, D <: Distance[D]](\n    queryable: Queryable[T, P, D],\n    queryTimeConfig: Seq[QueryTimeConfiguration[T, P]],\n    queries: Seq[Query[T]]\n  ): Future[Unit] = {\n    val indexQuery = new AnnIndexQueryLoadTest()\n    val duration = Duration(durationSec().toLong, TimeUnit.SECONDS)\n    indexQuery.performQueries(\n      queryable,\n      targetQPS(),\n      duration,\n      queries,\n      concurrencyLevel(),\n      queryTimeConfig\n    )\n  }\n\n  // provide index id injection based on argument\n  private[this] def getIndexIdInjection[T]: Injection[T, Array[Byte]] = {\n    val injection = indexIdType() match {\n      case \"long\" => AnnInjections.LongInjection\n      case \"string\" => AnnInjections.StringInjection\n      case \"int\" => AnnInjections.IntInjection\n      case entityKind => EntityKind.getEntityKind(entityKind).byteInjection\n    }\n    injection.asInstanceOf[Injection[T, Array[Byte]]]\n  }\n\n  private[this] def getRuntimeParamInjection[\n    P <: RuntimeParams\n  ]: Injection[P, ServiceRuntimeParams] = {\n    val injection = algo() match {\n      case \"annoy\" => AnnoyCommon.RuntimeParamsInjection\n      case \"hnsw\" => HnswCommon.RuntimeParamsInjection\n      case \"faiss\" => FaissCommon.RuntimeParamsInjection\n    }\n\n    injection.asInstanceOf[Injection[P, ServiceRuntimeParams]]\n  }\n\n  // provide distance injection based on argument\n  private[this] def getDistanceInjection[D <: Distance[D]]: Injection[D, ServiceDistance] = {\n    Metric.fromString(metric()).asInstanceOf[Injection[D, ServiceDistance]]\n  }\n\n  private[this] def getDistanceMetric[D <: Distance[D]]: Metric[D] = {\n    Metric.fromString(metric()).asInstanceOf[Metric[D]]\n  }\n\n  private[this] def buildQueryTimeConfig[T, P <: RuntimeParams](\n    numOfNeighbors: Int,\n    params: P,\n    config: Map[String, String]\n  ): QueryTimeConfiguration[T, P] = {\n    val printableQueryRecorder = new InMemoryLoadTestQueryRecorder[T]()\n    val scope = config.flatMap { case (key, value) => Seq(key, value.toString) }.toSeq\n    val statsLoadTestQueryRecorder = new StatsLoadTestQueryRecorder[T](\n      // Put the run time params in the stats receiver names so that we can tell the difference when\n      // we look at them later.\n      statsReceiver.scope(algo()).scope(scope: _*)\n    )\n    val queryRecorder = new ComposedLoadTestQueryRecorder(\n      Seq(printableQueryRecorder, statsLoadTestQueryRecorder)\n    )\n    QueryTimeConfiguration(\n      queryRecorder,\n      params,\n      numOfNeighbors,\n      printableQueryRecorder\n    )\n  }\n\n  override protected def modules: Seq[com.google.inject.Module] = Seq(ServiceIdentifierModule)\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/loadtest/AnnLoadTestWorker.scala",
    "content": "package com.twitter.ann.service.loadtest\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.Queryable\nimport com.twitter.ann.common.RuntimeParams\nimport com.twitter.concurrent.AsyncMeter\nimport com.twitter.concurrent.AsyncStream\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.logging.Logger\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Stopwatch\nimport com.twitter.util.Timer\nimport com.twitter.util.Try\nimport java.util.concurrent.atomic.AtomicInteger\n\nobject QueryTimeConfiguration {\n  val ResultHeader =\n    \"params\\tnumNeighbors\\trecall@1\\trecall@10\\trecall\\tavgLatencyMicros\\tp50LatencyMicros\\tp90LatencyMicros\\tp99LatencyMicros\\tavgRPS\"\n}\n\ncase class QueryTimeConfiguration[T, P <: RuntimeParams](\n  recorder: LoadTestQueryRecorder[T],\n  param: P,\n  numberOfNeighbors: Int,\n  private val results: InMemoryLoadTestQueryRecorder[T]) {\n  override def toString: String =\n    s\"QueryTimeConfiguration(param = $param, numberOfNeighbors = $numberOfNeighbors)\"\n\n  def printResults: String = {\n    val snapshot = results.computeSnapshot()\n    s\"$param\\t$numberOfNeighbors\\t${results.top1Recall}\\t${results.top10Recall}\\t${results.recall}\\t${snapshot.avgQueryLatencyMicros}\\t${snapshot.p50QueryLatencyMicros}\\t${snapshot.p90QueryLatencyMicros}\\t${snapshot.p99QueryLatencyMicros}\\t${results.avgRPS}\"\n  }\n}\n\n/**\n * Basic worker for ANN benchmark, send query with configured QPS and record results\n */\nclass AnnLoadTestWorker(\n  timer: Timer = DefaultTimer) {\n  private val logger = Logger()\n\n  /**\n   * @param queries List of tuple of query embedding and corresponding list of truth set of ids associated with the embedding\n   * @param qps the maximum number of request per second to send to the queryable. Note that if you\n   *            do not set the concurrency level high enough you may not be able to achieve this.\n   * @param duration         how long to perform the load test.\n   * @param configuration    Query configuration encapsulating runtime params and recorder.\n   * @param concurrencyLevel The maximum number of concurrent requests to the queryable at a time.\n   *                         Note that you may not be able to achieve this number of concurrent\n   *                         requests if you do not have the QPS set high enough.\n   *\n   * @return a Future that completes when the load test is over. It contains the number of requests\n   *         sent.\n   */\n  def runWithQps[T, P <: RuntimeParams, D <: Distance[D]](\n    queryable: Queryable[T, P, D],\n    queries: Seq[Query[T]],\n    qps: Int,\n    duration: Duration,\n    configuration: QueryTimeConfiguration[T, P],\n    concurrencyLevel: Int\n  ): Future[Int] = {\n    val elapsed = Stopwatch.start()\n    val atomicInteger = new AtomicInteger(0)\n    val fullStream = Stream.continually {\n      if (elapsed() <= duration) {\n        logger.ifDebug(s\"running with config: $configuration\")\n        Some(atomicInteger.getAndIncrement() % queries.size)\n      } else {\n        logger.ifDebug(s\"stopping with config: $configuration\")\n        None\n      }\n    }\n    val limitedStream = fullStream.takeWhile(_.isDefined).flatten\n    // at most we will have concurrencyLevel concurrent requests. So we should never have more than\n    // concurrency level waiters.\n    val asyncMeter = AsyncMeter.perSecond(qps, concurrencyLevel)(timer)\n\n    Future.Unit.before {\n      AsyncStream\n        .fromSeq(limitedStream).mapConcurrent(concurrencyLevel) { index =>\n          asyncMeter.await(1).flatMap { _ =>\n            performQuery(configuration, queryable, queries(index))\n          }\n        }.size\n    }\n  }\n\n  @VisibleForTesting\n  private[loadtest] def performQuery[T, P <: RuntimeParams, D <: Distance[D]](\n    configuration: QueryTimeConfiguration[T, P],\n    queryable: Queryable[T, P, D],\n    query: Query[T]\n  ): Future[Try[Unit]] = {\n    val elapsed = Stopwatch.start()\n    queryable\n      .query(query.embedding, configuration.numberOfNeighbors, configuration.param)\n      .onSuccess { res: List[T] =>\n        // underneath LoadTestRecorder will record results for load test\n        // knnMap should be truncated to be same size as query result\n        configuration.recorder.recordQueryResult(\n          query.trueNeighbours,\n          res,\n          elapsed.apply()\n        )\n        logger.ifDebug(s\"Successful query for $query\")\n      }\n      .onFailure { e =>\n        logger.error(s\"Failed query for $query: \" + e)\n      }\n      .unit\n      .liftToTry\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/loadtest/BUILD",
    "content": "scala_library(\n    name = \"loadtest\",\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ann/src/main/resources\",\n        \"ann/src/main/scala/com/twitter/ann/annoy\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/faiss\",\n        \"ann/src/main/scala/com/twitter/ann/hnsw\",\n        \"ann/src/main/scala/com/twitter/ann/util\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"finatra/inject/inject-server/src/main/scala\",\n        \"src/scala/com/twitter/cortex/ml/embeddings/common:Helpers\",\n        \"twitter-server-internal/src/main/scala\",\n        \"util/util-logging/src/main/scala\",\n    ],\n)\n\njvm_binary(\n    name = \"bin\",\n    basename = \"ann-loadtest\",\n    main = \"com.twitter.ann.service.loadtest.AnnLoadTestMain\",\n    runtime_platform = \"java11\",\n    dependencies = [\n        \":loadtest\",\n        \"3rdparty/jvm/org/slf4j:slf4j-jdk14\",\n        \"twitter-server/slf4j-jdk14/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/loadtest/EmbeddingIndexer.scala",
    "content": "package com.twitter.ann.service.loadtest\n\nimport com.twitter.ann.common.{Appendable, Distance, EntityEmbedding, Queryable, RuntimeParams}\nimport com.twitter.ann.util.IndexBuilderUtils\nimport com.twitter.util.{Future, Stopwatch}\n\nclass EmbeddingIndexer {\n  // Index embeddings into Appendable and return the (appendable, latency) pair\n  // we need to return appendable itself here because for Annoy, we need to build\n  // appendable and serialize it first, and then we could query with index directory\n  // once we are confident to remove Annoy, should clean up this method.\n  def indexEmbeddings[T, P <: RuntimeParams, D <: Distance[D]](\n    appendable: Appendable[T, P, D],\n    recorder: LoadTestBuildRecorder,\n    indexSet: Seq[EntityEmbedding[T]],\n    concurrencyLevel: Int\n  ): Future[Queryable[T, P, D]] = {\n    val indexBuildingTimeElapsed = Stopwatch.start()\n    val future = IndexBuilderUtils.addToIndex(appendable, indexSet, concurrencyLevel)\n    future.map { _ =>\n      val indexBuildingTime = indexBuildingTimeElapsed()\n      val toQueryableElapsed = Stopwatch.start()\n      val queryable = appendable.toQueryable\n      recorder.recordIndexCreation(indexSet.size, indexBuildingTime, toQueryableElapsed())\n      queryable\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/loadtest/LoadTestRecorder.scala",
    "content": "package com.twitter.ann.service.loadtest\n\nimport com.google.common.util.concurrent.AtomicDouble\nimport com.twitter.finagle.stats.{MetricsBucketedHistogram, Snapshot, StatsReceiver}\nimport com.twitter.util.{Duration, Stopwatch}\nimport java.util.concurrent.atomic.{AtomicInteger, AtomicReference}\n\ntrait LoadTestQueryRecorder[T] {\n  def recordQueryResult(\n    trueNeighbors: Seq[T],\n    foundNeighbors: Seq[T],\n    queryLatency: Duration\n  ): Unit\n}\n\ncase class LoadTestQueryResults(\n  numResults: Int,\n  top1Recall: Float,\n  top10Recall: Option[Float],\n  overallRecall: Float)\n\nprivate object LoadTestQueryRecorder {\n  def recordQueryResult[T](\n    trueNeighbors: Seq[T],\n    foundNeighbors: Seq[T]\n  ): LoadTestQueryResults = {\n    // record number of results returned\n    val numResults = foundNeighbors.size\n    if (trueNeighbors.isEmpty) {\n      LoadTestQueryResults(\n        numResults,\n        0f,\n        Option.empty,\n        0f\n      )\n    } else {\n      // record top 1, top 10 and overall recall\n      // recall here is computed as number of true neighbors within the returned points set\n      // divides by the number of required neighbors\n      val top1Recall = foundNeighbors.intersect(Seq(trueNeighbors.head)).size\n      val top10Recall = if (numResults >= 10 && trueNeighbors.size >= 10) {\n        Some(\n          trueNeighbors.take(10).intersect(foundNeighbors).size.toFloat / 10\n        )\n      } else {\n        None\n      }\n\n      val overallRecall = trueNeighbors\n        .take(foundNeighbors.size).intersect(foundNeighbors).size.toFloat /\n        Math.min(foundNeighbors.size, trueNeighbors.size)\n\n      LoadTestQueryResults(\n        numResults,\n        top1Recall,\n        top10Recall,\n        overallRecall\n      )\n    }\n  }\n}\n\nclass StatsLoadTestQueryRecorder[T](\n  statsReceiver: StatsReceiver)\n    extends LoadTestQueryRecorder[T] {\n  private[this] val numResultsStats = statsReceiver.stat(\"number_of_results\")\n  private[this] val recallStats = statsReceiver.stat(\"recall\")\n  private[this] val top1RecallStats = statsReceiver.stat(\"top_1_recall\")\n  private[this] val top10RecallStats = statsReceiver.stat(\"top_10_recall\")\n  private[this] val queryLatencyMicrosStats = statsReceiver.stat(\"query_latency_micros\")\n\n  override def recordQueryResult(\n    trueNeighbors: Seq[T],\n    foundNeighbors: Seq[T],\n    queryLatency: Duration\n  ): Unit = {\n    val results = LoadTestQueryRecorder.recordQueryResult(trueNeighbors, foundNeighbors)\n    numResultsStats.add(results.numResults)\n    recallStats.add(results.overallRecall * 100)\n    results.top10Recall.foreach { top10Recall =>\n      top10RecallStats.add(top10Recall * 100)\n    }\n    top1RecallStats.add(results.top1Recall * 100)\n    queryLatencyMicrosStats.add(queryLatency.inMicroseconds)\n  }\n}\n\ntrait LoadTestBuildRecorder {\n  def recordIndexCreation(\n    indexSize: Int,\n    indexLatency: Duration,\n    toQueryableLatency: Duration\n  ): Unit\n}\n\nclass StatsLoadTestBuildRecorder(\n  statsReceiver: StatsReceiver)\n    extends LoadTestBuildRecorder {\n  private[this] val indexLatencyGauge = statsReceiver.addGauge(\"index_latency_ms\")(_)\n  private[this] val indexSizeGauge = statsReceiver.addGauge(\"index_size\")(_)\n  private[this] val toQueryableGauge = statsReceiver.addGauge(\"to_queryable_latency_ms\")(_)\n\n  override def recordIndexCreation(\n    indexSize: Int,\n    indexLatency: Duration,\n    toQueryableLatency: Duration\n  ): Unit = {\n    indexLatencyGauge(indexLatency.inMillis)\n    indexSizeGauge(indexSize)\n    toQueryableGauge(toQueryableLatency.inMillis)\n  }\n}\n\nclass QueryRecorderSnapshot(snapshot: Snapshot) {\n  def avgQueryLatencyMicros: Double = snapshot.average\n  def p50QueryLatencyMicros: Double =\n    snapshot.percentiles.find(_.quantile == .5).get.value\n  def p90QueryLatencyMicros: Double =\n    snapshot.percentiles.find(_.quantile == .9).get.value\n  def p99QueryLatencyMicros: Double =\n    snapshot.percentiles.find(_.quantile == .99).get.value\n}\n\nclass InMemoryLoadTestQueryRecorder[T](\n  // You have to specify a name of the histogram even though it is not used\n  // Use latch period of bottom. We will compute a new snapshot every time we call computeSnapshot\n  private[this] val latencyHistogram: MetricsBucketedHistogram =\n    new MetricsBucketedHistogram(\"latencyhistogram\", latchPeriod = Duration.Bottom))\n    extends LoadTestQueryRecorder[T] {\n  private[this] val counter = new AtomicInteger(0)\n  private[this] val countMoreThan10Results = new AtomicInteger(0)\n  private[this] val recallSum = new AtomicDouble(0.0)\n  private[this] val top1RecallSum = new AtomicDouble(0.0)\n  private[this] val top10RecallSum = new AtomicDouble(0.0)\n  private[this] val elapsedTimeFun = new AtomicReference[(Stopwatch.Elapsed, Duration)]()\n  private[this] val elapsedTime = new AtomicReference[Duration](Duration.Zero)\n\n  /**\n   * Compute a snapshot of what happened between the time that this was called and the previous time\n   * it was called.\n   * @return\n   */\n  def computeSnapshot(): QueryRecorderSnapshot = {\n    new QueryRecorderSnapshot(latencyHistogram.snapshot())\n  }\n\n  def recall: Double =\n    if (counter.get() != 0) {\n      recallSum.get * 100 / counter.get()\n    } else { 0 }\n\n  def top1Recall: Double =\n    if (counter.get() != 0) {\n      top1RecallSum.get * 100 / counter.get()\n    } else { 0 }\n  def top10Recall: Double =\n    if (countMoreThan10Results.get() != 0) {\n      top10RecallSum.get * 100 / countMoreThan10Results.get()\n    } else { 0 }\n\n  def avgRPS: Double =\n    if (elapsedTime.get() != Duration.Zero) {\n      (counter.get().toDouble * 1e9) / elapsedTime.get().inNanoseconds\n    } else { 0 }\n\n  override def recordQueryResult(\n    trueNeighbors: Seq[T],\n    foundNeighbors: Seq[T],\n    queryLatency: Duration\n  ): Unit = {\n    elapsedTimeFun.compareAndSet(null, (Stopwatch.start(), queryLatency))\n    val results = LoadTestQueryRecorder.recordQueryResult(trueNeighbors, foundNeighbors)\n    top1RecallSum.addAndGet(results.top1Recall)\n    results.top10Recall.foreach { top10Recall =>\n      top10RecallSum.addAndGet(top10Recall)\n      countMoreThan10Results.incrementAndGet()\n    }\n    recallSum.addAndGet(results.overallRecall)\n    latencyHistogram.add(queryLatency.inMicroseconds)\n    counter.incrementAndGet()\n    // Requests are assumed to have started around the time time of the first time record was called\n    // plus the time it took for that query to hhave completed.\n    val (elapsedSinceFirstCall, firstQueryLatency) = elapsedTimeFun.get()\n    val durationSoFar = elapsedSinceFirstCall() + firstQueryLatency\n    elapsedTime.set(durationSoFar)\n  }\n}\n\nclass InMemoryLoadTestBuildRecorder extends LoadTestBuildRecorder {\n  var indexLatency: Duration = Duration.Zero\n  var indexSize: Int = 0\n  var toQueryableLatency: Duration = Duration.Zero\n\n  override def recordIndexCreation(\n    size: Int,\n    indexLatencyArg: Duration,\n    toQueryableLatencyArg: Duration\n  ): Unit = {\n    indexLatency = indexLatencyArg\n    indexSize = size\n    toQueryableLatency = toQueryableLatencyArg\n  }\n}\n\n/**\n * A LoadTestRecorder that be composed by other recorders\n */\nclass ComposedLoadTestQueryRecorder[T](\n  recorders: Seq[LoadTestQueryRecorder[T]])\n    extends LoadTestQueryRecorder[T] {\n  override def recordQueryResult(\n    trueNeighbors: Seq[T],\n    foundNeighbors: Seq[T],\n    queryLatency: Duration\n  ): Unit = recorders.foreach {\n    _.recordQueryResult(trueNeighbors, foundNeighbors, queryLatency)\n  }\n}\n\n/**\n * A LoadTestRecorder that be composed by other recorders\n */\nclass ComposedLoadTestBuildRecorder(\n  recorders: Seq[LoadTestBuildRecorder])\n    extends LoadTestBuildRecorder {\n  override def recordIndexCreation(\n    indexSize: Int,\n    indexLatency: Duration,\n    toQueryableLatency: Duration\n  ): Unit = recorders.foreach { _.recordIndexCreation(indexSize, indexLatency, toQueryableLatency) }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/loadtest/LoadTestUtils.scala",
    "content": "package com.twitter.ann.service.loadtest\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.ann.common.thriftscala.AnnQueryService\nimport com.twitter.ann.common.thriftscala.NearestNeighborQuery\nimport com.twitter.ann.common.thriftscala.NearestNeighborResult\nimport com.twitter.ann.common.thriftscala.{Distance => ServiceDistance}\nimport com.twitter.ann.common.thriftscala.{RuntimeParams => ServiceRuntimeParams}\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.EntityEmbedding\nimport com.twitter.ann.common.Queryable\nimport com.twitter.ann.common.RuntimeParams\nimport com.twitter.ann.common.ServiceClientQueryable\nimport com.twitter.bijection.Injection\nimport com.twitter.cortex.ml.embeddings.common.EntityKind\nimport com.twitter.finagle.builder.ClientBuilder\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient.MtlsThriftMuxClientSyntax\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.Service\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.ml.api.embedding.Embedding\nimport com.twitter.search.common.file.AbstractFile.Filter\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.search.common.file.FileUtils\nimport com.twitter.search.common.file.LocalFile\nimport com.twitter.util.Future\nimport com.twitter.util.logging.Logger\nimport java.io.File\nimport scala.collection.JavaConversions._\nimport scala.collection.mutable\nimport scala.util.Random\n\nobject LoadTestUtils {\n  lazy val Log = Logger(getClass.getName)\n\n  private[this] val LocalPath = \".\"\n  private[this] val RNG = new Random(100)\n\n  private[loadtest] def getTruthSetMap[Q, I](\n    directory: String,\n    queryIdType: String,\n    indexIdType: String\n  ): Map[Q, Seq[I]] = {\n    Log.info(s\"Loading truth set from ${directory}\")\n    val queryConverter = getKeyConverter[Q](queryIdType)\n    val indexConverter = getKeyConverter[I](indexIdType)\n    val res = loadKnnDirFileToMap(\n      getLocalFileHandle(directory),\n      // Knn truth file tsv format: [id neighbor:distance neighbor:distance ...]\n      arr => { arr.map(str => indexConverter(str.substring(0, str.lastIndexOf(\":\")))).toSeq },\n      queryConverter\n    )\n    assert(res.nonEmpty, s\"Must have some something in the truth set ${directory}\")\n    res\n  }\n\n  private[this] def getLocalFileHandle(\n    directory: String\n  ): AbstractFile = {\n    val fileHandle = FileUtils.getFileHandle(directory)\n    if (fileHandle.isInstanceOf[LocalFile]) {\n      fileHandle\n    } else {\n      val localFileHandle =\n        FileUtils.getFileHandle(s\"${LocalPath}${File.separator}${fileHandle.getName}\")\n      fileHandle.copyTo(localFileHandle)\n      localFileHandle\n    }\n  }\n\n  private[loadtest] def getEmbeddingsSet[T](\n    directory: String,\n    idType: String\n  ): Seq[EntityEmbedding[T]] = {\n    Log.info(s\"Loading embeddings from ${directory}\")\n    val res = loadKnnDirFileToMap(\n      getLocalFileHandle(directory),\n      arr => { arr.map(_.toFloat) },\n      getKeyConverter[T](idType)\n    ).map { case (key, value) => EntityEmbedding[T](key, Embedding(value.toArray)) }.toSeq\n    assert(res.nonEmpty, s\"Must have some something in the embeddings set ${directory}\")\n    res\n  }\n\n  private[this] def loadKnnDirFileToMap[K, V](\n    directory: AbstractFile,\n    f: Array[String] => Seq[V],\n    converter: String => K\n  ): Map[K, Seq[V]] = {\n    val map = mutable.HashMap[K, Seq[V]]()\n    directory\n      .listFiles(new Filter {\n        override def accept(file: AbstractFile): Boolean =\n          file.getName != AbstractFile.SUCCESS_FILE_NAME\n      }).foreach { file =>\n        asScalaBuffer(file.readLines()).foreach { line =>\n          addToMapFromKnnString(line, f, map, converter)\n        }\n      }\n    map.toMap\n  }\n\n  // Generating random float with value range bounded between minValue and maxValue\n  private[loadtest] def getRandomQuerySet(\n    dimension: Int,\n    totalQueries: Int,\n    minValue: Float,\n    maxValue: Float\n  ): Seq[EmbeddingVector] = {\n    Log.info(\n      s\"Generating $totalQueries random queries for dimension $dimension with value between $minValue and $maxValue...\")\n    assert(totalQueries > 0, s\"Total random queries $totalQueries should be greater than 0\")\n    assert(\n      maxValue > minValue,\n      s\"Random embedding max value should be greater than min value. min: $minValue max: $maxValue\")\n    (1 to totalQueries).map { _ =>\n      val embedding = Array.fill(dimension)(minValue + (maxValue - minValue) * RNG.nextFloat())\n      Embedding(embedding)\n    }\n  }\n\n  private[this] def getKeyConverter[T](idType: String): String => T = {\n    val converter = idType match {\n      case \"long\" =>\n        (s: String) => s.toLong\n      case \"string\" =>\n        (s: String) => s\n      case \"int\" =>\n        (s: String) => s.toInt\n      case entityKind =>\n        (s: String) => EntityKind.getEntityKind(entityKind).stringInjection.invert(s).get\n    }\n    converter.asInstanceOf[String => T]\n  }\n\n  private[loadtest] def buildRemoteServiceQueryClient[T, P <: RuntimeParams, D <: Distance[D]](\n    destination: String,\n    clientId: String,\n    statsReceiver: StatsReceiver,\n    serviceIdentifier: ServiceIdentifier,\n    runtimeParamInjection: Injection[P, ServiceRuntimeParams],\n    distanceInjection: Injection[D, ServiceDistance],\n    indexIdInjection: Injection[T, Array[Byte]]\n  ): Future[Queryable[T, P, D]] = {\n    val client: AnnQueryService.MethodPerEndpoint = new AnnQueryService.FinagledClient(\n      service = ClientBuilder()\n        .reportTo(statsReceiver)\n        .dest(destination)\n        .stack(ThriftMux.client.withMutualTls(serviceIdentifier).withClientId(ClientId(clientId)))\n        .build(),\n      stats = statsReceiver\n    )\n\n    val service = new Service[NearestNeighborQuery, NearestNeighborResult] {\n      override def apply(request: NearestNeighborQuery): Future[NearestNeighborResult] =\n        client.query(request)\n    }\n\n    Future.value(\n      new ServiceClientQueryable[T, P, D](\n        service,\n        runtimeParamInjection,\n        distanceInjection,\n        indexIdInjection\n      )\n    )\n  }\n\n  // helper method to convert a line in KNN file output format into map\n  @VisibleForTesting\n  def addToMapFromKnnString[K, V](\n    line: String,\n    f: Array[String] => Seq[V],\n    map: mutable.HashMap[K, Seq[V]],\n    converter: String => K\n  ): Unit = {\n    val items = line.split(\"\\t\")\n    map += converter(items(0)) -> f(items.drop(1))\n  }\n\n  def printResults(\n    inMemoryBuildRecorder: InMemoryLoadTestBuildRecorder,\n    queryTimeConfigurations: Seq[QueryTimeConfiguration[_, _]]\n  ): Seq[String] = {\n    val queryTimeConfigStrings = queryTimeConfigurations.map { config =>\n      config.printResults\n    }\n\n    Seq(\n      \"Build results\",\n      \"indexingTimeSecs\\ttoQueryableTimeMs\\tindexSize\",\n      s\"${inMemoryBuildRecorder.indexLatency.inSeconds}\\t${inMemoryBuildRecorder.toQueryableLatency.inMilliseconds}\\t${inMemoryBuildRecorder.indexSize}\",\n      \"Query results\",\n      QueryTimeConfiguration.ResultHeader\n    ) ++ queryTimeConfigStrings\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/loadtest/README.md",
    "content": "# Loadtest ANN query service with random embeddings\n\nAn ANN query service can be load-tested with random embeddings as queries, generated automatically by loadtest tool.\nExample script to load test a ANN query service with random embeddings:\n\n```bash\n$ aurora job create smf1/<role>/staging/ann-loadtest-service ann/src/main/aurora/loadtest/loadtest.aurora \\\n  --bind=profile.name=ann-loadtest-service \\\n  --bind=profile.role=<role> \\\n  --bind=profile.duration_sec=10 \\\n  --bind=profile.number_of_neighbors=10 \\\n  --bind=profile.qps=200 \\\n  --bind=profile.algo=hnsw \\\n  --bind=profile.metric=Cosine \\\n  --bind=profile.index_id_type=int \\\n  --bind=profile.hnsw_ef=400,600,800 \\\n  --bind=profile.embedding_dimension=3 \\\n  --bind=profile.concurrency_level=8 \\\n  --bind=profile.loadtest_type=remote \\\n  --bind=profile.service_destination=/srv#/staging/local/apoorvs/ann-server-test \\\n  --bind=profile.with_random_queries=True \\\n  --bind=profile.random_queries_count=50000 \\\n  --bind=profile.random_embedding_min_value=-10.0 \\\n  --bind=profile.random_embedding_max_value=10.0\n```\n\nIt will run the loadtest with `50000` random embeddings, where each embedding value will be range bounded between `random_embedding_min_value` and `random_embedding_max_value`.\nIn the above the case it will be bounded between `-10.0` and `10.0`.\nIf `random_embedding_min_value` and `random_embedding_max_value` are not supplied default value of `-1.0` and `1.0` will be used.\n\n## Results\n\nLoad test results will be printed to stdout of an aurora job.\n\n# Loadtest ANN query service with query set\n\nAn ANN query service can be load-tested with sample queries drawn from the embeddings dataset.\nFor creating sample queries i.e `query_set` refer this [section](#query-set-generator).\n\nTest is run with `live` version of loadtest binary that is already available in packer.\nExample script to load test a ANN query service:\n\n```bash\n$ aurora job create smf1/<role>/staging/ann-loadtest-service ann/src/main/aurora/loadtest/loadtest.aurora \\\n  --bind=profile.name=ann-loadtest-service \\\n  --bind=profile.role=<role> \\\n  --bind=profile.duration_sec=10 \\\n  --bind=profile.query_set_dir=hdfs:///user/cortex/ann_example/dataset/search/query_knn/query_set \\\n  --bind=profile.number_of_neighbors=10 \\\n  --bind=profile.qps=200 \\\n  --bind=profile.algo=hnsw \\\n  --bind=profile.query_id_type=string \\\n  --bind=profile.index_id_type=string \\\n  --bind=profile.metric=Cosine \\\n  --bind=profile.hnsw_ef=400,600,800 \\\n  --bind=profile.embedding_dimension=100 \\\n  --bind=profile.concurrency_level=8 \\\n  --bind=profile.loadtest_type=remote \\\n  --bind=profile.service_destination=/srv#/staging/local/apoorvs/ann-server-test\n```\n\n# In-Memory based loadtest for measuring recall\n\nLoad test can be with the above created dataset in memory.\nFor running in in-memory mode, index is created in memory, and for that you need `query_set/index_set/truth_set`.\nFor creating this dataset refer this [section](#knn-truth-set-generator).\n\nTest is run with `live` version loadtest binary that is already available in packer.\nExample script In-Memory index building and benchmarking:\n\n```bash\n$ aurora job create smf1/<role>/staging/ann-loadtest ann/src/main/aurora/loadtest/loadtest.aurora \\\n  --bind=profile.name=ann-loadtest \\\n  --bind=profile.role=<role> \\\n  --bind=profile.duration_sec=10 \\\n  --bind=profile.truth_set_dir=hdfs:///user/cortex/ann_example/dataset/search/query_knn/true_knn \\\n  --bind=profile.query_set_dir=hdfs:///user/cortex/ann_example/dataset/search/query_knn/query_set \\\n  --bind=profile.index_set_dir=hdfs:///user/cortex/ann_example/dataset/search/query_knn/index_set \\\n  --bind=profile.number_of_neighbors=10 \\\n  --bind=profile.qps=200 \\\n  --bind=profile.algo=hnsw \\\n  --bind=profile.query_id_type=string \\\n  --bind=profile.index_id_type=string \\\n  --bind=profile.metric=Cosine \\\n  --bind=profile.hnsw_ef_construction=15 \\\n  --bind=profile.hnsw_max_m=10 \\\n  --bind=profile.hnsw_ef=400,600,800 \\\n  --bind=profile.embedding_dimension=100 \\\n  --bind=profile.concurrency_level=8 \\\n  --bind=profile.loadtest_type=local\n```\n\n# Loadtest faiss\n\n```bash\n$ aurora job create smf1/<role>/staging/ann-loadtest-service ann/src/main/aurora/loadtest/loadtest.aurora \\\n  --bind=profile.name=ann-loadtest-service \\\n  --bind=profile.role=<role> \\\n  --bind=profile.duration_sec=10 \\\n  --bind=profile.number_of_neighbors=10 \\\n  --bind=profile.qps=200 \\\n  --bind=profile.algo=faiss \\ # Changed to faiss\n  --bind=profile.faiss_nprobe=1,3,9,27,81,128,256,512 \\ # Added\n  --bind=profile.faiss_quantizerKfactorRF=1,2 \\ # Pass a list to do grid search\n  --bind=profile.faiss_quantizerNprobe=128 \\ # Added\n  --bind=profile.metric=Cosine \\\n  --bind=profile.index_id_type=int \\\n  --bind=profile.embedding_dimension=3 \\\n  --bind=profile.concurrency_level=8 \\\n  --bind=profile.loadtest_type=remote \\\n  --bind=profile.service_destination=/srv#/staging/local/apoorvs/ann-server-test \\\n  --bind=profile.with_random_queries=True \\\n  --bind=profile.random_queries_count=50000 \\\n  --bind=profile.random_embedding_min_value=-10.0 \\\n  --bind=profile.random_embedding_max_value=10.0\n```\n\nFull list of faiss specific parameters. [Exact definition of all available parameters](https://github.com/facebookresearch/faiss/blob/36f2998a6469280cef3b0afcde2036935a29aa1f/faiss/AutoTune.cpp#L444). Please reach out if you need to use parameters which aren't shown below\n\n```\nfaiss_nprobe                = Default(String, '1')\nfaiss_quantizerEf           = Default(String, '0')\nfaiss_quantizerKfactorRF    = Default(String, '0')\nfaiss_quantizerNprobe       = Default(String, '0')\nfaiss_ht                    = Default(String, '0')\n```\n\n# Query Set Generator\n\nSample queries can be generated from the embeddings dataset and can be used directly with load test in tab format.\nTo generate sample queries `EmbeddingSamplingJob` can be used as follows.\n\n```bash\n$ ./bazel bundle cortex-core/entity-embeddings/src/scala/main/com/twitter/scalding/util/EmbeddingFormat:embeddingformat-deploy\n\n$ export INPUT_PATH=/user/cortex/embeddings/user/tfwproducersg/embedding_datarecords_on_data/2018/05/01\n$ export ENTITY_KIND=user\n$ export EMBEDDING_INPUT_FORMAT=usertensor\n$ export OUTPUT_PATH=/user/$USER/sample_embeddings\n$ export SAMPLE_PERCENT=0.1\n\n$ oscar hdfs \\\n    --screen --tee log.txt \\\n    --hadoop-client-memory 6000 \\\n    --hadoop-properties \"yarn.app.mapreduce.am.resource.mb=6000;yarn.app.mapreduce.am.command-opts='-Xmx7500m';mapreduce.map.memory.mb=7500;mapreduce.reduce.java.opts='-Xmx6000m';mapreduce.reduce.memory.mb=7500;mapred.task.timeout=36000000;\" \\\n    --min-split-size 284217728 \\\n    --bundle embeddingformat-deploy \\\n    --host hadoopnest1.smf1.twitter.com \\\n    --tool com.twitter.scalding.entityembeddings.util.EmbeddingFormat.EmbeddingSamplingJob -- \\\n    --entity_kind $ENTITY_KIND \\\n    --input.embedding_path $INPUT_PATH \\\n    --input.embedding_format $EMBEDDING_INPUT_FORMAT \\\n    --output.embedding_path $OUTPUT_PATH \\\n    --output.embedding_format tab \\\n    --sample_percent $SAMPLE_PERCENT\n```\n\nIt will sample 0.1% of embeddings and store them in `tab` format to hdfs that can be direcly used as `query_set` for loadtest.\n\n# Knn Truth Set Generator\n\nTo use load test framework to benchmark recall, you need to split your data set into index_set, query_set and knn_truth\n\n- index_set: data that will be indexed for ann\n- query_set: data that will be used for queries\n- truth_set: the real nearest neighbor used as truth to compute recall\n\nAnd also you need to figure out the dimension for your embedding vectors.\n\nKnnTruthSetGenerator can help to prepare data sets:\n\n```bash\n$ ./bazel bundle ann/src/main/scala/com/twitter/ann/scalding/offline:ann-offline-deploy\n\n$ export QUERY_EMBEDDINGS_PATH=/user/cortex-mlx/official_examples/ann/non_pii_random_user_embeddings_tab_format\n$ export INDEX_EMBEDDINGS_PATH=/user/cortex-mlx/official_examples/ann/non_pii_random_user_embeddings_tab_format\n$ export TRUTH_SET_PATH=/user/$USER/truth_set\n$ export INDEX_SET_PATH=/user/$USER/index_set\n$ export QUERY_SET_PATH=/user/$USER/query_set\n$ export METRIC=InnerProduct\n$ export QUERY_ENTITY_KIND=user\n$ export INDEX_ENTITY_KIND=user\n$ export NEIGHBOURS=10\n\n$ oscar hdfs \\\n  --screen --tee log.txt \\\n  --hadoop-client-memory 6000 \\\n  --hadoop-properties \"yarn.app.mapreduce.am.resource.mb=6000;yarn.app.mapreduce.am.command-opts='-Xmx7500m';mapreduce.map.memory.mb=7500;mapreduce.reduce.java.opts='-Xmx6000m';mapreduce.reduce.memory.mb=7500;mapred.task.timeout=36000000;\" \\\n  --bundle ann-offline-deploy \\\n  --min-split-size 284217728 \\\n  --host hadoopnest1.smf1.twitter.com \\\n  --tool com.twitter.ann.scalding.offline.KnnTruthSetGenerator -- \\\n  --neighbors $NEIGHBOURS \\\n  --metric $METRIC \\\n  --query_entity_kind $QUERY_ENTITY_KIND \\\n  --query.embedding_path $QUERY_EMBEDDINGS_PATH \\\n  --query.embedding_format tab \\\n  --query_sample_percent 50.0 \\\n  --index_entity_kind $INDEX_ENTITY_KIND \\\n  --index.embedding_path $INDEX_EMBEDDINGS_PATH \\\n  --index.embedding_format tab \\\n  --index_sample_percent 90.0 \\\n  --query_set_output.embedding_path $QUERY_SET_PATH \\\n  --query_set_output.embedding_format tab \\\n  --index_set_output.embedding_path $INDEX_SET_PATH \\\n  --index_set_output.embedding_format tab \\\n  --truth_set_output_path $TRUTH_SET_PATH \\\n  --reducers 100\n```\n\nIt will sample 90% of index set embeddings and 50% of query embeddings from total and then it will generate 3 datasets from the same that are index set, query set and true nearest neighbours from query to index in the tab format.\n`Note`: The reason for using high sample percent is due to the fact the sample embeddings dataset is small. For real use cases query set should be really small.\nSet `--reducers` according to the embeddings dataset size.\n\n# FAQ\n\nThere are multiple type of `query_id_type` and `index_id_type` that can be used. Some native types like string/int/long or related to entity embeddings\nlike tweet/word/user/url... for more info: [Link](https://cgit.twitter.biz/source/tree/src/scala/com/twitter/cortex/ml/embeddings/common/EntityKind.scala#n8)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/BUILD",
    "content": "scala_library(\n    name = \"common\",\n    sources = [\n        \"BaseQueryIndexServer.scala\",\n        \"Exceptions.scala\",\n        \"QueryIndexThriftController.scala\",\n        \"QueryableProvider.scala\",\n        \"RefreshableQueryable.scala\",\n        \"UnsafeQueryIndexServer.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":index_path_provider\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-zipkin-scribe\",\n        \"finatra-internal/decider\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"mediaservices/commons\",\n        \"scrooge/scrooge-core/src/main/scala\",\n        \"src/scala/com/twitter/cortex/ml/embeddings/common:Helpers\",\n        \"util/util-app/src/main/scala\",\n        \"util/util-logging/src/main/scala\",\n    ],\n)\n\nscala_library(\n    name = \"index_path_provider\",\n    sources = [\n        \"IndexPathProvider.scala\",\n        \"QueryServerUtil.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ann/src/main/scala/com/twitter/ann/hnsw\",\n        \"src/java/com/twitter/search/common/file\",\n        \"util/util-logging/src/main/scala\",\n        \"util/util-stats/src/main/scala/com/twitter/finagle/stats\",\n    ],\n)\n\nscala_library(\n    name = \"faiss_index_path_provider\",\n    sources = [\"FaissIndexPathProvider.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":index_path_provider\",\n        \"ann/src/main/scala/com/twitter/ann/faiss\",\n        \"src/java/com/twitter/search/common/file\",\n        \"util/util-logging/src/main/scala\",\n        \"util/util-stats/src/main/scala/com/twitter/finagle/stats\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/BaseQueryIndexServer.scala",
    "content": "package com.twitter.ann.service.query_server.common\n\nimport com.google.inject.Module\nimport com.twitter.ann.common.thriftscala.AnnQueryService\nimport com.twitter.app.Flag\nimport com.twitter.finatra.decider.modules.DeciderModule\nimport com.twitter.finatra.thrift.ThriftServer\nimport com.twitter.finatra.mtls.thriftmux.Mtls\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsThriftWebFormsModule\nimport com.twitter.finatra.thrift.filters.{\n  AccessLoggingFilter,\n  LoggingMDCFilter,\n  StatsFilter,\n  ThriftMDCFilter,\n  TraceIdMDCFilter\n}\nimport com.twitter.finatra.thrift.routing.ThriftRouter\n\n/**\n * This class provides most of the configuration needed for logging, stats, deciders etc.\n */\nabstract class BaseQueryIndexServer extends ThriftServer with Mtls {\n\n  protected val environment: Flag[String] = flag[String](\"environment\", \"service environment\")\n\n  /**\n   * Override with method to provide more module to guice.\n   */\n  protected def additionalModules: Seq[Module]\n\n  /**\n   * Override this method to add the controller to the thrift router. BaseQueryIndexServer takes\n   * care of most of the other configuration for you.\n   * @param router\n   */\n  protected def addController(router: ThriftRouter): Unit\n\n  override protected final lazy val modules: Seq[Module] = Seq(\n    DeciderModule,\n    new MtlsThriftWebFormsModule[AnnQueryService.MethodPerEndpoint](this)\n  ) ++ additionalModules\n\n  override protected final def configureThrift(router: ThriftRouter): Unit = {\n    router\n      .filter[LoggingMDCFilter]\n      .filter[TraceIdMDCFilter]\n      .filter[ThriftMDCFilter]\n      .filter[AccessLoggingFilter]\n      .filter[StatsFilter]\n\n    addController(router)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/Exceptions.scala",
    "content": "package com.twitter.ann.service.query_server.common\n\nimport com.twitter.ann.common.thriftscala.BadRequest\nimport com.twitter.mediaservices.commons._\n\nobject RuntimeExceptionTransform extends ExceptionTransformer {\n  override def transform = {\n    case e: BadRequest =>\n      MisuseExceptionInfo(e)\n  }\n\n  override def getStatName: PartialFunction[Exception, String] = {\n    case e: BadRequest => exceptionName(e, e.code.name)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/FaissIndexPathProvider.scala",
    "content": "package com.twitter.ann.service.query_server.common\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.search.common.file.AbstractFile\n\ncase class FaissIndexPathProvider(\n  override val minIndexSizeBytes: Long,\n  override val maxIndexSizeBytes: Long,\n  override val statsReceiver: StatsReceiver)\n    extends BaseIndexPathProvider {\n\n  override val log = Logger.get(\"FAISSIndexPathProvider\")\n\n  override def isValidIndex(dir: AbstractFile): Boolean = {\n    dir.isDirectory &&\n    dir.hasSuccessFile &&\n    dir.getChild(\"faiss.index\").exists()\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/IndexPathProvider.scala",
    "content": "package com.twitter.ann.service.query_server.common\n\nimport com.twitter.ann.common.IndexOutputFile\nimport com.twitter.ann.hnsw.HnswCommon._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.search.common.file.AbstractFile.Filter\nimport com.twitter.search.common.file.PathUtils\nimport com.twitter.util.Try\nimport java.io.IOException\nimport java.util.concurrent.atomic.AtomicReference\nimport scala.collection.JavaConverters._\nimport scala.math.Ordering.comparatorToOrdering\n\nabstract class IndexPathProvider {\n  def provideIndexPath(rootPath: AbstractFile, group: Boolean = false): Try[AbstractFile]\n  def provideIndexPathWithGroups(rootPath: AbstractFile): Try[Seq[AbstractFile]]\n}\n\nabstract class BaseIndexPathProvider extends IndexPathProvider {\n  protected val minIndexSizeBytes: Long\n  protected val maxIndexSizeBytes: Long\n  protected val statsReceiver: StatsReceiver\n  protected val log: Logger\n  private val invalidPathCounter = statsReceiver.counter(\"invalid_index\")\n  private val failToLocateDirectoryCounter = statsReceiver.counter(\"find_latest_path_fail\")\n  private val successProvidePathCounter = statsReceiver.counter(\"provide_path_success\")\n\n  private val latestGroupCount = new AtomicReference(0f)\n  private val latestIndexTimestamp = new AtomicReference(0f)\n  private val latestValidIndexTimestamp = new AtomicReference(0f)\n\n  private val INDEX_METADATA_FILE = \"ANN_INDEX_METADATA\"\n\n  private val latestIndexGauge = statsReceiver.addGauge(\"latest_index_timestamp\")(\n    latestIndexTimestamp.get()\n  )\n  private val latestValidIndexGauge = statsReceiver.addGauge(\"latest_valid_index_timestamp\")(\n    latestValidIndexTimestamp.get()\n  )\n  private val latestGroupCountGauge = statsReceiver.addGauge(\"latest_group_count\")(\n    latestGroupCount.get()\n  )\n\n  private val latestTimeStampDirectoryFilter = new AbstractFile.Filter {\n\n    /** Determines which files should be accepted when listing a directory. */\n    override def accept(file: AbstractFile): Boolean = {\n      val name = file.getName\n      PathUtils.TIMESTAMP_PATTERN.matcher(name).matches()\n    }\n  }\n\n  private def findLatestTimeStampValidSuccessDirectory(\n    path: AbstractFile,\n    group: Boolean\n  ): AbstractFile = {\n    log.info(s\"Calling findLatestTimeStampValidSuccessDirectory with ${path.getPath}\")\n    // Get all the timestamp directories\n    val dateDirs = path.listFiles(latestTimeStampDirectoryFilter).asScala.toSeq\n\n    if (dateDirs.nonEmpty) {\n      // Validate the indexes\n      val latestValidPath = {\n        if (group) {\n          // For grouped, check all the individual group indexes and stop as soon as a valid index\n          // is found.\n          dateDirs\n            .sorted(comparatorToOrdering(PathUtils.NEWEST_FIRST_COMPARATOR)).find(file => {\n              val indexMetadataFile = file.getChild(INDEX_METADATA_FILE)\n              val indexes = file.listFiles().asScala.filter(_.isDirectory)\n              val isValid = if (indexMetadataFile.exists()) {\n                // Metadata file exists. Check the number of groups and verify the index is\n                // complete\n                val indexMetadata = new IndexOutputFile(indexMetadataFile).loadIndexMetadata()\n                if (indexMetadata.numGroups.get != indexes.size) {\n                  log.info(\n                    s\"Grouped index ${file.getPath} should have ${indexMetadata.numGroups.get} groups but had ${indexes.size}\")\n                }\n                indexMetadata.numGroups.get == indexes.size\n              } else {\n                // True if the file doesn't exist. This is to make this change backwards\n                // compatible for clients using the old version of the dataflow job\n                true\n              }\n\n              isValid && indexes.forall(index => {\n                index.hasSuccessFile && isValidIndex(index) && QueryServerUtil\n                  .isValidIndexDirSize(index, minIndexSizeBytes, maxIndexSizeBytes)\n              })\n            })\n        } else {\n          // For non-grouped, find the first valid index.\n          dateDirs\n            .sorted(comparatorToOrdering(PathUtils.NEWEST_FIRST_COMPARATOR)).find(file => {\n              file.hasSuccessFile && QueryServerUtil\n                .isValidIndexDirSize(file, minIndexSizeBytes, maxIndexSizeBytes)\n            })\n        }\n      }\n\n      if (latestValidPath.nonEmpty) {\n        // Log the results\n        successProvidePathCounter.incr()\n        if (group) {\n          latestGroupCount.set(latestValidPath.get.listFiles().asScala.count(_.isDirectory))\n          log.info(\n            s\"findLatestTimeStampValidSuccessDirectory latestValidPath ${latestValidPath.get.getPath} and number of groups $latestGroupCount\")\n        } else {\n          val latestValidPathSize =\n            latestValidPath.get.listFiles(true).asScala.map(_.getSizeInBytes).sum\n          log.info(\n            s\"findLatestTimeStampValidSuccessDirectory latestValidPath ${latestValidPath.get.getPath} and size $latestValidPathSize\")\n        }\n        return latestValidPath.get\n      }\n    }\n\n    // Fail if no index or no valid index.\n    failToLocateDirectoryCounter.incr()\n    throw new IOException(s\"Cannot find any valid directory with SUCCESS file at ${path.getName}\")\n  }\n\n  def isValidIndex(index: AbstractFile): Boolean\n\n  override def provideIndexPath(\n    rootPath: AbstractFile,\n    group: Boolean = false\n  ): Try[AbstractFile] = {\n    Try {\n      val latestValidPath = findLatestTimeStampValidSuccessDirectory(rootPath, group)\n      if (!group) {\n        val latestPath = PathUtils.findLatestTimeStampSuccessDirectory(rootPath)\n        // since latestValidPath does not throw exception, latestPath must exist\n        assert(latestPath.isPresent)\n        val latestPathSize = latestPath.get.listFiles(true).asScala.map(_.getSizeInBytes).sum\n        log.info(s\"provideIndexPath latestPath ${latestPath\n          .get()\n          .getPath} and size $latestPathSize\")\n        latestIndexTimestamp.set(latestPath.get().getName.toFloat)\n        // latest directory is not valid, update counter for alerts\n        if (latestPath.get() != latestValidPath) {\n          invalidPathCounter.incr()\n        }\n      } else {\n        latestIndexTimestamp.set(latestValidPath.getName.toFloat)\n      }\n      latestValidIndexTimestamp.set(latestValidPath.getName.toFloat)\n      latestValidPath\n    }\n  }\n\n  override def provideIndexPathWithGroups(\n    rootPath: AbstractFile\n  ): Try[Seq[AbstractFile]] = {\n    val latestValidPath = provideIndexPath(rootPath, true)\n    latestValidPath.map { path =>\n      path\n        .listFiles(new Filter {\n          override def accept(file: AbstractFile): Boolean =\n            file.isDirectory && file.hasSuccessFile\n        }).asScala.toSeq\n    }\n  }\n}\n\ncase class ValidatedIndexPathProvider(\n  override val minIndexSizeBytes: Long,\n  override val maxIndexSizeBytes: Long,\n  override val statsReceiver: StatsReceiver)\n    extends BaseIndexPathProvider {\n\n  override val log = Logger.get(\"ValidatedIndexPathProvider\")\n\n  override def isValidIndex(dir: AbstractFile): Boolean = {\n    isValidHnswIndex(dir)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/QueryIndexThriftController.scala",
    "content": "package com.twitter.ann.service.query_server.common\n\nimport com.twitter.ann.common._\nimport com.twitter.ann.common.EmbeddingType._\nimport com.twitter.ann.common.thriftscala.AnnQueryService.Query\nimport com.twitter.ann.common.thriftscala.AnnQueryService\nimport com.twitter.ann.common.thriftscala.NearestNeighbor\nimport com.twitter.ann.common.thriftscala.NearestNeighborResult\nimport com.twitter.ann.common.thriftscala.{Distance => ServiceDistance}\nimport com.twitter.ann.common.thriftscala.{RuntimeParams => ServiceRuntimeParams}\nimport com.twitter.bijection.Injection\nimport com.twitter.finagle.Service\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.thrift.Controller\nimport com.twitter.mediaservices.commons.{ThriftServer => TServer}\nimport java.nio.ByteBuffer\nimport javax.inject.Inject\n\nclass QueryIndexThriftController[T, P <: RuntimeParams, D <: Distance[D]] @Inject() (\n  statsReceiver: StatsReceiver,\n  queryable: Queryable[T, P, D],\n  runtimeParamInjection: Injection[P, ServiceRuntimeParams],\n  distanceInjection: Injection[D, ServiceDistance],\n  idInjection: Injection[T, Array[Byte]])\n    extends Controller(AnnQueryService) {\n\n  private[this] val thriftServer = new TServer(statsReceiver, Some(RuntimeExceptionTransform))\n\n  val trackingStatName = \"ann_query\"\n\n  private[this] val stats = statsReceiver.scope(trackingStatName)\n  private[this] val numOfNeighboursRequested = stats.stat(\"num_of_neighbours_requested\")\n  private[this] val numOfNeighboursResponse = stats.stat(\"num_of_neighbours_response\")\n  private[this] val queryKeyNotFound = stats.stat(\"query_key_not_found\")\n\n  /**\n   * Implements AnnQueryService.query, returns nearest neighbours for a given query\n   */\n  val query: Service[Query.Args, Query.SuccessType] = { args: Query.Args =>\n    thriftServer.track(trackingStatName) {\n      val query = args.query\n      val key = query.key\n      val embedding = embeddingSerDe.fromThrift(query.embedding)\n      val numOfNeighbours = query.numberOfNeighbors\n      val withDistance = query.withDistance\n      val runtimeParams = runtimeParamInjection.invert(query.runtimeParams).get\n      numOfNeighboursRequested.add(numOfNeighbours)\n\n      val result = if (withDistance) {\n        val nearestNeighbors = if (queryable.isInstanceOf[QueryableGrouped[T, P, D]]) {\n          queryable\n            .asInstanceOf[QueryableGrouped[T, P, D]]\n            .queryWithDistance(embedding, numOfNeighbours, runtimeParams, key)\n        } else {\n          queryable\n            .queryWithDistance(embedding, numOfNeighbours, runtimeParams)\n        }\n\n        nearestNeighbors.map { list =>\n          list.map { nn =>\n            NearestNeighbor(\n              ByteBuffer.wrap(idInjection.apply(nn.neighbor)),\n              Some(distanceInjection.apply(nn.distance))\n            )\n          }\n        }\n      } else {\n\n        val nearestNeighbors = if (queryable.isInstanceOf[QueryableGrouped[T, P, D]]) {\n          queryable\n            .asInstanceOf[QueryableGrouped[T, P, D]]\n            .query(embedding, numOfNeighbours, runtimeParams, key)\n        } else {\n          queryable\n            .query(embedding, numOfNeighbours, runtimeParams)\n        }\n\n        nearestNeighbors\n          .map { list =>\n            list.map { nn =>\n              NearestNeighbor(ByteBuffer.wrap(idInjection.apply(nn)))\n            }\n          }\n      }\n\n      result.map(NearestNeighborResult(_)).onSuccess { r =>\n        numOfNeighboursResponse.add(r.nearestNeighbors.size)\n      }\n    }\n  }\n  handle(Query) { query }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/QueryServerUtil.scala",
    "content": "package com.twitter.ann.service.query_server.common\n\nimport com.twitter.logging.Logger\nimport com.twitter.search.common.file.AbstractFile\nimport scala.collection.JavaConverters._\n\nobject QueryServerUtil {\n\n  private val log = Logger.get(\"QueryServerUtil\")\n\n  /**\n   * Validate if the abstract file (directory) size is within the defined limits.\n   * @param dir Hdfs/Local directory\n   * @param minIndexSizeBytes minimum size of file in bytes (Exclusive)\n   * @param maxIndexSizeBytes minimum size of file in bytes (Exclusive)\n   * @return true if file size within minIndexSizeBytes and maxIndexSizeBytes else false\n   */\n  def isValidIndexDirSize(\n    dir: AbstractFile,\n    minIndexSizeBytes: Long,\n    maxIndexSizeBytes: Long\n  ): Boolean = {\n    val recursive = true\n    val dirSize = dir.listFiles(recursive).asScala.map(_.getSizeInBytes).sum\n\n    log.debug(s\"Ann index directory ${dir.getPath} size in bytes $dirSize\")\n\n    val isValid = (dirSize > minIndexSizeBytes) && (dirSize < maxIndexSizeBytes)\n    if (!isValid) {\n      log.info(s\"Ann index directory is invalid ${dir.getPath} size in bytes $dirSize\")\n    }\n    isValid\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/QueryableProvider.scala",
    "content": "package com.twitter.ann.service.query_server.common\n\nimport com.twitter.ann.common.{Distance, Queryable, RuntimeParams}\nimport com.twitter.search.common.file.AbstractFile\n\ntrait QueryableProvider[T, P <: RuntimeParams, D <: Distance[D]] {\n  def provideQueryable(indexDir: AbstractFile): Queryable[T, P, D]\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/RefreshableQueryable.scala",
    "content": "package com.twitter.ann.service.query_server.common\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.google.common.util.concurrent.ThreadFactoryBuilder\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.NeighborWithDistance\nimport com.twitter.ann.common.Queryable\nimport com.twitter.ann.common.QueryableGrouped\nimport com.twitter.ann.common.RuntimeParams\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport java.util.concurrent.atomic.AtomicReference\nimport java.util.concurrent.Executors\nimport java.util.concurrent.TimeUnit\nimport scala.util.Random\nimport scala.util.control.NonFatal\n\nclass RefreshableQueryable[T, P <: RuntimeParams, D <: Distance[D]](\n  grouped: Boolean,\n  rootDir: AbstractFile,\n  queryableProvider: QueryableProvider[T, P, D],\n  indexPathProvider: IndexPathProvider,\n  statsReceiver: StatsReceiver,\n  updateInterval: Duration = 10.minutes)\n    extends QueryableGrouped[T, P, D] {\n\n  private val log = Logger.get(\"RefreshableQueryable\")\n\n  private val loadCounter = statsReceiver.counter(\"load\")\n  private val loadFailCounter = statsReceiver.counter(\"load_error\")\n  private val newIndexCounter = statsReceiver.counter(\"new_index\")\n  protected val random = new Random(System.currentTimeMillis())\n\n  private val threadFactory = new ThreadFactoryBuilder()\n    .setNameFormat(\"refreshable-queryable-update-%d\")\n    .build()\n  // single thread to check and load index\n  private val executor = Executors.newScheduledThreadPool(1, threadFactory)\n\n  private[common] val indexPathRef: AtomicReference[AbstractFile] =\n    new AtomicReference(indexPathProvider.provideIndexPath(rootDir, grouped).get())\n  private[common] val queryableRef: AtomicReference[Map[Option[String], Queryable[T, P, D]]] = {\n    if (grouped) {\n      val mapping = getGroupMapping\n\n      new AtomicReference(mapping)\n    } else {\n      new AtomicReference(Map(None -> buildIndex(indexPathRef.get())))\n    }\n  }\n\n  private val servingIndexGauge = statsReceiver.addGauge(\"serving_index_timestamp\") {\n    indexPathRef.get().getName.toFloat\n  }\n\n  log.info(\"System.gc() before start\")\n  System.gc()\n\n  private val reloadTask = new Runnable {\n    override def run(): Unit = {\n      innerLoad()\n    }\n  }\n\n  def start(): Unit = {\n    executor.scheduleWithFixedDelay(\n      reloadTask,\n      // init reloading with random delay\n      computeRandomInitDelay().inSeconds,\n      updateInterval.inSeconds,\n      TimeUnit.SECONDS\n    )\n  }\n\n  private def buildIndex(indexPath: AbstractFile): Queryable[T, P, D] = {\n    log.info(s\"build index from ${indexPath.getPath}\")\n    queryableProvider.provideQueryable(indexPath)\n  }\n\n  @VisibleForTesting\n  private[common] def innerLoad(): Unit = {\n    log.info(\"Check and load for new index\")\n    loadCounter.incr()\n    try {\n      // Find the latest directory\n      val latestPath = indexPathProvider.provideIndexPath(rootDir, grouped).get()\n      if (indexPathRef.get() != latestPath) {\n        log.info(s\"loading index from: ${latestPath.getName}\")\n        newIndexCounter.incr()\n        if (grouped) {\n          val mapping = getGroupMapping\n          queryableRef.set(mapping)\n        } else {\n          val queryable = buildIndex(latestPath)\n          queryableRef.set(Map(None -> queryable))\n        }\n        indexPathRef.set(latestPath)\n      } else {\n        log.info(s\"Current index already up to date: ${indexPathRef.get.getName}\")\n      }\n    } catch {\n      case NonFatal(err) =>\n        loadFailCounter.incr()\n        log.error(s\"Failed to load index: $err\")\n    }\n    log.info(s\"Current index loaded from ${indexPathRef.get().getPath}\")\n  }\n\n  @VisibleForTesting\n  private[common] def computeRandomInitDelay(): Duration = {\n    val bound = 5.minutes\n    val nextUpdateSec = updateInterval + Duration.fromSeconds(\n      random.nextInt(bound.inSeconds)\n    )\n    nextUpdateSec\n  }\n\n  /**\n   * ANN query for ids with key as group id\n   * @param embedding: Embedding/Vector to be queried with.\n   * @param numOfNeighbors: Number of neighbours to be queried for.\n   * @param runtimeParams: Runtime params associated with index to control accuracy/latency etc.\n   * @param key: Optional key to lookup specific ANN index and perform query there\n   *  @return List of approximate nearest neighbour ids.\n   */\n  override def query(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: P,\n    key: Option[String]\n  ): Future[List[T]] = {\n    val mapping = queryableRef.get()\n\n    if (!mapping.contains(key)) {\n      Future.value(List())\n    } else {\n      mapping.get(key).get.query(embedding, numOfNeighbors, runtimeParams)\n    }\n  }\n\n  /**\n   * ANN query for ids with key as group id with distance\n   * @param embedding: Embedding/Vector to be queried with.\n   * @param numOfNeighbors: Number of neighbours to be queried for.\n   * @param runtimeParams: Runtime params associated with index to control accuracy/latency etc.\n   * @param key: Optional key to lookup specific ANN index and perform query there\n   *  @return List of approximate nearest neighbour ids with distance from the query embedding.\n   */\n  override def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: P,\n    key: Option[String]\n  ): Future[List[NeighborWithDistance[T, D]]] = {\n    val mapping = queryableRef.get()\n\n    if (!mapping.contains(key)) {\n      Future.value(List())\n    } else {\n      mapping.get(key).get.queryWithDistance(embedding, numOfNeighbors, runtimeParams)\n    }\n  }\n\n  private def getGroupMapping(): Map[Option[String], Queryable[T, P, D]] = {\n    val groupDirs = indexPathProvider.provideIndexPathWithGroups(rootDir).get()\n    val mapping = groupDirs.map { groupDir =>\n      val queryable = buildIndex(groupDir)\n      Option(groupDir.getName) -> queryable\n    }.toMap\n\n    mapping\n  }\n\n  /**\n   * ANN query for ids.\n   *\n   * @param embedding       : Embedding/Vector to be queried with.\n   * @param numOfNeighbors  : Number of neighbours to be queried for.\n   * @param runtimeParams   : Runtime params associated with index to control accuracy/latency etc.\n   *\n   * @return List of approximate nearest neighbour ids.\n   */\n  override def query(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Future[List[T]] = {\n    query(embedding, numOfNeighbors, runtimeParams, None)\n  }\n\n  /**\n   * ANN query for ids with distance.\n   *\n   * @param embedding      : Embedding/Vector to be queried with.\n   * @param numOfNeighbors : Number of neighbours to be queried for.\n   * @param runtimeParams  : Runtime params associated with index to control accuracy/latency etc.\n   *\n   * @return List of approximate nearest neighbour ids with distance from the query embedding.\n   */\n  override def queryWithDistance(\n    embedding: EmbeddingVector,\n    numOfNeighbors: Int,\n    runtimeParams: P\n  ): Future[List[NeighborWithDistance[T, D]]] = {\n    queryWithDistance(embedding, numOfNeighbors, runtimeParams, None)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/UnsafeQueryIndexServer.scala",
    "content": "package com.twitter.ann.service.query_server.common\n\nimport com.google.common.util.concurrent.MoreExecutors\nimport com.google.inject.Module\nimport com.twitter.ann.common._\nimport com.twitter.ann.common.thriftscala.{Distance => ServiceDistance}\nimport com.twitter.ann.common.thriftscala.{RuntimeParams => ServiceRuntimeParams}\nimport com.twitter.app.Flag\nimport com.twitter.bijection.Injection\nimport com.twitter.cortex.ml.embeddings.common.EntityKind\nimport com.twitter.finatra.thrift.routing.ThriftRouter\nimport java.util.concurrent.ExecutorService\nimport java.util.concurrent.Executors\nimport java.util.concurrent.ThreadPoolExecutor\nimport java.util.concurrent.TimeUnit\n\n/**\n * This class is used when you do not know the generic parameters of the Server at compile time.\n * If you want compile time checks that your parameters are correct use QueryIndexServer instead.\n * In particular, when you want to have these id, distance and the runtime params as cli options you\n * should extend this class.\n */\nabstract class UnsafeQueryIndexServer[R <: RuntimeParams] extends BaseQueryIndexServer {\n  private[this] val metricName = flag[String](\"metric\", \"metric\")\n  private[this] val idType = flag[String](\"id_type\", \"type of ids to use\")\n  private[query_server] val queryThreads =\n    flag[Int](\n      \"threads\",\n      System\n        .getProperty(\"mesos.resources.cpu\", s\"${Runtime.getRuntime.availableProcessors()}\").toInt,\n      \"Size of thread pool for concurrent querying\"\n    )\n  private[query_server] val dimension = flag[Int](\"dimension\", \"dimension\")\n  private[query_server] val indexDirectory = flag[String](\"index_directory\", \"index directory\")\n  private[query_server] val refreshable =\n    flag[Boolean](\"refreshable\", false, \"if index is refreshable or not\")\n  private[query_server] val refreshableInterval =\n    flag[Int](\"refreshable_interval_minutes\", 10, \"refreshable index update interval\")\n  private[query_server] val sharded =\n    flag[Boolean](\"sharded\", false, \"if index is sharded\")\n  private[query_server] val shardedHours =\n    flag[Int](\"sharded_hours\", \"how many shards load at once\")\n  private[query_server] val shardedWatchLookbackIndexes =\n    flag[Int](\"sharded_watch_lookback_indexes\", \"how many indexes backwards to watch\")\n  private[query_server] val shardedWatchIntervalMinutes =\n    flag[Int](\"sharded_watch_interval_minutes\", \"interval at which hdfs is watched for changes\")\n  private[query_server] val minIndexSizeBytes =\n    flag[Long](\"min_index_size_byte\", 0, \"min index size in bytes\")\n  private[query_server] val maxIndexSizeBytes =\n    flag[Long](\"max_index_size_byte\", Long.MaxValue, \"max index size in bytes\")\n  private[query_server] val grouped =\n    flag[Boolean](\"grouped\", false, \"if indexes are grouped\")\n  private[query_server] val qualityFactorEnabled =\n    flag[Boolean](\n      \"quality_factor_enabled\",\n      false,\n      \"Enable dynamically reducing search complexity when cgroups container is throttled. Useful to disable when load testing\"\n    )\n  private[query_server] val warmup_enabled: Flag[Boolean] =\n    flag(\"warmup\", false, \"Enable warmup before the query server starts up\")\n\n  // Time to wait for the executor to finish before terminating the JVM\n  private[this] val terminationTimeoutMs = 100\n  protected lazy val executor: ExecutorService = MoreExecutors.getExitingExecutorService(\n    Executors.newFixedThreadPool(queryThreads()).asInstanceOf[ThreadPoolExecutor],\n    terminationTimeoutMs,\n    TimeUnit.MILLISECONDS\n  )\n\n  protected lazy val unsafeMetric: Metric[_] with Injection[_, ServiceDistance] = {\n    Metric.fromString(metricName())\n  }\n\n  override protected val additionalModules: Seq[Module] = Seq()\n\n  override final def addController(router: ThriftRouter): Unit = {\n    router.add(queryIndexThriftController)\n  }\n\n  protected def unsafeQueryableMap[T, D <: Distance[D]]: Queryable[T, R, D]\n  protected val runtimeInjection: Injection[R, ServiceRuntimeParams]\n\n  private[this] def queryIndexThriftController[\n    T,\n    D <: Distance[D]\n  ]: QueryIndexThriftController[T, R, D] = {\n    val controller = new QueryIndexThriftController[T, R, D](\n      statsReceiver.scope(\"ann_server\"),\n      unsafeQueryableMap.asInstanceOf[Queryable[T, R, D]],\n      runtimeInjection,\n      unsafeMetric.asInstanceOf[Injection[D, ServiceDistance]],\n      idInjection[T]()\n    )\n\n    logger.info(\"QueryIndexThriftController created....\")\n    controller\n  }\n\n  protected final def idInjection[T](): Injection[T, Array[Byte]] = {\n    val idInjection = idType() match {\n      case \"string\" => AnnInjections.StringInjection\n      case \"long\" => AnnInjections.LongInjection\n      case \"int\" => AnnInjections.IntInjection\n      case entityKind => EntityKind.getEntityKind(entityKind).byteInjection\n    }\n\n    idInjection.asInstanceOf[Injection[T, Array[Byte]]]\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/throttling/AuroraCPUStatsReader.scala",
    "content": "package com.twitter.ann.service.query_server.common.throttling\n\nimport com.twitter.server.filter.CgroupsCpu\n\nclass AuroraCPUStatsReader() {\n\n  val cgroupsCpu = new CgroupsCpu()\n\n  def throttledTimeNanos(): Option[Long] = cgroupsCpu.cpuStat.map { cs =>\n    cs.throttledTimeNanos\n  }\n\n  /**\n   * Read assigned cpu number from Mesos files\n   *\n   * @return positive number is the number of CPUs (can be fractional).\n   * -1 means file read failed or it's not a valid Mesos environment.\n   */\n  def cpuQuota: Double = cgroupsCpu.cfsPeriodMicros match {\n    case -1L => -1.0\n    case 0L => 0.0 // avoid divide by 0\n    case periodMicros =>\n      cgroupsCpu.cfsQuotaMicros match {\n        case -1L => -1.0\n        case quotaMicros => quotaMicros.toDouble / periodMicros.toDouble\n      }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/throttling/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/faiss\",\n        \"ann/src/main/scala/com/twitter/ann/hnsw\",\n        \"twitter-server-internal/src/main/scala\",\n        \"util/util-stats/src/main/scala/com/twitter/finagle/stats\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/throttling/ThrottlingBasedQualityTask.scala",
    "content": "package com.twitter.ann.service.query_server.common.throttling\n\nimport com.twitter.ann.common.RuntimeParams\nimport com.twitter.ann.common.Task\nimport com.twitter.ann.faiss.FaissParams\nimport com.twitter.ann.hnsw.HnswParams\nimport com.twitter.ann.service.query_server.common.throttling.ThrottlingBasedQualityTask.SAMPLING_INTERVAL\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.logging.Logging\n\nobject ThrottlingBasedQualityTask {\n  private[throttling] val SAMPLING_INTERVAL = 100.milliseconds\n}\n\nclass ThrottlingBasedQualityTask(\n  override val statsReceiver: StatsReceiver,\n  // Parameters are taken from OverloadAdmissionController\n  instrument: ThrottlingInstrument = new WindowedThrottlingInstrument(SAMPLING_INTERVAL, 5,\n    new AuroraCPUStatsReader()))\n    extends Task\n    with Logging {\n  import ThrottlingBasedQualityTask._\n\n  // [0, 1] where 1 is fully throttled\n  // Quickly throttle, but dampen recovery to make sure we won't enter throttle/GC death spiral\n  @volatile private var dampenedThrottlingPercentage: Double = 0\n\n  protected[throttling] def task(): Future[Unit] = {\n    if (!instrument.disabled) {\n      instrument.sample()\n\n      val delta = instrument.percentageOfTimeSpentThrottling - dampenedThrottlingPercentage\n      if (delta > 0) {\n        // We want to start shedding load, do it quickly\n        dampenedThrottlingPercentage += delta\n      } else {\n        // Recover much slower\n        // At the rate of 100ms per sample, lookback is 2 minutes\n        val samplesToConverge = 1200.toDouble\n        dampenedThrottlingPercentage =\n          dampenedThrottlingPercentage + delta * (2 / (samplesToConverge + 1))\n      }\n\n      statsReceiver.stat(\"dampened_throttling\").add(dampenedThrottlingPercentage.toFloat * 100)\n    }\n\n    Future.Unit\n  }\n\n  protected def taskInterval: Duration = SAMPLING_INTERVAL\n\n  def discountParams[T <: RuntimeParams](params: T): T = {\n    // [0, 1] where 1 is best quality and lowest speed\n    // It's expected to run @1 majority of time\n    val qualityFactor = math.min(1, math.max(0, 1 - dampenedThrottlingPercentage))\n    def applyQualityFactor(param: Int) = math.max(1, math.ceil(param * qualityFactor).toInt)\n\n    params match {\n      case HnswParams(ef) => HnswParams(applyQualityFactor(ef)).asInstanceOf[T]\n      case FaissParams(nprobe, quantizerEf, quantizerKFactorRF, quantizerNprobe, ht) =>\n        FaissParams(\n          nprobe.map(applyQualityFactor),\n          quantizerEf.map(applyQualityFactor),\n          quantizerKFactorRF.map(applyQualityFactor),\n          quantizerNprobe.map(applyQualityFactor),\n          ht.map(applyQualityFactor)\n        ).asInstanceOf[T]\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/throttling/WindowedStats.scala",
    "content": "package com.twitter.ann.service.query_server.common.throttling\n\n/**\n * A simple ring buffer that keeps track of long values over `window`.\n */\nprivate[throttling] class WindowedStats(window: Int) {\n  private[this] val buffer = new Array[Long](window)\n  private[this] var index = 0\n  private[this] var sumValue = 0L\n  private[this] var count = 0\n\n  def add(v: Long): Unit = {\n    count = math.min(count + 1, window)\n    val old = buffer(index)\n    buffer(index) = v\n    index = (index + 1) % window\n    sumValue += v - old\n  }\n\n  def avg: Double = { sumValue.toDouble / count }\n  def sum: Long = { sumValue }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/throttling/WindowedThrottlingInstrument.scala",
    "content": "package com.twitter.ann.service.query_server.common.throttling\n\nimport com.twitter.util.Duration\n\ntrait ThrottlingInstrument {\n  def sample(): Unit\n  def percentageOfTimeSpentThrottling(): Double\n  def disabled: Boolean\n}\n\nclass WindowedThrottlingInstrument(\n  stepFrequency: Duration,\n  windowLengthInFrequencySteps: Int,\n  reader: AuroraCPUStatsReader)\n    extends ThrottlingInstrument {\n  private[this] val throttlingChangeHistory: WindowedStats = new WindowedStats(\n    windowLengthInFrequencySteps)\n\n  private[this] val cpuQuota: Double = reader.cpuQuota\n\n  // The total number of allotted CPU time per step (in nanos).\n  private[this] val assignedCpu: Duration = stepFrequency * cpuQuota\n  private[this] val assignedCpuNs: Long = assignedCpu.inNanoseconds\n\n  @volatile private[this] var previousThrottledTimeNs: Long = 0\n\n  /**\n   * If there isn't a limit on how much cpu the container can use, aurora\n   * throttling will never kick in.\n   */\n  final def disabled: Boolean = cpuQuota <= 0\n\n  def sample(): Unit = sampleThrottling() match {\n    case Some(load) =>\n      throttlingChangeHistory.add(load)\n    case None => ()\n  }\n\n  private[this] def sampleThrottling(): Option[Long] = reader.throttledTimeNanos().map {\n    throttledTimeNs =>\n      val throttlingChange = throttledTimeNs - previousThrottledTimeNs\n      previousThrottledTimeNs = throttledTimeNs\n      throttlingChange\n  }\n\n  // Time spent throttling over windowLength, normalized by number of CPUs\n  def percentageOfTimeSpentThrottling(): Double = {\n    math.min(1, throttlingChangeHistory.sum.toDouble / assignedCpuNs)\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/warmup/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"util/util-core:scala\",\n        \"util/util-slf4j-api/src/main/scala/com/twitter/util/logging\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/common/warmup/Warmup.scala",
    "content": "package com.twitter.ann.service.query_server.common.warmup\n\nimport com.twitter.ann.common.EmbeddingType.EmbeddingVector\nimport com.twitter.ml.api.embedding.Embedding\nimport com.twitter.util.Await\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\nimport scala.annotation.tailrec\nimport scala.util.Random\n\ntrait Warmup extends Logging {\n  protected def minSuccessfulTries: Int\n  protected def maxTries: Int\n  protected def randomQueryDimension: Int\n  protected def timeout: Duration\n\n  @tailrec\n  final protected def run(\n    iteration: Int = 0,\n    successes: Int = 0,\n    name: String,\n    f: => Future[_]\n  ): Unit = {\n    if (successes == minSuccessfulTries || iteration == maxTries) {\n      info(s\"Warmup finished after ${iteration} iterations with ${successes} successes\")\n    } else {\n      Try(Await.result(f.liftToTry, timeout)) match {\n        case Return(Return(_)) =>\n          debug(s\"[$name] Iteration $iteration Success\")\n          run(iteration + 1, successes + 1, name, f)\n        case Return(Throw(e)) =>\n          warn(s\"[$name] Iteration $iteration has failed: ${e.getMessage}. \", e)\n          run(iteration + 1, successes, name, f)\n        case Throw(e) =>\n          info(s\"[$name] Iteration $iteration was too slow: ${e.getMessage}. \", e)\n          run(iteration + 1, successes, name, f)\n      }\n    }\n  }\n\n  private val rng = new Random()\n  protected def randomQuery(): EmbeddingVector =\n    Embedding(Array.fill(randomQueryDimension)(-1 + 2 * rng.nextFloat()))\n\n  def warmup(): Unit\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/faiss/BUILD",
    "content": "scala_library(\n    name = \"server\",\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ann/src/main/java/com/twitter/ann/faiss\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/faiss\",\n        \"ann/src/main/scala/com/twitter/ann/service/query_server/common\",\n        \"ann/src/main/scala/com/twitter/ann/service/query_server/common:faiss_index_path_provider\",\n        \"ann/src/main/scala/com/twitter/ann/service/query_server/common/throttling\",\n        \"ann/src/main/scala/com/twitter/ann/service/query_server/common/warmup\",\n        \"src/java/com/twitter/search/common/file\",\n    ],\n)\n\njvm_binary(\n    name = \"faiss-query-server\",\n    main = \"com.twitter.ann.service.query_server.faiss.FaissQueryIndexServer\",\n    compiler_option_sets = [\"fatal_warnings\"],\n    runtime_platform = \"java11\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":server\",\n        \"3rdparty/jvm/ch/qos/logback:logback-classic\",\n        \"3rdparty/jvm/org/slf4j:jcl-over-slf4j\",\n        \"3rdparty/jvm/org/slf4j:jul-to-slf4j\",\n        \"3rdparty/jvm/org/slf4j:log4j-over-slf4j\",\n        \"ann/src/main/resources\",\n        \"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/faiss/FaissQueryIndexServer.scala",
    "content": "package com.twitter.ann.service.query_server.faiss\n\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common.QueryableOperations.Map\nimport com.twitter.ann.common._\nimport com.twitter.ann.common.thriftscala.{RuntimeParams => ServiceRuntimeParams}\nimport com.twitter.ann.faiss.FaissCommon\nimport com.twitter.ann.faiss.FaissIndex\nimport com.twitter.ann.faiss.FaissParams\nimport com.twitter.ann.faiss.HourlyShardedIndex\nimport com.twitter.ann.service.query_server.common.QueryableProvider\nimport com.twitter.ann.service.query_server.common.RefreshableQueryable\nimport com.twitter.ann.service.query_server.common.UnsafeQueryIndexServer\nimport com.twitter.ann.service.query_server.common.FaissIndexPathProvider\nimport com.twitter.ann.service.query_server.common.throttling.ThrottlingBasedQualityTask\nimport com.twitter.ann.service.query_server.common.warmup.Warmup\nimport com.twitter.bijection.Injection\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.search.common.file.FileUtils\nimport com.twitter.util.Duration\nimport java.util.concurrent.TimeUnit\n\nobject FaissQueryIndexServer extends FaissQueryableServer\n\nclass FaissQueryableServer extends UnsafeQueryIndexServer[FaissParams] {\n  // given a directory, how to load it as a queryable index\n  def queryableProvider[T, D <: Distance[D]]: QueryableProvider[T, FaissParams, D] =\n    new QueryableProvider[T, FaissParams, D] {\n      override def provideQueryable(\n        directory: AbstractFile\n      ): Queryable[T, FaissParams, D] = {\n        FaissIndex.loadIndex[T, D](\n          dimension(),\n          unsafeMetric.asInstanceOf[Metric[D]],\n          directory\n        )\n      }\n    }\n\n  private def buildSimpleQueryable[T, D <: Distance[D]](\n    dir: AbstractFile\n  ): Queryable[T, FaissParams, D] = {\n    val queryable = if (refreshable()) {\n      logger.info(s\"build refreshable queryable\")\n      val updatableQueryable = new RefreshableQueryable(\n        false,\n        dir,\n        queryableProvider.asInstanceOf[QueryableProvider[T, FaissParams, D]],\n        FaissIndexPathProvider(\n          minIndexSizeBytes(),\n          maxIndexSizeBytes(),\n          statsReceiver.scope(\"validated_index_provider\")\n        ),\n        statsReceiver.scope(\"refreshable_queryable\"),\n        updateInterval = refreshableInterval().minutes\n      )\n      // init first load of index and also schedule the following reloads\n      updatableQueryable.start()\n      updatableQueryable.asInstanceOf[QueryableGrouped[T, FaissParams, D]]\n    } else {\n      logger.info(s\"build non-refreshable queryable\")\n\n      logger.info(s\"Loading ${dir}\")\n      queryableProvider.provideQueryable(dir).asInstanceOf[Queryable[T, FaissParams, D]]\n    }\n\n    logger.info(\"Faiss queryable created....\")\n    queryable\n  }\n\n  private def buildShardedQueryable[T, D <: Distance[D]](\n    dir: AbstractFile\n  ): Queryable[T, FaissParams, D] = {\n    logger.info(s\"build sharded queryable\")\n\n    val queryable = HourlyShardedIndex.loadIndex[T, D](\n      dimension(),\n      unsafeMetric.asInstanceOf[Metric[D]],\n      dir,\n      shardedHours(),\n      Duration(shardedWatchIntervalMinutes(), TimeUnit.MINUTES),\n      shardedWatchLookbackIndexes(),\n      statsReceiver.scope(\"hourly_sharded_index\")\n    )\n\n    logger.info(\"Faiss sharded queryable created....\")\n\n    closeOnExit(queryable)\n    queryable.startImmediately()\n\n    logger.info(\"Directory watching is scheduled\")\n\n    queryable\n  }\n\n  // Readings come incorrect if reader is created too early in the lifecycle of a server\n  // hence lazy\n  private lazy val throttleSamplingTask = new ThrottlingBasedQualityTask(\n    statsReceiver.scope(\"throttling_task\"))\n\n  override def unsafeQueryableMap[T, D <: Distance[D]]: Queryable[T, FaissParams, D] = {\n    val dir = FileUtils.getFileHandle(indexDirectory())\n\n    val queryable = if (sharded()) {\n      require(shardedHours() > 0, \"Number of hourly shards must be specified\")\n      require(shardedWatchIntervalMinutes() > 0, \"Shard watch interval must be specified\")\n      require(shardedWatchLookbackIndexes() > 0, \"Index lookback must be specified\")\n      buildShardedQueryable[T, D](dir)\n    } else {\n      buildSimpleQueryable[T, D](dir)\n    }\n\n    if (qualityFactorEnabled()) {\n      logger.info(\"Quality Factor throttling is enabled\")\n      closeOnExit(throttleSamplingTask)\n      throttleSamplingTask.jitteredStart()\n\n      queryable.mapRuntimeParameters(throttleSamplingTask.discountParams)\n    } else {\n      queryable\n    }\n  }\n\n  override val runtimeInjection: Injection[FaissParams, ServiceRuntimeParams] =\n    FaissCommon.RuntimeParamsInjection\n\n  protected override def warmup(): Unit =\n    if (warmup_enabled())\n      new FaissWarmup(unsafeQueryableMap, dimension()).warmup()\n}\n\nclass FaissWarmup(faiss: Queryable[_, FaissParams, _], dimension: Int) extends Warmup {\n  protected def minSuccessfulTries: Int = 100\n  protected def maxTries: Int = 1000\n  protected def timeout: Duration = 50.milliseconds\n  protected def randomQueryDimension: Int = dimension\n\n  def warmup(): Unit = {\n    run(\n      name = \"queryWithDistance\",\n      f = faiss\n        .queryWithDistance(\n          randomQuery(),\n          100,\n          FaissParams(nprobe = Some(128), None, None, None, None))\n    )\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/hnsw/BUILD",
    "content": "scala_library(\n    name = \"server\",\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/hnsw\",\n        \"ann/src/main/scala/com/twitter/ann/service/query_server/common\",\n        \"ann/src/main/scala/com/twitter/ann/service/query_server/common/warmup\",\n        \"src/java/com/twitter/search/common/file\",\n    ],\n)\n\njvm_binary(\n    name = \"hnsw-query-server\",\n    main = \"com.twitter.ann.service.query_server.hnsw.HnswQueryIndexServer\",\n    compiler_option_sets = [\"fatal_warnings\"],\n    runtime_platform = \"java11\",\n    tags = [\n        \"bazel-compatible\",\n    ],\n    dependencies = [\n        \":server\",\n        \"3rdparty/jvm/ch/qos/logback:logback-classic\",\n        \"3rdparty/jvm/org/slf4j:jcl-over-slf4j\",\n        \"3rdparty/jvm/org/slf4j:jul-to-slf4j\",\n        \"3rdparty/jvm/org/slf4j:log4j-over-slf4j\",\n        \"ann/src/main/resources\",\n        \"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/service/query_server/hnsw/HnswQueryIndexServer.scala",
    "content": "package com.twitter.ann.service.query_server.hnsw\n\nimport com.twitter.ann.common.Distance\nimport com.twitter.ann.common._\nimport com.twitter.ann.common.thriftscala.{RuntimeParams => ServiceRuntimeParams}\nimport com.twitter.ann.hnsw.HnswCommon\nimport com.twitter.ann.hnsw.HnswParams\nimport com.twitter.ann.hnsw.TypedHnswIndex\nimport com.twitter.ann.service.query_server.common.QueryableProvider\nimport com.twitter.ann.service.query_server.common.RefreshableQueryable\nimport com.twitter.ann.service.query_server.common.UnsafeQueryIndexServer\nimport com.twitter.ann.service.query_server.common.ValidatedIndexPathProvider\nimport com.twitter.ann.service.query_server.common.warmup.Warmup\nimport com.twitter.bijection.Injection\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.search.common.file.AbstractFile\nimport com.twitter.search.common.file.FileUtils\nimport com.twitter.util.Duration\nimport com.twitter.util.FuturePool\n\n// Creating a separate hnsw query server object, since unit test require non singleton server.\nobject HnswQueryIndexServer extends HnswQueryableServer\n\nclass HnswQueryableServer extends UnsafeQueryIndexServer[HnswParams] {\n  private val IndexGroupPrefix = \"group_\"\n\n  // given a directory, how to load it as a queryable index\n  def queryableProvider[T, D <: Distance[D]]: QueryableProvider[T, HnswParams, D] =\n    new QueryableProvider[T, HnswParams, D] {\n      override def provideQueryable(\n        dir: AbstractFile\n      ): Queryable[T, HnswParams, D] = {\n        TypedHnswIndex.loadIndex[T, D](\n          dimension(),\n          unsafeMetric.asInstanceOf[Metric[D]],\n          idInjection[T](),\n          ReadWriteFuturePool(FuturePool.interruptible(executor)),\n          dir\n        )\n      }\n    }\n\n  private def buildQueryable[T, D <: Distance[D]](\n    dir: AbstractFile,\n    grouped: Boolean\n  ): Queryable[T, HnswParams, D] = {\n    val queryable = if (refreshable()) {\n      logger.info(s\"build refreshable queryable\")\n      val updatableQueryable = new RefreshableQueryable(\n        grouped,\n        dir,\n        queryableProvider.asInstanceOf[QueryableProvider[T, HnswParams, D]],\n        ValidatedIndexPathProvider(\n          minIndexSizeBytes(),\n          maxIndexSizeBytes(),\n          statsReceiver.scope(\"validated_index_provider\")\n        ),\n        statsReceiver.scope(\"refreshable_queryable\"),\n        updateInterval = refreshableInterval().minutes\n      )\n      // init first load of index and also schedule the following reloads\n      updatableQueryable.start()\n      updatableQueryable.asInstanceOf[QueryableGrouped[T, HnswParams, D]]\n    } else {\n      logger.info(s\"build non-refreshable queryable\")\n      queryableProvider.provideQueryable(dir).asInstanceOf[Queryable[T, HnswParams, D]]\n    }\n\n    logger.info(\"Hnsw queryable created....\")\n    queryable\n  }\n\n  override def unsafeQueryableMap[T, D <: Distance[D]]: Queryable[T, HnswParams, D] = {\n    val dir = FileUtils.getFileHandle(indexDirectory())\n    buildQueryable(dir, grouped())\n  }\n\n  override val runtimeInjection: Injection[HnswParams, ServiceRuntimeParams] =\n    HnswCommon.RuntimeParamsInjection\n\n  protected override def warmup(): Unit =\n    if (warmup_enabled()) new HNSWWarmup(unsafeQueryableMap, dimension()).warmup()\n}\n\nclass HNSWWarmup(hnsw: Queryable[_, HnswParams, _], dimension: Int) extends Warmup {\n  protected def minSuccessfulTries: Int = 100\n  protected def maxTries: Int = 1000\n  protected def timeout: Duration = 50.milliseconds\n  protected def randomQueryDimension: Int = dimension\n\n  def warmup(): Unit = {\n    run(\n      name = \"queryWithDistance\",\n      f = hnsw\n        .queryWithDistance(randomQuery(), 100, HnswParams(ef = 800))\n    )\n  }\n}\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/util/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"util/util-logging\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/scala/com/twitter/ann/util/IndexBuilderUtils.scala",
    "content": "package com.twitter.ann.util\n\nimport com.twitter.ann.common.{Appendable, EntityEmbedding}\nimport com.twitter.concurrent.AsyncStream\nimport com.twitter.logging.Logger\nimport com.twitter.util.Future\nimport java.util.concurrent.atomic.AtomicInteger\n\nobject IndexBuilderUtils {\n  val Log = Logger.apply()\n\n  def addToIndex[T](\n    appendable: Appendable[T, _, _],\n    embeddings: Seq[EntityEmbedding[T]],\n    concurrencyLevel: Int\n  ): Future[Int] = {\n    val count = new AtomicInteger()\n    // Async stream allows us to procss at most concurrentLevel futures at a time.\n    Future.Unit.before {\n      val stream = AsyncStream.fromSeq(embeddings)\n      val appendStream = stream.mapConcurrent(concurrencyLevel) { annEmbedding =>\n        val processed = count.incrementAndGet()\n        if (processed % 10000 == 0) {\n          Log.info(s\"Performed $processed updates\")\n        }\n        appendable.append(annEmbedding)\n      }\n      appendStream.size\n    }\n  }\n}\n"
  },
  {
    "path": "ann/src/main/thrift/com/twitter/ann/common/BUILD",
    "content": "create_thrift_libraries(\n    base_name = \"ann-common\",\n    sources = [\"*.thrift\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependency_roots = [\n        \"mediaservices/commons/src/main/thrift\",\n        \"src/thrift/com/twitter/ml/api:embedding\",\n    ],\n    generate_languages = [\n        \"java\",\n        \"python\",\n        \"scala\",\n        \"strato\",\n    ],\n    provides_java_name = \"ann-common-thrift-java\",\n    provides_scala_name = \"ann-common-thrift-scala\",\n)\n"
  },
  {
    "path": "ann/src/main/thrift/com/twitter/ann/common/ann_common.thrift",
    "content": "namespace java com.twitter.ann.common.thriftjava\n#@namespace scala com.twitter.ann.common.thriftscala\n#@namespace strato com.twitter.ann.common\nnamespace py gen.twitter.ann.common\n\ninclude \"com/twitter/mediaservices/commons/ServerCommon.thrift\"\ninclude \"com/twitter/ml/api/embedding.thrift\"\n\n/**\n* Thrift schema for storing file based Index mapping\n*/\nstruct FileBasedIndexIdStore {\n  1: optional map<i64, binary> indexIdMap\n}\n\nenum DistanceMetric {\n  L2, Cosine, InnerProduct, \n  RESERVED_4, RESERVED_5, RESERVED_6, RESERVED_7, EditDistance\n} (persisted = 'true',  strato.graphql.typename='DistanceMetric')\n\nstruct AnnoyIndexMetadata {\n  1: i32 dimension\n  2: DistanceMetric distanceMetric\n  3: i32 numOfTrees\n  4: i64 numOfVectorsIndexed\n} (persisted = 'true',  strato.graphql.typename='AnnoyIndexMetadata')\n\nstruct AnnoyRuntimeParam {\n  /* Number of vectors to evaluate while searching. A larger value will give more accurate results, but will take longer time to return.\n   * Default value would be numberOfTrees*numberOfNeigboursRequested\n   */\n  1: optional i32 numOfNodesToExplore\n}\n\nstruct HnswRuntimeParam {\n  // More the value of ef better the recall with but at cost of latency.\n  // Set it greater than equal to number of neighbours required.\n  1: i32 ef\n}\n\n// These options are subset of all possible parameters, defined by\n// https://github.com/facebookresearch/faiss/blob/36f2998a6469280cef3b0afcde2036935a29aa1f/faiss/AutoTune.cpp#L444\n// quantizer_ prefix changes IndexIVF.quantizer parameters instead\nstruct FaissRuntimeParam {\n  // How many cells to visit in IVFPQ. Higher is slower / more precise.\n  1: optional i32 nprobe\n  // Depth of search in HNSW. Higher is slower / more precise.\n  2: optional i32 quantizer_ef\n  // How many times more neighbours are requested from underlying index by IndexRefine.\n  3: optional i32 quantizer_kfactor_rf\n  // Same as 1: but for quantizer\n  4: optional i32 quantizer_nprobe\n  // Hamming distance threshold to filter neighbours when searching.\n  5: optional i32 ht\n}\n\n// Every ANN index will have this metadata and it'll be used by the query service for validation.\nstruct AnnIndexMetadata {\n 1: optional i64 timestamp\n 2: optional i32 index_size\n 3: optional bool withGroups\n 4: optional i32 numGroups\n} (persisted = 'true')\n\nstruct HnswIndexMetadata {\n 1: i32 dimension\n 2: DistanceMetric distanceMetric\n 3: i32 numElements\n} (persisted = 'true')\n\nstruct HnswInternalIndexMetadata {\n 1: i32 maxLevel\n 2: optional binary entryPoint\n 3: i32 efConstruction\n 4: i32 maxM\n 5: i32 numElements\n} (persisted = 'true')\n\nstruct HnswGraphEntry {\n  1: i32 level\n  2: binary key\n  3: list<binary> neighbours\n} (persisted = 'true', strato.graphql.typename='HnswGraphEntry')\n\nenum IndexType {\n   TWEET, \n   USER, \n   WORD, \n   LONG, \n   INT, \n   STRING, \n   RESERVED_7, RESERVED_8, RESERVED_9, RESERVED_10\n} (persisted = 'true',  strato.graphql.typename='IndexType')\n\nstruct CosineDistance {\n  1: required double distance\n}\n\nstruct L2Distance {\n  1: required double distance\n}\n\nstruct InnerProductDistance {\n  1: required double distance\n}\n\nstruct EditDistance {\n  1: required i32 distance\n}\n\nunion Distance {\n  1: CosineDistance cosineDistance\n  2: L2Distance l2Distance\n  3: InnerProductDistance innerProductDistance\n  4: EditDistance editDistance\n}\n\nstruct NearestNeighbor {\n  1: required binary id\n  2: optional Distance distance\n}\n\nstruct NearestNeighborResult {\n  // This list is ordered from nearest to furthest neighbor\n  1: required list<NearestNeighbor> nearestNeighbors\n}\n\n// Different runtime/tuning params while querying for indexes to control accuracy/latency etc..\nunion RuntimeParams {\n  1: AnnoyRuntimeParam annoyParam\n  2: HnswRuntimeParam hnswParam\n  3: FaissRuntimeParam faissParam\n}\n\nstruct NearestNeighborQuery {\n  1: required embedding.Embedding embedding\n  2: required bool with_distance\n  3: required RuntimeParams runtimeParams,\n  4: required i32 numberOfNeighbors,\n  // The purpose of the key here is to load the index in memory as a map of Option[key] to index\n  // If the key is not specified in the query, the map value corresponding to None key will be used\n  // as the queryable index to perform Nearest Neighbor search on\n  5: optional string key\n}\n\nenum BadRequestCode {\n  VECTOR_DIMENSION_MISMATCH,\n  RESERVED_2,\n  RESERVED_3,\n  RESERVED_4,\n  RESERVED_5,\n  RESERVED_6,\n  RESERVED_7,\n  RESERVED_8,\n  RESERVED_9\n}\n\nexception BadRequest {\n  1: string message\n  2: required BadRequestCode code\n}\n\nservice AnnQueryService {\n  /**\n  * Get approximate nearest neighbor for a given vector\n  */\n  NearestNeighborResult query(1: NearestNeighborQuery query)\n    throws (1: ServerCommon.ServerError serverError, 2: BadRequest badRequest)\n}\n"
  },
  {
    "path": "ann/src/main/thrift/com/twitter/ann/knn/BUILD",
    "content": "create_thrift_libraries(\n    base_name = \"thrift\",\n    sources = [\"*.thrift\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependency_roots = [\"src/thrift/com/twitter/ml/featurestore:ml-feature-store\"],\n    generate_languages = [\n        \"java\",\n        \"scala\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/thrift/com/twitter/ann/knn/knn.thrift",
    "content": "namespace java com.twitter.ann.knn.thriftjava\n#@namespace scala com.twitter.ann.knn.thriftscala\nnamespace py gen.twitter.ann.knn\n\ninclude \"com/twitter/ml/featurestore/entity.thrift\"\n\nstruct Neighbor {\n  1: required double distance\n  2: required entity.EntityId id\n} (persisted = \"true\")\n\nstruct Knn {\n  1: required entity.EntityId queryId\n  2: required list<Neighbor> neighbors\n}(persisted='true')\n"
  },
  {
    "path": "ann/src/main/thrift/com/twitter/ann/serialization/BUILD",
    "content": "create_thrift_libraries(\n    base_name = \"serialization\",\n    sources = [\"*.thrift\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependency_roots = [\n        \"src/thrift/com/twitter/ml/api:embedding\",\n    ],\n    generate_languages = [\n        \"java\",\n        \"scala\",\n    ],\n)\n"
  },
  {
    "path": "ann/src/main/thrift/com/twitter/ann/serialization/serialization.thrift",
    "content": "#@namespace scala com.twitter.ann.serialization.thriftscala\n\ninclude \"com/twitter/ml/api/embedding.thrift\"\n/**\n* Thrift schema for storing embeddings in a file\n*/\nstruct PersistedEmbedding {\n  1: required binary id\n  2: required embedding.Embedding embedding\n}(persisted = 'true')\n"
  },
  {
    "path": "ci/ci.sh",
    "content": "#!/bin/sh\n\nexit 0\n"
  },
  {
    "path": "cr-mixer/BUILD.bazel",
    "content": "jvm_binary(\n    name = \"bin\",\n    basename = \"cr-mixer\",\n    main = \"com.twitter.cr_mixer.CrMixerServerMain\",\n    runtime_platform = \"java11\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/ch/qos/logback:logback-classic\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer\",\n        \"finagle/finagle-zipkin-scribe/src/main/scala\",\n        \"finatra/inject/inject-logback/src/main/scala\",\n        \"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback\",\n        \"twitter-server-internal/src/main/scala\",\n        \"twitter-server/logback-classic/src/main/scala\",\n    ],\n)\n\n#  Aurora Workflows build phase convention requires a jvm_app named with ${project-name}-app\njvm_app(\n    name = \"cr-mixer-app\",\n    archive = \"zip\",\n    binary = \":bin\",\n    tags = [\"bazel-compatible\"],\n)\n"
  },
  {
    "path": "cr-mixer/README.md",
    "content": "# CR-Mixer\n\nCR-Mixer is a candidate generation service proposed as part of the Personalization Strategy vision for Twitter. Its aim is to speed up the iteration and development of candidate generation and light ranking. The service acts as a lightweight coordinating layer that delegates candidate generation tasks to underlying compute services. It focuses on Twitter's candidate generation use cases and offers a centralized platform for fetching, mixing, and managing candidate sources and light rankers. The overarching goal is to increase the speed and ease of testing and developing candidate generation pipelines, ultimately delivering more value to Twitter users.\n\nCR-Mixer acts as a configurator and delegator, providing abstractions for the challenging parts of candidate generation and handling performance issues. It will offer a 1-stop-shop for fetching and mixing candidate sources, a managed and shared performant platform, a light ranking layer, a common filtering layer, a version control system, a co-owned feature switch set, and peripheral tooling.\n\nCR-Mixer's pipeline consists of 4 steps: source signal extraction, candidate generation, filtering, and ranking. It also provides peripheral tooling like scribing, debugging, and monitoring. The service fetches source signals externally from stores like UserProfileService and RealGraph, calls external candidate generation services, and caches results. Filters are applied for deduping and pre-ranking, and a light ranking step follows.\n"
  },
  {
    "path": "cr-mixer/server/src/main/resources/BUILD.bazel",
    "content": "resources(\n    sources = [\n        \"*.xml\",\n        \"*.yml\",\n        \"config/*.yml\",\n    ],\n    tags = [\"bazel-compatible\"],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/resources/config/decider.yml",
    "content": "# The keys in this file correspond to the DeciderValues defined in\n# https://sourcegraph.twitter.biz/git.twitter.biz/source/-/blob/cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider/DeciderKey.scala\n\ndark_traffic_filter:\n  comment: Proportion of the requests that are forwarded as dark traffic to the proxy\n  default_availability: 0\n\nenable_tweet_recommendations_home_product:\n  comment: Proportion of requests where we return an actual response for TweetRecommendations Home product\n  default_availability: 10000\n\nenable_tweet_health_score:\n  comment: \"Enable the calculation for health scores in tweetInfo. By enabling this decider, we will compute TweetHealthModelScore\"\n  default_availability: 10000\n\nenable_user_agatha_score:\n  comment: \"Enable the calculation for health scores in tweetInfo. By enabling this decider, we will compute UserHealthModelScore\"\n  default_availability: 10000\n\nenable_user_tweet_entity_graph_traffic:\n  comment: \"Enable the traffic to user entity tweet graph to fetch liked-by tweets candidates\"\n  default_availability: 10000\n\nenable_user_tweet_graph_traffic:\n  comment: \"Enable the traffic to user tweet graph to fetch similar tweets candidates\"\n  default_availability: 10000\n\nenable_user_video_graph_traffic:\n  comment: \"Enable the traffic to user video graph to fetch similar tweets candidates\"\n  default_availability: 10000\n\nenable_user_ad_graph_traffic:\n  comment: \"Enable the traffic to user ad graph to fetch similar tweets candidates\"\n  default_availability: 10000\n\nenable_qig_similar_tweets_traffic:\n  comment: \"Enable the traffic to QIG to fetch similar tweet candidates\"\n  default_availability: 0\n\nenable_frs_traffic:\n  comment: \"Enable the traffic to FRS to fetch user follow recommendations\"\n  default_availability: 0\n\nenable_hydra_dark_traffic:\n  comment: \"Enable dark traffic to hydra\"\n  default_availability: 0\n\nenable_real_graph_mh_store:\n  comment: \"Enable traffic for the real graph manhattan based store\"\n  default_availability: 0\n\nenable_simclusters_ann_experimental_dark_traffic:\n  comment: \"Enable dark traffic to simclusters-ann-experimental\"\n  default_availability: 0\n\nenable_simclusters_ann_2_dark_traffic:\n  comment: \"Enable dark traffic to prod SimClustersANN2\"\n  default_availability: 0\n\nenable_user_state_store:\n  comment: \"Enable traffic user state store to hydrate user state\"\n  default_availability: 0\n\nupper_funnel_per_step_scribe_rate:\n  comment: \"Enable Upper Funnel Event Scribe Sampling (fetch, pre-rank, interleave etc.) for getTweetsRecommendations() endpoint\"\n  default_availability: 0\n\nkafka_message_scribe_sample_rate:\n  comment: \"Gates the production of forked scribe messages to kafka for the async feature hydrator\"\n  default_availability: 0\n\ntop_level_api_ddg_metrics_scribe_rate:\n  comment: \"Enable Top Level API DDG Metrics Scribe Sampling for getTweetsRecommendations() endpoint\"\n  default_availability: 0\n\nads_recommendations_per_experiment_scribe_rate:\n  comment: \"Percentage of DDG traffic to Scribe for getAdsRecommendations() endpoint\"\n  default_availability: 0\n\nenable_loadshedding_getTweetRecommendations:\n  comment: \"Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response\"\n  default_availability: 0\n\nenable_loadshedding_getTweetRecommendations_Home:\n  comment: \"Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response\"\n  default_availability: 0\n\nenable_loadshedding_getTweetRecommendations_Notifications:\n  comment: \"Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response\"\n  default_availability: 0\n\nenable_loadshedding_getTweetRecommendations_Email:\n  comment: \"Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response\"\n  default_availability: 0\n\nenable_loadshedding_getRelatedTweetsForQueryTweet:\n  comment: \"Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response\"\n  default_availability: 0\n\nenable_loadshedding_getRelatedTweetsForQueryTweet_Home:\n  comment: \"Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response\"\n  default_availability: 0\n\nenable_loadshedding_getRelatedTweetsForQueryTweet_MoreTweetsModule:\n  comment: \"Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response\"\n  default_availability: 0\n\nenable_loadshedding_getRelatedTweetsForQueryAuthor:\n  comment: \"Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response\"\n  default_availability: 0\n\nenable_loadshedding_getRelatedTweetsForQueryAuthor_MoreTweetsModule:\n  comment: \"Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response\"\n  default_availability: 0\n\nenable_loadshedding_getFrsBasedTweetRecommendations_Home:\n  comment: \"Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response\"\n  default_availability: 0\n\nenable_loadshedding_getFrsBasedTweetRecommendations_Notifications:\n  comment: \"Enable loadshedding (from 0% to 100%). Requests that have been shed will return an empty response\"\n  default_availability: 0\n\nenable_user_media_representation_store:\n  comment: \"Enable fetching user nudity rate signal from Media Understanding\"\n  default_availability: 0\n\nenable_magic_recs_real_time_aggregates_store:\n  comment: \"Enable fetching real time aggregates features from Magic Recs memcache\"\n  default_availability: 0\n\nenable_utg_realtime_tweet_engagement_score:\n  comment: \"Enable fetching real time tweet engagement score from utg-plus\"\n  default_availability: 0\n\nget_tweet_recommendations_cache_rate:\n  comment: \"Proportion of users where getTweetRecommendations() request and responses will be cached\"\n  default_availability: 1000\n\nenable_earlybird_traffic:\n  comment: \"Enable fetching tweet candidates from Earlybird\"\n  default_availability: 0\n\nenable_scribe_for_blue_verified_tweet_candidates:\n  comment: \"Enable scribing for tweet candidates from Blue Verified users\"\n  default_availability: 0\n"
  },
  {
    "path": "cr-mixer/server/src/main/resources/logback.xml",
    "content": "<configuration>\n  <shutdownHook class=\"ch.qos.logback.core.hook.DelayingShutdownHook\"/>\n\n  <!-- ===================================================== -->\n  <!-- Service Config -->\n  <!-- ===================================================== -->\n  <property name=\"DEFAULT_SERVICE_PATTERN\"\n            value=\"%-16X{traceId} %-12X{clientId:--} %-16X{method} %-25logger{0} %msg\"/>\n\n  <property name=\"DEFAULT_ACCESS_PATTERN\"\n            value=\"%msg\"/>\n\n  <!-- ===================================================== -->\n  <!-- Common Config -->\n  <!-- ===================================================== -->\n\n  <!-- JUL/JDK14 to Logback bridge -->\n  <contextListener class=\"ch.qos.logback.classic.jul.LevelChangePropagator\">\n    <resetJUL>true</resetJUL>\n  </contextListener>\n\n  <!-- ====================================================================================== -->\n  <!-- NOTE: The following appenders use a simple TimeBasedRollingPolicy configuration.       -->\n  <!--       You may want to consider using a more advanced SizeAndTimeBasedRollingPolicy.    -->\n  <!--       See: https://logback.qos.ch/manual/appenders.html#SizeAndTimeBasedRollingPolicy  -->\n  <!-- ====================================================================================== -->\n\n  <!-- Service Log (rollover daily, keep maximum of 21 days of gzip compressed logs) -->\n  <appender name=\"SERVICE\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n    <file>${log.service.output}</file>\n    <rollingPolicy class=\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\">\n      <!-- daily rollover -->\n      <fileNamePattern>${log.service.output}.%d.gz</fileNamePattern>\n      <!-- keep 21 days' worth of history -->\n      <maxHistory>21</maxHistory>\n      <cleanHistoryOnStart>true</cleanHistoryOnStart>\n    </rollingPolicy>\n    <encoder>\n      <pattern>%date %.-3level ${DEFAULT_SERVICE_PATTERN}%n</pattern>\n    </encoder>\n  </appender>\n\n  <!-- Access Log (rollover daily, keep maximum of 21 days of gzip compressed logs) -->\n  <appender name=\"ACCESS\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n    <file>${log.access.output}</file>\n    <rollingPolicy class=\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\">\n      <!-- daily rollover -->\n      <fileNamePattern>${log.access.output}.%d.gz</fileNamePattern>\n      <!-- keep 21 days' worth of history -->\n      <maxHistory>21</maxHistory>\n      <cleanHistoryOnStart>true</cleanHistoryOnStart>\n    </rollingPolicy>\n    <encoder>\n      <pattern>${DEFAULT_ACCESS_PATTERN}%n</pattern>\n    </encoder>\n  </appender>\n\n  <!--LogLens -->\n  <appender name=\"LOGLENS\" class=\"com.twitter.loglens.logback.LoglensAppender\">\n    <mdcAdditionalContext>true</mdcAdditionalContext>\n    <category>${log.lens.category}</category>\n    <index>${log.lens.index}</index>\n    <tag>${log.lens.tag}/service</tag>\n    <encoder>\n      <pattern>%msg</pattern>\n    </encoder>\n  </appender>\n\n  <!-- LogLens Access -->\n  <appender name=\"LOGLENS-ACCESS\" class=\"com.twitter.loglens.logback.LoglensAppender\">\n    <mdcAdditionalContext>true</mdcAdditionalContext>\n    <category>${log.lens.category}</category>\n    <index>${log.lens.index}</index>\n    <tag>${log.lens.tag}/access</tag>\n    <encoder>\n      <pattern>%msg</pattern>\n    </encoder>\n  </appender>\n\n  <!-- Pipeline Execution Logs -->\n  <appender name=\"ALLOW-LISTED-PIPELINE-EXECUTIONS\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n    <file>allow_listed_pipeline_executions.log</file>\n    <rollingPolicy class=\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\">\n      <!-- daily rollover -->\n      <fileNamePattern>allow_listed_pipeline_executions.log.%d.gz</fileNamePattern>\n      <!-- keep 7 days' worth of history -->\n      <maxHistory>7</maxHistory>\n      <cleanHistoryOnStart>true</cleanHistoryOnStart>\n    </rollingPolicy>\n    <encoder>\n      <pattern>%date %.-3level ${DEFAULT_SERVICE_PATTERN}%n</pattern>\n    </encoder>\n  </appender>\n\n  <!-- ===================================================== -->\n  <!-- Primary Async Appenders -->\n  <!-- ===================================================== -->\n\n  <property name=\"async_queue_size\" value=\"${queue.size:-50000}\"/>\n  <property name=\"async_max_flush_time\" value=\"${max.flush.time:-0}\"/>\n\n  <appender name=\"ASYNC-SERVICE\" class=\"com.twitter.inject.logback.AsyncAppender\">\n    <queueSize>${async_queue_size}</queueSize>\n    <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n    <appender-ref ref=\"SERVICE\"/>\n  </appender>\n\n  <appender name=\"ASYNC-ACCESS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n    <queueSize>${async_queue_size}</queueSize>\n    <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n    <appender-ref ref=\"ACCESS\"/>\n  </appender>\n\n  <appender name=\"ASYNC-ALLOW-LISTED-PIPELINE-EXECUTIONS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n    <queueSize>${async_queue_size}</queueSize>\n    <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n    <appender-ref ref=\"ALLOW-LISTED-PIPELINE-EXECUTIONS\"/>\n  </appender>\n\n  <appender name=\"ASYNC-LOGLENS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n    <queueSize>${async_queue_size}</queueSize>\n    <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n    <appender-ref ref=\"LOGLENS\"/>\n  </appender>\n\n  <appender name=\"ASYNC-LOGLENS-ACCESS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n    <queueSize>${async_queue_size}</queueSize>\n    <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n    <appender-ref ref=\"LOGLENS-ACCESS\"/>\n  </appender>\n\n  <!-- ===================================================== -->\n  <!-- Package Config -->\n  <!-- ===================================================== -->\n\n  <!-- Per-Package Config -->\n  <logger name=\"com.twitter\" level=\"info\"/>\n  <logger name=\"com.twitter.wilyns\" level=\"warn\"/>\n  <logger name=\"com.twitter.configbus.client.file\" level=\"off\"/>  \n  <logger name=\"com.twitter.finagle.mux\" level=\"warn\"/>\n  <logger name=\"com.twitter.finagle.serverset2\" level=\"warn\"/>\n  <logger name=\"com.twitter.logging.ScribeHandler\" level=\"off\"/>\n  <logger name=\"com.twitter.zookeeper.client.internal\" level=\"warn\"/>\n  <logger name=\"io.netty.handler.ssl.SslHandler\" level=\"OFF\"/>\n\n\n  <!-- Root Config -->\n  <root level=\"${log_level:-INFO}\">\n    <appender-ref ref=\"ASYNC-SERVICE\"/>\n    <appender-ref ref=\"ASYNC-LOGLENS\"/>\n  </root>\n\n  <!-- Access Logging -->\n  <logger name=\"com.twitter.finatra.thrift.filters.AccessLoggingFilter\"\n          level=\"info\"\n          additivity=\"false\">\n    <appender-ref ref=\"ASYNC-ACCESS\"/>\n    <appender-ref ref=\"ASYNC-LOGLENS-ACCESS\"/>\n  </logger>\n\n  <!-- Pipeline Executions Log -->\n  <logger name=\"com.twitter.product_mixer.core.service.pipeline_execution_logger\"\n          level=\"info\"\n          additivity=\"false\">\n    <appender-ref ref=\"ASYNC-ALLOW-LISTED-PIPELINE-EXECUTIONS\" />\n  </logger>\n\n</configuration>\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"cr-mixer/server/src/main/resources\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/controller\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/featureswitch\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/logging\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/scribe\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"decider/src/main/scala\",\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"finatra-internal/mtls-http/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/http-core/src/main/java/com/twitter/finatra/http\",\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-server/src/main/scala\",\n        \"finatra/inject/inject-utils/src/main/scala\",\n        \"finatra/utils/src/main/java/com/twitter/finatra/annotations\",\n        \"hydra/common/libraries/src/main/scala/com/twitter/hydra/common/model_config\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/filters\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n        \"thrift-web-forms/src/main/scala/com/twitter/thriftwebforms\",\n        \"thrift-web-forms/src/main/scala/com/twitter/thriftwebforms/view\",\n        \"timelines/src/main/scala/com/twitter/timelines/features/app\",\n        \"twitter-server-internal\",\n        \"twitter-server/server/src/main/scala\",\n        \"util/util-app/src/main/scala\",\n        \"util/util-core:scala\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/CrMixerHttpServerWarmupHandler.scala",
    "content": "package com.twitter.cr_mixer\n\nimport com.twitter.finatra.http.routing.HttpWarmup\nimport com.twitter.finatra.httpclient.RequestBuilder._\nimport com.twitter.inject.Logging\nimport com.twitter.inject.utils.Handler\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CrMixerHttpServerWarmupHandler @Inject() (warmup: HttpWarmup) extends Handler with Logging {\n\n  override def handle(): Unit = {\n    Try(warmup.send(get(\"/admin/cr-mixer/product-pipelines\"), admin = true)())\n      .onFailure(e => error(e.getMessage, e))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/CrMixerServer.scala",
    "content": "package com.twitter.cr_mixer\n\nimport com.google.inject.Module\nimport com.twitter.cr_mixer.controller.CrMixerThriftController\nimport com.twitter.cr_mixer.featureswitch.SetImpressedBucketsLocalContextFilter\nimport com.twitter.cr_mixer.module.ActivePromotedTweetStoreModule\nimport com.twitter.cr_mixer.module.CertoStratoStoreModule\nimport com.twitter.cr_mixer.module.CrMixerParamConfigModule\nimport com.twitter.cr_mixer.module.EmbeddingStoreModule\nimport com.twitter.cr_mixer.module.FrsStoreModule\nimport com.twitter.cr_mixer.module.MHMtlsParamsModule\nimport com.twitter.cr_mixer.module.OfflineCandidateStoreModule\nimport com.twitter.cr_mixer.module.RealGraphStoreMhModule\nimport com.twitter.cr_mixer.module.RealGraphOonStoreModule\nimport com.twitter.cr_mixer.module.RepresentationManagerModule\nimport com.twitter.cr_mixer.module.RepresentationScorerModule\nimport com.twitter.cr_mixer.module.TweetInfoStoreModule\nimport com.twitter.cr_mixer.module.TweetRecentEngagedUserStoreModule\nimport com.twitter.cr_mixer.module.TweetRecommendationResultsStoreModule\nimport com.twitter.cr_mixer.module.TripCandidateStoreModule\nimport com.twitter.cr_mixer.module.TwhinCollabFilterStratoStoreModule\nimport com.twitter.cr_mixer.module.UserSignalServiceColumnModule\nimport com.twitter.cr_mixer.module.UserSignalServiceStoreModule\nimport com.twitter.cr_mixer.module.UserStateStoreModule\nimport com.twitter.cr_mixer.module.core.ABDeciderModule\nimport com.twitter.cr_mixer.module.core.CrMixerFlagModule\nimport com.twitter.cr_mixer.module.core.CrMixerLoggingABDeciderModule\nimport com.twitter.cr_mixer.module.core.FeatureContextBuilderModule\nimport com.twitter.cr_mixer.module.core.FeatureSwitchesModule\nimport com.twitter.cr_mixer.module.core.KafkaProducerModule\nimport com.twitter.cr_mixer.module.core.LoggerFactoryModule\nimport com.twitter.cr_mixer.module.similarity_engine.ConsumerEmbeddingBasedTripSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.ConsumerEmbeddingBasedTwHINSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.ConsumerEmbeddingBasedTwoTowerSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.ConsumersBasedUserAdGraphSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.ConsumersBasedUserVideoGraphSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.ProducerBasedUserAdGraphSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.ProducerBasedUserTweetGraphSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.ProducerBasedUnifiedSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.SimClustersANNSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.TweetBasedUnifiedSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.TweetBasedQigSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.TweetBasedTwHINSimlarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.TweetBasedUserAdGraphSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.TweetBasedUserTweetGraphSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.TweetBasedUserVideoGraphSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.TwhinCollabFilterLookupSimilarityEngineModule\nimport com.twitter.cr_mixer.module.ConsumersBasedUserAdGraphStoreModule\nimport com.twitter.cr_mixer.module.ConsumersBasedUserTweetGraphStoreModule\nimport com.twitter.cr_mixer.module.ConsumersBasedUserVideoGraphStoreModule\nimport com.twitter.cr_mixer.module.DiffusionStoreModule\nimport com.twitter.cr_mixer.module.EarlybirdRecencyBasedCandidateStoreModule\nimport com.twitter.cr_mixer.module.TwiceClustersMembersStoreModule\nimport com.twitter.cr_mixer.module.StrongTiePredictionStoreModule\nimport com.twitter.cr_mixer.module.thrift_client.AnnQueryServiceClientModule\nimport com.twitter.cr_mixer.module.thrift_client.EarlybirdSearchClientModule\nimport com.twitter.cr_mixer.module.thrift_client.FrsClientModule\nimport com.twitter.cr_mixer.module.thrift_client.QigServiceClientModule\nimport com.twitter.cr_mixer.module.thrift_client.SimClustersAnnServiceClientModule\nimport com.twitter.cr_mixer.module.thrift_client.TweetyPieClientModule\nimport com.twitter.cr_mixer.module.thrift_client.UserTweetGraphClientModule\nimport com.twitter.cr_mixer.module.thrift_client.UserTweetGraphPlusClientModule\nimport com.twitter.cr_mixer.module.thrift_client.UserVideoGraphClientModule\nimport com.twitter.cr_mixer.{thriftscala => st}\nimport com.twitter.finagle.Filter\nimport com.twitter.finatra.annotations.DarkTrafficFilterType\nimport com.twitter.finatra.decider.modules.DeciderModule\nimport com.twitter.finatra.http.HttpServer\nimport com.twitter.finatra.http.routing.HttpRouter\nimport com.twitter.finatra.jackson.modules.ScalaObjectMapperModule\nimport com.twitter.finatra.mtls.http.{Mtls => HttpMtls}\nimport com.twitter.finatra.mtls.thriftmux.Mtls\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsThriftWebFormsModule\nimport com.twitter.finatra.thrift.ThriftServer\nimport com.twitter.finatra.thrift.filters._\nimport com.twitter.finatra.thrift.routing.ThriftRouter\nimport com.twitter.hydra.common.model_config.{ConfigModule => HydraConfigModule}\nimport com.twitter.inject.thrift.modules.ThriftClientIdModule\nimport com.twitter.product_mixer.core.module.LoggingThrowableExceptionMapper\nimport com.twitter.product_mixer.core.module.StratoClientModule\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule\nimport com.twitter.relevance_platform.common.filters.ClientStatsFilter\nimport com.twitter.relevance_platform.common.filters.DarkTrafficFilterModule\nimport com.twitter.cr_mixer.module.SimClustersANNServiceNameToClientMapper\nimport com.twitter.cr_mixer.module.SkitStratoStoreModule\nimport com.twitter.cr_mixer.module.BlueVerifiedAnnotationStoreModule\nimport com.twitter.cr_mixer.module.core.TimeoutConfigModule\nimport com.twitter.cr_mixer.module.grpc_client.NaviGRPCClientModule\nimport com.twitter.cr_mixer.module.similarity_engine.CertoTopicTweetSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.ConsumerBasedWalsSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.DiffusionBasedSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.EarlybirdSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.SkitTopicTweetSimilarityEngineModule\nimport com.twitter.cr_mixer.module.similarity_engine.UserTweetEntityGraphSimilarityEngineModule\nimport com.twitter.cr_mixer.module.thrift_client.HydraPartitionClientModule\nimport com.twitter.cr_mixer.module.thrift_client.HydraRootClientModule\nimport com.twitter.cr_mixer.module.thrift_client.UserAdGraphClientModule\nimport com.twitter.cr_mixer.module.thrift_client.UserTweetEntityGraphClientModule\nimport com.twitter.thriftwebforms.MethodOptions\n\nobject CrMixerServerMain extends CrMixerServer\n\nclass CrMixerServer extends ThriftServer with Mtls with HttpServer with HttpMtls {\n  override val name = \"cr-mixer-server\"\n\n  private val coreModules = Seq(\n    ABDeciderModule,\n    CrMixerFlagModule,\n    CrMixerLoggingABDeciderModule,\n    CrMixerParamConfigModule,\n    new DarkTrafficFilterModule[st.CrMixer.ReqRepServicePerEndpoint](),\n    DeciderModule,\n    FeatureContextBuilderModule,\n    FeatureSwitchesModule,\n    KafkaProducerModule,\n    LoggerFactoryModule,\n    MHMtlsParamsModule,\n    ProductMixerFlagModule,\n    ScalaObjectMapperModule,\n    ThriftClientIdModule\n  )\n\n  private val thriftClientModules = Seq(\n    AnnQueryServiceClientModule,\n    EarlybirdSearchClientModule,\n    FrsClientModule,\n    HydraPartitionClientModule,\n    HydraRootClientModule,\n    QigServiceClientModule,\n    SimClustersAnnServiceClientModule,\n    TweetyPieClientModule,\n    UserAdGraphClientModule,\n    UserTweetEntityGraphClientModule,\n    UserTweetGraphClientModule,\n    UserTweetGraphPlusClientModule,\n    UserVideoGraphClientModule,\n  )\n\n  private val grpcClientModules = Seq(\n    NaviGRPCClientModule\n  )\n\n  // Modules sorted alphabetically, please keep the order when adding a new module\n  override val modules: Seq[Module] =\n    coreModules ++ thriftClientModules ++ grpcClientModules ++\n      Seq(\n        ActivePromotedTweetStoreModule,\n        CertoStratoStoreModule,\n        CertoTopicTweetSimilarityEngineModule,\n        ConsumersBasedUserAdGraphSimilarityEngineModule,\n        ConsumersBasedUserTweetGraphStoreModule,\n        ConsumersBasedUserVideoGraphSimilarityEngineModule,\n        ConsumersBasedUserVideoGraphStoreModule,\n        ConsumerEmbeddingBasedTripSimilarityEngineModule,\n        ConsumerEmbeddingBasedTwHINSimilarityEngineModule,\n        ConsumerEmbeddingBasedTwoTowerSimilarityEngineModule,\n        ConsumersBasedUserAdGraphStoreModule,\n        ConsumerBasedWalsSimilarityEngineModule,\n        DiffusionStoreModule,\n        EmbeddingStoreModule,\n        EarlybirdSimilarityEngineModule,\n        EarlybirdRecencyBasedCandidateStoreModule,\n        FrsStoreModule,\n        HydraConfigModule,\n        OfflineCandidateStoreModule,\n        ProducerBasedUnifiedSimilarityEngineModule,\n        ProducerBasedUserAdGraphSimilarityEngineModule,\n        ProducerBasedUserTweetGraphSimilarityEngineModule,\n        RealGraphOonStoreModule,\n        RealGraphStoreMhModule,\n        RepresentationManagerModule,\n        RepresentationScorerModule,\n        SimClustersANNServiceNameToClientMapper,\n        SimClustersANNSimilarityEngineModule,\n        SkitStratoStoreModule,\n        SkitTopicTweetSimilarityEngineModule,\n        StratoClientModule,\n        StrongTiePredictionStoreModule,\n        TimeoutConfigModule,\n        TripCandidateStoreModule,\n        TwiceClustersMembersStoreModule,\n        TweetBasedQigSimilarityEngineModule,\n        TweetBasedTwHINSimlarityEngineModule,\n        TweetBasedUnifiedSimilarityEngineModule,\n        TweetBasedUserAdGraphSimilarityEngineModule,\n        TweetBasedUserTweetGraphSimilarityEngineModule,\n        TweetBasedUserVideoGraphSimilarityEngineModule,\n        TweetInfoStoreModule,\n        TweetRecentEngagedUserStoreModule,\n        TweetRecommendationResultsStoreModule,\n        TwhinCollabFilterStratoStoreModule,\n        TwhinCollabFilterLookupSimilarityEngineModule,\n        UserSignalServiceColumnModule,\n        UserSignalServiceStoreModule,\n        UserStateStoreModule,\n        UserTweetEntityGraphSimilarityEngineModule,\n        DiffusionBasedSimilarityEngineModule,\n        BlueVerifiedAnnotationStoreModule,\n        new MtlsThriftWebFormsModule[st.CrMixer.MethodPerEndpoint](this) {\n          override protected def defaultMethodAccess: MethodOptions.Access = {\n            MethodOptions.Access.ByLdapGroup(\n              Seq(\n                \"cr-mixer-admins\",\n                \"recosplat-sensitive-data-medium\",\n                \"recos-platform-admins\",\n              ))\n          }\n        }\n      )\n\n  def configureThrift(router: ThriftRouter): Unit = {\n    router\n      .filter[LoggingMDCFilter]\n      .filter[TraceIdMDCFilter]\n      .filter[ThriftMDCFilter]\n      .filter[ClientStatsFilter]\n      .filter[AccessLoggingFilter]\n      .filter[SetImpressedBucketsLocalContextFilter]\n      .filter[ExceptionMappingFilter]\n      .filter[Filter.TypeAgnostic, DarkTrafficFilterType]\n      .exceptionMapper[LoggingThrowableExceptionMapper]\n      .add[CrMixerThriftController]\n  }\n\n  override protected def warmup(): Unit = {\n    handle[CrMixerThriftServerWarmupHandler]()\n    handle[CrMixerHttpServerWarmupHandler]()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/CrMixerThriftServerWarmupHandler.scala",
    "content": "package com.twitter.cr_mixer\n\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finatra.thrift.routing.ThriftWarmup\nimport com.twitter.inject.Logging\nimport com.twitter.inject.utils.Handler\nimport com.twitter.product_mixer.core.{thriftscala => pt}\nimport com.twitter.cr_mixer.{thriftscala => st}\nimport com.twitter.scrooge.Request\nimport com.twitter.scrooge.Response\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CrMixerThriftServerWarmupHandler @Inject() (warmup: ThriftWarmup)\n    extends Handler\n    with Logging {\n\n  private val clientId = ClientId(\"thrift-warmup-client\")\n\n  def handle(): Unit = {\n    val testIds = Seq(1, 2, 3)\n    try {\n      clientId.asCurrent {\n        testIds.foreach { id =>\n          val warmupReq = warmupQuery(id)\n          info(s\"Sending warm-up request to service with query: $warmupReq\")\n          warmup.sendRequest(\n            method = st.CrMixer.GetTweetRecommendations,\n            req = Request(st.CrMixer.GetTweetRecommendations.Args(warmupReq)))(assertWarmupResponse)\n        }\n      }\n    } catch {\n      case e: Throwable =>\n        // we don't want a warmup failure to prevent start-up\n        error(e.getMessage, e)\n    }\n    info(\"Warm-up done.\")\n  }\n\n  private def warmupQuery(userId: Long): st.CrMixerTweetRequest = {\n    val clientContext = pt.ClientContext(\n      userId = Some(userId),\n      guestId = None,\n      appId = Some(258901L),\n      ipAddress = Some(\"0.0.0.0\"),\n      userAgent = Some(\"FAKE_USER_AGENT_FOR_WARMUPS\"),\n      countryCode = Some(\"US\"),\n      languageCode = Some(\"en\"),\n      isTwoffice = None,\n      userRoles = None,\n      deviceId = Some(\"FAKE_DEVICE_ID_FOR_WARMUPS\")\n    )\n    st.CrMixerTweetRequest(\n      clientContext = clientContext,\n      product = st.Product.Home,\n      productContext = Some(st.ProductContext.HomeContext(st.HomeContext())),\n    )\n  }\n\n  private def assertWarmupResponse(\n    result: Try[Response[st.CrMixer.GetTweetRecommendations.SuccessType]]\n  ): Unit = {\n    // we collect and log any exceptions from the result.\n    result match {\n      case Return(_) => // ok\n      case Throw(exception) =>\n        warn(\"Error performing warm-up request.\")\n        error(exception.getMessage, exception)\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/blender/AdsBlender.scala",
    "content": "package com.twitter.cr_mixer.blender\n\nimport com.twitter.cr_mixer.model.BlendedAdsCandidate\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.InitialAdsCandidate\nimport com.twitter.cr_mixer.util.InterleaveUtil\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.mutable\n\n@Singleton\ncase class AdsBlender @Inject() (globalStats: StatsReceiver) {\n\n  private val name: String = this.getClass.getCanonicalName\n  private val stats: StatsReceiver = globalStats.scope(name)\n\n  /**\n   * Interleaves candidates by iteratively choosing InterestedIn candidates and TWISTLY candidates\n   * in turn. InterestedIn candidates have no source signal, whereas TWISTLY candidates do. TWISTLY\n   * candidates themselves are interleaved by source before equal blending with InterestedIn\n   * candidates.\n   */\n  def blend(\n    inputCandidates: Seq[Seq[InitialAdsCandidate]],\n  ): Future[Seq[BlendedAdsCandidate]] = {\n\n    // Filter out empty candidate sequence\n    val candidates = inputCandidates.filter(_.nonEmpty)\n    val (interestedInCandidates, twistlyCandidates) =\n      candidates.partition(_.head.candidateGenerationInfo.sourceInfoOpt.isEmpty)\n    // First interleave twistly candidates\n    val interleavedTwistlyCandidates = InterleaveUtil.interleave(twistlyCandidates)\n\n    val twistlyAndInterestedInCandidates =\n      Seq(interestedInCandidates.flatten, interleavedTwistlyCandidates)\n\n    // then interleave  twistly candidates with interested in to make them even\n    val interleavedCandidates = InterleaveUtil.interleave(twistlyAndInterestedInCandidates)\n\n    stats.stat(\"candidates\").add(interleavedCandidates.size)\n\n    val blendedCandidates = buildBlendedAdsCandidate(inputCandidates, interleavedCandidates)\n    Future.value(blendedCandidates)\n  }\n  private def buildBlendedAdsCandidate(\n    inputCandidates: Seq[Seq[InitialAdsCandidate]],\n    interleavedCandidates: Seq[InitialAdsCandidate]\n  ): Seq[BlendedAdsCandidate] = {\n    val cgInfoLookupMap = buildCandidateToCGInfosMap(inputCandidates)\n    interleavedCandidates.map { interleavedCandidate =>\n      interleavedCandidate.toBlendedAdsCandidate(cgInfoLookupMap(interleavedCandidate.tweetId))\n    }\n  }\n\n  private def buildCandidateToCGInfosMap(\n    candidateSeq: Seq[Seq[InitialAdsCandidate]],\n  ): Map[TweetId, Seq[CandidateGenerationInfo]] = {\n    val tweetIdMap = mutable.HashMap[TweetId, Seq[CandidateGenerationInfo]]()\n\n    candidateSeq.foreach { candidates =>\n      candidates.foreach { candidate =>\n        val candidateGenerationInfoSeq = {\n          tweetIdMap.getOrElse(candidate.tweetId, Seq.empty)\n        }\n        val candidateGenerationInfo = candidate.candidateGenerationInfo\n        tweetIdMap.put(\n          candidate.tweetId,\n          candidateGenerationInfoSeq ++ Seq(candidateGenerationInfo))\n      }\n    }\n    tweetIdMap.toMap\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/blender/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"configapi/configapi-core\",\n        \"content-recommender/thrift/src/main/thrift:thrift-scala\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/core_workflows/user_model:user_model-scala\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/blender/BlendedCandidatesBuilder.scala",
    "content": "package com.twitter.cr_mixer.blender\n\nimport com.twitter.cr_mixer.model.BlendedCandidate\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.simclusters_v2.common.TweetId\nimport scala.collection.mutable\n\nobject BlendedCandidatesBuilder {\n\n  /**\n   * @param inputCandidates input candidate prior to interleaving\n   * @param interleavedCandidates after interleaving. These tweets are de-duplicated.\n   */\n  def build(\n    inputCandidates: Seq[Seq[InitialCandidate]],\n    interleavedCandidates: Seq[InitialCandidate]\n  ): Seq[BlendedCandidate] = {\n    val cgInfoLookupMap = buildCandidateToCGInfosMap(inputCandidates)\n    interleavedCandidates.map { interleavedCandidate =>\n      interleavedCandidate.toBlendedCandidate(cgInfoLookupMap(interleavedCandidate.tweetId))\n    }\n  }\n\n  /**\n   * The same tweet can be generated by different sources.\n   * This function tells you which CandidateGenerationInfo generated a given tweet\n   */\n  private def buildCandidateToCGInfosMap(\n    candidateSeq: Seq[Seq[InitialCandidate]],\n  ): Map[TweetId, Seq[CandidateGenerationInfo]] = {\n    val tweetIdMap = mutable.HashMap[TweetId, Seq[CandidateGenerationInfo]]()\n\n    candidateSeq.foreach { candidates =>\n      candidates.foreach { candidate =>\n        val candidateGenerationInfoSeq = {\n          tweetIdMap.getOrElse(candidate.tweetId, Seq.empty)\n        }\n        val candidateGenerationInfo = candidate.candidateGenerationInfo\n        tweetIdMap.put(\n          candidate.tweetId,\n          candidateGenerationInfoSeq ++ Seq(candidateGenerationInfo))\n      }\n    }\n    tweetIdMap.toMap\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/blender/ContentSignalBlender.scala",
    "content": "package com.twitter.cr_mixer.blender\n\nimport com.twitter.cr_mixer.model.BlendedCandidate\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.param.BlenderParams\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport javax.inject.Inject\n\ncase class ContentSignalBlender @Inject() (globalStats: StatsReceiver) {\n\n  private val name: String = this.getClass.getCanonicalName\n  private val stats: StatsReceiver = globalStats.scope(name)\n\n  /**\n   *  Exposes multiple types of sorting relying only on Content Based signals\n   *  Candidate Recency, Random, FavoriteCount and finally Standardized, which standardizes the scores\n   *  that come from the active SimilarityEngine and then sort on the standardized scores.\n   */\n  def blend(\n    params: Params,\n    inputCandidates: Seq[Seq[InitialCandidate]],\n  ): Future[Seq[BlendedCandidate]] = {\n    // Filter out empty candidate sequence\n    val candidates = inputCandidates.filter(_.nonEmpty)\n    val sortedCandidates = params(BlenderParams.ContentBlenderTypeSortingAlgorithmParam) match {\n      case BlenderParams.ContentBasedSortingAlgorithmEnum.CandidateRecency =>\n        candidates.flatten.sortBy(c => getSnowflakeTimeStamp(c.tweetId)).reverse\n      case BlenderParams.ContentBasedSortingAlgorithmEnum.RandomSorting =>\n        candidates.flatten.sortBy(_ => scala.util.Random.nextDouble())\n      case BlenderParams.ContentBasedSortingAlgorithmEnum.FavoriteCount =>\n        candidates.flatten.sortBy(-_.tweetInfo.favCount)\n      case BlenderParams.ContentBasedSortingAlgorithmEnum.SimilarityToSignalSorting =>\n        standardizeAndSortByScore(flattenAndGroupByEngineTypeOrFirstContribEngine(candidates))\n      case _ =>\n        candidates.flatten.sortBy(-_.tweetInfo.favCount)\n    }\n\n    stats.stat(\"candidates\").add(sortedCandidates.size)\n\n    val blendedCandidates =\n      BlendedCandidatesBuilder.build(inputCandidates, removeDuplicates(sortedCandidates))\n    Future.value(blendedCandidates)\n  }\n\n  private def removeDuplicates(candidates: Seq[InitialCandidate]): Seq[InitialCandidate] = {\n    val seen = collection.mutable.Set.empty[Long]\n    candidates.filter { c =>\n      if (seen.contains(c.tweetId)) {\n        false\n      } else {\n        seen += c.tweetId\n        true\n      }\n    }\n  }\n\n  private def groupByEngineTypeOrFirstContribEngine(\n    candidates: Seq[InitialCandidate]\n  ): Map[SimilarityEngineType, Seq[InitialCandidate]] = {\n    val grouped = candidates.groupBy { candidate =>\n      val contrib = candidate.candidateGenerationInfo.contributingSimilarityEngines\n      if (contrib.nonEmpty) {\n        contrib.head.similarityEngineType\n      } else {\n        candidate.candidateGenerationInfo.similarityEngineInfo.similarityEngineType\n      }\n    }\n    grouped\n  }\n\n  private def flattenAndGroupByEngineTypeOrFirstContribEngine(\n    candidates: Seq[Seq[InitialCandidate]]\n  ): Seq[Seq[InitialCandidate]] = {\n    val flat = candidates.flatten\n    val grouped = groupByEngineTypeOrFirstContribEngine(flat)\n    grouped.values.toSeq\n  }\n\n  private def standardizeAndSortByScore(\n    candidates: Seq[Seq[InitialCandidate]]\n  ): Seq[InitialCandidate] = {\n    candidates\n      .map { innerSeq =>\n        val meanScore = innerSeq\n          .map(c => c.candidateGenerationInfo.similarityEngineInfo.score.getOrElse(0.0))\n          .sum / innerSeq.length\n        val stdDev = scala.math\n          .sqrt(\n            innerSeq\n              .map(c => c.candidateGenerationInfo.similarityEngineInfo.score.getOrElse(0.0))\n              .map(a => a - meanScore)\n              .map(a => a * a)\n              .sum / innerSeq.length)\n        innerSeq\n          .map(c =>\n            (\n              c,\n              c.candidateGenerationInfo.similarityEngineInfo.score\n                .map { score =>\n                  if (stdDev != 0) (score - meanScore) / stdDev\n                  else 0.0\n                }\n                .getOrElse(0.0)))\n      }.flatten.sortBy { case (_, standardizedScore) => -standardizedScore }\n      .map { case (candidate, _) => candidate }\n  }\n\n  private def getSnowflakeTimeStamp(tweetId: Long): Time = {\n    val isSnowflake = SnowflakeId.isSnowflakeId(tweetId)\n    if (isSnowflake) {\n      SnowflakeId(tweetId).time\n    } else {\n      Time.fromMilliseconds(0L)\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/blender/CountWeightedInterleaveBlender.scala",
    "content": "package com.twitter.cr_mixer.blender\n\nimport com.twitter.cr_mixer.model.BlendedCandidate\nimport com.twitter.cr_mixer.model.CrCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.param.BlenderParams\nimport com.twitter.cr_mixer.util.CountWeightedInterleaveUtil\nimport com.twitter.cr_mixer.util.InterleaveUtil\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * A weighted round robin interleaving algorithm.\n * The weight of each blending group based on the count of candidates in each blending group.\n * The more candidates under a blending group, the more candidates are selected from it during round\n * robin, which in effect prioritizes this group.\n *\n * Weights sum up to 1. For example:\n * total candidates = 8\n *             Group                       Weight\n *         [A1, A2, A3, A4]          4/8 = 0.5  // select 50% of results from group A\n *         [B1, B2]                  2/8 = 0.25 // 25% from group B\n *         [C1, C2]                  2/8 = 0.25 // 25% from group C\n *\n * Blended results = [A1, A2, B1, C1, A3, A4, B2, C2]\n * See @linht's go/weighted-interleave\n */\n@Singleton\ncase class CountWeightedInterleaveBlender @Inject() (globalStats: StatsReceiver) {\n  import CountWeightedInterleaveBlender._\n\n  private val name: String = this.getClass.getCanonicalName\n  private val stats: StatsReceiver = globalStats.scope(name)\n\n  def blend(\n    query: CrCandidateGeneratorQuery,\n    inputCandidates: Seq[Seq[InitialCandidate]]\n  ): Future[Seq[BlendedCandidate]] = {\n    val weightedBlenderQuery = CountWeightedInterleaveBlender.paramToQuery(query.params)\n    countWeightedInterleave(weightedBlenderQuery, inputCandidates)\n  }\n\n  private[blender] def countWeightedInterleave(\n    query: WeightedBlenderQuery,\n    inputCandidates: Seq[Seq[InitialCandidate]],\n  ): Future[Seq[BlendedCandidate]] = {\n\n    val candidatesAndWeightKeyByIndexId: Seq[(Seq[InitialCandidate], Double)] = {\n      CountWeightedInterleaveUtil.buildInitialCandidatesWithWeightKeyByFeature(\n        inputCandidates,\n        query.rankerWeightShrinkage)\n    }\n\n    val interleavedCandidates =\n      InterleaveUtil.weightedInterleave(candidatesAndWeightKeyByIndexId, query.maxWeightAdjustments)\n\n    stats.stat(\"candidates\").add(interleavedCandidates.size)\n\n    val blendedCandidates = BlendedCandidatesBuilder.build(inputCandidates, interleavedCandidates)\n    Future.value(blendedCandidates)\n  }\n}\n\nobject CountWeightedInterleaveBlender {\n\n  /**\n   * We pass two parameters to the weighted interleaver:\n   * @param rankerWeightShrinkage shrinkage parameter between [0, 1] that determines how close we\n   *                              stay to uniform sampling. The bigger the shrinkage the\n   *                              closer we are to uniform round robin\n   * @param maxWeightAdjustments max number of weighted sampling to do prior to defaulting to\n   *                             uniform. Set so that we avoid infinite loops (e.g. if weights are\n   *                             0)\n   */\n  case class WeightedBlenderQuery(\n    rankerWeightShrinkage: Double,\n    maxWeightAdjustments: Int)\n\n  def paramToQuery(params: Params): WeightedBlenderQuery = {\n    val rankerWeightShrinkage: Double =\n      params(BlenderParams.RankingInterleaveWeightShrinkageParam)\n    val maxWeightAdjustments: Int =\n      params(BlenderParams.RankingInterleaveMaxWeightAdjustments)\n\n    WeightedBlenderQuery(rankerWeightShrinkage, maxWeightAdjustments)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/blender/InterleaveBlender.scala",
    "content": "package com.twitter.cr_mixer.blender\n\nimport com.twitter.cr_mixer.model.BlendedCandidate\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.util.InterleaveUtil\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class InterleaveBlender @Inject() (globalStats: StatsReceiver) {\n\n  private val name: String = this.getClass.getCanonicalName\n  private val stats: StatsReceiver = globalStats.scope(name)\n\n  /**\n   * Interleaves candidates, by taking 1 candidate from each Seq[Seq[InitialCandidate]] in sequence,\n   * until we run out of candidates.\n   */\n  def blend(\n    inputCandidates: Seq[Seq[InitialCandidate]],\n  ): Future[Seq[BlendedCandidate]] = {\n\n    val interleavedCandidates = InterleaveUtil.interleave(inputCandidates)\n\n    stats.stat(\"candidates\").add(interleavedCandidates.size)\n\n    val blendedCandidates = BlendedCandidatesBuilder.build(inputCandidates, interleavedCandidates)\n    Future.value(blendedCandidates)\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/blender/SourceTypeBackFillBlender.scala",
    "content": "package com.twitter.cr_mixer.blender\n\nimport com.twitter.cr_mixer.blender.ImplicitSignalBackFillBlender.BackFillSourceTypes\nimport com.twitter.cr_mixer.blender.ImplicitSignalBackFillBlender.BackFillSourceTypesWithVideo\nimport com.twitter.cr_mixer.model.BlendedCandidate\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.param.BlenderParams\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.cr_mixer.util.InterleaveUtil\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.Future\nimport javax.inject.Inject\n\ncase class SourceTypeBackFillBlender @Inject() (globalStats: StatsReceiver) {\n\n  private val name: String = this.getClass.getCanonicalName\n  private val stats: StatsReceiver = globalStats.scope(name)\n\n  /**\n   *  Partition the candidates based on source type\n   *  Interleave the two partitions of candidates separately\n   *  Then append the back fill candidates to the end\n   */\n  def blend(\n    params: Params,\n    inputCandidates: Seq[Seq[InitialCandidate]],\n  ): Future[Seq[BlendedCandidate]] = {\n\n    // Filter out empty candidate sequence\n    val candidates = inputCandidates.filter(_.nonEmpty)\n\n    val backFillSourceTypes =\n      if (params(BlenderParams.SourceTypeBackFillEnableVideoBackFill)) BackFillSourceTypesWithVideo\n      else BackFillSourceTypes\n    // partition candidates based on their source types\n    val (backFillCandidates, regularCandidates) =\n      candidates.partition(\n        _.head.candidateGenerationInfo.sourceInfoOpt\n          .exists(sourceInfo => backFillSourceTypes.contains(sourceInfo.sourceType)))\n\n    val interleavedRegularCandidates = InterleaveUtil.interleave(regularCandidates)\n    val interleavedBackFillCandidates =\n      InterleaveUtil.interleave(backFillCandidates)\n    stats.stat(\"backFillCandidates\").add(interleavedBackFillCandidates.size)\n    // Append interleaved backfill candidates to the end\n    val interleavedCandidates = interleavedRegularCandidates ++ interleavedBackFillCandidates\n\n    stats.stat(\"candidates\").add(interleavedCandidates.size)\n\n    val blendedCandidates = BlendedCandidatesBuilder.build(inputCandidates, interleavedCandidates)\n    Future.value(blendedCandidates)\n  }\n\n}\n\nobject ImplicitSignalBackFillBlender {\n  final val BackFillSourceTypesWithVideo: Set[SourceType] = Set(\n    SourceType.UserRepeatedProfileVisit,\n    SourceType.VideoTweetPlayback50,\n    SourceType.VideoTweetQualityView)\n\n  final val BackFillSourceTypes: Set[SourceType] = Set(SourceType.UserRepeatedProfileVisit)\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/blender/SwitchBlender.scala",
    "content": "package com.twitter.cr_mixer.blender\n\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.cr_mixer.model.BlendedCandidate\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.param.BlenderParams\nimport com.twitter.cr_mixer.param.BlenderParams.BlendingAlgorithmEnum\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class SwitchBlender @Inject() (\n  defaultBlender: InterleaveBlender,\n  sourceTypeBackFillBlender: SourceTypeBackFillBlender,\n  adsBlender: AdsBlender,\n  contentSignalBlender: ContentSignalBlender,\n  globalStats: StatsReceiver) {\n\n  private val stats = globalStats.scope(this.getClass.getCanonicalName)\n\n  def blend(\n    params: Params,\n    userState: UserState,\n    inputCandidates: Seq[Seq[InitialCandidate]],\n  ): Future[Seq[BlendedCandidate]] = {\n    // Take out empty seq\n    val nonEmptyCandidates = inputCandidates.collect {\n      case candidates if candidates.nonEmpty =>\n        candidates\n    }\n    stats.stat(\"num_of_sequences\").add(inputCandidates.size)\n\n    // Sort the seqs in an order\n    val innerSignalSorting = params(BlenderParams.SignalTypeSortingAlgorithmParam) match {\n      case BlenderParams.ContentBasedSortingAlgorithmEnum.SourceSignalRecency =>\n        SwitchBlender.TimestampOrder\n      case BlenderParams.ContentBasedSortingAlgorithmEnum.RandomSorting => SwitchBlender.RandomOrder\n      case _ => SwitchBlender.TimestampOrder\n    }\n\n    val candidatesToBlend = nonEmptyCandidates.sortBy(_.head)(innerSignalSorting)\n    // Blend based on specified blender rules\n    params(BlenderParams.BlendingAlgorithmParam) match {\n      case BlendingAlgorithmEnum.RoundRobin =>\n        defaultBlender.blend(candidatesToBlend)\n      case BlendingAlgorithmEnum.SourceTypeBackFill =>\n        sourceTypeBackFillBlender.blend(params, candidatesToBlend)\n      case BlendingAlgorithmEnum.SourceSignalSorting =>\n        contentSignalBlender.blend(params, candidatesToBlend)\n      case _ => defaultBlender.blend(candidatesToBlend)\n    }\n  }\n}\n\nobject SwitchBlender {\n\n  /**\n   * Prefers candidates generated from sources with the latest timestamps.\n   * The newer the source signal, the higher a candidate ranks.\n   * This ordering biases against consumer-based candidates because their timestamp defaults to 0\n   *\n   * Within a Seq[Seq[Candidate]], all candidates within a inner Seq\n   * are guaranteed to have the same sourceInfo because they are grouped by (sourceInfo, SE model).\n   * Hence, we can pick .headOption to represent the whole list when filtering by the internalId of the sourceInfoOpt.\n   * But of course the similarityEngine score in a CGInfo could be different.\n   */\n  val TimestampOrder: Ordering[InitialCandidate] =\n    math.Ordering\n      .by[InitialCandidate, Time](\n        _.candidateGenerationInfo.sourceInfoOpt\n          .flatMap(_.sourceEventTime)\n          .getOrElse(Time.fromMilliseconds(0L)))\n      .reverse\n\n  private val RandomOrder: Ordering[InitialCandidate] =\n    Ordering.by[InitialCandidate, Double](_ => scala.util.Random.nextDouble())\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation/AdsCandidateGenerator.scala",
    "content": "package com.twitter.cr_mixer.candidate_generation\n\nimport com.twitter.cr_mixer.blender.AdsBlender\nimport com.twitter.cr_mixer.logging.AdsRecommendationsScribeLogger\nimport com.twitter.cr_mixer.model.AdsCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.BlendedAdsCandidate\nimport com.twitter.cr_mixer.model.InitialAdsCandidate\nimport com.twitter.cr_mixer.model.RankedAdsCandidate\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.param.AdsParams\nimport com.twitter.cr_mixer.param.ConsumersBasedUserAdGraphParams\nimport com.twitter.cr_mixer.source_signal.RealGraphInSourceGraphFetcher\nimport com.twitter.cr_mixer.source_signal.SourceFetcher.FetcherQuery\nimport com.twitter.cr_mixer.source_signal.UssSourceSignalFetcher\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.util.Future\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AdsCandidateGenerator @Inject() (\n  ussSourceSignalFetcher: UssSourceSignalFetcher,\n  realGraphInSourceGraphFetcher: RealGraphInSourceGraphFetcher,\n  adsCandidateSourceRouter: AdsCandidateSourcesRouter,\n  adsBlender: AdsBlender,\n  scribeLogger: AdsRecommendationsScribeLogger,\n  globalStats: StatsReceiver) {\n\n  private val stats: StatsReceiver = globalStats.scope(this.getClass.getCanonicalName)\n  private val fetchSourcesStats = stats.scope(\"fetchSources\")\n  private val fetchRealGraphSeedsStats = stats.scope(\"fetchRealGraphSeeds\")\n  private val fetchCandidatesStats = stats.scope(\"fetchCandidates\")\n  private val interleaveStats = stats.scope(\"interleave\")\n  private val rankStats = stats.scope(\"rank\")\n\n  def get(query: AdsCandidateGeneratorQuery): Future[Seq[RankedAdsCandidate]] = {\n    val allStats = stats.scope(\"all\")\n    val perProductStats = stats.scope(\"perProduct\", query.product.toString)\n\n    StatsUtil.trackItemsStats(allStats) {\n      StatsUtil.trackItemsStats(perProductStats) {\n        for {\n          // fetch source signals\n          sourceSignals <- StatsUtil.trackBlockStats(fetchSourcesStats) {\n            fetchSources(query)\n          }\n          realGraphSeeds <- StatsUtil.trackItemMapStats(fetchRealGraphSeedsStats) {\n            fetchSeeds(query)\n          }\n          // get initial candidates from similarity engines\n          // hydrate lineItemInfo and filter out non active ads\n          initialCandidates <- StatsUtil.trackBlockStats(fetchCandidatesStats) {\n            fetchCandidates(query, sourceSignals, realGraphSeeds)\n          }\n\n          // blend candidates\n          blendedCandidates <- StatsUtil.trackItemsStats(interleaveStats) {\n            interleave(initialCandidates)\n          }\n\n          rankedCandidates <- StatsUtil.trackItemsStats(rankStats) {\n            rank(\n              blendedCandidates,\n              query.params(AdsParams.EnableScoreBoost),\n              query.params(AdsParams.AdsCandidateGenerationScoreBoostFactor),\n              rankStats)\n          }\n        } yield {\n          rankedCandidates.take(query.maxNumResults)\n        }\n      }\n    }\n\n  }\n\n  def fetchSources(\n    query: AdsCandidateGeneratorQuery\n  ): Future[Set[SourceInfo]] = {\n    val fetcherQuery =\n      FetcherQuery(query.userId, query.product, query.userState, query.params)\n    ussSourceSignalFetcher.get(fetcherQuery).map(_.getOrElse(Seq.empty).toSet)\n  }\n\n  private def fetchCandidates(\n    query: AdsCandidateGeneratorQuery,\n    sourceSignals: Set[SourceInfo],\n    realGraphSeeds: Map[UserId, Double]\n  ): Future[Seq[Seq[InitialAdsCandidate]]] = {\n    scribeLogger.scribeInitialAdsCandidates(\n      query,\n      adsCandidateSourceRouter\n        .fetchCandidates(query.userId, sourceSignals, realGraphSeeds, query.params),\n      query.params(AdsParams.EnableScribe)\n    )\n\n  }\n\n  private def fetchSeeds(\n    query: AdsCandidateGeneratorQuery\n  ): Future[Map[UserId, Double]] = {\n    if (query.params(ConsumersBasedUserAdGraphParams.EnableSourceParam)) {\n      realGraphInSourceGraphFetcher\n        .get(FetcherQuery(query.userId, query.product, query.userState, query.params))\n        .map(_.map(_.seedWithScores).getOrElse(Map.empty))\n    } else Future.value(Map.empty[UserId, Double])\n  }\n\n  private def interleave(\n    candidates: Seq[Seq[InitialAdsCandidate]]\n  ): Future[Seq[BlendedAdsCandidate]] = {\n    adsBlender\n      .blend(candidates)\n  }\n\n  private def rank(\n    candidates: Seq[BlendedAdsCandidate],\n    enableScoreBoost: Boolean,\n    scoreBoostFactor: Double,\n    statsReceiver: StatsReceiver,\n  ): Future[Seq[RankedAdsCandidate]] = {\n\n    val candidateSize = candidates.size\n    val rankedCandidates = candidates.zipWithIndex.map {\n      case (candidate, index) =>\n        val score = 0.5 + 0.5 * ((candidateSize - index).toDouble / candidateSize)\n        val boostedScore = if (enableScoreBoost) {\n          statsReceiver.stat(\"boostedScore\").add((100.0 * score * scoreBoostFactor).toFloat)\n          score * scoreBoostFactor\n        } else {\n          statsReceiver.stat(\"score\").add((100.0 * score).toFloat)\n          score\n        }\n        candidate.toRankedAdsCandidate(boostedScore)\n    }\n    Future.value(rankedCandidates)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation/AdsCandidateSourcesRouter.scala",
    "content": " package com.twitter.cr_mixer.candidate_generation\n\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.InitialAdsCandidate\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.model.TweetWithCandidateGenerationInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.ConsumersBasedUserAdGraphParams\nimport com.twitter.cr_mixer.param.ConsumerBasedWalsParams\nimport com.twitter.cr_mixer.param.ConsumerEmbeddingBasedCandidateGenerationParams\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.param.InterestedInParams\nimport com.twitter.cr_mixer.param.ProducerBasedCandidateGenerationParams\nimport com.twitter.cr_mixer.param.SimClustersANNParams\nimport com.twitter.cr_mixer.param.TweetBasedCandidateGenerationParams\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.ConsumerBasedWalsSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.ConsumersBasedUserAdGraphSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.FilterUtil\nimport com.twitter.cr_mixer.similarity_engine.HnswANNEngineQuery\nimport com.twitter.cr_mixer.similarity_engine.HnswANNSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.ProducerBasedUserAdGraphSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimClustersANNSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimClustersANNSimilarityEngine.Query\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TweetBasedUserAdGraphSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.LineItemInfo\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.simclusters_v2.common.ModelVersions\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.Future\n\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\ncase class AdsCandidateSourcesRouter @Inject() (\n  activePromotedTweetStore: ReadableStore[TweetId, Seq[LineItemInfo]],\n  decider: CrMixerDecider,\n  @Named(ModuleNames.SimClustersANNSimilarityEngine) simClustersANNSimilarityEngine: StandardSimilarityEngine[\n    Query,\n    TweetWithScore\n  ],\n  @Named(ModuleNames.TweetBasedUserAdGraphSimilarityEngine)\n  tweetBasedUserAdGraphSimilarityEngine: StandardSimilarityEngine[\n    TweetBasedUserAdGraphSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  @Named(ModuleNames.ConsumersBasedUserAdGraphSimilarityEngine)\n  consumersBasedUserAdGraphSimilarityEngine: StandardSimilarityEngine[\n    ConsumersBasedUserAdGraphSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  @Named(ModuleNames.ProducerBasedUserAdGraphSimilarityEngine)\n  producerBasedUserAdGraphSimilarityEngine: StandardSimilarityEngine[\n    ProducerBasedUserAdGraphSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  @Named(ModuleNames.TweetBasedTwHINANNSimilarityEngine)\n  tweetBasedTwHINANNSimilarityEngine: HnswANNSimilarityEngine,\n  @Named(ModuleNames.ConsumerEmbeddingBasedTwHINANNSimilarityEngine) consumerTwHINANNSimilarityEngine: HnswANNSimilarityEngine,\n  @Named(ModuleNames.ConsumerBasedWalsSimilarityEngine)\n  consumerBasedWalsSimilarityEngine: StandardSimilarityEngine[\n    ConsumerBasedWalsSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  globalStats: StatsReceiver,\n) {\n\n  import AdsCandidateSourcesRouter._\n\n  val stats: StatsReceiver = globalStats.scope(this.getClass.getSimpleName)\n\n  def fetchCandidates(\n    requestUserId: UserId,\n    sourceSignals: Set[SourceInfo],\n    realGraphSeeds: Map[UserId, Double],\n    params: configapi.Params\n  ): Future[Seq[Seq[InitialAdsCandidate]]] = {\n\n    val simClustersANN1ConfigId = params(SimClustersANNParams.SimClustersANN1ConfigId)\n\n    val tweetBasedSANNMinScore = params(\n      TweetBasedCandidateGenerationParams.SimClustersMinScoreParam)\n    val tweetBasedSANN1Candidates =\n      if (params(TweetBasedCandidateGenerationParams.EnableSimClustersANN1Param)) {\n        Future.collect(\n          CandidateSourcesRouter.getTweetBasedSourceInfo(sourceSignals).toSeq.map { sourceInfo =>\n            getSimClustersANNCandidates(\n              requestUserId,\n              Some(sourceInfo),\n              params,\n              simClustersANN1ConfigId,\n              tweetBasedSANNMinScore)\n          })\n      } else Future.value(Seq.empty)\n\n    val simClustersANN2ConfigId = params(SimClustersANNParams.SimClustersANN2ConfigId)\n    val tweetBasedSANN2Candidates =\n      if (params(TweetBasedCandidateGenerationParams.EnableSimClustersANN2Param)) {\n        Future.collect(\n          CandidateSourcesRouter.getTweetBasedSourceInfo(sourceSignals).toSeq.map { sourceInfo =>\n            getSimClustersANNCandidates(\n              requestUserId,\n              Some(sourceInfo),\n              params,\n              simClustersANN2ConfigId,\n              tweetBasedSANNMinScore)\n          })\n      } else Future.value(Seq.empty)\n\n    val tweetBasedUagCandidates =\n      if (params(TweetBasedCandidateGenerationParams.EnableUAGParam)) {\n        Future.collect(\n          CandidateSourcesRouter.getTweetBasedSourceInfo(sourceSignals).toSeq.map { sourceInfo =>\n            getTweetBasedUserAdGraphCandidates(Some(sourceInfo), params)\n          })\n      } else Future.value(Seq.empty)\n\n    val realGraphInNetworkBasedUagCandidates =\n      if (params(ConsumersBasedUserAdGraphParams.EnableSourceParam)) {\n        getRealGraphConsumersBasedUserAdGraphCandidates(realGraphSeeds, params).map(Seq(_))\n      } else Future.value(Seq.empty)\n\n    val producerBasedUagCandidates =\n      if (params(ProducerBasedCandidateGenerationParams.EnableUAGParam)) {\n        Future.collect(\n          CandidateSourcesRouter.getProducerBasedSourceInfo(sourceSignals).toSeq.map { sourceInfo =>\n            getProducerBasedUserAdGraphCandidates(Some(sourceInfo), params)\n          })\n      } else Future.value(Seq.empty)\n\n    val tweetBasedTwhinAdsCandidates =\n      if (params(TweetBasedCandidateGenerationParams.EnableTwHINParam)) {\n        Future.collect(\n          CandidateSourcesRouter.getTweetBasedSourceInfo(sourceSignals).toSeq.map { sourceInfo =>\n            getTwHINAdsCandidates(\n              tweetBasedTwHINANNSimilarityEngine,\n              SimilarityEngineType.TweetBasedTwHINANN,\n              requestUserId,\n              Some(sourceInfo),\n              ModelConfig.DebuggerDemo)\n          })\n      } else Future.value(Seq.empty)\n\n    val producerBasedSANNMinScore = params(\n      ProducerBasedCandidateGenerationParams.SimClustersMinScoreParam)\n    val producerBasedSANN1Candidates =\n      if (params(ProducerBasedCandidateGenerationParams.EnableSimClustersANN1Param)) {\n        Future.collect(\n          CandidateSourcesRouter.getProducerBasedSourceInfo(sourceSignals).toSeq.map { sourceInfo =>\n            getSimClustersANNCandidates(\n              requestUserId,\n              Some(sourceInfo),\n              params,\n              simClustersANN1ConfigId,\n              producerBasedSANNMinScore)\n          })\n      } else Future.value(Seq.empty)\n    val producerBasedSANN2Candidates =\n      if (params(ProducerBasedCandidateGenerationParams.EnableSimClustersANN2Param)) {\n        Future.collect(\n          CandidateSourcesRouter.getProducerBasedSourceInfo(sourceSignals).toSeq.map { sourceInfo =>\n            getSimClustersANNCandidates(\n              requestUserId,\n              Some(sourceInfo),\n              params,\n              simClustersANN2ConfigId,\n              producerBasedSANNMinScore)\n          })\n      } else Future.value(Seq.empty)\n\n    val interestedInMinScore = params(InterestedInParams.MinScoreParam)\n    val interestedInSANN1Candidates = if (params(InterestedInParams.EnableSimClustersANN1Param)) {\n      getSimClustersANNCandidates(\n        requestUserId,\n        None,\n        params,\n        simClustersANN1ConfigId,\n        interestedInMinScore).map(Seq(_))\n    } else Future.value(Seq.empty)\n\n    val interestedInSANN2Candidates = if (params(InterestedInParams.EnableSimClustersANN2Param)) {\n      getSimClustersANNCandidates(\n        requestUserId,\n        None,\n        params,\n        simClustersANN2ConfigId,\n        interestedInMinScore).map(Seq(_))\n    } else Future.value(Seq.empty)\n\n    val consumerTwHINAdsCandidates =\n      if (params(ConsumerEmbeddingBasedCandidateGenerationParams.EnableTwHINParam)) {\n        getTwHINAdsCandidates(\n          consumerTwHINANNSimilarityEngine,\n          SimilarityEngineType.ConsumerEmbeddingBasedTwHINANN,\n          requestUserId,\n          None,\n          ModelConfig.DebuggerDemo).map(Seq(_))\n      } else Future.value(Seq.empty)\n\n    val consumerBasedWalsCandidates =\n      if (params(\n          ConsumerBasedWalsParams.EnableSourceParam\n        )) {\n        getConsumerBasedWalsCandidates(sourceSignals, params)\n      }.map {\n        Seq(_)\n      }\n      else Future.value(Seq.empty)\n\n    Future\n      .collect(Seq(\n        tweetBasedSANN1Candidates,\n        tweetBasedSANN2Candidates,\n        tweetBasedUagCandidates,\n        tweetBasedTwhinAdsCandidates,\n        producerBasedUagCandidates,\n        producerBasedSANN1Candidates,\n        producerBasedSANN2Candidates,\n        realGraphInNetworkBasedUagCandidates,\n        interestedInSANN1Candidates,\n        interestedInSANN2Candidates,\n        consumerTwHINAdsCandidates,\n        consumerBasedWalsCandidates,\n      )).map(_.flatten).map { tweetsWithCGInfoSeq =>\n        Future.collect(\n          tweetsWithCGInfoSeq.map(candidates => convertToInitialCandidates(candidates, stats)))\n      }.flatten.map { candidatesLists =>\n        val result = candidatesLists.filter(_.nonEmpty)\n        stats.stat(\"numOfSequences\").add(result.size)\n        stats.stat(\"flattenCandidatesWithDup\").add(result.flatten.size)\n        result\n      }\n  }\n\n  private[candidate_generation] def convertToInitialCandidates(\n    candidates: Seq[TweetWithCandidateGenerationInfo],\n    stats: StatsReceiver\n  ): Future[Seq[InitialAdsCandidate]] = {\n    val tweetIds = candidates.map(_.tweetId).toSet\n    stats.stat(\"initialCandidateSizeBeforeLineItemFilter\").add(tweetIds.size)\n    Future.collect(activePromotedTweetStore.multiGet(tweetIds)).map { lineItemInfos =>\n      /** *\n       * If lineItemInfo does not exist, we will filter out the promoted tweet as it cannot be targeted and ranked in admixer\n       */\n      val filteredCandidates = candidates.collect {\n        case candidate if lineItemInfos.getOrElse(candidate.tweetId, None).isDefined =>\n          val lineItemInfo = lineItemInfos(candidate.tweetId)\n            .getOrElse(throw new IllegalStateException(\"Check previous line's condition\"))\n\n          InitialAdsCandidate(\n            tweetId = candidate.tweetId,\n            lineItemInfo = lineItemInfo,\n            candidate.candidateGenerationInfo\n          )\n      }\n      stats.stat(\"initialCandidateSizeAfterLineItemFilter\").add(filteredCandidates.size)\n      filteredCandidates\n    }\n  }\n\n  private[candidate_generation] def getSimClustersANNCandidates(\n    requestUserId: UserId,\n    sourceInfo: Option[SourceInfo],\n    params: configapi.Params,\n    configId: String,\n    minScore: Double\n  ) = {\n\n    val simClustersModelVersion =\n      ModelVersions.Enum.enumToSimClustersModelVersionMap(params(GlobalParams.ModelVersionParam))\n\n    val embeddingType =\n      if (sourceInfo.isEmpty) {\n        params(InterestedInParams.InterestedInEmbeddingIdParam).embeddingType\n      } else getSimClustersANNEmbeddingType(sourceInfo.get)\n    val query = SimClustersANNSimilarityEngine.fromParams(\n      if (sourceInfo.isEmpty) InternalId.UserId(requestUserId) else sourceInfo.get.internalId,\n      embeddingType,\n      simClustersModelVersion,\n      configId,\n      params\n    )\n\n    // dark traffic to simclusters-ann-2\n    if (decider.isAvailable(DeciderConstants.enableSimClustersANN2DarkTrafficDeciderKey)) {\n      val simClustersANN2ConfigId = params(SimClustersANNParams.SimClustersANN2ConfigId)\n      val sann2Query = SimClustersANNSimilarityEngine.fromParams(\n        if (sourceInfo.isEmpty) InternalId.UserId(requestUserId) else sourceInfo.get.internalId,\n        embeddingType,\n        simClustersModelVersion,\n        simClustersANN2ConfigId,\n        params\n      )\n      simClustersANNSimilarityEngine\n        .getCandidates(sann2Query)\n    }\n\n    simClustersANNSimilarityEngine\n      .getCandidates(query).map(_.getOrElse(Seq.empty)).map(_.filter(_.score > minScore).map {\n        tweetWithScore =>\n          val similarityEngineInfo = SimClustersANNSimilarityEngine\n            .toSimilarityEngineInfo(query, tweetWithScore.score)\n          TweetWithCandidateGenerationInfo(\n            tweetWithScore.tweetId,\n            CandidateGenerationInfo(\n              sourceInfo,\n              similarityEngineInfo,\n              Seq(similarityEngineInfo)\n            ))\n      })\n  }\n\n  private[candidate_generation] def getProducerBasedUserAdGraphCandidates(\n    sourceInfo: Option[SourceInfo],\n    params: configapi.Params\n  ) = {\n\n    val query = ProducerBasedUserAdGraphSimilarityEngine.fromParams(\n      sourceInfo.get.internalId,\n      params\n    )\n    producerBasedUserAdGraphSimilarityEngine\n      .getCandidates(query).map(_.getOrElse(Seq.empty)).map(_.map { tweetWithScore =>\n        val similarityEngineInfo = ProducerBasedUserAdGraphSimilarityEngine\n          .toSimilarityEngineInfo(tweetWithScore.score)\n        TweetWithCandidateGenerationInfo(\n          tweetWithScore.tweetId,\n          CandidateGenerationInfo(\n            sourceInfo,\n            similarityEngineInfo,\n            Seq(similarityEngineInfo)\n          ))\n      })\n  }\n\n  private[candidate_generation] def getTweetBasedUserAdGraphCandidates(\n    sourceInfo: Option[SourceInfo],\n    params: configapi.Params\n  ) = {\n\n    val query = TweetBasedUserAdGraphSimilarityEngine.fromParams(\n      sourceInfo.get.internalId,\n      params\n    )\n    tweetBasedUserAdGraphSimilarityEngine\n      .getCandidates(query).map(_.getOrElse(Seq.empty)).map(_.map { tweetWithScore =>\n        val similarityEngineInfo = TweetBasedUserAdGraphSimilarityEngine\n          .toSimilarityEngineInfo(tweetWithScore.score)\n        TweetWithCandidateGenerationInfo(\n          tweetWithScore.tweetId,\n          CandidateGenerationInfo(\n            sourceInfo,\n            similarityEngineInfo,\n            Seq(similarityEngineInfo)\n          ))\n      })\n  }\n\n  private[candidate_generation] def getRealGraphConsumersBasedUserAdGraphCandidates(\n    realGraphSeeds: Map[UserId, Double],\n    params: configapi.Params\n  ) = {\n\n    val query = ConsumersBasedUserAdGraphSimilarityEngine\n      .fromParams(realGraphSeeds, params)\n\n    // The internalId is a placeholder value. We do not plan to store the full seedUserId set.\n    val sourceInfo = SourceInfo(\n      sourceType = SourceType.RealGraphIn,\n      internalId = InternalId.UserId(0L),\n      sourceEventTime = None\n    )\n    consumersBasedUserAdGraphSimilarityEngine\n      .getCandidates(query).map(_.getOrElse(Seq.empty)).map(_.map { tweetWithScore =>\n        val similarityEngineInfo = ConsumersBasedUserAdGraphSimilarityEngine\n          .toSimilarityEngineInfo(tweetWithScore.score)\n        TweetWithCandidateGenerationInfo(\n          tweetWithScore.tweetId,\n          CandidateGenerationInfo(\n            Some(sourceInfo),\n            similarityEngineInfo,\n            Seq.empty // Atomic Similarity Engine. Hence it has no contributing SEs\n          )\n        )\n      })\n  }\n\n  private[candidate_generation] def getTwHINAdsCandidates(\n    similarityEngine: HnswANNSimilarityEngine,\n    similarityEngineType: SimilarityEngineType,\n    requestUserId: UserId,\n    sourceInfo: Option[SourceInfo], // if none, then it's consumer-based similarity engine\n    model: String\n  ): Future[Seq[TweetWithCandidateGenerationInfo]] = {\n    val internalId =\n      if (sourceInfo.nonEmpty) sourceInfo.get.internalId else InternalId.UserId(requestUserId)\n    similarityEngine\n      .getCandidates(buildHnswANNQuery(internalId, model)).map(_.getOrElse(Seq.empty)).map(_.map {\n        tweetWithScore =>\n          val similarityEngineInfo = SimilarityEngineInfo(\n            similarityEngineType = similarityEngineType,\n            modelId = Some(model),\n            score = Some(tweetWithScore.score))\n          TweetWithCandidateGenerationInfo(\n            tweetWithScore.tweetId,\n            CandidateGenerationInfo(\n              None,\n              similarityEngineInfo,\n              Seq(similarityEngineInfo)\n            ))\n      })\n  }\n\n  private[candidate_generation] def getConsumerBasedWalsCandidates(\n    sourceSignals: Set[SourceInfo],\n    params: configapi.Params\n  ): Future[Seq[TweetWithCandidateGenerationInfo]] = {\n    // Fetch source signals and filter them based on age.\n    val signals = FilterUtil.tweetSourceAgeFilter(\n      getConsumerBasedWalsSourceInfo(sourceSignals).toSeq,\n      params(ConsumerBasedWalsParams.MaxTweetSignalAgeHoursParam))\n\n    val candidatesOptFut = consumerBasedWalsSimilarityEngine.getCandidates(\n      ConsumerBasedWalsSimilarityEngine.fromParams(signals, params)\n    )\n    val tweetsWithCandidateGenerationInfoOptFut = candidatesOptFut.map {\n      _.map { tweetsWithScores =>\n        val sortedCandidates = tweetsWithScores.sortBy(-_.score)\n        val filteredCandidates =\n          FilterUtil.tweetAgeFilter(sortedCandidates, params(GlobalParams.MaxTweetAgeHoursParam))\n        consumerBasedWalsSimilarityEngine.getScopedStats\n          .stat(\"filteredCandidates_size\").add(filteredCandidates.size)\n\n        val tweetsWithCandidateGenerationInfo = filteredCandidates.map { tweetWithScore =>\n          {\n            val similarityEngineInfo =\n              ConsumerBasedWalsSimilarityEngine.toSimilarityEngineInfo(tweetWithScore.score)\n            TweetWithCandidateGenerationInfo(\n              tweetWithScore.tweetId,\n              CandidateGenerationInfo(\n                None,\n                similarityEngineInfo,\n                Seq.empty // Atomic Similarity Engine. Hence it has no contributing SEs\n              )\n            )\n          }\n        }\n        val maxCandidateNum = params(GlobalParams.MaxCandidateNumPerSourceKeyParam)\n        tweetsWithCandidateGenerationInfo.take(maxCandidateNum)\n      }\n    }\n    for {\n      tweetsWithCandidateGenerationInfoOpt <- tweetsWithCandidateGenerationInfoOptFut\n    } yield tweetsWithCandidateGenerationInfoOpt.toSeq.flatten\n  }\n}\n\nobject AdsCandidateSourcesRouter {\n  def getSimClustersANNEmbeddingType(\n    sourceInfo: SourceInfo\n  ): EmbeddingType = {\n    sourceInfo.sourceType match {\n      case SourceType.TweetFavorite | SourceType.Retweet | SourceType.OriginalTweet |\n          SourceType.Reply | SourceType.TweetShare | SourceType.NotificationClick |\n          SourceType.GoodTweetClick | SourceType.VideoTweetQualityView |\n          SourceType.VideoTweetPlayback50 =>\n        EmbeddingType.LogFavLongestL2EmbeddingTweet\n      case SourceType.UserFollow | SourceType.UserRepeatedProfileVisit | SourceType.RealGraphOon |\n          SourceType.FollowRecommendation | SourceType.UserTrafficAttributionProfileVisit |\n          SourceType.GoodProfileClick | SourceType.TwiceUserId =>\n        EmbeddingType.FavBasedProducer\n      case _ => throw new IllegalArgumentException(\"sourceInfo.sourceType not supported\")\n    }\n  }\n\n  def buildHnswANNQuery(internalId: InternalId, modelId: String): HnswANNEngineQuery = {\n    HnswANNEngineQuery(\n      sourceId = internalId,\n      modelId = modelId,\n      params = Params.Empty\n    )\n  }\n\n  def getConsumerBasedWalsSourceInfo(\n    sourceSignals: Set[SourceInfo]\n  ): Set[SourceInfo] = {\n    val AllowedSourceTypesForConsumerBasedWalsSE = Set(\n      SourceType.TweetFavorite.value,\n      SourceType.Retweet.value,\n      SourceType.TweetDontLike.value, //currently no-op\n      SourceType.TweetReport.value, //currently no-op\n      SourceType.AccountMute.value, //currently no-op\n      SourceType.AccountBlock.value //currently no-op\n    )\n    sourceSignals.collect {\n      case sourceInfo\n          if AllowedSourceTypesForConsumerBasedWalsSE.contains(sourceInfo.sourceType.value) =>\n        sourceInfo\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"ann/src/main/scala/com/twitter/ann/hnsw\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"configapi/configapi-core\",\n        \"content-recommender/thrift/src/main/thrift:thrift-scala\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/blender\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/config\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/logging\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/ranker\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"cuad/projects/hashspace/thrift:thrift-scala\",\n        \"decider/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"frigate/frigate-common:base\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/base\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/candidate\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/util:stats_util\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/constants\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/model\",\n        \"simclusters-ann/thrift/src/main/thrift:thrift-scala\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"src/scala/com/twitter/cortex/ml/embeddings/common:Helpers\",\n        \"src/scala/com/twitter/ml/featurestore/lib\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/frigate/data_pipeline/scalding:blue_verified_annotations-scala\",\n        \"src/thrift/com/twitter/ml/api:embedding-scala\",\n        \"src/thrift/com/twitter/recos/user_tweet_graph:user_tweet_graph-scala\",\n        \"src/thrift/com/twitter/recos/user_tweet_graph_plus:user_tweet_graph_plus-scala\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"src/thrift/com/twitter/search/query_interaction_graph/service:qig-service-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala\",\n        \"strato/config/columns/cuad/hashspace:hashspace-strato-client\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation/CandidateSourcesRouter.scala",
    "content": "package com.twitter.cr_mixer.candidate_generation\n\nimport com.twitter.contentrecommender.thriftscala.TweetInfo\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.GraphSourceInfo\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.model.TripTweetWithScore\nimport com.twitter.cr_mixer.model.TweetWithCandidateGenerationInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.model.TweetWithScoreAndSocialProof\nimport com.twitter.cr_mixer.param.ConsumerBasedWalsParams\nimport com.twitter.cr_mixer.param.ConsumerEmbeddingBasedCandidateGenerationParams\nimport com.twitter.cr_mixer.param.ConsumersBasedUserVideoGraphParams\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.similarity_engine.ConsumersBasedUserVideoGraphSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.ConsumerBasedWalsSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.ConsumerEmbeddingBasedTripSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.ConsumerEmbeddingBasedTwHINSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.ConsumerEmbeddingBasedTwoTowerSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.EngineQuery\nimport com.twitter.cr_mixer.similarity_engine.FilterUtil\nimport com.twitter.cr_mixer.similarity_engine.HnswANNEngineQuery\nimport com.twitter.cr_mixer.similarity_engine.HnswANNSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.ProducerBasedUnifiedSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TripEngineQuery\nimport com.twitter.cr_mixer.similarity_engine.TweetBasedUnifiedSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.UserTweetEntityGraphSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/**\n * Route the SourceInfo to the associated Candidate Engines.\n */\n@Singleton\ncase class CandidateSourcesRouter @Inject() (\n  customizedRetrievalCandidateGeneration: CustomizedRetrievalCandidateGeneration,\n  simClustersInterestedInCandidateGeneration: SimClustersInterestedInCandidateGeneration,\n  @Named(ModuleNames.TweetBasedUnifiedSimilarityEngine)\n  tweetBasedUnifiedSimilarityEngine: StandardSimilarityEngine[\n    TweetBasedUnifiedSimilarityEngine.Query,\n    TweetWithCandidateGenerationInfo\n  ],\n  @Named(ModuleNames.ProducerBasedUnifiedSimilarityEngine)\n  producerBasedUnifiedSimilarityEngine: StandardSimilarityEngine[\n    ProducerBasedUnifiedSimilarityEngine.Query,\n    TweetWithCandidateGenerationInfo\n  ],\n  @Named(ModuleNames.ConsumerEmbeddingBasedTripSimilarityEngine)\n  consumerEmbeddingBasedTripSimilarityEngine: StandardSimilarityEngine[\n    TripEngineQuery,\n    TripTweetWithScore\n  ],\n  @Named(ModuleNames.ConsumerEmbeddingBasedTwHINANNSimilarityEngine)\n  consumerBasedTwHINANNSimilarityEngine: HnswANNSimilarityEngine,\n  @Named(ModuleNames.ConsumerEmbeddingBasedTwoTowerANNSimilarityEngine)\n  consumerBasedTwoTowerSimilarityEngine: HnswANNSimilarityEngine,\n  @Named(ModuleNames.ConsumersBasedUserVideoGraphSimilarityEngine)\n  consumersBasedUserVideoGraphSimilarityEngine: StandardSimilarityEngine[\n    ConsumersBasedUserVideoGraphSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  @Named(ModuleNames.UserTweetEntityGraphSimilarityEngine) userTweetEntityGraphSimilarityEngine: StandardSimilarityEngine[\n    UserTweetEntityGraphSimilarityEngine.Query,\n    TweetWithScoreAndSocialProof\n  ],\n  @Named(ModuleNames.ConsumerBasedWalsSimilarityEngine)\n  consumerBasedWalsSimilarityEngine: StandardSimilarityEngine[\n    ConsumerBasedWalsSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  tweetInfoStore: ReadableStore[TweetId, TweetInfo],\n  globalStats: StatsReceiver,\n) {\n\n  import CandidateSourcesRouter._\n  val stats: StatsReceiver = globalStats.scope(this.getClass.getSimpleName)\n\n  def fetchCandidates(\n    requestUserId: UserId,\n    sourceSignals: Set[SourceInfo],\n    sourceGraphs: Map[String, Option[GraphSourceInfo]],\n    params: configapi.Params,\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n\n    val tweetBasedCandidatesFuture = getCandidates(\n      getTweetBasedSourceInfo(sourceSignals),\n      params,\n      TweetBasedUnifiedSimilarityEngine.fromParams,\n      tweetBasedUnifiedSimilarityEngine.getCandidates)\n\n    val producerBasedCandidatesFuture =\n      getCandidates(\n        getProducerBasedSourceInfo(sourceSignals),\n        params,\n        ProducerBasedUnifiedSimilarityEngine.fromParams(_, _),\n        producerBasedUnifiedSimilarityEngine.getCandidates\n      )\n\n    val simClustersInterestedInBasedCandidatesFuture =\n      getCandidatesPerSimilarityEngineModel(\n        requestUserId,\n        params,\n        SimClustersInterestedInCandidateGeneration.fromParams,\n        simClustersInterestedInCandidateGeneration.get)\n\n    val consumerEmbeddingBasedLogFavBasedTripCandidatesFuture =\n      if (params(\n          ConsumerEmbeddingBasedCandidateGenerationParams.EnableLogFavBasedSimClustersTripParam)) {\n        getSimClustersTripCandidates(\n          params,\n          ConsumerEmbeddingBasedTripSimilarityEngine.fromParams(\n            ModelConfig.ConsumerLogFavBasedInterestedInEmbedding,\n            InternalId.UserId(requestUserId),\n            params\n          ),\n          consumerEmbeddingBasedTripSimilarityEngine\n        ).map {\n          Seq(_)\n        }\n      } else\n        Future.Nil\n\n    val consumersBasedUvgRealGraphInCandidatesFuture =\n      if (params(ConsumersBasedUserVideoGraphParams.EnableSourceParam)) {\n        val realGraphInGraphSourceInfoOpt =\n          getGraphSourceInfoBySourceType(SourceType.RealGraphIn.name, sourceGraphs)\n\n        getGraphBasedCandidates(\n          params,\n          ConsumersBasedUserVideoGraphSimilarityEngine\n            .fromParamsForRealGraphIn(\n              realGraphInGraphSourceInfoOpt\n                .map { graphSourceInfo => graphSourceInfo.seedWithScores }.getOrElse(Map.empty),\n              params),\n          consumersBasedUserVideoGraphSimilarityEngine,\n          ConsumersBasedUserVideoGraphSimilarityEngine.toSimilarityEngineInfo,\n          realGraphInGraphSourceInfoOpt\n        ).map {\n          Seq(_)\n        }\n      } else Future.Nil\n\n    val consumerEmbeddingBasedFollowBasedTripCandidatesFuture =\n      if (params(\n          ConsumerEmbeddingBasedCandidateGenerationParams.EnableFollowBasedSimClustersTripParam)) {\n        getSimClustersTripCandidates(\n          params,\n          ConsumerEmbeddingBasedTripSimilarityEngine.fromParams(\n            ModelConfig.ConsumerFollowBasedInterestedInEmbedding,\n            InternalId.UserId(requestUserId),\n            params\n          ),\n          consumerEmbeddingBasedTripSimilarityEngine\n        ).map {\n          Seq(_)\n        }\n      } else\n        Future.Nil\n\n    val consumerBasedWalsCandidatesFuture =\n      if (params(\n          ConsumerBasedWalsParams.EnableSourceParam\n        )) {\n        getConsumerBasedWalsCandidates(sourceSignals, params)\n      }.map { Seq(_) }\n      else Future.Nil\n\n    val consumerEmbeddingBasedTwHINCandidatesFuture =\n      if (params(ConsumerEmbeddingBasedCandidateGenerationParams.EnableTwHINParam)) {\n        getHnswCandidates(\n          params,\n          ConsumerEmbeddingBasedTwHINSimilarityEngine.fromParams(\n            InternalId.UserId(requestUserId),\n            params),\n          consumerBasedTwHINANNSimilarityEngine\n        ).map { Seq(_) }\n      } else Future.Nil\n\n    val consumerEmbeddingBasedTwoTowerCandidatesFuture =\n      if (params(ConsumerEmbeddingBasedCandidateGenerationParams.EnableTwoTowerParam)) {\n        getHnswCandidates(\n          params,\n          ConsumerEmbeddingBasedTwoTowerSimilarityEngine.fromParams(\n            InternalId.UserId(requestUserId),\n            params),\n          consumerBasedTwoTowerSimilarityEngine\n        ).map {\n          Seq(_)\n        }\n      } else Future.Nil\n\n    val customizedRetrievalBasedCandidatesFuture =\n      getCandidatesPerSimilarityEngineModel(\n        requestUserId,\n        params,\n        CustomizedRetrievalCandidateGeneration.fromParams,\n        customizedRetrievalCandidateGeneration.get)\n\n    Future\n      .collect(\n        Seq(\n          tweetBasedCandidatesFuture,\n          producerBasedCandidatesFuture,\n          simClustersInterestedInBasedCandidatesFuture,\n          consumerBasedWalsCandidatesFuture,\n          consumerEmbeddingBasedLogFavBasedTripCandidatesFuture,\n          consumerEmbeddingBasedFollowBasedTripCandidatesFuture,\n          consumerEmbeddingBasedTwHINCandidatesFuture,\n          consumerEmbeddingBasedTwoTowerCandidatesFuture,\n          consumersBasedUvgRealGraphInCandidatesFuture,\n          customizedRetrievalBasedCandidatesFuture\n        )).map { candidatesList =>\n        // remove empty innerSeq\n        val result = candidatesList.flatten.filter(_.nonEmpty)\n        stats.stat(\"numOfSequences\").add(result.size)\n        stats.stat(\"flattenCandidatesWithDup\").add(result.flatten.size)\n\n        result\n      }\n  }\n\n  private def getGraphBasedCandidates[QueryType](\n    params: configapi.Params,\n    query: EngineQuery[QueryType],\n    engine: StandardSimilarityEngine[QueryType, TweetWithScore],\n    toSimilarityEngineInfo: Double => SimilarityEngineInfo,\n    graphSourceInfoOpt: Option[GraphSourceInfo] = None\n  ): Future[Seq[InitialCandidate]] = {\n    val candidatesOptFut = engine.getCandidates(query)\n    val tweetsWithCandidateGenerationInfoOptFut = candidatesOptFut.map {\n      _.map { tweetsWithScores =>\n        val sortedCandidates = tweetsWithScores.sortBy(-_.score)\n        engine.getScopedStats.stat(\"sortedCandidates_size\").add(sortedCandidates.size)\n        val tweetsWithCandidateGenerationInfo = sortedCandidates.map { tweetWithScore =>\n          {\n            val similarityEngineInfo = toSimilarityEngineInfo(tweetWithScore.score)\n            val sourceInfo = graphSourceInfoOpt.map { graphSourceInfo =>\n              // The internalId is a placeholder value. We do not plan to store the full seedUserId set.\n              SourceInfo(\n                sourceType = graphSourceInfo.sourceType,\n                internalId = InternalId.UserId(0L),\n                sourceEventTime = None\n              )\n            }\n            TweetWithCandidateGenerationInfo(\n              tweetWithScore.tweetId,\n              CandidateGenerationInfo(\n                sourceInfo,\n                similarityEngineInfo,\n                Seq.empty // Atomic Similarity Engine. Hence it has no contributing SEs\n              )\n            )\n          }\n        }\n        val maxCandidateNum = params(GlobalParams.MaxCandidateNumPerSourceKeyParam)\n        tweetsWithCandidateGenerationInfo.take(maxCandidateNum)\n      }\n    }\n    for {\n      tweetsWithCandidateGenerationInfoOpt <- tweetsWithCandidateGenerationInfoOptFut\n      initialCandidates <- convertToInitialCandidates(\n        tweetsWithCandidateGenerationInfoOpt.toSeq.flatten)\n    } yield initialCandidates\n  }\n\n  private def getCandidates[QueryType](\n    sourceSignals: Set[SourceInfo],\n    params: configapi.Params,\n    fromParams: (SourceInfo, configapi.Params) => QueryType,\n    getFunc: QueryType => Future[Option[Seq[TweetWithCandidateGenerationInfo]]]\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    val queries = sourceSignals.map { sourceInfo =>\n      fromParams(sourceInfo, params)\n    }.toSeq\n\n    Future\n      .collect {\n        queries.map { query =>\n          for {\n            candidates <- getFunc(query)\n            prefilterCandidates <- convertToInitialCandidates(candidates.toSeq.flatten)\n          } yield {\n            prefilterCandidates\n          }\n        }\n      }\n  }\n\n  private def getConsumerBasedWalsCandidates(\n    sourceSignals: Set[SourceInfo],\n    params: configapi.Params\n  ): Future[Seq[InitialCandidate]] = {\n    // Fetch source signals and filter them based on age.\n    val signals = FilterUtil.tweetSourceAgeFilter(\n      getConsumerBasedWalsSourceInfo(sourceSignals).toSeq,\n      params(ConsumerBasedWalsParams.MaxTweetSignalAgeHoursParam))\n\n    val candidatesOptFut = consumerBasedWalsSimilarityEngine.getCandidates(\n      ConsumerBasedWalsSimilarityEngine.fromParams(signals, params)\n    )\n    val tweetsWithCandidateGenerationInfoOptFut = candidatesOptFut.map {\n      _.map { tweetsWithScores =>\n        val sortedCandidates = tweetsWithScores.sortBy(-_.score)\n        val filteredCandidates =\n          FilterUtil.tweetAgeFilter(sortedCandidates, params(GlobalParams.MaxTweetAgeHoursParam))\n        consumerBasedWalsSimilarityEngine.getScopedStats\n          .stat(\"filteredCandidates_size\").add(filteredCandidates.size)\n\n        val tweetsWithCandidateGenerationInfo = filteredCandidates.map { tweetWithScore =>\n          {\n            val similarityEngineInfo =\n              ConsumerBasedWalsSimilarityEngine.toSimilarityEngineInfo(tweetWithScore.score)\n            TweetWithCandidateGenerationInfo(\n              tweetWithScore.tweetId,\n              CandidateGenerationInfo(\n                None,\n                similarityEngineInfo,\n                Seq.empty // Atomic Similarity Engine. Hence it has no contributing SEs\n              )\n            )\n          }\n        }\n        val maxCandidateNum = params(GlobalParams.MaxCandidateNumPerSourceKeyParam)\n        tweetsWithCandidateGenerationInfo.take(maxCandidateNum)\n      }\n    }\n    for {\n      tweetsWithCandidateGenerationInfoOpt <- tweetsWithCandidateGenerationInfoOptFut\n      initialCandidates <- convertToInitialCandidates(\n        tweetsWithCandidateGenerationInfoOpt.toSeq.flatten)\n    } yield initialCandidates\n  }\n\n  private def getSimClustersTripCandidates(\n    params: configapi.Params,\n    query: TripEngineQuery,\n    engine: StandardSimilarityEngine[\n      TripEngineQuery,\n      TripTweetWithScore\n    ],\n  ): Future[Seq[InitialCandidate]] = {\n    val tweetsWithCandidatesGenerationInfoOptFut =\n      engine.getCandidates(EngineQuery(query, params)).map {\n        _.map {\n          _.map { tweetWithScore =>\n            // define filters\n            TweetWithCandidateGenerationInfo(\n              tweetWithScore.tweetId,\n              CandidateGenerationInfo(\n                None,\n                SimilarityEngineInfo(\n                  SimilarityEngineType.ExploreTripOfflineSimClustersTweets,\n                  None,\n                  Some(tweetWithScore.score)),\n                Seq.empty\n              )\n            )\n          }\n        }\n      }\n    for {\n      tweetsWithCandidateGenerationInfoOpt <- tweetsWithCandidatesGenerationInfoOptFut\n      initialCandidates <- convertToInitialCandidates(\n        tweetsWithCandidateGenerationInfoOpt.toSeq.flatten)\n    } yield initialCandidates\n  }\n\n  private def getHnswCandidates(\n    params: configapi.Params,\n    query: HnswANNEngineQuery,\n    engine: HnswANNSimilarityEngine,\n  ): Future[Seq[InitialCandidate]] = {\n    val candidatesOptFut = engine.getCandidates(query)\n    val tweetsWithCandidateGenerationInfoOptFut = candidatesOptFut.map {\n      _.map { tweetsWithScores =>\n        val sortedCandidates = tweetsWithScores.sortBy(-_.score)\n        val filteredCandidates =\n          FilterUtil.tweetAgeFilter(sortedCandidates, params(GlobalParams.MaxTweetAgeHoursParam))\n        engine.getScopedStats.stat(\"filteredCandidates_size\").add(filteredCandidates.size)\n        val tweetsWithCandidateGenerationInfo = filteredCandidates.map { tweetWithScore =>\n          {\n            val similarityEngineInfo =\n              engine.toSimilarityEngineInfo(query, tweetWithScore.score)\n            TweetWithCandidateGenerationInfo(\n              tweetWithScore.tweetId,\n              CandidateGenerationInfo(\n                None,\n                similarityEngineInfo,\n                Seq.empty // Atomic Similarity Engine. Hence it has no contributing SEs\n              )\n            )\n          }\n        }\n        val maxCandidateNum = params(GlobalParams.MaxCandidateNumPerSourceKeyParam)\n        tweetsWithCandidateGenerationInfo.take(maxCandidateNum)\n      }\n    }\n    for {\n      tweetsWithCandidateGenerationInfoOpt <- tweetsWithCandidateGenerationInfoOptFut\n      initialCandidates <- convertToInitialCandidates(\n        tweetsWithCandidateGenerationInfoOpt.toSeq.flatten)\n    } yield initialCandidates\n  }\n\n  /**\n   * Returns candidates from each similarity engine separately.\n   * For 1 requestUserId, it will fetch results from each similarity engine e_i,\n   * and returns Seq[Seq[TweetCandidate]].\n   */\n  private def getCandidatesPerSimilarityEngineModel[QueryType](\n    requestUserId: UserId,\n    params: configapi.Params,\n    fromParams: (InternalId, configapi.Params) => QueryType,\n    getFunc: QueryType => Future[\n      Option[Seq[Seq[TweetWithCandidateGenerationInfo]]]\n    ]\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    val query = fromParams(InternalId.UserId(requestUserId), params)\n    getFunc(query).flatMap { candidatesPerSimilarityEngineModelOpt =>\n      val candidatesPerSimilarityEngineModel = candidatesPerSimilarityEngineModelOpt.toSeq.flatten\n      Future.collect {\n        candidatesPerSimilarityEngineModel.map(convertToInitialCandidates)\n      }\n    }\n  }\n\n  private[candidate_generation] def convertToInitialCandidates(\n    candidates: Seq[TweetWithCandidateGenerationInfo],\n  ): Future[Seq[InitialCandidate]] = {\n    val tweetIds = candidates.map(_.tweetId).toSet\n    Future.collect(tweetInfoStore.multiGet(tweetIds)).map { tweetInfos =>\n      /***\n       * If tweetInfo does not exist, we will filter out this tweet candidate.\n       */\n      candidates.collect {\n        case candidate if tweetInfos.getOrElse(candidate.tweetId, None).isDefined =>\n          val tweetInfo = tweetInfos(candidate.tweetId)\n            .getOrElse(throw new IllegalStateException(\"Check previous line's condition\"))\n\n          InitialCandidate(\n            tweetId = candidate.tweetId,\n            tweetInfo = tweetInfo,\n            candidate.candidateGenerationInfo\n          )\n      }\n    }\n  }\n}\n\nobject CandidateSourcesRouter {\n  def getGraphSourceInfoBySourceType(\n    sourceTypeStr: String,\n    sourceGraphs: Map[String, Option[GraphSourceInfo]]\n  ): Option[GraphSourceInfo] = {\n    sourceGraphs.getOrElse(sourceTypeStr, None)\n  }\n\n  def getTweetBasedSourceInfo(\n    sourceSignals: Set[SourceInfo]\n  ): Set[SourceInfo] = {\n    sourceSignals.collect {\n      case sourceInfo\n          if AllowedSourceTypesForTweetBasedUnifiedSE.contains(sourceInfo.sourceType.value) =>\n        sourceInfo\n    }\n  }\n\n  def getProducerBasedSourceInfo(\n    sourceSignals: Set[SourceInfo]\n  ): Set[SourceInfo] = {\n    sourceSignals.collect {\n      case sourceInfo\n          if AllowedSourceTypesForProducerBasedUnifiedSE.contains(sourceInfo.sourceType.value) =>\n        sourceInfo\n    }\n  }\n\n  def getConsumerBasedWalsSourceInfo(\n    sourceSignals: Set[SourceInfo]\n  ): Set[SourceInfo] = {\n    sourceSignals.collect {\n      case sourceInfo\n          if AllowedSourceTypesForConsumerBasedWalsSE.contains(sourceInfo.sourceType.value) =>\n        sourceInfo\n    }\n  }\n\n  /***\n   * Signal funneling should not exist in CG or even in any SimilarityEngine.\n   * They will be in Router, or eventually, in CrCandidateGenerator.\n   */\n  val AllowedSourceTypesForConsumerBasedWalsSE = Set(\n    SourceType.TweetFavorite.value,\n    SourceType.Retweet.value,\n    SourceType.TweetDontLike.value, //currently no-op\n    SourceType.TweetReport.value, //currently no-op\n    SourceType.AccountMute.value, //currently no-op\n    SourceType.AccountBlock.value //currently no-op\n  )\n  val AllowedSourceTypesForTweetBasedUnifiedSE = Set(\n    SourceType.TweetFavorite.value,\n    SourceType.Retweet.value,\n    SourceType.OriginalTweet.value,\n    SourceType.Reply.value,\n    SourceType.TweetShare.value,\n    SourceType.NotificationClick.value,\n    SourceType.GoodTweetClick.value,\n    SourceType.VideoTweetQualityView.value,\n    SourceType.VideoTweetPlayback50.value,\n    SourceType.TweetAggregation.value,\n  )\n  val AllowedSourceTypesForProducerBasedUnifiedSE = Set(\n    SourceType.UserFollow.value,\n    SourceType.UserRepeatedProfileVisit.value,\n    SourceType.RealGraphOon.value,\n    SourceType.FollowRecommendation.value,\n    SourceType.UserTrafficAttributionProfileVisit.value,\n    SourceType.GoodProfileClick.value,\n    SourceType.ProducerAggregation.value,\n  )\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation/CrCandidateGenerator.scala",
    "content": "package com.twitter.cr_mixer.candidate_generation\n\nimport com.twitter.cr_mixer.blender.SwitchBlender\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.filter.PostRankFilterRunner\nimport com.twitter.cr_mixer.filter.PreRankFilterRunner\nimport com.twitter.cr_mixer.logging.CrMixerScribeLogger\nimport com.twitter.cr_mixer.model.BlendedCandidate\nimport com.twitter.cr_mixer.model.CrCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.GraphSourceInfo\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.model.RankedCandidate\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.param.RankerParams\nimport com.twitter.cr_mixer.param.RecentNegativeSignalParams\nimport com.twitter.cr_mixer.ranker.SwitchRanker\nimport com.twitter.cr_mixer.source_signal.SourceInfoRouter\nimport com.twitter.cr_mixer.source_signal.UssStore.EnabledNegativeSourceTypes\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.util.Future\nimport com.twitter.util.JavaTimer\nimport com.twitter.util.Timer\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * For now it performs the main steps as follows:\n * 1. Source signal (via USS, FRS) fetch\n * 2. Candidate generation\n * 3. Filtering\n * 4. Interleave blender\n * 5. Ranker\n * 6. Post-ranker filter\n * 7. Truncation\n */\n@Singleton\nclass CrCandidateGenerator @Inject() (\n  sourceInfoRouter: SourceInfoRouter,\n  candidateSourceRouter: CandidateSourcesRouter,\n  switchBlender: SwitchBlender,\n  preRankFilterRunner: PreRankFilterRunner,\n  postRankFilterRunner: PostRankFilterRunner,\n  switchRanker: SwitchRanker,\n  crMixerScribeLogger: CrMixerScribeLogger,\n  timeoutConfig: TimeoutConfig,\n  globalStats: StatsReceiver) {\n  private val timer: Timer = new JavaTimer(true)\n\n  private val stats: StatsReceiver = globalStats.scope(this.getClass.getCanonicalName)\n\n  private val fetchSourcesStats = stats.scope(\"fetchSources\")\n  private val fetchPositiveSourcesStats = stats.scope(\"fetchPositiveSources\")\n  private val fetchNegativeSourcesStats = stats.scope(\"fetchNegativeSources\")\n  private val fetchCandidatesStats = stats.scope(\"fetchCandidates\")\n  private val fetchCandidatesAfterFilterStats = stats.scope(\"fetchCandidatesAfterFilter\")\n  private val preRankFilterStats = stats.scope(\"preRankFilter\")\n  private val interleaveStats = stats.scope(\"interleave\")\n  private val rankStats = stats.scope(\"rank\")\n  private val postRankFilterStats = stats.scope(\"postRankFilter\")\n  private val blueVerifiedTweetStats = stats.scope(\"blueVerifiedTweetStats\")\n  private val blueVerifiedTweetStatsPerSimilarityEngine =\n    stats.scope(\"blueVerifiedTweetStatsPerSimilarityEngine\")\n\n  def get(query: CrCandidateGeneratorQuery): Future[Seq[RankedCandidate]] = {\n    val allStats = stats.scope(\"all\")\n    val perProductStats = stats.scope(\"perProduct\", query.product.toString)\n    val perProductBlueVerifiedStats =\n      blueVerifiedTweetStats.scope(\"perProduct\", query.product.toString)\n\n    StatsUtil.trackItemsStats(allStats) {\n      trackResultStats(perProductStats) {\n        StatsUtil.trackItemsStats(perProductStats) {\n          val result = for {\n            (sourceSignals, sourceGraphsMap) <- StatsUtil.trackBlockStats(fetchSourcesStats) {\n              fetchSources(query)\n            }\n            initialCandidates <- StatsUtil.trackBlockStats(fetchCandidatesAfterFilterStats) {\n              // find the positive and negative signals\n              val (positiveSignals, negativeSignals) = sourceSignals.partition { signal =>\n                !EnabledNegativeSourceTypes.contains(signal.sourceType)\n              }\n              fetchPositiveSourcesStats.stat(\"size\").add(positiveSignals.size)\n              fetchNegativeSourcesStats.stat(\"size\").add(negativeSignals.size)\n\n              // find the positive signals to keep, removing block and muted users\n              val filteredSourceInfo =\n                if (negativeSignals.nonEmpty && query.params(\n                    RecentNegativeSignalParams.EnableSourceParam)) {\n                  filterSourceInfo(positiveSignals, negativeSignals)\n                } else {\n                  positiveSignals\n                }\n\n              // fetch candidates from the positive signals\n              StatsUtil.trackBlockStats(fetchCandidatesStats) {\n                fetchCandidates(query, filteredSourceInfo, sourceGraphsMap)\n              }\n            }\n            filteredCandidates <- StatsUtil.trackBlockStats(preRankFilterStats) {\n              preRankFilter(query, initialCandidates)\n            }\n            interleavedCandidates <- StatsUtil.trackItemsStats(interleaveStats) {\n              interleave(query, filteredCandidates)\n            }\n            rankedCandidates <- StatsUtil.trackItemsStats(rankStats) {\n              val candidatesToRank =\n                interleavedCandidates.take(query.params(RankerParams.MaxCandidatesToRank))\n              rank(query, candidatesToRank)\n            }\n            postRankFilterCandidates <- StatsUtil.trackItemsStats(postRankFilterStats) {\n              postRankFilter(query, rankedCandidates)\n            }\n          } yield {\n            trackTopKStats(\n              800,\n              postRankFilterCandidates,\n              isQueryK = false,\n              perProductBlueVerifiedStats)\n            trackTopKStats(\n              400,\n              postRankFilterCandidates,\n              isQueryK = false,\n              perProductBlueVerifiedStats)\n            trackTopKStats(\n              query.maxNumResults,\n              postRankFilterCandidates,\n              isQueryK = true,\n              perProductBlueVerifiedStats)\n\n            val (blueVerifiedTweets, remainingTweets) =\n              postRankFilterCandidates.partition(\n                _.tweetInfo.hasBlueVerifiedAnnotation.contains(true))\n            val topKBlueVerified = blueVerifiedTweets.take(query.maxNumResults)\n            val topKRemaining = remainingTweets.take(query.maxNumResults - topKBlueVerified.size)\n\n            trackBlueVerifiedTweetStats(topKBlueVerified, perProductBlueVerifiedStats)\n\n            if (topKBlueVerified.nonEmpty && query.params(RankerParams.EnableBlueVerifiedTopK)) {\n              topKBlueVerified ++ topKRemaining\n            } else {\n              postRankFilterCandidates\n            }\n          }\n          result.raiseWithin(timeoutConfig.serviceTimeout)(timer)\n        }\n      }\n    }\n  }\n\n  private def fetchSources(\n    query: CrCandidateGeneratorQuery\n  ): Future[(Set[SourceInfo], Map[String, Option[GraphSourceInfo]])] = {\n    crMixerScribeLogger.scribeSignalSources(\n      query,\n      sourceInfoRouter\n        .get(query.userId, query.product, query.userState, query.params))\n  }\n\n  private def filterSourceInfo(\n    positiveSignals: Set[SourceInfo],\n    negativeSignals: Set[SourceInfo]\n  ): Set[SourceInfo] = {\n    val filterUsers: Set[Long] = negativeSignals.flatMap {\n      case SourceInfo(_, InternalId.UserId(userId), _) => Some(userId)\n      case _ => None\n    }\n\n    positiveSignals.filter {\n      case SourceInfo(_, InternalId.UserId(userId), _) => !filterUsers.contains(userId)\n      case _ => true\n    }\n  }\n\n  def fetchCandidates(\n    query: CrCandidateGeneratorQuery,\n    sourceSignals: Set[SourceInfo],\n    sourceGraphs: Map[String, Option[GraphSourceInfo]]\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    val initialCandidates = candidateSourceRouter\n      .fetchCandidates(\n        query.userId,\n        sourceSignals,\n        sourceGraphs,\n        query.params\n      )\n\n    initialCandidates.map(_.flatten.map { candidate =>\n      if (candidate.tweetInfo.hasBlueVerifiedAnnotation.contains(true)) {\n        blueVerifiedTweetStatsPerSimilarityEngine\n          .scope(query.product.toString).scope(\n            candidate.candidateGenerationInfo.contributingSimilarityEngines.head.similarityEngineType.toString).counter(\n            candidate.tweetInfo.authorId.toString).incr()\n      }\n    })\n\n    crMixerScribeLogger.scribeInitialCandidates(\n      query,\n      initialCandidates\n    )\n  }\n\n  private def preRankFilter(\n    query: CrCandidateGeneratorQuery,\n    candidates: Seq[Seq[InitialCandidate]]\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    crMixerScribeLogger.scribePreRankFilterCandidates(\n      query,\n      preRankFilterRunner\n        .runSequentialFilters(query, candidates))\n  }\n\n  private def postRankFilter(\n    query: CrCandidateGeneratorQuery,\n    candidates: Seq[RankedCandidate]\n  ): Future[Seq[RankedCandidate]] = {\n    postRankFilterRunner.run(query, candidates)\n  }\n\n  private def interleave(\n    query: CrCandidateGeneratorQuery,\n    candidates: Seq[Seq[InitialCandidate]]\n  ): Future[Seq[BlendedCandidate]] = {\n    crMixerScribeLogger.scribeInterleaveCandidates(\n      query,\n      switchBlender\n        .blend(query.params, query.userState, candidates))\n  }\n\n  private def rank(\n    query: CrCandidateGeneratorQuery,\n    candidates: Seq[BlendedCandidate],\n  ): Future[Seq[RankedCandidate]] = {\n    crMixerScribeLogger.scribeRankedCandidates(\n      query,\n      switchRanker.rank(query, candidates)\n    )\n  }\n\n  private def trackResultStats(\n    stats: StatsReceiver\n  )(\n    fn: => Future[Seq[RankedCandidate]]\n  ): Future[Seq[RankedCandidate]] = {\n    fn.onSuccess { candidates =>\n      trackReasonChosenSourceTypeStats(candidates, stats)\n      trackReasonChosenSimilarityEngineStats(candidates, stats)\n      trackPotentialReasonsSourceTypeStats(candidates, stats)\n      trackPotentialReasonsSimilarityEngineStats(candidates, stats)\n    }\n  }\n\n  private def trackReasonChosenSourceTypeStats(\n    candidates: Seq[RankedCandidate],\n    stats: StatsReceiver\n  ): Unit = {\n    candidates\n      .groupBy(_.reasonChosen.sourceInfoOpt.map(_.sourceType))\n      .foreach {\n        case (sourceTypeOpt, rankedCands) =>\n          val sourceType = sourceTypeOpt.map(_.toString).getOrElse(\"RequesterId\") // default\n          stats.stat(\"reasonChosen\", \"sourceType\", sourceType, \"size\").add(rankedCands.size)\n      }\n  }\n\n  private def trackReasonChosenSimilarityEngineStats(\n    candidates: Seq[RankedCandidate],\n    stats: StatsReceiver\n  ): Unit = {\n    candidates\n      .groupBy(_.reasonChosen.similarityEngineInfo.similarityEngineType)\n      .foreach {\n        case (seInfoType, rankedCands) =>\n          stats\n            .stat(\"reasonChosen\", \"similarityEngine\", seInfoType.toString, \"size\").add(\n              rankedCands.size)\n      }\n  }\n\n  private def trackPotentialReasonsSourceTypeStats(\n    candidates: Seq[RankedCandidate],\n    stats: StatsReceiver\n  ): Unit = {\n    candidates\n      .flatMap(_.potentialReasons.map(_.sourceInfoOpt.map(_.sourceType)))\n      .groupBy(source => source)\n      .foreach {\n        case (sourceInfoOpt, seq) =>\n          val sourceType = sourceInfoOpt.map(_.toString).getOrElse(\"RequesterId\") // default\n          stats.stat(\"potentialReasons\", \"sourceType\", sourceType, \"size\").add(seq.size)\n      }\n  }\n\n  private def trackPotentialReasonsSimilarityEngineStats(\n    candidates: Seq[RankedCandidate],\n    stats: StatsReceiver\n  ): Unit = {\n    candidates\n      .flatMap(_.potentialReasons.map(_.similarityEngineInfo.similarityEngineType))\n      .groupBy(se => se)\n      .foreach {\n        case (seType, seq) =>\n          stats.stat(\"potentialReasons\", \"similarityEngine\", seType.toString, \"size\").add(seq.size)\n      }\n  }\n\n  private def trackBlueVerifiedTweetStats(\n    candidates: Seq[RankedCandidate],\n    statsReceiver: StatsReceiver\n  ): Unit = {\n    candidates.foreach { candidate =>\n      if (candidate.tweetInfo.hasBlueVerifiedAnnotation.contains(true)) {\n        statsReceiver.counter(candidate.tweetInfo.authorId.toString).incr()\n        statsReceiver\n          .scope(candidate.tweetInfo.authorId.toString).counter(candidate.tweetId.toString).incr()\n      }\n    }\n  }\n\n  private def trackTopKStats(\n    k: Int,\n    tweetCandidates: Seq[RankedCandidate],\n    isQueryK: Boolean,\n    statsReceiver: StatsReceiver\n  ): Unit = {\n    val (topK, beyondK) = tweetCandidates.splitAt(k)\n\n    val blueVerifiedIds = tweetCandidates.collect {\n      case candidate if candidate.tweetInfo.hasBlueVerifiedAnnotation.contains(true) =>\n        candidate.tweetInfo.authorId\n    }.toSet\n\n    blueVerifiedIds.foreach { blueVerifiedId =>\n      val numTweetsTopK = topK.count(_.tweetInfo.authorId == blueVerifiedId)\n      val numTweetsBeyondK = beyondK.count(_.tweetInfo.authorId == blueVerifiedId)\n\n      if (isQueryK) {\n        statsReceiver.scope(blueVerifiedId.toString).stat(s\"topK\").add(numTweetsTopK)\n        statsReceiver\n          .scope(blueVerifiedId.toString).stat(s\"beyondK\").add(numTweetsBeyondK)\n      } else {\n        statsReceiver.scope(blueVerifiedId.toString).stat(s\"top$k\").add(numTweetsTopK)\n        statsReceiver\n          .scope(blueVerifiedId.toString).stat(s\"beyond$k\").add(numTweetsBeyondK)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation/CustomizedRetrievalCandidateGeneration.scala",
    "content": "package com.twitter.cr_mixer.candidate_generation\n\nimport com.twitter.cr_mixer.candidate_generation.CustomizedRetrievalCandidateGeneration.Query\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithCandidateGenerationInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.CustomizedRetrievalBasedCandidateGenerationParams._\nimport com.twitter.cr_mixer.param.CustomizedRetrievalBasedTwhinParams._\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.similarity_engine.DiffusionBasedSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.LookupEngineQuery\nimport com.twitter.cr_mixer.similarity_engine.LookupSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TwhinCollabFilterSimilarityEngine\nimport com.twitter.cr_mixer.util.InterleaveUtil\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.Stats\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.mutable.ArrayBuffer\n\n/**\n * A candidate generator that fetches similar tweets from multiple customized retrieval based candidate sources\n *\n * Different from [[TweetBasedCandidateGeneration]], this store returns candidates from different\n * similarity engines without blending. In other words, this class shall not be thought of as a\n * Unified Similarity Engine. It is a CG that calls multiple singular Similarity Engines.\n */\n@Singleton\ncase class CustomizedRetrievalCandidateGeneration @Inject() (\n  @Named(ModuleNames.TwhinCollabFilterSimilarityEngine)\n  twhinCollabFilterSimilarityEngine: LookupSimilarityEngine[\n    TwhinCollabFilterSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  @Named(ModuleNames.DiffusionBasedSimilarityEngine)\n  diffusionBasedSimilarityEngine: LookupSimilarityEngine[\n    DiffusionBasedSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  statsReceiver: StatsReceiver)\n    extends CandidateSource[\n      Query,\n      Seq[TweetWithCandidateGenerationInfo]\n    ] {\n\n  override def name: String = this.getClass.getSimpleName\n\n  private val stats = statsReceiver.scope(name)\n  private val fetchCandidatesStat = stats.scope(\"fetchCandidates\")\n\n  /**\n   * For each Similarity Engine Model, return a list of tweet candidates\n   */\n  override def get(\n    query: Query\n  ): Future[Option[Seq[Seq[TweetWithCandidateGenerationInfo]]]] = {\n    query.internalId match {\n      case InternalId.UserId(_) =>\n        Stats.trackOption(fetchCandidatesStat) {\n          val twhinCollabFilterForFollowCandidatesFut = if (query.enableTwhinCollabFilter) {\n            twhinCollabFilterSimilarityEngine.getCandidates(query.twhinCollabFilterFollowQuery)\n          } else Future.None\n\n          val twhinCollabFilterForEngagementCandidatesFut =\n            if (query.enableTwhinCollabFilter) {\n              twhinCollabFilterSimilarityEngine.getCandidates(\n                query.twhinCollabFilterEngagementQuery)\n            } else Future.None\n\n          val twhinMultiClusterForFollowCandidatesFut = if (query.enableTwhinMultiCluster) {\n            twhinCollabFilterSimilarityEngine.getCandidates(query.twhinMultiClusterFollowQuery)\n          } else Future.None\n\n          val twhinMultiClusterForEngagementCandidatesFut =\n            if (query.enableTwhinMultiCluster) {\n              twhinCollabFilterSimilarityEngine.getCandidates(\n                query.twhinMultiClusterEngagementQuery)\n            } else Future.None\n\n          val diffusionBasedSimilarityEngineCandidatesFut = if (query.enableRetweetBasedDiffusion) {\n            diffusionBasedSimilarityEngine.getCandidates(query.diffusionBasedSimilarityEngineQuery)\n          } else Future.None\n\n          Future\n            .join(\n              twhinCollabFilterForFollowCandidatesFut,\n              twhinCollabFilterForEngagementCandidatesFut,\n              twhinMultiClusterForFollowCandidatesFut,\n              twhinMultiClusterForEngagementCandidatesFut,\n              diffusionBasedSimilarityEngineCandidatesFut\n            ).map {\n              case (\n                    twhinCollabFilterForFollowCandidates,\n                    twhinCollabFilterForEngagementCandidates,\n                    twhinMultiClusterForFollowCandidates,\n                    twhinMultiClusterForEngagementCandidates,\n                    diffusionBasedSimilarityEngineCandidates) =>\n                val maxCandidateNumPerSourceKey = 200\n                val twhinCollabFilterForFollowWithCGInfo =\n                  getTwhinCollabCandidatesWithCGInfo(\n                    twhinCollabFilterForFollowCandidates,\n                    maxCandidateNumPerSourceKey,\n                    query.twhinCollabFilterFollowQuery,\n                  )\n                val twhinCollabFilterForEngagementWithCGInfo =\n                  getTwhinCollabCandidatesWithCGInfo(\n                    twhinCollabFilterForEngagementCandidates,\n                    maxCandidateNumPerSourceKey,\n                    query.twhinCollabFilterEngagementQuery,\n                  )\n                val twhinMultiClusterForFollowWithCGInfo =\n                  getTwhinCollabCandidatesWithCGInfo(\n                    twhinMultiClusterForFollowCandidates,\n                    maxCandidateNumPerSourceKey,\n                    query.twhinMultiClusterFollowQuery,\n                  )\n                val twhinMultiClusterForEngagementWithCGInfo =\n                  getTwhinCollabCandidatesWithCGInfo(\n                    twhinMultiClusterForEngagementCandidates,\n                    maxCandidateNumPerSourceKey,\n                    query.twhinMultiClusterEngagementQuery,\n                  )\n                val retweetBasedDiffusionWithCGInfo =\n                  getDiffusionBasedCandidatesWithCGInfo(\n                    diffusionBasedSimilarityEngineCandidates,\n                    maxCandidateNumPerSourceKey,\n                    query.diffusionBasedSimilarityEngineQuery,\n                  )\n\n                val twhinCollabCandidateSourcesToBeInterleaved =\n                  ArrayBuffer[Seq[TweetWithCandidateGenerationInfo]](\n                    twhinCollabFilterForFollowWithCGInfo,\n                    twhinCollabFilterForEngagementWithCGInfo,\n                  )\n\n                val twhinMultiClusterCandidateSourcesToBeInterleaved =\n                  ArrayBuffer[Seq[TweetWithCandidateGenerationInfo]](\n                    twhinMultiClusterForFollowWithCGInfo,\n                    twhinMultiClusterForEngagementWithCGInfo,\n                  )\n\n                val interleavedTwhinCollabCandidates =\n                  InterleaveUtil.interleave(twhinCollabCandidateSourcesToBeInterleaved)\n\n                val interleavedTwhinMultiClusterCandidates =\n                  InterleaveUtil.interleave(twhinMultiClusterCandidateSourcesToBeInterleaved)\n\n                val twhinCollabFilterResults =\n                  if (interleavedTwhinCollabCandidates.nonEmpty) {\n                    Some(interleavedTwhinCollabCandidates.take(maxCandidateNumPerSourceKey))\n                  } else None\n\n                val twhinMultiClusterResults =\n                  if (interleavedTwhinMultiClusterCandidates.nonEmpty) {\n                    Some(interleavedTwhinMultiClusterCandidates.take(maxCandidateNumPerSourceKey))\n                  } else None\n\n                val diffusionResults =\n                  if (retweetBasedDiffusionWithCGInfo.nonEmpty) {\n                    Some(retweetBasedDiffusionWithCGInfo.take(maxCandidateNumPerSourceKey))\n                  } else None\n\n                Some(\n                  Seq(\n                    twhinCollabFilterResults,\n                    twhinMultiClusterResults,\n                    diffusionResults\n                  ).flatten)\n            }\n        }\n      case _ =>\n        throw new IllegalArgumentException(\"sourceId_is_not_userId_cnt\")\n    }\n  }\n\n  /** Returns a list of tweets that are generated less than `maxTweetAgeHours` hours ago */\n  private def tweetAgeFilter(\n    candidates: Seq[TweetWithScore],\n    maxTweetAgeHours: Duration\n  ): Seq[TweetWithScore] = {\n    // Tweet IDs are approximately chronological (see http://go/snowflake),\n    // so we are building the earliest tweet id once\n    // The per-candidate logic here then be candidate.tweetId > earliestPermittedTweetId, which is far cheaper.\n    val earliestTweetId = SnowflakeId.firstIdFor(Time.now - maxTweetAgeHours)\n    candidates.filter { candidate => candidate.tweetId >= earliestTweetId }\n  }\n\n  /**\n   * AgeFilters tweetCandidates with stats\n   * Only age filter logic is effective here (through tweetAgeFilter). This function acts mostly for metric logging.\n   */\n  private def ageFilterWithStats(\n    offlineInterestedInCandidates: Seq[TweetWithScore],\n    maxTweetAgeHours: Duration,\n    scopedStatsReceiver: StatsReceiver\n  ): Seq[TweetWithScore] = {\n    scopedStatsReceiver.stat(\"size\").add(offlineInterestedInCandidates.size)\n    val candidates = offlineInterestedInCandidates.map { candidate =>\n      TweetWithScore(candidate.tweetId, candidate.score)\n    }\n    val filteredCandidates = tweetAgeFilter(candidates, maxTweetAgeHours)\n    scopedStatsReceiver.stat(f\"filtered_size\").add(filteredCandidates.size)\n    if (filteredCandidates.isEmpty) scopedStatsReceiver.counter(f\"empty\").incr()\n\n    filteredCandidates\n  }\n\n  private def getTwhinCollabCandidatesWithCGInfo(\n    tweetCandidates: Option[Seq[TweetWithScore]],\n    maxCandidateNumPerSourceKey: Int,\n    twhinCollabFilterQuery: LookupEngineQuery[\n      TwhinCollabFilterSimilarityEngine.Query\n    ],\n  ): Seq[TweetWithCandidateGenerationInfo] = {\n    val twhinTweets = tweetCandidates match {\n      case Some(tweetsWithScores) =>\n        tweetsWithScores.map { tweetWithScore =>\n          TweetWithCandidateGenerationInfo(\n            tweetWithScore.tweetId,\n            CandidateGenerationInfo(\n              None,\n              TwhinCollabFilterSimilarityEngine\n                .toSimilarityEngineInfo(twhinCollabFilterQuery, tweetWithScore.score),\n              Seq.empty\n            )\n          )\n        }\n      case _ => Seq.empty\n    }\n    twhinTweets.take(maxCandidateNumPerSourceKey)\n  }\n\n  private def getDiffusionBasedCandidatesWithCGInfo(\n    tweetCandidates: Option[Seq[TweetWithScore]],\n    maxCandidateNumPerSourceKey: Int,\n    diffusionBasedSimilarityEngineQuery: LookupEngineQuery[\n      DiffusionBasedSimilarityEngine.Query\n    ],\n  ): Seq[TweetWithCandidateGenerationInfo] = {\n    val diffusionTweets = tweetCandidates match {\n      case Some(tweetsWithScores) =>\n        tweetsWithScores.map { tweetWithScore =>\n          TweetWithCandidateGenerationInfo(\n            tweetWithScore.tweetId,\n            CandidateGenerationInfo(\n              None,\n              DiffusionBasedSimilarityEngine\n                .toSimilarityEngineInfo(diffusionBasedSimilarityEngineQuery, tweetWithScore.score),\n              Seq.empty\n            )\n          )\n        }\n      case _ => Seq.empty\n    }\n    diffusionTweets.take(maxCandidateNumPerSourceKey)\n  }\n}\n\nobject CustomizedRetrievalCandidateGeneration {\n\n  case class Query(\n    internalId: InternalId,\n    maxCandidateNumPerSourceKey: Int,\n    maxTweetAgeHours: Duration,\n    // twhinCollabFilter\n    enableTwhinCollabFilter: Boolean,\n    twhinCollabFilterFollowQuery: LookupEngineQuery[\n      TwhinCollabFilterSimilarityEngine.Query\n    ],\n    twhinCollabFilterEngagementQuery: LookupEngineQuery[\n      TwhinCollabFilterSimilarityEngine.Query\n    ],\n    // twhinMultiCluster\n    enableTwhinMultiCluster: Boolean,\n    twhinMultiClusterFollowQuery: LookupEngineQuery[\n      TwhinCollabFilterSimilarityEngine.Query\n    ],\n    twhinMultiClusterEngagementQuery: LookupEngineQuery[\n      TwhinCollabFilterSimilarityEngine.Query\n    ],\n    enableRetweetBasedDiffusion: Boolean,\n    diffusionBasedSimilarityEngineQuery: LookupEngineQuery[\n      DiffusionBasedSimilarityEngine.Query\n    ],\n  )\n\n  def fromParams(\n    internalId: InternalId,\n    params: configapi.Params\n  ): Query = {\n    val twhinCollabFilterFollowQuery =\n      TwhinCollabFilterSimilarityEngine.fromParams(\n        internalId,\n        params(CustomizedRetrievalBasedTwhinCollabFilterFollowSource),\n        params)\n\n    val twhinCollabFilterEngagementQuery =\n      TwhinCollabFilterSimilarityEngine.fromParams(\n        internalId,\n        params(CustomizedRetrievalBasedTwhinCollabFilterEngagementSource),\n        params)\n\n    val twhinMultiClusterFollowQuery =\n      TwhinCollabFilterSimilarityEngine.fromParams(\n        internalId,\n        params(CustomizedRetrievalBasedTwhinMultiClusterFollowSource),\n        params)\n\n    val twhinMultiClusterEngagementQuery =\n      TwhinCollabFilterSimilarityEngine.fromParams(\n        internalId,\n        params(CustomizedRetrievalBasedTwhinMultiClusterEngagementSource),\n        params)\n\n    val diffusionBasedSimilarityEngineQuery =\n      DiffusionBasedSimilarityEngine.fromParams(\n        internalId,\n        params(CustomizedRetrievalBasedRetweetDiffusionSource),\n        params)\n\n    Query(\n      internalId = internalId,\n      maxCandidateNumPerSourceKey = params(GlobalParams.MaxCandidateNumPerSourceKeyParam),\n      maxTweetAgeHours = params(GlobalParams.MaxTweetAgeHoursParam),\n      // twhinCollabFilter\n      enableTwhinCollabFilter = params(EnableTwhinCollabFilterClusterParam),\n      twhinCollabFilterFollowQuery = twhinCollabFilterFollowQuery,\n      twhinCollabFilterEngagementQuery = twhinCollabFilterEngagementQuery,\n      enableTwhinMultiCluster = params(EnableTwhinMultiClusterParam),\n      twhinMultiClusterFollowQuery = twhinMultiClusterFollowQuery,\n      twhinMultiClusterEngagementQuery = twhinMultiClusterEngagementQuery,\n      enableRetweetBasedDiffusion = params(EnableRetweetBasedDiffusionParam),\n      diffusionBasedSimilarityEngineQuery = diffusionBasedSimilarityEngineQuery\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation/FrsTweetCandidateGenerator.scala",
    "content": "package com.twitter.cr_mixer.candidate_generation\n\nimport com.twitter.contentrecommender.thriftscala.TweetInfo\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.FrsTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithAuthor\nimport com.twitter.cr_mixer.param.FrsParams\nimport com.twitter.cr_mixer.similarity_engine.EarlybirdSimilarityEngineRouter\nimport com.twitter.cr_mixer.source_signal.FrsStore\nimport com.twitter.cr_mixer.source_signal.FrsStore.FrsQueryResult\nimport com.twitter.cr_mixer.thriftscala.FrsTweet\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.hermit.constants.AlgorithmFeedbackTokens\nimport com.twitter.hermit.constants.AlgorithmFeedbackTokens.AlgorithmToFeedbackTokenMap\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/**\n * TweetCandidateGenerator based on FRS seed users. For now this candidate generator fetches seed\n * users from FRS, and retrieves the seed users' past tweets from Earlybird with Earlybird light\n * ranking models.\n */\n@Singleton\nclass FrsTweetCandidateGenerator @Inject() (\n  @Named(ModuleNames.FrsStore) frsStore: ReadableStore[FrsStore.Query, Seq[FrsQueryResult]],\n  frsBasedSimilarityEngine: EarlybirdSimilarityEngineRouter,\n  tweetInfoStore: ReadableStore[TweetId, TweetInfo],\n  timeoutConfig: TimeoutConfig,\n  globalStats: StatsReceiver) {\n  import FrsTweetCandidateGenerator._\n\n  private val timer = DefaultTimer\n  private val stats: StatsReceiver = globalStats.scope(this.getClass.getCanonicalName)\n  private val fetchSeedsStats = stats.scope(\"fetchSeeds\")\n  private val fetchCandidatesStats = stats.scope(\"fetchCandidates\")\n  private val filterCandidatesStats = stats.scope(\"filterCandidates\")\n  private val hydrateCandidatesStats = stats.scope(\"hydrateCandidates\")\n  private val getCandidatesStats = stats.scope(\"getCandidates\")\n\n  /**\n   * The function retrieves the candidate for the given user as follows:\n   * 1. Seed user fetch from FRS.\n   * 2. Candidate fetch from Earlybird.\n   * 3. Filtering.\n   * 4. Candidate hydration.\n   * 5. Truncation.\n   */\n  def get(\n    frsTweetCandidateGeneratorQuery: FrsTweetCandidateGeneratorQuery\n  ): Future[Seq[FrsTweet]] = {\n    val userId = frsTweetCandidateGeneratorQuery.userId\n    val product = frsTweetCandidateGeneratorQuery.product\n    val allStats = stats.scope(\"all\")\n    val perProductStats = stats.scope(\"perProduct\", product.name)\n    StatsUtil.trackItemsStats(allStats) {\n      StatsUtil.trackItemsStats(perProductStats) {\n        val result = for {\n          seedAuthorWithScores <- StatsUtil.trackOptionItemMapStats(fetchSeedsStats) {\n            fetchSeeds(\n              userId,\n              frsTweetCandidateGeneratorQuery.impressedUserList,\n              frsTweetCandidateGeneratorQuery.languageCodeOpt,\n              frsTweetCandidateGeneratorQuery.countryCodeOpt,\n              frsTweetCandidateGeneratorQuery.params,\n            )\n          }\n          tweetCandidates <- StatsUtil.trackOptionItemsStats(fetchCandidatesStats) {\n            fetchCandidates(\n              userId,\n              seedAuthorWithScores.map(_.keys.toSeq).getOrElse(Seq.empty),\n              frsTweetCandidateGeneratorQuery.impressedTweetList,\n              seedAuthorWithScores.map(_.mapValues(_.score)).getOrElse(Map.empty),\n              frsTweetCandidateGeneratorQuery.params\n            )\n          }\n          filteredTweetCandidates <- StatsUtil.trackOptionItemsStats(filterCandidatesStats) {\n            filterCandidates(\n              tweetCandidates,\n              frsTweetCandidateGeneratorQuery.params\n            )\n          }\n          hydratedTweetCandidates <- StatsUtil.trackOptionItemsStats(hydrateCandidatesStats) {\n            hydrateCandidates(\n              seedAuthorWithScores,\n              filteredTweetCandidates\n            )\n          }\n        } yield {\n          hydratedTweetCandidates\n            .map(_.take(frsTweetCandidateGeneratorQuery.maxNumResults)).getOrElse(Seq.empty)\n        }\n        result.raiseWithin(timeoutConfig.frsBasedTweetEndpointTimeout)(timer)\n      }\n    }\n  }\n\n  /**\n   * Fetch recommended seed users from FRS\n   */\n  private def fetchSeeds(\n    userId: UserId,\n    userDenyList: Set[UserId],\n    languageCodeOpt: Option[String],\n    countryCodeOpt: Option[String],\n    params: Params\n  ): Future[Option[Map[UserId, FrsQueryResult]]] = {\n    frsStore\n      .get(\n        FrsStore.Query(\n          userId,\n          params(FrsParams.FrsBasedCandidateGenerationMaxSeedsNumParam),\n          params(FrsParams.FrsBasedCandidateGenerationDisplayLocationParam).displayLocation,\n          userDenyList.toSeq,\n          languageCodeOpt,\n          countryCodeOpt\n        )).map {\n        _.map { seedAuthors =>\n          seedAuthors.map(user => user.userId -> user).toMap\n        }\n      }\n  }\n\n  /**\n   * Fetch tweet candidates from Earlybird\n   */\n  private def fetchCandidates(\n    searcherUserId: UserId,\n    seedAuthors: Seq[UserId],\n    impressedTweetList: Set[TweetId],\n    frsUserToScores: Map[UserId, Double],\n    params: Params\n  ): Future[Option[Seq[TweetWithAuthor]]] = {\n    if (seedAuthors.nonEmpty) {\n      // call earlybird\n      val query = EarlybirdSimilarityEngineRouter.queryFromParams(\n        Some(searcherUserId),\n        seedAuthors,\n        impressedTweetList,\n        frsUserToScoresForScoreAdjustment = Some(frsUserToScores),\n        params\n      )\n      frsBasedSimilarityEngine.get(query)\n    } else Future.None\n  }\n\n  /**\n   * Filter candidates that do not pass visibility filter policy\n   */\n  private def filterCandidates(\n    candidates: Option[Seq[TweetWithAuthor]],\n    params: Params\n  ): Future[Option[Seq[TweetWithAuthor]]] = {\n    val tweetIds = candidates.map(_.map(_.tweetId).toSet).getOrElse(Set.empty)\n    if (params(FrsParams.FrsBasedCandidateGenerationEnableVisibilityFilteringParam))\n      Future\n        .collect(tweetInfoStore.multiGet(tweetIds)).map { tweetInfos =>\n          candidates.map {\n            // If tweetInfo does not exist, we will filter out this tweet candidate.\n            _.filter(candidate => tweetInfos.getOrElse(candidate.tweetId, None).isDefined)\n          }\n        }\n    else {\n      Future.value(candidates)\n    }\n  }\n\n  /**\n   * Hydrate the candidates with the FRS candidate sources and scores\n   */\n  private def hydrateCandidates(\n    frsAuthorWithScores: Option[Map[UserId, FrsQueryResult]],\n    candidates: Option[Seq[TweetWithAuthor]]\n  ): Future[Option[Seq[FrsTweet]]] = {\n    Future.value {\n      candidates.map {\n        _.map { tweetWithAuthor =>\n          val frsQueryResult = frsAuthorWithScores.flatMap(_.get(tweetWithAuthor.authorId))\n          FrsTweet(\n            tweetId = tweetWithAuthor.tweetId,\n            authorId = tweetWithAuthor.authorId,\n            frsPrimarySource = frsQueryResult.flatMap(_.primarySource),\n            frsAuthorScore = frsQueryResult.map(_.score),\n            frsCandidateSourceScores = frsQueryResult.flatMap { result =>\n              result.sourceWithScores.map {\n                _.collect {\n                  // see TokenStrToAlgorithmMap @ https://sourcegraph.twitter.biz/git.twitter.biz/source/-/blob/hermit/hermit-core/src/main/scala/com/twitter/hermit/constants/AlgorithmFeedbackTokens.scala\n                  // see Algorithm @ https://sourcegraph.twitter.biz/git.twitter.biz/source/-/blob/hermit/hermit-core/src/main/scala/com/twitter/hermit/model/Algorithm.scala\n                  case (candidateSourceAlgoStr, score)\n                      if AlgorithmFeedbackTokens.TokenStrToAlgorithmMap.contains(\n                        candidateSourceAlgoStr) =>\n                    AlgorithmToFeedbackTokenMap.getOrElse(\n                      AlgorithmFeedbackTokens.TokenStrToAlgorithmMap\n                        .getOrElse(candidateSourceAlgoStr, DefaultAlgo),\n                      DefaultAlgoToken) -> score\n                }\n              }\n            }\n          )\n        }\n      }\n    }\n  }\n\n}\n\nobject FrsTweetCandidateGenerator {\n  val DefaultAlgo: Algorithm.Value = Algorithm.Other\n  // 9999 is the token for Algorithm.Other\n  val DefaultAlgoToken: Int = AlgorithmToFeedbackTokenMap.getOrElse(DefaultAlgo, 9999)\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation/RelatedTweetCandidateGenerator.scala",
    "content": "package com.twitter.cr_mixer.candidate_generation\n\nimport com.twitter.contentrecommender.thriftscala.TweetInfo\nimport com.twitter.cr_mixer.filter.PreRankFilterRunner\nimport com.twitter.cr_mixer.logging.RelatedTweetScribeLogger\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.model.RelatedTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.TweetWithCandidateGenerationInfo\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.similarity_engine.ProducerBasedUnifiedSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TweetBasedUnifiedSimilarityEngine\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass RelatedTweetCandidateGenerator @Inject() (\n  @Named(ModuleNames.TweetBasedUnifiedSimilarityEngine) tweetBasedUnifiedSimilarityEngine: StandardSimilarityEngine[\n    TweetBasedUnifiedSimilarityEngine.Query,\n    TweetWithCandidateGenerationInfo\n  ],\n  @Named(ModuleNames.ProducerBasedUnifiedSimilarityEngine) producerBasedUnifiedSimilarityEngine: StandardSimilarityEngine[\n    ProducerBasedUnifiedSimilarityEngine.Query,\n    TweetWithCandidateGenerationInfo\n  ],\n  preRankFilterRunner: PreRankFilterRunner,\n  relatedTweetScribeLogger: RelatedTweetScribeLogger,\n  tweetInfoStore: ReadableStore[TweetId, TweetInfo],\n  globalStats: StatsReceiver) {\n\n  private val stats: StatsReceiver = globalStats.scope(this.getClass.getCanonicalName)\n  private val fetchCandidatesStats = stats.scope(\"fetchCandidates\")\n  private val preRankFilterStats = stats.scope(\"preRankFilter\")\n\n  def get(\n    query: RelatedTweetCandidateGeneratorQuery\n  ): Future[Seq[InitialCandidate]] = {\n\n    val allStats = stats.scope(\"all\")\n    val perProductStats = stats.scope(\"perProduct\", query.product.toString)\n    StatsUtil.trackItemsStats(allStats) {\n      StatsUtil.trackItemsStats(perProductStats) {\n        for {\n          initialCandidates <- StatsUtil.trackBlockStats(fetchCandidatesStats) {\n            fetchCandidates(query)\n          }\n          filteredCandidates <- StatsUtil.trackBlockStats(preRankFilterStats) {\n            preRankFilter(query, initialCandidates)\n          }\n        } yield {\n          filteredCandidates.headOption\n            .getOrElse(\n              throw new UnsupportedOperationException(\n                \"RelatedTweetCandidateGenerator results invalid\")\n            ).take(query.maxNumResults)\n        }\n      }\n    }\n  }\n\n  def fetchCandidates(\n    query: RelatedTweetCandidateGeneratorQuery\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    relatedTweetScribeLogger.scribeInitialCandidates(\n      query,\n      query.internalId match {\n        case InternalId.TweetId(_) =>\n          getCandidatesFromSimilarityEngine(\n            query,\n            TweetBasedUnifiedSimilarityEngine.fromParamsForRelatedTweet,\n            tweetBasedUnifiedSimilarityEngine.getCandidates)\n        case InternalId.UserId(_) =>\n          getCandidatesFromSimilarityEngine(\n            query,\n            ProducerBasedUnifiedSimilarityEngine.fromParamsForRelatedTweet,\n            producerBasedUnifiedSimilarityEngine.getCandidates)\n        case _ =>\n          throw new UnsupportedOperationException(\n            \"RelatedTweetCandidateGenerator gets invalid InternalId\")\n      }\n    )\n  }\n\n  /***\n   * fetch Candidates from TweetBased/ProducerBased Unified Similarity Engine,\n   * and apply VF filter based on TweetInfoStore\n   * To align with the downstream processing (filter, rank), we tend to return a Seq[Seq[InitialCandidate]]\n   * instead of a Seq[Candidate] even though we only have a Seq in it.\n   */\n  private def getCandidatesFromSimilarityEngine[QueryType](\n    query: RelatedTweetCandidateGeneratorQuery,\n    fromParamsForRelatedTweet: (InternalId, configapi.Params) => QueryType,\n    getFunc: QueryType => Future[Option[Seq[TweetWithCandidateGenerationInfo]]]\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n\n    /***\n     * We wrap the query to be a Seq of queries for the Sim Engine to ensure evolvability of candidate generation\n     * and as a result, it will return Seq[Seq[InitialCandidate]]\n     */\n    val engineQueries =\n      Seq(fromParamsForRelatedTweet(query.internalId, query.params))\n\n    Future\n      .collect {\n        engineQueries.map { query =>\n          for {\n            candidates <- getFunc(query)\n            prefilterCandidates <- convertToInitialCandidates(\n              candidates.toSeq.flatten\n            )\n          } yield prefilterCandidates\n        }\n      }\n  }\n\n  private def preRankFilter(\n    query: RelatedTweetCandidateGeneratorQuery,\n    candidates: Seq[Seq[InitialCandidate]]\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    relatedTweetScribeLogger.scribePreRankFilterCandidates(\n      query,\n      preRankFilterRunner\n        .runSequentialFilters(query, candidates))\n  }\n\n  private[candidate_generation] def convertToInitialCandidates(\n    candidates: Seq[TweetWithCandidateGenerationInfo],\n  ): Future[Seq[InitialCandidate]] = {\n    val tweetIds = candidates.map(_.tweetId).toSet\n    Future.collect(tweetInfoStore.multiGet(tweetIds)).map { tweetInfos =>\n      /***\n       * If tweetInfo does not exist, we will filter out this tweet candidate.\n       * This tweetInfo filter also acts as the VF filter\n       */\n      candidates.collect {\n        case candidate if tweetInfos.getOrElse(candidate.tweetId, None).isDefined =>\n          val tweetInfo = tweetInfos(candidate.tweetId)\n            .getOrElse(throw new IllegalStateException(\"Check previous line's condition\"))\n\n          InitialCandidate(\n            tweetId = candidate.tweetId,\n            tweetInfo = tweetInfo,\n            candidate.candidateGenerationInfo\n          )\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation/RelatedVideoTweetCandidateGenerator.scala",
    "content": "package com.twitter.cr_mixer.candidate_generation\n\nimport com.twitter.contentrecommender.thriftscala.TweetInfo\nimport com.twitter.cr_mixer.filter.PreRankFilterRunner\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.model.RelatedVideoTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.TweetWithCandidateGenerationInfo\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TweetBasedUnifiedSimilarityEngine\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass RelatedVideoTweetCandidateGenerator @Inject() (\n  @Named(ModuleNames.TweetBasedUnifiedSimilarityEngine) tweetBasedUnifiedSimilarityEngine: StandardSimilarityEngine[\n    TweetBasedUnifiedSimilarityEngine.Query,\n    TweetWithCandidateGenerationInfo\n  ],\n  preRankFilterRunner: PreRankFilterRunner,\n  tweetInfoStore: ReadableStore[TweetId, TweetInfo],\n  globalStats: StatsReceiver) {\n\n  private val stats: StatsReceiver = globalStats.scope(this.getClass.getCanonicalName)\n  private val fetchCandidatesStats = stats.scope(\"fetchCandidates\")\n  private val preRankFilterStats = stats.scope(\"preRankFilter\")\n\n  def get(\n    query: RelatedVideoTweetCandidateGeneratorQuery\n  ): Future[Seq[InitialCandidate]] = {\n\n    val allStats = stats.scope(\"all\")\n    val perProductStats = stats.scope(\"perProduct\", query.product.toString)\n    StatsUtil.trackItemsStats(allStats) {\n      StatsUtil.trackItemsStats(perProductStats) {\n        for {\n          initialCandidates <- StatsUtil.trackBlockStats(fetchCandidatesStats) {\n            fetchCandidates(query)\n          }\n          filteredCandidates <- StatsUtil.trackBlockStats(preRankFilterStats) {\n            preRankFilter(query, initialCandidates)\n          }\n        } yield {\n          filteredCandidates.headOption\n            .getOrElse(\n              throw new UnsupportedOperationException(\n                \"RelatedVideoTweetCandidateGenerator results invalid\")\n            ).take(query.maxNumResults)\n        }\n      }\n    }\n  }\n\n  def fetchCandidates(\n    query: RelatedVideoTweetCandidateGeneratorQuery\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    query.internalId match {\n      case InternalId.TweetId(_) =>\n        getCandidatesFromSimilarityEngine(\n          query,\n          TweetBasedUnifiedSimilarityEngine.fromParamsForRelatedVideoTweet,\n          tweetBasedUnifiedSimilarityEngine.getCandidates)\n      case _ =>\n        throw new UnsupportedOperationException(\n          \"RelatedVideoTweetCandidateGenerator gets invalid InternalId\")\n    }\n  }\n\n  /***\n   * fetch Candidates from TweetBased/ProducerBased Unified Similarity Engine,\n   * and apply VF filter based on TweetInfoStore\n   * To align with the downstream processing (filter, rank), we tend to return a Seq[Seq[InitialCandidate]]\n   * instead of a Seq[Candidate] even though we only have a Seq in it.\n   */\n  private def getCandidatesFromSimilarityEngine[QueryType](\n    query: RelatedVideoTweetCandidateGeneratorQuery,\n    fromParamsForRelatedVideoTweet: (InternalId, configapi.Params) => QueryType,\n    getFunc: QueryType => Future[Option[Seq[TweetWithCandidateGenerationInfo]]]\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n\n    /***\n     * We wrap the query to be a Seq of queries for the Sim Engine to ensure evolvability of candidate generation\n     * and as a result, it will return Seq[Seq[InitialCandidate]]\n     */\n    val engineQueries =\n      Seq(fromParamsForRelatedVideoTweet(query.internalId, query.params))\n\n    Future\n      .collect {\n        engineQueries.map { query =>\n          for {\n            candidates <- getFunc(query)\n            prefilterCandidates <- convertToInitialCandidates(\n              candidates.toSeq.flatten\n            )\n          } yield prefilterCandidates\n        }\n      }\n  }\n\n  private def preRankFilter(\n    query: RelatedVideoTweetCandidateGeneratorQuery,\n    candidates: Seq[Seq[InitialCandidate]]\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    preRankFilterRunner\n      .runSequentialFilters(query, candidates)\n  }\n\n  private[candidate_generation] def convertToInitialCandidates(\n    candidates: Seq[TweetWithCandidateGenerationInfo],\n  ): Future[Seq[InitialCandidate]] = {\n    val tweetIds = candidates.map(_.tweetId).toSet\n    Future.collect(tweetInfoStore.multiGet(tweetIds)).map { tweetInfos =>\n      /***\n       * If tweetInfo does not exist, we will filter out this tweet candidate.\n       * This tweetInfo filter also acts as the VF filter\n       */\n      candidates.collect {\n        case candidate if tweetInfos.getOrElse(candidate.tweetId, None).isDefined =>\n          val tweetInfo = tweetInfos(candidate.tweetId)\n            .getOrElse(throw new IllegalStateException(\"Check previous line's condition\"))\n\n          InitialCandidate(\n            tweetId = candidate.tweetId,\n            tweetInfo = tweetInfo,\n            candidate.candidateGenerationInfo\n          )\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation/SimClustersInterestedInCandidateGeneration.scala",
    "content": "package com.twitter.cr_mixer.candidate_generation\n\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.TweetWithCandidateGenerationInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.param.InterestedInParams\nimport com.twitter.cr_mixer.param.SimClustersANNParams\nimport com.twitter.cr_mixer.similarity_engine.EngineQuery\nimport com.twitter.cr_mixer.similarity_engine.SimClustersANNSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.common.ModelVersions\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport javax.inject.Named\nimport com.twitter.cr_mixer.model.ModuleNames\n\n/**\n * This store looks for similar tweets for a given UserId that generates UserInterestedIn\n * from SimClustersANN. It will be a standalone CandidateGeneration class moving forward.\n *\n * After the abstraction improvement (apply SimilarityEngine trait)\n * these CG will be subjected to change.\n */\n@Singleton\ncase class SimClustersInterestedInCandidateGeneration @Inject() (\n  @Named(ModuleNames.SimClustersANNSimilarityEngine)\n  simClustersANNSimilarityEngine: StandardSimilarityEngine[\n    SimClustersANNSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  statsReceiver: StatsReceiver)\n    extends CandidateSource[\n      SimClustersInterestedInCandidateGeneration.Query,\n      Seq[TweetWithCandidateGenerationInfo]\n    ] {\n\n  override def name: String = this.getClass.getSimpleName\n  private val stats = statsReceiver.scope(name)\n  private val fetchCandidatesStat = stats.scope(\"fetchCandidates\")\n\n  override def get(\n    query: SimClustersInterestedInCandidateGeneration.Query\n  ): Future[Option[Seq[Seq[TweetWithCandidateGenerationInfo]]]] = {\n\n    query.internalId match {\n      case _: InternalId.UserId =>\n        StatsUtil.trackOptionItemsStats(fetchCandidatesStat) {\n          // UserInterestedIn Queries\n          val userInterestedInCandidateResultFut =\n            if (query.enableUserInterestedIn && query.enableProdSimClustersANNSimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.interestedInSimClustersANNQuery,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userInterestedInExperimentalSANNCandidateResultFut =\n            if (query.enableUserInterestedIn && query.enableExperimentalSimClustersANNSimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.interestedInExperimentalSimClustersANNQuery,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userInterestedInSANN1CandidateResultFut =\n            if (query.enableUserInterestedIn && query.enableSimClustersANN1SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.interestedInSimClustersANN1Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userInterestedInSANN2CandidateResultFut =\n            if (query.enableUserInterestedIn && query.enableSimClustersANN2SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.interestedInSimClustersANN2Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userInterestedInSANN3CandidateResultFut =\n            if (query.enableUserInterestedIn && query.enableSimClustersANN3SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.interestedInSimClustersANN3Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userInterestedInSANN5CandidateResultFut =\n            if (query.enableUserInterestedIn && query.enableSimClustersANN5SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.interestedInSimClustersANN5Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userInterestedInSANN4CandidateResultFut =\n            if (query.enableUserInterestedIn && query.enableSimClustersANN4SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.interestedInSimClustersANN4Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n          // UserNextInterestedIn Queries\n          val userNextInterestedInCandidateResultFut =\n            if (query.enableUserNextInterestedIn && query.enableProdSimClustersANNSimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.nextInterestedInSimClustersANNQuery,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userNextInterestedInExperimentalSANNCandidateResultFut =\n            if (query.enableUserNextInterestedIn && query.enableExperimentalSimClustersANNSimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.nextInterestedInExperimentalSimClustersANNQuery,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userNextInterestedInSANN1CandidateResultFut =\n            if (query.enableUserNextInterestedIn && query.enableSimClustersANN1SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.nextInterestedInSimClustersANN1Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userNextInterestedInSANN2CandidateResultFut =\n            if (query.enableUserNextInterestedIn && query.enableSimClustersANN2SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.nextInterestedInSimClustersANN2Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userNextInterestedInSANN3CandidateResultFut =\n            if (query.enableUserNextInterestedIn && query.enableSimClustersANN3SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.nextInterestedInSimClustersANN3Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userNextInterestedInSANN5CandidateResultFut =\n            if (query.enableUserNextInterestedIn && query.enableSimClustersANN5SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.nextInterestedInSimClustersANN5Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userNextInterestedInSANN4CandidateResultFut =\n            if (query.enableUserNextInterestedIn && query.enableSimClustersANN4SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.nextInterestedInSimClustersANN4Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          // AddressBookInterestedIn Queries\n          val userAddressBookInterestedInCandidateResultFut =\n            if (query.enableAddressBookNextInterestedIn && query.enableProdSimClustersANNSimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.addressbookInterestedInSimClustersANNQuery,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userAddressBookExperimentalSANNCandidateResultFut =\n            if (query.enableAddressBookNextInterestedIn && query.enableExperimentalSimClustersANNSimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.addressbookInterestedInExperimentalSimClustersANNQuery,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userAddressBookSANN1CandidateResultFut =\n            if (query.enableAddressBookNextInterestedIn && query.enableSimClustersANN1SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.addressbookInterestedInSimClustersANN1Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userAddressBookSANN2CandidateResultFut =\n            if (query.enableAddressBookNextInterestedIn && query.enableSimClustersANN2SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.addressbookInterestedInSimClustersANN2Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userAddressBookSANN3CandidateResultFut =\n            if (query.enableAddressBookNextInterestedIn && query.enableSimClustersANN3SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.addressbookInterestedInSimClustersANN3Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userAddressBookSANN5CandidateResultFut =\n            if (query.enableAddressBookNextInterestedIn && query.enableSimClustersANN5SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.addressbookInterestedInSimClustersANN5Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          val userAddressBookSANN4CandidateResultFut =\n            if (query.enableAddressBookNextInterestedIn && query.enableSimClustersANN4SimilarityEngine)\n              getInterestedInCandidateResult(\n                simClustersANNSimilarityEngine,\n                query.addressbookInterestedInSimClustersANN4Query,\n                query.simClustersInterestedInMinScore)\n            else\n              Future.None\n\n          Future\n            .collect(\n              Seq(\n                userInterestedInCandidateResultFut,\n                userNextInterestedInCandidateResultFut,\n                userAddressBookInterestedInCandidateResultFut,\n                userInterestedInExperimentalSANNCandidateResultFut,\n                userNextInterestedInExperimentalSANNCandidateResultFut,\n                userAddressBookExperimentalSANNCandidateResultFut,\n                userInterestedInSANN1CandidateResultFut,\n                userNextInterestedInSANN1CandidateResultFut,\n                userAddressBookSANN1CandidateResultFut,\n                userInterestedInSANN2CandidateResultFut,\n                userNextInterestedInSANN2CandidateResultFut,\n                userAddressBookSANN2CandidateResultFut,\n                userInterestedInSANN3CandidateResultFut,\n                userNextInterestedInSANN3CandidateResultFut,\n                userAddressBookSANN3CandidateResultFut,\n                userInterestedInSANN5CandidateResultFut,\n                userNextInterestedInSANN5CandidateResultFut,\n                userAddressBookSANN5CandidateResultFut,\n                userInterestedInSANN4CandidateResultFut,\n                userNextInterestedInSANN4CandidateResultFut,\n                userAddressBookSANN4CandidateResultFut\n              )\n            ).map { candidateResults =>\n              Some(\n                candidateResults.map(candidateResult => candidateResult.getOrElse(Seq.empty))\n              )\n            }\n        }\n      case _ =>\n        stats.counter(\"sourceId_is_not_userId_cnt\").incr()\n        Future.None\n    }\n  }\n\n  private def simClustersCandidateMinScoreFilter(\n    simClustersAnnCandidates: Seq[TweetWithScore],\n    simClustersInterestedInMinScore: Double,\n    simClustersANNConfigId: String\n  ): Seq[TweetWithScore] = {\n    val filteredCandidates = simClustersAnnCandidates\n      .filter { candidate =>\n        candidate.score > simClustersInterestedInMinScore\n      }\n\n    stats.stat(simClustersANNConfigId, \"simClustersAnnCandidates_size\").add(filteredCandidates.size)\n    stats.counter(simClustersANNConfigId, \"simClustersAnnRequests\").incr()\n    if (filteredCandidates.isEmpty)\n      stats.counter(simClustersANNConfigId, \"emptyFilteredSimClustersAnnCandidates\").incr()\n\n    filteredCandidates.map { candidate =>\n      TweetWithScore(candidate.tweetId, candidate.score)\n    }\n  }\n\n  private def getInterestedInCandidateResult(\n    simClustersANNSimilarityEngine: StandardSimilarityEngine[\n      SimClustersANNSimilarityEngine.Query,\n      TweetWithScore\n    ],\n    simClustersANNQuery: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    simClustersInterestedInMinScore: Double,\n  ): Future[Option[Seq[TweetWithCandidateGenerationInfo]]] = {\n    val interestedInCandidatesFut =\n      simClustersANNSimilarityEngine.getCandidates(simClustersANNQuery)\n\n    val interestedInCandidateResultFut = interestedInCandidatesFut.map { interestedInCandidates =>\n      stats.stat(\"candidateSize\").add(interestedInCandidates.size)\n\n      val embeddingCandidatesStat = stats.scope(\n        simClustersANNQuery.storeQuery.simClustersANNQuery.sourceEmbeddingId.embeddingType.name)\n\n      embeddingCandidatesStat.stat(\"candidateSize\").add(interestedInCandidates.size)\n      if (interestedInCandidates.isEmpty) {\n        embeddingCandidatesStat.counter(\"empty_results\").incr()\n      }\n      embeddingCandidatesStat.counter(\"requests\").incr()\n\n      val filteredTweets = simClustersCandidateMinScoreFilter(\n        interestedInCandidates.toSeq.flatten,\n        simClustersInterestedInMinScore,\n        simClustersANNQuery.storeQuery.simClustersANNConfigId)\n\n      val interestedInTweetsWithCGInfo = filteredTweets.map { tweetWithScore =>\n        TweetWithCandidateGenerationInfo(\n          tweetWithScore.tweetId,\n          CandidateGenerationInfo(\n            None,\n            SimClustersANNSimilarityEngine\n              .toSimilarityEngineInfo(simClustersANNQuery, tweetWithScore.score),\n            Seq.empty // SANN is an atomic SE, and hence it has no contributing SEs\n          )\n        )\n      }\n\n      val interestedInResults = if (interestedInTweetsWithCGInfo.nonEmpty) {\n        Some(interestedInTweetsWithCGInfo)\n      } else None\n      interestedInResults\n    }\n    interestedInCandidateResultFut\n  }\n}\n\nobject SimClustersInterestedInCandidateGeneration {\n\n  case class Query(\n    internalId: InternalId,\n    enableUserInterestedIn: Boolean,\n    enableUserNextInterestedIn: Boolean,\n    enableAddressBookNextInterestedIn: Boolean,\n    enableProdSimClustersANNSimilarityEngine: Boolean,\n    enableExperimentalSimClustersANNSimilarityEngine: Boolean,\n    enableSimClustersANN1SimilarityEngine: Boolean,\n    enableSimClustersANN2SimilarityEngine: Boolean,\n    enableSimClustersANN3SimilarityEngine: Boolean,\n    enableSimClustersANN5SimilarityEngine: Boolean,\n    enableSimClustersANN4SimilarityEngine: Boolean,\n    simClustersInterestedInMinScore: Double,\n    simClustersNextInterestedInMinScore: Double,\n    simClustersAddressBookInterestedInMinScore: Double,\n    interestedInSimClustersANNQuery: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    nextInterestedInSimClustersANNQuery: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    addressbookInterestedInSimClustersANNQuery: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    interestedInExperimentalSimClustersANNQuery: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    nextInterestedInExperimentalSimClustersANNQuery: EngineQuery[\n      SimClustersANNSimilarityEngine.Query\n    ],\n    addressbookInterestedInExperimentalSimClustersANNQuery: EngineQuery[\n      SimClustersANNSimilarityEngine.Query\n    ],\n    interestedInSimClustersANN1Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    nextInterestedInSimClustersANN1Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    addressbookInterestedInSimClustersANN1Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    interestedInSimClustersANN2Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    nextInterestedInSimClustersANN2Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    addressbookInterestedInSimClustersANN2Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    interestedInSimClustersANN3Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    nextInterestedInSimClustersANN3Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    addressbookInterestedInSimClustersANN3Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    interestedInSimClustersANN5Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    nextInterestedInSimClustersANN5Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    addressbookInterestedInSimClustersANN5Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    interestedInSimClustersANN4Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    nextInterestedInSimClustersANN4Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    addressbookInterestedInSimClustersANN4Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n  )\n\n  def fromParams(\n    internalId: InternalId,\n    params: configapi.Params,\n  ): Query = {\n    // SimClusters common configs\n    val simClustersModelVersion =\n      ModelVersions.Enum.enumToSimClustersModelVersionMap(params(GlobalParams.ModelVersionParam))\n    val simClustersANNConfigId = params(SimClustersANNParams.SimClustersANNConfigId)\n    val experimentalSimClustersANNConfigId = params(\n      SimClustersANNParams.ExperimentalSimClustersANNConfigId)\n    val simClustersANN1ConfigId = params(SimClustersANNParams.SimClustersANN1ConfigId)\n    val simClustersANN2ConfigId = params(SimClustersANNParams.SimClustersANN2ConfigId)\n    val simClustersANN3ConfigId = params(SimClustersANNParams.SimClustersANN3ConfigId)\n    val simClustersANN5ConfigId = params(SimClustersANNParams.SimClustersANN5ConfigId)\n    val simClustersANN4ConfigId = params(SimClustersANNParams.SimClustersANN4ConfigId)\n\n    val simClustersInterestedInMinScore = params(InterestedInParams.MinScoreParam)\n    val simClustersNextInterestedInMinScore = params(\n      InterestedInParams.MinScoreSequentialModelParam)\n    val simClustersAddressBookInterestedInMinScore = params(\n      InterestedInParams.MinScoreAddressBookParam)\n\n    // InterestedIn embeddings parameters\n    val interestedInEmbedding = params(InterestedInParams.InterestedInEmbeddingIdParam)\n    val nextInterestedInEmbedding = params(InterestedInParams.NextInterestedInEmbeddingIdParam)\n    val addressbookInterestedInEmbedding = params(\n      InterestedInParams.AddressBookInterestedInEmbeddingIdParam)\n\n    // Prod SimClustersANN Query\n    val interestedInSimClustersANNQuery =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        interestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANNConfigId,\n        params)\n\n    val nextInterestedInSimClustersANNQuery =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        nextInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANNConfigId,\n        params)\n\n    val addressbookInterestedInSimClustersANNQuery =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        addressbookInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANNConfigId,\n        params)\n\n    // Experimental SANN cluster Query\n    val interestedInExperimentalSimClustersANNQuery =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        interestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        experimentalSimClustersANNConfigId,\n        params)\n\n    val nextInterestedInExperimentalSimClustersANNQuery =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        nextInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        experimentalSimClustersANNConfigId,\n        params)\n\n    val addressbookInterestedInExperimentalSimClustersANNQuery =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        addressbookInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        experimentalSimClustersANNConfigId,\n        params)\n\n    // SimClusters ANN cluster 1 Query\n    val interestedInSimClustersANN1Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        interestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN1ConfigId,\n        params)\n\n    val nextInterestedInSimClustersANN1Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        nextInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN1ConfigId,\n        params)\n\n    val addressbookInterestedInSimClustersANN1Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        addressbookInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN1ConfigId,\n        params)\n\n    // SimClusters ANN cluster 2 Query\n    val interestedInSimClustersANN2Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        interestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN2ConfigId,\n        params)\n\n    val nextInterestedInSimClustersANN2Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        nextInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN2ConfigId,\n        params)\n\n    val addressbookInterestedInSimClustersANN2Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        addressbookInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN2ConfigId,\n        params)\n\n    // SimClusters ANN cluster 3 Query\n    val interestedInSimClustersANN3Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        interestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN3ConfigId,\n        params)\n\n    val nextInterestedInSimClustersANN3Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        nextInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN3ConfigId,\n        params)\n\n    val addressbookInterestedInSimClustersANN3Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        addressbookInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN3ConfigId,\n        params)\n\n    // SimClusters ANN cluster 5 Query\n    val interestedInSimClustersANN5Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        interestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN5ConfigId,\n        params)\n    // SimClusters ANN cluster 4 Query\n    val interestedInSimClustersANN4Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        interestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN4ConfigId,\n        params)\n\n    val nextInterestedInSimClustersANN5Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        nextInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN5ConfigId,\n        params)\n\n    val nextInterestedInSimClustersANN4Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        nextInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN4ConfigId,\n        params)\n\n    val addressbookInterestedInSimClustersANN5Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        addressbookInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN5ConfigId,\n        params)\n\n    val addressbookInterestedInSimClustersANN4Query =\n      SimClustersANNSimilarityEngine.fromParams(\n        internalId,\n        addressbookInterestedInEmbedding.embeddingType,\n        simClustersModelVersion,\n        simClustersANN4ConfigId,\n        params)\n\n    Query(\n      internalId = internalId,\n      enableUserInterestedIn = params(InterestedInParams.EnableSourceParam),\n      enableUserNextInterestedIn = params(InterestedInParams.EnableSourceSequentialModelParam),\n      enableAddressBookNextInterestedIn = params(InterestedInParams.EnableSourceAddressBookParam),\n      enableProdSimClustersANNSimilarityEngine =\n        params(InterestedInParams.EnableProdSimClustersANNParam),\n      enableExperimentalSimClustersANNSimilarityEngine =\n        params(InterestedInParams.EnableExperimentalSimClustersANNParam),\n      enableSimClustersANN1SimilarityEngine = params(InterestedInParams.EnableSimClustersANN1Param),\n      enableSimClustersANN2SimilarityEngine = params(InterestedInParams.EnableSimClustersANN2Param),\n      enableSimClustersANN3SimilarityEngine = params(InterestedInParams.EnableSimClustersANN3Param),\n      enableSimClustersANN5SimilarityEngine = params(InterestedInParams.EnableSimClustersANN5Param),\n      enableSimClustersANN4SimilarityEngine = params(InterestedInParams.EnableSimClustersANN4Param),\n      simClustersInterestedInMinScore = simClustersInterestedInMinScore,\n      simClustersNextInterestedInMinScore = simClustersNextInterestedInMinScore,\n      simClustersAddressBookInterestedInMinScore = simClustersAddressBookInterestedInMinScore,\n      interestedInSimClustersANNQuery = interestedInSimClustersANNQuery,\n      nextInterestedInSimClustersANNQuery = nextInterestedInSimClustersANNQuery,\n      addressbookInterestedInSimClustersANNQuery = addressbookInterestedInSimClustersANNQuery,\n      interestedInExperimentalSimClustersANNQuery = interestedInExperimentalSimClustersANNQuery,\n      nextInterestedInExperimentalSimClustersANNQuery =\n        nextInterestedInExperimentalSimClustersANNQuery,\n      addressbookInterestedInExperimentalSimClustersANNQuery =\n        addressbookInterestedInExperimentalSimClustersANNQuery,\n      interestedInSimClustersANN1Query = interestedInSimClustersANN1Query,\n      nextInterestedInSimClustersANN1Query = nextInterestedInSimClustersANN1Query,\n      addressbookInterestedInSimClustersANN1Query = addressbookInterestedInSimClustersANN1Query,\n      interestedInSimClustersANN2Query = interestedInSimClustersANN2Query,\n      nextInterestedInSimClustersANN2Query = nextInterestedInSimClustersANN2Query,\n      addressbookInterestedInSimClustersANN2Query = addressbookInterestedInSimClustersANN2Query,\n      interestedInSimClustersANN3Query = interestedInSimClustersANN3Query,\n      nextInterestedInSimClustersANN3Query = nextInterestedInSimClustersANN3Query,\n      addressbookInterestedInSimClustersANN3Query = addressbookInterestedInSimClustersANN3Query,\n      interestedInSimClustersANN5Query = interestedInSimClustersANN5Query,\n      nextInterestedInSimClustersANN5Query = nextInterestedInSimClustersANN5Query,\n      addressbookInterestedInSimClustersANN5Query = addressbookInterestedInSimClustersANN5Query,\n      interestedInSimClustersANN4Query = interestedInSimClustersANN4Query,\n      nextInterestedInSimClustersANN4Query = nextInterestedInSimClustersANN4Query,\n      addressbookInterestedInSimClustersANN4Query = addressbookInterestedInSimClustersANN4Query,\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation/TopicTweetCandidateGenerator.scala",
    "content": "package com.twitter.cr_mixer.candidate_generation\n\nimport com.twitter.contentrecommender.thriftscala.TweetInfo\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TopicTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.TopicTweetWithScore\nimport com.twitter.cr_mixer.param.TopicTweetParams\nimport com.twitter.cr_mixer.similarity_engine.CertoTopicTweetSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SkitHighPrecisionTopicTweetSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SkitTopicTweetSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.cr_mixer.thriftscala.TopicTweet\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.servo.util.MemoizingStatsReceiver\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.TopicId\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Formerly CrTopic in legacy Content Recommender. This generator finds top Tweets per Topic.\n */\n@Singleton\nclass TopicTweetCandidateGenerator @Inject() (\n  certoTopicTweetSimilarityEngine: CertoTopicTweetSimilarityEngine,\n  skitTopicTweetSimilarityEngine: SkitTopicTweetSimilarityEngine,\n  skitHighPrecisionTopicTweetSimilarityEngine: SkitHighPrecisionTopicTweetSimilarityEngine,\n  tweetInfoStore: ReadableStore[TweetId, TweetInfo],\n  timeoutConfig: TimeoutConfig,\n  globalStats: StatsReceiver) {\n  private val timer = DefaultTimer\n  private val stats: StatsReceiver = globalStats.scope(this.getClass.getCanonicalName)\n  private val fetchCandidatesStats = stats.scope(\"fetchCandidates\")\n  private val filterCandidatesStats = stats.scope(\"filterCandidates\")\n  private val tweetyPieFilteredStats = filterCandidatesStats.stat(\"tweetypie_filtered\")\n  private val memoizedStatsReceiver = new MemoizingStatsReceiver(stats)\n\n  def get(\n    query: TopicTweetCandidateGeneratorQuery\n  ): Future[Map[Long, Seq[TopicTweet]]] = {\n    val maxTweetAge = query.params(TopicTweetParams.MaxTweetAge)\n    val product = query.product\n    val allStats = memoizedStatsReceiver.scope(\"all\")\n    val perProductStats = memoizedStatsReceiver.scope(\"perProduct\", product.name)\n    StatsUtil.trackMapValueStats(allStats) {\n      StatsUtil.trackMapValueStats(perProductStats) {\n        val result = for {\n          retrievedTweets <- fetchCandidates(query)\n          initialTweetCandidates <- convertToInitialCandidates(retrievedTweets)\n          filteredTweetCandidates <- filterCandidates(\n            initialTweetCandidates,\n            maxTweetAge,\n            query.isVideoOnly,\n            query.impressedTweetList)\n          rankedTweetCandidates = rankCandidates(filteredTweetCandidates)\n          hydratedTweetCandidates = hydrateCandidates(rankedTweetCandidates)\n        } yield {\n          hydratedTweetCandidates.map {\n            case (topicId, topicTweets) =>\n              val topKTweets = topicTweets.take(query.maxNumResults)\n              topicId -> topKTweets\n          }\n        }\n        result.raiseWithin(timeoutConfig.topicTweetEndpointTimeout)(timer)\n      }\n    }\n  }\n\n  private def fetchCandidates(\n    query: TopicTweetCandidateGeneratorQuery\n  ): Future[Map[TopicId, Option[Seq[TopicTweetWithScore]]]] = {\n    Future.collect {\n      query.topicIds.map { topicId =>\n        topicId -> StatsUtil.trackOptionStats(fetchCandidatesStats) {\n          Future\n            .join(\n              certoTopicTweetSimilarityEngine.get(CertoTopicTweetSimilarityEngine\n                .fromParams(topicId, query.isVideoOnly, query.params)),\n              skitTopicTweetSimilarityEngine\n                .get(SkitTopicTweetSimilarityEngine\n                  .fromParams(topicId, query.isVideoOnly, query.params)),\n              skitHighPrecisionTopicTweetSimilarityEngine\n                .get(SkitHighPrecisionTopicTweetSimilarityEngine\n                  .fromParams(topicId, query.isVideoOnly, query.params))\n            ).map {\n              case (certoTopicTweets, skitTfgTopicTweets, skitHighPrecisionTopicTweets) =>\n                val uniqueCandidates = (certoTopicTweets.getOrElse(Nil) ++\n                  skitTfgTopicTweets.getOrElse(Nil) ++\n                  skitHighPrecisionTopicTweets.getOrElse(Nil))\n                  .groupBy(_.tweetId).map {\n                    case (_, dupCandidates) => dupCandidates.head\n                  }.toSeq\n                Some(uniqueCandidates)\n            }\n        }\n      }.toMap\n    }\n  }\n\n  private def convertToInitialCandidates(\n    candidatesMap: Map[TopicId, Option[Seq[TopicTweetWithScore]]]\n  ): Future[Map[TopicId, Seq[InitialCandidate]]] = {\n    val initialCandidates = candidatesMap.map {\n      case (topicId, candidatesOpt) =>\n        val candidates = candidatesOpt.getOrElse(Nil)\n        val tweetIds = candidates.map(_.tweetId).toSet\n        val numTweetsPreFilter = tweetIds.size\n        Future.collect(tweetInfoStore.multiGet(tweetIds)).map { tweetInfos =>\n          /** *\n           * If tweetInfo does not exist, we will filter out this tweet candidate.\n           */\n          val tweetyPieFilteredInitialCandidates = candidates.collect {\n            case candidate if tweetInfos.getOrElse(candidate.tweetId, None).isDefined =>\n              val tweetInfo = tweetInfos(candidate.tweetId)\n                .getOrElse(throw new IllegalStateException(\"Check previous line's condition\"))\n\n              InitialCandidate(\n                tweetId = candidate.tweetId,\n                tweetInfo = tweetInfo,\n                CandidateGenerationInfo(\n                  None,\n                  SimilarityEngineInfo(\n                    similarityEngineType = candidate.similarityEngineType,\n                    modelId = None,\n                    score = Some(candidate.score)),\n                  Seq.empty\n                )\n              )\n          }\n          val numTweetsPostFilter = tweetyPieFilteredInitialCandidates.size\n          tweetyPieFilteredStats.add(numTweetsPreFilter - numTweetsPostFilter)\n          topicId -> tweetyPieFilteredInitialCandidates\n        }\n    }\n\n    Future.collect(initialCandidates.toSeq).map(_.toMap)\n  }\n\n  private def filterCandidates(\n    topicTweetMap: Map[TopicId, Seq[InitialCandidate]],\n    maxTweetAge: Duration,\n    isVideoOnly: Boolean,\n    excludeTweetIds: Set[TweetId]\n  ): Future[Map[TopicId, Seq[InitialCandidate]]] = {\n\n    val earliestTweetId = SnowflakeId.firstIdFor(Time.now - maxTweetAge)\n\n    val filteredResults = topicTweetMap.map {\n      case (topicId, tweetsWithScore) =>\n        topicId -> StatsUtil.trackItemsStats(filterCandidatesStats) {\n\n          val timeFilteredTweets =\n            tweetsWithScore.filter { tweetWithScore =>\n              tweetWithScore.tweetId >= earliestTweetId && !excludeTweetIds.contains(\n                tweetWithScore.tweetId)\n            }\n\n          filterCandidatesStats\n            .stat(\"exclude_and_time_filtered\").add(tweetsWithScore.size - timeFilteredTweets.size)\n\n          val tweetNudityFilteredTweets =\n            timeFilteredTweets.collect {\n              case tweet if tweet.tweetInfo.isPassTweetMediaNudityTag.contains(true) => tweet\n            }\n\n          filterCandidatesStats\n            .stat(\"tweet_nudity_filtered\").add(\n              timeFilteredTweets.size - tweetNudityFilteredTweets.size)\n\n          val userNudityFilteredTweets =\n            tweetNudityFilteredTweets.collect {\n              case tweet if tweet.tweetInfo.isPassUserNudityRateStrict.contains(true) => tweet\n            }\n\n          filterCandidatesStats\n            .stat(\"user_nudity_filtered\").add(\n              tweetNudityFilteredTweets.size - userNudityFilteredTweets.size)\n\n          val videoFilteredTweets = {\n            if (isVideoOnly) {\n              userNudityFilteredTweets.collect {\n                case tweet if tweet.tweetInfo.hasVideo.contains(true) => tweet\n              }\n            } else {\n              userNudityFilteredTweets\n            }\n          }\n\n          Future.value(videoFilteredTweets)\n        }\n    }\n    Future.collect(filteredResults)\n  }\n\n  private def rankCandidates(\n    tweetCandidatesMap: Map[TopicId, Seq[InitialCandidate]]\n  ): Map[TopicId, Seq[InitialCandidate]] = {\n    tweetCandidatesMap.mapValues { tweetCandidates =>\n      tweetCandidates.sortBy { candidate =>\n        -candidate.tweetInfo.favCount\n      }\n    }\n  }\n\n  private def hydrateCandidates(\n    topicCandidatesMap: Map[TopicId, Seq[InitialCandidate]]\n  ): Map[Long, Seq[TopicTweet]] = {\n    topicCandidatesMap.map {\n      case (topicId, tweetsWithScore) =>\n        topicId.entityId ->\n          tweetsWithScore.map { tweetWithScore =>\n            val similarityEngineType: SimilarityEngineType =\n              tweetWithScore.candidateGenerationInfo.similarityEngineInfo.similarityEngineType\n            TopicTweet(\n              tweetId = tweetWithScore.tweetId,\n              score = tweetWithScore.getSimilarityScore,\n              similarityEngineType = similarityEngineType\n            )\n          }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation/UtegTweetCandidateGenerator.scala",
    "content": "package com.twitter.cr_mixer.candidate_generation\n\nimport com.twitter.contentrecommender.thriftscala.TweetInfo\nimport com.twitter.cr_mixer.logging.UtegTweetScribeLogger\nimport com.twitter.cr_mixer.filter.UtegFilterRunner\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.RankedCandidate\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScoreAndSocialProof\nimport com.twitter.cr_mixer.model.UtegTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.similarity_engine.UserTweetEntityGraphSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.source_signal.RealGraphInSourceGraphFetcher\nimport com.twitter.cr_mixer.source_signal.SourceFetcher.FetcherQuery\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass UtegTweetCandidateGenerator @Inject() (\n  @Named(ModuleNames.UserTweetEntityGraphSimilarityEngine) userTweetEntityGraphSimilarityEngine: StandardSimilarityEngine[\n    UserTweetEntityGraphSimilarityEngine.Query,\n    TweetWithScoreAndSocialProof\n  ],\n  utegTweetScribeLogger: UtegTweetScribeLogger,\n  tweetInfoStore: ReadableStore[TweetId, TweetInfo],\n  realGraphInSourceGraphFetcher: RealGraphInSourceGraphFetcher,\n  utegFilterRunner: UtegFilterRunner,\n  globalStats: StatsReceiver) {\n\n  private val stats: StatsReceiver = globalStats.scope(this.getClass.getCanonicalName)\n  private val fetchSeedsStats = stats.scope(\"fetchSeeds\")\n  private val fetchCandidatesStats = stats.scope(\"fetchCandidates\")\n  private val utegFilterStats = stats.scope(\"utegFilter\")\n  private val rankStats = stats.scope(\"rank\")\n\n  def get(\n    query: UtegTweetCandidateGeneratorQuery\n  ): Future[Seq[TweetWithScoreAndSocialProof]] = {\n\n    val allStats = stats.scope(\"all\")\n    val perProductStats = stats.scope(\"perProduct\", query.product.toString)\n    StatsUtil.trackItemsStats(allStats) {\n      StatsUtil.trackItemsStats(perProductStats) {\n\n        /**\n         * The candidate we return in the end needs a social proof field, which isn't\n         * supported by the any existing Candidate type, so we created TweetWithScoreAndSocialProof\n         * instead.\n         *\n         * However, filters and light ranker expect Candidate-typed param to work. In order to minimise the\n         * changes to them, we are doing conversions from/to TweetWithScoreAndSocialProof to/from Candidate\n         * in this method.\n         */\n        for {\n          realGraphSeeds <- StatsUtil.trackItemMapStats(fetchSeedsStats) {\n            fetchSeeds(query)\n          }\n          initialTweets <- StatsUtil.trackItemsStats(fetchCandidatesStats) {\n            fetchCandidates(query, realGraphSeeds)\n          }\n          initialCandidates <- convertToInitialCandidates(initialTweets)\n          filteredCandidates <- StatsUtil.trackItemsStats(utegFilterStats) {\n            utegFilter(query, initialCandidates)\n          }\n          rankedCandidates <- StatsUtil.trackItemsStats(rankStats) {\n            rankCandidates(query, filteredCandidates)\n          }\n        } yield {\n          val topTweets = rankedCandidates.take(query.maxNumResults)\n          convertToTweets(topTweets, initialTweets.map(tweet => tweet.tweetId -> tweet).toMap)\n        }\n      }\n    }\n  }\n\n  private def utegFilter(\n    query: UtegTweetCandidateGeneratorQuery,\n    candidates: Seq[InitialCandidate]\n  ): Future[Seq[InitialCandidate]] = {\n    utegFilterRunner.runSequentialFilters(query, Seq(candidates)).map(_.flatten)\n  }\n\n  private def fetchSeeds(\n    query: UtegTweetCandidateGeneratorQuery\n  ): Future[Map[UserId, Double]] = {\n    realGraphInSourceGraphFetcher\n      .get(FetcherQuery(query.userId, query.product, query.userState, query.params))\n      .map(_.map(_.seedWithScores).getOrElse(Map.empty))\n  }\n\n  private[candidate_generation] def rankCandidates(\n    query: UtegTweetCandidateGeneratorQuery,\n    filteredCandidates: Seq[InitialCandidate],\n  ): Future[Seq[RankedCandidate]] = {\n    val blendedCandidates = filteredCandidates.map(candidate =>\n      candidate.toBlendedCandidate(Seq(candidate.candidateGenerationInfo)))\n\n    Future(\n      blendedCandidates.map { candidate =>\n        val score = candidate.getSimilarityScore\n        candidate.toRankedCandidate(score)\n      }\n    )\n\n  }\n\n  def fetchCandidates(\n    query: UtegTweetCandidateGeneratorQuery,\n    realGraphSeeds: Map[UserId, Double],\n  ): Future[Seq[TweetWithScoreAndSocialProof]] = {\n    val engineQuery = UserTweetEntityGraphSimilarityEngine.fromParams(\n      query.userId,\n      realGraphSeeds,\n      Some(query.impressedTweetList.toSeq),\n      query.params\n    )\n\n    utegTweetScribeLogger.scribeInitialCandidates(\n      query,\n      userTweetEntityGraphSimilarityEngine.getCandidates(engineQuery).map(_.toSeq.flatten)\n    )\n  }\n\n  private[candidate_generation] def convertToInitialCandidates(\n    candidates: Seq[TweetWithScoreAndSocialProof],\n  ): Future[Seq[InitialCandidate]] = {\n    val tweetIds = candidates.map(_.tweetId).toSet\n    Future.collect(tweetInfoStore.multiGet(tweetIds)).map { tweetInfos =>\n      /** *\n       * If tweetInfo does not exist, we will filter out this tweet candidate.\n       */\n      candidates.collect {\n        case candidate if tweetInfos.getOrElse(candidate.tweetId, None).isDefined =>\n          val tweetInfo = tweetInfos(candidate.tweetId)\n            .getOrElse(throw new IllegalStateException(\"Check previous line's condition\"))\n\n          InitialCandidate(\n            tweetId = candidate.tweetId,\n            tweetInfo = tweetInfo,\n            CandidateGenerationInfo(\n              None,\n              SimilarityEngineInfo(\n                similarityEngineType = SimilarityEngineType.Uteg,\n                modelId = None,\n                score = Some(candidate.score)),\n              Seq.empty\n            )\n          )\n      }\n    }\n  }\n\n  private[candidate_generation] def convertToTweets(\n    candidates: Seq[RankedCandidate],\n    tweetMap: Map[TweetId, TweetWithScoreAndSocialProof]\n  ): Seq[TweetWithScoreAndSocialProof] = {\n    candidates.map { candidate =>\n      tweetMap\n        .get(candidate.tweetId).map { tweet =>\n          TweetWithScoreAndSocialProof(\n            tweet.tweetId,\n            candidate.predictionScore,\n            tweet.socialProofByType\n          )\n        // The exception should never be thrown\n        }.getOrElse(throw new Exception(\"Cannot find ranked candidate in original UTEG tweets\"))\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/config/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/exception\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"simclusters-ann/thrift/src/main/thrift:thrift-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/config/SimClustersANNConfig.scala",
    "content": "package com.twitter.cr_mixer.config\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.exception.InvalidSANNConfigException\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclustersann.thriftscala.ScoringAlgorithm\nimport com.twitter.simclustersann.thriftscala.{SimClustersANNConfig => ThriftSimClustersANNConfig}\nimport com.twitter.util.Duration\n\ncase class SimClustersANNConfig(\n  maxNumResults: Int,\n  minScore: Double,\n  candidateEmbeddingType: EmbeddingType,\n  maxTopTweetsPerCluster: Int,\n  maxScanClusters: Int,\n  maxTweetCandidateAge: Duration,\n  minTweetCandidateAge: Duration,\n  annAlgorithm: ScoringAlgorithm) {\n  val toSANNConfigThrift: ThriftSimClustersANNConfig = ThriftSimClustersANNConfig(\n    maxNumResults = maxNumResults,\n    minScore = minScore,\n    candidateEmbeddingType = candidateEmbeddingType,\n    maxTopTweetsPerCluster = maxTopTweetsPerCluster,\n    maxScanClusters = maxScanClusters,\n    maxTweetCandidateAgeHours = maxTweetCandidateAge.inHours,\n    minTweetCandidateAgeHours = minTweetCandidateAge.inHours,\n    annAlgorithm = annAlgorithm,\n  )\n}\n\nobject SimClustersANNConfig {\n\n  final val DefaultConfig = SimClustersANNConfig(\n    maxNumResults = 200,\n    minScore = 0.0,\n    candidateEmbeddingType = EmbeddingType.LogFavBasedTweet,\n    maxTopTweetsPerCluster = 800,\n    maxScanClusters = 50,\n    maxTweetCandidateAge = 24.hours,\n    minTweetCandidateAge = 0.hours,\n    annAlgorithm = ScoringAlgorithm.CosineSimilarity,\n  )\n\n  /*\n  SimClustersANNConfigId: String\n  Format: Prod - “EmbeddingType_ModelVersion_Default”\n  Format: Experiment - “EmbeddingType_ModelVersion_Date_Two-Digit-Serial-Number”. Date : YYYYMMDD\n   */\n\n  private val FavBasedProducer_Model20m145k2020_Default = DefaultConfig.copy()\n\n  // Chunnan's exp on maxTweetCandidateAgeDays 2\n  private val FavBasedProducer_Model20m145k2020_20220617_06 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      maxTweetCandidateAge = 48.hours,\n    )\n\n  // Experimental SANN config\n  private val FavBasedProducer_Model20m145k2020_20220801 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.VideoPlayBack50LogFavBasedTweet,\n    )\n\n  // SANN-1 config\n  private val FavBasedProducer_Model20m145k2020_20220810 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-2 config\n  private val FavBasedProducer_Model20m145k2020_20220818 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavClickBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-3 config\n  private val FavBasedProducer_Model20m145k2020_20220819 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.PushOpenLogFavBasedTweet,\n    )\n\n  // SANN-5 config\n  private val FavBasedProducer_Model20m145k2020_20221221 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedRealTimeTweet,\n      maxTweetCandidateAge = 1.hours\n    )\n\n  // SANN-4 config\n  private val FavBasedProducer_Model20m145k2020_20221220 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedEvergreenTweet,\n      maxTweetCandidateAge = 48.hours\n    )\n  private val LogFavLongestL2EmbeddingTweet_Model20m145k2020_Default = DefaultConfig.copy()\n\n  // Chunnan's exp on maxTweetCandidateAgeDays 2\n  private val LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220617_06 =\n    LogFavLongestL2EmbeddingTweet_Model20m145k2020_Default.copy(\n      maxTweetCandidateAge = 48.hours,\n    )\n\n  // Experimental SANN config\n  private val LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220801 =\n    LogFavLongestL2EmbeddingTweet_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.VideoPlayBack50LogFavBasedTweet,\n    )\n\n  // SANN-1 config\n  private val LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220810 =\n    LogFavLongestL2EmbeddingTweet_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-2 config\n  private val LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220818 =\n    LogFavLongestL2EmbeddingTweet_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavClickBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-3 config\n  private val LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220819 =\n    LogFavLongestL2EmbeddingTweet_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.PushOpenLogFavBasedTweet,\n    )\n\n  // SANN-5 config\n  private val LogFavLongestL2EmbeddingTweet_Model20m145k2020_20221221 =\n    LogFavLongestL2EmbeddingTweet_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedRealTimeTweet,\n      maxTweetCandidateAge = 1.hours\n    )\n  // SANN-4 config\n  private val LogFavLongestL2EmbeddingTweet_Model20m145k2020_20221220 =\n    LogFavLongestL2EmbeddingTweet_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedEvergreenTweet,\n      maxTweetCandidateAge = 48.hours\n    )\n  private val UnfilteredUserInterestedIn_Model20m145k2020_Default = DefaultConfig.copy()\n\n  // Chunnan's exp on maxTweetCandidateAgeDays 2\n  private val UnfilteredUserInterestedIn_Model20m145k2020_20220617_06 =\n    UnfilteredUserInterestedIn_Model20m145k2020_Default.copy(\n      maxTweetCandidateAge = 48.hours,\n    )\n\n  // Experimental SANN config\n  private val UnfilteredUserInterestedIn_Model20m145k2020_20220801 =\n    UnfilteredUserInterestedIn_Model20m145k2020_20220617_06.copy(\n      candidateEmbeddingType = EmbeddingType.VideoPlayBack50LogFavBasedTweet,\n    )\n\n  // SANN-1 config\n  private val UnfilteredUserInterestedIn_Model20m145k2020_20220810 =\n    UnfilteredUserInterestedIn_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-2 config\n  private val UnfilteredUserInterestedIn_Model20m145k2020_20220818 =\n    UnfilteredUserInterestedIn_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavClickBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-3 config\n  private val UnfilteredUserInterestedIn_Model20m145k2020_20220819 =\n    UnfilteredUserInterestedIn_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.PushOpenLogFavBasedTweet,\n    )\n\n  // SANN-5 config\n  private val UnfilteredUserInterestedIn_Model20m145k2020_20221221 =\n    UnfilteredUserInterestedIn_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedRealTimeTweet,\n      maxTweetCandidateAge = 1.hours\n    )\n\n  // SANN-4 config\n  private val UnfilteredUserInterestedIn_Model20m145k2020_20221220 =\n    UnfilteredUserInterestedIn_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedEvergreenTweet,\n      maxTweetCandidateAge = 48.hours\n    )\n  private val LogFavBasedUserInterestedInFromAPE_Model20m145k2020_Default = DefaultConfig.copy()\n\n  // Chunnan's exp on maxTweetCandidateAgeDays 2\n  private val LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220617_06 =\n    LogFavBasedUserInterestedInFromAPE_Model20m145k2020_Default.copy(\n      maxTweetCandidateAge = 48.hours,\n    )\n\n  // Experimental SANN config\n  private val LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220801 =\n    LogFavBasedUserInterestedInFromAPE_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.VideoPlayBack50LogFavBasedTweet,\n    )\n\n  // SANN-1 config\n  private val LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220810 =\n    LogFavBasedUserInterestedInFromAPE_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-2 config\n  private val LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220818 =\n    LogFavBasedUserInterestedInFromAPE_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavClickBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-3 config\n  private val LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220819 =\n    LogFavBasedUserInterestedInFromAPE_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.PushOpenLogFavBasedTweet,\n    )\n\n  // SANN-5 config\n  private val LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20221221 =\n    LogFavBasedUserInterestedInFromAPE_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedRealTimeTweet,\n      maxTweetCandidateAge = 1.hours\n    )\n\n  // SANN-4 config\n  private val LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20221220 =\n    LogFavBasedUserInterestedInFromAPE_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedEvergreenTweet,\n      maxTweetCandidateAge = 48.hours\n    )\n  private val LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_Default =\n    DefaultConfig.copy()\n\n  // Chunnan's exp on maxTweetCandidateAgeDays 2\n  private val LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220617_06 =\n    LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_Default.copy(\n      maxTweetCandidateAge = 48.hours,\n    )\n\n  // Experimental SANN config\n  private val LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220801 =\n    LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.VideoPlayBack50LogFavBasedTweet,\n    )\n\n  // SANN-1 config\n  private val LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220810 =\n    LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-2 config\n  private val LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220818 =\n    LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavClickBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-3 config\n  private val LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220819 =\n    LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.PushOpenLogFavBasedTweet,\n    )\n\n  // SANN-5 config\n  private val LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20221221 =\n    LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedRealTimeTweet,\n      maxTweetCandidateAge = 1.hours\n    )\n\n  // SANN-4 config\n  private val LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20221220 =\n    LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedEvergreenTweet,\n      maxTweetCandidateAge = 48.hours\n    )\n  private val UserNextInterestedIn_Model20m145k2020_Default = DefaultConfig.copy()\n\n  // Chunnan's exp on maxTweetCandidateAgeDays 2\n  private val UserNextInterestedIn_Model20m145k2020_20220617_06 =\n    UserNextInterestedIn_Model20m145k2020_Default.copy(\n      maxTweetCandidateAge = 48.hours,\n    )\n\n  // Experimental SANN config\n  private val UserNextInterestedIn_Model20m145k2020_20220801 =\n    UserNextInterestedIn_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.VideoPlayBack50LogFavBasedTweet,\n    )\n\n  // SANN-1 config\n  private val UserNextInterestedIn_Model20m145k2020_20220810 =\n    UserNextInterestedIn_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-2 config\n  private val UserNextInterestedIn_Model20m145k2020_20220818 =\n    UserNextInterestedIn_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavClickBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-3 config\n  private val UserNextInterestedIn_Model20m145k2020_20220819 =\n    UserNextInterestedIn_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.PushOpenLogFavBasedTweet,\n    )\n\n  // SANN-5 config\n  private val UserNextInterestedIn_Model20m145k2020_20221221 =\n    UserNextInterestedIn_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedRealTimeTweet,\n      maxTweetCandidateAge = 1.hours\n    )\n\n  // SANN-4 config\n  private val UserNextInterestedIn_Model20m145k2020_20221220 =\n    UserNextInterestedIn_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedEvergreenTweet,\n      maxTweetCandidateAge = 48.hours\n    )\n  // Vincent's experiment on using FollowBasedProducer as query embedding type for UserFollow\n  private val FollowBasedProducer_Model20m145k2020_Default =\n    FavBasedProducer_Model20m145k2020_Default.copy()\n\n  // Experimental SANN config\n  private val FollowBasedProducer_Model20m145k2020_20220801 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.VideoPlayBack50LogFavBasedTweet,\n    )\n\n  // SANN-1 config\n  private val FollowBasedProducer_Model20m145k2020_20220810 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-2 config\n  private val FollowBasedProducer_Model20m145k2020_20220818 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      maxNumResults = 100,\n      candidateEmbeddingType = EmbeddingType.LogFavClickBasedAdsTweet,\n      maxTweetCandidateAge = 175200.hours,\n      maxTopTweetsPerCluster = 1600\n    )\n\n  // SANN-3 config\n  private val FollowBasedProducer_Model20m145k2020_20220819 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.PushOpenLogFavBasedTweet,\n    )\n\n  // SANN-5 config\n  private val FollowBasedProducer_Model20m145k2020_20221221 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedRealTimeTweet,\n      maxTweetCandidateAge = 1.hours\n    )\n\n  // SANN-4 config\n  private val FollowBasedProducer_Model20m145k2020_20221220 =\n    FavBasedProducer_Model20m145k2020_Default.copy(\n      candidateEmbeddingType = EmbeddingType.LogFavBasedEvergreenTweet,\n      maxTweetCandidateAge = 48.hours\n    )\n  val DefaultConfigMappings: Map[String, SimClustersANNConfig] = Map(\n    \"FavBasedProducer_Model20m145k2020_Default\" -> FavBasedProducer_Model20m145k2020_Default,\n    \"FavBasedProducer_Model20m145k2020_20220617_06\" -> FavBasedProducer_Model20m145k2020_20220617_06,\n    \"FavBasedProducer_Model20m145k2020_20220801\" -> FavBasedProducer_Model20m145k2020_20220801,\n    \"FavBasedProducer_Model20m145k2020_20220810\" -> FavBasedProducer_Model20m145k2020_20220810,\n    \"FavBasedProducer_Model20m145k2020_20220818\" -> FavBasedProducer_Model20m145k2020_20220818,\n    \"FavBasedProducer_Model20m145k2020_20220819\" -> FavBasedProducer_Model20m145k2020_20220819,\n    \"FavBasedProducer_Model20m145k2020_20221221\" -> FavBasedProducer_Model20m145k2020_20221221,\n    \"FavBasedProducer_Model20m145k2020_20221220\" -> FavBasedProducer_Model20m145k2020_20221220,\n    \"FollowBasedProducer_Model20m145k2020_Default\" -> FollowBasedProducer_Model20m145k2020_Default,\n    \"FollowBasedProducer_Model20m145k2020_20220801\" -> FollowBasedProducer_Model20m145k2020_20220801,\n    \"FollowBasedProducer_Model20m145k2020_20220810\" -> FollowBasedProducer_Model20m145k2020_20220810,\n    \"FollowBasedProducer_Model20m145k2020_20220818\" -> FollowBasedProducer_Model20m145k2020_20220818,\n    \"FollowBasedProducer_Model20m145k2020_20220819\" -> FollowBasedProducer_Model20m145k2020_20220819,\n    \"FollowBasedProducer_Model20m145k2020_20221221\" -> FollowBasedProducer_Model20m145k2020_20221221,\n    \"FollowBasedProducer_Model20m145k2020_20221220\" -> FollowBasedProducer_Model20m145k2020_20221220,\n    \"LogFavLongestL2EmbeddingTweet_Model20m145k2020_Default\" -> LogFavLongestL2EmbeddingTweet_Model20m145k2020_Default,\n    \"LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220617_06\" -> LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220617_06,\n    \"LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220801\" -> LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220801,\n    \"LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220810\" -> LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220810,\n    \"LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220818\" -> LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220818,\n    \"LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220819\" -> LogFavLongestL2EmbeddingTweet_Model20m145k2020_20220819,\n    \"LogFavLongestL2EmbeddingTweet_Model20m145k2020_20221221\" -> LogFavLongestL2EmbeddingTweet_Model20m145k2020_20221221,\n    \"LogFavLongestL2EmbeddingTweet_Model20m145k2020_20221220\" -> LogFavLongestL2EmbeddingTweet_Model20m145k2020_20221220,\n    \"UnfilteredUserInterestedIn_Model20m145k2020_Default\" -> UnfilteredUserInterestedIn_Model20m145k2020_Default,\n    \"UnfilteredUserInterestedIn_Model20m145k2020_20220617_06\" -> UnfilteredUserInterestedIn_Model20m145k2020_20220617_06,\n    \"UnfilteredUserInterestedIn_Model20m145k2020_20220801\" -> UnfilteredUserInterestedIn_Model20m145k2020_20220801,\n    \"UnfilteredUserInterestedIn_Model20m145k2020_20220810\" -> UnfilteredUserInterestedIn_Model20m145k2020_20220810,\n    \"UnfilteredUserInterestedIn_Model20m145k2020_20220818\" -> UnfilteredUserInterestedIn_Model20m145k2020_20220818,\n    \"UnfilteredUserInterestedIn_Model20m145k2020_20220819\" -> UnfilteredUserInterestedIn_Model20m145k2020_20220819,\n    \"UnfilteredUserInterestedIn_Model20m145k2020_20221221\" -> UnfilteredUserInterestedIn_Model20m145k2020_20221221,\n    \"UnfilteredUserInterestedIn_Model20m145k2020_20221220\" -> UnfilteredUserInterestedIn_Model20m145k2020_20221220,\n    \"LogFavBasedUserInterestedInFromAPE_Model20m145k2020_Default\" -> LogFavBasedUserInterestedInFromAPE_Model20m145k2020_Default,\n    \"LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220617_06\" -> LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220617_06,\n    \"LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220801\" -> LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220801,\n    \"LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220810\" -> LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220810,\n    \"LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220818\" -> LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220818,\n    \"LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220819\" -> LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20220819,\n    \"LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20221221\" -> LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20221221,\n    \"LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20221220\" -> LogFavBasedUserInterestedInFromAPE_Model20m145k2020_20221220,\n    \"LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_Default\" -> LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_Default,\n    \"LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220617_06\" -> LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220617_06,\n    \"LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220801\" -> LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220801,\n    \"LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220810\" -> LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220810,\n    \"LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220818\" -> LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220818,\n    \"LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220819\" -> LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20220819,\n    \"LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20221221\" -> LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20221221,\n    \"LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20221220\" -> LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020_20221220,\n    \"UserNextInterestedIn_Model20m145k2020_Default\" -> UserNextInterestedIn_Model20m145k2020_Default,\n    \"UserNextInterestedIn_Model20m145k2020_20220617_06\" -> UserNextInterestedIn_Model20m145k2020_20220617_06,\n    \"UserNextInterestedIn_Model20m145k2020_20220801\" -> UserNextInterestedIn_Model20m145k2020_20220801,\n    \"UserNextInterestedIn_Model20m145k2020_20220810\" -> UserNextInterestedIn_Model20m145k2020_20220810,\n    \"UserNextInterestedIn_Model20m145k2020_20220818\" -> UserNextInterestedIn_Model20m145k2020_20220818,\n    \"UserNextInterestedIn_Model20m145k2020_20220819\" -> UserNextInterestedIn_Model20m145k2020_20220819,\n    \"UserNextInterestedIn_Model20m145k2020_20221221\" -> UserNextInterestedIn_Model20m145k2020_20221221,\n    \"UserNextInterestedIn_Model20m145k2020_20221220\" -> UserNextInterestedIn_Model20m145k2020_20221220,\n  )\n\n  def getConfig(\n    embeddingType: String,\n    modelVersion: String,\n    id: String\n  ): SimClustersANNConfig = {\n    val configName = embeddingType + \"_\" + modelVersion + \"_\" + id\n    DefaultConfigMappings.get(configName) match {\n      case Some(config) => config\n      case None =>\n        throw InvalidSANNConfigException(s\"Incorrect config id passed in for SANN $configName\")\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/config/TimeoutConfig.scala",
    "content": "package com.twitter.cr_mixer.config\n\nimport com.twitter.util.Duration\n\ncase class TimeoutConfig(\n  /* Default timeouts for candidate generator */\n  serviceTimeout: Duration,\n  signalFetchTimeout: Duration,\n  similarityEngineTimeout: Duration,\n  annServiceClientTimeout: Duration,\n  /* For Uteg Candidate Generator */\n  utegSimilarityEngineTimeout: Duration,\n  /* For User State Store */\n  userStateUnderlyingStoreTimeout: Duration,\n  userStateStoreTimeout: Duration,\n  /* For FRS based tweets */\n  // Timeout passed to EarlyBird server\n  earlybirdServerTimeout: Duration,\n  // Timeout set on CrMixer side\n  earlybirdSimilarityEngineTimeout: Duration,\n  frsBasedTweetEndpointTimeout: Duration,\n  topicTweetEndpointTimeout: Duration,\n  // Timeout Settings for Navi gRPC Client\n  naviRequestTimeout: Duration)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/controller/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"content-recommender/thrift/src/main/thrift:content-recommender-common-scala\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/debug\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/featureswitch\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/logging\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"finagle/finagle-base-http/src/main\",\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"finatra/http-server/src/main/scala/com/twitter/finatra/http:controller\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift:controller\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/base\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"simclusters-ann/thrift/src/main/thrift:thrift-scala\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/ads/schema:common-scala\",\n        \"src/thrift/com/twitter/context:twitter-context-scala\",\n        \"src/thrift/com/twitter/core_workflows/user_model:user_model-scala\",\n        \"src/thrift/com/twitter/frigate/data_pipeline/scalding:blue_verified_annotations-scala\",\n        \"src/thrift/com/twitter/onboarding/relevance/coldstart_lookalike:coldstartlookalike-thrift-scala\",\n        \"src/thrift/com/twitter/recos:recos-common-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n        \"src/thrift/com/twitter/timelines/timeline_logging:thrift-scala\",\n        \"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala\",\n        \"stringcenter/client\",\n        \"timelines/src/main/scala/com/twitter/timelines/tracing/lensview\",\n        \"timelines/src/main/scala/com/twitter/timelines/tracing/lensview/funnelseries\",\n        \"twitter-context/src/main/scala\",\n        \"user-signal-service/thrift/src/main/thrift:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/controller/CrMixerThriftController.scala",
    "content": "package com.twitter.cr_mixer.controller\n\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.cr_mixer.candidate_generation.AdsCandidateGenerator\nimport com.twitter.cr_mixer.candidate_generation.CrCandidateGenerator\nimport com.twitter.cr_mixer.candidate_generation.FrsTweetCandidateGenerator\nimport com.twitter.cr_mixer.candidate_generation.RelatedTweetCandidateGenerator\nimport com.twitter.cr_mixer.candidate_generation.RelatedVideoTweetCandidateGenerator\nimport com.twitter.cr_mixer.candidate_generation.TopicTweetCandidateGenerator\nimport com.twitter.cr_mixer.candidate_generation.UtegTweetCandidateGenerator\nimport com.twitter.cr_mixer.featureswitch.ParamsBuilder\nimport com.twitter.cr_mixer.logging.CrMixerScribeLogger\nimport com.twitter.cr_mixer.logging.RelatedTweetScribeLogger\nimport com.twitter.cr_mixer.logging.AdsRecommendationsScribeLogger\nimport com.twitter.cr_mixer.logging.RelatedTweetScribeMetadata\nimport com.twitter.cr_mixer.logging.ScribeMetadata\nimport com.twitter.cr_mixer.logging.UtegTweetScribeLogger\nimport com.twitter.cr_mixer.model.AdsCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.CrCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.FrsTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.model.RankedAdsCandidate\nimport com.twitter.cr_mixer.model.RankedCandidate\nimport com.twitter.cr_mixer.model.RelatedTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.RelatedVideoTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.TopicTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.TweetWithScoreAndSocialProof\nimport com.twitter.cr_mixer.model.UtegTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.param.AdsParams\nimport com.twitter.cr_mixer.param.FrsParams.FrsBasedCandidateGenerationMaxCandidatesNumParam\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.param.RelatedTweetGlobalParams\nimport com.twitter.cr_mixer.param.RelatedVideoTweetGlobalParams\nimport com.twitter.cr_mixer.param.TopicTweetParams\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.param.decider.EndpointLoadShedder\nimport com.twitter.cr_mixer.thriftscala.AdTweetRecommendation\nimport com.twitter.cr_mixer.thriftscala.AdsRequest\nimport com.twitter.cr_mixer.thriftscala.AdsResponse\nimport com.twitter.cr_mixer.thriftscala.CrMixerTweetRequest\nimport com.twitter.cr_mixer.thriftscala.CrMixerTweetResponse\nimport com.twitter.cr_mixer.thriftscala.FrsTweetRequest\nimport com.twitter.cr_mixer.thriftscala.FrsTweetResponse\nimport com.twitter.cr_mixer.thriftscala.RelatedTweet\nimport com.twitter.cr_mixer.thriftscala.RelatedTweetRequest\nimport com.twitter.cr_mixer.thriftscala.RelatedTweetResponse\nimport com.twitter.cr_mixer.thriftscala.RelatedVideoTweet\nimport com.twitter.cr_mixer.thriftscala.RelatedVideoTweetRequest\nimport com.twitter.cr_mixer.thriftscala.RelatedVideoTweetResponse\nimport com.twitter.cr_mixer.thriftscala.TopicTweet\nimport com.twitter.cr_mixer.thriftscala.TopicTweetRequest\nimport com.twitter.cr_mixer.thriftscala.TopicTweetResponse\nimport com.twitter.cr_mixer.thriftscala.TweetRecommendation\nimport com.twitter.cr_mixer.thriftscala.UtegTweet\nimport com.twitter.cr_mixer.thriftscala.UtegTweetRequest\nimport com.twitter.cr_mixer.thriftscala.UtegTweetResponse\nimport com.twitter.cr_mixer.util.MetricTagUtil\nimport com.twitter.cr_mixer.util.SignalTimestampStatsUtil\nimport com.twitter.cr_mixer.{thriftscala => t}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.thrift.Controller\nimport com.twitter.hermit.store.common.ReadableWritableStore\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.simclusters_v2.thriftscala.TopicId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.timeline_logging.{thriftscala => thriftlog}\nimport com.twitter.timelines.tracing.lensview.funnelseries.TweetScoreFunnelSeries\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport java.util.UUID\nimport javax.inject.Inject\nimport org.apache.commons.lang.exception.ExceptionUtils\n\nclass CrMixerThriftController @Inject() (\n  crCandidateGenerator: CrCandidateGenerator,\n  relatedTweetCandidateGenerator: RelatedTweetCandidateGenerator,\n  relatedVideoTweetCandidateGenerator: RelatedVideoTweetCandidateGenerator,\n  utegTweetCandidateGenerator: UtegTweetCandidateGenerator,\n  frsTweetCandidateGenerator: FrsTweetCandidateGenerator,\n  topicTweetCandidateGenerator: TopicTweetCandidateGenerator,\n  crMixerScribeLogger: CrMixerScribeLogger,\n  relatedTweetScribeLogger: RelatedTweetScribeLogger,\n  utegTweetScribeLogger: UtegTweetScribeLogger,\n  adsRecommendationsScribeLogger: AdsRecommendationsScribeLogger,\n  adsCandidateGenerator: AdsCandidateGenerator,\n  decider: CrMixerDecider,\n  paramsBuilder: ParamsBuilder,\n  endpointLoadShedder: EndpointLoadShedder,\n  signalTimestampStatsUtil: SignalTimestampStatsUtil,\n  tweetRecommendationResultsStore: ReadableWritableStore[UserId, CrMixerTweetResponse],\n  userStateStore: ReadableStore[UserId, UserState],\n  statsReceiver: StatsReceiver)\n    extends Controller(t.CrMixer) {\n\n  lazy private val tweetScoreFunnelSeries = new TweetScoreFunnelSeries(statsReceiver)\n\n  private def logErrMessage(endpoint: String, e: Throwable): Unit = {\n    val msg = Seq(\n      s\"Failed endpoint $endpoint: ${e.getLocalizedMessage}\",\n      ExceptionUtils.getStackTrace(e)\n    ).mkString(\"\\n\")\n\n    /** *\n     * We chose logger.info() here to print message instead of logger.error since that\n     * logger.error sometimes suppresses detailed stacktrace.\n     */\n    logger.info(msg)\n  }\n\n  private def generateRequestUUID(): Long = {\n\n    /** *\n     * We generate unique UUID via bitwise operations. See the below link for more:\n     * https://stackoverflow.com/questions/15184820/how-to-generate-unique-positive-long-using-uuid\n     */\n    UUID.randomUUID().getMostSignificantBits & Long.MaxValue\n  }\n\n  handle(t.CrMixer.GetTweetRecommendations) { args: t.CrMixer.GetTweetRecommendations.Args =>\n    val endpointName = \"getTweetRecommendations\"\n\n    val requestUUID = generateRequestUUID()\n    val startTime = Time.now.inMilliseconds\n    val userId = args.request.clientContext.userId.getOrElse(\n      throw new IllegalArgumentException(\"userId must be present in the Thrift clientContext\")\n    )\n    val queryFut = buildCrCandidateGeneratorQuery(args.request, requestUUID, userId)\n    queryFut.flatMap { query =>\n      val scribeMetadata = ScribeMetadata.from(query)\n      endpointLoadShedder(endpointName, query.product.originalName) {\n\n        val response = crCandidateGenerator.get(query)\n\n        val blueVerifiedScribedResponse = response.flatMap { rankedCandidates =>\n          val hasBlueVerifiedCandidate = rankedCandidates.exists { tweet =>\n            tweet.tweetInfo.hasBlueVerifiedAnnotation.contains(true)\n          }\n\n          if (hasBlueVerifiedCandidate) {\n            crMixerScribeLogger.scribeGetTweetRecommendationsForBlueVerified(\n              scribeMetadata,\n              response)\n          } else {\n            response\n          }\n        }\n\n        val thriftResponse = blueVerifiedScribedResponse.map { candidates =>\n          if (query.product == t.Product.Home) {\n            scribeTweetScoreFunnelSeries(candidates)\n          }\n          buildThriftResponse(candidates)\n        }\n\n        cacheTweetRecommendationResults(args.request, thriftResponse)\n\n        crMixerScribeLogger.scribeGetTweetRecommendations(\n          args.request,\n          startTime,\n          scribeMetadata,\n          thriftResponse)\n      }.rescue {\n        case EndpointLoadShedder.LoadSheddingException =>\n          Future(CrMixerTweetResponse(Seq.empty))\n        case e =>\n          logErrMessage(endpointName, e)\n          Future(CrMixerTweetResponse(Seq.empty))\n      }\n    }\n\n  }\n\n  /** *\n   * GetRelatedTweetsForQueryTweet and GetRelatedTweetsForQueryAuthor are essentially\n   * doing very similar things, except that one passes in TweetId which calls TweetBased engine,\n   * and the other passes in AuthorId which calls ProducerBased engine.\n   */\n  handle(t.CrMixer.GetRelatedTweetsForQueryTweet) {\n    args: t.CrMixer.GetRelatedTweetsForQueryTweet.Args =>\n      val endpointName = \"getRelatedTweetsForQueryTweet\"\n      getRelatedTweets(endpointName, args.request)\n  }\n\n  handle(t.CrMixer.GetRelatedVideoTweetsForQueryTweet) {\n    args: t.CrMixer.GetRelatedVideoTweetsForQueryTweet.Args =>\n      val endpointName = \"getRelatedVideoTweetsForQueryVideoTweet\"\n      getRelatedVideoTweets(endpointName, args.request)\n\n  }\n\n  handle(t.CrMixer.GetRelatedTweetsForQueryAuthor) {\n    args: t.CrMixer.GetRelatedTweetsForQueryAuthor.Args =>\n      val endpointName = \"getRelatedTweetsForQueryAuthor\"\n      getRelatedTweets(endpointName, args.request)\n  }\n\n  private def getRelatedTweets(\n    endpointName: String,\n    request: RelatedTweetRequest\n  ): Future[RelatedTweetResponse] = {\n    val requestUUID = generateRequestUUID()\n    val startTime = Time.now.inMilliseconds\n    val queryFut = buildRelatedTweetQuery(request, requestUUID)\n\n    queryFut.flatMap { query =>\n      val relatedTweetScribeMetadata = RelatedTweetScribeMetadata.from(query)\n      endpointLoadShedder(endpointName, query.product.originalName) {\n        relatedTweetScribeLogger.scribeGetRelatedTweets(\n          request,\n          startTime,\n          relatedTweetScribeMetadata,\n          relatedTweetCandidateGenerator\n            .get(query)\n            .map(buildRelatedTweetResponse))\n      }.rescue {\n        case EndpointLoadShedder.LoadSheddingException =>\n          Future(RelatedTweetResponse(Seq.empty))\n        case e =>\n          logErrMessage(endpointName, e)\n          Future(RelatedTweetResponse(Seq.empty))\n      }\n    }\n\n  }\n\n  private def getRelatedVideoTweets(\n    endpointName: String,\n    request: RelatedVideoTweetRequest\n  ): Future[RelatedVideoTweetResponse] = {\n    val requestUUID = generateRequestUUID()\n    val queryFut = buildRelatedVideoTweetQuery(request, requestUUID)\n\n    queryFut.flatMap { query =>\n      endpointLoadShedder(endpointName, query.product.originalName) {\n        relatedVideoTweetCandidateGenerator.get(query).map { initialCandidateSeq =>\n          buildRelatedVideoTweetResponse(initialCandidateSeq)\n        }\n      }.rescue {\n        case EndpointLoadShedder.LoadSheddingException =>\n          Future(RelatedVideoTweetResponse(Seq.empty))\n        case e =>\n          logErrMessage(endpointName, e)\n          Future(RelatedVideoTweetResponse(Seq.empty))\n      }\n    }\n  }\n\n  handle(t.CrMixer.GetFrsBasedTweetRecommendations) {\n    args: t.CrMixer.GetFrsBasedTweetRecommendations.Args =>\n      val endpointName = \"getFrsBasedTweetRecommendations\"\n\n      val requestUUID = generateRequestUUID()\n      val queryFut = buildFrsBasedTweetQuery(args.request, requestUUID)\n      queryFut.flatMap { query =>\n        endpointLoadShedder(endpointName, query.product.originalName) {\n          frsTweetCandidateGenerator.get(query).map(FrsTweetResponse(_))\n        }.rescue {\n          case e =>\n            logErrMessage(endpointName, e)\n            Future(FrsTweetResponse(Seq.empty))\n        }\n      }\n  }\n\n  handle(t.CrMixer.GetTopicTweetRecommendations) {\n    args: t.CrMixer.GetTopicTweetRecommendations.Args =>\n      val endpointName = \"getTopicTweetRecommendations\"\n\n      val requestUUID = generateRequestUUID()\n      val query = buildTopicTweetQuery(args.request, requestUUID)\n\n      endpointLoadShedder(endpointName, query.product.originalName) {\n        topicTweetCandidateGenerator.get(query).map(TopicTweetResponse(_))\n      }.rescue {\n        case e =>\n          logErrMessage(endpointName, e)\n          Future(TopicTweetResponse(Map.empty[Long, Seq[TopicTweet]]))\n      }\n  }\n\n  handle(t.CrMixer.GetUtegTweetRecommendations) {\n    args: t.CrMixer.GetUtegTweetRecommendations.Args =>\n      val endpointName = \"getUtegTweetRecommendations\"\n\n      val requestUUID = generateRequestUUID()\n      val startTime = Time.now.inMilliseconds\n      val queryFut = buildUtegTweetQuery(args.request, requestUUID)\n      queryFut\n        .flatMap { query =>\n          val scribeMetadata = ScribeMetadata.from(query)\n          endpointLoadShedder(endpointName, query.product.originalName) {\n            utegTweetScribeLogger.scribeGetUtegTweetRecommendations(\n              args.request,\n              startTime,\n              scribeMetadata,\n              utegTweetCandidateGenerator\n                .get(query)\n                .map(buildUtegTweetResponse)\n            )\n          }.rescue {\n            case e =>\n              logErrMessage(endpointName, e)\n              Future(UtegTweetResponse(Seq.empty))\n          }\n        }\n  }\n\n  handle(t.CrMixer.GetAdsRecommendations) { args: t.CrMixer.GetAdsRecommendations.Args =>\n    val endpointName = \"getAdsRecommendations\"\n    val queryFut = buildAdsCandidateGeneratorQuery(args.request)\n    val startTime = Time.now.inMilliseconds\n    queryFut.flatMap { query =>\n      {\n        val scribeMetadata = ScribeMetadata.from(query)\n        val response = adsCandidateGenerator\n          .get(query).map { candidates =>\n            buildAdsResponse(candidates)\n          }\n        adsRecommendationsScribeLogger.scribeGetAdsRecommendations(\n          args.request,\n          startTime,\n          scribeMetadata,\n          response,\n          query.params(AdsParams.EnableScribe)\n        )\n      }.rescue {\n        case e =>\n          logErrMessage(endpointName, e)\n          Future(AdsResponse(Seq.empty))\n      }\n    }\n\n  }\n\n  private def buildCrCandidateGeneratorQuery(\n    thriftRequest: CrMixerTweetRequest,\n    requestUUID: Long,\n    userId: Long\n  ): Future[CrCandidateGeneratorQuery] = {\n\n    val product = thriftRequest.product\n    val productContext = thriftRequest.productContext\n    val scopedStats = statsReceiver\n      .scope(product.toString).scope(\"CrMixerTweetRequest\")\n\n    userStateStore\n      .get(userId).map { userStateOpt =>\n        val userState = userStateOpt\n          .getOrElse(UserState.EnumUnknownUserState(100))\n        scopedStats.scope(\"UserState\").counter(userState.toString).incr()\n\n        val params =\n          paramsBuilder.buildFromClientContext(\n            thriftRequest.clientContext,\n            thriftRequest.product,\n            userState\n          )\n\n        // Specify product-specific behavior mapping here\n        val maxNumResults = (product, productContext) match {\n          case (t.Product.Home, Some(t.ProductContext.HomeContext(homeContext))) =>\n            homeContext.maxResults.getOrElse(9999)\n          case (t.Product.Notifications, Some(t.ProductContext.NotificationsContext(cxt))) =>\n            params(GlobalParams.MaxCandidatesPerRequestParam)\n          case (t.Product.Email, None) =>\n            params(GlobalParams.MaxCandidatesPerRequestParam)\n          case (t.Product.ImmersiveMediaViewer, None) =>\n            params(GlobalParams.MaxCandidatesPerRequestParam)\n          case (t.Product.VideoCarousel, None) =>\n            params(GlobalParams.MaxCandidatesPerRequestParam)\n          case _ =>\n            throw new IllegalArgumentException(\n              s\"Product ${product} and ProductContext ${productContext} are not allowed in CrMixer\"\n            )\n        }\n\n        CrCandidateGeneratorQuery(\n          userId = userId,\n          product = product,\n          userState = userState,\n          maxNumResults = maxNumResults,\n          impressedTweetList = thriftRequest.excludedTweetIds.getOrElse(Nil).toSet,\n          params = params,\n          requestUUID = requestUUID,\n          languageCode = thriftRequest.clientContext.languageCode\n        )\n      }\n  }\n\n  private def buildRelatedTweetQuery(\n    thriftRequest: RelatedTweetRequest,\n    requestUUID: Long\n  ): Future[RelatedTweetCandidateGeneratorQuery] = {\n\n    val product = thriftRequest.product\n    val scopedStats = statsReceiver\n      .scope(product.toString).scope(\"RelatedTweetRequest\")\n    val userStateFut: Future[UserState] = (thriftRequest.clientContext.userId match {\n      case Some(userId) => userStateStore.get(userId)\n      case None => Future.value(Some(UserState.EnumUnknownUserState(100)))\n    }).map(_.getOrElse(UserState.EnumUnknownUserState(100)))\n\n    userStateFut.map { userState =>\n      scopedStats.scope(\"UserState\").counter(userState.toString).incr()\n      val params =\n        paramsBuilder.buildFromClientContext(\n          thriftRequest.clientContext,\n          thriftRequest.product,\n          userState)\n\n      // Specify product-specific behavior mapping here\n      // Currently, Home takes 10, and RUX takes 100\n      val maxNumResults = params(RelatedTweetGlobalParams.MaxCandidatesPerRequestParam)\n\n      RelatedTweetCandidateGeneratorQuery(\n        internalId = thriftRequest.internalId,\n        clientContext = thriftRequest.clientContext,\n        product = product,\n        maxNumResults = maxNumResults,\n        impressedTweetList = thriftRequest.excludedTweetIds.getOrElse(Nil).toSet,\n        params = params,\n        requestUUID = requestUUID\n      )\n    }\n  }\n\n  private def buildAdsCandidateGeneratorQuery(\n    thriftRequest: AdsRequest\n  ): Future[AdsCandidateGeneratorQuery] = {\n    val userId = thriftRequest.clientContext.userId.getOrElse(\n      throw new IllegalArgumentException(\"userId must be present in the Thrift clientContext\")\n    )\n    val product = thriftRequest.product\n    val requestUUID = generateRequestUUID()\n    userStateStore\n      .get(userId).map { userStateOpt =>\n        val userState = userStateOpt\n          .getOrElse(UserState.EnumUnknownUserState(100))\n        val params =\n          paramsBuilder.buildFromClientContext(\n            thriftRequest.clientContext,\n            thriftRequest.product,\n            userState)\n        val maxNumResults = params(AdsParams.AdsCandidateGenerationMaxCandidatesNumParam)\n        AdsCandidateGeneratorQuery(\n          userId = userId,\n          product = product,\n          userState = userState,\n          params = params,\n          maxNumResults = maxNumResults,\n          requestUUID = requestUUID\n        )\n      }\n  }\n\n  private def buildRelatedVideoTweetQuery(\n    thriftRequest: RelatedVideoTweetRequest,\n    requestUUID: Long\n  ): Future[RelatedVideoTweetCandidateGeneratorQuery] = {\n\n    val product = thriftRequest.product\n    val scopedStats = statsReceiver\n      .scope(product.toString).scope(\"RelatedVideoTweetRequest\")\n    val userStateFut: Future[UserState] = (thriftRequest.clientContext.userId match {\n      case Some(userId) => userStateStore.get(userId)\n      case None => Future.value(Some(UserState.EnumUnknownUserState(100)))\n    }).map(_.getOrElse(UserState.EnumUnknownUserState(100)))\n\n    userStateFut.map { userState =>\n      scopedStats.scope(\"UserState\").counter(userState.toString).incr()\n      val params =\n        paramsBuilder.buildFromClientContext(\n          thriftRequest.clientContext,\n          thriftRequest.product,\n          userState)\n\n      val maxNumResults = params(RelatedVideoTweetGlobalParams.MaxCandidatesPerRequestParam)\n\n      RelatedVideoTweetCandidateGeneratorQuery(\n        internalId = thriftRequest.internalId,\n        clientContext = thriftRequest.clientContext,\n        product = product,\n        maxNumResults = maxNumResults,\n        impressedTweetList = thriftRequest.excludedTweetIds.getOrElse(Nil).toSet,\n        params = params,\n        requestUUID = requestUUID\n      )\n    }\n\n  }\n\n  private def buildUtegTweetQuery(\n    thriftRequest: UtegTweetRequest,\n    requestUUID: Long\n  ): Future[UtegTweetCandidateGeneratorQuery] = {\n\n    val userId = thriftRequest.clientContext.userId.getOrElse(\n      throw new IllegalArgumentException(\"userId must be present in the Thrift clientContext\")\n    )\n    val product = thriftRequest.product\n    val productContext = thriftRequest.productContext\n    val scopedStats = statsReceiver\n      .scope(product.toString).scope(\"UtegTweetRequest\")\n\n    userStateStore\n      .get(userId).map { userStateOpt =>\n        val userState = userStateOpt\n          .getOrElse(UserState.EnumUnknownUserState(100))\n        scopedStats.scope(\"UserState\").counter(userState.toString).incr()\n\n        val params =\n          paramsBuilder.buildFromClientContext(\n            thriftRequest.clientContext,\n            thriftRequest.product,\n            userState\n          )\n\n        // Specify product-specific behavior mapping here\n        val maxNumResults = (product, productContext) match {\n          case (t.Product.Home, Some(t.ProductContext.HomeContext(homeContext))) =>\n            homeContext.maxResults.getOrElse(9999)\n          case _ =>\n            throw new IllegalArgumentException(\n              s\"Product ${product} and ProductContext ${productContext} are not allowed in CrMixer\"\n            )\n        }\n\n        UtegTweetCandidateGeneratorQuery(\n          userId = userId,\n          product = product,\n          userState = userState,\n          maxNumResults = maxNumResults,\n          impressedTweetList = thriftRequest.excludedTweetIds.getOrElse(Nil).toSet,\n          params = params,\n          requestUUID = requestUUID\n        )\n      }\n\n  }\n\n  private def buildTopicTweetQuery(\n    thriftRequest: TopicTweetRequest,\n    requestUUID: Long\n  ): TopicTweetCandidateGeneratorQuery = {\n    val userId = thriftRequest.clientContext.userId.getOrElse(\n      throw new IllegalArgumentException(\n        \"userId must be present in the TopicTweetRequest clientContext\")\n    )\n    val product = thriftRequest.product\n    val productContext = thriftRequest.productContext\n\n    // Specify product-specific behavior mapping here\n    val isVideoOnly = (product, productContext) match {\n      case (t.Product.ExploreTopics, Some(t.ProductContext.ExploreContext(context))) =>\n        context.isVideoOnly\n      case (t.Product.TopicLandingPage, None) =>\n        false\n      case (t.Product.HomeTopicsBackfill, None) =>\n        false\n      case (t.Product.TopicTweetsStrato, None) =>\n        false\n      case _ =>\n        throw new IllegalArgumentException(\n          s\"Product ${product} and ProductContext ${productContext} are not allowed in CrMixer\"\n        )\n    }\n\n    statsReceiver.scope(product.toString).counter(TopicTweetRequest.toString).incr()\n\n    val params =\n      paramsBuilder.buildFromClientContext(\n        thriftRequest.clientContext,\n        product,\n        UserState.EnumUnknownUserState(100)\n      )\n\n    val topicIds = thriftRequest.topicIds.map { topicId =>\n      TopicId(\n        entityId = topicId,\n        language = thriftRequest.clientContext.languageCode,\n        country = None\n      )\n    }.toSet\n\n    TopicTweetCandidateGeneratorQuery(\n      userId = userId,\n      topicIds = topicIds,\n      product = product,\n      maxNumResults = params(TopicTweetParams.MaxTopicTweetCandidatesParam),\n      impressedTweetList = thriftRequest.excludedTweetIds.getOrElse(Nil).toSet,\n      params = params,\n      requestUUID = requestUUID,\n      isVideoOnly = isVideoOnly\n    )\n  }\n\n  private def buildFrsBasedTweetQuery(\n    thriftRequest: FrsTweetRequest,\n    requestUUID: Long\n  ): Future[FrsTweetCandidateGeneratorQuery] = {\n    val userId = thriftRequest.clientContext.userId.getOrElse(\n      throw new IllegalArgumentException(\n        \"userId must be present in the FrsTweetRequest clientContext\")\n    )\n    val product = thriftRequest.product\n    val productContext = thriftRequest.productContext\n\n    val scopedStats = statsReceiver\n      .scope(product.toString).scope(\"FrsTweetRequest\")\n\n    userStateStore\n      .get(userId).map { userStateOpt =>\n        val userState = userStateOpt\n          .getOrElse(UserState.EnumUnknownUserState(100))\n        scopedStats.scope(\"UserState\").counter(userState.toString).incr()\n\n        val params =\n          paramsBuilder.buildFromClientContext(\n            thriftRequest.clientContext,\n            thriftRequest.product,\n            userState\n          )\n        val maxNumResults = (product, productContext) match {\n          case (t.Product.Home, Some(t.ProductContext.HomeContext(homeContext))) =>\n            homeContext.maxResults.getOrElse(\n              params(FrsBasedCandidateGenerationMaxCandidatesNumParam))\n          case _ =>\n            params(FrsBasedCandidateGenerationMaxCandidatesNumParam)\n        }\n\n        FrsTweetCandidateGeneratorQuery(\n          userId = userId,\n          product = product,\n          maxNumResults = maxNumResults,\n          impressedTweetList = thriftRequest.excludedTweetIds.getOrElse(Nil).toSet,\n          impressedUserList = thriftRequest.excludedUserIds.getOrElse(Nil).toSet,\n          params = params,\n          languageCodeOpt = thriftRequest.clientContext.languageCode,\n          countryCodeOpt = thriftRequest.clientContext.countryCode,\n          requestUUID = requestUUID\n        )\n      }\n  }\n\n  private def buildThriftResponse(\n    candidates: Seq[RankedCandidate]\n  ): CrMixerTweetResponse = {\n\n    val tweets = candidates.map { candidate =>\n      TweetRecommendation(\n        tweetId = candidate.tweetId,\n        score = candidate.predictionScore,\n        metricTags = Some(MetricTagUtil.buildMetricTags(candidate)),\n        latestSourceSignalTimestampInMillis =\n          SignalTimestampStatsUtil.buildLatestSourceSignalTimestamp(candidate)\n      )\n    }\n    signalTimestampStatsUtil.statsSignalTimestamp(tweets)\n    CrMixerTweetResponse(tweets)\n  }\n\n  private def scribeTweetScoreFunnelSeries(\n    candidates: Seq[RankedCandidate]\n  ): Seq[RankedCandidate] = {\n    // 202210210901 is a random number for code search of Lensview\n    tweetScoreFunnelSeries.startNewSpan(\n      name = \"GetTweetRecommendationsTopLevelTweetSimilarityEngineType\",\n      codePtr = 202210210901L) {\n      (\n        candidates,\n        candidates.map { candidate =>\n          thriftlog.TweetDimensionMeasure(\n            dimension = Some(\n              thriftlog\n                .RequestTweetDimension(\n                  candidate.tweetId,\n                  candidate.reasonChosen.similarityEngineInfo.similarityEngineType.value)),\n            measure = Some(thriftlog.RequestTweetMeasure(candidate.predictionScore))\n          )\n        }\n      )\n    }\n  }\n\n  private def buildRelatedTweetResponse(candidates: Seq[InitialCandidate]): RelatedTweetResponse = {\n    val tweets = candidates.map { candidate =>\n      RelatedTweet(\n        tweetId = candidate.tweetId,\n        score = Some(candidate.getSimilarityScore),\n        authorId = Some(candidate.tweetInfo.authorId)\n      )\n    }\n    RelatedTweetResponse(tweets)\n  }\n\n  private def buildRelatedVideoTweetResponse(\n    candidates: Seq[InitialCandidate]\n  ): RelatedVideoTweetResponse = {\n    val tweets = candidates.map { candidate =>\n      RelatedVideoTweet(\n        tweetId = candidate.tweetId,\n        score = Some(candidate.getSimilarityScore)\n      )\n    }\n    RelatedVideoTweetResponse(tweets)\n  }\n\n  private def buildUtegTweetResponse(\n    candidates: Seq[TweetWithScoreAndSocialProof]\n  ): UtegTweetResponse = {\n    val tweets = candidates.map { candidate =>\n      UtegTweet(\n        tweetId = candidate.tweetId,\n        score = candidate.score,\n        socialProofByType = candidate.socialProofByType\n      )\n    }\n    UtegTweetResponse(tweets)\n  }\n\n  private def buildAdsResponse(\n    candidates: Seq[RankedAdsCandidate]\n  ): AdsResponse = {\n    AdsResponse(ads = candidates.map { candidate =>\n      AdTweetRecommendation(\n        tweetId = candidate.tweetId,\n        score = candidate.predictionScore,\n        lineItems = Some(candidate.lineItemInfo))\n    })\n  }\n\n  private def cacheTweetRecommendationResults(\n    request: CrMixerTweetRequest,\n    response: Future[CrMixerTweetResponse]\n  ): Unit = {\n\n    val userId = request.clientContext.userId.getOrElse(\n      throw new IllegalArgumentException(\n        \"userId must be present in getTweetRecommendations() Thrift clientContext\"))\n\n    if (decider.isAvailableForId(userId, DeciderConstants.getTweetRecommendationsCacheRate)) {\n      response.map { crMixerTweetResponse =>\n        {\n          (\n            request.product,\n            request.clientContext.userId,\n            crMixerTweetResponse.tweets.nonEmpty) match {\n            case (t.Product.Home, Some(userId), true) =>\n              tweetRecommendationResultsStore.put((userId, crMixerTweetResponse))\n            case _ => Future.value(Unit)\n          }\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/exception/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/exception/InvalidSANNConfigException.scala",
    "content": "package com.twitter.cr_mixer\npackage exception\n\ncase class InvalidSANNConfigException(msg: String) extends Exception(msg)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/featureswitch/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"abdecider/src/main/scala\",\n        \"configapi/configapi-abdecider\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-featureswitches:v2\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"decider/src/main/scala\",\n        \"discovery-common/src/main/scala/com/twitter/discovery/common/configapi\",\n        \"featureswitches/featureswitches-core\",\n        \"featureswitches/featureswitches-core/src/main/scala/com/twitter/featureswitches/v2/builder\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"frigate/frigate-common:util\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/base\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/candidate\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/health\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/interests\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/strato\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/util\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"scribelib/marshallers/src/main/scala/com/twitter/scribelib/marshallers\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/core_workflows/user_model:user_model-scala\",\n        \"src/thrift/com/twitter/frigate:frigate-common-thrift-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/featureswitch/CrMixerLoggingABDecider.scala",
    "content": "package com.twitter.cr_mixer\npackage featureswitch\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.abdecider.Recipient\nimport com.twitter.abdecider.Bucket\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.util.Local\nimport scala.collection.concurrent.{Map => ConcurrentMap}\n\n/**\n * Wraps a LoggingABDecider, so all impressed buckets are recorded to a 'LocalContext' on a given request.\n *\n * Contexts (https://twitter.github.io/finagle/guide/Contexts.html) are Finagle's mechanism for\n * storing state/variables without having to pass these variables all around the request.\n *\n * In order for this class to be used the [[SetImpressedBucketsLocalContextFilter]] must be applied\n * at the beginning of the request, to initialize a concurrent map used to store impressed buckets.\n *\n * Whenever we get an a/b impression, the bucket information is logged to the concurrent hashmap.\n */\ncase class CrMixerLoggingABDecider(\n  loggingAbDecider: LoggingABDecider,\n  statsReceiver: StatsReceiver)\n    extends LoggingABDecider {\n\n  private val scopedStatsReceiver = statsReceiver.scope(\"cr_logging_ab_decider\")\n\n  override def impression(\n    experimentName: String,\n    recipient: Recipient\n  ): Option[Bucket] = {\n\n    StatsUtil.trackNonFutureBlockStats(scopedStatsReceiver.scope(\"log_impression\")) {\n      val maybeBuckets = loggingAbDecider.impression(experimentName, recipient)\n      maybeBuckets.foreach { b =>\n        scopedStatsReceiver.counter(\"impressions\").incr()\n        CrMixerImpressedBuckets.recordImpressedBucket(b)\n      }\n      maybeBuckets\n    }\n  }\n\n  override def track(\n    experimentName: String,\n    eventName: String,\n    recipient: Recipient\n  ): Unit = {\n    loggingAbDecider.track(experimentName, eventName, recipient)\n  }\n\n  override def bucket(\n    experimentName: String,\n    recipient: Recipient\n  ): Option[Bucket] = {\n    loggingAbDecider.bucket(experimentName, recipient)\n  }\n\n  override def experiments: Seq[String] = loggingAbDecider.experiments\n\n  override def experiment(experimentName: String) =\n    loggingAbDecider.experiment(experimentName)\n}\n\nobject CrMixerImpressedBuckets {\n  private[featureswitch] val localImpressedBucketsMap = new Local[ConcurrentMap[Bucket, Boolean]]\n\n  /**\n   * Gets all impressed buckets for this request.\n   **/\n  def getAllImpressedBuckets: Option[List[Bucket]] = {\n    localImpressedBucketsMap.apply().map(_.map { case (k, _) => k }.toList)\n  }\n\n  private[featureswitch] def recordImpressedBucket(bucket: Bucket) = {\n    localImpressedBucketsMap().foreach { m => m += bucket -> true }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/featureswitch/ParamsBuilder.scala",
    "content": "package com.twitter.cr_mixer.featureswitch\n\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.abdecider.UserRecipient\nimport com.twitter.cr_mixer.{thriftscala => t}\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.discovery.common.configapi.FeatureContextBuilder\nimport com.twitter.featureswitches.FSRecipient\nimport com.twitter.featureswitches.UserAgent\nimport com.twitter.featureswitches.{Recipient => FeatureSwitchRecipient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.thriftscala.ClientContext\nimport com.twitter.timelines.configapi.Config\nimport com.twitter.timelines.configapi.FeatureValue\nimport com.twitter.timelines.configapi.ForcedFeatureContext\nimport com.twitter.timelines.configapi.OrElseFeatureContext\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.timelines.configapi.RequestContext\nimport com.twitter.timelines.configapi.abdecider.LoggingABDeciderExperimentContext\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/** Singleton object for building [[Params]] to override */\n@Singleton\nclass ParamsBuilder @Inject() (\n  globalStats: StatsReceiver,\n  abDecider: LoggingABDecider,\n  featureContextBuilder: FeatureContextBuilder,\n  config: Config) {\n\n  private val stats = globalStats.scope(\"params\")\n\n  def buildFromClientContext(\n    clientContext: ClientContext,\n    product: t.Product,\n    userState: UserState,\n    userRoleOverride: Option[Set[String]] = None,\n    featureOverrides: Map[String, FeatureValue] = Map.empty,\n  ): Params = {\n    clientContext.userId match {\n      case Some(userId) =>\n        val userRecipient = buildFeatureSwitchRecipient(\n          userId,\n          userRoleOverride,\n          clientContext,\n          product,\n          userState\n        )\n\n        val featureContext = OrElseFeatureContext(\n          ForcedFeatureContext(featureOverrides),\n          featureContextBuilder(\n            Some(userId),\n            Some(userRecipient)\n          ))\n\n        config(\n          requestContext = RequestContext(\n            userId = Some(userId),\n            experimentContext = LoggingABDeciderExperimentContext(\n              abDecider,\n              Some(UserRecipient(userId, Some(userId)))),\n            featureContext = featureContext\n          ),\n          stats\n        )\n      case None =>\n        val guestRecipient =\n          buildFeatureSwitchRecipientWithGuestId(clientContext: ClientContext, product, userState)\n\n        val featureContext = OrElseFeatureContext(\n          ForcedFeatureContext(featureOverrides),\n          featureContextBuilder(\n            clientContext.userId,\n            Some(guestRecipient)\n          )\n        ) //ExperimentContext with GuestRecipient is not supported  as there is no active use-cases yet in CrMixer\n\n        config(\n          requestContext = RequestContext(\n            userId = clientContext.userId,\n            featureContext = featureContext\n          ),\n          stats\n        )\n    }\n  }\n\n  private def buildFeatureSwitchRecipientWithGuestId(\n    clientContext: ClientContext,\n    product: t.Product,\n    userState: UserState\n  ): FeatureSwitchRecipient = {\n\n    val recipient = FSRecipient(\n      userId = None,\n      userRoles = None,\n      deviceId = clientContext.deviceId,\n      guestId = clientContext.guestId,\n      languageCode = clientContext.languageCode,\n      countryCode = clientContext.countryCode,\n      userAgent = clientContext.userAgent.flatMap(UserAgent(_)),\n      isVerified = None,\n      isTwoffice = None,\n      tooClient = None,\n      highWaterMark = None\n    )\n\n    recipient.withCustomFields(\n      (ParamsBuilder.ProductCustomField, product.toString),\n      (ParamsBuilder.UserStateCustomField, userState.toString)\n    )\n  }\n\n  private def buildFeatureSwitchRecipient(\n    userId: Long,\n    userRolesOverride: Option[Set[String]],\n    clientContext: ClientContext,\n    product: t.Product,\n    userState: UserState\n  ): FeatureSwitchRecipient = {\n    val userRoles = userRolesOverride match {\n      case Some(overrides) => Some(overrides)\n      case _ => clientContext.userRoles.map(_.toSet)\n    }\n\n    val recipient = FSRecipient(\n      userId = Some(userId),\n      userRoles = userRoles,\n      deviceId = clientContext.deviceId,\n      guestId = clientContext.guestId,\n      languageCode = clientContext.languageCode,\n      countryCode = clientContext.countryCode,\n      userAgent = clientContext.userAgent.flatMap(UserAgent(_)),\n      isVerified = None,\n      isTwoffice = None,\n      tooClient = None,\n      highWaterMark = None\n    )\n\n    recipient.withCustomFields(\n      (ParamsBuilder.ProductCustomField, product.toString),\n      (ParamsBuilder.UserStateCustomField, userState.toString)\n    )\n  }\n}\n\nobject ParamsBuilder {\n  private val ProductCustomField = \"product_id\"\n  private val UserStateCustomField = \"user_state\"\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/featureswitch/SetImpressedBucketsLocalContextFilter.scala",
    "content": "package com.twitter.cr_mixer.featureswitch\n\nimport com.twitter.finagle.Filter\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.concurrent.TrieMap\nimport com.twitter.abdecider.Bucket\nimport com.twitter.finagle.Service\n\n@Singleton\nclass SetImpressedBucketsLocalContextFilter @Inject() () extends Filter.TypeAgnostic {\n  override def toFilter[Req, Rep]: Filter[Req, Rep, Req, Rep] =\n    (request: Req, service: Service[Req, Rep]) => {\n\n      val concurrentTrieMap = TrieMap\n        .empty[Bucket, Boolean] // Trie map has no locks and O(1) inserts\n      CrMixerImpressedBuckets.localImpressedBucketsMap.let(concurrentTrieMap) {\n        service(request)\n      }\n    }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"content-recommender/thrift/src/main/thrift:thrift-scala\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"finagle/finagle-core/src/main\",\n        \"frigate/frigate-common:util\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/core_workflows/user_model:user_model-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/FilterBase.scala",
    "content": "package com.twitter.cr_mixer.filter\n\nimport com.twitter.cr_mixer.model.CandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.util.Future\n\ntrait FilterBase {\n  def name: String\n\n  type ConfigType\n\n  def filter(\n    candidates: Seq[Seq[InitialCandidate]],\n    config: ConfigType\n  ): Future[Seq[Seq[InitialCandidate]]]\n\n  /**\n   * Build the config params here. passing in param() into the filter is strongly discouraged\n   * because param() can be slow when called many times\n   */\n  def requestToConfig[CGQueryType <: CandidateGeneratorQuery](request: CGQueryType): ConfigType\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/ImpressedTweetlistFilter.scala",
    "content": "package com.twitter.cr_mixer.filter\n\nimport com.twitter.cr_mixer.model.CandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n@Singleton\ncase class ImpressedTweetlistFilter() extends FilterBase {\n  import ImpressedTweetlistFilter._\n\n  override val name: String = this.getClass.getCanonicalName\n\n  override type ConfigType = FilterConfig\n\n  /*\n   Filtering removes some candidates based on configurable criteria.\n   */\n  override def filter(\n    candidates: Seq[Seq[InitialCandidate]],\n    config: FilterConfig\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    // Remove candidates which match a source tweet, or which are passed in impressedTweetList\n    val sourceTweetsMatch = candidates\n      .flatMap {\n\n        /***\n         * Within a Seq[Seq[InitialCandidate]], all candidates within a inner Seq\n         * are guaranteed to have the same sourceInfo. Hence, we can pick .headOption\n         * to represent the whole list when filtering by the internalId of the sourceInfoOpt.\n         * But of course the similarityEngineInfo could be different.\n         */\n        _.headOption.flatMap { candidate =>\n          candidate.candidateGenerationInfo.sourceInfoOpt.map(_.internalId)\n        }\n      }.collect {\n        case InternalId.TweetId(id) => id\n      }\n\n    val impressedTweetList: Set[TweetId] =\n      config.impressedTweetList ++ sourceTweetsMatch\n\n    val filteredCandidateMap: Seq[Seq[InitialCandidate]] =\n      candidates.map {\n        _.filterNot { candidate =>\n          impressedTweetList.contains(candidate.tweetId)\n        }\n      }\n    Future.value(filteredCandidateMap)\n  }\n\n  override def requestToConfig[CGQueryType <: CandidateGeneratorQuery](\n    request: CGQueryType\n  ): FilterConfig = {\n    FilterConfig(request.impressedTweetList)\n  }\n}\n\nobject ImpressedTweetlistFilter {\n  case class FilterConfig(impressedTweetList: Set[TweetId])\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/InNetworkFilter.scala",
    "content": "package com.twitter.cr_mixer.filter\n\nimport com.twitter.cr_mixer.model.CandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.UtegTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.param.UtegTweetGlobalParams\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport com.twitter.wtf.candidate.thriftscala.CandidateSeq\n\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/***\n * Filters in-network tweets\n */\n@Singleton\ncase class InNetworkFilter @Inject() (\n  @Named(ModuleNames.RealGraphInStore) realGraphStoreMh: ReadableStore[UserId, CandidateSeq],\n  globalStats: StatsReceiver)\n    extends FilterBase {\n  override val name: String = this.getClass.getCanonicalName\n  import InNetworkFilter._\n\n  override type ConfigType = FilterConfig\n  private val stats: StatsReceiver = globalStats.scope(this.getClass.getCanonicalName)\n  private val filterCandidatesStats = stats.scope(\"filter_candidates\")\n\n  override def filter(\n    candidates: Seq[Seq[InitialCandidate]],\n    filterConfig: FilterConfig,\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    StatsUtil.trackItemsStats(filterCandidatesStats) {\n      filterCandidates(candidates, filterConfig)\n    }\n  }\n\n  private def filterCandidates(\n    candidates: Seq[Seq[InitialCandidate]],\n    filterConfig: FilterConfig,\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n\n    if (!filterConfig.enableInNetworkFilter) {\n      Future.value(candidates)\n    } else {\n      filterConfig.userIdOpt match {\n        case Some(userId) =>\n          realGraphStoreMh\n            .get(userId).map(_.map(_.candidates.map(_.userId)).getOrElse(Seq.empty).toSet).map {\n              realGraphInNetworkAuthorsSet =>\n                candidates.map(_.filterNot { candidate =>\n                  realGraphInNetworkAuthorsSet.contains(candidate.tweetInfo.authorId)\n                })\n            }\n        case None => Future.value(candidates)\n      }\n    }\n  }\n\n  override def requestToConfig[CGQueryType <: CandidateGeneratorQuery](\n    request: CGQueryType\n  ): FilterConfig = {\n    request match {\n      case UtegTweetCandidateGeneratorQuery(userId, _, _, _, _, params, _) =>\n        FilterConfig(Some(userId), params(UtegTweetGlobalParams.EnableInNetworkFilterParam))\n      case _ => FilterConfig(None, false)\n    }\n  }\n}\n\nobject InNetworkFilter {\n  case class FilterConfig(\n    userIdOpt: Option[UserId],\n    enableInNetworkFilter: Boolean)\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/PostRankFilterRunner.scala",
    "content": "package com.twitter.cr_mixer.filter\nimport com.twitter.cr_mixer.model.CrCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.RankedCandidate\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class PostRankFilterRunner @Inject() (\n  globalStats: StatsReceiver) {\n\n  private val scopedStats = globalStats.scope(this.getClass.getCanonicalName)\n\n  private val beforeCount = scopedStats.stat(\"candidate_count\", \"before\")\n  private val afterCount = scopedStats.stat(\"candidate_count\", \"after\")\n\n  def run(\n    query: CrCandidateGeneratorQuery,\n    candidates: Seq[RankedCandidate]\n  ): Future[Seq[RankedCandidate]] = {\n\n    beforeCount.add(candidates.size)\n\n    Future(\n      removeBadRecentNotificationCandidates(candidates)\n    ).map { results =>\n      afterCount.add(results.size)\n      results\n    }\n  }\n\n  /**\n   * Remove \"bad\" quality candidates generated by recent notifications\n   * A candidate is bad when it is generated by a single RecentNotification\n   * SourceKey.\n   * e.x:\n   * tweetA {recent notification1} -> bad\n   * tweetB {recent notification1 recent notification2} -> good\n   *tweetC {recent notification1 recent follow1} -> bad\n   * SD-19397\n   */\n  private[filter] def removeBadRecentNotificationCandidates(\n    candidates: Seq[RankedCandidate]\n  ): Seq[RankedCandidate] = {\n    candidates.filterNot {\n      isBadQualityRecentNotificationCandidate\n    }\n  }\n\n  private def isBadQualityRecentNotificationCandidate(candidate: RankedCandidate): Boolean = {\n    candidate.potentialReasons.size == 1 &&\n    candidate.potentialReasons.head.sourceInfoOpt.nonEmpty &&\n    candidate.potentialReasons.head.sourceInfoOpt.get.sourceType == SourceType.NotificationClick\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/PreRankFilterRunner.scala",
    "content": "package com.twitter.cr_mixer.filter\n\nimport com.twitter.cr_mixer.model.CandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PreRankFilterRunner @Inject() (\n  impressedTweetListFilter: ImpressedTweetlistFilter,\n  tweetAgeFilter: TweetAgeFilter,\n  videoTweetFilter: VideoTweetFilter,\n  tweetReplyFilter: ReplyFilter,\n  globalStats: StatsReceiver) {\n\n  private val scopedStats = globalStats.scope(this.getClass.getCanonicalName)\n\n  /***\n   * The order of the filters does not matter as long as we do not apply .take(N) truncation\n   * across all filters. In other words, it is fine that we first do tweetAgeFilter, and then\n   * we do impressedTweetListFilter, or the other way around.\n   * Same idea applies to the signal based filter - it is ok that we apply signal based filters\n   * before impressedTweetListFilter.\n   *\n   * We move all signal based filters before tweetAgeFilter and impressedTweetListFilter\n   * as a set of early filters.\n   */\n  val orderedFilters = Seq(\n    tweetAgeFilter,\n    impressedTweetListFilter,\n    videoTweetFilter,\n    tweetReplyFilter\n  )\n\n  def runSequentialFilters[CGQueryType <: CandidateGeneratorQuery](\n    request: CGQueryType,\n    candidates: Seq[Seq[InitialCandidate]],\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    PreRankFilterRunner.runSequentialFilters(\n      request,\n      candidates,\n      orderedFilters,\n      scopedStats\n    )\n  }\n\n}\n\nobject PreRankFilterRunner {\n  private def recordCandidateStatsBeforeFilter(\n    candidates: Seq[Seq[InitialCandidate]],\n    statsReceiver: StatsReceiver\n  ): Unit = {\n    statsReceiver\n      .counter(\"empty_sources\", \"before\").incr(\n        candidates.count { _.isEmpty }\n      )\n    candidates.foreach { candidate =>\n      statsReceiver.counter(\"candidates\", \"before\").incr(candidate.size)\n    }\n  }\n\n  private def recordCandidateStatsAfterFilter(\n    candidates: Seq[Seq[InitialCandidate]],\n    statsReceiver: StatsReceiver\n  ): Unit = {\n    statsReceiver\n      .counter(\"empty_sources\", \"after\").incr(\n        candidates.count { _.isEmpty }\n      )\n    candidates.foreach { candidate =>\n      statsReceiver.counter(\"candidates\", \"after\").incr(candidate.size)\n    }\n  }\n\n  /*\n  Helper function for running some candidates through a sequence of filters\n   */\n  private[filter] def runSequentialFilters[CGQueryType <: CandidateGeneratorQuery](\n    request: CGQueryType,\n    candidates: Seq[Seq[InitialCandidate]],\n    filters: Seq[FilterBase],\n    statsReceiver: StatsReceiver\n  ): Future[Seq[Seq[InitialCandidate]]] =\n    filters.foldLeft(Future.value(candidates)) {\n      case (candsFut, filter) =>\n        candsFut.flatMap { cands =>\n          recordCandidateStatsBeforeFilter(cands, statsReceiver.scope(filter.name))\n          filter\n            .filter(cands, filter.requestToConfig(request))\n            .map { filteredCands =>\n              recordCandidateStatsAfterFilter(filteredCands, statsReceiver.scope(filter.name))\n              filteredCands\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/ReplyFilter.scala",
    "content": "package com.twitter.cr_mixer.filter\n\nimport com.twitter.cr_mixer.model.CandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.util.Future\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/***\n * Filters candidates that are replies\n */\n@Singleton\ncase class ReplyFilter @Inject() () extends FilterBase {\n  override def name: String = this.getClass.getCanonicalName\n  override type ConfigType = Boolean\n\n  override def filter(\n    candidates: Seq[Seq[InitialCandidate]],\n    config: ConfigType\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    if (config) {\n      Future.value(\n        candidates.map { candidateSeq =>\n          candidateSeq.filterNot { candidate =>\n            candidate.tweetInfo.isReply.getOrElse(false)\n          }\n        }\n      )\n    } else {\n      Future.value(candidates)\n    }\n  }\n\n  override def requestToConfig[CGQueryType <: CandidateGeneratorQuery](\n    query: CGQueryType\n  ): ConfigType = {\n    true\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/RetweetFilter.scala",
    "content": "package com.twitter.cr_mixer.filter\n\nimport com.twitter.cr_mixer.model.CandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.param.UtegTweetGlobalParams\nimport com.twitter.util.Future\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/***\n * Filters candidates that are retweets\n */\n@Singleton\ncase class RetweetFilter @Inject() () extends FilterBase {\n  override def name: String = this.getClass.getCanonicalName\n  override type ConfigType = Boolean\n\n  override def filter(\n    candidates: Seq[Seq[InitialCandidate]],\n    config: ConfigType\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    if (config) {\n      Future.value(\n        candidates.map { candidateSeq =>\n          candidateSeq.filterNot { candidate =>\n            candidate.tweetInfo.isRetweet.getOrElse(false)\n          }\n        }\n      )\n    } else {\n      Future.value(candidates)\n    }\n  }\n\n  override def requestToConfig[CGQueryType <: CandidateGeneratorQuery](\n    query: CGQueryType\n  ): ConfigType = {\n    query.params(UtegTweetGlobalParams.EnableRetweetFilterParam)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/TweetAgeFilter.scala",
    "content": "package com.twitter.cr_mixer.filter\n\nimport com.twitter.cr_mixer.model.CandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport javax.inject.Singleton\nimport com.twitter.conversions.DurationOps._\n\n@Singleton\ncase class TweetAgeFilter() extends FilterBase {\n  override val name: String = this.getClass.getCanonicalName\n\n  override type ConfigType = Duration\n\n  override def filter(\n    candidates: Seq[Seq[InitialCandidate]],\n    maxTweetAge: Duration\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    if (maxTweetAge >= 720.hours) {\n      Future.value(candidates)\n    } else {\n      // Tweet IDs are approximately chronological (see http://go/snowflake),\n      // so we are building the earliest tweet id once,\n      // and pass that as the value to filter candidates for each CandidateGenerationModel.\n      val earliestTweetId = SnowflakeId.firstIdFor(Time.now - maxTweetAge)\n      Future.value(candidates.map(_.filter(_.tweetId >= earliestTweetId)))\n    }\n  }\n\n  override def requestToConfig[CGQueryType <: CandidateGeneratorQuery](\n    query: CGQueryType\n  ): Duration = {\n    query.params(GlobalParams.MaxTweetAgeHoursParam)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/TweetInfoHealthFilterBase.scala",
    "content": "package com.twitter.cr_mixer.filter\n\nimport com.twitter.contentrecommender.thriftscala.TweetInfo\nimport com.twitter.cr_mixer.model.CandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.CrCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.HealthThreshold\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n@Singleton\ntrait TweetInfoHealthFilterBase extends FilterBase {\n  override def name: String = this.getClass.getCanonicalName\n  override type ConfigType = HealthThreshold.Enum.Value\n  def thresholdToPropertyMap: Map[HealthThreshold.Enum.Value, TweetInfo => Option[Boolean]]\n  def getFilterParamFn: CandidateGeneratorQuery => HealthThreshold.Enum.Value\n\n  override def filter(\n    candidates: Seq[Seq[InitialCandidate]],\n    config: HealthThreshold.Enum.Value\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    Future.value(candidates.map { seq =>\n      seq.filter(p => thresholdToPropertyMap(config)(p.tweetInfo).getOrElse(true))\n    })\n  }\n\n  /**\n   * Build the config params here. passing in param() into the filter is strongly discouraged\n   * because param() can be slow when called many times\n   */\n  override def requestToConfig[CGQueryType <: CandidateGeneratorQuery](\n    query: CGQueryType\n  ): HealthThreshold.Enum.Value = {\n    query match {\n      case q: CrCandidateGeneratorQuery => getFilterParamFn(q)\n      case _ => HealthThreshold.Enum.Off\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/UtegFilterRunner.scala",
    "content": "package com.twitter.cr_mixer.filter\n\nimport com.twitter.cr_mixer.model.CandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.Future\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/***\n *\n * Run filters sequentially for UTEG candidate generator. The structure is copied from PreRankFilterRunner.\n */\n@Singleton\nclass UtegFilterRunner @Inject() (\n  inNetworkFilter: InNetworkFilter,\n  utegHealthFilter: UtegHealthFilter,\n  retweetFilter: RetweetFilter,\n  globalStats: StatsReceiver) {\n\n  private val scopedStats = globalStats.scope(this.getClass.getCanonicalName)\n\n  val orderedFilters: Seq[FilterBase] = Seq(\n    inNetworkFilter,\n    utegHealthFilter,\n    retweetFilter\n  )\n\n  def runSequentialFilters[CGQueryType <: CandidateGeneratorQuery](\n    request: CGQueryType,\n    candidates: Seq[Seq[InitialCandidate]],\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    UtegFilterRunner.runSequentialFilters(\n      request,\n      candidates,\n      orderedFilters,\n      scopedStats\n    )\n  }\n\n}\n\nobject UtegFilterRunner {\n  private def recordCandidateStatsBeforeFilter(\n    candidates: Seq[Seq[InitialCandidate]],\n    statsReceiver: StatsReceiver\n  ): Unit = {\n    statsReceiver\n      .counter(\"empty_sources\", \"before\").incr(\n        candidates.count {\n          _.isEmpty\n        }\n      )\n    candidates.foreach { candidate =>\n      statsReceiver.counter(\"candidates\", \"before\").incr(candidate.size)\n    }\n  }\n\n  private def recordCandidateStatsAfterFilter(\n    candidates: Seq[Seq[InitialCandidate]],\n    statsReceiver: StatsReceiver\n  ): Unit = {\n    statsReceiver\n      .counter(\"empty_sources\", \"after\").incr(\n        candidates.count {\n          _.isEmpty\n        }\n      )\n    candidates.foreach { candidate =>\n      statsReceiver.counter(\"candidates\", \"after\").incr(candidate.size)\n    }\n  }\n\n  /*\n  Helper function for running some candidates through a sequence of filters\n   */\n  private[filter] def runSequentialFilters[CGQueryType <: CandidateGeneratorQuery](\n    request: CGQueryType,\n    candidates: Seq[Seq[InitialCandidate]],\n    filters: Seq[FilterBase],\n    statsReceiver: StatsReceiver\n  ): Future[Seq[Seq[InitialCandidate]]] =\n    filters.foldLeft(Future.value(candidates)) {\n      case (candsFut, filter) =>\n        candsFut.flatMap { cands =>\n          recordCandidateStatsBeforeFilter(cands, statsReceiver.scope(filter.name))\n          filter\n            .filter(cands, filter.requestToConfig(request))\n            .map { filteredCands =>\n              recordCandidateStatsAfterFilter(filteredCands, statsReceiver.scope(filter.name))\n              filteredCands\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/UtegHealthFilter.scala",
    "content": "package com.twitter.cr_mixer.filter\n\nimport com.twitter.cr_mixer.model.CandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.param.UtegTweetGlobalParams\nimport com.twitter.util.Future\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Remove unhealthy candidates\n * Currently Timeline Ranker applies a check on the following three scores:\n *  - toxicityScore\n *  - pBlockScore\n *  - pReportedTweetScore\n *\n * Where isPassTweetHealthFilterStrict checks two additions scores with the same threshold:\n *  - pSpammyTweetScore\n *  - spammyTweetContentScore\n *\n * We've verified that both filters behave very similarly.\n */\n@Singleton\ncase class UtegHealthFilter @Inject() () extends FilterBase {\n  override def name: String = this.getClass.getCanonicalName\n  override type ConfigType = Boolean\n\n  override def filter(\n    candidates: Seq[Seq[InitialCandidate]],\n    config: ConfigType\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    if (config) {\n      Future.value(\n        candidates.map { candidateSeq =>\n          candidateSeq.filter { candidate =>\n            candidate.tweetInfo.isPassTweetHealthFilterStrict.getOrElse(false)\n          }\n        }\n      )\n    } else {\n      Future.value(candidates)\n    }\n  }\n\n  override def requestToConfig[CGQueryType <: CandidateGeneratorQuery](\n    query: CGQueryType\n  ): ConfigType = {\n    query.params(UtegTweetGlobalParams.EnableTLRHealthFilterParam)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/filter/VideoTweetFilter.scala",
    "content": "package com.twitter.cr_mixer.filter\n\nimport com.twitter.cr_mixer.filter.VideoTweetFilter.FilterConfig\nimport com.twitter.cr_mixer.model.CandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.CrCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.model.RelatedTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.RelatedVideoTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.param.VideoTweetFilterParams\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n@Singleton\ncase class VideoTweetFilter() extends FilterBase {\n  override val name: String = this.getClass.getCanonicalName\n\n  override type ConfigType = FilterConfig\n\n  override def filter(\n    candidates: Seq[Seq[InitialCandidate]],\n    config: ConfigType\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    Future.value(candidates.map {\n      _.flatMap {\n        candidate =>\n          if (!config.enableVideoTweetFilter) {\n            Some(candidate)\n          } else {\n            // if hasVideo is true, hasImage, hasGif should be false\n            val hasVideo = checkTweetInfoAttribute(candidate.tweetInfo.hasVideo)\n            val isHighMediaResolution =\n              checkTweetInfoAttribute(candidate.tweetInfo.isHighMediaResolution)\n            val isQuoteTweet = checkTweetInfoAttribute(candidate.tweetInfo.isQuoteTweet)\n            val isReply = checkTweetInfoAttribute(candidate.tweetInfo.isReply)\n            val hasMultipleMedia = checkTweetInfoAttribute(candidate.tweetInfo.hasMultipleMedia)\n            val hasUrl = checkTweetInfoAttribute(candidate.tweetInfo.hasUrl)\n\n            if (hasVideo && isHighMediaResolution && !isQuoteTweet &&\n              !isReply && !hasMultipleMedia && !hasUrl) {\n              Some(candidate)\n            } else {\n              None\n            }\n          }\n      }\n    })\n  }\n\n  def checkTweetInfoAttribute(attributeOpt: => Option[Boolean]): Boolean = {\n    if (attributeOpt.isDefined)\n      attributeOpt.get\n    else {\n      // takes Quoted Tweet (TweetInfo.isQuoteTweet) as an example,\n      // if the attributeOpt is None, we by default say it is not a quoted tweet\n      // similarly, if TweetInfo.hasVideo is a None,\n      // we say it does not have video.\n      false\n    }\n  }\n\n  override def requestToConfig[CGQueryType <: CandidateGeneratorQuery](\n    query: CGQueryType\n  ): FilterConfig = {\n    val enableVideoTweetFilter = query match {\n      case _: CrCandidateGeneratorQuery | _: RelatedTweetCandidateGeneratorQuery |\n          _: RelatedVideoTweetCandidateGeneratorQuery =>\n        query.params(VideoTweetFilterParams.EnableVideoTweetFilterParam)\n      case _ => false // e.g., GetRelatedTweets()\n    }\n    FilterConfig(\n      enableVideoTweetFilter = enableVideoTweetFilter\n    )\n  }\n}\n\nobject VideoTweetFilter {\n  // extend the filterConfig to add more flags if needed.\n  // now they are hardcoded according to the prod setting\n  case class FilterConfig(\n    enableVideoTweetFilter: Boolean)\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/logging/AdsRecommendationsScribeLogger.scala",
    "content": "package com.twitter.cr_mixer.logging\n\nimport com.twitter.cr_mixer.model.AdsCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialAdsCandidate\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.logging.ScribeLoggerUtils._\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.thriftscala.AdsRecommendationTopLevelApiResult\nimport com.twitter.cr_mixer.thriftscala.AdsRecommendationsResult\nimport com.twitter.cr_mixer.thriftscala.AdsRequest\nimport com.twitter.cr_mixer.thriftscala.AdsResponse\nimport com.twitter.cr_mixer.thriftscala.FetchCandidatesResult\nimport com.twitter.cr_mixer.thriftscala.GetAdsRecommendationsScribe\nimport com.twitter.cr_mixer.thriftscala.PerformanceMetrics\nimport com.twitter.cr_mixer.thriftscala.TweetCandidateWithMetadata\nimport com.twitter.cr_mixer.util.CandidateGenerationKeyUtil\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.logging.Logger\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.util.Future\nimport com.twitter.util.Stopwatch\n\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\ncase class AdsRecommendationsScribeLogger @Inject() (\n  @Named(ModuleNames.AdsRecommendationsLogger) adsRecommendationsScribeLogger: Logger,\n  decider: CrMixerDecider,\n  statsReceiver: StatsReceiver) {\n\n  private val scopedStats = statsReceiver.scope(this.getClass.getCanonicalName)\n  private val upperFunnelsStats = scopedStats.scope(\"UpperFunnels\")\n  private val topLevelApiStats = scopedStats.scope(\"TopLevelApi\")\n\n  /*\n   * Scribe first step results after fetching initial ads candidate\n   * */\n  def scribeInitialAdsCandidates(\n    query: AdsCandidateGeneratorQuery,\n    getResultFn: => Future[Seq[Seq[InitialAdsCandidate]]],\n    enableScribe: Boolean // controlled by feature switch so that we can scribe for certain DDG\n  ): Future[Seq[Seq[InitialAdsCandidate]]] = {\n    val scribeMetadata = ScribeMetadata.from(query)\n    val timer = Stopwatch.start()\n    getResultFn.onSuccess { input =>\n      val latencyMs = timer().inMilliseconds\n      val result = convertFetchCandidatesResult(input, scribeMetadata.userId)\n      val traceId = Trace.id.traceId.toLong\n      val scribeMsg = buildScribeMessage(result, scribeMetadata, latencyMs, traceId)\n\n      if (enableScribe && decider.isAvailableForId(\n          scribeMetadata.userId,\n          DeciderConstants.adsRecommendationsPerExperimentScribeRate)) {\n        upperFunnelsStats.counter(scribeMetadata.product.originalName).incr()\n        scribeResult(scribeMsg)\n      }\n    }\n  }\n\n  /*\n   * Scribe top level API results\n   * */\n  def scribeGetAdsRecommendations(\n    request: AdsRequest,\n    startTime: Long,\n    scribeMetadata: ScribeMetadata,\n    getResultFn: => Future[AdsResponse],\n    enableScribe: Boolean\n  ): Future[AdsResponse] = {\n    val timer = Stopwatch.start()\n    getResultFn.onSuccess { response =>\n      val latencyMs = timer().inMilliseconds\n      val result = AdsRecommendationsResult.AdsRecommendationTopLevelApiResult(\n        AdsRecommendationTopLevelApiResult(\n          timestamp = startTime,\n          request = request,\n          response = response\n        ))\n      val traceId = Trace.id.traceId.toLong\n      val scribeMsg = buildScribeMessage(result, scribeMetadata, latencyMs, traceId)\n\n      if (enableScribe && decider.isAvailableForId(\n          scribeMetadata.userId,\n          DeciderConstants.adsRecommendationsPerExperimentScribeRate)) {\n        topLevelApiStats.counter(scribeMetadata.product.originalName).incr()\n        scribeResult(scribeMsg)\n      }\n    }\n  }\n\n  private def convertFetchCandidatesResult(\n    candidatesSeq: Seq[Seq[InitialAdsCandidate]],\n    requestUserId: UserId\n  ): AdsRecommendationsResult = {\n    val tweetCandidatesWithMetadata = candidatesSeq.flatMap { candidates =>\n      candidates.map { candidate =>\n        TweetCandidateWithMetadata(\n          tweetId = candidate.tweetId,\n          candidateGenerationKey = Some(\n            CandidateGenerationKeyUtil.toThrift(candidate.candidateGenerationInfo, requestUserId)),\n          score = Some(candidate.getSimilarityScore),\n          numCandidateGenerationKeys = None // not populated yet\n        )\n      }\n    }\n    AdsRecommendationsResult.FetchCandidatesResult(\n      FetchCandidatesResult(Some(tweetCandidatesWithMetadata)))\n  }\n\n  private def buildScribeMessage(\n    result: AdsRecommendationsResult,\n    scribeMetadata: ScribeMetadata,\n    latencyMs: Long,\n    traceId: Long\n  ): GetAdsRecommendationsScribe = {\n    GetAdsRecommendationsScribe(\n      uuid = scribeMetadata.requestUUID,\n      userId = scribeMetadata.userId,\n      result = result,\n      traceId = Some(traceId),\n      performanceMetrics = Some(PerformanceMetrics(Some(latencyMs))),\n      impressedBuckets = getImpressedBuckets(scopedStats)\n    )\n  }\n\n  private def scribeResult(\n    scribeMsg: GetAdsRecommendationsScribe\n  ): Unit = {\n    publish(\n      logger = adsRecommendationsScribeLogger,\n      codec = GetAdsRecommendationsScribe,\n      message = scribeMsg)\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/logging/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"abdecider/src/main/scala\",\n        \"content-recommender/thrift/src/main/thrift:content-recommender-common-scala\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/featureswitch\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/scribe\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"decider/src/main/scala\",\n        \"featureswitches/featureswitches-core/src/main/scala:experimentation-settings\",\n        \"finagle/finagle-core/src/main\",\n        \"frigate/frigate-common:base\",\n        \"frigate/frigate-common:util\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/base\",\n        \"kafka/finagle-kafka/finatra-kafka/src/main/scala\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"scribelib/marshallers/src/main/scala/com/twitter/scribelib/marshallers\",\n        \"scribelib/validators/src/main/scala/com/twitter/scribelib/validators\",\n        \"scrooge/scrooge-serializer/src/main/scala\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/ml/api:data-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"timelines/src/main/scala/com/twitter/timelines/clientevent\",\n        \"util-internal/scribe/src/main/scala/com/twitter/logging\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/logging/CrMixerScribeLogger.scala",
    "content": "package com.twitter.cr_mixer.logging\n\nimport com.google.common.base.CaseFormat\nimport com.twitter.abdecider.ScribingABDeciderUtil\nimport com.twitter.scribelib.marshallers.ClientDataProvider\nimport com.twitter.scribelib.marshallers.ScribeSerialization\nimport com.twitter.timelines.clientevent.MinimalClientDataProvider\nimport com.twitter.cr_mixer.model.BlendedCandidate\nimport com.twitter.cr_mixer.model.CrCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.model.RankedCandidate\nimport com.twitter.cr_mixer.logging.ScribeLoggerUtils._\nimport com.twitter.cr_mixer.model.GraphSourceInfo\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.scribe.ScribeCategories\nimport com.twitter.cr_mixer.thriftscala.CrMixerTweetRequest\nimport com.twitter.cr_mixer.thriftscala.CrMixerTweetResponse\nimport com.twitter.cr_mixer.thriftscala.FetchCandidatesResult\nimport com.twitter.cr_mixer.thriftscala.FetchSignalSourcesResult\nimport com.twitter.cr_mixer.thriftscala.GetTweetsRecommendationsScribe\nimport com.twitter.cr_mixer.thriftscala.InterleaveResult\nimport com.twitter.cr_mixer.thriftscala.PerformanceMetrics\nimport com.twitter.cr_mixer.thriftscala.PreRankFilterResult\nimport com.twitter.cr_mixer.thriftscala.Product\nimport com.twitter.cr_mixer.thriftscala.RankResult\nimport com.twitter.cr_mixer.thriftscala.Result\nimport com.twitter.cr_mixer.thriftscala.SourceSignal\nimport com.twitter.cr_mixer.thriftscala.TopLevelApiResult\nimport com.twitter.cr_mixer.thriftscala.TweetCandidateWithMetadata\nimport com.twitter.cr_mixer.thriftscala.VITTweetCandidateScribe\nimport com.twitter.cr_mixer.thriftscala.VITTweetCandidatesScribe\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.util.CandidateGenerationKeyUtil\nimport com.twitter.cr_mixer.util.MetricTagUtil\nimport com.twitter.decider.SimpleRecipient\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.finatra.kafka.producers.KafkaProducerBase\nimport com.twitter.logging.Logger\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.util.Future\nimport com.twitter.util.Stopwatch\nimport com.twitter.util.Time\n\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.util.Random\n\n@Singleton\ncase class CrMixerScribeLogger @Inject() (\n  decider: CrMixerDecider,\n  statsReceiver: StatsReceiver,\n  @Named(ModuleNames.TweetRecsLogger) tweetRecsScribeLogger: Logger,\n  @Named(ModuleNames.BlueVerifiedTweetRecsLogger) blueVerifiedTweetRecsScribeLogger: Logger,\n  @Named(ModuleNames.TopLevelApiDdgMetricsLogger) ddgMetricsLogger: Logger,\n  kafkaProducer: KafkaProducerBase[String, GetTweetsRecommendationsScribe]) {\n\n  import CrMixerScribeLogger._\n\n  private val scopedStats = statsReceiver.scope(\"CrMixerScribeLogger\")\n  private val topLevelApiStats = scopedStats.scope(\"TopLevelApi\")\n  private val upperFunnelsStats = scopedStats.scope(\"UpperFunnels\")\n  private val kafkaMessagesStats = scopedStats.scope(\"KafkaMessages\")\n  private val topLevelApiDdgMetricsStats = scopedStats.scope(\"TopLevelApiDdgMetrics\")\n  private val blueVerifiedTweetCandidatesStats = scopedStats.scope(\"BlueVerifiedTweetCandidates\")\n\n  private val serialization = new ScribeSerialization {}\n\n  def scribeSignalSources(\n    query: CrCandidateGeneratorQuery,\n    getResultFn: => Future[(Set[SourceInfo], Map[String, Option[GraphSourceInfo]])]\n  ): Future[(Set[SourceInfo], Map[String, Option[GraphSourceInfo]])] = {\n    scribeResultsAndPerformanceMetrics(\n      ScribeMetadata.from(query),\n      getResultFn,\n      convertToResultFn = convertFetchSignalSourcesResult\n    )\n  }\n\n  def scribeInitialCandidates(\n    query: CrCandidateGeneratorQuery,\n    getResultFn: => Future[Seq[Seq[InitialCandidate]]]\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    scribeResultsAndPerformanceMetrics(\n      ScribeMetadata.from(query),\n      getResultFn,\n      convertToResultFn = convertFetchCandidatesResult\n    )\n  }\n\n  def scribePreRankFilterCandidates(\n    query: CrCandidateGeneratorQuery,\n    getResultFn: => Future[Seq[Seq[InitialCandidate]]]\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    scribeResultsAndPerformanceMetrics(\n      ScribeMetadata.from(query),\n      getResultFn,\n      convertToResultFn = convertPreRankFilterResult\n    )\n  }\n\n  def scribeInterleaveCandidates(\n    query: CrCandidateGeneratorQuery,\n    getResultFn: => Future[Seq[BlendedCandidate]]\n  ): Future[Seq[BlendedCandidate]] = {\n    scribeResultsAndPerformanceMetrics(\n      ScribeMetadata.from(query),\n      getResultFn,\n      convertToResultFn = convertInterleaveResult,\n      enableKafkaScribe = true\n    )\n  }\n\n  def scribeRankedCandidates(\n    query: CrCandidateGeneratorQuery,\n    getResultFn: => Future[Seq[RankedCandidate]]\n  ): Future[Seq[RankedCandidate]] = {\n    scribeResultsAndPerformanceMetrics(\n      ScribeMetadata.from(query),\n      getResultFn,\n      convertToResultFn = convertRankResult\n    )\n  }\n\n  /**\n   * Scribe Top Level API Request / Response and performance metrics\n   * for the getTweetRecommendations() endpoint.\n   */\n  def scribeGetTweetRecommendations(\n    request: CrMixerTweetRequest,\n    startTime: Long,\n    scribeMetadata: ScribeMetadata,\n    getResultFn: => Future[CrMixerTweetResponse]\n  ): Future[CrMixerTweetResponse] = {\n    val timer = Stopwatch.start()\n    getResultFn.onSuccess { response =>\n      val latencyMs = timer().inMilliseconds\n      val result = convertTopLevelAPIResult(request, response, startTime)\n      val traceId = Trace.id.traceId.toLong\n      val scribeMsg = buildScribeMessage(result, scribeMetadata, latencyMs, traceId)\n\n      // We use upperFunnelPerStepScribeRate to cover TopLevelApi scribe logs\n      if (decider.isAvailableForId(\n          scribeMetadata.userId,\n          DeciderConstants.upperFunnelPerStepScribeRate)) {\n        topLevelApiStats.counter(scribeMetadata.product.originalName).incr()\n        scribeResult(scribeMsg)\n      }\n      if (decider.isAvailableForId(\n          scribeMetadata.userId,\n          DeciderConstants.topLevelApiDdgMetricsScribeRate)) {\n        topLevelApiDdgMetricsStats.counter(scribeMetadata.product.originalName).incr()\n        val topLevelDdgMetricsMetadata = TopLevelDdgMetricsMetadata.from(request)\n        publishTopLevelDdgMetrics(\n          logger = ddgMetricsLogger,\n          topLevelDdgMetricsMetadata = topLevelDdgMetricsMetadata,\n          latencyMs = latencyMs,\n          candidateSize = response.tweets.length)\n      }\n    }\n  }\n\n  /**\n   * Scribe all of the Blue Verified tweets that are candidates from cr-mixer\n   * from the getTweetRecommendations() endpoint for stats tracking/debugging purposes.\n   */\n  def scribeGetTweetRecommendationsForBlueVerified(\n    scribeMetadata: ScribeMetadata,\n    getResultFn: => Future[Seq[RankedCandidate]]\n  ): Future[Seq[RankedCandidate]] = {\n    getResultFn.onSuccess { rankedCandidates =>\n      if (decider.isAvailable(DeciderConstants.enableScribeForBlueVerifiedTweetCandidates)) {\n        blueVerifiedTweetCandidatesStats.counter(\"process_request\").incr()\n\n        val blueVerifiedTweetCandidates = rankedCandidates.filter { tweet =>\n          tweet.tweetInfo.hasBlueVerifiedAnnotation.contains(true)\n        }\n\n        val impressedBuckets = getImpressedBuckets(blueVerifiedTweetCandidatesStats).getOrElse(Nil)\n\n        val blueVerifiedCandidateScribes = blueVerifiedTweetCandidates.map { candidate =>\n          blueVerifiedTweetCandidatesStats\n            .scope(scribeMetadata.product.name).counter(\n              candidate.tweetInfo.authorId.toString).incr()\n          VITTweetCandidateScribe(\n            tweetId = candidate.tweetId,\n            authorId = candidate.tweetInfo.authorId,\n            score = candidate.predictionScore,\n            metricTags = MetricTagUtil.buildMetricTags(candidate)\n          )\n        }\n\n        val blueVerifiedScribe =\n          VITTweetCandidatesScribe(\n            uuid = scribeMetadata.requestUUID,\n            userId = scribeMetadata.userId,\n            candidates = blueVerifiedCandidateScribes,\n            product = scribeMetadata.product,\n            impressedBuckets = impressedBuckets\n          )\n\n        publish(\n          logger = blueVerifiedTweetRecsScribeLogger,\n          codec = VITTweetCandidatesScribe,\n          message = blueVerifiedScribe)\n      }\n    }\n  }\n\n  /**\n   * Scribe Per-step intermediate results and performance metrics\n   * for each step: fetch signals, fetch candidates, filters, ranker, etc\n   */\n  private[logging] def scribeResultsAndPerformanceMetrics[T](\n    scribeMetadata: ScribeMetadata,\n    getResultFn: => Future[T],\n    convertToResultFn: (T, UserId) => Result,\n    enableKafkaScribe: Boolean = false\n  ): Future[T] = {\n    val timer = Stopwatch.start()\n    getResultFn.onSuccess { input =>\n      val latencyMs = timer().inMilliseconds\n      val result = convertToResultFn(input, scribeMetadata.userId)\n      val traceId = Trace.id.traceId.toLong\n      val scribeMsg = buildScribeMessage(result, scribeMetadata, latencyMs, traceId)\n\n      if (decider.isAvailableForId(\n          scribeMetadata.userId,\n          DeciderConstants.upperFunnelPerStepScribeRate)) {\n        upperFunnelsStats.counter(scribeMetadata.product.originalName).incr()\n        scribeResult(scribeMsg)\n      }\n\n      // forks the scribe as a Kafka message for async feature hydration\n      if (enableKafkaScribe && shouldScribeKafkaMessage(\n          scribeMetadata.userId,\n          scribeMetadata.product)) {\n        kafkaMessagesStats.counter(scribeMetadata.product.originalName).incr()\n\n        val batchedKafkaMessages = downsampleKafkaMessage(scribeMsg)\n        batchedKafkaMessages.foreach { kafkaMessage =>\n          kafkaProducer.send(\n            topic = ScribeCategories.TweetsRecs.scribeCategory,\n            key = traceId.toString,\n            value = kafkaMessage,\n            timestamp = Time.now.inMilliseconds\n          )\n        }\n      }\n    }\n  }\n\n  private def convertTopLevelAPIResult(\n    request: CrMixerTweetRequest,\n    response: CrMixerTweetResponse,\n    startTime: Long\n  ): Result = {\n    Result.TopLevelApiResult(\n      TopLevelApiResult(\n        timestamp = startTime,\n        request = request,\n        response = response\n      ))\n  }\n\n  private def convertFetchSignalSourcesResult(\n    sourceInfoSetTuple: (Set[SourceInfo], Map[String, Option[GraphSourceInfo]]),\n    requestUserId: UserId\n  ): Result = {\n    val sourceSignals = sourceInfoSetTuple._1.map { sourceInfo =>\n      SourceSignal(id = Some(sourceInfo.internalId))\n    }\n    // For source graphs, we pass in requestUserId as a placeholder\n    val sourceGraphs = sourceInfoSetTuple._2.map {\n      case (_, _) =>\n        SourceSignal(id = Some(InternalId.UserId(requestUserId)))\n    }\n    Result.FetchSignalSourcesResult(\n      FetchSignalSourcesResult(\n        signals = Some(sourceSignals ++ sourceGraphs)\n      ))\n  }\n\n  private def convertFetchCandidatesResult(\n    candidatesSeq: Seq[Seq[InitialCandidate]],\n    requestUserId: UserId\n  ): Result = {\n    val tweetCandidatesWithMetadata = candidatesSeq.flatMap { candidates =>\n      candidates.map { candidate =>\n        TweetCandidateWithMetadata(\n          tweetId = candidate.tweetId,\n          candidateGenerationKey = Some(\n            CandidateGenerationKeyUtil.toThrift(candidate.candidateGenerationInfo, requestUserId)),\n          score = Some(candidate.getSimilarityScore),\n          numCandidateGenerationKeys = None // not populated yet\n        )\n      }\n    }\n    Result.FetchCandidatesResult(FetchCandidatesResult(Some(tweetCandidatesWithMetadata)))\n  }\n\n  private def convertPreRankFilterResult(\n    candidatesSeq: Seq[Seq[InitialCandidate]],\n    requestUserId: UserId\n  ): Result = {\n    val tweetCandidatesWithMetadata = candidatesSeq.flatMap { candidates =>\n      candidates.map { candidate =>\n        TweetCandidateWithMetadata(\n          tweetId = candidate.tweetId,\n          candidateGenerationKey = Some(\n            CandidateGenerationKeyUtil.toThrift(candidate.candidateGenerationInfo, requestUserId)),\n          score = Some(candidate.getSimilarityScore),\n          numCandidateGenerationKeys = None // not populated yet\n        )\n      }\n    }\n    Result.PreRankFilterResult(PreRankFilterResult(Some(tweetCandidatesWithMetadata)))\n  }\n\n  // We take InterleaveResult for Unconstrained dataset ML ranker training\n  private def convertInterleaveResult(\n    blendedCandidates: Seq[BlendedCandidate],\n    requestUserId: UserId\n  ): Result = {\n    val tweetCandidatesWithMetadata = blendedCandidates.map { blendedCandidate =>\n      val candidateGenerationKey =\n        CandidateGenerationKeyUtil.toThrift(blendedCandidate.reasonChosen, requestUserId)\n      TweetCandidateWithMetadata(\n        tweetId = blendedCandidate.tweetId,\n        candidateGenerationKey = Some(candidateGenerationKey),\n        authorId = Some(blendedCandidate.tweetInfo.authorId), // for ML pipeline training\n        score = Some(blendedCandidate.getSimilarityScore),\n        numCandidateGenerationKeys = Some(blendedCandidate.potentialReasons.size)\n      ) // hydrate fields for light ranking training data\n    }\n    Result.InterleaveResult(InterleaveResult(Some(tweetCandidatesWithMetadata)))\n  }\n\n  private def convertRankResult(\n    rankedCandidates: Seq[RankedCandidate],\n    requestUserId: UserId\n  ): Result = {\n    val tweetCandidatesWithMetadata = rankedCandidates.map { rankedCandidate =>\n      val candidateGenerationKey =\n        CandidateGenerationKeyUtil.toThrift(rankedCandidate.reasonChosen, requestUserId)\n      TweetCandidateWithMetadata(\n        tweetId = rankedCandidate.tweetId,\n        candidateGenerationKey = Some(candidateGenerationKey),\n        score = Some(rankedCandidate.getSimilarityScore),\n        numCandidateGenerationKeys = Some(rankedCandidate.potentialReasons.size)\n      )\n    }\n    Result.RankResult(RankResult(Some(tweetCandidatesWithMetadata)))\n  }\n\n  private def buildScribeMessage(\n    result: Result,\n    scribeMetadata: ScribeMetadata,\n    latencyMs: Long,\n    traceId: Long\n  ): GetTweetsRecommendationsScribe = {\n    GetTweetsRecommendationsScribe(\n      uuid = scribeMetadata.requestUUID,\n      userId = scribeMetadata.userId,\n      result = result,\n      traceId = Some(traceId),\n      performanceMetrics = Some(PerformanceMetrics(Some(latencyMs))),\n      impressedBuckets = getImpressedBuckets(scopedStats)\n    )\n  }\n\n  private def scribeResult(\n    scribeMsg: GetTweetsRecommendationsScribe\n  ): Unit = {\n    publish(\n      logger = tweetRecsScribeLogger,\n      codec = GetTweetsRecommendationsScribe,\n      message = scribeMsg)\n  }\n\n  /**\n   * Gate for producing messages to Kafka for async feature hydration\n   */\n  private def shouldScribeKafkaMessage(\n    userId: UserId,\n    product: Product\n  ): Boolean = {\n    val isEligibleUser = decider.isAvailable(\n      DeciderConstants.kafkaMessageScribeSampleRate,\n      Some(SimpleRecipient(userId)))\n    val isHomeProduct = (product == Product.Home)\n    isEligibleUser && isHomeProduct\n  }\n\n  /**\n   * Due to size limits of Strato (see SD-19028), each Kafka message must be downsampled\n   */\n  private[logging] def downsampleKafkaMessage(\n    scribeMsg: GetTweetsRecommendationsScribe\n  ): Seq[GetTweetsRecommendationsScribe] = {\n    val sampledResultSeq: Seq[Result] = scribeMsg.result match {\n      case Result.InterleaveResult(interleaveResult) =>\n        val sampledTweetsSeq = interleaveResult.tweets\n          .map { tweets =>\n            Random\n              .shuffle(tweets).take(KafkaMaxTweetsPerMessage)\n              .grouped(BatchSize).toSeq\n          }.getOrElse(Seq.empty)\n\n        sampledTweetsSeq.map { sampledTweets =>\n          Result.InterleaveResult(InterleaveResult(Some(sampledTweets)))\n        }\n\n      // if it's an unrecognized type, err on the side of sending no candidates\n      case _ =>\n        kafkaMessagesStats.counter(\"InvalidKafkaMessageResultType\").incr()\n        Seq(Result.InterleaveResult(InterleaveResult(None)))\n    }\n\n    sampledResultSeq.map { sampledResult =>\n      GetTweetsRecommendationsScribe(\n        uuid = scribeMsg.uuid,\n        userId = scribeMsg.userId,\n        result = sampledResult,\n        traceId = scribeMsg.traceId,\n        performanceMetrics = None,\n        impressedBuckets = None\n      )\n    }\n  }\n\n  /**\n   * Handles client_event serialization to log data into DDG metrics\n   */\n  private[logging] def publishTopLevelDdgMetrics(\n    logger: Logger,\n    topLevelDdgMetricsMetadata: TopLevelDdgMetricsMetadata,\n    candidateSize: Long,\n    latencyMs: Long,\n  ): Unit = {\n    val data = Map[Any, Any](\n      \"latency_ms\" -> latencyMs,\n      \"event_value\" -> candidateSize\n    )\n    val label: (String, String) = (\"tweetrec\", \"\")\n    val namespace = getNamespace(topLevelDdgMetricsMetadata, label) + (\"action\" -> \"candidates\")\n    val message =\n      serialization\n        .serializeClientEvent(namespace, getClientData(topLevelDdgMetricsMetadata), data)\n    logger.info(message)\n  }\n\n  private def getClientData(\n    topLevelDdgMetricsMetadata: TopLevelDdgMetricsMetadata\n  ): ClientDataProvider =\n    MinimalClientDataProvider(\n      userId = topLevelDdgMetricsMetadata.userId,\n      guestId = None,\n      clientApplicationId = topLevelDdgMetricsMetadata.clientApplicationId,\n      countryCode = topLevelDdgMetricsMetadata.countryCode\n    )\n\n  private def getNamespace(\n    topLevelDdgMetricsMetadata: TopLevelDdgMetricsMetadata,\n    label: (String, String)\n  ): Map[String, String] = {\n    val productName =\n      CaseFormat.UPPER_CAMEL\n        .to(CaseFormat.LOWER_UNDERSCORE, topLevelDdgMetricsMetadata.product.originalName)\n\n    Map(\n      \"client\" -> ScribingABDeciderUtil.clientForAppId(\n        topLevelDdgMetricsMetadata.clientApplicationId),\n      \"page\" -> \"cr-mixer\",\n      \"section\" -> productName,\n      \"component\" -> label._1,\n      \"element\" -> label._2\n    )\n  }\n}\n\nobject CrMixerScribeLogger {\n  val KafkaMaxTweetsPerMessage: Int = 200\n  val BatchSize: Int = 20\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/logging/RelatedTweetScribeLogger.scala",
    "content": "package com.twitter.cr_mixer.logging\n\nimport com.twitter.cr_mixer.model.RelatedTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.logging.ScribeLoggerUtils._\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.thriftscala.FetchCandidatesResult\nimport com.twitter.cr_mixer.thriftscala.GetRelatedTweetsScribe\nimport com.twitter.cr_mixer.thriftscala.PerformanceMetrics\nimport com.twitter.cr_mixer.thriftscala.PreRankFilterResult\nimport com.twitter.cr_mixer.thriftscala.RelatedTweetRequest\nimport com.twitter.cr_mixer.thriftscala.RelatedTweetResponse\nimport com.twitter.cr_mixer.thriftscala.RelatedTweetResult\nimport com.twitter.cr_mixer.thriftscala.RelatedTweetTopLevelApiResult\nimport com.twitter.cr_mixer.thriftscala.TweetCandidateWithMetadata\nimport com.twitter.cr_mixer.util.CandidateGenerationKeyUtil\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.logging.Logger\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.util.Future\nimport com.twitter.util.Stopwatch\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\ncase class RelatedTweetScribeLogger @Inject() (\n  decider: CrMixerDecider,\n  statsReceiver: StatsReceiver,\n  @Named(ModuleNames.RelatedTweetsLogger) relatedTweetsScribeLogger: Logger) {\n\n  private val scopedStats = statsReceiver.scope(\"RelatedTweetsScribeLogger\")\n  private val topLevelApiStats = scopedStats.scope(\"TopLevelApi\")\n  private val topLevelApiNoUserIdStats = scopedStats.scope(\"TopLevelApiNoUserId\")\n  private val upperFunnelsStats = scopedStats.scope(\"UpperFunnels\")\n  private val upperFunnelsNoUserIdStats = scopedStats.scope(\"UpperFunnelsNoUserId\")\n\n  def scribeInitialCandidates(\n    query: RelatedTweetCandidateGeneratorQuery,\n    getResultFn: => Future[Seq[Seq[InitialCandidate]]]\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    scribeResultsAndPerformanceMetrics(\n      RelatedTweetScribeMetadata.from(query),\n      getResultFn,\n      convertToResultFn = convertFetchCandidatesResult\n    )\n  }\n\n  def scribePreRankFilterCandidates(\n    query: RelatedTweetCandidateGeneratorQuery,\n    getResultFn: => Future[Seq[Seq[InitialCandidate]]]\n  ): Future[Seq[Seq[InitialCandidate]]] = {\n    scribeResultsAndPerformanceMetrics(\n      RelatedTweetScribeMetadata.from(query),\n      getResultFn,\n      convertToResultFn = convertPreRankFilterResult\n    )\n  }\n\n  /**\n   * Scribe Top Level API Request / Response and performance metrics\n   * for the getRelatedTweets endpoint.\n   */\n  def scribeGetRelatedTweets(\n    request: RelatedTweetRequest,\n    startTime: Long,\n    relatedTweetScribeMetadata: RelatedTweetScribeMetadata,\n    getResultFn: => Future[RelatedTweetResponse]\n  ): Future[RelatedTweetResponse] = {\n    val timer = Stopwatch.start()\n    getResultFn.onSuccess { response =>\n      relatedTweetScribeMetadata.clientContext.userId match {\n        case Some(userId) =>\n          if (decider.isAvailableForId(userId, DeciderConstants.upperFunnelPerStepScribeRate)) {\n            topLevelApiStats.counter(relatedTweetScribeMetadata.product.originalName).incr()\n            val latencyMs = timer().inMilliseconds\n            val result = convertTopLevelAPIResult(request, response, startTime)\n            val traceId = Trace.id.traceId.toLong\n            val scribeMsg =\n              buildScribeMessage(result, relatedTweetScribeMetadata, latencyMs, traceId)\n\n            scribeResult(scribeMsg)\n          }\n        case _ =>\n          topLevelApiNoUserIdStats.counter(relatedTweetScribeMetadata.product.originalName).incr()\n      }\n    }\n  }\n\n  /**\n   * Scribe Per-step intermediate results and performance metrics\n   * for each step: fetch candidates, filters.\n   */\n  private def scribeResultsAndPerformanceMetrics[T](\n    relatedTweetScribeMetadata: RelatedTweetScribeMetadata,\n    getResultFn: => Future[T],\n    convertToResultFn: (T, UserId) => RelatedTweetResult\n  ): Future[T] = {\n    val timer = Stopwatch.start()\n    getResultFn.onSuccess { input =>\n      relatedTweetScribeMetadata.clientContext.userId match {\n        case Some(userId) =>\n          if (decider.isAvailableForId(userId, DeciderConstants.upperFunnelPerStepScribeRate)) {\n            upperFunnelsStats.counter(relatedTweetScribeMetadata.product.originalName).incr()\n            val latencyMs = timer().inMilliseconds\n            val result = convertToResultFn(input, userId)\n            val traceId = Trace.id.traceId.toLong\n            val scribeMsg =\n              buildScribeMessage(result, relatedTweetScribeMetadata, latencyMs, traceId)\n            scribeResult(scribeMsg)\n          }\n        case _ =>\n          upperFunnelsNoUserIdStats.counter(relatedTweetScribeMetadata.product.originalName).incr()\n      }\n    }\n  }\n\n  private def convertTopLevelAPIResult(\n    request: RelatedTweetRequest,\n    response: RelatedTweetResponse,\n    startTime: Long\n  ): RelatedTweetResult = {\n    RelatedTweetResult.RelatedTweetTopLevelApiResult(\n      RelatedTweetTopLevelApiResult(\n        timestamp = startTime,\n        request = request,\n        response = response\n      ))\n  }\n\n  private def convertFetchCandidatesResult(\n    candidatesSeq: Seq[Seq[InitialCandidate]],\n    requestUserId: UserId\n  ): RelatedTweetResult = {\n    val tweetCandidatesWithMetadata = candidatesSeq.flatMap { candidates =>\n      candidates.map { candidate =>\n        TweetCandidateWithMetadata(\n          tweetId = candidate.tweetId,\n          candidateGenerationKey = None\n        ) // do not hydrate candidateGenerationKey to save cost\n      }\n    }\n    RelatedTweetResult.FetchCandidatesResult(\n      FetchCandidatesResult(Some(tweetCandidatesWithMetadata)))\n  }\n\n  private def convertPreRankFilterResult(\n    candidatesSeq: Seq[Seq[InitialCandidate]],\n    requestUserId: UserId\n  ): RelatedTweetResult = {\n    val tweetCandidatesWithMetadata = candidatesSeq.flatMap { candidates =>\n      candidates.map { candidate =>\n        val candidateGenerationKey =\n          CandidateGenerationKeyUtil.toThrift(candidate.candidateGenerationInfo, requestUserId)\n        TweetCandidateWithMetadata(\n          tweetId = candidate.tweetId,\n          candidateGenerationKey = Some(candidateGenerationKey),\n          authorId = Some(candidate.tweetInfo.authorId),\n          score = Some(candidate.getSimilarityScore),\n          numCandidateGenerationKeys = None\n        )\n      }\n    }\n    RelatedTweetResult.PreRankFilterResult(PreRankFilterResult(Some(tweetCandidatesWithMetadata)))\n  }\n\n  private def buildScribeMessage(\n    relatedTweetResult: RelatedTweetResult,\n    relatedTweetScribeMetadata: RelatedTweetScribeMetadata,\n    latencyMs: Long,\n    traceId: Long\n  ): GetRelatedTweetsScribe = {\n    GetRelatedTweetsScribe(\n      uuid = relatedTweetScribeMetadata.requestUUID,\n      internalId = relatedTweetScribeMetadata.internalId,\n      relatedTweetResult = relatedTweetResult,\n      requesterId = relatedTweetScribeMetadata.clientContext.userId,\n      guestId = relatedTweetScribeMetadata.clientContext.guestId,\n      traceId = Some(traceId),\n      performanceMetrics = Some(PerformanceMetrics(Some(latencyMs))),\n      impressedBuckets = getImpressedBuckets(scopedStats)\n    )\n  }\n\n  private def scribeResult(\n    scribeMsg: GetRelatedTweetsScribe\n  ): Unit = {\n    publish(logger = relatedTweetsScribeLogger, codec = GetRelatedTweetsScribe, message = scribeMsg)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/logging/ScribeLoggerUtils.scala",
    "content": "package com.twitter.cr_mixer.logging\n\nimport com.twitter.cr_mixer.featureswitch.CrMixerImpressedBuckets\nimport com.twitter.cr_mixer.thriftscala.ImpressesedBucketInfo\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.logging.Logger\nimport com.twitter.scrooge.BinaryThriftStructSerializer\nimport com.twitter.scrooge.ThriftStruct\nimport com.twitter.scrooge.ThriftStructCodec\n\nobject ScribeLoggerUtils {\n\n  /**\n   * Handles base64-encoding, serialization, and publish.\n   */\n  private[logging] def publish[T <: ThriftStruct](\n    logger: Logger,\n    codec: ThriftStructCodec[T],\n    message: T\n  ): Unit = {\n    logger.info(BinaryThriftStructSerializer(codec).toString(message))\n  }\n\n  private[logging] def getImpressedBuckets(\n    scopedStats: StatsReceiver\n  ): Option[List[ImpressesedBucketInfo]] = {\n    StatsUtil.trackNonFutureBlockStats(scopedStats.scope(\"getImpressedBuckets\")) {\n      CrMixerImpressedBuckets.getAllImpressedBuckets.map { listBuckets =>\n        val listBucketsSet = listBuckets.toSet\n        scopedStats.stat(\"impressed_buckets\").add(listBucketsSet.size)\n        listBucketsSet.map { bucket =>\n          ImpressesedBucketInfo(\n            experimentId = bucket.experiment.settings.experimentId.getOrElse(-1L),\n            bucketName = bucket.name,\n            version = bucket.experiment.settings.version,\n          )\n        }.toList\n      }\n    }\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/logging/ScribeMetadata.scala",
    "content": "package com.twitter.cr_mixer.logging\n\nimport com.twitter.cr_mixer.model.AdsCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.CrCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.RelatedTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.UtegTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.thriftscala.Product\nimport com.twitter.product_mixer.core.thriftscala.ClientContext\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\n\ncase class ScribeMetadata(\n  requestUUID: Long,\n  userId: UserId,\n  product: Product)\n\nobject ScribeMetadata {\n  def from(query: CrCandidateGeneratorQuery): ScribeMetadata = {\n    ScribeMetadata(query.requestUUID, query.userId, query.product)\n  }\n\n  def from(query: UtegTweetCandidateGeneratorQuery): ScribeMetadata = {\n    ScribeMetadata(query.requestUUID, query.userId, query.product)\n  }\n\n  def from(query: AdsCandidateGeneratorQuery): ScribeMetadata = {\n    ScribeMetadata(query.requestUUID, query.userId, query.product)\n  }\n}\n\ncase class RelatedTweetScribeMetadata(\n  requestUUID: Long,\n  internalId: InternalId,\n  clientContext: ClientContext,\n  product: Product)\n\nobject RelatedTweetScribeMetadata {\n  def from(query: RelatedTweetCandidateGeneratorQuery): RelatedTweetScribeMetadata = {\n    RelatedTweetScribeMetadata(\n      query.requestUUID,\n      query.internalId,\n      query.clientContext,\n      query.product)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/logging/TopLevelDdgMetricsMetadata.scala",
    "content": "package com.twitter.cr_mixer\npackage logging\n\nimport com.twitter.cr_mixer.thriftscala.CrMixerTweetRequest\nimport com.twitter.cr_mixer.thriftscala.Product\n\ncase class TopLevelDdgMetricsMetadata(\n  userId: Option[Long],\n  product: Product,\n  clientApplicationId: Option[Long],\n  countryCode: Option[String])\n\nobject TopLevelDdgMetricsMetadata {\n  def from(request: CrMixerTweetRequest): TopLevelDdgMetricsMetadata = {\n    TopLevelDdgMetricsMetadata(\n      userId = request.clientContext.userId,\n      product = request.product,\n      clientApplicationId = request.clientContext.appId,\n      countryCode = request.clientContext.countryCode\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/logging/UtegTweetScribeLogger.scala",
    "content": "package com.twitter.cr_mixer.logging\n\nimport com.twitter.cr_mixer.logging.ScribeLoggerUtils._\nimport com.twitter.cr_mixer.model.UtegTweetCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithScoreAndSocialProof\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.thriftscala.UtegTweetRequest\nimport com.twitter.cr_mixer.thriftscala.UtegTweetResponse\nimport com.twitter.cr_mixer.thriftscala.FetchCandidatesResult\nimport com.twitter.cr_mixer.thriftscala.GetUtegTweetsScribe\nimport com.twitter.cr_mixer.thriftscala.PerformanceMetrics\nimport com.twitter.cr_mixer.thriftscala.UtegTweetResult\nimport com.twitter.cr_mixer.thriftscala.UtegTweetTopLevelApiResult\nimport com.twitter.cr_mixer.thriftscala.TweetCandidateWithMetadata\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.logging.Logger\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.util.Future\nimport com.twitter.util.Stopwatch\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\ncase class UtegTweetScribeLogger @Inject() (\n  decider: CrMixerDecider,\n  statsReceiver: StatsReceiver,\n  @Named(ModuleNames.UtegTweetsLogger) utegTweetScribeLogger: Logger) {\n\n  private val scopedStats = statsReceiver.scope(\"UtegTweetScribeLogger\")\n  private val topLevelApiStats = scopedStats.scope(\"TopLevelApi\")\n  private val upperFunnelsStats = scopedStats.scope(\"UpperFunnels\")\n\n  def scribeInitialCandidates(\n    query: UtegTweetCandidateGeneratorQuery,\n    getResultFn: => Future[Seq[TweetWithScoreAndSocialProof]]\n  ): Future[Seq[TweetWithScoreAndSocialProof]] = {\n    scribeResultsAndPerformanceMetrics(\n      ScribeMetadata.from(query),\n      getResultFn,\n      convertToResultFn = convertFetchCandidatesResult\n    )\n  }\n\n  /**\n   * Scribe Top Level API Request / Response and performance metrics\n   * for the GetUtegTweetRecommendations() endpoint.\n   */\n  def scribeGetUtegTweetRecommendations(\n    request: UtegTweetRequest,\n    startTime: Long,\n    scribeMetadata: ScribeMetadata,\n    getResultFn: => Future[UtegTweetResponse]\n  ): Future[UtegTweetResponse] = {\n    val timer = Stopwatch.start()\n    getResultFn.onSuccess { response =>\n      if (decider.isAvailableForId(\n          scribeMetadata.userId,\n          DeciderConstants.upperFunnelPerStepScribeRate)) {\n        topLevelApiStats.counter(scribeMetadata.product.originalName).incr()\n        val latencyMs = timer().inMilliseconds\n        val result = convertTopLevelAPIResult(request, response, startTime)\n        val traceId = Trace.id.traceId.toLong\n        val scribeMsg =\n          buildScribeMessage(result, scribeMetadata, latencyMs, traceId)\n\n        scribeResult(scribeMsg)\n      }\n    }\n  }\n\n  private def convertTopLevelAPIResult(\n    request: UtegTweetRequest,\n    response: UtegTweetResponse,\n    startTime: Long\n  ): UtegTweetResult = {\n    UtegTweetResult.UtegTweetTopLevelApiResult(\n      UtegTweetTopLevelApiResult(\n        timestamp = startTime,\n        request = request,\n        response = response\n      ))\n  }\n\n  private def buildScribeMessage(\n    utegTweetResult: UtegTweetResult,\n    scribeMetadata: ScribeMetadata,\n    latencyMs: Long,\n    traceId: Long\n  ): GetUtegTweetsScribe = {\n    GetUtegTweetsScribe(\n      uuid = scribeMetadata.requestUUID,\n      userId = scribeMetadata.userId,\n      utegTweetResult = utegTweetResult,\n      traceId = Some(traceId),\n      performanceMetrics = Some(PerformanceMetrics(Some(latencyMs))),\n      impressedBuckets = getImpressedBuckets(scopedStats)\n    )\n  }\n\n  private def scribeResult(\n    scribeMsg: GetUtegTweetsScribe\n  ): Unit = {\n    publish(logger = utegTweetScribeLogger, codec = GetUtegTweetsScribe, message = scribeMsg)\n  }\n\n  private def convertFetchCandidatesResult(\n    candidates: Seq[TweetWithScoreAndSocialProof],\n    requestUserId: UserId\n  ): UtegTweetResult = {\n    val tweetCandidatesWithMetadata = candidates.map { candidate =>\n      TweetCandidateWithMetadata(\n        tweetId = candidate.tweetId,\n        candidateGenerationKey = None\n      ) // do not hydrate candidateGenerationKey to save cost\n    }\n    UtegTweetResult.FetchCandidatesResult(FetchCandidatesResult(Some(tweetCandidatesWithMetadata)))\n  }\n\n  /**\n   * Scribe Per-step intermediate results and performance metrics\n   * for each step: fetch candidates, filters.\n   */\n  private def scribeResultsAndPerformanceMetrics[T](\n    scribeMetadata: ScribeMetadata,\n    getResultFn: => Future[T],\n    convertToResultFn: (T, UserId) => UtegTweetResult\n  ): Future[T] = {\n    val timer = Stopwatch.start()\n    getResultFn.onSuccess { input =>\n      if (decider.isAvailableForId(\n          scribeMetadata.userId,\n          DeciderConstants.upperFunnelPerStepScribeRate)) {\n        upperFunnelsStats.counter(scribeMetadata.product.originalName).incr()\n        val latencyMs = timer().inMilliseconds\n        val result = convertToResultFn(input, scribeMetadata.userId)\n        val traceId = Trace.id.traceId.toLong\n        val scribeMsg =\n          buildScribeMessage(result, scribeMetadata, latencyMs, traceId)\n        scribeResult(scribeMsg)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core\",\n        \"content-recommender/thrift/src/main/thrift:thrift-scala\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/core_workflows/user_model:user_model-scala\",\n        \"src/thrift/com/twitter/recos:recos-common-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model/Candidate.scala",
    "content": "package com.twitter.cr_mixer.model\n\nimport com.twitter.contentrecommender.thriftscala.TweetInfo\nimport com.twitter.cr_mixer.thriftscala.LineItemInfo\nimport com.twitter.simclusters_v2.common.TweetId\n\nsealed trait Candidate {\n  val tweetId: TweetId\n\n  override def hashCode: Int = tweetId.toInt\n}\n\ncase class TweetWithCandidateGenerationInfo(\n  tweetId: TweetId,\n  candidateGenerationInfo: CandidateGenerationInfo)\n    extends Candidate {\n\n  def getSimilarityScore: Double =\n    candidateGenerationInfo.similarityEngineInfo.score.getOrElse(0.0)\n}\n\ncase class InitialCandidate(\n  tweetId: TweetId,\n  tweetInfo: TweetInfo,\n  candidateGenerationInfo: CandidateGenerationInfo)\n    extends Candidate {\n\n  /** *\n   * Get the Similarity Score of a Tweet from its CG Info. For instance,\n   * If it is from a UnifiedTweetBasedSimilarityEngine, the score will be the weighted combined score\n   * And if it is from a SimClustersANNSimilarityEngine, the score will be the SANN score\n   */\n  def getSimilarityScore: Double =\n    candidateGenerationInfo.similarityEngineInfo.score.getOrElse(0.0)\n\n  /**\n   * The same candidate can be generated by multiple algorithms.\n   * During blending, candidate deduping happens. In order to retain the candidateGenerationInfo\n   * from different algorithms, we attach them to a list of potentialReasons.\n   */\n  def toBlendedCandidate(\n    potentialReasons: Seq[CandidateGenerationInfo],\n  ): BlendedCandidate = {\n    BlendedCandidate(\n      tweetId,\n      tweetInfo,\n      candidateGenerationInfo,\n      potentialReasons,\n    )\n  }\n\n  // for experimental purposes only when bypassing interleave / ranking\n  def toRankedCandidate(): RankedCandidate = {\n    RankedCandidate(\n      tweetId,\n      tweetInfo,\n      0.0, // prediction score is default to 0.0 to help differentiate that it is a no-op\n      candidateGenerationInfo,\n      Seq(candidateGenerationInfo)\n    )\n  }\n}\n\ncase class InitialAdsCandidate(\n  tweetId: TweetId,\n  lineItemInfo: Seq[LineItemInfo],\n  candidateGenerationInfo: CandidateGenerationInfo)\n    extends Candidate {\n\n  /** *\n   * Get the Similarity Score of a Tweet from its CG Info. For instance,\n   * If it is from a UnifiedTweetBasedSimilarityEngine, the score will be the weighted combined score\n   * And if it is from a SimClustersANNSimilarityEngine, the score will be the SANN score\n   */\n  def getSimilarityScore: Double =\n    candidateGenerationInfo.similarityEngineInfo.score.getOrElse(0.0)\n\n  /**\n   * The same candidate can be generated by multiple algorithms.\n   * During blending, candidate deduping happens. In order to retain the candidateGenerationInfo\n   * from different algorithms, we attach them to a list of potentialReasons.\n   */\n  def toBlendedAdsCandidate(\n    potentialReasons: Seq[CandidateGenerationInfo],\n  ): BlendedAdsCandidate = {\n    BlendedAdsCandidate(\n      tweetId,\n      lineItemInfo,\n      candidateGenerationInfo,\n      potentialReasons,\n    )\n  }\n\n  // for experimental purposes only when bypassing interleave / ranking\n  def toRankedAdsCandidate(): RankedAdsCandidate = {\n    RankedAdsCandidate(\n      tweetId,\n      lineItemInfo,\n      0.0, // prediction score is default to 0.0 to help differentiate that it is a no-op\n      candidateGenerationInfo,\n      Seq(candidateGenerationInfo)\n    )\n  }\n}\n\ncase class BlendedCandidate(\n  tweetId: TweetId,\n  tweetInfo: TweetInfo,\n  reasonChosen: CandidateGenerationInfo,\n  potentialReasons: Seq[CandidateGenerationInfo])\n    extends Candidate {\n\n  /** *\n   * Get the Similarity Score of a Tweet from its CG Info. For instance,\n   * If it is from a UnifiedTweetBasedSimilarityEngine, the score will be the weighted combined score\n   * And if it is from a SimClustersANNSimilarityEngine, the score will be the SANN score\n   */\n  def getSimilarityScore: Double =\n    reasonChosen.similarityEngineInfo.score.getOrElse(0.0)\n\n  assert(potentialReasons.contains(reasonChosen))\n\n  def toRankedCandidate(predictionScore: Double): RankedCandidate = {\n    RankedCandidate(\n      tweetId,\n      tweetInfo,\n      predictionScore,\n      reasonChosen,\n      potentialReasons\n    )\n  }\n}\n\ncase class BlendedAdsCandidate(\n  tweetId: TweetId,\n  lineItemInfo: Seq[LineItemInfo],\n  reasonChosen: CandidateGenerationInfo,\n  potentialReasons: Seq[CandidateGenerationInfo])\n    extends Candidate {\n\n  /** *\n   * Get the Similarity Score of a Tweet from its CG Info. For instance,\n   * If it is from a UnifiedTweetBasedSimilarityEngine, the score will be the weighted combined score\n   * And if it is from a SimClustersANNSimilarityEngine, the score will be the SANN score\n   */\n  def getSimilarityScore: Double =\n    reasonChosen.similarityEngineInfo.score.getOrElse(0.0)\n\n  assert(potentialReasons.contains(reasonChosen))\n\n  def toRankedAdsCandidate(predictionScore: Double): RankedAdsCandidate = {\n    RankedAdsCandidate(\n      tweetId,\n      lineItemInfo,\n      predictionScore,\n      reasonChosen,\n      potentialReasons\n    )\n  }\n}\n\ncase class RankedCandidate(\n  tweetId: TweetId,\n  tweetInfo: TweetInfo,\n  predictionScore: Double,\n  reasonChosen: CandidateGenerationInfo,\n  potentialReasons: Seq[CandidateGenerationInfo])\n    extends Candidate {\n\n  /** *\n   * Get the Similarity Score of a Tweet from its CG Info. For instance,\n   * If it is from a UnifiedTweetBasedSimilarityEngine, the score will be the weighted combined score\n   * And if it is from a SimClustersANNSimilarityEngine, the score will be the SANN score\n   */\n  def getSimilarityScore: Double =\n    reasonChosen.similarityEngineInfo.score.getOrElse(0.0)\n\n  assert(potentialReasons.contains(reasonChosen))\n}\n\ncase class RankedAdsCandidate(\n  tweetId: TweetId,\n  lineItemInfo: Seq[LineItemInfo],\n  predictionScore: Double,\n  reasonChosen: CandidateGenerationInfo,\n  potentialReasons: Seq[CandidateGenerationInfo])\n    extends Candidate {\n\n  /** *\n   * Get the Similarity Score of a Tweet from its CG Info. For instance,\n   * If it is from a UnifiedTweetBasedSimilarityEngine, the score will be the weighted combined score\n   * And if it is from a SimClustersANNSimilarityEngine, the score will be the SANN score\n   */\n  def getSimilarityScore: Double =\n    reasonChosen.similarityEngineInfo.score.getOrElse(0.0)\n\n  assert(potentialReasons.contains(reasonChosen))\n}\n\ncase class TripTweetWithScore(tweetId: TweetId, score: Double) extends Candidate\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model/CandidateGenerationInfo.scala",
    "content": "package com.twitter.cr_mixer.model\n\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.util.Time\n\n/***\n * Tweet-level attributes. Represents the source used in candidate generation\n * Due to legacy reason, SourceType used to represent both SourceType and SimilarityEngineType\n * Moving forward, SourceType will be used for SourceType ONLY. eg., TweetFavorite, UserFollow, TwiceUserId\n * At the same time, We create a new SimilarityEngineType to separate them. eg., SimClustersANN\n *\n * Currently, one special case is that we have TwiceUserId as a source, which is not necessarily a \"signal\"\n * @param sourceType, e.g., SourceType.TweetFavorite, SourceType.UserFollow, SourceType.TwiceUserId\n * @param internalId, e.g., UserId(0L), TweetId(0L)\n */\ncase class SourceInfo(\n  sourceType: SourceType,\n  internalId: InternalId,\n  sourceEventTime: Option[Time])\n\n/***\n * Tweet-level attributes. Represents the source User Graph used in candidate generation\n * It is an intermediate product, and will not be stored, unlike SourceInfo.\n * Essentially, CrMixer queries a graph, and the graph returns a list of users to be used as sources.\n * For instance, RealGraph, EarlyBird, FRS, Stp, etc. The underlying similarity engines such as\n * UTG or UTEG will leverage these sources to build candidates.\n *\n * We extended the definition of SourceType to cover both \"Source Signal\" and \"Source Graph\"\n * See [CrMixer] Graph Based Source Fetcher Abstraction Proposal:\n *\n * consider making both SourceInfo and GraphSourceInfo extends the same trait to\n * have a unified interface.\n */\ncase class GraphSourceInfo(\n  sourceType: SourceType,\n  seedWithScores: Map[UserId, Double])\n\n/***\n * Tweet-level attributes. Represents the similarity engine (the algorithm) used for\n * candidate generation along with their metadata.\n * @param similarityEngineType, e.g., SimClustersANN, UserTweetGraph\n * @param modelId. e.g., UserTweetGraphConsumerEmbedding_ALL_20210708\n * @param score - a score generated by this sim engine\n */\ncase class SimilarityEngineInfo(\n  similarityEngineType: SimilarityEngineType,\n  modelId: Option[String], // ModelId can be a None. e.g., UTEG, UnifiedTweetBasedSE. etc\n  score: Option[Double])\n\n/****\n * Tweet-level attributes. A combination for both SourceInfo and SimilarityEngineInfo\n * SimilarityEngine is a composition, and it can be composed by many leaf Similarity Engines.\n * For instance, the TweetBasedUnified SE could be a composition of both UserTweetGraph SE, SimClustersANN SE.\n * Note that a SimilarityEngine (Composite) may call other SimilarityEngines (Atomic, Contributing)\n * to contribute to its final candidate list. We track these Contributing SEs in the contributingSimilarityEngines list\n *\n * @param sourceInfoOpt - this is optional as many consumerBased CG does not have a source\n * @param similarityEngineInfo - the similarity engine used in Candidate Generation (eg., TweetBasedUnifiedSE). It can be an atomic SE or an composite SE\n * @param contributingSimilarityEngines - only composite SE will have it (e.g., SANNN, UTG). Otherwise it is an empty Seq. All contributing SEs mst be atomic\n */\ncase class CandidateGenerationInfo(\n  sourceInfoOpt: Option[SourceInfo],\n  similarityEngineInfo: SimilarityEngineInfo,\n  contributingSimilarityEngines: Seq[SimilarityEngineInfo])\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model/CandidateGeneratorQuery.scala",
    "content": "package com.twitter.cr_mixer.model\n\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.cr_mixer.thriftscala.Product\nimport com.twitter.product_mixer.core.thriftscala.ClientContext\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.TopicId\nimport com.twitter.timelines.configapi.Params\n\nsealed trait CandidateGeneratorQuery {\n  val product: Product\n  val maxNumResults: Int\n  val impressedTweetList: Set[TweetId]\n  val params: Params\n  val requestUUID: Long\n}\n\nsealed trait HasUserId {\n  val userId: UserId\n}\n\ncase class CrCandidateGeneratorQuery(\n  userId: UserId,\n  product: Product,\n  userState: UserState,\n  maxNumResults: Int,\n  impressedTweetList: Set[TweetId],\n  params: Params,\n  requestUUID: Long,\n  languageCode: Option[String] = None)\n    extends CandidateGeneratorQuery\n    with HasUserId\n\ncase class UtegTweetCandidateGeneratorQuery(\n  userId: UserId,\n  product: Product,\n  userState: UserState,\n  maxNumResults: Int,\n  impressedTweetList: Set[TweetId],\n  params: Params,\n  requestUUID: Long)\n    extends CandidateGeneratorQuery\n    with HasUserId\n\ncase class RelatedTweetCandidateGeneratorQuery(\n  internalId: InternalId,\n  clientContext: ClientContext, // To scribe LogIn/LogOut requests\n  product: Product,\n  maxNumResults: Int,\n  impressedTweetList: Set[TweetId],\n  params: Params,\n  requestUUID: Long)\n    extends CandidateGeneratorQuery\n\ncase class RelatedVideoTweetCandidateGeneratorQuery(\n  internalId: InternalId,\n  clientContext: ClientContext, // To scribe LogIn/LogOut requests\n  product: Product,\n  maxNumResults: Int,\n  impressedTweetList: Set[TweetId],\n  params: Params,\n  requestUUID: Long)\n    extends CandidateGeneratorQuery\n\ncase class FrsTweetCandidateGeneratorQuery(\n  userId: UserId,\n  product: Product,\n  maxNumResults: Int,\n  impressedUserList: Set[UserId],\n  impressedTweetList: Set[TweetId],\n  params: Params,\n  languageCodeOpt: Option[String] = None,\n  countryCodeOpt: Option[String] = None,\n  requestUUID: Long)\n    extends CandidateGeneratorQuery\n\ncase class AdsCandidateGeneratorQuery(\n  userId: UserId,\n  product: Product,\n  userState: UserState,\n  maxNumResults: Int,\n  params: Params,\n  requestUUID: Long)\n\ncase class TopicTweetCandidateGeneratorQuery(\n  userId: UserId,\n  topicIds: Set[TopicId],\n  product: Product,\n  maxNumResults: Int,\n  impressedTweetList: Set[TweetId],\n  params: Params,\n  requestUUID: Long,\n  isVideoOnly: Boolean)\n    extends CandidateGeneratorQuery\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model/EarlybirdSimilarityEngineType.scala",
    "content": "package com.twitter.cr_mixer.model\n\nsealed trait EarlybirdSimilarityEngineType\nobject EarlybirdSimilarityEngineType_RecencyBased extends EarlybirdSimilarityEngineType\nobject EarlybirdSimilarityEngineType_ModelBased extends EarlybirdSimilarityEngineType\nobject EarlybirdSimilarityEngineType_TensorflowBased extends EarlybirdSimilarityEngineType\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model/HealthThreshold.scala",
    "content": "package com.twitter.cr_mixer.model\n\nobject HealthThreshold {\n  object Enum extends Enumeration {\n    val Off: Value = Value(1)\n    val Moderate: Value = Value(2)\n    val Strict: Value = Value(3)\n    val Stricter: Value = Value(4)\n    val StricterPlus: Value = Value(5)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model/ModelConfig.scala",
    "content": "package com.twitter.cr_mixer.model\n\n/**\n * A Configuration class for all Model Based Candidate Sources.\n *\n * The Model Name Guideline. Please your modelId as \"Algorithm_Product_Date\"\n * If your model is used for multiple product surfaces, name it as all\n * Don't name your algorithm as MBCG. All the algorithms here are MBCG =.=\n *\n * Don't forgot to add your new models into allHnswANNSimilarityEngineModelIds list.\n */\nobject ModelConfig {\n  // Offline SimClusters CG Experiment related Model Ids\n  val OfflineInterestedInFromKnownFor2020: String = \"OfflineIIKF_ALL_20220414\"\n  val OfflineInterestedInFromKnownFor2020Hl0El15: String = \"OfflineIIKF_ALL_20220414_Hl0_El15\"\n  val OfflineInterestedInFromKnownFor2020Hl2El15: String = \"OfflineIIKF_ALL_20220414_Hl2_El15\"\n  val OfflineInterestedInFromKnownFor2020Hl2El50: String = \"OfflineIIKF_ALL_20220414_Hl2_El50\"\n  val OfflineInterestedInFromKnownFor2020Hl8El50: String = \"OfflineIIKF_ALL_20220414_Hl8_El50\"\n  val OfflineMTSConsumerEmbeddingsFav90P20M: String =\n    \"OfflineMTSConsumerEmbeddingsFav90P20M_ALL_20220414\"\n\n  // Twhin Model Ids\n  val ConsumerBasedTwHINRegularUpdateAll20221024: String =\n    \"ConsumerBasedTwHINRegularUpdate_All_20221024\"\n\n  // Averaged Twhin Model Ids\n  val TweetBasedTwHINRegularUpdateAll20221024: String =\n    \"TweetBasedTwHINRegularUpdate_All_20221024\"\n\n  // Collaborative Filtering Twhin Model Ids\n  val TwhinCollabFilterForFollow: String =\n    \"TwhinCollabFilterForFollow\"\n  val TwhinCollabFilterForEngagement: String =\n    \"TwhinCollabFilterForEngagement\"\n  val TwhinMultiClusterForFollow: String =\n    \"TwhinMultiClusterForFollow\"\n  val TwhinMultiClusterForEngagement: String =\n    \"TwhinMultiClusterForEngagement\"\n\n  // Two Tower model Ids\n  val TwoTowerFavALL20220808: String =\n    \"TwoTowerFav_ALL_20220808\"\n\n  // Debugger Demo-Only Model Ids\n  val DebuggerDemo: String = \"DebuggerDemo\"\n\n  // ColdStartLookalike - this is not really a model name, it is as a placeholder to\n  // indicate ColdStartLookalike candidate source, which is currently being pluged into\n  // CustomizedRetrievalCandidateGeneration temporarily.\n  val ColdStartLookalikeModelName: String = \"ConsumersBasedUtgColdStartLookalike20220707\"\n\n  // consumersBasedUTG-RealGraphOon Model Id\n  val ConsumersBasedUtgRealGraphOon20220705: String = \"ConsumersBasedUtgRealGraphOon_All_20220705\"\n  // consumersBasedUAG-RealGraphOon Model Id\n  val ConsumersBasedUagRealGraphOon20221205: String = \"ConsumersBasedUagRealGraphOon_All_20221205\"\n\n  // FTR\n  val OfflineFavDecayedSum: String = \"OfflineFavDecayedSum\"\n  val OfflineFtrAt5Pop1000RnkDcy11: String = \"OfflineFtrAt5Pop1000RnkDcy11\"\n  val OfflineFtrAt5Pop10000RnkDcy11: String = \"OfflineFtrAt5Pop10000RnkDcy11\"\n\n  // All Model Ids of HnswANNSimilarityEngines\n  val allHnswANNSimilarityEngineModelIds = Seq(\n    ConsumerBasedTwHINRegularUpdateAll20221024,\n    TwoTowerFavALL20220808,\n    DebuggerDemo\n  )\n\n  val ConsumerLogFavBasedInterestedInEmbedding: String =\n    \"ConsumerLogFavBasedInterestedIn_ALL_20221228\"\n  val ConsumerFollowBasedInterestedInEmbedding: String =\n    \"ConsumerFollowBasedInterestedIn_ALL_20221228\"\n\n  val RetweetBasedDiffusion: String =\n    \"RetweetBasedDiffusion\"\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model/ModuleNames.scala",
    "content": "package com.twitter.cr_mixer.model\n\n/**\n * Define name annotated module names here\n */\nobject ModuleNames {\n\n  final val FrsStore = \"FrsStore\"\n  final val UssStore = \"UssStore\"\n  final val UssStratoColumn = \"UssStratoColumn\"\n  final val RsxStore = \"RsxStore\"\n  final val RmsTweetLogFavLongestL2EmbeddingStore = \"RmsTweetLogFavLongestL2EmbeddingStore\"\n  final val RmsUserFavBasedProducerEmbeddingStore = \"RmsUserFavBasedProducerEmbeddingStore\"\n  final val RmsUserLogFavInterestedInEmbeddingStore = \"RmsUserLogFavInterestedInEmbeddingStore\"\n  final val RmsUserFollowInterestedInEmbeddingStore = \"RmsUserFollowInterestedInEmbeddingStore\"\n  final val StpStore = \"StpStore\"\n  final val TwiceClustersMembersStore = \"TwiceClustersMembersStore\"\n  final val TripCandidateStore = \"TripCandidateStore\"\n\n  final val ConsumerEmbeddingBasedTripSimilarityEngine =\n    \"ConsumerEmbeddingBasedTripSimilarityEngine\"\n  final val ConsumerEmbeddingBasedTwHINANNSimilarityEngine =\n    \"ConsumerEmbeddingBasedTwHINANNSimilarityEngine\"\n  final val ConsumerEmbeddingBasedTwoTowerANNSimilarityEngine =\n    \"ConsumerEmbeddingBasedTwoTowerANNSimilarityEngine\"\n  final val ConsumersBasedUserAdGraphSimilarityEngine =\n    \"ConsumersBasedUserAdGraphSimilarityEngine\"\n  final val ConsumersBasedUserVideoGraphSimilarityEngine =\n    \"ConsumersBasedUserVideoGraphSimilarityEngine\"\n\n  final val ConsumerBasedWalsSimilarityEngine = \"ConsumerBasedWalsSimilarityEngine\"\n\n  final val TweetBasedTwHINANNSimilarityEngine = \"TweetBasedTwHINANNSimilarityEngine\"\n\n  final val SimClustersANNSimilarityEngine = \"SimClustersANNSimilarityEngine\"\n\n  final val ProdSimClustersANNServiceClientName = \"ProdSimClustersANNServiceClient\"\n  final val ExperimentalSimClustersANNServiceClientName = \"ExperimentalSimClustersANNServiceClient\"\n  final val SimClustersANNServiceClientName1 = \"SimClustersANNServiceClient1\"\n  final val SimClustersANNServiceClientName2 = \"SimClustersANNServiceClient2\"\n  final val SimClustersANNServiceClientName3 = \"SimClustersANNServiceClient3\"\n  final val SimClustersANNServiceClientName5 = \"SimClustersANNServiceClient5\"\n  final val SimClustersANNServiceClientName4 = \"SimClustersANNServiceClient4\"\n  final val UnifiedCache = \"unifiedCache\"\n  final val MLScoreCache = \"mlScoreCache\"\n  final val TweetRecommendationResultsCache = \"tweetRecommendationResultsCache\"\n  final val EarlybirdTweetsCache = \"earlybirdTweetsCache\"\n  final val EarlybirdRecencyBasedWithoutRetweetsRepliesTweetsCache =\n    \"earlybirdTweetsWithoutRetweetsRepliesCacheStore\"\n  final val EarlybirdRecencyBasedWithRetweetsRepliesTweetsCache =\n    \"earlybirdTweetsWithRetweetsRepliesCacheStore\"\n\n  final val AbDeciderLogger = \"abDeciderLogger\"\n  final val TopLevelApiDdgMetricsLogger = \"topLevelApiDdgMetricsLogger\"\n  final val TweetRecsLogger = \"tweetRecsLogger\"\n  final val BlueVerifiedTweetRecsLogger = \"blueVerifiedTweetRecsLogger\"\n  final val RelatedTweetsLogger = \"relatedTweetsLogger\"\n  final val UtegTweetsLogger = \"utegTweetsLogger\"\n  final val AdsRecommendationsLogger = \"adsRecommendationLogger\"\n\n  final val OfflineSimClustersANNInterestedInSimilarityEngine =\n    \"OfflineSimClustersANNInterestedInSimilarityEngine\"\n\n  final val RealGraphOonStore = \"RealGraphOonStore\"\n  final val RealGraphInStore = \"RealGraphInStore\"\n\n  final val OfflineTweet2020CandidateStore = \"OfflineTweet2020CandidateStore\"\n  final val OfflineTweet2020Hl0El15CandidateStore = \"OfflineTweet2020Hl0El15CandidateStore\"\n  final val OfflineTweet2020Hl2El15CandidateStore = \"OfflineTweet2020Hl2El15CandidateStore\"\n  final val OfflineTweet2020Hl2El50CandidateStore = \"OfflineTweet2020Hl2El50CandidateStore\"\n  final val OfflineTweet2020Hl8El50CandidateStore = \"OfflineTweet2020Hl8El50CandidateStore\"\n  final val OfflineTweetMTSCandidateStore = \"OfflineTweetMTSCandidateStore\"\n\n  final val OfflineFavDecayedSumCandidateStore = \"OfflineFavDecayedSumCandidateStore\"\n  final val OfflineFtrAt5Pop1000RankDecay11CandidateStore =\n    \"OfflineFtrAt5Pop1000RankDecay11CandidateStore\"\n  final val OfflineFtrAt5Pop10000RankDecay11CandidateStore =\n    \"OfflineFtrAt5Pop10000RankDecay11CandidateStore\"\n\n  final val TwhinCollabFilterStratoStoreForFollow = \"TwhinCollabFilterStratoStoreForFollow\"\n  final val TwhinCollabFilterStratoStoreForEngagement = \"TwhinCollabFilterStratoStoreForEngagement\"\n  final val TwhinMultiClusterStratoStoreForFollow = \"TwhinMultiClusterStratoStoreForFollow\"\n  final val TwhinMultiClusterStratoStoreForEngagement = \"TwhinMultiClusterStratoStoreForEngagement\"\n\n  final val ProducerBasedUserAdGraphSimilarityEngine =\n    \"ProducerBasedUserAdGraphSimilarityEngine\"\n  final val ProducerBasedUserTweetGraphSimilarityEngine =\n    \"ProducerBasedUserTweetGraphSimilarityEngine\"\n  final val ProducerBasedUnifiedSimilarityEngine = \"ProducerBasedUnifiedSimilarityEngine\"\n\n  final val TweetBasedUserAdGraphSimilarityEngine = \"TweetBasedUserAdGraphSimilarityEngine\"\n  final val TweetBasedUserTweetGraphSimilarityEngine = \"TweetBasedUserTweetGraphSimilarityEngine\"\n  final val TweetBasedUserVideoGraphSimilarityEngine = \"TweetBasedUserVideoGraphSimilarityEngine\"\n  final val TweetBasedQigSimilarityEngine = \"TweetBasedQigSimilarityEngine\"\n  final val TweetBasedUnifiedSimilarityEngine = \"TweetBasedUnifiedSimilarityEngine\"\n\n  final val TwhinCollabFilterSimilarityEngine = \"TwhinCollabFilterSimilarityEngine\"\n\n  final val ConsumerBasedUserTweetGraphStore = \"ConsumerBasedUserTweetGraphStore\"\n  final val ConsumerBasedUserVideoGraphStore = \"ConsumerBasedUserVideoGraphStore\"\n  final val ConsumerBasedUserAdGraphStore = \"ConsumerBasedUserAdGraphStore\"\n\n  final val UserTweetEntityGraphSimilarityEngine =\n    \"UserTweetEntityGraphSimilarityEngine\"\n\n  final val CertoTopicTweetSimilarityEngine = \"CertoTopicTweetSimilarityEngine\"\n  final val CertoStratoStoreName = \"CertoStratoStore\"\n\n  final val SkitTopicTweetSimilarityEngine = \"SkitTopicTweetSimilarityEngine\"\n  final val SkitHighPrecisionTopicTweetSimilarityEngine =\n    \"SkitHighPrecisionTopicTweetSimilarityEngine\"\n  final val SkitStratoStoreName = \"SkitStratoStore\"\n\n  final val HomeNaviGRPCClient = \"HomeNaviGRPCClient\"\n  final val AdsFavedNaviGRPCClient = \"AdsFavedNaviGRPCClient\"\n  final val AdsMonetizableNaviGRPCClient = \"AdsMonetizableNaviGRPCClient\"\n\n  final val RetweetBasedDiffusionRecsMhStore = \"RetweetBasedDiffusionRecsMhStore\"\n  final val DiffusionBasedSimilarityEngine = \"DiffusionBasedSimilarityEngine\"\n\n  final val BlueVerifiedAnnotationStore = \"BlueVerifiedAnnotationStore\"\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model/TopicTweetWithScore.scala",
    "content": "package com.twitter.cr_mixer.model\n\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.simclusters_v2.common.TweetId\n\n/***\n * Bind a tweetId with a raw score generated from one single Similarity Engine\n * @param similarityEngineType, which underlying topic source the topic tweet is from\n */\ncase class TopicTweetWithScore(\n  tweetId: TweetId,\n  score: Double,\n  similarityEngineType: SimilarityEngineType)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model/TweetWithAuthor.scala",
    "content": "package com.twitter.cr_mixer.model\n\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\n\ncase class TweetWithAuthor(tweetId: TweetId, authorId: UserId)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model/TweetWithScore.scala",
    "content": "package com.twitter.cr_mixer.model\n\nimport com.twitter.simclusters_v2.common.TweetId\n\n/***\n * Bind a tweetId with a raw score generated from one single Similarity Engine\n */\ncase class TweetWithScore(tweetId: TweetId, score: Double)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model/TweetWithScoreAndSocialProof.scala",
    "content": "package com.twitter.cr_mixer.model\n\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.recos.recos_common.thriftscala.SocialProofType\n\n/***\n * Bind a tweetId with a raw score and social proofs by type\n */\ncase class TweetWithScoreAndSocialProof(\n  tweetId: TweetId,\n  score: Double,\n  socialProofByType: Map[SocialProofType, Seq[Long]])\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/ActivePromotedTweetStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.bijection.thrift.CompactThriftCodec\nimport com.twitter.ads.entities.db.thriftscala.LineItemObjective\nimport com.twitter.bijection.Injection\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.thriftscala.LineItemInfo\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.store.common.ObservedCachedReadableStore\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.DataType\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.GeneralTensor\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.relevance_platform.common.injection.LZ4Injection\nimport com.twitter.relevance_platform.common.injection.SeqObjectInjection\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.ManhattanRO\nimport com.twitter.storehaus_internal.manhattan.ManhattanROConfig\nimport com.twitter.storehaus_internal.manhattan.Revenue\nimport com.twitter.storehaus_internal.util.ApplicationID\nimport com.twitter.storehaus_internal.util.DatasetName\nimport com.twitter.storehaus_internal.util.HDFSPath\nimport com.twitter.util.Future\nimport javax.inject.Named\nimport scala.collection.JavaConverters._\n\nobject ActivePromotedTweetStoreModule extends TwitterModule {\n\n  case class ActivePromotedTweetStore(\n    activePromotedTweetMHStore: ReadableStore[String, DataRecord],\n    statsReceiver: StatsReceiver)\n      extends ReadableStore[TweetId, Seq[LineItemInfo]] {\n    override def get(tweetId: TweetId): Future[Option[Seq[LineItemInfo]]] = {\n      activePromotedTweetMHStore.get(tweetId.toString).map {\n        _.map { dataRecord =>\n          val richDataRecord = new RichDataRecord(dataRecord)\n          val lineItemIdsFeature: Feature[GeneralTensor] =\n            new Feature.Tensor(\"active_promoted_tweets.line_item_ids\", DataType.INT64)\n\n          val lineItemObjectivesFeature: Feature[GeneralTensor] =\n            new Feature.Tensor(\"active_promoted_tweets.line_item_objectives\", DataType.INT64)\n\n          val lineItemIdsTensor: GeneralTensor = richDataRecord.getFeatureValue(lineItemIdsFeature)\n          val lineItemObjectivesTensor: GeneralTensor =\n            richDataRecord.getFeatureValue(lineItemObjectivesFeature)\n\n          val lineItemIds: Seq[Long] =\n            if (lineItemIdsTensor.getSetField == GeneralTensor._Fields.INT64_TENSOR && lineItemIdsTensor.getInt64Tensor.isSetLongs) {\n              lineItemIdsTensor.getInt64Tensor.getLongs.asScala.map(_.toLong)\n            } else Seq.empty\n\n          val lineItemObjectives: Seq[LineItemObjective] =\n            if (lineItemObjectivesTensor.getSetField == GeneralTensor._Fields.INT64_TENSOR && lineItemObjectivesTensor.getInt64Tensor.isSetLongs) {\n              lineItemObjectivesTensor.getInt64Tensor.getLongs.asScala.map(objective =>\n                LineItemObjective(objective.toInt))\n            } else Seq.empty\n\n          val lineItemInfo =\n            if (lineItemIds.size == lineItemObjectives.size) {\n              lineItemIds.zipWithIndex.map {\n                case (lineItemId, index) =>\n                  LineItemInfo(\n                    lineItemId = lineItemId,\n                    lineItemObjective = lineItemObjectives(index)\n                  )\n              }\n            } else Seq.empty\n\n          lineItemInfo\n        }\n      }\n    }\n  }\n\n  @Provides\n  @Singleton\n  def providesActivePromotedTweetStore(\n    manhattanKVClientMtlsParams: ManhattanKVClientMtlsParams,\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    crMixerStatsReceiver: StatsReceiver\n  ): ReadableStore[TweetId, Seq[LineItemInfo]] = {\n\n    val mhConfig = new ManhattanROConfig {\n      val hdfsPath = HDFSPath(\"\")\n      val applicationID = ApplicationID(\"ads_bigquery_features\")\n      val datasetName = DatasetName(\"active_promoted_tweets\")\n      val cluster = Revenue\n\n      override def statsReceiver: StatsReceiver =\n        crMixerStatsReceiver.scope(\"active_promoted_tweets_mh\")\n    }\n    val mhStore: ReadableStore[String, DataRecord] =\n      ManhattanRO\n        .getReadableStoreWithMtls[String, DataRecord](\n          mhConfig,\n          manhattanKVClientMtlsParams\n        )(\n          implicitly[Injection[String, Array[Byte]]],\n          CompactThriftCodec[DataRecord]\n        )\n\n    val underlyingStore =\n      ActivePromotedTweetStore(mhStore, crMixerStatsReceiver.scope(\"ActivePromotedTweetStore\"))\n    val memcachedStore = ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = underlyingStore,\n      cacheClient = crMixerUnifiedCacheClient,\n      ttl = 60.minutes,\n      asyncUpdate = false\n    )(\n      valueInjection = LZ4Injection.compose(SeqObjectInjection[LineItemInfo]()),\n      statsReceiver = crMixerStatsReceiver.scope(\"memCachedActivePromotedTweetStore\"),\n      keyToString = { k: TweetId => s\"apt/$k\" }\n    )\n\n    ObservedCachedReadableStore.from(\n      memcachedStore,\n      ttl = 30.minutes,\n      maxKeys = 250000, // size of promoted tweet is around 200,000\n      windowSize = 10000L,\n      cacheName = \"active_promoted_tweet_cache\",\n      maxMultiGetSize = 20\n    )(crMixerStatsReceiver.scope(\"inMemoryCachedActivePromotedTweetStore\"))\n\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n        \"core/*.scala\",\n        \"grpc_client/*.scala\",\n        \"similarity_engine/*.scala\",\n        \"source_signal/*.scala\",\n        \"thrift_client/*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/bijection:core\",\n        \"3rdparty/jvm/com/twitter/bijection:scrooge\",\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/com/twitter/storehaus:memcache\",\n        \"3rdparty/jvm/io/grpc:grpc-api\",\n        \"3rdparty/jvm/io/grpc:grpc-auth\",\n        \"3rdparty/jvm/io/grpc:grpc-core\",\n        \"3rdparty/jvm/io/grpc:grpc-netty\",\n        \"3rdparty/jvm/io/grpc:grpc-protobuf\",\n        \"3rdparty/jvm/io/grpc:grpc-stub\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/jvm/org/scalanlp:breeze\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"abdecider/src/main/scala\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"configapi/configapi-abdecider\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-featureswitches:v2\",\n        \"content-recommender/server/src/main/scala/com/twitter/contentrecommender:cr-mixer-deps\",\n        \"content-recommender/thrift/src/main/thrift:thrift-scala\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/candidate_generation\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/config\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/featureswitch\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/ranker\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/scribe\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"decider/src/main/scala\",\n        \"discovery-common/src/main/scala/com/twitter/discovery/common/configapi\",\n        \"featureswitches/featureswitches-core\",\n        \"featureswitches/featureswitches-core/src/main/scala/com/twitter/featureswitches/v2/builder\",\n        \"finagle-internal/finagle-grpc/src/main/scala\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finatra-internal/kafka/src/main/scala/com/twitter/finatra/kafka/consumers\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-modules/src/main/scala\",\n        \"finatra/inject/inject-thrift-client\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"frigate/frigate-common:util\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/base\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/candidate\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/health\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/interests\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/strato\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"hydra/partition/thrift/src/main/thrift:thrift-scala\",\n        \"hydra/root/thrift/src/main/thrift:thrift-scala\",\n        \"mediaservices/commons/src/main/scala:futuretracker\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"qig-ranker/thrift/src/main/thrift:thrift-scala\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/health_store\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/injection\",\n        \"relevance-platform/thrift/src/main/thrift:thrift-scala\",\n        \"representation-manager/client/src/main/scala/com/twitter/representation_manager\",\n        \"representation-manager/client/src/main/scala/com/twitter/representation_manager/config\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/migration\",\n        \"representation-manager/server/src/main/thrift:thrift-scala\",\n        \"representation-scorer/server/src/main/thrift:thrift-scala\",\n        \"servo/decider\",\n        \"servo/util/src/main/scala\",\n        \"simclusters-ann/thrift/src/main/thrift:thrift-scala\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/java/com/twitter/search/queryparser/query/search:search-query-nodes\",\n        \"src/scala/com/twitter/algebird_internal/injection\",\n        \"src/scala/com/twitter/cortex/ml/embeddings/common:Helpers\",\n        \"src/scala/com/twitter/ml/api/embedding\",\n        \"src/scala/com/twitter/ml/featurestore/lib\",\n        \"src/scala/com/twitter/scalding_internal/multiformat/format\",\n        \"src/scala/com/twitter/simclusters_v2/candidate_source\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/scala/com/twitter/storehaus_internal/manhattan\",\n        \"src/scala/com/twitter/storehaus_internal/manhattan/config\",\n        \"src/scala/com/twitter/storehaus_internal/memcache\",\n        \"src/scala/com/twitter/storehaus_internal/memcache/config\",\n        \"src/scala/com/twitter/storehaus_internal/offline\",\n        \"src/scala/com/twitter/storehaus_internal/util\",\n        \"src/scala/com/twitter/topic_recos/stores\",\n        \"src/thrift/com/twitter/core_workflows/user_model:user_model-scala\",\n        \"src/thrift/com/twitter/frigate:frigate-common-thrift-scala\",\n        \"src/thrift/com/twitter/frigate:frigate-thrift-scala\",\n        \"src/thrift/com/twitter/frigate/data_pipeline/scalding:blue_verified_annotations-scala\",\n        \"src/thrift/com/twitter/hermit/stp:hermit-stp-scala\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/ml/api:embedding-scala\",\n        \"src/thrift/com/twitter/ml/featurestore:ml-feature-store-embedding-scala\",\n        \"src/thrift/com/twitter/onboarding/relevance/coldstart_lookalike:coldstartlookalike-thrift-scala\",\n        \"src/thrift/com/twitter/recos:recos-common-scala\",\n        \"src/thrift/com/twitter/recos/user_ad_graph:user_ad_graph-scala\",\n        \"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala\",\n        \"src/thrift/com/twitter/recos/user_tweet_graph:user_tweet_graph-scala\",\n        \"src/thrift/com/twitter/recos/user_tweet_graph_plus:user_tweet_graph_plus-scala\",\n        \"src/thrift/com/twitter/recos/user_video_graph:user_video_graph-scala\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"src/thrift/com/twitter/search/query_interaction_graph/service:qig-service-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"src/thrift/com/twitter/topic_recos:topic_recos-thrift-scala\",\n        \"src/thrift/com/twitter/trends/trip_v1:trip-tweets-thrift-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"src/thrift/com/twitter/twistly:twistly-scala\",\n        \"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala\",\n        \"stitch/stitch-storehaus\",\n        \"stitch/stitch-tweetypie/src/main/scala\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"user-signal-service/thrift/src/main/thrift:thrift-scala\",\n        \"util-internal/scribe/src/main/scala/com/twitter/logging\",\n        \"util/util-hashing\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/BlueVerifiedAnnotationStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.google.inject.name.Named\nimport com.twitter.inject.TwitterModule\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.data_pipeline.scalding.thriftscala.BlueVerifiedAnnotationsV2\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.Athena\nimport com.twitter.storehaus_internal.manhattan.ManhattanRO\nimport com.twitter.storehaus_internal.manhattan.ManhattanROConfig\nimport com.twitter.storehaus_internal.util.ApplicationID\nimport com.twitter.storehaus_internal.util.DatasetName\nimport com.twitter.storehaus_internal.util.HDFSPath\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.hermit.store.common.ObservedCachedReadableStore\n\nobject BlueVerifiedAnnotationStoreModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.BlueVerifiedAnnotationStore)\n  def providesBlueVerifiedAnnotationStore(\n    statsReceiver: StatsReceiver,\n    manhattanKVClientMtlsParams: ManhattanKVClientMtlsParams,\n  ): ReadableStore[String, BlueVerifiedAnnotationsV2] = {\n\n    implicit val valueCodec = new BinaryScalaCodec(BlueVerifiedAnnotationsV2)\n\n    val underlyingStore = ManhattanRO\n      .getReadableStoreWithMtls[String, BlueVerifiedAnnotationsV2](\n        ManhattanROConfig(\n          HDFSPath(\"\"),\n          ApplicationID(\"content_recommender_athena\"),\n          DatasetName(\"blue_verified_annotations\"),\n          Athena),\n        manhattanKVClientMtlsParams\n      )\n\n    ObservedCachedReadableStore.from(\n      underlyingStore,\n      ttl = 24.hours,\n      maxKeys = 100000,\n      windowSize = 10000L,\n      cacheName = \"blue_verified_annotation_cache\"\n    )(statsReceiver.scope(\"inMemoryCachedBlueVerifiedAnnotationStore\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/CertoStratoStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.google.inject.name.Named\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.keyHasher\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.store.common.ObservedCachedReadableStore\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.relevance_platform.common.injection.LZ4Injection\nimport com.twitter.relevance_platform.common.injection.SeqObjectInjection\nimport com.twitter.simclusters_v2.thriftscala.TopicId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.Client\nimport com.twitter.topic_recos.stores.CertoTopicTopKTweetsStore\nimport com.twitter.topic_recos.thriftscala.TweetWithScores\n\nobject CertoStratoStoreModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.CertoStratoStoreName)\n  def providesCertoStratoStore(\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    stratoClient: Client,\n    statsReceiver: StatsReceiver\n  ): ReadableStore[TopicId, Seq[TweetWithScores]] = {\n    val certoStore = ObservedReadableStore(CertoTopicTopKTweetsStore.prodStore(stratoClient))(\n      statsReceiver.scope(ModuleNames.CertoStratoStoreName)).mapValues { topKTweetsWithScores =>\n      topKTweetsWithScores.topTweetsByFollowerL2NormalizedCosineSimilarityScore\n    }\n\n    val memCachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = certoStore,\n        cacheClient = crMixerUnifiedCacheClient,\n        ttl = 10.minutes\n      )(\n        valueInjection = LZ4Injection.compose(SeqObjectInjection[TweetWithScores]()),\n        statsReceiver = statsReceiver.scope(\"memcached_certo_store\"),\n        keyToString = { k => s\"certo:${keyHasher.hashKey(k.toString.getBytes)}\" }\n      )\n\n    ObservedCachedReadableStore.from[TopicId, Seq[TweetWithScores]](\n      memCachedStore,\n      ttl = 5.minutes,\n      maxKeys = 100000, // ~150MB max\n      cacheName = \"certo_in_memory_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"certo_in_memory_cache\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/ConsumersBasedUserAdGraphStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.inject.TwitterModule\nimport com.twitter.recos.user_ad_graph.thriftscala.ConsumersBasedRelatedAdRequest\nimport com.twitter.recos.user_ad_graph.thriftscala.RelatedAdResponse\nimport com.twitter.recos.user_ad_graph.thriftscala.UserAdGraph\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject ConsumersBasedUserAdGraphStoreModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.ConsumerBasedUserAdGraphStore)\n  def providesConsumerBasedUserAdGraphStore(\n    userAdGraphService: UserAdGraph.MethodPerEndpoint\n  ): ReadableStore[ConsumersBasedRelatedAdRequest, RelatedAdResponse] = {\n    new ReadableStore[ConsumersBasedRelatedAdRequest, RelatedAdResponse] {\n      override def get(\n        k: ConsumersBasedRelatedAdRequest\n      ): Future[Option[RelatedAdResponse]] = {\n        userAdGraphService.consumersBasedRelatedAds(k).map(Some(_))\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/ConsumersBasedUserTweetGraphStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.inject.TwitterModule\nimport com.twitter.recos.user_tweet_graph.thriftscala.ConsumersBasedRelatedTweetRequest\nimport com.twitter.recos.user_tweet_graph.thriftscala.RelatedTweetResponse\nimport com.twitter.recos.user_tweet_graph.thriftscala.UserTweetGraph\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject ConsumersBasedUserTweetGraphStoreModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.ConsumerBasedUserTweetGraphStore)\n  def providesConsumerBasedUserTweetGraphStore(\n    userTweetGraphService: UserTweetGraph.MethodPerEndpoint\n  ): ReadableStore[ConsumersBasedRelatedTweetRequest, RelatedTweetResponse] = {\n    new ReadableStore[ConsumersBasedRelatedTweetRequest, RelatedTweetResponse] {\n      override def get(\n        k: ConsumersBasedRelatedTweetRequest\n      ): Future[Option[RelatedTweetResponse]] = {\n        userTweetGraphService.consumersBasedRelatedTweets(k).map(Some(_))\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/ConsumersBasedUserVideoGraphStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.inject.TwitterModule\nimport com.twitter.recos.user_video_graph.thriftscala.ConsumersBasedRelatedTweetRequest\nimport com.twitter.recos.user_video_graph.thriftscala.RelatedTweetResponse\nimport com.twitter.recos.user_video_graph.thriftscala.UserVideoGraph\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject ConsumersBasedUserVideoGraphStoreModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.ConsumerBasedUserVideoGraphStore)\n  def providesConsumerBasedUserVideoGraphStore(\n    userVideoGraphService: UserVideoGraph.MethodPerEndpoint\n  ): ReadableStore[ConsumersBasedRelatedTweetRequest, RelatedTweetResponse] = {\n    new ReadableStore[ConsumersBasedRelatedTweetRequest, RelatedTweetResponse] {\n      override def get(\n        k: ConsumersBasedRelatedTweetRequest\n      ): Future[Option[RelatedTweetResponse]] = {\n        userVideoGraphService.consumersBasedRelatedTweets(k).map(Some(_))\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/CrMixerParamConfigModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.timelines.configapi.Config\nimport com.twitter.cr_mixer.param.CrMixerParamConfig\nimport com.twitter.inject.TwitterModule\nimport javax.inject.Singleton\n\nobject CrMixerParamConfigModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def provideConfig(): Config = {\n    CrMixerParamConfig.config\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/DiffusionStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.bijection.Injection\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.thriftscala.TweetsWithScore\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.Apollo\nimport com.twitter.storehaus_internal.manhattan.ManhattanRO\nimport com.twitter.storehaus_internal.manhattan.ManhattanROConfig\nimport com.twitter.storehaus_internal.util.ApplicationID\nimport com.twitter.storehaus_internal.util.DatasetName\nimport com.twitter.storehaus_internal.util.HDFSPath\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject DiffusionStoreModule extends TwitterModule {\n  type UserId = Long\n  implicit val longCodec = implicitly[Injection[Long, Array[Byte]]]\n  implicit val tweetRecsInjection: Injection[TweetsWithScore, Array[Byte]] =\n    BinaryScalaCodec(TweetsWithScore)\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.RetweetBasedDiffusionRecsMhStore)\n  def retweetBasedDiffusionRecsMhStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[Long, TweetsWithScore] = {\n    val manhattanROConfig = ManhattanROConfig(\n      HDFSPath(\"\"), // not needed\n      ApplicationID(\"cr_mixer_apollo\"),\n      DatasetName(\"diffusion_retweet_tweet_recs\"),\n      Apollo\n    )\n\n    buildTweetRecsStore(serviceIdentifier, manhattanROConfig)\n  }\n\n  private def buildTweetRecsStore(\n    serviceIdentifier: ServiceIdentifier,\n    manhattanROConfig: ManhattanROConfig\n  ): ReadableStore[Long, TweetsWithScore] = {\n\n    ManhattanRO\n      .getReadableStoreWithMtls[Long, TweetsWithScore](\n        manhattanROConfig,\n        ManhattanKVClientMtlsParams(serviceIdentifier)\n      )(longCodec, tweetRecsInjection)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/EarlybirdRecencyBasedCandidateStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.EarlybirdClientId\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.FacetsToFetch\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.GetCollectorTerminationParams\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.GetEarlybirdQuery\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.MetadataOptions\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.SeqLongInjection\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.search.common.query.thriftjava.thriftscala.CollectorParams\nimport com.twitter.search.earlybird.thriftscala.EarlybirdRequest\nimport com.twitter.search.earlybird.thriftscala.EarlybirdResponseCode\nimport com.twitter.search.earlybird.thriftscala.EarlybirdService\nimport com.twitter.search.earlybird.thriftscala.ThriftSearchQuery\nimport com.twitter.search.earlybird.thriftscala.ThriftSearchRankingMode\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport javax.inject.Named\n\nobject EarlybirdRecencyBasedCandidateStoreModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.EarlybirdRecencyBasedWithoutRetweetsRepliesTweetsCache)\n  def providesEarlybirdRecencyBasedWithoutRetweetsRepliesCandidateStore(\n    statsReceiver: StatsReceiver,\n    earlybirdSearchClient: EarlybirdService.MethodPerEndpoint,\n    @Named(ModuleNames.EarlybirdTweetsCache) earlybirdRecencyBasedTweetsCache: MemcachedClient,\n    timeoutConfig: TimeoutConfig\n  ): ReadableStore[UserId, Seq[TweetId]] = {\n    val stats = statsReceiver.scope(\"EarlybirdRecencyBasedWithoutRetweetsRepliesCandidateStore\")\n    val underlyingStore = new ReadableStore[UserId, Seq[TweetId]] {\n      override def get(userId: UserId): Future[Option[Seq[TweetId]]] = {\n        // Home based EB filters out retweets and replies\n        val earlybirdRequest =\n          buildEarlybirdRequest(\n            userId,\n            FilterOutRetweetsAndReplies,\n            DefaultMaxNumTweetPerUser,\n            timeoutConfig.earlybirdServerTimeout)\n        getEarlybirdSearchResult(earlybirdSearchClient, earlybirdRequest, stats)\n      }\n    }\n    ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = underlyingStore,\n      cacheClient = earlybirdRecencyBasedTweetsCache,\n      ttl = MemcacheKeyTimeToLiveDuration,\n      asyncUpdate = true\n    )(\n      valueInjection = SeqLongInjection,\n      statsReceiver = statsReceiver.scope(\"earlybird_recency_based_tweets_home_memcache\"),\n      keyToString = { k =>\n        f\"uEBRBHM:${keyHasher.hashKey(k.toString.getBytes)}%X\" // prefix = EarlyBirdRecencyBasedHoMe\n      }\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.EarlybirdRecencyBasedWithRetweetsRepliesTweetsCache)\n  def providesEarlybirdRecencyBasedWithRetweetsRepliesCandidateStore(\n    statsReceiver: StatsReceiver,\n    earlybirdSearchClient: EarlybirdService.MethodPerEndpoint,\n    @Named(ModuleNames.EarlybirdTweetsCache) earlybirdRecencyBasedTweetsCache: MemcachedClient,\n    timeoutConfig: TimeoutConfig\n  ): ReadableStore[UserId, Seq[TweetId]] = {\n    val stats = statsReceiver.scope(\"EarlybirdRecencyBasedWithRetweetsRepliesCandidateStore\")\n    val underlyingStore = new ReadableStore[UserId, Seq[TweetId]] {\n      override def get(userId: UserId): Future[Option[Seq[TweetId]]] = {\n        val earlybirdRequest = buildEarlybirdRequest(\n          userId,\n          // Notifications based EB keeps retweets and replies\n          NotFilterOutRetweetsAndReplies,\n          DefaultMaxNumTweetPerUser,\n          processingTimeout = timeoutConfig.earlybirdServerTimeout\n        )\n        getEarlybirdSearchResult(earlybirdSearchClient, earlybirdRequest, stats)\n      }\n    }\n    ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = underlyingStore,\n      cacheClient = earlybirdRecencyBasedTweetsCache,\n      ttl = MemcacheKeyTimeToLiveDuration,\n      asyncUpdate = true\n    )(\n      valueInjection = SeqLongInjection,\n      statsReceiver = statsReceiver.scope(\"earlybird_recency_based_tweets_notifications_memcache\"),\n      keyToString = { k =>\n        f\"uEBRBN:${keyHasher.hashKey(k.toString.getBytes)}%X\" // prefix = EarlyBirdRecencyBasedNotifications\n      }\n    )\n  }\n\n  private val keyHasher: KeyHasher = KeyHasher.FNV1A_64\n\n  /**\n   * Note the DefaultMaxNumTweetPerUser is used to adjust the result size per cache entry.\n   * If the value changes, it will increase the size of the memcache.\n   */\n  private val DefaultMaxNumTweetPerUser: Int = 100\n  private val FilterOutRetweetsAndReplies = true\n  private val NotFilterOutRetweetsAndReplies = false\n  private val MemcacheKeyTimeToLiveDuration: Duration = Duration.fromMinutes(15)\n\n  private def buildEarlybirdRequest(\n    seedUserId: UserId,\n    filterOutRetweetsAndReplies: Boolean,\n    maxNumTweetsPerSeedUser: Int,\n    processingTimeout: Duration\n  ): EarlybirdRequest =\n    EarlybirdRequest(\n      searchQuery = getThriftSearchQuery(\n        seedUserId = seedUserId,\n        filterOutRetweetsAndReplies = filterOutRetweetsAndReplies,\n        maxNumTweetsPerSeedUser = maxNumTweetsPerSeedUser,\n        processingTimeout = processingTimeout\n      ),\n      clientId = Some(EarlybirdClientId),\n      timeoutMs = processingTimeout.inMilliseconds.intValue(),\n      getOlderResults = Some(false),\n      adjustedProtectedRequestParams = None,\n      adjustedFullArchiveRequestParams = None,\n      getProtectedTweetsOnly = Some(false),\n      skipVeryRecentTweets = true,\n    )\n\n  private def getThriftSearchQuery(\n    seedUserId: UserId,\n    filterOutRetweetsAndReplies: Boolean,\n    maxNumTweetsPerSeedUser: Int,\n    processingTimeout: Duration\n  ): ThriftSearchQuery = ThriftSearchQuery(\n    serializedQuery = GetEarlybirdQuery(\n      None,\n      None,\n      Set.empty,\n      filterOutRetweetsAndReplies\n    ).map(_.serialize),\n    fromUserIDFilter64 = Some(Seq(seedUserId)),\n    numResults = maxNumTweetsPerSeedUser,\n    rankingMode = ThriftSearchRankingMode.Recency,\n    collectorParams = Some(\n      CollectorParams(\n        // numResultsToReturn defines how many results each EB shard will return to search root\n        numResultsToReturn = maxNumTweetsPerSeedUser,\n        // terminationParams.maxHitsToProcess is used for early terminating per shard results fetching.\n        terminationParams =\n          GetCollectorTerminationParams(maxNumTweetsPerSeedUser, processingTimeout)\n      )),\n    facetFieldNames = Some(FacetsToFetch),\n    resultMetadataOptions = Some(MetadataOptions),\n    searchStatusIds = None\n  )\n\n  private def getEarlybirdSearchResult(\n    earlybirdSearchClient: EarlybirdService.MethodPerEndpoint,\n    request: EarlybirdRequest,\n    statsReceiver: StatsReceiver\n  ): Future[Option[Seq[TweetId]]] = earlybirdSearchClient\n    .search(request)\n    .map { response =>\n      response.responseCode match {\n        case EarlybirdResponseCode.Success =>\n          val earlybirdSearchResult =\n            response.searchResults\n              .map {\n                _.results\n                  .map(searchResult => searchResult.id)\n              }\n          statsReceiver.scope(\"result\").stat(\"size\").add(earlybirdSearchResult.size)\n          earlybirdSearchResult\n        case e =>\n          statsReceiver.scope(\"failures\").counter(e.getClass.getSimpleName).incr()\n          Some(Seq.empty)\n      }\n    }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/EmbeddingStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.bijection.Injection\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.bijection.scrooge.CompactScalaCodec\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.inject.TwitterModule\nimport com.twitter.ml.api.{thriftscala => api}\nimport com.twitter.simclusters_v2.thriftscala.CandidateTweetsList\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.Apollo\nimport com.twitter.storehaus_internal.manhattan.ManhattanRO\nimport com.twitter.storehaus_internal.manhattan.ManhattanROConfig\nimport com.twitter.storehaus_internal.util.ApplicationID\nimport com.twitter.storehaus_internal.util.DatasetName\nimport com.twitter.storehaus_internal.util.HDFSPath\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject EmbeddingStoreModule extends TwitterModule {\n  type UserId = Long\n  implicit val mbcgUserEmbeddingInjection: Injection[api.Embedding, Array[Byte]] =\n    CompactScalaCodec(api.Embedding)\n  implicit val tweetCandidatesInjection: Injection[CandidateTweetsList, Array[Byte]] =\n    CompactScalaCodec(CandidateTweetsList)\n\n  final val TwHINEmbeddingRegularUpdateMhStoreName = \"TwHINEmbeddingRegularUpdateMhStore\"\n  @Provides\n  @Singleton\n  @Named(TwHINEmbeddingRegularUpdateMhStoreName)\n  def twHINEmbeddingRegularUpdateMhStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[InternalId, api.Embedding] = {\n    val binaryEmbeddingInjection: Injection[api.Embedding, Array[Byte]] =\n      BinaryScalaCodec(api.Embedding)\n\n    val longCodec = implicitly[Injection[Long, Array[Byte]]]\n\n    ManhattanRO\n      .getReadableStoreWithMtls[TweetId, api.Embedding](\n        ManhattanROConfig(\n          HDFSPath(\"\"), // not needed\n          ApplicationID(\"cr_mixer_apollo\"),\n          DatasetName(\"twhin_regular_update_tweet_embedding_apollo\"),\n          Apollo\n        ),\n        ManhattanKVClientMtlsParams(serviceIdentifier)\n      )(longCodec, binaryEmbeddingInjection).composeKeyMapping[InternalId] {\n        case InternalId.TweetId(tweetId) =>\n          tweetId\n        case _ =>\n          throw new UnsupportedOperationException(\"Invalid Internal Id\")\n      }\n  }\n\n  final val ConsumerBasedTwHINEmbeddingRegularUpdateMhStoreName =\n    \"ConsumerBasedTwHINEmbeddingRegularUpdateMhStore\"\n  @Provides\n  @Singleton\n  @Named(ConsumerBasedTwHINEmbeddingRegularUpdateMhStoreName)\n  def consumerBasedTwHINEmbeddingRegularUpdateMhStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[InternalId, api.Embedding] = {\n    val binaryEmbeddingInjection: Injection[api.Embedding, Array[Byte]] =\n      BinaryScalaCodec(api.Embedding)\n\n    val longCodec = implicitly[Injection[Long, Array[Byte]]]\n\n    ManhattanRO\n      .getReadableStoreWithMtls[UserId, api.Embedding](\n        ManhattanROConfig(\n          HDFSPath(\"\"), // not needed\n          ApplicationID(\"cr_mixer_apollo\"),\n          DatasetName(\"twhin_user_embedding_regular_update_apollo\"),\n          Apollo\n        ),\n        ManhattanKVClientMtlsParams(serviceIdentifier)\n      )(longCodec, binaryEmbeddingInjection).composeKeyMapping[InternalId] {\n        case InternalId.UserId(userId) =>\n          userId\n        case _ =>\n          throw new UnsupportedOperationException(\"Invalid Internal Id\")\n      }\n  }\n\n  final val TwoTowerFavConsumerEmbeddingMhStoreName = \"TwoTowerFavConsumerEmbeddingMhStore\"\n  @Provides\n  @Singleton\n  @Named(TwoTowerFavConsumerEmbeddingMhStoreName)\n  def twoTowerFavConsumerEmbeddingMhStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[InternalId, api.Embedding] = {\n    val binaryEmbeddingInjection: Injection[api.Embedding, Array[Byte]] =\n      BinaryScalaCodec(api.Embedding)\n\n    val longCodec = implicitly[Injection[Long, Array[Byte]]]\n\n    ManhattanRO\n      .getReadableStoreWithMtls[UserId, api.Embedding](\n        ManhattanROConfig(\n          HDFSPath(\"\"), // not needed\n          ApplicationID(\"cr_mixer_apollo\"),\n          DatasetName(\"two_tower_fav_user_embedding_apollo\"),\n          Apollo\n        ),\n        ManhattanKVClientMtlsParams(serviceIdentifier)\n      )(longCodec, binaryEmbeddingInjection).composeKeyMapping[InternalId] {\n        case InternalId.UserId(userId) =>\n          userId\n        case _ =>\n          throw new UnsupportedOperationException(\"Invalid Internal Id\")\n      }\n  }\n\n  final val DebuggerDemoUserEmbeddingMhStoreName = \"DebuggerDemoUserEmbeddingMhStoreName\"\n  @Provides\n  @Singleton\n  @Named(DebuggerDemoUserEmbeddingMhStoreName)\n  def debuggerDemoUserEmbeddingStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[InternalId, api.Embedding] = {\n    // This dataset is from src/scala/com/twitter/wtf/beam/bq_embedding_export/sql/MlfExperimentalUserEmbeddingScalaDataset.sql\n    // Change the above sql if you want to use a diff embedding\n    val manhattanROConfig = ManhattanROConfig(\n      HDFSPath(\"\"), // not needed\n      ApplicationID(\"cr_mixer_apollo\"),\n      DatasetName(\"experimental_user_embedding\"),\n      Apollo\n    )\n    buildUserEmbeddingStore(serviceIdentifier, manhattanROConfig)\n  }\n\n  final val DebuggerDemoTweetEmbeddingMhStoreName = \"DebuggerDemoTweetEmbeddingMhStore\"\n  @Provides\n  @Singleton\n  @Named(DebuggerDemoTweetEmbeddingMhStoreName)\n  def debuggerDemoTweetEmbeddingStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[InternalId, api.Embedding] = {\n    // This dataset is from src/scala/com/twitter/wtf/beam/bq_embedding_export/sql/MlfExperimentalTweetEmbeddingScalaDataset.sql\n    // Change the above sql if you want to use a diff embedding\n    val manhattanROConfig = ManhattanROConfig(\n      HDFSPath(\"\"), // not needed\n      ApplicationID(\"cr_mixer_apollo\"),\n      DatasetName(\"experimental_tweet_embedding\"),\n      Apollo\n    )\n    buildTweetEmbeddingStore(serviceIdentifier, manhattanROConfig)\n  }\n\n  private def buildUserEmbeddingStore(\n    serviceIdentifier: ServiceIdentifier,\n    manhattanROConfig: ManhattanROConfig\n  ): ReadableStore[InternalId, api.Embedding] = {\n    val binaryEmbeddingInjection: Injection[api.Embedding, Array[Byte]] =\n      BinaryScalaCodec(api.Embedding)\n\n    val longCodec = implicitly[Injection[Long, Array[Byte]]]\n    ManhattanRO\n      .getReadableStoreWithMtls[UserId, api.Embedding](\n        manhattanROConfig,\n        ManhattanKVClientMtlsParams(serviceIdentifier)\n      )(longCodec, binaryEmbeddingInjection).composeKeyMapping[InternalId] {\n        case InternalId.UserId(userId) =>\n          userId\n        case _ =>\n          throw new UnsupportedOperationException(\"Invalid Internal Id\")\n      }\n  }\n\n  private def buildTweetEmbeddingStore(\n    serviceIdentifier: ServiceIdentifier,\n    manhattanROConfig: ManhattanROConfig\n  ): ReadableStore[InternalId, api.Embedding] = {\n    val binaryEmbeddingInjection: Injection[api.Embedding, Array[Byte]] =\n      BinaryScalaCodec(api.Embedding)\n\n    val longCodec = implicitly[Injection[Long, Array[Byte]]]\n\n    ManhattanRO\n      .getReadableStoreWithMtls[TweetId, api.Embedding](\n        manhattanROConfig,\n        ManhattanKVClientMtlsParams(serviceIdentifier)\n      )(longCodec, binaryEmbeddingInjection).composeKeyMapping[InternalId] {\n        case InternalId.TweetId(tweetId) =>\n          tweetId\n        case _ =>\n          throw new UnsupportedOperationException(\"Invalid Internal Id\")\n      }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/FrsStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.source_signal.FrsStore\nimport com.twitter.cr_mixer.source_signal.FrsStore.FrsQueryResult\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.follow_recommendations.thriftscala.FollowRecommendationsThriftService\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\n\nobject FrsStoreModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.FrsStore)\n  def providesFrsStore(\n    frsClient: FollowRecommendationsThriftService.MethodPerEndpoint,\n    statsReceiver: StatsReceiver,\n    decider: CrMixerDecider\n  ): ReadableStore[FrsStore.Query, Seq[FrsQueryResult]] = {\n    ObservedReadableStore(FrsStore(frsClient, statsReceiver, decider))(\n      statsReceiver.scope(\"follow_recommendations_store\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/MHMtlsParamsModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport javax.inject.Singleton\n\nobject MHMtlsParamsModule extends TwitterModule {\n  @Singleton\n  @Provides\n  def providesManhattanMtlsParams(\n    serviceIdentifier: ServiceIdentifier\n  ): ManhattanKVClientMtlsParams = {\n    ManhattanKVClientMtlsParams(serviceIdentifier)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/OfflineCandidateStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.bijection.Injection\nimport com.twitter.bijection.scrooge.CompactScalaCodec\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.thriftscala.CandidateTweetsList\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.Apollo\nimport com.twitter.storehaus_internal.manhattan.ManhattanRO\nimport com.twitter.storehaus_internal.manhattan.ManhattanROConfig\nimport com.twitter.storehaus_internal.util.ApplicationID\nimport com.twitter.storehaus_internal.util.DatasetName\nimport com.twitter.storehaus_internal.util.HDFSPath\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject OfflineCandidateStoreModule extends TwitterModule {\n  type UserId = Long\n  implicit val tweetCandidatesInjection: Injection[CandidateTweetsList, Array[Byte]] =\n    CompactScalaCodec(CandidateTweetsList)\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.OfflineTweet2020CandidateStore)\n  def offlineTweet2020CandidateMhStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[UserId, CandidateTweetsList] = {\n    buildOfflineCandidateStore(\n      serviceIdentifier,\n      datasetName = \"offline_tweet_recommendations_from_interestedin_2020\"\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.OfflineTweet2020Hl0El15CandidateStore)\n  def offlineTweet2020Hl0El15CandidateMhStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[UserId, CandidateTweetsList] = {\n    buildOfflineCandidateStore(\n      serviceIdentifier,\n      datasetName = \"offline_tweet_recommendations_from_interestedin_2020_hl_0_el_15\"\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.OfflineTweet2020Hl2El15CandidateStore)\n  def offlineTweet2020Hl2El15CandidateMhStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[UserId, CandidateTweetsList] = {\n    buildOfflineCandidateStore(\n      serviceIdentifier,\n      datasetName = \"offline_tweet_recommendations_from_interestedin_2020_hl_2_el_15\"\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.OfflineTweet2020Hl2El50CandidateStore)\n  def offlineTweet2020Hl2El50CandidateMhStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[UserId, CandidateTweetsList] = {\n    buildOfflineCandidateStore(\n      serviceIdentifier,\n      datasetName = \"offline_tweet_recommendations_from_interestedin_2020_hl_2_el_50\"\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.OfflineTweet2020Hl8El50CandidateStore)\n  def offlineTweet2020Hl8El50CandidateMhStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[UserId, CandidateTweetsList] = {\n    buildOfflineCandidateStore(\n      serviceIdentifier,\n      datasetName = \"offline_tweet_recommendations_from_interestedin_2020_hl_8_el_50\"\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.OfflineTweetMTSCandidateStore)\n  def offlineTweetMTSCandidateMhStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[UserId, CandidateTweetsList] = {\n    buildOfflineCandidateStore(\n      serviceIdentifier,\n      datasetName = \"offline_tweet_recommendations_from_mts_consumer_embeddings\"\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.OfflineFavDecayedSumCandidateStore)\n  def offlineFavDecayedSumCandidateStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[UserId, CandidateTweetsList] = {\n    buildOfflineCandidateStore(\n      serviceIdentifier,\n      datasetName = \"offline_tweet_recommendations_from_decayed_sum\"\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.OfflineFtrAt5Pop1000RankDecay11CandidateStore)\n  def offlineFtrAt5Pop1000RankDecay11CandidateStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[UserId, CandidateTweetsList] = {\n    buildOfflineCandidateStore(\n      serviceIdentifier,\n      datasetName = \"offline_tweet_recommendations_from_ftrat5_pop1000_rank_decay_1_1\"\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.OfflineFtrAt5Pop10000RankDecay11CandidateStore)\n  def offlineFtrAt5Pop10000RankDecay11CandidateStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[UserId, CandidateTweetsList] = {\n    buildOfflineCandidateStore(\n      serviceIdentifier,\n      datasetName = \"offline_tweet_recommendations_from_ftrat5_pop10000_rank_decay_1_1\"\n    )\n  }\n\n  private def buildOfflineCandidateStore(\n    serviceIdentifier: ServiceIdentifier,\n    datasetName: String\n  ): ReadableStore[UserId, CandidateTweetsList] = {\n    ManhattanRO\n      .getReadableStoreWithMtls[Long, CandidateTweetsList](\n        ManhattanROConfig(\n          HDFSPath(\"\"), // not needed\n          ApplicationID(\"multi_type_simclusters\"),\n          DatasetName(datasetName),\n          Apollo\n        ),\n        ManhattanKVClientMtlsParams(serviceIdentifier)\n      )\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/RealGraphOonStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.app.Flag\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.wtf.candidate.thriftscala.CandidateSeq\n\nobject RealGraphOonStoreModule extends TwitterModule {\n\n  private val userRealGraphOonColumnPath: Flag[String] = flag[String](\n    name = \"crMixer.userRealGraphOonColumnPath\",\n    default = \"recommendations/twistly/userRealgraphOon\",\n    help = \"Strato column path for user real graph OON Store\"\n  )\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.RealGraphOonStore)\n  def providesRealGraphOonStore(\n    stratoClient: StratoClient,\n    statsReceiver: StatsReceiver\n  ): ReadableStore[UserId, CandidateSeq] = {\n    val realGraphOonStratoFetchableStore = StratoFetchableStore\n      .withUnitView[UserId, CandidateSeq](stratoClient, userRealGraphOonColumnPath())\n\n    ObservedReadableStore(\n      realGraphOonStratoFetchableStore\n    )(statsReceiver.scope(\"user_real_graph_oon_store\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/RealGraphStoreMhModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.google.inject.name.Named\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.Apollo\nimport com.twitter.storehaus_internal.manhattan.ManhattanRO\nimport com.twitter.storehaus_internal.manhattan.ManhattanROConfig\nimport com.twitter.storehaus_internal.util.ApplicationID\nimport com.twitter.storehaus_internal.util.DatasetName\nimport com.twitter.storehaus_internal.util.HDFSPath\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.cr_mixer.param.decider.DeciderKey\nimport com.twitter.hermit.store.common.DeciderableReadableStore\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.wtf.candidate.thriftscala.CandidateSeq\n\nobject RealGraphStoreMhModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.RealGraphInStore)\n  def providesRealGraphStoreMh(\n    decider: CrMixerDecider,\n    statsReceiver: StatsReceiver,\n    manhattanKVClientMtlsParams: ManhattanKVClientMtlsParams,\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n  ): ReadableStore[UserId, CandidateSeq] = {\n\n    implicit val valueCodec = new BinaryScalaCodec(CandidateSeq)\n    val underlyingStore = ManhattanRO\n      .getReadableStoreWithMtls[UserId, CandidateSeq](\n        ManhattanROConfig(\n          HDFSPath(\"\"),\n          ApplicationID(\"cr_mixer_apollo\"),\n          DatasetName(\"real_graph_scores_apollo\"),\n          Apollo),\n        manhattanKVClientMtlsParams\n      )\n\n    val memCachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = underlyingStore,\n        cacheClient = crMixerUnifiedCacheClient,\n        ttl = 24.hours,\n      )(\n        valueInjection = valueCodec,\n        statsReceiver = statsReceiver.scope(\"memCachedUserRealGraphMh\"),\n        keyToString = { k: UserId => s\"uRGraph/$k\" }\n      )\n\n    DeciderableReadableStore(\n      memCachedStore,\n      decider.deciderGateBuilder.idGate(DeciderKey.enableRealGraphMhStoreDeciderKey),\n      statsReceiver.scope(\"RealGraphMh\")\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/RepresentationManagerModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.representation_manager.thriftscala.SimClustersEmbeddingView\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport javax.inject.Named\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.simclusters_v2.thriftscala.{SimClustersEmbedding => ThriftSimClustersEmbedding}\n\nobject RepresentationManagerModule extends TwitterModule {\n  private val ColPathPrefix = \"recommendations/representation_manager/\"\n  private val SimclustersTweetColPath = ColPathPrefix + \"simClustersEmbedding.Tweet\"\n  private val SimclustersUserColPath = ColPathPrefix + \"simClustersEmbedding.User\"\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.RmsTweetLogFavLongestL2EmbeddingStore)\n  def providesRepresentationManagerTweetStore(\n    statsReceiver: StatsReceiver,\n    stratoClient: StratoClient,\n  ): ReadableStore[TweetId, SimClustersEmbedding] = {\n    ObservedReadableStore(\n      StratoFetchableStore\n        .withView[Long, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n          stratoClient,\n          SimclustersTweetColPath,\n          SimClustersEmbeddingView(\n            EmbeddingType.LogFavLongestL2EmbeddingTweet,\n            ModelVersion.Model20m145k2020))\n        .mapValues(SimClustersEmbedding(_)))(\n      statsReceiver.scope(\"rms_tweet_log_fav_longest_l2_store\"))\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.RmsUserFavBasedProducerEmbeddingStore)\n  def providesRepresentationManagerUserFavBasedProducerEmbeddingStore(\n    statsReceiver: StatsReceiver,\n    stratoClient: StratoClient,\n  ): ReadableStore[UserId, SimClustersEmbedding] = {\n    ObservedReadableStore(\n      StratoFetchableStore\n        .withView[Long, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n          stratoClient,\n          SimclustersUserColPath,\n          SimClustersEmbeddingView(\n            EmbeddingType.FavBasedProducer,\n            ModelVersion.Model20m145k2020\n          )\n        )\n        .mapValues(SimClustersEmbedding(_)))(\n      statsReceiver.scope(\"rms_user_fav_based_producer_store\"))\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.RmsUserLogFavInterestedInEmbeddingStore)\n  def providesRepresentationManagerUserLogFavConsumerEmbeddingStore(\n    statsReceiver: StatsReceiver,\n    stratoClient: StratoClient,\n  ): ReadableStore[UserId, SimClustersEmbedding] = {\n    ObservedReadableStore(\n      StratoFetchableStore\n        .withView[Long, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n          stratoClient,\n          SimclustersUserColPath,\n          SimClustersEmbeddingView(\n            EmbeddingType.LogFavBasedUserInterestedIn,\n            ModelVersion.Model20m145k2020\n          )\n        )\n        .mapValues(SimClustersEmbedding(_)))(\n      statsReceiver.scope(\"rms_user_log_fav_interestedin_store\"))\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.RmsUserFollowInterestedInEmbeddingStore)\n  def providesRepresentationManagerUserFollowInterestedInEmbeddingStore(\n    statsReceiver: StatsReceiver,\n    stratoClient: StratoClient,\n  ): ReadableStore[UserId, SimClustersEmbedding] = {\n    ObservedReadableStore(\n      StratoFetchableStore\n        .withView[Long, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n          stratoClient,\n          SimclustersUserColPath,\n          SimClustersEmbeddingView(\n            EmbeddingType.FollowBasedUserInterestedIn,\n            ModelVersion.Model20m145k2020\n          )\n        )\n        .mapValues(SimClustersEmbedding(_)))(\n      statsReceiver.scope(\"rms_user_follow_interestedin_store\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/RepresentationScorerModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.simclusters_v2.thriftscala.ScoringAlgorithm\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport javax.inject.Named\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.representationscorer.thriftscala.ListScoreId\n\nobject RepresentationScorerModule extends TwitterModule {\n\n  private val rsxColumnPath = \"recommendations/representation_scorer/listScore\"\n\n  private final val SimClusterModelVersion = ModelVersion.Model20m145k2020\n  private final val TweetEmbeddingType = EmbeddingType.LogFavBasedTweet\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.RsxStore)\n  def providesRepresentationScorerStore(\n    statsReceiver: StatsReceiver,\n    stratoClient: StratoClient,\n  ): ReadableStore[(UserId, TweetId), Double] = {\n    ObservedReadableStore(\n      StratoFetchableStore\n        .withUnitView[ListScoreId, Double](stratoClient, rsxColumnPath).composeKeyMapping[(\n          UserId,\n          TweetId\n        )] { key =>\n          representationScorerStoreKeyMapping(key._1, key._2)\n        }\n    )(statsReceiver.scope(\"rsx_store\"))\n  }\n\n  private def representationScorerStoreKeyMapping(t1: TweetId, t2: TweetId): ListScoreId = {\n    ListScoreId(\n      algorithm = ScoringAlgorithm.PairEmbeddingLogCosineSimilarity,\n      modelVersion = SimClusterModelVersion,\n      targetEmbeddingType = TweetEmbeddingType,\n      targetId = InternalId.TweetId(t1),\n      candidateEmbeddingType = TweetEmbeddingType,\n      candidateIds = Seq(InternalId.TweetId(t2))\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/SampleSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.LookupSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Singleton\n\n/**\n * In this example we build a [[StandardSimilarityEngine]] to wrap a dummy store\n */\nobject SimpleSimilarityEngineModule extends TwitterModule {\n  @Provides\n  @Singleton\n  def providesSimpleSimilarityEngine(\n    timeoutConfig: TimeoutConfig,\n    globalStats: StatsReceiver\n  ): StandardSimilarityEngine[UserId, (TweetId, Double)] = {\n    // Inject your readableStore implementation here\n    val dummyStore = ReadableStore.fromMap(\n      Map(\n        1L -> Seq((100L, 1.0), (101L, 1.0)),\n        2L -> Seq((200L, 2.0), (201L, 2.0)),\n        3L -> Seq((300L, 3.0), (301L, 3.0))\n      ))\n\n    new StandardSimilarityEngine[UserId, (TweetId, Double)](\n      implementingStore = dummyStore,\n      identifier = SimilarityEngineType.EnumUnknownSimilarityEngineType(9997),\n      globalStats = globalStats,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = None,\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n}\n\n/**\n * In this example we build a [[LookupSimilarityEngine]] to wrap a dummy store with 2 versions\n */\nobject LookupSimilarityEngineModule extends TwitterModule {\n  @Provides\n  @Singleton\n  def providesLookupSimilarityEngine(\n    timeoutConfig: TimeoutConfig,\n    globalStats: StatsReceiver\n  ): LookupSimilarityEngine[UserId, (TweetId, Double)] = {\n    // Inject your readableStore implementation here\n    val dummyStoreV1 = ReadableStore.fromMap(\n      Map(\n        1L -> Seq((100L, 1.0), (101L, 1.0)),\n        2L -> Seq((200L, 2.0), (201L, 2.0)),\n      ))\n\n    val dummyStoreV2 = ReadableStore.fromMap(\n      Map(\n        1L -> Seq((100L, 1.0), (101L, 1.0)),\n        2L -> Seq((200L, 2.0), (201L, 2.0)),\n      ))\n\n    new LookupSimilarityEngine[UserId, (TweetId, Double)](\n      versionedStoreMap = Map(\n        \"V1\" -> dummyStoreV1,\n        \"V2\" -> dummyStoreV2\n      ),\n      identifier = SimilarityEngineType.EnumUnknownSimilarityEngineType(9998),\n      globalStats = globalStats,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = None,\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/SimClustersANNServiceNameToClientMapper.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclustersann.thriftscala.SimClustersANNService\nimport javax.inject.Named\n\nobject SimClustersANNServiceNameToClientMapper extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def providesSimClustersANNServiceNameToClientMapping(\n    @Named(ModuleNames.ProdSimClustersANNServiceClientName) simClustersANNServiceProd: SimClustersANNService.MethodPerEndpoint,\n    @Named(ModuleNames.ExperimentalSimClustersANNServiceClientName) simClustersANNServiceExperimental: SimClustersANNService.MethodPerEndpoint,\n    @Named(ModuleNames.SimClustersANNServiceClientName1) simClustersANNService1: SimClustersANNService.MethodPerEndpoint,\n    @Named(ModuleNames.SimClustersANNServiceClientName2) simClustersANNService2: SimClustersANNService.MethodPerEndpoint,\n    @Named(ModuleNames.SimClustersANNServiceClientName3) simClustersANNService3: SimClustersANNService.MethodPerEndpoint,\n    @Named(ModuleNames.SimClustersANNServiceClientName5) simClustersANNService5: SimClustersANNService.MethodPerEndpoint,\n    @Named(ModuleNames.SimClustersANNServiceClientName4) simClustersANNService4: SimClustersANNService.MethodPerEndpoint\n  ): Map[String, SimClustersANNService.MethodPerEndpoint] = {\n    Map[String, SimClustersANNService.MethodPerEndpoint](\n      \"simclusters-ann\" -> simClustersANNServiceProd,\n      \"simclusters-ann-experimental\" -> simClustersANNServiceExperimental,\n      \"simclusters-ann-1\" -> simClustersANNService1,\n      \"simclusters-ann-2\" -> simClustersANNService2,\n      \"simclusters-ann-3\" -> simClustersANNService3,\n      \"simclusters-ann-5\" -> simClustersANNService5,\n      \"simclusters-ann-4\" -> simClustersANNService4\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/SkitStratoStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.google.inject.name.Named\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.keyHasher\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.hermit.store.common.ObservedCachedReadableStore\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.relevance_platform.common.injection.LZ4Injection\nimport com.twitter.relevance_platform.common.injection.SeqObjectInjection\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.Client\nimport com.twitter.topic_recos.thriftscala.TopicTopTweets\nimport com.twitter.topic_recos.thriftscala.TopicTweet\nimport com.twitter.topic_recos.thriftscala.TopicTweetPartitionFlatKey\n\n/**\n * Strato store that wraps the topic top tweets pipeline indexed from a Summingbird job\n */\nobject SkitStratoStoreModule extends TwitterModule {\n\n  val column = \"recommendations/topic_recos/topicTopTweets\"\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.SkitStratoStoreName)\n  def providesSkitStratoStore(\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    stratoClient: Client,\n    statsReceiver: StatsReceiver\n  ): ReadableStore[TopicTweetPartitionFlatKey, Seq[TopicTweet]] = {\n    val skitStore = ObservedReadableStore(\n      StratoFetchableStore\n        .withUnitView[TopicTweetPartitionFlatKey, TopicTopTweets](stratoClient, column))(\n      statsReceiver.scope(ModuleNames.SkitStratoStoreName)).mapValues { topicTopTweets =>\n      topicTopTweets.topTweets\n    }\n\n    val memCachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = skitStore,\n        cacheClient = crMixerUnifiedCacheClient,\n        ttl = 10.minutes\n      )(\n        valueInjection = LZ4Injection.compose(SeqObjectInjection[TopicTweet]()),\n        statsReceiver = statsReceiver.scope(\"memcached_skit_store\"),\n        keyToString = { k => s\"skit:${keyHasher.hashKey(k.toString.getBytes)}\" }\n      )\n\n    ObservedCachedReadableStore.from[TopicTweetPartitionFlatKey, Seq[TopicTweet]](\n      memCachedStore,\n      ttl = 5.minutes,\n      maxKeys = 100000, // ~150MB max\n      cacheName = \"skit_in_memory_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"skit_in_memory_cache\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/StrongTiePredictionStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.app.Flag\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.hermit.stp.thriftscala.STPResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport javax.inject.Named\n\nobject StrongTiePredictionStoreModule extends TwitterModule {\n\n  private val strongTiePredictionColumnPath: Flag[String] = flag[String](\n    name = \"crMixer.strongTiePredictionColumnPath\",\n    default = \"onboarding/userrecs/strong_tie_prediction_big\",\n    help = \"Strato column path for StrongTiePredictionStore\"\n  )\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.StpStore)\n  def providesStrongTiePredictionStore(\n    statsReceiver: StatsReceiver,\n    stratoClient: StratoClient,\n  ): ReadableStore[UserId, STPResult] = {\n    val strongTiePredictionStratoFetchableStore = StratoFetchableStore\n      .withUnitView[UserId, STPResult](stratoClient, strongTiePredictionColumnPath())\n\n    ObservedReadableStore(\n      strongTiePredictionStratoFetchableStore\n    )(statsReceiver.scope(\"strong_tie_prediction_big_store\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/TripCandidateStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripTweet\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripTweets\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripDomain\nimport javax.inject.Named\n\nobject TripCandidateStoreModule extends TwitterModule {\n  private val stratoColumn = \"trends/trip/tripTweetsDataflowProd\"\n\n  @Provides\n  @Named(ModuleNames.TripCandidateStore)\n  def providesSimClustersTripCandidateStore(\n    statsReceiver: StatsReceiver,\n    stratoClient: StratoClient\n  ): ReadableStore[TripDomain, Seq[TripTweet]] = {\n    val tripCandidateStratoFetchableStore =\n      StratoFetchableStore\n        .withUnitView[TripDomain, TripTweets](stratoClient, stratoColumn)\n        .mapValues(_.tweets)\n\n    ObservedReadableStore(\n      tripCandidateStratoFetchableStore\n    )(statsReceiver.scope(\"simclusters_trip_candidate_store\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/TweetInfoStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Module\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.contentrecommender.thriftscala.TweetInfo\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.frigate.common.store.health.TweetHealthModelStore\nimport com.twitter.frigate.common.store.health.TweetHealthModelStore.TweetHealthModelStoreConfig\nimport com.twitter.frigate.common.store.health.UserHealthModelStore\nimport com.twitter.frigate.thriftscala.TweetHealthScores\nimport com.twitter.frigate.thriftscala.UserAgathaScores\nimport com.twitter.hermit.store.common.DeciderableReadableStore\nimport com.twitter.hermit.store.common.ObservedCachedReadableStore\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.contentrecommender.store.TweetInfoStore\nimport com.twitter.contentrecommender.store.TweetyPieFieldsStore\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderKey\nimport com.twitter.frigate.data_pipeline.scalding.thriftscala.BlueVerifiedAnnotationsV2\nimport com.twitter.recos.user_tweet_graph_plus.thriftscala.UserTweetGraphPlus\nimport com.twitter.recos.user_tweet_graph_plus.thriftscala.TweetEngagementScores\nimport com.twitter.relevance_platform.common.health_store.UserMediaRepresentationHealthStore\nimport com.twitter.relevance_platform.common.health_store.MagicRecsRealTimeAggregatesStore\nimport com.twitter.relevance_platform.thriftscala.MagicRecsRealTimeAggregatesScores\nimport com.twitter.relevance_platform.thriftscala.UserMediaRepresentationScores\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.tweetypie.thriftscala.TweetService\nimport com.twitter.util.Future\nimport com.twitter.util.JavaTimer\nimport com.twitter.util.Timer\n\nimport javax.inject.Named\n\nobject TweetInfoStoreModule extends TwitterModule {\n  implicit val timer: Timer = new JavaTimer(true)\n  override def modules: Seq[Module] = Seq(UnifiedCacheClient)\n\n  @Provides\n  @Singleton\n  def providesTweetInfoStore(\n    statsReceiver: StatsReceiver,\n    serviceIdentifier: ServiceIdentifier,\n    stratoClient: StratoClient,\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    manhattanKVClientMtlsParams: ManhattanKVClientMtlsParams,\n    tweetyPieService: TweetService.MethodPerEndpoint,\n    userTweetGraphPlusService: UserTweetGraphPlus.MethodPerEndpoint,\n    @Named(ModuleNames.BlueVerifiedAnnotationStore) blueVerifiedAnnotationStore: ReadableStore[\n      String,\n      BlueVerifiedAnnotationsV2\n    ],\n    decider: CrMixerDecider\n  ): ReadableStore[TweetId, TweetInfo] = {\n\n    val tweetEngagementScoreStore: ReadableStore[TweetId, TweetEngagementScores] = {\n      val underlyingStore =\n        ObservedReadableStore(new ReadableStore[TweetId, TweetEngagementScores] {\n          override def get(\n            k: TweetId\n          ): Future[Option[TweetEngagementScores]] = {\n            userTweetGraphPlusService.tweetEngagementScore(k).map {\n              Some(_)\n            }\n          }\n        })(statsReceiver.scope(\"UserTweetGraphTweetEngagementScoreStore\"))\n\n      DeciderableReadableStore(\n        underlyingStore,\n        decider.deciderGateBuilder.idGate(\n          DeciderKey.enableUtgRealTimeTweetEngagementScoreDeciderKey),\n        statsReceiver.scope(\"UserTweetGraphTweetEngagementScoreStore\")\n      )\n\n    }\n\n    val tweetHealthModelStore: ReadableStore[TweetId, TweetHealthScores] = {\n      val underlyingStore = TweetHealthModelStore.buildReadableStore(\n        stratoClient,\n        Some(\n          TweetHealthModelStoreConfig(\n            enablePBlock = true,\n            enableToxicity = true,\n            enablePSpammy = true,\n            enablePReported = true,\n            enableSpammyTweetContent = true,\n            enablePNegMultimodal = true,\n          ))\n      )(statsReceiver.scope(\"UnderlyingTweetHealthModelStore\"))\n\n      DeciderableReadableStore(\n        ObservedMemcachedReadableStore.fromCacheClient(\n          backingStore = underlyingStore,\n          cacheClient = crMixerUnifiedCacheClient,\n          ttl = 2.hours\n        )(\n          valueInjection = BinaryScalaCodec(TweetHealthScores),\n          statsReceiver = statsReceiver.scope(\"memCachedTweetHealthModelStore\"),\n          keyToString = { k: TweetId => s\"tHMS/$k\" }\n        ),\n        decider.deciderGateBuilder.idGate(DeciderKey.enableHealthSignalsScoreDeciderKey),\n        statsReceiver.scope(\"TweetHealthModelStore\")\n      ) // use s\"tHMS/$k\" instead of s\"tweetHealthModelStore/$k\" to differentiate from CR cache\n    }\n\n    val userHealthModelStore: ReadableStore[UserId, UserAgathaScores] = {\n      val underlyingStore = UserHealthModelStore.buildReadableStore(stratoClient)(\n        statsReceiver.scope(\"UnderlyingUserHealthModelStore\"))\n      DeciderableReadableStore(\n        ObservedMemcachedReadableStore.fromCacheClient(\n          backingStore = underlyingStore,\n          cacheClient = crMixerUnifiedCacheClient,\n          ttl = 18.hours\n        )(\n          valueInjection = BinaryScalaCodec(UserAgathaScores),\n          statsReceiver = statsReceiver.scope(\"memCachedUserHealthModelStore\"),\n          keyToString = { k: UserId => s\"uHMS/$k\" }\n        ),\n        decider.deciderGateBuilder.idGate(DeciderKey.enableUserAgathaScoreDeciderKey),\n        statsReceiver.scope(\"UserHealthModelStore\")\n      )\n    }\n\n    val userMediaRepresentationHealthStore: ReadableStore[UserId, UserMediaRepresentationScores] = {\n      val underlyingStore =\n        UserMediaRepresentationHealthStore.buildReadableStore(\n          manhattanKVClientMtlsParams,\n          statsReceiver.scope(\"UnderlyingUserMediaRepresentationHealthStore\")\n        )\n      DeciderableReadableStore(\n        ObservedMemcachedReadableStore.fromCacheClient(\n          backingStore = underlyingStore,\n          cacheClient = crMixerUnifiedCacheClient,\n          ttl = 12.hours\n        )(\n          valueInjection = BinaryScalaCodec(UserMediaRepresentationScores),\n          statsReceiver = statsReceiver.scope(\"memCacheUserMediaRepresentationHealthStore\"),\n          keyToString = { k: UserId => s\"uMRHS/$k\" }\n        ),\n        decider.deciderGateBuilder.idGate(DeciderKey.enableUserMediaRepresentationStoreDeciderKey),\n        statsReceiver.scope(\"UserMediaRepresentationHealthStore\")\n      )\n    }\n\n    val magicRecsRealTimeAggregatesStore: ReadableStore[\n      TweetId,\n      MagicRecsRealTimeAggregatesScores\n    ] = {\n      val underlyingStore =\n        MagicRecsRealTimeAggregatesStore.buildReadableStore(\n          serviceIdentifier,\n          statsReceiver.scope(\"UnderlyingMagicRecsRealTimeAggregatesScores\")\n        )\n      DeciderableReadableStore(\n        underlyingStore,\n        decider.deciderGateBuilder.idGate(DeciderKey.enableMagicRecsRealTimeAggregatesStore),\n        statsReceiver.scope(\"MagicRecsRealTimeAggregatesStore\")\n      )\n    }\n\n    val tweetInfoStore: ReadableStore[TweetId, TweetInfo] = {\n      val underlyingStore = TweetInfoStore(\n        TweetyPieFieldsStore.getStoreFromTweetyPie(tweetyPieService),\n        userMediaRepresentationHealthStore,\n        magicRecsRealTimeAggregatesStore,\n        tweetEngagementScoreStore,\n        blueVerifiedAnnotationStore\n      )(statsReceiver.scope(\"tweetInfoStore\"))\n\n      val memcachedStore = ObservedMemcachedReadableStore.fromCacheClient(\n        backingStore = underlyingStore,\n        cacheClient = crMixerUnifiedCacheClient,\n        ttl = 15.minutes,\n        // Hydrating tweetInfo is now a required step for all candidates,\n        // hence we needed to tune these thresholds.\n        asyncUpdate = serviceIdentifier.environment == \"prod\"\n      )(\n        valueInjection = BinaryScalaCodec(TweetInfo),\n        statsReceiver = statsReceiver.scope(\"memCachedTweetInfoStore\"),\n        keyToString = { k: TweetId => s\"tIS/$k\" }\n      )\n\n      ObservedCachedReadableStore.from(\n        memcachedStore,\n        ttl = 15.minutes,\n        maxKeys = 8388607, // Check TweetInfo definition. size~92b. Around 736 MB\n        windowSize = 10000L,\n        cacheName = \"tweet_info_cache\",\n        maxMultiGetSize = 20\n      )(statsReceiver.scope(\"inMemoryCachedTweetInfoStore\"))\n    }\n    tweetInfoStore\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/TweetRecentEngagedUserStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.app.Flag\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.twistly.thriftscala.TweetRecentEngagedUsers\n\nobject TweetRecentEngagedUserStoreModule extends TwitterModule {\n\n  private val tweetRecentEngagedUsersStoreDefaultVersion =\n    0 // DefaultVersion for tweetEngagedUsersStore, whose key = (tweetId, DefaultVersion)\n  private val tweetRecentEngagedUsersColumnPath: Flag[String] = flag[String](\n    name = \"crMixer.tweetRecentEngagedUsersColumnPath\",\n    default = \"recommendations/twistly/tweetRecentEngagedUsers\",\n    help = \"Strato column path for TweetRecentEngagedUsersStore\"\n  )\n  private type Version = Long\n\n  @Provides\n  @Singleton\n  def providesTweetRecentEngagedUserStore(\n    statsReceiver: StatsReceiver,\n    stratoClient: StratoClient,\n  ): ReadableStore[TweetId, TweetRecentEngagedUsers] = {\n    val tweetRecentEngagedUsersStratoFetchableStore = StratoFetchableStore\n      .withUnitView[(TweetId, Version), TweetRecentEngagedUsers](\n        stratoClient,\n        tweetRecentEngagedUsersColumnPath()).composeKeyMapping[TweetId](tweetId =>\n        (tweetId, tweetRecentEngagedUsersStoreDefaultVersion))\n\n    ObservedReadableStore(\n      tweetRecentEngagedUsersStratoFetchableStore\n    )(statsReceiver.scope(\"tweet_recent_engaged_users_store\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/TweetRecommendationResultsStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.thriftscala.CrMixerTweetResponse\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.hermit.store.common.ReadableWritableStore\nimport com.twitter.hermit.store.common.ObservedReadableWritableMemcacheStore\nimport com.twitter.simclusters_v2.common.UserId\nimport javax.inject.Named\n\nobject TweetRecommendationResultsStoreModule extends TwitterModule {\n  @Provides\n  @Singleton\n  def providesTweetRecommendationResultsStore(\n    @Named(ModuleNames.TweetRecommendationResultsCache) tweetRecommendationResultsCacheClient: MemcachedClient,\n    statsReceiver: StatsReceiver\n  ): ReadableWritableStore[UserId, CrMixerTweetResponse] = {\n    ObservedReadableWritableMemcacheStore.fromCacheClient(\n      cacheClient = tweetRecommendationResultsCacheClient,\n      ttl = 24.hours)(\n      valueInjection = BinaryScalaCodec(CrMixerTweetResponse),\n      statsReceiver = statsReceiver.scope(\"TweetRecommendationResultsMemcacheStore\"),\n      keyToString = { k: UserId => k.toString }\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/TwhinCollabFilterStratoStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.inject.TwitterModule\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.cr_mixer.similarity_engine.TwhinCollabFilterSimilarityEngine.TwhinCollabFilterView\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\n\nobject TwhinCollabFilterStratoStoreModule extends TwitterModule {\n\n  val stratoColumnPath: String = \"cuad/twhin/getCollabFilterTweetCandidatesProd.User\"\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TwhinCollabFilterStratoStoreForFollow)\n  def providesTwhinCollabFilterStratoStoreForFollow(\n    stratoClient: StratoClient\n  ): ReadableStore[Long, Seq[TweetId]] = {\n    StratoFetchableStore.withView[Long, TwhinCollabFilterView, Seq[TweetId]](\n      stratoClient,\n      column = stratoColumnPath,\n      view = TwhinCollabFilterView(\"follow_2022_03_10_c_500K\")\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TwhinCollabFilterStratoStoreForEngagement)\n  def providesTwhinCollabFilterStratoStoreForEngagement(\n    stratoClient: StratoClient\n  ): ReadableStore[Long, Seq[TweetId]] = {\n    StratoFetchableStore.withView[Long, TwhinCollabFilterView, Seq[TweetId]](\n      stratoClient,\n      column = stratoColumnPath,\n      view = TwhinCollabFilterView(\"engagement_2022_04_10_c_500K\"))\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TwhinMultiClusterStratoStoreForFollow)\n  def providesTwhinMultiClusterStratoStoreForFollow(\n    stratoClient: StratoClient\n  ): ReadableStore[Long, Seq[TweetId]] = {\n    StratoFetchableStore.withView[Long, TwhinCollabFilterView, Seq[TweetId]](\n      stratoClient,\n      column = stratoColumnPath,\n      view = TwhinCollabFilterView(\"multiclusterFollow20220921\")\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TwhinMultiClusterStratoStoreForEngagement)\n  def providesTwhinMultiClusterStratoStoreForEngagement(\n    stratoClient: StratoClient\n  ): ReadableStore[Long, Seq[TweetId]] = {\n    StratoFetchableStore.withView[Long, TwhinCollabFilterView, Seq[TweetId]](\n      stratoClient,\n      column = stratoColumnPath,\n      view = TwhinCollabFilterView(\"multiclusterEng20220921\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/TwiceClustersMembersStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.app.Flag\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.simclusters_v2.thriftscala.OrderedClustersAndMembers\nimport javax.inject.Named\n\nobject TwiceClustersMembersStoreModule extends TwitterModule {\n\n  private val twiceClustersMembersColumnPath: Flag[String] = flag[String](\n    name = \"crMixer.twiceClustersMembersColumnPath\",\n    default =\n      \"recommendations/simclusters_v2/embeddings/TwiceClustersMembersLargestDimApeSimilarity\",\n    help = \"Strato column path for TweetRecentEngagedUsersStore\"\n  )\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TwiceClustersMembersStore)\n  def providesTweetRecentEngagedUserStore(\n    statsReceiver: StatsReceiver,\n    stratoClient: StratoClient,\n  ): ReadableStore[UserId, OrderedClustersAndMembers] = {\n    val twiceClustersMembersStratoFetchableStore = StratoFetchableStore\n      .withUnitView[UserId, OrderedClustersAndMembers](\n        stratoClient,\n        twiceClustersMembersColumnPath())\n\n    ObservedReadableStore(\n      twiceClustersMembersStratoFetchableStore\n    )(statsReceiver.scope(\"twice_clusters_members_largestDimApe_similarity_store\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/UnifiedCacheClient.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.app.Flag\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus_internal.memcache.MemcacheStore\nimport com.twitter.storehaus_internal.util.ClientName\nimport com.twitter.storehaus_internal.util.ZkEndPoint\nimport javax.inject.Named\n\nobject UnifiedCacheClient extends TwitterModule {\n\n  private val TIME_OUT = 20.milliseconds\n\n  val crMixerUnifiedCacheDest: Flag[String] = flag[String](\n    name = \"crMixer.unifiedCacheDest\",\n    default = \"/s/cache/content_recommender_unified_v2\",\n    help = \"Wily path to Content Recommender unified cache\"\n  )\n\n  val tweetRecommendationResultsCacheDest: Flag[String] = flag[String](\n    name = \"tweetRecommendationResults.CacheDest\",\n    default = \"/s/cache/tweet_recommendation_results\",\n    help = \"Wily path to CrMixer getTweetRecommendations() results cache\"\n  )\n\n  val earlybirdTweetsCacheDest: Flag[String] = flag[String](\n    name = \"earlybirdTweets.CacheDest\",\n    default = \"/s/cache/crmixer_earlybird_tweets\",\n    help = \"Wily path to CrMixer Earlybird Recency Based Similarity Engine result cache\"\n  )\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.UnifiedCache)\n  def provideUnifiedCacheClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver,\n  ): Client =\n    MemcacheStore.memcachedClient(\n      name = ClientName(\"memcache-content-recommender-unified\"),\n      dest = ZkEndPoint(crMixerUnifiedCacheDest()),\n      statsReceiver = statsReceiver.scope(\"cache_client\"),\n      serviceIdentifier = serviceIdentifier,\n      timeout = TIME_OUT\n    )\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TweetRecommendationResultsCache)\n  def providesTweetRecommendationResultsCache(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver,\n  ): Client =\n    MemcacheStore.memcachedClient(\n      name = ClientName(\"memcache-tweet-recommendation-results\"),\n      dest = ZkEndPoint(tweetRecommendationResultsCacheDest()),\n      statsReceiver = statsReceiver.scope(\"cache_client\"),\n      serviceIdentifier = serviceIdentifier,\n      timeout = TIME_OUT\n    )\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.EarlybirdTweetsCache)\n  def providesEarlybirdTweetsCache(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver,\n  ): Client =\n    MemcacheStore.memcachedClient(\n      name = ClientName(\"memcache-crmixer-earlybird-tweets\"),\n      dest = ZkEndPoint(earlybirdTweetsCacheDest()),\n      statsReceiver = statsReceiver.scope(\"cache_client\"),\n      serviceIdentifier = serviceIdentifier,\n      timeout = TIME_OUT\n    )\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/UserSignalServiceColumnModule.scala",
    "content": "package com.twitter.cr_mixer.module\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.usersignalservice.thriftscala.BatchSignalRequest\nimport com.twitter.usersignalservice.thriftscala.BatchSignalResponse\nimport javax.inject.Named\n\nobject UserSignalServiceColumnModule extends TwitterModule {\n  private val UssColumnPath = \"recommendations/user-signal-service/signals\"\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.UssStratoColumn)\n  def providesUserSignalServiceStore(\n    statsReceiver: StatsReceiver,\n    stratoClient: StratoClient,\n  ): ReadableStore[BatchSignalRequest, BatchSignalResponse] = {\n    ObservedReadableStore(\n      StratoFetchableStore\n        .withUnitView[BatchSignalRequest, BatchSignalResponse](stratoClient, UssColumnPath))(\n      statsReceiver.scope(\"user_signal_service_store\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/UserSignalServiceStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.source_signal.UssStore\nimport com.twitter.cr_mixer.source_signal.UssStore.Query\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.usersignalservice.thriftscala.BatchSignalRequest\nimport com.twitter.usersignalservice.thriftscala.BatchSignalResponse\nimport com.twitter.usersignalservice.thriftscala.SignalType\nimport com.twitter.usersignalservice.thriftscala.{Signal => UssSignal}\nimport javax.inject.Named\n\nobject UserSignalServiceStoreModule extends TwitterModule {\n\n  private val UssColumnPath = \"recommendations/user-signal-service/signals\"\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.UssStore)\n  def providesUserSignalServiceStore(\n    statsReceiver: StatsReceiver,\n    stratoClient: StratoClient,\n  ): ReadableStore[Query, Seq[(SignalType, Seq[UssSignal])]] = {\n    ObservedReadableStore(\n      UssStore(\n        StratoFetchableStore\n          .withUnitView[BatchSignalRequest, BatchSignalResponse](stratoClient, UssColumnPath),\n        statsReceiver))(statsReceiver.scope(\"user_signal_service_store\"))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/UserStateStoreModule.scala",
    "content": "package com.twitter.cr_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.bijection.Bufferable\nimport com.twitter.bijection.Injection\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.ManhattanRO\nimport com.twitter.storehaus_internal.manhattan.ManhattanROConfig\nimport com.twitter.storehaus_internal.util.HDFSPath\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.core_workflows.user_model.thriftscala.CondensedUserState\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderKey\nimport com.twitter.hermit.store.common.DeciderableReadableStore\nimport com.twitter.storehaus_internal.manhattan.Apollo\nimport com.twitter.storehaus_internal.util.ApplicationID\nimport com.twitter.storehaus_internal.util.DatasetName\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.JavaTimer\nimport com.twitter.util.Time\nimport com.twitter.util.TimeoutException\nimport com.twitter.util.Timer\nimport javax.inject.Named\n\nobject UserStateStoreModule extends TwitterModule {\n  implicit val timer: Timer = new JavaTimer(true)\n  final val NewUserCreateDaysThreshold = 7\n  final val DefaultUnknownUserStateValue = 100\n\n  // Convert CondensedUserState to UserState Enum\n  // If CondensedUserState is None, back fill by checking whether the user is new user\n  class UserStateStore(\n    userStateStore: ReadableStore[UserId, CondensedUserState],\n    timeout: Duration,\n    statsReceiver: StatsReceiver)\n      extends ReadableStore[UserId, UserState] {\n    override def get(userId: UserId): Future[Option[UserState]] = {\n      userStateStore\n        .get(userId).map(_.flatMap(_.userState)).map {\n          case Some(userState) => Some(userState)\n          case None =>\n            val isNewUser = SnowflakeId.timeFromIdOpt(userId).exists { userCreateTime =>\n              Time.now - userCreateTime < Duration.fromDays(NewUserCreateDaysThreshold)\n            }\n            if (isNewUser) Some(UserState.New)\n            else Some(UserState.EnumUnknownUserState(DefaultUnknownUserStateValue))\n\n        }.raiseWithin(timeout)(timer).rescue {\n          case _: TimeoutException =>\n            statsReceiver.counter(\"TimeoutException\").incr()\n            Future.None\n        }\n    }\n  }\n\n  @Provides\n  @Singleton\n  def providesUserStateStore(\n    crMixerDecider: CrMixerDecider,\n    statsReceiver: StatsReceiver,\n    manhattanKVClientMtlsParams: ManhattanKVClientMtlsParams,\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    timeoutConfig: TimeoutConfig\n  ): ReadableStore[UserId, UserState] = {\n\n    val underlyingStore = new UserStateStore(\n      ManhattanRO\n        .getReadableStoreWithMtls[UserId, CondensedUserState](\n          ManhattanROConfig(\n            HDFSPath(\"\"),\n            ApplicationID(\"cr_mixer_apollo\"),\n            DatasetName(\"condensed_user_state\"),\n            Apollo),\n          manhattanKVClientMtlsParams\n        )(\n          implicitly[Injection[Long, Array[Byte]]],\n          BinaryScalaCodec(CondensedUserState)\n        ),\n      timeoutConfig.userStateStoreTimeout,\n      statsReceiver.scope(\"UserStateStore\")\n    ).mapValues(_.value) // Read the value of Enum so that we only caches the Int\n\n    val memCachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = underlyingStore,\n        cacheClient = crMixerUnifiedCacheClient,\n        ttl = 24.hours,\n      )(\n        valueInjection = Bufferable.injectionOf[Int], // Cache Value is Enum Value for UserState\n        statsReceiver = statsReceiver.scope(\"memCachedUserStateStore\"),\n        keyToString = { k: UserId => s\"uState/$k\" }\n      ).mapValues(value => UserState.getOrUnknown(value))\n\n    DeciderableReadableStore(\n      memCachedStore,\n      crMixerDecider.deciderGateBuilder.idGate(DeciderKey.enableUserStateStoreDeciderKey),\n      statsReceiver.scope(\"UserStateStore\")\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/core/ABDeciderModule.scala",
    "content": "package com.twitter.cr_mixer.module.core\n\nimport com.google.inject.Provides\nimport com.google.inject.name.Named\nimport com.twitter.abdecider.ABDeciderFactory\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logging.Logger\nimport javax.inject.Singleton\n\nobject ABDeciderModule extends TwitterModule {\n\n  flag(\n    name = \"abdecider.path\",\n    default = \"/usr/local/config/abdecider/abdecider.yml\",\n    help = \"path to the abdecider Yml file location\"\n  )\n\n  @Provides\n  @Singleton\n  def provideABDecider(\n    @Flag(\"abdecider.path\") abDeciderYmlPath: String,\n    @Named(ModuleNames.AbDeciderLogger) scribeLogger: Logger\n  ): LoggingABDecider = {\n    ABDeciderFactory(\n      abDeciderYmlPath = abDeciderYmlPath,\n      scribeLogger = Some(scribeLogger),\n      environment = Some(\"production\")\n    ).buildWithLogging()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/core/CrMixerFlagModule.scala",
    "content": "package com.twitter.cr_mixer.module.core\n\nimport com.twitter.inject.TwitterModule\n\nobject CrMixerFlagName {\n  val SERVICE_FLAG = \"cr_mixer.flag\"\n  val DarkTrafficFilterDeciderKey = \"thrift.dark.traffic.filter.decider_key\"\n}\n\nobject CrMixerFlagModule extends TwitterModule {\n  import CrMixerFlagName._\n\n  flag[Boolean](name = SERVICE_FLAG, default = false, help = \"This is a CR Mixer flag\")\n\n  flag[String](\n    name = DarkTrafficFilterDeciderKey,\n    default = \"dark_traffic_filter\",\n    help = \"Dark traffic filter decider key\"\n  )\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/core/CrMixerLoggingABDeciderModule.scala",
    "content": "package com.twitter.cr_mixer.module.core\n\nimport com.google.inject.Provides\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.cr_mixer.featureswitch.CrMixerLoggingABDecider\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport javax.inject.Singleton\n\nobject CrMixerLoggingABDeciderModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def provideABDecider(\n    loggingABDecider: LoggingABDecider,\n    statsReceiver: StatsReceiver\n  ): CrMixerLoggingABDecider = {\n    CrMixerLoggingABDecider(loggingABDecider, statsReceiver)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/core/FeatureContextBuilderModule.scala",
    "content": "package com.twitter.cr_mixer.module.core\n\nimport com.google.inject.Provides\nimport com.twitter.discovery.common.configapi.FeatureContextBuilder\nimport com.twitter.featureswitches.v2.FeatureSwitches\nimport com.twitter.inject.TwitterModule\nimport javax.inject.Singleton\n\nobject FeatureContextBuilderModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def providesFeatureContextBuilder(featureSwitches: FeatureSwitches): FeatureContextBuilder = {\n    FeatureContextBuilder(featureSwitches)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/core/FeatureSwitchesModule.scala",
    "content": "package com.twitter.cr_mixer.module.core\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.featureswitch.CrMixerLoggingABDecider\nimport com.twitter.featureswitches.v2.FeatureSwitches\nimport com.twitter.featureswitches.v2.builder.FeatureSwitchesBuilder\nimport com.twitter.featureswitches.v2.experimentation.NullBucketImpressor\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.util.Duration\nimport javax.inject.Singleton\n\nobject FeatureSwitchesModule extends TwitterModule {\n\n  flag(\n    name = \"featureswitches.path\",\n    default = \"/features/cr-mixer/main\",\n    help = \"path to the featureswitch configuration directory\"\n  )\n  flag(\n    \"use_config_repo_mirror.bool\",\n    false,\n    \"If true, read config from a different directory, to facilitate testing.\")\n\n  val DefaultFastRefresh: Boolean = false\n  val AddServiceDetailsFromAurora: Boolean = true\n  val ImpressExperiments: Boolean = true\n\n  @Provides\n  @Singleton\n  def providesFeatureSwitches(\n    @Flag(\"featureswitches.path\") featureSwitchDirectory: String,\n    @Flag(\"use_config_repo_mirror.bool\") useConfigRepoMirrorFlag: Boolean,\n    abDecider: CrMixerLoggingABDecider,\n    statsReceiver: StatsReceiver\n  ): FeatureSwitches = {\n    val configRepoAbsPath =\n      getConfigRepoAbsPath(useConfigRepoMirrorFlag)\n    val fastRefresh =\n      shouldFastRefresh(useConfigRepoMirrorFlag)\n\n    val featureSwitches = FeatureSwitchesBuilder()\n      .abDecider(abDecider)\n      .statsReceiver(statsReceiver.scope(\"featureswitches-v2\"))\n      .configRepoAbsPath(configRepoAbsPath)\n      .featuresDirectory(featureSwitchDirectory)\n      .limitToReferencedExperiments(shouldLimit = true)\n      .experimentImpressionStatsEnabled(true)\n\n    if (!ImpressExperiments) featureSwitches.experimentBucketImpressor(NullBucketImpressor)\n    if (AddServiceDetailsFromAurora) featureSwitches.serviceDetailsFromAurora()\n    if (fastRefresh) featureSwitches.refreshPeriod(Duration.fromSeconds(10))\n\n    featureSwitches.build()\n  }\n\n  private def getConfigRepoAbsPath(\n    useConfigRepoMirrorFlag: Boolean\n  ): String = {\n    if (useConfigRepoMirrorFlag)\n      \"config_repo_mirror/\"\n    else \"/usr/local/config\"\n  }\n\n  private def shouldFastRefresh(\n    useConfigRepoMirrorFlag: Boolean\n  ): Boolean = {\n    if (useConfigRepoMirrorFlag)\n      true\n    else DefaultFastRefresh\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/core/KafkaProducerModule.scala",
    "content": "package com.twitter.cr_mixer.module.core\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.thriftscala.GetTweetsRecommendationsScribe\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finatra.kafka.producers.FinagleKafkaProducerBuilder\nimport com.twitter.finatra.kafka.producers.KafkaProducerBase\nimport com.twitter.finatra.kafka.producers.NullKafkaProducer\nimport com.twitter.finatra.kafka.serde.ScalaSerdes\nimport com.twitter.inject.TwitterModule\nimport javax.inject.Singleton\nimport org.apache.kafka.clients.CommonClientConfigs\nimport org.apache.kafka.common.config.SaslConfigs\nimport org.apache.kafka.common.config.SslConfigs\nimport org.apache.kafka.common.record.CompressionType\nimport org.apache.kafka.common.security.auth.SecurityProtocol\nimport org.apache.kafka.common.serialization.Serdes\n\nobject KafkaProducerModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def provideTweetRecsLoggerFactory(\n    serviceIdentifier: ServiceIdentifier,\n  ): KafkaProducerBase[String, GetTweetsRecommendationsScribe] = {\n    KafkaProducerFactory.getKafkaProducer(serviceIdentifier.environment)\n  }\n}\n\nobject KafkaProducerFactory {\n  private val jaasConfig =\n    \"\"\"com.sun.security.auth.module.Krb5LoginModule\n      |required \n      |principal=\"cr-mixer@TWITTER.BIZ\" \n      |debug=true \n      |useKeyTab=true \n      |storeKey=true \n      |keyTab=\"/var/lib/tss/keys/fluffy/keytabs/client/cr-mixer.keytab\" \n      |doNotPrompt=true;\n    \"\"\".stripMargin.replaceAll(\"\\n\", \" \")\n\n  private val trustStoreLocation = \"/etc/tw_truststore/messaging/kafka/client.truststore.jks\"\n\n  def getKafkaProducer(\n    environment: String\n  ): KafkaProducerBase[String, GetTweetsRecommendationsScribe] = {\n    if (environment == \"prod\") {\n      FinagleKafkaProducerBuilder()\n        .dest(\"/s/kafka/recommendations:kafka-tls\")\n        // kerberos params\n        .withConfig(SaslConfigs.SASL_JAAS_CONFIG, jaasConfig)\n        .withConfig(\n          CommonClientConfigs.SECURITY_PROTOCOL_CONFIG,\n          SecurityProtocol.SASL_SSL.toString)\n        .withConfig(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, trustStoreLocation)\n        .withConfig(SaslConfigs.SASL_MECHANISM, SaslConfigs.GSSAPI_MECHANISM)\n        .withConfig(SaslConfigs.SASL_KERBEROS_SERVICE_NAME, \"kafka\")\n        .withConfig(SaslConfigs.SASL_KERBEROS_SERVER_NAME, \"kafka\")\n        // Kafka params\n        .keySerializer(Serdes.String.serializer)\n        .valueSerializer(ScalaSerdes.CompactThrift[GetTweetsRecommendationsScribe].serializer())\n        .clientId(\"cr-mixer\")\n        .enableIdempotence(true)\n        .compressionType(CompressionType.LZ4)\n        .build()\n    } else {\n      new NullKafkaProducer[String, GetTweetsRecommendationsScribe]\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/core/LoggerFactoryModule.scala",
    "content": "package com.twitter.cr_mixer.module.core\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.scribe.ScribeCategories\nimport com.twitter.cr_mixer.scribe.ScribeCategory\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.logging.BareFormatter\nimport com.twitter.logging.Level\nimport com.twitter.logging.Logger\nimport com.twitter.logging.NullHandler\nimport com.twitter.logging.QueueingHandler\nimport com.twitter.logging.ScribeHandler\nimport com.twitter.logging.{LoggerFactory => TwitterLoggerFactory}\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject LoggerFactoryModule extends TwitterModule {\n\n  private val DefaultQueueSize = 10000\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.AbDeciderLogger)\n  def provideAbDeciderLogger(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Logger = {\n    buildLoggerFactory(\n      ScribeCategories.AbDecider,\n      serviceIdentifier.environment,\n      statsReceiver.scope(\"ScribeLogger\"))\n      .apply()\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TopLevelApiDdgMetricsLogger)\n  def provideTopLevelApiDdgMetricsLogger(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Logger = {\n    buildLoggerFactory(\n      ScribeCategories.TopLevelApiDdgMetrics,\n      serviceIdentifier.environment,\n      statsReceiver.scope(\"ScribeLogger\"))\n      .apply()\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TweetRecsLogger)\n  def provideTweetRecsLogger(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Logger = {\n    buildLoggerFactory(\n      ScribeCategories.TweetsRecs,\n      serviceIdentifier.environment,\n      statsReceiver.scope(\"ScribeLogger\"))\n      .apply()\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.BlueVerifiedTweetRecsLogger)\n  def provideVITTweetRecsLogger(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Logger = {\n    buildLoggerFactory(\n      ScribeCategories.VITTweetsRecs,\n      serviceIdentifier.environment,\n      statsReceiver.scope(\"ScribeLogger\"))\n      .apply()\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.RelatedTweetsLogger)\n  def provideRelatedTweetsLogger(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Logger = {\n    buildLoggerFactory(\n      ScribeCategories.RelatedTweets,\n      serviceIdentifier.environment,\n      statsReceiver.scope(\"ScribeLogger\"))\n      .apply()\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.UtegTweetsLogger)\n  def provideUtegTweetsLogger(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Logger = {\n    buildLoggerFactory(\n      ScribeCategories.UtegTweets,\n      serviceIdentifier.environment,\n      statsReceiver.scope(\"ScribeLogger\"))\n      .apply()\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.AdsRecommendationsLogger)\n  def provideAdsRecommendationsLogger(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Logger = {\n    buildLoggerFactory(\n      ScribeCategories.AdsRecommendations,\n      serviceIdentifier.environment,\n      statsReceiver.scope(\"ScribeLogger\"))\n      .apply()\n  }\n\n  private def buildLoggerFactory(\n    category: ScribeCategory,\n    environment: String,\n    statsReceiver: StatsReceiver\n  ): TwitterLoggerFactory = {\n    environment match {\n      case \"prod\" =>\n        TwitterLoggerFactory(\n          node = category.getProdLoggerFactoryNode,\n          level = Some(Level.INFO),\n          useParents = false,\n          handlers = List(\n            QueueingHandler(\n              maxQueueSize = DefaultQueueSize,\n              handler = ScribeHandler(\n                category = category.scribeCategory,\n                formatter = BareFormatter,\n                statsReceiver = statsReceiver.scope(category.getProdLoggerFactoryNode)\n              )\n            )\n          )\n        )\n      case _ =>\n        TwitterLoggerFactory(\n          node = category.getStagingLoggerFactoryNode,\n          level = Some(Level.DEBUG),\n          useParents = false,\n          handlers = List(\n            { () => NullHandler }\n          )\n        )\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/core/MemoizingStatsReceiverModule.scala",
    "content": "package com.twitter.cr_mixer.module.core\n\nimport com.twitter.finagle.stats.LoadedStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.servo.util.MemoizingStatsReceiver\n\nobject MemoizingStatsReceiverModule extends TwitterModule {\n  override def configure(): Unit = {\n    bind[StatsReceiver].toInstance(new MemoizingStatsReceiver(LoadedStatsReceiver))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/core/TimeoutConfigModule.scala",
    "content": "package com.twitter.cr_mixer.module.core\n\nimport com.twitter.inject.TwitterModule\nimport com.google.inject.Provides\nimport javax.inject.Singleton\nimport com.twitter.util.Duration\nimport com.twitter.app.Flag\nimport com.twitter.cr_mixer.config.TimeoutConfig\n\n/**\n * All timeout settings in CrMixer.\n * Timeout numbers are defined in source/cr-mixer/server/config/deploy.aurora\n */\nobject TimeoutConfigModule extends TwitterModule {\n\n  /**\n   * Flag names for client timeout\n   * These are used in modules extending ThriftMethodBuilderClientModule\n   * which cannot accept injection of TimeoutConfig\n   */\n  val EarlybirdClientTimeoutFlagName = \"earlybird.client.timeout\"\n  val FrsClientTimeoutFlagName = \"frsSignalFetch.client.timeout\"\n  val QigRankerClientTimeoutFlagName = \"qigRanker.client.timeout\"\n  val TweetypieClientTimeoutFlagName = \"tweetypie.client.timeout\"\n  val UserTweetGraphClientTimeoutFlagName = \"userTweetGraph.client.timeout\"\n  val UserTweetGraphPlusClientTimeoutFlagName = \"userTweetGraphPlus.client.timeout\"\n  val UserAdGraphClientTimeoutFlagName = \"userAdGraph.client.timeout\"\n  val UserVideoGraphClientTimeoutFlagName = \"userVideoGraph.client.timeout\"\n  val UtegClientTimeoutFlagName = \"uteg.client.timeout\"\n  val NaviRequestTimeoutFlagName = \"navi.client.request.timeout\"\n\n  /**\n   * Flags for timeouts\n   * These are defined and initialized only in this file\n   */\n  // timeout for the service\n  private val serviceTimeout: Flag[Duration] =\n    flag(\"service.timeout\", \"service total timeout\")\n\n  // timeout for signal fetch\n  private val signalFetchTimeout: Flag[Duration] =\n    flag[Duration](\"signalFetch.timeout\", \"signal fetch timeout\")\n\n  // timeout for similarity engine\n  private val similarityEngineTimeout: Flag[Duration] =\n    flag[Duration](\"similarityEngine.timeout\", \"similarity engine timeout\")\n  private val annServiceClientTimeout: Flag[Duration] =\n    flag[Duration](\"annService.client.timeout\", \"annQueryService client timeout\")\n\n  // timeout for user affinities fetcher\n  private val userStateUnderlyingStoreTimeout: Flag[Duration] =\n    flag[Duration](\"userStateUnderlyingStore.timeout\", \"user state underlying store timeout\")\n\n  private val userStateStoreTimeout: Flag[Duration] =\n    flag[Duration](\"userStateStore.timeout\", \"user state store timeout\")\n\n  private val utegSimilarityEngineTimeout: Flag[Duration] =\n    flag[Duration](\"uteg.similarityEngine.timeout\", \"uteg similarity engine timeout\")\n\n  private val earlybirdServerTimeout: Flag[Duration] =\n    flag[Duration](\"earlybird.server.timeout\", \"earlybird server timeout\")\n\n  private val earlybirdSimilarityEngineTimeout: Flag[Duration] =\n    flag[Duration](\"earlybird.similarityEngine.timeout\", \"Earlybird similarity engine timeout\")\n\n  private val frsBasedTweetEndpointTimeout: Flag[Duration] =\n    flag[Duration](\n      \"frsBasedTweet.endpoint.timeout\",\n      \"frsBasedTweet endpoint timeout\"\n    )\n\n  private val topicTweetEndpointTimeout: Flag[Duration] =\n    flag[Duration](\n      \"topicTweet.endpoint.timeout\",\n      \"topicTweet endpoint timeout\"\n    )\n\n  // timeout for Navi client\n  private val naviRequestTimeout: Flag[Duration] =\n    flag[Duration](\n      NaviRequestTimeoutFlagName,\n      Duration.fromMilliseconds(2000),\n      \"Request timeout for a single RPC Call\",\n    )\n\n  @Provides\n  @Singleton\n  def provideTimeoutBudget(): TimeoutConfig =\n    TimeoutConfig(\n      serviceTimeout = serviceTimeout(),\n      signalFetchTimeout = signalFetchTimeout(),\n      similarityEngineTimeout = similarityEngineTimeout(),\n      annServiceClientTimeout = annServiceClientTimeout(),\n      utegSimilarityEngineTimeout = utegSimilarityEngineTimeout(),\n      userStateUnderlyingStoreTimeout = userStateUnderlyingStoreTimeout(),\n      userStateStoreTimeout = userStateStoreTimeout(),\n      earlybirdServerTimeout = earlybirdServerTimeout(),\n      earlybirdSimilarityEngineTimeout = earlybirdSimilarityEngineTimeout(),\n      frsBasedTweetEndpointTimeout = frsBasedTweetEndpointTimeout(),\n      topicTweetEndpointTimeout = topicTweetEndpointTimeout(),\n      naviRequestTimeout = naviRequestTimeout()\n    )\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/grpc_client/NaviGRPCClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.grpc_client\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.finagle.Http\nimport com.twitter.finagle.grpc.FinagleChannelBuilder\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient.MtlsStackClientSyntax\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.util.Duration\nimport io.grpc.ManagedChannel\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject NaviGRPCClientModule extends TwitterModule {\n\n  val maxRetryAttempts = 3\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.HomeNaviGRPCClient)\n  def providesHomeNaviGRPCClient(\n    serviceIdentifier: ServiceIdentifier,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): ManagedChannel = {\n    val label = \"navi-wals-recommended-tweets-home-client\"\n    val dest = \"/s/ads-prediction/navi-wals-recommended-tweets-home\"\n    buildClient(serviceIdentifier, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.AdsFavedNaviGRPCClient)\n  def providesAdsFavedNaviGRPCClient(\n    serviceIdentifier: ServiceIdentifier,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): ManagedChannel = {\n    val label = \"navi-wals-ads-faved-tweets\"\n    val dest = \"/s/ads-prediction/navi-wals-ads-faved-tweets\"\n    buildClient(serviceIdentifier, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.AdsMonetizableNaviGRPCClient)\n  def providesAdsMonetizableNaviGRPCClient(\n    serviceIdentifier: ServiceIdentifier,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): ManagedChannel = {\n    val label = \"navi-wals-ads-monetizable-tweets\"\n    val dest = \"/s/ads-prediction/navi-wals-ads-monetizable-tweets\"\n    buildClient(serviceIdentifier, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  private def buildClient(\n    serviceIdentifier: ServiceIdentifier,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    dest: String,\n    label: String\n  ): ManagedChannel = {\n\n    val stats = statsReceiver.scope(\"clnt\").scope(label)\n\n    val client = Http.client\n      .withLabel(label)\n      .withMutualTls(serviceIdentifier)\n      .withRequestTimeout(timeoutConfig.naviRequestTimeout)\n      .withTransport.connectTimeout(Duration.fromMilliseconds(10000))\n      .withSession.acquisitionTimeout(Duration.fromMilliseconds(20000))\n      .withStatsReceiver(stats)\n      .withHttpStats\n\n    FinagleChannelBuilder\n      .forTarget(dest)\n      .overrideAuthority(\"rustserving\")\n      .maxRetryAttempts(maxRetryAttempts)\n      .enableRetryForStatus(io.grpc.Status.RESOURCE_EXHAUSTED)\n      .enableRetryForStatus(io.grpc.Status.UNKNOWN)\n      .enableUnsafeFullyBufferingMode()\n      .httpClient(client)\n      .build()\n\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/CertoTopicTweetSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TopicTweetWithScore\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.CertoTopicTweetSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.CertoTopicTweetSimilarityEngine.Query\nimport com.twitter.cr_mixer.similarity_engine.EngineQuery\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.DeciderConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.thriftscala.TopicId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.topic_recos.thriftscala.TweetWithScores\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject CertoTopicTweetSimilarityEngineModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.CertoTopicTweetSimilarityEngine)\n  def providesCertoTopicTweetSimilarityEngine(\n    @Named(ModuleNames.CertoStratoStoreName) certoStratoStore: ReadableStore[\n      TopicId,\n      Seq[TweetWithScores]\n    ],\n    timeoutConfig: TimeoutConfig,\n    decider: CrMixerDecider,\n    statsReceiver: StatsReceiver\n  ): StandardSimilarityEngine[\n    EngineQuery[Query],\n    TopicTweetWithScore\n  ] = {\n    new StandardSimilarityEngine[EngineQuery[Query], TopicTweetWithScore](\n      implementingStore = CertoTopicTweetSimilarityEngine(certoStratoStore, statsReceiver),\n      identifier = SimilarityEngineType.CertoTopicTweet,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.topicTweetEndpointTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig =\n            Some(DeciderConfig(decider, DeciderConstants.enableTopicTweetTrafficDeciderKey)),\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/ConsumerBasedWalsSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.similarity_engine.ConsumerBasedWalsSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport io.grpc.ManagedChannel\nimport javax.inject.Named\n\nobject ConsumerBasedWalsSimilarityEngineModule extends TwitterModule {\n  @Provides\n  @Named(ModuleNames.ConsumerBasedWalsSimilarityEngine)\n  def providesConsumerBasedWalsSimilarityEngine(\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    @Named(ModuleNames.HomeNaviGRPCClient) homeNaviGRPCClient: ManagedChannel,\n    @Named(ModuleNames.AdsFavedNaviGRPCClient) adsFavedNaviGRPCClient: ManagedChannel,\n    @Named(ModuleNames.AdsMonetizableNaviGRPCClient) adsMonetizableNaviGRPCClient: ManagedChannel,\n  ): StandardSimilarityEngine[\n    ConsumerBasedWalsSimilarityEngine.Query,\n    TweetWithScore\n  ] = {\n\n    val underlyingStore = new ConsumerBasedWalsSimilarityEngine(\n      homeNaviGRPCClient,\n      adsFavedNaviGRPCClient,\n      adsMonetizableNaviGRPCClient,\n      statsReceiver\n    )\n\n    new StandardSimilarityEngine[\n      ConsumerBasedWalsSimilarityEngine.Query,\n      TweetWithScore\n    ](\n      implementingStore = underlyingStore,\n      identifier = SimilarityEngineType.ConsumerBasedWalsANN,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = None,\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/ConsumerEmbeddingBasedTripSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TripTweetWithScore\nimport com.twitter.cr_mixer.similarity_engine.ConsumerEmbeddingBasedTripSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TripEngineQuery\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripTweet\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripDomain\nimport javax.inject.Named\n\nobject ConsumerEmbeddingBasedTripSimilarityEngineModule extends TwitterModule {\n  @Provides\n  @Named(ModuleNames.ConsumerEmbeddingBasedTripSimilarityEngine)\n  def providesConsumerEmbeddingBasedTripSimilarityEngineModule(\n    @Named(ModuleNames.RmsUserLogFavInterestedInEmbeddingStore)\n    userLogFavInterestedInEmbeddingStore: ReadableStore[UserId, SimClustersEmbedding],\n    @Named(ModuleNames.RmsUserFollowInterestedInEmbeddingStore)\n    userFollowInterestedInEmbeddingStore: ReadableStore[UserId, SimClustersEmbedding],\n    @Named(ModuleNames.TripCandidateStore)\n    tripCandidateStore: ReadableStore[TripDomain, Seq[TripTweet]],\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): StandardSimilarityEngine[TripEngineQuery, TripTweetWithScore] = {\n    val underlyingStore = ObservedReadableStore(\n      ConsumerEmbeddingBasedTripSimilarityEngine(\n        embeddingStoreLookUpMap = Map(\n          ModelConfig.ConsumerLogFavBasedInterestedInEmbedding -> userLogFavInterestedInEmbeddingStore,\n          ModelConfig.ConsumerFollowBasedInterestedInEmbedding -> userFollowInterestedInEmbeddingStore,\n        ),\n        tripCandidateSource = tripCandidateStore,\n        statsReceiver\n      ))(statsReceiver.scope(\"TripSimilarityEngine\"))\n\n    new StandardSimilarityEngine[TripEngineQuery, TripTweetWithScore](\n      implementingStore = underlyingStore,\n      identifier = SimilarityEngineType.ExploreTripOfflineSimClustersTweets,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = None,\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/ConsumerEmbeddingBasedTwHINSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.ann.common.thriftscala.AnnQueryService\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.cr_mixer.module.EmbeddingStoreModule\nimport com.twitter.cr_mixer.module.thrift_client.AnnQueryServiceClientModule\nimport com.twitter.cr_mixer.similarity_engine.HnswANNSimilarityEngine\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport com.twitter.ml.api.{thriftscala => api}\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\n\nobject ConsumerEmbeddingBasedTwHINSimilarityEngineModule extends TwitterModule {\n  @Provides\n  @Named(ModuleNames.ConsumerEmbeddingBasedTwHINANNSimilarityEngine)\n  def providesConsumerEmbeddingBasedTwHINANNSimilarityEngine(\n    // MH stores\n    @Named(EmbeddingStoreModule.ConsumerBasedTwHINEmbeddingRegularUpdateMhStoreName)\n    consumerBasedTwHINEmbeddingRegularUpdateMhStore: ReadableStore[InternalId, api.Embedding],\n    @Named(EmbeddingStoreModule.DebuggerDemoUserEmbeddingMhStoreName)\n    debuggerDemoUserEmbeddingMhStore: ReadableStore[InternalId, api.Embedding],\n    @Named(AnnQueryServiceClientModule.TwHINRegularUpdateAnnServiceClientName)\n    twHINRegularUpdateAnnService: AnnQueryService.MethodPerEndpoint,\n    @Named(AnnQueryServiceClientModule.DebuggerDemoAnnServiceClientName)\n    debuggerDemoAnnService: AnnQueryService.MethodPerEndpoint,\n    // Other configs\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver\n  ): HnswANNSimilarityEngine = {\n    new HnswANNSimilarityEngine(\n      embeddingStoreLookUpMap = Map(\n        ModelConfig.ConsumerBasedTwHINRegularUpdateAll20221024 -> consumerBasedTwHINEmbeddingRegularUpdateMhStore,\n        ModelConfig.DebuggerDemo -> debuggerDemoUserEmbeddingMhStore,\n      ),\n      annServiceLookUpMap = Map(\n        ModelConfig.ConsumerBasedTwHINRegularUpdateAll20221024 -> twHINRegularUpdateAnnService,\n        ModelConfig.DebuggerDemo -> debuggerDemoAnnService,\n      ),\n      globalStats = statsReceiver,\n      identifier = SimilarityEngineType.ConsumerEmbeddingBasedTwHINANN,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = None,\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/ConsumerEmbeddingBasedTwoTowerSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module\npackage similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.ann.common.thriftscala.AnnQueryService\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.cr_mixer.module.EmbeddingStoreModule\nimport com.twitter.cr_mixer.module.thrift_client.AnnQueryServiceClientModule\nimport com.twitter.cr_mixer.similarity_engine.HnswANNSimilarityEngine\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport com.twitter.ml.api.{thriftscala => api}\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\n\nobject ConsumerEmbeddingBasedTwoTowerSimilarityEngineModule extends TwitterModule {\n  @Provides\n  @Named(ModuleNames.ConsumerEmbeddingBasedTwoTowerANNSimilarityEngine)\n  def providesConsumerEmbeddingBasedTwoTowerANNSimilarityEngine(\n    @Named(EmbeddingStoreModule.TwoTowerFavConsumerEmbeddingMhStoreName)\n    twoTowerFavConsumerEmbeddingMhStore: ReadableStore[InternalId, api.Embedding],\n    @Named(AnnQueryServiceClientModule.TwoTowerFavAnnServiceClientName)\n    twoTowerFavAnnService: AnnQueryService.MethodPerEndpoint,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver\n  ): HnswANNSimilarityEngine = {\n    new HnswANNSimilarityEngine(\n      embeddingStoreLookUpMap = Map(\n        ModelConfig.TwoTowerFavALL20220808 -> twoTowerFavConsumerEmbeddingMhStore,\n      ),\n      annServiceLookUpMap = Map(\n        ModelConfig.TwoTowerFavALL20220808 -> twoTowerFavAnnService,\n      ),\n      globalStats = statsReceiver,\n      identifier = SimilarityEngineType.ConsumerEmbeddingBasedTwoTowerANN,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = None,\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/ConsumersBasedUserAdGraphSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.ConsumersBasedUserAdGraphSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.DeciderConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.recos.user_ad_graph.thriftscala.ConsumersBasedRelatedAdRequest\nimport com.twitter.recos.user_ad_graph.thriftscala.RelatedAdResponse\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject ConsumersBasedUserAdGraphSimilarityEngineModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.ConsumersBasedUserAdGraphSimilarityEngine)\n  def providesConsumersBasedUserAdGraphSimilarityEngine(\n    @Named(ModuleNames.ConsumerBasedUserAdGraphStore)\n    consumersBasedUserAdGraphStore: ReadableStore[\n      ConsumersBasedRelatedAdRequest,\n      RelatedAdResponse\n    ],\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    decider: CrMixerDecider\n  ): StandardSimilarityEngine[\n    ConsumersBasedUserAdGraphSimilarityEngine.Query,\n    TweetWithScore\n  ] = {\n\n    new StandardSimilarityEngine[\n      ConsumersBasedUserAdGraphSimilarityEngine.Query,\n      TweetWithScore\n    ](\n      implementingStore =\n        ConsumersBasedUserAdGraphSimilarityEngine(consumersBasedUserAdGraphStore, statsReceiver),\n      identifier = SimilarityEngineType.ConsumersBasedUserTweetGraph,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig =\n            Some(DeciderConfig(decider, DeciderConstants.enableUserTweetGraphTrafficDeciderKey)),\n          enableFeatureSwitch = None\n        )\n      ),\n      memCacheConfig = None\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/ConsumersBasedUserVideoGraphSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.ConsumersBasedUserVideoGraphSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.DeciderConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.recos.user_video_graph.thriftscala.ConsumersBasedRelatedTweetRequest\nimport com.twitter.recos.user_video_graph.thriftscala.RelatedTweetResponse\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject ConsumersBasedUserVideoGraphSimilarityEngineModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.ConsumersBasedUserVideoGraphSimilarityEngine)\n  def providesConsumersBasedUserVideoGraphSimilarityEngine(\n    @Named(ModuleNames.ConsumerBasedUserVideoGraphStore)\n    consumersBasedUserVideoGraphStore: ReadableStore[\n      ConsumersBasedRelatedTweetRequest,\n      RelatedTweetResponse\n    ],\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    decider: CrMixerDecider\n  ): StandardSimilarityEngine[\n    ConsumersBasedUserVideoGraphSimilarityEngine.Query,\n    TweetWithScore\n  ] = {\n\n    new StandardSimilarityEngine[\n      ConsumersBasedUserVideoGraphSimilarityEngine.Query,\n      TweetWithScore\n    ](\n      implementingStore = ConsumersBasedUserVideoGraphSimilarityEngine(\n        consumersBasedUserVideoGraphStore,\n        statsReceiver),\n      identifier = SimilarityEngineType.ConsumersBasedUserVideoGraph,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig =\n            Some(DeciderConfig(decider, DeciderConstants.enableUserVideoGraphTrafficDeciderKey)),\n          enableFeatureSwitch = None\n        )\n      ),\n      memCacheConfig = None\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/DiffusionBasedSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module\npackage similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.simclusters_v2.thriftscala.TweetsWithScore\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.similarity_engine.DiffusionBasedSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.DiffusionBasedSimilarityEngine.Query\nimport com.twitter.cr_mixer.similarity_engine.LookupSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject DiffusionBasedSimilarityEngineModule extends TwitterModule {\n  @Provides\n  @Singleton\n  @Named(ModuleNames.DiffusionBasedSimilarityEngine)\n  def providesDiffusionBasedSimilarityEngineModule(\n    @Named(ModuleNames.RetweetBasedDiffusionRecsMhStore)\n    retweetBasedDiffusionRecsMhStore: ReadableStore[Long, TweetsWithScore],\n    timeoutConfig: TimeoutConfig,\n    globalStats: StatsReceiver\n  ): LookupSimilarityEngine[Query, TweetWithScore] = {\n\n    val versionedStoreMap = Map(\n      ModelConfig.RetweetBasedDiffusion -> DiffusionBasedSimilarityEngine(\n        retweetBasedDiffusionRecsMhStore,\n        globalStats),\n    )\n\n    new LookupSimilarityEngine[Query, TweetWithScore](\n      versionedStoreMap = versionedStoreMap,\n      identifier = SimilarityEngineType.DiffusionBasedTweet,\n      globalStats = globalStats,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = None,\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/EarlybirdSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.EarlybirdModelBasedSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.EarlybirdRecencyBasedSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.EarlybirdSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.EarlybirdTensorflowBasedSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.DeciderConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport javax.inject.Singleton\n\nobject EarlybirdSimilarityEngineModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def providesRecencyBasedEarlybirdSimilarityEngine(\n    earlybirdRecencyBasedSimilarityEngine: EarlybirdRecencyBasedSimilarityEngine,\n    timeoutConfig: TimeoutConfig,\n    decider: CrMixerDecider,\n    statsReceiver: StatsReceiver\n  ): EarlybirdSimilarityEngine[\n    EarlybirdRecencyBasedSimilarityEngine.EarlybirdRecencyBasedSearchQuery,\n    EarlybirdRecencyBasedSimilarityEngine\n  ] = {\n    new EarlybirdSimilarityEngine[\n      EarlybirdRecencyBasedSimilarityEngine.EarlybirdRecencyBasedSearchQuery,\n      EarlybirdRecencyBasedSimilarityEngine\n    ](\n      implementingStore = earlybirdRecencyBasedSimilarityEngine,\n      identifier = SimilarityEngineType.EarlybirdRecencyBasedSimilarityEngine,\n      globalStats =\n        statsReceiver.scope(SimilarityEngineType.EarlybirdRecencyBasedSimilarityEngine.name),\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.earlybirdSimilarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = Some(\n            DeciderConfig(\n              decider = decider,\n              deciderString = DeciderConstants.enableEarlybirdTrafficDeciderKey\n            )),\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n\n  @Provides\n  @Singleton\n  def providesModelBasedEarlybirdSimilarityEngine(\n    earlybirdModelBasedSimilarityEngine: EarlybirdModelBasedSimilarityEngine,\n    timeoutConfig: TimeoutConfig,\n    decider: CrMixerDecider,\n    statsReceiver: StatsReceiver\n  ): EarlybirdSimilarityEngine[\n    EarlybirdModelBasedSimilarityEngine.EarlybirdModelBasedSearchQuery,\n    EarlybirdModelBasedSimilarityEngine\n  ] = {\n    new EarlybirdSimilarityEngine[\n      EarlybirdModelBasedSimilarityEngine.EarlybirdModelBasedSearchQuery,\n      EarlybirdModelBasedSimilarityEngine\n    ](\n      implementingStore = earlybirdModelBasedSimilarityEngine,\n      identifier = SimilarityEngineType.EarlybirdModelBasedSimilarityEngine,\n      globalStats =\n        statsReceiver.scope(SimilarityEngineType.EarlybirdModelBasedSimilarityEngine.name),\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.earlybirdSimilarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = Some(\n            DeciderConfig(\n              decider = decider,\n              deciderString = DeciderConstants.enableEarlybirdTrafficDeciderKey\n            )),\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n\n  @Provides\n  @Singleton\n  def providesTensorflowBasedEarlybirdSimilarityEngine(\n    earlybirdTensorflowBasedSimilarityEngine: EarlybirdTensorflowBasedSimilarityEngine,\n    timeoutConfig: TimeoutConfig,\n    decider: CrMixerDecider,\n    statsReceiver: StatsReceiver\n  ): EarlybirdSimilarityEngine[\n    EarlybirdTensorflowBasedSimilarityEngine.EarlybirdTensorflowBasedSearchQuery,\n    EarlybirdTensorflowBasedSimilarityEngine\n  ] = {\n    new EarlybirdSimilarityEngine[\n      EarlybirdTensorflowBasedSimilarityEngine.EarlybirdTensorflowBasedSearchQuery,\n      EarlybirdTensorflowBasedSimilarityEngine\n    ](\n      implementingStore = earlybirdTensorflowBasedSimilarityEngine,\n      identifier = SimilarityEngineType.EarlybirdTensorflowBasedSimilarityEngine,\n      globalStats =\n        statsReceiver.scope(SimilarityEngineType.EarlybirdTensorflowBasedSimilarityEngine.name),\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.earlybirdSimilarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = Some(\n            DeciderConfig(\n              decider = decider,\n              deciderString = DeciderConstants.enableEarlybirdTrafficDeciderKey\n            )),\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/ProducerBasedUnifiedSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithCandidateGenerationInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.similarity_engine.ProducerBasedUserTweetGraphSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.ProducerBasedUnifiedSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.cr_mixer.similarity_engine.SimClustersANNSimilarityEngine\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject ProducerBasedUnifiedSimilarityEngineModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.ProducerBasedUnifiedSimilarityEngine)\n  def providesProducerBasedUnifiedSimilarityEngine(\n    @Named(ModuleNames.ProducerBasedUserTweetGraphSimilarityEngine)\n    producerBasedUserTweetGraphSimilarityEngine: StandardSimilarityEngine[\n      ProducerBasedUserTweetGraphSimilarityEngine.Query,\n      TweetWithScore\n    ],\n    @Named(ModuleNames.SimClustersANNSimilarityEngine)\n    simClustersANNSimilarityEngine: StandardSimilarityEngine[\n      SimClustersANNSimilarityEngine.Query,\n      TweetWithScore\n    ],\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): StandardSimilarityEngine[\n    ProducerBasedUnifiedSimilarityEngine.Query,\n    TweetWithCandidateGenerationInfo\n  ] = {\n\n    val underlyingStore: ReadableStore[ProducerBasedUnifiedSimilarityEngine.Query, Seq[\n      TweetWithCandidateGenerationInfo\n    ]] = ProducerBasedUnifiedSimilarityEngine(\n      producerBasedUserTweetGraphSimilarityEngine,\n      simClustersANNSimilarityEngine,\n      statsReceiver\n    )\n\n    new StandardSimilarityEngine[\n      ProducerBasedUnifiedSimilarityEngine.Query,\n      TweetWithCandidateGenerationInfo\n    ](\n      implementingStore = underlyingStore,\n      identifier = SimilarityEngineType.ProducerBasedUnifiedSimilarityEngine,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = None,\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/ProducerBasedUserAdGraphSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.ProducerBasedUserAdGraphSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.DeciderConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine._\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.keyHasher\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.recos.user_ad_graph.thriftscala.UserAdGraph\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject ProducerBasedUserAdGraphSimilarityEngineModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.ProducerBasedUserAdGraphSimilarityEngine)\n  def providesProducerBasedUserAdGraphSimilarityEngine(\n    userAdGraphService: UserAdGraph.MethodPerEndpoint,\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    decider: CrMixerDecider\n  ): StandardSimilarityEngine[\n    ProducerBasedUserAdGraphSimilarityEngine.Query,\n    TweetWithScore\n  ] = {\n    new StandardSimilarityEngine[\n      ProducerBasedUserAdGraphSimilarityEngine.Query,\n      TweetWithScore\n    ](\n      implementingStore =\n        ProducerBasedUserAdGraphSimilarityEngine(userAdGraphService, statsReceiver),\n      identifier = SimilarityEngineType.ProducerBasedUserAdGraph,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig =\n            Some(DeciderConfig(decider, DeciderConstants.enableUserAdGraphTrafficDeciderKey)),\n          enableFeatureSwitch = None\n        )\n      ),\n      memCacheConfig = Some(\n        MemCacheConfig(\n          cacheClient = crMixerUnifiedCacheClient,\n          ttl = 10.minutes,\n          keyToString = { k =>\n            //Example Query CRMixer:ProducerBasedUTG:1234567890ABCDEF\n            f\"ProducerBasedUTG:${keyHasher.hashKey(k.toString.getBytes)}%X\"\n          }\n        ))\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/ProducerBasedUserTweetGraphSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.ProducerBasedUserTweetGraphSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine._\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.keyHasher\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.DeciderConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.recos.user_tweet_graph.thriftscala.UserTweetGraph\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject ProducerBasedUserTweetGraphSimilarityEngineModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.ProducerBasedUserTweetGraphSimilarityEngine)\n  def providesProducerBasedUserTweetGraphSimilarityEngine(\n    userTweetGraphService: UserTweetGraph.MethodPerEndpoint,\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    decider: CrMixerDecider\n  ): StandardSimilarityEngine[\n    ProducerBasedUserTweetGraphSimilarityEngine.Query,\n    TweetWithScore\n  ] = {\n    new StandardSimilarityEngine[\n      ProducerBasedUserTweetGraphSimilarityEngine.Query,\n      TweetWithScore\n    ](\n      implementingStore =\n        ProducerBasedUserTweetGraphSimilarityEngine(userTweetGraphService, statsReceiver),\n      identifier = SimilarityEngineType.ProducerBasedUserTweetGraph,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig =\n            Some(DeciderConfig(decider, DeciderConstants.enableUserTweetGraphTrafficDeciderKey)),\n          enableFeatureSwitch = None\n        )\n      ),\n      memCacheConfig = Some(\n        MemCacheConfig(\n          cacheClient = crMixerUnifiedCacheClient,\n          ttl = 10.minutes,\n          keyToString = { k =>\n            //Example Query CRMixer:ProducerBasedUTG:1234567890ABCDEF\n            f\"ProducerBasedUTG:${keyHasher.hashKey(k.toString.getBytes)}%X\"\n          }\n        ))\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/SimClustersANNSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.similarity_engine.SimClustersANNSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimClustersANNSimilarityEngine.Query\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.relevance_platform.common.injection.LZ4Injection\nimport com.twitter.relevance_platform.common.injection.SeqObjectInjection\nimport com.twitter.simclusters_v2.candidate_source.SimClustersANNCandidateSource.CacheableShortTTLEmbeddingTypes\nimport com.twitter.simclustersann.thriftscala.SimClustersANNService\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject SimClustersANNSimilarityEngineModule extends TwitterModule {\n\n  private val keyHasher: KeyHasher = KeyHasher.FNV1A_64\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.SimClustersANNSimilarityEngine)\n  def providesProdSimClustersANNSimilarityEngine(\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    simClustersANNServiceNameToClientMapper: Map[String, SimClustersANNService.MethodPerEndpoint],\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver\n  ): StandardSimilarityEngine[Query, TweetWithScore] = {\n\n    val underlyingStore =\n      SimClustersANNSimilarityEngine(simClustersANNServiceNameToClientMapper, statsReceiver)\n\n    val observedReadableStore =\n      ObservedReadableStore(underlyingStore)(statsReceiver.scope(\"SimClustersANNServiceStore\"))\n\n    val memCachedStore: ReadableStore[Query, Seq[TweetWithScore]] =\n      ObservedMemcachedReadableStore\n        .fromCacheClient(\n          backingStore = observedReadableStore,\n          cacheClient = crMixerUnifiedCacheClient,\n          ttl = 10.minutes\n        )(\n          valueInjection = LZ4Injection.compose(SeqObjectInjection[TweetWithScore]()),\n          statsReceiver = statsReceiver.scope(\"simclusters_ann_store_memcache\"),\n          keyToString = { k =>\n            //Example Query CRMixer:SCANN:1:2:1234567890ABCDEF:1234567890ABCDEF\n            f\"CRMixer:SCANN:${k.simClustersANNQuery.sourceEmbeddingId.embeddingType.getValue()}%X\" +\n              f\":${k.simClustersANNQuery.sourceEmbeddingId.modelVersion.getValue()}%X\" +\n              f\":${keyHasher.hashKey(k.simClustersANNQuery.sourceEmbeddingId.internalId.toString.getBytes)}%X\" +\n              f\":${keyHasher.hashKey(k.simClustersANNQuery.config.toString.getBytes)}%X\"\n          }\n        )\n\n    // Only cache the candidates if it's not Consumer-source. For example, TweetSource,\n    // ProducerSource, TopicSource\n    val wrapperStats = statsReceiver.scope(\"SimClustersANNWrapperStore\")\n\n    val wrapperStore: ReadableStore[Query, Seq[TweetWithScore]] =\n      buildWrapperStore(memCachedStore, observedReadableStore, wrapperStats)\n\n    new StandardSimilarityEngine[\n      Query,\n      TweetWithScore\n    ](\n      implementingStore = wrapperStore,\n      identifier = SimilarityEngineType.SimClustersANN,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = None,\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n\n  def buildWrapperStore(\n    memCachedStore: ReadableStore[Query, Seq[TweetWithScore]],\n    underlyingStore: ReadableStore[Query, Seq[TweetWithScore]],\n    wrapperStats: StatsReceiver\n  ): ReadableStore[Query, Seq[TweetWithScore]] = {\n\n    // Only cache the candidates if it's not Consumer-source. For example, TweetSource,\n    // ProducerSource, TopicSource\n    val wrapperStore: ReadableStore[Query, Seq[TweetWithScore]] =\n      new ReadableStore[Query, Seq[TweetWithScore]] {\n\n        override def multiGet[K1 <: Query](\n          queries: Set[K1]\n        ): Map[K1, Future[Option[Seq[TweetWithScore]]]] = {\n          val (cacheableQueries, nonCacheableQueries) =\n            queries.partition { query =>\n              CacheableShortTTLEmbeddingTypes.contains(\n                query.simClustersANNQuery.sourceEmbeddingId.embeddingType)\n            }\n          memCachedStore.multiGet(cacheableQueries) ++\n            underlyingStore.multiGet(nonCacheableQueries)\n        }\n      }\n    wrapperStore\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/SkitTopicTweetSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TopicTweetWithScore\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.EngineQuery\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.DeciderConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.SkitHighPrecisionTopicTweetSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SkitTopicTweetSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SkitTopicTweetSimilarityEngine.Query\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.topic_recos.thriftscala.TopicTweet\nimport com.twitter.topic_recos.thriftscala.TopicTweetPartitionFlatKey\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject SkitTopicTweetSimilarityEngineModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.SkitHighPrecisionTopicTweetSimilarityEngine)\n  def providesSkitHighPrecisionTopicTweetSimilarityEngine(\n    @Named(ModuleNames.SkitStratoStoreName) skitStratoStore: ReadableStore[\n      TopicTweetPartitionFlatKey,\n      Seq[TopicTweet]\n    ],\n    timeoutConfig: TimeoutConfig,\n    decider: CrMixerDecider,\n    statsReceiver: StatsReceiver\n  ): StandardSimilarityEngine[\n    EngineQuery[Query],\n    TopicTweetWithScore\n  ] = {\n    new StandardSimilarityEngine[EngineQuery[Query], TopicTweetWithScore](\n      implementingStore =\n        SkitHighPrecisionTopicTweetSimilarityEngine(skitStratoStore, statsReceiver),\n      identifier = SimilarityEngineType.SkitHighPrecisionTopicTweet,\n      globalStats = statsReceiver.scope(SimilarityEngineType.SkitHighPrecisionTopicTweet.name),\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.topicTweetEndpointTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig =\n            Some(DeciderConfig(decider, DeciderConstants.enableTopicTweetTrafficDeciderKey)),\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n  @Provides\n  @Singleton\n  @Named(ModuleNames.SkitTopicTweetSimilarityEngine)\n  def providesSkitTfgTopicTweetSimilarityEngine(\n    @Named(ModuleNames.SkitStratoStoreName) skitStratoStore: ReadableStore[\n      TopicTweetPartitionFlatKey,\n      Seq[TopicTweet]\n    ],\n    timeoutConfig: TimeoutConfig,\n    decider: CrMixerDecider,\n    statsReceiver: StatsReceiver\n  ): StandardSimilarityEngine[\n    EngineQuery[Query],\n    TopicTweetWithScore\n  ] = {\n    new StandardSimilarityEngine[EngineQuery[Query], TopicTweetWithScore](\n      implementingStore = SkitTopicTweetSimilarityEngine(skitStratoStore, statsReceiver),\n      identifier = SimilarityEngineType.SkitTfgTopicTweet,\n      globalStats = statsReceiver.scope(SimilarityEngineType.SkitTfgTopicTweet.name),\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.topicTweetEndpointTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig =\n            Some(DeciderConfig(decider, DeciderConstants.enableTopicTweetTrafficDeciderKey)),\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/TweetBasedQigSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine._\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.keyHasher\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.DeciderConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TweetBasedQigSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.qig_ranker.thriftscala.QigRanker\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject TweetBasedQigSimilarityEngineModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TweetBasedQigSimilarityEngine)\n  def providesTweetBasedQigSimilarTweetsCandidateSource(\n    qigRanker: QigRanker.MethodPerEndpoint,\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    decider: CrMixerDecider\n  ): StandardSimilarityEngine[\n    TweetBasedQigSimilarityEngine.Query,\n    TweetWithScore\n  ] = {\n    new StandardSimilarityEngine[\n      TweetBasedQigSimilarityEngine.Query,\n      TweetWithScore\n    ](\n      implementingStore = TweetBasedQigSimilarityEngine(qigRanker, statsReceiver),\n      identifier = SimilarityEngineType.Qig,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig =\n            Some(DeciderConfig(decider, DeciderConstants.enableQigSimilarTweetsTrafficDeciderKey)),\n          enableFeatureSwitch = None\n        )\n      ),\n      memCacheConfig = Some(\n        MemCacheConfig(\n          cacheClient = crMixerUnifiedCacheClient,\n          ttl = 10.minutes,\n          keyToString = { k =>\n            f\"TweetBasedQIGRanker:${keyHasher.hashKey(k.sourceId.toString.getBytes)}%X\"\n          }\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/TweetBasedTwHINSimlarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\nimport com.google.inject.Provides\nimport com.twitter.ann.common.thriftscala.AnnQueryService\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.cr_mixer.module.EmbeddingStoreModule\nimport com.twitter.cr_mixer.module.thrift_client.AnnQueryServiceClientModule\nimport com.twitter.cr_mixer.similarity_engine.HnswANNSimilarityEngine\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport com.twitter.ml.api.{thriftscala => api}\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.similarity_engine.HnswANNEngineQuery\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\n\nobject TweetBasedTwHINSimlarityEngineModule extends TwitterModule {\n  @Provides\n  @Named(ModuleNames.TweetBasedTwHINANNSimilarityEngine)\n  def providesTweetBasedTwHINANNSimilarityEngine(\n    // MH stores\n    @Named(EmbeddingStoreModule.TwHINEmbeddingRegularUpdateMhStoreName)\n    twHINEmbeddingRegularUpdateMhStore: ReadableStore[InternalId, api.Embedding],\n    @Named(EmbeddingStoreModule.DebuggerDemoTweetEmbeddingMhStoreName)\n    debuggerDemoTweetEmbeddingMhStore: ReadableStore[InternalId, api.Embedding],\n    // ANN clients\n    @Named(AnnQueryServiceClientModule.TwHINRegularUpdateAnnServiceClientName)\n    twHINRegularUpdateAnnService: AnnQueryService.MethodPerEndpoint,\n    @Named(AnnQueryServiceClientModule.DebuggerDemoAnnServiceClientName)\n    debuggerDemoAnnService: AnnQueryService.MethodPerEndpoint,\n    // Other configs\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver\n  ): HnswANNSimilarityEngine = {\n    new HnswANNSimilarityEngine(\n      embeddingStoreLookUpMap = Map(\n        ModelConfig.TweetBasedTwHINRegularUpdateAll20221024 -> twHINEmbeddingRegularUpdateMhStore,\n        ModelConfig.DebuggerDemo -> debuggerDemoTweetEmbeddingMhStore,\n      ),\n      annServiceLookUpMap = Map(\n        ModelConfig.TweetBasedTwHINRegularUpdateAll20221024 -> twHINRegularUpdateAnnService,\n        ModelConfig.DebuggerDemo -> debuggerDemoAnnService,\n      ),\n      globalStats = statsReceiver,\n      identifier = SimilarityEngineType.TweetBasedTwHINANN,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = None,\n          enableFeatureSwitch = None\n        )\n      ),\n      memCacheConfigOpt = Some(\n        SimilarityEngine.MemCacheConfig[HnswANNEngineQuery](\n          cacheClient = crMixerUnifiedCacheClient,\n          ttl = 30.minutes,\n          keyToString = (query: HnswANNEngineQuery) =>\n            SimilarityEngine.keyHasher.hashKey(query.cacheKey.getBytes).toString\n        ))\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/TweetBasedUnifiedSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithCandidateGenerationInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.similarity_engine.HnswANNSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimClustersANNSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TweetBasedQigSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TweetBasedUnifiedSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TweetBasedUserTweetGraphSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TweetBasedUserVideoGraphSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject TweetBasedUnifiedSimilarityEngineModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TweetBasedUnifiedSimilarityEngine)\n  def providesTweetBasedUnifiedSimilarityEngine(\n    @Named(ModuleNames.TweetBasedUserTweetGraphSimilarityEngine) tweetBasedUserTweetGraphSimilarityEngine: StandardSimilarityEngine[\n      TweetBasedUserTweetGraphSimilarityEngine.Query,\n      TweetWithScore\n    ],\n    @Named(ModuleNames.TweetBasedUserVideoGraphSimilarityEngine) tweetBasedUserVideoGraphSimilarityEngine: StandardSimilarityEngine[\n      TweetBasedUserVideoGraphSimilarityEngine.Query,\n      TweetWithScore\n    ],\n    @Named(ModuleNames.TweetBasedTwHINANNSimilarityEngine)\n    tweetBasedTwHINANNSimilarityEngine: HnswANNSimilarityEngine,\n    @Named(ModuleNames.TweetBasedQigSimilarityEngine) tweetBasedQigSimilarityEngine: StandardSimilarityEngine[\n      TweetBasedQigSimilarityEngine.Query,\n      TweetWithScore\n    ],\n    @Named(ModuleNames.SimClustersANNSimilarityEngine)\n    simClustersANNSimilarityEngine: StandardSimilarityEngine[\n      SimClustersANNSimilarityEngine.Query,\n      TweetWithScore\n    ],\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): StandardSimilarityEngine[\n    TweetBasedUnifiedSimilarityEngine.Query,\n    TweetWithCandidateGenerationInfo\n  ] = {\n\n    val underlyingStore: ReadableStore[TweetBasedUnifiedSimilarityEngine.Query, Seq[\n      TweetWithCandidateGenerationInfo\n    ]] = TweetBasedUnifiedSimilarityEngine(\n      tweetBasedUserTweetGraphSimilarityEngine,\n      tweetBasedUserVideoGraphSimilarityEngine,\n      simClustersANNSimilarityEngine,\n      tweetBasedQigSimilarityEngine,\n      tweetBasedTwHINANNSimilarityEngine,\n      statsReceiver\n    )\n\n    new StandardSimilarityEngine[\n      TweetBasedUnifiedSimilarityEngine.Query,\n      TweetWithCandidateGenerationInfo\n    ](\n      implementingStore = underlyingStore,\n      identifier = SimilarityEngineType.TweetBasedUnifiedSimilarityEngine,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = None,\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/TweetBasedUserAdGraphSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.DeciderConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TweetBasedUserAdGraphSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.recos.user_ad_graph.thriftscala.UserAdGraph\nimport com.twitter.relevance_platform.common.injection.LZ4Injection\nimport com.twitter.relevance_platform.common.injection.SeqObjectInjection\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.twistly.thriftscala.TweetRecentEngagedUsers\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject TweetBasedUserAdGraphSimilarityEngineModule extends TwitterModule {\n\n  private val keyHasher: KeyHasher = KeyHasher.FNV1A_64\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TweetBasedUserAdGraphSimilarityEngine)\n  def providesTweetBasedUserAdGraphSimilarityEngine(\n    userAdGraphService: UserAdGraph.MethodPerEndpoint,\n    tweetRecentEngagedUserStore: ReadableStore[TweetId, TweetRecentEngagedUsers],\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    decider: CrMixerDecider\n  ): StandardSimilarityEngine[\n    TweetBasedUserAdGraphSimilarityEngine.Query,\n    TweetWithScore\n  ] = {\n\n    val underlyingStore = TweetBasedUserAdGraphSimilarityEngine(\n      userAdGraphService,\n      tweetRecentEngagedUserStore,\n      statsReceiver)\n\n    val memCachedStore: ReadableStore[\n      TweetBasedUserAdGraphSimilarityEngine.Query,\n      Seq[\n        TweetWithScore\n      ]\n    ] =\n      ObservedMemcachedReadableStore\n        .fromCacheClient(\n          backingStore = underlyingStore,\n          cacheClient = crMixerUnifiedCacheClient,\n          ttl = 10.minutes\n        )(\n          valueInjection = LZ4Injection.compose(SeqObjectInjection[TweetWithScore]()),\n          statsReceiver = statsReceiver.scope(\"tweet_based_user_ad_graph_store_memcache\"),\n          keyToString = { k =>\n            //Example Query CRMixer:TweetBasedUTG:1234567890ABCDEF\n            f\"CRMixer:TweetBasedUAG:${keyHasher.hashKey(k.toString.getBytes)}%X\"\n          }\n        )\n\n    new StandardSimilarityEngine[\n      TweetBasedUserAdGraphSimilarityEngine.Query,\n      TweetWithScore\n    ](\n      implementingStore = memCachedStore,\n      identifier = SimilarityEngineType.TweetBasedUserAdGraph,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig =\n            Some(DeciderConfig(decider, DeciderConstants.enableUserAdGraphTrafficDeciderKey)),\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/TweetBasedUserTweetGraphSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module\npackage similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.DeciderConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TweetBasedUserTweetGraphSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.recos.user_tweet_graph.thriftscala.UserTweetGraph\nimport com.twitter.relevance_platform.common.injection.LZ4Injection\nimport com.twitter.relevance_platform.common.injection.SeqObjectInjection\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.twistly.thriftscala.TweetRecentEngagedUsers\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject TweetBasedUserTweetGraphSimilarityEngineModule extends TwitterModule {\n\n  private val keyHasher: KeyHasher = KeyHasher.FNV1A_64\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TweetBasedUserTweetGraphSimilarityEngine)\n  def providesTweetBasedUserTweetGraphSimilarityEngine(\n    userTweetGraphService: UserTweetGraph.MethodPerEndpoint,\n    tweetRecentEngagedUserStore: ReadableStore[TweetId, TweetRecentEngagedUsers],\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    decider: CrMixerDecider\n  ): StandardSimilarityEngine[\n    TweetBasedUserTweetGraphSimilarityEngine.Query,\n    TweetWithScore\n  ] = {\n\n    val underlyingStore = TweetBasedUserTweetGraphSimilarityEngine(\n      userTweetGraphService,\n      tweetRecentEngagedUserStore,\n      statsReceiver)\n\n    val memCachedStore: ReadableStore[\n      TweetBasedUserTweetGraphSimilarityEngine.Query,\n      Seq[\n        TweetWithScore\n      ]\n    ] =\n      ObservedMemcachedReadableStore\n        .fromCacheClient(\n          backingStore = underlyingStore,\n          cacheClient = crMixerUnifiedCacheClient,\n          ttl = 10.minutes\n        )(\n          valueInjection = LZ4Injection.compose(SeqObjectInjection[TweetWithScore]()),\n          statsReceiver = statsReceiver.scope(\"tweet_based_user_tweet_graph_store_memcache\"),\n          keyToString = { k =>\n            //Example Query CRMixer:TweetBasedUTG:1234567890ABCDEF\n            f\"CRMixer:TweetBasedUTG:${keyHasher.hashKey(k.toString.getBytes)}%X\"\n          }\n        )\n\n    new StandardSimilarityEngine[\n      TweetBasedUserTweetGraphSimilarityEngine.Query,\n      TweetWithScore\n    ](\n      implementingStore = memCachedStore,\n      identifier = SimilarityEngineType.TweetBasedUserTweetGraph,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig =\n            Some(DeciderConfig(decider, DeciderConstants.enableUserTweetGraphTrafficDeciderKey)),\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/TweetBasedUserVideoGraphSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.DeciderConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TweetBasedUserVideoGraphSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.recos.user_video_graph.thriftscala.UserVideoGraph\nimport com.twitter.relevance_platform.common.injection.LZ4Injection\nimport com.twitter.relevance_platform.common.injection.SeqObjectInjection\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.twistly.thriftscala.TweetRecentEngagedUsers\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject TweetBasedUserVideoGraphSimilarityEngineModule extends TwitterModule {\n\n  private val keyHasher: KeyHasher = KeyHasher.FNV1A_64\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TweetBasedUserVideoGraphSimilarityEngine)\n  def providesTweetBasedUserVideoGraphSimilarityEngine(\n    userVideoGraphService: UserVideoGraph.MethodPerEndpoint,\n    tweetRecentEngagedUserStore: ReadableStore[TweetId, TweetRecentEngagedUsers],\n    @Named(ModuleNames.UnifiedCache) crMixerUnifiedCacheClient: MemcachedClient,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    decider: CrMixerDecider\n  ): StandardSimilarityEngine[\n    TweetBasedUserVideoGraphSimilarityEngine.Query,\n    TweetWithScore\n  ] = {\n\n    val underlyingStore =\n      TweetBasedUserVideoGraphSimilarityEngine(\n        userVideoGraphService,\n        tweetRecentEngagedUserStore,\n        statsReceiver)\n\n    val memCachedStore: ReadableStore[\n      TweetBasedUserVideoGraphSimilarityEngine.Query,\n      Seq[\n        TweetWithScore\n      ]\n    ] =\n      ObservedMemcachedReadableStore\n        .fromCacheClient(\n          backingStore = underlyingStore,\n          cacheClient = crMixerUnifiedCacheClient,\n          ttl = 10.minutes\n        )(\n          valueInjection = LZ4Injection.compose(SeqObjectInjection[TweetWithScore]()),\n          statsReceiver = statsReceiver.scope(\"tweet_based_user_video_graph_store_memcache\"),\n          keyToString = { k =>\n            //Example Query CRMixer:TweetBasedUVG:1234567890ABCDEF\n            f\"CRMixer:TweetBasedUVG:${keyHasher.hashKey(k.toString.getBytes)}%X\"\n          }\n        )\n\n    new StandardSimilarityEngine[\n      TweetBasedUserVideoGraphSimilarityEngine.Query,\n      TweetWithScore\n    ](\n      implementingStore = memCachedStore,\n      identifier = SimilarityEngineType.TweetBasedUserVideoGraph,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig =\n            Some(DeciderConfig(decider, DeciderConstants.enableUserVideoGraphTrafficDeciderKey)),\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/TwhinCollabFilterLookupSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module\npackage similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.similarity_engine.LookupSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.TwhinCollabFilterSimilarityEngine.Query\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.TwhinCollabFilterSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/**\n * TwhinCandidatesLookupSimilarityEngineModule routes the request to the corresponding\n * twhin based candidate store which follow the same pattern as TwHIN Collaborative Filtering.\n */\n\nobject TwhinCollabFilterLookupSimilarityEngineModule extends TwitterModule {\n  @Provides\n  @Singleton\n  @Named(ModuleNames.TwhinCollabFilterSimilarityEngine)\n  def providesTwhinCollabFilterLookupSimilarityEngineModule(\n    @Named(ModuleNames.TwhinCollabFilterStratoStoreForFollow)\n    twhinCollabFilterStratoStoreForFollow: ReadableStore[Long, Seq[TweetId]],\n    @Named(ModuleNames.TwhinCollabFilterStratoStoreForEngagement)\n    twhinCollabFilterStratoStoreForEngagement: ReadableStore[Long, Seq[TweetId]],\n    @Named(ModuleNames.TwhinMultiClusterStratoStoreForFollow)\n    twhinMultiClusterStratoStoreForFollow: ReadableStore[Long, Seq[TweetId]],\n    @Named(ModuleNames.TwhinMultiClusterStratoStoreForEngagement)\n    twhinMultiClusterStratoStoreForEngagement: ReadableStore[Long, Seq[TweetId]],\n    timeoutConfig: TimeoutConfig,\n    globalStats: StatsReceiver\n  ): LookupSimilarityEngine[Query, TweetWithScore] = {\n    val versionedStoreMap = Map(\n      ModelConfig.TwhinCollabFilterForFollow -> TwhinCollabFilterSimilarityEngine(\n        twhinCollabFilterStratoStoreForFollow,\n        globalStats),\n      ModelConfig.TwhinCollabFilterForEngagement -> TwhinCollabFilterSimilarityEngine(\n        twhinCollabFilterStratoStoreForEngagement,\n        globalStats),\n      ModelConfig.TwhinMultiClusterForFollow -> TwhinCollabFilterSimilarityEngine(\n        twhinMultiClusterStratoStoreForFollow,\n        globalStats),\n      ModelConfig.TwhinMultiClusterForEngagement -> TwhinCollabFilterSimilarityEngine(\n        twhinMultiClusterStratoStoreForEngagement,\n        globalStats),\n    )\n\n    new LookupSimilarityEngine[Query, TweetWithScore](\n      versionedStoreMap = versionedStoreMap,\n      identifier = SimilarityEngineType.TwhinCollabFilter,\n      globalStats = globalStats,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.similarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = None,\n          enableFeatureSwitch = None\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/similarity_engine/UserTweetEntityGraphSimilarityEngineModule.scala",
    "content": "package com.twitter.cr_mixer.module.similarity_engine\n\nimport com.google.inject.Provides\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithScoreAndSocialProof\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.similarity_engine.UserTweetEntityGraphSimilarityEngine\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.DeciderConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.GatingConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.similarity_engine.StandardSimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.recos.user_tweet_entity_graph.thriftscala.UserTweetEntityGraph\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject UserTweetEntityGraphSimilarityEngineModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.UserTweetEntityGraphSimilarityEngine)\n  def providesUserTweetEntityGraphSimilarityEngine(\n    userTweetEntityGraphService: UserTweetEntityGraph.MethodPerEndpoint,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    decider: CrMixerDecider\n  ): StandardSimilarityEngine[\n    UserTweetEntityGraphSimilarityEngine.Query,\n    TweetWithScoreAndSocialProof\n  ] = {\n    new StandardSimilarityEngine[\n      UserTweetEntityGraphSimilarityEngine.Query,\n      TweetWithScoreAndSocialProof\n    ](\n      implementingStore =\n        UserTweetEntityGraphSimilarityEngine(userTweetEntityGraphService, statsReceiver),\n      identifier = SimilarityEngineType.Uteg,\n      globalStats = statsReceiver,\n      engineConfig = SimilarityEngineConfig(\n        timeout = timeoutConfig.utegSimilarityEngineTimeout,\n        gatingConfig = GatingConfig(\n          deciderConfig = Some(\n            DeciderConfig(decider, DeciderConstants.enableUserTweetEntityGraphTrafficDeciderKey)),\n          enableFeatureSwitch = None\n        )\n      ),\n      // We cannot use the key to cache anything in UTEG because the key contains a long list of userIds\n      memCacheConfig = None\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/AnnQueryServiceClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\n\nimport com.google.inject.Provides\nimport com.twitter.ann.common.thriftscala.AnnQueryService\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.inject.TwitterModule\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject AnnQueryServiceClientModule extends TwitterModule {\n  final val DebuggerDemoAnnServiceClientName = \"DebuggerDemoAnnServiceClient\"\n\n  @Provides\n  @Singleton\n  @Named(DebuggerDemoAnnServiceClientName)\n  def debuggerDemoAnnServiceClient(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    statsReceiver: StatsReceiver,\n    timeoutConfig: TimeoutConfig,\n  ): AnnQueryService.MethodPerEndpoint = {\n    // This ANN is built from the embeddings in src/scala/com/twitter/wtf/beam/bq_embedding_export/sql/MlfExperimentalTweetEmbeddingScalaDataset.sql\n    // Change the above sql if you want to build the index from a diff embedding\n    val dest = \"/s/cassowary/mlf-experimental-ann-service\"\n    val label = \"experimental-ann\"\n    buildClient(serviceIdentifier, clientId, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  final val TwHINUuaAnnServiceClientName = \"TwHINUuaAnnServiceClient\"\n  @Provides\n  @Singleton\n  @Named(TwHINUuaAnnServiceClientName)\n  def twhinUuaAnnServiceClient(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    statsReceiver: StatsReceiver,\n    timeoutConfig: TimeoutConfig,\n  ): AnnQueryService.MethodPerEndpoint = {\n    val dest = \"/s/cassowary/twhin-uua-ann-service\"\n    val label = \"twhin_uua_ann\"\n\n    buildClient(serviceIdentifier, clientId, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  final val TwHINRegularUpdateAnnServiceClientName = \"TwHINRegularUpdateAnnServiceClient\"\n  @Provides\n  @Singleton\n  @Named(TwHINRegularUpdateAnnServiceClientName)\n  def twHINRegularUpdateAnnServiceClient(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    statsReceiver: StatsReceiver,\n    timeoutConfig: TimeoutConfig,\n  ): AnnQueryService.MethodPerEndpoint = {\n    val dest = \"/s/cassowary/twhin-regular-update-ann-service\"\n    val label = \"twhin_regular_update\"\n\n    buildClient(serviceIdentifier, clientId, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  final val TwoTowerFavAnnServiceClientName = \"TwoTowerFavAnnServiceClient\"\n  @Provides\n  @Singleton\n  @Named(TwoTowerFavAnnServiceClientName)\n  def twoTowerFavAnnServiceClient(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    statsReceiver: StatsReceiver,\n    timeoutConfig: TimeoutConfig,\n  ): AnnQueryService.MethodPerEndpoint = {\n    val dest = \"/s/cassowary/tweet-rec-two-tower-fav-ann\"\n    val label = \"tweet_rec_two_tower_fav_ann\"\n\n    buildClient(serviceIdentifier, clientId, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  private def buildClient(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    dest: String,\n    label: String\n  ): AnnQueryService.MethodPerEndpoint = {\n    val thriftClient = ThriftMux.client\n      .withMutualTls(serviceIdentifier)\n      .withClientId(clientId)\n      .withLabel(label)\n      .withStatsReceiver(statsReceiver)\n      .withTransport.connectTimeout(500.milliseconds)\n      .withSession.acquisitionTimeout(500.milliseconds)\n      .methodBuilder(dest)\n      .withTimeoutPerRequest(timeoutConfig.annServiceClientTimeout)\n      .withRetryDisabled\n      .idempotent(5.percent)\n      .servicePerEndpoint[AnnQueryService.ServicePerEndpoint]\n\n    ThriftMux.Client.methodPerEndpoint(thriftClient)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/EarlybirdSearchClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\nimport com.twitter.app.Flag\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.search.earlybird.thriftscala.EarlybirdService\nimport com.twitter.inject.Injector\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.module.core.TimeoutConfigModule.EarlybirdClientTimeoutFlagName\nimport com.twitter.finagle.service.RetryBudget\nimport com.twitter.util.Duration\nimport org.apache.thrift.protocol.TCompactProtocol\n\nobject EarlybirdSearchClientModule\n    extends ThriftMethodBuilderClientModule[\n      EarlybirdService.ServicePerEndpoint,\n      EarlybirdService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override def label: String = \"earlybird\"\n  override def dest: String = \"/s/earlybird-root-superroot/root-superroot\"\n  private val requestTimeoutFlag: Flag[Duration] =\n    flag[Duration](EarlybirdClientTimeoutFlagName, \"Earlybird client timeout\")\n  override protected def requestTimeout: Duration = requestTimeoutFlag()\n\n  override def retryBudget: RetryBudget = RetryBudget.Empty\n\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client = {\n    super\n      .configureThriftMuxClient(injector, client)\n      .withProtocolFactory(new TCompactProtocol.Factory())\n      .withSessionQualifier\n      .successRateFailureAccrual(successRate = 0.9, window = 30.seconds)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/FrsClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\n\nimport com.twitter.app.Flag\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.module.core.TimeoutConfigModule.FrsClientTimeoutFlagName\nimport com.twitter.finagle.service.RetryBudget\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.follow_recommendations.thriftscala.FollowRecommendationsThriftService\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.util.Duration\n\nobject FrsClientModule\n    extends ThriftMethodBuilderClientModule[\n      FollowRecommendationsThriftService.ServicePerEndpoint,\n      FollowRecommendationsThriftService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override def label: String = \"follow-recommendations-service\"\n  override def dest: String = \"/s/follow-recommendations/follow-recos-service\"\n\n  private val frsSignalFetchTimeout: Flag[Duration] =\n    flag[Duration](FrsClientTimeoutFlagName, \"FRS signal fetch client timeout\")\n  override def requestTimeout: Duration = frsSignalFetchTimeout()\n\n  override def retryBudget: RetryBudget = RetryBudget.Empty\n\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client = {\n    super\n      .configureThriftMuxClient(injector, client)\n      .withStatsReceiver(injector.instance[StatsReceiver].scope(\"clnt\"))\n      .withSessionQualifier\n      .successRateFailureAccrual(successRate = 0.9, window = 30.seconds)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/HydraPartitionClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.hydra.partition.{thriftscala => ht}\n\nobject HydraPartitionClientModule\n    extends ThriftMethodBuilderClientModule[\n      ht.HydraPartition.ServicePerEndpoint,\n      ht.HydraPartition.MethodPerEndpoint\n    ]\n    with MtlsClient {\n  override def label: String = \"hydra-partition\"\n\n  override def dest: String = \"/s/hydra/hydra-partition\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = methodBuilder.withTimeoutTotal(500.milliseconds)\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/HydraRootClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.hydra.root.{thriftscala => ht}\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\n\nobject HydraRootClientModule\n    extends ThriftMethodBuilderClientModule[\n      ht.HydraRoot.ServicePerEndpoint,\n      ht.HydraRoot.MethodPerEndpoint\n    ]\n    with MtlsClient {\n  override def label: String = \"hydra-root\"\n\n  override def dest: String = \"/s/hydra/hydra-root\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = methodBuilder.withTimeoutTotal(500.milliseconds)\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/QigServiceClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\n\nimport com.twitter.app.Flag\nimport com.twitter.cr_mixer.module.core.TimeoutConfigModule.QigRankerClientTimeoutFlagName\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mux.ClientDiscardedRequestException\nimport com.twitter.finagle.service.ReqRep\nimport com.twitter.finagle.service.ResponseClass\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.qig_ranker.thriftscala.QigRanker\nimport com.twitter.util.Duration\nimport com.twitter.util.Throw\n\nobject QigServiceClientModule\n    extends ThriftMethodBuilderClientModule[\n      QigRanker.ServicePerEndpoint,\n      QigRanker.MethodPerEndpoint\n    ]\n    with MtlsClient {\n  override val label: String = \"qig-ranker\"\n  override val dest: String = \"/s/qig-shared/qig-ranker\"\n  private val qigRankerClientTimeout: Flag[Duration] =\n    flag[Duration](QigRankerClientTimeoutFlagName, \"ranking timeout\")\n\n  override def requestTimeout: Duration = qigRankerClientTimeout()\n\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client =\n    super\n      .configureThriftMuxClient(injector, client)\n      .withStatsReceiver(injector.instance[StatsReceiver].scope(\"clnt\"))\n      .withResponseClassifier {\n        case ReqRep(_, Throw(_: ClientDiscardedRequestException)) => ResponseClass.Ignorable\n      }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/SimClustersAnnServiceClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclustersann.{thriftscala => t}\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject SimClustersAnnServiceClientModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.ProdSimClustersANNServiceClientName)\n  def providesProdSimClustersANNServiceClient(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): t.SimClustersANNService.MethodPerEndpoint = {\n    val label = \"simclusters-ann-server\"\n    val dest = \"/s/simclusters-ann/simclusters-ann\"\n\n    buildClient(serviceIdentifier, clientId, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.ExperimentalSimClustersANNServiceClientName)\n  def providesExperimentalSimClustersANNServiceClient(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): t.SimClustersANNService.MethodPerEndpoint = {\n    val label = \"simclusters-ann-experimental-server\"\n    val dest = \"/s/simclusters-ann/simclusters-ann-experimental\"\n\n    buildClient(serviceIdentifier, clientId, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.SimClustersANNServiceClientName1)\n  def providesSimClustersANNServiceClient1(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): t.SimClustersANNService.MethodPerEndpoint = {\n    val label = \"simclusters-ann-server-1\"\n    val dest = \"/s/simclusters-ann/simclusters-ann-1\"\n\n    buildClient(serviceIdentifier, clientId, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.SimClustersANNServiceClientName2)\n  def providesSimClustersANNServiceClient2(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): t.SimClustersANNService.MethodPerEndpoint = {\n    val label = \"simclusters-ann-server-2\"\n    val dest = \"/s/simclusters-ann/simclusters-ann-2\"\n\n    buildClient(serviceIdentifier, clientId, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.SimClustersANNServiceClientName3)\n  def providesSimClustersANNServiceClient3(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): t.SimClustersANNService.MethodPerEndpoint = {\n    val label = \"simclusters-ann-server-3\"\n    val dest = \"/s/simclusters-ann/simclusters-ann-3\"\n\n    buildClient(serviceIdentifier, clientId, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.SimClustersANNServiceClientName5)\n  def providesSimClustersANNServiceClient5(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): t.SimClustersANNService.MethodPerEndpoint = {\n    val label = \"simclusters-ann-server-5\"\n    val dest = \"/s/simclusters-ann/simclusters-ann-5\"\n\n    buildClient(serviceIdentifier, clientId, timeoutConfig, statsReceiver, dest, label)\n  }\n\n  @Provides\n  @Singleton\n  @Named(ModuleNames.SimClustersANNServiceClientName4)\n  def providesSimClustersANNServiceClient4(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n  ): t.SimClustersANNService.MethodPerEndpoint = {\n    val label = \"simclusters-ann-server-4\"\n    val dest = \"/s/simclusters-ann/simclusters-ann-4\"\n\n    buildClient(serviceIdentifier, clientId, timeoutConfig, statsReceiver, dest, label)\n  }\n  private def buildClient(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    timeoutConfig: TimeoutConfig,\n    statsReceiver: StatsReceiver,\n    dest: String,\n    label: String\n  ): t.SimClustersANNService.MethodPerEndpoint = {\n    val stats = statsReceiver.scope(\"clnt\")\n\n    val thriftClient = ThriftMux.client\n      .withMutualTls(serviceIdentifier)\n      .withClientId(clientId)\n      .withLabel(label)\n      .withStatsReceiver(stats)\n      .methodBuilder(dest)\n      .idempotent(5.percent)\n      .withTimeoutPerRequest(timeoutConfig.annServiceClientTimeout)\n      .withRetryDisabled\n      .servicePerEndpoint[t.SimClustersANNService.ServicePerEndpoint]\n\n    ThriftMux.Client.methodPerEndpoint(thriftClient)\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/TweetyPieClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\n\nimport com.google.inject.Provides\nimport com.twitter.app.Flag\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.cr_mixer.module.core.TimeoutConfigModule.TweetypieClientTimeoutFlagName\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mux.ClientDiscardedRequestException\nimport com.twitter.finagle.service.ReqRep\nimport com.twitter.finagle.service.ResponseClass\nimport com.twitter.finagle.service.RetryBudget\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.stitch.tweetypie.{TweetyPie => STweetyPie}\nimport com.twitter.tweetypie.thriftscala.TweetService\nimport com.twitter.util.Duration\nimport com.twitter.util.Throw\nimport javax.inject.Singleton\n\nobject TweetyPieClientModule\n    extends ThriftMethodBuilderClientModule[\n      TweetService.ServicePerEndpoint,\n      TweetService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"tweetypie\"\n  override val dest = \"/s/tweetypie/tweetypie\"\n\n  private val tweetypieClientTimeout: Flag[Duration] =\n    flag[Duration](TweetypieClientTimeoutFlagName, \"tweetypie client timeout\")\n  override def requestTimeout: Duration = tweetypieClientTimeout()\n\n  override def retryBudget: RetryBudget = RetryBudget.Empty\n\n  // We bump the success rate from the default of 0.8 to 0.9 since we're dropping the\n  // consecutive failures part of the default policy.\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client =\n    super\n      .configureThriftMuxClient(injector, client)\n      .withStatsReceiver(injector.instance[StatsReceiver].scope(\"clnt\"))\n      .withSessionQualifier\n      .successRateFailureAccrual(successRate = 0.9, window = 30.seconds)\n      .withResponseClassifier {\n        case ReqRep(_, Throw(_: ClientDiscardedRequestException)) => ResponseClass.Ignorable\n      }\n\n  @Provides\n  @Singleton\n  def providesTweetyPie(\n    tweetyPieService: TweetService.MethodPerEndpoint\n  ): STweetyPie = {\n    STweetyPie(tweetyPieService)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/UserAdGraphClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\n\nimport com.twitter.app.Flag\nimport com.twitter.cr_mixer.module.core.TimeoutConfigModule.UserAdGraphClientTimeoutFlagName\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient.MtlsThriftMuxClientSyntax\nimport com.twitter.finagle.mux.ClientDiscardedRequestException\nimport com.twitter.finagle.service.ReqRep\nimport com.twitter.finagle.service.ResponseClass\nimport com.twitter.finagle.service.RetryBudget\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.recos.user_ad_graph.thriftscala.UserAdGraph\nimport com.twitter.util.Duration\nimport com.twitter.util.Throw\n\nobject UserAdGraphClientModule\n    extends ThriftMethodBuilderClientModule[\n      UserAdGraph.ServicePerEndpoint,\n      UserAdGraph.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"user-ad-graph\"\n  override val dest = \"/s/user-tweet-graph/user-ad-graph\"\n  private val userAdGraphClientTimeout: Flag[Duration] =\n    flag[Duration](UserAdGraphClientTimeoutFlagName, \"userAdGraph client timeout\")\n  override def requestTimeout: Duration = userAdGraphClientTimeout()\n\n  override def retryBudget: RetryBudget = RetryBudget.Empty\n\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client =\n    super\n      .configureThriftMuxClient(injector, client)\n      .withMutualTls(injector.instance[ServiceIdentifier])\n      .withStatsReceiver(injector.instance[StatsReceiver].scope(\"clnt\"))\n      .withResponseClassifier {\n        case ReqRep(_, Throw(_: ClientDiscardedRequestException)) => ResponseClass.Ignorable\n      }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/UserTweetEntityGraphClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\n\nimport com.twitter.app.Flag\nimport com.twitter.cr_mixer.module.core.TimeoutConfigModule.UtegClientTimeoutFlagName\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mux.ClientDiscardedRequestException\nimport com.twitter.finagle.service.ReqRep\nimport com.twitter.finagle.service.ResponseClass\nimport com.twitter.finagle.service.RetryBudget\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.recos.user_tweet_entity_graph.thriftscala.UserTweetEntityGraph\nimport com.twitter.util.Duration\nimport com.twitter.util.Throw\n\nobject UserTweetEntityGraphClientModule\n    extends ThriftMethodBuilderClientModule[\n      UserTweetEntityGraph.ServicePerEndpoint,\n      UserTweetEntityGraph.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"user-tweet-entity-graph\"\n  override val dest = \"/s/cassowary/user_tweet_entity_graph\"\n  private val userTweetEntityGraphClientTimeout: Flag[Duration] =\n    flag[Duration](UtegClientTimeoutFlagName, \"user tweet entity graph client timeout\")\n  override def requestTimeout: Duration = userTweetEntityGraphClientTimeout()\n\n  override def retryBudget: RetryBudget = RetryBudget.Empty\n\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client =\n    super\n      .configureThriftMuxClient(injector, client)\n      .withStatsReceiver(injector.instance[StatsReceiver].scope(\"clnt\"))\n      .withResponseClassifier {\n        case ReqRep(_, Throw(_: ClientDiscardedRequestException)) => ResponseClass.Ignorable\n      }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/UserTweetGraphClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\n\nimport com.twitter.app.Flag\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mux.ClientDiscardedRequestException\nimport com.twitter.finagle.service.ReqRep\nimport com.twitter.finagle.service.ResponseClass\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.recos.user_tweet_graph.thriftscala.UserTweetGraph\nimport com.twitter.util.Duration\nimport com.twitter.util.Throw\nimport com.twitter.cr_mixer.module.core.TimeoutConfigModule.UserTweetGraphClientTimeoutFlagName\nimport com.twitter.finagle.service.RetryBudget\n\nobject UserTweetGraphClientModule\n    extends ThriftMethodBuilderClientModule[\n      UserTweetGraph.ServicePerEndpoint,\n      UserTweetGraph.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"user-tweet-graph\"\n  override val dest = \"/s/user-tweet-graph/user-tweet-graph\"\n  private val userTweetGraphClientTimeout: Flag[Duration] =\n    flag[Duration](UserTweetGraphClientTimeoutFlagName, \"userTweetGraph client timeout\")\n  override def requestTimeout: Duration = userTweetGraphClientTimeout()\n\n  override def retryBudget: RetryBudget = RetryBudget.Empty\n\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client =\n    super\n      .configureThriftMuxClient(injector, client)\n      .withStatsReceiver(injector.instance[StatsReceiver].scope(\"clnt\"))\n      .withResponseClassifier {\n        case ReqRep(_, Throw(_: ClientDiscardedRequestException)) => ResponseClass.Ignorable\n      }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/UserTweetGraphPlusClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\n\nimport com.twitter.app.Flag\nimport com.twitter.cr_mixer.module.core.TimeoutConfigModule.UserTweetGraphPlusClientTimeoutFlagName\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mux.ClientDiscardedRequestException\nimport com.twitter.finagle.service.ReqRep\nimport com.twitter.finagle.service.ResponseClass\nimport com.twitter.finagle.service.RetryBudget\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.recos.user_tweet_graph_plus.thriftscala.UserTweetGraphPlus\nimport com.twitter.util.Duration\nimport com.twitter.util.Throw\n\nobject UserTweetGraphPlusClientModule\n    extends ThriftMethodBuilderClientModule[\n      UserTweetGraphPlus.ServicePerEndpoint,\n      UserTweetGraphPlus.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"user-tweet-graph-plus\"\n  override val dest = \"/s/user-tweet-graph/user-tweet-graph-plus\"\n  private val userTweetGraphPlusClientTimeout: Flag[Duration] =\n    flag[Duration](\n      UserTweetGraphPlusClientTimeoutFlagName,\n      \"userTweetGraphPlus client timeout\"\n    )\n  override def requestTimeout: Duration = userTweetGraphPlusClientTimeout()\n\n  override def retryBudget: RetryBudget = RetryBudget.Empty\n\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client =\n    super\n      .configureThriftMuxClient(injector, client)\n      .withStatsReceiver(injector.instance[StatsReceiver].scope(\"clnt\"))\n      .withResponseClassifier {\n        case ReqRep(_, Throw(_: ClientDiscardedRequestException)) => ResponseClass.Ignorable\n      }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/module/thrift_client/UserVideoGraphClientModule.scala",
    "content": "package com.twitter.cr_mixer.module.thrift_client\n\nimport com.twitter.app.Flag\nimport com.twitter.cr_mixer.module.core.TimeoutConfigModule.UserVideoGraphClientTimeoutFlagName\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mux.ClientDiscardedRequestException\nimport com.twitter.finagle.service.ReqRep\nimport com.twitter.finagle.service.ResponseClass\nimport com.twitter.finagle.service.RetryBudget\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.recos.user_video_graph.thriftscala.UserVideoGraph\nimport com.twitter.util.Duration\nimport com.twitter.util.Throw\n\nobject UserVideoGraphClientModule\n    extends ThriftMethodBuilderClientModule[\n      UserVideoGraph.ServicePerEndpoint,\n      UserVideoGraph.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"user-video-graph\"\n  override val dest = \"/s/user-tweet-graph/user-video-graph\"\n  private val userVideoGraphClientTimeout: Flag[Duration] =\n    flag[Duration](\n      UserVideoGraphClientTimeoutFlagName,\n      \"userVideoGraph client timeout\"\n    )\n  override def requestTimeout: Duration = userVideoGraphClientTimeout()\n\n  override def retryBudget: RetryBudget = RetryBudget.Empty\n\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client =\n    super\n      .configureThriftMuxClient(injector, client)\n      .withStatsReceiver(injector.instance[StatsReceiver].scope(\"clnt\"))\n      .withResponseClassifier {\n        case ReqRep(_, Throw(_: ClientDiscardedRequestException)) => ResponseClass.Ignorable\n      }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/AdsParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject AdsParams {\n  object AdsCandidateGenerationMaxCandidatesNumParam\n      extends FSBoundedParam[Int](\n        name = \"ads_candidate_generation_max_candidates_num\",\n        default = 400,\n        min = 0,\n        max = 2000\n      )\n\n  object EnableScoreBoost\n      extends FSParam[Boolean](\n        name = \"ads_candidate_generation_enable_score_boost\",\n        default = false\n      )\n\n  object AdsCandidateGenerationScoreBoostFactor\n      extends FSBoundedParam[Double](\n        name = \"ads_candidate_generation_score_boost_factor\",\n        default = 10000.0,\n        min = 1.0,\n        max = 100000.0\n      )\n\n  object EnableScribe\n      extends FSParam[Boolean](\n        name = \"ads_candidate_generation_enable_scribe\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    AdsCandidateGenerationMaxCandidatesNumParam,\n    EnableScoreBoost,\n    AdsCandidateGenerationScoreBoostFactor\n  )\n\n  lazy val config: BaseConfig = {\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      AdsCandidateGenerationMaxCandidatesNumParam)\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableScoreBoost,\n      EnableScribe\n    )\n\n    val doubleOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(AdsCandidateGenerationScoreBoostFactor)\n\n    BaseConfigBuilder()\n      .set(intOverrides: _*)\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"abdecider/src/main/scala\",\n        \"configapi/configapi-abdecider\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-featureswitches:v2\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"decider/src/main/scala\",\n        \"discovery-common/src/main/scala/com/twitter/discovery/common/configapi\",\n        \"featureswitches/featureswitches-core\",\n        \"featureswitches/featureswitches-core/src/main/scala/com/twitter/featureswitches/v2/builder\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"scribelib/marshallers/src/main/scala/com/twitter/scribelib/marshallers\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"user-signal-service/thrift/src/main/thrift:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/BlenderParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject BlenderParams {\n  object BlendingAlgorithmEnum extends Enumeration {\n    val RoundRobin: Value = Value\n    val SourceTypeBackFill: Value = Value\n    val SourceSignalSorting: Value = Value\n  }\n  object ContentBasedSortingAlgorithmEnum extends Enumeration {\n    val FavoriteCount: Value = Value\n    val SourceSignalRecency: Value = Value\n    val RandomSorting: Value = Value\n    val SimilarityToSignalSorting: Value = Value\n    val CandidateRecency: Value = Value\n  }\n\n  object BlendingAlgorithmParam\n      extends FSEnumParam[BlendingAlgorithmEnum.type](\n        name = \"blending_algorithm_id\",\n        default = BlendingAlgorithmEnum.RoundRobin,\n        enum = BlendingAlgorithmEnum\n      )\n\n  object RankingInterleaveWeightShrinkageParam\n      extends FSBoundedParam[Double](\n        name = \"blending_enable_ml_ranking_interleave_weights_shrinkage\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object RankingInterleaveMaxWeightAdjustments\n      extends FSBoundedParam[Int](\n        name = \"blending_interleave_max_weighted_adjustments\",\n        default = 3000,\n        min = 0,\n        max = 9999\n      )\n\n  object SignalTypeSortingAlgorithmParam\n      extends FSEnumParam[ContentBasedSortingAlgorithmEnum.type](\n        name = \"blending_algorithm_inner_signal_sorting_id\",\n        default = ContentBasedSortingAlgorithmEnum.SourceSignalRecency,\n        enum = ContentBasedSortingAlgorithmEnum\n      )\n\n  object ContentBlenderTypeSortingAlgorithmParam\n      extends FSEnumParam[ContentBasedSortingAlgorithmEnum.type](\n        name = \"blending_algorithm_content_blender_sorting_id\",\n        default = ContentBasedSortingAlgorithmEnum.FavoriteCount,\n        enum = ContentBasedSortingAlgorithmEnum\n      )\n\n  //UserAffinities Algo Param: whether to distributed the source type weights\n  object EnableDistributedSourceTypeWeightsParam\n      extends FSParam[Boolean](\n        name = \"blending_algorithm_enable_distributed_source_type_weights\",\n        default = false\n      )\n\n  object BlendGroupingMethodEnum extends Enumeration {\n    val SourceKeyDefault: Value = Value(\"SourceKey\")\n    val SourceTypeSimilarityEngine: Value = Value(\"SourceTypeSimilarityEngine\")\n    val AuthorId: Value = Value(\"AuthorId\")\n  }\n\n  object BlendGroupingMethodParam\n      extends FSEnumParam[BlendGroupingMethodEnum.type](\n        name = \"blending_grouping_method_id\",\n        default = BlendGroupingMethodEnum.SourceKeyDefault,\n        enum = BlendGroupingMethodEnum\n      )\n\n  object RecencyBasedRandomSamplingHalfLifeInDays\n      extends FSBoundedParam[Int](\n        name = \"blending_interleave_random_sampling_recency_based_half_life_in_days\",\n        default = 7,\n        min = 1,\n        max = 28\n      )\n\n  object RecencyBasedRandomSamplingDefaultWeight\n      extends FSBoundedParam[Double](\n        name = \"blending_interleave_random_sampling_recency_based_default_weight\",\n        default = 1.0,\n        min = 0.1,\n        max = 2.0\n      )\n\n  object SourceTypeBackFillEnableVideoBackFill\n      extends FSParam[Boolean](\n        name = \"blending_enable_video_backfill\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    BlendingAlgorithmParam,\n    RankingInterleaveWeightShrinkageParam,\n    RankingInterleaveMaxWeightAdjustments,\n    EnableDistributedSourceTypeWeightsParam,\n    BlendGroupingMethodParam,\n    RecencyBasedRandomSamplingHalfLifeInDays,\n    RecencyBasedRandomSamplingDefaultWeight,\n    SourceTypeBackFillEnableVideoBackFill,\n    SignalTypeSortingAlgorithmParam,\n    ContentBlenderTypeSortingAlgorithmParam,\n  )\n\n  lazy val config: BaseConfig = {\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n      BlendingAlgorithmParam,\n      BlendGroupingMethodParam,\n      SignalTypeSortingAlgorithmParam,\n      ContentBlenderTypeSortingAlgorithmParam\n    )\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableDistributedSourceTypeWeightsParam,\n      SourceTypeBackFillEnableVideoBackFill\n    )\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      RankingInterleaveMaxWeightAdjustments,\n      RecencyBasedRandomSamplingHalfLifeInDays\n    )\n\n    val doubleOverrides = FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(\n      RankingInterleaveWeightShrinkageParam,\n      RecencyBasedRandomSamplingDefaultWeight\n    )\n\n    BaseConfigBuilder()\n      .set(enumOverrides: _*)\n      .set(booleanOverrides: _*)\n      .set(intOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/BypassInterleaveAndRankParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject BypassInterleaveAndRankParams {\n  object EnableTwhinCollabFilterBypassParam\n      extends FSParam[Boolean](\n        name = \"bypass_interleave_and_rank_twhin_collab_filter\",\n        default = false\n      )\n\n  object EnableTwoTowerBypassParam\n      extends FSParam[Boolean](\n        name = \"bypass_interleave_and_rank_two_tower\",\n        default = false\n      )\n\n  object EnableConsumerBasedTwhinBypassParam\n      extends FSParam[Boolean](\n        name = \"bypass_interleave_and_rank_consumer_based_twhin\",\n        default = false\n      )\n\n  object EnableConsumerBasedWalsBypassParam\n      extends FSParam[Boolean](\n        name = \"bypass_interleave_and_rank_consumer_based_wals\",\n        default = false\n      )\n\n  object TwhinCollabFilterBypassPercentageParam\n      extends FSBoundedParam[Double](\n        name = \"bypass_interleave_and_rank_twhin_collab_filter_percentage\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object TwoTowerBypassPercentageParam\n      extends FSBoundedParam[Double](\n        name = \"bypass_interleave_and_rank_two_tower_percentage\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object ConsumerBasedTwhinBypassPercentageParam\n      extends FSBoundedParam[Double](\n        name = \"bypass_interleave_and_rank_consumer_based_twhin_percentage\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object ConsumerBasedWalsBypassPercentageParam\n      extends FSBoundedParam[Double](\n        name = \"bypass_interleave_and_rank_consumer_based_wals_percentage\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableTwhinCollabFilterBypassParam,\n    EnableTwoTowerBypassParam,\n    EnableConsumerBasedTwhinBypassParam,\n    EnableConsumerBasedWalsBypassParam,\n    TwhinCollabFilterBypassPercentageParam,\n    TwoTowerBypassPercentageParam,\n    ConsumerBasedTwhinBypassPercentageParam,\n    ConsumerBasedWalsBypassPercentageParam,\n  )\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableTwhinCollabFilterBypassParam,\n      EnableTwoTowerBypassParam,\n      EnableConsumerBasedTwhinBypassParam,\n      EnableConsumerBasedWalsBypassParam,\n    )\n\n    val doubleOverrides = FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(\n      TwhinCollabFilterBypassPercentageParam,\n      TwoTowerBypassPercentageParam,\n      ConsumerBasedTwhinBypassPercentageParam,\n      ConsumerBasedWalsBypassPercentageParam,\n    )\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/ConsumerBasedWalsParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\nobject ConsumerBasedWalsParams {\n\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"consumer_based_wals_enable_source\",\n        default = false\n      )\n\n  object ModelNameParam\n      extends FSParam[String](\n        name = \"consumer_based_wals_model_name\",\n        default = \"model_0\"\n      )\n\n  object WilyNsNameParam\n      extends FSParam[String](\n        name = \"consumer_based_wals_wily_ns_name\",\n        default = \"\"\n      )\n\n  object ModelInputNameParam\n      extends FSParam[String](\n        name = \"consumer_based_wals_model_input_name\",\n        default = \"examples\"\n      )\n\n  object ModelOutputNameParam\n      extends FSParam[String](\n        name = \"consumer_based_wals_model_output_name\",\n        default = \"all_tweet_ids\"\n      )\n\n  object ModelSignatureNameParam\n      extends FSParam[String](\n        name = \"consumer_based_wals_model_signature_name\",\n        default = \"serving_default\"\n      )\n\n  object MaxTweetSignalAgeHoursParam\n      extends FSBoundedParam[Duration](\n        name = \"consumer_based_wals_max_tweet_signal_age_hours\",\n        default = 72.hours,\n        min = 1.hours,\n        max = 720.hours\n      )\n      with HasDurationConversion {\n\n    override val durationConversion: DurationConversion = DurationConversion.FromHours\n  }\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableSourceParam,\n    ModelNameParam,\n    ModelInputNameParam,\n    ModelOutputNameParam,\n    ModelSignatureNameParam,\n    MaxTweetSignalAgeHoursParam,\n    WilyNsNameParam,\n  )\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam,\n    )\n    val stringOverrides = FeatureSwitchOverrideUtil.getStringFSOverrides(\n      ModelNameParam,\n      ModelInputNameParam,\n      ModelOutputNameParam,\n      ModelSignatureNameParam,\n      WilyNsNameParam\n    )\n\n    val boundedDurationFSOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDurationFSOverrides(MaxTweetSignalAgeHoursParam)\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(stringOverrides: _*)\n      .set(boundedDurationFSOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/ConsumerEmbeddingBasedCandidateGenerationParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject ConsumerEmbeddingBasedCandidateGenerationParams {\n\n  object EnableTwHINParam\n      extends FSParam[Boolean](\n        name = \"consumer_embedding_based_candidate_generation_enable_twhin\",\n        default = false\n      )\n\n  object EnableTwoTowerParam\n      extends FSParam[Boolean](\n        name = \"consumer_embedding_based_candidate_generation_enable_two_tower\",\n        default = false\n      )\n\n  object EnableLogFavBasedSimClustersTripParam\n      extends FSParam[Boolean](\n        name = \"consumer_embedding_based_candidate_generation_enable_logfav_based_simclusters_trip\",\n        default = false\n      )\n\n  object EnableFollowBasedSimClustersTripParam\n      extends FSParam[Boolean](\n        name = \"consumer_embedding_based_candidate_generation_enable_follow_based_simclusters_trip\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableTwHINParam,\n    EnableTwoTowerParam,\n    EnableFollowBasedSimClustersTripParam,\n    EnableLogFavBasedSimClustersTripParam\n  )\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableTwHINParam,\n      EnableTwoTowerParam,\n      EnableFollowBasedSimClustersTripParam,\n      EnableLogFavBasedSimClustersTripParam\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/ConsumerEmbeddingBasedTripParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject ConsumerEmbeddingBasedTripParams {\n  object SourceIdParam\n      extends FSParam[String](\n        name = \"consumer_embedding_based_trip_source_id\",\n        default = \"EXPLR_TOPK_VID_48H_V3\")\n\n  object MaxNumCandidatesParam\n      extends FSBoundedParam[Int](\n        name = \"consumer_embedding_based_trip_max_num_candidates\",\n        default = 80,\n        min = 0,\n        max = 200\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    SourceIdParam,\n    MaxNumCandidatesParam\n  )\n\n  lazy val config: BaseConfig = {\n    val stringFSOverrides =\n      FeatureSwitchOverrideUtil.getStringFSOverrides(\n        SourceIdParam\n      )\n\n    val intFSOverrides =\n      FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n        MaxNumCandidatesParam\n      )\n\n    BaseConfigBuilder()\n      .set(stringFSOverrides: _*)\n      .set(intFSOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/ConsumerEmbeddingBasedTwHINParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\n\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\n\nobject ConsumerEmbeddingBasedTwHINParams {\n  object ModelIdParam\n      extends FSParam[String](\n        name = \"consumer_embedding_based_twhin_model_id\",\n        default = ModelConfig.ConsumerBasedTwHINRegularUpdateAll20221024,\n      ) // Note: this default value does not match with ModelIds yet. This FS is a placeholder\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    ModelIdParam\n  )\n\n  lazy val config: BaseConfig = {\n    val stringFSOverrides =\n      FeatureSwitchOverrideUtil.getStringFSOverrides(\n        ModelIdParam\n      )\n\n    BaseConfigBuilder()\n      .set(stringFSOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/ConsumerEmbeddingBasedTwoTowerParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.cr_mixer.model.ModelConfig.TwoTowerFavALL20220808\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject ConsumerEmbeddingBasedTwoTowerParams {\n  object ModelIdParam\n      extends FSParam[String](\n        name = \"consumer_embedding_based_two_tower_model_id\",\n        default = TwoTowerFavALL20220808,\n      ) // Note: this default value does not match with ModelIds yet. This FS is a placeholder\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    ModelIdParam\n  )\n\n  lazy val config: BaseConfig = {\n    val stringFSOverrides =\n      FeatureSwitchOverrideUtil.getStringFSOverrides(\n        ModelIdParam\n      )\n\n    BaseConfigBuilder()\n      .set(stringFSOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/ConsumersBasedUserAdGraphParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject ConsumersBasedUserAdGraphParams {\n\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"consumers_based_user_ad_graph_enable_source\",\n        default = false\n      )\n\n  // UTG-Lookalike\n  object MinCoOccurrenceParam\n      extends FSBoundedParam[Int](\n        name = \"consumers_based_user_ad_graph_min_co_occurrence\",\n        default = 2,\n        min = 0,\n        max = 500\n      )\n\n  object MinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"consumers_based_user_ad_graph_min_score\",\n        default = 0.0,\n        min = 0.0,\n        max = 10.0\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableSourceParam,\n    MinCoOccurrenceParam,\n    MinScoreParam\n  )\n\n  lazy val config: BaseConfig = {\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(MinCoOccurrenceParam)\n    val doubleOverrides = FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(MinScoreParam)\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(EnableSourceParam)\n\n    BaseConfigBuilder()\n      .set(intOverrides: _*)\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/ConsumersBasedUserTweetGraphParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\n/**\n * ConsumersBasedUserTweetGraph Params, there are multiple ways (e.g. FRS, RealGraphOon) to generate consumersSeedSet for ConsumersBasedUserTweetGraph\n * for now we allow flexibility in tuning UTG params for different consumersSeedSet generation algo by giving the param name {consumerSeedSetAlgo}{ParamName}\n */\n\nobject ConsumersBasedUserTweetGraphParams {\n\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"consumers_based_user_tweet_graph_enable_source\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableSourceParam,\n  )\n\n  lazy val config: BaseConfig = {\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides()\n\n    val doubleOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides()\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam\n    )\n\n    BaseConfigBuilder()\n      .set(intOverrides: _*)\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/ConsumersBasedUserVideoGraphParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\n/**\n * ConsumersBasedUserVideoGraph Params: there are multiple ways (e.g. FRS, RealGraphIn) to generate consumersSeedSet for ConsumersBasedUserTweetGraph\n * for now we allow flexibility in tuning UVG params for different consumersSeedSet generation algo by giving the param name {consumerSeedSetAlgo}{ParamName}\n */\n\nobject ConsumersBasedUserVideoGraphParams {\n\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"consumers_based_user_video_graph_enable_source\",\n        default = false\n      )\n\n  // UTG-RealGraphIN\n  object RealGraphInMinCoOccurrenceParam\n      extends FSBoundedParam[Int](\n        name = \"consumers_based_user_video_graph_real_graph_in_min_co_occurrence\",\n        default = 3,\n        min = 0,\n        max = 500\n      )\n\n  object RealGraphInMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"consumers_based_user_video_graph_real_graph_in_min_score\",\n        default = 2.0,\n        min = 0.0,\n        max = 10.0\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableSourceParam,\n    RealGraphInMinCoOccurrenceParam,\n    RealGraphInMinScoreParam\n  )\n\n  lazy val config: BaseConfig = {\n\n    val intOverrides =\n      FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(RealGraphInMinCoOccurrenceParam)\n\n    val doubleOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(RealGraphInMinScoreParam)\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam\n    )\n\n    BaseConfigBuilder()\n      .set(intOverrides: _*)\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/CrMixerParamConfig.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.CompositeConfig\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.Param\n\nobject CrMixerParamConfig {\n\n  lazy val config: CompositeConfig = new CompositeConfig(\n    configs = Seq(\n      AdsParams.config,\n      BlenderParams.config,\n      BypassInterleaveAndRankParams.config,\n      RankerParams.config,\n      ConsumerBasedWalsParams.config,\n      ConsumerEmbeddingBasedCandidateGenerationParams.config,\n      ConsumerEmbeddingBasedTripParams.config,\n      ConsumerEmbeddingBasedTwHINParams.config,\n      ConsumerEmbeddingBasedTwoTowerParams.config,\n      ConsumersBasedUserAdGraphParams.config,\n      ConsumersBasedUserTweetGraphParams.config,\n      ConsumersBasedUserVideoGraphParams.config,\n      CustomizedRetrievalBasedCandidateGenerationParams.config,\n      CustomizedRetrievalBasedOfflineInterestedInParams.config,\n      CustomizedRetrievalBasedFTROfflineInterestedInParams.config,\n      CustomizedRetrievalBasedTwhinParams.config,\n      EarlybirdFrsBasedCandidateGenerationParams.config,\n      FrsParams.config,\n      GlobalParams.config,\n      InterestedInParams.config,\n      ProducerBasedCandidateGenerationParams.config,\n      ProducerBasedUserAdGraphParams.config,\n      ProducerBasedUserTweetGraphParams.config,\n      RecentFollowsParams.config,\n      RecentNegativeSignalParams.config,\n      RecentNotificationsParams.config,\n      RecentOriginalTweetsParams.config,\n      RecentReplyTweetsParams.config,\n      RecentRetweetsParams.config,\n      RecentTweetFavoritesParams.config,\n      RelatedTweetGlobalParams.config,\n      RelatedVideoTweetGlobalParams.config,\n      RelatedTweetProducerBasedParams.config,\n      RelatedTweetTweetBasedParams.config,\n      RelatedVideoTweetTweetBasedParams.config,\n      RealGraphInParams.config,\n      RealGraphOonParams.config,\n      RepeatedProfileVisitsParams.config,\n      SimClustersANNParams.config,\n      TopicTweetParams.config,\n      TweetBasedCandidateGenerationParams.config,\n      TweetBasedUserAdGraphParams.config,\n      TweetBasedUserTweetGraphParams.config,\n      TweetBasedUserVideoGraphParams.config,\n      TweetSharesParams.config,\n      TweetBasedTwHINParams.config,\n      RealGraphOonParams.config,\n      GoodTweetClickParams.config,\n      GoodProfileClickParams.config,\n      UtegTweetGlobalParams.config,\n      VideoTweetFilterParams.config,\n      VideoViewTweetsParams.config,\n      UnifiedUSSSignalParams.config,\n    ),\n    simpleName = \"CrMixerConfig\"\n  )\n\n  val allParams: Seq[Param[_] with FSName] = {\n    AdsParams.AllParams ++\n      BlenderParams.AllParams ++\n      BypassInterleaveAndRankParams.AllParams ++\n      RankerParams.AllParams ++\n      ConsumerBasedWalsParams.AllParams ++\n      ConsumerEmbeddingBasedCandidateGenerationParams.AllParams ++\n      ConsumerEmbeddingBasedTripParams.AllParams ++\n      ConsumerEmbeddingBasedTwHINParams.AllParams ++\n      ConsumerEmbeddingBasedTwoTowerParams.AllParams ++\n      ConsumersBasedUserAdGraphParams.AllParams ++\n      ConsumersBasedUserTweetGraphParams.AllParams ++\n      ConsumersBasedUserVideoGraphParams.AllParams ++\n      CustomizedRetrievalBasedCandidateGenerationParams.AllParams ++\n      CustomizedRetrievalBasedOfflineInterestedInParams.AllParams ++\n      CustomizedRetrievalBasedFTROfflineInterestedInParams.AllParams ++\n      CustomizedRetrievalBasedTwhinParams.AllParams ++\n      EarlybirdFrsBasedCandidateGenerationParams.AllParams ++\n      FrsParams.AllParams ++\n      GlobalParams.AllParams ++\n      InterestedInParams.AllParams ++\n      ProducerBasedCandidateGenerationParams.AllParams ++\n      ProducerBasedUserAdGraphParams.AllParams ++\n      ProducerBasedUserTweetGraphParams.AllParams ++\n      RecentFollowsParams.AllParams ++\n      RecentNegativeSignalParams.AllParams ++\n      RecentNotificationsParams.AllParams ++\n      RecentOriginalTweetsParams.AllParams ++\n      RecentReplyTweetsParams.AllParams ++\n      RecentRetweetsParams.AllParams ++\n      RecentTweetFavoritesParams.AllParams ++\n      RelatedTweetGlobalParams.AllParams ++\n      RelatedVideoTweetGlobalParams.AllParams ++\n      RelatedTweetProducerBasedParams.AllParams ++\n      RelatedTweetTweetBasedParams.AllParams ++\n      RelatedVideoTweetTweetBasedParams.AllParams ++\n      RepeatedProfileVisitsParams.AllParams ++\n      SimClustersANNParams.AllParams ++\n      TopicTweetParams.AllParams ++\n      TweetBasedCandidateGenerationParams.AllParams ++\n      TweetBasedUserAdGraphParams.AllParams ++\n      TweetBasedUserTweetGraphParams.AllParams ++\n      TweetBasedUserVideoGraphParams.AllParams ++\n      TweetSharesParams.AllParams ++\n      TweetBasedTwHINParams.AllParams ++\n      RealGraphOonParams.AllParams ++\n      RealGraphInParams.AllParams ++\n      GoodTweetClickParams.AllParams ++\n      GoodProfileClickParams.AllParams ++\n      UtegTweetGlobalParams.AllParams ++\n      VideoTweetFilterParams.AllParams ++\n      VideoViewTweetsParams.AllParams ++\n      UnifiedUSSSignalParams.AllParams\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/CustomizedRetrievalBasedCandidateGenerationParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject CustomizedRetrievalBasedCandidateGenerationParams {\n\n  // Offline SimClusters InterestedIn params\n  object EnableOfflineInterestedInParam\n      extends FSParam[Boolean](\n        name = \"customized_retrieval_based_candidate_generation_enable_offline_interestedin\",\n        default = false\n      )\n\n  // Offline SimClusters FTR-based InterestedIn\n  object EnableOfflineFTRInterestedInParam\n      extends FSParam[Boolean](\n        name = \"customized_retrieval_based_candidate_generation_enable_ftr_offline_interestedin\",\n        default = false\n      )\n\n  // TwHin Collab Filter Cluster params\n  object EnableTwhinCollabFilterClusterParam\n      extends FSParam[Boolean](\n        name = \"customized_retrieval_based_candidate_generation_enable_twhin_collab_filter_cluster\",\n        default = false\n      )\n\n  // TwHin Multi Cluster params\n  object EnableTwhinMultiClusterParam\n      extends FSParam[Boolean](\n        name = \"customized_retrieval_based_candidate_generation_enable_twhin_multi_cluster\",\n        default = false\n      )\n\n  object EnableRetweetBasedDiffusionParam\n      extends FSParam[Boolean](\n        name = \"customized_retrieval_based_candidate_generation_enable_retweet_based_diffusion\",\n        default = false\n      )\n  object CustomizedRetrievalBasedRetweetDiffusionSource\n      extends FSParam[String](\n        name =\n          \"customized_retrieval_based_candidate_generation_offline_retweet_based_diffusion_model_id\",\n        default = ModelConfig.RetweetBasedDiffusion\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableOfflineInterestedInParam,\n    EnableOfflineFTRInterestedInParam,\n    EnableTwhinCollabFilterClusterParam,\n    EnableTwhinMultiClusterParam,\n    EnableRetweetBasedDiffusionParam,\n    CustomizedRetrievalBasedRetweetDiffusionSource\n  )\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableOfflineInterestedInParam,\n      EnableOfflineFTRInterestedInParam,\n      EnableTwhinCollabFilterClusterParam,\n      EnableTwhinMultiClusterParam,\n      EnableRetweetBasedDiffusionParam\n    )\n\n    val stringFSOverrides =\n      FeatureSwitchOverrideUtil.getStringFSOverrides(\n        CustomizedRetrievalBasedRetweetDiffusionSource\n      )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(stringFSOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/CustomizedRetrievalBasedFTROfflineInterestedInParams.scala",
    "content": "package com.twitter.cr_mixer.param\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject CustomizedRetrievalBasedFTROfflineInterestedInParams {\n  object CustomizedRetrievalBasedFTROfflineInterestedInSource\n      extends FSParam[String](\n        name = \"customized_retrieval_based_ftr_offline_interestedin_model_id\",\n        default = ModelConfig.OfflineFavDecayedSum\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    CustomizedRetrievalBasedFTROfflineInterestedInSource)\n\n  lazy val config: BaseConfig = {\n\n    val stringFSOverrides =\n      FeatureSwitchOverrideUtil.getStringFSOverrides(\n        CustomizedRetrievalBasedFTROfflineInterestedInSource\n      )\n\n    BaseConfigBuilder()\n      .set(stringFSOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/CustomizedRetrievalBasedOfflineInterestedInParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject CustomizedRetrievalBasedOfflineInterestedInParams {\n\n  // Model slots available for offline InterestedIn candidate generation\n  object CustomizedRetrievalBasedOfflineInterestedInSource\n      extends FSParam[String](\n        name = \"customized_retrieval_based_offline_interestedin_model_id\",\n        default = ModelConfig.OfflineInterestedInFromKnownFor2020\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(CustomizedRetrievalBasedOfflineInterestedInSource)\n\n  lazy val config: BaseConfig = {\n\n    val stringFSOverrides =\n      FeatureSwitchOverrideUtil.getStringFSOverrides(\n        CustomizedRetrievalBasedOfflineInterestedInSource\n      )\n\n    BaseConfigBuilder()\n      .set(stringFSOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/CustomizedRetrievalBasedTwhinParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject CustomizedRetrievalBasedTwhinParams {\n\n  // Model slots available for TwhinCollab and MultiCluster\n  object CustomizedRetrievalBasedTwhinCollabFilterFollowSource\n      extends FSParam[String](\n        name = \"customized_retrieval_based_offline_twhin_collab_filter_follow_model_id\",\n        default = ModelConfig.TwhinCollabFilterForFollow\n      )\n\n  object CustomizedRetrievalBasedTwhinCollabFilterEngagementSource\n      extends FSParam[String](\n        name = \"customized_retrieval_based_offline_twhin_collab_filter_engagement_model_id\",\n        default = ModelConfig.TwhinCollabFilterForEngagement\n      )\n\n  object CustomizedRetrievalBasedTwhinMultiClusterFollowSource\n      extends FSParam[String](\n        name = \"customized_retrieval_based_offline_twhin_multi_cluster_follow_model_id\",\n        default = ModelConfig.TwhinMultiClusterForFollow\n      )\n\n  object CustomizedRetrievalBasedTwhinMultiClusterEngagementSource\n      extends FSParam[String](\n        name = \"customized_retrieval_based_offline_twhin_multi_cluster_engagement_model_id\",\n        default = ModelConfig.TwhinMultiClusterForEngagement\n      )\n\n  val AllParams: Seq[Param[_] with FSName] =\n    Seq(\n      CustomizedRetrievalBasedTwhinCollabFilterFollowSource,\n      CustomizedRetrievalBasedTwhinCollabFilterEngagementSource,\n      CustomizedRetrievalBasedTwhinMultiClusterFollowSource,\n      CustomizedRetrievalBasedTwhinMultiClusterEngagementSource,\n    )\n\n  lazy val config: BaseConfig = {\n\n    val stringFSOverrides =\n      FeatureSwitchOverrideUtil.getStringFSOverrides(\n        CustomizedRetrievalBasedTwhinCollabFilterFollowSource,\n        CustomizedRetrievalBasedTwhinCollabFilterEngagementSource,\n        CustomizedRetrievalBasedTwhinMultiClusterFollowSource,\n        CustomizedRetrievalBasedTwhinMultiClusterEngagementSource,\n      )\n\n    BaseConfigBuilder()\n      .set(stringFSOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/EarlybirdFrsBasedCandidateGenerationParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.EarlybirdSimilarityEngineType\nimport com.twitter.cr_mixer.model.EarlybirdSimilarityEngineType_ModelBased\nimport com.twitter.cr_mixer.model.EarlybirdSimilarityEngineType_RecencyBased\nimport com.twitter.cr_mixer.model.EarlybirdSimilarityEngineType_TensorflowBased\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\nobject EarlybirdFrsBasedCandidateGenerationParams {\n  object CandidateGenerationEarlybirdSimilarityEngineType extends Enumeration {\n    protected case class SimilarityEngineType(rankingMode: EarlybirdSimilarityEngineType)\n        extends super.Val\n    import scala.language.implicitConversions\n    implicit def valueToEarlybirdRankingMode(x: Value): SimilarityEngineType =\n      x.asInstanceOf[SimilarityEngineType]\n\n    val EarlybirdRankingMode_RecencyBased: SimilarityEngineType = SimilarityEngineType(\n      EarlybirdSimilarityEngineType_RecencyBased)\n    val EarlybirdRankingMode_ModelBased: SimilarityEngineType = SimilarityEngineType(\n      EarlybirdSimilarityEngineType_ModelBased)\n    val EarlybirdRankingMode_TensorflowBased: SimilarityEngineType = SimilarityEngineType(\n      EarlybirdSimilarityEngineType_TensorflowBased)\n  }\n\n  object FrsBasedCandidateGenerationEarlybirdSimilarityEngineTypeParam\n      extends FSEnumParam[CandidateGenerationEarlybirdSimilarityEngineType.type](\n        name = \"frs_based_candidate_generation_earlybird_ranking_mode_id\",\n        default =\n          CandidateGenerationEarlybirdSimilarityEngineType.EarlybirdRankingMode_RecencyBased,\n        enum = CandidateGenerationEarlybirdSimilarityEngineType\n      )\n\n  object FrsBasedCandidateGenerationRecencyBasedEarlybirdMaxTweetsPerUser\n      extends FSBoundedParam[Int](\n        name = \"frs_based_candidate_generation_earlybird_max_tweets_per_user\",\n        default = 100,\n        min = 0,\n        /**\n         * Note max should be equal to EarlybirdRecencyBasedCandidateStoreModule.DefaultMaxNumTweetPerUser.\n         * Which is the size of the memcached result list.\n         */\n        max = 100\n      )\n\n  object FrsBasedCandidateGenerationEarlybirdMaxTweetAge\n      extends FSBoundedParam[Duration](\n        name = \"frs_based_candidate_generation_earlybird_max_tweet_age_hours\",\n        default = 24.hours,\n        min = 12.hours,\n        /**\n         * Note max could be related to EarlybirdRecencyBasedCandidateStoreModule.DefaultMaxNumTweetPerUser.\n         * Which is the size of the memcached result list for recency based earlybird candidate source.\n         * E.g. if max = 720.hours, we may want to increase the DefaultMaxNumTweetPerUser.\n         */\n        max = 96.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromHours\n  }\n\n  object FrsBasedCandidateGenerationEarlybirdFilterOutRetweetsAndReplies\n      extends FSParam[Boolean](\n        name = \"frs_based_candidate_generation_earlybird_filter_out_retweets_and_replies\",\n        default = true\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    FrsBasedCandidateGenerationEarlybirdSimilarityEngineTypeParam,\n    FrsBasedCandidateGenerationRecencyBasedEarlybirdMaxTweetsPerUser,\n    FrsBasedCandidateGenerationEarlybirdMaxTweetAge,\n    FrsBasedCandidateGenerationEarlybirdFilterOutRetweetsAndReplies,\n  )\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      FrsBasedCandidateGenerationEarlybirdFilterOutRetweetsAndReplies,\n    )\n\n    val doubleOverrides = FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides()\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      FrsBasedCandidateGenerationRecencyBasedEarlybirdMaxTweetsPerUser\n    )\n\n    val durationFSOverrides =\n      FeatureSwitchOverrideUtil.getDurationFSOverrides(\n        FrsBasedCandidateGenerationEarlybirdMaxTweetAge\n      )\n\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n      FrsBasedCandidateGenerationEarlybirdSimilarityEngineTypeParam,\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .set(intOverrides: _*)\n      .set(enumOverrides: _*)\n      .set(durationFSOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/FrsParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.follow_recommendations.thriftscala.DisplayLocation\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.logging.Logger\nimport com.twitter.finagle.stats.NullStatsReceiver\n\nobject FrsParams {\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"signal_frs_enable_source\",\n        default = false\n      )\n\n  object EnableSourceGraphParam\n      extends FSParam[Boolean](\n        name = \"graph_frs_enable_source\",\n        default = false\n      )\n\n  object MinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"signal_frs_min_score\",\n        default = 0.4,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object MaxConsumerSeedsNumParam\n      extends FSBoundedParam[Int](\n        name = \"graph_frs_max_user_seeds_num\",\n        default = 200,\n        min = 0,\n        max = 1000\n      )\n\n  /**\n   * These params below are only used for FrsTweetCandidateGenerator and shouldn't be used in other endpoints\n   *    * FrsBasedCandidateGenerationMaxSeedsNumParam\n   *    * FrsCandidateGenerationDisplayLocationParam\n   *    * FrsCandidateGenerationDisplayLocation\n   *    * FrsBasedCandidateGenerationMaxCandidatesNumParam\n   */\n  object FrsBasedCandidateGenerationEnableVisibilityFilteringParam\n      extends FSParam[Boolean](\n        name = \"frs_based_candidate_generation_enable_vf\",\n        default = true\n      )\n\n  object FrsBasedCandidateGenerationMaxSeedsNumParam\n      extends FSBoundedParam[Int](\n        name = \"frs_based_candidate_generation_max_seeds_num\",\n        default = 100,\n        min = 0,\n        max = 800\n      )\n\n  object FrsBasedCandidateGenerationDisplayLocation extends Enumeration {\n    protected case class FrsDisplayLocationValue(displayLocation: DisplayLocation) extends super.Val\n    import scala.language.implicitConversions\n    implicit def valueToDisplayLocationValue(x: Value): FrsDisplayLocationValue =\n      x.asInstanceOf[FrsDisplayLocationValue]\n\n    val DisplayLocation_ContentRecommender: FrsDisplayLocationValue = FrsDisplayLocationValue(\n      DisplayLocation.ContentRecommender)\n    val DisplayLocation_Home: FrsDisplayLocationValue = FrsDisplayLocationValue(\n      DisplayLocation.HomeTimelineTweetRecs)\n    val DisplayLocation_Notifications: FrsDisplayLocationValue = FrsDisplayLocationValue(\n      DisplayLocation.TweetNotificationRecs)\n  }\n\n  object FrsBasedCandidateGenerationDisplayLocationParam\n      extends FSEnumParam[FrsBasedCandidateGenerationDisplayLocation.type](\n        name = \"frs_based_candidate_generation_display_location_id\",\n        default = FrsBasedCandidateGenerationDisplayLocation.DisplayLocation_Home,\n        enum = FrsBasedCandidateGenerationDisplayLocation\n      )\n\n  object FrsBasedCandidateGenerationMaxCandidatesNumParam\n      extends FSBoundedParam[Int](\n        name = \"frs_based_candidate_generation_max_candidates_num\",\n        default = 100,\n        min = 0,\n        max = 2000\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableSourceParam,\n    EnableSourceGraphParam,\n    MinScoreParam,\n    MaxConsumerSeedsNumParam,\n    FrsBasedCandidateGenerationMaxSeedsNumParam,\n    FrsBasedCandidateGenerationDisplayLocationParam,\n    FrsBasedCandidateGenerationMaxCandidatesNumParam,\n    FrsBasedCandidateGenerationEnableVisibilityFilteringParam\n  )\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam,\n      EnableSourceGraphParam,\n      FrsBasedCandidateGenerationEnableVisibilityFilteringParam\n    )\n\n    val doubleOverrides = FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(MinScoreParam)\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MaxConsumerSeedsNumParam,\n      FrsBasedCandidateGenerationMaxSeedsNumParam,\n      FrsBasedCandidateGenerationMaxCandidatesNumParam)\n\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n      FrsBasedCandidateGenerationDisplayLocationParam,\n    )\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .set(intOverrides: _*)\n      .set(enumOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/GlobalParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.simclusters_v2.common.ModelVersions\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\n/**\n * Instantiate Params that do not relate to a specific product.\n * The params in this file correspond to config repo file\n * [[https://sourcegraph.twitter.biz/config-git.twitter.biz/config/-/blob/features/cr-mixer/main/twistly_core.yml]]\n */\nobject GlobalParams {\n\n  object MaxCandidatesPerRequestParam\n      extends FSBoundedParam[Int](\n        name = \"twistly_core_max_candidates_per_request\",\n        default = 100,\n        min = 0,\n        max = 9000\n      )\n\n  object ModelVersionParam\n      extends FSEnumParam[ModelVersions.Enum.type](\n        name = \"twistly_core_simclusters_model_version_id\",\n        default = ModelVersions.Enum.Model20M145K2020,\n        enum = ModelVersions.Enum\n      )\n\n  object UnifiedMaxSourceKeyNum\n      extends FSBoundedParam[Int](\n        name = \"twistly_core_unified_max_sourcekey_num\",\n        default = 15,\n        min = 0,\n        max = 100\n      )\n\n  object MaxCandidateNumPerSourceKeyParam\n      extends FSBoundedParam[Int](\n        name = \"twistly_core_candidate_per_sourcekey_max_num\",\n        default = 200,\n        min = 0,\n        max = 1000\n      )\n\n  // 1 hours to 30 days\n  object MaxTweetAgeHoursParam\n      extends FSBoundedParam[Duration](\n        name = \"twistly_core_max_tweet_age_hours\",\n        default = 720.hours,\n        min = 1.hours,\n        max = 720.hours\n      )\n      with HasDurationConversion {\n\n    override val durationConversion: DurationConversion = DurationConversion.FromHours\n  }\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    MaxCandidatesPerRequestParam,\n    UnifiedMaxSourceKeyNum,\n    MaxCandidateNumPerSourceKeyParam,\n    ModelVersionParam,\n    MaxTweetAgeHoursParam\n  )\n\n  lazy val config: BaseConfig = {\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides()\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MaxCandidatesPerRequestParam,\n      UnifiedMaxSourceKeyNum,\n      MaxCandidateNumPerSourceKeyParam\n    )\n\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n      ModelVersionParam\n    )\n\n    val boundedDurationFSOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDurationFSOverrides(MaxTweetAgeHoursParam)\n\n    val seqOverrides = FeatureSwitchOverrideUtil.getLongSeqFSOverrides()\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(intOverrides: _*)\n      .set(boundedDurationFSOverrides: _*)\n      .set(enumOverrides: _*)\n      .set(seqOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/GoodProfileClickParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.usersignalservice.thriftscala.SignalType\n\nobject GoodProfileClickParams {\n\n  object ClickMinDwellTimeParam extends Enumeration {\n    protected case class SignalTypeValue(signalType: SignalType) extends super.Val\n    import scala.language.implicitConversions\n    implicit def valueToSignalTypeValue(x: Value): SignalTypeValue =\n      x.asInstanceOf[SignalTypeValue]\n\n    val TotalDwellTime10s = SignalTypeValue(SignalType.GoodProfileClick)\n    val TotalDwellTime20s = SignalTypeValue(SignalType.GoodProfileClick20s)\n    val TotalDwellTime30s = SignalTypeValue(SignalType.GoodProfileClick30s)\n\n  }\n\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"signal_good_profile_clicks_enable_source\",\n        default = false\n      )\n\n  object ClickMinDwellTimeType\n      extends FSEnumParam[ClickMinDwellTimeParam.type](\n        name = \"signal_good_profile_clicks_min_dwelltime_type_id\",\n        default = ClickMinDwellTimeParam.TotalDwellTime10s,\n        enum = ClickMinDwellTimeParam\n      )\n\n  val AllParams: Seq[Param[_] with FSName] =\n    Seq(EnableSourceParam, ClickMinDwellTimeType)\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam\n    )\n\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n      ClickMinDwellTimeType\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(enumOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/GoodTweetClickParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.usersignalservice.thriftscala.SignalType\n\nobject GoodTweetClickParams {\n\n  object ClickMinDwellTimeParam extends Enumeration {\n    protected case class SignalTypeValue(signalType: SignalType) extends super.Val\n    import scala.language.implicitConversions\n    implicit def valueToSignalTypeValue(x: Value): SignalTypeValue =\n      x.asInstanceOf[SignalTypeValue]\n\n    val TotalDwellTime2s = SignalTypeValue(SignalType.GoodTweetClick)\n    val TotalDwellTime5s = SignalTypeValue(SignalType.GoodTweetClick5s)\n    val TotalDwellTime10s = SignalTypeValue(SignalType.GoodTweetClick10s)\n    val TotalDwellTime30s = SignalTypeValue(SignalType.GoodTweetClick30s)\n\n  }\n\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"signal_good_tweet_clicks_enable_source\",\n        default = false\n      )\n\n  object ClickMinDwellTimeType\n      extends FSEnumParam[ClickMinDwellTimeParam.type](\n        name = \"signal_good_tweet_clicks_min_dwelltime_type_id\",\n        default = ClickMinDwellTimeParam.TotalDwellTime2s,\n        enum = ClickMinDwellTimeParam\n      )\n\n  object MaxSignalNumParam\n      extends FSBoundedParam[Int](\n        name = \"signal_good_tweet_clicks_max_signal_num\",\n        default = 15,\n        min = 0,\n        max = 15\n      )\n\n  val AllParams: Seq[Param[_] with FSName] =\n    Seq(EnableSourceParam, ClickMinDwellTimeType, MaxSignalNumParam)\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam\n    )\n\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n      ClickMinDwellTimeType\n    )\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MaxSignalNumParam\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(enumOverrides: _*)\n      .set(intOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/InterestedInParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.simclusters_v2.thriftscala.{EmbeddingType => SimClustersEmbeddingType}\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject InterestedInParams {\n\n  object SourceEmbedding extends Enumeration {\n    protected case class EmbeddingType(embeddingType: SimClustersEmbeddingType) extends super.Val\n    import scala.language.implicitConversions\n    implicit def valueToEmbeddingtype(x: Value): EmbeddingType = x.asInstanceOf[EmbeddingType]\n\n    val UserInterestedIn: Value = EmbeddingType(SimClustersEmbeddingType.FilteredUserInterestedIn)\n    val UnfilteredUserInterestedIn: Value = EmbeddingType(\n      SimClustersEmbeddingType.UnfilteredUserInterestedIn)\n    val FromProducerEmbedding: Value = EmbeddingType(\n      SimClustersEmbeddingType.FilteredUserInterestedInFromPE)\n    val LogFavBasedUserInterestedInFromAPE: Value = EmbeddingType(\n      SimClustersEmbeddingType.LogFavBasedUserInterestedInFromAPE)\n    val FollowBasedUserInterestedInFromAPE: Value = EmbeddingType(\n      SimClustersEmbeddingType.FollowBasedUserInterestedInFromAPE)\n    val UserNextInterestedIn: Value = EmbeddingType(SimClustersEmbeddingType.UserNextInterestedIn)\n    // AddressBook based InterestedIn\n    val LogFavBasedUserInterestedAverageAddressBookFromIIAPE: Value = EmbeddingType(\n      SimClustersEmbeddingType.LogFavBasedUserInterestedAverageAddressBookFromIIAPE)\n    val LogFavBasedUserInterestedMaxpoolingAddressBookFromIIAPE: Value = EmbeddingType(\n      SimClustersEmbeddingType.LogFavBasedUserInterestedMaxpoolingAddressBookFromIIAPE)\n    val LogFavBasedUserInterestedBooktypeMaxpoolingAddressBookFromIIAPE: Value = EmbeddingType(\n      SimClustersEmbeddingType.LogFavBasedUserInterestedBooktypeMaxpoolingAddressBookFromIIAPE)\n    val LogFavBasedUserInterestedLargestDimMaxpoolingAddressBookFromIIAPE: Value = EmbeddingType(\n      SimClustersEmbeddingType.LogFavBasedUserInterestedLargestDimMaxpoolingAddressBookFromIIAPE)\n    val LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE: Value = EmbeddingType(\n      SimClustersEmbeddingType.LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE)\n    val LogFavBasedUserInterestedConnectedMaxpoolingAddressBookFromIIAPE: Value = EmbeddingType(\n      SimClustersEmbeddingType.LogFavBasedUserInterestedConnectedMaxpoolingAddressBookFromIIAPE)\n  }\n\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"twistly_interestedin_enable_source\",\n        default = true\n      )\n\n  object InterestedInEmbeddingIdParam\n      extends FSEnumParam[SourceEmbedding.type](\n        name = \"twistly_interestedin_embedding_id\",\n        default = SourceEmbedding.UnfilteredUserInterestedIn,\n        enum = SourceEmbedding\n      )\n\n  object MinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"twistly_interestedin_min_score\",\n        default = 0.072,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object EnableSourceSequentialModelParam\n      extends FSParam[Boolean](\n        name = \"twistly_interestedin_sequential_model_enable_source\",\n        default = false\n      )\n\n  object NextInterestedInEmbeddingIdParam\n      extends FSEnumParam[SourceEmbedding.type](\n        name = \"twistly_interestedin_sequential_model_embedding_id\",\n        default = SourceEmbedding.UserNextInterestedIn,\n        enum = SourceEmbedding\n      )\n\n  object MinScoreSequentialModelParam\n      extends FSBoundedParam[Double](\n        name = \"twistly_interestedin_sequential_model_min_score\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object EnableSourceAddressBookParam\n      extends FSParam[Boolean](\n        name = \"twistly_interestedin_addressbook_enable_source\",\n        default = false\n      )\n\n  object AddressBookInterestedInEmbeddingIdParam\n      extends FSEnumParam[SourceEmbedding.type](\n        name = \"twistly_interestedin_addressbook_embedding_id\",\n        default = SourceEmbedding.LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE,\n        enum = SourceEmbedding\n      )\n\n  object MinScoreAddressBookParam\n      extends FSBoundedParam[Double](\n        name = \"twistly_interestedin_addressbook_min_score\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  // Prod SimClusters ANN param\n  // This is used to enable/disable querying of production SANN service. Useful when experimenting\n  // with replacements to it.\n  object EnableProdSimClustersANNParam\n      extends FSParam[Boolean](\n        name = \"twistly_interestedin_enable_prod_simclusters_ann\",\n        default = true\n      )\n\n  // Experimental SimClusters ANN params\n  object EnableExperimentalSimClustersANNParam\n      extends FSParam[Boolean](\n        name = \"twistly_interestedin_enable_experimental_simclusters_ann\",\n        default = false\n      )\n\n  // SimClusters ANN 1 cluster params\n  object EnableSimClustersANN1Param\n      extends FSParam[Boolean](\n        name = \"twistly_interestedin_enable_simclusters_ann_1\",\n        default = false\n      )\n\n  // SimClusters ANN 2 cluster params\n  object EnableSimClustersANN2Param\n      extends FSParam[Boolean](\n        name = \"twistly_interestedin_enable_simclusters_ann_2\",\n        default = false\n      )\n\n  // SimClusters ANN 3 cluster params\n  object EnableSimClustersANN3Param\n      extends FSParam[Boolean](\n        name = \"twistly_interestedin_enable_simclusters_ann_3\",\n        default = false\n      )\n\n  // SimClusters ANN 5 cluster params\n  object EnableSimClustersANN5Param\n      extends FSParam[Boolean](\n        name = \"twistly_interestedin_enable_simclusters_ann_5\",\n        default = false\n      )\n\n  // SimClusters ANN 4 cluster params\n  object EnableSimClustersANN4Param\n      extends FSParam[Boolean](\n        name = \"twistly_interestedin_enable_simclusters_ann_4\",\n        default = false\n      )\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableSourceParam,\n    EnableSourceSequentialModelParam,\n    EnableSourceAddressBookParam,\n    EnableProdSimClustersANNParam,\n    EnableExperimentalSimClustersANNParam,\n    EnableSimClustersANN1Param,\n    EnableSimClustersANN2Param,\n    EnableSimClustersANN3Param,\n    EnableSimClustersANN5Param,\n    EnableSimClustersANN4Param,\n    MinScoreParam,\n    MinScoreSequentialModelParam,\n    MinScoreAddressBookParam,\n    InterestedInEmbeddingIdParam,\n    NextInterestedInEmbeddingIdParam,\n    AddressBookInterestedInEmbeddingIdParam,\n  )\n\n  lazy val config: BaseConfig = {\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam,\n      EnableSourceSequentialModelParam,\n      EnableSourceAddressBookParam,\n      EnableProdSimClustersANNParam,\n      EnableExperimentalSimClustersANNParam,\n      EnableSimClustersANN1Param,\n      EnableSimClustersANN2Param,\n      EnableSimClustersANN3Param,\n      EnableSimClustersANN5Param,\n      EnableSimClustersANN4Param\n    )\n\n    val doubleOverrides = FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(\n      MinScoreParam,\n      MinScoreSequentialModelParam,\n      MinScoreAddressBookParam)\n\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n      InterestedInEmbeddingIdParam,\n      NextInterestedInEmbeddingIdParam,\n      AddressBookInterestedInEmbeddingIdParam\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .set(enumOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/ProducerBasedCandidateGenerationParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject ProducerBasedCandidateGenerationParams {\n  // Source params. Not being used. It is always set to true in prod\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"producer_based_candidate_generation_enable_source\",\n        default = false\n      )\n\n  object UtgCombinationMethodParam\n      extends FSEnumParam[UnifiedSETweetCombinationMethod.type](\n        name = \"producer_based_candidate_generation_utg_combination_method_id\",\n        default = UnifiedSETweetCombinationMethod.Frontload,\n        enum = UnifiedSETweetCombinationMethod\n      )\n\n  // UTG params\n  object EnableUTGParam\n      extends FSParam[Boolean](\n        name = \"producer_based_candidate_generation_enable_utg\",\n        default = false\n      )\n\n  object EnableUAGParam\n      extends FSParam[Boolean](\n        name = \"producer_based_candidate_generation_enable_uag\",\n        default = false\n      )\n\n  // SimClusters params\n  object EnableSimClustersANNParam\n      extends FSParam[Boolean](\n        name = \"producer_based_candidate_generation_enable_simclusters\",\n        default = true\n      )\n\n  // Filter params\n  object SimClustersMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"producer_based_candidate_generation_filter_simclusters_min_score\",\n        default = 0.7,\n        min = 0.0,\n        max = 1.0\n      )\n\n  // Experimental SimClusters ANN params\n  object EnableExperimentalSimClustersANNParam\n      extends FSParam[Boolean](\n        name = \"producer_based_candidate_generation_enable_experimental_simclusters_ann\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 1 params\n  object EnableSimClustersANN1Param\n      extends FSParam[Boolean](\n        name = \"producer_based_candidate_generation_enable_simclusters_ann_1\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 2 params\n  object EnableSimClustersANN2Param\n      extends FSParam[Boolean](\n        name = \"producer_based_candidate_generation_enable_simclusters_ann_2\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 3 params\n  object EnableSimClustersANN3Param\n      extends FSParam[Boolean](\n        name = \"producer_based_candidate_generation_enable_simclusters_ann_3\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 5 params\n  object EnableSimClustersANN5Param\n      extends FSParam[Boolean](\n        name = \"producer_based_candidate_generation_enable_simclusters_ann_5\",\n        default = false\n      )\n\n  object EnableSimClustersANN4Param\n      extends FSParam[Boolean](\n        name = \"producer_based_candidate_generation_enable_simclusters_ann_4\",\n        default = false\n      )\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableSourceParam,\n    EnableUAGParam,\n    EnableUTGParam,\n    EnableSimClustersANNParam,\n    EnableSimClustersANN1Param,\n    EnableSimClustersANN2Param,\n    EnableSimClustersANN3Param,\n    EnableSimClustersANN5Param,\n    EnableSimClustersANN4Param,\n    EnableExperimentalSimClustersANNParam,\n    SimClustersMinScoreParam,\n    UtgCombinationMethodParam\n  )\n\n  lazy val config: BaseConfig = {\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam,\n      EnableUAGParam,\n      EnableUTGParam,\n      EnableSimClustersANNParam,\n      EnableSimClustersANN1Param,\n      EnableSimClustersANN2Param,\n      EnableSimClustersANN3Param,\n      EnableSimClustersANN5Param,\n      EnableSimClustersANN4Param,\n      EnableExperimentalSimClustersANNParam\n    )\n\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n      UtgCombinationMethodParam,\n    )\n\n    val doubleOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(SimClustersMinScoreParam)\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .set(enumOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/ProducerBasedUserAdGraphParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject ProducerBasedUserAdGraphParams {\n\n  object MinCoOccurrenceParam\n      extends FSBoundedParam[Int](\n        name = \"producer_based_user_ad_graph_min_co_occurrence\",\n        default = 2,\n        min = 0,\n        max = 500\n      )\n\n  object MinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"producer_based_user_ad_graph_min_score\",\n        default = 3.0,\n        min = 0.0,\n        max = 10.0\n      )\n\n  object MaxNumFollowersParam\n      extends FSBoundedParam[Int](\n        name = \"producer_based_user_ad_graph_max_num_followers\",\n        default = 500,\n        min = 100,\n        max = 1000\n      )\n\n  val AllParams: Seq[Param[_] with FSName] =\n    Seq(MinCoOccurrenceParam, MaxNumFollowersParam, MinScoreParam)\n\n  lazy val config: BaseConfig = {\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MinCoOccurrenceParam,\n      MaxNumFollowersParam,\n    )\n\n    val doubleOverrides = FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(MinScoreParam)\n\n    BaseConfigBuilder()\n      .set(intOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/ProducerBasedUserTweetGraphParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject ProducerBasedUserTweetGraphParams {\n\n  object MinCoOccurrenceParam\n      extends FSBoundedParam[Int](\n        name = \"producer_based_user_tweet_graph_min_co_occurrence\",\n        default = 4,\n        min = 0,\n        max = 500\n      )\n\n  object MinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"producer_based_user_tweet_graph_min_score\",\n        default = 3.0,\n        min = 0.0,\n        max = 10.0\n      )\n\n  object MaxNumFollowersParam\n      extends FSBoundedParam[Int](\n        name = \"producer_based_user_tweet_graph_max_num_followers\",\n        default = 500,\n        min = 100,\n        max = 1000\n      )\n\n  val AllParams: Seq[Param[_] with FSName] =\n    Seq(MinCoOccurrenceParam, MaxNumFollowersParam, MinScoreParam)\n\n  lazy val config: BaseConfig = {\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MinCoOccurrenceParam,\n      MaxNumFollowersParam,\n    )\n\n    val doubleOverrides = FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(MinScoreParam)\n\n    BaseConfigBuilder()\n      .set(intOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RankerParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RankerParams {\n\n  object MaxCandidatesToRank\n      extends FSBoundedParam[Int](\n        name = \"twistly_core_max_candidates_to_rank\",\n        default = 2000,\n        min = 0,\n        max = 9999\n      )\n\n  object EnableBlueVerifiedTopK\n      extends FSParam[Boolean](\n        name = \"twistly_core_blue_verified_top_k\",\n        default = true\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    MaxCandidatesToRank,\n    EnableBlueVerifiedTopK\n  )\n\n  lazy val config: BaseConfig = {\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(EnableBlueVerifiedTopK)\n\n    val boundedDurationFSOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDurationFSOverrides()\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MaxCandidatesToRank\n    )\n\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n    )\n    val stringFSOverrides = FeatureSwitchOverrideUtil.getStringFSOverrides()\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(boundedDurationFSOverrides: _*)\n      .set(intOverrides: _*)\n      .set(enumOverrides: _*)\n      .set(stringFSOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RealGraphInParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi._\n\nobject RealGraphInParams {\n  object EnableSourceGraphParam\n      extends FSParam[Boolean](\n        name = \"graph_realgraphin_enable_source\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableSourceGraphParam,\n  )\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceGraphParam\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RealGraphOonParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RealGraphOonParams {\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"signal_realgraphoon_enable_source\",\n        default = false\n      )\n\n  object EnableSourceGraphParam\n      extends FSParam[Boolean](\n        name = \"graph_realgraphoon_enable_source\",\n        default = false\n      )\n\n  object MaxConsumerSeedsNumParam\n      extends FSBoundedParam[Int](\n        name = \"graph_realgraphoon_max_user_seeds_num\",\n        default = 200,\n        min = 0,\n        max = 1000\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableSourceParam,\n    EnableSourceGraphParam,\n    MaxConsumerSeedsNumParam\n  )\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam,\n      EnableSourceGraphParam\n    )\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(MaxConsumerSeedsNumParam)\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(intOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RecentFollowsParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RecentFollowsParams {\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"twistly_recentfollows_enable_source\",\n        default = true\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(EnableSourceParam)\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RecentNegativeSignalParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RecentNegativeSignalParams {\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"twistly_recentnegativesignals_enable_source\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableSourceParam\n  )\n\n  lazy val config: BaseConfig = {\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n    )\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam\n    )\n\n    val doubleOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides()\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*).set(doubleOverrides: _*).set(enumOverrides: _*).build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RecentNotificationsParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RecentNotificationsParams {\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"twistly_recentnotifications_enable_source\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(EnableSourceParam)\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RecentOriginalTweetsParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RecentOriginalTweetsParams {\n\n  // Source params\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"twistly_recentoriginaltweets_enable_source\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(EnableSourceParam)\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(EnableSourceParam)\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RecentReplyTweetsParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RecentReplyTweetsParams {\n  // Source params\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"twistly_recentreplytweets_enable_source\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(EnableSourceParam)\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(EnableSourceParam)\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RecentRetweetsParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RecentRetweetsParams {\n\n  // Source params\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"twistly_recentretweets_enable_source\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(EnableSourceParam)\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RecentTweetFavoritesParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RecentTweetFavoritesParams {\n  // Source params\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"twistly_recenttweetfavorites_enable_source\",\n        default = true\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(EnableSourceParam)\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RelatedTweetGlobalParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RelatedTweetGlobalParams {\n\n  object MaxCandidatesPerRequestParam\n      extends FSBoundedParam[Int](\n        name = \"related_tweet_core_max_candidates_per_request\",\n        default = 100,\n        min = 0,\n        max = 500\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(MaxCandidatesPerRequestParam)\n\n  lazy val config: BaseConfig = {\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MaxCandidatesPerRequestParam\n    )\n\n    BaseConfigBuilder()\n      .set(intOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RelatedTweetProducerBasedParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RelatedTweetProducerBasedParams {\n\n  // UTG params\n  object EnableUTGParam\n      extends FSParam[Boolean](\n        name = \"related_tweet_producer_based_enable_utg\",\n        default = false\n      )\n\n  // SimClusters params\n  object EnableSimClustersANNParam\n      extends FSParam[Boolean](\n        name = \"related_tweet_producer_based_enable_simclusters\",\n        default = true\n      )\n\n  // Filter params\n  object SimClustersMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"related_tweet_producer_based_filter_simclusters_min_score\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  // Experimental SimClusters ANN params\n  object EnableExperimentalSimClustersANNParam\n      extends FSParam[Boolean](\n        name = \"related_tweet_producer_based_enable_experimental_simclusters_ann\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 1 params\n  object EnableSimClustersANN1Param\n      extends FSParam[Boolean](\n        name = \"related_tweet_producer_based_enable_simclusters_ann_1\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 2 params\n  object EnableSimClustersANN2Param\n      extends FSParam[Boolean](\n        name = \"related_tweet_producer_based_enable_simclusters_ann_2\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 3 params\n  object EnableSimClustersANN3Param\n      extends FSParam[Boolean](\n        name = \"related_tweet_producer_based_enable_simclusters_ann_3\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 3 params\n  object EnableSimClustersANN5Param\n      extends FSParam[Boolean](\n        name = \"related_tweet_producer_based_enable_simclusters_ann_5\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 4 params\n  object EnableSimClustersANN4Param\n      extends FSParam[Boolean](\n        name = \"related_tweet_producer_based_enable_simclusters_ann_4\",\n        default = false\n      )\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableUTGParam,\n    EnableSimClustersANNParam,\n    EnableSimClustersANN1Param,\n    EnableSimClustersANN2Param,\n    EnableSimClustersANN3Param,\n    EnableSimClustersANN5Param,\n    EnableSimClustersANN4Param,\n    EnableExperimentalSimClustersANNParam,\n    SimClustersMinScoreParam\n  )\n\n  lazy val config: BaseConfig = {\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableUTGParam,\n      EnableSimClustersANNParam,\n      EnableSimClustersANN1Param,\n      EnableSimClustersANN2Param,\n      EnableSimClustersANN3Param,\n      EnableSimClustersANN5Param,\n      EnableSimClustersANN4Param,\n      EnableExperimentalSimClustersANNParam\n    )\n\n    val doubleOverrides = FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(\n      SimClustersMinScoreParam\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RelatedTweetTweetBasedParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RelatedTweetTweetBasedParams {\n\n  // UTG params\n  object EnableUTGParam\n      extends FSParam[Boolean](\n        name = \"related_tweet_tweet_based_enable_utg\",\n        default = false\n      )\n\n  // UVG params\n  object EnableUVGParam\n      extends FSParam[Boolean](\n        name = \"related_tweet_tweet_based_enable_uvg\",\n        default = false\n      )\n\n  // UAG params\n  object EnableUAGParam\n      extends FSParam[Boolean](\n        name = \"related_tweet_tweet_based_enable_uag\",\n        default = false\n      )\n\n  // SimClusters params\n  object EnableSimClustersANNParam\n      extends FSParam[Boolean](\n        name = \"related_tweet_tweet_based_enable_simclusters\",\n        default = true\n      )\n\n  // Experimental SimClusters ANN params\n  object EnableExperimentalSimClustersANNParam\n      extends FSParam[Boolean](\n        name = \"related_tweet_tweet_based_enable_experimental_simclusters_ann\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 1 params\n  object EnableSimClustersANN1Param\n      extends FSParam[Boolean](\n        name = \"related_tweet_tweet_based_enable_simclusters_ann_1\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 2 params\n  object EnableSimClustersANN2Param\n      extends FSParam[Boolean](\n        name = \"related_tweet_tweet_based_enable_simclusters_ann_2\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 3 params\n  object EnableSimClustersANN3Param\n      extends FSParam[Boolean](\n        name = \"related_tweet_tweet_based_enable_simclusters_ann_3\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 5 params\n  object EnableSimClustersANN5Param\n      extends FSParam[Boolean](\n        name = \"related_tweet_tweet_based_enable_simclusters_ann_5\",\n        default = false\n      )\n\n  object EnableSimClustersANN4Param\n      extends FSParam[Boolean](\n        name = \"related_tweet_tweet_based_enable_simclusters_ann_4\",\n        default = false\n      )\n  // TwHIN params\n  object EnableTwHINParam\n      extends FSParam[Boolean](\n        name = \"related_tweet_tweet_based_enable_twhin\",\n        default = false\n      )\n\n  // QIG params\n  object EnableQigSimilarTweetsParam\n      extends FSParam[Boolean](\n        name = \"related_tweet_tweet_based_enable_qig_similar_tweets\",\n        default = false\n      )\n\n  // Filter params\n  object SimClustersMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"related_tweet_tweet_based_filter_simclusters_min_score\",\n        default = 0.3,\n        min = 0.0,\n        max = 1.0\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableTwHINParam,\n    EnableQigSimilarTweetsParam,\n    EnableUTGParam,\n    EnableUVGParam,\n    EnableSimClustersANNParam,\n    EnableSimClustersANN2Param,\n    EnableSimClustersANN3Param,\n    EnableSimClustersANN5Param,\n    EnableSimClustersANN4Param,\n    EnableExperimentalSimClustersANNParam,\n    SimClustersMinScoreParam\n  )\n\n  lazy val config: BaseConfig = {\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableTwHINParam,\n      EnableQigSimilarTweetsParam,\n      EnableUTGParam,\n      EnableUVGParam,\n      EnableSimClustersANNParam,\n      EnableSimClustersANN2Param,\n      EnableSimClustersANN3Param,\n      EnableSimClustersANN5Param,\n      EnableSimClustersANN4Param,\n      EnableExperimentalSimClustersANNParam\n    )\n\n    val doubleOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(SimClustersMinScoreParam)\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RelatedVideoTweetGlobalParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RelatedVideoTweetGlobalParams {\n\n  object MaxCandidatesPerRequestParam\n      extends FSBoundedParam[Int](\n        name = \"related_video_tweet_core_max_candidates_per_request\",\n        default = 100,\n        min = 0,\n        max = 500\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(MaxCandidatesPerRequestParam)\n\n  lazy val config: BaseConfig = {\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MaxCandidatesPerRequestParam\n    )\n\n    BaseConfigBuilder()\n      .set(intOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RelatedVideoTweetTweetBasedParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RelatedVideoTweetTweetBasedParams {\n\n  // UTG params\n  object EnableUTGParam\n      extends FSParam[Boolean](\n        name = \"related_video_tweet_tweet_based_enable_utg\",\n        default = false\n      )\n\n  // SimClusters params\n  object EnableSimClustersANNParam\n      extends FSParam[Boolean](\n        name = \"related_video_tweet_tweet_based_enable_simclusters\",\n        default = true\n      )\n\n  // Experimental SimClusters ANN params\n  object EnableExperimentalSimClustersANNParam\n      extends FSParam[Boolean](\n        name = \"related_video_tweet_tweet_based_enable_experimental_simclusters_ann\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 1 params\n  object EnableSimClustersANN1Param\n      extends FSParam[Boolean](\n        name = \"related_video_tweet_tweet_based_enable_simclusters_ann_1\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 2 params\n  object EnableSimClustersANN2Param\n      extends FSParam[Boolean](\n        name = \"related_video_tweet_tweet_based_enable_simclusters_ann_2\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 3 params\n  object EnableSimClustersANN3Param\n      extends FSParam[Boolean](\n        name = \"related_video_tweet_tweet_based_enable_simclusters_ann_3\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 5 params\n  object EnableSimClustersANN5Param\n      extends FSParam[Boolean](\n        name = \"related_video_tweet_tweet_based_enable_simclusters_ann_5\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 4 params\n  object EnableSimClustersANN4Param\n      extends FSParam[Boolean](\n        name = \"related_video_tweet_tweet_based_enable_simclusters_ann_4\",\n        default = false\n      )\n  // TwHIN params\n  object EnableTwHINParam\n      extends FSParam[Boolean](\n        name = \"related_video_tweet_tweet_based_enable_twhin\",\n        default = false\n      )\n\n  // QIG params\n  object EnableQigSimilarTweetsParam\n      extends FSParam[Boolean](\n        name = \"related_video_tweet_tweet_based_enable_qig_similar_tweets\",\n        default = false\n      )\n\n  // Filter params\n  object SimClustersMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"related_video_tweet_tweet_based_filter_simclusters_min_score\",\n        default = 0.3,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object EnableUVGParam\n      extends FSParam[Boolean](\n        name = \"related_video_tweet_tweet_based_enable_uvg\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableTwHINParam,\n    EnableQigSimilarTweetsParam,\n    EnableUTGParam,\n    EnableUVGParam,\n    EnableSimClustersANNParam,\n    EnableSimClustersANN2Param,\n    EnableSimClustersANN3Param,\n    EnableSimClustersANN5Param,\n    EnableSimClustersANN4Param,\n    EnableExperimentalSimClustersANNParam,\n    SimClustersMinScoreParam\n  )\n\n  lazy val config: BaseConfig = {\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableTwHINParam,\n      EnableQigSimilarTweetsParam,\n      EnableUTGParam,\n      EnableUVGParam,\n      EnableSimClustersANNParam,\n      EnableSimClustersANN2Param,\n      EnableSimClustersANN3Param,\n      EnableSimClustersANN5Param,\n      EnableSimClustersANN4Param,\n      EnableExperimentalSimClustersANNParam\n    )\n\n    val doubleOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(SimClustersMinScoreParam)\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/RepeatedProfileVisitsParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.usersignalservice.thriftscala.SignalType\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject RepeatedProfileVisitsParams {\n  object ProfileMinVisitParam extends Enumeration {\n    protected case class SignalTypeValue(signalType: SignalType) extends super.Val\n    import scala.language.implicitConversions\n    implicit def valueToSignalTypeValue(x: Value): SignalTypeValue =\n      x.asInstanceOf[SignalTypeValue]\n\n    val TotalVisitsInPast180Days = SignalTypeValue(SignalType.RepeatedProfileVisit180dMinVisit6V1)\n    val TotalVisitsInPast90Days = SignalTypeValue(SignalType.RepeatedProfileVisit90dMinVisit6V1)\n    val TotalVisitsInPast14Days = SignalTypeValue(SignalType.RepeatedProfileVisit14dMinVisit2V1)\n    val TotalVisitsInPast180DaysNoNegative = SignalTypeValue(\n      SignalType.RepeatedProfileVisit180dMinVisit6V1NoNegative)\n    val TotalVisitsInPast90DaysNoNegative = SignalTypeValue(\n      SignalType.RepeatedProfileVisit90dMinVisit6V1NoNegative)\n    val TotalVisitsInPast14DaysNoNegative = SignalTypeValue(\n      SignalType.RepeatedProfileVisit14dMinVisit2V1NoNegative)\n  }\n\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"twistly_repeatedprofilevisits_enable_source\",\n        default = true\n      )\n\n  object MinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"twistly_repeatedprofilevisits_min_score\",\n        default = 0.5,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object ProfileMinVisitType\n      extends FSEnumParam[ProfileMinVisitParam.type](\n        name = \"twistly_repeatedprofilevisits_min_visit_type_id\",\n        default = ProfileMinVisitParam.TotalVisitsInPast14Days,\n        enum = ProfileMinVisitParam\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(EnableSourceParam, ProfileMinVisitType)\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam\n    )\n\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n      ProfileMinVisitType\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(enumOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/SimClustersANNParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject SimClustersANNParams {\n\n  // Different SimClusters ANN cluster has its own config id (model slot)\n  object SimClustersANNConfigId\n      extends FSParam[String](\n        name = \"similarity_simclusters_ann_simclusters_ann_config_id\",\n        default = \"Default\"\n      )\n\n  object SimClustersANN1ConfigId\n      extends FSParam[String](\n        name = \"similarity_simclusters_ann_simclusters_ann_1_config_id\",\n        default = \"20220810\"\n      )\n\n  object SimClustersANN2ConfigId\n      extends FSParam[String](\n        name = \"similarity_simclusters_ann_simclusters_ann_2_config_id\",\n        default = \"20220818\"\n      )\n\n  object SimClustersANN3ConfigId\n      extends FSParam[String](\n        name = \"similarity_simclusters_ann_simclusters_ann_3_config_id\",\n        default = \"20220819\"\n      )\n\n  object SimClustersANN5ConfigId\n      extends FSParam[String](\n        name = \"similarity_simclusters_ann_simclusters_ann_5_config_id\",\n        default = \"20221221\"\n      )\n  object SimClustersANN4ConfigId\n      extends FSParam[String](\n        name = \"similarity_simclusters_ann_simclusters_ann_4_config_id\",\n        default = \"20221220\"\n      )\n  object ExperimentalSimClustersANNConfigId\n      extends FSParam[String](\n        name = \"similarity_simclusters_ann_experimental_simclusters_ann_config_id\",\n        default = \"20220801\"\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    SimClustersANNConfigId,\n    SimClustersANN1ConfigId,\n    SimClustersANN2ConfigId,\n    SimClustersANN3ConfigId,\n    SimClustersANN5ConfigId,\n    ExperimentalSimClustersANNConfigId\n  )\n\n  lazy val config: BaseConfig = {\n    val stringOverrides = FeatureSwitchOverrideUtil.getStringFSOverrides(\n      SimClustersANNConfigId,\n      SimClustersANN1ConfigId,\n      SimClustersANN2ConfigId,\n      SimClustersANN3ConfigId,\n      SimClustersANN5ConfigId,\n      ExperimentalSimClustersANNConfigId\n    )\n\n    BaseConfigBuilder()\n      .set(stringOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/TopicTweetParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\nobject TopicTweetParams {\n  object MaxTweetAge\n      extends FSBoundedParam[Duration](\n        name = \"topic_tweet_candidate_generation_max_tweet_age_hours\",\n        default = 24.hours,\n        min = 12.hours,\n        max = 48.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromHours\n  }\n\n  object MaxTopicTweetCandidatesParam\n      extends FSBoundedParam[Int](\n        name = \"topic_tweet_max_candidates_num\",\n        default = 200,\n        min = 0,\n        max = 1000\n      )\n\n  object MaxSkitTfgCandidatesParam\n      extends FSBoundedParam[Int](\n        name = \"topic_tweet_skit_tfg_max_candidates_num\",\n        default = 100,\n        min = 0,\n        max = 1000\n      )\n\n  object MaxSkitHighPrecisionCandidatesParam\n      extends FSBoundedParam[Int](\n        name = \"topic_tweet_skit_high_precision_max_candidates_num\",\n        default = 100,\n        min = 0,\n        max = 1000\n      )\n\n  object MaxCertoCandidatesParam\n      extends FSBoundedParam[Int](\n        name = \"topic_tweet_certo_max_candidates_num\",\n        default = 100,\n        min = 0,\n        max = 1000\n      )\n\n  // The min prod score for Certo L2-normalized cosine candidates\n  object CertoScoreThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"topic_tweet_certo_score_threshold\",\n        default = 0.015,\n        min = 0,\n        max = 1\n      )\n\n  object SemanticCoreVersionIdParam\n      extends FSParam[Long](\n        name = \"semantic_core_version_id\",\n        default = 1380520918896713735L\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    CertoScoreThresholdParam,\n    MaxTopicTweetCandidatesParam,\n    MaxTweetAge,\n    MaxCertoCandidatesParam,\n    MaxSkitTfgCandidatesParam,\n    MaxSkitHighPrecisionCandidatesParam,\n    SemanticCoreVersionIdParam\n  )\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides()\n\n    val doubleOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(CertoScoreThresholdParam)\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MaxCertoCandidatesParam,\n      MaxSkitTfgCandidatesParam,\n      MaxSkitHighPrecisionCandidatesParam,\n      MaxTopicTweetCandidatesParam\n    )\n\n    val longOverrides = FeatureSwitchOverrideUtil.getLongFSOverrides(SemanticCoreVersionIdParam)\n\n    val durationFSOverrides = FeatureSwitchOverrideUtil.getDurationFSOverrides(MaxTweetAge)\n\n    val enumOverrides =\n      FeatureSwitchOverrideUtil.getEnumFSOverrides(NullStatsReceiver, Logger(getClass))\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .set(intOverrides: _*)\n      .set(longOverrides: _*)\n      .set(enumOverrides: _*)\n      .set(durationFSOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/TweetBasedCandidateGenerationParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject TweetBasedCandidateGenerationParams {\n\n  // Source params. Not being used. It is always set to true in prod\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_source\",\n        default = false\n      )\n\n  // UTG params\n  object EnableUTGParam\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_utg\",\n        default = true\n      )\n\n  // SimClusters params\n  object EnableSimClustersANNParam\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_simclusters\",\n        default = true\n      )\n\n  // Experimental SimClusters ANN params\n  object EnableExperimentalSimClustersANNParam\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_experimental_simclusters_ann\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 1 params\n  object EnableSimClustersANN1Param\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_simclusters_ann_1\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 2 params\n  object EnableSimClustersANN2Param\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_simclusters_ann_2\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 3 params\n  object EnableSimClustersANN3Param\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_simclusters_ann_3\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 3 params\n  object EnableSimClustersANN5Param\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_simclusters_ann_5\",\n        default = false\n      )\n\n  // SimClusters ANN cluster 4 params\n  object EnableSimClustersANN4Param\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_simclusters_ann_4\",\n        default = false\n      )\n  // TwHIN params\n  object EnableTwHINParam\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_twhin\",\n        default = false\n      )\n\n  // QIG params\n  object EnableQigSimilarTweetsParam\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_qig_similar_tweets\",\n        default = false\n      )\n\n  object QigMaxNumSimilarTweetsParam\n      extends FSBoundedParam[Int](\n        name = \"tweet_based_candidate_generation_qig_max_num_similar_tweets\",\n        default = 100,\n        min = 10,\n        max = 100\n      )\n\n  // UVG params\n  object EnableUVGParam\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_uvg\",\n        default = false\n      )\n\n  // UAG params\n  object EnableUAGParam\n      extends FSParam[Boolean](\n        name = \"tweet_based_candidate_generation_enable_uag\",\n        default = false\n      )\n\n  // Filter params\n  object SimClustersMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"tweet_based_candidate_generation_filter_simclusters_min_score\",\n        default = 0.5,\n        min = 0.0,\n        max = 1.0\n      )\n\n  // for learning DDG that has a higher threshold for video based SANN\n  object SimClustersVideoBasedMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"tweet_based_candidate_generation_filter_simclusters_video_based_min_score\",\n        default = 0.5,\n        min = 0.0,\n        max = 1.0\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableSourceParam,\n    EnableTwHINParam,\n    EnableQigSimilarTweetsParam,\n    EnableUTGParam,\n    EnableUVGParam,\n    EnableUAGParam,\n    EnableSimClustersANNParam,\n    EnableSimClustersANN1Param,\n    EnableSimClustersANN2Param,\n    EnableSimClustersANN3Param,\n    EnableSimClustersANN5Param,\n    EnableSimClustersANN4Param,\n    EnableExperimentalSimClustersANNParam,\n    SimClustersMinScoreParam,\n    SimClustersVideoBasedMinScoreParam,\n    QigMaxNumSimilarTweetsParam,\n  )\n\n  lazy val config: BaseConfig = {\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam,\n      EnableTwHINParam,\n      EnableQigSimilarTweetsParam,\n      EnableUTGParam,\n      EnableUVGParam,\n      EnableUAGParam,\n      EnableSimClustersANNParam,\n      EnableSimClustersANN1Param,\n      EnableSimClustersANN2Param,\n      EnableSimClustersANN3Param,\n      EnableSimClustersANN5Param,\n      EnableSimClustersANN4Param,\n      EnableExperimentalSimClustersANNParam,\n    )\n\n    val doubleOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(\n        SimClustersMinScoreParam,\n        SimClustersVideoBasedMinScoreParam)\n\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n    )\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      QigMaxNumSimilarTweetsParam\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(doubleOverrides: _*)\n      .set(enumOverrides: _*)\n      .set(intOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/TweetBasedTwHINParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.cr_mixer.model.ModelConfig\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject TweetBasedTwHINParams {\n  object ModelIdParam\n      extends FSParam[String](\n        name = \"tweet_based_twhin_model_id\",\n        default = ModelConfig.TweetBasedTwHINRegularUpdateAll20221024,\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(ModelIdParam)\n\n  lazy val config: BaseConfig = {\n    val stringFSOverrides =\n      FeatureSwitchOverrideUtil.getStringFSOverrides(\n        ModelIdParam\n      )\n\n    BaseConfigBuilder()\n      .set(stringFSOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/TweetBasedUserAdGraphParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject TweetBasedUserAdGraphParams {\n\n  object MinCoOccurrenceParam\n      extends FSBoundedParam[Int](\n        name = \"tweet_based_user_ad_graph_min_co_occurrence\",\n        default = 1,\n        min = 0,\n        max = 500\n      )\n\n  object ConsumersBasedMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"tweet_based_user_ad_graph_consumers_based_min_score\",\n        default = 0.0,\n        min = 0.0,\n        max = 10.0\n      )\n\n  object MaxConsumerSeedsNumParam\n      extends FSBoundedParam[Int](\n        name = \"tweet_based_user_ad_graph_max_user_seeds_num\",\n        default = 100,\n        min = 0,\n        max = 300\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    MinCoOccurrenceParam,\n    MaxConsumerSeedsNumParam,\n    ConsumersBasedMinScoreParam\n  )\n\n  lazy val config: BaseConfig = {\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MinCoOccurrenceParam,\n      MaxConsumerSeedsNumParam\n    )\n\n    val doubleOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(ConsumersBasedMinScoreParam)\n\n    BaseConfigBuilder()\n      .set(intOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/TweetBasedUserTweetGraphParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject TweetBasedUserTweetGraphParams {\n\n  object MinCoOccurrenceParam\n      extends FSBoundedParam[Int](\n        name = \"tweet_based_user_tweet_graph_min_co_occurrence\",\n        default = 3,\n        min = 0,\n        max = 500\n      )\n\n  object TweetBasedMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"tweet_based_user_tweet_graph_tweet_based_min_score\",\n        default = 0.5,\n        min = 0.0,\n        max = 10.0\n      )\n\n  object ConsumersBasedMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"tweet_based_user_tweet_graph_consumers_based_min_score\",\n        default = 4.0,\n        min = 0.0,\n        max = 10.0\n      )\n  object MaxConsumerSeedsNumParam\n      extends FSBoundedParam[Int](\n        name = \"tweet_based_user_tweet_graph_max_user_seeds_num\",\n        default = 100,\n        min = 0,\n        max = 300\n      )\n\n  object EnableCoverageExpansionOldTweetParam\n      extends FSParam[Boolean](\n        name = \"tweet_based_user_tweet_graph_enable_coverage_expansion_old_tweet\",\n        default = false\n      )\n\n  object EnableCoverageExpansionAllTweetParam\n      extends FSParam[Boolean](\n        name = \"tweet_based_user_tweet_graph_enable_coverage_expansion_all_tweet\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableCoverageExpansionAllTweetParam,\n    EnableCoverageExpansionOldTweetParam,\n    MinCoOccurrenceParam,\n    MaxConsumerSeedsNumParam,\n    TweetBasedMinScoreParam,\n    ConsumersBasedMinScoreParam\n  )\n\n  lazy val config: BaseConfig = {\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableCoverageExpansionAllTweetParam,\n      EnableCoverageExpansionOldTweetParam\n    )\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MinCoOccurrenceParam,\n      MaxConsumerSeedsNumParam\n    )\n\n    val doubleOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(\n        TweetBasedMinScoreParam,\n        ConsumersBasedMinScoreParam)\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(intOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/TweetBasedUserVideoGraphParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject TweetBasedUserVideoGraphParams {\n\n  object MinCoOccurrenceParam\n      extends FSBoundedParam[Int](\n        name = \"tweet_based_user_video_graph_min_co_occurrence\",\n        default = 5,\n        min = 0,\n        max = 500\n      )\n\n  object TweetBasedMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"tweet_based_user_video_graph_tweet_based_min_score\",\n        default = 0.0,\n        min = 0.0,\n        max = 100.0\n      )\n\n  object ConsumersBasedMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"tweet_based_user_video_graph_consumers_based_min_score\",\n        default = 4.0,\n        min = 0.0,\n        max = 10.0\n      )\n\n  object MaxConsumerSeedsNumParam\n      extends FSBoundedParam[Int](\n        name = \"tweet_based_user_video_graph_max_user_seeds_num\",\n        default = 200,\n        min = 0,\n        max = 500\n      )\n\n  object EnableCoverageExpansionOldTweetParam\n      extends FSParam[Boolean](\n        name = \"tweet_based_user_video_graph_enable_coverage_expansion_old_tweet\",\n        default = false\n      )\n\n  object EnableCoverageExpansionAllTweetParam\n      extends FSParam[Boolean](\n        name = \"tweet_based_user_video_graph_enable_coverage_expansion_all_tweet\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    MinCoOccurrenceParam,\n    MaxConsumerSeedsNumParam,\n    TweetBasedMinScoreParam,\n    EnableCoverageExpansionOldTweetParam,\n    EnableCoverageExpansionAllTweetParam\n  )\n\n  lazy val config: BaseConfig = {\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MinCoOccurrenceParam,\n      MaxConsumerSeedsNumParam\n    )\n\n    val doubleOverrides =\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(TweetBasedMinScoreParam)\n\n    BaseConfigBuilder()\n      .set(intOverrides: _*)\n      .set(doubleOverrides: _*)\n      .build()\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/TweetSharesParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\n\nobject TweetSharesParams {\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"twistly_tweetshares_enable_source\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(EnableSourceParam)\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam,\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .build()\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/UnifiedSETweetCombinationMethod.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport scala.language.implicitConversions\n\nobject UnifiedSETweetCombinationMethod extends Enumeration {\n\n  protected case class CombinationType(s: String) extends super.Val\n\n  implicit def valueToCombinationType(x: Value): CombinationType = x.asInstanceOf[CombinationType]\n\n  val Default: Value = CombinationType(\"\")\n  val Interleave: Value = CombinationType(\"Interleave\")\n  val Frontload: Value = CombinationType(\"Frontload\")\n  val Backfill: Value = CombinationType(\"Backfill\")\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/UnifiedUSSSignalParams.scala",
    "content": "package com.twitter.cr_mixer.param\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.usersignalservice.thriftscala.SignalType\nimport scala.language.implicitConversions\n\nobject UnifiedUSSSignalParams {\n\n  object TweetAggregationTypeParam extends Enumeration {\n    protected case class SignalTypeValue(signalType: SignalType) extends super.Val\n\n    implicit def valueToSignalTypeValue(x: Value): SignalTypeValue =\n      x.asInstanceOf[SignalTypeValue]\n\n    val UniformAggregation = SignalTypeValue(SignalType.TweetBasedUnifiedUniformSignal)\n    val EngagementAggregation = SignalTypeValue(\n      SignalType.TweetBasedUnifiedEngagementWeightedSignal)\n  }\n\n  object ProducerAggregationTypeParam extends Enumeration {\n    protected case class SignalTypeValue(signalType: SignalType) extends super.Val\n\n    import scala.language.implicitConversions\n\n    implicit def valueToSignalTypeValue(x: Value): SignalTypeValue =\n      x.asInstanceOf[SignalTypeValue]\n\n    val UniformAggregation = SignalTypeValue(SignalType.ProducerBasedUnifiedUniformSignal)\n    val EngagementAggregation = SignalTypeValue(\n      SignalType.ProducerBasedUnifiedEngagementWeightedSignal)\n\n  }\n\n  object ReplaceIndividualUSSSourcesParam\n      extends FSParam[Boolean](\n        name = \"twistly_agg_replace_enable_source\",\n        default = false\n      )\n\n  object EnableTweetAggSourceParam\n      extends FSParam[Boolean](\n        name = \"twistly_agg_tweet_agg_enable_source\",\n        default = false\n      )\n\n  object TweetAggTypeParam\n      extends FSEnumParam[TweetAggregationTypeParam.type](\n        name = \"twistly_agg_tweet_agg_type_id\",\n        default = TweetAggregationTypeParam.EngagementAggregation,\n        enum = TweetAggregationTypeParam\n      )\n\n  object UnifiedTweetSourceNumberParam\n      extends FSBoundedParam[Int](\n        name = \"twistly_agg_tweet_agg_source_number\",\n        default = 0,\n        min = 0,\n        max = 100,\n      )\n\n  object EnableProducerAggSourceParam\n      extends FSParam[Boolean](\n        name = \"twistly_agg_producer_agg_enable_source\",\n        default = false\n      )\n\n  object ProducerAggTypeParam\n      extends FSEnumParam[ProducerAggregationTypeParam.type](\n        name = \"twistly_agg_producer_agg_type_id\",\n        default = ProducerAggregationTypeParam.EngagementAggregation,\n        enum = ProducerAggregationTypeParam\n      )\n\n  object UnifiedProducerSourceNumberParam\n      extends FSBoundedParam[Int](\n        name = \"twistly_agg_producer_agg_source_number\",\n        default = 0,\n        min = 0,\n        max = 100,\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableTweetAggSourceParam,\n    EnableProducerAggSourceParam,\n    TweetAggTypeParam,\n    ProducerAggTypeParam,\n    UnifiedTweetSourceNumberParam,\n    UnifiedProducerSourceNumberParam,\n    ReplaceIndividualUSSSourcesParam\n  )\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableTweetAggSourceParam,\n      EnableProducerAggSourceParam,\n      ReplaceIndividualUSSSourcesParam,\n    )\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      UnifiedProducerSourceNumberParam,\n      UnifiedTweetSourceNumberParam)\n    val enumOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n      NullStatsReceiver,\n      Logger(getClass),\n      TweetAggTypeParam,\n      ProducerAggTypeParam\n    )\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(intOverrides: _*)\n      .set(enumOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/UtegTweetGlobalParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\nobject UtegTweetGlobalParams {\n\n  object MaxUtegCandidatesToRequestParam\n      extends FSBoundedParam[Int](\n        name = \"max_uteg_candidates_to_request\",\n        default = 800,\n        min = 10,\n        max = 200\n      )\n\n  object CandidateRefreshSinceTimeOffsetHoursParam\n      extends FSBoundedParam[Duration](\n        name = \"candidate_refresh_since_time_offset_hours\",\n        default = 48.hours,\n        min = 1.hours,\n        max = 96.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromHours\n  }\n\n  object EnableTLRHealthFilterParam\n      extends FSParam[Boolean](\n        name = \"enable_uteg_tlr_health_filter\",\n        default = true\n      )\n\n  object EnableRepliesToNonFollowedUsersFilterParam\n      extends FSParam[Boolean](\n        name = \"enable_uteg_replies_to_non_followed_users_filter\",\n        default = false\n      )\n\n  object EnableRetweetFilterParam\n      extends FSParam[Boolean](\n        name = \"enable_uteg_retweet_filter\",\n        default = true\n      )\n\n  object EnableInNetworkFilterParam\n      extends FSParam[Boolean](\n        name = \"enable_uteg_in_network_filter\",\n        default = true\n      )\n\n  val AllParams: Seq[Param[_] with FSName] =\n    Seq(\n      MaxUtegCandidatesToRequestParam,\n      CandidateRefreshSinceTimeOffsetHoursParam,\n      EnableTLRHealthFilterParam,\n      EnableRepliesToNonFollowedUsersFilterParam,\n      EnableRetweetFilterParam,\n      EnableInNetworkFilterParam\n    )\n\n  lazy val config: BaseConfig = {\n\n    val intOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n      MaxUtegCandidatesToRequestParam\n    )\n\n    val durationFSOverrides =\n      FeatureSwitchOverrideUtil.getDurationFSOverrides(\n        CandidateRefreshSinceTimeOffsetHoursParam\n      )\n\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableTLRHealthFilterParam,\n      EnableRepliesToNonFollowedUsersFilterParam,\n      EnableRetweetFilterParam,\n      EnableInNetworkFilterParam\n    )\n\n    BaseConfigBuilder()\n      .set(intOverrides: _*)\n      .set(durationFSOverrides: _*)\n      .set(booleanOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/VideoTweetFilterParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\n\nobject VideoTweetFilterParams {\n\n  object EnableVideoTweetFilterParam\n      extends FSParam[Boolean](\n        name = \"video_tweet_filter_enable_filter\",\n        default = false\n      )\n\n  val AllParams: Seq[Param[_] with FSName] = Seq(\n    EnableVideoTweetFilterParam\n  )\n\n  lazy val config: BaseConfig = {\n\n    val booleanOverrides =\n      FeatureSwitchOverrideUtil.getBooleanFSOverrides(EnableVideoTweetFilterParam)\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .build()\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/VideoViewTweetsParams.scala",
    "content": "package com.twitter.cr_mixer.param\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.timelines.configapi.BaseConfig\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.usersignalservice.thriftscala.SignalType\n\nobject VideoViewTweetsParams {\n  object EnableSourceParam\n      extends FSParam[Boolean](\n        name = \"signal_videoviewtweets_enable_source\",\n        default = false\n      )\n\n  object EnableSourceImpressionParam\n      extends FSParam[Boolean](\n        name = \"signal_videoviewtweets_enableimpression_source\",\n        default = false\n      )\n\n  object VideoViewTweetType extends Enumeration {\n    protected case class SignalTypeValue(signalType: SignalType) extends super.Val\n    import scala.language.implicitConversions\n    implicit def valueToSignalTypeValue(x: Value): SignalTypeValue =\n      x.asInstanceOf[SignalTypeValue]\n\n    val VideoTweetQualityView: SignalTypeValue = SignalTypeValue(SignalType.VideoView90dQualityV1)\n    val VideoTweetPlayback50: SignalTypeValue = SignalTypeValue(SignalType.VideoView90dPlayback50V1)\n  }\n\n  object VideoViewTweetTypeParam\n      extends FSEnumParam[VideoViewTweetType.type](\n        name = \"signal_videoviewtweets_videoviewtype_id\",\n        default = VideoViewTweetType.VideoTweetQualityView,\n        enum = VideoViewTweetType\n      )\n\n  val AllParams: Seq[Param[_] with FSName] =\n    Seq(EnableSourceParam, EnableSourceImpressionParam, VideoViewTweetTypeParam)\n\n  lazy val config: BaseConfig = {\n    val booleanOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n      EnableSourceParam,\n      EnableSourceImpressionParam,\n    )\n    val enumOverrides =\n      FeatureSwitchOverrideUtil.getEnumFSOverrides(\n        NullStatsReceiver,\n        Logger(getClass),\n        VideoViewTweetTypeParam)\n\n    BaseConfigBuilder()\n      .set(booleanOverrides: _*)\n      .set(enumOverrides: _*)\n      .build()\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"decider/src/main/scala\",\n        \"finagle/finagle-base-http/src/main\",\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"servo/decider\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider/CrMixerDecider.scala",
    "content": "package com.twitter.cr_mixer.param.decider\n\nimport com.twitter.decider.Decider\nimport com.twitter.decider.RandomRecipient\nimport com.twitter.decider.Recipient\nimport com.twitter.decider.SimpleRecipient\nimport com.twitter.simclusters_v2.common.DeciderGateBuilderWithIdHashing\nimport javax.inject.Inject\n\ncase class CrMixerDecider @Inject() (decider: Decider) {\n\n  def isAvailable(feature: String, recipient: Option[Recipient]): Boolean = {\n    decider.isAvailable(feature, recipient)\n  }\n\n  lazy val deciderGateBuilder = new DeciderGateBuilderWithIdHashing(decider)\n\n  /**\n   * When useRandomRecipient is set to false, the decider is either completely on or off.\n   * When useRandomRecipient is set to true, the decider is on for the specified % of traffic.\n   */\n  def isAvailable(feature: String, useRandomRecipient: Boolean = true): Boolean = {\n    if (useRandomRecipient) isAvailable(feature, Some(RandomRecipient))\n    else isAvailable(feature, None)\n  }\n\n  /***\n   * Decide whether the decider is available for a specific id using SimpleRecipient(id).\n   */\n  def isAvailableForId(\n    id: Long,\n    deciderConstants: String\n  ): Boolean = {\n    // Note: SimpleRecipient does expose a `val isUser = true` field which is not correct if the Id is not a user Id.\n    // However this field does not appear to be used anywhere in source.\n    decider.isAvailable(deciderConstants, Some(SimpleRecipient(id)))\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider/DeciderKey.scala",
    "content": "package com.twitter.cr_mixer.param.decider\n\nimport com.twitter.servo.decider.DeciderKeyEnum\n\nobject DeciderConstants {\n  val enableHealthSignalsScoreDeciderKey = \"enable_tweet_health_score\"\n  val enableUTGRealTimeTweetEngagementScoreDeciderKey = \"enable_utg_realtime_tweet_engagement_score\"\n  val enableUserAgathaScoreDeciderKey = \"enable_user_agatha_score\"\n  val enableUserTweetEntityGraphTrafficDeciderKey = \"enable_user_tweet_entity_graph_traffic\"\n  val enableUserTweetGraphTrafficDeciderKey = \"enable_user_tweet_graph_traffic\"\n  val enableUserVideoGraphTrafficDeciderKey = \"enable_user_video_graph_traffic\"\n  val enableUserAdGraphTrafficDeciderKey = \"enable_user_ad_graph_traffic\"\n  val enableSimClustersANN2DarkTrafficDeciderKey = \"enable_simclusters_ann_2_dark_traffic\"\n  val enableQigSimilarTweetsTrafficDeciderKey = \"enable_qig_similar_tweets_traffic\"\n  val enableFRSTrafficDeciderKey = \"enable_frs_traffic\"\n  val upperFunnelPerStepScribeRate = \"upper_funnel_per_step_scribe_rate\"\n  val kafkaMessageScribeSampleRate = \"kafka_message_scribe_sample_rate\"\n  val enableRealGraphMhStoreDeciderKey = \"enable_real_graph_mh_store\"\n  val topLevelApiDdgMetricsScribeRate = \"top_level_api_ddg_metrics_scribe_rate\"\n  val adsRecommendationsPerExperimentScribeRate = \"ads_recommendations_per_experiment_scribe_rate\"\n  val enableScribeForBlueVerifiedTweetCandidates =\n    \"enable_scribe_for_blue_verified_tweet_candidates\"\n\n  val enableUserStateStoreDeciderKey = \"enable_user_state_store\"\n  val enableUserMediaRepresentationStoreDeciderKey =\n    \"enable_user_media_representation_store\"\n  val enableMagicRecsRealTimeAggregatesStoreDeciderKey =\n    \"enable_magic_recs_real_time_aggregates_store\"\n\n  val enableEarlybirdTrafficDeciderKey = \"enable_earlybird_traffic\"\n\n  val enableTopicTweetTrafficDeciderKey = \"enable_topic_tweet_traffic\"\n\n  val getTweetRecommendationsCacheRate = \"get_tweet_recommendations_cache_rate\"\n}\n\nobject DeciderKey extends DeciderKeyEnum {\n\n  val enableHealthSignalsScoreDeciderKey: Value = Value(\n    DeciderConstants.enableHealthSignalsScoreDeciderKey\n  )\n\n  val enableUtgRealTimeTweetEngagementScoreDeciderKey: Value = Value(\n    DeciderConstants.enableUTGRealTimeTweetEngagementScoreDeciderKey\n  )\n  val enableUserAgathaScoreDeciderKey: Value = Value(\n    DeciderConstants.enableUserAgathaScoreDeciderKey\n  )\n  val enableUserMediaRepresentationStoreDeciderKey: Value = Value(\n    DeciderConstants.enableUserMediaRepresentationStoreDeciderKey\n  )\n\n  val enableMagicRecsRealTimeAggregatesStore: Value = Value(\n    DeciderConstants.enableMagicRecsRealTimeAggregatesStoreDeciderKey\n  )\n\n  val enableUserStateStoreDeciderKey: Value = Value(\n    DeciderConstants.enableUserStateStoreDeciderKey\n  )\n\n  val enableRealGraphMhStoreDeciderKey: Value = Value(\n    DeciderConstants.enableRealGraphMhStoreDeciderKey\n  )\n\n  val enableEarlybirdTrafficDeciderKey: Value = Value(\n    DeciderConstants.enableEarlybirdTrafficDeciderKey)\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider/EndpointLoadShedder.scala",
    "content": "package com.twitter.cr_mixer.param.decider\n\nimport com.twitter.decider.Decider\nimport com.twitter.decider.RandomRecipient\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport scala.util.control.NoStackTrace\n\n/*\n  Provides deciders-controlled load shedding for a given Product from a given endpoint.\n  The format of the decider keys is:\n\n    enable_loadshedding_<endpoint name>_<product name>\n  E.g.:\n    enable_loadshedding_getTweetRecommendations_Notifications\n\n  Deciders are fractional, so a value of 50.00 will drop 50% of responses. If a decider key is not\n  defined for a particular endpoint/product combination, those requests will always be\n  served.\n\n  We should therefore aim to define keys for the endpoints/product we care most about in decider.yml,\n  so that we can control them during incidents.\n */\ncase class EndpointLoadShedder @Inject() (\n  decider: Decider,\n  statsReceiver: StatsReceiver) {\n  import EndpointLoadShedder._\n\n  // Fall back to False for any undefined key\n  private val deciderWithFalseFallback: Decider = decider.orElse(Decider.False)\n  private val keyPrefix = \"enable_loadshedding\"\n  private val scopedStats = statsReceiver.scope(\"EndpointLoadShedder\")\n\n  def apply[T](endpointName: String, product: String)(serve: => Future[T]): Future[T] = {\n    /*\n    Checks if either per-product or top-level load shedding is enabled\n    If both are enabled at different percentages, load shedding will not be perfectly calculable due\n    to salting of hash (i.e. 25% load shed for Product x + 25% load shed for overall does not\n    result in 50% load shed for x)\n     */\n    val keyTyped = s\"${keyPrefix}_${endpointName}_$product\"\n    val keyTopLevel = s\"${keyPrefix}_${endpointName}\"\n\n    if (deciderWithFalseFallback.isAvailable(keyTopLevel, recipient = Some(RandomRecipient))) {\n      scopedStats.counter(keyTopLevel).incr\n      Future.exception(LoadSheddingException)\n    } else if (deciderWithFalseFallback.isAvailable(keyTyped, recipient = Some(RandomRecipient))) {\n      scopedStats.counter(keyTyped).incr\n      Future.exception(LoadSheddingException)\n    } else serve\n  }\n}\n\nobject EndpointLoadShedder {\n  object LoadSheddingException extends Exception with NoStackTrace\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/ranker/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"content-recommender/thrift/src/main/thrift:content-recommender-common-scala\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/config\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"decider/src/main/scala\",\n        \"frigate/frigate-common:base\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/base\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/util:stats_util\",\n        \"hydra/common/libraries/src/main/scala/com/twitter/hydra/common/model_config\",\n        \"hydra/partition/thrift/src/main/thrift:thrift-scala\",\n        \"hydra/root/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/core_workflows/user_model:user_model-scala\",\n        \"src/thrift/com/twitter/ml/api:data-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/ranker/DefaultRanker.scala",
    "content": "package com.twitter.cr_mixer.ranker\n\nimport com.twitter.cr_mixer.model.BlendedCandidate\nimport com.twitter.cr_mixer.model.RankedCandidate\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n/**\n * Keep the same order as the input.\n */\n@Singleton\nclass DefaultRanker() {\n  def rank(\n    candidates: Seq[BlendedCandidate],\n  ): Future[Seq[RankedCandidate]] = {\n    val candidateSize = candidates.size\n    val rankedCandidates = candidates.zipWithIndex.map {\n      case (candidate, index) =>\n        candidate.toRankedCandidate((candidateSize - index).toDouble)\n    }\n    Future.value(rankedCandidates)\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/ranker/SwitchRanker.scala",
    "content": "package com.twitter.cr_mixer.ranker\n\nimport com.twitter.cr_mixer.model.BlendedCandidate\nimport com.twitter.cr_mixer.model.CrCandidateGeneratorQuery\nimport com.twitter.cr_mixer.model.RankedCandidate\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.Future\nimport com.twitter.util.JavaTimer\nimport com.twitter.util.Time\nimport com.twitter.util.Timer\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * CR-Mixer internal ranker\n */\n@Singleton\nclass SwitchRanker @Inject() (\n  defaultRanker: DefaultRanker,\n  globalStats: StatsReceiver) {\n  private val stats: StatsReceiver = globalStats.scope(this.getClass.getCanonicalName)\n  implicit val timer: Timer = new JavaTimer(true)\n\n  def rank(\n    query: CrCandidateGeneratorQuery,\n    candidates: Seq[BlendedCandidate],\n  ): Future[Seq[RankedCandidate]] = {\n    defaultRanker.rank(candidates)\n  }\n\n}\n\nobject SwitchRanker {\n\n  /** Prefers candidates generated from sources with the latest timestamps.\n   * The newer the source signal, the higher a candidate ranks.\n   * This ordering biases against consumer-based candidates because their timestamp defaults to 0\n   */\n  val TimestampOrder: Ordering[RankedCandidate] =\n    math.Ordering\n      .by[RankedCandidate, Time](\n        _.reasonChosen.sourceInfoOpt\n          .flatMap(_.sourceEventTime)\n          .getOrElse(Time.fromMilliseconds(0L)))\n      .reverse\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/scribe/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"decider/src/main/scala\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finagle/finagle-core/src/main\",\n        \"frigate/frigate-common:base\",\n        \"frigate/frigate-common:util\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/base\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"scrooge/scrooge-serializer/src/main/scala\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"util-internal/scribe/src/main/scala/com/twitter/logging\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/scribe/ScribeCategory.scala",
    "content": "package com.twitter.cr_mixer.scribe\n\n/**\n * Categories define scribe categories used in cr-mixer service.\n */\nobject ScribeCategories {\n  lazy val AllCategories =\n    List(AbDecider, TopLevelApiDdgMetrics, TweetsRecs)\n\n  /**\n   * AbDecider represents scribe logs for experiments\n   */\n  lazy val AbDecider: ScribeCategory = ScribeCategory(\n    \"abdecider_scribe\",\n    \"client_event\"\n  )\n\n  /**\n   * Top-Level Client event scribe logs, to record changes in system metrics (e.g. latency,\n   * candidates returned, empty rate ) per experiment bucket, and store them in DDG metric group\n   */\n  lazy val TopLevelApiDdgMetrics: ScribeCategory = ScribeCategory(\n    \"top_level_api_ddg_metrics_scribe\",\n    \"client_event\"\n  )\n\n  lazy val TweetsRecs: ScribeCategory = ScribeCategory(\n    \"get_tweets_recommendations_scribe\",\n    \"cr_mixer_get_tweets_recommendations\"\n  )\n\n  lazy val VITTweetsRecs: ScribeCategory = ScribeCategory(\n    \"get_vit_tweets_recommendations_scribe\",\n    \"cr_mixer_get_vit_tweets_recommendations\"\n  )\n\n  lazy val RelatedTweets: ScribeCategory = ScribeCategory(\n    \"get_related_tweets_scribe\",\n    \"cr_mixer_get_related_tweets\"\n  )\n\n  lazy val UtegTweets: ScribeCategory = ScribeCategory(\n    \"get_uteg_tweets_scribe\",\n    \"cr_mixer_get_uteg_tweets\"\n  )\n\n  lazy val AdsRecommendations: ScribeCategory = ScribeCategory(\n    \"get_ads_recommendations_scribe\",\n    \"cr_mixer_get_ads_recommendations\"\n  )\n}\n\n/**\n * Category represents each scribe log data.\n *\n * @param loggerFactoryNode loggerFactory node name in cr-mixer associated with this scribe category\n * @param scribeCategory    scribe category name (globally unique at Twitter)\n */\ncase class ScribeCategory(\n  loggerFactoryNode: String,\n  scribeCategory: String) {\n  def getProdLoggerFactoryNode: String = loggerFactoryNode\n  def getStagingLoggerFactoryNode: String = \"staging_\" + loggerFactoryNode\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/service/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/service/CrMixerAlertNotificationConfig.scala",
    "content": "package com.twitter.cr_mixer.service\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.Destination\nimport com.twitter.product_mixer.core.functional_component.common.alert.NotificationGroup\n\n/**\n * Notifications (email, pagerduty, etc) can be specific per-alert but it is common for multiple\n * products to share notification configuration.\n *\n * Our configuration uses only email notifications because SampleMixer is a demonstration service\n * with neither internal nor customer-facing users. You will likely want to use a PagerDuty\n * destination instead. For example:\n * {{{\n *   critical = Destination(pagerDutyKey = Some(\"your-pagerduty-key\"))\n * }}}\n *\n *\n * For more information about how to get a PagerDuty key, see:\n * https://docbird.twitter.biz/mon/how-to-guides.html?highlight=notificationgroup#set-up-email-pagerduty-and-slack-notifications\n */\nobject CrMixerAlertNotificationConfig {\n  val DefaultNotificationGroup: NotificationGroup = NotificationGroup(\n    warn = Destination(emails = Seq(\"no-reply@twitter.com\")),\n    critical = Destination(emails = Seq(\"no-reply@twitter.com\"))\n  )\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/BUILD",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/com/twitter/storehaus:memcache\",\n        \"3rdparty/jvm/io/grpc:grpc-api\",\n        \"3rdparty/jvm/io/grpc:grpc-auth\",\n        \"3rdparty/jvm/io/grpc:grpc-core\",\n        \"3rdparty/jvm/io/grpc:grpc-netty\",\n        \"3rdparty/jvm/io/grpc:grpc-protobuf\",\n        \"3rdparty/jvm/io/grpc:grpc-stub\",\n        \"3rdparty/jvm/io/opil:tensorflow-serving-client\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"ann/src/main/scala/com/twitter/ann/hnsw\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"configapi/configapi-core\",\n        \"content-recommender/thrift/src/main/thrift:thrift-scala\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/config\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/exception\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"decider/src/main/scala\",\n        \"finagle-internal/finagle-grpc/src/main/scala\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/client\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"frigate/frigate-common:base\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/base\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/candidate\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/strato\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/util\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/util:stats_util\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"mediaservices/commons/src/main/scala:futuretracker\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"qig-ranker/thrift/src/main/thrift:thrift-scala\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/injection\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/simclustersann/multicluster\",\n        \"simclusters-ann/thrift/src/main/thrift:thrift-scala\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/java/com/twitter/search/queryparser/query/search:search-query-nodes\",\n        \"src/scala/com/twitter/cortex/ml/embeddings/common:Helpers\",\n        \"src/scala/com/twitter/ml/featurestore/lib\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/ml/api:embedding-scala\",\n        \"src/thrift/com/twitter/recos:recos-common-scala\",\n        \"src/thrift/com/twitter/recos/user_ad_graph:user_ad_graph-scala\",\n        \"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala\",\n        \"src/thrift/com/twitter/recos/user_tweet_graph:user_tweet_graph-scala\",\n        \"src/thrift/com/twitter/recos/user_video_graph:user_video_graph-scala\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"src/thrift/com/twitter/search/common:ranking-scala\",\n        \"src/thrift/com/twitter/search/query_interaction_graph/service:qig-service-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"src/thrift/com/twitter/topic_recos:topic_recos-thrift-scala\",\n        \"src/thrift/com/twitter/trends/trip_v1:trip-tweets-thrift-scala\",\n        \"src/thrift/com/twitter/twistly:twistly-scala\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/CertoTopicTweetSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.google.inject.name.Named\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TopicTweetWithScore\nimport com.twitter.cr_mixer.param.TopicTweetParams\nimport com.twitter.cr_mixer.similarity_engine.CertoTopicTweetSimilarityEngine._\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.thriftscala.TopicId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.topic_recos.thriftscala._\nimport com.twitter.util.Future\n\n@Singleton\ncase class CertoTopicTweetSimilarityEngine @Inject() (\n  @Named(ModuleNames.CertoStratoStoreName) certoStratoStore: ReadableStore[\n    TopicId,\n    Seq[TweetWithScores]\n  ],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[EngineQuery[Query], Seq[TopicTweetWithScore]] {\n\n  private val name: String = this.getClass.getSimpleName\n  private val stats = statsReceiver.scope(name)\n\n  override def get(query: EngineQuery[Query]): Future[Option[Seq[TopicTweetWithScore]]] = {\n    StatsUtil.trackOptionItemsStats(stats) {\n      topTweetsByFollowerL2NormalizedScore.get(query).map {\n        _.map { topicTopTweets =>\n          topicTopTweets.map { topicTweet =>\n            TopicTweetWithScore(\n              tweetId = topicTweet.tweetId,\n              score = topicTweet.scores.followerL2NormalizedCosineSimilarity8HrHalfLife,\n              similarityEngineType = SimilarityEngineType.CertoTopicTweet\n            )\n          }\n        }\n      }\n    }\n  }\n\n  private val topTweetsByFollowerL2NormalizedScore: ReadableStore[EngineQuery[Query], Seq[\n    TweetWithScores\n  ]] = {\n    ReadableStore.fromFnFuture { query: EngineQuery[Query] =>\n      StatsUtil.trackOptionItemsStats(stats) {\n        for {\n          topKTweetsWithScores <- certoStratoStore.get(query.storeQuery.topicId)\n        } yield {\n          topKTweetsWithScores.map(\n            _.filter(\n              _.scores.followerL2NormalizedCosineSimilarity8HrHalfLife >= query.storeQuery.certoScoreTheshold)\n              .take(query.storeQuery.maxCandidates))\n        }\n      }\n    }\n  }\n}\n\nobject CertoTopicTweetSimilarityEngine {\n\n  // Query is used as a cache key. Do not add any user level information in this.\n  case class Query(\n    topicId: TopicId,\n    maxCandidates: Int,\n    certoScoreTheshold: Double)\n\n  def fromParams(\n    topicId: TopicId,\n    isVideoOnly: Boolean,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n\n    val maxCandidates = if (isVideoOnly) {\n      params(TopicTweetParams.MaxCertoCandidatesParam) * 2\n    } else {\n      params(TopicTweetParams.MaxCertoCandidatesParam)\n    }\n\n    EngineQuery(\n      Query(\n        topicId = topicId,\n        maxCandidates = maxCandidates,\n        certoScoreTheshold = params(TopicTweetParams.CertoScoreThresholdParam)\n      ),\n      params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/ConsumerBasedWalsSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.ConsumerBasedWalsParams\nimport com.twitter.cr_mixer.similarity_engine.ConsumerBasedWalsSimilarityEngine.Query\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Future\nimport io.grpc.ManagedChannel\nimport tensorflow.serving.Predict.PredictRequest\nimport tensorflow.serving.Predict.PredictResponse\nimport tensorflow.serving.PredictionServiceGrpc\nimport org.tensorflow.example.Feature\nimport org.tensorflow.example.Int64List\nimport org.tensorflow.example.FloatList\nimport org.tensorflow.example.Features\nimport org.tensorflow.example.Example\nimport tensorflow.serving.Model\nimport org.tensorflow.framework.TensorProto\nimport org.tensorflow.framework.DataType\nimport org.tensorflow.framework.TensorShapeProto\nimport com.twitter.finagle.grpc.FutureConverters\nimport java.util.ArrayList\nimport java.lang\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport java.util.concurrent.ConcurrentHashMap\nimport scala.jdk.CollectionConverters._\n\n// Stats object maintain a set of stats that are specific to the Wals Engine.\ncase class WalsStats(scope: String, scopedStats: StatsReceiver) {\n\n  val requestStat = scopedStats.scope(scope)\n  val inputSignalSize = requestStat.stat(\"input_signal_size\")\n\n  val latency = requestStat.stat(\"latency_ms\")\n  val latencyOnError = requestStat.stat(\"error_latency_ms\")\n  val latencyOnSuccess = requestStat.stat(\"success_latency_ms\")\n\n  val requests = requestStat.counter(\"requests\")\n  val success = requestStat.counter(\"success\")\n  val failures = requestStat.scope(\"failures\")\n\n  def onFailure(t: Throwable, startTimeMs: Long) {\n    val duration = System.currentTimeMillis() - startTimeMs\n    latency.add(duration)\n    latencyOnError.add(duration)\n    failures.counter(t.getClass.getName).incr()\n  }\n\n  def onSuccess(startTimeMs: Long) {\n    val duration = System.currentTimeMillis() - startTimeMs\n    latency.add(duration)\n    latencyOnSuccess.add(duration)\n    success.incr()\n  }\n}\n\n// StatsMap maintains a mapping from Model's input signature to a stats receiver\n// The Wals model suports multiple input signature which can run different graphs internally and\n// can have a different performance profile.\n// Invoking StatsReceiver.stat() on each request can create a new stat object and can be expensive\n// in performance critical paths.\nobject WalsStatsMap {\n  val mapping = new ConcurrentHashMap[String, WalsStats]()\n\n  def get(scope: String, scopedStats: StatsReceiver): WalsStats = {\n    mapping.computeIfAbsent(scope, (scope) => WalsStats(scope, scopedStats))\n  }\n}\n\ncase class ConsumerBasedWalsSimilarityEngine(\n  homeNaviGRPCClient: ManagedChannel,\n  adsFavedNaviGRPCClient: ManagedChannel,\n  adsMonetizableNaviGRPCClient: ManagedChannel,\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[\n      Query,\n      Seq[TweetWithScore]\n    ] {\n\n  override def get(\n    query: ConsumerBasedWalsSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    val startTimeMs = System.currentTimeMillis()\n    val stats =\n      WalsStatsMap.get(\n        query.wilyNsName + \"/\" + query.modelSignatureName,\n        statsReceiver.scope(\"NaviPredictionService\")\n      )\n    stats.requests.incr()\n    stats.inputSignalSize.add(query.sourceIds.size)\n    try {\n      // avoid inference calls is source signals are empty\n      if (query.sourceIds.isEmpty) {\n        Future.value(Some(Seq.empty))\n      } else {\n        val grpcClient = query.wilyNsName match {\n          case \"navi-wals-recommended-tweets-home-client\" => homeNaviGRPCClient\n          case \"navi-wals-ads-faved-tweets\" => adsFavedNaviGRPCClient\n          case \"navi-wals-ads-monetizable-tweets\" => adsFavedNaviGRPCClient\n          // default to homeNaviGRPCClient\n          case _ => homeNaviGRPCClient\n        }\n        val stub = PredictionServiceGrpc.newFutureStub(grpcClient)\n        val inferRequest = getModelInput(query)\n\n        FutureConverters\n          .RichListenableFuture(stub.predict(inferRequest)).toTwitter\n          .transform {\n            case Return(resp) =>\n              stats.onSuccess(startTimeMs)\n              Future.value(Some(getModelOutput(query, resp)))\n            case Throw(e) =>\n              stats.onFailure(e, startTimeMs)\n              Future.exception(e)\n          }\n      }\n    } catch {\n      case e: Throwable => Future.exception(e)\n    }\n  }\n\n  def getFeaturesForRecommendations(query: ConsumerBasedWalsSimilarityEngine.Query): Example = {\n    val tweetIds = new ArrayList[lang.Long]()\n    val tweetFaveWeight = new ArrayList[lang.Float]()\n\n    query.sourceIds.foreach { sourceInfo =>\n      val weight = sourceInfo.sourceType match {\n        case SourceType.TweetFavorite | SourceType.Retweet => 1.0f\n        // currently no-op - as we do not get negative signals\n        case SourceType.TweetDontLike | SourceType.TweetReport | SourceType.AccountMute |\n            SourceType.AccountBlock =>\n          0.0f\n        case _ => 0.0f\n      }\n      sourceInfo.internalId match {\n        case InternalId.TweetId(tweetId) =>\n          tweetIds.add(tweetId)\n          tweetFaveWeight.add(weight)\n        case _ =>\n          throw new IllegalArgumentException(\n            s\"Invalid InternalID - does not contain TweetId for Source Signal: ${sourceInfo}\")\n      }\n    }\n\n    val tweetIdsFeature =\n      Feature\n        .newBuilder().setInt64List(\n          Int64List\n            .newBuilder().addAllValue(tweetIds).build()\n        ).build()\n\n    val tweetWeightsFeature = Feature\n      .newBuilder().setFloatList(\n        FloatList.newBuilder().addAllValue(tweetFaveWeight).build()).build()\n\n    val features = Features\n      .newBuilder()\n      .putFeature(\"tweet_ids\", tweetIdsFeature)\n      .putFeature(\"tweet_weights\", tweetWeightsFeature)\n      .build()\n    Example.newBuilder().setFeatures(features).build()\n  }\n\n  def getModelInput(query: ConsumerBasedWalsSimilarityEngine.Query): PredictRequest = {\n    val tfExample = getFeaturesForRecommendations(query)\n\n    val inferenceRequest = PredictRequest\n      .newBuilder()\n      .setModelSpec(\n        Model.ModelSpec\n          .newBuilder()\n          .setName(query.modelName)\n          .setSignatureName(query.modelSignatureName))\n      .putInputs(\n        query.modelInputName,\n        TensorProto\n          .newBuilder()\n          .setDtype(DataType.DT_STRING)\n          .setTensorShape(TensorShapeProto\n            .newBuilder()\n            .addDim(TensorShapeProto.Dim.newBuilder().setSize(1)))\n          .addStringVal(tfExample.toByteString)\n          .build()\n      ).build()\n    inferenceRequest\n  }\n\n  def getModelOutput(query: Query, response: PredictResponse): Seq[TweetWithScore] = {\n    val outputName = query.modelOutputName\n    if (response.containsOutputs(outputName)) {\n      val tweetList = response.getOutputsMap\n        .get(outputName)\n        .getInt64ValList.asScala\n      tweetList.zip(tweetList.size to 1 by -1).map { (tweetWithScore) =>\n        TweetWithScore(tweetWithScore._1, tweetWithScore._2.toLong)\n      }\n    } else {\n      Seq.empty\n    }\n  }\n}\n\nobject ConsumerBasedWalsSimilarityEngine {\n  case class Query(\n    sourceIds: Seq[SourceInfo],\n    modelName: String,\n    modelInputName: String,\n    modelOutputName: String,\n    modelSignatureName: String,\n    wilyNsName: String,\n  )\n\n  def fromParams(\n    sourceIds: Seq[SourceInfo],\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    EngineQuery(\n      Query(\n        sourceIds,\n        params(ConsumerBasedWalsParams.ModelNameParam),\n        params(ConsumerBasedWalsParams.ModelInputNameParam),\n        params(ConsumerBasedWalsParams.ModelOutputNameParam),\n        params(ConsumerBasedWalsParams.ModelSignatureNameParam),\n        params(ConsumerBasedWalsParams.WilyNsNameParam),\n      ),\n      params\n    )\n  }\n\n  def toSimilarityEngineInfo(\n    score: Double\n  ): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.ConsumerBasedWalsANN,\n      modelId = None,\n      score = Some(score))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/ConsumerEmbeddingBasedTripSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.TripTweetWithScore\nimport com.twitter.cr_mixer.param.ConsumerEmbeddingBasedTripParams\nimport com.twitter.cr_mixer.util.InterleaveUtil\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.common.ClusterId\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.Cluster\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.ClusterDomain\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripTweet\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripDomain\nimport com.twitter.util.Future\n\ncase class TripEngineQuery(\n  modelId: String,\n  sourceId: InternalId,\n  tripSourceId: String,\n  maxResult: Int,\n  params: Params)\n\ncase class ConsumerEmbeddingBasedTripSimilarityEngine(\n  embeddingStoreLookUpMap: Map[String, ReadableStore[UserId, SimClustersEmbedding]],\n  tripCandidateSource: ReadableStore[TripDomain, Seq[TripTweet]],\n  statsReceiver: StatsReceiver,\n) extends ReadableStore[TripEngineQuery, Seq[TripTweetWithScore]] {\n  import ConsumerEmbeddingBasedTripSimilarityEngine._\n\n  private val scopedStats = statsReceiver.scope(name)\n  private def fetchTopClusters(query: TripEngineQuery): Future[Option[Seq[ClusterId]]] = {\n    query.sourceId match {\n      case InternalId.UserId(userId) =>\n        val embeddingStore = embeddingStoreLookUpMap.getOrElse(\n          query.modelId,\n          throw new IllegalArgumentException(\n            s\"${this.getClass.getSimpleName}: \" +\n              s\"ModelId ${query.modelId} does not exist for embeddingStore\"\n          )\n        )\n        embeddingStore.get(userId).map(_.map(_.topClusterIds(MaxClusters)))\n      case _ =>\n        Future.None\n    }\n  }\n  private def fetchCandidates(\n    topClusters: Seq[ClusterId],\n    tripSourceId: String\n  ): Future[Seq[Seq[TripTweetWithScore]]] = {\n    Future\n      .collect {\n        topClusters.map { clusterId =>\n          tripCandidateSource\n            .get(\n              TripDomain(\n                sourceId = tripSourceId,\n                clusterDomain = Some(\n                  ClusterDomain(simCluster = Some(Cluster(clusterIntId = Some(clusterId))))))).map {\n              _.map {\n                _.collect {\n                  case TripTweet(tweetId, score) =>\n                    TripTweetWithScore(tweetId, score)\n                }\n              }.getOrElse(Seq.empty).take(MaxNumResultsPerCluster)\n            }\n        }\n      }\n  }\n\n  override def get(engineQuery: TripEngineQuery): Future[Option[Seq[TripTweetWithScore]]] = {\n    val fetchTopClustersStat = scopedStats.scope(engineQuery.modelId).scope(\"fetchTopClusters\")\n    val fetchCandidatesStat = scopedStats.scope(engineQuery.modelId).scope(\"fetchCandidates\")\n\n    for {\n      topClustersOpt <- StatsUtil.trackOptionStats(fetchTopClustersStat) {\n        fetchTopClusters(engineQuery)\n      }\n      candidates <- StatsUtil.trackItemsStats(fetchCandidatesStat) {\n        topClustersOpt match {\n          case Some(topClusters) => fetchCandidates(topClusters, engineQuery.tripSourceId)\n          case None => Future.Nil\n        }\n      }\n    } yield {\n      val interleavedTweets = InterleaveUtil.interleave(candidates)\n      val dedupCandidates = interleavedTweets\n        .groupBy(_.tweetId).flatMap {\n          case (_, tweetWithScoreSeq) => tweetWithScoreSeq.sortBy(-_.score).take(1)\n        }.toSeq.take(engineQuery.maxResult)\n      Some(dedupCandidates)\n    }\n  }\n}\n\nobject ConsumerEmbeddingBasedTripSimilarityEngine {\n  private val MaxClusters: Int = 8\n  private val MaxNumResultsPerCluster: Int = 25\n  private val name: String = this.getClass.getSimpleName\n\n  def fromParams(\n    modelId: String,\n    sourceId: InternalId,\n    params: configapi.Params\n  ): TripEngineQuery = {\n    TripEngineQuery(\n      modelId = modelId,\n      sourceId = sourceId,\n      tripSourceId = params(ConsumerEmbeddingBasedTripParams.SourceIdParam),\n      maxResult = params(ConsumerEmbeddingBasedTripParams.MaxNumCandidatesParam),\n      params = params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/ConsumerEmbeddingBasedTwHINSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.param.ConsumerEmbeddingBasedTwHINParams\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.timelines.configapi\n\nobject ConsumerEmbeddingBasedTwHINSimilarityEngine {\n  def fromParams(\n    sourceId: InternalId,\n    params: configapi.Params,\n  ): HnswANNEngineQuery = {\n    HnswANNEngineQuery(\n      sourceId = sourceId,\n      modelId = params(ConsumerEmbeddingBasedTwHINParams.ModelIdParam),\n      params = params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/ConsumerEmbeddingBasedTwoTowerSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.param.ConsumerEmbeddingBasedTwoTowerParams\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.timelines.configapi\n\nobject ConsumerEmbeddingBasedTwoTowerSimilarityEngine {\n  def fromParams(\n    sourceId: InternalId,\n    params: configapi.Params,\n  ): HnswANNEngineQuery = {\n    HnswANNEngineQuery(\n      sourceId = sourceId,\n      modelId = params(ConsumerEmbeddingBasedTwoTowerParams.ModelIdParam),\n      params = params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/ConsumersBasedUserAdGraphSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.ConsumersBasedUserAdGraphParams\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.user_ad_graph.thriftscala.ConsumersBasedRelatedAdRequest\nimport com.twitter.recos.user_ad_graph.thriftscala.RelatedAdResponse\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n/**\n * This store uses the graph based input (a list of userIds)\n * to query consumersBasedUserAdGraph and get their top engaged ad tweets\n */\n@Singleton\ncase class ConsumersBasedUserAdGraphSimilarityEngine(\n  consumersBasedUserAdGraphStore: ReadableStore[\n    ConsumersBasedRelatedAdRequest,\n    RelatedAdResponse\n  ],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[\n      ConsumersBasedUserAdGraphSimilarityEngine.Query,\n      Seq[TweetWithScore]\n    ] {\n\n  override def get(\n    query: ConsumersBasedUserAdGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    val consumersBasedRelatedAdRequest =\n      ConsumersBasedRelatedAdRequest(\n        query.seedWithScores.keySet.toSeq,\n        maxResults = Some(query.maxResults),\n        minCooccurrence = Some(query.minCooccurrence),\n        minScore = Some(query.minScore),\n        maxTweetAgeInHours = Some(query.maxTweetAgeInHours)\n      )\n    consumersBasedUserAdGraphStore\n      .get(consumersBasedRelatedAdRequest)\n      .map { relatedAdResponseOpt =>\n        relatedAdResponseOpt.map { relatedAdResponse =>\n          relatedAdResponse.adTweets.map { tweet =>\n            TweetWithScore(tweet.adTweetId, tweet.score)\n          }\n        }\n      }\n  }\n}\n\nobject ConsumersBasedUserAdGraphSimilarityEngine {\n\n  case class Query(\n    seedWithScores: Map[UserId, Double],\n    maxResults: Int,\n    minCooccurrence: Int,\n    minScore: Double,\n    maxTweetAgeInHours: Int)\n\n  def toSimilarityEngineInfo(\n    score: Double\n  ): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.ConsumersBasedUserAdGraph,\n      modelId = None,\n      score = Some(score))\n  }\n\n  def fromParams(\n    seedWithScores: Map[UserId, Double],\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n\n    EngineQuery(\n      Query(\n        seedWithScores = seedWithScores,\n        maxResults = params(GlobalParams.MaxCandidateNumPerSourceKeyParam),\n        minCooccurrence = params(ConsumersBasedUserAdGraphParams.MinCoOccurrenceParam),\n        minScore = params(ConsumersBasedUserAdGraphParams.MinScoreParam),\n        maxTweetAgeInHours = params(GlobalParams.MaxTweetAgeHoursParam).inHours,\n      ),\n      params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/ConsumersBasedUserVideoGraphSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.ConsumersBasedUserVideoGraphParams\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.user_video_graph.thriftscala.ConsumersBasedRelatedTweetRequest\nimport com.twitter.recos.user_video_graph.thriftscala.RelatedTweetResponse\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n/**\n * This store uses the graph based input (a list of userIds)\n * to query consumersBasedUserVideoGraph and get their top engaged tweets\n */\n@Singleton\ncase class ConsumersBasedUserVideoGraphSimilarityEngine(\n  consumersBasedUserVideoGraphStore: ReadableStore[\n    ConsumersBasedRelatedTweetRequest,\n    RelatedTweetResponse\n  ],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[\n      ConsumersBasedUserVideoGraphSimilarityEngine.Query,\n      Seq[TweetWithScore]\n    ] {\n\n  override def get(\n    query: ConsumersBasedUserVideoGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    val consumersBasedRelatedTweetRequest =\n      ConsumersBasedRelatedTweetRequest(\n        query.seedWithScores.keySet.toSeq,\n        maxResults = Some(query.maxResults),\n        minCooccurrence = Some(query.minCooccurrence),\n        minScore = Some(query.minScore),\n        maxTweetAgeInHours = Some(query.maxTweetAgeInHours)\n      )\n    consumersBasedUserVideoGraphStore\n      .get(consumersBasedRelatedTweetRequest)\n      .map { relatedTweetResponseOpt =>\n        relatedTweetResponseOpt.map { relatedTweetResponse =>\n          relatedTweetResponse.tweets.map { tweet =>\n            TweetWithScore(tweet.tweetId, tweet.score)\n          }\n        }\n      }\n  }\n}\n\nobject ConsumersBasedUserVideoGraphSimilarityEngine {\n\n  case class Query(\n    seedWithScores: Map[UserId, Double],\n    maxResults: Int,\n    minCooccurrence: Int,\n    minScore: Double,\n    maxTweetAgeInHours: Int)\n\n  def toSimilarityEngineInfo(\n    score: Double\n  ): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.ConsumersBasedUserVideoGraph,\n      modelId = None,\n      score = Some(score))\n  }\n\n  def fromParamsForRealGraphIn(\n    seedWithScores: Map[UserId, Double],\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n\n    EngineQuery(\n      Query(\n        seedWithScores = seedWithScores,\n        maxResults = params(GlobalParams.MaxCandidateNumPerSourceKeyParam),\n        minCooccurrence =\n          params(ConsumersBasedUserVideoGraphParams.RealGraphInMinCoOccurrenceParam),\n        minScore = params(ConsumersBasedUserVideoGraphParams.RealGraphInMinScoreParam),\n        maxTweetAgeInHours = params(GlobalParams.MaxTweetAgeHoursParam).inHours\n      ),\n      params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/DiffusionBasedSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.simclusters_v2.thriftscala.TweetsWithScore\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n@Singleton\ncase class DiffusionBasedSimilarityEngine(\n  retweetBasedDiffusionRecsMhStore: ReadableStore[Long, TweetsWithScore],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[\n      DiffusionBasedSimilarityEngine.Query,\n      Seq[TweetWithScore]\n    ] {\n\n  override def get(\n    query: DiffusionBasedSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n\n    query.sourceId match {\n      case InternalId.UserId(userId) =>\n        retweetBasedDiffusionRecsMhStore.get(userId).map {\n          _.map { tweetsWithScore =>\n            {\n              tweetsWithScore.tweets\n                .map(tweet => TweetWithScore(tweet.tweetId, tweet.score))\n            }\n          }\n        }\n      case _ =>\n        Future.None\n    }\n  }\n}\n\nobject DiffusionBasedSimilarityEngine {\n\n  val defaultScore: Double = 0.0\n\n  case class Query(\n    sourceId: InternalId,\n  )\n\n  def toSimilarityEngineInfo(\n    query: LookupEngineQuery[Query],\n    score: Double\n  ): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.DiffusionBasedTweet,\n      modelId = Some(query.lookupKey),\n      score = Some(score))\n  }\n\n  def fromParams(\n    sourceId: InternalId,\n    modelId: String,\n    params: configapi.Params,\n  ): LookupEngineQuery[Query] = {\n    LookupEngineQuery(\n      Query(sourceId = sourceId),\n      modelId,\n      params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/EarlybirdModelBasedSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.similarity_engine.EarlybirdModelBasedSimilarityEngine.EarlybirdModelBasedSearchQuery\nimport com.twitter.cr_mixer.similarity_engine.EarlybirdSimilarityEngineBase._\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.EarlybirdClientId\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.FacetsToFetch\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.MetadataOptions\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.search.common.ranking.thriftscala.ThriftRankingParams\nimport com.twitter.search.common.ranking.thriftscala.ThriftScoringFunctionType\nimport com.twitter.search.earlybird.thriftscala.EarlybirdRequest\nimport com.twitter.search.earlybird.thriftscala.EarlybirdService\nimport com.twitter.search.earlybird.thriftscala.ThriftSearchQuery\nimport com.twitter.search.earlybird.thriftscala.ThriftSearchRankingMode\nimport com.twitter.search.earlybird.thriftscala.ThriftSearchRelevanceOptions\nimport com.twitter.simclusters_v2.common.UserId\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class EarlybirdModelBasedSimilarityEngine @Inject() (\n  earlybirdSearchClient: EarlybirdService.MethodPerEndpoint,\n  timeoutConfig: TimeoutConfig,\n  stats: StatsReceiver)\n    extends EarlybirdSimilarityEngineBase[EarlybirdModelBasedSearchQuery] {\n  import EarlybirdModelBasedSimilarityEngine._\n  override val statsReceiver: StatsReceiver = stats.scope(this.getClass.getSimpleName)\n  override def getEarlybirdRequest(\n    query: EarlybirdModelBasedSearchQuery\n  ): Option[EarlybirdRequest] =\n    if (query.seedUserIds.nonEmpty)\n      Some(\n        EarlybirdRequest(\n          searchQuery = getThriftSearchQuery(query),\n          clientId = Some(EarlybirdClientId),\n          timeoutMs = timeoutConfig.earlybirdServerTimeout.inMilliseconds.intValue(),\n          clientRequestID = Some(s\"${Trace.id.traceId}\"),\n        ))\n    else None\n}\n\nobject EarlybirdModelBasedSimilarityEngine {\n  case class EarlybirdModelBasedSearchQuery(\n    seedUserIds: Seq[UserId],\n    maxNumTweets: Int,\n    oldestTweetTimestampInSec: Option[UserId],\n    frsUserToScoresForScoreAdjustment: Option[Map[UserId, Double]])\n      extends EarlybirdSearchQuery\n\n  /**\n   * Used by Push Service\n   */\n  val RealGraphScoringModel = \"frigate_unified_engagement_rg\"\n  val MaxHitsToProcess = 1000\n  val MaxConsecutiveSameUser = 1\n\n  private def getModelBasedRankingParams(\n    authorSpecificScoreAdjustments: Map[Long, Double]\n  ): ThriftRankingParams = ThriftRankingParams(\n    `type` = Some(ThriftScoringFunctionType.ModelBased),\n    selectedModels = Some(Map(RealGraphScoringModel -> 1.0)),\n    applyBoosts = false,\n    authorSpecificScoreAdjustments = Some(authorSpecificScoreAdjustments)\n  )\n\n  private def getRelevanceOptions(\n    authorSpecificScoreAdjustments: Map[Long, Double],\n  ): ThriftSearchRelevanceOptions = {\n    ThriftSearchRelevanceOptions(\n      maxConsecutiveSameUser = Some(MaxConsecutiveSameUser),\n      rankingParams = Some(getModelBasedRankingParams(authorSpecificScoreAdjustments)),\n      maxHitsToProcess = Some(MaxHitsToProcess),\n      orderByRelevance = true\n    )\n  }\n\n  private def getThriftSearchQuery(query: EarlybirdModelBasedSearchQuery): ThriftSearchQuery =\n    ThriftSearchQuery(\n      serializedQuery = Some(f\"(* [since_time ${query.oldestTweetTimestampInSec.getOrElse(0)}])\"),\n      fromUserIDFilter64 = Some(query.seedUserIds),\n      numResults = query.maxNumTweets,\n      maxHitsToProcess = MaxHitsToProcess,\n      rankingMode = ThriftSearchRankingMode.Relevance,\n      relevanceOptions =\n        Some(getRelevanceOptions(query.frsUserToScoresForScoreAdjustment.getOrElse(Map.empty))),\n      facetFieldNames = Some(FacetsToFetch),\n      resultMetadataOptions = Some(MetadataOptions),\n      searcherId = None\n    )\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/EarlybirdRecencyBasedSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TweetWithAuthor\nimport com.twitter.cr_mixer.similarity_engine.EarlybirdRecencyBasedSimilarityEngine.EarlybirdRecencyBasedSearchQuery\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\ncase class EarlybirdRecencyBasedSimilarityEngine @Inject() (\n  @Named(ModuleNames.EarlybirdRecencyBasedWithoutRetweetsRepliesTweetsCache)\n  earlybirdRecencyBasedWithoutRetweetsRepliesTweetsCacheStore: ReadableStore[\n    UserId,\n    Seq[TweetId]\n  ],\n  @Named(ModuleNames.EarlybirdRecencyBasedWithRetweetsRepliesTweetsCache)\n  earlybirdRecencyBasedWithRetweetsRepliesTweetsCacheStore: ReadableStore[\n    UserId,\n    Seq[TweetId]\n  ],\n  timeoutConfig: TimeoutConfig,\n  stats: StatsReceiver)\n    extends ReadableStore[EarlybirdRecencyBasedSearchQuery, Seq[TweetWithAuthor]] {\n  import EarlybirdRecencyBasedSimilarityEngine._\n  val statsReceiver: StatsReceiver = stats.scope(this.getClass.getSimpleName)\n\n  override def get(\n    query: EarlybirdRecencyBasedSearchQuery\n  ): Future[Option[Seq[TweetWithAuthor]]] = {\n    Future\n      .collect {\n        if (query.filterOutRetweetsAndReplies) {\n          query.seedUserIds.map { seedUserId =>\n            StatsUtil.trackOptionItemsStats(statsReceiver.scope(\"WithoutRetweetsAndReplies\")) {\n              earlybirdRecencyBasedWithoutRetweetsRepliesTweetsCacheStore\n                .get(seedUserId).map(_.map(_.map(tweetId =>\n                  TweetWithAuthor(tweetId = tweetId, authorId = seedUserId))))\n            }\n          }\n        } else {\n          query.seedUserIds.map { seedUserId =>\n            StatsUtil.trackOptionItemsStats(statsReceiver.scope(\"WithRetweetsAndReplies\")) {\n              earlybirdRecencyBasedWithRetweetsRepliesTweetsCacheStore\n                .get(seedUserId)\n                .map(_.map(_.map(tweetId =>\n                  TweetWithAuthor(tweetId = tweetId, authorId = seedUserId))))\n            }\n          }\n        }\n      }\n      .map { tweetWithAuthorList =>\n        val earliestTweetId = SnowflakeId.firstIdFor(Time.now - query.maxTweetAge)\n        tweetWithAuthorList\n          .flatMap(_.getOrElse(Seq.empty))\n          .filter(tweetWithAuthor =>\n            tweetWithAuthor.tweetId >= earliestTweetId // tweet age filter\n              && !query.excludedTweetIds\n                .contains(tweetWithAuthor.tweetId)) // excluded tweet filter\n          .sortBy(tweetWithAuthor =>\n            -SnowflakeId.unixTimeMillisFromId(tweetWithAuthor.tweetId)) // sort by recency\n          .take(query.maxNumTweets) // take most recent N tweets\n      }\n      .map(result => Some(result))\n  }\n\n}\n\nobject EarlybirdRecencyBasedSimilarityEngine {\n  case class EarlybirdRecencyBasedSearchQuery(\n    seedUserIds: Seq[UserId],\n    maxNumTweets: Int,\n    excludedTweetIds: Set[TweetId],\n    maxTweetAge: Duration,\n    filterOutRetweetsAndReplies: Boolean)\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/EarlybirdSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.TweetWithAuthor\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nclass EarlybirdSimilarityEngine[\n  Query,\n  EarlybirdSimilarityEngineStore <: ReadableStore[Query, Seq[TweetWithAuthor]]\n](\n  implementingStore: EarlybirdSimilarityEngineStore,\n  override val identifier: SimilarityEngineType,\n  globalStats: StatsReceiver,\n  engineConfig: SimilarityEngineConfig,\n) extends SimilarityEngine[EngineQuery[Query], TweetWithAuthor] {\n  private val scopedStats = globalStats.scope(\"similarityEngine\", identifier.toString)\n\n  def getScopedStats: StatsReceiver = scopedStats\n\n  def getCandidates(query: EngineQuery[Query]): Future[Option[Seq[TweetWithAuthor]]] = {\n    SimilarityEngine.getFromFn(\n      implementingStore.get,\n      query.storeQuery,\n      engineConfig,\n      query.params,\n      scopedStats\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/EarlybirdSimilarityEngineBase.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.TweetWithAuthor\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.search.earlybird.thriftscala.EarlybirdRequest\nimport com.twitter.search.earlybird.thriftscala.EarlybirdResponseCode\nimport com.twitter.search.earlybird.thriftscala.EarlybirdService\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\n/**\n * This trait is a base trait for Earlybird similarity engines. All Earlybird similarity\n * engines extend from it and override the construction method for EarlybirdRequest\n */\ntrait EarlybirdSimilarityEngineBase[EarlybirdSearchQuery]\n    extends ReadableStore[EarlybirdSearchQuery, Seq[TweetWithAuthor]] {\n  def earlybirdSearchClient: EarlybirdService.MethodPerEndpoint\n\n  def statsReceiver: StatsReceiver\n\n  def getEarlybirdRequest(query: EarlybirdSearchQuery): Option[EarlybirdRequest]\n\n  override def get(query: EarlybirdSearchQuery): Future[Option[Seq[TweetWithAuthor]]] = {\n    getEarlybirdRequest(query)\n      .map { earlybirdRequest =>\n        earlybirdSearchClient\n          .search(earlybirdRequest).map { response =>\n            response.responseCode match {\n              case EarlybirdResponseCode.Success =>\n                val earlybirdSearchResult =\n                  response.searchResults\n                    .map(\n                      _.results\n                        .map(searchResult =>\n                          TweetWithAuthor(\n                            searchResult.id,\n                            // fromUserId should be there since MetadataOptions.getFromUserId = true\n                            searchResult.metadata.map(_.fromUserId).getOrElse(0))).toSeq)\n                statsReceiver.scope(\"result\").stat(\"size\").add(earlybirdSearchResult.size)\n                earlybirdSearchResult\n              case e =>\n                statsReceiver.scope(\"failures\").counter(e.getClass.getSimpleName).incr()\n                Some(Seq.empty)\n            }\n          }\n      }.getOrElse(Future.None)\n  }\n}\n\nobject EarlybirdSimilarityEngineBase {\n  trait EarlybirdSearchQuery {\n    def seedUserIds: Seq[UserId]\n    def maxNumTweets: Int\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/EarlybirdSimilarityEngineRouter.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.EarlybirdSimilarityEngineType\nimport com.twitter.cr_mixer.model.EarlybirdSimilarityEngineType_ModelBased\nimport com.twitter.cr_mixer.model.EarlybirdSimilarityEngineType_RecencyBased\nimport com.twitter.cr_mixer.model.EarlybirdSimilarityEngineType_TensorflowBased\nimport com.twitter.cr_mixer.model.TweetWithAuthor\nimport com.twitter.cr_mixer.param.EarlybirdFrsBasedCandidateGenerationParams\nimport com.twitter.cr_mixer.param.EarlybirdFrsBasedCandidateGenerationParams.FrsBasedCandidateGenerationEarlybirdSimilarityEngineTypeParam\nimport com.twitter.cr_mixer.param.FrsParams.FrsBasedCandidateGenerationMaxCandidatesNumParam\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class EarlybirdSimilarityEngineRouter @Inject() (\n  earlybirdRecencyBasedSimilarityEngine: EarlybirdSimilarityEngine[\n    EarlybirdRecencyBasedSimilarityEngine.EarlybirdRecencyBasedSearchQuery,\n    EarlybirdRecencyBasedSimilarityEngine\n  ],\n  earlybirdModelBasedSimilarityEngine: EarlybirdSimilarityEngine[\n    EarlybirdModelBasedSimilarityEngine.EarlybirdModelBasedSearchQuery,\n    EarlybirdModelBasedSimilarityEngine\n  ],\n  earlybirdTensorflowBasedSimilarityEngine: EarlybirdSimilarityEngine[\n    EarlybirdTensorflowBasedSimilarityEngine.EarlybirdTensorflowBasedSearchQuery,\n    EarlybirdTensorflowBasedSimilarityEngine\n  ],\n  timeoutConfig: TimeoutConfig,\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[EarlybirdSimilarityEngineRouter.Query, Seq[TweetWithAuthor]] {\n  import EarlybirdSimilarityEngineRouter._\n\n  override def get(\n    k: EarlybirdSimilarityEngineRouter.Query\n  ): Future[Option[Seq[TweetWithAuthor]]] = {\n    k.rankingMode match {\n      case EarlybirdSimilarityEngineType_RecencyBased =>\n        earlybirdRecencyBasedSimilarityEngine.getCandidates(recencyBasedQueryFromParams(k))\n      case EarlybirdSimilarityEngineType_ModelBased =>\n        earlybirdModelBasedSimilarityEngine.getCandidates(modelBasedQueryFromParams(k))\n      case EarlybirdSimilarityEngineType_TensorflowBased =>\n        earlybirdTensorflowBasedSimilarityEngine.getCandidates(tensorflowBasedQueryFromParams(k))\n    }\n  }\n}\n\nobject EarlybirdSimilarityEngineRouter {\n  case class Query(\n    searcherUserId: Option[UserId],\n    seedUserIds: Seq[UserId],\n    maxNumTweets: Int,\n    excludedTweetIds: Set[TweetId],\n    rankingMode: EarlybirdSimilarityEngineType,\n    frsUserToScoresForScoreAdjustment: Option[Map[UserId, Double]],\n    maxTweetAge: Duration,\n    filterOutRetweetsAndReplies: Boolean,\n    params: configapi.Params)\n\n  def queryFromParams(\n    searcherUserId: Option[UserId],\n    seedUserIds: Seq[UserId],\n    excludedTweetIds: Set[TweetId],\n    frsUserToScoresForScoreAdjustment: Option[Map[UserId, Double]],\n    params: configapi.Params\n  ): Query =\n    Query(\n      searcherUserId,\n      seedUserIds,\n      maxNumTweets = params(FrsBasedCandidateGenerationMaxCandidatesNumParam),\n      excludedTweetIds,\n      rankingMode =\n        params(FrsBasedCandidateGenerationEarlybirdSimilarityEngineTypeParam).rankingMode,\n      frsUserToScoresForScoreAdjustment,\n      maxTweetAge = params(\n        EarlybirdFrsBasedCandidateGenerationParams.FrsBasedCandidateGenerationEarlybirdMaxTweetAge),\n      filterOutRetweetsAndReplies = params(\n        EarlybirdFrsBasedCandidateGenerationParams.FrsBasedCandidateGenerationEarlybirdFilterOutRetweetsAndReplies),\n      params\n    )\n\n  private def recencyBasedQueryFromParams(\n    query: Query\n  ): EngineQuery[EarlybirdRecencyBasedSimilarityEngine.EarlybirdRecencyBasedSearchQuery] =\n    EngineQuery(\n      EarlybirdRecencyBasedSimilarityEngine.EarlybirdRecencyBasedSearchQuery(\n        seedUserIds = query.seedUserIds,\n        maxNumTweets = query.maxNumTweets,\n        excludedTweetIds = query.excludedTweetIds,\n        maxTweetAge = query.maxTweetAge,\n        filterOutRetweetsAndReplies = query.filterOutRetweetsAndReplies\n      ),\n      query.params\n    )\n\n  private def tensorflowBasedQueryFromParams(\n    query: Query,\n  ): EngineQuery[EarlybirdTensorflowBasedSimilarityEngine.EarlybirdTensorflowBasedSearchQuery] =\n    EngineQuery(\n      EarlybirdTensorflowBasedSimilarityEngine.EarlybirdTensorflowBasedSearchQuery(\n        searcherUserId = query.searcherUserId,\n        seedUserIds = query.seedUserIds,\n        maxNumTweets = query.maxNumTweets,\n        // hard code the params below for now. Will move to FS after shipping the ddg\n        beforeTweetIdExclusive = None,\n        afterTweetIdExclusive =\n          Some(SnowflakeId.firstIdFor((Time.now - query.maxTweetAge).inMilliseconds)),\n        filterOutRetweetsAndReplies = query.filterOutRetweetsAndReplies,\n        useTensorflowRanking = true,\n        excludedTweetIds = query.excludedTweetIds,\n        maxNumHitsPerShard = 1000\n      ),\n      query.params\n    )\n  private def modelBasedQueryFromParams(\n    query: Query,\n  ): EngineQuery[EarlybirdModelBasedSimilarityEngine.EarlybirdModelBasedSearchQuery] =\n    EngineQuery(\n      EarlybirdModelBasedSimilarityEngine.EarlybirdModelBasedSearchQuery(\n        seedUserIds = query.seedUserIds,\n        maxNumTweets = query.maxNumTweets,\n        oldestTweetTimestampInSec = Some(query.maxTweetAge.ago.inSeconds),\n        frsUserToScoresForScoreAdjustment = query.frsUserToScoresForScoreAdjustment\n      ),\n      query.params\n    )\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/EarlybirdTensorflowBasedSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.search.earlybird.thriftscala.EarlybirdRequest\nimport com.twitter.search.earlybird.thriftscala.EarlybirdService\nimport com.twitter.search.earlybird.thriftscala.ThriftSearchQuery\nimport com.twitter.util.Time\nimport com.twitter.search.common.query.thriftjava.thriftscala.CollectorParams\nimport com.twitter.search.common.ranking.thriftscala.ThriftRankingParams\nimport com.twitter.search.common.ranking.thriftscala.ThriftScoringFunctionType\nimport com.twitter.search.earlybird.thriftscala.ThriftSearchRelevanceOptions\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport EarlybirdSimilarityEngineBase._\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.similarity_engine.EarlybirdTensorflowBasedSimilarityEngine.EarlybirdTensorflowBasedSearchQuery\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.EarlybirdClientId\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.FacetsToFetch\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.GetCollectorTerminationParams\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.GetEarlybirdQuery\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.MetadataOptions\nimport com.twitter.cr_mixer.util.EarlybirdSearchUtil.GetNamedDisjunctions\nimport com.twitter.search.earlybird.thriftscala.ThriftSearchRankingMode\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.util.Duration\n\n@Singleton\ncase class EarlybirdTensorflowBasedSimilarityEngine @Inject() (\n  earlybirdSearchClient: EarlybirdService.MethodPerEndpoint,\n  timeoutConfig: TimeoutConfig,\n  stats: StatsReceiver)\n    extends EarlybirdSimilarityEngineBase[EarlybirdTensorflowBasedSearchQuery] {\n  import EarlybirdTensorflowBasedSimilarityEngine._\n  override val statsReceiver: StatsReceiver = stats.scope(this.getClass.getSimpleName)\n  override def getEarlybirdRequest(\n    query: EarlybirdTensorflowBasedSearchQuery\n  ): Option[EarlybirdRequest] = {\n    if (query.seedUserIds.nonEmpty)\n      Some(\n        EarlybirdRequest(\n          searchQuery = getThriftSearchQuery(query, timeoutConfig.earlybirdServerTimeout),\n          clientHost = None,\n          clientRequestID = None,\n          clientId = Some(EarlybirdClientId),\n          clientRequestTimeMs = Some(Time.now.inMilliseconds),\n          cachingParams = None,\n          timeoutMs = timeoutConfig.earlybirdServerTimeout.inMilliseconds.intValue(),\n          facetRequest = None,\n          termStatisticsRequest = None,\n          debugMode = 0,\n          debugOptions = None,\n          searchSegmentId = None,\n          returnStatusType = None,\n          successfulResponseThreshold = None,\n          querySource = None,\n          getOlderResults = Some(false),\n          followedUserIds = Some(query.seedUserIds),\n          adjustedProtectedRequestParams = None,\n          adjustedFullArchiveRequestParams = None,\n          getProtectedTweetsOnly = Some(false),\n          retokenizeSerializedQuery = None,\n          skipVeryRecentTweets = true,\n          experimentClusterToUse = None\n        ))\n    else None\n  }\n}\n\nobject EarlybirdTensorflowBasedSimilarityEngine {\n  case class EarlybirdTensorflowBasedSearchQuery(\n    searcherUserId: Option[UserId],\n    seedUserIds: Seq[UserId],\n    maxNumTweets: Int,\n    beforeTweetIdExclusive: Option[TweetId],\n    afterTweetIdExclusive: Option[TweetId],\n    filterOutRetweetsAndReplies: Boolean,\n    useTensorflowRanking: Boolean,\n    excludedTweetIds: Set[TweetId],\n    maxNumHitsPerShard: Int)\n      extends EarlybirdSearchQuery\n\n  private def getThriftSearchQuery(\n    query: EarlybirdTensorflowBasedSearchQuery,\n    processingTimeout: Duration\n  ): ThriftSearchQuery =\n    ThriftSearchQuery(\n      serializedQuery = GetEarlybirdQuery(\n        query.beforeTweetIdExclusive,\n        query.afterTweetIdExclusive,\n        query.excludedTweetIds,\n        query.filterOutRetweetsAndReplies).map(_.serialize),\n      fromUserIDFilter64 = Some(query.seedUserIds),\n      numResults = query.maxNumTweets,\n      // Whether to collect conversation IDs. Remove it for now.\n      // collectConversationId = Gate.True(), // true for Home\n      rankingMode = ThriftSearchRankingMode.Relevance,\n      relevanceOptions = Some(getRelevanceOptions),\n      collectorParams = Some(\n        CollectorParams(\n          // numResultsToReturn defines how many results each EB shard will return to search root\n          numResultsToReturn = 1000,\n          // terminationParams.maxHitsToProcess is used for early terminating per shard results fetching.\n          terminationParams =\n            GetCollectorTerminationParams(query.maxNumHitsPerShard, processingTimeout)\n        )),\n      facetFieldNames = Some(FacetsToFetch),\n      resultMetadataOptions = Some(MetadataOptions),\n      searcherId = query.searcherUserId,\n      searchStatusIds = None,\n      namedDisjunctionMap = GetNamedDisjunctions(query.excludedTweetIds)\n    )\n\n  // The specific values of recap relevance/reranking options correspond to\n  // experiment: enable_recap_reranking_2988,timeline_internal_disable_recap_filter\n  // bucket    : enable_rerank,disable_filter\n  private def getRelevanceOptions: ThriftSearchRelevanceOptions = {\n    ThriftSearchRelevanceOptions(\n      proximityScoring = true,\n      maxConsecutiveSameUser = Some(2),\n      rankingParams = Some(getTensorflowBasedRankingParams),\n      maxHitsToProcess = Some(500),\n      maxUserBlendCount = Some(3),\n      proximityPhraseWeight = 9.0,\n      returnAllResults = Some(true)\n    )\n  }\n\n  private def getTensorflowBasedRankingParams: ThriftRankingParams = {\n    ThriftRankingParams(\n      `type` = Some(ThriftScoringFunctionType.TensorflowBased),\n      selectedTensorflowModel = Some(\"timelines_rectweet_replica\"),\n      minScore = -1.0e100,\n      applyBoosts = false,\n      authorSpecificScoreAdjustments = None\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/FilterUtil.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\n\nobject FilterUtil {\n\n  /** Returns a list of tweets that are generated less than `maxTweetAgeHours` hours ago */\n  def tweetAgeFilter(\n    candidates: Seq[TweetWithScore],\n    maxTweetAgeHours: Duration\n  ): Seq[TweetWithScore] = {\n    // Tweet IDs are approximately chronological (see http://go/snowflake),\n    // so we are building the earliest tweet id once\n    // The per-candidate logic here then be candidate.tweetId > earliestPermittedTweetId, which is far cheaper.\n    // See @cyao's phab on CrMixer generic age filter for reference https://phabricator.twitter.biz/D903188\n    val earliestTweetId = SnowflakeId.firstIdFor(Time.now - maxTweetAgeHours)\n    candidates.filter { candidate => candidate.tweetId >= earliestTweetId }\n  }\n\n  /** Returns a list of tweet sources that are generated less than `maxTweetAgeHours` hours ago */\n  def tweetSourceAgeFilter(\n    candidates: Seq[SourceInfo],\n    maxTweetSignalAgeHoursParam: Duration\n  ): Seq[SourceInfo] = {\n    // Tweet IDs are approximately chronological (see http://go/snowflake),\n    // so we are building the earliest tweet id once\n    // This filter applies to source signals. Some candidate source calls can be avoided if source signals\n    // can be filtered.\n    val earliestTweetId = SnowflakeId.firstIdFor(Time.now - maxTweetSignalAgeHoursParam)\n    candidates.filter { candidate =>\n      candidate.internalId match {\n        case InternalId.TweetId(tweetId) => tweetId >= earliestTweetId\n        case _ => false\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/HnswANNSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.ann.common.thriftscala.AnnQueryService\nimport com.twitter.ann.common.thriftscala.Distance\nimport com.twitter.ann.common.thriftscala.NearestNeighborQuery\nimport com.twitter.ann.hnsw.HnswCommon\nimport com.twitter.ann.hnsw.HnswParams\nimport com.twitter.bijection.Injection\nimport com.twitter.cortex.ml.embeddings.common.TweetKind\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.MemCacheConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.mediaservices.commons.codec.ArrayByteBufferCodec\nimport com.twitter.ml.api.thriftscala.{Embedding => ThriftEmbedding}\nimport com.twitter.ml.featurestore.lib\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.Future\n\ncase class HnswANNEngineQuery(\n  modelId: String,\n  sourceId: InternalId,\n  params: Params,\n) {\n  val cacheKey: String = s\"${modelId}_${sourceId.toString}\"\n}\n\n/**\n * This Engine looks for tweets whose similarity is close to a Source Dense Embedding.\n * Only support Long based embedding lookup. UserId or TweetId.\n *\n * It provides HNSW specific implementations\n *\n * @param memCacheConfigOpt   If specified, it will wrap the underlying store with a MemCache layer\n *                            You should only enable this for cacheable queries, e.x. TweetIds.\n *                            consumer based UserIds are generally not possible to cache.\n */\nclass HnswANNSimilarityEngine(\n  embeddingStoreLookUpMap: Map[String, ReadableStore[InternalId, ThriftEmbedding]],\n  annServiceLookUpMap: Map[String, AnnQueryService.MethodPerEndpoint],\n  globalStats: StatsReceiver,\n  override val identifier: SimilarityEngineType,\n  engineConfig: SimilarityEngineConfig,\n  memCacheConfigOpt: Option[MemCacheConfig[HnswANNEngineQuery]] = None)\n    extends SimilarityEngine[HnswANNEngineQuery, TweetWithScore] {\n\n  private val MaxNumResults: Int = 200\n  private val ef: Int = 800\n  private val TweetIdByteInjection: Injection[lib.TweetId, Array[Byte]] = TweetKind.byteInjection\n\n  private val scopedStats = globalStats.scope(\"similarityEngine\", identifier.toString)\n\n  def getScopedStats: StatsReceiver = scopedStats\n\n  private def fetchEmbedding(\n    query: HnswANNEngineQuery,\n  ): Future[Option[ThriftEmbedding]] = {\n    val embeddingStore = embeddingStoreLookUpMap.getOrElse(\n      query.modelId,\n      throw new IllegalArgumentException(\n        s\"${this.getClass.getSimpleName} ${identifier.toString}: \" +\n          s\"ModelId ${query.modelId} does not exist for embeddingStore\"\n      )\n    )\n\n    embeddingStore.get(query.sourceId)\n  }\n\n  private def fetchCandidates(\n    query: HnswANNEngineQuery,\n    embedding: ThriftEmbedding\n  ): Future[Seq[TweetWithScore]] = {\n    val annService = annServiceLookUpMap.getOrElse(\n      query.modelId,\n      throw new IllegalArgumentException(\n        s\"${this.getClass.getSimpleName} ${identifier.toString}: \" +\n          s\"ModelId ${query.modelId} does not exist for annStore\"\n      )\n    )\n\n    val hnswParams = HnswCommon.RuntimeParamsInjection.apply(HnswParams(ef))\n\n    val annQuery =\n      NearestNeighborQuery(embedding, withDistance = true, hnswParams, MaxNumResults)\n\n    annService\n      .query(annQuery)\n      .map(\n        _.nearestNeighbors\n          .map { nearestNeighbor =>\n            val candidateId = TweetIdByteInjection\n              .invert(ArrayByteBufferCodec.decode(nearestNeighbor.id))\n              .toOption\n              .map(_.tweetId)\n            (candidateId, nearestNeighbor.distance)\n          }.collect {\n            case (Some(candidateId), Some(distance)) =>\n              TweetWithScore(candidateId, toScore(distance))\n          })\n  }\n\n  // Convert Distance to a score such that higher scores mean more similar.\n  def toScore(distance: Distance): Double = {\n    distance match {\n      case Distance.EditDistance(editDistance) =>\n        // (-Infinite, 0.0]\n        0.0 - editDistance.distance\n      case Distance.L2Distance(l2Distance) =>\n        // (-Infinite, 0.0]\n        0.0 - l2Distance.distance\n      case Distance.CosineDistance(cosineDistance) =>\n        // [0.0 - 1.0]\n        1.0 - cosineDistance.distance\n      case Distance.InnerProductDistance(innerProductDistance) =>\n        // (-Infinite, Infinite)\n        1.0 - innerProductDistance.distance\n      case Distance.UnknownUnionField(_) =>\n        throw new IllegalStateException(\n          s\"${this.getClass.getSimpleName} does not recognize $distance.toString\"\n        )\n    }\n  }\n\n  private[similarity_engine] def getEmbeddingAndCandidates(\n    query: HnswANNEngineQuery\n  ): Future[Option[Seq[TweetWithScore]]] = {\n\n    val fetchEmbeddingStat = scopedStats.scope(query.modelId).scope(\"fetchEmbedding\")\n    val fetchCandidatesStat = scopedStats.scope(query.modelId).scope(\"fetchCandidates\")\n\n    for {\n      embeddingOpt <- StatsUtil.trackOptionStats(fetchEmbeddingStat) { fetchEmbedding(query) }\n      candidates <- StatsUtil.trackItemsStats(fetchCandidatesStat) {\n\n        embeddingOpt match {\n          case Some(embedding) => fetchCandidates(query, embedding)\n          case None => Future.Nil\n        }\n      }\n    } yield {\n      Some(candidates)\n    }\n  }\n\n  // Add memcache wrapper, if specified\n  private val store = {\n    val uncachedStore = ReadableStore.fromFnFuture(getEmbeddingAndCandidates)\n\n    memCacheConfigOpt match {\n      case Some(config) =>\n        SimilarityEngine.addMemCache(\n          underlyingStore = uncachedStore,\n          memCacheConfig = config,\n          statsReceiver = scopedStats\n        )\n      case _ => uncachedStore\n    }\n  }\n\n  def toSimilarityEngineInfo(\n    query: HnswANNEngineQuery,\n    score: Double\n  ): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = this.identifier,\n      modelId = Some(query.modelId),\n      score = Some(score))\n  }\n\n  override def getCandidates(\n    engineQuery: HnswANNEngineQuery\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    val versionedStats = globalStats.scope(engineQuery.modelId)\n    SimilarityEngine.getFromFn(\n      store.get,\n      engineQuery,\n      engineConfig,\n      engineQuery.params,\n      versionedStats\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/LookupSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.MemCacheConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.Future\n\ncase class LookupEngineQuery[Query](\n  storeQuery: Query, // the actual Query type of the underlying store\n  lookupKey: String,\n  params: Params,\n)\n\n/**\n * This Engine provides a map interface for looking up different model implementations.\n * It provides modelId level monitoring for free.\n *\n * Example use cases include OfflineSimClusters lookup\n *\n *\n * @param versionedStoreMap   A mapping from a modelId to a corresponding implementation\n * @param memCacheConfigOpt   If specified, it will wrap the underlying store with a MemCache layer\n *                            You should only enable this for cacheable queries, e.x. TweetIds.\n *                            consumer based UserIds are generally not possible to cache.\n */\nclass LookupSimilarityEngine[Query, Candidate <: Serializable](\n  versionedStoreMap: Map[String, ReadableStore[Query, Seq[Candidate]]], // key = modelId\n  override val identifier: SimilarityEngineType,\n  globalStats: StatsReceiver,\n  engineConfig: SimilarityEngineConfig,\n  memCacheConfigOpt: Option[MemCacheConfig[Query]] = None)\n    extends SimilarityEngine[LookupEngineQuery[Query], Candidate] {\n\n  private val scopedStats = globalStats.scope(\"similarityEngine\", identifier.toString)\n\n  private val underlyingLookupMap = {\n    memCacheConfigOpt match {\n      case Some(config) =>\n        versionedStoreMap.map {\n          case (modelId, store) =>\n            (\n              modelId,\n              SimilarityEngine.addMemCache(\n                underlyingStore = store,\n                memCacheConfig = config,\n                keyPrefix = Some(modelId),\n                statsReceiver = scopedStats\n              )\n            )\n        }\n      case _ => versionedStoreMap\n    }\n  }\n\n  override def getCandidates(\n    engineQuery: LookupEngineQuery[Query]\n  ): Future[Option[Seq[Candidate]]] = {\n    val versionedStore =\n      underlyingLookupMap\n        .getOrElse(\n          engineQuery.lookupKey,\n          throw new IllegalArgumentException(\n            s\"${this.getClass.getSimpleName} ${identifier.toString}: ModelId ${engineQuery.lookupKey} does not exist\"\n          )\n        )\n\n    SimilarityEngine.getFromFn(\n      fn = versionedStore.get,\n      storeQuery = engineQuery.storeQuery,\n      engineConfig = engineConfig,\n      params = engineQuery.params,\n      scopedStats = scopedStats.scope(engineQuery.lookupKey)\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/ModelBasedANNStore.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.ann.common.thriftscala.AnnQueryService\nimport com.twitter.ann.common.thriftscala.Distance\nimport com.twitter.ann.common.thriftscala.NearestNeighborQuery\nimport com.twitter.ann.common.thriftscala.NearestNeighborResult\nimport com.twitter.ann.hnsw.HnswCommon\nimport com.twitter.ann.hnsw.HnswParams\nimport com.twitter.bijection.Injection\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cortex.ml.embeddings.common.TweetKind\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.mediaservices.commons.codec.ArrayByteBufferCodec\nimport com.twitter.ml.api.thriftscala.{Embedding => ThriftEmbedding}\nimport com.twitter.ml.featurestore.lib\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n/**\n * This store looks for tweets whose similarity is close to a Source Dense Embedding.\n * Only support Long based embedding lookup. UserId or TweetId\n */\n@Singleton\nclass ModelBasedANNStore(\n  embeddingStoreLookUpMap: Map[String, ReadableStore[InternalId, ThriftEmbedding]],\n  annServiceLookUpMap: Map[String, AnnQueryService.MethodPerEndpoint],\n  globalStats: StatsReceiver)\n    extends ReadableStore[\n      ModelBasedANNStore.Query,\n      Seq[TweetWithScore]\n    ] {\n\n  import ModelBasedANNStore._\n\n  private val stats = globalStats.scope(this.getClass.getSimpleName)\n  private val fetchEmbeddingStat = stats.scope(\"fetchEmbedding\")\n  private val fetchCandidatesStat = stats.scope(\"fetchCandidates\")\n\n  override def get(query: Query): Future[Option[Seq[TweetWithScore]]] = {\n    for {\n      maybeEmbedding <- StatsUtil.trackOptionStats(fetchEmbeddingStat.scope(query.modelId)) {\n        fetchEmbedding(query)\n      }\n      maybeCandidates <- StatsUtil.trackOptionStats(fetchCandidatesStat.scope(query.modelId)) {\n        maybeEmbedding match {\n          case Some(embedding) =>\n            fetchCandidates(query, embedding)\n          case None =>\n            Future.None\n        }\n      }\n    } yield {\n      maybeCandidates.map(\n        _.nearestNeighbors\n          .map { nearestNeighbor =>\n            val candidateId = TweetIdByteInjection\n              .invert(ArrayByteBufferCodec.decode(nearestNeighbor.id))\n              .toOption\n              .map(_.tweetId)\n            (candidateId, nearestNeighbor.distance)\n          }.collect {\n            case (Some(candidateId), Some(distance)) =>\n              TweetWithScore(candidateId, toScore(distance))\n          })\n    }\n  }\n\n  private def fetchEmbedding(query: Query): Future[Option[ThriftEmbedding]] = {\n    embeddingStoreLookUpMap.get(query.modelId) match {\n      case Some(embeddingStore) =>\n        embeddingStore.get(query.sourceId)\n      case _ =>\n        Future.None\n    }\n  }\n\n  private def fetchCandidates(\n    query: Query,\n    embedding: ThriftEmbedding\n  ): Future[Option[NearestNeighborResult]] = {\n    val hnswParams = HnswCommon.RuntimeParamsInjection.apply(HnswParams(query.ef))\n\n    annServiceLookUpMap.get(query.modelId) match {\n      case Some(annService) =>\n        val annQuery =\n          NearestNeighborQuery(embedding, withDistance = true, hnswParams, MaxNumResults)\n        annService.query(annQuery).map(v => Some(v))\n      case _ =>\n        Future.None\n    }\n  }\n}\n\nobject ModelBasedANNStore {\n\n  val MaxNumResults: Int = 200\n  val MaxTweetCandidateAge: Duration = 1.day\n\n  val TweetIdByteInjection: Injection[lib.TweetId, Array[Byte]] = TweetKind.byteInjection\n\n  // For more information about HNSW algorithm: https://docbird.twitter.biz/ann/hnsw.html\n  case class Query(\n    sourceId: InternalId,\n    modelId: String,\n    similarityEngineType: SimilarityEngineType,\n    ef: Int = 800)\n\n  def toScore(distance: Distance): Double = {\n    distance match {\n      case Distance.L2Distance(l2Distance) =>\n        // (-Infinite, 0.0]\n        0.0 - l2Distance.distance\n      case Distance.CosineDistance(cosineDistance) =>\n        // [0.0 - 1.0]\n        1.0 - cosineDistance.distance\n      case Distance.InnerProductDistance(innerProductDistance) =>\n        // (-Infinite, Infinite)\n        1.0 - innerProductDistance.distance\n      case _ =>\n        0.0\n    }\n  }\n  def toSimilarityEngineInfo(query: Query, score: Double): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = query.similarityEngineType,\n      modelId = Some(query.modelId),\n      score = Some(score))\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/ProducerBasedUnifiedSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.model.TweetWithCandidateGenerationInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.param.ProducerBasedCandidateGenerationParams\nimport com.twitter.cr_mixer.param.UnifiedSETweetCombinationMethod\nimport com.twitter.cr_mixer.param.RelatedTweetProducerBasedParams\nimport com.twitter.cr_mixer.param.SimClustersANNParams\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.cr_mixer.util.InterleaveUtil\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.common.ModelVersions\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.mutable.ArrayBuffer\n\n/**\n * This store looks for similar tweets from UserTweetGraph for a Source ProducerId\n * For a query producerId,User Tweet Graph (UTG),\n * lets us find out which tweets the query producer's followers co-engaged\n */\n@Singleton\ncase class ProducerBasedUnifiedSimilarityEngine(\n  @Named(ModuleNames.ProducerBasedUserTweetGraphSimilarityEngine)\n  producerBasedUserTweetGraphSimilarityEngine: StandardSimilarityEngine[\n    ProducerBasedUserTweetGraphSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  simClustersANNSimilarityEngine: StandardSimilarityEngine[\n    SimClustersANNSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[ProducerBasedUnifiedSimilarityEngine.Query, Seq[\n      TweetWithCandidateGenerationInfo\n    ]] {\n\n  import ProducerBasedUnifiedSimilarityEngine._\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val fetchCandidatesStat = stats.scope(\"fetchCandidates\")\n\n  override def get(\n    query: Query\n  ): Future[Option[Seq[TweetWithCandidateGenerationInfo]]] = {\n    query.sourceInfo.internalId match {\n      case _: InternalId.UserId =>\n        StatsUtil.trackOptionItemsStats(fetchCandidatesStat) {\n          val sannCandidatesFut = if (query.enableSimClustersANN) {\n            simClustersANNSimilarityEngine.getCandidates(query.simClustersANNQuery)\n          } else Future.None\n\n          val sann1CandidatesFut =\n            if (query.enableSimClustersANN1) {\n              simClustersANNSimilarityEngine.getCandidates(query.simClustersANN1Query)\n            } else Future.None\n\n          val sann2CandidatesFut =\n            if (query.enableSimClustersANN2) {\n              simClustersANNSimilarityEngine.getCandidates(query.simClustersANN2Query)\n            } else Future.None\n\n          val sann3CandidatesFut =\n            if (query.enableSimClustersANN3) {\n              simClustersANNSimilarityEngine.getCandidates(query.simClustersANN3Query)\n            } else Future.None\n\n          val sann4CandidatesFut =\n            if (query.enableSimClustersANN4) {\n              simClustersANNSimilarityEngine.getCandidates(query.simClustersANN4Query)\n            } else Future.None\n\n          val sann5CandidatesFut =\n            if (query.enableSimClustersANN5) {\n              simClustersANNSimilarityEngine.getCandidates(query.simClustersANN5Query)\n            } else Future.None\n\n          val experimentalSANNCandidatesFut =\n            if (query.enableExperimentalSimClustersANN) {\n              simClustersANNSimilarityEngine.getCandidates(query.experimentalSimClustersANNQuery)\n            } else Future.None\n\n          val utgCandidatesFut = if (query.enableUtg) {\n            producerBasedUserTweetGraphSimilarityEngine.getCandidates(query.utgQuery)\n          } else Future.None\n\n          Future\n            .join(\n              sannCandidatesFut,\n              sann1CandidatesFut,\n              sann2CandidatesFut,\n              sann3CandidatesFut,\n              sann4CandidatesFut,\n              sann5CandidatesFut,\n              experimentalSANNCandidatesFut,\n              utgCandidatesFut\n            ).map {\n              case (\n                    simClustersAnnCandidates,\n                    simClustersAnn1Candidates,\n                    simClustersAnn2Candidates,\n                    simClustersAnn3Candidates,\n                    simClustersAnn4Candidates,\n                    simClustersAnn5Candidates,\n                    experimentalSANNCandidates,\n                    userTweetGraphCandidates) =>\n                val filteredSANNTweets = simClustersCandidateMinScoreFilter(\n                  simClustersAnnCandidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.simClustersANNQuery.storeQuery.simClustersANNConfigId)\n\n                val filteredExperimentalSANNTweets = simClustersCandidateMinScoreFilter(\n                  experimentalSANNCandidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.experimentalSimClustersANNQuery.storeQuery.simClustersANNConfigId)\n\n                val filteredSANN1Tweets = simClustersCandidateMinScoreFilter(\n                  simClustersAnn1Candidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.simClustersANN1Query.storeQuery.simClustersANNConfigId)\n\n                val filteredSANN2Tweets = simClustersCandidateMinScoreFilter(\n                  simClustersAnn2Candidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.simClustersANN2Query.storeQuery.simClustersANNConfigId)\n\n                val filteredSANN3Tweets = simClustersCandidateMinScoreFilter(\n                  simClustersAnn3Candidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.simClustersANN3Query.storeQuery.simClustersANNConfigId)\n\n                val filteredSANN4Tweets = simClustersCandidateMinScoreFilter(\n                  simClustersAnn4Candidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.simClustersANN4Query.storeQuery.simClustersANNConfigId)\n\n                val filteredSANN5Tweets = simClustersCandidateMinScoreFilter(\n                  simClustersAnn5Candidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.simClustersANN5Query.storeQuery.simClustersANNConfigId)\n\n                val filteredUTGTweets =\n                  userTweetGraphFilter(userTweetGraphCandidates.toSeq.flatten)\n\n                val sannTweetsWithCGInfo = filteredSANNTweets.map { tweetWithScore =>\n                  val similarityEngineInfo = SimClustersANNSimilarityEngine\n                    .toSimilarityEngineInfo(query.simClustersANNQuery, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n                val sann1TweetsWithCGInfo = filteredSANN1Tweets.map { tweetWithScore =>\n                  val similarityEngineInfo = SimClustersANNSimilarityEngine\n                    .toSimilarityEngineInfo(query.simClustersANN1Query, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n                val sann2TweetsWithCGInfo = filteredSANN2Tweets.map { tweetWithScore =>\n                  val similarityEngineInfo = SimClustersANNSimilarityEngine\n                    .toSimilarityEngineInfo(query.simClustersANN2Query, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n\n                val sann3TweetsWithCGInfo = filteredSANN3Tweets.map { tweetWithScore =>\n                  val similarityEngineInfo = SimClustersANNSimilarityEngine\n                    .toSimilarityEngineInfo(query.simClustersANN3Query, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n\n                val sann4TweetsWithCGInfo = filteredSANN4Tweets.map { tweetWithScore =>\n                  val similarityEngineInfo = SimClustersANNSimilarityEngine\n                    .toSimilarityEngineInfo(query.simClustersANN4Query, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n\n                val sann5TweetsWithCGInfo = filteredSANN5Tweets.map { tweetWithScore =>\n                  val similarityEngineInfo = SimClustersANNSimilarityEngine\n                    .toSimilarityEngineInfo(query.simClustersANN5Query, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n\n                val experimentalSANNTweetsWithCGInfo = filteredExperimentalSANNTweets.map {\n                  tweetWithScore =>\n                    val similarityEngineInfo = SimClustersANNSimilarityEngine\n                      .toSimilarityEngineInfo(\n                        query.experimentalSimClustersANNQuery,\n                        tweetWithScore.score)\n                    TweetWithCandidateGenerationInfo(\n                      tweetWithScore.tweetId,\n                      CandidateGenerationInfo(\n                        Some(query.sourceInfo),\n                        similarityEngineInfo,\n                        Seq(similarityEngineInfo)\n                      ))\n                }\n                val utgTweetsWithCGInfo = filteredUTGTweets.map { tweetWithScore =>\n                  val similarityEngineInfo =\n                    ProducerBasedUserTweetGraphSimilarityEngine\n                      .toSimilarityEngineInfo(tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n\n                val candidateSourcesToBeInterleaved =\n                  ArrayBuffer[Seq[TweetWithCandidateGenerationInfo]](\n                    sannTweetsWithCGInfo,\n                    sann1TweetsWithCGInfo,\n                    sann2TweetsWithCGInfo,\n                    sann3TweetsWithCGInfo,\n                    sann4TweetsWithCGInfo,\n                    sann5TweetsWithCGInfo,\n                    experimentalSANNTweetsWithCGInfo,\n                  )\n\n                if (query.utgCombinationMethod == UnifiedSETweetCombinationMethod.Interleave) {\n                  candidateSourcesToBeInterleaved += utgTweetsWithCGInfo\n                }\n\n                val interleavedCandidates =\n                  InterleaveUtil.interleave(candidateSourcesToBeInterleaved)\n\n                val candidateSourcesToBeOrdered =\n                  ArrayBuffer[Seq[TweetWithCandidateGenerationInfo]](interleavedCandidates)\n\n                if (query.utgCombinationMethod == UnifiedSETweetCombinationMethod.Frontload)\n                  candidateSourcesToBeOrdered.prepend(utgTweetsWithCGInfo)\n\n                val candidatesFromGivenOrderCombination =\n                  SimilaritySourceOrderingUtil.keepGivenOrder(candidateSourcesToBeOrdered)\n\n                val unifiedCandidatesWithUnifiedCGInfo = candidatesFromGivenOrderCombination.map {\n                  candidate =>\n                    /***\n                     * when a candidate was made by interleave/keepGivenOrder,\n                     * then we apply getProducerBasedUnifiedCGInfo() to override with the unified CGInfo\n                     *\n                     * in contributingSE list for interleave. We only have the chosen SE available.\n                     * This is hard to add for interleave, and we plan to add it later after abstraction improvement.\n                     */\n                    TweetWithCandidateGenerationInfo(\n                      tweetId = candidate.tweetId,\n                      candidateGenerationInfo = getProducerBasedUnifiedCGInfo(\n                        candidate.candidateGenerationInfo.sourceInfoOpt,\n                        candidate.getSimilarityScore,\n                        candidate.candidateGenerationInfo.contributingSimilarityEngines\n                      ) // getSimilarityScore comes from either unifiedScore or single score\n                    )\n                }\n                stats.stat(\"unified_candidate_size\").add(unifiedCandidatesWithUnifiedCGInfo.size)\n                val truncatedCandidates =\n                  unifiedCandidatesWithUnifiedCGInfo.take(query.maxCandidateNumPerSourceKey)\n                stats.stat(\"truncatedCandidates_size\").add(truncatedCandidates.size)\n\n                Some(truncatedCandidates)\n\n            }\n        }\n\n      case _ =>\n        stats.counter(\"sourceId_is_not_userId_cnt\").incr()\n        Future.None\n    }\n  }\n\n  private def simClustersCandidateMinScoreFilter(\n    simClustersAnnCandidates: Seq[TweetWithScore],\n    simClustersMinScore: Double,\n    simClustersANNConfigId: String\n  ): Seq[TweetWithScore] = {\n    val filteredCandidates = simClustersAnnCandidates\n      .filter { candidate =>\n        candidate.score > simClustersMinScore\n      }\n\n    stats.stat(simClustersANNConfigId, \"simClustersAnnCandidates_size\").add(filteredCandidates.size)\n    stats.counter(simClustersANNConfigId, \"simClustersAnnRequests\").incr()\n    if (filteredCandidates.isEmpty)\n      stats.counter(simClustersANNConfigId, \"emptyFilteredSimClustersAnnCandidates\").incr()\n\n    filteredCandidates.map { candidate =>\n      TweetWithScore(candidate.tweetId, candidate.score)\n    }\n  }\n\n  /** A no-op filter as UTG filter already happened at UTG service side */\n  private def userTweetGraphFilter(\n    userTweetGraphCandidates: Seq[TweetWithScore]\n  ): Seq[TweetWithScore] = {\n    val filteredCandidates = userTweetGraphCandidates\n\n    stats.stat(\"userTweetGraphCandidates_size\").add(userTweetGraphCandidates.size)\n    if (filteredCandidates.isEmpty) stats.counter(\"emptyFilteredUserTweetGraphCandidates\").incr()\n\n    filteredCandidates.map { candidate =>\n      TweetWithScore(candidate.tweetId, candidate.score)\n    }\n  }\n\n}\nobject ProducerBasedUnifiedSimilarityEngine {\n\n  /***\n   * Every candidate will have the CG Info with ProducerBasedUnifiedSimilarityEngine\n   * as they are generated by a composite of Similarity Engines.\n   * Additionally, we store the contributing SEs (eg., SANN, UTG).\n   */\n  private def getProducerBasedUnifiedCGInfo(\n    sourceInfoOpt: Option[SourceInfo],\n    unifiedScore: Double,\n    contributingSimilarityEngines: Seq[SimilarityEngineInfo]\n  ): CandidateGenerationInfo = {\n    CandidateGenerationInfo(\n      sourceInfoOpt,\n      SimilarityEngineInfo(\n        similarityEngineType = SimilarityEngineType.ProducerBasedUnifiedSimilarityEngine,\n        modelId = None, // We do not assign modelId for a unified similarity engine\n        score = Some(unifiedScore)\n      ),\n      contributingSimilarityEngines\n    )\n  }\n\n  case class Query(\n    sourceInfo: SourceInfo,\n    maxCandidateNumPerSourceKey: Int,\n    maxTweetAgeHours: Duration,\n    // SimClusters\n    enableSimClustersANN: Boolean,\n    simClustersANNQuery: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    enableExperimentalSimClustersANN: Boolean,\n    experimentalSimClustersANNQuery: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    enableSimClustersANN1: Boolean,\n    simClustersANN1Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    enableSimClustersANN2: Boolean,\n    simClustersANN2Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    enableSimClustersANN4: Boolean,\n    simClustersANN4Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    enableSimClustersANN3: Boolean,\n    simClustersANN3Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    enableSimClustersANN5: Boolean,\n    simClustersANN5Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    simClustersMinScore: Double,\n    // UTG\n    enableUtg: Boolean,\n    utgCombinationMethod: UnifiedSETweetCombinationMethod.Value,\n    utgQuery: EngineQuery[ProducerBasedUserTweetGraphSimilarityEngine.Query])\n\n  def fromParams(\n    sourceInfo: SourceInfo,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    val maxCandidateNumPerSourceKey = params(GlobalParams.MaxCandidateNumPerSourceKeyParam)\n    val maxTweetAgeHours = params(GlobalParams.MaxTweetAgeHoursParam)\n    // SimClusters\n    val enableSimClustersANN = params(\n      ProducerBasedCandidateGenerationParams.EnableSimClustersANNParam)\n    val simClustersModelVersion =\n      ModelVersions.Enum.enumToSimClustersModelVersionMap(params(GlobalParams.ModelVersionParam))\n    val simClustersANNConfigId = params(SimClustersANNParams.SimClustersANNConfigId)\n    // SimClusters - Experimental SANN Similarity Engine\n    val enableExperimentalSimClustersANN = params(\n      ProducerBasedCandidateGenerationParams.EnableExperimentalSimClustersANNParam)\n    val experimentalSimClustersANNConfigId = params(\n      SimClustersANNParams.ExperimentalSimClustersANNConfigId)\n    // SimClusters - SANN cluster 1 Similarity Engine\n    val enableSimClustersANN1 = params(\n      ProducerBasedCandidateGenerationParams.EnableSimClustersANN1Param)\n    val simClustersANN1ConfigId = params(SimClustersANNParams.SimClustersANN1ConfigId)\n    // SimClusters - SANN cluster 2 Similarity Engine\n    val enableSimClustersANN2 = params(\n      ProducerBasedCandidateGenerationParams.EnableSimClustersANN2Param)\n    val simClustersANN2ConfigId = params(SimClustersANNParams.SimClustersANN2ConfigId)\n    // SimClusters - SANN cluster 3 Similarity Engine\n    val enableSimClustersANN3 = params(\n      ProducerBasedCandidateGenerationParams.EnableSimClustersANN3Param)\n    val simClustersANN3ConfigId = params(SimClustersANNParams.SimClustersANN3ConfigId)\n    // SimClusters - SANN cluster 5 Similarity Engine\n    val enableSimClustersANN5 = params(\n      ProducerBasedCandidateGenerationParams.EnableSimClustersANN5Param)\n    val simClustersANN5ConfigId = params(SimClustersANNParams.SimClustersANN5ConfigId)\n    val enableSimClustersANN4 = params(\n      ProducerBasedCandidateGenerationParams.EnableSimClustersANN4Param)\n    val simClustersANN4ConfigId = params(SimClustersANNParams.SimClustersANN4ConfigId)\n\n    val simClustersMinScore = params(\n      ProducerBasedCandidateGenerationParams.SimClustersMinScoreParam)\n\n    // SimClusters ANN Query\n    val simClustersANNQuery = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      simClustersANNConfigId,\n      params\n    )\n    val experimentalSimClustersANNQuery = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      experimentalSimClustersANNConfigId,\n      params\n    )\n    val simClustersANN1Query = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      simClustersANN1ConfigId,\n      params\n    )\n    val simClustersANN2Query = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      simClustersANN2ConfigId,\n      params\n    )\n    val simClustersANN3Query = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      simClustersANN3ConfigId,\n      params\n    )\n    val simClustersANN5Query = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      simClustersANN5ConfigId,\n      params\n    )\n    val simClustersANN4Query = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      simClustersANN4ConfigId,\n      params\n    )\n    // UTG\n    val enableUtg = params(ProducerBasedCandidateGenerationParams.EnableUTGParam)\n    val utgCombinationMethod = params(\n      ProducerBasedCandidateGenerationParams.UtgCombinationMethodParam)\n\n    EngineQuery(\n      Query(\n        sourceInfo = sourceInfo,\n        maxCandidateNumPerSourceKey = maxCandidateNumPerSourceKey,\n        maxTweetAgeHours = maxTweetAgeHours,\n        enableSimClustersANN = enableSimClustersANN,\n        simClustersANNQuery = simClustersANNQuery,\n        enableExperimentalSimClustersANN = enableExperimentalSimClustersANN,\n        experimentalSimClustersANNQuery = experimentalSimClustersANNQuery,\n        enableSimClustersANN1 = enableSimClustersANN1,\n        simClustersANN1Query = simClustersANN1Query,\n        enableSimClustersANN2 = enableSimClustersANN2,\n        simClustersANN2Query = simClustersANN2Query,\n        enableSimClustersANN3 = enableSimClustersANN3,\n        simClustersANN3Query = simClustersANN3Query,\n        enableSimClustersANN5 = enableSimClustersANN5,\n        simClustersANN5Query = simClustersANN5Query,\n        enableSimClustersANN4 = enableSimClustersANN4,\n        simClustersANN4Query = simClustersANN4Query,\n        simClustersMinScore = simClustersMinScore,\n        enableUtg = enableUtg,\n        utgCombinationMethod = utgCombinationMethod,\n        utgQuery = ProducerBasedUserTweetGraphSimilarityEngine\n          .fromParams(sourceInfo.internalId, params)\n      ),\n      params\n    )\n  }\n\n  def fromParamsForRelatedTweet(\n    internalId: InternalId,\n    params: configapi.Params\n  ): EngineQuery[Query] = {\n    val maxCandidateNumPerSourceKey = params(GlobalParams.MaxCandidateNumPerSourceKeyParam)\n    val maxTweetAgeHours = params(GlobalParams.MaxTweetAgeHoursParam)\n    // SimClusters\n    val enableSimClustersANN = params(RelatedTweetProducerBasedParams.EnableSimClustersANNParam)\n    val simClustersModelVersion =\n      ModelVersions.Enum.enumToSimClustersModelVersionMap(params(GlobalParams.ModelVersionParam))\n    val simClustersANNConfigId = params(SimClustersANNParams.SimClustersANNConfigId)\n    val simClustersMinScore =\n      params(RelatedTweetProducerBasedParams.SimClustersMinScoreParam)\n    // SimClusters - Experimental SANN Similarity Engine\n    val enableExperimentalSimClustersANN = params(\n      RelatedTweetProducerBasedParams.EnableExperimentalSimClustersANNParam)\n    val experimentalSimClustersANNConfigId = params(\n      SimClustersANNParams.ExperimentalSimClustersANNConfigId)\n    // SimClusters - SANN cluster 1 Similarity Engine\n    val enableSimClustersANN1 = params(RelatedTweetProducerBasedParams.EnableSimClustersANN1Param)\n    val simClustersANN1ConfigId = params(SimClustersANNParams.SimClustersANN1ConfigId)\n    // SimClusters - SANN cluster 2 Similarity Engine\n    val enableSimClustersANN2 = params(RelatedTweetProducerBasedParams.EnableSimClustersANN2Param)\n    val simClustersANN2ConfigId = params(SimClustersANNParams.SimClustersANN2ConfigId)\n    // SimClusters - SANN cluster 3 Similarity Engine\n    val enableSimClustersANN3 = params(RelatedTweetProducerBasedParams.EnableSimClustersANN3Param)\n    val simClustersANN3ConfigId = params(SimClustersANNParams.SimClustersANN3ConfigId)\n    // SimClusters - SANN cluster 5 Similarity Engine\n    val enableSimClustersANN5 = params(RelatedTweetProducerBasedParams.EnableSimClustersANN5Param)\n    val simClustersANN5ConfigId = params(SimClustersANNParams.SimClustersANN5ConfigId)\n\n    val enableSimClustersANN4 = params(RelatedTweetProducerBasedParams.EnableSimClustersANN4Param)\n    val simClustersANN4ConfigId = params(SimClustersANNParams.SimClustersANN4ConfigId)\n    // Build SANN Query\n    val simClustersANNQuery = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      simClustersANNConfigId,\n      params\n    )\n    val experimentalSimClustersANNQuery = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      experimentalSimClustersANNConfigId,\n      params\n    )\n    val simClustersANN1Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      simClustersANN1ConfigId,\n      params\n    )\n    val simClustersANN2Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      simClustersANN2ConfigId,\n      params\n    )\n    val simClustersANN3Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      simClustersANN3ConfigId,\n      params\n    )\n    val simClustersANN5Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      simClustersANN5ConfigId,\n      params\n    )\n    val simClustersANN4Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.FavBasedProducer,\n      simClustersModelVersion,\n      simClustersANN4ConfigId,\n      params\n    )\n    // UTG\n    val enableUtg = params(RelatedTweetProducerBasedParams.EnableUTGParam)\n    val utgCombinationMethod = params(\n      ProducerBasedCandidateGenerationParams.UtgCombinationMethodParam)\n\n    // SourceType.RequestUserId is a placeholder.\n    val sourceInfo = SourceInfo(SourceType.RequestUserId, internalId, None)\n\n    EngineQuery(\n      Query(\n        sourceInfo = sourceInfo,\n        maxCandidateNumPerSourceKey = maxCandidateNumPerSourceKey,\n        maxTweetAgeHours = maxTweetAgeHours,\n        enableSimClustersANN = enableSimClustersANN,\n        simClustersANNQuery = simClustersANNQuery,\n        enableExperimentalSimClustersANN = enableExperimentalSimClustersANN,\n        experimentalSimClustersANNQuery = experimentalSimClustersANNQuery,\n        enableSimClustersANN1 = enableSimClustersANN1,\n        simClustersANN1Query = simClustersANN1Query,\n        enableSimClustersANN2 = enableSimClustersANN2,\n        simClustersANN2Query = simClustersANN2Query,\n        enableSimClustersANN3 = enableSimClustersANN3,\n        simClustersANN3Query = simClustersANN3Query,\n        enableSimClustersANN5 = enableSimClustersANN5,\n        simClustersANN5Query = simClustersANN5Query,\n        enableSimClustersANN4 = enableSimClustersANN4,\n        simClustersANN4Query = simClustersANN4Query,\n        simClustersMinScore = simClustersMinScore,\n        enableUtg = enableUtg,\n        utgQuery = ProducerBasedUserTweetGraphSimilarityEngine.fromParams(internalId, params),\n        utgCombinationMethod = utgCombinationMethod\n      ),\n      params\n    )\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/ProducerBasedUserAdGraphSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.param.ProducerBasedUserAdGraphParams\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.user_ad_graph.thriftscala.ProducerBasedRelatedAdRequest\nimport com.twitter.recos.user_ad_graph.thriftscala.UserAdGraph\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport javax.inject.Singleton\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.timelines.configapi\n\n/**\n * This store looks for similar tweets from UserAdGraph for a Source ProducerId\n * For a query producerId,User Tweet Graph (UAG),\n * lets us find out which ad tweets the query producer's followers co-engaged\n */\n@Singleton\ncase class ProducerBasedUserAdGraphSimilarityEngine(\n  userAdGraphService: UserAdGraph.MethodPerEndpoint,\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[ProducerBasedUserAdGraphSimilarityEngine.Query, Seq[\n      TweetWithScore\n    ]] {\n\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val fetchCandidatesStat = stats.scope(\"fetchCandidates\")\n\n  override def get(\n    query: ProducerBasedUserAdGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    query.sourceId match {\n      case InternalId.UserId(producerId) =>\n        StatsUtil.trackOptionItemsStats(fetchCandidatesStat) {\n          val relatedAdRequest =\n            ProducerBasedRelatedAdRequest(\n              producerId,\n              maxResults = Some(query.maxResults),\n              minCooccurrence = Some(query.minCooccurrence),\n              minScore = Some(query.minScore),\n              maxNumFollowers = Some(query.maxNumFollowers),\n              maxTweetAgeInHours = Some(query.maxTweetAgeInHours),\n            )\n\n          userAdGraphService.producerBasedRelatedAds(relatedAdRequest).map { relatedAdResponse =>\n            val candidates =\n              relatedAdResponse.adTweets.map(tweet => TweetWithScore(tweet.adTweetId, tweet.score))\n            Some(candidates)\n          }\n        }\n      case _ =>\n        Future.value(None)\n    }\n  }\n}\n\nobject ProducerBasedUserAdGraphSimilarityEngine {\n\n  def toSimilarityEngineInfo(score: Double): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.ProducerBasedUserAdGraph,\n      modelId = None,\n      score = Some(score))\n  }\n\n  case class Query(\n    sourceId: InternalId,\n    maxResults: Int,\n    minCooccurrence: Int, // require at least {minCooccurrence} lhs user engaged with returned tweet\n    minScore: Double,\n    maxNumFollowers: Int, // max number of lhs users\n    maxTweetAgeInHours: Int)\n\n  def fromParams(\n    sourceId: InternalId,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    EngineQuery(\n      Query(\n        sourceId = sourceId,\n        maxResults = params(GlobalParams.MaxCandidateNumPerSourceKeyParam),\n        minCooccurrence = params(ProducerBasedUserAdGraphParams.MinCoOccurrenceParam),\n        maxNumFollowers = params(ProducerBasedUserAdGraphParams.MaxNumFollowersParam),\n        maxTweetAgeInHours = params(GlobalParams.MaxTweetAgeHoursParam).inHours,\n        minScore = params(ProducerBasedUserAdGraphParams.MinScoreParam)\n      ),\n      params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/ProducerBasedUserTweetGraphSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.ProducerBasedUserTweetGraphParams\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.user_tweet_graph.thriftscala.ProducerBasedRelatedTweetRequest\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport javax.inject.Singleton\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.timelines.configapi\nimport com.twitter.recos.user_tweet_graph.thriftscala.UserTweetGraph\n\n/**\n * This store looks for similar tweets from UserTweetGraph for a Source ProducerId\n * For a query producerId,User Tweet Graph (UTG),\n * lets us find out which tweets the query producer's followers co-engaged\n */\n@Singleton\ncase class ProducerBasedUserTweetGraphSimilarityEngine(\n  userTweetGraphService: UserTweetGraph.MethodPerEndpoint,\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[ProducerBasedUserTweetGraphSimilarityEngine.Query, Seq[\n      TweetWithScore\n    ]] {\n\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val fetchCandidatesStat = stats.scope(\"fetchCandidates\")\n\n  override def get(\n    query: ProducerBasedUserTweetGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    query.sourceId match {\n      case InternalId.UserId(producerId) =>\n        StatsUtil.trackOptionItemsStats(fetchCandidatesStat) {\n          val relatedTweetRequest =\n            ProducerBasedRelatedTweetRequest(\n              producerId,\n              maxResults = Some(query.maxResults),\n              minCooccurrence = Some(query.minCooccurrence),\n              minScore = Some(query.minScore),\n              maxNumFollowers = Some(query.maxNumFollowers),\n              maxTweetAgeInHours = Some(query.maxTweetAgeInHours),\n            )\n\n          userTweetGraphService.producerBasedRelatedTweets(relatedTweetRequest).map {\n            relatedTweetResponse =>\n              val candidates =\n                relatedTweetResponse.tweets.map(tweet => TweetWithScore(tweet.tweetId, tweet.score))\n              Some(candidates)\n          }\n        }\n      case _ =>\n        Future.value(None)\n    }\n  }\n}\n\nobject ProducerBasedUserTweetGraphSimilarityEngine {\n\n  def toSimilarityEngineInfo(score: Double): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.ProducerBasedUserTweetGraph,\n      modelId = None,\n      score = Some(score))\n  }\n\n  case class Query(\n    sourceId: InternalId,\n    maxResults: Int,\n    minCooccurrence: Int, // require at least {minCooccurrence} lhs user engaged with returned tweet\n    minScore: Double,\n    maxNumFollowers: Int, // max number of lhs users\n    maxTweetAgeInHours: Int)\n\n  def fromParams(\n    sourceId: InternalId,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    EngineQuery(\n      Query(\n        sourceId = sourceId,\n        maxResults = params(GlobalParams.MaxCandidateNumPerSourceKeyParam),\n        minCooccurrence = params(ProducerBasedUserTweetGraphParams.MinCoOccurrenceParam),\n        maxNumFollowers = params(ProducerBasedUserTweetGraphParams.MaxNumFollowersParam),\n        maxTweetAgeInHours = params(GlobalParams.MaxTweetAgeHoursParam).inHours,\n        minScore = params(ProducerBasedUserTweetGraphParams.MinScoreParam)\n      ),\n      params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/SimClustersANNSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.config.SimClustersANNConfig\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclustersann.thriftscala.SimClustersANNService\nimport com.twitter.simclustersann.thriftscala.{Query => SimClustersANNQuery}\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Future\nimport javax.inject.Singleton\nimport com.twitter.cr_mixer.exception.InvalidSANNConfigException\nimport com.twitter.relevance_platform.simclustersann.multicluster.ServiceNameMapper\n\n@Singleton\ncase class SimClustersANNSimilarityEngine(\n  simClustersANNServiceNameToClientMapper: Map[String, SimClustersANNService.MethodPerEndpoint],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[\n      SimClustersANNSimilarityEngine.Query,\n      Seq[TweetWithScore]\n    ] {\n\n  private val name: String = this.getClass.getSimpleName\n  private val stats = statsReceiver.scope(name)\n  private val fetchCandidatesStat = stats.scope(\"fetchCandidates\")\n\n  private def getSimClustersANNService(\n    query: SimClustersANNQuery\n  ): Option[SimClustersANNService.MethodPerEndpoint] = {\n    ServiceNameMapper\n      .getServiceName(\n        query.sourceEmbeddingId.modelVersion,\n        query.config.candidateEmbeddingType).flatMap(serviceName =>\n        simClustersANNServiceNameToClientMapper.get(serviceName))\n  }\n\n  override def get(\n    query: SimClustersANNSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    StatsUtil.trackOptionItemsStats(fetchCandidatesStat) {\n\n      getSimClustersANNService(query.simClustersANNQuery) match {\n        case Some(simClustersANNService) =>\n          simClustersANNService.getTweetCandidates(query.simClustersANNQuery).map {\n            simClustersANNTweetCandidates =>\n              val tweetWithScores = simClustersANNTweetCandidates.map { candidate =>\n                TweetWithScore(candidate.tweetId, candidate.score)\n              }\n              Some(tweetWithScores)\n          }\n        case None =>\n          throw InvalidSANNConfigException(\n            \"No SANN Cluster configured to serve this query, check CandidateEmbeddingType and ModelVersion\")\n      }\n    }\n  }\n}\n\nobject SimClustersANNSimilarityEngine {\n  case class Query(\n    simClustersANNQuery: SimClustersANNQuery,\n    simClustersANNConfigId: String)\n\n  def toSimilarityEngineInfo(\n    query: EngineQuery[Query],\n    score: Double\n  ): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.SimClustersANN,\n      modelId = Some(\n        s\"SimClustersANN_${query.storeQuery.simClustersANNQuery.sourceEmbeddingId.embeddingType.toString}_\" +\n          s\"${query.storeQuery.simClustersANNQuery.sourceEmbeddingId.modelVersion.toString}_\" +\n          s\"${query.storeQuery.simClustersANNConfigId}\"),\n      score = Some(score)\n    )\n  }\n\n  def fromParams(\n    internalId: InternalId,\n    embeddingType: EmbeddingType,\n    modelVersion: ModelVersion,\n    simClustersANNConfigId: String,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n\n    // SimClusters EmbeddingId and ANNConfig\n    val simClustersEmbeddingId =\n      SimClustersEmbeddingId(embeddingType, modelVersion, internalId)\n    val simClustersANNConfig =\n      SimClustersANNConfig\n        .getConfig(embeddingType.toString, modelVersion.toString, simClustersANNConfigId)\n\n    EngineQuery(\n      Query(\n        SimClustersANNQuery(\n          sourceEmbeddingId = simClustersEmbeddingId,\n          config = simClustersANNConfig.toSANNConfigThrift\n        ),\n        simClustersANNConfigId\n      ),\n      params\n    )\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/SimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.GlobalRequestTimeoutException\nimport com.twitter.finagle.mux.ClientDiscardedRequestException\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.mux.ServerApplicationError\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.relevance_platform.common.injection.LZ4Injection\nimport com.twitter.relevance_platform.common.injection.SeqObjectInjection\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.TimeoutException\nimport com.twitter.util.logging.Logging\nimport org.apache.thrift.TApplicationException\n\n/**\n * A SimilarityEngine is a wrapper which, given a [[Query]], returns a list of [[Candidate]]\n * The main purposes of a SimilarityEngine is to provide a consistent interface for candidate\n * generation logic, and provides default functions, including:\n * - Identification\n * - Observability\n * - Timeout settings\n * - Exception Handling\n * - Gating by Deciders & FeatureSwitch settings\n * - (coming soon): Dark traffic\n *\n * Note:\n * A SimilarityEngine by itself is NOT meant to be cacheable.\n * Caching should be implemented in the underlying ReadableStore that provides the [[Candidate]]s\n *\n * Please keep extension of this class local this directory only\n *\n */\ntrait SimilarityEngine[Query, Candidate] {\n\n  /**\n   * Uniquely identifies a similarity engine.\n   * Avoid using the same engine type for more than one engine, it will cause stats to double count\n   */\n  private[similarity_engine] def identifier: SimilarityEngineType\n\n  def getCandidates(query: Query): Future[Option[Seq[Candidate]]]\n\n}\n\nobject SimilarityEngine extends Logging {\n  case class SimilarityEngineConfig(\n    timeout: Duration,\n    gatingConfig: GatingConfig)\n\n  /**\n   * Controls for whether or not this Engine is enabled.\n   * In our previous design, we were expecting a Sim Engine will only take one set of Params,\n   * and that’s why we decided to have GatingConfig and the EnableFeatureSwitch in the trait.\n   * However, we now have two candidate generation pipelines: Tweet Rec, Related Tweets\n   * and they are now having their own set of Params, but EnableFeatureSwitch can only put in 1 fixed value.\n   * We need some further refactor work to make it more flexible.\n   *\n   * @param deciderConfig Gate the Engine by a decider. If specified,\n   * @param enableFeatureSwitch. DO NOT USE IT FOR NOW. It needs some refactorting. Please set it to None (SD-20268)\n   */\n  case class GatingConfig(\n    deciderConfig: Option[DeciderConfig],\n    enableFeatureSwitch: Option[\n      FSParam[Boolean]\n    ]) // Do NOT use the enableFeatureSwitch. It needs some refactoring.\n\n  case class DeciderConfig(\n    decider: CrMixerDecider,\n    deciderString: String)\n\n  case class MemCacheConfig[K](\n    cacheClient: Client,\n    ttl: Duration,\n    asyncUpdate: Boolean = false,\n    keyToString: K => String)\n\n  private[similarity_engine] def isEnabled(\n    params: Params,\n    gatingConfig: GatingConfig\n  ): Boolean = {\n    val enabledByDecider =\n      gatingConfig.deciderConfig.forall { config =>\n        config.decider.isAvailable(config.deciderString)\n      }\n\n    val enabledByFS = gatingConfig.enableFeatureSwitch.forall(params.apply)\n\n    enabledByDecider && enabledByFS\n  }\n\n  // Default key hasher for memcache keys\n  val keyHasher: KeyHasher = KeyHasher.FNV1A_64\n\n  /**\n   * Add a MemCache wrapper to a ReadableStore with a preset key and value injection functions\n   * Note: The [[Query]] object needs to be cacheable,\n   * i.e. it cannot be a runtime objects or complex objects, for example, configapi.Params\n   *\n   * @param underlyingStore un-cached store implementation\n   * @param keyPrefix       a prefix differentiates 2 stores if they share the same key space.\n   *                        e.x. 2 implementations of ReadableStore[UserId, Seq[Candidiate] ]\n   *                        can use prefix \"store_v1\", \"store_v2\"\n   * @return                A ReadableStore with a MemCache wrapper\n   */\n  private[similarity_engine] def addMemCache[Query, Candidate <: Serializable](\n    underlyingStore: ReadableStore[Query, Seq[Candidate]],\n    memCacheConfig: MemCacheConfig[Query],\n    keyPrefix: Option[String] = None,\n    statsReceiver: StatsReceiver\n  ): ReadableStore[Query, Seq[Candidate]] = {\n    val prefix = keyPrefix.getOrElse(\"\")\n\n    ObservedMemcachedReadableStore.fromCacheClient[Query, Seq[Candidate]](\n      backingStore = underlyingStore,\n      cacheClient = memCacheConfig.cacheClient,\n      ttl = memCacheConfig.ttl,\n      asyncUpdate = memCacheConfig.asyncUpdate,\n    )(\n      valueInjection = LZ4Injection.compose(SeqObjectInjection[Candidate]()),\n      keyToString = { k: Query => s\"CRMixer:$prefix${memCacheConfig.keyToString(k)}\" },\n      statsReceiver = statsReceiver\n    )\n  }\n\n  private val timer = com.twitter.finagle.util.DefaultTimer\n\n  /**\n   * Applies runtime configs, like stats, timeouts, exception handling, onto fn\n   */\n  private[similarity_engine] def getFromFn[Query, Candidate](\n    fn: Query => Future[Option[Seq[Candidate]]],\n    storeQuery: Query,\n    engineConfig: SimilarityEngineConfig,\n    params: Params,\n    scopedStats: StatsReceiver\n  ): Future[Option[Seq[Candidate]]] = {\n    if (isEnabled(params, engineConfig.gatingConfig)) {\n      scopedStats.counter(\"gate_enabled\").incr()\n\n      StatsUtil\n        .trackOptionItemsStats(scopedStats) {\n          fn.apply(storeQuery).raiseWithin(engineConfig.timeout)(timer)\n        }\n        .rescue {\n          case _: TimeoutException | _: GlobalRequestTimeoutException | _: TApplicationException |\n              _: ClientDiscardedRequestException |\n              _: ServerApplicationError // TApplicationException inside\n              =>\n            debug(\"Failed to fetch. request aborted or timed out\")\n            Future.None\n          case e =>\n            error(\"Failed to fetch. request aborted or timed out\", e)\n            Future.None\n        }\n    } else {\n      scopedStats.counter(\"gate_disabled\").incr()\n      Future.None\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/SimilaritySourceOrderingUtil.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.TweetWithCandidateGenerationInfo\nimport com.twitter.simclusters_v2.common.TweetId\nimport scala.collection.mutable\nimport scala.collection.mutable.ArrayBuffer\n\nobject SimilaritySourceOrderingUtil {\n  /**\n   * This function flatten and dedup input candidates according to the order in the input Seq\n   * [[candidate10, candidate11], [candidate20, candidate21]] => [candidate10, candidate11, candidate20, candidate21]\n   */\n  def keepGivenOrder(\n    candidates: Seq[Seq[TweetWithCandidateGenerationInfo]],\n  ): Seq[TweetWithCandidateGenerationInfo] = {\n\n    val seen = mutable.Set[TweetId]()\n    val combinedCandidates = candidates.flatten\n    val result = ArrayBuffer[TweetWithCandidateGenerationInfo]()\n\n    combinedCandidates.foreach { candidate =>\n      val candidateTweetId = candidate.tweetId\n      val seenCandidate = seen.contains(candidateTweetId) // de-dup\n      if (!seenCandidate) {\n        result += candidate\n        seen.add(candidate.tweetId)\n      }\n    }\n    //convert result to immutable seq\n    result.toList\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/SkitHighPrecisionTopicTweetSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.google.inject.name.Named\nimport com.twitter.contentrecommender.thriftscala.AlgorithmType\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TopicTweetWithScore\nimport com.twitter.cr_mixer.param.TopicTweetParams\nimport com.twitter.cr_mixer.similarity_engine.SkitTopicTweetSimilarityEngine._\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.TopicId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.topic_recos.thriftscala.TopicTweet\nimport com.twitter.topic_recos.thriftscala.TopicTweetPartitionFlatKey\nimport com.twitter.util.Future\n\n@Singleton\ncase class SkitHighPrecisionTopicTweetSimilarityEngine @Inject() (\n  @Named(ModuleNames.SkitStratoStoreName) skitStratoStore: ReadableStore[\n    TopicTweetPartitionFlatKey,\n    Seq[TopicTweet]\n  ],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[EngineQuery[Query], Seq[TopicTweetWithScore]] {\n\n  private val name: String = this.getClass.getSimpleName\n  private val stats = statsReceiver.scope(name)\n\n  override def get(query: EngineQuery[Query]): Future[Option[Seq[TopicTweetWithScore]]] = {\n    StatsUtil.trackOptionItemsStats(stats) {\n      fetch(query).map { tweets =>\n        val topTweets =\n          tweets\n            .sortBy(-_.favCount)\n            .take(query.storeQuery.maxCandidates)\n            .map { tweet =>\n              TopicTweetWithScore(\n                tweetId = tweet.tweetId,\n                score = tweet.favCount,\n                similarityEngineType = SimilarityEngineType.SkitHighPrecisionTopicTweet\n              )\n            }\n        Some(topTweets)\n      }\n    }\n  }\n\n  private def fetch(query: EngineQuery[Query]): Future[Seq[SkitTopicTweet]] = {\n    val latestTweetTimeInHour = System.currentTimeMillis() / 1000 / 60 / 60\n\n    val earliestTweetTimeInHour = latestTweetTimeInHour -\n      math.min(MaxTweetAgeInHours, query.storeQuery.maxTweetAge.inHours)\n    val timedKeys = for (timePartition <- earliestTweetTimeInHour to latestTweetTimeInHour) yield {\n\n      TopicTweetPartitionFlatKey(\n        entityId = query.storeQuery.topicId.entityId,\n        timePartition = timePartition,\n        algorithmType = Some(AlgorithmType.SemanticCoreTweet),\n        tweetEmbeddingType = Some(EmbeddingType.LogFavBasedTweet),\n        language = query.storeQuery.topicId.language.getOrElse(\"\").toLowerCase,\n        country = None, // Disable country. It is not used.\n        semanticCoreAnnotationVersionId = Some(query.storeQuery.semanticCoreVersionId)\n      )\n    }\n\n    getTweetsForKeys(\n      timedKeys,\n      query.storeQuery.topicId\n    )\n  }\n\n  /**\n   * Given a set of keys, multiget the underlying Strato store, combine and flatten the results.\n   */\n  private def getTweetsForKeys(\n    keys: Seq[TopicTweetPartitionFlatKey],\n    sourceTopic: TopicId\n  ): Future[Seq[SkitTopicTweet]] = {\n    Future\n      .collect { skitStratoStore.multiGet(keys.toSet).values.toSeq }\n      .map { combinedResults =>\n        val topTweets = combinedResults.flatten.flatten\n        topTweets.map { tweet =>\n          SkitTopicTweet(\n            tweetId = tweet.tweetId,\n            favCount = tweet.scores.favCount.getOrElse(0L),\n            cosineSimilarityScore = tweet.scores.cosineSimilarity.getOrElse(0.0),\n            sourceTopic = sourceTopic\n          )\n        }\n      }\n  }\n}\n\nobject SkitHighPrecisionTopicTweetSimilarityEngine {\n\n  def fromParams(\n    topicId: TopicId,\n    isVideoOnly: Boolean,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    val maxCandidates = if (isVideoOnly) {\n      params(TopicTweetParams.MaxSkitHighPrecisionCandidatesParam) * 2\n    } else {\n      params(TopicTweetParams.MaxSkitHighPrecisionCandidatesParam)\n    }\n\n    EngineQuery(\n      Query(\n        topicId = topicId,\n        maxCandidates = maxCandidates,\n        maxTweetAge = params(TopicTweetParams.MaxTweetAge),\n        semanticCoreVersionId = params(TopicTweetParams.SemanticCoreVersionIdParam)\n      ),\n      params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/SkitTopicTweetSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.google.inject.name.Named\nimport com.twitter.contentrecommender.thriftscala.AlgorithmType\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.TopicTweetWithScore\nimport com.twitter.cr_mixer.param.TopicTweetParams\nimport com.twitter.cr_mixer.similarity_engine.SkitTopicTweetSimilarityEngine._\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.TopicId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.topic_recos.thriftscala.TopicTweet\nimport com.twitter.topic_recos.thriftscala.TopicTweetPartitionFlatKey\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\n\n@Singleton\ncase class SkitTopicTweetSimilarityEngine @Inject() (\n  @Named(ModuleNames.SkitStratoStoreName) skitStratoStore: ReadableStore[\n    TopicTweetPartitionFlatKey,\n    Seq[TopicTweet]\n  ],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[EngineQuery[Query], Seq[TopicTweetWithScore]] {\n\n  private val name: String = this.getClass.getSimpleName\n  private val stats = statsReceiver.scope(name)\n\n  override def get(query: EngineQuery[Query]): Future[Option[Seq[TopicTweetWithScore]]] = {\n    StatsUtil.trackOptionItemsStats(stats) {\n      fetch(query).map { tweets =>\n        val topTweets =\n          tweets\n            .sortBy(-_.cosineSimilarityScore)\n            .take(query.storeQuery.maxCandidates)\n            .map { tweet =>\n              TopicTweetWithScore(\n                tweetId = tweet.tweetId,\n                score = tweet.cosineSimilarityScore,\n                similarityEngineType = SimilarityEngineType.SkitTfgTopicTweet\n              )\n            }\n        Some(topTweets)\n      }\n    }\n  }\n\n  private def fetch(query: EngineQuery[Query]): Future[Seq[SkitTopicTweet]] = {\n    val latestTweetTimeInHour = System.currentTimeMillis() / 1000 / 60 / 60\n\n    val earliestTweetTimeInHour = latestTweetTimeInHour -\n      math.min(MaxTweetAgeInHours, query.storeQuery.maxTweetAge.inHours)\n    val timedKeys = for (timePartition <- earliestTweetTimeInHour to latestTweetTimeInHour) yield {\n\n      TopicTweetPartitionFlatKey(\n        entityId = query.storeQuery.topicId.entityId,\n        timePartition = timePartition,\n        algorithmType = Some(AlgorithmType.TfgTweet),\n        tweetEmbeddingType = Some(EmbeddingType.LogFavBasedTweet),\n        language = query.storeQuery.topicId.language.getOrElse(\"\").toLowerCase,\n        country = None, // Disable country. It is not used.\n        semanticCoreAnnotationVersionId = Some(query.storeQuery.semanticCoreVersionId),\n        simclustersModelVersion = Some(ModelVersion.Model20m145k2020)\n      )\n    }\n\n    getTweetsForKeys(\n      timedKeys,\n      query.storeQuery.topicId\n    )\n  }\n\n  /**\n   * Given a set of keys, multiget the underlying Strato store, combine and flatten the results.\n   */\n  private def getTweetsForKeys(\n    keys: Seq[TopicTweetPartitionFlatKey],\n    sourceTopic: TopicId\n  ): Future[Seq[SkitTopicTweet]] = {\n    Future\n      .collect { skitStratoStore.multiGet(keys.toSet).values.toSeq }\n      .map { combinedResults =>\n        val topTweets = combinedResults.flatten.flatten\n        topTweets.map { tweet =>\n          SkitTopicTweet(\n            tweetId = tweet.tweetId,\n            favCount = tweet.scores.favCount.getOrElse(0L),\n            cosineSimilarityScore = tweet.scores.cosineSimilarity.getOrElse(0.0),\n            sourceTopic = sourceTopic\n          )\n        }\n      }\n  }\n}\n\nobject SkitTopicTweetSimilarityEngine {\n\n  val MaxTweetAgeInHours: Int = 7.days.inHours // Simple guard to prevent overloading\n\n  // Query is used as a cache key. Do not add any user level information in this.\n  case class Query(\n    topicId: TopicId,\n    maxCandidates: Int,\n    maxTweetAge: Duration,\n    semanticCoreVersionId: Long)\n\n  case class SkitTopicTweet(\n    sourceTopic: TopicId,\n    tweetId: TweetId,\n    favCount: Long,\n    cosineSimilarityScore: Double)\n\n  def fromParams(\n    topicId: TopicId,\n    isVideoOnly: Boolean,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    val maxCandidates = if (isVideoOnly) {\n      params(TopicTweetParams.MaxSkitTfgCandidatesParam) * 2\n    } else {\n      params(TopicTweetParams.MaxSkitTfgCandidatesParam)\n    }\n\n    EngineQuery(\n      Query(\n        topicId = topicId,\n        maxCandidates = maxCandidates,\n        maxTweetAge = params(TopicTweetParams.MaxTweetAge),\n        semanticCoreVersionId = params(TopicTweetParams.SemanticCoreVersionIdParam)\n      ),\n      params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/StandardSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.MemCacheConfig\nimport com.twitter.cr_mixer.similarity_engine.SimilarityEngine.SimilarityEngineConfig\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.Future\n\n/**\n * @tparam Query ReadableStore's input type.\n */\ncase class EngineQuery[Query](\n  storeQuery: Query,\n  params: Params,\n)\n\n/**\n * A straight forward SimilarityEngine implementation that wraps a ReadableStore\n *\n * @param implementingStore   Provides the candidate retrieval's implementations\n * @param memCacheConfig      If specified, it will wrap the underlying store with a MemCache layer\n *                            You should only enable this for cacheable queries, e.x. TweetIds.\n *                            consumer based UserIds are generally not possible to cache.\n * @tparam Query              ReadableStore's input type\n * @tparam Candidate          ReadableStore's return type is Seq[[[Candidate]]]\n */\nclass StandardSimilarityEngine[Query, Candidate <: Serializable](\n  implementingStore: ReadableStore[Query, Seq[Candidate]],\n  override val identifier: SimilarityEngineType,\n  globalStats: StatsReceiver,\n  engineConfig: SimilarityEngineConfig,\n  memCacheConfig: Option[MemCacheConfig[Query]] = None)\n    extends SimilarityEngine[EngineQuery[Query], Candidate] {\n\n  private val scopedStats = globalStats.scope(\"similarityEngine\", identifier.toString)\n\n  def getScopedStats: StatsReceiver = scopedStats\n\n  // Add memcache wrapper, if specified\n  private val store = {\n    memCacheConfig match {\n      case Some(config) =>\n        SimilarityEngine.addMemCache(\n          underlyingStore = implementingStore,\n          memCacheConfig = config,\n          statsReceiver = scopedStats\n        )\n      case _ => implementingStore\n    }\n  }\n\n  override def getCandidates(\n    engineQuery: EngineQuery[Query]\n  ): Future[Option[Seq[Candidate]]] = {\n    SimilarityEngine.getFromFn(\n      store.get,\n      engineQuery.storeQuery,\n      engineConfig,\n      engineQuery.params,\n      scopedStats\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/TweetBasedQigSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.Stats\nimport com.twitter.product_mixer.core.thriftscala.ClientContext\nimport com.twitter.qig_ranker.thriftscala.Product\nimport com.twitter.qig_ranker.thriftscala.ProductContext\nimport com.twitter.qig_ranker.thriftscala.QigRanker\nimport com.twitter.qig_ranker.thriftscala.QigRankerProductResponse\nimport com.twitter.qig_ranker.thriftscala.QigRankerRequest\nimport com.twitter.qig_ranker.thriftscala.QigRankerResponse\nimport com.twitter.qig_ranker.thriftscala.TwistlySimilarTweetsProductContext\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n/**\n * This store looks for similar tweets from QueryInteractionGraph (QIG) for a source tweet id.\n * For a given query tweet, QIG returns us the similar tweets that have an overlap of engagements\n * (with the query tweet) on different search queries\n */\n@Singleton\ncase class TweetBasedQigSimilarityEngine(\n  qigRanker: QigRanker.MethodPerEndpoint,\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[\n      TweetBasedQigSimilarityEngine.Query,\n      Seq[TweetWithScore]\n    ] {\n\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val fetchCandidatesStat = stats.scope(\"fetchCandidates\")\n\n  override def get(\n    query: TweetBasedQigSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    query.sourceId match {\n      case InternalId.TweetId(tweetId) =>\n        val qigSimilarTweetsRequest = getQigSimilarTweetsRequest(tweetId)\n\n        Stats.trackOption(fetchCandidatesStat) {\n          qigRanker\n            .getSimilarCandidates(qigSimilarTweetsRequest)\n            .map { qigSimilarTweetsResponse =>\n              getCandidatesFromQigResponse(qigSimilarTweetsResponse)\n            }\n        }\n      case _ =>\n        Future.value(None)\n    }\n  }\n\n  private def getQigSimilarTweetsRequest(\n    tweetId: Long\n  ): QigRankerRequest = {\n    // Note: QigRanker needs a non-empty userId to be passed to return results.\n    // We are passing in a dummy userId until we fix this on QigRanker side\n    val clientContext = ClientContext(userId = Some(0L))\n    val productContext = ProductContext.TwistlySimilarTweetsProductContext(\n      TwistlySimilarTweetsProductContext(tweetId = tweetId))\n\n    QigRankerRequest(\n      clientContext = clientContext,\n      product = Product.TwistlySimilarTweets,\n      productContext = Some(productContext),\n    )\n  }\n\n  private def getCandidatesFromQigResponse(\n    qigSimilarTweetsResponse: QigRankerResponse\n  ): Option[Seq[TweetWithScore]] = {\n    qigSimilarTweetsResponse.productResponse match {\n      case QigRankerProductResponse\n            .TwistlySimilarTweetCandidatesResponse(response) =>\n        val tweetsWithScore = response.similarTweets\n          .map { similarTweetResult =>\n            TweetWithScore(\n              similarTweetResult.tweetResult.tweetId,\n              similarTweetResult.tweetResult.score.getOrElse(0L))\n          }\n        Some(tweetsWithScore)\n\n      case _ => None\n    }\n  }\n}\n\nobject TweetBasedQigSimilarityEngine {\n\n  def toSimilarityEngineInfo(score: Double): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.Qig,\n      modelId = None,\n      score = Some(score))\n  }\n\n  case class Query(sourceId: InternalId)\n\n  def fromParams(\n    sourceId: InternalId,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    EngineQuery(\n      Query(sourceId = sourceId),\n      params\n    )\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/TweetBasedUnifiedSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.model.TweetWithCandidateGenerationInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.param.RelatedTweetTweetBasedParams\nimport com.twitter.cr_mixer.param.RelatedVideoTweetTweetBasedParams\nimport com.twitter.cr_mixer.param.SimClustersANNParams\nimport com.twitter.cr_mixer.param.TweetBasedCandidateGenerationParams\nimport com.twitter.cr_mixer.param.TweetBasedTwHINParams\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.cr_mixer.util.InterleaveUtil\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.common.ModelVersions\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.mutable.ArrayBuffer\n\n/**\n * This store fetches similar tweets from multiple tweet based candidate sources\n * and combines them using different methods obtained from query params\n */\n@Singleton\ncase class TweetBasedUnifiedSimilarityEngine(\n  @Named(ModuleNames.TweetBasedUserTweetGraphSimilarityEngine)\n  tweetBasedUserTweetGraphSimilarityEngine: StandardSimilarityEngine[\n    TweetBasedUserTweetGraphSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  @Named(ModuleNames.TweetBasedUserVideoGraphSimilarityEngine)\n  tweetBasedUserVideoGraphSimilarityEngine: StandardSimilarityEngine[\n    TweetBasedUserVideoGraphSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  simClustersANNSimilarityEngine: StandardSimilarityEngine[\n    SimClustersANNSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  @Named(ModuleNames.TweetBasedQigSimilarityEngine)\n  tweetBasedQigSimilarTweetsSimilarityEngine: StandardSimilarityEngine[\n    TweetBasedQigSimilarityEngine.Query,\n    TweetWithScore\n  ],\n  @Named(ModuleNames.TweetBasedTwHINANNSimilarityEngine)\n  tweetBasedTwHINANNSimilarityEngine: HnswANNSimilarityEngine,\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[\n      TweetBasedUnifiedSimilarityEngine.Query,\n      Seq[TweetWithCandidateGenerationInfo]\n    ] {\n\n  import TweetBasedUnifiedSimilarityEngine._\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val fetchCandidatesStat = stats.scope(\"fetchCandidates\")\n\n  override def get(\n    query: Query\n  ): Future[Option[Seq[TweetWithCandidateGenerationInfo]]] = {\n\n    query.sourceInfo.internalId match {\n      case _: InternalId.TweetId =>\n        StatsUtil.trackOptionItemsStats(fetchCandidatesStat) {\n          val twhinQuery =\n            HnswANNEngineQuery(\n              sourceId = query.sourceInfo.internalId,\n              modelId = query.twhinModelId,\n              params = query.params)\n          val utgCandidatesFut =\n            if (query.enableUtg)\n              tweetBasedUserTweetGraphSimilarityEngine.getCandidates(query.utgQuery)\n            else Future.None\n\n          val uvgCandidatesFut =\n            if (query.enableUvg)\n              tweetBasedUserVideoGraphSimilarityEngine.getCandidates(query.uvgQuery)\n            else Future.None\n\n          val sannCandidatesFut = if (query.enableSimClustersANN) {\n            simClustersANNSimilarityEngine.getCandidates(query.simClustersANNQuery)\n          } else Future.None\n\n          val sann1CandidatesFut =\n            if (query.enableSimClustersANN1) {\n              simClustersANNSimilarityEngine.getCandidates(query.simClustersANN1Query)\n            } else Future.None\n\n          val sann2CandidatesFut =\n            if (query.enableSimClustersANN2) {\n              simClustersANNSimilarityEngine.getCandidates(query.simClustersANN2Query)\n            } else Future.None\n\n          val sann3CandidatesFut =\n            if (query.enableSimClustersANN3) {\n              simClustersANNSimilarityEngine.getCandidates(query.simClustersANN3Query)\n            } else Future.None\n\n          val sann5CandidatesFut =\n            if (query.enableSimClustersANN5) {\n              simClustersANNSimilarityEngine.getCandidates(query.simClustersANN5Query)\n            } else Future.None\n\n          val sann4CandidatesFut =\n            if (query.enableSimClustersANN4) {\n              simClustersANNSimilarityEngine.getCandidates(query.simClustersANN4Query)\n            } else Future.None\n\n          val experimentalSANNCandidatesFut =\n            if (query.enableExperimentalSimClustersANN) {\n              simClustersANNSimilarityEngine.getCandidates(query.experimentalSimClustersANNQuery)\n            } else Future.None\n\n          val qigCandidatesFut =\n            if (query.enableQig)\n              tweetBasedQigSimilarTweetsSimilarityEngine.getCandidates(query.qigQuery)\n            else Future.None\n\n          val twHINCandidateFut = if (query.enableTwHIN) {\n            tweetBasedTwHINANNSimilarityEngine.getCandidates(twhinQuery)\n          } else Future.None\n\n          Future\n            .join(\n              utgCandidatesFut,\n              sannCandidatesFut,\n              sann1CandidatesFut,\n              sann2CandidatesFut,\n              sann3CandidatesFut,\n              sann5CandidatesFut,\n              sann4CandidatesFut,\n              experimentalSANNCandidatesFut,\n              qigCandidatesFut,\n              twHINCandidateFut,\n              uvgCandidatesFut\n            ).map {\n              case (\n                    userTweetGraphCandidates,\n                    simClustersANNCandidates,\n                    simClustersANN1Candidates,\n                    simClustersANN2Candidates,\n                    simClustersANN3Candidates,\n                    simClustersANN5Candidates,\n                    simClustersANN4Candidates,\n                    experimentalSANNCandidates,\n                    qigSimilarTweetsCandidates,\n                    twhinCandidates,\n                    userVideoGraphCandidates) =>\n                val filteredUTGTweets =\n                  userTweetGraphFilter(userTweetGraphCandidates.toSeq.flatten)\n                val filteredUVGTweets =\n                  userVideoGraphFilter(userVideoGraphCandidates.toSeq.flatten)\n                val filteredSANNTweets = simClustersCandidateMinScoreFilter(\n                  simClustersANNCandidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.simClustersANNQuery.storeQuery.simClustersANNConfigId)\n\n                val filteredSANN1Tweets = simClustersCandidateMinScoreFilter(\n                  simClustersANN1Candidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.simClustersANN1Query.storeQuery.simClustersANNConfigId)\n\n                val filteredSANN2Tweets = simClustersCandidateMinScoreFilter(\n                  simClustersANN2Candidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.simClustersANN2Query.storeQuery.simClustersANNConfigId)\n\n                val filteredSANN3Tweets = simClustersCandidateMinScoreFilter(\n                  simClustersANN3Candidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.simClustersANN3Query.storeQuery.simClustersANNConfigId)\n\n                val filteredSANN4Tweets = simClustersCandidateMinScoreFilter(\n                  simClustersANN4Candidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.simClustersANN4Query.storeQuery.simClustersANNConfigId)\n\n                val filteredSANN5Tweets = simClustersCandidateMinScoreFilter(\n                  simClustersANN5Candidates.toSeq.flatten,\n                  query.simClustersMinScore,\n                  query.simClustersANN5Query.storeQuery.simClustersANNConfigId)\n\n                val filteredExperimentalSANNTweets = simClustersCandidateMinScoreFilter(\n                  experimentalSANNCandidates.toSeq.flatten,\n                  query.simClustersVideoBasedMinScore,\n                  query.experimentalSimClustersANNQuery.storeQuery.simClustersANNConfigId)\n\n                val filteredQigTweets = qigSimilarTweetsFilter(\n                  qigSimilarTweetsCandidates.toSeq.flatten,\n                  query.qigMaxTweetAgeHours,\n                  query.qigMaxNumSimilarTweets\n                )\n\n                val filteredTwHINTweets = twhinFilter(\n                  twhinCandidates.toSeq.flatten.sortBy(-_.score),\n                  query.twhinMaxTweetAgeHours,\n                  tweetBasedTwHINANNSimilarityEngine.getScopedStats\n                )\n                val utgTweetsWithCGInfo = filteredUTGTweets.map { tweetWithScore =>\n                  val similarityEngineInfo = TweetBasedUserTweetGraphSimilarityEngine\n                    .toSimilarityEngineInfo(tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n\n                val uvgTweetsWithCGInfo = filteredUVGTweets.map { tweetWithScore =>\n                  val similarityEngineInfo = TweetBasedUserVideoGraphSimilarityEngine\n                    .toSimilarityEngineInfo(tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n                val sannTweetsWithCGInfo = filteredSANNTweets.map { tweetWithScore =>\n                  val similarityEngineInfo = SimClustersANNSimilarityEngine\n                    .toSimilarityEngineInfo(query.simClustersANNQuery, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n                val sann1TweetsWithCGInfo = filteredSANN1Tweets.map { tweetWithScore =>\n                  val similarityEngineInfo = SimClustersANNSimilarityEngine\n                    .toSimilarityEngineInfo(query.simClustersANN1Query, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n                val sann2TweetsWithCGInfo = filteredSANN2Tweets.map { tweetWithScore =>\n                  val similarityEngineInfo = SimClustersANNSimilarityEngine\n                    .toSimilarityEngineInfo(query.simClustersANN2Query, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n                val sann3TweetsWithCGInfo = filteredSANN3Tweets.map { tweetWithScore =>\n                  val similarityEngineInfo = SimClustersANNSimilarityEngine\n                    .toSimilarityEngineInfo(query.simClustersANN3Query, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n                val sann4TweetsWithCGInfo = filteredSANN4Tweets.map { tweetWithScore =>\n                  val similarityEngineInfo = SimClustersANNSimilarityEngine\n                    .toSimilarityEngineInfo(query.simClustersANN4Query, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n                val sann5TweetsWithCGInfo = filteredSANN5Tweets.map { tweetWithScore =>\n                  val similarityEngineInfo = SimClustersANNSimilarityEngine\n                    .toSimilarityEngineInfo(query.simClustersANN5Query, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n\n                val experimentalSANNTweetsWithCGInfo = filteredExperimentalSANNTweets.map {\n                  tweetWithScore =>\n                    val similarityEngineInfo = SimClustersANNSimilarityEngine\n                      .toSimilarityEngineInfo(\n                        query.experimentalSimClustersANNQuery,\n                        tweetWithScore.score)\n                    TweetWithCandidateGenerationInfo(\n                      tweetWithScore.tweetId,\n                      CandidateGenerationInfo(\n                        Some(query.sourceInfo),\n                        similarityEngineInfo,\n                        Seq(similarityEngineInfo)\n                      ))\n                }\n                val qigTweetsWithCGInfo = filteredQigTweets.map { tweetWithScore =>\n                  val similarityEngineInfo = TweetBasedQigSimilarityEngine\n                    .toSimilarityEngineInfo(tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n\n                val twHINTweetsWithCGInfo = filteredTwHINTweets.map { tweetWithScore =>\n                  val similarityEngineInfo = tweetBasedTwHINANNSimilarityEngine\n                    .toSimilarityEngineInfo(twhinQuery, tweetWithScore.score)\n                  TweetWithCandidateGenerationInfo(\n                    tweetWithScore.tweetId,\n                    CandidateGenerationInfo(\n                      Some(query.sourceInfo),\n                      similarityEngineInfo,\n                      Seq(similarityEngineInfo)\n                    ))\n                }\n\n                val candidateSourcesToBeInterleaved =\n                  ArrayBuffer[Seq[TweetWithCandidateGenerationInfo]](\n                    sannTweetsWithCGInfo,\n                    experimentalSANNTweetsWithCGInfo,\n                    sann1TweetsWithCGInfo,\n                    sann2TweetsWithCGInfo,\n                    sann3TweetsWithCGInfo,\n                    sann5TweetsWithCGInfo,\n                    sann4TweetsWithCGInfo,\n                    qigTweetsWithCGInfo,\n                    uvgTweetsWithCGInfo,\n                    utgTweetsWithCGInfo,\n                    twHINTweetsWithCGInfo\n                  )\n\n                val interleavedCandidates =\n                  InterleaveUtil.interleave(candidateSourcesToBeInterleaved)\n\n                val unifiedCandidatesWithUnifiedCGInfo =\n                  interleavedCandidates.map { candidate =>\n                    /***\n                     * when a candidate was made by interleave/keepGivenOrder,\n                     * then we apply getTweetBasedUnifiedCGInfo() to override with the unified CGInfo\n                     *\n                     * we'll not have ALL SEs that generated the tweet\n                     * in contributingSE list for interleave. We only have the chosen SE available.\n                     */\n                    TweetWithCandidateGenerationInfo(\n                      tweetId = candidate.tweetId,\n                      candidateGenerationInfo = getTweetBasedUnifiedCGInfo(\n                        candidate.candidateGenerationInfo.sourceInfoOpt,\n                        candidate.getSimilarityScore,\n                        candidate.candidateGenerationInfo.contributingSimilarityEngines\n                      ) // getSimilarityScore comes from either unifiedScore or single score\n                    )\n                  }\n                stats\n                  .stat(\"unified_candidate_size\").add(unifiedCandidatesWithUnifiedCGInfo.size)\n\n                val truncatedCandidates =\n                  unifiedCandidatesWithUnifiedCGInfo.take(query.maxCandidateNumPerSourceKey)\n                stats.stat(\"truncatedCandidates_size\").add(truncatedCandidates.size)\n\n                Some(truncatedCandidates)\n            }\n        }\n\n      case _ =>\n        stats.counter(\"sourceId_is_not_tweetId_cnt\").incr()\n        Future.None\n    }\n  }\n\n  private def simClustersCandidateMinScoreFilter(\n    simClustersAnnCandidates: Seq[TweetWithScore],\n    simClustersMinScore: Double,\n    simClustersANNConfigId: String\n  ): Seq[TweetWithScore] = {\n    val filteredCandidates = simClustersAnnCandidates\n      .filter { candidate =>\n        candidate.score > simClustersMinScore\n      }\n\n    stats.stat(simClustersANNConfigId, \"simClustersAnnCandidates_size\").add(filteredCandidates.size)\n    stats.counter(simClustersANNConfigId, \"simClustersAnnRequests\").incr()\n    if (filteredCandidates.isEmpty)\n      stats.counter(simClustersANNConfigId, \"emptyFilteredSimClustersAnnCandidates\").incr()\n\n    filteredCandidates.map { candidate =>\n      TweetWithScore(candidate.tweetId, candidate.score)\n    }\n  }\n\n  /** Returns a list of tweets that are generated less than `maxTweetAgeHours` hours ago */\n  private def tweetAgeFilter(\n    candidates: Seq[TweetWithScore],\n    maxTweetAgeHours: Duration\n  ): Seq[TweetWithScore] = {\n    // Tweet IDs are approximately chronological (see http://go/snowflake),\n    // so we are building the earliest tweet id once\n    // The per-candidate logic here then be candidate.tweetId > earliestPermittedTweetId, which is far cheaper.\n    val earliestTweetId = SnowflakeId.firstIdFor(Time.now - maxTweetAgeHours)\n    candidates.filter { candidate => candidate.tweetId >= earliestTweetId }\n  }\n\n  private def twhinFilter(\n    twhinCandidates: Seq[TweetWithScore],\n    twhinMaxTweetAgeHours: Duration,\n    simEngineStats: StatsReceiver\n  ): Seq[TweetWithScore] = {\n    simEngineStats.stat(\"twhinCandidates_size\").add(twhinCandidates.size)\n    val candidates = twhinCandidates.map { candidate =>\n      TweetWithScore(candidate.tweetId, candidate.score)\n    }\n\n    val filteredCandidates = tweetAgeFilter(candidates, twhinMaxTweetAgeHours)\n    simEngineStats.stat(\"filteredTwhinCandidates_size\").add(filteredCandidates.size)\n    if (filteredCandidates.isEmpty) simEngineStats.counter(\"emptyFilteredTwhinCandidates\").incr()\n\n    filteredCandidates\n  }\n\n  /** A no-op filter as UTG filtering already happens on UTG service side */\n  private def userTweetGraphFilter(\n    userTweetGraphCandidates: Seq[TweetWithScore]\n  ): Seq[TweetWithScore] = {\n    val filteredCandidates = userTweetGraphCandidates\n\n    stats.stat(\"userTweetGraphCandidates_size\").add(userTweetGraphCandidates.size)\n    if (filteredCandidates.isEmpty) stats.counter(\"emptyFilteredUserTweetGraphCandidates\").incr()\n\n    filteredCandidates.map { candidate =>\n      TweetWithScore(candidate.tweetId, candidate.score)\n    }\n  }\n\n  /** A no-op filter as UVG filtering already happens on UVG service side */\n  private def userVideoGraphFilter(\n    userVideoGraphCandidates: Seq[TweetWithScore]\n  ): Seq[TweetWithScore] = {\n    val filteredCandidates = userVideoGraphCandidates\n\n    stats.stat(\"userVideoGraphCandidates_size\").add(userVideoGraphCandidates.size)\n    if (filteredCandidates.isEmpty) stats.counter(\"emptyFilteredUserVideoGraphCandidates\").incr()\n\n    filteredCandidates.map { candidate =>\n      TweetWithScore(candidate.tweetId, candidate.score)\n    }\n  }\n  private def qigSimilarTweetsFilter(\n    qigSimilarTweetsCandidates: Seq[TweetWithScore],\n    qigMaxTweetAgeHours: Duration,\n    qigMaxNumSimilarTweets: Int\n  ): Seq[TweetWithScore] = {\n    val ageFilteredCandidates = tweetAgeFilter(qigSimilarTweetsCandidates, qigMaxTweetAgeHours)\n    stats.stat(\"ageFilteredQigSimilarTweetsCandidates_size\").add(ageFilteredCandidates.size)\n\n    val filteredCandidates = ageFilteredCandidates.take(qigMaxNumSimilarTweets)\n    if (filteredCandidates.isEmpty) stats.counter(\"emptyFilteredQigSimilarTweetsCandidates\").incr()\n\n    filteredCandidates\n  }\n\n  /***\n   * Every candidate will have the CG Info with TweetBasedUnifiedSimilarityEngine\n   * as they are generated by a composite of Similarity Engines.\n   * Additionally, we store the contributing SEs (eg., SANN, UTG).\n   */\n  private def getTweetBasedUnifiedCGInfo(\n    sourceInfoOpt: Option[SourceInfo],\n    unifiedScore: Double,\n    contributingSimilarityEngines: Seq[SimilarityEngineInfo]\n  ): CandidateGenerationInfo = {\n    CandidateGenerationInfo(\n      sourceInfoOpt,\n      SimilarityEngineInfo(\n        similarityEngineType = SimilarityEngineType.TweetBasedUnifiedSimilarityEngine,\n        modelId = None, // We do not assign modelId for a unified similarity engine\n        score = Some(unifiedScore)\n      ),\n      contributingSimilarityEngines\n    )\n  }\n}\n\nobject TweetBasedUnifiedSimilarityEngine {\n\n  case class Query(\n    sourceInfo: SourceInfo,\n    maxCandidateNumPerSourceKey: Int,\n    enableSimClustersANN: Boolean,\n    simClustersANNQuery: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    enableExperimentalSimClustersANN: Boolean,\n    experimentalSimClustersANNQuery: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    enableSimClustersANN1: Boolean,\n    simClustersANN1Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    enableSimClustersANN2: Boolean,\n    simClustersANN2Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    enableSimClustersANN3: Boolean,\n    simClustersANN3Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    enableSimClustersANN5: Boolean,\n    simClustersANN5Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    enableSimClustersANN4: Boolean,\n    simClustersANN4Query: EngineQuery[SimClustersANNSimilarityEngine.Query],\n    simClustersMinScore: Double,\n    simClustersVideoBasedMinScore: Double,\n    twhinModelId: String,\n    enableTwHIN: Boolean,\n    twhinMaxTweetAgeHours: Duration,\n    qigMaxTweetAgeHours: Duration,\n    qigMaxNumSimilarTweets: Int,\n    enableUtg: Boolean,\n    utgQuery: EngineQuery[TweetBasedUserTweetGraphSimilarityEngine.Query],\n    enableUvg: Boolean,\n    uvgQuery: EngineQuery[TweetBasedUserVideoGraphSimilarityEngine.Query],\n    enableQig: Boolean,\n    qigQuery: EngineQuery[TweetBasedQigSimilarityEngine.Query],\n    params: configapi.Params)\n\n  def fromParams(\n    sourceInfo: SourceInfo,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    // SimClusters\n    val enableSimClustersANN =\n      params(TweetBasedCandidateGenerationParams.EnableSimClustersANNParam)\n\n    val simClustersModelVersion =\n      ModelVersions.Enum.enumToSimClustersModelVersionMap(params(GlobalParams.ModelVersionParam))\n    val simClustersMinScore = params(TweetBasedCandidateGenerationParams.SimClustersMinScoreParam)\n    val simClustersVideoBasedMinScore = params(\n      TweetBasedCandidateGenerationParams.SimClustersVideoBasedMinScoreParam)\n    val simClustersANNConfigId = params(SimClustersANNParams.SimClustersANNConfigId)\n    // SimClusters - Experimental SANN Similarity Engine (Video based SE)\n    val enableExperimentalSimClustersANN =\n      params(TweetBasedCandidateGenerationParams.EnableExperimentalSimClustersANNParam)\n\n    val experimentalSimClustersANNConfigId = params(\n      SimClustersANNParams.ExperimentalSimClustersANNConfigId)\n    // SimClusters - SANN cluster 1 Similarity Engine\n    val enableSimClustersANN1 =\n      params(TweetBasedCandidateGenerationParams.EnableSimClustersANN1Param)\n\n    val simClustersANN1ConfigId = params(SimClustersANNParams.SimClustersANN1ConfigId)\n    // SimClusters - SANN cluster 2 Similarity Engine\n    val enableSimClustersANN2 =\n      params(TweetBasedCandidateGenerationParams.EnableSimClustersANN2Param)\n    val simClustersANN2ConfigId = params(SimClustersANNParams.SimClustersANN2ConfigId)\n    // SimClusters - SANN cluster 3 Similarity Engine\n    val enableSimClustersANN3 =\n      params(TweetBasedCandidateGenerationParams.EnableSimClustersANN3Param)\n    val simClustersANN3ConfigId = params(SimClustersANNParams.SimClustersANN3ConfigId)\n    // SimClusters - SANN cluster 5 Similarity Engine\n    val enableSimClustersANN5 =\n      params(TweetBasedCandidateGenerationParams.EnableSimClustersANN5Param)\n    val simClustersANN5ConfigId = params(SimClustersANNParams.SimClustersANN5ConfigId)\n    // SimClusters - SANN cluster 4 Similarity Engine\n    val enableSimClustersANN4 =\n      params(TweetBasedCandidateGenerationParams.EnableSimClustersANN4Param)\n    val simClustersANN4ConfigId = params(SimClustersANNParams.SimClustersANN4ConfigId)\n    // SimClusters ANN Queries for different SANN clusters\n    val simClustersANNQuery = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANNConfigId,\n      params\n    )\n    val experimentalSimClustersANNQuery = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      experimentalSimClustersANNConfigId,\n      params\n    )\n    val simClustersANN1Query = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN1ConfigId,\n      params\n    )\n    val simClustersANN2Query = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN2ConfigId,\n      params\n    )\n    val simClustersANN3Query = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN3ConfigId,\n      params\n    )\n    val simClustersANN5Query = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN5ConfigId,\n      params\n    )\n    val simClustersANN4Query = SimClustersANNSimilarityEngine.fromParams(\n      sourceInfo.internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN4ConfigId,\n      params\n    )\n    // TweetBasedCandidateGeneration\n    val maxCandidateNumPerSourceKey = params(GlobalParams.MaxCandidateNumPerSourceKeyParam)\n    // TwHIN\n    val twhinModelId = params(TweetBasedTwHINParams.ModelIdParam)\n    val enableTwHIN =\n      params(TweetBasedCandidateGenerationParams.EnableTwHINParam)\n\n    val twhinMaxTweetAgeHours = params(GlobalParams.MaxTweetAgeHoursParam)\n\n    // QIG\n    val enableQig =\n      params(TweetBasedCandidateGenerationParams.EnableQigSimilarTweetsParam)\n    val qigMaxTweetAgeHours = params(GlobalParams.MaxTweetAgeHoursParam)\n    val qigMaxNumSimilarTweets = params(\n      TweetBasedCandidateGenerationParams.QigMaxNumSimilarTweetsParam)\n\n    // UTG\n    val enableUtg =\n      params(TweetBasedCandidateGenerationParams.EnableUTGParam)\n    // UVG\n    val enableUvg =\n      params(TweetBasedCandidateGenerationParams.EnableUVGParam)\n    EngineQuery(\n      Query(\n        sourceInfo = sourceInfo,\n        maxCandidateNumPerSourceKey = maxCandidateNumPerSourceKey,\n        enableSimClustersANN = enableSimClustersANN,\n        simClustersANNQuery = simClustersANNQuery,\n        enableExperimentalSimClustersANN = enableExperimentalSimClustersANN,\n        experimentalSimClustersANNQuery = experimentalSimClustersANNQuery,\n        enableSimClustersANN1 = enableSimClustersANN1,\n        simClustersANN1Query = simClustersANN1Query,\n        enableSimClustersANN2 = enableSimClustersANN2,\n        simClustersANN2Query = simClustersANN2Query,\n        enableSimClustersANN3 = enableSimClustersANN3,\n        simClustersANN3Query = simClustersANN3Query,\n        enableSimClustersANN5 = enableSimClustersANN5,\n        simClustersANN5Query = simClustersANN5Query,\n        enableSimClustersANN4 = enableSimClustersANN4,\n        simClustersANN4Query = simClustersANN4Query,\n        simClustersMinScore = simClustersMinScore,\n        simClustersVideoBasedMinScore = simClustersVideoBasedMinScore,\n        twhinModelId = twhinModelId,\n        enableTwHIN = enableTwHIN,\n        twhinMaxTweetAgeHours = twhinMaxTweetAgeHours,\n        qigMaxTweetAgeHours = qigMaxTweetAgeHours,\n        qigMaxNumSimilarTweets = qigMaxNumSimilarTweets,\n        enableUtg = enableUtg,\n        utgQuery = TweetBasedUserTweetGraphSimilarityEngine\n          .fromParams(sourceInfo.internalId, params),\n        enableQig = enableQig,\n        qigQuery = TweetBasedQigSimilarityEngine.fromParams(sourceInfo.internalId, params),\n        enableUvg = enableUvg,\n        uvgQuery =\n          TweetBasedUserVideoGraphSimilarityEngine.fromParams(sourceInfo.internalId, params),\n        params = params\n      ),\n      params\n    )\n  }\n\n  def fromParamsForRelatedTweet(\n    internalId: InternalId,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    // SimClusters\n    val enableSimClustersANN = params(RelatedTweetTweetBasedParams.EnableSimClustersANNParam)\n    val simClustersModelVersion =\n      ModelVersions.Enum.enumToSimClustersModelVersionMap(params(GlobalParams.ModelVersionParam))\n    val simClustersMinScore = params(RelatedTweetTweetBasedParams.SimClustersMinScoreParam)\n    val simClustersANNConfigId = params(SimClustersANNParams.SimClustersANNConfigId)\n    val enableExperimentalSimClustersANN =\n      params(RelatedTweetTweetBasedParams.EnableExperimentalSimClustersANNParam)\n    val experimentalSimClustersANNConfigId = params(\n      SimClustersANNParams.ExperimentalSimClustersANNConfigId)\n    // SimClusters - SANN cluster 1 Similarity Engine\n    val enableSimClustersANN1 = params(RelatedTweetTweetBasedParams.EnableSimClustersANN1Param)\n    val simClustersANN1ConfigId = params(SimClustersANNParams.SimClustersANN1ConfigId)\n    // SimClusters - SANN cluster 2 Similarity Engine\n    val enableSimClustersANN2 = params(RelatedTweetTweetBasedParams.EnableSimClustersANN2Param)\n    val simClustersANN2ConfigId = params(SimClustersANNParams.SimClustersANN2ConfigId)\n    // SimClusters - SANN cluster 3 Similarity Engine\n    val enableSimClustersANN3 = params(RelatedTweetTweetBasedParams.EnableSimClustersANN3Param)\n    val simClustersANN3ConfigId = params(SimClustersANNParams.SimClustersANN3ConfigId)\n    // SimClusters - SANN cluster 5 Similarity Engine\n    val enableSimClustersANN5 = params(RelatedTweetTweetBasedParams.EnableSimClustersANN5Param)\n    val simClustersANN5ConfigId = params(SimClustersANNParams.SimClustersANN5ConfigId)\n    // SimClusters - SANN cluster 4 Similarity Engine\n    val enableSimClustersANN4 = params(RelatedTweetTweetBasedParams.EnableSimClustersANN4Param)\n    val simClustersANN4ConfigId = params(SimClustersANNParams.SimClustersANN4ConfigId)\n    // SimClusters ANN Queries for different SANN clusters\n    val simClustersANNQuery = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANNConfigId,\n      params\n    )\n    val experimentalSimClustersANNQuery = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      experimentalSimClustersANNConfigId,\n      params\n    )\n    val simClustersANN1Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN1ConfigId,\n      params\n    )\n    val simClustersANN2Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN2ConfigId,\n      params\n    )\n    val simClustersANN3Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN3ConfigId,\n      params\n    )\n    val simClustersANN5Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN5ConfigId,\n      params\n    )\n    val simClustersANN4Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN4ConfigId,\n      params\n    )\n    // TweetBasedCandidateGeneration\n    val maxCandidateNumPerSourceKey = params(GlobalParams.MaxCandidateNumPerSourceKeyParam)\n    // TwHIN\n    val twhinModelId = params(TweetBasedTwHINParams.ModelIdParam)\n    val enableTwHIN = params(RelatedTweetTweetBasedParams.EnableTwHINParam)\n    val twhinMaxTweetAgeHours = params(GlobalParams.MaxTweetAgeHoursParam)\n    // QIG\n    val enableQig = params(RelatedTweetTweetBasedParams.EnableQigSimilarTweetsParam)\n    val qigMaxTweetAgeHours = params(GlobalParams.MaxTweetAgeHoursParam)\n    val qigMaxNumSimilarTweets = params(\n      TweetBasedCandidateGenerationParams.QigMaxNumSimilarTweetsParam)\n    // UTG\n    val enableUtg = params(RelatedTweetTweetBasedParams.EnableUTGParam)\n    // UVG\n    val enableUvg = params(RelatedTweetTweetBasedParams.EnableUVGParam)\n    // SourceType.RequestTweetId is a placeholder.\n    val sourceInfo = SourceInfo(SourceType.RequestTweetId, internalId, None)\n\n    EngineQuery(\n      Query(\n        sourceInfo = sourceInfo,\n        maxCandidateNumPerSourceKey = maxCandidateNumPerSourceKey,\n        enableSimClustersANN = enableSimClustersANN,\n        simClustersMinScore = simClustersMinScore,\n        simClustersVideoBasedMinScore = simClustersMinScore,\n        simClustersANNQuery = simClustersANNQuery,\n        enableExperimentalSimClustersANN = enableExperimentalSimClustersANN,\n        experimentalSimClustersANNQuery = experimentalSimClustersANNQuery,\n        enableSimClustersANN1 = enableSimClustersANN1,\n        simClustersANN1Query = simClustersANN1Query,\n        enableSimClustersANN2 = enableSimClustersANN2,\n        simClustersANN2Query = simClustersANN2Query,\n        enableSimClustersANN3 = enableSimClustersANN3,\n        simClustersANN3Query = simClustersANN3Query,\n        enableSimClustersANN5 = enableSimClustersANN5,\n        simClustersANN5Query = simClustersANN5Query,\n        enableSimClustersANN4 = enableSimClustersANN4,\n        simClustersANN4Query = simClustersANN4Query,\n        twhinModelId = twhinModelId,\n        enableTwHIN = enableTwHIN,\n        twhinMaxTweetAgeHours = twhinMaxTweetAgeHours,\n        qigMaxTweetAgeHours = qigMaxTweetAgeHours,\n        qigMaxNumSimilarTweets = qigMaxNumSimilarTweets,\n        enableUtg = enableUtg,\n        utgQuery = TweetBasedUserTweetGraphSimilarityEngine\n          .fromParams(sourceInfo.internalId, params),\n        enableQig = enableQig,\n        qigQuery = TweetBasedQigSimilarityEngine.fromParams(sourceInfo.internalId, params),\n        enableUvg = enableUvg,\n        uvgQuery =\n          TweetBasedUserVideoGraphSimilarityEngine.fromParams(sourceInfo.internalId, params),\n        params = params,\n      ),\n      params\n    )\n  }\n  def fromParamsForRelatedVideoTweet(\n    internalId: InternalId,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    // SimClusters\n    val enableSimClustersANN = params(RelatedVideoTweetTweetBasedParams.EnableSimClustersANNParam)\n    val simClustersModelVersion =\n      ModelVersions.Enum.enumToSimClustersModelVersionMap(params(GlobalParams.ModelVersionParam))\n    val simClustersMinScore = params(RelatedVideoTweetTweetBasedParams.SimClustersMinScoreParam)\n    val simClustersANNConfigId = params(SimClustersANNParams.SimClustersANNConfigId)\n    val enableExperimentalSimClustersANN = params(\n      RelatedVideoTweetTweetBasedParams.EnableExperimentalSimClustersANNParam)\n    val experimentalSimClustersANNConfigId = params(\n      SimClustersANNParams.ExperimentalSimClustersANNConfigId)\n    // SimClusters - SANN cluster 1 Similarity Engine\n    val enableSimClustersANN1 = params(RelatedVideoTweetTweetBasedParams.EnableSimClustersANN1Param)\n    val simClustersANN1ConfigId = params(SimClustersANNParams.SimClustersANN1ConfigId)\n    // SimClusters - SANN cluster 2 Similarity Engine\n    val enableSimClustersANN2 = params(RelatedVideoTweetTweetBasedParams.EnableSimClustersANN2Param)\n    val simClustersANN2ConfigId = params(SimClustersANNParams.SimClustersANN2ConfigId)\n    // SimClusters - SANN cluster 3 Similarity Engine\n    val enableSimClustersANN3 = params(RelatedVideoTweetTweetBasedParams.EnableSimClustersANN3Param)\n    val simClustersANN3ConfigId = params(SimClustersANNParams.SimClustersANN3ConfigId)\n    // SimClusters - SANN cluster 5 Similarity Engine\n    val enableSimClustersANN5 = params(RelatedVideoTweetTweetBasedParams.EnableSimClustersANN5Param)\n    val simClustersANN5ConfigId = params(SimClustersANNParams.SimClustersANN5ConfigId)\n\n    // SimClusters - SANN cluster 4 Similarity Engine\n    val enableSimClustersANN4 = params(RelatedVideoTweetTweetBasedParams.EnableSimClustersANN4Param)\n    val simClustersANN4ConfigId = params(SimClustersANNParams.SimClustersANN4ConfigId)\n    // SimClusters ANN Queries for different SANN clusters\n    val simClustersANNQuery = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANNConfigId,\n      params\n    )\n    val experimentalSimClustersANNQuery = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      experimentalSimClustersANNConfigId,\n      params\n    )\n    val simClustersANN1Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN1ConfigId,\n      params\n    )\n    val simClustersANN2Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN2ConfigId,\n      params\n    )\n    val simClustersANN3Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN3ConfigId,\n      params\n    )\n    val simClustersANN5Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN5ConfigId,\n      params\n    )\n\n    val simClustersANN4Query = SimClustersANNSimilarityEngine.fromParams(\n      internalId,\n      EmbeddingType.LogFavLongestL2EmbeddingTweet,\n      simClustersModelVersion,\n      simClustersANN4ConfigId,\n      params\n    )\n    // TweetBasedCandidateGeneration\n    val maxCandidateNumPerSourceKey = params(GlobalParams.MaxCandidateNumPerSourceKeyParam)\n    // TwHIN\n    val twhinModelId = params(TweetBasedTwHINParams.ModelIdParam)\n    val enableTwHIN = params(RelatedVideoTweetTweetBasedParams.EnableTwHINParam)\n    val twhinMaxTweetAgeHours = params(GlobalParams.MaxTweetAgeHoursParam)\n    // QIG\n    val enableQig = params(RelatedVideoTweetTweetBasedParams.EnableQigSimilarTweetsParam)\n    val qigMaxTweetAgeHours = params(GlobalParams.MaxTweetAgeHoursParam)\n    val qigMaxNumSimilarTweets = params(\n      TweetBasedCandidateGenerationParams.QigMaxNumSimilarTweetsParam)\n    // UTG\n    val enableUtg = params(RelatedVideoTweetTweetBasedParams.EnableUTGParam)\n\n    // SourceType.RequestTweetId is a placeholder.\n    val sourceInfo = SourceInfo(SourceType.RequestTweetId, internalId, None)\n\n    val enableUvg = params(RelatedVideoTweetTweetBasedParams.EnableUVGParam)\n    EngineQuery(\n      Query(\n        sourceInfo = sourceInfo,\n        maxCandidateNumPerSourceKey = maxCandidateNumPerSourceKey,\n        enableSimClustersANN = enableSimClustersANN,\n        simClustersMinScore = simClustersMinScore,\n        simClustersVideoBasedMinScore = simClustersMinScore,\n        simClustersANNQuery = simClustersANNQuery,\n        enableExperimentalSimClustersANN = enableExperimentalSimClustersANN,\n        experimentalSimClustersANNQuery = experimentalSimClustersANNQuery,\n        enableSimClustersANN1 = enableSimClustersANN1,\n        simClustersANN1Query = simClustersANN1Query,\n        enableSimClustersANN2 = enableSimClustersANN2,\n        simClustersANN2Query = simClustersANN2Query,\n        enableSimClustersANN3 = enableSimClustersANN3,\n        simClustersANN3Query = simClustersANN3Query,\n        enableSimClustersANN5 = enableSimClustersANN5,\n        simClustersANN5Query = simClustersANN5Query,\n        enableSimClustersANN4 = enableSimClustersANN4,\n        simClustersANN4Query = simClustersANN4Query,\n        twhinModelId = twhinModelId,\n        enableTwHIN = enableTwHIN,\n        twhinMaxTweetAgeHours = twhinMaxTweetAgeHours,\n        qigMaxTweetAgeHours = qigMaxTweetAgeHours,\n        qigMaxNumSimilarTweets = qigMaxNumSimilarTweets,\n        enableUtg = enableUtg,\n        utgQuery = TweetBasedUserTweetGraphSimilarityEngine\n          .fromParams(sourceInfo.internalId, params),\n        enableUvg = enableUvg,\n        uvgQuery =\n          TweetBasedUserVideoGraphSimilarityEngine.fromParams(sourceInfo.internalId, params),\n        enableQig = enableQig,\n        qigQuery = TweetBasedQigSimilarityEngine.fromParams(sourceInfo.internalId, params),\n        params = params\n      ),\n      params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/TweetBasedUserAdGraphSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.param.TweetBasedUserAdGraphParams\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.recos.user_ad_graph.thriftscala.ConsumersBasedRelatedAdRequest\nimport com.twitter.recos.user_ad_graph.thriftscala.RelatedAdResponse\nimport com.twitter.recos.user_ad_graph.thriftscala.UserAdGraph\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.twistly.thriftscala.TweetRecentEngagedUsers\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n/**\n * This store looks for similar tweets from UserAdGraph for a Source TweetId\n * For a query tweet,User Ad Graph (UAG)\n * lets us find out which other tweets share a lot of the same engagers with the query tweet\n */\n@Singleton\ncase class TweetBasedUserAdGraphSimilarityEngine(\n  userAdGraphService: UserAdGraph.MethodPerEndpoint,\n  tweetEngagedUsersStore: ReadableStore[TweetId, TweetRecentEngagedUsers],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[\n      TweetBasedUserAdGraphSimilarityEngine.Query,\n      Seq[TweetWithScore]\n    ] {\n\n  import TweetBasedUserAdGraphSimilarityEngine._\n\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val fetchCoverageExpansionCandidatesStat = stats.scope(\"fetchCoverageExpansionCandidates\")\n  override def get(\n    query: TweetBasedUserAdGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    query.sourceId match {\n      case InternalId.TweetId(tweetId) => getCandidates(tweetId, query)\n      case _ =>\n        Future.value(None)\n    }\n  }\n\n  // We first fetch tweet's recent engaged users as consumeSeedSet from MH store,\n  // then query consumersBasedUTG using the consumerSeedSet\n  private def getCandidates(\n    tweetId: TweetId,\n    query: TweetBasedUserAdGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    StatsUtil\n      .trackOptionItemsStats(fetchCoverageExpansionCandidatesStat) {\n        tweetEngagedUsersStore\n          .get(tweetId).flatMap {\n            _.map { tweetRecentEngagedUsers =>\n              val consumerSeedSet =\n                tweetRecentEngagedUsers.recentEngagedUsers\n                  .map { _.userId }.take(query.maxConsumerSeedsNum)\n              val consumersBasedRelatedAdRequest =\n                ConsumersBasedRelatedAdRequest(\n                  consumerSeedSet = consumerSeedSet,\n                  maxResults = Some(query.maxResults),\n                  minCooccurrence = Some(query.minCooccurrence),\n                  excludeTweetIds = Some(Seq(tweetId)),\n                  minScore = Some(query.consumersBasedMinScore),\n                  maxTweetAgeInHours = Some(query.maxTweetAgeInHours)\n                )\n              toTweetWithScore(userAdGraphService\n                .consumersBasedRelatedAds(consumersBasedRelatedAdRequest).map { Some(_) })\n            }.getOrElse(Future.value(None))\n          }\n      }\n  }\n\n}\n\nobject TweetBasedUserAdGraphSimilarityEngine {\n\n  def toSimilarityEngineInfo(score: Double): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.TweetBasedUserAdGraph,\n      modelId = None,\n      score = Some(score))\n  }\n  private def toTweetWithScore(\n    relatedAdResponseFut: Future[Option[RelatedAdResponse]]\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    relatedAdResponseFut.map { relatedAdResponseOpt =>\n      relatedAdResponseOpt.map { relatedAdResponse =>\n        val candidates =\n          relatedAdResponse.adTweets.map(tweet => TweetWithScore(tweet.adTweetId, tweet.score))\n\n        candidates\n      }\n    }\n  }\n\n  case class Query(\n    sourceId: InternalId,\n    maxResults: Int,\n    minCooccurrence: Int,\n    consumersBasedMinScore: Double,\n    maxTweetAgeInHours: Int,\n    maxConsumerSeedsNum: Int,\n  )\n\n  def fromParams(\n    sourceId: InternalId,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    EngineQuery(\n      Query(\n        sourceId = sourceId,\n        maxResults = params(GlobalParams.MaxCandidateNumPerSourceKeyParam),\n        minCooccurrence = params(TweetBasedUserAdGraphParams.MinCoOccurrenceParam),\n        consumersBasedMinScore = params(TweetBasedUserAdGraphParams.ConsumersBasedMinScoreParam),\n        maxTweetAgeInHours = params(GlobalParams.MaxTweetAgeHoursParam).inHours,\n        maxConsumerSeedsNum = params(TweetBasedUserAdGraphParams.MaxConsumerSeedsNumParam),\n      ),\n      params\n    )\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/TweetBasedUserTweetGraphSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.param.TweetBasedUserTweetGraphParams\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.recos.user_tweet_graph.thriftscala.RelatedTweetResponse\nimport com.twitter.recos.user_tweet_graph.thriftscala.TweetBasedRelatedTweetRequest\nimport com.twitter.recos.user_tweet_graph.thriftscala.ConsumersBasedRelatedTweetRequest\nimport com.twitter.recos.user_tweet_graph.thriftscala.UserTweetGraph\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.twistly.thriftscala.TweetRecentEngagedUsers\nimport com.twitter.util.Future\nimport javax.inject.Singleton\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\nimport scala.concurrent.duration.HOURS\n\n/**\n * This store looks for similar tweets from UserTweetGraph for a Source TweetId\n * For a query tweet,User Tweet Graph (UTG),\n * lets us find out which other tweets share a lot of the same engagers with the query tweet\n * one-pager: go/UTG\n */\n@Singleton\ncase class TweetBasedUserTweetGraphSimilarityEngine(\n  userTweetGraphService: UserTweetGraph.MethodPerEndpoint,\n  tweetEngagedUsersStore: ReadableStore[TweetId, TweetRecentEngagedUsers],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[\n      TweetBasedUserTweetGraphSimilarityEngine.Query,\n      Seq[TweetWithScore]\n    ] {\n\n  import TweetBasedUserTweetGraphSimilarityEngine._\n\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val fetchCandidatesStat = stats.scope(\"fetchCandidates\")\n  private val fetchCoverageExpansionCandidatesStat = stats.scope(\"fetchCoverageExpansionCandidates\")\n\n  override def get(\n    query: TweetBasedUserTweetGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    query.sourceId match {\n      case InternalId.TweetId(tweetId) if query.enableCoverageExpansionAllTweet =>\n        getCoverageExpansionCandidates(tweetId, query)\n\n      case InternalId.TweetId(tweetId) if query.enableCoverageExpansionOldTweet => // For Home\n        if (isOldTweet(tweetId)) getCoverageExpansionCandidates(tweetId, query)\n        else getCandidates(tweetId, query)\n\n      case InternalId.TweetId(tweetId) => getCandidates(tweetId, query)\n      case _ =>\n        Future.value(None)\n    }\n  }\n\n  // This is the main candidate source\n  private def getCandidates(\n    tweetId: TweetId,\n    query: TweetBasedUserTweetGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    StatsUtil.trackOptionItemsStats(fetchCandidatesStat) {\n      val tweetBasedRelatedTweetRequest = {\n        TweetBasedRelatedTweetRequest(\n          tweetId,\n          maxResults = Some(query.maxResults),\n          minCooccurrence = Some(query.minCooccurrence),\n          excludeTweetIds = Some(Seq(tweetId)),\n          minScore = Some(query.tweetBasedMinScore),\n          maxTweetAgeInHours = Some(query.maxTweetAgeInHours)\n        )\n      }\n      toTweetWithScore(\n        userTweetGraphService.tweetBasedRelatedTweets(tweetBasedRelatedTweetRequest).map {\n          Some(_)\n        })\n    }\n  }\n\n  // function for DDGs, for coverage expansion algo, we first fetch tweet's recent engaged users as consumeSeedSet from MH store,\n  // and query consumersBasedUTG using the consumeSeedSet\n  private def getCoverageExpansionCandidates(\n    tweetId: TweetId,\n    query: TweetBasedUserTweetGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    StatsUtil\n      .trackOptionItemsStats(fetchCoverageExpansionCandidatesStat) {\n        tweetEngagedUsersStore\n          .get(tweetId).flatMap {\n            _.map { tweetRecentEngagedUsers =>\n              val consumerSeedSet =\n                tweetRecentEngagedUsers.recentEngagedUsers\n                  .map { _.userId }.take(query.maxConsumerSeedsNum)\n              val consumersBasedRelatedTweetRequest =\n                ConsumersBasedRelatedTweetRequest(\n                  consumerSeedSet = consumerSeedSet,\n                  maxResults = Some(query.maxResults),\n                  minCooccurrence = Some(query.minCooccurrence),\n                  excludeTweetIds = Some(Seq(tweetId)),\n                  minScore = Some(query.consumersBasedMinScore),\n                  maxTweetAgeInHours = Some(query.maxTweetAgeInHours)\n                )\n\n              toTweetWithScore(userTweetGraphService\n                .consumersBasedRelatedTweets(consumersBasedRelatedTweetRequest).map { Some(_) })\n            }.getOrElse(Future.value(None))\n          }\n      }\n  }\n\n}\n\nobject TweetBasedUserTweetGraphSimilarityEngine {\n\n  def toSimilarityEngineInfo(score: Double): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.TweetBasedUserTweetGraph,\n      modelId = None,\n      score = Some(score))\n  }\n\n  private val oldTweetCap: Duration = Duration(48, HOURS)\n\n  private def toTweetWithScore(\n    relatedTweetResponseFut: Future[Option[RelatedTweetResponse]]\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    relatedTweetResponseFut.map { relatedTweetResponseOpt =>\n      relatedTweetResponseOpt.map { relatedTweetResponse =>\n        val candidates =\n          relatedTweetResponse.tweets.map(tweet => TweetWithScore(tweet.tweetId, tweet.score))\n        candidates\n      }\n    }\n  }\n\n  private def isOldTweet(tweetId: TweetId): Boolean = {\n    SnowflakeId\n      .timeFromIdOpt(tweetId).forall { tweetTime => tweetTime < Time.now - oldTweetCap }\n    // If there's no snowflake timestamp, we have no idea when this tweet happened.\n  }\n\n  case class Query(\n    sourceId: InternalId,\n    maxResults: Int,\n    minCooccurrence: Int,\n    tweetBasedMinScore: Double,\n    consumersBasedMinScore: Double,\n    maxTweetAgeInHours: Int,\n    maxConsumerSeedsNum: Int,\n    enableCoverageExpansionOldTweet: Boolean,\n    enableCoverageExpansionAllTweet: Boolean,\n  )\n\n  def fromParams(\n    sourceId: InternalId,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    EngineQuery(\n      Query(\n        sourceId = sourceId,\n        maxResults = params(GlobalParams.MaxCandidateNumPerSourceKeyParam),\n        minCooccurrence = params(TweetBasedUserTweetGraphParams.MinCoOccurrenceParam),\n        tweetBasedMinScore = params(TweetBasedUserTweetGraphParams.TweetBasedMinScoreParam),\n        consumersBasedMinScore = params(TweetBasedUserTweetGraphParams.ConsumersBasedMinScoreParam),\n        maxTweetAgeInHours = params(GlobalParams.MaxTweetAgeHoursParam).inHours,\n        maxConsumerSeedsNum = params(TweetBasedUserTweetGraphParams.MaxConsumerSeedsNumParam),\n        enableCoverageExpansionOldTweet =\n          params(TweetBasedUserTweetGraphParams.EnableCoverageExpansionOldTweetParam),\n        enableCoverageExpansionAllTweet =\n          params(TweetBasedUserTweetGraphParams.EnableCoverageExpansionAllTweetParam),\n      ),\n      params\n    )\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/TweetBasedUserVideoGraphSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.param.TweetBasedUserVideoGraphParams\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.recos.user_video_graph.thriftscala.RelatedTweetResponse\nimport com.twitter.recos.user_video_graph.thriftscala.ConsumersBasedRelatedTweetRequest\nimport com.twitter.recos.user_video_graph.thriftscala.TweetBasedRelatedTweetRequest\nimport com.twitter.recos.user_video_graph.thriftscala.UserVideoGraph\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.timelines.configapi\nimport com.twitter.twistly.thriftscala.TweetRecentEngagedUsers\nimport com.twitter.util.Duration\nimport javax.inject.Singleton\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport scala.concurrent.duration.HOURS\n\n/**\n * This store looks for similar tweets from UserVideoGraph for a Source TweetId\n * For a query tweet,User Video Graph (UVG),\n * lets us find out which other video tweets share a lot of the same engagers with the query tweet\n */\n@Singleton\ncase class TweetBasedUserVideoGraphSimilarityEngine(\n  userVideoGraphService: UserVideoGraph.MethodPerEndpoint,\n  tweetEngagedUsersStore: ReadableStore[TweetId, TweetRecentEngagedUsers],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[\n      TweetBasedUserVideoGraphSimilarityEngine.Query,\n      Seq[TweetWithScore]\n    ] {\n\n  import TweetBasedUserVideoGraphSimilarityEngine._\n\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val fetchCandidatesStat = stats.scope(\"fetchCandidates\")\n  private val fetchCoverageExpansionCandidatesStat = stats.scope(\"fetchCoverageExpansionCandidates\")\n\n  override def get(\n    query: TweetBasedUserVideoGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n\n    query.sourceId match {\n      case InternalId.TweetId(tweetId) if query.enableCoverageExpansionAllTweet =>\n        getCoverageExpansionCandidates(tweetId, query)\n\n      case InternalId.TweetId(tweetId) if query.enableCoverageExpansionOldTweet => // For Home\n        if (isOldTweet(tweetId)) getCoverageExpansionCandidates(tweetId, query)\n        else getCandidates(tweetId, query)\n\n      case InternalId.TweetId(tweetId) => getCandidates(tweetId, query)\n      case _ =>\n        Future.value(None)\n    }\n  }\n\n  private def getCandidates(\n    tweetId: TweetId,\n    query: TweetBasedUserVideoGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    StatsUtil.trackOptionItemsStats(fetchCandidatesStat) {\n      val tweetBasedRelatedTweetRequest = {\n        TweetBasedRelatedTweetRequest(\n          tweetId,\n          maxResults = Some(query.maxResults),\n          minCooccurrence = Some(query.minCooccurrence),\n          excludeTweetIds = Some(Seq(tweetId)),\n          minScore = Some(query.tweetBasedMinScore),\n          maxTweetAgeInHours = Some(query.maxTweetAgeInHours)\n        )\n      }\n      toTweetWithScore(\n        userVideoGraphService.tweetBasedRelatedTweets(tweetBasedRelatedTweetRequest).map {\n          Some(_)\n        })\n    }\n  }\n\n  private def getCoverageExpansionCandidates(\n    tweetId: TweetId,\n    query: TweetBasedUserVideoGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    StatsUtil\n      .trackOptionItemsStats(fetchCoverageExpansionCandidatesStat) {\n        tweetEngagedUsersStore\n          .get(tweetId).flatMap {\n            _.map { tweetRecentEngagedUsers =>\n              val consumerSeedSet =\n                tweetRecentEngagedUsers.recentEngagedUsers\n                  .map {\n                    _.userId\n                  }.take(query.maxConsumerSeedsNum)\n              val consumersBasedRelatedTweetRequest =\n                ConsumersBasedRelatedTweetRequest(\n                  consumerSeedSet = consumerSeedSet,\n                  maxResults = Some(query.maxResults),\n                  minCooccurrence = Some(query.minCooccurrence),\n                  excludeTweetIds = Some(Seq(tweetId)),\n                  minScore = Some(query.consumersBasedMinScore),\n                  maxTweetAgeInHours = Some(query.maxTweetAgeInHours)\n                )\n\n              toTweetWithScore(userVideoGraphService\n                .consumersBasedRelatedTweets(consumersBasedRelatedTweetRequest).map {\n                  Some(_)\n                })\n            }.getOrElse(Future.value(None))\n          }\n      }\n  }\n\n}\n\nobject TweetBasedUserVideoGraphSimilarityEngine {\n\n  private val oldTweetCap: Duration = Duration(24, HOURS)\n\n  def toSimilarityEngineInfo(score: Double): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.TweetBasedUserVideoGraph,\n      modelId = None,\n      score = Some(score))\n  }\n\n  private def toTweetWithScore(\n    relatedTweetResponseFut: Future[Option[RelatedTweetResponse]]\n  ): Future[Option[Seq[TweetWithScore]]] = {\n    relatedTweetResponseFut.map { relatedTweetResponseOpt =>\n      relatedTweetResponseOpt.map { relatedTweetResponse =>\n        val candidates =\n          relatedTweetResponse.tweets.map(tweet => TweetWithScore(tweet.tweetId, tweet.score))\n        candidates\n      }\n    }\n  }\n\n  private def isOldTweet(tweetId: TweetId): Boolean = {\n    SnowflakeId\n      .timeFromIdOpt(tweetId).forall { tweetTime => tweetTime < Time.now - oldTweetCap }\n    // If there's no snowflake timestamp, we have no idea when this tweet happened.\n  }\n\n  case class Query(\n    sourceId: InternalId,\n    maxResults: Int,\n    minCooccurrence: Int,\n    tweetBasedMinScore: Double,\n    consumersBasedMinScore: Double,\n    maxTweetAgeInHours: Int,\n    maxConsumerSeedsNum: Int,\n    enableCoverageExpansionOldTweet: Boolean,\n    enableCoverageExpansionAllTweet: Boolean)\n\n  def fromParams(\n    sourceId: InternalId,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    EngineQuery(\n      Query(\n        sourceId = sourceId,\n        maxResults = params(GlobalParams.MaxCandidateNumPerSourceKeyParam),\n        minCooccurrence = params(TweetBasedUserVideoGraphParams.MinCoOccurrenceParam),\n        tweetBasedMinScore = params(TweetBasedUserVideoGraphParams.TweetBasedMinScoreParam),\n        consumersBasedMinScore = params(TweetBasedUserVideoGraphParams.ConsumersBasedMinScoreParam),\n        maxTweetAgeInHours = params(GlobalParams.MaxTweetAgeHoursParam).inHours,\n        maxConsumerSeedsNum = params(TweetBasedUserVideoGraphParams.MaxConsumerSeedsNumParam),\n        enableCoverageExpansionOldTweet =\n          params(TweetBasedUserVideoGraphParams.EnableCoverageExpansionOldTweetParam),\n        enableCoverageExpansionAllTweet =\n          params(TweetBasedUserVideoGraphParams.EnableCoverageExpansionAllTweetParam)\n      ),\n      params\n    )\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/TwhinCollabFilterSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.cr_mixer.model.TweetWithScore\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n@Singleton\ncase class TwhinCollabFilterSimilarityEngine(\n  twhinCandidatesStratoStore: ReadableStore[Long, Seq[TweetId]],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[\n      TwhinCollabFilterSimilarityEngine.Query,\n      Seq[TweetWithScore]\n    ] {\n\n  import TwhinCollabFilterSimilarityEngine._\n  override def get(\n    query: TwhinCollabFilterSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScore]]] = {\n\n    query.sourceId match {\n      case InternalId.UserId(userId) =>\n        twhinCandidatesStratoStore.get(userId).map {\n          _.map {\n            _.map { tweetId => TweetWithScore(tweetId, defaultScore) }\n          }\n        }\n      case _ =>\n        Future.None\n    }\n  }\n}\n\nobject TwhinCollabFilterSimilarityEngine {\n\n  val defaultScore: Double = 1.0\n\n  case class TwhinCollabFilterView(clusterVersion: String)\n\n  case class Query(\n    sourceId: InternalId,\n  )\n\n  def toSimilarityEngineInfo(\n    query: LookupEngineQuery[Query],\n    score: Double\n  ): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.TwhinCollabFilter,\n      modelId = Some(query.lookupKey),\n      score = Some(score))\n  }\n\n  def fromParams(\n    sourceId: InternalId,\n    modelId: String,\n    params: configapi.Params,\n  ): LookupEngineQuery[Query] = {\n    LookupEngineQuery(\n      Query(sourceId = sourceId),\n      modelId,\n      params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/similarity_engine/UserTweetEntityGraphSimilarityEngine.scala",
    "content": "package com.twitter.cr_mixer.similarity_engine\n\nimport com.twitter.recos.recos_common.thriftscala.SocialProofType\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.TweetWithScoreAndSocialProof\nimport com.twitter.cr_mixer.param.UtegTweetGlobalParams\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.user_tweet_entity_graph.thriftscala.TweetEntityDisplayLocation\nimport com.twitter.recos.user_tweet_entity_graph.thriftscala.UserTweetEntityGraph\nimport com.twitter.recos.user_tweet_entity_graph.thriftscala.RecommendTweetEntityRequest\nimport com.twitter.recos.user_tweet_entity_graph.thriftscala.RecommendationType\nimport com.twitter.recos.user_tweet_entity_graph.thriftscala.UserTweetEntityRecommendationUnion.TweetRec\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n@Singleton\ncase class UserTweetEntityGraphSimilarityEngine(\n  userTweetEntityGraph: UserTweetEntityGraph.MethodPerEndpoint,\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[\n      UserTweetEntityGraphSimilarityEngine.Query,\n      Seq[TweetWithScoreAndSocialProof]\n    ] {\n\n  override def get(\n    query: UserTweetEntityGraphSimilarityEngine.Query\n  ): Future[Option[Seq[TweetWithScoreAndSocialProof]]] = {\n    val recommendTweetEntityRequest =\n      RecommendTweetEntityRequest(\n        requesterId = query.userId,\n        displayLocation = TweetEntityDisplayLocation.HomeTimeline,\n        recommendationTypes = Seq(RecommendationType.Tweet),\n        seedsWithWeights = query.seedsWithWeights,\n        maxResultsByType = Some(Map(RecommendationType.Tweet -> query.maxUtegCandidates)),\n        maxTweetAgeInMillis = Some(query.maxTweetAge.inMilliseconds),\n        excludedTweetIds = query.excludedTweetIds,\n        maxUserSocialProofSize = Some(UserTweetEntityGraphSimilarityEngine.MaxUserSocialProofSize),\n        maxTweetSocialProofSize =\n          Some(UserTweetEntityGraphSimilarityEngine.MaxTweetSocialProofSize),\n        minUserSocialProofSizes = Some(Map(RecommendationType.Tweet -> 1)),\n        tweetTypes = None,\n        socialProofTypes = query.socialProofTypes,\n        socialProofTypeUnions = None,\n        tweetAuthors = None,\n        maxEngagementAgeInMillis = None,\n        excludedTweetAuthors = None,\n      )\n\n    userTweetEntityGraph\n      .recommendTweets(recommendTweetEntityRequest)\n      .map { recommendTweetsResponse =>\n        val candidates = recommendTweetsResponse.recommendations.flatMap {\n          case TweetRec(recommendation) =>\n            Some(\n              TweetWithScoreAndSocialProof(\n                recommendation.tweetId,\n                recommendation.score,\n                recommendation.socialProofByType.toMap))\n          case _ => None\n        }\n        Some(candidates)\n      }\n  }\n}\n\nobject UserTweetEntityGraphSimilarityEngine {\n\n  private val MaxUserSocialProofSize = 10\n  private val MaxTweetSocialProofSize = 10\n\n  def toSimilarityEngineInfo(score: Double): SimilarityEngineInfo = {\n    SimilarityEngineInfo(\n      similarityEngineType = SimilarityEngineType.Uteg,\n      modelId = None,\n      score = Some(score))\n  }\n\n  case class Query(\n    userId: UserId,\n    seedsWithWeights: Map[UserId, Double],\n    excludedTweetIds: Option[Seq[Long]] = None,\n    maxUtegCandidates: Int,\n    maxTweetAge: Duration,\n    socialProofTypes: Option[Seq[SocialProofType]])\n\n  def fromParams(\n    userId: UserId,\n    seedsWithWeights: Map[UserId, Double],\n    excludedTweetIds: Option[Seq[TweetId]] = None,\n    params: configapi.Params,\n  ): EngineQuery[Query] = {\n    EngineQuery(\n      Query(\n        userId = userId,\n        seedsWithWeights = seedsWithWeights,\n        excludedTweetIds = excludedTweetIds,\n        maxUtegCandidates = params(UtegTweetGlobalParams.MaxUtegCandidatesToRequestParam),\n        maxTweetAge = params(UtegTweetGlobalParams.CandidateRefreshSinceTimeOffsetHoursParam),\n        socialProofTypes = Some(Seq(SocialProofType.Favorite))\n      ),\n      params\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"configapi/configapi-core\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/config\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param/decider\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"decider/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"frigate/frigate-common:base\",\n        \"frigate/frigate-common:util\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/base\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/candidate\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/core_workflows/user_model:user_model-scala\",\n        \"src/thrift/com/twitter/hermit/stp:hermit-stp-scala\",\n        \"src/thrift/com/twitter/onboarding/relevance/coldstart_lookalike:coldstartlookalike-thrift-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala\",\n        \"user-signal-service/thrift/src/main/thrift:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal/FrsSourceGraphFetcher.scala",
    "content": "package com.twitter.cr_mixer.source_signal\n\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.GraphSourceInfo\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.param.FrsParams\nimport com.twitter.cr_mixer.source_signal.FrsStore.FrsQueryResult\nimport com.twitter.cr_mixer.source_signal.SourceFetcher.FetcherQuery\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/***\n * This store fetches user recommendations from FRS (go/frs) for a given userId\n */\n@Singleton\ncase class FrsSourceGraphFetcher @Inject() (\n  @Named(ModuleNames.FrsStore) frsStore: ReadableStore[FrsStore.Query, Seq[FrsQueryResult]],\n  override val timeoutConfig: TimeoutConfig,\n  globalStats: StatsReceiver)\n    extends SourceGraphFetcher {\n\n  override protected val stats: StatsReceiver = globalStats.scope(identifier)\n  override protected val graphSourceType: SourceType = SourceType.FollowRecommendation\n\n  override def isEnabled(query: FetcherQuery): Boolean = {\n    query.params(FrsParams.EnableSourceGraphParam)\n  }\n\n  override def fetchAndProcess(\n    query: FetcherQuery,\n  ): Future[Option[GraphSourceInfo]] = {\n\n    val rawSignals = trackPerItemStats(query)(\n      frsStore\n        .get(\n          FrsStore\n            .Query(query.userId, query.params(FrsParams.MaxConsumerSeedsNumParam))).map {\n          _.map {\n            _.map { v => (v.userId, v.score) }\n          }\n        }\n    )\n    rawSignals.map {\n      _.map { userWithScores =>\n        convertGraphSourceInfo(userWithScores)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal/FrsSourceSignalFetcher.scala",
    "content": "package com.twitter.cr_mixer.source_signal\n\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.param.FrsParams\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.source_signal.FrsStore.FrsQueryResult\nimport com.twitter.cr_mixer.source_signal.SourceFetcher.FetcherQuery\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport javax.inject.Singleton\nimport javax.inject.Inject\nimport javax.inject.Named\n\n@Singleton\ncase class FrsSourceSignalFetcher @Inject() (\n  @Named(ModuleNames.FrsStore) frsStore: ReadableStore[FrsStore.Query, Seq[FrsQueryResult]],\n  override val timeoutConfig: TimeoutConfig,\n  globalStats: StatsReceiver)\n    extends SourceSignalFetcher {\n\n  override protected val stats: StatsReceiver = globalStats.scope(identifier)\n  override type SignalConvertType = UserId\n\n  override def isEnabled(query: FetcherQuery): Boolean = {\n    query.params(FrsParams.EnableSourceParam)\n  }\n\n  override def fetchAndProcess(query: FetcherQuery): Future[Option[Seq[SourceInfo]]] = {\n    // Fetch raw signals\n    val rawSignals = frsStore\n      .get(FrsStore.Query(query.userId, query.params(GlobalParams.UnifiedMaxSourceKeyNum)))\n      .map {\n        _.map {\n          _.map {\n            _.userId\n          }\n        }\n      }\n    // Process signals\n    rawSignals.map {\n      _.map { frsUsers =>\n        convertSourceInfo(SourceType.FollowRecommendation, frsUsers)\n      }\n    }\n  }\n\n  override def convertSourceInfo(\n    sourceType: SourceType,\n    signals: Seq[SignalConvertType]\n  ): Seq[SourceInfo] = {\n    signals.map { signal =>\n      SourceInfo(\n        sourceType = sourceType,\n        internalId = InternalId.UserId(signal),\n        sourceEventTime = None\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal/FrsStore.scala",
    "content": "package com.twitter.cr_mixer.source_signal\n\nimport com.twitter.cr_mixer.param.decider.CrMixerDecider\nimport com.twitter.cr_mixer.param.decider.DeciderConstants\nimport com.twitter.cr_mixer.source_signal.FrsStore.Query\nimport com.twitter.cr_mixer.source_signal.FrsStore.FrsQueryResult\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.thriftscala.ClientContext\nimport com.twitter.follow_recommendations.thriftscala.DisplayLocation\nimport com.twitter.follow_recommendations.thriftscala.FollowRecommendationsThriftService\nimport com.twitter.follow_recommendations.thriftscala.Recommendation\nimport com.twitter.follow_recommendations.thriftscala.RecommendationRequest\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Singleton\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.util.Future\n\n@Singleton\ncase class FrsStore(\n  frsClient: FollowRecommendationsThriftService.MethodPerEndpoint,\n  statsReceiver: StatsReceiver,\n  decider: CrMixerDecider)\n    extends ReadableStore[Query, Seq[FrsQueryResult]] {\n\n  override def get(\n    query: Query\n  ): Future[Option[Seq[FrsQueryResult]]] = {\n    if (decider.isAvailable(DeciderConstants.enableFRSTrafficDeciderKey)) {\n      val recommendationRequest =\n        buildFollowRecommendationRequest(query)\n\n      frsClient\n        .getRecommendations(recommendationRequest).map { recommendationResponse =>\n          Some(recommendationResponse.recommendations.collect {\n            case recommendation: Recommendation.User =>\n              FrsQueryResult(\n                recommendation.user.userId,\n                recommendation.user.scoringDetails\n                  .flatMap(_.score).getOrElse(0.0),\n                recommendation.user.scoringDetails\n                  .flatMap(_.candidateSourceDetails.flatMap(_.primarySource)),\n                recommendation.user.scoringDetails\n                  .flatMap(_.candidateSourceDetails.flatMap(_.candidateSourceScores)).map(_.toMap)\n              )\n          })\n        }\n    } else {\n      Future.None\n    }\n  }\n\n  private def buildFollowRecommendationRequest(\n    query: Query\n  ): RecommendationRequest = {\n    RecommendationRequest(\n      clientContext = ClientContext(\n        userId = Some(query.userId),\n        countryCode = query.countryCodeOpt,\n        languageCode = query.languageCodeOpt),\n      displayLocation = query.displayLocation,\n      maxResults = Some(query.maxConsumerSeedsNum),\n      excludedIds = Some(query.excludedUserIds)\n    )\n  }\n}\n\nobject FrsStore {\n  case class Query(\n    userId: UserId,\n    maxConsumerSeedsNum: Int,\n    displayLocation: DisplayLocation = DisplayLocation.ContentRecommender,\n    excludedUserIds: Seq[UserId] = Seq.empty,\n    languageCodeOpt: Option[String] = None,\n    countryCodeOpt: Option[String] = None)\n\n  case class FrsQueryResult(\n    userId: UserId,\n    score: Double,\n    primarySource: Option[Int],\n    sourceWithScores: Option[Map[String, Double]])\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal/RealGraphInSourceGraphFetcher.scala",
    "content": "package com.twitter.cr_mixer.source_signal\n\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.GraphSourceInfo\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.param.RealGraphInParams\nimport com.twitter.cr_mixer.source_signal.SourceFetcher.FetcherQuery\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport com.twitter.wtf.candidate.thriftscala.CandidateSeq\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/**\n * This store fetch user recommendations from In-Network RealGraph (go/realgraph) for a given userId\n */\n@Singleton\ncase class RealGraphInSourceGraphFetcher @Inject() (\n  @Named(ModuleNames.RealGraphInStore) realGraphStoreMh: ReadableStore[UserId, CandidateSeq],\n  override val timeoutConfig: TimeoutConfig,\n  globalStats: StatsReceiver)\n    extends SourceGraphFetcher {\n\n  override protected val stats: StatsReceiver = globalStats.scope(identifier)\n  override protected val graphSourceType: SourceType = SourceType.RealGraphIn\n\n  override def isEnabled(query: FetcherQuery): Boolean = {\n    query.params(RealGraphInParams.EnableSourceGraphParam)\n  }\n\n  override def fetchAndProcess(\n    query: FetcherQuery,\n  ): Future[Option[GraphSourceInfo]] = {\n    val rawSignals = trackPerItemStats(query)(\n      realGraphStoreMh.get(query.userId).map {\n        _.map { candidateSeq =>\n          candidateSeq.candidates\n            .map { candidate =>\n              // Bundle the userId with its score\n              (candidate.userId, candidate.score)\n            }\n        }\n      }\n    )\n    rawSignals.map {\n      _.map { userWithScores =>\n        convertGraphSourceInfo(userWithScores)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal/RealGraphOonSourceGraphFetcher.scala",
    "content": "package com.twitter.cr_mixer.source_signal\n\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.GraphSourceInfo\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.param.RealGraphOonParams\nimport com.twitter.cr_mixer.source_signal.SourceFetcher.FetcherQuery\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport com.twitter.wtf.candidate.thriftscala.CandidateSeq\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/**\n * This store fetch user recommendations from RealGraphOON (go/realgraph) for a given userId\n */\n@Singleton\ncase class RealGraphOonSourceGraphFetcher @Inject() (\n  @Named(ModuleNames.RealGraphOonStore) realGraphOonStore: ReadableStore[UserId, CandidateSeq],\n  override val timeoutConfig: TimeoutConfig,\n  globalStats: StatsReceiver)\n    extends SourceGraphFetcher {\n\n  override protected val stats: StatsReceiver = globalStats.scope(identifier)\n  override protected val graphSourceType: SourceType = SourceType.RealGraphOon\n\n  override def isEnabled(query: FetcherQuery): Boolean = {\n    query.params(RealGraphOonParams.EnableSourceGraphParam)\n  }\n\n  override def fetchAndProcess(\n    query: FetcherQuery,\n  ): Future[Option[GraphSourceInfo]] = {\n    val rawSignals = trackPerItemStats(query)(\n      realGraphOonStore.get(query.userId).map {\n        _.map { candidateSeq =>\n          candidateSeq.candidates\n            .map { candidate =>\n              // Bundle the userId with its score\n              (candidate.userId, candidate.score)\n            }.take(query.params(RealGraphOonParams.MaxConsumerSeedsNumParam))\n        }\n      }\n    )\n    rawSignals.map {\n      _.map { userWithScores =>\n        convertGraphSourceInfo(userWithScores)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal/SourceFetcher.scala",
    "content": "package com.twitter.cr_mixer.source_signal\n\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.source_signal.SourceFetcher.FetcherQuery\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.cr_mixer.thriftscala.{Product => TProduct}\nimport com.twitter.finagle.GlobalRequestTimeoutException\nimport com.twitter.finagle.mux.ClientDiscardedRequestException\nimport com.twitter.finagle.mux.ServerApplicationError\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport com.twitter.util.TimeoutException\nimport org.apache.thrift.TApplicationException\nimport com.twitter.util.logging.Logging\n\n/**\n * A SourceFetcher is a trait which, given a [[FetcherQuery]], returns [[ResultType]]\n * The main purposes of a SourceFetcher is to provide a consistent interface for source fetch\n * logic, and provides default functions, including:\n * - Identification\n * - Observability\n * - Timeout settings\n * - Exception Handling\n */\ntrait SourceFetcher[ResultType] extends ReadableStore[FetcherQuery, ResultType] with Logging {\n\n  protected final val timer = com.twitter.finagle.util.DefaultTimer\n  protected final def identifier: String = this.getClass.getSimpleName\n  protected def stats: StatsReceiver\n  protected def timeoutConfig: TimeoutConfig\n\n  /***\n   * Use FeatureSwitch to decide if a specific source is enabled.\n   */\n  def isEnabled(query: FetcherQuery): Boolean\n\n  /***\n   * This function fetches the raw sources and process them.\n   * Custom stats tracking can be added depending on the type of ResultType\n   */\n  def fetchAndProcess(\n    query: FetcherQuery,\n  ): Future[Option[ResultType]]\n\n  /***\n   * Side-effect function to track stats for signal fetching and processing.\n   */\n  def trackStats(\n    query: FetcherQuery\n  )(\n    func: => Future[Option[ResultType]]\n  ): Future[Option[ResultType]]\n\n  /***\n   * This function is called by the top level class to fetch sources. It executes the pipeline to\n   * fetch raw data, process and transform the sources. Exceptions, Stats, and timeout control are\n   * handled here.\n   */\n  override def get(\n    query: FetcherQuery\n  ): Future[Option[ResultType]] = {\n    val scopedStats = stats.scope(query.product.originalName)\n    if (isEnabled(query)) {\n      scopedStats.counter(\"gate_enabled\").incr()\n      trackStats(query)(fetchAndProcess(query))\n        .raiseWithin(timeoutConfig.signalFetchTimeout)(timer)\n        .onFailure { e =>\n          scopedStats.scope(\"exceptions\").counter(e.getClass.getSimpleName).incr()\n        }\n        .rescue {\n          case _: TimeoutException | _: GlobalRequestTimeoutException | _: TApplicationException |\n              _: ClientDiscardedRequestException |\n              _: ServerApplicationError // TApplicationException inside\n              =>\n            Future.None\n          case e =>\n            logger.info(e)\n            Future.None\n        }\n    } else {\n      scopedStats.counter(\"gate_disabled\").incr()\n      Future.None\n    }\n  }\n}\n\nobject SourceFetcher {\n\n  /***\n   * Every SourceFetcher all share the same input: FetcherQuery\n   */\n  case class FetcherQuery(\n    userId: UserId,\n    product: TProduct,\n    userState: UserState,\n    params: Params)\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal/SourceGraphFetcher.scala",
    "content": "package com.twitter.cr_mixer.source_signal\n\nimport com.twitter.cr_mixer.model.GraphSourceInfo\nimport com.twitter.cr_mixer.source_signal.SourceFetcher.FetcherQuery\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.util.Future\n\n/***\n * A SourceGraphFetcher is a trait that extends from `SourceFetcher`\n * and is specialized in tackling User Graph (eg., RealGraphOon, FRS) fetch.\n *\n * The [[ResultType]] of a SourceGraphFetcher is a `GraphSourceInfo` which contains a userSeedSet.\n * When we pass in userId, the underlying store returns one GraphSourceInfo.\n */\ntrait SourceGraphFetcher extends SourceFetcher[GraphSourceInfo] {\n  protected final val DefaultSeedScore = 1.0\n  protected def graphSourceType: SourceType\n\n  /***\n   * RawDataType contains a consumers seed UserId and a score (weight)\n   */\n  protected type RawDataType = (UserId, Double)\n\n  def trackStats(\n    query: FetcherQuery\n  )(\n    func: => Future[Option[GraphSourceInfo]]\n  ): Future[Option[GraphSourceInfo]] = {\n    val productScopedStats = stats.scope(query.product.originalName)\n    val productUserStateScopedStats = productScopedStats.scope(query.userState.toString)\n    StatsUtil\n      .trackOptionStats(productScopedStats) {\n        StatsUtil\n          .trackOptionStats(productUserStateScopedStats) {\n            func\n          }\n      }\n  }\n\n  // Track per item stats on the fetched graph results\n  def trackPerItemStats(\n    query: FetcherQuery\n  )(\n    func: => Future[Option[Seq[RawDataType]]]\n  ): Future[Option[Seq[RawDataType]]] = {\n    val productScopedStats = stats.scope(query.product.originalName)\n    val productUserStateScopedStats = productScopedStats.scope(query.userState.toString)\n    StatsUtil.trackOptionItemsStats(productScopedStats) {\n      StatsUtil.trackOptionItemsStats(productUserStateScopedStats) {\n        func\n      }\n    }\n  }\n\n  /***\n   * Convert Seq[RawDataType] into GraphSourceInfo\n   */\n  protected final def convertGraphSourceInfo(\n    userWithScores: Seq[RawDataType]\n  ): GraphSourceInfo = {\n    GraphSourceInfo(\n      sourceType = graphSourceType,\n      seedWithScores = userWithScores.map { userWithScore =>\n        userWithScore._1 -> userWithScore._2\n      }.toMap\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal/SourceInfoRouter.scala",
    "content": "package com.twitter.cr_mixer.source_signal\n\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.cr_mixer.model.GraphSourceInfo\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.source_signal.SourceFetcher.FetcherQuery\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.cr_mixer.thriftscala.{Product => TProduct}\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.timelines.configapi\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class SourceInfoRouter @Inject() (\n  ussSourceSignalFetcher: UssSourceSignalFetcher,\n  frsSourceSignalFetcher: FrsSourceSignalFetcher,\n  frsSourceGraphFetcher: FrsSourceGraphFetcher,\n  realGraphOonSourceGraphFetcher: RealGraphOonSourceGraphFetcher,\n  realGraphInSourceGraphFetcher: RealGraphInSourceGraphFetcher,\n) {\n\n  def get(\n    userId: UserId,\n    product: TProduct,\n    userState: UserState,\n    params: configapi.Params\n  ): Future[(Set[SourceInfo], Map[String, Option[GraphSourceInfo]])] = {\n\n    val fetcherQuery = FetcherQuery(userId, product, userState, params)\n    Future.join(\n      getSourceSignals(fetcherQuery),\n      getSourceGraphs(fetcherQuery)\n    )\n  }\n\n  private def getSourceSignals(\n    fetcherQuery: FetcherQuery\n  ): Future[Set[SourceInfo]] = {\n    Future\n      .join(\n        ussSourceSignalFetcher.get(fetcherQuery),\n        frsSourceSignalFetcher.get(fetcherQuery)).map {\n        case (ussSignalsOpt, frsSignalsOpt) =>\n          (ussSignalsOpt.getOrElse(Seq.empty) ++ frsSignalsOpt.getOrElse(Seq.empty)).toSet\n      }\n  }\n\n  private def getSourceGraphs(\n    fetcherQuery: FetcherQuery\n  ): Future[Map[String, Option[GraphSourceInfo]]] = {\n\n    Future\n      .join(\n        frsSourceGraphFetcher.get(fetcherQuery),\n        realGraphOonSourceGraphFetcher.get(fetcherQuery),\n        realGraphInSourceGraphFetcher.get(fetcherQuery)\n      ).map {\n        case (frsGraphOpt, realGraphOonGraphOpt, realGraphInGraphOpt) =>\n          Map(\n            SourceType.FollowRecommendation.name -> frsGraphOpt,\n            SourceType.RealGraphOon.name -> realGraphOonGraphOpt,\n            SourceType.RealGraphIn.name -> realGraphInGraphOpt,\n          )\n      }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal/SourceSignalFetcher.scala",
    "content": "package com.twitter.cr_mixer.source_signal\n\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.source_signal.SourceFetcher.FetcherQuery\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.util.Future\n\n/***\n * A SourceSignalFetcher is a trait that extends from `SourceFetcher`\n * and is specialized in tackling Signals (eg., USS, FRS) fetch.\n * Currently, we define Signals as (but not limited to) a set of past engagements that\n * the user makes, such as RecentFav, RecentFollow, etc.\n *\n * The [[ResultType]] of a SourceSignalFetcher is `Seq[SourceInfo]`. When we pass in userId,\n * the underlying store returns a list of signals.\n */\ntrait SourceSignalFetcher extends SourceFetcher[Seq[SourceInfo]] {\n\n  protected type SignalConvertType\n\n  def trackStats(\n    query: FetcherQuery\n  )(\n    func: => Future[Option[Seq[SourceInfo]]]\n  ): Future[Option[Seq[SourceInfo]]] = {\n    val productScopedStats = stats.scope(query.product.originalName)\n    val productUserStateScopedStats = productScopedStats.scope(query.userState.toString)\n    StatsUtil\n      .trackOptionItemsStats(productScopedStats) {\n        StatsUtil\n          .trackOptionItemsStats(productUserStateScopedStats) {\n            func\n          }\n      }\n  }\n\n  /***\n   * Convert a list of Signals of type [[SignalConvertType]] into SourceInfo\n   */\n  def convertSourceInfo(\n    sourceType: SourceType,\n    signals: Seq[SignalConvertType]\n  ): Seq[SourceInfo]\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal/UssSourceSignalFetcher.scala",
    "content": "package com.twitter.cr_mixer.source_signal\n\nimport com.twitter.cr_mixer.config.TimeoutConfig\nimport com.twitter.cr_mixer.model.ModuleNames\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.cr_mixer.source_signal.SourceFetcher.FetcherQuery\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.usersignalservice.thriftscala.{Signal => UssSignal}\nimport com.twitter.usersignalservice.thriftscala.SignalType\nimport com.twitter.frigate.common.util.StatsUtil.Size\nimport com.twitter.frigate.common.util.StatsUtil.Success\nimport com.twitter.frigate.common.util.StatsUtil.Empty\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport javax.inject.Singleton\nimport javax.inject.Inject\nimport javax.inject.Named\n\n@Singleton\ncase class UssSourceSignalFetcher @Inject() (\n  @Named(ModuleNames.UssStore) ussStore: ReadableStore[UssStore.Query, Seq[\n    (SignalType, Seq[UssSignal])\n  ]],\n  override val timeoutConfig: TimeoutConfig,\n  globalStats: StatsReceiver)\n    extends SourceSignalFetcher {\n\n  override protected val stats: StatsReceiver = globalStats.scope(identifier)\n  override type SignalConvertType = UssSignal\n\n  // always enable USS call. We have fine-grained FS to decider which signal to fetch\n  override def isEnabled(query: FetcherQuery): Boolean = true\n\n  override def fetchAndProcess(\n    query: FetcherQuery,\n  ): Future[Option[Seq[SourceInfo]]] = {\n    // Fetch raw signals\n    val rawSignals = ussStore.get(UssStore.Query(query.userId, query.params, query.product)).map {\n      _.map {\n        _.map {\n          case (signalType, signals) =>\n            trackUssSignalStatsPerSignalType(query, signalType, signals)\n            (signalType, signals)\n        }\n      }\n    }\n\n    /**\n     * Process signals:\n     * Transform a Seq of USS Signals with signalType specified to a Seq of SourceInfo\n     * We do case match to make sure the SignalType can correctly map to a SourceType defined in CrMixer\n     * and it should be simplified.\n     */\n    rawSignals.map {\n      _.map { nestedSignal =>\n        val sourceInfoList = nestedSignal.flatMap {\n          case (signalType, ussSignals) =>\n            signalType match {\n              case SignalType.TweetFavorite =>\n                convertSourceInfo(sourceType = SourceType.TweetFavorite, signals = ussSignals)\n              case SignalType.Retweet =>\n                convertSourceInfo(sourceType = SourceType.Retweet, signals = ussSignals)\n              case SignalType.Reply =>\n                convertSourceInfo(sourceType = SourceType.Reply, signals = ussSignals)\n              case SignalType.OriginalTweet =>\n                convertSourceInfo(sourceType = SourceType.OriginalTweet, signals = ussSignals)\n              case SignalType.AccountFollow =>\n                convertSourceInfo(sourceType = SourceType.UserFollow, signals = ussSignals)\n              case SignalType.RepeatedProfileVisit180dMinVisit6V1 |\n                  SignalType.RepeatedProfileVisit90dMinVisit6V1 |\n                  SignalType.RepeatedProfileVisit14dMinVisit2V1 =>\n                convertSourceInfo(\n                  sourceType = SourceType.UserRepeatedProfileVisit,\n                  signals = ussSignals)\n              case SignalType.NotificationOpenAndClickV1 =>\n                convertSourceInfo(sourceType = SourceType.NotificationClick, signals = ussSignals)\n              case SignalType.TweetShareV1 =>\n                convertSourceInfo(sourceType = SourceType.TweetShare, signals = ussSignals)\n              case SignalType.RealGraphOon =>\n                convertSourceInfo(sourceType = SourceType.RealGraphOon, signals = ussSignals)\n              case SignalType.GoodTweetClick | SignalType.GoodTweetClick5s |\n                  SignalType.GoodTweetClick10s | SignalType.GoodTweetClick30s =>\n                convertSourceInfo(sourceType = SourceType.GoodTweetClick, signals = ussSignals)\n              case SignalType.VideoView90dPlayback50V1 =>\n                convertSourceInfo(\n                  sourceType = SourceType.VideoTweetPlayback50,\n                  signals = ussSignals)\n              case SignalType.VideoView90dQualityV1 =>\n                convertSourceInfo(\n                  sourceType = SourceType.VideoTweetQualityView,\n                  signals = ussSignals)\n              case SignalType.GoodProfileClick | SignalType.GoodProfileClick20s |\n                  SignalType.GoodProfileClick30s =>\n                convertSourceInfo(sourceType = SourceType.GoodProfileClick, signals = ussSignals)\n              // negative signals\n              case SignalType.AccountBlock =>\n                convertSourceInfo(sourceType = SourceType.AccountBlock, signals = ussSignals)\n              case SignalType.AccountMute =>\n                convertSourceInfo(sourceType = SourceType.AccountMute, signals = ussSignals)\n              case SignalType.TweetReport =>\n                convertSourceInfo(sourceType = SourceType.TweetReport, signals = ussSignals)\n              case SignalType.TweetDontLike =>\n                convertSourceInfo(sourceType = SourceType.TweetDontLike, signals = ussSignals)\n              // Aggregated Signals\n              case SignalType.TweetBasedUnifiedEngagementWeightedSignal |\n                  SignalType.TweetBasedUnifiedUniformSignal =>\n                convertSourceInfo(sourceType = SourceType.TweetAggregation, signals = ussSignals)\n              case SignalType.ProducerBasedUnifiedEngagementWeightedSignal |\n                  SignalType.ProducerBasedUnifiedUniformSignal =>\n                convertSourceInfo(sourceType = SourceType.ProducerAggregation, signals = ussSignals)\n\n              // Default\n              case _ =>\n                Seq.empty[SourceInfo]\n            }\n        }\n        sourceInfoList\n      }\n    }\n  }\n\n  override def convertSourceInfo(\n    sourceType: SourceType,\n    signals: Seq[SignalConvertType]\n  ): Seq[SourceInfo] = {\n    signals.map { signal =>\n      SourceInfo(\n        sourceType = sourceType,\n        internalId = signal.targetInternalId.getOrElse(\n          throw new IllegalArgumentException(\n            s\"${sourceType.toString} Signal does not have internalId\")),\n        sourceEventTime =\n          if (signal.timestamp == 0L) None else Some(Time.fromMilliseconds(signal.timestamp))\n      )\n    }\n  }\n\n  private def trackUssSignalStatsPerSignalType(\n    query: FetcherQuery,\n    signalType: SignalType,\n    ussSignals: Seq[UssSignal]\n  ): Unit = {\n    val productScopedStats = stats.scope(query.product.originalName)\n    val productUserStateScopedStats = productScopedStats.scope(query.userState.toString)\n    val productStats = productScopedStats.scope(signalType.toString)\n    val productUserStateStats = productUserStateScopedStats.scope(signalType.toString)\n\n    productStats.counter(Success).incr()\n    productUserStateStats.counter(Success).incr()\n    val size = ussSignals.size\n    productStats.stat(Size).add(size)\n    productUserStateStats.stat(Size).add(size)\n    if (size == 0) {\n      productStats.counter(Empty).incr()\n      productUserStateStats.counter(Empty).incr()\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/source_signal/UssStore.scala",
    "content": "package com.twitter.cr_mixer.source_signal\n\nimport com.twitter.cr_mixer.param.GlobalParams\nimport com.twitter.cr_mixer.param.GoodProfileClickParams\nimport com.twitter.cr_mixer.param.GoodTweetClickParams\nimport com.twitter.cr_mixer.param.RealGraphOonParams\nimport com.twitter.cr_mixer.param.RecentFollowsParams\nimport com.twitter.cr_mixer.param.RecentNegativeSignalParams\nimport com.twitter.cr_mixer.param.RecentNotificationsParams\nimport com.twitter.cr_mixer.param.RecentOriginalTweetsParams\nimport com.twitter.cr_mixer.param.RecentReplyTweetsParams\nimport com.twitter.cr_mixer.param.RecentRetweetsParams\nimport com.twitter.cr_mixer.param.RecentTweetFavoritesParams\nimport com.twitter.cr_mixer.param.RepeatedProfileVisitsParams\nimport com.twitter.cr_mixer.param.TweetSharesParams\nimport com.twitter.cr_mixer.param.UnifiedUSSSignalParams\nimport com.twitter.cr_mixer.param.VideoViewTweetsParams\nimport com.twitter.cr_mixer.source_signal.UssStore.Query\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.usersignalservice.thriftscala.{Signal => UssSignal}\nimport com.twitter.usersignalservice.thriftscala.SignalType\nimport javax.inject.Singleton\nimport com.twitter.timelines.configapi\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.usersignalservice.thriftscala.BatchSignalRequest\nimport com.twitter.usersignalservice.thriftscala.BatchSignalResponse\nimport com.twitter.usersignalservice.thriftscala.SignalRequest\nimport com.twitter.util.Future\nimport com.twitter.cr_mixer.thriftscala.Product\nimport com.twitter.usersignalservice.thriftscala.ClientIdentifier\n\n@Singleton\ncase class UssStore(\n  stratoStore: ReadableStore[BatchSignalRequest, BatchSignalResponse],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[Query, Seq[(SignalType, Seq[UssSignal])]] {\n\n  import com.twitter.cr_mixer.source_signal.UssStore._\n\n  override def get(query: Query): Future[Option[Seq[(SignalType, Seq[UssSignal])]]] = {\n    val ussClientIdentifier = query.product match {\n      case Product.Home =>\n        ClientIdentifier.CrMixerHome\n      case Product.Notifications =>\n        ClientIdentifier.CrMixerNotifications\n      case Product.Email =>\n        ClientIdentifier.CrMixerEmail\n      case _ =>\n        ClientIdentifier.Unknown\n    }\n    val batchSignalRequest =\n      BatchSignalRequest(\n        query.userId,\n        buildUserSignalServiceRequests(query.params),\n        Some(ussClientIdentifier))\n\n    stratoStore\n      .get(batchSignalRequest)\n      .map {\n        _.map { batchSignalResponse =>\n          batchSignalResponse.signalResponse.toSeq.map {\n            case (signalType, ussSignals) =>\n              (signalType, ussSignals)\n          }\n        }\n      }\n  }\n\n  private def buildUserSignalServiceRequests(\n    param: Params,\n  ): Seq[SignalRequest] = {\n    val unifiedMaxSourceKeyNum = param(GlobalParams.UnifiedMaxSourceKeyNum)\n    val goodTweetClickMaxSignalNum = param(GoodTweetClickParams.MaxSignalNumParam)\n    val aggrTweetMaxSourceKeyNum = param(UnifiedUSSSignalParams.UnifiedTweetSourceNumberParam)\n    val aggrProducerMaxSourceKeyNum = param(UnifiedUSSSignalParams.UnifiedProducerSourceNumberParam)\n\n    val maybeRecentTweetFavorite =\n      if (param(RecentTweetFavoritesParams.EnableSourceParam))\n        Some(SignalRequest(Some(unifiedMaxSourceKeyNum), SignalType.TweetFavorite))\n      else None\n    val maybeRecentRetweet =\n      if (param(RecentRetweetsParams.EnableSourceParam))\n        Some(SignalRequest(Some(unifiedMaxSourceKeyNum), SignalType.Retweet))\n      else None\n    val maybeRecentReply =\n      if (param(RecentReplyTweetsParams.EnableSourceParam))\n        Some(SignalRequest(Some(unifiedMaxSourceKeyNum), SignalType.Reply))\n      else None\n    val maybeRecentOriginalTweet =\n      if (param(RecentOriginalTweetsParams.EnableSourceParam))\n        Some(SignalRequest(Some(unifiedMaxSourceKeyNum), SignalType.OriginalTweet))\n      else None\n    val maybeRecentFollow =\n      if (param(RecentFollowsParams.EnableSourceParam))\n        Some(SignalRequest(Some(unifiedMaxSourceKeyNum), SignalType.AccountFollow))\n      else None\n    val maybeRepeatedProfileVisits =\n      if (param(RepeatedProfileVisitsParams.EnableSourceParam))\n        Some(\n          SignalRequest(\n            Some(unifiedMaxSourceKeyNum),\n            param(RepeatedProfileVisitsParams.ProfileMinVisitType).signalType))\n      else None\n    val maybeRecentNotifications =\n      if (param(RecentNotificationsParams.EnableSourceParam))\n        Some(SignalRequest(Some(unifiedMaxSourceKeyNum), SignalType.NotificationOpenAndClickV1))\n      else None\n    val maybeTweetShares =\n      if (param(TweetSharesParams.EnableSourceParam)) {\n        Some(SignalRequest(Some(unifiedMaxSourceKeyNum), SignalType.TweetShareV1))\n      } else None\n    val maybeRealGraphOon =\n      if (param(RealGraphOonParams.EnableSourceParam)) {\n        Some(SignalRequest(Some(unifiedMaxSourceKeyNum), SignalType.RealGraphOon))\n      } else None\n\n    val maybeGoodTweetClick =\n      if (param(GoodTweetClickParams.EnableSourceParam))\n        Some(\n          SignalRequest(\n            Some(goodTweetClickMaxSignalNum),\n            param(GoodTweetClickParams.ClickMinDwellTimeType).signalType))\n      else None\n    val maybeVideoViewTweets =\n      if (param(VideoViewTweetsParams.EnableSourceParam)) {\n        Some(\n          SignalRequest(\n            Some(unifiedMaxSourceKeyNum),\n            param(VideoViewTweetsParams.VideoViewTweetTypeParam).signalType))\n      } else None\n    val maybeGoodProfileClick =\n      if (param(GoodProfileClickParams.EnableSourceParam))\n        Some(\n          SignalRequest(\n            Some(unifiedMaxSourceKeyNum),\n            param(GoodProfileClickParams.ClickMinDwellTimeType).signalType))\n      else None\n    val maybeAggTweetSignal =\n      if (param(UnifiedUSSSignalParams.EnableTweetAggSourceParam))\n        Some(\n          SignalRequest(\n            Some(aggrTweetMaxSourceKeyNum),\n            param(UnifiedUSSSignalParams.TweetAggTypeParam).signalType\n          )\n        )\n      else None\n    val maybeAggProducerSignal =\n      if (param(UnifiedUSSSignalParams.EnableProducerAggSourceParam))\n        Some(\n          SignalRequest(\n            Some(aggrProducerMaxSourceKeyNum),\n            param(UnifiedUSSSignalParams.ProducerAggTypeParam).signalType\n          )\n        )\n      else None\n\n    // negative signals\n    val maybeNegativeSignals = if (param(RecentNegativeSignalParams.EnableSourceParam)) {\n      EnabledNegativeSignalTypes\n        .map(negativeSignal => SignalRequest(Some(unifiedMaxSourceKeyNum), negativeSignal)).toSeq\n    } else Seq.empty\n\n    val allPositiveSignals =\n      if (param(UnifiedUSSSignalParams.ReplaceIndividualUSSSourcesParam))\n        Seq(\n          maybeRecentOriginalTweet,\n          maybeRecentNotifications,\n          maybeRealGraphOon,\n          maybeGoodTweetClick,\n          maybeGoodProfileClick,\n          maybeAggProducerSignal,\n          maybeAggTweetSignal,\n        )\n      else\n        Seq(\n          maybeRecentTweetFavorite,\n          maybeRecentRetweet,\n          maybeRecentReply,\n          maybeRecentOriginalTweet,\n          maybeRecentFollow,\n          maybeRepeatedProfileVisits,\n          maybeRecentNotifications,\n          maybeTweetShares,\n          maybeRealGraphOon,\n          maybeGoodTweetClick,\n          maybeVideoViewTweets,\n          maybeGoodProfileClick,\n          maybeAggProducerSignal,\n          maybeAggTweetSignal,\n        )\n    allPositiveSignals.flatten ++ maybeNegativeSignals\n  }\n\n}\n\nobject UssStore {\n  case class Query(\n    userId: UserId,\n    params: configapi.Params,\n    product: Product)\n\n  val EnabledNegativeSourceTypes: Set[SourceType] =\n    Set(SourceType.AccountBlock, SourceType.AccountMute)\n  private val EnabledNegativeSignalTypes: Set[SignalType] =\n    Set(SignalType.AccountBlock, SignalType.AccountMute)\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/org/lz4:lz4-java\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"configapi/configapi-core\",\n        \"content-recommender/thrift/src/main/thrift:thrift-scala\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/config\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/model\",\n        \"cr-mixer/server/src/main/scala/com/twitter/cr_mixer/param\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/util:stats_util\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/stats\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/java/com/twitter/search/queryparser/query/search:search-query-nodes\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/core_workflows/user_model:user_model-scala\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"src/thrift/com/twitter/search/common:ranking-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util/CandidateGenerationKeyUtil.scala",
    "content": "package com.twitter.cr_mixer.util\n\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.thriftscala.CandidateGenerationKey\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngine\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.simclusters_v2.common.UserId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.util.Time\n\nobject CandidateGenerationKeyUtil {\n  private val PlaceholderUserId = 0L // this default value will not be used\n\n  private val DefaultSourceInfo: SourceInfo = SourceInfo(\n    sourceType = SourceType.RequestUserId,\n    sourceEventTime = None,\n    internalId = InternalId.UserId(PlaceholderUserId)\n  )\n\n  def toThrift(\n    candidateGenerationInfo: CandidateGenerationInfo,\n    requestUserId: UserId\n  ): CandidateGenerationKey = {\n    CandidateGenerationKey(\n      sourceType = candidateGenerationInfo.sourceInfoOpt.getOrElse(DefaultSourceInfo).sourceType,\n      sourceEventTime = candidateGenerationInfo.sourceInfoOpt\n        .getOrElse(DefaultSourceInfo).sourceEventTime.getOrElse(Time.fromMilliseconds(0L)).inMillis,\n      id = candidateGenerationInfo.sourceInfoOpt\n        .map(_.internalId).getOrElse(InternalId.UserId(requestUserId)),\n      modelId = candidateGenerationInfo.similarityEngineInfo.modelId.getOrElse(\"\"),\n      similarityEngineType =\n        Some(candidateGenerationInfo.similarityEngineInfo.similarityEngineType),\n      contributingSimilarityEngine =\n        Some(candidateGenerationInfo.contributingSimilarityEngines.map(se =>\n          SimilarityEngine(se.similarityEngineType, se.modelId, se.score)))\n    )\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util/CountWeightedInterleaveUtil.scala",
    "content": "package com.twitter.cr_mixer.util\n\nimport com.twitter.cr_mixer.model.Candidate\nimport com.twitter.cr_mixer.model.InitialCandidate\nimport com.twitter.cr_mixer.model.RankedCandidate\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.param.BlenderParams.BlendGroupingMethodEnum\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.simclusters_v2.thriftscala.InternalId\n\nobject CountWeightedInterleaveUtil {\n\n  /**\n   * Grouping key for interleaving candidates\n   *\n   * @param sourceInfoOpt optional SourceInfo, containing the source information\n   * @param similarityEngineTypeOpt optional SimilarityEngineType, containing similarity engine\n   *                                information\n   * @param modelIdOpt optional modelId, containing the model ID\n   * @param authorIdOpt optional authorId, containing the tweet author ID\n   * @param groupIdOpt optional groupId, containing the ID corresponding to the blending group\n   */\n  case class GroupingKey(\n    sourceInfoOpt: Option[SourceInfo],\n    similarityEngineTypeOpt: Option[SimilarityEngineType],\n    modelIdOpt: Option[String],\n    authorIdOpt: Option[Long],\n    groupIdOpt: Option[Int])\n\n  /**\n   * Converts candidates to grouping key based upon the feature that we interleave with.\n   */\n  def toGroupingKey[CandidateType <: Candidate](\n    candidate: CandidateType,\n    interleaveFeature: Option[BlendGroupingMethodEnum.Value],\n    groupId: Option[Int],\n  ): GroupingKey = {\n    val grouping: GroupingKey = candidate match {\n      case c: RankedCandidate =>\n        interleaveFeature.getOrElse(BlendGroupingMethodEnum.SourceKeyDefault) match {\n          case BlendGroupingMethodEnum.SourceKeyDefault =>\n            GroupingKey(\n              sourceInfoOpt = c.reasonChosen.sourceInfoOpt,\n              similarityEngineTypeOpt =\n                Some(c.reasonChosen.similarityEngineInfo.similarityEngineType),\n              modelIdOpt = c.reasonChosen.similarityEngineInfo.modelId,\n              authorIdOpt = None,\n              groupIdOpt = groupId\n            )\n          // Some candidate sources don't have a sourceType, so it defaults to similarityEngine\n          case BlendGroupingMethodEnum.SourceTypeSimilarityEngine =>\n            val sourceInfoOpt = c.reasonChosen.sourceInfoOpt.map(_.sourceType).map { sourceType =>\n              SourceInfo(\n                sourceType = sourceType,\n                internalId = InternalId.UserId(0),\n                sourceEventTime = None)\n            }\n            GroupingKey(\n              sourceInfoOpt = sourceInfoOpt,\n              similarityEngineTypeOpt =\n                Some(c.reasonChosen.similarityEngineInfo.similarityEngineType),\n              modelIdOpt = c.reasonChosen.similarityEngineInfo.modelId,\n              authorIdOpt = None,\n              groupIdOpt = groupId\n            )\n          case BlendGroupingMethodEnum.AuthorId =>\n            GroupingKey(\n              sourceInfoOpt = None,\n              similarityEngineTypeOpt = None,\n              modelIdOpt = None,\n              authorIdOpt = Some(c.tweetInfo.authorId),\n              groupIdOpt = groupId\n            )\n          case _ =>\n            throw new UnsupportedOperationException(\n              s\"Unsupported interleave feature: $interleaveFeature\")\n        }\n      case _ =>\n        GroupingKey(\n          sourceInfoOpt = None,\n          similarityEngineTypeOpt = None,\n          modelIdOpt = None,\n          authorIdOpt = None,\n          groupIdOpt = groupId\n        )\n    }\n    grouping\n  }\n\n  /**\n   * Rather than manually calculating and maintaining the weights to rank with, we instead\n   * calculate the weights on the fly, based upon the frequencies of the candidates within each\n   * group. To ensure that diversity of the feature is maintained, we additionally employ a\n   * 'shrinkage' parameter which enforces more diversity by moving the weights closer to uniformity.\n   * More details are available at go/weighted-interleave.\n   *\n   * @param candidateSeqKeyByFeature candidate to key.\n   * @param rankerWeightShrinkage value between [0, 1] with 1 being complete uniformity.\n   * @return Interleaving weights keyed by feature.\n   */\n  private def calculateWeightsKeyByFeature[CandidateType <: Candidate](\n    candidateSeqKeyByFeature: Map[GroupingKey, Seq[CandidateType]],\n    rankerWeightShrinkage: Double\n  ): Map[GroupingKey, Double] = {\n    val maxNumberCandidates: Double = candidateSeqKeyByFeature.values\n      .map { candidates =>\n        candidates.size\n      }.max.toDouble\n    candidateSeqKeyByFeature.map {\n      case (featureKey: GroupingKey, candidateSeq: Seq[CandidateType]) =>\n        val observedWeight: Double = candidateSeq.size.toDouble / maxNumberCandidates\n        // How much to shrink empirical estimates to 1 (Default is to make all weights 1).\n        val finalWeight =\n          (1.0 - rankerWeightShrinkage) * observedWeight + rankerWeightShrinkage * 1.0\n        featureKey -> finalWeight\n    }\n  }\n\n  /**\n   * Builds out the groups and weights for weighted interleaving of the candidates.\n   * More details are available at go/weighted-interleave.\n   *\n   * @param rankedCandidateSeq candidates to interleave.\n   * @param rankerWeightShrinkage value between [0, 1] with 1 being complete uniformity.\n   * @return Candidates grouped by feature key and with calculated interleaving weights.\n   */\n  def buildRankedCandidatesWithWeightKeyByFeature(\n    rankedCandidateSeq: Seq[RankedCandidate],\n    rankerWeightShrinkage: Double,\n    interleaveFeature: BlendGroupingMethodEnum.Value\n  ): Seq[(Seq[RankedCandidate], Double)] = {\n    // To accommodate the re-grouping in InterleaveRanker\n    // In InterleaveBlender, we have already abandoned the grouping keys, and use Seq[Seq[]] to do interleave\n    // Since that we build the candidateSeq with groupingKey, we can guarantee there is no empty candidateSeq\n    val candidateSeqKeyByFeature: Map[GroupingKey, Seq[RankedCandidate]] =\n      rankedCandidateSeq.groupBy { candidate: RankedCandidate =>\n        toGroupingKey(candidate, Some(interleaveFeature), None)\n      }\n\n    // These weights [0, 1] are used to do weighted interleaving\n    // The default value of 1.0 ensures the group is always sampled.\n    val candidateWeightsKeyByFeature: Map[GroupingKey, Double] =\n      calculateWeightsKeyByFeature(candidateSeqKeyByFeature, rankerWeightShrinkage)\n\n    candidateSeqKeyByFeature.map {\n      case (groupingKey: GroupingKey, candidateSeq: Seq[RankedCandidate]) =>\n        Tuple2(\n          candidateSeq.sortBy(-_.predictionScore),\n          candidateWeightsKeyByFeature.getOrElse(groupingKey, 1.0))\n    }.toSeq\n  }\n\n  /**\n   * Takes current grouping (as implied by the outer Seq) and computes blending weights.\n   *\n   * @param initialCandidatesSeqSeq grouped candidates to interleave.\n   * @param rankerWeightShrinkage value between [0, 1] with 1 being complete uniformity.\n   * @return Grouped candidates with calculated interleaving weights.\n   */\n  def buildInitialCandidatesWithWeightKeyByFeature(\n    initialCandidatesSeqSeq: Seq[Seq[InitialCandidate]],\n    rankerWeightShrinkage: Double,\n  ): Seq[(Seq[InitialCandidate], Double)] = {\n    val candidateSeqKeyByFeature: Map[GroupingKey, Seq[InitialCandidate]] =\n      initialCandidatesSeqSeq.zipWithIndex.map(_.swap).toMap.map {\n        case (groupId: Int, initialCandidatesSeq: Seq[InitialCandidate]) =>\n          toGroupingKey(initialCandidatesSeq.head, None, Some(groupId)) -> initialCandidatesSeq\n      }\n\n    // These weights [0, 1] are used to do weighted interleaving\n    // The default value of 1.0 ensures the group is always sampled.\n    val candidateWeightsKeyByFeature =\n      calculateWeightsKeyByFeature(candidateSeqKeyByFeature, rankerWeightShrinkage)\n\n    candidateSeqKeyByFeature.map {\n      case (groupingKey: GroupingKey, candidateSeq: Seq[InitialCandidate]) =>\n        Tuple2(candidateSeq, candidateWeightsKeyByFeature.getOrElse(groupingKey, 1.0))\n    }.toSeq\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util/EarlybirdSearchUtil.scala",
    "content": "package com.twitter.cr_mixer.util\n\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant\nimport com.twitter.search.queryparser.query.search.SearchOperator\nimport com.twitter.search.queryparser.query.search.SearchOperatorConstants\nimport com.twitter.search.queryparser.query.{Query => EbQuery}\nimport com.twitter.search.queryparser.query.Conjunction\nimport scala.collection.JavaConverters._\nimport com.twitter.search.earlybird.thriftscala.ThriftSearchResultMetadataOptions\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.search.queryparser.query.Query\nimport com.twitter.util.Duration\nimport com.twitter.search.common.query.thriftjava.thriftscala.CollectorTerminationParams\n\nobject EarlybirdSearchUtil {\n  val EarlybirdClientId: String = \"cr-mixer.prod\"\n\n  val Mentions: String = EarlybirdFieldConstant.MENTIONS_FACET\n  val Hashtags: String = EarlybirdFieldConstant.HASHTAGS_FACET\n  val FacetsToFetch: Seq[String] = Seq(Mentions, Hashtags)\n\n  val MetadataOptions: ThriftSearchResultMetadataOptions = ThriftSearchResultMetadataOptions(\n    getTweetUrls = true,\n    getResultLocation = false,\n    getLuceneScore = false,\n    getInReplyToStatusId = true,\n    getReferencedTweetAuthorId = true,\n    getMediaBits = true,\n    getAllFeatures = true,\n    getFromUserId = true,\n    returnSearchResultFeatures = true,\n    // Set getExclusiveConversationAuthorId in order to retrieve Exclusive / SuperFollow tweets.\n    getExclusiveConversationAuthorId = true\n  )\n\n  // Filter out retweets and replies\n  val TweetTypesToExclude: Seq[String] =\n    Seq(\n      SearchOperatorConstants.NATIVE_RETWEETS,\n      SearchOperatorConstants.REPLIES)\n\n  def GetCollectorTerminationParams(\n    maxNumHitsPerShard: Int,\n    processingTimeout: Duration\n  ): Option[CollectorTerminationParams] = {\n    Some(\n      CollectorTerminationParams(\n        // maxHitsToProcess is used for early termination on each EB shard\n        maxHitsToProcess = Some(maxNumHitsPerShard),\n        timeoutMs = processingTimeout.inMilliseconds.toInt\n      ))\n  }\n\n  /**\n   * Get EarlybirdQuery\n   * This function creates a EBQuery based on the search input\n   */\n  def GetEarlybirdQuery(\n    beforeTweetIdExclusive: Option[TweetId],\n    afterTweetIdExclusive: Option[TweetId],\n    excludedTweetIds: Set[TweetId],\n    filterOutRetweetsAndReplies: Boolean\n  ): Option[EbQuery] =\n    CreateConjunction(\n      Seq(\n        CreateRangeQuery(beforeTweetIdExclusive, afterTweetIdExclusive),\n        CreateExcludedTweetIdsQuery(excludedTweetIds),\n        CreateTweetTypesFilters(filterOutRetweetsAndReplies)\n      ).flatten)\n\n  def CreateRangeQuery(\n    beforeTweetIdExclusive: Option[TweetId],\n    afterTweetIdExclusive: Option[TweetId]\n  ): Option[EbQuery] = {\n    val beforeIdClause = beforeTweetIdExclusive.map { beforeId =>\n      // MAX_ID is an inclusive value therefore we subtract 1 from beforeId.\n      new SearchOperator(SearchOperator.Type.MAX_ID, (beforeId - 1).toString)\n    }\n    val afterIdClause = afterTweetIdExclusive.map { afterId =>\n      new SearchOperator(SearchOperator.Type.SINCE_ID, afterId.toString)\n    }\n    CreateConjunction(Seq(beforeIdClause, afterIdClause).flatten)\n  }\n\n  def CreateTweetTypesFilters(filterOutRetweetsAndReplies: Boolean): Option[EbQuery] = {\n    if (filterOutRetweetsAndReplies) {\n      val tweetTypeFilters = TweetTypesToExclude.map { searchOperator =>\n        new SearchOperator(SearchOperator.Type.EXCLUDE, searchOperator)\n      }\n      CreateConjunction(tweetTypeFilters)\n    } else None\n  }\n\n  def CreateConjunction(clauses: Seq[EbQuery]): Option[EbQuery] = {\n    clauses.size match {\n      case 0 => None\n      case 1 => Some(clauses.head)\n      case _ => Some(new Conjunction(clauses.asJava))\n    }\n  }\n\n  def CreateExcludedTweetIdsQuery(tweetIds: Set[TweetId]): Option[EbQuery] = {\n    if (tweetIds.nonEmpty) {\n      Some(\n        new SearchOperator.Builder()\n          .setType(SearchOperator.Type.NAMED_MULTI_TERM_DISJUNCTION)\n          .addOperand(EarlybirdFieldConstant.ID_FIELD.getFieldName)\n          .addOperand(EXCLUDE_TWEET_IDS)\n          .setOccur(Query.Occur.MUST_NOT)\n          .build())\n    } else None\n  }\n\n  /**\n   * Get NamedDisjunctions with excludedTweetIds\n   */\n  def GetNamedDisjunctions(excludedTweetIds: Set[TweetId]): Option[Map[String, Seq[Long]]] =\n    if (excludedTweetIds.nonEmpty)\n      createNamedDisjunctionsExcludedTweetIds(excludedTweetIds)\n    else None\n\n  val EXCLUDE_TWEET_IDS = \"exclude_tweet_ids\"\n  private def createNamedDisjunctionsExcludedTweetIds(\n    tweetIds: Set[TweetId]\n  ): Option[Map[String, Seq[Long]]] = {\n    if (tweetIds.nonEmpty) {\n      Some(Map(EXCLUDE_TWEET_IDS -> tweetIds.toSeq))\n    } else None\n  }\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util/InterleaveUtil.scala",
    "content": "package com.twitter.cr_mixer.util\n\nimport com.twitter.cr_mixer.model.Candidate\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.RankedCandidate\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.simclusters_v2.common.TweetId\nimport scala.collection.mutable\nimport scala.collection.mutable.ArrayBuffer\n\nobject InterleaveUtil {\n\n  /**\n   * Interleaves candidates by iteratively taking one candidate from the 1st Seq and adding it to the result.\n   * Once we take a candidate from a Seq, we move this Seq to the end of the queue to process,\n   * and remove the candidate from that Seq.\n   *\n   * We keep a mutable.Set[TweetId] buffer to ensure there are no duplicates.\n   *\n   * @param candidates candidates assumed to be sorted by eventTime (latest event comes first)\n   * @return interleaved candidates\n   */\n  def interleave[CandidateType <: Candidate](\n    candidates: Seq[Seq[CandidateType]]\n  ): Seq[CandidateType] = {\n\n    // copy candidates into a mutable map so this method is thread-safe\n    val candidatesPerSequence = candidates.map { tweetCandidates =>\n      mutable.Queue() ++= tweetCandidates\n    }\n\n    val seen = mutable.Set[TweetId]()\n\n    val candidateSeqQueue = mutable.Queue() ++= candidatesPerSequence\n\n    val result = ArrayBuffer[CandidateType]()\n\n    while (candidateSeqQueue.nonEmpty) {\n      val candidatesQueue = candidateSeqQueue.head\n\n      if (candidatesQueue.nonEmpty) {\n        val candidate = candidatesQueue.dequeue()\n        val candidateTweetId = candidate.tweetId\n        val seenCandidate = seen.contains(candidateTweetId)\n        if (!seenCandidate) {\n          result += candidate\n          seen.add(candidate.tweetId)\n          candidateSeqQueue.enqueue(\n            candidateSeqQueue.dequeue()\n          ) // move this Seq to end\n        }\n      } else {\n        candidateSeqQueue.dequeue() //finished processing this Seq\n      }\n    }\n    //convert result to immutable seq\n    result.toList\n  }\n\n  /**\n   * Interleaves candidates by iteratively\n   * 1. Checking weight to see if enough accumulation has occurred to sample from\n   * 2. If yes, taking one candidate from the the Seq and adding it to the result.\n   * 3. Move this Seq to the end of the queue to process (and remove the candidate from that Seq if\n   *    we sampled it from step 2).\n   *\n   * We keep count of the iterations to prevent infinite loops.\n   * We keep a mutable.Set[TweetId] buffer to ensure there are no duplicates.\n   *\n   * @param candidatesAndWeight candidates assumed to be sorted by eventTime (latest event comes first),\n   *                            along with sampling weights to help prioritize important groups.\n   * @param maxWeightAdjustments Maximum number of iterations to account for weighting before\n   *                             defaulting to uniform interleaving.\n   * @return interleaved candidates\n   */\n  def weightedInterleave[CandidateType <: Candidate](\n    candidatesAndWeight: Seq[(Seq[CandidateType], Double)],\n    maxWeightAdjustments: Int = 0\n  ): Seq[CandidateType] = {\n\n    // Set to avoid numerical issues around 1.0\n    val min_weight = 1 - 1e-30\n\n    // copy candidates into a mutable map so this method is thread-safe\n    // adds a counter to use towards sampling\n    val candidatesAndWeightsPerSequence: Seq[\n      (mutable.Queue[CandidateType], InterleaveWeights)\n    ] =\n      candidatesAndWeight.map { candidatesAndWeight =>\n        (mutable.Queue() ++= candidatesAndWeight._1, InterleaveWeights(candidatesAndWeight._2, 0.0))\n      }\n\n    val seen: mutable.Set[TweetId] = mutable.Set[TweetId]()\n\n    val candidateSeqQueue: mutable.Queue[(mutable.Queue[CandidateType], InterleaveWeights)] =\n      mutable.Queue() ++= candidatesAndWeightsPerSequence\n\n    val result: ArrayBuffer[CandidateType] = ArrayBuffer[CandidateType]()\n    var number_iterations: Int = 0\n\n    while (candidateSeqQueue.nonEmpty) {\n      val (candidatesQueue, currentWeights) = candidateSeqQueue.head\n      if (candidatesQueue.nonEmpty) {\n        // Confirm weighting scheme\n        currentWeights.summed_weight += currentWeights.weight\n        number_iterations += 1\n        if (currentWeights.summed_weight >= min_weight || number_iterations >= maxWeightAdjustments) {\n          // If we sample, then adjust the counter\n          currentWeights.summed_weight -= 1.0\n          val candidate = candidatesQueue.dequeue()\n          val candidateTweetId = candidate.tweetId\n          val seenCandidate = seen.contains(candidateTweetId)\n          if (!seenCandidate) {\n            result += candidate\n            seen.add(candidate.tweetId)\n            candidateSeqQueue.enqueue(candidateSeqQueue.dequeue()) // move this Seq to end\n          }\n        } else {\n          candidateSeqQueue.enqueue(candidateSeqQueue.dequeue()) // move this Seq to end\n        }\n      } else {\n        candidateSeqQueue.dequeue() //finished processing this Seq\n      }\n    }\n    //convert result to immutable seq\n    result.toList\n  }\n\n  def buildCandidatesKeyByCGInfo(\n    candidates: Seq[RankedCandidate],\n  ): Seq[Seq[RankedCandidate]] = {\n    // To accommodate the re-grouping in InterleaveRanker\n    // In InterleaveBlender, we have already abandoned the grouping keys, and use Seq[Seq[]] to do interleave\n    // Since that we build the candidateSeq with groupingKey, we can guarantee there is no empty candidateSeq\n    val candidateSeqKeyByCG =\n      candidates.groupBy(candidate => GroupingKey.toGroupingKey(candidate.reasonChosen))\n    candidateSeqKeyByCG.map {\n      case (groupingKey, candidateSeq) =>\n        candidateSeq.sortBy(-_.predictionScore)\n    }.toSeq\n  }\n}\n\ncase class GroupingKey(\n  sourceInfoOpt: Option[SourceInfo],\n  similarityEngineType: SimilarityEngineType,\n  modelId: Option[String]) {}\n\nobject GroupingKey {\n  def toGroupingKey(candidateGenerationInfo: CandidateGenerationInfo): GroupingKey = {\n    GroupingKey(\n      candidateGenerationInfo.sourceInfoOpt,\n      candidateGenerationInfo.similarityEngineInfo.similarityEngineType,\n      candidateGenerationInfo.similarityEngineInfo.modelId\n    )\n  }\n}\n\ncase class InterleaveWeights(weight: Double, var summed_weight: Double)\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util/MetricTagUtil.scala",
    "content": "package com.twitter.cr_mixer.util\n\nimport com.twitter.cr_mixer.model.RankedCandidate\nimport com.twitter.cr_mixer.model.SimilarityEngineInfo\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.thriftscala.MetricTag\nimport com.twitter.cr_mixer.thriftscala.SimilarityEngineType\nimport com.twitter.cr_mixer.thriftscala.SourceType\n\nobject MetricTagUtil {\n\n  def buildMetricTags(candidate: RankedCandidate): Seq[MetricTag] = {\n    val interestedInMetricTag = isFromInterestedIn(candidate)\n\n    val cgInfoMetricTags = candidate.potentialReasons\n      .flatMap { cgInfo =>\n        val sourceMetricTag = cgInfo.sourceInfoOpt.flatMap { sourceInfo =>\n          toMetricTagFromSource(sourceInfo.sourceType)\n        }\n        val similarityEngineTags = toMetricTagFromSimilarityEngine(\n          cgInfo.similarityEngineInfo,\n          cgInfo.contributingSimilarityEngines)\n\n        val combinedMetricTag = cgInfo.sourceInfoOpt.flatMap { sourceInfo =>\n          toMetricTagFromSourceAndSimilarityEngine(sourceInfo, cgInfo.similarityEngineInfo)\n        }\n\n        Seq(sourceMetricTag) ++ similarityEngineTags ++ Seq(combinedMetricTag)\n      }.flatten.toSet\n    (interestedInMetricTag ++ cgInfoMetricTags).toSeq\n  }\n\n  /***\n   * match a sourceType to a metricTag\n   */\n  private def toMetricTagFromSource(sourceType: SourceType): Option[MetricTag] = {\n    sourceType match {\n      case SourceType.TweetFavorite => Some(MetricTag.TweetFavorite) // Personalized Topics in Home\n      case SourceType.Retweet => Some(MetricTag.Retweet) // Personalized Topics in Home\n      case SourceType.NotificationClick =>\n        Some(MetricTag.PushOpenOrNtabClick) // Health Filter in MR\n      case SourceType.OriginalTweet =>\n        Some(MetricTag.OriginalTweet)\n      case SourceType.Reply =>\n        Some(MetricTag.Reply)\n      case SourceType.TweetShare =>\n        Some(MetricTag.TweetShare)\n      case SourceType.UserFollow =>\n        Some(MetricTag.UserFollow)\n      case SourceType.UserRepeatedProfileVisit =>\n        Some(MetricTag.UserRepeatedProfileVisit)\n      case SourceType.TwiceUserId =>\n        Some(MetricTag.TwiceUserId)\n      case _ => None\n    }\n  }\n\n  /***\n   * If the SEInfo is built by a unified sim engine, we un-wrap the contributing sim engines.\n   * If not, we log the sim engine as usual.\n   * @param seInfo (CandidateGenerationInfo.similarityEngineInfo): SimilarityEngineInfo,\n   * @param cseInfo (CandidateGenerationInfo.contributingSimilarityEngines): Seq[SimilarityEngineInfo]\n   */\n  private def toMetricTagFromSimilarityEngine(\n    seInfo: SimilarityEngineInfo,\n    cseInfo: Seq[SimilarityEngineInfo]\n  ): Seq[Option[MetricTag]] = {\n    seInfo.similarityEngineType match {\n      case SimilarityEngineType.TweetBasedUnifiedSimilarityEngine => // un-wrap the unified sim engine\n        cseInfo.map { contributingSimEngine =>\n          toMetricTagFromSimilarityEngine(contributingSimEngine, Seq.empty)\n        }.flatten\n      case SimilarityEngineType.ProducerBasedUnifiedSimilarityEngine => // un-wrap the unified sim engine\n        cseInfo.map { contributingSimEngine =>\n          toMetricTagFromSimilarityEngine(contributingSimEngine, Seq.empty)\n        }.flatten\n      // SimClustersANN can either be called on its own, or be called under unified sim engine\n      case SimilarityEngineType.SimClustersANN => // the old \"UserInterestedIn\" will be replaced by this. Also, OfflineTwice\n        Seq(Some(MetricTag.SimClustersANN), seInfo.modelId.flatMap(toMetricTagFromModelId(_)))\n      case SimilarityEngineType.ConsumerEmbeddingBasedTwHINANN =>\n        Seq(Some(MetricTag.ConsumerEmbeddingBasedTwHINANN))\n      case SimilarityEngineType.TwhinCollabFilter => Seq(Some(MetricTag.TwhinCollabFilter))\n      // In the current implementation, TweetBasedUserTweetGraph/TweetBasedTwHINANN has a tag when\n      // it's either a base SE or a contributing SE. But for now they only show up in contributing SE.\n      case SimilarityEngineType.TweetBasedUserTweetGraph =>\n        Seq(Some(MetricTag.TweetBasedUserTweetGraph))\n      case SimilarityEngineType.TweetBasedTwHINANN =>\n        Seq(Some(MetricTag.TweetBasedTwHINANN))\n      case _ => Seq.empty\n    }\n  }\n\n  /***\n   * pass in a model id, and match it with the metric tag type.\n   */\n  private def toMetricTagFromModelId(\n    modelId: String\n  ): Option[MetricTag] = {\n\n    val pushOpenBasedModelRegex = \"(.*_Model20m145k2020_20220819)\".r\n\n    modelId match {\n      case pushOpenBasedModelRegex(_*) =>\n        Some(MetricTag.RequestHealthFilterPushOpenBasedTweetEmbedding)\n      case _ => None\n    }\n  }\n\n  private def toMetricTagFromSourceAndSimilarityEngine(\n    sourceInfo: SourceInfo,\n    seInfo: SimilarityEngineInfo\n  ): Option[MetricTag] = {\n    sourceInfo.sourceType match {\n      case SourceType.Lookalike\n          if seInfo.similarityEngineType == SimilarityEngineType.ConsumersBasedUserTweetGraph =>\n        Some(MetricTag.LookalikeUTG)\n      case _ => None\n    }\n  }\n\n  /**\n   * Special use case: used by Notifications team to generate the UserInterestedIn CRT push copy.\n   *\n   * if we have different types of InterestedIn (eg. UserInterestedIn, NextInterestedIn),\n   * this if statement will have to be refactored to contain the real UserInterestedIn.\n   * @return\n   */\n  private def isFromInterestedIn(candidate: RankedCandidate): Set[MetricTag] = {\n    if (candidate.reasonChosen.sourceInfoOpt.isEmpty\n      && candidate.reasonChosen.similarityEngineInfo.similarityEngineType == SimilarityEngineType.SimClustersANN) {\n      Set(MetricTag.UserInterestedIn)\n    } else Set.empty\n  }\n\n}\n"
  },
  {
    "path": "cr-mixer/server/src/main/scala/com/twitter/cr_mixer/util/SignalTimestampStatsUtil.scala",
    "content": "package com.twitter.cr_mixer.util\n\nimport com.twitter.cr_mixer.model.CandidateGenerationInfo\nimport com.twitter.cr_mixer.model.RankedCandidate\nimport com.twitter.cr_mixer.model.SourceInfo\nimport com.twitter.cr_mixer.thriftscala.SourceType\nimport com.twitter.cr_mixer.thriftscala.TweetRecommendation\nimport javax.inject.Inject\nimport com.twitter.finagle.stats.StatsReceiver\nimport javax.inject.Singleton\nimport com.twitter.relevance_platform.common.stats.BucketTimestampStats\n\n@Singleton\nclass SignalTimestampStatsUtil @Inject() (statsReceiver: StatsReceiver) {\n  import SignalTimestampStatsUtil._\n\n  private val signalDelayAgePerDayStats =\n    new BucketTimestampStats[TweetRecommendation](\n      BucketTimestampStats.MillisecondsPerDay,\n      _.latestSourceSignalTimestampInMillis.getOrElse(0),\n      Some(SignalTimestampMaxDays))(\n      statsReceiver.scope(\"signal_timestamp_per_day\")\n    ) // only stats past 90 days\n  private val signalDelayAgePerHourStats =\n    new BucketTimestampStats[TweetRecommendation](\n      BucketTimestampStats.MillisecondsPerHour,\n      _.latestSourceSignalTimestampInMillis.getOrElse(0),\n      Some(SignalTimestampMaxHours))(\n      statsReceiver.scope(\"signal_timestamp_per_hour\")\n    ) // only stats past 24 hours\n  private val signalDelayAgePerMinStats =\n    new BucketTimestampStats[TweetRecommendation](\n      BucketTimestampStats.MillisecondsPerMinute,\n      _.latestSourceSignalTimestampInMillis.getOrElse(0),\n      Some(SignalTimestampMaxMins))(\n      statsReceiver.scope(\"signal_timestamp_per_min\")\n    ) // only stats past 60 minutes\n\n  def statsSignalTimestamp(\n    tweets: Seq[TweetRecommendation],\n  ): Seq[TweetRecommendation] = {\n    signalDelayAgePerMinStats.count(tweets)\n    signalDelayAgePerHourStats.count(tweets)\n    signalDelayAgePerDayStats.count(tweets)\n  }\n}\n\nobject SignalTimestampStatsUtil {\n  val SignalTimestampMaxMins = 60 // stats at most 60 mins\n  val SignalTimestampMaxHours = 24 // stats at most 24 hours\n  val SignalTimestampMaxDays = 90 // stats at most 90 days\n\n  def buildLatestSourceSignalTimestamp(candidate: RankedCandidate): Option[Long] = {\n    val timestampSeq = candidate.potentialReasons\n      .collect {\n        case CandidateGenerationInfo(Some(SourceInfo(sourceType, _, Some(sourceEventTime))), _, _)\n            if sourceType == SourceType.TweetFavorite =>\n          sourceEventTime.inMilliseconds\n      }\n    if (timestampSeq.nonEmpty) {\n      Some(timestampSeq.max(Ordering.Long))\n    } else {\n      None\n    }\n  }\n}\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/BUILD",
    "content": "create_thrift_libraries(\n    base_name = \"thrift\",\n    sources = [\"*.thrift\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependency_roots = [\n        \"finatra-internal/thrift/src/main/thrift\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift\",\n        \"src/thrift/com/twitter/ads/schema:common\",\n        \"src/thrift/com/twitter/ml/api:data\",\n        \"src/thrift/com/twitter/recos:recos-common\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift\",\n        \"src/thrift/com/twitter/timelines/render:thrift\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql:api-media-graphql\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql:topics-graphql\",\n    ],\n    generate_languages = [\n        \"java\",\n        \"scala\",\n        \"strato\",\n    ],\n    provides_java_name = \"cr-mixer-thrift-java\",\n    provides_scala_name = \"cr-mixer-thrift-scala\",\n)\n\ncreate_thrift_libraries(\n    base_name = \"cr-mixer-scribe\",\n    sources = [\"*.thrift\"],\n    tags = [\"bazel-compatible\"],\n    dependency_roots = [\n        \"finatra-internal/thrift/src/main/thrift\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift\",\n        \"src/thrift/com/twitter/ads/schema:common\",\n        \"src/thrift/com/twitter/ml/api:data\",\n        \"src/thrift/com/twitter/recos:recos-common\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift\",\n        \"src/thrift/com/twitter/timelines/render:thrift\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql\",\n    ],\n    generate_languages = [\n        \"java\",\n        \"scala\",\n        \"strato\",\n    ],\n    provides_java_name = \"cr-mixer-scribe-java\",\n    provides_scala_name = \"cr-mixer-scribe-scala\",\n)\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/ads.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\ninclude \"product.thrift\"\ninclude \"product_context.thrift\"\n\ninclude \"com/twitter/product_mixer/core/client_context.thrift\"\ninclude \"com/twitter/ads/schema/shared.thrift\"\n\nstruct AdsRequest {\n\t1: required client_context.ClientContext clientContext\n\t2: required product.Product product\n\t# Product-specific parameters should be placed in the Product Context\n\t3: optional product_context.ProductContext productContext\n\t4: optional list<i64> excludedTweetIds (personalDataType = 'TweetId')\n} (persisted='true', hasPersonalData='true')\n\nstruct AdsResponse {\n  1: required list<AdTweetRecommendation> ads\n} (persisted='true')\n\nstruct AdTweetRecommendation {\n  1: required i64 tweetId (personalDataType = 'TweetId')\n  2: required double score\n  3: optional list<LineItemInfo> lineItems\n\n} (persisted='true')\n\nstruct LineItemInfo {\n  1: required i64 lineItemId (personalDataType = 'LineItemId')\n  2: required shared.LineItemObjective lineItemObjective\n} (persisted='true')\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/candidate_generation_key.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\ninclude \"source_type.thrift\"\ninclude \"com/twitter/simclusters_v2/identifier.thrift\"\n\nstruct SimilarityEngine {\n 1: required source_type.SimilarityEngineType similarityEngineType\n 2: optional string modelId\n 3: optional double score\n} (persisted='true')\n\nstruct CandidateGenerationKey {\n  1: required source_type.SourceType sourceType\n  2: required i64 sourceEventTime (personalDataType = 'PrivateTimestamp')\n  3: required identifier.InternalId id\n  4: required string modelId\n  5: optional source_type.SimilarityEngineType similarityEngineType\n  6: optional list<SimilarityEngine> contributingSimilarityEngine\n} (persisted='true')\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/cr_mixer.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\ninclude \"ads.thrift\"\ninclude \"candidate_generation_key.thrift\"\ninclude \"product.thrift\"\ninclude \"product_context.thrift\"\ninclude \"validation.thrift\"\ninclude \"metric_tags.thrift\"\ninclude \"related_tweet.thrift\"\ninclude \"uteg.thrift\"\ninclude \"frs_based_tweet.thrift\"\ninclude \"related_video_tweet.thrift\"\ninclude \"topic_tweet.thrift\"\n\ninclude \"com/twitter/product_mixer/core/client_context.thrift\"\ninclude \"com/twitter/timelines/render/response.thrift\"\ninclude \"finatra-thrift/finatra_thrift_exceptions.thrift\"\ninclude \"com/twitter/strato/graphql/slice.thrift\"\n\nstruct CrMixerTweetRequest {\n\t1: required client_context.ClientContext clientContext\n\t2: required product.Product product\n\t# Product-specific parameters should be placed in the Product Context\n\t3: optional product_context.ProductContext productContext\n\t4: optional list<i64> excludedTweetIds (personalDataType = 'TweetId')\n} (persisted='true', hasPersonalData='true')\n\nstruct TweetRecommendation {\n  1: required i64 tweetId (personalDataType = 'TweetId')\n  2: required double score\n  3: optional list<metric_tags.MetricTag> metricTags\n  # 4: the author of the tweet candidate. To be used by Content-Mixer to unblock the Hydra experiment.\n  4: optional i64 authorId (personalDataType = 'UserId')\n  # 5: extra info about candidate generation. To be used by Content-Mixer to unblock the Hydra experiment.\n  5: optional candidate_generation_key.CandidateGenerationKey candidateGenerationKey\n  # 1001: the latest timestamp of fav signals. If null, the candidate is not generated from fav signals\n  1001: optional i64 latestSourceSignalTimestampInMillis(personalDataType = 'PublicTimestamp')\n} (persisted='true', hasPersonalData = 'true')\n\nstruct CrMixerTweetResponse {\n 1: required list<TweetRecommendation> tweets\n} (persisted='true')\n\nservice CrMixer {\n  CrMixerTweetResponse getTweetRecommendations(1: CrMixerTweetRequest request) throws (\n    # Validation errors - the details of which will be reported to clients on failure\n    1: validation.ValidationExceptionList validationErrors;\n    # Server errors - the details of which will not be reported to clients\n    2: finatra_thrift_exceptions.ServerError serverError\n  )\n\n  # getRelatedTweetsForQueryTweet and getRelatedTweetsForQueryAuthor do very similar things\n  # We can merge these two endpoints into one unified endpoint\n  related_tweet.RelatedTweetResponse getRelatedTweetsForQueryTweet(1: related_tweet.RelatedTweetRequest request) throws (\n    # Validation errors - the details of which will be reported to clients on failure\n    1: validation.ValidationExceptionList validationErrors;\n    # Server errors - the details of which will not be reported to clients\n    2: finatra_thrift_exceptions.ServerError serverError\n  )\n\n  related_tweet.RelatedTweetResponse getRelatedTweetsForQueryAuthor(1: related_tweet.RelatedTweetRequest request) throws (\n    # Validation errors - the details of which will be reported to clients on failure\n    1: validation.ValidationExceptionList validationErrors;\n    # Server errors - the details of which will not be reported to clients\n    2: finatra_thrift_exceptions.ServerError serverError\n  )\n\n  uteg.UtegTweetResponse getUtegTweetRecommendations(1: uteg.UtegTweetRequest request) throws (\n    # Validation errors - the details of which will be reported to clients on failure\n    1: validation.ValidationExceptionList validationErrors;\n    # Server errors - the details of which will not be reported to clients\n    2: finatra_thrift_exceptions.ServerError serverError\n  )\n\n  frs_based_tweet.FrsTweetResponse getFrsBasedTweetRecommendations(1: frs_based_tweet.FrsTweetRequest request) throws (\n     # Validation errors - the details of which will be reported to clients on failure\n     1: validation.ValidationExceptionList validationErrors;\n     # Server errors - the details of which will not be reported to clients\n     2: finatra_thrift_exceptions.ServerError serverError\n  )\n\n  related_video_tweet.RelatedVideoTweetResponse getRelatedVideoTweetsForQueryTweet(1: related_video_tweet.RelatedVideoTweetRequest request) throws (\n      # Validation errors - the details of which will be reported to clients on failure\n      1: validation.ValidationExceptionList validationErrors;\n      # Server errors - the details of which will not be reported to clients\n      2: finatra_thrift_exceptions.ServerError serverError\n  )\n\n  ads.AdsResponse getAdsRecommendations(1: ads.AdsRequest request) throws (\n    # Validation errors - the details of which will be reported to clients on failure\n    1: validation.ValidationExceptionList validationErrors;\n    # Server errors - the details of which will not be reported to clients\n    2: finatra_thrift_exceptions.ServerError serverError\n  )\n\n  topic_tweet.TopicTweetResponse getTopicTweetRecommendations(1: topic_tweet.TopicTweetRequest request) throws (\n    # Validation errors - the details of which will be reported to clients on failure\n    1: validation.ValidationExceptionList validationErrors;\n    # Server errors - the details of which will not be reported to clients\n    2: finatra_thrift_exceptions.ServerError serverError\n  )\n}\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/frs_based_tweet.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\ninclude \"product.thrift\"\ninclude \"product_context.thrift\"\ninclude \"com/twitter/product_mixer/core/client_context.thrift\"\n\nstruct FrsTweetRequest {\n1: required client_context.ClientContext clientContext\n2: required product.Product product\n3: optional product_context.ProductContext productContext\n# excludedUserIds - user ids to be excluded from FRS candidate generation\n4: optional list<i64> excludedUserIds (personalDataType = 'UserId')\n# excludedTweetIds - tweet ids to be excluded from Earlybird candidate generation\n5: optional list<i64> excludedTweetIds (personalDataType = 'TweetId')\n} (persisted='true', hasPersonalData='true')\n\nstruct FrsTweet {\n1: required i64 tweetId (personalDataType = 'TweetId')\n2: required i64 authorId (personalDataType = 'UserId')\n# skip 3 in case we need tweet score in the future\n# frsPrimarySource - which FRS candidate source is the primary one to generate this author\n4: optional i32 frsPrimarySource\n# frsCandidateSourceScores - FRS candidate sources and the scores for this author\n# for i32 to algorithm mapping, see https://sourcegraph.twitter.biz/git.twitter.biz/source/-/blob/hermit/hermit-core/src/main/scala/com/twitter/hermit/constants/AlgorithmFeedbackTokens.scala?L12\n5: optional map<i32, double> frsCandidateSourceScores\n# frsPrimaryScore - the score of the FRS primary candidate source\n6: optional double frsAuthorScore\n} (persisted='true', hasPersonalData = 'true')\n\nstruct FrsTweetResponse {\n  1: required list<FrsTweet> tweets\n} (persisted='true')\n\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/metric_tags.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\n\n// NOTE: DO NOT depend on MetricTags for important ML Features or business logic.\n// MetricTags are meant for stats tracking & debugging purposes ONLY.\n// cr-mixer may change its definitions & how each candidate is tagged without public notice.\n// NOTE: TSPS needs the caller (Home) to specify which signal it uses to make Personalized Topics\nenum MetricTag {\n  // Source Signal Tags\n  TweetFavorite         = 0\n  Retweet               = 1\n  TrafficAttribution    = 2\n  OriginalTweet         = 3\n  Reply                 = 4\n  TweetShare            = 5\n\n  UserFollow            = 101\n  UserRepeatedProfileVisit = 102\n\n  PushOpenOrNtabClick   = 201\n\n  HomeTweetClick        = 301\n  HomeVideoView         = 302\n\n  // sim engine types\n  SimClustersANN        = 401\n  TweetBasedUserTweetGraph    = 402\n  TweetBasedTwHINANN          = 403\n  ConsumerEmbeddingBasedTwHINANN = 404\n\n\n  // combined engine types\n  UserInterestedIn      = 501 // Will deprecate soon\n  LookalikeUTG          = 502\n  TwhinCollabFilter     = 503\n\n  // Offline Twice\n  TwiceUserId           = 601\n\n  // Other Metric Tags\n  RequestHealthFilterPushOpenBasedTweetEmbedding = 701\n} (persisted='true', hasPersonalData='true')\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/product.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\n# In CrMixer, one org should only have one Product\nenum Product {\n  Home = 1\n  Notifications = 2\n  Email = 3\n  MoreTweetsModule = 4 # aka RUX\n  ImmersiveMediaViewer = 5\n  VideoCarousel = 6\n  ExploreTopics = 7\n  Ads = 8\n  HomeRealTime = 9 // Home Real-Time Tab is considered as a different Product surface to Home Tab. It's in early experiment phase.\n  TopicLandingPage = 10\n  HomeTopicsBackfill = 11\n  TopicTweetsStrato = 12\n}\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/product_context.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\nstruct HomeContext {\n\t2: optional i32 maxResults // enabled for QuaityFactor related DDGs only\n} (persisted='true', hasPersonalData='false')\n\nstruct NotificationsContext {\n\t1: optional i32 devNull // not being used. it's a placeholder\n} (persisted='true', hasPersonalData='false')\n\nstruct ExploreContext {\n  1: required bool isVideoOnly\n} (persisted='true', hasPersonalData='false')\n\nunion ProductContext {\n\t1: HomeContext homeContext\n\t2: NotificationsContext notificationsContext\n\t3: ExploreContext exploreContext\n} (persisted='true', hasPersonalData='false')\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/related_tweet.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\ninclude \"product.thrift\"\ninclude \"com/twitter/product_mixer/core/client_context.thrift\"\ninclude \"com/twitter/simclusters_v2/identifier.thrift\"\n\nstruct RelatedTweetRequest {\n  1: required identifier.InternalId internalId\n\t2: required product.Product product\n\t3: required client_context.ClientContext clientContext # RUX LogOut will have clientContext.userId = None\n\t4: optional list<i64> excludedTweetIds (personalDataType = 'TweetId')\n} (persisted='true', hasPersonalData='true')\n\nstruct RelatedTweet {\n  1: required i64 tweetId (personalDataType = 'TweetId')\n  2: optional double score\n  3: optional i64 authorId (personalDataType = 'UserId')\n} (persisted='true', hasPersonalData='true')\n\nstruct RelatedTweetResponse {\n  1: required list<RelatedTweet> tweets\n} (persisted='true')\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/related_video_tweet.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\ninclude \"product.thrift\"\ninclude \"com/twitter/product_mixer/core/client_context.thrift\"\ninclude \"com/twitter/simclusters_v2/identifier.thrift\"\n\nstruct RelatedVideoTweetRequest {\n  1: required identifier.InternalId internalId\n\t2: required product.Product product\n\t3: required client_context.ClientContext clientContext # RUX LogOut will have clientContext.userId = None\n\t4: optional list<i64> excludedTweetIds (personalDataType = 'TweetId')\n} (persisted='true', hasPersonalData='true')\n\nstruct RelatedVideoTweet {\n  1: required i64 tweetId (personalDataType = 'TweetId')\n  2: optional double score\n} (persisted='true', hasPersonalData='true')\n\nstruct RelatedVideoTweetResponse {\n  1: required list<RelatedVideoTweet> tweets\n} (persisted='true')\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/scribe.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\ninclude \"ads.thrift\"\ninclude \"candidate_generation_key.thrift\"\ninclude \"cr_mixer.thrift\"\ninclude \"metric_tags.thrift\"\ninclude \"product.thrift\"\ninclude \"related_tweet.thrift\"\ninclude \"source_type.thrift\"\ninclude \"uteg.thrift\"\ninclude \"com/twitter/ml/api/data.thrift\"\ninclude \"com/twitter/simclusters_v2/identifier.thrift\"\n\nstruct VITTweetCandidatesScribe {\n  1: required i64 uuid (personalDataType = 'UniversallyUniqueIdentifierUuid') # RequestUUID - unique scribe id for every request that comes in. Same request but different stages of scribe log (FetchCandidate, Filter, etc) share the same uuid\n  2: required i64 userId (personalDataType = 'UserId')\n  3: required list<VITTweetCandidateScribe> candidates\n  7: required product.Product product\n  8: required list<ImpressesedBucketInfo> impressedBuckets\n} (persisted='true', hasPersonalData = 'true')\n\nstruct VITTweetCandidateScribe {\n  1: required i64 tweetId (personalDataType = 'TweetId')\n  2: required i64 authorId (personalDataType = 'UserId')\n  3: required double score\n  4: required list<metric_tags.MetricTag> metricTags\n} (persisted='true', hasPersonalData = 'true')\n\nstruct GetTweetsRecommendationsScribe {\n  1: required i64 uuid (personalDataType = 'UniversallyUniqueIdentifierUuid') # RequestUUID - unique scribe id for every request that comes in. Same request but different stages of scribe log (FetchCandidate, Filter, etc) share the same uuid\n  2: required i64 userId (personalDataType = 'UserId')\n  3: required Result result\n  4: optional i64 traceId\n  5: optional PerformanceMetrics performanceMetrics\n  6: optional list<ImpressesedBucketInfo> impressedBuckets\n} (persisted='true', hasPersonalData = 'true')\n\nstruct SourceSignal {\n  # optional, since that the next step covers all info here\n  1: optional identifier.InternalId id\n} (persisted='true')\n\nstruct PerformanceMetrics {\n  1: optional i64 latencyMs\n} (persisted='true')\n\nstruct TweetCandidateWithMetadata {\n  1: required i64 tweetId (personalDataType = 'TweetId')\n  2: optional candidate_generation_key.CandidateGenerationKey candidateGenerationKey\n  3: optional i64 authorId (personalDataType = 'UserId') # only for InterleaveResult for hydrating training data\n  4: optional double score # score with respect to candidateGenerationKey\n  5: optional data.DataRecord dataRecord # attach any features to this candidate\n  6: optional i32 numCandidateGenerationKeys # num CandidateGenerationKeys generating this tweetId  \n} (persisted='true')\n\nstruct FetchSignalSourcesResult { \n  1: optional set<SourceSignal> signals\n} (persisted='true')\n\nstruct FetchCandidatesResult {\n  1: optional list<TweetCandidateWithMetadata> tweets\n} (persisted='true')\n\nstruct PreRankFilterResult {\n  1: optional list<TweetCandidateWithMetadata> tweets\n} (persisted='true')\n\nstruct InterleaveResult {\n  1: optional list<TweetCandidateWithMetadata> tweets\n} (persisted='true')\n\nstruct RankResult {\n  1: optional list<TweetCandidateWithMetadata> tweets\n} (persisted='true')\n\nstruct TopLevelApiResult {\n  1: required i64 timestamp (personalDataType = 'PrivateTimestamp')\n  2: required cr_mixer.CrMixerTweetRequest request\n  3: required cr_mixer.CrMixerTweetResponse response\n} (persisted='true')\n\nunion Result {\n  1: FetchSignalSourcesResult fetchSignalSourcesResult\n  2: FetchCandidatesResult fetchCandidatesResult\n  3: PreRankFilterResult preRankFilterResult\n  4: InterleaveResult interleaveResult\n  5: RankResult rankResult\n  6: TopLevelApiResult topLevelApiResult\n} (persisted='true', hasPersonalData = 'true')\n\nstruct ImpressesedBucketInfo {\n  1: required i64 experimentId (personalDataType = 'ExperimentId')\n  2: required string bucketName\n  3: required i32 version\n} (persisted='true')\n\n############# RelatedTweets Scribe #############\n\nstruct GetRelatedTweetsScribe {\n  1: required i64 uuid (personalDataType = 'UniversallyUniqueIdentifierUuid') # RequestUUID - unique scribe id for every request that comes in. Same request but different stages of scribe log (FetchCandidate, Filter, etc) share the same uuid\n  2: required identifier.InternalId internalId\n  3: required RelatedTweetResult relatedTweetResult\n  4: optional i64 requesterId (personalDataType = 'UserId')\n  5: optional i64 guestId (personalDataType = 'GuestId')\n  6: optional i64 traceId\n  7: optional PerformanceMetrics performanceMetrics\n  8: optional list<ImpressesedBucketInfo> impressedBuckets\n} (persisted='true', hasPersonalData = 'true')\n\nstruct RelatedTweetTopLevelApiResult {\n  1: required i64 timestamp (personalDataType = 'PrivateTimestamp')\n  2: required related_tweet.RelatedTweetRequest request\n  3: required related_tweet.RelatedTweetResponse response\n} (persisted='true')\n\nunion RelatedTweetResult {\n  1: RelatedTweetTopLevelApiResult relatedTweetTopLevelApiResult\n  2: FetchCandidatesResult fetchCandidatesResult\n  3: PreRankFilterResult preRankFilterResult # results after seqential filters\n  # if later we need rankResult, we can add it here\n} (persisted='true', hasPersonalData = 'true')\n\n############# UtegTweets Scribe #############\n\nstruct GetUtegTweetsScribe {\n  1: required i64 uuid (personalDataType = 'UniversallyUniqueIdentifierUuid') # RequestUUID - unique scribe id for every request that comes in. Same request but different stages of scribe log (FetchCandidate, Filter, etc) share the same uuid\n  2: required i64 userId (personalDataType = 'UserId')\n  3: required UtegTweetResult utegTweetResult\n  4: optional i64 traceId\n  5: optional PerformanceMetrics performanceMetrics\n  6: optional list<ImpressesedBucketInfo> impressedBuckets\n} (persisted='true', hasPersonalData = 'true')\n\nstruct UtegTweetTopLevelApiResult {\n  1: required i64 timestamp (personalDataType = 'PrivateTimestamp')\n  2: required uteg.UtegTweetRequest request\n  3: required uteg.UtegTweetResponse response\n} (persisted='true')\n\nunion UtegTweetResult {\n  1: UtegTweetTopLevelApiResult utegTweetTopLevelApiResult\n  2: FetchCandidatesResult fetchCandidatesResult\n  # if later we need rankResult, we can add it here\n} (persisted='true', hasPersonalData = 'true')\n\n############# getAdsRecommendations() Scribe #############\n\nstruct GetAdsRecommendationsScribe {\n  1: required i64 uuid (personalDataType = 'UniversallyUniqueIdentifierUuid') # RequestUUID - unique scribe id for every request that comes in. Same request but different stages of scribe log (FetchCandidate, Filter, etc) share the same uuid\n  2: required i64 userId (personalDataType = 'UserId')\n  3: required AdsRecommendationsResult result\n  4: optional i64 traceId\n  5: optional PerformanceMetrics performanceMetrics\n  6: optional list<ImpressesedBucketInfo> impressedBuckets\n} (persisted='true', hasPersonalData = 'true')\n\nstruct AdsRecommendationTopLevelApiResult {\n  1: required i64 timestamp (personalDataType = 'PrivateTimestamp')\n  2: required ads.AdsRequest request\n  3: required ads.AdsResponse response\n} (persisted='true')\n\nunion AdsRecommendationsResult{\n  1: AdsRecommendationTopLevelApiResult adsRecommendationTopLevelApiResult\n  2: FetchCandidatesResult fetchCandidatesResult\n}(persisted='true', hasPersonalData = 'true')\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/source_type.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\n// Due to legacy reason, SourceType used to represent both SourceSignalType and SimilarityEngineType\n// Hence, you can see several SourceType such as UserInterestedIn, HashSpace, etc.\n// Moving forward, SourceType will be used for SourceSignalType ONLY. eg., TweetFavorite, UserFollow\n// We will create a new SimilarityEngineType to separate them. eg., SimClustersANN\nenum SourceType {\n  // Tweet based Source Signal\n  TweetFavorite       = 0\n  Retweet             = 1\n  TrafficAttribution  = 2 // Traffic Attribution will be migrated over in Q3\n  OriginalTweet       = 3\n  Reply               = 4\n  TweetShare          = 5\n  GoodTweetClick      = 6 // total dwell time > N seconds after click on the tweet\n  VideoTweetQualityView = 7\n  VideoTweetPlayback50  = 8\n\n  // UserId based Source Signal (includes both Producer/Consumer)\n  UserFollow               = 101\n  UserRepeatedProfileVisit = 102\n\n  CurrentUser_DEPRECATED   = 103\n\n  RealGraphOon             = 104\n  FollowRecommendation     = 105\n\n  TwiceUserId              = 106\n  UserTrafficAttributionProfileVisit = 107\n  GoodProfileClick         = 108 // total dwell time > N seconds after click into the profile page\n\n  // (Notification) Tweet based Source Signal\n  NotificationClick   = 201\n\n  // (Home) Tweet based Source Signal\n  HomeTweetClick       = 301\n  HomeVideoView        = 302\n  HomeSongbirdShowMore = 303\n\n  // Topic based Source Signal\n  TopicFollow         = 401 // Deprecated\n  PopularTopic        = 402 // Deprecated\n\n  // Old CR code\n  UserInterestedIn    = 501 // Deprecated\n  TwiceInterestedIn   = 502 // Deprecated\n  MBCG                = 503 // Deprecated\n  HashSpace           = 504 // Deprecated\n\n  // Old CR code\n  Cluster             = 601 // Deprecated\n\n  // Search based Source Signal\n  SearchProfileClick  = 701 // Deprecated\n  SearchTweetClick    = 702 // Deprecated\n\n  // Graph based Source\n  StrongTiePrediction      = 801 // STP\n  TwiceClustersMembers     = 802\n  Lookalike                = 803 // Deprecated\n  RealGraphIn              = 804\n\n  // Current requester User Id. It is only used for scribing. Placeholder value\n  RequestUserId       = 1001\n  // Current request Tweet Id used in RelatedTweet. Placeholder value\n  RequestTweetId      = 1002\n\n  // Negative Signals\n  TweetReport = 1101\n  TweetDontLike = 1102\n  TweetSeeFewer = 1103\n  AccountBlock = 1104\n  AccountMute = 1105\n\n  // Aggregated Signals\n  TweetAggregation = 1201\n  ProducerAggregation = 1202\n} (persisted='true', hasPersonalData='true')\n\nenum SimilarityEngineType {\n  SimClustersANN              = 1\n  TweetBasedUserTweetGraph    = 2\n  TweetBasedTwHINANN          = 3\n  Follow2VecANN               = 4 // ConsumerEmbeddingBasedFollow2Vec\n  QIG                         = 5\n  OfflineSimClustersANN       = 6\n  LookalikeUTG_DEPRECATED     = 7\n  ProducerBasedUserTweetGraph = 8\n  FrsUTG_DEPRECATED           = 9\n  RealGraphOonUTG_DEPRECATED  = 10\n  ConsumerEmbeddingBasedTwHINANN = 11\n  TwhinCollabFilter           = 12\n  TwiceUTG_DEPRECATED         = 13\n  ConsumerEmbeddingBasedTwoTowerANN = 14\n  TweetBasedBeTANN            = 15\n  StpUTG_DEPRECATED           = 16\n  UTEG                        = 17\n  ROMR                        = 18\n  ConsumersBasedUserTweetGraph  = 19\n  TweetBasedUserVideoGraph    = 20\n  CertoTopicTweet             = 24\n  ConsumersBasedUserAdGraph   = 25\n  TweetBasedUserAdGraph       = 26\n  SkitTfgTopicTweet           = 27\n  ConsumerBasedWalsANN        = 28\n  ProducerBasedUserAdGraph    = 29\n  SkitHighPrecisionTopicTweet = 30\n  SkitInterestBrowserTopicTweet = 31\n  SkitProducerBasedTopicTweet   = 32\n  ExploreTripOfflineSimClustersTweets = 33\n  DiffusionBasedTweet = 34\n  ConsumersBasedUserVideoGraph  = 35\n\n  // In network\n  EarlybirdRecencyBasedSimilarityEngine = 21\n  EarlybirdModelBasedSimilarityEngine = 22\n  EarlybirdTensorflowBasedSimilarityEngine = 23\n  // Composite\n  TweetBasedUnifiedSimilarityEngine    = 1001\n  ProducerBasedUnifiedSimilarityEngine = 1002\n} (persisted='true')\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/topic_tweet.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\ninclude \"com/twitter/product_mixer/core/client_context.thrift\"\ninclude \"product.thrift\"\ninclude \"product_context.thrift\"\ninclude \"source_type.thrift\"\n\n\nstruct TopicTweetRequest {\n    1: required client_context.ClientContext clientContext\n    2: required product.Product product\n    3: required list<i64> topicIds\n    5: optional product_context.ProductContext productContext\n    6: optional list<i64> excludedTweetIds (personalDataType = 'TweetId')\n} (persisted='true', hasPersonalData='true')\n\nstruct TopicTweet {\n    1: required i64 tweetId (personalDataType = 'TweetId')\n    2: required double score\n    3: required source_type.SimilarityEngineType similarityEngineType\n} (persisted='true', hasPersonalData = 'true')\n\nstruct TopicTweetResponse {\n    1: required map<i64, list<TopicTweet>> tweets\n} (persisted='true')\n\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/uteg.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\ninclude \"product.thrift\"\ninclude \"product_context.thrift\"\n\ninclude \"com/twitter/product_mixer/core/client_context.thrift\"\ninclude \"com/twitter/recos/recos_common.thrift\"\n\nstruct UtegTweetRequest {\n\t1: required client_context.ClientContext clientContext\n\t2: required product.Product product\n\t# Product-specific parameters should be placed in the Product Context\n\t3: optional product_context.ProductContext productContext\n\t4: optional list<i64> excludedTweetIds (personalDataType = 'TweetId')\n} (persisted='true', hasPersonalData='true')\n\nstruct UtegTweet {\n  // tweet id\n  1: required i64 tweetId(personalDataType = 'TweetId')\n  // sum of weights of seed users who engaged with the tweet.\n  // If a user engaged with the same tweet twice, liked it and retweeted it, then his/her weight was counted twice.\n  2: required double score\n  // user social proofs per engagement type\n  3: required map<recos_common.SocialProofType, list<i64>> socialProofByType(personalDataTypeKey='EngagementTypePrivate', personalDataTypeValue='UserId')\n} (persisted='true', hasPersonalData = 'true')\n\nstruct UtegTweetResponse {\n  1: required list<UtegTweet> tweets\n} (persisted='true')\n"
  },
  {
    "path": "cr-mixer/thrift/src/main/thrift/validation.thrift",
    "content": "namespace java com.twitter.cr_mixer.thriftjava\n#@namespace scala com.twitter.cr_mixer.thriftscala\n#@namespace strato com.twitter.cr_mixer\n\n// ValidationErrorCode is used to identify classes of client errors returned from a Product Mixer\n// service. Use [[PipelineFailureExceptionMapper]] to adapt pipeline failures into thrift errors.\nenum ValidationErrorCode {\n  PRODUCT_DISABLED = 1\n  PLACEHOLDER_2 = 2\n} (hasPersonalData='false')\n\nexception ValidationException {\n  1: ValidationErrorCode errorCode\n  2: string msg\n} (hasPersonalData='false')\n\nexception ValidationExceptionList {\n  1: list<ValidationException> errors\n} (hasPersonalData='false')\n"
  },
  {
    "path": "follow-recommendations-service/BUILD",
    "content": "# Without this alias, library :follow-recommendations-service_lib would conflict with :bin\nalias(\n    name = \"follow-recommendations-service\",\n    target = \":follow-recommendations-service_lib\",\n)\n\ntarget(\n    name = \"follow-recommendations-service_lib\",\n    dependencies = [\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models\",\n    ],\n)\n\njvm_binary(\n    name = \"bin\",\n    basename = \"follow-recommendations-service\",\n    main = \"com.twitter.follow_recommendations.FollowRecommendationsServiceThriftServerMain\",\n    runtime_platform = \"java11\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":follow-recommendations-service\",\n        \"3rdparty/jvm/ch/qos/logback:logback-classic\",\n        \"finagle/finagle-zipkin-scribe/src/main/scala\",\n        \"finatra/inject/inject-logback/src/main/scala\",\n        \"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback\",\n        \"twitter-server-internal/src/main/scala\",\n        \"twitter-server/logback-classic/src/main/scala\",\n    ],\n)\n\n#  Aurora Workflows build phase convention requires a jvm_app named with ${project-name}-app\njvm_app(\n    name = \"follow-recommendations-service-app\",\n    archive = \"zip\",\n    binary = \":bin\",\n    bundles = [\n        bundle(\n            fileset = [\n                \"server/src/main/resources/*\",\n                \"server/src/main/resources/**/*\",\n            ],\n            owning_target = \"follow-recommendations-service/server/src/main/resources:frs_resources\",\n            relative_to = \"server/src/main/resources\",\n        ),\n    ],\n    tags = [\"bazel-compatible\"],\n)\n"
  },
  {
    "path": "follow-recommendations-service/CONFIG.ini",
    "content": "[code-coverage]\npackage = com.twitter.follow_recommendations\n\n[docbird]\nproject_name = follow-recommendations-service\nproject_type = service\n; example settings:\n; \n; project_name = fluffybird\n; description = fluffybird is a service for fluffing up feathers.\n; tags = python,documentation,fluffybird\n; project_type = service\n;  - allowed options: essay, library, service, hub, cookbook, styleguide, policy\n; owner_links = roster\n;  - allowed options: roster, find, email\n; scrolling_tocs = yes\n; comments = yes\n; verifications = yes\n; support_widget = yes\n; health_score = yes\n; sticky_sidebar = no\n\n[jira]\nproject = CJREL\n"
  },
  {
    "path": "follow-recommendations-service/README.md",
    "content": "# Follow Recommendations Service\n\n## Introduction to the Follow Recommendations Service (FRS)\nThe Follow Recommendations Service (FRS) is a robust recommendation engine designed to provide users with personalized suggestions for accounts to follow. At present, FRS supports Who-To-Follow (WTF) module recommendations across a variety of Twitter product interfaces. Additionally, by suggesting tweet authors, FRS also delivers FutureGraph tweet recommendations, which consist of tweets from accounts that users may be interested in following in the future.\n\n## Design\nThe system is tailored to accommodate diverse use cases, such as Post New-User-Experience (NUX), advertisements, FutureGraph tweets, and more. Each use case features a unique display location identifier. To view all display locations, refer to the following path: `follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/DisplayLocation.scala`.\n\nRecommendation steps are customized according to each display location. Common and high-level steps are encapsulated within the \"RecommendationFlow,\" which includes operations like candidate generation, ranker selection, filtering, transformation, and beyond. To explore all flows, refer to this path: `follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows`.\n\nFor each product (corresponding to a display location), one or multiple flows can be selected to generate candidates based on code and configurations. To view all products, refer to the following path: `follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline_tweet_recs`.\n\nThe FRS overview diagram is depicted below:\n\n![FRS_architecture.png](FRS_architecture.png)\n\n\n### Candidate Generation\nDuring this step, FRS utilizes various user signals and algorithms to identify candidates from all Twitter accounts. The candidate source folder is located at `follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/`, with a README file provided within each candidate source folder.\n\n### Filtering\nIn this phase, FRS applies different filtering logic after generating account candidates to improve quality and health. Filtering may occur before and/or after the ranking step, with heavier filtering logic (e.g., higher latency) typically applied after the ranking step. The filters' folder is located at `follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates`.\n\n### Ranking\nDuring this step, FRS employs both Machine Learning (ML) and heuristic rule-based candidate ranking. For the ML ranker, ML features are fetched beforehand (i.e., feature hydration),\nand a DataRecord (the Twitter-standard Machine Learning data format used to represent feature data, labels, and predictions when training or serving) is constructed for each <user, candidate> pair. \nThese pairs are then sent to a separate ML prediction service, which houses the ML model trained offline.\nThe ML prediction service returns a prediction score, representing the probability that a user will follow and engage with the candidate.\nThis score is a weighted sum of p(follow|recommendation) and p(positive engagement|follow), and FRS uses this score to rank the candidates.\n\nThe rankers' folder is located at `follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers`.\n\n### Transform\nIn this phase, the sequence of candidates undergoes necessary transformations, such as deduplication, attaching social proof (i.e., \"followed by XX user\"), adding tracking tokens, and more.\nThe transformers' folder can be found at `follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms`.\n\n### Truncation\nDuring this final step, FRS trims the candidate pool to a specified size. This process ensures that only the most relevant and engaging candidates are presented to users while maintaining an optimal user experience.\n\nBy implementing these comprehensive steps and adapting to various use cases, the Follow Recommendations Service (FRS) effectively curates tailored suggestions for Twitter users, enhancing their overall experience and promoting meaningful connections within the platform.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"configapi/configapi-core/src/main/scala/com/twitter/timelines/configapi\",\n        \"finagle/finagle-core/src/main\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/model\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/recommendation\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/recommendation\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base/CandidateSourceRegistry.scala",
    "content": "package com.twitter.follow_recommendations.common.base\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.EnrichedCandidateSource.toEnriched\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\n\n// a helper structure to register and select candidate sources based on identifiers\ntrait CandidateSourceRegistry[Target, Candidate] {\n\n  val statsReceiver: StatsReceiver\n\n  def sources: Set[CandidateSource[Target, Candidate]]\n\n  final lazy val candidateSources: Map[\n    CandidateSourceIdentifier,\n    CandidateSource[Target, Candidate]\n  ] = {\n    val map = sources.map { c =>\n      c.identifier -> c.observe(statsReceiver)\n    }.toMap\n\n    if (map.size != sources.size) {\n      throw new IllegalArgumentException(\"Duplicate Candidate Source Identifiers\")\n    }\n\n    map\n  }\n\n  def select(\n    identifiers: Set[CandidateSourceIdentifier]\n  ): Set[CandidateSource[Target, Candidate]] = {\n    // fails loud if the candidate source is not registered\n    identifiers.map(candidateSources(_))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base/EnrichedCandidateSource.scala",
    "content": "package com.twitter.follow_recommendations.common.base\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Duration\nimport com.twitter.util.TimeoutException\nimport scala.language.implicitConversions\n\nclass EnrichedCandidateSource[Target, Candidate](original: CandidateSource[Target, Candidate]) {\n\n  /**\n   * Gate the candidate source based on the Predicate of target.\n   * It returns results only if the predicate returns Valid.\n   *\n   * @param predicate\n   * @return\n   */\n  def gate(predicate: Predicate[Target]): CandidateSource[Target, Candidate] = {\n    throw new UnsupportedOperationException()\n  }\n\n  def observe(statsReceiver: StatsReceiver): CandidateSource[Target, Candidate] = {\n    val originalIdentifier = original.identifier\n    val stats = statsReceiver.scope(originalIdentifier.name)\n    new CandidateSource[Target, Candidate] {\n      val identifier = originalIdentifier\n      override def apply(target: Target): Stitch[Seq[Candidate]] = {\n        StatsUtil.profileStitchSeqResults[Candidate](original(target), stats)\n      }\n    }\n  }\n\n  /**\n   * Map target type into new target type (1 to optional mapping)\n   */\n  def stitchMapKey[Target2](\n    targetMapper: Target2 => Stitch[Option[Target]]\n  ): CandidateSource[Target2, Candidate] = {\n    val targetsMapper: Target2 => Stitch[Seq[Target]] = { target =>\n      targetMapper(target).map(_.toSeq)\n    }\n    stitchMapKeys(targetsMapper)\n  }\n\n  /**\n   * Map target type into new target type (1 to many mapping)\n   */\n  def stitchMapKeys[Target2](\n    targetMapper: Target2 => Stitch[Seq[Target]]\n  ): CandidateSource[Target2, Candidate] = {\n    new CandidateSource[Target2, Candidate] {\n      val identifier = original.identifier\n      override def apply(target: Target2): Stitch[Seq[Candidate]] = {\n        for {\n          mappedTargets <- targetMapper(target)\n          results <- Stitch.traverse(mappedTargets)(original(_))\n        } yield results.flatten\n      }\n    }\n  }\n\n  /**\n   * Map target type into new target type (1 to many mapping)\n   */\n  def mapKeys[Target2](\n    targetMapper: Target2 => Seq[Target]\n  ): CandidateSource[Target2, Candidate] = {\n    val stitchMapper: Target2 => Stitch[Seq[Target]] = { target =>\n      Stitch.value(targetMapper(target))\n    }\n    stitchMapKeys(stitchMapper)\n  }\n\n  /**\n   * Map candidate types to new type based on candidateMapper\n   */\n  def mapValues[Candidate2](\n    candidateMapper: Candidate => Stitch[Option[Candidate2]]\n  ): CandidateSource[Target, Candidate2] = {\n\n    new CandidateSource[Target, Candidate2] {\n      val identifier = original.identifier\n      override def apply(target: Target): Stitch[Seq[Candidate2]] = {\n        original(target).flatMap { candidates =>\n          val results = Stitch.traverse(candidates)(candidateMapper(_))\n          results.map(_.flatten)\n        }\n      }\n    }\n  }\n\n  /**\n   * Map candidate types to new type based on candidateMapper\n   */\n  def mapValue[Candidate2](\n    candidateMapper: Candidate => Candidate2\n  ): CandidateSource[Target, Candidate2] = {\n    val stitchMapper: Candidate => Stitch[Option[Candidate2]] = { c =>\n      Stitch.value(Some(candidateMapper(c)))\n    }\n    mapValues(stitchMapper)\n  }\n\n  /**\n   * This method wraps the candidate source in a designated timeout so that a single candidate\n   * source does not result in a timeout for the entire flow\n   */\n  def within(\n    candidateTimeout: Duration,\n    statsReceiver: StatsReceiver\n  ): CandidateSource[Target, Candidate] = {\n    val originalIdentifier = original.identifier\n    val timeoutCounter =\n      statsReceiver.counter(originalIdentifier.name, \"timeout\")\n\n    new CandidateSource[Target, Candidate] {\n      val identifier = originalIdentifier\n      override def apply(target: Target): Stitch[Seq[Candidate]] = {\n        original\n          .apply(target)\n          .within(candidateTimeout)(com.twitter.finagle.util.DefaultTimer)\n          .rescue {\n            case _: TimeoutException =>\n              timeoutCounter.incr()\n              Stitch.Nil\n          }\n      }\n    }\n  }\n\n  def failOpenWithin(\n    candidateTimeout: Duration,\n    statsReceiver: StatsReceiver\n  ): CandidateSource[Target, Candidate] = {\n    val originalIdentifier = original.identifier\n    val timeoutCounter =\n      statsReceiver.counter(originalIdentifier.name, \"timeout\")\n\n    new CandidateSource[Target, Candidate] {\n      val identifier = originalIdentifier\n      override def apply(target: Target): Stitch[Seq[Candidate]] = {\n        original\n          .apply(target)\n          .within(candidateTimeout)(com.twitter.finagle.util.DefaultTimer)\n          .handle {\n            case _: TimeoutException =>\n              timeoutCounter.incr()\n              Seq.empty\n            case e: Exception =>\n              statsReceiver\n                .scope(\"candidate_source_error\").scope(originalIdentifier.name).counter(\n                  e.getClass.getSimpleName).incr\n              Seq.empty\n          }\n      }\n    }\n  }\n}\n\nobject EnrichedCandidateSource {\n  implicit def toEnriched[K, V](original: CandidateSource[K, V]): EnrichedCandidateSource[K, V] =\n    new EnrichedCandidateSource(original)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base/ParamPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.base\n\nimport com.twitter.follow_recommendations.common.models.FilterReason.ParamReason\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Param\n\ncase class ParamPredicate[Request <: HasParams](param: Param[Boolean]) extends Predicate[Request] {\n\n  def apply(request: Request): Stitch[PredicateResult] = {\n    if (request.params(param)) {\n      Stitch.value(PredicateResult.Valid)\n    } else {\n      Stitch.value(PredicateResult.Invalid(Set(ParamReason(param.statName))))\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base/Predicate.scala",
    "content": "package com.twitter.follow_recommendations.common.base\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.FilterReason\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Stitch\n\ntrait Predicate[-Q] {\n\n  def apply(item: Q): Stitch[PredicateResult]\n  def arrow: Arrow[Q, PredicateResult] = Arrow.apply(apply)\n\n  def map[K](mapper: K => Q): Predicate[K] = Predicate(arrow.contramap(mapper))\n\n  /**\n   * check the predicate results for a batch of items for convenience.\n   *\n   * mark it as final to avoid potential abuse usage\n   */\n  final def batch(items: Seq[Q]): Stitch[Seq[PredicateResult]] = {\n    this.arrow.traverse(items)\n  }\n\n  /**\n   * Syntax sugar for functions which take in 2 inputs as a tuple.\n   */\n  def apply[Q1, Q2](item1: Q1, item2: Q2)(implicit ev: ((Q1, Q2)) => Q): Stitch[PredicateResult] = {\n    apply((item1, item2))\n  }\n\n  /**\n   * Runs the predicates in sequence. The returned predicate will return true iff both the predicates return true.\n   * ie. it is an AND operation\n   *\n   * We short-circuit the evaluation, ie we don't evaluate the 2nd predicate if the 1st is false\n   *\n   * @param p predicate to run in sequence\n   *\n   * @return a new predicate object that represents the logical AND of both predicates\n   */\n  def andThen[Q1 <: Q](p: Predicate[Q1]): Predicate[Q1] = {\n    Predicate({ query: Q1 =>\n      apply(query).flatMap {\n        case PredicateResult.Valid => p(query)\n        case PredicateResult.Invalid(reasons) => Stitch.value(PredicateResult.Invalid(reasons))\n      }\n    })\n  }\n\n  /**\n   * Creates a predicate which runs the current & given predicate in sequence.\n   * The returned predicate will return true if either current or given predicate returns true.\n   * That is, given predicate will be only run if current predicate returns false.\n   *\n   * @param p predicate to run in sequence\n   *\n   * @return new predicate object that represents the logical OR of both predicates.\n   *         if both are invalid, the reason would be the set of all invalid reasons.\n   */\n  def or[Q1 <: Q](p: Predicate[Q1]): Predicate[Q1] = {\n    Predicate({ query: Q1 =>\n      apply(query).flatMap {\n        case PredicateResult.Valid => Stitch.value(PredicateResult.Valid)\n        case PredicateResult.Invalid(reasons) =>\n          p(query).flatMap {\n            case PredicateResult.Valid => Stitch.value(PredicateResult.Valid)\n            case PredicateResult.Invalid(newReasons) =>\n              Stitch.value(PredicateResult.Invalid(reasons ++ newReasons))\n          }\n      }\n    })\n  }\n\n  /*\n   * Runs the predicate only if the provided predicate is valid, otherwise returns valid.\n   * */\n  def gate[Q1 <: Q](gatingPredicate: Predicate[Q1]): Predicate[Q1] = {\n    Predicate { query: Q1 =>\n      gatingPredicate(query).flatMap { result =>\n        if (result == PredicateResult.Valid) {\n          apply(query)\n        } else {\n          Stitch.value(PredicateResult.Valid)\n        }\n      }\n    }\n  }\n\n  def observe(statsReceiver: StatsReceiver): Predicate[Q] = Predicate(\n    StatsUtil.profilePredicateResult(this.arrow, statsReceiver))\n\n  def convertToFailOpenWithResultType(resultType: PredicateResult): Predicate[Q] = {\n    Predicate { query: Q =>\n      apply(query).handle {\n        case _: Exception =>\n          resultType\n      }\n\n    }\n  }\n\n}\n\nclass TruePredicate[Q] extends Predicate[Q] {\n  override def apply(item: Q): Stitch[PredicateResult] = Predicate.AlwaysTrueStitch\n}\n\nclass FalsePredicate[Q](reason: FilterReason) extends Predicate[Q] {\n  val InvalidResult = Stitch.value(PredicateResult.Invalid(Set(reason)))\n  override def apply(item: Q): Stitch[PredicateResult] = InvalidResult\n}\n\nobject Predicate {\n\n  val AlwaysTrueStitch = Stitch.value(PredicateResult.Valid)\n\n  val NumBatchesStat = \"num_batches_stats\"\n  val NumBatchesCount = \"num_batches\"\n\n  def apply[Q](func: Q => Stitch[PredicateResult]): Predicate[Q] = new Predicate[Q] {\n    override def apply(item: Q): Stitch[PredicateResult] = func(item)\n\n    override val arrow: Arrow[Q, PredicateResult] = Arrow(func)\n  }\n\n  def apply[Q](outerArrow: Arrow[Q, PredicateResult]): Predicate[Q] = new Predicate[Q] {\n    override def apply(item: Q): Stitch[PredicateResult] = arrow(item)\n\n    override val arrow: Arrow[Q, PredicateResult] = outerArrow\n  }\n\n  /**\n   * Given some items, this function\n   * 1. chunks them up in groups\n   * 2. lazily applies a predicate on each group\n   * 3. filters based on the predicate\n   * 4. takes first numToTake items.\n   *\n   * If numToTake is satisfied, then any later predicates are not called.\n   *\n   * @param items     items of type Q\n   * @param predicate predicate that determines whether an item is acceptable\n   * @param batchSize batch size to call the predicate with\n   * @param numToTake max number of items to return\n   * @param stats stats receiver\n   * @tparam Q type of item\n   *\n   * @return a future of K items\n   */\n  def batchFilterTake[Q](\n    items: Seq[Q],\n    predicate: Predicate[Q],\n    batchSize: Int,\n    numToTake: Int,\n    stats: StatsReceiver\n  ): Stitch[Seq[Q]] = {\n\n    def take(\n      input: Iterator[Stitch[Seq[Q]]],\n      prev: Seq[Q],\n      takeSize: Int,\n      numOfBatch: Int\n    ): Stitch[(Seq[Q], Int)] = {\n      if (input.hasNext) {\n        val currFut = input.next()\n        currFut.flatMap { curr =>\n          val taken = curr.take(takeSize)\n          val combined = prev ++ taken\n          if (taken.size < takeSize)\n            take(input, combined, takeSize - taken.size, numOfBatch + 1)\n          else Stitch.value((combined, numOfBatch + 1))\n        }\n      } else {\n        Stitch.value((prev, numOfBatch))\n      }\n    }\n\n    val batchedItems = items.view.grouped(batchSize)\n    val batchedFutures = batchedItems.map { batch =>\n      Stitch.traverse(batch)(predicate.apply).map { conds =>\n        (batch.zip(conds)).withFilter(_._2.value).map(_._1)\n      }\n    }\n    take(batchedFutures, Nil, numToTake, 0).map {\n      case (filtered: Seq[Q], numOfBatch: Int) =>\n        stats.stat(NumBatchesStat).add(numOfBatch)\n        stats.counter(NumBatchesCount).incr(numOfBatch)\n        filtered\n    }\n  }\n\n  /**\n   * filter a list of items based on the predicate\n   *\n   * @param items a list of items\n   * @param predicate predicate of the item\n   * @tparam Q item type\n   * @return the list of items that satisfy the predicate\n   */\n  def filter[Q](items: Seq[Q], predicate: Predicate[Q]): Stitch[Seq[Q]] = {\n    predicate.batch(items).map { results =>\n      items.zip(results).collect {\n        case (item, PredicateResult.Valid) => item\n      }\n    }\n  }\n\n  /**\n   * filter a list of items based on the predicate given the target\n   *\n   * @param target target item\n   * @param items a list of items\n   * @param predicate predicate of the (target, item) pair\n   * @tparam Q item type\n   * @return the list of items that satisfy the predicate given the target\n   */\n  def filter[T, Q](target: T, items: Seq[Q], predicate: Predicate[(T, Q)]): Stitch[Seq[Q]] = {\n    predicate.batch(items.map(i => (target, i))).map { results =>\n      items.zip(results).collect {\n        case (item, PredicateResult.Valid) => item\n      }\n    }\n  }\n\n  /**\n   * Returns a predicate, where an element is true iff it that element is true for all input predicates.\n   * ie. it is an AND operation\n   *\n   * This is done concurrently.\n   *\n   * @param predicates list of predicates\n   * @tparam Q Type parameter\n   *\n   * @return new predicate object that is the logical \"and\" of the input predicates\n   */\n  def andConcurrently[Q](predicates: Seq[Predicate[Q]]): Predicate[Q] = {\n    Predicate { query: Q =>\n      Stitch.traverse(predicates)(p => p(query)).map { predicateResults =>\n        val allInvalid = predicateResults\n          .collect {\n            case PredicateResult.Invalid(reason) =>\n              reason\n          }\n        if (allInvalid.isEmpty) {\n          PredicateResult.Valid\n        } else {\n          val allInvalidReasons = allInvalid.reduce(_ ++ _)\n          PredicateResult.Invalid(allInvalidReasons)\n        }\n      }\n    }\n  }\n}\n\n/**\n * applies the underlying predicate when the param is on.\n */\nabstract class GatedPredicateBase[Q](\n  underlyingPredicate: Predicate[Q],\n  stats: StatsReceiver = NullStatsReceiver)\n    extends Predicate[Q] {\n  def gate(item: Q): Boolean\n\n  val underlyingPredicateTotal = stats.counter(\"underlying_total\")\n  val underlyingPredicateValid = stats.counter(\"underlying_valid\")\n  val underlyingPredicateInvalid = stats.counter(\"underlying_invalid\")\n  val notGatedCounter = stats.counter(\"not_gated\")\n\n  val ValidStitch: Stitch[PredicateResult.Valid.type] = Stitch.value(PredicateResult.Valid)\n\n  override def apply(item: Q): Stitch[PredicateResult] = {\n    if (gate(item)) {\n      underlyingPredicateTotal.incr()\n      underlyingPredicate(item)\n    } else {\n      notGatedCounter.incr()\n      ValidStitch\n    }\n  }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base/PredicateResult.scala",
    "content": "package com.twitter.follow_recommendations.common.base\n\nimport com.twitter.follow_recommendations.common.models.FilterReason\n\nsealed trait PredicateResult {\n  def value: Boolean\n}\n\nobject PredicateResult {\n\n  case object Valid extends PredicateResult {\n    override val value = true\n  }\n\n  case class Invalid(reasons: Set[FilterReason] = Set.empty[FilterReason]) extends PredicateResult {\n    override val value = false\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base/Ranker.scala",
    "content": "package com.twitter.follow_recommendations.common.base\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Duration\nimport com.twitter.util.TimeoutException\n\n/**\n * Ranker is a special kind of transform that would only change the order of a list of items.\n * If a single item is given, it \"may\" attach additional scoring information to the item.\n *\n * @tparam Target target to recommend the candidates\n * @tparam Candidate candidate type to rank\n */\ntrait Ranker[Target, Candidate] extends Transform[Target, Candidate] { ranker =>\n\n  def rank(target: Target, candidates: Seq[Candidate]): Stitch[Seq[Candidate]]\n\n  override def transform(target: Target, candidates: Seq[Candidate]): Stitch[Seq[Candidate]] = {\n    rank(target, candidates)\n  }\n\n  override def observe(statsReceiver: StatsReceiver): Ranker[Target, Candidate] = {\n    val originalRanker = this\n    new Ranker[Target, Candidate] {\n      override def rank(target: Target, items: Seq[Candidate]): Stitch[Seq[Candidate]] = {\n        statsReceiver.counter(Transform.InputCandidatesCount).incr(items.size)\n        statsReceiver.stat(Transform.InputCandidatesStat).add(items.size)\n        StatsUtil.profileStitchSeqResults(originalRanker.rank(target, items), statsReceiver)\n      }\n    }\n  }\n\n  def reverse: Ranker[Target, Candidate] = new Ranker[Target, Candidate] {\n    def rank(target: Target, candidates: Seq[Candidate]): Stitch[Seq[Candidate]] =\n      ranker.rank(target, candidates).map(_.reverse)\n  }\n\n  def andThen(other: Ranker[Target, Candidate]): Ranker[Target, Candidate] = {\n    val original = this\n    new Ranker[Target, Candidate] {\n      def rank(target: Target, candidates: Seq[Candidate]): Stitch[Seq[Candidate]] = {\n        original.rank(target, candidates).flatMap { results => other.rank(target, results) }\n      }\n    }\n  }\n\n  /**\n   * This method wraps the Ranker in a designated timeout.\n   * If the ranker timeouts, it would return the original candidates directly,\n   * instead of failing the whole recommendation flow\n   */\n  def within(timeout: Duration, statsReceiver: StatsReceiver): Ranker[Target, Candidate] = {\n    val timeoutCounter = statsReceiver.counter(\"timeout\")\n    val original = this\n    new Ranker[Target, Candidate] {\n      override def rank(target: Target, candidates: Seq[Candidate]): Stitch[Seq[Candidate]] = {\n        original\n          .rank(target, candidates)\n          .within(timeout)(com.twitter.finagle.util.DefaultTimer)\n          .rescue {\n            case _: TimeoutException =>\n              timeoutCounter.incr()\n              Stitch.value(candidates)\n          }\n      }\n    }\n  }\n}\n\nobject Ranker {\n\n  def chain[Target, Candidate](\n    transformer: Transform[Target, Candidate],\n    ranker: Ranker[Target, Candidate]\n  ): Ranker[Target, Candidate] = {\n    new Ranker[Target, Candidate] {\n      def rank(target: Target, candidates: Seq[Candidate]): Stitch[Seq[Candidate]] = {\n        transformer\n          .transform(target, candidates)\n          .flatMap { results => ranker.rank(target, results) }\n      }\n    }\n  }\n}\n\nclass IdentityRanker[Target, Candidate] extends Ranker[Target, Candidate] {\n  def rank(target: Target, candidates: Seq[Candidate]): Stitch[Seq[Candidate]] =\n    Stitch.value(candidates)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base/RecommendationFlow.scala",
    "content": "package com.twitter.follow_recommendations.common.base\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.RecommendationPipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.recommendation.RecommendationPipelineResult\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorObserver\nimport com.twitter.stitch.Stitch\n\n/**\n * configs for results generated from the recommendation flow\n *\n * @param desiredCandidateCount num of desired candidates to return\n * @param batchForCandidatesCheck batch size for candidates check\n */\ncase class RecommendationResultsConfig(desiredCandidateCount: Int, batchForCandidatesCheck: Int)\n\ntrait BaseRecommendationFlow[Target, Candidate <: UniversalNoun[Long]] {\n  val identifier = RecommendationPipelineIdentifier(\"RecommendationFlow\")\n\n  def process(\n    pipelineRequest: Target\n  ): Stitch[RecommendationPipelineResult[Candidate, Seq[Candidate]]]\n\n  def mapKey[Target2](fn: Target2 => Target): BaseRecommendationFlow[Target2, Candidate] = {\n    val original = this\n    new BaseRecommendationFlow[Target2, Candidate] {\n      override def process(\n        pipelineRequest: Target2\n      ): Stitch[RecommendationPipelineResult[Candidate, Seq[Candidate]]] =\n        original.process(fn(pipelineRequest))\n    }\n  }\n}\n\n/**\n * Defines a typical recommendation flow to fetch, filter, rank and transform candidates.\n *\n * 1. targetEligibility: determine the eligibility of target request\n * 2. candidateSources: fetch candidates from candidate sources based on target type\n * 3. preRankerCandidateFilter: light filtering of candidates\n * 4. ranker: ranking of candidates (could be composed of multiple stages, light ranking, heavy ranking and etc)\n * 5. postRankerTransform: deduping, grouping, rule based promotion / demotions and etc\n * 6. validateCandidates: heavy filters to determine the eligibility of the candidates.\n *    will only be applied to candidates that we expect to return.\n * 7. transformResults: transform the individual candidates into desired format (e.g. hydrate social proof)\n *\n * Note that the actual implementations may not need to implement all the steps if not needed\n * (could just leave to IdentityRanker if ranking is not needed).\n *\n * Theoretically, the actual implementation could override the above flow to add\n * more steps (e.g. add a transform step before ranking).\n * But it is recommended to add the additional steps into this base flow if the step proves\n * to have significant justification, or merge it into an existing step if it is a minor change.\n *\n * @tparam Target type of target request\n * @tparam Candidate type of candidate to return\n */\ntrait RecommendationFlow[Target, Candidate <: UniversalNoun[Long]]\n    extends BaseRecommendationFlow[Target, Candidate]\n    with SideEffectsUtil[Target, Candidate] {\n\n  /**\n   * optionally update or enrich the request before executing the flows\n   */\n  protected def updateTarget(target: Target): Stitch[Target] = Stitch.value(target)\n\n  /**\n   *  check if the target is eligible for the flow\n   */\n  protected def targetEligibility: Predicate[Target]\n\n  /**\n   *  define the candidate sources that should be used for the given target\n   */\n  protected def candidateSources(target: Target): Seq[CandidateSource[Target, Candidate]]\n\n  /**\n   *  filter invalid candidates before the ranking phase.\n   */\n  protected def preRankerCandidateFilter: Predicate[(Target, Candidate)]\n\n  /**\n   * rank the candidates\n   */\n  protected def selectRanker(target: Target): Ranker[Target, Candidate]\n\n  /**\n   * transform the candidates after ranking (e.g. dedupping, grouping and etc)\n   */\n  protected def postRankerTransform: Transform[Target, Candidate]\n\n  /**\n   *  filter invalid candidates before returning the results.\n   *\n   *  Some heavy filters e.g. SGS filter could be applied in this step\n   */\n  protected def validateCandidates: Predicate[(Target, Candidate)]\n\n  /**\n   * transform the candidates into results and return\n   */\n  protected def transformResults: Transform[Target, Candidate]\n\n  /**\n   *  configuration for recommendation results\n   */\n  protected def resultsConfig(target: Target): RecommendationResultsConfig\n\n  /**\n   * track the quality factor the recommendation pipeline\n   */\n  protected def qualityFactorObserver: Option[QualityFactorObserver] = None\n\n  def statsReceiver: StatsReceiver\n\n  /**\n   * high level monitoring for the whole flow\n   * (make sure to add monitoring for each individual component by yourself)\n   *\n   * additional candidates: count, stats, non_empty_count\n   * target eligibility: latency, success, failures, request, count, valid_count, invalid_count, invalid_reasons\n   * candidate generation: latency, success, failures, request, count, non_empty_count, results_stat\n   * pre ranker filter: latency, success, failures, request, count, non_empty_count, results_stat\n   * ranker: latency, success, failures, request, count, non_empty_count, results_stat\n   * post ranker: latency, success, failures, request, count, non_empty_count, results_stat\n   * filter and take: latency, success, failures, request, count, non_empty_count, results_stat, batch count\n   * transform results: latency, success, failures, request, count, non_empty_count, results_stat\n   */\n  import RecommendationFlow._\n  lazy val additionalCandidatesStats = statsReceiver.scope(AdditionalCandidatesStats)\n  lazy val targetEligibilityStats = statsReceiver.scope(TargetEligibilityStats)\n  lazy val candidateGenerationStats = statsReceiver.scope(CandidateGenerationStats)\n  lazy val preRankerFilterStats = statsReceiver.scope(PreRankerFilterStats)\n  lazy val rankerStats = statsReceiver.scope(RankerStats)\n  lazy val postRankerTransformStats = statsReceiver.scope(PostRankerTransformStats)\n  lazy val filterAndTakeStats = statsReceiver.scope(FilterAndTakeStats)\n  lazy val transformResultsStats = statsReceiver.scope(TransformResultsStats)\n\n  lazy val overallStats = statsReceiver.scope(OverallStats)\n\n  import StatsUtil._\n\n  override def process(\n    pipelineRequest: Target\n  ): Stitch[RecommendationPipelineResult[Candidate, Seq[Candidate]]] = {\n\n    observeStitchQualityFactor(\n      profileStitchSeqResults(\n        updateTarget(pipelineRequest).flatMap { target =>\n          profilePredicateResult(targetEligibility(target), targetEligibilityStats).flatMap {\n            case PredicateResult.Valid => processValidTarget(target, Seq.empty)\n            case PredicateResult.Invalid(_) => Stitch.Nil\n          }\n        },\n        overallStats\n      ).map { candidates =>\n        RecommendationPipelineResult.empty.withResult(candidates)\n      },\n      qualityFactorObserver,\n      overallStats\n    )\n  }\n\n  protected def processValidTarget(\n    target: Target,\n    additionalCandidates: Seq[Candidate]\n  ): Stitch[Seq[Candidate]] = {\n\n    /**\n     * A basic recommendation flow looks like this:\n     *\n     * 1. fetch candidates from candidate sources\n     * 2. blend candidates with existing candidates\n     * 3. filter the candidates (light filters) before ranking\n     * 4. ranking\n     * 5. filter and truncate the candidates using postRankerCandidateFilter\n     * 6. transform the candidates based on product requirement\n     */\n    val candidateSourcesToFetch = candidateSources(target)\n    for {\n      candidates <- profileStitchSeqResults(\n        Stitch.traverse(candidateSourcesToFetch)(_(target)).map(_.flatten),\n        candidateGenerationStats\n      )\n      mergedCandidates =\n        profileSeqResults(additionalCandidates, additionalCandidatesStats) ++\n          candidates\n      filteredCandidates <- profileStitchSeqResults(\n        Predicate.filter(target, mergedCandidates, preRankerCandidateFilter),\n        preRankerFilterStats\n      )\n      rankedCandidates <- profileStitchSeqResults(\n        selectRanker(target).rank(target, filteredCandidates),\n        rankerStats\n      )\n      transformed <- profileStitchSeqResults(\n        postRankerTransform.transform(target, rankedCandidates),\n        postRankerTransformStats\n      )\n      truncated <- profileStitchSeqResults(\n        take(target, transformed, resultsConfig(target)),\n        filterAndTakeStats\n      )\n      results <- profileStitchSeqResults(\n        transformResults.transform(target, truncated),\n        transformResultsStats\n      )\n      _ <- applySideEffects(\n        target,\n        candidateSourcesToFetch,\n        candidates,\n        mergedCandidates,\n        filteredCandidates,\n        rankedCandidates,\n        transformed,\n        truncated,\n        results)\n    } yield results\n  }\n\n  private[this] def take(\n    target: Target,\n    candidates: Seq[Candidate],\n    config: RecommendationResultsConfig\n  ): Stitch[Seq[Candidate]] = {\n    Predicate\n      .batchFilterTake(\n        candidates.map(c => (target, c)),\n        validateCandidates,\n        config.batchForCandidatesCheck,\n        config.desiredCandidateCount,\n        statsReceiver\n      ).map(_.map(_._2))\n  }\n}\n\nobject RecommendationFlow {\n\n  val AdditionalCandidatesStats = \"additional_candidates\"\n  val TargetEligibilityStats = \"target_eligibility\"\n  val CandidateGenerationStats = \"candidate_generation\"\n  val PreRankerFilterStats = \"pre_ranker_filter\"\n  val RankerStats = \"ranker\"\n  val PostRankerTransformStats = \"post_ranker_transform\"\n  val FilterAndTakeStats = \"filter_and_take\"\n  val TransformResultsStats = \"transform_results\"\n  val OverallStats = \"overall\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base/SideEffectsUtil.scala",
    "content": "package com.twitter.follow_recommendations.common.base\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.stitch.Stitch\n\n/**\n * SideEffectsUtil applies side effects to the intermediate candidate results from a recommendation flow pipeline.\n *\n * @tparam Target target to recommend the candidates\n * @tparam Candidate candidate type to rank\n */\ntrait SideEffectsUtil[Target, Candidate] {\n  def applySideEffects(\n    target: Target,\n    candidateSources: Seq[CandidateSource[Target, Candidate]],\n    candidatesFromCandidateSources: Seq[Candidate],\n    mergedCandidates: Seq[Candidate],\n    filteredCandidates: Seq[Candidate],\n    rankedCandidates: Seq[Candidate],\n    transformedCandidates: Seq[Candidate],\n    truncatedCandidates: Seq[Candidate],\n    results: Seq[Candidate]\n  ): Stitch[Unit] = Stitch.Unit\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base/StatsUtil.scala",
    "content": "package com.twitter.follow_recommendations.common.base\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorObserver\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Stopwatch\nimport java.util.concurrent.TimeUnit\nimport scala.util.control.NonFatal\n\nobject StatsUtil {\n  val LatencyName = \"latency_ms\"\n  val RequestName = \"requests\"\n  val SuccessName = \"success\"\n  val FailureName = \"failures\"\n  val ResultsName = \"results\"\n  val ResultsStat = \"results_stat\"\n  val EmptyResultsName = \"empty\"\n  val NonEmptyResultsName = \"non_empty\"\n  val ValidCount = \"valid\"\n  val InvalidCount = \"invalid\"\n  val InvalidHasReasons = \"has_reasons\"\n  val Reasons = \"reasons\"\n  val QualityFactorStat = \"quality_factor_stat\"\n  val QualityFactorCounts = \"quality_factor_counts\"\n\n  /**\n   * Helper function for timing a stitch, returning the original stitch.\n   */\n  def profileStitch[T](stitch: Stitch[T], stat: StatsReceiver): Stitch[T] = {\n\n    Stitch\n      .time(stitch)\n      .map {\n        case (response, stitchRunDuration) =>\n          stat.counter(RequestName).incr()\n          stat.stat(LatencyName).add(stitchRunDuration.inMilliseconds)\n          response\n            .onSuccess { _ => stat.counter(SuccessName).incr() }\n            .onFailure { e =>\n              stat.counter(FailureName).incr()\n              stat.scope(FailureName).counter(getCleanClassName(e)).incr()\n            }\n      }\n      .lowerFromTry\n  }\n\n  /**\n   * Helper function for timing an arrow, returning the original arrow.\n   */\n  def profileArrow[T, U](arrow: Arrow[T, U], stat: StatsReceiver): Arrow[T, U] = {\n\n    Arrow\n      .time(arrow)\n      .map {\n        case (response, stitchRunDuration) =>\n          stat.counter(RequestName).incr()\n          stat.stat(LatencyName).add(stitchRunDuration.inMilliseconds)\n          response\n            .onSuccess { _ => stat.counter(SuccessName).incr() }\n            .onFailure { e =>\n              stat.counter(FailureName).incr()\n              stat.scope(FailureName).counter(getCleanClassName(e)).incr()\n            }\n      }\n      .lowerFromTry\n  }\n\n  /**\n   * Helper function to count and track the distribution of results\n   */\n  def profileResults[T](results: T, stat: StatsReceiver, size: T => Int): T = {\n    val numResults = size(results)\n    stat.counter(ResultsName).incr(numResults)\n    if (numResults == 0) {\n      stat.counter(EmptyResultsName).incr()\n      results\n    } else {\n      stat.stat(ResultsStat).add(numResults)\n      stat.counter(NonEmptyResultsName).incr()\n      results\n    }\n  }\n\n  /**\n   * Helper function to count and track the distribution of a list of results\n   */\n  def profileSeqResults[T](results: Seq[T], stat: StatsReceiver): Seq[T] = {\n    profileResults[Seq[T]](results, stat, _.size)\n  }\n\n  /**\n   * Helper function for timing a stitch and count the number of results, returning the original stitch.\n   */\n  def profileStitchResults[T](stitch: Stitch[T], stat: StatsReceiver, size: T => Int): Stitch[T] = {\n    profileStitch(stitch, stat).onSuccess { results => profileResults(results, stat, size) }\n  }\n\n  /**\n   * Helper function for timing an arrow and count the number of results, returning the original arrow.\n   */\n  def profileArrowResults[T, U](\n    arrow: Arrow[T, U],\n    stat: StatsReceiver,\n    size: U => Int\n  ): Arrow[T, U] = {\n    profileArrow(arrow, stat).onSuccess { results => profileResults(results, stat, size) }\n  }\n\n  /**\n   * Helper function for timing a stitch and count a seq of results, returning the original stitch.\n   */\n  def profileStitchSeqResults[T](stitch: Stitch[Seq[T]], stat: StatsReceiver): Stitch[Seq[T]] = {\n    profileStitchResults[Seq[T]](stitch, stat, _.size)\n  }\n\n  /**\n   * Helper function for timing a stitch and count optional results, returning the original stitch.\n   */\n  def profileStitchOptionalResults[T](\n    stitch: Stitch[Option[T]],\n    stat: StatsReceiver\n  ): Stitch[Option[T]] = {\n    profileStitchResults[Option[T]](stitch, stat, _.size)\n  }\n\n  /**\n   * Helper function for timing a stitch and count a map of results, returning the original stitch.\n   */\n  def profileStitchMapResults[K, V](\n    stitch: Stitch[Map[K, V]],\n    stat: StatsReceiver\n  ): Stitch[Map[K, V]] = {\n    profileStitchResults[Map[K, V]](stitch, stat, _.size)\n  }\n\n  def getCleanClassName(obj: Object): String =\n    obj.getClass.getSimpleName.stripSuffix(\"$\")\n\n  /**\n   * Helper function for timing a stitch and count a list of PredicateResult\n   */\n  def profilePredicateResults(\n    predicateResult: Stitch[Seq[PredicateResult]],\n    statsReceiver: StatsReceiver\n  ): Stitch[Seq[PredicateResult]] = {\n    profileStitch[Seq[PredicateResult]](\n      predicateResult,\n      statsReceiver\n    ).onSuccess {\n      _.map {\n        case PredicateResult.Valid =>\n          statsReceiver.counter(ValidCount).incr()\n        case PredicateResult.Invalid(reasons) =>\n          statsReceiver.counter(InvalidCount).incr()\n          reasons.map { filterReason =>\n            statsReceiver.counter(InvalidHasReasons).incr()\n            statsReceiver.scope(Reasons).counter(filterReason.reason).incr()\n          }\n      }\n    }\n  }\n\n  /**\n   * Helper function for timing a stitch and count individual PredicateResult\n   */\n  def profilePredicateResult(\n    predicateResult: Stitch[PredicateResult],\n    statsReceiver: StatsReceiver\n  ): Stitch[PredicateResult] = {\n    profilePredicateResults(\n      predicateResult.map(Seq(_)),\n      statsReceiver\n    ).map(_.head)\n  }\n\n  /**\n   * Helper function for timing an arrow and count a list of PredicateResult\n   */\n  def profilePredicateResults[Q](\n    predicateResult: Arrow[Q, Seq[PredicateResult]],\n    statsReceiver: StatsReceiver\n  ): Arrow[Q, Seq[PredicateResult]] = {\n    profileArrow[Q, Seq[PredicateResult]](\n      predicateResult,\n      statsReceiver\n    ).onSuccess {\n      _.map {\n        case PredicateResult.Valid =>\n          statsReceiver.counter(ValidCount).incr()\n        case PredicateResult.Invalid(reasons) =>\n          statsReceiver.counter(InvalidCount).incr()\n          reasons.map { filterReason =>\n            statsReceiver.counter(InvalidHasReasons).incr()\n            statsReceiver.scope(Reasons).counter(filterReason.reason).incr()\n          }\n      }\n    }\n  }\n\n  /**\n   * Helper function for timing an arrow and count individual PredicateResult\n   */\n  def profilePredicateResult[Q](\n    predicateResult: Arrow[Q, PredicateResult],\n    statsReceiver: StatsReceiver\n  ): Arrow[Q, PredicateResult] = {\n    profilePredicateResults(\n      predicateResult.map(Seq(_)),\n      statsReceiver\n    ).map(_.head)\n  }\n\n  /**\n   * Helper function for timing a stitch code block\n   */\n  def profileStitchSeqResults[T](\n    stats: StatsReceiver\n  )(\n    block: => Stitch[Seq[T]]\n  ): Stitch[Seq[T]] = {\n    stats.counter(RequestName).incr()\n    profileStitch(stats.stat(LatencyName), TimeUnit.MILLISECONDS) {\n      block onSuccess { r =>\n        if (r.isEmpty) stats.counter(EmptyResultsName).incr()\n        stats.stat(ResultsStat).add(r.size)\n      } onFailure { e =>\n        {\n          stats.counter(FailureName).incr()\n          stats.scope(FailureName).counter(e.getClass.getName).incr()\n        }\n      }\n    }\n  }\n\n  /**\n   * Time a given asynchronous `f` using the given `unit`.\n   */\n  def profileStitch[A](stat: Stat, unit: TimeUnit)(f: => Stitch[A]): Stitch[A] = {\n    val start = Stopwatch.timeNanos()\n    try {\n      f.respond { _ => stat.add(unit.convert(Stopwatch.timeNanos() - start, TimeUnit.NANOSECONDS)) }\n    } catch {\n      case NonFatal(e) =>\n        stat.add(unit.convert(Stopwatch.timeNanos() - start, TimeUnit.NANOSECONDS))\n        Stitch.exception(e)\n    }\n  }\n\n  def observeStitchQualityFactor[T](\n    stitch: Stitch[T],\n    qualityFactorObserverOption: Option[QualityFactorObserver],\n    statsReceiver: StatsReceiver\n  ): Stitch[T] = {\n    qualityFactorObserverOption\n      .map { observer =>\n        Stitch\n          .time(stitch)\n          .map {\n            case (response, stitchRunDuration) =>\n              observer(response, stitchRunDuration)\n              val qfVal = observer.qualityFactor.currentValue.floatValue() * 10000\n              statsReceiver.counter(QualityFactorCounts).incr()\n              statsReceiver\n                .stat(QualityFactorStat)\n                .add(qfVal)\n              response\n          }\n          .lowerFromTry\n      }.getOrElse(stitch)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base/Transform.scala",
    "content": "package com.twitter.follow_recommendations.common.base\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Param\n\n/**\n * transform a or a list of candidate for target T\n *\n * @tparam T target type\n * @tparam C candidate type\n */\ntrait Transform[-T, C] {\n\n  // you need to implement at least one of the two methods here.\n  def transformItem(target: T, item: C): Stitch[C] = {\n    transform(target, Seq(item)).map(_.head)\n  }\n\n  def transform(target: T, items: Seq[C]): Stitch[Seq[C]]\n\n  def mapTarget[T2](mapper: T2 => T): Transform[T2, C] = {\n    val original = this\n    new Transform[T2, C] {\n      override def transformItem(target: T2, item: C): Stitch[C] = {\n        original.transformItem(mapper(target), item)\n      }\n      override def transform(target: T2, items: Seq[C]): Stitch[Seq[C]] = {\n        original.transform(mapper(target), items)\n      }\n    }\n  }\n\n  /**\n   * sequential composition. we execute this' transform first, followed by the other's transform\n   */\n  def andThen[T1 <: T](other: Transform[T1, C]): Transform[T1, C] = {\n    val original = this\n    new Transform[T1, C] {\n      override def transformItem(target: T1, item: C): Stitch[C] =\n        original.transformItem(target, item).flatMap(other.transformItem(target, _))\n      override def transform(target: T1, items: Seq[C]): Stitch[Seq[C]] =\n        original.transform(target, items).flatMap(other.transform(target, _))\n    }\n  }\n\n  def observe(statsReceiver: StatsReceiver): Transform[T, C] = {\n    val originalTransform = this\n    new Transform[T, C] {\n      override def transform(target: T, items: Seq[C]): Stitch[Seq[C]] = {\n        statsReceiver.counter(Transform.InputCandidatesCount).incr(items.size)\n        statsReceiver.stat(Transform.InputCandidatesStat).add(items.size)\n        StatsUtil.profileStitchSeqResults(originalTransform.transform(target, items), statsReceiver)\n      }\n\n      override def transformItem(target: T, item: C): Stitch[C] = {\n        statsReceiver.counter(Transform.InputCandidatesCount).incr()\n        StatsUtil.profileStitch(originalTransform.transformItem(target, item), statsReceiver)\n      }\n    }\n  }\n}\n\ntrait GatedTransform[T <: HasParams, C] extends Transform[T, C] {\n  def gated(param: Param[Boolean]): Transform[T, C] = {\n    val original = this\n    (target: T, items: Seq[C]) => {\n      if (target.params(param)) {\n        original.transform(target, items)\n      } else {\n        Stitch.value(items)\n      }\n    }\n  }\n}\n\nobject Transform {\n  val InputCandidatesCount = \"input_candidates\"\n  val InputCandidatesStat = \"input_candidates_stat\"\n}\n\nclass IdentityTransform[T, C] extends Transform[T, C] {\n  override def transform(target: T, items: Seq[C]): Stitch[Seq[C]] = Stitch.value(items)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/addressbook/AddressBookParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.addressbook\n\nimport com.twitter.timelines.configapi.FSParam\n\nobject AddressBookParams {\n  // Used by display locations that want only to read from the ABV2 Client and ignore Manhattan\n  // Currently the only display location that does this is the ABUploadInjection DisplayLocation\n  object ReadFromABV2Only extends FSParam[Boolean](\"addressbook_read_only_from_abv2\", false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/addressbook/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/addressbook\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/email_storage_service\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/gizmoduck\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/phone_storage_service\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/deciders\",\n        \"src/thrift/com/twitter/hermit/candidate:hermit-candidate-scala\",\n        \"src/thrift/com/twitter/hermit/usercontacts:hermit-usercontacts-scala\",\n        \"strato/config/columns/onboarding/userrecs:userrecs-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/addressbook/ForwardEmailBookSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.addressbook\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.AddressBookParams.ReadFromABV2Only\nimport com.twitter.follow_recommendations.common.clients.addressbook.AddressbookClient\nimport com.twitter.follow_recommendations.common.clients.addressbook.models.EdgeType\nimport com.twitter.follow_recommendations.common.clients.addressbook.models.RecordIdentifier\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.utils.RescueWithStatsUtils.rescueWithStats\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.userrecs.ForwardEmailBookClientColumn\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForwardEmailBookSource @Inject() (\n  forwardEmailBookClientColumn: ForwardEmailBookClientColumn,\n  addressBookClient: AddressbookClient,\n  statsReceiver: StatsReceiver = NullStatsReceiver)\n    extends CandidateSource[HasParams with HasClientContext, CandidateUser] {\n\n  override val identifier: CandidateSourceIdentifier =\n    ForwardEmailBookSource.Identifier\n  private val stats: StatsReceiver = statsReceiver.scope(this.getClass.getSimpleName)\n\n  /**\n   * Generate a list of candidates for the target\n   */\n  override def apply(\n    target: HasParams with HasClientContext\n  ): Stitch[Seq[CandidateUser]] = {\n    val candidateUsers: Stitch[Seq[Long]] = target.getOptionalUserId\n      .map { userId =>\n        rescueWithStats(\n          addressBookClient.getUsers(\n            userId = userId,\n            identifiers =\n              Seq(RecordIdentifier(userId = Some(userId), email = None, phoneNumber = None)),\n            batchSize = AddressbookClient.AddressBook2BatchSize,\n            edgeType = ForwardEmailBookSource.DefaultEdgeType,\n            fetcherOption =\n              if (target.params.apply(ReadFromABV2Only)) None\n              else Some(forwardEmailBookClientColumn.fetcher),\n            queryOption = AddressbookClient\n              .createQueryOption(\n                edgeType = ForwardEmailBookSource.DefaultEdgeType,\n                isPhone = ForwardEmailBookSource.IsPhone)\n          ),\n          stats,\n          \"AddressBookClient\"\n        )\n      }.getOrElse(Stitch.Nil)\n\n    candidateUsers\n      .map(\n        _.take(ForwardEmailBookSource.NumEmailBookEntries)\n          .map(CandidateUser(_, score = Some(CandidateUser.DefaultCandidateScore))\n            .withCandidateSource(identifier)))\n  }\n}\n\nobject ForwardEmailBookSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.ForwardEmailBook.toString)\n  val NumEmailBookEntries: Int = 1000\n  val IsPhone = false\n  val DefaultEdgeType: EdgeType = EdgeType.Forward\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/addressbook/ForwardPhoneBookSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.addressbook\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.AddressBookParams.ReadFromABV2Only\nimport com.twitter.follow_recommendations.common.clients.addressbook.AddressbookClient\nimport com.twitter.follow_recommendations.common.clients.addressbook.models.EdgeType\nimport com.twitter.follow_recommendations.common.clients.addressbook.models.RecordIdentifier\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.utils.RescueWithStatsUtils.rescueWithStats\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.userrecs.ForwardPhoneContactsClientColumn\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForwardPhoneBookSource @Inject() (\n  forwardPhoneContactsClientColumn: ForwardPhoneContactsClientColumn,\n  addressBookClient: AddressbookClient,\n  statsReceiver: StatsReceiver = NullStatsReceiver)\n    extends CandidateSource[HasParams with HasClientContext, CandidateUser] {\n\n  override val identifier: CandidateSourceIdentifier =\n    ForwardPhoneBookSource.Identifier\n  private val stats: StatsReceiver = statsReceiver.scope(this.getClass.getSimpleName)\n\n  /**\n   * Generate a list of candidates for the target\n   */\n  override def apply(target: HasParams with HasClientContext): Stitch[Seq[CandidateUser]] = {\n    val candidateUsers: Stitch[Seq[Long]] = target.getOptionalUserId\n      .map { userId =>\n        rescueWithStats(\n          addressBookClient.getUsers(\n            userId,\n            identifiers =\n              Seq(RecordIdentifier(userId = Some(userId), email = None, phoneNumber = None)),\n            batchSize = AddressbookClient.AddressBook2BatchSize,\n            edgeType = ForwardPhoneBookSource.DefaultEdgeType,\n            fetcherOption =\n              if (target.params.apply(ReadFromABV2Only)) None\n              else Some(forwardPhoneContactsClientColumn.fetcher),\n            queryOption = AddressbookClient\n              .createQueryOption(\n                edgeType = ForwardPhoneBookSource.DefaultEdgeType,\n                isPhone = ForwardPhoneBookSource.IsPhone)\n          ),\n          stats,\n          \"AddressBookClient\"\n        )\n      }.getOrElse(Stitch.Nil)\n\n    candidateUsers\n      .map(\n        _.take(ForwardPhoneBookSource.NumPhoneBookEntries)\n          .map(CandidateUser(_, score = Some(CandidateUser.DefaultCandidateScore))\n            .withCandidateSource(identifier)))\n  }\n}\n\nobject ForwardPhoneBookSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.ForwardPhoneBook.toString)\n  val NumPhoneBookEntries: Int = 1000\n  val IsPhone = true\n  val DefaultEdgeType: EdgeType = EdgeType.Forward\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/addressbook/README.md",
    "content": "# Address Book Candidate Source\nProvides the accounts of a given user's forward and reverse phone and email book contacts.\nIt is only available when the user has synced their address book with the service.\n\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/addressbook/ReverseEmailBookSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.addressbook\n\nimport com.twitter.cds.contact_consent_state.thriftscala.PurposeOfProcessing\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.clients.addressbook.AddressbookClient\nimport com.twitter.follow_recommendations.common.clients.addressbook.models.EdgeType\nimport com.twitter.follow_recommendations.common.clients.addressbook.models.RecordIdentifier\nimport com.twitter.follow_recommendations.common.clients.email_storage_service.EmailStorageServiceClient\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.utils.RescueWithStatsUtils.rescueOptionalWithStats\nimport com.twitter.follow_recommendations.common.utils.RescueWithStatsUtils.rescueWithStats\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.userrecs.ReverseEmailContactsClientColumn\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ReverseEmailBookSource @Inject() (\n  reverseEmailContactsClientColumn: ReverseEmailContactsClientColumn,\n  essClient: EmailStorageServiceClient,\n  addressBookClient: AddressbookClient,\n  statsReceiver: StatsReceiver = NullStatsReceiver)\n    extends CandidateSource[HasParams with HasClientContext, CandidateUser] {\n  override val identifier: CandidateSourceIdentifier = ReverseEmailBookSource.Identifier\n  private val rescueStats = statsReceiver.scope(\"ReverseEmailBookSource\")\n\n  /**\n   * Generate a list of candidates for the target\n   */\n  override def apply(target: HasParams with HasClientContext): Stitch[Seq[CandidateUser]] = {\n    val reverseCandidatesFromEmail = target.getOptionalUserId\n      .map { userId =>\n        val verifiedEmailStitchOpt =\n          rescueOptionalWithStats(\n            essClient.getVerifiedEmail(userId, PurposeOfProcessing.ContentRecommendations),\n            rescueStats,\n            \"getVerifiedEmail\")\n        verifiedEmailStitchOpt.flatMap { emailOpt =>\n          rescueWithStats(\n            addressBookClient.getUsers(\n              userId = userId,\n              identifiers = emailOpt\n                .map(email =>\n                  RecordIdentifier(userId = None, email = Some(email), phoneNumber = None)).toSeq,\n              batchSize = ReverseEmailBookSource.NumEmailBookEntries,\n              edgeType = ReverseEmailBookSource.DefaultEdgeType,\n              fetcherOption =\n                if (target.params(AddressBookParams.ReadFromABV2Only)) None\n                else Some(reverseEmailContactsClientColumn.fetcher)\n            ),\n            rescueStats,\n            \"AddressBookClient\"\n          )\n        }\n      }.getOrElse(Stitch.Nil)\n\n    reverseCandidatesFromEmail.map(\n      _.take(ReverseEmailBookSource.NumEmailBookEntries)\n        .map(\n          CandidateUser(_, score = Some(CandidateUser.DefaultCandidateScore))\n            .withCandidateSource(identifier))\n    )\n  }\n}\n\nobject ReverseEmailBookSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.ReverseEmailBookIbis.toString)\n  val NumEmailBookEntries: Int = 500\n  val IsPhone = false\n  val DefaultEdgeType: EdgeType = EdgeType.Reverse\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/addressbook/ReversePhoneBookSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.addressbook\n\nimport com.twitter.cds.contact_consent_state.thriftscala.PurposeOfProcessing\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.clients.addressbook.AddressbookClient\nimport com.twitter.follow_recommendations.common.clients.addressbook.models.EdgeType\nimport com.twitter.follow_recommendations.common.clients.addressbook.models.RecordIdentifier\nimport com.twitter.follow_recommendations.common.clients.phone_storage_service.PhoneStorageServiceClient\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.utils.RescueWithStatsUtils.rescueWithStats\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.userrecs.ReversePhoneContactsClientColumn\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ReversePhoneBookSource @Inject() (\n  reversePhoneContactsClientColumn: ReversePhoneContactsClientColumn,\n  pssClient: PhoneStorageServiceClient,\n  addressBookClient: AddressbookClient,\n  statsReceiver: StatsReceiver = NullStatsReceiver)\n    extends CandidateSource[HasParams with HasClientContext, CandidateUser] {\n\n  override val identifier: CandidateSourceIdentifier = ReversePhoneBookSource.Identifier\n  private val stats: StatsReceiver = statsReceiver.scope(this.getClass.getSimpleName)\n\n  /**\n   * Generate a list of candidates for the target\n   */\n  override def apply(target: HasParams with HasClientContext): Stitch[Seq[CandidateUser]] = {\n    val reverseCandidatesFromPhones: Stitch[Seq[Long]] = target.getOptionalUserId\n      .map { userId =>\n        pssClient\n          .getPhoneNumbers(userId, PurposeOfProcessing.ContentRecommendations)\n          .flatMap { phoneNumbers =>\n            rescueWithStats(\n              addressBookClient.getUsers(\n                userId = userId,\n                identifiers = phoneNumbers.map(phoneNumber =>\n                  RecordIdentifier(userId = None, email = None, phoneNumber = Some(phoneNumber))),\n                batchSize = ReversePhoneBookSource.NumPhoneBookEntries,\n                edgeType = ReversePhoneBookSource.DefaultEdgeType,\n                fetcherOption =\n                  if (target.params(AddressBookParams.ReadFromABV2Only)) None\n                  else Some(reversePhoneContactsClientColumn.fetcher),\n                queryOption = AddressbookClient.createQueryOption(\n                  edgeType = ReversePhoneBookSource.DefaultEdgeType,\n                  isPhone = ReversePhoneBookSource.IsPhone)\n              ),\n              stats,\n              \"AddressBookClient\"\n            )\n          }\n      }.getOrElse(Stitch.Nil)\n\n    reverseCandidatesFromPhones.map(\n      _.take(ReversePhoneBookSource.NumPhoneBookEntries)\n        .map(\n          CandidateUser(_, score = Some(CandidateUser.DefaultCandidateScore))\n            .withCandidateSource(identifier))\n    )\n  }\n}\n\nobject ReversePhoneBookSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.ReversePhoneBook.toString)\n  val NumPhoneBookEntries: Int = 500\n  val IsPhone = true\n  val DefaultEdgeType: EdgeType = EdgeType.Reverse\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"content-recommender/thrift/src/main/thrift:thrift-scala\",\n        \"escherbird/src/scala/com/twitter/escherbird/util/stitchcache\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/modify_social_proof\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"src/scala/com/twitter/onboarding/relevance/features/ymbii\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/CachedCandidateSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.base\n\nimport com.twitter.escherbird.util.stitchcache.StitchCache\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Duration\n\nclass CachedCandidateSource[K <: Object, V <: Object](\n  candidateSource: CandidateSource[K, V],\n  maxCacheSize: Int,\n  cacheTTL: Duration,\n  statsReceiver: StatsReceiver,\n  override val identifier: CandidateSourceIdentifier)\n    extends CandidateSource[K, V] {\n\n  private val cache = StitchCache[K, Seq[V]](\n    maxCacheSize = maxCacheSize,\n    ttl = cacheTTL,\n    statsReceiver = statsReceiver.scope(identifier.name, \"cache\"),\n    underlyingCall = (k: K) => candidateSource(k)\n  )\n\n  override def apply(target: K): Stitch[Seq[V]] = cache.readThrough(target)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/ExperimentalCandidateSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.base\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\n\n/**\n * A wrapper of CandidateSource to make it easier to do experimentation\n * on new candidate generation algorithms\n *\n * @param baseSource base candidate source\n * @param darkreadAlgorithmParam controls whether or not to darkread candidates (fetch them even if they will not be included)\n * @param keepCandidatesParam controls whether or not to keep candidates from the base source\n * @param resultCountThresholdParam controls how many results the source must return to bucket the user and return results (greater-than-or-equal-to)\n * @tparam T request type. it must extend HasParams\n * @tparam V value type\n */\nclass ExperimentalCandidateSource[T <: HasParams, V](\n  baseSource: CandidateSource[T, V],\n  darkreadAlgorithmParam: Param[Boolean],\n  keepCandidatesParam: Param[Boolean],\n  resultCountThresholdParam: Param[Int],\n  baseStatsReceiver: StatsReceiver)\n    extends CandidateSource[T, V] {\n\n  override val identifier: CandidateSourceIdentifier = baseSource.identifier\n  private[base] val statsReceiver =\n    baseStatsReceiver.scope(s\"Experimental/${identifier.name}\")\n  private[base] val requestsCounter = statsReceiver.counter(\"requests\")\n  private[base] val resultCountGreaterThanThresholdCounter =\n    statsReceiver.counter(\"with_results_at_or_above_count_threshold\")\n  private[base] val keepResultsCounter = statsReceiver.counter(\"keep_results\")\n  private[base] val discardResultsCounter = statsReceiver.counter(\"discard_results\")\n\n  override def apply(request: T): Stitch[Seq[V]] = {\n    if (request.params(darkreadAlgorithmParam)) {\n      requestsCounter.incr()\n      fetchFromCandidateSourceAndProcessResults(request)\n    } else {\n      Stitch.Nil\n    }\n  }\n\n  private def fetchFromCandidateSourceAndProcessResults(request: T): Stitch[Seq[V]] = {\n    baseSource(request).map { results =>\n      if (results.length >= request.params(resultCountThresholdParam)) {\n        processResults(results, request.params(keepCandidatesParam))\n      } else {\n        Nil\n      }\n    }\n  }\n\n  private def processResults(results: Seq[V], keepResults: Boolean): Seq[V] = {\n    resultCountGreaterThanThresholdCounter.incr()\n    if (keepResults) {\n      keepResultsCounter.incr()\n      results\n    } else {\n      discardResultsCounter.incr()\n      Nil\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/RealGraphExpansionRepository.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.base\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.follow_recommendations.common.candidate_sources.base.RealGraphExpansionRepository.DefaultScore\nimport com.twitter.follow_recommendations.common.candidate_sources.base.RealGraphExpansionRepository.MaxNumIntermediateNodesToKeep\nimport com.twitter.follow_recommendations.common.candidate_sources.base.RealGraphExpansionRepository.FirstDegreeCandidatesTimeout\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models._\nimport com.twitter.onboarding.relevance.features.ymbii.ExpansionCandidateScores\nimport com.twitter.onboarding.relevance.features.ymbii.RawYMBIICandidateFeatures\nimport com.twitter.onboarding.relevance.store.thriftscala.CandidatesFollowedV1\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.util.Duration\nimport scala.collection.immutable\nimport scala.util.control.NonFatal\n\nprivate final case class InterestExpansionCandidate(\n  userID: Long,\n  score: Double,\n  features: RawYMBIICandidateFeatures)\n\nabstract class RealGraphExpansionRepository[Request](\n  realgraphExpansionStore: Fetcher[\n    Long,\n    Unit,\n    CandidatesFollowedV1\n  ],\n  override val identifier: CandidateSourceIdentifier,\n  statsReceiver: StatsReceiver = NullStatsReceiver,\n  maxUnderlyingCandidatesToQuery: Int = 50,\n  maxCandidatesToReturn: Int = 40,\n  overrideUnderlyingTimeout: Option[Duration] = None,\n  appendSocialProof: Boolean = false)\n    extends CandidateSource[\n      Request,\n      CandidateUser\n    ] {\n\n  val underlyingCandidateSource: Seq[\n    CandidateSource[\n      Request,\n      CandidateUser\n    ]\n  ]\n\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName).scope(identifier.name)\n  private val underlyingCandidateSourceFailureStats =\n    stats.scope(\"underlying_candidate_source_failure\")\n\n  def apply(\n    request: Request,\n  ): Stitch[Seq[CandidateUser]] = {\n\n    val candidatesFromUnderlyingSourcesStitch: Seq[Stitch[Seq[CandidateUser]]] =\n      underlyingCandidateSource.map { candidateSource =>\n        candidateSource\n          .apply(request)\n          .within(overrideUnderlyingTimeout.getOrElse(FirstDegreeCandidatesTimeout))(\n            DefaultTimer\n          )\n          .handle {\n            case NonFatal(e) =>\n              underlyingCandidateSourceFailureStats\n                .counter(candidateSource.identifier.name, e.getClass.getSimpleName).incr()\n              Seq.empty\n          }\n      }\n\n    for {\n      underlyingCandidatesFromEachAlgo <- Stitch.collect(candidatesFromUnderlyingSourcesStitch)\n      // The first algorithm in the list has the highest priority. Depending on if its not\n      // populated, fall back to other algorithms. Once a particular algorithm is chosen, only\n      // take the top few candidates from the underlying store for expansion.\n      underlyingCandidatesTuple =\n        underlyingCandidatesFromEachAlgo\n          .zip(underlyingCandidateSource)\n          .find(_._1.nonEmpty)\n\n      underlyingAlgorithmUsed: Option[CandidateSourceIdentifier] = underlyingCandidatesTuple.map {\n        case (_, candidateSource) => candidateSource.identifier\n      }\n\n      // Take maxUnderlyingCandidatesToQuery to query realgraphExpansionStore\n      underlyingCandidates =\n        underlyingCandidatesTuple\n          .map {\n            case (candidates, candidateSource) =>\n              stats\n                .scope(\"underlyingAlgorithmUsedScope\").counter(\n                  candidateSource.identifier.name).incr()\n              candidates\n          }\n          .getOrElse(Seq.empty)\n          .sortBy(_.score.getOrElse(DefaultScore))(Ordering.Double.reverse)\n          .take(maxUnderlyingCandidatesToQuery)\n\n      underlyingCandidateMap: Map[Long, Double] = underlyingCandidates.map { candidate =>\n        (candidate.id, candidate.score.getOrElse(DefaultScore))\n      }.toMap\n\n      expansionCandidates <-\n        Stitch\n          .traverse(underlyingCandidateMap.keySet.toSeq) { candidateId =>\n            Stitch.join(\n              Stitch.value(candidateId),\n              realgraphExpansionStore.fetch(candidateId).map(_.v))\n\n          }.map(_.toMap)\n\n      rerankedCandidates: Seq[InterestExpansionCandidate] =\n        rerankCandidateExpansions(underlyingCandidateMap, expansionCandidates)\n\n      rerankedCandidatesFiltered = rerankedCandidates.take(maxCandidatesToReturn)\n\n    } yield {\n      rerankedCandidatesFiltered.map { candidate =>\n        val socialProofReason = if (appendSocialProof) {\n          val socialProofIds = candidate.features.expansionCandidateScores\n            .map(_.intermediateCandidateId)\n          Some(\n            Reason(Some(\n              AccountProof(followProof = Some(FollowProof(socialProofIds, socialProofIds.size))))))\n        } else {\n          None\n        }\n        CandidateUser(\n          id = candidate.userID,\n          score = Some(candidate.score),\n          reason = socialProofReason,\n          userCandidateSourceDetails = Some(\n            UserCandidateSourceDetails(\n              primaryCandidateSource = Some(identifier),\n              candidateSourceFeatures = Map(identifier -> Seq(candidate.features))\n            ))\n        ).addAddressBookMetadataIfAvailable(underlyingAlgorithmUsed.toSeq)\n      }\n    }\n  }\n\n  /**\n   * Expands underlying candidates, returning them in sorted order.\n   *\n   * @param underlyingCandidatesMap A map from underlying candidate id to score\n   * @param expansionCandidateMap A map from underlying candidate id to optional expansion candidates\n   * @return A sorted sequence of expansion candidates and associated scores\n   */\n  private def rerankCandidateExpansions(\n    underlyingCandidatesMap: Map[Long, Double],\n    expansionCandidateMap: Map[Long, Option[CandidatesFollowedV1]]\n  ): Seq[InterestExpansionCandidate] = {\n\n    // extract features\n    val candidates: Seq[(Long, ExpansionCandidateScores)] = for {\n      (underlyingCandidateId, underlyingCandidateScore) <- underlyingCandidatesMap.toSeq\n      expansionCandidates =\n        expansionCandidateMap\n          .get(underlyingCandidateId)\n          .flatten\n          .map(_.candidatesFollowed)\n          .getOrElse(Seq.empty)\n      expansionCandidate <- expansionCandidates\n    } yield expansionCandidate.candidateID -> ExpansionCandidateScores(\n      underlyingCandidateId,\n      Some(underlyingCandidateScore),\n      Some(expansionCandidate.score)\n    )\n\n    // merge intermediate nodes for the same candidate\n    val dedupedCandidates: Seq[(Long, Seq[ExpansionCandidateScores])] =\n      candidates.groupBy(_._1).mapValues(_.map(_._2).sortBy(_.intermediateCandidateId)).toSeq\n\n    // score the candidate\n    val candidatesWithTotalScore: Seq[((Long, Seq[ExpansionCandidateScores]), Double)] =\n      dedupedCandidates.map { candidate: (Long, Seq[ExpansionCandidateScores]) =>\n        (\n          candidate,\n          candidate._2.map { ieScore: ExpansionCandidateScores =>\n            ieScore.scoreFromUserToIntermediateCandidate.getOrElse(DefaultScore) *\n              ieScore.scoreFromIntermediateToExpansionCandidate.getOrElse(DefaultScore)\n          }.sum)\n      }\n\n    // sort candidate by score\n    for {\n      ((candidate, edges), score) <- candidatesWithTotalScore.sortBy(_._2)(Ordering[Double].reverse)\n    } yield InterestExpansionCandidate(\n      candidate,\n      score,\n      RawYMBIICandidateFeatures(\n        edges.size,\n        edges.take(MaxNumIntermediateNodesToKeep).to[immutable.Seq])\n    )\n  }\n\n}\n\nobject RealGraphExpansionRepository {\n  private val FirstDegreeCandidatesTimeout: Duration = 250.milliseconds\n  private val MaxNumIntermediateNodesToKeep = 20\n  private val DefaultScore = 0.0d\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/SimilarUserExpanderParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.base\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject SimilarUserExpanderParams {\n\n  case object EnableNonDirectFollowExpansion\n      extends FSParam[Boolean](\"similar_user_enable_non_direct_follow_expansion\", true)\n\n  case object EnableSimsExpandSeedAccountsSort\n      extends FSParam[Boolean](\"similar_user_enable_sims_expander_seed_account_sort\", false)\n\n  case object DefaultExpansionInputCount\n      extends FSBoundedParam[Int](\n        name = \"similar_user_default_expansion_input_count\",\n        default = Integer.MAX_VALUE,\n        min = 0,\n        max = Integer.MAX_VALUE)\n\n  case object DefaultFinalCandidatesReturnedCount\n      extends FSBoundedParam[Int](\n        name = \"similar_user_default_final_candidates_returned_count\",\n        default = Integer.MAX_VALUE,\n        min = 0,\n        max = Integer.MAX_VALUE)\n\n  case object DefaultEnableImplicitEngagedExpansion\n      extends FSParam[Boolean](\"similar_user_enable_implicit_engaged_expansion\", true)\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/SimilarUserExpanderRepository.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.base\n\nimport com.twitter.follow_recommendations.common.candidate_sources.base.SimilarUserExpanderParams.DefaultEnableImplicitEngagedExpansion\nimport com.twitter.follow_recommendations.common.candidate_sources.base.SimilarUserExpanderParams.DefaultExpansionInputCount\nimport com.twitter.follow_recommendations.common.candidate_sources.base.SimilarUserExpanderParams.DefaultFinalCandidatesReturnedCount\nimport com.twitter.follow_recommendations.common.candidate_sources.base.SimilarUserExpanderParams.EnableNonDirectFollowExpansion\nimport com.twitter.follow_recommendations.common.candidate_sources.base.SimilarUserExpanderParams.EnableSimsExpandSeedAccountsSort\nimport com.twitter.follow_recommendations.common.candidate_sources.base.SimilarUserExpanderRepository.DefaultCandidateBuilder\nimport com.twitter.follow_recommendations.common.candidate_sources.base.SimilarUserExpanderRepository.DefaultScore\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.EngagementType\nimport com.twitter.follow_recommendations.common.models.FollowProof\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.follow_recommendations.common.models.SimilarToProof\nimport com.twitter.follow_recommendations.common.models.UserCandidateSourceDetails\nimport com.twitter.hermit.candidate.thriftscala.Candidates\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Params\n\ncase class SecondDegreeCandidate(userId: Long, score: Double, socialProof: Option[Seq[Long]])\n\nabstract class SimilarUserExpanderRepository[-Request <: HasParams](\n  override val identifier: CandidateSourceIdentifier,\n  similarToCandidatesFetcher: Fetcher[\n    Long,\n    Unit,\n    Candidates\n  ],\n  expansionInputSizeParam: FSBoundedParam[Int] = DefaultExpansionInputCount,\n  candidatesReturnedSizeParam: FSBoundedParam[Int] = DefaultFinalCandidatesReturnedCount,\n  enableImplicitEngagedExpansion: FSParam[Boolean] = DefaultEnableImplicitEngagedExpansion,\n  thresholdToAvoidExpansion: Int = 30,\n  maxExpansionPerCandidate: Option[Int] = None,\n  includingOriginalCandidates: Boolean = false,\n  scorer: (Double, Double) => Double = SimilarUserExpanderRepository.DefaultScorer,\n  aggregator: (Seq[Double]) => Double = ScoreAggregator.Max,\n  candidateBuilder: (Long, CandidateSourceIdentifier, Double, CandidateUser) => CandidateUser =\n    DefaultCandidateBuilder)\n    extends TwoHopExpansionCandidateSource[\n      Request,\n      CandidateUser,\n      SecondDegreeCandidate,\n      CandidateUser\n    ] {\n\n  val originalCandidateSource: CandidateSource[Request, CandidateUser]\n  val backupOriginalCandidateSource: Option[CandidateSource[Request, CandidateUser]] = None\n\n  override def firstDegreeNodes(request: Request): Stitch[Seq[CandidateUser]] = {\n\n    val originalCandidatesStitch: Stitch[Seq[CandidateUser]] =\n      originalCandidateSource(request)\n\n    val backupCandidatesStitch: Stitch[Seq[CandidateUser]] =\n      if (request.params(EnableNonDirectFollowExpansion)) {\n        backupOriginalCandidateSource.map(_.apply(request)).getOrElse(Stitch.Nil)\n      } else {\n        Stitch.Nil\n      }\n\n    val firstDegreeCandidatesCombinedStitch: Stitch[Seq[CandidateUser]] =\n      Stitch\n        .join(originalCandidatesStitch, backupCandidatesStitch).map {\n          case (firstDegreeOrigCandidates, backupFirstDegreeCandidates) =>\n            if (request.params(EnableSimsExpandSeedAccountsSort)) {\n              firstDegreeOrigCandidates ++ backupFirstDegreeCandidates sortBy {\n                -_.score.getOrElse(DefaultScore)\n              }\n            } else {\n              firstDegreeOrigCandidates ++ backupFirstDegreeCandidates\n            }\n        }\n\n    val candidatesAfterImplicitEngagementsRemovalStitch: Stitch[Seq[CandidateUser]] =\n      getCandidatesAfterImplicitEngagementFiltering(\n        request.params,\n        firstDegreeCandidatesCombinedStitch)\n\n    val firstDegreeCandidatesCombinedTrimmed = candidatesAfterImplicitEngagementsRemovalStitch.map {\n      candidates: Seq[CandidateUser] =>\n        candidates.take(request.params(expansionInputSizeParam))\n    }\n\n    firstDegreeCandidatesCombinedTrimmed.map { firstDegreeResults: Seq[CandidateUser] =>\n      if (firstDegreeResults.nonEmpty && firstDegreeResults.size < thresholdToAvoidExpansion) {\n        firstDegreeResults\n          .groupBy(_.id).mapValues(\n            _.maxBy(_.score)\n          ).values.toSeq\n      } else {\n        Nil\n      }\n    }\n\n  }\n\n  override def secondaryDegreeNodes(\n    request: Request,\n    firstDegreeCandidate: CandidateUser\n  ): Stitch[Seq[SecondDegreeCandidate]] = {\n    similarToCandidatesFetcher.fetch(firstDegreeCandidate.id).map(_.v).map { candidateListOption =>\n      candidateListOption\n        .map { candidatesList =>\n          candidatesList.candidates.map(candidate =>\n            SecondDegreeCandidate(candidate.userId, candidate.score, candidate.socialProof))\n        }.getOrElse(Nil)\n    }\n\n  }\n\n  override def aggregateAndScore(\n    req: Request,\n    firstDegreeToSecondDegreeNodesMap: Map[CandidateUser, Seq[SecondDegreeCandidate]]\n  ): Stitch[Seq[CandidateUser]] = {\n\n    val similarExpanderResults = firstDegreeToSecondDegreeNodesMap.flatMap {\n      case (firstDegreeCandidate, seqOfSecondDegreeCandidates) =>\n        val sourceScore = firstDegreeCandidate.score.getOrElse(DefaultScore)\n        val results: Seq[CandidateUser] = seqOfSecondDegreeCandidates.map { secondDegreeCandidate =>\n          val score = scorer(sourceScore, secondDegreeCandidate.score)\n          candidateBuilder(secondDegreeCandidate.userId, identifier, score, firstDegreeCandidate)\n        }\n        maxExpansionPerCandidate match {\n          case None => results\n          case Some(limit) => results.sortBy(-_.score.getOrElse(DefaultScore)).take(limit)\n        }\n    }.toSeq\n\n    val allCandidates = {\n      if (includingOriginalCandidates)\n        firstDegreeToSecondDegreeNodesMap.keySet.toSeq\n      else\n        Nil\n    } ++ similarExpanderResults\n\n    val groupedCandidates: Seq[CandidateUser] = allCandidates\n      .groupBy(_.id)\n      .flatMap {\n        case (_, candidates) =>\n          val finalScore = aggregator(candidates.map(_.score.getOrElse(DefaultScore)))\n          val candidateSourceDetailsCombined = aggregateCandidateSourceDetails(candidates)\n          val accountSocialProofcombined = aggregateAccountSocialProof(candidates)\n\n          candidates.headOption.map(\n            _.copy(\n              score = Some(finalScore),\n              reason = accountSocialProofcombined,\n              userCandidateSourceDetails = candidateSourceDetailsCombined)\n              .withCandidateSource(identifier))\n      }\n      .toSeq\n\n    Stitch.value(\n      groupedCandidates\n        .sortBy { -_.score.getOrElse(DefaultScore) }.take(req.params(candidatesReturnedSizeParam))\n    )\n  }\n\n  def aggregateCandidateSourceDetails(\n    candidates: Seq[CandidateUser]\n  ): Option[UserCandidateSourceDetails] = {\n    candidates\n      .map { candidate =>\n        candidate.userCandidateSourceDetails.map(_.candidateSourceScores).getOrElse(Map.empty)\n      }.reduceLeftOption { (scoreMap1, scoreMap2) =>\n        scoreMap1 ++ scoreMap2\n      }.map {\n        UserCandidateSourceDetails(primaryCandidateSource = None, _)\n      }\n\n  }\n\n  def aggregateAccountSocialProof(candidates: Seq[CandidateUser]): Option[Reason] = {\n    candidates\n      .map { candidate =>\n        (\n          candidate.reason\n            .flatMap(_.accountProof.flatMap(_.similarToProof.map(_.similarTo))).getOrElse(Nil),\n          candidate.reason\n            .flatMap(_.accountProof.flatMap(_.followProof.map(_.followedBy))).getOrElse(Nil),\n          candidate.reason\n            .flatMap(_.accountProof.flatMap(_.followProof.map(_.numIds))).getOrElse(0)\n        )\n      }.reduceLeftOption { (accountProofOne, accountProofTwo) =>\n        (\n          // merge similarToIds\n          accountProofOne._1 ++ accountProofTwo._1,\n          // merge followedByIds\n          accountProofOne._2 ++ accountProofTwo._2,\n          // add numIds\n          accountProofOne._3 + accountProofTwo._3)\n      }.map { proofs =>\n        Reason(accountProof = Some(\n          AccountProof(\n            similarToProof = Some(SimilarToProof(proofs._1)),\n            followProof = if (proofs._2.nonEmpty) Some(FollowProof(proofs._2, proofs._3)) else None\n          )))\n      }\n  }\n\n  def getCandidatesAfterImplicitEngagementFiltering(\n    params: Params,\n    firstDegreeCandidatesStitch: Stitch[Seq[CandidateUser]]\n  ): Stitch[Seq[CandidateUser]] = {\n\n    if (!params(enableImplicitEngagedExpansion)) {\n\n      /**\n       * Remove candidates whose engagement types only contain implicit engagements\n       * (e.g. Profile View, Tweet Click) and only expand those candidates who contain explicit\n       * engagements.\n       */\n      firstDegreeCandidatesStitch.map { candidates =>\n        candidates.filter { cand =>\n          cand.engagements.exists(engage =>\n            engage == EngagementType.Like || engage == EngagementType.Retweet || engage == EngagementType.Mention)\n        }\n      }\n    } else {\n      firstDegreeCandidatesStitch\n    }\n  }\n\n}\n\nobject SimilarUserExpanderRepository {\n  val DefaultScorer: (Double, Double) => Double = (sourceScore: Double, similarScore: Double) =>\n    similarScore\n  val MultiplyScorer: (Double, Double) => Double = (sourceScore: Double, similarScore: Double) =>\n    sourceScore * similarScore\n  val SourceScorer: (Double, Double) => Double = (sourceScore: Double, similarScore: Double) =>\n    sourceScore\n\n  val DefaultScore = 0.0d\n\n  val DefaultCandidateBuilder: (\n    Long,\n    CandidateSourceIdentifier,\n    Double,\n    CandidateUser\n  ) => CandidateUser =\n    (\n      userId: Long,\n      _: CandidateSourceIdentifier,\n      score: Double,\n      candidate: CandidateUser\n    ) => {\n      val originalCandidateSourceDetails =\n        candidate.userCandidateSourceDetails.flatMap { candSourceDetails =>\n          candSourceDetails.primaryCandidateSource.map { primaryCandidateSource =>\n            UserCandidateSourceDetails(\n              primaryCandidateSource = None,\n              candidateSourceScores = Map(primaryCandidateSource -> candidate.score))\n          }\n        }\n      CandidateUser(\n        id = userId,\n        score = Some(score),\n        userCandidateSourceDetails = originalCandidateSourceDetails,\n        reason =\n          Some(Reason(Some(AccountProof(similarToProof = Some(SimilarToProof(Seq(candidate.id)))))))\n      )\n    }\n\n  val FollowClusterCandidateBuilder: (\n    Long,\n    CandidateSourceIdentifier,\n    Double,\n    CandidateUser\n  ) => CandidateUser =\n    (userId: Long, _: CandidateSourceIdentifier, score: Double, candidate: CandidateUser) => {\n      val originalCandidateSourceDetails =\n        candidate.userCandidateSourceDetails.flatMap { candSourceDetails =>\n          candSourceDetails.primaryCandidateSource.map { primaryCandidateSource =>\n            UserCandidateSourceDetails(\n              primaryCandidateSource = None,\n              candidateSourceScores = Map(primaryCandidateSource -> candidate.score))\n          }\n        }\n\n      val originalFollowCluster = candidate.reason\n        .flatMap(_.accountProof.flatMap(_.followProof.map(_.followedBy)))\n\n      CandidateUser(\n        id = userId,\n        score = Some(score),\n        userCandidateSourceDetails = originalCandidateSourceDetails,\n        reason = Some(\n          Reason(\n            Some(\n              AccountProof(\n                similarToProof = Some(SimilarToProof(Seq(candidate.id))),\n                followProof = originalFollowCluster.map(follows =>\n                  FollowProof(follows, follows.size)))))\n        )\n      )\n    }\n}\n\nobject ScoreAggregator {\n  // aggregate the same candidates with same id by taking the one with largest score\n  val Max: Seq[Double] => Double = (candidateScores: Seq[Double]) => { candidateScores.max }\n\n  // aggregate the same candidates with same id by taking the sum of the scores\n  val Sum: Seq[Double] => Double = (candidateScores: Seq[Double]) => { candidateScores.sum }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/SocialProofEnforcedCandidateSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.base\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.transforms.modify_social_proof.ModifySocialProof\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.Duration\n\nabstract class SocialProofEnforcedCandidateSource(\n  candidateSource: CandidateSource[HasClientContext with HasParams, CandidateUser],\n  modifySocialProof: ModifySocialProof,\n  minNumSocialProofsRequired: Int,\n  override val identifier: CandidateSourceIdentifier,\n  baseStatsReceiver: StatsReceiver)\n    extends CandidateSource[HasClientContext with HasParams, CandidateUser] {\n\n  val statsReceiver = baseStatsReceiver.scope(identifier.name)\n\n  override def apply(target: HasClientContext with HasParams): Stitch[Seq[CandidateUser]] = {\n    val mustCallSgs: Boolean = target.params(SocialProofEnforcedCandidateSourceParams.MustCallSgs)\n    val callSgsCachedColumn: Boolean =\n      target.params(SocialProofEnforcedCandidateSourceParams.CallSgsCachedColumn)\n    val QueryIntersectionIdsNum: Int =\n      target.params(SocialProofEnforcedCandidateSourceParams.QueryIntersectionIdsNum)\n    val MaxNumCandidatesToAnnotate: Int =\n      target.params(SocialProofEnforcedCandidateSourceParams.MaxNumCandidatesToAnnotate)\n    val gfsIntersectionIdsNum: Int =\n      target.params(SocialProofEnforcedCandidateSourceParams.GfsIntersectionIdsNum)\n    val sgsIntersectionIdsNum: Int =\n      target.params(SocialProofEnforcedCandidateSourceParams.SgsIntersectionIdsNum)\n    val gfsLagDuration: Duration =\n      target.params(SocialProofEnforcedCandidateSourceParams.GfsLagDurationInDays)\n\n    candidateSource(target)\n      .flatMap { candidates =>\n        val candidatesWithoutEnoughSocialProof = candidates\n          .collect {\n            case candidate if !candidate.followedBy.exists(_.size >= minNumSocialProofsRequired) =>\n              candidate\n          }\n        statsReceiver\n          .stat(\"candidates_with_no_social_proofs\").add(candidatesWithoutEnoughSocialProof.size)\n        val candidatesToAnnotate =\n          candidatesWithoutEnoughSocialProof.take(MaxNumCandidatesToAnnotate)\n        statsReceiver.stat(\"candidates_to_annotate\").add(candidatesToAnnotate.size)\n\n        val annotatedCandidatesMapStitch = target.getOptionalUserId\n          .map { userId =>\n            modifySocialProof\n              .hydrateSocialProof(\n                userId,\n                candidatesToAnnotate,\n                Some(QueryIntersectionIdsNum),\n                mustCallSgs,\n                callSgsCachedColumn,\n                gfsLagDuration = gfsLagDuration,\n                gfsIntersectionIds = gfsIntersectionIdsNum,\n                sgsIntersectionIds = sgsIntersectionIdsNum\n              ).map { annotatedCandidates =>\n                annotatedCandidates\n                  .map(annotatedCandidate => (annotatedCandidate.id, annotatedCandidate)).toMap\n              }\n          }.getOrElse(Stitch.value(Map.empty[Long, CandidateUser]))\n\n        annotatedCandidatesMapStitch.map { annotatedCandidatesMap =>\n          candidates\n            .flatMap { candidate =>\n              if (candidate.followedBy.exists(_.size >= minNumSocialProofsRequired)) {\n                Some(candidate)\n              } else {\n                annotatedCandidatesMap.get(candidate.id).collect {\n                  case annotatedCandidate\n                      if annotatedCandidate.followedBy.exists(\n                        _.size >= minNumSocialProofsRequired) =>\n                    annotatedCandidate\n                }\n              }\n            }.map(_.withCandidateSource(identifier))\n        }\n      }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/SocialProofEnforcedCandidateSourceFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.base\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SocialProofEnforcedCandidateSourceFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] =\n    Seq(\n      SocialProofEnforcedCandidateSourceParams.MustCallSgs,\n      SocialProofEnforcedCandidateSourceParams.CallSgsCachedColumn,\n    )\n  override val intFSParams: Seq[FSBoundedParam[Int]] =\n    Seq(\n      SocialProofEnforcedCandidateSourceParams.QueryIntersectionIdsNum,\n      SocialProofEnforcedCandidateSourceParams.MaxNumCandidatesToAnnotate,\n      SocialProofEnforcedCandidateSourceParams.GfsIntersectionIdsNum,\n      SocialProofEnforcedCandidateSourceParams.SgsIntersectionIdsNum,\n    )\n\n  override val durationFSParams: Seq[FSBoundedParam[Duration] with HasDurationConversion] = Seq(\n    SocialProofEnforcedCandidateSourceParams.GfsLagDurationInDays\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/SocialProofEnforcedCandidateSourceParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.base\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.util.Duration\n\nobject SocialProofEnforcedCandidateSourceParams {\n  case object MustCallSgs\n      extends FSParam[Boolean](\"social_proof_enforced_candidate_source_must_call_sgs\", true)\n\n  case object CallSgsCachedColumn\n      extends FSParam[Boolean](\n        \"social_proof_enforced_candidate_source_call_sgs_cached_column\",\n        false)\n\n  case object QueryIntersectionIdsNum\n      extends FSBoundedParam[Int](\n        name = \"social_proof_enforced_candidate_source_query_intersection_ids_num\",\n        default = 3,\n        min = 0,\n        max = Integer.MAX_VALUE)\n\n  case object MaxNumCandidatesToAnnotate\n      extends FSBoundedParam[Int](\n        name = \"social_proof_enforced_candidate_source_max_num_candidates_to_annotate\",\n        default = 50,\n        min = 0,\n        max = Integer.MAX_VALUE)\n\n  case object GfsIntersectionIdsNum\n      extends FSBoundedParam[Int](\n        name = \"social_proof_enforced_candidate_source_gfs_intersection_ids_num\",\n        default = 3,\n        min = 0,\n        max = Integer.MAX_VALUE)\n\n  case object SgsIntersectionIdsNum\n      extends FSBoundedParam[Int](\n        name = \"social_proof_enforced_candidate_source_sgs_intersection_ids_num\",\n        default = 10,\n        min = 0,\n        max = Integer.MAX_VALUE)\n\n  case object GfsLagDurationInDays\n      extends FSBoundedParam[Duration](\n        name = \"social_proof_enforced_candidate_source_gfs_lag_duration_in_days\",\n        default = 14.days,\n        min = 1.days,\n        max = 60.days)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromDays\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/StratoFetcherSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.base\n\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\n\nabstract class StratoFetcherSource[K, U, V](\n  fetcher: Fetcher[K, U, V],\n  view: U,\n  override val identifier: CandidateSourceIdentifier)\n    extends CandidateSource[K, CandidateUser] {\n\n  def map(user: K, v: V): Seq[CandidateUser]\n\n  override def apply(target: K): Stitch[Seq[CandidateUser]] = {\n    fetcher\n      .fetch(target, view)\n      .map { result =>\n        result.v\n          .map { candidates => map(target, candidates) }\n          .getOrElse(Nil)\n          .map(_.withCandidateSource(identifier))\n      }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/StratoFetcherWithUnitViewSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.base\n\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\n\nabstract class StratoFetcherWithUnitViewSource[K, V](\n  fetcher: Fetcher[K, Unit, V],\n  override val identifier: CandidateSourceIdentifier)\n    extends StratoFetcherSource[K, Unit, V](fetcher, Unit, identifier)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/TweetAuthorsCandidateSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.base\n\nimport com.twitter.follow_recommendations.common.models.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.stitch.Stitch\n\n/**\n * base trait for tweet authors based algorithms, e.g. topical tweet authors, twistly, ...\n *\n * @tparam Target target type\n * @tparam Candidate output candidate types\n */\ntrait TweetAuthorsCandidateSource[-Target, +Candidate] extends CandidateSource[Target, Candidate] {\n\n  /**\n   * fetch Tweet candidates\n   */\n  def getTweetCandidates(target: Target): Stitch[Seq[TweetCandidate]]\n\n  /**\n   * fetch authorId\n   */\n  def getTweetAuthorId(tweetCandidate: TweetCandidate): Stitch[Option[Long]]\n\n  /**\n   * wrap candidate ID and TweetAuthorProof in Candidate\n   */\n  def toCandidate(authorId: Long, tweetIds: Seq[Long], score: Option[Double]): Candidate\n\n  /**\n   * aggregate scores, default to the first score\n   */\n  def aggregator(scores: Seq[Double]): Double =\n    scores.headOption.getOrElse(TweetAuthorsCandidateSource.DefaultScore)\n\n  /**\n   * aggregation method for a group of tweet candidates\n   */\n  def aggregateAndScore(\n    target: Target,\n    tweetCandidates: Seq[TweetCandidate]\n  ): Seq[Candidate]\n\n  /**\n   * generate a list of candidates for the target\n   */\n  def build(\n    target: Target\n  ): Stitch[Seq[Candidate]] = {\n    // Fetch Tweet candidates and hydrate author IDs\n    val tweetCandidatesStitch = for {\n      tweetCandidates <- getTweetCandidates(target)\n      authorIds <- Stitch.collect(tweetCandidates.map(getTweetAuthorId(_)))\n    } yield {\n      for {\n        (authorIdOpt, tweetCandidate) <- authorIds.zip(tweetCandidates)\n        authorId <- authorIdOpt\n      } yield tweetCandidate.copy(authorId = authorId)\n    }\n\n    // Aggregate and score, convert to candidate\n    tweetCandidatesStitch.map(aggregateAndScore(target, _))\n  }\n\n  def apply(target: Target): Stitch[Seq[Candidate]] =\n    build(target)\n}\n\nobject TweetAuthorsCandidateSource {\n  final val DefaultScore: Double = 0.0\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base/TwoHopExpansionCandidateSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.base\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.stitch.Stitch\n\n/**\n * base trait for two-hop expansion based algorithms, e.g. online_stp, phonebook_prediction,\n * recent following sims, recent engagement sims, ...\n *\n * @tparam Target target type\n * @tparam FirstDegree type of first degree nodes\n * @tparam SecondaryDegree type of secondary degree nodes\n * @tparam Candidate output candidate types\n */\ntrait TwoHopExpansionCandidateSource[-Target, FirstDegree, SecondaryDegree, +Candidate]\n    extends CandidateSource[Target, Candidate] {\n\n  /**\n   * fetch first degree nodes given request\n   */\n  def firstDegreeNodes(req: Target): Stitch[Seq[FirstDegree]]\n\n  /**\n   * fetch secondary degree nodes given request and first degree nodes\n   */\n  def secondaryDegreeNodes(req: Target, node: FirstDegree): Stitch[Seq[SecondaryDegree]]\n\n  /**\n   * aggregate and score the candidates to generate final results\n   */\n  def aggregateAndScore(\n    req: Target,\n    firstDegreeToSecondDegreeNodesMap: Map[FirstDegree, Seq[SecondaryDegree]]\n  ): Stitch[Seq[Candidate]]\n\n  /**\n   * Generate a list of candidates for the target\n   */\n  def apply(target: Target): Stitch[Seq[Candidate]] = {\n    for {\n      firstDegreeNodes <- firstDegreeNodes(target)\n      secondaryDegreeNodes <- Stitch.traverse(firstDegreeNodes)(secondaryDegreeNodes(target, _))\n      aggregated <- aggregateAndScore(target, firstDegreeNodes.zip(secondaryDegreeNodes).toMap)\n    } yield aggregated\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/crowd_search_accounts/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"escherbird/src/scala/com/twitter/escherbird/util/stitchcache\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/geoduck\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/model\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"src/thrift/com/twitter/onboarding/relevance/crowd_search_accounts:crowd_search_accounts-scala\",\n        \"strato/config/columns/onboarding/userrecs:userrecs-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-core/src/main/scala/com/twitter/conversions\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/crowd_search_accounts/CrowdSearchAccountsFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.crowd_search_accounts\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.Param\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CrowdSearchAccountsFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] = Seq(\n    CrowdSearchAccountsParams.CandidateSourceEnabled,\n  )\n  override val doubleFSParams: Seq[FSBoundedParam[Double]] = Seq(\n    CrowdSearchAccountsParams.CandidateSourceWeight,\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/crowd_search_accounts/CrowdSearchAccountsParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.crowd_search_accounts\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumSeqParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject CrowdSearchAccountsParams {\n  // whether or not to fetch CrowdSearchAccounts candidate sources\n  case object CandidateSourceEnabled\n      extends FSParam[Boolean](\"crowd_search_accounts_candidate_source_enabled\", false)\n\n  /**\n   *   Contains the logic key for account filtering and ranking. Currently we have 3 main logic keys\n   *    - new_daily: filtering top searched accounts with max daily searches based on new users\n   *    - new_weekly: filtering top searched accounts with max weekly searches based on new users\n   *    - daily: filtering top searched accounts with max daily searches\n   *    - weekly: filtering top searched accounts with max weekly searches\n   *    Mapping of the Logic Id to Logic key is done via @enum AccountsFilteringAndRankingLogic\n   */\n  case object AccountsFilteringAndRankingLogics\n      extends FSEnumSeqParam[AccountsFilteringAndRankingLogicId.type](\n        name = \"crowd_search_accounts_filtering_and_ranking_logic_ids\",\n        default = Seq(AccountsFilteringAndRankingLogicId.SearchesWeekly),\n        enum = AccountsFilteringAndRankingLogicId)\n\n  case object CandidateSourceWeight\n      extends FSBoundedParam[Double](\n        \"crowd_search_accounts_candidate_source_weight\",\n        default = 1200,\n        min = 0.001,\n        max = 2000)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/crowd_search_accounts/CrowdSearchAccountsSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.crowd_search_accounts\n\nimport com.twitter.escherbird.util.stitchcache.StitchCache\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.candidate_sources.crowd_search_accounts.CrowdSearchAccountsParams.AccountsFilteringAndRankingLogics\nimport com.twitter.follow_recommendations.common.candidate_sources.crowd_search_accounts.CrowdSearchAccountsParams.CandidateSourceEnabled\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasGeohashAndCountryCode\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.onboarding.relevance.crowd_search_accounts.thriftscala.CrowdSearchAccounts\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.userrecs.CrowdSearchAccountsClientColumn\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.Duration\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject AccountsFilteringAndRankingLogicId extends Enumeration {\n  type AccountsFilteringAndRankingLogicId = Value\n\n  val NewSearchesDaily: AccountsFilteringAndRankingLogicId = Value(\"new_searches_daily\")\n  val NewSearchesWeekly: AccountsFilteringAndRankingLogicId = Value(\"new_searches_weekly\")\n  val SearchesDaily: AccountsFilteringAndRankingLogicId = Value(\"searches_daily\")\n  val SearchesWeekly: AccountsFilteringAndRankingLogicId = Value(\"searches_weekly\")\n}\n\nobject CrowdSearchAccountsSource {\n  val MaxCacheSize = 500\n  val CacheTTL: Duration = Duration.fromHours(24)\n\n  type Target = HasParams with HasClientContext with HasGeohashAndCountryCode\n\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.CrowdSearchAccounts.toString)\n}\n\n@Singleton\nclass CrowdSearchAccountsSource @Inject() (\n  crowdSearchAccountsClientColumn: CrowdSearchAccountsClientColumn,\n  statsReceiver: StatsReceiver,\n) extends CandidateSource[CrowdSearchAccountsSource.Target, CandidateUser]\n    with Logging {\n\n  /** @see [[CandidateSourceIdentifier]] */\n  override val identifier: CandidateSourceIdentifier =\n    CrowdSearchAccountsSource.Identifier\n\n  private val stats = statsReceiver.scope(identifier.name)\n  private val requestsStats = stats.counter(\"requests\")\n  private val noCountryCodeStats = stats.counter(\"no_country_code\")\n  private val successStats = stats.counter(\"success\")\n  private val errorStats = stats.counter(\"error\")\n\n  private val cache = StitchCache[String, Option[CrowdSearchAccounts]](\n    maxCacheSize = CrowdSearchAccountsSource.MaxCacheSize,\n    ttl = CrowdSearchAccountsSource.CacheTTL,\n    statsReceiver = statsReceiver.scope(identifier.name, \"cache\"),\n    underlyingCall = (k: String) => {\n      crowdSearchAccountsClientColumn.fetcher\n        .fetch(k)\n        .map { result => result.v }\n    }\n  )\n\n  /** returns a Seq of ''potential'' content */\n  override def apply(\n    target: CrowdSearchAccountsSource.Target\n  ): Stitch[Seq[CandidateUser]] = {\n    if (!target.params(CandidateSourceEnabled)) {\n      return Stitch.value(Seq[CandidateUser]())\n    }\n    requestsStats.incr()\n    target.getCountryCode\n      .orElse(target.geohashAndCountryCode.flatMap(_.countryCode)).map { countryCode =>\n        Stitch\n          .collect(target\n            .params(AccountsFilteringAndRankingLogics).map(logic =>\n              cache.readThrough(countryCode.toUpperCase() + \"-\" + logic)))\n          .onSuccess(_ => {\n            successStats.incr()\n          })\n          .onFailure(t => {\n            debug(\"candidate source failed identifier = %s\".format(identifier), t)\n            errorStats.incr()\n          })\n          .map(transformCrowdSearchAccountsToCandidateSource)\n      }.getOrElse {\n        noCountryCodeStats.incr()\n        Stitch.value(Seq[CandidateUser]())\n      }\n  }\n\n  private def transformCrowdSearchAccountsToCandidateSource(\n    crowdSearchAccounts: Seq[Option[CrowdSearchAccounts]]\n  ): Seq[CandidateUser] = {\n    crowdSearchAccounts\n      .flatMap(opt =>\n        opt\n          .map(accounts =>\n            accounts.accounts.map(account =>\n              CandidateUser(\n                id = account.accountId,\n                score = Some(account.searchActivityScore),\n              ).withCandidateSource(identifier)))\n          .getOrElse(Seq[CandidateUser]()))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/crowd_search_accounts/README.md",
    "content": "# Crowd Search Candidate Source\nProvides the most searched accounts within a specific country over the past 1 and 7 days.\n* When we refer to \"most searched accounts\", we are referring to accounts that have been clicked on the most frequently by users after they see search results in both the typeahead and search results page.\n* The results returned by the service have undergone health filters.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/geoduck\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"src/thrift/com/twitter/hermit/pop_geo:hermit-pop-geo-scala\",\n        \"strato/config/columns/onboarding/userrecs:userrecs-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo/BasePopGeoHashSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.geo\n\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasGeohashAndCountryCode\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\n\n@Singleton\nclass BasePopGeohashSource @Inject() (\n  popGeoSource: CandidateSource[String, CandidateUser],\n  statsReceiver: StatsReceiver)\n    extends CandidateSource[\n      HasParams with HasClientContext with HasGeohashAndCountryCode,\n      CandidateUser\n    ]\n    with BasePopGeohashSourceConfig {\n\n  val stats: StatsReceiver = statsReceiver\n\n  // counter to check if we found a geohash value in the request\n  val foundGeohashCounter: Counter = stats.counter(\"found_geohash_value\")\n  // counter to check if we are missing a geohash value in the request\n  val missingGeohashCounter: Counter = stats.counter(\"missing_geohash_value\")\n\n  /** @see [[CandidateSourceIdentifier]] */\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    \"BasePopGeohashSource\")\n\n  override def apply(\n    target: HasParams with HasClientContext with HasGeohashAndCountryCode\n  ): Stitch[Seq[CandidateUser]] = {\n    if (!candidateSourceEnabled(target)) {\n      return Stitch.Nil\n    }\n    target.geohashAndCountryCode\n      .flatMap(_.geohash).map { geohash =>\n        foundGeohashCounter.incr()\n        val keys = (minGeohashLength(target) to math.min(maxGeohashLength(target), geohash.length))\n          .map(\"geohash_\" + geohash.take(_)).reverse\n        if (returnResultFromAllPrecision(target)) {\n          Stitch\n            .collect(keys.map(popGeoSource.apply)).map(\n              _.flatten.map(_.withCandidateSource(identifier))\n            )\n        } else {\n          Stitch\n            .collect(keys.map(popGeoSource.apply)).map(\n              _.find(_.nonEmpty)\n                .getOrElse(Nil)\n                .take(maxResults(target)).map(_.withCandidateSource(identifier))\n            )\n        }\n      }.getOrElse {\n        missingGeohashCounter.incr()\n        Stitch.Nil\n      }\n  }\n}\n\ntrait BasePopGeohashSourceConfig {\n  type Target = HasParams with HasClientContext\n  def maxResults(target: Target): Int = 200\n  def minGeohashLength(target: Target): Int = 2\n  def maxGeohashLength(target: Target): Int = 4\n  def returnResultFromAllPrecision(target: Target): Boolean = false\n  def candidateSourceEnabled(target: Target): Boolean = false\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo/PopCountryBackFillSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.geo\n\nimport com.google.inject.Singleton\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\n\n@Singleton\nclass PopCountryBackFillSource @Inject() (popGeoSource: PopGeoSource)\n    extends CandidateSource[HasClientContext with HasParams, CandidateUser] {\n\n  override val identifier: CandidateSourceIdentifier = PopCountryBackFillSource.Identifier\n\n  override def apply(target: HasClientContext with HasParams): Stitch[Seq[CandidateUser]] = {\n    target.getOptionalUserId\n      .map(_ =>\n        popGeoSource(PopCountryBackFillSource.DefaultKey)\n          .map(_.take(PopCountryBackFillSource.MaxResults).map(_.withCandidateSource(identifier))))\n      .getOrElse(Stitch.Nil)\n  }\n}\n\nobject PopCountryBackFillSource {\n  val Identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(Algorithm.PopCountryBackFill.toString)\n  val MaxResults = 40\n  val DefaultKey = \"country_US\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo/PopCountrySource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.geo\n\nimport com.google.inject.Singleton\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasGeohashAndCountryCode\nimport com.twitter.follow_recommendations.common.models.HasUserState\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\n\n@Singleton\nclass PopCountrySource @Inject() (\n  popGeoSource: PopGeoSource,\n  statsReceiver: StatsReceiver)\n    extends CandidateSource[\n      HasClientContext with HasParams with HasUserState with HasGeohashAndCountryCode,\n      CandidateUser\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = PopCountrySource.Identifier\n  val stats: StatsReceiver = statsReceiver.scope(\"PopCountrySource\")\n\n  // counter to check if we found a country code value in the request\n  val foundCountryCodeCounter: Counter = stats.counter(\"found_country_code_value\")\n  // counter to check if we are missing a country code value in the request\n  val missingCountryCodeCounter: Counter = stats.counter(\"missing_country_code_value\")\n\n  override def apply(\n    target: HasClientContext with HasParams with HasUserState with HasGeohashAndCountryCode\n  ): Stitch[Seq[CandidateUser]] = {\n    target.geohashAndCountryCode\n      .flatMap(_.countryCode).map { countryCode =>\n        foundCountryCodeCounter.incr()\n        if (target.userState.exists(PopCountrySource.BlacklistedTargetUserStates.contains)) {\n          Stitch.Nil\n        } else {\n          popGeoSource(\"country_\" + countryCode)\n            .map(_.take(PopCountrySource.MaxResults).map(_.withCandidateSource(identifier)))\n        }\n      }.getOrElse {\n        missingCountryCodeCounter.incr()\n        Stitch.Nil\n      }\n  }\n}\n\nobject PopCountrySource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.PopCountry.toString)\n  val MaxResults = 40\n  val BlacklistedTargetUserStates: Set[UserState] = Set(\n    UserState.HeavyTweeter,\n    UserState.HeavyNonTweeter,\n    UserState.MediumTweeter,\n    UserState.MediumNonTweeter)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo/PopGeoQualityFollowSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.geo\nimport com.google.inject.Singleton\nimport com.twitter.escherbird.util.stitchcache.StitchCache\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.PopularInGeoProof\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.hermit.pop_geo.thriftscala.PopUsersInPlace\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.userrecs.UniquePopQualityFollowUsersInPlaceClientColumn\nimport com.twitter.util.Duration\nimport javax.inject.Inject\n\n@Singleton\nclass PopGeohashQualityFollowSource @Inject() (\n  popGeoSource: PopGeoQualityFollowSource,\n  statsReceiver: StatsReceiver)\n    extends BasePopGeohashSource(\n      popGeoSource = popGeoSource,\n      statsReceiver = statsReceiver.scope(\"PopGeohashQualityFollowSource\"),\n    ) {\n  override val identifier: CandidateSourceIdentifier = PopGeohashQualityFollowSource.Identifier\n  override def maxResults(target: Target): Int = {\n    target.params(PopGeoQualityFollowSourceParams.PopGeoSourceMaxResultsPerPrecision)\n  }\n  override def minGeohashLength(target: Target): Int = {\n    target.params(PopGeoQualityFollowSourceParams.PopGeoSourceGeoHashMinPrecision)\n  }\n  override def maxGeohashLength(target: Target): Int = {\n    target.params(PopGeoQualityFollowSourceParams.PopGeoSourceGeoHashMaxPrecision)\n  }\n  override def returnResultFromAllPrecision(target: Target): Boolean = {\n    target.params(PopGeoQualityFollowSourceParams.PopGeoSourceReturnFromAllPrecisions)\n  }\n  override def candidateSourceEnabled(target: Target): Boolean = {\n    target.params(PopGeoQualityFollowSourceParams.CandidateSourceEnabled)\n  }\n}\n\nobject PopGeohashQualityFollowSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.PopGeohashQualityFollow.toString)\n}\n\nobject PopGeoQualityFollowSource {\n  val MaxCacheSize = 20000\n  val CacheTTL: Duration = Duration.fromHours(24)\n  val MaxResults = 200\n}\n\n@Singleton\nclass PopGeoQualityFollowSource @Inject() (\n  popGeoQualityFollowClientColumn: UniquePopQualityFollowUsersInPlaceClientColumn,\n  statsReceiver: StatsReceiver,\n) extends CandidateSource[String, CandidateUser] {\n\n  /** @see [[CandidateSourceIdentifier]] */\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    \"PopGeoQualityFollowSource\")\n\n  private val cache = StitchCache[String, Option[PopUsersInPlace]](\n    maxCacheSize = PopGeoQualityFollowSource.MaxCacheSize,\n    ttl = PopGeoQualityFollowSource.CacheTTL,\n    statsReceiver = statsReceiver.scope(identifier.name, \"cache\"),\n    underlyingCall = (k: String) => {\n      popGeoQualityFollowClientColumn.fetcher\n        .fetch(k)\n        .map { result => result.v }\n    }\n  )\n\n  override def apply(target: String): Stitch[Seq[CandidateUser]] = {\n    val result: Stitch[Option[PopUsersInPlace]] = cache.readThrough(target)\n    result.map { pu =>\n      pu.map { candidates =>\n          candidates.popUsers.sortBy(-_.score).take(PopGeoQualityFollowSource.MaxResults).map {\n            candidate =>\n              CandidateUser(\n                id = candidate.userId,\n                score = Some(candidate.score),\n                reason = Some(\n                  Reason(\n                    Some(\n                      AccountProof(\n                        popularInGeoProof = Some(PopularInGeoProof(location = candidates.place))\n                      )\n                    )\n                  )\n                )\n              )\n          }\n        }.getOrElse(Nil)\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo/PopGeoQualityFollowSourceFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.geo\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PopGeoQualityFollowSourceFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val intFSParams: Seq[FSBoundedParam[Int] with FSName] = Seq(\n    PopGeoQualityFollowSourceParams.PopGeoSourceGeoHashMaxPrecision,\n    PopGeoQualityFollowSourceParams.PopGeoSourceGeoHashMinPrecision,\n    PopGeoQualityFollowSourceParams.PopGeoSourceMaxResultsPerPrecision\n  )\n  override val doubleFSParams: Seq[FSBoundedParam[Double] with FSName] = Seq(\n    PopGeoQualityFollowSourceParams.CandidateSourceWeight\n  )\n  override val booleanFSParams: Seq[FSParam[Boolean] with FSName] = Seq(\n    PopGeoQualityFollowSourceParams.CandidateSourceEnabled,\n    PopGeoQualityFollowSourceParams.PopGeoSourceReturnFromAllPrecisions\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo/PopGeoQualityFollowSourceParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.geo\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject PopGeoQualityFollowSourceParams {\n  case object CandidateSourceEnabled\n      extends FSParam[Boolean](\"pop_geo_quality_follow_source_enabled\", false)\n\n  case object PopGeoSourceGeoHashMinPrecision\n      extends FSBoundedParam[Int](\n        \"pop_geo_quality_follow_source_geo_hash_min_precision\",\n        default = 2,\n        min = 0,\n        max = 10)\n\n  case object PopGeoSourceGeoHashMaxPrecision\n      extends FSBoundedParam[Int](\n        \"pop_geo_quality_follow_source_geo_hash_max_precision\",\n        default = 3,\n        min = 0,\n        max = 10)\n\n  case object PopGeoSourceReturnFromAllPrecisions\n      extends FSParam[Boolean](\n        \"pop_geo_quality_follow_source_return_from_all_precisions\",\n        default = false)\n\n  case object PopGeoSourceMaxResultsPerPrecision\n      extends FSBoundedParam[Int](\n        \"pop_geo_quality_follow_source_max_results_per_precision\",\n        default = 200,\n        min = 0,\n        max = 1000)\n\n  case object CandidateSourceWeight\n      extends FSBoundedParam[Double](\n        \"pop_geo_quality_follow_source_weight\",\n        default = 200,\n        min = 0.001,\n        max = 2000)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo/PopGeoSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.geo\n\nimport com.google.inject.Singleton\nimport com.google.inject.name.Named\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.candidate_sources.base.CachedCandidateSource\nimport com.twitter.follow_recommendations.common.candidate_sources.base.StratoFetcherWithUnitViewSource\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.PopularInGeoProof\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.hermit.pop_geo.thriftscala.PopUsersInPlace\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.util.Duration\nimport javax.inject.Inject\n\n@Singleton\nclass BasePopGeoSource @Inject() (\n  @Named(GuiceNamedConstants.POP_USERS_IN_PLACE_FETCHER) fetcher: Fetcher[\n    String,\n    Unit,\n    PopUsersInPlace\n  ]) extends StratoFetcherWithUnitViewSource[String, PopUsersInPlace](\n      fetcher,\n      BasePopGeoSource.Identifier) {\n\n  override def map(target: String, candidates: PopUsersInPlace): Seq[CandidateUser] =\n    BasePopGeoSource.map(target, candidates)\n}\n\nobject BasePopGeoSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"BasePopGeoSource\")\n  val MaxResults = 200\n\n  def map(target: String, candidates: PopUsersInPlace): Seq[CandidateUser] =\n    candidates.popUsers.sortBy(-_.score).take(BasePopGeoSource.MaxResults).view.map { candidate =>\n      CandidateUser(\n        id = candidate.userId,\n        score = Some(candidate.score),\n        reason = Some(\n          Reason(\n            Some(\n              AccountProof(\n                popularInGeoProof = Some(PopularInGeoProof(location = candidates.place))\n              )\n            )\n          )\n        )\n      )\n    }\n}\n\n@Singleton\nclass PopGeoSource @Inject() (basePopGeoSource: BasePopGeoSource, statsReceiver: StatsReceiver)\n    extends CachedCandidateSource[String, CandidateUser](\n      basePopGeoSource,\n      PopGeoSource.MaxCacheSize,\n      PopGeoSource.CacheTTL,\n      statsReceiver,\n      PopGeoSource.Identifier)\n\nobject PopGeoSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"PopGeoSource\")\n  val MaxCacheSize = 20000\n  val CacheTTL: Duration = 1.hours\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo/PopGeoSourceFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.geo\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PopGeoSourceFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val intFSParams: Seq[FSBoundedParam[Int] with FSName] = Seq(\n    PopGeoSourceParams.PopGeoSourceGeoHashMaxPrecision,\n    PopGeoSourceParams.PopGeoSourceMaxResultsPerPrecision,\n    PopGeoSourceParams.PopGeoSourceGeoHashMinPrecision,\n  )\n  override val booleanFSParams: Seq[FSParam[Boolean] with FSName] = Seq(\n    PopGeoSourceParams.PopGeoSourceReturnFromAllPrecisions,\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo/PopGeoSourceParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.geo\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject PopGeoSourceParams {\n  case object PopGeoSourceGeoHashMinPrecision\n      extends FSBoundedParam[Int](\n        \"pop_geo_source_geo_hash_min_precision\",\n        default = 2,\n        min = 0,\n        max = 10)\n\n  case object PopGeoSourceGeoHashMaxPrecision\n      extends FSBoundedParam[Int](\n        \"pop_geo_source_geo_hash_max_precision\",\n        default = 4,\n        min = 0,\n        max = 10)\n\n  case object PopGeoSourceReturnFromAllPrecisions\n      extends FSParam[Boolean](\"pop_geo_source_return_from_all_precisions\", default = false)\n\n  case object PopGeoSourceMaxResultsPerPrecision\n      extends FSBoundedParam[Int](\n        \"pop_geo_source_max_results_per_precision\",\n        default = 200,\n        min = 0,\n        max = 1000)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo/PopGeohashSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.geo\n\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport javax.inject.Inject\n\n@Singleton\nclass PopGeohashSource @Inject() (\n  popGeoSource: PopGeoSource,\n  statsReceiver: StatsReceiver)\n    extends BasePopGeohashSource(\n      popGeoSource = popGeoSource,\n      statsReceiver = statsReceiver.scope(\"PopGeohashSource\"),\n    ) {\n  override def candidateSourceEnabled(target: Target): Boolean = true\n  override val identifier: CandidateSourceIdentifier = PopGeohashSource.Identifier\n  override def minGeohashLength(target: Target): Int = {\n    target.params(PopGeoSourceParams.PopGeoSourceGeoHashMinPrecision)\n  }\n  override def maxResults(target: Target): Int = {\n    target.params(PopGeoSourceParams.PopGeoSourceMaxResultsPerPrecision)\n  }\n  override def maxGeohashLength(target: Target): Int = {\n    target.params(PopGeoSourceParams.PopGeoSourceGeoHashMaxPrecision)\n  }\n  override def returnResultFromAllPrecision(target: Target): Boolean = {\n    target.params(PopGeoSourceParams.PopGeoSourceReturnFromAllPrecisions)\n  }\n}\n\nobject PopGeohashSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.PopGeohash.toString)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo/README.md",
    "content": "# Pop Geo Candidate Source\nProvides the most followed / quality followed accounts in a specific country and a geolocation within past 2 weeks.\n* A \"quality follow\" refers to any follow that leads to visible engagement, such as favorites, mentions, retweets, direct messages, replies, and quote tweets. The engagement must be allowed in either direction, and must occur on the day of the follow or within one subsequent day. Additionally, there must be no unfollowing, blocking, muting, or reporting of the account in the same time period.\n* The minimum geolocation precision used is ±20 km (12 mi), and precise user geolocation is not utilized.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/ppmi_locale_follow/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"src/thrift/com/twitter/hermit/candidate:hermit-candidate-scala\",\n        \"strato/config/columns/onboarding:onboarding-strato-client\",\n        \"strato/config/columns/onboarding/userrecs:userrecs-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/ppmi_locale_follow/PPMILocaleFollowSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.ppmi_locale_follow\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.candidate_sources.ppmi_locale_follow.PPMILocaleFollowSourceParams.CandidateSourceEnabled\nimport com.twitter.follow_recommendations.common.candidate_sources.ppmi_locale_follow.PPMILocaleFollowSourceParams.LocaleToExcludeFromRecommendation\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.strato.generated.client.onboarding.UserPreferredLanguagesOnUserClientColumn\nimport com.twitter.strato.generated.client.onboarding.userrecs.LocaleFollowPpmiClientColumn\nimport com.twitter.timelines.configapi.HasParams\n\n/**\n * Fetches candidates based on the Positive Pointwise Mutual Information (PPMI) statistic\n * for a set of locales\n * */\n@Singleton\nclass PPMILocaleFollowSource @Inject() (\n  userPreferredLanguagesOnUserClientColumn: UserPreferredLanguagesOnUserClientColumn,\n  localeFollowPpmiClientColumn: LocaleFollowPpmiClientColumn,\n  statsReceiver: StatsReceiver)\n    extends CandidateSource[HasClientContext with HasParams, CandidateUser] {\n\n  override val identifier: CandidateSourceIdentifier = PPMILocaleFollowSource.Identifier\n  private val stats = statsReceiver.scope(\"PPMILocaleFollowSource\")\n\n  override def apply(target: HasClientContext with HasParams): Stitch[Seq[CandidateUser]] = {\n    (for {\n      countryCode <- target.getCountryCode\n      userId <- target.getOptionalUserId\n    } yield {\n      getPreferredLocales(userId, countryCode.toLowerCase())\n        .flatMap { locale =>\n          stats.addGauge(\"allLocale\") {\n            locale.length\n          }\n          val filteredLocale =\n            locale.filter(!target.params(LocaleToExcludeFromRecommendation).contains(_))\n          stats.addGauge(\"postFilterLocale\") {\n            filteredLocale.length\n          }\n          if (target.params(CandidateSourceEnabled)) {\n            getPPMILocaleFollowCandidates(filteredLocale)\n          } else Stitch(Seq.empty)\n        }\n        .map(_.sortBy(_.score)(Ordering[Option[Double]].reverse)\n          .take(PPMILocaleFollowSource.DefaultMaxCandidatesToReturn))\n    }).getOrElse(Stitch.Nil)\n  }\n\n  private def getPPMILocaleFollowCandidates(\n    locales: Seq[String]\n  ): Stitch[Seq[CandidateUser]] = {\n    Stitch\n      .traverse(locales) { locale =>\n        // Get PPMI candidates for each locale\n        localeFollowPpmiClientColumn.fetcher\n          .fetch(locale)\n          .map(_.v\n            .map(_.candidates).getOrElse(Nil).map { candidate =>\n              CandidateUser(id = candidate.userId, score = Some(candidate.score))\n            }.map(_.withCandidateSource(identifier)))\n      }.map(_.flatten)\n  }\n\n  private def getPreferredLocales(userId: Long, countryCode: String): Stitch[Seq[String]] = {\n    userPreferredLanguagesOnUserClientColumn.fetcher\n      .fetch(userId)\n      .map(_.v.map(_.languages).getOrElse(Nil).map { lang =>\n        s\"$countryCode-$lang\".toLowerCase\n      })\n  }\n}\n\nobject PPMILocaleFollowSource {\n  val Identifier = CandidateSourceIdentifier(Algorithm.PPMILocaleFollow.toString)\n  val DefaultMaxCandidatesToReturn = 100\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/ppmi_locale_follow/PPMILocaleFollowSourceFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.ppmi_locale_follow\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.Param\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PPMILocaleFollowSourceFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] = Seq(\n    PPMILocaleFollowSourceParams.CandidateSourceEnabled,\n  )\n\n  override val stringSeqFSParams: Seq[Param[Seq[String]] with FSName] = Seq(\n    PPMILocaleFollowSourceParams.LocaleToExcludeFromRecommendation,\n  )\n\n  override val doubleFSParams: Seq[FSBoundedParam[Double]] = Seq(\n    PPMILocaleFollowSourceParams.CandidateSourceWeight,\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/ppmi_locale_follow/PPMILocaleFollowSourceParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.ppmi_locale_follow\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\nclass PPMILocaleFollowSourceParams {}\nobject PPMILocaleFollowSourceParams {\n  case object LocaleToExcludeFromRecommendation\n      extends FSParam[Seq[String]](\n        \"ppmilocale_follow_source_locales_to_exclude_from_recommendation\",\n        default = Seq.empty)\n\n  case object CandidateSourceEnabled\n      extends FSParam[Boolean](\"ppmilocale_follow_source_enabled\", true)\n\n  case object CandidateSourceWeight\n      extends FSBoundedParam[Double](\n        \"ppmilocale_follow_source_candidate_source_weight\",\n        default = 1,\n        min = 0.001,\n        max = 2000)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/ppmi_locale_follow/README.md",
    "content": "# PPMI Locale Follow Candidate Source\nProvides accounts based on PPMI ([Positive Pointwise Mutual Information](https://en.wikipedia.org/wiki/Pointwise_mutual_information#Positive_PMI)) using follow actions as a feature for a specific local (language + country) within a week. In simpler terms, it provides a list of the most followed accounts for a given country and language input, based on the PPMI algorithm.\n\nPPMI is a statistical measure of the association between two events. In this case, it measures the association between the follow actions and the accounts being followed.\n\nIn summary, the service utilizes PPMI and follow actions to provide a list of the most followed accounts for a specific country and language input.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/promoted_accounts/BUILD",
    "content": "scala_library(\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/adserver\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/socialgraph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/promoted_accounts/PromotedAccountsCandidateSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.promoted_accounts\n\nimport com.twitter.adserver.thriftscala.AdServerException\nimport com.twitter.adserver.{thriftscala => adthrift}\nimport com.twitter.finagle.TimeoutException\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.clients.adserver.AdRequest\nimport com.twitter.follow_recommendations.common.clients.adserver.AdserverClient\nimport com.twitter.follow_recommendations.common.clients.socialgraph.SocialGraphClient\nimport com.twitter.follow_recommendations.common.models.FollowProof\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.inject.Logging\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase class PromotedCandidateUser(\n  id: Long,\n  position: Int,\n  adImpression: adthrift.AdImpression,\n  followProof: FollowProof,\n  primaryCandidateSource: Option[CandidateSourceIdentifier])\n\n@Singleton\nclass PromotedAccountsCandidateSource @Inject() (\n  adserverClient: AdserverClient,\n  sgsClient: SocialGraphClient,\n  statsReceiver: StatsReceiver)\n    extends CandidateSource[AdRequest, PromotedCandidateUser]\n    with Logging {\n\n  override val identifier: CandidateSourceIdentifier =\n    PromotedAccountsCandidateSource.Identifier\n\n  val stats: StatsReceiver = statsReceiver.scope(identifier.name)\n  val failureStat: StatsReceiver = stats.scope(\"failures\")\n  val adServerExceptionsCounter: Counter = failureStat.counter(\"AdServerException\")\n  val timeoutCounter: Counter = failureStat.counter(\"TimeoutException\")\n\n  def apply(request: AdRequest): Stitch[Seq[PromotedCandidateUser]] = {\n    adserverClient\n      .getAdImpressions(request)\n      .rescue {\n        case e: TimeoutException =>\n          timeoutCounter.incr()\n          logger.warn(\"Timeout on Adserver\", e)\n          Stitch.Nil\n        case e: AdServerException =>\n          adServerExceptionsCounter.incr()\n          logger.warn(\"Failed to fetch ads\", e)\n          Stitch.Nil\n      }\n      .flatMap { adImpressions: Seq[adthrift.AdImpression] =>\n        profileNumResults(adImpressions.size, \"results_from_ad_server\")\n        val idToImpMap = (for {\n          imp <- adImpressions\n          promotedAccountId <- imp.promotedAccountId\n        } yield promotedAccountId -> imp).toMap\n        request.clientContext.userId\n          .map { userId =>\n            sgsClient\n              .getIntersections(\n                userId,\n                adImpressions.filter(shouldShowSocialContext).flatMap(_.promotedAccountId),\n                PromotedAccountsCandidateSource.NumIntersections\n              ).map { promotedAccountWithIntersections =>\n                idToImpMap.map {\n                  case (promotedAccountId, imp) =>\n                    PromotedCandidateUser(\n                      promotedAccountId,\n                      imp.insertionPosition\n                        .map(_.toInt).getOrElse(\n                          getInsertionPositionDefaultValue(request.isTest.getOrElse(false))\n                        ),\n                      imp,\n                      promotedAccountWithIntersections\n                        .getOrElse(promotedAccountId, FollowProof(Nil, 0)),\n                      Some(identifier)\n                    )\n                }.toSeq\n              }.onSuccess(result => profileNumResults(result.size, \"final_results\"))\n          }.getOrElse(Stitch.Nil)\n      }\n  }\n\n  private def shouldShowSocialContext(imp: adthrift.AdImpression): Boolean =\n    imp.experimentValues.exists { expValues =>\n      expValues.get(\"display.display_style\").contains(\"show_social_context\")\n    }\n\n  private def getInsertionPositionDefaultValue(isTest: Boolean): Int = {\n    if (isTest) 0 else -1\n  }\n\n  private def profileNumResults(resultsSize: Int, statName: String): Unit = {\n    if (resultsSize <= 5) {\n      stats.scope(statName).counter(resultsSize.toString).incr()\n    } else {\n      stats.scope(statName).counter(\"more_than_5\").incr()\n    }\n  }\n}\n\nobject PromotedAccountsCandidateSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.PromotedAccount.toString)\n  val NumIntersections = 3\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/promoted_accounts/README.md",
    "content": "# Promoted Accounts Candidate Source\nPromoted accounts returned from Ads server.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/real_graph/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/real_time_real_graph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/stores\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"strato/config/columns/onboarding/realGraph:realGraph-strato-client\",\n        \"strato/config/columns/onboarding/userrecs:userrecs-strato-client\",\n        \"strato/config/columns/recommendations/twistly:twistly-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/real_graph/README.md",
    "content": "# RealGraph Candidate Source\nProvides out-of-network RealGraph candidates for a given user. RealGraph is a user-user graph dataset that aims to measure the strength of the relationship between two users.\n\nRealGraph comprises two components: a real-time pipeline that tracks various counts and relationships between user-user edges (such as the number of favorites, replies, retweets, clicks, whether followed, muted, or blocked), and an offline pipeline of a larger set of such user-user edge counts and relationships. Currently, the top k in-network scores have been exported for use by various teams.\n\nThe RealGraph dataset is used to predict user interactions at Twitter, and is based on the paper \"[Realgraph: User interaction prediction at Twitter](http://www.ueo-workshop.com/wp-content/uploads/2014/04/sig-alternate.pdf)\" by the UEO workshop at KDD'14.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/real_graph/RealGraphOonFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.real_graph\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.Param\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RealGraphOonFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] =\n    Seq(\n      RealGraphOonParams.IncludeRealGraphOonCandidates,\n      RealGraphOonParams.TryToReadRealGraphOonCandidates,\n      RealGraphOonParams.UseV2\n    )\n  override val doubleFSParams: Seq[FSBoundedParam[Double]] =\n    Seq(\n      RealGraphOonParams.ScoreThreshold\n    )\n  override val intFSParams: Seq[FSBoundedParam[Int]] =\n    Seq(\n      RealGraphOonParams.RealGraphOonResultCountThreshold,\n      RealGraphOonParams.MaxResults,\n    )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/real_graph/RealGraphOonParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.real_graph\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject RealGraphOonParams {\n  case object IncludeRealGraphOonCandidates\n      extends FSParam[Boolean](\n        \"real_graph_oon_include_candidates\",\n        false\n      )\n  case object TryToReadRealGraphOonCandidates\n      extends FSParam[Boolean](\n        \"real_graph_oon_try_to_read_candidates\",\n        false\n      )\n  case object RealGraphOonResultCountThreshold\n      extends FSBoundedParam[Int](\n        \"real_graph_oon_result_count_threshold\",\n        default = 1,\n        min = 0,\n        max = Integer.MAX_VALUE\n      )\n\n  case object UseV2\n      extends FSParam[Boolean](\n        \"real_graph_oon_use_v2\",\n        false\n      )\n\n  case object ScoreThreshold\n      extends FSBoundedParam[Double](\n        \"real_graph_oon_score_threshold\",\n        default = 0.26,\n        min = 0,\n        max = 1.0\n      )\n\n  case object MaxResults\n      extends FSBoundedParam[Int](\n        \"real_graph_oon_max_results\",\n        default = 200,\n        min = 0,\n        max = 1000\n      )\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/real_graph/RealGraphOonV2Source.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.real_graph\n\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.realGraph.UserRealgraphOonV2ClientColumn\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.wtf.candidate.thriftscala.CandidateSeq\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RealGraphOonV2Source @Inject() (\n  realGraphClientColumn: UserRealgraphOonV2ClientColumn)\n    extends CandidateSource[HasParams with HasClientContext, CandidateUser] {\n\n  override val identifier: CandidateSourceIdentifier =\n    RealGraphOonV2Source.Identifier\n\n  override def apply(request: HasParams with HasClientContext): Stitch[Seq[CandidateUser]] = {\n    request.getOptionalUserId\n      .map { userId =>\n        realGraphClientColumn.fetcher\n          .fetch(userId)\n          .map { result =>\n            result.v\n              .map { candidates => parseStratoResults(request, candidates) }\n              .getOrElse(Nil)\n              // returned candidates are sorted by score in descending order\n              .take(request.params(RealGraphOonParams.MaxResults))\n              .map(_.withCandidateSource(identifier))\n          }\n      }.getOrElse(Stitch(Seq.empty))\n  }\n\n  private def parseStratoResults(\n    request: HasParams with HasClientContext,\n    candidateSeqThrift: CandidateSeq\n  ): Seq[CandidateUser] = {\n    candidateSeqThrift.candidates.collect {\n      case candidate if candidate.score >= request.params(RealGraphOonParams.ScoreThreshold) =>\n        CandidateUser(\n          candidate.userId,\n          Some(candidate.score)\n        )\n    }\n  }\n\n}\n\nobject RealGraphOonV2Source {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.RealGraphOonV2.toString\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/real_graph/RealGraphSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.real_graph\n\nimport com.twitter.follow_recommendations.common.clients.real_time_real_graph.RealTimeRealGraphClient\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * This source gets the already followed edges from the real graph column as a candidate source.\n */\n@Singleton\nclass RealGraphSource @Inject() (\n  realGraph: RealTimeRealGraphClient)\n    extends CandidateSource[HasParams with HasClientContext, CandidateUser] {\n  override val identifier: CandidateSourceIdentifier = RealGraphSource.Identifier\n\n  override def apply(request: HasParams with HasClientContext): Stitch[Seq[CandidateUser]] = {\n    request.getOptionalUserId\n      .map { userId =>\n        realGraph.getRealGraphWeights(userId).map { scoreMap =>\n          scoreMap.map {\n            case (candidateId, realGraphScore) =>\n              CandidateUser(id = candidateId, score = Some(realGraphScore))\n                .withCandidateSource(identifier)\n          }.toSeq\n        }\n      }.getOrElse(Stitch.Nil)\n  }\n}\n\nobject RealGraphSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.RealGraphFollowed.toString)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/recent_engagement/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"discovery-ds/src/main/thrift/com/twitter/dds/jobs/repeated_profile_visits:profile_visit-scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/real_time_real_graph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/model\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"src/thrift/com/twitter/experiments/general_metrics:general_metrics-scala\",\n        \"strato/config/columns/rux:rux-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/recent_engagement/README.md",
    "content": "# Recent Engagement Candidate Source\nProvides recently engaged accounts for a given user:\n* Explicit engagements: like, retweet, reply\n* Implicit engagements: profile visit\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/recent_engagement/RecentEngagementDirectFollowSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.recent_engagement\n\nimport com.twitter.follow_recommendations.common.clients.real_time_real_graph.RealTimeRealGraphClient\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RecentEngagementDirectFollowSource @Inject() (\n  realTimeRealGraphClient: RealTimeRealGraphClient)\n    extends CandidateSource[Long, CandidateUser] {\n\n  val identifier: CandidateSourceIdentifier =\n    RecentEngagementDirectFollowSource.Identifier\n\n  /**\n   * Generate a list of candidates for the target using RealtimeGraphClient\n   * and RecentEngagementStore.\n   */\n  override def apply(targetUserId: Long): Stitch[Seq[CandidateUser]] = {\n    realTimeRealGraphClient\n      .getUsersRecentlyEngagedWith(\n        userId = targetUserId,\n        engagementScoreMap = RealTimeRealGraphClient.EngagementScoreMap,\n        includeDirectFollowCandidates = true,\n        includeNonDirectFollowCandidates = false\n      )\n      .map(_.map(_.withCandidateSource(identifier)).sortBy(-_.score.getOrElse(0.0)))\n  }\n}\n\nobject RecentEngagementDirectFollowSource {\n  val Identifier = CandidateSourceIdentifier(Algorithm.RecentEngagementDirectFollow.toString)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/recent_engagement/RecentEngagementNonDirectFollowSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.recent_engagement\n\nimport com.twitter.follow_recommendations.common.clients.real_time_real_graph.RealTimeRealGraphClient\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RecentEngagementNonDirectFollowSource @Inject() (\n  realTimeRealGraphClient: RealTimeRealGraphClient)\n    extends CandidateSource[Long, CandidateUser] {\n\n  val identifier: CandidateSourceIdentifier =\n    RecentEngagementNonDirectFollowSource.Identifier\n\n  /**\n   * Generate a list of candidates for the target using RealtimeGraphClient\n   * and RecentEngagementStore.\n   */\n  override def apply(targetUserId: Long): Stitch[Seq[CandidateUser]] = {\n    realTimeRealGraphClient\n      .getUsersRecentlyEngagedWith(\n        userId = targetUserId,\n        engagementScoreMap = RealTimeRealGraphClient.EngagementScoreMap,\n        includeDirectFollowCandidates = false,\n        includeNonDirectFollowCandidates = true\n      )\n      .map(_.map(_.withCandidateSource(identifier)).sortBy(-_.score.getOrElse(0.0)))\n  }\n}\n\nobject RecentEngagementNonDirectFollowSource {\n  val Identifier = CandidateSourceIdentifier(Algorithm.RecentEngagementNonDirectFollow.toString)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/recent_engagement/RepeatedProfileVisitsFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.recent_engagement\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.Param\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RepeatedProfileVisitsFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] =\n    Seq(\n      RepeatedProfileVisitsParams.IncludeCandidates,\n      RepeatedProfileVisitsParams.UseOnlineDataset,\n    )\n  override val intFSParams: Seq[FSBoundedParam[Int]] =\n    Seq(\n      RepeatedProfileVisitsParams.RecommendationThreshold,\n      RepeatedProfileVisitsParams.BucketingThreshold,\n    )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/recent_engagement/RepeatedProfileVisitsParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.recent_engagement\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject RepeatedProfileVisitsParams {\n\n  // If RepeatedProfileVisitsSource is run and there are recommended candidates for the target user, whether or not\n  // to actually include such candidates in our output recommendations. This FS will be used to control bucketing of\n  // users into control vs treatment buckets.\n  case object IncludeCandidates\n      extends FSParam[Boolean](name = \"repeated_profile_visits_include_candidates\", default = false)\n\n  // The threshold at or above which we will consider a profile to have been visited \"frequently enough\" to recommend\n  // the profile to the target user.\n  case object RecommendationThreshold\n      extends FSBoundedParam[Int](\n        name = \"repeated_profile_visits_recommendation_threshold\",\n        default = 3,\n        min = 0,\n        max = Integer.MAX_VALUE)\n\n  // The threshold at or above which we will consider a profile to have been visited \"frequently enough\" to recommend\n  // the profile to the target user.\n  case object BucketingThreshold\n      extends FSBoundedParam[Int](\n        name = \"repeated_profile_visits_bucketing_threshold\",\n        default = 3,\n        min = 0,\n        max = Integer.MAX_VALUE)\n\n  // Whether or not to use the online dataset (which has repeated profile visits information updated to within minutes)\n  // instead of the offline dataset (updated via offline jobs, which can have delays of hours to days).\n  case object UseOnlineDataset\n      extends FSParam[Boolean](name = \"repeated_profile_visits_use_online_dataset\", default = true)\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/recent_engagement/RepeatedProfileVisitsSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.recent_engagement\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.dds.jobs.repeated_profile_visits.thriftscala.ProfileVisitorInfo\nimport com.twitter.experiments.general_metrics.thriftscala.IdType\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.clients.real_time_real_graph.Engagement\nimport com.twitter.follow_recommendations.common.clients.real_time_real_graph.RealTimeRealGraphClient\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.inject.Logging\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.rux.RepeatedProfileVisitsAggregateClientColumn\n\n@Singleton\nclass RepeatedProfileVisitsSource @Inject() (\n  repeatedProfileVisitsAggregateClientColumn: RepeatedProfileVisitsAggregateClientColumn,\n  realTimeRealGraphClient: RealTimeRealGraphClient,\n  statsReceiver: StatsReceiver)\n    extends CandidateSource[HasParams with HasClientContext, CandidateUser]\n    with Logging {\n\n  val identifier: CandidateSourceIdentifier =\n    RepeatedProfileVisitsSource.Identifier\n\n  val sourceStatsReceiver = statsReceiver.scope(\"repeated_profile_visits_source\")\n  val offlineFetchErrorCounter = sourceStatsReceiver.counter(\"offline_fetch_error\")\n  val offlineFetchSuccessCounter = sourceStatsReceiver.counter(\"offline_fetch_success\")\n  val onlineFetchErrorCounter = sourceStatsReceiver.counter(\"online_fetch_error\")\n  val onlineFetchSuccessCounter = sourceStatsReceiver.counter(\"online_fetch_success\")\n  val noRepeatedProfileVisitsAboveBucketingThresholdCounter =\n    sourceStatsReceiver.counter(\"no_repeated_profile_visits_above_bucketing_threshold\")\n  val hasRepeatedProfileVisitsAboveBucketingThresholdCounter =\n    sourceStatsReceiver.counter(\"has_repeated_profile_visits_above_bucketing_threshold\")\n  val noRepeatedProfileVisitsAboveRecommendationsThresholdCounter =\n    sourceStatsReceiver.counter(\"no_repeated_profile_visits_above_recommendations_threshold\")\n  val hasRepeatedProfileVisitsAboveRecommendationsThresholdCounter =\n    sourceStatsReceiver.counter(\"has_repeated_profile_visits_above_recommendations_threshold\")\n  val includeCandidatesCounter = sourceStatsReceiver.counter(\"include_candidates\")\n  val noIncludeCandidatesCounter = sourceStatsReceiver.counter(\"no_include_candidates\")\n\n  // Returns visited user -> visit count, via off dataset.\n  def applyWithOfflineDataset(targetUserId: Long): Stitch[Map[Long, Int]] = {\n    repeatedProfileVisitsAggregateClientColumn.fetcher\n      .fetch(ProfileVisitorInfo(id = targetUserId, idType = IdType.User)).map(_.v)\n      .handle {\n        case e: Throwable =>\n          logger.error(\"Strato fetch for RepeatedProfileVisitsAggregateClientColumn failed: \" + e)\n          offlineFetchErrorCounter.incr()\n          None\n      }.onSuccess { result =>\n        offlineFetchSuccessCounter.incr()\n      }.map { resultOption =>\n        resultOption\n          .flatMap { result =>\n            result.profileVisitSet.map { profileVisitSet =>\n              profileVisitSet\n                .filter(profileVisit => profileVisit.totalTargetVisitsInLast14Days.getOrElse(0) > 0)\n                .filter(profileVisit => !profileVisit.doesSourceIdFollowTargetId.getOrElse(false))\n                .flatMap { profileVisit =>\n                  (profileVisit.targetId, profileVisit.totalTargetVisitsInLast14Days) match {\n                    case (Some(targetId), Some(totalVisitsInLast14Days)) =>\n                      Some(targetId -> totalVisitsInLast14Days)\n                    case _ => None\n                  }\n                }.toMap[Long, Int]\n            }\n          }.getOrElse(Map.empty)\n      }\n  }\n\n  // Returns visited user -> visit count, via online dataset.\n  def applyWithOnlineData(targetUserId: Long): Stitch[Map[Long, Int]] = {\n    val visitedUserToEngagementsStitch: Stitch[Map[Long, Seq[Engagement]]] =\n      realTimeRealGraphClient.getRecentProfileViewEngagements(targetUserId)\n    visitedUserToEngagementsStitch\n      .onFailure { f =>\n        onlineFetchErrorCounter.incr()\n      }.onSuccess { result =>\n        onlineFetchSuccessCounter.incr()\n      }.map { visitedUserToEngagements =>\n        visitedUserToEngagements\n          .mapValues(engagements => engagements.size)\n      }\n  }\n\n  def getRepeatedVisitedAccounts(params: Params, targetUserId: Long): Stitch[Map[Long, Int]] = {\n    var results: Stitch[Map[Long, Int]] = Stitch.value(Map.empty)\n    if (params.getBoolean(RepeatedProfileVisitsParams.UseOnlineDataset)) {\n      results = applyWithOnlineData(targetUserId)\n    } else {\n      results = applyWithOfflineDataset(targetUserId)\n    }\n    // Only keep users that had non-zero engagement counts.\n    results.map(_.filter(input => input._2 > 0))\n  }\n\n  def getRecommendations(params: Params, userId: Long): Stitch[Seq[CandidateUser]] = {\n    val recommendationThreshold = params.getInt(RepeatedProfileVisitsParams.RecommendationThreshold)\n    val bucketingThreshold = params.getInt(RepeatedProfileVisitsParams.BucketingThreshold)\n\n    // Get the list of repeatedly visited profilts. Only keep accounts with >= bucketingThreshold visits.\n    val repeatedVisitedAccountsStitch: Stitch[Map[Long, Int]] =\n      getRepeatedVisitedAccounts(params, userId).map(_.filter(kv => kv._2 >= bucketingThreshold))\n\n    repeatedVisitedAccountsStitch.map { candidates =>\n      // Now check if we should includeCandidates (e.g. whether user is in control bucket or treatment buckets).\n      if (candidates.isEmpty) {\n        // User has not visited any accounts above bucketing threshold. We will not bucket user into experiment. Just\n        // don't return no candidates.\n        noRepeatedProfileVisitsAboveBucketingThresholdCounter.incr()\n        Seq.empty\n      } else {\n        hasRepeatedProfileVisitsAboveBucketingThresholdCounter.incr()\n        if (!params.getBoolean(RepeatedProfileVisitsParams.IncludeCandidates)) {\n          // User has reached bucketing criteria. We check whether to include candidates (e.g. checking which bucket\n          // the user is in for the experiment). In this case the user is in a bucket to not include any candidates.\n          noIncludeCandidatesCounter.incr()\n          Seq.empty\n        } else {\n          includeCandidatesCounter.incr()\n          // We should include candidates. Include any candidates above recommendation thresholds.\n          val outputCandidatesSeq = candidates\n            .filter(kv => kv._2 >= recommendationThreshold).map { kv =>\n              val user = kv._1\n              val visitCount = kv._2\n              CandidateUser(user, Some(visitCount.toDouble))\n                .withCandidateSource(RepeatedProfileVisitsSource.Identifier)\n            }.toSeq\n          if (outputCandidatesSeq.isEmpty) {\n            noRepeatedProfileVisitsAboveRecommendationsThresholdCounter.incr()\n          } else {\n            hasRepeatedProfileVisitsAboveRecommendationsThresholdCounter.incr()\n          }\n          outputCandidatesSeq\n        }\n      }\n    }\n  }\n\n  override def apply(request: HasParams with HasClientContext): Stitch[Seq[CandidateUser]] = {\n    request.getOptionalUserId\n      .map { userId =>\n        getRecommendations(request.params, userId)\n      }.getOrElse(Stitch.Nil)\n  }\n}\n\nobject RepeatedProfileVisitsSource {\n  val Identifier = CandidateSourceIdentifier(Algorithm.RepeatedProfileVisits.toString)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/salsa/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/real_time_real_graph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"src/thrift/com/twitter/onboarding/relevance/candidates:candidates-scala\",\n        \"strato/config/columns/onboarding/userrecs:userrecs-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/salsa/README.md",
    "content": "# SALSA Candidate Source\nProvides an account expansion based on the SALSA PYMK (People You May Know) algorithm for a given account. The algorithm focuses on the mutual follow and address book graph, making it highly effective at providing good mutual follow recommendations.\n\nThe SALSA algorithm constructs a local graph and performs personalized random walks to identify the best recommendations for the user. The local graph represents the community of users that are most similar to or most relevant to the user, while the personalized random walk identifies the most popular interests among them.\n\nFor each target user, the local graph is a bipartite graph with a left-hand side (LHS) and a right-hand side (RHS). The LHS is built from several sources, including the target user, forward and reverse address books, mutual follows, recent followings, and recent followers. We choose a specified number of top candidates from these sources for each target user with different weights assigned to each source to favor the corresponding source, and build the LHS using the target user and those top candidates. The RHS consists of two parts: the top candidates from the sources mentioned above for the target user and the mutual follows of the other entries in the LHS.\n\nThe random walk starts from the target user in the LHS and adopts a restarting strategy to realize personalization.\n\nIn summary, the SALSA Candidate Source provides an account expansion based on the SALSA PYMK algorithm, utilizing a bipartite graph with personalized random walks to identify the most relevant and interesting recommendations for the user.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/salsa/RecentEngagementDirectFollowSalsaExpansionSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.salsa\n\nimport com.twitter.follow_recommendations.common.clients.real_time_real_graph.RealTimeRealGraphClient\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RecentEngagementDirectFollowSalsaExpansionSource @Inject() (\n  realTimeRealGraphClient: RealTimeRealGraphClient,\n  salsaExpander: SalsaExpander)\n    extends SalsaExpansionBasedCandidateSource[Long](salsaExpander) {\n\n  override val identifier: CandidateSourceIdentifier =\n    RecentEngagementDirectFollowSalsaExpansionSource.Identifier\n\n  override def firstDegreeNodes(target: Long): Stitch[Seq[Long]] = realTimeRealGraphClient\n    .getUsersRecentlyEngagedWith(\n      target,\n      RealTimeRealGraphClient.EngagementScoreMap,\n      includeDirectFollowCandidates = true,\n      includeNonDirectFollowCandidates = false\n    ).map { recentlyFollowed =>\n      recentlyFollowed\n        .take(RecentEngagementDirectFollowSalsaExpansionSource.NumFirstDegreeNodesToRetrieve)\n        .map(_.id)\n    }\n\n  override def maxResults(target: Long): Int =\n    RecentEngagementDirectFollowSalsaExpansionSource.OutputSize\n}\n\nobject RecentEngagementDirectFollowSalsaExpansionSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.RecentEngagementSarusOcCur.toString)\n  val NumFirstDegreeNodesToRetrieve = 10\n  val OutputSize = 200\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/salsa/SalsaExpander.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.salsa\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.strato.generated.client.onboarding.userrecs.SalsaFirstDegreeOnUserClientColumn\nimport com.twitter.strato.generated.client.onboarding.userrecs.SalsaSecondDegreeOnUserClientColumn\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FollowProof\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.stitch.Stitch\nimport com.twitter.wtf.candidate.thriftscala.Candidate\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase class SalsaExpandedCandidate(\n  candidateId: Long,\n  numberOfConnections: Int,\n  totalScore: Double,\n  connectingUsers: Seq[Long]) {\n  def toCandidateUser: CandidateUser =\n    CandidateUser(\n      id = candidateId,\n      score = Some(totalScore),\n      reason = Some(Reason(\n        Some(AccountProof(followProof = Some(FollowProof(connectingUsers, connectingUsers.size))))))\n    )\n}\n\ncase class SimilarUserCandidate(candidateId: Long, score: Double, similarToCandidate: Long)\n\n/**\n * Salsa expander uses pre-computed lists of candidates for each input user id and returns the highest scored candidates in the pre-computed lists as the expansion for the corresponding input id.\n */\n@Singleton\nclass SalsaExpander @Inject() (\n  statsReceiver: StatsReceiver,\n  firstDegreeClient: SalsaFirstDegreeOnUserClientColumn,\n  secondDegreeClient: SalsaSecondDegreeOnUserClientColumn,\n) {\n\n  val stats = statsReceiver.scope(\"salsa_expander\")\n\n  private def similarUsers(\n    input: Seq[Long],\n    neighbors: Seq[Option[Seq[Candidate]]]\n  ): Seq[SalsaExpandedCandidate] = {\n    input\n      .zip(neighbors).flatMap {\n        case (recId, Some(neighbors)) =>\n          neighbors.map(neighbor => SimilarUserCandidate(neighbor.userId, neighbor.score, recId))\n        case _ => Nil\n      }.groupBy(_.candidateId).map {\n        case (key, neighbors) =>\n          val scores = neighbors.map(_.score)\n          val connectingUsers = neighbors\n            .sortBy(-_.score)\n            .take(SalsaExpander.MaxConnectingUsersToOutputPerExpandedCandidate)\n            .map(_.similarToCandidate)\n\n          SalsaExpandedCandidate(key, scores.size, scores.sum, connectingUsers)\n      }\n      .filter(\n        _.numberOfConnections >= math\n          .min(SalsaExpander.MinConnectingUsersThreshold, input.size)\n      )\n      .toSeq\n  }\n\n  def apply(\n    firstDegreeInput: Seq[Long],\n    secondDegreeInput: Seq[Long],\n    maxNumOfCandidatesToReturn: Int\n  ): Stitch[Seq[CandidateUser]] = {\n\n    val firstDegreeNeighborsStitch =\n      Stitch\n        .collect(firstDegreeInput.map(firstDegreeClient.fetcher\n          .fetch(_).map(_.v.map(_.candidates.take(SalsaExpander.MaxDirectNeighbors))))).onSuccess {\n          firstDegreeNeighbors =>\n            stats.stat(\"first_degree_neighbors\").add(firstDegreeNeighbors.flatten.size)\n        }\n\n    val secondDegreeNeighborsStitch =\n      Stitch\n        .collect(\n          secondDegreeInput.map(\n            secondDegreeClient.fetcher\n              .fetch(_).map(\n                _.v.map(_.candidates.take(SalsaExpander.MaxIndirectNeighbors))))).onSuccess {\n          secondDegreeNeighbors =>\n            stats.stat(\"second_degree_neighbors\").add(secondDegreeNeighbors.flatten.size)\n        }\n\n    val neighborStitches =\n      Stitch.join(firstDegreeNeighborsStitch, secondDegreeNeighborsStitch).map {\n        case (first, second) => first ++ second\n      }\n\n    val similarUsersToInput = neighborStitches.map { neighbors =>\n      similarUsers(firstDegreeInput ++ secondDegreeInput, neighbors)\n    }\n\n    similarUsersToInput.map {\n      // Rank the candidate cot users by the combined weights from the connecting users. This is the default original implementation. It is unlikely to have weight ties and thus a second ranking function is not necessary.\n      _.sortBy(-_.totalScore)\n        .take(maxNumOfCandidatesToReturn)\n        .map(_.toCandidateUser)\n    }\n  }\n}\n\nobject SalsaExpander {\n  val MaxDirectNeighbors = 2000\n  val MaxIndirectNeighbors = 2000\n  val MinConnectingUsersThreshold = 2\n  val MaxConnectingUsersToOutputPerExpandedCandidate = 3\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/salsa/SalsaExpansionBasedCandidateSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.salsa\n\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.stitch.Stitch\n\nabstract class SalsaExpansionBasedCandidateSource[Target](salsaExpander: SalsaExpander)\n    extends CandidateSource[Target, CandidateUser] {\n\n  // Define first/second degree as empty sequences in cases of subclasses\n  // that don't implement one or the other.\n  // Example: MagicRecs only uses first degree nodes, and can ignore implementing secondDegreeNodes\n  //\n  // This allows apply(target) to combine both in the base class\n  def firstDegreeNodes(target: Target): Stitch[Seq[Long]] = Stitch.value(Seq())\n\n  def secondDegreeNodes(target: Target): Stitch[Seq[Long]] = Stitch.value(Seq())\n\n  // max number output results\n  def maxResults(target: Target): Int\n\n  override def apply(target: Target): Stitch[Seq[CandidateUser]] = {\n    val nodes = Stitch.join(firstDegreeNodes(target), secondDegreeNodes(target))\n\n    nodes.flatMap {\n      case (firstDegreeCandidates, secondDegreeCandidates) => {\n        salsaExpander(firstDegreeCandidates, secondDegreeCandidates, maxResults(target))\n          .map(_.map(_.withCandidateSource(identifier)).sortBy(-_.score.getOrElse(0.0)))\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"src/thrift/com/twitter/hermit/candidate:hermit-candidate-scala\",\n        \"strato/config/columns/onboarding/userrecs:userrecs-strato-client\",\n        \"strato/config/columns/recommendations/follow2vec:follow2vec-strato-client\",\n        \"strato/config/columns/recommendations/similarity:similarity-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/CacheBasedSimsStore.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims\n\nimport com.twitter.escherbird.util.stitchcache.StitchCache\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.hermit.candidate.thriftscala.Candidates\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.Duration\n\nimport java.lang.{Long => JLong}\n\nclass CacheBasedSimsStore(\n  id: CandidateSourceIdentifier,\n  fetcher: Fetcher[Long, Unit, Candidates],\n  maxCacheSize: Int,\n  cacheTtl: Duration,\n  statsReceiver: StatsReceiver)\n    extends CandidateSource[HasParams with HasSimilarToContext, CandidateUser] {\n\n  override val identifier: CandidateSourceIdentifier = id\n  private def getUsersFromSimsSource(userId: JLong): Stitch[Option[Candidates]] = {\n    fetcher\n      .fetch(userId)\n      .map(_.v)\n  }\n\n  private val simsCache = StitchCache[JLong, Option[Candidates]](\n    maxCacheSize = maxCacheSize,\n    ttl = cacheTtl,\n    statsReceiver = statsReceiver,\n    underlyingCall = getUsersFromSimsSource\n  )\n\n  override def apply(request: HasParams with HasSimilarToContext): Stitch[Seq[CandidateUser]] = {\n    Stitch\n      .traverse(request.similarToUserIds) { userId =>\n        simsCache.readThrough(userId).map { candidatesOpt =>\n          candidatesOpt\n            .map { candidates =>\n              StratoBasedSimsCandidateSource.map(userId, candidates)\n            }.getOrElse(Nil)\n        }\n      }.map(_.flatten.distinct.map(_.withCandidateSource(identifier)))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/DBV2SimsRefreshStore.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims\n\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.generated.client.onboarding.userrecs.NewSimsRefreshOnUserClientColumn\nimport com.twitter.util.Duration\n\nimport javax.inject.Inject\n\n@Singleton\nclass DBV2SimsRefreshStore @Inject() (\n  newSimsRefreshOnUserClientColumn: NewSimsRefreshOnUserClientColumn)\n    extends StratoBasedSimsCandidateSourceWithUnitView(\n      fetcher = newSimsRefreshOnUserClientColumn.fetcher,\n      identifier = DBV2SimsRefreshStore.Identifier)\n\n@Singleton\nclass CachedDBV2SimsRefreshStore @Inject() (\n  newSimsRefreshOnUserClientColumn: NewSimsRefreshOnUserClientColumn,\n  statsReceiver: StatsReceiver)\n    extends CacheBasedSimsStore(\n      id = DBV2SimsRefreshStore.Identifier,\n      fetcher = newSimsRefreshOnUserClientColumn.fetcher,\n      maxCacheSize = DBV2SimsRefreshStore.MaxCacheSize,\n      cacheTtl = DBV2SimsRefreshStore.CacheTTL,\n      statsReceiver = statsReceiver.scope(\"CachedDBV2SimsRefreshStore\", \"cache\")\n    )\n\nobject DBV2SimsRefreshStore {\n  val Identifier = CandidateSourceIdentifier(Algorithm.Sims.toString)\n  val MaxCacheSize = 5000\n  val CacheTTL: Duration = Duration.fromHours(24)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/DBV2SimsStore.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims\n\nimport com.google.inject.Singleton\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.hermit.candidate.thriftscala.Candidates\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.util.Duration\n\nimport javax.inject.Inject\n\n@Singleton\nclass DBV2SimsStore @Inject() (\n  @Named(GuiceNamedConstants.DBV2_SIMS_FETCHER) fetcher: Fetcher[Long, Unit, Candidates])\n    extends StratoBasedSimsCandidateSourceWithUnitView(\n      fetcher,\n      identifier = DBV2SimsStore.Identifier)\n\n@Singleton\nclass CachedDBV2SimsStore @Inject() (\n  @Named(GuiceNamedConstants.DBV2_SIMS_FETCHER) fetcher: Fetcher[Long, Unit, Candidates],\n  statsReceiver: StatsReceiver)\n    extends CacheBasedSimsStore(\n      id = DBV2SimsStore.Identifier,\n      fetcher = fetcher,\n      maxCacheSize = DBV2SimsStore.MaxCacheSize,\n      cacheTtl = DBV2SimsStore.CacheTTL,\n      statsReceiver = statsReceiver.scope(\"CachedDBV2SimsStore\", \"cache\")\n    )\n\nobject DBV2SimsStore {\n  val Identifier = CandidateSourceIdentifier(Algorithm.Sims.toString)\n  val MaxCacheSize = 1000\n  val CacheTTL: Duration = Duration.fromHours(24)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/Follow2vecNearestNeighborsStore.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims\n\nimport com.google.inject.Singleton\nimport com.twitter.follow_recommendations.common.candidate_sources.sims.Follow2vecNearestNeighborsStore.NearestNeighborParamsType\nimport com.twitter.hermit.candidate.thriftscala.Candidate\nimport com.twitter.hermit.candidate.thriftscala.Candidates\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.Fetch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.recommendations.follow2vec.LinearRegressionFollow2vecNearestNeighborsClientColumn\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport javax.inject.Inject\n\n@Singleton\nclass LinearRegressionFollow2vecNearestNeighborsStore @Inject() (\n  linearRegressionFollow2vecNearestNeighborsClientColumn: LinearRegressionFollow2vecNearestNeighborsClientColumn)\n    extends StratoBasedSimsCandidateSource[NearestNeighborParamsType](\n      Follow2vecNearestNeighborsStore.convertFetcher(\n        linearRegressionFollow2vecNearestNeighborsClientColumn.fetcher),\n      view = Follow2vecNearestNeighborsStore.defaultSearchParams,\n      identifier = Follow2vecNearestNeighborsStore.IdentifierF2vLinearRegression\n    )\n\nobject Follow2vecNearestNeighborsStore {\n  // (userid, feature store version for data)\n  type NearestNeighborKeyType = (Long, Long)\n  // (neighbors to be returned, ef value: accuracy / latency tradeoff, distance for filtering)\n  type NearestNeighborParamsType = (Option[Int], Option[Int], Option[Double])\n  // (seq(found neighbor id, score), distance for filtering)\n  type NearestNeighborValueType = (Seq[(Long, Option[Double])], Option[Double])\n\n  val IdentifierF2vLinearRegression: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.LinearRegressionFollow2VecNearestNeighbors.toString)\n\n  val defaultFeatureStoreVersion: Long = 20210708\n  val defaultSearchParams: NearestNeighborParamsType = (None, None, None)\n\n  def convertFetcher(\n    fetcher: Fetcher[NearestNeighborKeyType, NearestNeighborParamsType, NearestNeighborValueType]\n  ): Fetcher[Long, NearestNeighborParamsType, Candidates] = {\n    (key: Long, view: NearestNeighborParamsType) =>\n      {\n        def toCandidates(\n          results: Option[NearestNeighborValueType]\n        ): Option[Candidates] = {\n          results.flatMap { r =>\n            Some(\n              Candidates(\n                key,\n                r._1.map { neighbor =>\n                  Candidate(neighbor._1, neighbor._2.getOrElse(0))\n                }\n              )\n            )\n          }\n        }\n\n        val results: Stitch[Fetch.Result[NearestNeighborValueType]] =\n          fetcher.fetch(key = (key, defaultFeatureStoreVersion), view = view)\n        results.transform {\n          case Return(r) => Stitch.value(Fetch.Result(toCandidates(r.v)))\n          case Throw(e) => Stitch.exception(e)\n        }\n      }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/README.md",
    "content": "# Sims Candidate Source\nOffers various online sources for finding similar accounts based on a given user, whether it is the target user or an account candidate.\n\n## Sims\nThe objective is to identify a list of K users who are similar to a given user. In this scenario, we primarily focus on finding similar users as \"producers\" rather than \"consumers.\" Sims has two steps: candidate generation and ranking.\n\n### Sims Candidate Generation\n\nWith over 700 million users to consider, there are multiple ways to define similarities. Currently, we have three candidate sources for Sims:\n\n**CosineFollow** (based on user-user follow graph): The similarity between two users is defined as the cosine similarity between their followers. Despite sounding simple, computing all-pair similarity on the entire follow graph is computationally challenging. We are currently using the WHIMP algorithm to find the top 1000 similar users for each user ID. This candidate source has the largest coverage, as it can find similar user candidates for more than 700 million users.\n\n**CosineList** (based on user-list membership graph): The similarity between two users is defined as the cosine similarity between the lists they are included as members (e.g., [here](https://twitter.com/jack/lists/memberships) are the lists that @jack is on). The same algorithm as CosineFollow is used.\n\n**Follow2Vec** (essentially Word2Vec on user-user follow graph): We first train the Word2Vec model on follow sequence data to obtain users' embeddings and then find the most similar users based on the similarity of the embeddings. However, we need enough data for each user to learn a meaningful embedding for them, so we can only obtain embeddings for the top 10 million users (currently in production, testing 30 million users). Furthermore, Word2Vec model training is limited by memory and computation as it is trained on a single machine.\n\n##### Cosine Similarity\nA crucial component in Sims is calculating cosine similarities between users based on a user-X (X can be a user, list, or other entities) bipartite graph. This problem is technically challenging and took several years of effort to solve.\n\nThe current implementation uses the algorithm proposed in [When hashes met wedges: A distributed algorithm for finding high similarity vectors. WWW 2017](https://arxiv.org/pdf/1703.01054.pdf)\n\n### Sims Ranking\nAfter the candidate generation step, we can obtain dozens to hundreds of similar user candidates for each user. However, since these candidates come from different algorithms, we need a way to rank them. To do this, we collect user feedback.\n\nWe use the \"Profile Sidebar Impressions & Follow\" (a module with follow suggestions displayed when a user visits a profile page and scrolls down) to collect training data. To alleviate any system bias, we use 4% of traffic to show randomly shuffled candidates to users and collect positive (followed impression) and negative (impression only) data from this traffic. This data is used as an evaluation set. We use a portion of the remaining 96% of traffic for training data, filtering only for sets of impressions that had at least one follow, ensuring that the user taking action was paying attention to the impressions.\n\nThe examples are in the format of (profile_user, candidate_user, label). We add features for profile_users and candidate_users based on some high-level aggregated statistics in a feature dataset provided by the Customer Journey team, as well as features that represent the similarity between the profile_user and candidate_user.\n\nWe employ a multi-tower MLP model and optimize the logistic loss. The model is refreshed weekly using an ML workflow.\n\nWe recompute the candidates and rank them daily. The ranked results are published to the Manhattan dataset.\n\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/SimsExperimentalStore.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims\n\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.generated.client.recommendations.similarity.SimilarUsersBySimsExperimentalOnUserClientColumn\nimport com.twitter.util.Duration\n\nimport javax.inject.Inject\n\n@Singleton\nclass SimsExperimentalStore @Inject() (\n  simsExperimentalOnUserClientColumn: SimilarUsersBySimsExperimentalOnUserClientColumn)\n    extends StratoBasedSimsCandidateSourceWithUnitView(\n      fetcher = simsExperimentalOnUserClientColumn.fetcher,\n      identifier = SimsExperimentalStore.Identifier\n    )\n\n@Singleton\nclass CachedSimsExperimentalStore @Inject() (\n  simsExperimentalOnUserClientColumn: SimilarUsersBySimsExperimentalOnUserClientColumn,\n  statsReceiver: StatsReceiver)\n    extends CacheBasedSimsStore(\n      id = SimsExperimentalStore.Identifier,\n      fetcher = simsExperimentalOnUserClientColumn.fetcher,\n      maxCacheSize = SimsExperimentalStore.MaxCacheSize,\n      cacheTtl = SimsExperimentalStore.CacheTTL,\n      statsReceiver = statsReceiver.scope(\"CachedSimsExperimentalStore\", \"cache\")\n    )\n\nobject SimsExperimentalStore {\n  val Identifier = CandidateSourceIdentifier(Algorithm.Sims.toString)\n  val MaxCacheSize = 1000\n  val CacheTTL: Duration = Duration.fromHours(12)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/SimsSourceFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SimsSourceFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[FSParam[Boolean] with FSName] = Seq(\n    SimsSourceParams.DisableHeavyRanker\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/SimsSourceParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims\n\nimport com.twitter.timelines.configapi.FSParam\n\nobject SimsSourceParams {\n  case object EnableDBV2SimsStore extends FSParam[Boolean](\"sims_source_enable_dbv2_source\", false)\n\n  case object EnableDBV2SimsRefreshStore\n      extends FSParam[Boolean](\"sims_source_enable_dbv2_refresh_source\", false)\n\n  case object EnableExperimentalSimsStore\n      extends FSParam[Boolean](\"sims_source_enable_experimental_source\", false)\n\n  case object DisableHeavyRanker\n      extends FSParam[Boolean](\"sims_source_disable_heavy_ranker\", default = false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/SimsStore.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims\n\nimport com.google.inject.Singleton\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.hermit.candidate.thriftscala.Candidates\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.util.Duration\n\nimport javax.inject.Inject\n\n@Singleton\nclass SimsStore @Inject() (\n  @Named(GuiceNamedConstants.SIMS_FETCHER) fetcher: Fetcher[Long, Unit, Candidates])\n    extends StratoBasedSimsCandidateSourceWithUnitView(fetcher, identifier = SimsStore.Identifier)\n\n@Singleton\nclass CachedSimsStore @Inject() (\n  @Named(GuiceNamedConstants.SIMS_FETCHER) fetcher: Fetcher[Long, Unit, Candidates],\n  statsReceiver: StatsReceiver)\n    extends CacheBasedSimsStore(\n      id = SimsStore.Identifier,\n      fetcher = fetcher,\n      maxCacheSize = SimsStore.MaxCacheSize,\n      cacheTtl = SimsStore.CacheTTL,\n      statsReceiver = statsReceiver.scope(\"CachedSimsStore\", \"cache\")\n    )\n\nobject SimsStore {\n  val Identifier = CandidateSourceIdentifier(Algorithm.Sims.toString)\n  val MaxCacheSize = 50000\n  val CacheTTL: Duration = Duration.fromHours(24)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/StratoBasedSimsCandidateSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims\n\nimport com.twitter.follow_recommendations.common.candidate_sources.base.StratoFetcherSource\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.follow_recommendations.common.models.SimilarToProof\nimport com.twitter.hermit.candidate.thriftscala.Candidates\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\n\nabstract class StratoBasedSimsCandidateSource[U](\n  fetcher: Fetcher[Long, U, Candidates],\n  view: U,\n  override val identifier: CandidateSourceIdentifier)\n    extends StratoFetcherSource[Long, U, Candidates](fetcher, view, identifier) {\n\n  override def map(target: Long, candidates: Candidates): Seq[CandidateUser] =\n    StratoBasedSimsCandidateSource.map(target, candidates)\n}\n\nobject StratoBasedSimsCandidateSource {\n  def map(target: Long, candidates: Candidates): Seq[CandidateUser] = {\n    for {\n      candidate <- candidates.candidates\n    } yield CandidateUser(\n      id = candidate.userId,\n      score = Some(candidate.score),\n      reason = Some(\n        Reason(\n          Some(\n            AccountProof(\n              similarToProof = Some(SimilarToProof(Seq(target)))\n            )\n          )\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/StratoBasedSimsCandidateSourceWithUnitView.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims\n\nimport com.twitter.hermit.candidate.thriftscala.Candidates\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\n\nabstract class StratoBasedSimsCandidateSourceWithUnitView(\n  fetcher: Fetcher[Long, Unit, Candidates],\n  override val identifier: CandidateSourceIdentifier)\n    extends StratoBasedSimsCandidateSource[Unit](fetcher, Unit, identifier)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/SwitchingSimsSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SwitchingSimsSource @Inject() (\n  cachedDBV2SimsStore: CachedDBV2SimsStore,\n  cachedDBV2SimsRefreshStore: CachedDBV2SimsRefreshStore,\n  cachedSimsExperimentalStore: CachedSimsExperimentalStore,\n  cachedSimsStore: CachedSimsStore,\n  statsReceiver: StatsReceiver = NullStatsReceiver)\n    extends CandidateSource[HasParams with HasSimilarToContext, CandidateUser] {\n\n  override val identifier: CandidateSourceIdentifier = SwitchingSimsSource.Identifier\n\n  private val stats = statsReceiver.scope(\"SwitchingSimsSource\")\n  private val dbV2SimsStoreCounter = stats.counter(\"DBV2SimsStore\")\n  private val dbV2SimsRefreshStoreCounter = stats.counter(\"DBV2SimsRefreshStore\")\n  private val simsExperimentalStoreCounter = stats.counter(\"SimsExperimentalStore\")\n  private val simsStoreCounter = stats.counter(\"SimsStore\")\n\n  override def apply(request: HasParams with HasSimilarToContext): Stitch[Seq[CandidateUser]] = {\n    val selectedSimsStore =\n      if (request.params(SimsSourceParams.EnableDBV2SimsStore)) {\n        dbV2SimsStoreCounter.incr()\n        cachedDBV2SimsStore\n      } else if (request.params(SimsSourceParams.EnableDBV2SimsRefreshStore)) {\n        dbV2SimsRefreshStoreCounter.incr()\n        cachedDBV2SimsRefreshStore\n      } else if (request.params(SimsSourceParams.EnableExperimentalSimsStore)) {\n        simsExperimentalStoreCounter.incr()\n        cachedSimsExperimentalStore\n      } else {\n        simsStoreCounter.incr()\n        cachedSimsStore\n      }\n    stats.counter(\"total\").incr()\n    selectedSimsStore(request)\n  }\n}\n\nobject SwitchingSimsSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(Algorithm.Sims.toString)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/real_time_real_graph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/socialgraph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion/DBV2SimsExpansionParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims_expansion\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject DBV2SimsExpansionParams {\n  // Theses divisors are used to calibrate DBv2Sims extension candidates scores\n  case object RecentFollowingSimilarUsersDBV2CalibrateDivisor\n      extends FSBoundedParam[Double](\n        \"sims_expansion_recent_following_similar_users_dbv2_divisor\",\n        default = 1.0d,\n        min = 0.1d,\n        max = 100d)\n  case object RecentEngagementSimilarUsersDBV2CalibrateDivisor\n      extends FSBoundedParam[Double](\n        \"sims_expansion_recent_engagement_similar_users_dbv2_divisor\",\n        default = 1.0d,\n        min = 0.1d,\n        max = 100d)\n  case object DisableHeavyRanker\n      extends FSParam[Boolean](\"sims_expansion_disable_heavy_ranker\", default = false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion/README.md",
    "content": "# Sims Expansion Candidate Source\nprovides similar accounts based on the Sims algorithm for a given set of accounts.\n\nThis is a 2nd-hop expansion, meaning that the input accounts could be a user's recently engaged, followed, or algorithm-generated (such as RealGraph) accounts.\n\nFor more information on Sims and how it is utilized in the Follow Recommendations Service, please refer to the `follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims/README.md` file.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion/RecentEngagementSimilarUsersFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims_expansion\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSParam\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RecentEngagementSimilarUsersFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[FSParam[Boolean]] = Seq(\n    RecentEngagementSimilarUsersParams.FirstDegreeSortEnabled\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion/RecentEngagementSimilarUsersParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims_expansion\n\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject RecentEngagementSimilarUsersParams {\n\n  case object FirstDegreeSortEnabled\n      extends FSParam[Boolean](\n        name = \"sims_expansion_recent_engagement_first_degree_sort\",\n        default = true)\n  case object Aggregator\n      extends FSEnumParam[SimsExpansionSourceAggregatorId.type](\n        name = \"sims_expansion_recent_engagement_aggregator_id\",\n        default = SimsExpansionSourceAggregatorId.Sum,\n        enum = SimsExpansionSourceAggregatorId)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion/RecentEngagementSimilarUsersSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims_expansion\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.candidate_sources.sims.SwitchingSimsSource\nimport com.twitter.follow_recommendations.common.clients.real_time_real_graph.RealTimeRealGraphClient\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.follow_recommendations.common.models.SimilarToProof\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RecentEngagementSimilarUsersSource @Inject() (\n  realTimeRealGraphClient: RealTimeRealGraphClient,\n  switchingSimsSource: SwitchingSimsSource,\n  statsReceiver: StatsReceiver)\n    extends SimsExpansionBasedCandidateSource[HasClientContext with HasParams](\n      switchingSimsSource) {\n  override def maxSecondaryDegreeNodes(req: HasClientContext with HasParams): Int = Int.MaxValue\n\n  override def maxResults(req: HasClientContext with HasParams): Int =\n    RecentEngagementSimilarUsersSource.MaxResults\n\n  override val identifier: CandidateSourceIdentifier = RecentEngagementSimilarUsersSource.Identifier\n  private val stats = statsReceiver.scope(identifier.name)\n  private val calibratedScoreCounter = stats.counter(\"calibrated_scores_counter\")\n\n  override def scoreCandidate(sourceScore: Double, similarToScore: Double): Double = {\n    sourceScore * similarToScore\n  }\n\n  override def calibrateDivisor(req: HasClientContext with HasParams): Double = {\n    req.params(DBV2SimsExpansionParams.RecentEngagementSimilarUsersDBV2CalibrateDivisor)\n  }\n\n  override def calibrateScore(\n    candidateScore: Double,\n    req: HasClientContext with HasParams\n  ): Double = {\n    calibratedScoreCounter.incr()\n    candidateScore / calibrateDivisor(req)\n  }\n\n  /**\n   * fetch first degree nodes given request\n   */\n  override def firstDegreeNodes(\n    target: HasClientContext with HasParams\n  ): Stitch[Seq[CandidateUser]] = {\n    target.getOptionalUserId\n      .map { userId =>\n        realTimeRealGraphClient\n          .getUsersRecentlyEngagedWith(\n            userId,\n            RealTimeRealGraphClient.EngagementScoreMap,\n            includeDirectFollowCandidates = true,\n            includeNonDirectFollowCandidates = true\n          ).map(_.sortBy(-_.score.getOrElse(0.0d))\n            .take(RecentEngagementSimilarUsersSource.MaxFirstDegreeNodes))\n      }.getOrElse(Stitch.Nil)\n  }\n\n  override def aggregateAndScore(\n    request: HasClientContext with HasParams,\n    firstDegreeToSecondDegreeNodesMap: Map[CandidateUser, Seq[SimilarUser]]\n  ): Stitch[Seq[CandidateUser]] = {\n\n    val inputNodes = firstDegreeToSecondDegreeNodesMap.keys.map(_.id).toSet\n    val aggregator = request.params(RecentEngagementSimilarUsersParams.Aggregator) match {\n      case SimsExpansionSourceAggregatorId.Max =>\n        SimsExpansionBasedCandidateSource.ScoreAggregator.Max\n      case SimsExpansionSourceAggregatorId.Sum =>\n        SimsExpansionBasedCandidateSource.ScoreAggregator.Sum\n      case SimsExpansionSourceAggregatorId.MultiDecay =>\n        SimsExpansionBasedCandidateSource.ScoreAggregator.MultiDecay\n    }\n\n    val groupedCandidates = firstDegreeToSecondDegreeNodesMap.values.flatten\n      .filterNot(c => inputNodes.contains(c.candidateId))\n      .groupBy(_.candidateId)\n      .map {\n        case (id, candidates) =>\n          // Different aggregators for final score\n          val finalScore = aggregator(candidates.map(_.score).toSeq)\n          val proofs = candidates.map(_.similarTo).toSet\n\n          CandidateUser(\n            id = id,\n            score = Some(finalScore),\n            reason =\n              Some(Reason(Some(AccountProof(similarToProof = Some(SimilarToProof(proofs.toSeq))))))\n          ).withCandidateSource(identifier)\n      }\n      .toSeq\n      .sortBy(-_.score.getOrElse(0.0d))\n      .take(maxResults(request))\n\n    Stitch.value(groupedCandidates)\n  }\n}\n\nobject RecentEngagementSimilarUsersSource {\n  val Identifier = CandidateSourceIdentifier(Algorithm.RecentEngagementSimilarUser.toString)\n  val MaxFirstDegreeNodes = 10\n  val MaxResults = 200\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion/RecentFollowingSimilarUsersParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims_expansion\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject RecentFollowingSimilarUsersParams {\n  case object MaxFirstDegreeNodes\n      extends FSBoundedParam[Int](\n        name = \"sims_expansion_recent_following_max_first_degree_nodes\",\n        default = 10,\n        min = 0,\n        max = 200)\n  case object MaxSecondaryDegreeExpansionPerNode\n      extends FSBoundedParam[Int](\n        name = \"sims_expansion_recent_following_max_secondary_degree_nodes\",\n        default = 40,\n        min = 0,\n        max = 200)\n  case object MaxResults\n      extends FSBoundedParam[Int](\n        name = \"sims_expansion_recent_following_max_results\",\n        default = 200,\n        min = 0,\n        max = 200)\n  case object TimestampIntegrated\n      extends FSParam[Boolean](\n        name = \"sims_expansion_recent_following_integ_timestamp\",\n        default = false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion/RecentFollowingSimilarUsersSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims_expansion\n\nimport com.google.inject.Singleton\nimport com.twitter.follow_recommendations.common.candidate_sources.sims.SwitchingSimsSource\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedUserIds\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.clients.socialgraph.SocialGraphClient\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport javax.inject.Inject\n\nobject RecentFollowingSimilarUsersSource {\n\n  val Identifier = CandidateSourceIdentifier(Algorithm.NewFollowingSimilarUser.toString)\n}\n\n@Singleton\nclass RecentFollowingSimilarUsersSource @Inject() (\n  socialGraph: SocialGraphClient,\n  switchingSimsSource: SwitchingSimsSource,\n  statsReceiver: StatsReceiver)\n    extends SimsExpansionBasedCandidateSource[\n      HasParams with HasRecentFollowedUserIds with HasClientContext\n    ](switchingSimsSource) {\n\n  val identifier = RecentFollowingSimilarUsersSource.Identifier\n  private val stats = statsReceiver.scope(identifier.name)\n  private val maxResultsStats = stats.scope(\"max_results\")\n  private val calibratedScoreCounter = stats.counter(\"calibrated_scores_counter\")\n\n  override def firstDegreeNodes(\n    request: HasParams with HasRecentFollowedUserIds with HasClientContext\n  ): Stitch[Seq[CandidateUser]] = {\n    if (request.params(RecentFollowingSimilarUsersParams.TimestampIntegrated)) {\n      val recentFollowedUserIdsWithTimeStitch =\n        socialGraph.getRecentFollowedUserIdsWithTime(request.clientContext.userId.get)\n\n      recentFollowedUserIdsWithTimeStitch.map { results =>\n        val first_degree_nodes = results\n          .sortBy(-_.timeInMs).take(\n            request.params(RecentFollowingSimilarUsersParams.MaxFirstDegreeNodes))\n        val max_timestamp = first_degree_nodes.head.timeInMs\n        first_degree_nodes.map {\n          case userIdWithTime =>\n            CandidateUser(\n              userIdWithTime.userId,\n              score = Some(userIdWithTime.timeInMs.toDouble / max_timestamp))\n        }\n      }\n    } else {\n      Stitch.value(\n        request.recentFollowedUserIds\n          .getOrElse(Nil).take(\n            request.params(RecentFollowingSimilarUsersParams.MaxFirstDegreeNodes)).map(\n            CandidateUser(_, score = Some(1.0)))\n      )\n    }\n  }\n\n  override def maxSecondaryDegreeNodes(\n    req: HasParams with HasRecentFollowedUserIds with HasClientContext\n  ): Int = {\n    req.params(RecentFollowingSimilarUsersParams.MaxSecondaryDegreeExpansionPerNode)\n  }\n\n  override def maxResults(\n    req: HasParams with HasRecentFollowedUserIds with HasClientContext\n  ): Int = {\n    val firstDegreeNodes = req.params(RecentFollowingSimilarUsersParams.MaxFirstDegreeNodes)\n    val maxResultsNum = req.params(RecentFollowingSimilarUsersParams.MaxResults)\n    maxResultsStats\n      .stat(\n        s\"RecentFollowingSimilarUsersSource_firstDegreeNodes_${firstDegreeNodes}_maxResults_${maxResultsNum}\")\n      .add(1)\n    maxResultsNum\n  }\n\n  override def scoreCandidate(sourceScore: Double, similarToScore: Double): Double = {\n    sourceScore * similarToScore\n  }\n\n  override def calibrateDivisor(\n    req: HasParams with HasRecentFollowedUserIds with HasClientContext\n  ): Double = {\n    req.params(DBV2SimsExpansionParams.RecentFollowingSimilarUsersDBV2CalibrateDivisor)\n  }\n\n  override def calibrateScore(\n    candidateScore: Double,\n    req: HasParams with HasRecentFollowedUserIds with HasClientContext\n  ): Double = {\n    calibratedScoreCounter.incr()\n    candidateScore / calibrateDivisor(req)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion/RecentStrongEngagementDirectFollowSimilarUsersSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims_expansion\n\nimport com.google.inject.Singleton\nimport com.twitter.follow_recommendations.common.candidate_sources.sims.SwitchingSimsSource\nimport com.twitter.follow_recommendations.common.clients.real_time_real_graph.RealTimeRealGraphClient\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\nimport javax.inject.Inject\n\n@Singleton\nclass RecentStrongEngagementDirectFollowSimilarUsersSource @Inject() (\n  realTimeRealGraphClient: RealTimeRealGraphClient,\n  switchingSimsSource: SwitchingSimsSource)\n    extends SimsExpansionBasedCandidateSource[HasClientContext with HasParams](\n      switchingSimsSource) {\n\n  val identifier = RecentStrongEngagementDirectFollowSimilarUsersSource.Identifier\n\n  override def firstDegreeNodes(\n    request: HasClientContext with HasParams\n  ): Stitch[Seq[CandidateUser]] = request.getOptionalUserId\n    .map { userId =>\n      realTimeRealGraphClient\n        .getUsersRecentlyEngagedWith(\n          userId,\n          RealTimeRealGraphClient.StrongEngagementScoreMap,\n          includeDirectFollowCandidates = true,\n          includeNonDirectFollowCandidates = false\n        ).map(_.take(RecentStrongEngagementDirectFollowSimilarUsersSource.MaxFirstDegreeNodes))\n    }.getOrElse(Stitch.Nil)\n\n  override def maxSecondaryDegreeNodes(request: HasClientContext with HasParams): Int = Int.MaxValue\n\n  override def maxResults(request: HasClientContext with HasParams): Int =\n    RecentStrongEngagementDirectFollowSimilarUsersSource.MaxResults\n\n  override def scoreCandidate(sourceScore: Double, similarToScore: Double): Double = {\n    sourceScore * similarToScore\n  }\n\n  override def calibrateDivisor(req: HasClientContext with HasParams): Double = 1.0d\n}\n\nobject RecentStrongEngagementDirectFollowSimilarUsersSource {\n  val Identifier = CandidateSourceIdentifier(Algorithm.RecentStrongEngagementSimilarUser.toString)\n  val MaxFirstDegreeNodes = 10\n  val MaxResults = 200\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion/SimsExpansionBasedCandidateSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims_expansion\n\nimport com.twitter.follow_recommendations.common.candidate_sources.base.TwoHopExpansionCandidateSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims.SwitchingSimsSource\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.follow_recommendations.common.models.SimilarToProof\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport scala.math._\n\ncase class SimilarUser(candidateId: Long, similarTo: Long, score: Double)\n\nabstract class SimsExpansionBasedCandidateSource[-Target <: HasParams](\n  switchingSimsSource: SwitchingSimsSource)\n    extends TwoHopExpansionCandidateSource[Target, CandidateUser, SimilarUser, CandidateUser] {\n\n  // max number secondary degree nodes per first degree node\n  def maxSecondaryDegreeNodes(req: Target): Int\n\n  // max number output results\n  def maxResults(req: Target): Int\n\n  // scorer to score candidate based on first and second degree node scores\n  def scoreCandidate(source: Double, similarToScore: Double): Double\n\n  def calibrateDivisor(req: Target): Double\n\n  def calibrateScore(candidateScore: Double, req: Target): Double = {\n    candidateScore / calibrateDivisor(req)\n  }\n\n  override def secondaryDegreeNodes(req: Target, node: CandidateUser): Stitch[Seq[SimilarUser]] = {\n    switchingSimsSource(new HasParams with HasSimilarToContext {\n      override val similarToUserIds = Seq(node.id)\n      override val params = (req.params)\n    }).map(_.take(maxSecondaryDegreeNodes(req)).map { candidate =>\n      SimilarUser(\n        candidate.id,\n        node.id,\n        (node.score, candidate.score) match {\n          // only calibrated sims expanded candidates scores\n          case (Some(nodeScore), Some(candidateScore)) =>\n            calibrateScore(scoreCandidate(nodeScore, candidateScore), req)\n          case (Some(nodeScore), _) => nodeScore\n          // NewFollowingSimilarUser will enter this case\n          case _ => calibrateScore(candidate.score.getOrElse(0.0), req)\n        }\n      )\n    })\n  }\n\n  override def aggregateAndScore(\n    request: Target,\n    firstDegreeToSecondDegreeNodesMap: Map[CandidateUser, Seq[SimilarUser]]\n  ): Stitch[Seq[CandidateUser]] = {\n\n    val inputNodes = firstDegreeToSecondDegreeNodesMap.keys.map(_.id).toSet\n    val aggregator = request.params(SimsExpansionSourceParams.Aggregator) match {\n      case SimsExpansionSourceAggregatorId.Max =>\n        SimsExpansionBasedCandidateSource.ScoreAggregator.Max\n      case SimsExpansionSourceAggregatorId.Sum =>\n        SimsExpansionBasedCandidateSource.ScoreAggregator.Sum\n      case SimsExpansionSourceAggregatorId.MultiDecay =>\n        SimsExpansionBasedCandidateSource.ScoreAggregator.MultiDecay\n    }\n\n    val groupedCandidates = firstDegreeToSecondDegreeNodesMap.values.flatten\n      .filterNot(c => inputNodes.contains(c.candidateId))\n      .groupBy(_.candidateId)\n      .map {\n        case (id, candidates) =>\n          // Different aggregators for final score\n          val finalScore = aggregator(candidates.map(_.score).toSeq)\n          val proofs = candidates.map(_.similarTo).toSet\n\n          CandidateUser(\n            id = id,\n            score = Some(finalScore),\n            reason =\n              Some(Reason(Some(AccountProof(similarToProof = Some(SimilarToProof(proofs.toSeq))))))\n          ).withCandidateSource(identifier)\n      }\n      .toSeq\n      .sortBy(-_.score.getOrElse(0.0d))\n      .take(maxResults(request))\n\n    Stitch.value(groupedCandidates)\n  }\n}\n\nobject SimsExpansionBasedCandidateSource {\n  object ScoreAggregator {\n    val Max: Seq[Double] => Double = (candidateScores: Seq[Double]) => {\n      if (candidateScores.size > 0) candidateScores.max else 0.0\n    }\n    val Sum: Seq[Double] => Double = (candidateScores: Seq[Double]) => {\n      candidateScores.sum\n    }\n    val MultiDecay: Seq[Double] => Double = (candidateScores: Seq[Double]) => {\n      val alpha = 0.1\n      val beta = 0.1\n      val gamma = 0.8\n      val decay_scores: Seq[Double] =\n        candidateScores\n          .sorted(Ordering[Double].reverse)\n          .zipWithIndex\n          .map(x => x._1 * pow(gamma, x._2))\n      alpha * candidateScores.max + decay_scores.sum + beta * candidateScores.size\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion/SimsExpansionFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims_expansion\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SimsExpansionFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val intFSParams: Seq[FSBoundedParam[Int]] = Seq(\n    RecentFollowingSimilarUsersParams.MaxFirstDegreeNodes,\n    RecentFollowingSimilarUsersParams.MaxSecondaryDegreeExpansionPerNode,\n    RecentFollowingSimilarUsersParams.MaxResults\n  )\n\n  override val doubleFSParams: Seq[FSBoundedParam[Double]] = Seq(\n    DBV2SimsExpansionParams.RecentFollowingSimilarUsersDBV2CalibrateDivisor,\n    DBV2SimsExpansionParams.RecentEngagementSimilarUsersDBV2CalibrateDivisor\n  )\n\n  override val booleanFSParams: Seq[FSParam[Boolean]] = Seq(\n    DBV2SimsExpansionParams.DisableHeavyRanker,\n    RecentFollowingSimilarUsersParams.TimestampIntegrated\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion/SimsExpansionSourceParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.sims_expansion\nimport com.twitter.timelines.configapi.FSEnumParam\n\nobject SimsExpansionSourceParams {\n  case object Aggregator\n      extends FSEnumParam[SimsExpansionSourceAggregatorId.type](\n        name = \"sims_expansion_aggregator_id\",\n        default = SimsExpansionSourceAggregatorId.Sum,\n        enum = SimsExpansionSourceAggregatorId)\n}\n\nobject SimsExpansionSourceAggregatorId extends Enumeration {\n  type AggregatorId = Value\n  val Sum: AggregatorId = Value(\"sum\")\n  val Max: AggregatorId = Value(\"max\")\n  val MultiDecay: AggregatorId = Value(\"multi_decay\")\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/socialgraph/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/socialgraph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/socialgraph/README.md",
    "content": "# Social Graph Candidate Source\nProvides candidate expansion based on the Twitter social graph.\n\nCurrently, the expansion is mainly based on the \"follow\" social graph edge, which allows the service to identify recent following accounts for a given set of accounts. The input accounts could be a user's recent following, engaged, or other related accounts.\n\nIn summary, the Social Graph Candidate Source utilizes the Twitter social graph to provide candidate expansion, primarily focusing on recent following accounts.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/socialgraph/RecentFollowingRecentFollowingExpansionSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.socialgraph\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.candidate_sources.base.TwoHopExpansionCandidateSource\nimport com.twitter.follow_recommendations.common.clients.socialgraph.RecentEdgesQuery\nimport com.twitter.follow_recommendations.common.clients.socialgraph.SocialGraphClient\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FollowProof\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedUserIds\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.inject.Logging\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.socialgraph.thriftscala.RelationshipType\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * This candidate source is a two hop expansion over the follow graph. The candidates returned from this source is the users that get followed by the target user's recent followings. It will call SocialGraph `n` + 1 times where `n` is the number of recent followings of the target user to be considered.\n */\n@Singleton\nclass RecentFollowingRecentFollowingExpansionSource @Inject() (\n  socialGraphClient: SocialGraphClient,\n  statsReceiver: StatsReceiver)\n    extends TwoHopExpansionCandidateSource[\n      HasParams with HasRecentFollowedUserIds,\n      Long,\n      Long,\n      CandidateUser\n    ]\n    with Logging {\n\n  override val identifier: CandidateSourceIdentifier =\n    RecentFollowingRecentFollowingExpansionSource.Identifier\n\n  val stats = statsReceiver.scope(identifier.name)\n\n  override def firstDegreeNodes(\n    target: HasParams with HasRecentFollowedUserIds\n  ): Stitch[Seq[Long]] = Stitch.value(\n    target.recentFollowedUserIds\n      .getOrElse(Nil).take(\n        RecentFollowingRecentFollowingExpansionSource.NumFirstDegreeNodesToRetrieve)\n  )\n\n  override def secondaryDegreeNodes(\n    target: HasParams with HasRecentFollowedUserIds,\n    node: Long\n  ): Stitch[Seq[Long]] = socialGraphClient\n    .getRecentEdgesCached(\n      RecentEdgesQuery(\n        node,\n        Seq(RelationshipType.Following),\n        Some(RecentFollowingRecentFollowingExpansionSource.NumSecondDegreeNodesToRetrieve)),\n      useCachedStratoColumn =\n        target.params(RecentFollowingRecentFollowingExpansionSourceParams.CallSgsCachedColumn)\n    ).map(\n      _.take(RecentFollowingRecentFollowingExpansionSource.NumSecondDegreeNodesToRetrieve)).rescue {\n      case exception: Exception =>\n        logger.warn(\n          s\"${this.getClass} fails to retrieve second degree nodes for first degree node $node\",\n          exception)\n        stats.counter(\"second_degree_expansion_error\").incr()\n        Stitch.Nil\n    }\n\n  override def aggregateAndScore(\n    target: HasParams with HasRecentFollowedUserIds,\n    firstDegreeToSecondDegreeNodesMap: Map[Long, Seq[Long]]\n  ): Stitch[Seq[CandidateUser]] = {\n    val zipped = firstDegreeToSecondDegreeNodesMap.toSeq.flatMap {\n      case (firstDegreeId, secondDegreeIds) =>\n        secondDegreeIds.map(secondDegreeId => firstDegreeId -> secondDegreeId)\n    }\n    val candidateAndConnections = zipped\n      .groupBy { case (_, secondDegreeId) => secondDegreeId }\n      .mapValues { v => v.map { case (firstDegreeId, _) => firstDegreeId } }\n      .toSeq\n      .sortBy { case (_, connections) => -connections.size }\n      .map {\n        case (candidateId, connections) =>\n          CandidateUser(\n            id = candidateId,\n            score = Some(CandidateUser.DefaultCandidateScore),\n            reason = Some(\n              Reason(\n                Some(AccountProof(followProof = Some(FollowProof(connections, connections.size))))))\n          ).withCandidateSource(identifier)\n      }\n    Stitch.value(candidateAndConnections)\n  }\n}\n\nobject RecentFollowingRecentFollowingExpansionSource {\n  val Identifier = CandidateSourceIdentifier(Algorithm.NewFollowingNewFollowingExpansion.toString)\n\n  val NumFirstDegreeNodesToRetrieve = 5\n  val NumSecondDegreeNodesToRetrieve = 20\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/socialgraph/RecentFollowingRecentFollowingExpansionSourceFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.socialgraph\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RecentFollowingRecentFollowingExpansionSourceFSConfig @Inject() ()\n    extends FeatureSwitchConfig {\n\n  override val booleanFSParams: Seq[FSParam[Boolean] with FSName] = Seq(\n    RecentFollowingRecentFollowingExpansionSourceParams.CallSgsCachedColumn,\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/socialgraph/RecentFollowingRecentFollowingExpansionSourceParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.socialgraph\n\nimport com.twitter.timelines.configapi.FSParam\n\nobject RecentFollowingRecentFollowingExpansionSourceParams {\n  object CallSgsCachedColumn\n      extends FSParam[Boolean](\n        \"recent_following_recent_following_source_call_sgs_cached_column\",\n        true)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/addressbook\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/real_time_real_graph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/stores\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"src/scala/com/twitter/onboarding/relevance/features/strongtie\",\n        \"src/thrift/com/twitter/search/account_search/extended_network:extended_network_users-scala\",\n        \"strato/config/columns/hub:hub-strato-client\",\n        \"strato/config/columns/onboarding/userrecs:userrecs-strato-client\",\n        \"strato/config/columns/search/account_search:account_search-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/BaseOnlineSTPSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedUserIds\nimport com.twitter.follow_recommendations.common.models.STPGraph\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.logging.Logging\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.STPFeatureGenerator\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.STPRecord\n\nabstract class BaseOnlineSTPSource(\n  stpGraphBuilder: STPGraphBuilder,\n  baseStatsReceiver: StatsReceiver)\n    extends CandidateSource[\n      HasClientContext with HasParams with HasRecentFollowedUserIds,\n      CandidateUser\n    ]\n    with Logging {\n\n  protected val statsReceiver: StatsReceiver = baseStatsReceiver.scope(\"online_stp\")\n\n  override val identifier: CandidateSourceIdentifier = BaseOnlineSTPSource.Identifier\n\n  def getCandidates(\n    records: Seq[STPRecord],\n    request: HasClientContext with HasParams with HasRecentFollowedUserIds\n  ): Stitch[Seq[CandidateUser]]\n\n  override def apply(\n    request: HasClientContext with HasParams with HasRecentFollowedUserIds\n  ): Stitch[Seq[CandidateUser]] =\n    request.getOptionalUserId\n      .map { userId =>\n        stpGraphBuilder(request)\n          .flatMap { graph: STPGraph =>\n            logger.debug(graph)\n            val records = STPFeatureGenerator.constructFeatures(\n              userId,\n              graph.firstDegreeEdgeInfoList,\n              graph.secondDegreeEdgeInfoList)\n            getCandidates(records, request)\n          }\n      }.getOrElse(Stitch.Nil)\n}\n\nobject BaseOnlineSTPSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.OnlineStrongTiePredictionRecNoCaching.toString)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/Dbv2StpScorer.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.cortex.deepbird.runtime.prediction_engine.TensorflowPredictionEngine\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.ml.api.Feature.Continuous\nimport com.twitter.ml.api.util.SRichDataRecord\nimport com.twitter.ml.prediction_service.PredictionRequest\nimport com.twitter.stitch.Stitch\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.STPRecord\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.STPRecordAdapter\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/**\n * STP ML ranker trained using DeepBirdV2\n */\n@Singleton\nclass Dbv2StpScorer @Inject() (\n  @Named(GuiceNamedConstants.STP_DBV2_SCORER) tfPredictionEngine: TensorflowPredictionEngine) {\n  def getScoredResponse(record: STPRecord): Stitch[Option[Double]] = {\n    val request: PredictionRequest = new PredictionRequest(\n      STPRecordAdapter.adaptToDataRecord(record))\n    val responseStitch = Stitch.callFuture(tfPredictionEngine.getPrediction(request))\n    responseStitch.map { response =>\n      val richDr = SRichDataRecord(response.getPrediction)\n      richDr.getFeatureValueOpt(new Continuous(\"output\"))\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/EpStpScorer.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.bijection.thrift.BinaryThriftCodec\nimport com.twitter.relevance.ep_model.scorer.EPScorer\nimport com.twitter.relevance.ep_model.scorer.ScorerUtil\nimport com.twitter.relevance.ep_model.thrift\nimport com.twitter.relevance.ep_model.thriftscala.EPScoringOptions\nimport com.twitter.relevance.ep_model.thriftscala.EPScoringRequest\nimport com.twitter.relevance.ep_model.thriftscala.EPScoringResponse\nimport com.twitter.relevance.ep_model.thriftscala.Record\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\nimport scala.util.Success\n\ncase class ScoredResponse(score: Double, featuresBreakdown: Option[String] = None)\n\n/**\n * STP ML ranker trained using prehistoric ML framework\n */\n@Singleton\nclass EpStpScorer @Inject() (epScorer: EPScorer) {\n  private def getScore(responses: List[EPScoringResponse]): Option[ScoredResponse] =\n    responses.headOption\n      .flatMap { response =>\n        response.scores.flatMap {\n          _.headOption.map(score => ScoredResponse(ScorerUtil.normalize(score)))\n        }\n      }\n\n  def getScoredResponse(\n    record: Record,\n    details: Boolean = false\n  ): Stitch[Option[ScoredResponse]] = {\n    val scoringOptions = EPScoringOptions(\n      addFeaturesBreakDown = details,\n      addTransformerIntermediateRecords = details\n    )\n    val request = EPScoringRequest(auxFeatures = Some(Seq(record)), options = Some(scoringOptions))\n\n    Stitch.callFuture(\n      BinaryThriftCodec[thrift.EPScoringRequest]\n        .invert(BinaryScalaCodec(EPScoringRequest).apply(request))\n        .map { thriftRequest: thrift.EPScoringRequest =>\n          val responsesF = epScorer\n            .score(List(thriftRequest).asJava)\n            .map(\n              _.asScala.toList\n                .map(response =>\n                  BinaryScalaCodec(EPScoringResponse)\n                    .invert(BinaryThriftCodec[thrift.EPScoringResponse].apply(response)))\n                .collect { case Success(response) => response }\n            )\n          responsesF.map(getScore)\n        }\n        .getOrElse(Future(None)))\n  }\n}\n\nobject EpStpScorer {\n  val WithFeaturesBreakDown = false\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/MutualFollowStrongTiePredictionSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.follow_recommendations.common.clients.socialgraph.RecentEdgesQuery\nimport com.twitter.follow_recommendations.common.clients.socialgraph.SocialGraphClient\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedUserIds\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.socialgraph.thriftscala.RelationshipType\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.userrecs.StrongTiePredictionFeaturesOnUserClientColumn\nimport javax.inject.Singleton\nimport javax.inject.Inject\n\n/**\n * Returns mutual follows. It first gets mutual follows from recent 100 follows and followers, and then unions this\n * with mutual follows from STP features dataset.\n */\n@Singleton\nclass MutualFollowStrongTiePredictionSource @Inject() (\n  sgsClient: SocialGraphClient,\n  strongTiePredictionFeaturesOnUserClientColumn: StrongTiePredictionFeaturesOnUserClientColumn)\n    extends CandidateSource[HasClientContext with HasRecentFollowedUserIds, CandidateUser] {\n  val identifier: CandidateSourceIdentifier =\n    MutualFollowStrongTiePredictionSource.Identifier\n\n  override def apply(\n    target: HasClientContext with HasRecentFollowedUserIds\n  ): Stitch[Seq[CandidateUser]] = {\n    target.getOptionalUserId match {\n      case Some(userId) =>\n        val newFollowings = target.recentFollowedUserIds\n          .getOrElse(Nil)\n          .take(MutualFollowStrongTiePredictionSource.NumOfRecentFollowings)\n        val newFollowersStitch =\n          sgsClient\n            .getRecentEdges(RecentEdgesQuery(userId, Seq(RelationshipType.FollowedBy))).map(\n              _.take(MutualFollowStrongTiePredictionSource.NumOfRecentFollowers))\n        val mutualFollowsStitch =\n          strongTiePredictionFeaturesOnUserClientColumn.fetcher\n            .fetch(userId).map(_.v.flatMap(_.topMutualFollows.map(_.map(_.userId))).getOrElse(Nil))\n\n        Stitch.join(newFollowersStitch, mutualFollowsStitch).map {\n          case (newFollowers, mutualFollows) => {\n            (newFollowings.intersect(newFollowers) ++ mutualFollows).distinct\n              .map(id => CandidateUser(id, Some(CandidateUser.DefaultCandidateScore)))\n          }\n        }\n      case _ => Stitch.Nil\n    }\n  }\n}\n\nobject MutualFollowStrongTiePredictionSource {\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.MutualFollowSTP.toString)\n  val NumOfRecentFollowings = 100\n  val NumOfRecentFollowers = 100\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/OfflineMutualFollowExpansionSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.google.inject.Singleton\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.generated.client.onboarding.userrecs.MutualFollowExpansionClientColumn\nimport javax.inject.Inject\n\n/**\n * A source that finds the mutual follows of one's mutual follows that one isn't following already.\n */\n@Singleton\nclass OfflineMutualFollowExpansionSource @Inject() (\n  column: MutualFollowExpansionClientColumn)\n    extends OfflineStrongTiePredictionBaseSource(column.fetcher) {\n  override val identifier: CandidateSourceIdentifier =\n    OfflineMutualFollowExpansionSource.Identifier\n}\n\nobject OfflineMutualFollowExpansionSource {\n  val Identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(Algorithm.MutualFollowExpansion.toString)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/OfflineStpSourceFsConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass OfflineStpSourceFsConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[FSParam[Boolean] with FSName] = Seq(\n    OfflineStpSourceParams.UseDenserPmiMatrix\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/OfflineStpSourceParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.timelines.configapi.FSParam\n\nobject OfflineStpSourceParams {\n  // If enabled, we use the new, denser version of PMI matrix to generate OfflineSTP candidates.\n  case object UseDenserPmiMatrix\n      extends FSParam[Boolean](\"offline_stp_source_use_denser_pmi_matrix\", default = false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/OfflineStpSourceWithDensePmiMatrix.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.google.inject.Singleton\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.generated.client.hub.PpmiDenseMatrixCandidatesClientColumn\nimport javax.inject.Inject\n\n/**\n * Main source for strong-tie-prediction candidates generated offline.\n */\n@Singleton\nclass OfflineStpSourceWithDensePmiMatrix @Inject() (\n  stpColumn: PpmiDenseMatrixCandidatesClientColumn)\n    extends OfflineStrongTiePredictionBaseSource(stpColumn.fetcher) {\n  override val identifier: CandidateSourceIdentifier = OfflineStpSourceWithDensePmiMatrix.Identifier\n}\n\nobject OfflineStpSourceWithDensePmiMatrix {\n  val Identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(Algorithm.StrongTiePredictionRec.toString)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/OfflineStpSourceWithLegacyPmiMatrix.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.google.inject.Singleton\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.generated.client.onboarding.userrecs.StrongTiePredictionClientColumn\nimport javax.inject.Inject\n\n/**\n * Main source for strong-tie-prediction candidates generated offline.\n */\n@Singleton\nclass OfflineStpSourceWithLegacyPmiMatrix @Inject() (\n  stpColumn: StrongTiePredictionClientColumn)\n    extends OfflineStrongTiePredictionBaseSource(stpColumn.fetcher) {\n  override val identifier: CandidateSourceIdentifier =\n    OfflineStpSourceWithLegacyPmiMatrix.Identifier\n}\n\nobject OfflineStpSourceWithLegacyPmiMatrix {\n  val Identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(Algorithm.StrongTiePredictionRec.toString)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/OfflineStrongTiePredictionBaseSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FollowProof\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.hermit.stp.thriftscala.STPResult\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.timelines.configapi.HasParams\n\n/** Base class that all variants of our offline stp dataset can extend. Assumes the same STPResult\n *  value in the key and converts the result into the necessary internal model.\n */\nabstract class OfflineStrongTiePredictionBaseSource(\n  fetcher: Fetcher[Long, Unit, STPResult])\n    extends CandidateSource[HasParams with HasClientContext, CandidateUser] {\n\n  def fetch(\n    target: Long,\n  ): Stitch[Seq[CandidateUser]] = {\n    fetcher\n      .fetch(target)\n      .map { result =>\n        result.v\n          .map { candidates => OfflineStrongTiePredictionBaseSource.map(target, candidates) }\n          .getOrElse(Nil)\n          .map(_.withCandidateSource(identifier))\n      }\n  }\n\n  override def apply(request: HasParams with HasClientContext): Stitch[Seq[CandidateUser]] = {\n    request.getOptionalUserId.map(fetch).getOrElse(Stitch.Nil)\n  }\n}\n\nobject OfflineStrongTiePredictionBaseSource {\n  def map(target: Long, candidates: STPResult): Seq[CandidateUser] = {\n    for {\n      candidate <- candidates.strongTieUsers.sortBy(-_.score)\n    } yield CandidateUser(\n      id = candidate.userId,\n      score = Some(candidate.score),\n      reason = Some(\n        Reason(\n          Some(\n            AccountProof(\n              followProof = candidate.socialProof.map(proof => FollowProof(proof, proof.size))\n            )\n          )\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/OfflineStrongTiePredictionSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.google.inject.Singleton\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.OfflineStpSourceParams.UseDenserPmiMatrix\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.util.logging.Logging\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\n\nobject OfflineStpScore extends Feature[UserCandidate, Option[Double]]\n\n/**\n * Main source for strong-tie-prediction candidates generated offline.\n */\n@Singleton\nclass OfflineStrongTiePredictionSource @Inject() (\n  offlineStpSourceWithLegacyPmiMatrix: OfflineStpSourceWithLegacyPmiMatrix,\n  offlineStpSourceWithDensePmiMatrix: OfflineStpSourceWithDensePmiMatrix)\n    extends CandidateSource[HasParams with HasClientContext, CandidateUser]\n    with Logging {\n  override val identifier: CandidateSourceIdentifier = OfflineStrongTiePredictionSource.Identifier\n\n  override def apply(request: HasParams with HasClientContext): Stitch[Seq[CandidateUser]] = {\n    if (request.params(UseDenserPmiMatrix)) {\n      logger.info(\"Using dense PMI matrix.\")\n      offlineStpSourceWithDensePmiMatrix(request)\n    } else {\n      logger.info(\"Using legacy PMI matrix.\")\n      offlineStpSourceWithLegacyPmiMatrix(request)\n    }\n  }\n}\n\nobject OfflineStrongTiePredictionSource {\n  val Identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(Algorithm.StrongTiePredictionRec.toString)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/OnlineSTPSourceFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass OnlineSTPSourceFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[FSParam[Boolean] with FSName] = Seq(\n    OnlineSTPSourceParams.DisableHeavyRanker,\n    OnlineSTPSourceParams.UseDBv2Scorer,\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/OnlineSTPSourceParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\n\nobject OnlineSTPSourceParams {\n  // This replaces the old scorer module, located at EpStpScorer.scala, with the new scorer, located\n  // at Dbv2StpScorer.scala.\n  case object UseDBv2Scorer\n      extends FSParam[Boolean](\"online_stp_source_dbv2_scorer_enabled\", default = false)\n\n  // For experiments that test the impact of an improved OnlineSTP source, this controls the usage\n  // of the PostNux heavy-ranker. Note that this FS should *NOT* trigger bucket impressions.\n  case object DisableHeavyRanker\n      extends FSParam[Boolean](\"online_stp_source_disable_heavy_ranker\", default = false)\n\n  case object SetPredictionDetails extends Param[Boolean](default = false)\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/OnlineSTPSourceScorer.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedUserIds\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass OnlineSTPSourceScorer @Inject() (\n  onlineSTPSourceWithEPScorer: OnlineSTPSourceWithEPScorer)\n    extends CandidateSource[\n      HasClientContext with HasParams with HasRecentFollowedUserIds,\n      CandidateUser\n    ] {\n\n  override def apply(\n    request: HasClientContext with HasParams with HasRecentFollowedUserIds\n  ): Stitch[Seq[CandidateUser]] = {\n    onlineSTPSourceWithEPScorer(request)\n  }\n\n  override val identifier: CandidateSourceIdentifier = BaseOnlineSTPSource.Identifier\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/OnlineSTPSourceWithDeepbirdV2Scorer.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FollowProof\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedUserIds\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.onboarding.relevance.features.strongtie.{\n  StrongTieFeatures => StrongTieFeaturesWrapper\n}\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.STPRecord\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass OnlineSTPSourceWithDeepbirdV2Scorer @Inject() (\n  dbv2StpScorer: Dbv2StpScorer,\n  stpGraphBuilder: STPGraphBuilder,\n  baseStatReceiver: StatsReceiver)\n    extends BaseOnlineSTPSource(stpGraphBuilder, baseStatReceiver) {\n\n  private val dbv2ScorerUsedCounter = statsReceiver.counter(\"dbv2_scorer_used\")\n  private val dbv2ScorerFailureCounter = statsReceiver.counter(\"dbv2_scorer_failure\")\n  private val dbv2ScorerSuccessCounter = statsReceiver.counter(\"dbv2_scorer_success\")\n\n  override def getCandidates(\n    records: Seq[STPRecord],\n    request: HasClientContext with HasParams with HasRecentFollowedUserIds,\n  ): Stitch[Seq[CandidateUser]] = {\n    val possibleCandidates: Seq[Stitch[Option[CandidateUser]]] = records.map { trainingRecord =>\n      dbv2ScorerUsedCounter.incr()\n      val score = dbv2StpScorer.getScoredResponse(trainingRecord)\n      score.map {\n        case None =>\n          dbv2ScorerFailureCounter.incr()\n          None\n        case Some(scoreVal) =>\n          dbv2ScorerSuccessCounter.incr()\n          Some(\n            CandidateUser(\n              id = trainingRecord.destinationId,\n              score = Some(OnlineSTPSourceWithDeepbirdV2Scorer.logitSubtraction(scoreVal)),\n              reason = Some(\n                Reason(Some(\n                  AccountProof(followProof =\n                    Some(FollowProof(trainingRecord.socialProof, trainingRecord.socialProof.size)))\n                )))\n            ).withCandidateSourceAndFeatures(\n              identifier,\n              Seq(StrongTieFeaturesWrapper(trainingRecord.features)))\n          )\n      }\n    }\n    Stitch.collect(possibleCandidates).map { _.flatten.sortBy(-_.score.getOrElse(0.0)) }\n  }\n}\n\nobject OnlineSTPSourceWithDeepbirdV2Scorer {\n  // The following two variables are the means for the distribution of scores coming from the legacy\n  // and DBv2 OnlineSTP models. We need this to calibrate the DBv2 scores and align the two means.\n  // BQ Link: https://console.cloud.google.com/bigquery?sq=213005704923:e06ac27e4db74385a77a4b538c531f82\n  private val legacyMeanScore = 0.0478208871192468\n  private val dbv2MeanScore = 0.238666097210261\n\n  // In below are the necessary functions to calibrate the scores such that the means are aligned.\n  private val EPS: Double = 1e-8\n  private val e: Double = math.exp(1)\n  private def sigmoid(x: Double): Double = math.pow(e, x) / (math.pow(e, x) + 1)\n  // We add an EPS to the denominator to avoid division by 0.\n  private def logit(x: Double): Double = math.log(x / (1 - x + EPS))\n  def logitSubtraction(x: Double): Double = sigmoid(\n    logit(x) - (logit(dbv2MeanScore) - logit(legacyMeanScore)))\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/OnlineSTPSourceWithEPScorer.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.OnlineSTPSourceParams.SetPredictionDetails\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FollowProof\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedUserIds\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.onboarding.relevance.features.strongtie.{\n  StrongTieFeatures => StrongTieFeaturesWrapper\n}\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.logging.Logging\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.STPRecord\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass OnlineSTPSourceWithEPScorer @Inject() (\n  epStpScorer: EpStpScorer,\n  stpGraphBuilder: STPGraphBuilder,\n  baseStatReceiver: StatsReceiver)\n    extends BaseOnlineSTPSource(stpGraphBuilder, baseStatReceiver)\n    with Logging {\n  private val epScorerUsedCounter = statsReceiver.counter(\"ep_scorer_used\")\n\n  override def getCandidates(\n    records: Seq[STPRecord],\n    request: HasClientContext with HasParams with HasRecentFollowedUserIds,\n  ): Stitch[Seq[CandidateUser]] = {\n    epScorerUsedCounter.incr()\n\n    val possibleCandidates: Seq[Stitch[Option[CandidateUser]]] = records.map { trainingRecord =>\n      val scoredResponse =\n        epStpScorer.getScoredResponse(trainingRecord.record, request.params(SetPredictionDetails))\n      scoredResponse.map(_.map { response: ScoredResponse =>\n        logger.debug(response)\n        CandidateUser(\n          id = trainingRecord.destinationId,\n          score = Some(response.score),\n          reason = Some(\n            Reason(\n              Some(\n                AccountProof(followProof =\n                  Some(FollowProof(trainingRecord.socialProof, trainingRecord.socialProof.size)))\n              )))\n        ).withCandidateSourceAndFeatures(\n          identifier,\n          Seq(StrongTieFeaturesWrapper(trainingRecord.features)))\n      })\n    }\n\n    Stitch.collect(possibleCandidates).map { _.flatten.sortBy(-_.score.getOrElse(0.0)) }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/README.md",
    "content": "# Strong Tie Prediction (STP) Candidate Source\nProvides accounts with a high probability of potential mutual follows between the target user and the candidates.\n\n## STP: Strong Tie Prediction\nSTP refers to the prediction of p(MutualFollow) for a given pair of users, which powers the concept of People You May Know (PYMK).\n\nFor training, positives are existing mutual follows and negatives are mutual follows of your follows. Features help distinguish between friends and friends-of-friends.\n\nFor inference, candidates are the topK mutuals of your follows. These are rescored, and we only send the topN to the product or next re-ranker.\n\n\n### Online STP\nOnline STP generates a pool of candidates who are then ranked via a lightweight ranker.\nIt does this through a two-hop expansion of the mutual follow graph of users, where the first-degree neighbor is another user who has a link with the target user from following types:\n* Mutual Follow\n* Outbound phone contacts\n* Outbound email contacts\n* Inbound phone contacts\n* Inbound email contacts\n\nThe second-degree neighbor can only be a mutual follow link.\n\nCurrently, online STP can only perform the two-hop expansions on new users (<= 30 days since signup) due to compute resource constraints.\n\nFeatures used for the lightweight ranker:\n* realGraphWeight: real graph weight between user and first degree nodes\n* isForwardEmail: whether the candidate is in the user's email book\n* isReverseEmail: whether the user is in the candidate's email book\n* isForwardPhonebook: whether the candidate is in the user's phone book\n* isReversePhonebook: whether the user is in the candidate's phone book\n* numMutualFollowPath: number of mutual follow path between the user and the candidate\n* numLowTweepcredFollowPath: number of mutual low TweepCred path between the user and the candidate\n  * Tweepcred is a social network analysis tool that calculates the influence of Twitter users based on their interactions with other users. The tool uses the PageRank algorithm to rank users based on their influence.\n* hasForwardEmailPath: is there a third user x in the user's email book that connect user -> x -> candidate?\n* hasReverseEmailPath: is there a third user x in the user's reverse email book that connect user -> x -> candidate?\n* hasForwardPhonebookPath: is there a third user x in the user's phonebook that connect user -> x -> candidate?\n* hasReversePhonebookPath: is there a third user x in the user's reverse phonebook that connect user -> x -> candidate?\n\n### Offline STP\nOffline STP  is powered by Pointwise Mutual Information, which measures the association between two users based on Twitter's mutual follow graph.\nAn offline job generates candidates based on the overlap between their Mutual and Addressbook Follows and that of the target user. Candidates are then made available online.\nCandidates in OfflineSTP are \"accounts that have a high overlap of mutually-followed accounts with an account in your follow graph.\"\nThis can potentially mean that OfflineSTP has a bigger reach than OnlineSTP.\nFor example, in the provided diagram, B and C have a high overlap of mutual follows, so it would be considered a candidate for A that is three hops away.\n![img.png](img.png)\n\nOverall, STP is a useful candidate source for generating potential follow recommendations based on strong ties between users, but it should be used in conjunction with other candidate sources and re-rankers to provide a well-rounded set of recommendations for the target user.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/STPFirstDegreeFetcher.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ForwardEmailBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ForwardPhoneBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ReverseEmailBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ReversePhoneBookSource\nimport com.twitter.follow_recommendations.common.clients.real_time_real_graph.RealTimeRealGraphClient\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedUserIds\nimport com.twitter.follow_recommendations.common.models.PotentialFirstDegreeEdge\nimport com.twitter.follow_recommendations.common.stores.LowTweepCredFollowStore\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.hermit.model.Algorithm.Algorithm\nimport com.twitter.inject.Logging\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.Duration\nimport com.twitter.util.Timer\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.FirstDegreeEdge\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.FirstDegreeEdgeInfo\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.FirstDegreeEdgeInfoMonoid\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n// Grabs FirstDegreeNodes from Candidate Sources\n@Singleton\nclass STPFirstDegreeFetcher @Inject() (\n  realTimeGraphClient: RealTimeRealGraphClient,\n  reversePhoneBookSource: ReversePhoneBookSource,\n  reverseEmailBookSource: ReverseEmailBookSource,\n  forwardEmailBookSource: ForwardEmailBookSource,\n  forwardPhoneBookSource: ForwardPhoneBookSource,\n  mutualFollowStrongTiePredictionSource: MutualFollowStrongTiePredictionSource,\n  lowTweepCredFollowStore: LowTweepCredFollowStore,\n  timer: Timer,\n  statsReceiver: StatsReceiver)\n    extends Logging {\n\n  private val stats = statsReceiver.scope(\"STPFirstDegreeFetcher\")\n  private val stitchRequests = stats.scope(\"stitchRequests\")\n  private val allStitchRequests = stitchRequests.counter(\"all\")\n  private val timeoutStitchRequests = stitchRequests.counter(\"timeout\")\n  private val successStitchRequests = stitchRequests.counter(\"success\")\n\n  private implicit val firstDegreeEdgeInfoMonoid: FirstDegreeEdgeInfoMonoid =\n    new FirstDegreeEdgeInfoMonoid\n\n  /**\n   * Used to map from algorithm to the correct fetcher and firstDegreeEdgeInfo.\n   * Afterward, uses fetcher to get candidates and construct the correct FirstDegreeEdgeInfo.\n   * */\n  private def getPotentialFirstEdgesFromFetcher(\n    userId: Long,\n    target: HasClientContext with HasParams with HasRecentFollowedUserIds,\n    algorithm: Algorithm,\n    weight: Double\n  ): Stitch[Seq[PotentialFirstDegreeEdge]] = {\n    val (candidates, edgeInfo) = algorithm match {\n      case Algorithm.MutualFollowSTP =>\n        (\n          mutualFollowStrongTiePredictionSource(target),\n          Some(FirstDegreeEdgeInfo(mutualFollow = true)))\n      case Algorithm.ReverseEmailBookIbis =>\n        (reverseEmailBookSource(target), Some(FirstDegreeEdgeInfo(reverseEmail = true)))\n      case Algorithm.ReversePhoneBook =>\n        (reversePhoneBookSource(target), Some(FirstDegreeEdgeInfo(reversePhone = true)))\n      case Algorithm.ForwardEmailBook =>\n        (forwardEmailBookSource(target), Some(FirstDegreeEdgeInfo(forwardEmail = true)))\n      case Algorithm.ForwardPhoneBook =>\n        (forwardPhoneBookSource(target), Some(FirstDegreeEdgeInfo(forwardPhone = true)))\n      case Algorithm.LowTweepcredFollow =>\n        (\n          lowTweepCredFollowStore.getLowTweepCredUsers(target),\n          Some(FirstDegreeEdgeInfo(lowTweepcredFollow = true)))\n      case _ =>\n        (Stitch.Nil, None)\n    }\n    candidates.map(_.flatMap { candidate =>\n      edgeInfo.map(PotentialFirstDegreeEdge(userId, candidate.id, algorithm, weight, _))\n    })\n  }\n\n  /**\n   * Using the DefaultMap (AlgorithmToScore) we iterate through algorithm/weights to get\n   * candidates with a set weight. Then, given repeating candidates (by candidate id).\n   * Given those candidates we group by the candidateId and sum all below weights and combine\n   * the edgeInfos of into one. Then we choose the candidates with most weight. Finally,\n   * we attach the realGraphWeight score to those candidates.\n   * */\n  def getFirstDegreeEdges(\n    target: HasClientContext with HasParams with HasRecentFollowedUserIds\n  ): Stitch[Seq[FirstDegreeEdge]] = {\n    target.getOptionalUserId\n      .map { userId =>\n        allStitchRequests.incr()\n        val firstEdgesQueryStitch = Stitch\n          .collect(STPFirstDegreeFetcher.DefaultGraphBuilderAlgorithmToScore.map {\n            case (algorithm, candidateWeight) =>\n              getPotentialFirstEdgesFromFetcher(userId, target, algorithm, candidateWeight)\n          }.toSeq)\n          .map(_.flatten)\n\n        val destinationIdsToEdges = firstEdgesQueryStitch\n          .map(_.groupBy(_.connectingId).map {\n            case (destinationId: Long, edges: Seq[PotentialFirstDegreeEdge]) =>\n              val combinedDestScore = edges.map(_.score).sum\n              val combinedEdgeInfo: FirstDegreeEdgeInfo =\n                edges.map(_.edgeInfo).fold(firstDegreeEdgeInfoMonoid.zero) {\n                  (aggregatedInfo, currentInfo) =>\n                    firstDegreeEdgeInfoMonoid.plus(aggregatedInfo, currentInfo)\n                }\n              (destinationId, combinedEdgeInfo, combinedDestScore)\n          }).map(_.toSeq)\n\n        val topDestinationEdges = destinationIdsToEdges.map(_.sortBy {\n          case (_, _, combinedDestScore) => -combinedDestScore\n        }.take(STPFirstDegreeFetcher.MaxNumFirstDegreeEdges))\n\n        Stitch\n          .join(realTimeGraphClient.getRealGraphWeights(userId), topDestinationEdges).map {\n            case (realGraphWeights, topDestinationEdges) =>\n              successStitchRequests.incr()\n              topDestinationEdges.map {\n                case (destinationId, combinedEdgeInfo, _) =>\n                  val updatedEdgeInfo = combinedEdgeInfo.copy(\n                    realGraphWeight = realGraphWeights.getOrElse(destinationId, 0.0),\n                    lowTweepcredFollow =\n                      !combinedEdgeInfo.mutualFollow && combinedEdgeInfo.lowTweepcredFollow\n                  )\n                  FirstDegreeEdge(userId, destinationId, updatedEdgeInfo)\n              }\n          }.within(STPFirstDegreeFetcher.LongTimeoutFetcher)(timer).rescue {\n            case ex =>\n              timeoutStitchRequests.incr()\n              logger.error(\"Exception while loading direct edges in OnlineSTP: \", ex)\n              Stitch.Nil\n          }\n      }.getOrElse(Stitch.Nil)\n  }\n}\n\nobject STPFirstDegreeFetcher {\n  val MaxNumFirstDegreeEdges = 200\n  val DefaultGraphBuilderAlgorithmToScore = Map(\n    Algorithm.MutualFollowSTP -> 10.0,\n    Algorithm.LowTweepcredFollow -> 6.0,\n    Algorithm.ForwardEmailBook -> 7.0,\n    Algorithm.ForwardPhoneBook -> 9.0,\n    Algorithm.ReverseEmailBookIbis -> 5.0,\n    Algorithm.ReversePhoneBook -> 8.0\n  )\n  val LongTimeoutFetcher: Duration = 300.millis\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/STPGraphBuilder.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedUserIds\nimport com.twitter.follow_recommendations.common.models.STPGraph\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass STPGraphBuilder @Inject() (\n  stpFirstDegreeFetcher: STPFirstDegreeFetcher,\n  stpSecondDegreeFetcher: STPSecondDegreeFetcher,\n  statsReceiver: StatsReceiver) {\n  private val stats: StatsReceiver = statsReceiver.scope(this.getClass.getSimpleName)\n  private val firstDegreeStat: Stat = stats.stat(\"first_degree_edges\")\n  private val secondDegreeStat: Stat = stats.stat(\"second_degree_edges\")\n  def apply(\n    target: HasClientContext with HasParams with HasRecentFollowedUserIds\n  ): Stitch[STPGraph] = stpFirstDegreeFetcher\n    .getFirstDegreeEdges(target).flatMap { firstDegreeEdges =>\n      firstDegreeStat.add(firstDegreeEdges.size)\n      stpSecondDegreeFetcher\n        .getSecondDegreeEdges(target, firstDegreeEdges).map { secondDegreeEdges =>\n          secondDegreeStat.add(firstDegreeEdges.size)\n          STPGraph(firstDegreeEdges.toList, secondDegreeEdges.toList)\n        }\n    }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/STPSecondDegreeFetcher.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.twitter.follow_recommendations.common.models.IntermediateSecondDegreeEdge\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.userrecs.StrongTiePredictionFeaturesOnUserClientColumn\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.FirstDegreeEdge\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.SecondDegreeEdge\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.SecondDegreeEdgeInfo\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n// Link to code functionality we're migrating\n@Singleton\nclass STPSecondDegreeFetcher @Inject() (\n  strongTiePredictionFeaturesOnUserClientColumn: StrongTiePredictionFeaturesOnUserClientColumn) {\n\n  private def scoreSecondDegreeEdge(edge: SecondDegreeEdge): (Int, Int, Int) = {\n    def bool2int(b: Boolean): Int = if (b) 1 else 0\n    (\n      -edge.edgeInfo.numMutualFollowPath,\n      -edge.edgeInfo.numLowTweepcredFollowPath,\n      -(bool2int(edge.edgeInfo.forwardEmailPath) + bool2int(edge.edgeInfo.reverseEmailPath) +\n        bool2int(edge.edgeInfo.forwardPhonePath) + bool2int(edge.edgeInfo.reversePhonePath))\n    )\n  }\n\n  // Use each first-degree edge(w/ candidateId) to expand and find mutual follows.\n  // Then, with the mutual follows, group-by candidateId and join edge information\n  // to create secondDegree edges.\n  def getSecondDegreeEdges(\n    target: HasClientContext with HasParams,\n    firstDegreeEdges: Seq[FirstDegreeEdge]\n  ): Stitch[Seq[SecondDegreeEdge]] = {\n    target.getOptionalUserId\n      .map { userId =>\n        val firstDegreeConnectingIds = firstDegreeEdges.map(_.dstId)\n        val firstDegreeEdgeInfoMap = firstDegreeEdges.map(e => (e.dstId, e.edgeInfo)).toMap\n\n        val intermediateSecondDegreeEdgesStitch = Stitch\n          .traverse(firstDegreeConnectingIds) { connectingId =>\n            val stpFeaturesOptStitch = strongTiePredictionFeaturesOnUserClientColumn.fetcher\n              .fetch(connectingId)\n              .map(_.v)\n            stpFeaturesOptStitch.map { stpFeatureOpt =>\n              val intermediateSecondDegreeEdges = for {\n                edgeInfo <- firstDegreeEdgeInfoMap.get(connectingId)\n                stpFeatures <- stpFeatureOpt\n                topSecondDegreeUserIds =\n                  stpFeatures.topMutualFollows\n                    .getOrElse(Nil)\n                    .map(_.userId)\n                    .take(STPSecondDegreeFetcher.MaxNumOfMutualFollows)\n              } yield topSecondDegreeUserIds.map(\n                IntermediateSecondDegreeEdge(connectingId, _, edgeInfo))\n              intermediateSecondDegreeEdges.getOrElse(Nil)\n            }\n          }.map(_.flatten)\n\n        intermediateSecondDegreeEdgesStitch.map { intermediateSecondDegreeEdges =>\n          val secondaryDegreeEdges = intermediateSecondDegreeEdges.groupBy(_.candidateId).map {\n            case (candidateId, intermediateEdges) =>\n              SecondDegreeEdge(\n                srcId = userId,\n                dstId = candidateId,\n                edgeInfo = SecondDegreeEdgeInfo(\n                  numMutualFollowPath = intermediateEdges.count(_.edgeInfo.mutualFollow),\n                  numLowTweepcredFollowPath =\n                    intermediateEdges.count(_.edgeInfo.lowTweepcredFollow),\n                  forwardEmailPath = intermediateEdges.exists(_.edgeInfo.forwardEmail),\n                  reverseEmailPath = intermediateEdges.exists(_.edgeInfo.reverseEmail),\n                  forwardPhonePath = intermediateEdges.exists(_.edgeInfo.forwardPhone),\n                  reversePhonePath = intermediateEdges.exists(_.edgeInfo.reversePhone),\n                  socialProof = intermediateEdges\n                    .filter { e => e.edgeInfo.mutualFollow || e.edgeInfo.lowTweepcredFollow }\n                    .sortBy(-_.edgeInfo.realGraphWeight)\n                    .take(3)\n                    .map { c => (c.connectingId, c.edgeInfo.realGraphWeight) }\n                )\n              )\n          }\n          secondaryDegreeEdges.toSeq\n            .sortBy(scoreSecondDegreeEdge)\n            .take(STPSecondDegreeFetcher.MaxNumSecondDegreeEdges)\n        }\n      }.getOrElse(Stitch.Nil)\n  }\n}\n\nobject STPSecondDegreeFetcher {\n  val MaxNumSecondDegreeEdges = 200\n  val MaxNumOfMutualFollows = 50\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp/SocialProofEnforcedOfflineStrongTiePredictionSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.stp\n\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.candidate_sources.base.SocialProofEnforcedCandidateSource\nimport com.twitter.follow_recommendations.common.transforms.modify_social_proof.ModifySocialProof\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport javax.inject.Inject\n\n@Singleton\nclass SocialProofEnforcedOfflineStrongTiePredictionSource @Inject() (\n  offlineStrongTiePredictionSource: OfflineStrongTiePredictionSource,\n  modifySocialProof: ModifySocialProof,\n  statsReceiver: StatsReceiver)\n    extends SocialProofEnforcedCandidateSource(\n      offlineStrongTiePredictionSource,\n      modifySocialProof,\n      SocialProofEnforcedOfflineStrongTiePredictionSource.MinNumSocialProofsRequired,\n      SocialProofEnforcedOfflineStrongTiePredictionSource.Identifier,\n      statsReceiver)\n\nobject SocialProofEnforcedOfflineStrongTiePredictionSource {\n  val Identifier = CandidateSourceIdentifier(\n    Algorithm.StrongTiePredictionRecWithSocialProof.toString)\n\n  val MinNumSocialProofsRequired = 1\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/top_organic_follows_accounts/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"escherbird/src/scala/com/twitter/escherbird/util/stitchcache\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/geoduck\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/model\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"src/thrift/com/twitter/onboarding/relevance/organic_follows_accounts:organic_follows_accounts-scala\",\n        \"strato/config/columns/onboarding/userrecs:userrecs-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-core/src/main/scala/com/twitter/conversions\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/top_organic_follows_accounts/README.md",
    "content": "# Top Organic Follows Accounts\nProvides the most organically followed (i.e. not followed through the Who-To-Follow module) accounts for a given country.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/top_organic_follows_accounts/TopOrganicFollowsAccountsFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.top_organic_follows_accounts\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.Param\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TopOrganicFollowsAccountsFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] = Seq(\n    TopOrganicFollowsAccountsParams.CandidateSourceEnabled,\n  )\n  override val doubleFSParams: Seq[FSBoundedParam[Double]] = Seq(\n    TopOrganicFollowsAccountsParams.CandidateSourceWeight,\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/top_organic_follows_accounts/TopOrganicFollowsAccountsParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.top_organic_follows_accounts\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumSeqParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject TopOrganicFollowsAccountsParams {\n  // whether or not to fetch TopOrganicFollowsAccounts candidate sources\n  case object CandidateSourceEnabled\n      extends FSParam[Boolean](\"top_organic_follows_accounts_candidate_source_enabled\", false)\n\n  /**\n   *   Contains the logic key for account filtering and ranking. Currently we have 3 main logic keys\n   *    - new_organic_follows: filtering top organically followed accounts followed by new users\n   *    - non_new_organic_follows: filtering top organically followed accounts followed by non new users\n   *    - organic_follows: filtering top organically followed accounts followed by all users\n   *    Mapping of the Logic Id to Logic key is done via @enum AccountsFilteringAndRankingLogic\n   */\n  case object AccountsFilteringAndRankingLogics\n      extends FSEnumSeqParam[AccountsFilteringAndRankingLogicId.type](\n        name = \"top_organic_follows_accounts_filtering_and_ranking_logic_ids\",\n        default = Seq(AccountsFilteringAndRankingLogicId.OrganicFollows),\n        enum = AccountsFilteringAndRankingLogicId)\n\n  case object CandidateSourceWeight\n      extends FSBoundedParam[Double](\n        \"top_organic_follows_accounts_candidate_source_weight\",\n        default = 1200,\n        min = 0.001,\n        max = 2000)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/top_organic_follows_accounts/TopOrganicFollowsAccountsSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.top_organic_follows_accounts\n\nimport com.twitter.escherbird.util.stitchcache.StitchCache\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.candidate_sources.top_organic_follows_accounts.TopOrganicFollowsAccountsParams.AccountsFilteringAndRankingLogics\nimport com.twitter.follow_recommendations.common.candidate_sources.top_organic_follows_accounts.TopOrganicFollowsAccountsParams.CandidateSourceEnabled\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasGeohashAndCountryCode\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.onboarding.relevance.organic_follows_accounts.thriftscala.OrganicFollowsAccounts\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.userrecs.OrganicFollowsAccountsClientColumn\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.Duration\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject AccountsFilteringAndRankingLogicId extends Enumeration {\n  type AccountsFilteringAndRankingLogicId = Value\n\n  val NewOrganicFollows: AccountsFilteringAndRankingLogicId = Value(\"new_organic_follows\")\n  val NonNewOrganicFollows: AccountsFilteringAndRankingLogicId = Value(\"non_new_organic_follows\")\n  val OrganicFollows: AccountsFilteringAndRankingLogicId = Value(\"organic_follows\")\n}\n\nobject TopOrganicFollowsAccountsSource {\n  val MaxCacheSize = 500\n  val CacheTTL: Duration = Duration.fromHours(24)\n\n  type Target = HasParams with HasClientContext with HasGeohashAndCountryCode\n\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.OrganicFollowAccounts.toString)\n}\n\n@Singleton\nclass TopOrganicFollowsAccountsSource @Inject() (\n  organicFollowsAccountsClientColumn: OrganicFollowsAccountsClientColumn,\n  statsReceiver: StatsReceiver,\n) extends CandidateSource[TopOrganicFollowsAccountsSource.Target, CandidateUser]\n    with Logging {\n\n  /** @see [[CandidateSourceIdentifier]] */\n  override val identifier: CandidateSourceIdentifier =\n    TopOrganicFollowsAccountsSource.Identifier\n\n  private val stats = statsReceiver.scope(identifier.name)\n  private val requestsStats = stats.counter(\"requests\")\n  private val noCountryCodeStats = stats.counter(\"no_country_code\")\n  private val successStats = stats.counter(\"success\")\n  private val errorStats = stats.counter(\"error\")\n\n  private val cache = StitchCache[String, Option[OrganicFollowsAccounts]](\n    maxCacheSize = TopOrganicFollowsAccountsSource.MaxCacheSize,\n    ttl = TopOrganicFollowsAccountsSource.CacheTTL,\n    statsReceiver = statsReceiver.scope(identifier.name, \"cache\"),\n    underlyingCall = (k: String) => {\n      organicFollowsAccountsClientColumn.fetcher\n        .fetch(k)\n        .map { result => result.v }\n    }\n  )\n\n  /** returns a Seq of ''potential'' content */\n  override def apply(\n    target: TopOrganicFollowsAccountsSource.Target\n  ): Stitch[Seq[CandidateUser]] = {\n    if (!target.params(CandidateSourceEnabled)) {\n      return Stitch.value(Seq[CandidateUser]())\n    }\n    requestsStats.incr()\n    target.getCountryCode\n      .orElse(target.geohashAndCountryCode.flatMap(_.countryCode)).map { countryCode =>\n        Stitch\n          .collect(target\n            .params(AccountsFilteringAndRankingLogics).map(logic =>\n              cache.readThrough(countryCode.toUpperCase() + \"-\" + logic)))\n          .onSuccess(_ => {\n            successStats.incr()\n          })\n          .onFailure(t => {\n            debug(\"candidate source failed identifier = %s\".format(identifier), t)\n            errorStats.incr()\n          })\n          .map(transformOrganicFollowAccountssToCandidateSource)\n      }.getOrElse {\n        noCountryCodeStats.incr()\n        Stitch.value(Seq[CandidateUser]())\n      }\n  }\n\n  private def transformOrganicFollowAccountssToCandidateSource(\n    organicFollowsAccounts: Seq[Option[OrganicFollowsAccounts]]\n  ): Seq[CandidateUser] = {\n    organicFollowsAccounts\n      .flatMap(opt =>\n        opt\n          .map(accounts =>\n            accounts.accounts.map(account =>\n              CandidateUser(\n                id = account.accountId,\n                score = Some(account.followedCountScore),\n              ).withCandidateSource(identifier)))\n          .getOrElse(Seq[CandidateUser]()))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/triangular_loops/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"strato/config/columns/onboarding/userrecs:userrecs-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/triangular_loops/README.md",
    "content": "# Triangular Loops Candidate Source\nProvides account candidates based on the graph structures of the form u -> v -> w -> u,\nwhere the arrow indicates a follow edge. In other words, it looks for triangular loops in the user-user graph.\n\nIf the edge v -> u does not exist in the triangular loop, the Triangular Loops Candidate Source recommends u as a potential outbound mutual follow candidate for v.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/triangular_loops/TriangularLoopsFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.triangular_loops\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TriangularLoopsFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[FSParam[Boolean] with FSName] = Nil\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/triangular_loops/TriangularLoopsParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.triangular_loops\n\nimport com.twitter.timelines.configapi.FSParam\n\nobject TriangularLoopsParams {\n\n  object KeepOnlyCandidatesWhoFollowTargetUser\n      extends FSParam[Boolean](\n        \"triangular_loops_keep_only_candidates_who_follow_target_user\",\n        false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/triangular_loops/TriangularLoopsSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.triangular_loops\n\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FollowProof\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedByUserIds\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.userrecs.TriangularLoopsV2OnUserClientColumn\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.wtf.triangular_loop.thriftscala.Candidates\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TriangularLoopsSource @Inject() (\n  triangularLoopsV2Column: TriangularLoopsV2OnUserClientColumn)\n    extends CandidateSource[\n      HasParams with HasClientContext with HasRecentFollowedByUserIds,\n      CandidateUser\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = TriangularLoopsSource.Identifier\n\n  override def apply(\n    target: HasParams with HasClientContext with HasRecentFollowedByUserIds\n  ): Stitch[Seq[CandidateUser]] = {\n    val candidates = target.getOptionalUserId\n      .map { userId =>\n        val fetcher = triangularLoopsV2Column.fetcher\n        fetcher\n          .fetch(userId)\n          .map { result =>\n            result.v\n              .map(TriangularLoopsSource.mapCandidatesToCandidateUsers)\n              .getOrElse(Nil)\n          }\n      }.getOrElse(Stitch.Nil)\n    // Make sure recentFollowedByUserIds is populated within the RequestBuilder before enable it\n    if (target.params(TriangularLoopsParams.KeepOnlyCandidatesWhoFollowTargetUser))\n      filterOutCandidatesNotFollowingTargetUser(candidates, target.recentFollowedByUserIds)\n    else\n      candidates\n  }\n\n  def filterOutCandidatesNotFollowingTargetUser(\n    candidatesStitch: Stitch[Seq[CandidateUser]],\n    recentFollowings: Option[Seq[Long]]\n  ): Stitch[Seq[CandidateUser]] = {\n    candidatesStitch.map { candidates =>\n      val recentFollowingIdsSet = recentFollowings.getOrElse(Nil).toSet\n      candidates.filter(candidate => recentFollowingIdsSet.contains(candidate.id))\n    }\n  }\n}\n\nobject TriangularLoopsSource {\n\n  val Identifier = CandidateSourceIdentifier(Algorithm.TriangularLoop.toString)\n  val NumResults = 100\n\n  def mapCandidatesToCandidateUsers(candidates: Candidates): Seq[CandidateUser] = {\n    candidates.candidates\n      .map { candidate =>\n        CandidateUser(\n          id = candidate.incomingUserId,\n          score = Some(1.0 / math\n            .max(1, candidate.numFollowers.getOrElse(0) + candidate.numFollowings.getOrElse(0))),\n          reason = Some(\n            Reason(\n              Some(\n                AccountProof(\n                  followProof =\n                    if (candidate.socialProofUserIds.isEmpty) None\n                    else\n                      Some(\n                        FollowProof(\n                          candidate.socialProofUserIds,\n                          candidate.numSocialProof.getOrElse(candidate.socialProofUserIds.size)))\n                )\n              )\n            )\n          )\n        ).withCandidateSource(Identifier)\n      }.sortBy(-_.score.getOrElse(0.0)).take(NumResults)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/two_hop_random_walk/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"src/thrift/com/twitter/hermit/candidate:hermit-candidate-scala\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/two_hop_random_walk/README.md",
    "content": "# Two-hop Random Walk\nThe TwoHopRandomWalk algorithm re-ranks a user's second degree connections based on recent engagement strength. The algorithm works as follows:\n\n* Given a user `src`, find their top K first degree connections `fd(1)`, `fd(2)`, `fd(3)`,...,`fd(K)`. The ranking is based on real graph weights, which measure the recent engagement strength on the edges.\n* For each of the first degree connections `fd(i)`, expand to their top L connections via real graph, `sd(i,1)`, `sd(i,2)`,...,`sd(i,L)`. Note that sd nodes can also be `src`'s first degree nodes.\n* Aggregate all the nodes in step 2, filter out the first degree nodes, and calculate the weighted sum for the second degree.\n* Re-rank the second degree nodes and select the top M results as the algorithm output.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/two_hop_random_walk/TwoHopRandomWalkSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.two_hop_random_walk\n\nimport com.twitter.follow_recommendations.common.candidate_sources.base.StratoFetcherWithUnitViewSource\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.wtf.candidate.thriftscala.{CandidateSeq => TCandidateSeq}\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass TwoHopRandomWalkSource @Inject() (\n  @Named(GuiceNamedConstants.TWO_HOP_RANDOM_WALK_FETCHER) fetcher: Fetcher[\n    Long,\n    Unit,\n    TCandidateSeq\n  ]) extends StratoFetcherWithUnitViewSource[Long, TCandidateSeq](\n      fetcher,\n      TwoHopRandomWalkSource.Identifier) {\n\n  override def map(targetUserId: Long, tCandidateSeq: TCandidateSeq): Seq[CandidateUser] =\n    TwoHopRandomWalkSource.map(targetUserId, tCandidateSeq)\n\n}\n\nobject TwoHopRandomWalkSource {\n  def map(targetUserId: Long, tCandidateSeq: TCandidateSeq): Seq[CandidateUser] = {\n    tCandidateSeq.candidates\n      .sortBy(-_.score)\n      .map { tCandidate =>\n        CandidateUser(id = tCandidate.userId, score = Some(tCandidate.score))\n      }\n  }\n\n  val Identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(Algorithm.TwoHopRandomWalk.toString)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/user_user_graph/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"src/thrift/com/twitter/recos/user_user_graph:user_user_graph-scala\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/user_user_graph/README.md",
    "content": "# User-User Graph Candidate Source\nProvides account candidates generated from the User-User Graph (UUG).\n## User-User Graph (UUG)\nThe UUG algorithm reads User-Follow-User engagements that occurred in the past 24-48 hours, and provides accounts that the given user's recent followings have recently followed themselves. The UUG algorithm is implemented using the real-time graph processing library GraphJet.\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/user_user_graph/UserUserGraphCandidateSource.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.user_user_graph\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.follow_recommendations.common.models._\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.recos.recos_common.thriftscala.UserSocialProofType\nimport com.twitter.recos.user_user_graph.thriftscala.RecommendUserDisplayLocation\nimport com.twitter.recos.user_user_graph.thriftscala.RecommendUserRequest\nimport com.twitter.recos.user_user_graph.thriftscala.RecommendUserResponse\nimport com.twitter.recos.user_user_graph.thriftscala.RecommendedUser\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass UserUserGraphCandidateSource @Inject() (\n  @Named(GuiceNamedConstants.USER_USER_GRAPH_FETCHER)\n  fetcher: Fetcher[RecommendUserRequest, Unit, RecommendUserResponse],\n  statsReceiver: StatsReceiver)\n    extends CandidateSource[\n      UserUserGraphCandidateSource.Target,\n      CandidateUser\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = UserUserGraphCandidateSource.Identifier\n  val stats: StatsReceiver = statsReceiver.scope(\"UserUserGraph\")\n  val requestCounter: Counter = stats.counter(\"requests\")\n\n  override def apply(\n    target: UserUserGraphCandidateSource.Target\n  ): Stitch[Seq[CandidateUser]] = {\n    if (target.params(UserUserGraphParams.UserUserGraphCandidateSourceEnabledInWeightMap)) {\n      requestCounter.incr()\n      buildRecommendUserRequest(target)\n        .map { request =>\n          fetcher\n            .fetch(request)\n            .map(_.v)\n            .map { responseOpt =>\n              responseOpt\n                .map { response =>\n                  response.recommendedUsers\n                    .sortBy(-_.score)\n                    .map(convertToCandidateUsers)\n                    .map(_.withCandidateSource(identifier))\n                }.getOrElse(Nil)\n            }\n        }.getOrElse(Stitch.Nil)\n    } else {\n      Stitch.Nil\n    }\n  }\n\n  private[this] def buildRecommendUserRequest(\n    target: UserUserGraphCandidateSource.Target\n  ): Option[RecommendUserRequest] = {\n    (target.getOptionalUserId, target.recentFollowedUserIds) match {\n      case (Some(userId), Some(recentFollowedUserIds)) =>\n        // use recentFollowedUserIds as seeds for initial experiment\n        val seedsWithWeights: Map[Long, Double] = recentFollowedUserIds.map {\n          recentFollowedUserId =>\n            recentFollowedUserId -> UserUserGraphCandidateSource.DefaultSeedWeight\n        }.toMap\n        val request = RecommendUserRequest(\n          requesterId = userId,\n          displayLocation = UserUserGraphCandidateSource.DisplayLocation,\n          seedsWithWeights = seedsWithWeights,\n          excludedUserIds = Some(target.excludedUserIds),\n          maxNumResults = Some(target.params.getInt(UserUserGraphParams.MaxCandidatesToReturn)),\n          maxNumSocialProofs = Some(UserUserGraphCandidateSource.MaxNumSocialProofs),\n          minUserPerSocialProof = Some(UserUserGraphCandidateSource.MinUserPerSocialProof),\n          socialProofTypes = Some(Seq(UserUserGraphCandidateSource.SocialProofType))\n        )\n        Some(request)\n      case _ => None\n    }\n  }\n\n  private[this] def convertToCandidateUsers(\n    recommendedUser: RecommendedUser\n  ): CandidateUser = {\n    val socialProofUserIds =\n      recommendedUser.socialProofs.getOrElse(UserUserGraphCandidateSource.SocialProofType, Nil)\n    val reasonOpt = if (socialProofUserIds.nonEmpty) {\n      Some(\n        Reason(\n          Some(AccountProof(followProof =\n            Some(FollowProof(socialProofUserIds, socialProofUserIds.size)))))\n      )\n    } else {\n      None\n    }\n    CandidateUser(\n      id = recommendedUser.userId,\n      score = Some(recommendedUser.score),\n      reason = reasonOpt)\n  }\n}\n\nobject UserUserGraphCandidateSource {\n  type Target = HasParams\n    with HasClientContext\n    with HasRecentFollowedUserIds\n    with HasExcludedUserIds\n\n  val Identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    Algorithm.UserUserGraph.toString)\n  //Use HomeTimeline for experiment\n  val DisplayLocation: RecommendUserDisplayLocation = RecommendUserDisplayLocation.HomeTimeLine\n\n  //Default params used in MagicRecs\n  val DefaultSeedWeight: Double = 1.0\n  val SocialProofType = UserSocialProofType.Follow\n  val MaxNumSocialProofs = 10\n  val MinUserPerSocialProof: Map[UserSocialProofType, Int] =\n    Map[UserSocialProofType, Int]((SocialProofType, 2))\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/user_user_graph/UserUserGraphFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.user_user_graph\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.Param\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserUserGraphFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] = Seq(\n    UserUserGraphParams.UserUserGraphCandidateSourceEnabledInWeightMap,\n    UserUserGraphParams.UserUserGraphCandidateSourceEnabledInTransform\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/user_user_graph/UserUserGraphParams.scala",
    "content": "package com.twitter.follow_recommendations.common.candidate_sources.user_user_graph\n\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\n\nobject UserUserGraphParams {\n\n  // max number of candidates to return in total, 50 is the default param used in MagicRecs\n  object MaxCandidatesToReturn extends Param[Int](default = 50)\n\n  // whether or not to include UserUserGraph candidate source in the weighted blending step\n  case object UserUserGraphCandidateSourceEnabledInWeightMap\n      extends FSParam[Boolean](\"user_user_graph_candidate_source_enabled_in_weight_map\", true)\n\n  // whether or not to include UserUserGraph candidate source in the final transform step\n  case object UserUserGraphCandidateSourceEnabledInTransform\n      extends FSParam[Boolean](\"user_user_graph_candidate_source_enabled_in_transform\", true)\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/addressbook/AddressbookClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.addressbook\n\nimport com.twitter.addressbook.datatypes.thriftscala.QueryType\nimport com.twitter.addressbook.thriftscala.AddressBookGetRequest\nimport com.twitter.addressbook.thriftscala.AddressBookGetResponse\nimport com.twitter.addressbook.thriftscala.Addressbook2\nimport com.twitter.addressbook.thriftscala.ClientInfo\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.wtf.scalding.jobs.addressbook.thriftscala.STPResultFeature\nimport com.twitter.follow_recommendations.common.clients.addressbook.models.Contact\nimport com.twitter.follow_recommendations.common.clients.addressbook.models.EdgeType\nimport com.twitter.follow_recommendations.common.clients.addressbook.models.QueryOption\nimport com.twitter.follow_recommendations.common.clients.addressbook.models.RecordIdentifier\nimport com.twitter.wtf.scalding.jobs.address_book.ABUtil.hashContact\nimport com.twitter.wtf.scalding.jobs.address_book.ABUtil.normalizeEmail\nimport com.twitter.wtf.scalding.jobs.address_book.ABUtil.normalizePhoneNumber\nimport com.twitter.hermit.usercontacts.thriftscala.{UserContacts => tUserContacts}\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AddressbookClient @Inject() (\n  addressbookService: Addressbook2.MethodPerEndpoint,\n  statsReceiver: StatsReceiver = NullStatsReceiver) {\n\n  private val stats: StatsReceiver = statsReceiver.scope(this.getClass.getSimpleName)\n\n  private[this] def getResponseFromService(\n    identifiers: Seq[RecordIdentifier],\n    batchSize: Int,\n    edgeType: EdgeType,\n    maxFetches: Int,\n    queryOption: Option[QueryOption]\n  ): Stitch[Seq[AddressBookGetResponse]] = {\n    Stitch\n      .collect(\n        identifiers.map { identifier =>\n          Stitch.callFuture(\n            addressbookService.get(AddressBookGetRequest(\n              clientInfo = ClientInfo(None),\n              identifier = identifier.toThrift,\n              edgeType = edgeType.toThrift,\n              queryType = QueryType.UserId,\n              queryOption = queryOption.map(_.toThrift),\n              maxFetches = maxFetches,\n              resultBatchSize = batchSize\n            )))\n        }\n      )\n  }\n\n  private[this] def getContactsResponseFromService(\n    identifiers: Seq[RecordIdentifier],\n    batchSize: Int,\n    edgeType: EdgeType,\n    maxFetches: Int,\n    queryOption: Option[QueryOption]\n  ): Stitch[Seq[AddressBookGetResponse]] = {\n    Stitch\n      .collect(\n        identifiers.map { identifier =>\n          Stitch.callFuture(\n            addressbookService.get(AddressBookGetRequest(\n              clientInfo = ClientInfo(None),\n              identifier = identifier.toThrift,\n              edgeType = edgeType.toThrift,\n              queryType = QueryType.Contact,\n              queryOption = queryOption.map(_.toThrift),\n              maxFetches = maxFetches,\n              resultBatchSize = batchSize\n            )))\n        }\n      )\n  }\n\n  /** Mode of addressbook resolving logic\n   * ManhattanThenABV2: fetching manhattan cached result and backfill with addressbook v2\n   * ABV2Only: calling addressbook v2 directly without fetching manhattan cached result\n   * This can be controlled by passing a fetcher or not. Passing a fetcher will attempt to use it,\n   * if not then it won't.\n   */\n  def getUsers(\n    userId: Long,\n    identifiers: Seq[RecordIdentifier],\n    batchSize: Int,\n    edgeType: EdgeType,\n    fetcherOption: Option[Fetcher[Long, Unit, tUserContacts]] = None,\n    maxFetches: Int = 1,\n    queryOption: Option[QueryOption] = None,\n  ): Stitch[Seq[Long]] = {\n    fetcherOption match {\n      case Some(fetcher) =>\n        getUsersFromManhattan(userId, fetcher).flatMap { userContacts =>\n          if (userContacts.isEmpty) {\n            stats.counter(\"mhEmptyThenFromAbService\").incr()\n            getResponseFromService(identifiers, batchSize, edgeType, maxFetches, queryOption)\n              .map(_.flatMap(_.users).flatten.distinct)\n          } else {\n            stats.counter(\"fromManhattan\").incr()\n            Stitch.value(userContacts)\n          }\n        }\n      case None =>\n        stats.counter(\"fromAbService\").incr()\n        getResponseFromService(identifiers, batchSize, edgeType, maxFetches, queryOption)\n          .map(_.flatMap(_.users).flatten.distinct)\n    }\n  }\n\n  def getHashedContacts(\n    normalizeFn: String => String,\n    extractField: String,\n  )(\n    userId: Long,\n    identifiers: Seq[RecordIdentifier],\n    batchSize: Int,\n    edgeType: EdgeType,\n    fetcherOption: Option[Fetcher[String, Unit, STPResultFeature]] = None,\n    maxFetches: Int = 1,\n    queryOption: Option[QueryOption] = None,\n  ): Stitch[Seq[String]] = {\n\n    fetcherOption match {\n      case Some(fetcher) =>\n        getContactsFromManhattan(userId, fetcher).flatMap { userContacts =>\n          if (userContacts.isEmpty) {\n            getContactsResponseFromService(\n              identifiers,\n              batchSize,\n              edgeType,\n              maxFetches,\n              queryOption)\n              .map { response =>\n                for {\n                  resp <- response\n                  contacts <- resp.contacts\n                  contactsThrift = contacts.map(Contact.fromThrift)\n                  contactsSet = extractField match {\n                    case \"emails\" => contactsThrift.flatMap(_.emails.toSeq.flatten)\n                    case \"phoneNumbers\" => contactsThrift.flatMap(_.phoneNumbers.toSeq.flatten)\n                  }\n                  hashedAndNormalizedContacts = contactsSet.map(c => hashContact(normalizeFn(c)))\n                } yield hashedAndNormalizedContacts\n              }.map(_.flatten)\n          } else {\n            Stitch.Nil\n          }\n        }\n      case None => {\n        getContactsResponseFromService(identifiers, batchSize, edgeType, maxFetches, queryOption)\n          .map { response =>\n            for {\n              resp <- response\n              contacts <- resp.contacts\n              contactsThrift = contacts.map(Contact.fromThrift)\n              contactsSet = extractField match {\n                case \"emails\" => contactsThrift.flatMap(_.emails.toSeq.flatten)\n                case \"phoneNumbers\" => contactsThrift.flatMap(_.phoneNumbers.toSeq.flatten)\n              }\n              hashedAndNormalizedContacts = contactsSet.map(c => hashContact(normalizeFn(c)))\n            } yield hashedAndNormalizedContacts\n          }.map(_.flatten)\n      }\n    }\n  }\n\n  def getEmailContacts = getHashedContacts(normalizeEmail, \"emails\") _\n  def getPhoneContacts = getHashedContacts(normalizePhoneNumber, \"phoneNumbers\") _\n\n  private def getUsersFromManhattan(\n    userId: Long,\n    fetcher: Fetcher[Long, Unit, tUserContacts],\n  ): Stitch[Seq[Long]] = fetcher\n    .fetch(userId)\n    .map(_.v.map(_.destinationIds).toSeq.flatten.distinct)\n\n  private def getContactsFromManhattan(\n    userId: Long,\n    fetcher: Fetcher[String, Unit, STPResultFeature],\n  ): Stitch[Seq[String]] = fetcher\n    .fetch(userId.toString)\n    .map(_.v.map(_.strongTieUserFeature.map(_.destId)).toSeq.flatten.distinct)\n}\n\nobject AddressbookClient {\n  val AddressBook2BatchSize = 500\n\n  def createQueryOption(edgeType: EdgeType, isPhone: Boolean): Option[QueryOption] =\n    (edgeType, isPhone) match {\n      case (EdgeType.Reverse, _) =>\n        None\n      case (EdgeType.Forward, true) =>\n        Some(\n          QueryOption(\n            onlyDiscoverableInExpansion = false,\n            onlyConfirmedInExpansion = false,\n            onlyDiscoverableInResult = false,\n            onlyConfirmedInResult = false,\n            fetchGlobalApiNamespace = false,\n            isDebugRequest = false,\n            resolveEmails = false,\n            resolvePhoneNumbers = true\n          ))\n      case (EdgeType.Forward, false) =>\n        Some(\n          QueryOption(\n            onlyDiscoverableInExpansion = false,\n            onlyConfirmedInExpansion = false,\n            onlyDiscoverableInResult = false,\n            onlyConfirmedInResult = false,\n            fetchGlobalApiNamespace = false,\n            isDebugRequest = false,\n            resolveEmails = true,\n            resolvePhoneNumbers = false\n          ))\n    }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/addressbook/AddressbookModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.addressbook\n\nimport com.twitter.addressbook.thriftscala.Addressbook2\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.follow_recommendations.common.clients.common.BaseClientModule\n\nobject AddressbookModule extends BaseClientModule[Addressbook2.MethodPerEndpoint] with MtlsClient {\n  override val label = \"addressbook\"\n  override val dest = \"/s/addressbook/addressbook2\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/addressbook/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"addressbook/thrift/src/thrift/com/twitter/addressbook:thrift-scala\",\n        \"addressbook/thrift/src/thrift/com/twitter/addressbook/datatypes:thrift-scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-thrift-client\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/addressbook/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"src/scala/com/twitter/wtf/scalding/jobs/address_book:ab_util\",\n        \"src/thrift/com/twitter/hermit/usercontacts:hermit-usercontacts-scala\",\n        \"src/thrift/com/twitter/wtf/addressbook:addressbook-scala\",\n        \"stitch/stitch-core\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/addressbook/models/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"addressbook/thrift/src/thrift/com/twitter/addressbook:thrift-scala\",\n        \"addressbook/thrift/src/thrift/com/twitter/addressbook/datatypes:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/addressbook/models/Contact.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.addressbook.models\n\nimport com.twitter.addressbook.{thriftscala => t}\nimport com.twitter.util.Time\n\ncase class Contact(\n  id: Long,\n  emails: Option[Set[String]],\n  phoneNumbers: Option[Set[String]],\n  firstName: Option[String],\n  lastName: Option[String],\n  name: Option[String],\n  appId: Option[Long],\n  appIds: Option[Set[Long]],\n  importedTimestamp: Option[Time])\n\nobject Contact {\n  def fromThrift(thriftContact: t.Contact): Contact = Contact(\n    thriftContact.id,\n    thriftContact.emails.map(_.toSet),\n    thriftContact.phoneNumbers.map(_.toSet),\n    thriftContact.firstName,\n    thriftContact.lastName,\n    thriftContact.name,\n    thriftContact.appId,\n    thriftContact.appIds.map(_.toSet),\n    thriftContact.importedTimestamp.map(Time.fromMilliseconds)\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/addressbook/models/EdgeType.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.addressbook.models\n\nimport com.twitter.addressbook.datatypes.{thriftscala => t}\n\nsealed trait EdgeType {\n  def toThrift: t.EdgeType\n}\n\nobject EdgeType {\n  case object Forward extends EdgeType {\n    override val toThrift: t.EdgeType = t.EdgeType.Forward\n  }\n  case object Reverse extends EdgeType {\n    override val toThrift: t.EdgeType = t.EdgeType.Reverse\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/addressbook/models/QueryOption.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.addressbook.models\n\nimport com.twitter.addressbook.{thriftscala => t}\n\ncase class QueryOption(\n  onlyDiscoverableInExpansion: Boolean,\n  onlyConfirmedInExpansion: Boolean,\n  onlyDiscoverableInResult: Boolean,\n  onlyConfirmedInResult: Boolean,\n  fetchGlobalApiNamespace: Boolean,\n  isDebugRequest: Boolean,\n  resolveEmails: Boolean,\n  resolvePhoneNumbers: Boolean) {\n  def toThrift: t.QueryOption = t.QueryOption(\n    onlyDiscoverableInExpansion,\n    onlyConfirmedInExpansion,\n    onlyDiscoverableInResult,\n    onlyConfirmedInResult,\n    fetchGlobalApiNamespace,\n    isDebugRequest,\n    resolveEmails,\n    resolvePhoneNumbers\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/addressbook/models/RecordIdentifier.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.addressbook.models\n\nimport com.twitter.addressbook.datatypes.{thriftscala => t}\n\ncase class RecordIdentifier(\n  userId: Option[Long],\n  email: Option[String],\n  phoneNumber: Option[String]) {\n  def toThrift: t.RecordIdentifier = t.RecordIdentifier(userId, email, phoneNumber)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/adserver/AdRequest.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.adserver\n\nimport com.twitter.adserver.{thriftscala => t}\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\n\ncase class AdRequest(\n  clientContext: ClientContext,\n  displayLocation: DisplayLocation,\n  isTest: Option[Boolean],\n  profileUserId: Option[Long]) {\n  def toThrift: t.AdRequestParams = {\n\n    val request = t.AdRequest(\n      displayLocation = displayLocation.toAdDisplayLocation.getOrElse(\n        throw new MissingAdDisplayLocation(displayLocation)),\n      isTest = isTest,\n      countImpressionsOnCallback = Some(true),\n      numOrganicItems = Some(AdRequest.DefaultNumOrganicItems.toShort),\n      profileUserId = profileUserId\n    )\n\n    val clientInfo = t.ClientInfo(\n      clientId = clientContext.appId.map(_.toInt),\n      userIp = clientContext.ipAddress,\n      userId64 = clientContext.userId,\n      guestId = clientContext.guestId,\n      userAgent = clientContext.userAgent,\n      referrer = None,\n      deviceId = clientContext.deviceId,\n      languageCode = clientContext.languageCode,\n      countryCode = clientContext.countryCode\n    )\n\n    t.AdRequestParams(request, clientInfo)\n  }\n}\n\nobject AdRequest {\n  val DefaultNumOrganicItems = 10\n}\n\nclass MissingAdDisplayLocation(displayLocation: DisplayLocation)\n    extends Exception(\n      s\"Display Location ${displayLocation.toString} has no mapped AdsDisplayLocation set.\")\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/adserver/AdserverClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.adserver\n\nimport com.twitter.adserver.thriftscala.NewAdServer\nimport com.twitter.adserver.{thriftscala => t}\nimport com.twitter.stitch.Stitch\nimport javax.inject.{Inject, Singleton}\n\n@Singleton\nclass AdserverClient @Inject() (adserverService: NewAdServer.MethodPerEndpoint) {\n  def getAdImpressions(adRequest: AdRequest): Stitch[Seq[t.AdImpression]] = {\n    Stitch\n      .callFuture(\n        adserverService.makeAdRequest(adRequest.toThrift)\n      ).map(_.impressions)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/adserver/AdserverModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.adserver\n\nimport com.twitter.adserver.thriftscala.NewAdServer\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.follow_recommendations.common.clients.common.BaseClientModule\n\nobject AdserverModule extends BaseClientModule[NewAdServer.MethodPerEndpoint] with MtlsClient {\n  override val label = \"adserver\"\n  override val dest = \"/s/ads/adserver\"\n\n  override def configureThriftMuxClient(client: ThriftMux.Client): ThriftMux.Client =\n    client.withRequestTimeout(500.millis)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/adserver/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-thrift-client\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_rpc-scala\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/cache/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"cache/client\",\n        \"finagle/finagle-memcached/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"stitch/stitch-core\",\n        \"util/util-core:scala\",\n        \"util/util-thrift\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/cache/MemcacheClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.cache\n\nimport com.twitter.bijection.Bijection\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.Memcached.Client\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.io.Buf\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport java.security.MessageDigest\n\nobject MemcacheClient {\n  def apply[V](\n    client: Client,\n    dest: String,\n    valueBijection: Bijection[Buf, V],\n    ttl: Duration,\n    statsReceiver: StatsReceiver\n  ): MemcacheClient[V] = {\n    new MemcacheClient(client, dest, valueBijection, ttl, statsReceiver)\n  }\n}\n\nclass MemcacheClient[V](\n  client: Client,\n  dest: String,\n  valueBijection: Bijection[Buf, V],\n  ttl: Duration,\n  statsReceiver: StatsReceiver) {\n  val cache = client.newRichClient(dest).adapt[V](valueBijection)\n  val cacheTtl = Time.fromSeconds(ttl.inSeconds)\n\n  /**\n   * If cache contains key, return value from cache. Otherwise, run the underlying call\n   * to fetch the value, store it in cache, and then return the value.\n   */\n  def readThrough(\n    key: String,\n    underlyingCall: () => Stitch[V]\n  ): Stitch[V] = {\n    val cachedResult: Stitch[Option[V]] = Stitch\n      .callFuture(getIfPresent(key))\n      .within(70.millisecond)(DefaultTimer)\n      .rescue {\n        case e: Exception =>\n          statsReceiver.scope(\"rescued\").counter(e.getClass.getSimpleName).incr()\n          Stitch(None)\n      }\n    val resultStitch = cachedResult.map { resultOption =>\n      resultOption match {\n        case Some(cacheValue) => Stitch.value(cacheValue)\n        case None =>\n          val underlyingCallStitch = profileStitch(\n            underlyingCall(),\n            statsReceiver.scope(\"underlyingCall\")\n          )\n          underlyingCallStitch.map { result =>\n            put(key, result)\n            result\n          }\n      }\n    }.flatten\n    // profile the overall Stitch, and return the result\n    profileStitch(resultStitch, statsReceiver.scope(\"readThrough\"))\n  }\n\n  def getIfPresent(key: String): Future[Option[V]] = {\n    cache\n      .get(hashString(key))\n      .onSuccess {\n        case Some(value) => statsReceiver.counter(\"cache_hits\").incr()\n        case None => statsReceiver.counter(\"cache_misses\").incr()\n      }\n      .onFailure {\n        case e: Exception =>\n          statsReceiver.counter(\"cache_misses\").incr()\n          statsReceiver.scope(\"rescued\").counter(e.getClass.getSimpleName).incr()\n      }\n      .rescue {\n        case _ => Future.None\n      }\n  }\n\n  def put(key: String, value: V): Future[Unit] = {\n    cache.set(hashString(key), 0, cacheTtl, value)\n  }\n\n  /**\n   * Hash the input key string to a fixed length format using SHA-256 hash function.\n   */\n  def hashString(input: String): String = {\n    val bytes = MessageDigest.getInstance(\"SHA-256\").digest(input.getBytes(\"UTF-8\"))\n    bytes.map(\"%02x\".format(_)).mkString\n  }\n\n  /**\n   * Helper function for timing a stitch, returning the original stitch.\n   *\n   * Defining the profiling function here to keep the dependencies of this class\n   * generic and easy to export (i.e. copy-and-paste) into other services or packages.\n   */\n  def profileStitch[T](stitch: Stitch[T], stat: StatsReceiver): Stitch[T] = {\n    Stitch\n      .time(stitch)\n      .map {\n        case (response, stitchRunDuration) =>\n          stat.counter(\"requests\").incr()\n          stat.stat(\"latency_ms\").add(stitchRunDuration.inMilliseconds)\n          response\n            .onSuccess { _ => stat.counter(\"success\").incr() }\n            .onFailure { e =>\n              stat.counter(\"failures\").incr()\n              stat.scope(\"failures\").counter(e.getClass.getSimpleName).incr()\n            }\n      }\n      .lowerFromTry\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/cache/MemcacheModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.cache\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.Memcached\nimport com.twitter.finagle.Memcached.Client\nimport com.twitter.finagle.mtls.client.MtlsStackClient._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.service.Retries\nimport com.twitter.finagle.service.RetryPolicy\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport javax.inject.Singleton\n\nobject MemcacheModule extends TwitterModule {\n  @Provides\n  @Singleton\n  def provideMemcacheClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver,\n  ): Client = {\n    Memcached.client\n      .withMutualTls(serviceIdentifier)\n      .withStatsReceiver(statsReceiver.scope(\"twemcache\"))\n      .withTransport.connectTimeout(1.seconds)\n      .withRequestTimeout(1.seconds)\n      .withSession.acquisitionTimeout(10.seconds)\n      .configured(Retries.Policy(RetryPolicy.tries(1)))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/cache/ThriftBijection.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.cache\n\nimport com.twitter.bijection.Bijection\nimport com.twitter.io.Buf\nimport com.twitter.scrooge.CompactThriftSerializer\nimport com.twitter.scrooge.ThriftEnum\nimport com.twitter.scrooge.ThriftStruct\nimport java.nio.ByteBuffer\n\nabstract class ThriftBijection[T <: ThriftStruct] extends Bijection[Buf, T] {\n  val serializer: CompactThriftSerializer[T]\n\n  override def apply(b: Buf): T = {\n    val byteArray = Buf.ByteArray.Owned.extract(b)\n    serializer.fromBytes(byteArray)\n  }\n\n  override def invert(a: T): Buf = {\n    val byteArray = serializer.toBytes(a)\n    Buf.ByteArray.Owned(byteArray)\n  }\n}\n\nabstract class ThriftOptionBijection[T <: ThriftStruct] extends Bijection[Buf, Option[T]] {\n  val serializer: CompactThriftSerializer[T]\n\n  override def apply(b: Buf): Option[T] = {\n    if (b.isEmpty) {\n      None\n    } else {\n      val byteArray = Buf.ByteArray.Owned.extract(b)\n      Some(serializer.fromBytes(byteArray))\n    }\n  }\n\n  override def invert(a: Option[T]): Buf = {\n    a match {\n      case Some(t) =>\n        val byteArray = serializer.toBytes(t)\n        Buf.ByteArray.Owned(byteArray)\n      case None => Buf.Empty\n    }\n  }\n}\n\nclass ThriftEnumBijection[T <: ThriftEnum](constructor: Int => T) extends Bijection[Buf, T] {\n  override def apply(b: Buf): T = {    \n    val byteArray = Buf.ByteArray.Owned.extract(b)\n    val byteBuffer = ByteBuffer.wrap(byteArray)\n    constructor(byteBuffer.getInt())\n  }\n\n  override def invert(a: T): Buf = {      \n    val byteBuffer: ByteBuffer = ByteBuffer.allocate(4)\n    byteBuffer.putInt(a.getValue)\n    Buf.ByteArray.Owned(byteBuffer.array())\n  }\n}\n\nclass ThriftEnumOptionBijection[T <: ThriftEnum](constructor: Int => T) extends Bijection[Buf, Option[T]] {\n  override def apply(b: Buf): Option[T] = {      \n    if (b.isEmpty) {\n      None\n    } else {\n      val byteArray = Buf.ByteArray.Owned.extract(b)\n      val byteBuffer = ByteBuffer.wrap(byteArray)\n      Some(constructor(byteBuffer.getInt()))\n    }\n  }\n\n  override def invert(a: Option[T]): Buf = {\n    a match {\n      case Some(obj) => {\n        val byteBuffer: ByteBuffer = ByteBuffer.allocate(4)\n        byteBuffer.putInt(obj.getValue)\n        Buf.ByteArray.Owned(byteBuffer.array())\n      }\n      case None => Buf.Empty\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/common/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-thrift-client\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/common/BaseClientModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.common\n\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.thrift.Protocols\nimport com.twitter.follow_recommendations.common.constants.ServiceConstants._\nimport com.twitter.inject.thrift.modules.ThriftClientModule\nimport scala.reflect.ClassTag\n\n/**\n * basic client configurations that we apply for all of our clients go in here\n */\nabstract class BaseClientModule[T: ClassTag] extends ThriftClientModule[T] {\n  def configureThriftMuxClient(client: ThriftMux.Client): ThriftMux.Client = {\n    client\n      .withProtocolFactory(\n        Protocols.binaryFactory(\n          stringLengthLimit = StringLengthLimit,\n          containerLengthLimit = ContainerLengthLimit))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/deepbirdv2/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"cortex-deepbird/prediction/src/main/scala/com/twitter/cortex/deepbird/prediction\",\n        \"cortex-deepbird/thrift/src/main/thrift:thrift-java\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/common\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/deepbirdv2/DeepBirdV2PredictionServiceClientModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.deepbirdv2\n\nimport com.google.inject.Provides\nimport com.google.inject.name.Named\nimport com.twitter.bijection.scrooge.TBinaryProtocol\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cortex.deepbird.thriftjava.DeepbirdPredictionService\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.builder.ClientBuilder\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient._\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.thrift.RichClientParam\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.inject.TwitterModule\n\n/**\n * Module that provides multiple deepbirdv2 prediction service clients\n * We use the java api since data records are native java objects and we want to reduce overhead\n * while serializing/deserializing data.\n */\nobject DeepBirdV2PredictionServiceClientModule extends TwitterModule {\n\n  val RequestTimeout = 300.millis\n\n  private def getDeepbirdPredictionServiceClient(\n    clientId: ClientId,\n    label: String,\n    dest: String,\n    statsReceiver: StatsReceiver,\n    serviceIdentifier: ServiceIdentifier\n  ): DeepbirdPredictionService.ServiceToClient = {\n    val clientStatsReceiver = statsReceiver.scope(\"clnt\")\n    val mTlsClient = ThriftMux.client.withClientId(clientId).withMutualTls(serviceIdentifier)\n    new DeepbirdPredictionService.ServiceToClient(\n      ClientBuilder()\n        .name(label)\n        .stack(mTlsClient)\n        .dest(dest)\n        .requestTimeout(RequestTimeout)\n        .reportHostStats(NullStatsReceiver)\n        .build(),\n      RichClientParam(\n        new TBinaryProtocol.Factory(),\n        clientStats = clientStatsReceiver\n      )\n    )\n  }\n\n  @Provides\n  @Named(GuiceNamedConstants.WTF_PROD_DEEPBIRDV2_CLIENT)\n  def providesWtfProdDeepbirdV2PredictionService(\n    clientId: ClientId,\n    statsReceiver: StatsReceiver,\n    serviceIdentifier: ServiceIdentifier\n  ): DeepbirdPredictionService.ServiceToClient = {\n    getDeepbirdPredictionServiceClient(\n      clientId = clientId,\n      label = \"WtfProdDeepbirdV2PredictionService\",\n      dest = \"/s/cassowary/deepbirdv2-hermit-wtf\",\n      statsReceiver = statsReceiver,\n      serviceIdentifier = serviceIdentifier\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/dismiss_store/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/github/nscala_time\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"src/thrift/com/twitter/onboarding/relevance/store:store-scala\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/dismiss_store/DismissStore.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.dismiss_store\n\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.onboarding.relevance.store.thriftscala.WhoToFollowDismissEventDetails\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.Scan.Slice\nimport com.twitter.strato.client.Scanner\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/**\n * this store gets the list of dismissed candidates since a certain time\n * primarily used for filtering out accounts that a user has explicitly dismissed\n *\n * we fail open on timeouts, but loudly on other errors\n */\n@Singleton\nclass DismissStore @Inject() (\n  @Named(GuiceNamedConstants.DISMISS_STORE_SCANNER)\n  scanner: Scanner[(Long, Slice[\n      (Long, Long)\n    ]), Unit, (Long, (Long, Long)), WhoToFollowDismissEventDetails],\n  stats: StatsReceiver)\n    extends Logging {\n\n  private val MaxCandidatesToReturn = 100\n\n  // gets a list of dismissed candidates. if numCandidatesToFetchOption is none, we will fetch the default number of candidates\n  def get(\n    userId: Long,\n    negStartTimeMs: Long,\n    maxCandidatesToFetchOption: Option[Int]\n  ): Stitch[Seq[Long]] = {\n\n    val maxCandidatesToFetch = maxCandidatesToFetchOption.getOrElse(MaxCandidatesToReturn)\n\n    scanner\n      .scan(\n        (\n          userId,\n          Slice(\n            from = None,\n            to = Some((negStartTimeMs, Long.MaxValue)),\n            limit = Some(maxCandidatesToFetch)\n          )\n        )\n      )\n      .map {\n        case s: Seq[((Long, (Long, Long)), WhoToFollowDismissEventDetails)] if s.nonEmpty =>\n          s.map {\n            case ((_: Long, (_: Long, candidateId: Long)), _: WhoToFollowDismissEventDetails) =>\n              candidateId\n          }\n        case _ => Nil\n      }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/email_storage_service/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"emailstorage/server/src/main/thrift/com/twitter/emailstorage/api:email-storage-service-scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-thrift-client\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/email_storage_service/EmailStorageServiceClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.email_storage_service\n\nimport com.twitter.cds.contact_consent_state.thriftscala.PurposeOfProcessing\nimport com.twitter.emailstorage.api.thriftscala.EmailStorageService\nimport com.twitter.emailstorage.api.thriftscala.GetUsersEmailsRequest\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass EmailStorageServiceClient @Inject() (\n  val emailStorageService: EmailStorageService.MethodPerEndpoint) {\n\n  def getVerifiedEmail(\n    userId: Long,\n    purposeOfProcessing: PurposeOfProcessing\n  ): Stitch[Option[String]] = {\n    val req = GetUsersEmailsRequest(\n      userIds = Seq(userId),\n      clientIdentifier = Some(\"follow-recommendations-service\"),\n      purposesOfProcessing = Some(Seq(purposeOfProcessing))\n    )\n\n    Stitch.callFuture(emailStorageService.getUsersEmails(req)) map {\n      _.usersEmails.map(_.confirmedEmail.map(_.email)).head\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/email_storage_service/EmailStorageServiceModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.email_storage_service\n\nimport com.twitter.emailstorage.api.thriftscala.EmailStorageService\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.follow_recommendations.common.clients.common.BaseClientModule\n\nobject EmailStorageServiceModule\n    extends BaseClientModule[EmailStorageService.MethodPerEndpoint]\n    with MtlsClient {\n  override val label = \"email-storage-service\"\n  override val dest = \"/s/email-server/email-server\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/geoduck/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/github/nscala_time\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"src/thrift/com/twitter/geoduck:geoduck-scala\",\n        \"src/thrift/com/twitter/geoduck:geoduckpartnerplaces-thrift-scala\",\n        \"stitch/stitch-core\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/geoduck/LocationServiceClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.geoduck\n\nimport com.twitter.follow_recommendations.common.models.GeohashAndCountryCode\nimport com.twitter.geoduck.common.thriftscala.LocationSource\nimport com.twitter.geoduck.common.thriftscala.PlaceQuery\nimport com.twitter.geoduck.common.thriftscala.TransactionLocation\nimport com.twitter.geoduck.common.thriftscala.UserLocationRequest\nimport com.twitter.geoduck.thriftscala.LocationService\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass LocationServiceClient @Inject() (locationService: LocationService.MethodPerEndpoint) {\n  def getGeohashAndCountryCode(userId: Long): Stitch[GeohashAndCountryCode] = {\n    Stitch\n      .callFuture {\n        locationService\n          .userLocation(\n            UserLocationRequest(\n              Seq(userId),\n              Some(PlaceQuery(allPlaceTypes = Some(true))),\n              simpleReverseGeocode = true))\n          .map(_.found.get(userId)).map { transactionLocationOpt =>\n            val geohashOpt = transactionLocationOpt.flatMap(getGeohashFromTransactionLocation)\n            val countryCodeOpt =\n              transactionLocationOpt.flatMap(_.simpleRgcResult.flatMap(_.countryCodeAlpha2))\n            GeohashAndCountryCode(geohashOpt, countryCodeOpt)\n          }\n      }\n  }\n\n  private[this] def getGeohashFromTransactionLocation(\n    transactionLocation: TransactionLocation\n  ): Option[String] = {\n    transactionLocation.geohash.flatMap { geohash =>\n      val geohashPrefixLength = transactionLocation.locationSource match {\n        // if location source is logical, keep the first 4 chars in geohash\n        case Some(LocationSource.Logical) => Some(4)\n        // if location source is physical, keep the prefix according to accuracy\n        // accuracy is the accuracy of GPS readings in the unit of meter\n        case Some(LocationSource.Physical) =>\n          transactionLocation.coordinate.flatMap { coordinate =>\n            coordinate.accuracy match {\n              case Some(accuracy) if (accuracy < 50) => Some(7)\n              case Some(accuracy) if (accuracy < 200) => Some(6)\n              case Some(accuracy) if (accuracy < 1000) => Some(5)\n              case Some(accuracy) if (accuracy < 50000) => Some(4)\n              case Some(accuracy) if (accuracy < 100000) => Some(3)\n              case _ => None\n            }\n          }\n        case Some(LocationSource.Model) => Some(4)\n        case _ => None\n      }\n      geohashPrefixLength match {\n        case Some(l: Int) => geohash.stringGeohash.map(_.take(l))\n        case _ => None\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/geoduck/LocationServiceModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.geoduck\n\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.follow_recommendations.common.clients.common.BaseClientModule\nimport com.twitter.geoduck.thriftscala.LocationService\n\nobject LocationServiceModule\n    extends BaseClientModule[LocationService.MethodPerEndpoint]\n    with MtlsClient {\n  override val label = \"geoduck_locationservice\"\n  override val dest = \"/s/geo/geoduck_locationservice\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/geoduck/ReverseGeocodeClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.geoduck\n\nimport com.twitter.follow_recommendations.common.models.GeohashAndCountryCode\nimport com.twitter.geoduck.common.thriftscala.Location\nimport com.twitter.geoduck.common.thriftscala.PlaceQuery\nimport com.twitter.geoduck.common.thriftscala.ReverseGeocodeIPRequest\nimport com.twitter.geoduck.service.thriftscala.GeoContext\nimport com.twitter.geoduck.thriftscala.ReverseGeocoder\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ReverseGeocodeClient @Inject() (rgcService: ReverseGeocoder.MethodPerEndpoint) {\n  def getGeohashAndCountryCode(ipAddress: String): Stitch[GeohashAndCountryCode] = {\n    Stitch\n      .callFuture {\n        rgcService\n          .reverseGeocodeIp(\n            ReverseGeocodeIPRequest(\n              Seq(ipAddress),\n              PlaceQuery(None),\n              simpleReverseGeocode = true\n            ) // note: simpleReverseGeocode means that country code will be included in response\n          ).map { response =>\n            response.found.get(ipAddress) match {\n              case Some(location) => getGeohashAndCountryCodeFromLocation(location)\n              case _ => GeohashAndCountryCode(None, None)\n            }\n          }\n      }\n  }\n\n  private def getGeohashAndCountryCodeFromLocation(location: Location): GeohashAndCountryCode = {\n    val countryCode: Option[String] = location.simpleRgcResult.flatMap { _.countryCodeAlpha2 }\n\n    val geohashString: Option[String] = location.geohash.flatMap { hash =>\n      hash.stringGeohash.flatMap { hashString =>\n        Some(ReverseGeocodeClient.truncate(hashString))\n      }\n    }\n\n    GeohashAndCountryCode(geohashString, countryCode)\n  }\n\n}\n\nobject ReverseGeocodeClient {\n\n  val DefaultGeoduckIPRequestContext: GeoContext =\n    GeoContext(allPlaceTypes = true, includeGeohash = true, includeCountryCode = true)\n\n  // All these geohashes are guessed by IP (Logical Location Source).\n  // So take the four letters to make sure it is consistent with LocationServiceClient\n  val GeohashLengthAfterTruncation = 4\n  def truncate(geohash: String): String = geohash.take(GeohashLengthAfterTruncation)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/geoduck/UserLocationFetcher.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.geoduck\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.GeohashAndCountryCode\nimport com.twitter.stitch.Stitch\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserLocationFetcher @Inject() (\n  locationServiceClient: LocationServiceClient,\n  reverseGeocodeClient: ReverseGeocodeClient,\n  statsReceiver: StatsReceiver) {\n\n  private val stats: StatsReceiver = statsReceiver.scope(\"user_location_fetcher\")\n  private val totalRequestsCounter = stats.counter(\"requests\")\n  private val emptyResponsesCounter = stats.counter(\"empty\")\n  private val locationServiceExceptionCounter = stats.counter(\"location_service_exception\")\n  private val reverseGeocodeExceptionCounter = stats.counter(\"reverse_geocode_exception\")\n\n  def getGeohashAndCountryCode(\n    userId: Option[Long],\n    ipAddress: Option[String]\n  ): Stitch[Option[GeohashAndCountryCode]] = {\n    totalRequestsCounter.incr()\n    val lscLocationStitch = Stitch\n      .collect {\n        userId.map(locationServiceClient.getGeohashAndCountryCode)\n      }.rescue {\n        case _: Exception =>\n          locationServiceExceptionCounter.incr()\n          Stitch.None\n      }\n\n    val ipLocationStitch = Stitch\n      .collect {\n        ipAddress.map(reverseGeocodeClient.getGeohashAndCountryCode)\n      }.rescue {\n        case _: Exception =>\n          reverseGeocodeExceptionCounter.incr()\n          Stitch.None\n      }\n\n    Stitch.join(lscLocationStitch, ipLocationStitch).map {\n      case (lscLocation, ipLocation) => {\n        val geohash = lscLocation.flatMap(_.geohash).orElse(ipLocation.flatMap(_.geohash))\n        val countryCode =\n          lscLocation.flatMap(_.countryCode).orElse(ipLocation.flatMap(_.countryCode))\n        (geohash, countryCode) match {\n          case (None, None) =>\n            emptyResponsesCounter.incr()\n            None\n          case _ => Some(GeohashAndCountryCode(geohash, countryCode))\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/gizmoduck/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/github/nscala_time\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/common\",\n        \"src/thrift/com/twitter/gizmoduck:thrift-scala\",\n        \"stitch/stitch-gizmoduck\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/gizmoduck/GizmoduckClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.gizmoduck\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.gizmoduck.thriftscala.LookupContext\nimport com.twitter.gizmoduck.thriftscala.PerspectiveEdge\nimport com.twitter.gizmoduck.thriftscala.QueryFields\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.gizmoduck.Gizmoduck\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GizmoduckClient @Inject() (gizmoduckStitchClient: Gizmoduck, statsReceiver: StatsReceiver) {\n  val stats = statsReceiver.scope(\"gizmoduck_client\")\n  val getByIdStats = stats.scope(\"get_by_id\")\n  val getUserById = stats.scope(\"get_user_by_id\")\n\n  def isProtected(userId: Long): Stitch[Boolean] = {\n    // get latency metrics with StatsUtil.profileStitch when calling .getById\n    val response = StatsUtil.profileStitch(\n      gizmoduckStitchClient.getById(userId, Set(QueryFields.Safety)),\n      getByIdStats\n    )\n    response.map { result =>\n      result.user.flatMap(_.safety).map(_.isProtected).getOrElse(true)\n    }\n  }\n\n  def getUserName(userId: Long, forUserId: Long): Stitch[Option[String]] = {\n    val queryFields = GizmoduckClient.GetUserByIdUserNameQueryFields\n    val lookupContext = LookupContext(\n      forUserId = Some(forUserId),\n      perspectiveEdges = Some(GizmoduckClient.DefaultPerspectiveEdges)\n    )\n    // get latency metrics with StatsUtil.profileStitch when calling .getUserById\n    val response = StatsUtil.profileStitch(\n      gizmoduckStitchClient.getUserById(userId, queryFields, lookupContext),\n      getUserById\n    )\n    response.map(_.profile.map(_.name))\n  }\n}\n\nobject GizmoduckClient {\n  // Similar to GizmoduckUserRepository.DefaultPerspectiveEdges\n  val DefaultPerspectiveEdges: Set[PerspectiveEdge] =\n    Set(\n      PerspectiveEdge.Blocking,\n      PerspectiveEdge.BlockedBy,\n      PerspectiveEdge.DeviceFollowing,\n      PerspectiveEdge.FollowRequestSent,\n      PerspectiveEdge.Following,\n      PerspectiveEdge.FollowedBy,\n      PerspectiveEdge.LifelineFollowing,\n      PerspectiveEdge.LifelineFollowedBy,\n      PerspectiveEdge.Muting,\n      PerspectiveEdge.NoRetweetsFrom\n    )\n\n  // From GizmoduckUserRepository.DefaultQueryFields\n  val GetUserByIdQueryFields: Set[QueryFields] = Set(\n    QueryFields.Account,\n    QueryFields.Counts,\n    QueryFields.ExtendedProfile,\n    QueryFields.Perspective,\n    QueryFields.Profile,\n    QueryFields.ProfileDesign,\n    QueryFields.ProfileLocation,\n    QueryFields.Safety,\n    QueryFields.Roles,\n    QueryFields.Takedowns,\n    QueryFields.UrlEntities,\n    QueryFields.DirectMessageView,\n    QueryFields.MediaView\n  )\n\n  val GetUserByIdUserNameQueryFields: Set[QueryFields] = Set(\n    QueryFields.Profile\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/gizmoduck/GizmoduckModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.gizmoduck\n\nimport com.google.inject.Provides\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.follow_recommendations.common.clients.common.BaseClientModule\nimport com.twitter.gizmoduck.thriftscala.QueryFields\nimport com.twitter.gizmoduck.thriftscala.UserService\nimport com.twitter.stitch.gizmoduck.Gizmoduck\nimport javax.inject.Singleton\n\nobject GizmoduckModule extends BaseClientModule[UserService.MethodPerEndpoint] with MtlsClient {\n  override val label = \"gizmoduck\"\n  override val dest = \"/s/gizmoduck/gizmoduck\"\n\n  @Provides\n  @Singleton\n  def provideExtraGizmoduckQueryFields: Set[QueryFields] = Set.empty\n\n  @Provides\n  @Singleton\n  def providesStitchClient(futureIface: UserService.MethodPerEndpoint): Gizmoduck = {\n    Gizmoduck(futureIface)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/graph_feature_service/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-thrift-client\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"graph-feature-service/src/main/thrift/com/twitter/graph_feature_service:graph_feature_service_thrift-scala\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/graph_feature_service/GraphFeatureServiceClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.graph_feature_service\n\nimport com.twitter.follow_recommendations.common.models.FollowProof\nimport com.twitter.graph_feature_service.thriftscala.PresetFeatureTypes.WtfTwoHop\nimport com.twitter.graph_feature_service.thriftscala.EdgeType\nimport com.twitter.graph_feature_service.thriftscala.GfsIntersectionResponse\nimport com.twitter.graph_feature_service.thriftscala.GfsPresetIntersectionRequest\nimport com.twitter.graph_feature_service.thriftscala.{Server => GraphFeatureService}\nimport com.twitter.stitch.Stitch\nimport javax.inject.{Inject, Singleton}\n\n@Singleton\nclass GraphFeatureServiceClient @Inject() (\n  graphFeatureService: GraphFeatureService.MethodPerEndpoint) {\n\n  import GraphFeatureServiceClient._\n  def getIntersections(\n    userId: Long,\n    candidateIds: Seq[Long],\n    numIntersectionIds: Int\n  ): Stitch[Map[Long, FollowProof]] = {\n    Stitch\n      .callFuture(\n        graphFeatureService.getPresetIntersection(\n          GfsPresetIntersectionRequest(userId, candidateIds, WtfTwoHop, Some(numIntersectionIds))\n        )\n      ).map {\n        case GfsIntersectionResponse(gfsIntersectionResults) =>\n          (for {\n            candidateId <- candidateIds\n            gfsIntersectionResultForCandidate =\n              gfsIntersectionResults.filter(_.candidateUserId == candidateId)\n            followProof <- for {\n              result <- gfsIntersectionResultForCandidate\n              intersection <- result.intersectionValues\n              if leftEdgeTypes.contains(intersection.featureType.leftEdgeType)\n              if rightEdgeTypes.contains(intersection.featureType.rightEdgeType)\n              intersectionIds <- intersection.intersectionIds.toSeq\n            } yield FollowProof(intersectionIds, intersection.count.getOrElse(0))\n          } yield {\n            candidateId -> followProof\n          }).toMap\n      }\n  }\n}\n\nobject GraphFeatureServiceClient {\n  val leftEdgeTypes: Set[EdgeType] = Set(EdgeType.Following)\n  val rightEdgeTypes: Set[EdgeType] = Set(EdgeType.FollowedBy)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/graph_feature_service/GraphFeatureStoreModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.graph_feature_service\n\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.follow_recommendations.common.clients.common.BaseClientModule\nimport com.twitter.graph_feature_service.thriftscala.{Server => GraphFeatureService}\n\nobject GraphFeatureStoreModule\n    extends BaseClientModule[GraphFeatureService.MethodPerEndpoint]\n    with MtlsClient {\n  override val label = \"graph_feature_service\"\n  override val dest = \"/s/cassowary/graph_feature_service-server\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/impression_store/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/github/nscala_time\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"stitch/stitch-socialgraph\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/impression_store/ImpressionStoreModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.impression_store\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.follow_recommendations.thriftscala.DisplayLocation\nimport com.twitter.inject.TwitterModule\nimport com.twitter.strato.catalog.Scan.Slice\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.thrift.ScroogeConvImplicits._\n\nobject ImpressionStoreModule extends TwitterModule {\n\n  val columnPath: String = \"onboarding/userrecs/wtfImpressionCountsStore\"\n\n  type PKey = (Long, DisplayLocation)\n  type LKey = Long\n  type Value = (Long, Int)\n\n  @Provides\n  @Singleton\n  def providesImpressionStore(stratoClient: Client): WtfImpressionStore = {\n    new WtfImpressionStore(\n      stratoClient.scanner[\n        (PKey, Slice[LKey]),\n        Unit,\n        (PKey, LKey),\n        Value\n      ](columnPath)\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/impression_store/WtfImpressionStore.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.impression_store\n\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.common.models.WtfImpression\nimport com.twitter.follow_recommendations.thriftscala.{DisplayLocation => TDisplayLocation}\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.Scan.Slice\nimport com.twitter.strato.client.Scanner\nimport com.twitter.util.Time\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass WtfImpressionStore @Inject() (\n  scanner: Scanner[\n    ((Long, TDisplayLocation), Slice[Long]),\n    Unit,\n    ((Long, TDisplayLocation), Long),\n    (Long, Int)\n  ]) extends Logging {\n  def get(userId: Long, dl: DisplayLocation): Stitch[Seq[WtfImpression]] = {\n    val thriftDl = dl.toThrift\n    scanner.scan(((userId, thriftDl), Slice.all[Long])).map { impressionsPerDl =>\n      val wtfImpressions =\n        for {\n          (((_, _), candidateId), (latestTs, counts)) <- impressionsPerDl\n        } yield WtfImpression(\n          candidateId = candidateId,\n          displayLocation = dl,\n          latestTime = Time.fromMilliseconds(latestTs),\n          counts = counts\n        )\n      wtfImpressions\n    } rescue {\n      // fail open so that the request can still go through\n      case ex: Throwable =>\n        logger.warn(s\"$dl WtfImpressionsStore warn: \" + ex.getMessage)\n        Stitch.Nil\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/interests_service/BUILD",
    "content": "scala_library(\n    name = \"interests_service\",\n    sources = [\"InterestServiceClient.scala\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/interests\",\n        \"interests-service/thrift/src/main/thrift:thrift-scala\",\n        \"strato/src/main/scala/com/twitter/strato/catalog\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"strato/src/main/scala/com/twitter/strato/data\",\n        \"strato/src/main/scala/com/twitter/strato/thrift\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/interests_service/InterestServiceClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.interests_service\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.InterestedInInterestsFetchKey\nimport com.twitter.inject.Logging\nimport com.twitter.interests.thriftscala.InterestId\nimport com.twitter.interests.thriftscala.InterestRelationship\nimport com.twitter.interests.thriftscala.InterestedInInterestModel\nimport com.twitter.interests.thriftscala.UserInterest\nimport com.twitter.interests.thriftscala.UserInterestData\nimport com.twitter.interests.thriftscala.UserInterestsResponse\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.thrift.ScroogeConvImplicits._\n\n@Singleton\nclass InterestServiceClient @Inject() (\n  stratoClient: Client,\n  statsReceiver: StatsReceiver = NullStatsReceiver)\n    extends Logging {\n\n  val interestsServiceStratoColumnPath = \"interests/interestedInInterests\"\n  val stats = statsReceiver.scope(\"interest_service_client\")\n  val errorCounter = stats.counter(\"error\")\n\n  private val interestsFetcher =\n    stratoClient.fetcher[InterestedInInterestsFetchKey, UserInterestsResponse](\n      interestsServiceStratoColumnPath,\n      checkTypes = true\n    )\n\n  def fetchUttInterestIds(\n    userId: Long\n  ): Stitch[Seq[Long]] = {\n    fetchInterestRelationships(userId)\n      .map(_.toSeq.flatten.flatMap(extractUttInterest))\n  }\n\n  def extractUttInterest(\n    interestRelationShip: InterestRelationship\n  ): Option[Long] = {\n    interestRelationShip match {\n      case InterestRelationship.V1(relationshipV1) =>\n        relationshipV1.interestId match {\n          case InterestId.SemanticCore(semanticCoreInterest) => Some(semanticCoreInterest.id)\n          case _ => None\n        }\n      case _ => None\n    }\n  }\n\n  def fetchCustomInterests(\n    userId: Long\n  ): Stitch[Seq[String]] = {\n    fetchInterestRelationships(userId)\n      .map(_.toSeq.flatten.flatMap(extractCustomInterest))\n  }\n\n  def extractCustomInterest(\n    interestRelationShip: InterestRelationship\n  ): Option[String] = {\n    interestRelationShip match {\n      case InterestRelationship.V1(relationshipV1) =>\n        relationshipV1.interestId match {\n          case InterestId.FreeForm(freeFormInterest) => Some(freeFormInterest.interest)\n          case _ => None\n        }\n      case _ => None\n    }\n  }\n\n  def fetchInterestRelationships(\n    userId: Long\n  ): Stitch[Option[Seq[InterestRelationship]]] = {\n    interestsFetcher\n      .fetch(\n        InterestedInInterestsFetchKey(\n          userId = userId,\n          labels = None,\n          None\n        ))\n      .map(_.v)\n      .map {\n        case Some(response) =>\n          response.interests.interests.map { interests =>\n            interests.collect {\n              case UserInterest(_, Some(interestData)) =>\n                getInterestRelationship(interestData)\n            }.flatten\n          }\n        case _ => None\n      }\n      .rescue {\n        case e: Throwable => // we are swallowing all errors\n          logger.warn(s\"interests could not be retrieved for user $userId due to ${e.getCause}\")\n          errorCounter.incr\n          Stitch.None\n      }\n  }\n\n  private def getInterestRelationship(\n    interestData: UserInterestData\n  ): Seq[InterestRelationship] = {\n    interestData match {\n      case UserInterestData.InterestedIn(interestModels) =>\n        interestModels.collect {\n          case InterestedInInterestModel.ExplicitModel(model) => model\n        }\n      case _ => Nil\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/phone_storage_service/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-thrift-client\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"phonestorage/server/src/main/thrift/com/twitter/phonestorage/api:phone-storage-service-scala\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/phone_storage_service/PhoneStorageServiceClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.phone_storage_service\n\nimport com.twitter.cds.contact_consent_state.thriftscala.PurposeOfProcessing\nimport com.twitter.phonestorage.api.thriftscala.GetUserPhonesByUsersRequest\nimport com.twitter.phonestorage.api.thriftscala.PhoneStorageService\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PhoneStorageServiceClient @Inject() (\n  val phoneStorageService: PhoneStorageService.MethodPerEndpoint) {\n\n  /**\n   * PSS can potentially return multiple phone records.\n   * The current implementation of getUserPhonesByUsers returns only a single phone for a single user_id but\n   * we can trivially support handling multiple in case that changes in the future.\n   */\n  def getPhoneNumbers(\n    userId: Long,\n    purposeOfProcessing: PurposeOfProcessing,\n    forceCarrierLookup: Option[Boolean] = None\n  ): Stitch[Seq[String]] = {\n    val req = GetUserPhonesByUsersRequest(\n      userIds = Seq(userId),\n      forceCarrierLookup = forceCarrierLookup,\n      purposesOfProcessing = Some(Seq(purposeOfProcessing))\n    )\n\n    Stitch.callFuture(phoneStorageService.getUserPhonesByUsers(req)) map {\n      _.userPhones.map(_.phoneNumber)\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/phone_storage_service/PhoneStorageServiceModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.phone_storage_service\n\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.follow_recommendations.common.clients.common.BaseClientModule\nimport com.twitter.phonestorage.api.thriftscala.PhoneStorageService\n\nobject PhoneStorageServiceModule\n    extends BaseClientModule[PhoneStorageService.MethodPerEndpoint]\n    with MtlsClient {\n  override val label = \"phone-storage-service\"\n  override val dest = \"/s/ibis-ds-api/ibis-ds-api:thrift2\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/real_time_real_graph/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"strato/config/columns/ml/featureStore:featureStore-strato-client\",\n        \"strato/config/columns/onboarding/userrecs:userrecs-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/real_time_real_graph/Engagement.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.real_time_real_graph\n\nsealed trait EngagementType\n\n// We do not include SoftFollow since it's deprecated\nobject EngagementType {\n  object Click extends EngagementType\n  object Like extends EngagementType\n  object Mention extends EngagementType\n  object Retweet extends EngagementType\n  object ProfileView extends EngagementType\n}\n\ncase class Engagement(engagementType: EngagementType, timestamp: Long)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/real_time_real_graph/EngagementScorer.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.real_time_real_graph\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.util.Time\n\nobject EngagementScorer {\n  private[real_time_real_graph] val MemoryDecayHalfLife = 24.hour\n  private val ScoringFunctionBase = 0.5\n\n  def apply(\n    engagements: Map[Long, Seq[Engagement]],\n    engagementScoreMap: Map[EngagementType, Double],\n    minScore: Double = 0.0\n  ): Seq[(Long, Double, Seq[EngagementType])] = {\n    val now = Time.now\n    engagements\n      .mapValues { engags =>\n        val totalScore = engags.map { engagement => score(engagement, now, engagementScoreMap) }.sum\n        val engagementProof = getEngagementProof(engags, engagementScoreMap)\n        (totalScore, engagementProof)\n      }\n      .collect { case (uid, (score, proof)) if score > minScore => (uid, score, proof) }\n      .toSeq\n      .sortBy(-_._2)\n  }\n\n  /**\n   * The engagement score is the base score decayed via timestamp, loosely model the human memory forgetting\n   * curve, see https://en.wikipedia.org/wiki/Forgetting_curve\n   */\n  private[real_time_real_graph] def score(\n    engagement: Engagement,\n    now: Time,\n    engagementScoreMap: Map[EngagementType, Double]\n  ): Double = {\n    val timeLapse = math.max(now.inMillis - engagement.timestamp, 0)\n    val engagementScore = engagementScoreMap.getOrElse(engagement.engagementType, 0.0)\n    engagementScore * math.pow(\n      ScoringFunctionBase,\n      timeLapse.toDouble / MemoryDecayHalfLife.inMillis)\n  }\n\n  private def getEngagementProof(\n    engagements: Seq[Engagement],\n    engagementScoreMap: Map[EngagementType, Double]\n  ): Seq[EngagementType] = {\n\n    val filteredEngagement = engagements\n      .collectFirst {\n        case engagement\n            if engagement.engagementType != EngagementType.Click\n              && engagementScoreMap.get(engagement.engagementType).exists(_ > 0.0) =>\n          engagement.engagementType\n      }\n\n    Seq(filteredEngagement.getOrElse(EngagementType.Click))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/real_time_real_graph/RealTimeRealGraphClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.real_time_real_graph\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.ml.featureStore.TimelinesUserVertexOnUserClientColumn\nimport com.twitter.strato.generated.client.onboarding.userrecs.RealGraphScoresMhOnUserClientColumn\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\nimport com.twitter.wtf.real_time_interaction_graph.thriftscala._\n\n@Singleton\nclass RealTimeRealGraphClient @Inject() (\n  timelinesUserVertexOnUserClientColumn: TimelinesUserVertexOnUserClientColumn,\n  realGraphScoresMhOnUserClientColumn: RealGraphScoresMhOnUserClientColumn) {\n\n  def mapUserVertexToEngagementAndFilter(userVertex: UserVertex): Map[Long, Seq[Engagement]] = {\n    val minTimestamp = (Time.now - RealTimeRealGraphClient.MaxEngagementAge).inMillis\n    userVertex.outgoingInteractionMap.mapValues { interactions =>\n      interactions\n        .flatMap { interaction => RealTimeRealGraphClient.toEngagement(interaction) }.filter(\n          _.timestamp >= minTimestamp)\n    }.toMap\n  }\n\n  def getRecentProfileViewEngagements(userId: Long): Stitch[Map[Long, Seq[Engagement]]] = {\n    timelinesUserVertexOnUserClientColumn.fetcher\n      .fetch(userId).map(_.v).map { input =>\n        input\n          .map { userVertex =>\n            val targetToEngagements = mapUserVertexToEngagementAndFilter(userVertex)\n            targetToEngagements.mapValues { engagements =>\n              engagements.filter(engagement =>\n                engagement.engagementType == EngagementType.ProfileView)\n            }\n          }.getOrElse(Map.empty)\n      }\n  }\n\n  def getUsersRecentlyEngagedWith(\n    userId: Long,\n    engagementScoreMap: Map[EngagementType, Double],\n    includeDirectFollowCandidates: Boolean,\n    includeNonDirectFollowCandidates: Boolean\n  ): Stitch[Seq[CandidateUser]] = {\n    val isNewUser =\n      SnowflakeId.timeFromIdOpt(userId).exists { signupTime =>\n        (Time.now - signupTime) < RealTimeRealGraphClient.MaxNewUserAge\n      }\n    val updatedEngagementScoreMap =\n      if (isNewUser)\n        engagementScoreMap + (EngagementType.ProfileView -> RealTimeRealGraphClient.ProfileViewScore)\n      else engagementScoreMap\n\n    Stitch\n      .join(\n        timelinesUserVertexOnUserClientColumn.fetcher.fetch(userId).map(_.v),\n        realGraphScoresMhOnUserClientColumn.fetcher.fetch(userId).map(_.v)).map {\n        case (Some(userVertex), Some(neighbors)) =>\n          val engagements = mapUserVertexToEngagementAndFilter(userVertex)\n\n          val candidatesAndScores: Seq[(Long, Double, Seq[EngagementType])] =\n            EngagementScorer.apply(engagements, engagementScoreMap = updatedEngagementScoreMap)\n\n          val directNeighbors = neighbors.candidates.map(_._1).toSet\n          val (directFollows, nonDirectFollows) = candidatesAndScores\n            .partition {\n              case (id, _, _) => directNeighbors.contains(id)\n            }\n\n          val candidates =\n            (if (includeNonDirectFollowCandidates) nonDirectFollows else Seq.empty) ++\n              (if (includeDirectFollowCandidates)\n                 directFollows.take(RealTimeRealGraphClient.MaxNumDirectFollow)\n               else Seq.empty)\n\n          candidates.map {\n            case (id, score, proof) =>\n              CandidateUser(id, Some(score))\n          }\n\n        case _ => Nil\n      }\n  }\n\n  def getRealGraphWeights(userId: Long): Stitch[Map[Long, Double]] =\n    realGraphScoresMhOnUserClientColumn.fetcher\n      .fetch(userId)\n      .map(\n        _.v\n          .map(_.candidates.map(candidate => (candidate.userId, candidate.score)).toMap)\n          .getOrElse(Map.empty[Long, Double]))\n}\n\nobject RealTimeRealGraphClient {\n  private def toEngagement(interaction: Interaction): Option[Engagement] = {\n    // We do not include SoftFollow since it's deprecated\n    interaction match {\n      case Interaction.Retweet(Retweet(timestamp)) =>\n        Some(Engagement(EngagementType.Retweet, timestamp))\n      case Interaction.Favorite(Favorite(timestamp)) =>\n        Some(Engagement(EngagementType.Like, timestamp))\n      case Interaction.Click(Click(timestamp)) => Some(Engagement(EngagementType.Click, timestamp))\n      case Interaction.Mention(Mention(timestamp)) =>\n        Some(Engagement(EngagementType.Mention, timestamp))\n      case Interaction.ProfileView(ProfileView(timestamp)) =>\n        Some(Engagement(EngagementType.ProfileView, timestamp))\n      case _ => None\n    }\n  }\n\n  val MaxNumDirectFollow = 50\n  val MaxEngagementAge: Duration = 14.days\n  val MaxNewUserAge: Duration = 30.days\n  val ProfileViewScore = 0.4\n  val EngagementScoreMap = Map(\n    EngagementType.Like -> 1.0,\n    EngagementType.Retweet -> 1.0,\n    EngagementType.Mention -> 1.0\n  )\n  val StrongEngagementScoreMap = Map(\n    EngagementType.Like -> 1.0,\n    EngagementType.Retweet -> 1.0,\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/socialgraph/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/github/nscala_time\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"escherbird/src/scala/com/twitter/escherbird/util/stitchcache\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"socialgraph/server/src/main/scala/com/twitter/socialgraph/util\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"stitch/stitch-socialgraph\",\n        \"strato/config/columns/onboarding/socialGraphService:socialGraphService-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/socialgraph/SocialGraphClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.socialgraph\n\nimport com.twitter.escherbird.util.stitchcache.StitchCache\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.follow_recommendations.common.models.FollowProof\nimport com.twitter.follow_recommendations.common.models.UserIdWithTimestamp\nimport com.twitter.inject.Logging\nimport com.twitter.socialgraph.thriftscala.EdgesRequest\nimport com.twitter.socialgraph.thriftscala.IdsRequest\nimport com.twitter.socialgraph.thriftscala.IdsResult\nimport com.twitter.socialgraph.thriftscala.LookupContext\nimport com.twitter.socialgraph.thriftscala.OverCapacity\nimport com.twitter.socialgraph.thriftscala.PageRequest\nimport com.twitter.socialgraph.thriftscala.RelationshipType\nimport com.twitter.socialgraph.thriftscala.SrcRelationship\nimport com.twitter.socialgraph.util.ByteBufferUtil\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.SocialGraph\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.onboarding.socialGraphService.IdsClientColumn\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\nimport java.nio.ByteBuffer\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase class RecentEdgesQuery(\n  userId: Long,\n  relations: Seq[RelationshipType],\n  // prefer to default value to better utilize the caching function of stitch\n  count: Option[Int] = Some(SocialGraphClient.MaxQuerySize),\n  performUnion: Boolean = true,\n  recentEdgesWindowOpt: Option[Duration] = None,\n  targets: Option[Seq[Long]] = None)\n\ncase class EdgeRequestQuery(\n  userId: Long,\n  relation: RelationshipType,\n  count: Option[Int] = Some(SocialGraphClient.MaxQuerySize),\n  performUnion: Boolean = true,\n  recentEdgesWindowOpt: Option[Duration] = None,\n  targets: Option[Seq[Long]] = None)\n\n@Singleton\nclass SocialGraphClient @Inject() (\n  socialGraph: SocialGraph,\n  idsClientColumn: IdsClientColumn,\n  statsReceiver: StatsReceiver = NullStatsReceiver)\n    extends Logging {\n\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val cacheStats = stats.scope(\"cache\")\n  private val getIntersectionsStats = stats.scope(\"getIntersections\")\n  private val getIntersectionsFromCachedColumnStats =\n    stats.scope(\"getIntersectionsFromCachedColumn\")\n  private val getRecentEdgesStats = stats.scope(\"getRecentEdges\")\n  private val getRecentEdgesCachedStats = stats.scope(\"getRecentEdgesCached\")\n  private val getRecentEdgesFromCachedColumnStats = stats.scope(\"getRecentEdgesFromCachedColumn\")\n  private val getRecentEdgesCachedInternalStats = stats.scope(\"getRecentEdgesCachedInternal\")\n  private val getRecentEdgesWithTimeStats = stats.scope(\"getRecentEdgesWithTime\")\n\n  val sgsIdsFetcher: Fetcher[IdsRequest, Unit, IdsResult] = idsClientColumn.fetcher\n\n  private val recentEdgesCache = StitchCache[RecentEdgesQuery, Seq[Long]](\n    maxCacheSize = SocialGraphClient.MaxCacheSize,\n    ttl = SocialGraphClient.CacheTTL,\n    statsReceiver = cacheStats,\n    underlyingCall = getRecentEdges\n  )\n\n  def getRecentEdgesCached(\n    rq: RecentEdgesQuery,\n    useCachedStratoColumn: Boolean = true\n  ): Stitch[Seq[Long]] = {\n    getRecentEdgesCachedStats.counter(\"requests\").incr()\n    if (useCachedStratoColumn) {\n      getRecentEdgesFromCachedColumn(rq)\n    } else {\n      StatsUtil.profileStitch(\n        getRecentEdgesCachedInternal(rq),\n        getRecentEdgesCachedInternalStats\n      )\n    }\n  }\n\n  def getRecentEdgesCachedInternal(rq: RecentEdgesQuery): Stitch[Seq[Long]] = {\n    recentEdgesCache.readThrough(rq)\n  }\n\n  def getRecentEdgesFromCachedColumn(rq: RecentEdgesQuery): Stitch[Seq[Long]] = {\n    val pageRequest = rq.recentEdgesWindowOpt match {\n      case Some(recentEdgesWindow) =>\n        PageRequest(\n          count = rq.count,\n          cursor = Some(getEdgeCursor(recentEdgesWindow)),\n          selectAll = Some(true)\n        )\n      case _ => PageRequest(count = rq.count)\n    }\n    val idsRequest = IdsRequest(\n      rq.relations.map { relationshipType =>\n        SrcRelationship(\n          source = rq.userId,\n          relationshipType = relationshipType,\n          targets = rq.targets\n        )\n      },\n      pageRequest = Some(pageRequest),\n      context = Some(LookupContext(performUnion = Some(rq.performUnion)))\n    )\n\n    val socialGraphStitch = sgsIdsFetcher\n      .fetch(idsRequest, Unit)\n      .map(_.v)\n      .map { result =>\n        result\n          .map { idResult =>\n            val userIds: Seq[Long] = idResult.ids\n            getRecentEdgesFromCachedColumnStats.stat(\"num_edges\").add(userIds.size)\n            userIds\n          }.getOrElse(Seq.empty)\n      }\n      .rescue {\n        case e: Exception =>\n          stats.counter(e.getClass.getSimpleName).incr()\n          Stitch.Nil\n      }\n\n    StatsUtil.profileStitch(\n      socialGraphStitch,\n      getRecentEdgesFromCachedColumnStats\n    )\n  }\n\n  def getRecentEdges(rq: RecentEdgesQuery): Stitch[Seq[Long]] = {\n    val pageRequest = rq.recentEdgesWindowOpt match {\n      case Some(recentEdgesWindow) =>\n        PageRequest(\n          count = rq.count,\n          cursor = Some(getEdgeCursor(recentEdgesWindow)),\n          selectAll = Some(true)\n        )\n      case _ => PageRequest(count = rq.count)\n    }\n    val socialGraphStitch = socialGraph\n      .ids(\n        IdsRequest(\n          rq.relations.map { relationshipType =>\n            SrcRelationship(\n              source = rq.userId,\n              relationshipType = relationshipType,\n              targets = rq.targets\n            )\n          },\n          pageRequest = Some(pageRequest),\n          context = Some(LookupContext(performUnion = Some(rq.performUnion)))\n        )\n      )\n      .map { idsResult =>\n        val userIds: Seq[Long] = idsResult.ids\n        getRecentEdgesStats.stat(\"num_edges\").add(userIds.size)\n        userIds\n      }\n      .rescue {\n        case e: OverCapacity =>\n          stats.counter(e.getClass.getSimpleName).incr()\n          logger.warn(\"SGS Over Capacity\", e)\n          Stitch.Nil\n      }\n    StatsUtil.profileStitch(\n      socialGraphStitch,\n      getRecentEdgesStats\n    )\n  }\n\n  // This method return recent edges of (userId, timeInMs)\n  def getRecentEdgesWithTime(rq: EdgeRequestQuery): Stitch[Seq[UserIdWithTimestamp]] = {\n    val pageRequest = rq.recentEdgesWindowOpt match {\n      case Some(recentEdgesWindow) =>\n        PageRequest(\n          count = rq.count,\n          cursor = Some(getEdgeCursor(recentEdgesWindow)),\n          selectAll = Some(true)\n        )\n      case _ => PageRequest(count = rq.count)\n    }\n\n    val socialGraphStitch = socialGraph\n      .edges(\n        EdgesRequest(\n          SrcRelationship(\n            source = rq.userId,\n            relationshipType = rq.relation,\n            targets = rq.targets\n          ),\n          pageRequest = Some(pageRequest),\n          context = Some(LookupContext(performUnion = Some(rq.performUnion)))\n        )\n      )\n      .map { edgesResult =>\n        val userIds = edgesResult.edges.map { socialEdge =>\n          UserIdWithTimestamp(socialEdge.target, socialEdge.updatedAt)\n        }\n        getRecentEdgesWithTimeStats.stat(\"num_edges\").add(userIds.size)\n        userIds\n      }\n      .rescue {\n        case e: OverCapacity =>\n          stats.counter(e.getClass.getSimpleName).incr()\n          logger.warn(\"SGS Over Capacity\", e)\n          Stitch.Nil\n      }\n    StatsUtil.profileStitch(\n      socialGraphStitch,\n      getRecentEdgesWithTimeStats\n    )\n  }\n\n  // This method returns the cursor for a time duration, such that all the edges returned by SGS will be created\n  // in the range (now-window, now)\n  def getEdgeCursor(window: Duration): ByteBuffer = {\n    val cursorInLong = (-(Time.now - window).inMilliseconds) << 20\n    ByteBufferUtil.fromLong(cursorInLong)\n  }\n\n  // notice that this is more expensive but more realtime than the GFS one\n  def getIntersections(\n    userId: Long,\n    candidateIds: Seq[Long],\n    numIntersectionIds: Int\n  ): Stitch[Map[Long, FollowProof]] = {\n    val socialGraphStitch: Stitch[Map[Long, FollowProof]] = Stitch\n      .collect(candidateIds.map { candidateId =>\n        socialGraph\n          .ids(\n            IdsRequest(\n              Seq(\n                SrcRelationship(userId, RelationshipType.Following),\n                SrcRelationship(candidateId, RelationshipType.FollowedBy)\n              ),\n              pageRequest = Some(PageRequest(count = Some(numIntersectionIds)))\n            )\n          ).map { idsResult =>\n            getIntersectionsStats.stat(\"num_edges\").add(idsResult.ids.size)\n            (candidateId -> FollowProof(idsResult.ids, idsResult.ids.size))\n          }\n      }).map(_.toMap)\n      .rescue {\n        case e: OverCapacity =>\n          stats.counter(e.getClass.getSimpleName).incr()\n          logger.warn(\"social graph over capacity in hydrating social proof\", e)\n          Stitch.value(Map.empty)\n      }\n    StatsUtil.profileStitch(\n      socialGraphStitch,\n      getIntersectionsStats\n    )\n  }\n\n  def getIntersectionsFromCachedColumn(\n    userId: Long,\n    candidateIds: Seq[Long],\n    numIntersectionIds: Int\n  ): Stitch[Map[Long, FollowProof]] = {\n    val socialGraphStitch: Stitch[Map[Long, FollowProof]] = Stitch\n      .collect(candidateIds.map { candidateId =>\n        val idsRequest = IdsRequest(\n          Seq(\n            SrcRelationship(userId, RelationshipType.Following),\n            SrcRelationship(candidateId, RelationshipType.FollowedBy)\n          ),\n          pageRequest = Some(PageRequest(count = Some(numIntersectionIds)))\n        )\n\n        sgsIdsFetcher\n          .fetch(idsRequest, Unit)\n          .map(_.v)\n          .map { resultOpt =>\n            resultOpt.map { idsResult =>\n              getIntersectionsFromCachedColumnStats.stat(\"num_edges\").add(idsResult.ids.size)\n              candidateId -> FollowProof(idsResult.ids, idsResult.ids.size)\n            }\n          }\n      }).map(_.flatten.toMap)\n      .rescue {\n        case e: Exception =>\n          stats.counter(e.getClass.getSimpleName).incr()\n          Stitch.value(Map.empty)\n      }\n    StatsUtil.profileStitch(\n      socialGraphStitch,\n      getIntersectionsFromCachedColumnStats\n    )\n  }\n\n  def getInvalidRelationshipUserIds(\n    userId: Long,\n    maxNumRelationship: Int = SocialGraphClient.MaxNumInvalidRelationship\n  ): Stitch[Seq[Long]] = {\n    getRecentEdges(\n      RecentEdgesQuery(\n        userId,\n        SocialGraphClient.InvalidRelationshipTypes,\n        Some(maxNumRelationship)\n      )\n    )\n  }\n\n  def getInvalidRelationshipUserIdsFromCachedColumn(\n    userId: Long,\n    maxNumRelationship: Int = SocialGraphClient.MaxNumInvalidRelationship\n  ): Stitch[Seq[Long]] = {\n    getRecentEdgesFromCachedColumn(\n      RecentEdgesQuery(\n        userId,\n        SocialGraphClient.InvalidRelationshipTypes,\n        Some(maxNumRelationship)\n      )\n    )\n  }\n\n  def getRecentFollowedUserIds(userId: Long): Stitch[Seq[Long]] = {\n    getRecentEdges(\n      RecentEdgesQuery(\n        userId,\n        Seq(RelationshipType.Following)\n      )\n    )\n  }\n\n  def getRecentFollowedUserIdsFromCachedColumn(userId: Long): Stitch[Seq[Long]] = {\n    getRecentEdgesFromCachedColumn(\n      RecentEdgesQuery(\n        userId,\n        Seq(RelationshipType.Following)\n      )\n    )\n  }\n\n  def getRecentFollowedUserIdsWithTime(userId: Long): Stitch[Seq[UserIdWithTimestamp]] = {\n    getRecentEdgesWithTime(\n      EdgeRequestQuery(\n        userId,\n        RelationshipType.Following\n      )\n    )\n  }\n\n  def getRecentFollowedByUserIds(userId: Long): Stitch[Seq[Long]] = {\n    getRecentEdges(\n      RecentEdgesQuery(\n        userId,\n        Seq(RelationshipType.FollowedBy)\n      )\n    )\n  }\n\n  def getRecentFollowedByUserIdsFromCachedColumn(userId: Long): Stitch[Seq[Long]] = {\n    getRecentEdgesFromCachedColumn(\n      RecentEdgesQuery(\n        userId,\n        Seq(RelationshipType.FollowedBy)\n      )\n    )\n  }\n\n  def getRecentFollowedUserIdsWithTimeWindow(\n    userId: Long,\n    timeWindow: Duration\n  ): Stitch[Seq[Long]] = {\n    getRecentEdges(\n      RecentEdgesQuery(\n        userId,\n        Seq(RelationshipType.Following),\n        recentEdgesWindowOpt = Some(timeWindow)\n      )\n    )\n  }\n}\n\nobject SocialGraphClient {\n\n  val MaxQuerySize: Int = 500\n  val MaxCacheSize: Int = 5000000\n  // Ref: src/thrift/com/twitter/socialgraph/social_graph_service.thrift\n  val MaxNumInvalidRelationship: Int = 5000\n  val CacheTTL: Duration = Duration.fromHours(24)\n\n  val InvalidRelationshipTypes: Seq[RelationshipType] = Seq(\n    RelationshipType.HideRecommendations,\n    RelationshipType.Blocking,\n    RelationshipType.BlockedBy,\n    RelationshipType.Muting,\n    RelationshipType.MutedBy,\n    RelationshipType.ReportedAsSpam,\n    RelationshipType.ReportedAsSpamBy,\n    RelationshipType.ReportedAsAbuse,\n    RelationshipType.ReportedAsAbuseBy,\n    RelationshipType.FollowRequestOutgoing,\n    RelationshipType.Following,\n    RelationshipType.UsedToFollow,\n  )\n\n  /**\n   *\n   * Whether to call SGS to validate each candidate based on the number of invalid relationship users\n   * prefetched during request building step. This aims to not omit any invalid candidates that are\n   * not filtered out in previous steps.\n   *   If the number is 0, this might be a fail-opened SGS call.\n   *   If the number is larger or equal to 5000, this could hit SGS page size limit.\n   * Both cases account for a small percentage of the total traffic (<5%).\n   *\n   * @param numInvalidRelationshipUsers number of invalid relationship users fetched from getInvalidRelationshipUserIds\n   * @return whether to enable post-ranker SGS predicate\n   */\n  def enablePostRankerSgsPredicate(numInvalidRelationshipUsers: Int): Boolean = {\n    numInvalidRelationshipUsers == 0 || numInvalidRelationshipUsers >= MaxNumInvalidRelationship\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/socialgraph/SocialGraphModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.socialgraph\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.follow_recommendations.common.clients.common.BaseClientModule\nimport com.twitter.socialgraph.thriftscala.SocialGraphService\nimport com.twitter.stitch.socialgraph.SocialGraph\nimport javax.inject.Singleton\n\nobject SocialGraphModule\n    extends BaseClientModule[SocialGraphService.MethodPerEndpoint]\n    with MtlsClient {\n  override val label = \"social-graph-service\"\n  override val dest = \"/s/socialgraph/socialgraph\"\n\n  override def configureThriftMuxClient(client: ThriftMux.Client): ThriftMux.Client =\n    client.withSessionQualifier.noFailFast\n\n  @Provides\n  @Singleton\n  def providesStitchClient(futureIface: SocialGraphService.MethodPerEndpoint): SocialGraph = {\n    SocialGraph(futureIface)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"src/scala/com/twitter/onboarding/relevance/candidate_generation/utt/models\",\n        \"src/thrift/com/twitter/core_workflows/user_model:user_model-scala\",\n        \"src/thrift/com/twitter/frigate/data_pipeline:frigate-user-history-thrift-scala\",\n        \"src/thrift/com/twitter/hermit/candidate:hermit-candidate-scala\",\n        \"src/thrift/com/twitter/hermit/pop_geo:hermit-pop-geo-scala\",\n        \"src/thrift/com/twitter/onboarding/relevance/relatable_accounts:relatable_accounts-scala\",\n        \"src/thrift/com/twitter/onboarding/relevance/store:store-scala\",\n        \"src/thrift/com/twitter/recos/user_user_graph:user_user_graph-scala\",\n        \"src/thrift/com/twitter/search/account_search/extended_network:extended_network_users-scala\",\n        \"src/thrift/com/twitter/service/metastore/gen:thrift-scala\",\n        \"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala\",\n        \"src/thrift/com/twitter/wtf/ml:wtf-ml-output-thrift-scala\",\n        \"src/thrift/com/twitter/wtf/real_time_interaction_graph:wtf-real_time_interaction_graph-thrift-scala\",\n        \"src/thrift/com/twitter/wtf/triangular_loop:triangular_loop-scala\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato/StratoClientModule.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.strato\n\nimport com.google.inject.name.Named\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.core_workflows.user_model.thriftscala.CondensedUserState\nimport com.twitter.search.account_search.extended_network.thriftscala.ExtendedNetworkUserKey\nimport com.twitter.search.account_search.extended_network.thriftscala.ExtendedNetworkUserVal\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.thrift.Protocols\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.follow_recommendations.common.constants.ServiceConstants._\nimport com.twitter.frigate.data_pipeline.candidate_generation.thriftscala.LatestEvents\nimport com.twitter.hermit.candidate.thriftscala.{Candidates => HermitCandidates}\nimport com.twitter.hermit.pop_geo.thriftscala.PopUsersInPlace\nimport com.twitter.onboarding.relevance.relatable_accounts.thriftscala.RelatableAccounts\nimport com.twitter.inject.TwitterModule\nimport com.twitter.onboarding.relevance.candidates.thriftscala.InterestBasedUserRecommendations\nimport com.twitter.onboarding.relevance.candidates.thriftscala.UTTInterest\nimport com.twitter.onboarding.relevance.store.thriftscala.WhoToFollowDismissEventDetails\nimport com.twitter.recos.user_user_graph.thriftscala.RecommendUserRequest\nimport com.twitter.recos.user_user_graph.thriftscala.RecommendUserResponse\nimport com.twitter.service.metastore.gen.thriftscala.UserRecommendabilityFeatures\nimport com.twitter.strato.catalog.Scan.Slice\nimport com.twitter.strato.client.Strato.{Client => StratoClient}\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.client.Scanner\nimport com.twitter.strato.thrift.ScroogeConvImplicits._\nimport com.twitter.wtf.candidate.thriftscala.CandidateSeq\nimport com.twitter.wtf.ml.thriftscala.CandidateFeatures\nimport com.twitter.wtf.real_time_interaction_graph.thriftscala.Interaction\nimport com.twitter.wtf.triangular_loop.thriftscala.{Candidates => TriangularLoopCandidates}\nimport com.twitter.strato.opcontext.Attribution._\n\nobject StratoClientModule extends TwitterModule {\n\n  // column paths\n  val CosineFollowPath = \"recommendations/similarity/similarUsersByFollowGraph.User\"\n  val CosineListPath = \"recommendations/similarity/similarUsersByListGraph.User\"\n  val CuratedCandidatesPath = \"onboarding/curatedAccounts\"\n  val CuratedFilteredAccountsPath = \"onboarding/filteredAccountsFromRecommendations\"\n  val PopUsersInPlacePath = \"onboarding/userrecs/popUsersInPlace\"\n  val ProfileSidebarBlacklistPath = \"recommendations/hermit/profile-sidebar-blacklist\"\n  val RealTimeInteractionsPath = \"hmli/realTimeInteractions\"\n  val SimsPath = \"recommendations/similarity/similarUsersBySims.User\"\n  val DBV2SimsPath = \"onboarding/userrecs/newSims.User\"\n  val TriangularLoopsPath = \"onboarding/userrecs/triangularLoops.User\"\n  val TwoHopRandomWalkPath = \"onboarding/userrecs/twoHopRandomWalk.User\"\n  val UserRecommendabilityPath = \"onboarding/userRecommendabilityWithLongKeys.User\"\n  val UTTAccountRecommendationsPath = \"onboarding/userrecs/utt_account_recommendations\"\n  val UttSeedAccountsRecommendationPath = \"onboarding/userrecs/utt_seed_accounts\"\n  val UserStatePath = \"onboarding/userState.User\"\n  val WTFPostNuxFeaturesPath = \"ml/featureStore/onboarding/wtfPostNuxFeatures.User\"\n  val ElectionCandidatesPath = \"onboarding/electionAccounts\"\n  val UserUserGraphPath = \"recommendations/userUserGraph\"\n  val WtfDissmissEventsPath = \"onboarding/wtfDismissEvents\"\n  val RelatableAccountsPath = \"onboarding/userrecs/relatableAccounts\"\n  val ExtendedNetworkCandidatesPath = \"search/account_search/extendedNetworkCandidatesMH\"\n  val LabeledNotificationPath = \"frigate/magicrecs/labeledPushRecsAggregated.User\"\n\n  @Provides\n  @Singleton\n  def stratoClient(serviceIdentifier: ServiceIdentifier): Client = {\n    val timeoutBudget = 500.milliseconds\n    StratoClient(\n      ThriftMux.client\n        .withRequestTimeout(timeoutBudget)\n        .withProtocolFactory(Protocols.binaryFactory(\n          stringLengthLimit = StringLengthLimit,\n          containerLengthLimit = ContainerLengthLimit)))\n      .withMutualTls(serviceIdentifier)\n      .build()\n  }\n\n  // add strato putters, fetchers, scanners below:\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.COSINE_FOLLOW_FETCHER)\n  def cosineFollowFetcher(stratoClient: Client): Fetcher[Long, Unit, HermitCandidates] =\n    stratoClient.fetcher[Long, Unit, HermitCandidates](CosineFollowPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.COSINE_LIST_FETCHER)\n  def cosineListFetcher(stratoClient: Client): Fetcher[Long, Unit, HermitCandidates] =\n    stratoClient.fetcher[Long, Unit, HermitCandidates](CosineListPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.CURATED_COMPETITOR_ACCOUNTS_FETCHER)\n  def curatedBlacklistedAccountsFetcher(stratoClient: Client): Fetcher[String, Unit, Seq[Long]] =\n    stratoClient.fetcher[String, Unit, Seq[Long]](CuratedFilteredAccountsPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.CURATED_CANDIDATES_FETCHER)\n  def curatedCandidatesFetcher(stratoClient: Client): Fetcher[String, Unit, Seq[Long]] =\n    stratoClient.fetcher[String, Unit, Seq[Long]](CuratedCandidatesPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.POP_USERS_IN_PLACE_FETCHER)\n  def popUsersInPlaceFetcher(stratoClient: Client): Fetcher[String, Unit, PopUsersInPlace] =\n    stratoClient.fetcher[String, Unit, PopUsersInPlace](PopUsersInPlacePath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.RELATABLE_ACCOUNTS_FETCHER)\n  def relatableAccountsFetcher(stratoClient: Client): Fetcher[String, Unit, RelatableAccounts] =\n    stratoClient.fetcher[String, Unit, RelatableAccounts](RelatableAccountsPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.PROFILE_SIDEBAR_BLACKLIST_SCANNER)\n  def profileSidebarBlacklistScanner(\n    stratoClient: Client\n  ): Scanner[(Long, Slice[Long]), Unit, (Long, Long), Unit] =\n    stratoClient.scanner[(Long, Slice[Long]), Unit, (Long, Long), Unit](ProfileSidebarBlacklistPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.REAL_TIME_INTERACTIONS_FETCHER)\n  def realTimeInteractionsFetcher(\n    stratoClient: Client\n  ): Fetcher[(Long, Long), Unit, Seq[Interaction]] =\n    stratoClient.fetcher[(Long, Long), Unit, Seq[Interaction]](RealTimeInteractionsPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.SIMS_FETCHER)\n  def simsFetcher(stratoClient: Client): Fetcher[Long, Unit, HermitCandidates] =\n    stratoClient.fetcher[Long, Unit, HermitCandidates](SimsPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.DBV2_SIMS_FETCHER)\n  def dbv2SimsFetcher(stratoClient: Client): Fetcher[Long, Unit, HermitCandidates] =\n    stratoClient.fetcher[Long, Unit, HermitCandidates](DBV2SimsPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.TRIANGULAR_LOOPS_FETCHER)\n  def triangularLoopsFetcher(stratoClient: Client): Fetcher[Long, Unit, TriangularLoopCandidates] =\n    stratoClient.fetcher[Long, Unit, TriangularLoopCandidates](TriangularLoopsPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.TWO_HOP_RANDOM_WALK_FETCHER)\n  def twoHopRandomWalkFetcher(stratoClient: Client): Fetcher[Long, Unit, CandidateSeq] =\n    stratoClient.fetcher[Long, Unit, CandidateSeq](TwoHopRandomWalkPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.USER_RECOMMENDABILITY_FETCHER)\n  def userRecommendabilityFetcher(\n    stratoClient: Client\n  ): Fetcher[Long, Unit, UserRecommendabilityFeatures] =\n    stratoClient.fetcher[Long, Unit, UserRecommendabilityFeatures](UserRecommendabilityPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.USER_STATE_FETCHER)\n  def userStateFetcher(stratoClient: Client): Fetcher[Long, Unit, CondensedUserState] =\n    stratoClient.fetcher[Long, Unit, CondensedUserState](UserStatePath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.UTT_ACCOUNT_RECOMMENDATIONS_FETCHER)\n  def uttAccountRecommendationsFetcher(\n    stratoClient: Client\n  ): Fetcher[UTTInterest, Unit, InterestBasedUserRecommendations] =\n    stratoClient.fetcher[UTTInterest, Unit, InterestBasedUserRecommendations](\n      UTTAccountRecommendationsPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.UTT_SEED_ACCOUNTS_FETCHER)\n  def uttSeedAccountRecommendationsFetcher(\n    stratoClient: Client\n  ): Fetcher[UTTInterest, Unit, InterestBasedUserRecommendations] =\n    stratoClient.fetcher[UTTInterest, Unit, InterestBasedUserRecommendations](\n      UttSeedAccountsRecommendationPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.ELECTION_CANDIDATES_FETCHER)\n  def electionCandidatesFetcher(stratoClient: Client): Fetcher[String, Unit, Seq[Long]] =\n    stratoClient.fetcher[String, Unit, Seq[Long]](ElectionCandidatesPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.USER_USER_GRAPH_FETCHER)\n  def userUserGraphFetcher(\n    stratoClient: Client\n  ): Fetcher[RecommendUserRequest, Unit, RecommendUserResponse] =\n    stratoClient.fetcher[RecommendUserRequest, Unit, RecommendUserResponse](UserUserGraphPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.POST_NUX_WTF_FEATURES_FETCHER)\n  def wtfPostNuxFeaturesFetcher(stratoClient: Client): Fetcher[Long, Unit, CandidateFeatures] = {\n    val attribution = ManhattanAppId(\"starbuck\", \"wtf_starbuck\")\n    stratoClient\n      .fetcher[Long, Unit, CandidateFeatures](WTFPostNuxFeaturesPath)\n      .withAttribution(attribution)\n  }\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.EXTENDED_NETWORK)\n  def extendedNetworkFetcher(\n    stratoClient: Client\n  ): Fetcher[ExtendedNetworkUserKey, Unit, ExtendedNetworkUserVal] = {\n    stratoClient\n      .fetcher[ExtendedNetworkUserKey, Unit, ExtendedNetworkUserVal](ExtendedNetworkCandidatesPath)\n  }\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.DISMISS_STORE_SCANNER)\n  def dismissStoreScanner(\n    stratoClient: Client\n  ): Scanner[\n    (Long, Slice[(Long, Long)]),\n    Unit,\n    (Long, (Long, Long)),\n    WhoToFollowDismissEventDetails\n  ] =\n    stratoClient.scanner[\n      (Long, Slice[(Long, Long)]), // PKEY: userId, LKEY: (-ts, candidateId)\n      Unit,\n      (Long, (Long, Long)),\n      WhoToFollowDismissEventDetails\n    ](WtfDissmissEventsPath)\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.LABELED_NOTIFICATION_FETCHER)\n  def labeledNotificationFetcher(\n    stratoClient: Client\n  ): Fetcher[Long, Unit, LatestEvents] = {\n    stratoClient\n      .fetcher[Long, Unit, LatestEvents](LabeledNotificationPath)\n  }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/user_state/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/cache\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/deciders\",\n        \"stitch/stitch-core\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"user-signal-service/thrift/src/main/thrift:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/user_state/UserStateClient.scala",
    "content": "package com.twitter.follow_recommendations.common.clients.user_state\n\nimport com.google.inject.name.Named\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.core_workflows.user_model.thriftscala.CondensedUserState\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.decider.Decider\nimport com.twitter.decider.RandomRecipient\nimport com.twitter.finagle.Memcached.Client\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.follow_recommendations.common.clients.cache.MemcacheClient\nimport com.twitter.follow_recommendations.common.clients.cache.ThriftEnumOptionBijection\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.follow_recommendations.configapi.deciders.DeciderKey\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.util.Duration\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport java.lang.{Long => JLong}\n\n@Singleton\nclass UserStateClient @Inject() (\n  @Named(GuiceNamedConstants.USER_STATE_FETCHER) userStateFetcher: Fetcher[\n    Long,\n    Unit,\n    CondensedUserState\n  ],\n  client: Client,\n  statsReceiver: StatsReceiver,\n  decider: Decider = Decider.False) {\n\n  private val stats: StatsReceiver = statsReceiver.scope(\"user_state_client\")\n\n  // client to memcache cluster\n  val bijection = new ThriftEnumOptionBijection[UserState](UserState.apply)\n  val memcacheClient = MemcacheClient[Option[UserState]](\n    client = client,\n    dest = \"/s/cache/follow_recos_service:twemcaches\",\n    valueBijection = bijection,\n    ttl = UserStateClient.CacheTTL,\n    statsReceiver = stats.scope(\"twemcache\")\n  )\n\n  def getUserState(userId: Long): Stitch[Option[UserState]] = {\n    val deciderKey: String = DeciderKey.EnableDistributedCaching.toString\n    val enableDistributedCaching: Boolean = decider.isAvailable(deciderKey, Some(RandomRecipient))\n    val userStateStitch: Stitch[Option[UserState]] = \n      enableDistributedCaching match {\n        // read from memcache\n        case true => memcacheClient.readThrough(\n          // add a key prefix to address cache key collisions\n          key = \"UserStateClient\" + userId.toString,\n          underlyingCall = () => fetchUserState(userId)\n        )\n        case false => fetchUserState(userId)\n      }\n    val userStateStitchWithTimeout: Stitch[Option[UserState]] = \n      userStateStitch\n        // set a 150ms timeout limit for user state fetches\n        .within(150.milliseconds)(DefaultTimer)\n        .rescue {\n          case e: Exception =>\n            stats.scope(\"rescued\").counter(e.getClass.getSimpleName).incr()\n            Stitch(None)\n        }\n    // profile the latency of stitch call and return the result\n    StatsUtil.profileStitch(\n      userStateStitchWithTimeout,\n      stats.scope(\"getUserState\")\n    )\n  }\n\n  def fetchUserState(userId: JLong): Stitch[Option[UserState]] = {\n    userStateFetcher.fetch(userId).map(_.v.flatMap(_.userState))\n  }\n}\n\nobject UserStateClient {\n  val CacheTTL: Duration = Duration.fromHours(6)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"util/util-core/src/main/scala/com/twitter/conversions\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants/CandidateAlgorithmTypeConstants.scala",
    "content": "package com.twitter.follow_recommendations.common.constants\n\nimport com.twitter.hermit.constants.AlgorithmFeedbackTokens.AlgorithmToFeedbackTokenMap\nimport com.twitter.hermit.model.Algorithm._\nimport com.twitter.follow_recommendations.common.models.AlgorithmType\n\nobject CandidateAlgorithmTypeConstants {\n\n  /**\n   * Each algorithm is based on one, or more, of the 4 types of information we have on users,\n   * described in [[AlgorithmType]]. Assignment of algorithms to these categories are based on\n   */\n  private val AlgorithmIdToType: Map[String, Set[AlgorithmType.Value]] = Map(\n    // Activity Algorithms:\n    AlgorithmToFeedbackTokenMap(NewFollowingSimilarUser).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(Sims).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(NewFollowingSimilarUserSalsa).toString -> Set(\n      AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(RecentEngagementNonDirectFollow).toString -> Set(\n      AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(RecentEngagementSimilarUser).toString -> Set(\n      AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(RecentEngagementSarusOcCur).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(RecentSearchBasedRec).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(TwistlyTweetAuthors).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(Follow2VecNearestNeighbors).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(EmailTweetClick).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(RepeatedProfileVisits).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(GoodTweetClickEngagements).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(TweetShareEngagements).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(TweetSharerToShareRecipientEngagements).toString -> Set(\n      AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(TweetAuthorToShareRecipientEngagements).toString -> Set(\n      AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(LinearRegressionFollow2VecNearestNeighbors).toString -> Set(\n      AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(NUXLOHistory).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(TrafficAttributionAccounts).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(RealGraphOonV2).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(MagicRecsRecentEngagements).toString -> Set(AlgorithmType.Activity),\n    AlgorithmToFeedbackTokenMap(NotificationEngagement).toString -> Set(AlgorithmType.Activity),\n    // Social Algorithms:\n    AlgorithmToFeedbackTokenMap(TwoHopRandomWalk).toString -> Set(AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(RealTimeMutualFollow).toString -> Set(AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(ForwardPhoneBook).toString -> Set(AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(ForwardEmailBook).toString -> Set(AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(NewFollowingNewFollowingExpansion).toString -> Set(\n      AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(NewFollowingSarusCoOccurSocialProof).toString -> Set(\n      AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(ReverseEmailBookIbis).toString -> Set(AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(ReversePhoneBook).toString -> Set(AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(StrongTiePredictionRec).toString -> Set(AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(StrongTiePredictionRecWithSocialProof).toString -> Set(\n      AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(OnlineStrongTiePredictionRec).toString -> Set(AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(OnlineStrongTiePredictionRecNoCaching).toString -> Set(\n      AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(TriangularLoop).toString -> Set(AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(StrongTiePredictionPmi).toString -> Set(AlgorithmType.Social),\n    AlgorithmToFeedbackTokenMap(OnlineStrongTiePredictionRAB).toString -> Set(AlgorithmType.Social),\n    // Geo Algorithms:\n    AlgorithmToFeedbackTokenMap(PopCountryBackFill).toString -> Set(AlgorithmType.Geo),\n    AlgorithmToFeedbackTokenMap(PopCountry).toString -> Set(AlgorithmType.Geo),\n    AlgorithmToFeedbackTokenMap(PopGeohash).toString -> Set(AlgorithmType.Geo),\n//    AlgorithmToFeedbackTokenMap(PopGeohashRealGraph).toString -> Set(AlgorithmType.Geo),\n    AlgorithmToFeedbackTokenMap(EngagedFollowerRatio).toString -> Set(AlgorithmType.Geo),\n    AlgorithmToFeedbackTokenMap(CrowdSearchAccounts).toString -> Set(AlgorithmType.Geo),\n    AlgorithmToFeedbackTokenMap(OrganicFollowAccounts).toString -> Set(AlgorithmType.Geo),\n    AlgorithmToFeedbackTokenMap(PopGeohashQualityFollow).toString -> Set(AlgorithmType.Geo),\n    AlgorithmToFeedbackTokenMap(PPMILocaleFollow).toString -> Set(AlgorithmType.Geo),\n    // Interest Algorithms:\n    AlgorithmToFeedbackTokenMap(TttInterest).toString -> Set(AlgorithmType.Interest),\n    AlgorithmToFeedbackTokenMap(UttInterestRelatedUsers).toString -> Set(AlgorithmType.Interest),\n    AlgorithmToFeedbackTokenMap(UttSeedAccounts).toString -> Set(AlgorithmType.Interest),\n    AlgorithmToFeedbackTokenMap(UttProducerExpansion).toString -> Set(AlgorithmType.Interest),\n    // Hybrid (more than one type) Algorithms:\n    AlgorithmToFeedbackTokenMap(UttProducerOfflineMbcgV1).toString -> Set(\n      AlgorithmType.Interest,\n      AlgorithmType.Geo),\n    AlgorithmToFeedbackTokenMap(CuratedAccounts).toString -> Set(\n      AlgorithmType.Interest,\n      AlgorithmType.Geo),\n    AlgorithmToFeedbackTokenMap(UserUserGraph).toString -> Set(\n      AlgorithmType.Social,\n      AlgorithmType.Activity),\n  )\n  def getAlgorithmTypes(algoId: String): Set[String] = {\n    AlgorithmIdToType.get(algoId).map(_.map(_.toString)).getOrElse(Set.empty)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants/GuiceNamedConstants.scala",
    "content": "package com.twitter.follow_recommendations.common.constants\n\nobject GuiceNamedConstants {\n  final val PRODUCER_SIDE_FEATURE_SWITCHES = \"PRODUCER_SIDE_FEATURE_SWITCHES\"\n  final val CLIENT_EVENT_LOGGER = \"CLIENT_EVENT_LOGGER\"\n  final val COSINE_FOLLOW_FETCHER = \"cosine_follow_fetcher\"\n  final val COSINE_LIST_FETCHER = \"cosine_list_fetcher\"\n  final val CURATED_CANDIDATES_FETCHER = \"curated_candidates_fetcher\"\n  final val CURATED_COMPETITOR_ACCOUNTS_FETCHER = \"curated_competitor_accounts_fetcher\"\n  final val POP_USERS_IN_PLACE_FETCHER = \"pop_users_in_place_fetcher\"\n  final val PROFILE_SIDEBAR_BLACKLIST_SCANNER = \"profile_sidebar_blacklist_scanner\"\n  final val REQUEST_LOGGER = \"REQUEST_LOGGER\"\n  final val FLOW_LOGGER = \"FLOW_LOGGER\"\n  final val REAL_TIME_INTERACTIONS_FETCHER = \"real_time_interactions_fetcher\"\n  final val SIMS_FETCHER = \"sims_fetcher\"\n  final val DBV2_SIMS_FETCHER = \"dbv2_sims_fetcher\"\n\n  final val TRIANGULAR_LOOPS_FETCHER = \"triangular_loops_fetcher\"\n  final val TWO_HOP_RANDOM_WALK_FETCHER = \"two_hop_random_walk_fetcher\"\n  final val USER_RECOMMENDABILITY_FETCHER = \"user_recommendability_fetcher\"\n  final val USER_STATE_FETCHER = \"user_state_fetcher\"\n  final val UTT_ACCOUNT_RECOMMENDATIONS_FETCHER = \"utt_account_recomendations_fetcher\"\n  final val UTT_SEED_ACCOUNTS_FETCHER = \"utt_seed_accounts_fetcher\"\n\n  final val ELECTION_CANDIDATES_FETCHER = \"election_candidates_fetcher\"\n  final val POST_NUX_WTF_FEATURES_FETCHER = \"post_nux_wtf_features_fetcher\"\n\n  final val USER_USER_GRAPH_FETCHER = \"user_user_graph_fetcher\"\n  final val DISMISS_STORE_SCANNER = \"dismiss_store_scanner\"\n  final val LABELED_NOTIFICATION_FETCHER = \"labeled_notification_scanner\"\n\n  final val STP_EP_SCORER = \"stp_ep_scorer\"\n  final val STP_DBV2_SCORER = \"stp_dbv2_scorer\"\n  final val STP_RAB_DBV2_SCORER = \"stp_rab_dbv2_scorer\"\n\n  final val EXTENDED_NETWORK = \"extended_network_candidates\"\n\n  // scoring client constants\n  final val WTF_PROD_DEEPBIRDV2_CLIENT = \"wtf_prod_deepbirdv2_client\"\n\n  // ann clients\n  final val RELATABLE_ACCOUNTS_FETCHER = \"relatable_accounts_fetcher\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants/ServiceConstants.scala",
    "content": "package com.twitter.follow_recommendations.common.constants\n\nimport com.twitter.conversions.StorageUnitOps._\n\nobject ServiceConstants {\n\n  /** thrift client response size limits\n   *  these were estimated using monitoring dashboard\n   *  3MB network usage per second / 25 rps ~ 120KB/req << 1MB\n   *  we give some buffer here in case some requests require more data than others\n   */\n  val StringLengthLimit: Long =\n    10.megabyte.inBytes\n  val ContainerLengthLimit: Long = 1.megabyte.inBytes\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/adapters/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":candidate-algorithm-adapter\",\n        \":client-context-adapter\",\n        \":post-nux-algorithm-adapter\",\n        \":pre-fetched-feature-adapter\",\n    ],\n)\n\ntarget(\n    name = \"common\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/onboarding/relevance/util/metadata\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n\nscala_library(\n    name = \"candidate-algorithm-adapter\",\n    sources = [\n        \"CandidateAlgorithmAdapter.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":common\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/constants\",\n    ],\n)\n\nscala_library(\n    name = \"client-context-adapter\",\n    sources = [\n        \"ClientContextAdapter.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":common\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n    ],\n)\n\nscala_library(\n    name = \"post-nux-algorithm-adapter\",\n    sources = [\n        \"PostNuxAlgorithmAdapter.scala\",\n    ],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":common\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/customer_journey:post-nux-algorithm-aggregate\",\n    ],\n)\n\nscala_library(\n    name = \"pre-fetched-feature-adapter\",\n    sources = [\n        \"PreFetchedFeatureAdapter.scala\",\n    ],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":common\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/adapters/CandidateAlgorithmAdapter.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.adapters\n\nimport com.twitter.follow_recommendations.common.models.UserCandidateSourceDetails\nimport com.twitter.hermit.constants.AlgorithmFeedbackTokens.AlgorithmToFeedbackTokenMap\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.hermit.model.Algorithm.Algorithm\nimport com.twitter.hermit.model.Algorithm.UttProducerOfflineMbcgV1\nimport com.twitter.hermit.model.Algorithm.UttProducerOnlineMbcgV1\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.Feature.SparseBinary\nimport com.twitter.ml.api.Feature.SparseContinuous\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.IRecordOneToOneAdapter\nimport com.twitter.ml.api.util.FDsl._\n\nobject CandidateAlgorithmAdapter\n    extends IRecordOneToOneAdapter[Option[UserCandidateSourceDetails]] {\n\n  val CANDIDATE_ALGORITHMS: SparseBinary = new SparseBinary(\"candidate.source.algorithm_ids\")\n  val CANDIDATE_SOURCE_SCORES: SparseContinuous =\n    new SparseContinuous(\"candidate.source.scores\")\n  val CANDIDATE_SOURCE_RANKS: SparseContinuous =\n    new SparseContinuous(\"candidate.source.ranks\")\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(\n    CANDIDATE_ALGORITHMS,\n    CANDIDATE_SOURCE_SCORES,\n    CANDIDATE_SOURCE_RANKS\n  )\n\n  /** list of candidate source remaps to avoid creating different features for experimental sources.\n   *  the LHS should contain the experimental source, and the RHS should contain the prod source.\n   */\n  def remapCandidateSource(a: Algorithm): Algorithm = a match {\n    case UttProducerOnlineMbcgV1 => UttProducerOfflineMbcgV1\n    case _ => a\n  }\n\n  // add the list of algorithm feedback tokens (integers) as a sparse binary feature\n  override def adaptToDataRecord(\n    userCandidateSourceDetailsOpt: Option[UserCandidateSourceDetails]\n  ): DataRecord = {\n    val dr = new DataRecord()\n    userCandidateSourceDetailsOpt.foreach { userCandidateSourceDetails =>\n      val scoreMap = for {\n        (source, scoreOpt) <- userCandidateSourceDetails.candidateSourceScores\n        score <- scoreOpt\n        algo <- Algorithm.withNameOpt(source.name)\n        algoId <- AlgorithmToFeedbackTokenMap.get(remapCandidateSource(algo))\n      } yield algoId.toString -> score\n      val rankMap = for {\n        (source, rank) <- userCandidateSourceDetails.candidateSourceRanks\n        algo <- Algorithm.withNameOpt(source.name)\n        algoId <- AlgorithmToFeedbackTokenMap.get(remapCandidateSource(algo))\n      } yield algoId.toString -> rank.toDouble\n\n      val algoIds = scoreMap.keys.toSet ++ rankMap.keys.toSet\n\n      // hydrate if not empty\n      if (rankMap.nonEmpty) {\n        dr.setFeatureValue(CANDIDATE_SOURCE_RANKS, rankMap)\n      }\n      if (scoreMap.nonEmpty) {\n        dr.setFeatureValue(CANDIDATE_SOURCE_SCORES, scoreMap)\n      }\n      if (algoIds.nonEmpty) {\n        dr.setFeatureValue(CANDIDATE_ALGORITHMS, algoIds)\n      }\n    }\n    dr\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/adapters/ClientContextAdapter.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.adapters\n\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.ml.api.Feature.Binary\nimport com.twitter.ml.api.Feature.Continuous\nimport com.twitter.ml.api.Feature.Discrete\nimport com.twitter.ml.api.Feature.Text\nimport com.twitter.ml.api.util.FDsl._\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.IRecordOneToOneAdapter\nimport com.twitter.onboarding.relevance.util.metadata.LanguageUtil\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.snowflake.id.SnowflakeId\n\nobject ClientContextAdapter extends IRecordOneToOneAdapter[(ClientContext, DisplayLocation)] {\n\n  // we name features with `user.account` for relatively static user-related features\n  val USER_COUNTRY: Text = new Text(\"user.account.country\")\n  val USER_LANGUAGE: Text = new Text(\"user.account.language\")\n  // we name features with `user.context` for more dynamic user-related features\n  val USER_LANGUAGE_PREFIX: Text = new Text(\"user.context.language_prefix\")\n  val USER_CLIENT: Discrete = new Discrete(\"user.context.client\")\n  val USER_AGE: Continuous = new Continuous(\"user.context.age\")\n  val USER_IS_RECENT: Binary = new Binary(\"user.is.recent\")\n  // we name features with `meta` for meta info about the WTF recommendation request\n  val META_DISPLAY_LOCATION: Text = new Text(\"meta.display_location\")\n  val META_POSITION: Discrete = new Discrete(\"meta.position\")\n  // This indicates whether a data point is from a random serving policy\n  val META_IS_RANDOM: Binary = new Binary(\"prediction.engine.is_random\")\n\n  val RECENT_WIN_IN_DAYS: Int = 30\n  val GOAL_META_POSITION: Long = 1L\n  val GOAL_META_IS_RANDOM: Boolean = true\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(\n    USER_COUNTRY,\n    USER_LANGUAGE,\n    USER_AGE,\n    USER_LANGUAGE_PREFIX,\n    USER_CLIENT,\n    USER_IS_RECENT,\n    META_DISPLAY_LOCATION,\n    META_POSITION,\n    META_IS_RANDOM\n  )\n\n  /**\n   * we only want to set the relevant fields iff they exist to eliminate redundant information\n   * we do some simple normalization on the language code\n   * we set META_POSITION to 1 always\n   * we set META_IS_RANDOM to true always to simulate a random serving distribution\n   * @param record ClientContext and DisplayLocation from the request\n   */\n  override def adaptToDataRecord(target: (ClientContext, DisplayLocation)): DataRecord = {\n    val dr = new DataRecord()\n    val cc = target._1\n    val dl = target._2\n    cc.countryCode.foreach(countryCode => dr.setFeatureValue(USER_COUNTRY, countryCode))\n    cc.languageCode.foreach(rawLanguageCode => {\n      val userLanguage = LanguageUtil.simplifyLanguage(rawLanguageCode)\n      val userLanguagePrefix = userLanguage.take(2)\n      dr.setFeatureValue(USER_LANGUAGE, userLanguage)\n      dr.setFeatureValue(USER_LANGUAGE_PREFIX, userLanguagePrefix)\n    })\n    cc.appId.foreach(appId => dr.setFeatureValue(USER_CLIENT, appId))\n    cc.userId.foreach(id =>\n      SnowflakeId.timeFromIdOpt(id).map { signupTime =>\n        val userAge = signupTime.untilNow.inMillis.toDouble\n        dr.setFeatureValue(USER_AGE, userAge)\n        dr.setFeatureValue(USER_IS_RECENT, signupTime.untilNow.inDays <= RECENT_WIN_IN_DAYS)\n        signupTime.untilNow.inDays\n      })\n    dr.setFeatureValue(META_DISPLAY_LOCATION, dl.toFsName)\n    dr.setFeatureValue(META_POSITION, GOAL_META_POSITION)\n    dr.setFeatureValue(META_IS_RANDOM, GOAL_META_IS_RANDOM)\n    dr\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/adapters/PostNuxAlgorithmAdapter.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.adapters\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.Feature.Continuous\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.IRecordOneToOneAdapter\nimport com.twitter.ml.api.util.FDsl._\nimport com.twitter.ml.featurestore.catalog.features.customer_journey.PostNuxAlgorithmFeatures\nimport com.twitter.ml.featurestore.catalog.features.customer_journey.PostNuxAlgorithmIdAggregateFeatureGroup\nimport com.twitter.ml.featurestore.catalog.features.customer_journey.PostNuxAlgorithmTypeAggregateFeatureGroup\nimport scala.collection.JavaConverters._\n\nobject PostNuxAlgorithmIdAdapter extends PostNuxAlgorithmAdapter {\n  override val PostNuxAlgorithmFeatureGroup: PostNuxAlgorithmFeatures =\n    PostNuxAlgorithmIdAggregateFeatureGroup\n\n  // To keep the length of feature names reasonable, we remove the prefix added by FeatureStore.\n  override val FeatureStorePrefix: String =\n    \"wtf_algorithm_id.customer_journey.post_nux_algorithm_id_aggregate_feature_group.\"\n}\n\nobject PostNuxAlgorithmTypeAdapter extends PostNuxAlgorithmAdapter {\n  override val PostNuxAlgorithmFeatureGroup: PostNuxAlgorithmFeatures =\n    PostNuxAlgorithmTypeAggregateFeatureGroup\n\n  // To keep the length of feature names reasonable, we remove the prefix added by FeatureStore.\n  override val FeatureStorePrefix: String =\n    \"wtf_algorithm_type.customer_journey.post_nux_algorithm_type_aggregate_feature_group.\"\n}\n\ntrait PostNuxAlgorithmAdapter extends IRecordOneToOneAdapter[DataRecord] {\n\n  val PostNuxAlgorithmFeatureGroup: PostNuxAlgorithmFeatures\n\n  // The string that is attached to the feature name when it is fetched from feature store.\n  val FeatureStorePrefix: String\n\n  /**\n   *\n   * This stores transformed aggregate features for PostNux algorithm aggregate features. The\n   * transformation here is log-ratio, where ratio is the raw value divided by # of impressions.\n   */\n  case class TransformedAlgorithmFeatures(\n    ratioLog: Continuous) {\n    def getFeatures: Seq[Continuous] = Seq(ratioLog)\n  }\n\n  private def applyFeatureStorePrefix(feature: Continuous) = new Continuous(\n    s\"$FeatureStorePrefix${feature.getFeatureName}\")\n\n  // The list of input features WITH the prefix assigned to them by FeatureStore.\n  lazy val allInputFeatures: Seq[Seq[Continuous]] = Seq(\n    PostNuxAlgorithmFeatureGroup.Aggregate7DayFeatures.map(applyFeatureStorePrefix),\n    PostNuxAlgorithmFeatureGroup.Aggregate30DayFeatures.map(applyFeatureStorePrefix)\n  )\n\n  // This is a list of the features WITHOUT the prefix assigned to them by FeatureStore.\n  lazy val outputBaseFeatureNames: Seq[Seq[Continuous]] = Seq(\n    PostNuxAlgorithmFeatureGroup.Aggregate7DayFeatures,\n    PostNuxAlgorithmFeatureGroup.Aggregate30DayFeatures\n  )\n\n  // We use backend impression to calculate ratio values.\n  lazy val ratioDenominators: Seq[Continuous] = Seq(\n    applyFeatureStorePrefix(PostNuxAlgorithmFeatureGroup.BackendImpressions7Days),\n    applyFeatureStorePrefix(PostNuxAlgorithmFeatureGroup.BackendImpressions30Days)\n  )\n\n  /**\n   * A mapping from an original feature's ID to the corresponding set of transformed features.\n   * This is used to compute the transformed features for each of the original ones.\n   */\n  private lazy val TransformedFeaturesMap: Map[Continuous, TransformedAlgorithmFeatures] =\n    outputBaseFeatureNames.flatten.map { feature =>\n      (\n        // The input feature would have the FeatureStore prefix attached to it.\n        new Continuous(s\"$FeatureStorePrefix${feature.getFeatureName}\"),\n        // We don't keep the FeatureStore prefix to keep the length of feature names reasonable.\n        TransformedAlgorithmFeatures(\n          new Continuous(s\"${feature.getFeatureName}-ratio-log\")\n        ))\n    }.toMap\n\n  /**\n   * Given a denominator, number of impressions, this function returns another function that adds\n   * transformed features (log1p and ratio) of an input feature to a DataRecord.\n   */\n  private def addTransformedFeaturesToDataRecordFunc(\n    originalDr: DataRecord,\n    numImpressions: Double,\n  ): (DataRecord, Continuous) => DataRecord = { (record: DataRecord, feature: Continuous) =>\n    {\n      Option(originalDr.getFeatureValue(feature)) foreach { featureValue =>\n        TransformedFeaturesMap.get(feature).foreach { transformedFeatures =>\n          record.setFeatureValue(\n            transformedFeatures.ratioLog,\n            // We don't use log1p here since the values are ratios and adding 1 to the _ratio_ would\n            // lead to logarithm of values between 1 and 2, essentially making all values the same.\n            math.log((featureValue + 1) / numImpressions)\n          )\n        }\n      }\n      record\n    }\n  }\n\n  /**\n   * @param record: The input record whose PostNuxAlgorithm aggregates are to be transformed.\n   * @return the input [[DataRecord]] with transformed aggregates added.\n   */\n  override def adaptToDataRecord(record: DataRecord): DataRecord = {\n    if (record.continuousFeatures == null) {\n      // There are no base features available, and hence no transformations.\n      record\n    } else {\n\n      /**\n       * The `foldLeft` below goes through pairs of (1) Feature groups, such as those calculated over\n       * 7 days or 30 days, and (2) the number of impressions for each of these groups, which is the\n       * denominator when ratio is calculated.\n       */\n      ratioDenominators\n        .zip(allInputFeatures).foldLeft( /* initial empty DataRecord */ record)(\n          (\n            /* DataRecord with transformed features up to here */ transformedRecord,\n            /* A tuple with the denominator (#impressions) and features to be transformed */ numImpressionsAndFeatures\n          ) => {\n            val (numImpressionsFeature, features) = numImpressionsAndFeatures\n            Option(record.getFeatureValue(numImpressionsFeature)) match {\n              case Some(numImpressions) if numImpressions > 0.0 =>\n                /**\n                 * With the number of impressions fixed, we generate a function that adds log-ratio\n                 * for each feature in the current [[DataRecord]]. The `foldLeft` goes through all\n                 * such features and applies that function while updating the kept DataRecord.\n                 */\n                features.foldLeft(transformedRecord)(\n                  addTransformedFeaturesToDataRecordFunc(record, numImpressions))\n              case _ =>\n                transformedRecord\n            }\n          })\n    }\n  }\n\n  def getFeatures: Seq[Feature[_]] = TransformedFeaturesMap.values.flatMap(_.getFeatures).toSeq\n\n  override def getFeatureContext: FeatureContext =\n    new FeatureContext()\n      .addFeatures(this.getFeatures.asJava)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/adapters/PreFetchedFeatureAdapter.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.adapters\n\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.ml.api.Feature.Continuous\nimport com.twitter.ml.api.util.FDsl._\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.IRecordOneToOneAdapter\nimport com.twitter.util.Time\n\n/**\n * This adapter mimics UserRecentWTFImpressionsAndFollowsAdapter (for user) and\n * RecentWTFImpressionsFeatureAdapter (for candidate) for extracting recent impression\n * and follow features. This adapter extracts user, candidate, and pair-wise features.\n */\nobject PreFetchedFeatureAdapter\n    extends IRecordOneToOneAdapter[\n      (HasPreFetchedFeature, CandidateUser)\n    ] {\n\n  // impression features\n  val USER_NUM_RECENT_IMPRESSIONS: Continuous = new Continuous(\n    \"user.prefetch.num_recent_impressions\"\n  )\n  val USER_LAST_IMPRESSION_DURATION: Continuous = new Continuous(\n    \"user.prefetch.last_impression_duration\"\n  )\n  val CANDIDATE_NUM_RECENT_IMPRESSIONS: Continuous = new Continuous(\n    \"user-candidate.prefetch.num_recent_impressions\"\n  )\n  val CANDIDATE_LAST_IMPRESSION_DURATION: Continuous = new Continuous(\n    \"user-candidate.prefetch.last_impression_duration\"\n  )\n  // follow features\n  val USER_NUM_RECENT_FOLLOWERS: Continuous = new Continuous(\n    \"user.prefetch.num_recent_followers\"\n  )\n  val USER_NUM_RECENT_FOLLOWED_BY: Continuous = new Continuous(\n    \"user.prefetch.num_recent_followed_by\"\n  )\n  val USER_NUM_RECENT_MUTUAL_FOLLOWS: Continuous = new Continuous(\n    \"user.prefetch.num_recent_mutual_follows\"\n  )\n  // impression + follow features\n  val USER_NUM_RECENT_FOLLOWED_IMPRESSIONS: Continuous = new Continuous(\n    \"user.prefetch.num_recent_followed_impression\"\n  )\n  val USER_LAST_FOLLOWED_IMPRESSION_DURATION: Continuous = new Continuous(\n    \"user.prefetch.last_followed_impression_duration\"\n  )\n\n  override def adaptToDataRecord(\n    record: (HasPreFetchedFeature, CandidateUser)\n  ): DataRecord = {\n    val (target, candidate) = record\n    val dr = new DataRecord()\n    val t = Time.now\n    // set impression features for user, optionally for candidate\n    dr.setFeatureValue(USER_NUM_RECENT_IMPRESSIONS, target.numWtfImpressions.toDouble)\n    dr.setFeatureValue(\n      USER_LAST_IMPRESSION_DURATION,\n      (t - target.latestImpressionTime).inMillis.toDouble)\n    target.getCandidateImpressionCounts(candidate.id).foreach { counts =>\n      dr.setFeatureValue(CANDIDATE_NUM_RECENT_IMPRESSIONS, counts.toDouble)\n    }\n    target.getCandidateLatestTime(candidate.id).foreach { latestTime: Time =>\n      dr.setFeatureValue(CANDIDATE_LAST_IMPRESSION_DURATION, (t - latestTime).inMillis.toDouble)\n    }\n    // set recent follow features for user\n    dr.setFeatureValue(USER_NUM_RECENT_FOLLOWERS, target.numRecentFollowedUserIds.toDouble)\n    dr.setFeatureValue(USER_NUM_RECENT_FOLLOWED_BY, target.numRecentFollowedByUserIds.toDouble)\n    dr.setFeatureValue(USER_NUM_RECENT_MUTUAL_FOLLOWS, target.numRecentMutualFollows.toDouble)\n    dr.setFeatureValue(USER_NUM_RECENT_FOLLOWED_IMPRESSIONS, target.numFollowedImpressions.toDouble)\n    dr.setFeatureValue(\n      USER_LAST_FOLLOWED_IMPRESSION_DURATION,\n      target.lastFollowedImpressionDurationMs.getOrElse(Long.MaxValue).toDouble)\n    dr\n  }\n  override def getFeatureContext: FeatureContext = new FeatureContext(\n    USER_NUM_RECENT_IMPRESSIONS,\n    USER_LAST_IMPRESSION_DURATION,\n    CANDIDATE_NUM_RECENT_IMPRESSIONS,\n    CANDIDATE_LAST_IMPRESSION_DURATION,\n    USER_NUM_RECENT_FOLLOWERS,\n    USER_NUM_RECENT_FOLLOWED_BY,\n    USER_NUM_RECENT_MUTUAL_FOLLOWS,\n    USER_NUM_RECENT_FOLLOWED_IMPRESSIONS,\n    USER_LAST_FOLLOWED_IMPRESSION_DURATION,\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/common/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/common/FeatureSource.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.common\n\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\ntrait FeatureSource {\n  def id: FeatureSourceId\n  def featureContext: FeatureContext\n  def hydrateFeatures(\n    target: HasClientContext\n      with HasPreFetchedFeature\n      with HasParams\n      with HasSimilarToContext\n      with HasDisplayLocation,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Map[CandidateUser, DataRecord]]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/common/FeatureSourceId.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.common\n\nsealed trait FeatureSourceId\n\nobject FeatureSourceId {\n  object CandidateAlgorithmSourceId extends FeatureSourceId\n  object ClientContextSourceId extends FeatureSourceId\n  object FeatureStoreSourceId extends FeatureSourceId\n  object FeatureStoreTimelinesAuthorSourceId extends FeatureSourceId\n  object FeatureStoreGizmoduckSourceId extends FeatureSourceId\n  object FeatureStoreUserMetricCountsSourceId extends FeatureSourceId\n  object FeatureStoreNotificationSourceId extends FeatureSourceId\n\n  object FeatureStorePrecomputedNotificationSourceId extends FeatureSourceId\n  object FeatureStorePostNuxAlgorithmSourceId extends FeatureSourceId\n  @deprecated object StratoFeatureHydrationSourceId extends FeatureSourceId\n  object PreFetchedFeatureSourceId extends FeatureSourceId\n  object UserScoringFeatureSourceId extends FeatureSourceId\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/common/HasPreFetchedFeature.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.common\n\nimport com.twitter.follow_recommendations.common.models.HasMutualFollowedUserIds\nimport com.twitter.follow_recommendations.common.models.HasWtfImpressions\nimport com.twitter.follow_recommendations.common.models.WtfImpression\nimport com.twitter.util.Time\n\ntrait HasPreFetchedFeature extends HasMutualFollowedUserIds with HasWtfImpressions {\n\n  lazy val followedImpressions: Seq[WtfImpression] = {\n    for {\n      wtfImprList <- wtfImpressions.toSeq\n      wtfImpr <- wtfImprList\n      if recentFollowedUserIds.exists(_.contains(wtfImpr.candidateId))\n    } yield wtfImpr\n  }\n\n  lazy val numFollowedImpressions: Int = followedImpressions.size\n\n  lazy val lastFollowedImpressionDurationMs: Option[Long] = {\n    if (followedImpressions.nonEmpty) {\n      Some((Time.now - followedImpressions.map(_.latestTime).max).inMillis)\n    } else None\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"escherbird/src/scala/com/twitter/escherbird/util/stitchcache\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/adapters\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/constants\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/datasets/core:socialgraph\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/datasets/core:usersource\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/datasets/onboarding:mc-user-counting\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/datasets/onboarding:user-wtf-algorithm-aggregate\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/datasets/onboarding:wtf-impression\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/datasets/onboarding:wtf-post-nux\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/datasets/onboarding:wtf-user-algorithm-aggregate\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/datasets/timelines:timelines-author-features\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/entities/core\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/entities/onboarding\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/core:socialgraph\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/core:user\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/interests_discovery:user-topic-relationships\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/magicrecs:non-mr-notif-summmaries\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/magicrecs:non-mr-notif-summmary-aggregates\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/magicrecs:nonmr-ntab-summaries\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/onboarding:mc-user-counting\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/onboarding:post-nux-offline\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/onboarding:post-nux-offline-edge\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/onboarding:ratio\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/onboarding:simcluster-user-interested-in-candidate-known-for\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/onboarding:user-wtf-algorithm-aggregate\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/onboarding:wtf-impression\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/onboarding:wtf-user-algorithm-aggregate\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/rux:user-resurrection\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/timelines:aggregate\",\n        \"src/scala/com/twitter/ml/featurestore/lib\",\n        \"src/scala/com/twitter/ml/featurestore/lib/dynamic\",\n        \"src/scala/com/twitter/ml/featurestore/lib/embedding\",\n        \"src/scala/com/twitter/ml/featurestore/lib/feature\",\n        \"src/scala/com/twitter/ml/featurestore/lib/online\",\n        \"src/scala/com/twitter/ml/featurestore/lib/params\",\n        \"src/scala/com/twitter/onboarding/relevance/adapters/features/featurestore\",\n        \"strato/config/columns/ml/featureStore:featureStore-strato-client\",\n        \"strato/config/columns/ml/featureStore/onboarding:onboarding-strato-client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/CandidateAlgorithmSource.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.google.inject.Inject\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.feature_hydration.adapters.CandidateAlgorithmAdapter\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSource\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSourceId\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\n/**\n * This source only takes features from the candidate's source,\n * which is all the information we have about the candidate pre-feature-hydration\n */\n\n@Provides\n@Singleton\nclass CandidateAlgorithmSource @Inject() (stats: StatsReceiver) extends FeatureSource {\n\n  override val id: FeatureSourceId = FeatureSourceId.CandidateAlgorithmSourceId\n\n  override val featureContext: FeatureContext = CandidateAlgorithmAdapter.getFeatureContext\n\n  override def hydrateFeatures(\n    t: HasClientContext\n      with HasPreFetchedFeature\n      with HasParams\n      with HasSimilarToContext\n      with HasDisplayLocation, // we don't use the target here\n    candidates: Seq[CandidateUser]\n  ): Stitch[Map[CandidateUser, DataRecord]] = {\n    val featureHydrationStats = stats.scope(\"candidate_alg_source\")\n    val hasSourceDetailsStat = featureHydrationStats.counter(\"has_source_details\")\n    val noSourceDetailsStat = featureHydrationStats.counter(\"no_source_details\")\n    val noSourceRankStat = featureHydrationStats.counter(\"no_source_rank\")\n    val hasSourceRankStat = featureHydrationStats.counter(\"has_source_rank\")\n    val noSourceScoreStat = featureHydrationStats.counter(\"no_source_score\")\n    val hasSourceScoreStat = featureHydrationStats.counter(\"has_source_score\")\n\n    val candidatesToAlgoMap = for {\n      candidate <- candidates\n    } yield {\n      if (candidate.userCandidateSourceDetails.nonEmpty) {\n        hasSourceDetailsStat.incr()\n        candidate.userCandidateSourceDetails.foreach { details =>\n          if (details.candidateSourceRanks.isEmpty) {\n            noSourceRankStat.incr()\n          } else {\n            hasSourceRankStat.incr()\n          }\n          if (details.candidateSourceScores.isEmpty) {\n            noSourceScoreStat.incr()\n          } else {\n            hasSourceScoreStat.incr()\n          }\n        }\n      } else {\n        noSourceDetailsStat.incr()\n      }\n      candidate -> CandidateAlgorithmAdapter.adaptToDataRecord(candidate.userCandidateSourceDetails)\n    }\n    Stitch.value(candidatesToAlgoMap.toMap)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/ClientContextSource.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.follow_recommendations.common.feature_hydration.adapters.ClientContextAdapter\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSource\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSourceId\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\n/**\n * This source only takes features from the request (e.g. client context, WTF display location)\n * No external calls are made.\n */\n@Provides\n@Singleton\nclass ClientContextSource() extends FeatureSource {\n\n  override val id: FeatureSourceId = FeatureSourceId.ClientContextSourceId\n\n  override val featureContext: FeatureContext = ClientContextAdapter.getFeatureContext\n\n  override def hydrateFeatures(\n    t: HasClientContext\n      with HasPreFetchedFeature\n      with HasParams\n      with HasSimilarToContext\n      with HasDisplayLocation,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Map[CandidateUser, DataRecord]] = {\n    Stitch.value(\n      candidates\n        .map(_ -> ((t.clientContext, t.displayLocation))).toMap.mapValues(\n          ClientContextAdapter.adaptToDataRecord))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/FeatureHydrationSourcesFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FeatureHydrationSourcesFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] = Seq(\n    FeatureStoreSourceParams.EnableAlgorithmAggregateFeatures,\n    FeatureStoreSourceParams.EnableAuthorTopicAggregateFeatures,\n    FeatureStoreSourceParams.EnableCandidateClientFeatures,\n    FeatureStoreSourceParams.EnableCandidatePrecomputedNotificationFeatures,\n    FeatureStoreSourceParams.EnableCandidateUserAuthorRealTimeAggregateFeatures,\n    FeatureStoreSourceParams.EnableCandidateUserFeatures,\n    FeatureStoreSourceParams.EnableCandidateUserResurrectionFeatures,\n    FeatureStoreSourceParams.EnableCandidateUserTimelinesAuthorAggregateFeatures,\n    FeatureStoreSourceParams.EnableSeparateClientForTimelinesAuthors,\n    FeatureStoreSourceParams.EnableSeparateClientForGizmoduck,\n    FeatureStoreSourceParams.EnableSeparateClientForMetricCenterUserCounting,\n    FeatureStoreSourceParams.EnableSeparateClientForNotifications,\n    FeatureStoreSourceParams.EnableSimilarToUserFeatures,\n    FeatureStoreSourceParams.EnableTargetUserFeatures,\n    FeatureStoreSourceParams.EnableTargetUserResurrectionFeatures,\n    FeatureStoreSourceParams.EnableTargetUserWtfImpressionFeatures,\n    FeatureStoreSourceParams.EnableTopicAggregateFeatures,\n    FeatureStoreSourceParams.EnableUserCandidateEdgeFeatures,\n    FeatureStoreSourceParams.EnableUserCandidateWtfImpressionCandidateFeatures,\n    FeatureStoreSourceParams.EnableUserClientFeatures,\n    FeatureStoreSourceParams.EnableUserTopicFeatures,\n    FeatureStoreSourceParams.EnableUserWtfAlgEdgeFeatures,\n  )\n\n  override val durationFSParams: Seq[FSBoundedParam[Duration] with HasDurationConversion] = Seq(\n    FeatureStoreSourceParams.GlobalFetchTimeout\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/FeatureHydrationSourcesFeatureSwitchKeys.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nobject FeatureHydrationSourcesFeatureSwitchKeys {\n  val EnableAlgorithmAggregateFeatures = \"feature_store_source_enable_algorithm_aggregate_features\"\n  val EnableAuthorTopicAggregateFeatures =\n    \"feature_store_source_enable_author_topic_aggregate_features\"\n  val EnableCandidateClientFeatures = \"feature_store_source_enable_candidate_client_features\"\n  val EnableCandidateNotificationFeatures =\n    \"feature_store_source_enable_candidate_notification_features\"\n  val EnableCandidatePrecomputedNotificationFeatures =\n    \"feature_store_source_enable_candidate_precomputed_notification_features\"\n  val EnableCandidateUserFeatures = \"feature_store_source_enable_candidate_user_features\"\n  val EnableCandidateUserAuthorRealTimeAggregateFeatures =\n    \"feature_store_source_enable_candidate_user_author_rta_features\"\n  val EnableCandidateUserResurrectionFeatures =\n    \"feature_store_source_enable_candidate_user_resurrection_features\"\n  val EnableCandidateUserTimelinesAuthorAggregateFeatures =\n    \"feature_store_source_enable_candidate_user_timelines_author_aggregate_features\"\n  val EnableSimilarToUserFeatures = \"feature_store_source_enable_similar_to_user_features\"\n  val EnableTargetUserFeatures = \"feature_store_source_enable_target_user_features\"\n  val EnableTargetUserUserAuthorUserStateRealTimeAggregatesFeature =\n    \"feature_store_source_enable_target_user_user_author_user_state_rta_features\"\n  val EnableTargetUserResurrectionFeatures =\n    \"feature_store_source_enable_target_user_resurrection_features\"\n  val EnableTargetUserWtfImpressionFeatures =\n    \"feature_store_source_enable_target_user_wtf_impression_features\"\n  val EnableTopicAggregateFeatures = \"feature_store_source_enable_topic_aggregate_features\"\n  val EnableUserCandidateEdgeFeatures = \"feature_store_source_enable_user_candidate_edge_features\"\n  val EnableUserCandidateWtfImpressionCandidateFeatures =\n    \"feature_store_source_enable_user_candidate_wtf_impression_features\"\n  val EnableUserClientFeatures = \"feature_store_source_enable_user_client_features\"\n  val EnableUserNotificationFeatures = \"feature_store_source_enable_user_notification_features\"\n  val EnableUserTopicFeatures = \"feature_store_source_enable_user_topic_features\"\n  val EnableUserWtfAlgEdgeFeatures = \"feature_store_source_enable_user_wtf_alg_edge_features\"\n  val FeatureHydrationTimeout = \"feature_store_source_hydration_timeout_in_millis\"\n  val UseSeparateClientForTimelinesAuthor =\n    \"feature_store_source_separate_client_for_timelines_author_data\"\n  val UseSeparateClientMetricCenterUserCounting =\n    \"feature_store_source_separate_client_for_mc_user_counting_data\"\n  val UseSeparateClientForNotifications = \"feature_store_source_separate_client_for_notifications\"\n  val UseSeparateClientForGizmoduck = \"feature_store_source_separate_client_for_gizmoduck\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/FeatureStoreFeatures.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.featurestore.catalog.entities.core.{Author => AuthorEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{AuthorTopic => AuthorTopicEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{CandidateUser => CandidateUserEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{Topic => TopicEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{User => UserEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{UserCandidate => UserCandidateEntity}\nimport com.twitter.ml.featurestore.catalog.entities.onboarding.UserWtfAlgorithmEntity\nimport com.twitter.ml.featurestore.catalog.entities.onboarding.{\n  WtfAlgorithm => WtfAlgorithmIdEntity\n}\nimport com.twitter.ml.featurestore.catalog.entities.onboarding.{\n  WtfAlgorithmType => WtfAlgorithmTypeEntity\n}\nimport com.twitter.ml.featurestore.catalog.features.core.UserClients.FullPrimaryClientVersion\nimport com.twitter.ml.featurestore.catalog.features.core.UserClients.NumClients\nimport com.twitter.ml.featurestore.catalog.features.core.UserClients.PrimaryClient\nimport com.twitter.ml.featurestore.catalog.features.core.UserClients.PrimaryClientVersion\nimport com.twitter.ml.featurestore.catalog.features.core.UserClients.PrimaryDeviceManufacturer\nimport com.twitter.ml.featurestore.catalog.features.core.UserClients.PrimaryMobileSdkVersion\nimport com.twitter.ml.featurestore.catalog.features.core.UserClients.SecondaryClient\nimport com.twitter.ml.featurestore.catalog.features.core.UserCounts.Favorites\nimport com.twitter.ml.featurestore.catalog.features.core.UserCounts.Followers\nimport com.twitter.ml.featurestore.catalog.features.core.UserCounts.Following\nimport com.twitter.ml.featurestore.catalog.features.core.UserCounts.Tweets\nimport com.twitter.ml.featurestore.catalog.features.customer_journey.PostNuxAlgorithmIdAggregateFeatureGroup\nimport com.twitter.ml.featurestore.catalog.features.customer_journey.PostNuxAlgorithmTypeAggregateFeatureGroup\nimport com.twitter.ml.featurestore.catalog.features.customer_journey.{Utils => FeatureGroupUtils}\nimport com.twitter.ml.featurestore.catalog.features.interests_discovery.UserTopicRelationships.FollowedTopics\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumFavorites\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumFavoritesReceived\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumFollowBacks\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumFollows\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumFollowsReceived\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumLoginDays\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumLoginTweetImpressions\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumMuteBacks\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumMuted\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumOriginalTweets\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumQualityFollowReceived\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumQuoteRetweets\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumQuoteRetweetsReceived\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumReplies\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumRepliesReceived\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumRetweets\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumRetweetsReceived\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumSpamBlocked\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumSpamBlockedBacks\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumTweetImpressions\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumTweets\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumUnfollowBacks\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumUnfollows\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumUserActiveMinutes\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumWasMutualFollowed\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumWasMutualUnfollowed\nimport com.twitter.ml.featurestore.catalog.features.onboarding.MetricCenterUserCounts.NumWasUnfollowed\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOffline.Country\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOffline.FollowersOverFollowingRatio\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOffline.Language\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOffline.MutualFollowsOverFollowersRatio\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOffline.MutualFollowsOverFollowingRatio\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOffline.NumFollowers\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOffline.NumFollowings\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOffline.NumMutualFollows\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOffline.TweepCred\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOffline.UserState\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOfflineEdge.HaveSameCountry\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOfflineEdge.HaveSameLanguage\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOfflineEdge.HaveSameUserState\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOfflineEdge.NumFollowersGap\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOfflineEdge.NumFollowingsGap\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOfflineEdge.NumMutualFollowsGap\nimport com.twitter.ml.featurestore.catalog.features.onboarding.PostNuxOfflineEdge.TweepCredGap\nimport com.twitter.ml.featurestore.catalog.features.onboarding.Ratio.FollowersFollowings\nimport com.twitter.ml.featurestore.catalog.features.onboarding.Ratio.MutualFollowsFollowing\nimport com.twitter.ml.featurestore.catalog.features.onboarding.SimclusterUserInterestedInCandidateKnownFor.HasIntersection\nimport com.twitter.ml.featurestore.catalog.features.onboarding.SimclusterUserInterestedInCandidateKnownFor.IntersectionCandidateKnownForScore\nimport com.twitter.ml.featurestore.catalog.features.onboarding.SimclusterUserInterestedInCandidateKnownFor.IntersectionClusterIds\nimport com.twitter.ml.featurestore.catalog.features.onboarding.SimclusterUserInterestedInCandidateKnownFor.IntersectionUserFavCandidateKnownForScore\nimport com.twitter.ml.featurestore.catalog.features.onboarding.SimclusterUserInterestedInCandidateKnownFor.IntersectionUserFavScore\nimport com.twitter.ml.featurestore.catalog.features.onboarding.SimclusterUserInterestedInCandidateKnownFor.IntersectionUserFollowCandidateKnownForScore\nimport com.twitter.ml.featurestore.catalog.features.onboarding.SimclusterUserInterestedInCandidateKnownFor.IntersectionUserFollowScore\nimport com.twitter.ml.featurestore.catalog.features.onboarding.UserWtfAlgorithmAggregate\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WhoToFollowImpression.HomeTimelineWtfCandidateCounts\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WhoToFollowImpression.HomeTimelineWtfCandidateImpressionCounts\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WhoToFollowImpression.HomeTimelineWtfCandidateImpressionLatestTimestamp\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WhoToFollowImpression.HomeTimelineWtfLatestTimestamp\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.FollowRate\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.Follows\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.FollowsTweetFavRate\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.FollowsTweetReplies\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.FollowsTweetReplyRate\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.FollowsTweetRetweetRate\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.FollowsTweetRetweets\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.FollowsWithTweetFavs\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.FollowsWithTweetImpressions\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.HasAnyEngagements\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.HasForwardEngagements\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.HasReverseEngagements\nimport com.twitter.ml.featurestore.catalog.features.onboarding.WtfUserAlgorithmAggregate.Impressions\nimport com.twitter.ml.featurestore.catalog.features.rux.UserResurrection.DaysSinceRecentResurrection\nimport com.twitter.ml.featurestore.catalog.features.timelines.AuthorTopicAggregates\nimport com.twitter.ml.featurestore.catalog.features.timelines.EngagementsReceivedByAuthorRealTimeAggregates\nimport com.twitter.ml.featurestore.catalog.features.timelines.NegativeEngagementsReceivedByAuthorRealTimeAggregates\nimport com.twitter.ml.featurestore.catalog.features.timelines.OriginalAuthorAggregates\nimport com.twitter.ml.featurestore.catalog.features.timelines.TopicEngagementRealTimeAggregates\nimport com.twitter.ml.featurestore.catalog.features.timelines.TopicEngagementUserStateRealTimeAggregates\nimport com.twitter.ml.featurestore.catalog.features.timelines.TopicNegativeEngagementUserStateRealTimeAggregates\nimport com.twitter.ml.featurestore.catalog.features.timelines.UserEngagementAuthorUserStateRealTimeAggregates\nimport com.twitter.ml.featurestore.catalog.features.timelines.UserNegativeEngagementAuthorUserStateRealTimeAggregates\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.ml.featurestore.lib.feature.BoundFeature\nimport com.twitter.ml.featurestore.lib.feature.Feature\n\nobject FeatureStoreFeatures {\n  import FeatureStoreRawFeatures._\n  ///////////////////////////// Target user features ////////////////////////\n  val targetUserFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    (userKeyedFeatures ++ userAlgorithmAggregateFeatures).map(_.bind(UserEntity))\n\n  val targetUserResurrectionFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    userResurrectionFeatures.map(_.bind(UserEntity))\n  val targetUserWtfImpressionFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    wtfImpressionUserFeatures.map(_.bind(UserEntity))\n  val targetUserUserAuthorUserStateRealTimeAggregatesFeature: Set[BoundFeature[_ <: EntityId, _]] =\n    userAuthorUserStateRealTimeAggregatesFeature.map(_.bind(UserEntity))\n\n  val targetUserStatusFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    userStatusFeatures.map(_.bind(UserEntity).logarithm1p)\n  val targetUserMetricCountFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    mcFeatures.map(_.bind(UserEntity).logarithm1p)\n\n  val targetUserClientFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    clientFeatures.map(_.bind(UserEntity))\n\n  ///////////////////////////// Candidate user features ////////////////////////\n  val candidateUserFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    userKeyedFeatures.map(_.bind(CandidateUserEntity))\n  val candidateUserAuthorRealTimeAggregateFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    authorAggregateFeatures.map(_.bind(CandidateUserEntity))\n  val candidateUserResurrectionFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    userResurrectionFeatures.map(_.bind(CandidateUserEntity))\n\n  val candidateUserStatusFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    userStatusFeatures.map(_.bind(CandidateUserEntity).logarithm1p)\n  val candidateUserTimelinesAuthorAggregateFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    Set(timelinesAuthorAggregateFeatures.bind(CandidateUserEntity))\n  val candidateUserMetricCountFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    mcFeatures.map(_.bind(CandidateUserEntity).logarithm1p)\n\n  val candidateUserClientFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    clientFeatures.map(_.bind(CandidateUserEntity))\n\n  val similarToUserFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    (userKeyedFeatures ++ authorAggregateFeatures).map(_.bind(AuthorEntity))\n\n  val similarToUserStatusFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    userStatusFeatures.map(_.bind(AuthorEntity).logarithm1p)\n  val similarToUserTimelinesAuthorAggregateFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    Set(timelinesAuthorAggregateFeatures.bind(AuthorEntity))\n  val similarToUserMetricCountFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    mcFeatures.map(_.bind(AuthorEntity).logarithm1p)\n\n  val userCandidateEdgeFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    (simclusterUVIntersectionFeatures ++ userCandidatePostNuxEdgeFeatures).map(\n      _.bind(UserCandidateEntity))\n  val userCandidateWtfImpressionCandidateFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    wtfImpressionCandidateFeatures.map(_.bind(UserCandidateEntity))\n\n  /**\n   * Aggregate features based on candidate source algorithms.\n   */\n  val postNuxAlgorithmIdAggregateFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    Set(PostNuxAlgorithmIdAggregateFeatureGroup.FeaturesAsDataRecord)\n      .map(_.bind(WtfAlgorithmIdEntity))\n\n  /**\n   * Aggregate features based on candidate source algorithm types. There are 4 at the moment:\n   * Geo, Social, Activity and Interest.\n   */\n  val postNuxAlgorithmTypeAggregateFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    Set(PostNuxAlgorithmTypeAggregateFeatureGroup.FeaturesAsDataRecord)\n      .map(_.bind(WtfAlgorithmTypeEntity))\n\n  // user wtf-Algorithm features\n  val userWtfAlgorithmEdgeFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    FeatureGroupUtils.getTimelinesAggregationFrameworkCombinedFeatures(\n      UserWtfAlgorithmAggregate,\n      UserWtfAlgorithmEntity,\n      FeatureGroupUtils.getMaxSumAvgAggregate(UserWtfAlgorithmAggregate)\n    )\n\n  /**\n   * We have to add the max/sum/avg-aggregated features to the set of all features so that we can\n   * register them using FRS's [[FrsFeatureJsonExporter]].\n   *\n   * Any additional such aggregated features that are included in [[FeatureStoreSource]] client\n   * should be registered here as well.\n   */\n  val maxSumAvgAggregatedFeatureContext: FeatureContext = new FeatureContext()\n    .addFeatures(\n      UserWtfAlgorithmAggregate.getSecondaryAggregatedFeatureContext\n    )\n\n  // topic features\n  val topicAggregateFeatures: Set[BoundFeature[_ <: EntityId, _]] = Set(\n    TopicEngagementRealTimeAggregates.FeaturesAsDataRecord,\n    TopicNegativeEngagementUserStateRealTimeAggregates.FeaturesAsDataRecord,\n    TopicEngagementUserStateRealTimeAggregates.FeaturesAsDataRecord\n  ).map(_.bind(TopicEntity))\n  val userTopicFeatures: Set[BoundFeature[_ <: EntityId, _]] = Set(FollowedTopics.bind(UserEntity))\n  val authorTopicFeatures: Set[BoundFeature[_ <: EntityId, _]] = Set(\n    AuthorTopicAggregates.FeaturesAsDataRecord.bind(AuthorTopicEntity))\n  val topicFeatures = topicAggregateFeatures ++ userTopicFeatures ++ authorTopicFeatures\n\n}\n\nobject FeatureStoreRawFeatures {\n  val mcFeatures = Set(\n    NumTweets,\n    NumRetweets,\n    NumOriginalTweets,\n    NumRetweetsReceived,\n    NumFavoritesReceived,\n    NumRepliesReceived,\n    NumQuoteRetweetsReceived,\n    NumFollowsReceived,\n    NumFollowBacks,\n    NumFollows,\n    NumUnfollows,\n    NumUnfollowBacks,\n    NumQualityFollowReceived,\n    NumQuoteRetweets,\n    NumFavorites,\n    NumReplies,\n    NumLoginTweetImpressions,\n    NumTweetImpressions,\n    NumLoginDays,\n    NumUserActiveMinutes,\n    NumMuted,\n    NumSpamBlocked,\n    NumMuteBacks,\n    NumSpamBlockedBacks,\n    NumWasMutualFollowed,\n    NumWasMutualUnfollowed,\n    NumWasUnfollowed\n  )\n  // based off usersource, and each feature represents the cumulative 'sent' counts\n  val userStatusFeatures = Set(\n    Favorites,\n    Followers,\n    Following,\n    Tweets\n  )\n  // ratio features created from combining other features\n  val userRatioFeatures = Set(MutualFollowsFollowing, FollowersFollowings)\n  // features related to user login history\n  val userResurrectionFeatures: Set[Feature[UserId, Int]] = Set(\n    DaysSinceRecentResurrection\n  )\n\n  // real-time  aggregate features borrowed from timelines\n  val authorAggregateFeatures = Set(\n    EngagementsReceivedByAuthorRealTimeAggregates.FeaturesAsDataRecord,\n    NegativeEngagementsReceivedByAuthorRealTimeAggregates.FeaturesAsDataRecord,\n  )\n\n  val timelinesAuthorAggregateFeatures = OriginalAuthorAggregates.FeaturesAsDataRecord\n\n  val userAuthorUserStateRealTimeAggregatesFeature: Set[Feature[UserId, DataRecord]] = Set(\n    UserEngagementAuthorUserStateRealTimeAggregates.FeaturesAsDataRecord,\n    UserNegativeEngagementAuthorUserStateRealTimeAggregates.FeaturesAsDataRecord\n  )\n  // post nux per-user offline features\n  val userOfflineFeatures = Set(\n    NumFollowings,\n    NumFollowers,\n    NumMutualFollows,\n    TweepCred,\n    UserState,\n    Language,\n    Country,\n    MutualFollowsOverFollowingRatio,\n    MutualFollowsOverFollowersRatio,\n    FollowersOverFollowingRatio,\n  )\n  // matched post nux offline features between user and candidate\n  val userCandidatePostNuxEdgeFeatures = Set(\n    HaveSameUserState,\n    HaveSameLanguage,\n    HaveSameCountry,\n    NumFollowingsGap,\n    NumFollowersGap,\n    NumMutualFollowsGap,\n    TweepCredGap,\n  )\n  // user algorithm aggregate features\n  val userAlgorithmAggregateFeatures = Set(\n    Impressions,\n    Follows,\n    FollowRate,\n    FollowsWithTweetImpressions,\n    FollowsWithTweetFavs,\n    FollowsTweetFavRate,\n    FollowsTweetReplies,\n    FollowsTweetReplyRate,\n    FollowsTweetRetweets,\n    FollowsTweetRetweetRate,\n    HasForwardEngagements,\n    HasReverseEngagements,\n    HasAnyEngagements,\n  )\n  val userKeyedFeatures = userRatioFeatures ++ userOfflineFeatures\n  val wtfImpressionUserFeatures =\n    Set(HomeTimelineWtfCandidateCounts, HomeTimelineWtfLatestTimestamp)\n  val wtfImpressionCandidateFeatures =\n    Set(HomeTimelineWtfCandidateImpressionCounts, HomeTimelineWtfCandidateImpressionLatestTimestamp)\n  val simclusterUVIntersectionFeatures = Set(\n    IntersectionClusterIds,\n    HasIntersection,\n    IntersectionUserFollowScore,\n    IntersectionUserFavScore,\n    IntersectionCandidateKnownForScore,\n    IntersectionUserFollowCandidateKnownForScore,\n    IntersectionUserFavCandidateKnownForScore\n  )\n\n  // Client features\n  val clientFeatures = Set(\n    NumClients,\n    PrimaryClient,\n    PrimaryClientVersion,\n    FullPrimaryClientVersion,\n    PrimaryDeviceManufacturer,\n    PrimaryMobileSdkVersion,\n    SecondaryClient\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/FeatureStoreGizmoduckSource.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.github.benmanes.caffeine.cache.Caffeine\nimport com.google.inject.Inject\nimport com.twitter.finagle.TimeoutException\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSource\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSourceId\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.Utils.adaptAdditionalFeaturesToDataRecord\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.Utils.randomizedTTL\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.IRecordOneToOneAdapter\nimport com.twitter.ml.featurestore.catalog.datasets.core.UsersourceEntityDataset\nimport com.twitter.ml.featurestore.catalog.entities.core.{Author => AuthorEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{AuthorTopic => AuthorTopicEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{CandidateUser => CandidateUserEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{User => UserEntity}\nimport com.twitter.ml.featurestore.lib.EdgeEntityId\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.TopicId\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.ml.featurestore.lib.data.PredictionRecord\nimport com.twitter.ml.featurestore.lib.data.PredictionRecordAdapter\nimport com.twitter.ml.featurestore.lib.dataset.DatasetId\nimport com.twitter.ml.featurestore.lib.dataset.online.Hydrator.HydrationResponse\nimport com.twitter.ml.featurestore.lib.dataset.online.OnlineAccessDataset\nimport com.twitter.ml.featurestore.lib.dynamic.ClientConfig\nimport com.twitter.ml.featurestore.lib.dynamic.DynamicFeatureStoreClient\nimport com.twitter.ml.featurestore.lib.dynamic.DynamicHydrationConfig\nimport com.twitter.ml.featurestore.lib.dynamic.FeatureStoreParamsConfig\nimport com.twitter.ml.featurestore.lib.dynamic.GatedFeatures\nimport com.twitter.ml.featurestore.lib.feature.BoundFeature\nimport com.twitter.ml.featurestore.lib.feature.BoundFeatureSet\nimport com.twitter.ml.featurestore.lib.online.DatasetValuesCache\nimport com.twitter.ml.featurestore.lib.online.FeatureStoreRequest\nimport com.twitter.ml.featurestore.lib.online.OnlineFeatureGenerationStats\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport java.util.concurrent.TimeUnit\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\n\nclass FeatureStoreGizmoduckSource @Inject() (\n  serviceIdentifier: ServiceIdentifier,\n  stats: StatsReceiver)\n    extends FeatureSource {\n  import FeatureStoreGizmoduckSource._\n\n  val backupSourceStats = stats.scope(\"feature_store_hydration_gizmoduck\")\n  val adapterStats = backupSourceStats.scope(\"adapters\")\n  override def id: FeatureSourceId = FeatureSourceId.FeatureStoreGizmoduckSourceId\n  override def featureContext: FeatureContext = getFeatureContext\n\n  val clientConfig: ClientConfig[HasParams] = ClientConfig(\n    dynamicHydrationConfig = dynamicHydrationConfig,\n    featureStoreParamsConfig =\n      FeatureStoreParamsConfig(FeatureStoreParameters.featureStoreParams, Map.empty),\n    /**\n     * The smaller one between `timeoutProvider` and `FeatureStoreSourceParams.GlobalFetchTimeout`\n     * used below takes effect.\n     */\n    timeoutProvider = Function.const(800.millis),\n    serviceIdentifier = serviceIdentifier\n  )\n\n  private val datasetsToCache = Set(\n    UsersourceEntityDataset\n  ).asInstanceOf[Set[OnlineAccessDataset[_ <: EntityId, _]]]\n\n  private val datasetValuesCache: DatasetValuesCache =\n    DatasetValuesCache(\n      Caffeine\n        .newBuilder()\n        .expireAfterWrite(randomizedTTL(12.hours.inSeconds), TimeUnit.SECONDS)\n        .maximumSize(DefaultCacheMaxKeys)\n        .build[(_ <: EntityId, DatasetId), Stitch[HydrationResponse[_]]]\n        .asMap,\n      datasetsToCache,\n      DatasetCacheScope\n    )\n\n  private val dynamicFeatureStoreClient = DynamicFeatureStoreClient(\n    clientConfig,\n    backupSourceStats,\n    Set(datasetValuesCache)\n  )\n\n  private val adapter: IRecordOneToOneAdapter[PredictionRecord] =\n    PredictionRecordAdapter.oneToOne(\n      BoundFeatureSet(allFeatures),\n      OnlineFeatureGenerationStats(backupSourceStats)\n    )\n\n  override def hydrateFeatures(\n    target: HasClientContext\n      with HasPreFetchedFeature\n      with HasParams\n      with HasSimilarToContext\n      with HasDisplayLocation,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Map[CandidateUser, DataRecord]] = {\n    target.getOptionalUserId\n      .map { targetUserId =>\n        val featureRequests = candidates.map { candidate =>\n          val userEntityId = UserEntity.withId(UserId(targetUserId))\n          val candidateEntityId = CandidateUserEntity.withId(UserId(candidate.id))\n          val similarToUserId = target.similarToUserIds.map(id => AuthorEntity.withId(UserId(id)))\n          val topicProof = candidate.reason.flatMap(_.accountProof.flatMap(_.topicProof))\n          val authorTopicEntity = if (topicProof.isDefined) {\n            backupSourceStats.counter(\"candidates_with_topic_proof\").incr()\n            Set(\n              AuthorTopicEntity.withId(\n                EdgeEntityId(UserId(candidate.id), TopicId(topicProof.get.topicId))))\n          } else Nil\n\n          val entities =\n            Seq(userEntityId, candidateEntityId) ++ similarToUserId ++ authorTopicEntity\n          FeatureStoreRequest(entities)\n        }\n\n        val predictionRecordsFut = dynamicFeatureStoreClient(featureRequests, target)\n        val candidateFeatureMap = predictionRecordsFut.map { predictionRecords =>\n          // we can zip predictionRecords with candidates as the order is preserved in the client\n          candidates\n            .zip(predictionRecords).map {\n              case (candidate, predictionRecord) =>\n                candidate -> adaptAdditionalFeaturesToDataRecord(\n                  adapter.adaptToDataRecord(predictionRecord),\n                  adapterStats,\n                  FeatureStoreSource.featureAdapters)\n            }.toMap\n        }\n        Stitch\n          .callFuture(candidateFeatureMap)\n          .within(target.params(FeatureStoreSourceParams.GlobalFetchTimeout))(\n            com.twitter.finagle.util.DefaultTimer)\n          .rescue {\n            case _: TimeoutException =>\n              Stitch.value(Map.empty[CandidateUser, DataRecord])\n          }\n      }.getOrElse(Stitch.value(Map.empty[CandidateUser, DataRecord]))\n  }\n}\n\nobject FeatureStoreGizmoduckSource {\n  private val DatasetCacheScope = \"feature_store_local_cache_gizmoduck\"\n  private val DefaultCacheMaxKeys = 20000\n\n  val allFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    FeatureStoreFeatures.candidateUserStatusFeatures ++\n      FeatureStoreFeatures.similarToUserStatusFeatures ++\n      FeatureStoreFeatures.targetUserStatusFeatures\n\n  val getFeatureContext: FeatureContext =\n    BoundFeatureSet(allFeatures).toFeatureContext\n\n  val dynamicHydrationConfig: DynamicHydrationConfig[HasParams] =\n    DynamicHydrationConfig(\n      Set(\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(FeatureStoreFeatures.targetUserStatusFeatures),\n          gate = HasParams\n            .paramGate(FeatureStoreSourceParams.EnableSeparateClientForGizmoduck) &\n            HasParams.paramGate(FeatureStoreSourceParams.EnableTargetUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(FeatureStoreFeatures.candidateUserStatusFeatures),\n          gate =\n            HasParams\n              .paramGate(FeatureStoreSourceParams.EnableSeparateClientForGizmoduck) &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableCandidateUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(FeatureStoreFeatures.similarToUserStatusFeatures),\n          gate =\n            HasParams\n              .paramGate(FeatureStoreSourceParams.EnableSeparateClientForGizmoduck) &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableSimilarToUserFeatures)\n        ),\n      ))\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/FeatureStoreParameters.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.ml.featurestore.catalog.datasets.core.UserMobileSdkDataset\nimport com.twitter.ml.featurestore.catalog.datasets.core.UsersourceEntityDataset\nimport com.twitter.ml.featurestore.catalog.datasets.customer_journey.PostNuxAlgorithmIdAggregateDataset\nimport com.twitter.ml.featurestore.catalog.datasets.customer_journey.PostNuxAlgorithmTypeAggregateDataset\nimport com.twitter.ml.featurestore.catalog.datasets.magicrecs.NotificationSummariesEntityDataset\nimport com.twitter.ml.featurestore.catalog.datasets.onboarding.MetricCenterUserCountingFeaturesDataset\nimport com.twitter.ml.featurestore.catalog.datasets.onboarding.UserWtfAlgorithmAggregateFeaturesDataset\nimport com.twitter.ml.featurestore.catalog.datasets.onboarding.WhoToFollowPostNuxFeaturesDataset\nimport com.twitter.ml.featurestore.catalog.datasets.rux.UserRecentReactivationTimeDataset\nimport com.twitter.ml.featurestore.catalog.datasets.timelines.AuthorFeaturesEntityDataset\nimport com.twitter.ml.featurestore.lib.dataset.DatasetParams\nimport com.twitter.ml.featurestore.lib.dataset.online.BatchingPolicy\nimport com.twitter.ml.featurestore.lib.params.FeatureStoreParams\nimport com.twitter.strato.opcontext.Attribution.ManhattanAppId\nimport com.twitter.strato.opcontext.ServeWithin\n\nobject FeatureStoreParameters {\n\n  private val FeatureServiceBatchSize = 100\n\n  val featureStoreParams = FeatureStoreParams(\n    global = DatasetParams(\n      serveWithin = Some(ServeWithin(duration = 240.millis, roundTripAllowance = None)),\n      attributions = Seq(\n        ManhattanAppId(\"omega\", \"wtf_impression_store\"),\n        ManhattanAppId(\"athena\", \"wtf_athena\"),\n        ManhattanAppId(\"starbuck\", \"wtf_starbuck\"),\n        ManhattanAppId(\"apollo\", \"wtf_apollo\")\n      ),\n      batchingPolicy = Some(BatchingPolicy.Isolated(FeatureServiceBatchSize))\n    ),\n    perDataset = Map(\n      MetricCenterUserCountingFeaturesDataset.id ->\n        DatasetParams(\n          stratoSuffix = Some(\"onboarding\"),\n          batchingPolicy = Some(BatchingPolicy.Isolated(200))\n        ),\n      UsersourceEntityDataset.id ->\n        DatasetParams(\n          stratoSuffix = Some(\"onboarding\")\n        ),\n      WhoToFollowPostNuxFeaturesDataset.id ->\n        DatasetParams(\n          stratoSuffix = Some(\"onboarding\"),\n          batchingPolicy = Some(BatchingPolicy.Isolated(200))\n        ),\n      AuthorFeaturesEntityDataset.id ->\n        DatasetParams(\n          stratoSuffix = Some(\"onboarding\"),\n          batchingPolicy = Some(BatchingPolicy.Isolated(10))\n        ),\n      UserRecentReactivationTimeDataset.id -> DatasetParams(\n        stratoSuffix =\n          None // removed due to low hit rate. we should use a negative cache in the future\n      ),\n      UserWtfAlgorithmAggregateFeaturesDataset.id -> DatasetParams(\n        stratoSuffix = None\n      ),\n      NotificationSummariesEntityDataset.id -> DatasetParams(\n        stratoSuffix = Some(\"onboarding\"),\n        serveWithin = Some(ServeWithin(duration = 45.millis, roundTripAllowance = None)),\n        batchingPolicy = Some(BatchingPolicy.Isolated(10))\n      ),\n      UserMobileSdkDataset.id -> DatasetParams(\n        stratoSuffix = Some(\"onboarding\")\n      ),\n      PostNuxAlgorithmIdAggregateDataset.id -> DatasetParams(\n        stratoSuffix = Some(\"onboarding\")\n      ),\n      PostNuxAlgorithmTypeAggregateDataset.id -> DatasetParams(\n        stratoSuffix = Some(\"onboarding\")\n      ),\n    ),\n    enableFeatureGenerationStats = true\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/FeatureStorePostNuxAlgorithmSource.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.github.benmanes.caffeine.cache.Caffeine\nimport com.google.inject.Inject\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.TimeoutException\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.constants.CandidateAlgorithmTypeConstants\nimport com.twitter.follow_recommendations.common.feature_hydration.adapters.CandidateAlgorithmAdapter.remapCandidateSource\nimport com.twitter.follow_recommendations.common.feature_hydration.adapters.PostNuxAlgorithmIdAdapter\nimport com.twitter.follow_recommendations.common.feature_hydration.adapters.PostNuxAlgorithmTypeAdapter\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSource\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSourceId\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.Utils.adaptAdditionalFeaturesToDataRecord\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.Utils.randomizedTTL\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.hermit.constants.AlgorithmFeedbackTokens.AlgorithmToFeedbackTokenMap\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.DataRecordMerger\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.IRecordOneToOneAdapter\nimport com.twitter.ml.featurestore.catalog.datasets.customer_journey.PostNuxAlgorithmIdAggregateDataset\nimport com.twitter.ml.featurestore.catalog.datasets.customer_journey.PostNuxAlgorithmTypeAggregateDataset\nimport com.twitter.ml.featurestore.catalog.entities.onboarding.{WtfAlgorithm => OnboardingWtfAlgoId}\nimport com.twitter.ml.featurestore.catalog.entities.onboarding.{\n  WtfAlgorithmType => OnboardingWtfAlgoType\n}\nimport com.twitter.ml.featurestore.catalog.features.customer_journey.CombineAllFeaturesPolicy\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.WtfAlgorithmId\nimport com.twitter.ml.featurestore.lib.WtfAlgorithmType\nimport com.twitter.ml.featurestore.lib.data.PredictionRecord\nimport com.twitter.ml.featurestore.lib.data.PredictionRecordAdapter\nimport com.twitter.ml.featurestore.lib.dataset.DatasetId\nimport com.twitter.ml.featurestore.lib.dataset.online.Hydrator.HydrationResponse\nimport com.twitter.ml.featurestore.lib.dataset.online.OnlineAccessDataset\nimport com.twitter.ml.featurestore.lib.dynamic.ClientConfig\nimport com.twitter.ml.featurestore.lib.dynamic.DynamicFeatureStoreClient\nimport com.twitter.ml.featurestore.lib.dynamic.DynamicHydrationConfig\nimport com.twitter.ml.featurestore.lib.dynamic.FeatureStoreParamsConfig\nimport com.twitter.ml.featurestore.lib.dynamic.GatedFeatures\nimport com.twitter.ml.featurestore.lib.entity.EntityWithId\nimport com.twitter.ml.featurestore.lib.feature.BoundFeature\nimport com.twitter.ml.featurestore.lib.feature.BoundFeatureSet\nimport com.twitter.ml.featurestore.lib.online.DatasetValuesCache\nimport com.twitter.ml.featurestore.lib.online.FeatureStoreRequest\nimport com.twitter.ml.featurestore.lib.online.OnlineFeatureGenerationStats\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport java.util.concurrent.TimeUnit\nimport scala.collection.JavaConverters._\n\nclass FeatureStorePostNuxAlgorithmSource @Inject() (\n  serviceIdentifier: ServiceIdentifier,\n  stats: StatsReceiver)\n    extends FeatureSource {\n  import FeatureStorePostNuxAlgorithmSource._\n\n  val backupSourceStats = stats.scope(\"feature_store_hydration_post_nux_algorithm\")\n  val adapterStats = backupSourceStats.scope(\"adapters\")\n  override def id: FeatureSourceId = FeatureSourceId.FeatureStorePostNuxAlgorithmSourceId\n  override def featureContext: FeatureContext = getFeatureContext\n\n  private val dataRecordMerger = new DataRecordMerger\n\n  val clientConfig: ClientConfig[HasParams] = ClientConfig(\n    dynamicHydrationConfig = dynamicHydrationConfig,\n    featureStoreParamsConfig =\n      FeatureStoreParamsConfig(FeatureStoreParameters.featureStoreParams, Map.empty),\n    /**\n     * The smaller one between `timeoutProvider` and `FeatureStoreSourceParams.GlobalFetchTimeout`\n     * used below takes effect.\n     */\n    timeoutProvider = Function.const(800.millis),\n    serviceIdentifier = serviceIdentifier\n  )\n\n  private val datasetsToCache = Set(\n    PostNuxAlgorithmIdAggregateDataset,\n    PostNuxAlgorithmTypeAggregateDataset,\n  ).asInstanceOf[Set[OnlineAccessDataset[_ <: EntityId, _]]]\n\n  private val datasetValuesCache: DatasetValuesCache =\n    DatasetValuesCache(\n      Caffeine\n        .newBuilder()\n        .expireAfterWrite(randomizedTTL(12.hours.inSeconds), TimeUnit.SECONDS)\n        .maximumSize(DefaultCacheMaxKeys)\n        .build[(_ <: EntityId, DatasetId), Stitch[HydrationResponse[_]]]\n        .asMap,\n      datasetsToCache,\n      DatasetCacheScope\n    )\n\n  private val dynamicFeatureStoreClient = DynamicFeatureStoreClient(\n    clientConfig,\n    backupSourceStats,\n    Set(datasetValuesCache)\n  )\n\n  private val adapterToDataRecord: IRecordOneToOneAdapter[PredictionRecord] =\n    PredictionRecordAdapter.oneToOne(\n      BoundFeatureSet(allFeatures),\n      OnlineFeatureGenerationStats(backupSourceStats)\n    )\n\n  // These two calculate the rate for each feature by dividing it by the number of impressions, then\n  // apply a log transformation.\n  private val transformAdapters = Seq(PostNuxAlgorithmIdAdapter, PostNuxAlgorithmTypeAdapter)\n  override def hydrateFeatures(\n    target: HasClientContext\n      with HasPreFetchedFeature\n      with HasParams\n      with HasSimilarToContext\n      with HasDisplayLocation,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Map[CandidateUser, DataRecord]] = {\n    target.getOptionalUserId\n      .map { _: Long =>\n        val candidateAlgoIdEntities = candidates.map { candidate =>\n          candidate.id -> candidate.getAllAlgorithms\n            .flatMap { algo =>\n              AlgorithmToFeedbackTokenMap.get(remapCandidateSource(algo))\n            }.map(algoId => OnboardingWtfAlgoId.withId(WtfAlgorithmId(algoId)))\n        }.toMap\n\n        val candidateAlgoTypeEntities = candidateAlgoIdEntities.map {\n          case (candidateId, algoIdEntities) =>\n            candidateId -> algoIdEntities\n              .map(_.id.algoId)\n              .flatMap(algoId => CandidateAlgorithmTypeConstants.getAlgorithmTypes(algoId.toString))\n              .distinct\n              .map(algoType => OnboardingWtfAlgoType.withId(WtfAlgorithmType(algoType)))\n        }\n\n        val entities = {\n          candidateAlgoIdEntities.values.flatten ++ candidateAlgoTypeEntities.values.flatten\n        }.toSeq.distinct\n        val requests = entities.map(entity => FeatureStoreRequest(Seq(entity)))\n\n        val predictionRecordsFut = dynamicFeatureStoreClient(requests, target)\n        val candidateFeatureMap = predictionRecordsFut.map {\n          predictionRecords: Seq[PredictionRecord] =>\n            val entityFeatureMap: Map[EntityWithId[_], DataRecord] = entities\n              .zip(predictionRecords).map {\n                case (entity, predictionRecord) =>\n                  entity -> adaptAdditionalFeaturesToDataRecord(\n                    adapterToDataRecord.adaptToDataRecord(predictionRecord),\n                    adapterStats,\n                    transformAdapters)\n              }.toMap\n\n            // In case we have more than one algorithm ID, or type, for a candidate, we merge the\n            // resulting DataRecords using the two merging policies below.\n            val algoIdMergeFn =\n              CombineAllFeaturesPolicy(PostNuxAlgorithmIdAdapter.getFeatures).getMergeFn\n            val algoTypeMergeFn =\n              CombineAllFeaturesPolicy(PostNuxAlgorithmTypeAdapter.getFeatures).getMergeFn\n\n            val candidateAlgoIdFeaturesMap = candidateAlgoIdEntities.mapValues { entities =>\n              val features = entities.flatMap(e => Option(entityFeatureMap.getOrElse(e, null)))\n              algoIdMergeFn(features)\n            }\n\n            val candidateAlgoTypeFeaturesMap = candidateAlgoTypeEntities.mapValues { entities =>\n              val features = entities.flatMap(e => Option(entityFeatureMap.getOrElse(e, null)))\n              algoTypeMergeFn(features)\n            }\n\n            candidates.map { candidate =>\n              val idDrOpt = candidateAlgoIdFeaturesMap.getOrElse(candidate.id, None)\n              val typeDrOpt = candidateAlgoTypeFeaturesMap.getOrElse(candidate.id, None)\n\n              val featureDr = (idDrOpt, typeDrOpt) match {\n                case (None, Some(typeDataRecord)) => typeDataRecord\n                case (Some(idDataRecord), None) => idDataRecord\n                case (None, None) => new DataRecord()\n                case (Some(idDataRecord), Some(typeDataRecord)) =>\n                  dataRecordMerger.merge(idDataRecord, typeDataRecord)\n                  idDataRecord\n              }\n              candidate -> featureDr\n            }.toMap\n        }\n        Stitch\n          .callFuture(candidateFeatureMap)\n          .within(target.params(FeatureStoreSourceParams.GlobalFetchTimeout))(\n            com.twitter.finagle.util.DefaultTimer)\n          .rescue {\n            case _: TimeoutException =>\n              Stitch.value(Map.empty[CandidateUser, DataRecord])\n          }\n      }.getOrElse(Stitch.value(Map.empty[CandidateUser, DataRecord]))\n  }\n}\n\nobject FeatureStorePostNuxAlgorithmSource {\n  private val DatasetCacheScope = \"feature_store_local_cache_post_nux_algorithm\"\n  private val DefaultCacheMaxKeys = 1000 // Both of these datasets have <50 keys total.\n\n  val allFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    FeatureStoreFeatures.postNuxAlgorithmIdAggregateFeatures ++\n      FeatureStoreFeatures.postNuxAlgorithmTypeAggregateFeatures\n\n  val algoIdFinalFeatures = CombineAllFeaturesPolicy(\n    PostNuxAlgorithmIdAdapter.getFeatures).outputFeaturesPostMerge.toSeq\n  val algoTypeFinalFeatures = CombineAllFeaturesPolicy(\n    PostNuxAlgorithmTypeAdapter.getFeatures).outputFeaturesPostMerge.toSeq\n\n  val getFeatureContext: FeatureContext =\n    new FeatureContext().addFeatures((algoIdFinalFeatures ++ algoTypeFinalFeatures).asJava)\n\n  val dynamicHydrationConfig: DynamicHydrationConfig[HasParams] =\n    DynamicHydrationConfig(\n      Set(\n        GatedFeatures(\n          boundFeatureSet =\n            BoundFeatureSet(FeatureStoreFeatures.postNuxAlgorithmIdAggregateFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableAlgorithmAggregateFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet =\n            BoundFeatureSet(FeatureStoreFeatures.postNuxAlgorithmTypeAggregateFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableAlgorithmAggregateFeatures)\n        ),\n      ))\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/FeatureStoreSource.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.github.benmanes.caffeine.cache.Caffeine\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.TimeoutException\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.feature_hydration.adapters.CandidateAlgorithmAdapter.remapCandidateSource\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSource\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSourceId\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.Utils.adaptAdditionalFeaturesToDataRecord\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.Utils.randomizedTTL\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.hermit.constants.AlgorithmFeedbackTokens.AlgorithmToFeedbackTokenMap\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.IRecordOneToOneAdapter\nimport com.twitter.ml.featurestore.catalog.datasets.core.UsersourceEntityDataset\nimport com.twitter.ml.featurestore.catalog.datasets.magicrecs.NotificationSummariesEntityDataset\nimport com.twitter.ml.featurestore.catalog.datasets.onboarding.MetricCenterUserCountingFeaturesDataset\nimport com.twitter.ml.featurestore.catalog.datasets.timelines.AuthorFeaturesEntityDataset\nimport com.twitter.ml.featurestore.catalog.entities.core.{Author => AuthorEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{AuthorTopic => AuthorTopicEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{CandidateUser => CandidateUserEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{Topic => TopicEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{User => UserEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{UserCandidate => UserCandidateEntity}\nimport com.twitter.ml.featurestore.catalog.entities.onboarding.UserWtfAlgorithmEntity\nimport com.twitter.ml.featurestore.lib.data.PredictionRecord\nimport com.twitter.ml.featurestore.lib.data.PredictionRecordAdapter\nimport com.twitter.ml.featurestore.lib.dataset.online.Hydrator.HydrationResponse\nimport com.twitter.ml.featurestore.lib.dataset.online.OnlineAccessDataset\nimport com.twitter.ml.featurestore.lib.dataset.DatasetId\nimport com.twitter.ml.featurestore.lib.dynamic._\nimport com.twitter.ml.featurestore.lib.feature._\nimport com.twitter.ml.featurestore.lib.online.DatasetValuesCache\nimport com.twitter.ml.featurestore.lib.online.FeatureStoreRequest\nimport com.twitter.ml.featurestore.lib.online.OnlineFeatureGenerationStats\nimport com.twitter.ml.featurestore.lib.EdgeEntityId\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.TopicId\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.ml.featurestore.lib.WtfAlgorithmId\nimport com.twitter.onboarding.relevance.adapters.features.featurestore.CandidateAuthorTopicAggregatesAdapter\nimport com.twitter.onboarding.relevance.adapters.features.featurestore.CandidateTopicEngagementRealTimeAggregatesAdapter\nimport com.twitter.onboarding.relevance.adapters.features.featurestore.CandidateTopicEngagementUserStateRealTimeAggregatesAdapter\nimport com.twitter.onboarding.relevance.adapters.features.featurestore.CandidateTopicNegativeEngagementUserStateRealTimeAggregatesAdapter\nimport com.twitter.onboarding.relevance.adapters.features.featurestore.FeatureStoreAdapter\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\nimport java.util.concurrent.TimeUnit\n\n@Singleton\nclass FeatureStoreSource @Inject() (\n  serviceIdentifier: ServiceIdentifier,\n  stats: StatsReceiver)\n    extends FeatureSource {\n  import FeatureStoreSource._\n\n  override val id: FeatureSourceId = FeatureSourceId.FeatureStoreSourceId\n  override val featureContext: FeatureContext = FeatureStoreSource.getFeatureContext\n  val hydrateFeaturesStats = stats.scope(\"hydrate_features\")\n  val adapterStats = stats.scope(\"adapters\")\n  val featureSet: BoundFeatureSet = BoundFeatureSet(FeatureStoreSource.allFeatures)\n  val clientConfig: ClientConfig[HasParams] = ClientConfig(\n    dynamicHydrationConfig = FeatureStoreSource.dynamicHydrationConfig,\n    featureStoreParamsConfig =\n      FeatureStoreParamsConfig(FeatureStoreParameters.featureStoreParams, Map.empty),\n    /**\n     * The smaller one between `timeoutProvider` and `FeatureStoreSourceParams.GlobalFetchTimeout`\n     * used below takes effect.\n     */\n    timeoutProvider = Function.const(800.millis),\n    serviceIdentifier = serviceIdentifier\n  )\n\n  private val datasetsToCache = Set(\n    MetricCenterUserCountingFeaturesDataset,\n    UsersourceEntityDataset,\n    AuthorFeaturesEntityDataset,\n    NotificationSummariesEntityDataset\n  ).asInstanceOf[Set[OnlineAccessDataset[_ <: EntityId, _]]]\n\n  private val datasetValuesCache: DatasetValuesCache =\n    DatasetValuesCache(\n      Caffeine\n        .newBuilder()\n        .expireAfterWrite(randomizedTTL(12.hours.inSeconds), TimeUnit.SECONDS)\n        .maximumSize(DefaultCacheMaxKeys)\n        .build[(_ <: EntityId, DatasetId), Stitch[HydrationResponse[_]]]\n        .asMap,\n      datasetsToCache,\n      DatasetCacheScope\n    )\n\n  private val dynamicFeatureStoreClient = DynamicFeatureStoreClient(\n    clientConfig,\n    stats,\n    Set(datasetValuesCache)\n  )\n\n  private val adapter: IRecordOneToOneAdapter[PredictionRecord] =\n    PredictionRecordAdapter.oneToOne(\n      BoundFeatureSet(allFeatures),\n      OnlineFeatureGenerationStats(stats)\n    )\n\n  override def hydrateFeatures(\n    target: HasClientContext\n      with HasPreFetchedFeature\n      with HasParams\n      with HasSimilarToContext\n      with HasDisplayLocation,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Map[CandidateUser, DataRecord]] = {\n    target.getOptionalUserId\n      .map { targetUserId =>\n        val featureRequests = candidates.map { candidate =>\n          val userId = UserId(targetUserId)\n          val userEntityId = UserEntity.withId(userId)\n          val candidateEntityId = CandidateUserEntity.withId(UserId(candidate.id))\n          val userCandidateEdgeEntityId =\n            UserCandidateEntity.withId(EdgeEntityId(userId, UserId(candidate.id)))\n          val similarToUserId = target.similarToUserIds.map(id => AuthorEntity.withId(UserId(id)))\n          val topicProof = candidate.reason.flatMap(_.accountProof.flatMap(_.topicProof))\n          val topicEntities = if (topicProof.isDefined) {\n            hydrateFeaturesStats.counter(\"candidates_with_topic_proof\").incr()\n            val topicId = topicProof.get.topicId\n            val topicEntityId = TopicEntity.withId(TopicId(topicId))\n            val authorTopicEntityId =\n              AuthorTopicEntity.withId(EdgeEntityId(UserId(candidate.id), TopicId(topicId)))\n            Seq(topicEntityId, authorTopicEntityId)\n          } else Nil\n\n          val candidateAlgorithmsWithScores = candidate.getAllAlgorithms\n          val userWtfAlgEdgeEntities =\n            candidateAlgorithmsWithScores.flatMap(algo => {\n              val algoId = AlgorithmToFeedbackTokenMap.get(remapCandidateSource(algo))\n              algoId.map(id =>\n                UserWtfAlgorithmEntity.withId(EdgeEntityId(userId, WtfAlgorithmId(id))))\n            })\n\n          val entities = Seq(\n            userEntityId,\n            candidateEntityId,\n            userCandidateEdgeEntityId) ++ similarToUserId ++ topicEntities ++ userWtfAlgEdgeEntities\n          FeatureStoreRequest(entities)\n        }\n\n        val predictionRecordsFut = dynamicFeatureStoreClient(featureRequests, target)\n        val candidateFeatureMap = predictionRecordsFut.map { predictionRecords =>\n          // we can zip predictionRecords with candidates as the order is preserved in the client\n          candidates\n            .zip(predictionRecords).map {\n              case (candidate, predictionRecord) =>\n                candidate -> adaptAdditionalFeaturesToDataRecord(\n                  adapter.adaptToDataRecord(predictionRecord),\n                  adapterStats,\n                  FeatureStoreSource.featureAdapters)\n            }.toMap\n        }\n        Stitch\n          .callFuture(candidateFeatureMap)\n          .within(target.params(FeatureStoreSourceParams.GlobalFetchTimeout))(\n            com.twitter.finagle.util.DefaultTimer)\n          .rescue {\n            case _: TimeoutException =>\n              Stitch.value(Map.empty[CandidateUser, DataRecord])\n          }\n      }.getOrElse(Stitch.value(Map.empty[CandidateUser, DataRecord]))\n  }\n}\n\n// list of features that we will be fetching, even if we are only scribing but not scoring with them\nobject FeatureStoreSource {\n\n  private val DatasetCacheScope = \"feature_store_local_cache\"\n  private val DefaultCacheMaxKeys = 70000\n\n  import FeatureStoreFeatures._\n\n  ///////////////////// ALL hydrated features /////////////////////\n  val allFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    //target user\n    targetUserFeatures ++\n      targetUserUserAuthorUserStateRealTimeAggregatesFeature ++\n      targetUserResurrectionFeatures ++\n      targetUserWtfImpressionFeatures ++\n      targetUserStatusFeatures ++\n      targetUserMetricCountFeatures ++\n      //candidate user\n      candidateUserFeatures ++\n      candidateUserResurrectionFeatures ++\n      candidateUserAuthorRealTimeAggregateFeatures ++\n      candidateUserStatusFeatures ++\n      candidateUserMetricCountFeatures ++\n      candidateUserTimelinesAuthorAggregateFeatures ++\n      candidateUserClientFeatures ++\n      //similar to user\n      similarToUserFeatures ++\n      similarToUserStatusFeatures ++\n      similarToUserMetricCountFeatures ++\n      similarToUserTimelinesAuthorAggregateFeatures ++\n      //other\n      userCandidateEdgeFeatures ++\n      userCandidateWtfImpressionCandidateFeatures ++\n      topicFeatures ++\n      userWtfAlgorithmEdgeFeatures ++\n      targetUserClientFeatures\n\n  val dynamicHydrationConfig: DynamicHydrationConfig[HasParams] =\n    DynamicHydrationConfig(\n      Set(\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(topicAggregateFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableTopicAggregateFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(authorTopicFeatures),\n          gate =\n            HasParams\n              .paramGate(FeatureStoreSourceParams.EnableSeparateClientForTimelinesAuthors).unary_! &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableAuthorTopicAggregateFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(userTopicFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableUserTopicFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(targetUserFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableTargetUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(targetUserUserAuthorUserStateRealTimeAggregatesFeature),\n          gate = HasParams.paramGate(\n            FeatureStoreSourceParams.EnableTargetUserUserAuthorUserStateRealTimeAggregatesFeature)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(targetUserResurrectionFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableTargetUserResurrectionFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(targetUserWtfImpressionFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableTargetUserWtfImpressionFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(targetUserStatusFeatures),\n          gate =\n            HasParams.paramGate(FeatureStoreSourceParams.EnableSeparateClientForGizmoduck).unary_! &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableTargetUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(targetUserMetricCountFeatures),\n          gate = HasParams\n            .paramGate(\n              FeatureStoreSourceParams.EnableSeparateClientForMetricCenterUserCounting).unary_! &\n            HasParams.paramGate(FeatureStoreSourceParams.EnableTargetUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(candidateUserFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableCandidateUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(candidateUserAuthorRealTimeAggregateFeatures),\n          gate = HasParams.paramGate(\n            FeatureStoreSourceParams.EnableCandidateUserAuthorRealTimeAggregateFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(candidateUserResurrectionFeatures),\n          gate =\n            HasParams.paramGate(FeatureStoreSourceParams.EnableCandidateUserResurrectionFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(candidateUserStatusFeatures),\n          gate =\n            HasParams.paramGate(FeatureStoreSourceParams.EnableSeparateClientForGizmoduck).unary_! &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableCandidateUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(candidateUserTimelinesAuthorAggregateFeatures),\n          gate =\n            HasParams\n              .paramGate(FeatureStoreSourceParams.EnableSeparateClientForTimelinesAuthors).unary_! &\n              HasParams.paramGate(\n                FeatureStoreSourceParams.EnableCandidateUserTimelinesAuthorAggregateFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(candidateUserMetricCountFeatures),\n          gate =\n            HasParams\n              .paramGate(\n                FeatureStoreSourceParams.EnableSeparateClientForMetricCenterUserCounting).unary_! &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableCandidateUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(userCandidateEdgeFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableUserCandidateEdgeFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(userCandidateWtfImpressionCandidateFeatures),\n          gate = HasParams.paramGate(\n            FeatureStoreSourceParams.EnableUserCandidateWtfImpressionCandidateFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(userWtfAlgorithmEdgeFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableUserWtfAlgEdgeFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(similarToUserFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableSimilarToUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(similarToUserStatusFeatures),\n          gate =\n            HasParams.paramGate(FeatureStoreSourceParams.EnableSeparateClientForGizmoduck).unary_! &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableSimilarToUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(similarToUserTimelinesAuthorAggregateFeatures),\n          gate =\n            HasParams\n              .paramGate(FeatureStoreSourceParams.EnableSeparateClientForTimelinesAuthors).unary_! &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableSimilarToUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(similarToUserMetricCountFeatures),\n          gate =\n            HasParams\n              .paramGate(\n                FeatureStoreSourceParams.EnableSeparateClientForMetricCenterUserCounting).unary_! &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableSimilarToUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(candidateUserClientFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableCandidateClientFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(targetUserClientFeatures),\n          gate = HasParams.paramGate(FeatureStoreSourceParams.EnableUserClientFeatures)\n        ),\n      )\n    )\n  // for calibrating features, e.g. add log transformed topic features\n  val featureAdapters: Seq[FeatureStoreAdapter] = Seq(\n    CandidateTopicEngagementRealTimeAggregatesAdapter,\n    CandidateTopicNegativeEngagementUserStateRealTimeAggregatesAdapter,\n    CandidateTopicEngagementUserStateRealTimeAggregatesAdapter,\n    CandidateAuthorTopicAggregatesAdapter\n  )\n  val additionalFeatureContext: FeatureContext = FeatureContext.merge(\n    featureAdapters\n      .foldRight(new FeatureContext())((adapter, context) =>\n        context\n          .addFeatures(adapter.getFeatureContext))\n  )\n  val getFeatureContext: FeatureContext =\n    BoundFeatureSet(allFeatures).toFeatureContext\n      .addFeatures(additionalFeatureContext)\n      // The below are aggregated features that are aggregated for a second time over multiple keys.\n      .addFeatures(maxSumAvgAggregatedFeatureContext)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/FeatureStoreSourceParams.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.util.Duration\nimport com.twitter.conversions.DurationOps._\n\nobject FeatureStoreSourceParams {\n  case object EnableTopicAggregateFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableTopicAggregateFeatures,\n        default = true\n      )\n  case object EnableAlgorithmAggregateFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableAlgorithmAggregateFeatures,\n        default = false\n      )\n  case object EnableAuthorTopicAggregateFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableAuthorTopicAggregateFeatures,\n        default = true\n      )\n  case object EnableUserTopicFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableUserTopicFeatures,\n        default = false\n      )\n  case object EnableTargetUserFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableTargetUserFeatures,\n        default = true\n      )\n  case object EnableTargetUserUserAuthorUserStateRealTimeAggregatesFeature\n      extends FSParam[Boolean](\n        name =\n          FeatureHydrationSourcesFeatureSwitchKeys.EnableTargetUserUserAuthorUserStateRealTimeAggregatesFeature,\n        default = true\n      )\n  case object EnableTargetUserResurrectionFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableTargetUserResurrectionFeatures,\n        default = true\n      )\n  case object EnableTargetUserWtfImpressionFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableTargetUserWtfImpressionFeatures,\n        default = true\n      )\n  case object EnableCandidateUserFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableCandidateUserFeatures,\n        default = true\n      )\n  case object EnableCandidateUserAuthorRealTimeAggregateFeatures\n      extends FSParam[Boolean](\n        name =\n          FeatureHydrationSourcesFeatureSwitchKeys.EnableCandidateUserAuthorRealTimeAggregateFeatures,\n        default = true\n      )\n  case object EnableCandidateUserResurrectionFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableCandidateUserResurrectionFeatures,\n        default = true\n      )\n  case object EnableCandidateUserTimelinesAuthorAggregateFeatures\n      extends FSParam[Boolean](\n        name =\n          FeatureHydrationSourcesFeatureSwitchKeys.EnableCandidateUserTimelinesAuthorAggregateFeatures,\n        default = true\n      )\n  case object EnableUserCandidateEdgeFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableUserCandidateEdgeFeatures,\n        default = true\n      )\n  case object EnableUserCandidateWtfImpressionCandidateFeatures\n      extends FSParam[Boolean](\n        name =\n          FeatureHydrationSourcesFeatureSwitchKeys.EnableUserCandidateWtfImpressionCandidateFeatures,\n        default = true\n      )\n  case object EnableUserWtfAlgEdgeFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableUserWtfAlgEdgeFeatures,\n        default = false\n      )\n  case object EnableSimilarToUserFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableSimilarToUserFeatures,\n        default = true\n      )\n\n  case object EnableCandidatePrecomputedNotificationFeatures\n      extends FSParam[Boolean](\n        name =\n          FeatureHydrationSourcesFeatureSwitchKeys.EnableCandidatePrecomputedNotificationFeatures,\n        default = false\n      )\n\n  case object EnableCandidateClientFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableCandidateClientFeatures,\n        default = false\n      )\n\n  case object EnableUserClientFeatures\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.EnableUserClientFeatures,\n        default = false\n      )\n\n  case object EnableSeparateClientForTimelinesAuthors\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.UseSeparateClientForTimelinesAuthor,\n        default = false\n      )\n\n  case object EnableSeparateClientForMetricCenterUserCounting\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.UseSeparateClientMetricCenterUserCounting,\n        default = false\n      )\n\n  case object EnableSeparateClientForNotifications\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.UseSeparateClientForNotifications,\n        default = false\n      )\n\n  case object EnableSeparateClientForGizmoduck\n      extends FSParam[Boolean](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.UseSeparateClientForGizmoduck,\n        default = false\n      )\n\n  case object GlobalFetchTimeout\n      extends FSBoundedParam[Duration](\n        name = FeatureHydrationSourcesFeatureSwitchKeys.FeatureHydrationTimeout,\n        default = 240.millisecond,\n        min = 100.millisecond,\n        max = 400.millisecond)\n      with HasDurationConversion {\n    override def durationConversion: DurationConversion = DurationConversion.FromMillis\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/FeatureStoreTimelinesAuthorSource.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.github.benmanes.caffeine.cache.Caffeine\nimport com.google.inject.Inject\nimport com.twitter.finagle.TimeoutException\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSource\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSourceId\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.Utils.adaptAdditionalFeaturesToDataRecord\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.Utils.randomizedTTL\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.IRecordOneToOneAdapter\nimport com.twitter.ml.featurestore.catalog.datasets.timelines.AuthorFeaturesEntityDataset\nimport com.twitter.ml.featurestore.catalog.entities.core.{Author => AuthorEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{AuthorTopic => AuthorTopicEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{CandidateUser => CandidateUserEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{User => UserEntity}\nimport com.twitter.ml.featurestore.lib.EdgeEntityId\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.TopicId\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.ml.featurestore.lib.data.PredictionRecord\nimport com.twitter.ml.featurestore.lib.data.PredictionRecordAdapter\nimport com.twitter.ml.featurestore.lib.dataset.DatasetId\nimport com.twitter.ml.featurestore.lib.dataset.online.Hydrator.HydrationResponse\nimport com.twitter.ml.featurestore.lib.dataset.online.OnlineAccessDataset\nimport com.twitter.ml.featurestore.lib.dynamic.ClientConfig\nimport com.twitter.ml.featurestore.lib.dynamic.DynamicFeatureStoreClient\nimport com.twitter.ml.featurestore.lib.dynamic.DynamicHydrationConfig\nimport com.twitter.ml.featurestore.lib.dynamic.FeatureStoreParamsConfig\nimport com.twitter.ml.featurestore.lib.dynamic.GatedFeatures\nimport com.twitter.ml.featurestore.lib.feature.BoundFeature\nimport com.twitter.ml.featurestore.lib.feature.BoundFeatureSet\nimport com.twitter.ml.featurestore.lib.online.DatasetValuesCache\nimport com.twitter.ml.featurestore.lib.online.FeatureStoreRequest\nimport com.twitter.ml.featurestore.lib.online.OnlineFeatureGenerationStats\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport java.util.concurrent.TimeUnit\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\n\nclass FeatureStoreTimelinesAuthorSource @Inject() (\n  serviceIdentifier: ServiceIdentifier,\n  stats: StatsReceiver)\n    extends FeatureSource {\n  import FeatureStoreTimelinesAuthorSource._\n\n  val backupSourceStats = stats.scope(\"feature_store_hydration_timelines_author\")\n  val adapterStats = backupSourceStats.scope(\"adapters\")\n  override def id: FeatureSourceId = FeatureSourceId.FeatureStoreTimelinesAuthorSourceId\n  override def featureContext: FeatureContext = getFeatureContext\n\n  val clientConfig: ClientConfig[HasParams] = ClientConfig(\n    dynamicHydrationConfig = dynamicHydrationConfig,\n    featureStoreParamsConfig =\n      FeatureStoreParamsConfig(FeatureStoreParameters.featureStoreParams, Map.empty),\n    /**\n     * The smaller one between `timeoutProvider` and `FeatureStoreSourceParams.GlobalFetchTimeout`\n     * used below takes effect.\n     */\n    timeoutProvider = Function.const(800.millis),\n    serviceIdentifier = serviceIdentifier\n  )\n\n  private val datasetsToCache = Set(\n    AuthorFeaturesEntityDataset\n  ).asInstanceOf[Set[OnlineAccessDataset[_ <: EntityId, _]]]\n\n  private val datasetValuesCache: DatasetValuesCache =\n    DatasetValuesCache(\n      Caffeine\n        .newBuilder()\n        .expireAfterWrite(randomizedTTL(12.hours.inSeconds), TimeUnit.SECONDS)\n        .maximumSize(DefaultCacheMaxKeys)\n        .build[(_ <: EntityId, DatasetId), Stitch[HydrationResponse[_]]]\n        .asMap,\n      datasetsToCache,\n      DatasetCacheScope\n    )\n\n  private val dynamicFeatureStoreClient = DynamicFeatureStoreClient(\n    clientConfig,\n    backupSourceStats,\n    Set(datasetValuesCache)\n  )\n\n  private val adapter: IRecordOneToOneAdapter[PredictionRecord] =\n    PredictionRecordAdapter.oneToOne(\n      BoundFeatureSet(allFeatures),\n      OnlineFeatureGenerationStats(backupSourceStats)\n    )\n\n  override def hydrateFeatures(\n    target: HasClientContext\n      with HasPreFetchedFeature\n      with HasParams\n      with HasSimilarToContext\n      with HasDisplayLocation,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Map[CandidateUser, DataRecord]] = {\n    target.getOptionalUserId\n      .map { targetUserId =>\n        val featureRequests = candidates.map { candidate =>\n          val userEntityId = UserEntity.withId(UserId(targetUserId))\n          val candidateEntityId = CandidateUserEntity.withId(UserId(candidate.id))\n          val similarToUserId = target.similarToUserIds.map(id => AuthorEntity.withId(UserId(id)))\n          val topicProof = candidate.reason.flatMap(_.accountProof.flatMap(_.topicProof))\n          val authorTopicEntity = if (topicProof.isDefined) {\n            backupSourceStats.counter(\"candidates_with_topic_proof\").incr()\n            Set(\n              AuthorTopicEntity.withId(\n                EdgeEntityId(UserId(candidate.id), TopicId(topicProof.get.topicId))))\n          } else Nil\n\n          val entities =\n            Seq(userEntityId, candidateEntityId) ++ similarToUserId ++ authorTopicEntity\n          FeatureStoreRequest(entities)\n        }\n\n        val predictionRecordsFut = dynamicFeatureStoreClient(featureRequests, target)\n        val candidateFeatureMap = predictionRecordsFut.map { predictionRecords =>\n          // we can zip predictionRecords with candidates as the order is preserved in the client\n          candidates\n            .zip(predictionRecords).map {\n              case (candidate, predictionRecord) =>\n                candidate -> adaptAdditionalFeaturesToDataRecord(\n                  adapter.adaptToDataRecord(predictionRecord),\n                  adapterStats,\n                  FeatureStoreSource.featureAdapters)\n            }.toMap\n        }\n        Stitch\n          .callFuture(candidateFeatureMap)\n          .within(target.params(FeatureStoreSourceParams.GlobalFetchTimeout))(\n            com.twitter.finagle.util.DefaultTimer)\n          .rescue {\n            case _: TimeoutException =>\n              Stitch.value(Map.empty[CandidateUser, DataRecord])\n          }\n      }.getOrElse(Stitch.value(Map.empty[CandidateUser, DataRecord]))\n  }\n}\n\nobject FeatureStoreTimelinesAuthorSource {\n  private val DatasetCacheScope = \"feature_store_local_cache_timelines_author\"\n  private val DefaultCacheMaxKeys = 20000\n\n  import FeatureStoreFeatures._\n\n  val allFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    similarToUserTimelinesAuthorAggregateFeatures ++\n      candidateUserTimelinesAuthorAggregateFeatures ++\n      authorTopicFeatures\n\n  val getFeatureContext: FeatureContext =\n    BoundFeatureSet(allFeatures).toFeatureContext\n\n  val dynamicHydrationConfig: DynamicHydrationConfig[HasParams] =\n    DynamicHydrationConfig(\n      Set(\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(authorTopicFeatures),\n          gate =\n            HasParams\n              .paramGate(FeatureStoreSourceParams.EnableSeparateClientForTimelinesAuthors) &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableAuthorTopicAggregateFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(similarToUserTimelinesAuthorAggregateFeatures),\n          gate =\n            HasParams\n              .paramGate(FeatureStoreSourceParams.EnableSeparateClientForTimelinesAuthors) &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableSimilarToUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(candidateUserTimelinesAuthorAggregateFeatures),\n          gate =\n            HasParams\n              .paramGate(FeatureStoreSourceParams.EnableSeparateClientForTimelinesAuthors) &\n              HasParams.paramGate(\n                FeatureStoreSourceParams.EnableCandidateUserTimelinesAuthorAggregateFeatures)\n        ),\n      ))\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/FeatureStoreUserMetricCountsSource.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.github.benmanes.caffeine.cache.Caffeine\nimport com.google.inject.Inject\nimport com.twitter.finagle.TimeoutException\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSource\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSourceId\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.Utils.adaptAdditionalFeaturesToDataRecord\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.Utils.randomizedTTL\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.IRecordOneToOneAdapter\nimport com.twitter.ml.featurestore.catalog.datasets.onboarding.MetricCenterUserCountingFeaturesDataset\nimport com.twitter.ml.featurestore.catalog.entities.core.{Author => AuthorEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{AuthorTopic => AuthorTopicEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{CandidateUser => CandidateUserEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{User => UserEntity}\nimport com.twitter.ml.featurestore.lib.EdgeEntityId\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.TopicId\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.ml.featurestore.lib.data.PredictionRecord\nimport com.twitter.ml.featurestore.lib.data.PredictionRecordAdapter\nimport com.twitter.ml.featurestore.lib.dataset.DatasetId\nimport com.twitter.ml.featurestore.lib.dataset.online.Hydrator.HydrationResponse\nimport com.twitter.ml.featurestore.lib.dataset.online.OnlineAccessDataset\nimport com.twitter.ml.featurestore.lib.dynamic.ClientConfig\nimport com.twitter.ml.featurestore.lib.dynamic.DynamicFeatureStoreClient\nimport com.twitter.ml.featurestore.lib.dynamic.DynamicHydrationConfig\nimport com.twitter.ml.featurestore.lib.dynamic.FeatureStoreParamsConfig\nimport com.twitter.ml.featurestore.lib.dynamic.GatedFeatures\nimport com.twitter.ml.featurestore.lib.feature.BoundFeature\nimport com.twitter.ml.featurestore.lib.feature.BoundFeatureSet\nimport com.twitter.ml.featurestore.lib.online.DatasetValuesCache\nimport com.twitter.ml.featurestore.lib.online.FeatureStoreRequest\nimport com.twitter.ml.featurestore.lib.online.OnlineFeatureGenerationStats\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport java.util.concurrent.TimeUnit\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\n\nclass FeatureStoreUserMetricCountsSource @Inject() (\n  serviceIdentifier: ServiceIdentifier,\n  stats: StatsReceiver)\n    extends FeatureSource {\n  import FeatureStoreUserMetricCountsSource._\n\n  val backupSourceStats = stats.scope(\"feature_store_hydration_mc_counting\")\n  val adapterStats = backupSourceStats.scope(\"adapters\")\n  override def id: FeatureSourceId = FeatureSourceId.FeatureStoreUserMetricCountsSourceId\n  override def featureContext: FeatureContext = getFeatureContext\n\n  val clientConfig: ClientConfig[HasParams] = ClientConfig(\n    dynamicHydrationConfig = dynamicHydrationConfig,\n    featureStoreParamsConfig =\n      FeatureStoreParamsConfig(FeatureStoreParameters.featureStoreParams, Map.empty),\n    /**\n     * The smaller one between `timeoutProvider` and `FeatureStoreSourceParams.GlobalFetchTimeout`\n     * used below takes effect.\n     */\n    timeoutProvider = Function.const(800.millis),\n    serviceIdentifier = serviceIdentifier\n  )\n\n  private val datasetsToCache = Set(\n    MetricCenterUserCountingFeaturesDataset\n  ).asInstanceOf[Set[OnlineAccessDataset[_ <: EntityId, _]]]\n\n  private val datasetValuesCache: DatasetValuesCache =\n    DatasetValuesCache(\n      Caffeine\n        .newBuilder()\n        .expireAfterWrite(randomizedTTL(12.hours.inSeconds), TimeUnit.SECONDS)\n        .maximumSize(DefaultCacheMaxKeys)\n        .build[(_ <: EntityId, DatasetId), Stitch[HydrationResponse[_]]]\n        .asMap,\n      datasetsToCache,\n      DatasetCacheScope\n    )\n\n  private val dynamicFeatureStoreClient = DynamicFeatureStoreClient(\n    clientConfig,\n    backupSourceStats,\n    Set(datasetValuesCache)\n  )\n\n  private val adapter: IRecordOneToOneAdapter[PredictionRecord] =\n    PredictionRecordAdapter.oneToOne(\n      BoundFeatureSet(allFeatures),\n      OnlineFeatureGenerationStats(backupSourceStats)\n    )\n\n  override def hydrateFeatures(\n    target: HasClientContext\n      with HasPreFetchedFeature\n      with HasParams\n      with HasSimilarToContext\n      with HasDisplayLocation,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Map[CandidateUser, DataRecord]] = {\n    target.getOptionalUserId\n      .map { targetUserId =>\n        val featureRequests = candidates.map { candidate =>\n          val userEntityId = UserEntity.withId(UserId(targetUserId))\n          val candidateEntityId = CandidateUserEntity.withId(UserId(candidate.id))\n          val similarToUserId = target.similarToUserIds.map(id => AuthorEntity.withId(UserId(id)))\n          val topicProof = candidate.reason.flatMap(_.accountProof.flatMap(_.topicProof))\n          val authorTopicEntity = if (topicProof.isDefined) {\n            backupSourceStats.counter(\"candidates_with_topic_proof\").incr()\n            Set(\n              AuthorTopicEntity.withId(\n                EdgeEntityId(UserId(candidate.id), TopicId(topicProof.get.topicId))))\n          } else Nil\n\n          val entities =\n            Seq(userEntityId, candidateEntityId) ++ similarToUserId ++ authorTopicEntity\n          FeatureStoreRequest(entities)\n        }\n\n        val predictionRecordsFut = dynamicFeatureStoreClient(featureRequests, target)\n        val candidateFeatureMap = predictionRecordsFut.map { predictionRecords =>\n          // we can zip predictionRecords with candidates as the order is preserved in the client\n          candidates\n            .zip(predictionRecords).map {\n              case (candidate, predictionRecord) =>\n                candidate -> adaptAdditionalFeaturesToDataRecord(\n                  adapter.adaptToDataRecord(predictionRecord),\n                  adapterStats,\n                  FeatureStoreSource.featureAdapters)\n            }.toMap\n        }\n        Stitch\n          .callFuture(candidateFeatureMap)\n          .within(target.params(FeatureStoreSourceParams.GlobalFetchTimeout))(\n            com.twitter.finagle.util.DefaultTimer)\n          .rescue {\n            case _: TimeoutException =>\n              Stitch.value(Map.empty[CandidateUser, DataRecord])\n          }\n      }.getOrElse(Stitch.value(Map.empty[CandidateUser, DataRecord]))\n  }\n}\n\nobject FeatureStoreUserMetricCountsSource {\n  private val DatasetCacheScope = \"feature_store_local_cache_mc_user_counting\"\n  private val DefaultCacheMaxKeys = 20000\n\n  val allFeatures: Set[BoundFeature[_ <: EntityId, _]] =\n    FeatureStoreFeatures.candidateUserMetricCountFeatures ++\n      FeatureStoreFeatures.similarToUserMetricCountFeatures ++\n      FeatureStoreFeatures.targetUserMetricCountFeatures\n\n  val getFeatureContext: FeatureContext =\n    BoundFeatureSet(allFeatures).toFeatureContext\n\n  val dynamicHydrationConfig: DynamicHydrationConfig[HasParams] =\n    DynamicHydrationConfig(\n      Set(\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(FeatureStoreFeatures.targetUserMetricCountFeatures),\n          gate = HasParams\n            .paramGate(FeatureStoreSourceParams.EnableSeparateClientForMetricCenterUserCounting) &\n            HasParams.paramGate(FeatureStoreSourceParams.EnableTargetUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(FeatureStoreFeatures.candidateUserMetricCountFeatures),\n          gate =\n            HasParams\n              .paramGate(FeatureStoreSourceParams.EnableSeparateClientForMetricCenterUserCounting) &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableCandidateUserFeatures)\n        ),\n        GatedFeatures(\n          boundFeatureSet = BoundFeatureSet(FeatureStoreFeatures.similarToUserMetricCountFeatures),\n          gate =\n            HasParams\n              .paramGate(FeatureStoreSourceParams.EnableSeparateClientForMetricCenterUserCounting) &\n              HasParams.paramGate(FeatureStoreSourceParams.EnableSimilarToUserFeatures)\n        ),\n      ))\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/HydrationSourcesModule.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.escherbird.util.stitchcache.StitchCache\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.stitch.Stitch\nimport com.twitter.storage.client.manhattan.bijections.Bijections.BinaryCompactScalaInjection\nimport com.twitter.storage.client.manhattan.bijections.Bijections.LongInjection\nimport com.twitter.storage.client.manhattan.kv.Guarantee\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClient\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpoint\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpointBuilder\nimport com.twitter.storage.client.manhattan.kv.impl.Component\nimport com.twitter.storage.client.manhattan.kv.impl.Component0\nimport com.twitter.storage.client.manhattan.kv.impl.KeyDescriptor\nimport com.twitter.storage.client.manhattan.kv.impl.ValueDescriptor\nimport com.twitter.strato.generated.client.ml.featureStore.McUserCountingOnUserClientColumn\nimport com.twitter.strato.generated.client.ml.featureStore.onboarding.TimelinesAuthorFeaturesOnUserClientColumn\nimport com.twitter.timelines.author_features.v1.thriftscala.AuthorFeatures\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.onboarding.relevance.features.thriftscala.MCUserCountingFeatures\nimport java.lang.{Long => JLong}\nimport scala.util.Random\n\nobject HydrationSourcesModule extends TwitterModule {\n\n  val readFromManhattan = flag(\n    \"feature_hydration_enable_reading_from_manhattan\",\n    false,\n    \"Whether to read the data from Manhattan or Strato\")\n\n  val manhattanAppId =\n    flag(\"frs_readonly.appId\", \"ml_features_athena\", \"RO App Id used by the RO FRS service\")\n  val manhattanDestName = flag(\n    \"frs_readonly.destName\",\n    \"/s/manhattan/athena.native-thrift\",\n    \"manhattan Dest Name used by the RO FRS service\")\n\n  @Provides\n  @Singleton\n  def providesAthenaManhattanClient(\n    serviceIdentifier: ServiceIdentifier\n  ): ManhattanKVEndpoint = {\n    val client = ManhattanKVClient(\n      manhattanAppId(),\n      manhattanDestName(),\n      ManhattanKVClientMtlsParams(serviceIdentifier)\n    )\n    ManhattanKVEndpointBuilder(client)\n      .defaultGuarantee(Guarantee.Weak)\n      .build()\n  }\n\n  val manhattanAuthorDataset = \"timelines_author_features\"\n  private val defaultCacheMaxKeys = 60000\n  private val cacheTTL = 12.hours\n  private val earlyExpiration = 0.2\n\n  val authorKeyDesc = KeyDescriptor(Component(LongInjection), Component0)\n  val authorDatasetKey = authorKeyDesc.withDataset(manhattanAuthorDataset)\n  val authorValDesc = ValueDescriptor(BinaryCompactScalaInjection(AuthorFeatures))\n\n  @Provides\n  @Singleton\n  def timelinesAuthorStitchCache(\n    manhattanReadOnlyEndpoint: ManhattanKVEndpoint,\n    timelinesAuthorFeaturesColumn: TimelinesAuthorFeaturesOnUserClientColumn,\n    stats: StatsReceiver\n  ): StitchCache[JLong, Option[AuthorFeatures]] = {\n\n    val stitchCacheStats =\n      stats\n        .scope(\"direct_ds_source_feature_hydration_module\").scope(\"timelines_author\")\n\n    val stStat = stitchCacheStats.counter(\"readFromStrato-each\")\n    val mhtStat = stitchCacheStats.counter(\"readFromManhattan-each\")\n\n    val timelinesAuthorUnderlyingCall = if (readFromManhattan()) {\n      stitchCacheStats.counter(\"readFromManhattan\").incr()\n      val authorCacheUnderlyingManhattanCall: JLong => Stitch[Option[AuthorFeatures]] = id => {\n        mhtStat.incr()\n        val key = authorDatasetKey.withPkey(id)\n        manhattanReadOnlyEndpoint\n          .get(key = key, valueDesc = authorValDesc).map(_.map(value =>\n            clearUnsedFieldsForAuthorFeature(value.contents)))\n      }\n      authorCacheUnderlyingManhattanCall\n    } else {\n      stitchCacheStats.counter(\"readFromStrato\").incr()\n      val authorCacheUnderlyingStratoCall: JLong => Stitch[Option[AuthorFeatures]] = id => {\n        stStat.incr()\n        val timelinesAuthorFeaturesFetcher = timelinesAuthorFeaturesColumn.fetcher\n        timelinesAuthorFeaturesFetcher\n          .fetch(id).map(result => result.v.map(clearUnsedFieldsForAuthorFeature))\n      }\n      authorCacheUnderlyingStratoCall\n    }\n\n    StitchCache[JLong, Option[AuthorFeatures]](\n      underlyingCall = timelinesAuthorUnderlyingCall,\n      maxCacheSize = defaultCacheMaxKeys,\n      ttl = randomizedTTL(cacheTTL.inSeconds).seconds,\n      statsReceiver = stitchCacheStats\n    )\n\n  }\n\n  // Not adding manhattan since it didn't seem useful for Author Data, we can add in another phab\n  // if deemed helpful\n  @Provides\n  @Singleton\n  def metricCenterUserCountingStitchCache(\n    mcUserCountingFeaturesColumn: McUserCountingOnUserClientColumn,\n    stats: StatsReceiver\n  ): StitchCache[JLong, Option[MCUserCountingFeatures]] = {\n\n    val stitchCacheStats =\n      stats\n        .scope(\"direct_ds_source_feature_hydration_module\").scope(\"mc_user_counting\")\n\n    val stStat = stitchCacheStats.counter(\"readFromStrato-each\")\n    stitchCacheStats.counter(\"readFromStrato\").incr()\n\n    val mcUserCountingCacheUnderlyingCall: JLong => Stitch[Option[MCUserCountingFeatures]] = id => {\n      stStat.incr()\n      val mcUserCountingFeaturesFetcher = mcUserCountingFeaturesColumn.fetcher\n      mcUserCountingFeaturesFetcher.fetch(id).map(_.v)\n    }\n\n    StitchCache[JLong, Option[MCUserCountingFeatures]](\n      underlyingCall = mcUserCountingCacheUnderlyingCall,\n      maxCacheSize = defaultCacheMaxKeys,\n      ttl = randomizedTTL(cacheTTL.inSeconds).seconds,\n      statsReceiver = stitchCacheStats\n    )\n\n  }\n\n  // clear out fields we don't need to save cache space\n  private def clearUnsedFieldsForAuthorFeature(entry: AuthorFeatures): AuthorFeatures = {\n    entry.unsetUserTopics.unsetUserHealth.unsetAuthorCountryCodeAggregates.unsetOriginalAuthorCountryCodeAggregates\n  }\n\n  // To avoid a cache stampede. See https://en.wikipedia.org/wiki/Cache_stampede\n  private def randomizedTTL(ttl: Long): Long = {\n    (ttl - ttl * earlyExpiration * Random.nextDouble()).toLong\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/PreFetchedFeatureSource.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.google.inject.Inject\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.follow_recommendations.common.feature_hydration.adapters.PreFetchedFeatureAdapter\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSource\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSourceId\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\n@Provides\n@Singleton\nclass PreFetchedFeatureSource @Inject() () extends FeatureSource {\n  override def id: FeatureSourceId = FeatureSourceId.PreFetchedFeatureSourceId\n  override def featureContext: FeatureContext = PreFetchedFeatureAdapter.getFeatureContext\n  override def hydrateFeatures(\n    target: HasClientContext\n      with HasPreFetchedFeature\n      with HasParams\n      with HasSimilarToContext\n      with HasDisplayLocation,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Map[CandidateUser, DataRecord]] = {\n    Stitch.value(candidates.map { candidate =>\n      candidate -> PreFetchedFeatureAdapter.adaptToDataRecord((target, candidate))\n    }.toMap)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/UserScoringFeatureSource.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.google.inject.Inject\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSource\nimport com.twitter.follow_recommendations.common.feature_hydration.common.FeatureSourceId\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.DataRecordMerger\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\n/**\n * This source wraps around the separate sources that we hydrate features from\n * @param featureStoreSource        gets features that require a RPC call to feature store\n * @param stratoFeatureHydrationSource    gets features that require a RPC call to strato columns\n * @param clientContextSource       gets features that are already present in the request context\n * @param candidateAlgorithmSource  gets features that are already present from candidate generation\n * @param preFetchedFeatureSource   gets features that were prehydrated (shared in request lifecycle)\n */\n@Provides\n@Singleton\nclass UserScoringFeatureSource @Inject() (\n  featureStoreSource: FeatureStoreSource,\n  featureStoreGizmoduckSource: FeatureStoreGizmoduckSource,\n  featureStorePostNuxAlgorithmSource: FeatureStorePostNuxAlgorithmSource,\n  featureStoreTimelinesAuthorSource: FeatureStoreTimelinesAuthorSource,\n  featureStoreUserMetricCountsSource: FeatureStoreUserMetricCountsSource,\n  clientContextSource: ClientContextSource,\n  candidateAlgorithmSource: CandidateAlgorithmSource,\n  preFetchedFeatureSource: PreFetchedFeatureSource)\n    extends FeatureSource {\n\n  override val id: FeatureSourceId = FeatureSourceId.UserScoringFeatureSourceId\n\n  override val featureContext: FeatureContext = FeatureContext.merge(\n    featureStoreSource.featureContext,\n    featureStoreGizmoduckSource.featureContext,\n    featureStorePostNuxAlgorithmSource.featureContext,\n    featureStoreTimelinesAuthorSource.featureContext,\n    featureStoreUserMetricCountsSource.featureContext,\n    clientContextSource.featureContext,\n    candidateAlgorithmSource.featureContext,\n    preFetchedFeatureSource.featureContext,\n  )\n\n  val sources =\n    Seq(\n      featureStoreSource,\n      featureStorePostNuxAlgorithmSource,\n      featureStoreTimelinesAuthorSource,\n      featureStoreUserMetricCountsSource,\n      featureStoreGizmoduckSource,\n      clientContextSource,\n      candidateAlgorithmSource,\n      preFetchedFeatureSource\n    )\n\n  val dataRecordMerger = new DataRecordMerger\n\n  def hydrateFeatures(\n    target: HasClientContext\n      with HasPreFetchedFeature\n      with HasParams\n      with HasSimilarToContext\n      with HasDisplayLocation,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Map[CandidateUser, DataRecord]] = {\n    Stitch.collect(sources.map(_.hydrateFeatures(target, candidates))).map { featureMaps =>\n      (for {\n        candidate <- candidates\n      } yield {\n        val combinedDataRecord = new DataRecord\n        featureMaps\n          .flatMap(_.get(candidate).toSeq).foreach(dataRecordMerger.merge(combinedDataRecord, _))\n        candidate -> combinedDataRecord\n      }).toMap\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources/Utils.scala",
    "content": "package com.twitter.follow_recommendations.common.feature_hydration.sources\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.IRecordOneToOneAdapter\nimport scala.util.Random\n\n/**\n * Helper functions for FeatureStoreSource operations in FRS are available here.\n */\nobject Utils {\n\n  private val EarlyExpiration = 0.2\n\n  private[common] def adaptAdditionalFeaturesToDataRecord(\n    record: DataRecord,\n    adapterStats: StatsReceiver,\n    featureAdapters: Seq[IRecordOneToOneAdapter[DataRecord]]\n  ): DataRecord = {\n    featureAdapters.foldRight(record) { (adapter, record) =>\n      adapterStats.counter(adapter.getClass.getSimpleName).incr()\n      adapter.adaptToDataRecord(record)\n    }\n  }\n\n  // To avoid a cache stampede. See https://en.wikipedia.org/wiki/Cache_stampede\n  private[common] def randomizedTTL(ttl: Long): Long = {\n    (ttl - ttl * EarlyExpiration * Random.nextDouble()).toLong\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/features/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/features/LocationFeature.scala",
    "content": "package com.twitter.follow_recommendations.common.features\n\nimport com.twitter.follow_recommendations.common.models.GeohashAndCountryCode\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase object LocationFeature\n    extends FeatureWithDefaultOnFailure[PipelineQuery, Option[GeohashAndCountryCode]] {\n  override val defaultValue: Option[GeohashAndCountryCode] = None\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/features/TrackingTokenFeature.scala",
    "content": "package com.twitter.follow_recommendations.common.features\n\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase object TrackingTokenFeature extends FeatureWithDefaultOnFailure[PipelineQuery, Option[Int]] {\n  override val defaultValue: Option[Int] = None\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/features/UserStateFeature.scala",
    "content": "package com.twitter.follow_recommendations.common.features\n\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase object UserStateFeature extends Feature[PipelineQuery, Option[UserState]] {}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/AddressBookMetadata.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\n\n/**\n * contains information if a candidate is from a candidate source generated using the following signals.\n */\ncase class AddressBookMetadata(\n  inForwardPhoneBook: Boolean,\n  inReversePhoneBook: Boolean,\n  inForwardEmailBook: Boolean,\n  inReverseEmailBook: Boolean)\n\nobject AddressBookMetadata {\n\n  val ForwardPhoneBookCandidateSource = CandidateSourceIdentifier(\n    Algorithm.ForwardPhoneBook.toString)\n\n  val ReversePhoneBookCandidateSource = CandidateSourceIdentifier(\n    Algorithm.ReversePhoneBook.toString)\n\n  val ForwardEmailBookCandidateSource = CandidateSourceIdentifier(\n    Algorithm.ForwardEmailBook.toString)\n\n  val ReverseEmailBookCandidateSource = CandidateSourceIdentifier(\n    Algorithm.ReverseEmailBookIbis.toString)\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/AlgorithmType.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\n/**\n * Each candidate source algorithm could be based on one, or more, of the 4 general type of\n * information we have on a user:\n *   1. Social: the user's connections in Twitter's social graph.\n *   2. Geo: the user's geographical information.\n *   3. Interest: information on the user's chosen interests.\n *   4. Activity: information on the user's past activity.\n *\n * Note that an algorithm can fall under more than one of these categories.\n */\nobject AlgorithmType extends Enumeration {\n  type AlgorithmType = Value\n\n  val Social: Value = Value(\"social\")\n  val Geo: Value = Value(\"geo\")\n  val Activity: Value = Value(\"activity\")\n  val Interest: Value = Value(\"interest\")\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core/src/main/scala/com/twitter/timelines/configapi\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/constants\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/model\",\n        \"hermit/hermit-ml/src/main/scala/com/twitter/hermit/ml/models\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"scrooge/scrooge-serializer/src/main/scala\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/wtf/scalding/jobs/strong_tie_prediction\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_rpc-scala\",\n        \"src/thrift/com/twitter/timelines/author_features/user_health:thrift-scala\",\n        \"user-signal-service/thrift/src/main/thrift:thrift-scala\",\n        \"util/util-slf4j-api/src/main/scala/com/twitter/util/logging\",\n    ],\n    exports = [\n        \"util/util-slf4j-api/src/main/scala/com/twitter/util/logging\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/CandidateUser.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport com.twitter.hermit.constants.AlgorithmFeedbackTokens\nimport com.twitter.ml.api.thriftscala.{DataRecord => TDataRecord}\nimport com.twitter.ml.api.util.ScalaToJavaDataRecordConversions\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\n\ntrait FollowableEntity extends UniversalNoun[Long]\n\ntrait Recommendation\n    extends FollowableEntity\n    with HasReason\n    with HasAdMetadata\n    with HasTrackingToken {\n  val score: Option[Double]\n\n  def toThrift: t.Recommendation\n\n  def toOfflineThrift: offline.OfflineRecommendation\n}\n\ncase class CandidateUser(\n  override val id: Long,\n  override val score: Option[Double] = None,\n  override val reason: Option[Reason] = None,\n  override val userCandidateSourceDetails: Option[UserCandidateSourceDetails] = None,\n  override val adMetadata: Option[AdMetadata] = None,\n  override val trackingToken: Option[TrackingToken] = None,\n  override val dataRecord: Option[RichDataRecord] = None,\n  override val scores: Option[Scores] = None,\n  override val infoPerRankingStage: Option[scala.collection.Map[String, RankingInfo]] = None,\n  override val params: Params = Params.Invalid,\n  override val engagements: Seq[EngagementType] = Nil,\n  override val recommendationFlowIdentifier: Option[String] = None)\n    extends Recommendation\n    with HasUserCandidateSourceDetails\n    with HasDataRecord\n    with HasScores\n    with HasParams\n    with HasEngagements\n    with HasRecommendationFlowIdentifier\n    with HasInfoPerRankingStage {\n\n  val rankerIdsStr: Option[Seq[String]] = {\n    val strs = scores.map(_.scores.flatMap(_.rankerId.map(_.toString)))\n    if (strs.exists(_.nonEmpty)) strs else None\n  }\n\n  val thriftDataRecord: Option[TDataRecord] = for {\n    richDataRecord <- dataRecord\n    dr <- richDataRecord.dataRecord\n  } yield {\n    ScalaToJavaDataRecordConversions.javaDataRecord2ScalaDataRecord(dr)\n  }\n\n  val toOfflineUserThrift: offline.OfflineUserRecommendation = {\n    val scoringDetails =\n      if (userCandidateSourceDetails.isEmpty && score.isEmpty && thriftDataRecord.isEmpty) {\n        None\n      } else {\n        Some(\n          offline.ScoringDetails(\n            candidateSourceDetails = userCandidateSourceDetails.map(_.toOfflineThrift),\n            score = score,\n            dataRecord = thriftDataRecord,\n            rankerIds = rankerIdsStr,\n            infoPerRankingStage = infoPerRankingStage.map(_.mapValues(_.toOfflineThrift))\n          )\n        )\n      }\n    offline\n      .OfflineUserRecommendation(\n        id,\n        reason.map(_.toOfflineThrift),\n        adMetadata.map(_.adImpression),\n        trackingToken.map(_.toOfflineThrift),\n        scoringDetails = scoringDetails\n      )\n  }\n\n  override val toOfflineThrift: offline.OfflineRecommendation =\n    offline.OfflineRecommendation.User(toOfflineUserThrift)\n\n  val toUserThrift: t.UserRecommendation = {\n    val scoringDetails =\n      if (userCandidateSourceDetails.isEmpty && score.isEmpty && thriftDataRecord.isEmpty && scores.isEmpty) {\n        None\n      } else {\n        Some(\n          t.ScoringDetails(\n            candidateSourceDetails = userCandidateSourceDetails.map(_.toThrift),\n            score = score,\n            dataRecord = thriftDataRecord,\n            rankerIds = rankerIdsStr,\n            debugDataRecord = dataRecord.flatMap(_.debugDataRecord),\n            infoPerRankingStage = infoPerRankingStage.map(_.mapValues(_.toThrift))\n          )\n        )\n      }\n    t.UserRecommendation(\n      userId = id,\n      reason = reason.map(_.toThrift),\n      adImpression = adMetadata.map(_.adImpression),\n      trackingInfo = trackingToken.map(TrackingToken.serialize),\n      scoringDetails = scoringDetails,\n      recommendationFlowIdentifier = recommendationFlowIdentifier\n    )\n  }\n\n  override val toThrift: t.Recommendation =\n    t.Recommendation.User(toUserThrift)\n\n  def setFollowProof(followProofOpt: Option[FollowProof]): CandidateUser = {\n    this.copy(\n      reason = reason\n        .map { reason =>\n          reason.copy(\n            accountProof = reason.accountProof\n              .map { accountProof =>\n                accountProof.copy(followProof = followProofOpt)\n              }.orElse(Some(AccountProof(followProof = followProofOpt)))\n          )\n        }.orElse(Some(Reason(Some(AccountProof(followProof = followProofOpt)))))\n    )\n  }\n\n  def addScore(score: Score): CandidateUser = {\n    val newScores = scores match {\n      case Some(existingScores) => existingScores.copy(scores = existingScores.scores :+ score)\n      case None => Scores(Seq(score))\n    }\n    this.copy(scores = Some(newScores))\n  }\n}\n\nobject CandidateUser {\n  val DefaultCandidateScore = 1.0\n\n  // for converting candidate in ScoringUserRequest\n  def fromUserRecommendation(candidate: t.UserRecommendation): CandidateUser = {\n    // we only use the primary candidate source for now\n    val userCandidateSourceDetails = for {\n      scoringDetails <- candidate.scoringDetails\n      candidateSourceDetails <- scoringDetails.candidateSourceDetails\n    } yield UserCandidateSourceDetails(\n      primaryCandidateSource = candidateSourceDetails.primarySource\n        .flatMap(AlgorithmFeedbackTokens.TokenToAlgorithmMap.get).map { algo =>\n          CandidateSourceIdentifier(algo.toString)\n        },\n      candidateSourceScores = fromThriftScoreMap(candidateSourceDetails.candidateSourceScores),\n      candidateSourceRanks = fromThriftRankMap(candidateSourceDetails.candidateSourceRanks),\n      addressBookMetadata = None\n    )\n    CandidateUser(\n      id = candidate.userId,\n      score = candidate.scoringDetails.flatMap(_.score),\n      reason = candidate.reason.map(Reason.fromThrift),\n      userCandidateSourceDetails = userCandidateSourceDetails,\n      trackingToken = candidate.trackingInfo.map(TrackingToken.deserialize),\n      recommendationFlowIdentifier = candidate.recommendationFlowIdentifier,\n      infoPerRankingStage = candidate.scoringDetails.flatMap(\n        _.infoPerRankingStage.map(_.mapValues(RankingInfo.fromThrift)))\n    )\n  }\n\n  def fromThriftScoreMap(\n    thriftMapOpt: Option[scala.collection.Map[String, Double]]\n  ): Map[CandidateSourceIdentifier, Option[Double]] = {\n    (for {\n      thriftMap <- thriftMapOpt.toSeq\n      (algoName, score) <- thriftMap.toSeq\n    } yield {\n      CandidateSourceIdentifier(algoName) -> Some(score)\n    }).toMap\n  }\n\n  def fromThriftRankMap(\n    thriftMapOpt: Option[scala.collection.Map[String, Int]]\n  ): Map[CandidateSourceIdentifier, Int] = {\n    (for {\n      thriftMap <- thriftMapOpt.toSeq\n      (algoName, rank) <- thriftMap.toSeq\n    } yield {\n      CandidateSourceIdentifier(algoName) -> rank\n    }).toMap\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/ClientContextConverter.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.follow_recommendations.{thriftscala => frs}\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\n\nobject ClientContextConverter {\n  def toFRSOfflineClientContextThrift(\n    productMixerClientContext: ClientContext\n  ): offline.OfflineClientContext =\n    offline.OfflineClientContext(\n      productMixerClientContext.userId,\n      productMixerClientContext.guestId,\n      productMixerClientContext.appId,\n      productMixerClientContext.countryCode,\n      productMixerClientContext.languageCode,\n      productMixerClientContext.guestIdAds,\n      productMixerClientContext.guestIdMarketing\n    )\n\n  def fromThrift(clientContext: frs.ClientContext): ClientContext = ClientContext(\n    userId = clientContext.userId,\n    guestId = clientContext.guestId,\n    appId = clientContext.appId,\n    ipAddress = clientContext.ipAddress,\n    userAgent = clientContext.userAgent,\n    countryCode = clientContext.countryCode,\n    languageCode = clientContext.languageCode,\n    isTwoffice = clientContext.isTwoffice,\n    userRoles = clientContext.userRoles.map(_.toSet),\n    deviceId = clientContext.deviceId,\n    guestIdAds = clientContext.guestIdAds,\n    guestIdMarketing = clientContext.guestIdMarketing,\n    mobileDeviceId = None,\n    mobileDeviceAdId = None,\n    limitAdTracking = None\n  )\n\n  def toThrift(clientContext: ClientContext): frs.ClientContext = frs.ClientContext(\n    userId = clientContext.userId,\n    guestId = clientContext.guestIdAds,\n    appId = clientContext.appId,\n    ipAddress = clientContext.ipAddress,\n    userAgent = clientContext.userAgent,\n    countryCode = clientContext.countryCode,\n    languageCode = clientContext.languageCode,\n    isTwoffice = clientContext.isTwoffice,\n    userRoles = clientContext.userRoles,\n    deviceId = clientContext.deviceId,\n    guestIdAds = clientContext.guestIdAds,\n    guestIdMarketing = clientContext.guestIdMarketing\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/DisplayLocation.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.adserver.thriftscala.{DisplayLocation => AdDisplayLocation}\nimport com.twitter.follow_recommendations.logging.thriftscala.{\n  OfflineDisplayLocation => TOfflineDisplayLocation\n}\nimport com.twitter.follow_recommendations.thriftscala.{DisplayLocation => TDisplayLocation}\n\nsealed trait DisplayLocation {\n  def toThrift: TDisplayLocation\n\n  def toOfflineThrift: TOfflineDisplayLocation\n\n  def toFsName: String\n\n  // corresponding display location in adserver if available\n  // make sure to be consistent with the definition here\n  def toAdDisplayLocation: Option[AdDisplayLocation] = None\n}\n\n/**\n * Make sure you add the new DL to the following files and redeploy our attribution jobs\n *  - follow-recommendations-service/thrift/src/main/thrift/display_location.thrift\n *  - follow-recommendations-service/thrift/src/main/thrift/logging/display_location.thrift\n *  - follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/DisplayLocation.scala\n */\n\nobject DisplayLocation {\n\n  case object ProfileSidebar extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.ProfileSidebar\n    override val toOfflineThrift: TOfflineDisplayLocation = TOfflineDisplayLocation.ProfileSidebar\n    override val toFsName: String = \"ProfileSidebar\"\n\n    override val toAdDisplayLocation: Option[AdDisplayLocation] = Some(\n      AdDisplayLocation.ProfileAccountsSidebar\n    )\n  }\n\n  case object HomeTimeline extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.HomeTimeline\n    override val toOfflineThrift: TOfflineDisplayLocation = TOfflineDisplayLocation.HomeTimeline\n    override val toFsName: String = \"HomeTimeline\"\n    override val toAdDisplayLocation: Option[AdDisplayLocation] = Some(\n      // it is based on the logic that HTL DL should correspond to Sidebar:\n      AdDisplayLocation.WtfSidebar\n    )\n  }\n\n  case object ReactiveFollow extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.ReactiveFollow\n    override val toOfflineThrift: TOfflineDisplayLocation = TOfflineDisplayLocation.ReactiveFollow\n    override val toFsName: String = \"ReactiveFollow\"\n  }\n\n  case object ExploreTab extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.ExploreTab\n    override val toOfflineThrift: TOfflineDisplayLocation = TOfflineDisplayLocation.ExploreTab\n    override val toFsName: String = \"ExploreTab\"\n  }\n\n  case object MagicRecs extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.MagicRecs\n    override val toOfflineThrift: TOfflineDisplayLocation = TOfflineDisplayLocation.MagicRecs\n    override val toFsName: String = \"MagicRecs\"\n  }\n\n  case object AbUploadInjection extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.AbUploadInjection\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.AbUploadInjection\n    override val toFsName: String = \"AbUploadInjection\"\n  }\n\n  case object RuxLandingPage extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.RuxLandingPage\n    override val toOfflineThrift: TOfflineDisplayLocation = TOfflineDisplayLocation.RuxLandingPage\n    override val toFsName: String = \"RuxLandingPage\"\n  }\n\n  case object ProfileBonusFollow extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.ProfileBonusFollow\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.ProfileBonusFollow\n    override val toFsName: String = \"ProfileBonusFollow\"\n  }\n\n  case object ElectionExploreWtf extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.ElectionExploreWtf\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.ElectionExploreWtf\n    override val toFsName: String = \"ElectionExploreWtf\"\n  }\n\n  case object ClusterFollow extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.ClusterFollow\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.ClusterFollow\n    override val toFsName: String = \"ClusterFollow\"\n    override val toAdDisplayLocation: Option[AdDisplayLocation] = Some(\n      AdDisplayLocation.ClusterFollow\n    )\n  }\n\n  case object HtlBonusFollow extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.HtlBonusFollow\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.HtlBonusFollow\n    override val toFsName: String = \"HtlBonusFollow\"\n  }\n\n  case object TopicLandingPageHeader extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.TopicLandingPageHeader\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.TopicLandingPageHeader\n    override val toFsName: String = \"TopicLandingPageHeader\"\n  }\n\n  case object NewUserSarusBackfill extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.NewUserSarusBackfill\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.NewUserSarusBackfill\n    override val toFsName: String = \"NewUserSarusBackfill\"\n  }\n\n  case object NuxPymk extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.NuxPymk\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.NuxPymk\n    override val toFsName: String = \"NuxPymk\"\n  }\n\n  case object NuxInterests extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.NuxInterests\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.NuxInterests\n    override val toFsName: String = \"NuxInterests\"\n  }\n\n  case object NuxTopicBonusFollow extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.NuxTopicBonusFollow\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.NuxTopicBonusFollow\n    override val toFsName: String = \"NuxTopicBonusFollow\"\n  }\n\n  case object Sidebar extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.Sidebar\n    override val toOfflineThrift: TOfflineDisplayLocation = TOfflineDisplayLocation.Sidebar\n    override val toFsName: String = \"Sidebar\"\n\n    override val toAdDisplayLocation: Option[AdDisplayLocation] = Some(\n      AdDisplayLocation.WtfSidebar\n    )\n  }\n\n  case object CampaignForm extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.CampaignForm\n    override val toOfflineThrift: TOfflineDisplayLocation = TOfflineDisplayLocation.CampaignForm\n    override val toFsName: String = \"CampaignForm\"\n  }\n\n  case object ProfileTopFollowers extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.ProfileTopFollowers\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.ProfileTopFollowers\n    override val toFsName: String = \"ProfileTopFollowers\"\n  }\n\n  case object ProfileTopFollowing extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.ProfileTopFollowing\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.ProfileTopFollowing\n    override val toFsName: String = \"ProfileTopFollowing\"\n  }\n\n  case object RuxPymk extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.RuxPymk\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.RuxPymk\n    override val toFsName: String = \"RuxPymk\"\n  }\n\n  case object IndiaCovid19CuratedAccountsWtf extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.IndiaCovid19CuratedAccountsWtf\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.IndiaCovid19CuratedAccountsWtf\n    override val toFsName: String = \"IndiaCovid19CuratedAccountsWtf\"\n  }\n\n  case object PeoplePlusPlus extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.PeoplePlusPlus\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.PeoplePlusPlus\n    override val toFsName: String = \"PeoplePlusPlus\"\n  }\n\n  case object TweetNotificationRecs extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.TweetNotificationRecs\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.TweetNotificationRecs\n    override val toFsName: String = \"TweetNotificationRecs\"\n  }\n\n  case object ProfileDeviceFollow extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.ProfileDeviceFollow\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.ProfileDeviceFollow\n    override val toFsName: String = \"ProfileDeviceFollow\"\n  }\n\n  case object RecosBackfill extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.RecosBackfill\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.RecosBackfill\n    override val toFsName: String = \"RecosBackfill\"\n  }\n\n  case object HtlSpaceHosts extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.HtlSpaceHosts\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.HtlSpaceHosts\n    override val toFsName: String = \"HtlSpaceHosts\"\n  }\n\n  case object PostNuxFollowTask extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.PostNuxFollowTask\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.PostNuxFollowTask\n    override val toFsName: String = \"PostNuxFollowTask\"\n  }\n\n  case object TopicLandingPage extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.TopicLandingPage\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.TopicLandingPage\n    override val toFsName: String = \"TopicLandingPage\"\n  }\n\n  case object UserTypeaheadPrefetch extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.UserTypeaheadPrefetch\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.UserTypeaheadPrefetch\n    override val toFsName: String = \"UserTypeaheadPrefetch\"\n  }\n\n  case object HomeTimelineRelatableAccounts extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.HomeTimelineRelatableAccounts\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.HomeTimelineRelatableAccounts\n    override val toFsName: String = \"HomeTimelineRelatableAccounts\"\n  }\n\n  case object NuxGeoCategory extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.NuxGeoCategory\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.NuxGeoCategory\n    override val toFsName: String = \"NuxGeoCategory\"\n  }\n\n  case object NuxInterestsCategory extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.NuxInterestsCategory\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.NuxInterestsCategory\n    override val toFsName: String = \"NuxInterestsCategory\"\n  }\n\n  case object TopArticles extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.TopArticles\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.TopArticles\n    override val toFsName: String = \"TopArticles\"\n  }\n\n  case object NuxPymkCategory extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.NuxPymkCategory\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.NuxPymkCategory\n    override val toFsName: String = \"NuxPymkCategory\"\n  }\n\n  case object HomeTimelineTweetRecs extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.HomeTimelineTweetRecs\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.HomeTimelineTweetRecs\n    override val toFsName: String = \"HomeTimelineTweetRecs\"\n  }\n\n  case object HtlBulkFriendFollows extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.HtlBulkFriendFollows\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.HtlBulkFriendFollows\n    override val toFsName: String = \"HtlBulkFriendFollows\"\n  }\n\n  case object NuxAutoFollow extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.NuxAutoFollow\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.NuxAutoFollow\n    override val toFsName: String = \"NuxAutoFollow\"\n  }\n\n  case object SearchBonusFollow extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.SearchBonusFollow\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.SearchBonusFollow\n    override val toFsName: String = \"SearchBonusFollow\"\n  }\n\n  case object ContentRecommender extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.ContentRecommender\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.ContentRecommender\n    override val toFsName: String = \"ContentRecommender\"\n  }\n\n  case object HomeTimelineReverseChron extends DisplayLocation {\n    override val toThrift: TDisplayLocation = TDisplayLocation.HomeTimelineReverseChron\n    override val toOfflineThrift: TOfflineDisplayLocation =\n      TOfflineDisplayLocation.HomeTimelineReverseChron\n    override val toFsName: String = \"HomeTimelineReverseChron\"\n  }\n\n  def fromThrift(displayLocation: TDisplayLocation): DisplayLocation = displayLocation match {\n    case TDisplayLocation.ProfileSidebar => ProfileSidebar\n    case TDisplayLocation.HomeTimeline => HomeTimeline\n    case TDisplayLocation.MagicRecs => MagicRecs\n    case TDisplayLocation.AbUploadInjection => AbUploadInjection\n    case TDisplayLocation.RuxLandingPage => RuxLandingPage\n    case TDisplayLocation.ProfileBonusFollow => ProfileBonusFollow\n    case TDisplayLocation.ElectionExploreWtf => ElectionExploreWtf\n    case TDisplayLocation.ClusterFollow => ClusterFollow\n    case TDisplayLocation.HtlBonusFollow => HtlBonusFollow\n    case TDisplayLocation.ReactiveFollow => ReactiveFollow\n    case TDisplayLocation.TopicLandingPageHeader => TopicLandingPageHeader\n    case TDisplayLocation.NewUserSarusBackfill => NewUserSarusBackfill\n    case TDisplayLocation.NuxPymk => NuxPymk\n    case TDisplayLocation.NuxInterests => NuxInterests\n    case TDisplayLocation.NuxTopicBonusFollow => NuxTopicBonusFollow\n    case TDisplayLocation.ExploreTab => ExploreTab\n    case TDisplayLocation.Sidebar => Sidebar\n    case TDisplayLocation.CampaignForm => CampaignForm\n    case TDisplayLocation.ProfileTopFollowers => ProfileTopFollowers\n    case TDisplayLocation.ProfileTopFollowing => ProfileTopFollowing\n    case TDisplayLocation.RuxPymk => RuxPymk\n    case TDisplayLocation.IndiaCovid19CuratedAccountsWtf => IndiaCovid19CuratedAccountsWtf\n    case TDisplayLocation.PeoplePlusPlus => PeoplePlusPlus\n    case TDisplayLocation.TweetNotificationRecs => TweetNotificationRecs\n    case TDisplayLocation.ProfileDeviceFollow => ProfileDeviceFollow\n    case TDisplayLocation.RecosBackfill => RecosBackfill\n    case TDisplayLocation.HtlSpaceHosts => HtlSpaceHosts\n    case TDisplayLocation.PostNuxFollowTask => PostNuxFollowTask\n    case TDisplayLocation.TopicLandingPage => TopicLandingPage\n    case TDisplayLocation.UserTypeaheadPrefetch => UserTypeaheadPrefetch\n    case TDisplayLocation.HomeTimelineRelatableAccounts => HomeTimelineRelatableAccounts\n    case TDisplayLocation.NuxGeoCategory => NuxGeoCategory\n    case TDisplayLocation.NuxInterestsCategory => NuxInterestsCategory\n    case TDisplayLocation.TopArticles => TopArticles\n    case TDisplayLocation.NuxPymkCategory => NuxPymkCategory\n    case TDisplayLocation.HomeTimelineTweetRecs => HomeTimelineTweetRecs\n    case TDisplayLocation.HtlBulkFriendFollows => HtlBulkFriendFollows\n    case TDisplayLocation.NuxAutoFollow => NuxAutoFollow\n    case TDisplayLocation.SearchBonusFollow => SearchBonusFollow\n    case TDisplayLocation.ContentRecommender => ContentRecommender\n    case TDisplayLocation.HomeTimelineReverseChron => HomeTimelineReverseChron\n    case TDisplayLocation.EnumUnknownDisplayLocation(i) =>\n      throw new UnknownDisplayLocationException(\n        s\"Unknown display location thrift enum with value: ${i}\")\n  }\n\n  def fromOfflineThrift(displayLocation: TOfflineDisplayLocation): DisplayLocation =\n    displayLocation match {\n      case TOfflineDisplayLocation.ProfileSidebar => ProfileSidebar\n      case TOfflineDisplayLocation.HomeTimeline => HomeTimeline\n      case TOfflineDisplayLocation.MagicRecs => MagicRecs\n      case TOfflineDisplayLocation.AbUploadInjection => AbUploadInjection\n      case TOfflineDisplayLocation.RuxLandingPage => RuxLandingPage\n      case TOfflineDisplayLocation.ProfileBonusFollow => ProfileBonusFollow\n      case TOfflineDisplayLocation.ElectionExploreWtf => ElectionExploreWtf\n      case TOfflineDisplayLocation.ClusterFollow => ClusterFollow\n      case TOfflineDisplayLocation.HtlBonusFollow => HtlBonusFollow\n      case TOfflineDisplayLocation.TopicLandingPageHeader => TopicLandingPageHeader\n      case TOfflineDisplayLocation.NewUserSarusBackfill => NewUserSarusBackfill\n      case TOfflineDisplayLocation.NuxPymk => NuxPymk\n      case TOfflineDisplayLocation.NuxInterests => NuxInterests\n      case TOfflineDisplayLocation.NuxTopicBonusFollow => NuxTopicBonusFollow\n      case TOfflineDisplayLocation.ExploreTab => ExploreTab\n      case TOfflineDisplayLocation.ReactiveFollow => ReactiveFollow\n      case TOfflineDisplayLocation.Sidebar => Sidebar\n      case TOfflineDisplayLocation.CampaignForm => CampaignForm\n      case TOfflineDisplayLocation.ProfileTopFollowers => ProfileTopFollowers\n      case TOfflineDisplayLocation.ProfileTopFollowing => ProfileTopFollowing\n      case TOfflineDisplayLocation.RuxPymk => RuxPymk\n      case TOfflineDisplayLocation.IndiaCovid19CuratedAccountsWtf => IndiaCovid19CuratedAccountsWtf\n      case TOfflineDisplayLocation.PeoplePlusPlus => PeoplePlusPlus\n      case TOfflineDisplayLocation.TweetNotificationRecs => TweetNotificationRecs\n      case TOfflineDisplayLocation.ProfileDeviceFollow => ProfileDeviceFollow\n      case TOfflineDisplayLocation.RecosBackfill => RecosBackfill\n      case TOfflineDisplayLocation.HtlSpaceHosts => HtlSpaceHosts\n      case TOfflineDisplayLocation.PostNuxFollowTask => PostNuxFollowTask\n      case TOfflineDisplayLocation.TopicLandingPage => TopicLandingPage\n      case TOfflineDisplayLocation.UserTypeaheadPrefetch => UserTypeaheadPrefetch\n      case TOfflineDisplayLocation.HomeTimelineRelatableAccounts => HomeTimelineRelatableAccounts\n      case TOfflineDisplayLocation.NuxGeoCategory => NuxGeoCategory\n      case TOfflineDisplayLocation.NuxInterestsCategory => NuxInterestsCategory\n      case TOfflineDisplayLocation.TopArticles => TopArticles\n      case TOfflineDisplayLocation.NuxPymkCategory => NuxPymkCategory\n      case TOfflineDisplayLocation.HomeTimelineTweetRecs => HomeTimelineTweetRecs\n      case TOfflineDisplayLocation.HtlBulkFriendFollows => HtlBulkFriendFollows\n      case TOfflineDisplayLocation.NuxAutoFollow => NuxAutoFollow\n      case TOfflineDisplayLocation.SearchBonusFollow => SearchBonusFollow\n      case TOfflineDisplayLocation.ContentRecommender => ContentRecommender\n      case TOfflineDisplayLocation.HomeTimelineReverseChron => HomeTimelineReverseChron\n      case TOfflineDisplayLocation.EnumUnknownOfflineDisplayLocation(i) =>\n        throw new UnknownDisplayLocationException(\n          s\"Unknown offline display location thrift enum with value: ${i}\")\n    }\n}\n\nclass UnknownDisplayLocationException(message: String) extends Exception(message)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/EngagementType.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.thriftscala.{EngagementType => TEngagementType}\nimport com.twitter.follow_recommendations.logging.thriftscala.{\n  EngagementType => OfflineEngagementType\n}\nsealed trait EngagementType {\n  def toThrift: TEngagementType\n  def toOfflineThrift: OfflineEngagementType\n}\n\nobject EngagementType {\n  object Click extends EngagementType {\n    override val toThrift: TEngagementType = TEngagementType.Click\n\n    override val toOfflineThrift: OfflineEngagementType = OfflineEngagementType.Click\n  }\n  object Like extends EngagementType {\n    override val toThrift: TEngagementType = TEngagementType.Like\n\n    override val toOfflineThrift: OfflineEngagementType = OfflineEngagementType.Like\n  }\n  object Mention extends EngagementType {\n    override val toThrift: TEngagementType = TEngagementType.Mention\n\n    override val toOfflineThrift: OfflineEngagementType = OfflineEngagementType.Mention\n  }\n  object Retweet extends EngagementType {\n    override val toThrift: TEngagementType = TEngagementType.Retweet\n\n    override val toOfflineThrift: OfflineEngagementType = OfflineEngagementType.Retweet\n  }\n  object ProfileView extends EngagementType {\n    override val toThrift: TEngagementType = TEngagementType.ProfileView\n\n    override val toOfflineThrift: OfflineEngagementType = OfflineEngagementType.ProfileView\n  }\n\n  def fromThrift(engagementType: TEngagementType): EngagementType = engagementType match {\n    case TEngagementType.Click => Click\n    case TEngagementType.Like => Like\n    case TEngagementType.Mention => Mention\n    case TEngagementType.Retweet => Retweet\n    case TEngagementType.ProfileView => ProfileView\n    case TEngagementType.EnumUnknownEngagementType(i) =>\n      throw new UnknownEngagementTypeException(\n        s\"Unknown engagement type thrift enum with value: ${i}\")\n  }\n\n  def fromOfflineThrift(engagementType: OfflineEngagementType): EngagementType =\n    engagementType match {\n      case OfflineEngagementType.Click => Click\n      case OfflineEngagementType.Like => Like\n      case OfflineEngagementType.Mention => Mention\n      case OfflineEngagementType.Retweet => Retweet\n      case OfflineEngagementType.ProfileView => ProfileView\n      case OfflineEngagementType.EnumUnknownEngagementType(i) =>\n        throw new UnknownEngagementTypeException(\n          s\"Unknown engagement type offline thrift enum with value: ${i}\")\n    }\n}\nclass UnknownEngagementTypeException(message: String) extends Exception(message)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/FilterReason.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nsealed trait FilterReason {\n  def reason: String\n}\n\nobject FilterReason {\n\n  case object NoReason extends FilterReason {\n    override val reason: String = \"no_reason\"\n  }\n\n  case class ParamReason(paramName: String) extends FilterReason {\n    override val reason: String = s\"param_$paramName\"\n  }\n\n  case object ExcludedId extends FilterReason {\n    override val reason: String = \"excluded_id_from_request\"\n  }\n\n  case object ProfileSidebarBlacklist extends FilterReason {\n    override val reason: String = \"profile_sidebar_blacklisted_id\"\n  }\n\n  case object CuratedAccountsCompetitorList extends FilterReason {\n    override val reason: String = \"curated_blacklisted_id\"\n  }\n\n  case class InvalidRelationshipTypes(relationshipTypes: String) extends FilterReason {\n    override val reason: String = s\"invalid_relationship_types $relationshipTypes\"\n  }\n\n  case object ProfileId extends FilterReason {\n    override val reason: String = \"candidate_has_same_id_as_profile\"\n  }\n\n  case object DismissedId extends FilterReason {\n    override val reason: String = s\"dismissed_candidate\"\n  }\n\n  case object OptedOutId extends FilterReason {\n    override val reason: String = s\"candidate_opted_out_from_criteria_in_request\"\n  }\n\n  // gizmoduck predicates\n  case object NoUser extends FilterReason {\n    override val reason: String = \"no_user_result_from_gizmoduck\"\n  }\n\n  case object AddressBookUndiscoverable extends FilterReason {\n    override val reason: String = \"not_discoverable_via_address_book\"\n  }\n\n  case object PhoneBookUndiscoverable extends FilterReason {\n    override val reason: String = \"not_discoverable_via_phone_book\"\n  }\n\n  case object Deactivated extends FilterReason {\n    override val reason: String = \"deactivated\"\n  }\n\n  case object Suspended extends FilterReason {\n    override val reason: String = \"suspended\"\n  }\n\n  case object Restricted extends FilterReason {\n    override val reason: String = \"restricted\"\n  }\n\n  case object NsfwUser extends FilterReason {\n    override val reason: String = \"nsfwUser\"\n  }\n\n  case object NsfwAdmin extends FilterReason {\n    override val reason: String = \"nsfwAdmin\"\n  }\n\n  case object HssSignal extends FilterReason {\n    override val reason: String = \"hssSignal\"\n  }\n\n  case object IsProtected extends FilterReason {\n    override val reason: String = \"isProtected\"\n  }\n\n  case class CountryTakedown(countryCode: String) extends FilterReason {\n    override val reason: String = s\"takedown_in_$countryCode\"\n  }\n\n  case object Blink extends FilterReason {\n    override val reason: String = \"blink\"\n  }\n\n  case object AlreadyFollowed extends FilterReason {\n    override val reason: String = \"already_followed\"\n  }\n\n  case object InvalidRelationship extends FilterReason {\n    override val reason: String = \"invalid_relationship\"\n  }\n\n  case object NotFollowingTargetUser extends FilterReason {\n    override val reason: String = \"not_following_target_user\"\n  }\n\n  case object CandidateSideHoldback extends FilterReason {\n    override val reason: String = \"candidate_side_holdback\"\n  }\n\n  case object Inactive extends FilterReason {\n    override val reason: String = \"inactive\"\n  }\n\n  case object MissingRecommendabilityData extends FilterReason {\n    override val reason: String = \"missing_recommendability_data\"\n  }\n\n  case object HighTweetVelocity extends FilterReason {\n    override val reason: String = \"high_tweet_velocity\"\n  }\n\n  case object AlreadyRecommended extends FilterReason {\n    override val reason: String = \"already_recommended\"\n  }\n\n  case object MinStateNotMet extends FilterReason {\n    override val reason: String = \"min_state_user_not_met\"\n  }\n\n  case object FailOpen extends FilterReason {\n    override val reason: String = \"fail_open\"\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/FlowContext.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.follow_recommendations.{thriftscala => t}\n\ncase class FlowContext(steps: Seq[RecommendationStep]) {\n\n  def toThrift: t.FlowContext = t.FlowContext(steps = steps.map(_.toThrift))\n\n  def toOfflineThrift: offline.OfflineFlowContext =\n    offline.OfflineFlowContext(steps = steps.map(_.toOfflineThrift))\n}\n\nobject FlowContext {\n\n  def fromThrift(flowContext: t.FlowContext): FlowContext = {\n    FlowContext(steps = flowContext.steps.map(RecommendationStep.fromThrift))\n  }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/FlowRecommendation.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.follow_recommendations.{thriftscala => t}\n\ncase class FlowRecommendation(userId: Long) {\n\n  def toThrift: t.FlowRecommendation =\n    t.FlowRecommendation(userId = userId)\n\n  def toOfflineThrift: offline.OfflineFlowRecommendation =\n    offline.OfflineFlowRecommendation(userId = userId)\n\n}\n\nobject FlowRecommendation {\n  def fromThrift(flowRecommendation: t.FlowRecommendation): FlowRecommendation = {\n    FlowRecommendation(\n      userId = flowRecommendation.userId\n    )\n  }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/GeohashAndCountryCode.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ncase class GeohashAndCountryCode(geohash: Option[String], countryCode: Option[String])\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasAdMetadata.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.adserver.{thriftscala => t}\n\ncase class AdMetadata(\n  insertPosition: Int,\n  // use original ad impression info to avoid losing data in domain model translations\n  adImpression: t.AdImpression)\n\ntrait HasAdMetadata {\n\n  def adMetadata: Option[AdMetadata]\n\n  def adImpression: Option[t.AdImpression] = {\n    adMetadata.map(_.adImpression)\n  }\n\n  def insertPosition: Option[Int] = {\n    adMetadata.map(_.insertPosition)\n  }\n\n  def isPromotedAccount: Boolean = adMetadata.isDefined\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasByfSeedUserIds.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasByfSeedUserIds {\n  def byfSeedUserIds: Option[Seq[Long]]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasDataRecord.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.thriftscala.DebugDataRecord\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\nimport scala.collection.convert.ImplicitConversions._\n\n// contains the standard dataRecord struct, and the debug version if required\ncase class RichDataRecord(\n  dataRecord: Option[DataRecord] = None,\n  debugDataRecord: Option[DebugDataRecord] = None,\n)\n\ntrait HasDataRecord extends Logging {\n  def dataRecord: Option[RichDataRecord]\n\n  def toDebugDataRecord(dr: DataRecord, featureContext: FeatureContext): DebugDataRecord = {\n\n    val binaryFeatures: Option[Set[String]] = if (dr.isSetBinaryFeatures) {\n      Some(dr.getBinaryFeatures.flatMap { id =>\n        Try(featureContext.getFeature(id).getFeatureName).toOption\n      }.toSet)\n    } else None\n\n    val continuousFeatures: Option[Map[String, Double]] = if (dr.isSetContinuousFeatures) {\n      Some(dr.getContinuousFeatures.flatMap {\n        case (id, value) =>\n          Try(featureContext.getFeature(id).getFeatureName).toOption.map { id =>\n            id -> value.toDouble\n          }\n      }.toMap)\n    } else None\n\n    val discreteFeatures: Option[Map[String, Long]] = if (dr.isSetDiscreteFeatures) {\n      Some(dr.getDiscreteFeatures.flatMap {\n        case (id, value) =>\n          Try(featureContext.getFeature(id).getFeatureName).toOption.map { id =>\n            id -> value.toLong\n          }\n      }.toMap)\n    } else None\n\n    val stringFeatures: Option[Map[String, String]] = if (dr.isSetStringFeatures) {\n      Some(dr.getStringFeatures.flatMap {\n        case (id, value) =>\n          Try(featureContext.getFeature(id).getFeatureName).toOption.map { id =>\n            id -> value\n          }\n      }.toMap)\n    } else None\n\n    val sparseBinaryFeatures: Option[Map[String, Set[String]]] = if (dr.isSetSparseBinaryFeatures) {\n      Some(dr.getSparseBinaryFeatures.flatMap {\n        case (id, values) =>\n          Try(featureContext.getFeature(id).getFeatureName).toOption.map { id =>\n            id -> values.toSet\n          }\n      }.toMap)\n    } else None\n\n    val sparseContinuousFeatures: Option[Map[String, Map[String, Double]]] =\n      if (dr.isSetSparseContinuousFeatures) {\n        Some(dr.getSparseContinuousFeatures.flatMap {\n          case (id, values) =>\n            Try(featureContext.getFeature(id).getFeatureName).toOption.map { id =>\n              id -> values.map {\n                case (str, value) =>\n                  str -> value.toDouble\n              }.toMap\n            }\n        }.toMap)\n      } else None\n\n    DebugDataRecord(\n      binaryFeatures = binaryFeatures,\n      continuousFeatures = continuousFeatures,\n      discreteFeatures = discreteFeatures,\n      stringFeatures = stringFeatures,\n      sparseBinaryFeatures = sparseBinaryFeatures,\n      sparseContinuousFeatures = sparseContinuousFeatures,\n    )\n  }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasDebugOptions.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.thriftscala.DebugParams\n\ncase class DebugOptions(\n  randomizationSeed: Option[Long] = None,\n  fetchDebugInfo: Boolean = false,\n  doNotLog: Boolean = false)\n\nobject DebugOptions {\n  def fromDebugParamsThrift(debugParams: DebugParams): DebugOptions = {\n    DebugOptions(\n      debugParams.randomizationSeed,\n      debugParams.includeDebugInfoInResults.getOrElse(false),\n      debugParams.doNotLog.getOrElse(false)\n    )\n  }\n}\n\ntrait HasDebugOptions {\n  def debugOptions: Option[DebugOptions]\n\n  def getRandomizationSeed: Option[Long] = debugOptions.flatMap(_.randomizationSeed)\n\n  def fetchDebugInfo: Option[Boolean] = debugOptions.map(_.fetchDebugInfo)\n}\n\ntrait HasFrsDebugOptions {\n  def frsDebugOptions: Option[DebugOptions]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasDismissedUserIds.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasDismissedUserIds {\n  // user ids that are recently followed by the target user\n  def dismissedUserIds: Option[Seq[Long]]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasDisplayLocation.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasDisplayLocation {\n  def displayLocation: DisplayLocation\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasEngagements.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasEngagements {\n\n  def engagements: Seq[EngagementType]\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasExcludedUserIds.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasExcludedUserIds {\n  // user ids that are going to be excluded from recommendations\n  def excludedUserIds: Seq[Long]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasGeohashAndCountryCode.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasGeohashAndCountryCode {\n  def geohashAndCountryCode: Option[GeohashAndCountryCode]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasInfoPerRankingStage.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasInfoPerRankingStage {\n  def infoPerRankingStage: Option[scala.collection.Map[String, RankingInfo]]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasInterestIds.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasCustomInterests {\n  def customInterests: Option[Seq[String]]\n}\n\ntrait HasUttInterests {\n  def uttInterestIds: Option[Seq[Long]]\n}\n\ntrait HasInterestIds extends HasCustomInterests with HasUttInterests {}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasInvalidRelationshipUserIds.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasInvalidRelationshipUserIds {\n  // user ids that have invalid relationship with the target user\n  def invalidRelationshipUserIds: Option[Set[Long]]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasIsSoftUser.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasIsSoftUser {\n  def isSoftUser: Boolean\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasMutualFollowedUserIds.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\n// intersection of recent followers and followed by\ntrait HasMutualFollowedUserIds extends HasRecentFollowedUserIds with HasRecentFollowedByUserIds {\n\n  lazy val recentMutualFollows: Seq[Long] =\n    recentFollowedUserIds.getOrElse(Nil).intersect(recentFollowedByUserIds.getOrElse(Nil))\n\n  lazy val numRecentMutualFollows: Int = recentMutualFollows.size\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasPreviousRecommendationsContext.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasPreviousRecommendationsContext {\n\n  def previouslyRecommendedUserIDs: Set[Long]\n\n  def previouslyFollowedUserIds: Set[Long]\n\n  def skippedFollows: Set[Long] = {\n    previouslyRecommendedUserIDs.diff(previouslyFollowedUserIds)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasProfileId.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasProfileId {\n  def profileId: Long\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasQualityFactor.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasQualityFactor {\n  def qualityFactor: Option[Double]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasRecentFollowedByUserIds.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasRecentFollowedByUserIds {\n  // user ids that have recently followed the target user; target user has been \"followed by\" them.\n  def recentFollowedByUserIds: Option[Seq[Long]]\n\n  lazy val numRecentFollowedByUserIds: Int = recentFollowedByUserIds.map(_.size).getOrElse(0)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasRecentFollowedUserIds.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasRecentFollowedUserIds {\n  // user ids that are recently followed by the target user\n  def recentFollowedUserIds: Option[Seq[Long]]\n\n  // user ids that are recently followed by the target user in set data-structure\n  lazy val recentFollowedUserIdsSet: Option[Set[Long]] = recentFollowedUserIds match {\n    case Some(users) => Some(users.toSet)\n    case None => Some(Set.empty)\n  }\n\n  lazy val numRecentFollowedUserIds: Int = recentFollowedUserIds.map(_.size).getOrElse(0)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasRecentFollowedUserIdsWithTime.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasRecentFollowedUserIdsWithTime {\n  // user ids that are recently followed by the target user\n  def recentFollowedUserIdsWithTime: Option[Seq[UserIdWithTimestamp]]\n\n  lazy val numRecentFollowedUserIdsWithTime: Int =\n    recentFollowedUserIdsWithTime.map(_.size).getOrElse(0)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasRecentlyEngagedUserIds.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasRecentlyEngagedUserIds {\n  val recentlyEngagedUserIds: Option[Seq[RecentlyEngagedUserId]]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasRecommendationFlowIdentifier.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasRecommendationFlowIdentifier {\n  def recommendationFlowIdentifier: Option[String]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasScores.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasScores {\n  def scores: Option[Scores]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasSimilarToContext.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasSimilarToContext {\n\n  // user ids that are used to generate similar to recommendations\n  def similarToUserIds: Seq[Long]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasTopicId.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ntrait HasTopicId {\n  def topicId: Long\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasUserCandidateSourceDetails.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.hermit.ml.models.Feature\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.hermit.model.Algorithm.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\n\n/**\n * Used to keep track of a candidate's source not so much as a feature but for filtering candidate\n * from specific sources (eg. GizmoduckPredicate)\n */\ntrait HasUserCandidateSourceDetails { candidateUser: CandidateUser =>\n  def userCandidateSourceDetails: Option[UserCandidateSourceDetails]\n\n  def getAlgorithm: Algorithm = {\n    val algorithm = for {\n      details <- userCandidateSourceDetails\n      identifier <- details.primaryCandidateSource\n      algorithm <- Algorithm.withNameOpt(identifier.name)\n    } yield algorithm\n\n    algorithm.getOrElse(throw new Exception(\"Algorithm missing on candidate user!\"))\n  }\n\n  def getAllAlgorithms: Seq[Algorithm] = {\n    getCandidateSources.keys\n      .flatMap(identifier => Algorithm.withNameOpt(identifier.name)).toSeq\n  }\n\n  def getAddressBookMetadata: Option[AddressBookMetadata] = {\n    userCandidateSourceDetails.flatMap(_.addressBookMetadata)\n  }\n\n  def getCandidateSources: Map[CandidateSourceIdentifier, Option[Double]] = {\n    userCandidateSourceDetails.map(_.candidateSourceScores).getOrElse(Map.empty)\n  }\n\n  def getCandidateRanks: Map[CandidateSourceIdentifier, Int] = {\n    userCandidateSourceDetails.map(_.candidateSourceRanks).getOrElse(Map.empty)\n  }\n\n  def getCandidateFeatures: Map[CandidateSourceIdentifier, Seq[Feature]] = {\n    userCandidateSourceDetails.map(_.candidateSourceFeatures).getOrElse(Map.empty)\n  }\n\n  def getPrimaryCandidateSource: Option[CandidateSourceIdentifier] = {\n    userCandidateSourceDetails.flatMap(_.primaryCandidateSource)\n  }\n\n  def withCandidateSource(source: CandidateSourceIdentifier): CandidateUser = {\n    withCandidateSourceAndScore(source, candidateUser.score)\n  }\n\n  def withCandidateSourceAndScore(\n    source: CandidateSourceIdentifier,\n    score: Option[Double]\n  ): CandidateUser = {\n    withCandidateSourceScoreAndFeatures(source, score, Nil)\n  }\n\n  def withCandidateSourceAndFeatures(\n    source: CandidateSourceIdentifier,\n    features: Seq[Feature]\n  ): CandidateUser = {\n    withCandidateSourceScoreAndFeatures(source, candidateUser.score, features)\n  }\n\n  def withCandidateSourceScoreAndFeatures(\n    source: CandidateSourceIdentifier,\n    score: Option[Double],\n    features: Seq[Feature]\n  ): CandidateUser = {\n    val candidateSourceDetails =\n      candidateUser.userCandidateSourceDetails\n        .map { details =>\n          details.copy(\n            primaryCandidateSource = Some(source),\n            candidateSourceScores = details.candidateSourceScores + (source -> score),\n            candidateSourceFeatures = details.candidateSourceFeatures + (source -> features)\n          )\n        }.getOrElse(\n          UserCandidateSourceDetails(\n            Some(source),\n            Map(source -> score),\n            Map.empty,\n            None,\n            Map(source -> features)))\n    candidateUser.copy(\n      userCandidateSourceDetails = Some(candidateSourceDetails)\n    )\n  }\n\n  def addCandidateSourceScoresMap(\n    scoreMap: Map[CandidateSourceIdentifier, Option[Double]]\n  ): CandidateUser = {\n    val candidateSourceDetails = candidateUser.userCandidateSourceDetails\n      .map { details =>\n        details.copy(candidateSourceScores = details.candidateSourceScores ++ scoreMap)\n      }.getOrElse(UserCandidateSourceDetails(scoreMap.keys.headOption, scoreMap, Map.empty, None))\n    candidateUser.copy(\n      userCandidateSourceDetails = Some(candidateSourceDetails)\n    )\n  }\n\n  def addCandidateSourceRanksMap(\n    rankMap: Map[CandidateSourceIdentifier, Int]\n  ): CandidateUser = {\n    val candidateSourceDetails = candidateUser.userCandidateSourceDetails\n      .map { details =>\n        details.copy(candidateSourceRanks = details.candidateSourceRanks ++ rankMap)\n      }.getOrElse(UserCandidateSourceDetails(rankMap.keys.headOption, Map.empty, rankMap, None))\n    candidateUser.copy(\n      userCandidateSourceDetails = Some(candidateSourceDetails)\n    )\n  }\n\n  def addInfoPerRankingStage(\n    rankingStage: String,\n    scores: Option[Scores],\n    rank: Int\n  ): CandidateUser = {\n    val scoresOpt: Option[Scores] = scores.orElse(candidateUser.scores)\n    val originalInfoPerRankingStage =\n      candidateUser.infoPerRankingStage.getOrElse(Map[String, RankingInfo]())\n    candidateUser.copy(\n      infoPerRankingStage =\n        Some(originalInfoPerRankingStage + (rankingStage -> RankingInfo(scoresOpt, Some(rank))))\n    )\n  }\n\n  def addAddressBookMetadataIfAvailable(\n    candidateSources: Seq[CandidateSourceIdentifier]\n  ): CandidateUser = {\n\n    val addressBookMetadata = AddressBookMetadata(\n      inForwardPhoneBook =\n        candidateSources.contains(AddressBookMetadata.ForwardPhoneBookCandidateSource),\n      inReversePhoneBook =\n        candidateSources.contains(AddressBookMetadata.ReversePhoneBookCandidateSource),\n      inForwardEmailBook =\n        candidateSources.contains(AddressBookMetadata.ForwardEmailBookCandidateSource),\n      inReverseEmailBook =\n        candidateSources.contains(AddressBookMetadata.ReverseEmailBookCandidateSource)\n    )\n\n    val newCandidateSourceDetails = candidateUser.userCandidateSourceDetails\n      .map { details =>\n        details.copy(addressBookMetadata = Some(addressBookMetadata))\n      }.getOrElse(\n        UserCandidateSourceDetails(\n          None,\n          Map.empty,\n          Map.empty,\n          Some(addressBookMetadata),\n          Map.empty))\n\n    candidateUser.copy(\n      userCandidateSourceDetails = Some(newCandidateSourceDetails)\n    )\n  }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasUserState.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\n\ntrait HasUserState {\n  def userState: Option[UserState]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/HasWtfImpressions.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.util.Time\n\ntrait HasWtfImpressions {\n\n  def wtfImpressions: Option[Seq[WtfImpression]]\n\n  lazy val numWtfImpressions: Int = wtfImpressions.map(_.size).getOrElse(0)\n\n  lazy val candidateImpressions: Map[Long, WtfImpression] = wtfImpressions\n    .map { imprMap =>\n      imprMap.map { i =>\n        i.candidateId -> i\n      }.toMap\n    }.getOrElse(Map.empty)\n\n  lazy val latestImpressionTime: Time = {\n    if (wtfImpressions.exists(_.nonEmpty)) {\n      wtfImpressions.get.map(_.latestTime).max\n    } else Time.Top\n  }\n\n  def getCandidateImpressionCounts(id: Long): Option[Int] =\n    candidateImpressions.get(id).map(_.counts)\n\n  def getCandidateLatestTime(id: Long): Option[Time] = {\n    candidateImpressions.get(id).map(_.latestTime)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/OptimusRequest.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.timelines.configapi.HasParams\n\n/**\nConvenience trait to group together all traits needed for optimus ranking\n */\ntrait OptimusRequest\n    extends HasParams\n    with HasClientContext\n    with HasDisplayLocation\n    with HasInterestIds\n    with HasDebugOptions\n    with HasPreviousRecommendationsContext {}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/Product.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.product_mixer.core.model.common.identifier.ProductIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.{Product => ProductMixerProduct}\n\nobject Product {\n  case object MagicRecs extends ProductMixerProduct {\n    override val identifier: ProductIdentifier = ProductIdentifier(\"MagicRecs\")\n    override val stringCenterProject: Option[String] = Some(\"people-discovery\")\n  }\n\n  case object PlaceholderProductMixerProduct extends ProductMixerProduct {\n    override val identifier: ProductIdentifier = ProductIdentifier(\"PlaceholderProductMixerProduct\")\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/RankingInfo.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\n\ncase class RankingInfo(\n  scores: Option[Scores],\n  rank: Option[Int]) {\n\n  def toThrift: t.RankingInfo = {\n    t.RankingInfo(scores.map(_.toThrift), rank)\n  }\n\n  def toOfflineThrift: offline.RankingInfo = {\n    offline.RankingInfo(scores.map(_.toOfflineThrift), rank)\n  }\n}\n\nobject RankingInfo {\n\n  def fromThrift(rankingInfo: t.RankingInfo): RankingInfo = {\n    RankingInfo(\n      scores = rankingInfo.scores.map(Scores.fromThrift),\n      rank = rankingInfo.rank\n    )\n  }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/Reason.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\n\ncase class FollowProof(followedBy: Seq[Long], numIds: Int) {\n  def toThrift: t.FollowProof = {\n    t.FollowProof(followedBy, numIds)\n  }\n\n  def toOfflineThrift: offline.FollowProof = offline.FollowProof(followedBy, numIds)\n}\n\nobject FollowProof {\n\n  def fromThrift(proof: t.FollowProof): FollowProof = {\n    FollowProof(proof.userIds, proof.numIds)\n  }\n}\n\ncase class SimilarToProof(similarTo: Seq[Long]) {\n  def toThrift: t.SimilarToProof = {\n    t.SimilarToProof(similarTo)\n  }\n\n  def toOfflineThrift: offline.SimilarToProof = offline.SimilarToProof(similarTo)\n}\n\nobject SimilarToProof {\n  def fromThrift(proof: t.SimilarToProof): SimilarToProof = {\n    SimilarToProof(proof.userIds)\n  }\n}\n\ncase class PopularInGeoProof(location: String) {\n  def toThrift: t.PopularInGeoProof = {\n    t.PopularInGeoProof(location)\n  }\n\n  def toOfflineThrift: offline.PopularInGeoProof = offline.PopularInGeoProof(location)\n}\n\nobject PopularInGeoProof {\n\n  def fromThrift(proof: t.PopularInGeoProof): PopularInGeoProof = {\n    PopularInGeoProof(proof.location)\n  }\n}\n\ncase class TttInterestProof(interestId: Long, interestDisplayName: String) {\n  def toThrift: t.TttInterestProof = {\n    t.TttInterestProof(interestId, interestDisplayName)\n  }\n\n  def toOfflineThrift: offline.TttInterestProof =\n    offline.TttInterestProof(interestId, interestDisplayName)\n}\n\nobject TttInterestProof {\n\n  def fromThrift(proof: t.TttInterestProof): TttInterestProof = {\n    TttInterestProof(proof.interestId, proof.interestDisplayName)\n  }\n}\n\ncase class TopicProof(topicId: Long) {\n  def toThrift: t.TopicProof = {\n    t.TopicProof(topicId)\n  }\n\n  def toOfflineThrift: offline.TopicProof =\n    offline.TopicProof(topicId)\n}\n\nobject TopicProof {\n  def fromThrift(proof: t.TopicProof): TopicProof = {\n    TopicProof(proof.topicId)\n  }\n}\n\ncase class CustomInterest(query: String) {\n  def toThrift: t.CustomInterestProof = {\n    t.CustomInterestProof(query)\n  }\n\n  def toOfflineThrift: offline.CustomInterestProof =\n    offline.CustomInterestProof(query)\n}\n\nobject CustomInterest {\n  def fromThrift(proof: t.CustomInterestProof): CustomInterest = {\n    CustomInterest(proof.query)\n  }\n}\n\ncase class TweetsAuthorProof(tweetIds: Seq[Long]) {\n  def toThrift: t.TweetsAuthorProof = {\n    t.TweetsAuthorProof(tweetIds)\n  }\n\n  def toOfflineThrift: offline.TweetsAuthorProof =\n    offline.TweetsAuthorProof(tweetIds)\n}\n\nobject TweetsAuthorProof {\n  def fromThrift(proof: t.TweetsAuthorProof): TweetsAuthorProof = {\n    TweetsAuthorProof(proof.tweetIds)\n  }\n}\n\ncase class DeviceFollowProof(isDeviceFollow: Boolean) {\n  def toThrift: t.DeviceFollowProof = {\n    t.DeviceFollowProof(isDeviceFollow)\n  }\n  def toOfflineThrift: offline.DeviceFollowProof =\n    offline.DeviceFollowProof(isDeviceFollow)\n}\n\nobject DeviceFollowProof {\n  def fromThrift(proof: t.DeviceFollowProof): DeviceFollowProof = {\n    DeviceFollowProof(proof.isDeviceFollow)\n  }\n\n}\n\ncase class AccountProof(\n  followProof: Option[FollowProof] = None,\n  similarToProof: Option[SimilarToProof] = None,\n  popularInGeoProof: Option[PopularInGeoProof] = None,\n  tttInterestProof: Option[TttInterestProof] = None,\n  topicProof: Option[TopicProof] = None,\n  customInterestProof: Option[CustomInterest] = None,\n  tweetsAuthorProof: Option[TweetsAuthorProof] = None,\n  deviceFollowProof: Option[DeviceFollowProof] = None) {\n  def toThrift: t.AccountProof = {\n    t.AccountProof(\n      followProof.map(_.toThrift),\n      similarToProof.map(_.toThrift),\n      popularInGeoProof.map(_.toThrift),\n      tttInterestProof.map(_.toThrift),\n      topicProof.map(_.toThrift),\n      customInterestProof.map(_.toThrift),\n      tweetsAuthorProof.map(_.toThrift),\n      deviceFollowProof.map(_.toThrift)\n    )\n  }\n\n  def toOfflineThrift: offline.AccountProof = {\n    offline.AccountProof(\n      followProof.map(_.toOfflineThrift),\n      similarToProof.map(_.toOfflineThrift),\n      popularInGeoProof.map(_.toOfflineThrift),\n      tttInterestProof.map(_.toOfflineThrift),\n      topicProof.map(_.toOfflineThrift),\n      customInterestProof.map(_.toOfflineThrift),\n      tweetsAuthorProof.map(_.toOfflineThrift),\n      deviceFollowProof.map(_.toOfflineThrift)\n    )\n  }\n}\n\nobject AccountProof {\n  def fromThrift(proof: t.AccountProof): AccountProof = {\n    AccountProof(\n      proof.followProof.map(FollowProof.fromThrift),\n      proof.similarToProof.map(SimilarToProof.fromThrift),\n      proof.popularInGeoProof.map(PopularInGeoProof.fromThrift),\n      proof.tttInterestProof.map(TttInterestProof.fromThrift),\n      proof.topicProof.map(TopicProof.fromThrift),\n      proof.customInterestProof.map(CustomInterest.fromThrift),\n      proof.tweetsAuthorProof.map(TweetsAuthorProof.fromThrift),\n      proof.deviceFollowProof.map(DeviceFollowProof.fromThrift)\n    )\n  }\n}\n\ncase class Reason(accountProof: Option[AccountProof]) {\n  def toThrift: t.Reason = {\n    t.Reason(accountProof.map(_.toThrift))\n  }\n\n  def toOfflineThrift: offline.Reason = {\n    offline.Reason(accountProof.map(_.toOfflineThrift))\n  }\n}\n\nobject Reason {\n\n  def fromThrift(reason: t.Reason): Reason = {\n    Reason(reason.accountProof.map(AccountProof.fromThrift))\n  }\n}\n\ntrait HasReason {\n\n  def reason: Option[Reason]\n  // helper methods below\n\n  def followedBy: Option[Seq[Long]] = {\n    for {\n      reason <- reason\n      accountProof <- reason.accountProof\n      followProof <- accountProof.followProof\n    } yield { followProof.followedBy }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/RecentlyEngagedUserId.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.follow_recommendations.{thriftscala => t}\n\ncase class RecentlyEngagedUserId(id: Long, engagementType: EngagementType) {\n  def toThrift: t.RecentlyEngagedUserId =\n    t.RecentlyEngagedUserId(id = id, engagementType = engagementType.toThrift)\n\n  def toOfflineThrift: offline.RecentlyEngagedUserId =\n    offline.RecentlyEngagedUserId(id = id, engagementType = engagementType.toOfflineThrift)\n}\n\nobject RecentlyEngagedUserId {\n  def fromThrift(recentlyEngagedUserId: t.RecentlyEngagedUserId): RecentlyEngagedUserId = {\n    RecentlyEngagedUserId(\n      id = recentlyEngagedUserId.id,\n      engagementType = EngagementType.fromThrift(recentlyEngagedUserId.engagementType)\n    )\n  }\n\n  def fromOfflineThrift(\n    recentlyEngagedUserId: offline.RecentlyEngagedUserId\n  ): RecentlyEngagedUserId = {\n    RecentlyEngagedUserId(\n      id = recentlyEngagedUserId.id,\n      engagementType = EngagementType.fromOfflineThrift(recentlyEngagedUserId.engagementType)\n    )\n  }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/RecommendationStep.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\n\ncase class RecommendationStep(\n  recommendations: Seq[FlowRecommendation],\n  followedUserIds: Set[Long]) {\n\n  def toThrift: t.RecommendationStep = t.RecommendationStep(\n    recommendations = recommendations.map(_.toThrift),\n    followedUserIds = followedUserIds\n  )\n\n  def toOfflineThrift: offline.OfflineRecommendationStep =\n    offline.OfflineRecommendationStep(\n      recommendations = recommendations.map(_.toOfflineThrift),\n      followedUserIds = followedUserIds)\n\n}\n\nobject RecommendationStep {\n\n  def fromThrift(recommendationStep: t.RecommendationStep): RecommendationStep = {\n    RecommendationStep(\n      recommendations = recommendationStep.recommendations.map(FlowRecommendation.fromThrift),\n      followedUserIds = recommendationStep.followedUserIds.toSet)\n  }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/STPGraph.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.hermit.model.Algorithm.Algorithm\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.FirstDegreeEdge\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.FirstDegreeEdgeInfo\nimport com.twitter.wtf.scalding.jobs.strong_tie_prediction.SecondDegreeEdge\n\ncase class PotentialFirstDegreeEdge(\n  userId: Long,\n  connectingId: Long,\n  algorithm: Algorithm,\n  score: Double,\n  edgeInfo: FirstDegreeEdgeInfo)\n\ncase class IntermediateSecondDegreeEdge(\n  connectingId: Long,\n  candidateId: Long,\n  edgeInfo: FirstDegreeEdgeInfo)\n\ncase class STPGraph(\n  firstDegreeEdgeInfoList: List[FirstDegreeEdge],\n  secondDegreeEdgeInfoList: List[SecondDegreeEdge])\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/SafetyLevel.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.spam.rtf.thriftscala.{SafetyLevel => ThriftSafetyLevel}\n\nsealed trait SafetyLevel {\n  def toThrift: ThriftSafetyLevel\n}\n\nobject SafetyLevel {\n  case object Recommendations extends SafetyLevel {\n    override val toThrift = ThriftSafetyLevel.Recommendations\n  }\n\n  case object TopicsLandingPageTopicRecommendations extends SafetyLevel {\n    override val toThrift = ThriftSafetyLevel.TopicsLandingPageTopicRecommendations\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/Score.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId.RankerId\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.follow_recommendations.{thriftscala => t}\n\n/**\n * Type of Score. This is used to differentiate scores.\n *\n * Define it as a trait so it is possible to add more information for different score types.\n */\nsealed trait ScoreType {\n  def getName: String\n}\n\n/**\n * Existing Score Types\n */\nobject ScoreType {\n\n  /**\n   * the score is calculated based on heuristics and most likely not normalized\n   */\n  case object HeuristicBasedScore extends ScoreType {\n    override def getName: String = \"HeuristicBasedScore\"\n  }\n\n  /**\n   * probability of follow after the candidate is recommended to the user\n   */\n  case object PFollowGivenReco extends ScoreType {\n    override def getName: String = \"PFollowGivenReco\"\n  }\n\n  /**\n   * probability of engage after the user follows the candidate\n   */\n  case object PEngagementGivenFollow extends ScoreType {\n    override def getName: String = \"PEngagementGivenFollow\"\n  }\n\n  /**\n   * probability of engage per tweet impression\n   */\n  case object PEngagementPerImpression extends ScoreType {\n    override def getName: String = \"PEngagementPerImpression\"\n  }\n\n  /**\n   * probability of engage per tweet impression\n   */\n  case object PEngagementGivenReco extends ScoreType {\n    override def getName: String = \"PEngagementGivenReco\"\n  }\n\n  def fromScoreTypeString(scoreTypeName: String): ScoreType = scoreTypeName match {\n    case \"HeuristicBasedScore\" => HeuristicBasedScore\n    case \"PFollowGivenReco\" => PFollowGivenReco\n    case \"PEngagementGivenFollow\" => PEngagementGivenFollow\n    case \"PEngagementPerImpression\" => PEngagementPerImpression\n    case \"PEngagementGivenReco\" => PEngagementGivenReco\n  }\n}\n\n/**\n * Represent the output from a certain ranker or scorer. All the fields are optional\n *\n * @param value value of the score\n * @param rankerId ranker id\n * @param scoreType score type\n */\nfinal case class Score(\n  value: Double,\n  rankerId: Option[RankerId] = None,\n  scoreType: Option[ScoreType] = None) {\n\n  def toThrift: t.Score = t.Score(\n    value = value,\n    rankerId = rankerId.map(_.toString),\n    scoreType = scoreType.map(_.getName)\n  )\n\n  def toOfflineThrift: offline.Score =\n    offline.Score(\n      value = value,\n      rankerId = rankerId.map(_.toString),\n      scoreType = scoreType.map(_.getName)\n    )\n}\n\nobject Score {\n\n  val RandomScore = Score(0.0d, Some(RankerId.RandomRanker))\n\n  def optimusScore(score: Double, scoreType: ScoreType): Score = {\n    Score(value = score, scoreType = Some(scoreType))\n  }\n\n  def predictionScore(score: Double, rankerId: RankerId): Score = {\n    Score(value = score, rankerId = Some(rankerId))\n  }\n\n  def fromThrift(thriftScore: t.Score): Score =\n    Score(\n      value = thriftScore.value,\n      rankerId = thriftScore.rankerId.flatMap(RankerId.getRankerByName),\n      scoreType = thriftScore.scoreType.map(ScoreType.fromScoreTypeString)\n    )\n}\n\n/**\n * a list of scores\n */\nfinal case class Scores(\n  scores: Seq[Score],\n  selectedRankerId: Option[RankerId] = None,\n  isInProducerScoringExperiment: Boolean = false) {\n\n  def toThrift: t.Scores =\n    t.Scores(\n      scores = scores.map(_.toThrift),\n      selectedRankerId = selectedRankerId.map(_.toString),\n      isInProducerScoringExperiment = isInProducerScoringExperiment\n    )\n\n  def toOfflineThrift: offline.Scores =\n    offline.Scores(\n      scores = scores.map(_.toOfflineThrift),\n      selectedRankerId = selectedRankerId.map(_.toString),\n      isInProducerScoringExperiment = isInProducerScoringExperiment\n    )\n}\n\nobject Scores {\n  val Empty: Scores = Scores(Nil)\n\n  def fromThrift(thriftScores: t.Scores): Scores =\n    Scores(\n      scores = thriftScores.scores.map(Score.fromThrift),\n      selectedRankerId = thriftScores.selectedRankerId.flatMap(RankerId.getRankerByName),\n      isInProducerScoringExperiment = thriftScores.isInProducerScoringExperiment\n    )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/Session.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.finagle.tracing.Trace\n\nobject Session {\n\n  /**\n   * The sessionId in FRS is the finagle trace id which is static within the lifetime of a single\n   * request.\n   *\n   * It is used when generating per-candidate tokens (in TrackingTokenTransform) and is also passed\n   * in to downstream Optimus ranker requests.\n   *\n   */\n  def getSessionId: Long = Trace.id.traceId.toLong\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/SignalData.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.usersignalservice.thriftscala.SignalType\nimport com.twitter.usersignalservice.thriftscala.Signal\n\ntrait SignalData {\n  val userId: Long\n  val signalType: SignalType\n}\n\ncase class RecentFollowsSignal(\n  override val userId: Long,\n  override val signalType: SignalType,\n  followedUserId: Long,\n  timestamp: Long)\n    extends SignalData\n\nobject RecentFollowsSignal {\n\n  def fromUssSignal(targetUserId: Long, signal: Signal): RecentFollowsSignal = {\n    val InternalId.UserId(followedUserId) = signal.targetInternalId.getOrElse(\n      throw new IllegalArgumentException(\"RecentFollow Signal does not have internalId\"))\n\n    RecentFollowsSignal(\n      userId = targetUserId,\n      followedUserId = followedUserId,\n      timestamp = signal.timestamp,\n      signalType = signal.signalType\n    )\n  }\n\n  def getRecentFollowedUserIds(\n    signalDataMap: Option[Map[SignalType, Seq[SignalData]]]\n  ): Option[Seq[Long]] = {\n    signalDataMap.map(_.getOrElse(SignalType.AccountFollow, default = Seq.empty).flatMap {\n      case RecentFollowsSignal(userId, signalType, followedUserId, timestamp) =>\n        Some(followedUserId)\n      case _ => None\n    })\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/TrackingToken.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport com.twitter.scrooge.BinaryThriftStructSerializer\nimport com.twitter.suggests.controller_data.thriftscala.ControllerData\nimport com.twitter.util.Base64StringEncoder\n\n/**\n * used for attribution per target-candidate pair\n * @param sessionId         trace-id of the finagle request\n * @param controllerData    64-bit encoded binary attributes of our recommendation\n * @param algorithmId       id for identifying a candidate source. maintained for backwards compatibility\n */\ncase class TrackingToken(\n  sessionId: Long,\n  displayLocation: Option[DisplayLocation],\n  controllerData: Option[ControllerData],\n  algorithmId: Option[Int]) {\n\n  def toThrift: t.TrackingToken = {\n    Trace.id.traceId.toLong\n    t.TrackingToken(\n      sessionId = sessionId,\n      displayLocation = displayLocation.map(_.toThrift),\n      controllerData = controllerData,\n      algoId = algorithmId\n    )\n  }\n\n  def toOfflineThrift: offline.TrackingToken = {\n    offline.TrackingToken(\n      sessionId = sessionId,\n      displayLocation = displayLocation.map(_.toOfflineThrift),\n      controllerData = controllerData,\n      algoId = algorithmId\n    )\n  }\n}\n\nobject TrackingToken {\n  val binaryThriftSerializer = BinaryThriftStructSerializer[t.TrackingToken](t.TrackingToken)\n  def serialize(trackingToken: TrackingToken): String = {\n    Base64StringEncoder.encode(binaryThriftSerializer.toBytes(trackingToken.toThrift))\n  }\n  def deserialize(trackingTokenStr: String): TrackingToken = {\n    fromThrift(binaryThriftSerializer.fromBytes(Base64StringEncoder.decode(trackingTokenStr)))\n  }\n  def fromThrift(token: t.TrackingToken): TrackingToken = {\n    TrackingToken(\n      sessionId = token.sessionId,\n      displayLocation = token.displayLocation.map(DisplayLocation.fromThrift),\n      controllerData = token.controllerData,\n      algorithmId = token.algoId\n    )\n  }\n}\n\ntrait HasTrackingToken {\n  def trackingToken: Option[TrackingToken]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/TweetCandidate.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ncase class TweetCandidate(\n  tweetId: Long,\n  authorId: Long,\n  score: Option[Double])\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/UserCandidateSourceDetails.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport com.twitter.hermit.constants.AlgorithmFeedbackTokens._\nimport com.twitter.hermit.ml.models.Feature\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\n\n/**\n * primaryCandidateSource param is showing the candidate source that responsible for generating this\n * candidate, as the candidate might have gone through multiple candidate sources to get generated\n * (for example if it has generated by a composite source). WeightedCandidateSourceRanker uses this\n * field to do the sampling over candidate sources. All the sources used for generating this\n * candidate (including the primary source) and their corresponding score exist in the\n * candidateSourceScores field.\n */\ncase class UserCandidateSourceDetails(\n  primaryCandidateSource: Option[CandidateSourceIdentifier],\n  candidateSourceScores: Map[CandidateSourceIdentifier, Option[Double]] = Map.empty,\n  candidateSourceRanks: Map[CandidateSourceIdentifier, Int] = Map.empty,\n  addressBookMetadata: Option[AddressBookMetadata] = None,\n  candidateSourceFeatures: Map[CandidateSourceIdentifier, Seq[Feature]] = Map.empty,\n) {\n\n  def toThrift: t.CandidateSourceDetails = {\n    t.CandidateSourceDetails(\n      candidateSourceScores = Some(candidateSourceScores.map {\n        case (identifier, score) =>\n          (identifier.name, score.getOrElse(0.0d))\n      }),\n      primarySource = for {\n        identifier <- primaryCandidateSource\n        algo <- Algorithm.withNameOpt(identifier.name)\n        feedbackToken <- AlgorithmToFeedbackTokenMap.get(algo)\n      } yield feedbackToken\n    )\n  }\n\n  def toOfflineThrift: offline.CandidateSourceDetails = {\n    offline.CandidateSourceDetails(\n      candidateSourceScores = Some(candidateSourceScores.map {\n        case (identifier, score) =>\n          (identifier.name, score.getOrElse(0.0d))\n      }),\n      primarySource = for {\n        identifier <- primaryCandidateSource\n        algo <- Algorithm.withNameOpt(identifier.name)\n        feedbackToken <- AlgorithmToFeedbackTokenMap.get(algo)\n      } yield feedbackToken\n    )\n  }\n}\n\nobject UserCandidateSourceDetails {\n  val algorithmNameMap: Map[String, Algorithm.Value] = Algorithm.values.map {\n    algorithmValue: Algorithm.Value =>\n      (algorithmValue.toString, algorithmValue)\n  }.toMap\n\n  /**\n   * This method is used to parse the candidate source of the candidates, which is only passed from\n   * the scoreUserCandidates endpoint. We create custom candidate source identifiers which\n   * CandidateAlgorithmSource will read from to hydrate the algorithm id feature.\n   * candidateSourceScores will not be populated from the endpoint, but we add the conversion for\n   * completeness. Note that the conversion uses the raw string of the Algorithm rather than the\n   * assigned strings that we give to our own candidate sources in the FRS.\n   */\n  def fromThrift(details: t.CandidateSourceDetails): UserCandidateSourceDetails = {\n    val primaryCandidateSource: Option[CandidateSourceIdentifier] = for {\n      primarySourceToken <- details.primarySource\n      algo <- TokenToAlgorithmMap.get(primarySourceToken)\n    } yield CandidateSourceIdentifier(algo.toString)\n\n    val candidateSourceScores = for {\n      scoreMap <- details.candidateSourceScores.toSeq\n      (name, score) <- scoreMap\n      algo <- algorithmNameMap.get(name)\n    } yield {\n      CandidateSourceIdentifier(algo.toString) -> Some(score)\n    }\n    val candidateSourceRanks = for {\n      rankMap <- details.candidateSourceRanks.toSeq\n      (name, rank) <- rankMap\n      algo <- algorithmNameMap.get(name)\n    } yield {\n      CandidateSourceIdentifier(algo.toString) -> rank\n    }\n    UserCandidateSourceDetails(\n      primaryCandidateSource = primaryCandidateSource,\n      candidateSourceScores = candidateSourceScores.toMap,\n      candidateSourceRanks = candidateSourceRanks.toMap,\n      addressBookMetadata = None,\n      candidateSourceFeatures = Map.empty\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/UserIdAndTimestamp.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\ncase class UserIdWithTimestamp(userId: Long, timeInMs: Long)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/WtfImpression.scala",
    "content": "package com.twitter.follow_recommendations.common.models\n\nimport com.twitter.util.Time\n\n/**\n * Domain model for representing impressions on wtf recommendations in the past 16 days\n */\ncase class WtfImpression(\n  candidateId: Long,\n  displayLocation: DisplayLocation,\n  latestTime: Time,\n  counts: Int)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"escherbird/src/scala/com/twitter/escherbird/util/stitchcache\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/user_state\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/features\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/stores\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/CandidateParamPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates\n\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.models.FilterReason\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Param\n\nclass CandidateParamPredicate[A <: HasParams](\n  param: Param[Boolean],\n  reason: FilterReason)\n    extends Predicate[A] {\n  override def apply(candidate: A): Stitch[PredicateResult] = {\n    if (candidate.params(param)) {\n      Stitch.value(PredicateResult.Valid)\n    } else {\n      Stitch.value(PredicateResult.Invalid(Set(reason)))\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/CandidateSourceParamPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates\n\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FilterReason\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * This predicate allows us to filter candidates given its source.\n * To avoid bucket dilution, we only want to evaluate the param (which would implicitly trigger\n * bucketing for FSParams) only if the candidate source fn yields true.\n * The param provided should be true when we want to keep the candidate and false otherwise.\n */\nclass CandidateSourceParamPredicate(\n  val param: Param[Boolean],\n  val reason: FilterReason,\n  candidateSources: Set[CandidateSourceIdentifier])\n    extends Predicate[CandidateUser] {\n  override def apply(candidate: CandidateUser): Stitch[PredicateResult] = {\n    // we want to avoid evaluating the param if the candidate source fn yields false\n    if (candidate.getCandidateSources.keys.exists(candidateSources.contains) && !candidate.params(\n        param)) {\n      Stitch.value(PredicateResult.Invalid(Set(reason)))\n    } else {\n      Stitch.value(PredicateResult.Valid)\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/CuratedCompetitorListPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.follow_recommendations.common.models.FilterReason.CuratedAccountsCompetitorList\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.escherbird.util.stitchcache.StitchCache\n\n@Singleton\ncase class CuratedCompetitorListPredicate @Inject() (\n  statsReceiver: StatsReceiver,\n  @Named(GuiceNamedConstants.CURATED_COMPETITOR_ACCOUNTS_FETCHER) competitorAccountFetcher: Fetcher[\n    String,\n    Unit,\n    Seq[Long]\n  ]) extends Predicate[CandidateUser] {\n\n  private val stats: StatsReceiver = statsReceiver.scope(this.getClass.getName)\n  private val cacheStats = stats.scope(\"cache\")\n\n  private val cache = StitchCache[String, Set[Long]](\n    maxCacheSize = CuratedCompetitorListPredicate.CacheNumberOfEntries,\n    ttl = CuratedCompetitorListPredicate.CacheTTL,\n    statsReceiver = cacheStats,\n    underlyingCall = (competitorListPrefix: String) => query(competitorListPrefix)\n  )\n\n  private def query(prefix: String): Stitch[Set[Long]] =\n    competitorAccountFetcher.fetch(prefix).map(_.v.getOrElse(Nil).toSet)\n\n  /**\n   * Caveat here is that though the similarToUserIds allows for a Seq[Long], in practice we would\n   * only return 1 userId. Multiple userId's would result in filtering candidates associated with\n   * a different similarToUserId. For example:\n   *   - similarToUser1 -> candidate1, candidate2\n   *   - similarToUser2 -> candidate3\n   *   and in the competitorList store we have:\n   *   - similarToUser1 -> candidate3\n   *   we'll be filtering candidate3 on account of similarToUser1, even though it was generated\n   *   with similarToUser2. This might still be desirable at a product level (since we don't want\n   *   to show these accounts anyway), but might not achieve what you intend to code-wise.\n   */\n  override def apply(candidate: CandidateUser): Stitch[PredicateResult] = {\n    cache.readThrough(CuratedCompetitorListPredicate.DefaultKey).map { competitorListAccounts =>\n      if (competitorListAccounts.contains(candidate.id)) {\n        PredicateResult.Invalid(Set(CuratedAccountsCompetitorList))\n      } else {\n        PredicateResult.Valid\n      }\n    }\n  }\n}\n\nobject CuratedCompetitorListPredicate {\n  val DefaultKey: String = \"default_list\"\n  val CacheTTL = 5.minutes\n  val CacheNumberOfEntries = 5\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/ExcludedUserIdPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates\n\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.models.FilterReason.ExcludedId\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasExcludedUserIds\nimport com.twitter.stitch.Stitch\n\nobject ExcludedUserIdPredicate extends Predicate[(HasExcludedUserIds, CandidateUser)] {\n\n  val ValidStitch: Stitch[PredicateResult.Valid.type] = Stitch.value(PredicateResult.Valid)\n  val ExcludedStitch: Stitch[PredicateResult.Invalid] =\n    Stitch.value(PredicateResult.Invalid(Set(ExcludedId)))\n\n  override def apply(pair: (HasExcludedUserIds, CandidateUser)): Stitch[PredicateResult] = {\n    val (excludedUserIds, candidate) = pair\n    if (excludedUserIds.excludedUserIds.contains(candidate.id)) {\n      ExcludedStitch\n    } else {\n      ValidStitch\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/InactivePredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates\n\nimport com.google.inject.name.Named\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FilterReason\nimport com.twitter.follow_recommendations.common.predicates.InactivePredicateParams._\nimport com.twitter.service.metastore.gen.thriftscala.UserRecommendabilityFeatures\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.escherbird.util.stitchcache.StitchCache\nimport com.twitter.follow_recommendations.common.models.HasUserState\nimport com.twitter.follow_recommendations.common.predicates.InactivePredicateParams.DefaultInactivityThreshold\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\n\nimport java.lang.{Long => JLong}\n\n@Singleton\ncase class InactivePredicate @Inject() (\n  statsReceiver: StatsReceiver,\n  @Named(GuiceNamedConstants.USER_RECOMMENDABILITY_FETCHER) userRecommendabilityFetcher: Fetcher[\n    Long,\n    Unit,\n    UserRecommendabilityFeatures\n  ]) extends Predicate[(HasParams with HasClientContext with HasUserState, CandidateUser)] {\n\n  private val stats: StatsReceiver = statsReceiver.scope(\"InactivePredicate\")\n  private val cacheStats = stats.scope(\"cache\")\n\n  private def queryUserRecommendable(userId: Long): Stitch[Option[UserRecommendabilityFeatures]] =\n    userRecommendabilityFetcher.fetch(userId).map(_.v)\n\n  private val userRecommendableCache =\n    StitchCache[JLong, Option[UserRecommendabilityFeatures]](\n      maxCacheSize = 100000,\n      ttl = 12.hours,\n      statsReceiver = cacheStats.scope(\"UserRecommendable\"),\n      underlyingCall = (userId: JLong) => queryUserRecommendable(userId)\n    )\n\n  override def apply(\n    targetAndCandidate: (HasParams with HasClientContext with HasUserState, CandidateUser)\n  ): Stitch[PredicateResult] = {\n    val (target, candidate) = targetAndCandidate\n\n    userRecommendableCache\n      .readThrough(candidate.id).map {\n        case recFeaturesFetchResult =>\n          recFeaturesFetchResult match {\n            case None =>\n              PredicateResult.Invalid(Set(FilterReason.MissingRecommendabilityData))\n            case Some(recFeatures) =>\n              if (disableInactivityPredicate(target, target.userState, recFeatures.userState)) {\n                PredicateResult.Valid\n              } else {\n                val defaultInactivityThreshold = target.params(DefaultInactivityThreshold).days\n                val hasBeenActiveRecently = recFeatures.lastStatusUpdateMs\n                  .map(Time.now - Time.fromMilliseconds(_)).getOrElse(\n                    Duration.Top) < defaultInactivityThreshold\n                stats\n                  .scope(defaultInactivityThreshold.toString).counter(\n                    if (hasBeenActiveRecently)\n                      \"active\"\n                    else\n                      \"inactive\"\n                  ).incr()\n                if (hasBeenActiveRecently && (!target\n                    .params(UseEggFilter) || recFeatures.isNotEgg.contains(1))) {\n                  PredicateResult.Valid\n                } else {\n                  PredicateResult.Invalid(Set(FilterReason.Inactive))\n                }\n              }\n          }\n      }.rescue {\n        case e: Exception =>\n          stats.counter(e.getClass.getSimpleName).incr()\n          Stitch(PredicateResult.Invalid(Set(FilterReason.FailOpen)))\n      }\n  }\n\n  private[this] def disableInactivityPredicate(\n    target: HasParams,\n    consumerState: Option[UserState],\n    candidateState: Option[UserState]\n  ): Boolean = {\n    target.params(MightBeDisabled) &&\n    consumerState.exists(InactivePredicate.ValidConsumerStates.contains) &&\n    (\n      (\n        candidateState.exists(InactivePredicate.ValidCandidateStates.contains) &&\n        !target.params(OnlyDisableForNewUserStateCandidates)\n      ) ||\n      (\n        candidateState.contains(UserState.New) &&\n        target.params(OnlyDisableForNewUserStateCandidates)\n      )\n    )\n  }\n}\n\nobject InactivePredicate {\n  val ValidConsumerStates: Set[UserState] = Set(\n    UserState.HeavyNonTweeter,\n    UserState.MediumNonTweeter,\n    UserState.HeavyTweeter,\n    UserState.MediumTweeter\n  )\n  val ValidCandidateStates: Set[UserState] =\n    Set(UserState.New, UserState.VeryLight, UserState.Light, UserState.NearZero)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/InactivePredicateParams.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\n\nobject InactivePredicateParams {\n  case object DefaultInactivityThreshold\n      extends FSBoundedParam[Int](\n        name = \"inactive_predicate_default_inactivity_threshold\",\n        default = 60,\n        min = 1,\n        max = 500\n      )\n  case object UseEggFilter extends Param[Boolean](true)\n  case object MightBeDisabled extends FSParam[Boolean](\"inactive_predicate_might_be_disabled\", true)\n  case object OnlyDisableForNewUserStateCandidates\n      extends FSParam[Boolean](\n        \"inactive_predicate_only_disable_for_new_user_state_candidates\",\n        false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/PreviouslyRecommendedUserIdsPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates\n\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FilterReason\nimport com.twitter.follow_recommendations.common.models.HasPreviousRecommendationsContext\nimport com.twitter.stitch.Stitch\nimport javax.inject.Singleton\n\n@Singleton\nclass PreviouslyRecommendedUserIdsPredicate\n    extends Predicate[(HasPreviousRecommendationsContext, CandidateUser)] {\n  override def apply(\n    pair: (HasPreviousRecommendationsContext, CandidateUser)\n  ): Stitch[PredicateResult] = {\n\n    val (targetUser, candidate) = pair\n\n    val previouslyRecommendedUserIDs = targetUser.previouslyRecommendedUserIDs\n\n    if (!previouslyRecommendedUserIDs.contains(candidate.id)) {\n      PreviouslyRecommendedUserIdsPredicate.ValidStitch\n    } else {\n      PreviouslyRecommendedUserIdsPredicate.AlreadyRecommendedStitch\n    }\n  }\n}\n\nobject PreviouslyRecommendedUserIdsPredicate {\n  val ValidStitch: Stitch[PredicateResult.Valid.type] = Stitch.value(PredicateResult.Valid)\n  val AlreadyRecommendedStitch: Stitch[PredicateResult.Invalid] =\n    Stitch.value(PredicateResult.Invalid(Set(FilterReason.AlreadyRecommended)))\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/dismiss/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/dismiss_store\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/dismiss/DismissedCandidatePredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.dismiss\n\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.models.FilterReason.DismissedId\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDismissedUserIds\nimport com.twitter.stitch.Stitch\nimport javax.inject.Singleton\n\n@Singleton\nclass DismissedCandidatePredicate extends Predicate[(HasDismissedUserIds, CandidateUser)] {\n\n  override def apply(pair: (HasDismissedUserIds, CandidateUser)): Stitch[PredicateResult] = {\n\n    val (targetUser, candidate) = pair\n    targetUser.dismissedUserIds\n      .map { dismissedUserIds =>\n        if (!dismissedUserIds.contains(candidate.id)) {\n          DismissedCandidatePredicate.ValidStitch\n        } else {\n          DismissedCandidatePredicate.DismissedStitch\n        }\n      }.getOrElse(DismissedCandidatePredicate.ValidStitch)\n  }\n}\n\nobject DismissedCandidatePredicate {\n  val ValidStitch: Stitch[PredicateResult.Valid.type] = Stitch.value(PredicateResult.Valid)\n  val DismissedStitch: Stitch[PredicateResult.Invalid] =\n    Stitch.value(PredicateResult.Invalid(Set(DismissedId)))\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/dismiss/DismissedCandidatePredicateParams.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.dismiss\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\nobject DismissedCandidatePredicateParams {\n  case object LookBackDuration extends Param[Duration](180.days)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/gizmoduck/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"escherbird/src/scala/com/twitter/escherbird/util/stitchcache\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/cache\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/gizmoduck\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/deciders\",\n        \"stitch/stitch-gizmoduck\",\n        \"util/util-slf4j-api/src/main/scala\",\n        \"util/util-thrift\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/gizmoduck/GizmoduckPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.gizmoduck\n\nimport com.twitter.decider.Decider\nimport com.twitter.decider.RandomRecipient\nimport com.twitter.escherbird.util.stitchcache.StitchCache\nimport com.twitter.finagle.Memcached.Client\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.clients.cache.MemcacheClient\nimport com.twitter.follow_recommendations.common.clients.cache.ThriftBijection\nimport com.twitter.follow_recommendations.common.models.FilterReason._\nimport com.twitter.follow_recommendations.common.models.AddressBookMetadata\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FilterReason\nimport com.twitter.follow_recommendations.common.predicates.gizmoduck.GizmoduckPredicate._\nimport com.twitter.follow_recommendations.common.predicates.gizmoduck.GizmoduckPredicateParams._\nimport com.twitter.follow_recommendations.configapi.deciders.DeciderKey\nimport com.twitter.gizmoduck.thriftscala.LabelValue.BlinkBad\nimport com.twitter.gizmoduck.thriftscala.LabelValue.BlinkWorst\nimport com.twitter.gizmoduck.thriftscala.LabelValue\nimport com.twitter.gizmoduck.thriftscala.LookupContext\nimport com.twitter.gizmoduck.thriftscala.QueryFields\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.gizmoduck.thriftscala.UserResult\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.scrooge.CompactThriftSerializer\nimport com.twitter.spam.rtf.thriftscala.SafetyLevel\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.gizmoduck.Gizmoduck\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.Duration\nimport com.twitter.util.logging.Logging\nimport java.lang.{Long => JLong}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * In this filter, we want to check 4 categories of conditions:\n *   - if candidate is discoverable given that it's from an address-book/phone-book based source\n *   - if candidate is unsuitable based on it's safety sub-fields in gizmoduck\n *   - if candidate is withheld because of country-specific take-down policies\n *   - if candidate is marked as bad/worst based on blink labels\n * We fail close on the query as this is a product-critical filter\n */\n@Singleton\ncase class GizmoduckPredicate @Inject() (\n  gizmoduck: Gizmoduck,\n  client: Client,\n  statsReceiver: StatsReceiver,\n  decider: Decider = Decider.False)\n    extends Predicate[(HasClientContext with HasParams, CandidateUser)]\n    with Logging {\n  private val stats: StatsReceiver = statsReceiver.scope(this.getClass.getName)\n\n  // track # of Gizmoduck predicate queries that yielded valid & invalid predicate results\n  private val validPredicateResultCounter = stats.counter(\"predicate_valid\")\n  private val invalidPredicateResultCounter = stats.counter(\"predicate_invalid\")\n\n  // track # of cases where no Gizmoduck user was found\n  private val noGizmoduckUserCounter = stats.counter(\"no_gizmoduck_user_found\")\n\n  private val gizmoduckCache = StitchCache[JLong, UserResult](\n    maxCacheSize = MaxCacheSize,\n    ttl = CacheTTL,\n    statsReceiver = stats.scope(\"cache\"),\n    underlyingCall = getByUserId\n  )\n\n  // Distributed Twemcache to store UserResult objects keyed on user IDs\n  val bijection = new ThriftBijection[UserResult] {\n    override val serializer = CompactThriftSerializer(UserResult)\n  }\n  val memcacheClient = MemcacheClient[UserResult](\n    client = client,\n    dest = \"/s/cache/frs:twemcaches\",\n    valueBijection = bijection,\n    ttl = CacheTTL,\n    statsReceiver = stats.scope(\"twemcache\")\n  )\n\n  // main method used to apply GizmoduckPredicate to a candidate user\n  override def apply(\n    pair: (HasClientContext with HasParams, CandidateUser)\n  ): Stitch[PredicateResult] = {\n    val (request, candidate) = pair\n    // measure the latency of the getGizmoduckPredicateResult, since this predicate\n    // check is product-critical and relies on querying a core service (Gizmoduck)\n    StatsUtil.profileStitch(\n      getGizmoduckPredicateResult(request, candidate),\n      stats.scope(\"getGizmoduckPredicateResult\")\n    )\n  }\n\n  private def getGizmoduckPredicateResult(\n    request: HasClientContext with HasParams,\n    candidate: CandidateUser\n  ): Stitch[PredicateResult] = {\n    val timeout: Duration = request.params(GizmoduckGetTimeout)\n\n    val deciderKey: String = DeciderKey.EnableGizmoduckCaching.toString\n    val enableDistributedCaching: Boolean = decider.isAvailable(deciderKey, Some(RandomRecipient))\n\n    // try getting an existing UserResult from cache if possible\n    val userResultStitch: Stitch[UserResult] = \n      enableDistributedCaching match {\n        // read from memcache\n        case true => memcacheClient.readThrough(\n          // add a key prefix to address cache key collisions\n          key = \"GizmoduckPredicate\" + candidate.id.toString,\n          underlyingCall = () => getByUserId(candidate.id)\n        )\n        // read from local cache\n        case false => gizmoduckCache.readThrough(candidate.id)\n      }\n\n    val predicateResultStitch = userResultStitch.map {\n      userResult => {\n        val predicateResult = getPredicateResult(request, candidate, userResult)\n        if (enableDistributedCaching) {\n          predicateResult match {\n            case PredicateResult.Valid => \n              stats.scope(\"twemcache\").counter(\"predicate_valid\").incr()\n            case PredicateResult.Invalid(reasons) => \n              stats.scope(\"twemcache\").counter(\"predicate_invalid\").incr()\n          }\n          // log metrics to check if local cache value matches distributed cache value  \n          logPredicateResultEquality(\n            request,\n            candidate,\n            predicateResult\n          )\n        } else {\n          predicateResult match {\n            case PredicateResult.Valid => \n              stats.scope(\"cache\").counter(\"predicate_valid\").incr()\n            case PredicateResult.Invalid(reasons) => \n              stats.scope(\"cache\").counter(\"predicate_invalid\").incr()\n          }\n        }\n        predicateResult\n      }\n    }\n    predicateResultStitch\n      .within(timeout)(DefaultTimer)\n      .rescue { // fail-open when timeout or exception\n        case e: Exception =>\n          stats.scope(\"rescued\").counter(e.getClass.getSimpleName).incr()\n          invalidPredicateResultCounter.incr()\n          Stitch(PredicateResult.Invalid(Set(FailOpen)))\n      }\n  }\n\n  private def logPredicateResultEquality(\n    request: HasClientContext with HasParams,\n    candidate: CandidateUser,\n    predicateResult: PredicateResult\n  ): Unit = {\n    val localCachedUserResult = Option(gizmoduckCache.cache.getIfPresent(candidate.id))\n    if (localCachedUserResult.isDefined) {\n      val localPredicateResult = getPredicateResult(request, candidate, localCachedUserResult.get)\n      localPredicateResult.equals(predicateResult) match {\n        case true => stats.scope(\"has_equal_predicate_value\").counter(\"true\").incr()\n        case false => stats.scope(\"has_equal_predicate_value\").counter(\"false\").incr()\n      }\n    } else {\n      stats.scope(\"has_equal_predicate_value\").counter(\"undefined\").incr()\n    }\n  }\n\n  // method to get PredicateResult from UserResult\n  def getPredicateResult(\n    request: HasClientContext with HasParams,\n    candidate: CandidateUser,\n    userResult: UserResult,\n  ): PredicateResult = {\n    userResult.user match {\n      case Some(user) =>\n        val abPbReasons = getAbPbReason(user, candidate.getAddressBookMetadata)\n        val safetyReasons = getSafetyReasons(user)\n        val countryTakedownReasons = getCountryTakedownReasons(user, request.getCountryCode)\n        val blinkReasons = getBlinkReasons(user)\n        val allReasons =\n          abPbReasons ++ safetyReasons ++ countryTakedownReasons ++ blinkReasons\n        if (allReasons.nonEmpty) {\n          invalidPredicateResultCounter.incr()\n          PredicateResult.Invalid(allReasons)\n        } else {\n          validPredicateResultCounter.incr()\n          PredicateResult.Valid\n        }\n      case None =>\n        noGizmoduckUserCounter.incr()\n        invalidPredicateResultCounter.incr()\n        PredicateResult.Invalid(Set(NoUser))\n    }\n  }\n\n  private def getByUserId(userId: JLong): Stitch[UserResult] = {\n    StatsUtil.profileStitch(\n      gizmoduck.getById(userId = userId, queryFields = queryFields, context = lookupContext),\n      stats.scope(\"getByUserId\")\n    )\n  }\n}\n\nobject GizmoduckPredicate {\n\n  private[gizmoduck] val lookupContext: LookupContext =\n    LookupContext(`includeDeactivated` = true, `safetyLevel` = Some(SafetyLevel.Recommendations))\n\n  private[gizmoduck] val queryFields: Set[QueryFields] =\n    Set(\n      QueryFields.Discoverability, // needed for Address Book / Phone Book discoverability checks in getAbPbReason\n      QueryFields.Safety, // needed for user state safety checks in getSafetyReasons, getCountryTakedownReasons\n      QueryFields.Labels, // needed for user label checks in getBlinkReasons\n      QueryFields.Takedowns // needed for checking takedown labels for a user in getCountryTakedownReasons\n    )\n\n  private[gizmoduck] val BlinkLabels: Set[LabelValue] = Set(BlinkBad, BlinkWorst)\n\n  private[gizmoduck] def getAbPbReason(\n    user: User,\n    abMetadataOpt: Option[AddressBookMetadata]\n  ): Set[FilterReason] = {\n    (for {\n      discoverability <- user.discoverability\n      abMetadata <- abMetadataOpt\n    } yield {\n      val AddressBookMetadata(fwdPb, rvPb, fwdAb, rvAb) = abMetadata\n      val abReason: Set[FilterReason] =\n        if ((!discoverability.discoverableByEmail) && (fwdAb || rvAb))\n          Set(AddressBookUndiscoverable)\n        else Set.empty\n      val pbReason: Set[FilterReason] =\n        if ((!discoverability.discoverableByMobilePhone) && (fwdPb || rvPb))\n          Set(PhoneBookUndiscoverable)\n        else Set.empty\n      abReason ++ pbReason\n    }).getOrElse(Set.empty)\n  }\n\n  private[gizmoduck] def getSafetyReasons(user: User): Set[FilterReason] = {\n    user.safety\n      .map { s =>\n        val deactivatedReason: Set[FilterReason] =\n          if (s.deactivated) Set(Deactivated) else Set.empty\n        val suspendedReason: Set[FilterReason] = if (s.suspended) Set(Suspended) else Set.empty\n        val restrictedReason: Set[FilterReason] = if (s.restricted) Set(Restricted) else Set.empty\n        val nsfwUserReason: Set[FilterReason] = if (s.nsfwUser) Set(NsfwUser) else Set.empty\n        val nsfwAdminReason: Set[FilterReason] = if (s.nsfwAdmin) Set(NsfwAdmin) else Set.empty\n        val isProtectedReason: Set[FilterReason] = if (s.isProtected) Set(IsProtected) else Set.empty\n        deactivatedReason ++ suspendedReason ++ restrictedReason ++ nsfwUserReason ++ nsfwAdminReason ++ isProtectedReason\n      }.getOrElse(Set.empty)\n  }\n\n  private[gizmoduck] def getCountryTakedownReasons(\n    user: User,\n    countryCodeOpt: Option[String]\n  ): Set[FilterReason] = {\n    (for {\n      safety <- user.safety.toSeq\n      if safety.hasTakedown\n      takedowns <- user.takedowns.toSeq\n      takedownCountry <- takedowns.countryCodes\n      requestingCountry <- countryCodeOpt\n      if takedownCountry.toLowerCase == requestingCountry.toLowerCase\n    } yield Set(CountryTakedown(takedownCountry.toLowerCase))).flatten.toSet\n  }\n\n  private[gizmoduck] def getBlinkReasons(user: User): Set[FilterReason] = {\n    user.labels\n      .map(_.labels.map(_.labelValue))\n      .getOrElse(Nil)\n      .exists(BlinkLabels.contains)\n    for {\n      labels <- user.labels.toSeq\n      label <- labels.labels\n      if (BlinkLabels.contains(label.labelValue))\n    } yield Set(Blink)\n  }.flatten.toSet\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/gizmoduck/GizmoduckPredicateCache.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.gizmoduck\n\nimport java.util.concurrent.TimeUnit\n\nimport com.google.common.base.Ticker\nimport com.google.common.cache.CacheBuilder\nimport com.google.common.cache.Cache\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.Time\nimport com.twitter.util.Duration\n\n/**\n * In-memory cache used for caching GizmoduckPredicate query calls in\n * com.twitter.follow_recommendations.common.predicates.gizmoduck.GizmoduckPredicate.\n * \n * References the cache implementation in com.twitter.escherbird.util.stitchcache,\n * but without the underlying Stitch call.\n */\nobject GizmoduckPredicateCache {\n\n  private[GizmoduckPredicateCache] class TimeTicker extends Ticker {\n    override def read(): Long = Time.now.inNanoseconds\n  }\n\n  def apply[K, V](\n    maxCacheSize: Int,\n    ttl: Duration,\n    statsReceiver: StatsReceiver\n  ): Cache[K, V] = {\n\n    val cache: Cache[K, V] =\n      CacheBuilder\n        .newBuilder()\n        .maximumSize(maxCacheSize)\n        .asInstanceOf[CacheBuilder[K, V]]\n        .expireAfterWrite(ttl.inSeconds, TimeUnit.SECONDS)\n        .recordStats()\n        .ticker(new TimeTicker())\n        .build()\n\n    // metrics for tracking cache usage\n    statsReceiver.provideGauge(\"cache_size\") { cache.size.toFloat }\n    statsReceiver.provideGauge(\"cache_hits\") { cache.stats.hitCount.toFloat }\n    statsReceiver.provideGauge(\"cache_misses\") { cache.stats.missCount.toFloat }\n    statsReceiver.provideGauge(\"cache_hit_rate\") { cache.stats.hitRate.toFloat }\n    statsReceiver.provideGauge(\"cache_evictions\") { cache.stats.evictionCount.toFloat }\n\n    cache\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/gizmoduck/GizmoduckPredicateFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.gizmoduck\n\nimport com.twitter.follow_recommendations.common.predicates.gizmoduck.GizmoduckPredicateParams._\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.util.Duration\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GizmoduckPredicateFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val durationFSParams: Seq[FSBoundedParam[Duration] with HasDurationConversion] = Seq(\n    GizmoduckGetTimeout\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/gizmoduck/GizmoduckPredicateParams.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.gizmoduck\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.util.Duration\nimport com.twitter.conversions.DurationOps._\n\nobject GizmoduckPredicateParams {\n  case object GizmoduckGetTimeout\n      extends FSBoundedParam[Duration](\n        name = \"gizmoduck_predicate_timeout_in_millis\",\n        default = 200.millisecond,\n        min = 1.millisecond,\n        max = 500.millisecond)\n      with HasDurationConversion {\n    override def durationConversion: DurationConversion = DurationConversion.FromMillis\n  }\n  val MaxCacheSize: Int = 250000\n  val CacheTTL: Duration = Duration.fromHours(6)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/health/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"escherbird/src/scala/com/twitter/escherbird/util/stitchcache\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/cache\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"strato/config/columns/hss/user_signals/api:api-strato-client\",\n        \"util/util-slf4j-api/src/main/scala\",\n        \"util/util-thrift\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/health/HssPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.hss\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FilterReason\nimport com.twitter.follow_recommendations.common.models.FilterReason.FailOpen\nimport com.twitter.hss.api.thriftscala.SignalValue\nimport com.twitter.hss.api.thriftscala.UserHealthSignal.AgathaCseDouble\nimport com.twitter.hss.api.thriftscala.UserHealthSignal.NsfwAgathaUserScoreDouble\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.hss.user_signals.api.HealthSignalsOnUserClientColumn\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.logging.Logging\nimport com.twitter.util.Duration\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Filter out candidates based on Health Signal Store (HSS) health signals\n */\n@Singleton\ncase class HssPredicate @Inject() (\n  healthSignalsOnUserClientColumn: HealthSignalsOnUserClientColumn,\n  statsReceiver: StatsReceiver)\n    extends Predicate[(HasClientContext with HasParams, CandidateUser)]\n    with Logging {\n\n  private val stats: StatsReceiver = statsReceiver.scope(this.getClass.getName)\n\n  override def apply(\n    pair: (HasClientContext with HasParams, CandidateUser)\n  ): Stitch[PredicateResult] = {\n    val (request, candidate) = pair\n    StatsUtil.profileStitch(\n      getHssPredicateResult(request, candidate),\n      stats.scope(\"getHssPredicateResult\")\n    )\n  }\n\n  private def getHssPredicateResult(\n    request: HasClientContext with HasParams,\n    candidate: CandidateUser\n  ): Stitch[PredicateResult] = {\n\n    val hssCseScoreThreshold: Double = request.params(HssPredicateParams.HssCseScoreThreshold)\n    val hssNsfwScoreThreshold: Double = request.params(HssPredicateParams.HssNsfwScoreThreshold)\n    val timeout: Duration = request.params(HssPredicateParams.HssApiTimeout)\n\n    healthSignalsOnUserClientColumn.fetcher\n      .fetch(candidate.id, Seq(AgathaCseDouble, NsfwAgathaUserScoreDouble))\n      .map { fetchResult =>\n        fetchResult.v match {\n          case Some(response) =>\n            val agathaCseScoreDouble: Double = userHealthSignalValueToDoubleOpt(\n              response.signalValues.get(AgathaCseDouble)).getOrElse(0d)\n            val agathaNsfwScoreDouble: Double = userHealthSignalValueToDoubleOpt(\n              response.signalValues.get(NsfwAgathaUserScoreDouble)).getOrElse(0d)\n\n            stats.stat(\"agathaCseScoreDistribution\").add(agathaCseScoreDouble.toFloat)\n            stats.stat(\"agathaNsfwScoreDistribution\").add(agathaNsfwScoreDouble.toFloat)\n\n            /**\n             * Only filter out the candidate when it has both high Agatha CSE score and NSFW score, as the Agatha CSE\n             * model is an old one that may not be precise or have high recall.\n             */\n            if (agathaCseScoreDouble >= hssCseScoreThreshold && agathaNsfwScoreDouble >= hssNsfwScoreThreshold) {\n              PredicateResult.Invalid(Set(FilterReason.HssSignal))\n            } else {\n              PredicateResult.Valid\n            }\n          case None =>\n            PredicateResult.Valid\n        }\n      }\n      .within(timeout)(DefaultTimer)\n      .rescue {\n        case e: Exception =>\n          stats.scope(\"rescued\").counter(e.getClass.getSimpleName).incr()\n          Stitch(PredicateResult.Invalid(Set(FailOpen)))\n      }\n  }\n\n  private def userHealthSignalValueToDoubleOpt(signalValue: Option[SignalValue]): Option[Double] = {\n    signalValue match {\n      case Some(SignalValue.DoubleValue(value)) => Some(value)\n      case _ => None\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/health/HssPredicateFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.hss\n\nimport com.twitter.follow_recommendations.common.predicates.hss.HssPredicateParams._\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.util.Duration\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HssPredicateFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val doubleFSParams: Seq[FSBoundedParam[Double]] = Seq(\n    HssCseScoreThreshold,\n    HssNsfwScoreThreshold,\n  )\n\n  override val durationFSParams: Seq[FSBoundedParam[Duration] with HasDurationConversion] = Seq(\n    HssApiTimeout\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/health/HssPredicateParams.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.hss\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.util.Duration\n\nobject HssPredicateParams {\n  object HssCseScoreThreshold\n      extends FSBoundedParam[Double](\n        \"hss_predicate_cse_score_threshold\",\n        default = 0.992d,\n        min = 0.0d,\n        max = 1.0d)\n\n  object HssNsfwScoreThreshold\n      extends FSBoundedParam[Double](\n        \"hss_predicate_nsfw_score_threshold\",\n        default = 1.5d,\n        min = -100.0d,\n        max = 100.0d)\n\n  object HssApiTimeout\n      extends FSBoundedParam[Duration](\n        name = \"hss_predicate_timeout_in_millis\",\n        default = 200.millisecond,\n        min = 1.millisecond,\n        max = 500.millisecond)\n      with HasDurationConversion {\n    override def durationConversion: DurationConversion = DurationConversion.FromMillis\n  }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/sgs/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/socialgraph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/sgs/InvalidRelationshipPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.sgs\n\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FilterReason\nimport com.twitter.follow_recommendations.common.models.HasInvalidRelationshipUserIds\nimport com.twitter.stitch.Stitch\nimport javax.inject.Singleton\n\n@Singleton\nclass InvalidRelationshipPredicate\n    extends Predicate[(HasInvalidRelationshipUserIds, CandidateUser)] {\n\n  override def apply(\n    pair: (HasInvalidRelationshipUserIds, CandidateUser)\n  ): Stitch[PredicateResult] = {\n\n    val (targetUser, candidate) = pair\n    targetUser.invalidRelationshipUserIds match {\n      case Some(users) =>\n        if (!users.contains(candidate.id)) {\n          InvalidRelationshipPredicate.ValidStitch\n        } else {\n          Stitch.value(InvalidRelationshipPredicate.InvalidRelationshipStitch)\n        }\n      case None => Stitch.value(PredicateResult.Valid)\n    }\n  }\n}\n\nobject InvalidRelationshipPredicate {\n  val ValidStitch: Stitch[PredicateResult.Valid.type] = Stitch.value(PredicateResult.Valid)\n  val InvalidRelationshipStitch: PredicateResult.Invalid =\n    PredicateResult.Invalid(Set(FilterReason.InvalidRelationship))\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/sgs/RecentFollowingPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.sgs\n\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FilterReason\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedUserIds\nimport com.twitter.stitch.Stitch\nimport javax.inject.Singleton\n\n@Singleton\nclass RecentFollowingPredicate extends Predicate[(HasRecentFollowedUserIds, CandidateUser)] {\n\n  override def apply(pair: (HasRecentFollowedUserIds, CandidateUser)): Stitch[PredicateResult] = {\n\n    val (targetUser, candidate) = pair\n    targetUser.recentFollowedUserIdsSet match {\n      case Some(users) =>\n        if (!users.contains(candidate.id)) {\n          RecentFollowingPredicate.ValidStitch\n        } else {\n          RecentFollowingPredicate.AlreadyFollowedStitch\n        }\n      case None => RecentFollowingPredicate.ValidStitch\n    }\n  }\n}\n\nobject RecentFollowingPredicate {\n  val ValidStitch: Stitch[PredicateResult.Valid.type] = Stitch.value(PredicateResult.Valid)\n  val AlreadyFollowedStitch: Stitch[PredicateResult.Invalid] =\n    Stitch.value(PredicateResult.Invalid(Set(FilterReason.AlreadyFollowed)))\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/sgs/SgsPredicateFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.sgs\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.util.Duration\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SgsPredicateFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val durationFSParams: Seq[FSBoundedParam[Duration] with HasDurationConversion] = Seq(\n    SgsPredicateParams.SgsRelationshipsPredicateTimeout\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/sgs/SgsPredicateParams.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.sgs\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.util.Duration\nimport com.twitter.conversions.DurationOps._\n\nobject SgsPredicateParams {\n  case object SgsRelationshipsPredicateTimeout\n      extends FSBoundedParam[Duration](\n        name = \"sgs_predicate_relationships_timeout_in_millis\",\n        default = 300.millisecond,\n        min = 1.millisecond,\n        max = 1000.millisecond)\n      with HasDurationConversion {\n    override def durationConversion: DurationConversion = DurationConversion.FromMillis\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/sgs/SgsRelationshipsByUserIdPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.sgs\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FilterReason.InvalidRelationshipTypes\nimport com.twitter.socialgraph.thriftscala.ExistsRequest\nimport com.twitter.socialgraph.thriftscala.ExistsResult\nimport com.twitter.socialgraph.thriftscala.LookupContext\nimport com.twitter.socialgraph.thriftscala.Relationship\nimport com.twitter.socialgraph.thriftscala.RelationshipType\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.SocialGraph\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nclass SgsRelationshipsByUserIdPredicate(\n  socialGraph: SocialGraph,\n  relationshipMappings: Seq[RelationshipMapping],\n  statsReceiver: StatsReceiver)\n    extends Predicate[(Option[Long], CandidateUser)]\n    with Logging {\n  private val InvalidFromPrimaryCandidateSourceName = \"invalid_from_primary_candidate_source\"\n  private val InvalidFromCandidateSourceName = \"invalid_from_candidate_source\"\n  private val NoPrimaryCandidateSource = \"no_primary_candidate_source\"\n\n  private val stats: StatsReceiver = statsReceiver.scope(this.getClass.getName)\n\n  override def apply(\n    pair: (Option[Long], CandidateUser)\n  ): Stitch[PredicateResult] = {\n    val (idOpt, candidate) = pair\n    val relationships = relationshipMappings.map { relationshipMapping: RelationshipMapping =>\n      Relationship(\n        relationshipMapping.relationshipType,\n        relationshipMapping.includeBasedOnRelationship)\n    }\n    idOpt\n      .map { id: Long =>\n        val existsRequest = ExistsRequest(\n          id,\n          candidate.id,\n          relationships = relationships,\n          context = SgsRelationshipsByUserIdPredicate.UnionLookupContext\n        )\n        socialGraph\n          .exists(existsRequest).map { existsResult: ExistsResult =>\n            if (existsResult.exists) {\n              candidate.getPrimaryCandidateSource match {\n                case Some(candidateSource) =>\n                  stats\n                    .scope(InvalidFromPrimaryCandidateSourceName).counter(\n                      candidateSource.name).incr()\n                case None =>\n                  stats\n                    .scope(InvalidFromPrimaryCandidateSourceName).counter(\n                      NoPrimaryCandidateSource).incr()\n              }\n              candidate.getCandidateSources.foreach({\n                case (candidateSource, _) =>\n                  stats\n                    .scope(InvalidFromCandidateSourceName).counter(candidateSource.name).incr()\n              })\n              PredicateResult.Invalid(Set(InvalidRelationshipTypes(relationshipMappings\n                .map { relationshipMapping: RelationshipMapping =>\n                  relationshipMapping.relationshipType\n                }.mkString(\", \"))))\n            } else {\n              PredicateResult.Valid\n            }\n          }\n      }\n      // if no user id is present, return true by default\n      .getOrElse(Stitch.value(PredicateResult.Valid))\n  }\n}\n\nobject SgsRelationshipsByUserIdPredicate {\n  // OR Operation\n  @VisibleForTesting\n  private[follow_recommendations] val UnionLookupContext = Some(\n    LookupContext(performUnion = Some(true)))\n}\n\n@Singleton\nclass ExcludeNonFollowersSgsPredicate @Inject() (\n  socialGraph: SocialGraph,\n  statsReceiver: StatsReceiver)\n    extends SgsRelationshipsByUserIdPredicate(\n      socialGraph,\n      Seq(RelationshipMapping(RelationshipType.FollowedBy, includeBasedOnRelationship = false)),\n      statsReceiver)\n\n@Singleton\nclass ExcludeNonFollowingSgsPredicate @Inject() (\n  socialGraph: SocialGraph,\n  statsReceiver: StatsReceiver)\n    extends SgsRelationshipsByUserIdPredicate(\n      socialGraph,\n      Seq(RelationshipMapping(RelationshipType.Following, includeBasedOnRelationship = false)),\n      statsReceiver)\n\n@Singleton\nclass ExcludeFollowingSgsPredicate @Inject() (\n  socialGraph: SocialGraph,\n  statsReceiver: StatsReceiver)\n    extends SgsRelationshipsByUserIdPredicate(\n      socialGraph,\n      Seq(RelationshipMapping(RelationshipType.Following, includeBasedOnRelationship = true)),\n      statsReceiver)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/sgs/SgsRelationshipsPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.sgs\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasProfileId\nimport com.twitter.follow_recommendations.common.models.FilterReason.FailOpen\nimport com.twitter.follow_recommendations.common.models.FilterReason.InvalidRelationshipTypes\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.socialgraph.thriftscala.ExistsRequest\nimport com.twitter.socialgraph.thriftscala.ExistsResult\nimport com.twitter.socialgraph.thriftscala.LookupContext\nimport com.twitter.socialgraph.thriftscala.Relationship\nimport com.twitter.socialgraph.thriftscala.RelationshipType\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.SocialGraph\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.TimeoutException\nimport com.twitter.util.logging.Logging\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase class RelationshipMapping(\n  relationshipType: RelationshipType,\n  includeBasedOnRelationship: Boolean)\n\nclass SgsRelationshipsPredicate(\n  socialGraph: SocialGraph,\n  relationshipMappings: Seq[RelationshipMapping],\n  statsReceiver: StatsReceiver = NullStatsReceiver)\n    extends Predicate[(HasClientContext with HasParams, CandidateUser)]\n    with Logging {\n\n  private val stats: StatsReceiver = statsReceiver.scope(this.getClass.getSimpleName)\n\n  override def apply(\n    pair: (HasClientContext with HasParams, CandidateUser)\n  ): Stitch[PredicateResult] = {\n    val (target, candidate) = pair\n    val timeout = target.params(SgsPredicateParams.SgsRelationshipsPredicateTimeout)\n    SgsRelationshipsPredicate\n      .extractUserId(target)\n      .map { id =>\n        val relationships = relationshipMappings.map { relationshipMapping: RelationshipMapping =>\n          Relationship(\n            relationshipMapping.relationshipType,\n            relationshipMapping.includeBasedOnRelationship)\n        }\n        val existsRequest = ExistsRequest(\n          id,\n          candidate.id,\n          relationships = relationships,\n          context = SgsRelationshipsPredicate.UnionLookupContext\n        )\n        socialGraph\n          .exists(existsRequest).map { existsResult: ExistsResult =>\n            if (existsResult.exists) {\n              PredicateResult.Invalid(Set(InvalidRelationshipTypes(relationshipMappings\n                .map { relationshipMapping: RelationshipMapping =>\n                  relationshipMapping.relationshipType\n                }.mkString(\", \"))))\n            } else {\n              PredicateResult.Valid\n            }\n          }\n          .within(timeout)(com.twitter.finagle.util.DefaultTimer)\n      }\n      // if no user id is present, return true by default\n      .getOrElse(Stitch.value(PredicateResult.Valid))\n      .rescue {\n        case e: TimeoutException =>\n          stats.counter(\"timeout\").incr()\n          Stitch(PredicateResult.Invalid(Set(FailOpen)))\n        case e: Exception =>\n          stats.counter(e.getClass.getSimpleName).incr()\n          Stitch(PredicateResult.Invalid(Set(FailOpen)))\n      }\n\n  }\n}\n\nobject SgsRelationshipsPredicate {\n  // OR Operation\n  @VisibleForTesting\n  private[follow_recommendations] val UnionLookupContext = Some(\n    LookupContext(performUnion = Some(true)))\n\n  private def extractUserId(target: HasClientContext with HasParams): Option[Long] = target match {\n    case profRequest: HasProfileId => Some(profRequest.profileId)\n    case userRequest: HasClientContext with HasParams => userRequest.getOptionalUserId\n    case _ => None\n  }\n}\n\n@Singleton\nclass InvalidTargetCandidateRelationshipTypesPredicate @Inject() (\n  socialGraph: SocialGraph)\n    extends SgsRelationshipsPredicate(\n      socialGraph,\n      InvalidRelationshipTypesPredicate.InvalidRelationshipTypes) {}\n\n@Singleton\nclass NoteworthyAccountsSgsPredicate @Inject() (\n  socialGraph: SocialGraph)\n    extends SgsRelationshipsPredicate(\n      socialGraph,\n      InvalidRelationshipTypesPredicate.NoteworthyAccountsInvalidRelationshipTypes)\n\nobject InvalidRelationshipTypesPredicate {\n\n  val InvalidRelationshipTypesExcludeFollowing: Seq[RelationshipMapping] = Seq(\n    RelationshipMapping(RelationshipType.HideRecommendations, true),\n    RelationshipMapping(RelationshipType.Blocking, true),\n    RelationshipMapping(RelationshipType.BlockedBy, true),\n    RelationshipMapping(RelationshipType.Muting, true),\n    RelationshipMapping(RelationshipType.MutedBy, true),\n    RelationshipMapping(RelationshipType.ReportedAsSpam, true),\n    RelationshipMapping(RelationshipType.ReportedAsSpamBy, true),\n    RelationshipMapping(RelationshipType.ReportedAsAbuse, true),\n    RelationshipMapping(RelationshipType.ReportedAsAbuseBy, true)\n  )\n\n  val InvalidRelationshipTypes: Seq[RelationshipMapping] = Seq(\n    RelationshipMapping(RelationshipType.FollowRequestOutgoing, true),\n    RelationshipMapping(RelationshipType.Following, true),\n    RelationshipMapping(\n      RelationshipType.UsedToFollow,\n      true\n    ) // this data is accessible for 90 days.\n  ) ++ InvalidRelationshipTypesExcludeFollowing\n\n  val NoteworthyAccountsInvalidRelationshipTypes: Seq[RelationshipMapping] = Seq(\n    RelationshipMapping(RelationshipType.Blocking, true),\n    RelationshipMapping(RelationshipType.BlockedBy, true),\n    RelationshipMapping(RelationshipType.Muting, true),\n    RelationshipMapping(RelationshipType.MutedBy, true),\n    RelationshipMapping(RelationshipType.ReportedAsSpam, true),\n    RelationshipMapping(RelationshipType.ReportedAsSpamBy, true),\n    RelationshipMapping(RelationshipType.ReportedAsAbuse, true),\n    RelationshipMapping(RelationshipType.ReportedAsAbuseBy, true)\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/user_activity/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/cache\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/deciders\",\n        \"strato/config/columns/onboarding:onboarding-strato-client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/user_activity/UserActivityPredicate.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.user_activity\n\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.decider.Decider\nimport com.twitter.decider.RandomRecipient\nimport com.twitter.finagle.Memcached.Client\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.PredicateResult\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.follow_recommendations.common.clients.cache.MemcacheClient\nimport com.twitter.follow_recommendations.common.clients.cache.ThriftEnumOptionBijection\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FilterReason\nimport com.twitter.follow_recommendations.configapi.deciders.DeciderKey\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.UserRecommendabilityWithLongKeysOnUserClientColumn\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nabstract case class UserStateActivityPredicate(\n  userRecommendabilityClient: UserRecommendabilityWithLongKeysOnUserClientColumn,\n  validCandidateStates: Set[UserState],\n  client: Client,\n  statsReceiver: StatsReceiver,\n  decider: Decider = Decider.False)\n    extends Predicate[(HasParams with HasClientContext, CandidateUser)] {\n\n  private val stats: StatsReceiver = statsReceiver.scope(this.getClass.getSimpleName)\n\n  // client to memcache cluster\n  val bijection = new ThriftEnumOptionBijection[UserState](UserState.apply)\n  val memcacheClient = MemcacheClient[Option[UserState]](\n    client = client,\n    dest = \"/s/cache/follow_recos_service:twemcaches\",\n    valueBijection = bijection,\n    ttl = UserActivityPredicateParams.CacheTTL,\n    statsReceiver = stats.scope(\"twemcache\")\n  )\n\n  override def apply(\n    targetAndCandidate: (HasParams with HasClientContext, CandidateUser)\n  ): Stitch[PredicateResult] = {\n    val userRecommendabilityFetcher = userRecommendabilityClient.fetcher\n    val (_, candidate) = targetAndCandidate\n\n    val deciderKey: String = DeciderKey.EnableExperimentalCaching.toString\n    val enableDistributedCaching: Boolean = decider.isAvailable(deciderKey, Some(RandomRecipient))\n    val userStateStitch: Stitch[Option[UserState]] = \n      enableDistributedCaching match {\n        case true => {\n          memcacheClient.readThrough(\n            // add a key prefix to address cache key collisions\n            key = \"UserActivityPredicate\" + candidate.id.toString,\n            underlyingCall = () => queryUserRecommendable(candidate.id)\n          )\n        }\n        case false => queryUserRecommendable(candidate.id)\n      }\n    val resultStitch: Stitch[PredicateResult] = \n      userStateStitch.map { userStateOpt =>\n        userStateOpt match {\n          case Some(userState) => {\n            if (validCandidateStates.contains(userState)) {\n              PredicateResult.Valid\n            } else {\n              PredicateResult.Invalid(Set(FilterReason.MinStateNotMet))\n            }\n          }\n          case None => {\n            PredicateResult.Invalid(Set(FilterReason.MissingRecommendabilityData))\n          }\n        }\n      }\n    \n    StatsUtil.profileStitch(resultStitch, stats.scope(\"apply\"))\n      .rescue {\n        case e: Exception =>\n          stats.scope(\"rescued\").counter(e.getClass.getSimpleName).incr()\n          Stitch(PredicateResult.Invalid(Set(FilterReason.FailOpen)))\n      }\n  }\n\n  def queryUserRecommendable(\n    userId: Long\n  ): Stitch[Option[UserState]] = {\n    val userRecommendabilityFetcher = userRecommendabilityClient.fetcher\n    userRecommendabilityFetcher.fetch(userId).map { userCandidate => \n      userCandidate.v.flatMap(_.userState)\n    }\n  }\n}\n\n@Singleton\nclass MinStateUserActivityPredicate @Inject() (\n  userRecommendabilityClient: UserRecommendabilityWithLongKeysOnUserClientColumn,\n  client: Client,\n  statsReceiver: StatsReceiver)\n    extends UserStateActivityPredicate(\n      userRecommendabilityClient,\n      Set(\n        UserState.Light,\n        UserState.HeavyNonTweeter,\n        UserState.MediumNonTweeter,\n        UserState.HeavyTweeter,\n        UserState.MediumTweeter\n      ),\n      client,\n      statsReceiver\n    )\n\n@Singleton\nclass AllTweeterUserActivityPredicate @Inject() (\n  userRecommendabilityClient: UserRecommendabilityWithLongKeysOnUserClientColumn,\n  client: Client,\n  statsReceiver: StatsReceiver)\n    extends UserStateActivityPredicate(\n      userRecommendabilityClient,\n      Set(\n        UserState.HeavyTweeter,\n        UserState.MediumTweeter\n      ),\n      client,\n      statsReceiver\n    )\n\n@Singleton\nclass HeavyTweeterUserActivityPredicate @Inject() (\n  userRecommendabilityClient: UserRecommendabilityWithLongKeysOnUserClientColumn,\n  client: Client,\n  statsReceiver: StatsReceiver)\n    extends UserStateActivityPredicate(\n      userRecommendabilityClient,\n      Set(\n        UserState.HeavyTweeter\n      ),\n      client,\n      statsReceiver\n    )\n\n@Singleton\nclass NonNearZeroUserActivityPredicate @Inject() (\n  userRecommendabilityClient: UserRecommendabilityWithLongKeysOnUserClientColumn,\n  client: Client,\n  statsReceiver: StatsReceiver)\n    extends UserStateActivityPredicate(\n      userRecommendabilityClient,\n      Set(\n        UserState.New,\n        UserState.VeryLight,\n        UserState.Light,\n        UserState.MediumNonTweeter,\n        UserState.MediumTweeter,\n        UserState.HeavyNonTweeter,\n        UserState.HeavyTweeter\n      ),\n      client,\n      statsReceiver\n    )\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/user_activity/UserActivityPredicateParams.scala",
    "content": "package com.twitter.follow_recommendations.common.predicates.user_activity\n\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.util.Duration\n\nobject UserActivityPredicateParams {\n  case object HeavyTweeterEnabled\n      extends FSParam[Boolean](\"user_activity_predicate_heavy_tweeter_enabled\", false)\n  val CacheTTL: Duration = Duration.fromHours(6)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common/AdhocScoreModificationType.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.common\n\n/**\n * To manage the extent of adhoc score modifications, we set a hard limit that from each of the\n * types below *ONLY ONE* adhoc scorer can be applied to candidates' scores. More details about the\n * usage is available in [[AdhocRanker]]\n */\n\nobject AdhocScoreModificationType extends Enumeration {\n  type AdhocScoreModificationType = Value\n\n  // This type of scorer increases the score of a subset of candidates through various policies.\n  val BoostingScorer: AdhocScoreModificationType = Value(\"boosting\")\n\n  // This type of scorer shuffles candidates randomly according to some distribution.\n  val WeightedRandomSamplingScorer: AdhocScoreModificationType = Value(\"weighted_random_sampling\")\n\n  // This is added solely for testing purposes and should not be used in production.\n  val InvalidAdhocScorer: AdhocScoreModificationType = Value(\"invalid_adhoc_scorer\")\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common/DedupCandidates.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.common\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport scala.collection.mutable\n\nobject DedupCandidates {\n  def apply[C <: UniversalNoun[Long]](input: Seq[C]): Seq[C] = {\n    val seen = mutable.HashSet[Long]()\n    input.filter { candidate => seen.add(candidate.id) }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common/RankerId.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.common\n\nobject RankerId extends Enumeration {\n  type RankerId = Value\n\n  val RandomRanker: RankerId = Value(\"random\")\n  // The production PostNUX ML warm-start auto-retraining model ranker\n  val PostNuxProdRanker: RankerId = Value(\"postnux_prod\")\n  val None: RankerId = Value(\"none\")\n\n  // Sampling from the Placket-Luce distribution. Applied after ranker step. Its ranker id is mainly used for logging.\n  val PlacketLuceSamplingTransformer: RankerId = Value(\"placket_luce_sampling_transformer\")\n\n  def getRankerByName(name: String): Option[RankerId] =\n    RankerId.values.toSeq.find(_.equals(Value(name)))\n\n}\n\n/**\n * ML model based heavy ranker ids.\n */\nobject ModelBasedHeavyRankerId {\n  import RankerId._\n  val HeavyRankerIds: Set[String] = Set(\n    PostNuxProdRanker.toString,\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/fatigue_ranker/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/impression_store\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/utils\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/fatigue_ranker/ImpressionBasedFatigueRanker.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.fatigue_ranker\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.Ranker\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasWtfImpressions\nimport com.twitter.follow_recommendations.common.models.WtfImpression\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId.RankerId\nimport com.twitter.follow_recommendations.common.rankers.utils.Utils\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.servo.util.MemoizingStatsReceiver\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.Time\n\n/**\n * Ranks candidates based on the given weights for each algorithm while preserving the ranks inside each algorithm.\n * Reorders the ranked list based on recent impressions from recentImpressionRepo\n *\n * Note that the penalty is added to the rank of each candidate. To make producer-side experiments\n * with multiple rankers possible, we modify the scores for each candidate and ranker as:\n *     NewScore(C, R) = -(Rank(C, R) + Impression(C, U) x FatigueFactor),\n * where C is a candidate, R a ranker and U the target user.\n * Note also that fatigue penalty is independent of any of the rankers.\n */\nclass ImpressionBasedFatigueRanker[\n  Target <: HasClientContext with HasDisplayLocation with HasParams with HasWtfImpressions\n](\n  fatigueFactor: Int,\n  statsReceiver: StatsReceiver)\n    extends Ranker[Target, CandidateUser] {\n\n  val name: String = this.getClass.getSimpleName\n  val stats = statsReceiver.scope(\"impression_based_fatigue_ranker\")\n  val droppedStats: MemoizingStatsReceiver = new MemoizingStatsReceiver(stats.scope(\"hard_drops\"))\n  val impressionStats: StatsReceiver = stats.scope(\"wtf_impressions\")\n  val noImpressionCounter: Counter = impressionStats.counter(\"no_impressions\")\n  val oldestImpressionStat: Stat = impressionStats.stat(\"oldest_sec\")\n\n  override def rank(target: Target, candidates: Seq[CandidateUser]): Stitch[Seq[CandidateUser]] = {\n    StatsUtil.profileStitch(\n      Stitch.value(rankCandidates(target, candidates)),\n      stats.scope(\"rank\")\n    )\n  }\n\n  private def trackTimeSinceOldestImpression(impressions: Seq[WtfImpression]): Unit = {\n    val timeSinceOldest = Time.now - impressions.map(_.latestTime).min\n    oldestImpressionStat.add(timeSinceOldest.inSeconds)\n  }\n\n  private def rankCandidates(\n    target: Target,\n    candidates: Seq[CandidateUser]\n  ): Seq[CandidateUser] = {\n    target.wtfImpressions\n      .map { wtfImpressions =>\n        if (wtfImpressions.isEmpty) {\n          noImpressionCounter.incr()\n          candidates\n        } else {\n          val rankerIds =\n            candidates.flatMap(_.scores.map(_.scores.flatMap(_.rankerId))).flatten.sorted.distinct\n\n          /**\n           * In below we create a Map from each CandidateUser's ID to a Map from each Ranker that\n           * the user has a score for, and candidate's corresponding rank when candidates are sorted\n           * by that Ranker (Only candidates who have this Ranker are considered for ranking).\n           */\n          val candidateRanks: Map[Long, Map[RankerId, Int]] = rankerIds\n            .flatMap { rankerId =>\n              // Candidates with no scores from this Ranker is first removed to calculate ranks.\n              val relatedCandidates =\n                candidates.filter(_.scores.exists(_.scores.exists(_.rankerId.contains(rankerId))))\n              relatedCandidates\n                .sortBy(-_.scores\n                  .flatMap(_.scores.find(_.rankerId.contains(rankerId)).map(_.value)).getOrElse(\n                    0.0)).zipWithIndex.map {\n                  case (candidate, rank) => (candidate.id, rankerId, rank)\n                }\n            }.groupBy(_._1).map {\n              case (candidate, ranksForAllRankers) =>\n                (\n                  candidate,\n                  ranksForAllRankers.map { case (_, rankerId, rank) => (rankerId, rank) }.toMap)\n            }\n\n          val idFatigueCountMap =\n            wtfImpressions.groupBy(_.candidateId).mapValues(_.map(_.counts).sum)\n          trackTimeSinceOldestImpression(wtfImpressions)\n          val rankedCandidates: Seq[CandidateUser] = candidates\n            .map { candidate =>\n              val candidateImpressions = idFatigueCountMap.getOrElse(candidate.id, 0)\n              val fatiguedScores = candidate.scores.map { ss =>\n                ss.copy(scores = ss.scores.map { s =>\n                  s.rankerId match {\n                    // We set the new score as -rank after fatigue penalty is applied.\n                    case Some(rankerId) =>\n                      // If the candidate's ID is not in the candidate->ranks map, or there is no\n                      // rank for this specific ranker and this candidate, we use maximum possible\n                      // rank instead. Note that this indicates that there is a problem.\n                      s.copy(value = -(candidateRanks\n                        .getOrElse(candidate.id, Map()).getOrElse(rankerId, candidates.length) +\n                        candidateImpressions * fatigueFactor))\n                    // In case a score exists without a RankerId, we pass on the score as is.\n                    case None => s\n                  }\n                })\n              }\n              candidate.copy(scores = fatiguedScores)\n            }.zipWithIndex.map {\n              // We re-rank candidates with their input ordering (which is done by the request-level\n              // ranker) and fatigue penalty.\n              case (candidate, inputRank) =>\n                val candidateImpressions = idFatigueCountMap.getOrElse(candidate.id, 0)\n                (candidate, inputRank + candidateImpressions * fatigueFactor)\n            }.sortBy(_._2).map(_._1)\n          // Only populate ranking info when WTF impression info present\n          val scribeRankingInfo: Boolean =\n            target.params(ImpressionBasedFatigueRankerParams.ScribeRankingInfoInFatigueRanker)\n          if (scribeRankingInfo) Utils.addRankingInfo(rankedCandidates, name) else rankedCandidates\n        }\n      }.getOrElse(candidates) // no reranking/filtering when wtf impressions not present\n  }\n}\n\nobject ImpressionBasedFatigueRanker {\n  val DefaultFatigueFactor = 5\n\n  def build[\n    Target <: HasClientContext with HasDisplayLocation with HasParams with HasWtfImpressions\n  ](\n    baseStatsReceiver: StatsReceiver,\n    fatigueFactor: Int = DefaultFatigueFactor\n  ): ImpressionBasedFatigueRanker[Target] =\n    new ImpressionBasedFatigueRanker(fatigueFactor, baseStatsReceiver)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/fatigue_ranker/ImpressionBasedFatigueRankerFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.fatigue_ranker\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ImpressionBasedFatigueRankerFSConfig @Inject() extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[FSParam[Boolean]] =\n    Seq(ImpressionBasedFatigueRankerParams.ScribeRankingInfoInFatigueRanker)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/fatigue_ranker/ImpressionBasedFatigueRankerParams.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.fatigue_ranker\n\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\n\nobject ImpressionBasedFatigueRankerParams {\n  // Whether to enable hard dropping of impressed candidates\n  object DropImpressedCandidateEnabled extends Param[Boolean](false)\n  // At what # of impressions to hard drop candidates.\n  object DropCandidateImpressionThreshold extends Param[Int](default = 10)\n  // Whether to scribe candidate ranking/scoring info per ranking stage\n  object ScribeRankingInfoInFatigueRanker\n      extends FSParam[Boolean](\"fatigue_ranker_scribe_ranking_info\", true)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/first_n_ranker/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/utils\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/first_n_ranker/FirstNRanker.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.first_n_ranker\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.Ranker\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasQualityFactor\nimport com.twitter.follow_recommendations.common.rankers.utils.Utils\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\n/**\n * This class is meant to filter candidates between stages of our ranker by taking the first N\n * candidates, merging any candidate source information for candidates with multiple entries.\n * To allow us to chain this truncation operation any number of times sequentially within the main\n * ranking builder, we abstract the truncation as a separate Ranker\n */\n@Singleton\nclass FirstNRanker[Target <: HasClientContext with HasParams with HasQualityFactor] @Inject() (\n  stats: StatsReceiver)\n    extends Ranker[Target, CandidateUser] {\n\n  val name: String = this.getClass.getSimpleName\n  private val baseStats = stats.scope(\"first_n_ranker\")\n  val scaledDownByQualityFactorCounter =\n    baseStats.counter(\"scaled_down_by_quality_factor\")\n  private val mergeStat = baseStats.scope(\"merged_candidates\")\n  private val mergeStat2 = mergeStat.counter(\"2\")\n  private val mergeStat3 = mergeStat.counter(\"3\")\n  private val mergeStat4 = mergeStat.counter(\"4+\")\n  private val candidateSizeStats = baseStats.scope(\"candidate_size\")\n\n  private case class CandidateSourceScore(\n    candidateId: Long,\n    sourceId: CandidateSourceIdentifier,\n    score: Option[Double])\n\n  /**\n   * Adds the rank of each candidate based on the primary candidate source's score.\n   * In the event where the provided ordering of candidates do not align with the score,\n   * we will respect the score, since the ordering might have been mixed up due to other previous\n   * steps like the shuffleFn in the `WeightedCandidateSourceRanker`.\n   * @param candidates  ordered list of candidates\n   * @return            same ordered list of candidates, but with the rank information appended\n   */\n  def addRank(candidates: Seq[CandidateUser]): Seq[CandidateUser] = {\n    val candidateSourceRanks = for {\n      (sourceIdOpt, sourceCandidates) <- candidates.groupBy(_.getPrimaryCandidateSource)\n      (candidate, rank) <- sourceCandidates.sortBy(-_.score.getOrElse(0.0)).zipWithIndex\n    } yield {\n      (candidate, sourceIdOpt) -> rank\n    }\n    candidates.map { c =>\n      c.getPrimaryCandidateSource\n        .map { sourceId =>\n          val sourceRank = candidateSourceRanks((c, c.getPrimaryCandidateSource))\n          c.addCandidateSourceRanksMap(Map(sourceId -> sourceRank))\n        }.getOrElse(c)\n    }\n  }\n\n  override def rank(target: Target, candidates: Seq[CandidateUser]): Stitch[Seq[CandidateUser]] = {\n\n    val scaleDownFactor = Math.max(\n      target.qualityFactor.getOrElse(1.0d),\n      target.params(FirstNRankerParams.MinNumCandidatesScoredScaleDownFactor)\n    )\n\n    if (scaleDownFactor < 1.0d)\n      scaledDownByQualityFactorCounter.incr()\n\n    val n = (target.params(FirstNRankerParams.CandidatesToRank) * scaleDownFactor).toInt\n    val scribeRankingInfo: Boolean =\n      target.params(FirstNRankerParams.ScribeRankingInfoInFirstNRanker)\n    candidateSizeStats.counter(s\"n$n\").incr()\n    val candidatesWithRank = addRank(candidates)\n    if (target.params(FirstNRankerParams.GroupDuplicateCandidates)) {\n      val groupedCandidates: Map[Long, Seq[CandidateUser]] = candidatesWithRank.groupBy(_.id)\n      val topN = candidates\n        .map { c =>\n          merge(groupedCandidates(c.id))\n        }.distinct.take(n)\n      Stitch.value(if (scribeRankingInfo) Utils.addRankingInfo(topN, name) else topN)\n    } else {\n      Stitch.value(\n        if (scribeRankingInfo) Utils.addRankingInfo(candidatesWithRank, name).take(n)\n        else candidatesWithRank.take(n))\n    } // for efficiency, if don't need to deduplicate\n  }\n\n  /**\n   * we use the primary candidate source of the first entry, and aggregate all of the other entries'\n   * candidate source scores into the first entry's candidateSourceScores\n   * @param candidates list of candidates with the same id\n   * @return           a single merged candidate\n   */\n  private[first_n_ranker] def merge(candidates: Seq[CandidateUser]): CandidateUser = {\n    if (candidates.size == 1) {\n      candidates.head\n    } else {\n      candidates.size match {\n        case 2 => mergeStat2.incr()\n        case 3 => mergeStat3.incr()\n        case i if i >= 4 => mergeStat4.incr()\n        case _ =>\n      }\n      val allSources = candidates.flatMap(_.getCandidateSources).toMap\n      val allRanks = candidates.flatMap(_.getCandidateRanks).toMap\n      candidates.head.addCandidateSourceScoresMap(allSources).addCandidateSourceRanksMap(allRanks)\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/first_n_ranker/FirstNRankerFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.first_n_ranker\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\n@Singleton\nclass FirstNRankerFSConfig @Inject() extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[FSParam[Boolean]] =\n    Seq(FirstNRankerParams.ScribeRankingInfoInFirstNRanker)\n\n  override val intFSParams: Seq[FSBoundedParam[Int]] = Seq(\n    FirstNRankerParams.CandidatesToRank\n  )\n\n  override val doubleFSParams: Seq[FSBoundedParam[Double]] = Seq(\n    FirstNRankerParams.MinNumCandidatesScoredScaleDownFactor\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/first_n_ranker/FirstNRankerFeatureSwitchKeys.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.first_n_ranker\n\nobject FirstNRankerFeatureSwitchKeys {\n  val CandidatePoolSize = \"first_n_ranker_candidate_pool_size\"\n  val ScribeRankingInfo = \"first_n_ranker_scribe_ranking_info\"\n  val MinNumCandidatesScoredScaleDownFactor =\n    \"first_n_ranker_min_scale_down_factor\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/first_n_ranker/FirstNRankerParams.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.first_n_ranker\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\n\nobject FirstNRankerParams {\n  case object CandidatesToRank\n      extends FSBoundedParam[Int](\n        FirstNRankerFeatureSwitchKeys.CandidatePoolSize,\n        default = 100,\n        min = 50,\n        max = 600)\n\n  case object GroupDuplicateCandidates extends Param[Boolean](true)\n  case object ScribeRankingInfoInFirstNRanker\n      extends FSParam[Boolean](FirstNRankerFeatureSwitchKeys.ScribeRankingInfo, true)\n\n  // the minimum of candidates to score in each request.\n  object MinNumCandidatesScoredScaleDownFactor\n      extends FSBoundedParam[Double](\n        name = FirstNRankerFeatureSwitchKeys.MinNumCandidatesScoredScaleDownFactor,\n        default = 0.3,\n        min = 0.1,\n        max = 1.0)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/interleave_ranker/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/ranking\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/utils\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/interleave_ranker/InterleaveRanker.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.interleave_ranker\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.Ranker\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId\nimport com.twitter.follow_recommendations.common.rankers.utils.Utils\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\n@Singleton\nclass InterleaveRanker[Target <: HasParams] @Inject() (\n  statsReceiver: StatsReceiver)\n    extends Ranker[Target, CandidateUser] {\n\n  val name: String = this.getClass.getSimpleName\n  private val stats = statsReceiver.scope(\"interleave_ranker\")\n  private val inputStats = stats.scope(\"input\")\n  private val interleavingStats = stats.scope(\"interleave\")\n\n  override def rank(\n    target: Target, \n    candidates: Seq[CandidateUser]\n  ): Stitch[Seq[CandidateUser]] = {\n    StatsUtil.profileStitch(\n      Stitch.value(rankCandidates(target, candidates)),\n      stats.scope(\"rank\")\n    )\n  }\n\n  private def rankCandidates(\n    target: Target,\n    candidates: Seq[CandidateUser]\n  ): Seq[CandidateUser] = {\n\n    /**\n     * By this stage, all valid candidates should have:\n     *   1. Their Scores field populated.\n     *   2. Their selectedRankerId set.\n     *   3. Have a score associated to their selectedRankerId.\n     * If there is any candidate that doesn't meet the conditions above, there is a problem in one\n     * of the previous rankers. Since no new scoring is done in this ranker, we simply remove them.\n     */\n    val validCandidates =\n      candidates.filter { c =>\n        c.scores.isDefined &&\n        c.scores.exists(_.selectedRankerId.isDefined) &&\n        getCandidateScoreByRankerId(c, c.scores.flatMap(_.selectedRankerId)).isDefined\n      }\n\n    // To monitor the percentage of valid candidates, as defined above, we track the following:\n    inputStats.counter(\"candidates_with_no_scores\").incr(candidates.count(_.scores.isEmpty))\n    inputStats\n      .counter(\"candidates_with_no_selected_ranker\").incr(candidates.count { c =>\n        c.scores.isEmpty || c.scores.exists(_.selectedRankerId.isEmpty)\n      })\n    inputStats\n      .counter(\"candidates_with_no_score_for_selected_ranker\").incr(candidates.count { c =>\n        c.scores.isEmpty ||\n        c.scores.exists(_.selectedRankerId.isEmpty) ||\n        getCandidateScoreByRankerId(c, c.scores.flatMap(_.selectedRankerId)).isEmpty\n      })\n    inputStats.counter(\"total_num_candidates\").incr(candidates.length)\n    inputStats.counter(\"total_valid_candidates\").incr(validCandidates.length)\n\n    // We only count rankerIds from those candidates who are valid to exclude those candidates with\n    // a valid selectedRankerId that don't have an associated score for it.\n    val rankerIds = validCandidates.flatMap(_.scores.flatMap(_.selectedRankerId)).sorted.distinct\n    rankerIds.foreach { rankerId =>\n      inputStats\n        .counter(s\"valid_scores_for_${rankerId.toString}\").incr(\n          candidates.count(getCandidateScoreByRankerId(_, Some(rankerId)).isDefined))\n      inputStats.counter(s\"total_candidates_for_${rankerId.toString}\").incr(candidates.length)\n    }\n    inputStats.counter(s\"num_ranker_ids=${rankerIds.length}\").incr()\n    val scribeRankingInfo: Boolean =\n      target.params(InterleaveRankerParams.ScribeRankingInfoInInterleaveRanker)\n    if (rankerIds.length <= 1)\n      // In the case of \"Number of RankerIds = 0\", we pass on the candidates even though there is\n      // a problem in a previous ranker that provided the scores.\n      if (scribeRankingInfo) Utils.addRankingInfo(candidates, name) else candidates\n    else      \n      if (scribeRankingInfo)\n        Utils.addRankingInfo(interleaveCandidates(validCandidates, rankerIds), name)\n      else interleaveCandidates(validCandidates, rankerIds)\n  }\n\n  @VisibleForTesting\n  private[interleave_ranker] def interleaveCandidates(\n    candidates: Seq[CandidateUser],\n    rankerIds: Seq[RankerId.RankerId]\n  ): Seq[CandidateUser] = {\n    val candidatesWithRank = rankerIds\n      .flatMap { ranker =>\n        candidates\n        // We first sort all candidates using this ranker.\n          .sortBy(-getCandidateScoreByRankerId(_, Some(ranker)).getOrElse(Double.MinValue))\n          .zipWithIndex.filter(\n            // but only hold those candidates whose selected ranker is this ranker.\n            // These ranks will be forced in the final ordering.\n            _._1.scores.flatMap(_.selectedRankerId).contains(ranker))\n      }\n\n    // Only candidates who have isInProducerScoringExperiment set to true will have their position enforced. We\n    // separate candidates into two groups: (1) Production and (2) Experiment.\n    val (expCandidates, prodCandidates) =\n      candidatesWithRank.partition(_._1.scores.exists(_.isInProducerScoringExperiment))\n\n    // We resolve (potential) conflicts between the enforced ranks of experimental models.\n    val expCandidatesFinalPos = resolveConflicts(expCandidates)\n\n    // Retrieve non-occupied positions and assign them to candidates who use production ranker.\n    val occupiedPos = expCandidatesFinalPos.map(_._2).toSet\n    val prodCandidatesFinalPos =\n      prodCandidates\n        .map(_._1).zip(\n          candidates.indices.filterNot(occupiedPos.contains).sorted.take(prodCandidates.length))\n\n    // Merge the two groups and sort them by their corresponding positions.\n    val finalCandidates = (prodCandidatesFinalPos ++ expCandidatesFinalPos).sortBy(_._2).map(_._1)\n\n    // We count the presence of each ranker in the top-3 final positions.\n    finalCandidates.zip(0 until 3).foreach {\n      case (c, r) =>\n        // We only do so for candidates that are in a producer-side experiment.\n        if (c.scores.exists(_.isInProducerScoringExperiment))\n          c.scores.flatMap(_.selectedRankerId).map(_.toString).foreach { rankerName =>\n            interleavingStats\n              .counter(s\"num_final_position_${r}_$rankerName\")\n              .incr()\n          }\n    }\n\n    finalCandidates\n  }\n\n  @VisibleForTesting\n  private[interleave_ranker] def resolveConflicts(\n    candidatesWithRank: Seq[(CandidateUser, Int)]\n  ): Seq[(CandidateUser, Int)] = {\n    // The following two metrics will allow us to calculate the rate of conflicts occurring.\n    // Example: If overall there are 10 producers in different bucketing experiments, and 3 of them\n    // are assigned to the same position. The rate would be 3/10, 30%.\n    val numCandidatesWithConflicts = interleavingStats.counter(\"candidates_with_conflict\")\n    val numCandidatesNoConflicts = interleavingStats.counter(\"candidates_without_conflict\")\n    val candidatesGroupedByRank = candidatesWithRank.groupBy(_._2).toSeq.sortBy(_._1).map {\n      case (rank, candidatesWithRank) => (rank, candidatesWithRank.map(_._1))\n    }\n\n    candidatesGroupedByRank.foldLeft(Seq[(CandidateUser, Int)]()) { (upToHere, nextGroup) =>\n      val (rank, candidates) = nextGroup\n      if (candidates.length > 1)\n        numCandidatesWithConflicts.incr(candidates.length)\n      else\n        numCandidatesNoConflicts.incr()\n\n      // We use the position after the last-assigned candidate as a starting point, or 0 otherwise.\n      // If candidates' position is after this \"starting point\", we enforce that position instead.\n      val minAvailableIndex = scala.math.max(upToHere.lastOption.map(_._2).getOrElse(-1) + 1, rank)\n      val enforcedPos =\n        (minAvailableIndex until minAvailableIndex + candidates.length).toList\n      val shuffledEnforcedPos =\n        if (candidates.length > 1) scala.util.Random.shuffle(enforcedPos) else enforcedPos\n      if (shuffledEnforcedPos.length > 1) {\n        candidates.zip(shuffledEnforcedPos).sortBy(_._2).map(_._1).zipWithIndex.foreach {\n          case (c, r) =>\n            c.scores.flatMap(_.selectedRankerId).map(_.toString).foreach { rankerName =>\n              // For each ranker, we count the total number of times it has been in a conflict.\n              interleavingStats\n                .counter(s\"num_${shuffledEnforcedPos.length}-way_conflicts_$rankerName\")\n                .incr()\n              // We also count the positions each of the rankers have fallen randomly into. In any\n              // experiment this should converge to uniform distribution given enough occurrences.\n              // Note that the position here is relative to the other candidates in the conflict and\n              // not the overall position of each candidate.\n              interleavingStats\n                .counter(\n                  s\"num_position_${r}_after_${shuffledEnforcedPos.length}-way_conflict_$rankerName\")\n                .incr()\n            }\n        }\n      }\n      upToHere ++ candidates.zip(shuffledEnforcedPos).sortBy(_._2)\n    }\n  }\n\n  @VisibleForTesting\n  private[interleave_ranker] def getCandidateScoreByRankerId(\n    candidate: CandidateUser,\n    rankerIdOpt: Option[RankerId.RankerId]\n  ): Option[Double] = {\n    rankerIdOpt match {\n      case None => None\n      case Some(rankerId) =>\n        candidate.scores.flatMap {\n          _.scores.find(_.rankerId.contains(rankerId)).map(_.value)\n        }\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/interleave_ranker/InterleaveRankerFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.interleave_ranker\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSParam\n\n@Singleton\nclass InterleaveRankerFSConfig @Inject() extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[FSParam[Boolean]] =\n    Seq(InterleaveRankerParams.ScribeRankingInfoInInterleaveRanker)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/interleave_ranker/InterleaveRankerParams.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.interleave_ranker\n\nimport com.twitter.timelines.configapi.FSParam\n\nobject InterleaveRankerParams {\n  case object ScribeRankingInfoInInterleaveRanker\n      extends FSParam[Boolean](\"interleave_ranker_scribe_ranking_info\", true)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/ranking/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/deepbirdv2\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/scoring\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/utils\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n\n# This is to import only the params from MlRanker, for instance to get request-level heavy ranker.\nscala_library(\n    name = \"ml_ranker_params\",\n    sources = [\n        \"MlRankerParams.scala\",\n    ],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common\",\n        \"timelines/src/main/scala/com/twitter/timelines/config/configapi\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/ranking/HydrateFeaturesTransform.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.ml_ranker.ranking\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.GatedTransform\nimport com.twitter.follow_recommendations.common.base.StatsUtil.profileStitchMapResults\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.UserScoringFeatureSource\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDebugOptions\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasSimilarToContext\nimport com.twitter.follow_recommendations.common.models.RichDataRecord\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.logging.Logging\n\n/**\n * Hydrate features given target and candidates lists.\n * This is a required step before MlRanker.\n * If a feature is not hydrated before MlRanker is triggered, a runtime exception will be thrown\n */\n@Singleton\nclass HydrateFeaturesTransform[\n  Target <: HasClientContext with HasParams with HasDebugOptions with HasPreFetchedFeature with HasSimilarToContext with HasDisplayLocation] @Inject() (\n  userScoringFeatureSource: UserScoringFeatureSource,\n  stats: StatsReceiver)\n    extends GatedTransform[Target, CandidateUser]\n    with Logging {\n\n  private val hydrateFeaturesStats = stats.scope(\"hydrate_features\")\n\n  def transform(target: Target, candidates: Seq[CandidateUser]): Stitch[Seq[CandidateUser]] = {\n    // get features\n    val featureMapStitch: Stitch[Map[CandidateUser, DataRecord]] =\n      profileStitchMapResults(\n        userScoringFeatureSource.hydrateFeatures(target, candidates),\n        hydrateFeaturesStats)\n\n    featureMapStitch.map { featureMap =>\n      candidates\n        .map { candidate =>\n          val dataRecord = featureMap(candidate)\n          // add debugRecord only when the request parameter is set\n          val debugDataRecord = if (target.debugOptions.exists(_.fetchDebugInfo)) {\n            Some(candidate.toDebugDataRecord(dataRecord, userScoringFeatureSource.featureContext))\n          } else None\n          candidate.copy(\n            dataRecord = Some(RichDataRecord(Some(dataRecord), debugDataRecord))\n          )\n        }\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/ranking/MlRanker.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.ml_ranker.ranking\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.Ranker\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.follow_recommendations.common.base.StatsUtil.profileSeqResults\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasDebugOptions\nimport com.twitter.follow_recommendations.common.models.Scores\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId.RankerId\nimport com.twitter.follow_recommendations.common.rankers.utils.Utils\nimport com.twitter.follow_recommendations.common.rankers.ml_ranker.scoring.AdhocScorer\nimport com.twitter.follow_recommendations.common.rankers.ml_ranker.scoring.Scorer\nimport com.twitter.follow_recommendations.common.rankers.ml_ranker.scoring.ScorerFactory\nimport com.twitter.follow_recommendations.common.utils.CollectionUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.logging.Logging\n\n/**\n * This class has a rank function that will perform 4 steps:\n *   - choose which scorer to use for each candidate\n *   - score candidates given their respective features\n *   - add scoring information to the candidate\n *   - sort candidates by their respective scores\n *   The feature source and scorer will depend on the request's params\n */\n@Singleton\nclass MlRanker[\n  Target <: HasClientContext with HasParams with HasDisplayLocation with HasDebugOptions] @Inject() (\n  scorerFactory: ScorerFactory,\n  statsReceiver: StatsReceiver)\n    extends Ranker[Target, CandidateUser]\n    with Logging {\n\n  private val stats: StatsReceiver = statsReceiver.scope(\"ml_ranker\")\n\n  private val inputStat = stats.scope(\"1_input\")\n  private val selectScorerStat = stats.scope(\"2_select_scorer\")\n  private val scoreStat = stats.scope(\"3_score\")\n\n  override def rank(\n    target: Target,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Seq[CandidateUser]] = {\n    profileSeqResults(candidates, inputStat)\n    val requestRankerId = target.params(MlRankerParams.RequestScorerIdParam)\n    val rankerIds = chooseRankerByCandidate(candidates, requestRankerId)\n\n    val scoreStitch = score(candidates, rankerIds, requestRankerId).map { scoredCandidates =>\n      {\n        // sort the candidates by score\n        val sortedCandidates = sort(target, scoredCandidates)\n        // add scribe field to candidates (if applicable) and return candidates\n        scribeCandidates(target, sortedCandidates)\n      }\n    }\n    StatsUtil.profileStitch(scoreStitch, stats.scope(\"rank\"))\n  }\n\n  /**\n   * @param target: The WTF request for a given consumer.\n   * @param candidates A list of candidates considered for recommendation.\n   * @return A map from each candidate to a tuple that includes:\n   *          (1) The selected scorer that should be used to rank this candidate\n   *          (2) a flag determining whether the candidate is in a producer-side experiment.\n   */\n  private[ranking] def chooseRankerByCandidate(\n    candidates: Seq[CandidateUser],\n    requestRankerId: RankerId\n  ): Map[CandidateUser, RankerId] = {\n    candidates.map { candidate =>\n      val selectedCandidateRankerId =\n        if (candidate.params == Params.Invalid || candidate.params == Params.Empty) {\n          selectScorerStat.counter(\"candidate_params_empty\").incr()\n          requestRankerId\n        } else {\n          val candidateRankerId = candidate.params(MlRankerParams.CandidateScorerIdParam)\n          if (candidateRankerId == RankerId.None) {\n            // This candidate is a not part of any producer-side experiment.\n            selectScorerStat.counter(\"default_to_request_ranker\").incr()\n            requestRankerId\n          } else {\n            // This candidate is in a treatment bucket of a producer-side experiment.\n            selectScorerStat.counter(\"use_candidate_ranker\").incr()\n            candidateRankerId\n          }\n        }\n      selectScorerStat.scope(\"selected\").counter(selectedCandidateRankerId.toString).incr()\n      candidate -> selectedCandidateRankerId\n    }.toMap\n  }\n\n  @VisibleForTesting\n  private[ranking] def score(\n    candidates: Seq[CandidateUser],\n    rankerIds: Map[CandidateUser, RankerId],\n    requestRankerId: RankerId\n  ): Stitch[Seq[CandidateUser]] = {\n    val features = candidates.map(_.dataRecord.flatMap(_.dataRecord))\n\n    require(features.forall(_.nonEmpty), \"features are not hydrated for all the candidates\")\n\n    val scorers = scorerFactory.getScorers(rankerIds.values.toSeq.sorted.distinct)\n\n    // Scorers are split into ML-based and Adhoc (defined as a scorer that does not need to call an\n    // ML prediction service and scores candidates using locally-available data).\n    val (adhocScorers, mlScorers) = scorers.partition {\n      case _: AdhocScorer => true\n      case _ => false\n    }\n\n    // score candidates\n    val scoresStitch = score(features.map(_.get), mlScorers)\n    val candidatesWithMlScoresStitch = scoresStitch.map { scoresSeq =>\n      candidates\n        .zip(scoresSeq).map { // copy datarecord and score into candidate object\n          case (candidate, scores) =>\n            val selectedRankerId = rankerIds(candidate)\n            val useRequestRanker =\n              candidate.params == Params.Invalid ||\n                candidate.params == Params.Empty ||\n                candidate.params(MlRankerParams.CandidateScorerIdParam) == RankerId.None\n            candidate.copy(\n              score = scores.scores.find(_.rankerId.contains(requestRankerId)).map(_.value),\n              scores = if (scores.scores.nonEmpty) {\n                Some(\n                  scores.copy(\n                    scores = scores.scores,\n                    selectedRankerId = Some(selectedRankerId),\n                    isInProducerScoringExperiment = !useRequestRanker\n                  ))\n              } else None\n            )\n        }\n    }\n\n    candidatesWithMlScoresStitch.map { candidates =>\n      // The basis for adhoc scores are the \"request-level\" ML ranker. We add the base score here\n      // while adhoc scorers are applied in [[AdhocRanker]].\n      addMlBaseScoresForAdhocScorers(candidates, requestRankerId, adhocScorers)\n    }\n  }\n\n  @VisibleForTesting\n  private[ranking] def addMlBaseScoresForAdhocScorers(\n    candidates: Seq[CandidateUser],\n    requestRankerId: RankerId,\n    adhocScorers: Seq[Scorer]\n  ): Seq[CandidateUser] = {\n    candidates.map { candidate =>\n      candidate.scores match {\n        case Some(oldScores) =>\n          // 1. We fetch the ML score that is the basis of adhoc scores:\n          val baseMlScoreOpt = Utils.getCandidateScoreByRankerId(candidate, requestRankerId)\n\n          // 2. For each adhoc scorer, we copy the ML score object, changing only the ID and type.\n          val newScores = adhocScorers flatMap { adhocScorer =>\n            baseMlScoreOpt.map(\n              _.copy(rankerId = Some(adhocScorer.id), scoreType = adhocScorer.scoreType))\n          }\n\n          // 3. We add the new adhoc score entries to the candidate.\n          candidate.copy(scores = Some(oldScores.copy(scores = oldScores.scores ++ newScores)))\n        case _ =>\n          // Since there is no base ML score, there should be no adhoc score modification as well.\n          candidate\n      }\n    }\n  }\n\n  private[this] def score(\n    dataRecords: Seq[DataRecord],\n    scorers: Seq[Scorer]\n  ): Stitch[Seq[Scores]] = {\n    val scoredResponse = scorers.map { scorer =>\n      StatsUtil.profileStitch(scorer.score(dataRecords), scoreStat.scope(scorer.id.toString))\n    }\n    // If we could score a candidate with too many rankers, it is likely to blow up the whole system.\n    // and fail back to default production model\n    StatsUtil.profileStitch(Stitch.collect(scoredResponse), scoreStat).map { scoresByScorerId =>\n      CollectionUtil.transposeLazy(scoresByScorerId).map { scoresPerCandidate =>\n        Scores(scoresPerCandidate)\n      }\n    }\n  }\n\n  // sort candidates using score in descending order\n  private[this] def sort(\n    target: Target,\n    candidates: Seq[CandidateUser]\n  ): Seq[CandidateUser] = {\n    candidates.sortBy(c => -c.score.getOrElse(MlRanker.DefaultScore))\n  }\n\n  private[this] def scribeCandidates(\n    target: Target,\n    candidates: Seq[CandidateUser]\n  ): Seq[CandidateUser] = {\n    val scribeRankingInfo: Boolean = target.params(MlRankerParams.ScribeRankingInfoInMlRanker)\n    scribeRankingInfo match {\n      case true => Utils.addRankingInfo(candidates, \"MlRanker\")\n      case false => candidates\n    }\n  }\n}\n\nobject MlRanker {\n  // this is to ensure candidates with absent scores are ranked the last\n  val DefaultScore: Double = Double.MinValue\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/ranking/MlRankerFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.ml_ranker.ranking\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSParam\n\n@Singleton\nclass MlRankerFSConfig @Inject() extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[FSParam[Boolean]] =\n    Seq(MlRankerParams.ScribeRankingInfoInMlRanker)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/ranking/MlRankerParams.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.ml_ranker.ranking\n\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSParam\n\n/**\n * When adding Producer side experiments, make sure to register the FS Key in [[ProducerFeatureFilter]]\n * in [[FeatureSwitchesModule]], otherwise, the FS will not work.\n */\nobject MlRankerParams {\n  // which ranker to use by default for the given request\n  case object RequestScorerIdParam\n      extends FSEnumParam[RankerId.type](\n        name = \"post_nux_ml_flow_ml_ranker_id\",\n        default = RankerId.PostNuxProdRanker,\n        enum = RankerId\n      )\n\n  // which ranker to use for the given candidate\n  case object CandidateScorerIdParam\n      extends FSEnumParam[RankerId.type](\n        name = \"post_nux_ml_flow_candidate_user_scorer_id\",\n        default = RankerId.None,\n        enum = RankerId\n      )\n\n  case object ScribeRankingInfoInMlRanker\n      extends FSParam[Boolean](\"post_nux_ml_flow_scribe_ranking_info_in_ml_ranker\", true)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/scoring/AdhocScorer.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.ml_ranker.scoring\n\nimport com.twitter.follow_recommendations.common.rankers.common.AdhocScoreModificationType.AdhocScoreModificationType\nimport com.twitter.follow_recommendations.common.models.Score\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.stitch.Stitch\n\ntrait AdhocScorer extends Scorer {\n\n  /**\n   * NOTE: For instances of [[AdhocScorer]] this function SHOULD NOT be used.\n   * Please use:\n   *   [[score(target: HasClientContext with HasParams, candidates: Seq[CandidateUser])]]\n   * instead.\n   */\n  @Deprecated\n  override def score(records: Seq[DataRecord]): Stitch[Seq[Score]] =\n    throw new UnsupportedOperationException(\n      \"For instances of AdhocScorer this operation is not defined. Please use \" +\n        \"`def score(target: HasClientContext with HasParams, candidates: Seq[CandidateUser])` \" +\n        \"instead.\")\n\n  /**\n   * This helps us manage the extend of adhoc modification on candidates' score. There is a hard\n   * limit of applying ONLY ONE scorer of each type to a score.\n   */\n  val scoreModificationType: AdhocScoreModificationType\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/scoring/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/deepbirdv2\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/ranking:ml_ranker_params\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/pluck/source/core_workflows/user_model:condensed_user_state-scala\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/scoring/DeepbirdScorer.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.ml_ranker.scoring\n\nimport com.twitter.cortex.deepbird.thriftjava.DeepbirdPredictionService\nimport com.twitter.cortex.deepbird.thriftjava.ModelSelector\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDebugOptions\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.Score\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.prediction_service.{BatchPredictionRequest => JBatchPredictionRequest}\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.Future\nimport com.twitter.util.TimeoutException\nimport scala.collection.JavaConversions._\nimport scala.collection.JavaConverters._\n\n/**\n * Generic trait that implements the scoring given a deepbirdClient\n * To test out a new model, create a scorer extending this trait, override the modelName and inject the scorer\n */\ntrait DeepbirdScorer extends Scorer {\n  def modelName: String\n  def predictionFeature: Feature.Continuous\n  // Set a default batchSize of 100 when making model prediction calls to the Deepbird V2 prediction server\n  def batchSize: Int = 100\n  def deepbirdClient: DeepbirdPredictionService.ServiceToClient\n  def baseStats: StatsReceiver\n\n  def modelSelector: ModelSelector = new ModelSelector().setId(modelName)\n  def stats: StatsReceiver = baseStats.scope(this.getClass.getSimpleName).scope(modelName)\n\n  private def requestCount = stats.counter(\"requests\")\n  private def emptyRequestCount = stats.counter(\"empty_requests\")\n  private def successCount = stats.counter(\"success\")\n  private def failureCount = stats.counter(\"failures\")\n  private def inputRecordsStat = stats.stat(\"input_records\")\n  private def outputRecordsStat = stats.stat(\"output_records\")\n\n  // Counters for tracking batch-prediction statistics when making DBv2 prediction calls\n  //\n  // numBatchRequests tracks the number of batch prediction requests made to DBv2 prediction servers\n  private def numBatchRequests = stats.counter(\"batches\")\n  // numEmptyBatchRequests tracks the number of batch prediction requests made to DBv2 prediction servers\n  // that had an empty input DataRecord\n  private def numEmptyBatchRequests = stats.counter(\"empty_batches\")\n  // numTimedOutBatchRequests tracks the number of batch prediction requests made to DBv2 prediction servers\n  // that had timed-out\n  private def numTimedOutBatchRequests = stats.counter(\"timeout_batches\")\n\n  private def batchPredictionLatency = stats.stat(\"batch_prediction_latency\")\n  private def predictionLatency = stats.stat(\"prediction_latency\")\n\n  private def numEmptyModelPredictions = stats.counter(\"empty_model_predictions\")\n  private def numNonEmptyModelPredictions = stats.counter(\"non_empty_model_predictions\")\n\n  private val DefaultPredictionScore = 0.0\n\n  /**\n   * NOTE: For instances of [[DeepbirdScorer]] this function SHOULD NOT be used.\n   * Please use [[score(records: Seq[DataRecord])]] instead.\n   */\n  @Deprecated\n  def score(\n    target: HasClientContext with HasParams with HasDisplayLocation with HasDebugOptions,\n    candidates: Seq[CandidateUser]\n  ): Seq[Option[Score]] =\n    throw new UnsupportedOperationException(\n      \"For instances of DeepbirdScorer this operation is not defined. Please use \" +\n        \"`def score(records: Seq[DataRecord]): Stitch[Seq[Score]]` \" +\n        \"instead.\")\n\n  override def score(records: Seq[DataRecord]): Stitch[Seq[Score]] = {\n    requestCount.incr()\n    if (records.isEmpty) {\n      emptyRequestCount.incr()\n      Stitch.Nil\n    } else {\n      inputRecordsStat.add(records.size)\n      Stitch.callFuture(\n        batchPredict(records, batchSize)\n          .map { recordList =>\n            val scores = recordList.map { record =>\n              Score(\n                value = record.getOrElse(DefaultPredictionScore),\n                rankerId = Some(id),\n                scoreType = scoreType)\n            }\n            outputRecordsStat.add(scores.size)\n            scores\n          }.onSuccess(_ => successCount.incr())\n          .onFailure(_ => failureCount.incr()))\n    }\n  }\n\n  def batchPredict(\n    dataRecords: Seq[DataRecord],\n    batchSize: Int\n  ): Future[Seq[Option[Double]]] = {\n    Stat\n      .timeFuture(predictionLatency) {\n        val batchedDataRecords = dataRecords.grouped(batchSize).toSeq\n        numBatchRequests.incr(batchedDataRecords.size)\n        Future\n          .collect(batchedDataRecords.map(batch => predict(batch)))\n          .map(res => res.reduce(_ ++ _))\n      }\n  }\n\n  def predict(dataRecords: Seq[DataRecord]): Future[Seq[Option[Double]]] = {\n    Stat\n      .timeFuture(batchPredictionLatency) {\n        if (dataRecords.isEmpty) {\n          numEmptyBatchRequests.incr()\n          Future.Nil\n        } else {\n          deepbirdClient\n            .batchPredictFromModel(new JBatchPredictionRequest(dataRecords.asJava), modelSelector)\n            .map { response =>\n              response.predictions.toSeq.map { prediction =>\n                val predictionFeatureOption = Option(\n                  new RichDataRecord(prediction).getFeatureValue(predictionFeature)\n                )\n                predictionFeatureOption match {\n                  case Some(predictionValue) =>\n                    numNonEmptyModelPredictions.incr()\n                    Option(predictionValue.toDouble)\n                  case None =>\n                    numEmptyModelPredictions.incr()\n                    Option(DefaultPredictionScore)\n                }\n              }\n            }\n            .rescue {\n              case e: TimeoutException => // DBv2 prediction calls that timed out\n                numTimedOutBatchRequests.incr()\n                stats.counter(e.getClass.getSimpleName).incr()\n                Future.value(dataRecords.map(_ => Option(DefaultPredictionScore)))\n              case e: Exception => // other generic DBv2 prediction call failures\n                stats.counter(e.getClass.getSimpleName).incr()\n                Future.value(dataRecords.map(_ => Option(DefaultPredictionScore)))\n            }\n        }\n      }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/scoring/PostnuxDeepbirdProdScorer.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.ml_ranker.scoring\n\nimport com.twitter.cortex.deepbird.thriftjava.DeepbirdPredictionService\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId\nimport com.twitter.ml.api.Feature\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n// This is a standard DeepbirdV2 ML Ranker scoring config that should be extended by all ML scorers\n//\n// Only modify this trait when adding new fields to DeepbirdV2 scorers which\ntrait DeepbirdProdScorer extends DeepbirdScorer {\n  override val batchSize = 20\n}\n\n// Feature.Continuous(\"prediction\") is specific to ClemNet architecture, we can change it to be more informative in the next iteration\ntrait PostNuxV1DeepbirdProdScorer extends DeepbirdProdScorer {\n  override val predictionFeature: Feature.Continuous =\n    new Feature.Continuous(\"prediction\")\n}\n\n// The current, primary PostNUX DeepbirdV2 scorer used in production\n@Singleton\nclass PostnuxDeepbirdProdScorer @Inject() (\n  @Named(GuiceNamedConstants.WTF_PROD_DEEPBIRDV2_CLIENT)\n  override val deepbirdClient: DeepbirdPredictionService.ServiceToClient,\n  override val baseStats: StatsReceiver)\n    extends PostNuxV1DeepbirdProdScorer {\n  override val id = RankerId.PostNuxProdRanker\n  override val modelName = \"PostNUX14531GafClemNetWarmStart\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/scoring/RandomScorer.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.ml_ranker.scoring\n\nimport com.twitter.cortex.deepbird.thriftjava.DeepbirdPredictionService\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.Feature\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/**\n * This scorer assigns random values between 0 and 1 to each candidate as scores.\n */\n\n@Singleton\nclass RandomScorer @Inject() (\n  @Named(GuiceNamedConstants.WTF_PROD_DEEPBIRDV2_CLIENT)\n  override val deepbirdClient: DeepbirdPredictionService.ServiceToClient,\n  override val baseStats: StatsReceiver)\n    extends DeepbirdScorer {\n  override val id = RankerId.RandomRanker\n  private val rnd = new scala.util.Random(System.currentTimeMillis())\n\n  override def predict(dataRecords: Seq[DataRecord]): Future[Seq[Option[Double]]] = {\n    if (dataRecords.isEmpty) {\n      Future.Nil\n    } else {\n      // All candidates are assigned a random value between 0 and 1 as score.\n      Future.value(dataRecords.map(_ => Option(rnd.nextDouble())))\n    }\n  }\n\n  override val modelName = \"PostNuxRandomRanker\"\n\n  // This is not needed since we are overriding the `predict` function, but we have to override\n  // `predictionFeature` anyway.\n  override val predictionFeature: Feature.Continuous =\n    new Feature.Continuous(\"prediction.pfollow_pengagement\")\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/scoring/Scorer.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.ml_ranker.scoring\n\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasDebugOptions\nimport com.twitter.follow_recommendations.common.models.Score\nimport com.twitter.follow_recommendations.common.models.ScoreType\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\ntrait Scorer {\n\n  // unique id of the scorer\n  def id: RankerId.Value\n\n  // type of the output scores\n  def scoreType: Option[ScoreType] = None\n\n  // Scoring when an ML model is used.\n  def score(records: Seq[DataRecord]): Stitch[Seq[Score]]\n\n  /**\n   * Scoring when a non-ML method is applied. E.g: Boosting, randomized reordering, etc.\n   * This method assumes that candidates' scores are already retrieved from heavy-ranker models and\n   * are available for use.\n   */\n  def score(\n    target: HasClientContext with HasParams with HasDisplayLocation with HasDebugOptions,\n    candidates: Seq[CandidateUser]\n  ): Seq[Option[Score]]\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/scoring/ScorerFactory.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.ml_ranker.scoring\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId.RankerId\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScorerFactory @Inject() (\n  postnuxProdScorer: PostnuxDeepbirdProdScorer,\n  randomScorer: RandomScorer,\n  stats: StatsReceiver) {\n\n  private val scorerFactoryStats = stats.scope(\"scorer_factory\")\n  private val scorerStat = scorerFactoryStats.scope(\"scorer\")\n\n  def getScorers(\n    rankerIds: Seq[RankerId]\n  ): Seq[Scorer] = {\n    rankerIds.map { scorerId =>\n      val scorer: Scorer = getScorerById(scorerId)\n      // count # of times a ranker has been requested\n      scorerStat.counter(scorer.id.toString).incr()\n      scorer\n    }\n  }\n\n  def getScorerById(scorerId: RankerId): Scorer = scorerId match {\n    case RankerId.PostNuxProdRanker =>\n      postnuxProdScorer\n    case RankerId.RandomRanker =>\n      randomScorer\n    case _ =>\n      scorerStat.counter(\"invalid_scorer_type\").incr()\n      throw new IllegalArgumentException(\"unknown_scorer_type\")\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/utils/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/utils/Utils.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.utils\n\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.Score\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId.RankerId\n\nobject Utils {\n\n  /**\n   * Add the ranking and scoring info for a list of candidates on a given ranking stage.\n   * @param candidates A list of CandidateUser\n   * @param rankingStage Should use `Ranker.name` as the ranking stage.\n   * @return The list of CandidateUser with ranking/scoring info added.\n   */\n  def addRankingInfo(candidates: Seq[CandidateUser], rankingStage: String): Seq[CandidateUser] = {\n    candidates.zipWithIndex.map {\n      case (candidate, rank) =>\n        // 1-based ranking for better readability\n        candidate.addInfoPerRankingStage(rankingStage, candidate.scores, rank + 1)\n    }\n  }\n\n  def getCandidateScoreByRankerId(candidate: CandidateUser, rankerId: RankerId): Option[Score] =\n    candidate.scores.flatMap { ss => ss.scores.find(_.rankerId.contains(rankerId)) }\n\n  def getAllRankerIds(candidates: Seq[CandidateUser]): Seq[RankerId] =\n    candidates.flatMap(_.scores.map(_.scores.flatMap(_.rankerId))).flatten.distinct\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/weighted_candidate_source_ranker/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/utils\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/weighted_candidate_source_ranker/CandidateShuffle.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker\n\nimport com.twitter.follow_recommendations.common.utils.RandomUtil\nimport scala.util.Random\n\nsealed trait CandidateShuffler[T] {\n  def shuffle(seed: Option[Long])(input: Seq[T]): Seq[T]\n}\n\nclass NoShuffle[T]() extends CandidateShuffler[T] {\n  def shuffle(seed: Option[Long])(input: Seq[T]): Seq[T] = input\n}\n\nclass RandomShuffler[T]() extends CandidateShuffler[T] {\n  def shuffle(seed: Option[Long])(input: Seq[T]): Seq[T] = {\n    seed.map(new Random(_)).getOrElse(Random).shuffle(input)\n  }\n}\n\ntrait RankWeightedRandomShuffler[T] extends CandidateShuffler[T] {\n\n  def rankToWeight(rank: Int): Double\n  def shuffle(seed: Option[Long])(input: Seq[T]): Seq[T] = {\n    val candWeights = input.zipWithIndex.map {\n      case (candidate, rank) => (candidate, rankToWeight(rank))\n    }\n    RandomUtil.weightedRandomShuffle(candWeights, seed.map(new Random(_))).unzip._1\n  }\n}\n\nclass ExponentialShuffler[T]() extends RankWeightedRandomShuffler[T] {\n  def rankToWeight(rank: Int): Double = {\n    1 / math\n      .pow(rank.toDouble, 2.0) // this function was proved to be effective in previous DDGs\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/weighted_candidate_source_ranker/WeightMethod.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker\n\nobject WeightMethod extends Enumeration {\n  type WeightMethod = Value\n  val WeightedRandomSampling, WeightedRoundRobin = Value\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/weighted_candidate_source_ranker/WeightedCandidateSourceBaseRanker.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker\n\nimport com.twitter.follow_recommendations.common.utils.RandomUtil\nimport com.twitter.follow_recommendations.common.utils.MergeUtil\nimport com.twitter.follow_recommendations.common.utils.Weighted\nimport com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker.WeightMethod._\nimport scala.util.Random\n\n/**\n * This ranker selects the next candidate source to select a candidate from. It supports\n * two kinds of algorithm, WeightedRandomSampling or WeightedRoundRobin. WeightedRandomSampling\n * pick the next candidate source randomly, WeightedRoundRobin picked the next candidate source\n * sequentially based on the weight of the candidate source. It is default to WeightedRandomSampling\n * if no weight method is provided.\n *\n * Example usage of this class:\n *\n * When use WeightedRandomSampling:\n * Input candidate sources and their weights are: {{CS1: 3}, {CS2: 2}, {CS3: 5}}\n * Ranked candidates sequence is not determined because of random sampling.\n * One possible output candidate sequence is: (CS1_candidate1, CS2_candidate1, CS2_candidate2,\n * CS3_candidate1, CS3_candidates2, CS3_candidate3, CS1_candidate2, CS1_candidate3,\n * CS3_candidate4, CS3_candidate5, CS1_candidate4, CS1_candidate5, CS2_candidate6, CS2_candidate3,...)\n *\n * When use WeightedRoundRobin:\n * Input candidate sources and their weights are: {{CS1: 3}, {CS2: 2}, {CS3: 5}}\n * Output candidate sequence is: (CS1_candidate1, CS1_candidate2, CS1_candidate3,\n * CS2_candidate1, CS2_candidates2, CS3_candidate1, CS3_candidate2, CS3_candidate3,\n * CS3_candidate4, CS3_candidate5, CS1_candidate4, CS1_candidate5, CS1_candidate6, CS2_candidate3,...)\n *\n * Note: CS1_candidate1 means the first candidate in CS1 candidate source.\n *\n * @tparam A candidate source type\n * @tparam Rec Recommendation type\n * @param candidateSourceWeights relative weights for different candidate sources\n */\nclass WeightedCandidateSourceBaseRanker[A, Rec](\n  candidateSourceWeights: Map[A, Double],\n  weightMethod: WeightMethod = WeightedRandomSampling,\n  randomSeed: Option[Long]) {\n\n  /**\n   * Creates a iterator over algorithms and calls next to return a Stream of candidates\n   *\n   *\n   * @param candidateSources the set of candidate sources that are being sampled\n   * @param candidateSourceWeights map of candidate source to weight\n   * @param candidates the map of candidate source to the iterator of its results\n   * @param weightMethod a enum to indict which weight method to use. Two values are supported\n   * currently. When WeightedRandomSampling is set, the next candidate is picked from a candidate\n   * source that is randomly chosen. When WeightedRoundRobin is set, the next candidate is picked\n   * from current candidate source until the number of candidates reaches to the assigned weight of\n   * the candidate source. The next call of this function will return a candidate from the next\n   * candidate source which is after previous candidate source based on the order input\n   * candidate source sequence.\n\n   * @return stream of candidates\n   */\n  def stream(\n    candidateSources: Set[A],\n    candidateSourceWeights: Map[A, Double],\n    candidates: Map[A, Iterator[Rec]],\n    weightMethod: WeightMethod = WeightedRandomSampling,\n    random: Option[Random] = None\n  ): Stream[Rec] = {\n    val weightedCandidateSource: Weighted[A] = new Weighted[A] {\n      override def apply(a: A): Double = candidateSourceWeights.getOrElse(a, 0)\n    }\n\n    /**\n     * Generates a stream of candidates.\n     *\n     * @param candidateSourceIter an iterator over candidate sources returned by the sampling procedure\n     * @return stream of candidates\n     */\n    def next(candidateSourceIter: Iterator[A]): Stream[Rec] = {\n      val source = candidateSourceIter.next()\n      val it = candidates(source)\n      if (it.hasNext) {\n        val currCand = it.next()\n        currCand #:: next(candidateSourceIter)\n      } else {\n        assert(candidateSources.contains(source), \"Selected source is not in candidate sources\")\n        // Remove the depleted candidate source and re-sample\n        stream(candidateSources - source, candidateSourceWeights, candidates, weightMethod, random)\n      }\n    }\n    if (candidateSources.isEmpty)\n      Stream.empty\n    else {\n      val candidateSourceSeq = candidateSources.toSeq\n      val candidateSourceIter =\n        if (weightMethod == WeightMethod.WeightedRoundRobin) {\n          MergeUtil.weightedRoundRobin(candidateSourceSeq)(weightedCandidateSource).iterator\n        } else {\n          //default to weighted random sampling if no other weight method is provided\n          RandomUtil\n            .weightedRandomSamplingWithReplacement(\n              candidateSourceSeq,\n              random\n            )(weightedCandidateSource).iterator\n        }\n      next(candidateSourceIter)\n    }\n  }\n\n  def apply(input: Map[A, TraversableOnce[Rec]]): Stream[Rec] = {\n    stream(\n      input.keySet,\n      candidateSourceWeights,\n      input.map {\n        case (k, v) => k -> v.toIterator\n      }, // cannot do mapValues here, as that only returns a view\n      weightMethod,\n      randomSeed.map(new Random(_))\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/weighted_candidate_source_ranker/WeightedCandidateSourceRanker.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker\nimport com.twitter.follow_recommendations.common.base.Ranker\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.rankers.common.DedupCandidates\nimport com.twitter.follow_recommendations.common.rankers.utils.Utils\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\n/**\n * Candidate Ranker that mixes and ranks multiple candidate lists from different candidate sources with the\n * following steps:\n *  1) generate a ranked candidate list of each candidate source by sorting and shuffling the candidate list\n *     of the algorithm.\n *  2) merge the ranked lists generated in 1) into a single list using weighted randomly sampling.\n *  3) If dedup is required, dedup the output from 2) by candidate id.\n *\n * @param basedRanker base ranker\n * @param shuffleFn the shuffle function that will be used to shuffle each algorithm's sorted candidate list.\n * @param dedup whether to remove duplicated candidates from the final output.\n */\nclass WeightedCandidateSourceRanker[Target <: HasParams](\n  basedRanker: WeightedCandidateSourceBaseRanker[\n    CandidateSourceIdentifier,\n    CandidateUser\n  ],\n  shuffleFn: Seq[CandidateUser] => Seq[CandidateUser],\n  dedup: Boolean)\n    extends Ranker[Target, CandidateUser] {\n\n  val name: String = this.getClass.getSimpleName\n\n  override def rank(target: Target, candidates: Seq[CandidateUser]): Stitch[Seq[CandidateUser]] = {\n    val scribeRankingInfo: Boolean =\n      target.params(WeightedCandidateSourceRankerParams.ScribeRankingInfoInWeightedRanker)\n    val rankedCands = rankCandidates(group(candidates))\n    Stitch.value(if (scribeRankingInfo) Utils.addRankingInfo(rankedCands, name) else rankedCands)\n  }\n\n  private def group(\n    candidates: Seq[CandidateUser]\n  ): Map[CandidateSourceIdentifier, Seq[CandidateUser]] = {\n    val flattened = for {\n      candidate <- candidates\n      identifier <- candidate.getPrimaryCandidateSource\n    } yield (identifier, candidate)\n    flattened.groupBy(_._1).mapValues(_.map(_._2))\n  }\n\n  private def rankCandidates(\n    input: Map[CandidateSourceIdentifier, Seq[CandidateUser]]\n  ): Seq[CandidateUser] = {\n    // Sort and shuffle candidates per candidate source.\n    // Note 1: Using map instead mapValue here since mapValue somehow caused infinite loop when used as part of Stream.\n    val sortAndShuffledCandidates = input.map {\n      case (source, candidates) =>\n        // Note 2: toList is required here since candidates is a view, and it will result in infinit loop when used as part of Stream.\n        // Note 3: there is no real sorting logic here, it assumes the input is already sorted by candidate sources\n        val sortedCandidates = candidates.toList\n        source -> shuffleFn(sortedCandidates).iterator\n    }\n    val rankedCandidates = basedRanker(sortAndShuffledCandidates)\n\n    if (dedup) DedupCandidates(rankedCandidates) else rankedCandidates\n  }\n}\n\nobject WeightedCandidateSourceRanker {\n\n  def build[Target <: HasParams](\n    candidateSourceWeight: Map[CandidateSourceIdentifier, Double],\n    shuffleFn: Seq[CandidateUser] => Seq[CandidateUser] = identity,\n    dedup: Boolean = false,\n    randomSeed: Option[Long] = None\n  ): WeightedCandidateSourceRanker[Target] = {\n    new WeightedCandidateSourceRanker(\n      new WeightedCandidateSourceBaseRanker(\n        candidateSourceWeight,\n        WeightMethod.WeightedRandomSampling,\n        randomSeed = randomSeed),\n      shuffleFn,\n      dedup\n    )\n  }\n}\n\nobject WeightedCandidateSourceRankerWithoutRandomSampling {\n  def build[Target <: HasParams](\n    candidateSourceWeight: Map[CandidateSourceIdentifier, Double]\n  ): WeightedCandidateSourceRanker[Target] = {\n    new WeightedCandidateSourceRanker(\n      new WeightedCandidateSourceBaseRanker(\n        candidateSourceWeight,\n        WeightMethod.WeightedRoundRobin,\n        randomSeed = None),\n      identity,\n      false,\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/weighted_candidate_source_ranker/WeightedCandidateSourceRankerFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSParam\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass WeightedCandidateSourceRankerFSConfig @Inject() extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[FSParam[Boolean]] =\n    Seq(WeightedCandidateSourceRankerParams.ScribeRankingInfoInWeightedRanker)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/weighted_candidate_source_ranker/WeightedCandidateSourceRankerParams.scala",
    "content": "package com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker\n\nimport com.twitter.timelines.configapi.FSParam\n\nobject WeightedCandidateSourceRankerParams {\n  case object ScribeRankingInfoInWeightedRanker\n      extends FSParam[Boolean](\"weighted_ranker_scribe_ranking_info\", false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/stores/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/socialgraph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"strato/config/columns/onboarding/userrecs:userrecs-strato-client\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/stores/LowTweepCredFollowStore.scala",
    "content": "package com.twitter.follow_recommendations.common.stores\n\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedUserIds\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.onboarding.userrecs.TweepCredOnUserClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n// Not a candidate source since it's a intermediary.\n@Singleton\nclass LowTweepCredFollowStore @Inject() (tweepCredOnUserClientColumn: TweepCredOnUserClientColumn) {\n\n  def getLowTweepCredUsers(target: HasRecentFollowedUserIds): Stitch[Seq[CandidateUser]] = {\n    val newFollowings =\n      target.recentFollowedUserIds.getOrElse(Nil).take(LowTweepCredFollowStore.NumFlockToRetrieve)\n\n    val validTweepScoreUserIdsStitch: Stitch[Seq[Long]] = Stitch\n      .traverse(newFollowings) { newFollowingUserId =>\n        val tweepCredScoreOptStitch = tweepCredOnUserClientColumn.fetcher\n          .fetch(newFollowingUserId)\n          .map(_.v)\n        tweepCredScoreOptStitch.map(_.flatMap(tweepCred =>\n          if (tweepCred < LowTweepCredFollowStore.TweepCredThreshold) {\n            Some(newFollowingUserId)\n          } else {\n            None\n          }))\n      }.map(_.flatten)\n\n    validTweepScoreUserIdsStitch\n      .map(_.map(CandidateUser(_, Some(CandidateUser.DefaultCandidateScore))))\n  }\n}\n\nobject LowTweepCredFollowStore {\n  val NumFlockToRetrieve = 500\n  val TweepCredThreshold = 40\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/dedup/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/dedup/DedupTransform.scala",
    "content": "package com.twitter.follow_recommendations.common.transforms.dedup\n\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.stitch.Stitch\nimport scala.collection.mutable\n\nclass DedupTransform[Request, Candidate <: UniversalNoun[Long]]()\n    extends Transform[Request, Candidate] {\n  override def transform(target: Request, candidates: Seq[Candidate]): Stitch[Seq[Candidate]] = {\n    val seen = mutable.HashSet[Long]()\n    Stitch.value(candidates.filter(candidate => seen.add(candidate.id)))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/modify_social_proof/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"configapi/configapi-core\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/graph_feature_service\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/socialgraph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/deciders\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/modify_social_proof/ModifySocialProofTransform.scala",
    "content": "package com.twitter.follow_recommendations.common.transforms.modify_social_proof\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.decider.Decider\nimport com.twitter.decider.RandomRecipient\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.follow_recommendations.common.base.GatedTransform\nimport com.twitter.follow_recommendations.common.clients.graph_feature_service.GraphFeatureServiceClient\nimport com.twitter.follow_recommendations.common.clients.socialgraph.SocialGraphClient\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FollowProof\nimport com.twitter.follow_recommendations.configapi.deciders.DeciderKey\nimport com.twitter.graph_feature_service.thriftscala.EdgeType\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.logging.Logging\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject ModifySocialProof {\n  val GfsLagDuration: Duration = 14.days\n  val GfsIntersectionIds: Int = 3\n  val SgsIntersectionIds: Int = 10\n  val LeftEdgeTypes: Set[EdgeType] = Set(EdgeType.Following)\n  val RightEdgeTypes: Set[EdgeType] = Set(EdgeType.FollowedBy)\n\n  /**\n   * Given the intersection ID's for a particular candidate, update the candidate's social proof\n   * @param candidate          candidate object\n   * @param followProof        follow proof to be added (includes id's and count)\n   * @param stats              stats for tracking\n   * @return                   updated candidate object\n   */\n  def addIntersectionIdsToCandidate(\n    candidate: CandidateUser,\n    followProof: FollowProof,\n    stats: StatsReceiver\n  ): CandidateUser = {\n    // create updated set of social proof\n    val updatedFollowedByOpt = candidate.followedBy match {\n      case Some(existingFollowedBy) => Some((followProof.followedBy ++ existingFollowedBy).distinct)\n      case None if followProof.followedBy.nonEmpty => Some(followProof.followedBy.distinct)\n      case _ => None\n    }\n\n    val updatedFollowProof = updatedFollowedByOpt.map { updatedFollowedBy =>\n      val updatedCount = followProof.numIds.max(updatedFollowedBy.size)\n      // track stats\n      val numSocialProofAdded = updatedFollowedBy.size - candidate.followedBy.size\n      addCandidatesWithSocialContextCountStat(stats, numSocialProofAdded)\n      FollowProof(updatedFollowedBy, updatedCount)\n    }\n\n    candidate.setFollowProof(updatedFollowProof)\n  }\n\n  private def addCandidatesWithSocialContextCountStat(\n    statsReceiver: StatsReceiver,\n    count: Int\n  ): Unit = {\n    if (count > 3) {\n      statsReceiver.counter(\"4_and_more\").incr()\n    } else {\n      statsReceiver.counter(count.toString).incr()\n    }\n  }\n}\n\n/**\n * This class makes a request to gfs/sgs for hydrating additional social proof on each of the\n * provided candidates.\n */\n@Singleton\nclass ModifySocialProof @Inject() (\n  gfsClient: GraphFeatureServiceClient,\n  socialGraphClient: SocialGraphClient,\n  statsReceiver: StatsReceiver,\n  decider: Decider = Decider.True)\n    extends Logging {\n  import ModifySocialProof._\n\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val addedStats = stats.scope(\"num_social_proof_added_per_candidate\")\n  private val gfsStats = stats.scope(\"graph_feature_service\")\n  private val sgsStats = stats.scope(\"social_graph_service\")\n  private val previousProofEmptyCounter = stats.counter(\"previous_proof_empty\")\n  private val emptyFollowProofCounter = stats.counter(\"empty_followed_proof\")\n\n  /**\n   * For each candidate provided, we get the intersectionIds between the user and the candidate,\n   * appending the unique results to the social proof (followedBy field) if not already previously\n   * seen we query GFS for all users, except for cases specified via the mustCallSgs field or for\n   * very new users, who would not have any data in GFS, due to the lag duration of the service's\n   * processing. this is determined by GfsLagDuration\n   * @param userId id of the target user whom we provide recommendations for\n   * @param candidates list of candidates\n   * @param intersectionIdsNum if provided, determines the maximum number of accounts we want to be hydrated for social proof\n   * @param mustCallSgs Determines if we should query SGS regardless of user age or not.\n   * @return list of candidates updated with additional social proof\n   */\n  def hydrateSocialProof(\n    userId: Long,\n    candidates: Seq[CandidateUser],\n    intersectionIdsNum: Option[Int] = None,\n    mustCallSgs: Boolean = false,\n    callSgsCachedColumn: Boolean = false,\n    gfsLagDuration: Duration = GfsLagDuration,\n    gfsIntersectionIds: Int = GfsIntersectionIds,\n    sgsIntersectionIds: Int = SgsIntersectionIds,\n  ): Stitch[Seq[CandidateUser]] = {\n    addCandidatesWithSocialContextCountStat(\n      stats.scope(\"social_context_count_before_hydration\"),\n      candidates.count(_.followedBy.isDefined)\n    )\n    val candidateIds = candidates.map(_.id)\n    val userAgeOpt = SnowflakeId.timeFromIdOpt(userId).map(Time.now - _)\n\n    // this decider gate is used to determine what % of requests is allowed to call\n    // Graph Feature Service. this is useful for ramping down requests to Graph Feature Service\n    // when necessary\n    val deciderKey: String = DeciderKey.EnableGraphFeatureServiceRequests.toString\n    val enableGfsRequests: Boolean = decider.isAvailable(deciderKey, Some(RandomRecipient))\n\n    // if new query sgs\n    val (candidateToIntersectionIdsMapFut, isGfs) =\n      if (!enableGfsRequests || mustCallSgs || userAgeOpt.exists(_ < gfsLagDuration)) {\n        (\n          if (callSgsCachedColumn)\n            socialGraphClient.getIntersectionsFromCachedColumn(\n              userId,\n              candidateIds,\n              intersectionIdsNum.getOrElse(sgsIntersectionIds)\n            )\n          else\n            socialGraphClient.getIntersections(\n              userId,\n              candidateIds,\n              intersectionIdsNum.getOrElse(sgsIntersectionIds)),\n          false)\n      } else {\n        (\n          gfsClient.getIntersections(\n            userId,\n            candidateIds,\n            intersectionIdsNum.getOrElse(gfsIntersectionIds)),\n          true)\n      }\n    val finalCandidates = candidateToIntersectionIdsMapFut\n      .map { candidateToIntersectionIdsMap =>\n        {\n          previousProofEmptyCounter.incr(candidates.count(_.followedBy.exists(_.isEmpty)))\n          candidates.map { candidate =>\n            addIntersectionIdsToCandidate(\n              candidate,\n              candidateToIntersectionIdsMap.getOrElse(candidate.id, FollowProof(Seq.empty, 0)),\n              addedStats)\n          }\n        }\n      }\n      .within(250.milliseconds)(DefaultTimer)\n      .rescue {\n        case e: Exception =>\n          error(e.getMessage)\n          if (isGfs) {\n            gfsStats.scope(\"rescued\").counter(e.getClass.getSimpleName).incr()\n          } else {\n            sgsStats.scope(\"rescued\").counter(e.getClass.getSimpleName).incr()\n          }\n          Stitch.value(candidates)\n      }\n\n    finalCandidates.onSuccess { candidatesSeq =>\n      emptyFollowProofCounter.incr(candidatesSeq.count(_.followedBy.exists(_.isEmpty)))\n      addCandidatesWithSocialContextCountStat(\n        stats.scope(\"social_context_count_after_hydration\"),\n        candidatesSeq.count(_.followedBy.isDefined)\n      )\n    }\n  }\n}\n\n/**\n * This transform uses ModifySocialProof (which makes a request to gfs/sgs) for hydrating additional\n * social proof on each of the provided candidates.\n */\n@Singleton\nclass ModifySocialProofTransform @Inject() (modifySocialProof: ModifySocialProof)\n    extends GatedTransform[HasClientContext with HasParams, CandidateUser]\n    with Logging {\n\n  override def transform(\n    target: HasClientContext with HasParams,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Seq[CandidateUser]] =\n    target.getOptionalUserId\n      .map(modifySocialProof.hydrateSocialProof(_, candidates)).getOrElse(Stitch.value(candidates))\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/modify_social_proof/RemoveAccountProofTransform.scala",
    "content": "package com.twitter.follow_recommendations.common.transforms.modify_social_proof\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.GatedTransform\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RemoveAccountProofTransform @Inject() (statsReceiver: StatsReceiver)\n    extends GatedTransform[HasClientContext with HasParams, CandidateUser] {\n\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val removedProofsCounter = stats.counter(\"num_removed_proofs\")\n\n  override def transform(\n    target: HasClientContext with HasParams,\n    items: Seq[CandidateUser]\n  ): Stitch[Seq[CandidateUser]] =\n    Stitch.value(items.map { candidate =>\n      removedProofsCounter.incr()\n      candidate.copy(reason = None)\n    })\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/ranker_id/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/constants\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/ranker_id/RandomRankerIdTransform.scala",
    "content": "package com.twitter.follow_recommendations.common.transforms.ranker_id\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.follow_recommendations.common.base.GatedTransform\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.Score\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\n\n/**\n * This class appends each candidate's rankerIds with the RandomRankerId.\n * This is primarily for determining if a candidate was generated via random shuffling.\n */\n@Singleton\nclass RandomRankerIdTransform @Inject() () extends GatedTransform[HasParams, CandidateUser] {\n\n  override def transform(\n    target: HasParams,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Seq[CandidateUser]] = {\n    Stitch.value(candidates.map(_.addScore(Score.RandomScore)))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/recommendation_flow_identifier/AddRecommendationFlowIdentifierTransform.scala",
    "content": "package com.twitter.follow_recommendations.common.transforms.recommendation_flow_identifier\n\nimport com.google.inject.Inject\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasRecommendationFlowIdentifier\nimport com.twitter.stitch.Stitch\n\nclass AddRecommendationFlowIdentifierTransform @Inject()\n    extends Transform[HasRecommendationFlowIdentifier, CandidateUser] {\n\n  override def transform(\n    target: HasRecommendationFlowIdentifier,\n    items: Seq[CandidateUser]\n  ): Stitch[Seq[CandidateUser]] = {\n    Stitch.value(items.map { candidateUser =>\n      candidateUser.copy(recommendationFlowIdentifier = target.recommendationFlowIdentifier)\n    })\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/recommendation_flow_identifier/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/tracking_token/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/constants\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/tracking_token/TrackingTokenTransform.scala",
    "content": "package com.twitter.follow_recommendations.common.transforms.tracking_token\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.Session\nimport com.twitter.follow_recommendations.common.models.TrackingToken\nimport com.twitter.hermit.constants.AlgorithmFeedbackTokens.AlgorithmToFeedbackTokenMap\nimport com.twitter.hermit.model.Algorithm\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.logging.Logging\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * This transform adds the tracking token for all candidates\n * Since this happens in the same request, we use the same trace-id for all candidates\n * There are no RPC calls in this transform so it's safe to chain it with `andThen` at the end of\n * all other product-specific transforms\n */\n@Singleton\nclass TrackingTokenTransform @Inject() (baseStatsReceiver: StatsReceiver)\n    extends Transform[HasDisplayLocation with HasClientContext, CandidateUser]\n    with Logging {\n\n  def profileResults(\n    target: HasDisplayLocation with HasClientContext,\n    candidates: Seq[CandidateUser]\n  ) = {\n    // Metrics to track # results per candidate source\n    val stats = baseStatsReceiver.scope(target.displayLocation.toString + \"/final_results\")\n    stats.stat(\"total\").add(candidates.size)\n\n    stats.counter(target.displayLocation.toString).incr()\n\n    val flattenedCandidates: Seq[(CandidateSourceIdentifier, CandidateUser)] = for {\n      candidate <- candidates\n      identifier <- candidate.getPrimaryCandidateSource\n    } yield (identifier, candidate)\n    val candidatesGroupedBySource: Map[CandidateSourceIdentifier, Seq[CandidateUser]] =\n      flattenedCandidates.groupBy(_._1).mapValues(_.map(_._2))\n    candidatesGroupedBySource map {\n      case (source, candidates) => stats.stat(source.name).add(candidates.size)\n    }\n  }\n\n  override def transform(\n    target: HasDisplayLocation with HasClientContext,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Seq[CandidateUser]] = {\n    profileResults(target, candidates)\n\n    Stitch.value(\n      target.getOptionalUserId\n        .map { _ =>\n          candidates.map {\n            candidate =>\n              val token = Some(TrackingToken(\n                sessionId = Session.getSessionId,\n                displayLocation = Some(target.displayLocation),\n                controllerData = None,\n                algorithmId = candidate.userCandidateSourceDetails.flatMap(_.primaryCandidateSource\n                  .flatMap { identifier =>\n                    Algorithm.withNameOpt(identifier.name).flatMap(AlgorithmToFeedbackTokenMap.get)\n                  })\n              ))\n              candidate.copy(trackingToken = token)\n          }\n        }.getOrElse(candidates))\n\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/weighted_sampling/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/utils\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/weighted_sampling/SamplingTransform.scala",
    "content": "package com.twitter.follow_recommendations.common.transforms.weighted_sampling\nimport com.twitter.follow_recommendations.common.base.GatedTransform\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDebugOptions\nimport com.twitter.follow_recommendations.common.models.Score\nimport com.twitter.follow_recommendations.common.models.Scores\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId\nimport com.twitter.follow_recommendations.common.rankers.utils.Utils\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SamplingTransform @Inject() ()\n    extends GatedTransform[HasClientContext with HasParams with HasDebugOptions, CandidateUser] {\n\n  val name: String = this.getClass.getSimpleName\n\n  /*\n  Description: This function takes in a set of candidate users and ranks them for a who-to-follow\n  request by sampling from the Placket-Luce distribution\n  (https://cran.rstudio.com/web/packages/PlackettLuce/vignettes/Overview.html) with a three\n  variations. The first variation is that the scores of the candidates are multiplied by\n  multiplicativeFactor before sampling. The second variation is that the scores are\n  exponentiated before sampling. The third variation is that depending on how many who-to-follow\n  positions are being requested, the first k positions are reserved for the candidates with the\n  highest scores (and they are sorted in decreasing order of score) and the remaining positions\n  are sampled from a Placket-Luce. We use the efficient algorithm proposed in this blog\n  https://medium.com/swlh/going-old-school-designing-algorithms-for-fast-weighted-sampling-in-production-c48fc1f40051\n  to sample from a Plackett-Luce. Because of numerical stability reasons, before sampling from this\n  distribution, (1) we subtract off the maximum score from all the scores and (2) if after\n  this subtraction and multiplication by the multiplicative factor the resulting score is <= -10,\n  we force the candidate's transformed score under the above algorithm to be 0 (so r^(1/w) = 0)\n  where r is a random number and w is the transformed score.\n\n  inputs:\n  - target: HasClientContext (WTF request)\n  - candidates: sequence of CandidateUsers (users that need to be ranked from a who-to-follow\n                request) each of which has a score\n\n  inputs accessed through feature switches, i.e. through target.params (see the following file:\n  \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/\n  transforms/weighted_sampling/SamplingTransformParams.scala\"):\n  - topKFixed: the first k positions of the who-to-follow ranking correspond to the users with the k\n               highest scores and are not sampled from the Placket-Luce distribution\n  - multiplicativeFactor: multiplicativeFactor is used to transform the scores of each candidate by\n                          multiplying that user's score by multiplicativeFactor\n\n  output:\n  - Sequence of CandidateUser whose order represents the ranking of users in a who-to-follow request\n    This ranking is sampled from a Placket-Luce distribution.\n   */\n  override def transform(\n    target: HasClientContext with HasParams with HasDebugOptions,\n    candidates: Seq[CandidateUser]\n  ): Stitch[Seq[CandidateUser]] = {\n\n    // the first k positions of the who-to-follow ranking correspond to the users with the k\n    // highest scores and are not sampled from the Placket-Luce distribution\n    val topKFixed = target.params(SamplingTransformParams.TopKFixed)\n\n    // multiplicativeFactor is used to transform the scores of each candidate by\n    // multiplying that user's score by multiplicativeFactor\n    val multiplicativeFactor = target.params(SamplingTransformParams.MultiplicativeFactor)\n\n    // sort candidates by their score\n    val candidatesSorted = candidates.sortBy(-1 * _.score.getOrElse(0.0))\n\n    // pick the top K candidates by score and the remaining candidates\n    val (topKFixedCandidates, candidatesOutsideOfTopK) =\n      candidatesSorted.zipWithIndex.partition { case (value, index) => index < topKFixed }\n\n    val randomNumGenerator =\n      new scala.util.Random(target.getRandomizationSeed.getOrElse(System.currentTimeMillis))\n\n    // we need to subtract the maximum score off the scores for numerical stability reasons\n    // subtracting the max score off does not effect the underlying distribution we are sampling\n    // the candidates from\n    // we need the if statement since you cannot take the max of an empty sequence\n    val maximum_score = if (candidatesOutsideOfTopK.nonEmpty) {\n      candidatesOutsideOfTopK.map(x => x._1.score.getOrElse(0.0)).max\n    } else {\n      0.0\n    }\n\n    // for candidates in candidatesOutsideOfTopK, we transform their score by subtracting off\n    // maximum_score and then multiply by multiplicativeFactor\n    val candidatesOutsideOfTopKTransformedScore = candidatesOutsideOfTopK.map(x =>\n      (x._1, multiplicativeFactor * (x._1.score.getOrElse(0.0) - maximum_score)))\n\n    // for each candidate with score transformed and clip score w, sample a random number r,\n    // create a new score r^(1/w) and sort the candidates to get the final ranking.\n    // for numerical stability reasons if the score is <=-10, we force r^(1/w) = 0.\n    // this samples the candidates from the modified Plackett-Luce distribution. See\n    // https://medium.com/swlh/going-old-school-designing-algorithms-for-fast-weighted-sampling-in-production-c48fc1f40051\n\n    val candidatesOutsideOfTopKSampled = candidatesOutsideOfTopKTransformedScore\n      .map(x =>\n        (\n          x._1,\n          if (x._2 <= -10.0)\n            0.0\n          else\n            scala.math.pow(\n              randomNumGenerator.nextFloat(),\n              1 / (scala.math\n                .exp(x._2))))).sortBy(-1 * _._2)\n\n    val topKCandidates: Seq[CandidateUser] = topKFixedCandidates.map(_._1)\n\n    val scribeRankingInfo: Boolean =\n      target.params(SamplingTransformParams.ScribeRankingInfoInSamplingTransform)\n\n    val transformedCandidates: Seq[CandidateUser] = if (scribeRankingInfo) {\n      val topKCandidatesWithRankingInfo: Seq[CandidateUser] =\n        Utils.addRankingInfo(topKCandidates, name)\n      val candidatesOutsideOfTopKSampledWithRankingInfo: Seq[CandidateUser] =\n        candidatesOutsideOfTopKSampled.zipWithIndex.map {\n          case ((candidate, score), rank) =>\n            val newScore = Seq(Score(score, Some(RankerId.PlacketLuceSamplingTransformer)))\n            val newScores: Option[Scores] = candidate.scores\n              .map { scores =>\n                scores.copy(scores = scores.scores ++ newScore)\n              }.orElse(Some(Scores(newScore, Some(RankerId.PlacketLuceSamplingTransformer))))\n            val globalRank = rank + topKFixed + 1\n            candidate.addInfoPerRankingStage(name, newScores, globalRank)\n        }\n\n      topKCandidatesWithRankingInfo ++ candidatesOutsideOfTopKSampledWithRankingInfo\n    } else {\n      topKCandidates ++ candidatesOutsideOfTopKSampled.map(_._1)\n    }\n\n    Stitch.value(transformedCandidates)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/weighted_sampling/SamplingTransformFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.common.transforms.weighted_sampling\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SamplingTransformFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val intFSParams: Seq[FSBoundedParam[Int]] = Seq(SamplingTransformParams.TopKFixed)\n\n  override val doubleFSParams: Seq[FSBoundedParam[Double]] = Seq(\n    SamplingTransformParams.MultiplicativeFactor)\n\n  override val booleanFSParams: Seq[FSParam[Boolean]] = Seq(\n    SamplingTransformParams.ScribeRankingInfoInSamplingTransform)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/weighted_sampling/SamplingTransformParams.scala",
    "content": "package com.twitter.follow_recommendations.common.transforms.weighted_sampling\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject SamplingTransformParams {\n\n  case object TopKFixed // indicates how many of the fisrt K who-to-follow recommendations are reserved for the candidates with largest K CandidateUser.score where these candidates are sorted in decreasing order of score\n      extends FSBoundedParam[Int](\n        name = \"post_nux_ml_flow_weighted_sampling_top_k_fixed\",\n        default = 0,\n        min = 0,\n        max = 100)\n\n  case object MultiplicativeFactor // CandidateUser.score gets transformed to multiplicativeFactor*CandidateUser.score before sampling from the Plackett-Luce distribution\n      extends FSBoundedParam[Double](\n        name = \"post_nux_ml_flow_weighted_sampling_multiplicative_factor\",\n        default = 1.0,\n        min = -1000.0,\n        max = 1000.0)\n\n  case object ScribeRankingInfoInSamplingTransform\n      extends FSParam[Boolean](\"sampling_transform_scribe_ranking_info\", false)\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils/BUILD",
    "content": "scala_library(\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/request\",\n        \"stitch/stitch-core\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils/CollectionUtil.scala",
    "content": "package com.twitter.follow_recommendations.common.utils\n\nobject CollectionUtil {\n\n  /**\n   * Transposes a sequence of sequences. As opposed to the Scala collection library version\n   * of transpose, the sequences do not have to have the same length.\n   *\n   * Example:\n   * transpose(immutable.Seq(immutable.Seq(1,2,3), immutable.Seq(4,5), immutable.Seq(6,7)))\n   *   => immutable.Seq(immutable.Seq(1, 4, 6), immutable.Seq(2, 5, 7), immutable.Seq(3))\n   *\n   * @param seq a sequence of sequences\n   * @tparam A the type of elements in the seq\n   * @return the transposed sequence of sequences\n   */\n  def transposeLazy[A](seq: Seq[Seq[A]]): Stream[Seq[A]] =\n    seq.filter(_.nonEmpty) match {\n      case Nil => Stream.empty\n      case ys => ys.map(_.head) #:: transposeLazy(ys.map(_.tail))\n    }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils/DisplayLocationProductConverterUtil.scala",
    "content": "package com.twitter.follow_recommendations.common.utils\n\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.common.models.Product\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\n\nobject DisplayLocationProductConverterUtil {\n  def productToDisplayLocation(product: Product): DisplayLocation = {\n    product match {\n      case Product.MagicRecs => DisplayLocation.MagicRecs\n      case _ =>\n        throw UnconvertibleProductMixerProductException(\n          s\"Cannot convert Product Mixer Product ${product.identifier.name} into a FRS DisplayLocation.\")\n    }\n  }\n\n  def displayLocationToProduct(displayLocation: DisplayLocation): Product = {\n    displayLocation match {\n      case DisplayLocation.MagicRecs => Product.MagicRecs\n      case _ =>\n        throw UnconvertibleProductMixerProductException(\n          s\"Cannot convert DisplayLocation ${displayLocation.toFsName} into a Product Mixer Product.\")\n    }\n  }\n}\n\ncase class UnconvertibleProductMixerProductException(message: String) extends Exception(message)\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils/MergeUtil.scala",
    "content": "package com.twitter.follow_recommendations.common.utils\n\nobject MergeUtil {\n\n  /**\n   * Takes a seq of items which have weights. Returns an infinite stream of each item\n   * by their weights. All weights need to be greater than or equal to zero. In addition,\n   * the sum of weights should be greater than zero.\n   *\n   * Example usage of this function:\n   * Input weighted Item {{CS1, 3}, {CS2, 2}, {CS3, 5}}\n   * Output stream: (CS1, CS1, CS1, CS2, CS2, CS3, CS3, CS3, CS3, CS3, CS1, CS1, CS1, CS2,...}\n   *\n   * @param items    items\n   * @param weighted provides weights for items\n   * @tparam T type of item\n   *\n   * @return Stream of Ts\n   */\n  def weightedRoundRobin[T](\n    items: Seq[T]\n  )(\n    implicit weighted: Weighted[T]\n  ): Stream[T] = {\n    if (items.isEmpty) {\n      Stream.empty\n    } else {\n      val weights = items.map { i => weighted(i) }\n      assert(\n        weights.forall {\n          _ >= 0\n        },\n        \"Negative weight exists for sampling\")\n      val cumulativeWeight = weights.scanLeft(0.0)(_ + _).tail\n      assert(cumulativeWeight.last > 0, \"Sum of the sampling weights is not positive\")\n\n      var weightIdx = 0\n      var weight = 0\n\n      def next(): Stream[T] = {\n        val tmpIdx = weightIdx\n        weight = weight + 1\n        weight = if (weight >= weights(weightIdx)) 0 else weight\n        weightIdx = if (weight == 0) weightIdx + 1 else weightIdx\n        weightIdx = if (weightIdx == weights.length) 0 else weightIdx\n        items(tmpIdx) #:: next()\n      }\n      next()\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils/RandomUtil.scala",
    "content": "package com.twitter.follow_recommendations.common.utils\nimport scala.util.Random\n\nobject RandomUtil {\n\n  /**\n   * Takes a seq of items which have weights. Returns an infinite stream that is\n   * sampled with replacement using the weights for each item. All weights need\n   * to be greater than or equal to zero. In addition, the sum of weights\n   * should be greater than zero.\n   *\n   * @param items items\n   * @param weighted provides weights for items\n   * @tparam T type of item\n   * @return Stream of Ts\n   */\n  def weightedRandomSamplingWithReplacement[T](\n    items: Seq[T],\n    random: Option[Random] = None\n  )(\n    implicit weighted: Weighted[T]\n  ): Stream[T] = {\n    if (items.isEmpty) {\n      Stream.empty\n    } else {\n      val weights = items.map { i => weighted(i) }\n      assert(weights.forall { _ >= 0 }, \"Negative weight exists for sampling\")\n      val cumulativeWeight = weights.scanLeft(0.0)(_ + _).tail\n      assert(cumulativeWeight.last > 0, \"Sum of the sampling weights is not positive\")\n      val cumulativeProbability = cumulativeWeight map (_ / cumulativeWeight.last)\n      def next(): Stream[T] = {\n        val rand = random.getOrElse(Random).nextDouble()\n        val idx = cumulativeProbability.indexWhere(_ >= rand)\n        items(if (idx == -1) items.length - 1 else idx) #:: next()\n      }\n      next()\n    }\n  }\n\n  /**\n   * Takes a seq of items and their weights. Returns a lazy weighted shuffle of\n   * the elements in the list. All weights should be greater than zero.\n   *\n   * @param items items\n   * @param weighted provides weights for items\n   * @tparam T type of item\n   * @return Stream of Ts\n   */\n  def weightedRandomShuffle[T](\n    items: Seq[T],\n    random: Option[Random] = None\n  )(\n    implicit weighted: Weighted[T]\n  ): Stream[T] = {\n    assert(items.forall { i => weighted(i) > 0 }, \"Non-positive weight exists for shuffling\")\n    def next(it: Seq[T]): Stream[T] = {\n      if (it.isEmpty)\n        Stream.empty\n      else {\n        val cumulativeWeight = it.scanLeft(0.0)((acc: Double, curr: T) => acc + weighted(curr)).tail\n        val cutoff = random.getOrElse(Random).nextDouble() * cumulativeWeight.last\n        val idx = cumulativeWeight.indexWhere(_ >= cutoff)\n        val (left, right) = it.splitAt(idx)\n        it(if (idx == -1) it.size - 1 else idx) #:: next(left ++ right.drop(1))\n      }\n    }\n    next(items)\n  }\n\n  /**\n   * Takes a seq of items and a weight function, returns a lazy weighted shuffle of\n   * the elements in the list.The weight function is based on the rank of the element\n   * in the original lst.\n   * @param items\n   * @param rankToWeight\n   * @param random\n   * @tparam T\n   * @return\n   */\n  def weightedRandomShuffleByRank[T](\n    items: Seq[T],\n    rankToWeight: Int => Double,\n    random: Option[Random] = None\n  ): Stream[T] = {\n    val candWeights = items.zipWithIndex.map { case (item, rank) => (item, rankToWeight(rank)) }\n    RandomUtil.weightedRandomShuffle(candWeights, random).map(_._1)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils/RescueWithStatsUtils.scala",
    "content": "package com.twitter.follow_recommendations.common.utils\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Duration\nimport com.twitter.util.TimeoutException\n\nobject RescueWithStatsUtils {\n  def rescueWithStats[T](\n    s: Stitch[Seq[T]],\n    stats: StatsReceiver,\n    source: String\n  ): Stitch[Seq[T]] = {\n    StatsUtil.profileStitchSeqResults(s, stats.scope(source)).rescue {\n      case _: Exception => Stitch.Nil\n    }\n  }\n\n  def rescueOptionalWithStats[T](\n    s: Stitch[Option[T]],\n    stats: StatsReceiver,\n    source: String\n  ): Stitch[Option[T]] = {\n    StatsUtil.profileStitchOptionalResults(s, stats.scope(source)).rescue {\n      case _: Exception => Stitch.None\n    }\n  }\n\n  def rescueWithStatsWithin[T](\n    s: Stitch[Seq[T]],\n    stats: StatsReceiver,\n    source: String,\n    timeout: Duration\n  ): Stitch[Seq[T]] = {\n    val hydratedScopeSource = stats.scope(source)\n    StatsUtil\n      .profileStitchSeqResults(\n        s.within(timeout)(com.twitter.finagle.util.DefaultTimer),\n        hydratedScopeSource)\n      .rescue {\n        case _: TimeoutException =>\n          hydratedScopeSource.counter(\"timeout\").incr()\n          Stitch.Nil\n        case _: Exception =>\n          hydratedScopeSource.counter(\"exception\").incr()\n          Stitch.Nil\n      }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils/UserSignupUtil.scala",
    "content": "package com.twitter.follow_recommendations.common.utils\n\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\n\nobject UserSignupUtil {\n  def signupTime(hasClientContext: HasClientContext): Option[Time] =\n    hasClientContext.clientContext.userId.flatMap(SnowflakeId.timeFromIdOpt)\n\n  def userSignupAge(hasClientContext: HasClientContext): Option[Duration] =\n    signupTime(hasClientContext).map(Time.now - _)\n}\n"
  },
  {
    "path": "follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils/Weighted.scala",
    "content": "package com.twitter.follow_recommendations.common.utils\n\n/**\n * Typeclass for any Recommendation type that has a weight\n *\n */\ntrait Weighted[-Rec] {\n  def apply(rec: Rec): Double\n}\n\nobject Weighted {\n  implicit object WeightedTuple extends Weighted[(_, Double)] {\n    override def apply(rec: (_, Double)): Double = rec._2\n  }\n\n  def fromFunction[Rec](f: Rec => Double): Weighted[Rec] = {\n    new Weighted[Rec] {\n      override def apply(rec: Rec): Double = f(rec)\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/resources/BUILD",
    "content": "resources(\n    sources = [\n        \"*.tsv\",\n        \"*.xml\",\n        \"**/*\",\n        \"config/*.yml\",\n    ],\n)\n\n# Created for Bazel compatibility.\n# In Bazel, loose files must be part of a target to be included into a bundle.\nfiles(\n    name = \"frs_resources\",\n    sources = [\n        \"*.tsv\",\n        \"*.xml\",\n        \"*.yml\",\n        \"**/*\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/resources/config/decider.yml",
    "content": "enable_recommendations:\n  comment: Proportion of requests where we return an actual response as part. Decreasing the value will increase the portion of empty responses (in order to disable the service) as part of the graceful degradation.\n  default_availability: 10000\nenable_score_user_candidates:\n  comment: Proportion of requests where score user candidates from the scoreUserCandidates endpoint\n  default_availability: 10000\nenable_profile_sidebar_product:\n  comment: Proportion of requests where we return an actual response for profile sidebar product\n  default_availability: 10000\nenable_magic_recs_product:\n  comment: Proportion of requests where we return an actual response for magic recs product\n  default_availability: 10000\nenable_rux_landing_page_product:\n  comment: Proportion of requests where we return an actual response for rux landing page product\n  default_availability: 10000\nenable_rux_pymk_product:\n  comment: Proportion of requests where we return an actual response for rux pymk product\n  default_availability: 10000\nenable_profile_bonus_follow_product:\n  comment: Proportion of requests where we return an actual response for profile bonus follow product\n  default_availability: 10000\nenable_election_explore_wtf_product:\n  comment: Proportion of requests where we return an actual response for election explore wtf product\n  default_availability: 10000\nenable_cluster_follow_product:\n  comment: Proportion of requests where we return an actual response for cluster follow product\n  default_availability: 10000\nenable_home_timeline_product:\n  comment: Proportion of requests where we return an actual response for htl wtf product\n  default_availability: 10000\nenable_htl_bonus_follow_product:\n  comment: Proportion of requests where we return an actual response for htl bonus follow product\n  default_availability: 10000\nenable_explore_tab_product:\n  comment: Proportion of requests where we return an actual response for explore tab product\n  default_availability: 10000\nenable_sidebar_product:\n  comment: Proportion of requests where we return an actual response for sidebar product\n  default_availability: 10000\nenable_campaign_form_product:\n  comment: Proportion of requests where we return an actual response for campaign form product\n  default_availability: 10000\nenable_reactive_follow_product:\n  comment: Proportion of requests where we return an actual response for reactive follow product\n  default_availability: 10000\nenable_nux_pymk_product:\n  comment: Proportion of requests where we return an actual response for nux pymk product\n  default_availability: 10000\nenable_nux_interests_product:\n  comment: Proportion of requests where we return an actual response for nux interests product\n  default_availability: 10000\nenable_nux_topic_bonus_follow_product:\n  comment: Proportion of requests where we return an actual response for nux topic-based bonus follow product\n  default_availability: 10000\nenable_india_covid19_curated_accounts_wtf_product:\n  comment: Proportion of requests where we return an actual response for india covid19 curated accounts wtf product\n  default_availability: 10000\nenable_ab_upload_product:\n  comment: Proportion of requests where we return an actual response for the address book upload product\n  default_availability: 10000\nenable_people_plus_plus_product:\n  comment: Proportion of requests where we return an actual response for the PeoplePlusPlus/Connect Tab product\n  default_availability: 10000\nenable_tweet_notification_recs_product:\n  comment: Proportion of requests where we return an actual response for the Tweet Notification Recommendations product\n  default_availability: 10000\nenable_profile_device_follow_product:\n  comment: Proportion of requests where we return an actual response for the ProfileDeviceFollow product\n  default_availability: 10000\nenable_diffy_module_dark_reading:\n  comment: Percentage of dark read traffic routed to diffy thrift\n  default_availability: 0\nenable_recos_backfill_product:\n  comment: Proportion of requests where we return an actual response for the RecosBackfill product\n  default_availability: 10000\nenable_post_nux_follow_task_product:\n  comment: Proportion of requests where we return an actual response for post NUX follow task product\n  default_availability: 10000\nenable_curated_space_hosts_product:\n  comment: Proportion of requests where we return an actual response for curated space hosts product\n  default_availability: 10000\nenable_nux_geo_category_product:\n  comment: Proportion of requests where we return an actual response for nux geo category product\n  default_availability: 10000\nenable_nux_interests_category_product:\n  comment: Proportion of requests where we return an actual response for nux interests category product\n  default_availability: 10000\nenable_nux_pymk_category_product:\n  comment: Proportion of requests where we return an actual response for nux pymk category product\n  default_availability: 10000\nenable_home_timeline_tweet_recs_product:\n  comment: Proportion of requests where we return an actual response for the Home Timeline Tweet Recs product\n  default_availability: 10000\nenable_htl_bulk_friend_follows_product:\n  comment: Proportion of requests where we return an actual response for the HTL bulk friend follows product\n  default_availability: 10000\nenable_nux_auto_follow_product:\n  comment: Proportion of requests where we return an actual response for the NUX auto follow product\n  default_availability: 10000\nenable_search_bonus_follow_product:\n  comment: Proportion of requests where we return an actual response for search bonus follow product\n  default_availability: 10000\nenable_fetch_user_in_request_builder:\n  comment: Proportion of requests where we fetch user object from gizmoduck in request builder\n  default_availability: 0\nenable_product_mixer_magic_recs_product:\n  comment: Proportion of requests where we enable the product mixer magic recs product\n  default_availability: 10000\nenable_home_timeline_reverse_chron_product:\n  comment: Proportion of requests where we return an actual response for Home timeline reverse chron product\n  default_availability: 10000\nenable_product_mixer_pipeline_magic_recs_dark_read:\n  comment: Compare product mixer pipeline responses to current FRS pipeline responses for Magic Recs\n  default_availability: 0\nenable_experimental_caching:\n  comment: Proportion of requests we use experimental caching for data caching\n  default_availability: 0\nenable_distributed_caching:\n  comment: Proportion of requests we use a distributed cache cluster for data caching\n  default_availability: 10000\nenable_gizmoduck_caching:\n  comment: Proportion of requests we use a distributed cache cluster for data caching in Gizmoduck\n  default_availability: 10000\nenable_traffic_dark_reading:\n  comment: Proportion of requests where we replicate the request for traffic dark reading\n  default_availability: 0\nenable_graph_feature_service_requests:\n  comment: Proportion of requests where we allow request calls to Graph Feature Service\n  default_availability: 10000\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/resources/logback.xml",
    "content": "<configuration>\n  <shutdownHook class=\"ch.qos.logback.core.hook.DelayingShutdownHook\"/>\n\n  <!-- ===================================================== -->\n  <!-- Service Config -->\n  <!-- ===================================================== -->\n  <property name=\"DEFAULT_SERVICE_PATTERN\"\n            value=\"%-16X{traceId} %-12X{serviceIdentifier:--} %-16X{method} %-12X{product:--} %-25logger{0} %msg\"/>\n\n  <property name=\"DEFAULT_ACCESS_PATTERN\"\n            value=\"%msg %-12X{serviceIdentifier:--} %X{traceId} %X{product:--}\"/>\n\n  <!-- ===================================================== -->\n  <!-- Common Config -->\n  <!-- ===================================================== -->\n\n  <!-- JUL/JDK14 to Logback bridge -->\n  <contextListener class=\"ch.qos.logback.classic.jul.LevelChangePropagator\">\n    <resetJUL>true</resetJUL>\n  </contextListener>\n\n  <!-- Service Log (Rollover every 50MB, max 5 logs) -->\n  <appender name=\"SERVICE\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n    <file>${log.service.output}</file>\n    <rollingPolicy class=\"ch.qos.logback.core.rolling.FixedWindowRollingPolicy\">\n      <fileNamePattern>${log.service.output}.%i</fileNamePattern>\n      <minIndex>1</minIndex>\n      <maxIndex>5</maxIndex>\n    </rollingPolicy>\n    <triggeringPolicy class=\"ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy\">\n      <maxFileSize>50MB</maxFileSize>\n    </triggeringPolicy>\n    <encoder>\n      <pattern>%date %.-3level ${DEFAULT_SERVICE_PATTERN}%n</pattern>\n    </encoder>\n  </appender>\n\n  <!-- Access Log (Rollover every 50MB, max 5 logs) -->\n  <appender name=\"ACCESS\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n    <file>${log.access.output}</file>\n    <rollingPolicy class=\"ch.qos.logback.core.rolling.FixedWindowRollingPolicy\">\n      <fileNamePattern>${log.access.output}.%i</fileNamePattern>\n      <minIndex>1</minIndex>\n      <maxIndex>5</maxIndex>\n    </rollingPolicy>\n    <triggeringPolicy class=\"ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy\">\n      <maxFileSize>50MB</maxFileSize>\n    </triggeringPolicy>\n    <encoder>\n      <pattern>${DEFAULT_ACCESS_PATTERN}%n</pattern>\n    </encoder>\n  </appender>\n\n  <!--LogLens -->\n  <appender name=\"LOGLENS\" class=\"com.twitter.loglens.logback.LoglensAppender\">\n    <mdcAdditionalContext>true</mdcAdditionalContext>\n    <category>${log.lens.category}</category>\n    <index>${log.lens.index}</index>\n    <tag>${log.lens.tag}/service</tag>\n    <encoder>\n      <pattern>%msg</pattern>\n    </encoder>\n  </appender>\n\n  <!-- LogLens Access -->\n  <appender name=\"LOGLENS-ACCESS\" class=\"com.twitter.loglens.logback.LoglensAppender\">\n    <mdcAdditionalContext>true</mdcAdditionalContext>\n    <category>${log.lens.category}</category>\n    <index>${log.lens.index}</index>\n    <tag>${log.lens.tag}/access</tag>\n    <encoder>\n      <pattern>%msg</pattern>\n    </encoder>\n  </appender>\n\n  <!-- ===================================================== -->\n  <!-- Primary Async Appenders -->\n  <!-- ===================================================== -->\n\n  <property name=\"async_queue_size\" value=\"${queue.size:-50000}\"/>\n  <property name=\"async_max_flush_time\" value=\"${max.flush.time:-0}\"/>\n\n  <appender name=\"ASYNC-SERVICE\" class=\"com.twitter.inject.logback.AsyncAppender\">\n    <queueSize>${async_queue_size}</queueSize>\n    <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n    <appender-ref ref=\"SERVICE\"/>\n  </appender>\n\n  <appender name=\"ASYNC-ACCESS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n    <queueSize>${async_queue_size}</queueSize>\n    <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n    <appender-ref ref=\"ACCESS\"/>\n  </appender>\n\n  <appender name=\"ASYNC-LOGLENS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n    <queueSize>${async_queue_size}</queueSize>\n    <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n    <appender-ref ref=\"LOGLENS\"/>\n  </appender>\n\n  <appender name=\"ASYNC-LOGLENS-ACCESS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n    <queueSize>${async_queue_size}</queueSize>\n    <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n    <appender-ref ref=\"LOGLENS-ACCESS\"/>\n  </appender>\n\n  <!-- ===================================================== -->\n  <!-- Package Config -->\n  <!-- ===================================================== -->\n\n  <!-- Per-Package Config -->\n  <logger name=\"com.twitter\" level=\"info\"/>\n  <logger name=\"com.twitter.wilyns\" level=\"warn\"/>\n  <logger name=\"com.twitter.finagle.mux\" level=\"warn\"/>\n  <logger name=\"com.twitter.finagle.serverset2\" level=\"warn\"/>\n  <logger name=\"com.twitter.logging.ScribeHandler\" level=\"warn\"/>\n  <logger name=\"com.twitter.zookeeper.client.internal\" level=\"warn\"/>\n\n  <!-- Root Config -->\n  <root level=\"${log_level:-INFO}\">\n    <appender-ref ref=\"ASYNC-SERVICE\"/>\n    <appender-ref ref=\"ASYNC-LOGLENS\"/>\n  </root>\n\n  <!-- Access Logging -->\n  <logger name=\"com.twitter.finatra.thrift.filters.AccessLoggingFilter\"\n          level=\"info\"\n          additivity=\"false\">\n    <appender-ref ref=\"ASYNC-ACCESS\"/>\n    <appender-ref ref=\"ASYNC-LOGLENS-ACCESS\"/>\n  </logger>\n\n</configuration>\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/resources/quality/stp_models/20141223/epModel",
    "content": "# OWNER = jdeng\n# Date = 20141223_153423\n# Training Size = 16744473\n# Testing Size = 16767335\n#  trained with ElasticNetCV alpha=0.05 cv_folds=5 best_lambda=1.0E-7\n# num base features: 10\n# num nonzero weights: 30\n{bias:-5.67151,featureMetadataMap:[\"fwd_email\":{metadata:{featureWeight:{weight:2.47389}}},\"rev_phone\":{metadata:{featureWeight:{weight:1.88433}}},\"mutual_follow_path\":{metadata:{featureWeight:{intervalWeights:[{left:47.0,weight:6.31809},{left:11.0,right:16.0,weight:4.52959},{left:31.0,right:47.0,weight:5.7101},{right:2.0,weight:0.383515},{left:24.0,right:31.0,weight:5.26515},{left:3.0,right:4.0,weight:2.91751},{left:2.0,right:3.0,weight:2.22851},{left:4.0,right:5.0,weight:3.28515},{left:8.0,right:11.0,weight:4.14731},{left:5.0,right:8.0,weight:3.73588},{left:16.0,right:24.0,weight:4.90908}]}}},\"fwd_phone\":{metadata:{featureWeight:{weight:2.07327}}},\"fwd_email_path\":{metadata:{featureWeight:{weight:0.961773}}},\"rev_phone_path\":{metadata:{featureWeight:{weight:0.354484}}},\"low_tweepcred_follow_path\":{metadata:{featureWeight:{intervalWeights:[{left:4.0,right:5.0,weight:0.177209},{left:7.0,right:8.0,weight:0.12378},{left:3.0,right:4.0,weight:0.197566},{left:5.0,right:6.0,weight:0.15867},{left:2.0,right:3.0,weight:0.196539},{right:2.0,weight:0.1805},{left:75.0,weight:-0.424598},{left:6.0,right:7.0,weight:0.143698},{left:10.0,right:13.0,weight:0.0458502},{left:8.0,right:10.0,weight:0.0919314},{left:13.0,right:75.0,weight:-0.111484}]}}},\"rev_email_path\":{metadata:{featureWeight:{weight:0.654451}}},\"rev_email\":{metadata:{featureWeight:{weight:2.33859}}},\"fwd_phone_path\":{metadata:{featureWeight:{weight:0.210418}}}]}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/resources/quality/stp_models/20141223/trainingConfig",
    "content": "{input:{context:\"discover.prod\",startDateTime:\"\",endDateTime:\"\",trainingFeatures:[\"STP_FEATURES\":[\"fwd_email\",\"mutual_follow_path\",\"fwd_email_path\",\"rev_phone_path\",\"low_tweepcred_follow_path\",\"rev_phone\",\"fwd_phone\",\"rev_email_path\",\"rev_email\",\"fwd_phone_path\"]],engagementActions:[\"click\",\"favorite\",\"open_link\",\"open\",\"send_tweet\",\"send_reply\",\"retweet\",\"reply\",\"profile_click\",\"follow\"],impressionActions:[\"discard\",\"results\",\"impression\"],dataFormat:1,dataPath:\"\",isLabeled:0},sample:{positiveSampleRatio:1.0,negativeSampleRatio:1.0,sampleType:1},split:{trainingDataSplitSize:0.5,testingDataSplitSize:0.5,splitType:2},transform:{},filter:{featureOptions:[]},join:{engagementRules:[\"discover\"],contentIdType:\"tweet\",groupBucketSize:3600000},discretize:{}}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"finatra-internal/decider/src/main/scala\",\n        \"finatra-internal/international/src/main/scala/com/twitter/finatra/international/modules\",\n        \"finatra-internal/mtls-http/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/http-core/src/main/java/com/twitter/finatra/http\",\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-server/src/main/scala\",\n        \"finatra/inject/inject-thrift-client\",\n        \"finatra/jackson/src/main/scala/com/twitter/finatra/jackson/modules\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/addressbook\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/adserver\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/cache\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/deepbirdv2\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/email_storage_service\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/geoduck\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/gizmoduck\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/graph_feature_service\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/phone_storage_service\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/socialgraph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/strato\",\n        \"follow-recommendations-service/server/src/main/resources\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/controllers\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/modules\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services/exceptions\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"geoduck/service/src/main/scala/com/twitter/geoduck/service/common/clientmodules\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/stringcenter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice\",\n        \"twitter-server/server/src/main/scala\",\n        \"util/util-app/src/main/scala\",\n        \"util/util-core:scala\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/FollowRecommendationsServiceThriftServer.scala",
    "content": "package com.twitter.follow_recommendations\n\nimport com.google.inject.Module\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finatra.decider.modules.DeciderModule\nimport com.twitter.finatra.http.HttpServer\nimport com.twitter.finatra.http.routing.HttpRouter\nimport com.twitter.finatra.international.modules.I18nFactoryModule\nimport com.twitter.finatra.international.modules.LanguagesModule\nimport com.twitter.finatra.jackson.modules.ScalaObjectMapperModule\nimport com.twitter.finatra.mtls.http.{Mtls => HttpMtls}\nimport com.twitter.finatra.mtls.thriftmux.Mtls\nimport com.twitter.finatra.thrift.ThriftServer\nimport com.twitter.finatra.thrift.filters._\nimport com.twitter.finagle.thrift.Protocols\nimport com.twitter.finatra.thrift.routing.ThriftRouter\nimport com.twitter.follow_recommendations.common.clients.addressbook.AddressbookModule\nimport com.twitter.follow_recommendations.common.clients.adserver.AdserverModule\nimport com.twitter.follow_recommendations.common.clients.cache.MemcacheModule\nimport com.twitter.follow_recommendations.common.clients.deepbirdv2.DeepBirdV2PredictionServiceClientModule\nimport com.twitter.follow_recommendations.common.clients.email_storage_service.EmailStorageServiceModule\nimport com.twitter.follow_recommendations.common.clients.geoduck.LocationServiceModule\nimport com.twitter.follow_recommendations.common.clients.gizmoduck.GizmoduckModule\nimport com.twitter.follow_recommendations.common.clients.graph_feature_service.GraphFeatureStoreModule\nimport com.twitter.follow_recommendations.common.clients.impression_store.ImpressionStoreModule\nimport com.twitter.follow_recommendations.common.clients.phone_storage_service.PhoneStorageServiceModule\nimport com.twitter.follow_recommendations.common.clients.socialgraph.SocialGraphModule\nimport com.twitter.follow_recommendations.common.clients.strato.StratoClientModule\nimport com.twitter.follow_recommendations.common.constants.ServiceConstants._\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.HydrationSourcesModule\nimport com.twitter.follow_recommendations.controllers.ThriftController\nimport com.twitter.follow_recommendations.modules._\nimport com.twitter.follow_recommendations.service.exceptions.UnknownLoggingExceptionMapper\nimport com.twitter.follow_recommendations.services.FollowRecommendationsServiceWarmupHandler\nimport com.twitter.follow_recommendations.thriftscala.FollowRecommendationsThriftService\nimport com.twitter.geoduck.service.common.clientmodules.ReverseGeocoderThriftClientModule\nimport com.twitter.inject.thrift.filters.DarkTrafficFilter\nimport com.twitter.inject.thrift.modules.ThriftClientIdModule\nimport com.twitter.product_mixer.core.controllers.ProductMixerController\nimport com.twitter.product_mixer.core.module.PipelineExecutionLoggerModule\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule\nimport com.twitter.product_mixer.core.module.stringcenter.ProductScopeStringCenterModule\nimport com.twitter.product_mixer.core.product.guice.ProductScopeModule\n\nobject FollowRecommendationsServiceThriftServerMain extends FollowRecommendationsServiceThriftServer\n\nclass FollowRecommendationsServiceThriftServer\n    extends ThriftServer\n    with Mtls\n    with HttpServer\n    with HttpMtls {\n  override val name: String = \"follow-recommendations-service-server\"\n\n  override val modules: Seq[Module] =\n    Seq(\n      ABDeciderModule,\n      AddressbookModule,\n      AdserverModule,\n      ConfigApiModule,\n      DeciderModule,\n      DeepBirdV2PredictionServiceClientModule,\n      DiffyModule,\n      EmailStorageServiceModule,\n      FeaturesSwitchesModule,\n      FlagsModule,\n      GizmoduckModule,\n      GraphFeatureStoreModule,\n      HydrationSourcesModule,\n      I18nFactoryModule,\n      ImpressionStoreModule,\n      LanguagesModule,\n      LocationServiceModule,\n      MemcacheModule,\n      PhoneStorageServiceModule,\n      PipelineExecutionLoggerModule,\n      ProductMixerFlagModule,\n      ProductRegistryModule,\n      new ProductScopeModule(),\n      new ProductScopeStringCenterModule(),\n      new ReverseGeocoderThriftClientModule,\n      ScalaObjectMapperModule,\n      ScorerModule,\n      ScribeModule,\n      SocialGraphModule,\n      StratoClientModule,\n      ThriftClientIdModule,\n      TimerModule,\n    )\n\n  def configureThrift(router: ThriftRouter): Unit = {\n    router\n      .filter[LoggingMDCFilter]\n      .filter[TraceIdMDCFilter]\n      .filter[ThriftMDCFilter]\n      .filter[StatsFilter]\n      .filter[AccessLoggingFilter]\n      .filter[ExceptionMappingFilter]\n      .exceptionMapper[UnknownLoggingExceptionMapper]\n      .filter[DarkTrafficFilter[FollowRecommendationsThriftService.ReqRepServicePerEndpoint]]\n      .add[ThriftController]\n  }\n\n  override def configureThriftServer(server: ThriftMux.Server): ThriftMux.Server = {\n    server.withProtocolFactory(\n      Protocols.binaryFactory(\n        stringLengthLimit = StringLengthLimit,\n        containerLengthLimit = ContainerLengthLimit))\n  }\n\n  override def configureHttp(router: HttpRouter): Unit = router.add(\n    ProductMixerController[FollowRecommendationsThriftService.MethodPerEndpoint](\n      this.injector,\n      FollowRecommendationsThriftService.ExecutePipeline))\n\n  override def warmup(): Unit = {\n    handle[FollowRecommendationsServiceWarmupHandler]()\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models/Action.scala",
    "content": "package com.twitter.follow_recommendations.assembler.models\n\nimport com.twitter.follow_recommendations.{thriftscala => t}\n\ncase class Action(text: String, actionURL: String) {\n  lazy val toThrift: t.Action = {\n    t.Action(text, actionURL)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"stringcenter/client\",\n    ],\n    exports = [\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models/Config.scala",
    "content": "package com.twitter.follow_recommendations.assembler.models\n\nimport com.twitter.stringcenter.client.core.ExternalString\n\ncase class HeaderConfig(title: TitleConfig)\ncase class TitleConfig(text: ExternalString)\ncase class FooterConfig(actionConfig: Option[ActionConfig])\ncase class ActionConfig(footerText: ExternalString, actionURL: String)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models/FeedbackAction.scala",
    "content": "package com.twitter.follow_recommendations.assembler.models\n\nimport com.twitter.follow_recommendations.{thriftscala => t}\n\ntrait FeedbackAction {\n  def toThrift: t.FeedbackAction\n}\n\ncase class DismissUserId() extends FeedbackAction {\n  override lazy val toThrift: t.FeedbackAction = {\n    t.FeedbackAction.DismissUserId(t.DismissUserId())\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models/Footer.scala",
    "content": "package com.twitter.follow_recommendations.assembler.models\n\nimport com.twitter.follow_recommendations.{thriftscala => t}\n\ncase class Footer(action: Option[Action]) {\n  lazy val toThrift: t.Footer = {\n    t.Footer(action.map(_.toThrift))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models/Header.scala",
    "content": "package com.twitter.follow_recommendations.assembler.models\n\nimport com.twitter.follow_recommendations.{thriftscala => t}\n\ncase class Header(title: Title) {\n  lazy val toThrift: t.Header = {\n    t.Header(title.toThrift)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models/Layout.scala",
    "content": "package com.twitter.follow_recommendations.assembler.models\n\nsealed trait Layout\n\ncase class UserListLayout(\n  header: Option[HeaderConfig],\n  userListOptions: UserListOptions,\n  socialProofs: Option[Seq[SocialProof]],\n  footer: Option[FooterConfig])\n    extends Layout\n\ncase class CarouselLayout(\n  header: Option[HeaderConfig],\n  carouselOptions: CarouselOptions,\n  socialProofs: Option[Seq[SocialProof]])\n    extends Layout\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models/RecommendationOptions.scala",
    "content": "package com.twitter.follow_recommendations.assembler.models\n\nsealed trait RecommendationOptions\n\ncase class UserListOptions(\n  userBioEnabled: Boolean,\n  userBioTruncated: Boolean,\n  userBioMaxLines: Option[Long],\n) extends RecommendationOptions\n\ncase class CarouselOptions() extends RecommendationOptions\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models/SocialProof.scala",
    "content": "package com.twitter.follow_recommendations.assembler.models\n\nimport com.twitter.stringcenter.client.core.ExternalString\n\nsealed trait SocialProof\n\ncase class GeoContextProof(popularInCountryText: ExternalString) extends SocialProof\ncase class FollowedByUsersProof(text1: ExternalString, text2: ExternalString, textN: ExternalString)\n    extends SocialProof\n\nsealed trait SocialText {\n  def text: String\n}\n\ncase class GeoSocialText(text: String) extends SocialText\ncase class FollowedByUsersText(text: String) extends SocialText\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models/Title.scala",
    "content": "package com.twitter.follow_recommendations.assembler.models\n\nimport com.twitter.follow_recommendations.{thriftscala => t}\n\ncase class Title(text: String) {\n  lazy val toThrift: t.Title = {\n    t.Title(text)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models/WTFPresentation.scala",
    "content": "package com.twitter.follow_recommendations.assembler.models\n\nimport com.twitter.follow_recommendations.{thriftscala => t}\n\ntrait WTFPresentation {\n  def toThrift: t.WTFPresentation\n}\n\ncase class UserList(\n  userBioEnabled: Boolean,\n  userBioTruncated: Boolean,\n  userBioMaxLines: Option[Long],\n  feedbackAction: Option[FeedbackAction])\n    extends WTFPresentation {\n  def toThrift: t.WTFPresentation = {\n    t.WTFPresentation.UserBioList(\n      t.UserList(userBioEnabled, userBioTruncated, userBioMaxLines, feedbackAction.map(_.toThrift)))\n  }\n}\n\nobject UserList {\n  def fromUserListOptions(\n    userListOptions: UserListOptions\n  ): UserList = {\n    UserList(\n      userListOptions.userBioEnabled,\n      userListOptions.userBioTruncated,\n      userListOptions.userBioMaxLines,\n      None)\n  }\n}\n\ncase class Carousel(\n  feedbackAction: Option[FeedbackAction])\n    extends WTFPresentation {\n  def toThrift: t.WTFPresentation = {\n    t.WTFPresentation.Carousel(t.Carousel(feedbackAction.map(_.toThrift)))\n  }\n}\n\nobject Carousel {\n  def fromCarouselOptions(\n    carouselOptions: CarouselOptions\n  ): Carousel = {\n    Carousel(None)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/blenders/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/blenders/PromotedAccountsBlender.scala",
    "content": "package com.twitter.follow_recommendations.blenders\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.follow_recommendations.common.models.AdMetadata\nimport com.twitter.follow_recommendations.common.models.Recommendation\nimport com.twitter.inject.Logging\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PromotedAccountsBlender @Inject() (statsReceiver: StatsReceiver)\n    extends Transform[Int, Recommendation]\n    with Logging {\n\n  import PromotedAccountsBlender._\n  val stats = statsReceiver.scope(Name)\n  val inputOrganicAccounts = stats.counter(InputOrganic)\n  val inputPromotedAccounts = stats.counter(InputPromoted)\n  val outputOrganicAccounts = stats.counter(OutputOrganic)\n  val outputPromotedAccounts = stats.counter(OutputPromoted)\n  val promotedAccountsStats = stats.scope(NumPromotedAccounts)\n\n  override def transform(\n    maxResults: Int,\n    items: Seq[Recommendation]\n  ): Stitch[Seq[Recommendation]] = {\n    val (promoted, organic) = items.partition(_.isPromotedAccount)\n    val promotedIds = promoted.map(_.id).toSet\n    val dedupedOrganic = organic.filterNot(u => promotedIds.contains(u.id))\n    val blended = blendPromotedAccount(dedupedOrganic, promoted, maxResults)\n    val (outputPromoted, outputOrganic) = blended.partition(_.isPromotedAccount)\n    inputOrganicAccounts.incr(dedupedOrganic.size)\n    inputPromotedAccounts.incr(promoted.size)\n    outputOrganicAccounts.incr(outputOrganic.size)\n    val size = outputPromoted.size\n    outputPromotedAccounts.incr(size)\n    if (size <= 5) {\n      promotedAccountsStats.counter(outputPromoted.size.toString).incr()\n    } else {\n      promotedAccountsStats.counter(MoreThan5Promoted).incr()\n    }\n    Stitch.value(blended)\n  }\n\n  /**\n   * Merge Promoted results and organic results. Promoted result dictates the position\n   * in the merge list.\n   *\n   * merge a list of positioned users, aka. promoted, and a list of organic\n   * users.  The positioned promoted users are pre-sorted with regards to their\n   * position ascendingly.  Only requirement about position is to be within the\n   * range, i.e, can not exceed the combined length if merge is successful, ok\n   * to be at the last position, but not beyond.\n   * For more detailed description of location position:\n   * http://confluence.local.twitter.com/display/ADS/Promoted+Tweets+in+Timeline+Design+Document\n   */\n  @VisibleForTesting\n  private[blenders] def mergePromotedAccounts(\n    organicUsers: Seq[Recommendation],\n    promotedUsers: Seq[Recommendation]\n  ): Seq[Recommendation] = {\n    def mergeAccountWithIndex(\n      organicUsers: Seq[Recommendation],\n      promotedUsers: Seq[Recommendation],\n      index: Int\n    ): Stream[Recommendation] = {\n      if (promotedUsers.isEmpty) organicUsers.toStream\n      else {\n        val promotedHead = promotedUsers.head\n        val promotedTail = promotedUsers.tail\n        promotedHead.adMetadata match {\n          case Some(AdMetadata(position, _)) =>\n            if (position < 0) mergeAccountWithIndex(organicUsers, promotedTail, index)\n            else if (position == index)\n              promotedHead #:: mergeAccountWithIndex(organicUsers, promotedTail, index)\n            else if (organicUsers.isEmpty) organicUsers.toStream\n            else {\n              val organicHead = organicUsers.head\n              val organicTail = organicUsers.tail\n              organicHead #:: mergeAccountWithIndex(organicTail, promotedUsers, index + 1)\n            }\n          case _ =>\n            logger.error(\"Unknown Candidate type in mergePromotedAccounts\")\n            Stream.empty\n        }\n      }\n    }\n\n    mergeAccountWithIndex(organicUsers, promotedUsers, 0)\n  }\n\n  private[this] def blendPromotedAccount(\n    organic: Seq[Recommendation],\n    promoted: Seq[Recommendation],\n    maxResults: Int\n  ): Seq[Recommendation] = {\n\n    val merged = mergePromotedAccounts(organic, promoted)\n    val mergedServed = merged.take(maxResults)\n    val promotedServed = promoted.intersect(mergedServed)\n\n    if (isBlendPromotedNeeded(\n        mergedServed.size - promotedServed.size,\n        promotedServed.size,\n        maxResults\n      )) {\n      mergedServed\n    } else {\n      organic.take(maxResults)\n    }\n  }\n\n  @VisibleForTesting\n  private[blenders] def isBlendPromotedNeeded(\n    organicSize: Int,\n    promotedSize: Int,\n    maxResults: Int\n  ): Boolean = {\n    (organicSize > 1) &&\n    (promotedSize > 0) &&\n    (promotedSize < organicSize) &&\n    (promotedSize <= 2) &&\n    (maxResults > 1)\n  }\n}\n\nobject PromotedAccountsBlender {\n  val Name = \"promoted_accounts_blender\"\n  val InputOrganic = \"input_organic_accounts\"\n  val InputPromoted = \"input_promoted_accounts\"\n  val OutputOrganic = \"output_organic_accounts\"\n  val OutputPromoted = \"output_promoted_accounts\"\n  val NumPromotedAccounts = \"num_promoted_accounts\"\n  val MoreThan5Promoted = \"more_than_5\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-decider\",\n        \"configapi/configapi-featureswitches:v2\",\n        \"featureswitches/featureswitches-core\",\n        \"featureswitches/featureswitches-core:v2\",\n        \"featureswitches/featureswitches-core/src/main/scala/com/twitter/featureswitches/v2/builder\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/crowd_search_accounts\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/real_graph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/recent_engagement\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/top_organic_follows_accounts\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/triangular_loops\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/ranking\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/deciders\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/params\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/content_recommender_flow\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/ConfigBuilder.scala",
    "content": "package com.twitter.follow_recommendations.configapi\n\nimport com.twitter.timelines.configapi.CompositeConfig\nimport com.twitter.timelines.configapi.Config\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ConfigBuilder @Inject() (\n  deciderConfigs: DeciderConfigs,\n  featureSwitchConfigs: FeatureSwitchConfigs) {\n  // The order of configs added to `CompositeConfig` is important. The config will be matched with\n  // the first possible rule. So, current setup will give priority to Deciders instead of FS\n  def build(): Config =\n    new CompositeConfig(Seq(deciderConfigs.config, featureSwitchConfigs.config))\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/DeciderConfigs.scala",
    "content": "package com.twitter.follow_recommendations.configapi\n\nimport com.twitter.decider.Recipient\nimport com.twitter.decider.SimpleRecipient\nimport com.twitter.follow_recommendations.configapi.deciders.DeciderKey\nimport com.twitter.follow_recommendations.configapi.deciders.DeciderParams\nimport com.twitter.follow_recommendations.products.home_timeline_tweet_recs.configapi.HomeTimelineTweetRecsParams\nimport com.twitter.servo.decider.DeciderGateBuilder\nimport com.twitter.timelines.configapi._\nimport com.twitter.timelines.configapi.decider.DeciderSwitchOverrideValue\nimport com.twitter.timelines.configapi.decider.GuestRecipient\nimport com.twitter.timelines.configapi.decider.RecipientBuilder\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass DeciderConfigs @Inject() (deciderGateBuilder: DeciderGateBuilder) {\n  val overrides: Seq[OptionalOverride[_]] = DeciderConfigs.ParamsToDeciderMap.map {\n    case (params, deciderKey) =>\n      params.optionalOverrideValue(\n        DeciderSwitchOverrideValue(\n          feature = deciderGateBuilder.keyToFeature(deciderKey),\n          enabledValue = true,\n          recipientBuilder = DeciderConfigs.UserOrGuestOrRequest\n        )\n      )\n  }.toSeq\n\n  val config: BaseConfig = BaseConfigBuilder(overrides).build(\"FollowRecommendationServiceDeciders\")\n}\n\nobject DeciderConfigs {\n  val ParamsToDeciderMap = Map(\n    DeciderParams.EnableRecommendations -> DeciderKey.EnableRecommendations,\n    DeciderParams.EnableScoreUserCandidates -> DeciderKey.EnableScoreUserCandidates,\n    HomeTimelineTweetRecsParams.EnableProduct -> DeciderKey.EnableHomeTimelineTweetRecsProduct,\n  )\n\n  object UserOrGuestOrRequest extends RecipientBuilder {\n\n    def apply(requestContext: BaseRequestContext): Option[Recipient] = requestContext match {\n      case c: WithUserId if c.userId.isDefined =>\n        c.userId.map(SimpleRecipient)\n      case c: WithGuestId if c.guestId.isDefined =>\n        c.guestId.map(GuestRecipient)\n      case c: WithGuestId =>\n        RecipientBuilder.Request(c)\n      case _ =>\n        throw new UndefinedUserIdNorGuestIDException(requestContext)\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/FeatureSwitchConfigs.scala",
    "content": "package com.twitter.follow_recommendations.configapi\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.candidate_sources.base.SocialProofEnforcedCandidateSourceFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.crowd_search_accounts.CrowdSearchAccountsFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopGeoQualityFollowSourceFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.top_organic_follows_accounts.TopOrganicFollowsAccountsFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopGeoSourceFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.ppmi_locale_follow.PPMILocaleFollowSourceFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.real_graph.RealGraphOonFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.recent_engagement.RepeatedProfileVisitsFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.sims.SimsSourceFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentEngagementSimilarUsersFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.SimsExpansionFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.socialgraph.RecentFollowingRecentFollowingExpansionSourceFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.OfflineStpSourceFsConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.OnlineSTPSourceFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.triangular_loops.TriangularLoopsFSConfig\nimport com.twitter.follow_recommendations.common.candidate_sources.user_user_graph.UserUserGraphFSConfig\nimport com.twitter.follow_recommendations.common.feature_hydration.sources.FeatureHydrationSourcesFSConfig\nimport com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker.WeightedCandidateSourceRankerFSConfig\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.follow_recommendations.flows.content_recommender_flow.ContentRecommenderFlowFSConfig\nimport com.twitter.follow_recommendations.common.predicates.gizmoduck.GizmoduckPredicateFSConfig\nimport com.twitter.follow_recommendations.common.predicates.hss.HssPredicateFSConfig\nimport com.twitter.follow_recommendations.common.predicates.sgs.SgsPredicateFSConfig\nimport com.twitter.follow_recommendations.flows.post_nux_ml.PostNuxMlFlowFSConfig\nimport com.twitter.logging.Logger\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FeatureSwitchConfigs @Inject() (\n  globalFeatureSwitchConfig: GlobalFeatureSwitchConfig,\n  featureHydrationSourcesFSConfig: FeatureHydrationSourcesFSConfig,\n  weightedCandidateSourceRankerFSConfig: WeightedCandidateSourceRankerFSConfig,\n  // Flow related config\n  contentRecommenderFlowFSConfig: ContentRecommenderFlowFSConfig,\n  postNuxMlFlowFSConfig: PostNuxMlFlowFSConfig,\n  // Candidate source related config\n  crowdSearchAccountsFSConfig: CrowdSearchAccountsFSConfig,\n  offlineStpSourceFsConfig: OfflineStpSourceFsConfig,\n  onlineSTPSourceFSConfig: OnlineSTPSourceFSConfig,\n  popGeoSourceFSConfig: PopGeoSourceFSConfig,\n  popGeoQualityFollowFSConfig: PopGeoQualityFollowSourceFSConfig,\n  realGraphOonFSConfig: RealGraphOonFSConfig,\n  repeatedProfileVisitsFSConfig: RepeatedProfileVisitsFSConfig,\n  recentEngagementSimilarUsersFSConfig: RecentEngagementSimilarUsersFSConfig,\n  recentFollowingRecentFollowingExpansionSourceFSConfig: RecentFollowingRecentFollowingExpansionSourceFSConfig,\n  simsExpansionFSConfig: SimsExpansionFSConfig,\n  simsSourceFSConfig: SimsSourceFSConfig,\n  socialProofEnforcedCandidateSourceFSConfig: SocialProofEnforcedCandidateSourceFSConfig,\n  triangularLoopsFSConfig: TriangularLoopsFSConfig,\n  userUserGraphFSConfig: UserUserGraphFSConfig,\n  // Predicate related configs\n  gizmoduckPredicateFSConfig: GizmoduckPredicateFSConfig,\n  hssPredicateFSConfig: HssPredicateFSConfig,\n  sgsPredicateFSConfig: SgsPredicateFSConfig,\n  ppmiLocaleSourceFSConfig: PPMILocaleFollowSourceFSConfig,\n  topOrganicFollowsAccountsFSConfig: TopOrganicFollowsAccountsFSConfig,\n  statsReceiver: StatsReceiver) {\n\n  val logger = Logger(classOf[FeatureSwitchConfigs])\n\n  val mergedFSConfig =\n    FeatureSwitchConfig.merge(\n      Seq(\n        globalFeatureSwitchConfig,\n        featureHydrationSourcesFSConfig,\n        weightedCandidateSourceRankerFSConfig,\n        // Flow related config\n        contentRecommenderFlowFSConfig,\n        postNuxMlFlowFSConfig,\n        // Candidate source related config\n        crowdSearchAccountsFSConfig,\n        offlineStpSourceFsConfig,\n        onlineSTPSourceFSConfig,\n        popGeoSourceFSConfig,\n        popGeoQualityFollowFSConfig,\n        realGraphOonFSConfig,\n        repeatedProfileVisitsFSConfig,\n        recentEngagementSimilarUsersFSConfig,\n        recentFollowingRecentFollowingExpansionSourceFSConfig,\n        simsExpansionFSConfig,\n        simsSourceFSConfig,\n        socialProofEnforcedCandidateSourceFSConfig,\n        triangularLoopsFSConfig,\n        userUserGraphFSConfig,\n        // Predicate related configs:\n        gizmoduckPredicateFSConfig,\n        hssPredicateFSConfig,\n        sgsPredicateFSConfig,\n        ppmiLocaleSourceFSConfig,\n        topOrganicFollowsAccountsFSConfig,\n      )\n    )\n\n  /**\n   * enum params have to be listed in this main file together as otherwise we'll have to pass in\n   * some signature like `Seq[FSEnumParams[_]]` which are generics of generics and won't compile.\n   * we only have enumFsParams from globalFeatureSwitchConfig at the moment\n   */\n  val enumOverrides = globalFeatureSwitchConfig.enumFsParams.flatMap { enumParam =>\n    FeatureSwitchOverrideUtil.getEnumFSOverrides(statsReceiver, logger, enumParam)\n  }\n\n  val gatedOverrides = mergedFSConfig.gatedOverridesMap.flatMap {\n    case (fsName, overrides) =>\n      FeatureSwitchOverrideUtil.gatedOverrides(fsName, overrides: _*)\n  }\n\n  val enumSeqOverrides = globalFeatureSwitchConfig.enumSeqFsParams.flatMap { enumSeqParam =>\n    FeatureSwitchOverrideUtil.getEnumSeqFSOverrides(statsReceiver, logger, enumSeqParam)\n  }\n\n  val overrides =\n    FeatureSwitchOverrideUtil\n      .getBooleanFSOverrides(mergedFSConfig.booleanFSParams: _*) ++\n      FeatureSwitchOverrideUtil\n        .getBoundedIntFSOverrides(mergedFSConfig.intFSParams: _*) ++\n      FeatureSwitchOverrideUtil\n        .getBoundedLongFSOverrides(mergedFSConfig.longFSParams: _*) ++\n      FeatureSwitchOverrideUtil\n        .getBoundedDoubleFSOverrides(mergedFSConfig.doubleFSParams: _*) ++\n      FeatureSwitchOverrideUtil\n        .getDurationFSOverrides(mergedFSConfig.durationFSParams: _*) ++\n      FeatureSwitchOverrideUtil\n        .getBoundedOptionalDoubleOverrides(mergedFSConfig.optionalDoubleFSParams: _*) ++\n      FeatureSwitchOverrideUtil.getStringSeqFSOverrides(mergedFSConfig.stringSeqFSParams: _*) ++\n      enumOverrides ++\n      gatedOverrides ++\n      enumSeqOverrides\n\n  val config = BaseConfigBuilder(overrides).build(\"FollowRecommendationServiceFeatureSwitches\")\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/GlobalFeatureSwitchConfig.scala",
    "content": "package com.twitter.follow_recommendations.configapi\n\nimport com.twitter.follow_recommendations.common.candidate_sources.crowd_search_accounts.CrowdSearchAccountsParams.AccountsFilteringAndRankingLogics\nimport com.twitter.follow_recommendations.common.candidate_sources.top_organic_follows_accounts.TopOrganicFollowsAccountsParams.{\n  AccountsFilteringAndRankingLogics => OrganicAccountsFilteringAndRankingLogics\n}\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentEngagementSimilarUsersParams\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.SimsExpansionSourceParams\nimport com.twitter.follow_recommendations.common.rankers.ml_ranker.ranking.MlRankerParams.CandidateScorerIdParam\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams.CandidateSourcesToFilter\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams.EnableCandidateParamHydrations\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams.EnableGFSSocialProofTransform\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams.EnableRecommendationFlowLogs\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams.EnableWhoToFollowProducts\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams.KeepSocialUserCandidate\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams.KeepUserCandidate\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.Param\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GlobalFeatureSwitchConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] = {\n    Seq(\n      EnableCandidateParamHydrations,\n      KeepUserCandidate,\n      KeepSocialUserCandidate,\n      EnableGFSSocialProofTransform,\n      EnableWhoToFollowProducts,\n      EnableRecommendationFlowLogs\n    )\n  }\n\n  val enumFsParams =\n    Seq(\n      CandidateScorerIdParam,\n      SimsExpansionSourceParams.Aggregator,\n      RecentEngagementSimilarUsersParams.Aggregator,\n      CandidateSourcesToFilter,\n    )\n\n  val enumSeqFsParams =\n    Seq(\n      AccountsFilteringAndRankingLogics,\n      OrganicAccountsFilteringAndRankingLogics\n    )\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/ParamsFactory.scala",
    "content": "package com.twitter.follow_recommendations.configapi\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.servo.util.MemoizingStatsReceiver\nimport com.twitter.timelines.configapi.Config\nimport com.twitter.timelines.configapi.FeatureValue\nimport com.twitter.timelines.configapi.Params\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ParamsFactory @Inject() (\n  config: Config,\n  requestContextFactory: RequestContextFactory,\n  statsReceiver: StatsReceiver) {\n\n  private val stats = new MemoizingStatsReceiver(statsReceiver.scope(\"configapi\"))\n  def apply(followRecommendationServiceRequestContext: RequestContext): Params =\n    config(followRecommendationServiceRequestContext, stats)\n\n  def apply(\n    clientContext: ClientContext,\n    displayLocation: DisplayLocation,\n    featureOverrides: Map[String, FeatureValue]\n  ): Params =\n    apply(requestContextFactory(clientContext, displayLocation, featureOverrides))\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/RequestContext.scala",
    "content": "package com.twitter.follow_recommendations.configapi\n\nimport com.twitter.timelines.configapi.BaseRequestContext\nimport com.twitter.timelines.configapi.FeatureContext\nimport com.twitter.timelines.configapi.NullFeatureContext\nimport com.twitter.timelines.configapi.GuestId\nimport com.twitter.timelines.configapi.UserId\nimport com.twitter.timelines.configapi.WithFeatureContext\nimport com.twitter.timelines.configapi.WithGuestId\nimport com.twitter.timelines.configapi.WithUserId\n\ncase class RequestContext(\n  userId: Option[UserId],\n  guestId: Option[GuestId],\n  featureContext: FeatureContext = NullFeatureContext)\n    extends BaseRequestContext\n    with WithUserId\n    with WithGuestId\n    with WithFeatureContext\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/RequestContextFactory.scala",
    "content": "package com.twitter.follow_recommendations.configapi\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.google.inject.Inject\nimport com.twitter.decider.Decider\nimport com.twitter.featureswitches.v2.FeatureSwitches\nimport com.twitter.featureswitches.{Recipient => FeatureSwitchRecipient}\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.timelines.configapi.FeatureContext\nimport com.twitter.timelines.configapi.FeatureValue\nimport com.twitter.timelines.configapi.ForcedFeatureContext\nimport com.twitter.timelines.configapi.OrElseFeatureContext\nimport com.twitter.timelines.configapi.featureswitches.v2.FeatureSwitchResultsFeatureContext\nimport javax.inject.Singleton\n\n/*\n * Request Context Factory is used to build RequestContext objects which are used\n * by the config api to determine the param overrides to apply to the request.\n * The param overrides are determined per request by configs which specify which\n * FS/Deciders/AB translate to what param overrides.\n */\n@Singleton\nclass RequestContextFactory @Inject() (featureSwitches: FeatureSwitches, decider: Decider) {\n  def apply(\n    clientContext: ClientContext,\n    displayLocation: DisplayLocation,\n    featureOverrides: Map[String, FeatureValue]\n  ): RequestContext = {\n    val featureContext = getFeatureContext(clientContext, displayLocation, featureOverrides)\n    RequestContext(clientContext.userId, clientContext.guestId, featureContext)\n  }\n\n  private[configapi] def getFeatureContext(\n    clientContext: ClientContext,\n    displayLocation: DisplayLocation,\n    featureOverrides: Map[String, FeatureValue]\n  ): FeatureContext = {\n    val recipient =\n      getFeatureSwitchRecipient(clientContext)\n        .withCustomFields(\"display_location\" -> displayLocation.toFsName)\n\n    // userAgeOpt is going to be set to None for logged out users and defaulted to Some(Int.MaxValue) for non-snowflake users\n    val userAgeOpt = clientContext.userId.map { userId =>\n      SnowflakeId.timeFromIdOpt(userId).map(_.untilNow.inDays).getOrElse(Int.MaxValue)\n    }\n    val recipientWithAccountAge =\n      userAgeOpt\n        .map(age => recipient.withCustomFields(\"account_age_in_days\" -> age)).getOrElse(recipient)\n\n    val results = featureSwitches.matchRecipient(recipientWithAccountAge)\n    OrElseFeatureContext(\n      ForcedFeatureContext(featureOverrides),\n      new FeatureSwitchResultsFeatureContext(results))\n  }\n\n  @VisibleForTesting\n  private[configapi] def getFeatureSwitchRecipient(\n    clientContext: ClientContext\n  ): FeatureSwitchRecipient = {\n    FeatureSwitchRecipient(\n      userId = clientContext.userId,\n      userRoles = clientContext.userRoles,\n      deviceId = clientContext.deviceId,\n      guestId = clientContext.guestId,\n      languageCode = clientContext.languageCode,\n      countryCode = clientContext.countryCode,\n      isVerified = None,\n      clientApplicationId = clientContext.appId,\n      isTwoffice = clientContext.isTwoffice\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/candidates/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-decider\",\n        \"configapi/configapi-featureswitches:v2\",\n        \"featureswitches/featureswitches-core\",\n        \"featureswitches/featureswitches-core:v2\",\n        \"featureswitches/featureswitches-core/src/main/scala/com/twitter/featureswitches/v2/builder\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/deciders\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/params\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/candidates/CandidateUserContext.scala",
    "content": "package com.twitter.follow_recommendations.configapi.candidates\n\nimport com.twitter.timelines.configapi.BaseRequestContext\nimport com.twitter.timelines.configapi.FeatureContext\nimport com.twitter.timelines.configapi.NullFeatureContext\nimport com.twitter.timelines.configapi.WithFeatureContext\nimport com.twitter.timelines.configapi.WithUserId\n\n/**\n * represent the context for a recommendation candidate (producer side)\n * @param userId id of the recommended user\n * @param featureContext feature context\n */\ncase class CandidateUserContext(\n  override val userId: Option[Long],\n  featureContext: FeatureContext = NullFeatureContext)\n    extends BaseRequestContext\n    with WithUserId\n    with WithFeatureContext\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/candidates/CandidateUserContextFactory.scala",
    "content": "package com.twitter.follow_recommendations.configapi.candidates\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.google.inject.Inject\nimport com.twitter.decider.Decider\nimport com.twitter.featureswitches.v2.FeatureSwitches\nimport com.twitter.featureswitches.{Recipient => FeatureSwitchRecipient}\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants.PRODUCER_SIDE_FEATURE_SWITCHES\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.timelines.configapi.FeatureContext\nimport com.twitter.timelines.configapi.featureswitches.v2.FeatureSwitchResultsFeatureContext\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass CandidateUserContextFactory @Inject() (\n  @Named(PRODUCER_SIDE_FEATURE_SWITCHES) featureSwitches: FeatureSwitches,\n  decider: Decider) {\n  def apply(\n    candidateUser: CandidateUser,\n    displayLocation: DisplayLocation\n  ): CandidateUserContext = {\n    val featureContext = getFeatureContext(candidateUser, displayLocation)\n\n    CandidateUserContext(Some(candidateUser.id), featureContext)\n  }\n\n  private[configapi] def getFeatureContext(\n    candidateUser: CandidateUser,\n    displayLocation: DisplayLocation\n  ): FeatureContext = {\n\n    val recipient = getFeatureSwitchRecipient(candidateUser).withCustomFields(\n      \"display_location\" -> displayLocation.toFsName)\n    new FeatureSwitchResultsFeatureContext(featureSwitches.matchRecipient(recipient))\n  }\n\n  @VisibleForTesting\n  private[configapi] def getFeatureSwitchRecipient(\n    candidateUser: CandidateUser\n  ): FeatureSwitchRecipient = {\n    FeatureSwitchRecipient(\n      userId = Some(candidateUser.id),\n      userRoles = None,\n      deviceId = None,\n      guestId = None,\n      languageCode = None,\n      countryCode = None,\n      isVerified = None,\n      clientApplicationId = None,\n      isTwoffice = None\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/candidates/CandidateUserParamsFactory.scala",
    "content": "package com.twitter.follow_recommendations.configapi.candidates\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams\nimport com.twitter.servo.util.MemoizingStatsReceiver\nimport com.twitter.timelines.configapi.Config\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Params\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * CandidateParamsFactory is primarily used for \"producer side\" experiments, don't use it on consumer side experiments\n */\n@Singleton\nclass CandidateUserParamsFactory[T <: HasParams with HasDisplayLocation] @Inject() (\n  config: Config,\n  candidateContextFactory: CandidateUserContextFactory,\n  statsReceiver: StatsReceiver) {\n  private val stats = new MemoizingStatsReceiver(statsReceiver.scope(\"configapi_candidate_params\"))\n  def apply(candidateContext: CandidateUser, request: T): CandidateUser = {\n    if (candidateContext.params == Params.Invalid) {\n      if (request.params(GlobalParams.EnableCandidateParamHydrations)) {\n        candidateContext.copy(params =\n          config(candidateContextFactory(candidateContext, request.displayLocation), stats))\n      } else {\n        candidateContext.copy(params = Params.Empty)\n      }\n    } else {\n      candidateContext\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/candidates/HydrateCandidateParamsTransform.scala",
    "content": "package com.twitter.follow_recommendations.configapi.candidates\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.logging.Logging\n\n@Singleton\nclass HydrateCandidateParamsTransform[Target <: HasParams with HasDisplayLocation] @Inject() (\n  candidateParamsFactory: CandidateUserParamsFactory[Target])\n    extends Transform[Target, CandidateUser]\n    with Logging {\n\n  def transform(target: Target, candidates: Seq[CandidateUser]): Stitch[Seq[CandidateUser]] = {\n    Stitch.value(candidates.map(candidateParamsFactory.apply(_, target)))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common/FeatureSwitchConfig.scala",
    "content": "package com.twitter.follow_recommendations.configapi.common\n\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil.DefinedFeatureName\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil.ValueFeatureName\nimport com.twitter.timelines.configapi.BoundedParam\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.OptionalOverride\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\ntrait FeatureSwitchConfig {\n  def booleanFSParams: Seq[Param[Boolean] with FSName] = Nil\n\n  def intFSParams: Seq[FSBoundedParam[Int]] = Nil\n\n  def longFSParams: Seq[FSBoundedParam[Long]] = Nil\n\n  def doubleFSParams: Seq[FSBoundedParam[Double]] = Nil\n\n  def durationFSParams: Seq[FSBoundedParam[Duration] with HasDurationConversion] = Nil\n\n  def optionalDoubleFSParams: Seq[\n    (BoundedParam[Option[Double]], DefinedFeatureName, ValueFeatureName)\n  ] = Nil\n\n  def stringSeqFSParams: Seq[Param[Seq[String]] with FSName] = Nil\n\n  /**\n   * Apply overrides in list when the given FS Key is enabled.\n   * This override type does NOT work with experiments. Params here will be evaluated for every\n   * request IMMEDIATELY, not upon param.apply. If you would like to use an experiment pls use\n   * the primitive type or ENUM overrides.\n   */\n  def gatedOverridesMap: Map[String, Seq[OptionalOverride[_]]] = Map.empty\n}\n\nobject FeatureSwitchConfig {\n  def merge(configs: Seq[FeatureSwitchConfig]): FeatureSwitchConfig = new FeatureSwitchConfig {\n    override def booleanFSParams: Seq[Param[Boolean] with FSName] =\n      configs.flatMap(_.booleanFSParams)\n    override def intFSParams: Seq[FSBoundedParam[Int]] =\n      configs.flatMap(_.intFSParams)\n    override def longFSParams: Seq[FSBoundedParam[Long]] =\n      configs.flatMap(_.longFSParams)\n    override def durationFSParams: Seq[FSBoundedParam[Duration] with HasDurationConversion] =\n      configs.flatMap(_.durationFSParams)\n    override def gatedOverridesMap: Map[String, Seq[OptionalOverride[_]]] =\n      configs.flatMap(_.gatedOverridesMap).toMap\n    override def doubleFSParams: Seq[FSBoundedParam[Double]] =\n      configs.flatMap(_.doubleFSParams)\n    override def optionalDoubleFSParams: Seq[\n      (BoundedParam[Option[Double]], DefinedFeatureName, ValueFeatureName)\n    ] =\n      configs.flatMap(_.optionalDoubleFSParams)\n    override def stringSeqFSParams: Seq[Param[Seq[String]] with FSName] =\n      configs.flatMap(_.stringSeqFSParams)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/deciders/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-decider\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/deciders/DeciderKey.scala",
    "content": "package com.twitter.follow_recommendations.configapi.deciders\n\nimport com.twitter.servo.decider.DeciderKeyEnum\n\nobject DeciderKey extends DeciderKeyEnum {\n  val EnableDiffyModuleDarkReading = Value(\"enable_diffy_module_dark_reading\")\n  val EnableRecommendations = Value(\"enable_recommendations\")\n  val EnableScoreUserCandidates = Value(\"enable_score_user_candidates\")\n  val EnableProfileSidebarProduct = Value(\"enable_profile_sidebar_product\")\n  val EnableMagicRecsProduct = Value(\"enable_magic_recs_product\")\n  val EnableRuxLandingPageProduct = Value(\"enable_rux_landing_page_product\")\n  val EnableRuxPymkProduct = Value(\"enable_rux_pymk_product\")\n  val EnableProfileBonusFollowProduct = Value(\"enable_profile_bonus_follow_product\")\n  val EnableElectionExploreWtfProduct = Value(\"enable_election_explore_wtf_product\")\n  val EnableClusterFollowProduct = Value(\"enable_cluster_follow_product\")\n  val EnableHomeTimelineProduct = Value(\"enable_home_timeline_product\")\n  val EnableHtlBonusFollowProduct = Value(\"enable_htl_bonus_follow_product\")\n  val EnableExploreTabProduct = Value(\"enable_explore_tab_product\")\n  val EnableSidebarProduct = Value(\"enable_sidebar_product\")\n  val EnableNuxPymkProduct = Value(\"enable_nux_pymk_product\")\n  val EnableNuxInterestsProduct = Value(\"enable_nux_interests_product\")\n  val EnableNuxTopicBonusFollowProduct = Value(\"enable_nux_topic_bonus_follow_product\")\n  val EnableCampaignFormProduct = Value(\"enable_campaign_form_product\")\n  val EnableReactiveFollowProduct = Value(\"enable_reactive_follow_product\")\n  val EnableIndiaCovid19CuratedAccountsWtfProduct = Value(\n    \"enable_india_covid19_curated_accounts_wtf_product\")\n  val EnableAbUploadProduct = Value(\"enable_ab_upload_product\")\n  val EnablePeolePlusPlusProduct = Value(\"enable_people_plus_plus_product\")\n  val EnableTweetNotificationRecsProduct = Value(\"enable_tweet_notification_recs_product\")\n  val EnableProfileDeviceFollow = Value(\"enable_profile_device_follow_product\")\n  val EnableRecosBackfillProduct = Value(\"enable_recos_backfill_product\")\n  val EnablePostNuxFollowTaskProduct = Value(\"enable_post_nux_follow_task_product\")\n  val EnableCuratedSpaceHostsProduct = Value(\"enable_curated_space_hosts_product\")\n  val EnableNuxGeoCategoryProduct = Value(\"enable_nux_geo_category_product\")\n  val EnableNuxInterestsCategoryProduct = Value(\"enable_nux_interests_category_product\")\n  val EnableNuxPymkCategoryProduct = Value(\"enable_nux_pymk_category_product\")\n  val EnableHomeTimelineTweetRecsProduct = Value(\"enable_home_timeline_tweet_recs_product\")\n  val EnableHtlBulkFriendFollowsProduct = Value(\"enable_htl_bulk_friend_follows_product\")\n  val EnableNuxAutoFollowProduct = Value(\"enable_nux_auto_follow_product\")\n  val EnableSearchBonusFollowProduct = Value(\"enable_search_bonus_follow_product\")\n  val EnableFetchUserInRequestBuilder = Value(\"enable_fetch_user_in_request_builder\")\n  val EnableProductMixerMagicRecsProduct = Value(\"enable_product_mixer_magic_recs_product\")\n  val EnableHomeTimelineReverseChronProduct = Value(\"enable_home_timeline_reverse_chron_product\")\n  val EnableProductMixerPipelineMagicRecsDarkRead = Value(\n    \"enable_product_mixer_pipeline_magic_recs_dark_read\")\n  val EnableExperimentalCaching = Value(\"enable_experimental_caching\")\n  val EnableDistributedCaching = Value(\"enable_distributed_caching\")\n  val EnableGizmoduckCaching = Value(\"enable_gizmoduck_caching\")\n  val EnableTrafficDarkReading = Value(\"enable_traffic_dark_reading\")\n  val EnableGraphFeatureServiceRequests = Value(\"enable_graph_feature_service_requests\")\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/deciders/DeciderParams.scala",
    "content": "package com.twitter.follow_recommendations.configapi.deciders\n\nimport com.twitter.timelines.configapi.Param\n\nobject DeciderParams {\n  object EnableRecommendations extends Param[Boolean](false)\n  object EnableScoreUserCandidates extends Param[Boolean](false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/params/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-decider\",\n        \"configapi/configapi-featureswitches:v2\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/deciders\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/params/GlobalParams.scala",
    "content": "package com.twitter.follow_recommendations.configapi.params\n\nimport com.twitter.follow_recommendations.models.CandidateSourceType\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSParam\n\n/**\n * When adding Producer side experiments, make sure to register the FS Key in [[ProducerFeatureFilter]]\n * in [[FeatureSwitchesModule]], otherwise, the FS will not work.\n */\nobject GlobalParams {\n\n  object EnableCandidateParamHydrations\n      extends FSParam[Boolean](\"frs_receiver_enable_candidate_params\", false)\n\n  object KeepUserCandidate\n      extends FSParam[Boolean](\"frs_receiver_holdback_keep_user_candidate\", true)\n\n  object KeepSocialUserCandidate\n      extends FSParam[Boolean](\"frs_receiver_holdback_keep_social_user_candidate\", true)\n\n  case object EnableGFSSocialProofTransform\n      extends FSParam(\"social_proof_transform_use_graph_feature_service\", true)\n\n  case object EnableWhoToFollowProducts extends FSParam(\"who_to_follow_product_enabled\", true)\n\n  case object CandidateSourcesToFilter\n      extends FSEnumParam[CandidateSourceType.type](\n        \"candidate_sources_type_filter_id\",\n        CandidateSourceType.None,\n        CandidateSourceType)\n\n  object EnableRecommendationFlowLogs\n      extends FSParam[Boolean](\"frs_recommendation_flow_logs_enabled\", false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/controllers/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"decider/src/main/scala\",\n        \"finagle/finagle-core/src/main\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift:controller\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/exceptions\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/filters\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/modules\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/response\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/routing\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/debug_query\",\n        \"scrooge/scrooge-core/src/main/scala\",\n        \"util/util-core:scala\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/controllers/CandidateUserDebugParamsBuilder.scala",
    "content": "package com.twitter.follow_recommendations.controllers\n\nimport com.twitter.follow_recommendations.common.models._\nimport com.twitter.follow_recommendations.configapi.ParamsFactory\nimport com.twitter.follow_recommendations.models.CandidateUserDebugParams\nimport com.twitter.follow_recommendations.models.FeatureValue\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CandidateUserDebugParamsBuilder @Inject() (paramsFactory: ParamsFactory) {\n  def fromThrift(req: t.ScoringUserRequest): CandidateUserDebugParams = {\n    val clientContext = ClientContextConverter.fromThrift(req.clientContext)\n    val displayLocation = DisplayLocation.fromThrift(req.displayLocation)\n\n    CandidateUserDebugParams(req.candidates.map { candidate =>\n      candidate.userId -> paramsFactory(\n        clientContext,\n        displayLocation,\n        candidate.featureOverrides\n          .map(_.mapValues(FeatureValue.fromThrift).toMap).getOrElse(Map.empty))\n    }.toMap)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/controllers/RecommendationRequestBuilder.scala",
    "content": "package com.twitter.follow_recommendations.controllers\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.ClientContextConverter\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.models.DebugParams\nimport com.twitter.follow_recommendations.models.DisplayContext\nimport com.twitter.follow_recommendations.models.RecommendationRequest\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport com.twitter.gizmoduck.thriftscala.UserType\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RecommendationRequestBuilder @Inject() (\n  requestBuilderUserFetcher: RequestBuilderUserFetcher,\n  statsReceiver: StatsReceiver) {\n  private val scopedStats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val isSoftUserCounter = scopedStats.counter(\"is_soft_user\")\n\n  def fromThrift(tRequest: t.RecommendationRequest): Stitch[RecommendationRequest] = {\n    requestBuilderUserFetcher.fetchUser(tRequest.clientContext.userId).map { userOpt =>\n      val isSoftUser = userOpt.exists(_.userType == UserType.Soft)\n      if (isSoftUser) isSoftUserCounter.incr()\n      RecommendationRequest(\n        clientContext = ClientContextConverter.fromThrift(tRequest.clientContext),\n        displayLocation = DisplayLocation.fromThrift(tRequest.displayLocation),\n        displayContext = tRequest.displayContext.map(DisplayContext.fromThrift),\n        maxResults = tRequest.maxResults,\n        cursor = tRequest.cursor,\n        excludedIds = tRequest.excludedIds,\n        fetchPromotedContent = tRequest.fetchPromotedContent,\n        debugParams = tRequest.debugParams.map(DebugParams.fromThrift),\n        userLocationState = tRequest.userLocationState,\n        isSoftUser = isSoftUser\n      )\n    }\n\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/controllers/RequestBuilderUserFetcher.scala",
    "content": "package com.twitter.follow_recommendations.controllers\n\nimport com.twitter.decider.Decider\nimport com.twitter.decider.SimpleRecipient\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.follow_recommendations.configapi.deciders.DeciderKey\nimport com.twitter.gizmoduck.thriftscala.LookupContext\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.gizmoduck.Gizmoduck\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RequestBuilderUserFetcher @Inject() (\n  gizmoduck: Gizmoduck,\n  statsReceiver: StatsReceiver,\n  decider: Decider) {\n  private val scopedStats = statsReceiver.scope(this.getClass.getSimpleName)\n\n  def fetchUser(userIdOpt: Option[Long]): Stitch[Option[User]] = {\n    userIdOpt match {\n      case Some(userId) if enableDecider(userId) =>\n        val stitch = gizmoduck\n          .getUserById(\n            userId = userId,\n            context = LookupContext(\n              forUserId = Some(userId),\n              includeProtected = true,\n              includeSoftUsers = true\n            )\n          ).map(user => Some(user))\n        StatsUtil\n          .profileStitch(stitch, scopedStats)\n          .handle {\n            case _: Throwable => None\n          }\n      case _ => Stitch.None\n    }\n  }\n\n  private def enableDecider(userId: Long): Boolean = {\n    decider.isAvailable(\n      DeciderKey.EnableFetchUserInRequestBuilder.toString,\n      Some(SimpleRecipient(userId)))\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/controllers/ScoringUserRequestBuilder.scala",
    "content": "package com.twitter.follow_recommendations.controllers\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.ClientContextConverter\nimport com.twitter.follow_recommendations.common.models.DebugOptions\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.models.DebugParams\nimport com.twitter.follow_recommendations.models.ScoringUserRequest\nimport com.twitter.timelines.configapi.Params\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport com.twitter.gizmoduck.thriftscala.UserType\nimport com.twitter.stitch.Stitch\n\n@Singleton\nclass ScoringUserRequestBuilder @Inject() (\n  requestBuilderUserFetcher: RequestBuilderUserFetcher,\n  candidateUserDebugParamsBuilder: CandidateUserDebugParamsBuilder,\n  statsReceiver: StatsReceiver) {\n  private val scopedStats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val isSoftUserCounter = scopedStats.counter(\"is_soft_user\")\n\n  def fromThrift(req: t.ScoringUserRequest): Stitch[ScoringUserRequest] = {\n    requestBuilderUserFetcher.fetchUser(req.clientContext.userId).map { userOpt =>\n      val isSoftUser = userOpt.exists(_.userType == UserType.Soft)\n      if (isSoftUser) isSoftUserCounter.incr()\n\n      val candidateUsersParamsMap = candidateUserDebugParamsBuilder.fromThrift(req)\n      val candidates = req.candidates.map { candidate =>\n        CandidateUser\n          .fromUserRecommendation(candidate).copy(params =\n            candidateUsersParamsMap.paramsMap.getOrElse(candidate.userId, Params.Invalid))\n      }\n\n      ScoringUserRequest(\n        clientContext = ClientContextConverter.fromThrift(req.clientContext),\n        displayLocation = DisplayLocation.fromThrift(req.displayLocation),\n        params = Params.Empty,\n        debugOptions = req.debugParams.map(DebugOptions.fromDebugParamsThrift),\n        recentFollowedUserIds = None,\n        recentFollowedByUserIds = None,\n        wtfImpressions = None,\n        similarToUserIds = Nil,\n        candidates = candidates,\n        debugParams = req.debugParams.map(DebugParams.fromThrift),\n        isSoftUser = isSoftUser\n      )\n    }\n  }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/controllers/ThriftController.scala",
    "content": "package com.twitter.follow_recommendations.controllers\n\nimport com.twitter.finatra.thrift.Controller\nimport com.twitter.follow_recommendations.configapi.ParamsFactory\nimport com.twitter.follow_recommendations.services.ProductPipelineSelector\nimport com.twitter.follow_recommendations.services.UserScoringService\nimport com.twitter.follow_recommendations.thriftscala.FollowRecommendationsThriftService\nimport com.twitter.follow_recommendations.thriftscala.FollowRecommendationsThriftService._\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\n\nclass ThriftController @Inject() (\n  userScoringService: UserScoringService,\n  recommendationRequestBuilder: RecommendationRequestBuilder,\n  scoringUserRequestBuilder: ScoringUserRequestBuilder,\n  productPipelineSelector: ProductPipelineSelector,\n  paramsFactory: ParamsFactory)\n    extends Controller(FollowRecommendationsThriftService) {\n\n  handle(GetRecommendations) { args: GetRecommendations.Args =>\n    val stitch = recommendationRequestBuilder.fromThrift(args.request).flatMap { request =>\n      val params = paramsFactory(\n        request.clientContext,\n        request.displayLocation,\n        request.debugParams.flatMap(_.featureOverrides).getOrElse(Map.empty))\n      productPipelineSelector.selectPipeline(request, params).map(_.toThrift)\n    }\n    Stitch.run(stitch)\n  }\n\n  handle(ScoreUserCandidates) { args: ScoreUserCandidates.Args =>\n    val stitch = scoringUserRequestBuilder.fromThrift(args.request).flatMap { request =>\n      val params = paramsFactory(\n        request.clientContext,\n        request.displayLocation,\n        request.debugParams.flatMap(_.featureOverrides).getOrElse(Map.empty))\n      userScoringService.get(request.copy(params = params)).map(_.toThrift)\n    }\n    Stitch.run(stitch)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/ads/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/promoted_accounts\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/tracking_token\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/ads/PromotedAccountsFlow.scala",
    "content": "package com.twitter.follow_recommendations.flows.ads\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.EnrichedCandidateSource\nimport com.twitter.follow_recommendations.common.base.IdentityRanker\nimport com.twitter.follow_recommendations.common.base.IdentityTransform\nimport com.twitter.follow_recommendations.common.base.ParamPredicate\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.Ranker\nimport com.twitter.follow_recommendations.common.base.RecommendationFlow\nimport com.twitter.follow_recommendations.common.base.RecommendationResultsConfig\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.follow_recommendations.common.base.TruePredicate\nimport com.twitter.follow_recommendations.common.candidate_sources.promoted_accounts.PromotedAccountsCandidateSource\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.predicates.ExcludedUserIdPredicate\nimport com.twitter.follow_recommendations.common.transforms.tracking_token.TrackingTokenTransform\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.util.Duration\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PromotedAccountsFlow @Inject() (\n  promotedAccountsCandidateSource: PromotedAccountsCandidateSource,\n  trackingTokenTransform: TrackingTokenTransform,\n  baseStatsReceiver: StatsReceiver,\n  @Flag(\"fetch_prod_promoted_accounts\") fetchProductionPromotedAccounts: Boolean)\n    extends RecommendationFlow[PromotedAccountsFlowRequest, CandidateUser] {\n\n  protected override def targetEligibility: Predicate[PromotedAccountsFlowRequest] =\n    new ParamPredicate[PromotedAccountsFlowRequest](\n      PromotedAccountsFlowParams.TargetEligibility\n    )\n\n  protected override def candidateSources(\n    target: PromotedAccountsFlowRequest\n  ): Seq[CandidateSource[PromotedAccountsFlowRequest, CandidateUser]] = {\n    import EnrichedCandidateSource._\n    val candidateSourceStats = statsReceiver.scope(\"candidate_sources\")\n    val budget: Duration = target.params(PromotedAccountsFlowParams.FetchCandidateSourceBudget)\n    val candidateSources = Seq(\n      promotedAccountsCandidateSource\n        .mapKeys[PromotedAccountsFlowRequest](r =>\n          Seq(r.toAdsRequest(fetchProductionPromotedAccounts)))\n        .mapValue(PromotedAccountsUtil.toCandidateUser)\n    ).map { candidateSource =>\n      candidateSource\n        .failOpenWithin(budget, candidateSourceStats).observe(candidateSourceStats)\n    }\n    candidateSources\n  }\n\n  protected override def preRankerCandidateFilter: Predicate[\n    (PromotedAccountsFlowRequest, CandidateUser)\n  ] = {\n    val preRankerFilterStats = statsReceiver.scope(\"pre_ranker\")\n    ExcludedUserIdPredicate.observe(preRankerFilterStats.scope(\"exclude_user_id_predicate\"))\n  }\n\n  /**\n   * rank the candidates\n   */\n  protected override def selectRanker(\n    target: PromotedAccountsFlowRequest\n  ): Ranker[PromotedAccountsFlowRequest, CandidateUser] = {\n    new IdentityRanker[PromotedAccountsFlowRequest, CandidateUser]\n  }\n\n  /**\n   * transform the candidates after ranking (e.g. dedupping, grouping and etc)\n   */\n  protected override def postRankerTransform: Transform[\n    PromotedAccountsFlowRequest,\n    CandidateUser\n  ] = {\n    new IdentityTransform[PromotedAccountsFlowRequest, CandidateUser]\n  }\n\n  /**\n   *  filter invalid candidates before returning the results.\n   *\n   *  Some heavy filters e.g. SGS filter could be applied in this step\n   */\n  protected override def validateCandidates: Predicate[\n    (PromotedAccountsFlowRequest, CandidateUser)\n  ] = {\n    new TruePredicate[(PromotedAccountsFlowRequest, CandidateUser)]\n  }\n\n  /**\n   * transform the candidates into results and return\n   */\n  protected override def transformResults: Transform[PromotedAccountsFlowRequest, CandidateUser] = {\n    trackingTokenTransform\n  }\n\n  /**\n   *  configuration for recommendation results\n   */\n  protected override def resultsConfig(\n    target: PromotedAccountsFlowRequest\n  ): RecommendationResultsConfig = {\n    RecommendationResultsConfig(\n      target.params(PromotedAccountsFlowParams.ResultSizeParam),\n      target.params(PromotedAccountsFlowParams.BatchSizeParam)\n    )\n  }\n\n  override val statsReceiver: StatsReceiver = baseStatsReceiver.scope(\"promoted_accounts_flow\")\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/ads/PromotedAccountsFlowParams.scala",
    "content": "package com.twitter.follow_recommendations.flows.ads\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\nabstract class PromotedAccountsFlowParams[A](default: A) extends Param[A](default) {\n  override val statName: String = \"ads/\" + this.getClass.getSimpleName\n}\n\nobject PromotedAccountsFlowParams {\n\n  // number of total slots returned to the end user, available to put ads\n  case object TargetEligibility extends PromotedAccountsFlowParams[Boolean](true)\n  case object ResultSizeParam extends PromotedAccountsFlowParams[Int](Int.MaxValue)\n  case object BatchSizeParam extends PromotedAccountsFlowParams[Int](Int.MaxValue)\n  case object FetchCandidateSourceBudget\n      extends PromotedAccountsFlowParams[Duration](1000.millisecond)\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/ads/PromotedAccountsFlowRequest.scala",
    "content": "package com.twitter.follow_recommendations.flows.ads\nimport com.twitter.follow_recommendations.common.clients.adserver.AdRequest\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasExcludedUserIds\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Params\n\ncase class PromotedAccountsFlowRequest(\n  override val clientContext: ClientContext,\n  override val params: Params,\n  displayLocation: DisplayLocation,\n  profileId: Option[Long],\n  // note we also add userId and profileId to excludeUserIds\n  excludeIds: Seq[Long])\n    extends HasParams\n    with HasClientContext\n    with HasExcludedUserIds\n    with HasDisplayLocation {\n  def toAdsRequest(fetchProductionPromotedAccounts: Boolean): AdRequest = {\n    AdRequest(\n      clientContext = clientContext,\n      displayLocation = displayLocation,\n      isTest = Some(!fetchProductionPromotedAccounts),\n      profileUserId = profileId\n    )\n  }\n  override val excludedUserIds: Seq[Long] = {\n    excludeIds ++ clientContext.userId.toSeq ++ profileId.toSeq\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/ads/PromotedAccountsUtil.scala",
    "content": "package com.twitter.follow_recommendations.flows.ads\nimport com.twitter.follow_recommendations.common.candidate_sources.promoted_accounts.PromotedCandidateUser\nimport com.twitter.follow_recommendations.common.models.AccountProof\nimport com.twitter.follow_recommendations.common.models.AdMetadata\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.Reason\nimport com.twitter.follow_recommendations.common.models.UserCandidateSourceDetails\n\nobject PromotedAccountsUtil {\n  def toCandidateUser(promotedCandidateUser: PromotedCandidateUser): CandidateUser = {\n    CandidateUser(\n      id = promotedCandidateUser.id,\n      score = None,\n      adMetadata =\n        Some(AdMetadata(promotedCandidateUser.position, promotedCandidateUser.adImpression)),\n      reason = Some(\n        Reason(\n          accountProof = Some(AccountProof(followProof = Some(promotedCandidateUser.followProof))))\n      ),\n      userCandidateSourceDetails = Some(\n        UserCandidateSourceDetails(\n          promotedCandidateUser.primaryCandidateSource,\n          Map.empty,\n          Map.empty,\n          None))\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/content_recommender_flow/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/addressbook\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/crowd_search_accounts\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/real_graph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/top_organic_follows_accounts\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/user_user_graph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/gizmoduck\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/sgs\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/weighted_candidate_source_ranker\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/dedup\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/tracking_token\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/candidates\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/params\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/common\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/utils\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/content_recommender_flow/ContentRecommenderFlow.scala",
    "content": "package com.twitter.follow_recommendations.flows.content_recommender_flow\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.EnrichedCandidateSource\nimport com.twitter.follow_recommendations.common.base.GatedPredicateBase\nimport com.twitter.follow_recommendations.common.base.ParamPredicate\nimport com.twitter.follow_recommendations.common.base.Predicate\nimport com.twitter.follow_recommendations.common.base.Ranker\nimport com.twitter.follow_recommendations.common.base.RecommendationFlow\nimport com.twitter.follow_recommendations.common.base.RecommendationResultsConfig\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.predicates.ExcludedUserIdPredicate\nimport com.twitter.follow_recommendations.common.predicates.InactivePredicate\nimport com.twitter.follow_recommendations.common.predicates.gizmoduck.GizmoduckPredicate\nimport com.twitter.follow_recommendations.common.predicates.sgs.InvalidRelationshipPredicate\nimport com.twitter.follow_recommendations.common.predicates.sgs.InvalidTargetCandidateRelationshipTypesPredicate\nimport com.twitter.follow_recommendations.common.predicates.sgs.RecentFollowingPredicate\nimport com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker.WeightedCandidateSourceRanker\nimport com.twitter.follow_recommendations.common.transforms.dedup.DedupTransform\nimport com.twitter.follow_recommendations.common.transforms.tracking_token.TrackingTokenTransform\nimport com.twitter.follow_recommendations.utils.CandidateSourceHoldbackUtil\nimport com.twitter.follow_recommendations.utils.RecommendationFlowBaseSideEffectsUtil\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.quality_factor.BoundsWithDefault\nimport com.twitter.product_mixer.core.quality_factor.LinearLatencyQualityFactor\nimport com.twitter.product_mixer.core.quality_factor.LinearLatencyQualityFactorConfig\nimport com.twitter.product_mixer.core.quality_factor.LinearLatencyQualityFactorObserver\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorObserver\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ContentRecommenderFlow @Inject() (\n  contentRecommenderFlowCandidateSourceRegistry: ContentRecommenderFlowCandidateSourceRegistry,\n  recentFollowingPredicate: RecentFollowingPredicate,\n  gizmoduckPredicate: GizmoduckPredicate,\n  inactivePredicate: InactivePredicate,\n  sgsPredicate: InvalidTargetCandidateRelationshipTypesPredicate,\n  invalidRelationshipPredicate: InvalidRelationshipPredicate,\n  trackingTokenTransform: TrackingTokenTransform,\n  baseStatsReceiver: StatsReceiver)\n    extends RecommendationFlow[ContentRecommenderRequest, CandidateUser]\n    with RecommendationFlowBaseSideEffectsUtil[ContentRecommenderRequest, CandidateUser]\n    with CandidateSourceHoldbackUtil {\n\n  override val statsReceiver: StatsReceiver = baseStatsReceiver.scope(\"content_recommender_flow\")\n\n  override val qualityFactorObserver: Option[QualityFactorObserver] = {\n    val config = LinearLatencyQualityFactorConfig(\n      qualityFactorBounds =\n        BoundsWithDefault(minInclusive = 0.1, maxInclusive = 1.0, default = 1.0),\n      initialDelay = 60.seconds,\n      targetLatency = 100.milliseconds,\n      targetLatencyPercentile = 95.0,\n      delta = 0.001\n    )\n    val qualityFactor = LinearLatencyQualityFactor(config)\n    val observer = LinearLatencyQualityFactorObserver(qualityFactor)\n    statsReceiver.provideGauge(\"quality_factor\")(qualityFactor.currentValue.toFloat)\n    Some(observer)\n  }\n\n  protected override def targetEligibility: Predicate[ContentRecommenderRequest] =\n    new ParamPredicate[ContentRecommenderRequest](\n      ContentRecommenderParams.TargetEligibility\n    )\n\n  protected override def candidateSources(\n    target: ContentRecommenderRequest\n  ): Seq[CandidateSource[ContentRecommenderRequest, CandidateUser]] = {\n    import EnrichedCandidateSource._\n    val identifiers = ContentRecommenderFlowCandidateSourceWeights.getWeights(target.params).keySet\n    val selected = contentRecommenderFlowCandidateSourceRegistry.select(identifiers)\n    val budget =\n      target.params(ContentRecommenderParams.FetchCandidateSourceBudgetInMillisecond).millisecond\n    filterCandidateSources(target, selected.map(c => c.failOpenWithin(budget, statsReceiver)).toSeq)\n  }\n\n  protected override val preRankerCandidateFilter: Predicate[\n    (ContentRecommenderRequest, CandidateUser)\n  ] = {\n    val preRankerFilterStats = statsReceiver.scope(\"pre_ranker\")\n    val recentFollowingPredicateStats = preRankerFilterStats.scope(\"recent_following_predicate\")\n    val invalidRelationshipPredicateStats =\n      preRankerFilterStats.scope(\"invalid_relationship_predicate\")\n\n    object recentFollowingGatedPredicate\n        extends GatedPredicateBase[(ContentRecommenderRequest, CandidateUser)](\n          recentFollowingPredicate,\n          recentFollowingPredicateStats\n        ) {\n      override def gate(item: (ContentRecommenderRequest, CandidateUser)): Boolean =\n        item._1.params(ContentRecommenderParams.EnableRecentFollowingPredicate)\n    }\n\n    object invalidRelationshipGatedPredicate\n        extends GatedPredicateBase[(ContentRecommenderRequest, CandidateUser)](\n          invalidRelationshipPredicate,\n          invalidRelationshipPredicateStats\n        ) {\n      override def gate(item: (ContentRecommenderRequest, CandidateUser)): Boolean =\n        item._1.params(ContentRecommenderParams.EnableInvalidRelationshipPredicate)\n    }\n\n    ExcludedUserIdPredicate\n      .observe(preRankerFilterStats.scope(\"exclude_user_id_predicate\"))\n      .andThen(recentFollowingGatedPredicate.observe(recentFollowingPredicateStats))\n      .andThen(invalidRelationshipGatedPredicate.observe(invalidRelationshipPredicateStats))\n  }\n\n  /**\n   * rank the candidates\n   */\n  protected override def selectRanker(\n    target: ContentRecommenderRequest\n  ): Ranker[ContentRecommenderRequest, CandidateUser] = {\n    val rankersStatsReceiver = statsReceiver.scope(\"rankers\")\n    WeightedCandidateSourceRanker\n      .build[ContentRecommenderRequest](\n        ContentRecommenderFlowCandidateSourceWeights.getWeights(target.params),\n        randomSeed = target.getRandomizationSeed\n      ).observe(rankersStatsReceiver.scope(\"weighted_candidate_source_ranker\"))\n  }\n\n  /**\n   * transform the candidates after ranking\n   */\n  protected override def postRankerTransform: Transform[\n    ContentRecommenderRequest,\n    CandidateUser\n  ] = {\n    new DedupTransform[ContentRecommenderRequest, CandidateUser]\n      .observe(statsReceiver.scope(\"dedupping\"))\n  }\n\n  protected override def validateCandidates: Predicate[\n    (ContentRecommenderRequest, CandidateUser)\n  ] = {\n    val stats = statsReceiver.scope(\"validate_candidates\")\n    val gizmoduckPredicateStats = stats.scope(\"gizmoduck_predicate\")\n    val inactivePredicateStats = stats.scope(\"inactive_predicate\")\n    val sgsPredicateStats = stats.scope(\"sgs_predicate\")\n\n    val includeGizmoduckPredicate =\n      new ParamPredicate[ContentRecommenderRequest](\n        ContentRecommenderParams.EnableGizmoduckPredicate)\n        .map[(ContentRecommenderRequest, CandidateUser)] {\n          case (request: ContentRecommenderRequest, _) =>\n            request\n        }\n\n    val includeInactivePredicate =\n      new ParamPredicate[ContentRecommenderRequest](\n        ContentRecommenderParams.EnableInactivePredicate)\n        .map[(ContentRecommenderRequest, CandidateUser)] {\n          case (request: ContentRecommenderRequest, _) =>\n            request\n        }\n\n    val includeInvalidTargetCandidateRelationshipTypesPredicate =\n      new ParamPredicate[ContentRecommenderRequest](\n        ContentRecommenderParams.EnableInvalidTargetCandidateRelationshipPredicate)\n        .map[(ContentRecommenderRequest, CandidateUser)] {\n          case (request: ContentRecommenderRequest, _) =>\n            request\n        }\n\n    Predicate\n      .andConcurrently[(ContentRecommenderRequest, CandidateUser)](\n        Seq(\n          gizmoduckPredicate.observe(gizmoduckPredicateStats).gate(includeGizmoduckPredicate),\n          inactivePredicate.observe(inactivePredicateStats).gate(includeInactivePredicate),\n          sgsPredicate\n            .observe(sgsPredicateStats).gate(\n              includeInvalidTargetCandidateRelationshipTypesPredicate),\n        )\n      )\n  }\n\n  /**\n   * transform the candidates into results and return\n   */\n  protected override def transformResults: Transform[ContentRecommenderRequest, CandidateUser] = {\n    trackingTokenTransform\n  }\n\n  /**\n   *  configuration for recommendation results\n   */\n  protected override def resultsConfig(\n    target: ContentRecommenderRequest\n  ): RecommendationResultsConfig = {\n    RecommendationResultsConfig(\n      target.maxResults.getOrElse(target.params(ContentRecommenderParams.ResultSizeParam)),\n      target.params(ContentRecommenderParams.BatchSizeParam)\n    )\n  }\n\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/content_recommender_flow/ContentRecommenderFlowCandidateSourceRegistry.scala",
    "content": "package com.twitter.follow_recommendations.flows.content_recommender_flow\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.CandidateSourceRegistry\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ForwardEmailBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ForwardPhoneBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ReverseEmailBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ReversePhoneBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopCountryBackFillSource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopCountrySource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopGeohashSource\nimport com.twitter.follow_recommendations.common.candidate_sources.crowd_search_accounts.CrowdSearchAccountsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.ppmi_locale_follow.PPMILocaleFollowSource\nimport com.twitter.follow_recommendations.common.candidate_sources.top_organic_follows_accounts.TopOrganicFollowsAccountsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.real_graph.RealGraphOonV2Source\nimport com.twitter.follow_recommendations.common.candidate_sources.recent_engagement.RepeatedProfileVisitsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentEngagementSimilarUsersSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentFollowingSimilarUsersSource\nimport com.twitter.follow_recommendations.common.candidate_sources.socialgraph.RecentFollowingRecentFollowingExpansionSource\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.OfflineStrongTiePredictionSource\nimport com.twitter.follow_recommendations.common.candidate_sources.triangular_loops.TriangularLoopsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.user_user_graph.UserUserGraphCandidateSource\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ContentRecommenderFlowCandidateSourceRegistry @Inject() (\n  // social based\n  forwardPhoneBookSource: ForwardPhoneBookSource,\n  forwardEmailBookSource: ForwardEmailBookSource,\n  reversePhoneBookSource: ReversePhoneBookSource,\n  reverseEmailBookSource: ReverseEmailBookSource,\n  offlineStrongTiePredictionSource: OfflineStrongTiePredictionSource,\n  triangularLoopsSource: TriangularLoopsSource,\n  userUserGraphCandidateSource: UserUserGraphCandidateSource,\n  realGraphOonSource: RealGraphOonV2Source,\n  recentFollowingRecentFollowingExpansionSource: RecentFollowingRecentFollowingExpansionSource,\n  // activity based\n  recentFollowingSimilarUsersSource: RecentFollowingSimilarUsersSource,\n  recentEngagementSimilarUsersSource: RecentEngagementSimilarUsersSource,\n  repeatedProfileVisitsSource: RepeatedProfileVisitsSource,\n  // geo based\n  popCountrySource: PopCountrySource,\n  popGeohashSource: PopGeohashSource,\n  popCountryBackFillSource: PopCountryBackFillSource,\n  crowdSearchAccountsSource: CrowdSearchAccountsSource,\n  topOrganicFollowsAccountsSource: TopOrganicFollowsAccountsSource,\n  ppmiLocaleFollowSource: PPMILocaleFollowSource,\n  baseStatsReceiver: StatsReceiver)\n    extends CandidateSourceRegistry[ContentRecommenderRequest, CandidateUser] {\n\n  override val statsReceiver = baseStatsReceiver\n    .scope(\"content_recommender_flow\", \"candidate_sources\")\n\n  override val sources: Set[CandidateSource[ContentRecommenderRequest, CandidateUser]] = Seq(\n    forwardPhoneBookSource,\n    forwardEmailBookSource,\n    reversePhoneBookSource,\n    reverseEmailBookSource,\n    offlineStrongTiePredictionSource,\n    triangularLoopsSource,\n    userUserGraphCandidateSource,\n    realGraphOonSource,\n    recentFollowingRecentFollowingExpansionSource,\n    recentFollowingSimilarUsersSource,\n    recentEngagementSimilarUsersSource,\n    repeatedProfileVisitsSource,\n    popCountrySource,\n    popGeohashSource,\n    popCountryBackFillSource,\n    crowdSearchAccountsSource,\n    topOrganicFollowsAccountsSource,\n    ppmiLocaleFollowSource,\n  ).toSet\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/content_recommender_flow/ContentRecommenderFlowCandidateSourceWeights.scala",
    "content": "package com.twitter.follow_recommendations.flows.content_recommender_flow\n\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ForwardEmailBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ForwardPhoneBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ReverseEmailBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ReversePhoneBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopCountryBackFillSource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopCountrySource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopGeohashSource\nimport com.twitter.follow_recommendations.common.candidate_sources.real_graph.RealGraphOonV2Source\nimport com.twitter.follow_recommendations.common.candidate_sources.recent_engagement.RepeatedProfileVisitsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentEngagementSimilarUsersSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentFollowingSimilarUsersSource\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.OfflineStrongTiePredictionSource\nimport com.twitter.follow_recommendations.common.candidate_sources.triangular_loops.TriangularLoopsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.user_user_graph.UserUserGraphCandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.follow_recommendations.common.candidate_sources.crowd_search_accounts.CrowdSearchAccountsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.ppmi_locale_follow.PPMILocaleFollowSource\nimport com.twitter.follow_recommendations.common.candidate_sources.socialgraph.RecentFollowingRecentFollowingExpansionSource\nimport com.twitter.follow_recommendations.common.candidate_sources.top_organic_follows_accounts.TopOrganicFollowsAccountsSource\nimport com.twitter.timelines.configapi.Params\n\nobject ContentRecommenderFlowCandidateSourceWeights {\n\n  def getWeights(\n    params: Params\n  ): Map[CandidateSourceIdentifier, Double] = {\n    Map[CandidateSourceIdentifier, Double](\n      // Social based\n      UserUserGraphCandidateSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.UserUserGraphSourceWeight),\n      ForwardPhoneBookSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.ForwardPhoneBookSourceWeight),\n      ReversePhoneBookSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.ReversePhoneBookSourceWeight),\n      ForwardEmailBookSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.ForwardEmailBookSourceWeight),\n      ReverseEmailBookSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.ReverseEmailBookSourceWeight),\n      TriangularLoopsSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.TriangularLoopsSourceWeight),\n      OfflineStrongTiePredictionSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.OfflineStrongTiePredictionSourceWeight),\n      RecentFollowingRecentFollowingExpansionSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.NewFollowingNewFollowingExpansionSourceWeight),\n      RecentFollowingSimilarUsersSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.NewFollowingSimilarUserSourceWeight),\n      // Activity based\n      RealGraphOonV2Source.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.RealGraphOonSourceWeight),\n      RecentEngagementSimilarUsersSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.RecentEngagementSimilarUserSourceWeight),\n      RepeatedProfileVisitsSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.RepeatedProfileVisitsSourceWeight),\n      // Geo based\n      PopCountrySource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.PopCountrySourceWeight),\n      PopGeohashSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.PopGeohashSourceWeight),\n      PopCountryBackFillSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.PopCountryBackfillSourceWeight),\n      PPMILocaleFollowSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.PPMILocaleFollowSourceWeight),\n      CrowdSearchAccountsSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.CrowdSearchAccountSourceWeight),\n      TopOrganicFollowsAccountsSource.Identifier -> params(\n        ContentRecommenderFlowCandidateSourceWeightsParams.TopOrganicFollowsAccountsSourceWeight),\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/content_recommender_flow/ContentRecommenderFlowCandidateSourceWeightsParams.scala",
    "content": "package com.twitter.follow_recommendations.flows.content_recommender_flow\n\nimport com.twitter.timelines.configapi.FSBoundedParam\n\nobject ContentRecommenderFlowCandidateSourceWeightsParams {\n  // Social based\n  case object ForwardPhoneBookSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.ForwardPhoneBookSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object ForwardEmailBookSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.ForwardEmailBookSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object ReversePhoneBookSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.ReversePhoneBookSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object ReverseEmailBookSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.ReverseEmailBookSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object OfflineStrongTiePredictionSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.OfflineStrongTiePredictionSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object TriangularLoopsSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.TriangularLoopsSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object UserUserGraphSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.UserUserGraphSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object NewFollowingNewFollowingExpansionSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.NewFollowingNewFollowingExpansionSourceWeight,\n        1d,\n        0d,\n        1000d)\n  // Activity based\n  case object NewFollowingSimilarUserSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.NewFollowingSimilarUserSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object RecentEngagementSimilarUserSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.RecentEngagementSimilarUserSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object RepeatedProfileVisitsSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.RepeatedProfileVisitsSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object RealGraphOonSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.RealGraphOonSourceWeight,\n        1d,\n        0d,\n        1000d)\n  // Geo based\n  case object PopCountrySourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.PopCountrySourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object PopGeohashSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.PopGeohashSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object PopCountryBackfillSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.PopCountryBackfillSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object PPMILocaleFollowSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.PPMILocaleFollowSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object TopOrganicFollowsAccountsSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.TopOrganicFollowsAccountsSourceWeight,\n        1d,\n        0d,\n        1000d)\n  case object CrowdSearchAccountSourceWeight\n      extends FSBoundedParam[Double](\n        ContentRecommenderFlowFeatureSwitchKeys.CrowdSearchAccountSourceWeight,\n        1d,\n        0d,\n        1000d)\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/content_recommender_flow/ContentRecommenderFlowFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.flows.content_recommender_flow\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.Param\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ContentRecommenderFlowFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] =\n    Seq(\n      ContentRecommenderParams.IncludeActivityBasedCandidateSource,\n      ContentRecommenderParams.IncludeSocialBasedCandidateSource,\n      ContentRecommenderParams.IncludeGeoBasedCandidateSource,\n      ContentRecommenderParams.IncludeHomeTimelineTweetRecsCandidateSource,\n      ContentRecommenderParams.IncludeSocialProofEnforcedCandidateSource,\n      ContentRecommenderParams.EnableRecentFollowingPredicate,\n      ContentRecommenderParams.EnableGizmoduckPredicate,\n      ContentRecommenderParams.EnableInactivePredicate,\n      ContentRecommenderParams.EnableInvalidTargetCandidateRelationshipPredicate,\n      ContentRecommenderParams.IncludeNewFollowingNewFollowingExpansionCandidateSource,\n      ContentRecommenderParams.IncludeMoreGeoBasedCandidateSource,\n      ContentRecommenderParams.TargetEligibility,\n      ContentRecommenderParams.GetFollowersFromSgs,\n      ContentRecommenderParams.EnableInvalidRelationshipPredicate,\n    )\n\n  override val intFSParams: Seq[FSBoundedParam[Int]] =\n    Seq(\n      ContentRecommenderParams.ResultSizeParam,\n      ContentRecommenderParams.BatchSizeParam,\n      ContentRecommenderParams.FetchCandidateSourceBudgetInMillisecond,\n      ContentRecommenderParams.RecentFollowingPredicateBudgetInMillisecond,\n    )\n\n  override val doubleFSParams: Seq[FSBoundedParam[Double]] =\n    Seq(\n      ContentRecommenderFlowCandidateSourceWeightsParams.ForwardPhoneBookSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.ForwardEmailBookSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.ReversePhoneBookSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.ReverseEmailBookSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.OfflineStrongTiePredictionSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.TriangularLoopsSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.UserUserGraphSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.NewFollowingNewFollowingExpansionSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.NewFollowingSimilarUserSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.RecentEngagementSimilarUserSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.RepeatedProfileVisitsSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.RealGraphOonSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.PopCountrySourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.PopGeohashSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.PopCountryBackfillSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.PPMILocaleFollowSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.TopOrganicFollowsAccountsSourceWeight,\n      ContentRecommenderFlowCandidateSourceWeightsParams.CrowdSearchAccountSourceWeight,\n    )\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/content_recommender_flow/ContentRecommenderFlowFeatureSwitchKeys.scala",
    "content": "package com.twitter.follow_recommendations.flows.content_recommender_flow\n\nobject ContentRecommenderFlowFeatureSwitchKeys {\n  val TargetUserEligible = \"content_recommender_flow_target_eligible\"\n  val ResultSize = \"content_recommender_flow_result_size\"\n  val BatchSize = \"content_recommender_flow_batch_size\"\n  val RecentFollowingPredicateBudgetInMillisecond =\n    \"content_recommender_flow_recent_following_predicate_budget_in_ms\"\n  val CandidateGenerationBudgetInMillisecond =\n    \"content_recommender_flow_candidate_generation_budget_in_ms\"\n  val EnableRecentFollowingPredicate = \"content_recommender_flow_enable_recent_following_predicate\"\n  val EnableGizmoduckPredicate = \"content_recommender_flow_enable_gizmoduck_predicate\"\n  val EnableInactivePredicate = \"content_recommender_flow_enable_inactive_predicate\"\n  val EnableInvalidTargetCandidateRelationshipPredicate =\n    \"content_recommender_flow_enable_invalid_target_candidate_relationship_predicate\"\n  val IncludeActivityBasedCandidateSource =\n    \"content_recommender_flow_include_activity_based_candidate_source\"\n  val IncludeSocialBasedCandidateSource =\n    \"content_recommender_flow_include_social_based_candidate_source\"\n  val IncludeGeoBasedCandidateSource =\n    \"content_recommender_flow_include_geo_based_candidate_source\"\n  val IncludeHomeTimelineTweetRecsCandidateSource =\n    \"content_recommender_flow_include_home_timeline_tweet_recs_candidate_source\"\n  val IncludeSocialProofEnforcedCandidateSource =\n    \"content_recommender_flow_include_social_proof_enforced_candidate_source\"\n  val IncludeNewFollowingNewFollowingExpansionCandidateSource =\n    \"content_recommender_flow_include_new_following_new_following_expansion_candidate_source\"\n  val IncludeMoreGeoBasedCandidateSource =\n    \"content_recommender_flow_include_more_geo_based_candidate_source\"\n  val GetFollowersFromSgs = \"content_recommender_flow_get_followers_from_sgs\"\n  val EnableInvalidRelationshipPredicate =\n    \"content_recommender_flow_enable_invalid_relationship_predicate\"\n\n  // Candidate source weight param keys\n  // Social based\n  val ForwardPhoneBookSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_forward_phone_book\"\n  val ForwardEmailBookSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_forward_email_book\"\n  val ReversePhoneBookSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_reverse_phone_book\"\n  val ReverseEmailBookSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_reverse_email_book\"\n  val OfflineStrongTiePredictionSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_offline_stp\"\n  val TriangularLoopsSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_triangular_loops\"\n  val UserUserGraphSourceWeight = \"content_recommender_flow_candidate_source_weight_user_user_graph\"\n  val NewFollowingNewFollowingExpansionSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_new_following_new_following_expansion\"\n  // Activity based\n  val NewFollowingSimilarUserSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_new_following_similar_user\"\n  val RecentEngagementSimilarUserSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_recent_engagement_similar_user\"\n  val RepeatedProfileVisitsSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_repeated_profile_visits\"\n  val RealGraphOonSourceWeight = \"content_recommender_flow_candidate_source_weight_real_graph_oon\"\n  // Geo based\n  val PopCountrySourceWeight = \"content_recommender_flow_candidate_source_weight_pop_country\"\n  val PopGeohashSourceWeight = \"content_recommender_flow_candidate_source_weight_pop_geohash\"\n  val PopCountryBackfillSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_pop_country_backfill\"\n  val PPMILocaleFollowSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_ppmi_locale_follow\"\n  val TopOrganicFollowsAccountsSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_top_organic_follow_account\"\n  val CrowdSearchAccountSourceWeight =\n    \"content_recommender_flow_candidate_source_weight_crowd_search_account\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/content_recommender_flow/ContentRecommenderParams.scala",
    "content": "package com.twitter.follow_recommendations.flows.content_recommender_flow\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\n\nabstract class ContentRecommenderParams[A](default: A) extends Param[A](default) {\n  override val statName: String = \"content_recommender/\" + this.getClass.getSimpleName\n}\n\nobject ContentRecommenderParams {\n\n  case object TargetEligibility\n      extends FSParam[Boolean](ContentRecommenderFlowFeatureSwitchKeys.TargetUserEligible, true)\n\n  case object ResultSizeParam\n      extends FSBoundedParam[Int](ContentRecommenderFlowFeatureSwitchKeys.ResultSize, 15, 1, 500)\n  case object BatchSizeParam\n      extends FSBoundedParam[Int](ContentRecommenderFlowFeatureSwitchKeys.BatchSize, 15, 1, 500)\n  case object RecentFollowingPredicateBudgetInMillisecond\n      extends FSBoundedParam[Int](\n        ContentRecommenderFlowFeatureSwitchKeys.RecentFollowingPredicateBudgetInMillisecond,\n        8,\n        1,\n        50)\n  case object FetchCandidateSourceBudgetInMillisecond\n      extends FSBoundedParam[Int](\n        ContentRecommenderFlowFeatureSwitchKeys.CandidateGenerationBudgetInMillisecond,\n        60,\n        1,\n        80)\n  case object EnableRecentFollowingPredicate\n      extends FSParam[Boolean](\n        ContentRecommenderFlowFeatureSwitchKeys.EnableRecentFollowingPredicate,\n        true)\n  case object EnableGizmoduckPredicate\n      extends FSParam[Boolean](\n        ContentRecommenderFlowFeatureSwitchKeys.EnableGizmoduckPredicate,\n        false)\n  case object EnableInactivePredicate\n      extends FSParam[Boolean](\n        ContentRecommenderFlowFeatureSwitchKeys.EnableInactivePredicate,\n        false)\n  case object EnableInvalidTargetCandidateRelationshipPredicate\n      extends FSParam[Boolean](\n        ContentRecommenderFlowFeatureSwitchKeys.EnableInvalidTargetCandidateRelationshipPredicate,\n        false)\n  case object IncludeActivityBasedCandidateSource\n      extends FSParam[Boolean](\n        ContentRecommenderFlowFeatureSwitchKeys.IncludeActivityBasedCandidateSource,\n        true)\n  case object IncludeSocialBasedCandidateSource\n      extends FSParam[Boolean](\n        ContentRecommenderFlowFeatureSwitchKeys.IncludeSocialBasedCandidateSource,\n        true)\n  case object IncludeGeoBasedCandidateSource\n      extends FSParam[Boolean](\n        ContentRecommenderFlowFeatureSwitchKeys.IncludeGeoBasedCandidateSource,\n        true)\n  case object IncludeHomeTimelineTweetRecsCandidateSource\n      extends FSParam[Boolean](\n        ContentRecommenderFlowFeatureSwitchKeys.IncludeHomeTimelineTweetRecsCandidateSource,\n        false)\n  case object IncludeSocialProofEnforcedCandidateSource\n      extends FSParam[Boolean](\n        ContentRecommenderFlowFeatureSwitchKeys.IncludeSocialProofEnforcedCandidateSource,\n        false)\n  case object IncludeNewFollowingNewFollowingExpansionCandidateSource\n      extends FSParam[Boolean](\n        ContentRecommenderFlowFeatureSwitchKeys.IncludeNewFollowingNewFollowingExpansionCandidateSource,\n        false)\n\n  case object IncludeMoreGeoBasedCandidateSource\n      extends FSParam[Boolean](\n        ContentRecommenderFlowFeatureSwitchKeys.IncludeMoreGeoBasedCandidateSource,\n        false)\n\n  case object GetFollowersFromSgs\n      extends FSParam[Boolean](ContentRecommenderFlowFeatureSwitchKeys.GetFollowersFromSgs, false)\n\n  case object EnableInvalidRelationshipPredicate\n      extends FSParam[Boolean](\n        ContentRecommenderFlowFeatureSwitchKeys.EnableInvalidRelationshipPredicate,\n        false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/content_recommender_flow/ContentRecommenderRequest.scala",
    "content": "package com.twitter.follow_recommendations.flows.content_recommender_flow\n\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.follow_recommendations.common.models.DebugOptions\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.common.models.GeohashAndCountryCode\nimport com.twitter.follow_recommendations.common.models.HasDebugOptions\nimport com.twitter.follow_recommendations.common.models.HasDisplayLocation\nimport com.twitter.follow_recommendations.common.models.HasExcludedUserIds\nimport com.twitter.follow_recommendations.common.models.HasGeohashAndCountryCode\nimport com.twitter.follow_recommendations.common.models.HasInvalidRelationshipUserIds\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedByUserIds\nimport com.twitter.follow_recommendations.common.models.HasRecentFollowedUserIds\nimport com.twitter.follow_recommendations.common.models.HasUserState\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Params\n\ncase class ContentRecommenderRequest(\n  override val params: Params,\n  override val clientContext: ClientContext,\n  inputExcludeUserIds: Seq[Long],\n  override val recentFollowedUserIds: Option[Seq[Long]],\n  override val recentFollowedByUserIds: Option[Seq[Long]],\n  override val invalidRelationshipUserIds: Option[Set[Long]],\n  override val displayLocation: DisplayLocation,\n  maxResults: Option[Int] = None,\n  override val debugOptions: Option[DebugOptions] = None,\n  override val geohashAndCountryCode: Option[GeohashAndCountryCode] = None,\n  override val userState: Option[UserState] = None)\n    extends HasParams\n    with HasClientContext\n    with HasDisplayLocation\n    with HasDebugOptions\n    with HasRecentFollowedUserIds\n    with HasRecentFollowedByUserIds\n    with HasInvalidRelationshipUserIds\n    with HasExcludedUserIds\n    with HasUserState\n    with HasGeohashAndCountryCode {\n  override val excludedUserIds: Seq[Long] = {\n    inputExcludeUserIds ++ clientContext.userId.toSeq\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/content_recommender_flow/ContentRecommenderRequestBuilder.scala",
    "content": "package com.twitter.follow_recommendations.flows.content_recommender_flow\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.clients.geoduck.UserLocationFetcher\nimport com.twitter.follow_recommendations.common.clients.socialgraph.SocialGraphClient\nimport com.twitter.follow_recommendations.common.clients.user_state.UserStateClient\nimport com.twitter.follow_recommendations.common.utils.RescueWithStatsUtils.rescueOptionalWithStats\nimport com.twitter.follow_recommendations.common.utils.RescueWithStatsUtils.rescueWithStats\nimport com.twitter.follow_recommendations.common.utils.RescueWithStatsUtils.rescueWithStatsWithin\nimport com.twitter.follow_recommendations.products.common.ProductRequest\nimport com.twitter.stitch.Stitch\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ContentRecommenderRequestBuilder @Inject() (\n  socialGraph: SocialGraphClient,\n  userLocationFetcher: UserLocationFetcher,\n  userStateClient: UserStateClient,\n  statsReceiver: StatsReceiver) {\n\n  val stats: StatsReceiver = statsReceiver.scope(\"content_recommender_request_builder\")\n  val invalidRelationshipUsersStats: StatsReceiver = stats.scope(\"invalidRelationshipUserIds\")\n  private val invalidRelationshipUsersMaxSizeCounter =\n    invalidRelationshipUsersStats.counter(\"maxSize\")\n  private val invalidRelationshipUsersNotMaxSizeCounter =\n    invalidRelationshipUsersStats.counter(\"notMaxSize\")\n\n  def build(req: ProductRequest): Stitch[ContentRecommenderRequest] = {\n    val userStateStitch = Stitch\n      .collect(req.recommendationRequest.clientContext.userId.map(userId =>\n        userStateClient.getUserState(userId))).map(_.flatten)\n    val recentFollowedUserIdsStitch =\n      Stitch\n        .collect(req.recommendationRequest.clientContext.userId.map { userId =>\n          rescueWithStatsWithin(\n            socialGraph.getRecentFollowedUserIds(userId),\n            stats,\n            \"recentFollowedUserIds\",\n            req\n              .params(\n                ContentRecommenderParams.RecentFollowingPredicateBudgetInMillisecond).millisecond\n          )\n        })\n    val recentFollowedByUserIdsStitch =\n      if (req.params(ContentRecommenderParams.GetFollowersFromSgs)) {\n        Stitch\n          .collect(\n            req.recommendationRequest.clientContext.userId.map(userId =>\n              rescueWithStatsWithin(\n                socialGraph.getRecentFollowedByUserIdsFromCachedColumn(userId),\n                stats,\n                \"recentFollowedByUserIds\",\n                req\n                  .params(ContentRecommenderParams.RecentFollowingPredicateBudgetInMillisecond)\n                  .millisecond\n              )))\n      } else Stitch.None\n    val invalidRelationshipUserIdsStitch: Stitch[Option[Seq[Long]]] =\n      if (req.params(ContentRecommenderParams.EnableInvalidRelationshipPredicate)) {\n        Stitch\n          .collect(\n            req.recommendationRequest.clientContext.userId.map { userId =>\n              rescueWithStats(\n                socialGraph\n                  .getInvalidRelationshipUserIdsFromCachedColumn(userId)\n                  .onSuccess(ids =>\n                    if (ids.size >= SocialGraphClient.MaxNumInvalidRelationship) {\n                      invalidRelationshipUsersMaxSizeCounter.incr()\n                    } else {\n                      invalidRelationshipUsersNotMaxSizeCounter.incr()\n                    }),\n                stats,\n                \"invalidRelationshipUserIds\"\n              )\n            }\n          )\n      } else {\n        Stitch.None\n      }\n    val locationStitch =\n      rescueOptionalWithStats(\n        userLocationFetcher.getGeohashAndCountryCode(\n          req.recommendationRequest.clientContext.userId,\n          req.recommendationRequest.clientContext.ipAddress\n        ),\n        stats,\n        \"userLocation\"\n      )\n    Stitch\n      .join(\n        recentFollowedUserIdsStitch,\n        recentFollowedByUserIdsStitch,\n        invalidRelationshipUserIdsStitch,\n        locationStitch,\n        userStateStitch)\n      .map {\n        case (\n              recentFollowedUserIds,\n              recentFollowedByUserIds,\n              invalidRelationshipUserIds,\n              location,\n              userState) =>\n          ContentRecommenderRequest(\n            req.params,\n            req.recommendationRequest.clientContext,\n            req.recommendationRequest.excludedIds.getOrElse(Nil),\n            recentFollowedUserIds,\n            recentFollowedByUserIds,\n            invalidRelationshipUserIds.map(_.toSet),\n            req.recommendationRequest.displayLocation,\n            req.recommendationRequest.maxResults,\n            req.recommendationRequest.debugParams.flatMap(_.debugOptions),\n            location,\n            userState\n          )\n      }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/addressbook\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/crowd_search_accounts\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/real_graph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/recent_engagement\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/salsa\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/top_organic_follows_accounts\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/triangular_loops\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/two_hop_random_walk\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/user_user_graph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/deepbirdv2\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/geoduck\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/impression_store\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/interests_service\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/user_state\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/dismiss\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/gizmoduck\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/health\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/sgs\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/user_activity\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/fatigue_ranker\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/first_n_ranker\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/interleave_ranker\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/ranking\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/scoring\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/weighted_candidate_source_ranker\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/dedup\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/modify_social_proof\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/ranker_id\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/tracking_token\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/weighted_sampling\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/candidates\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/params\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/logging\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/common\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/utils\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/PostNuxMlCandidateSourceRegistry.scala",
    "content": "package com.twitter.follow_recommendations.flows.post_nux_ml\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.CandidateSourceRegistry\nimport com.twitter.follow_recommendations.common.base.EnrichedCandidateSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ForwardEmailBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ForwardPhoneBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ReverseEmailBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ReversePhoneBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.crowd_search_accounts.CrowdSearchAccountsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.top_organic_follows_accounts.TopOrganicFollowsAccountsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopCountrySource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopCountryBackFillSource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopGeohashQualityFollowSource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopGeohashSource\nimport com.twitter.follow_recommendations.common.candidate_sources.ppmi_locale_follow.PPMILocaleFollowSource\nimport com.twitter.follow_recommendations.common.candidate_sources.real_graph.RealGraphOonV2Source\nimport com.twitter.follow_recommendations.common.candidate_sources.recent_engagement.RecentEngagementNonDirectFollowSource\nimport com.twitter.follow_recommendations.common.candidate_sources.recent_engagement.RepeatedProfileVisitsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.salsa.RecentEngagementDirectFollowSalsaExpansionSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims.LinearRegressionFollow2vecNearestNeighborsStore\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentEngagementSimilarUsersSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentFollowingSimilarUsersSource\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.OnlineSTPSourceScorer\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.OfflineStrongTiePredictionSource\nimport com.twitter.follow_recommendations.common.candidate_sources.triangular_loops.TriangularLoopsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.user_user_graph.UserUserGraphCandidateSource\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PostNuxMlCandidateSourceRegistry @Inject() (\n  crowdSearchAccountsCandidateSource: CrowdSearchAccountsSource,\n  topOrganicFollowsAccountsSource: TopOrganicFollowsAccountsSource,\n  linearRegressionfollow2vecNearestNeighborsStore: LinearRegressionFollow2vecNearestNeighborsStore,\n  forwardEmailBookSource: ForwardEmailBookSource,\n  forwardPhoneBookSource: ForwardPhoneBookSource,\n  offlineStrongTiePredictionSource: OfflineStrongTiePredictionSource,\n  onlineSTPSource: OnlineSTPSourceScorer,\n  popCountrySource: PopCountrySource,\n  popCountryBackFillSource: PopCountryBackFillSource,\n  popGeohashSource: PopGeohashSource,\n  recentEngagementDirectFollowSimilarUsersSource: RecentEngagementSimilarUsersSource,\n  recentEngagementNonDirectFollowSource: RecentEngagementNonDirectFollowSource,\n  recentEngagementDirectFollowSalsaExpansionSource: RecentEngagementDirectFollowSalsaExpansionSource,\n  recentFollowingSimilarUsersSource: RecentFollowingSimilarUsersSource,\n  realGraphOonV2Source: RealGraphOonV2Source,\n  repeatedProfileVisitSource: RepeatedProfileVisitsSource,\n  reverseEmailBookSource: ReverseEmailBookSource,\n  reversePhoneBookSource: ReversePhoneBookSource,\n  triangularLoopsSource: TriangularLoopsSource,\n  userUserGraphCandidateSource: UserUserGraphCandidateSource,\n  ppmiLocaleFollowSource: PPMILocaleFollowSource,\n  popGeohashQualityFollowSource: PopGeohashQualityFollowSource,\n  baseStatsReceiver: StatsReceiver,\n) extends CandidateSourceRegistry[PostNuxMlRequest, CandidateUser] {\n  import EnrichedCandidateSource._\n\n  override val statsReceiver = baseStatsReceiver\n    .scope(\"post_nux_ml_flow\", \"candidate_sources\")\n\n  // sources primarily based on social graph signals\n  private[this] val socialSources = Seq(\n    linearRegressionfollow2vecNearestNeighborsStore.mapKeys[PostNuxMlRequest](\n      _.getOptionalUserId.toSeq),\n    forwardEmailBookSource,\n    forwardPhoneBookSource,\n    offlineStrongTiePredictionSource,\n    onlineSTPSource,\n    reverseEmailBookSource,\n    reversePhoneBookSource,\n    triangularLoopsSource,\n  )\n\n  // sources primarily based on geo signals\n  private[this] val geoSources = Seq(\n    popCountrySource,\n    popCountryBackFillSource,\n    popGeohashSource,\n    popGeohashQualityFollowSource,\n    topOrganicFollowsAccountsSource,\n    crowdSearchAccountsCandidateSource,\n    ppmiLocaleFollowSource,\n  )\n\n  // sources primarily based on recent activity signals\n  private[this] val activitySources = Seq(\n    repeatedProfileVisitSource,\n    recentEngagementDirectFollowSalsaExpansionSource.mapKeys[PostNuxMlRequest](\n      _.getOptionalUserId.toSeq),\n    recentEngagementDirectFollowSimilarUsersSource,\n    recentEngagementNonDirectFollowSource.mapKeys[PostNuxMlRequest](_.getOptionalUserId.toSeq),\n    recentFollowingSimilarUsersSource,\n    realGraphOonV2Source,\n    userUserGraphCandidateSource,\n  )\n\n  override val sources: Set[CandidateSource[PostNuxMlRequest, CandidateUser]] = (\n    geoSources ++ socialSources ++ activitySources\n  ).toSet\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/PostNuxMlCandidateSourceWeightParams.scala",
    "content": "package com.twitter.follow_recommendations.flows.post_nux_ml\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.Param\n\nabstract class PostNuxMlCandidateSourceWeightParams[A](default: A) extends Param[A](default) {\n  override val statName: String = \"post_nux_ml/\" + this.getClass.getSimpleName\n}\n\nobject PostNuxMlCandidateSourceWeightParams {\n\n  case object CandidateWeightCrowdSearch\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightCrowdSearch,\n        1.0,\n        0.0,\n        1000.0\n      )\n\n  case object CandidateWeightTopOrganicFollow\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightTopOrganicFollow,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightPPMILocaleFollow\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightPPMILocaleFollow,\n        1.0,\n        0.0,\n        1000.0\n      )\n\n  case object CandidateWeightForwardEmailBook\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightForwardEmailBook,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightForwardPhoneBook\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightForwardPhoneBook,\n        1.0,\n        0.0,\n        1000.0\n      )\n\n  case object CandidateWeightOfflineStrongTiePrediction\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightOfflineStrongTiePrediction,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightOnlineStp\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightOnlineStp,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightPopCountry\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightPopCountry,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightPopGeohash\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightPopGeohash,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightPopGeohashQualityFollow\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightPopGeohashQualityFollow,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightPopGeoBackfill\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightPopGeoBackfill,\n        1,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightRecentFollowingSimilarUsers\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightRecentFollowingSimilarUsers,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightRecentEngagementDirectFollowSalsaExpansion\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightRecentEngagementDirectFollowSalsaExpansion,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightRecentEngagementNonDirectFollow\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightRecentEngagementNonDirectFollow,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightRecentEngagementSimilarUsers\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightRecentEngagementSimilarUsers,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightRepeatedProfileVisits\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightRepeatedProfileVisits,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightFollow2vecNearestNeighbors\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightFollow2vecNearestNeighbors,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightReverseEmailBook\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightReverseEmailBook,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightReversePhoneBook\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightReversePhoneBook,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightTriangularLoops\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightTriangularLoops,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightTwoHopRandomWalk\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightTwoHopRandomWalk,\n        1.0,\n        0.0,\n        1000.0\n      )\n  case object CandidateWeightUserUserGraph\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightUserUserGraph,\n        1.0,\n        0.0,\n        1000.0\n      )\n\n  case object CandidateWeightRealGraphOonV2\n      extends FSBoundedParam[Double](\n        PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.CandidateWeightRealGraphOonV2,\n        1.0,\n        0.0,\n        2000.0\n      )\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/PostNuxMlCombinedRankerBuilder.scala",
    "content": "package com.twitter.follow_recommendations.flows.post_nux_ml\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.IdentityRanker\nimport com.twitter.follow_recommendations.common.base.IdentityTransform\nimport com.twitter.follow_recommendations.common.base.Ranker\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.models._\nimport com.twitter.follow_recommendations.common.rankers.common.RankerId\nimport com.twitter.follow_recommendations.common.rankers.fatigue_ranker.ImpressionBasedFatigueRanker\nimport com.twitter.follow_recommendations.common.rankers.first_n_ranker.FirstNRanker\nimport com.twitter.follow_recommendations.common.rankers.first_n_ranker.FirstNRankerParams\nimport com.twitter.follow_recommendations.common.rankers.interleave_ranker.InterleaveRanker\nimport com.twitter.follow_recommendations.common.rankers.ml_ranker.ranking.HydrateFeaturesTransform\nimport com.twitter.follow_recommendations.common.rankers.ml_ranker.ranking.MlRanker\nimport com.twitter.follow_recommendations.common.rankers.ml_ranker.ranking.MlRankerParams\nimport com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker.WeightedCandidateSourceRanker\nimport com.twitter.follow_recommendations.configapi.candidates.HydrateCandidateParamsTransform\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.timelines.configapi.HasParams\n\n/**\n * Used to build the combined ranker comprising 4 stages of ranking:\n * - weighted sampler\n * - truncating to the top N merged results for ranking\n * - ML ranker\n * - Interleaving ranker for producer-side experiments\n * - impression-based fatigueing\n */\n@Singleton\nclass PostNuxMlCombinedRankerBuilder[\n  T <: HasParams with HasSimilarToContext with HasClientContext with HasExcludedUserIds with HasDisplayLocation with HasDebugOptions with HasPreFetchedFeature with HasDismissedUserIds with HasQualityFactor] @Inject() (\n  firstNRanker: FirstNRanker[T],\n  hydrateFeaturesTransform: HydrateFeaturesTransform[T],\n  hydrateCandidateParamsTransform: HydrateCandidateParamsTransform[T],\n  mlRanker: MlRanker[T],\n  statsReceiver: StatsReceiver) {\n  private[this] val stats: StatsReceiver = statsReceiver.scope(\"post_nux_ml_ranker\")\n\n  // we construct each ranker independently and chain them together\n  def build(\n    request: T,\n    candidateSourceWeights: Map[CandidateSourceIdentifier, Double]\n  ): Ranker[T, CandidateUser] = {\n    val displayLocationStats = stats.scope(request.displayLocation.toString)\n    val weightedRankerStats: StatsReceiver =\n      displayLocationStats.scope(\"weighted_candidate_source_ranker\")\n    val firstNRankerStats: StatsReceiver =\n      displayLocationStats.scope(\"first_n_ranker\")\n    val hydrateCandidateParamsStats =\n      displayLocationStats.scope(\"hydrate_candidate_params\")\n    val fatigueRankerStats = displayLocationStats.scope(\"fatigue_ranker\")\n    val interleaveRankerStats =\n      displayLocationStats.scope(\"interleave_ranker\")\n    val allRankersStats = displayLocationStats.scope(\"all_rankers\")\n\n    // Checking if the heavy-ranker is an experimental model.\n    // If it is, InterleaveRanker and candidate parameter hydration are disabled.\n    // *NOTE* that consumer-side experiments should at any time take a small % of traffic, less\n    // than 20% for instance, to leave enough room for producer experiments. Increasing bucket\n    // size for producer experiments lead to other issues and is not a viable option for faster\n    // experiments.\n    val requestRankerId = request.params(MlRankerParams.RequestScorerIdParam)\n    if (requestRankerId != RankerId.PostNuxProdRanker) {\n      hydrateCandidateParamsStats.counter(s\"disabled_by_${requestRankerId.toString}\").incr()\n      interleaveRankerStats.counter(s\"disabled_by_${requestRankerId.toString}\").incr()\n    }\n\n    // weighted ranker that samples from the candidate sources\n    val weightedRanker = WeightedCandidateSourceRanker\n      .build[T](\n        candidateSourceWeights,\n        request.params(PostNuxMlParams.CandidateShuffler).shuffle(request.getRandomizationSeed),\n        randomSeed = request.getRandomizationSeed\n      ).observe(weightedRankerStats)\n\n    // ranker that takes the first n results (ie truncates output) while merging duplicates\n    val firstNRankerObs = firstNRanker.observe(firstNRankerStats)\n    // either ML ranker that uses deepbirdv2 to score or no ranking\n    val mainRanker: Ranker[T, CandidateUser] =\n      buildMainRanker(request, requestRankerId == RankerId.PostNuxProdRanker, displayLocationStats)\n    // fatigue ranker that uses wtf impressions to fatigue\n    val fatigueRanker = buildFatigueRanker(request, fatigueRankerStats).observe(fatigueRankerStats)\n\n    // interleaveRanker combines rankings from several rankers and enforces candidates' ranks in\n    // experiment buckets according to their assigned ranker model.\n    val interleaveRanker =\n      buildInterleaveRanker(\n        request,\n        requestRankerId == RankerId.PostNuxProdRanker,\n        interleaveRankerStats)\n        .observe(interleaveRankerStats)\n\n    weightedRanker\n      .andThen(firstNRankerObs)\n      .andThen(mainRanker)\n      .andThen(fatigueRanker)\n      .andThen(interleaveRanker)\n      .observe(allRankersStats)\n  }\n\n  def buildMainRanker(\n    request: T,\n    isMainRankerPostNuxProd: Boolean,\n    displayLocationStats: StatsReceiver\n  ): Ranker[T, CandidateUser] = {\n\n    // note that we may be disabling heavy ranker for users not bucketed\n    // (due to empty results from the new candidate source)\n    // need a better solution in the future\n    val mlRankerStats = displayLocationStats.scope(\"ml_ranker\")\n    val noMlRankerStats = displayLocationStats.scope(\"no_ml_ranker\")\n    val hydrateFeaturesStats =\n      displayLocationStats.scope(\"hydrate_features\")\n    val hydrateCandidateParamsStats =\n      displayLocationStats.scope(\"hydrate_candidate_params\")\n    val notHydrateCandidateParamsStats =\n      displayLocationStats.scope(\"not_hydrate_candidate_params\")\n    val rankerStats = displayLocationStats.scope(\"ranker\")\n    val mlRankerDisabledByExperimentsCounter =\n      mlRankerStats.counter(\"disabled_by_experiments\")\n    val mlRankerDisabledByQualityFactorCounter =\n      mlRankerStats.counter(\"disabled_by_quality_factor\")\n\n    val disabledByQualityFactor = request.qualityFactor\n      .exists(_ <= request.params(PostNuxMlParams.TurnoffMLScorerQFThreshold))\n\n    if (disabledByQualityFactor)\n      mlRankerDisabledByQualityFactorCounter.incr()\n\n    if (request.params(PostNuxMlParams.UseMlRanker) && !disabledByQualityFactor) {\n\n      val hydrateFeatures = hydrateFeaturesTransform\n        .observe(hydrateFeaturesStats)\n\n      val optionalHydratedParamsTransform: Transform[T, CandidateUser] = {\n        // We disable candidate parameter hydration for experimental heavy-ranker models.\n        if (isMainRankerPostNuxProd &&\n          request.params(PostNuxMlParams.EnableCandidateParamHydration)) {\n          hydrateCandidateParamsTransform\n            .observe(hydrateCandidateParamsStats)\n        } else {\n          new IdentityTransform[T, CandidateUser]()\n            .observe(notHydrateCandidateParamsStats)\n        }\n      }\n      val candidateSize = request.params(FirstNRankerParams.CandidatesToRank)\n      Ranker\n        .chain(\n          hydrateFeatures.andThen(optionalHydratedParamsTransform),\n          mlRanker.observe(mlRankerStats),\n        )\n        .within(\n          request.params(PostNuxMlParams.MlRankerBudget),\n          rankerStats.scope(s\"n$candidateSize\"))\n    } else {\n      new IdentityRanker[T, CandidateUser].observe(noMlRankerStats)\n    }\n  }\n\n  def buildInterleaveRanker(\n    request: T,\n    isMainRankerPostNuxProd: Boolean,\n    interleaveRankerStats: StatsReceiver\n  ): Ranker[T, CandidateUser] = {\n    // InterleaveRanker is enabled only for display locations powered by the PostNux heavy-ranker.\n    if (request.params(PostNuxMlParams.EnableInterleaveRanker) &&\n      // InterleaveRanker is disabled for requests with experimental heavy-rankers.\n      isMainRankerPostNuxProd) {\n      new InterleaveRanker[T](interleaveRankerStats)\n    } else {\n      new IdentityRanker[T, CandidateUser]()\n    }\n  }\n\n  def buildFatigueRanker(\n    request: T,\n    fatigueRankerStats: StatsReceiver\n  ): Ranker[T, CandidateUser] = {\n    if (request.params(PostNuxMlParams.EnableFatigueRanker)) {\n      ImpressionBasedFatigueRanker\n        .build[T](\n          fatigueRankerStats\n        ).within(request.params(PostNuxMlParams.FatigueRankerBudget), fatigueRankerStats)\n    } else {\n      new IdentityRanker[T, CandidateUser]()\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/PostNuxMlFlow.scala",
    "content": "package com.twitter.follow_recommendations.flows.post_nux_ml\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.EnrichedCandidateSource._\nimport com.twitter.follow_recommendations.common.base._\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.FilterReason\nimport com.twitter.follow_recommendations.common.predicates.dismiss.DismissedCandidatePredicate\nimport com.twitter.follow_recommendations.common.predicates.gizmoduck.GizmoduckPredicate\nimport com.twitter.follow_recommendations.common.transforms.ranker_id.RandomRankerIdTransform\nimport com.twitter.follow_recommendations.common.predicates.sgs.InvalidTargetCandidateRelationshipTypesPredicate\nimport com.twitter.follow_recommendations.common.predicates.sgs.RecentFollowingPredicate\nimport com.twitter.follow_recommendations.common.predicates.CandidateParamPredicate\nimport com.twitter.follow_recommendations.common.predicates.CandidateSourceParamPredicate\nimport com.twitter.follow_recommendations.common.predicates.CuratedCompetitorListPredicate\nimport com.twitter.follow_recommendations.common.predicates.ExcludedUserIdPredicate\nimport com.twitter.follow_recommendations.common.predicates.InactivePredicate\nimport com.twitter.follow_recommendations.common.predicates.PreviouslyRecommendedUserIdsPredicate\nimport com.twitter.follow_recommendations.common.predicates.user_activity.NonNearZeroUserActivityPredicate\nimport com.twitter.follow_recommendations.common.transforms.dedup.DedupTransform\nimport com.twitter.follow_recommendations.common.transforms.modify_social_proof.ModifySocialProofTransform\nimport com.twitter.follow_recommendations.common.transforms.tracking_token.TrackingTokenTransform\nimport com.twitter.follow_recommendations.common.transforms.weighted_sampling.SamplingTransform\nimport com.twitter.follow_recommendations.configapi.candidates.CandidateUserParamsFactory\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams.EnableGFSSocialProofTransform\nimport com.twitter.follow_recommendations.utils.CandidateSourceHoldbackUtil\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.Duration\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.follow_recommendations.common.clients.socialgraph.SocialGraphClient\nimport com.twitter.follow_recommendations.common.predicates.hss.HssPredicate\nimport com.twitter.follow_recommendations.common.predicates.sgs.InvalidRelationshipPredicate\nimport com.twitter.follow_recommendations.common.transforms.modify_social_proof.RemoveAccountProofTransform\nimport com.twitter.follow_recommendations.logging.FrsLogger\nimport com.twitter.follow_recommendations.models.RecommendationFlowData\nimport com.twitter.follow_recommendations.utils.RecommendationFlowBaseSideEffectsUtil\nimport com.twitter.product_mixer.core.model.common.identifier.RecommendationPipelineIdentifier\nimport com.twitter.product_mixer.core.quality_factor.BoundsWithDefault\nimport com.twitter.product_mixer.core.quality_factor.LinearLatencyQualityFactor\nimport com.twitter.product_mixer.core.quality_factor.LinearLatencyQualityFactorConfig\nimport com.twitter.product_mixer.core.quality_factor.LinearLatencyQualityFactorObserver\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorObserver\nimport com.twitter.stitch.Stitch\n\n/**\n * We use this flow for all post-nux display locations that would use a machine-learning-based-ranker\n * eg HTL, Sidebar, etc\n * Note that the RankedPostNuxFlow is used primarily for scribing/data collection, and doesn't\n * incorporate all of the other components in a flow (candidate source generation, predicates etc)\n */\n@Singleton\nclass PostNuxMlFlow @Inject() (\n  postNuxMlCandidateSourceRegistry: PostNuxMlCandidateSourceRegistry,\n  postNuxMlCombinedRankerBuilder: PostNuxMlCombinedRankerBuilder[PostNuxMlRequest],\n  curatedCompetitorListPredicate: CuratedCompetitorListPredicate,\n  gizmoduckPredicate: GizmoduckPredicate,\n  sgsPredicate: InvalidTargetCandidateRelationshipTypesPredicate,\n  hssPredicate: HssPredicate,\n  invalidRelationshipPredicate: InvalidRelationshipPredicate,\n  recentFollowingPredicate: RecentFollowingPredicate,\n  nonNearZeroUserActivityPredicate: NonNearZeroUserActivityPredicate,\n  inactivePredicate: InactivePredicate,\n  dismissedCandidatePredicate: DismissedCandidatePredicate,\n  previouslyRecommendedUserIdsPredicate: PreviouslyRecommendedUserIdsPredicate,\n  modifySocialProofTransform: ModifySocialProofTransform,\n  removeAccountProofTransform: RemoveAccountProofTransform,\n  trackingTokenTransform: TrackingTokenTransform,\n  randomRankerIdTransform: RandomRankerIdTransform,\n  candidateParamsFactory: CandidateUserParamsFactory[PostNuxMlRequest],\n  samplingTransform: SamplingTransform,\n  frsLogger: FrsLogger,\n  baseStatsReceiver: StatsReceiver)\n    extends RecommendationFlow[PostNuxMlRequest, CandidateUser]\n    with RecommendationFlowBaseSideEffectsUtil[PostNuxMlRequest, CandidateUser]\n    with CandidateSourceHoldbackUtil {\n  override protected val targetEligibility: Predicate[PostNuxMlRequest] =\n    new ParamPredicate[PostNuxMlRequest](PostNuxMlParams.TargetEligibility)\n\n  override val statsReceiver: StatsReceiver = baseStatsReceiver.scope(\"post_nux_ml_flow\")\n\n  override val qualityFactorObserver: Option[QualityFactorObserver] = {\n    val config = LinearLatencyQualityFactorConfig(\n      qualityFactorBounds =\n        BoundsWithDefault(minInclusive = 0.1, maxInclusive = 1.0, default = 1.0),\n      initialDelay = 60.seconds,\n      targetLatency = 700.milliseconds,\n      targetLatencyPercentile = 95.0,\n      delta = 0.001\n    )\n    val qualityFactor = LinearLatencyQualityFactor(config)\n    val observer = LinearLatencyQualityFactorObserver(qualityFactor)\n    statsReceiver.provideGauge(\"quality_factor\")(qualityFactor.currentValue.toFloat)\n    Some(observer)\n  }\n\n  override protected def updateTarget(request: PostNuxMlRequest): Stitch[PostNuxMlRequest] = {\n    Stitch.value(\n      request.copy(qualityFactor = qualityFactorObserver.map(_.qualityFactor.currentValue))\n    )\n  }\n\n  private[post_nux_ml] def getCandidateSourceIdentifiers(\n    params: Params\n  ): Set[CandidateSourceIdentifier] = {\n    PostNuxMlFlowCandidateSourceWeights.getWeights(params).keySet\n  }\n\n  override protected def candidateSources(\n    request: PostNuxMlRequest\n  ): Seq[CandidateSource[PostNuxMlRequest, CandidateUser]] = {\n    val identifiers = getCandidateSourceIdentifiers(request.params)\n    val selected: Set[CandidateSource[PostNuxMlRequest, CandidateUser]] =\n      postNuxMlCandidateSourceRegistry.select(identifiers)\n    val budget: Duration = request.params(PostNuxMlParams.FetchCandidateSourceBudget)\n    filterCandidateSources(\n      request,\n      selected.map(c => c.failOpenWithin(budget, statsReceiver)).toSeq)\n  }\n\n  override protected val preRankerCandidateFilter: Predicate[(PostNuxMlRequest, CandidateUser)] = {\n    val stats = statsReceiver.scope(\"pre_ranker\")\n\n    object excludeNearZeroUserPredicate\n        extends GatedPredicateBase[(PostNuxMlRequest, CandidateUser)](\n          nonNearZeroUserActivityPredicate,\n          stats.scope(\"exclude_near_zero_predicate\")\n        ) {\n      override def gate(item: (PostNuxMlRequest, CandidateUser)): Boolean =\n        item._1.params(PostNuxMlParams.ExcludeNearZeroCandidates)\n    }\n\n    object invalidRelationshipGatedPredicate\n        extends GatedPredicateBase[(PostNuxMlRequest, CandidateUser)](\n          invalidRelationshipPredicate,\n          stats.scope(\"invalid_relationship_predicate\")\n        ) {\n      override def gate(item: (PostNuxMlRequest, CandidateUser)): Boolean =\n        item._1.params(PostNuxMlParams.EnableInvalidRelationshipPredicate)\n    }\n\n    ExcludedUserIdPredicate\n      .observe(stats.scope(\"exclude_user_id_predicate\"))\n      .andThen(\n        recentFollowingPredicate.observe(stats.scope(\"recent_following_predicate\"))\n      )\n      .andThen(\n        dismissedCandidatePredicate.observe(stats.scope(\"dismissed_candidate_predicate\"))\n      )\n      .andThen(\n        previouslyRecommendedUserIdsPredicate.observe(\n          stats.scope(\"previously_recommended_user_ids_predicate\"))\n      )\n      .andThen(\n        invalidRelationshipGatedPredicate.observe(stats.scope(\"invalid_relationship_predicate\"))\n      )\n      .andThen(\n        excludeNearZeroUserPredicate.observe(stats.scope(\"exclude_near_zero_user_state\"))\n      )\n      .observe(stats.scope(\"overall_pre_ranker_candidate_filter\"))\n  }\n\n  override protected def selectRanker(\n    request: PostNuxMlRequest\n  ): Ranker[PostNuxMlRequest, CandidateUser] = {\n    postNuxMlCombinedRankerBuilder.build(\n      request,\n      PostNuxMlFlowCandidateSourceWeights.getWeights(request.params))\n  }\n\n  override protected val postRankerTransform: Transform[PostNuxMlRequest, CandidateUser] = {\n    new DedupTransform[PostNuxMlRequest, CandidateUser]\n      .observe(statsReceiver.scope(\"dedupping\"))\n      .andThen(\n        samplingTransform\n          .gated(PostNuxMlParams.SamplingTransformEnabled)\n          .observe(statsReceiver.scope(\"samplingtransform\")))\n  }\n\n  override protected val validateCandidates: Predicate[(PostNuxMlRequest, CandidateUser)] = {\n    val stats = statsReceiver.scope(\"validate_candidates\")\n    val competitorPredicate =\n      curatedCompetitorListPredicate.map[(PostNuxMlRequest, CandidateUser)](_._2)\n\n    val producerHoldbackPredicate = new CandidateParamPredicate[CandidateUser](\n      GlobalParams.KeepUserCandidate,\n      FilterReason.CandidateSideHoldback\n    ).map[(PostNuxMlRequest, CandidateUser)] {\n      case (request, user) => candidateParamsFactory(user, request)\n    }\n    val pymkProducerHoldbackPredicate = new CandidateSourceParamPredicate(\n      GlobalParams.KeepSocialUserCandidate,\n      FilterReason.CandidateSideHoldback,\n      CandidateSourceHoldbackUtil.SocialCandidateSourceIds\n    ).map[(PostNuxMlRequest, CandidateUser)] {\n      case (request, user) => candidateParamsFactory(user, request)\n    }\n    val sgsPredicateStats = stats.scope(\"sgs_predicate\")\n    object sgsGatedPredicate\n        extends GatedPredicateBase[(PostNuxMlRequest, CandidateUser)](\n          sgsPredicate.observe(sgsPredicateStats),\n          sgsPredicateStats\n        ) {\n\n      /**\n       * When SGS predicate is turned off, only query SGS exists API for (user, candidate, relationship)\n       * when the user's number of invalid relationships exceeds the threshold during request\n       * building step. This is to minimize load to SGS and underlying Flock DB.\n       */\n      override def gate(item: (PostNuxMlRequest, CandidateUser)): Boolean =\n        item._1.params(PostNuxMlParams.EnableSGSPredicate) ||\n          SocialGraphClient.enablePostRankerSgsPredicate(\n            item._1.invalidRelationshipUserIds.getOrElse(Set.empty).size)\n    }\n\n    val hssPredicateStats = stats.scope(\"hss_predicate\")\n    object hssGatedPredicate\n        extends GatedPredicateBase[(PostNuxMlRequest, CandidateUser)](\n          hssPredicate.observe(hssPredicateStats),\n          hssPredicateStats\n        ) {\n      override def gate(item: (PostNuxMlRequest, CandidateUser)): Boolean =\n        item._1.params(PostNuxMlParams.EnableHssPredicate)\n    }\n\n    Predicate\n      .andConcurrently[(PostNuxMlRequest, CandidateUser)](\n        Seq(\n          competitorPredicate.observe(stats.scope(\"curated_competitor_predicate\")),\n          gizmoduckPredicate.observe(stats.scope(\"gizmoduck_predicate\")),\n          sgsGatedPredicate,\n          hssGatedPredicate,\n          inactivePredicate.observe(stats.scope(\"inactive_predicate\")),\n        )\n      )\n      // to avoid dilutions, we need to apply the receiver holdback predicates at the very last step\n      .andThen(pymkProducerHoldbackPredicate.observe(stats.scope(\"pymk_receiver_side_holdback\")))\n      .andThen(producerHoldbackPredicate.observe(stats.scope(\"receiver_side_holdback\")))\n      .observe(stats.scope(\"overall_validate_candidates\"))\n  }\n\n  override protected val transformResults: Transform[PostNuxMlRequest, CandidateUser] = {\n    modifySocialProofTransform\n      .gated(EnableGFSSocialProofTransform)\n      .andThen(trackingTokenTransform)\n      .andThen(randomRankerIdTransform.gated(PostNuxMlParams.LogRandomRankerId))\n      .andThen(removeAccountProofTransform.gated(PostNuxMlParams.EnableRemoveAccountProofTransform))\n  }\n\n  override protected def resultsConfig(request: PostNuxMlRequest): RecommendationResultsConfig = {\n    RecommendationResultsConfig(\n      request.maxResults.getOrElse(request.params(PostNuxMlParams.ResultSizeParam)),\n      request.params(PostNuxMlParams.BatchSizeParam)\n    )\n  }\n\n  override def applySideEffects(\n    target: PostNuxMlRequest,\n    candidateSources: Seq[CandidateSource[PostNuxMlRequest, CandidateUser]],\n    candidatesFromCandidateSources: Seq[CandidateUser],\n    mergedCandidates: Seq[CandidateUser],\n    filteredCandidates: Seq[CandidateUser],\n    rankedCandidates: Seq[CandidateUser],\n    transformedCandidates: Seq[CandidateUser],\n    truncatedCandidates: Seq[CandidateUser],\n    results: Seq[CandidateUser]\n  ): Stitch[Unit] = {\n    frsLogger.logRecommendationFlowData[PostNuxMlRequest](\n      target,\n      RecommendationFlowData[PostNuxMlRequest](\n        target,\n        PostNuxMlFlow.identifier,\n        candidateSources,\n        candidatesFromCandidateSources,\n        mergedCandidates,\n        filteredCandidates,\n        rankedCandidates,\n        transformedCandidates,\n        truncatedCandidates,\n        results\n      )\n    )\n    super.applySideEffects(\n      target,\n      candidateSources,\n      candidatesFromCandidateSources,\n      mergedCandidates,\n      filteredCandidates,\n      rankedCandidates,\n      transformedCandidates,\n      truncatedCandidates,\n      results\n    )\n  }\n}\n\nobject PostNuxMlFlow {\n  val identifier = RecommendationPipelineIdentifier(\"PostNuxMlFlow\")\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/PostNuxMlFlowCandidateSourceWeights.scala",
    "content": "package com.twitter.follow_recommendations.flows.post_nux_ml\n\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ForwardEmailBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ForwardPhoneBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ReverseEmailBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook.ReversePhoneBookSource\nimport com.twitter.follow_recommendations.common.candidate_sources.crowd_search_accounts.CrowdSearchAccountsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopCountryBackFillSource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopCountrySource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopGeohashQualityFollowSource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopGeohashSource\nimport com.twitter.follow_recommendations.common.candidate_sources.ppmi_locale_follow.PPMILocaleFollowSource\nimport com.twitter.follow_recommendations.common.candidate_sources.real_graph.RealGraphOonV2Source\nimport com.twitter.follow_recommendations.common.candidate_sources.recent_engagement.RecentEngagementNonDirectFollowSource\nimport com.twitter.follow_recommendations.common.candidate_sources.recent_engagement.RepeatedProfileVisitsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.salsa.RecentEngagementDirectFollowSalsaExpansionSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentEngagementSimilarUsersSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentFollowingSimilarUsersSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims.Follow2vecNearestNeighborsStore\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.BaseOnlineSTPSource\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.OfflineStrongTiePredictionSource\nimport com.twitter.follow_recommendations.common.candidate_sources.top_organic_follows_accounts.TopOrganicFollowsAccountsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.triangular_loops.TriangularLoopsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.two_hop_random_walk.TwoHopRandomWalkSource\nimport com.twitter.follow_recommendations.common.candidate_sources.user_user_graph.UserUserGraphCandidateSource\nimport com.twitter.follow_recommendations.flows.post_nux_ml.PostNuxMlCandidateSourceWeightParams._\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.timelines.configapi.Params\n\nobject PostNuxMlFlowCandidateSourceWeights {\n\n  def getWeights(params: Params): Map[CandidateSourceIdentifier, Double] = {\n    Map[CandidateSourceIdentifier, Double](\n      // Social based\n      PPMILocaleFollowSource.Identifier -> params(CandidateWeightPPMILocaleFollow),\n      Follow2vecNearestNeighborsStore.IdentifierF2vLinearRegression -> params(\n        CandidateWeightFollow2vecNearestNeighbors),\n      RecentFollowingSimilarUsersSource.Identifier -> params(\n        CandidateWeightRecentFollowingSimilarUsers),\n      BaseOnlineSTPSource.Identifier -> params(CandidateWeightOnlineStp),\n      OfflineStrongTiePredictionSource.Identifier -> params(\n        CandidateWeightOfflineStrongTiePrediction),\n      ForwardEmailBookSource.Identifier -> params(CandidateWeightForwardEmailBook),\n      ForwardPhoneBookSource.Identifier -> params(CandidateWeightForwardPhoneBook),\n      ReverseEmailBookSource.Identifier -> params(CandidateWeightReverseEmailBook),\n      ReversePhoneBookSource.Identifier -> params(CandidateWeightReversePhoneBook),\n      TriangularLoopsSource.Identifier -> params(CandidateWeightTriangularLoops),\n      TwoHopRandomWalkSource.Identifier -> params(CandidateWeightTwoHopRandomWalk),\n      UserUserGraphCandidateSource.Identifier -> params(CandidateWeightUserUserGraph),\n      // Geo based\n      PopCountrySource.Identifier -> params(CandidateWeightPopCountry),\n      PopCountryBackFillSource.Identifier -> params(CandidateWeightPopGeoBackfill),\n      PopGeohashSource.Identifier -> params(CandidateWeightPopGeohash),\n      PopGeohashQualityFollowSource.Identifier -> params(CandidateWeightPopGeohashQualityFollow),\n      CrowdSearchAccountsSource.Identifier -> params(CandidateWeightCrowdSearch),\n      TopOrganicFollowsAccountsSource.Identifier -> params(CandidateWeightTopOrganicFollow),\n      // Engagement based\n      RealGraphOonV2Source.Identifier -> params(CandidateWeightRealGraphOonV2),\n      RecentEngagementNonDirectFollowSource.Identifier -> params(\n        CandidateWeightRecentEngagementNonDirectFollow),\n      RecentEngagementSimilarUsersSource.Identifier -> params(\n        CandidateWeightRecentEngagementSimilarUsers),\n      RepeatedProfileVisitsSource.Identifier -> params(CandidateWeightRepeatedProfileVisits),\n      RecentEngagementDirectFollowSalsaExpansionSource.Identifier -> params(\n        CandidateWeightRecentEngagementDirectFollowSalsaExpansion),\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys.scala",
    "content": "package com.twitter.follow_recommendations.flows.post_nux_ml\n\nobject PostNuxMlFlowCandidateSourceWeightsFeatureSwitchKeys {\n  val CandidateWeightCrowdSearch = \"post_nux_ml_flow_candidate_source_weights_user_crowd_search\"\n  val CandidateWeightTopOrganicFollow =\n    \"post_nux_ml_flow_candidate_source_weights_top_organic_follow\"\n  val CandidateWeightPPMILocaleFollow =\n    \"post_nux_ml_flow_candidate_source_weights_user_ppmi_locale_follow\"\n  val CandidateWeightForwardEmailBook =\n    \"post_nux_ml_flow_candidate_source_weights_user_forward_email_book\"\n  val CandidateWeightForwardPhoneBook =\n    \"post_nux_ml_flow_candidate_source_weights_user_forward_phone_book\"\n  val CandidateWeightOfflineStrongTiePrediction =\n    \"post_nux_ml_flow_candidate_source_weights_user_offline_strong_tie_prediction\"\n  val CandidateWeightOnlineStp = \"post_nux_ml_flow_candidate_source_weights_user_online_stp\"\n  val CandidateWeightPopCountry = \"post_nux_ml_flow_candidate_source_weights_user_pop_country\"\n  val CandidateWeightPopGeohash = \"post_nux_ml_flow_candidate_source_weights_user_pop_geohash\"\n  val CandidateWeightPopGeohashQualityFollow =\n    \"post_nux_ml_flow_candidate_source_weights_user_pop_geohash_quality_follow\"\n  val CandidateWeightPopGeoBackfill =\n    \"post_nux_ml_flow_candidate_source_weights_user_pop_geo_backfill\"\n  val CandidateWeightRecentFollowingSimilarUsers =\n    \"post_nux_ml_flow_candidate_source_weights_user_recent_following_similar_users\"\n  val CandidateWeightRecentEngagementDirectFollowSalsaExpansion =\n    \"post_nux_ml_flow_candidate_source_weights_user_recent_engagement_direct_follow_salsa_expansion\"\n  val CandidateWeightRecentEngagementNonDirectFollow =\n    \"post_nux_ml_flow_candidate_source_weights_user_recent_engagement_non_direct_follow\"\n  val CandidateWeightRecentEngagementSimilarUsers =\n    \"post_nux_ml_flow_candidate_source_weights_user_recent_engagement_similar_users\"\n  val CandidateWeightRepeatedProfileVisits =\n    \"post_nux_ml_flow_candidate_source_weights_user_repeated_profile_visits\"\n  val CandidateWeightFollow2vecNearestNeighbors =\n    \"post_nux_ml_flow_candidate_source_weights_user_follow2vec_nearest_neighbors\"\n  val CandidateWeightReverseEmailBook =\n    \"post_nux_ml_flow_candidate_source_weights_user_reverse_email_book\"\n  val CandidateWeightReversePhoneBook =\n    \"post_nux_ml_flow_candidate_source_weights_user_reverse_phone_book\"\n  val CandidateWeightTriangularLoops =\n    \"post_nux_ml_flow_candidate_source_weights_user_triangular_loops\"\n  val CandidateWeightTwoHopRandomWalk =\n    \"post_nux_ml_flow_candidate_source_weights_user_two_hop_random_walk\"\n  val CandidateWeightUserUserGraph =\n    \"post_nux_ml_flow_candidate_source_weights_user_user_user_graph\"\n  val CandidateWeightRealGraphOonV2 =\n    \"post_nux_ml_flow_candidate_source_weights_user_real_graph_oon_v2\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/PostNuxMlFlowFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.flows.post_nux_ml\n\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker.NoShuffle\nimport com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker.RandomShuffler\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PostNuxMlFlowFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] = Seq(\n    PostNuxMlParams.OnlineSTPEnabled,\n    PostNuxMlParams.SamplingTransformEnabled,\n    PostNuxMlParams.Follow2VecLinearRegressionEnabled,\n    PostNuxMlParams.UseMlRanker,\n    PostNuxMlParams.EnableCandidateParamHydration,\n    PostNuxMlParams.EnableInterleaveRanker,\n    PostNuxMlParams.EnableAdhocRanker,\n    PostNuxMlParams.ExcludeNearZeroCandidates,\n    PostNuxMlParams.IncludeRepeatedProfileVisitsCandidateSource,\n    PostNuxMlParams.EnableInterestsOptOutPredicate,\n    PostNuxMlParams.EnableSGSPredicate,\n    PostNuxMlParams.EnableInvalidRelationshipPredicate,\n    PostNuxMlParams.EnableRemoveAccountProofTransform,\n    PostNuxMlParams.EnablePPMILocaleFollowSourceInPostNux,\n    PostNuxMlParams.EnableRealGraphOonV2,\n    PostNuxMlParams.GetFollowersFromSgs,\n    PostNuxMlRequestBuilderParams.EnableInvalidRelationshipPredicate\n  )\n\n  override val doubleFSParams: Seq[FSBoundedParam[Double]] = Seq(\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightCrowdSearch,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightTopOrganicFollow,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightPPMILocaleFollow,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightForwardEmailBook,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightForwardPhoneBook,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightOfflineStrongTiePrediction,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightOnlineStp,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightPopCountry,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightPopGeohash,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightPopGeohashQualityFollow,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightPopGeoBackfill,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightRecentFollowingSimilarUsers,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightRecentEngagementDirectFollowSalsaExpansion,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightRecentEngagementNonDirectFollow,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightRecentEngagementSimilarUsers,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightRepeatedProfileVisits,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightFollow2vecNearestNeighbors,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightReverseEmailBook,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightReversePhoneBook,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightTriangularLoops,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightTwoHopRandomWalk,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightUserUserGraph,\n    PostNuxMlCandidateSourceWeightParams.CandidateWeightRealGraphOonV2,\n    PostNuxMlParams.TurnoffMLScorerQFThreshold\n  )\n\n  override val durationFSParams: Seq[FSBoundedParam[Duration] with HasDurationConversion] = Seq(\n    PostNuxMlParams.MlRankerBudget,\n    PostNuxMlRequestBuilderParams.TopicIdFetchBudget,\n    PostNuxMlRequestBuilderParams.DismissedIdScanBudget,\n    PostNuxMlRequestBuilderParams.WTFImpressionsScanBudget\n  )\n\n  override val gatedOverridesMap = Map(\n    PostNuxMlFlowFeatureSwitchKeys.EnableRandomDataCollection -> Seq(\n      PostNuxMlParams.CandidateShuffler := new RandomShuffler[CandidateUser],\n      PostNuxMlParams.LogRandomRankerId := true\n    ),\n    PostNuxMlFlowFeatureSwitchKeys.EnableNoShuffler -> Seq(\n      PostNuxMlParams.CandidateShuffler := new NoShuffle[CandidateUser]\n    ),\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/PostNuxMlFlowFeatureSwitchKeys.scala",
    "content": "package com.twitter.follow_recommendations.flows.post_nux_ml\n\nobject PostNuxMlFlowFeatureSwitchKeys {\n  val UseMlRanker = \"post_nux_ml_flow_use_ml_ranker\"\n  val EnableCandidateParamHydration = \"post_nux_ml_flow_enable_candidate_param_hydration\"\n  val OnlineSTPEnabled = \"post_nux_ml_flow_online_stp_source_enabled\"\n  val Follow2VecLinearRegressionEnabled = \"post_nux_ml_flow_follow_to_vec_lr_source_enabled\"\n  val EnableRandomDataCollection = \"post_nux_ml_flow_random_data_collection_enabled\"\n  val EnableAdhocRanker = \"post_nux_ml_flow_adhoc_ranker_enabled\"\n  val EnableFatigueRanker = \"post_nux_ml_flow_fatigue_ranker_enabled\"\n  val EnableInterleaveRanker = \"post_nux_ml_flow_interleave_ranker_enabled\"\n  val IncludeRepeatedProfileVisitsCandidateSource =\n    \"post_nux_ml_flow_include_repeated_profile_visits_candidate_source\"\n  val MLRankerBudget = \"post_nux_ml_flow_ml_ranker_budget_millis\"\n  val EnableNoShuffler = \"post_nux_ml_flow_no_shuffler\"\n  val SamplingTransformEnabled = \"post_nux_ml_flow_sampling_transform_enabled\"\n  val ExcludeNearZeroCandidates = \"post_nux_ml_flow_exclude_near_zero_candidates\"\n  val EnableInterestsOptOutPredicate = \"post_nux_ml_flow_enable_interests_opt_out_predicate\"\n  val EnableRemoveAccountProofTransform = \"post_nux_ml_flow_enable_remove_account_proof_transform\"\n  val EnablePPMILocaleFollowSourceInPostNux = \"post_nux_ml_flow_enable_ppmilocale_follow_source\"\n  val EnableInvalidRelationshipPredicate = \"post_nux_ml_flow_enable_invalid_relationship_predicate\"\n  val EnableRealGraphOonV2 = \"post_nux_ml_flow_enable_real_graph_oon_v2\"\n  val EnableSGSPredicate = \"post_nux_ml_flow_enable_sgs_predicate\"\n  val EnableHssPredicate = \"post_nux_ml_flow_enable_hss_predicate\"\n  val GetFollowersFromSgs = \"post_nux_ml_flow_get_followers_from_sgs\"\n  val TurnOffMLScorerQFThreshold = \"post_nux_ml_flow_turn_off_ml_scorer_threhsold\"\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/PostNuxMlParams.scala",
    "content": "package com.twitter.follow_recommendations.flows.post_nux_ml\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker.CandidateShuffler\nimport com.twitter.follow_recommendations.common.rankers.weighted_candidate_source_ranker.ExponentialShuffler\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\nabstract class PostNuxMlParams[A](default: A) extends Param[A](default) {\n  override val statName: String = \"post_nux_ml/\" + this.getClass.getSimpleName\n}\n\nobject PostNuxMlParams {\n\n  // infra params:\n  case object FetchCandidateSourceBudget extends PostNuxMlParams[Duration](90.millisecond)\n\n  // WTF Impression Store has very high tail latency (p9990 or p9999), but p99 latency is pretty good (~100ms)\n  // set the time budget for this step to be 200ms to make the performance of service more predictable\n  case object FatigueRankerBudget extends PostNuxMlParams[Duration](200.millisecond)\n\n  case object MlRankerBudget\n      extends FSBoundedParam[Duration](\n        name = PostNuxMlFlowFeatureSwitchKeys.MLRankerBudget,\n        default = 400.millisecond,\n        min = 100.millisecond,\n        max = 800.millisecond)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMillis\n  }\n\n  // product params:\n  case object TargetEligibility extends PostNuxMlParams[Boolean](true)\n\n  case object ResultSizeParam extends PostNuxMlParams[Int](3)\n  case object BatchSizeParam extends PostNuxMlParams[Int](12)\n\n  case object CandidateShuffler\n      extends PostNuxMlParams[CandidateShuffler[CandidateUser]](\n        new ExponentialShuffler[CandidateUser])\n  case object LogRandomRankerId extends PostNuxMlParams[Boolean](false)\n\n  // whether or not to use the ml ranker at all (feature hydration + ranker)\n  case object UseMlRanker\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.UseMlRanker, false)\n\n  // whether or not to enable candidate param hydration in postnux_ml_flow\n  case object EnableCandidateParamHydration\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.EnableCandidateParamHydration, false)\n\n  // Whether or not OnlineSTP candidates are considered in the final pool of candidates.\n  // If set to `false`, the candidate source will be removed *after* all other considerations.\n  case object OnlineSTPEnabled\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.OnlineSTPEnabled, false)\n\n  // Whether or not the candidates are sampled from a Plackett-Luce model\n  case object SamplingTransformEnabled\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.SamplingTransformEnabled, false)\n\n  // Whether or not Follow2Vec candidates are considered in the final pool of candidates.\n  // If set to `false`, the candidate source will be removed *after* all other considerations.\n  case object Follow2VecLinearRegressionEnabled\n      extends FSParam[Boolean](\n        PostNuxMlFlowFeatureSwitchKeys.Follow2VecLinearRegressionEnabled,\n        false)\n\n  // Whether or not to enable AdhocRanker to allow adhoc, non-ML, score modifications.\n  case object EnableAdhocRanker\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.EnableAdhocRanker, false)\n\n  // Whether the impression-based fatigue ranker is enabled or not.\n  case object EnableFatigueRanker\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.EnableFatigueRanker, true)\n\n  // whether or not to enable InterleaveRanker for producer-side experiments.\n  case object EnableInterleaveRanker\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.EnableInterleaveRanker, false)\n\n  // whether to exclude users in near zero user state\n  case object ExcludeNearZeroCandidates\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.ExcludeNearZeroCandidates, false)\n\n  case object EnablePPMILocaleFollowSourceInPostNux\n      extends FSParam[Boolean](\n        PostNuxMlFlowFeatureSwitchKeys.EnablePPMILocaleFollowSourceInPostNux,\n        false)\n\n  case object EnableInterestsOptOutPredicate\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.EnableInterestsOptOutPredicate, false)\n\n  case object EnableInvalidRelationshipPredicate\n      extends FSParam[Boolean](\n        PostNuxMlFlowFeatureSwitchKeys.EnableInvalidRelationshipPredicate,\n        false)\n\n  // Totally disabling SGS predicate need to disable EnableInvalidRelationshipPredicate as well\n  case object EnableSGSPredicate\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.EnableSGSPredicate, true)\n\n  case object EnableHssPredicate\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.EnableHssPredicate, true)\n\n  // Whether or not to include RepeatedProfileVisits as one of the candidate sources in the PostNuxMlFlow. If false,\n  // RepeatedProfileVisitsSource would not be run for the users in candidate_generation.\n  case object IncludeRepeatedProfileVisitsCandidateSource\n      extends FSParam[Boolean](\n        PostNuxMlFlowFeatureSwitchKeys.IncludeRepeatedProfileVisitsCandidateSource,\n        false)\n\n  case object EnableRealGraphOonV2\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.EnableRealGraphOonV2, false)\n\n  case object GetFollowersFromSgs\n      extends FSParam[Boolean](PostNuxMlFlowFeatureSwitchKeys.GetFollowersFromSgs, false)\n\n  case object EnableRemoveAccountProofTransform\n      extends FSParam[Boolean](\n        PostNuxMlFlowFeatureSwitchKeys.EnableRemoveAccountProofTransform,\n        false)\n\n  // quality factor threshold to turn off ML ranker completely\n  object TurnoffMLScorerQFThreshold\n      extends FSBoundedParam[Double](\n        name = PostNuxMlFlowFeatureSwitchKeys.TurnOffMLScorerQFThreshold,\n        default = 0.3,\n        min = 0.1,\n        max = 1.0)\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/PostNuxMlRequest.scala",
    "content": "package com.twitter.follow_recommendations.flows.post_nux_ml\n\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.models._\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Params\n\ncase class PostNuxMlRequest(\n  override val params: Params,\n  override val clientContext: ClientContext,\n  override val similarToUserIds: Seq[Long],\n  inputExcludeUserIds: Seq[Long],\n  override val recentFollowedUserIds: Option[Seq[Long]],\n  override val invalidRelationshipUserIds: Option[Set[Long]],\n  override val recentFollowedByUserIds: Option[Seq[Long]],\n  override val dismissedUserIds: Option[Seq[Long]],\n  override val displayLocation: DisplayLocation,\n  maxResults: Option[Int] = None,\n  override val debugOptions: Option[DebugOptions] = None,\n  override val wtfImpressions: Option[Seq[WtfImpression]],\n  override val uttInterestIds: Option[Seq[Long]] = None,\n  override val customInterests: Option[Seq[String]] = None,\n  override val geohashAndCountryCode: Option[GeohashAndCountryCode] = None,\n  inputPreviouslyRecommendedUserIds: Option[Set[Long]] = None,\n  inputPreviouslyFollowedUserIds: Option[Set[Long]] = None,\n  override val isSoftUser: Boolean = false,\n  override val userState: Option[UserState] = None,\n  override val qualityFactor: Option[Double] = None)\n    extends HasParams\n    with HasSimilarToContext\n    with HasClientContext\n    with HasExcludedUserIds\n    with HasDisplayLocation\n    with HasDebugOptions\n    with HasGeohashAndCountryCode\n    with HasPreFetchedFeature\n    with HasDismissedUserIds\n    with HasInterestIds\n    with HasPreviousRecommendationsContext\n    with HasIsSoftUser\n    with HasUserState\n    with HasInvalidRelationshipUserIds\n    with HasQualityFactor {\n  override val excludedUserIds: Seq[Long] = {\n    inputExcludeUserIds ++ clientContext.userId.toSeq ++ similarToUserIds\n  }\n  override val previouslyRecommendedUserIDs: Set[Long] =\n    inputPreviouslyRecommendedUserIds.getOrElse(Set.empty)\n  override val previouslyFollowedUserIds: Set[Long] =\n    inputPreviouslyFollowedUserIds.getOrElse(Set.empty)\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/PostNuxMlRequestBuilder.scala",
    "content": "package com.twitter.follow_recommendations.flows.post_nux_ml\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.clients.dismiss_store.DismissStore\nimport com.twitter.follow_recommendations.common.clients.geoduck.UserLocationFetcher\nimport com.twitter.follow_recommendations.common.clients.impression_store.WtfImpressionStore\nimport com.twitter.follow_recommendations.common.clients.interests_service.InterestServiceClient\nimport com.twitter.follow_recommendations.common.clients.socialgraph.SocialGraphClient\nimport com.twitter.follow_recommendations.common.clients.user_state.UserStateClient\nimport com.twitter.follow_recommendations.common.predicates.dismiss.DismissedCandidatePredicateParams\nimport com.twitter.follow_recommendations.common.utils.RescueWithStatsUtils._\nimport com.twitter.follow_recommendations.flows.post_nux_ml.PostNuxMlRequestBuilderParams.DismissedIdScanBudget\nimport com.twitter.follow_recommendations.flows.post_nux_ml.PostNuxMlRequestBuilderParams.TopicIdFetchBudget\nimport com.twitter.follow_recommendations.flows.post_nux_ml.PostNuxMlRequestBuilderParams.WTFImpressionsScanBudget\nimport com.twitter.follow_recommendations.products.common.ProductRequest\nimport com.twitter.inject.Logging\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PostNuxMlRequestBuilder @Inject() (\n  socialGraph: SocialGraphClient,\n  wtfImpressionStore: WtfImpressionStore,\n  dismissStore: DismissStore,\n  userLocationFetcher: UserLocationFetcher,\n  interestServiceClient: InterestServiceClient,\n  userStateClient: UserStateClient,\n  statsReceiver: StatsReceiver)\n    extends Logging {\n\n  val stats: StatsReceiver = statsReceiver.scope(\"post_nux_ml_request_builder\")\n  val invalidRelationshipUsersStats: StatsReceiver = stats.scope(\"invalidRelationshipUserIds\")\n  private val invalidRelationshipUsersMaxSizeCounter =\n    invalidRelationshipUsersStats.counter(\"maxSize\")\n  private val invalidRelationshipUsersNotMaxSizeCounter =\n    invalidRelationshipUsersStats.counter(\"notMaxSize\")\n\n  def build(\n    req: ProductRequest,\n    previouslyRecommendedUserIds: Option[Set[Long]] = None,\n    previouslyFollowedUserIds: Option[Set[Long]] = None\n  ): Stitch[PostNuxMlRequest] = {\n    val dl = req.recommendationRequest.displayLocation\n    val resultsStitch = Stitch.collect(\n      req.recommendationRequest.clientContext.userId\n        .map { userId =>\n          val lookBackDuration = req.params(DismissedCandidatePredicateParams.LookBackDuration)\n          val negativeStartTs = -(Time.now - lookBackDuration).inMillis\n          val recentFollowedUserIdsStitch =\n            rescueWithStats(\n              socialGraph.getRecentFollowedUserIds(userId),\n              stats,\n              \"recentFollowedUserIds\")\n          val invalidRelationshipUserIdsStitch =\n            if (req.params(PostNuxMlParams.EnableInvalidRelationshipPredicate)) {\n              rescueWithStats(\n                socialGraph\n                  .getInvalidRelationshipUserIds(userId)\n                  .onSuccess(ids =>\n                    if (ids.size >= SocialGraphClient.MaxNumInvalidRelationship) {\n                      invalidRelationshipUsersMaxSizeCounter.incr()\n                    } else {\n                      invalidRelationshipUsersNotMaxSizeCounter.incr()\n                    }),\n                stats,\n                \"invalidRelationshipUserIds\"\n              )\n            } else {\n              Stitch.value(Seq.empty)\n            }\n          // recentFollowedByUserIds are only used in experiment candidate sources\n          val recentFollowedByUserIdsStitch = if (req.params(PostNuxMlParams.GetFollowersFromSgs)) {\n            rescueWithStats(\n              socialGraph.getRecentFollowedByUserIdsFromCachedColumn(userId),\n              stats,\n              \"recentFollowedByUserIds\")\n          } else Stitch.value(Seq.empty)\n          val wtfImpressionsStitch =\n            rescueWithStatsWithin(\n              wtfImpressionStore.get(userId, dl),\n              stats,\n              \"wtfImpressions\",\n              req.params(WTFImpressionsScanBudget))\n          val dismissedUserIdsStitch =\n            rescueWithStatsWithin(\n              dismissStore.get(userId, negativeStartTs, None),\n              stats,\n              \"dismissedUserIds\",\n              req.params(DismissedIdScanBudget))\n          val locationStitch =\n            rescueOptionalWithStats(\n              userLocationFetcher.getGeohashAndCountryCode(\n                Some(userId),\n                req.recommendationRequest.clientContext.ipAddress),\n              stats,\n              \"userLocation\"\n            )\n          val topicIdsStitch =\n            rescueWithStatsWithin(\n              interestServiceClient.fetchUttInterestIds(userId),\n              stats,\n              \"topicIds\",\n              req.params(TopicIdFetchBudget))\n          val userStateStitch =\n            rescueOptionalWithStats(userStateClient.getUserState(userId), stats, \"userState\")\n          Stitch.join(\n            recentFollowedUserIdsStitch,\n            invalidRelationshipUserIdsStitch,\n            recentFollowedByUserIdsStitch,\n            dismissedUserIdsStitch,\n            wtfImpressionsStitch,\n            locationStitch,\n            topicIdsStitch,\n            userStateStitch\n          )\n        })\n\n    resultsStitch.map {\n      case Some(\n            (\n              recentFollowedUserIds,\n              invalidRelationshipUserIds,\n              recentFollowedByUserIds,\n              dismissedUserIds,\n              wtfImpressions,\n              locationInfo,\n              topicIds,\n              userState)) =>\n        PostNuxMlRequest(\n          params = req.params,\n          clientContext = req.recommendationRequest.clientContext,\n          similarToUserIds = Nil,\n          inputExcludeUserIds = req.recommendationRequest.excludedIds.getOrElse(Nil),\n          recentFollowedUserIds = Some(recentFollowedUserIds),\n          invalidRelationshipUserIds = Some(invalidRelationshipUserIds.toSet),\n          recentFollowedByUserIds = Some(recentFollowedByUserIds),\n          dismissedUserIds = Some(dismissedUserIds),\n          displayLocation = dl,\n          maxResults = req.recommendationRequest.maxResults,\n          debugOptions = req.recommendationRequest.debugParams.flatMap(_.debugOptions),\n          wtfImpressions = Some(wtfImpressions),\n          geohashAndCountryCode = locationInfo,\n          uttInterestIds = Some(topicIds),\n          inputPreviouslyRecommendedUserIds = previouslyRecommendedUserIds,\n          inputPreviouslyFollowedUserIds = previouslyFollowedUserIds,\n          isSoftUser = req.recommendationRequest.isSoftUser,\n          userState = userState\n        )\n      case _ =>\n        PostNuxMlRequest(\n          params = req.params,\n          clientContext = req.recommendationRequest.clientContext,\n          similarToUserIds = Nil,\n          inputExcludeUserIds = req.recommendationRequest.excludedIds.getOrElse(Nil),\n          recentFollowedUserIds = None,\n          invalidRelationshipUserIds = None,\n          recentFollowedByUserIds = None,\n          dismissedUserIds = None,\n          displayLocation = dl,\n          maxResults = req.recommendationRequest.maxResults,\n          debugOptions = req.recommendationRequest.debugParams.flatMap(_.debugOptions),\n          wtfImpressions = None,\n          geohashAndCountryCode = None,\n          inputPreviouslyRecommendedUserIds = previouslyRecommendedUserIds,\n          inputPreviouslyFollowedUserIds = previouslyFollowedUserIds,\n          isSoftUser = req.recommendationRequest.isSoftUser,\n          userState = None\n        )\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml/PostNuxMlRequestBuilderParams.scala",
    "content": "package com.twitter.follow_recommendations.flows.post_nux_ml\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.util.Duration\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.HasDurationConversion\n\nobject PostNuxMlRequestBuilderParams {\n  case object TopicIdFetchBudget\n      extends FSBoundedParam[Duration](\n        name = \"post_nux_ml_request_builder_topic_id_fetch_budget_millis\",\n        default = 200.millisecond,\n        min = 80.millisecond,\n        max = 400.millisecond)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMillis\n  }\n\n  case object DismissedIdScanBudget\n      extends FSBoundedParam[Duration](\n        name = \"post_nux_ml_request_builder_dismissed_id_scan_budget_millis\",\n        default = 200.millisecond,\n        min = 80.millisecond,\n        max = 400.millisecond)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMillis\n  }\n\n  case object WTFImpressionsScanBudget\n      extends FSBoundedParam[Duration](\n        name = \"post_nux_ml_request_builder_wtf_impressions_scan_budget_millis\",\n        default = 200.millisecond,\n        min = 80.millisecond,\n        max = 400.millisecond)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMillis\n  }\n\n  case object EnableInvalidRelationshipPredicate\n      extends FSParam[Boolean](\n        name = \"post_nux_ml_request_builder_enable_invalid_relationship_predicate\",\n        false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/logging/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/params\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"scribelib/marshallers/src/main/scala/com/twitter/scribelib/marshallers\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/logging/FrsLogger.scala",
    "content": "package com.twitter.follow_recommendations.logging\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.follow_recommendations.common.models.HasIsSoftUser\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams\nimport com.twitter.follow_recommendations.logging.thriftscala.RecommendationLog\nimport com.twitter.follow_recommendations.models.DebugParams\nimport com.twitter.follow_recommendations.models.RecommendationFlowData\nimport com.twitter.follow_recommendations.models.RecommendationRequest\nimport com.twitter.follow_recommendations.models.RecommendationResponse\nimport com.twitter.follow_recommendations.models.ScoringUserRequest\nimport com.twitter.follow_recommendations.models.ScoringUserResponse\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logging.LoggerFactory\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.scribelib.marshallers.ClientDataProvider\nimport com.twitter.scribelib.marshallers.ExternalRefererDataProvider\nimport com.twitter.scribelib.marshallers.ScribeSerialization\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/**\n * This is the standard logging class we use to log data into:\n * 1) logs.follow_recommendations_logs\n *\n * This logger logs data for 2 endpoints: getRecommendations, scoreUserCandidates\n * All data scribed via this logger have to be converted into the same thrift type: RecommendationLog\n *\n * 2) logs.frs_recommendation_flow_logs\n *\n * This logger logs recommendation flow data for getRecommendations requests\n * All data scribed via this logger have to be converted into the same thrift type: FrsRecommendationFlowLog\n */\n@Singleton\nclass FrsLogger @Inject() (\n  @Named(GuiceNamedConstants.REQUEST_LOGGER) loggerFactory: LoggerFactory,\n  @Named(GuiceNamedConstants.FLOW_LOGGER) flowLoggerFactory: LoggerFactory,\n  stats: StatsReceiver,\n  @Flag(\"log_results\") serviceShouldLogResults: Boolean)\n    extends ScribeSerialization {\n  private val logger = loggerFactory.apply()\n  private val flowLogger = flowLoggerFactory.apply()\n  private val logRecommendationCounter = stats.counter(\"scribe_recommendation\")\n  private val logScoringCounter = stats.counter(\"scribe_scoring\")\n  private val logRecommendationFlowCounter = stats.counter(\"scribe_recommendation_flow\")\n\n  def logRecommendationResult(\n    request: RecommendationRequest,\n    response: RecommendationResponse\n  ): Unit = {\n    if (!request.isSoftUser) {\n      val log =\n        RecommendationLog(request.toOfflineThrift, response.toOfflineThrift, Time.now.inMillis)\n      logRecommendationCounter.incr()\n      logger.info(\n        serializeThrift(\n          log,\n          FrsLogger.LogCategory,\n          FrsLogger.mkProvider(request.clientContext)\n        ))\n    }\n  }\n\n  def logScoringResult(request: ScoringUserRequest, response: ScoringUserResponse): Unit = {\n    if (!request.isSoftUser) {\n      val log =\n        RecommendationLog(\n          request.toRecommendationRequest.toOfflineThrift,\n          response.toRecommendationResponse.toOfflineThrift,\n          Time.now.inMillis)\n      logScoringCounter.incr()\n      logger.info(\n        serializeThrift(\n          log,\n          FrsLogger.LogCategory,\n          FrsLogger.mkProvider(request.toRecommendationRequest.clientContext)\n        ))\n    }\n  }\n\n  def logRecommendationFlowData[Target <: HasClientContext with HasIsSoftUser with HasParams](\n    request: Target,\n    flowData: RecommendationFlowData[Target]\n  ): Unit = {\n    if (!request.isSoftUser && request.params(GlobalParams.EnableRecommendationFlowLogs)) {\n      val log = flowData.toRecommendationFlowLogOfflineThrift\n      logRecommendationFlowCounter.incr()\n      flowLogger.info(\n        serializeThrift(\n          log,\n          FrsLogger.FlowLogCategory,\n          FrsLogger.mkProvider(request.clientContext)\n        ))\n    }\n  }\n\n  // We prefer the settings given in the user request, and if none provided we default to the\n  // aurora service configuration.\n  def shouldLog(debugParamsOpt: Option[DebugParams]): Boolean =\n    debugParamsOpt match {\n      case Some(debugParams) =>\n        debugParams.debugOptions match {\n          case Some(debugOptions) =>\n            !debugOptions.doNotLog\n          case None =>\n            serviceShouldLogResults\n        }\n      case None =>\n        serviceShouldLogResults\n    }\n\n}\n\nobject FrsLogger {\n  val LogCategory = \"follow_recommendations_logs\"\n  val FlowLogCategory = \"frs_recommendation_flow_logs\"\n\n  def mkProvider(clientContext: ClientContext) = new ClientDataProvider {\n\n    /** The id of the current user. When the user is logged out, this method should return None. */\n    override val userId: Option[Long] = clientContext.userId\n\n    /** The id of the guest, which is present in logged-in or loged-out states */\n    override val guestId: Option[Long] = clientContext.guestId\n\n    /** The personalization id (pid) of the user, used to personalize Twitter services */\n    override val personalizationId: Option[String] = None\n\n    /** The id of the individual device the user is currently using. This id will be unique for different users' devices. */\n    override val deviceId: Option[String] = clientContext.deviceId\n\n    /** The OAuth application id of the application the user is currently using */\n    override val clientApplicationId: Option[Long] = clientContext.appId\n\n    /** The OAuth parent application id of the application the user is currently using */\n    override val parentApplicationId: Option[Long] = None\n\n    /** The two-letter, upper-case country code used to designate the country from which the scribe event occurred */\n    override val countryCode: Option[String] = clientContext.countryCode\n\n    /** The two-letter, lower-case language code used to designate the probably language spoken by the scribe event initiator */\n    override val languageCode: Option[String] = clientContext.languageCode\n\n    /** The user-agent header used to identify the client browser or device that the user is currently active on */\n    override val userAgent: Option[String] = clientContext.userAgent\n\n    /** Whether the user is accessing Twitter via a secured connection */\n    override val isSsl: Option[Boolean] = Some(true)\n\n    /** The referring URL to the current page for web-based clients, if applicable */\n    override val referer: Option[String] = None\n\n    /**\n     * The external site, partner, or email that lead to the current Twitter application. Returned value consists of a\n     * tuple including the encrypted referral data and the type of referral\n     */\n    override val externalReferer: Option[ExternalRefererDataProvider] = None\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core/src/main/scala/com/twitter/timelines/configapi\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/common\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/CandidateSourceType.scala",
    "content": "package com.twitter.follow_recommendations.models\n\nobject CandidateSourceType extends Enumeration {\n  type CandidateSourceType = Value\n  val Social = Value(\"social\")\n  val GeoAndInterests = Value(\"geo_and_interests\")\n  val ActivityContextual = Value(\"activity_contextual\")\n  val None = Value(\"none\")\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/CandidateUserDebugParams.scala",
    "content": "package com.twitter.follow_recommendations.models\n\nimport com.twitter.timelines.configapi.Params\n\ncase class CandidateUserDebugParams(paramsMap: Map[Long, Params])\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/DebugParams.scala",
    "content": "package com.twitter.follow_recommendations.models\n\nimport com.twitter.follow_recommendations.common.models.DebugOptions\nimport com.twitter.follow_recommendations.common.models.DebugOptions.fromDebugParamsThrift\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport com.twitter.timelines.configapi.{FeatureValue => ConfigApiFeatureValue}\n\ncase class DebugParams(\n  featureOverrides: Option[Map[String, ConfigApiFeatureValue]],\n  debugOptions: Option[DebugOptions])\n\nobject DebugParams {\n  def fromThrift(thrift: t.DebugParams): DebugParams = DebugParams(\n    featureOverrides = thrift.featureOverrides.map { map =>\n      map.mapValues(FeatureValue.fromThrift).toMap\n    },\n    debugOptions = Some(\n      fromDebugParamsThrift(thrift)\n    )\n  )\n  def toOfflineThrift(model: DebugParams): offline.OfflineDebugParams =\n    offline.OfflineDebugParams(randomizationSeed = model.debugOptions.flatMap(_.randomizationSeed))\n}\n\ntrait HasFrsDebugParams {\n  def frsDebugParams: Option[DebugParams]\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/DisplayContext.scala",
    "content": "package com.twitter.follow_recommendations.models\n\nimport com.twitter.follow_recommendations.common.models.FlowContext\nimport com.twitter.follow_recommendations.common.models.RecentlyEngagedUserId\nimport com.twitter.follow_recommendations.logging.thriftscala.OfflineDisplayContext\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport scala.reflect.ClassTag\nimport scala.reflect.classTag\n\ntrait DisplayContext {\n  def toOfflineThrift: offline.OfflineDisplayContext\n}\n\nobject DisplayContext {\n  case class Profile(profileId: Long) extends DisplayContext {\n    override val toOfflineThrift: OfflineDisplayContext =\n      offline.OfflineDisplayContext.Profile(offline.OfflineProfile(profileId))\n  }\n  case class Search(searchQuery: String) extends DisplayContext {\n    override val toOfflineThrift: OfflineDisplayContext =\n      offline.OfflineDisplayContext.Search(offline.OfflineSearch(searchQuery))\n  }\n  case class Rux(focalAuthorId: Long) extends DisplayContext {\n    override val toOfflineThrift: OfflineDisplayContext =\n      offline.OfflineDisplayContext.Rux(offline.OfflineRux(focalAuthorId))\n  }\n\n  case class Topic(topicId: Long) extends DisplayContext {\n    override val toOfflineThrift: OfflineDisplayContext =\n      offline.OfflineDisplayContext.Topic(offline.OfflineTopic(topicId))\n  }\n\n  case class ReactiveFollow(followedUserIds: Seq[Long]) extends DisplayContext {\n    override val toOfflineThrift: OfflineDisplayContext =\n      offline.OfflineDisplayContext.ReactiveFollow(offline.OfflineReactiveFollow(followedUserIds))\n  }\n\n  case class NuxInterests(flowContext: Option[FlowContext], uttInterestIds: Option[Seq[Long]])\n      extends DisplayContext {\n    override val toOfflineThrift: OfflineDisplayContext =\n      offline.OfflineDisplayContext.NuxInterests(\n        offline.OfflineNuxInterests(flowContext.map(_.toOfflineThrift)))\n  }\n\n  case class PostNuxFollowTask(flowContext: Option[FlowContext]) extends DisplayContext {\n    override val toOfflineThrift: OfflineDisplayContext =\n      offline.OfflineDisplayContext.PostNuxFollowTask(\n        offline.OfflinePostNuxFollowTask(flowContext.map(_.toOfflineThrift)))\n  }\n\n  case class AdCampaignTarget(similarToUserIds: Seq[Long]) extends DisplayContext {\n    override val toOfflineThrift: OfflineDisplayContext =\n      offline.OfflineDisplayContext.AdCampaignTarget(\n        offline.OfflineAdCampaignTarget(similarToUserIds))\n  }\n\n  case class ConnectTab(\n    byfSeedUserIds: Seq[Long],\n    similarToUserIds: Seq[Long],\n    engagedUserIds: Seq[RecentlyEngagedUserId])\n      extends DisplayContext {\n    override val toOfflineThrift: OfflineDisplayContext =\n      offline.OfflineDisplayContext.ConnectTab(\n        offline.OfflineConnectTab(\n          byfSeedUserIds,\n          similarToUserIds,\n          engagedUserIds.map(user => user.toOfflineThrift)))\n  }\n\n  case class SimilarToUser(similarToUserId: Long) extends DisplayContext {\n    override val toOfflineThrift: OfflineDisplayContext =\n      offline.OfflineDisplayContext.SimilarToUser(offline.OfflineSimilarToUser(similarToUserId))\n  }\n\n  def fromThrift(tDisplayContext: t.DisplayContext): DisplayContext = tDisplayContext match {\n    case t.DisplayContext.Profile(p) => Profile(p.profileId)\n    case t.DisplayContext.Search(s) => Search(s.searchQuery)\n    case t.DisplayContext.Rux(r) => Rux(r.focalAuthorId)\n    case t.DisplayContext.Topic(t) => Topic(t.topicId)\n    case t.DisplayContext.ReactiveFollow(f) => ReactiveFollow(f.followedUserIds)\n    case t.DisplayContext.NuxInterests(n) =>\n      NuxInterests(n.flowContext.map(FlowContext.fromThrift), n.uttInterestIds)\n    case t.DisplayContext.AdCampaignTarget(a) =>\n      AdCampaignTarget(a.similarToUserIds)\n    case t.DisplayContext.ConnectTab(connect) =>\n      ConnectTab(\n        connect.byfSeedUserIds,\n        connect.similarToUserIds,\n        connect.recentlyEngagedUserIds.map(RecentlyEngagedUserId.fromThrift))\n    case t.DisplayContext.SimilarToUser(r) =>\n      SimilarToUser(r.similarToUserId)\n    case t.DisplayContext.PostNuxFollowTask(p) =>\n      PostNuxFollowTask(p.flowContext.map(FlowContext.fromThrift))\n    case t.DisplayContext.UnknownUnionField(t) =>\n      throw new UnknownDisplayContextException(t.field.name)\n  }\n\n  def getDisplayContextAs[T <: DisplayContext: ClassTag](displayContext: DisplayContext): T =\n    displayContext match {\n      case context: T => context\n      case _ =>\n        throw new UnexpectedDisplayContextTypeException(\n          displayContext,\n          classTag[T].getClass.getSimpleName)\n    }\n}\n\nclass UnknownDisplayContextException(name: String)\n    extends Exception(s\"Unknown DisplayContext in Thrift: ${name}\")\n\nclass UnexpectedDisplayContextTypeException(displayContext: DisplayContext, expectedType: String)\n    extends Exception(s\"DisplayContext ${displayContext} not of expected type ${expectedType}\")\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/FeatureValue.scala",
    "content": "package com.twitter.follow_recommendations.models\n\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport com.twitter.timelines.configapi._\n\nobject FeatureValue {\n  def fromThrift(thriftFeatureValue: t.FeatureValue): FeatureValue = thriftFeatureValue match {\n    case t.FeatureValue.PrimitiveValue(t.PrimitiveFeatureValue.BoolValue(bool)) =>\n      BooleanFeatureValue(bool)\n    case t.FeatureValue.PrimitiveValue(t.PrimitiveFeatureValue.StrValue(string)) =>\n      StringFeatureValue(string)\n    case t.FeatureValue.PrimitiveValue(t.PrimitiveFeatureValue.IntValue(int)) =>\n      NumberFeatureValue(int)\n    case t.FeatureValue.PrimitiveValue(t.PrimitiveFeatureValue.LongValue(long)) =>\n      NumberFeatureValue(long)\n    case t.FeatureValue.PrimitiveValue(t.PrimitiveFeatureValue.UnknownUnionField(field)) =>\n      throw new UnknownFeatureValueException(s\"Primitive: ${field.field.name}\")\n    case t.FeatureValue.UnknownUnionField(field) =>\n      throw new UnknownFeatureValueException(field.field.name)\n  }\n}\n\nclass UnknownFeatureValueException(fieldName: String)\n    extends Exception(s\"Unknown FeatureValue name in thrift: ${fieldName}\")\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/RecommendationFlowData.scala",
    "content": "package com.twitter.follow_recommendations.models\n\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.ClientContextConverter\nimport com.twitter.follow_recommendations.common.models.HasUserState\nimport com.twitter.follow_recommendations.common.utils.UserSignupUtil\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.RecommendationPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.util.Time\n\ncase class RecommendationFlowData[Target <: HasClientContext](\n  request: Target,\n  recommendationFlowIdentifier: RecommendationPipelineIdentifier,\n  candidateSources: Seq[CandidateSource[Target, CandidateUser]],\n  candidatesFromCandidateSources: Seq[CandidateUser],\n  mergedCandidates: Seq[CandidateUser],\n  filteredCandidates: Seq[CandidateUser],\n  rankedCandidates: Seq[CandidateUser],\n  transformedCandidates: Seq[CandidateUser],\n  truncatedCandidates: Seq[CandidateUser],\n  results: Seq[CandidateUser])\n    extends HasMarshalling {\n\n  import RecommendationFlowData._\n\n  lazy val toRecommendationFlowLogOfflineThrift: offline.RecommendationFlowLog = {\n    val userMetadata = userToOfflineRecommendationFlowUserMetadata(request)\n    val signals = userToOfflineRecommendationFlowSignals(request)\n    val filteredCandidateSourceCandidates =\n      candidatesToOfflineRecommendationFlowCandidateSourceCandidates(\n        candidateSources,\n        filteredCandidates\n      )\n    val rankedCandidateSourceCandidates =\n      candidatesToOfflineRecommendationFlowCandidateSourceCandidates(\n        candidateSources,\n        rankedCandidates\n      )\n    val truncatedCandidateSourceCandidates =\n      candidatesToOfflineRecommendationFlowCandidateSourceCandidates(\n        candidateSources,\n        truncatedCandidates\n      )\n\n    offline.RecommendationFlowLog(\n      ClientContextConverter.toFRSOfflineClientContextThrift(request.clientContext),\n      userMetadata,\n      signals,\n      Time.now.inMillis,\n      recommendationFlowIdentifier.name,\n      Some(filteredCandidateSourceCandidates),\n      Some(rankedCandidateSourceCandidates),\n      Some(truncatedCandidateSourceCandidates)\n    )\n  }\n}\n\nobject RecommendationFlowData {\n  def userToOfflineRecommendationFlowUserMetadata[Target <: HasClientContext](\n    request: Target\n  ): Option[offline.OfflineRecommendationFlowUserMetadata] = {\n    val userSignupAge = UserSignupUtil.userSignupAge(request).map(_.inDays)\n    val userState = request match {\n      case req: HasUserState => req.userState.map(_.name)\n      case _ => None\n    }\n    Some(offline.OfflineRecommendationFlowUserMetadata(userSignupAge, userState))\n  }\n\n  def userToOfflineRecommendationFlowSignals[Target <: HasClientContext](\n    request: Target\n  ): Option[offline.OfflineRecommendationFlowSignals] = {\n    val countryCode = request.getCountryCode\n    Some(offline.OfflineRecommendationFlowSignals(countryCode))\n  }\n\n  def candidatesToOfflineRecommendationFlowCandidateSourceCandidates[Target <: HasClientContext](\n    candidateSources: Seq[CandidateSource[Target, CandidateUser]],\n    candidates: Seq[CandidateUser],\n  ): Seq[offline.OfflineRecommendationFlowCandidateSourceCandidates] = {\n    val candidatesGroupedByCandidateSources =\n      candidates.groupBy(\n        _.getPrimaryCandidateSource.getOrElse(CandidateSourceIdentifier(\"NoCandidateSource\")))\n\n    candidateSources.map(candidateSource => {\n      val candidates =\n        candidatesGroupedByCandidateSources.get(candidateSource.identifier).toSeq.flatten\n      val candidateUserIds = candidates.map(_.id)\n      val candidateUserScores = candidates.map(_.score).exists(_.nonEmpty) match {\n        case true => Some(candidates.map(_.score.getOrElse(-1.0)))\n        case false => None\n      }\n      offline.OfflineRecommendationFlowCandidateSourceCandidates(\n        candidateSource.identifier.name,\n        candidateUserIds,\n        candidateUserScores\n      )\n    })\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/RecommendationRequest.scala",
    "content": "package com.twitter.follow_recommendations.models\n\nimport com.twitter.follow_recommendations.common.models.ClientContextConverter\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\n\ncase class RecommendationRequest(\n  clientContext: ClientContext,\n  displayLocation: DisplayLocation,\n  displayContext: Option[DisplayContext],\n  maxResults: Option[Int],\n  cursor: Option[String],\n  excludedIds: Option[Seq[Long]],\n  fetchPromotedContent: Option[Boolean],\n  debugParams: Option[DebugParams] = None,\n  userLocationState: Option[String] = None,\n  isSoftUser: Boolean = false) {\n  def toOfflineThrift: offline.OfflineRecommendationRequest = offline.OfflineRecommendationRequest(\n    ClientContextConverter.toFRSOfflineClientContextThrift(clientContext),\n    displayLocation.toOfflineThrift,\n    displayContext.map(_.toOfflineThrift),\n    maxResults,\n    cursor,\n    excludedIds,\n    fetchPromotedContent,\n    debugParams.map(DebugParams.toOfflineThrift)\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/RecommendationResponse.scala",
    "content": "package com.twitter.follow_recommendations.models\n\nimport com.twitter.follow_recommendations.{thriftscala => t}\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.follow_recommendations.common.models.Recommendation\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\n\ncase class RecommendationResponse(recommendations: Seq[Recommendation]) extends HasMarshalling {\n  lazy val toThrift: t.RecommendationResponse =\n    t.RecommendationResponse(recommendations.map(_.toThrift))\n\n  lazy val toOfflineThrift: offline.OfflineRecommendationResponse =\n    offline.OfflineRecommendationResponse(recommendations.map(_.toOfflineThrift))\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/Request.scala",
    "content": "package com.twitter.follow_recommendations.models\n\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.product_mixer.core.model.marshalling.request\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.product_mixer.core.model.marshalling.request.ProductContext\nimport com.twitter.product_mixer.core.model.marshalling.request.{Request => ProductMixerRequest}\n\ncase class Request(\n  override val maxResults: Option[Int],\n  override val debugParams: Option[request.DebugParams],\n  override val productContext: Option[ProductContext],\n  override val product: request.Product,\n  override val clientContext: ClientContext,\n  override val serializedRequestCursor: Option[String],\n  override val frsDebugParams: Option[DebugParams],\n  displayLocation: DisplayLocation,\n  excludedIds: Option[Seq[Long]],\n  fetchPromotedContent: Option[Boolean],\n  userLocationState: Option[String] = None)\n    extends ProductMixerRequest\n    with HasFrsDebugParams {}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/ScoringUserRequest.scala",
    "content": "package com.twitter.follow_recommendations.models\n\nimport com.twitter.follow_recommendations.common.feature_hydration.common.HasPreFetchedFeature\nimport com.twitter.follow_recommendations.common.models._\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Params\n\ncase class ScoringUserRequest(\n  override val clientContext: ClientContext,\n  override val displayLocation: DisplayLocation,\n  override val params: Params,\n  override val debugOptions: Option[DebugOptions] = None,\n  override val recentFollowedUserIds: Option[Seq[Long]],\n  override val recentFollowedByUserIds: Option[Seq[Long]],\n  override val wtfImpressions: Option[Seq[WtfImpression]],\n  override val similarToUserIds: Seq[Long],\n  candidates: Seq[CandidateUser],\n  debugParams: Option[DebugParams] = None,\n  isSoftUser: Boolean = false)\n    extends HasClientContext\n    with HasDisplayLocation\n    with HasParams\n    with HasDebugOptions\n    with HasPreFetchedFeature\n    with HasSimilarToContext {\n  def toOfflineThrift: offline.OfflineScoringUserRequest = offline.OfflineScoringUserRequest(\n    ClientContextConverter.toFRSOfflineClientContextThrift(clientContext),\n    displayLocation.toOfflineThrift,\n    candidates.map(_.toOfflineUserThrift)\n  )\n  def toRecommendationRequest: RecommendationRequest = RecommendationRequest(\n    clientContext = clientContext,\n    displayLocation = displayLocation,\n    displayContext = None,\n    maxResults = None,\n    cursor = None,\n    excludedIds = None,\n    fetchPromotedContent = None,\n    debugParams = debugParams,\n    isSoftUser = isSoftUser\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/ScoringUserResponse.scala",
    "content": "package com.twitter.follow_recommendations.models\n\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.logging.{thriftscala => offline}\nimport com.twitter.follow_recommendations.{thriftscala => t}\n\ncase class ScoringUserResponse(candidates: Seq[CandidateUser]) {\n  lazy val toThrift: t.ScoringUserResponse =\n    t.ScoringUserResponse(candidates.map(_.toUserThrift))\n\n  lazy val toRecommendationResponse: RecommendationResponse = RecommendationResponse(candidates)\n\n  lazy val toOfflineThrift: offline.OfflineScoringUserResponse =\n    offline.OfflineScoringUserResponse(candidates.map(_.toOfflineUserThrift))\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/failures/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models/failures/TimeoutPipelineFailure.scala",
    "content": "package com.twitter.follow_recommendations.models.failures\n\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.CandidateSourceTimeout\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\n\nobject TimeoutPipelineFailure {\n  def apply(candidateSourceName: String): PipelineFailure = {\n    PipelineFailure(\n      CandidateSourceTimeout,\n      s\"Candidate Source $candidateSourceName timed out before returning candidates\")\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/modules/ABDeciderModule.scala",
    "content": "package com.twitter.follow_recommendations.modules\n\nimport com.google.inject.Provides\nimport com.google.inject.name.Named\nimport com.twitter.abdecider.ABDeciderFactory\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.inject.TwitterModule\nimport com.twitter.logging.LoggerFactory\nimport javax.inject.Singleton\n\nobject ABDeciderModule extends TwitterModule {\n  @Provides\n  @Singleton\n  def provideABDecider(\n    stats: StatsReceiver,\n    @Named(GuiceNamedConstants.CLIENT_EVENT_LOGGER) factory: LoggerFactory\n  ): LoggingABDecider = {\n\n    val ymlPath = \"/usr/local/config/abdecider/abdecider.yml\"\n\n    val abDeciderFactory = ABDeciderFactory(\n      abDeciderYmlPath = ymlPath,\n      scribeLogger = Some(factory()),\n      environment = Some(\"production\")\n    )\n\n    abDeciderFactory.buildWithLogging()\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/modules/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/constants\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/transforms/modify_social_proof\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry\",\n        \"twml/runtime/src/main/scala/com/twitter/deepbird/runtime/prediction_engine\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/modules/ConfigApiModule.scala",
    "content": "package com.twitter.follow_recommendations.modules\n\nimport com.google.inject.Provides\nimport com.twitter.decider.Decider\nimport com.twitter.follow_recommendations.configapi.ConfigBuilder\nimport com.twitter.inject.TwitterModule\nimport com.twitter.servo.decider.DeciderGateBuilder\nimport com.twitter.timelines.configapi.Config\nimport javax.inject.Singleton\n\nobject ConfigApiModule extends TwitterModule {\n  @Provides\n  @Singleton\n  def providesDeciderGateBuilder(decider: Decider): DeciderGateBuilder =\n    new DeciderGateBuilder(decider)\n\n  @Provides\n  @Singleton\n  def providesConfig(configBuilder: ConfigBuilder): Config = configBuilder.build()\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/modules/DiffyModule.scala",
    "content": "package com.twitter.follow_recommendations.modules\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.decider.RandomRecipient\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient.MtlsThriftMuxClientSyntax\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finatra.annotations.DarkTrafficService\nimport com.twitter.follow_recommendations.configapi.deciders.DeciderKey\nimport com.twitter.follow_recommendations.thriftscala.FollowRecommendationsThriftService\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.thrift.filters.DarkTrafficFilter\nimport com.twitter.servo.decider.DeciderGateBuilder\n\nobject DiffyModule extends TwitterModule {\n  // diffy.dest is defined in the Follow Recommendations Service aurora file\n  // and points to the Dark Traffic Proxy server\n  private val destFlag =\n    flag[String](\"diffy.dest\", \"/$/nil\", \"Resolvable name of diffy-service or proxy\")\n\n  @Provides\n  @Singleton\n  @DarkTrafficService\n  def provideDarkTrafficService(\n    serviceIdentifier: ServiceIdentifier\n  ): FollowRecommendationsThriftService.ReqRepServicePerEndpoint = {\n    ThriftMux.client\n      .withClientId(ClientId(\"follow_recos_service_darktraffic_proxy_client\"))\n      .withMutualTls(serviceIdentifier)\n      .servicePerEndpoint[FollowRecommendationsThriftService.ReqRepServicePerEndpoint](\n        dest = destFlag(),\n        label = \"darktrafficproxy\"\n      )\n  }\n\n  @Provides\n  @Singleton\n  def provideDarkTrafficFilter(\n    @DarkTrafficService darkService: FollowRecommendationsThriftService.ReqRepServicePerEndpoint,\n    deciderGateBuilder: DeciderGateBuilder,\n    statsReceiver: StatsReceiver,\n    @Flag(\"environment\") env: String\n  ): DarkTrafficFilter[FollowRecommendationsThriftService.ReqRepServicePerEndpoint] = {\n    // sampleFunction is used to determine which requests should get replicated\n    // to the dark traffic proxy server\n    val sampleFunction: Any => Boolean = { _ =>\n      // check whether the current FRS instance is deployed in production\n      env match {\n        case \"prod\" =>\n          statsReceiver.scope(\"provideDarkTrafficFilter\").counter(\"prod\").incr()\n          destFlag.isDefined && deciderGateBuilder\n            .keyToFeature(DeciderKey.EnableTrafficDarkReading).isAvailable(RandomRecipient)\n        case _ =>\n          statsReceiver.scope(\"provideDarkTrafficFilter\").counter(\"devel\").incr()\n          // replicate zero requests if in non-production environment\n          false\n      }\n    }\n    new DarkTrafficFilter[FollowRecommendationsThriftService.ReqRepServicePerEndpoint](\n      darkService,\n      sampleFunction,\n      forwardAfterService = true,\n      statsReceiver.scope(\"DarkTrafficFilter\"),\n      lookupByMethod = true\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/modules/FeatureSwitchesModule.scala",
    "content": "package com.twitter.follow_recommendations.modules\n\nimport com.google.inject.Provides\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.featureswitches.v2.Feature\nimport com.twitter.featureswitches.v2.FeatureFilter\nimport com.twitter.featureswitches.v2.FeatureSwitches\nimport com.twitter.featureswitches.v2.builder.FeatureSwitchesBuilder\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants.PRODUCER_SIDE_FEATURE_SWITCHES\nimport com.twitter.inject.TwitterModule\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject FeaturesSwitchesModule extends TwitterModule {\n  private val DefaultConfigRepoPath = \"/usr/local/config\"\n  private val FeaturesPath = \"/features/onboarding/follow-recommendations-service/main\"\n  val isLocal = flag(\"configrepo.local\", false, \"Is the server running locally or in a DC\")\n  val localConfigRepoPath = flag(\n    \"local.configrepo\",\n    System.getProperty(\"user.home\") + \"/workspace/config\",\n    \"Path to your local config repo\"\n  )\n\n  @Provides\n  @Singleton\n  def providesFeatureSwitches(\n    abDecider: LoggingABDecider,\n    statsReceiver: StatsReceiver\n  ): FeatureSwitches = {\n    val configRepoPath = if (isLocal()) {\n      localConfigRepoPath()\n    } else {\n      DefaultConfigRepoPath\n    }\n\n    FeatureSwitchesBuilder\n      .createDefault(FeaturesPath, abDecider, Some(statsReceiver))\n      .configRepoAbsPath(configRepoPath)\n      .serviceDetailsFromAurora()\n      .build()\n  }\n\n  @Provides\n  @Singleton\n  @Named(PRODUCER_SIDE_FEATURE_SWITCHES)\n  def providesProducerFeatureSwitches(\n    abDecider: LoggingABDecider,\n    statsReceiver: StatsReceiver\n  ): FeatureSwitches = {\n    val configRepoPath = if (isLocal()) {\n      localConfigRepoPath()\n    } else {\n      DefaultConfigRepoPath\n    }\n\n    /**\n     * Feature Switches evaluate all tied FS Keys on Params construction time, which is very inefficient\n     * for producer/candidate side holdbacks because we have 100s of candidates, and 100s of FS which result\n     * in 10,000 FS evaluations when we want 1 per candidate (100 total), so we create a new FS Client\n     * which has a [[ProducerFeatureFilter]] set for feature filter to reduce the FS Keys we evaluate.\n     */\n    FeatureSwitchesBuilder\n      .createDefault(FeaturesPath, abDecider, Some(statsReceiver.scope(\"producer_side_fs\")))\n      .configRepoAbsPath(configRepoPath)\n      .serviceDetailsFromAurora()\n      .addFeatureFilter(ProducerFeatureFilter)\n      .build()\n  }\n}\n\ncase object ProducerFeatureFilter extends FeatureFilter {\n  private val AllowedKeys = Set(\n    \"post_nux_ml_flow_candidate_user_scorer_id\",\n    \"frs_receiver_holdback_keep_social_user_candidate\",\n    \"frs_receiver_holdback_keep_user_candidate\")\n\n  override def filter(feature: Feature): Option[Feature] = {\n    if (AllowedKeys.exists(feature.parameters.contains)) {\n      Some(feature)\n    } else {\n      None\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/modules/FlagsModule.scala",
    "content": "package com.twitter.follow_recommendations.modules\nimport com.twitter.inject.TwitterModule\n\nobject FlagsModule extends TwitterModule {\n  flag[Boolean](\n    name = \"fetch_prod_promoted_accounts\",\n    help = \"Whether or not to fetch production promoted accounts (true / false)\"\n  )\n  flag[Boolean](\n    name = \"interests_opt_out_prod_enabled\",\n    help = \"Whether to fetch intersts opt out data from the prod strato column or not\"\n  )\n  flag[Boolean](\n    name = \"log_results\",\n    default = false,\n    help = \"Whether to log results such that we use them for scoring or metrics\"\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/modules/ProductRegistryModule.scala",
    "content": "package com.twitter.follow_recommendations.modules\n\nimport com.twitter.follow_recommendations.products.ProdProductRegistry\nimport com.twitter.follow_recommendations.products.common.ProductRegistry\nimport com.twitter.inject.TwitterModule\nimport javax.inject.Singleton\n\nobject ProductRegistryModule extends TwitterModule {\n  override protected def configure(): Unit = {\n    bind[ProductRegistry].to[ProdProductRegistry].in[Singleton]\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/modules/ScorerModule.scala",
    "content": "package com.twitter.follow_recommendations.modules\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.inject.TwitterModule\nimport com.twitter.relevance.ep_model.common.CommonConstants\nimport com.twitter.relevance.ep_model.scorer.EPScorer\nimport com.twitter.relevance.ep_model.scorer.EPScorerBuilder\nimport java.io.File\nimport java.io.FileOutputStream\nimport scala.language.postfixOps\n\nobject ScorerModule extends TwitterModule {\n  private val STPScorerPath = \"/quality/stp_models/20141223\"\n\n  private def fileFromResource(resource: String): File = {\n    val inputStream = getClass.getResourceAsStream(resource)\n    val file = File.createTempFile(resource, \"temp\")\n    val fos = new FileOutputStream(file)\n    Iterator\n      .continually(inputStream.read)\n      .takeWhile(-1 !=)\n      .foreach(fos.write)\n    file\n  }\n\n  @Provides\n  @Singleton\n  def provideEpScorer: EPScorer = {\n    val modelPath =\n      fileFromResource(STPScorerPath + \"/\" + CommonConstants.EP_MODEL_FILE_NAME).getAbsolutePath\n    val trainingConfigPath =\n      fileFromResource(STPScorerPath + \"/\" + CommonConstants.TRAINING_CONFIG).getAbsolutePath\n    val epScorer = new EPScorerBuilder\n    epScorer\n      .withModelPath(modelPath)\n      .withTrainingConfig(trainingConfigPath)\n      .build()\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/modules/ScribeModule.scala",
    "content": "package com.twitter.follow_recommendations.modules\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.constants.GuiceNamedConstants\nimport com.twitter.inject.TwitterModule\nimport com.twitter.logging.BareFormatter\nimport com.twitter.logging.HandlerFactory\nimport com.twitter.logging.Level\nimport com.twitter.logging.LoggerFactory\nimport com.twitter.logging.NullHandler\nimport com.twitter.logging.QueueingHandler\nimport com.twitter.logging.ScribeHandler\n\nobject ScribeModule extends TwitterModule {\n  val useProdLogger = flag(\n    name = \"scribe.use_prod_loggers\",\n    default = false,\n    help = \"whether to use production logging for service\"\n  )\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.CLIENT_EVENT_LOGGER)\n  def provideClientEventsLoggerFactory(stats: StatsReceiver): LoggerFactory = {\n    val loggerCategory = \"client_event\"\n    val clientEventsHandler: HandlerFactory = if (useProdLogger()) {\n      QueueingHandler(\n        maxQueueSize = 10000,\n        handler = ScribeHandler(\n          category = loggerCategory,\n          formatter = BareFormatter,\n          level = Some(Level.INFO),\n          statsReceiver = stats.scope(\"client_event_scribe\")\n        )\n      )\n    } else { () => NullHandler }\n    LoggerFactory(\n      node = \"abdecider\",\n      level = Some(Level.INFO),\n      useParents = false,\n      handlers = clientEventsHandler :: Nil\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.REQUEST_LOGGER)\n  def provideFollowRecommendationsLoggerFactory(stats: StatsReceiver): LoggerFactory = {\n    val loggerCategory = \"follow_recommendations_logs\"\n    val handlerFactory: HandlerFactory = if (useProdLogger()) {\n      QueueingHandler(\n        maxQueueSize = 10000,\n        handler = ScribeHandler(\n          category = loggerCategory,\n          formatter = BareFormatter,\n          level = Some(Level.INFO),\n          statsReceiver = stats.scope(\"follow_recommendations_logs_scribe\")\n        )\n      )\n    } else { () => NullHandler }\n    LoggerFactory(\n      node = loggerCategory,\n      level = Some(Level.INFO),\n      useParents = false,\n      handlers = handlerFactory :: Nil\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(GuiceNamedConstants.FLOW_LOGGER)\n  def provideFrsRecommendationFlowLoggerFactory(stats: StatsReceiver): LoggerFactory = {\n    val loggerCategory = \"frs_recommendation_flow_logs\"\n    val handlerFactory: HandlerFactory = if (useProdLogger()) {\n      QueueingHandler(\n        maxQueueSize = 10000,\n        handler = ScribeHandler(\n          category = loggerCategory,\n          formatter = BareFormatter,\n          level = Some(Level.INFO),\n          statsReceiver = stats.scope(\"frs_recommendation_flow_logs_scribe\")\n        )\n      )\n    } else { () => NullHandler }\n    LoggerFactory(\n      node = loggerCategory,\n      level = Some(Level.INFO),\n      useParents = false,\n      handlers = handlerFactory :: Nil\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/modules/TimerModule.scala",
    "content": "package com.twitter.follow_recommendations.modules\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.finagle.memcached.ZookeeperStateMonitor.DefaultTimer\nimport com.twitter.inject.TwitterModule\nimport com.twitter.util.Timer\n\nobject TimerModule extends TwitterModule {\n  @Provides\n  @Singleton\n  def providesTimer: Timer = DefaultTimer\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/common\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/explore_tab\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline_tweet_recs\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/sidebar\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/ProdProductRegistry.scala",
    "content": "package com.twitter.follow_recommendations.products\n\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.products.common.ProductRegistry\nimport com.twitter.follow_recommendations.products.explore_tab.ExploreTabProduct\nimport com.twitter.follow_recommendations.products.home_timeline.HomeTimelineProduct\nimport com.twitter.follow_recommendations.products.home_timeline_tweet_recs.HomeTimelineTweetRecsProduct\nimport com.twitter.follow_recommendations.products.sidebar.SidebarProduct\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ProdProductRegistry @Inject() (\n  exploreTabProduct: ExploreTabProduct,\n  homeTimelineProduct: HomeTimelineProduct,\n  homeTimelineTweetRecsProduct: HomeTimelineTweetRecsProduct,\n  sidebarProduct: SidebarProduct,\n) extends ProductRegistry {\n\n  override val products: Seq[common.Product] =\n    Seq(\n      exploreTabProduct,\n      homeTimelineProduct,\n      homeTimelineTweetRecsProduct,\n      sidebarProduct\n    )\n\n  override val displayLocationProductMap: Map[DisplayLocation, common.Product] =\n    products.groupBy(_.displayLocation).flatMap {\n      case (loc, products) =>\n        assert(products.size == 1, s\"Found more than 1 Product for ${loc}\")\n        products.headOption.map { product => loc -> product }\n    }\n\n  override def getProductByDisplayLocation(displayLocation: DisplayLocation): common.Product = {\n    displayLocationProductMap.getOrElse(\n      displayLocation,\n      throw new MissingProductException(displayLocation))\n  }\n}\n\nclass MissingProductException(displayLocation: DisplayLocation)\n    extends Exception(s\"No Product found for ${displayLocation}\")\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/common/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/common/Exceptions.scala",
    "content": "package com.twitter.follow_recommendations.products.common\n\nabstract class ProductException(message: String) extends Exception(message)\n\nclass MissingFieldException(productRequest: ProductRequest, fieldName: String)\n    extends ProductException(\n      s\"Missing ${fieldName} field for ${productRequest.recommendationRequest.displayLocation} request\")\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/common/Product.scala",
    "content": "package com.twitter.follow_recommendations.products.common\n\nimport com.twitter.follow_recommendations.assembler.models.Layout\nimport com.twitter.follow_recommendations.common.base.BaseRecommendationFlow\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.common.models.Recommendation\nimport com.twitter.follow_recommendations.models.RecommendationRequest\nimport com.twitter.product_mixer.core.model.marshalling.request.{Product => ProductMixerProduct}\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Params\n\ntrait Product {\n\n  /** Each product also requires a human-readable name.\n   * You can change this at any time\n   */\n  def name: String\n\n  /**\n   * Every product needs a machine-friendly identifier for internal use.\n   * You should use the same name as the product package name.\n   * Except dashes are better than underscore\n   *\n   * Avoid changing this once it's in production.\n   */\n  def identifier: String\n\n  def displayLocation: DisplayLocation\n\n  def selectWorkflows(\n    request: ProductRequest\n  ): Stitch[Seq[BaseRecommendationFlow[ProductRequest, _ <: Recommendation]]]\n\n  /**\n   * Blender is responsible for blending together the candidates generated by different flows used\n   * in a product. For example, if a product uses two flows, it is blender's responsibility to\n   * interleave their generated candidates together and make a unified sequence of candidates.\n   */\n  def blender: Transform[ProductRequest, Recommendation]\n\n  /**\n   * It is resultsTransformer job to do any final transformations needed on the final list of\n   * candidates generated by a product. For example, if a final quality check on candidates needed,\n   * resultsTransformer will handle it.\n   */\n  def resultsTransformer(request: ProductRequest): Stitch[Transform[ProductRequest, Recommendation]]\n\n  def enabled(request: ProductRequest): Stitch[Boolean]\n\n  def layout: Option[Layout] = None\n\n  def productMixerProduct: Option[ProductMixerProduct] = None\n}\n\ncase class ProductRequest(recommendationRequest: RecommendationRequest, params: Params)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/common/ProductRegistry.scala",
    "content": "package com.twitter.follow_recommendations.products.common\n\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\n\ntrait ProductRegistry {\n  def products: Seq[Product]\n  def displayLocationProductMap: Map[DisplayLocation, Product]\n  def getProductByDisplayLocation(displayLocation: DisplayLocation): Product\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/explore_tab/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/blenders\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/ads\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/common\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/explore_tab/configapi\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/explore_tab/ExploreTabProduct.scala",
    "content": "package com.twitter.follow_recommendations.products.explore_tab\n\nimport com.twitter.follow_recommendations.common.base.BaseRecommendationFlow\nimport com.twitter.follow_recommendations.common.base.IdentityTransform\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.common.models.Recommendation\nimport com.twitter.follow_recommendations.flows.post_nux_ml.PostNuxMlFlow\nimport com.twitter.follow_recommendations.flows.post_nux_ml.PostNuxMlRequestBuilder\nimport com.twitter.follow_recommendations.products.common.Product\nimport com.twitter.follow_recommendations.products.common.ProductRequest\nimport com.twitter.follow_recommendations.products.explore_tab.configapi.ExploreTabParams\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ExploreTabProduct @Inject() (\n  postNuxMlFlow: PostNuxMlFlow,\n  postNuxMlRequestBuilder: PostNuxMlRequestBuilder)\n    extends Product {\n  override val name: String = \"Explore Tab\"\n\n  override val identifier: String = \"explore-tab\"\n\n  override val displayLocation: DisplayLocation = DisplayLocation.ExploreTab\n\n  override def selectWorkflows(\n    request: ProductRequest\n  ): Stitch[Seq[BaseRecommendationFlow[ProductRequest, _ <: Recommendation]]] = {\n    postNuxMlRequestBuilder.build(request).map { postNuxMlRequest =>\n      Seq(postNuxMlFlow.mapKey({ _: ProductRequest => postNuxMlRequest }))\n    }\n  }\n\n  override val blender: Transform[ProductRequest, Recommendation] =\n    new IdentityTransform[ProductRequest, Recommendation]\n\n  override def resultsTransformer(\n    request: ProductRequest\n  ): Stitch[Transform[ProductRequest, Recommendation]] =\n    Stitch.value(new IdentityTransform[ProductRequest, Recommendation])\n\n  override def enabled(request: ProductRequest): Stitch[Boolean] = {\n    // Ideally we should hook up is_soft_user as custom FS field and disable the product through FS\n    val enabledForUserType = !request.recommendationRequest.isSoftUser || request.params(\n      ExploreTabParams.EnableProductForSoftUser)\n    Stitch.value(request.params(ExploreTabParams.EnableProduct) && enabledForUserType)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/explore_tab/configapi/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/explore_tab/configapi/ExploreTabFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.products.explore_tab.configapi\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.follow_recommendations.products.explore_tab.configapi.ExploreTabParams._\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.Param\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ExploreTabFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] =\n    Seq(EnableProductForSoftUser)\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/explore_tab/configapi/ExploreTabParams.scala",
    "content": "package com.twitter.follow_recommendations.products.explore_tab.configapi\n\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelines.configapi.FSParam\n\nobject ExploreTabParams {\n  object EnableProduct extends Param[Boolean](false)\n  object EnableProductForSoftUser\n      extends FSParam[Boolean](\"explore_tab_enable_product_for_soft_user\", false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/blenders\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/ads\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/common\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline/configapi\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline/HTLProductMixer.scala",
    "content": "package com.twitter.follow_recommendations.products.home_timeline\n\nimport com.twitter.product_mixer.core.model.common.identifier.ProductIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\n\ncase object HTLProductMixer extends Product {\n  override val identifier: ProductIdentifier = ProductIdentifier(\"HomeTimeline\")\n  override val stringCenterProject: Option[String] = Some(\"people-discovery\")\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline/HomeTimelineProduct.scala",
    "content": "package com.twitter.follow_recommendations.products.home_timeline\n\nimport com.twitter.follow_recommendations.assembler.models.ActionConfig\nimport com.twitter.follow_recommendations.assembler.models.FollowedByUsersProof\nimport com.twitter.follow_recommendations.assembler.models.FooterConfig\nimport com.twitter.follow_recommendations.assembler.models.GeoContextProof\nimport com.twitter.follow_recommendations.assembler.models.HeaderConfig\nimport com.twitter.follow_recommendations.assembler.models.Layout\nimport com.twitter.follow_recommendations.assembler.models.TitleConfig\nimport com.twitter.follow_recommendations.assembler.models.UserListLayout\nimport com.twitter.follow_recommendations.assembler.models.UserListOptions\nimport com.twitter.follow_recommendations.common.base.BaseRecommendationFlow\nimport com.twitter.follow_recommendations.common.base.IdentityTransform\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.follow_recommendations.flows.ads.PromotedAccountsFlow\nimport com.twitter.follow_recommendations.flows.ads.PromotedAccountsFlowRequest\nimport com.twitter.follow_recommendations.blenders.PromotedAccountsBlender\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.common.models.Recommendation\nimport com.twitter.follow_recommendations.flows.post_nux_ml.PostNuxMlFlow\nimport com.twitter.follow_recommendations.flows.post_nux_ml.PostNuxMlRequestBuilder\nimport com.twitter.follow_recommendations.products.common.Product\nimport com.twitter.follow_recommendations.products.common.ProductRequest\nimport com.twitter.follow_recommendations.products.home_timeline.configapi.HomeTimelineParams._\nimport com.twitter.inject.Injector\nimport com.twitter.product_mixer.core.model.marshalling.request\nimport com.twitter.product_mixer.core.product.guice.ProductScope\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeTimelineProduct @Inject() (\n  postNuxMlFlow: PostNuxMlFlow,\n  postNuxMlRequestBuilder: PostNuxMlRequestBuilder,\n  promotedAccountsFlow: PromotedAccountsFlow,\n  promotedAccountsBlender: PromotedAccountsBlender,\n  productScope: ProductScope,\n  injector: Injector,\n) extends Product {\n\n  override val name: String = \"Home Timeline\"\n\n  override val identifier: String = \"home-timeline\"\n\n  override val displayLocation: DisplayLocation = DisplayLocation.HomeTimeline\n\n  override def selectWorkflows(\n    request: ProductRequest\n  ): Stitch[Seq[BaseRecommendationFlow[ProductRequest, _ <: Recommendation]]] = {\n    postNuxMlRequestBuilder.build(request).map { postNuxMlRequest =>\n      Seq(\n        postNuxMlFlow.mapKey({ request: ProductRequest => postNuxMlRequest }),\n        promotedAccountsFlow.mapKey(mkPromotedAccountsRequest))\n    }\n  }\n\n  override val blender: Transform[ProductRequest, Recommendation] = {\n    promotedAccountsBlender.mapTarget[ProductRequest](getMaxResults)\n  }\n\n  private val identityTransform = new IdentityTransform[ProductRequest, Recommendation]\n\n  override def resultsTransformer(\n    request: ProductRequest\n  ): Stitch[Transform[ProductRequest, Recommendation]] = Stitch.value(identityTransform)\n\n  override def enabled(request: ProductRequest): Stitch[Boolean] =\n    Stitch.value(request.params(EnableProduct))\n\n  override def layout: Option[Layout] = {\n    productMixerProduct.map { product =>\n      val homeTimelineStrings = productScope.let(product) {\n        injector.instance[HomeTimelineStrings]\n      }\n      UserListLayout(\n        header = Some(HeaderConfig(TitleConfig(homeTimelineStrings.whoToFollowModuleTitle))),\n        userListOptions = UserListOptions(userBioEnabled = true, userBioTruncated = true, None),\n        socialProofs = Some(\n          Seq(\n            FollowedByUsersProof(\n              homeTimelineStrings.whoToFollowFollowedByManyUserSingleString,\n              homeTimelineStrings.whoToFollowFollowedByManyUserDoubleString,\n              homeTimelineStrings.whoToFollowFollowedByManyUserMultipleString\n            ),\n            GeoContextProof(homeTimelineStrings.whoToFollowPopularInCountryKey)\n          )),\n        footer = Some(\n          FooterConfig(\n            Some(ActionConfig(homeTimelineStrings.whoToFollowModuleFooter, \"http://twitter.com\"))))\n      )\n    }\n  }\n\n  override def productMixerProduct: Option[request.Product] = Some(HTLProductMixer)\n\n  private[home_timeline] def mkPromotedAccountsRequest(\n    req: ProductRequest\n  ): PromotedAccountsFlowRequest = {\n    PromotedAccountsFlowRequest(\n      req.recommendationRequest.clientContext,\n      req.params,\n      req.recommendationRequest.displayLocation,\n      None,\n      req.recommendationRequest.excludedIds.getOrElse(Nil)\n    )\n  }\n\n  private[home_timeline] def getMaxResults(req: ProductRequest): Int = {\n    req.recommendationRequest.maxResults.getOrElse(\n      req.params(DefaultMaxResults)\n    )\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline/HomeTimelineStrings.scala",
    "content": "package com.twitter.follow_recommendations.products.home_timeline\n\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.ExternalStringRegistry\nimport com.twitter.stringcenter.client.core.ExternalString\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeTimelineStrings @Inject() (\n  @ProductScoped externalStringRegistryProvider: Provider[ExternalStringRegistry]) {\n  private val externalStringRegistry = externalStringRegistryProvider.get()\n  val whoToFollowFollowedByManyUserSingleString: ExternalString =\n    externalStringRegistry.createProdString(\"WtfRecommendationContext.followedByManyUserSingle\")\n  val whoToFollowFollowedByManyUserDoubleString: ExternalString =\n    externalStringRegistry.createProdString(\"WtfRecommendationContext.followedByManyUserDouble\")\n  val whoToFollowFollowedByManyUserMultipleString: ExternalString =\n    externalStringRegistry.createProdString(\"WtfRecommendationContext.followedByManyUserMultiple\")\n  val whoToFollowPopularInCountryKey: ExternalString =\n    externalStringRegistry.createProdString(\"WtfRecommendationContext.popularInCountry\")\n  val whoToFollowModuleTitle: ExternalString =\n    externalStringRegistry.createProdString(\"WhoToFollowModule.title\")\n  val whoToFollowModuleFooter: ExternalString =\n    externalStringRegistry.createProdString(\"WhoToFollowModule.pivot\")\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline/configapi/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/common\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline/configapi/HomeTimelineFSConfig.scala",
    "content": "package com.twitter.follow_recommendations.products.home_timeline.configapi\n\nimport com.twitter.follow_recommendations.configapi.common.FeatureSwitchConfig\nimport com.twitter.follow_recommendations.products.home_timeline.configapi.HomeTimelineParams._\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeTimelineFSConfig @Inject() () extends FeatureSwitchConfig {\n  override val booleanFSParams: Seq[Param[Boolean] with FSName] =\n    Seq(EnableWritingServingHistory)\n\n  override val durationFSParams: Seq[FSBoundedParam[Duration] with HasDurationConversion] = Seq(\n    DurationGuardrailToForceSuggest,\n    SuggestBasedFatigueDuration\n  )\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline/configapi/HomeTimelineParams.scala",
    "content": "package com.twitter.follow_recommendations.products.home_timeline.configapi\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\nobject HomeTimelineParams {\n  object EnableProduct extends Param[Boolean](false)\n\n  object DefaultMaxResults extends Param[Int](20)\n\n  object EnableWritingServingHistory\n      extends FSParam[Boolean](\"home_timeline_enable_writing_serving_history\", false)\n\n  object DurationGuardrailToForceSuggest\n      extends FSBoundedParam[Duration](\n        name = \"home_timeline_duration_guardrail_to_force_suggest_in_hours\",\n        default = 0.hours,\n        min = 0.hours,\n        max = 1000.hours)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromHours\n  }\n\n  object SuggestBasedFatigueDuration\n      extends FSBoundedParam[Duration](\n        name = \"home_timeline_suggest_based_fatigue_duration_in_hours\",\n        default = 0.hours,\n        min = 0.hours,\n        max = 1000.hours)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromHours\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline_tweet_recs/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/blenders\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/content_recommender_flow\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/common\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline_tweet_recs/configapi\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline_tweet_recs/HomeTimelineTweetRecsProduct.scala",
    "content": "package com.twitter.follow_recommendations.products.home_timeline_tweet_recs\n\nimport com.twitter.follow_recommendations.common.base.BaseRecommendationFlow\nimport com.twitter.follow_recommendations.common.base.IdentityTransform\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.common.models.Recommendation\nimport com.twitter.follow_recommendations.flows.content_recommender_flow.ContentRecommenderFlow\nimport com.twitter.follow_recommendations.flows.content_recommender_flow.ContentRecommenderRequestBuilder\nimport com.twitter.follow_recommendations.products.common.Product\nimport com.twitter.follow_recommendations.products.common.ProductRequest\nimport com.twitter.follow_recommendations.products.home_timeline_tweet_recs.configapi.HomeTimelineTweetRecsParams._\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/*\n * This \"DisplayLocation\" is used to generate user recommendations using the ContentRecommenderFlow. These recommendations are later used downstream\n * to generate recommended tweets on Home Timeline.\n */\n@Singleton\nclass HomeTimelineTweetRecsProduct @Inject() (\n  contentRecommenderFlow: ContentRecommenderFlow,\n  contentRecommenderRequestBuilder: ContentRecommenderRequestBuilder)\n    extends Product {\n  override val name: String = \"Home Timeline Tweet Recs\"\n\n  override val identifier: String = \"home-timeline-tweet-recs\"\n\n  override val displayLocation: DisplayLocation = DisplayLocation.HomeTimelineTweetRecs\n\n  override def selectWorkflows(\n    request: ProductRequest\n  ): Stitch[Seq[BaseRecommendationFlow[ProductRequest, _ <: Recommendation]]] = {\n    contentRecommenderRequestBuilder.build(request).map { contentRecommenderRequest =>\n      Seq(contentRecommenderFlow.mapKey({ request: ProductRequest => contentRecommenderRequest }))\n    }\n  }\n\n  override val blender: Transform[ProductRequest, Recommendation] =\n    new IdentityTransform[ProductRequest, Recommendation]\n\n  override def resultsTransformer(\n    request: ProductRequest\n  ): Stitch[Transform[ProductRequest, Recommendation]] =\n    Stitch.value(new IdentityTransform[ProductRequest, Recommendation])\n\n  override def enabled(request: ProductRequest): Stitch[Boolean] =\n    Stitch.value(request.params(EnableProduct))\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline_tweet_recs/configapi/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core\",\n        \"configapi/configapi-decider\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/common\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/home_timeline_tweet_recs/configapi/HomeTimelineTweetRecsParams.scala",
    "content": "package com.twitter.follow_recommendations.products.home_timeline_tweet_recs.configapi\n\nimport com.twitter.timelines.configapi.Param\n\nobject HomeTimelineTweetRecsParams {\n  object EnableProduct extends Param[Boolean](false)\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/sidebar/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/blenders\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/ads\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/flows/post_nux_ml\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/common\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/sidebar/configapi\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/sidebar/SidebarProduct.scala",
    "content": "package com.twitter.follow_recommendations.products.sidebar\n\nimport com.twitter.follow_recommendations.common.base.BaseRecommendationFlow\nimport com.twitter.follow_recommendations.common.base.IdentityTransform\nimport com.twitter.follow_recommendations.common.base.Transform\nimport com.twitter.follow_recommendations.flows.ads.PromotedAccountsFlow\nimport com.twitter.follow_recommendations.flows.ads.PromotedAccountsFlowRequest\nimport com.twitter.follow_recommendations.blenders.PromotedAccountsBlender\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.follow_recommendations.common.models.Recommendation\nimport com.twitter.follow_recommendations.flows.post_nux_ml.PostNuxMlFlow\nimport com.twitter.follow_recommendations.flows.post_nux_ml.PostNuxMlRequestBuilder\nimport com.twitter.follow_recommendations.products.common.Product\nimport com.twitter.follow_recommendations.products.common.ProductRequest\nimport com.twitter.follow_recommendations.products.sidebar.configapi.SidebarParams\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SidebarProduct @Inject() (\n  postNuxMlFlow: PostNuxMlFlow,\n  postNuxMlRequestBuilder: PostNuxMlRequestBuilder,\n  promotedAccountsFlow: PromotedAccountsFlow,\n  promotedAccountsBlender: PromotedAccountsBlender)\n    extends Product {\n  override val name: String = \"Sidebar\"\n\n  override val identifier: String = \"sidebar\"\n\n  override val displayLocation: DisplayLocation = DisplayLocation.Sidebar\n\n  override def selectWorkflows(\n    request: ProductRequest\n  ): Stitch[Seq[BaseRecommendationFlow[ProductRequest, _ <: Recommendation]]] = {\n    postNuxMlRequestBuilder.build(request).map { postNuxMlRequest =>\n      Seq(\n        postNuxMlFlow.mapKey({ _: ProductRequest => postNuxMlRequest }),\n        promotedAccountsFlow.mapKey(mkPromotedAccountsRequest)\n      )\n    }\n  }\n\n  override val blender: Transform[ProductRequest, Recommendation] = {\n    promotedAccountsBlender.mapTarget[ProductRequest](getMaxResults)\n  }\n\n  private[sidebar] def mkPromotedAccountsRequest(\n    req: ProductRequest\n  ): PromotedAccountsFlowRequest = {\n    PromotedAccountsFlowRequest(\n      req.recommendationRequest.clientContext,\n      req.params,\n      req.recommendationRequest.displayLocation,\n      None,\n      req.recommendationRequest.excludedIds.getOrElse(Nil)\n    )\n  }\n\n  private[sidebar] def getMaxResults(req: ProductRequest): Int = {\n    req.recommendationRequest.maxResults.getOrElse(\n      req.params(SidebarParams.DefaultMaxResults)\n    )\n  }\n\n  override def resultsTransformer(\n    request: ProductRequest\n  ): Stitch[Transform[ProductRequest, Recommendation]] =\n    Stitch.value(new IdentityTransform[ProductRequest, Recommendation])\n\n  override def enabled(request: ProductRequest): Stitch[Boolean] =\n    Stitch.value(request.params(SidebarParams.EnableProduct))\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/sidebar/configapi/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products/sidebar/configapi/SidebarParams.scala",
    "content": "package com.twitter.follow_recommendations.products.sidebar.configapi\n\nimport com.twitter.timelines.configapi.Param\n\nobject SidebarParams {\n  object EnableProduct extends Param[Boolean](false)\n\n  object DefaultMaxResults extends Param[Int](20)\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-server/src/main/scala\",\n        \"finatra/inject/inject-thrift-client\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/crowd_search_accounts\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/real_graph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/recent_engagement\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/top_organic_follows_accounts\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/clients/impression_store\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/predicates/sgs\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/ranking\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/rankers/ml_ranker/scoring\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/utils\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/assembler/models\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/deciders\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/logging\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/products\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry\",\n        \"twitter-server/server/src/main/scala\",\n        \"util/util-app/src/main/scala\",\n        \"util/util-core:scala\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services/FollowRecommendationsServiceWarmupHandler.scala",
    "content": "package com.twitter.follow_recommendations.services\n\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finatra.thrift.routing.ThriftWarmup\nimport com.twitter.follow_recommendations.thriftscala.FollowRecommendationsThriftService.GetRecommendations\nimport com.twitter.follow_recommendations.thriftscala.ClientContext\nimport com.twitter.follow_recommendations.thriftscala.DebugParams\nimport com.twitter.follow_recommendations.thriftscala.DisplayContext\nimport com.twitter.follow_recommendations.thriftscala.DisplayLocation\nimport com.twitter.follow_recommendations.thriftscala.Profile\nimport com.twitter.follow_recommendations.thriftscala.RecommendationRequest\nimport com.twitter.inject.Logging\nimport com.twitter.inject.utils.Handler\nimport com.twitter.scrooge.Request\nimport com.twitter.scrooge.Response\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FollowRecommendationsServiceWarmupHandler @Inject() (warmup: ThriftWarmup)\n    extends Handler\n    with Logging {\n\n  private val clientId = ClientId(\"thrift-warmup-client\")\n\n  override def handle(): Unit = {\n    val testIds = Seq(1L)\n    def warmupQuery(userId: Long, displayLocation: DisplayLocation): RecommendationRequest = {\n      val clientContext = ClientContext(\n        userId = Some(userId),\n        guestId = None,\n        appId = Some(258901L),\n        ipAddress = Some(\"0.0.0.0\"),\n        userAgent = Some(\"FAKE_USER_AGENT_FOR_WARMUPS\"),\n        countryCode = Some(\"US\"),\n        languageCode = Some(\"en\"),\n        isTwoffice = None,\n        userRoles = None,\n        deviceId = Some(\"FAKE_DEVICE_ID_FOR_WARMUPS\")\n      )\n      RecommendationRequest(\n        clientContext = clientContext,\n        displayLocation = displayLocation,\n        displayContext = None,\n        maxResults = Some(3),\n        fetchPromotedContent = Some(false),\n        debugParams = Some(DebugParams(doNotLog = Some(true)))\n      )\n    }\n\n    // Add FRS display locations here if they should be targeted for warm-up\n    // when FRS is starting from a fresh state after a deploy\n    val displayLocationsToWarmUp: Seq[DisplayLocation] = Seq(\n      DisplayLocation.HomeTimeline,\n      DisplayLocation.HomeTimelineReverseChron,\n      DisplayLocation.ProfileSidebar,\n      DisplayLocation.NuxInterests,\n      DisplayLocation.NuxPymk\n    )\n\n    try {\n      clientId.asCurrent {\n        // Iterate over each user ID created for testing\n        testIds foreach { id =>\n          // Iterate over each display location targeted for warm-up\n          displayLocationsToWarmUp foreach { displayLocation =>\n            val warmupReq = warmupQuery(id, displayLocation)\n            info(s\"Sending warm-up request to service with query: $warmupReq\")\n            warmup.sendRequest(\n              method = GetRecommendations,\n              req = Request(GetRecommendations.Args(warmupReq)))(assertWarmupResponse)\n            // send the request one more time so that it goes through cache hits\n            warmup.sendRequest(\n              method = GetRecommendations,\n              req = Request(GetRecommendations.Args(warmupReq)))(assertWarmupResponse)\n          }\n        }\n      }\n    } catch {\n      case e: Throwable =>\n        // we don't want a warmup failure to prevent start-up\n        error(e.getMessage, e)\n    }\n    info(\"Warm-up done.\")\n  }\n\n  /* Private */\n\n  private def assertWarmupResponse(result: Try[Response[GetRecommendations.SuccessType]]): Unit = {\n    // we collect and log any exceptions from the result.\n    result match {\n      case Return(_) => // ok\n      case Throw(exception) =>\n        warn()\n        error(s\"Error performing warm-up request: ${exception.getMessage}\", exception)\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services/ProductMixerRecommendationService.scala",
    "content": "package com.twitter.follow_recommendations.services\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.follow_recommendations.common.utils.DisplayLocationProductConverterUtil\nimport com.twitter.follow_recommendations.configapi.deciders.DeciderParams\nimport com.twitter.follow_recommendations.logging.FrsLogger\nimport com.twitter.follow_recommendations.models.{DebugParams => FrsDebugParams}\nimport com.twitter.follow_recommendations.models.RecommendationRequest\nimport com.twitter.follow_recommendations.models.RecommendationResponse\nimport com.twitter.follow_recommendations.models.Request\nimport com.twitter.product_mixer.core.model.marshalling.request.{\n  DebugParams => ProductMixerDebugParams\n}\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineRequest\nimport com.twitter.stitch.Stitch\n\n@Singleton\nclass ProductMixerRecommendationService @Inject() (\n  productPipelineRegistry: ProductPipelineRegistry,\n  resultLogger: FrsLogger,\n  baseStats: StatsReceiver) {\n\n  private val stats = baseStats.scope(\"product_mixer_recos_service_stats\")\n  private val loggingStats = stats.scope(\"logged\")\n\n  def get(request: RecommendationRequest, params: Params): Stitch[RecommendationResponse] = {\n    if (params(DeciderParams.EnableRecommendations)) {\n      val productMixerRequest = convertToProductMixerRequest(request)\n\n      productPipelineRegistry\n        .getProductPipeline[Request, RecommendationResponse](productMixerRequest.product)\n        .process(ProductPipelineRequest(productMixerRequest, params)).onSuccess { response =>\n          if (resultLogger.shouldLog(request.debugParams)) {\n            loggingStats.counter().incr()\n            resultLogger.logRecommendationResult(request, response)\n          }\n        }\n    } else {\n      Stitch.value(RecommendationResponse(Nil))\n    }\n\n  }\n\n  def convertToProductMixerRequest(frsRequest: RecommendationRequest): Request = {\n    Request(\n      maxResults = frsRequest.maxResults,\n      debugParams = convertToProductMixerDebugParams(frsRequest.debugParams),\n      productContext = None,\n      product =\n        DisplayLocationProductConverterUtil.displayLocationToProduct(frsRequest.displayLocation),\n      clientContext = frsRequest.clientContext,\n      serializedRequestCursor = frsRequest.cursor,\n      frsDebugParams = frsRequest.debugParams,\n      displayLocation = frsRequest.displayLocation,\n      excludedIds = frsRequest.excludedIds,\n      fetchPromotedContent = frsRequest.fetchPromotedContent,\n      userLocationState = frsRequest.userLocationState\n    )\n  }\n\n  private def convertToProductMixerDebugParams(\n    frsDebugParams: Option[FrsDebugParams]\n  ): Option[ProductMixerDebugParams] = {\n    frsDebugParams.map { debugParams =>\n      ProductMixerDebugParams(debugParams.featureOverrides, None)\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services/ProductPipelineSelector.scala",
    "content": "package com.twitter.follow_recommendations.services\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.common.models.DebugOptions\nimport com.twitter.follow_recommendations.models.DebugParams\nimport com.twitter.follow_recommendations.models.RecommendationRequest\nimport com.twitter.follow_recommendations.models.RecommendationResponse\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Params\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.util.Random\n\n@Singleton\nclass ProductPipelineSelector @Inject() (\n  recommendationsService: RecommendationsService,\n  productMixerRecommendationService: ProductMixerRecommendationService,\n  productPipelineSelectorConfig: ProductPipelineSelectorConfig,\n  baseStats: StatsReceiver) {\n\n  private val frsStats = baseStats.scope(\"follow_recommendations_service\")\n  private val stats = frsStats.scope(\"product_pipeline_selector_parity\")\n\n  private val readFromProductMixerCounter = stats.counter(\"select_product_mixer\")\n  private val readFromOldFRSCounter = stats.counter(\"select_old_frs\")\n\n  def selectPipeline(\n    request: RecommendationRequest,\n    params: Params\n  ): Stitch[RecommendationResponse] = {\n    productPipelineSelectorConfig\n      .getDarkReadAndExpParams(request.displayLocation).map { darkReadAndExpParam =>\n        if (params(darkReadAndExpParam.expParam)) {\n          readFromProductMixerPipeline(request, params)\n        } else if (params(darkReadAndExpParam.darkReadParam)) {\n          darkReadAndReturnResult(request, params)\n        } else {\n          readFromOldFrsPipeline(request, params)\n        }\n      }.getOrElse(readFromOldFrsPipeline(request, params))\n  }\n\n  private def readFromProductMixerPipeline(\n    request: RecommendationRequest,\n    params: Params\n  ): Stitch[RecommendationResponse] = {\n    readFromProductMixerCounter.incr()\n    productMixerRecommendationService.get(request, params)\n  }\n\n  private def readFromOldFrsPipeline(\n    request: RecommendationRequest,\n    params: Params\n  ): Stitch[RecommendationResponse] = {\n    readFromOldFRSCounter.incr()\n    recommendationsService.get(request, params)\n  }\n\n  private def darkReadAndReturnResult(\n    request: RecommendationRequest,\n    params: Params\n  ): Stitch[RecommendationResponse] = {\n    val darkReadStats = stats.scope(\"select_dark_read\", request.displayLocation.toFsName)\n    darkReadStats.counter(\"count\").incr()\n\n    // If no seed is set, create a random one that both requests will use to remove differences\n    // in randomness for the WeightedCandidateSourceRanker\n    val randomizationSeed = new Random().nextLong()\n\n    val oldFRSPiplelineRequest = request.copy(\n      debugParams = Some(\n        request.debugParams.getOrElse(\n          DebugParams(None, Some(DebugOptions(randomizationSeed = Some(randomizationSeed))))))\n    )\n    val productMixerPipelineRequest = request.copy(\n      debugParams = Some(\n        request.debugParams.getOrElse(\n          DebugParams(\n            None,\n            Some(DebugOptions(doNotLog = true, randomizationSeed = Some(randomizationSeed))))))\n    )\n\n    StatsUtil\n      .profileStitch(\n        readFromOldFrsPipeline(oldFRSPiplelineRequest, params),\n        darkReadStats.scope(\"frs_timing\")).applyEffect { frsOldPipelineResponse =>\n        Stitch.async(\n          StatsUtil\n            .profileStitch(\n              readFromProductMixerPipeline(productMixerPipelineRequest, params),\n              darkReadStats.scope(\"product_mixer_timing\")).liftToOption().map {\n              case Some(frsProductMixerResponse) =>\n                darkReadStats.counter(\"product_mixer_pipeline_success\").incr()\n                compare(request, frsOldPipelineResponse, frsProductMixerResponse)\n              case None =>\n                darkReadStats.counter(\"product_mixer_pipeline_failure\").incr()\n            }\n        )\n      }\n  }\n\n  def compare(\n    request: RecommendationRequest,\n    frsOldPipelineResponse: RecommendationResponse,\n    frsProductMixerResponse: RecommendationResponse\n  ): Unit = {\n    val compareStats = stats.scope(\"pipeline_comparison\", request.displayLocation.toFsName)\n    compareStats.counter(\"total-comparisons\").incr()\n\n    val oldFrsMap = frsOldPipelineResponse.recommendations.map { user => user.id -> user }.toMap\n    val productMixerMap = frsProductMixerResponse.recommendations.map { user =>\n      user.id -> user\n    }.toMap\n\n    compareTopNResults(3, frsOldPipelineResponse, frsProductMixerResponse, compareStats)\n    compareTopNResults(5, frsOldPipelineResponse, frsProductMixerResponse, compareStats)\n    compareTopNResults(25, frsOldPipelineResponse, frsProductMixerResponse, compareStats)\n    compareTopNResults(50, frsOldPipelineResponse, frsProductMixerResponse, compareStats)\n    compareTopNResults(75, frsOldPipelineResponse, frsProductMixerResponse, compareStats)\n\n    // Compare individual matching candidates\n    oldFrsMap.keys.foreach(userId => {\n      if (productMixerMap.contains(userId)) {\n        (oldFrsMap(userId), productMixerMap(userId)) match {\n          case (oldFrsUser: CandidateUser, productMixerUser: CandidateUser) =>\n            compareStats.counter(\"matching-user-count\").incr()\n            compareUser(oldFrsUser, productMixerUser, compareStats)\n          case _ =>\n            compareStats.counter(\"unknown-user-type-count\").incr()\n        }\n      } else {\n        compareStats.counter(\"missing-user-count\").incr()\n      }\n    })\n  }\n\n  private def compareTopNResults(\n    n: Int,\n    frsOldPipelineResponse: RecommendationResponse,\n    frsProductMixerResponse: RecommendationResponse,\n    compareStats: StatsReceiver\n  ): Unit = {\n    if (frsOldPipelineResponse.recommendations.size >= n && frsProductMixerResponse.recommendations.size >= n) {\n      val oldFrsPipelineFirstN = frsOldPipelineResponse.recommendations.take(n).map(_.id)\n      val productMixerPipelineFirstN = frsProductMixerResponse.recommendations.take(n).map(_.id)\n\n      if (oldFrsPipelineFirstN.sorted == productMixerPipelineFirstN.sorted)\n        compareStats.counter(s\"first-$n-sorted-equal-ids\").incr()\n      if (oldFrsPipelineFirstN == productMixerPipelineFirstN)\n        compareStats.counter(s\"first-$n-unsorted-ids-equal\").incr()\n      else\n        compareStats.counter(s\"first-$n-unsorted-ids-unequal\").incr()\n    }\n  }\n\n  private def compareUser(\n    oldFrsUser: CandidateUser,\n    productMixerUser: CandidateUser,\n    stats: StatsReceiver\n  ): Unit = {\n    val userStats = stats.scope(\"matching-user\")\n\n    if (oldFrsUser.score != productMixerUser.score)\n      userStats.counter(\"mismatch-score\").incr()\n    if (oldFrsUser.reason != productMixerUser.reason)\n      userStats.counter(\"mismatch-reason\").incr()\n    if (oldFrsUser.userCandidateSourceDetails != productMixerUser.userCandidateSourceDetails)\n      userStats.counter(\"mismatch-userCandidateSourceDetails\").incr()\n    if (oldFrsUser.adMetadata != productMixerUser.adMetadata)\n      userStats.counter(\"mismatch-adMetadata\").incr()\n    if (oldFrsUser.trackingToken != productMixerUser.trackingToken)\n      userStats.counter(\"mismatch-trackingToken\").incr()\n    if (oldFrsUser.dataRecord != productMixerUser.dataRecord)\n      userStats.counter(\"mismatch-dataRecord\").incr()\n    if (oldFrsUser.scores != productMixerUser.scores)\n      userStats.counter(\"mismatch-scores\").incr()\n    if (oldFrsUser.infoPerRankingStage != productMixerUser.infoPerRankingStage)\n      userStats.counter(\"mismatch-infoPerRankingStage\").incr()\n    if (oldFrsUser.params != productMixerUser.params)\n      userStats.counter(\"mismatch-params\").incr()\n    if (oldFrsUser.engagements != productMixerUser.engagements)\n      userStats.counter(\"mismatch-engagements\").incr()\n    if (oldFrsUser.recommendationFlowIdentifier != productMixerUser.recommendationFlowIdentifier)\n      userStats.counter(\"mismatch-recommendationFlowIdentifier\").incr()\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services/ProductPipelineSelectorConfig.scala",
    "content": "package com.twitter.follow_recommendations.services\n\nimport com.twitter.follow_recommendations.common.models.DisplayLocation\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\nimport javax.inject.Singleton\n\n@Singleton\nclass ProductPipelineSelectorConfig {\n  private val paramsMap: Map[DisplayLocation, DarkReadAndExpParams] = Map.empty\n\n  def getDarkReadAndExpParams(\n    displayLocation: DisplayLocation\n  ): Option[DarkReadAndExpParams] = {\n    paramsMap.get(displayLocation)\n  }\n}\n\ncase class DarkReadAndExpParams(darkReadParam: Param[Boolean], expParam: FSParam[Boolean])\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services/ProductRecommenderService.scala",
    "content": "package com.twitter.follow_recommendations.services\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.StatsUtil\nimport com.twitter.follow_recommendations.common.models.Recommendation\nimport com.twitter.follow_recommendations.models.RecommendationRequest\nimport com.twitter.follow_recommendations.products.common.ProductRegistry\nimport com.twitter.follow_recommendations.products.common.ProductRequest\nimport com.twitter.stitch.Stitch\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams.EnableWhoToFollowProducts\nimport com.twitter.timelines.configapi.Params\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ProductRecommenderService @Inject() (\n  productRegistry: ProductRegistry,\n  statsReceiver: StatsReceiver) {\n\n  private val stats = statsReceiver.scope(\"ProductRecommenderService\")\n\n  def getRecommendations(\n    request: RecommendationRequest,\n    params: Params\n  ): Stitch[Seq[Recommendation]] = {\n    val displayLocation = request.displayLocation\n    val displayLocationStatName = displayLocation.toString\n    val locationStats = stats.scope(displayLocationStatName)\n    val loggedInOrOutStats = if (request.clientContext.userId.isDefined) {\n      stats.scope(\"logged_in\").scope(displayLocationStatName)\n    } else {\n      stats.scope(\"logged_out\").scope(displayLocationStatName)\n    }\n\n    loggedInOrOutStats.counter(\"requests\").incr()\n    val product = productRegistry.getProductByDisplayLocation(displayLocation)\n    val productRequest = ProductRequest(request, params)\n    val productEnabledStitch =\n      StatsUtil.profileStitch(product.enabled(productRequest), locationStats.scope(\"enabled\"))\n    productEnabledStitch.flatMap { productEnabled =>\n      if (productEnabled && params(EnableWhoToFollowProducts)) {\n        loggedInOrOutStats.counter(\"enabled\").incr()\n        val stitch = for {\n          workflows <- StatsUtil.profileStitch(\n            product.selectWorkflows(productRequest),\n            locationStats.scope(\"select_workflows\"))\n          workflowRecos <- StatsUtil.profileStitch(\n            Stitch.collect(\n              workflows.map(_.process(productRequest).map(_.result.getOrElse(Seq.empty)))),\n            locationStats.scope(\"execute_workflows\")\n          )\n          blendedCandidates <- StatsUtil.profileStitch(\n            product.blender.transform(productRequest, workflowRecos.flatten),\n            locationStats.scope(\"blend_results\"))\n          resultsTransformer <- StatsUtil.profileStitch(\n            product.resultsTransformer(productRequest),\n            locationStats.scope(\"results_transformer\"))\n          transformedCandidates <- StatsUtil.profileStitch(\n            resultsTransformer.transform(productRequest, blendedCandidates),\n            locationStats.scope(\"execute_results_transformer\"))\n        } yield {\n          transformedCandidates\n        }\n        StatsUtil.profileStitchResults[Seq[Recommendation]](stitch, locationStats, _.size)\n      } else {\n        loggedInOrOutStats.counter(\"disabled\").incr()\n        locationStats.counter(\"disabled_product\").incr()\n        Stitch.Nil\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services/RecommendationsService.scala",
    "content": "package com.twitter.follow_recommendations.services\n\nimport com.twitter.follow_recommendations.configapi.deciders.DeciderParams\nimport com.twitter.follow_recommendations.logging.FrsLogger\nimport com.twitter.follow_recommendations.models.RecommendationRequest\nimport com.twitter.follow_recommendations.models.RecommendationResponse\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Params\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RecommendationsService @Inject() (\n  productRecommenderService: ProductRecommenderService,\n  resultLogger: FrsLogger) {\n  def get(request: RecommendationRequest, params: Params): Stitch[RecommendationResponse] = {\n    if (params(DeciderParams.EnableRecommendations)) {\n      productRecommenderService\n        .getRecommendations(request, params).map(RecommendationResponse).onSuccess { response =>\n          if (resultLogger.shouldLog(request.debugParams)) {\n            resultLogger.logRecommendationResult(request, response)\n          }\n        }\n    } else {\n      Stitch.value(RecommendationResponse(Nil))\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services/UserScoringService.scala",
    "content": "package com.twitter.follow_recommendations.services\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.common.base.StatsUtil.profileStitchSeqResults\nimport com.twitter.follow_recommendations.common.clients.impression_store.WtfImpressionStore\nimport com.twitter.follow_recommendations.common.clients.socialgraph.SocialGraphClient\nimport com.twitter.follow_recommendations.common.rankers.ml_ranker.ranking.HydrateFeaturesTransform\nimport com.twitter.follow_recommendations.common.rankers.ml_ranker.ranking.MlRanker\nimport com.twitter.follow_recommendations.common.utils.RescueWithStatsUtils.rescueWithStats\nimport com.twitter.follow_recommendations.configapi.deciders.DeciderParams\nimport com.twitter.follow_recommendations.logging.FrsLogger\nimport com.twitter.follow_recommendations.models.ScoringUserRequest\nimport com.twitter.follow_recommendations.models.ScoringUserResponse\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserScoringService @Inject() (\n  socialGraph: SocialGraphClient,\n  wtfImpressionStore: WtfImpressionStore,\n  hydrateFeaturesTransform: HydrateFeaturesTransform[ScoringUserRequest],\n  mlRanker: MlRanker[ScoringUserRequest],\n  resultLogger: FrsLogger,\n  stats: StatsReceiver) {\n\n  private val scopedStats: StatsReceiver = stats.scope(this.getClass.getSimpleName)\n  private val disabledCounter: Counter = scopedStats.counter(\"disabled\")\n\n  def get(request: ScoringUserRequest): Stitch[ScoringUserResponse] = {\n    if (request.params(DeciderParams.EnableScoreUserCandidates)) {\n      val hydratedRequest = hydrate(request)\n      val candidatesStitch = hydratedRequest.flatMap { req =>\n        hydrateFeaturesTransform.transform(req, request.candidates).flatMap {\n          candidateWithFeatures =>\n            mlRanker.rank(req, candidateWithFeatures)\n        }\n      }\n      profileStitchSeqResults(candidatesStitch, scopedStats)\n        .map(ScoringUserResponse)\n        .onSuccess { response =>\n          if (resultLogger.shouldLog(request.debugParams)) {\n            resultLogger.logScoringResult(request, response)\n          }\n        }\n    } else {\n      disabledCounter.incr()\n      Stitch.value(ScoringUserResponse(Nil))\n    }\n  }\n\n  private def hydrate(request: ScoringUserRequest): Stitch[ScoringUserRequest] = {\n    val allStitches = Stitch.collect(request.clientContext.userId.map { userId =>\n      val recentFollowedUserIdsStitch =\n        rescueWithStats(\n          socialGraph.getRecentFollowedUserIds(userId),\n          stats,\n          \"recentFollowedUserIds\")\n      val recentFollowedByUserIdsStitch =\n        rescueWithStats(\n          socialGraph.getRecentFollowedByUserIds(userId),\n          stats,\n          \"recentFollowedByUserIds\")\n      val wtfImpressionsStitch =\n        rescueWithStats(\n          wtfImpressionStore.get(userId, request.displayLocation),\n          stats,\n          \"wtfImpressions\")\n      Stitch.join(recentFollowedUserIdsStitch, recentFollowedByUserIdsStitch, wtfImpressionsStitch)\n    })\n    allStitches.map {\n      case Some((recentFollowedUserIds, recentFollowedByUserIds, wtfImpressions)) =>\n        request.copy(\n          recentFollowedUserIds =\n            if (recentFollowedUserIds.isEmpty) None else Some(recentFollowedUserIds),\n          recentFollowedByUserIds =\n            if (recentFollowedByUserIds.isEmpty) None else Some(recentFollowedByUserIds),\n          wtfImpressions = if (wtfImpressions.isEmpty) None else Some(wtfImpressions)\n        )\n      case _ => request\n    }\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services/exceptions/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift:controller\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/exceptions\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/filters\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/modules\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/response\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/routing\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/services/exceptions/UnknownExceptionMapper.scala",
    "content": "package com.twitter.follow_recommendations.service.exceptions\n\nimport com.twitter.finatra.thrift.exceptions.ExceptionMapper\nimport com.twitter.inject.Logging\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n@Singleton\nclass UnknownLoggingExceptionMapper extends ExceptionMapper[Exception, Throwable] with Logging {\n  def handleException(throwable: Exception): Future[Throwable] = {\n    error(\n      s\"Unmapped Exception: ${throwable.getMessage} - ${throwable.getStackTrace.mkString(\", \\n\\t\")}\",\n      throwable\n    )\n\n    Future.exception(throwable)\n  }\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/utils/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-server/src/main/scala\",\n        \"finatra/inject/inject-thrift-client\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/base\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/addressbook\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/geo\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/ppmi_locale_follow\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/recent_engagement\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/sims_expansion\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/socialgraph\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/stp\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/triangular_loops\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/candidate_sources/two_hop_random_walk\",\n        \"follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/feature_hydration/sources\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/configapi/params\",\n        \"follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/models\",\n        \"twitter-server/server/src/main/scala\",\n        \"util/util-app/src/main/scala\",\n        \"util/util-core:scala\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/utils/CandidateSourceHoldbackUtil.scala",
    "content": "package com.twitter.follow_recommendations.utils\n\nimport com.twitter.follow_recommendations.common.candidate_sources.addressbook._\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopCountrySource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopCountryBackFillSource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopGeoSource\nimport com.twitter.follow_recommendations.common.candidate_sources.geo.PopGeohashSource\nimport com.twitter.follow_recommendations.common.candidate_sources.ppmi_locale_follow.PPMILocaleFollowSource\nimport com.twitter.follow_recommendations.common.candidate_sources.recent_engagement.RecentEngagementNonDirectFollowSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims.SwitchingSimsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentEngagementSimilarUsersSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentFollowingSimilarUsersSource\nimport com.twitter.follow_recommendations.common.candidate_sources.sims_expansion.RecentStrongEngagementDirectFollowSimilarUsersSource\nimport com.twitter.follow_recommendations.common.candidate_sources.socialgraph.RecentFollowingRecentFollowingExpansionSource\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.MutualFollowStrongTiePredictionSource\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.OfflineStrongTiePredictionSource\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.BaseOnlineSTPSource\nimport com.twitter.follow_recommendations.common.candidate_sources.stp.SocialProofEnforcedOfflineStrongTiePredictionSource\nimport com.twitter.follow_recommendations.common.candidate_sources.triangular_loops.TriangularLoopsSource\nimport com.twitter.follow_recommendations.common.candidate_sources.two_hop_random_walk.TwoHopRandomWalkSource\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.follow_recommendations.configapi.params.GlobalParams\nimport com.twitter.follow_recommendations.models.CandidateSourceType\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.timelines.configapi.HasParams\n\ntrait CandidateSourceHoldbackUtil {\n  import CandidateSourceHoldbackUtil._\n  def filterCandidateSources[T <: HasParams](\n    request: T,\n    sources: Seq[CandidateSource[T, CandidateUser]]\n  ): Seq[CandidateSource[T, CandidateUser]] = {\n    val typeToFilter = request.params(GlobalParams.CandidateSourcesToFilter)\n    val sourcesToFilter = CandidateSourceTypeToMap.get(typeToFilter).getOrElse(Set.empty)\n    sources.filterNot { source => sourcesToFilter.contains(source.identifier) }\n  }\n}\n\nobject CandidateSourceHoldbackUtil {\n  final val ContextualActivityCandidateSourceIds: Set[CandidateSourceIdentifier] =\n    Set(\n      RecentFollowingSimilarUsersSource.Identifier,\n      RecentEngagementNonDirectFollowSource.Identifier,\n      RecentEngagementSimilarUsersSource.Identifier,\n      RecentStrongEngagementDirectFollowSimilarUsersSource.Identifier,\n      SwitchingSimsSource.Identifier,\n    )\n\n  final val SocialCandidateSourceIds: Set[CandidateSourceIdentifier] =\n    Set(\n      ForwardEmailBookSource.Identifier,\n      ForwardPhoneBookSource.Identifier,\n      ReverseEmailBookSource.Identifier,\n      ReversePhoneBookSource.Identifier,\n      RecentFollowingRecentFollowingExpansionSource.Identifier,\n      BaseOnlineSTPSource.Identifier,\n      MutualFollowStrongTiePredictionSource.Identifier,\n      OfflineStrongTiePredictionSource.Identifier,\n      SocialProofEnforcedOfflineStrongTiePredictionSource.Identifier,\n      TriangularLoopsSource.Identifier,\n      TwoHopRandomWalkSource.Identifier\n    )\n\n  final val GeoCandidateSourceIds: Set[CandidateSourceIdentifier] =\n    Set(\n      PPMILocaleFollowSource.Identifier,\n      PopCountrySource.Identifier,\n      PopGeohashSource.Identifier,\n      PopCountryBackFillSource.Identifier,\n      PopGeoSource.Identifier,\n    )\n\n  final val CandidateSourceTypeToMap: Map[CandidateSourceType.Value, Set[\n    CandidateSourceIdentifier\n  ]] =\n    Map(\n      CandidateSourceType.Social -> SocialCandidateSourceIds,\n      CandidateSourceType.ActivityContextual -> ContextualActivityCandidateSourceIds,\n      CandidateSourceType.GeoAndInterests -> GeoCandidateSourceIds\n    )\n}\n"
  },
  {
    "path": "follow-recommendations-service/server/src/main/scala/com/twitter/follow_recommendations/utils/RecommendationFlowBaseSideEffectsUtil.scala",
    "content": "package com.twitter.follow_recommendations.utils\n\nimport com.twitter.follow_recommendations.common.base.RecommendationFlow\nimport com.twitter.follow_recommendations.common.base.SideEffectsUtil\nimport com.twitter.follow_recommendations.common.models.CandidateUser\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\n\ntrait RecommendationFlowBaseSideEffectsUtil[Target <: HasClientContext, Candidate <: CandidateUser]\n    extends SideEffectsUtil[Target, Candidate] {\n  recommendationFlow: RecommendationFlow[Target, Candidate] =>\n\n  override def applySideEffects(\n    target: Target,\n    candidateSources: Seq[CandidateSource[Target, Candidate]],\n    candidatesFromCandidateSources: Seq[Candidate],\n    mergedCandidates: Seq[Candidate],\n    filteredCandidates: Seq[Candidate],\n    rankedCandidates: Seq[Candidate],\n    transformedCandidates: Seq[Candidate],\n    truncatedCandidates: Seq[Candidate],\n    results: Seq[Candidate]\n  ): Stitch[Unit] = {\n    Stitch.async(\n      Stitch.collect(\n        Seq(\n          applySideEffectsCandidateSourceCandidates(\n            target,\n            candidateSources,\n            candidatesFromCandidateSources),\n          applySideEffectsMergedCandidates(target, mergedCandidates),\n          applySideEffectsFilteredCandidates(target, filteredCandidates),\n          applySideEffectsRankedCandidates(target, rankedCandidates),\n          applySideEffectsTransformedCandidates(target, transformedCandidates),\n          applySideEffectsTruncatedCandidates(target, truncatedCandidates),\n          applySideEffectsResults(target, results)\n        )\n      ))\n  }\n\n  /*\n  In subclasses, override functions below to apply custom side effects at each step in pipeline.\n  Call super.applySideEffectsXYZ to scribe basic scribes implemented in this parent class\n   */\n  def applySideEffectsCandidateSourceCandidates(\n    target: Target,\n    candidateSources: Seq[CandidateSource[Target, Candidate]],\n    candidatesFromCandidateSources: Seq[Candidate]\n  ): Stitch[Unit] = {\n    val candidatesGroupedByCandidateSources =\n      candidatesFromCandidateSources.groupBy(\n        _.getPrimaryCandidateSource.getOrElse(CandidateSourceIdentifier(\"NoCandidateSource\")))\n\n    target.getOptionalUserId match {\n      case Some(userId) =>\n        val userAgeOpt = SnowflakeId.timeFromIdOpt(userId).map(_.untilNow.inDays)\n        userAgeOpt match {\n          case Some(userAge) if userAge <= 30 =>\n            candidateSources.map { candidateSource =>\n              {\n                val candidateSourceStats = statsReceiver.scope(candidateSource.identifier.name)\n\n                val isEmpty =\n                  !candidatesGroupedByCandidateSources.keySet.contains(candidateSource.identifier)\n\n                if (userAge <= 1)\n                  candidateSourceStats\n                    .scope(\"user_age\", \"1\", \"empty\").counter(isEmpty.toString).incr()\n                if (userAge <= 7)\n                  candidateSourceStats\n                    .scope(\"user_age\", \"7\", \"empty\").counter(isEmpty.toString).incr()\n                if (userAge <= 30)\n                  candidateSourceStats\n                    .scope(\"user_age\", \"30\", \"empty\").counter(isEmpty.toString).incr()\n              }\n            }\n          case _ => Nil\n        }\n      case None => Nil\n    }\n    Stitch.Unit\n  }\n\n  def applySideEffectsBaseCandidates(\n    target: Target,\n    candidates: Seq[Candidate]\n  ): Stitch[Unit] = Stitch.Unit\n\n  def applySideEffectsMergedCandidates(\n    target: Target,\n    candidates: Seq[Candidate]\n  ): Stitch[Unit] = applySideEffectsBaseCandidates(target, candidates)\n\n  def applySideEffectsFilteredCandidates(\n    target: Target,\n    candidates: Seq[Candidate]\n  ): Stitch[Unit] = applySideEffectsBaseCandidates(target, candidates)\n\n  def applySideEffectsRankedCandidates(\n    target: Target,\n    candidates: Seq[Candidate]\n  ): Stitch[Unit] = applySideEffectsBaseCandidates(target, candidates)\n\n  def applySideEffectsTransformedCandidates(\n    target: Target,\n    candidates: Seq[Candidate]\n  ): Stitch[Unit] = applySideEffectsBaseCandidates(target, candidates)\n\n  def applySideEffectsTruncatedCandidates(\n    target: Target,\n    candidates: Seq[Candidate]\n  ): Stitch[Unit] = applySideEffectsBaseCandidates(target, candidates)\n\n  def applySideEffectsResults(\n    target: Target,\n    candidates: Seq[Candidate]\n  ): Stitch[Unit] = applySideEffectsBaseCandidates(target, candidates)\n}\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/BUILD",
    "content": "create_thrift_libraries(\n    base_name = \"thrift\",\n    sources = [\"*.thrift\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependency_roots = [\n        \"finatra-internal/thrift/src/main/thrift\",\n        \"follow-recommendations-service/thrift/src/main/thrift/logging:thrift\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_common\",\n        \"src/thrift/com/twitter/ml/api:data\",\n        \"src/thrift/com/twitter/suggests/controller_data\",\n    ],\n    generate_languages = [\n        \"java\",\n        \"scala\",\n        \"strato\",\n    ],\n    provides_java_name = \"follow-recommendations-java\",\n    provides_scala_name = \"follow-recommendations-scala\",\n)\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/assembler.thrift",
    "content": "namespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\nstruct Header {\n 1: required Title title\n}\n\nstruct Title {\n 1: required string text\n}\n\nstruct Footer {\n 1: optional Action action\n}\n\nstruct Action {\n 1: required string text\n 2: required string actionURL\n}\n\nstruct UserList {\n  1: required bool userBioEnabled\n  2: required bool userBioTruncated\n  3: optional i64 userBioMaxLines\n  4: optional FeedbackAction feedbackAction\n}\n\nstruct Carousel {\n  1: optional FeedbackAction feedbackAction\n}\n\nunion WTFPresentation {\n  1: UserList userBioList\n  2: Carousel carousel\n}\n\nstruct DismissUserId {}\n\nunion FeedbackAction {\n 1: DismissUserId dismissUserId\n}\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/client_context.thrift",
    "content": "namespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\n// Caller/Client level specific context (e.g, user id/guest id/app id).\nstruct ClientContext {\n  1: optional i64 userId(personalDataType='UserId')\n  2: optional i64 guestId(personalDataType='GuestId')\n  3: optional i64 appId(personalDataType='AppId')\n  4: optional string ipAddress(personalDataType='IpAddress')\n  5: optional string userAgent(personalDataType='UserAgent')\n  6: optional string countryCode(personalDataType='InferredCountry')\n  7: optional string languageCode(personalDataType='InferredLanguage')\n  9: optional bool isTwoffice(personalDataType='InferredLocation')\n  10: optional set<string> userRoles\n  11: optional string deviceId(personalDataType='DeviceId')\n  12: optional i64 guestIdAds(personalDataType='GuestId')\n  13: optional i64 guestIdMarketing(personalDataType='GuestId')\n}(hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/debug.thrift",
    "content": "namespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendation\n\n// These are broken into their own union\n// because we can have features that are\n// complex flavors of these (such as Seq)\nunion PrimitiveFeatureValue {\n    1: i32 intValue\n    2: i64 longValue\n    3: string strValue\n    4: bool boolValue\n}\n\nunion FeatureValue {\n    1: PrimitiveFeatureValue primitiveValue\n}\n\nstruct DebugParams {\n    1: optional map<string, FeatureValue> featureOverrides\n    2: optional i64 randomizationSeed\n    3: optional bool includeDebugInfoInResults\n    4: optional bool doNotLog\n}\n\nenum DebugCandidateSourceIdentifier {\n  UTT_INTERESTS_RELATED_USERS_SOURCE = 0\n  UTT_PRODUCER_EXPANSION_SOURCE = 1\n  UTT_SEED_ACCOUNT_SOURCE = 2\n  BYF_USER_FOLLOW_CLUSTER_SIMS_SOURCE = 3\n  BYF_USER_FOLLOW_CLUSTER_SOURCE = 4\n  USER_FOLLOW_CLUSTER_SOURCE = 5\n  RECENT_SEARCH_BASED_SOURCE = 6\n  PEOPLE_ACTIVITY_RECENT_ENGAGEMENT_SOURCE = 7\n  PEOPLE_ACTIVITY_RECENT_ENGAGEMENT_SIMS_SOURCE = 8,\n  REVERSE_PHONE_BOOK_SOURCE = 9,\n  REVERSE_EMAIL_BOOK_SOURCE = 10,\n  SIMS_DEBUG_STORE = 11,\n  UTT_PRODUCER_ONLINE_MBCG_SOURCE = 12,\n  BONUS_FOLLOW_CONDITIONAL_ENGAGEMENT_STORE = 13,\n  // 14 (BONUS_FOLLOW_PMI_STORE) was deleted as it's not used anymore\n  FOLLOW2VEC_NEAREST_NEIGHBORS_STORE = 15,\n  OFFLINE_STP = 16,\n  OFFLINE_STP_BIG = 17,\n  OFFLINE_MUTUAL_FOLLOW_EXPANSION = 18,\n  REPEATED_PROFILE_VISITS = 19,\n  TIME_DECAY_FOLLOW2VEC_NEAREST_NEIGHBORS_STORE = 20,\n  LINEAR_REGRESSION_FOLLOW2VEC_NEAREST_NEIGHBORS_STORE = 21,\n  REAL_GRAPH_EXPANSION_SOURCE = 22,\n  RELATABLE_ACCOUNTS_BY_INTEREST = 23,\n  EMAIL_TWEET_CLICK = 24,\n  GOOD_TWEET_CLICK_ENGAGEMENTS = 25,\n  ENGAGED_FOLLOWER_RATIO = 26,\n  TWEET_SHARE_ENGAGEMENTS = 27,\n  BULK_FRIEND_FOLLOWS = 28,\n  REAL_GRAPH_OON_V2_SOURCE = 30,\n  CROWD_SEARCH_ACCOUNTS = 31,\n  POP_GEOHASH = 32,\n  POP_COUNTRY = 33,\n  POP_COUNTRY_BACKFILL = 34,\n  TWEET_SHARER_TO_SHARE_RECIPIENT_ENGAGEMENTS = 35,\n  TWEET_AUTHOR_TO_SHARE_RECIPIENT_ENGAGEMENTS = 36,\n  BULK_FRIEND_FOLLOWS_NEW_USER = 37,\n  ONLINE_STP_EPSCORER = 38,\n  ORGANIC_FOLLOW_ACCOUNTS = 39,\n  NUX_LO_HISTORY = 40,\n  TRAFFIC_ATTRIBUTION_ACCOUNTS = 41,\n  ONLINE_STP_RAW_ADDRESS_BOOK = 42,\n  POP_GEOHASH_QUALITY_FOLLOW = 43,\n  NOTIFICATION_ENGAGEMENT = 44,\n  EFR_BY_WORLDWIDE_PICTURE_PRODUCER = 45,\n  POP_GEOHASH_REAL_GRAPH = 46,\n}\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/display_context.thrift",
    "content": "include \"flows.thrift\"\ninclude \"recently_engaged_user_id.thrift\"\n\nnamespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\nstruct Profile {\n    1: required i64 profileId(personalDataType='UserId')\n}(hasPersonalData='true')\n\nstruct Search {\n    1: required string searchQuery(personalDataType='SearchQuery')\n}(hasPersonalData='true')\n\nstruct Rux {\n    1: required i64 focalAuthorId(personalDataType='UserId')\n}(hasPersonalData='true')\n\nstruct Topic {\n  1: required i64 topicId(personalDataType = 'TopicFollow')\n}(hasPersonalData='true')\n\nstruct ReactiveFollow {\n    1: required list<i64> followedUserIds(personalDataType='UserId')\n}(hasPersonalData='true')\n\nstruct NuxInterests {\n    1: optional flows.FlowContext flowContext // set for recommendation inside an interactive flow\n    2: optional list<i64> uttInterestIds // if provided, we use these interestIds for generating candidates instead of for example fetching user selected interests\n}(hasPersonalData='true')\n\nstruct AdCampaignTarget {\n    1: required list<i64> similarToUserIds(personalDataType='UserId')\n}(hasPersonalData='true')\n\nstruct ConnectTab {\n    1: required list<i64> byfSeedUserIds(personalDataType='UserId')\n    2: required list<i64> similarToUserIds(personalDataType='UserId')\n    3: required list<recently_engaged_user_id.RecentlyEngagedUserId> recentlyEngagedUserIds\n}(hasPersonalData='true')\n\nstruct SimilarToUser {\n    1: required i64 similarToUserId(personalDataType='UserId')\n}(hasPersonalData='true')\n\nstruct PostNuxFollowTask {\n    1: optional flows.FlowContext flowContext // set for recommendation inside an interactive flow\n}(hasPersonalData='true')\n\nunion DisplayContext {\n    1: Profile profile\n    2: Search search\n    3: Rux rux\n    4: Topic topic\n    5: ReactiveFollow reactiveFollow\n    6: NuxInterests nuxInterests\n    7: AdCampaignTarget adCampaignTarget\n    8: ConnectTab connectTab\n    9: SimilarToUser similarToUser\n    10: PostNuxFollowTask postNuxFollowTask\n}(hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/display_location.thrift",
    "content": "namespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\nenum DisplayLocation {\n    SIDEBAR = 0\n    PROFILE_SIDEBAR = 2\n    CLUSTER_FOLLOW = 7\n    NEW_USER_SARUS_BACKFILL = 12\n    PROFILE_DEVICE_FOLLOW = 23\n    RECOS_BACKFILL = 32\n    HOME_TIMELINE = 39    # HOME_TIMELINE_WTF in Hermit\n    PROFILE_TOP_FOLLOWING = 42\n    PROFILE_TOP_FOLLOWERS = 43\n    PEOPLE_PLUS_PLUS = 47\n    EXPLORE_TAB = 57\n    MagicRecs = 59        # Account recommendation in notification\n    AB_UPLOAD_INJECTION = 60\n    /**\n     * To prevent setting 2 display locations with the same index in FRS.\n     *\n     * The display location should be added to the following files:\n     *  - follow-recommendations-service/thrift/src/main/thrift/display_location.thrift\n     *  - follow-recommendations-service/thrift/src/main/thrift/logging/display_location.thrift\n     *  - follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/DisplayLocation.scala\n   */\n    CAMPAIGN_FORM = 61\n    RUX_LANDING_PAGE = 62\n    PROFILE_BONUS_FOLLOW = 63\n    ELECTION_EXPLORE_WTF = 64\n    HTL_BONUS_FOLLOW = 65\n    TOPIC_LANDING_PAGE_HEADER = 66\n    NUX_PYMK = 67\n    NUX_INTERESTS = 68\n    REACTIVE_FOLLOW = 69\n    RUX_PYMK = 70\n    INDIA_COVID19_CURATED_ACCOUNTS_WTF = 71\n    NUX_TOPIC_BONUS_FOLLOW = 72\n    TWEET_NOTIFICATION_RECS = 73\n    HTL_SPACE_HOSTS = 74\n    POST_NUX_FOLLOW_TASK = 75\n    TOPIC_LANDING_PAGE = 76\n    USER_TYPEAHEAD_PREFETCH = 77\n    HOME_TIMELINE_RELATABLE_ACCOUNTS = 78\n    NUX_GEO_CATEGORY = 79\n    NUX_INTERESTS_CATEGORY = 80\n    NUX_PYMK_CATEGORY = 81\n    TOP_ARTICLES = 82\n    HOME_TIMELINE_TWEET_RECS = 83\n    HTL_BULK_FRIEND_FOLLOWS = 84\n    NUX_AUTO_FOLLOW = 85\n    SEARCH_BONUS_FOLLOW = 86\n    CONTENT_RECOMMENDER = 87\n    HOME_TIMELINE_REVERSE_CHRON = 88\n}\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/engagementType.thrift",
    "content": "namespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\nenum EngagementType {\n    Click = 0\n    Like = 1\n    Mention = 2\n    Retweet = 3\n    ProfileView = 4\n}\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/flows.thrift",
    "content": "/*\n * This file defines additional thrift objects that should be specified in FRS request for context of recommendation, specifically the previous recommendations / new interactions in an interactive flow (series of follow steps). These typically are sent from OCF\n */\n\nnamespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\nstruct FlowRecommendation {\n  1: required i64 userId(personalDataType='UserId')\n}(hasPersonalData='true')\n\nstruct RecommendationStep {\n  1: required list<FlowRecommendation> recommendations\n  2: required set<i64> followedUserIds(personalDataType='UserId')\n}(hasPersonalData='true')\n\nstruct FlowContext {\n  1: required list<RecommendationStep> steps\n}(hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/follow-recommendations-service.thrift",
    "content": "namespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\ninclude \"assembler.thrift\"\ninclude \"client_context.thrift\"\ninclude \"debug.thrift\"\ninclude \"display_context.thrift\"\ninclude \"display_location.thrift\"\ninclude \"recommendations.thrift\"\ninclude \"recently_engaged_user_id.thrift\"\n\ninclude \"finatra-thrift/finatra_thrift_exceptions.thrift\"\ninclude \"com/twitter/product_mixer/core/pipeline_execution_result.thrift\"\n\nstruct RecommendationRequest {\n    1: required client_context.ClientContext clientContext\n    2: required display_location.DisplayLocation displayLocation\n    3: optional display_context.DisplayContext displayContext\n    // Max results to return\n    4: optional i32 maxResults\n    // Cursor to continue returning results if any\n    5: optional string cursor\n    // IDs of Content to exclude from recommendations\n    6: optional list<i64> excludedIds(personalDataType='UserId')\n    // Whether to also get promoted content\n    7: optional bool fetchPromotedContent\n    8: optional debug.DebugParams debugParams\n    9: optional string userLocationState(personalDataType='InferredLocation')\n}(hasPersonalData='true')\n\n\nstruct RecommendationResponse {\n    1: required list<recommendations.Recommendation> recommendations\n}(hasPersonalData='true')\n\n// for scoring a list of candidates, while logging hydrated features\nstruct ScoringUserRequest {\n  1: required client_context.ClientContext clientContext\n  2: required display_location.DisplayLocation displayLocation\n  3: required list<recommendations.UserRecommendation> candidates\n  4: optional debug.DebugParams debugParams\n}(hasPersonalData='true')\n\nstruct ScoringUserResponse {\n  1: required list<recommendations.UserRecommendation> candidates // empty for now\n}(hasPersonalData='true')\n\n// for getting the list of candidates generated by a single candidate source\nstruct DebugCandidateSourceRequest {\n  1: required client_context.ClientContext clientContext\n  2: required debug.DebugCandidateSourceIdentifier candidateSource\n  3: optional list<i64> uttInterestIds\n  4: optional debug.DebugParams debugParams\n  5: optional list<i64> recentlyFollowedUserIds\n  6: optional list<recently_engaged_user_id.RecentlyEngagedUserId> recentlyEngagedUserIds\n  7: optional list<i64> byfSeedUserIds\n  8: optional list<i64> similarToUserIds\n  9: required bool applySgsPredicate\n  10: optional i32 maxResults\n}(hasPersonalData='true')\n\nservice FollowRecommendationsThriftService {\n  RecommendationResponse getRecommendations(1: RecommendationRequest request) throws (\n    1: finatra_thrift_exceptions.ServerError serverError,\n    2: finatra_thrift_exceptions.UnknownClientIdError unknownClientIdError,\n    3: finatra_thrift_exceptions.NoClientIdError noClientIdError\n  )\n  RecommendationDisplayResponse getRecommendationDisplayResponse(1: RecommendationRequest request) throws (\n    1: finatra_thrift_exceptions.ServerError serverError,\n    2: finatra_thrift_exceptions.UnknownClientIdError unknownClientIdError,\n    3: finatra_thrift_exceptions.NoClientIdError noClientIdError\n  )\n  // temporary endpoint for feature hydration and logging for data collection.\n  ScoringUserResponse scoreUserCandidates(1: ScoringUserRequest request) throws (\n    1: finatra_thrift_exceptions.ServerError serverError,\n    2: finatra_thrift_exceptions.UnknownClientIdError unknownClientIdError,\n    3: finatra_thrift_exceptions.NoClientIdError noClientIdError\n  )\n  // Debug endpoint for getting recommendations of a single candidate source. We can remove this endpoint when ProMix provide this functionality and we integrate with it.\n  RecommendationResponse debugCandidateSource(1: DebugCandidateSourceRequest request) throws (\n      1: finatra_thrift_exceptions.ServerError serverError,\n      2: finatra_thrift_exceptions.UnknownClientIdError unknownClientIdError,\n      3: finatra_thrift_exceptions.NoClientIdError noClientIdError\n  )\n\n  // Get the full execution log for a pipeline (used by our debugging tools)\n  pipeline_execution_result.PipelineExecutionResult executePipeline(1: RecommendationRequest request) throws (\n    1: finatra_thrift_exceptions.ServerError serverError,\n    2: finatra_thrift_exceptions.UnknownClientIdError unknownClientIdError,\n    3: finatra_thrift_exceptions.NoClientIdError noClientIdError\n  )\n}\n\nstruct RecommendationDisplayResponse {\n 1: required list<recommendations.HydratedRecommendation> hydratedRecommendation\n 2: optional assembler.Header header\n 3: optional assembler.Footer footer\n 4: optional assembler.WTFPresentation wtfPresentation\n}(hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/follow_recommendations_serving_history.thrift",
    "content": "namespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\n// struct used for storing the history of computing and serving of recommendations to a user\nstruct FollowRecommendationsServingHistory {\n  1: required i64 lastComputationTimeMs (personalDataType = 'PrivateTimestamp')\n  2: required i64 lastServingTimeMs (personalDataType = 'PrivateTimestamp')\n}(persisted='true', hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/BUILD",
    "content": "create_thrift_libraries(\n    base_name = \"thrift\",\n    sources = [\"*.thrift\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependency_roots = [\n        \"src/thrift/com/twitter/ads/adserver:adserver_common\",\n        \"src/thrift/com/twitter/ml/api:data\",\n        \"src/thrift/com/twitter/suggests/controller_data\",\n    ],\n    generate_languages = [\n        \"java\",\n        \"scala\",\n        \"strato\",\n    ],\n    provides_java_name = \"follow-recommendations-logging-java\",\n    provides_scala_name = \"follow-recommendations-logging-scala\",\n)\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/client_context.thrift",
    "content": "namespace java com.twitter.follow_recommendations.logging.thriftjava\n#@namespace scala com.twitter.follow_recommendations.logging.thriftscala\n#@namespace strato com.twitter.follow_recommendations.logging\n\n// Offline equal of ClientContext\nstruct OfflineClientContext {\n  1: optional i64 userId(personalDataType='UserId')\n  2: optional i64 guestId(personalDataType='GuestId')\n  3: optional i64 appId(personalDataType='AppId')\n  4: optional string countryCode(personalDataType='InferredCountry')\n  5: optional string languageCode(personalDataType='InferredLanguage')\n  6: optional i64 guestIdAds(personalDataType='GuestId')\n  7: optional i64 guestIdMarketing(personalDataType='GuestId')\n}(persisted='true', hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/debug.thrift",
    "content": "namespace java com.twitter.follow_recommendations.logging.thriftjava\n#@namespace scala com.twitter.follow_recommendations.logging.thriftscala\n#@namespace strato com.twitter.follow_recommendation.logging\n\n// subset of DebugParams\nstruct OfflineDebugParams {\n    1: optional i64 randomizationSeed // track if the request was randomly ranked or not\n}(persisted='true', hasPersonalData='false')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/display_context.thrift",
    "content": "include \"logging/flows.thrift\"\ninclude \"logging/recently_engaged_user_id.thrift\"\n\nnamespace java com.twitter.follow_recommendations.logging.thriftjava\n#@namespace scala com.twitter.follow_recommendations.logging.thriftscala\n#@namespace strato com.twitter.follow_recommendations.logging\n\n// Offline equal of Profile DisplayContext\nstruct OfflineProfile {\n    1: required i64 profileId(personalDataType='UserId')\n}(persisted='true', hasPersonalData='true')\n\n// Offline equal of Search DisplayContext\nstruct OfflineSearch {\n    1: required string searchQuery(personalDataType='SearchQuery')\n}(persisted='true', hasPersonalData='true')\n\n// Offline equal of Rux Landing Page DisplayContext\nstruct OfflineRux {\n  1: required i64 focalAuthorId(personalDataType=\"UserId\")\n}(persisted='true', hasPersonalData='true')\n\n// Offline equal of Topic DisplayContext\nstruct OfflineTopic {\n  1: required i64 topicId(personalDataType = 'TopicFollow')\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineReactiveFollow {\n    1: required list<i64> followedUserIds(personalDataType='UserId')\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineNuxInterests {\n    1: optional flows.OfflineFlowContext flowContext // set for recommendation inside an interactive flow\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineAdCampaignTarget {\n    1: required list<i64> similarToUserIds(personalDataType='UserId')\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineConnectTab {\n    1: required list<i64> byfSeedUserIds(personalDataType='UserId')\n    2: required list<i64> similarToUserIds(personalDataType='UserId')\n    3: required list<recently_engaged_user_id.RecentlyEngagedUserId> recentlyEngagedUserIds\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineSimilarToUser {\n    1: required i64 similarToUserId(personalDataType='UserId')\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflinePostNuxFollowTask {\n    1: optional flows.OfflineFlowContext flowContext // set for recommendation inside an interactive flow\n}(persisted='true', hasPersonalData='true')\n\n// Offline equal of DisplayContext\nunion OfflineDisplayContext {\n    1: OfflineProfile profile\n    2: OfflineSearch search\n    3: OfflineRux rux\n    4: OfflineTopic topic\n    5: OfflineReactiveFollow reactiveFollow\n    6: OfflineNuxInterests nuxInterests\n    7: OfflineAdCampaignTarget adCampaignTarget\n    8: OfflineConnectTab connectTab\n    9: OfflineSimilarToUser similarToUser\n    10: OfflinePostNuxFollowTask postNuxFollowTask\n}(persisted='true', hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/display_location.thrift",
    "content": "namespace java com.twitter.follow_recommendations.logging.thriftjava\n#@namespace scala com.twitter.follow_recommendations.logging.thriftscala\n#@namespace strato com.twitter.follow_recommendations.logging\n\n/**\n * Make sure you add the new DL to the following files and redeploy our attribution jobs\n *  - follow-recommendations-service/thrift/src/main/thrift/display_location.thrift\n *  - follow-recommendations-service/thrift/src/main/thrift/logging/display_location.thrift\n *  - follow-recommendations-service/common/src/main/scala/com/twitter/follow_recommendations/common/models/DisplayLocation.scala\n */\n\n// Offline equal of DisplayLocation\nenum OfflineDisplayLocation {\n    SIDEBAR = 0\n    PROFILE_SIDEBAR = 2\n    CLUSTER_FOLLOW = 7\n    NEW_USER_SARUS_BACKFILL = 12\n    PROFILE_DEVICE_FOLLOW = 23\n    RECOS_BACKFILL = 32\n    HOME_TIMELINE = 39\n    PROFILE_TOP_FOLLOWING = 42\n    PROFILE_TOP_FOLLOWERS = 43\n    PEOPLE_PLUS_PLUS = 47\n    EXPLORE_TAB = 57\n    MagicRecs = 59\n  AB_UPLOAD_INJECTION = 60\n    CAMPAIGN_FORM = 61\n    RUX_LANDING_PAGE = 62\n    PROFILE_BONUS_FOLLOW = 63\n    ELECTION_EXPLORE_WTF = 64\n    HTL_BONUS_FOLLOW = 65\n    TOPIC_LANDING_PAGE_HEADER = 66\n    NUX_PYMK = 67\n    NUX_INTERESTS = 68\n    REACTIVE_FOLLOW = 69\n    RUX_PYMK = 70\n    INDIA_COVID19_CURATED_ACCOUNTS_WTF=71\n    NUX_TOPIC_BONUS_FOLLOW = 72\n    TWEET_NOTIFICATION_RECS = 73\n    HTL_SPACE_HOSTS = 74\n    POST_NUX_FOLLOW_TASK = 75\n    TOPIC_LANDING_PAGE = 76\n    USER_TYPEAHEAD_PREFETCH = 77\n    HOME_TIMELINE_RELATABLE_ACCOUNTS = 78\n    NUX_GEO_CATEGORY = 79\n    NUX_INTERESTS_CATEGORY = 80\n    NUX_PYMK_CATEGORY = 81\n    TOP_ARTICLES = 82\n    HOME_TIMELINE_TWEET_RECS = 83\n    HTL_BULK_FRIEND_FOLLOWS = 84\n    NUX_AUTO_FOLLOW = 85\n    SEARCH_BONUS_FOLLOW = 86\n    CONTENT_RECOMMENDER = 87\n    HOME_TIMELINE_REVERSE_CHRON = 88\n}(persisted='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/engagementType.thrift",
    "content": "namespace java com.twitter.follow_recommendations.logging.thriftjava\n#@namespace scala com.twitter.follow_recommendations.logging.thriftscala\n#@namespace strato com.twitter.follow_recommendations.logging\n\nenum EngagementType {\n    Click = 0\n    Like = 1\n    Mention = 2\n    Retweet = 3\n    ProfileView = 4\n}\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/flows.thrift",
    "content": "namespace java com.twitter.follow_recommendations.logging.thriftjava\n#@namespace scala com.twitter.follow_recommendations.logging.thriftscala\n#@namespace strato com.twitter.follow_recommendations.logging\n\nstruct OfflineFlowRecommendation {\n  1: required i64 userId(personalDataType='UserId')\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineRecommendationStep {\n  1: required list<OfflineFlowRecommendation> recommendations\n  2: required set<i64> followedUserIds(personalDataType='UserId')\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineFlowContext {\n  1: required list<OfflineRecommendationStep> steps\n}(persisted='true', hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/logs.thrift",
    "content": "namespace java com.twitter.follow_recommendations.logging.thriftjava\n#@namespace scala com.twitter.follow_recommendations.logging.thriftscala\n#@namespace strato com.twitter.follow_recommendations.logging\n\ninclude \"client_context.thrift\"\ninclude \"debug.thrift\"\ninclude \"display_context.thrift\"\ninclude \"display_location.thrift\"\ninclude \"recommendations.thrift\"\n\nstruct OfflineRecommendationRequest {\n    1: required client_context.OfflineClientContext clientContext\n    2: required display_location.OfflineDisplayLocation displayLocation\n    3: optional display_context.OfflineDisplayContext displayContext\n    4: optional i32 maxResults\n    5: optional string cursor\n    6: optional list<i64> excludedIds(personalDataType='UserId')\n    7: optional bool fetchPromotedContent\n    8: optional debug.OfflineDebugParams debugParams\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineRecommendationResponse {\n    1: required list<recommendations.OfflineRecommendation> recommendations\n}(persisted='true', hasPersonalData='true')\n\nstruct RecommendationLog {\n    1: required OfflineRecommendationRequest request\n    2: required OfflineRecommendationResponse response\n    3: required i64 timestampMs\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineScoringUserRequest {\n  1: required client_context.OfflineClientContext clientContext\n  2: required display_location.OfflineDisplayLocation displayLocation\n  3: required list<recommendations.OfflineUserRecommendation> candidates\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineScoringUserResponse {\n  1: required list<recommendations.OfflineUserRecommendation> candidates\n}(persisted='true', hasPersonalData='true')\n\nstruct ScoredUsersLog {\n  1: required OfflineScoringUserRequest request\n  2: required OfflineScoringUserResponse response\n    3: required i64 timestampMs\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineRecommendationFlowUserMetadata {\n  1: optional i32 userSignupAge(personalDataType = 'AgeOfAccount')\n  2: optional string userState(personalDataType = 'UserState')\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineRecommendationFlowSignals {\n  1: optional string countryCode(personalDataType='InferredCountry')\n}(persisted='true', hasPersonalData='true')\n\nstruct OfflineRecommendationFlowCandidateSourceCandidates {\n  1: required string candidateSourceName\n  2: required list<i64> candidateUserIds(personalDataType='UserId')\n  3: optional list<double> candidateUserScores\n}(persisted='true', hasPersonalData='true')\n\nstruct RecommendationFlowLog {\n  1: required client_context.OfflineClientContext clientContext\n  2: optional OfflineRecommendationFlowUserMetadata userMetadata\n  3: optional OfflineRecommendationFlowSignals signals\n  4: required i64 timestampMs\n  5: required string recommendationFlowIdentifier\n  6: optional list<OfflineRecommendationFlowCandidateSourceCandidates> filteredCandidates\n  7: optional list<OfflineRecommendationFlowCandidateSourceCandidates> rankedCandidates\n  8: optional list<OfflineRecommendationFlowCandidateSourceCandidates> truncatedCandidates\n}(persisted='true', hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/reasons.thrift",
    "content": "namespace java com.twitter.follow_recommendations.logging.thriftjava\n#@namespace scala com.twitter.follow_recommendations.logging.thriftscala\n#@namespace strato com.twitter.follow_recommendations.logging\n\n// Proof based on Follow relationship\nstruct FollowProof {\n  1: required list<i64> userIds(personalDataType='UserId')\n  2: required i32 numIds(personalDataType='CountOfFollowersAndFollowees')\n}(persisted='true', hasPersonalData='true')\n\n// Similar to userIds in the context (e.g. profileId)\nstruct SimilarToProof {\n  1: required list<i64> userIds(personalDataType='UserId')\n}(persisted='true', hasPersonalData='true')\n\n// Proof based on geo location\nstruct PopularInGeoProof {\n  1: required string location(personalDataType='InferredLocation')\n}(persisted='true', hasPersonalData='true')\n\n// Proof based on ttt interest\nstruct TttInterestProof {\n  1: required i64 interestId(personalDataType='ProvidedInterests')\n  2: required string interestDisplayName(personalDataType='ProvidedInterests')\n}(persisted='true', hasPersonalData='true')\n\n// Proof based on topics\nstruct TopicProof {\n  1: required i64 topicId(personalDataType='ProvidedInterests')\n}(persisted='true', hasPersonalData='true')\n\n// Proof based on custom interest / search queries\nstruct CustomInterestProof {\n  1: required string customerInterest(personalDataType='SearchQuery')\n}(persisted='true', hasPersonalData='true')\n\n// Proof based on tweet authors\nstruct TweetsAuthorProof {\n  1: required list<i64> tweetIds(personalDataType='TweetId')\n}(persisted='true', hasPersonalData='true')\n\n// Proof candidate is of device follow type\nstruct DeviceFollowProof {\n  1: required bool isDeviceFollow(personalDataType='OtherDeviceInfo')\n}(persisted='true', hasPersonalData='true')\n\n// Account level proof that should be attached to each candidate\nstruct AccountProof {\n  1: optional FollowProof followProof\n  2: optional SimilarToProof similarToProof\n  3: optional PopularInGeoProof popularInGeoProof\n  4: optional TttInterestProof tttInterestProof\n  5: optional TopicProof topicProof\n  6: optional CustomInterestProof customInterestProof\n  7: optional TweetsAuthorProof tweetsAuthorProof\n  8: optional DeviceFollowProof deviceFollowProof\n\n}(persisted='true', hasPersonalData='true')\n\nstruct Reason {\n  1: optional AccountProof accountProof  \n}(persisted='true', hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/recently_engaged_user_id.thrift",
    "content": "namespace java com.twitter.follow_recommendations.logging.thriftjava\n#@namespace scala com.twitter.follow_recommendations.logging.thriftscala\n#@namespace strato com.twitter.follow_recommendations.logging\n\ninclude \"engagementType.thrift\"\n\nstruct RecentlyEngagedUserId {\n  1: required i64 id(personalDataType='UserId')\n  2: required engagementType.EngagementType engagementType \n}(persisted='true', hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/recommendations.thrift",
    "content": "namespace java com.twitter.follow_recommendations.logging.thriftjava\n#@namespace scala com.twitter.follow_recommendations.logging.thriftscala\n#@namespace strato com.twitter.follow_recommendations.logging\n\ninclude \"com/twitter/ads/adserver/adserver_common.thrift\"\ninclude \"reasons.thrift\"\ninclude \"tracking.thrift\"\ninclude \"scoring.thrift\"\n\n// Offline equal of UserRecommendation\nstruct OfflineUserRecommendation {\n    1: required i64 userId(personalDataType='UserId')\n    // reason for this suggestions, eg: social context\n    2: optional reasons.Reason reason\n    // present if it is a promoted account\n    3: optional adserver_common.AdImpression adImpression\n  // tracking token (unserialized) for attribution\n  4: optional tracking.TrackingToken trackingToken\n    // scoring details\n    5: optional scoring.ScoringDetails scoringDetails\n}(persisted='true', hasPersonalData='true')\n\n// Offline equal of Recommendation\nunion OfflineRecommendation {\n    1: OfflineUserRecommendation user\n}(persisted='true', hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/scoring.thrift",
    "content": "namespace java com.twitter.follow_recommendations.logging.thriftjava\n#@namespace scala com.twitter.follow_recommendations.logging.thriftscala\n#@namespace strato com.twitter.follow_recommendations.logging\n\ninclude \"com/twitter/ml/api/data.thrift\"\n\nstruct CandidateSourceDetails {\n  1: optional map<string, double> candidateSourceScores\n  2: optional i32 primarySource\n}(persisted='true', hasPersonalData='false')\n\nstruct Score {\n  1: required double value\n  2: optional string rankerId\n  3: optional string scoreType\n}(persisted='true', hasPersonalData='false') // scoring and ranking info per ranking stage\n\n// Contains (1) the ML-based heavy ranker and score (2) scores and rankers in producer experiment framework\nstruct Scores {\n  1: required list<Score> scores\n  2: optional string selectedRankerId\n  3: required bool isInProducerScoringExperiment\n}(persisted='true', hasPersonalData='false')\n\nstruct RankingInfo {\n  1: optional Scores scores\n  2: optional i32 rank\n}(persisted='true', hasPersonalData='false')\n\n// this encapsulates all information related to the ranking process from generation to scoring\nstruct ScoringDetails {\n    1: optional CandidateSourceDetails candidateSourceDetails\n    2: optional double score  // The ML-based heavy ranker score\n    3: optional data.DataRecord dataRecord\n    4: optional list<string> rankerIds  // all ranker ids, including (1) ML-based heavy ranker (2) non-ML adhoc rankers\n    5: optional map<string, RankingInfo> infoPerRankingStage  // scoring and ranking info per ranking stage\n}(persisted='true', hasPersonalData='true')\n\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/logging/tracking.thrift",
    "content": "namespace java com.twitter.follow_recommendations.logging.thriftjava\n#@namespace scala com.twitter.follow_recommendations.logging.thriftscala\n#@namespace strato com.twitter.follow_recommendations.logging\n\ninclude \"com/twitter/suggests/controller_data/controller_data.thrift\"\ninclude \"display_location.thrift\"\n\nstruct TrackingToken {\n  // trace-id of the request\n  1: required i64 sessionId (personalDataType='SessionId')\n  2: optional display_location.OfflineDisplayLocation displayLocation\n  // 64-bit encoded binary attributes of our recommendation\n  3: optional controller_data.ControllerData controllerData\n  // WTF Algorithm Id (backward compatibility)\n  4: optional i32 algoId\n}(persisted='true', hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/reasons.thrift",
    "content": "namespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\n// Proof based on Follow relationship\nstruct FollowProof {\n  1: required list<i64> userIds(personalDataType='UserId')\n  2: required i32 numIds(personalDataType='CountOfFollowersAndFollowees')\n}(hasPersonalData='true')\n\n// Similar to userIds in the context (e.g. profileId)\nstruct SimilarToProof {\n  1: required list<i64> userIds(personalDataType='UserId')\n}(hasPersonalData='true')\n\n// Proof based on geo location\nstruct PopularInGeoProof {\n  1: required string location(personalDataType='InferredLocation')\n}(hasPersonalData='true')\n\n// Proof based on ttt interest\nstruct TttInterestProof {\n  1: required i64 interestId(personalDataType='ProvidedInterests')\n  2: required string interestDisplayName(personalDataType='ProvidedInterests')\n}(hasPersonalData='true')\n\n// Proof based on topics\nstruct TopicProof {\n  1: required i64 topicId(personalDataType='ProvidedInterests')\n}(hasPersonalData='true')\n\n// Proof based on custom interest / search queries\nstruct CustomInterestProof {\n  1: required string query(personalDataType='SearchQuery')\n}(hasPersonalData='true')\n\n// Proof based on tweet authors\nstruct TweetsAuthorProof {\n  1: required list<i64> tweetIds(personalDataType='TweetId')\n}(hasPersonalData='true')\n\n// Proof candidate is of device follow type\nstruct DeviceFollowProof {\n  1: required bool isDeviceFollow(personalDataType='OtherDeviceInfo')\n}(hasPersonalData='true')\n\n// Account level proof that should be attached to each candidate\nstruct AccountProof {\n  1: optional FollowProof followProof\n  2: optional SimilarToProof similarToProof\n  3: optional PopularInGeoProof popularInGeoProof\n  4: optional TttInterestProof tttInterestProof\n  5: optional TopicProof topicProof\n  6: optional CustomInterestProof customInterestProof\n  7: optional TweetsAuthorProof tweetsAuthorProof\n  8: optional DeviceFollowProof deviceFollowProof\n}(hasPersonalData='true')\n\nstruct Reason {\n  1: optional AccountProof accountProof \n}(hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/recently_engaged_user_id.thrift",
    "content": "namespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\ninclude \"engagementType.thrift\"\n\nstruct RecentlyEngagedUserId {\n  1: required i64 id(personalDataType='UserId')\n  2: required engagementType.EngagementType engagementType \n}(persisted='true', hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/recommendations.thrift",
    "content": "namespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\ninclude \"com/twitter/ads/adserver/adserver_common.thrift\"\ninclude \"debug.thrift\"\ninclude \"reasons.thrift\"\ninclude \"scoring.thrift\"\n\nstruct UserRecommendation {\n    1: required i64 userId(personalDataType='UserId')\n    // reason for this suggestions, eg: social context\n    2: optional reasons.Reason reason\n    // present if it is a promoted account\n    3: optional adserver_common.AdImpression adImpression\n    // tracking token for attribution\n    4: optional string trackingInfo\n    // scoring details\n    5: optional scoring.ScoringDetails scoringDetails\n    6: optional string recommendationFlowIdentifier\n    // FeatureSwitch overrides for candidates:\n    7: optional map<string, debug.FeatureValue> featureOverrides\n}(hasPersonalData='true')\n\nunion Recommendation {\n    1: UserRecommendation user\n}(hasPersonalData='true')\n\nstruct HydratedUserRecommendation {\n  1: required i64 userId(personalDataType='UserId')\n  2: optional string socialProof\n  // present if it is a promoted account, used by clients for determining ad impression\n  3: optional adserver_common.AdImpression adImpression\n  // tracking token for attribution\n  4: optional string trackingInfo\n}(hasPersonalData='true')\n\nunion HydratedRecommendation {\n  1: HydratedUserRecommendation hydratedUserRecommendation\n}\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/scoring.thrift",
    "content": "namespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\ninclude \"com/twitter/ml/api/data.thrift\"\n\nstruct CandidateSourceDetails {\n  1: optional map<string, double> candidateSourceScores\n  2: optional i32 primarySource\n  3: optional map<string, i32> candidateSourceRanks\n}(hasPersonalData='false')\n\nstruct Score {\n  1: required double value\n  2: optional string rankerId\n  3: optional string scoreType\n}(hasPersonalData='false')\n\n// Contains (1) the ML-based heavy ranker and score (2) scores and rankers in producer experiment framework\nstruct Scores {\n  1: required list<Score> scores\n  2: optional string selectedRankerId\n  3: required bool isInProducerScoringExperiment\n}(hasPersonalData='false')\n\nstruct RankingInfo {\n  1: optional Scores scores\n  2: optional i32 rank\n}(hasPersonalData='false')\n\n// this encapsulates all information related to the ranking process from generation to scoring\nstruct ScoringDetails {\n    1: optional CandidateSourceDetails candidateSourceDetails\n    2: optional double score\n    3: optional data.DataRecord dataRecord\n    4: optional list<string> rankerIds\n    5: optional DebugDataRecord debugDataRecord // this field is not logged as it's only used for debugging\n    6: optional map<string, RankingInfo> infoPerRankingStage  // scoring and ranking info per ranking stage\n}(hasPersonalData='true')\n\n// exactly the same as a data record, except that we store the feature name instead of the id\nstruct DebugDataRecord {\n  1: optional set<string> binaryFeatures;                     // stores BINARY features\n  2: optional map<string, double> continuousFeatures;         // stores CONTINUOUS features\n  3: optional map<string, i64> discreteFeatures;              // stores DISCRETE features\n  4: optional map<string, string> stringFeatures;             // stores STRING features\n  5: optional map<string, set<string>> sparseBinaryFeatures;  // stores sparse BINARY features\n  6: optional map<string, map<string, double>> sparseContinuousFeatures; // sparse CONTINUOUS features\n}(hasPersonalData='true')\n"
  },
  {
    "path": "follow-recommendations-service/thrift/src/main/thrift/tracking.thrift",
    "content": "namespace java com.twitter.follow_recommendations.thriftjava\n#@namespace scala com.twitter.follow_recommendations.thriftscala\n#@namespace strato com.twitter.follow_recommendations\n\ninclude \"com/twitter/suggests/controller_data/controller_data.thrift\"\ninclude \"display_location.thrift\"\n\n// struct used for tracking/attribution purposes in our offline pipelines\nstruct TrackingToken {\n  // trace-id of the request\n  1: required i64 sessionId (personalDataType='SessionId')\n  2: optional display_location.DisplayLocation displayLocation\n  // 64-bit encoded binary attributes of our recommendation\n  3: optional controller_data.ControllerData controllerData\n  // WTF Algorithm Id (backward compatibility)\n  4: optional i32 algoId\n}(hasPersonalData='true')\n"
  },
  {
    "path": "graph-feature-service/BUILD.bazel",
    "content": "alias(\n    name = \"graph_feature_service-server\",\n    target = \":graph_feature_service-server_lib\",\n)\n\ntarget(\n    name = \"graph_feature_service-server_lib\",\n    dependencies = [\n        \"graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server\",\n    ],\n)\n\nalias(\n    name = \"graph_feature_service-worker\",\n    target = \":graph_feature_service-worker_lib\",\n)\n\ntarget(\n    name = \"graph_feature_service-worker_lib\",\n    dependencies = [\n        \"graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker\",\n    ],\n)\n\njvm_binary(\n    name = \"server-bin\",\n    basename = \"graph_feature_service-server\",\n    main = \"com.twitter.graph_feature_service.server.Main\",\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":graph_feature_service-server\",\n        \"3rdparty/jvm/ch/qos/logback:logback-classic\",\n        \"finagle/finagle-zipkin-scribe/src/main/scala\",\n        \"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback\",\n        \"twitter-server/logback-classic/src/main/scala\",\n    ],\n)\n\njvm_binary(\n    name = \"worker-bin\",\n    basename = \"graph_feature_service-worker\",\n    main = \"com.twitter.graph_feature_service.worker.Main\",\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":graph_feature_service-worker\",\n        \"3rdparty/jvm/ch/qos/logback:logback-classic\",\n        \"finagle/finagle-zipkin-scribe/src/main/scala\",\n        \"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback\",\n        \"twitter-server/logback-classic/src/main/scala\",\n    ],\n)\n\njvm_app(\n    name = \"server-bundle\",\n    basename = \"graph_feature_service-server-dist\",\n    binary = \":server-bin\",\n    tags = [\"bazel-compatible\"],\n)\n\njvm_app(\n    name = \"worker-bundle\",\n    basename = \"graph_feature_service-worker-dist\",\n    binary = \":worker-bin\",\n    tags = [\"bazel-compatible\"],\n)\n"
  },
  {
    "path": "graph-feature-service/README.md",
    "content": "# Graph Feature Service\n\nGraph Feature Service (GFS) is a distributed system that can provide various graph features for given pairs of users. For instance, given source user A and candidate user C, GFS can answer questions like “how many of A’s followings have favorited C”, “how many of A’s followings are following C”, and “how much C is similar to the users that A has favorited“."
  },
  {
    "path": "graph-feature-service/doc/common.md",
    "content": "# Common thrift types\n\nGFS uses several thrift datastructures which are common to multiple queries. They are listed below.\n\n## EdgeType\n\n`EdgeType` is a thrift enum which specifies which edge types to query for the graph.\n\n```thrift\nenum EdgeType {\n  FOLLOWING,\n  FOLLOWED_BY,\n  FAVORITE,\n  FAVORITED_BY,\n  RETWEET,\n  RETWEETED_BY,\n  REPLY,\n  REPLYED_BY,\n  MENTION,\n  MENTIONED_BY,\n  MUTUAL_FOLLOW,\n  SIMILAR_TO, // more edge types (like block, report, etc.) can be supported later.\n  RESERVED_12,\n  RESERVED_13,\n  RESERVED_14,\n  RESERVED_15,\n  RESERVED_16,\n  RESERVED_17,\n  RESERVED_18,\n  RESERVED_19,\n  RESERVED_20\n}\n```\n\nFor an example of how this is used, consider the `GetNeighbors` query. If we set the `edgeType` field\nof the `GfsNeighborsRequest`, the response will contain all the users that the specified user follows.\nIf, on the other hand, we set `edgeType` to be `FollowedBy` it will return all the users who are\nfollowed by the specified user.\n\n## FeatureType\n\n`FeatureType` is a thrift struct which is used in queries which require two edge types.\n\n```thrift\nstruct FeatureType {\n  1: required EdgeType leftEdgeType // edge type from source user\n  2: required EdgeType rightEdgeType // edge type from candidate user\n}(persisted=\"true\")\n```\n\n## UserWithScore\n\nThe candidate generation queries return lists of candidates together with a computed score for the\nrelevant feature. `UserWithScore` is a thrift struct which bundles together a candidate's ID with\nthe score.\n\n```thrift\nstruct UserWithScore {\n  1: required i64 userId\n  2: required double score\n}\n```\n"
  },
  {
    "path": "graph-feature-service/doc/getintersection.md",
    "content": "# GetIntersection\n\n## Request and response syntax\n\nA `GetIntersection` call takes as input a `GfsIntersectionRequest` thrift struct. \n\n```thrift\nstruct GfsIntersectionRequest {\n  1: required i64 userId\n  2: required list<i64> candidateUserIds\n  3: required list<FeatureType> featureTypes\n}\n```\n\nThe response is returned in a `GfsIntersectionResponse` thrift struct.\n\n```thrift\nstruct GfsIntersectionResponse {\n  1: required i64 userId\n  2: required list<GfsIntersectionResult> results\n}\n\nstruct GfsIntersectionResult {\n  1: required i64 candidateUserId\n  2: required list<IntersectionValue> intersectionValues\n}\n\nstruct IntersectionValue {\n  1: required FeatureType featureType\n  2: optional i32 count\n  3: optional list<i64> intersectionIds\n  4: optional i32 leftNodeDegree\n  5: optional i32 rightNodeDegree\n}(persisted=\"true\")\n```\n\n## Behavior\n\nThe `GfsIntersectionResponse` contains in its `results` field a `GfsIntersectionResult` for every candidate in `candidateIds` which contains an  `IntersectionValue` for every `FeatureType` in the request's `featureTypes` field. \n\nThe `IntersectionValue` contains the size of the intersection between the `leftEdgeType` edges from `userId` and the `rightEdgeType` edges from `candidateId` in the `count` field, as well as their respective degrees in the graphs in `leftNodeDegree` and `rightNodeDegree` respectively.\n\n**Note:** the `intersectionIds` field currently only contains `Nil`.\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/common/BUILD.bazel",
    "content": "scala_library(\n    platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n    ],\n    dependencies = [\"src/scala/com/twitter/storehaus_internal/util\"],\n)\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/common/Configs.scala",
    "content": "package com.twitter.graph_feature_service.common\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\nimport java.nio.ByteBuffer\nimport scala.util.hashing.MurmurHash3\n\nobject Configs {\n\n  // NOTE: notify #recos-platform slack room, if you want to change this.\n  // This SHOULD be updated together with NUM_SHARDS in worker.aurora\n  final val NumGraphShards: Int = 40\n\n  final val TopKRealGraph: Int = 512\n\n  final val BaseHdfsPath: String = \"/user/cassowary/processed/gfs/constant_db/\"\n\n  // whether or not to write in_value and out_value graphs. Used in the scalding job.\n  final val EnableValueGraphs: Boolean = true\n  // whether or not to write in_key and out_key graphs. Used in the scalding job.\n  final val EnableKeyGraphs: Boolean = false\n\n  final val FollowOutValPath: String = \"follow_out_val/\"\n  final val FollowOutKeyPath: String = \"follow_out_key/\"\n  final val FollowInValPath: String = \"follow_in_val/\"\n  final val FollowInKeyPath: String = \"follow_in_key/\"\n\n  final val MutualFollowValPath: String = \"mutual_follow_val/\"\n  final val MutualFollowKeyPath: String = \"mutual_follow_key/\"\n\n  final val FavoriteOutValPath: String = \"favorite_out_val/\"\n  final val FavoriteInValPath: String = \"favorite_in_val/\"\n  final val FavoriteOutKeyPath: String = \"favorite_out_key/\"\n  final val FavoriteInKeyPath: String = \"favorite_in_key/\"\n\n  final val RetweetOutValPath: String = \"retweet_out_val/\"\n  final val RetweetInValPath: String = \"retweet_in_val/\"\n  final val RetweetOutKeyPath: String = \"retweet_out_key/\"\n  final val RetweetInKeyPath: String = \"retweet_in_key/\"\n\n  final val MentionOutValPath: String = \"mention_out_val/\"\n  final val MentionInValPath: String = \"mention_in_val/\"\n  final val MentionOutKeyPath: String = \"mention_out_key/\"\n  final val MentionInKeyPath: String = \"mention_in_key/\"\n\n  final val MemCacheTTL: Duration = 8.hours\n\n  final val RandomSeed: Int = 39582942\n\n  def getTimedHdfsShardPath(shardId: Int, path: String, time: Time): String = {\n    val timeStr = time.format(\"yyyy/MM/dd\")\n    s\"$path/$timeStr/shard_$shardId\"\n  }\n\n  def getHdfsPath(path: String, overrideBaseHdfsPath: Option[String] = None): String = {\n    val basePath = overrideBaseHdfsPath.getOrElse(BaseHdfsPath)\n    s\"$basePath$path\"\n  }\n\n  private def hash(kArr: Array[Byte], seed: Int): Int = {\n    MurmurHash3.bytesHash(kArr, seed) & 0x7fffffff // keep positive\n  }\n\n  private def hashLong(l: Long, seed: Int): Int = {\n    hash(ByteBuffer.allocate(8).putLong(l).array(), seed)\n  }\n\n  def shardForUser(userId: Long): Int = {\n    hashLong(userId, RandomSeed) % NumGraphShards\n  }\n\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"**/*.scala\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/lz4:lz4-java\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"discovery-common/src/main/scala/com/twitter/discovery/common/stats\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"finatra-internal/decider/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-server/src/main/scala\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"finatra/inject/inject-utils/src/main/scala\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift:controller\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/filters\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/routing\",\n        \"graph-feature-service/src/main/resources\",\n        \"graph-feature-service/src/main/scala/com/twitter/graph_feature_service/common\",\n        \"graph-feature-service/src/main/scala/com/twitter/graph_feature_service/util\",\n        \"graph-feature-service/src/main/thrift/com/twitter/graph_feature_service:graph_feature_service_thrift-scala\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"servo/request/src/main/scala\",\n        \"src/scala/com/twitter/storehaus_internal/memcache\",\n        \"util/util-app/src/main/scala\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server/Main.scala",
    "content": "package com.twitter.graph_feature_service.server\n\nimport com.google.inject.Module\nimport com.twitter.finatra.decider.modules.DeciderModule\nimport com.twitter.finatra.mtls.thriftmux.Mtls\nimport com.twitter.finatra.thrift.ThriftServer\nimport com.twitter.finatra.thrift.filters.{\n  AccessLoggingFilter,\n  LoggingMDCFilter,\n  StatsFilter,\n  ThriftMDCFilter,\n  TraceIdMDCFilter\n}\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsThriftWebFormsModule\nimport com.twitter.finatra.thrift.routing.ThriftRouter\nimport com.twitter.graph_feature_service.server.controllers.ServerController\nimport com.twitter.graph_feature_service.server.handlers.ServerWarmupHandler\nimport com.twitter.graph_feature_service.server.modules.{\n  GetIntersectionStoreModule,\n  GraphFeatureServiceWorkerClientsModule,\n  ServerFlagsModule\n}\nimport com.twitter.graph_feature_service.thriftscala\nimport com.twitter.inject.thrift.modules.ThriftClientIdModule\n\nobject Main extends ServerMain\n\nclass ServerMain extends ThriftServer with Mtls {\n\n  override val name = \"graph_feature_service-server\"\n\n  override val modules: Seq[Module] = {\n    Seq(\n      ServerFlagsModule,\n      DeciderModule,\n      ThriftClientIdModule,\n      GraphFeatureServiceWorkerClientsModule,\n      GetIntersectionStoreModule,\n      new MtlsThriftWebFormsModule[thriftscala.Server.MethodPerEndpoint](this)\n    )\n  }\n\n  override def configureThrift(router: ThriftRouter): Unit = {\n    router\n      .filter[LoggingMDCFilter]\n      .filter[TraceIdMDCFilter]\n      .filter[ThriftMDCFilter]\n      .filter[AccessLoggingFilter]\n      .filter[StatsFilter]\n      .add[ServerController]\n  }\n\n  override protected def warmup(): Unit = {\n    handle[ServerWarmupHandler]()\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server/controllers/ServerController.scala",
    "content": "package com.twitter.graph_feature_service.server.controllers\n\nimport com.twitter.discovery.common.stats.DiscoveryStatsFilter\nimport com.twitter.finagle.Service\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.thrift.Controller\nimport com.twitter.graph_feature_service.server.handlers.ServerGetIntersectionHandler.GetIntersectionRequest\nimport com.twitter.graph_feature_service.server.handlers.ServerGetIntersectionHandler\nimport com.twitter.graph_feature_service.thriftscala\nimport com.twitter.graph_feature_service.thriftscala.Server.GetIntersection\nimport com.twitter.graph_feature_service.thriftscala.Server.GetPresetIntersection\nimport com.twitter.graph_feature_service.thriftscala._\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ServerController @Inject() (\n  serverGetIntersectionHandler: ServerGetIntersectionHandler\n)(\n  implicit statsReceiver: StatsReceiver)\n    extends Controller(thriftscala.Server) {\n\n  private val getIntersectionService: Service[GetIntersectionRequest, GfsIntersectionResponse] =\n    new DiscoveryStatsFilter(statsReceiver.scope(\"srv\").scope(\"get_intersection\"))\n      .andThen(Service.mk(serverGetIntersectionHandler))\n\n  val getIntersection: Service[GetIntersection.Args, GfsIntersectionResponse] = { args =>\n    // TODO: Disable updateCache after HTL switch to use PresetIntersection endpoint.\n    getIntersectionService(\n      GetIntersectionRequest.fromGfsIntersectionRequest(args.request, cacheable = true))\n  }\n  handle(GetIntersection) { getIntersection }\n\n  def getPresetIntersection: Service[\n    GetPresetIntersection.Args,\n    GfsIntersectionResponse\n  ] = { args =>\n    // TODO: Refactor after HTL switch to PresetIntersection\n    val cacheable = args.request.presetFeatureTypes == PresetFeatureTypes.HtlTwoHop\n    getIntersectionService(\n      GetIntersectionRequest.fromGfsPresetIntersectionRequest(args.request, cacheable))\n  }\n\n  handle(GetPresetIntersection) { getPresetIntersection }\n\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server/handlers/ServerGetIntersectionHandler.scala",
    "content": "package com.twitter.graph_feature_service.server.handlers\n\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.graph_feature_service.server.handlers.ServerGetIntersectionHandler.GetIntersectionRequest\nimport com.twitter.graph_feature_service.server.stores.FeatureTypesEncoder\nimport com.twitter.graph_feature_service.server.stores.GetIntersectionStore.GetIntersectionQuery\nimport com.twitter.graph_feature_service.thriftscala.PresetFeatureTypes\nimport com.twitter.graph_feature_service.thriftscala._\nimport com.twitter.graph_feature_service.util.FeatureTypesCalculator\nimport com.twitter.servo.request.RequestHandler\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport com.twitter.util.Memoize\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass ServerGetIntersectionHandler @Inject() (\n  @Named(\"ReadThroughGetIntersectionStore\")\n  readThroughStore: ReadableStore[GetIntersectionQuery, CachedIntersectionResult],\n  @Named(\"BypassCacheGetIntersectionStore\")\n  readOnlyStore: ReadableStore[GetIntersectionQuery, CachedIntersectionResult]\n)(\n  implicit statsReceiver: StatsReceiver)\n    extends RequestHandler[GetIntersectionRequest, GfsIntersectionResponse] {\n\n  import ServerGetIntersectionHandler._\n\n  // TODO: Track all the stats based on PresetFeatureType and update the dashboard\n  private val stats: StatsReceiver = statsReceiver.scope(\"srv\").scope(\"get_intersection\")\n  private val numCandidatesCount = stats.counter(\"total_num_candidates\")\n  private val numCandidatesStat = stats.stat(\"num_candidates\")\n  private val numFeaturesStat = stats.stat(\"num_features\")\n  private val userEmptyCount = stats.counter(\"user_empty_count\")\n  private val candidateEmptyRateStat = stats.stat(\"candidate_empty_rate\")\n  private val candidateNumEmptyStat = stats.stat(\"candidate_num_empty\")\n  private val missedRateStat = stats.stat(\"miss_rate\")\n  private val numMissedStat = stats.stat(\"num_missed\")\n\n  // Assume the order from HTL doesn't change. Only log the HTL query now.\n  private val featureStatMap = FeatureTypesCalculator.presetFeatureTypes.map { feature =>\n    val featureString = s\"${feature.leftEdgeType.name}_${feature.rightEdgeType.name}\"\n    feature -> Array(\n      stats.counter(s\"feature_type_${featureString}_total\"),\n      stats.counter(s\"feature_type_${featureString}_count_zero\"),\n      stats.counter(s\"feature_type_${featureString}_left_zero\"),\n      stats.counter(s\"feature_type_${featureString}_right_zero\")\n    )\n  }.toMap\n\n  private val sourceCandidateNumStats = Memoize[PresetFeatureTypes, Stat] { presetFeature =>\n    stats.stat(s\"source_candidate_num_${presetFeature.name}\")\n  }\n\n  override def apply(request: GetIntersectionRequest): Future[GfsIntersectionResponse] = {\n    val featureTypes = request.calculatedFeatureTypes\n    val numCandidates = request.candidateUserIds.length\n    val numFeatures = featureTypes.length\n\n    numCandidatesCount.incr(numCandidates)\n    numCandidatesStat.add(numCandidates)\n    numFeaturesStat.add(numFeatures)\n    sourceCandidateNumStats(request.presetFeatureTypes).add(numCandidates)\n\n    // Note: do not change the orders of features and candidates.\n    val candidateIds = request.candidateUserIds\n\n    if (featureTypes.isEmpty || candidateIds.isEmpty) {\n      Future.value(DefaultGfsIntersectionResponse)\n    } else {\n      Future\n        .collect {\n          val getIntersectionStore = if (request.cacheable) readThroughStore else readOnlyStore\n          getIntersectionStore.multiGet(GetIntersectionQuery.buildQueries(request))\n        }.map { responses =>\n          val results = responses.collect {\n            case (query, Some(result)) =>\n              query.candidateId -> GfsIntersectionResult(\n                query.candidateId,\n                query.calculatedFeatureTypes.zip(result.values).map {\n                  case (featureType, value) =>\n                    IntersectionValue(\n                      featureType,\n                      Some(value.count),\n                      if (value.intersectionIds.isEmpty) None else Some(value.intersectionIds),\n                      Some(value.leftNodeDegree),\n                      Some(value.rightNodeDegree)\n                    )\n                }\n              )\n          }\n\n          // Keep the response order same as input\n          val processedResults = candidateIds.map { candidateId =>\n            results.getOrElse(candidateId, GfsIntersectionResult(candidateId, List.empty))\n          }\n\n          val candidateEmptyNum =\n            processedResults.count(\n              _.intersectionValues.exists(value => isZero(value.rightNodeDegree)))\n\n          val numMissed = processedResults.count(_.intersectionValues.size != numFeatures)\n\n          if (processedResults.exists(\n              _.intersectionValues.forall(value => isZero(value.leftNodeDegree)))) {\n            userEmptyCount.incr()\n          }\n\n          candidateNumEmptyStat.add(candidateEmptyNum)\n          candidateEmptyRateStat.add(candidateEmptyNum.toFloat / numCandidates)\n          numMissedStat.add(numMissed)\n          missedRateStat.add(numMissed.toFloat / numCandidates)\n\n          processedResults.foreach { result =>\n            result.intersectionValues.zip(featureTypes).foreach {\n              case (value, featureType) =>\n                featureStatMap.get(featureType).foreach { statsArray =>\n                  statsArray(TotalIndex).incr()\n                  if (isZero(value.count)) {\n                    statsArray(CountIndex).incr()\n                  }\n                  if (isZero(value.leftNodeDegree)) {\n                    statsArray(LeftIndex).incr()\n                  }\n                  if (isZero(value.rightNodeDegree)) {\n                    statsArray(RightIndex).incr()\n                  }\n                }\n            }\n          }\n\n          GfsIntersectionResponse(processedResults)\n        }\n    }\n\n  }\n\n}\n\nprivate[graph_feature_service] object ServerGetIntersectionHandler {\n\n  case class GetIntersectionRequest(\n    userId: Long,\n    candidateUserIds: Seq[Long],\n    featureTypes: Seq[FeatureType],\n    presetFeatureTypes: PresetFeatureTypes,\n    intersectionIdLimit: Option[Int],\n    cacheable: Boolean) {\n\n    lazy val calculatedFeatureTypes: Seq[FeatureType] =\n      FeatureTypesCalculator.getFeatureTypes(presetFeatureTypes, featureTypes)\n\n    lazy val calculatedFeatureTypesString: String =\n      FeatureTypesEncoder(calculatedFeatureTypes)\n  }\n\n  object GetIntersectionRequest {\n\n    def fromGfsIntersectionRequest(\n      request: GfsIntersectionRequest,\n      cacheable: Boolean\n    ): GetIntersectionRequest = {\n      GetIntersectionRequest(\n        request.userId,\n        request.candidateUserIds,\n        request.featureTypes,\n        PresetFeatureTypes.Empty,\n        request.intersectionIdLimit,\n        cacheable)\n    }\n\n    def fromGfsPresetIntersectionRequest(\n      request: GfsPresetIntersectionRequest,\n      cacheable: Boolean\n    ): GetIntersectionRequest = {\n      GetIntersectionRequest(\n        request.userId,\n        request.candidateUserIds,\n        List.empty,\n        request.presetFeatureTypes,\n        request.intersectionIdLimit,\n        cacheable)\n    }\n  }\n\n  private val DefaultGfsIntersectionResponse = GfsIntersectionResponse()\n\n  private val TotalIndex = 0\n  private val CountIndex = 1\n  private val LeftIndex = 2\n  private val RightIndex = 3\n\n  def isZero(opt: Option[Int]): Boolean = {\n    !opt.exists(_ != 0)\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server/handlers/ServerWarmupHandler.scala",
    "content": "package com.twitter.graph_feature_service.server.handlers\n\nimport com.twitter.finatra.thrift.routing.ThriftWarmup\nimport com.twitter.graph_feature_service.thriftscala.EdgeType.FavoritedBy\nimport com.twitter.graph_feature_service.thriftscala.EdgeType.FollowedBy\nimport com.twitter.graph_feature_service.thriftscala.EdgeType.Following\nimport com.twitter.graph_feature_service.thriftscala.Server.GetIntersection\nimport com.twitter.graph_feature_service.thriftscala.FeatureType\nimport com.twitter.graph_feature_service.thriftscala.GfsIntersectionRequest\nimport com.twitter.inject.utils.Handler\nimport com.twitter.scrooge.Request\nimport com.twitter.util.logging.Logger\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.util.Random\n\n@Singleton\nclass ServerWarmupHandler @Inject() (warmup: ThriftWarmup) extends Handler {\n\n  val logger: Logger = Logger(\"WarmupHandler\")\n\n  // TODO: Add the testing accounts to warm-up the service.\n  private val testingAccounts: Array[Long] = Seq.empty.toArray\n\n  private def getRandomRequest: GfsIntersectionRequest = {\n    GfsIntersectionRequest(\n      testingAccounts(Random.nextInt(testingAccounts.length)),\n      testingAccounts,\n      Seq(FeatureType(Following, FollowedBy), FeatureType(Following, FavoritedBy))\n    )\n  }\n\n  override def handle(): Unit = {\n    warmup.sendRequest(\n      GetIntersection,\n      Request(\n        GetIntersection.Args(\n          getRandomRequest\n        )),\n      10\n    )()\n\n    logger.info(\"Warmup Done!\")\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server/modules/GetIntersectionStoreModule.scala",
    "content": "package com.twitter.graph_feature_service.server.modules\n\nimport com.google.inject.Provides\nimport com.twitter.bijection.scrooge.CompactScalaCodec\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.graph_feature_service.common.Configs._\nimport com.twitter.graph_feature_service.server.stores.GetIntersectionStore\nimport com.twitter.graph_feature_service.server.stores.GetIntersectionStore.GetIntersectionQuery\nimport com.twitter.graph_feature_service.thriftscala.CachedIntersectionResult\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.memcache.MemcacheStore\nimport com.twitter.storehaus_internal.util.{ClientName, ZkEndPoint}\nimport com.twitter.util.Duration\nimport javax.inject.{Named, Singleton}\n\n/**\n * Initialize the MemCache based GetIntersectionStore.\n * The Key of MemCache is UserId~CandidateId~FeatureTypes~IntersectionIdLimit.\n */\nobject GetIntersectionStoreModule extends TwitterModule {\n\n  private[this] val requestTimeout: Duration = 25.millis\n  private[this] val retries: Int = 0\n\n  @Provides\n  @Named(\"ReadThroughGetIntersectionStore\")\n  @Singleton\n  def provideReadThroughGetIntersectionStore(\n    graphFeatureServiceWorkerClients: GraphFeatureServiceWorkerClients,\n    serviceIdentifier: ServiceIdentifier,\n    @Flag(ServerFlagNames.MemCacheClientName) memCacheName: String,\n    @Flag(ServerFlagNames.MemCachePath) memCachePath: String\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): ReadableStore[GetIntersectionQuery, CachedIntersectionResult] = {\n    buildMemcacheStore(\n      graphFeatureServiceWorkerClients,\n      memCacheName,\n      memCachePath,\n      serviceIdentifier)\n  }\n\n  @Provides\n  @Named(\"BypassCacheGetIntersectionStore\")\n  @Singleton\n  def provideReadOnlyGetIntersectionStore(\n    graphFeatureServiceWorkerClients: GraphFeatureServiceWorkerClients,\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): ReadableStore[GetIntersectionQuery, CachedIntersectionResult] = {\n    // Bypass the Memcache.\n    GetIntersectionStore(graphFeatureServiceWorkerClients, statsReceiver)\n  }\n\n  private[this] def buildMemcacheStore(\n    graphFeatureServiceWorkerClients: GraphFeatureServiceWorkerClients,\n    memCacheName: String,\n    memCachePath: String,\n    serviceIdentifier: ServiceIdentifier,\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): ReadableStore[GetIntersectionQuery, CachedIntersectionResult] = {\n    val backingStore = GetIntersectionStore(graphFeatureServiceWorkerClients, statsReceiver)\n\n    val cacheClient = MemcacheStore.memcachedClient(\n      name = ClientName(memCacheName),\n      dest = ZkEndPoint(memCachePath),\n      timeout = requestTimeout,\n      retries = retries,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver\n    )\n\n    ObservedMemcachedReadableStore.fromCacheClient[GetIntersectionQuery, CachedIntersectionResult](\n      backingStore = backingStore,\n      cacheClient = cacheClient,\n      ttl = MemCacheTTL\n    )(\n      valueInjection = LZ4Injection.compose(CompactScalaCodec(CachedIntersectionResult)),\n      statsReceiver = statsReceiver.scope(\"mem_cache\"),\n      keyToString = { key =>\n        s\"L~${key.userId}~${key.candidateId}~${key.featureTypesString}~${key.intersectionIdLimit}\"\n      }\n    )\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server/modules/GraphFeatureServiceWorkerClientsModule.scala",
    "content": "package com.twitter.graph_feature_service.server.modules\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient._\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.service.RetryBudget\nimport com.twitter.graph_feature_service.thriftscala\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.util.{Await, Duration}\nimport javax.inject.Singleton\n\ncase class GraphFeatureServiceWorkerClients(\n  workers: Seq[thriftscala.Worker.MethodPerEndpoint])\n\nobject GraphFeatureServiceWorkerClientsModule extends TwitterModule {\n  private[this] val closeableGracePeriod: Duration = 1.second\n  private[this] val requestTimeout: Duration = 25.millis\n\n  @Provides\n  @Singleton\n  def provideGraphFeatureServiceWorkerClient(\n    @Flag(ServerFlagNames.NumWorkers) numWorkers: Int,\n    @Flag(ServerFlagNames.ServiceRole) serviceRole: String,\n    @Flag(ServerFlagNames.ServiceEnv) serviceEnv: String,\n    serviceIdentifier: ServiceIdentifier\n  ): GraphFeatureServiceWorkerClients = {\n\n    val workers: Seq[thriftscala.Worker.MethodPerEndpoint] =\n      (0 until numWorkers).map { id =>\n        val dest = s\"/srv#/$serviceEnv/local/$serviceRole/graph_feature_service-worker-$id\"\n\n        val client = ThriftMux.client\n          .withRequestTimeout(requestTimeout)\n          .withRetryBudget(RetryBudget.Empty)\n          .withMutualTls(serviceIdentifier)\n          .build[thriftscala.Worker.MethodPerEndpoint](dest, s\"worker-$id\")\n\n        onExit {\n          val closeable = client.asClosable\n          Await.result(closeable.close(closeableGracePeriod), closeableGracePeriod)\n        }\n\n        client\n      }\n\n    GraphFeatureServiceWorkerClients(workers)\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server/modules/LZ4Injection.scala",
    "content": "package com.twitter.graph_feature_service.server.modules\n\nimport com.twitter.bijection.Injection\nimport scala.util.Try\nimport net.jpountz.lz4.{LZ4CompressorWithLength, LZ4DecompressorWithLength, LZ4Factory}\n\nobject LZ4Injection extends Injection[Array[Byte], Array[Byte]] {\n  private val lz4Factory = LZ4Factory.fastestInstance()\n  private val fastCompressor = new LZ4CompressorWithLength(lz4Factory.fastCompressor())\n  private val decompressor = new LZ4DecompressorWithLength(lz4Factory.fastDecompressor())\n\n  override def apply(a: Array[Byte]): Array[Byte] = LZ4Injection.fastCompressor.compress(a)\n\n  override def invert(b: Array[Byte]): Try[Array[Byte]] = Try {\n    LZ4Injection.decompressor.decompress(b)\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server/modules/ServerFlagModule.scala",
    "content": "package com.twitter.graph_feature_service.server.modules\n\nimport com.twitter.inject.TwitterModule\n\nobject ServerFlagNames {\n  final val NumWorkers = \"service.num_workers\"\n  final val ServiceRole = \"service.role\"\n  final val ServiceEnv = \"service.env\"\n\n  final val MemCacheClientName = \"service.mem_cache_client_name\"\n  final val MemCachePath = \"service.mem_cache_path\"\n}\n\n/**\n * Initializes references to the flag values defined in the aurora.deploy file.\n * To check what the flag values are initialized in runtime, search FlagsModule in stdout\n */\nobject ServerFlagsModule extends TwitterModule {\n\n  import ServerFlagNames._\n\n  flag[Int](NumWorkers, \"Num of workers\")\n\n  flag[String](ServiceRole, \"Service Role\")\n\n  flag[String](ServiceEnv, \"Service Env\")\n\n  flag[String](MemCacheClientName, \"MemCache Client Name\")\n\n  flag[String](MemCachePath, \"MemCache Path\")\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server/stores/FeatureTypesEncoder.scala",
    "content": "package com.twitter.graph_feature_service.server.stores\n\nimport com.twitter.graph_feature_service.common.Configs.RandomSeed\nimport com.twitter.graph_feature_service.thriftscala.FeatureType\nimport scala.util.hashing.MurmurHash3\n\nobject FeatureTypesEncoder {\n\n  def apply(featureTypes: Seq[FeatureType]): String = {\n    val byteArray = featureTypes.flatMap { featureType =>\n      Array(featureType.leftEdgeType.getValue.toByte, featureType.rightEdgeType.getValue.toByte)\n    }.toArray\n    (MurmurHash3.bytesHash(byteArray, RandomSeed) & 0x7fffffff).toString // keep positive\n  }\n\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/server/stores/GetIntersectionStore.scala",
    "content": "package com.twitter.graph_feature_service.server.stores\n\nimport com.twitter.finagle.RequestTimeoutException\nimport com.twitter.finagle.stats.{Stat, StatsReceiver}\nimport com.twitter.graph_feature_service.server.handlers.ServerGetIntersectionHandler.GetIntersectionRequest\nimport com.twitter.graph_feature_service.server.modules.GraphFeatureServiceWorkerClients\nimport com.twitter.graph_feature_service.server.stores.GetIntersectionStore.GetIntersectionQuery\nimport com.twitter.graph_feature_service.thriftscala._\nimport com.twitter.inject.Logging\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport javax.inject.Singleton\nimport scala.collection.mutable.ArrayBuffer\n\n@Singleton\ncase class GetIntersectionStore(\n  graphFeatureServiceWorkerClients: GraphFeatureServiceWorkerClients,\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[GetIntersectionQuery, CachedIntersectionResult]\n    with Logging {\n\n  import GetIntersectionStore._\n\n  private val stats = statsReceiver.scope(\"get_intersection_store\")\n  private val requestCount = stats.counter(name = \"request_count\")\n  private val aggregatorLatency = stats.stat(\"aggregator_latency\")\n  private val timeOutCounter = stats.counter(\"worker_timeouts\")\n  private val unknownErrorCounter = stats.counter(\"unknown_errors\")\n\n  override def multiGet[K1 <: GetIntersectionQuery](\n    ks: Set[K1]\n  ): Map[K1, Future[Option[CachedIntersectionResult]]] = {\n    if (ks.isEmpty) {\n      Map.empty\n    } else {\n      requestCount.incr()\n\n      val head = ks.head\n      // We assume all the GetIntersectionQuery use the same userId and featureTypes\n      val userId = head.userId\n      val featureTypes = head.featureTypes\n      val presetFeatureTypes = head.presetFeatureTypes\n      val calculatedFeatureTypes = head.calculatedFeatureTypes\n      val intersectionIdLimit = head.intersectionIdLimit\n\n      val request = WorkerIntersectionRequest(\n        userId,\n        ks.map(_.candidateId).toArray,\n        featureTypes,\n        presetFeatureTypes,\n        intersectionIdLimit\n      )\n\n      val resultFuture = Future\n        .collect(\n          graphFeatureServiceWorkerClients.workers.map { worker =>\n            worker\n              .getIntersection(request)\n              .rescue {\n                case _: RequestTimeoutException =>\n                  timeOutCounter.incr()\n                  Future.value(DefaultWorkerIntersectionResponse)\n                case e =>\n                  unknownErrorCounter.incr()\n                  logger.error(\"Failure to load result.\", e)\n                  Future.value(DefaultWorkerIntersectionResponse)\n              }\n          }\n        ).map { responses =>\n          Stat.time(aggregatorLatency) {\n            gfsIntersectionResponseAggregator(\n              responses,\n              calculatedFeatureTypes,\n              request.candidateUserIds,\n              intersectionIdLimit\n            )\n          }\n        }\n\n      ks.map { query =>\n        query -> resultFuture.map(_.get(query.candidateId))\n      }.toMap\n    }\n  }\n\n  /**\n   * Function to merge GfsIntersectionResponse from workers into one result.\n   */\n  private def gfsIntersectionResponseAggregator(\n    responseList: Seq[WorkerIntersectionResponse],\n    features: Seq[FeatureType],\n    candidates: Seq[Long],\n    intersectionIdLimit: Int\n  ): Map[Long, CachedIntersectionResult] = {\n\n    // Map of (candidate -> features -> type -> value)\n    val cube = Array.fill[Int](candidates.length, features.length, 3)(0)\n    // Map of (candidate -> features -> intersectionIds)\n    val ids = Array.fill[Option[ArrayBuffer[Long]]](candidates.length, features.length)(None)\n    val notZero = intersectionIdLimit != 0\n\n    for {\n      response <- responseList\n      (features, candidateIndex) <- response.results.zipWithIndex\n      (workerValue, featureIndex) <- features.zipWithIndex\n    } {\n      cube(candidateIndex)(featureIndex)(CountIndex) += workerValue.count\n      cube(candidateIndex)(featureIndex)(LeftDegreeIndex) += workerValue.leftNodeDegree\n      cube(candidateIndex)(featureIndex)(RightDegreeIndex) += workerValue.rightNodeDegree\n\n      if (notZero && workerValue.intersectionIds.nonEmpty) {\n        val arrayBuffer = ids(candidateIndex)(featureIndex) match {\n          case Some(buffer) => buffer\n          case None =>\n            val buffer = ArrayBuffer[Long]()\n            ids(candidateIndex)(featureIndex) = Some(buffer)\n            buffer\n        }\n        val intersectionIds = workerValue.intersectionIds\n\n        // Scan the intersectionId based on the Shard. The response order is consistent.\n        if (arrayBuffer.size < intersectionIdLimit) {\n          if (intersectionIds.size > intersectionIdLimit - arrayBuffer.size) {\n            arrayBuffer ++= intersectionIds.slice(0, intersectionIdLimit - arrayBuffer.size)\n          } else {\n            arrayBuffer ++= intersectionIds\n          }\n        }\n      }\n    }\n\n    candidates.zipWithIndex.map {\n      case (candidate, candidateIndex) =>\n        candidate -> CachedIntersectionResult(features.indices.map { featureIndex =>\n          WorkerIntersectionValue(\n            cube(candidateIndex)(featureIndex)(CountIndex),\n            cube(candidateIndex)(featureIndex)(LeftDegreeIndex),\n            cube(candidateIndex)(featureIndex)(RightDegreeIndex),\n            ids(candidateIndex)(featureIndex).getOrElse(Nil)\n          )\n        })\n    }.toMap\n  }\n\n}\n\nobject GetIntersectionStore {\n\n  private[graph_feature_service] case class GetIntersectionQuery(\n    userId: Long,\n    candidateId: Long,\n    featureTypes: Seq[FeatureType],\n    presetFeatureTypes: PresetFeatureTypes,\n    featureTypesString: String,\n    calculatedFeatureTypes: Seq[FeatureType],\n    intersectionIdLimit: Int)\n\n  private[graph_feature_service] object GetIntersectionQuery {\n    def buildQueries(request: GetIntersectionRequest): Set[GetIntersectionQuery] = {\n      request.candidateUserIds.toSet.map { candidateId: Long =>\n        GetIntersectionQuery(\n          request.userId,\n          candidateId,\n          request.featureTypes,\n          request.presetFeatureTypes,\n          request.calculatedFeatureTypesString,\n          request.calculatedFeatureTypes,\n          request.intersectionIdLimit.getOrElse(DefaultIntersectionIdLimit)\n        )\n      }\n    }\n  }\n\n  // Don't return the intersectionId for better performance\n  private val DefaultIntersectionIdLimit = 0\n  private val DefaultWorkerIntersectionResponse = WorkerIntersectionResponse()\n\n  private val CountIndex = 0\n  private val LeftDegreeIndex = 1\n  private val RightDegreeIndex = 2\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/util/BUILD",
    "content": "scala_library(\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"graph-feature-service/src/main/thrift/com/twitter/graph_feature_service:graph_feature_service_thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/util/FeatureTypesCalculator.scala",
    "content": "package com.twitter.graph_feature_service.util\n\nimport com.twitter.graph_feature_service.thriftscala.EdgeType._\nimport com.twitter.graph_feature_service.thriftscala.{FeatureType, PresetFeatureTypes}\n\nobject FeatureTypesCalculator {\n\n  final val DefaultTwoHop = Seq(\n    FeatureType(Following, FollowedBy),\n    FeatureType(Following, FavoritedBy),\n    FeatureType(Following, RetweetedBy),\n    FeatureType(Following, MentionedBy),\n    FeatureType(Following, MutualFollow),\n    FeatureType(Favorite, FollowedBy),\n    FeatureType(Favorite, FavoritedBy),\n    FeatureType(Favorite, RetweetedBy),\n    FeatureType(Favorite, MentionedBy),\n    FeatureType(Favorite, MutualFollow),\n    FeatureType(MutualFollow, FollowedBy),\n    FeatureType(MutualFollow, FavoritedBy),\n    FeatureType(MutualFollow, RetweetedBy),\n    FeatureType(MutualFollow, MentionedBy),\n    FeatureType(MutualFollow, MutualFollow)\n  )\n\n  final val SocialProofTwoHop = Seq(FeatureType(Following, FollowedBy))\n\n  final val HtlTwoHop = DefaultTwoHop\n\n  final val WtfTwoHop = SocialProofTwoHop\n\n  final val SqTwoHop = DefaultTwoHop\n\n  final val RuxTwoHop = DefaultTwoHop\n\n  final val MRTwoHop = DefaultTwoHop\n\n  final val UserTypeaheadTwoHop = SocialProofTwoHop\n\n  final val presetFeatureTypes =\n    (HtlTwoHop ++ WtfTwoHop ++ SqTwoHop ++ RuxTwoHop ++ MRTwoHop ++ UserTypeaheadTwoHop).toSet\n\n  def getFeatureTypes(\n    presetFeatureTypes: PresetFeatureTypes,\n    featureTypes: Seq[FeatureType]\n  ): Seq[FeatureType] = {\n    presetFeatureTypes match {\n      case PresetFeatureTypes.HtlTwoHop => HtlTwoHop\n      case PresetFeatureTypes.WtfTwoHop => WtfTwoHop\n      case PresetFeatureTypes.SqTwoHop => SqTwoHop\n      case PresetFeatureTypes.RuxTwoHop => RuxTwoHop\n      case PresetFeatureTypes.MrTwoHop => MRTwoHop\n      case PresetFeatureTypes.UserTypeaheadTwoHop => UserTypeaheadTwoHop\n      case _ => featureTypes\n    }\n  }\n\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/util/IntersectionValueCalculator.scala",
    "content": "package com.twitter.graph_feature_service.util\n\nimport com.twitter.graph_feature_service.thriftscala.{\n  FeatureType,\n  IntersectionValue,\n  WorkerIntersectionValue\n}\nimport java.nio.ByteBuffer\nimport scala.collection.mutable.ArrayBuffer\n\n/**\n * Functions for computing feature values based on the values returned by constantDB.\n */\nobject IntersectionValueCalculator {\n\n  /**\n   * Compute the size of the array in a ByteBuffer.\n   * Note that this function assumes the ByteBuffer is encoded using Injections.seqLong2ByteBuffer\n   */\n  def computeArraySize(x: ByteBuffer): Int = {\n    x.remaining() >> 3 // divide 8\n  }\n\n  /**\n   *\n   */\n  def apply(x: ByteBuffer, y: ByteBuffer, intersectionIdLimit: Int): WorkerIntersectionValue = {\n\n    val xSize = computeArraySize(x)\n    val ySize = computeArraySize(y)\n\n    val largerArray = if (xSize > ySize) x else y\n    val smallerArray = if (xSize > ySize) y else x\n\n    if (intersectionIdLimit == 0) {\n      val result = computeIntersectionUsingBinarySearchOnLargerByteBuffer(smallerArray, largerArray)\n      WorkerIntersectionValue(result, xSize, ySize)\n    } else {\n      val (result, ids) = computeIntersectionWithIds(smallerArray, largerArray, intersectionIdLimit)\n      WorkerIntersectionValue(result, xSize, ySize, ids)\n    }\n  }\n\n  /**\n   * Note that this function assumes the ByteBuffer is encoded using Injections.seqLong2ByteBuffer\n   *\n   */\n  def computeIntersectionUsingBinarySearchOnLargerByteBuffer(\n    smallArray: ByteBuffer,\n    largeArray: ByteBuffer\n  ): Int = {\n    var res: Int = 0\n    var i: Int = 0\n\n    while (i < smallArray.remaining()) {\n      if (binarySearch(largeArray, smallArray.getLong(i)) >= 0) {\n        res += 1\n      }\n      i += 8\n    }\n    res\n  }\n\n  def computeIntersectionWithIds(\n    smallArray: ByteBuffer,\n    largeArray: ByteBuffer,\n    intersectionLimit: Int\n  ): (Int, Seq[Long]) = {\n    var res: Int = 0\n    var i: Int = 0\n    // Most of the intersectionLimit is smaller than default size: 16\n    val idBuffer = ArrayBuffer[Long]()\n\n    while (i < smallArray.remaining()) {\n      val value = smallArray.getLong(i)\n      if (binarySearch(largeArray, value) >= 0) {\n        res += 1\n        // Always get the smaller ids\n        if (idBuffer.size < intersectionLimit) {\n          idBuffer += value\n        }\n      }\n      i += 8\n    }\n    (res, idBuffer)\n  }\n\n  /**\n   * Note that this function assumes the ByteBuffer is encoded using Injections.seqLong2ByteBuffer\n   *\n   */\n  private[util] def binarySearch(arr: ByteBuffer, value: Long): Int = {\n    var start = 0\n    var end = arr.remaining()\n\n    while (start <= end && start < arr.remaining()) {\n      val mid = ((start + end) >> 1) & ~7 // take mid - mid % 8\n      if (arr.getLong(mid) == value) {\n        return mid // return the index of the value\n      } else if (arr.getLong(mid) < value) {\n        start = mid + 8\n      } else {\n        end = mid - 1\n      }\n    }\n    // if not existed, return -1\n    -1\n  }\n\n  /**\n   * TODO: for now it only computes intersection size. Will add more feature types (e.g., dot\n   * product, maximum value).\n   *\n   * NOTE that this function assumes both x and y are SORTED arrays.\n   * In graph feature service, the sorting is done in the offline Scalding job.\n   *\n   * @param x                     source user's array\n   * @param y                     candidate user's array\n   * @param featureType           feature type\n   * @return\n   */\n  def apply(x: Array[Long], y: Array[Long], featureType: FeatureType): IntersectionValue = {\n\n    val xSize = x.length\n    val ySize = y.length\n\n    val intersection =\n      if (xSize.min(ySize) * math.log(xSize.max(ySize)) < (xSize + ySize).toDouble) {\n        if (xSize < ySize) {\n          computeIntersectionUsingBinarySearchOnLargerArray(x, y)\n        } else {\n          computeIntersectionUsingBinarySearchOnLargerArray(y, x)\n        }\n      } else {\n        computeIntersectionUsingListMerging(x, y)\n      }\n\n    IntersectionValue(\n      featureType,\n      Some(intersection.toInt),\n      None, // return None for now\n      Some(xSize),\n      Some(ySize)\n    )\n  }\n\n  /**\n   * Function for computing the intersections of two SORTED arrays by list merging.\n   *\n   * @param x one array\n   * @param y another array\n   * @param ordering ordering function for comparing values of T\n   * @tparam T type\n   * @return The intersection size and the list of intersected elements\n   */\n  private[util] def computeIntersectionUsingListMerging[T](\n    x: Array[T],\n    y: Array[T]\n  )(\n    implicit ordering: Ordering[T]\n  ): Int = {\n\n    var res: Int = 0\n    var i: Int = 0\n    var j: Int = 0\n\n    while (i < x.length && j < y.length) {\n      val comp = ordering.compare(x(i), y(j))\n      if (comp > 0) j += 1\n      else if (comp < 0) i += 1\n      else {\n        res += 1\n        i += 1\n        j += 1\n      }\n    }\n    res\n  }\n\n  /**\n   * Function for computing the intersections of two arrays by binary search on the larger array.\n   * Note that the larger array MUST be SORTED.\n   *\n   * @param smallArray            smaller array\n   * @param largeArray            larger array\n   * @param ordering ordering function for comparing values of T\n   * @tparam T type\n   *\n   * @return The intersection size and the list of intersected elements\n   */\n  private[util] def computeIntersectionUsingBinarySearchOnLargerArray[T](\n    smallArray: Array[T],\n    largeArray: Array[T]\n  )(\n    implicit ordering: Ordering[T]\n  ): Int = {\n    var res: Int = 0\n    var i: Int = 0\n    while (i < smallArray.length) {\n      val currentValue: T = smallArray(i)\n      if (binarySearch(largeArray, currentValue) >= 0) {\n        res += 1\n      }\n      i += 1\n    }\n    res\n  }\n\n  /**\n   * Function for doing the binary search\n   *\n   * @param arr array\n   * @param value the target value for searching\n   * @param ordering ordering function\n   * @tparam T type\n   * @return the index of element in the larger array.\n   *         If there is no such element in the array, return -1.\n   */\n  private[util] def binarySearch[T](\n    arr: Array[T],\n    value: T\n  )(\n    implicit ordering: Ordering[T]\n  ): Int = {\n    var start = 0\n    var end = arr.length - 1\n\n    while (start <= end) {\n      val mid = (start + end) >> 1\n      val comp = ordering.compare(arr(mid), value)\n      if (comp == 0) {\n        return mid // return the index of the value\n      } else if (comp < 0) {\n        start = mid + 1\n      } else {\n        end = mid - 1\n      }\n    }\n    // if not existed, return -1\n    -1\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"**/*.scala\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"discovery-common/src/main/scala/com/twitter/discovery/common/stats\",\n        \"finatra-internal/decider/src/main/scala\",\n        \"finatra-internal/gizmoduck/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-server/src/main/scala\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"finatra/inject/inject-utils/src/main/scala\",\n        \"frigate/frigate-common:constdb_util\",\n        \"graph-feature-service/src/main/resources\",\n        \"graph-feature-service/src/main/scala/com/twitter/graph_feature_service/common\",\n        \"graph-feature-service/src/main/scala/com/twitter/graph_feature_service/util\",\n        \"graph-feature-service/src/main/thrift/com/twitter/graph_feature_service:graph_feature_service_thrift-scala\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"servo/request/src/main/scala\",\n        \"twitter-server-internal/src/main/scala\",\n        \"twitter-server/server/src/main/scala\",\n        \"util/util-app/src/main/scala\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker/Main.scala",
    "content": "package com.twitter.graph_feature_service.worker\n\nimport com.google.inject.Module\nimport com.twitter.finatra.decider.modules.DeciderModule\nimport com.twitter.finatra.gizmoduck.modules.TimerModule\nimport com.twitter.finatra.mtls.thriftmux.Mtls\nimport com.twitter.finatra.thrift.ThriftServer\nimport com.twitter.finatra.thrift.filters.{\n  LoggingMDCFilter,\n  StatsFilter,\n  ThriftMDCFilter,\n  TraceIdMDCFilter\n}\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsThriftWebFormsModule\nimport com.twitter.finatra.thrift.routing.ThriftRouter\nimport com.twitter.graph_feature_service.thriftscala\nimport com.twitter.graph_feature_service.worker.controllers.WorkerController\nimport com.twitter.graph_feature_service.worker.handlers.WorkerWarmupHandler\nimport com.twitter.graph_feature_service.worker.modules.{\n  GraphContainerProviderModule,\n  WorkerFlagModule\n}\nimport com.twitter.graph_feature_service.worker.util.GraphContainer\nimport com.twitter.inject.thrift.modules.ThriftClientIdModule\nimport com.twitter.util.Await\n\nobject Main extends WorkerMain\n\nclass WorkerMain extends ThriftServer with Mtls {\n\n  override val name = \"graph_feature_service-worker\"\n\n  override val modules: Seq[Module] = {\n    Seq(\n      WorkerFlagModule,\n      DeciderModule,\n      TimerModule,\n      ThriftClientIdModule,\n      GraphContainerProviderModule,\n      new MtlsThriftWebFormsModule[thriftscala.Worker.MethodPerEndpoint](this)\n    )\n  }\n\n  override def configureThrift(router: ThriftRouter): Unit = {\n    router\n      .filter[LoggingMDCFilter]\n      .filter[TraceIdMDCFilter]\n      .filter[ThriftMDCFilter]\n      .filter[StatsFilter]\n      .add[WorkerController]\n  }\n\n  override protected def warmup(): Unit = {\n    val graphContainer = injector.instance[GraphContainer]\n    Await.result(graphContainer.warmup)\n    handle[WorkerWarmupHandler]()\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker/controllers/WorkerController.scala",
    "content": "package com.twitter.graph_feature_service.worker.controllers\n\nimport com.twitter.discovery.common.stats.DiscoveryStatsFilter\nimport com.twitter.finagle.Service\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.thrift.Controller\nimport com.twitter.graph_feature_service.thriftscala\nimport com.twitter.graph_feature_service.thriftscala.Worker.GetIntersection\nimport com.twitter.graph_feature_service.thriftscala._\nimport com.twitter.graph_feature_service.worker.handlers._\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass WorkerController @Inject() (\n  workerGetIntersectionHandler: WorkerGetIntersectionHandler\n)(\n  implicit statsReceiver: StatsReceiver)\n    extends Controller(thriftscala.Worker) {\n\n  // use DiscoveryStatsFilter to filter out exceptions out of our control\n  private val getIntersectionService: Service[\n    WorkerIntersectionRequest,\n    WorkerIntersectionResponse\n  ] =\n    new DiscoveryStatsFilter[WorkerIntersectionRequest, WorkerIntersectionResponse](\n      statsReceiver.scope(\"srv\").scope(\"get_intersection\")\n    ).andThen(Service.mk(workerGetIntersectionHandler))\n\n  val getIntersection: Service[GetIntersection.Args, WorkerIntersectionResponse] = { args =>\n    getIntersectionService(args.request).onFailure { throwable =>\n      logger.error(s\"Failure to get intersection for request $args.\", throwable)\n    }\n  }\n\n  handle(GetIntersection) { getIntersection }\n\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker/handlers/WorkerGetIntersectionHandler.scala",
    "content": "package com.twitter.graph_feature_service.worker.handlers\n\nimport com.twitter.finagle.stats.{Stat, StatsReceiver}\nimport com.twitter.graph_feature_service.thriftscala.{\n  WorkerIntersectionRequest,\n  WorkerIntersectionResponse,\n  WorkerIntersectionValue\n}\nimport com.twitter.graph_feature_service.util.{FeatureTypesCalculator, IntersectionValueCalculator}\nimport com.twitter.graph_feature_service.util.IntersectionValueCalculator._\nimport com.twitter.graph_feature_service.worker.util.GraphContainer\nimport com.twitter.servo.request.RequestHandler\nimport com.twitter.util.Future\nimport java.nio.ByteBuffer\nimport javax.inject.{Inject, Singleton}\n\n@Singleton\nclass WorkerGetIntersectionHandler @Inject() (\n  graphContainer: GraphContainer,\n  statsReceiver: StatsReceiver)\n    extends RequestHandler[WorkerIntersectionRequest, WorkerIntersectionResponse] {\n\n  import WorkerGetIntersectionHandler._\n\n  private val stats: StatsReceiver = statsReceiver.scope(\"srv/get_intersection\")\n  private val numCandidatesCount = stats.counter(\"total_num_candidates\")\n  private val toPartialGraphQueryStat = stats.stat(\"to_partial_graph_query_latency\")\n  private val fromPartialGraphQueryStat = stats.stat(\"from_partial_graph_query_latency\")\n  private val intersectionCalculationStat = stats.stat(\"computation_latency\")\n\n  override def apply(request: WorkerIntersectionRequest): Future[WorkerIntersectionResponse] = {\n\n    numCandidatesCount.incr(request.candidateUserIds.length)\n\n    val userId = request.userId\n\n    // NOTE: do not change the order of candidates\n    val candidateIds = request.candidateUserIds\n\n    // NOTE: do not change the order of features\n    val featureTypes =\n      FeatureTypesCalculator.getFeatureTypes(request.presetFeatureTypes, request.featureTypes)\n\n    val leftEdges = featureTypes.map(_.leftEdgeType).distinct\n    val rightEdges = featureTypes.map(_.rightEdgeType).distinct\n\n    val rightEdgeMap = Stat.time(toPartialGraphQueryStat) {\n      rightEdges.map { rightEdge =>\n        val map = graphContainer.toPartialMap.get(rightEdge) match {\n          case Some(graph) =>\n            candidateIds.flatMap { candidateId =>\n              graph.apply(candidateId).map(candidateId -> _)\n            }.toMap\n          case None =>\n            Map.empty[Long, ByteBuffer]\n        }\n        rightEdge -> map\n      }.toMap\n    }\n\n    val leftEdgeMap = Stat.time(fromPartialGraphQueryStat) {\n      leftEdges.flatMap { leftEdge =>\n        graphContainer.toPartialMap.get(leftEdge).flatMap(_.apply(userId)).map(leftEdge -> _)\n      }.toMap\n    }\n\n    val res = Stat.time(intersectionCalculationStat) {\n      WorkerIntersectionResponse(\n        // NOTE that candidate ordering is important\n        candidateIds.map { candidateId =>\n          // NOTE that the featureTypes ordering is important\n          featureTypes.map {\n            featureType =>\n              val leftNeighborsOpt = leftEdgeMap.get(featureType.leftEdgeType)\n              val rightNeighborsOpt =\n                rightEdgeMap.get(featureType.rightEdgeType).flatMap(_.get(candidateId))\n\n              if (leftNeighborsOpt.isEmpty && rightNeighborsOpt.isEmpty) {\n                EmptyWorkerIntersectionValue\n              } else if (rightNeighborsOpt.isEmpty) {\n                EmptyWorkerIntersectionValue.copy(\n                  leftNodeDegree = computeArraySize(leftNeighborsOpt.get)\n                )\n              } else if (leftNeighborsOpt.isEmpty) {\n                EmptyWorkerIntersectionValue.copy(\n                  rightNodeDegree = computeArraySize(rightNeighborsOpt.get)\n                )\n              } else {\n                IntersectionValueCalculator(\n                  leftNeighborsOpt.get,\n                  rightNeighborsOpt.get,\n                  request.intersectionIdLimit)\n              }\n          }\n        }\n      )\n    }\n\n    Future.value(res)\n  }\n}\n\nobject WorkerGetIntersectionHandler {\n  val EmptyWorkerIntersectionValue: WorkerIntersectionValue = WorkerIntersectionValue(0, 0, 0, Nil)\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker/handlers/WorkerWarmupHandler.scala",
    "content": "package com.twitter.graph_feature_service.worker.handlers\n\nimport com.twitter.finatra.thrift.routing.ThriftWarmup\nimport com.twitter.inject.Logging\nimport com.twitter.inject.utils.Handler\nimport javax.inject.{Inject, Singleton}\n\n@Singleton\nclass WorkerWarmupHandler @Inject() (warmup: ThriftWarmup) extends Handler with Logging {\n\n  override def handle(): Unit = {\n    info(\"Warmup Done!\")\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker/modules/GraphContainerProviderModule.scala",
    "content": "package com.twitter.graph_feature_service.worker.modules\n\nimport com.google.inject.Provides\nimport com.twitter.concurrent.AsyncSemaphore\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.graph_feature_service.common.Configs._\nimport com.twitter.graph_feature_service.worker.util\nimport com.twitter.graph_feature_service.worker.util.AutoUpdatingGraph\nimport com.twitter.graph_feature_service.worker.util.FollowedByPartialValueGraph\nimport com.twitter.graph_feature_service.worker.util.FollowingPartialValueGraph\nimport com.twitter.graph_feature_service.worker.util.GraphContainer\nimport com.twitter.graph_feature_service.worker.util.GraphKey\nimport com.twitter.graph_feature_service.worker.util.MutualFollowPartialValueGraph\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.util.Timer\nimport javax.inject.Singleton\n\nobject GraphContainerProviderModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def provideAutoUpdatingGraphs(\n    @Flag(WorkerFlagNames.HdfsCluster) hdfsCluster: String,\n    @Flag(WorkerFlagNames.HdfsClusterUrl) hdfsClusterUrl: String,\n    @Flag(WorkerFlagNames.ShardId) shardId: Int\n  )(\n    implicit statsReceiver: StatsReceiver,\n    timer: Timer\n  ): GraphContainer = {\n\n    // NOTE that we do not load some the graphs for saving RAM at this moment.\n    val enabledGraphPaths: Map[GraphKey, String] =\n      Map(\n        FollowingPartialValueGraph -> FollowOutValPath,\n        FollowedByPartialValueGraph -> FollowInValPath\n      )\n\n    // Only allow one graph to update at the same time.\n    val sharedSemaphore = new AsyncSemaphore(1)\n\n    val graphs: Map[GraphKey, AutoUpdatingGraph] =\n      enabledGraphPaths.map {\n        case (graphKey, path) =>\n          graphKey -> AutoUpdatingGraph(\n            dataPath = getHdfsPath(path),\n            hdfsCluster = hdfsCluster,\n            hdfsClusterUrl = hdfsClusterUrl,\n            shard = shardId,\n            minimumSizeForCompleteGraph = 1e6.toLong,\n            sharedSemaphore = Some(sharedSemaphore)\n          )(\n            statsReceiver\n              .scope(\"graphs\")\n              .scope(graphKey.getClass.getSimpleName),\n            timer\n          )\n      }\n\n    util.GraphContainer(graphs)\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker/modules/WorkerFlagModule.scala",
    "content": "package com.twitter.graph_feature_service.worker.modules\n\nimport com.twitter.inject.TwitterModule\n\nobject WorkerFlagNames {\n  final val ServiceRole = \"service.role\"\n  final val ServiceEnv = \"service.env\"\n  final val ShardId = \"service.shardId\"\n  final val NumShards = \"service.numShards\"\n  final val HdfsCluster = \"service.hdfsCluster\"\n  final val HdfsClusterUrl = \"service.hdfsClusterUrl\"\n}\n\n/**\n * Initializes references to the flag values defined in the aurora.deploy file.\n * To check what the flag values are initialized in runtime, search FlagsModule in stdout\n */\nobject WorkerFlagModule extends TwitterModule {\n\n  import WorkerFlagNames._\n\n  flag[Int](ShardId, \"Shard Id\")\n\n  flag[Int](NumShards, \"Num of Graph Shards\")\n\n  flag[String](ServiceRole, \"Service Role\")\n\n  flag[String](ServiceEnv, \"Service Env\")\n\n  flag[String](HdfsCluster, \"Hdfs cluster to download graph files from\")\n\n  flag[String](HdfsClusterUrl, \"Hdfs cluster url to download graph files from\")\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker/util/AutoUpdatingGraph.scala",
    "content": "package com.twitter.graph_feature_service.worker.util\n\nimport com.twitter.bijection.Injection\nimport com.twitter.concurrent.AsyncSemaphore\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.constdb_util.{\n  AutoUpdatingReadOnlyGraph,\n  ConstDBImporter,\n  Injections\n}\nimport com.twitter.graph_feature_service.common.Configs\nimport com.twitter.util.{Duration, Future, Timer}\nimport java.nio.ByteBuffer\n\n/**\n * @param dataPath                    the path to the data on HDFS\n * @param hdfsCluster                 cluster where we check for updates and download graph files from\n * @param hdfsClusterUrl              url to HDFS cluster\n * @param shard                       The shard of the graph to download\n * @param minimumSizeForCompleteGraph minimumSize for complete graph - otherwise we don't load it\n * @param updateIntervalMin           The interval after which the first update is tried and the interval between such updates\n * @param updateIntervalMax           the maximum time before an update is triggered\n * @param deleteInterval              The interval after which older data is deleted from disk\n * @param sharedSemaphore             The semaphore controls the number of graph loads at same time on the instance.\n */\ncase class AutoUpdatingGraph(\n  dataPath: String,\n  hdfsCluster: String,\n  hdfsClusterUrl: String,\n  shard: Int,\n  minimumSizeForCompleteGraph: Long,\n  updateIntervalMin: Duration = 1.hour,\n  updateIntervalMax: Duration = 12.hours,\n  deleteInterval: Duration = 2.seconds,\n  sharedSemaphore: Option[AsyncSemaphore] = None\n)(\n  implicit statsReceiver: StatsReceiver,\n  timer: Timer)\n    extends AutoUpdatingReadOnlyGraph[Long, ByteBuffer](\n      hdfsCluster,\n      hdfsClusterUrl,\n      shard,\n      minimumSizeForCompleteGraph,\n      updateIntervalMin,\n      updateIntervalMax,\n      deleteInterval,\n      sharedSemaphore\n    )\n    with ConstDBImporter[Long, ByteBuffer] {\n\n  override def numGraphShards: Int = Configs.NumGraphShards\n\n  override def basePath: String = dataPath\n\n  override val keyInj: Injection[Long, ByteBuffer] = Injections.long2Varint\n\n  override val valueInj: Injection[ByteBuffer, ByteBuffer] = Injection.identity\n\n  override def get(targetId: Long): Future[Option[ByteBuffer]] =\n    super\n      .get(targetId)\n      .map { res =>\n        res.foreach(r => arraySizeStat.add(r.remaining()))\n        res\n      }\n\n  private val arraySizeStat = stats.scope(\"get\").stat(\"size\")\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker/util/GfsQuery.scala",
    "content": "package com.twitter.graph_feature_service.worker.util\n\nimport com.twitter.graph_feature_service.thriftscala.EdgeType\n\nsealed trait GfsQuery {\n  def edgeType: EdgeType\n  def userId: Long\n}\n\n/**\n * Search for edges for any users to users in local partition.\n */\ncase class ToPartialQuery(edgeType: EdgeType, userId: Long) extends GfsQuery\n\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker/util/GraphContainer.scala",
    "content": "package com.twitter.graph_feature_service.worker.util\n\nimport com.twitter.graph_feature_service.thriftscala.EdgeType\nimport com.twitter.util.Future\n\ncase class GraphContainer(\n  graphs: Map[GraphKey, AutoUpdatingGraph]) {\n\n  final val toPartialMap: Map[EdgeType, AutoUpdatingGraph] =\n    graphs.collect {\n      case (partialValueGraph: PartialValueGraph, graph) =>\n        partialValueGraph.edgeType -> graph\n    }\n\n  // load all the graphs from constantDB format to memory\n  def warmup: Future[Unit] = {\n    Future.collect(graphs.mapValues(_.warmup())).unit\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker/util/GraphKey.scala",
    "content": "package com.twitter.graph_feature_service.worker.util\n\nimport com.twitter.graph_feature_service.thriftscala.EdgeType\nimport com.twitter.graph_feature_service.thriftscala.EdgeType._\n\nsealed trait GraphKey {\n\n  def edgeType: EdgeType\n}\n\nsealed trait PartialValueGraph extends GraphKey\n\n/**\n * Follow Graphs\n */\nobject FollowingPartialValueGraph extends PartialValueGraph {\n\n  override def edgeType: EdgeType = Following\n}\n\nobject FollowedByPartialValueGraph extends PartialValueGraph {\n\n  override def edgeType: EdgeType = FollowedBy\n}\n\n/**\n * Mutual Follow Graphs\n */\nobject MutualFollowPartialValueGraph extends PartialValueGraph {\n\n  override def edgeType: EdgeType = MutualFollow\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scala/com/twitter/graph_feature_service/worker/util/GraphType.scala",
    "content": "package com.twitter.graph_feature_service.worker.util\n\n//These classes are to help the GraphContainer choose the right data structure to answer queries\nsealed trait GraphType\n\nobject FollowGraph extends GraphType\n\nobject FavoriteGraph extends GraphType\n\nobject RetweetGraph extends GraphType\n\nobject ReplyGraph extends GraphType\n\nobject MentionGraph extends GraphType\n\nobject MutualFollowGraph extends GraphType\n"
  },
  {
    "path": "graph-feature-service/src/main/scalding/com/twitter/graph_feature_service/scalding/BUILD.bazel",
    "content": "scala_library(\n    platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/bijection:core\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/constdb_util\",\n        \"graph-feature-service/src/main/scala/com/twitter/graph_feature_service/common\",\n        \"src/scala/com/twitter/interaction_graph/scio/agg_all:interaction_graph_history_aggregated_edge_snapshot-scala\",\n        \"src/scala/com/twitter/interaction_graph/scio/ml/scores:real_graph_in_scores-scala\",\n        \"src/scala/com/twitter/pluck/source/user_audits:user_audit_final-scala\",\n        \"src/scala/com/twitter/scalding_internal/dalv2\",\n        \"src/scala/com/twitter/scalding_internal/job\",\n        \"src/scala/com/twitter/scalding_internal/job/analytics_batch\",\n    ],\n)\n\nscalding_job(\n    name = \"graph_feature_service_adhoc_job\",\n    main = \"com.twitter.graph_feature_service.scalding.GraphFeatureServiceAdhocApp\",\n    args = [\n        \"--date 2022-10-24\",\n    ],\n    config = [\n        (\"hadoop.map.jvm.total-memory\", \"3072m\"),\n        (\"hadoop.reduce.jvm.total-memory\", \"3072m\"),\n        (\"hadoop.submitter.jvm.total-memory\", \"5120m\"),\n        (\"submitter.tier\", \"preemptible\"),\n    ],\n    contact = \"recos-platform-alerts@twitter.com\",\n    hadoop_cluster = \"atla-proc\",\n    hadoop_properties = [(\"mapreduce.job.hdfs-servers\", \"/atla/proc/user/cassowary\")],\n    platform = \"java8\",\n    role = \"cassowary\",\n    runtime_platform = \"java8\",\n    tags = [\n        \"bazel-compatible:migrated\",\n        \"bazel-only\",\n    ],\n    dependencies = [\":scalding\"],\n)\n\nscalding_job(\n    name = \"graph_feature_service_daily_job\",\n    main = \"com.twitter.graph_feature_service.scalding.GraphFeatureServiceScheduledApp\",\n    config = [\n        (\"hadoop.map.jvm.total-memory\", \"3072m\"),\n        (\"hadoop.reduce.jvm.total-memory\", \"3072m\"),\n        (\"hadoop.submitter.jvm.total-memory\", \"5120m\"),\n        (\"submitter.tier\", \"preemptible\"),\n    ],\n    contact = \"recos-platform-alerts@twitter.com\",\n    cron = \"01,31 * * * *\",\n    hadoop_cluster = \"atla-proc\",\n    hadoop_properties = [(\"mapreduce.job.hdfs-servers\", \"/atla/proc/user/cassowary\")],\n    platform = \"java8\",\n    role = \"cassowary\",\n    runtime_platform = \"java8\",\n    tags = [\n        \"bazel-compatible:migrated\",\n        \"bazel-only\",\n    ],\n    dependencies = [\":scalding\"],\n)\n"
  },
  {
    "path": "graph-feature-service/src/main/scalding/com/twitter/graph_feature_service/scalding/EdgeFeature.scala",
    "content": "package com.twitter.graph_feature_service.scalding\n\ncase class EdgeFeature(\n  realGraphScore: Float,\n  followScore: Option[Float] = None,\n  mutualFollowScore: Option[Float] = None,\n  favoriteScore: Option[Float] = None,\n  retweetScore: Option[Float] = None,\n  mentionScore: Option[Float] = None)\n"
  },
  {
    "path": "graph-feature-service/src/main/scalding/com/twitter/graph_feature_service/scalding/GraphFeatureServiceAppBase.scala",
    "content": "package com.twitter.graph_feature_service.scalding\n\nimport com.twitter.scalding._\nimport com.twitter.scalding_internal.job.TwitterExecutionApp\nimport com.twitter.scalding_internal.job.analytics_batch.{\n  AnalyticsBatchExecution,\n  AnalyticsBatchExecutionArgs,\n  BatchDescription,\n  BatchFirstTime,\n  BatchIncrement,\n  TwitterScheduledExecutionApp\n}\nimport java.util.TimeZone\n\n/**\n * Each job only needs to implement this runOnDateRange() function. It makes it easier for testing.\n */\ntrait GraphFeatureServiceBaseJob {\n  implicit val timeZone: TimeZone = DateOps.UTC\n  implicit val dateParser: DateParser = DateParser.default\n\n  def runOnDateRange(\n    enableValueGraphs: Option[Boolean] = None,\n    enableKeyGraphs: Option[Boolean] = None\n  )(\n    implicit dateRange: DateRange,\n    timeZone: TimeZone,\n    uniqueID: UniqueID\n  ): Execution[Unit]\n\n  /**\n   * Print customized counters in the log\n   */\n  def printerCounters[T](execution: Execution[T]): Execution[Unit] = {\n    execution.getCounters\n      .flatMap {\n        case (_, counters) =>\n          counters.toMap.toSeq\n            .sortBy(e => (e._1.group, e._1.counter))\n            .foreach {\n              case (statKey, value) =>\n                println(s\"${statKey.group}\\t${statKey.counter}\\t$value\")\n            }\n          Execution.unit\n      }\n  }\n}\n\n/**\n * Trait that wraps things about adhoc jobs.\n */\ntrait GraphFeatureServiceAdhocBaseApp extends TwitterExecutionApp with GraphFeatureServiceBaseJob {\n  override def job: Execution[Unit] = Execution.withId { implicit uniqueId =>\n    Execution.getArgs.flatMap { args: Args =>\n      implicit val dateRange: DateRange = DateRange.parse(args.list(\"date\"))(timeZone, dateParser)\n      printerCounters(runOnDateRange())\n    }\n  }\n}\n\n/**\n * Trait that wraps things about scheduled jobs.\n *\n * A new daily app only needs to declare the starting date.\n */\ntrait GraphFeatureServiceScheduledBaseApp\n    extends TwitterScheduledExecutionApp\n    with GraphFeatureServiceBaseJob {\n\n  def firstTime: RichDate // for example: RichDate(\"2018-02-21\")\n\n  def batchIncrement: Duration = Days(1)\n\n  override def scheduledJob: Execution[Unit] = Execution.withId { implicit uniqueId =>\n    val analyticsArgs = AnalyticsBatchExecutionArgs(\n      batchDesc = BatchDescription(getClass.getName),\n      firstTime = BatchFirstTime(firstTime),\n      batchIncrement = BatchIncrement(batchIncrement)\n    )\n\n    AnalyticsBatchExecution(analyticsArgs) { implicit dateRange =>\n      printerCounters(runOnDateRange())\n    }\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scalding/com/twitter/graph_feature_service/scalding/GraphFeatureServiceApps.scala",
    "content": "package com.twitter.graph_feature_service.scalding\n\nimport com.twitter.scalding.DateRange\nimport com.twitter.scalding.Execution\nimport com.twitter.scalding.RichDate\nimport com.twitter.scalding.UniqueID\nimport java.util.Calendar\nimport java.util.TimeZone\nimport sun.util.calendar.BaseCalendar\n\n/**\n * To launch an adhoc run:\n *\n  scalding remote run --target graph-feature-service/src/main/scalding/com/twitter/graph_feature_service/scalding:graph_feature_service_adhoc_job\n */\nobject GraphFeatureServiceAdhocApp\n    extends GraphFeatureServiceMainJob\n    with GraphFeatureServiceAdhocBaseApp {}\n\n/**\n * To schedule the job, upload the workflows config (only required for the first time and subsequent config changes):\n * scalding workflow upload --jobs graph-feature-service/src/main/scalding/com/twitter/graph_feature_service/scalding:graph_feature_service_daily_job --autoplay --build-cron-schedule \"20 23 1 * *\"\n * You can then build from the UI by clicking \"Build\" and pasting in your remote branch, or leave it empty if you're redeploying from master.\n * The workflows config above should automatically trigger once each month.\n */\nobject GraphFeatureServiceScheduledApp\n    extends GraphFeatureServiceMainJob\n    with GraphFeatureServiceScheduledBaseApp {\n  override def firstTime: RichDate = RichDate(\"2018-05-18\")\n\n  override def runOnDateRange(\n    enableValueGraphs: Option[Boolean],\n    enableKeyGraphs: Option[Boolean]\n  )(\n    implicit dateRange: DateRange,\n    timeZone: TimeZone,\n    uniqueID: UniqueID\n  ): Execution[Unit] = {\n    // Only run the value Graphs on Tuesday, Thursday, Saturday\n    val overrideEnableValueGraphs = {\n      val dayOfWeek = dateRange.start.toCalendar.get(Calendar.DAY_OF_WEEK)\n      dayOfWeek == BaseCalendar.TUESDAY |\n        dayOfWeek == BaseCalendar.THURSDAY |\n        dayOfWeek == BaseCalendar.SATURDAY\n    }\n\n    super.runOnDateRange(\n      Some(true),\n      Some(false) // disable key Graphs since we are not using them in production\n    )\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scalding/com/twitter/graph_feature_service/scalding/GraphFeatureServiceMainJob.scala",
    "content": "package com.twitter.graph_feature_service.scalding\n\nimport com.twitter.bijection.Injection\nimport com.twitter.frigate.common.constdb_util.Injections\nimport com.twitter.frigate.common.constdb_util.ScaldingUtil\nimport com.twitter.graph_feature_service.common.Configs\nimport com.twitter.graph_feature_service.common.Configs._\nimport com.twitter.interaction_graph.scio.agg_all.InteractionGraphHistoryAggregatedEdgeSnapshotScalaDataset\nimport com.twitter.interaction_graph.scio.ml.scores.RealGraphInScoresScalaDataset\nimport com.twitter.interaction_graph.thriftscala.FeatureName\nimport com.twitter.interaction_graph.thriftscala.{EdgeFeature => TEdgeFeature}\nimport com.twitter.pluck.source.user_audits.UserAuditFinalScalaDataset\nimport com.twitter.scalding.DateRange\nimport com.twitter.scalding.Days\nimport com.twitter.scalding.Execution\nimport com.twitter.scalding.Stat\nimport com.twitter.scalding.UniqueID\nimport com.twitter.scalding.typed.TypedPipe\nimport com.twitter.scalding_internal.dalv2.DAL\nimport com.twitter.scalding_internal.dalv2.remote_access.AllowCrossClusterSameDC\nimport com.twitter.scalding_internal.multiformat.format.keyval.KeyVal\nimport com.twitter.util.Time\nimport com.twitter.wtf.candidate.thriftscala.CandidateSeq\nimport java.nio.ByteBuffer\nimport java.util.TimeZone\n\ntrait GraphFeatureServiceMainJob extends GraphFeatureServiceBaseJob {\n\n  // keeping hdfsPath as a separate variable in order to override it in unit tests\n  protected val hdfsPath: String = BaseHdfsPath\n\n  protected def getShardIdForUser(userId: Long): Int = shardForUser(userId)\n\n  protected implicit val keyInj: Injection[Long, ByteBuffer] = Injections.long2Varint\n\n  protected implicit val valueInj: Injection[Long, ByteBuffer] = Injections.long2ByteBuffer\n\n  protected val bufferSize: Int = 1 << 26\n\n  protected val maxNumKeys: Int = 1 << 24\n\n  protected val numReducers: Int = NumGraphShards\n\n  protected val outputStreamBufferSize: Int = 1 << 26\n\n  protected final val shardingByKey = { (k: Long, _: Long) =>\n    getShardIdForUser(k)\n  }\n\n  protected final val shardingByValue = { (_: Long, v: Long) =>\n    getShardIdForUser(v)\n  }\n\n  private def writeGraphToDB(\n    graph: TypedPipe[(Long, Long)],\n    shardingFunction: (Long, Long) => Int,\n    path: String\n  )(\n    implicit dateRange: DateRange\n  ): Execution[TypedPipe[(Int, Unit)]] = {\n    ScaldingUtil\n      .writeConstDB[Long, Long](\n        graph.withDescription(s\"sharding $path\"),\n        shardingFunction,\n        shardId =>\n          getTimedHdfsShardPath(\n            shardId,\n            getHdfsPath(path, Some(hdfsPath)),\n            Time.fromMilliseconds(dateRange.end.timestamp)\n          ),\n        Int.MaxValue,\n        bufferSize,\n        maxNumKeys,\n        numReducers,\n        outputStreamBufferSize\n      )(\n        keyInj,\n        valueInj,\n        Ordering[(Long, Long)]\n      )\n      .forceToDiskExecution\n  }\n\n  def extractFeature(\n    featureList: Seq[TEdgeFeature],\n    featureName: FeatureName\n  ): Option[Float] = {\n    featureList\n      .find(_.name == featureName)\n      .map(_.tss.ewma.toFloat)\n      .filter(_ > 0.0)\n  }\n\n  /**\n   * Function to extract a subgraph (e.g., follow graph) from real graph and take top K by real graph\n   * weight.\n   *\n   * @param input input real graph\n   * @param edgeFilter filter function to only get the edges needed (e.g., only follow edges)\n   * @param counter counter\n   * @return a subgroup that contains topK, e.g., follow graph for each user.\n   */\n  private def getSubGraph(\n    input: TypedPipe[(Long, Long, EdgeFeature)],\n    edgeFilter: EdgeFeature => Boolean,\n    counter: Stat\n  ): TypedPipe[(Long, Long)] = {\n    input\n      .filter(c => edgeFilter(c._3))\n      .map {\n        case (srcId, destId, features) =>\n          (srcId, (destId, features.realGraphScore))\n      }\n      .group\n      // auto reducer estimation only allocates 15 reducers, so setting an explicit number here\n      .withReducers(2000)\n      .sortedReverseTake(TopKRealGraph)(Ordering.by(_._2))\n      .flatMap {\n        case (srcId, topKNeighbors) =>\n          counter.inc()\n          topKNeighbors.map {\n            case (destId, _) =>\n              (srcId, destId)\n          }\n      }\n  }\n\n  def getMauIds()(implicit dateRange: DateRange, uniqueID: UniqueID): TypedPipe[Long] = {\n    val numMAUs = Stat(\"NUM_MAUS\")\n    val uniqueMAUs = Stat(\"UNIQUE_MAUS\")\n\n    DAL\n      .read(UserAuditFinalScalaDataset)\n      .withRemoteReadPolicy(AllowCrossClusterSameDC)\n      .toTypedPipe\n      .collect {\n        case user_audit if user_audit.isValid =>\n          numMAUs.inc()\n          user_audit.userId\n      }\n      .distinct\n      .map { u =>\n        uniqueMAUs.inc()\n        u\n      }\n  }\n\n  def getRealGraphWithMAUOnly(\n    implicit dateRange: DateRange,\n    timeZone: TimeZone,\n    uniqueID: UniqueID\n  ): TypedPipe[(Long, Long, EdgeFeature)] = {\n    val numMAUs = Stat(\"NUM_MAUS\")\n    val uniqueMAUs = Stat(\"UNIQUE_MAUS\")\n\n    val monthlyActiveUsers = DAL\n      .read(UserAuditFinalScalaDataset)\n      .withRemoteReadPolicy(AllowCrossClusterSameDC)\n      .toTypedPipe\n      .collect {\n        case user_audit if user_audit.isValid =>\n          numMAUs.inc()\n          user_audit.userId\n      }\n      .distinct\n      .map { u =>\n        uniqueMAUs.inc()\n        u\n      }\n      .asKeys\n\n    val realGraphAggregates = DAL\n      .readMostRecentSnapshot(\n        InteractionGraphHistoryAggregatedEdgeSnapshotScalaDataset,\n        dateRange.embiggen(Days(5)))\n      .withRemoteReadPolicy(AllowCrossClusterSameDC)\n      .toTypedPipe\n      .map { edge =>\n        val featureList = edge.features\n        val edgeFeature = EdgeFeature(\n          edge.weight.getOrElse(0.0).toFloat,\n          extractFeature(featureList, FeatureName.NumMutualFollows),\n          extractFeature(featureList, FeatureName.NumFavorites),\n          extractFeature(featureList, FeatureName.NumRetweets),\n          extractFeature(featureList, FeatureName.NumMentions)\n        )\n        (edge.sourceId, (edge.destinationId, edgeFeature))\n      }\n      .join(monthlyActiveUsers)\n      .map {\n        case (srcId, ((destId, feature), _)) =>\n          (destId, (srcId, feature))\n      }\n      .join(monthlyActiveUsers)\n      .map {\n        case (destId, ((srcId, feature), _)) =>\n          (srcId, destId, feature)\n      }\n    realGraphAggregates\n  }\n\n  def getTopKFollowGraph(\n    implicit dateRange: DateRange,\n    timeZone: TimeZone,\n    uniqueID: UniqueID\n  ): TypedPipe[(Long, Long)] = {\n    val followGraphMauStat = Stat(\"NumFollowEdges_MAU\")\n    val mau: TypedPipe[Long] = getMauIds()\n    DAL\n      .readMostRecentSnapshot(RealGraphInScoresScalaDataset, dateRange.embiggen(Days(7)))\n      .withRemoteReadPolicy(AllowCrossClusterSameDC)\n      .toTypedPipe\n      .groupBy(_.key)\n      .join(mau.asKeys)\n      .withDescription(\"filtering srcId by mau\")\n      .flatMap {\n        case (_, (KeyVal(srcId, CandidateSeq(candidates)), _)) =>\n          followGraphMauStat.inc()\n          val topK = candidates.sortBy(-_.score).take(TopKRealGraph)\n          topK.map { c => (srcId, c.userId) }\n      }\n  }\n\n  override def runOnDateRange(\n    enableValueGraphs: Option[Boolean],\n    enableKeyGraphs: Option[Boolean]\n  )(\n    implicit dateRange: DateRange,\n    timeZone: TimeZone,\n    uniqueID: UniqueID\n  ): Execution[Unit] = {\n\n    val processValueGraphs = enableValueGraphs.getOrElse(Configs.EnableValueGraphs)\n    val processKeyGraphs = enableKeyGraphs.getOrElse(Configs.EnableKeyGraphs)\n\n    if (!processKeyGraphs && !processValueGraphs) {\n      // Skip the batch job\n      Execution.unit\n    } else {\n      // val favoriteGraphStat = Stat(\"NumFavoriteEdges\")\n      // val retweetGraphStat = Stat(\"NumRetweetEdges\")\n      // val mentionGraphStat = Stat(\"NumMentionEdges\")\n\n      // val realGraphAggregates = getRealGraphWithMAUOnly\n\n      val followGraph = getTopKFollowGraph\n      // val mutualFollowGraph = followGraph.asKeys.join(followGraph.swap.asKeys).keys\n\n      // val favoriteGraph =\n      //   getSubGraph(realGraphAggregates, _.favoriteScore.isDefined, favoriteGraphStat)\n\n      // val retweetGraph =\n      //   getSubGraph(realGraphAggregates, _.retweetScore.isDefined, retweetGraphStat)\n\n      // val mentionGraph =\n      //   getSubGraph(realGraphAggregates, _.mentionScore.isDefined, mentionGraphStat)\n\n      val writeValDataSetExecutions = if (processValueGraphs) {\n        Seq(\n          (followGraph, shardingByValue, FollowOutValPath),\n          (followGraph.swap, shardingByValue, FollowInValPath)\n          // (mutualFollowGraph, shardingByValue, MutualFollowValPath),\n          // (favoriteGraph, shardingByValue, FavoriteOutValPath),\n          // (favoriteGraph.swap, shardingByValue, FavoriteInValPath),\n          // (retweetGraph, shardingByValue, RetweetOutValPath),\n          // (retweetGraph.swap, shardingByValue, RetweetInValPath),\n          // (mentionGraph, shardingByValue, MentionOutValPath),\n          // (mentionGraph.swap, shardingByValue, MentionInValPath)\n        )\n      } else {\n        Seq.empty\n      }\n\n      val writeKeyDataSetExecutions = if (processKeyGraphs) {\n        Seq(\n          (followGraph, shardingByKey, FollowOutKeyPath),\n          (followGraph.swap, shardingByKey, FollowInKeyPath)\n          // (favoriteGraph, shardingByKey, FavoriteOutKeyPath),\n          // (favoriteGraph.swap, shardingByKey, FavoriteInKeyPath),\n          // (retweetGraph, shardingByKey, RetweetOutKeyPath),\n          // (retweetGraph.swap, shardingByKey, RetweetInKeyPath),\n          // (mentionGraph, shardingByKey, MentionOutKeyPath),\n          // (mentionGraph.swap, shardingByKey, MentionInKeyPath),\n          // (mutualFollowGraph, shardingByKey, MutualFollowKeyPath)\n        )\n      } else {\n        Seq.empty\n      }\n\n      Execution\n        .sequence((writeValDataSetExecutions ++ writeKeyDataSetExecutions).map {\n          case (graph, shardingMethod, path) =>\n            writeGraphToDB(graph, shardingMethod, path)\n        }).unit\n    }\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/scalding/com/twitter/graph_feature_service/scalding/adhoc/BUILD.bazel",
    "content": "scala_library(\n    platform = \"java8\",\n    tags = [\"bazel-only\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/bijection:core\",\n        \"3rdparty/jvm/com/twitter/bijection:scrooge\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/constdb_util\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/scalding_internal/job\",\n        \"src/scala/com/twitter/scalding_internal/job/analytics_batch\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n    ],\n)\n\nhadoop_binary(\n    name = \"gfs_random_request-adhoc\",\n    main = \"com.twitter.graph_feature_service.scalding.adhoc.RandomRequestGenerationApp\",\n    platform = \"java8\",\n    runtime_platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-compatible:migrated\",\n        \"bazel-only\",\n    ],\n    dependencies = [\":adhoc\"],\n)\n"
  },
  {
    "path": "graph-feature-service/src/main/scalding/com/twitter/graph_feature_service/scalding/adhoc/RandomRequestGenerationApp.scala",
    "content": "package com.twitter.graph_feature_service.scalding.adhoc\n\nimport com.twitter.bijection.Injection\nimport com.twitter.frigate.common.constdb_util.Injections\nimport com.twitter.ml.api.Feature.Discrete\nimport com.twitter.ml.api.{DailySuffixFeatureSource, DataSetPipe, RichDataRecord}\nimport com.twitter.scalding._\nimport com.twitter.scalding_internal.job.TwitterExecutionApp\nimport java.nio.ByteBuffer\nimport java.util.TimeZone\n\nobject RandomRequestGenerationJob {\n  implicit val timeZone: TimeZone = DateOps.UTC\n  implicit val dateParser: DateParser = DateParser.default\n\n  val timelineRecapDataSetPath: String =\n    \"/atla/proc2/user/timelines/processed/suggests/recap/data_records\"\n\n  val USER_ID = new Discrete(\"meta.user_id\")\n  val AUTHOR_ID = new Discrete(\"meta.author_id\")\n\n  val timelineRecapOutPutPath: String = \"/user/cassowary/gfs/adhoc/timeline_data\"\n\n  implicit val inj: Injection[Long, ByteBuffer] = Injections.long2Varint\n\n  def run(\n    dataSetPath: String,\n    outPutPath: String,\n    numOfPairsToTake: Int\n  )(\n    implicit dateRange: DateRange,\n    uniqueID: UniqueID\n  ): Execution[Unit] = {\n\n    val NumUserAuthorPairs = Stat(\"NumUserAuthorPairs\")\n\n    val dataSet: DataSetPipe = DailySuffixFeatureSource(dataSetPath).read\n\n    val userAuthorPairs: TypedPipe[(Long, Long)] = dataSet.records.map { record =>\n      val richRecord = new RichDataRecord(record, dataSet.featureContext)\n\n      val userId = richRecord.getFeatureValue(USER_ID)\n      val authorId = richRecord.getFeatureValue(AUTHOR_ID)\n      NumUserAuthorPairs.inc()\n      (userId, authorId)\n    }\n\n    userAuthorPairs\n      .limit(numOfPairsToTake)\n      .writeExecution(\n        TypedTsv[(Long, Long)](outPutPath)\n      )\n  }\n}\n\n/**\n * ./bazel bundle graph-feature-service/src/main/scalding/com/twitter/graph_feature_service/scalding/adhoc:all\n *\n * oscar hdfs --screen --user cassowary --tee gfs_log --bundle gfs_random_request-adhoc \\\n      --tool com.twitter.graph_feature_service.scalding.adhoc.RandomRequestGenerationApp \\\n      -- --date 2018-08-11  \\\n      --input /atla/proc2/user/timelines/processed/suggests/recap/data_records \\\n      --output /user/cassowary/gfs/adhoc/timeline_data\n */\nobject RandomRequestGenerationApp extends TwitterExecutionApp {\n  import RandomRequestGenerationJob._\n  override def job: Execution[Unit] = Execution.withId { implicit uniqueId =>\n    Execution.getArgs.flatMap { args: Args =>\n      implicit val dateRange: DateRange = DateRange.parse(args.list(\"date\"))(timeZone, dateParser)\n      run(\n        args.optional(\"input\").getOrElse(timelineRecapDataSetPath),\n        args.optional(\"output\").getOrElse(timelineRecapOutPutPath),\n        args.int(\"num_pairs\", 3000)\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "graph-feature-service/src/main/thrift/com/twitter/graph_feature_service/BUILD",
    "content": "create_thrift_libraries(\n    base_name = \"graph_feature_service_thrift\",\n    sources = [\"*.thrift\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    generate_languages = [\n        \"java\",\n        # ruby is added due to ruby dependees in timelines\n        \"ruby\",\n        \"scala\",\n        \"strato\",\n    ],\n    provides_java_name = \"graph_feature_service_thrift_java\",\n    provides_scala_name = \"graph_feature_service_thrift_scala\",\n)\n"
  },
  {
    "path": "graph-feature-service/src/main/thrift/com/twitter/graph_feature_service/graph_feature_service.thrift",
    "content": "namespace java com.twitter.graph_feature_service.thriftjava\n#@namespace scala com.twitter.graph_feature_service.thriftscala\n#@namespace strato com.twitter.graph_feature_service.thriftscala\n\n// edge type to differentiate different types of graphs (we can also add a lot of other types of edges)\nenum EdgeType {\n  FOLLOWING,\n  FOLLOWED_BY,\n  FAVORITE,\n  FAVORITED_BY,\n  RETWEET,\n  RETWEETED_BY,\n  REPLY,\n  REPLYED_BY,\n  MENTION,\n  MENTIONED_BY,\n  MUTUAL_FOLLOW,\n  SIMILAR_TO, // more edge types (like block, report, etc.) can be supported later.\n  RESERVED_12,\n  RESERVED_13,\n  RESERVED_14,\n  RESERVED_15,\n  RESERVED_16,\n  RESERVED_17,\n  RESERVED_18,\n  RESERVED_19,\n  RESERVED_20\n}\n\nenum PresetFeatureTypes {\n  EMPTY,\n  HTL_TWO_HOP,\n  WTF_TWO_HOP,\n  SQ_TWO_HOP,\n  RUX_TWO_HOP,\n  MR_TWO_HOP,\n  USER_TYPEAHEAD_TWO_HOP\n}\n\nstruct UserWithCount {\n  1: required i64 userId(personalDataType = 'UserId')\n  2: required i32 count\n}(hasPersonalData = 'true')\n\nstruct UserWithScore {\n  1: required i64 userId(personalDataType = 'UserId')\n  2: required double score\n}(hasPersonalData = 'true')\n\n// Feature Type\n// For example, to compute how many of source user's following's have favorited candidate user,\n// we need to compute the intersection between source user's FOLLOWING edges, and candidate user's\n// FAVORITED_BY edge. In this case, we should user FeatureType(FOLLOWING, FAVORITED_BY)\nstruct FeatureType {\n  1: required EdgeType leftEdgeType // edge type from source user\n  2: required EdgeType rightEdgeType // edge type from candidate user\n}(persisted=\"true\")\n\nstruct IntersectionValue {\n  1: required FeatureType featureType\n  2: optional i32 count\n  3: optional list<i64> intersectionIds(personalDataType = 'UserId')\n  4: optional i32 leftNodeDegree\n  5: optional i32 rightNodeDegree\n}(persisted=\"true\", hasPersonalData = 'true')\n\nstruct GfsIntersectionResult {\n  1: required i64 candidateUserId(personalDataType = 'UserId')\n  2: required list<IntersectionValue> intersectionValues\n}(hasPersonalData = 'true')\n\nstruct GfsIntersectionRequest {\n  1: required i64 userId(personalDataType = 'UserId')\n  2: required list<i64> candidateUserIds(personalDataType = 'UserId')\n  3: required list<FeatureType> featureTypes\n  4: optional i32 intersectionIdLimit\n}\n\nstruct GfsPresetIntersectionRequest {\n  1: required i64 userId(personalDataType = 'UserId')\n  2: required list<i64> candidateUserIds(personalDataType = 'UserId')\n  3: required PresetFeatureTypes presetFeatureTypes\n  4: optional i32 intersectionIdLimit\n}(hasPersonalData = 'true')\n\nstruct GfsIntersectionResponse {\n  1: required list<GfsIntersectionResult> results\n}\n\nservice Server {\n  GfsIntersectionResponse getIntersection(1: GfsIntersectionRequest request)\n  GfsIntersectionResponse getPresetIntersection(1: GfsPresetIntersectionRequest request)\n}\n\n###################################################################################################\n##  For internal usage only\n###################################################################################################\nstruct WorkerIntersectionRequest {\n  1: required i64 userId(personalDataType = 'UserId')\n  2: required list<i64> candidateUserIds(personalDataType = 'UserId')\n  3: required list<FeatureType> featureTypes\n  4: required PresetFeatureTypes presetFeatureTypes\n  5: required i32 intersectionIdLimit\n}(hasPersonalData = 'true')\n\nstruct WorkerIntersectionResponse {\n  1: required list<list<WorkerIntersectionValue>> results\n}\n\nstruct WorkerIntersectionValue {\n  1: i32 count\n  2: i32 leftNodeDegree\n  3: i32 rightNodeDegree\n  4: list<i64> intersectionIds(personalDataType = 'UserId')\n}(hasPersonalData = 'true')\n\nstruct CachedIntersectionResult {\n  1: required list<WorkerIntersectionValue> values\n}\n\nservice Worker {\n  WorkerIntersectionResponse getIntersection(1: WorkerIntersectionRequest request)\n}\n"
  },
  {
    "path": "home-mixer/BUILD.bazel",
    "content": "jvm_binary(\n    name = \"bin\",\n    basename = \"home-mixer\",\n    main = \"com.twitter.home_mixer.HomeMixerServerMain\",\n    runtime_platform = \"java11\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/ch/qos/logback:logback-classic\",\n        \"finagle/finagle-zipkin-scribe/src/main/scala\",\n        \"finatra/inject/inject-logback/src/main/scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer\",\n        \"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback\",\n        \"twitter-server-internal/src/main/scala\",\n        \"twitter-server/logback-classic/src/main/scala\",\n    ],\n)\n\n#  Aurora Workflows build phase convention requires a jvm_app named with home-mixer-app\njvm_app(\n    name = \"home-mixer-app\",\n    archive = \"zip\",\n    binary = \":bin\",\n    bundles = [\n        bundle(\n            fileset = [\"config/**/*\"],\n            owning_target = \"home-mixer/config:files\",\n        ),\n    ],\n    tags = [\"bazel-compatible\"],\n)\n"
  },
  {
    "path": "home-mixer/README.md",
    "content": "Home Mixer\n==========\n\nHome Mixer is the main service used to construct and serve Twitter's Home Timelines. It currently\npowers:\n- For you - best Tweets from people you follow + recommended out-of-network content\n- Following - reverse chronological Tweets from people you follow\n- Lists - reverse chronological Tweets from List members\n\nHome Mixer is built on Product Mixer, our custom Scala framework that facilitates building\nfeeds of content.\n\n## Overview\n\nThe For You recommendation algorithm in Home Mixer involves the following stages:\n\n- Candidate Generation - fetch Tweets from various Candidate Sources. For example:\n    - Earlybird Search Index\n    - User Tweet Entity Graph\n    - Cr Mixer\n    - Follow Recommendations Service\n- Feature Hydration\n    - Fetch the ~6000 features needed for ranking\n- Scoring and Ranking using ML model\n- Filters and Heuristics. For example:\n    - Author Diversity\n    - Content Balance (In network vs Out of Network)\n    - Feedback fatigue\n    - Deduplication / previously seen Tweets removal\n    - Visibility Filtering (blocked, muted authors/tweets, NSFW settings)\n- Mixing - integrate Tweets with non-Tweet content\n    - Ads\n    - Who-to-follow modules\n    - Prompts\n- Product Features and Serving\n    - Conversation Modules for replies\n    - Social Context\n    - Timeline Navigation\n    - Edited Tweets\n    - Feedback options\n    - Pagination and cursoring\n    - Observability and logging\n    - Client instructions and content marshalling\n\n## Pipeline Structure\n\n### General\n\nProduct Mixer services like Home Mixer are structured around Pipelines that split the execution\ninto transparent and structured steps.\n\nRequests first go to Product Pipelines, which are used to select which Mixer Pipeline or\nRecommendation Pipeline to run for a given request. Each Mixer or Recommendation\nPipeline may run multiple Candidate Pipelines to fetch candidates to include in the response.\n\nMixer Pipelines combine the results of multiple heterogeneous Candidate Pipelines together\n(e.g. ads, tweets, users) while Recommendation Pipelines are used to score (via Scoring Pipelines)\nand rank the results of homogenous Candidate Pipelines so that the top ranked ones can be returned.\nThese pipelines also marshall candidates into a domain object and then into a transport object\nto return to the caller.\n\nCandidate Pipelines fetch candidates from underlying Candidate Sources and perform some basic\noperations on the Candidates, such as filtering out unwanted candidates, applying decorations,\nand hydrating features.\n\nThe sections below describe the high level pipeline structure (non-exhaustive) for the main Home\nTimeline tabs powered by Home Mixer.\n\n### For You\n\n- ForYouProductPipelineConfig\n    - ForYouScoredTweetsMixerPipelineConfig (main orchestration layer - mixes Tweets with ads and users)\n        - ForYouScoredTweetsCandidatePipelineConfig (fetch Tweets)\n            - ScoredTweetsRecommendationPipelineConfig (main Tweet recommendation layer)\n                - Fetch Tweet Candidates\n                    - ScoredTweetsInNetworkCandidatePipelineConfig\n                    - ScoredTweetsTweetMixerCandidatePipelineConfig\n                    - ScoredTweetsUtegCandidatePipelineConfig\n                    - ScoredTweetsFrsCandidatePipelineConfig\n                - Feature Hydration and Scoring\n                    - ScoredTweetsScoringPipelineConfig\n        - ForYouConversationServiceCandidatePipelineConfig (backup reverse chron pipeline in case Scored Tweets fails)\n        - ForYouAdsCandidatePipelineConfig (fetch ads)\n        - ForYouWhoToFollowCandidatePipelineConfig (fetch users to recommend)\n\n### Following\n\n- FollowingProductPipelineConfig\n    - FollowingMixerPipelineConfig\n        - FollowingEarlybirdCandidatePipelineConfig (fetch tweets from Search Index)\n        - ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)\n        - FollowingAdsCandidatePipelineConfig (fetch ads)\n        - FollowingWhoToFollowCandidatePipelineConfig (fetch users to recommend)\n\n### Lists\n\n- ListTweetsProductPipelineConfig\n    - ListTweetsMixerPipelineConfig\n        - ListTweetsTimelineServiceCandidatePipelineConfig (fetch tweets from timeline service)\n        - ConversationServiceCandidatePipelineConfig (fetch ancestors for conversation modules)\n        - ListTweetsAdsCandidatePipelineConfig (fetch ads)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-netty4/src/main/scala\",\n        \"finatra-internal/mtls-http/src/main/scala\",\n        \"home-mixer/server/src/main/resources\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/controller\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/federated\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/module\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"joinkey/src/main/scala/com/twitter/joinkey/context\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/stringcenter\",\n        \"strato/src/main/scala/com/twitter/strato/fed/server\",\n        \"timelines/src/main/scala/com/twitter/timelines/config\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/HomeMixerHttpServerWarmupHandler.scala",
    "content": "package com.twitter.home_mixer\n\nimport com.twitter.finatra.http.routing.HttpWarmup\nimport com.twitter.finatra.httpclient.RequestBuilder._\nimport com.twitter.util.logging.Logging\nimport com.twitter.inject.utils.Handler\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeMixerHttpServerWarmupHandler @Inject() (warmup: HttpWarmup) extends Handler with Logging {\n\n  override def handle(): Unit = {\n    Try(warmup.send(get(\"/admin/product-mixer/product-pipelines\"), admin = true)())\n      .onFailure(e => error(e.getMessage, e))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/HomeMixerServer.scala",
    "content": "package com.twitter.home_mixer\n\nimport com.google.inject.Module\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.StorageUnitOps.richStorageUnitFromInt\nimport com.twitter.finagle.Filter\nimport com.twitter.finagle.Http\nimport com.twitter.finagle.Thrift\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.server.MtlsStackServer.MtlsHttpServerSyntax\nimport com.twitter.finagle.netty4.param.TrackWorkerPool\nimport com.twitter.finatra.annotations.DarkTrafficFilterType\nimport com.twitter.finatra.http.HttpServer\nimport com.twitter.finatra.http.routing.HttpRouter\nimport com.twitter.finatra.mtls.http.{Mtls => HttpMtls}\nimport com.twitter.finatra.mtls.thriftmux.Mtls\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsThriftWebFormsModule\nimport com.twitter.finatra.thrift.ThriftServer\nimport com.twitter.finatra.thrift.filters._\nimport com.twitter.finatra.thrift.routing.ThriftRouter\nimport com.twitter.home_mixer.controller.HomeHttpController\nimport com.twitter.home_mixer.controller.HomeThriftController\nimport com.twitter.home_mixer.federated.HomeMixerColumn\nimport com.twitter.home_mixer.module._\nimport com.twitter.home_mixer.param.GlobalParamConfigModule\nimport com.twitter.home_mixer.product.HomeMixerProductModule\nimport com.twitter.home_mixer.{thriftscala => st}\nimport com.twitter.joinkey.context.CreateRequestJoinKeyContextFilter\nimport com.twitter.product_mixer.component_library.module.AccountRecommendationsMixerModule\nimport com.twitter.product_mixer.component_library.module.CommunitiesMixerClientModule\nimport com.twitter.product_mixer.component_library.module.DarkTrafficFilterModule\nimport com.twitter.product_mixer.component_library.module.EarlybirdModule\nimport com.twitter.product_mixer.component_library.module.FeedbackHistoryClientModule\nimport com.twitter.product_mixer.component_library.module.GizmoduckClientModule\nimport com.twitter.product_mixer.component_library.module.MemcachedImpressionBloomFilterStoreModule\nimport com.twitter.product_mixer.component_library.module.OnboardingTaskServiceModule\nimport com.twitter.product_mixer.component_library.module.SocialGraphServiceModule\nimport com.twitter.product_mixer.component_library.module.StaleTweetsCacheModule\nimport com.twitter.product_mixer.component_library.module.TestUserMapperConfigModule\nimport com.twitter.product_mixer.component_library.module.TimelineRankerClientModule\nimport com.twitter.product_mixer.component_library.module.TimelineServiceClientModule\nimport com.twitter.product_mixer.component_library.module.TweetImpressionStoreModule\nimport com.twitter.product_mixer.component_library.module.TweetMixerClientModule\nimport com.twitter.product_mixer.component_library.module.UserSessionStoreModule\nimport com.twitter.product_mixer.component_library.module.UtegClientModule\nimport com.twitter.product_mixer.component_library.module.UtvgClientModule\nimport com.twitter.product_mixer.core.controllers.ProductMixerController\nimport com.twitter.product_mixer.core.module.LoggingThrowableExceptionMapper\nimport com.twitter.product_mixer.core.module.ProductMixerModule\nimport com.twitter.product_mixer.core.module.stringcenter.ProductScopeStringCenterModule\nimport com.twitter.scrooge.TUnsafeBinaryProtocolFactory\nimport com.twitter.strato.fed.StratoFed\nimport com.twitter.strato.fed.server.StratoFedServer\n\nobject HomeMixerServerMain extends HomeMixerServer\n\nclass HomeMixerServer\n    extends StratoFedServer\n    with ThriftServer\n    with Mtls\n    with HttpServer\n    with HttpMtls {\n  override val name = \"home-mixer-server\"\n\n  override val modules: Seq[Module] = Seq(\n    AccountRecommendationsMixerModule,\n    AdvertiserBrandSafetySettingsStoreModule,\n    ClientSentImpressionsPublisherModule,\n    ClusterDetailsModule,\n    CommunitiesMixerClientModule,\n    ConversationServiceModule,\n    EarlybirdModule,\n    EarlybirdRealtimeCGModule,\n    EventsRecosClientModule,\n    FeedbackHistoryClientModule,\n    GizmoduckClientModule,\n    GizmoduckTimelinesCacheClientModule,\n    GlobalParamConfigModule,\n    HomeAdsCandidateSourceModule,\n    HomeMixerFeaturesModule,\n    HomeMixerFlagsModule,\n    HomeMixerProductModule,\n    HomeMixerResourcesModule,\n    InMemoryCacheModule,\n    InjectionHistoryClientModule,\n    LimiterModule,\n    ManhattanClientsModule,\n    ManhattanFeatureRepositoryModule,\n    MediaClusterId88Module,\n    MediaClusterId95Module,\n    MemcachedFeatureRepositoryModule,\n    MemcachedImpressionBloomFilterStoreModule,\n    MemcachedScoredCandidateFeaturesStoreModule,\n    NaviModelClientModule,\n    OnboardingTaskServiceModule,\n    OptimizedStratoClientModule,\n    PeopleDiscoveryServiceModule,\n    PhoenixClientModule,\n    ProductMixerModule,\n    RealGraphInNetworkScoresModule,\n    RealtimeAggregateFeatureRepositoryModule,\n    ScoredTweetsMemcacheModule,\n    ScoredVideoTweetsMemcacheModule,\n    ScribeEventPublisherModule,\n    SimClustersRecentEngagementsClientModule,\n    SocialGraphServiceModule,\n    StaleTweetsCacheModule,\n    TestUserMapperConfigModule,\n    ThriftFeatureRepositoryModule,\n    TimelineRankerClientModule,\n    TimelineServiceClientModule,\n    TimelinesPersistenceStoreClientModule,\n    TopicSocialProofClientModule,\n    TvWatchHistoryCacheClientModule,\n    TweetImpressionStoreModule,\n    TweetMixerClientModule,\n    TweetWatchTimeMetadataModule,\n    TweetypieClientModule,\n    TweetypieStaticEntitiesCacheClientModule,\n    TwhinEmbeddingsModule,\n    UserSessionStoreModule,\n    UtegClientModule,\n    UttTopicModule,\n    UtvgClientModule,\n    VideoEmbeddingModule,\n    new DarkTrafficFilterModule[st.HomeMixer.ReqRepServicePerEndpoint](),\n    new MtlsThriftWebFormsModule[st.HomeMixer.MethodPerEndpoint](this),\n    new ProductScopeStringCenterModule()\n  )\n\n  val requestJoinKeyContextFilter = new CreateRequestJoinKeyContextFilter\n\n  override def configureThrift(router: ThriftRouter): Unit = {\n    router\n      .filter[LoggingMDCFilter]\n      .filter[TraceIdMDCFilter]\n      .filter[ThriftMDCFilter]\n      .filter[StatsFilter]\n      .filter[AccessLoggingFilter]\n      .filter[ExceptionMappingFilter]\n      .filter[Filter.TypeAgnostic, DarkTrafficFilterType]\n      .filter(requestJoinKeyContextFilter)\n      .exceptionMapper[LoggingThrowableExceptionMapper]\n      .exceptionMapper[PipelineFailureExceptionMapper]\n      .add[HomeThriftController]\n  }\n\n  override def configureStratoThriftServer(server: ThriftMux.Server): ThriftMux.Server = {\n    super\n      .configureStratoThriftServer(server)\n      .configured(\n        TrackWorkerPool(\n          enableTracking = true,\n          trackingTaskPeriod = 20.milliseconds,\n          threadDumpThreshold = 0.milliseconds\n        ))\n      .withMaxReusableBufferSize(1.megabyte.bytes.toInt)\n      .withProtocolFactory(new TUnsafeBinaryProtocolFactory(Thrift.param.protocolFactory))\n  }\n\n  override def configureHttp(router: HttpRouter): Unit =\n    router\n      .add(\n        ProductMixerController[st.HomeMixer.MethodPerEndpoint](\n          this.injector,\n          st.HomeMixer.ExecutePipeline\n        )\n      ).add[HomeHttpController]\n\n  override def configureHttpsServer(server: Http.Server): Http.Server = {\n    val serviceIdentifier: ServiceIdentifier = injector.instance[ServiceIdentifier]\n    server.withMutualTls(serviceIdentifier.copy(role = \"home-mixer\"))\n  }\n\n  override val dest: String = \"/s/home-mixer/home-mixer:strato\"\n\n  override val columns: Seq[Class[_ <: StratoFed.Column]] =\n    Seq(classOf[HomeMixerColumn])\n\n  override protected def warmup(): Unit = {\n    handle[HomeMixerThriftServerWarmupHandler]()\n    handle[HomeMixerHttpServerWarmupHandler]()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/HomeMixerThriftServerWarmupHandler.scala",
    "content": "package com.twitter.home_mixer\n\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finatra.thrift.routing.ThriftWarmup\nimport com.twitter.home_mixer.{thriftscala => st}\nimport com.twitter.inject.utils.Handler\nimport com.twitter.product_mixer.core.{thriftscala => pt}\nimport com.twitter.scrooge.Request\nimport com.twitter.scrooge.Response\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeMixerThriftServerWarmupHandler @Inject() (warmup: ThriftWarmup)\n    extends Handler\n    with Logging {\n\n  private val CurrentClientId = ClientId(\"thrift-warmup-client\")\n  private val SleepThreshold = 10000 // millis\n\n  private val TestIds = Seq.empty\n\n  private val BaseClientContext = pt.ClientContext(\n    userId = TestIds.headOption,\n    guestId = None,\n    appId = Some(1L),\n    ipAddress = Some(\"0.0.0.0\"),\n    userAgent = Some(\"FAKE_USER_AGENT_FOR_WARMUPS\"),\n    countryCode = Some(\"US\"),\n    languageCode = Some(\"en\"),\n    isTwoffice = None,\n    userRoles = None,\n    deviceId = Some(\"FAKE_DEVICE_ID_FOR_WARMUPS\")\n  )\n\n  def handle(): Unit = {\n    try {\n      CurrentClientId.asCurrent {\n        TestIds.foreach { id =>\n          val warmupReqs = warmupQuery(id)\n          warmupReqs.foreach { warmupReq =>\n            info(s\"Sending warm-up request to service with query: $warmupReq\")\n            warmup.sendRequest(\n              method = st.HomeMixer.GetUrtResponse,\n              req = Request(st.HomeMixer.GetUrtResponse.Args(warmupReq)),\n            )(assertWarmupResponse)\n          }\n        }\n      }\n    } catch {\n      case e: Throwable =>\n        error(e.getMessage, e)\n    }\n    info(\"Warm-up done.\")\n  }\n\n  private def warmupQuery(userId: Long): Seq[st.HomeMixerRequest] = {\n    val clientContext = BaseClientContext.copy(userId = Some(userId))\n\n    val scoredTweets = st.HomeMixerRequest(\n      clientContext = clientContext,\n      product = st.Product.ScoredTweets,\n      productContext = Some(st.ProductContext.ScoredTweets(st.ScoredTweets())),\n    )\n\n    val scoredVideoTweets = st.HomeMixerRequest(\n      clientContext = clientContext,\n      product = st.Product.ScoredVideoTweets,\n      productContext = Some(st.ProductContext.ScoredVideoTweets(st.ScoredVideoTweets())),\n    )\n\n    val forYou = st.HomeMixerRequest(\n      clientContext = clientContext,\n      product = st.Product.ForYou,\n      productContext = Some(st.ProductContext.ForYou(st.ForYou())),\n    )\n\n    val following = st.HomeMixerRequest(\n      clientContext = clientContext,\n      product = st.Product.Following,\n      productContext = Some(st.ProductContext.Following(st.Following())),\n    )\n\n    Seq(scoredTweets, scoredVideoTweets, forYou, following)\n  }\n\n  private def assertWarmupResponse(\n    result: Try[Response[st.HomeMixer.GetUrtResponse.SuccessType]]\n  ): Unit = {\n    result match {\n      case Return(_) => // ok\n      case Throw(exception) =>\n        warn(\"Error performing warm-up request.\")\n        error(exception.getMessage, exception)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/stale_tweets\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/param_gated\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/transformer/stale_tweets\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline/ConversationServiceCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.candidate_pipeline\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.InNetworkFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetTypeMetricsFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.InvalidConversationModuleFilter\nimport com.twitter.home_mixer.functional_component.filter.InvalidSubscriptionTweetFilter\nimport com.twitter.home_mixer.functional_component.filter.LocationFilter\nimport com.twitter.home_mixer.functional_component.filter.RetweetDeduplicationFilter\nimport com.twitter.home_mixer.functional_component.filter.TweetHydrationFilter\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedTweetDroppedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSource\nimport com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSourceRequest\nimport com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.TweetWithConversationMetadata\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.communities.CommunityNamesFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedBulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.filter.FeatureFilter\nimport com.twitter.product_mixer.component_library.filter.PredicateFeatureFilter\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.DependentCandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\n\n/**\n * Candidate Pipeline Config that fetches tweets from the Conversation Service Candidate Source\n */\nclass ConversationServiceCandidatePipelineConfig[Query <: PipelineQuery](\n  conversationServiceCandidateSource: ConversationServiceCandidateSource,\n  tweetypieFeatureHydrator: TweetypieFeatureHydrator,\n  namesFeatureHydrator: NamesFeatureHydrator,\n  communityNamesFeatureHydrator: CommunityNamesFeatureHydrator,\n  invalidSubscriptionTweetFilter: InvalidSubscriptionTweetFilter,\n  override val gates: Seq[BaseGate[Query]],\n  override val decorator: Option[CandidateDecorator[Query, TweetCandidate]],\n  servedType: hmt.ServedType,\n  paramGatedPostContextFeatureHydrator: ParamGatedBulkCandidateFeatureHydrator[Query, TweetCandidate])\n    extends DependentCandidatePipelineConfig[\n      Query,\n      ConversationServiceCandidateSourceRequest,\n      TweetWithConversationMetadata,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ConversationService\")\n\n  private val InNetworkFilterId = \"InNetwork\"\n  private val QuotedTweetDroppedFilterId = \"QuotedTweetDropped\"\n\n  override val candidateSource: BaseCandidateSource[\n    ConversationServiceCandidateSourceRequest,\n    TweetWithConversationMetadata\n  ] = conversationServiceCandidateSource\n\n  override val queryTransformer: DependentCandidatePipelineQueryTransformer[\n    Query,\n    ConversationServiceCandidateSourceRequest\n  ] = { (_, candidates) =>\n    val tweetsWithConversationMetadata = candidates.collect {\n      case candidate if candidate.isCandidateType[TweetCandidate]() =>\n        TweetWithConversationMetadata(\n          tweetId = candidate.candidateIdLong,\n          userId = candidate.features.getOrElse(AuthorIdFeature, None),\n          sourceTweetId = candidate.features.getOrElse(SourceTweetIdFeature, None),\n          sourceUserId = candidate.features.getOrElse(SourceUserIdFeature, None),\n          inReplyToTweetId = candidate.features.getOrElse(InReplyToTweetIdFeature, None),\n          conversationId = None,\n          ancestors = Seq.empty\n        )\n    }\n    ConversationServiceCandidateSourceRequest(tweetsWithConversationMetadata)\n  }\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[TweetWithConversationMetadata]\n  ] = Seq(ConversationServiceResponseFeatureTransformer(servedType))\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    TweetWithConversationMetadata,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.tweetId) }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[Query, TweetCandidate, _]\n  ] = Seq(\n    tweetypieFeatureHydrator,\n    InNetworkFeatureHydrator,\n  )\n\n  override def filters: Seq[Filter[Query, TweetCandidate]] = Seq(\n    RetweetDeduplicationFilter,\n    FeatureFilter.fromFeature(FilterIdentifier(InNetworkFilterId), InNetworkFeature),\n    TweetHydrationFilter,\n    PredicateFeatureFilter.fromPredicate(\n      FilterIdentifier(QuotedTweetDroppedFilterId),\n      shouldKeepCandidate = { features => !features.getOrElse(QuotedTweetDroppedFeature, false) }\n    ),\n    LocationFilter,\n    invalidSubscriptionTweetFilter,\n    InvalidConversationModuleFilter\n  )\n\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[Query, TweetCandidate, _]\n  ] = Seq(\n    communityNamesFeatureHydrator,\n    namesFeatureHydrator,\n    TweetTypeMetricsFeatureHydrator,\n    paramGatedPostContextFeatureHydrator\n  )\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),\n    HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline/ConversationServiceCandidatePipelineConfigBuilder.scala",
    "content": "package com.twitter.home_mixer.candidate_pipeline\n\nimport com.twitter.home_mixer.functional_component.decorator.HomeConversationServiceCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeTweetContextBuilder\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.PostContextFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.InvalidSubscriptionTweetFilter\nimport com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSource\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.communities.CommunityNamesFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedBulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.gate.NonEmptyCandidatesGate\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ConversationServiceCandidatePipelineConfigBuilder[Query <: PipelineQuery] @Inject() (\n  conversationServiceCandidateSource: ConversationServiceCandidateSource,\n  tweetypieFeatureHydrator: TweetypieFeatureHydrator,\n  communityNamesFeatureHydrator: CommunityNamesFeatureHydrator,\n  postContextFeatureHydrator: PostContextFeatureHydrator,\n  invalidSubscriptionTweetFilter: InvalidSubscriptionTweetFilter,\n  namesFeatureHydrator: NamesFeatureHydrator,\n  homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder,\n  homeTweetContextBuilder: HomeTweetContextBuilder) {\n\n  def build(\n    nonEmptyCandidateScope: CandidateScope,\n    servedType: hmt.ServedType,\n    enablePostContextFeatureHydratorParam: Param[Boolean]\n  ): ConversationServiceCandidatePipelineConfig[Query] = {\n    val paramGatedPostContextFeatureHydrator = ParamGatedBulkCandidateFeatureHydrator(\n      enablePostContextFeatureHydratorParam,\n      postContextFeatureHydrator\n    )\n    \n    new ConversationServiceCandidatePipelineConfig(\n      conversationServiceCandidateSource,\n      tweetypieFeatureHydrator,\n      namesFeatureHydrator,\n      communityNamesFeatureHydrator,\n      invalidSubscriptionTweetFilter,\n      Seq(NonEmptyCandidatesGate(nonEmptyCandidateScope)),\n      HomeConversationServiceCandidateDecorator(\n        homeFeedbackActionInfoBuilder,\n        homeTweetContextBuilder,\n        servedType\n      ),\n      servedType,\n      paramGatedPostContextFeatureHydrator\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline/ConversationServiceResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.candidate_pipeline\n\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.TweetWithConversationMetadata\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\ncase class ConversationServiceResponseFeatureTransformer(servedType: hmt.ServedType)\n    extends CandidateFeatureTransformer[TweetWithConversationMetadata] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ConversationServiceResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    AuthorIdFeature,\n    InReplyToTweetIdFeature,\n    IsRetweetFeature,\n    SourceTweetIdFeature,\n    SourceUserIdFeature,\n    ConversationModuleFocalTweetIdFeature,\n    AncestorsFeature,\n    ServedTypeFeature\n  )\n\n  override def transform(candidate: TweetWithConversationMetadata): FeatureMap = FeatureMapBuilder()\n    .add(AuthorIdFeature, candidate.userId)\n    .add(InReplyToTweetIdFeature, candidate.inReplyToTweetId)\n    .add(IsRetweetFeature, candidate.sourceTweetId.isDefined)\n    .add(SourceTweetIdFeature, candidate.sourceTweetId)\n    .add(SourceUserIdFeature, candidate.sourceUserId)\n    .add(ConversationModuleFocalTweetIdFeature, candidate.conversationId)\n    .add(AncestorsFeature, candidate.ancestors)\n    .add(ServedTypeFeature, servedType)\n    .build()\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline/EditedTweetsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.candidate_pipeline\n\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator\nimport com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.component_library.candidate_source.stale_tweets.StaleTweetsCacheCandidateSource\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref.ContextualTweetRefBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.EmptyClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.transformer.stale_tweets.EditedTweetsCandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.TimelineFocalTweetSafetyLevel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.TweetHydrationContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Candidate Pipeline Config that fetches edited tweets from the Stale Tweets Cache\n */\n@Singleton\ncase class EditedTweetsCandidatePipelineConfig @Inject() (\n  staleTweetsCacheCandidateSource: StaleTweetsCacheCandidateSource,\n  namesFeatureHydrator: NamesFeatureHydrator,\n  homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder)\n    extends DependentCandidatePipelineConfig[\n      PipelineQuery,\n      Seq[Long],\n      Long,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\"EditedTweets\")\n\n  override val candidateSource: BaseCandidateSource[Seq[Long], Long] =\n    staleTweetsCacheCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[PipelineQuery, Seq[Long]] =\n    EditedTweetsCandidatePipelineQueryTransformer(PersistenceEntriesFeature)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[Long, TweetCandidate] = {\n    candidate => TweetCandidate(id = candidate)\n  }\n\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[PipelineQuery, TweetCandidate, _]\n  ] = Seq(namesFeatureHydrator)\n\n  override val decorator: Option[CandidateDecorator[PipelineQuery, TweetCandidate]] = {\n    val tweetItemBuilder = TweetCandidateUrtItemBuilder[PipelineQuery, TweetCandidate](\n      clientEventInfoBuilder = EmptyClientEventInfoBuilder,\n      entryIdToReplaceBuilder = Some((_, candidate, _) =>\n        Some(s\"${TweetItem.TweetEntryNamespace}-${candidate.id.toString}\")),\n      contextualTweetRefBuilder = Some(\n        ContextualTweetRefBuilder(\n          TweetHydrationContext(\n            // Apply safety level that includes canonical VF treatments that apply regardless of context.\n            safetyLevelOverride = Some(TimelineFocalTweetSafetyLevel),\n            outerTweetContext = None\n          )\n        )\n      ),\n      feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder)\n    )\n\n    Some(UrtItemCandidateDecorator(tweetItemBuilder))\n  }\n\n  override val alerts =\n    Seq(HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.5, 50, 60, 60))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline/NewTweetsPillCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.candidate_pipeline\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.functional_component.gate.RequestContextNotGate\nimport com.twitter.home_mixer.model.HomeFeatures.GetNewerFeature\nimport com.twitter.home_mixer.model.request.DeviceContext\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.DurationParamBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.ShowAlertCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.StaticShowAlertColorConfigurationBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.StaticShowAlertDisplayLocationBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.StaticShowAlertIconDisplayInfoBuilder\nimport com.twitter.product_mixer.component_library.gate.FeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.StaticCandidateSource\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseDurationBuilder\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.NewTweets\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertColorConfiguration\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertIconDisplayInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.Top\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.UpArrow\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.TwitterBlueRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.WhiteRosettaColor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.util.Duration\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Candidate Pipeline Config that creates the New Tweets Pill\n */\n@Singleton\nclass NewTweetsPillCandidatePipelineConfig[Query <: PipelineQuery with HasDeviceContext] @Inject() (\n) extends DependentCandidatePipelineConfig[\n      Query,\n      Unit,\n      ShowAlertCandidate,\n      ShowAlertCandidate\n    ] {\n  import NewTweetsPillCandidatePipelineConfig._\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"NewTweetsPill\")\n\n  override val gates: Seq[Gate[Query]] = Seq(\n    RequestContextNotGate(Seq(DeviceContext.RequestContext.PullToRefresh)),\n    FeatureGate.fromFeature(GetNewerFeature)\n  )\n\n  override val candidateSource: CandidateSource[Unit, ShowAlertCandidate] =\n    StaticCandidateSource(\n      CandidateSourceIdentifier(identifier.name),\n      Seq(ShowAlertCandidate(id = identifier.name, userIds = Seq.empty))\n    )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[Query, Unit] = { _ => Unit }\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    ShowAlertCandidate,\n    ShowAlertCandidate\n  ] = { candidate => candidate }\n\n  override val decorator: Option[CandidateDecorator[Query, ShowAlertCandidate]] = {\n    val triggerDelayBuilder = new BaseDurationBuilder[Query] {\n      override def apply(\n        query: Query,\n        candidate: ShowAlertCandidate,\n        features: FeatureMap\n      ): Option[Duration] = {\n        val delay = query.deviceContext.flatMap(_.requestContextValue) match {\n          case Some(DeviceContext.RequestContext.TweetSelfThread) => 0.millis\n          case Some(DeviceContext.RequestContext.ManualRefresh) => 0.millis\n          case _ => TriggerDelay\n        }\n\n        Some(delay)\n      }\n    }\n\n    val homeShowAlertCandidateBuilder = ShowAlertCandidateUrtItemBuilder(\n      alertType = NewTweets,\n      colorConfigBuilder = StaticShowAlertColorConfigurationBuilder(DefaultColorConfig),\n      displayLocationBuilder = StaticShowAlertDisplayLocationBuilder(Top),\n      triggerDelayBuilder = Some(triggerDelayBuilder),\n      displayDurationBuilder = Some(DurationParamBuilder(StaticParam(DisplayDuration))),\n      iconDisplayInfoBuilder = Some(StaticShowAlertIconDisplayInfoBuilder(DefaultIconDisplayInfo))\n    )\n\n    Some(UrtItemCandidateDecorator(homeShowAlertCandidateBuilder))\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),\n    HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()\n  )\n}\n\nobject NewTweetsPillCandidatePipelineConfig {\n  val DefaultColorConfig: ShowAlertColorConfiguration = ShowAlertColorConfiguration(\n    background = TwitterBlueRosettaColor,\n    text = WhiteRosettaColor,\n    border = Some(WhiteRosettaColor)\n  )\n\n  val DefaultIconDisplayInfo: ShowAlertIconDisplayInfo =\n    ShowAlertIconDisplayInfo(icon = UpArrow, tint = WhiteRosettaColor)\n\n  // Unlimited display time (until user takes action)\n  val DisplayDuration = -1.millisecond\n  val TriggerDelay = 4.minutes\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline/TimelineServiceResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.candidate_pipeline\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.timelineservice.{thriftscala => t}\n\nobject TimelineServiceResponseFeatureTransformer extends CandidateFeatureTransformer[t.Tweet] {\n\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"TimelineServiceResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    AuthorIdFeature,\n    InReplyToTweetIdFeature,\n    IsRetweetFeature,\n    SourceTweetIdFeature,\n    SourceUserIdFeature,\n  )\n\n  override def transform(candidate: t.Tweet): FeatureMap = FeatureMapBuilder()\n    .add(AuthorIdFeature, candidate.userId)\n    .add(InReplyToTweetIdFeature, candidate.inReplyToStatusId)\n    .add(IsRetweetFeature, candidate.sourceStatusId.isDefined)\n    .add(SourceTweetIdFeature, candidate.sourceStatusId)\n    .add(SourceUserIdFeature, candidate.sourceUserId)\n    .build()\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline/VerifiedPromptCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.candidate_pipeline\n\nimport com.twitter.home_mixer.functional_component.decorator.builder.VerifiedPromptBuilder\nimport com.twitter.home_mixer.functional_component.gate.RateLimitNotGate\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.InlinePromptCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.IdentityCandidateExtractor\nimport com.twitter.product_mixer.core.functional_component.candidate_source.PassthroughCandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.ExternalStringRegistry\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass VerifiedPromptCandidatePipelineConfig @Inject() (\n  identityCandidateExtractor: IdentityCandidateExtractor[PipelineQuery],\n  @ProductScoped externalStringRegistryProvider: Provider[ExternalStringRegistry],\n  @ProductScoped stringCenterProvider: Provider[StringCenter])\n    extends DependentCandidatePipelineConfig[\n      PipelineQuery,\n      PipelineQuery,\n      PipelineQuery,\n      InlinePromptCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"VerifiedPrompt\")\n\n  override val gates: Seq[Gate[PipelineQuery]] = Seq(RateLimitNotGate)\n\n  override def candidateSource: CandidateSource[\n    PipelineQuery,\n    PipelineQuery\n  ] = PassthroughCandidateSource(\n    CandidateSourceIdentifier(\"VerifiedPassthroughCandidateSource\"),\n    identityCandidateExtractor\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[PipelineQuery, PipelineQuery] =\n    identity\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    PipelineQuery,\n    InlinePromptCandidate\n  ] = { query => InlinePromptCandidate(id = query.getRequiredUserId.toString) }\n\n  override val decorator: Option[\n    CandidateDecorator[PipelineQuery, InlinePromptCandidate]\n  ] = {\n    val clientEventInfoBuilder =\n      ClientEventInfoBuilder[PipelineQuery, InlinePromptCandidate](\"verified_prompt\")\n    val stringCenter = stringCenterProvider.get()\n    val externalStringRegistry = externalStringRegistryProvider.get()\n\n    val verifiedPromptBuilder = VerifiedPromptBuilder(\n      clientEventInfoBuilder,\n      stringCenter,\n      externalStringRegistry\n    )\n\n    Some(UrtItemCandidateDecorator(verifiedPromptBuilder))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/controller/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/http-server/src/main/scala/com/twitter/finatra/http\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift:controller\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/debug_query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/urt\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/controller/HomeHttpController.scala",
    "content": "package com.twitter.home_mixer.controller\n\nimport com.twitter.finagle.http.Request\nimport com.twitter.finatra.http.Controller\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.model.request.ScoredTweetsProduct\nimport com.twitter.home_mixer.model.request.ScoredTweetsProductContext\nimport com.twitter.home_mixer.service.ScoredTweetsService\nimport com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.jackson.ScalaObjectMapper\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeHttpController @Inject() (\n  scoredTweetsService: ScoredTweetsService,\n  paramsBuilder: ParamsBuilder,\n  mapper: ScalaObjectMapper)\n    extends Controller {\n\n  private val UserIdParam = \"userId\"\n  private val CountryCode = \"US\"\n  private val LanguageCode = \"en\"\n  private val AppId = 1L\n  private val DefaultUserId = 1L\n  private val UserAgent = \"\"\n\n  private val BaseClientContext = ClientContext(\n    userId = Some(DefaultUserId),\n    guestId = None,\n    appId = Some(AppId),\n    ipAddress = None,\n    userAgent = Some(UserAgent),\n    countryCode = Some(CountryCode),\n    languageCode = Some(LanguageCode),\n    isTwoffice = None,\n    userRoles = None,\n    deviceId = None,\n    mobileDeviceId = None,\n    mobileDeviceAdId = None,\n    limitAdTracking = None,\n    guestIdAds = None,\n    guestIdMarketing = None,\n    authenticatedUserId = None,\n    isVerifiedCrawler = None\n  )\n\n  case class ScoredTweetMetadata(tweetId: Long, authorId: Long, inNetwork: Boolean, text: String)\n\n  get(\"/scoredTweets\") { request: Request =>\n    val userId = request.getLongParam(UserIdParam)\n    val hmRequest = HomeMixerRequest(\n      clientContext = BaseClientContext.copy(userId = Some(userId)),\n      product = ScoredTweetsProduct,\n      productContext =\n        Some(ScoredTweetsProductContext(None, None, None, None, None, None, None, None)),\n      serializedRequestCursor = None,\n      maxResults = None,\n      debugParams = None,\n      homeRequestParam = false\n    )\n\n    val params = buildParams(hmRequest)\n    val response = scoredTweetsService.getScoredTweetsResponse[HomeMixerRequest](hmRequest, params)\n    Stitch.run(response).map { scoredTweetsResponse =>\n      scoredTweetsResponse.scoredTweets.map { tweet =>\n        val tweetData = ScoredTweetMetadata(\n          tweet.tweetId,\n          tweet.authorId,\n          tweet.inNetwork.getOrElse(false),\n          tweet.tweetText.getOrElse(\"\")\n        )\n        mapper.writeValueAsString(tweetData)\n      }\n    }\n  }\n\n  private def buildParams(request: HomeMixerRequest): Params = {\n    val userAgeOpt = request.clientContext.userId.map { userId =>\n      SnowflakeId.timeFromIdOpt(userId).map(_.untilNow.inDays).getOrElse(Int.MaxValue)\n    }\n    val fsCustomMapInput = userAgeOpt.map(\"account_age_in_days\" -> _).toMap\n    paramsBuilder.build(\n      clientContext = request.clientContext,\n      product = request.product,\n      featureOverrides = request.debugParams.flatMap(_.featureOverrides).getOrElse(Map.empty),\n      fsCustomMapInput = fsCustomMapInput\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/controller/HomeThriftController.scala",
    "content": "package com.twitter.home_mixer.controller\n\nimport com.twitter.finatra.thrift.Controller\nimport com.twitter.home_mixer.marshaller.request.HomeMixerRequestUnmarshaller\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.service.ScoredTweetsService\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.core.controllers.DebugTwitterContext\nimport com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder\nimport com.twitter.product_mixer.core.service.debug_query.DebugQueryService\nimport com.twitter.product_mixer.core.service.urt.UrtService\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Params\nimport javax.inject.Inject\nimport server.src.main.scala.com.twitter.home_mixer.service.HeavyRankerScoresService\n\nclass HomeThriftController @Inject() (\n  homeRequestUnmarshaller: HomeMixerRequestUnmarshaller,\n  debugQueryService: DebugQueryService,\n  urtService: UrtService,\n  scoredTweetsService: ScoredTweetsService,\n  heavyRankerScoresService: HeavyRankerScoresService,\n  paramsBuilder: ParamsBuilder)\n    extends Controller(t.HomeMixer)\n    with DebugTwitterContext {\n\n  handle(t.HomeMixer.GetUrtResponse) { args: t.HomeMixer.GetUrtResponse.Args =>\n    val request = homeRequestUnmarshaller(args.request)\n    val params = buildParams(request)\n    Stitch.run(urtService.getUrtResponse[HomeMixerRequest](request, params))\n  }\n\n  handle(t.HomeMixer.DebugGetUrtResponse) { args: t.HomeMixer.DebugGetUrtResponse.Args =>\n    val request = homeRequestUnmarshaller(args.request)\n    val params = buildParams(request)\n    withDebugTwitterContext(request.clientContext) {\n      Stitch.run(urtService.getUrtResponse[HomeMixerRequest](request, params))\n    }\n  }\n\n  // Handle debug requests\n  handle(t.HomeMixer.ExecutePipeline)\n    .withService(debugQueryService(homeRequestUnmarshaller.apply))\n\n  handle(t.HomeMixer.GetScoredTweetsResponse) { args: t.HomeMixer.GetScoredTweetsResponse.Args =>\n    val request = homeRequestUnmarshaller(args.request)\n    val params = buildParams(request)\n    withDebugTwitterContext(request.clientContext) {\n      Stitch.run(scoredTweetsService.getScoredTweetsResponse[HomeMixerRequest](request, params))\n    }\n  }\n\n  handle(t.HomeMixer.GetHeavyRankerScoresResponse) {\n    args: t.HomeMixer.GetHeavyRankerScoresResponse.Args =>\n      val request = homeRequestUnmarshaller(args.request)\n      val params = buildParams(request)\n      withDebugTwitterContext(request.clientContext) {\n        Stitch.run(\n          heavyRankerScoresService.getHeavyRankerScoresResponse[HomeMixerRequest](request, params))\n      }\n  }\n\n  private def buildParams(request: HomeMixerRequest): Params = {\n    val userAgeOpt = request.clientContext.userId.map { userId =>\n      // Setting to Int.MaxValue for cases where id is not snowflake id as they are pretty old accounts\n      SnowflakeId.timeFromIdOpt(userId).map(_.untilNow.inDays).getOrElse(Int.MaxValue)\n    }\n    val fsCustomMapInput = userAgeOpt.map(\"account_age_in_days\" -> _).toMap\n    paramsBuilder.build(\n      clientContext = request.clientContext,\n      product = request.product,\n      featureOverrides = request.debugParams.flatMap(_.featureOverrides).getOrElse(Map.empty),\n      fsCustomMapInput = fsCustomMapInput\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/federated/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"joinkey/src/main/scala/com/twitter/joinkey/context\",\n        \"joinkey/src/main/thrift/com/twitter/joinkey/context:joinkey-context-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"stitch/stitch-gizmoduck\",\n        \"strato/config/columns/auth-context:auth-context-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/callcontext\",\n        \"strato/src/main/scala/com/twitter/strato/fed/server\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/federated/HomeMixerColumn.scala",
    "content": "package com.twitter.home_mixer.federated\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.filter.SLODefinition\nimport com.twitter.finagle.filter.SLOStatsFilter\nimport com.twitter.finagle.service.ResponseClassifier\nimport com.twitter.finagle.stats.DefaultStatsReceiver\nimport com.twitter.gizmoduck.{thriftscala => gd}\nimport com.twitter.home_mixer.marshaller.request.HomeMixerRequestUnmarshaller\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.{thriftscala => hm}\nimport com.twitter.joinkey.context.thriftscala.{RequestJoinKeyContext => ThriftJoinKeyContext}\nimport com.twitter.joinkey.context.RequestJoinKeyContext\nimport com.twitter.joinkey.context.ThreadedLocalJoinKeyRandomGenerator\nimport com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineRequest\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineResult\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry\nimport com.twitter.product_mixer.core.{thriftscala => pm}\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.Stitch.Letter\nimport com.twitter.stitch.gizmoduck.Gizmoduck\nimport com.twitter.strato.callcontext.CallContext\nimport com.twitter.strato.catalog.OpMetadata\nimport com.twitter.strato.config._\nimport com.twitter.strato.data._\nimport com.twitter.strato.fed.StratoFed\nimport com.twitter.strato.generated.client.auth_context.AuditIpClientColumn\nimport com.twitter.strato.graphql.timelines.{thriftscala => gql}\nimport com.twitter.strato.thrift.ScroogeConv\nimport com.twitter.timelines.render.thriftscala.Timeline\nimport com.twitter.timelines.render.{thriftscala => tr}\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeMixerColumn @Inject() (\n  homeMixerRequestUnmarshaller: HomeMixerRequestUnmarshaller,\n  gizmoduck: Gizmoduck,\n  auditIpClientColumn: AuditIpClientColumn,\n  paramsBuilder: ParamsBuilder,\n  productPipelineRegistry: ProductPipelineRegistry)\n    extends StratoFed.Column(HomeMixerColumn.Path)\n    with StratoFed.Fetch.Arrow {\n\n  override val contactInfo: ContactInfo = ContactInfo(\n    contactEmail = \"\",\n    ldapGroup = \"\",\n    jiraProject = \"\",\n    slackRoomId = \"\"\n  )\n\n  override val metadata: OpMetadata =\n    OpMetadata(\n      lifecycle = Some(Lifecycle.Production),\n      description =\n        Some(Description.PlainText(\"Federated Strato column for Timelines served via Home Mixer\"))\n    )\n\n  private val bouncerAccess: Seq[Policy] = Seq(BouncerAccess())\n  private val finatraTestServiceIdentifiers: Seq[Policy] = Seq(\n    ServiceIdentifierPattern(\n      role = \"\",\n      service = \"\",\n      env = \"\",\n      zone = Seq(\"\"))\n  )\n\n  private val sloStatsFilter = {\n    val requestToSLODefinition: PartialFunction[Any, SLODefinition] = {\n      case gql.TimelineKey.HomeTimeline(_) | gql.TimelineKey.HomeTimelineV2(_) =>\n        SLODefinition(\"ForYou\", 1000.millis)\n    }\n    new SLOStatsFilter[gql.TimelineKey, Result[Timeline]](\n      requestToSLODefinition = requestToSLODefinition,\n      responseClassifier = ResponseClassifier.Default,\n      statsReceiver = DefaultStatsReceiver.scope(\"slo\")\n    )\n  }\n\n  override val policy: Policy = AnyOf(bouncerAccess ++ finatraTestServiceIdentifiers)\n\n  override type Key = gql.TimelineKey\n  override type View = gql.HomeTimelineView\n  override type Value = tr.Timeline\n\n  override val keyConv: Conv[Key] = ScroogeConv.fromStruct[gql.TimelineKey]\n  override val viewConv: Conv[View] = ScroogeConv.fromStruct[gql.HomeTimelineView]\n  override val valueConv: Conv[Value] = ScroogeConv.fromStruct[tr.Timeline]\n\n  // For populating requestJoinId\n  private val joinIdGenerator = new ThreadedLocalJoinKeyRandomGenerator\n  // For populating user roles\n  private val queryFieldsRoles = gd.QueryFields.Roles\n  private object RequestJoinKeyLetter extends Letter[Option[Long]] {\n    override def let[A](requestJoinId: Option[Long])(fn: => A): A =\n      RequestJoinKeyContext.let(ThriftJoinKeyContext(requestJoinId))(fn)\n  }\n\n  private def createHomeMixerRequestArrow(\n    auditIpClientColumn: AuditIpClientColumn,\n  ): Arrow[(Key, View), hm.HomeMixerRequest] = {\n\n    val populateUserRolesAndIp: Arrow[(Key, View), (Option[Set[String]], Option[String])] = {\n      val populateUserRoles = Arrow\n        .flatMap[(Key, View), Option[Set[String]]] { _ =>\n          Stitch.collect {\n            CallContext.twitterUserId.map { userId =>\n              gizmoduck\n                .getUserById(\n                  userId = userId,\n                  queryFields = Set(queryFieldsRoles),\n                  context = gd.LookupContext(forUserId = Some(userId))\n                ).map(_.roles.map(_.roles.toSet).getOrElse(Set.empty))\n            }\n          }\n        }\n\n      val populateIpAddress = Arrow\n        .flatMap[(Key, View), Option[String]](_ =>\n          auditIpClientColumn.fetcher\n            .callStack(HomeMixerColumn.FetchCallstack)\n            .fetch((), ()).map(_.v))\n\n      Arrow.join(\n        populateUserRoles,\n        populateIpAddress\n      )\n    }\n\n    Arrow.zipWithArg(populateUserRolesAndIp).map {\n      case ((key, view), (roles, ipAddress)) =>\n        val deviceContextOpt = Some(\n          hm.DeviceContext(\n            isPolling = CallContext.isPolling,\n            requestContext = view.requestContext,\n            latestControlAvailable = view.latestControlAvailable,\n            autoplayEnabled = view.autoplayEnabled\n          ))\n        val seenTweetIds = view.seenTweetIds.filter(_.nonEmpty)\n\n        val (product, productContext) = key match {\n          case gql.TimelineKey.HomeTimeline(_) | gql.TimelineKey.HomeTimelineV2(_) =>\n            (\n              hm.Product.ForYou,\n              hm.ProductContext.ForYou(\n                hm.ForYou(\n                  deviceContextOpt,\n                  seenTweetIds,\n                  view.dspClientContext,\n                  view.pushToHomeTweetId\n                )\n              ))\n          case gql.TimelineKey.HomeLatestTimeline(_) | gql.TimelineKey.HomeLatestTimelineV2(_) =>\n            (\n              hm.Product.Following,\n              hm.ProductContext.Following(\n                hm.Following(deviceContextOpt, seenTweetIds, view.dspClientContext)))\n          case gql.TimelineKey.CreatorSubscriptionsTimeline(_) =>\n            (\n              hm.Product.Subscribed,\n              hm.ProductContext.Subscribed(hm.Subscribed(deviceContextOpt, seenTweetIds)))\n          case _ => throw new UnsupportedOperationException(s\"Unknown product: $key\")\n        }\n\n        val clientContext = pm.ClientContext(\n          userId = CallContext.twitterUserId,\n          guestId = CallContext.guestId,\n          guestIdAds = CallContext.guestIdAds,\n          guestIdMarketing = CallContext.guestIdMarketing,\n          appId = CallContext.clientApplicationId,\n          ipAddress = ipAddress,\n          userAgent = CallContext.userAgent,\n          countryCode = CallContext.requestCountryCode,\n          languageCode = CallContext.requestLanguageCode,\n          isTwoffice = CallContext.isInternalOrTwoffice,\n          userRoles = roles,\n          deviceId = CallContext.deviceId,\n          mobileDeviceId = CallContext.mobileDeviceId,\n          mobileDeviceAdId = CallContext.adId,\n          limitAdTracking = CallContext.limitAdTracking,\n          authenticatedUserId = CallContext.authenticatedTwitterUserId,\n          isVerifiedCrawler = CallContext.isVerifiedCrawler\n        )\n\n        hm.HomeMixerRequest(\n          clientContext = clientContext,\n          product = product,\n          productContext = Some(productContext),\n          maxResults = Try(view.count.get.toInt).toOption.orElse(HomeMixerColumn.MaxCount),\n          cursor = view.cursor.filter(_.nonEmpty)\n        )\n    }\n  }\n\n  override val fetch: Arrow[(Key, View), Result[Value]] = {\n    val transformThriftIntoPipelineRequest: Arrow[\n      (Key, View),\n      ProductPipelineRequest[HomeMixerRequest]\n    ] = {\n      Arrow\n        .identity[(Key, View)]\n        .andThen {\n          createHomeMixerRequestArrow(auditIpClientColumn)\n        }\n        .map { thriftRequest =>\n          val request = homeMixerRequestUnmarshaller(thriftRequest)\n          val params = paramsBuilder.build(\n            clientContext = request.clientContext,\n            product = request.product,\n            featureOverrides = request.debugParams.flatMap(_.featureOverrides).getOrElse(Map.empty),\n          )\n          ProductPipelineRequest(request, params)\n        }\n    }\n\n    val underlyingProduct: Arrow[\n      ProductPipelineRequest[HomeMixerRequest],\n      ProductPipelineResult[tr.TimelineResponse]\n    ] = Arrow\n      .identity[ProductPipelineRequest[HomeMixerRequest]]\n      .map { pipelineRequest =>\n        val pipelineArrow =\n          Arrow.let(RequestJoinKeyLetter)(joinIdGenerator.get().getNewRequestJoinId) { // Populate requestJoinId\n            productPipelineRegistry\n              .getProductPipeline[HomeMixerRequest, tr.TimelineResponse](\n                pipelineRequest.request.product)\n              .arrow\n          }\n        (pipelineArrow, pipelineRequest)\n      }.applyArrow\n\n    Arrow\n      .zipWithArg(\n        Arrow.time {\n          transformThriftIntoPipelineRequest\n            .andThen(underlyingProduct)\n            .map {\n              _.result match {\n                case Some(result) => found(result.timeline)\n                case _ => missing\n              }\n            }\n        }\n      ).map {\n        case ((key, _), (result, duration)) =>\n          sloStatsFilter.record(key, result, duration.inNanoseconds)\n          result\n      }.lowerFromTry\n  }\n}\n\nobject HomeMixerColumn {\n  val Path = \"home-mixer/homeMixer.Timeline\"\n  private val FetchCallstack = s\"$Path:fetch\"\n  private val MaxCount: Option[Int] = Some(100)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/candidate_source/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finagle/finagle-memcached/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"stitch/stitch-timelineservice/src/main/scala\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/candidate_source/EarlybirdCandidateSource.scala",
    "content": "package com.twitter.home_mixer.functional_component.candidate_source\n\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.search.earlybird.{thriftscala => t}\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase object EarlybirdResponseTruncatedFeature\n    extends FeatureWithDefaultOnFailure[t.EarlybirdRequest, Boolean] {\n  override val defaultValue: Boolean = false\n}\n\ncase object EarlybirdBottomTweetFeature\n    extends FeatureWithDefaultOnFailure[t.EarlybirdRequest, Option[Long]] {\n  override val defaultValue: Option[Long] = None\n}\n\n@Singleton\ncase class EarlybirdCandidateSource @Inject() (\n  earlybird: t.EarlybirdService.MethodPerEndpoint)\n    extends CandidateSourceWithExtractedFeatures[t.EarlybirdRequest, t.ThriftSearchResult] {\n\n  override val identifier = CandidateSourceIdentifier(\"Earlybird\")\n\n  override def apply(\n    request: t.EarlybirdRequest\n  ): Stitch[CandidatesWithSourceFeatures[t.ThriftSearchResult]] = {\n    Stitch.callFuture(earlybird.search(request)).map { response =>\n      val candidates = response.searchResults.map(_.results).getOrElse(Seq.empty)\n\n      val features = FeatureMapBuilder()\n        .add(EarlybirdResponseTruncatedFeature, candidates.size == request.searchQuery.numResults)\n        .add(EarlybirdBottomTweetFeature, candidates.lastOption.map(_.id))\n        .build()\n\n      CandidatesWithSourceFeatures(candidates, features)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/candidate_source/StaleTweetsCacheCandidateSource.scala",
    "content": "package com.twitter.home_mixer.functional_component.candidate_source\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.StaleTweetsCache\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass StaleTweetsCacheCandidateSource @Inject() (\n  @Named(StaleTweetsCache) staleTweetsCache: MemcachedClient)\n    extends CandidateSource[Seq[Long], Long] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"StaleTweetsCache\")\n\n  private val StaleTweetsCacheKeyPrefix = \"v1_\"\n\n  override def apply(request: Seq[Long]): Stitch[Seq[Long]] = {\n    val keys = request.map(StaleTweetsCacheKeyPrefix + _)\n\n    Stitch.callFuture(staleTweetsCache.get(keys).map { tweets =>\n      tweets.map {\n        case (k, _) => k.replaceFirst(StaleTweetsCacheKeyPrefix, \"\").toLong\n      }.toSeq\n    })\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/pivot\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/trends_events\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/EntryPointPivotModuleDecorator.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator\n\nimport com.twitter.home_mixer.functional_component.decorator.EntryPointPivotModuleCandidateDecorator.EntryNamespaceString\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemInModuleDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.pivot.PivotCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.SuggestTypeClientEventDetailsBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleHeaderBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.pivot.PivotCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseStr\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Vertical\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\n\ncase class StrEntryPointPivotCategoryText(\n  defaultText: String)\n    extends BaseStr[PipelineQuery, PivotCandidate] {\n  def apply(\n    query: PipelineQuery,\n    candidate: PivotCandidate,\n    candidateFeatures: FeatureMap\n  ): String =\n    candidate.categoryText.getOrElse(defaultText)\n}\nobject EntryPointPivotModuleCandidateDecorator {\n  val EntryNamespaceString = \"entry-point-pivot\"\n}\ncase class EntryPointPivotModuleCandidateDecorator(\n  component: String,\n  headerText: BaseStr[PipelineQuery, PivotCandidate]) {\n\n  private val clientEventsBuilder =\n    ClientEventInfoBuilder[PipelineQuery, PivotCandidate](component)\n  private val clientEventDetailsBuilder =\n    SuggestTypeClientEventDetailsBuilder(st.SuggestType.EntryPointPivot)\n\n  private val itemBuilder =\n    PivotCandidateUrtItemBuilder(clientEventInfoBuilder = Some(clientEventDetailsBuilder))\n  private val itemDecorator = UrtItemCandidateDecorator(itemBuilder)\n\n  private val moduleHeaderBuilder = ModuleHeaderBuilder(\n    textBuilder = headerText,\n    isSticky = Some(false),\n    urlBuilder = None\n  )\n\n  private val moduleBuilder = TimelineModuleBuilder(\n    entryNamespace = EntryNamespace(EntryNamespaceString),\n    displayTypeBuilder = StaticModuleDisplayTypeBuilder(Vertical),\n    clientEventInfoBuilder = clientEventsBuilder,\n    headerBuilder = Some(moduleHeaderBuilder),\n    footerBuilder = None\n  )\n\n  val moduleDecorator: UrtItemInModuleDecorator[PipelineQuery, PivotCandidate, Nothing] =\n    UrtItemInModuleDecorator(itemDecorator, moduleBuilder)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/ForYouTweetCandidateDecorator.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator\n\nimport com.twitter.home_mixer.functional_component.decorator.ForYouTweetCandidateDecorator.ConvoModuleEntryNamespace\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeClientEventInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeConversationModuleMetadataBuilder\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeTimelinesScoreInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeTweetContextBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeTweetSocialContextBuilder\nimport com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ManualModuleId\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject ForYouTweetCandidateDecorator {\n  val ConvoModuleEntryNamespace = EntryNamespace(\"home-conversation\")\n}\n\n@Singleton\nclass ForYouTweetCandidateDecorator @Inject() (\n  homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder,\n  homeTweetSocialContextBuilder: HomeTweetSocialContextBuilder,\n  homeTweetContextBuilder: HomeTweetContextBuilder) {\n\n  val clientEventInfoBuilder = HomeClientEventInfoBuilder()\n\n  val tweetItemBuilder = TweetCandidateUrtItemBuilder(\n    clientEventInfoBuilder = clientEventInfoBuilder,\n    socialContextBuilder = Some(homeTweetSocialContextBuilder),\n    timelinesScoreInfoBuilder = Some(HomeTimelinesScoreInfoBuilder),\n    feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder),\n    tweetContext = Some(homeTweetContextBuilder)\n  )\n\n  val tweetDecorator = UrtItemCandidateDecorator(tweetItemBuilder)\n\n  val moduleBuilder = TimelineModuleBuilder(\n    entryNamespace = ConvoModuleEntryNamespace,\n    clientEventInfoBuilder = clientEventInfoBuilder,\n    moduleIdGeneration = ManualModuleId(0L),\n    displayTypeBuilder = StaticModuleDisplayTypeBuilder(VerticalConversation),\n    metadataBuilder = Some(HomeConversationModuleMetadataBuilder())\n  )\n\n  val decorator: UrtMultipleModulesDecorator[PipelineQuery, TweetCandidate, Long] =\n    UrtMultipleModulesDecorator(\n      urtItemCandidateDecorator = tweetDecorator,\n      moduleBuilder = moduleBuilder,\n      groupByKey = (_, _, candidateFeatures) =>\n        candidateFeatures.getOrElse(ConversationModuleFocalTweetIdFeature, None)\n    )\n\n  def build(): UrtMultipleModulesDecorator[PipelineQuery, TweetCandidate, Long] = decorator\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/HomeConversationServiceCandidateDecorator.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator\n\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeConversationModuleMetadataBuilder\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeTimelinesScoreInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.{HomeFeedbackActionInfoBuilder, HomeTweetContextBuilder}\nimport com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.social_context.CommunitiesSocialContextBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject HomeConversationServiceCandidateDecorator {\n\n  private val ConversationModuleNamespace = EntryNamespace(\"home-conversation\")\n\n  def apply(\n    homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder,\n    homeTweetContextBuilder: HomeTweetContextBuilder,\n    servedType: hmt.ServedType\n  ): Some[UrtMultipleModulesDecorator[PipelineQuery, TweetCandidate, Long]] = {\n    val component = servedType.originalName\n    val clientEventInfoBuilder = ClientEventInfoBuilder(component)\n    val tweetItemBuilder = TweetCandidateUrtItemBuilder(\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      timelinesScoreInfoBuilder = Some(HomeTimelinesScoreInfoBuilder),\n      feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder),\n      socialContextBuilder = Some(CommunitiesSocialContextBuilder),\n      tweetContext = Some(homeTweetContextBuilder)\n    )\n\n    val moduleBuilder = TimelineModuleBuilder(\n      entryNamespace = ConversationModuleNamespace,\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      displayTypeBuilder = StaticModuleDisplayTypeBuilder(VerticalConversation),\n      metadataBuilder = Some(HomeConversationModuleMetadataBuilder())\n    )\n\n    Some(\n      UrtMultipleModulesDecorator(\n        urtItemCandidateDecorator = UrtItemCandidateDecorator(tweetItemBuilder),\n        moduleBuilder = moduleBuilder,\n        groupByKey = (_, _, candidateFeatures) =>\n          candidateFeatures.getOrElse(ConversationModuleFocalTweetIdFeature, None)\n      ))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/HomeQueryTypePredicates.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator\n\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\n\nobject HomeQueryTypePredicates {\n  private[this] val QueryPredicates: Seq[(String, FeatureMap => Boolean)] = Seq(\n    (\"request\", _ => true),\n    (\"get_initial\", _.getOrElse(GetInitialFeature, false)),\n    (\"get_newer\", _.getOrElse(GetNewerFeature, false)),\n    (\"get_older\", _.getOrElse(GetOlderFeature, false)),\n    (\"pull_to_refresh\", _.getOrElse(PullToRefreshFeature, false)),\n    (\"request_context_launch\", _.getOrElse(IsLaunchRequestFeature, false)),\n    (\"request_context_foreground\", _.getOrElse(IsForegroundRequestFeature, false))\n  )\n\n  val PredicateMap = QueryPredicates.toMap\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/KeywordTrendsModuleCandidateDecorator.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator\n\nimport com.twitter.home_mixer.functional_component.decorator.KeywordTrendsModuleCandidateDecorator.Component\nimport com.twitter.home_mixer.functional_component.decorator.KeywordTrendsModuleCandidateDecorator.KeywordTrendsEntryNamespace\nimport com.twitter.home_mixer.functional_component.decorator.KeywordTrendsModuleCandidateDecorator.TrendsUrl\nimport com.twitter.home_mixer.functional_component.decorator.builder.KeywordTrendMetaDescriptionBuilder\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemInModuleDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.trend.TrendCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.trend.TrendPromotedMetadataBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.trend.TrendRankBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.StaticUrlBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.Str\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleHeaderBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.UnifiedTrendCandidate\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ExternalUrl\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Vertical\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject KeywordTrendsModuleCandidateDecorator {\n  val KeywordTrendsEntryNamespace = EntryNamespace(\"keyword-trends\")\n  private val Component = \"trend\"\n  private val TrendsUrl = \"\"\n}\n\n@Singleton\nclass KeywordTrendsModuleCandidateDecorator @Inject() (\n  keywordTrendMetaDescriptionBuilder: KeywordTrendMetaDescriptionBuilder,\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings) {\n\n  private val clientEventInfoBuilder =\n    ClientEventInfoBuilder[PipelineQuery, UnifiedTrendCandidate](Component)\n\n  private val trendItemBuilder = TrendCandidateUrtItemBuilder(\n    keywordTrendMetaDescriptionBuilder,\n    TrendPromotedMetadataBuilder,\n    clientEventInfoBuilder,\n    Some(TrendRankBuilder[PipelineQuery, UnifiedTrendCandidate]())\n  )\n  private val trendItemDecorator = UrtItemCandidateDecorator(trendItemBuilder)\n\n  private val moduleHeaderBuilder = ModuleHeaderBuilder(\n    textBuilder = Str(\n      text = externalStrings.TrendingString,\n      stringCenter = stringCenter\n    ),\n    isSticky = Some(false),\n    urlBuilder = Some(StaticUrlBuilder(TrendsUrl, ExternalUrl))\n  )\n  private val moduleBuilder = TimelineModuleBuilder(\n    entryNamespace = KeywordTrendsEntryNamespace,\n    displayTypeBuilder = StaticModuleDisplayTypeBuilder(Vertical),\n    clientEventInfoBuilder = clientEventInfoBuilder,\n    headerBuilder = Some(moduleHeaderBuilder)\n  )\n\n  val moduleDecorator = UrtItemInModuleDecorator(trendItemDecorator, moduleBuilder)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/PinnedTweetBroadcastCandidateDecorator.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator\n\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeClientEventInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeTimelinesScoreInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.PinnedTweetsModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.TweetCarouselModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.TweetCarouselType\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableLandingPage\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnablePinnedTweetsCarouselParam\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.social_context.GeneralSocialContextBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.Str\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ExternalUrl\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.PinGeneralContextType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stitch.Stitch\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass PinnedTweetBroadcastCandidateDecorator @Inject() (\n  tweetCarouselModuleCandidateDecorator: TweetCarouselModuleCandidateDecorator,\n  @ProductScoped stringCenterProvider: Provider[StringCenter],\n  externalStrings: HomeMixerExternalStrings,\n  homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder)\n    extends CandidateDecorator[PipelineQuery, TweetCandidate] {\n\n  private val stringCenter = stringCenterProvider.get()\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[Decoration]] = {\n    if (candidates.size > 1 && query.params(EnablePinnedTweetsCarouselParam)) {\n      tweetCarouselModuleCandidateDecorator\n        .build(TweetCarouselType.PinnedTweets).apply(query, candidates)\n    } else {\n      val clientEventInfoBuilder = HomeClientEventInfoBuilder[PipelineQuery, TweetCandidate]()\n\n      val landingUrl = if (query.params(EnableLandingPage)) {\n        PinnedTweetsModuleCandidateDecorator.headerLink.map(link => Url(ExternalUrl, link))\n      } else None\n\n      val socialContextBuilder = GeneralSocialContextBuilder(\n        textBuilder = Str(externalStrings.BroadcastedPinnedTweetSocialContextString, stringCenter),\n        contextType = PinGeneralContextType,\n        landingUrl = landingUrl\n      )\n\n      val tweetItemBuilder = TweetCandidateUrtItemBuilder[PipelineQuery, TweetCandidate](\n        clientEventInfoBuilder = clientEventInfoBuilder,\n        socialContextBuilder = Some(socialContextBuilder),\n        timelinesScoreInfoBuilder = Some(HomeTimelinesScoreInfoBuilder),\n        feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder)\n      )\n\n      UrtItemCandidateDecorator(tweetItemBuilder).apply(query, candidates)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/StoriesModuleCandidateDecorator.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator\n\nimport com.twitter.home_mixer.functional_component.decorator.StoriesModuleCandidateDecorator.Component\nimport com.twitter.home_mixer.functional_component.decorator.StoriesModuleCandidateDecorator.TrendsEntryNamespace\nimport com.twitter.home_mixer.functional_component.decorator.StoriesModuleCandidateDecorator.TrendsUrl\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemInModuleDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.trend.AiTrendCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.StaticUrlBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.Str\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleFooterBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleHeaderBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.UnifiedTrendCandidate\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DeepLink\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Vertical\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject StoriesModuleCandidateDecorator {\n  val TrendsEntryNamespace = EntryNamespace(\"stories\")\n  private val Component = \"stories\"\n  private val TrendsUrl = \"\"\n}\n\n@Singleton\nclass StoriesModuleCandidateDecorator @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings) {\n  private val clientEventsBuilder =\n    ClientEventInfoBuilder[PipelineQuery, UnifiedTrendCandidate](Component)\n\n  private val trendItemBuilder = AiTrendCandidateUrtItemBuilder(clientEventsBuilder)\n  private val trendItemDecorator = UrtItemCandidateDecorator(trendItemBuilder)\n\n  private val moduleHeaderBuilder = ModuleHeaderBuilder(\n    textBuilder = Str(\n      text = externalStrings.NewsHeaderString,\n      stringCenter = stringCenter\n    ),\n    isSticky = Some(false),\n    urlBuilder = Some(StaticUrlBuilder(TrendsUrl, DeepLink))\n  )\n\n  private val moduleFooterBuilder = ModuleFooterBuilder(\n    textBuilder = Str(\n      text = externalStrings.NewsFooterString,\n      stringCenter = stringCenter\n    ),\n    urlBuilder = Some(StaticUrlBuilder(TrendsUrl, DeepLink))\n  )\n\n  private val moduleBuilder = TimelineModuleBuilder(\n    entryNamespace = TrendsEntryNamespace,\n    displayTypeBuilder = StaticModuleDisplayTypeBuilder(Vertical),\n    clientEventInfoBuilder = clientEventsBuilder,\n    headerBuilder = Some(moduleHeaderBuilder),\n    footerBuilder = Some(moduleFooterBuilder)\n  )\n\n  val moduleDecorator: UrtItemInModuleDecorator[PipelineQuery, UnifiedTrendCandidate, Nothing] =\n    UrtItemInModuleDecorator(trendItemDecorator, moduleBuilder)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/TuneFeedModuleCandidateDecorator.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator\n\nimport com.twitter.home_mixer.functional_component.decorator.TuneFeedModuleCandidateDecorator.EntryNamespaceString\nimport com.twitter.home_mixer.functional_component.decorator.TuneFeedModuleCandidateDecorator.GrokTopicBaseUrl\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeClientEventInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.TuneFeedFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.model.HomeFeatures.CurrentDisplayedGrokTopicFeature\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemInModuleDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.icon.HorizonIconBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.StaticUrlBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.social_context.GeneralModuleSocialContextBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.ModuleStrStatic\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.StrStatic\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleFooterBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleFooterDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleHeaderBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleHeaderDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.FeedbackStroke\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ExternalUrl\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TextOnlyGeneralContextType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Feedback\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.FeedbackList\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Separator\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Base64UrlSafeStringEncoder\nimport java.nio.charset.Charset\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject TuneFeedModuleCandidateDecorator {\n  val EntryNamespaceString = \"tune-feed\"\n  val GrokTopicBaseUrl = \"\"\n}\n\n@Singleton\nclass TuneFeedModuleCandidateDecorator @Inject() (\n  tuneFeedFeedbackActionInfoBuilder: TuneFeedFeedbackActionInfoBuilder)\n    extends CandidateDecorator[PipelineQuery, TweetCandidate] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[Decoration]] = {\n\n    val clientEventInfoBuilder = HomeClientEventInfoBuilder[PipelineQuery, TweetCandidate]()\n\n    val tuneFeedItemBuilder =\n      TweetCandidateUrtItemBuilder[PipelineQuery, TweetCandidate](\n        clientEventInfoBuilder = clientEventInfoBuilder,\n        feedbackActionInfoBuilder = Some(tuneFeedFeedbackActionInfoBuilder)\n      )\n\n    val tuneFeedItemDecorator =\n      UrtItemCandidateDecorator(tuneFeedItemBuilder)\n\n    val tuneFeedModuleBuilder = {\n      val generalModuleSocialContextBuilder = GeneralModuleSocialContextBuilder(\n        textBuilder = ModuleStrStatic(text = \"Tune your feed\"),\n        contextType = TextOnlyGeneralContextType\n      )\n\n      // Safe .get, enforced in TuneFeedModuleGate\n      val topicId = query.features.get.getOrElse(CurrentDisplayedGrokTopicFeature, None).get._1\n      val topicUrl = GrokTopicBaseUrl\n\n      val tuneFeedModuleHeaderBuilder = ModuleHeaderBuilder(\n        textBuilder = StrStatic(text = query.features.get\n          .getOrElse(CurrentDisplayedGrokTopicFeature, None).map(_._2).getOrElse(\n            \"Help us show you more of what you love\")),\n        moduleSocialContextBuilder = Some(generalModuleSocialContextBuilder),\n        moduleHeaderIconBuilder = Some(HorizonIconBuilder(FeedbackStroke)),\n        moduleHeaderDisplayTypeBuilder = ModuleHeaderDisplayTypeBuilder(Feedback),\n        urlBuilder = Some(StaticUrlBuilder(topicUrl, ExternalUrl)),\n        isSticky = Some(false)\n      )\n\n      val tuneFeedModuleFooterBuilder = ModuleFooterBuilder(\n        textBuilder = StrStatic(text = \"\"),\n        urlBuilder = None,\n        moduleFooterDisplayTypeBuilder = ModuleFooterDisplayTypeBuilder(Separator)\n      )\n\n      TimelineModuleBuilder(\n        entryNamespace = EntryNamespace(EntryNamespaceString),\n        clientEventInfoBuilder = clientEventInfoBuilder,\n        displayTypeBuilder = StaticModuleDisplayTypeBuilder(FeedbackList),\n        headerBuilder = Some(tuneFeedModuleHeaderBuilder),\n        footerBuilder = Some(tuneFeedModuleFooterBuilder)\n      )\n    }\n    UrtItemInModuleDecorator(tuneFeedItemDecorator, tuneFeedModuleBuilder)\n      .apply(query, candidates)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/TweetCarouselModuleCandidateDecorator.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeClientEventInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeTimelinesScoreInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.TweetCarouselType.BookmarkedTweets\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.TweetCarouselType.CarouselType\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.TweetCarouselType.PinnedTweets\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableLandingPage\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemInModuleDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.Str\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleHeaderBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.Frown\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.CondensedTweet\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Carousel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Classic\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleHeader\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.stringcenter.client.core.ExternalString\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\ntrait ClientComponents {\n  val clientEventComponent: String\n  val entryNamespaceString: String\n  val headerLink: Option[String] = None\n}\n\nobject BookmarksModuleCandidateDecorator extends ClientComponents {\n  override val clientEventComponent = \"bookmarked-tweet\"\n  override val entryNamespaceString = \"bookmarked-tweet\"\n  override val headerLink = Some(\"\")\n}\n\nobject PinnedTweetsModuleCandidateDecorator extends ClientComponents {\n  override val clientEventComponent = \"pinned-tweets\"\n  override val entryNamespaceString = \"pinned-tweets\"\n  override val headerLink = Some(\"\")\n}\n\nobject TweetCarouselType extends Enumeration {\n  type CarouselType = Value\n  val BookmarkedTweets, PinnedTweets = Value\n}\n\n@Singleton\nclass TweetCarouselModuleCandidateDecorator @Inject() (\n  homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder,\n  @ProductScoped stringCenterProvider: Provider[StringCenter],\n  externalStrings: HomeMixerExternalStrings,\n  feedbackStrings: FeedbackStrings) {\n\n  private val stringCenter = stringCenterProvider.get()\n\n  private def getClientComponents(carouselType: CarouselType): ClientComponents = {\n    carouselType match {\n      case BookmarkedTweets => BookmarksModuleCandidateDecorator\n      case PinnedTweets => PinnedTweetsModuleCandidateDecorator\n    }\n  }\n\n  def build(\n    carouselType: CarouselType\n  ): UrtItemInModuleDecorator[PipelineQuery, TweetCandidate, Nothing] = {\n    val components = getClientComponents(carouselType)\n\n    val clientEventInfoBuilder = HomeClientEventInfoBuilder[PipelineQuery, TweetCandidate]()\n\n    val tweetItemBuilder = TweetCandidateUrtItemBuilder[PipelineQuery, TweetCandidate](\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      displayType = CondensedTweet,\n      timelinesScoreInfoBuilder = Some(HomeTimelinesScoreInfoBuilder),\n      feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder)\n    )\n\n    val tweetItemDecorator = UrtItemCandidateDecorator(tweetItemBuilder)\n\n    val headerBuilder =\n      TweetCarouselModuleHeaderBuilder(carouselType, components, stringCenter, externalStrings)\n\n    val tweetCarouselModuleBuilder = TimelineModuleBuilder(\n      entryNamespace = EntryNamespace(components.entryNamespaceString),\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      displayTypeBuilder = StaticModuleDisplayTypeBuilder(Carousel),\n      headerBuilder = Some(headerBuilder),\n      footerBuilder = None,\n      feedbackActionInfoBuilder = Some(\n        FeedbackActionInfoBuilder(\n          seeLessOftenFeedbackString = feedbackStrings.seeLessOftenFeedbackString,\n          seeLessOftenConfirmationFeedbackString = feedbackStrings.seeLessOftenFeedbackString,\n          stringCenter = stringCenter,\n          encodedFeedbackRequest = None\n        )\n      ),\n      showMoreBehaviorBuilder = None\n    )\n\n    UrtItemInModuleDecorator(tweetItemDecorator, tweetCarouselModuleBuilder)\n  }\n}\n\ncase class TweetCarouselModuleHeaderBuilder(\n  carouselType: CarouselType,\n  components: ClientComponents,\n  stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends BaseModuleHeaderBuilder[PipelineQuery, TweetCandidate] {\n\n  private def getLandingUrl(query: PipelineQuery): Option[Url] = {\n    val landingPageUrl = components.headerLink.map { url => Url(ExternalUrl, url) }\n\n    carouselType match {\n      case BookmarkedTweets => landingPageUrl\n      case PinnedTweets if query.params(EnableLandingPage) => landingPageUrl\n      case _ => None\n    }\n  }\n\n  def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Option[ModuleHeader] = {\n    candidates.headOption.map { candidate =>\n      val title = carouselType match {\n        case BookmarkedTweets =>\n          Str(externalStrings.BookmarksHeaderString, stringCenter)\n            .apply(query, candidate.candidate, candidate.features)\n        case PinnedTweets =>\n          Str(externalStrings.PinnedTweetsHeaderString, stringCenter)\n            .apply(query, candidate.candidate, candidate.features)\n      }\n\n      val landingUrl = getLandingUrl(query)\n\n      ModuleHeader(\n        text = title,\n        sticky = Some(false),\n        customIcon = None,\n        socialContext = None,\n        icon = None,\n        moduleHeaderDisplayType = Classic,\n        landingUrl = landingUrl\n      )\n    }\n  }\n}\n\ncase class FeedbackActionInfoBuilder(\n  seeLessOftenFeedbackString: ExternalString,\n  seeLessOftenConfirmationFeedbackString: ExternalString,\n  stringCenter: StringCenter,\n  encodedFeedbackRequest: Option[String])\n    extends BaseFeedbackActionInfoBuilder[PipelineQuery, TweetCandidate] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackActionInfo] = Some(\n    FeedbackActionInfo(\n      feedbackActions = Seq(\n        FeedbackAction(\n          feedbackType = SeeFewer,\n          prompt = Some(\n            Str(seeLessOftenFeedbackString, stringCenter, None)\n              .apply(query, candidate, candidateFeatures)),\n          confirmation = Some(\n            Str(seeLessOftenConfirmationFeedbackString, stringCenter, None)\n              .apply(query, candidate, candidateFeatures)),\n          childFeedbackActions = None,\n          feedbackUrl = None,\n          confirmationDisplayType = None,\n          clientEventInfo = None,\n          richBehavior = None,\n          subprompt = None,\n          icon = Some(Frown),\n          hasUndoAction = Some(true),\n          encodedFeedbackRequest = encodedFeedbackRequest\n        )\n      ),\n      feedbackMetadata = None,\n      displayContext = None,\n      clientEventInfo = None\n    )\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/VideoCarouselModuleCandidateDecorator.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator\n\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeClientEventInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.FeedbackActionInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.FeedbackStrings\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.model.HomeFeatures.VideoDisplayTypeFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemInModuleDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.Str\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.FeatureModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtItemPresentation\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleFooterBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleHeaderBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.Media\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.MediaShort\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Carousel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Classic\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.CompactCarousel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.MediaHighCarousel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleFooter\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleHeader\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stitch.Stitch\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.configapi.Param\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\nobject VideoCarouselModuleCandidateDecorator {\n  val entryNamespaceString = \"video-carousel\"\n  val deepLink = \"https://twitter.com/i/video\"\n}\n\n@Singleton\nclass VideoCarouselModuleCandidateDecorator @Inject() (\n  homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder,\n  @ProductScoped stringCenterProvider: Provider[StringCenter],\n  externalStrings: HomeMixerExternalStrings,\n  feedbackStrings: FeedbackStrings) {\n  import VideoCarouselModuleCandidateDecorator._\n\n  private val stringCenter = stringCenterProvider.get()\n\n  def build(\n    enableVideoCarouselFooter: Param[Boolean]\n  ): UrtItemInModuleDecorator[PipelineQuery, TweetCandidate, Nothing] = {\n    val clientEventInfoBuilder = HomeClientEventInfoBuilder[PipelineQuery, TweetCandidate]()\n    val tweetItemDecorator =\n      VideoItemCandidateDecorator(clientEventInfoBuilder, homeFeedbackActionInfoBuilder)\n    val headerBuilder =\n      VideoCarouselModuleHeaderBuilder(\n        stringCenter,\n        externalStrings,\n        deepLink,\n        enableVideoCarouselFooter)\n\n    val footerBuilder =\n      VideoCarouselModuleFooterBuilder(\n        stringCenter,\n        externalStrings,\n        deepLink,\n        enableVideoCarouselFooter)\n\n    val videoCarouselModuleBuilder = TimelineModuleBuilder(\n      entryNamespace = EntryNamespace(entryNamespaceString),\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      displayTypeBuilder =\n        FeatureModuleDisplayTypeBuilder(VideoDisplayTypeFeature, MediaHighCarousel),\n      headerBuilder = Some(headerBuilder),\n      footerBuilder = Some(footerBuilder),\n      feedbackActionInfoBuilder = Some(\n        FeedbackActionInfoBuilder(\n          seeLessOftenFeedbackString = feedbackStrings.seeLessOftenFeedbackString,\n          seeLessOftenConfirmationFeedbackString = feedbackStrings.seeLessOftenFeedbackString,\n          stringCenter = stringCenter,\n          encodedFeedbackRequest = None\n        )\n      ),\n      showMoreBehaviorBuilder = None\n    )\n\n    UrtItemInModuleDecorator(tweetItemDecorator, videoCarouselModuleBuilder)\n  }\n}\n\ncase class VideoItemCandidateDecorator(\n  clientEventInfoBuilder: HomeClientEventInfoBuilder[PipelineQuery, TweetCandidate],\n  homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder)\n    extends CandidateDecorator[PipelineQuery, TweetCandidate] {\n\n  private val MediaVideoItemBuilder = TweetCandidateUrtItemBuilder[PipelineQuery, TweetCandidate](\n    clientEventInfoBuilder = clientEventInfoBuilder,\n    displayType = Media,\n    feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder)\n  )\n  private val MediaShortVideoItemBuilder =\n    TweetCandidateUrtItemBuilder[PipelineQuery, TweetCandidate](\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      displayType = MediaShort,\n      feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder)\n    )\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[Decoration]] = {\n    val moduleDisplayType = candidates\n      .flatMap { candidate =>\n        candidate.features.getOrElse(VideoDisplayTypeFeature, None)\n      }.headOption.getOrElse(Carousel)\n    // If display type of first video is Carousel, we need a horizontal video module\n    // If display type of first video is CompactCarousel, we need a vertical video module\n    val builder = moduleDisplayType match {\n      case CompactCarousel => MediaShortVideoItemBuilder\n      case _ => MediaVideoItemBuilder\n    }\n    val candidatePresentations = candidates.map { candidate =>\n      val itemPresentation = UrtItemPresentation(\n        timelineItem = builder(query, candidate.candidate, candidate.features)\n      )\n\n      Decoration(candidate.candidate, itemPresentation)\n    }\n\n    Stitch.value(candidatePresentations)\n  }\n}\n\ncase class VideoCarouselModuleHeaderBuilder(\n  stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings,\n  deepLink: String,\n  enableVideoCarouselFooter: Param[Boolean])\n    extends BaseModuleHeaderBuilder[PipelineQuery, TweetCandidate] {\n\n  def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Option[ModuleHeader] = {\n    candidates.headOption.map { candidate =>\n      val title = Str(externalStrings.VideoCarouselHeaderString, stringCenter)\n        .apply(query, candidate.candidate, candidate.features)\n      // If footer is enabled, do not add landing URL so footer can have the deep link\n      val landingUrl = if (!query.params(enableVideoCarouselFooter)) {\n        Some(Url(ExternalUrl, deepLink))\n      } else {\n        None\n      }\n\n      ModuleHeader(\n        text = title,\n        sticky = Some(false),\n        customIcon = None,\n        socialContext = None,\n        icon = None,\n        moduleHeaderDisplayType = Classic,\n        landingUrl = landingUrl\n      )\n    }\n  }\n}\n\ncase class VideoCarouselModuleFooterBuilder(\n  stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings,\n  deepLink: String,\n  enableVideoCarouselFooter: Param[Boolean])\n    extends BaseModuleFooterBuilder[PipelineQuery, TweetCandidate] {\n\n  def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Option[ModuleFooter] = {\n    candidates.headOption.flatMap { candidate =>\n      // Add footer only if footer is enabled\n      if (query.params(enableVideoCarouselFooter)) {\n        val footerText = Str(externalStrings.VideoCarouselFooterString, stringCenter)\n          .apply(query, candidate.candidate, candidate.features)\n        Some(\n          ModuleFooter(\n            text = footerText,\n            landingUrl = Some(Url(ExternalUrl, deepLink))\n          ))\n      } else {\n        None\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/location\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/trends_events\",\n        \"src/scala/com/twitter/suggests/controller_data\",\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate\",\n        \"trends/trending_content/src/main/scala/com/twitter/trends/trending_content/util:compacting-number-localizer\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/HomeAdsClientEventDetailsBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.builder\n\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.suggests.controller_data.home_tweets.v1.{thriftscala => v1ht}\nimport com.twitter.suggests.controller_data.home_tweets.{thriftscala => ht}\nimport com.twitter.suggests.controller_data.thriftscala.ControllerData\nimport com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}\n\ncase class HomeAdsClientEventDetailsBuilder(injectionType: Option[String])\n    extends BaseClientEventDetailsBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: UniversalNoun[Any],\n    candidateFeatures: FeatureMap\n  ): Option[ClientEventDetails] = {\n    val homeTweetsControllerDataV1 = v1ht.HomeTweetsControllerData(\n      tweetTypesBitmap = 0L,\n      traceId = Some(Trace.id.traceId.toLong),\n      requestJoinId = None)\n\n    val serializedControllerData = HomeClientEventDetailsBuilder.ControllerDataSerializer(\n      ControllerData.V2(\n        ControllerDataV2.HomeTweets(ht.HomeTweetsControllerData.V1(homeTweetsControllerDataV1))))\n\n    val clientEventDetails = ClientEventDetails(\n      aiTrendDetails = None,\n      conversationDetails = None,\n      timelinesDetails = Some(\n        TimelinesDetails(\n          injectionType = injectionType,\n          controllerData = Some(serializedControllerData),\n          sourceData = None)),\n      articleDetails = None,\n      liveEventDetails = None,\n      commerceDetails = None,\n    )\n\n    Some(clientEventDetails)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/HomeClientEventDetailsBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.builder\n\nimport com.twitter.bijection.Base64String\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.bijection.{Injection => Serializer}\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.home_mixer.model.HomeFeatures.PositionFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PredictionRequestIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.suggests.controller_data.Home\nimport com.twitter.suggests.controller_data.TweetTypeGenerator\nimport com.twitter.suggests.controller_data.home_tweets.v1.{thriftscala => v1ht}\nimport com.twitter.suggests.controller_data.home_tweets.{thriftscala => ht}\nimport com.twitter.suggests.controller_data.thriftscala.ControllerData\nimport com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}\n\nobject HomeClientEventDetailsBuilder {\n  implicit val ByteSerializer: Serializer[ControllerData, Array[Byte]] =\n    BinaryScalaCodec(ControllerData)\n\n  val ControllerDataSerializer: Serializer[ControllerData, String] =\n    Serializer.connect[ControllerData, Array[Byte], Base64String, String]\n\n  /**\n   * RequestJoinId field in HomeTweetsControllerData is repurposed to pass PredictionRequestId\n   * ReqeustJoinId is no longer used. If wish to switch back, uncomment the below method and\n   * update homeTweetsControllerDataV1.requestJoinId\n   *\n   * define getRequestJoinId as a method(def) rather than a val because each new request\n   * needs to call the context to update the id.\n\n   * private def getRequestJoinId(): Option[Long] =\n   *  RequestJoinKeyContext.current.flatMap(_.requestJoinId)\n   **/\n}\n\ncase class HomeClientEventDetailsBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n) extends BaseClientEventDetailsBuilder[Query, Candidate]\n    with TweetTypeGenerator[FeatureMap] {\n\n  import HomeClientEventDetailsBuilder._\n\n  override def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[ClientEventDetails] = {\n\n    val tweetTypesBitmaps = mkTweetTypesBitmaps(\n      Home.TweetTypeIdxMap,\n      HomeTweetTypePredicates.PredicateMap,\n      candidateFeatures)\n\n    val tweetTypesListBytes = mkItemTypesBitmapsV2(\n      Home.TweetTypeIdxMap,\n      HomeTweetTypePredicates.PredicateMap,\n      candidateFeatures)\n\n    val homeTweetsControllerDataV1 = v1ht.HomeTweetsControllerData(\n      tweetTypesBitmap = tweetTypesBitmaps.getOrElse(0, 0L),\n      tweetTypesBitmapContinued1 = tweetTypesBitmaps.get(1),\n      traceId = Some(Trace.id.traceId.toLong),\n      injectedPosition = candidateFeatures.getOrElse(PositionFeature, None),\n      tweetTypesListBytes = Some(tweetTypesListBytes),\n      // Repurpose requestJoinId field to avoid adding additional payload\n      // Use RequestJoinId to pass PredictionRequestId for model training data join\n      requestJoinId = candidateFeatures.getOrElse(PredictionRequestIdFeature, None),\n      servedId = candidateFeatures.getOrElse(ServedIdFeature, None)\n    )\n\n    val serializedControllerData = ControllerDataSerializer(\n      ControllerData.V2(\n        ControllerDataV2.HomeTweets(ht.HomeTweetsControllerData.V1(homeTweetsControllerDataV1))))\n\n    val clientEventDetails = ClientEventDetails(\n      conversationDetails = None,\n      timelinesDetails = Some(\n        TimelinesDetails(\n          injectionType = Some(candidateFeatures.get(ServedTypeFeature).name),\n          controllerData = Some(serializedControllerData),\n          sourceData = None)),\n      articleDetails = None,\n      liveEventDetails = None,\n      commerceDetails = None,\n      aiTrendDetails = None\n    )\n\n    Some(clientEventDetails)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/HomeClientEventInfoBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.EntityTokenFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Sets the [[ClientEventInfo]] with the `component` field set to the Suggest Type assigned to each candidate\n */\ncase class HomeClientEventInfoBuilder[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n  detailsBuilder: Option[BaseClientEventDetailsBuilder[Query, Candidate]] = None)\n    extends BaseClientEventInfoBuilder[Query, Candidate] {\n\n  override def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap,\n    element: Option[String]\n  ): Option[ClientEventInfo] = {\n    val servedType = candidateFeatures.get(ServedTypeFeature)\n\n    Some(\n      ClientEventInfo(\n        component = Some(servedType.originalName),\n        element = element,\n        details = detailsBuilder.flatMap(_.apply(query, candidate, candidateFeatures)),\n        action = None,\n        /**\n         * A backend entity encoded by the Client Entities Encoding Library.\n         * Placeholder string for now\n         */\n        entityToken = candidateFeatures.getOrElse(EntityTokenFeature, None)\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/HomeConversationModuleMetadataBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleMetadataBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleConversationMetadata\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleMetadata\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class HomeConversationModuleMetadataBuilder[\n  -Query <: PipelineQuery,\n  -Candidate <: BaseTweetCandidate\n]() extends BaseModuleMetadataBuilder[Query, Candidate] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): ModuleMetadata = ModuleMetadata(\n    adsMetadata = None,\n    conversationMetadata = Some(\n      ModuleConversationMetadata(\n        allTweetIds = Some((candidates.last.candidate.id +:\n          candidates.last.features.getOrElse(AncestorsFeature, Seq.empty).map(_.tweetId)).reverse),\n        socialContext = None,\n        enableDeduplication = Some(true)\n      )),\n    gridCarouselMetadata = None,\n    pillGroupMetadata = None\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/HomeTimelinesScoreInfoBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableSendScoresToClient\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.tweet.BaseTimelinesScoreInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TimelinesScoreInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject HomeTimelinesScoreInfoBuilder\n    extends BaseTimelinesScoreInfoBuilder[PipelineQuery, TweetCandidate] {\n\n  private val UndefinedTweetScore = -1.0\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[TimelinesScoreInfo] = {\n    if (query.params(EnableSendScoresToClient)) {\n      val score = candidateFeatures.getOrElse(ScoreFeature, None).getOrElse(UndefinedTweetScore)\n      Some(TimelinesScoreInfo(score))\n    } else None\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/HomeTweetTypePredicates.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.builder\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationIdFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BasicTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType\nimport com.twitter.timelinemixer.injection.model.candidate.SemanticCoreFeatures\nimport com.twitter.tweetypie.{thriftscala => tpt}\n\nobject HomeTweetTypePredicates {\n\n  /**\n   * The predicates defined in this file are used purely for metrics tracking purposes to \n   * measure how often we serve posts with various attributes.\n   */\n  private val CandidatePredicates: Seq[(String, FeatureMap => Boolean)] = Seq(\n    (\"with_candidate\", _ => true),\n    (\"retweet\", _.getOrElse(IsRetweetFeature, false)),\n    (\"reply\", _.getOrElse(InReplyToTweetIdFeature, None).nonEmpty),\n    (\"image\", _.getOrElse(HasImageFeature, false)),\n    (\"video\", _.getOrElse(HasVideoFeature, false)),\n    (\"link\", _.getOrElse(EarlybirdFeature, None).exists(_.hasVisibleLink)),\n    (\"quote\", _.getOrElse(EarlybirdFeature, None).exists(_.hasQuote.contains(true))),\n    (\"like_social_context\", _.getOrElse(NonSelfFavoritedByUserIdsFeature, Seq.empty).nonEmpty),\n    (\"protected\", _.getOrElse(AuthorIsProtectedFeature, false)),\n    (\n      \"has_exclusive_conversation_author_id\",\n      _.getOrElse(ExclusiveConversationAuthorIdFeature, None).nonEmpty),\n    (\"is_eligible_for_connect_boost\", _ => false),\n    (\"hashtag\", _.getOrElse(EarlybirdFeature, None).exists(_.numHashtags > 0)),\n    (\"has_scheduled_space\", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isScheduled)),\n    (\"has_recorded_space\", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isRecorded)),\n    (\"is_read_from_cache\", _.getOrElse(IsReadFromCacheFeature, false)),\n    (\"get_initial\", _.getOrElse(GetInitialFeature, false)),\n    (\"get_newer\", _.getOrElse(GetNewerFeature, false)),\n    (\"get_middle\", _.getOrElse(GetMiddleFeature, false)),\n    (\"get_older\", _.getOrElse(GetOlderFeature, false)),\n    (\"pull_to_refresh\", _.getOrElse(PullToRefreshFeature, false)),\n    (\"polling\", _.getOrElse(PollingFeature, false)),\n    (\"near_empty\", _.getOrElse(ServedSizeFeature, None).exists(_ < 3)),\n    (\"is_request_context_launch\", _.getOrElse(IsLaunchRequestFeature, false)),\n    (\"mutual_follow\", _.getOrElse(EarlybirdFeature, None).exists(_.fromMutualFollow)),\n    (\n      \"less_than_10_mins_since_lnpt\",\n      _.getOrElse(LastNonPollingTimeFeature, None).exists(_.untilNow < 10.minutes)),\n    (\"served_in_conversation_module\", _.getOrElse(ServedInConversationModuleFeature, false)),\n    (\"has_ticketed_space\", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.hasTickets)),\n    (\"in_utis_top5\", _.getOrElse(PositionFeature, None).exists(_ < 5)),\n    (\n      \"conversation_module_has_2_displayed_tweets\",\n      _.getOrElse(ConversationModule2DisplayedTweetsFeature, false)),\n    (\"empty_request\", _.getOrElse(ServedSizeFeature, None).exists(_ == 0)),\n    (\"served_size_less_than_50\", _.getOrElse(ServedSizeFeature, None).exists(_ < 50)),\n    (\n      \"served_size_between_50_and_100\",\n      _.getOrElse(ServedSizeFeature, None).exists(size => size >= 50 && size < 100)),\n    (\"authored_by_contextual_user\", _.getOrElse(AuthoredByContextualUserFeature, false)),\n    (\"is_self_thread_tweet\", _.getOrElse(IsSelfThreadFeature, false)),\n    (\"has_ancestors\", _.getOrElse(AncestorsFeature, Seq.empty).nonEmpty),\n    (\"full_scoring_succeeded\", _.getOrElse(FullScoringSucceededFeature, false)),\n    (\"served_size_less_than_20\", _.getOrElse(ServedSizeFeature, None).exists(_ < 20)),\n    (\"served_size_less_than_10\", _.getOrElse(ServedSizeFeature, None).exists(_ < 10)),\n    (\"served_size_less_than_5\", _.getOrElse(ServedSizeFeature, None).exists(_ < 5)),\n    (\n      \"account_age_less_than_30_minutes\",\n      _.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 30.minutes)),\n    (\"conversation_module_has_gap\", _.getOrElse(ConversationModuleHasGapFeature, false)),\n    (\n      \"directed_at_user_is_in_first_degree\",\n      _.getOrElse(EarlybirdFeature, None).exists(_.directedAtUserIdIsInFirstDegree.contains(true))),\n    (\n      \"has_semantic_core_annotation\",\n      _.getOrElse(EarlybirdFeature, None).exists(_.semanticCoreAnnotations.nonEmpty)),\n    (\"is_request_context_foreground\", _.getOrElse(IsForegroundRequestFeature, false)),\n    (\n      \"account_age_less_than_1_day\",\n      _.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 1.day)),\n    (\n      \"account_age_less_than_7_days\",\n      _.getOrElse(AccountAgeFeature, None).exists(_.untilNow < 7.days)),\n    (\n      \"part_of_utt\",\n      _.getOrElse(EarlybirdFeature, None)\n        .exists(_.semanticCoreAnnotations.exists(_.exists(annotation =>\n          annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy)))),\n    (\n      \"has_home_latest_request_past_week\",\n      _.getOrElse(FollowingLastNonPollingTimeFeature, None).exists(_.untilNow < 7.days)),\n    (\"is_utis_pos0\", _.getOrElse(PositionFeature, None).exists(_ == 0)),\n    (\"is_utis_pos1\", _.getOrElse(PositionFeature, None).exists(_ == 1)),\n    (\"is_utis_pos2\", _.getOrElse(PositionFeature, None).exists(_ == 2)),\n    (\"is_utis_pos3\", _.getOrElse(PositionFeature, None).exists(_ == 3)),\n    (\"is_utis_pos4\", _.getOrElse(PositionFeature, None).exists(_ == 4)),\n    (\"is_random_tweet\", _ => false),\n    (\"has_random_tweet_in_response\", _ => false),\n    (\"is_random_tweet_above_in_utis\", _ => false),\n    (\n      \"has_ancestor_authored_by_viewer\",\n      candidate =>\n        candidate\n          .getOrElse(AncestorsFeature, Seq.empty).exists(ancestor =>\n            candidate.getOrElse(ViewerIdFeature, 0L) == ancestor.userId)),\n    (\"ancestor\", _.getOrElse(IsAncestorCandidateFeature, false)),\n    (\n      \"deep_reply\",\n      candidate =>\n        candidate.getOrElse(InReplyToTweetIdFeature, None).nonEmpty && candidate\n          .getOrElse(AncestorsFeature, Seq.empty).size > 2),\n    (\n      \"has_simcluster_embeddings\",\n      _.getOrElse(\n        SimclustersTweetTopKClustersWithScoresFeature,\n        Map.empty[String, Double]).nonEmpty),\n    (\n      \"tweet_age_less_than_15_seconds\",\n      _.getOrElse(TweetAgeFeature, None)\n        .exists(_ <= 15.seconds.inMillis)),\n    (\n      \"less_than_1_hour_since_lnpt\",\n      _.getOrElse(LastNonPollingTimeFeature, None).exists(_.untilNow < 1.hour)),\n    (\"has_gte_10_favs\", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 10))),\n    (\n      \"device_language_matches_tweet_language\",\n      candidate =>\n        candidate.getOrElse(TweetLanguageFeature, None) ==\n          candidate.getOrElse(DeviceLanguageFeature, None)),\n    (\n      \"root_ancestor\",\n      candidate =>\n        candidate.getOrElse(IsAncestorCandidateFeature, false) && candidate\n          .getOrElse(InReplyToTweetIdFeature, None).isEmpty),\n    (\"question\", _.getOrElse(EarlybirdFeature, None).exists(_.hasQuestion.contains(true))),\n    (\"in_network\", _.getOrElse(InNetworkFeature, true)),\n    (\n      \"has_political_annotation\",\n      _.getOrElse(EarlybirdFeature, None).exists(\n        _.semanticCoreAnnotations.exists(\n          _.exists(annotation =>\n            SemanticCoreFeatures.PoliticalDomains.contains(annotation.domainId) ||\n              (annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy &&\n                annotation.entityId == SemanticCoreFeatures.UttPoliticsEntityId))))),\n    (\n      \"is_dont_at_me_by_invitation\",\n      _.getOrElse(EarlybirdFeature, None).exists(\n        _.conversationControl.exists(_.isInstanceOf[tpt.ConversationControl.ByInvitation]))),\n    (\n      \"is_dont_at_me_community\",\n      _.getOrElse(EarlybirdFeature, None)\n        .exists(_.conversationControl.exists(_.isInstanceOf[tpt.ConversationControl.Community]))),\n    (\"has_zero_score\", _.getOrElse(ScoreFeature, None).exists(_ == 0.0)),\n    (\n      \"is_followed_topic_tweet\",\n      _.getOrElse(TopicContextFunctionalityTypeFeature, None)\n        .exists(_ == BasicTopicContextFunctionalityType)),\n    (\n      \"is_recommended_topic_tweet\",\n      _.getOrElse(TopicContextFunctionalityTypeFeature, None)\n        .exists(_ == RecommendationTopicContextFunctionalityType)),\n    (\"has_gte_100_favs\", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 100))),\n    (\"has_gte_1k_favs\", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 1000))),\n    (\n      \"has_gte_10k_favs\",\n      _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 10000))),\n    (\n      \"has_gte_100k_favs\",\n      _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 100000))),\n    (\"has_audio_space\", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.hasSpace)),\n    (\"has_live_audio_space\", _.getOrElse(AudioSpaceMetaDataFeature, None).exists(_.isLive)),\n    (\n      \"has_gte_10_retweets\",\n      _.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 10))),\n    (\n      \"has_gte_100_retweets\",\n      _.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 100))),\n    (\n      \"has_gte_1k_retweets\",\n      _.getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= 1000))),\n    (\n      \"has_us_political_annotation\",\n      _.getOrElse(EarlybirdFeature, None)\n        .exists(_.semanticCoreAnnotations.exists(_.exists(annotation =>\n          annotation.domainId == SemanticCoreFeatures.UnifiedTwitterTaxonomy &&\n            annotation.entityId == SemanticCoreFeatures.usPoliticalTweetEntityId &&\n            annotation.groupId == SemanticCoreFeatures.UsPoliticalTweetAnnotationGroupIds.BalancedV0)))),\n    (\n      \"has_toxicity_score_above_threshold\",\n      _.getOrElse(EarlybirdFeature, None).exists(_.toxicityScore.exists(_ > 0.91))),\n    (\"is_topic_tweet\", _.getOrElse(TopicIdSocialContextFeature, None).isDefined),\n    (\n      \"text_only\",\n      candidate =>\n        candidate.getOrElse(HasDisplayedTextFeature, false) &&\n          !(candidate.getOrElse(HasImageFeature, false) ||\n            candidate.getOrElse(HasVideoFeature, false) ||\n            candidate.getOrElse(EarlybirdFeature, None).exists(_.hasCard))),\n    (\n      \"image_only\",\n      candidate =>\n        candidate.getOrElse(HasImageFeature, false) &&\n          !candidate.getOrElse(HasDisplayedTextFeature, false)),\n    (\"has_1_image\", _.getOrElse(NumImagesFeature, None).exists(_ == 1)),\n    (\"has_2_images\", _.getOrElse(NumImagesFeature, None).exists(_ == 2)),\n    (\"has_3_images\", _.getOrElse(NumImagesFeature, None).exists(_ == 3)),\n    (\"has_4_images\", _.getOrElse(NumImagesFeature, None).exists(_ == 4)),\n    (\"has_card\", _.getOrElse(EarlybirdFeature, None).exists(_.hasCard)),\n    (\"user_follow_count_gte_50\", _.getOrElse(UserFollowingCountFeature, None).exists(_ > 50)),\n    (\n      \"has_liked_by_social_context\",\n      candidateFeatures =>\n        candidateFeatures\n          .getOrElse(ValidLikedByUserIdsFeature, Seq.empty).nonEmpty),\n    (\n      \"has_followed_by_social_context\",\n      _.getOrElse(SGSValidFollowedByUserIdsFeature, Seq.empty).nonEmpty),\n    (\n      \"has_topic_social_context\",\n      candidateFeatures =>\n        candidateFeatures\n          .getOrElse(TopicIdSocialContextFeature, None)\n          .isDefined &&\n          candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None).isDefined),\n    (\"video_lte_10_sec\", _.getOrElse(VideoDurationMsFeature, None).exists(_ <= 10000)),\n    (\n      \"video_bt_10_60_sec\",\n      _.getOrElse(VideoDurationMsFeature, None).exists(duration =>\n        duration > 10000 && duration <= 60000)),\n    (\"video_gt_60_sec\", _.getOrElse(VideoDurationMsFeature, None).exists(_ > 60000)),\n    (\n      \"tweet_age_lte_30_minutes\",\n      _.getOrElse(TweetAgeFeature, None)\n        .exists(_ <= 30.minutes.inMillis)),\n    (\n      \"tweet_age_lte_1_hour\",\n      _.getOrElse(TweetAgeFeature, None)\n        .exists(_ <= 1.hour.inMillis)),\n    (\n      \"tweet_age_lte_6_hours\",\n      _.getOrElse(TweetAgeFeature, None)\n        .exists(_ <= 6.hours.inMillis)),\n    (\n      \"tweet_age_lte_12_hours\",\n      _.getOrElse(TweetAgeFeature, None)\n        .exists(_ <= 12.hours.inMillis)),\n    (\n      \"tweet_age_gte_24_hours\",\n      _.getOrElse(TweetAgeFeature, None)\n        .exists(_ >= 24.hours.inMillis)),\n    (\"author_is_blue_verified\", _.getOrElse(AuthorIsBlueVerifiedFeature, false)),\n    (\"author_is_gold_verified\", _.getOrElse(AuthorIsGoldVerifiedFeature, false)),\n    (\"author_is_gray_verified\", _.getOrElse(AuthorIsGrayVerifiedFeature, false)),\n    (\"author_is_legacy_verified\", _.getOrElse(AuthorIsLegacyVerifiedFeature, false)),\n    (\"author_is_creator\", _.getOrElse(AuthorIsCreatorFeature, false)),\n    (\n      \"viral_content_creator_in_network\",\n      candidate =>\n        candidate.getOrElse(ViralContentCreatorFeature, false) &&\n          candidate.getOrElse(InNetworkFeature, true)),\n    (\n      \"viral_content_creator_out_of_network\",\n      candidate =>\n        candidate.getOrElse(ViralContentCreatorFeature, false) &&\n          !candidate.getOrElse(InNetworkFeature, true)),\n    (\n      \"grok_content_creator_in_network\",\n      candidate =>\n        candidate.getOrElse(GrokContentCreatorFeature, false) &&\n          candidate.getOrElse(InNetworkFeature, true)),\n    (\n      \"grok_content_creator_out_of_network\",\n      candidate =>\n        candidate.getOrElse(GrokContentCreatorFeature, false) &&\n          !candidate.getOrElse(InNetworkFeature, true)),\n    (\n      \"gork_content_creator_in_network\",\n      candidate =>\n        candidate.getOrElse(GorkContentCreatorFeature, false) &&\n          candidate.getOrElse(InNetworkFeature, true)),\n    (\n      \"gork_content_creator_out_of_network\",\n      candidate =>\n        candidate.getOrElse(GorkContentCreatorFeature, false) &&\n          !candidate.getOrElse(InNetworkFeature, true)),\n    (\"has_location\", _.getOrElse(LocationIdFeature, None).isDefined),\n    (\"article\", _.getOrElse(IsArticleFeature, false)),\n    (\n      \"grok_category_news\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_sports\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\"grok_category_entertainment\", _ => false),\n    (\n      \"grok_category_business_&_finance\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_technology\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_gaming\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_movies_&_tv\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_music\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_travel\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_food\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_fashion\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_health_&_fitness\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_anime\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_celebrity\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_cryptocurrency\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_science\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_memes\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\"grok_category_art\", _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_religion\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_shopping\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_cars\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_aviation\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_motorcycles\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_beauty\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_nature_&_outdoors\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_pets\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_relationships\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_home_&_garden\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_career\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_dance\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_education\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_podcasts\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\n      \"grok_category_streaming\",\n      _.getOrElse(GrokTopCategoryFeature, None).contains(<removed_id>)),\n    (\"grok_is_gore\", _.getOrElse(GrokIsGoreFeature, None).getOrElse(false)),\n    (\"grok_is_nsfw\", _.getOrElse(GrokIsNsfwFeature, None).getOrElse(false)),\n    (\"grok_is_spam\", _.getOrElse(GrokIsSpamFeature, None).getOrElse(false)),\n    (\"grok_is_violent\", _.getOrElse(GrokIsViolentFeature, None).getOrElse(false)),\n    (\"grok_is_low_quality\", _.getOrElse(GrokIsLowQualityFeature, None).getOrElse(false)),\n    (\"grok_is_ocr\", _.getOrElse(GrokIsOcrFeature, None).getOrElse(false)),    \n    (\n      \"grok_politics_neutral\", // Purely for metrics tracking. Does not affect the recommendations.\n      _.getOrElse(GrokPoliticalInclinationFeature, None).contains(hmt.PoliticalInclination.Neutral)\n    ),\n    (\n      \"grok_politics_left\", // Purely for metrics tracking. Does not affect the recommendations.\n      _.getOrElse(GrokPoliticalInclinationFeature, None).contains(hmt.PoliticalInclination.Left)\n    ),\n    (\n      \"grok_politics_right\", // Purely for metrics tracking. Does not affect the recommendations.\n      _.getOrElse(GrokPoliticalInclinationFeature, None).contains(hmt.PoliticalInclination.Right)\n    ),\n    (\"is_slop_lte_0\", _.getOrElse(SlopAuthorScoreFeature, None).exists(_ <= 0.0)),\n    (\"is_slop_lte_0_2\", _.getOrElse(SlopAuthorScoreFeature, None).exists(_ <= 0.2)),\n    (\"is_slop_gt_0\", _.getOrElse(SlopAuthorScoreFeature, None).exists(_ > 0.0)),\n    (\"is_slop_gt_0_2\", _.getOrElse(SlopAuthorScoreFeature, None).exists(_ > 0.2)),\n    (\"is_slop_gt_0_4\", _.getOrElse(SlopAuthorScoreFeature, None).exists(_ > 0.4)),\n    (\"is_slop_gt_0_6\", _.getOrElse(SlopAuthorScoreFeature, None).exists(_ > 0.6)),\n    (\n      \"unique_author_ratio_lte_50_pct\",\n      features => {\n        val uniqueAuthorCount = features.getOrElse(UniqueAuthorCountFeature, None).getOrElse(0)\n        val servedSize = features.getOrElse(ServedSizeFeature, None).getOrElse(1)\n        uniqueAuthorCount.toDouble / servedSize <= 0.5\n      }\n    ),\n    (\"unique_author_lte_5\", _.getOrElse(UniqueAuthorCountFeature, None).exists(_ <= 5)),\n    (\"unique_author_lte_10\", _.getOrElse(UniqueAuthorCountFeature, None).exists(_ <= 10)),\n    (\"unique_author_lte_15\", _.getOrElse(UniqueAuthorCountFeature, None).exists(_ <= 15)),\n    (\n      \"single_author_gte_25_pct\",\n      features => {\n        val maxCount = features.getOrElse(MaxSingleAuthorCountFeature, None).getOrElse(0)\n        val servedSize = features.getOrElse(ServedSizeFeature, None).getOrElse(0)\n        servedSize > 0 && maxCount.toDouble / servedSize >= 0.25\n      }),\n    (\n      \"single_author_gte_50_pct\",\n      features => {\n        val maxCount = features.getOrElse(MaxSingleAuthorCountFeature, None).getOrElse(0)\n        val servedSize = features.getOrElse(ServedSizeFeature, None).getOrElse(0)\n        servedSize > 0 && maxCount.toDouble / servedSize >= 0.5\n      }),\n    (\n      \"unique_category_ratio_lte_50_pct\",\n      features => {\n        val uniqueAuthorCount = features.getOrElse(UniqueCategoryCountFeature, None).getOrElse(0)\n        val servedSize = features.getOrElse(ServedSizeFeature, None).getOrElse(1)\n        uniqueAuthorCount.toDouble / servedSize <= 0.5\n      }\n    ),\n    (\"unique_category_lte_5\", _.getOrElse(UniqueCategoryCountFeature, None).exists(_ <= 5)),\n    (\"unique_category_lte_10\", _.getOrElse(UniqueCategoryCountFeature, None).exists(_ <= 10)),\n    (\"unique_category_lte_15\", _.getOrElse(UniqueCategoryCountFeature, None).exists(_ <= 15)),\n    (\n      \"single_category_gte_25_pct\",\n      features => {\n        val maxCount = features.getOrElse(MaxSingleCategoryCountFeature, None).getOrElse(0)\n        val servedSize = features.getOrElse(ServedSizeFeature, None).getOrElse(0)\n        servedSize > 0 && maxCount.toDouble / servedSize >= 0.25\n      }),\n    (\n      \"single_category_gte_50_pct\",\n      features => {\n        val maxCount = features.getOrElse(MaxSingleCategoryCountFeature, None).getOrElse(0)\n        val servedSize = features.getOrElse(ServedSizeFeature, None).getOrElse(0)\n        servedSize > 0 && maxCount.toDouble / servedSize >= 0.5\n      }),\n    (\"is_grokslopscore_low_1\", _.getOrElse(GrokSlopScoreFeature, None).contains(1L)),\n    (\"is_grokslopscore_med_2\", _.getOrElse(GrokSlopScoreFeature, None).contains(2L)),\n    (\"is_grokslopscore_high_3\", _.getOrElse(GrokSlopScoreFeature, None).contains(3L)),\n    (\"is_boosted\", _.getOrElse(IsBoostedCandidateFeature, false)),\n    (\n      \"has_source_signal_tweet_favorite\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"TweetFavorite\"))),\n    (\n      \"has_source_signal_retweet\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"Retweet\"))),\n    (\n      \"has_source_signal_reply\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"Reply\"))),\n    (\n      \"has_source_signal_tweet_bookmark_v1\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"TweetBookmarkV1\"))),\n    (\n      \"has_source_signal_original_tweet\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"OriginalTweet\"))),\n    (\n      \"has_source_signal_account_follow\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"AccountFollow\"))),\n    (\n      \"has_source_signal_tweet_share_v1\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"TweetShareV1\"))),\n    (\n      \"has_source_signal_tweet_photo_expand\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"TweetPhotoExpand\"))),\n    (\n      \"has_source_signal_search_tweet_click\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"SearchTweetClick\"))),\n    (\n      \"has_source_signal_profile_tweet_click\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"ProfileTweetClick\"))),\n    (\n      \"has_source_signal_tweet_video_open\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"TweetVideoOpen\"))),\n    (\n      \"has_source_signal_video_view_90d_quality_v1_all_surfaces\",\n      _.getOrElse(SourceSignalFeature, None)\n        .exists(_.signalType.contains(\"VideoView90dQualityV1AllSurfaces\"))),\n    (\n      \"has_source_signal_video_view_90d_quality_v2\",\n      _.getOrElse(SourceSignalFeature, None)\n        .exists(_.signalType.contains(\"VideoView90dQualityV2\"))),\n    (\n      \"has_source_signal_video_view_90d_quality_v2_visibility_75\",\n      _.getOrElse(SourceSignalFeature, None)\n        .exists(_.signalType.contains(\"VideoView90dQualityV2Visibility75\"))),\n    (\n      \"has_source_signal_video_view_90d_quality_v2_visibility_100\",\n      _.getOrElse(SourceSignalFeature, None)\n        .exists(_.signalType.contains(\"VideoView90dQualityV2Visibility100\"))),\n    (\n      \"has_source_signal_video_view_90d_quality_v3\",\n      _.getOrElse(SourceSignalFeature, None)\n        .exists(_.signalType.contains(\"VideoView90dQualityV3\"))),\n    (\n      \"has_source_signal_account_block\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"AccountBlock\"))),\n    (\n      \"has_source_signal_account_mute\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"AccountMute\"))),\n    (\n      \"has_source_signal_tweet_report\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"TweetReport\"))),\n    (\n      \"has_source_signal_tweet_dont_like\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"TweetDontLike\"))),\n    (\n      \"has_source_signal_tweet_report_v2\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"TweetReportV2\"))),\n    (\n      \"has_source_signal_tweet_dont_like_v2\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"TweetDontLikeV2\"))),\n    (\n      \"has_source_signal_notification_open_and_click_v1\",\n      _.getOrElse(SourceSignalFeature, None)\n        .exists(_.signalType.contains(\"NotificationOpenAndClickV1\"))),\n    (\n      \"has_source_signal_feedback_notrelevant\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"FeedbackNotrelevant\"))),\n    (\n      \"has_source_signal_feedback_relevant\",\n      _.getOrElse(SourceSignalFeature, None).exists(_.signalType.contains(\"FeedbackRelevant\"))),\n    (\n      \"has_source_signal_high_quality_source_tweet\",\n      _.getOrElse(SourceSignalFeature, None)\n        .exists(_.signalType.contains(\"HighQualitySourceTweet\"))),\n    (\n      \"has_source_signal_high_quality_source_user\",\n      _.getOrElse(SourceSignalFeature, None)\n        .exists(_.signalType.contains(\"HighQualitySourceUser\"))),\n  )\n\n  val PredicateMap = CandidatePredicates.toMap\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/KeywordTrendMetaDescriptionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.builder\n\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.trend.BaseTrendMetaDescriptionBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.Str\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendTweetCountFeature\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.UnifiedTrendCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.trends.trending_content.util.CompactingNumberLocalizer\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass KeywordTrendMetaDescriptionBuilder @Inject() (\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter])\n    extends BaseTrendMetaDescriptionBuilder[PipelineQuery, UnifiedTrendCandidate] {\n\n  private val stringCenter = stringCenterProvider.get()\n\n  private val tweetCountStr = Str(\n    text = externalStrings.KeywordTrendsTweetCountDescriptionString,\n    stringCenter = stringCenter\n  )\n\n  private val compactingNumberLocalizer = new CompactingNumberLocalizer()\n\n  def apply(\n    query: PipelineQuery,\n    candidate: UnifiedTrendCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[String] = {\n    // E.g. \"23.4K posts\"\n    candidateFeatures.getOrElse(TrendTweetCountFeature, None).map { tweetCount =>\n      val compactedTweetCount = compactingNumberLocalizer.localizeAndCompact(\n        query.getLanguageCode\n          .getOrElse(\"en\"),\n        tweetCount)\n      tweetCountStr(query, candidate, candidateFeatures).format(compactedTweetCount)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/ListClientEventDetailsBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.builder\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\n\ncase class ListClientEventDetailsBuilder(suggestType: st.SuggestType)\n    extends BaseClientEventDetailsBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: UniversalNoun[Any],\n    candidateFeatures: FeatureMap\n  ): Option[ClientEventDetails] = {\n    val clientEventDetails = ClientEventDetails(\n      conversationDetails = None,\n      timelinesDetails = Some(\n        TimelinesDetails(\n          injectionType = Some(suggestType.name),\n          controllerData = None,\n          sourceData = None)),\n      articleDetails = None,\n      liveEventDetails = None,\n      commerceDetails = None\n    )\n\n    Some(clientEventDetails)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder/VerifiedPromptBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.builder\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.message.InlinePromptCandidateUrtItemStringCenterBuilder.InlinePromptClientEventInfoElement\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.message.MessageTextActionBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.Str\nimport com.twitter.product_mixer.component_library.model.candidate.InlinePromptCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.InlinePromptMessageContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessagePromptItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stringcenter.client.ExternalStringRegistry\nimport com.twitter.stringcenter.client.StringCenter\n\ncase class VerifiedPromptBuilder(\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[PipelineQuery, InlinePromptCandidate],\n  stringCenter: StringCenter,\n  externalStringRegistry: ExternalStringRegistry)\n    extends CandidateUrtEntryBuilder[\n      PipelineQuery,\n      InlinePromptCandidate,\n      MessagePromptItem\n    ] {\n  private val VerifiedUrl = \"\"\n\n  private val headerExternalStr = externalStringRegistry.createProdString(\"verified_prompt_header\")\n  private val bodyExternalStr = externalStringRegistry.createProdString(\"verified_prompt_body\")\n  private val buttonExternalStr =\n    externalStringRegistry.createProdString(\"creator_subscriptions_teaser_button\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: InlinePromptCandidate,\n    candidateFeatures: FeatureMap\n  ): MessagePromptItem = {\n    val headerStr = Str(headerExternalStr, stringCenter).apply(query, candidate, candidateFeatures)\n    val bodyStr = Str(bodyExternalStr, stringCenter).apply(query, candidate, candidateFeatures)\n    val buttonStrBuilder = Str(buttonExternalStr, stringCenter)\n\n    val button = MessageTextActionBuilder(\n      textBuilder = buttonStrBuilder,\n      dismissOnClick = false,\n      url = Some(VerifiedUrl)\n    ).apply(query, candidate, candidateFeatures)\n\n    MessagePromptItem(\n      id = candidate.id,\n      sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n      clientEventInfo = clientEventInfoBuilder(\n        query = query,\n        candidate = candidate,\n        candidateFeatures = candidateFeatures,\n        element = Some(InlinePromptClientEventInfoElement)\n      ),\n      isPinned = None,\n      content = InlinePromptMessageContent(\n        headerText = headerStr,\n        bodyText = Some(bodyStr),\n        primaryButtonAction = Some(button),\n        secondaryButtonAction = None,\n        headerRichText = None,\n        bodyRichText = None,\n        socialContext = None,\n        userFacepile = None\n      ),\n      impressionCallbacks = None,\n      feedbackActionInfo = None\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/AddEntriesWithReplaceAndShowAlertAndShowCoverInstructionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.AlwaysInclude\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.IncludeInstruction\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtInstructionBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Cover\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ShowAlert\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, AddEntriesTimelineInstruction] {\n\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[AddEntriesTimelineInstruction] = {\n    if (includeInstruction(query, entries)) {\n      val entriesToAdd = entries\n        .filterNot(_.isInstanceOf[ShowAlert])\n        .filterNot(_.isInstanceOf[Cover])\n        .filter(_.entryIdToReplace.isEmpty)\n      if (entriesToAdd.nonEmpty) Seq(AddEntriesTimelineInstruction(entriesToAdd))\n      else Seq.empty\n    } else\n      Seq.empty\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/AuthorChildFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.service.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class AuthorChildFeedbackActionBuilder @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings) {\n\n  def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {\n    CandidatesUtil.getOriginalAuthorId(candidateFeatures).flatMap { authorId =>\n      FeedbackUtil.buildUserSeeFewerChildFeedbackAction(\n        userId = authorId,\n        namesByUserId = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String]),\n        promptExternalString = externalStrings.showFewerTweetsString,\n        confirmationExternalString = externalStrings.showFewerTweetsConfirmationString,\n        engagementType = t.FeedbackEngagementType.Tweet,\n        stringCenter = stringCenter\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder\",\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/BlockUserChildFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BottomSheet\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorBlockUser\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class BlockUserChildFeedbackActionBuilder @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings) {\n\n  def apply(\n    candidateFeatures: FeatureMap\n  ): Option[ChildFeedbackAction] = {\n    val userIdOpt =\n      if (candidateFeatures.getOrElse(IsRetweetFeature, false))\n        candidateFeatures.getOrElse(SourceUserIdFeature, None)\n      else candidateFeatures.getOrElse(AuthorIdFeature, None)\n\n    userIdOpt.flatMap { userId =>\n      val screenNamesMap = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])\n      val userScreenNameOpt = screenNamesMap.get(userId)\n      userScreenNameOpt.map { userScreenName =>\n        val prompt = stringCenter.prepare(\n          externalStrings.blockUserString,\n          Map(\"username\" -> userScreenName)\n        )\n        val confirmation = stringCenter.prepare(\n          externalStrings.blockUserConfirmationString,\n          Map(\"username\" -> userScreenName)\n        )\n\n        ChildFeedbackAction(\n          feedbackType = RichBehavior,\n          prompt = Some(prompt),\n          confirmation = Some(confirmation),\n          subprompt = None,\n          feedbackUrl = None,\n          hasUndoAction = Some(true),\n          confirmationDisplayType = Some(BottomSheet),\n          clientEventInfo = Some(\n            ClientEventInfo(\n              component = None,\n              element = Some(\"block\"),\n              details = None,\n              action = Some(\"click\"),\n              entityToken = None\n            )\n          ),\n          icon = Some(icon.No),\n          richBehavior = Some(RichFeedbackBehaviorBlockUser(userId))\n        )\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/ChildFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.param.HomeGlobalParams.PostFeedbackPromptNegativeParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.PostFeedbackPromptNeutralParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.PostFeedbackPromptPositiveParam\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.stringcenter.client.core.ExternalString\nimport com.twitter.timelines.common.{thriftscala => tlc}\nimport com.twitter.timelineservice.model.FeedbackInfo\nimport com.twitter.timelineservice.model.FeedbackMetadata\nimport com.twitter.timelineservice.{thriftscala => tlst}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ntrait ChildFeedbackActionBuilder {\n  val stringCenter: StringCenter\n  def feedbackType: tlst.FeedbackType\n  def internalFeedbackType: FeedbackType\n  def promptString: ExternalString\n  def confirmationString: ExternalString\n  def clientEventElement: String\n  def clientEventAction: String\n  def clientEventComponent: Option[String] = None\n  def hasUndoAction: Boolean = true\n  def confirmationDisplayType: Option[ConfirmationDisplayType] = None\n  def getPrompt(query: PipelineQuery): String = {\n    stringCenter.prepare(promptString)\n  }\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap,\n  ): Option[ChildFeedbackAction] = {\n    val prompt = getPrompt(query)\n    val confirmation = stringCenter.prepare(confirmationString)\n    val feedbackMetadata = FeedbackMetadata(\n      engagementType = None,\n      entityIds = Seq(tlc.FeedbackEntity.TweetId(candidate.id)),\n      ttl = Some(FeedbackUtil.FeedbackTtl)\n    )\n    val feedbackUrl = FeedbackInfo.feedbackUrl(\n      feedbackType = feedbackType,\n      feedbackMetadata = feedbackMetadata,\n      injectionType = None\n    )\n\n    Some(\n      ChildFeedbackAction(\n        feedbackType = internalFeedbackType,\n        prompt = Some(prompt),\n        confirmation = Some(confirmation),\n        feedbackUrl = Some(feedbackUrl),\n        hasUndoAction = Some(hasUndoAction),\n        confirmationDisplayType = confirmationDisplayType,\n        clientEventInfo = Some(\n          ClientEventInfo(\n            component = clientEventComponent,\n            element = Some(clientEventElement),\n            details = None,\n            action = Some(clientEventAction),\n            entityToken = None\n          )\n        ),\n        icon = None,\n        richBehavior = None,\n        subprompt = None\n      )\n    )\n  }\n}\n\n@Singleton\ncase class SeeMoreChildFeedbackActionBuilder @Inject() (\n  @ProductScoped override val stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends ChildFeedbackActionBuilder {\n  override def feedbackType: tlst.FeedbackType = tlst.FeedbackType.Relevant\n  override def promptString: ExternalString = externalStrings.seeMoreString\n  override def confirmationString: ExternalString =\n    externalStrings.genericConfirmationString\n  override def clientEventComponent: Option[String] = Some(\"for_you_post_followup\")\n  override def clientEventElement: String = \"feedback_relevant\"\n  override def clientEventAction: String = \"click\"\n  override def internalFeedbackType: FeedbackType = Relevant\n  override def hasUndoAction: Boolean = false\n  override def confirmationDisplayType: Option[ConfirmationDisplayType] = Some(BottomSheet)\n}\n\n@Singleton\ncase class SeeLessChildFeedbackActionBuilder @Inject() (\n  @ProductScoped override val stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends ChildFeedbackActionBuilder {\n  override def feedbackType: tlst.FeedbackType = tlst.FeedbackType.NotRelevant\n  override def promptString: ExternalString = externalStrings.seeLessString\n  override def confirmationString: ExternalString =\n    externalStrings.genericConfirmationString\n  override def clientEventComponent: Option[String] = Some(\"for_you_post_followup\")\n  override def clientEventElement: String = \"feedback_notrelevant\"\n  override def clientEventAction: String = \"click\"\n  override def internalFeedbackType: FeedbackType = NotRelevant\n  override def hasUndoAction: Boolean = false\n  override def confirmationDisplayType: Option[ConfirmationDisplayType] = Some(BottomSheet)\n}\n\n@Singleton\ncase class RelevantChildFeedbackActionBuilder @Inject() (\n  @ProductScoped override val stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends ChildFeedbackActionBuilder {\n  override def feedbackType: tlst.FeedbackType = tlst.FeedbackType.Relevant\n  override def promptString: ExternalString = externalStrings.relevantString\n  override def confirmationString: ExternalString =\n    externalStrings.relevantConfirmationString\n  override def clientEventComponent: Option[String] = Some(\"for_you_post_relevance_prompt\")\n  override def clientEventElement: String = \"feedback_relevant\"\n  override def clientEventAction: String = \"click\"\n  override def internalFeedbackType: FeedbackType = Relevant\n  override def hasUndoAction: Boolean = false\n  override def confirmationDisplayType: Option[ConfirmationDisplayType] = Some(BottomSheet)\n  override def getPrompt(query: PipelineQuery): String = {\n    query.params(PostFeedbackPromptPositiveParam)\n  }\n}\n\n@Singleton\ncase class NotRelevantChildFeedbackActionBuilder @Inject() (\n  @ProductScoped override val stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends ChildFeedbackActionBuilder {\n  override def feedbackType: tlst.FeedbackType = tlst.FeedbackType.NotRelevant\n  override def promptString: ExternalString = externalStrings.notRelevantString\n  override def confirmationString: ExternalString =\n    externalStrings.notRelevantConfirmationString\n  override def clientEventComponent: Option[String] = Some(\"for_you_post_relevance_prompt\")\n  override def clientEventElement: String = \"feedback_notrelevant\"\n  override def clientEventAction: String = \"click\"\n  override def internalFeedbackType: FeedbackType = NotRelevant\n  override def hasUndoAction: Boolean = false\n  override def confirmationDisplayType: Option[ConfirmationDisplayType] = Some(BottomSheet)\n  override def getPrompt(query: PipelineQuery): String = {\n    query.params(PostFeedbackPromptNegativeParam)\n  }\n\n}\n\n@Singleton\ncase class NeutralChildFeedbackActionBuilder @Inject() (\n  @ProductScoped override val stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends ChildFeedbackActionBuilder {\n  override def feedbackType: tlst.FeedbackType = tlst.FeedbackType.Neutral\n  override def promptString: ExternalString = externalStrings.neutralString\n  override def confirmationString: ExternalString =\n    externalStrings.neutralConfirmationString\n  override def clientEventComponent: Option[String] = Some(\"for_you_post_relevance_prompt\")\n  override def clientEventElement: String = \"feedback_neutral\"\n  override def clientEventAction: String = \"click\"\n  override def internalFeedbackType: FeedbackType = Neutral\n  override def hasUndoAction: Boolean = false\n  override def confirmationDisplayType: Option[ConfirmationDisplayType] = Some(BottomSheet)\n  override def getPrompt(query: PipelineQuery): String = {\n    query.params(PostFeedbackPromptNeutralParam)\n  }\n}\n\n@Singleton\ncase class DontlikeNotRelevantChildFeedbackActionBuilder @Inject() (\n  @ProductScoped override val stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends ChildFeedbackActionBuilder {\n  override def feedbackType: tlst.FeedbackType = tlst.FeedbackType.NotRelevant\n  override def promptString: ExternalString = externalStrings.notRelevantString\n  override def confirmationString: ExternalString =\n    externalStrings.notRelevantConfirmationString\n  override def clientEventElement: String = \"feedback_notrelevant\"\n  override def clientEventAction: String = \"click\"\n  override def internalFeedbackType: FeedbackType = NotRelevant\n}\n\n@Singleton\ncase class HatefulChildFeedbackActionBuilder @Inject() (\n  @ProductScoped override val stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends ChildFeedbackActionBuilder {\n  override def feedbackType: tlst.FeedbackType = tlst.FeedbackType.Hateful\n  override def promptString: ExternalString = externalStrings.hatefulString\n  override def confirmationString: ExternalString =\n    externalStrings.hatefulConfirmationString\n  override def clientEventElement: String = \"feedback_hateful\"\n  override def clientEventAction: String = \"click\"\n  override def internalFeedbackType: FeedbackType = Hateful\n}\n\n@Singleton\ncase class BoringChildFeedbackActionBuilder @Inject() (\n  @ProductScoped override val stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends ChildFeedbackActionBuilder {\n  override def feedbackType: tlst.FeedbackType = tlst.FeedbackType.Boring\n  override def promptString: ExternalString = externalStrings.boringString\n  override def confirmationString: ExternalString =\n    externalStrings.boringConfirmationString\n  override def clientEventElement: String = \"feedback_boring\"\n  override def clientEventAction: String = \"click\"\n  override def internalFeedbackType: FeedbackType = Boring\n}\n\n@Singleton\ncase class ConfusingChildFeedbackActionBuilder @Inject() (\n  @ProductScoped override val stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends ChildFeedbackActionBuilder {\n  override def feedbackType: tlst.FeedbackType = tlst.FeedbackType.Confusing\n  override def promptString: ExternalString = externalStrings.confusingString\n  override def confirmationString: ExternalString =\n    externalStrings.confusingConfirmationString\n  override def clientEventElement: String = \"feedback_confusing\"\n  override def clientEventAction: String = \"click\"\n  override def internalFeedbackType: FeedbackType = Confusing\n}\n\n@Singleton\ncase class ClickbaitChildFeedbackActionBuilder @Inject() (\n  @ProductScoped override val stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends ChildFeedbackActionBuilder {\n  override def feedbackType: tlst.FeedbackType = tlst.FeedbackType.Clickbait\n  override def promptString: ExternalString = externalStrings.clickbaitString\n  override def confirmationString: ExternalString =\n    externalStrings.clickbaitConfirmationString\n  override def clientEventElement: String = \"feedback_clickbait\"\n  override def clientEventAction: String = \"click\"\n  override def internalFeedbackType: FeedbackType = Clickbait\n}\n\n@Singleton\ncase class RagebaitChildFeedbackActionBuilder @Inject() (\n  @ProductScoped override val stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends ChildFeedbackActionBuilder {\n  override def feedbackType: tlst.FeedbackType = tlst.FeedbackType.Ragebait\n  override def promptString: ExternalString = externalStrings.ragebaitString\n  override def confirmationString: ExternalString =\n    externalStrings.ragebaitConfirmationString\n  override def clientEventElement: String = \"feedback_ragebait\"\n  override def clientEventAction: String = \"click\"\n  override def internalFeedbackType: FeedbackType = Ragebait\n}\n\n@Singleton\ncase class RegretChildFeedbackActionBuilder @Inject() (\n  @ProductScoped override val stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings)\n    extends ChildFeedbackActionBuilder {\n  override def feedbackType: tlst.FeedbackType = tlst.FeedbackType.Regret\n  override def promptString: ExternalString = externalStrings.regretString\n  override def confirmationString: ExternalString =\n    externalStrings.regretConfirmationString\n  override def clientEventElement: String = \"feedback_regret\"\n  override def clientEventAction: String = \"click\"\n  override def internalFeedbackType: FeedbackType = Regret\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/DebugSocialContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.DebugStringFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableDebugString\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.CommunityGeneralContextType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DeepLink\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.GeneralContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.util.Try\n\nobject DebugSocialContextBuilder extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {\n\n  val TweetUrl = \"\"\n  val UserUrl = \"\"\n  val TrendsUrl = \"\"\n  val UserSignals = Set(\"Follow\", \"Profile\")\n  val Trends = \"Trends\"\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext] = {\n    if (query.params(EnableDebugString)) {\n      candidateFeatures.getOrElse(DebugStringFeature, None).map { debugString =>\n        val signalId = Try(debugString.split(\" \").head.toLong).toOption\n        val baseUrl =\n          if (UserSignals.exists(debugString.contains)) UserUrl\n          else if (debugString.contains(Trends)) TrendsUrl\n          else TweetUrl\n\n        val url = signalId.map { id =>\n          Url(\n            urlType = DeepLink,\n            url = s\"$baseUrl$id\",\n            urtEndpointOptions = None\n          )\n        }\n\n        GeneralContext(\n          contextType = CommunityGeneralContextType,\n          text = debugString,\n          url = None,\n          contextImageUrls = None,\n          landingUrl = url\n        )\n      }\n    } else None\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/DontLikeFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableAdditionalChildFeedbackParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableBlockMuteReportChildFeedbackParam\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.Frown\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DontLike\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.common.{thriftscala => tlc}\nimport com.twitter.timelineservice.model.FeedbackMetadata\nimport com.twitter.timelineservice.model.FeedbackInfo\nimport com.twitter.timelineservice.{thriftscala => tls}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class DontLikeFeedbackActionBuilder @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings,\n  authorChildFeedbackActionBuilder: AuthorChildFeedbackActionBuilder,\n  retweeterChildFeedbackActionBuilder: RetweeterChildFeedbackActionBuilder,\n  notRelevantChildFeedbackActionBuilder: DontlikeNotRelevantChildFeedbackActionBuilder,\n  hatefulChildFeedbackActionBuilder: HatefulChildFeedbackActionBuilder,\n  boringChildFeedbackActionBuilder: BoringChildFeedbackActionBuilder,\n  confusingChildFeedbackActionBuilder: ConfusingChildFeedbackActionBuilder,\n  clickbaitChildFeedbackActionBuilder: ClickbaitChildFeedbackActionBuilder,\n  ragebaitChildFeedbackActionBuilder: RagebaitChildFeedbackActionBuilder,\n  regretChildFeedbackActionBuilder: RegretChildFeedbackActionBuilder,\n  blockUserChildFeedbackActionBuilder: BlockUserChildFeedbackActionBuilder,\n  muteUserChildFeedbackActionBuilder: MuteUserChildFeedbackActionBuilder) {\n\n  private val DontLikeClientEventInfo =\n    ClientEventInfo(None, Some(\"feedback_dontlike\"), None, Some(\"click\"), None)\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackAction] = {\n    CandidatesUtil.getOriginalAuthorId(candidateFeatures).map { authorId =>\n      val feedbackEntities = Seq(\n        tlc.FeedbackEntity.TweetId(candidate.id),\n        tlc.FeedbackEntity.UserId(authorId)\n      )\n      val feedbackMetadata = FeedbackMetadata(\n        engagementType = None,\n        entityIds = feedbackEntities,\n        ttl = Some(30.days)\n      )\n      val feedbackUrl = FeedbackInfo.feedbackUrl(\n        feedbackType = tls.FeedbackType.DontLike,\n        feedbackMetadata = feedbackMetadata,\n        injectionType = None\n      )\n\n      val additionalChildFeedbackActions: Seq[ChildFeedbackAction] =\n        if (query.params(EnableAdditionalChildFeedbackParam)) {\n          Seq(\n            hatefulChildFeedbackActionBuilder(query, candidate, candidateFeatures),\n            boringChildFeedbackActionBuilder(query, candidate, candidateFeatures),\n            confusingChildFeedbackActionBuilder(query, candidate, candidateFeatures),\n            clickbaitChildFeedbackActionBuilder(query, candidate, candidateFeatures),\n            ragebaitChildFeedbackActionBuilder(query, candidate, candidateFeatures),\n            regretChildFeedbackActionBuilder(query, candidate, candidateFeatures)\n          ).flatten\n        } else Seq.empty\n\n      val blockMuteReportChildFeedbackActions: Seq[ChildFeedbackAction] =\n        if (query.params(EnableBlockMuteReportChildFeedbackParam)) {\n          Seq(\n            blockUserChildFeedbackActionBuilder(candidateFeatures),\n            muteUserChildFeedbackActionBuilder(candidateFeatures)\n          ).flatten\n        } else Seq.empty\n\n      val childFeedbackActions = Seq(\n        authorChildFeedbackActionBuilder(candidateFeatures),\n        retweeterChildFeedbackActionBuilder(candidateFeatures),\n        notRelevantChildFeedbackActionBuilder(query, candidate, candidateFeatures),\n      ).flatten ++ additionalChildFeedbackActions ++ blockMuteReportChildFeedbackActions\n\n      FeedbackAction(\n        feedbackType = DontLike,\n        prompt = Some(stringCenter.prepare(externalStrings.dontLikeString)),\n        confirmation = Some(stringCenter.prepare(externalStrings.dontLikeConfirmationString)),\n        childFeedbackActions =\n          if (childFeedbackActions.nonEmpty) Some(childFeedbackActions) else None,\n        feedbackUrl = Some(feedbackUrl),\n        hasUndoAction = Some(true),\n        confirmationDisplayType = None,\n        clientEventInfo = Some(DontLikeClientEventInfo),\n        icon = Some(Frown),\n        richBehavior = None,\n        subprompt = None,\n        encodedFeedbackRequest = None\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/EngagerSocialContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.stringcenter.client.core.ExternalString\n\nprivate[decorator] case class SocialContextIdAndScreenName(\n  socialContextId: Long,\n  screenName: String)\n\nobject EngagerSocialContextBuilder {\n  private val UserIdRequestParamName = \"user_id\"\n  private val DirectInjectionContentSourceRequestParamName = \"dis\"\n  private val DirectInjectionIdRequestParamName = \"diid\"\n  private val DirectInjectionContentSourceSocialProofUsers = \"socialproofusers\"\n  private val SocialProofUrl = \"/2/timeline/social_proof.json\"\n}\n\ncase class EngagerSocialContextBuilder(\n  contextType: GeneralContextType,\n  stringCenter: StringCenter,\n  oneUserString: ExternalString,\n  twoUsersString: ExternalString,\n  moreUsersString: ExternalString,\n  timelineTitle: ExternalString) {\n  import EngagerSocialContextBuilder._\n\n  def apply(\n    socialContextIds: Seq[Long],\n    query: PipelineQuery,\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext] = {\n    val realNames = candidateFeatures.getOrElse(RealNamesFeature, Map.empty[Long, String])\n    val validSocialContextIdAndScreenNames = socialContextIds.flatMap { socialContextId =>\n      realNames\n        .get(socialContextId)\n        .map(screenName => SocialContextIdAndScreenName(socialContextId, screenName))\n    }\n\n    validSocialContextIdAndScreenNames match {\n      case Seq(user) =>\n        val socialContextString =\n          stringCenter.prepare(oneUserString, Map(\"user\" -> user.screenName))\n        Some(mkOneUserSocialContext(socialContextString, user.socialContextId))\n      case Seq(firstUser, secondUser) =>\n        val socialContextString =\n          stringCenter.prepare(\n            twoUsersString,\n            Map(\"user1\" -> firstUser.screenName, \"user2\" -> secondUser.screenName)\n          )\n        Some(\n          mkManyUserSocialContext(\n            socialContextString,\n            query.getRequiredUserId,\n            validSocialContextIdAndScreenNames.map(_.socialContextId)\n          )\n        )\n\n      case firstUser +: otherUsers =>\n        val otherUsersCount = otherUsers.size\n        val socialContextString =\n          stringCenter.prepare(\n            moreUsersString,\n            Map(\"user\" -> firstUser.screenName, \"count\" -> otherUsersCount)\n          )\n        Some(\n          mkManyUserSocialContext(\n            socialContextString,\n            query.getRequiredUserId,\n            validSocialContextIdAndScreenNames.map(_.socialContextId)\n          )\n        )\n      case _ => None\n    }\n  }\n\n  private def mkOneUserSocialContext(socialContextString: String, userId: Long): GeneralContext = {\n    GeneralContext(\n      contextType = contextType,\n      text = socialContextString,\n      url = None,\n      contextImageUrls = None,\n      landingUrl = Some(\n        Url(\n          urlType = DeepLink,\n          url = \"\",\n          urtEndpointOptions = None\n        )\n      )\n    )\n  }\n\n  private def mkManyUserSocialContext(\n    socialContextString: String,\n    viewerId: Long,\n    socialContextIds: Seq[Long]\n  ): GeneralContext = {\n    GeneralContext(\n      contextType = contextType,\n      text = socialContextString,\n      url = None,\n      contextImageUrls = None,\n      landingUrl = Some(\n        Url(\n          urlType = UrtEndpoint,\n          url = SocialProofUrl,\n          urtEndpointOptions = Some(UrtEndpointOptions(\n            requestParams = Some(Map(\n              UserIdRequestParamName -> viewerId.toString,\n              DirectInjectionContentSourceRequestParamName -> DirectInjectionContentSourceSocialProofUsers,\n              DirectInjectionIdRequestParamName -> socialContextIds.mkString(\",\")\n            )),\n            title = Some(stringCenter.prepare(timelineTitle)),\n            cacheId = None,\n            subtitle = None\n          ))\n        ))\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/ExtendedReplySocialContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.FocalTweetAuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FocalTweetRealNamesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n/**\n * Use '@A replied' when the root tweet is out-of-network and the reply is in network.\n *\n * This function should only be called for the root Tweet of convo modules. This is enforced by\n * [[HomeTweetSocialContextBuilder]].\n */\n@Singleton\ncase class ExtendedReplySocialContextBuilder @Inject() (\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter])\n    extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {\n\n  private val stringCenter = stringCenterProvider.get()\n  private val extendedReplyString = externalStrings.socialContextExtendedReply\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext] = {\n\n    // If these values are missing default to not showing an extended reply banner\n    val inNetworkRoot = candidateFeatures.getOrElse(InNetworkFeature, true)\n\n    val inNetworkFocalTweet =\n      candidateFeatures.getOrElse(FocalTweetInNetworkFeature, None).getOrElse(false)\n\n    if (!inNetworkRoot && inNetworkFocalTweet) {\n\n      val focalTweetAuthorIdOpt = candidateFeatures.getOrElse(FocalTweetAuthorIdFeature, None)\n      val focalTweetRealNames =\n        candidateFeatures\n          .getOrElse(FocalTweetRealNamesFeature, None).getOrElse(Map.empty[Long, String])\n      val focalTweetAuthorNameOpt = focalTweetAuthorIdOpt.flatMap(focalTweetRealNames.get)\n\n      (focalTweetAuthorIdOpt, focalTweetAuthorNameOpt) match {\n        case (Some(focalTweetAuthorId), Some(focalTweetAuthorName)) =>\n          Some(\n            GeneralContext(\n              contextType = ConversationGeneralContextType,\n              text = stringCenter\n                .prepare(extendedReplyString, placeholders = Map(\"user1\" -> focalTweetAuthorName)),\n              url = None,\n              contextImageUrls = None,\n              landingUrl = Some(\n                Url(\n                  urlType = DeepLink,\n                  url = \"\",\n                  urtEndpointOptions = None\n                ))\n            ))\n        case _ =>\n          None\n      }\n    } else {\n      None\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/FeedbackStrings.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.ExternalStringRegistry\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass FeedbackStrings @Inject() (\n  @ProductScoped externalStringRegistryProvider: Provider[ExternalStringRegistry]) {\n  private val externalStringRegistry = externalStringRegistryProvider.get()\n\n  val seeLessOftenFeedbackString =\n    externalStringRegistry.createProdString(\"Feedback.seeLessOften\")\n  val seeLessOftenConfirmationFeedbackString =\n    externalStringRegistry.createProdString(\"Feedback.seeLessOftenConfirmation\")\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/FeedbackUtil.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SeeFewer\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.stringcenter.client.core.ExternalString\nimport com.twitter.timelines.common.{thriftscala => tlc}\nimport com.twitter.timelines.service.{thriftscala => t}\nimport com.twitter.timelineservice.model.FeedbackInfo\nimport com.twitter.timelineservice.model.FeedbackMetadata\nimport com.twitter.timelineservice.{thriftscala => tlst}\n\nobject FeedbackUtil {\n\n  val FeedbackTtl = 30.days\n\n  def buildUserSeeFewerChildFeedbackAction(\n    userId: Long,\n    namesByUserId: Map[Long, String],\n    promptExternalString: ExternalString,\n    confirmationExternalString: ExternalString,\n    engagementType: t.FeedbackEngagementType,\n    stringCenter: StringCenter\n  ): Option[ChildFeedbackAction] = {\n    namesByUserId.get(userId).map { userScreenName =>\n      val prompt = stringCenter.prepare(\n        promptExternalString,\n        Map(\"user\" -> userScreenName)\n      )\n      val confirmation = stringCenter.prepare(\n        confirmationExternalString,\n        Map(\"user\" -> userScreenName)\n      )\n      val feedbackMetadata = FeedbackMetadata(\n        engagementType = Some(engagementType),\n        entityIds = Seq(tlc.FeedbackEntity.UserId(userId)),\n        ttl = Some(FeedbackTtl))\n      val feedbackUrl = FeedbackInfo.feedbackUrl(\n        feedbackType = tlst.FeedbackType.SeeFewer,\n        feedbackMetadata = feedbackMetadata,\n        injectionType = None\n      )\n\n      ChildFeedbackAction(\n        feedbackType = SeeFewer,\n        prompt = Some(prompt),\n        confirmation = Some(confirmation),\n        feedbackUrl = Some(feedbackUrl),\n        hasUndoAction = Some(true),\n        confirmationDisplayType = None,\n        clientEventInfo = None,\n        icon = None,\n        richBehavior = None,\n        subprompt = None\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/FollowedBySocialContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\ncase class FollowedBySocialContextBuilder @Inject() (\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter])\n    extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {\n\n  private val stringCenter = stringCenterProvider.get()\n\n  private val engagerSocialContextBuilder = EngagerSocialContextBuilder(\n    contextType = FollowGeneralContextType,\n    stringCenter = stringCenter,\n    oneUserString = externalStrings.socialContextOneUserFollowsString,\n    twoUsersString = externalStrings.socialContextTwoUsersFollowString,\n    moreUsersString = externalStrings.socialContextMoreUsersFollowString,\n    timelineTitle = externalStrings.socialContextFollowedByTimelineTitle\n  )\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext] = {\n    // Only apply followed-by social context for OON Tweets\n    val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)\n    if (!inNetwork) {\n      val validFollowedByUserIds =\n        candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Nil)\n      engagerSocialContextBuilder(\n        socialContextIds = validFollowedByUserIds,\n        query = query,\n        candidateFeatures = candidateFeatures\n      )\n    } else {\n      None\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/HomeFeedbackActionInfoBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnablePostFeedbackParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnablePostFollowupParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnablePostDetailsNegativeFeedbackParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.PostFeedbackThresholdParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.PostFollowupThresholdParam\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.util.Random\n\n@Singleton\nclass HomeFeedbackActionInfoBuilder @Inject() (\n  postFeedbackActionBuilder: PostFeedbackActionBuilder,\n  postFollowupFeedbackActionBuilder: PostFollowupFeedbackActionBuilder,\n  dontLikeFeedbackActionBuilder: DontLikeFeedbackActionBuilder,\n  postDetailsNegativeFeedbackActionBuilder: PostDetailsNegativeFeedbackActionBuilder)\n    extends BaseFeedbackActionInfoBuilder[PipelineQuery, TweetCandidate] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackActionInfo] = {\n    val supportedProduct = query.product match {\n      case ForYouProduct => true\n      case _ => false\n    }\n    val isAuthoredByViewer = CandidatesUtil.isAuthoredByViewer(query, candidateFeatures)\n\n    if (supportedProduct && !isAuthoredByViewer) {\n      // avoid showing post feedback for every candidate\n      val shouldShowPostFeedback =\n        query.params(EnablePostFeedbackParam) &&\n          Random.nextDouble() < query.params(PostFeedbackThresholdParam)\n\n      val shouldShowPostFollowup =\n        query.params(EnablePostFollowupParam) &&\n          Random.nextDouble() < query.params(PostFollowupThresholdParam)\n\n      val shouldShowPostDetailsNegative =\n        query.params(EnablePostDetailsNegativeFeedbackParam)\n\n      val feedbackActions =\n        Seq(\n          if (shouldShowPostFeedback)\n            postFeedbackActionBuilder(query, candidate, candidateFeatures)\n          else None,\n          if (shouldShowPostFollowup)\n            postFollowupFeedbackActionBuilder(query, candidate, candidateFeatures)\n          else None,\n          dontLikeFeedbackActionBuilder(\n            query,\n            candidate,\n            candidateFeatures\n          ),\n          if (shouldShowPostDetailsNegative)\n            postDetailsNegativeFeedbackActionBuilder(\n              query,\n              candidate,\n              candidateFeatures\n            )\n          else None\n        ).flatten\n\n      Some(\n        FeedbackActionInfo(\n          feedbackActions = feedbackActions,\n          feedbackMetadata = None,\n          displayContext = None,\n          clientEventInfo = None\n        ))\n    } else None\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/HomeTweetContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.BasketballContextFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GenericPostContextFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseTweetContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeTweetContextBuilder @Inject() ()\n    extends BaseTweetContextBuilder[PipelineQuery, TweetCandidate] {\n\n  private val grokCom = \"grok.com\"\n\n  private def mapBasketballStatus(originalStatus: Option[String]): Option[String] = {\n    originalStatus.flatMap {\n      case \"Inprogress\" | \"Halftime\" => Some(\"Live\")\n      case \"Closed\" | \"Completed\" => Some(\"Final\")\n      case \"Created\" | \"Scheduled\" => Some(\"Upcoming\")\n      case _ => None\n    }\n  }\n\n  private def mapPoints(status: String, points: Option[Short]): Option[Short] = {\n    if (status == \"Live\" || status == \"Final\") {\n      points\n    } else {\n      None\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    features: FeatureMap\n  ): Option[TweetContext] = {\n    features\n      .getOrElse(BasketballContextFeature, None).flatMap { basketballContext =>\n        val mappedStatus = mapBasketballStatus(basketballContext.status)\n\n        mappedStatus.map { status =>\n          TweetContext(\n            contextType = ContextType.Topic,\n            text = \"\",\n            landingUrl = None,\n            contextImageUrls = None,\n            context = Some(TweetContextDetails.Basketball(BasketballContext(\n              clock = basketballContext.clock,\n              homeTeamScore = mapPoints(status, basketballContext.homeTeamScore),\n              awayTeamScore = mapPoints(status, basketballContext.awayTeamScore),\n              homeTeamName = basketballContext.homeTeamName,\n              awayTeamName = basketballContext.awayTeamName,\n              status = Some(status),\n              url = Url(\n                urlType = DeepLink,\n                url = basketballContext.url.url,\n                urtEndpointOptions = None\n              )\n            ))),\n            icon = None\n          )\n        }\n      }.orElse {\n        features.getOrElse(GenericPostContextFeature, None).map { genericContext =>\n          val grokIconOpt = if (genericContext.url.url.contains(grokCom)) Some(icon.Grok) else None\n\n          TweetContext(\n            contextType = ContextType.Topic,\n            text = genericContext.primaryText,\n            landingUrl = None,\n            contextImageUrls = None,\n            context = Some(TweetContextDetails.Generic(GenericContext(\n              primaryText = genericContext.primaryText,\n              secondaryText = genericContext.secondaryText,\n              url = Url(\n                urlType = DeepLink,\n                url = genericContext.url.url,\n                urtEndpointOptions = None\n              )\n            ))),\n            icon = grokIconOpt\n          )\n        }\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/HomeTweetSocialContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ConversationModuleIdFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableCommunitiesContextParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableSocialContextParam\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.social_context.CommunitiesSocialContextBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class HomeTweetSocialContextBuilder @Inject() (\n  likedBySocialContextBuilder: LikedBySocialContextBuilder,\n  servedTypeSocialContextBuilder: ServedTypeSocialContextBuilder,\n  followedBySocialContextBuilder: FollowedBySocialContextBuilder,\n  extendedReplySocialContextBuilder: ExtendedReplySocialContextBuilder,\n  receivedReplySocialContextBuilder: ReceivedReplySocialContextBuilder)\n    extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    features: FeatureMap\n  ): Option[SocialContext] = {\n    val communitiesSocialContextBuilder =\n      if (query.params(EnableCommunitiesContextParam))\n        CommunitiesSocialContextBuilder(query, candidate, features)\n      else None\n\n    if (query.params(EnableSocialContextParam)) {\n      features.getOrElse(ConversationModuleFocalTweetIdFeature, None) match {\n        case None =>\n          DebugSocialContextBuilder(query, candidate, features)\n            .orElse(communitiesSocialContextBuilder)\n            .orElse(servedTypeSocialContextBuilder(query, candidate, features))\n            .orElse(followedBySocialContextBuilder(query, candidate, features))\n        case Some(_) =>\n          val conversationId = features.getOrElse(ConversationModuleIdFeature, None)\n          // Only hydrate the social context into the root tweet in a conversation module\n          if (conversationId.contains(candidate.id)) {\n            communitiesSocialContextBuilder\n              .orElse(extendedReplySocialContextBuilder(query, candidate, features))\n              .orElse(receivedReplySocialContextBuilder(query, candidate, features))\n          } else None\n      }\n    } else None\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/HomeWhoToFollowFeedbackActionInfoBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.WhoToFollowFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.service.{thriftscala => tl}\nimport com.twitter.timelines.util.FeedbackRequestSerializer\nimport com.twitter.timelineservice.suggests.thriftscala.SuggestType\nimport com.twitter.timelineservice.thriftscala.FeedbackType\n\nobject HomeWhoToFollowFeedbackActionInfoBuilder {\n  private val FeedbackMetadata = tl.FeedbackMetadata(\n    injectionType = Some(SuggestType.WhoToFollow),\n    engagementType = None,\n    entityIds = Seq.empty,\n    ttlMs = None\n  )\n  private val FeedbackRequest =\n    tl.DefaultFeedbackRequest2(FeedbackType.SeeFewer, FeedbackMetadata)\n  private val EncodedFeedbackRequest =\n    FeedbackRequestSerializer.serialize(tl.FeedbackRequest.DefaultFeedbackRequest2(FeedbackRequest))\n}\n\n@Singleton\ncase class HomeWhoToFollowFeedbackActionInfoBuilder @Inject() (\n  feedbackStrings: FeedbackStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter])\n    extends BaseFeedbackActionInfoBuilder[PipelineQuery, UserCandidate] {\n\n  private val whoToFollowFeedbackActionInfoBuilder = WhoToFollowFeedbackActionInfoBuilder(\n    seeLessOftenFeedbackString = feedbackStrings.seeLessOftenFeedbackString,\n    seeLessOftenConfirmationFeedbackString = feedbackStrings.seeLessOftenConfirmationFeedbackString,\n    stringCenter = stringCenterProvider.get(),\n    encodedFeedbackRequest = Some(HomeWhoToFollowFeedbackActionInfoBuilder.EncodedFeedbackRequest)\n  )\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: UserCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackActionInfo] =\n    whoToFollowFeedbackActionInfoBuilder.apply(query, candidate, candidateFeatures)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/HomeWhoToSubscribeFeedbackActionInfoBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.WhoToFollowFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.service.{thriftscala => tl}\nimport com.twitter.timelines.util.FeedbackRequestSerializer\nimport com.twitter.timelineservice.suggests.thriftscala.SuggestType\nimport com.twitter.timelineservice.thriftscala.FeedbackType\n\nobject HomeWhoToSubscribeFeedbackActionInfoBuilder {\n  private val FeedbackMetadata = tl.FeedbackMetadata(\n    injectionType = Some(SuggestType.WhoToSubscribe),\n    engagementType = None,\n    entityIds = Seq.empty,\n    ttlMs = None\n  )\n  private val FeedbackRequest =\n    tl.DefaultFeedbackRequest2(FeedbackType.SeeFewer, FeedbackMetadata)\n  private val EncodedFeedbackRequest =\n    FeedbackRequestSerializer.serialize(tl.FeedbackRequest.DefaultFeedbackRequest2(FeedbackRequest))\n}\n\n@Singleton\ncase class HomeWhoToSubscribeFeedbackActionInfoBuilder @Inject() (\n  feedbackStrings: FeedbackStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter])\n    extends BaseFeedbackActionInfoBuilder[PipelineQuery, UserCandidate] {\n\n  private val whoToSubscribeFeedbackActionInfoBuilder = WhoToFollowFeedbackActionInfoBuilder(\n    seeLessOftenFeedbackString = feedbackStrings.seeLessOftenFeedbackString,\n    seeLessOftenConfirmationFeedbackString = feedbackStrings.seeLessOftenConfirmationFeedbackString,\n    stringCenter = stringCenterProvider.get(),\n    encodedFeedbackRequest =\n      Some(HomeWhoToSubscribeFeedbackActionInfoBuilder.EncodedFeedbackRequest)\n  )\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: UserCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackActionInfo] =\n    whoToSubscribeFeedbackActionInfoBuilder.apply(query, candidate, candidateFeatures)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/LikedBySocialContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.ValidLikedByUserIdsFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.LikeGeneralContextType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\ncase class LikedBySocialContextBuilder @Inject() (\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter])\n    extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {\n\n  private val stringCenter = stringCenterProvider.get()\n\n  private val engagerSocialContextBuilder = EngagerSocialContextBuilder(\n    contextType = LikeGeneralContextType,\n    stringCenter = stringCenter,\n    oneUserString = externalStrings.socialContextOneUserLikedString,\n    twoUsersString = externalStrings.socialContextTwoUsersLikedString,\n    moreUsersString = externalStrings.socialContextMoreUsersLikedString,\n    timelineTitle = externalStrings.socialContextLikedByTimelineTitle\n  )\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext] = {\n\n    // Liked by users are valid only if they pass both the SGS and Perspective filters.\n    val validLikedByUserIds = candidateFeatures.getOrElse(ValidLikedByUserIdsFeature, Nil)\n\n    engagerSocialContextBuilder(\n      socialContextIds = validLikedByUserIds,\n      query = query,\n      candidateFeatures = candidateFeatures\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/ListsSocialContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserScreenNameFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelineservice.suggests.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n/**\n * \"Your Lists\" will be rendered for the context and a url link for your lists.\n */\n@Singleton\ncase class ListsSocialContextBuilder @Inject() (\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter])\n    extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {\n\n  private val stringCenter = stringCenterProvider.get()\n  private val listString = externalStrings.ownedSubscribedListsModuleHeaderString\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext] = {\n    candidateFeatures.get(SuggestTypeFeature) match {\n      case Some(suggestType) if suggestType == t.SuggestType.RankedListTweet =>\n        val userName = query.features.flatMap(_.getOrElse(UserScreenNameFeature, None))\n        Some(\n          GeneralContext(\n            contextType = ListGeneralContextType,\n            text = stringCenter.prepare(listString),\n            url = userName.map(name => \"\"),\n            contextImageUrls = None,\n            landingUrl = None\n          ))\n      case _ => None\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/MuteUserChildFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BottomSheet\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleMuteUser\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class MuteUserChildFeedbackActionBuilder @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings) {\n\n  def apply(\n    candidateFeatures: FeatureMap\n  ): Option[ChildFeedbackAction] = {\n    val userIdOpt =\n      if (candidateFeatures.getOrElse(IsRetweetFeature, false))\n        candidateFeatures.getOrElse(SourceUserIdFeature, None)\n      else candidateFeatures.getOrElse(AuthorIdFeature, None)\n\n    userIdOpt.flatMap { userId =>\n      val screenNamesMap = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])\n      val userScreenNameOpt = screenNamesMap.get(userId)\n      userScreenNameOpt.map { userScreenName =>\n        val prompt = stringCenter.prepare(\n          externalStrings.muteUserString,\n          Map(\"username\" -> userScreenName)\n        )\n        val confirmation = stringCenter.prepare(\n          externalStrings.muteUserConfirmationString,\n          Map(\"username\" -> userScreenName)\n        )\n        ChildFeedbackAction(\n          feedbackType = RichBehavior,\n          prompt = Some(prompt),\n          confirmation = Some(confirmation),\n          subprompt = None,\n          feedbackUrl = None,\n          hasUndoAction = Some(true),\n          confirmationDisplayType = Some(BottomSheet),\n          clientEventInfo = Some(\n            ClientEventInfo(\n              component = None,\n              element = Some(\"mute\"),\n              details = None,\n              action = Some(\"click\"),\n              entityToken = None\n            )\n          ),\n          icon = Some(icon.SpeakerOff),\n          richBehavior = Some(RichFeedbackBehaviorToggleMuteUser(userId)),\n        )\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/NotInterestedTopicFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecWithEducationTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorMarkNotInterestedTopic\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class NotInterestedTopicFeedbackActionBuilder @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings) {\n\n  def apply(\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackAction] = {\n    val isOutOfNetwork = !candidateFeatures.getOrElse(InNetworkFeature, true)\n    val validFollowedByUserIds =\n      candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Nil)\n    val validLikedByUserIds =\n      candidateFeatures\n        .getOrElse(SGSValidLikedByUserIdsFeature, Nil)\n        .filter(\n          candidateFeatures.getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Nil).toSet.contains)\n\n    if (isOutOfNetwork && validLikedByUserIds.isEmpty && validFollowedByUserIds.isEmpty) {\n      val topicIdSocialContext = candidateFeatures.getOrElse(TopicIdSocialContextFeature, None)\n      val topicContextFunctionalityType =\n        candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None)\n\n      (topicIdSocialContext, topicContextFunctionalityType) match {\n        case (Some(topicId), Some(topicContextFunctionalityType))\n            if topicContextFunctionalityType == RecommendationTopicContextFunctionalityType ||\n              topicContextFunctionalityType == RecWithEducationTopicContextFunctionalityType =>\n          Some(\n            FeedbackAction(\n              feedbackType = RichBehavior,\n              prompt = None,\n              confirmation = None,\n              childFeedbackActions = None,\n              feedbackUrl = None,\n              hasUndoAction = Some(true),\n              confirmationDisplayType = None,\n              clientEventInfo = None,\n              icon = None,\n              richBehavior =\n                Some(RichFeedbackBehaviorMarkNotInterestedTopic(topicId = topicId.toString)),\n              subprompt = None,\n              encodedFeedbackRequest = None\n            )\n          )\n        case _ => None\n      }\n    } else {\n      None\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/NotRelevantChildFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.NotRelevant\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.common.{thriftscala => tlc}\nimport com.twitter.timelineservice.model.FeedbackInfo\nimport com.twitter.timelineservice.model.FeedbackMetadata\nimport com.twitter.timelineservice.{thriftscala => tlst}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class NotRelevantChildFeedbackActionBuilder @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings) {\n\n  def apply(\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[ChildFeedbackAction] = {\n    val prompt = stringCenter.prepare(externalStrings.notRelevantString)\n    val confirmation = stringCenter.prepare(externalStrings.notRelevantConfirmationString)\n    val feedbackMetadata = FeedbackMetadata(\n      engagementType = None,\n      entityIds = Seq(tlc.FeedbackEntity.TweetId(candidate.id)),\n      ttl = Some(FeedbackUtil.FeedbackTtl))\n    val feedbackUrl = FeedbackInfo.feedbackUrl(\n      feedbackType = tlst.FeedbackType.NotRelevant,\n      feedbackMetadata = feedbackMetadata,\n      injectionType = candidateFeatures.getOrElse(SuggestTypeFeature, None)\n    )\n\n    Some(\n      ChildFeedbackAction(\n        feedbackType = NotRelevant,\n        prompt = Some(prompt),\n        confirmation = Some(confirmation),\n        feedbackUrl = Some(feedbackUrl),\n        hasUndoAction = Some(true),\n        confirmationDisplayType = None,\n        clientEventInfo = None,\n        icon = None,\n        richBehavior = None,\n        subprompt = None\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/PopularInYourAreaSocialContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\ncase class PopularInYourAreaSocialContextBuilder @Inject() (\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter])\n    extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {\n\n  private val stringCenter = stringCenterProvider.get()\n  private val popularInYourAreaString = externalStrings.socialContextPopularInYourAreaString\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext] = {\n    val suggestTypeOpt = candidateFeatures.getOrElse(SuggestTypeFeature, None)\n    if (suggestTypeOpt.contains(st.SuggestType.RecommendedTrendTweet)) {\n      Some(\n        GeneralContext(\n          contextType = LocationGeneralContextType,\n          text = stringCenter.prepare(popularInYourAreaString),\n          url = None,\n          contextImageUrls = None,\n          landingUrl = None\n        ))\n    } else None\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/PopularVideoSocialContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\ncase class PopularVideoSocialContextBuilder @Inject() (\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter])\n    extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {\n\n  private val stringCenter = stringCenterProvider.get()\n  private val popularVideoString = externalStrings.socialContextPopularVideoString\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext] = {\n    val suggestTypeOpt = candidateFeatures.getOrElse(SuggestTypeFeature, None)\n    if (suggestTypeOpt.contains(st.SuggestType.MediaTweet)) {\n      Some(\n        GeneralContext(\n          contextType = SparkleGeneralContextType,\n          text = stringCenter.prepare(popularVideoString),\n          url = None,\n          contextImageUrls = None,\n          landingUrl = Some(\n            Url(\n              urlType = DeepLink,\n              url = \"\"\n            )\n          )\n        ))\n    } else None\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/PostDetailsNegativeFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.Frown\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BottomSheet\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.NotRelevant\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.common.{thriftscala => tlc}\nimport com.twitter.timelineservice.model.FeedbackInfo\nimport com.twitter.timelineservice.model.FeedbackMetadata\nimport com.twitter.timelineservice.{thriftscala => tls}\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class PostDetailsNegativeFeedbackActionBuilder @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings) {\n\n  private val PostDetailsNegativeClientEventInfo =\n    ClientEventInfo(None, Some(\"feedback_notrelevant\"), None, Some(\"click\"), None)\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackAction] = {\n    CandidatesUtil.getOriginalAuthorId(candidateFeatures).map { authorId =>\n      val feedbackEntities = Seq(\n        tlc.FeedbackEntity.TweetId(candidate.id),\n        tlc.FeedbackEntity.UserId(authorId)\n      )\n      val feedbackMetadata = FeedbackMetadata(\n        engagementType = None,\n        entityIds = feedbackEntities,\n        ttl = Some(30.days)\n      )\n      val feedbackUrl = FeedbackInfo.feedbackUrl(\n        feedbackType = tls.FeedbackType.NotRelevant,\n        feedbackMetadata = feedbackMetadata,\n        injectionType = None\n      )\n\n      FeedbackAction(\n        feedbackType = NotRelevant,\n        prompt = Some(stringCenter.prepare(externalStrings.notRelevantString)),\n        confirmation = Some(stringCenter.prepare(externalStrings.notRelevantConfirmationString)),\n        childFeedbackActions = None,\n        feedbackUrl = Some(feedbackUrl),\n        hasUndoAction = Some(false),\n        confirmationDisplayType = Some(BottomSheet),\n        clientEventInfo = Some(PostDetailsNegativeClientEventInfo),\n        icon = Some(Frown),\n        richBehavior = None,\n        subprompt = None,\n        encodedFeedbackRequest = None\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/PostFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.param.HomeGlobalParams.PostFeedbackPromptTitleParam\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.Smile\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BottomSheet\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Generic\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.common.{thriftscala => tlc}\nimport com.twitter.timelineservice.model.FeedbackInfo\nimport com.twitter.timelineservice.model.FeedbackMetadata\nimport com.twitter.timelineservice.{thriftscala => tls}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class PostFeedbackActionBuilder @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings,\n  relevantChildFeedbackActionBuilder: RelevantChildFeedbackActionBuilder,\n  notRelevantChildFeedbackActionBuilder: NotRelevantChildFeedbackActionBuilder,\n  neutralChildFeedbackActionBuilder: NeutralChildFeedbackActionBuilder) {\n\n  val ClientEventInfoComponent: String = \"for_you_post_relevance_prompt\"\n  val ClientEventInfoElement: String = \"relevance_prompt\"\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackAction] = {\n    CandidatesUtil.getOriginalAuthorId(candidateFeatures).map { authorId =>\n      val feedbackEntities = Seq(\n        tlc.FeedbackEntity.TweetId(candidate.id),\n        tlc.FeedbackEntity.UserId(authorId)\n      )\n      val feedbackMetadata = FeedbackMetadata(\n        engagementType = None,\n        entityIds = feedbackEntities,\n        ttl = Some(30.days)\n      )\n      val feedbackUrl = FeedbackInfo.feedbackUrl(\n        feedbackType = tls.FeedbackType.Generic,\n        feedbackMetadata = feedbackMetadata,\n        injectionType = None\n      )\n\n      val childFeedbackActions: Seq[ChildFeedbackAction] = {\n        Seq(\n          relevantChildFeedbackActionBuilder(query, candidate, candidateFeatures),\n          notRelevantChildFeedbackActionBuilder(query, candidate, candidateFeatures),\n          // neutralChildFeedbackActionBuilder(query, candidate, candidateFeatures)\n        ).flatten\n      }\n\n      FeedbackAction(\n        feedbackType = Generic,\n        prompt = Some(query.params(PostFeedbackPromptTitleParam)),\n        confirmation = Some(\n          stringCenter.prepare(externalStrings.genericConfirmationString)\n        ),\n        childFeedbackActions = Some(childFeedbackActions),\n        feedbackUrl = Some(feedbackUrl),\n        hasUndoAction = None,\n        confirmationDisplayType = Some(BottomSheet),\n        clientEventInfo = Some(\n          ClientEventInfo(\n            component = Some(ClientEventInfoComponent),\n            element = Some(ClientEventInfoElement),\n            details = None,\n            action = None,\n            entityToken = None\n          )),\n        icon = Some(Smile),\n        richBehavior = None,\n        subprompt = None,\n        encodedFeedbackRequest = None\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/PostFollowupFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.param.HomeGlobalParams.PostFeedbackPromptTitleParam\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BottomSheet\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.GiveFeedback\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.common.{thriftscala => tlc}\nimport com.twitter.timelineservice.model.FeedbackInfo\nimport com.twitter.timelineservice.model.FeedbackMetadata\nimport com.twitter.timelineservice.{thriftscala => tls}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class PostFollowupFeedbackActionBuilder @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings,\n  seeMoreChildFeedbackActionBuilder: SeeMoreChildFeedbackActionBuilder,\n  seeLessChildFeedbackActionBuilder: SeeLessChildFeedbackActionBuilder) {\n\n  val ClientEventInfoComponent: String = \"for_you_post_followup\"\n  val ClientEventInfoElement: String = \"followup\"\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackAction] = {\n    CandidatesUtil.getOriginalAuthorId(candidateFeatures).map { authorId =>\n      val feedbackEntities = Seq(\n        tlc.FeedbackEntity.TweetId(candidate.id),\n        tlc.FeedbackEntity.UserId(authorId)\n      )\n      val feedbackMetadata = FeedbackMetadata(\n        engagementType = None,\n        entityIds = feedbackEntities,\n        ttl = Some(30.days)\n      )\n      val feedbackUrl = FeedbackInfo.feedbackUrl(\n        feedbackType = tls.FeedbackType.Generic,\n        feedbackMetadata = feedbackMetadata,\n        injectionType = None\n      )\n\n      val childFeedbackActions: Seq[ChildFeedbackAction] = {\n        Seq(\n          seeLessChildFeedbackActionBuilder(query, candidate, candidateFeatures),\n          seeMoreChildFeedbackActionBuilder(query, candidate, candidateFeatures)\n        ).flatten\n      }\n\n      FeedbackAction(\n        feedbackType = GiveFeedback,\n        prompt = Some(query.params(PostFeedbackPromptTitleParam)),\n        confirmation = Some(\n          stringCenter.prepare(externalStrings.genericConfirmationString)\n        ),\n        childFeedbackActions = Some(childFeedbackActions),\n        feedbackUrl = Some(feedbackUrl),\n        hasUndoAction = None,\n        confirmationDisplayType = Some(BottomSheet),\n        clientEventInfo = Some(\n          ClientEventInfo(\n            component = Some(ClientEventInfoComponent),\n            element = Some(ClientEventInfoElement),\n            details = None,\n            action = None,\n            entityToken = None\n          )),\n        icon = None,\n        richBehavior = None,\n        subprompt = None,\n        encodedFeedbackRequest = None\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/ReceivedReplySocialContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n/**\n * Use '@A received a reply' as social context when the root Tweet is in network and the focal tweet is OON.\n *\n * This function should only be called for the root Tweet of convo modules. This is enforced by\n * [[HomeTweetSocialContextBuilder]].\n */\n@Singleton\ncase class ReceivedReplySocialContextBuilder @Inject() (\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter])\n    extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {\n\n  private val stringCenter = stringCenterProvider.get()\n  private val receivedReplyString = externalStrings.socialContextReceivedReply\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext] = {\n\n    // If these values are missing default to not showing a received a reply banner\n    val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, false)\n    val inNetworkFocalTweet =\n      candidateFeatures.getOrElse(FocalTweetInNetworkFeature, None).getOrElse(true)\n\n    if (inNetwork && !inNetworkFocalTweet) {\n\n      val authorIdOpt = candidateFeatures.getOrElse(AuthorIdFeature, None)\n      val realNames = candidateFeatures.getOrElse(RealNamesFeature, Map.empty[Long, String])\n      val authorNameOpt = authorIdOpt.flatMap(realNames.get)\n\n      (authorIdOpt, authorNameOpt) match {\n        case (Some(authorId), Some(authorName)) =>\n          Some(\n            GeneralContext(\n              contextType = ConversationGeneralContextType,\n              text = stringCenter\n                .prepare(receivedReplyString, placeholders = Map(\"user1\" -> authorName)),\n              url = None,\n              contextImageUrls = None,\n              landingUrl = Some(\n                Url(\n                  urlType = DeepLink,\n                  url = \"\",\n                  urtEndpointOptions = None\n                )\n              )\n            )\n          )\n        case _ => None\n      }\n    } else {\n      None\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/RelevancePromptCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.HomeFeatures.ServedIdFeature\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.relevance_prompt.RelevancePromptCandidateUrtItemStringCenterBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.StrStatic\nimport com.twitter.product_mixer.component_library.model.candidate.RelevancePromptCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.Compact\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.PromptItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.common.{thriftscala => thriftCommon}\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.service.thriftscala.FeedbackEngagementType\nimport com.twitter.timelineservice.model.FeedbackInfo\nimport com.twitter.timelineservice.model.FeedbackMetadata\nimport com.twitter.timelineservice.suggests.thriftscala.SuggestType\nimport com.twitter.timelineservice.{thriftscala => thrift}\n\nobject RelevancePromptCandidateUrtItemBuilder {\n\n  val ClientEventInfoComponent: String = \"for_you_relevance_prompt\"\n\n  val ClientEventBuilder = ClientEventInfoBuilder[PipelineQuery, RelevancePromptCandidate](\n    component = ClientEventInfoComponent\n  )\n\n  val ConfirmationStr = \"Thank you for your feedback!\"\n}\n\ncase class RelevancePromptCandidateUrtItemBuilder(\n  titleParam: FSParam[String],\n  relevantPromptParam: FSParam[String],\n  notRelevantPromptParam: FSParam[String],\n  neutralPromptParam: FSParam[String],\n) extends CandidateUrtEntryBuilder[\n      PipelineQuery,\n      RelevancePromptCandidate,\n      PromptItem\n    ] {\n\n  import RelevancePromptCandidateUrtItemBuilder._\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: RelevancePromptCandidate,\n    candidateFeatures: FeatureMap\n  ): PromptItem = {\n\n    val servedId: Long = query.features.flatMap(_.getOrElse(ServedIdFeature, None)).getOrElse(0L)\n\n    // Use user 0 as a dummy entity ID until we have a proper session ID to track\n    val feedbackMetadata = FeedbackMetadata(\n      Some(FeedbackEngagementType.RelevancePrompt),\n      entityIds = Seq(thriftCommon.FeedbackEntity.FeedbackId(servedId)),\n      ttl = Some(30.days)\n    )\n\n    val positiveFeedbackUrl = FeedbackInfo.feedbackUrl(\n      feedbackType = thrift.FeedbackType.Relevant,\n      feedbackMetadata = feedbackMetadata,\n      injectionType = Some(SuggestType.RelevancePrompt)\n    )\n    val negativeFeedbackUrl = FeedbackInfo.feedbackUrl(\n      feedbackType = thrift.FeedbackType.NotRelevant,\n      feedbackMetadata = feedbackMetadata,\n      injectionType = Some(SuggestType.RelevancePrompt)\n    )\n    val neutralFeedbackUrl = FeedbackInfo.feedbackUrl(\n      feedbackType = thrift.FeedbackType.Neutral,\n      feedbackMetadata = feedbackMetadata,\n      injectionType = Some(SuggestType.RelevancePrompt)\n    )\n\n    val relevancePromptCandidateUrtItemStringCenterBuilder =\n      RelevancePromptCandidateUrtItemStringCenterBuilder(\n        clientEventInfoBuilder = ClientEventBuilder,\n        titleTextBuilder = StrStatic(query.params(titleParam)),\n        confirmationTextBuilder = StrStatic(ConfirmationStr),\n        isRelevantTextBuilder = StrStatic(query.params(relevantPromptParam)),\n        notRelevantTextBuilder = StrStatic(query.params(notRelevantPromptParam)),\n        displayType = Compact,\n        isRelevantCallback = Callback(positiveFeedbackUrl),\n        notRelevantCallback = Callback(negativeFeedbackUrl),\n        neutralTextBuilder = Some(StrStatic(query.params(neutralPromptParam))),\n        neutralCallback = Some(Callback(neutralFeedbackUrl))\n      )\n\n    relevancePromptCandidateUrtItemStringCenterBuilder(\n      query = query,\n      relevancePromptCandidate = candidate,\n      candidateFeatures = candidateFeatures\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/ReportTweetChildFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorReportTweet\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class ReportTweetChildFeedbackActionBuilder @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings) {\n\n  def apply(\n    candidate: TweetCandidate,\n  ): Option[ChildFeedbackAction] = {\n    Some(\n      ChildFeedbackAction(\n        feedbackType = RichBehavior,\n        prompt = Some(stringCenter.prepare(externalStrings.reportTweetString)),\n        confirmation = None,\n        feedbackUrl = None,\n        hasUndoAction = Some(true),\n        confirmationDisplayType = None,\n        clientEventInfo = Some(\n          ClientEventInfo(\n            component = None,\n            element = Some(\"report_tweet\"),\n            details = None,\n            action = Some(\"click\"),\n            entityToken = None\n          )\n        ),\n        icon = Some(icon.Flag),\n        richBehavior = Some(RichFeedbackBehaviorReportTweet(candidate.id)),\n        subprompt = None\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/RetweeterChildFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.service.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class RetweeterChildFeedbackActionBuilder @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings) {\n\n  def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {\n    val isRetweet = candidateFeatures.getOrElse(IsRetweetFeature, false)\n\n    if (isRetweet) {\n      candidateFeatures.getOrElse(AuthorIdFeature, None).flatMap { retweeterId =>\n        FeedbackUtil.buildUserSeeFewerChildFeedbackAction(\n          userId = retweeterId,\n          namesByUserId = candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String]),\n          promptExternalString = externalStrings.showFewerRetweetsString,\n          confirmationExternalString = externalStrings.showFewerRetweetsConfirmationString,\n          engagementType = t.FeedbackEngagementType.Retweet,\n          stringCenter = stringCenter\n        )\n      }\n    } else None\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/ServedTypeSocialContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityIdFeature\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityNameFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n/**\n * Builds social context for tweets that have a 1:1 relationship\n * between served type and social context. e.g. Lists/Communities\n */\n@Singleton\ncase class ServedTypeSocialContextBuilder @Inject() (\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter])\n    extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {\n\n  private val ListsUrl = \"\"\n  private val CommunitiesUrl = \"\"\n\n  private val stringCenter = stringCenterProvider.get()\n  private val listString = externalStrings.ownedSubscribedListsModuleHeaderString\n  private val popularGeoString = externalStrings.socialContextPopularInYourAreaString\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext] = candidateFeatures.get(ServedTypeFeature) match {\n\n    case t.ServedType.ForYouCommunity =>\n      val communityId = candidateFeatures.getOrElse(CommunityIdFeature, None)\n      val communityName = candidateFeatures.getOrElse(CommunityNameFeature, None)\n      val context = GeneralContext(\n        contextType = CommunityGeneralContextType,\n        text = communityName.getOrElse(\"\"),\n        url = None,\n        contextImageUrls = None,\n        landingUrl = communityId.map(id => Url(ExternalUrl, CommunitiesUrl + id))\n      )\n      Some(context)\n\n    case t.ServedType.ForYouPopularGeo =>\n      val context = GeneralContext(\n        contextType = LocationGeneralContextType,\n        text = stringCenter.prepare(popularGeoString),\n        url = None,\n        contextImageUrls = None,\n        landingUrl = None\n      )\n      Some(context)\n\n    case _ => None\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/TopicSocialContextBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContext\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class TopicSocialContextBuilder @Inject() ()\n    extends BaseSocialContextBuilder[PipelineQuery, TweetCandidate] {\n\n  def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext] = {\n    val inNetwork = candidateFeatures.getOrElse(InNetworkFeature, true)\n    if (!inNetwork) {\n      val topicIdSocialContextOpt = candidateFeatures.getOrElse(TopicIdSocialContextFeature, None)\n      val topicContextFunctionalityTypeOpt =\n        candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None)\n      (topicIdSocialContextOpt, topicContextFunctionalityTypeOpt) match {\n        case (Some(topicId), Some(topicContextFunctionalityType)) =>\n          Some(\n            TopicContext(\n              topicId = topicId.toString,\n              functionalityType = Some(topicContextFunctionalityType)\n            ))\n        case _ => None\n      }\n    } else {\n      None\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/TuneFeedFeedbackActionInfoBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TuneFeedFeedbackActionInfoBuilder @Inject() (\n  postFollowupFeedbackActionBuilder: PostFollowupFeedbackActionBuilder,\n  dontLikeFeedbackActionBuilder: DontLikeFeedbackActionBuilder)\n    extends BaseFeedbackActionInfoBuilder[PipelineQuery, TweetCandidate] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackActionInfo] = {\n    Some(\n      FeedbackActionInfo(\n        feedbackActions = Seq(\n          postFollowupFeedbackActionBuilder(query, candidate, candidateFeatures),\n          dontLikeFeedbackActionBuilder(query, candidate, candidateFeatures)).flatten,\n        feedbackMetadata = None,\n        displayContext = None,\n        clientEventInfo = None\n      ))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder/UnfollowUserChildFeedbackActionBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.decorator.urt.builder\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleFollowUser\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class UnfollowUserChildFeedbackActionBuilder @Inject() (\n  @ProductScoped stringCenter: StringCenter,\n  externalStrings: HomeMixerExternalStrings) {\n\n  def apply(candidateFeatures: FeatureMap): Option[ChildFeedbackAction] = {\n    val isInNetwork = candidateFeatures.getOrElse(InNetworkFeature, false)\n    val userIdOpt = candidateFeatures.getOrElse(AuthorIdFeature, None)\n\n    if (isInNetwork) {\n      userIdOpt.flatMap { userId =>\n        val screenNamesMap =\n          candidateFeatures.getOrElse(ScreenNamesFeature, Map.empty[Long, String])\n        val userScreenNameOpt = screenNamesMap.get(userId)\n        userScreenNameOpt.map { userScreenName =>\n          val prompt = stringCenter.prepare(\n            externalStrings.unfollowUserString,\n            Map(\"username\" -> userScreenName)\n          )\n          val confirmation = stringCenter.prepare(\n            externalStrings.unfollowUserConfirmationString,\n            Map(\"username\" -> userScreenName)\n          )\n          ChildFeedbackAction(\n            feedbackType = RichBehavior,\n            prompt = Some(prompt),\n            confirmation = Some(confirmation),\n            feedbackUrl = None,\n            hasUndoAction = Some(true),\n            confirmationDisplayType = None,\n            clientEventInfo = None,\n            icon = Some(icon.Unfollow),\n            richBehavior = Some(RichFeedbackBehaviorToggleFollowUser(userId)),\n            subprompt = None\n          )\n        }\n      }\n    } else None\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/AncestorFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.tweetconvosvc.tweet_ancestor.{thriftscala => ta}\nimport com.twitter.tweetconvosvc.{thriftscala => tcs}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AncestorFeatureHydrator @Inject() (\n  conversationServiceClient: tcs.ConversationService.MethodPerEndpoint)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"Ancestor\")\n\n  override val features: Set[Feature[_, _]] = Set(AncestorsFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val candidatesWithReplies = candidates.collect {\n      case candidate if candidate.features.getOrElse(InReplyToTweetIdFeature, None).isDefined =>\n        candidate.candidate.id\n    }\n    val candidateIsReplyIndexMap = candidatesWithReplies.zipWithIndex.toMap\n    val ancestorsRequest = tcs.GetAncestorsRequest(candidatesWithReplies)\n    conversationServiceClient.getAncestors(ancestorsRequest).map { getAncestorsResponse =>\n      candidates.map { candidate =>\n        val resultIndex = candidateIsReplyIndexMap.get(candidate.candidate.id)\n        val ancestors = resultIndex\n          .map { index =>\n            getAncestorsResponse.ancestors(index) match {\n              case tcs.TweetAncestorsResult.TweetAncestors(ancestorsResult)\n                  if ancestorsResult.nonEmpty =>\n                ancestorsResult.head.ancestors ++ getTruncatedRootTweet(ancestorsResult.head)\n              case _ => Seq.empty\n            }\n          }.getOrElse(Seq.empty)\n        FeatureMap(AncestorsFeature, ancestors)\n      }\n    }\n  }\n\n  private def getTruncatedRootTweet(\n    ancestors: ta.TweetAncestors,\n  ): Option[ta.TweetAncestor] = {\n    ancestors.conversationRootAuthorId.collect {\n      case rootAuthorId\n          if ancestors.state == ta.ReplyState.Partial &&\n            ancestors.ancestors.last.tweetId != ancestors.conversationId =>\n        ta.TweetAncestor(ancestors.conversationId, rootAuthorId)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/AuthorFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.author_features.AuthorFeaturesAdapter\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.AuthorFeatureRepository\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.servo.repository.KeyValueResult\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.author_features.v1.{thriftjava => af}\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject AuthorFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass AuthorFeatureHydrator @Inject() (\n  @Named(AuthorFeatureRepository) client: KeyValueRepository[Seq[Long], Long, af.AuthorFeatures],\n  override val statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"AuthorFeature\")\n\n  override val features: Set[Feature[_, _]] = Set(AuthorFeature)\n\n  override val statScope: String = identifier.toString\n\n  private val DefaultDataRecord = new DataRecord()\n\n  private val DefaultFeatureMap = FeatureMap(AuthorFeature, DefaultDataRecord)\n\n  private val authorIdsCountStat = statsReceiver.scope(statScope).stat(\"authorIdsSize\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val inNetworkCandidates =\n      candidates.filter(_.features.getOrElse(FromInNetworkSourceFeature, false))\n    val possiblyAuthorIds = inNetworkCandidates.map(extractKey)\n    val authorIds = possiblyAuthorIds.flatten.distinct\n    authorIdsCountStat.add(authorIds.size)\n\n    val response: Future[KeyValueResult[Long, DataRecord]] =\n      if (authorIds.nonEmpty)\n        client(authorIds)\n          .map {\n            _.mapFound {\n              AuthorFeaturesAdapter.adaptToDataRecords(_).asScala.head\n            }\n          }\n      else Future.value(KeyValueResult.empty)\n\n    response.map { result =>\n      candidates.map { candidate =>\n        val authorId = extractKey(candidate)\n        val authorDR = observedGet(key = authorId, keyValueResult = result)\n        authorDR.toOption\n          .flatMap {\n            _.map { features =>\n              FeatureMap(AuthorFeature, features)\n            }\n          }.getOrElse(DefaultFeatureMap)\n      }\n    }\n  }\n\n  private def extractKey(\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Option[Long] = {\n    CandidatesUtil.getOriginalAuthorId(candidate.features)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/AuthorLargeEmbeddingsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeLargeEmbeddingsFeatures.AuthorLargeEmbeddingsFeature\nimport com.twitter.home_mixer.model.HomeLargeEmbeddingsFeatures.AuthorLargeEmbeddingsKeyFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableLargeEmbeddingsFeatureHydrationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelNameParam\nimport com.twitter.home_mixer_features.{thriftscala => hmf}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.large_embeddings.AuthorLargeEmbeddingsAdapter\nimport com.twitter.timelines.prediction.adapters.large_embeddings.HashingFeatureParams\nimport com.twitter.timelines.prediction.adapters.large_embeddings.HomeMixerLargeEmbeddingsFeatureHydrator\nimport com.twitter.timelines.prediction.adapters.large_embeddings.LargeEmbeddingsAdapter\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AuthorLargeEmbeddingsFeatureHydrator @Inject() (\n  statsReceiver: StatsReceiver,\n  override val homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery]\n    with HomeMixerLargeEmbeddingsFeatureHydrator {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"AuthorLargeEmbeddings\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(AuthorLargeEmbeddingsFeature, AuthorLargeEmbeddingsKeyFeature)\n\n  override val adapter: LargeEmbeddingsAdapter = AuthorLargeEmbeddingsAdapter\n\n  override val cacheType: hmf.Cache = hmf.Cache.AuthorLargeEmbeddings\n\n  override val scopedStatsReceiver: StatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableLargeEmbeddingsFeatureHydrationParam)\n\n  // Hashing Features\n  override val defaultHashingFeatureParams: HashingFeatureParams = HashingFeatureParams(\n    scales = Seq(3384241453L, 3372414709L),\n    biases = Seq(1649585795L, 3131243219L),\n    modulus = 3957384397L,\n    bucketSize = 3000000L,\n  )\n\n  override val modelName2HashingFeatureParams: Map[String, HashingFeatureParams] = Map(\n    \"hr_video_prod__v3_realtime\" -> HashingFeatureParams(\n      scales = Seq(113294449L, 601841083L),\n      biases = Seq(2231299001L, 841367196L),\n      modulus = 2343760591L,\n      bucketSize = 3000000L,\n    ),\n    \"hr_video_prod__v2_lembeds\" -> HashingFeatureParams(\n      scales = Seq(787140070L, 633713480L),\n      biases = Seq(427768658L, 911091889L),\n      modulus = 2888480981L,\n      bucketSize = 300000L,\n    ),\n    \"hr_prod__v4_embeds_230M\" -> HashingFeatureParams(\n      scales = Seq(371965780L, 328930218L),\n      biases = Seq(139686260L, 37755056L),\n      modulus = 631860353L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_prod__v5_embeds_230M_and_transformer\" -> HashingFeatureParams(\n      scales = Seq(371965780L, 328930218L),\n      biases = Seq(139686260L, 37755056L),\n      modulus = 631860353L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_prod__v5_watchtime\" -> HashingFeatureParams(\n      scales = Seq(2328530078L, 2844016377L),\n      biases = Seq(1352496802L, 3011003330L),\n      modulus = 3979826519L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_prod__v6_transformer_v2\" -> HashingFeatureParams(\n      scales = Seq(371965780L, 328930218L),\n      biases = Seq(139686260L, 37755056L),\n      modulus = 631860353L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_prod__v6_mixed_training\" -> HashingFeatureParams(\n      scales = Seq(371965780L, 328930218L),\n      biases = Seq(139686260L, 37755056L),\n      modulus = 631860353L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_prod__v6_transformer_v2_kafka_merge_join\" -> HashingFeatureParams(\n      scales = Seq(371965780L, 328930218L),\n      biases = Seq(139686260L, 37755056L),\n      modulus = 631860353L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_prod__v6_transformer_v2_realtime_debias_21apr\" -> HashingFeatureParams(\n      scales = Seq(371965780L, 328930218L),\n      biases = Seq(139686260L, 37755056L),\n      modulus = 631860353L,\n      bucketSize = 30000000L,\n    ),\n  )\n  private val batchSize = 25\n\n  private def getBatchedFeatureMap(\n    modelName: String,\n    candidatesBatch: Seq[CandidateWithFeatures[TweetCandidate]],\n  ): Future[Seq[FeatureMap]] = {\n    val authorIds =\n      candidatesBatch.map { candidate =>\n        candidate.features.getOrElse(AuthorIdFeature, None).getOrElse(0L)\n      }\n\n    getLargeEmbeddings(authorIds, modelName).map { responses =>\n      responses.map { largeEmbeddingResponse =>\n        FeatureMapBuilder()\n          .add(AuthorLargeEmbeddingsFeature, largeEmbeddingResponse.dataRecord)\n          .add(AuthorLargeEmbeddingsKeyFeature, largeEmbeddingResponse.hashedKeys)\n          .build()\n      }\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val modelName = query.params(ModelNameParam)\n    OffloadFuturePools.offloadBatchSeqToFutureSeq(\n      candidates,\n      getBatchedFeatureMap(modelName, _),\n      batchSize\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/knuddels/jtokkit\",\n        \"3rdparty/jvm/io/grpc:grpc-protobuf\",\n        \"3rdparty/jvm/org/scalanlp:breeze\",\n        \"hmli/hss/src/main/thrift/com/twitter/hss:thrift-scala\",\n        \"home-mixer-features/thrift/src/main/thrift:thrift-java\",\n        \"home-mixer-features/thrift/src/main/thrift:thrift-scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/author_features\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/content\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/gizmoduck_features\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/inferred_topic\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/simclusters_features\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/transformer_embeddings\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/twhin_embeddings\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/module\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie/content\",\n        \"joinkey/src/main/scala/com/twitter/joinkey/context\",\n        \"joinkey/src/main/thrift/com/twitter/joinkey/context:joinkey-context-scala\",\n        \"language/thrift:types-lib-scala\",\n        \"limiter/thrift-only/src/main/thrift:thrift-scala\",\n        \"media-understanding/embeddings/src/main/thrift/com/twitter/media-understanding/embeddings:thrift-scala\",\n        \"media-understanding/video-summary/thrift/src/main/thrift:thrift-scala\",\n        \"periscope/api-proxy-thrift/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/recommendations\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/location\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_is_nsfw\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_visibility_reason\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"representation-scorer/server/src/main/scala/com/twitter/representationscorer/common\",\n        \"search/search-router/thrift/src/main/thrift:thrift-scala\",\n        \"servo/repo/src/main/scala\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/ml/api/constant\",\n        \"src/java/com/twitter/search/common/util/lang\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/scala/com/twitter/suggests/controller_data\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/large_embeddings\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/real_graph\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/realtime_interaction_graph\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/request_context\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/twistly\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/two_hop_features\",\n        \"src/scala/com/twitter/timelines/prediction/common/util\",\n        \"src/scala/com/twitter/timelines/prediction/features/common\",\n        \"src/scala/com/twitter/timelines/prediction/features/location\",\n        \"src/scala/com/twitter/timelines/prediction/features/realtime_interaction_graph\",\n        \"src/scala/com/twitter/timelines/prediction/features/simcluster\",\n        \"src/scala/com/twitter/timelines/prediction/features/time_features\",\n        \"src/scala/com/twitter/topic_recos/common\",\n        \"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"src/thrift/com/twitter/timelines/content_understanding/user_interests:user_interests-scala\",\n        \"src/thrift/com/twitter/timelines/impression_bloom_filter:thrift-scala\",\n        \"src/thrift/com/twitter/user_session_store:thrift-java\",\n        \"strato/config/columns/analytics/video:video-strato-client\",\n        \"strato/config/columns/audiencerewards/audienceRewardsService:getSuperFollowEligibility-strato-client\",\n        \"strato/config/columns/content_understanding:content_understanding-strato-client\",\n        \"strato/config/columns/content_understanding/internal/manhattan:manhattan-strato-client\",\n        \"strato/config/columns/events/experiences/basketball:basketball-strato-client\",\n        \"strato/config/columns/events/urt:urt-strato-client\",\n        \"strato/config/columns/geo/service:service-strato-client\",\n        \"strato/config/columns/heartbeat_optimizer:heartbeat_optimizer-strato-client\",\n        \"strato/config/columns/hss/user_scores/api:api-strato-client\",\n        \"strato/config/columns/language/tweet:tweet-strato-client\",\n        \"strato/config/columns/language/user:user-strato-client\",\n        \"strato/config/columns/ml/featureStore:featureStore-strato-client\",\n        \"strato/config/columns/periscope:periscope-strato-client\",\n        \"strato/config/columns/recommendations/interaction_graph/on_prem:on-prem-interaction-graph-user-features-strato-client\",\n        \"strato/config/columns/recommendations/simclusters_v2:simclusters_v2-strato-client\",\n        \"strato/config/columns/recommendations/user-signal-service:user-signal-service-strato-client\",\n        \"strato/config/columns/searchai/grok:grok-strato-client\",\n        \"strato/config/columns/searchai/storage:storage-strato-client\",\n        \"strato/config/columns/subscription-services/subscription-verification:subscription-verification-strato-client\",\n        \"strato/config/columns/trends/trip:trip-strato-client\",\n        \"strato/config/columns/tweetypie/federated:federated-strato-client\",\n        \"strato/config/columns/tweetypie/managed:managed-strato-client\",\n        \"strato/config/columns/unified-counter/service:service-strato-client\",\n        \"strato/config/columns/user-history-transformer/user-actions:user-actions-strato-client\",\n        \"strato/config/columns/videoRecommendations/twitterClip:twitterClip-strato-client\",\n        \"strato/config/columns/viewcounts:viewcounts-strato-client\",\n        \"strato/config/src/thrift/com/twitter/strato/columns/heartbeat_optimizer:heartbeat_optimizer-scala\",\n        \"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/feedback\",\n        \"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan\",\n        \"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/persistence\",\n        \"timelines/src/main/scala/com/twitter/timelines/clients/strato/topics\",\n        \"timelines/src/main/scala/com/twitter/timelines/clients/strato/twistly\",\n        \"timelineservice/common/src/main/scala/com/twitter/timelineservice/model\",\n        \"topic-social-proof/server/src/main/scala/com/twitter/tsp/stores\",\n        \"topic-social-proof/server/src/main/thrift:thrift-scala\",\n        \"topiclisting/topiclisting-core/src/main/scala/com/twitter/topiclisting\",\n        \"tweetconvosvc/thrift/src/main/thrift:thrift-scala\",\n        \"tweetsource/common/src/main/thrift:thrift-scala\",\n        \"unified_user_actions/thrift/src/main/thrift/com/twitter/unified_user_actions:unified_user_actions-scala\",\n        \"user-signal-service/thrift/src/main/thrift:thrift-scala\",\n        \"user_history_transformer/common/src/main/scala/com/twitter/user_history_transformer/common\",\n        \"user_history_transformer/common/src/main/scala/com/twitter/user_history_transformer/util\",\n        \"user_history_transformer/domain/src/main/scala:aggregation\",\n        \"user_history_transformer/domain/src/main/scala:backbone\",\n        \"user_history_transformer/domain/src/main/scala:user-history\",\n        \"user_history_transformer/service/src/main/java/com/x/user_action_sequence\",\n        \"user_history_transformer/thrift/src/main/thrift/com/x/user_action_sequence:user_action_sequence-scala\",\n    ],\n    exports = [\n        \"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala\",\n        \"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/BasketballContextFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.BasketballContextFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.BasketballTeamAccountIdsParam\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport com.twitter.strato.catalog.Fetch\nimport com.twitter.strato.generated.client.events.experiences.basketball.PostBasketballContextClientColumn\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass BasketballContextFeatureHydrator @Inject() (\n  postBasketballContextClientColumn: PostBasketballContextClientColumn\n) extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"BasketballContext\")\n\n  override val features: Set[Feature[_, _]] = Set(BasketballContextFeature)\n\n  private val fetcher: Fetcher[Long, Unit, urt.BasketballContext] = postBasketballContextClientColumn.fetcher\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    val basketballAuthorIds = query.params(BasketballTeamAccountIdsParam)\n\n    Stitch.collect {\n      candidates.map { candidate =>\n        val servedType = candidate.features.getOrElse(ServedTypeFeature, hmt.ServedType.Undefined)\n        val isPromoted = (servedType == hmt.ServedType.ForYouPromoted || servedType == hmt.ServedType.FollowingPromoted)\n\n        val authorId = candidate.features.getOrElse(AuthorIdFeature, None)\n        val isBasketballAuthor = authorId.exists(id => basketballAuthorIds.contains(id))\n\n        // Skip hydration if the post is an ad or not from a basketball account\n        if (isPromoted || !isBasketballAuthor) {\n          Stitch.value(FeatureMapBuilder().add(BasketballContextFeature, None).build())\n        } else {\n          fetcher.fetch(candidate.candidate.id, Unit).map {\n            case Fetch.Result(Some(basketballContext), _) =>\n              FeatureMapBuilder().add(BasketballContextFeature, Some(basketballContext)).build()\n            case _ =>\n              FeatureMapBuilder().add(BasketballContextFeature, None).build()\n          }\n        }\n      }\n    }\n  }\n} \n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/BroadcastStateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.TweetUrlsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.Fetch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.periscope.CoreOnBroadcastClientColumn\nimport com.twitter.ubs.{thriftscala => ubs}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject BroadcastStateFeature extends Feature[TweetCandidate, Option[ubs.BroadcastState]]\n\n@Singleton\nclass BroadcastStateFeatureHydrator @Inject() (\n  coreOnBroadcastClientColumn: CoreOnBroadcastClientColumn)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with WithDefaultFeatureMap {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"BroadcastState\")\n\n  override val features: Set[Feature[_, _]] = Set(BroadcastStateFeature)\n\n  private val pattern = \"\".r\n\n  private val fetcher: Fetcher[String, Unit, ubs.Broadcast] = coreOnBroadcastClientColumn.fetcher\n\n  override val defaultFeatureMap: FeatureMap = FeatureMap(BroadcastStateFeature, None)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    val broadcastIdMap = candidates.flatMap { candidate =>\n      candidate.features\n        .getOrElse(TweetUrlsFeature, Seq.empty)\n        .collectFirst { case pattern(broadcastId) => broadcastId }\n        .map(broadcastId => candidate.candidate.id -> broadcastId)\n    }.toMap\n\n    val responses = Stitch.collect {\n      broadcastIdMap.values.toSeq.distinct.map { broadcastId =>\n        fetcher.fetch(broadcastId).map {\n          case Fetch.Result(Some(broadcast), _) if broadcast.broadcastId.nonEmpty =>\n            Some(broadcastId -> broadcast)\n          case _ => None\n        }\n      }\n    }\n\n    responses.map { results =>\n      val broadcastMap = results.flatten.toMap\n      candidates.map { candidate =>\n        val broadcastState = broadcastIdMap.get(candidate.candidate.id).flatMap { broadcastId =>\n          broadcastMap.get(broadcastId).flatMap(_.state)\n        }\n        FeatureMapBuilder().add(BroadcastStateFeature, broadcastState).build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/CategoryDiversityRescoringFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.simclusters_features.SimclustersFeaturesAdapter.SimclustersSparseTweetEmbeddingsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.CategoryDiversityRescoringWeightParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.CategoryDiversityKParam\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport scala.jdk.CollectionConverters._\n\nobject CategoryDiversityRescoringFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"CategoryDiversityRescoring\")\n\n  override val features: Set[Feature[_, _]] = Set(ScoreFeature)\n\n  final val EmptyDataRecord = new DataRecord()\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offload {\n\n    diversityPenalty(query, candidates).map(score => FeatureMap(ScoreFeature, Some(score)))\n  }\n\n  private def diversityPenalty(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n  ): Seq[Double] = {\n    val n = candidates.length\n    val selected = scala.collection.mutable.Set[Int]()\n    val newScores = Array.fill(n)(0.0)\n    val weight = query.params(CategoryDiversityRescoringWeightParam)\n    val k = query.params(CategoryDiversityKParam)\n\n    val topKCategories = getCandidateCategory(k, candidates)\n    val clusterCntMap = scala.collection.mutable.Map[String, Int]()\n\n    val candidatesWithIndexWithCategories: Seq[\n      (CandidateWithFeatures[TweetCandidate], Int, Seq[String])\n    ] = candidates.zipWithIndex.zip(topKCategories).map {\n      case ((candidate, index), categories) =>\n        (candidate, index, categories)\n    }\n    for (i <- 0 until n) {\n      var maxScore = Double.NegativeInfinity\n      var bestCandidateIndex = -1\n      var candidateCategories: Seq[String] = Seq.empty\n\n      for ((candidate, index, categories) <- candidatesWithIndexWithCategories) {\n        if (!selected.contains(index)) {\n          val relevance = candidate.features.getOrElse(ScoreFeature, None).getOrElse(0.0)\n          var penalty = 0.0\n          var totalCategoryCnt = 0\n          \n          categories.foreach { category =>\n            val cnt = clusterCntMap.getOrElse(category, 0)\n            penalty += math.log(cnt + 1) / math.log(2)\n          }\n\n          val score = math.max(relevance - weight * penalty, 0.00001)\n\n          if (score > maxScore) {\n            maxScore = score\n            bestCandidateIndex = index\n            candidateCategories = categories\n          }\n        }\n      }\n      if (bestCandidateIndex != -1) {\n        selected += bestCandidateIndex\n        newScores(bestCandidateIndex) = maxScore\n        candidateCategories.foreach { category =>\n          clusterCntMap.put(category, clusterCntMap.getOrElse(category, 0) + 1)\n        }\n      }\n    }\n\n    newScores.toSeq\n  }\n\n  private def getCandidateCategory(\n    k: Int,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Seq[String]] = {\n    val topKClusters: Seq[Seq[String]] = candidates.map { candidate =>\n      val topKClustersMap =\n        Option(\n          candidate.features\n            .getOrElse(SimClustersLogFavBasedTweetFeature, new DataRecord())\n            .getSparseContinuousFeatures\n        ).flatMap(mapOpt =>\n          Option(mapOpt.get(SimclustersSparseTweetEmbeddingsFeature.getFeatureId)))\n      topKClustersMap\n        .map(_.asScala.toSeq.sortBy(-_._2).take(k).map(_._1))\n        .getOrElse(Seq.empty)\n    }\n    topKClusters\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/ClipEmbeddingFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.content.ClipEmbeddingFeaturesAdapter\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableClipEmbeddingFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableVideoClipEmbeddingFeatureHydrationDeciderParam\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.videoRecommendations.twitterClip.TwitterClipEmbeddingMhClientColumn\nimport com.twitter.util.logging.Logging\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\n@Singleton\nclass ClipEmbeddingFeatureHydrator @Inject() (\n  twitterClipEmbeddingMhClientColumn: TwitterClipEmbeddingMhClientColumn,\n  statsReceiver: StatsReceiver)\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery]\n    with Logging {\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableClipEmbeddingFeaturesParam) &&\n      query.params(EnableVideoClipEmbeddingFeatureHydrationDeciderParam)\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"ClipEmbedding\")\n\n  private val fetcher: Fetcher[Long, Unit, Seq[Double]] =\n    twitterClipEmbeddingMhClientColumn.fetcher\n\n  override val features: Set[Feature[_, _]] = Set(ClipEmbeddingFeature)\n\n  private val DefaultFeatureMap =\n    FeatureMapBuilder().add(ClipEmbeddingFeature, new DataRecord()).build()\n\n  private val scopedStatsReceiver: StatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val clipEmbeddingNotFoundCounter = scopedStatsReceiver.counter(\"clipEmbeddingNotFound\")\n  private val clipEmbeddingFoundCounter = scopedStatsReceiver.counter(\"clipEmbeddingFound\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    fetcher\n      .fetch(candidate.id).map { manhattanResult =>\n        manhattanResult.v match {\n          case Some(embeddings) =>\n            clipEmbeddingFoundCounter.incr()\n            val dataRecord =\n              ClipEmbeddingFeaturesAdapter.adaptToDataRecords(embeddings).asScala.head\n            FeatureMapBuilder().add(ClipEmbeddingFeature, dataRecord).build()\n          case None =>\n            clipEmbeddingNotFoundCounter.incr()\n            DefaultFeatureMap\n        }\n      }.onFailure(e => {\n        error(s\"Error fetching VideoClipEmbedding: $e\")\n      })\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/ClipEmbeddingMediaUnderstandingFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.content.ClipEmbeddingFeaturesAdapter\nimport com.twitter.home_mixer.model.HomeFeatures.MediaCategoryFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MediaIdFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableClipEmbeddingMediaUnderstandingFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableVideoClipEmbeddingMediaUnderstandingFeatureHydrationDeciderParam\nimport com.twitter.media_understanding.embeddings.thriftscala.MediaEmbedding\nimport com.twitter.media_understanding.embeddings.thriftscala.MediaEmbeddingInfo\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject ClipEmbeddingFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass ClipEmbeddingMediaUnderstandingFeatureHydrator @Inject() (\n  stratoClient: Client,\n  statsReceiver: StatsReceiver)\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery]\n    with Logging {\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableClipEmbeddingMediaUnderstandingFeaturesParam) &&\n      query.params(EnableVideoClipEmbeddingMediaUnderstandingFeatureHydrationDeciderParam)\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"ClipEmbeddingMediaUnderstanding\")\n\n  val MediaModelName: String = \"twitter_clip_256\"\n  val MediaVersion: String = \"0\"\n\n  val mediaEmbeddingFetcher: Fetcher[(String, (String, String)), Unit, MediaEmbeddingInfo] =\n    stratoClient.fetcher[(String, (String, String)), Unit, MediaEmbeddingInfo](\n      \"media-understanding/embeddings/prod/embeddings\"\n    )\n\n  def getTweetMediaEmbeddings(key: String): Stitch[Option[Seq[Double]]] = {\n    mediaEmbeddingFetchCounter.incr()\n    mediaEmbeddingFetcher\n      .fetch((key, (MediaModelName, MediaVersion)))\n      .map { result =>\n        result.v match {\n          case Some(mediaData) =>\n            mediaData.embedding match {\n              case Some(MediaEmbedding.DoubleVector(doubles)) =>\n                mediaEmbeddingFetchSuccessCounter.incr()\n                Some(doubles.map(_.toDouble))\n              case _ =>\n                mediaEmbeddingFetchFailureCounter.incr()\n                None\n            }\n          case None =>\n            mediaEmbeddingFetchFailureCounter.incr()\n            None\n        }\n      }\n      .rescue {\n        case e: Exception =>\n          mediaEmbeddingFetchFailureCounter.incr()\n          Stitch.None\n      }\n  }\n\n  def getClipEmbeddings(\n    tweetId: Long,\n    existingFeatures: FeatureMap\n  ): Stitch[Option[Seq[Double]]] = {\n    val mediaId = existingFeatures.getOrElse(MediaIdFeature, None)\n    val mediaCategory = existingFeatures.getOrElse(MediaCategoryFeature, None)\n\n    (mediaId, mediaCategory) match {\n      case (Some(id), Some(category)) =>\n        val key = s\"${category.getValue}/$id\"\n        getTweetMediaEmbeddings(key).map {\n          case Some(embeddings) => Some(embeddings)\n          case _ =>\n            clipEmbeddingNotFoundNoneCounter.incr()\n            None\n        }\n      case _ => Stitch.value(None)\n    }\n  }\n\n  override val features: Set[Feature[_, _]] = Set(ClipEmbeddingFeature)\n\n  private val DefaultFeatureMap =\n    FeatureMapBuilder().add(ClipEmbeddingFeature, new DataRecord()).build()\n\n  private val scopedStatsReceiver: StatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val clipEmbeddingNotFoundNoneCounter =\n    scopedStatsReceiver.counter(\"clipEmbeddingNotFoundNoneCounter\")\n  private val clipEmbeddingNotFoundCounter = scopedStatsReceiver.counter(\"clipEmbeddingNotFound\")\n  private val clipEmbeddingFoundCounter = scopedStatsReceiver.counter(\"clipEmbeddingFound\")\n\n  val mediaEmbeddingFetchCounter = statsReceiver.counter(\"media_embedding_fetch_total\")\n  val mediaEmbeddingFetchSuccessCounter = statsReceiver.counter(\"media_embedding_fetch_success\")\n  val mediaEmbeddingFetchFailureCounter = statsReceiver.counter(\"media_embedding_fetch_failure\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    getClipEmbeddings(candidate.id, existingFeatures)\n      .map {\n        case Some(embeddings) =>\n          clipEmbeddingFoundCounter.incr()\n          val dataRecord =\n            ClipEmbeddingFeaturesAdapter.adaptToDataRecords(embeddings).asScala.head\n          FeatureMapBuilder().add(ClipEmbeddingFeature, dataRecord).build()\n        case None =>\n          clipEmbeddingNotFoundCounter.incr()\n          DefaultFeatureMap\n      }.onFailure {\n        case e: Exception => error(s\"Error fetching VideoClipEmbedding: $e\")\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/ClipImageClusterIdFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.ClipImageClusterIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaIdsFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.ImageClipClusterIdInMemCache\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.InProcessCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.videoRecommendations.twitterClip.TwitterClipImageClusterIdMh95ClientColumn\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass ClipImageClusterIdFeatureHydrator @Inject() (\n  twitterClipImageClusterIdMh95ClientColumn: TwitterClipImageClusterIdMh95ClientColumn,\n  @Named(ImageClipClusterIdInMemCache) imageClipClusterIdInMemCache: InProcessCache[\n    Long,\n    Option[Option[Long]]\n  ],\n  statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"ClipImageClusterId\")\n\n  override val features: Set[Feature[_, _]] = Set(ClipImageClusterIdsFeature)\n\n  private val clusterIdFetcher = twitterClipImageClusterIdMh95ClientColumn.fetcher\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  private val cacheHitCounter = scopedStatsReceiver.counter(\"cache/hit\")\n  private val cacheMissCounter = scopedStatsReceiver.counter(\"cache/miss\")\n  private val fetchExceptionCounter =\n    scopedStatsReceiver.counter(\"getFromCacheOrFetch/exception\")\n\n  private def getFromCacheOrFetch(mediaId: Long): Stitch[Option[Option[Long]]] = {\n    imageClipClusterIdInMemCache\n      .get(mediaId)\n      .map { cachedValue =>\n        cacheHitCounter.incr()\n        Stitch.value(cachedValue)\n      }.getOrElse {\n        cacheMissCounter.incr()\n        clusterIdFetcher\n          .fetch(mediaId)\n          .map(_.v)\n          .liftToOption()\n          .flatMap { clusterIdOpt =>\n            imageClipClusterIdInMemCache.set(mediaId, clusterIdOpt)\n            Stitch.value(clusterIdOpt)\n          }\n      }.handle {\n        case _: Exception =>\n          fetchExceptionCounter.incr()\n          None\n      }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    Stitch.collect(\n      candidates.map { candidate =>\n        val mediaIds = candidate.features.getOrElse(TweetMediaIdsFeature, Seq.empty[Long])\n        val items: Seq[Stitch[Option[(Long, Long)]]] = mediaIds.map { mediaId =>\n          getFromCacheOrFetch(mediaId).map {\n            case Some(Some(mediaClusterId)) =>\n              keyFoundCounter.incr()\n              Some(mediaId -> mediaClusterId)\n            case _ =>\n              keyNotFoundCounter.incr()\n              None\n          }\n        }\n\n        Stitch.collect(items).map { results =>\n          val mediaClusterMap = results.flatten.toMap\n          FeatureMap(ClipImageClusterIdsFeature, mediaClusterMap)\n        }\n      }\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/DependentBulkCandidateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\n\ncase class DependentBulkCandidateFeatureHydrator[Query <: PipelineQuery](\n  parentCandidateFeatureHydrator: BulkCandidateFeatureHydrator[Query, TweetCandidate],\n  childrenCandidateFeatureHydrators: Seq[\n    BulkCandidateFeatureHydrator[Query, TweetCandidate] with WithDefaultFeatureMap\n  ]) extends BulkCandidateFeatureHydrator[Query, TweetCandidate]\n    with Conditionally[Query] {\n\n  override final val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"Dependent\" +\n      parentCandidateFeatureHydrator.identifier.name)\n\n  override def onlyIf(query: Query): Boolean = {\n    parentCandidateFeatureHydrator match {\n      case candidateHydrator: BulkCandidateFeatureHydrator[_, _] with Conditionally[Query] =>\n        candidateHydrator.onlyIf(query)\n      case _ => true\n    }\n  }\n\n  override val features: Set[Feature[_, _]] =\n    childrenCandidateFeatureHydrators.foldLeft(parentCandidateFeatureHydrator.features) {\n      (features, childFeatureHydrator) => features ++ childFeatureHydrator.features\n    }\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] =\n    OffloadFuturePools.offloadStitch {\n      val parentFeatureMapsStitch = parentCandidateFeatureHydrator.apply(query, candidates)\n      parentFeatureMapsStitch.flatMap { parentFeatureMaps =>\n        val updatedCandidates = candidates.zip(parentFeatureMaps).map {\n          case (tweetCandidate, featureMap) =>\n            new CandidateWithFeatures[TweetCandidate] {\n              override val candidate = tweetCandidate.candidate\n              override val features = tweetCandidate.features ++ featureMap\n            }\n        }\n        val (childrenCandidateFeatureHydratorsFiltered, childrenCandidateFeatureHydratorsDefault) =\n          childrenCandidateFeatureHydrators.partition {\n            case candidateFeatureHydrator: BulkCandidateFeatureHydrator[_, _] with Conditionally[\n                  Query\n                ] =>\n              candidateFeatureHydrator.onlyIf(query)\n\n            case _: BulkCandidateFeatureHydrator[_, _] =>\n              true\n\n            case _ => false\n          }\n        val childrenFeatureMapsDefault = childrenCandidateFeatureHydratorsDefault.map {\n          featureHydrator => Seq.fill(candidates.size)(featureHydrator.defaultFeatureMap)\n        }\n        val childrenFeatureMapsStitch = Stitch.traverse(childrenCandidateFeatureHydratorsFiltered) {\n          _.apply(query, updatedCandidates)\n        }\n        childrenFeatureMapsStitch.map { childrenFeatureMaps =>\n          val allFeatureMaps =\n            (parentFeatureMaps +: childrenFeatureMaps) ++ childrenFeatureMapsDefault\n          allFeatureMaps.transpose.map(FeatureMap.merge(_))\n        }\n      }\n    }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/DismissInfoQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.DismissInfoFeature\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.timelinemixer.clients.manhattan.InjectionHistoryClient\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelinemixer.clients.manhattan.DismissInfo\nimport com.twitter.timelineservice.suggests.thriftscala.SuggestType\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject DismissInfoQueryFeatureHydrator {\n  val DismissInfoSuggestTypes = Seq(SuggestType.WhoToFollow)\n}\n\n@Singleton\ncase class DismissInfoQueryFeatureHydrator @Inject() (\n  dismissInfoClient: InjectionHistoryClient)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"DismissInfo\")\n\n  override val features: Set[Feature[_, _]] = Set(DismissInfoFeature)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] =\n    Stitch.callFuture {\n      dismissInfoClient\n        .readDismissInfoEntries(\n          query.getRequiredUserId,\n          DismissInfoQueryFeatureHydrator.DismissInfoSuggestTypes).map { response =>\n          val dismissInfoMap = response.mapValues(DismissInfo.fromThrift)\n          FeatureMapBuilder().add(DismissInfoFeature, dismissInfoMap).build()\n        }\n    }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8, 50, 60, 60)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/DiversityRescoringFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport breeze.linalg._\nimport breeze.numerics.sqrt\n//import com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinEmbeddingsFeatures.TwhinTweetEmbeddingsFeature\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.TransformerEmbeddingsFeatures.PostTransformerEmbeddingsJointBlueFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.TwhinDiversityRescoringWeightParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.TwhinDiversityRescoringRatioParam\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport scala.jdk.CollectionConverters.collectionAsScalaIterableConverter\n\nobject DiversityRescoringFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"DiversityRescoring\")\n\n  override val features: Set[Feature[_, _]] = Set(ScoreFeature)\n\n  final val EmptyDataRecord = new DataRecord()\n\n  private val embeddingsSize = 128\n\n  private val defaultEmbeddings = Seq.fill(embeddingsSize)(0.0)\n\n  private val defaultDenseVector =\n    DenseVector(Array.fill(embeddingsSize)(1.0)) / sqrt(embeddingsSize)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offload {\n    val embeddings = candidates.map { candidate =>\n      val embeddingsTensor =\n        Option(\n          candidate.features\n            .getOrElse(TransformerPostEmbeddingJointBlueFeature, EmptyDataRecord)\n            .getTensors)\n          .flatMap(tensorsOpt =>\n            Option(tensorsOpt.get(PostTransformerEmbeddingsJointBlueFeature.getFeatureId)))\n      embeddingsTensor\n        .map(_.getFloatTensor.floats.asScala.map(_.doubleValue).toSeq)\n        .getOrElse(defaultEmbeddings)\n    }\n\n    val denseEmbeddingsNormalized = embeddings.map { seq =>\n      val denseVector = DenseVector(seq.toArray)\n      val normVal = norm(denseVector)\n      if (normVal != 0)\n        denseVector / normVal\n      else defaultDenseVector\n    }\n\n    val distanceMatrix =\n      DenseMatrix.zeros[Double](denseEmbeddingsNormalized.length, denseEmbeddingsNormalized.length)\n\n    for (i <- denseEmbeddingsNormalized.indices; j <- denseEmbeddingsNormalized.indices) {\n      distanceMatrix(i, j) = norm(denseEmbeddingsNormalized(i) - denseEmbeddingsNormalized(j))\n    }\n    mmr(query, candidates, distanceMatrix).map(score => FeatureMap(ScoreFeature, Some(score)))\n  }\n\n  def mmr(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n    distanceMatrix: DenseMatrix[Double]\n  ): Seq[Double] = {\n    val n = candidates.length\n    val diversityRatio = query.params(TwhinDiversityRescoringRatioParam)\n    val diversityWeight = query.params(TwhinDiversityRescoringWeightParam)\n    val selected = scala.collection.mutable.Set[Int]()\n    val newScores = Array.fill(n)(0.0)\n\n    val candidatesWithIndex = candidates.zipWithIndex\n    for (i <- 0 until n) {\n      var maxScore = Double.NegativeInfinity\n      var bestCandidateIndex = -1\n\n      for ((candidate, index) <- candidatesWithIndex) {\n        if (!selected.contains(index)) {\n          val relevance = candidate.features.getOrElse(ScoreFeature, None).getOrElse(0.0)\n          val minDistance = {\n            if (selected.isEmpty || selected.size < (1 - diversityRatio) * n) None\n            else {\n              selected\n                .map(j => distanceMatrix(index, j))\n                .reduceOption { (a, b) =>\n                  if (a < b) a else b\n                }\n            }\n          }\n          val score = relevance + diversityWeight * minDistance.getOrElse(2.0)\n          if (score > maxScore) {\n            maxScore = score\n            bestCandidateIndex = index\n          }\n        }\n      }\n\n      if (bestCandidateIndex != -1) {\n        selected += bestCandidateIndex\n        newScores(bestCandidateIndex) = maxScore\n      }\n    }\n\n    newScores.toSeq\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/EarlybirdSearchResultFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdSearchResultFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.EarlybirdRepository\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport com.twitter.servo.keyvalue.KeyValueResult\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Return\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass EarlybirdSearchResultFeatureHydrator @Inject() (\n  @Named(EarlybirdRepository) client: KeyValueRepository[\n    (Seq[Long], Long),\n    Long,\n    eb.ThriftSearchResult\n  ],\n  override val statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"EarlybirdSearchResult\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    EarlybirdScoreFeature,\n    EarlybirdSearchResultFeature\n  )\n\n  override val statScope: String = identifier.toString\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    client((candidates.map(_.candidate.id), query.getRequiredUserId))\n      .map(handleResponse(candidates, _))\n  }\n\n  private def handleResponse(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n    results: KeyValueResult[Long, eb.ThriftSearchResult]\n  ): Seq[FeatureMap] = {\n    candidates\n      .map { candidate =>\n        observedGet(Some(candidate.candidate.id), results)\n      }.map {\n        case Return(Some(searchResult)) =>\n          FeatureMapBuilder()\n            .add(EarlybirdScoreFeature, searchResult.metadata.flatMap(_.score))\n            .add(EarlybirdSearchResultFeature, Some(searchResult))\n            .build()\n        case other =>\n          FeatureMapBuilder()\n            .add(EarlybirdScoreFeature, None)\n            .add(EarlybirdSearchResultFeature, other)\n            .build()\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/FeedbackHistoryQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.FeedbackHistoryFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasRecentFeedbackSinceCacheTtlFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CachedScoredTweets\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelinemixer.clients.feedback.FeedbackHistoryManhattanClient\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class FeedbackHistoryQueryFeatureHydrator @Inject() (\n  feedbackHistoryClient: FeedbackHistoryManhattanClient)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"FeedbackHistory\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(FeedbackHistoryFeature, HasRecentFeedbackSinceCacheTtlFeature)\n\n  override def hydrate(\n    query: PipelineQuery\n  ): Stitch[FeatureMap] =\n    Stitch\n      .callFuture(feedbackHistoryClient.get(query.getRequiredUserId))\n      .map { feedbackHistory =>\n        val latestFeedbackTimestamp =\n          if (feedbackHistory.nonEmpty) Some(feedbackHistory.map(_.timestamp.inMilliseconds).max)\n          else None\n        val cachedScoredTweetsTtl = query.params(CachedScoredTweets.TTLParam)\n        val hasRecentFeedbackSinceCacheTtl = latestFeedbackTimestamp match {\n          case Some(timestamp) =>\n            Time.fromMilliseconds(timestamp).untilNow < cachedScoredTweetsTtl\n          case None => false\n        }\n\n        val featureMapBuilder = FeatureMapBuilder()\n          .add(FeedbackHistoryFeature, feedbackHistory)\n          .add(HasRecentFeedbackSinceCacheTtlFeature, hasRecentFeedbackSinceCacheTtl)\n\n        featureMapBuilder.build()\n      }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/FollowableUttTopicsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.topic_recos.common.LocaleUtil\nimport com.twitter.topiclisting.SemanticCoreEntityId\nimport com.twitter.topiclisting.TopicListingViewerContext\nimport com.twitter.tsp.stores.UttTopicFilterStore\nimport com.twitter.tsp.{thriftscala => tsp}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject FollowableUttTopicsFeatures\n    extends Feature[PipelineQuery, Option[Map[SemanticCoreEntityId, Option[tsp.TopicFollowType]]]]\n\n@Singleton\nclass FollowableUttTopicsQueryFeatureHydrator @Inject() (\n  uttStore: UttTopicFilterStore,\n  override val statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"FollowableUttTopics\")\n\n  override val features: Set[Feature[_, _]] = Set(FollowableUttTopicsFeatures)\n\n  override val statScope: String = identifier.toString\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val context = TopicListingViewerContext.fromClientContext(query.clientContext)\n\n    Stitch.callFuture {\n      uttStore\n        .getAllowListTopicsForUser(\n          userId = query.getRequiredUserId,\n          topicListingSetting = tsp.TopicListingSetting.Followable,\n          context = context\n            .copy(languageCode = LocaleUtil.getStandardLanguageCode(context.languageCode)),\n          bypassModes = None\n        ).map { topics =>\n          FeatureMapBuilder().add(FollowableUttTopicsFeatures, Some(topics)).build()\n        }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/FrsSeedUsersQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.follow_recommendations.{thriftscala => frs}\nimport com.twitter.product_mixer.component_library.candidate_source.recommendations.UserFollowRecommendationsCandidateSource\nimport com.twitter.product_mixer.component_library.candidate_source.recommendations.CachedUserFollowRecommendationsCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyView\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject FrsSeedUserIdsFeature extends Feature[TweetCandidate, Option[Seq[Long]]]\nobject FrsUserToFollowedByUserIdsFeature extends Feature[TweetCandidate, Map[Long, Seq[Long]]]\n\n@Singleton\ncase class FrsSeedUsersQueryFeatureHydrator @Inject() (\n  userFollowRecommendationsCandidateSource: UserFollowRecommendationsCandidateSource,\n  cachedUserFollowRecommendationsCandidateSource: CachedUserFollowRecommendationsCandidateSource)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"FrsSeedUsers\")\n\n  private val maxUsersToFetch = 100\n\n  override val features: Set[Feature[_, _]] = Set(\n    FrsSeedUserIdsFeature,\n    FrsUserToFollowedByUserIdsFeature\n  )\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val frsRequest = frs.RecommendationRequest(\n      clientContext = frs.ClientContext(query.getOptionalUserId),\n      displayLocation = frs.DisplayLocation.HomeTimelineTweetRecs,\n      maxResults = Some(maxUsersToFetch)\n    )\n\n    userFollowRecommendationsCandidateSource(StratoKeyView(frsRequest, Unit))\n      .map { userRecommendations: Seq[frs.UserRecommendation] =>\n        val seedUserIds = userRecommendations.map(_.userId)\n        val seedUserIdsSet = seedUserIds.toSet\n\n        val userToFollowedByUserIds: Map[Long, Seq[Long]] = userRecommendations.flatMap {\n          userRecommendation =>\n            if (seedUserIdsSet.contains(userRecommendation.userId)) {\n              val followProof =\n                userRecommendation.reason.flatMap(_.accountProof).flatMap(_.followProof)\n              val followedByUserIds = followProof.map(_.userIds).getOrElse(Seq.empty)\n              Some(userRecommendation.userId -> followedByUserIds)\n            } else {\n              None\n            }\n        }.toMap\n\n        FeatureMapBuilder()\n          .add(FrsSeedUserIdsFeature, Some(seedUserIds))\n          .add(FrsUserToFollowedByUserIdsFeature, userToFollowedByUserIds)\n          .build()\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/GeoduckAuthorLocationHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.dal.personal_data.{thriftjava => pd}\nimport com.twitter.geoduck.common.thriftscala.TransactionLocation\nimport com.twitter.geoduck.common.{thriftscala => t}\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.product_mixer.component_library.feature.location.Location\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordOptionalFeature\nimport com.twitter.product_mixer.core.feature.datarecord.LongDiscreteDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.ExpiringLruInProcessCache\nimport com.twitter.servo.cache.InProcessCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.geo.service.UserLocationClientColumn\nimport com.twitter.timelines.prediction.features.location.LocationFeatures\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject AuthorLocationNeighborhoodFeature\n    extends DataRecordOptionalFeature[PipelineQuery, Long]\n    with LongDiscreteDataRecordCompatible {\n  override val featureName: String =\n    LocationFeatures.AUTHOR_LOCATION_NEIGHBORDHOOD.getFeatureName\n  override val personalDataTypes: Set[pd.PersonalDataType] =\n    Set(pd.PersonalDataType.InferredLocation)\n}\n\nobject AuthorLocationCityFeature\n    extends DataRecordOptionalFeature[PipelineQuery, Long]\n    with LongDiscreteDataRecordCompatible {\n  override val featureName: String =\n    LocationFeatures.AUTHOR_LOCATION_CITY.getFeatureName\n  override val personalDataTypes: Set[pd.PersonalDataType] =\n    Set(pd.PersonalDataType.InferredLocation)\n}\n\nobject AuthorLocationMetroFeature\n    extends DataRecordOptionalFeature[PipelineQuery, Long]\n    with LongDiscreteDataRecordCompatible {\n  override val featureName: String =\n    LocationFeatures.AUTHOR_LOCATION_METRO.getFeatureName\n  override val personalDataTypes: Set[pd.PersonalDataType] =\n    Set(pd.PersonalDataType.InferredLocation)\n}\n\nobject AuthorLocationRegionFeature\n    extends DataRecordOptionalFeature[PipelineQuery, Long]\n    with LongDiscreteDataRecordCompatible {\n  override val featureName: String =\n    LocationFeatures.AUTHOR_LOCATION_REGION.getFeatureName\n  override val personalDataTypes: Set[pd.PersonalDataType] =\n    Set(pd.PersonalDataType.InferredLocation)\n}\n\nobject AuthorLocationCountryFeature\n    extends DataRecordOptionalFeature[PipelineQuery, Long]\n    with LongDiscreteDataRecordCompatible {\n  override val featureName: String =\n    LocationFeatures.AUTHOR_LOCATION_COUNTRY.getFeatureName\n  override val personalDataTypes: Set[pd.PersonalDataType] =\n    Set(pd.PersonalDataType.InferredLocation)\n}\n\nobject GeoduckAuthorLocationHydrator {\n  private val BaseTTLMinutes = 60 * 24\n  private val TTL = (BaseTTLMinutes + scala.util.Random.nextInt(60)).minutes\n\n  val cache: InProcessCache[Long, Option[TransactionLocation]] =\n    new ExpiringLruInProcessCache[Long, Option[TransactionLocation]](\n      ttl = TTL,\n      maximumSize = 150 * 1000 // Cache up to 150k users\n    )\n}\n\n@Singleton\nclass GeoduckAuthorLocationHydrator @Inject() (\n  userLocationClientColumn: UserLocationClientColumn)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"GeoduckAuthorLocationHydrator\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    LocationFeature,\n    AuthorLocationNeighborhoodFeature,\n    AuthorLocationCityFeature,\n    AuthorLocationMetroFeature,\n    AuthorLocationRegionFeature,\n    AuthorLocationCountryFeature\n  )\n\n  private val PlaceQuery = t.PlaceQuery(\n    placeTypes = Some(\n      Set(\n        t.PlaceType.Neighborhood,\n        t.PlaceType.City,\n        t.PlaceType.Metro,\n        t.PlaceType.Admin1,\n        t.PlaceType.Country\n      )\n    )\n  )\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    val authorIds = candidates.flatMap(_.features.getOrElse(AuthorIdFeature, None)).distinct\n\n    val (hitIds, missIds) = authorIds.partition(id =>\n      GeoduckAuthorLocationHydrator.cache\n        .get(id)\n        .isDefined)\n\n    val cachedOptionMap: Map[Long, Option[TransactionLocation]] =\n      hitIds.map(id => id -> GeoduckAuthorLocationHydrator.cache.get(id).get).toMap\n\n    val fetchCacheMisses: Stitch[Map[Long, TransactionLocation]] =\n      if (missIds.isEmpty) Stitch.value(Map.empty[Long, TransactionLocation])\n      else {\n        userLocationClientColumn.fetcher\n          .fetch(\n            key = Unit,\n            t.UserLocationRequest(\n              userIds = missIds,\n              placeQuery = Some(PlaceQuery)\n            )\n          )\n          .map { response =>\n            response.v.toList.flatMap(_._1).toMap\n          }\n          .handle { case _ => Map.empty[Long, TransactionLocation] }\n      }\n\n    val allLocationsStitch: Stitch[Map[Long, TransactionLocation]] =\n      fetchCacheMisses.map { fetchedMap =>\n        missIds.foreach { id =>\n          val locOpt: Option[TransactionLocation] = fetchedMap.get(id)\n          GeoduckAuthorLocationHydrator.cache.set(id, locOpt)\n        }\n\n        val cachedLocs: Map[Long, TransactionLocation] =\n          cachedOptionMap.collect { case (id, Some(loc)) => id -> loc }\n\n        cachedLocs ++ fetchedMap\n      }\n\n    allLocationsStitch.map { allLocations =>\n      candidates.map { candidate =>\n        val locOpt = for {\n          authorId <- candidate.features.getOrElse(AuthorIdFeature, None)\n          loc <- allLocations.get(authorId)\n        } yield loc\n\n        locOpt\n          .map { transactionLocation =>\n            val placeMap = transactionLocation.placeMap\n            val locationDetails = Location(\n              neighborhood = placeMap\n                .flatMap(_.get(t.PlaceType.Neighborhood))\n                .flatMap(_.headOption),\n              city = placeMap.flatMap(_.get(t.PlaceType.City)).flatMap(_.headOption),\n              metro = placeMap.flatMap(_.get(t.PlaceType.Metro)).flatMap(_.headOption),\n              region = placeMap.flatMap(_.get(t.PlaceType.Admin1)).flatMap(_.headOption),\n              country = placeMap.flatMap(_.get(t.PlaceType.Country)).flatMap(_.headOption)\n            )\n\n            FeatureMapBuilder()\n              .add(LocationFeature, Option(locationDetails))\n              .add(AuthorLocationNeighborhoodFeature, locationDetails.neighborhood)\n              .add(AuthorLocationCityFeature, locationDetails.city)\n              .add(AuthorLocationMetroFeature, locationDetails.metro)\n              .add(AuthorLocationRegionFeature, locationDetails.region)\n              .add(AuthorLocationCountryFeature, locationDetails.country)\n              .build()\n          }\n          .getOrElse {\n            FeatureMapBuilder()\n              .add(LocationFeature, None)\n              .add(AuthorLocationNeighborhoodFeature, None)\n              .add(AuthorLocationCityFeature, None)\n              .add(AuthorLocationMetroFeature, None)\n              .add(AuthorLocationRegionFeature, None)\n              .add(AuthorLocationCountryFeature, None)\n              .build()\n          }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/GizmoduckAuthorFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.github.nscala_time.time.Imports.LocalDate\nimport com.twitter.ads.entities.db.{thriftscala => ae}\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.gizmoduck.{thriftscala => gt}\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.gizmoduck_features.GFeatures\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.gizmoduck_features.GizmoduckFeaturesAdapter\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorAccountAge\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorFollowersFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIsBlueVerifiedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIsProtectedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsSupportAccountReplyFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorSafetyLabels\nimport com.twitter.home_mixer.module.SupportAccountsConfig\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.GizmoduckTimelinesCache\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.TtlCache\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nprivate case class UserFeatures(\n  isBlueVerified: Boolean,\n  isVerifiedOrganization: Boolean,\n  isVerifiedOrganizationAffiliate: Boolean,\n  isProtected: Boolean,\n  isSupportAccount: Boolean,\n  followersCount: Option[Long],\n  accountAge: Option[Duration],\n  labels: Option[Seq[String]])\n    extends GFeatures\n\nobject GizmoduckAuthorFeatures\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass GizmoduckAuthorFeatureHydrator @Inject() (\n  gizmoduck: gt.UserService.MethodPerEndpoint,\n  @Named(GizmoduckTimelinesCache) cacheClient: TtlCache[Long, gt.User],\n  supportAccounts: SupportAccountsConfig)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"GizmoduckAuthor\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    AuthorAccountAge,\n    AuthorFollowersFeature,\n    AuthorIsBlueVerifiedFeature,\n    AuthorIsProtectedFeature,\n    IsSupportAccountReplyFeature,\n    GizmoduckAuthorFeatures,\n    AuthorSafetyLabels\n  )\n\n  private val CacheTTL = 24.hours\n\n  private val queryFields: Set[gt.QueryFields] = Set(\n    gt.QueryFields.AdvertiserAccount,\n    gt.QueryFields.Profile,\n    gt.QueryFields.Safety,\n    gt.QueryFields.Labels,\n    gt.QueryFields.Counts\n  )\n\n  private val lookupContext = gt.LookupContext(isRequestSheddable = Some(true))\n\n  // Advertiser service levels which are assumed to provide support via replies\n  private val AdvertiserServiceLevels = Set[ae.ServiceLevel](\n    ae.ServiceLevel.Dso,\n    ae.ServiceLevel.Mms,\n    ae.ServiceLevel.Reseller,\n    ae.ServiceLevel.Smb\n  )\n\n  private val SupportAccounts = supportAccounts.accounts\n\n  private val DefaultUserFeatures = UserFeatures(\n    isBlueVerified = false,\n    isVerifiedOrganization = false,\n    isVerifiedOrganizationAffiliate = false,\n    isProtected = false,\n    isSupportAccount = false,\n    followersCount = None,\n    accountAge = None,\n    labels = None\n  )\n\n  private def extractFeaturesFromUser(\n    user: gt.User\n  ): UserFeatures = {\n    val isBlueVerified = user.safety.flatMap(_.isBlueVerified).getOrElse(false)\n    val safetyLabels = user.labels.map(_.labels.map(_.labelValue.name))\n    val verifiedOrganizationDetails = user.safety.flatMap(_.verifiedOrganizationDetails)\n    val isVerifiedOrganization =\n      verifiedOrganizationDetails.flatMap(_.isVerifiedOrganization).getOrElse(false)\n    val isVerifiedOrganizationAffiliate =\n      verifiedOrganizationDetails.flatMap(_.isVerifiedOrganizationAffiliate).getOrElse(false)\n\n    val isProtected = user.safety.exists(_.isProtected)\n    val isSupportAccount =\n      user.profile.exists(_.businessProfileState == gt.BusinessProfileState.Enabled) ||\n        (user.advertiserAccount.flatMap(_.advertiserType).nonEmpty &&\n          user.advertiserAccount\n            .flatMap(_.serviceLevels).getOrElse(Seq.empty)\n            .exists(AdvertiserServiceLevels.contains))\n    val followersCount = user.counts.map(_.followers)\n    val accountAge = Duration.fromSeconds(\n      (LocalDate.now().toDate.toInstant.getEpochSecond - (user.createdAtMsec / 1000)).toInt)\n\n    UserFeatures(\n      isBlueVerified,\n      isVerifiedOrganization,\n      isVerifiedOrganizationAffiliate,\n      isProtected,\n      isSupportAccount,\n      followersCount,\n      Some(accountAge),\n      safetyLabels\n    )\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val authorIds = candidates.flatMap(_.features.getOrElse(AuthorIdFeature, None)).distinct\n    cacheClient\n      .get(authorIds)\n      .flatMap { cacheResponse =>\n        val cacheHydratedUsersMap = cacheResponse.found\n        val notFoundUsers = cacheResponse.notFound.toSeq.distinct\n\n        val gizmoduckResponseFuture = if (notFoundUsers.nonEmpty) {\n          gizmoduck.get(lookupContext, notFoundUsers, queryFields)\n        } else Future.value(Seq.empty)\n\n        gizmoduckResponseFuture.map { gizmoduckResponse =>\n          val gizmoduckHydratedUsersMap = gizmoduckResponse.collect {\n            case userResult if userResult.user.isDefined =>\n              val user = userResult.user.get\n              cacheClient.add(user.id, user, CacheTTL)\n              user.id -> user\n          }.toMap\n\n          val hydratedUsersMap = (cacheHydratedUsersMap ++ gizmoduckHydratedUsersMap)\n            .mapValues(extractFeaturesFromUser)\n\n          candidates.map { candidate =>\n            val authorIdOpt = candidate.features.getOrElse(AuthorIdFeature, None)\n            val userFeatures =\n              authorIdOpt.flatMap(hydratedUsersMap.get).getOrElse(DefaultUserFeatures)\n\n            // Some accounts run promotions and send replies automatically.\n            // We assume that a reply that took more than one minute is not an auto-reply.\n            // If time difference doesn't exist, this means that one of the tweets was\n            // not snowflake and therefore much older, and therefore OK as an extended reply.\n            val timeDifference = candidate.features.getOrElse(InReplyToTweetIdFeature, None).map {\n              SnowflakeId.timeFromId(candidate.candidate.id) - SnowflakeId.timeFromId(_)\n            }\n            val isAutoReply = timeDifference.exists(_ < 1.minute)\n\n            val isSupportAccountReply =\n              candidate.features.getOrElse(InReplyToTweetIdFeature, None).nonEmpty &&\n                !candidate.features.getOrElse(IsRetweetFeature, false) &&\n                candidate.features.getOrElse(FromInNetworkSourceFeature, false) &&\n                (userFeatures.isSupportAccount ||\n                  authorIdOpt.exists(SupportAccounts.contains) || isAutoReply)\n\n            val gizmoduckFeaturesRichDataRecords =\n              GizmoduckFeaturesAdapter.adaptToDataRecords(userFeatures).asScala.head\n\n            FeatureMapBuilder()\n              .add(AuthorAccountAge, userFeatures.accountAge)\n              .add(AuthorFollowersFeature, userFeatures.followersCount)\n              .add(AuthorIsBlueVerifiedFeature, userFeatures.isBlueVerified)\n              .add(AuthorIsProtectedFeature, userFeatures.isProtected)\n              .add(IsSupportAccountReplyFeature, isSupportAccountReply)\n              .add(GizmoduckAuthorFeatures, gizmoduckFeaturesRichDataRecords)\n              .add(AuthorSafetyLabels, userFeatures.labels)\n              .build()\n          }\n        }\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/GizmoduckUserQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.gizmoduck.{thriftscala => gt}\nimport com.twitter.home_mixer.model.HomeFeatures.SignupCountryFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SignupSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserFollowersCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserFollowingCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserScreenNameFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerAllowsAdsPersonalizationFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerAllowsDataSharingFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerAllowsForYouRecommendationsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerHasPremiumTier\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerSafetyLabels\nimport com.twitter.home_mixer.model.signup.MarchMadness\nimport com.twitter.home_mixer.model.signup.Onboard\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.gizmoduck.Gizmoduck\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class GizmoduckUserQueryFeatureHydrator @Inject() (gizmoduck: Gizmoduck)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"GizmoduckUser\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    UserFollowingCountFeature,\n    UserFollowersCountFeature,\n    UserTypeFeature,\n    UserScreenNameFeature,\n    ViewerHasPremiumTier,\n    SignupCountryFeature,\n    SignupSourceFeature,\n    ViewerAllowsForYouRecommendationsFeature,\n    ViewerAllowsDataSharingFeature,\n    ViewerAllowsAdsPersonalizationFeature,\n    ViewerSafetyLabels\n  )\n\n  private val queryFields: Set[gt.QueryFields] = Set(\n    gt.QueryFields.Counts,\n    gt.QueryFields.Safety,\n    gt.QueryFields.Profile,\n    gt.QueryFields.Account,\n    gt.QueryFields.Labels\n  )\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n    gizmoduck\n      .getUserById(\n        userId = userId,\n        queryFields = queryFields,\n        context = gt.LookupContext(forUserId = Some(userId), includeSoftUsers = true))\n      .map { user =>\n        val premiumTier = user.safety\n          .map { safety =>\n            safety.skipRateLimit.contains(true) ||\n            safety.isBlueVerified.contains(true) ||\n            safety.verifiedType.contains(gt.VerifiedType.Business) ||\n            safety.verifiedType.contains(gt.VerifiedType.Government) ||\n            safety.verifiedOrganizationDetails.exists(_.isVerifiedOrganization.getOrElse(false)) ||\n            safety.verifiedOrganizationDetails\n              .exists(_.isVerifiedOrganizationAffiliate.contains(true))\n          }.getOrElse(false)\n\n        val signupSource = user.safety.flatMap(_.signupCreationSource).flatMap {\n          case gt.SignupCreationSource.MarchMadness => Some(MarchMadness)\n          case gt.SignupCreationSource.Onboard => Some(Onboard)\n          case _ => None\n        }\n\n        FeatureMapBuilder()\n          .add(UserFollowingCountFeature, user.counts.map(_.following.toInt))\n          .add(UserFollowersCountFeature, user.counts.map(_.followers.toInt))\n          .add(UserTypeFeature, Some(user.userType))\n          .add(UserScreenNameFeature, user.profile.map(_.screenName))\n          .add(ViewerHasPremiumTier, premiumTier)\n          .add(SignupCountryFeature, user.safety.flatMap(_.signupCountryCode))\n          .add(SignupSourceFeature, signupSource)\n          .add(\n            ViewerAllowsForYouRecommendationsFeature,\n            user.account.flatMap(_.allowForYouRecommendations)\n          )\n          .add(\n            ViewerAllowsDataSharingFeature,\n            user.account.map(_.allowSharingDataForThirdPartyPersonalization)\n          ).add(\n            ViewerAllowsAdsPersonalizationFeature,\n            user.account.map(_.allowAdsPersonalization)\n          ).add(\n            ViewerSafetyLabels,\n            user.labels.map(_.labels.map(_.labelValue.name))\n          )\n          .build()\n      }\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.7)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/GraphTwoHopFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.graph_feature_service.{thriftscala => gfs}\nimport com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.GraphTwoHopRepository\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.two_hop_features.TwoHopFeaturesAdapter\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject GraphTwoHopFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass GraphTwoHopFeatureHydrator @Inject() (\n  @Named(GraphTwoHopRepository) client: KeyValueRepository[\n    (Seq[Long], Long),\n    Long,\n    Seq[gfs.IntersectionValue]\n  ],\n  override val statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"GraphTwoHop\")\n\n  override val features: Set[Feature[_, _]] = Set(GraphTwoHopFeature, FollowedByUserIdsFeature)\n\n  override val statScope: String = identifier.toString\n\n  private val twoHopFeaturesAdapter = new TwoHopFeaturesAdapter\n\n  private val FollowFeatureType = gfs.FeatureType(gfs.EdgeType.Following, gfs.EdgeType.FollowedBy)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    // Apply filters to in network candidates for retweets only.\n    val (inNetworkCandidates, oonCandidates) = candidates.partition { candidate =>\n      candidate.features.getOrElse(FromInNetworkSourceFeature, false)\n    }\n\n    val inNetworkCandidatesToHydrate =\n      inNetworkCandidates.filter(_.features.getOrElse(IsRetweetFeature, false))\n\n    val candidatesToHydrate = (inNetworkCandidatesToHydrate ++ oonCandidates)\n      .flatMap(candidate => CandidatesUtil.getOriginalAuthorId(candidate.features)).distinct\n\n    val response = client((candidatesToHydrate, query.getRequiredUserId))\n\n    response.map { result =>\n      candidates.map { candidate =>\n        val originalAuthorId = CandidatesUtil.getOriginalAuthorId(candidate.features)\n\n        val value = observedGet(key = originalAuthorId, keyValueResult = result)\n        val transformedValue = postTransformer(value)\n        val followedByUserIds = value.toOption\n          .flatMap(getFollowedByUserIds(_))\n          .getOrElse(Seq.empty)\n\n        FeatureMapBuilder()\n          .add(GraphTwoHopFeature, transformedValue)\n          .add(FollowedByUserIdsFeature, followedByUserIds)\n          .build()\n      }\n    }\n  }\n\n  private def getFollowedByUserIds(input: Option[Seq[gfs.IntersectionValue]]): Option[Seq[Long]] =\n    input.map {\n      _.filter(_.featureType == FollowFeatureType).flatMap(_.intersectionIds).flatten.distinct\n    }\n\n  private def postTransformer(input: Try[Option[Seq[gfs.IntersectionValue]]]): Try[DataRecord] =\n    input.map(twoHopFeaturesAdapter.adaptToDataRecords(_).asScala.head)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/GrokAnnotationsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.GrokTopics.GrokCategoryIdToNameMap\nimport com.twitter.home_mixer.model.HomeFeatures.DebugStringFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokAnnotationsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokCategoryDataRecordFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokTopCategoryFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokIsGoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokIsLowQualityFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokIsNsfwFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokIsOcrFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokIsSpamFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokIsViolentFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokPoliticalInclinationFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokSlopScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokSunnyScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokTagsFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableGrokAnnotations\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer_features.{thriftscala => hmf}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GrokAnnotationsFeatureHydrator @Inject() (\n  homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"GrokAnnotations\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(\n      GrokAnnotationsFeature,\n      GrokCategoryDataRecordFeature,\n      GrokTopCategoryFeature,\n      GrokTagsFeature,\n      GrokIsGoreFeature,\n      GrokIsNsfwFeature,\n      GrokIsSpamFeature,\n      GrokIsViolentFeature,\n      GrokIsLowQualityFeature,\n      GrokIsOcrFeature,\n      GrokSunnyScoreFeature,\n      GrokPoliticalInclinationFeature,\n      GrokSlopScoreFeature,\n      DebugStringFeature\n    )\n\n  override def onlyIf(query: PipelineQuery): Boolean = query.params(EnableGrokAnnotations)\n\n  private val batchSize = 64\n\n  private def getGrokAnnotationsFromHMF(\n    tweetIdsToHydrate: Seq[Long],\n  ): Future[Seq[Option[hmt.GrokAnnotations]]] = {\n    val keySerialized = tweetIdsToHydrate.map(_.toString)\n    val request = hmf.HomeMixerFeaturesRequest(\n      keySerialized,\n      hmf.Cache.GrokPostAnnotations\n    )\n\n    val responseFut = homeMixerFeatureService.getHomeMixerFeatures(request)\n    responseFut\n      .map { response =>\n        response.homeMixerFeatures\n          .map { homeMixerFeaturesOpt =>\n            homeMixerFeaturesOpt.homeMixerFeaturesType.map {\n              case hmf.HomeMixerFeaturesType.GrokPostAnnotations(data) =>\n                val metadata = data.annotations.tweetBoolMetadata.map { metadata =>\n                  hmt.GrokMetadata(\n                    isNsfw = metadata.isNsfw.getOrElse(false),\n                    isGore = metadata.isGore.getOrElse(false),\n                    isViolent = metadata.isViolent.getOrElse(false),\n                    isSpam = metadata.isSpam.getOrElse(false),\n                    isSoftNsfw = metadata.isSoftNsfw.getOrElse(false),\n                    isLowQuality = metadata.isHighQuality.exists(!_),\n                    isOcr = metadata.isOcr.getOrElse(false),\n                  )\n                }\n                val categoryScoreMap =\n                  data.annotations.entities // This gives Option[List[EntityWithMetadata]]\n                    .map(\n                      _.map(entity =>\n                        entity.qualifiedId._2.toString -> entity.score\n                          .getOrElse(0.0) // Convert each entity to (String, Double)\n                      ).toMap\n                    ) // Convert List[(String, Double)] to Map[String, Double]\n\n                hmt.GrokAnnotations(\n                  topics = data.topics,\n                  tags = data.annotations.tags.getOrElse(Seq.empty).map(_.tag),\n                  metadata = metadata,\n                  categoryScores = categoryScoreMap,\n                  sunnyScore = data.annotations.sunnyScore,\n                  politicalInclination = data.annotations.politicalInclination.flatMap {\n                    inclination =>\n                      scala.util\n                        .Try(hmt.PoliticalInclination.valueOf(inclination.name)).toOption.flatten\n                  },\n                  slopScore = data.annotations.slopScore,\n                )\n              case _ => throw new Exception(\"Unknown type returned\")\n            }\n          }\n      }.handle { case _ => Seq.fill(tweetIdsToHydrate.size)(None) }\n  }\n\n  def getFeatureMaps(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n  ): Future[Seq[FeatureMap]] = {\n\n    val tweetIdsToHydrate = candidates.map(CandidatesUtil.getOriginalTweetId)\n    val debugStringFeatures =\n      candidates.map(_.features.getOrElse(DebugStringFeature, None).getOrElse(\"\"))\n\n    val responseMap = getGrokAnnotationsFromHMF(tweetIdsToHydrate)\n    responseMap.map { result =>\n      result.zip(debugStringFeatures).map {\n        case (annotations, debugStringFeature) =>\n          val categoryScoreMap: Option[Map[String, Double]] =\n            annotations.flatMap(_.categoryScores.map(_.toMap))\n          val tags: Set[String] = annotations.map(_.tags).getOrElse(Seq.empty).toSet\n          val grokSlopFeature = annotations.flatMap(_.slopScore.map(_.toLong))\n\n          FeatureMapBuilder()\n            .add(GrokAnnotationsFeature, annotations)\n            .add(GrokCategoryDataRecordFeature, categoryScoreMap)\n            .add(\n              GrokTopCategoryFeature,\n              annotations\n                .flatMap(_.categoryScores)\n                .flatMap { scores =>\n                  val validCategories = scores.collect {\n                    case (category, score)\n                        if category.forall(_.isDigit) && GrokCategoryIdToNameMap.contains(category.toLong) =>\n                      (category.toLong, score)\n                  }\n                  if (validCategories.nonEmpty) {\n                    Some(validCategories.maxBy(_._2)._1)\n                  } else {\n                    None\n                  }\n                }\n            )\n            .add(GrokTagsFeature, tags.map(_.toLowerCase))\n            .add(GrokIsGoreFeature, annotations.flatMap(_.metadata.map(_.isGore)))\n            .add(GrokIsNsfwFeature, annotations.flatMap(_.metadata.map(_.isNsfw)))\n            .add(GrokIsSpamFeature, annotations.flatMap(_.metadata.map(_.isSpam)))\n            .add(GrokIsViolentFeature, annotations.flatMap(_.metadata.map(_.isViolent)))\n            .add(GrokIsLowQualityFeature, annotations.flatMap(_.metadata.map(_.isLowQuality)))\n            .add(GrokIsOcrFeature, annotations.flatMap(_.metadata.map(_.isOcr)))\n            .add(GrokSunnyScoreFeature, annotations.flatMap(_.sunnyScore))\n            // Used only for metrics tracking. Does not affect the recommendations.\n            .add(GrokPoliticalInclinationFeature, annotations.flatMap(_.politicalInclination))\n            .add(GrokSlopScoreFeature, grokSlopFeature)\n            .add(\n              DebugStringFeature,\n              Some(\"%s GrokSlop:%s\"\n                .format(debugStringFeature, grokSlopFeature.map(_.toString).getOrElse(\"None\"))))\n            .build()\n        case _ =>\n          FeatureMapBuilder()\n            .add(GrokAnnotationsFeature, None)\n            .add(GrokCategoryDataRecordFeature, None)\n            .add(GrokTopCategoryFeature, None)\n            .add(GrokTagsFeature, Set.empty[String])\n            .add(GrokIsGoreFeature, None)\n            .add(GrokIsNsfwFeature, None)\n            .add(GrokIsSpamFeature, None)\n            .add(GrokIsViolentFeature, None)\n            .add(GrokIsLowQualityFeature, None)\n            .add(GrokIsOcrFeature, None)\n            .add(GrokSunnyScoreFeature, None)\n            .add(GrokPoliticalInclinationFeature, None)\n            .add(GrokSlopScoreFeature, None)\n            .add(DebugStringFeature, None)\n            .build()\n      }\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    OffloadFuturePools.offloadBatchSeqToFutureSeq(\n      candidates,\n      getFeatureMaps,\n      batchSize\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/GrokGorkContentCreatorFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokContentCreatorFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GorkContentCreatorFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\n\n/**\n * Used only for metrics tracking to measure how often we are serving these posts\n */\nobject GrokGorkContentCreatorFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"GrokGorkContentCreator\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(GrokContentCreatorFeature, GorkContentCreatorFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offload {\n\n    candidates.map { candidate =>\n      val authorIdOpt = candidate.features.getOrElse(AuthorIdFeature, None)\n      FeatureMap(\n        GrokContentCreatorFeature,\n        authorIdOpt.contains(GrokCreatorId),\n        GorkContentCreatorFeature,\n        authorIdOpt.contains(GorkCreatorId),\n      )\n    }\n  }\n\n  private val GrokCreatorId = <removed_id> // @grok\n  private val GorkCreatorId = <removed_id> // @gork\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/GrokTranslatedPostIsCachedFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.GrokTranslatedPostIsCachedFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableGrokAutoTranslateLanguageFilter\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.searchai.grok.TranslatedPostClientColumn\nimport com.twitter.search_router.thriftscala.GrokTranslateData\nimport com.twitter.util.logging.Logging\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GrokTranslatedPostIsCachedFeatureHydrator @Inject() (\n  translatedPostClientColumn: TranslatedPostClientColumn,\n  statsReceiver: StatsReceiver)\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery]\n    with Logging {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"GrokTranslatedPostIsCached\")\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val successCounter = scopedStatsReceiver.counter(\"cacheFetch/success\")\n  private val failedCounter = scopedStatsReceiver.counter(\"cacheFetch/failure\")\n  private val cacheHitCounter = scopedStatsReceiver.counter(\"cacheHit\")\n  private val cacheMissCounter = scopedStatsReceiver.counter(\"cacheMiss\")\n  private val DefaultFeatureMap =\n    FeatureMapBuilder().add(GrokTranslatedPostIsCachedFeature, true).build()\n\n  private val fetcher: Fetcher[(Long, String), Unit, GrokTranslateData] =\n    translatedPostClientColumn.fetcher\n\n  override val features: Set[Feature[_, _]] = Set(GrokTranslatedPostIsCachedFeature)\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableGrokAutoTranslateLanguageFilter)\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    val tweetId = candidate.id\n    val dstLangOpt = query.getLanguageCode\n    dstLangOpt match {\n      case Some(dstLang) =>\n        fetcher\n          .fetch((tweetId, dstLang), ()).map { result =>\n            successCounter.incr()\n            val cacheAvailable = result.v.isDefined\n            if (cacheAvailable) cacheHitCounter.incr()\n            else cacheMissCounter.incr()\n            FeatureMapBuilder()\n              .add(GrokTranslatedPostIsCachedFeature, cacheAvailable)\n              .build()\n          }.rescue {\n            case _ =>\n              failedCounter.incr()\n              Stitch.value(DefaultFeatureMap)\n          }\n      case None => Stitch.value(DefaultFeatureMap)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/HeartbeatOptimizerParamsHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.HeartbeatOptimizerParamsMHPkey\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.columns.heartbeat_optimizer.thriftscala.OptimizerMHParamsValue\nimport com.twitter.strato.columns.heartbeat_optimizer.thriftscala.ParameterAndValue\nimport com.twitter.strato.generated.client.heartbeat_optimizer.HeartbeatOptimizerParamsMHClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject HeartbeatOptimizerWeightsFeature extends Feature[PipelineQuery, Option[OptimizerParams]]\n\ncase class OptimizerParams(\n  epochTimestamp: Long,\n  nBuckets: Int,\n  optimizerWeights: Seq[Map[String, Double]])\n\n@Singleton\nclass HeartbeatOptimizerParamsHydrator @Inject() (\n  heartbeatOptimizerParamsMHClientColumn: HeartbeatOptimizerParamsMHClientColumn,\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"HeartbeatOptimizerParams\")\n\n  private val fetcher: Fetcher[(String, String), Unit, OptimizerMHParamsValue] =\n    heartbeatOptimizerParamsMHClientColumn.fetcher\n\n  override val features: Set[Feature[_, _]] = Set(HeartbeatOptimizerWeightsFeature)\n\n  private val LKEY = \"0\"\n\n  private val sucessStat: Counter = statsReceiver.counter(\"success\")\n  private val failureStat: Counter = statsReceiver.counter(\"failure\")\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val pkey = query.params(HeartbeatOptimizerParamsMHPkey)\n    fetcher\n      .fetch((pkey, LKEY)).map { manhattanResult =>\n        manhattanResult.v match {\n          case Some(optimizerMHParamsValue) =>\n            val parameterAndValueList: Seq[Seq[ParameterAndValue]] =\n              optimizerMHParamsValue.parameterAndValueList\n\n            val paramMaps: Seq[Map[String, Double]] = parameterAndValueList.map {\n              paramAndValueList =>\n                paramAndValueList.map { paramAndValue =>\n                  (paramAndValue.parameter, paramAndValue.value)\n                }.toMap\n            }\n            val optimizerWeights = OptimizerParams(\n              epochTimestamp = optimizerMHParamsValue.epochTimestamp.toLong,\n              nBuckets = paramMaps.size,\n              optimizerWeights = paramMaps\n            )\n            sucessStat.incr(1)\n            FeatureMapBuilder()\n              .add(HeartbeatOptimizerWeightsFeature, Some(optimizerWeights))\n              .build()\n          case _ =>\n            failureStat.incr(1)\n            FeatureMapBuilder()\n              .add(HeartbeatOptimizerWeightsFeature, None)\n              .build()\n        }\n      }.handle {\n        case _ =>\n          failureStat.incr(1)\n          FeatureMapBuilder()\n            .add(HeartbeatOptimizerWeightsFeature, None)\n            .build()\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/HeavyRankerWeightsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport breeze.stats.distributions.Beta\nimport breeze.stats.distributions.Rand\nimport com.github.nscala_time.time.Imports.LocalDate\nimport com.twitter.home_mixer.model.PredictedScoreFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.AddNoiseInWeightsPerLabel\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.EnableDailyFrozenNoisyWeights\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.NoisyWeightAlphaParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.NoisyWeightBetaParam\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport java.lang.{Long => JLong}\nimport java.util.Objects\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.ml.api.thriftscala.GeneralTensor\nimport com.twitter.ml.api.thriftscala.DoubleTensor\n\n@Singleton\nclass HeavyRankerWeightsQueryFeatureHydrator @Inject() ()\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"HeavyRankerWeights\")\n\n  private def doubleToGeneralTensor(value: Double): GeneralTensor = {\n    val tensor = DoubleTensor(doubles = List(value), shape = Some(List(1L)))\n    GeneralTensor.DoubleTensor(tensor)\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val builder = FeatureMapBuilder()\n    // Ensure that for the same userId and start of day, we get the same sequence of random numbers\n    // Let's us freeze the weight for one day and observe UAS impact\n    val startOfDay = LocalDate.now().toDateTimeAtStartOfDay.toInstant.getMillis\n    val seed =\n      Objects.hash(JLong.valueOf(query.getRequiredUserId), JLong.valueOf(startOfDay)).toLong\n    val alpha = query.params(NoisyWeightAlphaParam)\n    val beta = query.params(NoisyWeightBetaParam)\n    val betaDist = new Beta(alpha, beta)\n    if (query.params(EnableDailyFrozenNoisyWeights)) Rand.generator.setSeed(seed)\n    for (predictedScoreFeature <- PredictedScoreFeature.PredictedScoreFeatures) {\n      val presetWeight = query.params(predictedScoreFeature.modelWeightParam)\n      val weight = if (query.params(AddNoiseInWeightsPerLabel)) {\n        // Apply noise to each weight\n        presetWeight * (1 + betaDist.draw())\n      } else {\n        presetWeight\n      }\n      builder.add(predictedScoreFeature.weightQueryFeature, Some(weight))\n      for (biasQueryFeature <- predictedScoreFeature.biasQueryFeature;\n        modelBiasParam <- predictedScoreFeature.modelBiasParam) {\n        builder.add(biasQueryFeature, Some(query.params(modelBiasParam)))\n      }\n      for (debiasQueryFeature <- predictedScoreFeature.debiasQueryFeature;\n        modelDebiasParam <- predictedScoreFeature.modelDebiasParam) {\n        val debiasValue = query.params(modelDebiasParam)\n        builder.add(debiasQueryFeature, Some(doubleToGeneralTensor((debiasValue))))\n      }\n    }\n    Stitch.value(builder.build())\n  }\n\n  override val features: Set[Feature[_, _]] = {\n    val weightFeatures = PredictedScoreFeature.PredictedScoreFeatureSet.map(_.weightQueryFeature)\n    val biasFeatures = PredictedScoreFeature.PredictedScoreFeatureSet.flatMap(_.biasQueryFeature)\n    val debiasFeatures = PredictedScoreFeature.PredictedScoreFeatureSet\n      .flatMap(_.debiasQueryFeature)\n    (weightFeatures ++ biasFeatures ++ debiasFeatures).toSet\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/ImpressedImageClusterIdsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.ImageClipClusterIdInMemCache\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.InProcessCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.impressionstore.thriftscala.ImpressionList\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\ncase object ImpressedImageClusterIds extends FeatureWithDefaultOnFailure[PipelineQuery, Seq[Long]] {\n  override val defaultValue: Seq[Long] = Seq.empty\n}\n\n/**\n * Get the list of image cluster ids that the user has already seen.\n */\n@Singleton\ncase class ImpressedImageClusterIdsQueryFeatureHydrator @Inject() (\n  stratoClient: Client,\n  tweetImpressionStore: ReadableStore[Long, ImpressionList],\n  @Named(ImageClipClusterIdInMemCache) imageClipClusterIdInMemCache: InProcessCache[\n    Long,\n    Option[Option[Long]]\n  ],\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"ImpressedImageClusterIdsQuery\")\n\n  override val features: Set[Feature[_, _]] = Set(ImpressedImageClusterIds)\n\n  val clusterIdFetcher: Fetcher[Long, Unit, Long] =\n    stratoClient.fetcher[Long, Unit, Long](\n      \"videoRecommendations/twitterClip/twitterClipImageClusterIdMh95\")\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"impressionKey/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"impressionKey/notFound\")\n  private val cacheHitCounter = scopedStatsReceiver.counter(\"cache/hit\")\n  private val cacheMissCounter = scopedStatsReceiver.counter(\"cache/miss\")\n  private val fetchExceptionCounter =\n    scopedStatsReceiver.counter(\"getFromCacheOrFetch/exception\")\n\n  private def getImageClipClusterId(tweetId: Long): Stitch[Option[Option[Long]]] = {\n    imageClipClusterIdInMemCache\n      .get(tweetId)\n      .map { cachedValue =>\n        cacheHitCounter.incr()\n        Stitch.value(cachedValue)\n      }.getOrElse {\n        cacheMissCounter.incr()\n        clusterIdFetcher\n          .fetch(tweetId)\n          .map(_.v)\n          .liftToOption()\n          .flatMap { clusterIdOpt =>\n            imageClipClusterIdInMemCache.set(tweetId, clusterIdOpt)\n            Stitch.value(clusterIdOpt)\n          }\n      }.handle {\n        case _: Exception =>\n          fetchExceptionCounter.incr()\n          None\n      }\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    query.getOptionalUserId match {\n      case Some(userId) =>\n        val featureMapResult: Future[Seq[Long]] = tweetImpressionStore\n          .get(userId).map { impressionListOpt =>\n            if (impressionListOpt.isEmpty) {\n              keyNotFoundCounter.incr()\n            }\n            keyFoundCounter.incr()\n\n            val tweetIdsOpt = for {\n              impressionList <- impressionListOpt\n              impressions <- impressionList.impressions\n            } yield {\n              impressions.take(250).map(_.tweetId)\n            }\n            tweetIdsOpt.getOrElse(Seq.empty)\n          }\n\n        Stitch.callFuture(featureMapResult).flatMap { tweetIds =>\n          Stitch\n            .traverse(tweetIds) { tweetId =>\n              getImageClipClusterId(tweetId)\n            }.map { results: Seq[Option[Option[Long]]] =>\n              FeatureMapBuilder().add(ImpressedImageClusterIds, results.flatten.flatten).build()\n            }\n        }\n\n      case None =>\n        val featureMapResult = FeatureMapBuilder().add(ImpressedImageClusterIds, Seq.empty).build()\n        Stitch.value(featureMapResult)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/ImpressedMediaClusterIdsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MediaClusterId95Store\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MediaClipClusterIdInMemCache\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.InProcessCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.impressionstore.thriftscala.ImpressionList\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\ncase object ImpressedMediaClusterIds extends FeatureWithDefaultOnFailure[PipelineQuery, Seq[Long]] {\n  override val defaultValue: Seq[Long] = Seq.empty\n}\n\n/**\n * Get the list of media cluster ids that the user has already seen.\n */\n@Singleton\ncase class ImpressedMediaClusterIdsQueryFeatureHydrator @Inject() (\n  @Named(MediaClusterId95Store) clusterIdStore: ReadableStore[Long, Long],\n  tweetImpressionStore: ReadableStore[Long, ImpressionList],\n  @Named(MediaClipClusterIdInMemCache) mediaClipClusterIdInMemCache: InProcessCache[\n    Long,\n    Option[Option[Long]]\n  ],\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"ImpressedMediaClusterIdsQuery\")\n\n  override val features: Set[Feature[_, _]] = Set(ImpressedMediaClusterIds)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"impressionKey/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"impressionKey/notFound\")\n  private val cacheHitCounter = scopedStatsReceiver.counter(\"cache/hit\")\n  private val cacheMissCounter = scopedStatsReceiver.counter(\"cache/miss\")\n  private val storeHitCounter = scopedStatsReceiver.counter(\"store/hit\")\n  private val storeMissCounter = scopedStatsReceiver.counter(\"store/miss\")\n  private val fetchExceptionCounter = scopedStatsReceiver.counter(\"fetch/exception\")\n\n  private def getmediaClipClusterId(tweetId: Long): Stitch[Option[Option[Long]]] = {\n    mediaClipClusterIdInMemCache\n      .get(tweetId)\n      .map { cachedValue =>\n        cacheHitCounter.incr()\n        Stitch.value(cachedValue)\n      }.getOrElse {\n        cacheMissCounter.incr()\n        Stitch\n          .callFuture(clusterIdStore.get(tweetId))\n          .flatMap { clusterIdOpt =>\n            if (clusterIdOpt.isDefined) {\n              storeHitCounter.incr()\n            } else {\n              storeMissCounter.incr()\n            }\n            val wrappedClusterIdOpt = Some(clusterIdOpt)\n            mediaClipClusterIdInMemCache.set(tweetId, wrappedClusterIdOpt)\n            Stitch.value(wrappedClusterIdOpt)\n          }\n      }.handle {\n        case _: Exception =>\n          fetchExceptionCounter.incr()\n          None\n      }\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    query.getOptionalUserId match {\n      case Some(userId) =>\n        val featureMapResult: Future[Seq[Long]] = tweetImpressionStore\n          .get(userId).map { impressionListOpt =>\n            if (impressionListOpt.isEmpty) {\n              keyNotFoundCounter.incr()\n            }\n            keyFoundCounter.incr()\n\n            val tweetIdsOpt = for {\n              impressionList <- impressionListOpt\n              impressions <- impressionList.impressions\n            } yield {\n              impressions.take(250).map(_.tweetId)\n            }\n            tweetIdsOpt.getOrElse(Seq.empty)\n          }\n\n        Stitch.callFuture(featureMapResult).flatMap { tweetIds =>\n          Stitch\n            .traverse(tweetIds) { tweetId =>\n              getmediaClipClusterId(tweetId)\n            }.map { results: Seq[Option[Option[Long]]] =>\n              FeatureMapBuilder().add(ImpressedMediaClusterIds, results.flatten.flatten).build()\n            }\n        }\n\n      case None =>\n        val featureMapResult = FeatureMapBuilder().add(ImpressedMediaClusterIds, Seq.empty).build()\n        Stitch.value(featureMapResult)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/ImpressionBloomFilterQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.ImpressionBloomFilterFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MemcachedImpressionBloomFilterStore\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.Store\nimport com.twitter.timelines.impressionbloomfilter.{thriftscala => blm}\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\ncase class ImpressionBloomFilterQueryFeatureHydrator @Inject() (\n  @Named(MemcachedImpressionBloomFilterStore) bloomFilterClient: Store[\n    blm.ImpressionBloomFilterKey,\n    blm.ImpressionBloomFilterSeq\n  ]) extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"ImpressionBloomFilter\")\n\n  override val features: Set[Feature[_, _]] = Set(ImpressionBloomFilterFeature)\n\n  private val SurfaceArea = blm.SurfaceArea.HomeTimeline\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n    Stitch.callFuture {\n      bloomFilterClient\n        .get(blm.ImpressionBloomFilterKey(userId, SurfaceArea))\n        .map(_.getOrElse(blm.ImpressionBloomFilterSeq(Seq.empty)))\n        .map { bloomFilterSeq =>\n          FeatureMapBuilder().add(ImpressionBloomFilterFeature, bloomFilterSeq).build()\n        }\n    }\n  }\n\n  override val alerts = Seq(HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/InNetworkFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject InNetworkFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"InNetwork\")\n\n  override val features: Set[Feature[_, _]] = Set(InNetworkFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val viewerId = query.getRequiredUserId\n    val followedUserIds = query.features.get.get(SGSFollowedUsersFeature).toSet\n\n    val featureMaps = candidates.map { candidate =>\n      // We use authorId and not sourceAuthorId here so that retweets are defined as in network\n      val isInNetworkOpt = candidate.features.getOrElse(AuthorIdFeature, None).map { authorId =>\n        // Users cannot follow themselves but this is in network by definition\n        val isSelfTweet = authorId == viewerId\n        isSelfTweet || followedUserIds.contains(authorId)\n      }\n      FeatureMapBuilder().add(InNetworkFeature, isInNetworkOpt.getOrElse(true)).build()\n    }\n    Stitch.value(featureMaps)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/LastNegativeFeedbackTimeQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.LastNegativeFeedbackTimeFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.recommendations.user_signal_service.SignalsClientColumn\nimport com.twitter.usersignalservice.thriftscala.BatchSignalRequest\nimport com.twitter.usersignalservice.thriftscala.ClientIdentifier\nimport com.twitter.usersignalservice.thriftscala.SignalRequest\nimport com.twitter.usersignalservice.thriftscala.SignalType\nimport com.twitter.usersignalservice.thriftscala.{Signal => UssSignal}\nimport com.twitter.util.Time\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass LastNegativeFeedbackTimeQueryFeatureHydrator @Inject() (\n  signalsClientColumn: SignalsClientColumn)\n    extends QueryFeatureHydrator[PipelineQuery]\n    with Logging {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"USSSignals\")\n  private val fetcher = signalsClientColumn.fetcher\n\n  override val features: Set[Feature[_, _]] = Set(\n    LastNegativeFeedbackTimeFeature,\n  )\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val lastNegativeFeedbackTime = getLastNegativeFeedbackTime(query.getRequiredUserId)\n    lastNegativeFeedbackTime.map { FeatureMap(LastNegativeFeedbackTimeFeature, _) }\n  }\n\n  def getLastNegativeFeedbackTime(userId: Long): Stitch[Option[Time]] = {\n    val enabledNegativeSignalTypes = Seq(\n      SignalType.AccountBlock,\n      SignalType.AccountMute,\n      SignalType.TweetSeeFewer,\n      SignalType.TweetReport,\n      SignalType.TweetDontLike)\n\n    // negative signals\n    val maybeNegativeSignals =\n      enabledNegativeSignalTypes.map { negativeSignal =>\n        SignalRequest(\n          maxResults = Some(1), // Only most recent needed\n          signalType = negativeSignal\n        )\n      }\n\n    val batchSignalRequest =\n      BatchSignalRequest(userId, maybeNegativeSignals, Some(ClientIdentifier.CrMixerHome))\n\n    val signalsStitch = fetcher\n      .fetch(batchSignalRequest)\n      .map { result =>\n        result.v\n          .map(_.signalResponse.toSeq.flatMap {\n            case (_, signals) =>\n              signals\n          }).getOrElse(Seq.empty)\n      }\n    val getTimestamp: UssSignal => Option[Time] = signal =>\n      if (signal.timestamp == 0L) None else Some(Time.fromMilliseconds(signal.timestamp))\n    signalsStitch.map {\n      _.map(getTimestamp).flatten.reduceOption((a, b) => if (a < b) a else b)\n    }\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/LastNonPollingTimeQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.FollowingLastNonPollingTimeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.LastNonPollingTimeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.NonPollingTimesFeature\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.user_session_store.ReadRequest\nimport com.twitter.user_session_store.ReadWriteUserSessionStore\nimport com.twitter.user_session_store.UserSessionDataset\nimport com.twitter.user_session_store.UserSessionDataset.UserSessionDataset\nimport com.twitter.util.Time\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class LastNonPollingTimeQueryFeatureHydrator @Inject() (\n  userSessionStore: ReadWriteUserSessionStore)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"LastNonPollingTime\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    FollowingLastNonPollingTimeFeature,\n    LastNonPollingTimeFeature,\n    NonPollingTimesFeature\n  )\n\n  private val datasets: Set[UserSessionDataset] = Set(UserSessionDataset.NonPollingTimes)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    userSessionStore\n      .read(ReadRequest(query.getRequiredUserId, datasets))\n      .map { userSession =>\n        val nonPollingTimestamps = userSession.flatMap(_.nonPollingTimestamps)\n\n        val lastNonPollingTime = nonPollingTimestamps\n          .flatMap(_.nonPollingTimestampsMs.headOption)\n          .map(Time.fromMilliseconds)\n\n        val followingLastNonPollingTime = nonPollingTimestamps\n          .flatMap(_.mostRecentHomeLatestNonPollingTimestampMs)\n          .map(Time.fromMilliseconds)\n\n        val nonPollingTimes = nonPollingTimestamps\n          .map(_.nonPollingTimestampsMs)\n          .getOrElse(Seq.empty)\n\n        FeatureMapBuilder()\n          .add(FollowingLastNonPollingTimeFeature, followingLastNonPollingTime)\n          .add(LastNonPollingTimeFeature, lastNonPollingTime)\n          .add(NonPollingTimesFeature, nonPollingTimes)\n          .build()\n      }\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.9)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/ListIdsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.param.HomeGlobalParams.ListMandarinTweetsParams.ListMandarinTweetsEnable\nimport com.twitter.home_mixer.param.HomeGlobalParams.ListMandarinTweetsParams.ListMandarinTweetsLists\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.socialgraph.{thriftscala => sg}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.SocialGraph\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase object ListIdsFeature extends FeatureWithDefaultOnFailure[PipelineQuery, Seq[Long]] {\n  override val defaultValue: Seq[Long] = Seq.empty\n}\n\n@Singleton\nclass ListIdsQueryFeatureHydrator @Inject() (socialGraph: SocialGraph)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"ListIds\")\n\n  override val features: Set[Feature[_, _]] = Set(ListIdsFeature)\n\n  private val MaxListsToFetch = 20\n\n  private def buildIdsRequest(userId: Long, relationshipType: sg.RelationshipType) = sg.IdsRequest(\n    relationships = Seq(\n      sg.SrcRelationship(userId, relationshipType, hasRelationship = true),\n      sg.SrcRelationship(userId, sg.RelationshipType.ListMuting, hasRelationship = false)\n    ),\n    pageRequest = Some(sg.PageRequest(selectAll = Some(false), count = Some(MaxListsToFetch))),\n    context = Some(sg.LookupContext(performUnion = Some(false), includeAll = Some(false)))\n  )\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n\n    val subscribedRequest = buildIdsRequest(userId, sg.RelationshipType.ListIsSubscriber)\n    val ownedRequest = buildIdsRequest(userId, sg.RelationshipType.ListOwning)\n\n    Stitch.join(socialGraph.ids(ownedRequest), socialGraph.ids(subscribedRequest)).map {\n      case (ownedResponse, subscribedResponse) =>\n        val recommendedListIds =\n          if (query.params(ListMandarinTweetsEnable))\n            query.params(ListMandarinTweetsLists)\n          else\n            Seq.empty\n\n        val ids = (ownedResponse.ids ++ subscribedResponse.ids ++ recommendedListIds).distinct\n\n        FeatureMapBuilder().add(ListIdsFeature, ids).build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/MediaClusterIdFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.MediaCategoryFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaClusterIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaIdsFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MediaClipClusterIdInMemCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MediaClusterId95Store\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableMediaClusterFeatureHydrationParam\nimport com.twitter.mediaservices.commons.thriftscala.MediaCategory\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.InProcessCache\nimport com.twitter.stitch.Stitch\n\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport com.twitter.storehaus.ReadableStore\n\n@Singleton\nclass MediaClusterIdFeatureHydrator @Inject() (\n  @Named(MediaClusterId95Store) clusterIdStore: ReadableStore[Long, Long],\n  @Named(MediaClipClusterIdInMemCache) mediaClipClusterIdInMemCache: InProcessCache[\n    Long,\n    Option[Option[Long]]\n  ],\n  statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery]\n    with WithDefaultFeatureMap {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"MediaClusterId\")\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableMediaClusterFeatureHydrationParam)\n\n  override val features: Set[Feature[_, _]] = Set(TweetMediaClusterIdsFeature)\n\n  override val defaultFeatureMap: FeatureMap =\n    FeatureMap(TweetMediaClusterIdsFeature, Map.empty[Long, Long])\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  private val cacheHitCounter = scopedStatsReceiver.counter(\"cache/hit\")\n  private val cacheMissCounter = scopedStatsReceiver.counter(\"cache/miss\")\n  private val fetchExceptionCounter =\n    scopedStatsReceiver.counter(\"getFromCacheOrFetch/exception\")\n\n  private def getFromCacheOrFetch(tweetId: Long): Stitch[Option[Option[Long]]] = {\n    mediaClipClusterIdInMemCache\n      .get(tweetId)\n      .map { cachedValue =>\n        cacheHitCounter.incr()\n        Stitch.value(cachedValue)\n      }.getOrElse {\n        cacheMissCounter.incr()\n        // Use the store which handles memcache + Manhattan fallback\n        Stitch\n          .callFuture(clusterIdStore.get(tweetId)).map { result =>\n            result match {\n              case Some(clusterId) =>\n                keyFoundCounter.incr()\n                val finalResult = Some(Some(clusterId))\n                mediaClipClusterIdInMemCache.set(tweetId, finalResult)\n                finalResult\n              case None =>\n                keyNotFoundCounter.incr()\n                mediaClipClusterIdInMemCache.set(tweetId, Some(None))\n                Some(None)\n            }\n          }.handle {\n            case ex: Exception =>\n              fetchExceptionCounter.incr()\n              ex.printStackTrace()\n              None\n          }\n      }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    Stitch.collect(\n      candidates.map { candidate =>\n        val mediaIds = candidate.features.getOrElse(TweetMediaIdsFeature, Seq.empty[Long])\n        val mediaCategory = candidate.features.getOrElse(MediaCategoryFeature, None)\n        if (mediaCategory.contains(MediaCategory.TweetVideo) || mediaCategory.contains(\n            MediaCategory.AmplifyVideo)) {\n          val items: Seq[Stitch[Option[(Long, Long)]]] = mediaIds.map { mediaId =>\n            getFromCacheOrFetch(candidate.candidate.id).map {\n              case Some(Some(mediaClusterId)) =>\n                keyFoundCounter.incr()\n                Some(mediaId -> mediaClusterId)\n              case _ =>\n                keyNotFoundCounter.incr()\n                None\n            }\n          }\n\n          Stitch.collect(items).map { results =>\n            val mediaClusterMap =\n              results.flatten.toMap // Flatten to remove None and create Map[Long, Long]\n            FeatureMapBuilder().add(TweetMediaClusterIdsFeature, mediaClusterMap).build()\n          }\n        } else {\n          Stitch.value(\n            FeatureMapBuilder().add(TweetMediaClusterIdsFeature, Map.empty[Long, Long]).build())\n        }\n      }\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/MediaCompletionRateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaCompletionRateFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaIdsFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithModerateTimeout\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MediaCompletionRateInMemCache\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.InProcessCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.generated.client.analytics.video.VideoCompletionRateOnApiMediaClientColumn\nimport com.twitter.strato.graphql.thriftscala.ApiMediaKey.TweetVideo\nimport com.twitter.strato.graphql.thriftscala.TweetVideoKey\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass MediaCompletionRateFeatureHydrator @Inject() (\n  @Named(BatchedStratoClientWithModerateTimeout) stratoClient: Client,\n  @Named(MediaCompletionRateInMemCache) mediaCompletionRateInMemCache: InProcessCache[Long, Double],\n  statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"MediaCompletionRate\")\n\n  override val features: Set[Feature[_, _]] = Set(TweetMediaCompletionRateFeature)\n\n  private val completionRateFetcher = stratoClient.fetcher[\n    VideoCompletionRateOnApiMediaClientColumn.Key,\n    VideoCompletionRateOnApiMediaClientColumn.View,\n    VideoCompletionRateOnApiMediaClientColumn.Value\n  ](VideoCompletionRateOnApiMediaClientColumn.Path)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  private val cacheHitCounter = scopedStatsReceiver.counter(\"cache/hit\")\n  private val cacheMissCounter = scopedStatsReceiver.counter(\"cache/miss\")\n  private val fetchExceptionCounter =\n    scopedStatsReceiver.counter(\"getFromCacheOrFetch/exception\")\n\n  private def getFromCacheOrFetch(mediaId: Long): Stitch[Option[Double]] = {\n    mediaCompletionRateInMemCache\n      .get(mediaId)\n      .map { cachedValue =>\n        cacheHitCounter.incr()\n        Stitch.value(Some(cachedValue))\n      }.getOrElse {\n        cacheMissCounter.incr()\n        val key = TweetVideo(TweetVideoKey(mediaId = mediaId, tweetId = 0L))\n        completionRateFetcher\n          .fetch(key)\n          .flatMap { result =>\n            val completionRate = result.v.map(_.organic)\n            mediaCompletionRateInMemCache.set(mediaId, completionRate.getOrElse(0.0))\n            Stitch.value(completionRate)\n          }\n      }.handle {\n        case _: Exception =>\n          fetchExceptionCounter.incr()\n          None\n      }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    val featureMaps = candidates.map { candidate =>\n      val hasVideo = candidate.features.getOrElse(HasVideoFeature, false)\n      val mediaIds = candidate.features.getOrElse(TweetMediaIdsFeature, Seq.empty[Long])\n      val inNetwork = candidate.features.getOrElse(FromInNetworkSourceFeature, false)\n      val completionRates = if (hasVideo && !inNetwork) {\n        mediaIds.map { mediaId =>\n          getFromCacheOrFetch(mediaId).map {\n            case Some(completionRate) =>\n              keyFoundCounter.incr()\n              Some(completionRate)\n            case _ =>\n              keyNotFoundCounter.incr()\n              None\n          }\n        }\n      } else Seq.empty[Stitch[Option[Double]]]\n\n      Stitch.collectToTry(completionRates).map { results =>\n        val completionRates = results.map(_.toOption.flatten).flatten\n        val maxCompletionRate = if (completionRates.nonEmpty) Some(completionRates.max) else None\n        FeatureMap(TweetMediaCompletionRateFeature, maxCompletionRate)\n      }\n    }\n    Stitch.collect(featureMaps)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/MultiModalEmbeddingsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.MultiModalEmbeddingsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.ExpiringLruInProcessCache\nimport com.twitter.servo.cache.InProcessCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.searchai.storage.PostMultimodalEmbeddingMhClientColumn\nimport com.twitter.util.Duration\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.util.Random\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.util.Try\n\nobject MultiModalEmbeddingsFeatureHydrator {\n  private val BaseTTL = 15\n  private val TTL: Duration = (BaseTTL + Random.nextInt(10)).minutes\n  private val maximumCacheSize = 15000 // 15000 * 1536 * 8 bytes = <200Mb\n\n  val cache: InProcessCache[Long, Option[Option[Seq[Double]]]] =\n    new ExpiringLruInProcessCache(ttl = TTL, maximumSize = maximumCacheSize)\n\n  private val embeddingFetcherModelVersion = \"v1\"\n}\n\n@Singleton\nclass MultiModalEmbeddingsFeatureHydrator @Inject() (\n  postMultimodalEmbeddingMhClientColumn: PostMultimodalEmbeddingMhClientColumn,\n  statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"MultiModalEmbeddings\")\n\n  override val features: Set[Feature[_, _]] = Set(MultiModalEmbeddingsFeature)\n\n  private val embeddingFetcher = postMultimodalEmbeddingMhClientColumn.fetcher\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  private val fetchExceptionCounter = scopedStatsReceiver.counter(\"getFromCacheOrFetch/exception\")\n\n  private def getEmbedding(\n    tweetId: Long\n  ): Stitch[Option[Option[Seq[Double]]]] = {\n    val embeddingStitchFromCache: Option[Stitch[Option[Option[Seq[Double]]]]] =\n      MultiModalEmbeddingsFeatureHydrator.cache\n        .get(tweetId)\n        .map(Stitch.value)\n\n    embeddingStitchFromCache match {\n      case Some(stitch: Stitch[Option[Option[Seq[Double]]]]) => stitch\n      case None =>\n        val embeddingStitch = embeddingFetcher\n          .fetch((tweetId, MultiModalEmbeddingsFeatureHydrator.embeddingFetcherModelVersion)).map {\n            result =>\n              result.v match {\n                case Some(tweetEmbedding)\n                    if tweetEmbedding != null && tweetEmbedding.embedding1 != null && tweetEmbedding.embedding1.isDefined =>\n                  keyFoundCounter.incr()\n                  val embedding = tweetEmbedding.embedding1\n                  MultiModalEmbeddingsFeatureHydrator.cache.set(tweetId, Some(embedding))\n                  Some(embedding)\n                case _ =>\n                  keyNotFoundCounter.incr()\n                  MultiModalEmbeddingsFeatureHydrator.cache.set(tweetId, Some(None))\n                  None\n              }\n          }.handle {\n            case ex: Exception =>\n              fetchExceptionCounter.incr()\n              None\n          }\n        embeddingStitch\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    val stitchesWithTry: Stitch[Seq[Try[Option[Option[Seq[Double]]]]]] = Stitch.collectToTry {\n      candidates.map { candidate =>\n        val tweetId: Long = candidate.candidate.id\n        getEmbedding(tweetId)\n      }\n    }\n\n    stitchesWithTry.map { tryVal =>\n      tryVal.flatMap(_.toOption).map {\n        case Some(embedding) => FeatureMap(MultiModalEmbeddingsFeature, embedding)\n        case _ => FeatureMap(MultiModalEmbeddingsFeature, None)\n      }\n    }\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/NamesFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.gizmoduck.{thriftscala => gt}\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.gizmoduck.Gizmoduck\nimport com.twitter.util.Return\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nprotected case class ProfileNames(screenName: String, realName: String)\n\n@Singleton\nclass NamesFeatureHydrator @Inject() (gizmoduck: Gizmoduck)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"Names\")\n\n  override val features: Set[Feature[_, _]] = Set(ScreenNamesFeature, RealNamesFeature)\n\n  override def onlyIf(query: PipelineQuery): Boolean = query.product match {\n    case FollowingProduct => false\n    case _ => true\n  }\n\n  private val queryFields: Set[gt.QueryFields] = Set(gt.QueryFields.Profile)\n\n  /**\n   * The UI currently only ever displays the first 2 names in social context lines\n   * E.g. \"User and 3 others like\" or \"UserA and UserB liked\"\n   */\n  private val MaxCountUsers = 2\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n\n    val candidateUserIdsMap = candidates.map { candidate =>\n      candidate.candidate.id ->\n        (candidate.features.getOrElse(FavoritedByUserIdsFeature, Nil).take(MaxCountUsers) ++\n          candidate.features.getOrElse(FollowedByUserIdsFeature, Nil).take(MaxCountUsers) ++\n          candidate.features.getOrElse(AuthorIdFeature, None) ++\n          candidate.features.getOrElse(SourceUserIdFeature, None)).distinct\n    }.toMap\n\n    val distinctUserIds = candidateUserIdsMap.values.flatten.toSeq.distinct\n\n    Stitch\n      .collectToTry(distinctUserIds.map(userId => gizmoduck.getUserById(userId, queryFields)))\n      .map { allUsers =>\n        val idToProfileNamesMap = allUsers.flatMap {\n          case Return(allUser) =>\n            allUser.profile\n              .map(profile => allUser.id -> ProfileNames(profile.screenName, profile.name))\n          case _ => None\n        }.toMap\n\n        val validUserIds = idToProfileNamesMap.keySet\n\n        candidates.map { candidate =>\n          val combinedMap = candidateUserIdsMap\n            .getOrElse(candidate.candidate.id, Nil)\n            .flatMap {\n              case userId if validUserIds.contains(userId) =>\n                idToProfileNamesMap.get(userId).map(profileNames => userId -> profileNames)\n              case _ => None\n            }\n\n          val perCandidateRealNameMap = combinedMap.map { case (k, v) => k -> v.realName }.toMap\n          val perCandidateScreenNameMap = combinedMap.map { case (k, v) => k -> v.screenName }.toMap\n\n          FeatureMapBuilder()\n            .add(ScreenNamesFeature, perCandidateScreenNameMap)\n            .add(RealNamesFeature, perCandidateRealNameMap)\n            .build()\n        }\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/NaviClientConfigQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.NaviClientConfigFeature\nimport com.twitter.home_mixer.model.NaviClientConfig\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelIdParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.NaviGPUBatchSizeParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ProdModelIdParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.UseGPUNaviClusterParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.UseGPUNaviClusterTestUsersParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.UseRealtimeNaviClusterParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.UseSecondaryNaviClusterParam\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecap\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecapGPU\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecapRealtime\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecapSecondary\nimport com.twitter.product_mixer.component_library.module.TestUserMapper\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass NaviClientConfigQueryFeatureHydrator @Inject() (\n  testUserMapper: TestUserMapper)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"NaviClientConfig\")\n\n  override val features: Set[Feature[_, _]] = Set(NaviClientConfigFeature)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    // Use experimental realtime cluster\n    val useRealtimeCluster = query.params(UseRealtimeNaviClusterParam)\n    // Use secondary navi cluster for handling peak traffic\n    val useSecondaryCluster = query.params(UseSecondaryNaviClusterParam)\n\n    val modelId = query.params(ModelIdParam)\n    val prodModelId = query.params(ProdModelIdParam)\n    val isTestUser = testUserMapper.isTestUser(query.clientContext)\n    val useGPUCluster = prodModelId == modelId && (query.params(UseGPUNaviClusterParam) ||\n      (isTestUser && query.params(UseGPUNaviClusterTestUsersParam)))\n\n    val gpuBatchSize =\n      if (useGPUCluster) Some(query.params(NaviGPUBatchSizeParam).toInt)\n      else None\n\n    val clusterClientPriorityList = Seq(\n      (useGPUCluster, NaviModelClientHomeRecapGPU),\n      (useRealtimeCluster, NaviModelClientHomeRecapRealtime),\n      (useSecondaryCluster, NaviModelClientHomeRecapSecondary)\n    )\n\n    val clientName = clusterClientPriorityList\n      .collectFirst {\n        case (true, name) => name\n      }.getOrElse(NaviModelClientHomeRecap)\n\n    val clusterStr = clientName match {\n      case NaviModelClientHomeRecapGPU => \"GPU\"\n      case NaviModelClientHomeRecapRealtime => \"Realtime\"\n      case _ => \"\"\n    }\n\n    val naviConfig = NaviClientConfig(clientName, gpuBatchSize, clusterStr)\n    Stitch.value(FeatureMap(NaviClientConfigFeature, naviConfig))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/NaviVideoClientConfigQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.NaviClientConfigFeature\nimport com.twitter.home_mixer.model.NaviClientConfig\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring._\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecapVideo\nimport com.twitter.product_mixer.component_library.module.TestUserMapper\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass NaviVideoClientConfigQueryFeatureHydrator @Inject() (\n  testUserMapper: TestUserMapper)\n    extends NaviClientConfigQueryFeatureHydrator(testUserMapper) {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"NaviVideoClientConfig\")\n\n  override val features: Set[Feature[_, _]] = Set(NaviClientConfigFeature)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    query.params(UseVideoNaviClusterParam) match {\n      case true =>\n        val config = NaviClientConfig(\n          clientName = NaviModelClientHomeRecapVideo,\n          customizedBatchSize = None,\n          clusterStr = \"Video\"\n        )\n        Stitch.value(FeatureMap(NaviClientConfigFeature, config))\n      case _ => super.hydrate(query)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/OnPremRealGraphQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableOnPremRealGraphQueryFeatures\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.recommendations.interaction_graph.on_prem.RealGraphUserFeaturesOnUserClientColumn\nimport com.twitter.timelines.real_graph.{thriftscala => rg}\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass OnPremRealGraphQueryFeatureHydrator @Inject() (\n  realGraphUserFeaturesClientColumn: RealGraphUserFeaturesOnUserClientColumn)\n    extends QueryFeatureHydrator[PipelineQuery]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"OnPremRealGraphFeatures\")\n\n  override val features: Set[Feature[_, _]] = Set(RealGraphFeatures)\n\n  override def onlyIf(\n    query: PipelineQuery\n  ): Boolean = {\n    query.params(EnableOnPremRealGraphQueryFeatures)\n  }\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    realGraphUserFeaturesClientColumn.fetcher\n      .fetch(query.getRequiredUserId).map { userSession =>\n        val realGraphFeaturesMap = userSession.v.flatMap { userSession =>\n          userSession.realGraphFeatures.collect {\n            case rg.RealGraphFeatures.V1(realGraphFeatures) =>\n              val edgeFeatures =\n                realGraphFeatures.edgeFeatures ++ realGraphFeatures.oonEdgeFeatures\n              edgeFeatures.map { edge => edge.destId -> edge }.toMap\n          }\n        }\n\n        FeatureMapBuilder().add(RealGraphFeatures, realGraphFeaturesMap).build()\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/OptimizerWeightsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.PredictedScoreFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\n\n/*\n * This Hydrator updates the model scoring weights with the optimizer weights.\n */\n\nclass OptimizerWeightsQueryFeatureHydrator @Inject() (\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"OptimizerWeights\")\n\n  private val EnabledPredictedScoreFeatures: Seq[PredictedScoreFeature] =\n    PredictedScoreFeature.PredictedScoreFeatures\n\n  override val features: Set[Feature[_, _]] = {\n    EnabledPredictedScoreFeatures.map(_.weightQueryFeature).toSet\n  }\n\n  private val BIG_PRIME = 65537\n  private def hashFn(userId: Long, seed: Long, numBuckets: Int): Int = {\n    val x = userId % BIG_PRIME\n    val y = seed % BIG_PRIME\n    val h = (31 * x + 41 * y) * 53\n    (h % BIG_PRIME).toInt % numBuckets\n  }\n\n  private val successCounter =\n    statsReceiver.counter(\"success\")\n  private val failureCounter = statsReceiver.counter(\"failure\")\n\n  private def get_user_weights(\n    userId: Long,\n    optimizerParams: OptimizerParams\n  ): Map[String, Double] = {\n    val weightIndex =\n      hashFn(userId, seed = optimizerParams.epochTimestamp, numBuckets = optimizerParams.nBuckets)\n    val userWeights = optimizerParams.optimizerWeights(weightIndex)\n    userWeights\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val optimizerParams = query.features\n      .flatMap(_.getOrElse(HeartbeatOptimizerWeightsFeature, None))\n\n    val featureMapBuilder = FeatureMapBuilder()\n\n    optimizerParams match {\n      case Some(optimizerParams) =>\n        if (optimizerParams.optimizerWeights.size != optimizerParams.nBuckets) {\n          failureCounter.incr(1)\n          // Fall back to default weights when size mismatch\n          for (predictedScoreFeature <- EnabledPredictedScoreFeatures) {\n            val currentWeight = query.params(predictedScoreFeature.modelWeightParam)\n            featureMapBuilder.add(predictedScoreFeature.weightQueryFeature, Some(currentWeight))\n          }\n        } else {\n          val userWeights: Map[String, Double] =\n            get_user_weights(query.getRequiredUserId, optimizerParams)\n\n          for (predictedScoreFeature <- EnabledPredictedScoreFeatures) {\n            val currentWeight = query.params(predictedScoreFeature.modelWeightParam)\n            val weight_name = predictedScoreFeature.modelWeightParam.name\n            val optimizerWeight =\n              userWeights.getOrElse(weight_name, currentWeight)\n            featureMapBuilder.add(predictedScoreFeature.weightQueryFeature, Some(optimizerWeight))\n          }\n          successCounter.incr(1)\n        }\n\n      case _ =>\n        // When HeartbeatOptimizerWeightsFeature is None, use default weights\n        for (predictedScoreFeature <- EnabledPredictedScoreFeatures) {\n          val currentWeight = query.params(predictedScoreFeature.modelWeightParam)\n          featureMapBuilder.add(predictedScoreFeature.weightQueryFeature, Some(currentWeight))\n        }\n        failureCounter.incr(1)\n    }\n\n    Stitch.value(featureMapBuilder.build())\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/OriginalAuthorLargeEmbeddingsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeLargeEmbeddingsFeatures.OriginalAuthorLargeEmbeddingsFeature\nimport com.twitter.home_mixer.model.HomeLargeEmbeddingsFeatures.OriginalAuthorLargeEmbeddingsKeyFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableLargeEmbeddingsFeatureHydrationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelNameParam\nimport com.twitter.home_mixer.util.CandidatesUtil.getOriginalAuthorId\nimport com.twitter.home_mixer_features.{thriftscala => hmf}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.large_embeddings.HashingFeatureParams\nimport com.twitter.timelines.prediction.adapters.large_embeddings.HomeMixerLargeEmbeddingsFeatureHydrator\nimport com.twitter.timelines.prediction.adapters.large_embeddings.LargeEmbeddingsAdapter\nimport com.twitter.timelines.prediction.adapters.large_embeddings.OriginalAuthorLargeEmbeddingsAdapter\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass OriginalAuthorLargeEmbeddingsFeatureHydrator @Inject() (\n  statsReceiver: StatsReceiver,\n  override val homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery]\n    with HomeMixerLargeEmbeddingsFeatureHydrator {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"OriginalAuthorLargeEmbeddings\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(OriginalAuthorLargeEmbeddingsFeature, OriginalAuthorLargeEmbeddingsKeyFeature)\n\n  override val adapter: LargeEmbeddingsAdapter = OriginalAuthorLargeEmbeddingsAdapter\n\n  override val cacheType: hmf.Cache = hmf.Cache.AuthorLargeEmbeddings\n\n  override val scopedStatsReceiver: StatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableLargeEmbeddingsFeatureHydrationParam)\n\n  // Hashing Features\n  override val defaultHashingFeatureParams: HashingFeatureParams = HashingFeatureParams(\n    scales = Seq(3384241453L, 3372414709L),\n    biases = Seq(1649585795L, 3131243219L),\n    modulus = 3957384397L,\n    bucketSize = 3000000L,\n  )\n\n  override val modelName2HashingFeatureParams: Map[String, HashingFeatureParams] = Map(\n    \"hr_video_prod__v3_realtime\" -> HashingFeatureParams(\n      scales = Seq(113294449L, 601841083L),\n      biases = Seq(2231299001L, 841367196L),\n      modulus = 2343760591L,\n      bucketSize = 3000000L,\n    ),\n    \"hr_video_prod__v2_lembeds\" -> HashingFeatureParams(\n      scales = Seq(787140070L, 633713480L),\n      biases = Seq(427768658L, 911091889L),\n      modulus = 2888480981L,\n      bucketSize = 300000L,\n    ),\n    \"hr_prod__v4_embeds_230M\" -> HashingFeatureParams(\n      scales = Seq(371965780L, 328930218L),\n      biases = Seq(139686260L, 37755056L),\n      modulus = 631860353L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_prod__v5_embeds_230M_and_transformer\" -> HashingFeatureParams(\n      scales = Seq(371965780L, 328930218L),\n      biases = Seq(139686260L, 37755056L),\n      modulus = 631860353L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_prod__v5_watchtime\" -> HashingFeatureParams(\n      scales = Seq(2328530078L, 2844016377L),\n      biases = Seq(1352496802L, 3011003330L),\n      modulus = 3979826519L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_prod__v6_transformer_v2\" -> HashingFeatureParams(\n      scales = Seq(371965780L, 328930218L),\n      biases = Seq(139686260L, 37755056L),\n      modulus = 631860353L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_prod__v6_mixed_training\" -> HashingFeatureParams(\n      scales = Seq(371965780L, 328930218L),\n      biases = Seq(139686260L, 37755056L),\n      modulus = 631860353L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_prod__v6_transformer_v2_kafka_merge_join\" -> HashingFeatureParams(\n      scales = Seq(371965780L, 328930218L),\n      biases = Seq(139686260L, 37755056L),\n      modulus = 631860353L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_prod__v6_transformer_v2_realtime_debias_21apr\" -> HashingFeatureParams(\n      scales = Seq(371965780L, 328930218L),\n      biases = Seq(139686260L, 37755056L),\n      modulus = 631860353L,\n      bucketSize = 30000000L,\n    ),\n    \"hr_video_prod__v4_realtime\" -> HashingFeatureParams(\n      scales = Seq(113294449L, 601841083L),\n      biases = Seq(2231299001L, 841367196L),\n      modulus = 2343760591L,\n      bucketSize = 3000000L,\n    ),\n    \"hr_video_prod__v4_realtime_mergehead\" -> HashingFeatureParams(\n      scales = Seq(113294449L, 601841083L),\n      biases = Seq(2231299001L, 841367196L),\n      modulus = 2343760591L,\n      bucketSize = 3000000L,\n    ),\n  )\n\n  private val batchSize = 25\n\n  private def getBatchedFeatureMap(\n    modelName: String,\n    candidatesBatch: Seq[CandidateWithFeatures[TweetCandidate]],\n  ): Future[Seq[FeatureMap]] = {\n    val originalAuthorIds =\n      candidatesBatch.map { candidate =>\n        getOriginalAuthorId(candidate.features).getOrElse(0L)\n      }\n\n    getLargeEmbeddings(originalAuthorIds, modelName).map { responses =>\n      responses.map { largeEmbeddingResponse =>\n        FeatureMapBuilder()\n          .add(OriginalAuthorLargeEmbeddingsFeature, largeEmbeddingResponse.dataRecord)\n          .add(OriginalAuthorLargeEmbeddingsKeyFeature, largeEmbeddingResponse.hashedKeys)\n          .build()\n      }\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val modelName = query.params(ModelNameParam)\n    OffloadFuturePools.offloadBatchSeqToFutureSeq(\n      candidates,\n      getBatchedFeatureMap(modelName, _),\n      batchSize\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/OriginalTweetLargeEmbeddingsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeLargeEmbeddingsFeatures.OriginalTweetLargeEmbeddingsFeature\nimport com.twitter.home_mixer.model.HomeLargeEmbeddingsFeatures.OriginalTweetLargeEmbeddingsKeyFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableLargeEmbeddingsFeatureHydrationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelNameParam\nimport com.twitter.home_mixer.util.CandidatesUtil.getOriginalTweetId\nimport com.twitter.home_mixer_features.{thriftscala => hmf}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.large_embeddings.HashingFeatureParams\nimport com.twitter.timelines.prediction.adapters.large_embeddings.HomeMixerLargeEmbeddingsFeatureHydrator\nimport com.twitter.timelines.prediction.adapters.large_embeddings.LargeEmbeddingsAdapter\nimport com.twitter.timelines.prediction.adapters.large_embeddings.OriginalTweetLargeEmbeddingsAdapter\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass OriginalTweetLargeEmbeddingsFeatureHydrator @Inject() (\n  statsReceiver: StatsReceiver,\n  override val homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery]\n    with HomeMixerLargeEmbeddingsFeatureHydrator {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"OriginalTweetLargeEmbeddings\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(OriginalTweetLargeEmbeddingsFeature, OriginalTweetLargeEmbeddingsKeyFeature)\n\n  override val adapter: LargeEmbeddingsAdapter = OriginalTweetLargeEmbeddingsAdapter\n\n  override val cacheType: hmf.Cache = hmf.Cache.TweetLargeEmbeddings\n\n  override val scopedStatsReceiver: StatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableLargeEmbeddingsFeatureHydrationParam)\n\n  // Hashing Features\n  override val defaultHashingFeatureParams: HashingFeatureParams = HashingFeatureParams(\n    scales = Seq(1131302000L, 303023026L),\n    biases = Seq(799473858L, 600426834L),\n    modulus = 3588720353L,\n    bucketSize = 10000000L,\n  )\n\n  override val modelName2HashingFeatureParams: Map[String, HashingFeatureParams] = Map(\n    \"hr_video_prod__v3_realtime\" -> HashingFeatureParams(\n      scales = Seq(1487022661L, 1399245971L),\n      biases = Seq(1372992088L, 632996194L),\n      modulus = 2865175829L,\n      bucketSize = 10000000L,\n    ),\n    \"hr_video_prod__v2_lembeds\" -> HashingFeatureParams(\n      scales = Seq(2516541900L, 2376187492L),\n      biases = Seq(3022238687L, 1571354734L),\n      modulus = 3047336911L,\n      bucketSize = 1000000L,\n    ),\n    \"hr_prod__v4_embeds_230M\" -> HashingFeatureParams(\n      scales = Seq(2161410491L, 1754358832L),\n      biases = Seq(296686044L, 1959990826L),\n      modulus = 2361375383L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v5_embeds_230M_and_transformer\" -> HashingFeatureParams(\n      scales = Seq(2161410491L, 1754358832L),\n      biases = Seq(296686044L, 1959990826L),\n      modulus = 2361375383L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v5_watchtime\" -> HashingFeatureParams(\n      scales = Seq(407033648L, 940305868L),\n      biases = Seq(494266171L, 269596788L),\n      modulus = 949146421L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v6_transformer_v2\" -> HashingFeatureParams(\n      scales = Seq(2161410491L, 1754358832L),\n      biases = Seq(296686044L, 1959990826L),\n      modulus = 2361375383L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v6_mixed_training\" -> HashingFeatureParams(\n      scales = Seq(2161410491L, 1754358832L),\n      biases = Seq(296686044L, 1959990826L),\n      modulus = 2361375383L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v6_transformer_v2_kafka_merge_join\" -> HashingFeatureParams(\n      scales = Seq(2161410491L, 1754358832L),\n      biases = Seq(296686044L, 1959990826L),\n      modulus = 2361375383L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v6_transformer_v2_realtime_debias_21apr\" -> HashingFeatureParams(\n      scales = Seq(2161410491L, 1754358832L),\n      biases = Seq(296686044L, 1959990826L),\n      modulus = 2361375383L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_video_prod__v4_realtime\" -> HashingFeatureParams(\n      scales = Seq(1487022661L, 1399245971L),\n      biases = Seq(1372992088L, 632996194L),\n      modulus = 2865175829L,\n      bucketSize = 10000000L,\n    ),\n    \"hr_video_prod__v4_realtime_mergehead\" -> HashingFeatureParams(\n      scales = Seq(1487022661L, 1399245971L),\n      biases = Seq(1372992088L, 632996194L),\n      modulus = 2865175829L,\n      bucketSize = 10000000L,\n    ),\n  )\n\n  private val batchSize = 25\n\n  private def getBatchedFeatureMap(\n    modelName: String,\n    candidatesBatch: Seq[CandidateWithFeatures[TweetCandidate]],\n  ): Future[Seq[FeatureMap]] = {\n    val originalTweetIds =\n      candidatesBatch.map { candidate =>\n        getOriginalTweetId(candidate.candidate, candidate.features)\n      }\n\n    getLargeEmbeddings(originalTweetIds, modelName).map { responses =>\n      responses.map { largeEmbeddingResponse =>\n        FeatureMapBuilder()\n          .add(OriginalTweetLargeEmbeddingsFeature, largeEmbeddingResponse.dataRecord)\n          .add(OriginalTweetLargeEmbeddingsKeyFeature, largeEmbeddingResponse.hashedKeys)\n          .build()\n      }\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val modelName = query.params(ModelNameParam)\n    OffloadFuturePools.offloadBatchSeqToFutureSeq(\n      candidates,\n      getBatchedFeatureMap(modelName, _),\n      batchSize\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/PersistenceStoreQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.common_internal.analytics.twitter_client_user_agent_parser.UserAgent\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedAuthorIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTweetIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTweetPreviewIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.WhoToFollowExcludedUserIdsFeature\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.param.HomeGlobalParams.ExcludeServedAuthorIdsDurationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.ExcludeServedTweetIdsDurationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.ExcludeServedTweetIdsNumberParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelinemixer.clients.persistence.TimelineResponseBatchesClient\nimport com.twitter.timelinemixer.clients.persistence.TimelineResponseV3\nimport com.twitter.timelines.util.client_info.ClientPlatform\nimport com.twitter.timelineservice.model.TimelineQuery\nimport com.twitter.timelineservice.model.core.TimelineKind\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport com.twitter.util.Time\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class PersistenceStoreQueryFeatureHydrator @Inject() (\n  timelineResponseBatchesClient: TimelineResponseBatchesClient[TimelineResponseV3],\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"PersistenceStore\")\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val servedTweetIdsSizeStat = scopedStatsReceiver.stat(\"ServedTweetIdsSize\")\n\n  private val WhoToFollowExcludedUserIdsLimit = 1000\n  private val ServedTweetPreviewIdsDuration = 10.hours\n  private val ServedTweetPreviewIdsLimit = 10\n\n  override val features: Set[Feature[_, _]] = Set(\n    ServedTweetIdsFeature,\n    ServedTweetPreviewIdsFeature,\n    PersistenceEntriesFeature,\n    WhoToFollowExcludedUserIdsFeature,\n    ServedAuthorIdsFeature\n  )\n\n  private val supportedClients = Seq(\n    ClientPlatform.IPhone,\n    ClientPlatform.IPad,\n    ClientPlatform.Mac,\n    ClientPlatform.Android,\n    ClientPlatform.Web,\n    ClientPlatform.RWeb,\n    ClientPlatform.TweetDeckGryphon\n  )\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val timelineKind = query.product match {\n      case FollowingProduct => TimelineKind.homeLatest\n      case ForYouProduct => TimelineKind.home\n      case other => throw new UnsupportedOperationException(s\"Unknown product: $other\")\n    }\n    val timelineQuery = TimelineQuery(id = query.getRequiredUserId, kind = timelineKind)\n\n    Stitch.callFuture {\n      timelineResponseBatchesClient\n        .get(query = timelineQuery, clientPlatforms = supportedClients)\n        .map { timelineResponses =>\n          // Note that the WTF entries are not being scoped by ClientPlatform\n          val whoToFollowUserIds = timelineResponses\n            .flatMap { timelineResponse =>\n              timelineResponse.entries\n                .filter(_.entityIdType == EntityIdType.WhoToFollow)\n                .flatMap(_.itemIds.toSeq.flatMap(_.flatMap(_.userId)))\n            }.take(WhoToFollowExcludedUserIdsLimit)\n\n          val clientPlatform = ClientPlatform.fromQueryOptions(\n            clientAppId = query.clientContext.appId,\n            userAgent = query.clientContext.userAgent.flatMap(UserAgent.fromString))\n\n          val recentTimelineItems = timelineResponses\n            .filter(_.clientPlatform == clientPlatform)\n            .sortBy(-_.servedTime.inMilliseconds)\n\n          val servedTweetIds = recentTimelineItems\n            .filter(_.servedTime >= Time.now - query.params(ExcludeServedTweetIdsDurationParam))\n            .flatMap(_.entries\n              .flatMap(_.tweetIds(includeSourceTweets = true)).take(\n                query.params(ExcludeServedTweetIdsNumberParam)))\n\n          servedTweetIdsSizeStat.add(servedTweetIds.size)\n\n          val servedTweetPreviewIds = recentTimelineItems\n            .filter(_.servedTime >= Time.now - ServedTweetPreviewIdsDuration)\n            .flatMap(\n              _.entries\n                .filter(_.entityIdType == EntityIdType.TweetPreview)\n                .flatMap(_.tweetIds(includeSourceTweets = true)).take(ServedTweetPreviewIdsLimit))\n\n          val servedAuthorIds: Map[Long, Seq[Long]] = recentTimelineItems\n            .filter(_.servedTime >= Time.now - query.params(ExcludeServedAuthorIdsDurationParam))\n            .flatMap { timelineResponse =>\n              timelineResponse.entries\n                .filter(_.entityIdType == EntityIdType.Tweet)\n                .flatMap { entry =>\n                  val authorId = entry.sourceAuthorIds.headOption.getOrElse(-1L)\n                  if (authorId != -1L) {\n                    // only include entries with valid author IDs\n                    entry.tweetIds(includeSourceTweets = true).map(tweetId => (authorId, tweetId))\n                  } else Seq.empty\n                }\n            }\n            .groupBy(_._1)\n            .mapValues(_.map(_._2).distinct)\n\n          FeatureMapBuilder()\n            .add(ServedTweetIdsFeature, servedTweetIds)\n            .add(ServedTweetPreviewIdsFeature, servedTweetPreviewIds)\n            .add(PersistenceEntriesFeature, timelineResponses)\n            .add(WhoToFollowExcludedUserIdsFeature, whoToFollowUserIds)\n            .add(ServedAuthorIdsFeature, servedAuthorIds)\n            .build()\n        }\n    }\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.7, 50, 60, 60)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/PerspectiveFilteredSocialContextFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PerspectiveFilteredLikedByUserIdsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.timelineservice.TimelineService\nimport com.twitter.stitch.timelineservice.TimelineService.GetPerspectives\nimport com.twitter.timelineservice.thriftscala.PerspectiveType\nimport com.twitter.timelineservice.thriftscala.PerspectiveType.Favorited\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Filter out unlike edges from liked-by tweets\n * Useful if the likes come from a cache and because UTEG does not fully remove unlike edges.\n */\n@Singleton\nclass PerspectiveFilteredSocialContextFeatureHydrator @Inject() (timelineService: TimelineService)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"PerspectiveFilteredSocialContext\")\n\n  override val features: Set[Feature[_, _]] = Set(PerspectiveFilteredLikedByUserIdsFeature)\n\n  private val MaxCountUsers = 10\n  private val favoritePerspectiveSet: Set[PerspectiveType] = Set(Favorited)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    val engagingUserIdtoTweetId = candidates.flatMap { candidate =>\n      candidate.features\n        .getOrElse(FavoritedByUserIdsFeature, Seq.empty).take(MaxCountUsers)\n        .map(favoritedBy => favoritedBy -> candidate.candidate.id)\n    }\n\n    val queries = engagingUserIdtoTweetId.map {\n      case (userId, tweetId) =>\n        GetPerspectives.Query(userId = userId, tweetId = tweetId, types = favoritePerspectiveSet)\n    }\n\n    Stitch.collect(queries.map(timelineService.getPerspective)).map { perspectiveResults =>\n      val validUserIdTweetIds: Set[(Long, Long)] =\n        queries\n          .zip(perspectiveResults)\n          .collect { case (query, perspective) if perspective.favorited => query }\n          .map(query => (query.userId, query.tweetId))\n          .toSet\n\n      candidates.map { candidate =>\n        val perspectiveFilteredFavoritedByUserIds: Seq[Long] = candidate.features\n          .getOrElse(FavoritedByUserIdsFeature, Seq.empty).take(MaxCountUsers)\n          .filter { userId => validUserIdTweetIds.contains((userId, candidate.candidate.id)) }\n\n        FeatureMapBuilder()\n          .add(PerspectiveFilteredLikedByUserIdsFeature, perspectiveFilteredFavoritedByUserIds)\n          .build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/PhoenixRescoringFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.PhoenixScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.WeightedModelScoreFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnablePhoenixScorerParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnablePhoenixRescoreParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnablePhoenixScoreParam\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject PhoenixRescoringFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"PhoenixRescoring\")\n\n  override val features: Set[Feature[_, _]] = Set(ScoreFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    // Use rescoring to change scoreFeature only when scoring didn't happen using Phoenix\n    val usePhoenixRescoring =\n      query.params(EnablePhoenixScorerParam) &&\n        query.params(EnablePhoenixRescoreParam) &&\n        !query.params(EnablePhoenixScoreParam)\n    val finalScores = candidates.map { candidate =>\n      val score = candidate.features.getOrElse(ScoreFeature, None).getOrElse(0.0)\n      val weightedModelScore =\n        candidate.features.getOrElse(WeightedModelScoreFeature, None).getOrElse(0.0)\n      val phoenixScore = candidate.features.getOrElse(PhoenixScoreFeature, None).getOrElse(0.0)\n      // The multiplier is for heuristics, it might not always be accurate for listwise heuristics\n      if (score == 0.0 | weightedModelScore == 0.0) 0.0\n      else if (usePhoenixRescoring) phoenixScore * (score / weightedModelScore)\n      else score\n    }\n    Stitch.value(finalScores.map(score => FeatureMap(ScoreFeature, Some(score))))\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/PostContextFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GenericPostContextFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.MaxPostContextDuplicatesPerRequest\nimport com.twitter.home_mixer.param.HomeGlobalParams.MaxPostContextPostsPerRequest\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport com.twitter.strato.catalog.Fetch\nimport com.twitter.strato.generated.client.events.urt.PostContextClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.util.Random\n\n@Singleton\nclass PostContextFeatureHydrator @Inject() (\n  postContextClientColumn: PostContextClientColumn)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"PostContext\")\n\n  override val features: Set[Feature[_, _]] = Set(GenericPostContextFeature)\n\n  private val fetcher: Fetcher[Long, Unit, urt.GenericContext] = postContextClientColumn.fetcher\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    val maxPosts = query.params(MaxPostContextPostsPerRequest)\n    val maxDuplicates = query.params(MaxPostContextDuplicatesPerRequest)\n\n    val contextsStitch: Stitch[Seq[Option[urt.GenericContext]]] = Stitch.collect {\n      candidates.map { candidate =>\n        val servedType = candidate.features.getOrElse(ServedTypeFeature, hmt.ServedType.Undefined)\n        val isPromoted = {\n          servedType == hmt.ServedType.ForYouPromoted || servedType == hmt.ServedType.FollowingPromoted\n        }\n\n        val isOriginal = (!candidate.features.getOrElse(IsRetweetFeature, false)) &&\n          candidate.features.getOrElse(InReplyToTweetIdFeature, None).isEmpty\n\n        if (isPromoted || !isOriginal) {\n          // Promoted tweets are not eligible for Post Context.\n          Stitch.value(None)\n        } else {\n          fetcher.fetch(candidate.candidate.id, ()).map {\n            case Fetch.Result(Some(postContext), _) => Some(postContext)\n            case _ => None\n          }\n        }\n      }\n    }\n\n    contextsStitch.map { rawContexts =>\n      val urlCount = scala.collection.mutable.Map.empty[String, Int]\n      val afterDupFilter: Vector[Option[urt.GenericContext]] =\n        rawContexts.map {\n          case Some(ctx) =>\n            val url = ctx.url.url\n            val seen = urlCount.getOrElse(url, 0)\n            if (seen < maxDuplicates) {\n              urlCount.update(url, seen + 1)\n              Some(ctx)\n            } else None // drop: duplicate overflow\n          case None => None\n        }(collection.breakOut)\n\n      val keptIndices: Vector[Int] =\n        afterDupFilter.iterator.zipWithIndex.collect { case (Some(_), idx) => idx }.toVector\n\n      val indicesToDrop: Set[Int] =\n        if (keptIndices.size <= maxPosts) Set.empty\n        else {\n          val numToDrop = keptIndices.size - maxPosts\n          Random.shuffle(keptIndices).take(numToDrop).toSet\n        }\n\n      afterDupFilter.zipWithIndex.map {\n        case (Some(ctx), idx) if !indicesToDrop.contains(idx) =>\n          FeatureMapBuilder().add(GenericPostContextFeature, Some(ctx)).build()\n\n        case _ =>\n          FeatureMapBuilder().add(GenericPostContextFeature, None).build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RateLimitQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerIsRateLimited\nimport com.twitter.home_mixer.param.HomeGlobalParams.RateLimitTestIdsParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.limiter.{thriftscala => t}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class RateLimitQueryFeatureHydrator @Inject() (limiterClient: t.LimitService.MethodPerEndpoint)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"RateLimit\")\n\n  override val features: Set[Feature[_, _]] = Set(ViewerIsRateLimited)\n\n  val RegularFeature = \"graphql_global_regular_tweets_read\"\n  val NewFeature = \"graphql_global_new_tweets_read\"\n  val SoftFeature = \"graphql_global_soft_user_tweets_read\"\n  val SuspendedFeature = \"graphql_global_suspended_user_tweets_read\"\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getOptionalUserId\n    val appId = query.clientContext.appId\n\n    val rateLimitTestIds = query.params(RateLimitTestIdsParam)\n\n    val usagesStitch = Seq(\n      t.FeatureRequest(RegularFeature, userId, applicationId = appId),\n      t.FeatureRequest(NewFeature, userId, applicationId = appId),\n      t.FeatureRequest(SoftFeature, userId, applicationId = appId),\n      t.FeatureRequest(SuspendedFeature, userId, applicationId = appId)\n    ).map { request => Stitch.callFuture(limiterClient.getLimitUsage(None, Some(request))) }\n\n    Stitch.collect(usagesStitch).map { usage =>\n      val limited =\n        if (rateLimitTestIds.contains(userId.get))\n          usage.map(u => u.remaining.toDouble / u.limit).exists(_ < 0.999)\n        else usage.map(_.remaining).exists(_ == 0)\n\n      FeatureMapBuilder().add(ViewerIsRateLimited, limited).build()\n    }\n  }\n\n  override val alerts = Seq(HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(70))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealGraphInNetworkScoresQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RealGraphInNetworkScoresOnPrem\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.wtf.candidate.{thriftscala => wtf}\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\ncase class RealGraphInNetworkScoresQueryFeatureHydrator @Inject() (\n  @Named(RealGraphInNetworkScoresOnPrem) realGraphMhStore: ReadableStore[Long, Seq[wtf.Candidate]])\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RealGraphInNetworkScores\")\n\n  override val features: Set[Feature[_, _]] = Set(RealGraphInNetworkScoresFeature)\n\n  private val RealGraphCandidateCount = 1000\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n\n    Stitch.callFuture(realGraphMhStore.get(query.getRequiredUserId)).map { realGraphFollowedUsers =>\n      val realGraphScoresFeatures = realGraphFollowedUsers\n        .getOrElse(Seq.empty)\n        .sortBy(-_.score)\n        .map(candidate => candidate.userId -> scaleScore(candidate.score))\n        .take(RealGraphCandidateCount)\n        .toMap\n\n      FeatureMapBuilder().add(RealGraphInNetworkScoresFeature, realGraphScoresFeatures).build()\n    }\n  }\n\n  // Rescale Real Graph v2 scores from [0,1] to the v1 scores distribution [1,2.97]\n  // v1 logic: src/scala/com/twitter/interaction_graph/scalding/jobs/scoring/InteractionGraphScoringJob.scala?L77-80\n  private def scaleScore(score: Double): Double =\n    if (score >= 0.0 && score <= 1.0) score * 1.97 + 1.0 else score\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealGraphQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableOnPremRealGraphQueryFeatures\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RealGraphFeatureRepository\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.repository.Repository\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.model.UserId\nimport com.twitter.timelines.real_graph.v1.thriftscala.RealGraphEdgeFeatures\nimport com.twitter.timelines.real_graph.{thriftscala => rg}\nimport com.twitter.user_session_store.{thriftscala => uss}\n\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject RealGraphFeatures extends Feature[PipelineQuery, Option[Map[UserId, RealGraphEdgeFeatures]]]\n\n@Singleton\nclass RealGraphQueryFeatureHydrator @Inject() (\n  @Named(RealGraphFeatureRepository) repository: Repository[Long, Option[uss.UserSession]])\n    extends QueryFeatureHydrator[PipelineQuery]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RealGraphFeatures\")\n\n  override val features: Set[Feature[_, _]] = Set(RealGraphFeatures)\n\n  override def onlyIf(\n    query: PipelineQuery\n  ): Boolean = {\n    !query.params(EnableOnPremRealGraphQueryFeatures)\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    Stitch.callFuture {\n      repository(query.getRequiredUserId).map { userSession =>\n        val realGraphFeaturesMap = userSession.flatMap { userSession =>\n          userSession.realGraphFeatures.collect {\n            case rg.RealGraphFeatures.V1(realGraphFeatures) =>\n              val edgeFeatures = realGraphFeatures.edgeFeatures ++ realGraphFeatures.oonEdgeFeatures\n              edgeFeatures.map { edge => edge.destId -> edge }.toMap\n          }\n        }\n\n        FeatureMapBuilder().add(RealGraphFeatures, realGraphFeaturesMap).build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealGraphViewerAuthorFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.RealGraphViewerAuthorFeatureHydrator.getCombinedRealGraphFeatures\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature\nimport com.twitter.home_mixer.util.MissingKeyException\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.real_graph.RealGraphEdgeFeaturesCombineAdapter\nimport com.twitter.timelines.prediction.adapters.real_graph.RealGraphFeaturesAdapter\nimport com.twitter.timelines.real_graph.v1.{thriftscala => v1}\nimport com.twitter.timelines.real_graph.{thriftscala => rg}\nimport com.twitter.util.Throw\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject RealGraphViewerAuthorDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject RealGraphViewerAuthorsDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass RealGraphViewerAuthorFeatureHydrator @Inject() ()\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RealGraphViewerAuthor\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(RealGraphViewerAuthorDataRecordFeature, RealGraphViewerAuthorsDataRecordFeature)\n\n  private val realGraphEdgeFeaturesAdapter = new RealGraphFeaturesAdapter\n  private val realGraphEdgeFeaturesCombineAdapter =\n    new RealGraphEdgeFeaturesCombineAdapter(prefix = \"authors.realgraph\")\n\n  private val MissingKeyFeatureMap = FeatureMapBuilder()\n    .add(RealGraphViewerAuthorDataRecordFeature, Throw(MissingKeyException))\n    .add(RealGraphViewerAuthorsDataRecordFeature, Throw(MissingKeyException))\n    .build()\n\n  private val batchSize = 64\n\n  def getFeatureMap(\n    candidate: CandidateWithFeatures[TweetCandidate],\n    viewerId: Long,\n    realGraphFeatures: Map[Long, v1.RealGraphEdgeFeatures]\n  ): FeatureMap =\n    candidate.features.getOrElse(AuthorIdFeature, None) match {\n      case Some(authorId) =>\n        val realGraphAuthorFeatures =\n          getRealGraphViewerAuthorFeatures(viewerId, authorId, realGraphFeatures)\n        val realGraphAuthorDataRecord = realGraphEdgeFeaturesAdapter\n          .adaptToDataRecords(realGraphAuthorFeatures).asScala.headOption.getOrElse(new DataRecord)\n\n        val combinedRealGraphFeaturesDataRecord = for {\n          inReplyToAuthorId <- candidate.features.getOrElse(InReplyToUserIdFeature, None)\n        } yield {\n          val combinedRealGraphFeatures =\n            getCombinedRealGraphFeatures(Seq(authorId, inReplyToAuthorId), realGraphFeatures)\n          realGraphEdgeFeaturesCombineAdapter\n            .adaptToDataRecords(Some(combinedRealGraphFeatures)).asScala.headOption\n            .getOrElse(new DataRecord)\n        }\n\n        FeatureMap(\n          RealGraphViewerAuthorDataRecordFeature,\n          realGraphAuthorDataRecord,\n          RealGraphViewerAuthorsDataRecordFeature,\n          combinedRealGraphFeaturesDataRecord.getOrElse(new DataRecord)\n        )\n      case _ => MissingKeyFeatureMap\n    }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val viewerId = query.getRequiredUserId\n    val realGraphFeatures = query.features\n      .flatMap(_.getOrElse(RealGraphFeatures, None))\n      .getOrElse(Map.empty[Long, v1.RealGraphEdgeFeatures])\n\n    OffloadFuturePools.offloadBatchElementToElement(\n      candidates,\n      getFeatureMap(_, viewerId, realGraphFeatures),\n      batchSize)\n  }\n\n  private def getRealGraphViewerAuthorFeatures(\n    viewerId: Long,\n    authorId: Long,\n    realGraphEdgeFeaturesMap: Map[Long, v1.RealGraphEdgeFeatures]\n  ): rg.UserRealGraphFeatures = {\n    realGraphEdgeFeaturesMap.get(authorId) match {\n      case Some(realGraphEdgeFeatures) =>\n        rg.UserRealGraphFeatures(\n          srcId = viewerId,\n          features = rg.RealGraphFeatures.V1(\n            v1.RealGraphFeatures(edgeFeatures = Seq(realGraphEdgeFeatures))))\n      case _ =>\n        rg.UserRealGraphFeatures(\n          srcId = viewerId,\n          features = rg.RealGraphFeatures.V1(v1.RealGraphFeatures(edgeFeatures = Seq.empty)))\n    }\n  }\n}\n\nobject RealGraphViewerAuthorFeatureHydrator {\n  def getCombinedRealGraphFeatures(\n    userIds: Seq[Long],\n    realGraphEdgeFeaturesMap: Map[Long, v1.RealGraphEdgeFeatures]\n  ): rg.RealGraphFeatures = {\n    val edgeFeatures = userIds.flatMap(realGraphEdgeFeaturesMap.get)\n    rg.RealGraphFeatures.V1(v1.RealGraphFeatures(edgeFeatures = edgeFeatures))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealGraphViewerRelatedUsersFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MentionUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableRealGraphViewerRelatedUsersFeaturesParam\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.real_graph.RealGraphEdgeFeaturesCombineAdapter\nimport com.twitter.timelines.real_graph.v1.{thriftscala => v1}\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject RealGraphViewerRelatedUsersDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass RealGraphViewerRelatedUsersFeatureHydrator @Inject() ()\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery]\n    with WithDefaultFeatureMap {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RealGraphViewerRelatedUsers\")\n\n  override val features: Set[Feature[_, _]] = Set(RealGraphViewerRelatedUsersDataRecordFeature)\n\n  override val defaultFeatureMap: FeatureMap = FeatureMap(\n    RealGraphViewerRelatedUsersDataRecordFeature,\n    RealGraphViewerRelatedUsersDataRecordFeature.defaultValue)\n\n  private val RealGraphEdgeFeaturesCombineAdapter = new RealGraphEdgeFeaturesCombineAdapter\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableRealGraphViewerRelatedUsersFeaturesParam)\n\n  val batchSize = 64\n\n  def getFeatureMap(\n    candidate: CandidateWithFeatures[TweetCandidate],\n    realGraphQueryFeatures: Map[Long, v1.RealGraphEdgeFeatures]\n  ): FeatureMap = {\n    val allRelatedUserIds = getRelatedUserIds(candidate.features)\n    val realGraphFeatures =\n      RealGraphViewerAuthorFeatureHydrator.getCombinedRealGraphFeatures(\n        allRelatedUserIds,\n        realGraphQueryFeatures\n      )\n    val realGraphFeaturesDataRecord = RealGraphEdgeFeaturesCombineAdapter\n      .adaptToDataRecords(Some(realGraphFeatures)).asScala.headOption\n      .getOrElse(new DataRecord)\n\n    FeatureMap(RealGraphViewerRelatedUsersDataRecordFeature, realGraphFeaturesDataRecord)\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val realGraphQueryFeatures = query.features\n      .flatMap(_.getOrElse(RealGraphFeatures, None))\n      .getOrElse(Map.empty[Long, v1.RealGraphEdgeFeatures])\n\n    OffloadFuturePools.offloadBatchElementToElement(\n      candidates,\n      getFeatureMap(_, realGraphQueryFeatures),\n      batchSize)\n  }\n\n  private def getRelatedUserIds(features: FeatureMap): Seq[Long] = {\n    (CandidatesUtil.getEngagerUserIds(features) ++\n      features.getOrElse(AuthorIdFeature, None) ++\n      features.getOrElse(MentionUserIdFeature, Seq.empty) ++\n      features.getOrElse(SourceUserIdFeature, None) ++\n      features.getOrElse(DirectedAtUserIdFeature, None)).distinct\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealTimeEntityRealGraphQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.EntityRealGraphClientStore\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.recos.entities.{thriftscala => ent}\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.wtf.entity_real_graph.{thriftscala => erg}\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject RealTimeEntityRealGraphFeatures\n    extends Feature[PipelineQuery, Option[Map[ent.Entity, Map[erg.EngagementType, erg.Features]]]]\n\n@Singleton\nclass RealTimeEntityRealGraphQueryFeatureHydrator @Inject() (\n  @Named(EntityRealGraphClientStore) client: ReadableStore[\n    erg.EntityRealGraphRequest,\n    erg.EntityRealGraphResponse\n  ],\n  override val statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RealTimeEntityRealGraphFeatures\")\n\n  override val features: Set[Feature[_, _]] = Set(RealTimeEntityRealGraphFeatures)\n\n  override val statScope: String = identifier.toString\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val entityRealGraphRequest = erg.EntityRealGraphRequest(\n      userId = query.getRequiredUserId,\n      entityTypes = Set(erg.EntityType.SemanticCore),\n      normalizeCounts = Some(true))\n    Stitch.callFuture {\n      client.get(entityRealGraphRequest).map { response =>\n        val engagements = response.map(_.response.mapValues(_.toMap).toMap)\n        FeatureMapBuilder().add(RealTimeEntityRealGraphFeatures, engagements).build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealTimeInteractionGraphEdgeFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.realtime_interaction_graph.RealTimeInteractionGraphFeaturesAdapter\nimport com.twitter.timelines.prediction.features.realtime_interaction_graph.RealTimeInteractionGraphEdgeFeatures\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject RealTimeInteractionGraphEdgeFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass RealTimeInteractionGraphEdgeFeatureHydrator @Inject() ()\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RealTimeInteractionGraphEdge\")\n\n  override val features: Set[Feature[_, _]] = Set(RealTimeInteractionGraphEdgeFeature)\n\n  private val realTimeInteractionGraphFeaturesAdapter = new RealTimeInteractionGraphFeaturesAdapter\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offload {\n    val userVertex =\n      query.features.flatMap(_.getOrElse(RealTimeInteractionGraphUserVertexQueryFeature, None))\n    val realTimeInteractionGraphFeaturesMap =\n      userVertex.map(RealTimeInteractionGraphEdgeFeatures(_, Time.now))\n\n    candidates.map { candidate =>\n      val feature = candidate.features.getOrElse(AuthorIdFeature, None).flatMap { authorId =>\n        realTimeInteractionGraphFeaturesMap.flatMap(_.get(authorId))\n      }\n\n      val dataRecordFeature =\n        realTimeInteractionGraphFeaturesAdapter.adaptToDataRecords(feature).asScala.head\n\n      FeatureMap(RealTimeInteractionGraphEdgeFeature, dataRecordFeature)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RealTimeInteractionGraphUserVertexQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RealTimeInteractionGraphUserVertexCache\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.wtf.real_time_interaction_graph.{thriftscala => ig}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject RealTimeInteractionGraphUserVertexQueryFeature\n    extends Feature[PipelineQuery, Option[ig.UserVertex]]\n\n@Singleton\nclass RealTimeInteractionGraphUserVertexQueryFeatureHydrator @Inject() (\n  @Named(RealTimeInteractionGraphUserVertexCache) client: ReadCache[Long, ig.UserVertex],\n  override val statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RealTimeInteractionGraphUserVertex\")\n\n  override val features: Set[Feature[_, _]] = Set(RealTimeInteractionGraphUserVertexQueryFeature)\n\n  override val statScope: String = identifier.toString\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n\n    Stitch.callFuture(\n      client.get(Seq(userId)).map { results =>\n        val feature = observedGet(key = Some(userId), keyValueResult = results)\n        FeatureMapBuilder()\n          .add(RealTimeInteractionGraphUserVertexQueryFeature, feature)\n          .build()\n      }\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RequestQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.tracing.Annotation.BinaryAnnotation\nimport com.twitter.finagle.tracing.ForwardAnnotation\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.model.request.DeviceContext.RequestContext\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.joinkey.context.RequestJoinKeyContext\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.BadRequest\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.search.common.util.lang.ThriftLanguageUtil\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.request_context.RequestContextAdapter.dowFromTimestamp\nimport com.twitter.timelines.prediction.adapters.request_context.RequestContextAdapter.hourFromTimestamp\nimport java.util.UUID\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RequestQueryFeatureHydrator[\n  Query <: PipelineQuery with HasPipelineCursor[UrtOrderedCursor] with HasDeviceContext] @Inject() (\n) extends QueryFeatureHydrator[Query] {\n\n  override val features: Set[Feature[_, _]] = Set(\n    AccountAgeFeature,\n    ClientIdFeature,\n    DeviceCountryFeature,\n    DeviceLanguageFeature,\n    GetInitialFeature,\n    GetMiddleFeature,\n    GetNewerFeature,\n    GetOlderFeature,\n    GuestIdFeature,\n    HasDarkRequestFeature,\n    IsForegroundRequestFeature,\n    IsLaunchRequestFeature,\n    PollingFeature,\n    PullToRefreshFeature,\n    RequestJoinIdFeature,\n    ServedIdFeature,\n    TimestampFeature,\n    TimestampGMTDowFeature,\n    TimestampGMTHourFeature,\n    ViewerIdFeature\n  )\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"Request\")\n\n  private val DarkRequestAnnotation = \"clnt/has_dark_request\"\n\n  // Convert Language code to ISO 639-3 format\n  private def getLanguageISOFormatByCode(languageCode: String): String =\n    ThriftLanguageUtil.getLanguageCodeOf(ThriftLanguageUtil.getThriftLanguageOf(languageCode))\n\n  private def getRequestJoinId: Option[Long] =\n    RequestJoinKeyContext.current.flatMap(_.requestJoinId)\n\n  private def hasDarkRequest: Option[Boolean] = ForwardAnnotation.current\n    .getOrElse(Seq[BinaryAnnotation]())\n    .find(_.key == DarkRequestAnnotation)\n    .map(_.value.asInstanceOf[Boolean])\n\n  override def hydrate(query: Query): Stitch[FeatureMap] = {\n    val requestContext = query.deviceContext.flatMap(_.requestContextValue)\n    val servedId = UUID.randomUUID.getMostSignificantBits\n    val timestamp = query.queryTime.inMilliseconds\n\n    val featureMap = FeatureMapBuilder()\n      .add(AccountAgeFeature, query.getOptionalUserId.flatMap(SnowflakeId.timeFromIdOpt))\n      .add(ClientIdFeature, query.clientContext.appId)\n      .add(DeviceCountryFeature, query.getCountryCode)\n      .add(DeviceLanguageFeature, query.getLanguageCode.map(getLanguageISOFormatByCode))\n      .add(\n        GetInitialFeature,\n        query.pipelineCursor.forall(cursor => cursor.id.isEmpty && cursor.gapBoundaryId.isEmpty))\n      .add(\n        GetMiddleFeature,\n        query.pipelineCursor.exists(cursor =>\n          cursor.id.isDefined && cursor.gapBoundaryId.isDefined &&\n            cursor.cursorType.contains(GapCursor)))\n      .add(\n        GetNewerFeature,\n        query.pipelineCursor.exists(cursor =>\n          cursor.id.isDefined && cursor.gapBoundaryId.isEmpty &&\n            cursor.cursorType.contains(TopCursor)))\n      .add(\n        GetOlderFeature,\n        query.pipelineCursor.exists(cursor =>\n          cursor.id.isDefined && cursor.gapBoundaryId.isEmpty &&\n            cursor.cursorType.contains(BottomCursor)))\n      .add(GuestIdFeature, query.clientContext.guestId)\n      .add(IsForegroundRequestFeature, requestContext.contains(RequestContext.Foreground))\n      .add(IsLaunchRequestFeature, requestContext.contains(RequestContext.Launch))\n      .add(PollingFeature, query.deviceContext.exists(_.isPolling.contains(true)))\n      .add(PullToRefreshFeature, requestContext.contains(RequestContext.PullToRefresh))\n      .add(ServedIdFeature, Some(servedId))\n      .add(RequestJoinIdFeature, getRequestJoinId)\n      .add(TimestampFeature, timestamp)\n      .add(TimestampGMTDowFeature, dowFromTimestamp(timestamp))\n      .add(TimestampGMTHourFeature, hourFromTimestamp(timestamp))\n      .add(HasDarkRequestFeature, hasDarkRequest)\n      .add(\n        ViewerIdFeature,\n        query.getOptionalUserId\n          .orElse(query.getGuestId).getOrElse(\n            throw PipelineFailure(BadRequest, \"Missing viewer id\")))\n      .build()\n\n    Stitch.value(featureMap)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RequestTimeQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.HomeFeatures.FollowingLastNonPollingTimeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.LastNonPollingTimeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.NonPollingTimesFeature\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.util.FDsl._\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.features.time_features.AccountAgeInterval\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.ACCOUNT_AGE_INTERVAL\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.IS_12_MONTH_NEW_USER\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.IS_30_DAY_NEW_USER\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.TIME_BETWEEN_NON_POLLING_REQUESTS_AVG\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.TIME_SINCE_LAST_NON_POLLING_REQUEST\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.TIME_SINCE_VIEWER_ACCOUNT_CREATION_SECS\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.USER_ID_IS_SNOWFLAKE_ID\nimport com.twitter.user_session_store.ReadRequest\nimport com.twitter.user_session_store.ReadWriteUserSessionStore\nimport com.twitter.user_session_store.UserSessionDataset\nimport com.twitter.user_session_store.UserSessionDataset.UserSessionDataset\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject RequestTimeDataRecordFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\ncase class RequestTimeQueryFeatureHydrator @Inject() (\n  userSessionStore: ReadWriteUserSessionStore)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"RequestTime\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    FollowingLastNonPollingTimeFeature,\n    LastNonPollingTimeFeature,\n    NonPollingTimesFeature,\n    RequestTimeDataRecordFeature\n  )\n\n  private val datasets: Set[UserSessionDataset] = Set(UserSessionDataset.NonPollingTimes)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    userSessionStore\n      .read(ReadRequest(query.getRequiredUserId, datasets))\n      .map { userSession =>\n        val nonPollingTimestamps = userSession.flatMap(_.nonPollingTimestamps)\n\n        val lastNonPollingTime = nonPollingTimestamps\n          .flatMap(_.nonPollingTimestampsMs.headOption)\n          .map(Time.fromMilliseconds)\n\n        val followingLastNonPollingTime = nonPollingTimestamps\n          .flatMap(_.mostRecentHomeLatestNonPollingTimestampMs)\n          .map(Time.fromMilliseconds)\n\n        val nonPollingTimes = nonPollingTimestamps\n          .map(_.nonPollingTimestampsMs)\n          .getOrElse(Seq.empty)\n\n        val requestTimeDataRecord = getRequestTimeDataRecord(query, nonPollingTimes)\n\n        FeatureMapBuilder()\n          .add(FollowingLastNonPollingTimeFeature, followingLastNonPollingTime)\n          .add(LastNonPollingTimeFeature, lastNonPollingTime)\n          .add(NonPollingTimesFeature, nonPollingTimes)\n          .add(RequestTimeDataRecordFeature, requestTimeDataRecord)\n          .build()\n      }\n  }\n\n  def getRequestTimeDataRecord(query: PipelineQuery, nonPollingTimes: Seq[Long]): DataRecord = {\n    val requestTimeMs = query.queryTime.inMillis\n    val accountAge = SnowflakeId.timeFromIdOpt(query.getRequiredUserId)\n    val timeSinceAccountCreation = accountAge.map(query.queryTime.since)\n    val timeSinceEarliestNonPollingRequest =\n      nonPollingTimes.lastOption.map(requestTimeMs - _)\n    val timeSinceLastNonPollingRequest =\n      nonPollingTimes.headOption.map(requestTimeMs - _)\n\n    new DataRecord()\n      .setFeatureValue(USER_ID_IS_SNOWFLAKE_ID, accountAge.isDefined)\n      .setFeatureValue(\n        IS_30_DAY_NEW_USER,\n        timeSinceAccountCreation.map(_ < 30.days).getOrElse(false)\n      )\n      .setFeatureValue(\n        IS_12_MONTH_NEW_USER,\n        timeSinceAccountCreation.map(_ < 365.days).getOrElse(false)\n      )\n      .setFeatureValueFromOption(\n        ACCOUNT_AGE_INTERVAL,\n        timeSinceAccountCreation.flatMap(AccountAgeInterval.fromDuration).map(_.id.toLong)\n      )\n      .setFeatureValueFromOption(\n        TIME_SINCE_VIEWER_ACCOUNT_CREATION_SECS,\n        timeSinceAccountCreation.map(_.inSeconds.toDouble)\n      )\n      .setFeatureValueFromOption(\n        TIME_BETWEEN_NON_POLLING_REQUESTS_AVG,\n        timeSinceEarliestNonPollingRequest.map(_.toDouble / math.max(1.0, nonPollingTimes.size))\n      )\n      .setFeatureValueFromOption(\n        TIME_SINCE_LAST_NON_POLLING_REQUEST,\n        timeSinceLastNonPollingRequest.map(_.toDouble)\n      )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/SGSValidSocialContextFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.socialgraph.{thriftscala => sg}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.SocialGraph\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * This hydrator takes liked-by and followed-by user ids and checks via SGS that the viewer is\n * following the engager, that the viewer is not blocking the engager, that the engager is not\n * blocking the viewer, and that the viewer has not muted the engager.\n */\n@Singleton\nclass SGSValidSocialContextFeatureHydrator @Inject() (\n  socialGraph: SocialGraph)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with WithDefaultFeatureMap {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"SGSValidSocialContext\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    SGSValidFollowedByUserIdsFeature,\n    SGSValidLikedByUserIdsFeature\n  )\n\n  override val defaultFeatureMap: FeatureMap = FeatureMap(\n    SGSValidFollowedByUserIdsFeature,\n    Seq.empty,\n    SGSValidLikedByUserIdsFeature,\n    Seq.empty\n  )\n\n  private val MaxCountUsers = 10\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    val allSocialContextUserIds =\n      candidates.flatMap { candidate =>\n        candidate.features.getOrElse(FavoritedByUserIdsFeature, Nil).take(MaxCountUsers) ++\n          candidate.features.getOrElse(FollowedByUserIdsFeature, Nil).take(MaxCountUsers)\n      }.distinct\n\n    getValidUserIds(query.getRequiredUserId, allSocialContextUserIds).map { validUserIds =>\n      candidates.map { candidate =>\n        val sgsFilteredLikedByUserIds =\n          candidate.features\n            .getOrElse(FavoritedByUserIdsFeature, Nil).take(MaxCountUsers)\n            .filter(validUserIds.contains)\n\n        val sgsFilteredFollowedByUserIds =\n          candidate.features\n            .getOrElse(FollowedByUserIdsFeature, Nil).take(MaxCountUsers)\n            .filter(validUserIds.contains)\n\n        FeatureMapBuilder()\n          .add(SGSValidFollowedByUserIdsFeature, sgsFilteredFollowedByUserIds)\n          .add(SGSValidLikedByUserIdsFeature, sgsFilteredLikedByUserIds)\n          .build()\n      }\n    }\n  }\n\n  private def getValidUserIds(\n    viewerId: Long,\n    socialProofUserIds: Seq[Long]\n  ): Stitch[Seq[Long]] = {\n    if (socialProofUserIds.nonEmpty) {\n      val request = sg.IdsRequest(\n        relationships = Seq(\n          sg.SrcRelationship(\n            viewerId,\n            sg.RelationshipType.Following,\n            targets = Some(socialProofUserIds),\n            hasRelationship = true),\n          sg.SrcRelationship(\n            viewerId,\n            sg.RelationshipType.Blocking,\n            targets = Some(socialProofUserIds),\n            hasRelationship = false),\n          sg.SrcRelationship(\n            viewerId,\n            sg.RelationshipType.BlockedBy,\n            targets = Some(socialProofUserIds),\n            hasRelationship = false),\n          sg.SrcRelationship(\n            viewerId,\n            sg.RelationshipType.Muting,\n            targets = Some(socialProofUserIds),\n            hasRelationship = false)\n        ),\n        pageRequest = Some(sg.PageRequest(selectAll = Some(true)))\n      )\n      socialGraph.ids(request).map(_.ids)\n    } else Stitch.Nil\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/SimClustersEngagementSimilarityFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableSimClustersSimilarityFeaturesDeciderParam\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.clients.strato.twistly.SimClustersRecentEngagementSimilarityClient\nimport com.twitter.timelines.configapi.decider.BooleanDeciderParam\nimport com.twitter.timelines.prediction.adapters.twistly.SimClustersRecentEngagementSimilarityFeaturesAdapter\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject SimClustersEngagementSimilarityFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass SimClustersEngagementSimilarityFeatureHydrator @Inject() (\n  simClustersEngagementSimilarityClient: SimClustersRecentEngagementSimilarityClient)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"SimClustersEngagementSimilarity\")\n\n  override val features: Set[Feature[_, _]] = Set(SimClustersEngagementSimilarityFeature)\n\n  private val simClustersRecentEngagementSimilarityFeaturesAdapter =\n    new SimClustersRecentEngagementSimilarityFeaturesAdapter\n\n  override def onlyIf(query: PipelineQuery): Boolean = {\n    val param: BooleanDeciderParam = EnableSimClustersSimilarityFeaturesDeciderParam\n    query.params.apply(param)\n  }\n\n  def getFeatureMaps(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n    userId: Long\n  ): Future[Seq[FeatureMap]] = {\n    val tweetToCandidates =\n      candidates.map(candidate => candidate.candidate.id -> candidate).toMap\n    val tweetIds = tweetToCandidates.keySet.toSeq\n    val userTweetEdges = tweetIds.map(tweetId => (userId, tweetId))\n\n    simClustersEngagementSimilarityClient\n      .getSimClustersRecentEngagementSimilarityScores(userTweetEdges).map {\n        simClustersRecentEngagementSimilarityScoresMap =>\n          candidates.map { candidate =>\n            val similarityFeatureOpt = simClustersRecentEngagementSimilarityScoresMap\n              .get(userId -> candidate.candidate.id).flatten\n            val dataRecordOpt = similarityFeatureOpt.map { similarityFeature =>\n              simClustersRecentEngagementSimilarityFeaturesAdapter\n                .adaptToDataRecords(similarityFeature)\n                .get(0)\n            }\n            FeatureMap(\n              SimClustersEngagementSimilarityFeature,\n              dataRecordOpt.getOrElse(new DataRecord))\n          }\n      }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] =\n    Stitch.callFuture {\n      getFeatureMaps(candidates, query.getRequiredUserId)\n    }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/SimClustersLogFavBasedTweetFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.DiversityRescoringFeatureHydrator.EmptyDataRecord\nimport com.twitter.home_mixer_features.thriftjava.HomeMixerFeaturesRequest\nimport com.twitter.home_mixer_features.{thriftjava => t}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject SimClustersLogFavBasedTweetFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass SimClustersLogFavBasedTweetFeatureHydrator @Inject() (\n  homeMixerFeatureService: t.HomeMixerFeatures.ServiceToClient,\n  statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"SimClustersLogFavBasedTweet\")\n\n  override val features: Set[Feature[_, _]] = Set(SimClustersLogFavBasedTweetFeature)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyTotalCounter = scopedStatsReceiver.counter(\"key/total\")\n\n  private val batchSize = 50\n\n  private def getEmbeddingsFromHMF(\n    tweetIds: Seq[Long]\n  ): Future[Seq[DataRecord]] = {\n    val keysSerialized = tweetIds.map(_.toString)\n    val request = new HomeMixerFeaturesRequest()\n    request.setKeys(keysSerialized.asJava)\n    request.setCache(t.Cache.LOG_FAV_BASED_TWEET_20M145K2020_EMBEDDINGS)\n    homeMixerFeatureService\n      .getHomeMixerFeatures(request)\n      .map { resp => unmarshallHomeMixerFeaturesResponse(resp) }\n  }\n\n  private def unmarshallHomeMixerFeaturesResponse(\n    response: t.HomeMixerFeaturesResponse\n  ): Seq[DataRecord] = {\n    response.getHomeMixerFeatures.asScala.map { homeMixerFeatureOpt =>\n      if (homeMixerFeatureOpt.isSetHomeMixerFeaturesType) {\n        val homeMixerFeature = homeMixerFeatureOpt.getHomeMixerFeaturesType\n        if (homeMixerFeature.isSet(t.HomeMixerFeaturesType._Fields.DATA_RECORD)) {\n          homeMixerFeature.getDataRecord\n        } else {\n          throw new Exception(\"Unexpected type\")\n        }\n      } else EmptyDataRecord\n    }\n  }\n\n  private def getBatchedFeatureMap(\n    candidatesBatch: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Future[Seq[FeatureMap]] = {\n    val tweetIds =\n      candidatesBatch.map { candidate =>\n        keyTotalCounter.incr()\n        candidate.candidate.id\n      }\n\n    getEmbeddingsFromHMF(tweetIds).map { response =>\n      response.map { dataRecordOpt =>\n        keyFoundCounter.incr()\n        FeatureMap(SimClustersLogFavBasedTweetFeature, dataRecordOpt)\n      }\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    OffloadFuturePools.offloadBatchSeqToFutureSeq(candidates, getBatchedFeatureMap, batchSize)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/SimClustersUserSparseEmbeddingsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.dal.personal_data.{thriftjava => pd}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordOptionalFeature\nimport com.twitter.product_mixer.core.feature.datarecord.SparseContinuousDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.recommendations.simclusters_v2.InterestedIn20M145K2020OnUserClientColumn\nimport com.twitter.timelines.prediction.features.simcluster.SimclusterFeatures\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject SimClustersUserLogFavSparseEmbeddingsDataRecordFeature\n    extends DataRecordOptionalFeature[PipelineQuery, Map[String, Double]]\n    with SparseContinuousDataRecordCompatible {\n  override val featureName: String =\n    SimclusterFeatures.SIMCLUSTER_USER_LOG_FAV_CLUSTER_SCORES.getFeatureName\n  override val personalDataTypes: Set[pd.PersonalDataType] =\n    Set(pd.PersonalDataType.InferredInterests)\n}\n\nobject SimClustersUserFollowSparseEmbeddingsDataRecordFeature\n    extends DataRecordOptionalFeature[PipelineQuery, Map[String, Double]]\n    with SparseContinuousDataRecordCompatible {\n  override val featureName: String =\n    SimclusterFeatures.SIMCLUSTER_USER_FOLLOW_CLUSTER_SCORES.getFeatureName\n  override val personalDataTypes: Set[pd.PersonalDataType] =\n    Set(pd.PersonalDataType.InferredInterests)\n}\n\n@Singleton\nclass SimClustersUserSparseEmbeddingsQueryFeatureHydrator @Inject() (\n  interestedIn20M145K2020OnUserClientColumn: InterestedIn20M145K2020OnUserClientColumn)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"SimClustersUserSparseEmbeddingsQuery\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(\n      SimClustersUserLogFavSparseEmbeddingsDataRecordFeature,\n      SimClustersUserFollowSparseEmbeddingsDataRecordFeature)\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(SimClustersUserLogFavSparseEmbeddingsDataRecordFeature, None)\n    .add(SimClustersUserFollowSparseEmbeddingsDataRecordFeature, None)\n    .build()\n\n  override def hydrate(\n    query: PipelineQuery,\n  ): Stitch[FeatureMap] = {\n    val interestedInEmbeddingsOptStitch = interestedIn20M145K2020OnUserClientColumn.fetcher\n      .fetch(query.getRequiredUserId)\n      .map { result =>\n        result.v.map { interestedInEmbeddings =>\n          interestedInEmbeddings.clusterIdToScores\n        }\n      }\n    interestedInEmbeddingsOptStitch\n      .map { interestedInEmbeddingsOpt =>\n        val logFavEmbeddings = interestedInEmbeddingsOpt.map { interestedInEmbeddings =>\n          interestedInEmbeddings.map {\n            case (key, value) => (key.toString, value.logFavScore.getOrElse(0.0))\n          }.toMap\n        }\n        val followEmbeddings = interestedInEmbeddingsOpt.map { interestedInEmbeddings =>\n          interestedInEmbeddings.map {\n            case (key, value) => (key.toString, value.followScore.getOrElse(0.0))\n          }.toMap\n        }\n        FeatureMapBuilder()\n          .add(SimClustersUserLogFavSparseEmbeddingsDataRecordFeature, logFavEmbeddings)\n          .add(SimClustersUserFollowSparseEmbeddingsDataRecordFeature, followEmbeddings)\n          .build()\n      }.handle { case _ => DefaultFeatureMap }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/SimClustersUserTweetScoresHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.dal.personal_data.{thriftjava => pd}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordOptionalFeature\nimport com.twitter.product_mixer.core.feature.datarecord.DoubleDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.Fetch\nimport com.twitter.strato.generated.client.ml.featureStore.SimClustersUserInterestedInTweetEmbeddingDotProduct20M145K2020OnUserTweetEdgeClientColumn\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject SimClustersUserInterestedInTweetEmbeddingDataRecordFeature\n    extends DataRecordOptionalFeature[TweetCandidate, Double]\n    with DoubleDataRecordCompatible {\n  override val featureName: String =\n    \"user-tweet.recommendations.sim_clusters_scores.user_interested_in_tweet_embedding_dot_product_20m_145k_2020\"\n  override val personalDataTypes: Set[pd.PersonalDataType] =\n    Set(pd.PersonalDataType.InferredInterests)\n}\n\n@Singleton\nclass SimClustersUserTweetScoresHydrator @Inject() (\n  simClustersColumn: SimClustersUserInterestedInTweetEmbeddingDotProduct20M145K2020OnUserTweetEdgeClientColumn,\n  statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"SimClustersUserTweetScores\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(SimClustersUserInterestedInTweetEmbeddingDataRecordFeature)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  private val keyFailureCounter = scopedStatsReceiver.counter(\"key/failure\")\n  private val keySkipCounter = scopedStatsReceiver.counter(\"key/skip\")\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(SimClustersUserInterestedInTweetEmbeddingDataRecordFeature, None)\n    .build()\n  private val MinFavToHydrate = 9\n  private val batchSize = 64\n\n  def getFeatureMaps(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n    userId: Long\n  ): Future[Seq[FeatureMap]] = {\n    val featureMapStitch = Stitch.traverse(candidates) { candidate =>\n      val ebFeatures = candidate.features.getOrElse(EarlybirdFeature, None)\n      val favCount = ebFeatures.flatMap(_.favCountV2).getOrElse(0)\n\n      if (ebFeatures.isEmpty || favCount >= MinFavToHydrate) {\n        simClustersColumn.fetcher\n          .fetch((userId, candidate.candidate.id), Unit)\n          .map {\n            case Fetch.Result(response, _) =>\n              if (response.nonEmpty) keyFoundCounter.incr()\n              else keyNotFoundCounter.incr()\n              FeatureMapBuilder()\n                .add(SimClustersUserInterestedInTweetEmbeddingDataRecordFeature, response)\n                .build()\n            case _ =>\n              keyFailureCounter.incr()\n              DefaultFeatureMap\n          }\n      } else {\n        keySkipCounter.incr()\n        Stitch.value(DefaultFeatureMap)\n      }\n    }\n    Stitch.run(featureMapStitch)\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    OffloadFuturePools.offloadBatchSeqToFutureSeq(\n      candidates,\n      getFeatureMaps(_, query.getRequiredUserId),\n      batchSize,\n      offload = true)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/SimclusterBasedTopAuthorsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ExpiringLruInProcessCache\nimport com.twitter.servo.cache.InProcessCache\nimport com.twitter.simclusters_v2.thriftscala.ClusterDetails\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.util.Random\n\nobject SimclustersFavBasedTopAuthors extends Feature[PipelineQuery, Seq[(Long, Double)]]\nobject SimclustersFollowBasedTopAuthors extends Feature[PipelineQuery, Seq[(Long, Double)]]\n\nobject SimclusterBasedTopAuthorsQueryFeatureHydrator {\n  private val BaseTTL = 60 * 24\n  private val TTL = (BaseTTL + Random.nextInt(60)).minutes\n\n  val cache: InProcessCache[String, Seq[(Long, Double)]] =\n    new ExpiringLruInProcessCache(ttl = TTL, maximumSize = 150 * 1000)\n}\n\n@Singleton\nclass SimclusterBasedTopAuthorsQueryFeatureHydrator @Inject() (\n  store: ReadableStore[String, ClusterDetails])\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  import SimclusterBasedTopAuthorsQueryFeatureHydrator._\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"SimclusterBasedTopAuthors\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(SimclustersFavBasedTopAuthors, SimclustersFollowBasedTopAuthors)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val favBasedEmbeddings =\n      query.features\n        .flatMap(_.getOrElse(SimClustersUserLogFavSparseEmbeddingsDataRecordFeature, None))\n        .getOrElse(Map.empty[String, Double])\n    val followBasedEmbeddings =\n      query.features\n        .flatMap(_.getOrElse(SimClustersUserFollowSparseEmbeddingsDataRecordFeature, None))\n        .getOrElse(Map.empty[String, Double])\n\n    val favBasedTopAuthorsStitch = Stitch.callFuture(getTopAuthorsWithScores(favBasedEmbeddings))\n    val followBasedTopAuthorsStitch =\n      Stitch.callFuture(getTopAuthorsWithScores(followBasedEmbeddings))\n    Stitch.join(favBasedTopAuthorsStitch, followBasedTopAuthorsStitch).map {\n      case (favBasedTopAuthors, followBasedTopAuthors) =>\n        FeatureMap(\n          SimclustersFavBasedTopAuthors,\n          favBasedTopAuthors,\n          SimclustersFollowBasedTopAuthors,\n          followBasedTopAuthors\n        )\n    }\n  }\n\n  private def getTopAuthorsWithScores(\n    embeddings: Map[String, Double]\n  ): Future[Seq[(Long, Double)]] = {\n    val flattenedAuthorsWithScoresFut = Future\n      .collect {\n        embeddings.map {\n          case (clusterId, seedScore) =>\n            getTopAuthorsWithScoresForCluster(clusterId).map { topAuthors =>\n              topAuthors.map {\n                case (author, score) => (author, score * seedScore)\n              }\n            }\n        }.toSeq\n      }.map(_.flatten)\n    flattenedAuthorsWithScoresFut.map { flattenedAuthorsWithScores =>\n      val authorsWithScores =\n        flattenedAuthorsWithScores.groupBy(_._1).mapValues(_.map(_._2).sum).toSeq\n      authorsWithScores.sortBy(-_._2)\n    }\n  }\n\n  private def getTopAuthorsWithScoresForCluster(\n    clusterId: String\n  ): Future[Seq[(Long, Double)]] = {\n    cache\n      .get(clusterId)\n      .map(Future.value(_))\n      .getOrElse {\n        store\n          .get(clusterId).map { clusterDetailsOpt =>\n            val authorsWithScores = clusterDetailsOpt\n              .flatMap { clusterDetails =>\n                clusterDetails.knownForUsersAndScores.map { knownForUsersAndScores =>\n                  knownForUsersAndScores.map { userAndScore =>\n                    (userAndScore.userId, userAndScore.score)\n                  }\n                }\n              }.getOrElse(Seq.empty)\n            cache.set(clusterId, authorsWithScores)\n            authorsWithScores\n          }.handle { case _ => Seq.empty }\n      }\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/SlopAuthorFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.abuse.detection.scoring.{thriftscala => t}\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DebugStringFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SlopAuthorFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SlopAuthorScoreFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.SlopMaxScore\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.hss.user_scores.api.HealthModelScoresOnUserClientColumn\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SlopAuthorFeatureHydrator @Inject() (\n  healthModelScoresOnUserClientColumn: HealthModelScoresOnUserClientColumn)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"SlopAuthor\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(SlopAuthorFeature, DebugStringFeature, SlopAuthorScoreFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    val authorIds = candidates.flatMap(_.features.getOrElse(AuthorIdFeature, None)).distinct\n    val slopThreshold = query.params(SlopMaxScore)\n    Stitch\n      .collectToTry {\n        authorIds.map { authorId =>\n          healthModelScoresOnUserClientColumn.fetcher\n            .fetch(authorId, Seq(t.UserHealthModel.NsfwConsumerFollowerScore))\n            .map { response =>\n              authorId -> response.v.flatMap { scoresMap =>\n                scoresMap.get(t.UserHealthModel.NsfwConsumerFollowerScore)\n              }\n            }\n        }\n      }.map { authorNsfwScores =>\n        val authorIdsToNsfwScoresMap = authorNsfwScores.flatMap(_.toOption).toMap\n        candidates.map { candidate =>\n          val debugStringFeature =\n            candidate.features.getOrElse(DebugStringFeature, None).getOrElse(\"\")\n          candidate.features.getOrElse(AuthorIdFeature, None) match {\n            case Some(authorId) =>\n              val slopAuthorScore =\n                authorIdsToNsfwScoresMap.getOrElse(authorId, None).getOrElse(0.0)\n              FeatureMap(\n                SlopAuthorFeature,\n                slopAuthorScore > slopThreshold,\n                DebugStringFeature,\n                Some(\"%s Slop %.3f\".format(debugStringFeature, slopAuthorScore)),\n                SlopAuthorScoreFeature,\n                Some(slopAuthorScore)\n              )\n            case _ =>\n              FeatureMap(\n                SlopAuthorFeature,\n                false,\n                DebugStringFeature,\n                Some(debugStringFeature),\n                SlopAuthorScoreFeature,\n                None\n              )\n          }\n        }\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/SpaceStateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.TweetUrlsFeature\nimport com.twitter.periscope.api.{thriftscala => ps}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.Fetch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.periscope.CoreOnAudioSpaceClientColumn\nimport com.twitter.ubs.{thriftscala => ubs}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject SpaceStateFeature extends Feature[TweetCandidate, Option[ubs.BroadcastState]]\n\n@Singleton\nclass SpaceStateFeatureHydrator @Inject() (\n  coreOnAudioSpaceClientColumn: CoreOnAudioSpaceClientColumn)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with WithDefaultFeatureMap {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"SpaceState\")\n\n  override val features: Set[Feature[_, _]] = Set(SpaceStateFeature)\n\n  private val pattern = \"\"\"https?://(?:x|twitter).com/i/spaces/(\\w+).*\"\"\".r\n\n  private val fetcher: Fetcher[String, ps.AudioSpacesLookupContext, ubs.AudioSpace] =\n    coreOnAudioSpaceClientColumn.fetcher\n\n  private val lookupContext = ps.AudioSpacesLookupContext(participantHydrationLevel =\n    Some(ps.ParticipantHydrationLevel.NoParticipantInfo))\n\n  override val defaultFeatureMap: FeatureMap = FeatureMap(SpaceStateFeature, None)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] =\n    OffloadFuturePools.offloadStitch {\n      val spaceIdMap = candidates.flatMap { candidate =>\n        candidate.features\n          .getOrElse(TweetUrlsFeature, Seq.empty)\n          .collectFirst { case pattern(spaceId) => spaceId }\n          .map(spaceId => candidate.candidate.id -> spaceId)\n      }.toMap\n\n      Stitch\n        .collect {\n          spaceIdMap.values.toSeq.distinct.map { spaceId =>\n            fetcher\n              .fetch(spaceId, lookupContext)\n              .map {\n                case Fetch.Result(Some(audioSpace), _) if audioSpace.broadcastId.nonEmpty =>\n                  Some(spaceId -> audioSpace)\n                case _ => None\n              }\n          }\n        }.map { results =>\n          val audioSpaceMap = results.flatten.toMap\n          candidates.map { candidate =>\n            val spaceState = spaceIdMap.get(candidate.candidate.id).flatMap { spaceId =>\n              audioSpaceMap.get(spaceId).flatMap(_.state)\n            }\n\n            FeatureMapBuilder().add(SpaceStateFeature, spaceState).build()\n          }\n        }\n    }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TSPInferredTopicFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.contentrecommender.{thriftscala => cr}\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.inferred_topic.InferredTopicAdapter\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TSPMetricTagFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BasicTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContextFunctionalityType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.clients.strato.topics.TopicSocialProofClient\nimport com.twitter.topiclisting.TopicListingViewerContext\nimport com.twitter.tsp.thriftscala.TopicFollowType\nimport com.twitter.tsp.{thriftscala => tsp}\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TSPInferredTopicFeature extends Feature[TweetCandidate, Map[Long, Double]]\nobject TSPInferredTopicDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TSPInferredTopicFeatureHydrator @Inject() (\n  topicSocialProofClient: TopicSocialProofClient)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TSPInferredTopic\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    TSPInferredTopicFeature,\n    TSPInferredTopicDataRecordFeature,\n    TopicIdSocialContextFeature,\n    TopicContextFunctionalityTypeFeature\n  )\n\n  private val topK = 3\n\n  private val SuggestTypesToSetSocialProof: Set[hmt.ServedType] = Set(\n    hmt.ServedType.ForYouTweetMixer,\n    hmt.ServedType.ForYouSimclusters,\n    hmt.ServedType.ForYouTwhin,\n    hmt.ServedType.ForYouUtg,\n    hmt.ServedType.ForYouUvg,\n    hmt.ServedType.ForYouUteg,\n    hmt.ServedType.ForYouPopularGeo,\n    hmt.ServedType.ForYouPopularTopic,\n    hmt.ServedType.ForYouDeepRetrieval,\n    hmt.ServedType.ForYouEvergreenDeepRetrieval,\n    hmt.ServedType.ForYouRelatedCreator,\n    hmt.ServedType.ForYouLocal,\n    hmt.ServedType.ForYouTrends,\n    hmt.ServedType.ForYouHistoryAuthor,\n  )\n\n  private val DefaultFeatureMap = FeatureMap(\n    TSPInferredTopicFeature,\n    Map.empty[Long, Double],\n    TSPInferredTopicDataRecordFeature,\n    new DataRecord(),\n    TopicIdSocialContextFeature,\n    None,\n    TopicContextFunctionalityTypeFeature,\n    None\n  )\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val tags = candidates.collect {\n      case candidate if candidate.features.getTry(TSPMetricTagFeature).isReturn =>\n        candidate.candidate.id -> candidate.features\n          .getOrElse(TSPMetricTagFeature, Set.empty[tsp.MetricTag])\n    }.toMap\n\n    val followableUttTopics = query.features\n      .flatMap(_.getOrElse(FollowableUttTopicsFeatures, None))\n      .map(_.mapValues(_.getOrElse(TopicFollowType.ImplicitFollow)))\n\n    val topicSocialProofRequest = tsp.TopicSocialProofRequest(\n      userId = query.getRequiredUserId,\n      tweetIds = candidates.map(_.candidate.id).toSet,\n      displayLocation = cr.DisplayLocation.HomeTimeline,\n      topicListingSetting = tsp.TopicListingSetting.Followable,\n      context = TopicListingViewerContext.fromClientContext(query.clientContext).toThrift,\n      bypassModes = None,\n      allowlist = followableUttTopics,\n      // Only TweetMixer source has this data. Convert the TweetMixer metric tag to tsp metric tag.\n      tags = if (tags.isEmpty) None else Some(tags)\n    )\n\n    topicSocialProofClient\n      .getTopicTweetSocialProofResponse(topicSocialProofRequest)\n      .map {\n        case Some(response) =>\n          handleResponse(response, candidates)\n        case _ => candidates.map { _ => DefaultFeatureMap }\n      }\n  }\n\n  private def handleResponse(\n    response: tsp.TopicSocialProofResponse,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[FeatureMap] = {\n    candidates.map { candidate =>\n      val topicWithScores = response.socialProofs.getOrElse(candidate.candidate.id, Seq.empty)\n      if (topicWithScores.nonEmpty) {\n        val (socialProofId, socialProofFunctionalityType) =\n          if (SuggestTypesToSetSocialProof.contains(\n              candidate.features.getOrElse(ServedTypeFeature, hmt.ServedType.Undefined))) {\n            getSocialProof(topicWithScores)\n          } else (None, None)\n\n        val inferredTopicFeatures =\n          topicWithScores.sortBy(-_.score).take(topK).map(a => (a.topicId, a.score)).toMap\n\n        val inferredTopicDataRecord =\n          InferredTopicAdapter.adaptToDataRecords(inferredTopicFeatures).asScala.head\n\n        FeatureMap(\n          TSPInferredTopicFeature,\n          inferredTopicFeatures,\n          TSPInferredTopicDataRecordFeature,\n          inferredTopicDataRecord,\n          TopicIdSocialContextFeature,\n          socialProofId,\n          TopicContextFunctionalityTypeFeature,\n          socialProofFunctionalityType\n        )\n      } else DefaultFeatureMap\n    }\n  }\n\n  private def getSocialProof(\n    topicWithScores: Seq[tsp.TopicWithScore]\n  ): (Option[Long], Option[TopicContextFunctionalityType]) = {\n    val followingTopicId = topicWithScores.collectFirst {\n      case tsp.TopicWithScore(topicId, _, _, Some(tsp.TopicFollowType.Following)) => topicId\n    }\n\n    if (followingTopicId.nonEmpty) {\n      return (followingTopicId, Some(BasicTopicContextFunctionalityType))\n    }\n\n    val implicitFollowingId = topicWithScores.collectFirst {\n      case tsp.TopicWithScore(topicId, _, _, Some(tsp.TopicFollowType.ImplicitFollow)) =>\n        topicId\n    }\n\n    if (implicitFollowingId.nonEmpty) {\n      return (implicitFollowingId, Some(RecommendationTopicContextFunctionalityType))\n    }\n\n    (None, None)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TransformerPostEmbeddingFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.PostTransformerEmbeddingsHomeBlueAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.PostTransformerEmbeddingsHomeGreenAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.PostTransformerEmbeddingsJointBlueAdapter\nimport com.twitter.home_mixer_features.{thriftscala => hmf}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TransformerPostEmbeddingHomeBlueFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject TransformerPostEmbeddingHomeGreenFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject TransformerPostEmbeddingJointBlueFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TransformerPostEmbeddingHomeBlueFeatureHydrator @Inject() (\n  homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint,\n  statsReceiver: StatsReceiver)\n    extends TransformerPostEmbeddingFeatureHydrator(\n      homeMixerFeatureService,\n      statsReceiver,\n      hmf.Cache.TransformerPostEmbeddings,\n      TransformerPostEmbeddingHomeBlueFeature,\n      PostTransformerEmbeddingsHomeBlueAdapter\n    ) {\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"TransformerPostEmbeddingBlue\")\n}\n\n@Singleton\nclass TransformerPostEmbeddingHomeGreenFeatureHydrator @Inject() (\n  homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint,\n  statsReceiver: StatsReceiver)\n    extends TransformerPostEmbeddingFeatureHydrator(\n      homeMixerFeatureService,\n      statsReceiver,\n      hmf.Cache.TransformerPostEmbeddingsGreen,\n      TransformerPostEmbeddingHomeGreenFeature,\n      PostTransformerEmbeddingsHomeGreenAdapter\n    ) {\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"TransformerPostEmbeddingGreen\")\n}\n\n@Singleton\nclass TransformerPostEmbeddingJointBlueFeatureHydrator @Inject() (\n  homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint,\n  statsReceiver: StatsReceiver)\n    extends TransformerPostEmbeddingFeatureHydrator(\n      homeMixerFeatureService,\n      statsReceiver,\n      hmf.Cache.TransformerPostJointEmbeddingsBlue,\n      TransformerPostEmbeddingJointBlueFeature,\n      PostTransformerEmbeddingsJointBlueAdapter\n    ) {\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"TransformerPostEmbeddingGreen\")\n}\n\nabstract class TransformerPostEmbeddingFeatureHydrator(\n  homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint,\n  statsReceiver: StatsReceiver,\n  cache: hmf.Cache,\n  feature: DataRecordInAFeature[TweetCandidate],\n  dataRecordAdapter: TimelinesMutatingAdapterBase[Option[ml.FloatTensor]])\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val features: Set[Feature[_, _]] = Set(feature)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyTotalCounter = scopedStatsReceiver.counter(\"key/total\")\n\n  private val batchSize = 50\n\n  private def getEmbeddingsFromHMF(\n    tweetIds: Seq[Long]\n  ): Future[Seq[Option[Seq[Double]]]] = {\n    val keysSerialized = tweetIds.map(_.toString)\n    val request = hmf.HomeMixerFeaturesRequest(keysSerialized, cache)\n    val responseFut =\n      homeMixerFeatureService.getHomeMixerFeatures(request)\n    responseFut\n      .map { response =>\n        response.homeMixerFeatures\n          .map { homeMixerFeatureOpt =>\n            homeMixerFeatureOpt.homeMixerFeaturesType.map {\n              case hmf.HomeMixerFeaturesType.RawEmbedding(rawEmbedding) =>\n                rawEmbedding\n              case _ => throw new Exception(\"Unknown type returned\")\n            }\n          }\n      }.handle { case _ => Seq.fill(tweetIds.size)(None) }\n  }\n\n  private def getBatchedFeatureMap(\n    candidatesBatch: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Future[Seq[FeatureMap]] = {\n    val tweetIds =\n      candidatesBatch.map { candidate =>\n        keyTotalCounter.incr()\n        candidate.candidate.id\n      }\n\n    getEmbeddingsFromHMF(tweetIds).map { response =>\n      response.map { embeddingOpt =>\n        val floatTensor =\n          embeddingOpt.map { embedding =>\n            keyFoundCounter.incr()\n            ml.FloatTensor(embedding)\n          }\n        val dataRecord =\n          dataRecordAdapter.adaptToDataRecords(floatTensor).asScala.head\n        FeatureMap(feature, dataRecord)\n      }\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    OffloadFuturePools.offloadBatchSeqToFutureSeq(candidates, getBatchedFeatureMap, batchSize)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TweetEntityServiceContentFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.knuddels.jtokkit.Encodings\nimport com.knuddels.jtokkit.api.Encoding\nimport com.knuddels.jtokkit.api.ModelType\nimport com.twitter.escherbird.{thriftscala => esb}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.content.ContentFeatureAdapter\nimport com.twitter.home_mixer.model.HomeFeatures.HasImageFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasMultipleMedia\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsSelfThreadFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MediaCategoryFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MediaIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MediaUnderstandingAnnotationIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SemanticAnnotationIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetLanguageFromTweetypieFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTextFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTextTokensFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoAspectRatioFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoDurationMsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoHeightFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoWidthFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTweetypieContentFeaturesDeciderParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTweetypieContentFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTweetypieContentMediaEntityFeaturesParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableContentFeatureFromTesService\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.home_mixer.util.tweetypie.content.FeatureExtractionHelper\nimport com.twitter.home_mixer_features.{thriftscala => hmf}\nimport com.twitter.mediaservices.commons.thriftscala.MediaCategory\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.common.util.MediaUnderstandingAnnotations\nimport com.twitter.tweetypie.{thriftscala => tp}\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TweetypieContentDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\ncase class TweetContentExtractionResult(\n  annotations: Seq[esb.TweetEntityAnnotation] = Seq.empty,\n  contentDataRecord: DataRecord = new DataRecord(),\n  videoDurationMs: Option[Int] = None,\n  tweetLanguage: Option[String] = None,\n  tweetText: Option[String] = None,\n  tweetTextTokens: Option[Seq[Int]] = None,\n  aspectRatio: Option[Float] = None,\n  height: Option[Short] = None,\n  width: Option[Short] = None,\n  isSelfThread: Boolean = false,\n  mediaId: Option[Long] = None,\n  mediaCategory: Option[MediaCategory] = None,\n  hasMultipleMedia: Option[Boolean] = None,\n  hasImage: Option[Boolean] = None,\n  hasVideo: Option[Boolean] = None)\n\n@Singleton\nclass TweetEntityServiceContentFeatureHydrator @Inject() (\n  homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint,\n  override val statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with ObservedKeyValueResultHandler\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"TweetEntityServiceContent\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    MediaUnderstandingAnnotationIdsFeature,\n    SemanticAnnotationIdsFeature,\n    TweetypieContentDataRecordFeature,\n    VideoDurationMsFeature,\n    TweetLanguageFromTweetypieFeature,\n    TweetTextFeature,\n    TweetTextTokensFeature,\n    VideoAspectRatioFeature,\n    VideoHeightFeature,\n    VideoWidthFeature,\n    IsSelfThreadFeature,\n    MediaIdFeature,\n    MediaCategoryFeature,\n    HasMultipleMedia,\n    HasImageFeature,\n    HasVideoFeature\n  )\n\n  override def onlyIf(\n    query: PipelineQuery\n  ): Boolean = query.params(EnableTweetypieContentFeaturesDeciderParam) &&\n    query.params(EnableTweetypieContentFeaturesParam)\n\n  override val statScope: String = identifier.toString\n  private val batchSize = 64\n  val tokenizer: Encoding =\n    Encodings.newLazyEncodingRegistry().getEncodingForModel(ModelType.GPT_4)\n\n  private def getContentFeaturesFromHMF(\n    tweetIdsToHydrate: Seq[Long],\n    getFromTES: Boolean = false\n  ): Future[Seq[Option[tp.Tweet]]] = {\n    val keysSerialized = tweetIdsToHydrate.map(_.toString)\n    val request = hmf.HomeMixerFeaturesRequest(\n      keysSerialized,\n      hmf.Cache.TweetypieContent,\n      Some(\n        hmf.HomeMixerFeaturesRequestContext.ContentFeatureRequestContext(\n          hmf.ContentFeatureRequestContext(Some(getFromTES))\n        ))\n    )\n    val responseFut =\n      homeMixerFeatureService.getHomeMixerFeatures(request)\n    responseFut\n      .map { response =>\n        response.homeMixerFeatures\n          .map { homeMixerFeatureOpt =>\n            homeMixerFeatureOpt.homeMixerFeaturesType.map {\n              case hmf.HomeMixerFeaturesType.TweetypieContent(homeMixerFeature) =>\n                homeMixerFeature\n              case _ => throw new Exception(\"Unknown type returned\")\n            }\n          }\n      }.handle { case _ => Seq.fill(tweetIdsToHydrate.size)(None) }\n  }\n\n  private def getFeatureMaps(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n    query: PipelineQuery,\n    getFromTes: Boolean\n  ): Future[Seq[FeatureMap]] = {\n    val tweetIdsToHydrate = candidates.map(CandidatesUtil.getOriginalTweetId)\n\n    val isExtractMediaEntities = query.params(EnableTweetypieContentMediaEntityFeaturesParam)\n\n    val responseMap = getContentFeaturesFromHMF(tweetIdsToHydrate, getFromTes)\n\n    responseMap.map { result =>\n      result.map { tweetContent =>\n        val transformed = postTransformer(tweetContent, isExtractMediaEntities)\n        val annotationIds = transformed.annotations.map(_.entityId)\n        val mediaUnderstandingAnnotationIds =\n          getNonSensitiveHighRecallMediaUnderstandingAnnotationEntityIds(transformed.annotations)\n        FeatureMapBuilder(sizeHint = 13)\n          .add(MediaUnderstandingAnnotationIdsFeature, mediaUnderstandingAnnotationIds)\n          .add(SemanticAnnotationIdsFeature, annotationIds)\n          .add(TweetypieContentDataRecordFeature, transformed.contentDataRecord)\n          .add(VideoDurationMsFeature, transformed.videoDurationMs)\n          .add(TweetLanguageFromTweetypieFeature, transformed.tweetLanguage)\n          .add(TweetTextFeature, transformed.tweetText)\n          .add(TweetTextTokensFeature, transformed.tweetTextTokens)\n          .add(VideoAspectRatioFeature, transformed.aspectRatio)\n          .add(VideoHeightFeature, transformed.height)\n          .add(VideoWidthFeature, transformed.width)\n          .add(IsSelfThreadFeature, transformed.isSelfThread)\n          .add(MediaIdFeature, transformed.mediaId)\n          .add(MediaCategoryFeature, transformed.mediaCategory)\n          .add(HasMultipleMedia, transformed.hasMultipleMedia.getOrElse(false))\n          .add(HasImageFeature, transformed.hasImage.getOrElse(false))\n          .add(HasVideoFeature, transformed.hasVideo.getOrElse(false))\n          .build()\n      }\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val getFromTes = query.params(EnableContentFeatureFromTesService)\n    OffloadFuturePools.offloadBatchSeqToFutureSeq(\n      candidates,\n      getFeatureMaps(_, query, getFromTes),\n      batchSize)\n  }\n\n  private def postTransformer(\n    result: Option[tp.Tweet],\n    isExtractMediaEntities: Boolean = true\n  ): TweetContentExtractionResult = {\n    val transformedValue =\n      result.map(FeatureExtractionHelper.extractFeatures(_, isExtractMediaEntities))\n    val semanticAnnotations =\n      transformedValue.flatMap { _.semanticCoreAnnotations }.getOrElse(Seq.empty)\n    val dataRecord = ContentFeatureAdapter.adaptToDataRecords(transformedValue).asScala.head\n    val videoDurationMs = transformedValue.flatMap { _.videoDurationMs }\n\n    val mediaId = transformedValue.flatMap { _.media.flatMap(_.headOption).map(_.mediaId) }\n    val hasMultipleMedia =\n      Some(transformedValue.map(_.media.map(_.size > 1).getOrElse(false)).getOrElse(false))\n    val mediaCategory = transformedValue.flatMap {\n      _.media.flatMap(_.headOption).flatMap(_.mediaKey).map(_.mediaCategory)\n    }\n    val tweetLanguage = result.flatMap { _.language.map(_.language) }\n    val tweetText = result.flatMap { _.coreData.map(_.text) }\n    val tweetTextTokens = tweetText.map { text =>\n      tokenizer.encodeOrdinary(text, 1024).getTokens.toArray.toSeq\n    }\n    val aspectRatioNum = transformedValue.flatMap { _.aspectRatioNum }\n    val aspectRatioDen = transformedValue.flatMap { _.aspectRatioDen }\n    val aspectRatio = aspectRatioNum\n      .zip(aspectRatioDen).map {\n        case (num, den) =>\n          if (den != 0) num.toFloat / den.toFloat\n          else -1\n      }.find(_ > 0)\n    val mediaHeight = transformedValue.flatMap { _.heights.flatMap(_.headOption) }\n    val mediaWidth = transformedValue.flatMap { _.widths.flatMap(_.headOption) }\n    val isSelfThread = transformedValue.exists(_.selfThreadMetadata.nonEmpty)\n    val hasImage = transformedValue.flatMap(_.hasImage)\n    val hasVideo = transformedValue.flatMap(_.hasVideo)\n    TweetContentExtractionResult(\n      semanticAnnotations,\n      dataRecord,\n      videoDurationMs,\n      tweetLanguage,\n      tweetText,\n      tweetTextTokens,\n      aspectRatio,\n      mediaHeight,\n      mediaWidth,\n      isSelfThread,\n      mediaId,\n      mediaCategory,\n      hasMultipleMedia,\n      hasImage,\n      hasVideo\n    )\n  }\n\n  private def getNonSensitiveHighRecallMediaUnderstandingAnnotationEntityIds(\n    semanticCoreAnnotations: Seq[esb.TweetEntityAnnotation]\n  ): Seq[Long] = semanticCoreAnnotations\n    .filter(MediaUnderstandingAnnotations.isEligibleNonSensitiveHighRecallMUAnnotation)\n    .map(_.entityId)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TweetEntityServiceFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.conversions.DurationOps.RichDuration\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ExclusiveConversationAuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FirstMediaIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasImageFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsArticleFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsInReplyToReplyOrDirectedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsInReplyToRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MentionScreenNameFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MentionUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.OriginalTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetUrlsFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TesBatchedStratoClient\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetypieStaticEntitiesCache\nimport com.twitter.home_mixer.util.tweetypie.content.TweetMediaFeaturesExtractor\nimport com.twitter.mediaservices.commons.thriftscala.MediaKey\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityIdFeature\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.TtlCache\nimport com.twitter.servo.keyvalue.KeyValueResult\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.client.CommunityId\nimport com.twitter.strato.client.UserId\nimport com.twitter.strato.generated.client.tweetypie.federated.ArticleOnTweetClientColumn\nimport com.twitter.strato.generated.client.tweetypie.federated.CommunityOnTweetClientColumn\nimport com.twitter.strato.generated.client.tweetypie.federated.ContextualQuoteTweetRefOnTweetClientColumn\nimport com.twitter.strato.generated.client.tweetypie.federated.DirectedAtUserOnTweetClientColumn\nimport com.twitter.strato.generated.client.tweetypie.federated.ExclusiveTweetControlOnTweetClientColumn\nimport com.twitter.strato.generated.client.tweetypie.federated.MediaKeysOnTweetClientColumn\nimport com.twitter.strato.generated.client.tweetypie.federated.MentionsOnTweetClientColumn\nimport com.twitter.strato.generated.client.tweetypie.federated.NarrowcastPlaceOnTweetClientColumn\nimport com.twitter.strato.generated.client.tweetypie.federated.PureCoreDataOnTweetClientColumn\nimport com.twitter.strato.generated.client.tweetypie.federated.UrlsOnTweetClientColumn\nimport com.twitter.strato.graphql.contextual_refs.thriftscala.ContextualTweetRef\nimport com.twitter.tweetypie.{thriftscala => tp}\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetEntityServiceFeatureHydrator @Inject() (\n  @Named(TweetypieStaticEntitiesCache) cacheClient: TtlCache[Long, tp.Tweet],\n  @Named(TesBatchedStratoClient) stratoClient: Client,\n  statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n  import TweetEntityServiceFeatureHydrator._\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(IdentifierName)\n\n  override val features: Set[Feature[_, _]] = Features\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val pureCoreDataNonEmptyCounter = scopedStatsReceiver.counter(\"pureCoreDataNonEmpty\")\n  private val pureCoreDataEmptyCounter = scopedStatsReceiver.counter(\"pureCoreDataEmpty\")\n\n  private lazy val getTESRecord: Arrow[Long, TESRecord] = getTESRecordArrow(stratoClient)\n\n  private lazy val getFromTES: Arrow[Seq[Long], Map[Long, tp.Tweet]] = Arrow\n    .sequence(getTESRecord)\n    .map { record =>\n      record\n        .map(_.getTweetOpt).filter { tweetOpt =>\n          if (tweetOpt.nonEmpty) pureCoreDataNonEmptyCounter.incr()\n          else pureCoreDataEmptyCounter.incr()\n          tweetOpt.nonEmpty\n        }.map(tweet => (tweet.get.id, tweet.get)).toMap\n    }\n\n  private lazy val getHydratedTweetMapWithCacheWriteBack: Arrow[\n    KeyValueResult[Long, tp.Tweet],\n    Map[Long, tp.Tweet]\n  ] = Arrow\n    .zipWithArg(\n      Arrow\n        .identity[KeyValueResult[Long, tp.Tweet]]\n        .andThen(getFromTES.contramap[KeyValueResult[Long, tp.Tweet]](kv => kv.notFound.toSeq))\n    ).map {\n      case (fromCache, fromTES) =>\n        fromTES.map(kv => cacheClient.set(kv._1, kv._2, CacheTTL))\n        fromCache.found ++ fromTES\n    }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    val tweetIds: Seq[Long] = candidates.map(_.candidate.id)\n    val inReplyToIds = candidates.flatMap(_.features.getOrElse(InReplyToTweetIdFeature, None))\n    val idsToHydrate = (tweetIds ++ inReplyToIds).distinct\n\n    Stitch\n      .callFuture(cacheClient.get(idsToHydrate))\n      .flatMap(cacheResult => getHydratedTweetMapWithCacheWriteBack(cacheResult))\n      .map { tweetMap: Map[Long, tp.Tweet] =>\n        candidates.map { candidate =>\n          tweetMap\n            .get(candidate.candidate.id).map { tweet =>\n              getFeatureMapFromTweet(\n                tweet,\n                candidate.features.getOrElse(InReplyToTweetIdFeature, None).flatMap(tweetMap.get)\n              )\n            }.getOrElse(DefaultFeatureMap)\n        }\n      }\n  }\n}\n\nobject TweetEntityServiceFeatureHydrator {\n  private val IdentifierName = \"TweetEntityService\"\n  private val CacheTTL = 48.hours\n\n  private val Features: Set[Feature[_, _]] = Set(\n    AuthorIdFeature,\n    CommunityIdFeature,\n    DirectedAtUserIdFeature,\n    ExclusiveConversationAuthorIdFeature,\n    FirstMediaIdFeature,\n    HasImageFeature,\n    HasVideoFeature,\n    IsArticleFeature,\n    InReplyToTweetIdFeature,\n    InReplyToUserIdFeature,\n    IsInReplyToReplyOrDirectedFeature,\n    IsInReplyToRetweetFeature,\n    IsRetweetFeature,\n    LocationIdFeature,\n    MentionScreenNameFeature,\n    MentionUserIdFeature,\n    QuotedTweetIdFeature,\n    OriginalTweetIdFeature,\n    SourceTweetIdFeature,\n    SourceUserIdFeature,\n    TweetMediaIdsFeature,\n    TweetUrlsFeature,\n  )\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(AuthorIdFeature, None)\n    .add(DirectedAtUserIdFeature, None)\n    .add(ExclusiveConversationAuthorIdFeature, None)\n    .add(HasImageFeature, false)\n    .add(HasVideoFeature, false)\n    .add(IsArticleFeature, false)\n    .add(InReplyToTweetIdFeature, None)\n    .add(InReplyToUserIdFeature, None)\n    .add(IsInReplyToReplyOrDirectedFeature, false)\n    .add(IsInReplyToRetweetFeature, false)\n    .add(IsRetweetFeature, false)\n    .add(LocationIdFeature, None)\n    .add(MentionScreenNameFeature, Seq.empty)\n    .add(MentionUserIdFeature, Seq.empty)\n    .add(QuotedTweetIdFeature, None)\n    .add(OriginalTweetIdFeature, None)\n    .add(SourceTweetIdFeature, None)\n    .add(SourceUserIdFeature, None)\n    .add(CommunityIdFeature, None)\n    .add(TweetMediaIdsFeature, Seq.empty)\n    .add(FirstMediaIdFeature, None)\n    .add(TweetUrlsFeature, Seq.empty)\n    .build()\n\n  private def getFeatureMapFromTweet(\n    tweet: tp.Tweet,\n    inReplyToTweet: Option[tp.Tweet]\n  ): FeatureMap = {\n    val coreData = tweet.coreData\n    val quotedTweet = tweet.quotedTweet\n    val mentions = tweet.mentions.getOrElse(Seq.empty)\n    val share = coreData.flatMap(_.share)\n    val reply = coreData.flatMap(_.reply)\n    val urls = tweet.urls.map(_.flatMap(_.expanded)).toSeq.flatten\n\n    val (isInReplyToReplyOrDirected, isInReplyToRetweet) = inReplyToTweet\n      .map { tweet =>\n        (\n          // when inReplyToUserId exists, it can be a reply or a directedAt tweet,\n          // depending on whether inReplyToTweetId exists\n          tweet.coreData.flatMap(_.reply).map(_.inReplyToUserId).isDefined,\n          tweet.coreData.flatMap(_.share).isDefined\n        )\n      }.getOrElse((false, false))\n\n    // There are cases where the inReplyToUserId exists while inReplyToStatusId does not.\n    // They're usually directed tweets that are not replies.\n    val inReplyToTweetId = reply.flatMap(_.inReplyToStatusId)\n    val inReplyToUserId = if (inReplyToTweetId.nonEmpty) reply.map(_.inReplyToUserId) else None\n    val tweetMediaIds = TweetMediaFeaturesExtractor.getMediaIds(tweet)\n\n    FeatureMapBuilder()\n      .add(AuthorIdFeature, coreData.map(_.userId))\n      .add(DirectedAtUserIdFeature, coreData.flatMap(_.directedAtUser.map(_.userId)))\n      .add(\n        ExclusiveConversationAuthorIdFeature,\n        tweet.exclusiveTweetControl.map(_.conversationAuthorId))\n      .add(HasImageFeature, TweetMediaFeaturesExtractor.hasImage(tweet))\n      .add(HasVideoFeature, TweetMediaFeaturesExtractor.hasVideo(tweet))\n      .add(IsArticleFeature, tweet.article.isDefined)\n      .add(InReplyToTweetIdFeature, inReplyToTweetId)\n      .add(InReplyToUserIdFeature, inReplyToUserId)\n      .add(IsRetweetFeature, share.isDefined)\n      .add(IsInReplyToReplyOrDirectedFeature, isInReplyToReplyOrDirected)\n      .add(IsInReplyToRetweetFeature, isInReplyToRetweet)\n      .add(LocationIdFeature, tweet.narrowcastPlace.map(_.id))\n      .add(MentionScreenNameFeature, mentions.map(_.screenName))\n      .add(MentionUserIdFeature, mentions.flatMap(_.userId))\n      .add(QuotedTweetIdFeature, quotedTweet.map(_.tweetId))\n      .add(OriginalTweetIdFeature, Some(share.map(_.sourceStatusId).getOrElse(tweet.id)))\n      .add(SourceTweetIdFeature, share.map(_.sourceStatusId))\n      .add(SourceUserIdFeature, share.map(_.sourceUserId))\n      .add(CommunityIdFeature, tweet.communities.flatMap(_.communityIds.headOption))\n      .add(TweetMediaIdsFeature, tweetMediaIds)\n      .add(FirstMediaIdFeature, tweetMediaIds.headOption)\n      .add(TweetUrlsFeature, urls)\n      .build()\n  }\n\n  private def getTESRecordArrow(stratoClient: Client): Arrow[Long, TESRecord] = {\n    val pureCoreDataArrow: Arrow[Long, Option[tp.PureCoreData]] =\n      new PureCoreDataOnTweetClientColumn(stratoClient).fetcher.asArrow\n        .contramap[Long](tweetId => (tweetId, ()))\n        .map(_.v)\n\n    val communityArrow: Arrow[Long, Option[CommunityId]] =\n      new CommunityOnTweetClientColumn(stratoClient).fetcher.asArrow\n        .contramap[Long](tweetId => (tweetId, ()))\n        .map(_.v)\n\n    val directedAtUserArrow: Arrow[Long, Option[UserId]] =\n      new DirectedAtUserOnTweetClientColumn(stratoClient).fetcher.asArrow\n        .contramap[Long](tweetId => (tweetId, ()))\n        .map(_.v)\n\n    val exclusiveTweetControlArrow: Arrow[Long, Option[tp.ExclusiveTweetControl]] =\n      new ExclusiveTweetControlOnTweetClientColumn(stratoClient).fetcher.asArrow\n        .contramap[Long](tweetId => (tweetId, ()))\n        .map(_.v)\n\n    val mediaKeysArrow: Arrow[Long, Option[Seq[MediaKey]]] =\n      new MediaKeysOnTweetClientColumn(stratoClient).fetcher.asArrow\n        .contramap[Long](tweetId => (tweetId, ()))\n        .map(_.v)\n\n    val mentionsArrow: Arrow[Long, Option[Seq[tp.MentionEntity]]] =\n      new MentionsOnTweetClientColumn(stratoClient).fetcher.asArrow\n        .contramap[Long](tweetId => (tweetId, ()))\n        .map(_.v)\n\n    val contextualQuoteTweetRefArrow: Arrow[Long, Option[ContextualTweetRef]] =\n      new ContextualQuoteTweetRefOnTweetClientColumn(stratoClient).fetcher.asArrow\n        .contramap[Long](tweetId => (tweetId, ()))\n        .map(_.v)\n\n    val narrowcastPlaceArrow: Arrow[Long, Option[tp.NarrowcastPlace]] =\n      new NarrowcastPlaceOnTweetClientColumn(stratoClient).fetcher.asArrow\n        .contramap[Long](tweetId => (tweetId, ()))\n        .map(_.v)\n\n    val articleArrow: Arrow[Long, Option[tp.Article]] =\n      new ArticleOnTweetClientColumn(stratoClient).fetcher.asArrow\n        .contramap[Long](tweetId => (tweetId, ()))\n        .map(_.v)\n\n    val urlsArrow: Arrow[Long, Option[Seq[tp.UrlEntity]]] =\n      new UrlsOnTweetClientColumn(stratoClient).fetcher.asArrow\n        .contramap[Long](tweetId => (tweetId, ()))\n        .map(_.v)\n\n    Arrow\n      .zipWithArg(\n        Arrow\n          .identity[Long].andThen(Arrow.join(\n            pureCoreDataArrow,\n            articleArrow,\n            communityArrow,\n            directedAtUserArrow,\n            exclusiveTweetControlArrow,\n            mediaKeysArrow,\n            mentionsArrow,\n            narrowcastPlaceArrow,\n            contextualQuoteTweetRefArrow,\n            urlsArrow\n          ))\n      ).map {\n        case (\n              tweetId,\n              (\n                pureCoreDataOpt: Option[tp.PureCoreData],\n                articleOpt: Option[tp.Article],\n                communityIdOpt: Option[CommunityId],\n                directedAtUserIdOpt: Option[UserId],\n                exclusiveTweetControls: Option[tp.ExclusiveTweetControl],\n                mediaKeysOpt: Option[Seq[MediaKey]],\n                mentionEntitiesOpt: Option[Seq[tp.MentionEntity]],\n                narrowcastPlaceOpt: Option[tp.NarrowcastPlace],\n                quotedTweetOpt: Option[ContextualTweetRef],\n                urlsOpt: Option[Seq[tp.UrlEntity]]\n              )) =>\n          TESRecord(\n            tweetId,\n            pureCoreDataOpt,\n            articleOpt,\n            communityIdOpt,\n            directedAtUserIdOpt,\n            exclusiveTweetControls,\n            mediaKeysOpt,\n            mentionEntitiesOpt,\n            narrowcastPlaceOpt,\n            quotedTweetOpt,\n            urlsOpt\n          )\n      }\n  }\n}\n\ncase class TESRecord(\n  tweetId: Long,\n  pureCoreDataOpt: Option[tp.PureCoreData],\n  articleOpt: Option[tp.Article],\n  communityIdOpt: Option[CommunityId],\n  directedAtUserIdOpt: Option[UserId],\n  exclusiveTweetControls: Option[tp.ExclusiveTweetControl],\n  mediaKeysOpt: Option[Seq[MediaKey]],\n  mentionEntitiesOpt: Option[Seq[tp.MentionEntity]],\n  narrowcastPlaceOpt: Option[tp.NarrowcastPlace],\n  quotedTweetOpt: Option[ContextualTweetRef],\n  urlsOpt: Option[Seq[tp.UrlEntity]]) {\n\n  def getTweetOpt: Option[tp.Tweet] = pureCoreDataOpt.map { pureCoreData =>\n    tp.Tweet(\n      id = tweetId,\n      coreData = Some(\n        tp.TweetCoreData.unsafeEmpty.copy(\n          userId = pureCoreData.userId,\n          share = pureCoreData.share,\n          reply = pureCoreData.reply,\n          directedAtUser = directedAtUserIdOpt.map(id => tp.DirectedAtUser(id.value, \"\"))\n        )),\n      article = articleOpt,\n      communities = communityIdOpt.map(community => tp.Communities(Seq(community.value))),\n      exclusiveTweetControl = exclusiveTweetControls\n        .map(control => tp.ExclusiveTweetControl(control.conversationAuthorId)),\n      media = mediaKeysOpt.map { mediaKeys =>\n        mediaKeys.map { key =>\n          tp.MediaEntity.unsafeEmpty.copy(mediaId = key.mediaId, mediaKey = Some(key))\n        }\n      },\n      mentions = mentionEntitiesOpt,\n      narrowcastPlace = narrowcastPlaceOpt,\n      quotedTweet = quotedTweetOpt.map(qt => tp.QuotedTweet.unsafeEmpty.copy(tweetId = qt.id)),\n      urls = urlsOpt\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TweetImpressionsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.HomeFeatures.TweetImpressionsFeature\nimport com.twitter.home_mixer.model.request.HasSeenTweetIds\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.impression.{thriftscala => t}\nimport com.twitter.timelines.impressionstore.store.ManhattanTweetImpressionStoreClient\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class TweetImpressionsQueryFeatureHydrator[\n  Query <: PipelineQuery with HasSeenTweetIds] @Inject() (\n  manhattanTweetImpressionStoreClient: ManhattanTweetImpressionStoreClient)\n    extends QueryFeatureHydrator[Query] {\n\n  private val TweetImpressionTTL = 2.days\n  private val TweetImpressionCap = 5000\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TweetImpressions\")\n\n  override val features: Set[Feature[_, _]] = Set(TweetImpressionsFeature)\n\n  override def hydrate(query: Query): Stitch[FeatureMap] = {\n    manhattanTweetImpressionStoreClient.get(query.getRequiredUserId).map { entriesOpt =>\n      val entries = entriesOpt.map(_.entries).toSeq.flatten\n      val updatedImpressions =\n        if (query.seenTweetIds.forall(_.isEmpty)) entries\n        else updateTweetImpressions(entries, query.seenTweetIds.get)\n\n      FeatureMapBuilder().add(TweetImpressionsFeature, updatedImpressions).build()\n    }\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8)\n  )\n\n  /**\n   * 1) Check timestamps and remove expired tweets based on [[TweetImpressionTTL]]\n   * 2) Filter duplicates between current tweets and those in the impression store (remove older ones)\n   * 3) Prepend new (Timestamp, Seq[TweetIds]) to the tweets from the impression store\n   * 4) Truncate older tweets if sum of all tweets across timestamps >= [[TweetImpressionCap]],\n   */\n  private[feature_hydrator] def updateTweetImpressions(\n    tweetImpressionsFromStore: Seq[t.TweetImpressionsEntry],\n    seenIdsFromClient: Seq[Long],\n    currentTime: Long = Time.now.inMilliseconds,\n    tweetImpressionTTL: Duration = TweetImpressionTTL,\n    tweetImpressionCap: Int = TweetImpressionCap,\n  ): Seq[t.TweetImpressionsEntry] = {\n    val seenIdsFromClientSet = seenIdsFromClient.toSet\n    val dedupedTweetImpressionsFromStore: Seq[t.TweetImpressionsEntry] = tweetImpressionsFromStore\n      .collect {\n        case t.TweetImpressionsEntry(ts, tweetIds)\n            if Time.fromMilliseconds(ts).untilNow < tweetImpressionTTL =>\n          t.TweetImpressionsEntry(ts, tweetIds.filterNot(seenIdsFromClientSet.contains))\n      }.filter { _.tweetIds.nonEmpty }\n\n    val mergedTweetImpressionsEntries =\n      t.TweetImpressionsEntry(currentTime, seenIdsFromClient) +: dedupedTweetImpressionsFromStore\n    val initialTweetImpressionsWithCap = (Seq.empty[t.TweetImpressionsEntry], tweetImpressionCap)\n\n    val (truncatedTweetImpressionsEntries: Seq[t.TweetImpressionsEntry], _) =\n      mergedTweetImpressionsEntries\n        .foldLeft(initialTweetImpressionsWithCap) {\n          case (\n                (tweetImpressions: Seq[t.TweetImpressionsEntry], remainingCap),\n                t.TweetImpressionsEntry(ts, tweetIds)) if remainingCap > 0 =>\n            (\n              t.TweetImpressionsEntry(ts, tweetIds.take(remainingCap)) +: tweetImpressions,\n              remainingCap - tweetIds.size)\n          case (tweetImpressionsWithCap, _) => tweetImpressionsWithCap\n        }\n    truncatedTweetImpressionsEntries.reverse\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TweetLanguageFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.TweetLanguageFromLanguageSignalFeature\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.language.tweet.LanguageOnTweetClientColumn\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetLanguageFeatureHydrator @Inject() (\n  languageOnTweetClientColumn: LanguageOnTweetClientColumn,\n  statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetLanguage\")\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val failedCounter = scopedStatsReceiver.scope(getClass.getSimpleName).counter(\"failure\")\n\n  private val DefaultFeatureMap =\n    FeatureMapBuilder().add(TweetLanguageFromLanguageSignalFeature, None).build()\n\n  override def features: Set[Feature[_, _]] = Set(TweetLanguageFromLanguageSignalFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    OffloadFuturePools.offloadStitch {\n      Stitch.collect {\n        candidates.map { candidate =>\n          languageOnTweetClientColumn.fetcher\n            .fetch(\n              CandidatesUtil.getOriginalTweetId(candidate),\n              LanguageOnTweetClientColumn\n                .View(true)).map { result =>\n              FeatureMapBuilder()\n                .add(TweetLanguageFromLanguageSignalFeature, result.v)\n                .build()\n            }.rescue {\n              case _ =>\n                failedCounter.incr()\n                Stitch.value(DefaultFeatureMap)\n            }\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TweetLargeEmbeddingsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeLargeEmbeddingsFeatures.TweetLargeEmbeddingsFeature\nimport com.twitter.home_mixer.model.HomeLargeEmbeddingsFeatures.TweetLargeEmbeddingsKeyFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableLargeEmbeddingsFeatureHydrationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelNameParam\nimport com.twitter.home_mixer_features.{thriftscala => hmf}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.large_embeddings.HashingFeatureParams\nimport com.twitter.timelines.prediction.adapters.large_embeddings.HomeMixerLargeEmbeddingsFeatureHydrator\nimport com.twitter.timelines.prediction.adapters.large_embeddings.LargeEmbeddingsAdapter\nimport com.twitter.timelines.prediction.adapters.large_embeddings.TweetLargeEmbeddingsAdapter\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetLargeEmbeddingsFeatureHydrator @Inject() (\n  statsReceiver: StatsReceiver,\n  override val homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery]\n    with HomeMixerLargeEmbeddingsFeatureHydrator {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetLargeEmbeddings\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(TweetLargeEmbeddingsFeature, TweetLargeEmbeddingsKeyFeature)\n\n  override val adapter: LargeEmbeddingsAdapter = TweetLargeEmbeddingsAdapter\n\n  override val cacheType: hmf.Cache = hmf.Cache.TweetLargeEmbeddings\n\n  override val scopedStatsReceiver: StatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableLargeEmbeddingsFeatureHydrationParam)\n\n  override val modelName2HashingFeatureParams: Map[String, HashingFeatureParams] = Map(\n    \"hr_video_prod__v3_realtime\" -> HashingFeatureParams(\n      scales = Seq(1487022661L, 1399245971L),\n      biases = Seq(1372992088L, 632996194L),\n      modulus = 2865175829L,\n      bucketSize = 10000000L,\n    ),\n    \"hr_video_prod__v2_lembeds\" -> HashingFeatureParams(\n      scales = Seq(2516541900L, 2376187492L),\n      biases = Seq(3022238687L, 1571354734L),\n      modulus = 3047336911L,\n      bucketSize = 1000000L,\n    ),\n    \"hr_prod__v4_embeds_230M\" -> HashingFeatureParams(\n      scales = Seq(2161410491L, 1754358832L),\n      biases = Seq(296686044L, 1959990826L),\n      modulus = 2361375383L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v5_embeds_230M_and_transformer\" -> HashingFeatureParams(\n      scales = Seq(2161410491L, 1754358832L),\n      biases = Seq(296686044L, 1959990826L),\n      modulus = 2361375383L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v5_watchtime\" -> HashingFeatureParams(\n      scales = Seq(407033648L, 940305868L),\n      biases = Seq(494266171L, 269596788L),\n      modulus = 949146421L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v6_transformer_v2\" -> HashingFeatureParams(\n      scales = Seq(2161410491L, 1754358832L),\n      biases = Seq(296686044L, 1959990826L),\n      modulus = 2361375383L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v6_mixed_training\" -> HashingFeatureParams(\n      scales = Seq(2161410491L, 1754358832L),\n      biases = Seq(296686044L, 1959990826L),\n      modulus = 2361375383L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v6_transformer_v2_kafka_merge_join\" -> HashingFeatureParams(\n      scales = Seq(2161410491L, 1754358832L),\n      biases = Seq(296686044L, 1959990826L),\n      modulus = 2361375383L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v6_transformer_v2_realtime_debias_21apr\" -> HashingFeatureParams(\n      scales = Seq(2161410491L, 1754358832L),\n      biases = Seq(296686044L, 1959990826L),\n      modulus = 2361375383L,\n      bucketSize = 100000000L,\n    ),\n  )\n\n  // Hashing Features\n  override val defaultHashingFeatureParams: HashingFeatureParams = HashingFeatureParams(\n    scales = Seq(1131302000L, 303023026L),\n    biases = Seq(799473858L, 600426834L),\n    modulus = 3588720353L,\n    bucketSize = 10000000L,\n  )\n\n  private val batchSize = 25\n\n  private def getBatchedFeatureMap(\n    modelName: String,\n    candidatesBatch: Seq[CandidateWithFeatures[TweetCandidate]],\n  ): Future[Seq[FeatureMap]] = {\n    val tweetIds = candidatesBatch.map { candidate => candidate.candidate.id }\n\n    getLargeEmbeddings(tweetIds, modelName).map { responses =>\n      responses.map { largeEmbeddingResponse =>\n        FeatureMapBuilder()\n          .add(TweetLargeEmbeddingsFeature, largeEmbeddingResponse.dataRecord)\n          .add(TweetLargeEmbeddingsKeyFeature, largeEmbeddingResponse.hashedKeys)\n          .build()\n      }\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val modelName = query.params(ModelNameParam)\n    OffloadFuturePools.offloadBatchSeqToFutureSeq(\n      candidates,\n      getBatchedFeatureMap(modelName, _),\n      batchSize\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TweetMetaDataFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.constant.SharedFeatures\nimport com.twitter.ml.api.util.DataRecordConverters._\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport java.lang.{Long => JLong}\n\nobject TweetMetaDataDataRecord\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject TweetMetaDataFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TweetMetaData\")\n\n  override def features: Set[Feature[_, _]] = Set(TweetMetaDataDataRecord)\n\n  private val batchSize = 64\n\n  def getFeatureMap(candidate: CandidateWithFeatures[TweetCandidate]): FeatureMap = {\n    val richDataRecord = new RichDataRecord()\n    setFeatures(richDataRecord, candidate.candidate, candidate.features)\n    FeatureMap(TweetMetaDataDataRecord, richDataRecord.getRecord)\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    OffloadFuturePools.offloadBatchElementToElement(candidates, getFeatureMap, batchSize)\n  }\n\n  private def setFeatures(\n    richDataRecord: RichDataRecord,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Unit = {\n    richDataRecord.setFeatureValue[JLong](SharedFeatures.TWEET_ID, candidate.id)\n\n    richDataRecord.setFeatureValueFromOption(\n      TimelinesSharedFeatures.ORIGINAL_AUTHOR_ID,\n      CandidatesUtil.getOriginalAuthorId(existingFeatures))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TweetTimeFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.NonPollingTimesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetAgeFeature\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.util.FDsl._\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures._\nimport com.twitter.util.Duration\n\nimport scala.collection.Searching._\n\nobject TweetTimeDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject TweetTimeFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with WithDefaultFeatureMap {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TweetTime\")\n\n  override val features: Set[Feature[_, _]] = Set(TweetTimeDataRecordFeature, TweetAgeFeature)\n\n  override val defaultFeatureMap: FeatureMap =\n    FeatureMap(\n      TweetTimeDataRecordFeature,\n      TweetTimeDataRecordFeature.defaultValue,\n      TweetAgeFeature,\n      None\n    )\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offload {\n\n    val nonPollingTimestampsMs = query.features.get.getOrElse(NonPollingTimesFeature, Seq.empty)\n\n    candidates.map { candidate =>\n      val tweetFeatures = candidate.features.getOrElse(EarlybirdFeature, None)\n      val timeSinceTweetCreation =\n        SnowflakeId.timeFromIdOpt(candidate.candidate.id).map(query.queryTime.since)\n      val timeSinceTweetCreationMs = timeSinceTweetCreation.map(_.inMillis)\n\n      val timeSinceSourceTweetCreationOpt = candidate.features\n        .getOrElse(SourceTweetIdFeature, None)\n        .flatMap { sourceTweetId =>\n          SnowflakeId.timeFromIdOpt(sourceTweetId).map(query.queryTime.since)\n        }.orElse(timeSinceTweetCreation)\n\n      val lastFavSinceCreationHrs =\n        tweetFeatures.flatMap(_.lastFavSinceCreationHrs).map(_.toDouble)\n      val lastRetweetSinceCreationHrs =\n        tweetFeatures.flatMap(_.lastRetweetSinceCreationHrs).map(_.toDouble)\n      val lastReplySinceCreationHrs =\n        tweetFeatures.flatMap(_.lastReplySinceCreationHrs).map(_.toDouble)\n      val lastQuoteSinceCreationHrs =\n        tweetFeatures.flatMap(_.lastQuoteSinceCreationHrs).map(_.toDouble)\n      val timeSinceLastFavoriteHrs =\n        getTimeSinceLastEngagementHrs(lastFavSinceCreationHrs, timeSinceSourceTweetCreationOpt)\n      val timeSinceLastRetweetHrs =\n        getTimeSinceLastEngagementHrs(lastRetweetSinceCreationHrs, timeSinceSourceTweetCreationOpt)\n      val timeSinceLastReplyHrs =\n        getTimeSinceLastEngagementHrs(lastReplySinceCreationHrs, timeSinceSourceTweetCreationOpt)\n      val timeSinceLastQuoteHrs =\n        getTimeSinceLastEngagementHrs(lastQuoteSinceCreationHrs, timeSinceSourceTweetCreationOpt)\n\n      val timeSinceLastNonPollingRequest =\n        nonPollingTimestampsMs.headOption.map(query.queryTime.inMillis - _)\n\n      val nonPollingRequestsSinceTweetCreation =\n        if (nonPollingTimestampsMs.nonEmpty && timeSinceTweetCreationMs.isDefined) {\n          nonPollingTimestampsMs\n            .search(timeSinceTweetCreationMs.get)(Ordering[Long].reverse)\n            .insertionPoint\n        } else 0.0\n\n      val tweetAgeRatio =\n        if (timeSinceTweetCreationMs.exists(_ > 0.0) && timeSinceLastNonPollingRequest.isDefined) {\n          timeSinceLastNonPollingRequest.get / timeSinceTweetCreationMs.get.toDouble\n        } else 0.0\n\n      val dataRecord = new DataRecord()\n        .setFeatureValue(IS_TWEET_RECYCLED, false)\n        .setFeatureValue(TWEET_AGE_RATIO, tweetAgeRatio)\n        .setFeatureValueFromOption(\n          TIME_SINCE_TWEET_CREATION,\n          timeSinceTweetCreationMs.map(_.toDouble)\n        )\n        .setFeatureValue(\n          NON_POLLING_REQUESTS_SINCE_TWEET_CREATION,\n          nonPollingRequestsSinceTweetCreation\n        )\n        .setFeatureValueFromOption(LAST_FAVORITE_SINCE_CREATION_HRS, lastFavSinceCreationHrs)\n        .setFeatureValueFromOption(LAST_RETWEET_SINCE_CREATION_HRS, lastRetweetSinceCreationHrs)\n        .setFeatureValueFromOption(LAST_REPLY_SINCE_CREATION_HRS, lastReplySinceCreationHrs)\n        .setFeatureValueFromOption(LAST_QUOTE_SINCE_CREATION_HRS, lastQuoteSinceCreationHrs)\n        .setFeatureValueFromOption(TIME_SINCE_LAST_FAVORITE_HRS, timeSinceLastFavoriteHrs)\n        .setFeatureValueFromOption(TIME_SINCE_LAST_RETWEET_HRS, timeSinceLastRetweetHrs)\n        .setFeatureValueFromOption(TIME_SINCE_LAST_REPLY_HRS, timeSinceLastReplyHrs)\n        .setFeatureValueFromOption(TIME_SINCE_LAST_QUOTE_HRS, timeSinceLastQuoteHrs)\n\n      FeatureMap(TweetTimeDataRecordFeature, dataRecord, TweetAgeFeature, timeSinceTweetCreationMs)\n    }\n  }\n\n  private def getTimeSinceLastEngagementHrs(\n    lastEngagementTimeSinceCreationHrsOpt: Option[Double],\n    timeSinceTweetCreation: Option[Duration]\n  ): Option[Double] = lastEngagementTimeSinceCreationHrsOpt.flatMap { lastEngagementTimeHrs =>\n    timeSinceTweetCreation.map(_.inHours - lastEngagementTimeHrs)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TweetTypeMetricsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeTweetTypePredicates\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTypeMetricsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.suggests.controller_data.Home\n\nobject TweetTypeMetricsFeatureHydrator\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TweetTypeMetrics\")\n\n  override val features: Set[Feature[_, _]] = Set(TweetTypeMetricsFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    // Tweet type metrics are already available for cached tweets and shouldn't be overwritten\n    val tweetTypeMetricsFeature = existingFeatures.getOrElse(TweetTypeMetricsFeature, None)\n    val queryFeatures = query.features.getOrElse(FeatureMap.empty)\n\n    val tweetTypesByteList = if (tweetTypeMetricsFeature.isEmpty) {\n      val bitset = new java.util.BitSet()\n\n      val trueTweetTypes = HomeTweetTypePredicates.PredicateMap.collect {\n        // Not combining query and candidate features to reduce cost, instead running predicate separately\n        case (predicateName, predicate)\n            if (predicate(existingFeatures) | predicate(queryFeatures)) =>\n          predicateName\n      }.toSet\n\n      Home.TweetTypeIdxMap.collect {\n        case (tweetType, index) if trueTweetTypes.contains(tweetType) => bitset.set(index)\n      }\n\n      Some(bitset.toByteArray.toList)\n    } else tweetTypeMetricsFeature\n\n    Stitch.value(FeatureMapBuilder().add(TweetTypeMetricsFeature, tweetTypesByteList).build())\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TweetypieFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ExclusiveConversationAuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsArticleFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedTweetDroppedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetLanguageFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTextFeature\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.model.request.ScoredTweetsProduct\nimport com.twitter.home_mixer.model.request.SubscribedProduct\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableTweetEntityServiceMigrationParam\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithLongTimeout\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityIdFeature\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationIdFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_is_nsfw.IsNsfw\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_visibility_reason.VisibilityReason\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.spam.rtf.{thriftscala => rtf}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.generated.client.tweetypie.managed.HomeMixerOnTweetClientColumn\nimport com.twitter.tweetypie.{thriftscala => tp}\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetypieFeatureHydrator @Inject() (\n  tweetypieStitchClient: TweetypieStitchClient,\n  statsReceiver: StatsReceiver,\n  @Named(BatchedStratoClientWithLongTimeout) stratoClient: Client)\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Logging {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"Tweetypie\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    AuthorIdFeature,\n    CommunityIdFeature,\n    ExclusiveConversationAuthorIdFeature,\n    InReplyToTweetIdFeature,\n    IsArticleFeature,\n    IsHydratedFeature,\n    IsNsfw,\n    IsRetweetFeature,\n    LocationIdFeature,\n    QuotedTweetDroppedFeature,\n    QuotedTweetIdFeature,\n    QuotedUserIdFeature,\n    SourceTweetIdFeature,\n    SourceUserIdFeature,\n    TweetTextFeature,\n    TweetLanguageFeature,\n    VisibilityReason\n  )\n\n  val HydrationFields: Set[tp.TweetInclude] = Set(\n    tp.TweetInclude.TweetFieldId(tp.Tweet.CommunitiesField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.CoreDataField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.SelfThreadMetadataField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.ExclusiveTweetControlField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.IdField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.LanguageField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.QuotedTweetField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.ArticleField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.NarrowcastPlaceField.id)\n  )\n\n  private val tweetypieTweetsFoundCounter =\n    statsReceiver.counter(\"TweetypieTweetsFound\")\n  private val tweetypieTweetsNotFoundCounter =\n    statsReceiver.counter(\"TweetypieTweetsNotFound\")\n  private val tesTweetsFoundCounter =\n    statsReceiver.counter(\"TesTweetsFound\")\n  private val tesTweetsNotFoundCounter =\n    statsReceiver.counter(\"TesTweetsNotFound\")\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(CommunityIdFeature, None)\n    .add(IsArticleFeature, false)\n    .add(IsHydratedFeature, false)\n    .add(IsNsfw, None)\n    .add(LocationIdFeature, None)\n    .add(QuotedTweetDroppedFeature, false)\n    .add(TweetTextFeature, None)\n    .add(VisibilityReason, None)\n    .build()\n\n  private def buildFeatureMap(\n    gtfResult: Stitch[tp.GetTweetFieldsResult],\n    fromTes: Boolean,\n    exclusiveAuthorIdOpt: Option[Long],\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    gtfResult.map {\n      case tp.GetTweetFieldsResult(_, tp.TweetFieldsResultState.Found(found), quoteOpt, _) =>\n        if (fromTes) tesTweetsFoundCounter.incr()\n        else tweetypieTweetsFoundCounter.incr()\n\n        val coreData = found.tweet.coreData\n        val isNsfwAdmin = coreData.exists(_.nsfwAdmin)\n        val isNsfwUser = coreData.exists(_.nsfwUser)\n\n        val quotedTweetDropped = quoteOpt.exists {\n          case _: tp.TweetFieldsResultState.Filtered => true\n          case _: tp.TweetFieldsResultState.NotFound => true\n          case _ => false\n        }\n        val quotedTweetIsNsfw = quoteOpt.exists {\n          case quoteTweet: tp.TweetFieldsResultState.Found =>\n            quoteTweet.found.tweet.coreData.exists(data => data.nsfwAdmin || data.nsfwUser)\n          case _ => false\n        }\n\n        val sourceTweetIsNsfw =\n          found.retweetedTweet.exists(_.coreData.exists(data => data.nsfwAdmin || data.nsfwUser))\n\n        val tweetText = coreData.map(_.text)\n        val tweetLanguage = found.tweet.language.map(_.language)\n\n        val tweetAuthorId = coreData.map(_.userId)\n        val inReplyToTweetId = coreData.flatMap(_.reply.flatMap(_.inReplyToStatusId))\n        val retweetedTweetId = found.retweetedTweet.map(_.id)\n        val quotedTweetId = quoteOpt.flatMap {\n          case quoteTweet: tp.TweetFieldsResultState.Found =>\n            Some(quoteTweet.found.tweet.id)\n          case _ => None\n        }\n\n        val retweetedTweetUserId = found.retweetedTweet.flatMap(_.coreData).map(_.userId)\n        val quotedTweetUserId = quoteOpt.flatMap {\n          case quoteTweet: tp.TweetFieldsResultState.Found =>\n            quoteTweet.found.tweet.coreData.map(_.userId)\n          case _ => None\n        }\n\n        val isNsfw = isNsfwAdmin || isNsfwUser || sourceTweetIsNsfw || quotedTweetIsNsfw\n\n        val tpExclusiveAuthorIdOpt = found.tweet.exclusiveTweetControl.map(_.conversationAuthorId)\n        val updatedExclusiveAuthorId = tpExclusiveAuthorIdOpt.orElse(exclusiveAuthorIdOpt)\n\n        val communityId = found.tweet.communities.flatMap(_.communityIds.headOption)\n\n        FeatureMapBuilder()\n          .add(AuthorIdFeature, tweetAuthorId)\n          .add(CommunityIdFeature, communityId)\n          .add(ExclusiveConversationAuthorIdFeature, updatedExclusiveAuthorId)\n          .add(InReplyToTweetIdFeature, inReplyToTweetId)\n          .add(IsArticleFeature, found.tweet.article.nonEmpty)\n          .add(IsHydratedFeature, true)\n          .add(IsNsfw, Some(isNsfw))\n          .add(IsRetweetFeature, retweetedTweetId.isDefined)\n          .add(LocationIdFeature, found.tweet.narrowcastPlace.map(_.id))\n          .add(QuotedTweetDroppedFeature, quotedTweetDropped)\n          .add(QuotedTweetIdFeature, quotedTweetId)\n          .add(QuotedUserIdFeature, quotedTweetUserId)\n          .add(SourceTweetIdFeature, retweetedTweetId)\n          .add(SourceUserIdFeature, retweetedTweetUserId)\n          .add(TweetLanguageFeature, tweetLanguage)\n          .add(TweetTextFeature, tweetText)\n          .add(VisibilityReason, found.suppressReason)\n          .build()\n\n      case _ =>\n        if (fromTes) tesTweetsNotFoundCounter.incr()\n        else tweetypieTweetsNotFoundCounter.incr()\n\n        DefaultFeatureMap ++ FeatureMapBuilder()\n          .add(AuthorIdFeature, existingFeatures.getOrElse(AuthorIdFeature, None))\n          .add(ExclusiveConversationAuthorIdFeature, exclusiveAuthorIdOpt)\n          .add(InReplyToTweetIdFeature, existingFeatures.getOrElse(InReplyToTweetIdFeature, None))\n          .add(IsRetweetFeature, existingFeatures.getOrElse(IsRetweetFeature, false))\n          .add(LocationIdFeature, existingFeatures.getOrElse(LocationIdFeature, None))\n          .add(QuotedTweetIdFeature, existingFeatures.getOrElse(QuotedTweetIdFeature, None))\n          .add(QuotedUserIdFeature, existingFeatures.getOrElse(QuotedUserIdFeature, None))\n          .add(SourceTweetIdFeature, existingFeatures.getOrElse(SourceTweetIdFeature, None))\n          .add(SourceUserIdFeature, existingFeatures.getOrElse(SourceUserIdFeature, None))\n          .add(TweetLanguageFeature, existingFeatures.getOrElse(TweetLanguageFeature, None))\n          .build()\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    val safetyLevel = query.product match {\n      case FollowingProduct => rtf.SafetyLevel.TimelineHomeLatest\n      case ForYouProduct =>\n        val inNetwork = existingFeatures.getOrElse(InNetworkFeature, true)\n        if (inNetwork) rtf.SafetyLevel.TimelineHome else rtf.SafetyLevel.TimelineHomeRecommendations\n      case ScoredTweetsProduct => rtf.SafetyLevel.TimelineHome\n      case SubscribedProduct => rtf.SafetyLevel.TimelineHomeSubscribed\n      case unknown => throw new UnsupportedOperationException(s\"Unknown product: $unknown\")\n    }\n\n    val tweetFieldsOptions = tp.GetTweetFieldsOptions(\n      tweetIncludes = HydrationFields,\n      includeRetweetedTweet = true,\n      includeQuotedTweet = true,\n      visibilityPolicy = tp.TweetVisibilityPolicy.UserVisible,\n      safetyLevel = Some(safetyLevel),\n      forUserId = query.getOptionalUserId\n    )\n\n    val exclusiveAuthorIdOpt =\n      existingFeatures.getOrElse(ExclusiveConversationAuthorIdFeature, None)\n\n    if (query.params(EnableTweetEntityServiceMigrationParam)) {\n      val fetcher = new HomeMixerOnTweetClientColumn(stratoClient).fetcher\n      fetcher\n        .fetch(\n          candidate.id,\n          tweetFieldsOptions\n        ).map(_.v).flatMap {\n          case Some(result) =>\n            buildFeatureMap(Stitch.value(result), true, exclusiveAuthorIdOpt, existingFeatures)\n          case None =>\n            tesTweetsNotFoundCounter.incr()\n            Stitch.value(DefaultFeatureMap)\n        }\n    } else {\n      val gtfResult: Stitch[tp.GetTweetFieldsResult] =\n        tweetypieStitchClient.getTweetFields(tweetId = candidate.id, options = tweetFieldsOptions)\n      buildFeatureMap(gtfResult, fromTes = false, exclusiveAuthorIdOpt, existingFeatures)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TwhinAuthorFollowFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinAuthorFollowEmbeddingsAdapter\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinAuthorFollowFeatureRepository\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.servo.repository.KeyValueResult\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinAuthorFollowFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinAuthorFollowFeatureHydrator @Inject() (\n  @Named(TwhinAuthorFollowFeatureRepository)\n  client: KeyValueRepository[Seq[Long], Long, ml.FloatTensor],\n  override val statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TwhinAuthorFollow\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinAuthorFollowFeature)\n\n  override val statScope: String = identifier.toString\n\n  private val emptyDataRecord = new DataRecord()\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val possiblyAuthorIds = extractKeys(candidates)\n    val authorIds = possiblyAuthorIds.flatten.distinct\n\n    val response: Future[KeyValueResult[Long, DataRecord]] =\n      if (authorIds.isEmpty) Future.value(KeyValueResult.empty)\n      else client(authorIds).map(_.mapFound(postTransformer))\n\n    response.map { result =>\n      possiblyAuthorIds.map { possiblyAuthorId =>\n        val value =\n          observedGet(key = possiblyAuthorId, keyValueResult = result)\n            .map(_.getOrElse(emptyDataRecord))\n\n        FeatureMapBuilder().add(TwhinAuthorFollowFeature, value).build()\n      }\n    }\n  }\n\n  private def postTransformer(embedding: ml.FloatTensor): DataRecord = {\n    TwhinAuthorFollowEmbeddingsAdapter.adaptToDataRecords(Some(embedding)).asScala.head\n  }\n\n  private def extractKeys(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] = {\n    candidates.map { candidate =>\n      CandidatesUtil.getOriginalAuthorId(candidate.features)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TwhinRebuildTweetFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinRebuildTweetEmbeddingsAdapter\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinRebuildTweetEmbeddingsStore\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableHomeMixerFeaturesService\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer_features.{thriftscala => hmf}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.util.Future\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.simclusters_v2.thriftscala.TwhinTweetEmbedding\nimport com.twitter.simclusters_v2.thriftscala.TwhinEmbeddingDataset\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinRebuildTweetFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinRebuildTweetFeatureHydrator @Inject() (\n  homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint,\n  @Named(TwhinRebuildTweetEmbeddingsStore) store: ReadableStore[(Long, Long), TwhinTweetEmbedding],\n  statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"TwhinRebuildTweet\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinRebuildTweetFeature)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyTotalCounter = scopedStatsReceiver.counter(\"key/total\")\n\n  private val batchSize = 50\n\n  private val versionId = TwhinEmbeddingDataset.RefreshedTwhinTweet.value.toLong\n\n  private def getTwhinEmbeddingsFromHMF(\n    originalTweetIds: Seq[Long]\n  ): Future[Seq[Option[TwhinTweetEmbedding]]] = {\n    val keysSerialized = originalTweetIds.map(_.toString)\n    val request = hmf.HomeMixerFeaturesRequest(keysSerialized, hmf.Cache.TwhinRebuild)\n    val responseFut =\n      homeMixerFeatureService.getHomeMixerFeatures(request)\n    responseFut\n      .map { response =>\n        response.homeMixerFeatures\n          .map { homeMixerFeatureOpt =>\n            homeMixerFeatureOpt.homeMixerFeaturesType.map {\n              case hmf.HomeMixerFeaturesType.TwhinTweetEmbedding(homeMixerFeature) =>\n                homeMixerFeature\n              case _ => throw new Exception(\"Unknown type returned\")\n            }\n          }\n      }.handle { case _ => Seq.fill(originalTweetIds.size)(None) }\n  }\n\n  private def getTwhinEmbeddingsFromReadableStore(\n    originalTweetIds: Seq[Long]\n  ): Future[Seq[Option[TwhinTweetEmbedding]]] = {\n    val tweetIdVersionIdPairs = originalTweetIds.map(tweetId => (tweetId, versionId))\n    Future.collect(store.multiGet(tweetIdVersionIdPairs.toSet)).map { storeResponse =>\n      tweetIdVersionIdPairs.map {\n        storeResponse.getOrElse(_, None)\n      }\n    }\n  }\n\n  private def getBatchedFeatureMap(\n    candidatesBatch: Seq[CandidateWithFeatures[TweetCandidate]],\n    callMiddleMan: Boolean\n  ): Future[Seq[FeatureMap]] = {\n    val originalTweetIds =\n      candidatesBatch.map { candidate =>\n        {\n          keyTotalCounter.incr()\n          CandidatesUtil.getOriginalTweetId(candidate.candidate, candidate.features)\n        }\n      }\n\n    val responseMap =\n      if (callMiddleMan) getTwhinEmbeddingsFromHMF(originalTweetIds)\n      else getTwhinEmbeddingsFromReadableStore(originalTweetIds)\n\n    responseMap.map { response =>\n      response.map { twhinEmbeddingOpt =>\n        val floatTensor = {\n          keyFoundCounter.incr()\n          twhinEmbeddingOpt.map(twhinEmbedding => ml.FloatTensor(twhinEmbedding.embedding))\n        }\n        val dataRecord =\n          TwhinRebuildTweetEmbeddingsAdapter.adaptToDataRecords(floatTensor).asScala.head\n        FeatureMap(TwhinRebuildTweetFeature, dataRecord)\n      }\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val callMiddleMan = query.params(EnableHomeMixerFeaturesService)\n    OffloadFuturePools.offloadBatchSeqToFutureSeq(\n      candidates,\n      getBatchedFeatureMap(_, callMiddleMan),\n      batchSize)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TwhinRebuildUserEngagementQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinRebuildUserEngagementEmbeddingsAdapter\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinRebuildUserEngagementFeatureRepository\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinRebuildUserEngagementFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinRebuildUserEngagementQueryFeatureHydrator @Inject() (\n  @Named(TwhinRebuildUserEngagementFeatureRepository)\n  client: KeyValueRepository[Seq[Long], Long, ml.FloatTensor],\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TwhinRebuildUserEngagement\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinRebuildUserEngagementFeature)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  private val keyFailureCounter = scopedStatsReceiver.counter(\"key/failure\")\n  private val keyTotalCounter = scopedStatsReceiver.counter(\"key/total\")\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n    Stitch.callFuture(client(Seq(userId))).map { results =>\n      keyTotalCounter.incr()\n      val embedding: Option[ml.FloatTensor] = results(userId) match {\n        case Return(value) =>\n          if (value.exists(_.floats.nonEmpty)) keyFoundCounter.incr()\n          else keyNotFoundCounter.incr()\n          value\n        case Throw(_) =>\n          keyFailureCounter.incr()\n          None\n        case _ =>\n          None\n      }\n\n      val dataRecord =\n        TwhinRebuildUserEngagementEmbeddingsAdapter.adaptToDataRecords(embedding).asScala.head\n\n      FeatureMapBuilder()\n        .add(TwhinRebuildUserEngagementFeature, dataRecord)\n        .build()\n    }\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TwhinRebuildUserPositiveQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinRebuildUserPositiveEmbeddingsAdapter\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinRebuildUserPositiveEmbeddingsStore\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.simclusters_v2.thriftscala.TwhinTweetEmbedding\nimport com.twitter.simclusters_v2.thriftscala.TwhinEmbeddingDataset\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinRebuildUserPositiveFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinRebuildUserPositiveQueryFeatureHydrator @Inject() (\n  @Named(TwhinRebuildUserPositiveEmbeddingsStore) store: ReadableStore[\n    (Long, Long),\n    TwhinTweetEmbedding\n  ],\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TwhinRebuildUserPositive\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinRebuildUserPositiveFeature)\n\n  private val versionId = TwhinEmbeddingDataset.RefreshedTwhinTweet.value\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  private val keyTotalCounter = scopedStatsReceiver.counter(\"key/total\")\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    Stitch\n      .callFuture(store.get((query.getRequiredUserId, versionId))).map { resultOpt =>\n        keyTotalCounter.incr()\n        resultOpt match {\n          case Some(_) =>\n            keyFoundCounter.incr()\n          case None => keyNotFoundCounter.incr()\n        }\n        val floatTensor = resultOpt.map(result => ml.FloatTensor(result.embedding))\n        val dataRecord = TwhinRebuildUserPositiveEmbeddingsAdapter\n          .adaptToDataRecords(floatTensor).asScala.head\n        FeatureMapBuilder().add(TwhinRebuildUserPositiveFeature, dataRecord).build()\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TwhinTweetFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinTweetEmbeddingsAdapter\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinTweetEmbeddingsStore\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableHomeMixerFeaturesService\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer_features.{thriftscala => hmf}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.util.Future\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.simclusters_v2.thriftscala.TwhinTweetEmbedding\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinTweetFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinTweetFeatureHydrator @Inject() (\n  homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint,\n  @Named(TwhinTweetEmbeddingsStore) store: ReadableStore[Long, TwhinTweetEmbedding])\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TwhinTweet\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinTweetFeature)\n\n  private val batchSize = 50\n\n  private def getTwhinEmbeddingsFromHMF(\n    originalTweetIds: Seq[Long]\n  ): Future[Seq[Option[TwhinTweetEmbedding]]] = {\n    val keysSerialized = originalTweetIds.map(_.toString)\n    val request = hmf.HomeMixerFeaturesRequest(keysSerialized, hmf.Cache.Twhin)\n    val responseFut =\n      homeMixerFeatureService.getHomeMixerFeatures(request)\n    responseFut\n      .map { response =>\n        response.homeMixerFeatures\n          .map { homeMixerFeatureOpt =>\n            homeMixerFeatureOpt.homeMixerFeaturesType.map {\n              case hmf.HomeMixerFeaturesType.TwhinTweetEmbedding(homeMixerFeature) =>\n                homeMixerFeature\n              case _ => throw new Exception(\"Unknown type returned\")\n            }\n          }\n      }.handle { case _ => Seq.fill(originalTweetIds.size)(None) }\n  }\n\n  private def getTwhinEmbeddingsFromReadableStore(\n    originalTweetIds: Seq[Long]\n  ): Future[Seq[Option[TwhinTweetEmbedding]]] = {\n    Future.collect(store.multiGet(originalTweetIds.toSet)).map { storeResponse =>\n      originalTweetIds.map {\n        storeResponse.getOrElse(_, None)\n      }\n    }\n  }\n\n  private def getBatchedFeatureMap(\n    candidatesBatch: Seq[CandidateWithFeatures[TweetCandidate]],\n    callMiddleMan: Boolean\n  ): Future[Seq[FeatureMap]] = {\n    val originalTweetIds =\n      candidatesBatch.map { candidate =>\n        CandidatesUtil.getOriginalTweetId(candidate.candidate, candidate.features)\n      }\n\n    val responseMap =\n      if (callMiddleMan) getTwhinEmbeddingsFromHMF(originalTweetIds)\n      else getTwhinEmbeddingsFromReadableStore(originalTweetIds)\n\n    responseMap.map { response =>\n      response.map { twhinEmbeddingOpt =>\n        val floatTensor =\n          twhinEmbeddingOpt.map(twhinEmbedding => ml.FloatTensor(twhinEmbedding.embedding))\n        val dataRecord =\n          TwhinTweetEmbeddingsAdapter.adaptToDataRecords(floatTensor).asScala.head\n        FeatureMap(TwhinTweetFeature, dataRecord)\n      }\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val callMiddleMan = query.params(EnableHomeMixerFeaturesService)\n    OffloadFuturePools.offloadBatchSeqToFutureSeq(\n      candidates,\n      getBatchedFeatureMap(_, callMiddleMan),\n      batchSize)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TwhinUserEngagementQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinUserEngagementEmbeddingsAdapter\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinUserEngagementFeatureRepository\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinUserEngagementFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinUserEngagementQueryFeatureHydrator @Inject() (\n  @Named(TwhinUserEngagementFeatureRepository)\n  client: KeyValueRepository[Seq[Long], Long, ml.FloatTensor],\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TwhinUserEngagement\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinUserEngagementFeature)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  private val keyFailureCounter = scopedStatsReceiver.counter(\"key/failure\")\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n    Stitch.callFuture(client(Seq(userId))).map { results =>\n      val embedding: Option[ml.FloatTensor] = results(userId) match {\n        case Return(value) =>\n          if (value.exists(_.floats.nonEmpty)) keyFoundCounter.incr()\n          else keyNotFoundCounter.incr()\n          value\n        case Throw(_) =>\n          keyFailureCounter.incr()\n          None\n        case _ =>\n          None\n      }\n\n      val dataRecord =\n        TwhinUserEngagementEmbeddingsAdapter.adaptToDataRecords(embedding).asScala.head\n\n      FeatureMapBuilder()\n        .add(TwhinUserEngagementFeature, dataRecord)\n        .build()\n    }\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TwhinUserFollowQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinUserFollowEmbeddingsAdapter\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinUserFollowFeatureRepository\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinUserFollowFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinUserFollowQueryFeatureHydrator @Inject() (\n  @Named(TwhinUserFollowFeatureRepository)\n  client: KeyValueRepository[Seq[Long], Long, ml.FloatTensor],\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TwhinUserFollow\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinUserFollowFeature)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  private val keyFailureCounter = scopedStatsReceiver.counter(\"key/failure\")\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n    Stitch\n      .callFuture(client(Seq(userId)))\n      .map { results =>\n        val embedding: Option[ml.FloatTensor] = results(userId) match {\n          case Return(value) =>\n            if (value.exists(_.floats.nonEmpty)) keyFoundCounter.incr()\n            else keyNotFoundCounter.incr()\n            value\n          case Throw(_) =>\n            keyFailureCounter.incr()\n            None\n          case _ =>\n            None\n        }\n\n        val dataRecord = TwhinUserFollowEmbeddingsAdapter.adaptToDataRecords(embedding).asScala.head\n\n        FeatureMapBuilder()\n          .add(TwhinUserFollowFeature, dataRecord)\n          .build()\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TwhinUserNegativeFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinUserNegativeEmbeddingsAdapter\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinUserNegativeEmbeddingsStore\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.simclusters_v2.thriftscala.TwhinTweetEmbedding\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinUserNegativeFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinUserNegativeQueryFeatureHydrator @Inject() (\n  @Named(TwhinUserNegativeEmbeddingsStore) store: ReadableStore[Long, TwhinTweetEmbedding])\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TwhinUserNegative\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinUserNegativeFeature)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    Stitch.callFuture(store.get(query.getRequiredUserId)).map { resultOpt =>\n      val floatTensor = resultOpt.map(result => ml.FloatTensor(result.embedding))\n      val dataRecord = TwhinUserNegativeEmbeddingsAdapter\n        .adaptToDataRecords(floatTensor).asScala.head\n      FeatureMapBuilder().add(TwhinUserNegativeFeature, dataRecord).build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TwhinUserPositiveFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinUserPositiveEmbeddingsAdapter\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinUserPositiveEmbeddingsStore\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.simclusters_v2.thriftscala.TwhinTweetEmbedding\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinUserPositiveFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinUserPositiveQueryFeatureHydrator @Inject() (\n  @Named(TwhinUserPositiveEmbeddingsStore) store: ReadableStore[Long, TwhinTweetEmbedding])\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TwhinUserPositive\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinUserPositiveFeature)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    Stitch.callFuture(store.get(query.getRequiredUserId)).map { resultOpt =>\n      val floatTensor = resultOpt.map(result => ml.FloatTensor(result.embedding))\n      val dataRecord = TwhinUserPositiveEmbeddingsAdapter\n        .adaptToDataRecords(floatTensor).asScala.head\n      FeatureMapBuilder().add(TwhinUserPositiveFeature, dataRecord).build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/TwhinVideoFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinVideoEmbeddingsAdapter\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinVideoEmbeddingsStore\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.simclusters_v2.thriftscala.TwhinTweetEmbedding\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinVideoFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinVideoFeatureHydrator @Inject() (\n  @Named(TwhinVideoEmbeddingsStore) store: ReadableStore[Long, TwhinTweetEmbedding])\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TwhinVideo\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinVideoFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = OffloadFuturePools.offloadFuture {\n    val originalTweetId = CandidatesUtil.getOriginalTweetId(candidate, existingFeatures)\n\n    store.get(originalTweetId).map { resultOpt =>\n      val floatTensor = resultOpt.map(result => ml.FloatTensor(result.embedding))\n      val dataRecord = TwhinVideoEmbeddingsAdapter.adaptToDataRecords(floatTensor).asScala.head\n      FeatureMapBuilder().add(TwhinVideoFeature, dataRecord).build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UnifiedUserActionsUserIdentifierFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.unified_counter.service.UuaUserIdentifierClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UnifiedUserActionsUserIdentifierFeatureHydrator @Inject() (\n  uuaUserIdentifierClientColumn: UuaUserIdentifierClientColumn)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UnifiedUserActionsUserIdentifier\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    UuaUserGenderFeature,\n    UuaUserStateFeature,\n    UuaUserAgeBucketFeature\n  )\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(UuaUserGenderFeature, None)\n    .add(UuaUserStateFeature, None)\n    .add(UuaUserAgeBucketFeature, None)\n    .build()\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    uuaUserIdentifierClientColumn.fetcher\n      .fetch(query.getRequiredUserId.toString)\n      .map { response =>\n        response.v match {\n          case Some(userInfo) =>\n            val gender = userInfo.userGender.map(_.toString)\n            val state = userInfo.userState.map(_.value.toLong)\n            val ageBucket = userInfo.userAgeBucket.map(_.toString)\n\n            FeatureMapBuilder()\n              .add(UuaUserGenderFeature, gender)\n              .add(UuaUserStateFeature, state)\n              .add(UuaUserAgeBucketFeature, ageBucket)\n              .build()\n\n          case _ =>\n            DefaultFeatureMap\n        }\n      }\n      .rescue {\n        case _: Throwable =>\n          Stitch.value(DefaultFeatureMap)\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserActionByteArrayQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.UserActionsByteArrayFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.UserActionsMaxCount\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.user_history_transformer.user_actions.UserActionSequenceMhClientColumn\nimport com.twitter.user_history_transformer.domain.AggregationAlgorithmV1\nimport com.twitter.user_history_transformer.domain.AggregationConfig\nimport com.twitter.user_history_transformer.domain.AggregationProcessor\nimport com.twitter.user_history_transformer.domain.UserActionSequenceUtils\nimport com.twitter.user_history_transformer.util.SchemaUtils\nimport com.x.user_action_sequence.thriftscala.UserActionSequenceDataContainer.OrderedAggregatedUserActionList\nimport com.x.user_action_sequence.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserActionsArrayByteQueryFeatureHydrator @Inject() (\n  userActionSequenceMhClientColumn: UserActionSequenceMhClientColumn,\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserActionsByteArray\")\n\n  override val features: Set[Feature[_, _]] = Set(UserActionsByteArrayFeature)\n\n  private val DefaultFeatureMap = FeatureMap(UserActionsByteArrayFeature, None)\n\n  private val windowTimeMs = 5 * 60 * 1000\n\n  private val aggregationProcessor = new AggregationProcessor(\n    AggregationConfig(\n      postProcessorSeq = Seq.empty,\n      windowTimeMs = windowTimeMs,\n      maxLength = 1024,\n      aggregationAlgorithm = AggregationAlgorithmV1\n    )\n  )\n\n  private def hasNegativeValue(value: Option[Long]): Boolean = value.exists(_ < 0)\n\n  private def hasNegativeValues(aggregatedUserAction: t.AggregatedUserAction): Boolean = {\n    if (hasNegativeValue(aggregatedUserAction.userId)) return true\n\n    aggregatedUserAction.tweetInfo.exists { tweetInfo =>\n      val fieldsToCheck = List(\n        tweetInfo.tweetId,\n        tweetInfo.authorId,\n        tweetInfo.retweetingTweetId,\n        tweetInfo.quotingTweetId,\n        tweetInfo.replyingTweetId,\n        tweetInfo.quotedTweetId,\n        tweetInfo.inReplyToTweetId,\n        tweetInfo.retweetedTweetId,\n        tweetInfo.editedTweetId\n      )\n\n      fieldsToCheck.exists(hasNegativeValue)\n    }\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] =\n    OffloadFuturePools.offloadStitch {\n      userActionSequenceMhClientColumn.fetcher.fetch(query.getRequiredUserId).map { response =>\n        val featureMap = response.v.map { userActionsSeq =>\n          val decompressedUserActionSeq =\n            UserActionSequenceUtils.expand(userActionsSeq, statsReceiver)\n          val downsampledUserActionSeq =\n            UserActionSequenceUtils.mhToKafka(decompressedUserActionSeq)\n          val aggregatedUserActions = aggregationProcessor\n            .process(downsampledUserActionSeq)\n            .filterNot(hasNegativeValues)\n            .takeRight(query.params(UserActionsMaxCount))\n\n          val filteredUserActionSeq = userActionsSeq.copy(\n            userActionsData = Some(\n              OrderedAggregatedUserActionList(\n                t.AggregatedUserActionList(aggregatedUserActions = Some(aggregatedUserActions))\n              )\n            )\n          )\n\n          val actions = SchemaUtils.convertUserActionSequenceThriftToProtobuf(filteredUserActionSeq)\n\n          FeatureMap(UserActionsByteArrayFeature, Some(actions.toByteArray))\n        }\n\n        featureMap.getOrElse(DefaultFeatureMap)\n      }\n    }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserActionsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.UserActionsContainsExplicitSignalsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserActionsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserActionsSizeFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.UserActionsMaxCount\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableDenseUserActionsHydrationParam\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.user_history_transformer.user_actions.UserActionSequenceMhClientColumn\nimport com.twitter.user_history_transformer.domain.AggregationAlgorithmV1\nimport com.twitter.user_history_transformer.domain.AggregationAlgorithmWithoutHomeFilter\nimport com.twitter.user_history_transformer.domain.AggregationConfig\nimport com.twitter.user_history_transformer.domain.AggregationProcessor\nimport com.twitter.user_history_transformer.domain.UserActionSequenceUtils\nimport com.twitter.user_history_transformer.util.SchemaUtils\nimport com.x.user_action_sequence.thriftscala.ActionName.ClientTweetRecapDwelled\nimport com.x.user_action_sequence.thriftscala.ActionName.ClientTweetRecapNotDwelled\nimport com.x.user_action_sequence.thriftscala.ActionName\nimport com.x.user_action_sequence.thriftscala.AggregatedUserAction\nimport com.x.user_action_sequence.thriftscala.UserActionSequenceDataContainer.OrderedAggregatedUserActionList\nimport com.x.user_action_sequence.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserActionsQueryFeatureHydrator @Inject() (\n  userActionSequenceMhClientColumn: UserActionSequenceMhClientColumn,\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"UserActions\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    UserActionsFeature,\n    UserActionsSizeFeature,\n    UserActionsContainsExplicitSignalsFeature\n  )\n\n  private val userAggregatedActionSeqlengthStat =\n    statsReceiver.stat(\"UserActionsQueryFeatureHydrator\", \"length\")\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(UserActionsFeature, None)\n    .add(UserActionsSizeFeature, None)\n    .add(UserActionsContainsExplicitSignalsFeature, false)\n    .build()\n\n  private val windowTimeMs = 5 * 60 * 1000\n\n  private val ExcludedDwellActions: Set[ActionName] =\n    Set(ClientTweetRecapDwelled, ClientTweetRecapNotDwelled)\n\n  private def filterDwells(\n    aggAction: AggregatedUserAction\n  ): Boolean = {\n    aggAction.actions\n      .getOrElse(Seq.empty)\n      .exists {\n        _.actionName.exists { name =>\n          !ExcludedDwellActions.contains(name)\n        }\n      }\n  }\n\n  private val aggregationProcessor = new AggregationProcessor(\n    AggregationConfig(\n      postProcessorSeq = Seq.empty,\n      windowTimeMs = windowTimeMs,\n      maxLength = 1024,\n      aggregationAlgorithm = AggregationAlgorithmV1,\n    )\n  )\n\n  private val denseAggregationProcessor = new AggregationProcessor(\n    AggregationConfig(\n      postProcessorSeq = Seq.empty,\n      windowTimeMs = windowTimeMs,\n      maxLength = 1024,\n      aggregationAlgorithm = AggregationAlgorithmWithoutHomeFilter,\n      aggActionFilterFuncGenerator = (_, _) => {\n        aggAction => filterDwells(aggAction)\n      }\n    )\n  )\n\n  private def hasNegativeValue(value: Option[Long]): Boolean = value.exists(_ < 0)\n  private def hasNegativeValues(aggregatedUserAction: t.AggregatedUserAction): Boolean = {\n    if (hasNegativeValue(aggregatedUserAction.userId)) return true\n\n    aggregatedUserAction.tweetInfo.exists { tweetInfo =>\n      val fieldsToCheck = List(\n        tweetInfo.tweetId,\n        tweetInfo.authorId,\n        tweetInfo.retweetingTweetId,\n        tweetInfo.quotingTweetId,\n        tweetInfo.replyingTweetId,\n        tweetInfo.quotedTweetId,\n        tweetInfo.inReplyToTweetId,\n        tweetInfo.retweetedTweetId,\n        tweetInfo.editedTweetId\n      )\n\n      fieldsToCheck.exists(hasNegativeValue)\n    }\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val processor =\n      if (query.params(EnableDenseUserActionsHydrationParam)) denseAggregationProcessor\n      else aggregationProcessor\n    OffloadFuturePools.offloadStitch {\n      userActionSequenceMhClientColumn.fetcher.fetch(query.getRequiredUserId).map { response =>\n        val featureMap = response.v.map { userActionsSeq =>\n          val decompressedUserActionSeq =\n            UserActionSequenceUtils.expand(userActionsSeq, statsReceiver)\n          val aggregatedUserActions = processor\n            .process(decompressedUserActionSeq)\n            .filterNot(hasNegativeValues)\n            .takeRight(query.params(UserActionsMaxCount))\n\n          val size = aggregatedUserActions.length\n          userAggregatedActionSeqlengthStat.add(size)\n\n          val hasExplicitSignals =\n            UserActionSequenceUtils.hasExplicitSignals(decompressedUserActionSeq)\n\n          val filteredUserActionSeq = userActionsSeq.copy(\n            userActionsData = Some(\n              OrderedAggregatedUserActionList(\n                t.AggregatedUserActionList(aggregatedUserActions = Some(aggregatedUserActions))\n              )\n            )\n          )\n\n          val actions = SchemaUtils.convertUserActionSequenceThriftToProtobuf(filteredUserActionSeq)\n\n          FeatureMapBuilder()\n            .add(UserActionsFeature, Some(actions))\n            .add(UserActionsSizeFeature, Some(size))\n            .add(UserActionsContainsExplicitSignalsFeature, hasExplicitSignals)\n            .build()\n        }\n\n        featureMap.getOrElse(DefaultFeatureMap)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserEngagedGrokCategoriesFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.trends.trip.EngagedGrokTopicsAndTagsMonthlyOnUserClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject UserSubLevelCategoriesFeature extends Feature[TweetCandidate, Seq[(Long, Double)]]\n\n@Singleton\nclass UserEngagedGrokCategoriesFeatureHydrator @Inject() (\n  engagedGrokTopicsAndTagMonthlysOnUserClientColumn: EngagedGrokTopicsAndTagsMonthlyOnUserClientColumn)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserEngagedGrokCategories\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(UserSubLevelCategoriesFeature)\n\n  private def fetchSubLevelEntities(userId: Long): Stitch[Seq[(Long, Double)]] = {\n    engagedGrokTopicsAndTagMonthlysOnUserClientColumn.fetcher.fetch(userId).map { result =>\n      result.v\n        .flatMap(_.subLevelEntities).getOrElse(Seq.empty)\n        .take(3)\n        .map(entityInfo => (entityInfo.entityId, entityInfo.score))\n    }\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n    fetchSubLevelEntities(userId).map { subEntities =>\n      FeatureMapBuilder()\n        .add(UserSubLevelCategoriesFeature, subEntities)\n        .build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserEngagedLanguagesFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.UserEngagedLanguagesFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.language.user.NormalizedEngagedTweetLanguagesOnUserClientColumn\nimport com.twitter.language.types.{thriftscala => lg}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserEngagedLanguagesFeatureHydrator @Inject() (\n  engagedLanguageOnUserColumn: NormalizedEngagedTweetLanguagesOnUserClientColumn)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserEngagedLanguages\")\n\n  override def features: Set[Feature[_, _]] = Set(UserEngagedLanguagesFeature)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    engagedLanguageOnUserColumn.fetcher\n      .fetch(query.getRequiredUserId, Some(lg.LanguageType.User)).map { result =>\n        FeatureMapBuilder()\n          .add(UserEngagedLanguagesFeature, result.v.getOrElse(Seq.empty).toSet)\n          .build()\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserEngagementGrokTagFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.content_understanding.UserTopTagsMhClientColumn\nimport com.twitter.strato.generated.client.trends.trip.UserAssociatedTopicsClientColumn\nimport com.twitter.trends.trip_v1.user_topics.{thriftscala => ut}\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject RealtimeUserEngagedGrokTagFeature\n    extends Feature[TweetCandidate, Seq[(String, Option[Double])]]\nobject EvergreenUserEngagedGrokTagFeature\n    extends Feature[TweetCandidate, Seq[(String, Option[Double])]]\n\n@Singleton\nclass UserEngagementGrokTagFeatureHydrator @Inject() (\n  evergreenUserTopTags: UserTopTagsMhClientColumn,\n  tripUserTopTags: UserAssociatedTopicsClientColumn)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"UserEngagementGrokTag\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(RealtimeUserEngagedGrokTagFeature, EvergreenUserEngagedGrokTagFeature)\n\n  def fetchRealTimeUserTags(userId: Long): Stitch[Seq[(String, Option[Double])]] = {\n    tripUserTopTags.fetcher.fetch(ut.UserTopicDomain(userId = userId, sourceId = None)).map { result =>\n      result.v.flatMap(_.tags).getOrElse(Seq.empty[ut.TagCandidate]).map(tc => (tc.tag, tc.score))\n    }\n  }\n\n  def fetchEvergreenUserTags(userId: Long): Stitch[Seq[(String, Option[Double])]] = {\n    evergreenUserTopTags.fetcher.fetch(userId).map { view =>\n      view.v.map(_.tags.map(tc => (tc.tag, Some(tc.score)))).getOrElse(Seq.empty)\n    }\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n    Stitch.join(fetchRealTimeUserTags(userId), fetchEvergreenUserTags(userId)).map {\n      case (realtime, evergreen) =>\n        FeatureMapBuilder()\n          .add(RealtimeUserEngagedGrokTagFeature, realtime)\n          .add(EvergreenUserEngagedGrokTagFeature, evergreen)\n          .build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserFrequentLocationHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.geoduck.common.thriftscala.TransactionLocation\nimport com.twitter.geoduck.common.{thriftscala => t}\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.product_mixer.component_library.feature.location.{\n  Location => ProductMixerLocation\n}\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.ExpiringLruInProcessCache\nimport com.twitter.servo.cache.InProcessCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.geo.service.UserLocationClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject UserFrequentLocationHydrator {\n  private val BaseTTLMinutes = 60 * 24\n  private val TTL = (BaseTTLMinutes + scala.util.Random.nextInt(60)).minutes\n\n  val cache: InProcessCache[Long, Option[TransactionLocation]] =\n    new ExpiringLruInProcessCache[Long, Option[TransactionLocation]](\n      ttl = TTL,\n      maximumSize = 150 * 1000 // Cache up to 150k users\n    )\n}\n\n@Singleton\nclass UserFrequentLocationHydrator @Inject() (\n  userLocationClientColumn: UserLocationClientColumn)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"UserFrequentLocationHydrator\")\n\n  override val features: Set[Feature[_, _]] = Set(LocationFeature)\n\n  private val PlaceQuery = t.PlaceQuery(\n    placeTypes = Some(\n      Set(\n        t.PlaceType.Neighborhood,\n        t.PlaceType.City,\n        t.PlaceType.Metro,\n        t.PlaceType.Admin1,\n        t.PlaceType.Country\n      )\n    )\n  )\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    val authorIds = candidates.flatMap(_.features.getOrElse(AuthorIdFeature, None)).distinct\n\n    val (hitIds, missIds) =\n      authorIds.partition(id => UserFrequentLocationHydrator.cache.get(id).isDefined)\n\n    val cachedOptionMap: Map[Long, Option[TransactionLocation]] =\n      hitIds.map(id => id -> UserFrequentLocationHydrator.cache.get(id).get).toMap\n\n    val fetchCacheMisses: Stitch[Map[Long, TransactionLocation]] =\n      if (missIds.isEmpty) {\n        Stitch.value(Map.empty[Long, TransactionLocation])\n      } else {\n        userLocationClientColumn.fetcher\n          .fetch(\n            key = Unit,\n            t.UserLocationRequest(\n              userIds = missIds,\n              placeQuery = Some(PlaceQuery)\n            )\n          )\n          .map { response =>\n            response.v.toList.flatMap(_._1).toMap\n          }\n          .handle {\n            case e =>\n              Map.empty[Long, TransactionLocation]\n          }\n      }\n\n    val allLocationsStitch: Stitch[Map[Long, TransactionLocation]] =\n      fetchCacheMisses.map { fetchedMap =>\n        missIds.foreach { id =>\n          val locOpt: Option[TransactionLocation] = fetchedMap.get(id)\n          UserFrequentLocationHydrator.cache.set(id, locOpt)\n        }\n\n        val cachedLocs: Map[Long, TransactionLocation] =\n          cachedOptionMap.collect { case (id, Some(loc)) => id -> loc }\n\n        cachedLocs ++ fetchedMap\n      }\n\n    allLocationsStitch.map { allLocations =>\n      candidates.map { candidate =>\n        val locOpt = for {\n          authorId <- candidate.features.getOrElse(AuthorIdFeature, None)\n          loc <- allLocations.get(authorId)\n        } yield loc\n\n        locOpt\n          .map { transactionLocation =>\n            val placeMap = transactionLocation.placeMap\n            val locationDetails = ProductMixerLocation(\n              neighborhood = placeMap\n                .flatMap(_.get(t.PlaceType.Neighborhood))\n                .flatMap(_.headOption),\n              city = placeMap\n                .flatMap(_.get(t.PlaceType.City))\n                .flatMap(_.headOption),\n              metro = placeMap\n                .flatMap(_.get(t.PlaceType.Metro))\n                .flatMap(_.headOption),\n              region = placeMap\n                .flatMap(_.get(t.PlaceType.Admin1))\n                .flatMap(_.headOption),\n              country = placeMap\n                .flatMap(_.get(t.PlaceType.Country))\n                .flatMap(_.headOption)\n            )\n            FeatureMapBuilder()\n              .add(LocationFeature, Option(locationDetails))\n              .build()\n          }\n          .getOrElse {\n            FeatureMapBuilder()\n              .add(LocationFeature, None)\n              .build()\n          }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserFrequentLocationQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.product_mixer.component_library.feature.location.{Location => ProductLocation}\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.geoduck.common.{thriftscala => t}\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.geo.service.UserLocationClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserFrequentLocationQueryFeatureHydrator @Inject() (\n  userLocationClientColumn: UserLocationClientColumn)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserFrequentLocationQuery\")\n\n  override val features: Set[Feature[_, _]] = Set(LocationFeature)\n\n  private val PlaceQuery = t.PlaceQuery(\n    placeTypes = Some(\n      Set(\n        t.PlaceType.Neighborhood,\n        t.PlaceType.City,\n        t.PlaceType.Metro,\n        t.PlaceType.Admin1,\n        t.PlaceType.Country\n      )\n    )\n  )\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(LocationFeature, None)\n    .build()\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n    val locationStitch = userLocationClientColumn.fetcher\n      .fetch(\n        key = Unit,\n        t.UserLocationRequest(\n          userIds = Seq(userId),\n          placeQuery = Some(PlaceQuery)\n        )\n      )\n      .map { response =>\n        response.v.toList.flatMap(_._1).toMap.get(userId)\n      }\n    locationStitch\n      .map { locationOpt =>\n        locationOpt\n          .map { location =>\n            val placeMap = location.placeMap\n            val locationDetails = ProductLocation(\n              neighborhood = placeMap\n                .flatMap(_.get(t.PlaceType.Neighborhood))\n                .flatMap(_.headOption),\n              city = placeMap\n                .flatMap(_.get(t.PlaceType.City))\n                .flatMap(_.headOption),\n              metro = placeMap\n                .flatMap(_.get(t.PlaceType.Metro))\n                .flatMap(_.headOption),\n              region = placeMap\n                .flatMap(_.get(t.PlaceType.Admin1))\n                .flatMap(_.headOption),\n              country = placeMap\n                .flatMap(_.get(t.PlaceType.Country))\n                .flatMap(_.headOption)\n            )\n            FeatureMapBuilder()\n              .add(LocationFeature, Option(locationDetails))\n              .build()\n          }\n          .getOrElse {\n            DefaultFeatureMap\n          }\n      }.handle {\n        case e =>\n          DefaultFeatureMap\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserHistoryTransformerEmbeddingQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.TransformerByteEmbeddingsAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.TransformerEmbeddingsAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.UserHistoryTransformerEmbeddingsHomeBlueAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.UserHistoryTransformerEmbeddingsHomeGreenAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.UserHistoryTransformerEmbeddingsJointBlueAdapter\nimport com.twitter.home_mixer_features.thriftscala.HomeMixerFeaturesType\nimport com.twitter.home_mixer_features.{thriftscala => hmf}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport java.nio.ByteBuffer\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject UserHistoryTransformerEmbeddingHomeBlueFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\nobject UserHistoryTransformerEmbeddingHomeGreenFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\nobject UserHistoryTransformerEmbeddingJointBlueFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass UserHistoryTransformerEmbeddingQueryFeatureHydratorBuilder @Inject() (\n  homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint,\n  statsReceiver: StatsReceiver) {\n  def buildHomeBlueHydrator(): UserHistoryTransformerFloatEmbeddingQueryFeatureHydrator = {\n    UserHistoryTransformerFloatEmbeddingQueryFeatureHydrator(\n      FeatureHydratorIdentifier(\"HomeBlueTransformerEmbeddingQueryFeatureHydrator\"),\n      homeMixerFeatureService,\n      statsReceiver,\n      UserHistoryTransformerEmbeddingHomeBlueFeature,\n      hmf.Cache.TransformerUserEmbeddings,\n      UserHistoryTransformerEmbeddingsHomeBlueAdapter\n    )\n  }\n\n  def buildHomeGreenHydrator(): UserHistoryTransformerByteEmbeddingQueryFeatureHydrator = {\n    UserHistoryTransformerByteEmbeddingQueryFeatureHydrator(\n      FeatureHydratorIdentifier(\"HomeGreenTransformerEmbeddingQueryFeatureHydrator\"),\n      homeMixerFeatureService,\n      statsReceiver,\n      UserHistoryTransformerEmbeddingHomeGreenFeature,\n      hmf.Cache.TransformerUserEmbeddingsGreen,\n      UserHistoryTransformerEmbeddingsHomeGreenAdapter\n    )\n  }\n\n  def buildJointBlueHydrator(): UserHistoryTransformerByteEmbeddingQueryFeatureHydrator = {\n    UserHistoryTransformerByteEmbeddingQueryFeatureHydrator(\n      FeatureHydratorIdentifier(\"JointBlueTransformerEmbeddingQueryFeatureHydrator\"),\n      homeMixerFeatureService,\n      statsReceiver,\n      UserHistoryTransformerEmbeddingJointBlueFeature,\n      hmf.Cache.TransformerUserJointEmbeddingsBlue,\n      UserHistoryTransformerEmbeddingsJointBlueAdapter\n    )\n  }\n}\n\ncase class UserHistoryTransformerFloatEmbeddingQueryFeatureHydrator(\n  override val identifier: FeatureHydratorIdentifier,\n  override val homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint,\n  override val statsReceiver: StatsReceiver,\n  override val embeddingFeature: DataRecordInAFeature[PipelineQuery],\n  override val cacheType: hmf.Cache,\n  transformerEmbeddingsAdapter: TransformerEmbeddingsAdapter)\n    extends BaseUserHistoryTransformerEmbeddingQueryFeatureHydrator {\n\n  override def responseToDataRecord(homeMixerFeaturesType: HomeMixerFeaturesType): DataRecord = {\n    val embedding = homeMixerFeaturesType match {\n      case hmf.HomeMixerFeaturesType.RawEmbedding(floatEmbedding) =>\n        floatEmbedding\n      case other =>\n        wrongTypeCounter.incr()\n        throw new Exception(\n          f\"Type not matching. Expected RawEmbedding but got ${other.getClass.getSimpleName}\")\n    }\n    val tensor = ml.FloatTensor(embedding)\n    transformerEmbeddingsAdapter.adaptToDataRecords(Some(tensor)).asScala.head\n  }\n}\n\ncase class UserHistoryTransformerByteEmbeddingQueryFeatureHydrator(\n  override val identifier: FeatureHydratorIdentifier,\n  override val homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint,\n  override val statsReceiver: StatsReceiver,\n  override val embeddingFeature: DataRecordInAFeature[PipelineQuery],\n  override val cacheType: hmf.Cache,\n  transformerByteEmbeddingsAdapter: TransformerByteEmbeddingsAdapter)\n    extends BaseUserHistoryTransformerEmbeddingQueryFeatureHydrator {\n\n  override def responseToDataRecord(homeMixerFeaturesType: HomeMixerFeaturesType): DataRecord = {\n    val embedding = homeMixerFeaturesType match {\n      case hmf.HomeMixerFeaturesType.RawByteEmbedding(byteEmbedding) =>\n        byteEmbedding\n      case other =>\n        wrongTypeCounter.incr()\n        throw new Exception(\n          f\"Type not matching. Expected RawByteEmbedding but got ${other.getClass.getSimpleName}\")\n    }\n    val tensor = ml.RawTypedTensor(ml.DataType.Byte, convertToByteBuffer(embedding))\n    transformerByteEmbeddingsAdapter.adaptToDataRecords(Some(tensor)).asScala.head\n  }\n\n  private def convertToByteBuffer(byteArray: Seq[Byte]): ByteBuffer = {\n    val buffer = ByteBuffer.allocate(byteArray.size)\n    byteArray.foreach(buffer.put)\n    buffer.flip\n    buffer\n  }\n}\n\ntrait BaseUserHistoryTransformerEmbeddingQueryFeatureHydrator\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  def homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint\n  def statsReceiver: StatsReceiver\n  def embeddingFeature: DataRecordInAFeature[PipelineQuery]\n\n  def cacheType: hmf.Cache\n\n  override val features: Set[Feature[_, _]] = Set(embeddingFeature)\n\n  protected val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  protected val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  protected val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  protected val wrongTypeCounter = scopedStatsReceiver.counter(\"key/wrongType\")\n\n  def responseToDataRecord(homeMixerFeaturesType: HomeMixerFeaturesType): DataRecord\n\n  private def getFeatureMap(\n    pipelineQuery: PipelineQuery\n  ): Future[FeatureMap] = {\n    val keysSerialized = Seq(pipelineQuery.getRequiredUserId.toString)\n    val request = hmf.HomeMixerFeaturesRequest(keysSerialized, cacheType)\n    val responseFut =\n      homeMixerFeatureService.getHomeMixerFeatures(request)\n    responseFut\n      .map { response =>\n        response.homeMixerFeatures.headOption.flatMap { homeMixerFeatureOpt =>\n          homeMixerFeatureOpt.homeMixerFeaturesType match {\n            case Some(homeMixerFeaturesType) =>\n              keyFoundCounter.incr()\n              Some(responseToDataRecord(homeMixerFeaturesType))\n            case None =>\n              keyNotFoundCounter.incr()\n              None\n          }\n        }\n      }.handle { case _ => None }\n      .map {\n        case Some(dataRecord) =>\n          FeatureMap(embeddingFeature, dataRecord)\n        case None =>\n          FeatureMap(embeddingFeature, new DataRecord())\n      }\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    Stitch.callFuture(getFeatureMap(query))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserLanguagesFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UserLanguagesRepository\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.search.common.constants.{thriftscala => scc}\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject UserLanguagesFeature extends Feature[PipelineQuery, Seq[scc.ThriftLanguage]]\n\n@Singleton\ncase class UserLanguagesFeatureHydrator @Inject() (\n  @Named(UserLanguagesRepository) client: KeyValueRepository[Seq[Long], Long, Seq[\n    scc.ThriftLanguage\n  ]],\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"UserLanguages\")\n\n  override val features: Set[Feature[_, _]] = Set(UserLanguagesFeature)\n\n  override val statScope: String = identifier.toString\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val key = query.getRequiredUserId\n    Stitch.callFuture(client(Seq(key))).map { result =>\n      val feature =\n        observedGet(key = Some(key), keyValueResult = result).map(_.getOrElse(Seq.empty))\n      FeatureMapBuilder()\n        .add(UserLanguagesFeature, feature)\n        .build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserLargeEmbeddingsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeLargeEmbeddingsFeatures.UserLargeEmbeddingsFeature\nimport com.twitter.home_mixer.model.HomeLargeEmbeddingsFeatures.UserLargeEmbeddingsKeyFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableLargeEmbeddingsFeatureHydrationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelNameParam\nimport com.twitter.home_mixer_features.{thriftscala => hmf}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.large_embeddings.HashingFeatureParams\nimport com.twitter.timelines.prediction.adapters.large_embeddings.HomeMixerLargeEmbeddingsFeatureHydrator\nimport com.twitter.timelines.prediction.adapters.large_embeddings.LargeEmbeddingsAdapter\nimport com.twitter.timelines.prediction.adapters.large_embeddings.UserLargeEmbeddingsAdapter\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserLargeEmbeddingsFeatureHydrator @Inject() (\n  statsReceiver: StatsReceiver,\n  override val homeMixerFeatureService: hmf.HomeMixerFeatures.MethodPerEndpoint)\n    extends QueryFeatureHydrator[PipelineQuery]\n    with Conditionally[PipelineQuery]\n    with HomeMixerLargeEmbeddingsFeatureHydrator {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserLargeEmbeddings\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(UserLargeEmbeddingsFeature, UserLargeEmbeddingsKeyFeature)\n\n  override val adapter: LargeEmbeddingsAdapter = UserLargeEmbeddingsAdapter\n\n  override val cacheType: hmf.Cache = hmf.Cache.UserLargeEmbeddings\n\n  override val scopedStatsReceiver: StatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableLargeEmbeddingsFeatureHydrationParam)\n\n  // Hashing Features\n  override val defaultHashingFeatureParams: HashingFeatureParams = HashingFeatureParams(\n    scales = Seq(1681734645L, 1546314972L),\n    biases = Seq(2701200313L, 1873259806L),\n    modulus = 3055559939L,\n    bucketSize = 10000000L,\n  )\n\n  override val modelName2HashingFeatureParams: Map[String, HashingFeatureParams] = Map(\n    \"hr_video_prod__v3_realtime\" -> HashingFeatureParams(\n      scales = Seq(1341131000L, 519927459L),\n      biases = Seq(2924993425L, 294422133L),\n      modulus = 3109207999L,\n      bucketSize = 10000000L,\n    ),\n    \"hr_video_prod__v2_lembeds\" -> HashingFeatureParams(\n      scales = Seq(214226227L, 561611689L),\n      biases = Seq(182790211L, 330327483L),\n      modulus = 816016163L,\n      bucketSize = 1000000L,\n    ),\n    \"hr_prod__v4_embeds_230M\" -> HashingFeatureParams(\n      scales = Seq(196742702L, 1852108266L),\n      biases = Seq(1935840681L, 167407236L),\n      modulus = 2859568897L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v5_embeds_230M_and_transformer\" -> HashingFeatureParams(\n      scales = Seq(196742702L, 1852108266L),\n      biases = Seq(1935840681L, 167407236L),\n      modulus = 2859568897L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v5_watchtime\" -> HashingFeatureParams(\n      scales = Seq(45230244L, 676046872L),\n      biases = Seq(866394657L, 1019127517L),\n      modulus = 1047809363L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v6_transformer_v2\" -> HashingFeatureParams(\n      scales = Seq(196742702L, 1852108266L),\n      biases = Seq(1935840681L, 167407236L),\n      modulus = 2859568897L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v6_mixed_training\" -> HashingFeatureParams(\n      scales = Seq(196742702L, 1852108266L),\n      biases = Seq(1935840681L, 167407236L),\n      modulus = 2859568897L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v6_transformer_v2_kafka_merge_join\" -> HashingFeatureParams(\n      scales = Seq(196742702L, 1852108266L),\n      biases = Seq(1935840681L, 167407236L),\n      modulus = 2859568897L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_prod__v6_transformer_v2_realtime_debias_21apr\" -> HashingFeatureParams(\n      scales = Seq(196742702L, 1852108266L),\n      biases = Seq(1935840681L, 167407236L),\n      modulus = 2859568897L,\n      bucketSize = 100000000L,\n    ),\n    \"hr_video_prod__v4_realtime\" -> HashingFeatureParams(\n      scales = Seq(1341131000L, 519927459L),\n      biases = Seq(2924993425L, 294422133L),\n      modulus = 3109207999L,\n      bucketSize = 10000000L,\n    ),\n    \"hr_video_prod__v4_realtime_mergehead\" -> HashingFeatureParams(\n      scales = Seq(1341131000L, 519927459L),\n      biases = Seq(2924993425L, 294422133L),\n      modulus = 3109207999L,\n      bucketSize = 10000000L,\n    ),\n  )\n\n  private def getFeatureMap(\n    pipelineQuery: PipelineQuery\n  ): Future[FeatureMap] = {\n    val userId = pipelineQuery.getRequiredUserId\n    val modelName = pipelineQuery.params(ModelNameParam)\n    val responseMap = getLargeEmbeddings(userId, modelName)\n    responseMap.map { response =>\n      FeatureMapBuilder()\n        .add(UserLargeEmbeddingsFeature, response.dataRecord)\n        .add(UserLargeEmbeddingsKeyFeature, response.hashedKeys)\n        .build()\n    }\n  }\n\n  override def hydrate(\n    query: PipelineQuery\n  ): Stitch[FeatureMap] = Stitch.callFuture(getFeatureMap(query))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserStateQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.UserStateFeature\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.user_health.v1.{thriftscala => uhv1}\nimport com.twitter.timelines.user_health.{thriftscala => uh}\nimport com.twitter.user_session_store.ReadOnlyUserSessionStore\nimport com.twitter.user_session_store.ReadRequest\nimport com.twitter.user_session_store.UserSessionDataset\nimport com.twitter.user_session_store.UserSessionDataset.UserSessionDataset\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class UserStateQueryFeatureHydrator @Inject() (\n  userSessionStore: ReadOnlyUserSessionStore)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"UserState\")\n\n  override val features: Set[Feature[_, _]] = Set(UserStateFeature)\n\n  private val datasets: Set[UserSessionDataset] = Set(UserSessionDataset.UserHealth)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    userSessionStore\n      .read(ReadRequest(query.getRequiredUserId, datasets))\n      .map { userSession =>\n        val userState = userSession.flatMap {\n          _.userHealth match {\n            case Some(uh.UserHealth.V1(uhv1.UserHealth(userState))) => userState\n            case _ => None\n          }\n        }\n\n        FeatureMapBuilder()\n          .add(UserStateFeature, userState)\n          .build()\n      }\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserSubscriptionQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.subscription_services.subscription_verification.HasNoAdsBenefitOnUserClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject NoAdsTierFeature extends FeatureWithDefaultOnFailure[PipelineQuery, Boolean] {\n  override val defaultValue: Boolean = false\n}\n\n@Singleton\ncase class UserSubscriptionQueryFeatureHydrator @Inject() (\n  hasNoAdsBenefitOnUserClientColumn: HasNoAdsBenefitOnUserClientColumn)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"UserSubscription\")\n\n  override val features: Set[Feature[_, _]] = Set(NoAdsTierFeature)\n\n  override def hydrate(\n    query: PipelineQuery\n  ): Stitch[FeatureMap] = hasNoAdsBenefitOnUserClientColumn.fetcher\n    .fetch(query.getRequiredUserId)\n    .map { result =>\n      FeatureMapBuilder()\n        .add(NoAdsTierFeature, result.v.getOrElse(false))\n        .build()\n    }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UserUnderstandableLangaugesFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.UserUnderstandableLanguagesFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableGrokAutoTranslateLanguageFilter\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.language.user.GrokAutoTranslateUnderstandableLanguagesOnUserClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserUnderstandableLanguagesFeatureHydrator @Inject() (\n  understandableLanguagesClientColumn: GrokAutoTranslateUnderstandableLanguagesOnUserClientColumn,\n  override val statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery]\n    with Conditionally[PipelineQuery]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserUnderstandableLanguages\")\n\n  override val features: Set[Feature[_, _]] = Set(UserUnderstandableLanguagesFeature)\n\n  override val statScope: String = identifier.toString\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val successCounter = scopedStatsReceiver.counter(\"success\")\n  private val failedCounter = scopedStatsReceiver.counter(\"failure\")\n\n  private val DefaultFeatureMap = FeatureMap(UserUnderstandableLanguagesFeature, Seq.empty)\n\n  private val fetcher: Fetcher[Long, Unit, Seq[String]] =\n    understandableLanguagesClientColumn.fetcher\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableGrokAutoTranslateLanguageFilter)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val key = query.getRequiredUserId\n    fetcher\n      .fetch(key, ()).map { result =>\n        successCounter.incr()\n        FeatureMap(UserUnderstandableLanguagesFeature, result.v.getOrElse(Seq.empty))\n      }.rescue {\n        case _ =>\n          failedCounter.incr()\n          Stitch.value(DefaultFeatureMap)\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/UtegFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.FavoritedByCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RepliedByCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RepliedByEngagerIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RetweetedByCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RetweetedByEngagerIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UtegSocialProofRepository\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.recos.recos_common.{thriftscala => rc}\nimport com.twitter.recos.user_tweet_entity_graph.{thriftscala => uteg}\nimport com.twitter.servo.keyvalue.KeyValueResult\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass UtegFeatureHydrator @Inject() (\n  @Named(UtegSocialProofRepository) client: KeyValueRepository[\n    (Seq[Long], (Long, Map[Long, Double])),\n    Long,\n    uteg.TweetRecommendation\n  ]) extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"Uteg\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    FavoritedByUserIdsFeature,\n    RetweetedByEngagerIdsFeature,\n    RepliedByEngagerIdsFeature,\n    FavoritedByCountFeature,\n    RetweetedByCountFeature,\n    RepliedByCountFeature\n  )\n\n  override def onlyIf(query: PipelineQuery): Boolean = query.features\n    .exists(_.getOrElse(RealGraphInNetworkScoresFeature, Map.empty[Long, Double]).nonEmpty)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val seedUserWeights = query.features.map(_.get(RealGraphInNetworkScoresFeature)).get\n\n    val sourceTweetIds = candidates.flatMap(_.features.getOrElse(SourceTweetIdFeature, None))\n    val inReplyToTweetIds = candidates.flatMap(_.features.getOrElse(InReplyToTweetIdFeature, None))\n    val tweetIds = candidates.map(_.candidate.id)\n    val tweetIdsToSend = (tweetIds ++ sourceTweetIds ++ inReplyToTweetIds).distinct\n\n    val utegQuery = (tweetIdsToSend, (query.getRequiredUserId, seedUserWeights))\n\n    client(utegQuery).map(handleResponse(candidates, _))\n  }\n\n  private def handleResponse(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n    results: KeyValueResult[Long, uteg.TweetRecommendation],\n  ): Seq[FeatureMap] = {\n    candidates.map { candidate =>\n      val inNetwork = candidate.features.getOrElse(FromInNetworkSourceFeature, false)\n      val candidateProof = results(candidate.candidate.id).toOption.flatten\n      val sourceProof = candidate.features\n        .getOrElse(SourceTweetIdFeature, None).flatMap(results(_).toOption.flatten)\n      val proofs = Seq(candidateProof, sourceProof).flatten.map(_.socialProofByType)\n\n      val favoritedBy = proofs.flatMap(_.get(rc.SocialProofType.Favorite)).flatten\n      val retweetedBy = proofs.flatMap(_.get(rc.SocialProofType.Retweet)).flatten\n      val repliedBy = proofs.flatMap(_.get(rc.SocialProofType.Reply)).flatten\n\n      val (favoritedByCount, retweetedByCount, repliedByCount) =\n        if (!inNetwork) {\n          (favoritedBy.size.toDouble, retweetedBy.size.toDouble, repliedBy.size.toDouble)\n        } else { (0.0, 0.0, 0.0) }\n\n      FeatureMapBuilder(sizeHint = 6)\n        .add(FavoritedByUserIdsFeature, favoritedBy)\n        .add(RetweetedByEngagerIdsFeature, retweetedBy)\n        .add(RepliedByEngagerIdsFeature, repliedBy)\n        .add(FavoritedByCountFeature, favoritedByCount)\n        .add(RetweetedByCountFeature, retweetedByCount)\n        .add(RepliedByCountFeature, repliedByCount)\n        .build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/VideoSummaryEmbeddingFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.content.VideoSummaryEmbeddingFeaturesAdaptor\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaIdsFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableVideoSummaryEmbeddingFeatureDeciderParam\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.VideoEmbeddingMHStore\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport com.twitter.media_understanding.video_summary.thriftscala.VideoEmbedding\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.util.logging.Logging\nimport scala.collection.JavaConverters._\n\nobject VideoSummaryEmbeddingFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass VideoSummaryEmbeddingFeatureHydrator @Inject() (\n  @Named(VideoEmbeddingMHStore) videoEmbeddingStore: ReadableStore[Long, VideoEmbedding],\n  statsReceiver: StatsReceiver)\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery]\n    with Logging {\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"VideoSummaryEmbedding\")\n\n  override val features: Set[Feature[_, _]] = Set(VideoSummaryEmbeddingFeature)\n\n  private val DefaultFeatureMap =\n    FeatureMapBuilder().add(VideoSummaryEmbeddingFeature, new DataRecord()).build()\n\n  private val scopedCounter = statsReceiver.scope(\"VideoSummaryEmbeddingHydration\")\n  private val nonEmptyCounter = scopedCounter.counter(\"nonEmpty\")\n  private val emptyCounter = scopedCounter.counter(\"empty\")\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableVideoSummaryEmbeddingFeatureDeciderParam)\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    getFirstMediaId(existingFeatures).fold(Stitch.value(DefaultFeatureMap)) { mediaId =>\n      Stitch\n        .callFuture(videoEmbeddingStore.get(mediaId)).map { resultOpt =>\n          resultOpt match {\n            case Some(embedding) =>\n              val dataRecord = VideoSummaryEmbeddingFeaturesAdaptor\n                .adaptToDataRecords(embedding.embedding).asScala.head\n              nonEmptyCounter.incr()\n              FeatureMapBuilder().add(VideoSummaryEmbeddingFeature, dataRecord).build()\n            case None =>\n              emptyCounter.incr()\n              DefaultFeatureMap\n          }\n        }.onFailure { e =>\n          error(s\"Error fetching VideoSummaryEmbedding: $e\")\n          Stitch.value(DefaultFeatureMap)\n        }\n    }\n  }\n\n  private def getFirstMediaId(featureMap: FeatureMap): Option[Long] =\n    featureMap.getOrElse(TweetMediaIdsFeature, Seq.empty[Long]).headOption\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/ViewCountsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.ViewCountFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.Fetch\nimport com.twitter.strato.generated.client.viewcounts.ViewCountOnTweetClientColumn\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ViewCountsFeatureHydrator @Inject() (\n  viewCountsColumn: ViewCountOnTweetClientColumn,\n  statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"ViewCounts\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(ViewCountFeature)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  private val keyFailureCounter = scopedStatsReceiver.counter(\"key/failure\")\n\n  private val DefaultFeatureMap = FeatureMap(ViewCountFeature, None)\n  private val batchSize = 64\n\n  def getFeatureMaps(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n  ): Future[Seq[FeatureMap]] = {\n    val featureMapStitch = Stitch.traverse(candidates) { candidate =>\n      viewCountsColumn.fetcher\n        .fetch(candidate.candidate.id, Unit)\n        .map {\n          case Fetch.Result(response, _) =>\n            if (response.nonEmpty) keyFoundCounter.incr()\n            else keyNotFoundCounter.incr()\n            FeatureMap(ViewCountFeature, response)\n          case _ =>\n            keyFailureCounter.incr()\n            DefaultFeatureMap\n        }\n    }\n    Stitch.run(featureMapStitch)\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    OffloadFuturePools.offloadBatchSeqToFutureSeq(\n      candidates,\n      getFeatureMaps,\n      batchSize,\n      offload = true)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/ViralContentCreatorMetricsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ViralContentCreatorFeature\nimport com.twitter.home_mixer.module.ViralContentCreatorsConfig\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n *  Track metrics on how often we serve posts from viral content creators\n */\n@Singleton\ncase class ViralContentCreatorMetricsFeatureHydrator @Inject() (\n  viralContentCreatorsConfig: ViralContentCreatorsConfig)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"ViralContentCreatorMetrics\")\n\n  override val features: Set[Feature[_, _]] = Set(ViralContentCreatorFeature)\n\n  private val ViralContentCreators = viralContentCreatorsConfig.creators\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offload {\n    candidates.map { candidate =>\n      val authorIdOpt = candidate.features.getOrElse(AuthorIdFeature, None)\n      FeatureMap(ViralContentCreatorFeature, authorIdOpt.exists(ViralContentCreators.contains))\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/WithDefaultFeatureMap.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\n\ntrait WithDefaultFeatureMap {\n  // Make sure that default Feature Map has same features as defined in the feature hydrator\n  val defaultFeatureMap: FeatureMap\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/author_features/AuthorFeaturesAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.author_features\n\nimport com.twitter.home_mixer.util.DataRecordUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.util.CompactDataRecordConverter\nimport com.twitter.ml.api.util.FDsl._\nimport com.twitter.timelines.author_features.v1.{thriftjava => af}\nimport com.twitter.timelines.prediction.common.adapters.TimelinesAdapterBase\nimport com.twitter.timelines.prediction.common.aggregates.TimelinesAggregationConfig\nimport com.twitter.timelines.prediction.features.user_health.UserHealthFeatures\nimport scala.collection.JavaConverters._\n\nobject AuthorFeaturesAdapter extends TimelinesAdapterBase[af.AuthorFeatures] {\n\n  private val Prefix = \"original_author.timelines.original_author_aggregates.\"\n\n  private val typedAggregateGroups =\n    TimelinesAggregationConfig.originalAuthorAggregatesV1.buildTypedAggregateGroups()\n\n  private val aggregateFeaturesRenameMap: Map[Feature[_], Feature[_]] =\n    typedAggregateGroups.map(_.outputFeaturesToRenamedOutputFeatures(Prefix)).reduce(_ ++ _)\n\n  private val prefixedOriginalAuthorAggregateFeatures =\n    typedAggregateGroups.flatMap(_.allOutputFeatures).map { feature =>\n      aggregateFeaturesRenameMap.getOrElse(feature, feature)\n    }\n\n  private val authorFeatures = prefixedOriginalAuthorAggregateFeatures ++ Seq(\n    UserHealthFeatures.AuthorState,\n    UserHealthFeatures.NumAuthorFollowers,\n    UserHealthFeatures.NumAuthorConnectDays,\n    UserHealthFeatures.NumAuthorConnect\n  )\n\n  private val aggregateFeatureContext: FeatureContext =\n    new FeatureContext(typedAggregateGroups.flatMap(_.allOutputFeatures).asJava)\n\n  private lazy val prefixedAggregateFeatureContext: FeatureContext =\n    new FeatureContext(prefixedOriginalAuthorAggregateFeatures.asJava)\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(authorFeatures: _*)\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  private val compactDataRecordConverter = new CompactDataRecordConverter()\n\n  override def adaptToDataRecords(\n    authorFeatures: af.AuthorFeatures\n  ): java.util.List[DataRecord] = {\n    val dataRecord =\n      if (authorFeatures.aggregates != null) {\n        val originalAuthorAggregatesDataRecord =\n          compactDataRecordConverter.compactDataRecordToDataRecord(authorFeatures.aggregates)\n\n        DataRecordUtil.applyRename(\n          originalAuthorAggregatesDataRecord,\n          aggregateFeatureContext,\n          prefixedAggregateFeatureContext,\n          aggregateFeaturesRenameMap)\n      } else new DataRecord\n\n    if (authorFeatures.user_health != null) {\n      val userHealth = authorFeatures.user_health\n\n      if (userHealth.user_state != null) {\n        dataRecord.setFeatureValue(\n          UserHealthFeatures.AuthorState,\n          userHealth.user_state.getValue.toLong\n        )\n      }\n\n      dataRecord.setFeatureValue(\n        UserHealthFeatures.NumAuthorFollowers,\n        userHealth.num_followers.toDouble\n      )\n\n      dataRecord.setFeatureValue(\n        UserHealthFeatures.NumAuthorConnectDays,\n        userHealth.num_connect_days.toDouble\n      )\n\n      dataRecord.setFeatureValue(\n        UserHealthFeatures.NumAuthorConnect,\n        userHealth.num_connect.toDouble\n      )\n    }\n\n    List(dataRecord).asJava\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/author_features/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/scala/com/twitter/timelines/prediction/common/aggregates\",\n        \"src/scala/com/twitter/timelines/prediction/features/user_health\",\n        \"src/thrift/com/twitter/timelines/author_features:thrift-java\",\n        \"timelines/data_processing/ml_util/aggregation_framework\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/content/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters\",\n        \"src/scala/com/twitter/timelines/prediction/features/common\",\n        \"src/scala/com/twitter/timelines/prediction/features/conversation_features\",\n        \"src/scala/com/twitter/timelines/prediction/features/recap\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/content/ClipEmbeddingFeaturesAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.content\n\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.FloatTensor\nimport com.twitter.ml.api.GeneralTensor\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport scala.collection.JavaConverters._\n\nobject ClipEmbeddingFeaturesAdapter extends TimelinesMutatingAdapterBase[Seq[Double]] {\n\n  val ClipEmbeddingsFeature: Feature.Tensor = TimelinesSharedFeatures.CLIP_EMBEDDING\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(ClipEmbeddingsFeature)\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def setFeatures(\n    clipEmbedding: Seq[Double],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    val clipEmbeddingTensor = new GeneralTensor()\n    clipEmbeddingTensor.setFloatTensor(new FloatTensor(clipEmbedding.map(Double.box).asJava))\n    richDataRecord.setFeatureValue(\n      ClipEmbeddingsFeature,\n      clipEmbeddingTensor\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/content/ContentFeatureAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.content\n\nimport com.twitter.home_mixer.model.ContentFeatures\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.util.DataRecordConverters._\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.common.adapters.TweetLengthType\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport com.twitter.timelines.prediction.features.conversation_features.ConversationFeatures\nimport com.twitter.timelines.prediction.features.recap.RecapFeatures\nimport scala.collection.JavaConverters._\n\nobject ContentFeatureAdapter extends TimelinesMutatingAdapterBase[Option[ContentFeatures]] {\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(\n    ConversationFeatures.IS_SELF_THREAD_TWEET,\n    ConversationFeatures.IS_LEAF_IN_SELF_THREAD,\n    TimelinesSharedFeatures.ASPECT_RATIO_DEN,\n    TimelinesSharedFeatures.ASPECT_RATIO_NUM,\n    TimelinesSharedFeatures.BIT_RATE,\n    TimelinesSharedFeatures.CLASSIFICATION_LABELS,\n    TimelinesSharedFeatures.COLOR_1_BLUE,\n    TimelinesSharedFeatures.COLOR_1_GREEN,\n    TimelinesSharedFeatures.COLOR_1_PERCENTAGE,\n    TimelinesSharedFeatures.COLOR_1_RED,\n    TimelinesSharedFeatures.FACE_AREAS,\n    TimelinesSharedFeatures.HAS_APP_INSTALL_CALL_TO_ACTION,\n    TimelinesSharedFeatures.HAS_DESCRIPTION,\n    TimelinesSharedFeatures.HAS_QUESTION,\n    TimelinesSharedFeatures.HAS_SELECTED_PREVIEW_IMAGE,\n    TimelinesSharedFeatures.HAS_TITLE,\n    TimelinesSharedFeatures.HAS_VISIT_SITE_CALL_TO_ACTION,\n    TimelinesSharedFeatures.HAS_WATCH_NOW_CALL_TO_ACTION,\n    TimelinesSharedFeatures.HEIGHT_1,\n    TimelinesSharedFeatures.HEIGHT_2,\n    TimelinesSharedFeatures.HEIGHT_3,\n    TimelinesSharedFeatures.HEIGHT_4,\n    TimelinesSharedFeatures.IS_360,\n    TimelinesSharedFeatures.IS_EMBEDDABLE,\n    TimelinesSharedFeatures.IS_MANAGED,\n    TimelinesSharedFeatures.IS_MONETIZABLE,\n    TimelinesSharedFeatures.MEDIA_PROVIDERS,\n    TimelinesSharedFeatures.NUM_CAPS,\n    TimelinesSharedFeatures.NUM_COLOR_PALLETTE_ITEMS,\n    TimelinesSharedFeatures.NUM_FACES,\n    TimelinesSharedFeatures.NUM_MEDIA_TAGS,\n    TimelinesSharedFeatures.NUM_NEWLINES,\n    TimelinesSharedFeatures.NUM_STICKERS,\n    TimelinesSharedFeatures.NUM_WHITESPACES,\n    TimelinesSharedFeatures.RESIZE_METHOD_1,\n    TimelinesSharedFeatures.RESIZE_METHOD_2,\n    TimelinesSharedFeatures.RESIZE_METHOD_3,\n    TimelinesSharedFeatures.RESIZE_METHOD_4,\n    TimelinesSharedFeatures.TWEET_LENGTH,\n    TimelinesSharedFeatures.TWEET_LENGTH_TYPE,\n    TimelinesSharedFeatures.VIDEO_DURATION,\n    TimelinesSharedFeatures.VIEW_COUNT,\n    TimelinesSharedFeatures.WIDTH_1,\n    TimelinesSharedFeatures.WIDTH_2,\n    TimelinesSharedFeatures.WIDTH_3,\n    TimelinesSharedFeatures.WIDTH_4,\n    RecapFeatures.HAS_VIDEO,\n    RecapFeatures.HAS_IMAGE,\n  )\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  private def getTweetLengthType(tweetLength: Int): Long = {\n    tweetLength match {\n      case x if 0 > x || 280 < x => TweetLengthType.INVALID\n      case x if 0 <= x && x <= 30 => TweetLengthType.VERY_SHORT\n      case x if 30 < x && x <= 60 => TweetLengthType.SHORT\n      case x if 60 < x && x <= 90 => TweetLengthType.MEDIUM\n      case x if 90 < x && x <= 140 => TweetLengthType.LENGTHY\n      case x if 140 < x && x <= 210 => TweetLengthType.VERY_LENGTHY\n      case x if x > 210 => TweetLengthType.MAXIMUM_LENGTH\n    }\n  }\n\n  override def setFeatures(\n    contentFeatures: Option[ContentFeatures],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    if (contentFeatures.nonEmpty) {\n      val features = contentFeatures.get\n      // Conversation Features\n      richDataRecord.setFeatureValueFromOption(\n        ConversationFeatures.IS_SELF_THREAD_TWEET,\n        Some(features.selfThreadMetadata.nonEmpty)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        ConversationFeatures.IS_LEAF_IN_SELF_THREAD,\n        features.selfThreadMetadata.map(_.isLeaf)\n      )\n\n      // Media Features\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.ASPECT_RATIO_DEN,\n        features.aspectRatioDen.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.ASPECT_RATIO_NUM,\n        features.aspectRatioNum.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.BIT_RATE,\n        features.bitRate.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HEIGHT_1,\n        features.heights.flatMap(_.lift(0)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HEIGHT_2,\n        features.heights.flatMap(_.lift(1)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HEIGHT_3,\n        features.heights.flatMap(_.lift(2)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HEIGHT_4,\n        features.heights.flatMap(_.lift(3)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_MEDIA_TAGS,\n        features.numMediaTags.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.RESIZE_METHOD_1,\n        features.resizeMethods.flatMap(_.lift(0)).map(_.toLong)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.RESIZE_METHOD_2,\n        features.resizeMethods.flatMap(_.lift(1)).map(_.toLong)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.RESIZE_METHOD_3,\n        features.resizeMethods.flatMap(_.lift(2)).map(_.toLong)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.RESIZE_METHOD_4,\n        features.resizeMethods.flatMap(_.lift(3)).map(_.toLong)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.VIDEO_DURATION,\n        features.videoDurationMs.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.WIDTH_1,\n        features.widths.flatMap(_.lift(0)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.WIDTH_2,\n        features.widths.flatMap(_.lift(1)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.WIDTH_3,\n        features.widths.flatMap(_.lift(2)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.WIDTH_4,\n        features.widths.flatMap(_.lift(3)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_COLOR_PALLETTE_ITEMS,\n        features.numColors.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.COLOR_1_RED,\n        features.dominantColorRed.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.COLOR_1_BLUE,\n        features.dominantColorBlue.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.COLOR_1_GREEN,\n        features.dominantColorGreen.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.COLOR_1_PERCENTAGE,\n        features.dominantColorPercentage\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.MEDIA_PROVIDERS,\n        features.mediaOriginProviders.map(_.toSet.asJava)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.IS_360,\n        features.is360\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.VIEW_COUNT,\n        features.viewCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.IS_MANAGED,\n        features.isManaged\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.IS_MONETIZABLE,\n        features.isMonetizable\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.IS_EMBEDDABLE,\n        features.isEmbeddable\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_STICKERS,\n        features.stickerIds.map(_.length.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_FACES,\n        features.faceAreas.map(_.length.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.FACE_AREAS,\n        // guard for exception from max on empty seq\n        features.faceAreas.map(faceAreas =>\n          faceAreas.map(_.toDouble).reduceOption(_ max _).getOrElse(0.0))\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_SELECTED_PREVIEW_IMAGE,\n        features.hasSelectedPreviewImage\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_TITLE,\n        features.hasTitle\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_DESCRIPTION,\n        features.hasDescription\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_VISIT_SITE_CALL_TO_ACTION,\n        features.hasVisitSiteCallToAction\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_APP_INSTALL_CALL_TO_ACTION,\n        features.hasAppInstallCallToAction\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_WATCH_NOW_CALL_TO_ACTION,\n        features.hasWatchNowCallToAction\n      )\n      // text features\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_CAPS,\n        Some(features.numCaps.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.TWEET_LENGTH,\n        Some(features.length.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.TWEET_LENGTH_TYPE,\n        Some(getTweetLengthType(features.length.toInt))\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_WHITESPACES,\n        Some(features.numWhiteSpaces.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_QUESTION,\n        Some(features.hasQuestion)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_NEWLINES,\n        features.numNewlines.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        RecapFeatures.HAS_IMAGE,\n        features.hasImage\n      )\n      richDataRecord.setFeatureValueFromOption(\n        RecapFeatures.HAS_VIDEO,\n        features.hasVideo\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/content/TextTokensFeaturesAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.content\n\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.thriftscala.Int32Tensor\nimport com.twitter.ml.api.util.ScalaToJavaDataRecordConversions\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport com.twitter.ml.api.{thriftscala => ml}\n\nobject TextTokensFeaturesAdapter extends TimelinesMutatingAdapterBase[Seq[Int]] {\n\n  private val TextTokenFeature: Feature.Tensor =\n    TimelinesSharedFeatures.TEXT_TOKENS_EMBEDDING\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(TextTokenFeature)\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def setFeatures(\n    textTokens: Seq[Int],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    richDataRecord.setFeatureValue(\n      TextTokenFeature,\n      ScalaToJavaDataRecordConversions.scalaTensor2Java(\n        ml.GeneralTensor.Int32Tensor(Int32Tensor(ints = textTokens))\n      ))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/content/VideoSummaryEmbeddingFeaturesAdaptor.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.content\n\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.FloatTensor\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.GeneralTensor\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport scala.collection.JavaConverters._\n\nobject VideoSummaryEmbeddingFeaturesAdaptor extends TimelinesMutatingAdapterBase[Seq[Double]] {\n\n  val VideoSummaryEmbeddingFeature: Feature.Tensor = TimelinesSharedFeatures.VIDEO_SUMMARY_EMBEDDING\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(VideoSummaryEmbeddingFeature)\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def setFeatures(\n    videoSummaryEmbedding: Seq[Double],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    val summaryEmbeddingTensor = new GeneralTensor()\n    summaryEmbeddingTensor.setFloatTensor(\n      new FloatTensor(videoSummaryEmbedding.map(Double.box).asJava))\n    richDataRecord.setFeatureValue(\n      VideoSummaryEmbeddingFeature,\n      summaryEmbeddingTensor\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/gizmoduck_features/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/scala/com/twitter/timelines/prediction/features/gizmoduck\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/gizmoduck_features/GizmoduckFeaturesAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.gizmoduck_features\n\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.gizmoduck.GizmoduckFeatures\nimport java.lang.{Boolean => JBoolean}\n\nobject GizmoduckFeaturesAdapter extends TimelinesMutatingAdapterBase[GFeatures] {\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(\n    GizmoduckFeatures.AUTHOR_IS_BLUE_VERIFIED,\n    GizmoduckFeatures.AUTHOR_IS_VERIFIED_ORGANIZATION,\n    GizmoduckFeatures.AUTHOR_IS_VERIFIED_ORGANIZATION_AFFILIATE,\n    GizmoduckFeatures.AUTHOR_IS_PROTECTED,\n  )\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def candidateFeatures: Set[Feature[_]] = super.candidateFeatures\n  override def setFeatures(\n    contentFeatures: GFeatures,\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    richDataRecord.setFeatureValue[JBoolean](\n      GizmoduckFeatures.AUTHOR_IS_BLUE_VERIFIED,\n      contentFeatures.isBlueVerified\n    )\n    richDataRecord.setFeatureValue[JBoolean](\n      GizmoduckFeatures.AUTHOR_IS_VERIFIED_ORGANIZATION,\n      contentFeatures.isVerifiedOrganization\n    )\n    richDataRecord.setFeatureValue[JBoolean](\n      GizmoduckFeatures.AUTHOR_IS_VERIFIED_ORGANIZATION_AFFILIATE,\n      contentFeatures.isVerifiedOrganizationAffiliate\n    )\n    richDataRecord.setFeatureValue[JBoolean](\n      GizmoduckFeatures.AUTHOR_IS_PROTECTED,\n      contentFeatures.isProtected\n    )\n  }\n}\n\ntrait GFeatures {\n  def isBlueVerified: Boolean\n  def isVerifiedOrganization: Boolean\n  def isVerifiedOrganizationAffiliate: Boolean\n  def isProtected: Boolean\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/inferred_topic/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/scala/com/twitter/timelines/prediction/features/common\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/inferred_topic/InferredTopicAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.inferred_topic\n\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport scala.collection.JavaConverters._\n\nobject InferredTopicAdapter extends TimelinesMutatingAdapterBase[Map[Long, Double]] {\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(\n    TimelinesSharedFeatures.INFERRED_TOPIC_IDS)\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def setFeatures(\n    inferredTopicFeatures: Map[Long, Double],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    richDataRecord.setFeatureValue(\n      TimelinesSharedFeatures.INFERRED_TOPIC_IDS,\n      inferredTopicFeatures.keys.map(_.toString).toSet.asJava)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/light_ranking_features/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/scala/com/twitter/timelines/prediction/features/common\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/light_ranking_features/LightRankingCandidateFeaturesAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.light_ranking_features\n\nimport com.twitter.home_mixer.thriftscala.ServedType\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport java.lang.{Boolean => JBoolean}\nimport java.lang.{Long => JLong}\nimport java.lang.{String => JString}\n\ncase class LightRankingCandidateFeatures(\n  isSelected: Boolean,\n  isSelectedByHeavyRanker: Boolean,\n  rankByHeavyRanker: Int,\n  servedType: ServedType,\n  candidateSourcePosition: Long,\n)\n\n/**\n * define light ranking features adapter to create a data record which includes many light ranking features\n * e.g. predictionRequestId, userId, tweetId to be used as joined key in batch pipeline.\n */\nobject LightRankingCandidateFeaturesAdapter\n    extends TimelinesMutatingAdapterBase[LightRankingCandidateFeatures] {\n\n  val featureContext = new FeatureContext(\n    TimelinesSharedFeatures.IS_SELECTED,\n    TimelinesSharedFeatures.IS_SELECTED_BY_HEAVY_RANKER,\n    TimelinesSharedFeatures.RANK_BY_HEAVY_RANKER,\n    TimelinesSharedFeatures.SERVED_TYPE_ID,\n    TimelinesSharedFeatures.SERVED_TYPE,\n    TimelinesSharedFeatures.CANDIDATE_SOURCE_POSITION,\n  )\n\n  override def getFeatureContext: FeatureContext = featureContext\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def setFeatures(\n    lightRankingCandidateFeatures: LightRankingCandidateFeatures,\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    richDataRecord.setFeatureValue[JBoolean](\n      TimelinesSharedFeatures.IS_SELECTED,\n      lightRankingCandidateFeatures.isSelected)\n    richDataRecord.setFeatureValue[JBoolean](\n      TimelinesSharedFeatures.IS_SELECTED_BY_HEAVY_RANKER,\n      lightRankingCandidateFeatures.isSelectedByHeavyRanker)\n    richDataRecord.setFeatureValue[JLong](\n      TimelinesSharedFeatures.RANK_BY_HEAVY_RANKER,\n      lightRankingCandidateFeatures.rankByHeavyRanker)\n    richDataRecord.setFeatureValue[JString](\n      TimelinesSharedFeatures.SERVED_TYPE,\n      lightRankingCandidateFeatures.servedType.name)\n    richDataRecord.setFeatureValue[JLong](\n      TimelinesSharedFeatures.SERVED_TYPE_ID,\n      lightRankingCandidateFeatures.servedType.getValue())\n    richDataRecord.setFeatureValue[JLong](\n      TimelinesSharedFeatures.CANDIDATE_SOURCE_POSITION,\n      lightRankingCandidateFeatures.candidateSourcePosition)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/non_ml_features/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/ml/api/constant\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/scala/com/twitter/timelines/prediction/features/common\",\n        \"src/scala/com/twitter/timelines/prediction/features/request_context\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/non_ml_features/NonMLCandidateFeaturesAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.non_ml_features\n\nimport com.twitter.ml.api.constant.SharedFeatures\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport java.lang.{Long => JLong}\n\ncase class NonMLCandidateFeatures(\n  tweetId: Long,\n  sourceTweetId: Long,\n  originalAuthorId: Option[Long],\n)\n\n/**\n * define non ml features adapter to create a data record which includes many non ml features\n * e.g. predictionRequestId, userId, tweetId to be used as joined key in batch pipeline.\n */\nobject NonMLCandidateFeaturesAdapter extends TimelinesMutatingAdapterBase[NonMLCandidateFeatures] {\n\n  private val featureContext = new FeatureContext(\n    SharedFeatures.TWEET_ID,\n    // For Secondary Engagement data generation\n    TimelinesSharedFeatures.SOURCE_TWEET_ID,\n    TimelinesSharedFeatures.ORIGINAL_AUTHOR_ID,\n  )\n\n  override def getFeatureContext: FeatureContext = featureContext\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def setFeatures(\n    nonMLCandidateFeatures: NonMLCandidateFeatures,\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    richDataRecord.setFeatureValue[JLong](SharedFeatures.TWEET_ID, nonMLCandidateFeatures.tweetId)\n    richDataRecord.setFeatureValue[JLong](\n      TimelinesSharedFeatures.SOURCE_TWEET_ID,\n      nonMLCandidateFeatures.sourceTweetId)\n    nonMLCandidateFeatures.originalAuthorId.foreach(\n      richDataRecord.setFeatureValue[JLong](TimelinesSharedFeatures.ORIGINAL_AUTHOR_ID, _))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/non_ml_features/NonMLCommonFeaturesAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.non_ml_features\n\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.constant.SharedFeatures\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport com.twitter.timelines.prediction.features.request_context.RequestContextFeatures\nimport java.lang.{Long => JLong}\nimport java.lang.{String => JString}\n\ncase class NonMLCommonFeatures(\n  userId: Long,\n  guestId: Option[Long],\n  clientId: Option[Long],\n  countryCode: Option[String],\n  predictionRequestId: Option[Long],\n  productSurface: String,\n  servedTimestamp: Long,\n)\n\n/**\n * define non ml features adapter to create a data record which includes many non ml features\n * e.g. predictionRequestId, userId, tweetId to be used as joined key in batch pipeline.\n */\nobject NonMLCommonFeaturesAdapter extends TimelinesMutatingAdapterBase[NonMLCommonFeatures] {\n\n  private val featureContext = new FeatureContext(\n    SharedFeatures.USER_ID,\n    SharedFeatures.GUEST_ID,\n    SharedFeatures.CLIENT_ID,\n    TimelinesSharedFeatures.PREDICTION_REQUEST_ID,\n    TimelinesSharedFeatures.PRODUCT_SURFACE,\n    TimelinesSharedFeatures.SERVED_TIMESTAMP,\n    RequestContextFeatures.COUNTRY_CODE,\n  )\n\n  override def getFeatureContext: FeatureContext = featureContext\n\n  override val commonFeatures: Set[Feature[_]] = Set(\n    SharedFeatures.USER_ID,\n    SharedFeatures.GUEST_ID,\n    SharedFeatures.CLIENT_ID,\n    TimelinesSharedFeatures.PREDICTION_REQUEST_ID,\n    TimelinesSharedFeatures.PRODUCT_SURFACE,\n    TimelinesSharedFeatures.SERVED_TIMESTAMP,\n    RequestContextFeatures.COUNTRY_CODE,\n  )\n\n  override def setFeatures(\n    nonMLCommonFeatures: NonMLCommonFeatures,\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    richDataRecord.setFeatureValue[JLong](SharedFeatures.USER_ID, nonMLCommonFeatures.userId)\n    nonMLCommonFeatures.guestId.foreach(\n      richDataRecord.setFeatureValue[JLong](SharedFeatures.GUEST_ID, _))\n    nonMLCommonFeatures.clientId.foreach(\n      richDataRecord.setFeatureValue[JLong](SharedFeatures.CLIENT_ID, _))\n    nonMLCommonFeatures.predictionRequestId.foreach(\n      richDataRecord.setFeatureValue[JLong](TimelinesSharedFeatures.PREDICTION_REQUEST_ID, _))\n    nonMLCommonFeatures.countryCode.foreach(\n      richDataRecord.setFeatureValue[JString](RequestContextFeatures.COUNTRY_CODE, _))\n    richDataRecord.setFeatureValue(\n      TimelinesSharedFeatures.PRODUCT_SURFACE,\n      nonMLCommonFeatures.productSurface)\n    richDataRecord.setFeatureValue[JLong](\n      TimelinesSharedFeatures.SERVED_TIMESTAMP,\n      nonMLCommonFeatures.servedTimestamp)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/offline_aggregates/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"timelines/data_processing/ml_util/aggregation_framework/conversion:for-timelines\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/offline_aggregates/PassThroughAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.offline_aggregates\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.IRecordOneToOneAdapter\n\nobject PassThroughAdapter extends IRecordOneToOneAdapter[Seq[DataRecord]] {\n  override def adaptToDataRecord(record: Seq[DataRecord]): DataRecord =\n    record.headOption.getOrElse(new DataRecord)\n\n  // This is not necessary and should not be used.\n  override def getFeatureContext = ???\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/offline_aggregates/SparseAggregatesToDenseAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.offline_aggregates\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.conversion.CombineCountsPolicy\nimport com.twitter.timelines.prediction.common.adapters.TimelinesIRecordAdapter\n\nclass SparseAggregatesToDenseAdapter(policy: CombineCountsPolicy)\n    extends TimelinesIRecordAdapter[Seq[DataRecord]] {\n\n  override def setFeatures(input: Seq[DataRecord], mutableDataRecord: RichDataRecord): Unit =\n    policy.defaultMergeRecord(mutableDataRecord.getRecord, input)\n\n  override val getFeatureContext: FeatureContext =\n    new FeatureContext(policy.outputFeaturesPostMerge.toSeq: _*)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/simclusters_features/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters\",\n        \"src/scala/com/twitter/timelines/prediction/features/simcluster\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/simclusters_features/SimclustersFeaturesAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.simclusters_features\n\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbedding\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.simcluster.SimclusterFeatures\nimport java.util\nimport java.util.Collections\n\nobject SimclustersFeaturesAdapter\n    extends TimelinesMutatingAdapterBase[Option[SimClustersEmbedding]] {\n\n  val SimclustersSparseTweetEmbeddingsFeature: Feature.SparseContinuous =\n    SimclusterFeatures.SIMCLUSTER_TWEET_CLUSTER_SCORES\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(\n    SimclustersSparseTweetEmbeddingsFeature)\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def setFeatures(\n    simClustersEmbedding: Option[SimClustersEmbedding],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    val simclustersWithScoresMap = simClustersEmbedding match {\n      case Some(value) =>\n        val result = new util.HashMap[String, java.lang.Double](value.embedding.size, 1.0f)\n        value.embedding.foreach { simclusterWithScore =>\n          result.put(simclusterWithScore.clusterId.toString, simclusterWithScore.score)\n        }\n        result\n      case None =>\n        Collections.emptyMap[String, java.lang.Double]()\n    }\n    richDataRecord.setFeatureValue(\n      SimclustersSparseTweetEmbeddingsFeature,\n      simclustersWithScoresMap\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/transformer_embeddings/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"user_history_transformer/thrift/src/main/thrift/com/twitter/user_history_transformer:user_history_transformer-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/transformer_embeddings/TransformerEmbeddingsAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings\n\nimport com.twitter.ml.api.DataType\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.util.ScalaToJavaDataRecordConversions\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport scala.collection.JavaConverters._\n\nsealed trait TransformerEmbeddingsAdapter\n    extends TimelinesMutatingAdapterBase[Option[ml.FloatTensor]] {\n  def embeddingsFeature: Feature.Tensor\n\n  override def getFeatureContext: FeatureContext = new FeatureContext(\n    embeddingsFeature\n  )\n\n  override def setFeatures(\n    embedding: Option[ml.FloatTensor],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    embedding.foreach { floatTensor =>\n      richDataRecord.setFeatureValue(\n        embeddingsFeature,\n        ScalaToJavaDataRecordConversions.scalaTensor2Java(\n          ml.GeneralTensor\n            .FloatTensor(floatTensor)))\n    }\n  }\n}\n\nsealed trait TransformerByteEmbeddingsAdapter\n    extends TimelinesMutatingAdapterBase[Option[ml.RawTypedTensor]] {\n  def embeddingsFeature: Feature.Tensor\n\n  override def getFeatureContext: FeatureContext = new FeatureContext(\n    embeddingsFeature\n  )\n\n  override def setFeatures(\n    embedding: Option[ml.RawTypedTensor],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    embedding.foreach { tensor =>\n      richDataRecord.setFeatureValue(\n        embeddingsFeature,\n        ScalaToJavaDataRecordConversions.scalaTensor2Java(\n          ml.GeneralTensor\n            .RawTypedTensor(tensor)))\n    }\n  }\n}\n\nobject TransformerEmbeddingsFeatures {\n  val UserHistoryTransformerEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.transformer.user_history_as_float_tensor\",\n    DataType.FLOAT,\n    List(128L).map(long2Long).asJava,\n  )\n  val UserHistoryTransformerEmbeddingsGreenFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.transformer.green.user_history_as_float_tensor\",\n    DataType.BYTE,\n    List(1024L).map(long2Long).asJava,\n  )\n  val UserHistoryTransformerEmbeddingsJointBlueFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.transformer.joint.blue.user_history_as_float_tensor\",\n    DataType.BYTE,\n    List(1024L).map(long2Long).asJava,\n  )\n  val PostTransformerEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"tweet_id.transformer.post_as_float_tensor\",\n    DataType.FLOAT,\n    List(128L).map(long2Long).asJava,\n  )\n  val PostTransformerEmbeddingsGreenFeature: Feature.Tensor = new Feature.Tensor(\n    \"tweet_id.transformer.green.post_as_float_tensor\",\n    DataType.FLOAT,\n    List(128L).map(long2Long).asJava,\n  )\n  val PostTransformerEmbeddingsJointBlueFeature: Feature.Tensor = new Feature.Tensor(\n    \"tweet_id.transformer.joint.blue.post_as_float_tensor\",\n    DataType.FLOAT,\n    List(128L).map(long2Long).asJava,\n  )\n}\n\nobject UserHistoryTransformerEmbeddingsHomeBlueAdapter extends TransformerEmbeddingsAdapter {\n  override val embeddingsFeature: Feature.Tensor =\n    TransformerEmbeddingsFeatures.UserHistoryTransformerEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n}\n\nobject UserHistoryTransformerEmbeddingsHomeGreenAdapter extends TransformerByteEmbeddingsAdapter {\n  override val embeddingsFeature: Feature.Tensor =\n    TransformerEmbeddingsFeatures.UserHistoryTransformerEmbeddingsGreenFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n}\n\nobject UserHistoryTransformerEmbeddingsJointBlueAdapter extends TransformerByteEmbeddingsAdapter {\n  override val embeddingsFeature: Feature.Tensor =\n    TransformerEmbeddingsFeatures.UserHistoryTransformerEmbeddingsJointBlueFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n}\n\nobject PostTransformerEmbeddingsHomeBlueAdapter extends TransformerEmbeddingsAdapter {\n  override val embeddingsFeature: Feature.Tensor =\n    TransformerEmbeddingsFeatures.PostTransformerEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n}\n\nobject PostTransformerEmbeddingsHomeGreenAdapter extends TransformerEmbeddingsAdapter {\n  override val embeddingsFeature: Feature.Tensor =\n    TransformerEmbeddingsFeatures.PostTransformerEmbeddingsGreenFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n}\n\nobject PostTransformerEmbeddingsJointBlueAdapter extends TransformerEmbeddingsAdapter {\n  override val embeddingsFeature: Feature.Tensor =\n    TransformerEmbeddingsFeatures.PostTransformerEmbeddingsJointBlueFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/transformer_embeddings/UserHistoryEventsAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings\n\nimport com.twitter.ml.api.DataType\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.GeneralTensor\nimport com.twitter.ml.api.Int32Tensor\nimport com.twitter.ml.api.Int64Tensor\nimport com.twitter.ml.api.RawTypedTensor\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.user_history_transformer.thriftscala.UserHistory\nimport scala.collection.JavaConverters._\nimport java.nio.ByteBuffer\n\nobject UserHistoryEventsFeatures {\n  val TweetIdsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.history.user_history_events_tweet_ids\",\n    DataType.INT64,\n    List(100L).map(long2Long).asJava,\n  )\n\n  val AuthorIdsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.history.user_history_events_author_ids\",\n    DataType.INT64,\n    List(100L).map(long2Long).asJava,\n  )\n\n  val ActionTypesFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.history.user_history_events_action_types\",\n    DataType.INT32,\n    List(100L).map(long2Long).asJava,\n  )\n\n  val ActionTimestampsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.history.user_history_events_action_timestamps\",\n    DataType.INT64,\n    List(100L).map(long2Long).asJava,\n  )\n\n  val SemanticIdsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.history.user_history_events_semantic_ids\",\n    DataType.UINT8,\n    List(500L).map(long2Long).asJava,\n  )\n}\n\ntrait BaseUserHistoryEventsAdapter extends TimelinesMutatingAdapterBase[Seq[UserHistory]] {\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  private val DefaultSemanticId: Seq[Short] = Seq(0, 0, 0, 0, 0)\n\n  def setAdditionalFeatures(\n    record: Seq[UserHistory],\n    mutableDataRecord: RichDataRecord\n  ): Unit = {}\n\n  override def setFeatures(\n    record: Seq[UserHistory],\n    mutableDataRecord: RichDataRecord\n  ): Unit = {\n    val tweetIdsFeature = GeneralTensor.int64Tensor(\n      new Int64Tensor(\n        record\n          .map(history => history.sourceTweetId.getOrElse(history.tweetId)).map(long2Long).asJava))\n\n    val authorIdsFeature = GeneralTensor.int64Tensor(\n      new Int64Tensor(record.map(_.authorId.getOrElse(-1L)).map(long2Long).asJava))\n\n    val actionTypesFeature = GeneralTensor.int32Tensor(\n      new Int32Tensor(record.map(_.actionType.getValue()).map(int2Integer).asJava))\n\n    val actionTimestampsFeature = GeneralTensor.int64Tensor(\n      new Int64Tensor(record.map(_.engagedTimestampMs).map(long2Long).asJava))\n\n    val semanticIdsFeature = GeneralTensor.rawTypedTensor(\n      new RawTypedTensor(\n        DataType.UINT8,\n        ByteBuffer.wrap(\n          record\n            .flatMap(_.metadata.flatMap(_.semanticId).getOrElse(DefaultSemanticId))\n            .map(_.toByte).toArray)\n      )\n    )\n\n    mutableDataRecord\n      .setFeatureValue(UserHistoryEventsFeatures.TweetIdsFeature, tweetIdsFeature)\n    mutableDataRecord\n      .setFeatureValue(UserHistoryEventsFeatures.AuthorIdsFeature, authorIdsFeature)\n    mutableDataRecord\n      .setFeatureValue(UserHistoryEventsFeatures.ActionTypesFeature, actionTypesFeature)\n    mutableDataRecord\n      .setFeatureValue(UserHistoryEventsFeatures.ActionTimestampsFeature, actionTimestampsFeature)\n    mutableDataRecord\n      .setFeatureValue(UserHistoryEventsFeatures.SemanticIdsFeature, semanticIdsFeature)\n\n    setAdditionalFeatures(record, mutableDataRecord)\n\n  }\n\n  override def getFeatureContext: FeatureContext = new FeatureContext(\n    UserHistoryEventsFeatures.TweetIdsFeature,\n    UserHistoryEventsFeatures.AuthorIdsFeature,\n    UserHistoryEventsFeatures.ActionTypesFeature,\n    UserHistoryEventsFeatures.ActionTimestampsFeature,\n    UserHistoryEventsFeatures.SemanticIdsFeature,\n  )\n}\nobject UserHistoryEventsAdapter extends BaseUserHistoryEventsAdapter {}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/transformer_embeddings/VideoUserHistoryEventsAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings\n\nimport com.twitter.ml.api.DataType\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.GeneralTensor\nimport com.twitter.ml.api.Int32Tensor\nimport com.twitter.ml.api.Int64Tensor\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.StringTensor\nimport com.twitter.user_history_transformer.thriftscala.UserHistory\nimport scala.collection.JavaConverters._\n\nobject VideoUserHistoryEventsFeatures {\n  val VideoWatchTimesFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.history.user_history_events_video_watch_times\",\n    DataType.INT64,\n    List(100L).map(long2Long).asJava,\n  )\n  val VideoMediaIdsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.history.user_history_events_video_media_ids\",\n    DataType.INT64,\n    List(100L).map(long2Long).asJava,\n  )\n\n  val VideoMediaCategoriesFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.history.user_history_events_video_media_categories\",\n    DataType.INT32,\n    List(100L).map(long2Long).asJava,\n  )\n\n  val VideoDurationsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.history.user_history_events_video_durations\",\n    DataType.INT32,\n    List(100L).map(long2Long).asJava,\n  )\n  val VideoTopEntityFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.history.user_history_events_video_top_entity_info\",\n    DataType.STRING,\n    List(100L).map(long2Long).asJava,\n  )\n  val VideoGrokTagsInfoFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.history.user_history_events_video_grok_tags_info\",\n    DataType.STRING,\n    List(100L).map(long2Long).asJava,\n  )\n  val VideoCluster95IdsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.history.user_history_events_video_cluster95_ids_info\",\n    DataType.INT64,\n    List(100L).map(long2Long).asJava,\n  )\n}\n\nobject VideoUserHistoryEventsAdapter extends BaseUserHistoryEventsAdapter {\n  override def getFeatureContext: FeatureContext = {\n    super.getFeatureContext\n      .addFeatures(VideoUserHistoryEventsFeatures.VideoWatchTimesFeature)\n      .addFeatures(VideoUserHistoryEventsFeatures.VideoMediaIdsFeature)\n      .addFeatures(VideoUserHistoryEventsFeatures.VideoMediaCategoriesFeature)\n      .addFeatures(VideoUserHistoryEventsFeatures.VideoDurationsFeature)\n      .addFeatures(VideoUserHistoryEventsFeatures.VideoCluster95IdsFeature)\n      .addFeatures(VideoUserHistoryEventsFeatures.VideoTopEntityFeature)\n      .addFeatures(VideoUserHistoryEventsFeatures.VideoGrokTagsInfoFeature)\n  }\n\n  override def setAdditionalFeatures(\n    record: Seq[UserHistory],\n    mutableDataRecord: RichDataRecord\n  ): Unit = {\n    val videoWatchTimes = new GeneralTensor()\n    videoWatchTimes.setInt64Tensor(\n      new Int64Tensor(\n        record.flatMap(_.metadata.flatMap(_.watchTime).map(_.toLong)).map(long2Long).asJava)\n    )\n    mutableDataRecord.setFeatureValue(\n      VideoUserHistoryEventsFeatures.VideoWatchTimesFeature,\n      videoWatchTimes\n    )\n    val videoMediaIds = new GeneralTensor()\n    videoMediaIds.setInt64Tensor(\n      new Int64Tensor(record.flatMap(_.metadata.flatMap(_.mediaId)).map(long2Long).asJava)\n    )\n    mutableDataRecord.setFeatureValue(\n      VideoUserHistoryEventsFeatures.VideoMediaIdsFeature,\n      videoMediaIds\n    )\n\n    val videoMediaCategories = new GeneralTensor()\n    videoMediaCategories.setInt32Tensor(\n      new Int32Tensor(\n        record.flatMap(_.metadata).flatMap(_.mediaCategory).map(_.value).map(int2Integer).asJava))\n    mutableDataRecord.setFeatureValue(\n      VideoUserHistoryEventsFeatures.VideoMediaCategoriesFeature,\n      videoMediaCategories\n    )\n\n    val videoDurations = new GeneralTensor()\n    videoDurations.setInt32Tensor(\n      new Int32Tensor(record.flatMap(_.metadata).flatMap(_.videoDuration).map(int2Integer).asJava))\n    mutableDataRecord.setFeatureValue(\n      VideoUserHistoryEventsFeatures.VideoDurationsFeature,\n      videoDurations\n    )\n\n    val topEntitiesInfo = new GeneralTensor()\n    val topEntitiesAsString: Seq[String] = record.map { userHistory =>\n      userHistory.metadata\n        .flatMap(_.entities)\n        .map(_.filter(_.score.isDefined))\n        .filter(_.nonEmpty)\n        .map(_.maxBy(_.score.get))\n        .headOption\n        .map(entity => entity.qualifiedId._1.toString + \":\" + entity.qualifiedId._2.toString)\n        .getOrElse(\"\")\n    }\n\n    topEntitiesInfo.setStringTensor(new StringTensor(topEntitiesAsString.asJava))\n    mutableDataRecord.setFeatureValue(\n      VideoUserHistoryEventsFeatures.VideoTopEntityFeature,\n      topEntitiesInfo\n    )\n\n    val grokTagsInfo = new GeneralTensor()\n    val grokTagsAsString: Seq[String] = record.map { userHistory =>\n      userHistory.metadata\n        .flatMap(_.tags)\n        .map(_.map(_.tag))\n        .getOrElse(Nil)\n        .mkString(\":\")\n    }\n    grokTagsInfo.setStringTensor(new StringTensor(grokTagsAsString.asJava))\n\n    mutableDataRecord.setFeatureValue(\n      VideoUserHistoryEventsFeatures.VideoGrokTagsInfoFeature,\n      grokTagsInfo\n    )\n\n    val clusterIdsFeature = new GeneralTensor()\n    clusterIdsFeature.setInt64Tensor(\n      new Int64Tensor(record.flatMap(_.metadata).flatMap(_.clusterId).map(long2Long).asJava))\n    mutableDataRecord.setFeatureValue(\n      VideoUserHistoryEventsFeatures.VideoCluster95IdsFeature,\n      clusterIdsFeature\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/twhin_embeddings/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/ml/api:embedding-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/twhin_embeddings/TwhinEmbeddingsAdapter.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings\n\nimport com.twitter.ml.api.DataType\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.util.ScalaToJavaDataRecordConversions\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\n\nsealed trait TwhinEmbeddingsAdapter extends TimelinesMutatingAdapterBase[Option[ml.FloatTensor]] {\n  def twhinEmbeddingsFeature: Feature.Tensor\n\n  override def getFeatureContext: FeatureContext = new FeatureContext(\n    twhinEmbeddingsFeature\n  )\n\n  override def setFeatures(\n    embedding: Option[ml.FloatTensor],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    embedding.foreach { floatTensor =>\n      richDataRecord.setFeatureValue(\n        twhinEmbeddingsFeature,\n        ScalaToJavaDataRecordConversions.scalaTensor2Java(\n          ml.GeneralTensor\n            .FloatTensor(floatTensor)))\n    }\n  }\n}\n\nobject TwhinEmbeddingsFeatures {\n  val TwhinAuthorFollowEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"original_author.twhin.tw_hi_n.author_follow_as_float_tensor\",\n    DataType.FLOAT\n  )\n\n  val TwhinUserEngagementEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.twhin.tw_hi_n.user_engagement_as_float_tensor\",\n    DataType.FLOAT\n  )\n\n  val TwhinRebuildUserEngagementEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.twhin.tw_hi_n.rebuild_user_engagement_as_float_tensor\",\n    DataType.FLOAT\n  )\n\n  val TwhinUserFollowEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.twhin.tw_hi_n.user_follow_as_float_tensor\",\n    DataType.FLOAT\n  )\n\n  val TwhinUserPositiveEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.twhin.tw_hi_n.user_positive_as_float_tensor\",\n    DataType.FLOAT\n  )\n\n  val TwhinRebuildUserPositiveEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.twhin.tw_hi_n.rebuild_user_positive_as_float_tensor\",\n    DataType.FLOAT\n  )\n\n  val TwhinUserNegativeEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.twhin.tw_hi_n.user_negative_as_float_tensor\",\n    DataType.FLOAT\n  )\n\n  val TwhinTweetEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"original_tweet.twhin.tw_hi_n.tweet_v_2_as_float_tensor\",\n    DataType.FLOAT\n  )\n\n  val TwhinRebuildTweetEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"original_tweet.twhin.tw_hi_n.rebuild_tweet_as_float_tensor\",\n    DataType.FLOAT\n  )\n\n  val TwhinVideoEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"original_tweet.twhin.tw_hi_n.video_as_float_tensor\",\n    DataType.FLOAT\n  )\n}\n\nobject TwhinAuthorFollowEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinAuthorFollowEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n}\n\nobject TwhinUserEngagementEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinUserEngagementEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set(twhinEmbeddingsFeature)\n}\n\nobject TwhinRebuildUserEngagementEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinRebuildUserEngagementEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set(twhinEmbeddingsFeature)\n}\n\nobject TwhinUserFollowEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinUserFollowEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set(twhinEmbeddingsFeature)\n}\n\nobject TwhinUserPositiveEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinUserPositiveEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set(twhinEmbeddingsFeature)\n}\n\nobject TwhinRebuildUserPositiveEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinRebuildUserPositiveEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set(twhinEmbeddingsFeature)\n}\n\nobject TwhinTweetEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinTweetEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n}\n\nobject TwhinRebuildTweetEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinRebuildTweetEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n}\n\nobject TwhinVideoEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinVideoEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n}\n\nobject TwhinUserNegativeEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinUserNegativeEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set(twhinEmbeddingsFeature)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates/AggregateFeatureInfo.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates\n\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.offline_aggregates.BaseAggregateRootFeature\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateType.AggregateType\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.TypedAggregateGroup\nimport scala.jdk.CollectionConverters.asJavaIterableConverter\n\n// A helper class deriving aggregate feature info from the given configuration parameters.\nclass AggregateFeatureInfo(\n  val aggregateGroups: Set[AggregateGroup],\n  val aggregateType: AggregateType) {\n\n  private val typedAggregateGroups = aggregateGroups.flatMap(_.buildTypedAggregateGroups()).toList\n\n  val featureContext: FeatureContext =\n    new FeatureContext(\n      (typedAggregateGroups.flatMap(_.allOutputFeatures) ++\n        typedAggregateGroups.flatMap(_.allOutputKeys) ++\n        Seq(TypedAggregateGroup.timestampFeature)).asJava)\n\n  val feature: BaseAggregateRootFeature =\n    AggregateFeatureInfo.pickFeature(aggregateType)\n}\n\nobject AggregateFeatureInfo {\n  val features: Set[BaseAggregateRootFeature] =\n    Set(PartAAggregateRootFeature, PartBAggregateRootFeature)\n\n  def pickFeature(aggregateType: AggregateType): BaseAggregateRootFeature = {\n    val filtered = features.filter(_.aggregateTypes.contains(aggregateType))\n    require(\n      filtered.size == 1,\n      \"requested AggregateType must be backed by exactly one physical store.\")\n    filtered.head\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/offline_aggregates\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/offline_aggregates\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"servo/repo/src/main/scala\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/request_context\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/scala/com/twitter/timelines/prediction/common/aggregates\",\n        \"src/thrift/com/twitter/user_session_store:thrift-java\",\n        \"timelines/data_processing/jobs/timeline_ranking_user_features:mini\",\n        \"timelines/data_processing/ml_util/aggregation_framework:common_types\",\n        \"timelines/data_processing/ml_util/aggregation_framework/conversion:for-timelines\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates/BaseEdgeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.WithDefaultFeatureMap\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.IRecordOneToOneAdapter\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.offline_aggregates.AggregateFeaturesToDecodeWithMetadata\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.offline_aggregates.BaseAggregateRootFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateType.AggregateType\nimport com.twitter.timelines.suggests.common.dense_data_record.thriftjava.DenseCompactDataRecord\nimport com.twitter.util.Future\nimport java.lang.{Long => JLong}\nimport java.util.{Map => JMap}\n\nabstract case class BaseEdgeAggregateFeature(\n  name: String,\n  aggregateGroups: Set[AggregateGroup],\n  aggregateType: AggregateType,\n  extractMapFn: AggregateFeaturesToDecodeWithMetadata => JMap[JLong, DenseCompactDataRecord],\n  adapter: IRecordOneToOneAdapter[Seq[DataRecord]],\n  getSecondaryKeysFn: CandidateWithFeatures[TweetCandidate] => Seq[Long])\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord\n\n  private val rootFeatureInfo = new AggregateFeatureInfo(aggregateGroups, aggregateType)\n  val featureContext: FeatureContext = rootFeatureInfo.featureContext\n  val rootFeature: BaseAggregateRootFeature = rootFeatureInfo.feature\n\n  override def hashCode(): Int = name.hashCode\n}\n\ntrait BaseEdgeAggregateFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with WithDefaultFeatureMap {\n\n  def aggregateFeatures: Set[BaseEdgeAggregateFeature]\n\n  override def features = aggregateFeatures.asInstanceOf[Set[Feature[_, _]]]\n\n  private val batchSize = 32\n\n  override lazy val defaultFeatureMap: FeatureMap = {\n    val featureMapBuilder = new FeatureMapBuilder()\n    aggregateFeatures.foreach(feature => featureMapBuilder.add(feature, feature.defaultValue))\n    featureMapBuilder.build()\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val featuresSeq = aggregateFeatures.toSeq\n    val dataRecordsFuture = featuresSeq.map { feature =>\n      hydrateAggregateFeature(query, candidates, feature)\n    }\n\n    Future.collect(dataRecordsFuture).map { dataRecordsPerFeature =>\n      val dataRecordsPerCandidate = dataRecordsPerFeature.transpose\n      dataRecordsPerCandidate.map { dataRecords =>\n        assert(featuresSeq.size == dataRecords.size)\n        val builder = FeatureMapBuilder(sizeHint = featuresSeq.size)\n        featuresSeq.zip(dataRecords).map {\n          case (feature, dataRecord) => builder.add(feature, dataRecord)\n        }\n        builder.build()\n      }\n    }\n  }\n\n  private def getDataRecord(\n    ids: Seq[Long],\n    decoded: Map[Long, DataRecord],\n    feature: BaseEdgeAggregateFeature\n  ) = {\n    val dataRecords = ids.flatMap(decoded.get)\n    feature.adapter.adaptToDataRecord(dataRecords)\n  }\n\n  private def hydrateAggregateFeature(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n    feature: BaseEdgeAggregateFeature\n  ): Future[Seq[DataRecord]] = {\n    val rootFeature = feature.rootFeature\n    val extractMapFn = feature.extractMapFn\n    val featureContext = feature.featureContext\n    val secondaryIds: Seq[Seq[Long]] = candidates.map(feature.getSecondaryKeysFn)\n\n    val featuresToDecodeWithMetadata = query.features\n      .flatMap(_.getOrElse(rootFeature, None))\n      .getOrElse(AggregateFeaturesToDecodeWithMetadata.empty)\n\n    // Decode the DenseCompactDataRecords into DataRecords for each required secondary id.\n    val decoded: Map[Long, DataRecord] = Utils.selectAndTransform(\n      secondaryIds.flatten.distinct,\n      featuresToDecodeWithMetadata.toDataRecord,\n      extractMapFn(featuresToDecodeWithMetadata)\n    )\n\n    // Remove unnecessary features in-place. This is safe because the underlying DataRecords\n    // are unique and have just been generated in the previous step.\n    decoded.values.foreach(Utils.filterDataRecord(_, featureContext))\n\n    // Put features into the FeatureMapBuilders\n\n    OffloadFuturePools.offloadBatchElementToElement(\n      secondaryIds,\n      getDataRecord(_, decoded, feature),\n      batchSize)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates/EdgeAggregateFeatures.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TSPInferredTopicFeature\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.offline_aggregates.PassThroughAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.offline_aggregates.SparseAggregatesToDenseAdapter\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MentionScreenNameFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateType\nimport com.twitter.timelines.prediction.common.aggregates.TimelinesAggregationConfig\nimport com.twitter.timelines.prediction.common.aggregates.TimelinesAggregationConfig.CombineCountPolicies\n\nobject EdgeAggregateFeatures {\n\n  object UserAuthorAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        name = \"UserAuthorAggregateFeature\",\n        aggregateGroups = TimelinesAggregationConfig.userAuthorAggregatesV2 ++ Set(\n          TimelinesAggregationConfig.userAuthorAggregatesV5,\n          TimelinesAggregationConfig.tweetSourceUserAuthorAggregatesV1,\n          TimelinesAggregationConfig.twitterWideUserAuthorAggregates\n        ),\n        aggregateType = AggregateType.UserAuthor,\n        extractMapFn = _.userAuthorAggregates,\n        adapter = PassThroughAdapter,\n        getSecondaryKeysFn = _.features.getOrElse(AuthorIdFeature, None).toSeq\n      )\n\n  object UserOriginalAuthorAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        name = \"UserOriginalAuthorAggregateFeature\",\n        aggregateGroups = Set(TimelinesAggregationConfig.userOriginalAuthorAggregatesV1),\n        aggregateType = AggregateType.UserOriginalAuthor,\n        extractMapFn = _.userOriginalAuthorAggregates,\n        adapter = PassThroughAdapter,\n        getSecondaryKeysFn = candidate =>\n          CandidatesUtil.getOriginalAuthorId(candidate.features).toSeq\n      )\n\n  object UserTopicAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        name = \"UserTopicAggregateFeature\",\n        aggregateGroups = Set(\n          TimelinesAggregationConfig.userTopicAggregates,\n          TimelinesAggregationConfig.userTopicAggregatesV2,\n        ),\n        aggregateType = AggregateType.UserTopic,\n        extractMapFn = _.userTopicAggregates,\n        adapter = PassThroughAdapter,\n        getSecondaryKeysFn = candidate =>\n          candidate.features.getOrElse(TopicIdSocialContextFeature, None).toSeq\n      )\n\n  object UserMentionAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        name = \"UserMentionAggregateFeature\",\n        aggregateGroups = Set(TimelinesAggregationConfig.userMentionAggregates),\n        aggregateType = AggregateType.UserMention,\n        extractMapFn = _.userMentionAggregates,\n        adapter = new SparseAggregatesToDenseAdapter(CombineCountPolicies.MentionCountsPolicy),\n        getSecondaryKeysFn = candidate =>\n          candidate.features.getOrElse(MentionScreenNameFeature, Seq.empty).map(_.hashCode.toLong)\n      )\n\n  object UserInferredTopicAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        name = \"UserInferredTopicAggregateFeature\",\n        aggregateGroups = Set(\n          TimelinesAggregationConfig.userInferredTopicAggregates,\n        ),\n        aggregateType = AggregateType.UserInferredTopic,\n        extractMapFn = _.userInferredTopicAggregates,\n        adapter = new SparseAggregatesToDenseAdapter(\n          CombineCountPolicies.UserInferredTopicCountsPolicy),\n        getSecondaryKeysFn = candidate =>\n          candidate.features.getOrElse(TSPInferredTopicFeature, Map.empty[Long, Double]).keys.toSeq\n      )\n\n  object UserInferredTopicAggregateV2Feature\n      extends BaseEdgeAggregateFeature(\n        name = \"UserInferredTopicAggregateV2Feature\",\n        aggregateGroups = Set(\n          TimelinesAggregationConfig.userInferredTopicAggregatesV2\n        ),\n        aggregateType = AggregateType.UserInferredTopic,\n        extractMapFn = _.userInferredTopicAggregates,\n        adapter = new SparseAggregatesToDenseAdapter(\n          CombineCountPolicies.UserInferredTopicV2CountsPolicy),\n        getSecondaryKeysFn = candidate =>\n          candidate.features.getOrElse(TSPInferredTopicFeature, Map.empty[Long, Double]).keys.toSeq\n      )\n\n  object UserMediaUnderstandingAnnotationAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        name = \"UserMediaUnderstandingAnnotationAggregateFeature\",\n        aggregateGroups = Set(\n          TimelinesAggregationConfig.userMediaUnderstandingAnnotationAggregates),\n        aggregateType = AggregateType.UserMediaUnderstandingAnnotation,\n        extractMapFn = _.userMediaUnderstandingAnnotationAggregates,\n        adapter = new SparseAggregatesToDenseAdapter(\n          CombineCountPolicies.UserMediaUnderstandingAnnotationCountsPolicy),\n        getSecondaryKeysFn = candidate =>\n          CandidatesUtil.getMediaUnderstandingAnnotationIds(candidate.features)\n      )\n\n  object UserEngagerAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        name = \"UserEngagerAggregateFeature\",\n        aggregateGroups = Set(TimelinesAggregationConfig.userEngagerAggregates),\n        aggregateType = AggregateType.UserEngager,\n        extractMapFn = _.userEngagerAggregates,\n        adapter = new SparseAggregatesToDenseAdapter(CombineCountPolicies.EngagerCountsPolicy),\n        getSecondaryKeysFn = candidate => CandidatesUtil.getEngagerUserIds(candidate.features)\n      )\n\n  object UserEngagerGoodClickAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        name = \"UserEngagerGoodClickAggregateFeature\",\n        aggregateGroups = Set(TimelinesAggregationConfig.userEngagerGoodClickAggregates),\n        aggregateType = AggregateType.UserEngager,\n        extractMapFn = _.userEngagerAggregates,\n        adapter = new SparseAggregatesToDenseAdapter(\n          CombineCountPolicies.EngagerGoodClickCountsPolicy),\n        getSecondaryKeysFn = candidate => CandidatesUtil.getEngagerUserIds(candidate.features)\n      )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates/PartAAggregateQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TimelineAggregateMetadataRepository\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TimelineAggregatePartARepository\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.offline_aggregates.BaseAggregateQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.offline_aggregates.BaseAggregateRootFeature\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.servo.repository.Repository\nimport com.twitter.timelines.data_processing.jobs.timeline_ranking_user_features.TimelinesPartAStoreRegister\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.StoreConfig\nimport com.twitter.timelines.suggests.common.dense_data_record.thriftscala.DenseFeatureMetadata\nimport com.twitter.user_session_store.thriftjava.UserSession\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject PartAAggregateRootFeature extends BaseAggregateRootFeature {\n  override val aggregateStores: Set[StoreConfig[_]] = TimelinesPartAStoreRegister.allStores\n}\n\n@Singleton\nclass PartAAggregateQueryFeatureHydrator @Inject() (\n  @Named(TimelineAggregatePartARepository)\n  repository: Repository[Long, Option[UserSession]],\n  @Named(TimelineAggregateMetadataRepository)\n  metadataRepository: Repository[Int, Option[DenseFeatureMetadata]])\n    extends BaseAggregateQueryFeatureHydrator(\n      repository,\n      metadataRepository,\n      PartAAggregateRootFeature\n    ) {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"PartAAggregateQuery\")\n\n  override val features = Set(PartAAggregateRootFeature)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates/PartBAggregateQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TimelineAggregateMetadataRepository\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TimelineAggregatePartBRepository\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.DataRecordMerger\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.offline_aggregates.BaseAggregateQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.offline_aggregates.BaseAggregateRootFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.repository.Repository\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.data_processing.jobs.timeline_ranking_user_features.TimelinesPartBStoreRegister\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateType\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.StoreConfig\nimport com.twitter.timelines.prediction.adapters.request_context.RequestContextAdapter\nimport com.twitter.timelines.prediction.common.aggregates.TimelinesAggregationConfig\nimport com.twitter.timelines.suggests.common.dense_data_record.thriftscala.DenseFeatureMetadata\nimport com.twitter.user_session_store.thriftjava.UserSession\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject PartBAggregateRootFeature extends BaseAggregateRootFeature {\n  override val aggregateStores: Set[StoreConfig[_]] = TimelinesPartBStoreRegister.allStores\n}\n\nobject UserAggregateFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass PartBAggregateQueryFeatureHydrator @Inject() (\n  @Named(TimelineAggregatePartBRepository)\n  repository: Repository[Long, Option[UserSession]],\n  @Named(TimelineAggregateMetadataRepository)\n  metadataRepository: Repository[Int, Option[DenseFeatureMetadata]])\n    extends BaseAggregateQueryFeatureHydrator(\n      repository,\n      metadataRepository,\n      PartBAggregateRootFeature\n    ) {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"PartBAggregateQuery\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(PartBAggregateRootFeature, UserAggregateFeature)\n\n  private val userAggregateFeatureInfo = new AggregateFeatureInfo(\n    aggregateGroups = Set(\n      TimelinesAggregationConfig.userAggregatesV2,\n      TimelinesAggregationConfig.userAggregatesV5Continuous,\n      TimelinesAggregationConfig.userAggregatesV6,\n      TimelinesAggregationConfig.twitterWideUserAggregates,\n    ),\n    aggregateType = AggregateType.User\n  )\n\n  private val userHourAggregateFeatureInfo = new AggregateFeatureInfo(\n    aggregateGroups = Set(\n      TimelinesAggregationConfig.userRequestHourAggregates,\n    ),\n    aggregateType = AggregateType.UserRequestHour\n  )\n\n  private val userDowAggregateFeatureInfo = new AggregateFeatureInfo(\n    aggregateGroups = Set(\n      TimelinesAggregationConfig.userRequestDowAggregates\n    ),\n    aggregateType = AggregateType.UserRequestDow\n  )\n\n  require(\n    userAggregateFeatureInfo.feature == PartBAggregateRootFeature,\n    \"UserAggregates feature must be provided by the PartB data source.\")\n  require(\n    userHourAggregateFeatureInfo.feature == PartBAggregateRootFeature,\n    \"UserRequstHourAggregates feature must be provided by the PartB data source.\")\n  require(\n    userDowAggregateFeatureInfo.feature == PartBAggregateRootFeature,\n    \"UserRequestDowAggregates feature must be provided by the PartB data source.\")\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    // Hydrate TimelineAggregatePartBFeature and UserAggregateFeature sequentially.\n    super.hydrate(query).map { featureMap =>\n      val time: Time = Time.now\n      val hourOfDay = RequestContextAdapter.hourFromTimestamp(time.inMilliseconds)\n      val dayOfWeek = RequestContextAdapter.dowFromTimestamp(time.inMilliseconds)\n\n      val dr = featureMap\n        .get(PartBAggregateRootFeature).map { featuresWithMetadata =>\n          val userAggregatesDr =\n            featuresWithMetadata.userAggregatesOpt\n              .map(featuresWithMetadata.toDataRecord)\n          val userRequestHourAggregatesDr =\n            Option(featuresWithMetadata.userRequestHourAggregates.get(hourOfDay))\n              .map(featuresWithMetadata.toDataRecord)\n          val userRequestDowAggregatesDr =\n            Option(featuresWithMetadata.userRequestDowAggregates.get(dayOfWeek))\n              .map(featuresWithMetadata.toDataRecord)\n\n          dropUnknownFeatures(userAggregatesDr, userAggregateFeatureInfo.featureContext)\n\n          dropUnknownFeatures(\n            userRequestHourAggregatesDr,\n            userHourAggregateFeatureInfo.featureContext)\n\n          dropUnknownFeatures(\n            userRequestDowAggregatesDr,\n            userDowAggregateFeatureInfo.featureContext)\n\n          mergeDataRecordOpts(\n            userAggregatesDr,\n            userRequestHourAggregatesDr,\n            userRequestDowAggregatesDr)\n\n        }.getOrElse(new DataRecord())\n\n      featureMap + (UserAggregateFeature, dr)\n    }\n  }\n\n  private val drMerger = new DataRecordMerger\n  private def mergeDataRecordOpts(dataRecordOpts: Option[DataRecord]*): DataRecord =\n    dataRecordOpts.flatten.foldLeft(new DataRecord) { (l, r) =>\n      drMerger.merge(l, r)\n      l\n    }\n\n  private def dropUnknownFeatures(\n    dataRecordOpt: Option[DataRecord],\n    featureContext: FeatureContext\n  ): Unit =\n    dataRecordOpt.foreach(new RichDataRecord(_, featureContext).dropUnknownFeatures())\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates/TopicEdgeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserInferredTopicAggregateFeature\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserInferredTopicAggregateV2Feature\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserTopicAggregateFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableTopicEdgeAggregateFeatureHydratorParam\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject TopicEdgeAggregateFeatureHydrator\n    extends BaseEdgeAggregateFeatureHydrator\n    with Conditionally[PipelineQuery] {\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableTopicEdgeAggregateFeatureHydratorParam)\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TopicEdgeAggregate\")\n\n  override val aggregateFeatures: Set[BaseEdgeAggregateFeature] = Set(\n    UserInferredTopicAggregateFeature,\n    UserInferredTopicAggregateV2Feature,\n    UserTopicAggregateFeature,\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates/TopicEdgeTruncatedAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserInferredTopicAggregateFeature\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserInferredTopicAggregateV2Feature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableTopicEdgeAggregateFeatureHydratorParam\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject TopicEdgeTruncatedAggregateFeatureHydrator\n    extends BaseEdgeAggregateFeatureHydrator\n    with Conditionally[PipelineQuery] {\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    !query.params(EnableTopicEdgeAggregateFeatureHydratorParam)\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TopicEdgeTruncatedAggregate\")\n\n  override val aggregateFeatures: Set[BaseEdgeAggregateFeature] = Set(\n    UserInferredTopicAggregateFeature,\n    UserInferredTopicAggregateV2Feature,\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates/TweetContentEdgeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserMediaUnderstandingAnnotationAggregateFeature\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\n\nobject TweetContentEdgeAggregateFeatureHydrator extends BaseEdgeAggregateFeatureHydrator {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetContentEdgeAggregate\")\n\n  override val aggregateFeatures: Set[BaseEdgeAggregateFeature] = Set(\n    UserMediaUnderstandingAnnotationAggregateFeature\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates/UserEngagerEdgeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserEngagerAggregateFeature\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserEngagerGoodClickAggregateFeature\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\n\nobject UserEngagerEdgeAggregateFeatureHydrator extends BaseEdgeAggregateFeatureHydrator {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserEngagerEdgeAggregate\")\n\n  override val aggregateFeatures: Set[BaseEdgeAggregateFeature] = Set(\n    UserEngagerAggregateFeature,\n    UserEngagerGoodClickAggregateFeature,\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates/UserEntityEdgeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserAuthorAggregateFeature\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserMentionAggregateFeature\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserOriginalAuthorAggregateFeature\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\n\nobject UserEntityAggregateFeatureHydrator extends BaseEdgeAggregateFeatureHydrator {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserEntityEdgeAggregate\")\n\n  override val aggregateFeatures: Set[BaseEdgeAggregateFeature] = Set(\n    UserAuthorAggregateFeature,\n    UserOriginalAuthorAggregateFeature,\n    UserMentionAggregateFeature\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates/Utils.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.suggests.common.dense_data_record.thriftjava.DenseCompactDataRecord\n\nprivate[offline_aggregates] object Utils {\n\n  /**\n   * Selects only those values in map that correspond to the keys in ids and apply the provided\n   * transform to the selected values. This is a convenience method for use by Timelines Aggregation\n   * Framework based features.\n   *\n   * @param idsToSelect The set of ids to extract values for.\n   * @param transform A transform to apply to the selected values.\n   * @param map Map[Long, DenseCompactDataRecord]\n   */\n  def selectAndTransform(\n    idsToSelect: Seq[Long],\n    transform: DenseCompactDataRecord => DataRecord,\n    map: java.util.Map[java.lang.Long, DenseCompactDataRecord],\n  ): Map[Long, DataRecord] = {\n    val filtered: Seq[(Long, DataRecord)] =\n      for {\n        id <- idsToSelect if map.containsKey(id)\n      } yield {\n        id -> transform(map.get(id))\n      }\n    filtered.toMap\n  }\n\n  def filterDataRecord(dr: DataRecord, featureContext: FeatureContext): Unit = {\n    new RichDataRecord(dr, featureContext).dropUnknownFeatures()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"home-mixer-features/thrift/src/main/thrift:thrift-java\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/module\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/real_time_aggregates\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/real_time_aggregates\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"servo/repo/src/main/scala\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api\",\n        \"src/scala/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/storehaus_internal/store\",\n        \"src/scala/com/twitter/timelines/prediction/common/aggregates/real_time:base-config\",\n        \"src/scala/com/twitter/timelines/prediction/common/aggregates/real_time/tv:base-config\",\n        \"src/thrift/com/twitter/timelines/realtime_aggregates:thrift-scala\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/real_time_aggregates\",\n        \"timelines/data_processing/ml_util/aggregation_framework/real_time_aggregates\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates/EngagementsReceivedByAuthorRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.authorIdFeature\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.keyTransformD1\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.EngagementsReceivedByAuthorCache\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer_features.{thriftjava => t}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject EngagementsReceivedByAuthorRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass EngagementsReceivedByAuthorRealTimeAggregateFeatureHydrator @Inject() (\n  override val homeMixerFeatureService: t.HomeMixerFeatures.ServiceToClient,\n  @Named(EngagementsReceivedByAuthorCache) override val client: ReadCache[Long, DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends FlagBasedRealTimeAggregateBulkCandidateFeatureHydrator[Long] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"EngagementsReceivedByAuthorRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    EngagementsReceivedByAuthorRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    authorEngagementRealTimeAggregatesProd,\n    authorShareEngagementsRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    authorShareEngagementsRealTimeAggregates -> \"original_author.timelines.author_share_engagements_real_time_aggregates.\"\n  )\n\n  def serializeKey(key: Long): String = {\n    keyTransformD1(authorIdFeature)(key)\n  }\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] =\n    candidates.map(candidate => CandidatesUtil.getOriginalAuthorId(candidate.features))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates/FlagBasedRealTimeAggregateBulkCandidateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.real_time_aggregates\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.WithDefaultFeatureMap\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableHomeMixerFeaturesService\nimport com.twitter.home_mixer.util.MissingKeyException\nimport com.twitter.home_mixer_features.thriftjava.HomeMixerFeaturesRequest\nimport com.twitter.home_mixer_features.{thriftjava => t}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.real_time_aggregates.BaseRealTimeAggregateBulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\nimport scala.jdk.CollectionConverters.iterableAsScalaIterableConverter\nimport scala.jdk.CollectionConverters.seqAsJavaListConverter\n\ntrait FlagBasedRealTimeAggregateBulkCandidateFeatureHydrator[K]\n    extends BaseRealTimeAggregateBulkCandidateFeatureHydrator[K, TweetCandidate]\n    with WithDefaultFeatureMap {\n\n  def EmptyDataRecord: DataRecord = new DataRecord()\n\n  override lazy val defaultFeatureMap: FeatureMap = FeatureMap(outputFeature, EmptyDataRecord)\n\n  def serializeKey(key: K): String\n\n  val homeMixerFeatureService: t.HomeMixerFeatures.ServiceToClient\n\n  private val batchSize = 64\n\n  private def createHomeMixerFeaturesRequest(keys: Seq[K]): t.HomeMixerFeaturesRequest = {\n    val keysSerialized = keys.map(serializeKey)\n    val request = new HomeMixerFeaturesRequest()\n    request.setKeys(keysSerialized.asJava)\n    request.setCache(t.Cache.RTA)\n  }\n\n  private def unmarshallHomeMixerFeaturesResponse(\n    response: t.HomeMixerFeaturesResponse\n  ): Iterable[DataRecord] = {\n    response.getHomeMixerFeatures.asScala\n      .map { homeMixerFeatureOpt =>\n        if (homeMixerFeatureOpt.isSetHomeMixerFeaturesType) {\n          val homeMixerFeature = homeMixerFeatureOpt.getHomeMixerFeaturesType\n          if (homeMixerFeature.isSet(t.HomeMixerFeaturesType._Fields.DATA_RECORD)) {\n            postTransformer(homeMixerFeature.getDataRecord)\n          } else {\n            throw new Exception(\"Unexpected type\")\n          }\n        } else EmptyDataRecord\n      }\n  }\n\n  private def getFeatureMaps(\n    possiblyKeys: Seq[Option[K]],\n    dataRecordMap: Map[K, DataRecord]\n  ): Future[Seq[Try[DataRecord]]] = {\n    val transformer = { key: Option[K] =>\n      if (key.nonEmpty)\n        Return(dataRecordMap.getOrElse(key.get, EmptyDataRecord))\n      else Throw(MissingKeyException)\n    }\n    OffloadFuturePools.offloadBatchElementToElement(possiblyKeys, transformer, batchSize)\n  }\n\n  def fetchRecordsFromHomeMixerFeaturesService(\n    possiblyKeys: Seq[Option[K]]\n  ): Future[Seq[Try[DataRecord]]] = {\n    val keys = possiblyKeys.flatten.distinct\n\n    val transformer = { keyGroup: Seq[K] =>\n      val request = createHomeMixerFeaturesRequest(keyGroup)\n      val responseFut =\n        homeMixerFeatureService.getHomeMixerFeatures(request)\n      responseFut\n        .map { response =>\n          keyGroup.zip(unmarshallHomeMixerFeaturesResponse(response))\n        }.handle { case _ => Seq.empty }\n    }\n\n    val response =\n      OffloadFuturePools.offloadBatchSeqToFutureSeq(keys, transformer, batchSize)\n\n    response.map(_.toMap).flatMap { keyValueResult =>\n      getFeatureMaps(possiblyKeys, keyValueResult)\n    }\n  }\n\n  def fetchRecords(\n    possiblyKeys: Seq[Option[K]],\n    callMiddleMan: Boolean\n  ): Future[Seq[Try[DataRecord]]] = {\n    if (callMiddleMan) fetchRecordsFromHomeMixerFeaturesService(possiblyKeys)\n    else fetchAndConstructDataRecords(possiblyKeys) // cache is default\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val possiblyKeys = keysFromQueryAndCandidates(query, candidates)\n    val callMiddleMan = query.params(EnableHomeMixerFeaturesService)\n    fetchRecords(possiblyKeys, callMiddleMan).map { dataRecords =>\n      val featureMaps = dataRecords.map { dataRecord =>\n        FeatureMapBuilder().add(outputFeature, dataRecord).build()\n      }\n      featureMaps\n    }\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates/TopicCountryEngagementRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.countryCodeFeature\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.keyTransformD1T1\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.topicIdFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableTopicCountryBasedRealTimeAggregateFeatureHydratorParam\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TopicCountryEngagementCache\nimport com.twitter.home_mixer_features.{thriftjava => t}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject TopicCountryEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TopicCountryEngagementRealTimeAggregateFeatureHydrator @Inject() (\n  override val homeMixerFeatureService: t.HomeMixerFeatures.ServiceToClient,\n  @Named(TopicCountryEngagementCache) override val client: ReadCache[(Long, String), DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends FlagBasedRealTimeAggregateBulkCandidateFeatureHydrator[(Long, String)]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TopicCountryEngagementRealTimeAggregate\")\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableTopicCountryBasedRealTimeAggregateFeatureHydratorParam)\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    TopicCountryEngagementRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    topicCountryRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    topicCountryRealTimeAggregates -> \"topic-country_code.timelines.topic_country_engagement_real_time_aggregates.\"\n  )\n\n  def serializeKey(key: (Long, String)): String = {\n    keyTransformD1T1(topicIdFeature, countryCodeFeature)(key)\n  }\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[(Long, String)]] = {\n    candidates.map { candidate =>\n      val maybeTopicId = candidate.features\n        .getTry(TopicIdSocialContextFeature)\n        .toOption\n        .flatten\n\n      val maybeCountryCode = query.clientContext.countryCode\n\n      for {\n        topicId <- maybeTopicId\n        countryCode <- maybeCountryCode\n      } yield (topicId, countryCode)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates/TopicEngagementRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.real_time_aggregates\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.keyTransformD1\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.topicIdFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableTopicBasedRealTimeAggregateFeatureHydratorParam\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TopicEngagementCache\nimport com.twitter.home_mixer_features.{thriftjava => t}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject TopicEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TopicEngagementRealTimeAggregateFeatureHydrator @Inject() (\n  override val homeMixerFeatureService: t.HomeMixerFeatures.ServiceToClient,\n  @Named(TopicEngagementCache) override val client: ReadCache[Long, DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends FlagBasedRealTimeAggregateBulkCandidateFeatureHydrator[Long]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TopicEngagementRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    TopicEngagementRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    topicEngagementRealTimeAggregatesProd,\n    topicEngagement24HourRealTimeAggregatesProd,\n    topicShareEngagementsRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    topicEngagement24HourRealTimeAggregatesProd -> \"topic.timelines.topic_engagement_24_hour_real_time_aggregates.\",\n    topicShareEngagementsRealTimeAggregates -> \"topic.timelines.topic_share_engagements_real_time_aggregates.\"\n  )\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableTopicBasedRealTimeAggregateFeatureHydratorParam)\n\n  def serializeKey(key: Long): String = {\n    keyTransformD1(topicIdFeature)(key)\n  }\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] = {\n    candidates.map { candidate =>\n      candidate.features\n        .getTry(TopicIdSocialContextFeature)\n        .toOption\n        .flatten\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates/TweetCountryEngagementRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.countryCodeFeature\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.keyTransformD1T1\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.tweetIdFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableTweetCountryRTAMhFallbackParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableTweetCountryRTAMhOnlyParam\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RTAManhattanStore\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetCountryEngagementCache\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer_features.{thriftjava => t}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregationKey\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport com.twitter.util.Future\nimport com.twitter.util.Try\nimport com.twitter.timelines.realtime_aggregates.{thriftscala => thrift}\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.stitch.Stitch\n\nobject TweetCountryEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TweetCountryEngagementRealTimeAggregateFeatureHydrator @Inject() (\n  override val homeMixerFeatureService: t.HomeMixerFeatures.ServiceToClient,\n  @Named(TweetCountryEngagementCache) override val client: ReadCache[(Long, String), DataRecord],\n  @Named(RTAManhattanStore) mhClient: Option[ReadableStore[thrift.AggregationKey, DataRecord]],\n  override val statsReceiver: StatsReceiver)\n    extends FlagBasedRealTimeAggregateBulkCandidateFeatureHydrator[(Long, String)] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetCountryEngagementRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    TweetCountryEngagementRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    tweetCountryRealTimeAggregates,\n    tweetCountryPrivateEngagementsRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    tweetCountryRealTimeAggregates -> \"tweet-country_code.timelines.tweet_country_engagement_real_time_aggregates.\",\n    tweetCountryPrivateEngagementsRealTimeAggregates -> \"tweet-country_code.timelines.tweet_country_private_engagement_real_time_aggregates.\"\n  )\n\n  def serializeKey(key: (Long, String)): String = {\n    keyTransformD1T1(tweetIdFeature, countryCodeFeature)(key)\n  }\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[(Long, String)]] = {\n    val countryCode = query.clientContext.countryCode\n    candidates.map { candidate =>\n      val originalTweetId = CandidatesUtil.getOriginalTweetId(candidate)\n      countryCode.map((originalTweetId, _))\n    }\n  }\n\n  def convert(key: (Long, String)): thrift.AggregationKey = {\n    val ak = AggregationKey(Map(tweetIdFeature -> key._1), Map(countryCodeFeature -> key._2))\n    thrift.AggregationKey(\n      ak.discreteFeaturesById,\n      ak.textFeaturesById\n    )\n  }\n\n  def fetchAndConstructDataRecordFromMh(\n    possiblyKeys: Seq[Option[(Long, String)]]\n  ): Future[Seq[Try[DataRecord]]] = {\n    Future\n      .collect {\n        possiblyKeys.flatten\n          .map {\n            convert(_)\n          }\n          .grouped(64).map { keyGroup =>\n            val results = mhClient.get.multiGet(keyGroup.toSet)\n            Future.collect(keyGroup.flatMap(results.get)).map { drSeq =>\n              drSeq.map { drOpt =>\n                if (drOpt.isEmpty) statsReceiver.scope(\"mhTweetCountryRTA\").counter(\"empty\").incr()\n                else statsReceiver.scope(\"mhTweetCountryRTA\").counter(\"non_empty\").incr()\n                Try(drOpt.map(postTransformer).getOrElse(EmptyDataRecord))\n              }\n            }\n          }.toSeq\n      }.map(_.flatten)\n  }\n\n  def getFetchFunc(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Future[Seq[Try[DataRecord]]] = {\n    val fetchFromMhOnly = query.params.getBoolean(EnableTweetCountryRTAMhOnlyParam)\n    val fetchFromMhAsFallBack = query.params.getBoolean(EnableTweetCountryRTAMhFallbackParam)\n    val possiblyKeys = keysFromQueryAndCandidates(query, candidates)\n    val stats = statsReceiver.scope(\"tweet_country_real_time_rta\")\n    if (fetchFromMhOnly) {\n      fetchAndConstructDataRecordFromMh(possiblyKeys)\n    } else if (fetchFromMhAsFallBack) {\n      fetchAndConstructDataRecordsWithFallback(\n        possiblyKeys,\n        stats,\n        fetchAndConstructDataRecords,\n        fetchAndConstructDataRecordFromMh,\n      )\n    } else {\n      fetchAndConstructDataRecords(possiblyKeys)\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    getFetchFunc(query, candidates).map { dataRecords =>\n      val featureMaps = dataRecords.map { dataRecord =>\n        FeatureMapBuilder().add(outputFeature, dataRecord).build()\n      }\n      featureMaps\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates/TweetEngagementRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.keyTransformD1\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.tweetIdFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableTweetRTAMhFallbackParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableTweetRTAMhOnlyParam\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RTAManhattanStore\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetEngagementCache\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer_features.{thriftjava => t}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregationKey\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport com.twitter.util.Future\nimport com.twitter.util.Try\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.realtime_aggregates.{thriftscala => thrift}\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject TweetEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TweetEngagementRealTimeAggregateFeatureHydrator @Inject() (\n  override val homeMixerFeatureService: t.HomeMixerFeatures.ServiceToClient,\n  @Named(TweetEngagementCache) override val client: ReadCache[Long, DataRecord],\n  @Named(RTAManhattanStore) mhClient: Option[ReadableStore[thrift.AggregationKey, DataRecord]],\n  override val statsReceiver: StatsReceiver)\n    extends FlagBasedRealTimeAggregateBulkCandidateFeatureHydrator[Long] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetEngagementRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    TweetEngagementRealTimeAggregateFeature\n\n  val batchSize = 64\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    tweetEngagement30MinuteCountsProd,\n    tweetEngagementTotalCountsProd,\n    tweetEngagementUserStateRealTimeAggregatesProd,\n    tweetNegativeEngagementUserStateRealTimeAggregates,\n    tweetNegativeEngagement6HourCounts,\n    tweetNegativeEngagementTotalCounts,\n    tweetShareEngagementsRealTimeAggregates,\n    tweetBCEDwellEngagementsRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    tweetShareEngagementsRealTimeAggregates -> \"original_tweet.timelines.tweet_share_engagements_real_time_aggregates.\",\n    tweetBCEDwellEngagementsRealTimeAggregates -> \"original_tweet.timelines.tweet_bce_dwell_engagements_real_time_aggregates.\"\n  )\n\n  def serializeKey(key: Long): String = {\n    keyTransformD1(tweetIdFeature)(key)\n  }\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] = {\n    val keys = candidates\n      .map(candidate => Some(CandidatesUtil.getOriginalTweetId(candidate)))\n    keys\n  }\n\n  def convert(key: Long): thrift.AggregationKey = {\n    val ak = AggregationKey(Map(tweetIdFeature -> key), Map.empty)\n    thrift.AggregationKey(\n      ak.discreteFeaturesById,\n      ak.textFeaturesById\n    )\n  }\n\n  def fetchAndConstructDataRecordFromMh(\n    possiblyKeys: Seq[Option[Long]]\n  ): Future[Seq[Try[DataRecord]]] = {\n    Future\n      .collect {\n        possiblyKeys.flatten\n          .map { convert(_) }\n          .grouped(batchSize).map { keyGroup =>\n            val results = mhClient.get.multiGet(keyGroup.toSet)\n            Future.collect(keyGroup.flatMap(results.get)).map { drSeq =>\n              drSeq.map { drOpt =>\n                if (drOpt.isEmpty) statsReceiver.scope(\"mhTweetRTA\").counter(\"empty\").incr()\n                else statsReceiver.scope(\"mhTweetRTA\").counter(\"non_empty\").incr()\n                Try(drOpt.map(postTransformer).getOrElse(EmptyDataRecord))\n              }\n            }\n          }.toSeq\n      }.map(_.flatten)\n  }\n\n  def getFetchFunc(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Future[Seq[Try[DataRecord]]] = {\n    val fetchFromMhOnly = query.params.getBoolean(EnableTweetRTAMhOnlyParam)\n    val fetchFromMhAsFallBack = query.params.getBoolean(EnableTweetRTAMhFallbackParam)\n    val possiblyKeys = keysFromQueryAndCandidates(query, candidates)\n    val stats = statsReceiver.scope(\"tweet_real_time_rta\")\n    if (fetchFromMhOnly) {\n      fetchAndConstructDataRecordFromMh(possiblyKeys)\n    } else if (fetchFromMhAsFallBack) {\n      fetchAndConstructDataRecordsWithFallback(\n        possiblyKeys,\n        stats,\n        fetchAndConstructDataRecords,\n        fetchAndConstructDataRecordFromMh,\n      )\n    } else {\n      fetchAndConstructDataRecords(possiblyKeys)\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    getFetchFunc(query, candidates).map { dataRecords =>\n      val featureMaps = dataRecords.map { dataRecord =>\n        FeatureMapBuilder().add(outputFeature, dataRecord).build()\n      }\n      featureMaps\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates/TwitterListEngagementRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.ListIdFeature\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.keyTransformD1\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.listIdFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwitterListEngagementCache\nimport com.twitter.home_mixer_features.{thriftjava => t}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject TwitterListEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwitterListEngagementRealTimeAggregateFeatureHydrator @Inject() (\n  override val homeMixerFeatureService: t.HomeMixerFeatures.ServiceToClient,\n  @Named(TwitterListEngagementCache) override val client: ReadCache[Long, DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends FlagBasedRealTimeAggregateBulkCandidateFeatureHydrator[Long] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TwitterListEngagementRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    TwitterListEngagementRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] =\n    Seq(listEngagementRealTimeAggregatesProd)\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    listEngagementRealTimeAggregatesProd -> \"twitter_list.timelines.twitter_list_engagement_real_time_aggregates.\"\n  )\n\n  def serializeKey(key: Long): String = {\n    keyTransformD1(listIdFeature)(key)\n  }\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] = candidates.map { candidate =>\n    candidate.features.getTry(ListIdFeature).toOption.flatten\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates/UserAuthorEngagementRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.authorIdFeature\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.keyTransformD2\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.keyTransformD2AggregationKey\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.userIdFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableUserAuthorRTAMhFallbackParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableUserAuthorRTAMhOnlyParam\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RTAManhattanStore\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UserAuthorEngagementCache\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer_features.{thriftjava => t}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport com.twitter.timelines.realtime_aggregates.{thriftscala => thrift}\nimport com.twitter.util.Future\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject UserAuthorEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass UserAuthorEngagementRealTimeAggregateFeatureHydrator @Inject() (\n  override val homeMixerFeatureService: t.HomeMixerFeatures.ServiceToClient,\n  @Named(UserAuthorEngagementCache) override val client: ReadCache[(Long, Long), DataRecord],\n  @Named(RTAManhattanStore) mhClient: Option[\n    ReadableStore[thrift.AggregationKey, DataRecord]\n  ],\n  override val statsReceiver: StatsReceiver)\n    extends FlagBasedRealTimeAggregateBulkCandidateFeatureHydrator[(Long, Long)] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserAuthorEngagementRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    UserAuthorEngagementRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    userAuthorEngagementRealTimeAggregatesProd,\n    userAuthorShareEngagementsRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    userAuthorEngagementRealTimeAggregatesProd -> \"user-author.timelines.user_author_engagement_real_time_aggregates.\",\n    userAuthorShareEngagementsRealTimeAggregates -> \"user-author.timelines.user_author_share_engagements_real_time_aggregates.\"\n  )\n\n  def serializeKey(key: (Long, Long)): String = {\n    keyTransformD2(userIdFeature, authorIdFeature)(key)\n  }\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[(Long, Long)]] = {\n    val userId = query.getRequiredUserId\n    candidates.map { candidate =>\n      CandidatesUtil\n        .getOriginalAuthorId(candidate.features)\n        .map((userId, _))\n    }\n  }\n\n  def convert(key: (Long, Long)): thrift.AggregationKey = {\n    val ak = keyTransformD2AggregationKey(userIdFeature, authorIdFeature)((key._1, key._2))\n    thrift.AggregationKey(\n      ak.discreteFeaturesById,\n      ak.textFeaturesById\n    )\n  }\n\n  def fetchAndConstructDataRecordFromMh(\n    possiblyKeys: Seq[Option[(Long, Long)]]\n  ): Future[Seq[Try[DataRecord]]] = {\n    Future\n      .collect {\n        possiblyKeys.flatten\n          .map {\n            convert(_)\n          }\n          .grouped(64).map { keyGroup =>\n            val results = mhClient.get.multiGet(keyGroup.toSet)\n            Future.collect(keyGroup.flatMap(results.get)).map { drSeq =>\n              drSeq.map { drOpt =>\n                if (drOpt.isEmpty) statsReceiver.scope(\"mhUserAuthorRTA\").counter(\"empty\").incr()\n                else statsReceiver.scope(\"mhUserAuthorRTA\").counter(\"non_empty\").incr()\n                Try(drOpt.map(postTransformer).getOrElse(EmptyDataRecord))\n              }\n            }\n          }.toSeq\n      }.map(_.flatten)\n  }\n\n  def getFetchFunc(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Future[Seq[Try[DataRecord]]] = {\n    val fetchFromMhOnly = query.params.getBoolean(EnableUserAuthorRTAMhOnlyParam)\n    val fetchFromMhAsFallBack = query.params.getBoolean(EnableUserAuthorRTAMhFallbackParam)\n    val possiblyKeys = keysFromQueryAndCandidates(query, candidates)\n    val stats = statsReceiver.scope(\"user_author_real_time_rta\")\n    if (fetchFromMhOnly) {\n      fetchAndConstructDataRecordFromMh(possiblyKeys)\n    } else if (fetchFromMhAsFallBack) {\n      fetchAndConstructDataRecordsWithFallback(\n        possiblyKeys,\n        stats,\n        fetchAndConstructDataRecords,\n        fetchAndConstructDataRecordFromMh,\n      )\n    } else {\n      fetchAndConstructDataRecords(possiblyKeys)\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    getFetchFunc(query, candidates).map { dataRecords =>\n      val featureMaps = dataRecords.map { dataRecord =>\n        FeatureMapBuilder().add(outputFeature, dataRecord).build()\n      }\n      featureMaps\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates/UserEngagementRealTimeAggregatesFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.module.RealtimeAggregateFeatureRepositoryModule.userIdFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableUserRTAMhFallbackParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableUserRTAMhOnlyParam\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RTAManhattanStore\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UserEngagementCache\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.real_time_aggregates.BaseRealTimeAggregateQueryFeatureHydrator\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregationKey\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport com.twitter.util.Future\nimport com.twitter.util.Try\nimport com.twitter.timelines.realtime_aggregates.{thriftscala => thrift}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject UserEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass UserEngagementRealTimeAggregatesFeatureHydrator @Inject() (\n  @Named(UserEngagementCache) override val client: ReadCache[Long, DataRecord],\n  @Named(RTAManhattanStore) mhClient: Option[ReadableStore[thrift.AggregationKey, DataRecord]],\n  override val statsReceiver: StatsReceiver)\n    extends BaseRealTimeAggregateQueryFeatureHydrator[Long] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserEngagementRealTimeAggregates\")\n\n  override val outputFeature: DataRecordInAFeature[PipelineQuery] =\n    UserEngagementRealTimeAggregateFeature\n\n  val aggregateGroups: Seq[AggregateGroup] = Seq(\n    userEngagementRealTimeAggregatesProd,\n    userShareEngagementsRealTimeAggregates,\n    userBCEDwellEngagementsRealTimeAggregates,\n    userEngagement48HourRealTimeAggregatesProd,\n    userNegativeEngagementAuthorUserState72HourRealTimeAggregates,\n    userNegativeEngagementAuthorUserStateRealTimeAggregates,\n    userProfileEngagementRealTimeAggregates,\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    userShareEngagementsRealTimeAggregates -> \"user.timelines.user_share_engagements_real_time_aggregates.\",\n    userBCEDwellEngagementsRealTimeAggregates -> \"user.timelines.user_bce_dwell_engagements_real_time_aggregates.\",\n    userEngagement48HourRealTimeAggregatesProd -> \"user.timelines.user_engagement_48_hour_real_time_aggregates.\",\n    userNegativeEngagementAuthorUserState72HourRealTimeAggregates -> \"user.timelines.user_negative_engagement_author_user_state_72_hour_real_time_aggregates.\",\n    userProfileEngagementRealTimeAggregates -> \"user.timelines.user_profile_engagement_real_time_aggregates.\"\n  )\n\n  override def keysFromQueryAndCandidates(query: PipelineQuery): Option[Long] = {\n    Some(query.getRequiredUserId)\n  }\n\n  val EmptyDataRecord = new DataRecord\n\n  override def fetchAndConstructDataRecords(\n    possiblyKeys: Seq[Option[Long]]\n  ): Future[Seq[Try[DataRecord]]] = {\n    val keys = possiblyKeys.flatten.map { k =>\n      val ak = AggregationKey(Map(userIdFeature -> k), Map.empty)\n      thrift.AggregationKey(\n        ak.discreteFeaturesById,\n        ak.textFeaturesById\n      )\n    }\n    Future\n      .collect {\n        keys.map { key =>\n          statsReceiver.scope(\"mhUserRTA\").counter(\"mh_called\").incr()\n          val result = mhClient.get.get(key)\n          result.map { drOpt =>\n            if (drOpt.isEmpty) statsReceiver.scope(\"mhUserRTA\").counter(\"empty\").incr()\n            else statsReceiver.scope(\"mhUserRTA\").counter(\"non_empty\").incr()\n            Try(drOpt.map(postTransformer).getOrElse(EmptyDataRecord))\n          }\n        }\n      }\n  }\n\n  def convert(key: Long): thrift.AggregationKey = {\n    val ak = AggregationKey(Map(userIdFeature -> key), Map.empty)\n    thrift.AggregationKey(\n      ak.discreteFeaturesById,\n      ak.textFeaturesById\n    )\n  }\n\n  def fetchAndConstructDataRecordFromMh(\n    possiblyKeys: Seq[Option[Long]]\n  ): Future[Seq[Try[DataRecord]]] = {\n    Future\n      .collect {\n        possiblyKeys.flatten\n          .map {\n            convert(_)\n          }\n          .grouped(64).map { keyGroup =>\n            val results = mhClient.get.multiGet(keyGroup.toSet)\n            Future.collect(keyGroup.flatMap(results.get)).map { drSeq =>\n              drSeq.map { drOpt =>\n                if (drOpt.isEmpty) statsReceiver.scope(\"mhUserRTA\").counter(\"empty\").incr()\n                else statsReceiver.scope(\"mhUserRTA\").counter(\"non_empty\").incr()\n                Try(drOpt.map(postTransformer).getOrElse(EmptyDataRecord))\n              }\n            }\n          }.toSeq\n      }.map(_.flatten)\n  }\n\n  def getFetchFunc(\n    query: PipelineQuery\n  ): Future[Seq[Try[DataRecord]]] = {\n    val fetchFromMhOnly = query.params.getBoolean(EnableUserRTAMhOnlyParam)\n    val fetchFromMhAsFallBack = query.params.getBoolean(EnableUserRTAMhFallbackParam)\n    val possiblyKeys = Seq(keysFromQueryAndCandidates(query))\n    val stats = statsReceiver.scope(\"user_author_real_time_rta\")\n    if (fetchFromMhOnly) {\n      fetchAndConstructDataRecordFromMh(possiblyKeys)\n    } else if (fetchFromMhAsFallBack) {\n      fetchAndConstructDataRecordsWithFallback(\n        possiblyKeys,\n        stats,\n        fetchAndConstructDataRecords,\n        fetchAndConstructDataRecordFromMh,\n      )\n    } else {\n      fetchAndConstructDataRecords(possiblyKeys)\n    }\n  }\n\n  override def hydrate(\n    query: PipelineQuery\n  ): Stitch[FeatureMap] = OffloadFuturePools.offloadFuture {\n    getFetchFunc(query).map { dataRecords =>\n      val featureMaps = dataRecords.map { dataRecord =>\n        FeatureMapBuilder().add(outputFeature, dataRecord).build()\n      }\n      featureMaps.head\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates/UserTweetTvVideoRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.real_time_aggregates\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TvVideoByUserTweetCache\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.real_time_aggregates.BaseRealTimeAggregateBulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.metrics.AggregateFeature\nimport com.twitter.timelines.prediction.common.aggregates.real_time.tv.TvOnlineAggregationFeaturesOnlyConfig\nimport com.twitter.util.Return\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject DummyFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = {\n    new DataRecord()\n  }\n}\n\n// The feature value is keyed by the RTA aggregation group prefix, then the AggregateFeature\n// for ease of access.\nobject UserTweetTvVideoRealtimeAggregatesFeatures\n    extends Feature[TweetCandidate, Option[Map[String, Map[AggregateFeature[_], Double]]]]\n\n@Singleton\nclass UserTweetTvVideoRealTimeAggregateFeatureHydrator @Inject() (\n  @Named(TvVideoByUserTweetCache) override val client: ReadCache[(Long, Long), DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends BaseRealTimeAggregateBulkCandidateFeatureHydrator[(Long, Long), TweetCandidate]\n    with Conditionally[ScoredTweetsQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserTweetTvVideoRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] = DummyFeature\n\n  override def onlyIf(\n    query: ScoredTweetsQuery\n  ): Boolean = query.videoType.contains(hmt.VideoType.LongForm)\n\n  override def features: Set[Feature[_, _]] = Set(\n    UserTweetTvVideoRealtimeAggregatesFeatures\n  )\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    TvOnlineAggregationFeaturesOnlyConfig.userTweetViewRealTimeAggregates,\n    TvOnlineAggregationFeaturesOnlyConfig.userTweetPlaybackRealTimeAggregates,\n    TvOnlineAggregationFeaturesOnlyConfig.userTweetImpressionRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map.empty\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[(Long, Long)]] = {\n    candidates.map(candidate =>\n      Some((query.getRequiredUserId, CandidatesUtil.getOriginalTweetId(candidate))))\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    fetchValues(keysFromQueryAndCandidates(query, candidates)).map { featureToAggValueMaps =>\n      featureToAggValueMaps.map { featureToAggValueMap =>\n        val value = featureToAggValueMap match {\n          case Return(r) => Some(r)\n          case _ => None\n        }\n        FeatureMapBuilder()\n          .add(UserTweetTvVideoRealtimeAggregatesFeatures, value)\n          .build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/user_history/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"configapi/configapi-decider\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/transformer_embeddings\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/module\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"media-understanding/embeddings/src/main/thrift/com/twitter/media-understanding/embeddings:thrift-scala\",\n        \"servo/repo/src/main/scala\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"strato/config/columns/content_understanding:content_understanding-strato-client\",\n        \"strato/config/columns/tweetypie/managed:managed-strato-client\",\n        \"strato/config/columns/user-history-transformer/unhydrated-user-history:unhydrated-user-history-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"user_history_transformer/common/src/main/scala/com/twitter/user_history_transformer/util\",\n        \"user_history_transformer/thrift/src/main/thrift/com/twitter/user_history_transformer:user_history_transformer-scala\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/real_time_aggregates\",\n        \"timelines/data_processing/ml_util/aggregation_framework/real_time_aggregates\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/user_history/BaseUserHistoryEventsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.user_history\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.BaseUserHistoryEventsAdapter\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.user_history_transformer.thriftscala.UnhydratedUserHistory\nimport com.twitter.user_history_transformer.thriftscala.UserHistory\nimport com.twitter.user_history_transformer.util.UserHistoryCompressionUtils\nimport scala.collection.JavaConverters._\n\nobject UserHistoryEventsFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\ntrait BaseUserHistoryEventsQueryFeatureHydrator extends QueryFeatureHydrator[PipelineQuery] {\n  private val DefaultDataRecord = new DataRecord()\n\n  override def features: Set[Feature[_, _]] = Set(UserHistoryEventsFeature)\n\n  def historyFetcher: Fetcher[Long, Unit, UnhydratedUserHistory]\n  def filterEvents(query: PipelineQuery, historyEvents: Seq[UserHistory]): Stitch[Seq[UserHistory]]\n  def adapter: BaseUserHistoryEventsAdapter\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    historyFetcher.fetch(query.getRequiredUserId).flatMap { result =>\n      result.v match {\n        case Some(unhydratedUserHistory: UnhydratedUserHistory) =>\n          val expandedValue: UnhydratedUserHistory =\n            UserHistoryCompressionUtils.expand(value = unhydratedUserHistory)\n          filterEvents(query, expandedValue.events)\n            .map { filteredEvents: Seq[UserHistory] =>\n              val record =\n                if (filteredEvents.nonEmpty)\n                  adapter\n                    .adaptToDataRecords(filteredEvents)\n                    .asScala\n                    .headOption\n                    .getOrElse(DefaultDataRecord)\n                else\n                  DefaultDataRecord\n              FeatureMap(UserHistoryEventsFeature, record)\n            }\n        case None =>\n          Stitch.value(FeatureMap(UserHistoryEventsFeature, DefaultDataRecord))\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/user_history/ScoredTweetsUserHistoryEventsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.user_history\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.BaseUserHistoryEventsAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.UserHistoryEventsAdapter\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.UserHistoryEventsLengthParam\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.user_history_transformer.unhydrated_user_history.UnhydratedUserHistoryMHProdClientColumn\nimport com.twitter.user_history_transformer.thriftscala.SourceType\nimport com.twitter.user_history_transformer.thriftscala.UnhydratedUserHistory\nimport com.twitter.user_history_transformer.thriftscala.UserHistory\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class ScoredTweetsUserHistoryEventsQueryFeatureHydrator @Inject() (\n  unhydratedUserHistoryMHProdClientColumn: UnhydratedUserHistoryMHProdClientColumn)\n    extends BaseUserHistoryEventsQueryFeatureHydrator {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"ScoredTweetsUserHistoryEvents\")\n\n  override val historyFetcher: Fetcher[Long, Unit, UnhydratedUserHistory] =\n    unhydratedUserHistoryMHProdClientColumn.fetcher\n\n  private val allowedSourceTypes: Set[SourceType] = Set(SourceType.Home, SourceType.Uua)\n  override val adapter: BaseUserHistoryEventsAdapter = UserHistoryEventsAdapter\n  override def filterEvents(\n    query: PipelineQuery,\n    historyEvents: Seq[UserHistory]\n  ): Stitch[Seq[UserHistory]] = {\n    val filteredEvents = historyEvents.filter { event =>\n      allowedSourceTypes.contains(event.metadata.flatMap(_.source).getOrElse(SourceType.Uua))\n    }\n    Stitch.value(filteredEvents.takeRight(query.params(UserHistoryEventsLengthParam)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/user_history/ScoredVideoTweetsUserHistoryEventsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.functional_component.feature_hydrator.user_history\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.BaseUserHistoryEventsAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.VideoUserHistoryEventsAdapter\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableScoredVideoTweetsUserHistoryEventsQueryFeatureHydrationDeciderParam\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MediaClipClusterIdInMemCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MediaClusterId95Store\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.UserHistoryEventsLengthParam\nimport com.twitter.mediaservices.commons.thriftscala.MediaCategory\nimport com.twitter.mediaservices.commons.tweetmedia.thriftscala.MediaInfo\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.InProcessCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.columns.content_understanding.thriftscala.EntityWithMetadata\nimport com.twitter.strato.columns.content_understanding.thriftscala.TagWithMetadata\nimport com.twitter.strato.generated.client.content_understanding.DeserializedVideoAnnotationClientColumn\nimport com.twitter.strato.generated.client.tweetypie.managed.ImmersiveExploreClientEventsJobTesOnTweetClientColumn\nimport com.twitter.strato.generated.client.user_history_transformer.unhydrated_user_history.UnhydratedUserHistoryMHProdClientColumn\nimport com.twitter.unified_user_actions.thriftscala.ActionType\nimport com.twitter.user_history_transformer.thriftscala.SourceType\nimport com.twitter.user_history_transformer.thriftscala.UnhydratedUserHistory\nimport com.twitter.user_history_transformer.thriftscala.UserHistory\nimport com.twitter.user_history_transformer.thriftscala.UserHistoryMetadata\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.tweetypie.{thriftscala => tp}\nimport com.twitter.strato.catalog.Fetch\nimport javax.inject.Named\nimport com.twitter.storehaus.ReadableStore\n\n@Singleton\ncase class ScoredVideoTweetsUserHistoryEventsQueryFeatureHydrator @Inject() (\n  immersiveExploreClientEventsJobTesOnTweetClientColumn: ImmersiveExploreClientEventsJobTesOnTweetClientColumn,\n  unhydratedUserHistoryMHProdClientColumn: UnhydratedUserHistoryMHProdClientColumn,\n  deserializedVideoAnnotationClientColumn: DeserializedVideoAnnotationClientColumn,\n  @Named(MediaClusterId95Store) clusterIdStore: ReadableStore[Long, Long],\n  @Named(MediaClipClusterIdInMemCache) mediaClipClusterIdInMemCache: InProcessCache[\n    Long,\n    Option[Option[Long]]\n  ],\n  statsReceiver: StatsReceiver)\n    extends BaseUserHistoryEventsQueryFeatureHydrator\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"ScoredVideoTweetsUserHistoryEvents\")\n\n  override def onlyIf(\n    query: PipelineQuery\n  ): Boolean =\n    query.params(EnableScoredVideoTweetsUserHistoryEventsQueryFeatureHydrationDeciderParam)\n\n  override val historyFetcher: Fetcher[Long, Unit, UnhydratedUserHistory] =\n    unhydratedUserHistoryMHProdClientColumn.fetcher\n\n  val tesFetcher: Fetcher[Long, tp.GetTweetFieldsOptions, tp.GetTweetFieldsResult] =\n    immersiveExploreClientEventsJobTesOnTweetClientColumn.fetcher\n\n  val TweetypieContentHydrationFields: Set[tp.TweetInclude] = Set(\n    tp.TweetInclude.TweetFieldId(tp.Tweet.CoreDataField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.IdField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.MediaField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.MediaKeysField.id)\n  )\n\n  private val scopedStatsReceiver: StatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val immersiveClientActionTweetIdsCounter =\n    scopedStatsReceiver.counter(\"immersiveClientActionTweetIdsCounter\")\n  private val validMediaInfoCounter = scopedStatsReceiver.counter(\"validMediaInfoCounter\")\n  private val tesFetchCounter = statsReceiver.counter(\"tes_fetch_total\")\n  private val tesFetchSuccessCounter = statsReceiver.counter(\"tes_fetch_success\")\n  private val tesFetchFailureCounter = statsReceiver.counter(\"tes_fetch_failure\")\n  private val combinedTweetIdsCounter = scopedStatsReceiver.counter(\"combinedTweetIdsCounter\")\n  private val inValidMediaInfoCounter = scopedStatsReceiver.counter(\"inValidMediaInfoCounter\")\n  private val inMemCacheHitCounter = scopedStatsReceiver.counter(\"cache/hit\")\n  private val inMemCacheMissCounter = scopedStatsReceiver.counter(\"cache/miss\")\n\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  private val keyFailureCounter = scopedStatsReceiver.counter(\"key/failure\")\n\n  private val clusterIdFoundCounter = scopedStatsReceiver.counter(\"clusterIdFoundCounter\")\n  private val clusterIdNotFoundCounter = scopedStatsReceiver.counter(\"clusterIdNotFoundCounter\")\n  private val clusterIdFailureCounter = scopedStatsReceiver.counter(\"clusterIdFailureCountere\")\n\n  override val adapter: BaseUserHistoryEventsAdapter = VideoUserHistoryEventsAdapter\n\n  override def filterEvents(\n    query: PipelineQuery,\n    historyEvents: Seq[UserHistory]\n  ): Stitch[Seq[UserHistory]] = {\n\n    val immersiveActions = query match {\n      case query: PipelineQuery with ScoredTweetsQuery =>\n        query.immersiveClientMetadata\n          .map(metadata => metadata.immersiveClientActions)\n          .getOrElse(Seq.empty)\n          .filter { action => action.tweetId.isDefined && action.watchTimeMs.isDefined }\n      case _ =>\n        Seq.empty\n    }\n\n    val immersiveActionMap: Map[Long, Int] = immersiveActions.map { action =>\n      action.tweetId.get -> action.watchTimeMs.get\n    }.toMap\n\n    val finalHistoryEvents =\n      historyEvents\n        .filter { event =>\n          (event.actionType == ActionType.ClientTweetVideoWatchTime ||\n          event.actionType == ActionType.ClientTweetVideoHeartbeat) &&\n          !immersiveActionMap.contains(event.tweetId)\n        }\n        .takeRight(query.params(UserHistoryEventsLengthParam))\n\n    val combinedTweetIds =\n      (finalHistoryEvents.map(_.tweetId) ++ immersiveActions.map(_.tweetId.get)).distinct\n    combinedTweetIdsCounter.incr(combinedTweetIds.size)\n\n    val fetchedData: Stitch[Map[Long, tp.GetTweetFieldsResult]] =\n      Stitch\n        .traverse(combinedTweetIds) { tweetId =>\n          tesFetchCounter.incr()\n          tesFetcher\n            .fetch(\n              tweetId,\n              tp.GetTweetFieldsOptions(tweetIncludes = TweetypieContentHydrationFields)\n            )\n            .map {\n              case Fetch.Result(Some(resolvedResult), _) =>\n                tesFetchSuccessCounter.incr()\n                (tweetId, resolvedResult)\n              case _ =>\n                tesFetchFailureCounter.incr()\n                (\n                  tweetId,\n                  tp.GetTweetFieldsResult(\n                    tweetId,\n                    tp.TweetFieldsResultState.NotFound(tp.TweetFieldsResultNotFound())\n                  )\n                )\n            }\n        }.map(_.toMap)\n\n    val grokMetadataFetched: Stitch[\n      Map[Long, (Option[Seq[TagWithMetadata]], Option[Seq[EntityWithMetadata]])]\n    ] =\n      Stitch\n        .traverse(combinedTweetIds) { tweetId =>\n          deserializedVideoAnnotationClientColumn.fetcher\n            .fetch(tweetId, Unit)\n            .map {\n              case Fetch.Result(response, _) =>\n                if (response.nonEmpty) keyFoundCounter.incr()\n                else keyNotFoundCounter.incr()\n                (\n                  tweetId,\n                  (response.flatMap(_.tags), response.flatMap(_.entities))\n                )\n              case _ =>\n                keyFailureCounter.incr()\n                (\n                  tweetId,\n                  (None: Option[List[TagWithMetadata]], None: Option[List[EntityWithMetadata]])\n                )\n            }\n            .handle {\n              case _ =>\n                keyFailureCounter.incr()\n                (\n                  tweetId,\n                  (None: Option[List[TagWithMetadata]], None: Option[List[EntityWithMetadata]])\n                )\n            }\n        }.map(_.toMap)\n\n    val cluster95IdsFetched: Stitch[Map[Long, Long]] =\n      Stitch\n        .traverse(combinedTweetIds) { tweetId =>\n          getFromCacheOrFetch(tweetId)\n            .map {\n              case Some(Some(clusterId)) =>\n                clusterIdFoundCounter.incr()\n                (tweetId, clusterId)\n              case Some(None) | None =>\n                clusterIdNotFoundCounter.incr()\n                (tweetId, -1L)\n            }\n            .handle {\n              case _ =>\n                clusterIdFailureCounter.incr()\n                (tweetId, -1L)\n            }\n        }.map(_.toMap)\n\n    for {\n      tweetResults <- fetchedData\n      grokMetadata <- grokMetadataFetched\n      clusterIds <- cluster95IdsFetched\n    } yield {\n      val tweetToMediaInfoMap = tweetResults.foldLeft(\n        Map.empty[Long, (Long, Option[MediaCategory], Option[Int], Option[Long], Long)]\n      ) {\n        case (mediaInfoMap, (tweetId, tweetResult)) =>\n          tweetResult.tweetResult match {\n            case tp.TweetFieldsResultState.Found(found) =>\n              val media = found.tweet.media.flatMap(_.headOption)\n              val authorId = found.tweet.coreData.map(_.userId)\n              val mediaId = media.map(_.mediaId)\n              val mediaCategory = media.flatMap(_.mediaKey).map(_.mediaCategory)\n              val videoDuration = media.flatMap(_.mediaInfo match {\n                case Some(MediaInfo.VideoInfo(videoInfo)) => Some(videoInfo.durationMillis)\n                case _ => None\n              })\n              val clusterId = clusterIds.getOrElse(tweetId, -1L)\n\n              mediaId match {\n                case Some(id) =>\n                  mediaInfoMap + (tweetId -> (\n                    (\n                      id,\n                      mediaCategory,\n                      videoDuration,\n                      authorId,\n                      clusterId\n                    )\n                  ))\n                case None => mediaInfoMap\n              }\n            case _ =>\n              mediaInfoMap\n          }\n      }\n\n      val immersiveActionsUserHistory = immersiveActions.map { action =>\n        val tweetId = action.tweetId.get\n        val (tagsOpt, entitiesOpt) = grokMetadata.getOrElse(tweetId, (None, None))\n        val mediaInfo = tweetToMediaInfoMap.get(tweetId)\n        val finalTags = tagsOpt.getOrElse(List.empty[TagWithMetadata])\n        val finalEntities = entitiesOpt.getOrElse(List.empty[EntityWithMetadata])\n        val finalAuthorId: Option[Long] = mediaInfo.flatMap(_._4)\n        val finalClusterId: Long = mediaInfo.map(_._5).getOrElse(-1L)\n\n        val (finalMediaId, finalMediaCategory, finalVideoDuration) = mediaInfo match {\n          case Some((mediaId, mediaCategory, videoDuration, _, _)) =>\n            validMediaInfoCounter.incr()\n            (Some(mediaId), mediaCategory, videoDuration)\n          case None =>\n            inValidMediaInfoCounter.incr()\n            (Some(-1L), Some(MediaCategory.TweetVideo), Some(-1))\n        }\n\n        UserHistory(\n          userId = query.getRequiredUserId,\n          tweetId = tweetId,\n          authorId = finalAuthorId,\n          actionType = ActionType.ClientTweetVideoWatchTime,\n          engagedTimestampMs = query.queryTime.inMilliseconds,\n          textTokens = None,\n          sourceTweetId = Some(tweetId),\n          metadata = Some(\n            UserHistoryMetadata(\n              source = Some(SourceType.ImmersiveVideo),\n              watchTime = Some(action.watchTimeMs.get.toDouble),\n              mediaId = finalMediaId,\n              mediaCategory = finalMediaCategory,\n              videoDuration = finalVideoDuration,\n              tags = Some(finalTags),\n              entities = Some(finalEntities),\n              clusterId = Some(finalClusterId)\n            )\n          )\n        )\n      }\n\n      immersiveClientActionTweetIdsCounter.incr(immersiveActionsUserHistory.size)\n\n      (finalHistoryEvents ++ immersiveActionsUserHistory).map { event =>\n        val tweetId = event.tweetId\n        val (tagsOpt, entitiesOpt) = grokMetadata.getOrElse(tweetId, (None, None))\n        val mediaInfo = tweetToMediaInfoMap.get(tweetId)\n        val finalTags = tagsOpt.getOrElse(List.empty[TagWithMetadata])\n        val finalEntities = entitiesOpt.getOrElse(List.empty[EntityWithMetadata])\n        val finalAuthorId: Option[Long] = mediaInfo.flatMap(_._4)\n        val finalClusterId: Long = mediaInfo.map(_._5).getOrElse(-1L)\n\n        val (finalMediaId, finalMediaCategory, finalVideoDuration) = mediaInfo match {\n          case Some((mediaId, mediaCategory, videoDuration, _, _)) =>\n            validMediaInfoCounter.incr()\n            (Some(mediaId), mediaCategory, videoDuration)\n          case None =>\n            inValidMediaInfoCounter.incr()\n            (Some(-1L), Some(MediaCategory.TweetVideo), Some(-1))\n        }\n\n        event.copy(\n          authorId = finalAuthorId.orElse(event.authorId),\n          metadata = event.metadata.map { metadata =>\n            metadata.copy(\n              mediaId = finalMediaId,\n              mediaCategory = finalMediaCategory,\n              videoDuration = finalVideoDuration,\n              tags = Some(finalTags),\n              entities = Some(finalEntities),\n              clusterId = Some(finalClusterId)\n            )\n          }\n        )\n      }\n    }\n  }\n\n  private def getFromCacheOrFetch(tweetId: Long): Stitch[Option[Option[Long]]] = {\n    mediaClipClusterIdInMemCache\n      .get(tweetId)\n      .map { cachedValue =>\n        inMemCacheHitCounter.incr()\n        Stitch.value(cachedValue)\n      }.getOrElse {\n        inMemCacheMissCounter.incr()\n        Stitch\n          .callFuture(clusterIdStore.get(tweetId))\n          .map { result =>\n            val finalResult = Some(result)\n            mediaClipClusterIdInMemCache.set(tweetId, finalResult)\n            finalResult\n          }\n          .handle {\n            case _ =>\n              None\n          }\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/AuthorDedupFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Keep only 1 tweet per author\n */\nobject AuthorDedupFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"AuthorDedup\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val keptIds = candidates\n      .groupBy { candidate =>\n        candidate.features.getOrElse(AuthorIdFeature, None).getOrElse(0L)\n      }.map {\n        case (_, candidates) => candidates.head.candidate.id\n      }.toSet\n\n    val (kept, removed) = candidates.partition(c => keptIds.contains(c.candidate.id))\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/transformer_embeddings\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/user_history\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/location\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"src/thrift/com/twitter/timelines/impression_bloom_filter:thrift-scala\",\n        \"stitch/stitch-socialgraph\",\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence\",\n        \"timelines/src/main/scala/com/twitter/timelines/impressionstore/impressionbloomfilter\",\n        \"timelineservice/common/src/main/scala/com/twitter/timelineservice/model\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n    exports = [\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/ClipClusterDeduplicationFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.ClipImageClusterIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaClusterIdsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\ntrait ClipClusterDeduplicationFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  val clusterIdFeature: Feature[TweetCandidate, Map[Long, Long]]\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (removedCandidateIds, _) = candidates.foldLeft((Set.empty[Long], Set.empty[Long])) {\n      case ((removedIds, seenClusterIds), candidate) =>\n        val clusterIds = candidate.features\n          .getOrElse(clusterIdFeature, Map.empty[Long, Long])\n          .values\n          .toSet\n\n        if (clusterIds.size == 1) {\n          val clusterId = clusterIds.head\n          if (seenClusterIds.contains(clusterId)) {\n            (removedIds + candidate.candidate.id, seenClusterIds)\n          } else {\n            (removedIds, seenClusterIds + clusterId)\n          }\n        } else {\n          (removedIds, seenClusterIds)\n        }\n    }\n\n    val (removed, kept) = candidates\n      .map(_.candidate)\n      .partition(c => removedCandidateIds.contains(c.id))\n\n    Stitch.value(FilterResult(kept = kept, removed = removed))\n  }\n}\n\nobject ClipImageClusterDeduplicationFilter extends ClipClusterDeduplicationFilter {\n  override val identifier: FilterIdentifier = FilterIdentifier(\"ClipImageClusterDeduplication\")\n  override val clusterIdFeature: Feature[TweetCandidate, Map[Long, Long]] =\n    ClipImageClusterIdsFeature\n}\n\nobject ClipVideoClusterDeduplicationFilter extends ClipClusterDeduplicationFilter {\n  override val identifier: FilterIdentifier = FilterIdentifier(\"ClipVideoClusterDeduplication\")\n  override val clusterIdFeature: Feature[TweetCandidate, Map[Long, Long]] =\n    TweetMediaClusterIdsFeature\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/ClusterBasedDedupFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.UserHistoryEventsFeatures\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.transformer_embeddings.VideoUserHistoryEventsFeatures\nimport com.twitter.home_mixer.functional_component.feature_hydrator.user_history.UserHistoryEventsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DedupClusterId88Feature\nimport com.twitter.home_mixer.model.HomeFeatures.DedupClusterIdFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.DedupHistoricalEventsTimeWindowParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableClusterBased88DedupFilter\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableClusterBasedDedupFilter\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableNoClusterFilter\nimport scala.collection.convert.ImplicitConversions._\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nobject ClusterBasedDedupFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"ClusterBasedDedup\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = {\n    query.params(EnableClusterBasedDedupFilter)\n  }\n\n  private def getViewedClusterIds(query: PipelineQuery): Set[Long] = {\n    val currentTimeMs = System.currentTimeMillis()\n    val dedupHistoricalEventsTimeWindow = query.params(DedupHistoricalEventsTimeWindowParam)\n\n    val dataRecord = query.features.get\n      .getOrElse(UserHistoryEventsFeature, new DataRecord())\n\n    val clusterIds = dataRecord.getTensors match {\n      case tensors if tensors != null =>\n        val clusterIdsTensor =\n          tensors.get(VideoUserHistoryEventsFeatures.VideoCluster95IdsFeature.getFeatureId)\n        val actionTimestampTensor =\n          tensors.get(UserHistoryEventsFeatures.ActionTimestampsFeature.getFeatureId)\n\n        if (clusterIdsTensor != null && clusterIdsTensor.isSet() &&\n          actionTimestampTensor != null && actionTimestampTensor.isSet()) {\n          val actionTimestampsBuffer = actionTimestampTensor.getInt64Tensor.longs\n            .map(_.longValue())\n          val clusterIdsBuffer = clusterIdsTensor.getInt64Tensor.longs.map(_.longValue())\n\n          clusterIdsBuffer\n            .zip(actionTimestampsBuffer)\n            .collect {\n              case (clusterId, timestamp)\n                  if timestamp > 0 && (currentTimeMs - timestamp) <= dedupHistoricalEventsTimeWindow =>\n                clusterId\n            }\n            .toSet\n        } else {\n          Set.empty[Long]\n        }\n      case _ =>\n        Set.empty[Long]\n    }\n    clusterIds\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val historicalClusters = getViewedClusterIds(query) // Still based on 0.95 threshold\n    val enable88Dedup = query.params(EnableClusterBased88DedupFilter)\n    val enableNoClusterFilter = query.params(EnableNoClusterFilter)\n\n    val (kept, removed) = candidates.foldLeft(\n      (\n        Seq[CandidateWithFeatures[TweetCandidate]](), // kept accumulator\n        Seq[CandidateWithFeatures[TweetCandidate]](), // removed accumulator\n        Set[Long](), // seen cluster IDs for 0.95 threshold\n        Set[Long]() // seen cluster IDs for 0.88 threshold\n      )\n    ) {\n      case ((keptAcc, removedAcc, seenCluster95Ids, seenCluster88Ids), candidate) =>\n        val clusterId95 = candidate.features\n          .getOrElse(DedupClusterIdFeature, None) // 0.95 threshold\n        val clusterId88 = candidate.features\n          .getOrElse(DedupClusterId88Feature, None) // 0.88 threshold\n\n        (clusterId95, clusterId88) match {\n          case (Some(c95), _) if historicalClusters.contains(c95) =>\n            // Historical match at 0.95 threshold, remove it\n            (keptAcc, removedAcc :+ candidate, seenCluster95Ids, seenCluster88Ids)\n          case (Some(c95), _) if seenCluster95Ids.contains(c95) =>\n            // Duplicate at 0.95 threshold (non-historical), remove it\n            (keptAcc, removedAcc :+ candidate, seenCluster95Ids, seenCluster88Ids)\n          case (Some(c95), Some(c88)) if enable88Dedup && seenCluster88Ids.contains(c88) =>\n            // Unique at 0.95 but duplicate at 0.88 threshold, remove it\n            (keptAcc, removedAcc :+ candidate, seenCluster95Ids + c95, seenCluster88Ids)\n          case (Some(c95), Some(c88)) =>\n            // First occurrence at both levels, keep it\n            val updatedSeen88Ids = if (enable88Dedup) seenCluster88Ids + c88 else seenCluster88Ids\n            (keptAcc :+ candidate, removedAcc, seenCluster95Ids + c95, updatedSeen88Ids)\n          case (Some(c95), None) =>\n            // Only 0.95 cluster ID, keep it if not seen\n            (keptAcc :+ candidate, removedAcc, seenCluster95Ids + c95, seenCluster88Ids)\n          case (None, Some(c88)) if enable88Dedup && seenCluster88Ids.contains(c88) =>\n            // Only 0.88 cluster ID, remove if duplicate\n            (keptAcc, removedAcc :+ candidate, seenCluster95Ids, seenCluster88Ids)\n          case (None, Some(c88)) =>\n            // Only 0.88 cluster ID, keep if not seen\n            val updatedSeen88Ids = if (enable88Dedup) seenCluster88Ids + c88 else seenCluster88Ids\n            (keptAcc :+ candidate, removedAcc, seenCluster95Ids, updatedSeen88Ids)\n          case (None, None) =>\n            // No cluster IDs, keep or remove based on enableNoClusterFilter\n            if (enableNoClusterFilter) {\n              (keptAcc, removedAcc :+ candidate, seenCluster95Ids, seenCluster88Ids) // Remove it\n            } else {\n              (keptAcc :+ candidate, removedAcc, seenCluster95Ids, seenCluster88Ids) // Keep it\n            }\n        }\n    } match {\n      case (keptCandidates, removedCandidates, _, _) =>\n        (keptCandidates, removedCandidates)\n    }\n\n    Stitch.value(\n      FilterResult(\n        kept = kept.map(_.candidate),\n        removed = removed.map(_.candidate)\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/ConsistentAspectRatioFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.VideoAspectRatioFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * Filter for making sure all video posts in a carousel has consistent aspect ratio\n */\ncase class ConsistentAspectRatioFilter(\n  allowVerticalVideosParam: Param[Boolean],\n  allowHorizontalVideosParam: Param[Boolean])\n    extends Filter[PipelineQuery, TweetCandidate] {\n\n  /** @see [[FilterIdentifier]] */\n  override val identifier: FilterIdentifier = FilterIdentifier(\"ConsistentAspectRatio\")\n\n  /**\n   * Filter the list of candidates\n   *\n   * @return a FilterResult including both the list of kept candidate and the list of removed candidates\n   */\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (hasAspectRatio, doesNotHaveAspectRatio) = candidates\n      .partition(_.features.getOrElse(VideoAspectRatioFeature, None).nonEmpty)\n    if (hasAspectRatio.nonEmpty) {\n      val isHorizontal =\n        (query.params(allowVerticalVideosParam), query.params(allowHorizontalVideosParam)) match {\n          case (false, _) => false\n          case (_, false) => true\n          case _ => // If both video types are allowed, take the first video's aspect ratio\n            val aspectRatioFirstVideo =\n              hasAspectRatio.head.features.getOrElse(VideoAspectRatioFeature, None).get\n            aspectRatioFirstVideo > 1.0\n        }\n      val (consistentAspectRatio, differentAspectRatio) = hasAspectRatio.partition { candidate =>\n        candidate.features.getOrElse(VideoAspectRatioFeature, None).get > 1.0 == isHorizontal\n      }\n      Stitch.value(\n        FilterResult(\n          kept = consistentAspectRatio.map(_.candidate),\n          removed = ((differentAspectRatio ++ doesNotHaveAspectRatio)).map(_.candidate)))\n    } else {\n      Stitch.value(\n        FilterResult(kept = Seq.empty, removed = doesNotHaveAspectRatio.map(_.candidate)))\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/CountryFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableCountryFilter\n\n/**\n * Filters tweets based on matching country location between user query and tweet candidates\n */\nobject CountryFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"Country\")\n\n  /**\n   * Determines if the filter should be applied based on the EnableCountryFilter parameter\n   */\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = {\n    query.params(EnableCountryFilter)\n  }\n\n  /**\n   * Filters candidates based on exact country location matching\n   * @param query The pipeline query containing user location features\n   * @param candidates The sequence of tweet candidates with their features\n   * @return A Stitch containing FilterResult with kept and removed candidates\n   */\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val userCountryOpt = query.features\n      .map(_.getOrElse(LocationFeature, None))\n      .flatMap(_.flatMap(_.country))\n\n    val (kept, removed) = candidates.partition { candidate =>\n      val postLocationOpt = candidate.features.getOrElse(LocationFeature, None).flatMap(_.country)\n      val keep = (postLocationOpt, userCountryOpt) match {\n        case (Some(postLocation), Some(userCountry)) =>\n          postLocation == userCountry\n        case (Some(_), None) =>\n          false\n        case (None, _) =>\n          true\n        case _ =>\n          true\n      }\n      keep\n    }\n\n    Stitch.value(\n      FilterResult(\n        kept = kept.map(_.candidate),\n        removed = removed.map(_.candidate)\n      ))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/CurrentPinnedTweetFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.CurrentPinnedTweetFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Keep only the currently pinned tweet per author\n */\nobject CurrentPinnedTweetFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"CurrentPinnedTweet\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val (kept, removed) = candidates.partition { candidate =>\n      val currentPinnedTweet = candidate.features.getOrElse(CurrentPinnedTweetFeature, None)\n      currentPinnedTweet.contains(candidate.candidate.id)\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/DropMaxCandidatesFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.FSBoundedParam\n\ncase class DropMaxCandidatesFilter[Candidate <: UniversalNoun[Any]](\n  maxCandidatesParam: FSBoundedParam[Int])\n    extends Filter[PipelineQuery, Candidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"DropMaxCandidates\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n    val maxCandidates = query.params(maxCandidatesParam)\n    val (kept, removed) = candidates.map(_.candidate).splitAt(maxCandidates)\n\n    Stitch.value(FilterResult(kept, removed))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/FeedbackFatigueFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FeedbackHistoryFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeedbackFatigueFilteringDurationParam\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.common.thriftscala.FeedbackEntity\nimport com.twitter.timelineservice.model.FeedbackEntry\nimport com.twitter.timelineservice.{thriftscala => tls}\n\nobject FeedbackFatigueFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"FeedbackFatigue\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean =\n    query.features.exists(_.getOrElse(FeedbackHistoryFeature, Seq.empty).nonEmpty)\n\n  override def apply(\n    query: pipeline.PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val feedbackEntriesByEngagementType =\n      query.features\n        .getOrElse(FeatureMap.empty).getOrElse(FeedbackHistoryFeature, Seq.empty)\n        .filter { entry =>\n          val timeSinceFeedback = query.queryTime.minus(entry.timestamp)\n          timeSinceFeedback < query.params(FeedbackFatigueFilteringDurationParam) &&\n          entry.feedbackType == tls.FeedbackType.SeeFewer\n        }.groupBy(_.engagementType)\n\n    val authorsToFilter =\n      getUserIds(\n        feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Tweet, Seq.empty))\n    val likersToFilter =\n      getUserIds(\n        feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Like, Seq.empty))\n    val followersToFilter =\n      getUserIds(\n        feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Follow, Seq.empty))\n    val retweetersToFilter =\n      getUserIds(\n        feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Retweet, Seq.empty))\n\n    val (removed, kept) = candidates.partition { candidate =>\n      val originalAuthorId = CandidatesUtil.getOriginalAuthorId(candidate.features)\n      val authorId = candidate.features.getOrElse(AuthorIdFeature, None)\n\n      val likers = candidate.features.getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty)\n      val eligibleLikers = likers.filterNot(likersToFilter.contains)\n\n      val followers = candidate.features.getOrElse(SGSValidFollowedByUserIdsFeature, Seq.empty)\n      val eligibleFollowers = followers.filterNot(followersToFilter.contains)\n\n      originalAuthorId.exists(authorsToFilter.contains) ||\n      (likers.nonEmpty && eligibleLikers.isEmpty) ||\n      (followers.nonEmpty && eligibleFollowers.isEmpty && likers.isEmpty) ||\n      (candidate.features.getOrElse(IsRetweetFeature, false) &&\n      authorId.exists(retweetersToFilter.contains))\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n\n  private def getUserIds(\n    feedbackEntries: Seq[FeedbackEntry],\n  ): Set[Long] =\n    feedbackEntries.collect {\n      case FeedbackEntry(_, _, FeedbackEntity.UserId(userId), _, _, _) => userId\n    }.toSet\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/GrokGoreFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableGrokGoreFilter\nimport com.twitter.home_mixer.model.HomeFeatures.GrokAnnotationsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Filter out gore tweets based on grok annotations\n */\nobject GrokGoreFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"GrokGore\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = query.params(EnableGrokGoreFilter)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val (removed, kept) = candidates.partition { candidate =>\n      val annotations = candidate.features.getOrElse(GrokAnnotationsFeature, None)\n      annotations.flatMap(_.metadata.map(_.isGore)).getOrElse(false)\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/GrokNsfwFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableNsfwFilter\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableSoftNsfwFilter\nimport com.twitter.home_mixer.model.HomeFeatures.GrokAnnotationsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Filter out NSFW tweets based on grok annotations\n */\nobject GrokNsfwFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"GrokNsfw\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = query.params(EnableNsfwFilter)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val (removed, kept) = candidates.partition { candidate =>\n      val annotations = candidate.features.getOrElse(GrokAnnotationsFeature, None)\n      val isNsfw = annotations.flatMap(_.metadata.map(_.isNsfw)).getOrElse(false)\n      val isSoftNsfw = annotations.flatMap(_.metadata.map(_.isSoftNsfw)).getOrElse(false)\n\n      if (query.params(EnableSoftNsfwFilter)) {\n        isNsfw || isSoftNsfw\n      } else {\n        isNsfw\n      }\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/GrokSpamFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableGrokSpamFilter\nimport com.twitter.home_mixer.model.HomeFeatures.GrokAnnotationsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Filter out spam tweets based on grok annotations\n */\nobject GrokSpamFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"GrokSpam\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = query.params(EnableGrokSpamFilter)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val (removed, kept) = candidates.partition { candidate =>\n      val annotations = candidate.features.getOrElse(GrokAnnotationsFeature, None)\n      annotations.flatMap(_.metadata.map(_.isSpam)).getOrElse(false)\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/GrokViolentFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableGrokViolentFilter\nimport com.twitter.home_mixer.model.HomeFeatures.GrokAnnotationsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Filter out violent tweets based on grok annotations\n */\nobject GrokViolentFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"GrokViolent\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = query.params(EnableGrokViolentFilter)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val (removed, kept) = candidates.partition { candidate =>\n      val annotations = candidate.features.getOrElse(GrokAnnotationsFeature, None)\n      annotations.flatMap(_.metadata.map(_.isViolent)).getOrElse(false)\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/HasAuthorFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject HasAuthorFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"HasAuthor\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (kept, removed) = candidates.partition { candidate =>\n      candidate.features.getOrElse(AuthorIdFeature, None).isDefined\n    }\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/HasMultipleMediaFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.HasMultipleMedia\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableHasMultipleMediaFilter\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject HasMultipleMediaFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"HasMultipleMedia\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = {\n    query.params(EnableHasMultipleMediaFilter)\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (removed, kept) = candidates.partition { candidate =>\n      candidate.features.getOrElse(HasMultipleMedia, false)\n    }\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/InvalidConversationModuleFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Exclude conversation modules where Tweets have been dropped by other filters\n *\n * Largest conversation modules have 3 Tweets, so if all 3 are present, module is valid.\n * For 2 Tweet modules, check if the head is the root (not a reply) and the last item\n * is actually replying to the root directly with no missing intermediate tweets\n */\nobject InvalidConversationModuleFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"InvalidConversationModule\")\n\n  val ValidThreeTweetModuleSize = 3\n  val ValidTwoTweetModuleSize = 2\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val allowedTweetIds = candidates\n      .groupBy(_.features.getOrElse(ConversationModuleFocalTweetIdFeature, None))\n      .map { case (id, candidates) => (id, candidates.sortBy(_.candidate.id)) }\n      .filter {\n        case (Some(_), conversation) if conversation.size == ValidThreeTweetModuleSize => true\n        case (Some(focalId), conversation) if conversation.size == ValidTwoTweetModuleSize =>\n          conversation.head.features.getOrElse(InReplyToTweetIdFeature, None).isEmpty &&\n            conversation.last.candidate.id == focalId &&\n            conversation.last.features\n              .getOrElse(InReplyToTweetIdFeature, None)\n              .contains(conversation.head.candidate.id)\n        case (None, _) => true\n        case _ => false\n      }.values.flatten.toSeq.map(_.candidate.id).toSet\n\n    val (kept, removed) =\n      candidates.map(_.candidate).partition(candidate => allowedTweetIds.contains(candidate.id))\n    Stitch.value(FilterResult(kept = kept, removed = removed))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/InvalidSubscriptionTweetFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.home_mixer.model.HomeFeatures.ExclusiveConversationAuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidatePipelines\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.socialgraph.{thriftscala => sg}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.SocialGraph\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.immutable.ListSet\n\n/**\n * Exclude invalid subscription tweets - cases where the viewer is not subscribed to the author\n *\n * If SGS hydration fails, `SGSInvalidSubscriptionTweetFeature` will be set to None for\n * subscription tweets, so we explicitly filter those tweets out.\n */\n@Singleton\ncase class InvalidSubscriptionTweetFilter @Inject() (\n  socialGraphClient: SocialGraph,\n  statsReceiver: StatsReceiver)\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Logging {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"InvalidSubscriptionTweet\")\n\n  private val scopedStatsReceiver = statsReceiver.scope(identifier.toString)\n  private val servedTypeStatsReceiver = scopedStatsReceiver.scope(\"ServedType\")\n  private val validCounter = scopedStatsReceiver.counter(\"validExclusiveTweet\")\n  private val invalidCounter = scopedStatsReceiver.counter(\"invalidExclusiveTweet\")\n\n  private val forYouScoredTweetsCandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouScoredTweets\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = Stitch\n    .traverse(candidates) { candidate =>\n      val exclusiveAuthorId =\n        candidate.features.getOrElse(ExclusiveConversationAuthorIdFeature, None)\n\n      if (exclusiveAuthorId.isDefined) {\n        val request = sg.ExistsRequest(\n          source = query.getRequiredUserId,\n          target = exclusiveAuthorId.get,\n          relationships =\n            Seq(sg.Relationship(sg.RelationshipType.TierOneSuperFollowing, hasRelationship = true)),\n        )\n        socialGraphClient.exists(request).map(_.exists).map { valid =>\n          if (!valid) {\n            invalidCounter.incr()\n            val candidatePipelines = candidate.features\n              .getOrElse(CandidatePipelines, ListSet.empty[CandidatePipelineIdentifier])\n            // Temporary debugging code\n            if (candidatePipelines.contains(forYouScoredTweetsCandidatePipelineIdentifier)) {\n              val servedType =\n                candidate.features.getOrElse(ServedTypeFeature, hmt.ServedType.Undefined)\n              servedTypeStatsReceiver.counter(servedType.name).incr()\n              logger.info(\n                s\"Removing subscription Tweet ${candidate.candidate.id} \" +\n                  s\"for User ${query.getRequiredUserId} from Author $exclusiveAuthorId \" +\n                  s\"with traceId: ${Trace.id.traceId.toLong}\"\n              )\n            }\n          } else validCounter.incr()\n          valid\n        }\n      } else Stitch.value(true)\n    }.map { validResults =>\n      val (kept, removed) = candidates\n        .map(_.candidate)\n        .zip(validResults)\n        .partition { case (candidate, valid) => valid }\n\n      val keptCandidates = kept.map { case (candidate, _) => candidate }\n      val removedCandidates = removed.map { case (candidate, _) => candidate }\n\n      FilterResult(kept = keptCandidates, removed = removedCandidates)\n    }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/LocationFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationFeature\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject LocationFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"Location\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val userLocationOpt = query.features.get.getOrElse(LocationFeature, None)\n\n    val (kept, removed) = candidates.partition { candidate =>\n      val postLocationOpt = candidate.features.getOrElse(LocationIdFeature, None)\n      (postLocationOpt, userLocationOpt) match {\n        case (Some(postLocation), Some(userLocation)) => userLocation.matches(postLocation)\n        case (Some(postLocation), None) => false\n        case _ => true\n      }\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/MaxVideoDurationFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoDurationMsFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableMaxVideoDurationFilter\nimport com.twitter.home_mixer.param.HomeGlobalParams.MaxVideoDurationThresholdParam\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Filter out tweets based on video duration\n */\nobject MaxVideoDurationFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"MaxVideoDuration\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = query.params(EnableMaxVideoDurationFilter)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val maxVideoDurationThresh = query.params(MaxVideoDurationThresholdParam)\n    val (kept, removed) = candidates.partition { candidate =>\n      val hasVideoFeature = candidate.features.getOrElse(HasVideoFeature, false)\n      val videoDuration = candidate.features.getOrElse(VideoDurationMsFeature, None).getOrElse(0)\n\n      hasVideoFeature && (videoDuration <= maxVideoDurationThresh)\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/MediaDeduplicationFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaIdsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Remove duplicate media in the same payload\n */\nobject MediaIdDeduplicationFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"MediaDeduplication\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (removedCandidateIds, _) = candidates.foldLeft((Set.empty[Long], Set.empty[Long])) {\n      case ((removedIds, seenMediaIds), candidate) =>\n        val mediaIds = candidate.features.getOrElse(TweetMediaIdsFeature, Seq.empty[Long])\n        if (mediaIds.size == 1) {\n          val mediaId = mediaIds.head\n          if (seenMediaIds.contains(mediaId)) {\n            (removedIds + candidate.candidate.id, seenMediaIds)\n          } else {\n            (removedIds, seenMediaIds + mediaId)\n          }\n        } else {\n          (removedIds, seenMediaIds)\n        }\n    }\n\n    val (removed, kept) = candidates.map(_.candidate)\n      .partition(c => removedCandidateIds.contains(c.id))\n    Stitch.value(FilterResult(kept = kept, removed = removed))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/MinVideoDurationFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaCompletionRateFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoDurationMsFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableMinVideoDurationFilter\nimport com.twitter.home_mixer.param.HomeGlobalParams.MinVideoDurationThresholdParam\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Filter out tweets based on video duration\n */\nobject MinVideoDurationFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"MinVideoDuration\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = query.params(EnableMinVideoDurationFilter)\n\n  private val CompletionRateThreshold = 90\n  private val FavThreshold = 30\n  private val LongDuration = 5000\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val minVideoDurationThresh = query.params(MinVideoDurationThresholdParam)\n    val (removed, kept) = candidates.partition { candidate =>\n      val hasVideoFeature = candidate.features.getOrElse(HasVideoFeature, false)\n      val completionRate =\n        candidate.features.getOrElse(TweetMediaCompletionRateFeature, None).getOrElse(0.0)\n      val videoDuration = candidate.features.getOrElse(VideoDurationMsFeature, None).getOrElse(0)\n      val ebFeatures = candidate.features.getOrElse(EarlybirdFeature, None)\n      val favCount = ebFeatures.flatMap(_.favCountV2)\n\n      val sketchyCompletionRate =\n        favCount.forall(_ > FavThreshold) &&\n          completionRate > CompletionRateThreshold &&\n          videoDuration > LongDuration\n\n      val lowDuration = videoDuration <= minVideoDurationThresh\n\n      hasVideoFeature && (lowDuration || sketchyCompletionRate)\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/PredicateGatedFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\ntrait FilterPredicate[-Query <: PipelineQuery] {\n  def apply(query: Query): Boolean\n}\n\n/**\n * A [[Filter]] with [[Conditionally]] based on a [[FilterPredicate]]\n *\n * @param predicate the predicate to turn this filter on and off\n * @param filter the underlying filter to run when `predicate` is true\n * @tparam Query The domain model for the query or request\n * @tparam Candidate The type of the candidates\n */\ncase class PredicateGatedFilter[-Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n  predicate: FilterPredicate[Query],\n  filter: Filter[Query, Candidate])\n    extends Filter[Query, Candidate]\n    with Filter.Conditionally[Query, Candidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\n    PredicateGatedFilter.IdentifierPrefix + filter.identifier.name)\n\n  override val alerts: Seq[Alert] = filter.alerts\n\n  override def onlyIf(query: Query, candidates: Seq[CandidateWithFeatures[Candidate]]): Boolean =\n    Conditionally.and(Filter.Input(query, candidates), filter, predicate(query))\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = filter.apply(query, candidates)\n}\n\nobject PredicateGatedFilter {\n  val IdentifierPrefix = \"PredicateGated\"\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/PreviouslySeenMediaIdsFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.ImpressedMediaIds\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaIdsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Filter out users' previously seen mediaIds\n */\nobject PreviouslySeenMediaIdsFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"PreviouslySeenMediaIds\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val seenMediaIds =\n      query.features.map(_.getOrElse(ImpressedMediaIds, Seq.empty)).toSet.flatten\n\n    val (removed, kept) = candidates.partition { candidate =>\n      candidate.features.getOrElse(TweetMediaIdsFeature, Seq.empty).exists(seenMediaIds.contains)\n    }\n\n    val filterResult = FilterResult(\n      kept = kept.map(_.candidate),\n      removed = removed.map(_.candidate)\n    )\n\n    Stitch.value(filterResult)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/PreviouslySeenTweetsFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.ImpressionBloomFilterFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserRecentEngagementTweetIdsFeature\nimport com.twitter.home_mixer.model.request.HasSeenTweetIds\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.impressionstore.impressionbloomfilter.ImpressionBloomFilterItem\n\n/**\n * Filter out users' previously seen tweets from Impression Bloom Filter\n */\nobject PreviouslySeenTweetsFilter\n    extends Filter[PipelineQuery with HasSeenTweetIds, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"PreviouslySeenTweets\")\n\n  override def apply(\n    query: PipelineQuery with HasSeenTweetIds,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val bloomFilterSeq = query.features.map(_.get(ImpressionBloomFilterFeature)).get\n    val bloomFilters =\n      bloomFilterSeq.entries.map(ImpressionBloomFilterItem.fromThrift(_).bloomFilter)\n\n    val seenTweetIds = query.seenTweetIds.getOrElse(Seq.empty).toSet\n    val engagedTweetIds =\n      query.features\n        .map(_.getOrElse(UserRecentEngagementTweetIdsFeature, Seq.empty).toSet)\n        .getOrElse(Set.empty)\n\n    val (removed, kept) = candidates.partition { candidate =>\n      CandidatesUtil.getTweetIdAndSourceId(candidate).exists { tweetId =>\n        seenTweetIds.contains(tweetId) || engagedTweetIds.contains(tweetId) ||\n        bloomFilters.exists(filter => filter.mayContain(tweetId))\n      }\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/PreviouslyServedAncestorsFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.common_internal.analytics.twitter_client_user_agent_parser.UserAgent\nimport com.twitter.home_mixer.model.HomeFeatures.IsAncestorCandidateFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelinemixer.injection.store.persistence.TimelinePersistenceUtils\nimport com.twitter.timelines.util.client_info.ClientPlatform\n\nobject PreviouslyServedAncestorsFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with TimelinePersistenceUtils {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"PreviouslyServedAncestors\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val clientPlatform = ClientPlatform.fromQueryOptions(\n      clientAppId = query.clientContext.appId,\n      userAgent = query.clientContext.userAgent.flatMap(UserAgent.fromString))\n    val entries =\n      query.features.map(_.getOrElse(PersistenceEntriesFeature, Seq.empty)).toSeq.flatten\n    val tweetIds = applicableResponses(clientPlatform, entries)\n      .flatMap(_.entries.flatMap(_.tweetIds(includeSourceTweets = true))).toSet\n    val ancestorIds =\n      candidates\n        .filter(_.features.getOrElse(IsAncestorCandidateFeature, false)).map(_.candidate.id).toSet\n\n    val (removed, kept) =\n      candidates\n        .map(_.candidate).partition(candidate =>\n          tweetIds.contains(candidate.id) && ancestorIds.contains(candidate.id))\n\n    Stitch.value(FilterResult(kept = kept, removed = removed))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/PreviouslyServedTweetPreviewsFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTweetPreviewIdsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject PreviouslyServedTweetPreviewsFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"PreviouslyServedTweetPreviews\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val servedTweetPreviewIds =\n      query.features.map(_.getOrElse(ServedTweetPreviewIdsFeature, Seq.empty)).toSeq.flatten.toSet\n\n    val (removed, kept) = candidates.partition { candidate =>\n      servedTweetPreviewIds.contains(candidate.candidate.id)\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/PreviouslyServedTweetsFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.GetOlderFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTweetIdsFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableServedFilterAllRequests\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject PreviouslyServedTweetsFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"PreviouslyServedTweets\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = {\n    query.features.exists(_.getOrElse(GetOlderFeature, false)) ||\n    query.params(EnableServedFilterAllRequests)\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val servedTweetIds =\n      query.features.map(_.getOrElse(ServedTweetIdsFeature, Seq.empty)).toSeq.flatten.toSet\n\n    val (removed, kept) = candidates.partition { candidate =>\n      val tweetIdAndSourceId = CandidatesUtil.getTweetIdAndSourceId(candidate)\n      tweetIdAndSourceId.exists(servedTweetIds.contains)\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/QuoteDeduplicationFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Remove base tweet if it is quoted in another candidate\n */\nobject QuoteDeduplicationFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"QuoteDeduplication\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val quotedTweets = candidates.flatMap(_.features.getOrElse(QuotedTweetIdFeature, None)).toSet\n\n    val (removed, kept) =\n      candidates.map(_.candidate).partition(candidate => quotedTweets.contains(candidate.id))\n\n    Stitch.value(FilterResult(kept = kept, removed = removed))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/RegionFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableRegionFilter\n\n/**\n * Filters tweets based on matching region location between user query and tweet candidates\n */\nobject RegionFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"Region\")\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = {\n    query.params(EnableRegionFilter)\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val userRegionOpt = query.features\n      .map(_.getOrElse(LocationFeature, None))\n      .flatMap(_.flatMap(_.region))\n\n    val (kept, removed) = candidates.partition { candidate =>\n      val postLocationOpt = candidate.features.getOrElse(LocationFeature, None).flatMap(_.region)\n      val keep = (postLocationOpt, userRegionOpt) match {\n        case (Some(postLocation), Some(userRegion)) =>\n          postLocation == userRegion\n        case (Some(_), None) =>\n          false\n        case (None, _) =>\n          true\n        case _ =>\n          true\n      }\n      keep\n    }\n\n    Stitch.value(\n      FilterResult(\n        kept = kept.map(_.candidate),\n        removed = removed.map(_.candidate)\n      ))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/RejectTweetFromViewerFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject RejectTweetFromViewerFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"RejectTweetFromViewer\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (removed, kept) = candidates.partition(candidate =>\n      CandidatesUtil.isAuthoredByViewer(query, candidate.features))\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/ReplyFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject ReplyFilter extends Filter[PipelineQuery, TweetCandidate] {\n  override val identifier: FilterIdentifier = FilterIdentifier(\"Reply\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (kept, removed) = candidates.partition { candidate =>\n      candidate.features.getOrElse(InReplyToTweetIdFeature, None).isEmpty\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/RetweetDeduplicationFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport scala.collection.mutable\n\nobject RetweetDeduplicationFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"RetweetDeduplication\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    // If there are 2 retweets of the same native tweet, we will choose the first one\n    // The tweets are returned in descending score order, so we will choose the higher scored tweet\n    val dedupedTweetIdsSet =\n      candidates.partition(_.features.getOrElse(IsRetweetFeature, false)) match {\n        case (retweets, nativeTweets) =>\n          val nativeTweetIds = nativeTweets.map(_.candidate.id)\n          val seenTweetIds = mutable.Set[Long]() ++ nativeTweetIds\n          val dedupedRetweets = retweets.filter { retweet =>\n            val tweetIdAndSourceId = CandidatesUtil.getTweetIdAndSourceId(retweet)\n            val retweetIsUnique = tweetIdAndSourceId.forall(!seenTweetIds.contains(_))\n            if (retweetIsUnique) {\n              seenTweetIds ++= tweetIdAndSourceId\n            }\n            retweetIsUnique\n          }\n          (nativeTweets ++ dedupedRetweets).map(_.candidate.id).toSet\n      }\n\n    val (kept, removed) =\n      candidates.map(_.candidate).partition(candidate => dedupedTweetIdsSet.contains(candidate.id))\n\n    Stitch.value(FilterResult(kept = kept, removed = removed))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/RetweetFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject RetweetFilter extends Filter[PipelineQuery, TweetCandidate] {\n  override val identifier: FilterIdentifier = FilterIdentifier(\"Retweet\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val (kept, removed) = candidates\n      .partition { candidate =>\n        !candidate.features.getOrElse(IsRetweetFeature, false)\n      }\n\n    val filterResult = FilterResult(\n      kept = kept.map(_.candidate),\n      removed = removed.map(_.candidate)\n    )\n\n    Stitch.value(filterResult)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/SlopFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.core_workflows.user_model.{thriftscala => um}\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorFollowersFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SlopAuthorFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserStateFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableSlopFilter\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableSlopFilterEligibleUserStateParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableSlopFilterLowSignalUsers\nimport com.twitter.home_mixer.param.HomeGlobalParams.SlopMinFollowers\nimport com.twitter.home_mixer.util.SignalUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject SlopFilter\n    extends Filter[PipelineQuery, TweetCandidate]\n    with Filter.Conditionally[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"Slop\")\n\n  private val MinFollowingThreshold = 5\n  private val EligibleUserStates: Set[um.UserState] =\n    Set(um.UserState.NearZero, um.UserState.New, um.UserState.VeryLight)\n\n  override def onlyIf(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = {\n    val numSlopAuthorsFollowed = candidates\n      .filter { candidate =>\n        candidate.features.getOrElse(SlopAuthorFeature, false) &&\n        candidate.features.getOrElse(InNetworkFeature, false)\n      }.flatMap(_.features.getOrElse(AuthorIdFeature, None)).distinct.size\n\n    val userState = query.features.flatMap(_.getOrElse(UserStateFeature, None))\n\n    val lowSignalUser = SignalUtil.isLowSignalUser(query)\n\n    numSlopAuthorsFollowed < MinFollowingThreshold &&\n    (\n      (userState.forall(EligibleUserStates.contains) &&\n      query.params(EnableSlopFilterEligibleUserStateParam)) ||\n      (lowSignalUser && query.params(EnableSlopFilterLowSignalUsers)) ||\n      query.params(EnableSlopFilter)\n    )\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val minFollowersThreshold = query.params(SlopMinFollowers)\n\n    val (removed, kept) = candidates.partition { candidate =>\n      candidate.features.getOrElse(SlopAuthorFeature, false) &&\n      !candidate.features.getOrElse(InNetworkFeature, false) &&\n      candidate.features.getOrElse(AuthorFollowersFeature, None).forall(_ > minFollowersThreshold)\n    }\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/TweetHydrationFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Filter out tweets that fail Tweetypie VF hydration\n */\nobject TweetHydrationFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"TweetHydration\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (kept, removed) = candidates.partition { candidate =>\n      candidate.features.getOrElse(IsHydratedFeature, false)\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter/WeeklyBookmarkFilter.scala",
    "content": "package com.twitter.home_mixer.functional_component.filter\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.HomeFeatures.BookmarkedTweetTimestamp\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Time\n\nobject WeeklyBookmarkFilter extends Filter[PipelineQuery, TweetCandidate] {\n  override val identifier: FilterIdentifier = FilterIdentifier(\"WeeklyBookmark\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (keptCandidates, removedCandidates) = candidates.partition { filterCandidate =>\n      filterCandidate.features\n        .get(BookmarkedTweetTimestamp).exists { timestamp =>\n          val aWeekAgo = Time.now - 7.days\n          Time.fromMilliseconds(timestamp) >= aWeekAgo\n        }\n    }\n\n    Stitch.value(\n      FilterResult(\n        kept = keptCandidates.map(_.candidate),\n        removed = removedCandidates.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/AllowForYouRecommendationsGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerAllowsForYouRecommendationsFeature\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * This gate disables out of network candidate pipelines when the AllowForYouRecommendations\n * user preference is set to false.\n * Defaults to true if the preference is not set.\n */\nobject AllowForYouRecommendationsGate extends Gate[PipelineQuery] {\n  override val identifier: GateIdentifier = GateIdentifier(\"AllowForYouRecommendations\")\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val allowForYouRecommendations = query.features\n      .flatMap(_.getOrElse(ViewerAllowsForYouRecommendationsFeature, Some(true))).getOrElse(\n        true\n      )\n    Stitch.value(allowForYouRecommendations)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/param\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module:test-user-mapper\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence\",\n        \"timelineservice/common/src/main/scala/com/twitter/timelineservice/model\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/BookmarksTimeGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.home_mixer.model.HomeFeatures.HasDarkRequestFeature\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableBookmarksModuleWeekendGate\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport java.time.DayOfWeek\nimport java.time.ZonedDateTime\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass BookmarksTimeGate @Inject() (serviceIdentifier: ServiceIdentifier)\n    extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"BookmarksTime\")\n\n  // Serve on the weekends, with 48 hours injection time since persistence store is only 2 days. This way we have 2\n  // chances to show the module both on Saturday and Sunday\n  private def isWeekend(zonedDateTime: ZonedDateTime) = {\n    zonedDateTime.getDayOfWeek match {\n      case DayOfWeek.SATURDAY => true\n      case DayOfWeek.SUNDAY => true\n      case _ => false\n    }\n\n  }\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val gateDisabled = !query.params(EnableBookmarksModuleWeekendGate)\n    val isDevel = serviceIdentifier.environment.toLowerCase != \"prod\"\n    val isDarkRequest = query.features.flatMap { _.get(HasDarkRequestFeature) }.getOrElse(false)\n    Stitch.value(\n      gateDisabled || isDarkRequest || isDevel || isWeekend(query.queryTime.toZonedDateTime))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/DismissFatigueGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelinemixer.clients.manhattan.DismissInfo\nimport com.twitter.timelineservice.suggests.thriftscala.SuggestType\nimport com.twitter.util.Duration\n\nobject DismissFatigueGate {\n  // how long a dismiss action from user needs to be respected\n  val DefaultBaseDismissDuration = 7.days\n  val MaximumDismissalCountMultiplier = 4\n}\n\ncase class DismissFatigueGate(\n  suggestType: SuggestType,\n  dismissInfoFeature: Feature[PipelineQuery, Map[SuggestType, Option[DismissInfo]]],\n  baseDismissDuration: Duration = DismissFatigueGate.DefaultBaseDismissDuration,\n) extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"DismissFatigue\")\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val dismissInfoMap = query.features.map(\n      _.getOrElse(dismissInfoFeature, Map.empty[SuggestType, Option[DismissInfo]]))\n\n    val isVisible = dismissInfoMap\n      .flatMap(_.get(suggestType))\n      .flatMap(_.map { info =>\n        val currentDismissalDuration = query.queryTime.since(info.lastDismissed)\n        val targetDismissalDuration = dismissDurationForCount(info.count, baseDismissDuration)\n\n        currentDismissalDuration > targetDismissalDuration\n      }).getOrElse(true)\n    Stitch.value(isVisible)\n  }\n\n  private def dismissDurationForCount(\n    dismissCount: Int,\n    dismissDuration: Duration\n  ): Duration =\n    // limit to maximum dismissal duration\n    dismissDuration * Math.min(dismissCount, DismissFatigueGate.MaximumDismissalCountMultiplier)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/ExcludeSoftUserGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.gizmoduck.{thriftscala => t}\nimport com.twitter.home_mixer.model.HomeFeatures.UserTypeFeature\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * A Soft User is a user who is in the gradual onboarding state. This gate can be\n * used to turn off certain functionality like ads for these users.\n */\nobject ExcludeSoftUserGate extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"ExcludeSoftUser\")\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val softUser = query.features\n      .exists(_.getOrElse(UserTypeFeature, None).exists(_ == t.UserType.Soft))\n    Stitch.value(!softUser)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/ExcludeSyntheticUserGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * A Synthetic User is a user who is created and managed on behalf of XCC's\n * Loadtesting framework. This gate can be used to turn off certain functionality like ads for\n * these users.\n */\nobject ExcludeSyntheticUserGate extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"ExcludeSyntheticUser\")\n  private val STRESS_TEST_APP_ID: Long = 1L\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val isSyntheticUser = query.clientContext.appId.contains(STRESS_TEST_APP_ID)\n    Stitch.value(!isSyntheticUser)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/PersistenceStoreDurationValidationGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.common_internal.analytics.twitter_client_user_agent_parser.UserAgent\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelinemixer.injection.store.persistence.TimelinePersistenceUtils\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelines.util.client_info.ClientPlatform\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\n\n/**\n * For users who max out the persistence store, we may serve certain modules too frequently.\n * Use this gate to prevent that.\n *\n * Gate stops the request if the time since the oldest entry < input duration\n * (or if there aren't enough entries, which can also cause a small duration)\n *\n * @param minInjectionIntervalParam the desired minimum interval between injections\n */\ncase class PersistenceStoreDurationValidationGate(\n  minDuration: Param[Duration] = StaticParam(48.hours))\n    extends Gate[PipelineQuery]\n    with TimelinePersistenceUtils {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"PersistenceStoreDurationValidation\")\n\n  private val MinEntries = 1500\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val continue = query.features.map { featureMap =>\n      val timelineResponses = featureMap.getOrElse(PersistenceEntriesFeature, Seq.empty)\n      val clientPlatform = ClientPlatform.fromQueryOptions(\n        clientAppId = query.clientContext.appId,\n        userAgent = query.clientContext.userAgent.flatMap(UserAgent.fromString)\n      )\n      val sortedResponses = responseByClient(clientPlatform, timelineResponses)\n      val entryCount = sortedResponses.flatMap(_.entries).size\n\n      val oldestTime = sortedResponses.lastOption.map(_.servedTime).getOrElse(Time.Bottom)\n\n      val duration = Time.now.since(oldestTime)\n      duration > query.params(minDuration) || entryCount < MinEntries\n    }\n\n    Stitch.value(continue.getOrElse(true))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/RateLimitGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerHasPremiumTier\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerIsRateLimited\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject RateLimitGate extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"RateLimit\")\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val isRateLimited = query.features.map(_.getOrElse(ViewerIsRateLimited, false))\n    val hasPremiumTier = query.features.map(_.getOrElse(ViewerHasPremiumTier, false))\n    Stitch.value(isRateLimited.contains(false) || hasPremiumTier.contains(true))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/RateLimitNotGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerHasPremiumTier\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerIsRateLimited\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject RateLimitNotGate extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"RateLimitNot\")\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val isRateLimited = query.features.map(_.getOrElse(ViewerIsRateLimited, false))\n    val hasPremiumTier = query.features.map(_.getOrElse(ViewerHasPremiumTier, false))\n    Stitch.value(isRateLimited.contains(true) && hasPremiumTier.contains(false))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/RecentlyServedByServedTypeGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelinemixer.injection.store.persistence.TimelinePersistenceUtils\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport com.twitter.timelineservice.model.TweetScoreV1\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\nimport com.twitter.home_mixer.{thriftscala => hmt}\n\n/**\n * Gate used to reduce the frequency of injections based on specific served types.\n * This gate checks if any tweets in the persistence store have a served type equal to the specified targetServedType.\n * Note that the actual interval between injections may be less than the specified minInjectionIntervalParam \n * if data is unavailable or missing. For example, being deleted by the persistence store via a TTL or similar mechanism.\n *\n * @param minInjectionIntervalParam the desired minimum interval between injections\n * @param targetServedType the served type to check for in persisted tweets\n */\ncase class RecentlyServedByServedTypeGate(\n  minInjectionIntervalParam: Param[Duration],\n  targetServedType: hmt.ServedType)\n    extends Gate[PipelineQuery]\n    with TimelinePersistenceUtils {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"RecentlyServedByServedType\")\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] =\n    Stitch(\n      query.queryTime.since(getLastInjectionTime(query)) > query.params(minInjectionIntervalParam))\n\n  private def getLastInjectionTime(query: PipelineQuery) = query.features\n    .flatMap { featureMap =>\n      val timelineResponses = featureMap.getOrElse(PersistenceEntriesFeature, Seq.empty)\n      val latestResponseWithTargetServedTypeEntry =\n        timelineResponses.find { response =>\n          response.entries.exists { entry =>\n            entry.entityIdType == EntityIdType.Tweet && \n            entry.itemIds.exists { itemIds =>\n              itemIds.exists { itemId =>\n                itemId.tweetScore.exists {\n                  case tweetScore: TweetScoreV1 =>\n                    tweetScore.servedType.contains(targetServedType.originalName)\n                  case _ => false\n                }\n              }\n            }\n          }\n        }\n\n      latestResponseWithTargetServedTypeEntry.map(_.servedTime)\n    }.getOrElse(Time.Bottom)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/RequestContextGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.home_mixer.model.request.DeviceContext.RequestContext\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Gate that fetches the request context from the device context and\n * continues if the request context matches *any* of the specified ones.\n */\ncase class RequestContextGate(requestContexts: Seq[RequestContext.Value])\n    extends Gate[PipelineQuery with HasDeviceContext] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"RequestContext\")\n\n  override def shouldContinue(query: PipelineQuery with HasDeviceContext): Stitch[Boolean] =\n    Stitch.value(\n      requestContexts.exists(query.deviceContext.flatMap(_.requestContextValue).contains))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/RequestContextNotGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.home_mixer.model.request.DeviceContext.RequestContext\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Gate that fetches the request context from the device context and\n * continues if the request context does not match any of the specified ones.\n *\n * If no input request context is specified, the gate continues\n */\ncase class RequestContextNotGate(requestContexts: Seq[RequestContext.Value])\n    extends Gate[PipelineQuery with HasDeviceContext] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"RequestContextNot\")\n\n  override def shouldContinue(query: PipelineQuery with HasDeviceContext): Stitch[Boolean] =\n    Stitch.value(\n      !requestContexts.exists(query.deviceContext.flatMap(_.requestContextValue).contains))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/SupportedLanguagesGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject SupportedLanguagesGate extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"SupportedLanguages\")\n\n  // Production languages which have high translation coverage for strings used in Home Timeline.\n  private val supportedLanguages: Set[String] = Set(\n    \"ar\", // Arabic\n    \"ar-x-fm\", // Arabic (Female)\n    \"bg\", // Bulgarian\n    \"bn\", // Bengali\n    \"ca\", // Catalan\n    \"cs\", // Czech\n    \"da\", // Danish\n    \"de\", // German\n    \"el\", // Greek\n    \"en\", // English\n    \"en-gb\", // British English\n    \"en-ss\", // English Screen shot\n    \"en-xx\", // English Pseudo\n    \"es\", // Spanish\n    \"eu\", // Basque\n    \"fa\", // Farsi (Persian)\n    \"fi\", // Finnish\n    \"fil\", // Filipino\n    \"fr\", // French\n    \"ga\", // Irish\n    \"gl\", // Galician\n    \"gu\", // Gujarati\n    \"he\", // Hebrew\n    \"hi\", // Hindi\n    \"hr\", // Croatian\n    \"hu\", // Hungarian\n    \"id\", // Indonesian\n    \"it\", // Italian\n    \"ja\", // Japanese\n    \"kn\", // Kannada\n    \"ko\", // Korean\n    \"mr\", // Marathi\n    \"msa\", // Malay\n    \"nl\", // Dutch\n    \"no\", // Norwegian\n    \"pl\", // Polish\n    \"pt\", // Portuguese\n    \"ro\", // Romanian\n    \"ru\", // Russian\n    \"sk\", // Slovak\n    \"sr\", // Serbian\n    \"sv\", // Swedish\n    \"ta\", // Tamil\n    \"th\", // Thai\n    \"tr\", // Turkish\n    \"uk\", // Ukrainian\n    \"ur\", // Urdu\n    \"vi\", // Vietnamese\n    \"zh-cn\", // Simplified Chinese\n    \"zh-tw\" // Traditional Chinese\n  )\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] =\n    Stitch.value(query.getLanguageCode.forall(supportedLanguages.contains))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/TestUserProbabilisticGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nimport com.twitter.product_mixer.component_library.module.TestUserMapper\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Generic gate used to assign a probability that the associated Component / Pipeline operates\n * on a Synthetic test user.\n *\n * @param testUserMapper the testUserMapper utility object that evaluates if this is a test user\n */\n@Singleton\nclass TestUserProbabilisticGate @Inject() (testUserMapper: TestUserMapper)\n    extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"TestUserProbabilistic\")\n  private val TEST_USERS_GATE_PROBABILITY = 0.05\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    if (!testUserMapper.isTestUser(\n        query.clientContext) || math.random < TEST_USERS_GATE_PROBABILITY) {\n      Stitch.True\n    } else {\n      Stitch.False\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate/TimelinesPersistenceStoreLastInjectionGate.scala",
    "content": "package com.twitter.home_mixer.functional_component.gate\n\nimport com.twitter.common_internal.analytics.twitter_client_user_agent_parser.UserAgent\nimport com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelinemixer.injection.store.persistence.TimelinePersistenceUtils\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelines.util.client_info.ClientPlatform\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\n\n/**\n * Gate used to reduce the frequency of injections. Note that the actual interval between injections may be\n * less than the specified minInjectionIntervalParam if data is unavailable or missing. For example, being deleted by\n * the persistence store via a TTL or similar mechanism.\n *\n * @param minInjectionIntervalParam the desired minimum interval between injections\n * @param persistenceEntriesFeature the feature for retrieving persisted timeline responses\n */\ncase class TimelinesPersistenceStoreLastInjectionGate(\n  minInjectionIntervalParam: Param[Duration],\n  entityIdType: EntityIdType.Value)\n    extends Gate[PipelineQuery]\n    with TimelinePersistenceUtils {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"TimelinesPersistenceStoreLastInjection\")\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] =\n    Stitch(\n      query.queryTime.since(getLastInjectionTime(query)) > query.params(minInjectionIntervalParam))\n\n  private def getLastInjectionTime(query: PipelineQuery) = query.features\n    .flatMap { featureMap =>\n      val timelineResponses = featureMap.getOrElse(PersistenceEntriesFeature, Seq.empty)\n      val clientPlatform = ClientPlatform.fromQueryOptions(\n        clientAppId = query.clientContext.appId,\n        userAgent = query.clientContext.userAgent.flatMap(UserAgent.fromString)\n      )\n      val sortedResponses = responseByClient(clientPlatform, timelineResponses)\n      val latestResponseWithEntityIdTypeEntry =\n        sortedResponses.find(_.entries.exists(_.entityIdType == entityIdType))\n\n      latestResponseWithEntityIdTypeEntry.map(_.servedTime)\n    }.getOrElse(Time.Bottom)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/query_transformer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"common-internal/analytics/twitter-client-user-agent-parser/src/main/scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence\",\n        \"timelineservice/common:model\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/query_transformer/EditedTweetsCandidatePipelineQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.functional_component.query_transformer\n\nimport com.twitter.common_internal.analytics.twitter_client_user_agent_parser.UserAgent\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelinemixer.clients.persistence.EntryWithItemIds\nimport com.twitter.timelines.persistence.thriftscala.RequestType\nimport com.twitter.timelines.util.client_info.ClientPlatform\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport com.twitter.util.Time\n\nobject EditedTweetsCandidatePipelineQueryTransformer\n    extends CandidatePipelineQueryTransformer[PipelineQuery, Seq[Long]] {\n\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"EditedTweets\")\n\n  // The time window for which a tweet remains editable after creation.\n  private val EditTimeWindow = 60.minutes\n\n  override def transform(query: PipelineQuery): Seq[Long] = {\n    val applicableCandidates = getApplicableCandidates(query)\n\n    if (applicableCandidates.nonEmpty) {\n      // Include the response corresponding with the Previous Timeline Load (PTL).\n      // Any tweets in it could have become stale since being served.\n      val previousTimelineLoadTime = applicableCandidates.head.servedTime\n\n      // The time window for editing a tweet is 60 minutes,\n      // so we ignore responses older than (PTL Time - 60 mins).\n      val inWindowCandidates: Seq[PersistenceStoreEntry] = applicableCandidates\n        .takeWhile(_.servedTime.until(previousTimelineLoadTime) < EditTimeWindow)\n\n      // Exclude the tweet IDs for which ReplaceEntry instructions have already been sent.\n      val (tweetsAlreadyReplaced, tweetsToCheck) = inWindowCandidates\n        .partition(_.entryWithItemIds.itemIds.exists(_.head.entryIdToReplace.nonEmpty))\n\n      val tweetIdFromEntry: PartialFunction[PersistenceStoreEntry, Long] = {\n        case entry if entry.tweetId.nonEmpty => entry.tweetId.get\n      }\n\n      val tweetIdsAlreadyReplaced: Set[Long] = tweetsAlreadyReplaced.collect(tweetIdFromEntry).toSet\n      val tweetIdsToCheck: Seq[Long] = tweetsToCheck.collect(tweetIdFromEntry)\n\n      tweetIdsToCheck.filterNot(tweetIdsAlreadyReplaced.contains).distinct\n    } else Seq.empty\n  }\n\n  // The candidates here come from the Timelines Persistence Store, via a query feature\n  private def getApplicableCandidates(query: PipelineQuery): Seq[PersistenceStoreEntry] = {\n    val userAgent = UserAgent.fromString(query.clientContext.userAgent.getOrElse(\"\"))\n    val clientPlatform = ClientPlatform.fromQueryOptions(query.clientContext.appId, userAgent)\n\n    val sortedResponses = query.features\n      .getOrElse(FeatureMap.empty)\n      .getOrElse(PersistenceEntriesFeature, Seq.empty)\n      .filter(_.clientPlatform == clientPlatform)\n      .sortBy(-_.servedTime.inMilliseconds)\n\n    val recentResponses = sortedResponses.indexWhere(_.requestType == RequestType.Initial) match {\n      case -1 => sortedResponses\n      case lastGetInitialIndex => sortedResponses.take(lastGetInitialIndex + 1)\n    }\n\n    recentResponses.flatMap { r =>\n      r.entries.collect {\n        case entry if entry.entityIdType == EntityIdType.Tweet =>\n          PersistenceStoreEntry(entry, r.servedTime, r.clientPlatform, r.requestType)\n      }\n    }.distinct\n  }\n}\n\ncase class PersistenceStoreEntry(\n  entryWithItemIds: EntryWithItemIds,\n  servedTime: Time,\n  clientPlatform: ClientPlatform,\n  requestType: RequestType) {\n\n  // Timelines Persistence Store currently includes 1 tweet ID per entryWithItemIds for tweets\n  val tweetId: Option[Long] = entryWithItemIds.itemIds.flatMap(_.head.tweetId)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/scorer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    dependencies = [\n        \"3rdparty/jvm/io/grpc:grpc-protobuf\",\n        \"3rdparty/jvm/io/grpc:grpc-stub\",\n        \"finagle-internal/finagle-grpc/src/main/scala\",\n        \"finatra/inject/inject-utils/src/main/scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/user_history\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module:test-user-mapper\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"timelines/src/main/scala/com/twitter/timelines/clients/predictionservice\",\n        \"timelineservice/common:model\",\n        \"user_history_transformer/service/src/main/java/com/x/user_action_sequence\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/scorer/FeedbackFatigueScorer.scala",
    "content": "package com.twitter.home_mixer.functional_component.scorer\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FeedbackHistoryFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.common.{thriftscala => tl}\nimport com.twitter.timelineservice.model.FeedbackEntry\nimport com.twitter.timelineservice.{thriftscala => tls}\nimport com.twitter.util.Time\nimport scala.collection.mutable\n\nobject FeedbackFatigueScorer\n    extends Scorer[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"FeedbackFatigue\")\n\n  override def features: Set[Feature[_, _]] = Set(ScoreFeature)\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.features.exists(_.getOrElse(FeedbackHistoryFeature, Seq.empty).nonEmpty)\n\n  val DurationForDiscounting = 140.days\n  private val ScoreMultiplierLowerBound = 0.2\n  private val ScoreMultiplierUpperBound = 1.0\n  private val ScoreMultiplierIncrementsCount = 4\n  private val ScoreMultiplierIncrement =\n    (ScoreMultiplierUpperBound - ScoreMultiplierLowerBound) / ScoreMultiplierIncrementsCount\n  private val ScoreMultiplierIncrementDurationInDays =\n    DurationForDiscounting.inDays / ScoreMultiplierIncrementsCount.toDouble\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val feedbackEntriesByEngagementType =\n      query.features\n        .getOrElse(FeatureMap.empty).getOrElse(FeedbackHistoryFeature, Seq.empty)\n        .filter { entry =>\n          val timeSinceFeedback = query.queryTime.minus(entry.timestamp)\n          timeSinceFeedback < DurationForDiscounting &&\n          entry.feedbackType == tls.FeedbackType.SeeFewer\n        }.groupBy(_.engagementType)\n\n    val authorsToDiscount =\n      getUserDiscounts(\n        query.queryTime,\n        feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Tweet, Seq.empty))\n    val likersToDiscount =\n      getUserDiscounts(\n        query.queryTime,\n        feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Like, Seq.empty))\n    val followersToDiscount =\n      getUserDiscounts(\n        query.queryTime,\n        feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Follow, Seq.empty))\n    val retweetersToDiscount =\n      getUserDiscounts(\n        query.queryTime,\n        feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Retweet, Seq.empty))\n\n    val featureMaps = candidates.map { candidate =>\n      val multiplier = getScoreMultiplier(\n        candidate,\n        authorsToDiscount,\n        likersToDiscount,\n        followersToDiscount,\n        retweetersToDiscount\n      )\n      val score = candidate.features.getOrElse(ScoreFeature, None)\n      FeatureMapBuilder().add(ScoreFeature, score.map(_ * multiplier)).build()\n    }\n\n    Stitch.value(featureMaps)\n  }\n\n  def getScoreMultiplier(\n    candidate: CandidateWithFeatures[TweetCandidate],\n    authorsToDiscount: Map[Long, Double],\n    likersToDiscount: Map[Long, Double],\n    followersToDiscount: Map[Long, Double],\n    retweetersToDiscount: Map[Long, Double],\n  ): Double = {\n    val originalAuthorId =\n      CandidatesUtil.getOriginalAuthorId(candidate.features).getOrElse(0L)\n    val originalAuthorMultiplier = authorsToDiscount.getOrElse(originalAuthorId, 1.0)\n\n    val likers = candidate.features.getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty)\n    val likerMultipliers = likers.flatMap(likersToDiscount.get)\n    val likerMultiplier =\n      if (likerMultipliers.nonEmpty && likers.size == likerMultipliers.size)\n        likerMultipliers.max\n      else 1.0\n\n    val followers = candidate.features.getOrElse(SGSValidFollowedByUserIdsFeature, Seq.empty)\n    val followerMultipliers = followers.flatMap(followersToDiscount.get)\n    val followerMultiplier =\n      if (followerMultipliers.nonEmpty && followers.size == followerMultipliers.size &&\n        likers.isEmpty)\n        followerMultipliers.max\n      else 1.0\n\n    val authorId = candidate.features.getOrElse(AuthorIdFeature, None).getOrElse(0L)\n    val retweeterMultiplier =\n      if (candidate.features.getOrElse(IsRetweetFeature, false))\n        retweetersToDiscount.getOrElse(authorId, 1.0)\n      else 1.0\n\n    originalAuthorMultiplier * likerMultiplier * followerMultiplier * retweeterMultiplier\n  }\n\n  def getUserDiscounts(\n    queryTime: Time,\n    feedbackEntries: Seq[FeedbackEntry],\n  ): Map[Long, Double] = {\n    val userDiscounts = mutable.Map[Long, Double]()\n    feedbackEntries\n      .collect {\n        case FeedbackEntry(_, _, tl.FeedbackEntity.UserId(userId), timestamp, _, _) =>\n          val timeSinceFeedback = queryTime.minus(timestamp)\n          val multiplier = ((timeSinceFeedback.inDays / ScoreMultiplierIncrementDurationInDays)\n            * ScoreMultiplierIncrement + ScoreMultiplierLowerBound)\n          userDiscounts.update(userId, multiplier)\n      }\n    userDiscounts.toMap\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/scorer/NaviModelScorer.scala",
    "content": "package com.twitter.home_mixer.functional_component.scorer\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.NaviClientConfigFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PredictionRequestIdFeature\nimport com.twitter.home_mixer.model.PredictedScoreFeature.PredictedScoreFeatureSet\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelIdParam\nimport com.twitter.home_mixer.util.NaviScorerStatsHandler\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.module.TestUserMapper\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.AllFeatures\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordConverter\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordExtractor\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.FeatureMapSanitizer\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport com.twitter.util.Return\nimport java.util.UUID\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.product_mixer.core.feature.datarecord.BaseDataRecordFeature\n\nobject CommonFeaturesDataRecordFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject CandidateFeaturesDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\ncase class NaviModelScorer @Inject() (\n  predictClientFactory: PredictClientFactory,\n  testUserMapper: TestUserMapper,\n  statsReceiver: StatsReceiver)\n    extends Scorer[PipelineQuery, TweetCandidate] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"NaviModel\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    CommonFeaturesDataRecordFeature,\n    CandidateFeaturesDataRecordFeature,\n    PredictionRequestIdFeature,\n  ) ++ PredictedScoreFeatureSet.asInstanceOf[Set[Feature[_, _]]]\n\n  private val queryDataRecordAdapter = new DataRecordConverter(AllFeatures())\n  private val candidatesDataRecordAdapter = new DataRecordConverter(AllFeatures())\n\n  private val resultDataRecordExtractor = new DataRecordExtractor(PredictedScoreFeatureSet)\n  private val modelStatsHandler = new NaviScorerStatsHandler(statsReceiver, getClass.getSimpleName)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val modelId = query.params(ModelIdParam)\n    val naviClientConfig =\n      query.features.map(_.get(NaviClientConfigFeature)).get // Should always be present\n\n    val modelStats =\n      modelStatsHandler.getModelStats(query)\n\n    val modelClient =\n      predictClientFactory.getClient(\n        naviClientConfig.clientName,\n        naviClientConfig.customizedBatchSize)\n\n    val predictionRequestId = UUID.randomUUID.getMostSignificantBits\n    val candidateAdapter = candidatesDataRecordAdapter.toDataRecord(_)\n\n    val commonRecord = query.features.map(queryDataRecordAdapter.toDataRecord)\n\n    def getScores(\n      candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n    ): Future[Seq[FeatureMap]] = {\n      val features = candidates.map(_.features)\n      val records = features.map(candidateAdapter)\n\n      val responsesFut =\n        modelClient.getPredictionsForBatch(records, commonRecord, modelId = Some(modelId))\n      responsesFut.map { responses =>\n        modelStats.failuresStat.add(responses.count(_.isThrow))\n        modelStats.responsesStat.add(responses.size)\n\n        if (responses.size == candidates.size) {\n          val predictedScoreFeatureMaps = responses.map {\n            case Return(dataRecord) => resultDataRecordExtractor.fromDataRecord(dataRecord)\n            case _ => resultDataRecordExtractor.fromDataRecord(new DataRecord())\n          }\n\n          val featureMapSanitizer = new FeatureMapSanitizer[BaseDataRecordFeature[_, _]](\n            includeFeatures = PredictedScoreFeatureSet.toSet, // Ensure this is a Set[DRFeature]\n            statsReceiver = statsReceiver\n          )\n\n          // Sanitize the FeatureMaps\n          val sanitizedFeatureMaps = featureMapSanitizer.sanitize(predictedScoreFeatureMaps)\n\n          // Add Data Record to candidate Feature Map for logging in later stages\n          sanitizedFeatureMaps.zip(records).map {\n            case (predictedScoreFeatureMap, candidateRecord) =>\n              predictedScoreFeatureMap ++\n                FeatureMapBuilder()\n                  .add(CandidateFeaturesDataRecordFeature, candidateRecord)\n                  .add(CommonFeaturesDataRecordFeature, commonRecord.getOrElse(new DataRecord()))\n                  .add(PredictionRequestIdFeature, Some(predictionRequestId))\n                  .build()\n          }\n        } else {\n          modelStats.invalidResponsesCounter.incr()\n          throw PipelineFailure(IllegalStateFailure, \"Result size mismatched candidates size\")\n        }\n      }\n    }\n\n    val scores = OffloadFuturePools.offloadBatchSeqToFutureSeq(\n      candidates,\n      getScores(_),\n      naviClientConfig.customizedBatchSize.getOrElse(predictClientFactory.DefaultRequestBatchSize),\n      offload = true\n    )\n    Stitch.callFuture(scores)\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/scorer/OONTweetScalingScorer.scala",
    "content": "package com.twitter.home_mixer.functional_component.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Scales scores of each out-of-network tweet by the specified scale factor\n */\nobject OONTweetScalingScorer extends Scorer[PipelineQuery, TweetCandidate] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"OONTweetScaling\")\n\n  override val features: Set[Feature[_, _]] = Set(ScoreFeature)\n\n  private val ScaleFactor = 0.75\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    Stitch.value {\n      candidates.map { candidate =>\n        val score = candidate.features.getOrElse(ScoreFeature, None)\n        val updatedScore = if (selector(candidate)) score.map(_ * ScaleFactor) else score\n        FeatureMapBuilder().add(ScoreFeature, updatedScore).build()\n      }\n    }\n  }\n\n  /**\n   * We should only be applying this multiplier to Out-Of-Network tweets.\n   * In-Network Retweets of Out-Of-Network tweets should not have this multiplier applied\n   */\n  private def selector(candidate: CandidateWithFeatures[TweetCandidate]): Boolean = {\n    !candidate.features.getOrElse(InNetworkFeature, false) &&\n    !candidate.features.getOrElse(IsRetweetFeature, false)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/scorer/PhoenixModelRerankingScorer.scala",
    "content": "package com.twitter.home_mixer.functional_component.scorer\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.DebugStringFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PhoenixScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedScoreFeature.PhoenixPredictedScoreFeatures\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixInferenceClusterParam\nimport com.twitter.home_mixer.util.PhoenixScorerStatsHandler\nimport com.twitter.home_mixer.util.RerankerUtil._\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class PhoenixModelRerankingScorer @Inject() (statsReceiver: StatsReceiver)\n    extends Scorer[PipelineQuery, TweetCandidate] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"PhoenixModelReranking\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    PhoenixScoreFeature,\n    // WeightedModelScoreFeature, remove temporarily to avoid overwriting navi weighted score\n    DebugStringFeature\n  )\n\n  private val StatsReadabilityMultiplier = 1000\n  private val modelStatsHandler =\n    new PhoenixScorerStatsHandler(statsReceiver, getClass.getSimpleName)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val cluster = query.params(PhoenixInferenceClusterParam).toString\n    val modelStats = modelStatsHandler.getModelStats(query, cluster)\n\n    val scoresAndWeightsSeq = candidates.map { candidate =>\n      PhoenixPredictedScoreFeatures.map { feature =>\n        val predictions = feature.extractScore(candidate.features, query)\n        modelStats.trackPredictedScoreStats(feature, predictions)\n        val isEligible = feature.isEligible(candidate.features)\n        val score = if (predictions.nonEmpty && isEligible) predictions.max else 0.0\n        val weight = query.params(feature.modelWeightParam)\n        (score, weight)\n      }\n    }\n    val transformedScoresAndWeightsSeq = getScoresWithPerHeadMax(scoresAndWeightsSeq)\n\n    val debugStrings: Seq[String] =\n      candidates.map(_.features.getOrElse(DebugStringFeature, None).getOrElse(\"\"))\n\n    val featureMaps = transformedScoresAndWeightsSeq\n      .zip(debugStrings)\n      .map {\n        case (transformedScores, debugStr) =>\n          val finalScore =\n            aggregateWeightedScores(query, transformedScores, modelStats.negativeFilterCounter)\n          val featureNames = PhoenixPredictedScoreFeatures.map(_.featureName)\n          modelStats.scoreStat.add((finalScore * StatsReadabilityMultiplier).toFloat)\n\n          val updatedDebugStr =\n            computeDebugMetadata(debugStr, featureNames, transformedScores, finalScore)\n\n          FeatureMapBuilder()\n            .add(PhoenixScoreFeature, Some(finalScore))\n            .add(DebugStringFeature, Some(updatedDebugStr))\n            .build()\n      }\n\n    Stitch.value(featureMaps)\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/scorer/PhoenixScorer.scala",
    "content": "package com.twitter.home_mixer.functional_component.scorer\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserActionsFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedScoreFeature.PhoenixPredictedScoreFeatureSet\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixCluster\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixInferenceClusterParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixTimeoutInMsParam\nimport com.twitter.home_mixer.util.PhoenixUtils.createCandidateSets\nimport com.twitter.home_mixer.util.PhoenixUtils.getPredictionResponseMap\nimport com.twitter.home_mixer.util.PhoenixUtils.getTweetInfoFromCandidates\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.util.MemoizingStatsReceiver\nimport com.twitter.stitch.Stitch\nimport io.grpc.ManagedChannel\nimport javax.inject.Inject\nimport javax.inject.Singleton\n@Singleton\ncase class PhoenixScorer @Inject() (\n  @Named(\"PhoenixClient\") channelsMap: Map[PhoenixCluster.Value, Seq[ManagedChannel]],\n  statsReceiver: StatsReceiver)\n    extends Scorer[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"Phoenix\")\n\n  override val features: Set[Feature[_, _]] =\n    PhoenixPredictedScoreFeatureSet.asInstanceOf[Set[Feature[_, _]]]\n\n  val memoizingStatsReceiver: MemoizingStatsReceiver = new MemoizingStatsReceiver(\n    statsReceiver.scope(this.getClass.getSimpleName))\n\n  override def onlyIf(query: PipelineQuery): Boolean = {\n    query.features.flatMap(_.getOrElse(UserActionsFeature, None)).isDefined\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadStitch {\n    val phoenixCluster = query.params(PhoenixInferenceClusterParam)\n    val channels = channelsMap(phoenixCluster)\n\n    val tweetInfos =\n      getTweetInfoFromCandidates(candidates.map(_.candidate), candidates.map(_.features))\n    val request = createCandidateSets(query, tweetInfos)\n    val timeoutMs = query.params(PhoenixTimeoutInMsParam)\n    val predictionsMapStitch =\n      getPredictionResponseMap(\n        request,\n        channels,\n        phoenixCluster.toString,\n        timeoutMs,\n        memoizingStatsReceiver)\n\n    predictionsMapStitch.map { predictionsMap =>\n      candidates.map { candidate =>\n        val sourceTweetId =\n          candidate.features.getOrElse(SourceTweetIdFeature, None) match {\n            case Some(sourceTweetId) => sourceTweetId\n            case _ => candidate.candidate.id\n          }\n        val actionPredictionsMap = predictionsMap.getOrElse(sourceTweetId, Map.empty)\n        val fmBuilder = FeatureMapBuilder()\n        PhoenixPredictedScoreFeatureSet.map { feature =>\n          val predictions = feature.actions.flatMap(actionPredictionsMap.get)\n          val score = if (predictions.nonEmpty) Some(predictions.max) else None\n          fmBuilder.add(feature, score)\n        }\n        fmBuilder.build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/scorer/PredictClientFactory.scala",
    "content": "package com.twitter.home_mixer.functional_component.scorer\n\nimport com.twitter.finagle.stats.BroadcastStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecap\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecapGPU\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecapRealtime\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecapSecondary\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecapVideo\nimport com.twitter.timelines.clients.predictionservice.PredictionGRPCService\nimport com.twitter.timelines.clients.predictionservice.PredictionServiceGRPCClient\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\ncase class PredictClientFactory @Inject() (\n  @Named(NaviModelClientHomeRecap) homeRecapPredictionGRPCService: PredictionGRPCService,\n  @Named(NaviModelClientHomeRecapSecondary) homeRecapSecondaryPredictionGRPCService: PredictionGRPCService,\n  @Named(NaviModelClientHomeRecapRealtime) homeRecapRealtimePredictionGRPCService: PredictionGRPCService,\n  @Named(NaviModelClientHomeRecapGPU) homeRecapGPUPredictionGRPCService: PredictionGRPCService,\n  @Named(NaviModelClientHomeRecapVideo) homeRecapVideoPredictionGRPCService: PredictionGRPCService,\n  statsReceiver: StatsReceiver) {\n\n  val DefaultRequestBatchSize = 32\n\n  def getClient(\n    clientName: String,\n    customizedBatchSize: Option[Int]\n  ): PredictionServiceGRPCClient = {\n    clientName match {\n      case NaviModelClientHomeRecap =>\n        homeRecapModelClient\n      case NaviModelClientHomeRecapSecondary =>\n        homeRecapSecondaryModelClient\n      case NaviModelClientHomeRecapRealtime =>\n        homeRecapRealtimeModelClient\n      case NaviModelClientHomeRecapVideo =>\n        homeRecapVideoModelClient\n      case NaviModelClientHomeRecapGPU =>\n        val gpuBatchSize = customizedBatchSize.getOrElse(DefaultRequestBatchSize)\n        new PredictionServiceGRPCClient(\n          service = homeRecapGPUPredictionGRPCService,\n          statsReceiver = BroadcastStatsReceiver(\n            Seq(statsReceiver, statsReceiver.scope(\"home_recap_gpu\"))),\n          requestBatchSize = gpuBatchSize,\n          useCompact = false\n        )\n      case _ =>\n        throw new IllegalArgumentException(s\"Unknown clientName: $clientName\")\n    }\n  }\n\n  private lazy val homeRecapModelClient = new PredictionServiceGRPCClient(\n    service = homeRecapPredictionGRPCService,\n    statsReceiver = BroadcastStatsReceiver(Seq(statsReceiver, statsReceiver.scope(\"home_recap\"))),\n    requestBatchSize = DefaultRequestBatchSize,\n    useCompact = false\n  )\n\n  private lazy val homeRecapSecondaryModelClient = new PredictionServiceGRPCClient(\n    service = homeRecapSecondaryPredictionGRPCService,\n    statsReceiver = BroadcastStatsReceiver(Seq(statsReceiver, statsReceiver.scope(\"home_recap\"))),\n    requestBatchSize = DefaultRequestBatchSize,\n    useCompact = false\n  )\n\n  private lazy val homeRecapRealtimeModelClient = new PredictionServiceGRPCClient(\n    service = homeRecapRealtimePredictionGRPCService,\n    statsReceiver = BroadcastStatsReceiver(\n      Seq(statsReceiver, statsReceiver.scope(\"home_recap_realtime\"))),\n    requestBatchSize = DefaultRequestBatchSize,\n    useCompact = false\n  )\n\n  private lazy val homeRecapVideoModelClient = new PredictionServiceGRPCClient(\n    service = homeRecapVideoPredictionGRPCService,\n    statsReceiver = BroadcastStatsReceiver(\n      Seq(statsReceiver, statsReceiver.scope(\"home_recap_video\"))),\n    requestBatchSize = DefaultRequestBatchSize,\n    useCompact = false\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/scorer/WeighedModelRerankingScorer.scala",
    "content": "package com.twitter.home_mixer.functional_component.scorer\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.DebugStringFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.WeightedModelScoreFeature\nimport com.twitter.home_mixer.model.PredictedScoreFeature.PredictedScoreFeatures\nimport com.twitter.home_mixer.util.NaviScorerStatsHandler\nimport com.twitter.home_mixer.util.RerankerUtil._\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class WeighedModelRerankingScorer @Inject() (\n  statsReceiver: StatsReceiver)\n    extends Scorer[PipelineQuery, TweetCandidate] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"WeightedModelReranking\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    ScoreFeature,\n    WeightedModelScoreFeature,\n    DebugStringFeature\n  )\n\n  private val modelStatsHandler = new NaviScorerStatsHandler(statsReceiver, getClass.getSimpleName)\n  private val StatsReadabilityMultiplier = 1000\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val modelStats = modelStatsHandler.getModelStats(query)\n    val scoresAndWeightsSeq = candidates.map(computeModelScores(query, _, Some(modelStats)))\n    val transformedScoresAndWeightsSeq = getScoresWithPerHeadMax(scoresAndWeightsSeq)\n\n    val debugStrings: Seq[String] =\n      candidates.map(_.features.getOrElse(DebugStringFeature, None).getOrElse(\"\"))\n\n    val featureMaps = transformedScoresAndWeightsSeq\n      .zip(debugStrings)\n      .map {\n        case (transformedScores, debugStr) =>\n          val finalScore =\n            aggregateWeightedScores(query, transformedScores, modelStats.negativeFilterCounter)\n          val featureNames = PredictedScoreFeatures.map(_.statName)\n          modelStats.scoreStat.add((finalScore * StatsReadabilityMultiplier).toFloat)\n\n          val updatedDebugStr =\n            computeDebugMetadata(debugStr, featureNames, transformedScores, finalScore)\n\n          FeatureMapBuilder()\n            .add(ScoreFeature, Some(finalScore))\n            .add(WeightedModelScoreFeature, Some(finalScore))\n            .add(DebugStringFeature, Some(updatedDebugStr))\n            .build()\n      }\n\n    Stitch.value(featureMaps)\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"src/scala/com/twitter/suggests/controller_data\",\n        \"stringcenter/client\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector/DebunchCandidates.scala",
    "content": "package com.twitter.home_mixer.functional_component.selector\n\nimport com.twitter.home_mixer.functional_component.selector.DebunchCandidates.TrailingTweetsMinSize\nimport com.twitter.home_mixer.functional_component.selector.DebunchCandidates.TrailingTweetsPortionToKeep\nimport com.twitter.home_mixer.model.HomeFeatures.GetNewerFeature\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope.PartitionedCandidates\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait MustDebunch {\n  def apply(candidate: CandidateWithDetails): Boolean\n}\n\nobject DebunchCandidates {\n  val TrailingTweetsMinSize = 5\n  val TrailingTweetsPortionToKeep = 0.1\n}\n\n/**\n * This selector rearranges the candidates to only allow bunches of size [[maxBunchSize]], where a\n * bunch is a consecutive sequence of candidates that meet [[mustDebunch]].\n */\ncase class DebunchCandidates(\n  override val pipelineScope: CandidateScope,\n  mustDebunch: MustDebunch,\n  maxBunchSize: Int)\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val PartitionedCandidates(selectedCandidates, otherCandidates) =\n      pipelineScope.partition(remainingCandidates)\n    val mutableCandidates = collection.mutable.ListBuffer(selectedCandidates: _*)\n\n    var candidatePointer = 0\n    var nonDebunchPointer = 0\n    var bunchSize = 0\n    var finalNonDebunch = -1\n\n    while (candidatePointer < mutableCandidates.size) {\n      if (mustDebunch(mutableCandidates(candidatePointer))) bunchSize += 1\n      else {\n        bunchSize = 0\n        finalNonDebunch = candidatePointer\n      }\n\n      if (bunchSize > maxBunchSize) {\n        nonDebunchPointer = Math.max(candidatePointer, nonDebunchPointer)\n        while (nonDebunchPointer < mutableCandidates.size &&\n          mustDebunch(mutableCandidates(nonDebunchPointer))) {\n          nonDebunchPointer += 1\n        }\n        if (nonDebunchPointer == mutableCandidates.size)\n          candidatePointer = mutableCandidates.size\n        else {\n          val nextNonDebunch = mutableCandidates(nonDebunchPointer)\n          mutableCandidates.remove(nonDebunchPointer)\n          mutableCandidates.insert(candidatePointer, nextNonDebunch)\n          bunchSize = 0\n          finalNonDebunch = candidatePointer\n        }\n      }\n\n      candidatePointer += 1\n    }\n\n    val debunchedCandidates = if (query.features.exists(_.getOrElse(GetNewerFeature, false))) {\n      val trailingTweetsSize = mutableCandidates.size - finalNonDebunch - 1\n      val keepCandidates = finalNonDebunch + 1 +\n        Math.max(TrailingTweetsMinSize, TrailingTweetsPortionToKeep * trailingTweetsSize).toInt\n      mutableCandidates.toList.take(keepCandidates)\n    } else mutableCandidates.toList\n\n    val updatedCandidates = otherCandidates ++ debunchedCandidates\n    SelectorResult(remainingCandidates = updatedCandidates, result = result)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector/RandomShuffleCandidates.scala",
    "content": "package com.twitter.home_mixer.functional_component.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport scala.util.Random\n\nobject RandomShuffleCandidates extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val shuffledRemainingCandidates = Random.shuffle(remainingCandidates)\n    SelectorResult(remainingCandidates = shuffledRemainingCandidates, result = result)\n  }\n\n  override def pipelineScope: CandidateScope = AllPipelines\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector/ScoreAveragingPositionSelector.scala",
    "content": "package com.twitter.home_mixer.functional_component.selector\n\nimport com.twitter.home_mixer.model.HomeFeatures.IsBoostedCandidateFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserFollowersCountFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CategoryColdStartProbabilisticReturnParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CategoryColdStartTierOneProbabilityParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.ContentExplorationBoostPosParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.ContentExplorationViewerMaxFollowersParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.DeepRetrievalBoostPosParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.DeepRetrievalI2iProbabilityParam\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport scala.util.Random\n\n/**\n * Abstract base class specifically for fixed position selectors that need score averaging\n * Contains complete selector logic, supports removing target candidates from remaining candidates\n * and assigning them the average score of adjacent candidates\n */\nabstract class ScoreAveragingPositionSelector extends Selector[PipelineQuery] {\n\n  // Abstract methods that subclasses need to implement\n  def getOffset(query: PipelineQuery): Int\n  def getNumCandidatesToBoost(query: PipelineQuery): Int\n  def selectEligibleCandidates(query: PipelineQuery, candidate: CandidateWithDetails): Boolean\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val eligibleCandidates = remainingCandidates.filter(c => selectEligibleCandidates(query, c))\n    val targetCandidates = eligibleCandidates.take(getNumCandidatesToBoost(query))\n\n    if (targetCandidates.nonEmpty) {\n      val filteredRemainingCandidates = remainingCandidates.diff(targetCandidates)\n      val offset = getOffset(query)\n\n      // Get scores from adjacent positions for average calculation\n      val adjacentScores = getAdjacentScores(filteredRemainingCandidates, offset)\n\n      // Assign target candidates scores based on the average of adjacent candidates\n      val scoredTargetCandidates =\n        targetCandidates.map(candidate => assignAverageScore(candidate, adjacentScores))\n\n      val updatedRemainingCandidates =\n        if (offset >= 0 && offset <= filteredRemainingCandidates.length) {\n          filteredRemainingCandidates.slice(0, offset) ++ scoredTargetCandidates ++\n            filteredRemainingCandidates.slice(offset, filteredRemainingCandidates.length)\n        } else {\n          filteredRemainingCandidates ++ scoredTargetCandidates\n        }\n\n      SelectorResult(remainingCandidates = updatedRemainingCandidates, result = result)\n    } else {\n      SelectorResult(remainingCandidates = remainingCandidates, result = result)\n    }\n  }\n\n  // Get adjacent position scores\n  def getAdjacentScores(\n    filteredRemainingCandidates: Seq[CandidateWithDetails],\n    offset: Int\n  ): Seq[Option[Double]] = {\n    if (offset >= 1 && offset < filteredRemainingCandidates.length) {\n      Seq(\n        filteredRemainingCandidates(offset - 1).features.getOrElse(ScoreFeature, None),\n        filteredRemainingCandidates(offset).features.getOrElse(ScoreFeature, None)\n      )\n    } else if (offset == 0 && filteredRemainingCandidates.nonEmpty) {\n      Seq(filteredRemainingCandidates.head.features.getOrElse(ScoreFeature, None))\n    } else if (offset >= filteredRemainingCandidates.length && filteredRemainingCandidates.nonEmpty) {\n      Seq(filteredRemainingCandidates.last.features.getOrElse(ScoreFeature, None))\n    } else {\n      Seq.empty\n    }\n  }\n\n  // Assign scores to candidates based on the average score of adjacent candidates\n  def assignAverageScore(\n    candidate: CandidateWithDetails,\n    adjacentScores: Seq[Option[Double]]\n  ): CandidateWithDetails = {\n    val validScores = adjacentScores.flatten\n    val avgScore = if (validScores.nonEmpty) {\n      Some(validScores.sum / validScores.size)\n    } else None\n\n    candidate match {\n      case item: ItemCandidateWithDetails if avgScore.isDefined =>\n        val updatedFeatures = FeatureMapBuilder()\n          .add(ScoreFeature, avgScore)\n          .add(IsBoostedCandidateFeature, true)\n          .build()\n        item.copy(features = item.features ++ updatedFeatures)\n      case _ => candidate\n    }\n  }\n\n  def pipelineScope: CandidateScope = AllPipelines\n}\n\n/**\n * Concrete implementations of score averaging position selectors\n */\nobject SortFixedPositionContentExplorationSimclusterColdPostsCandidates\n    extends ScoreAveragingPositionSelector {\n\n  override def getOffset(query: PipelineQuery): Int =\n    query.params.getInt(ContentExplorationBoostPosParam)\n\n  override def getNumCandidatesToBoost(query: PipelineQuery): Int = 1\n\n  override def selectEligibleCandidates(\n    query: PipelineQuery,\n    candidate: CandidateWithDetails\n  ): Boolean = {\n    val servedType = candidate.features.getOrElse(ServedTypeFeature, hmt.ServedType.Undefined)\n    servedType == hmt.ServedType.ForYouContentExplorationSimclusterColdPosts\n  }\n}\n\nobject SortFixedPositionContentExplorationMixedCandidates extends ScoreAveragingPositionSelector {\n\n  override def getOffset(query: PipelineQuery): Int =\n    query.params.getInt(ContentExplorationBoostPosParam)\n\n  override def getNumCandidatesToBoost(query: PipelineQuery): Int = 1\n\n  override def selectEligibleCandidates(\n    query: PipelineQuery,\n    candidate: CandidateWithDetails\n  ): Boolean = true\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val chosenServedType =\n      if (Random.nextDouble() < query.params.getDouble(CategoryColdStartTierOneProbabilityParam))\n        hmt.ServedType.ForYouContentExploration\n      else hmt.ServedType.ForYouContentExplorationTier2\n\n    val eligibleCandidates = remainingCandidates\n      .filter(_.features.getOrElse(ServedTypeFeature, hmt.ServedType.Undefined) == chosenServedType)\n\n    val viewerMaxFollowers = query.params(ContentExplorationViewerMaxFollowersParam)\n    val viewerFollowers = query.features.flatMap(_.getOrElse(UserFollowersCountFeature, None))\n\n    val returnCandidatesRandomDraw =\n      Random.nextDouble() < query.params.getDouble(CategoryColdStartProbabilisticReturnParam)\n\n    if (viewerFollowers.forall(_ < viewerMaxFollowers) &&\n      eligibleCandidates.nonEmpty && returnCandidatesRandomDraw) {\n      val targetCandidates = eligibleCandidates.take(getNumCandidatesToBoost(query))\n\n      val filteredRemainingCandidates = remainingCandidates.diff(targetCandidates)\n      val offset = getOffset(query)\n\n      // Get scores from adjacent positions for average calculation\n      val adjacentScores = getAdjacentScores(filteredRemainingCandidates, offset)\n\n      // Assign target candidates scores based on the average of adjacent candidates\n      val scoredTargetCandidates =\n        targetCandidates.map(candidate => assignAverageScore(candidate, adjacentScores))\n\n      val updatedRemainingCandidates =\n        if (offset >= 0 && offset <= filteredRemainingCandidates.length) {\n          filteredRemainingCandidates.slice(0, offset) ++ scoredTargetCandidates ++\n            filteredRemainingCandidates.slice(offset, filteredRemainingCandidates.length)\n        } else filteredRemainingCandidates ++ scoredTargetCandidates\n\n      SelectorResult(remainingCandidates = updatedRemainingCandidates, result = result)\n    } else SelectorResult(remainingCandidates = remainingCandidates, result = result)\n  }\n}\n\nobject SortFixedPositionDeepRetrievalMixedCandidates extends ScoreAveragingPositionSelector {\n\n  override def getOffset(query: PipelineQuery): Int =\n    query.params.getInt(DeepRetrievalBoostPosParam)\n\n  override def getNumCandidatesToBoost(query: PipelineQuery): Int = 1\n\n  override def selectEligibleCandidates(\n    query: PipelineQuery,\n    candidate: CandidateWithDetails\n  ): Boolean = true\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val chosenServedType =\n      if (Random.nextDouble() < query.params.getDouble(DeepRetrievalI2iProbabilityParam))\n        hmt.ServedType.ForYouContentExplorationDeepRetrievalI2i\n      else hmt.ServedType.ForYouContentExplorationTier2DeepRetrievalI2i\n\n    val eligibleCandidates = remainingCandidates\n      .filter(_.features.getOrElse(ServedTypeFeature, hmt.ServedType.Undefined) == chosenServedType)\n\n    val viewerMaxFollowers = query.params(ContentExplorationViewerMaxFollowersParam)\n    val viewerFollowers = query.features.flatMap(_.getOrElse(UserFollowersCountFeature, None))\n\n    if (viewerFollowers.forall(_ < viewerMaxFollowers) && eligibleCandidates.nonEmpty) {\n      val targetCandidates = Seq(eligibleCandidates(Random.nextInt(eligibleCandidates.size)))\n\n      val filteredRemainingCandidates = remainingCandidates.diff(targetCandidates)\n      val offset = getOffset(query)\n\n      // Get scores from adjacent positions for average calculation\n      val adjacentScores = getAdjacentScores(filteredRemainingCandidates, offset)\n\n      // Assign target candidates scores based on the average of adjacent candidates\n      val scoredTargetCandidates =\n        targetCandidates.map(candidate => assignAverageScore(candidate, adjacentScores))\n\n      val updatedRemainingCandidates =\n        if (offset >= 0 && offset <= filteredRemainingCandidates.length) {\n          filteredRemainingCandidates.slice(0, offset) ++ scoredTargetCandidates ++\n            filteredRemainingCandidates.slice(offset, filteredRemainingCandidates.length)\n        } else filteredRemainingCandidates ++ scoredTargetCandidates\n\n      SelectorResult(remainingCandidates = updatedRemainingCandidates, result = result)\n    } else SelectorResult(remainingCandidates = remainingCandidates, result = result)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector/SortFixedPositionCandidates.scala",
    "content": "package com.twitter.home_mixer.functional_component.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nabstract class SortFixedPositionCandidates extends Selector[PipelineQuery] {\n\n  def getOffset(query: PipelineQuery): Int\n\n  def getNumCandidatesToBoost(query: PipelineQuery): Int\n\n  def selectEligibleCandidates(\n    query: PipelineQuery,\n    candidate: CandidateWithDetails\n  ): Boolean\n\n  def selectTargetFromRemainingCandidates(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails]\n  ): Seq[CandidateWithDetails] = {\n    val numCandidatesToBoost = getNumCandidatesToBoost(query)\n    val eligibleCandidates = remainingCandidates\n      .filter(candidate => selectEligibleCandidates(query, candidate))\n\n    eligibleCandidates.take(numCandidatesToBoost)\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val targetCandidates = selectTargetFromRemainingCandidates(query, remainingCandidates)\n    val offset = getOffset(query)\n    if (targetCandidates.nonEmpty) {\n      val updatedRemainingCandidates = if (offset >= 0 && offset < remainingCandidates.length) {\n        remainingCandidates.slice(0, offset) ++ targetCandidates ++ remainingCandidates.slice(\n          offset,\n          remainingCandidates.length)\n      } else {\n        remainingCandidates ++ targetCandidates\n      }\n      SelectorResult(\n        remainingCandidates = updatedRemainingCandidates.distinct,\n        result = result\n      )\n    } else {\n      SelectorResult(remainingCandidates = remainingCandidates, result = result)\n    }\n  }\n\n  override def pipelineScope: CandidateScope = AllPipelines\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector/UpdateConversationModuleId.scala",
    "content": "package com.twitter.home_mixer.functional_component.selector\n\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtModulePresentation\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope.PartitionedCandidates\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * This selector updates the id of the conversation modules to be the head of the module's id.\n */\ncase class UpdateConversationModuleId(\n  override val pipelineScope: CandidateScope)\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val PartitionedCandidates(selectedCandidates, otherCandidates) =\n      pipelineScope.partition(remainingCandidates)\n\n    val updatedCandidates = selectedCandidates.map {\n      case module @ ModuleCandidateWithDetails(candidates, presentationOpt, _) =>\n        val updatedPresentation = presentationOpt.map {\n          case urtModule @ UrtModulePresentation(timelineModule) =>\n            urtModule.copy(timelineModule =\n              timelineModule.copy(id = candidates.head.candidateIdLong))\n        }\n        module.copy(presentation = updatedPresentation)\n      case candidate => candidate\n    }\n\n    SelectorResult(remainingCandidates = updatedCandidates ++ otherCandidates, result = result)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector/UpdateHomeClientEventDetails.scala",
    "content": "package com.twitter.home_mixer.functional_component.selector\n\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeClientEventDetailsBuilder\nimport com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ConversationModule2DisplayedTweetsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ConversationModuleHasGapFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokCategoryDataRecordFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MaxSingleAuthorCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MaxSingleCategoryCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PositionFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedInConversationModuleFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedSizeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UniqueAuthorCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UniqueCategoryCountFeature\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtItemPresentation\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtModulePresentation\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Builds serialized tweet type metrics controller data and updates Client Event Details\n * and Candidate Presentations with this info.\n *\n * Currently only updates presentation of Item Candidates. This needs to be updated\n * when modules are added.\n *\n * This is implemented as a Selector instead of a Decorator in the Candidate Pipeline\n * because we need to add controller data that looks at the final timeline as a whole\n * (e.g. served size, final candidate positions).\n *\n * @param candidatePipelines - only candidates from the specified pipeline will be updated\n */\ncase class UpdateHomeClientEventDetails(candidatePipelines: Set[CandidatePipelineIdentifier])\n    extends Selector[PipelineQuery] {\n\n  override val pipelineScope: CandidateScope = SpecificPipelines(candidatePipelines)\n\n  private val detailsBuilder = HomeClientEventDetailsBuilder()\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val selectedCandidates = result.filter(pipelineScope.contains)\n\n    val authorCounts: Map[Long, Int] = selectedCandidates\n      .flatMap(_.features.getOrElse(AuthorIdFeature, None))\n      .groupBy(identity)\n      .map { case (authorId, ids) => authorId -> ids.length }\n\n    val categoryCounts: Map[String, Int] = selectedCandidates\n      .flatMap(_.features.getOrElse(GrokCategoryDataRecordFeature, None).getOrElse(Map.empty).keys)\n      .groupBy(identity)\n      .map { case (category, categories) => category -> categories.length }\n\n    val resultFeatures = FeatureMapBuilder()\n      .add(ServedSizeFeature, Some(selectedCandidates.size))\n      .add(UniqueAuthorCountFeature, Some(authorCounts.size))\n      .add(\n        MaxSingleAuthorCountFeature,\n        Some(if (authorCounts.values.nonEmpty) authorCounts.values.max else 0))\n      .add(UniqueCategoryCountFeature, Some(categoryCounts.size))\n      .add(\n        MaxSingleCategoryCountFeature,\n        Some(if (categoryCounts.values.nonEmpty) categoryCounts.values.max else 0))\n      .build()\n\n    val updatedResult = result.zipWithIndex.map {\n      case (item @ ItemCandidateWithDetails(candidate, _, _), position)\n          if pipelineScope.contains(item) =>\n        val resultCandidateFeatures = FeatureMapBuilder()\n          .add(PositionFeature, Some(position))\n          .build()\n\n        updateItemPresentation(query, item, resultFeatures, resultCandidateFeatures)\n\n      case (module @ ModuleCandidateWithDetails(candidates, presentation, features), position)\n          if pipelineScope.contains(module) =>\n        val resultCandidateFeatures = FeatureMapBuilder()\n          .add(PositionFeature, Some(position))\n          .add(ServedInConversationModuleFeature, true)\n          .add(ConversationModule2DisplayedTweetsFeature, module.candidates.size == 2)\n          .add(\n            ConversationModuleHasGapFeature,\n            module.candidates.last.features.getOrElse(AncestorsFeature, Seq.empty).size > 2)\n          .add(ServedTypeFeature, module.candidates.last.features.get(ServedTypeFeature))\n          .build()\n\n        val updatedItemCandidates =\n          candidates.map(updateItemPresentation(query, _, resultFeatures, resultCandidateFeatures))\n\n        val updatedCandidateFeatures = features ++ resultFeatures ++ resultCandidateFeatures\n\n        val updatedPresentation = presentation.map {\n          case urtModule @ UrtModulePresentation(timelineModule) =>\n            val clientEventDetails =\n              detailsBuilder(\n                query,\n                candidates.last.candidate,\n                query.features.get ++ updatedCandidateFeatures)\n            val updatedClientEventInfo =\n              timelineModule.clientEventInfo.map(_.copy(details = clientEventDetails))\n            val updatedTimelineModule =\n              timelineModule.copy(clientEventInfo = updatedClientEventInfo)\n            urtModule.copy(timelineModule = updatedTimelineModule)\n        }\n\n        module.copy(\n          candidates = updatedItemCandidates,\n          presentation = updatedPresentation,\n          features = updatedCandidateFeatures\n        )\n\n      case (any, position) => any\n    }\n\n    SelectorResult(remainingCandidates = remainingCandidates, result = updatedResult)\n  }\n\n  private def updateItemPresentation(\n    query: PipelineQuery,\n    item: ItemCandidateWithDetails,\n    resultCandidateFeatures: FeatureMap,\n    resultFeatures: FeatureMap,\n  ): ItemCandidateWithDetails = {\n    val updatedItemCandidateFeatures = item.features ++ resultFeatures ++ resultCandidateFeatures\n\n    val updatedPresentation = item.presentation.map {\n      case urtItem @ UrtItemPresentation(timelineItem: TweetItem, _) =>\n        val clientEventDetails =\n          detailsBuilder(query, item.candidate, query.features.get ++ updatedItemCandidateFeatures)\n        val updatedClientEventInfo =\n          timelineItem.clientEventInfo.map(_.copy(details = clientEventDetails))\n        val updatedTimelineItem = timelineItem.copy(clientEventInfo = updatedClientEventInfo)\n        urtItem.copy(timelineItem = updatedTimelineItem)\n      case any => any\n    }\n    item.copy(presentation = updatedPresentation, features = updatedItemCandidateFeatures)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector/UpdateNewTweetsPillDecoration.scala",
    "content": "package com.twitter.home_mixer.functional_component.selector\n\nimport com.twitter.home_mixer.functional_component.selector.UpdateNewTweetsPillDecoration.NumAvatars\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableNewTweetsPillAvatarsParam\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtItemPresentation\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ShowAlert\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.stringcenter.client.core.ExternalString\n\nobject UpdateNewTweetsPillDecoration {\n  val NumAvatars = 3\n}\n\ncase class UpdateNewTweetsPillDecoration[Query <: PipelineQuery with HasDeviceContext](\n  override val pipelineScope: CandidateScope,\n  stringCenter: StringCenter,\n  seeNewTweetsString: ExternalString,\n  tweetedString: ExternalString)\n    extends Selector[Query] {\n\n  override def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val (alerts, otherCandidates) =\n      remainingCandidates.partition(candidate =>\n        candidate.isCandidateType[ShowAlertCandidate]() && pipelineScope.contains(candidate))\n    val updatedCandidates = alerts\n      .collectFirst {\n        case newTweetsPill: ItemCandidateWithDetails =>\n          val userIds = CandidatesUtil\n            .getItemCandidatesWithOnlyModuleLast(result)\n            .filter(candidate =>\n              candidate.isCandidateType[TweetCandidate]() && pipelineScope.contains(candidate))\n            .filterNot(_.features.getOrElse(IsRetweetFeature, false))\n            .flatMap(_.features.getOrElse(AuthorIdFeature, None))\n            .filterNot(_ == query.getRequiredUserId)\n            .distinct\n\n          val updatedPresentation = newTweetsPill.presentation.map {\n            case presentation: UrtItemPresentation =>\n              presentation.timelineItem match {\n                case alert: ShowAlert =>\n                  val text = if (useAvatars(query, userIds)) tweetedString else seeNewTweetsString\n                  val richText = RichText(\n                    text = stringCenter.prepare(text),\n                    entities = List.empty,\n                    rtl = None,\n                    alignment = None)\n\n                  val updatedAlert =\n                    alert.copy(userIds = Some(userIds.take(NumAvatars)), richText = Some(richText))\n                  presentation.copy(timelineItem = updatedAlert)\n              }\n          }\n          otherCandidates :+ newTweetsPill.copy(presentation = updatedPresentation)\n      }.getOrElse(remainingCandidates)\n\n    SelectorResult(remainingCandidates = updatedCandidates, result = result)\n  }\n\n  private def useAvatars(query: Query, userIds: Seq[Long]): Boolean = {\n    val enableAvatars = query.params(EnableNewTweetsPillAvatarsParam)\n    enableAvatars && userIds.size >= NumAvatars\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect/BUILD.bazel",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"eventbus/client/src/main/scala/com/twitter/eventbus/client\",\n        \"finagle/finagle-mysql/src/main/scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/content\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/light_ranking_features\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/non_ml_features\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/twhin_embeddings\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/scorer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timeline_logging\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweet_mixer\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/communities_to_join\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/job\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/recruiting_organization\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_subscribe_module\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"scribelib/marshallers/src/main/scala/com/twitter/scribelib/marshallers\",\n        \"src/scala/com/twitter/suggests/controller_data\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/twistly\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/two_hop_features\",\n        \"src/scala/com/twitter/timelines/prediction/features/large_embeddings\",\n        \"src/thrift/com/twitter/timelines/impression_store:thrift-scala\",\n        \"src/thrift/com/twitter/timelines/served_candidates_logging:served_candidates_logging-scala\",\n        \"src/thrift/com/twitter/timelines/suggests/common:data_record_metadata-scala\",\n        \"src/thrift/com/twitter/timelines/suggests/common:poly_data_record-java\",\n        \"src/thrift/com/twitter/timelines/timeline_logging:thrift-scala\",\n        \"strato/config/columns/videoRecommendations/twitterClip:twitterClip-strato-client\",\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence\",\n        \"timelines/ml:pldr-client\",\n        \"timelines/ml:pldr-conversion\",\n        \"timelines/ml/cont_train/common/domain/src/main/scala/com/twitter/timelines/ml/cont_train/common/domain/non_scalding\",\n        \"timelines/src/main/scala/com/twitter/timelines/clientconfig\",\n        \"timelines/src/main/scala/com/twitter/timelines/util/stats\",\n        \"timelineservice/common:model\",\n        \"tweet-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"user_session_store/src/main/scala/com/twitter/user_session_store\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect/BaseCacheCandidateFeaturesSideEffect.scala",
    "content": "package com.twitter.home_mixer.functional_component.side_effect\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.filter.OffloadFilter\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mysql.Client\nimport com.twitter.finagle.mysql.Transactions\nimport com.twitter.finagle.offload.OffloadFuturePool\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.transport.Transport\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.content.ClipEmbeddingFeaturesAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.content.TextTokensFeaturesAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.light_ranking_features.LightRankingCandidateFeatures\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.light_ranking_features.LightRankingCandidateFeaturesAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.non_ml_features.NonMLCandidateFeatures\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.non_ml_features.NonMLCandidateFeaturesAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.non_ml_features.NonMLCommonFeatures\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.non_ml_features.NonMLCommonFeaturesAdapter\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.twhin_embeddings.TwhinVideoEmbeddingsAdapter\nimport com.twitter.home_mixer.functional_component.scorer.CandidateFeaturesDataRecordFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ClientIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GuestIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PredictionRequestIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTextTokensFeature\nimport com.twitter.home_mixer.model.HomeFeatures.WeightedModelScoreFeature\nimport com.twitter.home_mixer.model.PredictedScoreFeature.PredictedScoreFeatureSet\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.model.request.ScoredTweetsProduct\nimport com.twitter.home_mixer.model.request.ScoredVideoTweetsProduct\nimport com.twitter.home_mixer.model.request.SubscribedProduct\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableClipEmbeddingFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTweetTextTokensEmbeddingFeatureScribingParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTweetVideoAggregatedWatchTimeFeatureScribingParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinVideoFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableVideoClipEmbeddingFeatureHydrationDeciderParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.IsSelectedByHeavyRankerCountParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.util.CandidatesUtil.getOriginalAuthorId\nimport com.twitter.home_mixer.util.CandidatesUtil.getOriginalTweetId\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.DataRecordMerger\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordConverter\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.SpecificFeatures\nimport com.twitter.product_mixer.core.functional_component.common.alert.SuccessRateAlert\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateSourcePosition\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.simclusters_v2.thriftscala.TwhinTweetEmbedding\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus.Store\nimport com.twitter.strato.generated.client.videoRecommendations.twitterClip.TwitterClipEmbeddingMhClientColumn\nimport com.twitter.timelines.ml.cont_train.common.domain.non_scalding.CandidateAndCommonFeaturesStreamingUtils\nimport com.twitter.timelines.ml.pldr.client.MysqlClientUtils\nimport com.twitter.timelines.ml.pldr.client.VersionedMetadataCacheClient\nimport com.twitter.timelines.ml.pldr.conversion.VersionIdAndFeatures\nimport com.twitter.timelines.prediction.adapters.twistly.VideoAggregatedWatchTimeFeaturesAdapter\nimport com.twitter.timelines.prediction.features.large_embeddings.LargeEmbeddingsFeatures.AllCandidateLargeEmbeddingsFeatures\nimport com.twitter.timelines.served_candidates_logging.{thriftscala => sc}\nimport com.twitter.timelines.suggests.common.data_record_metadata.{thriftscala => drmd}\nimport com.twitter.timelines.suggests.common.poly_data_record.{thriftjava => pldr}\nimport com.twitter.timelines.util.stats.FutureObserver\nimport com.twitter.timelines.util.stats.OptionObserver\nimport com.twitter.twistly.thriftscala.VideoViewEngagementType\nimport com.twitter.twistly.thriftscala.WatchTimeMetadata\nimport com.twitter.twistly.{thriftscala => ts}\nimport com.twitter.util.Future\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\nimport scala.collection.mutable.ArrayBuffer\n\n@Singleton\nclass BaseCacheCandidateFeaturesSideEffect @Inject() (\n  dataRecordMetadataStoreConfigsYml: String,\n  store: Store[\n    sc.CandidateFeatureKey,\n    pldr.PolyDataRecord\n  ],\n  tweetWatchTimeMetadataStore: ReadableStore[(Long, VideoViewEngagementType), WatchTimeMetadata],\n  twitterClipEmbeddingMhClientColumn: TwitterClipEmbeddingMhClientColumn,\n  twhinVideoStore: ReadableStore[Long, TwhinTweetEmbedding],\n  statsReceiver: StatsReceiver)\n    extends PipelineResultSideEffect[PipelineQuery, HasMarshalling]\n    with Conditionally[PipelineQuery, HasMarshalling]\n    with Logging {\n\n  override def onlyIf(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Boolean = {\n    val serviceIdentifier = ServiceIdentifier.fromCertificate(Transport.peerCertificate)\n    (selectedCandidates.nonEmpty || remainingCandidates.nonEmpty || droppedCandidates.nonEmpty) && serviceIdentifier.role != \"video-mixer\"\n  }\n\n  override val alerts: Seq[SuccessRateAlert] = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(98.5))\n\n  val identifier: SideEffectIdentifier =\n    SideEffectIdentifier(\"BaseCacheCandidateFeaturesSideEffect\")\n  val statScope: String = this.getClass.getSimpleName\n\n  private val scopedStatsReceiver = statsReceiver.scope(statScope)\n  private val metadataFetchFailedCounter = scopedStatsReceiver.counter(\"metadataFetchFailed\")\n  private val writesRequestCounter = scopedStatsReceiver.counter(\"writesRequests\")\n  private val writesFailedCounter = scopedStatsReceiver.counter(\"writesFailed\")\n\n  private val twhinVideoEmbeddingFutureObserver = FutureObserver(\n    scopedStatsReceiver.scope(\"twhinVideoEmbedding\"))\n  private val twhinVideoEmbeddingOptionObserver = OptionObserver(\n    scopedStatsReceiver.scope(\"twhinVideoEmbedding\"))\n\n  private val clipEmbeddingCounter = scopedStatsReceiver.counter(\"clipEmbeddingCounter\")\n\n  private val tweetWatchTimeMetadataRequestCounter =\n    scopedStatsReceiver.counter(\"tweetWatchTimeMetadataRequests\")\n  private val tweetWatchTimeMetadataSuccessCounter =\n    scopedStatsReceiver.counter(\"tweetWatchTimeMetadataSuccessCounter\")\n  private val tweetWatchTimeMetadataFailureCounter =\n    scopedStatsReceiver.counter(\"tweetWatchTimeMetadataFailureCounter\")\n\n  private val drMerger = new DataRecordMerger\n  private val predictedScoreFeaturesDataRecordAdapter =\n    new DataRecordConverter(SpecificFeatures(PredictedScoreFeatureSet))\n\n  lazy private val dataRecordMetadataStoreClient: Option[Client with Transactions] = Try {\n    try {\n      val c = MysqlClientUtils.parseConfigFromYaml(dataRecordMetadataStoreConfigsYml)\n      logger.info(s\"pldr mysql config: ${c.host} ${c.port} ${c.user} ${c.database}\")\n    } catch {\n      case e: Throwable =>\n        logger.error(\"pldr mysql error: \" + e.toString)\n    }\n\n    MysqlClientUtils.mysqlClientProvider(\n      MysqlClientUtils.parseConfigFromYaml(dataRecordMetadataStoreConfigsYml)\n    )\n  }.toOption\n\n  lazy private val versionedMetadataCacheClientOpt: Option[\n    VersionedMetadataCacheClient[Map[drmd.FeaturesCategory, Option[VersionIdAndFeatures]]]\n  ] = dataRecordMetadataStoreClient.map { mysqlClient =>\n    new VersionedMetadataCacheClient[Map[drmd.FeaturesCategory, Option[VersionIdAndFeatures]]](\n      maximumSize = 1,\n      expireDurationOpt = None,\n      mysqlClient = mysqlClient,\n      transform = CandidateAndCommonFeaturesStreamingUtils.metadataTransformer,\n      statsReceiver = statsReceiver\n    )\n  }\n\n  versionedMetadataCacheClientOpt.foreach {\n    _.metadataFetchTimerTask(\n      CandidateAndCommonFeaturesStreamingUtils.metadataFetchKey,\n      metadataFetchTimer = DefaultTimer,\n      metadataFetchInterval = 90.seconds,\n      metadataFetchFailedCounter = metadataFetchFailedCounter\n    )\n  }\n\n  override def apply(\n    inputs: PipelineResultSideEffect.Inputs[PipelineQuery, HasMarshalling]\n  ): Stitch[Unit] = Stitch.let(OffloadFuturePool.lowPriorityLocal)(()) {\n    OffloadFuturePools.offloadStitch {\n\n      // Exclude unscored candidates (e.g. because of scoring QF truncation or cache reads)\n      val selectedCandidates =\n        (inputs.selectedCandidates ++ inputs.remainingCandidates)\n          .filter { candidate =>\n            candidate.features.contains(CandidateFeaturesDataRecordFeature)\n          }\n      val droppedCandidates =\n        inputs.droppedCandidates\n          .filter { candidate =>\n            candidate.features.contains(CandidateFeaturesDataRecordFeature)\n          }\n\n      val candidates = (selectedCandidates ++ droppedCandidates)\n      val isSelectedCandidateIds: Set[Long] = selectedCandidates.map(_.candidateIdLong).toSet\n      val candidatesHeavyRankerScoreBasedRank: Map[Long, Int] = candidates\n        .sortBy(-_.features\n          .getOrElse(WeightedModelScoreFeature, None).getOrElse(Double.NegativeInfinity)).map(\n          _.candidateIdLong).zipWithIndex.toMap\n      val isSelectedByHeavyRankerCount = inputs.query.params(IsSelectedByHeavyRankerCountParam)\n\n      val predictionRequestId = candidates.headOption.flatMap { candidate =>\n        candidate.features.getOrElse(PredictionRequestIdFeature, None)\n      }\n\n      val productSurface = inputs.query.product match {\n        case FollowingProduct => hmt.Product.Following\n        case ForYouProduct => hmt.Product.ForYou\n        case ScoredTweetsProduct => hmt.Product.ScoredTweets\n        case ScoredVideoTweetsProduct => hmt.Product.ScoredVideoTweets\n        case SubscribedProduct => hmt.Product.Subscribed\n        case other => throw new UnsupportedOperationException(s\"Unknown product: $other\")\n      }\n\n      val nonMLCommonFeatures = NonMLCommonFeatures(\n        userId = inputs.query.getRequiredUserId,\n        guestId = inputs.query.features.flatMap(_.getOrElse(GuestIdFeature, None)),\n        clientId = inputs.query.features.flatMap(_.getOrElse(ClientIdFeature, None)),\n        countryCode = inputs.query.getCountryCode,\n        predictionRequestId = predictionRequestId,\n        productSurface = productSurface.toString,\n        servedTimestamp = inputs.query.queryTime.inMilliseconds\n      )\n\n      val nonMLCommonFeaturesDataRecord =\n        NonMLCommonFeaturesAdapter.adaptToDataRecords(nonMLCommonFeatures).asScala.head\n\n      val sideEffectStitch = fetchExperimentalFeatures(inputs.query, candidates)\n        .map { candidatesAndFeatures =>\n          // Further writes are cheap and do not have callbacks,\n          // therefore it's safe to bypass offload filter.\n          OffloadFilter.withOffloadsDisabled {\n            candidatesAndFeatures.foreach {\n              case (candidate, experimentalFeatures) =>\n                val candidateFeaturesPldr = buildCandidateFeaturesPldr(\n                  query = inputs.query,\n                  candidate = candidate,\n                  isSelected = isSelectedCandidateIds.contains(candidate.candidateIdLong),\n                  isSelectedByHeavyRanker = candidatesHeavyRankerScoreBasedRank\n                    .getOrElse(\n                      candidate.candidateIdLong,\n                      candidates.size) < isSelectedByHeavyRankerCount,\n                  rankByHeavyRanker = candidatesHeavyRankerScoreBasedRank\n                    .getOrElse(candidate.candidateIdLong, candidates.size),\n                  nonMLCommonFeaturesDataRecord = nonMLCommonFeaturesDataRecord,\n                  experimentalFeaturesDataRecords = experimentalFeatures,\n                )\n\n                writesRequestCounter.incr()\n                val candidateFeaturesKey = sc.CandidateFeatureKey(\n                  tweetId = candidate.candidateIdLong,\n                  viewerId = inputs.query.getRequiredUserId,\n                  servedId = predictionRequestId.getOrElse(-1L)\n                )\n\n                store.put(candidateFeaturesKey -> candidateFeaturesPldr).rescue {\n                  case _: Throwable =>\n                    writesFailedCounter.incr()\n                    Future.Unit\n                }\n            }\n          }\n        }\n\n      Stitch.run(sideEffectStitch)\n\n      Stitch.Unit\n    }\n  }\n\n  private def buildCandidateFeaturesPldr(\n    query: PipelineQuery,\n    candidate: CandidateWithDetails,\n    isSelected: Boolean,\n    isSelectedByHeavyRanker: Boolean,\n    rankByHeavyRanker: Int,\n    nonMLCommonFeaturesDataRecord: DataRecord,\n    experimentalFeaturesDataRecords: Seq[DataRecord]\n  ): Option[pldr.PolyDataRecord] = {\n    // Step 1) Set candidate features to all existing candidate features used in ranking\n    val candidateFeaturesDataRecord =\n      candidate.features.get(CandidateFeaturesDataRecordFeature)\n\n    // Step 2) Remove all large embeddings features from DataRecord\n    AllCandidateLargeEmbeddingsFeatures.foreach { feature =>\n      candidateFeaturesDataRecord.tensors.remove(feature.getFeatureId)\n    }\n\n    // Step 3) Add prediction score\n    val predictedScoreFeaturesDataRecord =\n      predictedScoreFeaturesDataRecordAdapter.toDataRecord(candidate.features)\n    drMerger.merge(candidateFeaturesDataRecord, predictedScoreFeaturesDataRecord)\n\n    // Step 4) Add non-ML common features\n    drMerger.merge(candidateFeaturesDataRecord, nonMLCommonFeaturesDataRecord)\n\n    // Step 5) Add non-ML candidate features, including light ranking features\n    val nonMLCandidateFeatures = NonMLCandidateFeatures(\n      tweetId = candidate.candidateIdLong,\n      sourceTweetId = getOriginalTweetId(candidate.candidateIdLong, candidate.features),\n      originalAuthorId = getOriginalAuthorId(candidate.features)\n    )\n    val nonMLCandidateFeaturesDataRecord =\n      NonMLCandidateFeaturesAdapter.adaptToDataRecords(nonMLCandidateFeatures).asScala.head\n    val lightRankingCandidateFeatures = LightRankingCandidateFeatures(\n      isSelected = isSelected,\n      isSelectedByHeavyRanker = isSelectedByHeavyRanker,\n      rankByHeavyRanker = rankByHeavyRanker,\n      servedType = candidate.features.get(ServedTypeFeature),\n      candidateSourcePosition = candidate.features.get(CandidateSourcePosition).toLong\n    )\n    val lightRankingCandidateFeaturesDataRecord =\n      LightRankingCandidateFeaturesAdapter\n        .adaptToDataRecords(lightRankingCandidateFeatures).asScala.head\n    drMerger.merge(nonMLCandidateFeaturesDataRecord, lightRankingCandidateFeaturesDataRecord)\n    drMerger.merge(candidateFeaturesDataRecord, nonMLCandidateFeaturesDataRecord)\n\n    // Step 5) Add experimental features (including twhin)\n    experimentalFeaturesDataRecords.foreach(drMerger.merge(candidateFeaturesDataRecord, _))\n\n    CandidateAndCommonFeaturesStreamingUtils.candidateFeaturesToPolyDataRecord(\n      versionedMetadataCacheClientOpt = versionedMetadataCacheClientOpt,\n      candidateFeatures = candidateFeaturesDataRecord,\n      valueFormat = pldr.PolyDataRecord._Fields.LITE_COMPACT_DATA_RECORD\n    )\n  }\n\n  private def fetchExperimentalFeatures(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithDetails],\n  ): Stitch[Map[CandidateWithDetails, Seq[DataRecord]]] = {\n    val stitches = Seq(\n      fetchTwhinVideoEmbeddingDataRecords(query, candidates),\n      fetchTweetTextTokensDataRecord(query, candidates),\n      fetchClipEmbeddings(query, candidates),\n      fetchTweetVideoAggregatedWatchTimeDataRecord(query, candidates),\n    )\n    Stitch\n      .collect(stitches)\n      .map { candidatesMaps =>\n        val candidatesToDataRecords =\n          new java.util.HashMap[CandidateWithDetails, ArrayBuffer[DataRecord]](\n            candidates.size * 3 / 4)\n        candidatesMaps.foreach { candidatesToDataRecord =>\n          candidatesToDataRecord.foreach {\n            case (candidate, dataRecord) =>\n              if (dataRecord.isDefined) {\n                candidatesToDataRecords\n                  .computeIfAbsent(\n                    candidate,\n                    _ => new ArrayBuffer[DataRecord](stitches.size)\n                  ).append(dataRecord.get)\n              }\n          }\n        }\n        candidatesToDataRecords.asScala.toMap\n      }\n  }\n\n  private def fetchTwhinVideoEmbeddingDataRecords(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithDetails],\n  ): Stitch[Map[CandidateWithDetails, Option[DataRecord]]] = {\n    if (!query.params(EnableTwhinVideoFeaturesParam)) {\n      Stitch.value(Map.empty)\n    } else {\n      val originalTweetToCandidates = candidates.groupBy { CandidatesUtil.getOriginalTweetId(_) }\n      Stitch.callFuture {\n        twhinVideoEmbeddingFutureObserver(\n          Future.collect(twhinVideoStore.multiGet(originalTweetToCandidates.keySet)).map {\n            originalTweetIdToEmbeddings =>\n              originalTweetIdToEmbeddings.flatMap {\n                case (originalTweetId, embeddingOpt) =>\n                  val floatTensor = twhinVideoEmbeddingOptionObserver(embeddingOpt)\n                    .map { e => ml.FloatTensor(e.embedding) }\n                  val dataRecord =\n                    TwhinVideoEmbeddingsAdapter.adaptToDataRecords(floatTensor).asScala.headOption\n                  originalTweetToCandidates(originalTweetId).map { candidate =>\n                    candidate -> dataRecord\n                  }\n              }\n          }\n        )\n      }\n    }\n  }\n\n  private def fetchClipEmbeddings(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithDetails],\n  ): Stitch[Map[CandidateWithDetails, Option[DataRecord]]] = {\n    Stitch\n      .collect(\n        candidates.map { candidate =>\n          if (query.params(EnableClipEmbeddingFeaturesParam) &&\n            candidate.features.getOrElse(HasVideoFeature, false) &&\n            !query.params(\n              EnableVideoClipEmbeddingFeatureHydrationDeciderParam\n            )) { // If it's not enabled through feature hydrator\n            clipEmbeddingCounter.incr()\n            twitterClipEmbeddingMhClientColumn.fetcher\n              .fetch(candidate.candidateIdLong)\n              .map { result =>\n                candidate -> result.v.flatMap { record =>\n                  ClipEmbeddingFeaturesAdapter\n                    .adaptToDataRecords(record)\n                    .asScala\n                    .headOption\n                }\n              }\n          } else {\n            Stitch.value(candidate -> None)\n          }\n        }\n      ).map(_.toMap)\n  }\n\n  private def fetchTweetTextTokensDataRecord(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithDetails],\n  ): Stitch[Map[CandidateWithDetails, Option[DataRecord]]] = {\n    Stitch.value {\n      if (query.params(EnableTweetTextTokensEmbeddingFeatureScribingParam)) {\n        Map.empty\n      } else {\n        candidates.map { candidate =>\n          candidate -> candidate.features.getOrElse(TweetTextTokensFeature, None).flatMap {\n            textTokens =>\n              TextTokensFeaturesAdapter\n                .adaptToDataRecords(textTokens)\n                .asScala\n                .headOption\n          }\n        }.toMap\n      }\n    }\n  }\n  private def fetchTweetVideoAggregatedWatchTimeDataRecord(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithDetails],\n  ): Stitch[Map[CandidateWithDetails, Option[DataRecord]]] = {\n    Stitch\n      .collect(\n        candidates.map { candidate =>\n          if (query.params(\n              EnableTweetVideoAggregatedWatchTimeFeatureScribingParam) && candidate.features\n              .getOrElse(HasVideoFeature, false)) {\n            tweetWatchTimeMetadataRequestCounter.incr()\n            Stitch.callFuture(\n              tweetWatchTimeMetadataStore\n                .get(\n                  (candidate.candidateIdLong, ts.VideoViewEngagementType.ImmersiveVideoWatchTime)\n                ).onSuccess { _ => tweetWatchTimeMetadataSuccessCounter.incr() }\n                .onFailure { _ => tweetWatchTimeMetadataFailureCounter.incr() }\n                .map { opt =>\n                  candidate -> opt.flatMap(VideoAggregatedWatchTimeFeaturesAdapter\n                    .adaptToDataRecords(_).asScala.headOption)\n                }\n            )\n          } else {\n            Stitch.value(candidate -> None)\n          }\n        }\n      ).map(_.toMap)\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect/ClientEventsBuilder.scala",
    "content": "package com.twitter.home_mixer.functional_component.side_effect\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.functional_component.decorator.HomeQueryTypePredicates\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeTweetTypePredicates\nimport com.twitter.home_mixer.model.HomeFeatures.AccountAgeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTypeMetricsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoDurationMsFeature\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.model.request.SubscribedProduct\nimport com.twitter.product_mixer.component_library.side_effect.ScribeClientEventSideEffect.ClientEvent\nimport com.twitter.product_mixer.component_library.side_effect.ScribeClientEventSideEffect.EventNamespace\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.suggests.controller_data.Home\n\nprivate[side_effect] sealed trait ClientEventsBuilder {\n  private val FollowingSection = Some(\"latest\")\n  private val ForYouSection = Some(\"home\")\n  private val SubscribedSection = Some(\"subscribed\")\n\n  protected def section(query: PipelineQuery): Option[String] = {\n    query.product match {\n      case FollowingProduct => FollowingSection\n      case ForYouProduct => ForYouSection\n      case SubscribedProduct => SubscribedSection\n      case other => throw new UnsupportedOperationException(s\"Unknown product: $other\")\n    }\n  }\n\n  protected def count(\n    candidates: Seq[CandidateWithDetails],\n    predicate: FeatureMap => Boolean = _ => true,\n    queryFeatures: FeatureMap = FeatureMap.empty\n  ): Option[Long] = Some(candidates.view.count(item => predicate(item.features ++ queryFeatures)))\n\n  protected def sum(\n    candidates: Seq[CandidateWithDetails],\n    predicate: FeatureMap => Option[Int],\n    queryFeatures: FeatureMap = FeatureMap.empty\n  ): Option[Long] =\n    Some(candidates.view.flatMap(item => predicate(item.features ++ queryFeatures)).sum)\n}\n\nprivate[side_effect] object ServedEventsBuilder extends ClientEventsBuilder {\n\n  private val ServedTweetsAction = Some(\"served_tweets\")\n  private val ServedUsersAction = Some(\"served_users\")\n  private val ServedCommunitiesAction = Some(\"served_communities\")\n  private val ServedPromptsAction = Some(\"served_prompts\")\n  private val InjectedComponent = Some(\"injected\")\n  private val PromotedComponent = Some(\"promoted\")\n  private val WhoToFollowComponent = Some(\"who_to_follow\")\n  private val WhoToSubscribeComponent = Some(\"who_to_subscribe\")\n  private val CommunitiesToJoinComponent = Some(\"communities_to_join\")\n  private val RelevancePromptComponent = Some(\"for_you_survey_feed\")\n  private val WithVideoDurationComponent = Some(\"with_video_duration\")\n  private val VideoDurationSumElement = Some(\"video_duration_sum\")\n  private val NumVideosElement = Some(\"num_videos\")\n\n  def tweetTypePredicate(predicateName: String): FeatureMap => Boolean = {\n    val predicateIdxOpt = Home.ItemTypeIdxMap.get(predicateName)\n    if (predicateIdxOpt.nonEmpty) {\n      val predicateIdx = predicateIdxOpt.get\n      featureMap: FeatureMap => {\n        featureMap\n          .getOrElse(TweetTypeMetricsFeature, None)\n          .map { tweetTypeMetrics =>\n            java.util.BitSet\n              .valueOf(tweetTypeMetrics.toArray)\n              .get(predicateIdx)\n          }.getOrElse(false)\n      }\n    } else _ => false\n  }\n  def build(\n    query: PipelineQuery,\n    injectedTweets: Seq[ItemCandidateWithDetails],\n    promotedTweets: Seq[ItemCandidateWithDetails],\n    whoToFollowUsers: Seq[ItemCandidateWithDetails],\n    whoToSubscribeUsers: Seq[ItemCandidateWithDetails],\n    communititesToJoin: Seq[ItemCandidateWithDetails],\n    relevancePrompt: Seq[ItemCandidateWithDetails]\n  ): Seq[ClientEvent] = {\n    val baseEventNamespace = EventNamespace(\n      section = section(query),\n      action = ServedTweetsAction\n    )\n    val overallServedEvents = Seq(\n      ClientEvent(baseEventNamespace, eventValue = count(injectedTweets ++ promotedTweets)),\n      ClientEvent(\n        baseEventNamespace.copy(component = InjectedComponent),\n        eventValue = count(injectedTweets)),\n      ClientEvent(\n        baseEventNamespace.copy(component = PromotedComponent),\n        eventValue = count(promotedTweets)),\n      ClientEvent(\n        baseEventNamespace.copy(component = WhoToFollowComponent, action = ServedUsersAction),\n        eventValue = count(whoToFollowUsers)),\n      ClientEvent(\n        baseEventNamespace.copy(component = WhoToSubscribeComponent, action = ServedUsersAction),\n        eventValue = count(whoToSubscribeUsers)),\n      ClientEvent(\n        baseEventNamespace\n          .copy(component = CommunitiesToJoinComponent, action = ServedCommunitiesAction),\n        eventValue = count(communititesToJoin)),\n      ClientEvent(\n        baseEventNamespace\n          .copy(component = RelevancePromptComponent, action = ServedPromptsAction),\n        eventValue = count(relevancePrompt)),\n    )\n\n    val tweetTypeServedEvents = HomeTweetTypePredicates.PredicateMap.map {\n      case (tweetType, predicate) =>\n        ClientEvent(\n          baseEventNamespace.copy(component = InjectedComponent, element = Some(tweetType)),\n          eventValue = count(\n            injectedTweets,\n            tweetTypePredicate(tweetType),\n            query.features.getOrElse(FeatureMap.empty))\n        )\n    }.toSeq\n\n    val servedTypeServedEvents = injectedTweets\n      .map(_.features.get(ServedTypeFeature))\n      .groupBy(identity).map {\n        case (servedType, group) =>\n          ClientEvent(\n            baseEventNamespace.copy(component = Some(servedType.originalName)),\n            eventValue = Some(group.size.toLong))\n      }.toSeq\n\n    // Video duration events\n    val numVideosEvent = ClientEvent(\n      baseEventNamespace.copy(component = WithVideoDurationComponent, element = NumVideosElement),\n      eventValue = count(injectedTweets, _.getOrElse(VideoDurationMsFeature, None).nonEmpty)\n    )\n    val videoDurationSumEvent = ClientEvent(\n      baseEventNamespace\n        .copy(component = WithVideoDurationComponent, element = VideoDurationSumElement),\n      eventValue = sum(injectedTweets, _.getOrElse(VideoDurationMsFeature, None))\n    )\n    val videoEvents = Seq(numVideosEvent, videoDurationSumEvent)\n\n    overallServedEvents ++ tweetTypeServedEvents ++ servedTypeServedEvents ++ videoEvents\n  }\n}\n\nprivate[side_effect] object EmptyTimelineEventsBuilder extends ClientEventsBuilder {\n  private val EmptyAction = Some(\"empty\")\n  private val AccountAgeLessThan30MinutesComponent = Some(\"account_age_less_than_30_minutes\")\n  private val ServedNonPromotedTweetElement = Some(\"served_non_promoted_tweet\")\n\n  def build(\n    query: PipelineQuery,\n    injectedTweets: Seq[ItemCandidateWithDetails]\n  ): Seq[ClientEvent] = {\n    val baseEventNamespace = EventNamespace(\n      section = section(query),\n      action = EmptyAction\n    )\n\n    // Empty timeline events\n    val accountAgeLessThan30Minutes = query.features\n      .flatMap(_.getOrElse(AccountAgeFeature, None))\n      .exists(_.untilNow < 30.minutes)\n    val isEmptyTimeline = count(injectedTweets).contains(0L)\n    val predicates = Seq(\n      None -> isEmptyTimeline,\n      AccountAgeLessThan30MinutesComponent -> (isEmptyTimeline && accountAgeLessThan30Minutes)\n    )\n    for {\n      (component, predicate) <- predicates\n      if predicate\n    } yield ClientEvent(\n      baseEventNamespace.copy(component = component, element = ServedNonPromotedTweetElement))\n  }\n}\n\nprivate[side_effect] object QueryEventsBuilder extends ClientEventsBuilder {\n\n  private val ServedSizePredicateMap: Map[String, Int => Boolean] = Map(\n    (\"size_is_empty\", _ <= 0),\n    (\"size_at_most_5\", _ <= 5),\n    (\"size_at_most_10\", _ <= 10),\n    (\"size_at_most_35\", _ <= 35)\n  )\n\n  def build(\n    query: PipelineQuery,\n    injectedTweets: Seq[ItemCandidateWithDetails]\n  ): Seq[ClientEvent] = {\n    val baseEventNamespace = EventNamespace(\n      section = section(query)\n    )\n    val queryFeatureMap = query.features.getOrElse(FeatureMap.empty)\n    val servedSizeQueryEvents =\n      for {\n        (queryPredicateName, queryPredicate) <- HomeQueryTypePredicates.PredicateMap\n        if queryPredicate(queryFeatureMap)\n        (servedSizePredicateName, servedSizePredicate) <- ServedSizePredicateMap\n        if servedSizePredicate(injectedTweets.size)\n      } yield ClientEvent(\n        baseEventNamespace\n          .copy(component = Some(servedSizePredicateName), action = Some(queryPredicateName)))\n    servedSizeQueryEvents.toSeq\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect/CommonFeaturesPldrConverter.scala",
    "content": "package com.twitter.home_mixer.functional_component.side_effect\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mysql.Client\nimport com.twitter.finagle.mysql.Transactions\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.non_ml_features.NonMLCommonFeatures\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.non_ml_features.NonMLCommonFeaturesAdapter\nimport com.twitter.home_mixer.functional_component.scorer.CommonFeaturesDataRecordFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ClientIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GuestIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PredictionRequestIdFeature\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.model.request.ScoredTweetsProduct\nimport com.twitter.home_mixer.model.request.ScoredVideoTweetsProduct\nimport com.twitter.home_mixer.model.request.SubscribedProduct\nimport com.twitter.home_mixer.param.HomeGlobalParams\nimport com.twitter.home_mixer.param.HomeMixerFlagName.DataRecordMetadataStoreConfigsYmlFlag\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.ml.api.DataRecordMerger\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.ml.cont_train.common.domain.non_scalding.CandidateAndCommonFeaturesStreamingUtils\nimport com.twitter.timelines.ml.pldr.client.MysqlClientUtils\nimport com.twitter.timelines.ml.pldr.client.VersionedMetadataCacheClient\nimport com.twitter.timelines.ml.pldr.conversion.VersionIdAndFeatures\nimport com.twitter.timelines.prediction.features.large_embeddings.LargeEmbeddingsFeatures.AllCommonLargeEmbeddingsFeatures\nimport com.twitter.timelines.suggests.common.data_record_metadata.{thriftscala => drmd}\nimport com.twitter.timelines.suggests.common.poly_data_record.{thriftjava => pldr}\nimport com.twitter.timelines.util.stats.OptionObserver\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\n@Singleton\nclass CommonFeaturesPldrConverter @Inject() (\n  @Flag(DataRecordMetadataStoreConfigsYmlFlag) dataRecordMetadataStoreConfigsYml: String,\n  statsReceiver: StatsReceiver) {\n\n  private val drMerger = new DataRecordMerger\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val metadataFetchFailedCounter = scopedStatsReceiver.counter(\"metadataFetchFailed\")\n\n  private val commonFeaturesPLDROptionObserver =\n    OptionObserver(scopedStatsReceiver.scope(\"commonFeaturesPLDR\"))\n\n  private lazy val dataRecordMetadataStoreClient: Option[Client with Transactions] = Try {\n    MysqlClientUtils.mysqlClientProvider(\n      MysqlClientUtils.parseConfigFromYaml(dataRecordMetadataStoreConfigsYml)\n    )\n  }.toOption\n\n  private lazy val versionedMetadataCacheClientOpt: Option[\n    VersionedMetadataCacheClient[Map[drmd.FeaturesCategory, Option[VersionIdAndFeatures]]]\n  ] = dataRecordMetadataStoreClient.map { mysqlClient =>\n    new VersionedMetadataCacheClient[Map[drmd.FeaturesCategory, Option[VersionIdAndFeatures]]](\n      maximumSize = 1,\n      expireDurationOpt = None,\n      mysqlClient = mysqlClient,\n      transform = CandidateAndCommonFeaturesStreamingUtils.metadataTransformer,\n      statsReceiver = statsReceiver\n    )\n  }\n\n  versionedMetadataCacheClientOpt.foreach {\n    _.metadataFetchTimerTask(\n      CandidateAndCommonFeaturesStreamingUtils.metadataFetchKey,\n      metadataFetchTimer = DefaultTimer,\n      metadataFetchInterval = 90.seconds,\n      metadataFetchFailedCounter = metadataFetchFailedCounter\n    )\n  }\n\n  /**\n   * Get the common features data record converted to PLDR format with prediction request ID\n   *\n   * @param query\n   * @param selectedCandidates\n   * @return prediction request ID and the common features PLDR for the given request\n   */\n  def getCommonFeaturesPldr(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails]\n  ): Option[(Long, pldr.PolyDataRecord)] = {\n    // Exclude unscored candidates (e.g. because of scoring QF truncation or cache reads)\n    val candidatesHead = selectedCandidates\n      .find { candidate =>\n        candidate.features.contains(CommonFeaturesDataRecordFeature)\n      }\n\n    if (candidatesHead.nonEmpty) {\n      val candidateFeaturesHead = candidatesHead.get.features\n      val predictionRequestId = candidateFeaturesHead.getOrElse(PredictionRequestIdFeature, None)\n      val productSurface = query.product match {\n        case FollowingProduct => hmt.Product.Following\n        case ForYouProduct => hmt.Product.ForYou\n        case ScoredTweetsProduct => hmt.Product.ScoredTweets\n        case ScoredVideoTweetsProduct => hmt.Product.ScoredVideoTweets\n        case SubscribedProduct => hmt.Product.Subscribed\n        case other => throw new UnsupportedOperationException(s\"Unknown product: $other\")\n      }\n      val nonMLCommonFeatures = NonMLCommonFeatures(\n        userId = query.getRequiredUserId,\n        guestId = candidateFeaturesHead.getOrElse(GuestIdFeature, None),\n        clientId = candidateFeaturesHead.getOrElse(ClientIdFeature, None),\n        countryCode = query.getCountryCode,\n        predictionRequestId = predictionRequestId,\n        productSurface = productSurface.toString,\n        servedTimestamp = query.queryTime.inMilliseconds\n      )\n      val nonMLCommonFeaturesDataRecord =\n        NonMLCommonFeaturesAdapter.adaptToDataRecords(nonMLCommonFeatures).asScala.head\n\n      val commonFeaturesDataRecord = candidateFeaturesHead.get(CommonFeaturesDataRecordFeature)\n      //Remove large embeddings from dataRecord\n      AllCommonLargeEmbeddingsFeatures.foreach { feature =>\n        commonFeaturesDataRecord.tensors.remove(feature.getFeatureId)\n      }\n      drMerger.merge(commonFeaturesDataRecord, nonMLCommonFeaturesDataRecord)\n\n      val commonFeaturesPLDROpt = CandidateAndCommonFeaturesStreamingUtils\n        .commonFeaturesToPolyDataRecord(\n          versionedMetadataCacheClientOpt = versionedMetadataCacheClientOpt,\n          commonFeatures = commonFeaturesDataRecord,\n          valueFormat = pldr.PolyDataRecord._Fields.LITE_COMPACT_DATA_RECORD,\n          createNew = query.params(\n            HomeGlobalParams.EnableCommonFeaturesDataRecordCopyDuringPldrConversionParam)\n        )\n\n      commonFeaturesPLDROptionObserver(commonFeaturesPLDROpt).flatMap { pldr =>\n        predictionRequestId.map(_ -> pldr)\n      }\n    } else None\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect/HomeScribeClientEventSideEffect.scala",
    "content": "package com.twitter.home_mixer.functional_component.side_effect\n\nimport com.twitter.clientapp.thriftscala.LogEvent\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.side_effect.ScribeClientEventSideEffect\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Side effect that logs served tweet metrics to Scribe as client events.\n */\ncase class HomeScribeClientEventSideEffect(\n  enableScribeClientEvents: Boolean,\n  override val logPipelinePublisher: EventPublisher[LogEvent],\n  injectedTweetsCandidatePipelineIdentifiers: Seq[CandidatePipelineIdentifier],\n  adsCandidatePipelineIdentifier: Option[CandidatePipelineIdentifier] = None,\n  whoToFollowCandidatePipelineIdentifier: Option[CandidatePipelineIdentifier] = None,\n  whoToSubscribeCandidatePipelineIdentifier: Option[CandidatePipelineIdentifier] = None,\n  forYouCommunitiesToJoinCandidatePipelineIdentifier: Option[CandidatePipelineIdentifier] = None,\n  forYouRelevancePromptCandidatePipelineIdentifier: Option[CandidatePipelineIdentifier] = None)\n    extends ScribeClientEventSideEffect[PipelineQuery, Timeline]\n    with PipelineResultSideEffect.Conditionally[\n      PipelineQuery,\n      Timeline\n    ] {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"HomeScribeClientEvent\")\n\n  override val page = \"home\"\n\n  override def onlyIf(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: Timeline\n  ): Boolean = enableScribeClientEvents\n\n  override def buildClientEvents(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: Timeline\n  ): Seq[ScribeClientEventSideEffect.ClientEvent] = {\n\n    val itemCandidates = CandidatesUtil.getItemCandidates(selectedCandidates)\n    val sources = itemCandidates.groupBy(_.source)\n    val injectedTweets =\n      injectedTweetsCandidatePipelineIdentifiers.flatMap(sources.getOrElse(_, Seq.empty))\n    val promotedTweets = adsCandidatePipelineIdentifier.flatMap(sources.get).toSeq.flatten\n\n    // WhoToFollow and WhoToSubscribe modules are not required for all home-mixer products, e.g. list tweets timeline.\n    val whoToFollowUsers = whoToFollowCandidatePipelineIdentifier.flatMap(sources.get).toSeq.flatten\n    val whoToSubscribeUsers =\n      whoToSubscribeCandidatePipelineIdentifier.flatMap(sources.get).toSeq.flatten\n\n    val communititesToJoin =\n      forYouCommunitiesToJoinCandidatePipelineIdentifier.flatMap(sources.get).toSeq.flatten\n\n    val relevancePrompt =\n      forYouRelevancePromptCandidatePipelineIdentifier.flatMap(sources.get).toSeq.flatten\n\n    val servedEvents = ServedEventsBuilder\n      .build(\n        query,\n        injectedTweets,\n        promotedTweets,\n        whoToFollowUsers,\n        whoToSubscribeUsers,\n        communititesToJoin,\n        relevancePrompt)\n\n    val emptyTimelineEvents = EmptyTimelineEventsBuilder.build(query, injectedTweets)\n\n    val queryEvents = QueryEventsBuilder.build(query, injectedTweets)\n\n    (servedEvents ++ emptyTimelineEvents ++ queryEvents).filter(_.eventValue.forall(_ > 0))\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.9)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect/HomeScribeServedCandidatesSideEffect.scala",
    "content": "package com.twitter.home_mixer.functional_component.side_effect\n\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.home_mixer.marshaller.timeline_logging.PromotedTweetDetailsMarshaller\nimport com.twitter.home_mixer.marshaller.timeline_logging.TweetDetailsMarshaller\nimport com.twitter.home_mixer.marshaller.timeline_logging.WhoToFollowDetailsMarshaller\nimport com.twitter.home_mixer.model.HomeFeatures.GetInitialFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GetMiddleFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GetNewerFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GetOlderFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasDarkRequestFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RequestJoinIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceSignalFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserActionsByteArrayFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserActionsContainsExplicitSignalsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserActionsSizeFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedScoreFeature\nimport com.twitter.home_mixer.model.PredictedScoreFeature\nimport com.twitter.home_mixer.model.request.DeviceContext.RequestContext\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.model.request.HasSeenTweetIds\nimport com.twitter.home_mixer.model.request.SubscribedProduct\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableScribeServedCandidatesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixInferenceClusterParam\nimport com.twitter.home_mixer.param.HomeMixerFlagName.ScribeServedCandidatesFlag\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.BaseUserCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowCandidateDecorator\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_subscribe_module.WhoToSubscribeCandidateDecorator\nimport com.twitter.product_mixer.component_library.side_effect.ScribeLogEventSideEffect\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ModuleItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.UserItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.timeline_logging.thriftscala.Prediction\nimport com.twitter.timelines.timeline_logging.{thriftscala => thrift}\nimport com.twitter.util.Time\nimport java.nio.ByteBuffer\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Side effect that logs home timeline served candidates to Scribe.\n */\n@Singleton\nclass HomeScribeServedCandidatesSideEffect @Inject() (\n  @Flag(ScribeServedCandidatesFlag) enableScribeServedCandidates: Boolean,\n  scribeEventPublisher: EventPublisher[thrift.ServedEntry])\n    extends ScribeLogEventSideEffect[\n      thrift.ServedEntry,\n      PipelineQuery with HasSeenTweetIds with HasDeviceContext,\n      Timeline\n    ]\n    with PipelineResultSideEffect.Conditionally[\n      PipelineQuery with HasSeenTweetIds with HasDeviceContext,\n      Timeline\n    ] {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"HomeScribeServedCandidates\")\n\n  override def onlyIf(\n    query: PipelineQuery with HasSeenTweetIds with HasDeviceContext,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: Timeline\n  ): Boolean = enableScribeServedCandidates && query.params(EnableScribeServedCandidatesParam)\n\n  override def buildLogEvents(\n    query: PipelineQuery with HasSeenTweetIds with HasDeviceContext,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: Timeline\n  ): Seq[thrift.ServedEntry] = {\n    val timelineType = query.product match {\n      case FollowingProduct => thrift.TimelineType.HomeLatest\n      case ForYouProduct => thrift.TimelineType.Home\n      case SubscribedProduct => thrift.TimelineType.HomeSubscribed\n      case other => throw new UnsupportedOperationException(s\"Unknown product: $other\")\n    }\n    val requestProvenance = query.deviceContext.map { deviceContext =>\n      deviceContext.requestContextValue match {\n        case RequestContext.Foreground => thrift.RequestProvenance.Foreground\n        case RequestContext.Launch => thrift.RequestProvenance.Launch\n        case RequestContext.PullToRefresh => thrift.RequestProvenance.Ptr\n        case _ => thrift.RequestProvenance.Other\n      }\n    }\n    val queryType = query.features.map { featureMap =>\n      if (featureMap.getOrElse(GetOlderFeature, false)) thrift.QueryType.GetOlder\n      else if (featureMap.getOrElse(GetNewerFeature, false)) thrift.QueryType.GetNewer\n      else if (featureMap.getOrElse(GetMiddleFeature, false)) thrift.QueryType.GetMiddle\n      else if (featureMap.getOrElse(GetInitialFeature, false)) thrift.QueryType.GetInitial\n      else thrift.QueryType.Other\n    }\n\n    val phoenixCluster =\n      s\"phoenix.${query.params(PhoenixInferenceClusterParam).toString.toLowerCase}.\"\n\n    val tweetIdToItemCandidateMap: Map[Long, ItemCandidateWithDetails] =\n      selectedCandidates.flatMap {\n        case item: ItemCandidateWithDetails if item.candidate.isInstanceOf[BaseTweetCandidate] =>\n          Seq((item.candidateIdLong, item))\n        case module: ModuleCandidateWithDetails\n            if module.candidates.headOption.exists(_.candidate.isInstanceOf[BaseTweetCandidate]) =>\n          module.candidates.map(item => (item.candidateIdLong, item))\n        case _ => Seq.empty\n      }.toMap\n\n    val userIdToItemCandidateMap: Map[Long, ItemCandidateWithDetails] =\n      selectedCandidates.flatMap {\n        case module: ModuleCandidateWithDetails\n            if module.candidates.forall(_.candidate.isInstanceOf[BaseUserCandidate]) =>\n          module.candidates.map { item =>\n            (item.candidateIdLong, item)\n          }\n        case _ => Seq.empty\n      }.toMap\n\n    val userActions = selectedCandidates.collectFirst {\n      case candidate if candidate.features.getOrElse(UserActionsSizeFeature, None).isDefined =>\n        (\n          candidate.features.getOrElse(UserActionsSizeFeature, None),\n          candidate.features.getOrElse(UserActionsContainsExplicitSignalsFeature, false)\n        )\n    }\n\n    val userActionsByteArrayOpt =\n      query.features.flatMap(_.getOrElse(UserActionsByteArrayFeature, None))\n    val userActionsBuffer = userActionsByteArrayOpt.map(ua => ByteBuffer.wrap(ua))\n\n    val requestInfo = thrift.RequestInfo(\n      requestTimeMs = query.queryTime.inMilliseconds,\n      traceId = Trace.id.traceId.toLong,\n      userId = query.getOptionalUserId,\n      clientAppId = query.clientContext.appId,\n      hasDarkRequest = query.features.flatMap(_.getOrElse(HasDarkRequestFeature, None)),\n      parentId = Some(Trace.id.parentId.toLong),\n      spanId = Some(Trace.id.spanId.toLong),\n      timelineType = Some(timelineType),\n      ipAddress = query.clientContext.ipAddress,\n      userAgent = query.clientContext.userAgent,\n      queryType = queryType,\n      requestProvenance = requestProvenance,\n      languageCode = query.clientContext.languageCode,\n      countryCode = query.clientContext.countryCode,\n      requestEndTimeMs = Some(Time.now.inMilliseconds),\n      servedRequestId = query.features.flatMap(_.getOrElse(ServedIdFeature, None)),\n      requestJoinId = query.features.flatMap(_.getOrElse(RequestJoinIdFeature, None)),\n      userActionsSize = userActions.flatMap(_._1),\n      userActionsContainsExplicitSignals = userActions.map(_._2),\n      userActions = userActionsBuffer\n    )\n\n    response.instructions\n      .collect {\n        case AddEntriesTimelineInstruction(entries) =>\n          entries.zipWithIndex.collect {\n            case (entry: TweetItem, index) if entry.promotedMetadata.isDefined =>\n              val promotedTweetDetails = PromotedTweetDetailsMarshaller(entry, index)\n              Seq(\n                thrift.EntryInfo(\n                  id = entry.id,\n                  position = index.shortValue(),\n                  entryId = entry.entryIdentifier,\n                  entryType = thrift.EntryType.PromotedTweet,\n                  sortIndex = entry.sortIndex,\n                  verticalSize = Some(1),\n                  displayType = Some(entry.displayType.toString),\n                  details = Some(thrift.ItemDetails.PromotedTweetDetails(promotedTweetDetails))\n                ))\n            case (entry: TweetItem, index) =>\n              val candidate = tweetIdToItemCandidateMap(entry.id)\n              val tweetDetails = TweetDetailsMarshaller(entry, candidate)\n              Seq(\n                thrift.EntryInfo(\n                  id = candidate.candidateIdLong,\n                  position = index.shortValue(),\n                  entryId = entry.entryIdentifier,\n                  entryType = thrift.EntryType.Tweet,\n                  sortIndex = entry.sortIndex,\n                  verticalSize = Some(1),\n                  score = candidate.features.getOrElse(ScoreFeature, None),\n                  displayType = Some(entry.displayType.toString),\n                  details = Some(thrift.ItemDetails.TweetDetails(tweetDetails)),\n                  predictionScores = Some(extractPredictionScores(candidate, phoenixCluster)),\n                  sourceSignal = candidate.features.getOrElse(SourceSignalFeature, None).map {\n                    signal =>\n                      thrift.SourceSignal(\n                        id = Some(signal.id),\n                        signalType = signal.signalType,\n                        signalEntity = signal.signalEntity,\n                        authorId = signal.authorId,\n                      )\n                  }\n                ))\n            case (module: TimelineModule, _)\n                if module.entryNamespace.toString == WhoToFollowCandidateDecorator.EntryNamespaceString =>\n              module.items.zipWithIndex.collect {\n                case (ModuleItem(entry: UserItem, _, _, _), index) =>\n                  val candidate = userIdToItemCandidateMap(entry.id)\n                  val whoToFollowDetails = WhoToFollowDetailsMarshaller(entry, candidate)\n                  thrift.EntryInfo(\n                    id = entry.id,\n                    position = index.shortValue(),\n                    entryId = module.entryIdentifier,\n                    entryType = thrift.EntryType.WhoToFollowModule,\n                    sortIndex = module.sortIndex,\n                    score = candidate.features.getOrElse(ScoreFeature, None),\n                    displayType = Some(entry.displayType.toString),\n                    details = Some(thrift.ItemDetails.WhoToFollowDetails(whoToFollowDetails))\n                  )\n              }\n            case (module: TimelineModule, _)\n                if module.entryNamespace.toString == WhoToSubscribeCandidateDecorator.EntryNamespaceString =>\n              module.items.zipWithIndex.collect {\n                case (ModuleItem(entry: UserItem, _, _, _), index) =>\n                  val candidate = userIdToItemCandidateMap(entry.id)\n                  val whoToSubscribeDetails = WhoToFollowDetailsMarshaller(entry, candidate)\n                  thrift.EntryInfo(\n                    id = entry.id,\n                    position = index.shortValue(),\n                    entryId = module.entryIdentifier,\n                    entryType = thrift.EntryType.WhoToSubscribeModule,\n                    sortIndex = module.sortIndex,\n                    score = candidate.features.getOrElse(ScoreFeature, None),\n                    displayType = Some(entry.displayType.toString),\n                    details = Some(thrift.ItemDetails.WhoToFollowDetails(whoToSubscribeDetails))\n                  )\n              }\n            case (module: TimelineModule, _)\n                if module.sortIndex.isDefined && module.items.headOption.exists(\n                  _.item.isInstanceOf[TweetItem]) =>\n              module.items.zipWithIndex.collect {\n                case (ModuleItem(entry: TweetItem, _, _, _), index) =>\n                  val candidate = tweetIdToItemCandidateMap(entry.id)\n                  thrift.EntryInfo(\n                    id = entry.id,\n                    position = index.shortValue(),\n                    entryId = module.entryIdentifier,\n                    entryType = thrift.EntryType.ConversationModule,\n                    sortIndex = module.sortIndex,\n                    score = candidate.features.getOrElse(ScoreFeature, None),\n                    displayType = Some(entry.displayType.toString),\n                    predictionScores = Some(extractPredictionScores(candidate, phoenixCluster)),\n                    sourceSignal =\n                      candidate.features.getOrElse(SourceSignalFeature, None).map { signal =>\n                        thrift.SourceSignal(\n                          id = Some(signal.id),\n                          signalType = signal.signalType\n                        )\n                      }\n                  )\n              }\n            case _ => Seq.empty\n          }.flatten\n        // Other instructions\n        case _ => Seq.empty[thrift.EntryInfo]\n      }.flatten.map { entryInfo =>\n        thrift.ServedEntry(\n          entry = Some(entryInfo),\n          request = requestInfo\n        )\n      }\n  }\n\n  override val logPipelinePublisher: EventPublisher[thrift.ServedEntry] =\n    scribeEventPublisher\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert()\n  )\n\n  private def extractPredictionScores(\n    candidate: ItemCandidateWithDetails,\n    phoenixCluster: String\n  ): Set[Prediction] = {\n    val naviPredictionScores = PredictedScoreFeature.PredictedScoreFeatureSet.map { feature =>\n      Prediction(Some(feature.featureName), candidate.features.getOrElse(feature, None))\n    }\n\n    val phoenixPredictionScores =\n      PhoenixPredictedScoreFeature.PhoenixPredictedScoreFeatureSet.map { feature =>\n        Prediction(\n          Some(phoenixCluster + feature.featureName),\n          candidate.features.getOrElse(feature, None)\n        )\n      }\n\n    (naviPredictionScores ++ phoenixPredictionScores).filter(_.score.isDefined)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect/PublishClientSentImpressionsEventBusSideEffect.scala",
    "content": "package com.twitter.home_mixer.functional_component.side_effect\n\nimport com.twitter.eventbus.client.EventBusPublisher\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.GetNewerFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GetOlderFeature\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.model.request.SubscribedProduct\nimport com.twitter.home_mixer.model.request.HasSeenTweetIds\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.impressionstore.thriftscala.Impression\nimport com.twitter.timelines.impressionstore.thriftscala.ImpressionList\nimport com.twitter.timelines.impressionstore.thriftscala.PublishedImpressionList\nimport com.twitter.timelines.impressionstore.thriftscala.SurfaceArea\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject PublishClientSentImpressionsEventBusSideEffect {\n  val HomeSurfaceArea: Option[Set[SurfaceArea]] = Some(Set(SurfaceArea.HomeTimeline))\n  val HomeLatestSurfaceArea: Option[Set[SurfaceArea]] = Some(Set(SurfaceArea.HomeLatestTimeline))\n  val HomeSubscribedSurfaceArea: Option[Set[SurfaceArea]] = Some(Set(SurfaceArea.HomeSubscribed))\n}\n\n/**\n * Side effect that publishes seen tweet IDs sent from clients. The seen tweet IDs are sent to a\n * heron topology which writes to a memcache dataset.\n */\n@Singleton\nclass PublishClientSentImpressionsEventBusSideEffect @Inject() (\n  eventBusPublisher: EventBusPublisher[PublishedImpressionList],\n  statsReceiver: StatsReceiver)\n    extends PipelineResultSideEffect[PipelineQuery with HasSeenTweetIds, HasMarshalling]\n    with PipelineResultSideEffect.Conditionally[\n      PipelineQuery with HasSeenTweetIds,\n      HasMarshalling\n    ] {\n  import PublishClientSentImpressionsEventBusSideEffect._\n\n  override val identifier: SideEffectIdentifier =\n    SideEffectIdentifier(\"PublishClientSentImpressionsEventBus\")\n\n  private val seenIdsStatsReceiver = statsReceiver.scope(identifier.toString).scope(\"SeenIds\")\n\n  override def onlyIf(\n    query: PipelineQuery with HasSeenTweetIds,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Boolean = query.seenTweetIds.exists(_.nonEmpty)\n\n  def buildEvents(\n    query: PipelineQuery with HasSeenTweetIds,\n    currentTime: Long\n  ): Option[Seq[Impression]] = {\n    val surfaceArea = query.product match {\n      case ForYouProduct => HomeSurfaceArea\n      case FollowingProduct => HomeLatestSurfaceArea\n      case SubscribedProduct => HomeSubscribedSurfaceArea\n      case _ => None\n    }\n\n    val device = query.clientContext.appId.getOrElse(0L).toString\n\n    query.seenTweetIds.map { seenTweetIds =>\n      val getNewer = query.features.map(_.getOrElse(GetNewerFeature, false)).getOrElse(false)\n      val getOlder = query.features.map(_.getOrElse(GetOlderFeature, false)).getOrElse(false)\n      val requestType = if (getNewer) \"newer\" else if (getOlder) \"older\" else \"none\"\n      seenIdsStatsReceiver\n        .scope(query.product.identifier.name).scope(requestType).stat(device)\n        .add(seenTweetIds.distinct.size)\n\n      seenTweetIds.map { tweetId =>\n        Impression(\n          tweetId = tweetId,\n          impressionTime = Some(currentTime),\n          surfaceAreas = surfaceArea\n        )\n      }\n    }\n  }\n\n  final override def apply(\n    inputs: PipelineResultSideEffect.Inputs[PipelineQuery with HasSeenTweetIds, HasMarshalling]\n  ): Stitch[Unit] = {\n    val currentTime = Time.now.inMilliseconds\n    val impressions = buildEvents(inputs.query, currentTime)\n\n    Stitch.callFuture(\n      eventBusPublisher.publish(\n        PublishedImpressionList(\n          inputs.query.getRequiredUserId,\n          ImpressionList(impressions),\n          currentTime\n        )\n      )\n    )\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.4)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect/PublishClientSentImpressionsManhattanSideEffect.scala",
    "content": "package com.twitter.home_mixer.functional_component.side_effect\n\nimport com.twitter.home_mixer.model.HomeFeatures.TweetImpressionsFeature\nimport com.twitter.home_mixer.model.request.HasSeenTweetIds\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.impression.{thriftscala => t}\nimport com.twitter.timelines.impressionstore.store.ManhattanTweetImpressionStoreClient\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Side effect that updates the timelines tweet impression\n * store (Manhattan) with seen tweet IDs sent from clients\n */\n@Singleton\nclass PublishClientSentImpressionsManhattanSideEffect @Inject() (\n  manhattanTweetImpressionStoreClient: ManhattanTweetImpressionStoreClient)\n    extends PipelineResultSideEffect[PipelineQuery with HasSeenTweetIds, HasMarshalling]\n    with PipelineResultSideEffect.Conditionally[\n      PipelineQuery with HasSeenTweetIds,\n      HasMarshalling\n    ] {\n\n  override val identifier: SideEffectIdentifier =\n    SideEffectIdentifier(\"PublishClientSentImpressionsManhattan\")\n\n  override def onlyIf(\n    query: PipelineQuery with HasSeenTweetIds,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Boolean = query.seenTweetIds.exists(_.nonEmpty)\n\n  def buildEvents(query: PipelineQuery): Option[(Long, t.TweetImpressionsEntries)] = {\n    query.features.flatMap { featureMap =>\n      val impressions = featureMap.getOrElse(TweetImpressionsFeature, Seq.empty)\n      if (impressions.nonEmpty)\n        Some((query.getRequiredUserId, t.TweetImpressionsEntries(impressions)))\n      else None\n    }\n  }\n\n  final override def apply(\n    inputs: PipelineResultSideEffect.Inputs[PipelineQuery with HasSeenTweetIds, HasMarshalling]\n  ): Stitch[Unit] = {\n    val events = buildEvents(inputs.query)\n\n    Stitch\n      .traverse(events) {\n        case (key, value) => manhattanTweetImpressionStoreClient.write(key, value)\n      }\n      .unit\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.4)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect/PublishImpressionBloomFilterSideEffect.scala",
    "content": "package com.twitter.home_mixer.functional_component.side_effect\n\nimport com.twitter.home_mixer.model.HomeFeatures.ImpressionBloomFilterFeature\nimport com.twitter.home_mixer.model.request.HasSeenTweetIds\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableImpressionBloomFilter\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.clients.manhattan.store.ManhattanStoreClient\nimport com.twitter.timelines.impressionbloomfilter.{thriftscala => blm}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PublishImpressionBloomFilterSideEffect @Inject() (\n  bloomFilterClient: ManhattanStoreClient[\n    blm.ImpressionBloomFilterKey,\n    blm.ImpressionBloomFilterSeq\n  ]) extends PipelineResultSideEffect[PipelineQuery with HasSeenTweetIds, HasMarshalling]\n    with PipelineResultSideEffect.Conditionally[\n      PipelineQuery with HasSeenTweetIds,\n      HasMarshalling\n    ] {\n\n  override val identifier: SideEffectIdentifier =\n    SideEffectIdentifier(\"PublishImpressionBloomFilter\")\n\n  private val SurfaceArea = blm.SurfaceArea.HomeTimeline\n\n  override def onlyIf(\n    query: PipelineQuery with HasSeenTweetIds,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Boolean =\n    query.params.getBoolean(EnableImpressionBloomFilter) && query.seenTweetIds.exists(_.nonEmpty)\n\n  def buildEvents(query: PipelineQuery): Option[blm.ImpressionBloomFilterSeq] = {\n    query.features.flatMap { featureMap =>\n      val impressionBloomFilterSeq = featureMap.get(ImpressionBloomFilterFeature)\n      if (impressionBloomFilterSeq.entries.nonEmpty) Some(impressionBloomFilterSeq)\n      else None\n    }\n  }\n\n  override def apply(\n    inputs: PipelineResultSideEffect.Inputs[PipelineQuery with HasSeenTweetIds, HasMarshalling]\n  ): Stitch[Unit] = {\n    buildEvents(inputs.query)\n      .map { updatedBloomFilterSeq =>\n        bloomFilterClient.write(\n          blm.ImpressionBloomFilterKey(inputs.query.getRequiredUserId, SurfaceArea),\n          updatedBloomFilterSeq)\n      }.getOrElse(Stitch.Unit)\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect/TruncateTimelinesPersistenceStoreSideEffect.scala",
    "content": "package com.twitter.home_mixer.functional_component.side_effect\n\nimport com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.param.HomeGlobalParams.TimelinesPersistenceStoreMaxEntriesPerClient\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelinemixer.clients.persistence.TimelineResponseBatchesClient\nimport com.twitter.timelinemixer.clients.persistence.TimelineResponseV3\nimport com.twitter.timelineservice.model.TimelineQuery\nimport com.twitter.timelineservice.model.core.TimelineKind\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Side effect that truncates entries in the Timelines Persistence store\n * based on the number of entries per client.\n */\n@Singleton\nclass TruncateTimelinesPersistenceStoreSideEffect @Inject() (\n  timelineResponseBatchesClient: TimelineResponseBatchesClient[TimelineResponseV3])\n    extends PipelineResultSideEffect[PipelineQuery, Timeline] {\n\n  override val identifier: SideEffectIdentifier =\n    SideEffectIdentifier(\"TruncateTimelinesPersistenceStore\")\n\n  def getResponsesToDelete(query: PipelineQuery): Seq[TimelineResponseV3] = {\n    val responses =\n      query.features.map(_.getOrElse(PersistenceEntriesFeature, Seq.empty)).toSeq.flatten\n    val responsesByClient = responses.groupBy(_.clientPlatform).values.toSeq\n    val maxEntriesPerClient = query.params(TimelinesPersistenceStoreMaxEntriesPerClient)\n\n    responsesByClient.flatMap {\n      _.sortBy(_.servedTime.inMilliseconds)\n        .foldRight((Seq.empty[TimelineResponseV3], maxEntriesPerClient)) {\n          case (response, (responsesToDelete, remainingCap)) =>\n            if (remainingCap > 0) (responsesToDelete, remainingCap - response.entries.size)\n            else (response +: responsesToDelete, remainingCap)\n        } match { case (responsesToDelete, _) => responsesToDelete }\n    }\n  }\n\n  final override def apply(\n    inputs: PipelineResultSideEffect.Inputs[PipelineQuery, Timeline]\n  ): Stitch[Unit] = {\n    val timelineKind = inputs.query.product match {\n      case FollowingProduct => TimelineKind.homeLatest\n      case ForYouProduct => TimelineKind.home\n      case other => throw new UnsupportedOperationException(s\"Unknown product: $other\")\n    }\n    val timelineQuery = TimelineQuery(id = inputs.query.getRequiredUserId, kind = timelineKind)\n\n    val responsesToDelete = getResponsesToDelete(inputs.query)\n\n    if (responsesToDelete.nonEmpty)\n      Stitch.callFuture(timelineResponseBatchesClient.delete(timelineQuery, responsesToDelete))\n    else Stitch.Unit\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect/UpdateLastNonPollingTimeSideEffect.scala",
    "content": "package com.twitter.home_mixer.functional_component.side_effect\n\nimport com.twitter.home_mixer.model.HomeFeatures.FollowingLastNonPollingTimeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.NonPollingTimesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PollingFeature\nimport com.twitter.home_mixer.model.request.DeviceContext\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.component_library.side_effect.UserSessionStoreUpdateSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelineservice.model.util.FinagleRequestContext\nimport com.twitter.user_session_store.ReadWriteUserSessionStore\nimport com.twitter.user_session_store.WriteRequest\nimport com.twitter.user_session_store.thriftscala.NonPollingTimestamps\nimport com.twitter.user_session_store.thriftscala.UserSessionField\nimport com.twitter.util.Time\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Side effect that updates the User Session Store (Manhattan) with the timestamps of non polling requests.\n */\n@Singleton\nclass UpdateLastNonPollingTimeSideEffect[\n  Query <: PipelineQuery with HasDeviceContext,\n  ResponseType <: HasMarshalling] @Inject() (\n  override val userSessionStore: ReadWriteUserSessionStore)\n    extends UserSessionStoreUpdateSideEffect[\n      WriteRequest,\n      Query,\n      ResponseType\n    ] {\n  private val MaxNonPollingTimes = 10\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"UpdateLastNonPollingTime\")\n\n  /**\n   * When the request is non polling and is not a background fetch request, update\n   * the list of non polling timestamps with the timestamp of the current request\n   */\n  override def buildWriteRequest(query: Query): Option[WriteRequest] = {\n    val isBackgroundFetch = query.deviceContext\n      .exists(_.requestContextValue.contains(DeviceContext.RequestContext.BackgroundFetch))\n\n    if (!query.features.exists(_.getOrElse(PollingFeature, false)) && !isBackgroundFetch) {\n      val fields = Seq(UserSessionField.NonPollingTimestamps(makeLastNonPollingTimestamps(query)))\n      Some(WriteRequest(query.getRequiredUserId, fields))\n    } else None\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.96)\n  )\n\n  private def makeLastNonPollingTimestamps(query: Query): NonPollingTimestamps = {\n    val priorNonPollingTimestamps =\n      query.features.map(_.getOrElse(NonPollingTimesFeature, Seq.empty)).toSeq.flatten\n\n    val lastNonPollingTimeMs =\n      FinagleRequestContext.default.requestStartTime.get.getOrElse(Time.now).inMillis\n\n    val followingLastNonPollingTime = query.features\n      .flatMap(features => features.getOrElse(FollowingLastNonPollingTimeFeature, None))\n      .map(_.inMillis)\n\n    NonPollingTimestamps(\n      nonPollingTimestampsMs =\n        (lastNonPollingTimeMs +: priorNonPollingTimestamps).take(MaxNonPollingTimes),\n      mostRecentHomeLatestNonPollingTimestampMs =\n        if (query.product == FollowingProduct) Some(lastNonPollingTimeMs)\n        else followingLastNonPollingTime\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect/UpdateTimelinesPersistenceStoreSideEffect.scala",
    "content": "package com.twitter.home_mixer.functional_component.side_effect\n\nimport com.twitter.home_mixer.functional_component.decorator.EntryPointPivotModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.ForYouTweetCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.KeywordTrendsModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.StoriesModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.TuneFeedModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.VideoCarouselModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.BookmarksModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.PinnedTweetsModuleCandidateDecorator\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.model.PredictedDwellScoreFeature\nimport com.twitter.home_mixer.model.PredictedFavoriteScoreFeature\nimport com.twitter.home_mixer.model.PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature\nimport com.twitter.home_mixer.model.PredictedGoodClickConvoDescUamGt2ScoreFeature\nimport com.twitter.home_mixer.model.PredictedGoodProfileClickScoreFeature\nimport com.twitter.home_mixer.model.PredictedNegativeFeedbackV2ScoreFeature\nimport com.twitter.home_mixer.model.PredictedReplyEngagedByAuthorScoreFeature\nimport com.twitter.home_mixer.model.PredictedReplyScoreFeature\nimport com.twitter.home_mixer.model.PredictedRetweetScoreFeature\nimport com.twitter.home_mixer.model.PredictedShareScoreFeature\nimport com.twitter.home_mixer.model.PredictedVideoQualityViewScoreFeature\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnablePersistenceDebug\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.pipeline.candidate.communities_to_join.CommunitiesToJoinCandidateDecorator\nimport com.twitter.product_mixer.component_library.pipeline.candidate.job.RecommendedJobsCandidateDecorator\nimport com.twitter.product_mixer.component_library.pipeline.candidate.recruiting_organization.RecommendedRecruitingOrganizationsCandidateDecorator\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowCandidateDecorator\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_subscribe_module.WhoToSubscribeCandidateDecorator\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ReplaceEntryTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ShowCoverInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.PromptItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PromotedMetadata\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelinemixer.clients.persistence.EntryWithItemIds\nimport com.twitter.timelinemixer.clients.persistence.ItemIds\nimport com.twitter.timelinemixer.clients.persistence.TimelineResponseBatchesClient\nimport com.twitter.timelinemixer.clients.persistence.TimelineResponseV3\nimport com.twitter.timelines.persistence.{thriftscala => persistence}\nimport com.twitter.timelineservice.model.PredictedScores\nimport com.twitter.timelineservice.model.TimelineQuery\nimport com.twitter.timelineservice.model.TimelineQueryOptions\nimport com.twitter.timelineservice.model.TweetScoreV1\nimport com.twitter.timelineservice.model.core.TimelineKind\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport com.twitter.util.Time\nimport com.twitter.{timelineservice => tls}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Side effect that updates the Timelines Persistence Store (Manhattan) with the entries being returned.\n */\n@Singleton\nclass UpdateTimelinesPersistenceStoreSideEffect @Inject() (\n  timelineResponseBatchesClient: TimelineResponseBatchesClient[TimelineResponseV3])\n    extends PipelineResultSideEffect[PipelineQuery, Timeline] {\n\n  override val identifier: SideEffectIdentifier =\n    SideEffectIdentifier(\"UpdateTimelinesPersistenceStore\")\n\n  final override def apply(\n    inputs: PipelineResultSideEffect.Inputs[PipelineQuery, Timeline]\n  ): Stitch[Unit] = {\n    if (inputs.response.instructions.nonEmpty) {\n      val timelineKind = inputs.query.product match {\n        case FollowingProduct => TimelineKind.homeLatest\n        case ForYouProduct => TimelineKind.home\n        case other => throw new UnsupportedOperationException(s\"Unknown product: $other\")\n      }\n\n      val debugEnabled = inputs.query.params(EnablePersistenceDebug)\n\n      val timelineQuery = TimelineQuery(\n        id = inputs.query.getRequiredUserId,\n        kind = timelineKind,\n        options = TimelineQueryOptions(\n          contextualUserId = inputs.query.getOptionalUserId,\n          deviceContext = tls.DeviceContext.empty.copy(\n            userAgent = inputs.query.clientContext.userAgent,\n            clientAppId = inputs.query.clientContext.appId)\n        )\n      )\n\n      val tweetIdToItemCandidateMap: Map[Long, ItemCandidateWithDetails] =\n        inputs.selectedCandidates.flatMap {\n          case item: ItemCandidateWithDetails if item.candidate.id.isInstanceOf[Long] =>\n            Seq((item.candidateIdLong, item))\n          case module: ModuleCandidateWithDetails\n              if module.candidates.headOption.exists(_.candidate.id.isInstanceOf[Long]) =>\n            module.candidates.map(item => (item.candidateIdLong, item))\n          case _ => Seq.empty\n        }.toMap\n\n      val entries = inputs.response.instructions.collect {\n        case AddEntriesTimelineInstruction(entries) =>\n          entries.collect {\n            // includes tweets, tweet previews, and promoted tweets\n            case entry: TweetItem if entry.sortIndex.isDefined =>\n              val tweetEntry = buildTweetEntryWithItemIds(\n                tweetIdToItemCandidateMap(entry.id),\n                entry.sortIndex.get,\n                debugEnabled,\n                entry.promotedMetadata\n              )\n              Seq(tweetEntry)\n            case entry: PromptItem =>\n              Seq(\n                EntryWithItemIds(\n                  // Temporarily use annotation here as entity type until we come up with a proper one\n                  entityIdType = EntityIdType.Annotation,\n                  sortIndex = entry.sortIndex.getOrElse(0L),\n                  size = 1,\n                  itemIds = None\n                ))\n            case module: TimelineModule if module.sortIndex.isDefined && module.items.nonEmpty =>\n              if (module.entryNamespace == ForYouTweetCandidateDecorator.ConvoModuleEntryNamespace) {\n                module.items.map { item =>\n                  buildTweetEntryWithItemIds(\n                    tweetIdToItemCandidateMap(item.item.id.asInstanceOf[Long]),\n                    module.sortIndex.get,\n                    debugEnabled\n                  )\n                }\n              } else if (module.entryNamespace == StoriesModuleCandidateDecorator.TrendsEntryNamespace\n                || module.entryNamespace == KeywordTrendsModuleCandidateDecorator.KeywordTrendsEntryNamespace) {\n                Seq(\n                  EntryWithItemIds(\n                    entityIdType = EntityIdType.Trends,\n                    sortIndex = module.sortIndex.get,\n                    size = module.items.size.toShort,\n                    itemIds = None\n                  )\n                )\n              } else {\n                val moduleItems = module.items.map(_.item.id.asInstanceOf[Long])\n\n                val (entityId, items) = module.entryNamespace.toString match {\n                  case WhoToFollowCandidateDecorator.EntryNamespaceString =>\n                    val itemIds = moduleItems.map(id => ItemIds(userId = Some(id)))\n                    (EntityIdType.WhoToFollow, itemIds)\n                  case WhoToSubscribeCandidateDecorator.EntryNamespaceString =>\n                    val itemIds = moduleItems.map(id => ItemIds(userId = Some(id)))\n                    (EntityIdType.WhoToSubscribe, itemIds)\n                  case CommunitiesToJoinCandidateDecorator.EntryNamespaceString =>\n                    val itemIds = moduleItems.map(id => ItemIds(communityId = Some(id)))\n                    (EntityIdType.CommunityModule, itemIds)\n                  case RecommendedJobsCandidateDecorator.EntryNamespaceString =>\n                    val itemIds = moduleItems.map(id => ItemIds(jobId = Some(id)))\n                    (EntityIdType.JobModule, itemIds)\n                  case RecommendedRecruitingOrganizationsCandidateDecorator.EntryNamespaceString =>\n                    val itemIds =\n                      moduleItems.map(id => ItemIds(recruitingOrganizationId = Some(id)))\n                    (EntityIdType.RecruitingOrganizationModule, itemIds)\n                  case BookmarksModuleCandidateDecorator.entryNamespaceString =>\n                    (EntityIdType.BookmarksModule, Seq.empty)\n                  case PinnedTweetsModuleCandidateDecorator.entryNamespaceString =>\n                    (EntityIdType.PinnedTweetsModule, Seq.empty)\n                  case VideoCarouselModuleCandidateDecorator.entryNamespaceString =>\n                    (EntityIdType.VideoCarouselModule, Seq.empty)\n                  case EntryPointPivotModuleCandidateDecorator.EntryNamespaceString =>\n                    (EntityIdType.EntryPointPivot, Seq.empty)\n                  case TuneFeedModuleCandidateDecorator.EntryNamespaceString =>\n                    (EntityIdType.TuneFeedModule, Seq.empty)\n                  case other => throw new IllegalStateException(\"Invalid namespace: \" + other)\n                }\n\n                val entry = EntryWithItemIds(\n                  entityIdType = entityId,\n                  sortIndex = module.sortIndex.get,\n                  size = module.items.size.toShort,\n                  itemIds = Some(items)\n                )\n                Seq(entry)\n              }\n          }.flatten\n        case ShowCoverInstruction(cover) =>\n          val entry = EntryWithItemIds(\n            entityIdType = EntityIdType.Prompt,\n            sortIndex = cover.sortIndex.get,\n            size = 1,\n            itemIds = None\n          )\n          Seq(entry)\n        case ReplaceEntryTimelineInstruction(replace) =>\n          val namespaceLength = TweetItem.TweetEntryNamespace.toString.length\n          val itemId = ItemIds(\n            tweetId = replace.entryIdToReplace.map(_.substring(namespaceLength + 1).toLong),\n            entryIdToReplace = replace.entryIdToReplace,\n          )\n          val entry = EntryWithItemIds(\n            entityIdType = EntityIdType.Tweet,\n            sortIndex = replace.sortIndex.get,\n            size = 1,\n            itemIds = Some(Seq(itemId))\n          )\n          Seq(entry)\n      }.flatten\n\n      val servedId = inputs.query.features.flatMap(_.getOrElse(ServedIdFeature, None))\n\n      val response = TimelineResponseV3(\n        clientPlatform = timelineQuery.clientPlatform,\n        servedTime = Time.now,\n        requestType = requestTypeFromQuery(inputs.query),\n        entries = entries,\n        servedId = servedId,\n      )\n\n      Stitch.callFuture(timelineResponseBatchesClient.insertResponse(timelineQuery, response))\n    } else Stitch.Unit\n  }\n\n  private def buildTweetEntryWithItemIds(\n    candidate: ItemCandidateWithDetails,\n    sortIndex: Long,\n    debug: Boolean,\n    promotedMetadata: Option[PromotedMetadata] = None\n  ): EntryWithItemIds = {\n    val features = candidate.features\n    val sourceAuthorId =\n      if (features.getOrElse(IsRetweetFeature, false)) features.getOrElse(SourceUserIdFeature, None)\n      else features.getOrElse(AuthorIdFeature, None)\n    val quoteAuthorId =\n      if (features.getOrElse(QuotedTweetIdFeature, None).nonEmpty)\n        features.getOrElse(SourceUserIdFeature, None)\n      else None\n    val tweetScore = features.getOrElse(ScoreFeature, None)\n    val debugInfo = features.getOrElse(DebugStringFeature, None)\n    val predictionRequestId = features.getOrElse(PredictionRequestIdFeature, None)\n    val servedType = candidate.features.getOrElse(ServedTypeFeature, hmt.ServedType.Undefined).name\n    val isPreview = features.getOrElse(IsTweetPreviewFeature, default = false)\n    val entityType = if (isPreview) EntityIdType.TweetPreview else EntityIdType.Tweet\n    val grokAnnotations = if (debug) features.getOrElse(GrokAnnotationsFeature, None) else None\n    val topics = grokAnnotations.map(_.topics)\n    val tags = grokAnnotations.map(_.tags)\n    val impressionId = if (debug) promotedMetadata.flatMap(_.impressionString) else None\n    val predictedScores =\n      if (debug)\n        Some(\n          PredictedScores(\n            favoriteScore = features.getOrElse(PredictedFavoriteScoreFeature, None),\n            replyScore = features.getOrElse(PredictedReplyScoreFeature, None),\n            retweetScore = features.getOrElse(PredictedRetweetScoreFeature, None),\n            replyEngagedByAuthorScore =\n              features.getOrElse(PredictedReplyEngagedByAuthorScoreFeature, None),\n            goodClickConvoDescFavoritedOrRepliedScore = features\n              .getOrElse(PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature, None),\n            goodClickConvoDescUamGt2Score =\n              features.getOrElse(PredictedGoodClickConvoDescUamGt2ScoreFeature, None),\n            goodProfileClickScore = features.getOrElse(PredictedGoodProfileClickScoreFeature, None),\n            videoQualityViewScore = features.getOrElse(PredictedVideoQualityViewScoreFeature, None),\n            shareScore = features.getOrElse(PredictedShareScoreFeature, None),\n            dwellScore = features.getOrElse(PredictedDwellScoreFeature, None),\n            negativeFeedbackV2Score =\n              features.getOrElse(PredictedNegativeFeedbackV2ScoreFeature, None)\n          ))\n      else None\n\n    val itemId = ItemIds(\n      tweetId = Some(candidate.candidateIdLong),\n      sourceTweetId = features.getOrElse(SourceTweetIdFeature, None),\n      quoteTweetId = features.getOrElse(QuotedTweetIdFeature, None),\n      sourceAuthorId = sourceAuthorId,\n      quoteAuthorId = quoteAuthorId,\n      inReplyToTweetId = features.getOrElse(InReplyToTweetIdFeature, None),\n      inReplyToAuthorId = features.getOrElse(DirectedAtUserIdFeature, None),\n      semanticCoreId = features.getOrElse(SemanticCoreIdFeature, None),\n      tweetScore = tweetScore.map(\n        TweetScoreV1(\n          _,\n          Some(servedType),\n          debugInfo,\n          predictionRequestId,\n          topics,\n          tags,\n          predictedScores)\n      ),\n      impressionId = impressionId\n    )\n\n    EntryWithItemIds(\n      entityIdType = entityType,\n      sortIndex = sortIndex,\n      size = 1.toShort,\n      itemIds = Some(Seq(itemId))\n    )\n  }\n\n  private def requestTypeFromQuery(query: PipelineQuery): persistence.RequestType = {\n    val features = query.features.getOrElse(FeatureMap.empty)\n\n    val featureToRequestType = Seq(\n      (PollingFeature, persistence.RequestType.Polling),\n      (GetInitialFeature, persistence.RequestType.Initial),\n      (GetNewerFeature, persistence.RequestType.Newer),\n      (GetMiddleFeature, persistence.RequestType.Middle),\n      (GetOlderFeature, persistence.RequestType.Older)\n    )\n\n    featureToRequestType\n      .collectFirst {\n        case (feature, requestType) if features.getOrElse(feature, false) => requestType\n      }.getOrElse(persistence.RequestType.Other)\n  }\n\n  override val alerts = Seq(HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.8))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n    ],\n    exports = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request/DeviceContextUnmarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.request\n\nimport com.twitter.home_mixer.model.request.DeviceContext\nimport com.twitter.home_mixer.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass DeviceContextUnmarshaller @Inject() () {\n\n  def apply(deviceContext: t.DeviceContext): DeviceContext = {\n    DeviceContext(\n      isPolling = deviceContext.isPolling,\n      requestContext = deviceContext.requestContext,\n      latestControlAvailable = deviceContext.latestControlAvailable,\n      autoplayEnabled = deviceContext.autoplayEnabled\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request/HomeMixerDebugParamsUnmarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.request\n\nimport com.twitter.home_mixer.model.request.HomeMixerDebugOptions\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.core.functional_component.marshaller.request.FeatureValueUnmarshaller\nimport com.twitter.product_mixer.core.model.marshalling.request.DebugParams\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeMixerDebugParamsUnmarshaller @Inject() (\n  featureValueUnmarshaller: FeatureValueUnmarshaller) {\n\n  def apply(debugParams: t.DebugParams): DebugParams = {\n    DebugParams(\n      featureOverrides = debugParams.featureOverrides.map { map =>\n        map.mapValues(featureValueUnmarshaller(_)).toMap\n      },\n      debugOptions = debugParams.debugOptions.map { options =>\n        HomeMixerDebugOptions(\n          requestTimeOverride = options.requestTimeOverrideMillis.map(Time.fromMilliseconds),\n          showIntermediateLogs = options.showIntermediateLogs.orElse(Some(false))\n        )\n      }\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request/HomeMixerProductContextUnmarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.request\n\nimport com.twitter.home_mixer.model.request.FollowingProductContext\nimport com.twitter.home_mixer.model.request.ForYouProductContext\nimport com.twitter.home_mixer.model.request.HeavyRankerScoresProductContext\nimport com.twitter.home_mixer.model.request.ScoredTweetsProductContext\nimport com.twitter.home_mixer.model.request.ScoredVideoTweetsProductContext\nimport com.twitter.home_mixer.model.request.SubscribedProductContext\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.core.model.marshalling.request.ProductContext\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeMixerProductContextUnmarshaller @Inject() (\n  deviceContextUnmarshaller: DeviceContextUnmarshaller) {\n\n  def apply(productContext: t.ProductContext): ProductContext = productContext match {\n    case t.ProductContext.Following(p) =>\n      FollowingProductContext(\n        deviceContext = p.deviceContext.map(deviceContextUnmarshaller(_)),\n        seenTweetIds = p.seenTweetIds,\n        dspClientContext = p.dspClientContext\n      )\n    case t.ProductContext.ForYou(p) =>\n      ForYouProductContext(\n        deviceContext = p.deviceContext.map(deviceContextUnmarshaller(_)),\n        seenTweetIds = p.seenTweetIds,\n        dspClientContext = p.dspClientContext\n      )\n    case t.ProductContext.ListManagement(p) =>\n      throw new UnsupportedOperationException(s\"This product is no longer used\")\n    case t.ProductContext.ScoredTweets(p) =>\n      ScoredTweetsProductContext(\n        deviceContext = p.deviceContext.map(deviceContextUnmarshaller(_)),\n        seenTweetIds = p.seenTweetIds,\n        servedTweetIds = p.servedTweetIds,\n        backfillTweetIds = p.backfillTweetIds,\n        signupCountryCode = p.signupCountryCode,\n        allowForYouRecommendations = p.allowForYouRecommendations,\n        signupSource = None, // not exposed in thrift interface\n        followerCount = p.followerCount\n      )\n    case t.ProductContext.ScoredVideoTweets(p) =>\n      ScoredVideoTweetsProductContext(\n        deviceContext = p.deviceContext.map(deviceContextUnmarshaller(_)),\n        seenTweetIds = p.seenTweetIds,\n        videoType = p.videoType,\n        pinnedRelatedTweetIds = p.pinnedRelatedTweetIds,\n        scorePinnedTweetsOnly = p.scorePinnedTweetsOnly,\n        immersiveClientMetadata = p.immersiveClientMetadata\n      )\n    case t.ProductContext.ListTweets(p) =>\n      throw new UnsupportedOperationException(s\"This product is no longer used\")\n    case t.ProductContext.ListRecommendedUsers(p) =>\n      throw new UnsupportedOperationException(s\"This product is no longer used\")\n    case t.ProductContext.Subscribed(p) =>\n      SubscribedProductContext(\n        deviceContext = p.deviceContext.map(deviceContextUnmarshaller(_)),\n        seenTweetIds = p.seenTweetIds,\n      )\n    case t.ProductContext.HeavyRankerScores(p) =>\n      HeavyRankerScoresProductContext(\n        deviceContext = p.tweetScoringRequestContext\n          .flatMap(_.deviceContext.map(deviceContextUnmarshaller(_))),\n        tweetIds = p.tweetIds\n      )\n    case t.ProductContext.UnknownUnionField(field) =>\n      throw new UnsupportedOperationException(s\"Unknown display context: ${field.field.name}\")\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request/HomeMixerProductUnmarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.request\n\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.model.request.HeavyRankerScoresProduct\nimport com.twitter.home_mixer.model.request.ScoredTweetsProduct\nimport com.twitter.home_mixer.model.request.ScoredVideoTweetsProduct\nimport com.twitter.home_mixer.model.request.SubscribedProduct\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeMixerProductUnmarshaller @Inject() () {\n\n  def apply(product: t.Product): Product = product match {\n    case t.Product.Following => FollowingProduct\n    case t.Product.ForYou => ForYouProduct\n    case t.Product.ListManagement =>\n      throw new UnsupportedOperationException(s\"This product is no longer used\")\n    case t.Product.ScoredTweets => ScoredTweetsProduct\n    case t.Product.ScoredVideoTweets => ScoredVideoTweetsProduct\n    case t.Product.ListTweets =>\n      throw new UnsupportedOperationException(s\"This product is no longer used\")\n    case t.Product.ListRecommendedUsers =>\n      throw new UnsupportedOperationException(s\"This product is no longer used\")\n    case t.Product.Subscribed => SubscribedProduct\n    case t.Product.HeavyRankerScores => HeavyRankerScoresProduct\n    case t.Product.EnumUnknownProduct(value) =>\n      throw new UnsupportedOperationException(s\"Unknown product: $value\")\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/request/HomeMixerRequestUnmarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.request\n\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.core.functional_component.marshaller.request.ClientContextUnmarshaller\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeMixerRequestUnmarshaller @Inject() (\n  clientContextUnmarshaller: ClientContextUnmarshaller,\n  homeProductUnmarshaller: HomeMixerProductUnmarshaller,\n  homeProductContextUnmarshaller: HomeMixerProductContextUnmarshaller,\n  homeDebugParamsUnmarshaller: HomeMixerDebugParamsUnmarshaller) {\n\n  def apply(homeRequest: t.HomeMixerRequest): HomeMixerRequest = {\n    HomeMixerRequest(\n      clientContext = clientContextUnmarshaller(homeRequest.clientContext),\n      product = homeProductUnmarshaller(homeRequest.product),\n      productContext = homeRequest.productContext.map(homeProductContextUnmarshaller(_)),\n      // Avoid de-serializing cursors in the request unmarshaller. The unmarshaller should never\n      // fail, which is often a possibility when trying to de-serialize a cursor. Cursors can also\n      // be product-specific and more appropriately handled in individual product pipelines.\n      serializedRequestCursor = homeRequest.cursor,\n      maxResults = homeRequest.maxResults,\n      debugParams = homeRequest.debugParams.map(homeDebugParamsUnmarshaller(_)),\n      homeRequestParam = false\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timeline_logging/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n        \"src/thrift/com/twitter/timelines/timeline_logging:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timeline_logging/PromotedTweetDetailsMarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.timeline_logging\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.timelines.timeline_logging.{thriftscala => thriftlog}\n\nobject PromotedTweetDetailsMarshaller {\n\n  def apply(entry: TweetItem, position: Int): thriftlog.PromotedTweetDetails = {\n    thriftlog.PromotedTweetDetails(\n      advertiserId = Some(entry.promotedMetadata.map(_.advertiserId).getOrElse(0L)),\n      insertPosition = Some(position),\n      impressionId = entry.promotedMetadata.flatMap(_.impressionString)\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timeline_logging/TweetDetailsMarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.timeline_logging\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtItemPresentation\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtModulePresentation\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.GeneralContextTypeMarshaller\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ConversationGeneralContextType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.GeneralContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContext\nimport com.twitter.timelines.service.{thriftscala => tst}\nimport com.twitter.timelines.timeline_logging.{thriftscala => thriftlog}\n\nobject TweetDetailsMarshaller {\n\n  private val generalContextTypeMarshaller = new GeneralContextTypeMarshaller()\n\n  def apply(entry: TweetItem, candidate: CandidateWithDetails): thriftlog.TweetDetails = {\n    val socialContext = candidate.presentation.flatMap {\n      case _ @UrtItemPresentation(timelineItem: TweetItem, _) => timelineItem.socialContext\n      case _ @UrtModulePresentation(timelineModule) =>\n        timelineModule.items.head.item match {\n          case timelineItem: TweetItem => timelineItem.socialContext\n          case _ => Some(ConversationGeneralContextType)\n        }\n    }\n\n    val socialContextType = socialContext match {\n      case Some(GeneralContext(contextType, _, _, _, _)) =>\n        Some(generalContextTypeMarshaller(contextType).value.toShort)\n      case Some(TopicContext(_, _)) => Some(tst.ContextType.Topic.value.toShort)\n      case _ => None\n    }\n\n    thriftlog.TweetDetails(\n      sourceTweetId = candidate.features.getOrElse(SourceTweetIdFeature, None),\n      socialContextType = socialContextType,\n      suggestType = Some(candidate.features.get(ServedTypeFeature).name),\n      authorId = candidate.features.getOrElse(AuthorIdFeature, None),\n      sourceAuthorId = candidate.features.getOrElse(SourceUserIdFeature, None)\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timeline_logging/WhoToFollowDetailsMarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.timeline_logging\n\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.UserItem\nimport com.twitter.timelines.timeline_logging.{thriftscala => thriftlog}\n\nobject WhoToFollowDetailsMarshaller {\n\n  def apply(entry: UserItem, candidate: ItemCandidateWithDetails): thriftlog.WhoToFollowDetails =\n    thriftlog.WhoToFollowDetails(\n      enableReactiveBlending = entry.enableReactiveBlending,\n      impressionId = entry.promotedMetadata.flatMap(_.impressionString),\n      advertiserId = entry.promotedMetadata.map(_.advertiserId)\n    )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"src/thrift/com/twitter/timelineservice:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines/ChronologicalCursorMarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.timelines\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.timelines.service.{thriftscala => t}\n\nobject ChronologicalCursorMarshaller {\n\n  def apply(cursor: UrtOrderedCursor): Option[t.ChronologicalCursor] = {\n    cursor.cursorType match {\n      case Some(TopCursor) => Some(t.ChronologicalCursor(bottom = cursor.id))\n      case Some(BottomCursor) => Some(t.ChronologicalCursor(top = cursor.id))\n      case Some(GapCursor) =>\n        Some(t.ChronologicalCursor(top = cursor.id, bottom = cursor.gapBoundaryId))\n      case _ => None\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines/ChronologicalCursorUnmarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.timelines\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.timelines.service.{thriftscala => t}\n\nobject ChronologicalCursorUnmarshaller {\n\n  def apply(requestCursor: t.RequestCursor): Option[UrtOrderedCursor] = {\n    requestCursor match {\n      case t.RequestCursor.ChronologicalCursor(cursor) =>\n        (cursor.top, cursor.bottom) match {\n          case (Some(top), None) =>\n            Some(UrtOrderedCursor(top, cursor.top, Some(BottomCursor)))\n          case (None, Some(bottom)) =>\n            Some(UrtOrderedCursor(bottom, cursor.bottom, Some(TopCursor)))\n          case (Some(top), Some(bottom)) =>\n            Some(UrtOrderedCursor(top, cursor.top, Some(GapCursor), cursor.bottom))\n          case _ => None\n        }\n      case _ => None\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines/DeviceContextMarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.timelines\n\nimport com.twitter.home_mixer.model.request.DeviceContext\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.timelineservice.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass DeviceContextMarshaller @Inject() () {\n\n  def apply(deviceContext: DeviceContext, clientContext: ClientContext): t.DeviceContext = {\n    t.DeviceContext(\n      countryCode = clientContext.countryCode,\n      languageCode = clientContext.languageCode,\n      clientAppId = clientContext.appId,\n      ipAddress = clientContext.ipAddress,\n      guestId = clientContext.guestId,\n      userAgent = clientContext.userAgent,\n      deviceId = clientContext.deviceId,\n      isPolling = deviceContext.isPolling,\n      requestContext = deviceContext.requestContext,\n      referrer = None,\n      tfeAuthHeader = None,\n      mobileDeviceId = clientContext.mobileDeviceId,\n      isSessionStart = None,\n      latestControlAvailable = deviceContext.latestControlAvailable,\n      guestIdMarketing = clientContext.guestIdMarketing,\n      isInternalOrTwoffice = clientContext.isTwoffice,\n      guestIdAds = clientContext.guestIdAds,\n      isUrtRequest = Some(true)\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines/RecommendedUsersCursorUnmarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.timelines\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtUnorderedExcludeIdsCursor\nimport com.twitter.timelines.service.{thriftscala => t}\nimport com.twitter.util.Time\n\nobject RecommendedUsersCursorUnmarshaller {\n\n  def apply(requestCursor: t.RequestCursor): Option[UrtUnorderedExcludeIdsCursor] = {\n    requestCursor match {\n      case t.RequestCursor.RecommendedUsersCursor(cursor) =>\n        Some(\n          UrtUnorderedExcludeIdsCursor(\n            initialSortIndex = cursor.minSortIndex.getOrElse(Time.now.inMilliseconds),\n            excludedIds = cursor.previouslyRecommendedUserIds\n          ))\n      case _ => None\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines/TimelineServiceCursorMarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.timelines\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.timelineservice.{thriftscala => t}\n\nobject TimelineServiceCursorMarshaller {\n\n  def apply(cursor: UrtOrderedCursor): Option[t.Cursor2] = {\n    val id = cursor.id.map(_.toString)\n    val gapBoundaryId = cursor.gapBoundaryId.map(_.toString)\n    cursor.cursorType match {\n      case Some(TopCursor) => Some(t.Cursor2(bottom = id))\n      case Some(BottomCursor) => Some(t.Cursor2(top = id))\n      case Some(GapCursor) => Some(t.Cursor2(top = id, bottom = gapBoundaryId))\n      case _ => None\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines/TopicContextFunctionalityTypeUnmarshaller.scala",
    "content": "package com.twitter.home_mixer.marshaller.timelines\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BasicTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecWithEducationTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContextFunctionalityType\nimport com.twitter.timelines.render.{thriftscala => urt}\n\nobject TopicContextFunctionalityTypeUnmarshaller {\n\n  def apply(\n    topicContextFunctionalityType: urt.TopicContextFunctionalityType\n  ): TopicContextFunctionalityType = topicContextFunctionalityType match {\n    case urt.TopicContextFunctionalityType.Basic => BasicTopicContextFunctionalityType\n    case urt.TopicContextFunctionalityType.Recommendation =>\n      RecommendationTopicContextFunctionalityType\n    case urt.TopicContextFunctionalityType.RecWithEducation =>\n      RecWithEducationTopicContextFunctionalityType\n    case urt.TopicContextFunctionalityType.EnumUnknownTopicContextFunctionalityType(field) =>\n      throw new UnsupportedOperationException(s\"Unknown topic context functionality type: $field\")\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/io/grpc:grpc-protobuf\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/signup\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord\",\n        \"search/search-router/thrift/src/main/thrift:thrift-scala\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/ml/api/constant\",\n        \"src/scala/com/twitter/timelines/prediction/features/common\",\n        \"src/scala/com/twitter/timelines/prediction/features/conversation_features\",\n        \"src/scala/com/twitter/timelines/prediction/features/engaged_language_features\",\n        \"src/scala/com/twitter/timelines/prediction/features/recap\",\n        \"src/scala/com/twitter/timelines/prediction/features/request_context\",\n        \"src/scala/com/twitter/timelines/prediction/features/user_health\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"src/thrift/com/twitter/timelines/impression_bloom_filter:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala\",\n        \"strato/config/src/thrift/com/twitter/strato/columns/content_understanding:content_understanding-scala\",\n        \"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan\",\n        \"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/persistence\",\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate\",\n        \"topic-social-proof/server/src/main/thrift:thrift-scala\",\n        \"tweetconvosvc/common/src/main/thrift/com/twitter/tweetconvosvc/tweet_ancestor:thrift-scala\",\n        \"user_history_transformer/service/src/main/java/com/x/user_action_sequence\",\n    ],\n    exports = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/signup\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala\",\n        \"tweetconvosvc/common/src/main/thrift/com/twitter/tweetconvosvc/tweet_ancestor:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/ClearCacheIncludeInstruction.scala",
    "content": "package com.twitter.home_mixer.model\n\nimport com.twitter.home_mixer.model.request.DeviceContext.RequestContext\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.IncludeInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\n/**\n * Include a clear cache timeline instruction when we satisfy these criteria:\n * - Request Provenance\n * - At least N non-ad tweet entries in the response\n *\n * This is to ensure that we have sufficient new content to justify jumping users to the\n * top of the new timelines response and don't add unnecessary load to backend systems\n */\ncase class ClearCacheIncludeInstruction(\n  ptrEnableParam: FSParam[Boolean],\n  coldStartEnableParam: FSParam[Boolean],\n  warmStartEnableParam: FSParam[Boolean],\n  manualRefreshEnableParam: FSParam[Boolean],\n  navigateEnableParam: FSParam[Boolean],\n  minEntriesParam: FSBoundedParam[Int])\n    extends IncludeInstruction[PipelineQuery with HasDeviceContext] {\n\n  override def apply(\n    query: PipelineQuery with HasDeviceContext,\n    entries: Seq[TimelineEntry]\n  ): Boolean = {\n    val requestContext = query.deviceContext.flatMap(_.requestContextValue)\n\n    val ptrEnabled =\n      query.params(ptrEnableParam) && requestContext.contains(RequestContext.PullToRefresh)\n    val coldStartEnabled =\n      query.params(coldStartEnableParam) && requestContext.contains(RequestContext.Launch)\n    val warmStartEnabled =\n      query.params(warmStartEnableParam) && requestContext.contains(RequestContext.Foreground)\n    val manualRefreshEnabled =\n      query.params(manualRefreshEnableParam) && requestContext.contains(\n        RequestContext.ManualRefresh)\n    val navigateEnabled =\n      query.params(navigateEnableParam) && requestContext.contains(RequestContext.Navigate)\n\n    val minTweets = query.params(minEntriesParam) <= entries.collect {\n      case item: TweetItem if item.promotedMetadata.isEmpty => 1\n      case module: TimelineModule if module.items.head.item.isInstanceOf[TweetItem] =>\n        module.items.size\n    }.sum\n\n    (ptrEnabled || coldStartEnabled || warmStartEnabled || manualRefreshEnabled || navigateEnabled) && minTweets\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/ContentFeatures.scala",
    "content": "package com.twitter.home_mixer.model\n\nimport com.twitter.escherbird.{thriftscala => esb}\nimport com.twitter.search.common.features.{thriftscala => sc}\nimport com.twitter.tweetypie.{thriftscala => tp}\n\nobject ContentFeatures {\n  val Empty: ContentFeatures = ContentFeatures(\n    0.toShort,\n    false,\n    0.toShort,\n    0.toShort,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None\n  )\n\n  def fromThrift(ebFeatures: sc.ThriftTweetFeatures): ContentFeatures =\n    ContentFeatures(\n      length = ebFeatures.tweetLength.getOrElse(0).toShort,\n      hasQuestion = ebFeatures.hasQuestion.getOrElse(false),\n      numCaps = ebFeatures.numCaps.getOrElse(0).toShort,\n      numWhiteSpaces = ebFeatures.numWhitespaces.getOrElse(0).toShort,\n      numNewlines = ebFeatures.numNewlines,\n      videoDurationMs = ebFeatures.videoDurationMs,\n      bitRate = ebFeatures.bitRate,\n      aspectRatioNum = ebFeatures.aspectRatioNum,\n      aspectRatioDen = ebFeatures.aspectRatioDen,\n      widths = ebFeatures.widths.map(_.map(_.toShort)),\n      heights = ebFeatures.heights.map(_.map(_.toShort)),\n      resizeMethods = ebFeatures.resizeMethods.map(_.map(_.toShort)),\n      numMediaTags = ebFeatures.numMediaTags.map(_.toShort),\n      mediaTagScreenNames = ebFeatures.mediaTagScreenNames,\n      emojiTokens = ebFeatures.emojiTokens.map(_.toSet),\n      emoticonTokens = ebFeatures.emoticonTokens.map(_.toSet),\n      faceAreas = ebFeatures.faceAreas,\n      dominantColorRed = ebFeatures.dominantColorRed,\n      dominantColorBlue = ebFeatures.dominantColorBlue,\n      dominantColorGreen = ebFeatures.dominantColorGreen,\n      numColors = ebFeatures.numColors.map(_.toShort),\n      stickerIds = ebFeatures.stickerIds,\n      mediaOriginProviders = ebFeatures.mediaOriginProviders,\n      isManaged = ebFeatures.isManaged,\n      is360 = ebFeatures.is360,\n      viewCount = ebFeatures.viewCount,\n      isMonetizable = ebFeatures.isMonetizable,\n      isEmbeddable = ebFeatures.isEmbeddable,\n      hasSelectedPreviewImage = ebFeatures.hasSelectedPreviewImage,\n      hasTitle = ebFeatures.hasTitle,\n      hasDescription = ebFeatures.hasDescription,\n      hasVisitSiteCallToAction = ebFeatures.hasVisitSiteCallToAction,\n      hasAppInstallCallToAction = ebFeatures.hasAppInstallCallToAction,\n      hasWatchNowCallToAction = ebFeatures.hasWatchNowCallToAction,\n      dominantColorPercentage = ebFeatures.dominantColorPercentage,\n      posUnigrams = ebFeatures.posUnigrams.map(_.toSet),\n      posBigrams = ebFeatures.posBigrams.map(_.toSet),\n      semanticCoreAnnotations = ebFeatures.semanticCoreAnnotations,\n      tokens = ebFeatures.textTokens.map(_.toSeq),\n      conversationControl = ebFeatures.conversationControl,\n      // media and selfThreadMetadata not carried by ThriftTweetFeatures\n      media = None,\n      selfThreadMetadata = None,\n      hasImage = None,\n      hasVideo = None\n    )\n}\n\ncase class ContentFeatures(\n  length: Short,\n  hasQuestion: Boolean,\n  numCaps: Short,\n  numWhiteSpaces: Short,\n  numNewlines: Option[Short],\n  videoDurationMs: Option[Int],\n  bitRate: Option[Int],\n  aspectRatioNum: Option[Short],\n  aspectRatioDen: Option[Short],\n  widths: Option[Seq[Short]],\n  heights: Option[Seq[Short]],\n  resizeMethods: Option[Seq[Short]],\n  numMediaTags: Option[Short],\n  mediaTagScreenNames: Option[Seq[String]],\n  emojiTokens: Option[Set[String]],\n  emoticonTokens: Option[Set[String]],\n  faceAreas: Option[Seq[Int]],\n  dominantColorRed: Option[Short],\n  dominantColorBlue: Option[Short],\n  dominantColorGreen: Option[Short],\n  numColors: Option[Short],\n  stickerIds: Option[Seq[Long]],\n  mediaOriginProviders: Option[Seq[String]],\n  isManaged: Option[Boolean],\n  is360: Option[Boolean],\n  viewCount: Option[Long],\n  isMonetizable: Option[Boolean],\n  isEmbeddable: Option[Boolean],\n  hasSelectedPreviewImage: Option[Boolean],\n  hasTitle: Option[Boolean],\n  hasDescription: Option[Boolean],\n  hasVisitSiteCallToAction: Option[Boolean],\n  hasAppInstallCallToAction: Option[Boolean],\n  hasWatchNowCallToAction: Option[Boolean],\n  media: Option[Seq[tp.MediaEntity]],\n  dominantColorPercentage: Option[Double],\n  posUnigrams: Option[Set[String]],\n  posBigrams: Option[Set[String]],\n  semanticCoreAnnotations: Option[Seq[esb.TweetEntityAnnotation]],\n  selfThreadMetadata: Option[tp.SelfThreadMetadata],\n  tokens: Option[Seq[String]],\n  conversationControl: Option[tp.ConversationControl],\n  hasImage: Option[Boolean],\n  hasVideo: Option[Boolean],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/GapIncludeInstruction.scala",
    "content": "package com.twitter.home_mixer.model\n\nimport com.twitter.home_mixer.functional_component.candidate_source.EarlybirdBottomTweetFeature\nimport com.twitter.home_mixer.functional_component.candidate_source.EarlybirdResponseTruncatedFeature\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.IncludeInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Determine whether to include a Gap Cursor in the response based on whether a timeline\n * is truncated because it has more entries than the max response size.\n * There are two ways this can happen:\n *  1) There are unused entries in Earlybird. This is determined by a flag returned from Earlybird.\n *     We respect the Earlybird flag only if there are some entries after deduping and filtering\n *     to ensure that we do not get stuck repeatedly serving gaps which lead to no tweets.\n *  2) Ads injection can take the response size over the max count. Goldfinch truncates tweet\n *     entries in this case. We can check if the bottom tweet from Earlybird is in the response to\n *     determine if all Earlybird tweets have been used.\n *\n * While scrolling down to get older tweets (BottomCursor), responses will generally be\n * truncated, but we don't want to render a gap cursor there, so we need to ensure we only\n * apply the truncation check to newer (TopCursor) or middle (GapCursor) requests.\n *\n * We return either a Gap Cursor or a Bottom Cursor, but not both, so the include instruction\n * for Bottom should be the inverse of Gap.\n */\nobject GapIncludeInstruction\n    extends IncludeInstruction[PipelineQuery with HasPipelineCursor[UrtOrderedCursor]] {\n\n  override def apply(\n    query: PipelineQuery with HasPipelineCursor[UrtOrderedCursor],\n    entries: Seq[TimelineEntry]\n  ): Boolean = {\n    val wasTruncated = query.features.exists(_.getOrElse(EarlybirdResponseTruncatedFeature, false))\n\n    // Get oldest tweet or tweets within oldest conversation module\n    val tweetEntries = entries.view.reverse\n      .collectFirst {\n        case item: TweetItem if item.promotedMetadata.isEmpty => Seq(item.id.toString)\n        case module: TimelineModule if module.items.head.item.isInstanceOf[TweetItem] =>\n          module.items.map(_.item.id.toString)\n      }.toSeq.flatten\n\n    val bottomCursor =\n      query.features.flatMap(_.getOrElse(EarlybirdBottomTweetFeature, None)).map(_.toString)\n\n    // Ads truncation happened if we have at least max count entries and bottom tweet is missing\n    val adsTruncation = query.requestedMaxResults.exists(_ <= entries.size) &&\n      !bottomCursor.exists(tweetEntries.contains)\n\n    query.pipelineCursor.exists(_.cursorType match {\n      case Some(TopCursor) | Some(GapCursor) =>\n        (wasTruncated && tweetEntries.nonEmpty) || adsTruncation\n      case _ => false\n    })\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/GrokTopics.scala",
    "content": "package com.twitter.home_mixer.model\n\nobject GrokTopics {\n  val GrokCategoryIdToNameMap = Map(\n    <removed_id> -> \"Sports\",\n    <removed_id> -> \"Anime\",\n    <removed_id> -> \"Celebrity\",\n    <removed_id> -> \"Music\",\n    <removed_id> -> \"News\",\n    <removed_id> -> \"Business & Finance\",\n    <removed_id> -> \"Cryptocurrency\",\n    <removed_id> -> \"Technology\",\n    <removed_id> -> \"Science\",\n    <removed_id> -> \"Gaming\",\n    <removed_id> -> \"Movies & TV\",\n    <removed_id> -> \"Travel\",\n    <removed_id> -> \"Food\",\n    <removed_id> -> \"Health & Fitness\",\n    <removed_id> -> \"Memes\",\n    <removed_id> -> \"Art\",\n    <removed_id> -> \"Fashion\",\n    <removed_id> -> \"Religion\",\n    <removed_id> -> \"Shopping\",\n    <removed_id> -> \"Cars\",\n    <removed_id> -> \"Aviation\",\n    <removed_id> -> \"Motorcycles\",\n    <removed_id> -> \"Beauty\",\n    <removed_id> -> \"Nature & Outdoors\",\n    <removed_id> -> \"Pets\",\n    <removed_id> -> \"Relationships\",\n    <removed_id> -> \"Home & Garden\",\n    <removed_id> -> \"Career\",\n    <removed_id> -> \"Dance\",\n    <removed_id> -> \"Education\",\n    <removed_id> -> \"Podcasts\",\n    <removed_id> -> \"Streaming\"\n  ) \n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/HomeAdsQuery.scala",
    "content": "package com.twitter.home_mixer.model\n\nimport com.twitter.adserver.thriftscala.RequestTriggerType\nimport com.twitter.home_mixer.model.HomeFeatures.GetInitialFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GetNewerFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GetOlderFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PollingFeature\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.component_library.model.query.ads.AdsQuery\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * These are for feeds needed for ads only.\n */\ntrait HomeAdsQuery\n    extends AdsQuery\n    with PipelineQuery\n    with HasDeviceContext\n    with HasPipelineCursor[UrtOrderedCursor] {\n\n  private val featureToRequestTriggerType = Seq(\n    (GetInitialFeature, RequestTriggerType.Initial),\n    (GetNewerFeature, RequestTriggerType.Scroll),\n    (GetOlderFeature, RequestTriggerType.Scroll),\n    (PollingFeature, RequestTriggerType.AutoRefresh)\n  )\n\n  override val autoplayEnabled: Option[Boolean] = deviceContext.flatMap(_.autoplayEnabled)\n\n  override def requestTriggerType: Option[RequestTriggerType] = {\n    val features = this.features.getOrElse(FeatureMap.empty)\n\n    featureToRequestTriggerType.collectFirst {\n      case (feature, requestType) if features.get(feature) => Some(requestType)\n    }.flatten\n  }\n\n  override val disableNsfwAvoidance: Option[Boolean] = Some(true)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/HomeFeatures.scala",
    "content": "package com.twitter.home_mixer.model\n\nimport com.twitter.core_workflows.user_model.{thriftscala => um}\nimport com.twitter.dal.personal_data.thriftjava.PersonalDataType\nimport com.twitter.dal.personal_data.{thriftjava => pd}\nimport com.twitter.gizmoduck.{thriftscala => gt}\nimport com.twitter.home_mixer.model.candidate_source.SourceSignal\nimport com.twitter.home_mixer.model.signup.SignupSource\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport com.twitter.mediaservices.commons.thriftscala.MediaCategory\nimport com.twitter.ml.api.constant.SharedFeatures\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.BoolDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordFeature\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordOptionalFeature\nimport com.twitter.product_mixer.core.feature.datarecord.DoubleDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.datarecord.LongDiscreteDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.datarecord.SparseBinaryDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.datarecord.SparseContinuousDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.datarecord.StringDataRecordCompatible\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleDisplayType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.search.common.features.{thriftscala => sc}\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport com.twitter.strato.columns.content_understanding.thriftscala.AnnotatedVideoDeserialized\nimport com.twitter.timelinemixer.clients.manhattan.DismissInfo\nimport com.twitter.timelinemixer.clients.persistence.TimelineResponseV3\nimport com.twitter.timelinemixer.injection.model.candidate.AudioSpaceMetaData\nimport com.twitter.timelines.conversation_features.v1.thriftscala.ConversationFeatures\nimport com.twitter.timelines.impressionbloomfilter.{thriftscala => blm}\nimport com.twitter.timelines.model.UserId\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport com.twitter.timelines.prediction.features.conversation_features\nimport com.twitter.timelines.prediction.features.engaged_language_features.LanguageFeatures\nimport com.twitter.timelines.prediction.features.engagement_features.EngagementDataRecordFeatures\nimport com.twitter.timelines.prediction.features.recap.RecapFeatures\nimport com.twitter.timelines.prediction.features.request_context.RequestContextFeatures\nimport com.twitter.timelines.prediction.features.user_health.UserHealthFeatures\nimport com.twitter.timelineservice.model.FeedbackEntry\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\nimport com.twitter.tsp.{thriftscala => tsp}\nimport com.twitter.tweetconvosvc.tweet_ancestor.{thriftscala => ta}\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\nimport com.x.user_action_sequence.UserActionSequence\n\nobject HomeFeatures {\n  // Candidate Features\n  object AncestorsFeature extends Feature[TweetCandidate, Seq[ta.TweetAncestor]]\n  object AudioSpaceMetaDataFeature extends Feature[TweetCandidate, Option[AudioSpaceMetaData]]\n  object ListIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object ListNameFeature extends Feature[TweetCandidate, Option[String]]\n  object ValidLikedByUserIdsFeature extends Feature[TweetCandidate, Seq[Long]]\n  object BookmarkedTweetTimestamp extends Feature[TweetCandidate, Option[Long]]\n  object ArticleIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object ArticlePreviewTextFeature extends Feature[TweetCandidate, Option[String]]\n\n  /**\n   * For Retweets, this should refer to the retweeting user. Use [[SourceUserIdFeature]] if you want to know\n   * who created the Tweet that was retweeted.\n   */\n  object AuthorIdFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Long]\n      with LongDiscreteDataRecordCompatible {\n    override val featureName: String = SharedFeatures.AUTHOR_ID.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set(pd.PersonalDataType.UserId)\n  }\n\n  object AuthorAccountAge extends Feature[TweetCandidate, Option[Duration]]\n  object AuthorIsBlueVerifiedFeature extends Feature[TweetCandidate, Boolean]\n  object AuthorIsGoldVerifiedFeature extends Feature[TweetCandidate, Boolean]\n  object AuthorIsGrayVerifiedFeature extends Feature[TweetCandidate, Boolean]\n  object AuthorIsLegacyVerifiedFeature extends Feature[TweetCandidate, Boolean]\n  object AuthorIsCreatorFeature extends Feature[TweetCandidate, Boolean]\n  object AuthorIsProtectedFeature extends Feature[TweetCandidate, Boolean]\n  object AuthorFollowersFeature extends Feature[TweetCandidate, Option[Long]]\n  object ViralContentCreatorFeature extends Feature[TweetCandidate, Boolean]\n  object GrokContentCreatorFeature extends Feature[TweetCandidate, Boolean]\n  object GorkContentCreatorFeature extends Feature[TweetCandidate, Boolean]\n  object AuthoredByContextualUserFeature extends Feature[TweetCandidate, Boolean]\n  object CachedCandidatePipelineIdentifierFeature extends Feature[TweetCandidate, Option[String]]\n  object ConversationFeature extends Feature[TweetCandidate, Option[ConversationFeatures]]\n  object AuthorSafetyLabels extends Feature[TweetCandidate, Option[Seq[String]]]\n\n  /**\n   * This field should be set to the focal Tweet's tweetId for all tweets which are expected to\n   * be rendered in the same convo module. For non-convo module Tweets, this will be\n   * set to None. Note this is different from how TweetyPie defines ConversationId which is defined\n   * on all Tweets and points to the root tweet. This feature is used for grouping convo modules together.\n   */\n  object ConversationModuleFocalTweetIdFeature extends Feature[TweetCandidate, Option[Long]]\n\n  /**\n   * This field should always be set to the root Tweet in a conversation for all Tweets. For replies, this will\n   * point back to the root Tweet. For non-replies, this will be the candidate's Tweet id. This is consistent with\n   * the TweetyPie definition of ConversationModuleId.\n   */\n  object ConversationModuleIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object DirectedAtUserIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object EarlybirdFeature extends Feature[TweetCandidate, Option[sc.ThriftTweetFeatures]]\n  object EarlybirdScoreFeature extends Feature[TweetCandidate, Option[Double]]\n  object EarlybirdSearchResultFeature extends Feature[TweetCandidate, Option[eb.ThriftSearchResult]]\n  object EntityTokenFeature extends Feature[TweetCandidate, Option[String]]\n  object ExclusiveConversationAuthorIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object FavoritedByCountFeature\n      extends DataRecordFeature[TweetCandidate, Double]\n      with DoubleDataRecordCompatible {\n    override val featureName: String =\n      EngagementDataRecordFeatures.InNetworkFavoritesCount.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] =\n      Set(pd.PersonalDataType.CountOfPrivateLikes, pd.PersonalDataType.CountOfPublicLikes)\n  }\n  object FavoritedByUserIdsFeature extends Feature[TweetCandidate, Seq[Long]]\n  object FeedbackHistoryFeature extends Feature[TweetCandidate, Seq[FeedbackEntry]]\n  // A boolean feature indicating whether the latest feedback timestamp till now is newer than the cached scored tweets TTL\n  object HasRecentFeedbackSinceCacheTtlFeature extends Feature[PipelineQuery, Boolean]\n\n  object SlopAuthorFeature extends Feature[TweetCandidate, Boolean]\n\n  object SlopAuthorScoreFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Double]\n      with DoubleDataRecordCompatible {\n    override val featureName: String = RecapFeatures.SLOP_AUTHOR_SCORE.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n\n  object GrokTranslatedPostIsCachedFeature extends Feature[TweetCandidate, Boolean]\n\n  object GrokVideoMetadataFeature\n      extends FeatureWithDefaultOnFailure[TweetCandidate, Option[AnnotatedVideoDeserialized]] {\n    override def defaultValue: Option[AnnotatedVideoDeserialized] = None\n  }\n\n  object GrokCategoryDataRecordFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Map[String, Double]]\n      with SparseContinuousDataRecordCompatible {\n    override val featureName: String =\n      RecapFeatures.GROK_CATEGORY_SCORES.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] =\n      Set(pd.PersonalDataType.InferredInterests)\n  }\n\n  object GrokTagsFeature\n      extends Feature[TweetCandidate, Set[String]]\n      with SparseBinaryDataRecordCompatible {\n    override val featureName: String = RecapFeatures.GROK_TAGS.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set(\n      pd.PersonalDataType.InferredLanguage)\n  }\n\n  object GrokIsGoreFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Boolean]\n      with BoolDataRecordCompatible {\n    override val featureName: String =\n      RecapFeatures.GROK_IS_GORE.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n\n  object GrokIsNsfwFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Boolean]\n      with BoolDataRecordCompatible {\n    override val featureName: String =\n      RecapFeatures.GROK_IS_NSFW.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n\n  object GrokIsSoftNsfwFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Boolean]\n      with BoolDataRecordCompatible {\n    override val featureName: String =\n      RecapFeatures.GROK_IS_SOFT_NSFW.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n\n  object GrokIsSpamFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Boolean]\n      with BoolDataRecordCompatible {\n    override val featureName: String =\n      RecapFeatures.GROK_IS_SPAM.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n\n  object GrokIsViolentFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Boolean]\n      with BoolDataRecordCompatible {\n    override val featureName: String =\n      RecapFeatures.GROK_IS_VIOLENT.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n\n  object GrokIsLowQualityFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Boolean]\n      with BoolDataRecordCompatible {\n    override val featureName: String =\n      RecapFeatures.GROK_IS_LOW_QUALITY.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] =\n      Set(pd.PersonalDataType.InferredInterests)\n  }\n\n  object GrokIsOcrFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Boolean]\n      with BoolDataRecordCompatible {\n    override val featureName: String =\n      RecapFeatures.GROK_IS_OCR.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] =\n      Set(pd.PersonalDataType.InferredInterests)\n  }\n\n  object GrokSunnyScoreFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Double]\n      with DoubleDataRecordCompatible {\n    override val featureName: String = RecapFeatures.GROK_SUNNY_SCORE.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set(\n      pd.PersonalDataType.EngagementScore)\n  }\n\n  // Used only for metrics tracking. Does not affect the recommendations.\n  object GrokPoliticalInclinationFeature\n      extends Feature[TweetCandidate, Option[hmt.PoliticalInclination]]\n\n  object GrokSlopScoreFeature extends Feature[TweetCandidate, Option[Long]]\n\n  object DedupClusterIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object DedupClusterId88Feature extends Feature[TweetCandidate, Option[Long]]\n  object RetweetedByCountFeature\n      extends DataRecordFeature[TweetCandidate, Double]\n      with DoubleDataRecordCompatible {\n    override val featureName: String =\n      EngagementDataRecordFeatures.InNetworkRetweetsCount.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] =\n      Set(pd.PersonalDataType.CountOfPrivateRetweets, pd.PersonalDataType.CountOfPublicRetweets)\n  }\n  object RetweetedByEngagerIdsFeature extends Feature[TweetCandidate, Seq[Long]]\n  object RepliedByCountFeature\n      extends DataRecordFeature[TweetCandidate, Double]\n      with DoubleDataRecordCompatible {\n    override val featureName: String =\n      EngagementDataRecordFeatures.InNetworkRepliesCount.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] =\n      Set(pd.PersonalDataType.CountOfPrivateReplies, pd.PersonalDataType.CountOfPublicReplies)\n  }\n  object RepliedByEngagerIdsFeature extends Feature[TweetCandidate, Seq[Long]]\n  object FollowedByUserIdsFeature extends Feature[TweetCandidate, Seq[Long]]\n\n  object TopicIdSocialContextFeature extends Feature[TweetCandidate, Option[Long]]\n  object TopicContextFunctionalityTypeFeature\n      extends Feature[TweetCandidate, Option[TopicContextFunctionalityType]]\n  object BasketballContextFeature extends Feature[TweetCandidate, Option[urt.BasketballContext]]\n\n  object GenericPostContextFeature extends Feature[TweetCandidate, Option[urt.GenericContext]]\n  object FromInNetworkSourceFeature extends Feature[TweetCandidate, Boolean]\n  object FullScoringSucceededFeature extends Feature[TweetCandidate, Boolean]\n  object HasDisplayedTextFeature extends Feature[TweetCandidate, Boolean]\n  object InReplyToTweetIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object InReplyToUserIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object IsArticleFeature extends Feature[TweetCandidate, Boolean]\n  object IsAncestorCandidateFeature extends Feature[TweetCandidate, Boolean]\n  object IsBoostedCandidateFeature extends Feature[TweetCandidate, Boolean]\n  object IsExtendedReplyFeature\n      extends DataRecordFeature[TweetCandidate, Boolean]\n      with BoolDataRecordCompatible {\n    override val featureName: String = RecapFeatures.IS_EXTENDED_REPLY.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n  object IsInReplyToReplyOrDirectedFeature extends Feature[TweetCandidate, Boolean]\n  object IsInReplyToRetweetFeature extends Feature[TweetCandidate, Boolean]\n  object IsRandomTweetFeature\n      extends DataRecordFeature[TweetCandidate, Boolean]\n      with BoolDataRecordCompatible {\n    override val featureName: String = TimelinesSharedFeatures.IS_RANDOM_TWEET.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n  object IsReadFromCacheFeature extends Feature[TweetCandidate, Boolean]\n  object IsRetweetFeature extends Feature[TweetCandidate, Boolean]\n  object IsRetweetedReplyFeature extends Feature[TweetCandidate, Boolean]\n  object IsSupportAccountReplyFeature extends Feature[TweetCandidate, Boolean]\n  object LastScoredTimestampMsFeature extends Feature[TweetCandidate, Option[Long]]\n  object NonSelfFavoritedByUserIdsFeature extends Feature[TweetCandidate, Seq[Long]]\n  object NumImagesFeature extends Feature[TweetCandidate, Option[Int]]\n  object PositionFeature extends Feature[TweetCandidate, Option[Int]]\n\n  // Internal id generated per prediction service request\n  object PredictionRequestIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object QuotedTweetIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object QuotedUserIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object PhoenixScoreFeature extends Feature[TweetCandidate, Option[Double]]\n  object ScoreFeature extends Feature[TweetCandidate, Option[Double]]\n  object IsColdStartPostFeature extends Feature[TweetCandidate, Boolean]\n  object SemanticCoreIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object SimclustersTweetTopKClustersWithScoresFeature\n      extends Feature[TweetCandidate, Map[String, Double]]\n  // Tweet ID of the source tweet if the candidate is a retweet\n  object SourceTweetIdFeature extends Feature[TweetCandidate, Option[Long]]\n\n  // Tweet ID of the source tweet if the candidate is a retweet. Tweet id of the candidate otherwise\n  object OriginalTweetIdFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Long]\n      with LongDiscreteDataRecordCompatible {\n    override val featureName: String = TimelinesSharedFeatures.SOURCE_TWEET_ID.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set(pd.PersonalDataType.TweetId)\n  }\n  object SourceUserIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object ServedTypeFeature extends Feature[TweetCandidate, hmt.ServedType]\n  object TSPMetricTagFeature extends Feature[TweetCandidate, Set[tsp.MetricTag]]\n  object TweetLanguageFeature extends Feature[TweetCandidate, Option[String]]\n  object TweetMediaIdsFeature extends Feature[TweetCandidate, Seq[Long]]\n  object TweetMediaClusterIdsFeature extends Feature[TweetCandidate, Map[Long, Long]]\n  object TweetMediaCompletionRateFeature extends Feature[TweetCandidate, Option[Double]]\n  object ClipImageClusterIdsFeature extends Feature[TweetCandidate, Map[Long, Long]]\n  object MultiModalEmbeddingsFeature extends Feature[TweetCandidate, Option[Seq[Double]]]\n  object FirstMediaIdFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Long]\n      with LongDiscreteDataRecordCompatible {\n    override val featureName: String = TimelinesSharedFeatures.FIRST_MEDIA_ID.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set(pd.PersonalDataType.TweetId)\n  }\n  object TweetUrlsFeature extends Feature[TweetCandidate, Seq[String]]\n\n  object ViewCountFeature extends Feature[TweetCandidate, Option[Long]]\n  object VideoDurationMsFeature extends Feature[TweetCandidate, Option[Int]]\n  object ViewerIdFeature\n      extends DataRecordFeature[TweetCandidate, Long]\n      with LongDiscreteDataRecordCompatible {\n    override def featureName: String = SharedFeatures.USER_ID.getFeatureName\n    override def personalDataTypes: Set[pd.PersonalDataType] = Set(pd.PersonalDataType.UserId)\n  }\n  object WeightedModelScoreFeature extends Feature[TweetCandidate, Option[Double]]\n  object MentionUserIdFeature extends Feature[TweetCandidate, Seq[Long]]\n  object MentionScreenNameFeature extends Feature[TweetCandidate, Seq[String]]\n  object HasImageFeature extends Feature[TweetCandidate, Boolean]\n  object HasVideoFeature extends Feature[TweetCandidate, Boolean]\n  object VideoAspectRatioFeature extends Feature[TweetCandidate, Option[Float]]\n  object VideoDisplayTypeFeature extends Feature[TweetCandidate, Option[ModuleDisplayType]]\n  object VideoHeightFeature extends Feature[TweetCandidate, Option[Short]]\n  object VideoWidthFeature extends Feature[TweetCandidate, Option[Short]]\n  object MediaIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object HasMultipleMedia extends Feature[TweetCandidate, Boolean]\n  object MediaCategoryFeature extends Feature[TweetCandidate, Option[MediaCategory]]\n  object SemanticAnnotationIdsFeature extends Feature[TweetCandidate, Seq[Long]]\n  object TweetTypeMetricsFeature extends Feature[TweetCandidate, Option[Seq[Byte]]]\n  object CurrentPinnedTweetFeature extends Feature[TweetCandidate, Option[Long]]\n\n  // Tweetypie VF Features\n  object IsHydratedFeature extends Feature[TweetCandidate, Boolean]\n  object OonNsfwFeature extends Feature[TweetCandidate, Boolean]\n  object QuotedTweetDroppedFeature extends Feature[TweetCandidate, Boolean]\n  // Raw Tweet Text from Tweetypie\n  object TweetTextFeature extends Feature[TweetCandidate, Option[String]]\n  object TweetTextTokensFeature extends Feature[TweetCandidate, Option[Seq[Int]]]\n  object AuthorEnabledPreviewsFeature extends Feature[TweetCandidate, Boolean]\n  object IsTweetPreviewFeature extends Feature[TweetCandidate, Boolean]\n\n  // SGS Features\n  /**\n   * By convention, this is set to true for retweets of non-followed authors\n   * E.g. where somebody the viewer follows retweets a Tweet from somebody the viewer doesn't follow\n   */\n  object InNetworkFeature extends FeatureWithDefaultOnFailure[TweetCandidate, Boolean] {\n    override val defaultValue: Boolean = true\n  }\n\n  // Query Features\n  object AccountAgeFeature extends Feature[PipelineQuery, Option[Time]]\n  object ClientIdFeature\n      extends DataRecordOptionalFeature[PipelineQuery, Long]\n      with LongDiscreteDataRecordCompatible {\n    override def featureName: String = SharedFeatures.CLIENT_ID.getFeatureName\n    override def personalDataTypes: Set[pd.PersonalDataType] = Set(pd.PersonalDataType.ClientType)\n  }\n  object CachedScoredTweetsFeature extends Feature[PipelineQuery, Seq[hmt.ScoredTweet]]\n\n  object FollowsSportsAccountFeature extends Feature[PipelineQuery, Boolean]\n\n  object DeviceCountryFeature\n      extends DataRecordOptionalFeature[PipelineQuery, String]\n      with StringDataRecordCompatible {\n    override def featureName: String = RequestContextFeatures.COUNTRY_CODE.getFeatureName\n\n    override def personalDataTypes: Set[pd.PersonalDataType] =\n      Set(pd.PersonalDataType.PrivateCountryOrRegion, pd.PersonalDataType.InferredCountry)\n  }\n\n  object DeviceLanguageFeature\n      extends DataRecordOptionalFeature[PipelineQuery, String]\n      with StringDataRecordCompatible {\n    override def featureName: String = RequestContextFeatures.LANGUAGE_CODE.getFeatureName\n\n    override def personalDataTypes: Set[pd.PersonalDataType] = Set(\n      pd.PersonalDataType.GeneralSettings,\n      pd.PersonalDataType.ProvidedLanguage,\n      pd.PersonalDataType.InferredLanguage)\n  }\n\n  object UuaUserGenderFeature\n      extends DataRecordOptionalFeature[PipelineQuery, String]\n      with StringDataRecordCompatible {\n    override def featureName: String = UserHealthFeatures.UserGender.getFeatureName\n\n    override def personalDataTypes: Set[pd.PersonalDataType] = Set(\n      pd.PersonalDataType.GeneralSettings,\n      pd.PersonalDataType.ProvidedGender,\n      pd.PersonalDataType.InferredGender)\n  }\n\n  object UuaUserStateFeature\n      extends DataRecordOptionalFeature[PipelineQuery, Long]\n      with LongDiscreteDataRecordCompatible {\n    override def featureName: String = UserHealthFeatures.UserState.getFeatureName\n\n    override def personalDataTypes: Set[pd.PersonalDataType] = Set(\n      pd.PersonalDataType.UserState,\n      pd.PersonalDataType.UserType\n    )\n  }\n\n  object UuaUserAgeBucketFeature\n      extends DataRecordOptionalFeature[PipelineQuery, String]\n      with StringDataRecordCompatible {\n    override def featureName: String = UserHealthFeatures.UserAgeBucket.getFeatureName\n\n    override def personalDataTypes: Set[pd.PersonalDataType] = Set(\n      pd.PersonalDataType.GeneralSettings,\n      pd.PersonalDataType.ProvidedAge,\n      pd.PersonalDataType.InferredAge\n    )\n  }\n\n  object DismissInfoFeature\n      extends FeatureWithDefaultOnFailure[PipelineQuery, Map[st.SuggestType, Option[DismissInfo]]] {\n    override def defaultValue: Map[st.SuggestType, Option[DismissInfo]] = Map.empty\n  }\n  object FollowingLastNonPollingTimeFeature extends Feature[PipelineQuery, Option[Time]]\n  object GetInitialFeature\n      extends DataRecordFeature[PipelineQuery, Boolean]\n      with BoolDataRecordCompatible {\n    override def featureName: String = RequestContextFeatures.IS_GET_INITIAL.getFeatureName\n    override def personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n  object GetMiddleFeature\n      extends DataRecordFeature[PipelineQuery, Boolean]\n      with BoolDataRecordCompatible {\n    override def featureName: String = RequestContextFeatures.IS_GET_MIDDLE.getFeatureName\n    override def personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n  object GetNewerFeature\n      extends DataRecordFeature[PipelineQuery, Boolean]\n      with BoolDataRecordCompatible {\n    override def featureName: String = RequestContextFeatures.IS_GET_NEWER.getFeatureName\n    override def personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n  object GetOlderFeature\n      extends DataRecordFeature[PipelineQuery, Boolean]\n      with BoolDataRecordCompatible {\n    override def featureName: String = RequestContextFeatures.IS_GET_OLDER.getFeatureName\n    override def personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n  object GuestIdFeature\n      extends DataRecordOptionalFeature[PipelineQuery, Long]\n      with LongDiscreteDataRecordCompatible {\n    override def featureName: String = SharedFeatures.GUEST_ID.getFeatureName\n    override def personalDataTypes: Set[pd.PersonalDataType] = Set(pd.PersonalDataType.GuestId)\n  }\n\n  object GrokAnnotationsFeature extends Feature[TweetCandidate, Option[hmt.GrokAnnotations]]\n\n  object GrokTopCategoryFeature extends Feature[TweetCandidate, Option[Long]]\n\n  object HasDarkRequestFeature extends Feature[PipelineQuery, Option[Boolean]]\n  object ImpressionBloomFilterFeature\n      extends FeatureWithDefaultOnFailure[PipelineQuery, blm.ImpressionBloomFilterSeq] {\n    override def defaultValue: blm.ImpressionBloomFilterSeq =\n      blm.ImpressionBloomFilterSeq(Seq.empty)\n  }\n  object ImpressedMediaIds extends FeatureWithDefaultOnFailure[PipelineQuery, Seq[Long]] {\n    override val defaultValue: Seq[Long] = Seq.empty\n  }\n  object IsForegroundRequestFeature extends Feature[PipelineQuery, Boolean]\n  object IsLaunchRequestFeature extends Feature[PipelineQuery, Boolean]\n  object LastNonPollingTimeFeature extends Feature[PipelineQuery, Option[Time]]\n  object LastNegativeFeedbackTimeFeature extends Feature[PipelineQuery, Option[Time]]\n  object LowSignalUserFeature extends Feature[PipelineQuery, Boolean]\n  object NaviClientConfigFeature extends Feature[PipelineQuery, NaviClientConfig]\n  object NonPollingTimesFeature extends Feature[PipelineQuery, Seq[Long]]\n  object PersistenceEntriesFeature extends Feature[PipelineQuery, Seq[TimelineResponseV3]]\n  object PollingFeature extends Feature[PipelineQuery, Boolean]\n  object PullToRefreshFeature extends Feature[PipelineQuery, Boolean]\n  // Scores from Real Graph representing the relationship between the viewer and another user\n  object RealGraphInNetworkScoresFeature extends Feature[PipelineQuery, Map[UserId, Double]]\n  object ImmersiveClientEmbeddingsFeature extends Feature[PipelineQuery, Map[Long, Seq[Double]]]\n  object RequestJoinIdFeature extends Feature[TweetCandidate, Option[Long]]\n  // Internal id generated per request\n  object ServedIdFeature extends Feature[PipelineQuery, Option[Long]]\n  object ServedTweetIdsFeature extends Feature[PipelineQuery, Seq[Long]]\n  object ServedAuthorIdsFeature extends Feature[PipelineQuery, Map[Long, Seq[Long]]]\n  object ServedTweetPreviewIdsFeature extends Feature[PipelineQuery, Seq[Long]]\n\n  object SignupSourceFeature extends Feature[PipelineQuery, Option[SignupSource]]\n  object SignupCountryFeature extends Feature[PipelineQuery, Option[String]]\n  object ViewerAllowsAdsPersonalizationFeature extends Feature[PipelineQuery, Option[Boolean]]\n  object ViewerAllowsForYouRecommendationsFeature extends Feature[PipelineQuery, Option[Boolean]]\n  object ViewerAllowsDataSharingFeature extends Feature[PipelineQuery, Option[Boolean]]\n  object TimelineServiceTweetsFeature extends Feature[PipelineQuery, Seq[Long]]\n  object TLSOriginalTweetsWithAuthorFeature\n      extends Feature[PipelineQuery, Seq[(Long, Option[Long])]]\n\n  object TLSOriginalTweetsWithConfirmedAuthorFeature\n      extends Feature[PipelineQuery, Seq[(Long, Long)]]\n\n  object TweetAuthorFollowersFeature extends Feature[PipelineQuery, Map[Long, Option[Long]]]\n\n  object TimestampFeature\n      extends DataRecordFeature[PipelineQuery, Long]\n      with LongDiscreteDataRecordCompatible {\n    override val featureName: String = SharedFeatures.TIMESTAMP.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n  object TimestampGMTDowFeature\n      extends DataRecordFeature[PipelineQuery, Long]\n      with LongDiscreteDataRecordCompatible {\n    override val featureName: String = RequestContextFeatures.TIMESTAMP_GMT_DOW.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n  object TimestampGMTHourFeature\n      extends DataRecordFeature[PipelineQuery, Long]\n      with LongDiscreteDataRecordCompatible {\n    override val featureName: String = RequestContextFeatures.TIMESTAMP_GMT_HOUR.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n  object TweetMixerScoreFeature\n      extends DataRecordOptionalFeature[TweetCandidate, Double]\n      with DoubleDataRecordCompatible {\n    override val featureName: String = TimelinesSharedFeatures.CANDIDATE_SOURCE_SCORE.getFeatureName\n    override val personalDataTypes: Set[PersonalDataType] = Set(pd.PersonalDataType.EngagementScore)\n  }\n\n  object SourceSignalFeature extends Feature[TweetCandidate, Option[SourceSignal]]\n\n  object DebugStringFeature extends Feature[TweetCandidate, Option[String]]\n\n  object IsSelfThreadFeature\n      extends DataRecordFeature[PipelineQuery, Boolean]\n      with BoolDataRecordCompatible {\n    override val featureName: String =\n      conversation_features.ConversationFeatures.IS_SELF_THREAD_TWEET.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  }\n\n  object TweetLanguageFromTweetypieFeature\n      extends DataRecordOptionalFeature[TweetCandidate, String]\n      with StringDataRecordCompatible {\n    override val featureName: String =\n      LanguageFeatures.TweetLanguageFromTweetypieFeature.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] =\n      Set(pd.PersonalDataType.InferredLanguage)\n  }\n  object TweetLanguageFromLanguageSignalFeature\n      extends DataRecordOptionalFeature[PipelineQuery, String]\n      with StringDataRecordCompatible {\n    override val featureName: String =\n      LanguageFeatures.TweetLanguageFromLanguageSignalFeature.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set(\n      pd.PersonalDataType.InferredLanguage)\n  }\n  object UserActionsFeature extends Feature[PipelineQuery, Option[UserActionSequence]]\n  object UserActionsByteArrayFeature extends Feature[PipelineQuery, Option[Array[Byte]]]\n  object UserActionsSizeFeature extends Feature[PipelineQuery, Option[Int]]\n  object UserActionsContainsExplicitSignalsFeature extends Feature[PipelineQuery, Boolean]\n  object UserEngagedLanguagesFeature\n      extends Feature[PipelineQuery, Set[String]]\n      with SparseBinaryDataRecordCompatible {\n    override val featureName: String = LanguageFeatures.EngagedLanguages.getFeatureName\n    override val personalDataTypes: Set[pd.PersonalDataType] = Set(\n      pd.PersonalDataType.InferredLanguage)\n  }\n  object UserFollowedTopicsCountFeature extends Feature[PipelineQuery, Option[Int]]\n  object UserFollowingCountFeature extends Feature[PipelineQuery, Option[Int]]\n  object UserFollowersCountFeature extends Feature[PipelineQuery, Option[Int]]\n  object UserRecentEngagementTweetIdsFeature extends Feature[PipelineQuery, Seq[Long]]\n  object UserLastExplicitSignalTimeFeature extends Feature[PipelineQuery, Option[Time]]\n  object UserUnderstandableLanguagesFeature extends Feature[PipelineQuery, Seq[String]]\n  object UserScreenNameFeature extends Feature[PipelineQuery, Option[String]]\n  object UserStateFeature extends Feature[PipelineQuery, Option[um.UserState]]\n  object UserTypeFeature extends Feature[PipelineQuery, Option[gt.UserType]]\n  object ViewerSafetyLabels extends Feature[PipelineQuery, Option[Seq[String]]]\n  object ViewerIsRateLimited extends Feature[PipelineQuery, Boolean]\n  object ViewerHasJobRecommendationsEnabled extends Feature[PipelineQuery, Boolean]\n  object ViewerHasPremiumTier extends Feature[PipelineQuery, Boolean]\n  object ViewerHasRecruitingOrganizationRecommendationsEnabled\n      extends Feature[PipelineQuery, Boolean]\n  object WhoToFollowExcludedUserIdsFeature\n      extends FeatureWithDefaultOnFailure[PipelineQuery, Seq[Long]] {\n    override def defaultValue = Seq.empty\n  }\n\n  object NSFWConsumerScoreFeature extends Feature[PipelineQuery, Double]\n  object NSFWConsumerFollowerScoreFeature extends Feature[PipelineQuery, Double]\n\n  object CurrentDisplayedGrokTopicFeature\n      extends FeatureWithDefaultOnFailure[PipelineQuery, Option[(Long, String)]] {\n    override val defaultValue: Option[(Long, String)] = None\n  }\n\n  // Result Features\n  object ServedSizeFeature extends Feature[PipelineQuery, Option[Int]]\n  object UniqueAuthorCountFeature extends Feature[PipelineQuery, Option[Int]]\n  object MaxSingleAuthorCountFeature extends Feature[PipelineQuery, Option[Int]]\n  object UniqueCategoryCountFeature extends Feature[PipelineQuery, Option[Int]]\n  object MaxSingleCategoryCountFeature extends Feature[PipelineQuery, Option[Int]]\n  object ServedInConversationModuleFeature extends Feature[TweetCandidate, Boolean]\n  object ConversationModule2DisplayedTweetsFeature extends Feature[TweetCandidate, Boolean]\n  object ConversationModuleHasGapFeature extends Feature[TweetCandidate, Boolean]\n  object SGSValidLikedByUserIdsFeature extends Feature[TweetCandidate, Seq[Long]]\n  object SGSValidFollowedByUserIdsFeature extends Feature[TweetCandidate, Seq[Long]]\n  object ScreenNamesFeature extends Feature[TweetCandidate, Map[Long, String]]\n  object RealNamesFeature extends Feature[TweetCandidate, Map[Long, String]]\n  object TweetAgeFeature extends Feature[TweetCandidate, Option[Long]]\n\n  /**\n   * Features around the focal Tweet for Tweets which should be rendered in convo modules.\n   * These are needed in order to render social context above the root tweet in a convo modules.\n   * For example if we have a convo module A-B-C (A Tweets, B replies to A, C replies to B), the descendant features are\n   * for the Tweet C. These features are None except for the root Tweet for Tweets which should render into\n   * convo modules.\n   */\n  object FocalTweetAuthorIdFeature extends Feature[TweetCandidate, Option[Long]]\n  object FocalTweetInNetworkFeature extends Feature[TweetCandidate, Option[Boolean]]\n  object FocalTweetRealNamesFeature extends Feature[TweetCandidate, Option[Map[Long, String]]]\n  object FocalTweetScreenNamesFeature extends Feature[TweetCandidate, Option[Map[Long, String]]]\n  object MediaUnderstandingAnnotationIdsFeature extends Feature[TweetCandidate, Seq[Long]]\n  object AdTagUrlFeature extends Feature[TweetCandidate, Option[String]]\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/HomeLargeEmbeddingsFeatures.scala",
    "content": "package com.twitter.home_mixer.model\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject HomeLargeEmbeddingsFeatures {\n  object AuthorLargeEmbeddingsFeature\n      extends DataRecordInAFeature[TweetCandidate]\n      with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n    override def defaultValue: DataRecord = new DataRecord()\n  }\n\n  object AuthorLargeEmbeddingsKeyFeature extends Feature[TweetCandidate, Seq[Long]]\n\n  object OriginalAuthorLargeEmbeddingsFeature\n      extends DataRecordInAFeature[TweetCandidate]\n      with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n    override def defaultValue: DataRecord = new DataRecord()\n  }\n\n  object OriginalAuthorLargeEmbeddingsKeyFeature extends Feature[TweetCandidate, Seq[Long]]\n\n  object TweetLargeEmbeddingsFeature\n      extends DataRecordInAFeature[TweetCandidate]\n      with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n    override def defaultValue: DataRecord = new DataRecord()\n  }\n\n  object TweetLargeEmbeddingsKeyFeature extends Feature[TweetCandidate, Seq[Long]]\n\n  object OriginalTweetLargeEmbeddingsFeature\n      extends DataRecordInAFeature[TweetCandidate]\n      with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n    override def defaultValue: DataRecord = new DataRecord()\n  }\n\n  object OriginalTweetLargeEmbeddingsKeyFeature extends Feature[TweetCandidate, Seq[Long]]\n\n  object UserLargeEmbeddingsFeature\n      extends DataRecordInAFeature[PipelineQuery]\n      with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n    override def defaultValue: DataRecord = new DataRecord()\n  }\n\n  object UserLargeEmbeddingsKeyFeature extends Feature[PipelineQuery, Seq[Long]]\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/NaviClientConfig.scala",
    "content": "package com.twitter.home_mixer.model\n\ncase class NaviClientConfig(\n  clientName: String,\n  customizedBatchSize: Option[Int],\n  clusterStr: String)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/NavigationIncludeInstruction.scala",
    "content": "package com.twitter.home_mixer.model\n\nimport com.twitter.home_mixer.model.request.DeviceContext.RequestContext\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.IncludeInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\n\n/**\n * Include a navigation timeline instruction when we satisfy criteria\n */\ncase class NavigationIncludeInstruction(\n  ptrEnableParam: FSParam[Boolean],\n  coldStartEnableParam: FSParam[Boolean],\n  warmStartEnableParam: FSParam[Boolean],\n  navigateEnableParam: FSParam[Boolean],\n  manualRefreshEnableParam: FSParam[Boolean])\n    extends IncludeInstruction[PipelineQuery with HasDeviceContext] {\n\n  override def apply(\n    query: PipelineQuery with HasDeviceContext,\n    entries: Seq[TimelineEntry]\n  ): Boolean = {\n    val requestContext = query.deviceContext.flatMap(_.requestContextValue)\n\n    val ptrEnabled =\n      query.params(ptrEnableParam) && requestContext.contains(RequestContext.PullToRefresh)\n    val coldStartEnabled =\n      query.params(coldStartEnableParam) && requestContext.contains(RequestContext.Launch)\n    val warmStartEnabled =\n      query.params(warmStartEnableParam) && requestContext.contains(RequestContext.Foreground)\n    val manualRefreshEnabled =\n      query.params(manualRefreshEnableParam) && requestContext.contains(\n        RequestContext.ManualRefresh)\n    val navigateEnabled =\n      query.params(navigateEnableParam) && requestContext.contains(RequestContext.Navigate)\n\n    ptrEnabled || coldStartEnabled || warmStartEnabled || manualRefreshEnabled || navigateEnabled\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/PhoenixPredictedScoreFeature.scala",
    "content": "package com.twitter.home_mixer.model\n\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoDurationMsFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelWeights\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.UseProdInPhoenixParams\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.x.user_action_sequence.ActionName\nimport com.x.user_action_sequence.ActionName._\n\nsealed trait PhoenixPredictedScoreFeature extends Feature[TweetCandidate, Option[Double]] {\n  def featureName: String\n  def modelWeightParam: FSBoundedParam[Double]\n  def actions: Seq[ActionName]\n  def isEligible(features: FeatureMap): Boolean = true\n  def prodScoreFeature: PredictedScoreFeature\n  def useProdFeatureParam: FSParam[Boolean]\n  def extractScore(features: FeatureMap, query: PipelineQuery): Option[Double] = {\n    if (query.params(useProdFeatureParam))\n      prodScoreFeature.extractScore(features, query)\n    else features.getOrElse(this, None).orElse(prodScoreFeature.extractScore(features, query))\n  }\n}\n\nobject PhoenixPredictedFavoriteScoreFeature extends PhoenixPredictedScoreFeature {\n  override val featureName = \"fav\"\n  override val modelWeightParam = ModelWeights.FavParam\n  override val actions = Seq(SERVER_TWEET_FAV)\n\n  override def prodScoreFeature = PredictedFavoriteScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdFavForPhoenixParam\n}\n\nobject PhoenixPredictedReplyScoreFeature extends PhoenixPredictedScoreFeature {\n  override val featureName = \"reply\"\n  override val modelWeightParam = ModelWeights.ReplyParam\n  override val actions = Seq(SERVER_TWEET_REPLY)\n\n  override def prodScoreFeature = PredictedReplyScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdReplyForPhoenixParam\n}\n\nobject PhoenixPredictedRetweetScoreFeature extends PhoenixPredictedScoreFeature {\n  override val featureName = \"retweet\"\n  override val modelWeightParam = ModelWeights.RetweetParam\n  override val actions = Seq(SERVER_TWEET_QUOTE, SERVER_TWEET_RETWEET)\n\n  override def prodScoreFeature = PredictedRetweetScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdRetweetForPhoenixParam\n}\n\nobject PhoenixPredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature\n    extends PhoenixPredictedScoreFeature {\n  override val featureName = \"click_engage\"\n  override val modelWeightParam = ModelWeights.GoodClickV1Param\n  override val actions = Seq(CLIENT_TWEET_PHOTO_EXPAND)\n\n  override def prodScoreFeature = PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdGoodClickV1ForPhoenixParam\n}\n\nobject PhoenixPredictedGoodClickConvoDescUamGt2ScoreFeature extends PhoenixPredictedScoreFeature {\n  override val featureName = \"click_dwell\"\n  override val modelWeightParam = ModelWeights.GoodClickV2Param\n  override val actions = Seq(CLIENT_TWEET_CLICK)\n\n  override def prodScoreFeature = PredictedGoodClickConvoDescUamGt2ScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdGoodClickV2ForPhoenixParam\n}\n\nobject PhoenixPredictedGoodProfileClickScoreFeature extends PhoenixPredictedScoreFeature {\n  override val featureName = \"good_profile_click\"\n  override val modelWeightParam = ModelWeights.GoodProfileClickParam\n  override val actions = Seq(CLIENT_TWEET_CLICK_PROFILE)\n\n  override def prodScoreFeature = PredictedGoodProfileClickScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdProfileClickForPhoenixParam\n}\n\nobject PhoenixPredictedVideoQualityViewScoreFeature extends PhoenixPredictedScoreFeature {\n  override val featureName = \"vqv\"\n  override val modelWeightParam = ModelWeights.VideoQualityViewParam\n  override val actions = Seq(CLIENT_TWEET_VIDEO_QUALITY_VIEW)\n\n  override def isEligible(features: FeatureMap): Boolean = {\n    val isVideoDurationGte10Seconds =\n      (features.getOrElse(VideoDurationMsFeature, None).getOrElse(0) / 1000.0) >= 10\n    val hasVideoFeature = features.getOrElse(HasVideoFeature, false)\n    hasVideoFeature && isVideoDurationGte10Seconds\n  }\n  override def prodScoreFeature = PredictedVideoQualityViewScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdVQVForPhoenixParam\n}\n\nobject PhoenixPredictedShareScoreFeature extends PhoenixPredictedScoreFeature {\n  override val featureName = \"share\"\n  override val modelWeightParam = ModelWeights.ShareParam\n  override val actions = Seq(\n    CLIENT_TWEET_SHARE_VIA_COPY_LINK,\n    CLIENT_TWEET_CLICK_SEND_VIA_DIRECT_MESSAGE,\n    CLIENT_TWEET_SHARE\n  )\n\n  override def prodScoreFeature = PredictedShareScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdShareForPhoenixParam\n}\n\nobject PhoenixPredictedDwellScoreFeature extends PhoenixPredictedScoreFeature {\n  override val featureName = \"dwell\"\n  override val modelWeightParam = ModelWeights.DwellParam\n  override val actions = Seq(CLIENT_TWEET_RECAP_DWELLED)\n\n  override def isEligible(features: FeatureMap): Boolean = {\n    val isVideoDurationGte10Seconds =\n      (features.getOrElse(VideoDurationMsFeature, None).getOrElse(0) / 1000.0) >= 10\n    val hasVideoFeature = features.getOrElse(HasVideoFeature, false)\n    !(hasVideoFeature && isVideoDurationGte10Seconds)\n  }\n  override def prodScoreFeature = PredictedDwellScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdDwellForPhoenixParam\n}\n\nobject PhoenixPredictedOpenLinkScoreFeature extends PhoenixPredictedScoreFeature {\n  override val featureName = \"open_link\"\n  override val modelWeightParam = ModelWeights.OpenLinkParam\n  override val actions = Seq(CLIENT_TWEET_OPEN_LINK)\n\n  // Placeholder prod score feature. This should not be used.\n  override def prodScoreFeature = PredictedShareScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdOpenLinkForPhoenixParam\n}\n\nobject PhoenixPredictedScreenshotScoreFeature extends PhoenixPredictedScoreFeature {\n  override val featureName = \"screenshot\"\n  override val modelWeightParam = ModelWeights.OpenLinkParam\n  override val actions = Seq(CLIENT_TWEET_TAKE_SCREENSHOT)\n\n  // Placeholder prod score feature. This should not be used.\n  override def prodScoreFeature = PredictedShareScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdScreenshotForPhoenixParam\n}\n\nobject PhoenixPredictedBookmarkScoreFeature extends PhoenixPredictedScoreFeature {\n  override val featureName = \"bookmark\"\n  override val modelWeightParam = ModelWeights.BookmarkParam\n  override val actions = Seq(CLIENT_TWEET_BOOKMARK)\n\n  // Placeholder prod score feature. This should not be used.\n  override def prodScoreFeature = PredictedBookmarkScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdBookmarkForPhoenixParam\n}\n\n// Negative Engagements\nobject PhoenixPredictedNegativeFeedbackV2ScoreFeature extends PhoenixPredictedScoreFeature {\n  override val featureName = \"negative_feedback_v2\"\n  override val modelWeightParam = ModelWeights.NegativeFeedbackV2Param\n  override val actions = Seq(\n    CLIENT_TWEET_NOT_INTERESTED_IN,\n    CLIENT_TWEET_BLOCK_AUTHOR,\n    CLIENT_TWEET_MUTE_AUTHOR,\n    CLIENT_TWEET_REPORT,\n  )\n\n  override def prodScoreFeature = PredictedNegativeFeedbackV2ScoreFeature\n  override def useProdFeatureParam = UseProdInPhoenixParams.EnableProdNegForPhoenixParam\n}\n\nobject PhoenixPredictedScoreFeature {\n  val PhoenixPredictedScoreFeatures: Seq[PhoenixPredictedScoreFeature] = Seq(\n    PhoenixPredictedFavoriteScoreFeature,\n    PhoenixPredictedReplyScoreFeature,\n    PhoenixPredictedRetweetScoreFeature,\n    PhoenixPredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature,\n    PhoenixPredictedGoodClickConvoDescUamGt2ScoreFeature,\n    PhoenixPredictedGoodProfileClickScoreFeature,\n    PhoenixPredictedVideoQualityViewScoreFeature,\n    PhoenixPredictedShareScoreFeature,\n    PhoenixPredictedDwellScoreFeature,\n    PhoenixPredictedOpenLinkScoreFeature,\n    PhoenixPredictedScreenshotScoreFeature,\n    PhoenixPredictedBookmarkScoreFeature,\n    // Negative Engagements\n    PhoenixPredictedNegativeFeedbackV2ScoreFeature,\n  )\n\n  val PhoenixPredictedScoreFeatureSet: Set[PhoenixPredictedScoreFeature] =\n    PhoenixPredictedScoreFeatures.toSet\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/PredictedScoreFeature.scala",
    "content": "package com.twitter.home_mixer.model\n\nimport com.twitter.dal.personal_data.thriftjava.PersonalDataType\nimport com.twitter.dal.personal_data.{thriftjava => pd}\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoDurationMsFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableImmersiveVQV\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableTenSecondsLogicForVQV\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ScoreThresholdForVQVParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.EnableBinarySchemeForVQVParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.BinarySchemeConstantForVQVParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ScoreThresholdForDwellParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.EnableDwellOrVQVParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.EnableBinarySchemeForDwellParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelBiases\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelDebiases\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelWeights\nimport com.twitter.ml.api.DataType\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordOptionalFeature\nimport com.twitter.product_mixer.core.feature.datarecord.DoubleDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.prediction.features.recap.RecapFeatures\nimport com.twitter.ml.api.thriftscala.GeneralTensor\nimport com.twitter.product_mixer.core.feature.datarecord.GeneralTensorDataRecordCompatible\n\nsealed trait PredictedScoreFeature\n    extends DataRecordOptionalFeature[TweetCandidate, Double]\n    with DoubleDataRecordCompatible {\n\n  override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n\n  // Note: Determine whether the score prediction should contribute to the weighted score\n  // Currently, it is being used for PredictedVideoQualityViewScoreFeature\n  def isEligible(\n    features: FeatureMap,\n    query: PipelineQuery\n  ): Boolean = true\n  def statName: String\n  def modelWeightParam: FSBoundedParam[Double]\n  def modelBiasParam: Option[FSBoundedParam[Double]] = None\n  def modelDebiasParam: Option[FSBoundedParam[Double]] = None\n  def extractScore(features: FeatureMap, query: PipelineQuery): Option[Double] =\n    features.getOrElse(this, None)\n\n  def weightQueryFeatureName: String\n  lazy val weightQueryFeature: Feature[PipelineQuery, Option[Double]] =\n    PredictedScoreFeature.getDataRecordFeatureFromName(weightQueryFeatureName)\n  def biasQueryFeatureName: Option[String] = None\n  lazy val biasQueryFeature: Option[Feature[PipelineQuery, Option[Double]]] =\n    biasQueryFeatureName.map(PredictedScoreFeature.getDataRecordFeatureFromName)\n\n  def debiasQueryFeatureName: Option[String] = None\n\n  lazy val debiasQueryFeature: Option[Feature[PipelineQuery, Option[GeneralTensor]]] =\n    debiasQueryFeatureName.map(PredictedScoreFeature.getGeneralTensorFeatureFromName)\n}\n\nobject PredictedFavoriteScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_FAVORITED.getFeatureName\n  override val statName = \"fav\"\n  override val modelWeightParam = ModelWeights.FavParam\n  override val weightQueryFeatureName = RecapFeatures.WEIGHT_IS_FAVORITED.getFeatureName\n  override val modelDebiasParam = Some(ModelDebiases.FavParam)\n  override val debiasQueryFeatureName = Some(RecapFeatures.DEBIAS_IS_FAVORITED.getFeatureName)\n}\n\nobject PredictedReplyScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_REPLIED.getFeatureName\n  override val statName = \"reply\"\n  override val modelWeightParam = ModelWeights.ReplyParam\n  override val weightQueryFeatureName = RecapFeatures.WEIGHT_IS_REPLIED.getFeatureName\n  override val modelDebiasParam = Some(ModelDebiases.ReplyParam)\n  override val debiasQueryFeatureName = Some(RecapFeatures.DEBIAS_IS_REPLIED.getFeatureName)\n}\n\nobject PredictedRetweetScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_RETWEETED.getFeatureName\n  override val statName = \"retweet\"\n  override val modelWeightParam = ModelWeights.RetweetParam\n  override val weightQueryFeatureName = RecapFeatures.WEIGHT_IS_RETWEETED.getFeatureName\n  override val modelDebiasParam = Some(ModelDebiases.RetweetParam)\n  override val debiasQueryFeatureName = Some(RecapFeatures.DEBIAS_IS_RETWEETED.getFeatureName)\n}\n\nobject PredictedReplyEngagedByAuthorScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_REPLIED_REPLY_ENGAGED_BY_AUTHOR.getFeatureName\n  override val statName = \"reply_engaged_by_author\"\n  override val modelWeightParam = ModelWeights.ReplyEngagedByAuthorParam\n  override val weightQueryFeatureName =\n    RecapFeatures.WEIGHT_IS_REPLIED_REPLY_ENGAGED_BY_AUTHOR.getFeatureName\n  override val modelDebiasParam = Some(ModelDebiases.ReplyEngagedByAuthorParam)\n  override val debiasQueryFeatureName =\n    Some(RecapFeatures.DEBIAS_IS_REPLIED_REPLY_ENGAGED_BY_AUTHOR.getFeatureName)\n}\n\nobject PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_GOOD_CLICKED_V1.getFeatureName\n  override val statName = \"click_engaged\" // click_convo_desc_favorited_or_replied\n  override val modelWeightParam = ModelWeights.GoodClickV1Param\n  override val weightQueryFeatureName = RecapFeatures.WEIGHT_IS_GOOD_CLICKED_V1.getFeatureName\n  override val modelDebiasParam = Some(ModelDebiases.GoodClickV1Param)\n  override val debiasQueryFeatureName = Some(RecapFeatures.DEBIAS_IS_GOOD_CLICKED_V1.getFeatureName)\n}\n\nobject PredictedGoodClickConvoDescUamGt2ScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_GOOD_CLICKED_V2.getFeatureName\n  override val statName = \"click_dwell\" // good_click_convo_desc_uam_gt_2\n  override val modelWeightParam = ModelWeights.GoodClickV2Param\n  override val weightQueryFeatureName = RecapFeatures.WEIGHT_IS_GOOD_CLICKED_V2.getFeatureName\n  override val modelDebiasParam = Some(ModelDebiases.GoodClickV2Param)\n  override val debiasQueryFeatureName = Some(RecapFeatures.DEBIAS_IS_GOOD_CLICKED_V2.getFeatureName)\n}\n\nobject PredictedGoodProfileClickScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_PROFILE_CLICKED_AND_PROFILE_ENGAGED.getFeatureName\n  override val statName = \"good_profile_click\"\n  override val modelWeightParam = ModelWeights.GoodProfileClickParam\n  override val weightQueryFeatureName =\n    RecapFeatures.WEIGHT_IS_PROFILE_CLICKED_AND_PROFILE_ENGAGED.getFeatureName\n  override val modelDebiasParam = Some(ModelDebiases.GoodProfileClickParam)\n  override val debiasQueryFeatureName =\n    Some(RecapFeatures.DEBIAS_IS_PROFILE_CLICKED_AND_PROFILE_ENGAGED.getFeatureName)\n}\n\nobject PredictedVideoQualityViewImmersiveScoreFeature extends PredictedScoreFeature {\n  override def isEligible(\n    features: FeatureMap,\n    query: PipelineQuery\n  ): Boolean = {\n    query.params(EnableImmersiveVQV)\n  }\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_VIDEO_QUALITY_VIEWED_IMMERSIVE.getFeatureName\n  override val statName = \"vqv_immersive\" // video_quality_viewed_immersive\n  override val modelWeightParam = ModelWeights.VideoQualityViewImmersiveParam\n  override val weightQueryFeatureName =\n    RecapFeatures.WEIGHT_IS_VIDEO_QUALITY_VIEWED_IMMERSIVE.getFeatureName\n  override val modelBiasParam = Some(ModelBiases.VideoQualityViewImmersiveParam)\n  override val biasQueryFeatureName = Some(\n    RecapFeatures.BIAS_IS_VIDEO_QUALITY_VIEWED_IMMERSIVE.getFeatureName)\n  override val modelDebiasParam = Some(ModelDebiases.VideoQualityViewImmersiveParam)\n  override val debiasQueryFeatureName =\n    Some(RecapFeatures.DEBIAS_IS_VIDEO_QUALITY_VIEWED_IMMERSIVE.getFeatureName)\n\n}\n\nobject PredictedVideoQualityViewScoreFeature extends PredictedScoreFeature {\n  override def isEligible(\n    features: FeatureMap,\n    query: PipelineQuery\n  ): Boolean = {\n    val isTenSecondsLogicEnabled = query.params(EnableTenSecondsLogicForVQV)\n    val isVideoDurationGte10Seconds =\n      (features.getOrElse(VideoDurationMsFeature, None).getOrElse(0) / 1000.0) >= 10\n    val hasVideoFeature = features.getOrElse(HasVideoFeature, false)\n\n    hasVideoFeature && (!isTenSecondsLogicEnabled || isVideoDurationGte10Seconds)\n  }\n  override val featureName: String = RecapFeatures.PREDICTED_IS_VIDEO_QUALITY_VIEWED.getFeatureName\n  override val statName = \"vqv\" // video_quality_viewed\n  override val modelWeightParam = ModelWeights.VideoQualityViewParam\n  override val weightQueryFeatureName = RecapFeatures.WEIGHT_IS_VIDEO_QUALITY_VIEWED.getFeatureName\n  override val modelBiasParam = Some(ModelBiases.VideoQualityViewParam)\n  override val biasQueryFeatureName = Some(\n    RecapFeatures.BIAS_IS_VIDEO_QUALITY_VIEWED.getFeatureName)\n  override val modelDebiasParam = Some(ModelDebiases.VideoQualityViewParam)\n  override val debiasQueryFeatureName = Some(\n    RecapFeatures.DEBIAS_IS_VIDEO_QUALITY_VIEWED.getFeatureName)\n\n  override def extractScore(features: FeatureMap, query: PipelineQuery): Some[Double] = {\n    // For VQV, if the score is below a threshold, we return 0\n    val vqvScore = features.getOrElse(this, None).getOrElse(0.0)\n    if (vqvScore < query.params(ScoreThresholdForVQVParam)) {\n      // the default threshold is 0.0, vqvScore should be always non-negative\n      Some(0.0)\n    } else if (query.params(EnableBinarySchemeForVQVParam)) {\n      // If the binary scheme is enabled, we return a constant\n      Some(query.params(BinarySchemeConstantForVQVParam))\n    } else {\n      Some(vqvScore)\n    }\n  }\n\n}\n\nobject PredictedBookmarkScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_BOOKMARKED.getFeatureName\n  override val statName = \"bookmark\"\n  override val modelWeightParam = ModelWeights.BookmarkParam\n  override val weightQueryFeatureName = RecapFeatures.WEIGHT_IS_BOOKMARKED.getFeatureName\n  override val modelDebiasParam = Some(ModelDebiases.BookmarkParam)\n  override val debiasQueryFeatureName = Some(RecapFeatures.DEBIAS_IS_BOOKMARKED.getFeatureName)\n}\n\nobject PredictedShareScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_SHARED.getFeatureName\n  override val statName = \"share\"\n  override val modelWeightParam = ModelWeights.ShareParam\n  override val weightQueryFeatureName = RecapFeatures.WEIGHT_IS_SHARED.getFeatureName\n  override val modelDebiasParam = Some(ModelDebiases.ShareParam)\n  override val debiasQueryFeatureName = Some(RecapFeatures.DEBIAS_IS_SHARED.getFeatureName)\n}\n\nobject PredictedDwellScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_DWELLED.getFeatureName\n  override val statName = \"dwell\"\n  override val modelWeightParam = ModelWeights.DwellParam\n  override val weightQueryFeatureName = RecapFeatures.WEIGHT_IS_DWELLED.getFeatureName\n  override val modelDebiasParam = Some(ModelDebiases.DwellParam)\n  override val debiasQueryFeatureName = Some(RecapFeatures.DEBIAS_IS_DWELLED.getFeatureName)\n\n  override def isEligible(\n    features: FeatureMap,\n    query: PipelineQuery\n  ): Boolean = {\n    val isTenSecondsLogicEnabled = query.params(EnableTenSecondsLogicForVQV)\n    val isVideoDurationGte10Seconds =\n      (features.getOrElse(VideoDurationMsFeature, None).getOrElse(0) / 1000.0) >= 10\n    val hasVideoFeature = features.getOrElse(HasVideoFeature, false)\n\n    val isEligibleForVqv =\n      hasVideoFeature && (!isTenSecondsLogicEnabled || isVideoDurationGte10Seconds)\n    !(query.params(EnableDwellOrVQVParam) && isEligibleForVqv)\n  }\n\n  override def extractScore(features: FeatureMap, query: PipelineQuery): Some[Double] = {\n    // For Dwell, if the score is below a threshold, we return 0\n    val dwellScore = features.getOrElse(this, None).getOrElse(0.0)\n    if (dwellScore < query.params(ScoreThresholdForDwellParam)) {\n      // the default threshold is 0.0, dwellScore should be always non-negative\n      Some(0.0)\n    } else if (query.params(EnableBinarySchemeForDwellParam)) {\n      // If the binary scheme is enabled, we return a constant\n      Some(query.params(ScoreThresholdForDwellParam))\n    } else {\n      Some(dwellScore)\n    }\n  }\n}\n\nobject PredictedVideoWatchTimeScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_VIDEO_WATCH_TIME_MS.getFeatureName\n  override val statName = \"video_watch_time_ms\"\n  override val modelWeightParam = ModelWeights.VideoWatchTimeMsParam\n  override val weightQueryFeatureName = RecapFeatures.WEIGHT_VIDEO_WATCH_TIME_MS.getFeatureName\n  override val modelDebiasParam = Some(ModelDebiases.VideoWatchTimeMsParam)\n  override val debiasQueryFeatureName = Some(\n    RecapFeatures.DEBIAS_VIDEO_WATCH_TIME_MS.getFeatureName)\n}\n\n// Negative Engagements\nobject PredictedNegativeFeedbackV2ScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_NEGATIVE_FEEDBACK_V2.getFeatureName\n  override val statName = \"negative_feedback_v2\"\n  override val modelWeightParam = ModelWeights.NegativeFeedbackV2Param\n  override val weightQueryFeatureName = RecapFeatures.WEIGHT_IS_NEGATIVE_FEEDBACK_V2.getFeatureName\n  override val modelDebiasParam = Some(ModelDebiases.NegativeFeedbackV2Param)\n  override val debiasQueryFeatureName = Some(\n    RecapFeatures.DEBIAS_IS_NEGATIVE_FEEDBACK_V2.getFeatureName)\n}\n\nobject PredictedVideoQualityWatchScoreFeature extends PredictedScoreFeature {\n  override def isEligible(\n    features: FeatureMap,\n    query: PipelineQuery\n  ) = {\n    features.getOrElse(HasVideoFeature, false) && (features\n      .getOrElse(VideoDurationMsFeature, None).getOrElse(0) / 1000.0) >= 10\n  }\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_VIDEO_QUALITY_WATCH.getFeatureName\n  override val statName = \"video_quality_watched\"\n  override val modelWeightParam = ModelWeights.VideoQualityWatchParam\n  override val weightQueryFeatureName = RecapFeatures.WEIGHT_IS_VIDEO_QUALITY_WATCHED.getFeatureName\n  override val modelBiasParam = Some(ModelBiases.VideoQualityWatchParam)\n  override val biasQueryFeatureName = Some(\n    RecapFeatures.BIAS_IS_VIDEO_QUALITY_WATCHED.getFeatureName)\n  override val modelDebiasParam = Some(ModelDebiases.VideoQualityWatchParam)\n  override val debiasQueryFeatureName = Some(\n    RecapFeatures.DEBIAS_IS_VIDEO_QUALITY_WATCHED.getFeatureName)\n}\n\nobject PredictedScoreFeature {\n  val PredictedScoreFeatures: Seq[PredictedScoreFeature] = Seq(\n    PredictedFavoriteScoreFeature,\n    PredictedReplyScoreFeature,\n    PredictedRetweetScoreFeature,\n    PredictedReplyEngagedByAuthorScoreFeature,\n    PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature,\n    PredictedGoodClickConvoDescUamGt2ScoreFeature,\n    PredictedGoodProfileClickScoreFeature,\n    PredictedVideoQualityViewScoreFeature,\n    PredictedVideoQualityViewImmersiveScoreFeature,\n    PredictedBookmarkScoreFeature,\n    PredictedShareScoreFeature,\n    PredictedDwellScoreFeature,\n    PredictedVideoQualityWatchScoreFeature,\n    PredictedVideoWatchTimeScoreFeature,\n    // Negative Engagements\n    PredictedNegativeFeedbackV2ScoreFeature,\n  )\n\n  val PredictedScoreFeatureSet: Set[PredictedScoreFeature] = PredictedScoreFeatures.toSet\n\n  def getGeneralTensorFeatureFromName(\n    name: String\n  ): Feature[PipelineQuery, Option[GeneralTensor]] = {\n    new DataRecordOptionalFeature[PipelineQuery, GeneralTensor]\n      with GeneralTensorDataRecordCompatible {\n      override val featureName: String = name\n      override val personalDataTypes: Set[PersonalDataType] = Set.empty\n      override def toString: String = name\n      override def dataType: DataType = DataType.DOUBLE\n    }\n  }\n\n  def getDataRecordFeatureFromName(\n    name: String\n  ): Feature[PipelineQuery, Option[Double]] = {\n    new DataRecordOptionalFeature[PipelineQuery, Double] with DoubleDataRecordCompatible {\n      override val featureName: String = name\n      override val personalDataTypes: Set[PersonalDataType] = Set.empty\n      override def toString: String = name\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/candidate_source/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\"user-signal-service/thrift/src/main/thrift:thrift-scala\"],\n    exports = [],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/candidate_source/SourceSignal.scala",
    "content": "package com.twitter.home_mixer.model.candidate_source\n\nimport com.twitter.usersignalservice.{thriftscala => se}\n\ncase class SourceSignal(\n  id: Long,\n  signalType: Option[String],\n  signalEntity: Option[se.SignalEntity],\n  authorId: Option[Long])\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/signup\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"timelineservice/common:model\",\n    ],\n    exports = [\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request/DeviceContext.scala",
    "content": "package com.twitter.home_mixer.model.request\n\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.{timelineservice => tls}\n\ncase class DeviceContext(\n  isPolling: Option[Boolean],\n  requestContext: Option[String],\n  latestControlAvailable: Option[Boolean],\n  autoplayEnabled: Option[Boolean]) {\n\n  lazy val requestContextValue: Option[DeviceContext.RequestContext.Value] =\n    requestContext.flatMap { value =>\n      val normalizedValue = value.trim.toLowerCase()\n      DeviceContext.RequestContext.values.find(_.toString == normalizedValue)\n    }\n\n  def toTimelineServiceDeviceContext(clientContext: ClientContext): tls.DeviceContext =\n    tls.DeviceContext(\n      countryCode = clientContext.countryCode,\n      languageCode = clientContext.languageCode,\n      clientAppId = clientContext.appId,\n      ipAddress = clientContext.ipAddress,\n      guestId = clientContext.guestId,\n      sessionId = None,\n      timezone = None,\n      userAgent = clientContext.userAgent,\n      deviceId = clientContext.deviceId,\n      isPolling = isPolling,\n      requestProvenance = requestContext,\n      referrer = None,\n      tfeAuthHeader = None,\n      mobileDeviceId = clientContext.mobileDeviceId,\n      isSessionStart = None,\n      displaySize = None,\n      isURTRequest = Some(true),\n      latestControlAvailable = latestControlAvailable,\n      guestIdMarketing = clientContext.guestIdMarketing,\n      isInternalOrTwoffice = clientContext.isTwoffice,\n      browserNotificationPermission = None,\n      guestIdAds = clientContext.guestIdAds,\n    )\n}\n\nobject DeviceContext {\n  val Empty: DeviceContext = DeviceContext(\n    isPolling = None,\n    requestContext = None,\n    latestControlAvailable = None,\n    autoplayEnabled = None\n  )\n\n  /**\n   * Constants which reflect valid client request provenances (why a request was initiated, encoded\n   * by the \"request_context\" HTTP parameter).\n   */\n  object RequestContext extends Enumeration {\n    val Auto = Value(\"auto\")\n    val Foreground = Value(\"foreground\")\n    val Gap = Value(\"gap\")\n    val Launch = Value(\"launch\")\n    val ManualRefresh = Value(\"manual_refresh\")\n    val Navigate = Value(\"navigate\")\n    val Polling = Value(\"polling\")\n    val PullToRefresh = Value(\"ptr\")\n    val Signup = Value(\"signup\")\n    val TweetSelfThread = Value(\"tweet_self_thread\")\n    val BackgroundFetch = Value(\"background_fetch\")\n  }\n}\n\ntrait HasDeviceContext {\n  def deviceContext: Option[DeviceContext]\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request/HasListId.scala",
    "content": "package com.twitter.home_mixer.model.request\n\n/**\n * [[HasListId]] enables shared components to access the list id shared by all list timeline products.\n */\ntrait HasListId {\n  def listId: Long\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request/HasSeenTweetIds.scala",
    "content": "package com.twitter.home_mixer.model.request\n\n/**\n * [[HasSeenTweetIds]] enables shared components to access the list of impressed tweet IDs\n * sent by clients across different Home Mixer query types (e.g. FollowingQuery, ForYouQuery)\n */\ntrait HasSeenTweetIds {\n  def seenTweetIds: Option[Seq[Long]]\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request/HomeMixerDebugOptions.scala",
    "content": "package com.twitter.home_mixer.model.request\n\nimport com.twitter.product_mixer.core.model.marshalling.request.DebugOptions\nimport com.twitter.util.Time\n\ncase class HomeMixerDebugOptions(\n  override val requestTimeOverride: Option[Time],\n  override val showIntermediateLogs: Option[Boolean])\n    extends DebugOptions\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request/HomeMixerProduct.scala",
    "content": "package com.twitter.home_mixer.model.request\n\nimport com.twitter.product_mixer.core.model.common.identifier.ProductIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\n\n/**\n * Identifier names on products can be used to create Feature Switch rules by product,\n * which useful if bucketing occurs in a component shared by multiple products.\n * @see [[Product.identifier]]\n */\n\ncase object FollowingProduct extends Product {\n  override val identifier: ProductIdentifier = ProductIdentifier(\"Following\")\n  override val stringCenterProject: Option[String] = Some(\"timelinemixer\")\n}\n\ncase object ForYouProduct extends Product {\n  override val identifier: ProductIdentifier = ProductIdentifier(\"ForYou\")\n  override val stringCenterProject: Option[String] = Some(\"timelinemixer\")\n}\n\ncase object ScoredTweetsProduct extends Product {\n  override val identifier: ProductIdentifier = ProductIdentifier(\"ScoredTweets\")\n  override val stringCenterProject: Option[String] = Some(\"timelinemixer\")\n}\n\ncase object ScoredVideoTweetsProduct extends Product {\n  override val identifier: ProductIdentifier = ProductIdentifier(\"ScoredVideoTweets\")\n  override val stringCenterProject: Option[String] = Some(\"timelinemixer\")\n}\n\ncase object SubscribedProduct extends Product {\n  override val identifier: ProductIdentifier = ProductIdentifier(\"Subscribed\")\n  override val stringCenterProject: Option[String] = Some(\"timelinemixer\")\n}\n\ncase object HeavyRankerScoresProduct extends Product {\n  override val identifier: ProductIdentifier = ProductIdentifier(\"HeavyRankerScores\")\n  override val stringCenterProject: Option[String] = Some(\"timelinemixer\")\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request/HomeMixerProductContext.scala",
    "content": "package com.twitter.home_mixer.model.request\n\nimport com.twitter.dspbidder.commons.thriftscala.DspClientContext\nimport com.twitter.home_mixer.model.signup.SignupSource\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.core.model.marshalling.request.ProductContext\n\ncase class FollowingProductContext(\n  deviceContext: Option[DeviceContext],\n  seenTweetIds: Option[Seq[Long]],\n  dspClientContext: Option[DspClientContext])\n    extends ProductContext\n\ncase class ForYouProductContext(\n  deviceContext: Option[DeviceContext],\n  seenTweetIds: Option[Seq[Long]],\n  dspClientContext: Option[DspClientContext])\n    extends ProductContext\n\ncase class ScoredTweetsProductContext(\n  deviceContext: Option[DeviceContext],\n  seenTweetIds: Option[Seq[Long]],\n  servedTweetIds: Option[Seq[Long]],\n  backfillTweetIds: Option[Seq[Long]],\n  signupCountryCode: Option[String],\n  allowForYouRecommendations: Option[Boolean],\n  signupSource: Option[SignupSource],\n  followerCount: Option[Int],\n  servedAuthorIds: Option[Map[Long, Seq[Long]]] = None)\n    extends ProductContext\n\ncase class ScoredVideoTweetsProductContext(\n  deviceContext: Option[DeviceContext],\n  seenTweetIds: Option[Seq[Long]],\n  videoType: Option[t.VideoType],\n  pinnedRelatedTweetIds: Option[Seq[Long]],\n  scorePinnedTweetsOnly: Option[Boolean],\n  immersiveClientMetadata: Option[t.ImmersiveClientMetadata])\n    extends ProductContext\n\ncase class SubscribedProductContext(\n  deviceContext: Option[DeviceContext],\n  seenTweetIds: Option[Seq[Long]])\n    extends ProductContext\n\ncase class HeavyRankerScoresProductContext(\n  deviceContext: Option[DeviceContext],\n  tweetIds: Option[Seq[Long]])\n    extends ProductContext\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request/HomeMixerRequest.scala",
    "content": "package com.twitter.home_mixer.model.request\n\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.product_mixer.core.model.marshalling.request.DebugParams\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport com.twitter.product_mixer.core.model.marshalling.request.ProductContext\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\n\ncase class HomeMixerRequest(\n  override val clientContext: ClientContext,\n  override val product: Product,\n  // Product-specific parameters should be placed in the Product Context\n  override val productContext: Option[ProductContext],\n  override val serializedRequestCursor: Option[String],\n  override val maxResults: Option[Int],\n  override val debugParams: Option[DebugParams],\n  // Parameters that apply to all products can be promoted to the request-level\n  homeRequestParam: Boolean)\n    extends Request\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/signup/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/model/signup/SignupSource.scala",
    "content": "package com.twitter.home_mixer.model.signup\n\nsealed trait SignupSource\n\ncase object Onboard extends SignupSource\ncase object MarchMadness extends SignupSource\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/AdvertiserBrandSafetySettingsStoreModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\n\nimport com.twitter.adserver.{thriftscala => ads}\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storage.client.manhattan.kv.Guarantee\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.ManhattanCluster\nimport com.twitter.storehaus_internal.manhattan.ManhattanClusters\nimport com.twitter.timelines.clients.ads.AdvertiserBrandSafetySettingsStore\nimport com.twitter.timelines.clients.manhattan.mhv3.ManhattanClientBuilder\nimport com.twitter.timelines.clients.manhattan.mhv3.ManhattanClientConfigWithDataset\nimport com.twitter.util.Duration\n\nimport javax.inject.Singleton\n\nobject AdvertiserBrandSafetySettingsStoreModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def providesAdvertiserBrandSafetySettingsStore(\n    injectedServiceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): ReadableStore[Long, ads.AdvertiserBrandSafetySettings] = {\n    val advertiserBrandSafetySettingsManhattanClientConfig = new ManhattanClientConfigWithDataset {\n      override val cluster: ManhattanCluster = ManhattanClusters.apollo\n      override val appId: String = \"brand_safety_apollo\"\n      override val dataset = \"advertiser_brand_safety_settings\"\n      override val statsScope: String = \"AdvertiserBrandSafetySettingsManhattanClient\"\n      override val defaultGuarantee = Guarantee.Weak\n      override val defaultMaxTimeout: Duration = 100.milliseconds\n      override val maxRetryCount: Int = 1\n      override val isReadOnly: Boolean = true\n      override val serviceIdentifier: ServiceIdentifier = injectedServiceIdentifier\n    }\n\n    val advertiserBrandSafetySettingsManhattanEndpoint = ManhattanClientBuilder\n      .buildManhattanEndpoint(advertiserBrandSafetySettingsManhattanClientConfig, statsReceiver)\n\n    val advertiserBrandSafetySettingsStore: ReadableStore[Long, ads.AdvertiserBrandSafetySettings] =\n      AdvertiserBrandSafetySettingsStore\n        .cached(\n          advertiserBrandSafetySettingsManhattanEndpoint,\n          advertiserBrandSafetySettingsManhattanClientConfig.dataset,\n          ttl = 60.minutes,\n          maxKeys = 100000,\n          windowSize = 10L\n        )(statsReceiver)\n\n    advertiserBrandSafetySettingsStore\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/bijection:scrooge\",\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/io/grpc:grpc-netty\",\n        \"cproxy/thrift/src/main/thrift:thrift-scala\",\n        \"deferredrpc/client/src/main/scala\",\n        \"deferredrpc/client/src/main/thrift:thrift-scala\",\n        \"escherbird/src/scala/com/twitter/escherbird/util/uttclient\",\n        \"escherbird/src/thrift/com/twitter/escherbird/utt:strato-columns-scala\",\n        \"eventbus/client/src/main/scala/com/twitter/eventbus/client\",\n        \"events-recos/events-recos-service/src/main/thrift:events-recos-thrift-scala\",\n        \"finagle-internal/finagle-grpc/src/main/scala\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/interests:package\",\n        \"graph-feature-service/src/main/thrift/com/twitter/graph_feature_service:graph_feature_service_thrift-scala\",\n        \"home-mixer-features/thrift/src/main/thrift:thrift-java\",\n        \"home-mixer-features/thrift/src/main/thrift:thrift-scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/store\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/earlybird\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"interests-service/thrift/src/main/thrift:thrift-scala\",\n        \"kafka/finagle-kafka/finatra-kafka/src/main/scala\",\n        \"limiter/thrift-only/src/main/thrift:thrift-scala\",\n        \"media-understanding/video-summary/thrift/src/main/thrift:thrift-scala\",\n        \"people-discovery/api/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module\",\n        \"product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/manhattan_client\",\n        \"product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/memcached_client\",\n        \"product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/thrift_client\",\n        \"servo/manhattan\",\n        \"src/java/com/twitter/logpipeline/client:event-publisher-client-lib\",\n        \"src/java/com/twitter/logpipeline/client/common:event-publisher-core\",\n        \"src/java/com/twitter/logpipeline/client/serializers:event-publisher-serializers\",\n        \"src/scala/com/twitter/ml/featurestore/lib\",\n        \"src/scala/com/twitter/scalding_internal/multiformat/format\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/scala/com/twitter/storehaus_internal\",\n        \"src/scala/com/twitter/storehaus_internal/offline\",\n        \"src/scala/com/twitter/summingbird_internal/bijection:bijection-implicits\",\n        \"src/scala/com/twitter/summingbird_internal/runner/store_config\",\n        \"src/scala/com/twitter/timelines/util\",\n        \"src/scala/com/twitter/wtf/entity_real_graph/summingbird/client\",\n        \"src/scala/com/twitter/wtf/entity_real_graph/summingbird/common/config\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_rpc-scala\",\n        \"src/thrift/com/twitter/clientapp/gen:clientapp-scala\",\n        \"src/thrift/com/twitter/manhattan:v1-scala\",\n        \"src/thrift/com/twitter/manhattan:v2-scala\",\n        \"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"src/thrift/com/twitter/service/metastore/gen:thrift-scala\",\n        \"src/thrift/com/twitter/timelines/impression_store:thrift-scala\",\n        \"src/thrift/com/twitter/timelines/realtime_aggregates:thrift-scala\",\n        \"src/thrift/com/twitter/timelines/served_candidates_logging:served_candidates_logging-scala\",\n        \"src/thrift/com/twitter/timelines/suggests/common:poly_data_record-java\",\n        \"src/thrift/com/twitter/timelines/timeline_logging:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala\",\n        \"src/thrift/com/twitter/user_session_store:thrift-java\",\n        \"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala\",\n        \"stitch/stitch-tweetypie\",\n        \"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/manhattan\",\n        \"timelinemixer/common/src/main/scala/com/twitter/timelinemixer/clients/rankedtweetcaching\",\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/store/persistence\",\n        \"timelines/ml/cont_train/common/client/src/main/scala/com/twitter/timelines/ml/cont_train/common/client/scored_candidate_features_cache\",\n        \"timelines/src/main/scala/com/twitter/timelines/clients/ads\",\n        \"timelines/src/main/scala/com/twitter/timelines/clients/manhattan\",\n        \"timelines/src/main/scala/com/twitter/timelines/clients/memcache_common\",\n        \"timelines/src/main/scala/com/twitter/timelines/clients/predictionservice\",\n        \"timelines/src/main/scala/com/twitter/timelines/clients/strato/topics\",\n        \"timelines/src/main/scala/com/twitter/timelines/clients/strato/twistly\",\n        \"timelines/src/main/scala/com/twitter/timelines/config\",\n        \"timelines/src/main/scala/com/twitter/timelines/util/stats\",\n        \"timelineservice/common/src/main/scala/com/twitter/timelineservice/model\",\n        \"topic-social-proof/server/src/main/scala/com/twitter/tsp/stores\",\n        \"topiclisting/common/src/main/scala/com/twitter/topiclisting/clients/utt\",\n        \"topiclisting/topiclisting-core/src/main/scala/com/twitter/topiclisting\",\n        \"topiclisting/topiclisting-utt/src/main/scala/com/twitter/topiclisting/utt\",\n        \"tweetconvosvc/client/src/main/scala/com/twitter/tweetconvosvc/client/builder\",\n        \"user_history_transformer/protobuf/src/main/protobuf/com/x/user_action_sequence\",\n        \"user_history_transformer/service/src/main/java/com/x/user_action_sequence\",\n    ],\n    exports = [\n        \"timelines/src/main/scala/com/twitter/timelines/clients/predictionservice\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/BlenderClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.shared_library.thrift_client.FinagleThriftClientBuilder\nimport com.twitter.product_mixer.shared_library.thrift_client.NonIdempotent\nimport com.twitter.search.blender.thriftscala.BlenderService\nimport javax.inject.Singleton\n\nobject BlenderClientModule extends TwitterModule {\n\n  @Singleton\n  @Provides\n  def providesBlenderClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): BlenderService.MethodPerEndpoint = {\n    val clientId = serviceIdentifier.environment.toLowerCase match {\n      case \"prod\" => ClientId(\"\")\n      case _ => ClientId(\"\")\n    }\n\n    FinagleThriftClientBuilder.buildFinagleMethodPerEndpoint[\n      BlenderService.ServicePerEndpoint,\n      BlenderService.MethodPerEndpoint\n    ](\n      serviceIdentifier = serviceIdentifier,\n      clientId = clientId,\n      dest = \"/s/blender-universal/blender\",\n      label = \"blender\",\n      statsReceiver = statsReceiver,\n      idempotency = NonIdempotent,\n      timeoutPerRequest = 1000.milliseconds,\n      timeoutTotal = 1000.milliseconds,\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/ClientSentImpressionsPublisherModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.eventbus.client.EventBusPublisher\nimport com.twitter.eventbus.client.EventBusPublisherBuilder\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.timelines.config.ConfigUtils\nimport com.twitter.timelines.config.Env\nimport com.twitter.timelines.impressionstore.thriftscala.PublishedImpressionList\nimport javax.inject.Singleton\n\nobject ClientSentImpressionsPublisherModule extends TwitterModule with ConfigUtils {\n  private val serviceName = \"home-mixer\"\n\n  @Singleton\n  @Provides\n  def providesClientSentImpressionsPublisher(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): EventBusPublisher[PublishedImpressionList] = {\n    val env = serviceIdentifier.environment.toLowerCase match {\n      case \"prod\" => Env.prod\n      case \"staging\" => Env.staging\n      case \"local\" => Env.local\n      case _ => Env.devel\n    }\n\n    val streamName = env match {\n      case Env.prod => \"timelinemixer_client_sent_impressions_prod\"\n      case _ => \"timelinemixer_client_sent_impressions_devel\"\n    }\n\n    EventBusPublisherBuilder()\n      .clientId(clientIdWithScopeOpt(serviceName, env))\n      .serviceIdentifier(serviceIdentifier)\n      .streamName(streamName)\n      .statsReceiver(statsReceiver.scope(\"eventbus\"))\n      .thriftStruct(PublishedImpressionList)\n      .tcpConnectTimeout(20.milliseconds)\n      .connectTimeout(100.milliseconds)\n      .requestTimeout(1.second)\n      .publishTimeout(1.second)\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/ClusterDetailsModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.bijection.Bufferable\nimport com.twitter.bijection.Injection\nimport com.twitter.bijection.scrooge.CompactScalaCodec\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.thriftscala.ClusterDetails\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.Athena\nimport com.twitter.storehaus_internal.manhattan.ManhattanRO\nimport com.twitter.storehaus_internal.manhattan.ManhattanROConfig\nimport com.twitter.storehaus_internal.util.ApplicationID\nimport com.twitter.storehaus_internal.util.DatasetName\nimport com.twitter.storehaus_internal.util.HDFSPath\nimport javax.inject.Singleton\n\nobject ClusterDetailsModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def providesClusterDetailsStore(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[String, ClusterDetails] = {\n    implicit val keyInjection: Injection[(String, Int), Array[Byte]] =\n      Bufferable.injectionOf[(String, Int)]\n    implicit val valueInjection: Injection[ClusterDetails, Array[Byte]] =\n      CompactScalaCodec(ClusterDetails)\n\n    val modelName = \"20M_145K_2020\"\n\n    ManhattanRO\n      .getReadableStoreWithMtls[(String, Int), ClusterDetails](\n        ManhattanROConfig(\n          HDFSPath(\"\"),\n          ApplicationID(\"simclusters_v2\"),\n          DatasetName(\"simclusters_v2_cluster_details_20m_145k_2020\"),\n          Athena\n        ),\n        ManhattanKVClientMtlsParams(serviceIdentifier)\n      ).composeKeyMapping(clusterIdString => (modelName, clusterIdString.toInt))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/ConversationServiceModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.tweetconvosvc.thriftscala.ConversationService\nimport com.twitter.util.Duration\nimport org.apache.thrift.protocol.TCompactProtocol\n\nobject ConversationServiceModule\n    extends ThriftMethodBuilderClientModule[\n      ConversationService.ServicePerEndpoint,\n      ConversationService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label: String = \"tweetconvosvc\"\n  override val dest: String = \"/s/tweetconvosvc/tweetconvosvc\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = methodBuilder.withTimeoutPerRequest(100.milliseconds)\n\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client =\n    super\n      .configureThriftMuxClient(injector, client)\n      .withProtocolFactory(new TCompactProtocol.Factory())\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/EarlybirdRealtimeCGModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.EarlybirdRealtimCGEndpoint\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.shared_library.thrift_client.FinagleThriftClientBuilder\nimport com.twitter.product_mixer.shared_library.thrift_client.Idempotent\nimport com.twitter.search.earlybird.{thriftscala => t}\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport org.apache.thrift.protocol.TCompactProtocol\n\nobject EarlybirdRealtimeCGModule extends TwitterModule {\n\n  val Label: String = \"earlybird-rootrealtimecg\"\n  val Dest: String = \"/s/earlybird-rootrealtimecg/root-realtime_cg\"\n\n  @Provides\n  @Singleton\n  @Named(EarlybirdRealtimCGEndpoint)\n  def providesEarlybirdRealtimeCGService(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    statsReceiver: StatsReceiver\n  ): t.EarlybirdService.MethodPerEndpoint = {\n\n    FinagleThriftClientBuilder.buildFinagleMethodPerEndpoint[\n      t.EarlybirdService.ServicePerEndpoint,\n      t.EarlybirdService.MethodPerEndpoint\n    ](\n      serviceIdentifier = serviceIdentifier,\n      clientId = clientId,\n      dest = Dest,\n      label = Label,\n      statsReceiver = statsReceiver,\n      protocolFactoryOverride = Some(new TCompactProtocol.Factory),\n      idempotency = Idempotent(1.percent),\n      timeoutPerRequest = 600.milliseconds,\n      timeoutTotal = 650.milliseconds,\n      acquisitionTimeout = 1.seconds\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/EventsRecosClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.events.recos.thriftscala.EventsRecosService\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.util.Duration\n\nobject EventsRecosClientModule\n    extends ThriftMethodBuilderClientModule[\n      EventsRecosService.ServicePerEndpoint,\n      EventsRecosService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label: String = \"events-recos\"\n  override val dest: String = \"/s/events-recos/events-recos-service\"\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(450.millis)\n      .withTimeoutTotal(450.millis)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/FeedbackHistoryClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.timelinemixer.clients.feedback.FeedbackHistoryManhattanClient\nimport com.twitter.timelinemixer.clients.feedback.FeedbackHistoryManhattanClientConfig\nimport com.twitter.timelines.clients.manhattan.mhv3.ManhattanClientBuilder\nimport com.twitter.util.Duration\nimport javax.inject.Singleton\n\nobject FeedbackHistoryClientModule extends TwitterModule {\n  private val ProdDataset = \"feedback_history\"\n  private val StagingDataset = \"feedback_history_nonprod\"\n  private final val Timeout = \"mh_feedback_history.timeout\"\n\n  flag[Duration](Timeout, 150.millis, \"Timeout per request\")\n\n  @Provides\n  @Singleton\n  def providesFeedbackHistoryClient(\n    @Flag(Timeout) timeout: Duration,\n    serviceId: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ) = {\n    val manhattanDataset = serviceId.environment.toLowerCase match {\n      case \"prod\" => ProdDataset\n      case _ => StagingDataset\n    }\n\n    val config = new FeedbackHistoryManhattanClientConfig {\n      val dataset = manhattanDataset\n      val isReadOnly = true\n      val serviceIdentifier = serviceId\n      override val defaultMaxTimeout = timeout\n    }\n\n    new FeedbackHistoryManhattanClient(\n      ManhattanClientBuilder.buildManhattanEndpoint(config, statsReceiver),\n      manhattanDataset,\n      statsReceiver\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/GizmoduckTimelinesCacheClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps.RichDuration\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.gizmoduck.{thriftscala => gt}\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.GizmoduckTimelinesCache\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.shared_library.memcached_client.MemcachedClientBuilder\nimport com.twitter.servo.cache.FinagleMemcache\nimport com.twitter.servo.cache.KeyValueTransformingTtlCache\nimport com.twitter.servo.cache.ObservableTtlCache\nimport com.twitter.servo.cache.Serializer\nimport com.twitter.servo.cache.ThriftSerializer\nimport com.twitter.servo.cache.TtlCache\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport org.apache.thrift.protocol.TCompactProtocol\nimport com.twitter.finagle.memcached.compressing.scheme.Lz4\n\nobject GizmoduckTimelinesCacheClientModule extends TwitterModule {\n\n  private val ScopeName = \"GizmoduckTimelinesCache\"\n  private val ProdDest = \"/srv#/prod/local/cache/timelines_gizmoduck_secure:twemcaches\"\n\n  private val userSerializer: Serializer[gt.User] = {\n    new ThriftSerializer[gt.User](gt.User, new TCompactProtocol.Factory())\n  }\n\n  @Provides\n  @Singleton\n  @Named(GizmoduckTimelinesCache)\n  def providesGizmoduckTimelinesCache(\n    statsReceiver: StatsReceiver,\n    serviceIdentifier: ServiceIdentifier\n  ): TtlCache[Long, gt.User] = {\n    val memCacheClient = MemcachedClientBuilder.buildMemcachedClient(\n      destName = ProdDest,\n      numTries = 1,\n      numConnections = 1,\n      requestTimeout = 100.milliseconds,\n      globalTimeout = 100.milliseconds,\n      connectTimeout = 100.milliseconds,\n      acquisitionTimeout = 100.milliseconds,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver,\n      compressionScheme = Lz4\n    )\n    mkCache(new FinagleMemcache(memCacheClient), statsReceiver)\n  }\n\n  private def mkCache(\n    finagleMemcache: FinagleMemcache,\n    statsReceiver: StatsReceiver\n  ): TtlCache[Long, gt.User] = {\n    val baseCache: KeyValueTransformingTtlCache[Long, String, gt.User, Array[Byte]] =\n      new KeyValueTransformingTtlCache(\n        underlyingCache = finagleMemcache,\n        transformer = userSerializer,\n        underlyingKey = { key: Long => key.toString }\n      )\n    ObservableTtlCache(\n      underlyingCache = baseCache,\n      statsReceiver = statsReceiver.scope(ScopeName),\n      windowSize = 1000,\n      name = ScopeName\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/HomeAdsCandidateSourceModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.twitter.adserver.thriftscala.NewAdServer\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.util.Duration\n\nobject HomeAdsCandidateSourceModule\n    extends ThriftMethodBuilderClientModule[\n      NewAdServer.ServicePerEndpoint,\n      NewAdServer.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"adserver\"\n  override val dest = \"/s/ads/adserver\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(1200.milliseconds)\n      .withTimeoutTotal(1200.milliseconds)\n      .withMaxRetries(2)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 150.milliseconds\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/HomeMixerFeaturesModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient.MtlsThriftMuxClientSyntax\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.thrift.RichClientParam\nimport com.twitter.inject.TwitterModule\nimport com.twitter.home_mixer_features.{thriftjava => tj}\nimport com.twitter.home_mixer_features.{thriftscala => ts}\nimport com.twitter.product_mixer.shared_library.thrift_client.FinagleThriftClientBuilder\nimport com.twitter.product_mixer.shared_library.thrift_client.Idempotent\nimport javax.inject.Singleton\n\nobject HomeMixerFeaturesModule extends TwitterModule {\n\n  val Label: String = \"home-mixer-features\"\n  val Dest: String = \"/s/home-mixer/home-mixer-features\"\n\n  @Provides\n  @Singleton\n  def providesHomeMixerFeaturesService(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    statsReceiver: StatsReceiver,\n  ): tj.HomeMixerFeatures.ServiceToClient = {\n    buildClient(serviceIdentifier, clientId, statsReceiver, Dest, Label)\n  }\n\n  @Provides\n  @Singleton\n  def providesHomeMixerFeaturesScalaService(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    statsReceiver: StatsReceiver,\n  ): ts.HomeMixerFeatures.MethodPerEndpoint = {\n    FinagleThriftClientBuilder.buildFinagleMethodPerEndpoint[\n      ts.HomeMixerFeatures.ServicePerEndpoint,\n      ts.HomeMixerFeatures.MethodPerEndpoint\n    ](\n      serviceIdentifier = serviceIdentifier,\n      clientId = clientId,\n      dest = Dest,\n      label = Label,\n      statsReceiver = statsReceiver,\n      idempotency = Idempotent(1.percent),\n      timeoutPerRequest = 300.milliseconds,\n      timeoutTotal = 300.milliseconds,\n      acquisitionTimeout = 1.seconds\n    )\n  }\n\n  private def buildClient(\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    statsReceiver: StatsReceiver,\n    dest: String,\n    label: String\n  ): tj.HomeMixerFeatures.ServiceToClient = {\n    val stats = statsReceiver.scope(\"clnt\")\n    val thriftClient = ThriftMux.client\n      .withMutualTls(serviceIdentifier)\n      .withClientId(clientId)\n      .withLabel(label)\n      .withStatsReceiver(stats)\n      .withRequestTimeout(300.milliseconds)\n      .withSession.acquisitionTimeout(1.second)\n      .methodBuilder(dest)\n      .withTimeoutTotal(300.milliseconds)\n      .idempotent(1.percent)\n      .newService\n\n    new tj.HomeMixerFeatures.ServiceToClient(\n      thriftClient,\n      RichClientParam()\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/HomeMixerFlagsModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.twitter.conversions.DurationOps.RichDuration\nimport com.twitter.home_mixer.param.HomeMixerFlagName\nimport com.twitter.inject.TwitterModule\nimport com.twitter.util.Duration\n\nobject HomeMixerFlagsModule extends TwitterModule {\n\n  import HomeMixerFlagName._\n\n  flag[Boolean](\n    name = ScribeClientEventsFlag,\n    default = false,\n    help = \"Toggles logging client events to Scribe\"\n  )\n\n  flag[Boolean](\n    name = ScribeServedCandidatesFlag,\n    default = false,\n    help = \"Toggles logging served candidates to Scribe\"\n  )\n\n  flag[Boolean](\n    name = ScribeScoredCandidatesFlag,\n    default = false,\n    help = \"Toggles logging scored candidates to Scribe\"\n  )\n\n  flag[Boolean](\n    name = ScribeFeaturesFlag,\n    default = false,\n    help = \"Toggles logging served common features and candidates features to Scribe\"\n  )\n\n  flag[String](\n    name = DataRecordMetadataStoreConfigsYmlFlag,\n    default = \"mysql_timelines_ro_prod.yml\",\n    help = \"The YML file that contains the necessary info for creating metadata store MySQL client.\"\n  )\n\n  flag[String](\n    name = DarkTrafficFilterDeciderKey,\n    default = \"dark_traffic_filter\",\n    help = \"Dark traffic filter decider key\"\n  )\n\n  flag[Duration](\n    TargetScoringLatency,\n    700.millis,\n    \"Target scoring latency for Quality Factor\"\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/HomeMixerResourcesModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.config.yaml.YamlMap\nimport com.twitter.inject.TwitterModule\nimport javax.inject.Singleton\n\ncase class ViralContentCreatorsConfig(creators: Set[Long])\ncase class SupportAccountsConfig(accounts: Set[Long])\n\nobject HomeMixerResourcesModule extends TwitterModule {\n\n  private val ConfigFilePath = \"/config/ids.yml\"\n  private val SupportAccountsKey = \"support_accounts\"\n  private val ViralContentCreatorsKey = \"viral_content_creators\"\n\n  private val yaml: YamlMap = YamlMap.load(ConfigFilePath)\n\n  @Singleton\n  @Provides\n  def providesViralContentCreatorsConfig: ViralContentCreatorsConfig = {\n    val contentCreators = yaml.longSeq(ViralContentCreatorsKey).toSet\n    ViralContentCreatorsConfig(contentCreators)\n  }\n\n  @Singleton\n  @Provides\n  def providesSupportAccountsConfig: SupportAccountsConfig = {\n    val supportAccounts = yaml.longSeq(SupportAccountsKey).toSet\n    SupportAccountsConfig(supportAccounts)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/ImpressionBloomFilterModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.storage.client.manhattan.kv.Guarantee\nimport com.twitter.storehaus_internal.manhattan.ManhattanClusters\nimport com.twitter.timelines.clients.manhattan.store._\nimport com.twitter.timelines.impressionbloomfilter.{thriftscala => blm}\nimport com.twitter.timelines.impressionstore.impressionbloomfilter.ImpressionBloomFilterManhattanKeyValueDescriptor\nimport com.twitter.util.Duration\nimport javax.inject.Singleton\n\nobject ImpressionBloomFilterModule extends TwitterModule {\n\n  private val ProdAppId = \"impression_bloom_filter_store\"\n  private val ProdDataset = \"impression_bloom_filter\"\n  private val StagingAppId = \"impression_bloom_filter_store_staging\"\n  private val StagingDataset = \"impression_bloom_filter_staging\"\n  private val ClientStatsScope = \"tweetBloomFilterImpressionManhattanClient\"\n  private val DefaultTTL = 7.days\n  private final val Timeout = \"mh_impression_store_bloom_filter.timeout\"\n\n  flag[Duration](Timeout, 150.millis, \"Timeout per request\")\n\n  @Provides\n  @Singleton\n  def providesImpressionBloomFilter(\n    @Flag(Timeout) timeout: Duration,\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): ManhattanStoreClient[blm.ImpressionBloomFilterKey, blm.ImpressionBloomFilterSeq] = {\n    val (appId, dataset) = serviceIdentifier.environment.toLowerCase match {\n      case \"prod\" => (ProdAppId, ProdDataset)\n      case _ => (StagingAppId, StagingDataset)\n    }\n\n    implicit val manhattanKeyValueDescriptor: ImpressionBloomFilterManhattanKeyValueDescriptor =\n      ImpressionBloomFilterManhattanKeyValueDescriptor(\n        dataset = dataset,\n        ttl = DefaultTTL\n      )\n\n    ManhattanStoreClientBuilder.buildManhattanClient(\n      serviceIdentifier = serviceIdentifier,\n      cluster = ManhattanClusters.nash,\n      appId = appId,\n      defaultMaxTimeout = timeout,\n      maxRetryCount = 2,\n      defaultGuarantee = Some(Guarantee.SoftDcReadMyWrites),\n      isReadOnly = false,\n      statsScope = ClientStatsScope,\n      statsReceiver = statsReceiver\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/InMemoryCacheModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.ImageClipClusterIdInMemCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.IsColdStartPostInMemCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MediaClipClusterIdInMemCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MediaCompletionRateInMemCache\nimport com.twitter.inject.TwitterModule\nimport com.twitter.servo.cache.ExpiringLruInProcessCache\nimport com.twitter.servo.cache.InProcessCache\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.util.Random\n\nobject InMemoryCacheModule extends TwitterModule {\n  @Singleton\n  @Provides\n  @Named(MediaClipClusterIdInMemCache)\n  def providesMediaClipClusterIdInMemCache(\n  ): InProcessCache[Long, Option[Option[Long]]] = {\n    val BaseTTL = 4\n    val TTL = (BaseTTL + Random.nextInt(3)).minutes\n    val cache: InProcessCache[Long, Option[Option[Long]]] =\n      new ExpiringLruInProcessCache(ttl = TTL, maximumSize = 500000)\n    cache\n  }\n\n  @Singleton\n  @Provides\n  @Named(ImageClipClusterIdInMemCache)\n  def providesImageClipClusterIdInMemCache(\n  ): InProcessCache[Long, Option[Option[Long]]] = {\n    val BaseTTL = 4\n    val TTL = (BaseTTL + Random.nextInt(3)).minutes\n    val cache: InProcessCache[Long, Option[Option[Long]]] =\n      new ExpiringLruInProcessCache(ttl = TTL, maximumSize = 500000)\n    cache\n  }\n\n  @Singleton\n  @Provides\n  @Named(MediaCompletionRateInMemCache)\n  def providesMediaCompletionRateInMemCache(\n  ): InProcessCache[Long, Double] = {\n    val BaseTTL = 20\n    val TTL = (BaseTTL + Random.nextInt(15)).minutes\n    val cache: InProcessCache[Long, Double] =\n      new ExpiringLruInProcessCache(ttl = TTL, maximumSize = 10000000)\n    cache\n  }\n\n  @Singleton\n  @Provides\n  @Named(IsColdStartPostInMemCache)\n  def providesIsColdStartPostInMemCache(\n  ): InProcessCache[Long, Boolean] = {\n    val BaseTTL = 4\n    val TTL = (BaseTTL + Random.nextInt(3)).minutes\n    val cache: InProcessCache[Long, Boolean] =\n      new ExpiringLruInProcessCache(ttl = TTL, maximumSize = 500000)\n    cache\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/InjectionHistoryClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.builder.ClientBuilder\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient._\nimport com.twitter.finagle.service.RetryPolicy\nimport com.twitter.finagle.ssl.OpportunisticTls\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.manhattan.v2.thriftscala.{ManhattanCoordinator => ManhattanV2}\nimport com.twitter.timelinemixer.clients.manhattan.InjectionHistoryClient\nimport com.twitter.timelinemixer.clients.manhattan.ManhattanDatasetConfig\nimport com.twitter.timelines.clients.manhattan.Dataset\nimport com.twitter.timelines.clients.manhattan.ManhattanClient\nimport com.twitter.timelines.util.stats.RequestScope\nimport javax.inject.Singleton\nimport org.apache.thrift.protocol.TBinaryProtocol\nimport com.twitter.timelines.config.TimelinesUnderlyingClientConfiguration.ConnectTimeout\nimport com.twitter.timelines.config.TimelinesUnderlyingClientConfiguration.TCPConnectTimeout\n\nobject InjectionHistoryClientModule extends TwitterModule {\n  private val ProdDataset = \"suggestion_history\"\n  private val StagingDataset = \"suggestion_history_nonprod\"\n  private val AppId = \"twitter_suggests\"\n  private val ServiceName = \"manhattan.omega\"\n  private val OmegaManhattanDest = \"/s/manhattan/omega.native-thrift\"\n  private val InjectionRequestScope = RequestScope(\"injectionHistoryClient\")\n  private val RequestTimeout = 75.millis\n  private val Timeout = 150.millis\n\n  val retryPolicy = RetryPolicy.tries(\n    2,\n    RetryPolicy.TimeoutAndWriteExceptionsOnly\n      .orElse(RetryPolicy.ChannelClosedExceptionsOnly))\n\n  @Provides\n  @Singleton\n  def providesInjectionHistoryClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ) = {\n    val dataset = serviceIdentifier.environment.toLowerCase match {\n      case \"prod\" => ProdDataset\n      case _ => StagingDataset\n    }\n\n    val thriftMuxClient = ClientBuilder()\n      .name(ServiceName)\n      .daemon(daemonize = true)\n      .failFast(enabled = true)\n      .retryPolicy(retryPolicy)\n      .tcpConnectTimeout(TCPConnectTimeout)\n      .connectTimeout(ConnectTimeout)\n      .dest(OmegaManhattanDest)\n      .requestTimeout(RequestTimeout)\n      .timeout(Timeout)\n      .stack(ThriftMux.client\n        .withMutualTls(serviceIdentifier)\n        .withOpportunisticTls(OpportunisticTls.Required))\n      .build()\n\n    val manhattanOmegaClient = new ManhattanV2.FinagledClient(\n      service = thriftMuxClient,\n      protocolFactory = new TBinaryProtocol.Factory(),\n      serviceName = ServiceName,\n    )\n\n    val readOnlyMhClient = new ManhattanClient(\n      appId = AppId,\n      manhattan = manhattanOmegaClient,\n      requestScope = InjectionRequestScope,\n      serviceName = ServiceName,\n      statsReceiver = statsReceiver\n    ).readOnly\n\n    val mhDatasetConfig = new ManhattanDatasetConfig {\n      override val SuggestionHistoryDataset = Dataset(dataset)\n    }\n\n    new InjectionHistoryClient(\n      readOnlyMhClient,\n      mhDatasetConfig\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/LimiterModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.limiter.{thriftscala => t}\nimport com.twitter.util.Duration\n\nobject LimiterModule\n    extends ThriftMethodBuilderClientModule[\n      t.LimitService.ServicePerEndpoint,\n      t.LimitService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label: String = \"limiter\"\n  override val dest: String = \"/s/limiter/limiter\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = methodBuilder.withTimeoutPerRequest(200.milliseconds)\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/ManhattanClientsModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RTAManhattanEndpoint\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RTAManhattanStore\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RealGraphManhattanEndpoint\nimport com.twitter.home_mixer.store.RTAMHStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.storage.client.manhattan.kv._\nimport com.twitter.timelines.config.ConfigUtils\nimport com.twitter.util.Duration\nimport com.twitter.timelines.realtime_aggregates.{thriftscala => thrift}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.storehaus.ReadableStore\n\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject ManhattanClientsModule extends TwitterModule with ConfigUtils {\n\n  private val ApolloDest = \"/s/manhattan/apollo.native-thrift\"\n  private val BalterDest = \"/s/manhattan/baltar.native-thrift\"\n  private final val Timeout = \"mh_real_graph.timeout\"\n\n  flag[Duration](Timeout, 150.millis, \"Timeout total\")\n\n  @Provides\n  @Singleton\n  @Named(RealGraphManhattanEndpoint)\n  def providesRealGraphManhattanEndpoint(\n    @Flag(Timeout) timeout: Duration,\n    serviceIdentifier: ServiceIdentifier\n  ): ManhattanKVEndpoint = {\n    lazy val client = ManhattanKVClient(\n      appId = \"real_graph\",\n      dest = ApolloDest,\n      mtlsParams = ManhattanKVClientMtlsParams(serviceIdentifier = serviceIdentifier),\n      label = \"real-graph-data\"\n    )\n\n    ManhattanKVEndpointBuilder(client)\n      .maxRetryCount(2)\n      .defaultMaxTimeout(timeout)\n      .build()\n  }\n\n  @Provides\n  @Singleton\n  @Named(RTAManhattanEndpoint)\n  def providesRTAManhattanEndpoint(\n    @Flag(Timeout) timeout: Duration,\n    serviceIdentifier: ServiceIdentifier\n  ): ManhattanKVEndpoint = {\n    lazy val client = ManhattanKVClient(\n      appId = \"timelines_real_time_aggregates\",\n      dest = BalterDest,\n      mtlsParams = ManhattanKVClientMtlsParams(serviceIdentifier = serviceIdentifier),\n      label = \"rta-test\"\n    )\n\n    ManhattanKVEndpointBuilder(client)\n      .maxRetryCount(2)\n      .defaultMaxTimeout(timeout)\n      .build()\n  }\n\n  @Provides\n  @Singleton\n  @Named(RTAManhattanStore)\n  def providesRTAManhattanStore(\n    @Named(RTAManhattanEndpoint) manhattanKVEndpoint: ManhattanKVEndpoint\n  ): Option[ReadableStore[thrift.AggregationKey, DataRecord]] = {\n    Some(new RTAMHStore(manhattanKVEndpoint))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/ManhattanFeatureRepositoryModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.common.primitives.Longs\nimport com.google.inject.Provides\nimport com.twitter.bijection.Injection\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.bijection.scrooge.CompactScalaCodec\nimport com.twitter.bijection.thrift.ThriftCodec\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames._\nimport com.twitter.home_mixer.util.InjectionTransformerImplicits._\nimport com.twitter.home_mixer.util.LanguageUtil\nimport com.twitter.home_mixer.util.TensorFlowUtil\nimport com.twitter.inject.TwitterModule\nimport com.twitter.manhattan.v1.{thriftscala => mh}\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.ml.featurestore.{thriftscala => fs}\nimport com.twitter.product_mixer.shared_library.manhattan_client.ManhattanClientBuilder\nimport com.twitter.scalding_internal.multiformat.format.keyval.KeyValInjection.ScalaBinaryThrift\nimport com.twitter.search.common.constants.{thriftscala => scc}\nimport com.twitter.service.metastore.gen.{thriftscala => smg}\nimport com.twitter.servo.cache._\nimport com.twitter.servo.manhattan.ManhattanKeyValueRepository\nimport com.twitter.servo.repository.CachingKeyValueRepository\nimport com.twitter.servo.repository.ChunkingStrategy\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.servo.repository.Repository\nimport com.twitter.servo.repository.keysAsQuery\nimport com.twitter.servo.util.Transformer\nimport com.twitter.storage.client.manhattan.bijections.Bijections\nimport com.twitter.storehaus_internal.manhattan.ManhattanClusters\nimport com.twitter.timelines.author_features.v1.{thriftjava => af}\nimport com.twitter.timelines.suggests.common.dense_data_record.{thriftscala => ddr}\nimport com.twitter.user_session_store.{thriftjava => uss}\nimport com.twitter.user_session_store.{thriftscala => uss_scala}\nimport com.twitter.util.Duration\nimport com.twitter.util.Try\nimport java.nio.ByteBuffer\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport org.apache.thrift.protocol.TCompactProtocol\nimport org.apache.thrift.transport.TMemoryInputTransport\nimport org.apache.thrift.transport.TTransport\n\nobject ManhattanFeatureRepositoryModule extends TwitterModule {\n\n  private val DEFAULT_RPC_CHUNK_SIZE = 50\n\n  private val ThriftEntityIdInjection = ScalaBinaryThrift(fs.EntityId)\n\n  private val FeatureStoreUserIdKeyTransformer = new Transformer[Long, ByteBuffer] {\n    override def to(userId: Long): Try[ByteBuffer] = {\n      Try(ByteBuffer.wrap(ThriftEntityIdInjection.apply(UserId(userId).toThrift)))\n    }\n    override def from(b: ByteBuffer): Try[Long] = ???\n  }\n\n  private val LongUserIdKeyTransformer = new Transformer[Long, ByteBuffer] {\n    override def to(userId: Long): Try[ByteBuffer] = {\n      Try(ByteBuffer.wrap(Longs.toByteArray(userId)))\n    }\n    override def from(b: ByteBuffer): Try[Long] = ???\n  }\n\n  private val FloatTensorTransformer = new Transformer[ByteBuffer, ml.FloatTensor] {\n    override def to(input: ByteBuffer): Try[ml.FloatTensor] = {\n      val floatTensor = TensorFlowUtil.embeddingByteBufferToFloatTensor(input)\n      Try(floatTensor)\n    }\n    override def from(b: ml.FloatTensor): Try[ByteBuffer] = ???\n  }\n\n  private val EmbeddingTransformer = new Transformer[ByteBuffer, ml.FloatTensor] {\n    override def to(input: ByteBuffer): Try[ml.FloatTensor] = {\n      Try.fromScala(\n        Bijections\n          .BinaryScalaInjection(ml.Embedding).andThen(Bijections.byteBuffer2Buf.inverse).invert(\n            input).map { embedding =>\n            embedding.tensor.map { tensor =>\n              TensorFlowUtil.embeddingNoHeaderByteBufferToFloatTensor(tensor.content)\n            }.get\n          }\n      )\n    }\n\n    override def from(b: ml.FloatTensor): Try[ByteBuffer] = ???\n  }\n\n  private val LanguageTransformer = new Transformer[ByteBuffer, Seq[scc.ThriftLanguage]] {\n    override def to(input: ByteBuffer): Try[Seq[scc.ThriftLanguage]] = {\n      Try.fromScala(\n        Bijections\n          .BinaryScalaInjection(smg.UserLanguages)\n          .andThen(Bijections.byteBuffer2Buf.inverse)\n          .invert(input).map(LanguageUtil.computeLanguages(_)))\n    }\n\n    override def from(b: Seq[scc.ThriftLanguage]): Try[ByteBuffer] = ???\n  }\n\n  private val LongKeyTransformer = Injection\n    .connect[Long, Array[Byte]]\n    .toByteBufferTransformer()\n\n  // manhattan clients\n\n  @Provides\n  @Singleton\n  @Named(ManhattanApolloClient)\n  def providesManhattanApolloClient(\n    serviceIdentifier: ServiceIdentifier\n  ): mh.ManhattanCoordinator.MethodPerEndpoint = {\n    ManhattanClientBuilder\n      .buildManhattanV1FinagleClient(\n        ManhattanClusters.apollo,\n        serviceIdentifier\n      )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ManhattanAthenaClient)\n  def providesManhattanAthenaClient(\n    serviceIdentifier: ServiceIdentifier\n  ): mh.ManhattanCoordinator.MethodPerEndpoint = {\n    ManhattanClientBuilder\n      .buildManhattanV1FinagleClient(\n        ManhattanClusters.athena,\n        serviceIdentifier\n      )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ManhattanOmegaClient)\n  def providesManhattanOmegaClient(\n    serviceIdentifier: ServiceIdentifier\n  ): mh.ManhattanCoordinator.MethodPerEndpoint = {\n    ManhattanClientBuilder\n      .buildManhattanV1FinagleClient(\n        ManhattanClusters.omega,\n        serviceIdentifier\n      )\n  }\n\n  @Provides\n  @Singleton\n  @Named(ManhattanStarbuckClient)\n  def providesManhattanStarbuckClient(\n    serviceIdentifier: ServiceIdentifier\n  ): mh.ManhattanCoordinator.MethodPerEndpoint = {\n    ManhattanClientBuilder\n      .buildManhattanV1FinagleClient(\n        ManhattanClusters.starbuck,\n        serviceIdentifier\n      )\n  }\n\n  // non-cached manhattan repositories\n  /**\n   * A repository of the offline aggregate feature metadata necessary to decode\n   * DenseCompactDataRecords.\n   *\n   * This repository is expected to virtually always pick up the metadata form the local cache with\n   * nearly 0 latency.\n   */\n  @Provides\n  @Singleton\n  @Named(TimelineAggregateMetadataRepository)\n  def providesTimelineAggregateMetadataRepository(\n    @Named(ManhattanAthenaClient) client: mh.ManhattanCoordinator.MethodPerEndpoint\n  ): Repository[Int, Option[ddr.DenseFeatureMetadata]] = {\n\n    val keyTransformer = Injection\n      .connect[Int, Array[Byte]]\n      .toByteBufferTransformer()\n\n    val valueTransformer = new Transformer[ByteBuffer, ddr.DenseFeatureMetadata] {\n      private val compactProtocolFactory = new TCompactProtocol.Factory\n\n      def to(buffer: ByteBuffer): Try[ddr.DenseFeatureMetadata] = Try {\n        val transport = transportFromByteBuffer(buffer)\n        ddr.DenseFeatureMetadata.decode(compactProtocolFactory.getProtocol(transport))\n      }\n\n      // Encoding intentionally not implemented as it is never used\n      def from(metadata: ddr.DenseFeatureMetadata): Try[ByteBuffer] = ???\n    }\n\n    val inProcessCache: Cache[Int, Cached[ddr.DenseFeatureMetadata]] = InProcessLruCacheFactory(\n      ttl = Duration.fromMinutes(20),\n      lruSize = 30\n    ).apply(serializer = Transformer(_ => ???, _ => ???)) // Serialization is not necessary here.\n\n    val keyValueRepository = new ManhattanKeyValueRepository(\n      client = client,\n      keyTransformer = keyTransformer,\n      valueTransformer = valueTransformer,\n      appId = \"timelines_dense_aggregates_encoding_metadata\", // Expected QPS is negligible.\n      dataset = \"user_session_dense_feature_metadata\",\n      timeoutInMillis = 100\n    )\n\n    KeyValueRepository\n      .singular(\n        new CachingKeyValueRepository[Seq[Int], Int, ddr.DenseFeatureMetadata](\n          keyValueRepository,\n          new NonLockingCache(inProcessCache),\n          keysAsQuery[Int]\n        )\n      )\n  }\n\n  @Provides\n  @Singleton\n  @Named(RealGraphFeatureRepository)\n  def providesRealGraphFeatureRepository(\n    @Named(ManhattanApolloClient) client: mh.ManhattanCoordinator.MethodPerEndpoint\n  ): Repository[Long, Option[uss_scala.UserSession]] = {\n    val valueTransformer = CompactScalaCodec(uss_scala.UserSession).toByteBufferTransformer().flip\n\n    KeyValueRepository.singular(\n      new ManhattanKeyValueRepository(\n        client = client,\n        keyTransformer = LongKeyTransformer,\n        valueTransformer = valueTransformer,\n        appId = \"real_graph\",\n        dataset = \"real_graph_user_features\",\n        timeoutInMillis = 100,\n      )\n    )\n  }\n\n  // cached manhattan repositories\n\n  @Provides\n  @Singleton\n  @Named(AuthorFeatureRepository)\n  def providesAuthorFeatureRepository(\n    @Named(ManhattanAthenaClient) client: mh.ManhattanCoordinator.MethodPerEndpoint,\n    @Named(HomeAuthorFeaturesCacheClient) cacheClient: Memcache\n  ): KeyValueRepository[Seq[Long], Long, af.AuthorFeatures] = {\n\n    val valueInjection = ThriftCodec\n      .toCompact[af.AuthorFeatures]\n\n    val keyValueRepository = batchedManhattanKeyValueRepository(\n      client = client,\n      keyTransformer = LongKeyTransformer,\n      valueTransformer = valueInjection.toByteBufferTransformer().flip,\n      appId = \"timelines_author_feature_store_athena\",\n      dataset = \"timelines_author_features\",\n      timeoutInMillis = 100\n    )\n\n    val remoteCacheRepo = buildMemCachedRepository(\n      keyValueRepository = keyValueRepository,\n      cacheClient = cacheClient,\n      cachePrefix = \"AuthorFeatureHydrator\",\n      ttl = 12.hours,\n      valueInjection = valueInjection\n    )\n\n    remoteCacheRepo\n  }\n\n  @Provides\n  @Singleton\n  @Named(TwhinAuthorFollowFeatureRepository)\n  def providesTwhinAuthorFollowFeatureRepository(\n    @Named(ManhattanApolloClient) client: mh.ManhattanCoordinator.MethodPerEndpoint,\n    @Named(TwhinAuthorFollowFeatureCacheClient) cacheClient: Memcache\n  ): KeyValueRepository[Seq[Long], Long, ml.FloatTensor] = {\n    val keyValueRepository =\n      batchedManhattanKeyValueRepository(\n        client = client,\n        keyTransformer = FeatureStoreUserIdKeyTransformer,\n        valueTransformer = FloatTensorTransformer,\n        appId = \"ml_features_apollo\",\n        dataset = \"twhin_author_follow_embedding_fsv1__v1_thrift__embedding\",\n        timeoutInMillis = 100\n      )\n\n    val valueInjection: Injection[ml.FloatTensor, Array[Byte]] =\n      BinaryScalaCodec(ml.FloatTensor)\n\n    buildMemCachedRepository(\n      keyValueRepository = keyValueRepository,\n      cacheClient = cacheClient,\n      cachePrefix = \"twhinAuthorFollows\",\n      ttl = 24.hours,\n      valueInjection = valueInjection\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(UserLanguagesRepository)\n  def providesUserLanguagesFeatureRepository(\n    @Named(ManhattanStarbuckClient) client: mh.ManhattanCoordinator.MethodPerEndpoint\n  ): KeyValueRepository[Seq[Long], Long, Seq[scc.ThriftLanguage]] = {\n    batchedManhattanKeyValueRepository(\n      client = client,\n      keyTransformer = LongKeyTransformer,\n      valueTransformer = LanguageTransformer,\n      appId = \"user_metadata\",\n      dataset = \"languages\",\n      timeoutInMillis = 70\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(TwhinUserFollowFeatureRepository)\n  def providesTwhinUserFollowFeatureRepository(\n    @Named(ManhattanApolloClient) client: mh.ManhattanCoordinator.MethodPerEndpoint\n  ): KeyValueRepository[Seq[Long], Long, ml.FloatTensor] = {\n    batchedManhattanKeyValueRepository(\n      client = client,\n      keyTransformer = FeatureStoreUserIdKeyTransformer,\n      valueTransformer = FloatTensorTransformer,\n      appId = \"ml_features_apollo\",\n      dataset = \"twhin_user_follow_embedding_fsv1__v1_thrift__embedding\",\n      timeoutInMillis = 100\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(TimelineAggregatePartARepository)\n  def providesTimelineAggregatePartARepository(\n    @Named(ManhattanApolloClient) client: mh.ManhattanCoordinator.MethodPerEndpoint,\n  ): Repository[Long, Option[uss.UserSession]] =\n    timelineAggregateRepository(\n      mhClient = client,\n      mhDataset = \"timelines_aggregates_v2_features_by_user_part_a_apollo\",\n      mhAppId = \"timelines_aggregates_v2_features_by_user_part_a_apollo\"\n    )\n\n  @Provides\n  @Singleton\n  @Named(TimelineAggregatePartBRepository)\n  def providesTimelineAggregatePartBRepository(\n    @Named(ManhattanApolloClient) client: mh.ManhattanCoordinator.MethodPerEndpoint,\n  ): Repository[Long, Option[uss.UserSession]] =\n    timelineAggregateRepository(\n      mhClient = client,\n      mhDataset = \"timelines_aggregates_v2_features_by_user_part_b_apollo\",\n      mhAppId = \"timelines_aggregates_v2_features_by_user_part_b_apollo\"\n    )\n\n  @Provides\n  @Singleton\n  @Named(TwhinUserEngagementFeatureRepository)\n  def providesTwhinUserEngagementFeatureRepository(\n    @Named(ManhattanApolloClient) client: mh.ManhattanCoordinator.MethodPerEndpoint\n  ): KeyValueRepository[Seq[Long], Long, ml.FloatTensor] = {\n\n    batchedManhattanKeyValueRepository(\n      client = client,\n      keyTransformer = FeatureStoreUserIdKeyTransformer,\n      valueTransformer = FloatTensorTransformer,\n      appId = \"ml_features_apollo\",\n      dataset = \"twhin_user_engagement_embedding_fsv1__v1_thrift__embedding\",\n      timeoutInMillis = 100\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(TwhinRebuildUserEngagementFeatureRepository)\n  def providesTwhinRebuildUserEngagementFeatureRepository(\n    @Named(ManhattanApolloClient) client: mh.ManhattanCoordinator.MethodPerEndpoint\n  ): KeyValueRepository[Seq[Long], Long, ml.FloatTensor] = {\n\n    batchedManhattanKeyValueRepository(\n      client = client,\n      keyTransformer = LongUserIdKeyTransformer,\n      valueTransformer = EmbeddingTransformer,\n      appId = \"twhin_embeddings_apollo\",\n      dataset = \"twhin_refreshed_user_eng_emb\",\n      timeoutInMillis = 100\n    )\n  }\n\n  private def buildMemCachedRepository[K, V](\n    keyValueRepository: KeyValueRepository[Seq[K], K, V],\n    cacheClient: Memcache,\n    cachePrefix: String,\n    ttl: Duration,\n    valueInjection: Injection[V, Array[Byte]]\n  ): CachingKeyValueRepository[Seq[K], K, V] = {\n    val cachedSerializer = CachedSerializer.binary(\n      valueInjection.toByteArrayTransformer()\n    )\n\n    val cache = MemcacheCacheFactory(\n      cacheClient,\n      ttl,\n      PrefixKeyTransformerFactory(cachePrefix)\n    )[K, Cached[V]](cachedSerializer)\n\n    new CachingKeyValueRepository(\n      keyValueRepository,\n      new NonLockingCache(cache),\n      keysAsQuery[K]\n    )\n  }\n\n  private def buildInProcessCachedRepository[K, V](\n    keyValueRepository: KeyValueRepository[Seq[K], K, V],\n    ttl: Duration,\n    size: Int,\n    valueInjection: Injection[V, Array[Byte]]\n  ): CachingKeyValueRepository[Seq[K], K, V] = {\n    val cachedSerializer = CachedSerializer.binary(\n      valueInjection.toByteArrayTransformer()\n    )\n\n    val cache = InProcessLruCacheFactory(\n      ttl = ttl,\n      lruSize = size\n    )[K, Cached[V]](cachedSerializer)\n\n    new CachingKeyValueRepository(\n      keyValueRepository,\n      new NonLockingCache(cache),\n      keysAsQuery[K]\n    )\n  }\n\n  private def batchedManhattanKeyValueRepository[K, V](\n    client: mh.ManhattanCoordinator.MethodPerEndpoint,\n    keyTransformer: Transformer[K, ByteBuffer],\n    valueTransformer: Transformer[ByteBuffer, V],\n    appId: String,\n    dataset: String,\n    timeoutInMillis: Int,\n    chunkSize: Int = DEFAULT_RPC_CHUNK_SIZE\n  ): KeyValueRepository[Seq[K], K, V] =\n    KeyValueRepository.chunked(\n      new ManhattanKeyValueRepository(\n        client = client,\n        keyTransformer = keyTransformer,\n        valueTransformer = valueTransformer,\n        appId = appId,\n        dataset = dataset,\n        timeoutInMillis = timeoutInMillis\n      ),\n      chunker = ChunkingStrategy.equalSize(chunkSize)\n    )\n\n  private def transportFromByteBuffer(buffer: ByteBuffer): TTransport =\n    new TMemoryInputTransport(\n      buffer.array(),\n      buffer.arrayOffset() + buffer.position(),\n      buffer.remaining())\n\n  private def timelineAggregateRepository(\n    mhClient: mh.ManhattanCoordinator.MethodPerEndpoint,\n    mhDataset: String,\n    mhAppId: String\n  ): Repository[Long, Option[uss.UserSession]] = {\n    val valueInjection = ThriftCodec\n      .toCompact[uss.UserSession]\n\n    KeyValueRepository.singular(\n      new ManhattanKeyValueRepository(\n        client = mhClient,\n        keyTransformer = LongKeyTransformer,\n        valueTransformer = valueInjection.toByteBufferTransformer().flip,\n        appId = mhAppId,\n        dataset = mhDataset,\n        timeoutInMillis = 100\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/ManhattanTweetImpressionStoreModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.storage.client.manhattan.kv.Guarantee\nimport com.twitter.storehaus_internal.manhattan.ManhattanClusters\nimport com.twitter.timelines.clients.manhattan.mhv3.ManhattanClientBuilder\nimport com.twitter.timelines.impressionstore.store.ManhattanTweetImpressionStoreClientConfig\nimport com.twitter.timelines.impressionstore.store.ManhattanTweetImpressionStoreClient\nimport com.twitter.util.Duration\nimport javax.inject.Singleton\n\nobject ManhattanTweetImpressionStoreModule extends TwitterModule {\n\n  private val ProdAppId = \"timelines_tweet_impression_store_v2\"\n  private val ProdDataset = \"timelines_tweet_impressions_v2\"\n  private val StagingAppId = \"timelines_tweet_impression_store_staging\"\n  private val StagingDataset = \"timelines_tweet_impressions_staging\"\n  private val StatsScope = \"manhattanTweetImpressionStoreClient\"\n  private val DefaultTTL = 2.days\n  private final val Timeout = \"mh_impression_store.timeout\"\n\n  flag[Duration](Timeout, 150.millis, \"Timeout per request\")\n\n  @Provides\n  @Singleton\n  def providesManhattanTweetImpressionStoreClient(\n    @Flag(Timeout) timeout: Duration,\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): ManhattanTweetImpressionStoreClient = {\n\n    val (appId, dataset) = serviceIdentifier.environment.toLowerCase match {\n      case \"prod\" => (ProdAppId, ProdDataset)\n      case _ => (StagingAppId, StagingDataset)\n    }\n\n    val config = ManhattanTweetImpressionStoreClientConfig(\n      cluster = ManhattanClusters.nash,\n      appId = appId,\n      dataset = dataset,\n      statsScope = StatsScope,\n      defaultGuarantee = Guarantee.SoftDcReadMyWrites,\n      defaultMaxTimeout = timeout,\n      maxRetryCount = 2,\n      isReadOnly = false,\n      serviceIdentifier = serviceIdentifier,\n      ttl = DefaultTTL\n    )\n\n    val manhattanEndpoint = ManhattanClientBuilder.buildManhattanEndpoint(config, statsReceiver)\n    ManhattanTweetImpressionStoreClient(config, manhattanEndpoint, statsReceiver)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/MediaClusterId88Module.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MediaClusterId88Store\nimport com.twitter.home_mixer.store.MediaClusterId88Store\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject MediaClusterId88Module extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(MediaClusterId88Store)\n  def providesMediaClusterId88Store(\n    mediaClusterId88Store: MediaClusterId88Store\n  ): ReadableStore[Long, Long] = {\n    mediaClusterId88Store.clusterIdStore\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/MediaClusterIdModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MediaClusterId95Store\nimport com.twitter.home_mixer.store.MediaClusterId95Store\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject MediaClusterId95Module extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(MediaClusterId95Store)\n  def providesMediaClusterId95Store(\n    mediaClusterId95Store: MediaClusterId95Store\n  ): ReadableStore[Long, Long] = {\n    mediaClusterId95Store.clusterIdStore\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/MemcachedFeatureRepositoryModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.Memcached\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.EntityRealGraphClientStore\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.HomeAuthorFeaturesCacheClient\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RealTimeInteractionGraphUserVertexClient\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TimelinesRealTimeAggregateClient\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinAuthorFollowFeatureCacheClient\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.shared_library.memcached_client.MemcachedClientBuilder\nimport com.twitter.servo.cache.ExpiringLruInProcessCache\nimport com.twitter.servo.cache.FinagleMemcache\nimport com.twitter.servo.cache.FinagleMemcacheFactory\nimport com.twitter.servo.cache.HotKeyMemcacheClient\nimport com.twitter.servo.cache.Memcache\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.wtf.entity_real_graph.summingbird.client.EntityRealGraphClient\nimport com.twitter.wtf.entity_real_graph.summingbird.common.config.Configs.Environment\nimport com.twitter.wtf.entity_real_graph.{thriftscala => erg}\nimport com.twitter.finagle.memcached.compressing.scheme.Lz4\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TvRealTimeAggregateClient\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject MemcachedFeatureRepositoryModule extends TwitterModule {\n\n  // This must match the respective parameter on the write path. Note that servo sets a different\n  // hasher by default. See [[com.twitter.hashing.KeyHasher]] for the list of other available\n  // hashers.\n  private val memcacheKeyHasher = \"ketama\"\n\n  @Provides\n  @Singleton\n  @Named(TimelinesRealTimeAggregateClient)\n  def providesTimelinesRealTimeAggregateClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Memcache = {\n    val cacheClient = MemcachedClientBuilder.buildMemcachedClient(\n      destName = \"/s/cache/timelines_real_time_aggregates:twemcaches\",\n      numTries = 3,\n      numConnections = 1,\n      requestTimeout = 100.milliseconds,\n      globalTimeout = 300.milliseconds,\n      connectTimeout = 200.milliseconds,\n      acquisitionTimeout = 200.milliseconds,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver\n    )\n\n    val hotkeyCacheClient = new HotKeyMemcacheClient(\n      proxyClient = cacheClient,\n      inProcessCache = new ExpiringLruInProcessCache(ttl = 15.minute, maximumSize = 75000),\n      statsReceiver = statsReceiver.scope(TimelinesRealTimeAggregateClient).scope(\"inProcess\")\n    )\n\n    new FinagleMemcache(hotkeyCacheClient, memcacheKeyHasher)\n  }\n\n  @Provides\n  @Singleton\n  @Named(HomeAuthorFeaturesCacheClient)\n  def providesHomeAuthorFeaturesCacheClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Memcache = {\n    val cacheClient = MemcachedClientBuilder.buildRawMemcachedClient(\n      numTries = 2,\n      numConnections = 1,\n      requestTimeout = 150.milliseconds,\n      globalTimeout = 300.milliseconds,\n      connectTimeout = 200.milliseconds,\n      acquisitionTimeout = 200.milliseconds,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver,\n      compressionScheme = Lz4\n    )\n\n    buildMemcacheClient(cacheClient, \"/s/cache/timelines_author_features:twemcaches\")\n  }\n\n  @Provides\n  @Singleton\n  @Named(TwhinAuthorFollowFeatureCacheClient)\n  def providesTwhinAuthorFollowFeatureCacheClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Memcache = {\n    val cacheClient = MemcachedClientBuilder.buildRawMemcachedClient(\n      numTries = 2,\n      numConnections = 1,\n      requestTimeout = 150.milliseconds,\n      globalTimeout = 300.milliseconds,\n      connectTimeout = 200.milliseconds,\n      acquisitionTimeout = 200.milliseconds,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver\n    )\n\n    buildMemcacheClient(cacheClient, \"/s/cache/home_twhin_author_features:twemcaches\")\n  }\n\n  @Provides\n  @Singleton\n  @Named(RealTimeInteractionGraphUserVertexClient)\n  def providesRealTimeInteractionGraphUserVertexClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Memcache = {\n    val cacheClient = MemcachedClientBuilder.buildRawMemcachedClient(\n      numTries = 2,\n      numConnections = 1,\n      requestTimeout = 150.milliseconds,\n      globalTimeout = 300.milliseconds,\n      connectTimeout = 200.milliseconds,\n      acquisitionTimeout = 200.milliseconds,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver\n    )\n\n    buildMemcacheClient(cacheClient, \"/s/cache/realtime_interactive_graph_prod_v2:twemcaches\")\n  }\n\n  @Provides\n  @Singleton\n  @Named(TvRealTimeAggregateClient)\n  def providesTvRealTimeAggregateClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Memcache = {\n    val cacheClient = MemcachedClientBuilder.buildRawMemcachedClient(\n      numTries = 1,\n      numConnections = 1,\n      requestTimeout = 200.milliseconds,\n      globalTimeout = 200.milliseconds,\n      connectTimeout = 200.milliseconds,\n      acquisitionTimeout = 200.milliseconds,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver\n    )\n    buildMemcacheClient(cacheClient, \"/srv#/prod/local/cache/tv_real_time_aggregates:twemcaches\")\n  }\n\n  @Provides\n  @Singleton\n  @Named(EntityRealGraphClientStore)\n  def providesEntityRealGraphClient(\n    serviceIdentifier: ServiceIdentifier\n  ): ReadableStore[erg.EntityRealGraphRequest, erg.EntityRealGraphResponse] = {\n    EntityRealGraphClient(Environment.withName(\"prod\"), serviceIdentifier, Some(250.millis))\n  }\n\n  private def buildMemcacheClient(cacheClient: Memcached.Client, dest: String): Memcache =\n    FinagleMemcacheFactory(\n      client = cacheClient,\n      dest = dest,\n      hashName = memcacheKeyHasher\n    )()\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/MemcachedScoredCandidateFeaturesStoreModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.Store\nimport com.twitter.timelines.clients.memcache_common.StorehausMemcacheConfig\nimport com.twitter.timelines.ml.cont_train.common.client.scored_candidate_features_cache.ScoredCandidateFeaturesMemcacheBuilder\nimport com.twitter.timelines.served_candidates_logging.{thriftscala => scl}\nimport com.twitter.timelines.suggests.common.poly_data_record.{thriftjava => pdr}\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MemcacheCandidateFeaturesStore\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MemcacheVideoCandidateFeaturesStore\nimport com.twitter.finagle.memcached.compressing.scheme.Lz4\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject MemcachedScoredCandidateFeaturesStoreModule extends TwitterModule {\n\n  private val ScoredTweetsProdDestName =\n    \"/srv#/prod/local/cache/timelines_scored_candidate_features:twemcaches\"\n  private val ScoredTweetsStagingDestName =\n    \"/srv#/test/local/cache/twemcache_timelines_scored_candidate_features:twemcaches\"\n\n  private val ScoredVideoProdDestName =\n    \"/srv#/prod/local/cache/timelines_scored_video_candidate_features:twemcaches\"\n  private val ScoredVideoStagingDestName =\n    \"/srv#/test/local/cache/twemcache_timelines_scored_video_candidate_features:twemcaches\"\n\n  @Singleton\n  @Provides\n  @Named(MemcacheCandidateFeaturesStore)\n  def providesMemcachedScoredCandidateFeaturesStore(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Store[scl.CandidateFeatureKey, pdr.PolyDataRecord] = {\n    val destName = serviceIdentifier.environment.toLowerCase match {\n      case \"prod\" => ScoredTweetsProdDestName\n      case _ => ScoredTweetsStagingDestName\n    }\n    buildCacheClient(serviceIdentifier, statsReceiver, destName)\n  }\n\n  @Singleton\n  @Provides\n  @Named(MemcacheVideoCandidateFeaturesStore)\n  def providesMemcachedScoredVideoCandidateFeaturesStore(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Store[scl.CandidateFeatureKey, pdr.PolyDataRecord] = {\n    val destName = serviceIdentifier.environment.toLowerCase match {\n      case \"prod\" => ScoredVideoProdDestName\n      case _ => ScoredVideoStagingDestName\n    }\n    buildCacheClient(serviceIdentifier, statsReceiver, destName)\n  }\n\n  private def buildCacheClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver,\n    destName: String,\n  ): Store[scl.CandidateFeatureKey, pdr.PolyDataRecord] = {\n\n    new ScoredCandidateFeaturesMemcacheBuilder(\n      config = StorehausMemcacheConfig(\n        destName = destName,\n        keyPrefix = \"\",\n        requestTimeout = 200.milliseconds,\n        numTries = 2,\n        globalTimeout = 500.milliseconds,\n        tcpConnectTimeout = 20.milliseconds,\n        connectionAcquisitionTimeout = 150.milliseconds,\n        numPendingRequests = 200,\n        isReadOnly = false,\n        serviceIdentifier = serviceIdentifier,\n        numConnections = 1,\n        compressionScheme = Lz4\n      ),\n      ttl = 5.minute,\n      statsReceiver = statsReceiver\n    ).build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/NaviModelClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.Http\nimport com.twitter.finagle.grpc.FinagleChannelBuilder\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient.MtlsStackClientSyntax\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecap\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecapGPU\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecapRealtime\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecapSecondary\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.NaviModelClientHomeRecapVideo\nimport com.twitter.inject.TwitterModule\nimport com.twitter.timelines.clients.predictionservice.PredictionGRPCService\nimport com.twitter.util.Duration\nimport io.grpc.ManagedChannel\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject NaviModelClientModule extends TwitterModule {\n\n  private val Authority = \"rustserving\"\n  private val MaxRetryAttempts = 2\n  private val MaxPredictionTimeoutMs: Duration = 700.millis\n  private val ConnectTimeoutMs: Duration = 200.millis\n  private val AcquisitionTimeoutMs: Duration = 500.millis\n\n  @Singleton\n  @Named(NaviModelClientHomeRecap)\n  @Provides\n  def providesHomeRecapPredictionGRPCService(\n    serviceIdentifier: ServiceIdentifier,\n  ): PredictionGRPCService = {\n    providesPredictionGRPCService(serviceIdentifier, \"navi_home_recap_onnx\")\n  }\n\n  @Singleton\n  @Named(NaviModelClientHomeRecapSecondary)\n  @Provides\n  def providesHomeRecapSecondaryPredictionGRPCService(\n    serviceIdentifier: ServiceIdentifier,\n  ): PredictionGRPCService = {\n    providesPredictionGRPCService(serviceIdentifier, \"navi_home_recap_onnx_2\")\n  }\n\n  @Singleton\n  @Named(NaviModelClientHomeRecapRealtime)\n  @Provides\n  def providesHomeRecapRealtimePredictionGRPCService(\n    serviceIdentifier: ServiceIdentifier,\n  ): PredictionGRPCService = {\n    providesPredictionGRPCService(serviceIdentifier, \"navi_home_realtime_recap_onnx\")\n  }\n\n  @Singleton\n  @Named(NaviModelClientHomeRecapGPU)\n  @Provides\n  def providesHomeRecapGPUPredictionGRPCService(\n    serviceIdentifier: ServiceIdentifier,\n  ): PredictionGRPCService = {\n    providesPredictionGRPCService(serviceIdentifier, \"navi_home_recap_onnx_v100\")\n  }\n\n  @Singleton\n  @Named(NaviModelClientHomeRecapVideo)\n  @Provides\n  def providesHomeRecapVideoPredictionGRPCService(\n    serviceIdentifier: ServiceIdentifier,\n  ): PredictionGRPCService = {\n    providesPredictionGRPCService(serviceIdentifier, \"navi_home_recap_video_onnx\")\n  }\n\n  private def providesPredictionGRPCService(\n    serviceIdentifier: ServiceIdentifier,\n    naviClusterName: String\n  ): PredictionGRPCService = {\n    val modelPath = s\"/s/ml-serving/$naviClusterName\"\n    val client = Http.client\n      .withLabel(modelPath)\n      .withMutualTls(serviceIdentifier)\n      .withRequestTimeout(MaxPredictionTimeoutMs)\n      .withTransport.connectTimeout(ConnectTimeoutMs)\n      .withSession.acquisitionTimeout(AcquisitionTimeoutMs)\n      .withHttpStats\n\n    val channel: ManagedChannel = FinagleChannelBuilder\n      .forTarget(modelPath)\n      .overrideAuthority(Authority)\n      .maxRetryAttempts(MaxRetryAttempts)\n      .enableRetryForStatus(io.grpc.Status.RESOURCE_EXHAUSTED)\n      .enableRetryForStatus(io.grpc.Status.UNKNOWN)\n      .enableUnsafeFullyBufferingMode()\n      .httpClient(client)\n      .build()\n\n    new PredictionGRPCService(channel)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/OptimizedStratoClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.service.Retries\nimport com.twitter.finagle.service.RetryPolicy\nimport com.twitter.finagle.ssl.OpportunisticTls\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithDefaultTimeout\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.StratoClientWithLongTimeout\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithLongTimeout\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithModerateTimeout\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TesBatchedStratoClient\nimport com.twitter.inject.TwitterModule\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.client.Strato\nimport com.twitter.strato.rpc.ClientBucketingStrategy\nimport com.twitter.util.Try\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/**\n * Strato Finagle config is copied from TLX\n */\nobject OptimizedStratoClientModule extends TwitterModule {\n\n  private val StratoClientConnectionTimeout = 200.millis\n  private val StratoClientAcquisitionTimeout = 500.millis\n  private val DefaultStratoClientRequestTimeout = 280.millis\n  private val StratoClientLongRequestTimeout = 1000.millis\n  private val ModerateStratoClientRequestTimeout = 600.millis\n  private val LongStratoClientRequestTimeout = 1000.millis\n\n  private val DefaultRetryPartialFunction: PartialFunction[Try[Nothing], Boolean] =\n    RetryPolicy.TimeoutAndWriteExceptionsOnly\n      .orElse(RetryPolicy.ChannelClosedExceptionsOnly)\n\n  protected def mkRetryPolicy(tries: Int): RetryPolicy[Try[Nothing]] =\n    RetryPolicy.tries(tries, DefaultRetryPartialFunction)\n\n  @Singleton\n  @Provides\n  @Named(BatchedStratoClientWithDefaultTimeout)\n  def providesDefaultStratoClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Client = {\n    Strato.client\n      .withMutualTls(serviceIdentifier, opportunisticLevel = OpportunisticTls.Required)\n      .withSession.acquisitionTimeout(StratoClientAcquisitionTimeout)\n      .withTransport.connectTimeout(StratoClientConnectionTimeout)\n      .withRequestTimeout(DefaultStratoClientRequestTimeout)\n      .withPerRequestTimeout(DefaultStratoClientRequestTimeout)\n      .configured(Retries.Policy(mkRetryPolicy(1)))\n      .withStatsReceiver(statsReceiver.scope(\"default_strato_client\"))\n      .build()\n  }\n\n  @Singleton\n  @Provides\n  @Named(StratoClientWithLongTimeout)\n  def providesLongStratoClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Client = {\n    Strato.client\n      .withMutualTls(serviceIdentifier, opportunisticLevel = OpportunisticTls.Required)\n      .withSession.acquisitionTimeout(StratoClientAcquisitionTimeout)\n      .withTransport.connectTimeout(StratoClientConnectionTimeout)\n      .withRequestTimeout(StratoClientLongRequestTimeout)\n      .withPerRequestTimeout(StratoClientLongRequestTimeout)\n      .configured(Retries.Policy(mkRetryPolicy(10)))\n      .withStatsReceiver(statsReceiver.scope(\"long_strato_client\"))\n      .build()\n  }\n\n  @Singleton\n  @Provides\n  @Named(BatchedStratoClientWithModerateTimeout)\n  def providesModerateTimeoutStratoClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Client = {\n    Strato.client\n      .withMutualTls(serviceIdentifier, opportunisticLevel = OpportunisticTls.Required)\n      .withSession.acquisitionTimeout(StratoClientAcquisitionTimeout)\n      .withTransport.connectTimeout(StratoClientConnectionTimeout)\n      .withRequestTimeout(ModerateStratoClientRequestTimeout)\n      .withPerRequestTimeout(ModerateStratoClientRequestTimeout)\n      .withRpcBatchSize(64)\n      .configured(Retries.Policy(mkRetryPolicy(1)))\n      .withStatsReceiver(statsReceiver.scope(\"moderate_timeout_strato_client\"))\n      .build()\n  }\n\n  @Singleton\n  @Provides\n  @Named(BatchedStratoClientWithLongTimeout)\n  def providesLongTimeoutStratoClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Client = {\n    Strato.client\n      .withMutualTls(serviceIdentifier, opportunisticLevel = OpportunisticTls.Required)\n      .withSession.acquisitionTimeout(StratoClientAcquisitionTimeout)\n      .withTransport.connectTimeout(StratoClientConnectionTimeout)\n      .withRequestTimeout(LongStratoClientRequestTimeout)\n      .withPerRequestTimeout(LongStratoClientRequestTimeout)\n      .withRpcBatchSize(5)\n      .configured(Retries.Policy(mkRetryPolicy(1)))\n      .withStatsReceiver(statsReceiver.scope(\"long_timeout_strato_client\"))\n      .build()\n  }\n\n  @Singleton\n  @Provides\n  @Named(TesBatchedStratoClient)\n  def providesTesStratoClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Client = {\n    Strato.client\n      .withMutualTls(serviceIdentifier, opportunisticLevel = OpportunisticTls.Required)\n      .withSession.acquisitionTimeout(StratoClientAcquisitionTimeout)\n      .withTransport.connectTimeout(StratoClientConnectionTimeout)\n      .withRequestTimeout(ModerateStratoClientRequestTimeout)\n      .withPerRequestTimeout(ModerateStratoClientRequestTimeout)\n      .withRpcBatchSize(140)\n      .withBucketingStrategy(ClientBucketingStrategy.ByArg)\n      .configured(Retries.Policy(mkRetryPolicy(1)))\n      .withStatsReceiver(statsReceiver.scope(\"tes_strato_client\"))\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/PeopleDiscoveryServiceModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.peoplediscovery.api.thriftscala.ThriftPeopleDiscoveryService\nimport com.twitter.util.Duration\n\n/**\n * Copy of com.twitter.product_mixer.component_library.module.PeopleDiscoveryServiceModule\n */\nobject PeopleDiscoveryServiceModule\n    extends ThriftMethodBuilderClientModule[\n      ThriftPeopleDiscoveryService.ServicePerEndpoint,\n      ThriftPeopleDiscoveryService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label: String = \"people-discovery-api\"\n\n  override val dest: String = \"/s/people-discovery-api/people-discovery-api:thrift\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(350.millis)\n      .withTimeoutTotal(350.millis)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/PhoenixClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.name.Named\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixCluster\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixCluster.Experiment1\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixCluster.Experiment2\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixCluster.Experiment3\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixCluster.Experiment4\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixCluster.Experiment5\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixCluster.Experiment6\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixCluster.Experiment7\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixCluster.Experiment8\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixCluster.Prod\nimport com.twitter.inject.TwitterModule\nimport io.grpc.ManagedChannel\nimport io.grpc.netty.NettyChannelBuilder\nimport java.util.concurrent.TimeUnit\nimport javax.inject.Singleton\n\nobject PhoenixClientModule extends TwitterModule {\n\n  val ChannelsPerHost = 10\n  val XAIGrpcPort = 80\n  val ProdEndpoint = \"\"\n\n  private def buildChannels(endpoint: String): Seq[ManagedChannel] = {\n    val endpoints = Seq.fill(ChannelsPerHost)(endpoint)\n    endpoints.map { host =>\n      NettyChannelBuilder\n        .forAddress(host, XAIGrpcPort)\n        .usePlaintext()\n        .keepAliveTime(60, TimeUnit.SECONDS)\n        .keepAliveTimeout(20, TimeUnit.SECONDS)\n        .keepAliveWithoutCalls(true)\n        .initialFlowControlWindow(128 * 1024 * 1024)\n        .flowControlWindow(1024 * 1024 * 128)\n        .maxInboundMessageSize(20 * 1024 * 1024)\n        .build()\n    }\n  }\n\n  @Provides\n  @Singleton\n  @Named(\"PhoenixClient\")\n  private def getStub(): Map[PhoenixCluster.Value, Seq[ManagedChannel]] = {\n    val endpointMap = Map(\n      Prod -> ProdEndpoint,\n      Experiment1 -> \"\",\n      Experiment2 -> \"\",\n      Experiment3 -> \"\",\n      Experiment4 -> \"\",\n      Experiment5 -> \"\",\n      Experiment6 -> \"\",\n      Experiment7 -> \"\",\n      Experiment8 -> \"\",\n    ).withDefaultValue(ProdEndpoint)\n\n    endpointMap.mapValues(buildChannels)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/PipelineFailureExceptionMapper.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.twitter.finatra.thrift.exceptions.ExceptionMapper\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.util.logging.Logging\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ProductDisabled\nimport com.twitter.scrooge.ThriftException\nimport com.twitter.util.Future\nimport javax.inject.Singleton\n\n@Singleton\nclass PipelineFailureExceptionMapper\n    extends ExceptionMapper[PipelineFailure, ThriftException]\n    with Logging {\n\n  def handleException(throwable: PipelineFailure): Future[ThriftException] = {\n    throwable match {\n      // SliceService (unlike UrtService) throws an exception when the requested product is disabled\n      case PipelineFailure(ProductDisabled, reason, _, _) =>\n        Future.exception(\n          t.ValidationExceptionList(errors =\n            Seq(t.ValidationException(t.ValidationErrorCode.ProductDisabled, reason))))\n      case _ =>\n        error(\"Unhandled PipelineFailure\", throwable)\n        Future.exception(throwable)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/RealGraphInNetworkScoresModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.name.Named\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RealGraphInNetworkScoresOnPrem\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RealGraphManhattanEndpoint\nimport com.twitter.home_mixer.store.RealGraphInNetworkScoresStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpoint\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.util.CommonTypes.ViewerId\nimport com.twitter.wtf.candidate.thriftscala.Candidate\nimport javax.inject.Singleton\n\nobject RealGraphInNetworkScoresModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(RealGraphInNetworkScoresOnPrem)\n  def providesRealGraphInNetworkScoresFeaturesStore(\n    @Named(RealGraphManhattanEndpoint) realGraphInNetworkScoresManhattanKVEndpoint: ManhattanKVEndpoint\n  ): ReadableStore[ViewerId, Seq[Candidate]] = {\n    new RealGraphInNetworkScoresStore(realGraphInNetworkScoresManhattanKVEndpoint)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/RealtimeAggregateFeatureRepositoryModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.name.Named\nimport com.twitter.bijection.Injection\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.bijection.thrift.ThriftCodec\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.EngagementsReceivedByAuthorCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RealTimeInteractionGraphUserVertexCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RealTimeInteractionGraphUserVertexClient\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TimelinesRealTimeAggregateClient\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TopicCountryEngagementCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TopicEngagementCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TvRealTimeAggregateClient\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TvVideoByUserTweetCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetCountryEngagementCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetEngagementCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwitterListEngagementCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UserAuthorEngagementCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UserEngagementCache\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UserTopicEngagementForNewUserCache\nimport com.twitter.home_mixer.util.InjectionTransformerImplicits._\nimport com.twitter.inject.TwitterModule\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.{api => ml}\nimport com.twitter.servo.cache.KeyValueTransformingReadCache\nimport com.twitter.servo.cache.Memcache\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.servo.util.Transformer\nimport com.twitter.storehaus_internal.memcache.MemcacheHelper\nimport com.twitter.summingbird.batch.Batcher\nimport com.twitter.summingbird_internal.bijection.BatchPairImplicits\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregationKey\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregationKeyInjection\nimport com.twitter.wtf.real_time_interaction_graph.{thriftscala => ig}\nimport javax.inject.Singleton\n\nobject RealtimeAggregateFeatureRepositoryModule\n    extends TwitterModule\n    with RealtimeAggregateHelpers {\n\n  @Provides\n  @Singleton\n  @Named(UserTopicEngagementForNewUserCache)\n  def providesUserTopicEngagementForNewUserCache(\n    @Named(TimelinesRealTimeAggregateClient) client: Memcache\n  ): ReadCache[(Long, Long), ml.DataRecord] = {\n    new KeyValueTransformingReadCache(\n      client,\n      dataRecordValueTransformer,\n      keyTransformD2(userIdFeature, topicIdFeature)\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(TwitterListEngagementCache)\n  def providesTwitterListEngagementCache(\n    @Named(TimelinesRealTimeAggregateClient) client: Memcache\n  ): ReadCache[Long, ml.DataRecord] = {\n    new KeyValueTransformingReadCache(\n      client,\n      dataRecordValueTransformer,\n      keyTransformD1(listIdFeature)\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(TopicEngagementCache)\n  def providesTopicEngagementCache(\n    @Named(TimelinesRealTimeAggregateClient) client: Memcache\n  ): ReadCache[Long, ml.DataRecord] = {\n    new KeyValueTransformingReadCache(\n      client,\n      dataRecordValueTransformer,\n      keyTransformD1(topicIdFeature)\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(UserAuthorEngagementCache)\n  def providesUserAuthorEngagementCache(\n    @Named(TimelinesRealTimeAggregateClient) client: Memcache\n  ): ReadCache[(Long, Long), ml.DataRecord] = {\n    new KeyValueTransformingReadCache(\n      client,\n      dataRecordValueTransformer,\n      keyTransformD2(userIdFeature, authorIdFeature)\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(UserEngagementCache)\n  def providesUserEngagementCache(\n    @Named(TimelinesRealTimeAggregateClient) client: Memcache\n  ): ReadCache[Long, ml.DataRecord] = {\n    new KeyValueTransformingReadCache(\n      client,\n      dataRecordValueTransformer,\n      keyTransformD1(userIdFeature)\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(TweetCountryEngagementCache)\n  def providesTweetCountryEngagementCache(\n    @Named(TimelinesRealTimeAggregateClient) client: Memcache\n  ): ReadCache[(Long, String), ml.DataRecord] = {\n\n    new KeyValueTransformingReadCache(\n      client,\n      dataRecordValueTransformer,\n      keyTransformD1T1(tweetIdFeature, countryCodeFeature)\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(TweetEngagementCache)\n  def providesTweetEngagementCache(\n    @Named(TimelinesRealTimeAggregateClient) client: Memcache\n  ): ReadCache[Long, ml.DataRecord] = {\n    new KeyValueTransformingReadCache(\n      client,\n      dataRecordValueTransformer,\n      keyTransformD1(tweetIdFeature)\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(EngagementsReceivedByAuthorCache)\n  def providesEngagementsReceivedByAuthorCache(\n    @Named(TimelinesRealTimeAggregateClient) client: Memcache\n  ): ReadCache[Long, ml.DataRecord] = {\n    new KeyValueTransformingReadCache(\n      client,\n      dataRecordValueTransformer,\n      keyTransformD1(authorIdFeature)\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(TopicCountryEngagementCache)\n  def providesTopicCountryEngagementCache(\n    @Named(TimelinesRealTimeAggregateClient) client: Memcache\n  ): ReadCache[(Long, String), ml.DataRecord] = {\n    new KeyValueTransformingReadCache(\n      client,\n      dataRecordValueTransformer,\n      keyTransformD1T1(topicIdFeature, countryCodeFeature)\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(RealTimeInteractionGraphUserVertexCache)\n  def providesRealTimeInteractionGraphUserVertexCache(\n    @Named(RealTimeInteractionGraphUserVertexClient) client: Memcache\n  ): ReadCache[Long, ig.UserVertex] = {\n\n    val valueTransformer = BinaryScalaCodec(ig.UserVertex).toByteArrayTransformer()\n\n    val underlyingKey: Long => String = {\n      val cacheKeyPrefix = \"user_vertex\"\n      val defaultBatchID = Batcher.unit.currentBatch\n      val batchPairInjection = BatchPairImplicits.keyInjection(Injection.connect[Long, Array[Byte]])\n      MemcacheHelper\n        .keyEncoder(cacheKeyPrefix)(batchPairInjection)\n        .compose((k: Long) => (k, defaultBatchID))\n    }\n\n    new KeyValueTransformingReadCache(\n      client,\n      valueTransformer,\n      underlyingKey\n    )\n  }\n\n  @Provides\n  @Singleton\n  @Named(TvVideoByUserTweetCache)\n  def providesTvVideoImpressionByUserTweetCache(\n    @Named(TvRealTimeAggregateClient) client: Memcache\n  ): ReadCache[(Long, Long), ml.DataRecord] = {\n    new KeyValueTransformingReadCache(\n      client,\n      dataRecordValueTransformer,\n      keyTransformD2(userIdFeature, tweetIdFeature)\n    )\n  }\n}\n\ntrait RealtimeAggregateHelpers {\n\n  val authorIdFeature = new Feature.Discrete(\"entities.source_author_id\").getFeatureId\n  val countryCodeFeature = new Feature.Text(\"geo.user_location.country_code\").getFeatureId\n  val listIdFeature = new Feature.Discrete(\"list.id\").getFeatureId\n  val userIdFeature = new Feature.Discrete(\"meta.user_id\").getFeatureId\n  val topicIdFeature = new Feature.Discrete(\"entities.topic_id\").getFeatureId\n  val tweetIdFeature = new Feature.Discrete(\"entities.source_tweet_id\").getFeatureId\n\n  private def customKeyBuilder[K](prefix: String, f: K => Array[Byte]): K => String = {\n    // intentionally not implementing injection inverse because it is never used\n    def g(arr: Array[Byte]) = ???\n\n    MemcacheHelper.keyEncoder(prefix)(Injection.build(f)(g))\n  }\n\n  private val keyEncoder: AggregationKey => String = {\n    val cacheKeyPrefix = \"\"\n    val defaultBatchID = Batcher.unit.currentBatch\n\n    val batchPairInjection = BatchPairImplicits.keyInjection(AggregationKeyInjection)\n    customKeyBuilder(cacheKeyPrefix, batchPairInjection)\n      .compose((k: AggregationKey) => (k, defaultBatchID))\n  }\n\n  def keyTransformD1AggregationKey(f1: Long)(key: Long): AggregationKey = {\n    AggregationKey(Map(f1 -> key), Map.empty)\n  }\n\n  def keyTransformD1(f1: Long)(key: Long): String = {\n    val aggregationKey = AggregationKey(Map(f1 -> key), Map.empty)\n    keyEncoder(aggregationKey)\n  }\n\n  def keyTransformD2(f1: Long, f2: Long)(keys: (Long, Long)): String = {\n    val (k1, k2) = keys\n    val aggregationKey = AggregationKey(Map(f1 -> k1, f2 -> k2), Map.empty)\n    keyEncoder(aggregationKey)\n  }\n\n  def keyTransformD2AggregationKey(f1: Long, f2: Long)(keys: (Long, Long)): AggregationKey = {\n    val (k1, k2) = keys\n    AggregationKey(Map(f1 -> k1, f2 -> k2), Map.empty)\n  }\n\n  def keyTransformD1T1(f1: Long, f2: Long)(keys: (Long, String)): String = {\n    val (k1, k2) = keys\n    val aggregationKey = AggregationKey(Map(f1 -> k1), Map(f2 -> k2))\n    keyEncoder(aggregationKey)\n  }\n\n  def keyTransformD1T1AggregationKey(f1: Long, f2: Long)(keys: (Long, String)): AggregationKey = {\n    val (k1, k2) = keys\n    AggregationKey(Map(f1 -> k1), Map(f2 -> k2))\n  }\n\n  val dataRecordValueTransformer: Transformer[DataRecord, Array[Byte]] = ThriftCodec\n    .toCompact[ml.DataRecord]\n    .toByteArrayTransformer()\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/ScoredTweetsMemcacheModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.ScoredTweetsCache\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.shared_library.memcached_client.MemcachedClientBuilder\nimport com.twitter.servo.cache.FinagleMemcache\nimport com.twitter.servo.cache.KeyTransformer\nimport com.twitter.servo.cache.KeyValueTransformingTtlCache\nimport com.twitter.servo.cache.Serializer\nimport com.twitter.servo.cache.ThriftSerializer\nimport com.twitter.servo.cache.TtlCache\nimport com.twitter.timelines.model.UserId\nimport org.apache.thrift.protocol.TCompactProtocol\nimport com.twitter.finagle.memcached.compressing.scheme.Lz4\n\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject ScoredTweetsMemcacheModule extends TwitterModule {\n\n  private val ScopeName = \"ScoredTweetsCache\"\n  private val ProdDestName = \"/srv#/prod/local/cache/home_scored_tweets:twemcaches\"\n  private val StagingDestName = \"/srv#/test/local/cache/twemcache_home_scored_tweets:twemcaches\"\n  private val scoredTweetsSerializer: Serializer[t.ScoredTweetsResponse] =\n    new ThriftSerializer[t.ScoredTweetsResponse](\n      t.ScoredTweetsResponse,\n      new TCompactProtocol.Factory())\n  private val userIdKeyTransformer: KeyTransformer[UserId] = (userId: UserId) => userId.toString\n\n  @Singleton\n  @Named(ScoredTweetsCache)\n  @Provides\n  def providesScoredTweetsCache(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): TtlCache[UserId, t.ScoredTweetsResponse] = {\n    val destName = serviceIdentifier.environment.toLowerCase match {\n      case \"prod\" => ProdDestName\n      case _ => StagingDestName\n    }\n    val client = MemcachedClientBuilder.buildMemcachedClient(\n      destName = destName,\n      numTries = 2,\n      numConnections = 1,\n      requestTimeout = 200.milliseconds,\n      globalTimeout = 400.milliseconds,\n      connectTimeout = 100.milliseconds,\n      acquisitionTimeout = 100.milliseconds,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver.scope(ScopeName),\n      compressionScheme = Lz4\n    )\n    val underlyingCache = new FinagleMemcache(client)\n\n    new KeyValueTransformingTtlCache(\n      underlyingCache = underlyingCache,\n      transformer = scoredTweetsSerializer,\n      underlyingKey = userIdKeyTransformer\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/ScoredVideoTweetsMemcacheModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.memcached.compressing.scheme.Lz4\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.ScoredVideoTweetsCache\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.shared_library.memcached_client.MemcachedClientBuilder\nimport com.twitter.servo.cache._\nimport com.twitter.timelines.model.UserId\nimport org.apache.thrift.protocol.TCompactProtocol\n\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject ScoredVideoTweetsMemcacheModule extends TwitterModule {\n\n  private val ScopeName = \"ScoredVideoTweetsCache\"\n  private val destName = \"/s/cache/explore_served_tweet_impressions:twemcaches\"\n  private val scoredTweetsSerializer: Serializer[t.ScoredTweetsResponse] =\n    new ThriftSerializer[t.ScoredTweetsResponse](\n      t.ScoredTweetsResponse,\n      new TCompactProtocol.Factory())\n\n  private val userIdKeyTransformer: KeyTransformer[UserId] = (userId: UserId) =>\n    userId.toString + \":home-mixer\"\n\n  @Singleton\n  @Named(ScoredVideoTweetsCache)\n  @Provides\n  def providesScoredTweetsCache(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): TtlCache[UserId, t.ScoredTweetsResponse] = {\n    val client = MemcachedClientBuilder.buildMemcachedClient(\n      destName = destName,\n      numTries = 2,\n      numConnections = 1,\n      requestTimeout = 200.milliseconds,\n      globalTimeout = 400.milliseconds,\n      connectTimeout = 100.milliseconds,\n      acquisitionTimeout = 100.milliseconds,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver.scope(ScopeName),\n      compressionScheme = Lz4\n    )\n    val underlyingCache = new FinagleMemcache(client)\n\n    new KeyValueTransformingTtlCache(\n      underlyingCache = underlyingCache,\n      transformer = scoredTweetsSerializer,\n      underlyingKey = userIdKeyTransformer\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/ScribeEventPublisherModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.clientapp.{thriftscala => ca}\nimport com.twitter.finatra.kafka.interceptors.InstanceMetadataProducerInterceptor\nimport com.twitter.finatra.kafka.interceptors.PublishTimeProducerInterceptor\nimport com.twitter.finatra.kafka.producers.FinagleKafkaProducerBuilder\nimport com.twitter.finatra.kafka.serde.ScalaSerdes\nimport com.twitter.finatra.kafka.serde.UnKeyedSerde\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.CommonFeaturesScribeEventPublisher\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.CommonFeaturesScribeVideoEventPublisher\nimport com.twitter.inject.TwitterModule\nimport com.twitter.logpipeline.client.EventPublisherManager\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.logpipeline.client.serializers.EventLogMsgTBinarySerializer\nimport com.twitter.timelines.suggests.common.poly_data_record.{thriftjava => pldr}\nimport com.twitter.timelines.timeline_logging.{thriftscala => tl}\nimport com.twitter.util.Duration\nimport com.twitter.util.StorageUnit\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport org.apache.kafka.clients.CommonClientConfigs\nimport org.apache.kafka.common.config.SaslConfigs\nimport org.apache.kafka.common.config.SslConfigs\nimport org.apache.kafka.common.security.auth.SecurityProtocol\n\nobject ScribeEventPublisherModule extends TwitterModule {\n\n  val ClientEventLogCategory = \"client_event\"\n  val ServedCandidatesLogCategory = \"home_timeline_served_candidates_flattened\"\n  val ScoredCandidatesLogCategory = \"home_timeline_scored_candidates\"\n  val ServedCommonFeaturesLogCategory = \"tq_served_common_features_offline\"\n  val ServedVideoCommonFeaturesLogCategory = \"tq_served_video_common_features_offline\"\n\n  @Provides\n  @Singleton\n  def providesClientEventsScribeEventPublisher: EventPublisher[ca.LogEvent] = {\n    val builder = FinagleKafkaProducerBuilder()\n      .dest(s\"/s/kafka/client-events:kafka-tls\")\n      .keySerializer(UnKeyedSerde.serializer)\n      .valueSerializer(ScalaSerdes.Thrift[ca.LogEvent].serializer)\n      .clientId(\"home_mixer_client_event_publisher\")\n      .linger(Duration.fromMilliseconds(16))\n      .batchSize(StorageUnit.fromKilobytes(64))\n      .deliveryTimeout(Duration.fromMilliseconds(30000))\n      .requestTimeout(Duration.fromMilliseconds(25000))\n      .withConfig(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, SecurityProtocol.SASL_SSL.toString)\n      .interceptor[PublishTimeProducerInterceptor]\n      .interceptor[InstanceMetadataProducerInterceptor]\n      .withConfig(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, \"\")\n      .withConfig(SaslConfigs.SASL_MECHANISM, SaslConfigs.GSSAPI_MECHANISM)\n      .withConfig(SaslConfigs.SASL_KERBEROS_SERVICE_NAME, \"kafka\")\n      .withConfig(SaslConfigs.SASL_KERBEROS_SERVER_NAME, \"kafka\")\n    EventPublisherManager.buildKafkaLogPipelinePublisher(builder, ClientEventLogCategory)\n  }\n\n  @Provides\n  @Singleton\n  @Named(CommonFeaturesScribeEventPublisher)\n  def providesCommonFeaturesScribeEventPublisher: EventPublisher[pldr.PolyDataRecord] = {\n    val serializer = EventLogMsgTBinarySerializer.getNewSerializer\n    EventPublisherManager.buildScribeLogPipelinePublisher(\n      ServedCommonFeaturesLogCategory,\n      serializer)\n  }\n\n  @Provides\n  @Singleton\n  @Named(CommonFeaturesScribeVideoEventPublisher)\n  def providesCommonFeaturesScribeVideoEventPublisher: EventPublisher[pldr.PolyDataRecord] = {\n    val serializer = EventLogMsgTBinarySerializer.getNewSerializer\n    EventPublisherManager.buildScribeLogPipelinePublisher(\n      ServedVideoCommonFeaturesLogCategory,\n      serializer)\n  }\n\n  @Provides\n  @Singleton\n  def providesServedCandidatesScribeEventPublisher: EventPublisher[tl.ServedEntry] = {\n    val builder = FinagleKafkaProducerBuilder()\n      .dest(\"/s/kafka/timeline:kafka-tls\")\n      .keySerializer(UnKeyedSerde.serializer)\n      .valueSerializer(ScalaSerdes.Thrift[tl.ServedEntry].serializer)\n      .clientId(s\"$ServedCandidatesLogCategory-publisher\")\n      .linger(Duration.fromMilliseconds(16))\n      .batchSize(StorageUnit.fromKilobytes(64))\n      .deliveryTimeout(Duration.fromMilliseconds(30000))\n      .requestTimeout(Duration.fromMilliseconds(25000))\n      .withConfig(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, SecurityProtocol.SASL_SSL.toString)\n      .interceptor[PublishTimeProducerInterceptor]\n      .interceptor[InstanceMetadataProducerInterceptor]\n      .withConfig(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, \"\")\n      .withConfig(SaslConfigs.SASL_MECHANISM, SaslConfigs.GSSAPI_MECHANISM)\n      .withConfig(SaslConfigs.SASL_KERBEROS_SERVICE_NAME, \"kafka\")\n      .withConfig(SaslConfigs.SASL_KERBEROS_SERVER_NAME, \"kafka\")\n    EventPublisherManager.buildKafkaLogPipelinePublisher(builder, ServedCandidatesLogCategory)\n  }\n\n  @Provides\n  @Singleton\n  def provideScoredCandidatesScribeEventPublisher: EventPublisher[tl.ScoredCandidate] = {\n    val builder = FinagleKafkaProducerBuilder()\n      .dest(\"/s/kafka/timeline:kafka-tls\")\n      .keySerializer(UnKeyedSerde.serializer)\n      .valueSerializer(ScalaSerdes.Thrift[tl.ScoredCandidate].serializer)\n      .clientId(s\"$ScoredCandidatesLogCategory-publisher\")\n      .linger(Duration.fromMilliseconds(16))\n      .batchSize(StorageUnit.fromKilobytes(64))\n      .deliveryTimeout(Duration.fromMilliseconds(30000))\n      .requestTimeout(Duration.fromMilliseconds(25000))\n      .withConfig(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, SecurityProtocol.SASL_SSL.toString)\n      .interceptor[PublishTimeProducerInterceptor]\n      .interceptor[InstanceMetadataProducerInterceptor]\n      .withConfig(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, \"\")\n      .withConfig(SaslConfigs.SASL_MECHANISM, SaslConfigs.GSSAPI_MECHANISM)\n      .withConfig(SaslConfigs.SASL_KERBEROS_SERVICE_NAME, \"kafka\")\n      .withConfig(SaslConfigs.SASL_KERBEROS_SERVER_NAME, \"kafka\")\n    EventPublisherManager.buildKafkaLogPipelinePublisher(builder, ScoredCandidatesLogCategory)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/SimClustersRecentEngagementsClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithModerateTimeout\nimport com.twitter.inject.TwitterModule\nimport com.twitter.strato.client.Client\nimport com.twitter.timelines.clients.strato.twistly.SimClustersRecentEngagementSimilarityClient\nimport com.twitter.timelines.clients.strato.twistly.SimClustersRecentEngagementSimilarityClientImpl\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject SimClustersRecentEngagementsClientModule extends TwitterModule {\n  @Singleton\n  @Provides\n  def providesSimilarityClient(\n    @Named(BatchedStratoClientWithModerateTimeout)\n    stratoClient: Client,\n    statsReceiver: StatsReceiver\n  ): SimClustersRecentEngagementSimilarityClient = {\n    new SimClustersRecentEngagementSimilarityClientImpl(stratoClient, statsReceiver)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/StaleTweetsCacheModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.name.Named\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.StaleTweetsCache\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.shared_library.memcached_client.MemcachedClientBuilder\nimport javax.inject.Singleton\n\nobject StaleTweetsCacheModule extends TwitterModule {\n\n  @Singleton\n  @Provides\n  @Named(StaleTweetsCache)\n  def providesCache(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): MemcachedClient = {\n    MemcachedClientBuilder.buildMemcachedClient(\n      destName = \"/srv#/prod/local/cache/staletweetscache:twemcaches\",\n      numTries = 3,\n      numConnections = 1,\n      requestTimeout = 200.milliseconds,\n      globalTimeout = 500.milliseconds,\n      connectTimeout = 200.milliseconds,\n      acquisitionTimeout = 200.milliseconds,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver,\n      failureAccrualPolicy = None,\n      keyHasher = Some(KeyHasher.FNV1_32)\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/ThriftFeatureRepositoryModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.graph_feature_service.{thriftscala => gfs}\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.EarlybirdRepository\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.GraphTwoHopRepository\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.InterestsThriftServiceClient\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UserFollowedTopicIdsRepository\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UtegSocialProofRepository\nimport com.twitter.home_mixer.util.earlybird.EarlybirdRequestUtil\nimport com.twitter.inject.TwitterModule\nimport com.twitter.interests.{thriftscala => int}\nimport com.twitter.product_mixer.shared_library.thrift_client.FinagleThriftClientBuilder\nimport com.twitter.product_mixer.shared_library.thrift_client.Idempotent\nimport com.twitter.recos.recos_common.{thriftscala => rc}\nimport com.twitter.recos.user_tweet_entity_graph.{thriftscala => uteg}\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport com.twitter.servo.cache._\nimport com.twitter.servo.keyvalue.KeyValueResultBuilder\nimport com.twitter.servo.repository.ChunkingStrategy\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.servo.repository.KeyValueResult\nimport com.twitter.util.Future\nimport com.twitter.util.Return\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject ThriftFeatureRepositoryModule extends TwitterModule {\n\n  private val DefaultRPCChunkSize = 50\n  private val GFSInteractionIdsLimit = 10\n\n  type EarlybirdQuery = (Seq[Long], Long)\n  type UtegQuery = (Seq[Long], (Long, Map[Long, Double]))\n\n  @Provides\n  @Singleton\n  @Named(InterestsThriftServiceClient)\n  def providesInterestsThriftServiceClient(\n    clientId: ClientId,\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): int.InterestsThriftService.MethodPerEndpoint = {\n    FinagleThriftClientBuilder\n      .buildFinagleMethodPerEndpoint[\n        int.InterestsThriftService.ServicePerEndpoint,\n        int.InterestsThriftService.MethodPerEndpoint](\n        serviceIdentifier = serviceIdentifier,\n        clientId = clientId,\n        dest = \"/s/interests-thrift-service/interests-thrift-service\",\n        label = \"interests\",\n        statsReceiver = statsReceiver,\n        idempotency = Idempotent(1.percent),\n        timeoutPerRequest = 350.milliseconds,\n        timeoutTotal = 350.milliseconds\n      )\n  }\n\n  @Provides\n  @Singleton\n  @Named(UserFollowedTopicIdsRepository)\n  def providesUserFollowedTopicIdsRepository(\n    @Named(InterestsThriftServiceClient) client: int.InterestsThriftService.MethodPerEndpoint\n  ): KeyValueRepository[Seq[Long], Long, Seq[Long]] = {\n\n    val lookupContext = Some(\n      int.ExplicitInterestLookupContext(Some(Seq(int.InterestRelationType.Followed)))\n    )\n\n    def lookup(userId: Long): Future[Seq[Long]] = {\n      client.getUserExplicitInterests(userId, lookupContext).map { interests =>\n        interests.flatMap {\n          _.interestId match {\n            case int.InterestId.SemanticCore(semanticCoreInterest) => Some(semanticCoreInterest.id)\n            case _ => None\n          }\n        }\n      }\n    }\n\n    val keyValueRepository = toRepository(lookup)\n\n    keyValueRepository\n  }\n\n  @Provides\n  @Singleton\n  @Named(UtegSocialProofRepository)\n  def providesUtegSocialProofRepository(\n    clientId: ClientId,\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): KeyValueRepository[UtegQuery, Long, uteg.TweetRecommendation] = {\n    val client = FinagleThriftClientBuilder.buildFinagleMethodPerEndpoint[\n      uteg.UserTweetEntityGraph.ServicePerEndpoint,\n      uteg.UserTweetEntityGraph.MethodPerEndpoint](\n      serviceIdentifier = serviceIdentifier,\n      clientId = clientId,\n      dest = \"/s/cassowary/user_tweet_entity_graph\",\n      label = \"uteg-social-proof-repo\",\n      statsReceiver = statsReceiver,\n      idempotency = Idempotent(1.percent),\n      timeoutPerRequest = 150.milliseconds,\n      timeoutTotal = 250.milliseconds\n    )\n\n    val utegSocialProofTypes = Seq(\n      rc.SocialProofType.Favorite,\n      rc.SocialProofType.Retweet,\n      rc.SocialProofType.Reply\n    )\n\n    def lookup(\n      tweetIds: Seq[Long],\n      view: (Long, Map[Long, Double])\n    ): Future[Seq[Option[uteg.TweetRecommendation]]] = {\n      val (userId, seedsWithWeights) = view\n      val socialProofRequest = uteg.SocialProofRequest(\n        requesterId = Some(userId),\n        seedsWithWeights = seedsWithWeights,\n        inputTweets = tweetIds,\n        socialProofTypes = Some(utegSocialProofTypes)\n      )\n      client.findTweetSocialProofs(socialProofRequest).map { result =>\n        val resultMap = result.socialProofResults.map(t => t.tweetId -> t).toMap\n        tweetIds.map(resultMap.get)\n      }\n    }\n\n    toRepositoryBatchWithView(lookup, chunkSize = 200)\n  }\n\n  @Provides\n  @Singleton\n  @Named(GraphTwoHopRepository)\n  def providesGraphTwoHopRepository(\n    clientId: ClientId,\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): KeyValueRepository[(Seq[Long], Long), Long, Seq[gfs.IntersectionValue]] = {\n    val client = FinagleThriftClientBuilder\n      .buildFinagleMethodPerEndpoint[gfs.Server.ServicePerEndpoint, gfs.Server.MethodPerEndpoint](\n        serviceIdentifier = serviceIdentifier,\n        clientId = clientId,\n        dest = \"/s/cassowary/graph_feature_service-server\",\n        label = \"gfs-repo\",\n        statsReceiver = statsReceiver,\n        idempotency = Idempotent(1.percent),\n        timeoutPerRequest = 350.milliseconds,\n        timeoutTotal = 500.milliseconds\n      )\n\n    def lookup(\n      userIds: Seq[Long],\n      viewerId: Long\n    ): Future[Seq[Option[Seq[gfs.IntersectionValue]]]] = {\n      val gfsIntersectionRequest = gfs.GfsPresetIntersectionRequest(\n        userId = viewerId,\n        candidateUserIds = userIds,\n        presetFeatureTypes = gfs.PresetFeatureTypes.HtlTwoHop,\n        intersectionIdLimit = Some(GFSInteractionIdsLimit)\n      )\n\n      client\n        .getPresetIntersection(gfsIntersectionRequest)\n        .map { graphFeatureServiceResponse =>\n          val resultMap = graphFeatureServiceResponse.results\n            .map(result => result.candidateUserId -> result.intersectionValues).toMap\n          userIds.map(resultMap.get(_))\n        }\n    }\n\n    toRepositoryBatchWithView(lookup, chunkSize = 200)\n  }\n\n  @Provides\n  @Singleton\n  @Named(EarlybirdRepository)\n  def providesEarlybirdSearchRepository(\n    client: eb.EarlybirdService.MethodPerEndpoint,\n    clientId: ClientId\n  ): KeyValueRepository[EarlybirdQuery, Long, eb.ThriftSearchResult] = {\n\n    def lookup(\n      tweetIds: Seq[Long],\n      viewerId: Long\n    ): Future[Seq[Option[eb.ThriftSearchResult]]] = {\n      val request = EarlybirdRequestUtil.getTweetsFeaturesRequest(\n        userId = Some(viewerId),\n        tweetIds = Some(tweetIds),\n        clientId = Some(clientId.name),\n        authorScoreMap = None,\n        tensorflowModel = Some(\"timelines_unified_prod\")\n      )\n\n      client\n        .search(request).map { response =>\n          val resultMap = response.searchResults\n            .map(_.results.map { result => result.id -> result }.toMap).getOrElse(Map.empty)\n          tweetIds.map(resultMap.get)\n        }\n    }\n    toRepositoryBatchWithView(lookup)\n  }\n\n  protected def toRepository[K, V](\n    hydrate: K => Future[V]\n  ): KeyValueRepository[Seq[K], K, V] = {\n    def asRepository(keys: Seq[K]): Future[KeyValueResult[K, V]] = {\n      Future.collect(keys.map(hydrate(_).liftToTry)).map { results =>\n        keys\n          .zip(results)\n          .foldLeft(new KeyValueResultBuilder[K, V]()) {\n            case (bldr, (k, result)) =>\n              result match {\n                case Return(v) => bldr.addFound(k, v)\n                case _ => bldr.addNotFound(k)\n              }\n          }.result\n      }\n    }\n\n    asRepository\n  }\n\n  protected def toRepositoryBatch[K, V](\n    hydrate: Seq[K] => Future[Seq[Option[V]]],\n    chunkSize: Int = DefaultRPCChunkSize\n  ): KeyValueRepository[Seq[K], K, V] = {\n    def repository(keys: Seq[K]): Future[KeyValueResult[K, V]] =\n      batchRepositoryProcess(keys, hydrate(keys))\n\n    KeyValueRepository.chunked(repository, ChunkingStrategy.equalSize(chunkSize))\n  }\n\n  protected def toRepositoryBatchWithView[K, T, V](\n    hydrate: (Seq[K], T) => Future[Seq[Option[V]]],\n    chunkSize: Int = DefaultRPCChunkSize\n  ): KeyValueRepository[(Seq[K], T), K, V] = {\n    def repository(input: (Seq[K], T)): Future[KeyValueResult[K, V]] = {\n      val (keys, view) = input\n      batchRepositoryProcess(keys, hydrate(keys, view))\n    }\n\n    KeyValueRepository.chunked(repository, CustomChunkingStrategy.equalSizeWithView(chunkSize))\n  }\n\n  private def batchRepositoryProcess[K, V](\n    keys: Seq[K],\n    f: Future[Seq[Option[V]]]\n  ): Future[KeyValueResult[K, V]] = {\n    f.liftToTry\n      .map {\n        case Return(values) =>\n          keys\n            .zip(values)\n            .foldLeft(new KeyValueResultBuilder[K, V]()) {\n              case (bldr, (k, value)) =>\n                value match {\n                  case Some(v) => bldr.addFound(k, v)\n                  case _ => bldr.addNotFound(k)\n                }\n            }.result\n        case _ =>\n          keys\n            .foldLeft(new KeyValueResultBuilder[K, V]()) {\n              case (bldr, k) => bldr.addNotFound(k)\n            }.result\n      }\n  }\n\n  // Use only for cases not already covered by Servo's [[ChunkingStrategy]]\n  object CustomChunkingStrategy {\n    def equalSizeWithView[K, T](maxSize: Int): ((Seq[K], T)) => Seq[(Seq[K], T)] = {\n      case (keys, view) =>\n        ChunkingStrategy\n          .equalSize[K](maxSize)(keys)\n          .map { chunk: Seq[K] => (chunk, view) }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/TimelinesPersistenceStoreClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.timelinemixer.clients.persistence.TimelinePersistenceManhattanClientBuilder\nimport com.twitter.timelinemixer.clients.persistence.TimelinePersistenceManhattanClientConfig\nimport com.twitter.timelinemixer.clients.persistence.TimelineResponseBatchesClient\nimport com.twitter.timelinemixer.clients.persistence.TimelineResponseV3\nimport com.twitter.util.Duration\nimport javax.inject.Singleton\n\nobject TimelinesPersistenceStoreClientModule extends TwitterModule {\n  private val StagingDataset = \"timeline_response_batches_v5_nonprod\"\n  private val ProdDataset = \"timeline_response_batches_v5\"\n  private final val Timeout = \"mh_persistence_store.timeout\"\n\n  flag[Duration](Timeout, 300.millis, \"Timeout per request\")\n\n  @Provides\n  @Singleton\n  def providesTimelinesPersistenceStoreClient(\n    @Flag(Timeout) timeout: Duration,\n    injectedServiceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): TimelineResponseBatchesClient[TimelineResponseV3] = {\n    val timelineResponseBatchesDataset =\n      injectedServiceIdentifier.environment.toLowerCase match {\n        case \"prod\" => ProdDataset\n        case _ => StagingDataset\n      }\n\n    val timelineResponseBatchesConfig = new TimelinePersistenceManhattanClientConfig {\n      val dataset = timelineResponseBatchesDataset\n      val isReadOnly = false\n      val serviceIdentifier = injectedServiceIdentifier\n      override val defaultMaxTimeout = timeout\n      override val maxRetryCount = 2\n    }\n\n    TimelinePersistenceManhattanClientBuilder.buildTimelineResponseV3BatchesClient(\n      timelineResponseBatchesConfig,\n      statsReceiver\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/TopicSocialProofClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithModerateTimeout\nimport com.twitter.inject.TwitterModule\nimport com.twitter.strato.client.Client\nimport com.twitter.timelines.clients.strato.topics.TopicSocialProofClient\nimport com.twitter.timelines.clients.strato.topics.TopicSocialProofClientImpl\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject TopicSocialProofClientModule extends TwitterModule {\n\n  @Singleton\n  @Provides\n  def providesSimilarityClient(\n    @Named(BatchedStratoClientWithModerateTimeout)\n    stratoClient: Client,\n    statsReceiver: StatsReceiver\n  ): TopicSocialProofClient = new TopicSocialProofClientImpl(stratoClient, statsReceiver)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/TvWatchHistoryCacheClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TvWatchedVideoIdsKeyCacheStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.shared_library.memcached_client.MemcachedClientBuilder\nimport com.twitter.servo.cache.FinagleMemcache\nimport com.twitter.servo.cache.KeyValueTransformingTtlCache\nimport com.twitter.servo.cache.ObservableTtlCache\nimport com.twitter.servo.cache.TtlCache\nimport com.twitter.servo.util.Transformer\nimport com.twitter.util.Try\nimport java.nio.ByteBuffer\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.mutable.ArrayBuffer\n\nobject TvWatchHistoryCacheClientModule extends TwitterModule {\n  @Provides\n  @Singleton\n  @Named(TvWatchedVideoIdsKeyCacheStore)\n  def providesEntityRealGraphClient(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): TtlCache[Long, Seq[Long]] = {\n    val scopeName = \"TvWatchedVideoIds\"\n    val memCacheClient = MemcachedClientBuilder.buildMemcachedClient(\n      destName = \"/srv#/prod/local/cache/related_videos:twemcaches\",\n      numTries = 1,\n      numConnections = 1,\n      requestTimeout = 50.milliseconds,\n      globalTimeout = 100.milliseconds,\n      connectTimeout = 100.milliseconds,\n      acquisitionTimeout = 100.milliseconds,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver,\n      keyHasher = Some(KeyHasher.FNV1A_64)\n    )\n\n    val longSeqTransformer: Transformer[Seq[Long], Array[Byte]] =\n      new Transformer[Seq[Long], Array[Byte]] {\n        def deserialize(input: Array[Byte]): ArrayBuffer[Long] = {\n          val buffer = ByteBuffer.wrap(input)\n          val resultBuffer = ArrayBuffer[Long]()\n          while (buffer.hasRemaining) {\n            val longValue = buffer.getLong()\n            resultBuffer += longValue\n          }\n          resultBuffer\n        }\n\n        override def from(input: Array[Byte]): Try[Seq[Long]] = Try(deserialize(input))\n\n        override def to(b: Seq[Long]): Try[Array[Byte]] = ???\n      }\n    val keyValueCache = new KeyValueTransformingTtlCache[Long, String, Seq[Long], Array[Byte]](\n      underlyingCache = new FinagleMemcache(memCacheClient),\n      transformer = longSeqTransformer,\n      underlyingKey = { key: Long => key.toString }\n    )\n    ObservableTtlCache(\n      underlyingCache = keyValueCache,\n      statsReceiver = statsReceiver.scope(scopeName),\n      windowSize = 1000,\n      name = scopeName\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/TweetWatchTimeMetadataModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetWatchTimeMetadataStore\nimport com.twitter.home_mixer.store.TweetWatchTimeMetadataStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.twistly.thriftscala.VideoViewEngagementType\nimport com.twitter.twistly.thriftscala.WatchTimeMetadata\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject TweetWatchTimeMetadataModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(TweetWatchTimeMetadataStore)\n  def providesTweetWatchTimeMetadataStore(\n    tweetWatchTimeMetadataStore: TweetWatchTimeMetadataStore\n  ): ReadableStore[(Long, VideoViewEngagementType), WatchTimeMetadata] = {\n    tweetWatchTimeMetadataStore.tweetWatchTimeMetadataStore\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/TweetyPieClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.annotations.Flags\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.stitch.tweetypie.TweetyPie\nimport com.twitter.tweetypie.thriftscala.TweetService\nimport com.twitter.util.Duration\nimport javax.inject.Singleton\n\n/**\n * Idempotent Tweetypie Thrift and Stitch client.\n */\nobject TweetypieClientModule\n    extends ThriftMethodBuilderClientModule[\n      TweetService.ServicePerEndpoint,\n      TweetService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  private val TimeoutRequest = \"tweetypie.timeout_request\"\n  private val TimeoutTotal = \"tweetypie.timeout_total\"\n\n  flag[Duration](TimeoutRequest, 1000.millis, \"Timeout per request\")\n  flag[Duration](TimeoutTotal, 1000.millis, \"Total timeout\")\n\n  override val label: String = \"tweetypie\"\n  override val dest: String = \"/s/tweetypie/tweetypie\"\n\n  @Singleton\n  @Provides\n  def providesTweetypieStitchClient(tweetService: TweetService.MethodPerEndpoint): TweetyPie =\n    new TweetyPie(tweetService)\n\n  /**\n   * TweetyPie client id must be in the form of {service.env} or it will not be treated as an\n   * unauthorized client\n   */\n  override protected def clientId(injector: Injector): ClientId = {\n    val serviceIdentifier = injector.instance[ServiceIdentifier]\n    ClientId(s\"${serviceIdentifier.service}.${serviceIdentifier.environment}\")\n  }\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    val timeoutRequest = injector.instance[Duration](Flags.named(TimeoutRequest))\n    val timeoutTotal = injector.instance[Duration](Flags.named(TimeoutTotal))\n\n    methodBuilder\n      .withTimeoutPerRequest(timeoutRequest)\n      .withTimeoutTotal(timeoutTotal)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.millis\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/TweetypieStaticEntitiesCacheClientModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.google.inject.name.Named\nimport com.twitter.conversions.DurationOps.RichDuration\nimport com.twitter.cproxy.{thriftscala => cp}\nimport com.twitter.deferredrpc.client.DeferredThriftService\nimport com.twitter.deferredrpc.thrift.Datacenter\nimport com.twitter.deferredrpc.thrift.DeferredRPC\nimport com.twitter.deferredrpc.thrift.Target\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.builder.ClientBuilder\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient.MtlsThriftMuxClientSyntax\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetypieStaticEntitiesCache\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.shared_library.memcached_client.MemcachedClientBuilder\nimport com.twitter.servo.cache.ExpiringLruInProcessCache\nimport com.twitter.servo.cache.FinagleMemcache\nimport com.twitter.servo.cache.HotKeyMemcacheClient\nimport com.twitter.servo.cache.KeyTransformer\nimport com.twitter.servo.cache.KeyValueTransformingTtlCache\nimport com.twitter.servo.cache.ObservableTtlCache\nimport com.twitter.servo.cache.Serializer\nimport com.twitter.servo.cache.ThriftSerializer\nimport com.twitter.servo.cache.TtlCache\nimport com.twitter.servo.util.Gate\nimport com.twitter.timelinemixer.clients.rankedtweetcaching.CproxyTtlCacheWrapper\nimport com.twitter.tweetypie.{thriftscala => tp}\nimport javax.inject.Singleton\nimport org.apache.thrift.protocol.TCompactProtocol\n\nobject TweetypieStaticEntitiesCacheClientModule extends TwitterModule {\n\n  private val ScopeName = \"TweetypieStaticEntitiesMemcache\"\n  private val ProdDest = \"/s/cache/timelinescorer_tweet_core_data:twemcaches\"\n  private val ProdKrpcDest = \"/s/kafka-shared/krpc-server-cache\"\n  private val DevelKrpcDest = \"/srv#/staging/local/kafka-shared/krpc-server-custdevel\"\n\n  private val tweetsSerializer: Serializer[tp.Tweet] =\n    new ThriftSerializer[tp.Tweet](tp.Tweet, new TCompactProtocol.Factory())\n\n  private val keyTransformer: KeyTransformer[Long] = { tweetId => tweetId.toString }\n\n  @Provides\n  @Singleton\n  @Named(TweetypieStaticEntitiesCache)\n  def providesTweetypieStaticEntitiesCache(\n    statsReceiver: StatsReceiver,\n    serviceIdentifier: ServiceIdentifier\n  ): TtlCache[Long, tp.Tweet] = {\n    val memCacheClient = MemcachedClientBuilder.buildMemcachedClient(\n      destName = ProdDest,\n      numTries = 1,\n      numConnections = 1,\n      requestTimeout = 200.milliseconds,\n      globalTimeout = 200.milliseconds,\n      connectTimeout = 200.milliseconds,\n      acquisitionTimeout = 200.milliseconds,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver\n    )\n\n    val hotkeyCacheClient = new HotKeyMemcacheClient(\n      proxyClient = memCacheClient,\n      inProcessCache = new ExpiringLruInProcessCache(ttl = 15.minute, maximumSize = 75000),\n      statsReceiver = statsReceiver.scope(ScopeName).scope(\"inProcess\")\n    )\n\n    val krpcDest = serviceIdentifier.environment.toLowerCase match {\n      case \"prod\" => ProdKrpcDest\n      case _ => DevelKrpcDest\n    }\n\n    val deferredRpcClient = new DeferredRPC.FinagledClient(\n      ClientBuilder()\n        .name(\"deferredRpc\")\n        .dest(krpcDest)\n        .requestTimeout(200.milliseconds)\n        .hostConnectionLimit(3)\n        .stack(ThriftMux.client.withMutualTls(serviceIdentifier))\n        .build()\n    )\n\n    val cproxyClient = new cp.Cproxy.FinagledClient(\n      service = new DeferredThriftService(\n        deferredrpcService = deferredRpcClient,\n        target = Target(Datacenter.AllOthers, \"cproxy\"),\n        statsReceiver = statsReceiver.scope(\"deferredThriftService\")\n      ),\n      stats = statsReceiver.scope(\"cproxy\")\n    )\n\n    val baseCache: KeyValueTransformingTtlCache[Long, String, tp.Tweet, Array[Byte]] =\n      new KeyValueTransformingTtlCache(\n        underlyingCache = new FinagleMemcache(hotkeyCacheClient),\n        transformer = tweetsSerializer,\n        underlyingKey = keyTransformer\n      )\n\n    val observableTtlCache = ObservableTtlCache(\n      underlyingCache = baseCache,\n      statsReceiver = statsReceiver.scope(ScopeName),\n      windowSize = 1000,\n      name = ScopeName\n    )\n\n    new CproxyTtlCacheWrapper[Long, tp.Tweet](\n      underlyingCache = observableTtlCache,\n      cProxyCache = cproxyClient,\n      cType = cp.CachePoolType.HomeTweetCoreData,\n      valueSerializer = tweetsSerializer,\n      keyTransformer = keyTransformer,\n      replicationAvailable = Gate.True\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/TwhinEmbeddingsModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinRebuildTweetEmbeddingsStore\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinRebuildUserPositiveEmbeddingsStore\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinTweetEmbeddingsStore\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinUserNegativeEmbeddingsStore\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinUserPositiveEmbeddingsStore\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinVideoEmbeddingsStore\nimport com.twitter.home_mixer.store.TwhinEmbeddingsStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.{thriftscala => t}\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject TwhinEmbeddingsModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(TwhinTweetEmbeddingsStore)\n  def providesTwhinTweetEmbeddingFeaturesStore(\n    twhinEmbeddingsStore: TwhinEmbeddingsStore\n  ): ReadableStore[Long, t.TwhinTweetEmbedding] = twhinEmbeddingsStore.cachedTweetStore\n\n  @Provides\n  @Singleton\n  @Named(TwhinVideoEmbeddingsStore)\n  def providesTwhinVideoEmbeddingFeaturesStore(\n    twhinEmbeddingsStore: TwhinEmbeddingsStore\n  ): ReadableStore[Long, t.TwhinTweetEmbedding] = twhinEmbeddingsStore.cachedVideoStore\n\n  @Provides\n  @Singleton\n  @Named(TwhinUserPositiveEmbeddingsStore)\n  def providesTwhinUserPositiveEmbeddingFeaturesStore(\n    twhinEmbeddingsStore: TwhinEmbeddingsStore\n  ): ReadableStore[Long, t.TwhinTweetEmbedding] = twhinEmbeddingsStore.mhUserPositiveStore\n\n  @Provides\n  @Singleton\n  @Named(TwhinRebuildUserPositiveEmbeddingsStore)\n  def providesTwhinRebuildUserPositiveEmbeddingFeatureStore(\n    twhinEmbeddingStore: TwhinEmbeddingsStore\n  ): ReadableStore[(Long, Long), t.TwhinTweetEmbedding] =\n    twhinEmbeddingStore.mhRebuildUserPositiveStore\n\n  @Provides\n  @Singleton\n  @Named(TwhinUserNegativeEmbeddingsStore)\n  def providesTwhinUserNegativeEmbeddingFeaturesStore(\n    twhinEmbeddingsStore: TwhinEmbeddingsStore\n  ): ReadableStore[Long, t.TwhinTweetEmbedding] = twhinEmbeddingsStore.mhUserNegativeStore\n\n  @Provides\n  @Singleton\n  @Named(TwhinRebuildTweetEmbeddingsStore)\n  def providesTwhinRebuildTweetEmbeddingFeaturesStore(\n    twhinEmbeddingsStore: TwhinEmbeddingsStore\n  ): ReadableStore[(Long, Long), t.TwhinTweetEmbedding] =\n    twhinEmbeddingsStore.cachedTweetRebuildStore\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/UttTopicModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.escherbird.util.uttclient.CacheConfigV2\nimport com.twitter.escherbird.util.uttclient.CachedUttClientV2\nimport com.twitter.escherbird.util.uttclient.UttClientCacheConfigsV2\nimport com.twitter.escherbird.utt.strato.{thriftscala => utt}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithModerateTimeout\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ConfigRepoLocalPath\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ServiceLocal\nimport com.twitter.strato.client.Client\nimport com.twitter.topiclisting.TopicListing\nimport com.twitter.topiclisting.TopicListingBuilder\nimport com.twitter.topiclisting.clients.utt.UttClient\nimport com.twitter.topiclisting.utt.UttLocalization\nimport com.twitter.topiclisting.utt.UttLocalizationImpl\nimport com.twitter.tsp.stores.LocalizedUttRecommendableTopicsStore\nimport com.twitter.tsp.stores.TopicStore\nimport com.twitter.tsp.stores.UttTopicFilterStore\nimport com.twitter.util.JavaTimer\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject UttTopicModule extends TwitterModule {\n\n  private val StatsScope = \"UttTopic\"\n\n  private val optOutStratoStorePath: String = \"interests/optOutInterests\"\n\n  private val notInterestedInStorePath: String = \"interests/notInterestedTopicsGetter\"\n\n  @Provides\n  @Singleton\n  def providesTopicListing(\n    statsReceiver: StatsReceiver,\n    @Flag(ServiceLocal) isServiceLocal: Boolean,\n    @Flag(ConfigRepoLocalPath) localConfigRepoPath: String\n  ): TopicListing = {\n    val configSourceBasePath = if (isServiceLocal) Some(localConfigRepoPath) else None\n    new TopicListingBuilder(statsReceiver, configSourceBasePath).build\n  }\n\n  @Provides\n  @Singleton\n  def providesUttLocalization(\n    @Named(BatchedStratoClientWithModerateTimeout)\n    stratoClient: Client,\n    topicListing: TopicListing,\n    statsReceiver: StatsReceiver\n  ): UttLocalization = {\n    // used the same capacity as in currently used client from TSP\n    val defaultCacheConfigV2 = CacheConfigV2(capacity = 262143)\n    val uttClientCacheConfigsV2 = UttClientCacheConfigsV2(\n      getTaxonomyConfig = defaultCacheConfigV2,\n      getUttTaxonomyConfig = defaultCacheConfigV2,\n      getLeafIds = defaultCacheConfigV2,\n      getLeafUttEntities = defaultCacheConfigV2\n    )\n\n    lazy val cachedUttClientV2 = new CachedUttClientV2(\n      stratoClient = stratoClient,\n      env = utt.Environment.Prod,\n      cacheConfigs = uttClientCacheConfigsV2,\n      statsReceiver = statsReceiver.scope(\"CachedUttClient\")\n    )\n    val uttClient = new UttClient(cachedUttClientV2, statsReceiver)\n\n    new UttLocalizationImpl(topicListing, uttClient, statsReceiver.scope(StatsScope))\n  }\n\n  @Provides\n  @Singleton\n  def providesUttTopicFilterStore(\n    @Named(BatchedStratoClientWithModerateTimeout)\n    stratoClient: Client,\n    uttLocalization: UttLocalization,\n    topicListing: TopicListing,\n    statsReceiver: StatsReceiver\n  ): UttTopicFilterStore = {\n    val userOptOutTopicsStore =\n      TopicStore.userOptOutTopicStore(stratoClient, optOutStratoStorePath)(\n        statsReceiver.scope(\"interests_opt_out_store\"))\n    val explicitFollowingTopicsStore =\n      TopicStore.explicitFollowingTopicStore(stratoClient)(\n        statsReceiver.scope(\"explicit_following_interests_store\"))\n    val userNotInterestedInTopicsStore =\n      TopicStore.notInterestedInTopicsStore(stratoClient, notInterestedInStorePath)(\n        statsReceiver.scope(\"not_interested_in_store\"))\n    val localizedUttRecommendableTopicsStore = new LocalizedUttRecommendableTopicsStore(\n      uttLocalization)\n    val timer = new JavaTimer(isDaemon = true)\n\n    new UttTopicFilterStore(\n      topicListing = topicListing,\n      userOptOutTopicsStore = userOptOutTopicsStore,\n      explicitFollowingTopicsStore = explicitFollowingTopicsStore,\n      notInterestedTopicsStore = userNotInterestedInTopicsStore,\n      localizedUttRecommendableTopicsStore = localizedUttRecommendableTopicsStore,\n      timer = timer,\n      stats = statsReceiver.scope(\"UttTopicFilterStore\")\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/module/VideoEmbeddingModule.scala",
    "content": "package com.twitter.home_mixer.module\n\nimport com.google.inject.Provides\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.VideoEmbeddingMHStore\nimport com.twitter.home_mixer.store.VideoEmbeddingMHStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.media_understanding.video_summary.thriftscala.VideoEmbedding\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject VideoEmbeddingModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  @Named(VideoEmbeddingMHStore)\n  def providesVideoEmbeddingMHStore(\n    videoEmbeddingMHStore: VideoEmbeddingMHStore\n  ): ReadableStore[Long, VideoEmbedding] = {\n    videoEmbeddingMHStore.videoEmbeddingMHStore\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/param/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param/decider\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/param/GlobalParamConfigModule.scala",
    "content": "package com.twitter.home_mixer.param\n\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.core.functional_component.configapi.registry.GlobalParamConfig\n\nobject GlobalParamConfigModule extends TwitterModule {\n  override def configure(): Unit = {\n    bind[GlobalParamConfig].to[HomeGlobalParamConfig]\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/param/HomeGlobalParamConfig.scala",
    "content": "package com.twitter.home_mixer.param\n\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableClipEmbeddingFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableClipEmbeddingMediaUnderstandingFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableDedupClusterId88FeatureHydrationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableDedupClusterIdFeatureHydrationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableGeoduckAuthorLocationHydatorParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableGrokVideoMetadataFeatureHydrationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableImmersiveClientActionsClipEmbeddingQueryFeatureHydrationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableImmersiveClientActionsQueryFeatureHydrationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableLargeEmbeddingsFeatureHydrationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableOnPremRealGraphQueryFeatures\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableRealGraphQueryFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableRealGraphViewerRelatedUsersFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableSimclustersSparseTweetFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTransformerPostEmbeddingJointBlueFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTweetLanguageFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTweetTextTokensEmbeddingFeatureScribingParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTweetVideoAggregatedWatchTimeFeatureScribingParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTweetypieContentFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTweetypieContentMediaEntityFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinRebuildTweetFeaturesOnlineParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinRebuildUserEngagementFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinRebuildUserPositiveFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinTweetFeaturesOnlineParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinUserNegativeFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinUserPositiveFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinVideoFeaturesOnlineParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinVideoFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableUserFavAvgTextEmbeddingsQueryFeatureParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableUserHistoryTransformerJointBlueEmbeddingFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableViewCountFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring\nimport com.twitter.home_mixer.param.HomeGlobalParams._\nimport com.twitter.product_mixer.core.functional_component.configapi.registry.GlobalParamConfig\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Register Params that do not relate to a specific product. See GlobalParamConfig -> ParamConfig\n * for hooks to register Params based on type.\n */\n@Singleton\nclass HomeGlobalParamConfig @Inject() () extends GlobalParamConfig {\n\n  override val booleanDeciderOverrides = Seq(\n    EnableServedCandidateFeatureKeysKafkaPublishingParam,\n    FeatureHydration.EnableSimClustersSimilarityFeaturesDeciderParam,\n    FeatureHydration.EnableTweetypieContentFeaturesDeciderParam,\n    FeatureHydration.EnableVideoSummaryEmbeddingFeatureDeciderParam,\n    FeatureHydration.EnableVideoClipEmbeddingFeatureHydrationDeciderParam,\n    FeatureHydration.EnableScoredVideoTweetsUserHistoryEventsQueryFeatureHydrationDeciderParam,\n    FeatureHydration.EnableVideoClipEmbeddingMediaUnderstandingFeatureHydrationDeciderParam,\n    EnableCommonFeaturesDataRecordCopyDuringPldrConversionParam,\n    Scoring.UseSecondaryNaviClusterParam,\n    Scoring.UseGPUNaviClusterTestUsersParam\n  )\n\n  override val boundedDoubleDeciderOverrides = Seq(Scoring.NaviGPUBatchSizeParam)\n\n  override val stringFSOverrides =\n    Seq(\n      Scoring.ModelNameParam,\n      Scoring.ModelIdParam,\n      Scoring.ProdModelIdParam,\n      PostFeedbackPromptTitleParam,\n      PostFeedbackPromptPositiveParam,\n      PostFeedbackPromptNegativeParam,\n      PostFeedbackPromptNeutralParam,\n    )\n\n  override val booleanFSOverrides = Seq(\n    AdsDisableInjectionBasedOnUserRoleParam,\n    EnableTweetEntityServiceMigrationParam,\n    EnableTweetEntityServiceVisibilityMigrationParam,\n    EnableAdvertiserBrandSafetySettingsFeatureHydratorParam,\n    EnableSSPAdsBrandSafetySettingsFeatureHydratorParam,\n    EnableDebugString,\n    EnablePersistenceDebug,\n    EnableLargeEmbeddingsFeatureHydrationParam,\n    EnableNewTweetsPillAvatarsParam,\n    EnableOnPremRealGraphQueryFeatures,\n    EnableRealGraphQueryFeaturesParam,\n    EnableRealGraphViewerRelatedUsersFeaturesParam,\n    EnableScribeServedCandidatesParam,\n    EnableSendScoresToClient,\n    EnableSimclustersSparseTweetFeaturesParam,\n    EnableCommunitiesContextParam,\n    EnableSocialContextParam,\n    EnableTwhinRebuildUserEngagementFeaturesParam,\n    EnableTwhinRebuildUserPositiveFeaturesParam,\n    EnableTwhinVideoFeaturesParam,\n    EnableTwhinUserNegativeFeaturesParam,\n    EnableTwhinUserPositiveFeaturesParam,\n    EnableTwhinVideoFeaturesOnlineParam,\n    EnableTwhinTweetFeaturesOnlineParam,\n    EnableTwhinRebuildTweetFeaturesOnlineParam,\n    EnableBasketballContextFeatureHydratorParam,\n    EnablePostContextFeatureHydratorParam,\n    EnableTransformerPostEmbeddingJointBlueFeaturesParam,\n    EnableClipEmbeddingFeaturesParam,\n    EnableClipEmbeddingMediaUnderstandingFeaturesParam,\n    EnableUserHistoryTransformerJointBlueEmbeddingFeaturesParam,\n    EnableLandingPage,\n    EnableTenSecondsLogicForVQV,\n    EnableImmersiveVQV,\n    EnableExploreSimclustersLandingPage,\n    EnableTweetLanguageFeaturesParam,\n    EnableTweetypieContentFeaturesParam,\n    EnableTweetypieContentMediaEntityFeaturesParam,\n    EnableViewCountFeaturesParam,\n    ListMandarinTweetsParams.ListMandarinTweetsEnable,\n    Scoring.EnableNoNegHeuristicParam,\n    Scoring.EnableNegSectionRankingParam,\n    Scoring.EnableBinarySchemeForVQVParam,\n    Scoring.EnableBinarySchemeForDwellParam,\n    Scoring.EnableDwellOrVQVParam,\n    Scoring.UseRealtimeNaviClusterParam,\n    Scoring.UseGPUNaviClusterParam,\n    Scoring.RequestNormalizedScoresParam,\n    Scoring.AddNoiseInWeightsPerLabel,\n    Scoring.EnableDailyFrozenNoisyWeights,\n    Scoring.TwhinDiversityRescoringParam,\n    Scoring.CategoryDiversityRescoringParam,\n    Scoring.UseVideoNaviClusterParam,\n    Scoring.NormalizedNegativeHead,\n    Scoring.UseWeightForNegHeadParam,\n    Scoring.ConstantNegativeHead,\n    Scoring.UseProdInPhoenixParams.EnableProdDwellForPhoenixParam,\n    Scoring.UseProdInPhoenixParams.EnableProdFavForPhoenixParam,\n    Scoring.UseProdInPhoenixParams.EnableProdGoodClickV1ForPhoenixParam,\n    Scoring.UseProdInPhoenixParams.EnableProdGoodClickV2ForPhoenixParam,\n    Scoring.UseProdInPhoenixParams.EnableProdNegForPhoenixParam,\n    Scoring.UseProdInPhoenixParams.EnableProdProfileClickForPhoenixParam,\n    Scoring.UseProdInPhoenixParams.EnableProdReplyForPhoenixParam,\n    Scoring.UseProdInPhoenixParams.EnableProdRetweetForPhoenixParam,\n    Scoring.UseProdInPhoenixParams.EnableProdShareForPhoenixParam,\n    Scoring.UseProdInPhoenixParams.EnableProdVQVForPhoenixParam,\n    Scoring.UseProdInPhoenixParams.EnableProdOpenLinkForPhoenixParam,\n    Scoring.UseProdInPhoenixParams.EnableProdScreenshotForPhoenixParam,\n    Scoring.UseProdInPhoenixParams.EnableProdBookmarkForPhoenixParam,\n    EnableUserFavAvgTextEmbeddingsQueryFeatureParam,\n    EnableTweetTextTokensEmbeddingFeatureScribingParam,\n    EnableTweetVideoAggregatedWatchTimeFeatureScribingParam,\n    EnableImmersiveClientActionsQueryFeatureHydrationParam,\n    EnableImmersiveClientActionsClipEmbeddingQueryFeatureHydrationParam,\n    EnableGrokVideoMetadataFeatureHydrationParam,\n    EnableDedupClusterIdFeatureHydrationParam,\n    EnableDedupClusterId88FeatureHydrationParam,\n    EnableServedFilterAllRequests,\n    EnablePinnedTweetsCarouselParam,\n    EnablePostDetailsNegativeFeedbackParam,\n    EnablePostFeedbackParam,\n    EnablePostFollowupParam,\n    EnableSlopFilter,\n    EnableNsfwFilter,\n    EnableSoftNsfwFilter,\n    EnableGrokGoreFilter,\n    EnableMinVideoDurationFilter,\n    EnableMaxVideoDurationFilter,\n    EnableGrokSpamFilter,\n    EnableGrokViolentFilter,\n    EnableClusterBasedDedupFilter,\n    EnableCountryFilter,\n    EnableRegionFilter,\n    EnableHasMultipleMediaFilter,\n    EnableClusterBased88DedupFilter,\n    EnableNoClusterFilter,\n    EnableSlopFilterLowSignalUsers,\n    EnableSlopFilterEligibleUserStateParam,\n    EnableGrokAnnotations,\n    EnableTopicBasedRealTimeAggregateFeatureHydratorParam,\n    EnableTopicCountryBasedRealTimeAggregateFeatureHydratorParam,\n    EnableTopicEdgeAggregateFeatureHydratorParam,\n    EnableAdditionalChildFeedbackParam,\n    EnableGeoduckAuthorLocationHydatorParam,\n    EnableTweetRTAMhOnlyParam,\n    EnableTweetRTAMhFallbackParam,\n    EnableTweetCountryRTAMhOnlyParam,\n    EnableTweetCountryRTAMhFallbackParam,\n    EnableUserRTAMhOnlyParam,\n    EnableUserRTAMhFallbackParam,\n    EnableUserAuthorRTAMhOnlyParam,\n    EnableUserAuthorRTAMhFallbackParam,\n    EnableBlockMuteReportChildFeedbackParam,\n    EnablePhoenixScorerParam,\n    EnableUserActionsShadowScribeParam,\n  )\n\n  override val boundedIntFSOverrides = Seq(\n    MaxNumberReplaceInstructionsParam,\n    TimelinesPersistenceStoreMaxEntriesPerClient,\n    ExcludeServedTweetIdsNumberParam,\n    IsSelectedByHeavyRankerCountParam,\n    SlopMinFollowers,\n    UserActionsMaxCount,\n    PhoenixTimeoutInMsParam,\n  )\n\n  override val boundedLongFSOverrides =\n    Seq(\n      DedupHistoricalEventsTimeWindowParam,\n      MinVideoDurationThresholdParam,\n      MaxVideoDurationThresholdParam)\n  override val boundedDoubleFSOverrides = Seq(\n    SlopMaxScore,\n    PostFeedbackThresholdParam,\n    PostFollowupThresholdParam,\n    // Model Weights\n    Scoring.ModelWeights.BookmarkParam,\n    Scoring.ModelWeights.Dwell0Param,\n    Scoring.ModelWeights.Dwell1Param,\n    Scoring.ModelWeights.Dwell2Param,\n    Scoring.ModelWeights.Dwell3Param,\n    Scoring.ModelWeights.Dwell4Param,\n    Scoring.ModelWeights.DwellParam,\n    Scoring.ModelWeights.FavParam,\n    Scoring.ModelWeights.GoodClickParam,\n    Scoring.ModelWeights.GoodClickV1Param,\n    Scoring.ModelWeights.GoodClickV2Param,\n    Scoring.ModelWeights.GoodProfileClickParam,\n    Scoring.ModelWeights.NegativeFeedbackV2Param,\n    Scoring.ModelWeights.OpenLinkParam,\n    Scoring.ModelWeights.ProfileDwelledParam,\n    Scoring.ModelWeights.ReplyEngagedByAuthorParam,\n    Scoring.ModelWeights.ReplyParam,\n    Scoring.ModelWeights.ReportParam,\n    Scoring.ModelWeights.RetweetParam,\n    Scoring.ModelWeights.ScreenshotParam,\n    Scoring.ModelWeights.ShareMenuClickParam,\n    Scoring.ModelWeights.ShareParam,\n    Scoring.ModelWeights.StrongNegativeFeedbackParam,\n    Scoring.ModelWeights.TweetDetailDwellParam,\n    Scoring.ModelWeights.VideoPlayback50Param,\n    Scoring.ModelWeights.VideoQualityViewImmersiveParam,\n    Scoring.ModelWeights.VideoQualityViewParam,\n    Scoring.ModelWeights.VideoQualityWatchParam,\n    Scoring.ModelWeights.VideoWatchTimeMsParam,\n    Scoring.ModelWeights.WeakNegativeFeedbackParam,\n    // Model Biases\n    Scoring.ModelBiases.VideoQualityViewParam,\n    Scoring.ModelBiases.VideoQualityViewImmersiveParam,\n    Scoring.ModelBiases.VideoQualityWatchParam,\n    Scoring.NoisyWeightAlphaParam,\n    Scoring.NoisyWeightBetaParam,\n    Scoring.NegativeScoreConstantFilterThresholdParam,\n    Scoring.NegativeScoreNormFilterThresholdParam,\n    Scoring.RequestRankDecayFactorParam,\n    Scoring.ScoreThresholdForVQVParam,\n    Scoring.ScoreThresholdForDwellParam,\n    Scoring.BinarySchemeConstantForVQVParam,\n    Scoring.ImpressedMediaClusterBasedRescoringParam,\n    // ModelDebiases\n    Scoring.ModelDebiases.FavParam,\n    Scoring.ModelDebiases.ReplyParam,\n    Scoring.ModelDebiases.RetweetParam,\n    Scoring.ModelDebiases.GoodClickV1Param,\n    Scoring.ModelDebiases.GoodClickV2Param,\n    Scoring.ModelDebiases.GoodProfileClickParam,\n    Scoring.ModelDebiases.ReplyEngagedByAuthorParam,\n    Scoring.ModelDebiases.VideoQualityViewParam,\n    Scoring.ModelDebiases.VideoQualityViewImmersiveParam,\n    Scoring.ModelDebiases.NegativeFeedbackV2Param,\n    Scoring.ModelDebiases.BookmarkParam,\n    Scoring.ModelDebiases.ShareParam,\n    Scoring.ModelDebiases.DwellParam,\n    Scoring.ModelDebiases.VideoQualityWatchParam,\n    Scoring.ModelDebiases.VideoWatchTimeMsParam\n  )\n\n  override val longSetFSOverrides = Seq(\n    RateLimitTestIdsParam,\n    BasketballTeamAccountIdsParam,\n    Scoring.AuthorListForDataCollectionParam\n  )\n\n  override val boundedDurationFSOverrides = Seq(\n    FeedbackFatigueFilteringDurationParam,\n    ExcludeServedTweetIdsDurationParam,\n    ExcludeServedAuthorIdsDurationParam\n  )\n\n  override val enumFSOverrides = Seq(\n    PhoenixInferenceClusterParam\n  )\n\n  override val longSeqFSOverrides = Seq(\n    ListMandarinTweetsParams.ListMandarinTweetsLists\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/param/HomeGlobalParams.scala",
    "content": "package com.twitter.home_mixer.param\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.param.decider.DeciderKey\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.decider.BooleanDeciderParam\nimport com.twitter.timelines.configapi.decider.DeciderBoundedParam\nimport com.twitter.util.Duration\n\n/**\n * Instantiate Params that do not relate to a specific product.\n *\n * @see [[com.twitter.product_mixer.core.product.ProductParamConfig.supportedClientFSName]]\n */\nobject HomeGlobalParams {\n\n  /**\n   * This param is used to disable ads injection for timelines served by home-mixer.\n   * It is currently used to maintain user-role based no-ads lists for automation accounts,\n   * and should NOT be used for other purposes.\n   */\n  object AdsDisableInjectionBasedOnUserRoleParam\n      extends FSParam(\n        name = \"home_mixer_ads_disable_injection_based_on_user_role\",\n        default = false\n      )\n\n  object EnableTweetEntityServiceMigrationParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_tweet_entity_service_migration\",\n        default = false\n      )\n\n  object EnableTweetEntityServiceVisibilityMigrationParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_tweet_entity_service_visibility_migration\",\n        default = false\n      )\n\n  object EnableSendScoresToClient\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_send_scores_to_client\",\n        default = false\n      )\n\n  object EnableDebugString\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_debug_string\",\n        default = false\n      )\n\n  object EnablePersistenceDebug\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_persistence_debug\",\n        default = false\n      )\n\n  object MaxNumberReplaceInstructionsParam\n      extends FSBoundedParam[Int](\n        name = \"home_mixer_max_number_replace_instructions\",\n        default = 10,\n        min = 1,\n        max = 20\n      )\n\n  object TimelinesPersistenceStoreMaxEntriesPerClient\n      extends FSBoundedParam[Int](\n        name = \"home_mixer_timelines_persistence_store_max_entries_per_client\",\n        default = 1800,\n        min = 500,\n        max = 5000\n      )\n\n  object EnableNewTweetsPillAvatarsParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_new_tweets_pill_avatars\",\n        default = true\n      )\n\n  object EnableSocialContextParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_social_context\",\n        default = false\n      )\n\n  object EnableCommunitiesContextParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_communities_context\",\n        default = true\n      )\n\n  object EnableAdvertiserBrandSafetySettingsFeatureHydratorParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_advertiser_brand_safety_settings_feature_hydrator\",\n        default = true\n      )\n\n  object EnableBasketballContextFeatureHydratorParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_basketball_context_feature_hydrator\",\n        default = false\n      )\n\n  object EnablePostContextFeatureHydratorParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_post_context_feature_hydrator\",\n        default = false\n      )\n\n  object BasketballTeamAccountIdsParam\n      extends FSParam[Set[Long]](\n        name = \"home_mixer_basketball_team_account_ids\",\n        default = Set()\n      )\n\n  object EnableSSPAdsBrandSafetySettingsFeatureHydratorParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_ssp_ads_brand_safety_settings_feature_hydrator\",\n        default = true\n      )\n\n  object ExcludeServedTweetIdsNumberParam\n      extends FSBoundedParam[Int](\n        name = \"home_mixer_exclude_served_tweet_ids_number\",\n        default = 100,\n        min = 0,\n        max = 100\n      )\n\n  object ExcludeServedTweetIdsDurationParam\n      extends FSBoundedParam[Duration](\n        \"home_mixer_exclude_served_tweet_ids_in_minutes\",\n        default = 10.minutes,\n        min = 1.minute,\n        max = 60.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object ExcludeServedAuthorIdsDurationParam\n      extends FSBoundedParam[Duration](\n        \"home_mixer_exclude_served_author_ids_in_minutes\",\n        default = 60.minutes,\n        min = 1.minute,\n        max = 60.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object EnableServedFilterAllRequests\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_served_filter_all_requests\",\n        default = false\n      )\n\n  object EnableScribeServedCandidatesParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_served_tweets_enable_scribing\",\n        default = false\n      )\n\n  object EnableServedCandidateFeatureKeysKafkaPublishingParam\n      extends BooleanDeciderParam(\n        decider = DeciderKey.EnableServedCandidateFeatureKeysKafkaPublishing)\n\n  object RateLimitTestIdsParam\n      extends FSParam[Set[Long]](\n        name = \"home_mixer_rate_limit_test_ids\",\n        default = Set.empty\n      )\n\n  object IsSelectedByHeavyRankerCountParam\n      extends FSBoundedParam[Int](\n        name = \"home_mixer_is_selected_by_heavy_ranker_count\",\n        default = 100,\n        min = 0,\n        max = 2000\n      )\n\n  object EnableAdditionalChildFeedbackParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_additional_child_feedback\",\n        default = false\n      )\n\n  object EnableBlockMuteReportChildFeedbackParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_block_mute_report_child_feedback\",\n        default = false\n      )\n\n  object ListMandarinTweetsParams {\n    object ListMandarinTweetsEnable\n        extends FSParam[Boolean](\n          name = \"home_mixer_mandarin_list_tweets_enabled\",\n          default = false\n        )\n\n    object ListMandarinTweetsLists\n        extends FSParam[Seq[Long]](\n          name = \"home_mixer_mandarin_tweets_lists\",\n          default = Seq.empty\n        )\n  }\n\n  object FeatureHydration {\n    object EnableLargeEmbeddingsFeatureHydrationParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_large_embeddings\",\n          default = false\n        )\n\n    object EnableSimClustersSimilarityFeaturesDeciderParam\n        extends BooleanDeciderParam(\n          decider = DeciderKey.EnableSimClustersSimilarityFeatureHydration\n        )\n\n    object EnableOnPremRealGraphQueryFeatures\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_on_prem_real_graph_query_features\",\n          default = false\n        )\n\n    object EnableRealGraphQueryFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_real_graph_query_features\",\n          default = false\n        )\n\n    object EnableRealGraphViewerRelatedUsersFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_real_graph_viewer_related_users_features\",\n          default = false\n        )\n\n    object EnableSimclustersSparseTweetFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_simclusters_sparse_tweet_features\",\n          default = false\n        )\n\n    object EnableTwhinUserPositiveFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_twhin_user_positive_features\",\n          default = false\n        )\n\n    object EnableTwhinVideoFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_twhin_video_features\",\n          default = false\n        )\n\n    object EnableTwhinUserNegativeFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_twhin_user_negative_features\",\n          default = false\n        )\n\n    object EnableTwhinVideoFeaturesOnlineParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_twhin_video_online_features\",\n          default = false\n        )\n\n    object EnableTwhinRebuildUserEngagementFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_twhin_rebuild_user_engagement_features\",\n          default = false\n        )\n\n    object EnableTwhinRebuildUserPositiveFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_twhin_rebuild_user_positive_features\",\n          default = false\n        )\n\n    object EnableClipEmbeddingFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_clip_embedding_features\",\n          default = false\n        )\n\n    object EnableClipEmbeddingMediaUnderstandingFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_clip_embedding_media_understanding_features\",\n          default = false\n        )\n\n    object EnableUserHistoryTransformerJointBlueEmbeddingFeaturesParam\n        extends FSParam[Boolean](\n          name =\n            \"home_mixer_feature_hydration_enable_user_history_transformer_joint_blue_embedding_features\",\n          default = false\n        )\n\n    object EnableTweetLanguageFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_tweet_language_features\",\n          default = false\n        )\n\n    object EnableTwhinTweetFeaturesOnlineParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_twhin_tweet_online_features\",\n          default = false\n        )\n\n    object EnableTwhinRebuildTweetFeaturesOnlineParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_twhin_rebuild_tweet_online_features\",\n          default = false\n        )\n\n    object EnableTransformerPostEmbeddingJointBlueFeaturesParam\n        extends FSParam[Boolean](\n          name =\n            \"home_mixer_feature_hydration_enable_transformer_post_embedding_features_joint_blue\",\n          default = false\n        )\n\n    object EnableTweetypieContentFeaturesDeciderParam\n        extends BooleanDeciderParam(\n          decider = DeciderKey.EnableTweetypieContentFeatures\n        )\n\n    object EnableTweetypieContentFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_tweetypie_content_features\",\n          default = true\n        )\n\n    object EnableTweetypieContentMediaEntityFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_tweetypie_content_media_entity_features\",\n          default = true\n        )\n\n    object EnableUserFavAvgTextEmbeddingsQueryFeatureParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_user_fav_avg_text_embeddings_query_feature\",\n          default = false\n        )\n\n    object EnableTweetTextTokensEmbeddingFeatureScribingParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_tweet_text_tokens_embedding_feature_scribing\",\n          default = false\n        )\n\n    object EnableTweetVideoAggregatedWatchTimeFeatureScribingParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_tweet_video_aggregated_watch_time\",\n          default = false\n        )\n\n    object EnableImmersiveClientActionsQueryFeatureHydrationParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_immersive_client_actions\",\n          default = false\n        )\n\n    object EnableImmersiveClientActionsClipEmbeddingQueryFeatureHydrationParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_immersive_client_actions_clip_embedding\",\n          default = false\n        )\n\n    object EnableGrokVideoMetadataFeatureHydrationParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_grok_video_metadata\",\n          default = false\n        )\n\n    object EnableDedupClusterIdFeatureHydrationParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_dedup_cluster_id\",\n          default = false\n        )\n\n    object EnableDedupClusterId88FeatureHydrationParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_dedup_cluster_id_88\",\n          default = false\n        )\n\n    object EnableGeoduckAuthorLocationHydatorParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_geoduck_author_location_hydrator\",\n          default = false\n        )\n\n    object EnableViewCountFeaturesParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_feature_hydration_enable_view_count_features\",\n          default = false\n        )\n\n    object EnableVideoSummaryEmbeddingFeatureDeciderParam\n        extends BooleanDeciderParam(\n          decider = DeciderKey.EnableVideoSummaryEmbeddingHydration\n        )\n\n    object EnableVideoClipEmbeddingFeatureHydrationDeciderParam\n        extends BooleanDeciderParam(\n          decider = DeciderKey.EnableVideoClipEmbeddingHydration\n        )\n\n    object EnableScoredVideoTweetsUserHistoryEventsQueryFeatureHydrationDeciderParam\n        extends BooleanDeciderParam(\n          decider =\n            DeciderKey.EnableScoredVideoTweetsUserHistoryEventsQueryFeatureHydrationDeciderParam\n        )\n\n    object EnableVideoClipEmbeddingMediaUnderstandingFeatureHydrationDeciderParam\n        extends BooleanDeciderParam(\n          decider = DeciderKey.EnableVideoClipEmbeddingMediaUnderstandingHydration\n        )\n  }\n\n  object Scoring {\n\n    object AuthorListForDataCollectionParam\n        extends FSParam[Set[Long]](\n          name = \"home_mixer_author_list_for_data_collection\",\n          default = Set.empty[Long]\n        )\n\n    object ModelNameParam\n        extends FSParam[String](\n          name = \"home_mixer_model_name\",\n          default = \"\"\n        )\n\n    object ImpressedMediaClusterBasedRescoringParam\n        extends FSBoundedParam[Double](\n          name = \"home_mixer_impressed_media_cluster_based_rescoring\",\n          default = 0.0,\n          min = 0.0,\n          max = 0.2\n        )\n\n    object ImpressedImageClusterBasedRescoringParam\n        extends FSBoundedParam[Double](\n          name = \"home_mixer_impressed_image_cluster_based_rescoring\",\n          default = 0.0,\n          min = 0.0,\n          max = 1.0\n        )\n\n    object ModelIdParam\n        extends FSParam[String](\n          name = \"home_mixer_model_id\",\n          default = \"Home\"\n        )\n\n    object ProdModelIdParam\n        extends FSParam[String](\n          name = \"home_mixer_model_prod_model_id\",\n          default = \"Home\"\n        )\n\n    object UseRealtimeNaviClusterParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_model_use_realtime_navi_cluster\",\n          default = false\n        )\n\n    object UseGPUNaviClusterParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_model_use_gpu_navi_cluster\",\n          default = false\n        )\n\n    object UseSecondaryNaviClusterParam\n        extends BooleanDeciderParam(decider = DeciderKey.EnableSecondaryNaviRecapCluster)\n\n    object UseGPUNaviClusterTestUsersParam\n        extends BooleanDeciderParam(decider = DeciderKey.EnableGPUNaviRecapClusterTestUsers)\n\n    object UseVideoNaviClusterParam\n        extends FSParam[Boolean](\"home_mixer_model_use_video_navi_cluster\", false)\n\n    object NaviGPUBatchSizeParam\n        extends DeciderBoundedParam[Double](\n          decider = DeciderKey.NaviGPUClusterRequestBatchSize,\n          default = 1800.0,\n          min = 0.0,\n          max = 10000.0\n        )\n\n    object AddNoiseInWeightsPerLabel\n        extends FSParam[Boolean](\n          name = \"home_mixer_add_noise_in_weights_per_label\",\n          default = false\n        )\n\n    object EnableDailyFrozenNoisyWeights\n        extends FSParam[Boolean](\n          name = \"home_mixer_enable_daily_frozen_weights\",\n          default = false\n        )\n\n    object NoisyWeightAlphaParam\n        extends FSBoundedParam[Double](\n          name = \"home_mixer_noisy_weight_alpha_param\",\n          default = 2,\n          min = 0.0,\n          max = 10.0\n        )\n\n    object NoisyWeightBetaParam\n        extends FSBoundedParam[Double](\n          name = \"home_mixer_noisy_weight_beta_param\",\n          default = 2,\n          min = 0.0,\n          max = 10.0\n        )\n    object NegativeScoreConstantFilterThresholdParam\n        extends FSBoundedParam[Double](\n          name = \"home_mixer_negative_score_constant_filter_threshold\",\n          default = 1e-3,\n          min = 0,\n          max = 1\n        )\n\n    object NegativeScoreNormFilterThresholdParam\n        extends FSBoundedParam[Double](\n          name = \"home_mixer_negative_score_norm_filter_threshold\",\n          default = 0.15,\n          min = 0,\n          max = 1\n        )\n\n    object RequestNormalizedScoresParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_request_normalized_scores\",\n          default = false\n        )\n\n    object NormalizedNegativeHead\n        extends FSParam[Boolean](\n          name = \"home_mixer_normalized_negative_head\",\n          default = false\n        )\n\n    object UseWeightForNegHeadParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_use_weight_for_neg_head\",\n          default = false\n        )\n\n    object ConstantNegativeHead\n        extends FSParam[Boolean](\n          name = \"home_mixer_constant_negative_head\",\n          default = false\n        )\n\n    object EnableNoNegHeuristicParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_no_neg_heuristic\",\n          default = false\n        )\n\n    object EnableNegSectionRankingParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_neg_section_ranking\",\n          default = false\n        )\n\n    object RequestRankDecayFactorParam\n        extends FSBoundedParam[Double](\n          name = \"home_mixer_request_rank_decay_factor\",\n          default = 0.95,\n          min = 0,\n          max = 1\n        )\n\n    object ScoreThresholdForVQVParam\n        extends FSBoundedParam[Double](\n          name = \"home_mixer_score_threshold_for_vqv\",\n          default = 0.0,\n          min = 0.0,\n          max = 1.0\n        )\n\n    object ScoreThresholdForDwellParam\n        extends FSBoundedParam[Double](\n          name = \"home_mixer_score_threshold_for_dwell\",\n          default = 0.0,\n          min = 0.0,\n          max = 1.0\n        )\n\n    object EnableBinarySchemeForVQVParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_enable_binary_scheme_for_vqv\",\n          default = false\n        )\n\n    object BinarySchemeConstantForVQVParam\n        extends FSBoundedParam[Double](\n          name = \"home_mixer_constant_binary_scheme_for_vqv\",\n          default = 0.0,\n          min = 0.0,\n          max = 1.0\n        )\n\n    object EnableBinarySchemeForDwellParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_enable_binary_scheme_for_dwell\",\n          default = false\n        )\n\n    object EnableDwellOrVQVParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_enable_dwell_or_video_watch_time\",\n          default = false\n        )\n\n    object TwhinDiversityRescoringParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_twhin_diversity_rescoring\",\n          default = false\n        )\n\n    object CategoryDiversityRescoringParam\n        extends FSParam[Boolean](\n          name = \"home_mixer_category_diversity_rescoring\",\n          default = false\n        )\n\n    object ModelBiases {\n      object VideoQualityViewParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_bias_video_quality_viewed\",\n            default = 0.0,\n            min = 0.0,\n            max = 100.0\n          )\n\n      object VideoQualityViewImmersiveParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_bias_video_quality_viewed_immersive\",\n            default = 0.0,\n            min = 0.0,\n            max = 100.0\n          )\n\n      object VideoQualityWatchParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_bias_video_quality_watched\",\n            default = 0.0,\n            min = 0.0,\n            max = 100.0\n          )\n    }\n\n    object ModelDebiases {\n\n      object FavParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_fav\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object RetweetParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_retweet\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object ReplyParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_reply\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object DwellParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_dwell\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object GoodProfileClickParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_good_profile_click\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object VideoWatchTimeMsParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_video_watch_time_ms\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object VideoQualityViewParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_video_quality_viewed\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object VideoQualityViewImmersiveParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_video_quality_viewed_immersive\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object ReplyEngagedByAuthorParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_reply_engaged_by_author\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object GoodClickV1Param\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_good_click_v1\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object GoodClickV2Param\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_good_click_v2\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object BookmarkParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_bookmark\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object ShareParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_share\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object NegativeFeedbackV2Param\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_negative_feedback_v2\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object VideoQualityWatchParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_debias_video_quality_watched\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n    }\n\n    object ModelWeights {\n\n      object FavParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_fav\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object RetweetParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_retweet\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object ReplyParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_reply\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object GoodProfileClickParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_good_profile_click\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object VideoPlayback50Param\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_video_playback50\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object VideoQualityViewParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_video_quality_viewed\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object VideoQualityViewImmersiveParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_video_quality_viewed_immersive\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object ReplyEngagedByAuthorParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_reply_engaged_by_author\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object GoodClickParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_good_click\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object GoodClickV1Param\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_good_click_v1\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object GoodClickV2Param\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_good_click_v2\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object TweetDetailDwellParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_tweet_detail_dwell\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object ProfileDwelledParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_profile_dwelled\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object BookmarkParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_bookmark\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object ShareParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_share\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object ShareMenuClickParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_share_menu_click\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object NegativeFeedbackV2Param\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_negative_feedback_v2\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object ReportParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_report\",\n            default = 0.0,\n            min = -20000.0,\n            max = 0.0\n          )\n\n      object WeakNegativeFeedbackParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_weak_negative_feedback\",\n            default = 0.0,\n            min = -1000.0,\n            max = 0.0\n          )\n\n      object StrongNegativeFeedbackParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_strong_negative_feedback\",\n            default = 0.0,\n            min = -1000.0,\n            max = 0.0\n          )\n\n      object DwellParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_dwell\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object OpenLinkParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_open_link\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object ScreenshotParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_screenshot\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      object VideoWatchTimeMsParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_video_watch_time_ms\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n\n      // Categorical Dwell Params\n      object Dwell0Param\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_dwell_0\",\n            default = 0.0,\n            min = 0.0,\n            max = 1000.0\n          )\n\n      object Dwell1Param\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_dwell_1\",\n            default = 0.0,\n            min = 0.0,\n            max = 1000.0\n          )\n\n      object Dwell2Param\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_dwell_2\",\n            default = 0.0,\n            min = 0.0,\n            max = 1000.0\n          )\n\n      object Dwell3Param\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_dwell_3\",\n            default = 0.0,\n            min = 0.0,\n            max = 1000.0\n          )\n\n      object Dwell4Param\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_dwell_4\",\n            default = 0.0,\n            min = 0.0,\n            max = 1000.0\n          )\n\n      object VideoQualityWatchParam\n          extends FSBoundedParam[Double](\n            name = \"home_mixer_model_weight_video_quality_watched\",\n            default = 0.0,\n            min = -10000.0,\n            max = 10000.0\n          )\n    }\n\n    object UseProdInPhoenixParams {\n      object EnableProdFavForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_fav_for_phoenix\",\n            default = false\n          )\n\n      object EnableProdReplyForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_reply_for_phoenix\",\n            default = false\n          )\n\n      object EnableProdShareForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_share_for_phoenix\",\n            default = false\n          )\n\n      object EnableProdRetweetForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_retweet_for_phoenix\",\n            default = false\n          )\n\n      object EnableProdVQVForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_vqv_for_phoenix\",\n            default = false\n          )\n\n      object EnableProdDwellForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_dwell_for_phoenix\",\n            default = false\n          )\n\n      object EnableProdNegForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_neg_for_phoenix\",\n            default = false\n          )\n\n      object EnableProdProfileClickForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_profile_click_for_phoenix\",\n            default = false\n          )\n\n      object EnableProdGoodClickV1ForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_good_click_v1_for_phoenix\",\n            default = false\n          )\n\n      object EnableProdGoodClickV2ForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_good_click_v2_for_phoenix\",\n            default = false\n          )\n\n      object EnableProdOpenLinkForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_open_link_for_phoenix\",\n            default = false\n          )\n\n      object EnableProdScreenshotForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_screenshot_for_phoenix\",\n            default = false\n          )\n\n      object EnableProdBookmarkForPhoenixParam\n          extends FSParam[Boolean](\n            name = \"home_mixer_enable_prod_bookmark_for_phoenix\",\n            default = false\n          )\n    }\n  }\n\n  object EnableTenSecondsLogicForVQV\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_ten_seconds_logic_for_vqv\",\n        default = true\n      )\n\n  object EnableImmersiveVQV\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_immersive_vqv\",\n        default = false\n      )\n  object EnableLandingPage\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_landing_page\",\n        default = false\n      )\n\n  object EnableExploreSimclustersLandingPage\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_explore_simclusters_landing_page\",\n        default = false\n      )\n\n  object EnableTopicBasedRealTimeAggregateFeatureHydratorParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_topic_based_real_time_aggregate_feature_hydrator_param\",\n        default = true\n      )\n\n  object EnableTopicCountryBasedRealTimeAggregateFeatureHydratorParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_topic_country_based_real_time_aggregate_feature_hydrator_param\",\n        default = true\n      )\n\n  object EnableTopicEdgeAggregateFeatureHydratorParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_topic_edge_aggregate_feature_hydrator_param\",\n        default = true\n      )\n\n  object FeedbackFatigueFilteringDurationParam\n      extends FSBoundedParam[Duration](\n        name = \"home_mixer_feedback_fatigue_filtering_duration_in_days\",\n        default = 14.days,\n        min = 0.days,\n        max = 100.days\n      )\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromDays\n  }\n\n  object EnableCommonFeaturesDataRecordCopyDuringPldrConversionParam\n      extends BooleanDeciderParam(\n        decider = DeciderKey.EnableCommonFeaturesDataRecordCopyDuringPldrConversion)\n\n  object EnablePinnedTweetsCarouselParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_pinned_tweets_carousel\",\n        default = false\n      )\n\n  object EnablePostFeedbackParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_post_feedback\",\n        default = false\n      )\n\n  object PostFeedbackThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"home_mixer_post_feedback_threshold\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object PostFeedbackPromptTitleParam\n      extends FSParam[String](\n        name = \"home_mixer_post_feedback_prompt_title\",\n        default = \"Are you interested in this post?\"\n      )\n\n  object PostFeedbackPromptPositiveParam\n      extends FSParam[String](\n        name = \"home_mixer_post_feedback_prompt_positive\",\n        default = \"Yes\"\n      )\n\n  object PostFeedbackPromptNegativeParam\n      extends FSParam[String](\n        name = \"home_mixer_post_feedback_prompt_negative\",\n        default = \"No\"\n      )\n\n  object PostFeedbackPromptNeutralParam\n      extends FSParam[String](\n        name = \"home_mixer_post_feedback_prompt_neutral\",\n        default = \"Not sure\"\n      )\n\n  object EnablePostFollowupParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_post_followup\",\n        default = false\n      )\n\n  object EnablePostDetailsNegativeFeedbackParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_post_details_negative_feedback\",\n        default = false\n      )\n\n  object PostFollowupThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"home_mixer_post_followup_threshold\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object EnableSlopFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_slop_filter\",\n        default = false\n      )\n\n  object EnableNsfwFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_nsfw_filter\",\n        default = false\n      )\n\n  object EnableSoftNsfwFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_soft_nsfw_filter\",\n        default = false\n      )\n\n  object EnableGrokSpamFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_grok_spam_filter\",\n        default = false\n      )\n\n  object EnableGrokViolentFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_grok_violent_filter\",\n        default = false\n      )\n\n  object EnableGrokGoreFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_grok_gore_filter\",\n        default = false\n      )\n\n  object EnableMinVideoDurationFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_min_video_duration_filter\",\n        default = false\n      )\n\n  object EnableMaxVideoDurationFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_max_video_duration_filter\",\n        default = false\n      )\n\n  object EnableClusterBasedDedupFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_cluster_based_dedup_filter\",\n        default = false\n      )\n\n  object EnableCountryFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_country_filter\",\n        default = false\n      )\n\n  object EnableRegionFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_region_filter\",\n        default = false\n      )\n\n  object EnableHasMultipleMediaFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_has_multiple_media_filter\",\n        default = false\n      )\n\n  object EnableClusterBased88DedupFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_cluster_based_88_dedup_filter\",\n        default = false\n      )\n\n  object EnableNoClusterFilter\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_no_cluster_filter\",\n        default = false\n      )\n\n  object DedupHistoricalEventsTimeWindowParam\n      extends FSBoundedParam[Long](\n        name = \"home_mixer_dedup_historical_events_time_window\",\n        default = 43200000L, // 12 * 60 * 60 * 1000 = 12hrs in milliseconds\n        min = 0L,\n        max = 604800000L // 7 days\n      )\n\n  object MinVideoDurationThresholdParam\n      extends FSBoundedParam[Long](\n        name = \"home_mixer_min_video_duration_threshold\",\n        default = 0L, // 0 second\n        min = 0L,\n        max = 604800000L // 7 days\n      )\n\n  object MaxVideoDurationThresholdParam\n      extends FSBoundedParam[Long](\n        name = \"home_mixer_max_video_duration_threshold\",\n        default = 604800000L, // 7 days\n        min = 0L,\n        max = 604800000L // 7 days\n      )\n\n  object EnableSlopFilterLowSignalUsers\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_slop_filter_low_signal_users\",\n        default = false\n      )\n\n  object EnableSlopFilterEligibleUserStateParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_slop_filter_eligible_user_state_param\",\n        default = true\n      )\n\n  object SlopMaxScore\n      extends FSBoundedParam[Double](\n        name = \"home_mixer_slop_max_score\",\n        default = 0.3,\n        min = 0.0,\n        max = 4.0\n      )\n\n  object SlopMinFollowers\n      extends FSBoundedParam[Int](\n        name = \"home_mixer_slop_min_followers\",\n        default = 100,\n        min = 0,\n        max = 10000000\n      )\n\n  object EnableGrokAnnotations\n      extends FSParam[Boolean](\n        name = \"home_mixer_feature_hydration_enable_grok_annotations\",\n        default = false\n      )\n\n  object UserActionsMaxCount\n      extends FSBoundedParam[Int](\n        name = \"home_mixer_user_actions_max_count\",\n        default = 522,\n        min = 0,\n        max = 10000\n      )\n\n  object EnableTweetRTAMhOnlyParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_feature_hydration_enable_tweet_rta_read_from_mh\",\n        default = false\n      )\n\n  object EnableTweetRTAMhFallbackParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_feature_hydration_enable_tweet_rta_read_fallback_to_mh\",\n        default = false\n      )\n\n  object EnableTweetCountryRTAMhOnlyParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_feature_hydration_enable_tweet_country_rta_read_from_mh\",\n        default = false\n      )\n\n  object EnableTweetCountryRTAMhFallbackParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_feature_hydration_enable_tweet_country_rta_read_fallback_to_mh\",\n        default = false\n      )\n\n  object EnableUserRTAMhOnlyParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_feature_hydration_enable_user_rta_read_from_mh\",\n        default = false\n      )\n\n  object EnableUserRTAMhFallbackParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_feature_hydration_enable_user_rta_read_fallback_to_mh\",\n        default = false\n      )\n\n  object EnableUserAuthorRTAMhOnlyParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_feature_hydration_enable_user_author_rta_read_from_mh\",\n        default = false\n      )\n\n  object EnableUserAuthorRTAMhFallbackParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_feature_hydration_enable_user_author_rta_read_fallback_to_mh\",\n        default = false\n      )\n\n  object MaxPostContextPostsPerRequest\n      extends FSParam[Int](\n        name = \"home_mixer_feature_hydration_max_post_context_posts\",\n        default = 5\n      )\n\n  object MaxPostContextDuplicatesPerRequest\n      extends FSParam[Int](\n        name = \"home_mixer_feature_hydration_max_post_context_duplicates\",\n        default = 2\n      )\n\n  object PhoenixCluster extends Enumeration {\n    val Prod = Value\n    val Experiment1 = Value\n    val Experiment2 = Value\n    val Experiment3 = Value\n    val Experiment4 = Value\n    val Experiment5 = Value\n    val Experiment6 = Value\n    val Experiment7 = Value\n    val Experiment8 = Value\n  }\n\n  object PhoenixInferenceClusterParam\n      extends FSEnumParam[PhoenixCluster.type](\n        name = \"home_mixer_model_phoenix_inference_cluster_id\",\n        default = PhoenixCluster.Prod,\n        enum = PhoenixCluster\n      )\n\n  object PhoenixTimeoutInMsParam\n      extends FSBoundedParam[Int](\n        name = \"home_mixer_model_phoenix_timeout_in_ms\",\n        default = 500,\n        min = 10,\n        max = 10000\n      )\n\n  object EnablePhoenixScorerParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_model_enable_phoenix_scorer\",\n        default = false\n      )\n\n  object EnableUserActionsShadowScribeParam\n      extends FSParam[Boolean](\n        name = \"home_mixer_enable_user_actions_shadow_scribe\",\n        default = false\n      )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/param/HomeMixerFlagName.scala",
    "content": "package com.twitter.home_mixer.param\n\nobject HomeMixerFlagName {\n  final val ScribeClientEventsFlag = \"scribe.client_events\"\n  final val ScribeServedCandidatesFlag = \"scribe.served_candidates\"\n  final val ScribeScoredCandidatesFlag = \"scribe.scored_candidates\"\n  final val ScribeFeaturesFlag = \"scribe.features\"\n  final val DataRecordMetadataStoreConfigsYmlFlag = \"data.record.metadata.store.configs.yml\"\n  final val DarkTrafficFilterDeciderKey = \"thrift.dark.traffic.filter.decider_key\"\n  final val TargetScoringLatency = \"target.scoring.latency\"\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/param/HomeMixerInjectionNames.scala",
    "content": "package com.twitter.home_mixer.param\n\nobject HomeMixerInjectionNames {\n  final val AuthorFeatureRepository = \"AuthorFeatureRepository\"\n  final val CommonFeaturesScribeEventPublisher = \"CommonFeaturesScribeEventPublisher\"\n  final val CommonFeaturesScribeVideoEventPublisher = \"CommonFeaturesScribeVideoEventPublisher\"\n  final val EarlybirdRepository = \"EarlybirdRepository\"\n  final val EarlybirdRealtimCGEndpoint = \"EarlybirdRealtimCGEndpoint\"\n  final val EngagementsReceivedByAuthorCache = \"EngagementsReceivedByAuthorCache\"\n  final val EntityRealGraphClientStore = \"EntityRealGraphClientStore\"\n  final val GraphTwoHopRepository = \"GraphTwoHopRepository\"\n  final val GizmoduckTimelinesCache = \"GizmoduckTimelinesCache\"\n  final val HomeAuthorFeaturesCacheClient = \"HomeAuthorFeaturesCacheClient\"\n  final val InterestsThriftServiceClient = \"InterestsThriftServiceClient\"\n  final val BatchedStratoClientWithDefaultTimeout = \"BatchedStratoClientWithDefaultTimeout\"\n  final val StratoClientWithLongTimeout = \"StratoClientWithLongTimeout\"\n  final val BatchedStratoClientWithModerateTimeout = \"BatchedStratoClientWithModerateTimeout\"\n  final val BatchedStratoClientWithLongTimeout = \"BatchedStratoClientWithLongTimeout\"\n  final val ManhattanApolloClient = \"ManhattanApolloClient\"\n  final val ManhattanAthenaClient = \"ManhattanAthenaClient\"\n  final val ManhattanOmegaClient = \"ManhattanOmegaClient\"\n  final val ManhattanStarbuckClient = \"ManhattanStarbuckClient\"\n  final val MediaClipClusterIdInMemCache = \"MediaClipClusterIdInMemCache\"\n  final val ImageClipClusterIdInMemCache = \"ImageClipClusterIdInMemCache\"\n  final val MediaCompletionRateInMemCache = \"MediaCompletionRateInMemCache\"\n  final val IsColdStartPostInMemCache = \"IsColdStartPostInMemCache\"\n  final val MemcacheCandidateFeaturesStore = \"MemcacheCandidateFeaturesStore\"\n  final val MemcacheVideoCandidateFeaturesStore = \"MemcacheVideoCandidateFeaturesStore\"\n  final val MemcachedImpressionBloomFilterStore = \"MemcachedImpressionBloomFilterStore\"\n  final val NaviModelClientHomeRecap = \"NaviModelClientHomeRecap\"\n  final val NaviModelClientHomeRecapSecondary = \"NaviModelClientHomeRecapSecondary\"\n  final val NaviModelClientHomeRecapRealtime = \"NaviModelClientHomeRecapRealtime\"\n  final val NaviModelClientHomeRecapGPU = \"NaviModelClientHomeRecapGPU\"\n  final val NaviModelClientHomeRecapVideo = \"NaviModelClientHomeRecapVideo\"\n  final val RealGraphInNetworkScoresOnPrem = \"RealGraphInNetworkScoresOnPrem\"\n  final val RealGraphManhattanEndpoint = \"RealGraphFeaturesManhattanEndpoint\"\n  final val RTAManhattanEndpoint = \"RTAManhattanEndpoint\"\n  final val RTAManhattanStore = \"RTAManhattanStore\"\n  final val RealGraphFeatureRepository = \"RealGraphFeatureRepository\"\n  final val RealTimeInteractionGraphUserVertexCache = \"RealTimeInteractionGraphUserVertexCache\"\n  final val RealTimeInteractionGraphUserVertexClient = \"RealTimeInteractionGraphUserVertexClient\"\n  final val ScoredTweetsCache = \"ScoredTweetsCache\"\n  final val ScoredVideoTweetsCache = \"ScoredVideoTweetsCache\"\n  final val TesBatchedStratoClient = \"TesBatchedStratoClient\"\n  final val TimelineAggregateMetadataRepository = \"TimelineAggregateMetadataRepository\"\n  final val TimelineAggregatePartARepository = \"TimelineAggregatePartARepository\"\n  final val TimelineAggregatePartBRepository = \"TimelineAggregatePartBRepository\"\n  final val TimelinesRealTimeAggregateClient = \"TimelinesRealTimeAggregateClient\"\n  final val TopicCountryEngagementCache = \"TopicCountryEngagementCache\"\n  final val TopicEngagementCache = \"TopicEngagementCache\"\n  final val TweetCountryEngagementCache = \"TweetCountryEngagementCache\"\n  final val TweetEngagementCache = \"TweetEngagementCache\"\n  final val TweetypieStaticEntitiesCache = \"TweetypieStaticEntitiesCache\"\n  final val TwhinAuthorFollowFeatureCacheClient = \"TwhinAuthorFollowFeatureCacheClient\"\n  final val TwhinAuthorFollowFeatureRepository = \"TwhinAuthorFollowFeatureRepository\"\n  final val TwhinUserEngagementFeatureRepository = \"TwhinUserEngagementFeatureRepository\"\n  final val TwhinRebuildUserEngagementFeatureRepository =\n    \"TwhinRebuildUserEngagementFeatureRepository\"\n  final val TwhinUserFollowFeatureRepository = \"TwhinUserFollowFeatureRepository\"\n  final val TwhinTweetEmbeddingsStore = \"TwhinTweetEmbeddingsStore\"\n  final val TwhinRebuildTweetEmbeddingsStore = \"TwhinRebuildTweetEmbeddingsStore\"\n  final val TwhinVideoEmbeddingsStore = \"TwhinVideoEmbeddingsStore\"\n  final val TwhinUserNegativeEmbeddingsStore = \"TwhinUserNegativeEmbeddingsStore\"\n  final val TwhinUserPositiveEmbeddingsStore = \"TwhinUserPositiveEmbeddingsStore\"\n  final val TwhinRebuildUserPositiveEmbeddingsStore = \"TwhinRebuildUserPositiveEmbeddingsStore\"\n  final val TwitterListEngagementCache = \"TwitterListEngagementCache\"\n  final val UserAuthorEngagementCache = \"UserAuthorEngagementCache\"\n  final val UserEngagementCache = \"UserEngagementCache\"\n  final val UserFollowedTopicIdsRepository = \"UserFollowedTopicIdsRepository\"\n  final val UserLanguagesRepository = \"UserLanguagesRepository\"\n  final val UserTopicEngagementForNewUserCache = \"UserTopicEngagementForNewUserCache\"\n  final val UtegSocialProofRepository = \"UtegSocialProofRepository\"\n  final val TvRealTimeAggregateClient = \"TvRealTimeAggregateClient\"\n  final val TvVideoByUserTweetCache = \"TvVideoByUserTweetCache\"\n  final val TvWatchedVideoIdsKeyCacheStore = \"TvWatchedVideoIdsKeyCacheStore\"\n  final val MediaClusterId95Store = \"MediaClusterId95Store\"\n  final val MediaClusterId88Store = \"MediaClusterId88Store\"\n  final val TweetWatchTimeMetadataStore = \"TweetWatchTimeMetadataStore\"\n  final val VideoEmbeddingMHStore = \"VideoEmbeddingMHStore\"\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/param/decider/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\"servo/decider\"],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/param/decider/DeciderKey.scala",
    "content": "package com.twitter.home_mixer.param.decider\n\nimport com.twitter.servo.decider.DeciderKeyEnum\n\n/**\n * These values must correspond to the deciders configured in the\n * home-mixer/server/src/main/resources/config/decider.yml file\n *\n * @see [[com.twitter.product_mixer.core.product.ProductParamConfig.enabledDeciderKey]]\n */\nobject DeciderKey extends DeciderKeyEnum {\n\n  val EnableForYouProduct = Value(\"enable_for_you_product\")\n\n  val EnableFollowingProduct = Value(\"enable_following_product\")\n\n  val EnableScoredTweetsProduct = Value(\"enable_scored_tweets_product\")\n\n  val EnableScoredVideoTweetsProduct = Value(\"enable_scored_video_tweets_product\")\n\n  val EnableSubscribedProduct = Value(\"enable_subscribed_product\")\n\n  val EnableHeavyRankerScoresProduct = Value(\"enable_heavy_ranker_scores_product\")\n\n  val EnableSimClustersSimilarityFeatureHydration =\n    Value(\"enable_simclusters_similarity_feature_hydration\")\n\n  val EnableServedCandidateFeatureKeysKafkaPublishing =\n    Value(\"enable_served_candidate_feature_keys_kafka_publishing\")\n\n  val EnablePublishCommonFeaturesKafka =\n    Value(\"enable_publish_common_features_kafka\")\n\n  val LiveSpacesFactor = Value(\"live_spaces_factor\")\n\n  val MtlNormalizationAlpha = Value(\"mtl_normalization_alpha\")\n\n  val EnableTweetypieContentFeatures = Value(\"enable_tweetypie_content_features\")\n\n  val EnableCommonFeaturesDataRecordCopyDuringPldrConversion =\n    Value(\"enable_common_features_data_record_copy_during_pldr_conversion\")\n\n  val EnableGetTweetsFromArchiveIndex = Value(\"enable_get_tweets_from_archive_index\")\n\n  val EnableSecondaryNaviRecapCluster = Value(\"enable_secondary_navi_recap_cluster\")\n\n  val EnableGPUNaviRecapClusterTestUsers = Value(\"enable_gpu_navi_recap_cluster_test_users\")\n\n  val NaviGPUClusterRequestBatchSize = Value(\"navi_gpu_cluster_request_batch_size\")\n\n  val EnableVideoSummaryEmbeddingHydration = Value(\n    \"enable_video_summary_embedding_feature_hydration\")\n\n  val EnableVideoClipEmbeddingHydration = Value(\"enable_video_clip_embedding_feature_hydration\")\n\n  val EnableScoredVideoTweetsUserHistoryEventsQueryFeatureHydrationDeciderParam = Value(\n    \"enable_scored_video_tweets_user_history_events_query_feature_hydration\")\n\n  val EnableVideoClipEmbeddingMediaUnderstandingHydration = Value(\n    \"enable_video_clip_embedding_media_understanding_feature_hydration\")\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/BUILD.bazel",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/heavy_ranker_scores\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/heavy_ranker_scores/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_video_tweets\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/model\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/HomeMixerProductModule.scala",
    "content": "package com.twitter.home_mixer.product\n\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistryConfig\n\nobject HomeMixerProductModule extends TwitterModule {\n\n  override def configure(): Unit = {\n    bind[ProductPipelineRegistryConfig].to[HomeProductPipelineRegistryConfig]\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/HomeProductPipelineRegistryConfig.scala",
    "content": "package com.twitter.home_mixer.product\n\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.model.request.HeavyRankerScoresProduct\nimport com.twitter.home_mixer.model.request.ScoredTweetsProduct\nimport com.twitter.home_mixer.model.request.ScoredVideoTweetsProduct\nimport com.twitter.home_mixer.model.request.SubscribedProduct\nimport com.twitter.home_mixer.product.following.FollowingProductPipelineConfig\nimport com.twitter.home_mixer.product.for_you.ForYouProductPipelineConfig\nimport com.twitter.home_mixer.product.heavy_ranker_scores.HeavyRankerScoresProductPipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.ScoredTweetsProductPipelineConfig\nimport com.twitter.home_mixer.product.scored_video_tweets.ScoredVideoTweetsProductPipelineConfig\nimport com.twitter.home_mixer.product.subscribed.SubscribedProductPipelineConfig\nimport com.twitter.inject.Injector\nimport com.twitter.product_mixer.core.product.guice.ProductScope\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistryConfig\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeProductPipelineRegistryConfig @Inject() (\n  injector: Injector,\n  productScope: ProductScope)\n    extends ProductPipelineRegistryConfig {\n\n  private val followingProductPipelineConfig = productScope.let(FollowingProduct) {\n    injector.instance[FollowingProductPipelineConfig]\n  }\n  private val forYouProductPipelineConfig = productScope.let(ForYouProduct) {\n    injector.instance[ForYouProductPipelineConfig]\n  }\n\n  private val scoredTweetsProductPipelineConfig = productScope.let(ScoredTweetsProduct) {\n    injector.instance[ScoredTweetsProductPipelineConfig]\n  }\n\n  private val scoredVideoTweetsProductPipelineConfig = productScope.let(ScoredVideoTweetsProduct) {\n    injector.instance[ScoredVideoTweetsProductPipelineConfig]\n  }\n\n  private val subscribedProductPipelineConfig = productScope.let(SubscribedProduct) {\n    injector.instance[SubscribedProductPipelineConfig]\n  }\n\n  private val heavyRankerScoresProductPipelineConfig = productScope.let(HeavyRankerScoresProduct) {\n    injector.instance[HeavyRankerScoresProductPipelineConfig]\n  }\n\n  override val productPipelineConfigs = Seq(\n    followingProductPipelineConfig,\n    forYouProductPipelineConfig,\n    scoredTweetsProductPipelineConfig,\n    scoredVideoTweetsProductPipelineConfig,\n    subscribedProductPipelineConfig,\n    heavyRankerScoresProductPipelineConfig\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/api\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/account_recommendations_mixer\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/earlybird\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/param_gated\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/async\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/impressed_tweets\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/location\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/param_gated\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/selector\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/cursor/timelines\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/earlybird\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/java/com/twitter/search/queryparser/query/search:search-query-nodes\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate\",\n    ],\n    exports = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/param\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/FollowingAdsCandidatePipelineBuilder.scala",
    "content": "package com.twitter.home_mixer.product.following\n\nimport com.twitter.adserver.{thriftscala => ads}\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeAdsClientEventDetailsBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator.NoAdsTierFeature\nimport com.twitter.home_mixer.functional_component.gate.ExcludeSoftUserGate\nimport com.twitter.home_mixer.functional_component.gate.ExcludeSyntheticUserGate\nimport com.twitter.home_mixer.param.HomeGlobalParams\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableAdvertiserBrandSafetySettingsFeatureHydratorParam\nimport com.twitter.home_mixer.product.following.model.FollowingQuery\nimport com.twitter.home_mixer.product.following.param.FollowingParam.EnableFastAds\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.candidate_source.ads.AdsProdThriftCandidateSource\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref.ContextualTweetRefBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.ad.AdsCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.ads.AdvertiserBrandSafetySettingsFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.gate.FeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.AdsCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.AdsCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.ValidAdImpressionIdFilter\nimport com.twitter.product_mixer.core.gate.ParamNotGate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.TimelineHomePromotedHydrationSafetyLevel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.TweetHydrationContext\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FollowingAdsCandidatePipelineBuilder @Inject() (\n  adsCandidatePipelineConfigBuilder: AdsCandidatePipelineConfigBuilder,\n  adsCandidateSource: AdsProdThriftCandidateSource,\n  advertiserBrandSafetySettingsFeatureHydrator: AdvertiserBrandSafetySettingsFeatureHydrator[\n    FollowingQuery,\n    AdsCandidate\n  ]) {\n\n  private val identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\"FollowingAds\")\n\n  private val EstimatedNumOrganicItems = 100.toShort\n\n  private val servedType = hmt.ServedType.FollowingPromoted\n\n  private val clientEventInfoBuilder = ClientEventInfoBuilder(\n    component = servedType.originalName,\n    detailsBuilder = Some(HomeAdsClientEventDetailsBuilder(Some(servedType.originalName)))\n  )\n\n  private val contextualTweetRefBuilder = ContextualTweetRefBuilder(\n    TweetHydrationContext(\n      safetyLevelOverride = Some(TimelineHomePromotedHydrationSafetyLevel),\n      outerTweetContext = None\n    ))\n\n  private val decorator = UrtItemCandidateDecorator(\n    AdsCandidateUrtItemBuilder(\n      tweetClientEventInfoBuilder = Some(clientEventInfoBuilder),\n      contextualTweetRefBuilder = Some(contextualTweetRefBuilder)\n    ))\n\n  private val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),\n    HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()\n  )\n\n  def build(): AdsCandidatePipelineConfig[FollowingQuery] =\n    adsCandidatePipelineConfigBuilder.build[FollowingQuery](\n      adsCandidateSource = adsCandidateSource,\n      identifier = identifier,\n      adsDisplayLocationBuilder = query =>\n        if (query.params.getBoolean(EnableFastAds)) ads.DisplayLocation.TimelineHomeReverseChron\n        else ads.DisplayLocation.TimelineHome,\n      estimateNumOrganicItems = _ => EstimatedNumOrganicItems,\n      gates = Seq(\n        ParamNotGate(\n          name = \"AdsDisableInjectionBasedOnUserRole\",\n          param = HomeGlobalParams.AdsDisableInjectionBasedOnUserRoleParam\n        ),\n        ExcludeSoftUserGate,\n        ExcludeSyntheticUserGate,\n        FeatureGate.fromNegatedFeature(NoAdsTierFeature)\n      ),\n      filters = Seq(ValidAdImpressionIdFilter),\n      postFilterFeatureHydration = Seq(\n        ParamGatedCandidateFeatureHydrator(\n          EnableAdvertiserBrandSafetySettingsFeatureHydratorParam,\n          advertiserBrandSafetySettingsFeatureHydrator\n        )\n      ),\n      decorator = Some(decorator),\n      alerts = alerts,\n      urtRequest = Some(true)\n    )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/FollowingAdsDependentCandidatePipelineBuilder.scala",
    "content": "package com.twitter.home_mixer.product.following\n\nimport com.twitter.adserver.{thriftscala => ads}\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeAdsClientEventDetailsBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator.NoAdsTierFeature\nimport com.twitter.home_mixer.functional_component.gate.ExcludeSoftUserGate\nimport com.twitter.home_mixer.functional_component.gate.ExcludeSyntheticUserGate\nimport com.twitter.home_mixer.model.HomeFeatures.TweetLanguageFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTextFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableAdvertiserBrandSafetySettingsFeatureHydratorParam\nimport com.twitter.home_mixer.product.following.model.FollowingQuery\nimport com.twitter.home_mixer.product.following.param.FollowingParam.EnableFastAds\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.candidate_source.ads.AdsProdThriftCandidateSource\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref.ContextualTweetRefBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.ad.AdsCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.ads.AdvertiserBrandSafetySettingsFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.gate.FeatureGate\nimport com.twitter.product_mixer.component_library.gate.NonEmptyCandidatesGate\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.AdsDependentCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.AdsDependentCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.CountCandidatesFromPipelines\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.PipelineScopedOrganicItems\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.ValidAdImpressionIdFilter\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.gate.ParamNotGate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.TimelineHomePromotedHydrationSafetyLevel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.TweetHydrationContext\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FollowingAdsDependentCandidatePipelineBuilder @Inject() (\n  adsCandidatePipelineConfigBuilder: AdsDependentCandidatePipelineConfigBuilder,\n  adsCandidateSource: AdsProdThriftCandidateSource,\n  advertiserBrandSafetySettingsFeatureHydrator: AdvertiserBrandSafetySettingsFeatureHydrator[\n    FollowingQuery,\n    AdsCandidate\n  ]) {\n\n  private val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"FollowingAdsDependent\")\n\n  private val servedType = hmt.ServedType.FollowingPromoted\n\n  private val clientEventInfoBuilder = ClientEventInfoBuilder(\n    component = servedType.originalName,\n    detailsBuilder = Some(HomeAdsClientEventDetailsBuilder(Some(servedType.originalName)))\n  )\n\n  private val contextualTweetRefBuilder = ContextualTweetRefBuilder(\n    TweetHydrationContext(\n      safetyLevelOverride = Some(TimelineHomePromotedHydrationSafetyLevel),\n      outerTweetContext = None\n    ))\n\n  private val decorator = UrtItemCandidateDecorator(\n    AdsCandidateUrtItemBuilder(\n      tweetClientEventInfoBuilder = Some(clientEventInfoBuilder),\n      contextualTweetRefBuilder = Some(contextualTweetRefBuilder)\n    ))\n\n  private val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),\n    HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()\n  )\n\n  def build(\n    organicCandidatePipelines: CandidateScope\n  ): AdsDependentCandidatePipelineConfig[FollowingQuery] =\n    adsCandidatePipelineConfigBuilder.build[FollowingQuery](\n      adsCandidateSource = adsCandidateSource,\n      identifier = identifier,\n      adsDisplayLocationBuilder = query =>\n        if (query.params.getBoolean(EnableFastAds)) ads.DisplayLocation.TimelineHomeReverseChron\n        else ads.DisplayLocation.TimelineHome,\n      getOrganicItems = PipelineScopedOrganicItems(\n        pipelines = organicCandidatePipelines,\n        textFeature = TweetTextFeature,\n        languageFeature = TweetLanguageFeature\n      ),\n      countNumOrganicItems = CountCandidatesFromPipelines(organicCandidatePipelines),\n      gates = Seq(\n        ParamNotGate(\n          name = \"AdsDisableInjectionBasedOnUserRole\",\n          param = HomeGlobalParams.AdsDisableInjectionBasedOnUserRoleParam\n        ),\n        ExcludeSoftUserGate,\n        ExcludeSyntheticUserGate,\n        FeatureGate.fromNegatedFeature(NoAdsTierFeature),\n        NonEmptyCandidatesGate(organicCandidatePipelines)\n      ),\n      filters = Seq(ValidAdImpressionIdFilter),\n      postFilterFeatureHydration = Seq(\n        ParamGatedCandidateFeatureHydrator(\n          EnableAdvertiserBrandSafetySettingsFeatureHydratorParam,\n          advertiserBrandSafetySettingsFeatureHydrator\n        )\n      ),\n      decorator = Some(decorator),\n      alerts = alerts,\n      urtRequest = Some(true),\n    )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/FollowingDependentAdsMixerPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.following\n\nimport com.twitter.clientapp.{thriftscala => ca}\nimport com.twitter.goldfinch.api.AdsInjectionSurfaceAreas\nimport com.twitter.home_mixer.candidate_pipeline.ConversationServiceCandidatePipelineConfigBuilder\nimport com.twitter.home_mixer.candidate_pipeline.EditedTweetsCandidatePipelineConfig\nimport com.twitter.home_mixer.candidate_pipeline.NewTweetsPillCandidatePipelineConfig\nimport com.twitter.home_mixer.candidate_pipeline.VerifiedPromptCandidatePipelineConfig\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator._\nimport com.twitter.home_mixer.functional_component.selector.UpdateHomeClientEventDetails\nimport com.twitter.home_mixer.functional_component.selector.UpdateNewTweetsPillDecoration\nimport com.twitter.home_mixer.functional_component.side_effect._\nimport com.twitter.home_mixer.model.ClearCacheIncludeInstruction\nimport com.twitter.home_mixer.model.NavigationIncludeInstruction\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableSSPAdsBrandSafetySettingsFeatureHydratorParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.MaxNumberReplaceInstructionsParam\nimport com.twitter.home_mixer.param.HomeMixerFlagName.ScribeClientEventsFlag\nimport com.twitter.home_mixer.product.following.model.FollowingQuery\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.product.following.param.FollowingParam.ClearCache\nimport com.twitter.home_mixer.product.following.param.FollowingParam.EnableFlipInjectionModuleCandidatePipelineParam\nimport com.twitter.home_mixer.product.following.param.FollowingParam.EnablePostContextFeatureHydratorParam\nimport com.twitter.home_mixer.product.following.param.FollowingParam.Navigation\nimport com.twitter.home_mixer.product.following.param.FollowingParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.product.following.param.FollowingParam.StaticParamValueFive\nimport com.twitter.home_mixer.product.following.param.FollowingParam.StaticParamValueZero\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.ads.SSPAdsBrandSafetySettingsFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.async.AsyncQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.impressed_tweets.ImpressedTweetsQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.location.UserLocationQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated.ParamGatedQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.FlipPromptDependentCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.selector.FlipPromptDynamicInsertionPosition\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowArmCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.premarshaller.urt.UrtDomainMarshaller\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ClearCacheInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.NavigationInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedBottomCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedCursorIdSelector.TweetIdSelector\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedGapCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedTopCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceAllEntries\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceEntryInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ShowAlertInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ShowCoverInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.StaticTimelineScribeConfigBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtMetadataBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.earlybird.EarlybirdGapIncludeInstruction\nimport com.twitter.product_mixer.component_library.selector.DropMaxCandidates\nimport com.twitter.product_mixer.component_library.selector.DropMaxModuleItemCandidates\nimport com.twitter.product_mixer.component_library.selector.DropModuleTooFewModuleItemResults\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.component_library.selector.InsertDynamicPositionResults\nimport com.twitter.product_mixer.component_library.selector.InsertFixedPositionResults\nimport com.twitter.product_mixer.component_library.selector.SelectConditionally\nimport com.twitter.product_mixer.component_library.selector.UpdateSortCandidates\nimport com.twitter.product_mixer.component_library.selector.ads.AdsInjector\nimport com.twitter.product_mixer.component_library.selector.ads.InsertAdResults\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.UrtTransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.MixerPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineConfig\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass FollowingDependentAdsMixerPipelineConfig @Inject() (\n  followingEarlybirdCandidatePipelineConfig: FollowingEarlybirdCandidatePipelineConfig,\n  conversationServiceCandidatePipelineConfigBuilder: ConversationServiceCandidatePipelineConfigBuilder[\n    FollowingQuery\n  ],\n  followingAdsDependentCandidatePipelineBuilder: FollowingAdsDependentCandidatePipelineBuilder,\n  followingWhoToFollowCandidatePipelineConfigBuilder: FollowingWhoToFollowCandidatePipelineConfigBuilder,\n  flipPromptDependentCandidatePipelineConfigBuilder: FlipPromptDependentCandidatePipelineConfigBuilder,\n  editedTweetsCandidatePipelineConfig: EditedTweetsCandidatePipelineConfig,\n  newTweetsPillCandidatePipelineConfig: NewTweetsPillCandidatePipelineConfig[FollowingQuery],\n  verifiedPromptCandidatePipelineConfig: VerifiedPromptCandidatePipelineConfig,\n  dismissInfoQueryFeatureHydrator: DismissInfoQueryFeatureHydrator,\n  gizmoduckUserQueryFeatureHydrator: GizmoduckUserQueryFeatureHydrator,\n  persistenceStoreQueryFeatureHydrator: PersistenceStoreQueryFeatureHydrator,\n  rateLimitQueryFeatureHydrator: RateLimitQueryFeatureHydrator,\n  requestQueryFeatureHydrator: RequestQueryFeatureHydrator[FollowingQuery],\n  sgsFollowedUsersQueryFeatureHydrator: SGSFollowedUsersQueryFeatureHydrator,\n  memcacheTweetImpressionsQueryFeatureHydrator: ImpressedTweetsQueryFeatureHydrator,\n  lastNonPollingTimeQueryFeatureHydrator: LastNonPollingTimeQueryFeatureHydrator,\n  userLocationQueryFeatureHydrator: UserLocationQueryFeatureHydrator,\n  userSubscriptionQueryFeatureHydrator: UserSubscriptionQueryFeatureHydrator,\n  sspAdsBrandSafetySettingsFeatureHydrator: SSPAdsBrandSafetySettingsFeatureHydrator,\n  adsInjector: AdsInjector,\n  updateLastNonPollingTimeSideEffect: UpdateLastNonPollingTimeSideEffect[FollowingQuery, Timeline],\n  publishClientSentImpressionsEventBusSideEffect: PublishClientSentImpressionsEventBusSideEffect,\n  updateTimelinesPersistenceStoreSideEffect: UpdateTimelinesPersistenceStoreSideEffect,\n  truncateTimelinesPersistenceStoreSideEffect: TruncateTimelinesPersistenceStoreSideEffect,\n  homeTimelineServedCandidatesSideEffect: HomeScribeServedCandidatesSideEffect,\n  clientEventsScribeEventPublisher: EventPublisher[ca.LogEvent],\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter],\n  urtTransportMarshaller: UrtTransportMarshaller,\n  @Flag(ScribeClientEventsFlag) enableScribeClientEvents: Boolean)\n    extends MixerPipelineConfig[FollowingQuery, Timeline, urt.TimelineResponse] {\n\n  override val identifier: MixerPipelineIdentifier =\n    MixerPipelineIdentifier(\"FollowingDependentAds\")\n\n  private val dependentCandidatesStep = MixerPipelineConfig.dependentCandidatePipelinesStep\n  private val resultSelectorsStep = MixerPipelineConfig.resultSelectorsStep\n\n  override val fetchQueryFeatures: Seq[QueryFeatureHydrator[FollowingQuery]] = Seq(\n    requestQueryFeatureHydrator,\n    sgsFollowedUsersQueryFeatureHydrator,\n    rateLimitQueryFeatureHydrator,\n    gizmoduckUserQueryFeatureHydrator,\n    userSubscriptionQueryFeatureHydrator,\n    ParamGatedQueryFeatureHydrator(\n      EnableSSPAdsBrandSafetySettingsFeatureHydratorParam,\n      sspAdsBrandSafetySettingsFeatureHydrator\n    ),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, dismissInfoQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, persistenceStoreQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, lastNonPollingTimeQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, userLocationQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(resultSelectorsStep, memcacheTweetImpressionsQueryFeatureHydrator),\n  )\n\n  private val earlybirdCandidatePipelineScope =\n    SpecificPipeline(followingEarlybirdCandidatePipelineConfig.identifier)\n\n  private val conversationServiceCandidatePipelineConfig =\n    conversationServiceCandidatePipelineConfigBuilder.build(\n      earlybirdCandidatePipelineScope,\n      hmt.ServedType.FollowingInNetwork,\n      EnablePostContextFeatureHydratorParam\n    )\n\n  private val followingAdsCandidatePipelineConfig =\n    followingAdsDependentCandidatePipelineBuilder.build(earlybirdCandidatePipelineScope)\n\n  private val followingWhoToFollowCandidatePipelineConfig =\n    followingWhoToFollowCandidatePipelineConfigBuilder.build(earlybirdCandidatePipelineScope)\n\n  private val flipPromptCandidatePipelineConfig =\n    flipPromptDependentCandidatePipelineConfigBuilder.build[FollowingQuery](\n      supportedClientParam = Some(EnableFlipInjectionModuleCandidatePipelineParam)\n    )\n\n  override val candidatePipelines: Seq[CandidatePipelineConfig[FollowingQuery, _, _, _]] =\n    Seq(followingEarlybirdCandidatePipelineConfig)\n\n  override val dependentCandidatePipelines: Seq[\n    DependentCandidatePipelineConfig[FollowingQuery, _, _, _]\n  ] = Seq(\n    conversationServiceCandidatePipelineConfig,\n    followingAdsCandidatePipelineConfig,\n    followingWhoToFollowCandidatePipelineConfig,\n    flipPromptCandidatePipelineConfig,\n    editedTweetsCandidatePipelineConfig,\n    newTweetsPillCandidatePipelineConfig,\n    verifiedPromptCandidatePipelineConfig\n  )\n\n  override val failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy] = Map(\n    followingAdsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    followingWhoToFollowCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    flipPromptCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    editedTweetsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    newTweetsPillCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n  )\n\n  override val resultSelectors: Seq[Selector[FollowingQuery]] = Seq(\n    UpdateSortCandidates(\n      ordering = CandidatesUtil.reverseChronTweetsOrdering,\n      candidatePipeline = conversationServiceCandidatePipelineConfig.identifier\n    ),\n    DropMaxCandidates(\n      candidatePipeline = editedTweetsCandidatePipelineConfig.identifier,\n      maxSelectionsParam = MaxNumberReplaceInstructionsParam\n    ),\n    DropMaxCandidates(\n      candidatePipeline = conversationServiceCandidatePipelineConfig.identifier,\n      maxSelectionsParam = ServerMaxResultsParam\n    ),\n    DropModuleTooFewModuleItemResults(\n      candidatePipeline = followingWhoToFollowCandidatePipelineConfig.identifier,\n      minModuleItemsParam = StaticParam(WhoToFollowArmCandidatePipelineConfig.MinCandidatesSize)\n    ),\n    DropMaxModuleItemCandidates(\n      candidatePipeline = followingWhoToFollowCandidatePipelineConfig.identifier,\n      maxModuleItemsParam = StaticParam(WhoToFollowArmCandidatePipelineConfig.MaxCandidatesSize)\n    ),\n    InsertAppendResults(candidatePipeline = conversationServiceCandidatePipelineConfig.identifier),\n    InsertFixedPositionResults(\n      candidatePipeline = verifiedPromptCandidatePipelineConfig.identifier,\n      positionParam = StaticParamValueZero\n    ),\n    InsertDynamicPositionResults(\n      candidatePipeline = flipPromptCandidatePipelineConfig.identifier,\n      dynamicInsertionPosition = FlipPromptDynamicInsertionPosition(StaticParamValueZero)\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = followingWhoToFollowCandidatePipelineConfig.identifier,\n      positionParam = StaticParamValueFive\n    ),\n    InsertAdResults(\n      surfaceAreaName = AdsInjectionSurfaceAreas.HomeTimeline,\n      adsInjector = adsInjector.forSurfaceArea(AdsInjectionSurfaceAreas.HomeTimeline),\n      adsCandidatePipeline = followingAdsCandidatePipelineConfig.identifier\n    ),\n    // This selector must come after the tweets are inserted into the results\n    UpdateNewTweetsPillDecoration(\n      pipelineScope = SpecificPipelines(\n        conversationServiceCandidatePipelineConfig.identifier,\n        newTweetsPillCandidatePipelineConfig.identifier\n      ),\n      stringCenter = stringCenterProvider.get(),\n      seeNewTweetsString = externalStrings.seeNewTweetsString,\n      tweetedString = externalStrings.tweetedString\n    ),\n    InsertAppendResults(candidatePipeline = editedTweetsCandidatePipelineConfig.identifier),\n    SelectConditionally(\n      selector =\n        InsertAppendResults(candidatePipeline = newTweetsPillCandidatePipelineConfig.identifier),\n      includeSelector = (_, _, results) => CandidatesUtil.containsType[TweetCandidate](results)\n    ),\n    UpdateHomeClientEventDetails(\n      candidatePipelines = Set(conversationServiceCandidatePipelineConfig.identifier)\n    )\n  )\n\n  private val homeScribeClientEventSideEffect = HomeScribeClientEventSideEffect(\n    enableScribeClientEvents = enableScribeClientEvents,\n    logPipelinePublisher = clientEventsScribeEventPublisher,\n    injectedTweetsCandidatePipelineIdentifiers =\n      Seq(conversationServiceCandidatePipelineConfig.identifier),\n    adsCandidatePipelineIdentifier = Some(followingAdsCandidatePipelineConfig.identifier),\n    whoToFollowCandidatePipelineIdentifier =\n      Some(followingWhoToFollowCandidatePipelineConfig.identifier),\n  )\n\n  override val resultSideEffects: Seq[PipelineResultSideEffect[FollowingQuery, Timeline]] = Seq(\n    homeScribeClientEventSideEffect,\n    homeTimelineServedCandidatesSideEffect,\n    publishClientSentImpressionsEventBusSideEffect,\n    truncateTimelinesPersistenceStoreSideEffect,\n    updateLastNonPollingTimeSideEffect,\n    updateTimelinesPersistenceStoreSideEffect,\n  )\n\n  override val domainMarshaller: DomainMarshaller[FollowingQuery, Timeline] = {\n    val instructionBuilders = Seq(\n      ClearCacheInstructionBuilder(\n        ClearCacheIncludeInstruction(\n          ClearCache.PtrEnableParam,\n          ClearCache.ColdStartEnableParam,\n          ClearCache.WarmStartEnableParam,\n          ClearCache.ManualRefreshEnableParam,\n          ClearCache.NavigateEnableParam,\n          ClearCache.MinEntriesParam\n        )),\n      ReplaceEntryInstructionBuilder(ReplaceAllEntries),\n      AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder(),\n      ShowAlertInstructionBuilder(),\n      ShowCoverInstructionBuilder(),\n      NavigationInstructionBuilder(\n        NavigationIncludeInstruction(\n          Navigation.PtrEnableParam,\n          Navigation.ColdStartEnableParam,\n          Navigation.WarmStartEnableParam,\n          Navigation.ManualRefreshEnableParam,\n          Navigation.NavigateEnableParam\n        ))\n    )\n\n    val topCursorBuilder = OrderedTopCursorBuilder(TweetIdSelector)\n    val bottomCursorBuilder =\n      OrderedBottomCursorBuilder(TweetIdSelector, EarlybirdGapIncludeInstruction.inverse())\n    val gapCursorBuilder = OrderedGapCursorBuilder(TweetIdSelector, EarlybirdGapIncludeInstruction)\n\n    val scribeConfigBuilder =\n      StaticTimelineScribeConfigBuilder(TimelineScribeConfig(Some(\"following\"), None, None))\n    val metadataBuilder = UrtMetadataBuilder(scribeConfigBuilder = Some(scribeConfigBuilder))\n\n    UrtDomainMarshaller(\n      instructionBuilders = instructionBuilders,\n      metadataBuilder = Some(metadataBuilder),\n      cursorBuilders = Seq(topCursorBuilder, bottomCursorBuilder, gapCursorBuilder)\n    )\n  }\n\n  override val transportMarshaller: TransportMarshaller[Timeline, urt.TimelineResponse] =\n    urtTransportMarshaller\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/FollowingEarlybirdCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.following\n\nimport com.twitter.home_mixer.candidate_pipeline.FollowingEarlybirdResponseFeatureTransformer\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.product.following.model.FollowingQuery\nimport com.twitter.product_mixer.component_library.candidate_source.earlybird.EarlybirdTweetCandidateSource\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.component_library.gate.NonEmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.search.earlybird.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FollowingEarlybirdCandidatePipelineConfig @Inject() (\n  earlybirdTweetCandidateSource: EarlybirdTweetCandidateSource,\n  followingEarlybirdQueryTransformer: FollowingEarlybirdQueryTransformer)\n    extends CandidatePipelineConfig[\n      FollowingQuery,\n      t.EarlybirdRequest,\n      t.ThriftSearchResult,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"FollowingEarlybird\")\n\n  override val candidateSource: BaseCandidateSource[t.EarlybirdRequest, t.ThriftSearchResult] =\n    earlybirdTweetCandidateSource\n\n  override val gates: Seq[Gate[FollowingQuery]] = Seq(\n    RateLimitGate,\n    NonEmptySeqFeatureGate(SGSFollowedUsersFeature)\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    FollowingQuery,\n    t.EarlybirdRequest\n  ] = followingEarlybirdQueryTransformer\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[t.ThriftSearchResult]\n  ] = Seq(FollowingEarlybirdResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    t.ThriftSearchResult,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.id) }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/FollowingEarlybirdQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.following\n\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.home_mixer.product.following.model.FollowingQuery\nimport com.twitter.home_mixer.product.following.param.FollowingParam.ServerMaxResultsParam\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MalformedCursor\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant\nimport com.twitter.search.earlybird.{thriftscala => t}\nimport com.twitter.search.queryparser.query.Conjunction\nimport com.twitter.search.queryparser.query.search.SearchOperator\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.jdk.CollectionConverters.asJavaIterableConverter\n\n@Singleton\ncase class FollowingEarlybirdQueryTransformer @Inject() (clientId: ClientId)\n    extends CandidatePipelineQueryTransformer[FollowingQuery, t.EarlybirdRequest] {\n\n  override def transform(query: FollowingQuery): t.EarlybirdRequest = {\n    val followedUserIds =\n      query.features.map(_.get(SGSFollowedUsersFeature)).getOrElse(Seq.empty).toSet\n    val userId = query.getRequiredUserId\n    val combinedUserIds = userId +: followedUserIds.toSeq\n\n    val baseFollowedUsersSearchOperator = new SearchOperator.Builder()\n      .setType(SearchOperator.Type.FEATURE_VALUE_IN_ACCEPT_LIST_OR_UNSET)\n      .addOperand(EarlybirdFieldConstant.DIRECTED_AT_USER_ID_CSF.getFieldName)\n\n    val followedUsersQuery =\n      baseFollowedUsersSearchOperator.addOperands(combinedUserIds.map(_.toString).asJava).build()\n\n    val searchQuery = query.pipelineCursor\n      .map { cursor =>\n        val sinceIdQuery =\n          (id: Long) => new SearchOperator(SearchOperator.Type.SINCE_ID, id.toString)\n        val maxIdQuery = // max ID is inclusive, so subtract 1\n          (id: Long) => new SearchOperator(SearchOperator.Type.MAX_ID, (id - 1).toString)\n\n        (cursor.cursorType, cursor.id, cursor.gapBoundaryId) match {\n          case (Some(TopCursor), Some(sinceId), _) =>\n            new Conjunction(sinceIdQuery(sinceId), followedUsersQuery)\n          case (Some(BottomCursor), Some(maxId), _) =>\n            new Conjunction(maxIdQuery(maxId), followedUsersQuery)\n          case (Some(GapCursor), Some(maxId), Some(sinceId)) =>\n            new Conjunction(sinceIdQuery(sinceId), maxIdQuery(maxId), followedUsersQuery)\n          case (Some(GapCursor), _, _) =>\n            throw PipelineFailure(MalformedCursor, \"Invalid cursor \" + cursor.toString)\n          case _ => followedUsersQuery\n        }\n      }.getOrElse(followedUsersQuery)\n\n    val metadataOptions = t.ThriftSearchResultMetadataOptions(\n      getInReplyToStatusId = true,\n      getReferencedTweetAuthorId = true,\n      getFromUserId = true\n    )\n\n    t.EarlybirdRequest(\n      searchQuery = t.ThriftSearchQuery(\n        serializedQuery = Some(searchQuery.serialize),\n        fromUserIDFilter64 = Some(combinedUserIds),\n        numResults = query.requestedMaxResults.getOrElse(query.params(ServerMaxResultsParam)),\n        rankingMode = t.ThriftSearchRankingMode.Recency,\n        resultMetadataOptions = Some(metadataOptions),\n        searcherId = query.getOptionalUserId,\n      ),\n      getOlderResults = Some(true), // needed for archive access to older tweets\n      clientRequestID = Some(s\"${Trace.id.traceId}\"),\n      followedUserIds = Some(combinedUserIds),\n      numResultsToReturnAtRoot = Some(query.params(ServerMaxResultsParam)),\n      clientId = Some(clientId.name),\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/FollowingEarlybirdResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.candidate_pipeline\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.search.earlybird.{thriftscala => t}\n\nobject FollowingEarlybirdResponseFeatureTransformer\n    extends CandidateFeatureTransformer[t.ThriftSearchResult] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"FollowingEarlybirdResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    AuthorIdFeature,\n    InReplyToTweetIdFeature,\n    IsRetweetFeature,\n    SourceTweetIdFeature,\n    SourceUserIdFeature,\n  )\n\n  override def transform(candidate: t.ThriftSearchResult): FeatureMap = FeatureMapBuilder()\n    .add(AuthorIdFeature, candidate.tweetypieTweet.flatMap(_.coreData.map(_.userId)))\n    .add(\n      InReplyToTweetIdFeature,\n      candidate.tweetypieTweet.flatMap(_.coreData.flatMap(_.reply.flatMap(_.inReplyToStatusId))))\n    .add(IsRetweetFeature, candidate.metadata.exists(_.isRetweet.contains(true)))\n    .add(SourceTweetIdFeature, candidate.sourceTweetypieTweet.map(_.id))\n    .add(SourceUserIdFeature, candidate.sourceTweetypieTweet.flatMap(_.coreData.map(_.userId)))\n    .build()\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/FollowingMixerPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.following\n\nimport com.twitter.clientapp.{thriftscala => ca}\nimport com.twitter.goldfinch.api.AdsInjectionSurfaceAreas\nimport com.twitter.home_mixer.candidate_pipeline.ConversationServiceCandidatePipelineConfigBuilder\nimport com.twitter.home_mixer.candidate_pipeline.EditedTweetsCandidatePipelineConfig\nimport com.twitter.home_mixer.candidate_pipeline.NewTweetsPillCandidatePipelineConfig\nimport com.twitter.home_mixer.candidate_pipeline.VerifiedPromptCandidatePipelineConfig\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator._\nimport com.twitter.home_mixer.functional_component.selector.UpdateHomeClientEventDetails\nimport com.twitter.home_mixer.functional_component.selector.UpdateNewTweetsPillDecoration\nimport com.twitter.home_mixer.functional_component.side_effect._\nimport com.twitter.home_mixer.model.ClearCacheIncludeInstruction\nimport com.twitter.home_mixer.model.NavigationIncludeInstruction\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableSSPAdsBrandSafetySettingsFeatureHydratorParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.MaxNumberReplaceInstructionsParam\nimport com.twitter.home_mixer.param.HomeMixerFlagName.ScribeClientEventsFlag\nimport com.twitter.home_mixer.product.following.model.FollowingQuery\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.product.following.param.FollowingParam.ClearCache\nimport com.twitter.home_mixer.product.following.param.FollowingParam.EnableFlipInjectionModuleCandidatePipelineParam\nimport com.twitter.home_mixer.product.following.param.FollowingParam.EnablePostContextFeatureHydratorParam\nimport com.twitter.home_mixer.product.following.param.FollowingParam.Navigation\nimport com.twitter.home_mixer.product.following.param.FollowingParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.product.following.param.FollowingParam.StaticParamValueFive\nimport com.twitter.home_mixer.product.following.param.FollowingParam.StaticParamValueZero\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.ads.SSPAdsBrandSafetySettingsFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.async.AsyncQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.impressed_tweets.ImpressedTweetsQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.location.UserLocationQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated.ParamGatedQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.FlipPromptDependentCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.selector.FlipPromptDynamicInsertionPosition\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowArmCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.premarshaller.urt.UrtDomainMarshaller\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ClearCacheInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.NavigationInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedBottomCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedCursorIdSelector.TweetIdSelector\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedGapCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedTopCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceAllEntries\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceEntryInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ShowAlertInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ShowCoverInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.StaticTimelineScribeConfigBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtMetadataBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.earlybird.EarlybirdGapIncludeInstruction\nimport com.twitter.product_mixer.component_library.selector.DropMaxCandidates\nimport com.twitter.product_mixer.component_library.selector.DropMaxModuleItemCandidates\nimport com.twitter.product_mixer.component_library.selector.DropModuleTooFewModuleItemResults\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.component_library.selector.InsertDynamicPositionResults\nimport com.twitter.product_mixer.component_library.selector.InsertFixedPositionResults\nimport com.twitter.product_mixer.component_library.selector.SelectConditionally\nimport com.twitter.product_mixer.component_library.selector.UpdateSortCandidates\nimport com.twitter.product_mixer.component_library.selector.ads.AdsInjector\nimport com.twitter.product_mixer.component_library.selector.ads.InsertAdResults\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.UrtTransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.MixerPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineConfig\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass FollowingMixerPipelineConfig @Inject() (\n  followingEarlybirdCandidatePipelineConfig: FollowingEarlybirdCandidatePipelineConfig,\n  conversationServiceCandidatePipelineConfigBuilder: ConversationServiceCandidatePipelineConfigBuilder[\n    FollowingQuery\n  ],\n  followingAdsCandidatePipelineBuilder: FollowingAdsCandidatePipelineBuilder,\n  followingWhoToFollowCandidatePipelineConfigBuilder: FollowingWhoToFollowCandidatePipelineConfigBuilder,\n  flipPromptDependentCandidatePipelineConfigBuilder: FlipPromptDependentCandidatePipelineConfigBuilder,\n  editedTweetsCandidatePipelineConfig: EditedTweetsCandidatePipelineConfig,\n  newTweetsPillCandidatePipelineConfig: NewTweetsPillCandidatePipelineConfig[FollowingQuery],\n  verifiedPromptCandidatePipelineConfig: VerifiedPromptCandidatePipelineConfig,\n  dismissInfoQueryFeatureHydrator: DismissInfoQueryFeatureHydrator,\n  gizmoduckUserQueryFeatureHydrator: GizmoduckUserQueryFeatureHydrator,\n  persistenceStoreQueryFeatureHydrator: PersistenceStoreQueryFeatureHydrator,\n  rateLimitQueryFeatureHydrator: RateLimitQueryFeatureHydrator,\n  requestQueryFeatureHydrator: RequestQueryFeatureHydrator[FollowingQuery],\n  sgsFollowedUsersQueryFeatureHydrator: SGSFollowedUsersQueryFeatureHydrator,\n  memcacheTweetImpressionsQueryFeatureHydrator: ImpressedTweetsQueryFeatureHydrator,\n  lastNonPollingTimeQueryFeatureHydrator: LastNonPollingTimeQueryFeatureHydrator,\n  userLocationQueryFeatureHydrator: UserLocationQueryFeatureHydrator,\n  userSubscriptionQueryFeatureHydrator: UserSubscriptionQueryFeatureHydrator,\n  sspAdsBrandSafetySettingsFeatureHydrator: SSPAdsBrandSafetySettingsFeatureHydrator,\n  adsInjector: AdsInjector,\n  updateLastNonPollingTimeSideEffect: UpdateLastNonPollingTimeSideEffect[FollowingQuery, Timeline],\n  publishClientSentImpressionsEventBusSideEffect: PublishClientSentImpressionsEventBusSideEffect,\n  updateTimelinesPersistenceStoreSideEffect: UpdateTimelinesPersistenceStoreSideEffect,\n  truncateTimelinesPersistenceStoreSideEffect: TruncateTimelinesPersistenceStoreSideEffect,\n  homeTimelineServedCandidatesSideEffect: HomeScribeServedCandidatesSideEffect,\n  clientEventsScribeEventPublisher: EventPublisher[ca.LogEvent],\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter],\n  urtTransportMarshaller: UrtTransportMarshaller,\n  @Flag(ScribeClientEventsFlag) enableScribeClientEvents: Boolean)\n    extends MixerPipelineConfig[FollowingQuery, Timeline, urt.TimelineResponse] {\n\n  override val identifier: MixerPipelineIdentifier = MixerPipelineIdentifier(\"Following\")\n\n  private val dependentCandidatesStep = MixerPipelineConfig.dependentCandidatePipelinesStep\n  private val resultSelectorsStep = MixerPipelineConfig.resultSelectorsStep\n\n  override val fetchQueryFeatures: Seq[QueryFeatureHydrator[FollowingQuery]] = Seq(\n    requestQueryFeatureHydrator,\n    sgsFollowedUsersQueryFeatureHydrator,\n    rateLimitQueryFeatureHydrator,\n    gizmoduckUserQueryFeatureHydrator,\n    userSubscriptionQueryFeatureHydrator,\n    ParamGatedQueryFeatureHydrator(\n      EnableSSPAdsBrandSafetySettingsFeatureHydratorParam,\n      sspAdsBrandSafetySettingsFeatureHydrator\n    ),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, dismissInfoQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, persistenceStoreQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, lastNonPollingTimeQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, userLocationQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(resultSelectorsStep, memcacheTweetImpressionsQueryFeatureHydrator),\n  )\n\n  private val earlybirdCandidatePipelineScope =\n    SpecificPipeline(followingEarlybirdCandidatePipelineConfig.identifier)\n\n  private val conversationServiceCandidatePipelineConfig =\n    conversationServiceCandidatePipelineConfigBuilder.build(\n      earlybirdCandidatePipelineScope,\n      hmt.ServedType.FollowingInNetwork,\n      EnablePostContextFeatureHydratorParam\n    )\n\n  private val followingAdsCandidatePipelineConfig = followingAdsCandidatePipelineBuilder.build()\n\n  private val followingWhoToFollowCandidatePipelineConfig =\n    followingWhoToFollowCandidatePipelineConfigBuilder.build(earlybirdCandidatePipelineScope)\n\n  private val flipPromptCandidatePipelineConfig =\n    flipPromptDependentCandidatePipelineConfigBuilder.build[FollowingQuery](\n      supportedClientParam = Some(EnableFlipInjectionModuleCandidatePipelineParam)\n    )\n\n  override val candidatePipelines: Seq[CandidatePipelineConfig[FollowingQuery, _, _, _]] = Seq(\n    followingEarlybirdCandidatePipelineConfig,\n    followingAdsCandidatePipelineConfig\n  )\n\n  override val dependentCandidatePipelines: Seq[\n    DependentCandidatePipelineConfig[FollowingQuery, _, _, _]\n  ] = Seq(\n    conversationServiceCandidatePipelineConfig,\n    followingWhoToFollowCandidatePipelineConfig,\n    flipPromptCandidatePipelineConfig,\n    editedTweetsCandidatePipelineConfig,\n    newTweetsPillCandidatePipelineConfig,\n    verifiedPromptCandidatePipelineConfig\n  )\n\n  override val failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy] = Map(\n    followingAdsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    followingWhoToFollowCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    flipPromptCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    editedTweetsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    newTweetsPillCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n  )\n\n  override val resultSelectors: Seq[Selector[FollowingQuery]] = Seq(\n    UpdateSortCandidates(\n      ordering = CandidatesUtil.reverseChronTweetsOrdering,\n      candidatePipeline = conversationServiceCandidatePipelineConfig.identifier\n    ),\n    DropMaxCandidates(\n      candidatePipeline = editedTweetsCandidatePipelineConfig.identifier,\n      maxSelectionsParam = MaxNumberReplaceInstructionsParam\n    ),\n    DropMaxCandidates(\n      candidatePipeline = conversationServiceCandidatePipelineConfig.identifier,\n      maxSelectionsParam = ServerMaxResultsParam\n    ),\n    DropModuleTooFewModuleItemResults(\n      candidatePipeline = followingWhoToFollowCandidatePipelineConfig.identifier,\n      minModuleItemsParam = StaticParam(WhoToFollowArmCandidatePipelineConfig.MinCandidatesSize)\n    ),\n    DropMaxModuleItemCandidates(\n      candidatePipeline = followingWhoToFollowCandidatePipelineConfig.identifier,\n      maxModuleItemsParam = StaticParam(WhoToFollowArmCandidatePipelineConfig.MaxCandidatesSize)\n    ),\n    InsertAppendResults(candidatePipeline = conversationServiceCandidatePipelineConfig.identifier),\n    InsertFixedPositionResults(\n      candidatePipeline = verifiedPromptCandidatePipelineConfig.identifier,\n      positionParam = StaticParamValueZero\n    ),\n    InsertDynamicPositionResults(\n      candidatePipeline = flipPromptCandidatePipelineConfig.identifier,\n      dynamicInsertionPosition = FlipPromptDynamicInsertionPosition(StaticParamValueZero)\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = followingWhoToFollowCandidatePipelineConfig.identifier,\n      positionParam = StaticParamValueFive\n    ),\n    InsertAdResults(\n      surfaceAreaName = AdsInjectionSurfaceAreas.HomeTimeline,\n      adsInjector = adsInjector.forSurfaceArea(AdsInjectionSurfaceAreas.HomeTimeline),\n      adsCandidatePipeline = followingAdsCandidatePipelineConfig.identifier\n    ),\n    // This selector must come after the tweets are inserted into the results\n    UpdateNewTweetsPillDecoration(\n      pipelineScope = SpecificPipelines(\n        conversationServiceCandidatePipelineConfig.identifier,\n        newTweetsPillCandidatePipelineConfig.identifier\n      ),\n      stringCenter = stringCenterProvider.get(),\n      seeNewTweetsString = externalStrings.seeNewTweetsString,\n      tweetedString = externalStrings.tweetedString\n    ),\n    InsertAppendResults(candidatePipeline = editedTweetsCandidatePipelineConfig.identifier),\n    SelectConditionally(\n      selector =\n        InsertAppendResults(candidatePipeline = newTweetsPillCandidatePipelineConfig.identifier),\n      includeSelector = (_, _, results) => CandidatesUtil.containsType[TweetCandidate](results)\n    ),\n    UpdateHomeClientEventDetails(\n      candidatePipelines = Set(conversationServiceCandidatePipelineConfig.identifier)\n    )\n  )\n\n  private val homeScribeClientEventSideEffect = HomeScribeClientEventSideEffect(\n    enableScribeClientEvents = enableScribeClientEvents,\n    logPipelinePublisher = clientEventsScribeEventPublisher,\n    injectedTweetsCandidatePipelineIdentifiers =\n      Seq(conversationServiceCandidatePipelineConfig.identifier),\n    adsCandidatePipelineIdentifier = Some(followingAdsCandidatePipelineConfig.identifier),\n    whoToFollowCandidatePipelineIdentifier =\n      Some(followingWhoToFollowCandidatePipelineConfig.identifier),\n  )\n\n  override val resultSideEffects: Seq[PipelineResultSideEffect[FollowingQuery, Timeline]] = Seq(\n    homeScribeClientEventSideEffect,\n    homeTimelineServedCandidatesSideEffect,\n    publishClientSentImpressionsEventBusSideEffect,\n    truncateTimelinesPersistenceStoreSideEffect,\n    updateLastNonPollingTimeSideEffect,\n    updateTimelinesPersistenceStoreSideEffect,\n  )\n\n  override val domainMarshaller: DomainMarshaller[FollowingQuery, Timeline] = {\n    val instructionBuilders = Seq(\n      ClearCacheInstructionBuilder(\n        ClearCacheIncludeInstruction(\n          ClearCache.PtrEnableParam,\n          ClearCache.ColdStartEnableParam,\n          ClearCache.WarmStartEnableParam,\n          ClearCache.ManualRefreshEnableParam,\n          ClearCache.NavigateEnableParam,\n          ClearCache.MinEntriesParam\n        )),\n      ReplaceEntryInstructionBuilder(ReplaceAllEntries),\n      AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder(),\n      ShowAlertInstructionBuilder(),\n      ShowCoverInstructionBuilder(),\n      NavigationInstructionBuilder(\n        NavigationIncludeInstruction(\n          Navigation.PtrEnableParam,\n          Navigation.ColdStartEnableParam,\n          Navigation.WarmStartEnableParam,\n          Navigation.ManualRefreshEnableParam,\n          Navigation.NavigateEnableParam\n        ))\n    )\n\n    val topCursorBuilder = OrderedTopCursorBuilder(TweetIdSelector)\n    val bottomCursorBuilder =\n      OrderedBottomCursorBuilder(TweetIdSelector, EarlybirdGapIncludeInstruction.inverse())\n    val gapCursorBuilder = OrderedGapCursorBuilder(TweetIdSelector, EarlybirdGapIncludeInstruction)\n\n    val scribeConfigBuilder =\n      StaticTimelineScribeConfigBuilder(TimelineScribeConfig(Some(\"following\"), None, None))\n    val metadataBuilder = UrtMetadataBuilder(scribeConfigBuilder = Some(scribeConfigBuilder))\n\n    UrtDomainMarshaller(\n      instructionBuilders = instructionBuilders,\n      metadataBuilder = Some(metadataBuilder),\n      cursorBuilders = Seq(topCursorBuilder, bottomCursorBuilder, gapCursorBuilder)\n    )\n  }\n\n  override val transportMarshaller: TransportMarshaller[Timeline, urt.TimelineResponse] =\n    urtTransportMarshaller\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/FollowingProductPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.following\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.home_mixer.model.request.FollowingProductContext\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.product.following.model.FollowingQuery\nimport com.twitter.home_mixer.product.following.param.FollowingParam.EnableDependentAdsParam\nimport com.twitter.home_mixer.product.following.param.FollowingParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.product.following.param.FollowingParamConfig\nimport com.twitter.home_mixer.service.HomeMixerAccessPolicy.DefaultHomeMixerAccessPolicy\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig.DefaultNotificationGroup\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.timelines.ChronologicalCursorUnmarshaller\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AccessPolicy\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.common.alert.EmptyResponseRateAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.LatencyAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.P99\nimport com.twitter.product_mixer.core.functional_component.common.alert.SuccessRateAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.ThroughputAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfAbove\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfBelow\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfLatencyAbove\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ProductPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineConfig\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.BadRequest\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MalformedCursor\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineConfig\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.product_mixer.core.util.SortIndexBuilder\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport com.twitter.timelines.util.RequestCursorSerializer\nimport com.twitter.util.Time\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FollowingProductPipelineConfig @Inject() (\n  followingDependentAdsMixerPipelineConfig: FollowingDependentAdsMixerPipelineConfig,\n  followingMixerPipelineConfig: FollowingMixerPipelineConfig,\n  followingParamConfig: FollowingParamConfig)\n    extends ProductPipelineConfig[HomeMixerRequest, FollowingQuery, urt.TimelineResponse] {\n\n  override val identifier: ProductPipelineIdentifier = ProductPipelineIdentifier(\"Following\")\n\n  override val product: Product = FollowingProduct\n  override val paramConfig: ProductParamConfig = followingParamConfig\n\n  override def pipelineQueryTransformer(\n    request: HomeMixerRequest,\n    params: Params\n  ): FollowingQuery = {\n    val context = request.productContext match {\n      case Some(context: FollowingProductContext) => context\n      case _ => throw PipelineFailure(BadRequest, \"FollowingProductContext not found\")\n    }\n\n    val debugOptions = request.debugParams.flatMap(_.debugOptions)\n\n    /**\n     * Unlike other clients, newly created tweets on Android have the sort index set to the current\n     * time instead of the top sort index + 1, so these tweets get stuck at the top of the timeline\n     * if subsequent timeline responses use the sort index from the previous response instead of\n     * the current time.\n     */\n    val pipelineCursor = request.serializedRequestCursor.flatMap { cursor =>\n      Try(UrtCursorSerializer.deserializeOrderedCursor(cursor))\n        .getOrElse(ChronologicalCursorUnmarshaller(RequestCursorSerializer.deserialize(cursor)))\n        .map {\n          case UrtOrderedCursor(_, id, Some(GapCursor), gapBoundaryId)\n              if id.isEmpty || gapBoundaryId.isEmpty =>\n            throw PipelineFailure(MalformedCursor, \"Gap Cursor bounds not defined\")\n          case topCursor @ UrtOrderedCursor(_, _, Some(TopCursor), _) =>\n            val queryTime = debugOptions.flatMap(_.requestTimeOverride).getOrElse(Time.now)\n            topCursor.copy(initialSortIndex = SortIndexBuilder.timeToId(queryTime))\n          case cursor => cursor\n        }\n    }\n\n    FollowingQuery(\n      params = params,\n      clientContext = request.clientContext,\n      features = None,\n      pipelineCursor = pipelineCursor,\n      requestedMaxResults = Some(params(ServerMaxResultsParam)),\n      debugOptions = debugOptions,\n      deviceContext = context.deviceContext,\n      seenTweetIds = context.seenTweetIds,\n      dspClientContext = context.dspClientContext\n    )\n  }\n\n  override val pipelines: Seq[PipelineConfig] =\n    Seq(followingMixerPipelineConfig, followingDependentAdsMixerPipelineConfig)\n\n  override def pipelineSelector(query: FollowingQuery): ComponentIdentifier =\n    if (query.params(EnableDependentAdsParam)) followingDependentAdsMixerPipelineConfig.identifier\n    else followingMixerPipelineConfig.identifier\n\n  override val alerts: Seq[Alert] = Seq(\n    SuccessRateAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfBelow(99.9, 20, 30),\n      criticalPredicate = TriggerIfBelow(99.9, 30, 30),\n    ),\n    LatencyAlert(\n      notificationGroup = DefaultNotificationGroup,\n      percentile = P99,\n      warnPredicate = TriggerIfLatencyAbove(1200.millis, 15, 30),\n      criticalPredicate = TriggerIfLatencyAbove(1500.millis, 15, 30)\n    ),\n    ThroughputAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfAbove(18000),\n      criticalPredicate = TriggerIfAbove(20000)\n    ),\n    EmptyResponseRateAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfAbove(65),\n      criticalPredicate = TriggerIfAbove(80)\n    )\n  )\n\n  override val debugAccessPolicies: Set[AccessPolicy] = DefaultHomeMixerAccessPolicy\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/FollowingWhoToFollowCandidatePipelineConfigBuilder.scala",
    "content": "package com.twitter.home_mixer.product.following\n\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeWhoToFollowFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.functional_component.gate.DismissFatigueGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.model.HomeFeatures.DismissInfoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.WhoToFollowExcludedUserIdsFeature\nimport com.twitter.home_mixer.product.following.model.FollowingQuery\nimport com.twitter.home_mixer.product.following.param.FollowingParam.EnableWhoToFollowCandidatePipelineParam\nimport com.twitter.home_mixer.product.following.param.FollowingParam.WhoToFollowDisplayLocationParam\nimport com.twitter.home_mixer.product.following.param.FollowingParam.WhoToFollowDisplayTypeIdParam\nimport com.twitter.home_mixer.product.following.param.FollowingParam.WhoToFollowMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.following.param.FollowingParam.WhoToFollowUserDisplayTypeIdParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ParamWhoToFollowModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.gate.NonEmptyCandidatesGate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.ParamWhoToFollowUserDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowArmCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowArmDependentCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowArmDependentCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport com.twitter.timelineservice.suggests.thriftscala.SuggestType\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FollowingWhoToFollowCandidatePipelineConfigBuilder @Inject() (\n  whoToFollowArmDependentCandidatePipelineConfigBuilder: WhoToFollowArmDependentCandidatePipelineConfigBuilder,\n  homeWhoToFollowFeedbackActionInfoBuilder: HomeWhoToFollowFeedbackActionInfoBuilder) {\n\n  def build(\n    requiredNonEmptyPipelines: CandidateScope\n  ): WhoToFollowArmDependentCandidatePipelineConfig[FollowingQuery] = {\n    val gates: Seq[BaseGate[PipelineQuery]] = Seq(\n      TimelinesPersistenceStoreLastInjectionGate(\n        WhoToFollowMinInjectionIntervalParam,\n        EntityIdType.WhoToFollow\n      ),\n      DismissFatigueGate(SuggestType.WhoToFollow, DismissInfoFeature),\n      NonEmptyCandidatesGate(requiredNonEmptyPipelines)\n    )\n\n    whoToFollowArmDependentCandidatePipelineConfigBuilder.build[FollowingQuery](\n      identifier = WhoToFollowArmCandidatePipelineConfig.identifier,\n      supportedClientParam = Some(EnableWhoToFollowCandidatePipelineParam),\n      alerts = alerts,\n      gates = gates,\n      moduleDisplayTypeBuilder =\n        ParamWhoToFollowModuleDisplayTypeBuilder(WhoToFollowDisplayTypeIdParam),\n      feedbackActionInfoBuilder = Some(homeWhoToFollowFeedbackActionInfoBuilder),\n      userDisplayTypeBuilder =\n        ParamWhoToFollowUserDisplayTypeBuilder(WhoToFollowUserDisplayTypeIdParam),\n      displayLocationParam = StaticParam(WhoToFollowDisplayLocationParam.default),\n      excludedUserIdsFeature = Some(WhoToFollowExcludedUserIdsFeature),\n      profileUserIdFeature = None\n    )\n  }\n\n  private val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(70),\n    HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline\",\n        \"stringcenter/client\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model/FollowingQuery.scala",
    "content": "package com.twitter.home_mixer.product.following.model\n\nimport com.twitter.adserver.thriftscala.HomeTimelineType\nimport com.twitter.adserver.thriftscala.TimelineRequestParams\nimport com.twitter.home_mixer.model.HomeAdsQuery\nimport com.twitter.dspbidder.commons.{thriftscala => dsp}\nimport com.twitter.home_mixer.model.request.DeviceContext\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.model.request.HasSeenTweetIds\nimport com.twitter.home_mixer.model.request.FollowingProduct\nimport com.twitter.onboarding.task.service.{thriftscala => ots}\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.HasFlipInjectionParams\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.request._\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Params\n\ncase class FollowingQuery(\n  override val params: Params,\n  override val clientContext: ClientContext,\n  override val pipelineCursor: Option[UrtOrderedCursor],\n  override val requestedMaxResults: Option[Int],\n  override val debugOptions: Option[DebugOptions],\n  override val features: Option[FeatureMap],\n  override val deviceContext: Option[DeviceContext],\n  override val seenTweetIds: Option[Seq[Long]],\n  override val dspClientContext: Option[dsp.DspClientContext])\n    extends PipelineQuery\n    with HasPipelineCursor[UrtOrderedCursor]\n    with HasDeviceContext\n    with HasSeenTweetIds\n    with HasFlipInjectionParams\n    with HomeAdsQuery {\n  override val product: Product = FollowingProduct\n\n  override def withFeatureMap(features: FeatureMap): FollowingQuery =\n    copy(features = Some(features))\n\n  override val timelineRequestParams: Option[TimelineRequestParams] =\n    Some(TimelineRequestParams(homeTimelineType = Some(HomeTimelineType.HomeLatest)))\n\n  // Fields below are used for FLIP Injection in Onboarding Task Service (OTS)\n  override val displayLocation: ots.DisplayLocation = ots.DisplayLocation.HomeLatestTimeline\n  override val rankingDisablerWithLatestControlsAvailable: Option[Boolean] = None\n  override val isEmptyState: Option[Boolean] = None\n  override val isFirstRequestAfterSignup: Option[Boolean] = None\n  override val isEndOfTimeline: Option[Boolean] = None\n  override val timelineId: Option[Long] = None\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model/HomeMixerExternalStrings.scala",
    "content": "package com.twitter.home_mixer.product.following.model\n\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.ExternalStringRegistry\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass HomeMixerExternalStrings @Inject() (\n  @ProductScoped externalStringRegistryProvider: Provider[ExternalStringRegistry]) {\n  val seeNewTweetsString =\n    externalStringRegistryProvider.get().createProdString(\"SeeNewTweets\")\n  val tweetedString =\n    externalStringRegistryProvider.get().createProdString(\"Tweeted\")\n  val muteUserString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.muteUser\")\n  val muteUserConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.muteUserConfirmation\")\n  val blockUserString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.blockUser\")\n  val blockUserConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.blockUserConfirmation\")\n  val unfollowUserString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.unfollowUser\")\n  val unfollowUserConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.unfollowUserConfirmation\")\n  val reportTweetString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.reportTweet\")\n  val genericString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.generic\")\n  val genericConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.genericConfirmation\")\n  val relevantString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.relevant\")\n  val relevantConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.relevantConfirmation\")\n  val dontLikeString = externalStringRegistryProvider.get().createProdString(\"Feedback.dontLike\")\n  val dontLikeConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.dontLikeConfirmation\")\n  val showFewerTweetsString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.showFewerTweets\")\n  val showFewerTweetsConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.showFewerTweetsConfirmation\")\n  val showFewerRetweetsString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.showFewerRetweets\")\n  val showFewerRetweetsConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.showFewerRetweetsConfirmation\")\n  val notRelevantString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.notRelevant\")\n  val notRelevantConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.notRelevantConfirmation\")\n  val hatefulString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.hateful\")\n  val hatefulConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.hatefulConfirmation\")\n  val boringString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.boring\")\n  val boringConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.boringConfirmation\")\n  val confusingString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.confusing\")\n  val confusingConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.confusingConfirmation\")\n  val clickbaitString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.clickbait\")\n  val clickbaitConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.clickbaitConfirmation\")\n  val ragebaitString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.ragebait\")\n  val ragebaitConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.ragebaitConfirmation\")\n  val regretString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.regret\")\n  val regretConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.regretConfirmation\")\n  val neutralString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.neutral\")\n  val neutralConfirmationString =\n    externalStringRegistryProvider.get().createProdString(\"Feedback.neutralConfirmation\")\n\n  val seeMoreString =\n    externalStringRegistryProvider.get().createProdString(\"PagedCarouselModule.showMoreText\")\n  val seeLessString =\n    externalStringRegistryProvider.get().createProdString(\"PagedCarouselModule.showLessText\")\n\n  val socialContextOneUserLikedString =\n    externalStringRegistryProvider.get().createProdString(\"SocialContext.oneUserLiked\")\n  val socialContextTwoUsersLikedString =\n    externalStringRegistryProvider.get().createProdString(\"SocialContext.twoUsersLiked\")\n  val socialContextMoreUsersLikedString =\n    externalStringRegistryProvider.get().createProdString(\"SocialContext.moreUsersLiked\")\n  val socialContextLikedByTimelineTitle =\n    externalStringRegistryProvider.get().createProdString(\"SocialContext.likedByTimelineTitle\")\n\n  val socialContextOneUserFollowsString =\n    externalStringRegistryProvider.get().createProdString(\"SocialContext.oneUserFollows\")\n  val socialContextTwoUsersFollowString =\n    externalStringRegistryProvider.get().createProdString(\"SocialContext.twoUsersFollow\")\n  val socialContextMoreUsersFollowString =\n    externalStringRegistryProvider.get().createProdString(\"SocialContext.moreUsersFollow\")\n  val socialContextFollowedByTimelineTitle =\n    externalStringRegistryProvider.get().createProdString(\"SocialContext.followedByTimelineTitle\")\n\n  val socialContextYouMightLikeString =\n    externalStringRegistryProvider.get().createProdString(\"SocialContext.youMightLike\")\n\n  val socialContextExtendedReply =\n    externalStringRegistryProvider.get().createProdString(\"SocialContext.extendedReply\")\n  val socialContextReceivedReply =\n    externalStringRegistryProvider.get().createProdString(\"SocialContext.receivedReply\")\n\n  val socialContextPopularInYourAreaString =\n    externalStringRegistryProvider.get().createProdString(\"PopgeoTweet.socialProof\")\n\n  val ownedSubscribedListsModuleHeaderString =\n    externalStringRegistryProvider.get().createProdString(\"OwnedSubscribedListModule.header\")\n\n  val CommunityToJoinHeaderString =\n    externalStringRegistryProvider.get().createProdString(\"CommunityToJoinModule.header\")\n  val CommunityToJoinFooterString =\n    externalStringRegistryProvider.get().createProdString(\"CommunityToJoinModule.footer\")\n\n  val RecommendedJobHeaderString =\n    externalStringRegistryProvider.get().createProdString(\"RecommendedJobModule.header\")\n  val RecommendedJobFooterString =\n    externalStringRegistryProvider.get().createProdString(\"RecommendedJobModule.footer\")\n\n  val RecommendedRecruitingOrganizationHeaderString =\n    externalStringRegistryProvider\n      .get().createProdString(\"RecommendedRecruitingOrganizationModule.header\")\n  val RecommendedRecruitingOrganizationFooterString =\n    externalStringRegistryProvider\n      .get().createProdString(\"RecommendedRecruitingOrganizationModule.footer\")\n\n  val BookmarksHeaderString =\n    externalStringRegistryProvider.get().createProdString(\"RecentBookmarks.header\")\n\n  val PinnedTweetsHeaderString =\n    externalStringRegistryProvider.get().createProdString(\"PinnedTweetsModule.header\")\n  val BroadcastedPinnedTweetSocialContextString =\n    externalStringRegistryProvider.get().createProdString(\"BroadcastedPinnedTweet.context\")\n  val VideoCarouselHeaderString =\n    externalStringRegistryProvider.get().createProdString(\"VideoCarouselModule.header\")\n  val VideoCarouselFooterString =\n    externalStringRegistryProvider.get().createProdString(\"VideoCarouselModule.footer\")\n\n  val TrendingString =\n    externalStringRegistryProvider.get().createProdString(\"Trending\")\n  val KeywordTrendsTweetCountDescriptionString =\n    externalStringRegistryProvider.get().createProdString(\"KeywordTrends.tweetCountDescription\")\n\n  val NewsHeaderString =\n    externalStringRegistryProvider.get().createProdString(\"News.header\")\n  val NewsFooterString =\n    externalStringRegistryProvider.get().createProdString(\"News.footer\")\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/param/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param/decider\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/param/FollowingParam.scala",
    "content": "package com.twitter.home_mixer.product.following.param\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.WhoToFollowModuleDisplayType\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowUserDisplayType\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.util.Duration\n\nobject FollowingParam {\n  val SupportedClientFSName = \"following_supported_client\"\n  val StaticParamValueZero = StaticParam(0)\n  val StaticParamValueFive = StaticParam(5)\n\n  object ServerMaxResultsParam\n      extends FSBoundedParam[Int](\n        name = \"following_server_max_results\",\n        default = 100,\n        min = 1,\n        max = 500\n      )\n\n  object EnableWhoToFollowCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"following_enable_who_to_follow\",\n        default = true\n      )\n\n  object EnableFlipInjectionModuleCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"following_enable_flip_inline_injection_module\",\n        default = true\n      )\n\n  object EnablePostContextFeatureHydratorParam\n      extends FSParam[Boolean](\n        name = \"following_enable_post_context_feature_hydrator\",\n        default = false\n      )\n\n  object WhoToFollowMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"following_who_to_follow_min_injection_interval_in_minutes\",\n        default = 1800.minutes,\n        min = 0.minutes,\n        max = 6000.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object WhoToFollowDisplayTypeIdParam\n      extends FSEnumParam[WhoToFollowModuleDisplayType.type](\n        name = \"following_enable_who_to_follow_display_type_id\",\n        default = WhoToFollowModuleDisplayType.Vertical,\n        enum = WhoToFollowModuleDisplayType\n      )\n\n  object WhoToFollowUserDisplayTypeIdParam\n      extends FSEnumParam[WhoToFollowUserDisplayType.type](\n        name = \"following_enable_who_to_follow_user_display_type_id\",\n        default = WhoToFollowUserDisplayType.User,\n        enum = WhoToFollowUserDisplayType\n      )\n\n  object WhoToFollowDisplayLocationParam\n      extends FSParam[String](\n        name = \"following_who_to_follow_display_location\",\n        default = \"timeline_reverse_chron\"\n      )\n\n  object EnableFastAds\n      extends FSParam[Boolean](\n        name = \"following_enable_fast_ads\",\n        default = true\n      )\n\n  object EnableDependentAdsParam\n      extends FSParam[Boolean](\n        name = \"following_enable_dependent_ads\",\n        default = true\n      )\n\n  object EnableNavigationInstructionParam\n      extends FSParam[Boolean](\n        name = \"following_enable_navigation_instruction\",\n        default = false\n      )\n\n  object ClearCache {\n    object PtrEnableParam\n        extends FSParam[Boolean](\n          name = \"following_clear_cache_ptr_enable\",\n          default = false\n        )\n\n    object ColdStartEnableParam\n        extends FSParam[Boolean](\n          name = \"following_clear_cache_cold_start_enable\",\n          default = false\n        )\n\n    object WarmStartEnableParam\n        extends FSParam[Boolean](\n          name = \"following_clear_cache_warm_start_enable\",\n          default = false\n        )\n\n    object ManualRefreshEnableParam\n        extends FSParam[Boolean](\n          name = \"following_clear_cache_manual_refresh_enable\",\n          default = false\n        )\n\n    object NavigateEnableParam\n        extends FSParam[Boolean](\n          name = \"following_clear_cache_navigate_enable\",\n          default = false\n        )\n\n    case object MinEntriesParam\n        extends FSBoundedParam[Int](\n          name = \"following_clear_cache_min_entries\",\n          default = 10,\n          min = 0,\n          max = 35\n        )\n  }\n\n  object Navigation {\n    object PtrEnableParam\n        extends FSParam[Boolean](\n          name = \"following_navigation_ptr_enable\",\n          default = false\n        )\n\n    object ColdStartEnableParam\n        extends FSParam[Boolean](\n          name = \"following_navigation_cold_start_enable\",\n          default = false\n        )\n\n    object WarmStartEnableParam\n        extends FSParam[Boolean](\n          name = \"following_navigation_warm_start_enable\",\n          default = false\n        )\n\n    object ManualRefreshEnableParam\n        extends FSParam[Boolean](\n          name = \"following_navigation_manual_refresh_enable\",\n          default = false\n        )\n\n    object NavigateEnableParam\n        extends FSParam[Boolean](\n          name = \"following_navigation_navigate_enable\",\n          default = false\n        )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/param/FollowingParamConfig.scala",
    "content": "package com.twitter.home_mixer.product.following.param\n\nimport com.twitter.home_mixer.param.decider.DeciderKey\nimport com.twitter.home_mixer.product.following.param.FollowingParam._\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.servo.decider.DeciderKeyName\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FollowingParamConfig @Inject() () extends ProductParamConfig {\n  override val enabledDeciderKey: DeciderKeyName = DeciderKey.EnableFollowingProduct\n  override val supportedClientFSName: String = SupportedClientFSName\n\n  override val booleanFSOverrides = Seq(\n    ClearCache.PtrEnableParam,\n    ClearCache.ColdStartEnableParam,\n    ClearCache.WarmStartEnableParam,\n    ClearCache.ManualRefreshEnableParam,\n    ClearCache.NavigateEnableParam,\n    EnableFlipInjectionModuleCandidatePipelineParam,\n    EnableWhoToFollowCandidatePipelineParam,\n    EnablePostContextFeatureHydratorParam,\n    EnableFastAds,\n    EnableDependentAdsParam,\n    EnableNavigationInstructionParam,\n    Navigation.PtrEnableParam,\n    Navigation.ColdStartEnableParam,\n    Navigation.WarmStartEnableParam,\n    Navigation.ManualRefreshEnableParam,\n    Navigation.NavigateEnableParam,\n  )\n\n  override val boundedIntFSOverrides = Seq(\n    ClearCache.MinEntriesParam,\n    ServerMaxResultsParam\n  )\n\n  override val stringFSOverrides = Seq(WhoToFollowDisplayLocationParam)\n\n  override val boundedDurationFSOverrides = Seq(WhoToFollowMinInjectionIntervalParam)\n\n  override val enumFSOverrides = Seq(\n    WhoToFollowDisplayTypeIdParam,\n    WhoToFollowUserDisplayTypeIdParam\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/api\",\n        \"communities-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"events-recos/events-recos-service/src/main/thrift:events-recos-thrift-scala\",\n        \"geoduck/util/country\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/filter\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/query_transformer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/response_transformer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/scorer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/selector\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/side_effect\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/earlybird\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/entry_point_pivot\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/location\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/frame\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/param_gated\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_is_nsfw\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_visibility_reason\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/async\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/param_gated\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/pivot\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/trends_events\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/communities_to_join\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/selector\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/jetfuel_entry_point\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/job\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/recruiting_organization\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_subscribe_module\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/cursor/timelines\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/product_pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/request\",\n        \"recruiting/candidate-service/src/main/thrift:thrift-scala\",\n        \"recruiting/src/main/thrift:thrift-scala\",\n        \"src/thrift/com/twitter/frigate/bookmarks:bookmarks-thrift-scala\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"strato/config/columns/events/entryPoint:entryPoint-strato-client\",\n        \"strato/config/columns/events/experiences/grok:grok-strato-client\",\n        \"strato/config/columns/trendsai/ordered:ordered-strato-client\",\n        \"strato/config/src/thrift/com/twitter/strato/columns/jetfuel:jetfuel-scala\",\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate\",\n    ],\n    exports = [\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouAdsCandidatePipelineBuilder.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.adserver.{thriftscala => ads}\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeAdsClientEventDetailsBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator.NoAdsTierFeature\nimport com.twitter.home_mixer.functional_component.gate.ExcludeSoftUserGate\nimport com.twitter.home_mixer.functional_component.gate.ExcludeSyntheticUserGate\nimport com.twitter.home_mixer.param.HomeGlobalParams\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableAdvertiserBrandSafetySettingsFeatureHydratorParam\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.AdsNumOrganicItemsParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.candidate_source.ads.AdsProdThriftCandidateSource\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref.ContextualTweetRefBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.ad.AdsCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.ads.AdvertiserBrandSafetySettingsFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.gate.FeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.AdsCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.AdsCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.StaticAdsDisplayLocationBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.ValidAdImpressionIdFilter\nimport com.twitter.product_mixer.core.gate.ParamNotGate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.TimelineHomePromotedHydrationSafetyLevel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.TweetHydrationContext\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouAdsCandidatePipelineBuilder @Inject() (\n  adsCandidatePipelineConfigBuilder: AdsCandidatePipelineConfigBuilder,\n  adsCandidateSource: AdsProdThriftCandidateSource,\n  advertiserBrandSafetySettingsFeatureHydrator: AdvertiserBrandSafetySettingsFeatureHydrator[\n    ForYouQuery,\n    AdsCandidate\n  ]) {\n\n  private val identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\"ForYouAds\")\n\n  private val servedType = hmt.ServedType.ForYouPromoted\n\n  private val clientEventInfoBuilder = ClientEventInfoBuilder(\n    component = servedType.originalName,\n    detailsBuilder = Some(HomeAdsClientEventDetailsBuilder(Some(servedType.name)))\n  )\n\n  private val contextualTweetRefBuilder = ContextualTweetRefBuilder(\n    TweetHydrationContext(\n      safetyLevelOverride = Some(TimelineHomePromotedHydrationSafetyLevel),\n      outerTweetContext = None\n    ))\n\n  private val decorator = UrtItemCandidateDecorator(\n    AdsCandidateUrtItemBuilder(\n      tweetClientEventInfoBuilder = Some(clientEventInfoBuilder),\n      contextualTweetRefBuilder = Some(contextualTweetRefBuilder)\n    ))\n\n  private val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),\n    HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()\n  )\n\n  def build(): AdsCandidatePipelineConfig[ForYouQuery] =\n    adsCandidatePipelineConfigBuilder.build[ForYouQuery](\n      adsCandidateSource = adsCandidateSource,\n      identifier = identifier,\n      adsDisplayLocationBuilder = StaticAdsDisplayLocationBuilder(ads.DisplayLocation.TimelineHome),\n      estimateNumOrganicItems = _.params(AdsNumOrganicItemsParam).toShort,\n      gates = Seq(\n        ParamNotGate(\n          name = \"AdsDisableInjectionBasedOnUserRole\",\n          param = HomeGlobalParams.AdsDisableInjectionBasedOnUserRoleParam\n        ),\n        ExcludeSoftUserGate,\n        ExcludeSyntheticUserGate,\n        FeatureGate.fromNegatedFeature(NoAdsTierFeature)\n      ),\n      filters = Seq(ValidAdImpressionIdFilter),\n      postFilterFeatureHydration = Seq(\n        ParamGatedCandidateFeatureHydrator(\n          EnableAdvertiserBrandSafetySettingsFeatureHydratorParam,\n          advertiserBrandSafetySettingsFeatureHydrator\n        )\n      ),\n      decorator = Some(decorator),\n      alerts = alerts,\n      urtRequest = Some(true)\n    )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouAdsDependentCandidatePipelineBuilder.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.adserver.{thriftscala => ads}\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeAdsClientEventDetailsBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator.NoAdsTierFeature\nimport com.twitter.home_mixer.functional_component.gate.ExcludeSoftUserGate\nimport com.twitter.home_mixer.functional_component.gate.ExcludeSyntheticUserGate\nimport com.twitter.home_mixer.model.HomeFeatures.TweetLanguageFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTextFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableAdvertiserBrandSafetySettingsFeatureHydratorParam\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.candidate_source.ads.AdsProdThriftCandidateSource\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref.ContextualTweetRefBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.ad.AdsCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.ads.AdvertiserBrandSafetySettingsFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.gate.FeatureGate\nimport com.twitter.product_mixer.component_library.gate.NonEmptyCandidatesGate\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.AdsDependentCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.AdsDependentCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.CountTruncatedItemCandidatesFromPipelines\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.StaticAdsDisplayLocationBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.TruncatedPipelineScopedOrganicItems\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.ValidAdImpressionIdFilter\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.gate.ParamNotGate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.TimelineHomePromotedHydrationSafetyLevel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.TweetHydrationContext\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouAdsDependentCandidatePipelineBuilder @Inject() (\n  adsCandidatePipelineConfigBuilder: AdsDependentCandidatePipelineConfigBuilder,\n  adsCandidateSource: AdsProdThriftCandidateSource,\n  advertiserBrandSafetySettingsFeatureHydrator: AdvertiserBrandSafetySettingsFeatureHydrator[\n    ForYouQuery,\n    AdsCandidate\n  ]) {\n\n  private val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouAdsDependent\")\n\n  private val servedType = hmt.ServedType.ForYouPromoted\n\n  private val MaxOrganicTweets = 35\n\n  private val clientEventInfoBuilder = ClientEventInfoBuilder(\n    component = servedType.originalName,\n    detailsBuilder = Some(HomeAdsClientEventDetailsBuilder(Some(servedType.name)))\n  )\n\n  private val contextualTweetRefBuilder = ContextualTweetRefBuilder(\n    TweetHydrationContext(\n      safetyLevelOverride = Some(TimelineHomePromotedHydrationSafetyLevel),\n      outerTweetContext = None\n    ))\n\n  private val decorator = UrtItemCandidateDecorator(\n    AdsCandidateUrtItemBuilder(\n      tweetClientEventInfoBuilder = Some(clientEventInfoBuilder),\n      contextualTweetRefBuilder = Some(contextualTweetRefBuilder)\n    ))\n\n  private val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),\n    HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()\n  )\n\n  def build(\n    organicCandidatePipelines: CandidateScope\n  ): AdsDependentCandidatePipelineConfig[ForYouQuery] =\n    adsCandidatePipelineConfigBuilder.build[ForYouQuery](\n      adsCandidateSource = adsCandidateSource,\n      identifier = identifier,\n      adsDisplayLocationBuilder = StaticAdsDisplayLocationBuilder(ads.DisplayLocation.TimelineHome),\n      getOrganicItems = TruncatedPipelineScopedOrganicItems(\n        pipelines = organicCandidatePipelines,\n        textFeature = TweetTextFeature,\n        languageFeature = TweetLanguageFeature,\n        ordering = CandidatesUtil.scoreOrdering,\n        maxCount = MaxOrganicTweets\n      ),\n      countNumOrganicItems =\n        CountTruncatedItemCandidatesFromPipelines(organicCandidatePipelines, MaxOrganicTweets),\n      gates = Seq(\n        ParamNotGate(\n          name = \"AdsDisableInjectionBasedOnUserRole\",\n          param = HomeGlobalParams.AdsDisableInjectionBasedOnUserRoleParam\n        ),\n        ExcludeSoftUserGate,\n        ExcludeSyntheticUserGate,\n        FeatureGate.fromNegatedFeature(NoAdsTierFeature),\n        NonEmptyCandidatesGate(organicCandidatePipelines)\n      ),\n      filters = Seq(ValidAdImpressionIdFilter),\n      postFilterFeatureHydration = Seq(\n        ParamGatedCandidateFeatureHydrator(\n          EnableAdvertiserBrandSafetySettingsFeatureHydratorParam,\n          advertiserBrandSafetySettingsFeatureHydrator\n        )\n      ),\n      decorator = Some(decorator),\n      alerts = alerts,\n      urtRequest = Some(true)\n    )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouBookmarksCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.frigate.bookmarks.{thriftscala => t}\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.TweetCarouselModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.TweetCarouselType\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.TweetHydrationFilter\nimport com.twitter.home_mixer.functional_component.filter.WeeklyBookmarkFilter\nimport com.twitter.home_mixer.functional_component.gate.BookmarksTimeGate\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.product.for_you.candidate_source.BookmarksCandidateSource\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.BookmarksModuleMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableBookmarksCandidatePipelineParam\nimport com.twitter.home_mixer.product.for_you.response_transformer.BookmarksResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.gate.DefinedUserIdGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouBookmarksCandidatePipelineConfig @Inject() (\n  bookmarksTimeGate: BookmarksTimeGate,\n  bookmarksCandidateSource: BookmarksCandidateSource,\n  tweetypieFeatureHydrator: TweetypieFeatureHydrator,\n  tweetCarouselModuleCandidateDecorator: TweetCarouselModuleCandidateDecorator)\n    extends CandidatePipelineConfig[\n      PipelineQuery,\n      Long,\n      t.BookmarkedTweet,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouBookmarks\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] =\n    Some(EnableBookmarksCandidatePipelineParam)\n\n  override val gates: Seq[Gate[PipelineQuery]] = Seq(\n    DefinedUserIdGate,\n    RateLimitGate,\n    bookmarksTimeGate,\n    TimelinesPersistenceStoreLastInjectionGate(\n      BookmarksModuleMinInjectionIntervalParam,\n      EntityIdType.BookmarksModule\n    )\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[PipelineQuery, Long] =\n    query => query.getRequiredUserId\n\n  override val candidateSource: BaseCandidateSource[Long, t.BookmarkedTweet] =\n    bookmarksCandidateSource\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[t.BookmarkedTweet]\n  ] = Seq(BookmarksResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    t.BookmarkedTweet,\n    TweetCandidate\n  ] = tweet => TweetCandidate(tweet.tweetId)\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[PipelineQuery, TweetCandidate, _]\n  ] = Seq(tweetypieFeatureHydrator)\n\n  override val filters: Seq[Filter[PipelineQuery, TweetCandidate]] =\n    Seq(TweetHydrationFilter, WeeklyBookmarkFilter)\n\n  override val decorator: Option[\n    CandidateDecorator[PipelineQuery, TweetCandidate]\n  ] = Some(tweetCarouselModuleCandidateDecorator.build(TweetCarouselType.BookmarkedTweets))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouCommunitiesToJoinCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.communities_mixer.{thriftscala => cmt}\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.FeedbackStrings\nimport com.twitter.home_mixer.functional_component.filter.DropMaxCandidatesFilter\nimport com.twitter.home_mixer.functional_component.gate.DismissFatigueGate\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.model.HomeFeatures.DismissInfoFeature\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.CommunitiesToJoinDisplayTypeIdParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.CommunitiesToJoinMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableCommunitiesToJoinCandidatePipelineParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.MaxCommunitiesToJoinCandidatesParam\nimport com.twitter.product_mixer.component_library.candidate_source.communities.CommunitiesToJoinCandidateSource\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.IsAvailableToJoinFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.communities.IsAvailableToJoinFeatureHydrator\nimport com.twitter.product_mixer.component_library.filter.FeatureFilter\nimport com.twitter.product_mixer.component_library.gate.DefinedUserIdGate\nimport com.twitter.product_mixer.component_library.gate.communities.CommunitiesJoinLimitGate\nimport com.twitter.product_mixer.component_library.model.candidate.CommunityCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.communities_to_join.CommunitiesToJoinCandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.marshaller.request.ClientContextMarshaller\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouCommunitiesToJoinCandidatePipelineConfig @Inject() (\n  communitiesToJoinCandidateSource: CommunitiesToJoinCandidateSource,\n  communitiesJoinLimitGate: CommunitiesJoinLimitGate,\n  isAvailableToJoinFeatureHydrator: IsAvailableToJoinFeatureHydrator,\n  @ProductScoped stringCenterProvider: Provider[StringCenter],\n  externalStrings: HomeMixerExternalStrings,\n  feedbackStrings: FeedbackStrings)\n    extends CandidatePipelineConfig[\n      PipelineQuery with HasPipelineCursor[_],\n      cmt.CommunitiesMixerRequest,\n      Long,\n      CommunityCandidate\n    ] {\n\n  private val stringCenter = stringCenterProvider.get()\n\n  private val MaxCommunities = 10\n\n  override val supportedClientParam: Option[FSParam[Boolean]] = Some(\n    EnableCommunitiesToJoinCandidatePipelineParam)\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouCommunitiesToJoin\")\n\n  private val IsAvailableToJoinFilterId = \"IsAvailableToJoin\"\n\n  override val gates = Seq(\n    DefinedUserIdGate,\n    RateLimitGate,\n    TimelinesPersistenceStoreLastInjectionGate(\n      CommunitiesToJoinMinInjectionIntervalParam,\n      EntityIdType.CommunityModule\n    ),\n    DismissFatigueGate(st.SuggestType.CommunityToJoin, DismissInfoFeature),\n    communitiesJoinLimitGate\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    PipelineQuery,\n    cmt.CommunitiesMixerRequest\n  ] = { query =>\n    cmt.CommunitiesMixerRequest(\n      clientContext = ClientContextMarshaller(query.clientContext),\n      product = cmt.Product.CommunityRecs,\n      productContext = Some(\n        cmt.ProductContext.CommunityRecs(\n          cmt.CommunityRecs(displayFormat = cmt.DisplayFormat.Module))),\n      cursor = None,\n      maxResults = Some(MaxCommunities),\n      debugParams = None,\n      displayLocation = None\n    )\n  }\n\n  override val candidateSource: CandidateSource[cmt.CommunitiesMixerRequest, Long] =\n    communitiesToJoinCandidateSource\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[PipelineQuery, CommunityCandidate, _]\n  ] = Seq(isAvailableToJoinFeatureHydrator)\n\n  override def filters: Seq[Filter[PipelineQuery, CommunityCandidate]] = Seq(\n    FeatureFilter\n      .fromFeature(FilterIdentifier(IsAvailableToJoinFilterId), IsAvailableToJoinFeature),\n    DropMaxCandidatesFilter(MaxCommunitiesToJoinCandidatesParam)\n  )\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[Long, CommunityCandidate] = {\n    communityResult => CommunityCandidate(id = communityResult)\n  }\n\n  override val decorator: Option[\n    CandidateDecorator[PipelineQuery, CommunityCandidate]\n  ] = {\n    Some(\n      CommunitiesToJoinCandidateDecorator(\n        moduleDisplayTypeParam = CommunitiesToJoinDisplayTypeIdParam,\n        stringCenter = stringCenter,\n        headerString = externalStrings.CommunityToJoinHeaderString,\n        footerString = Some(externalStrings.CommunityToJoinFooterString),\n        seeLessOftenString = Some(feedbackStrings.seeLessOftenFeedbackString),\n        seeLessOftenConfirmationString =\n          Some(feedbackStrings.seeLessOftenConfirmationFeedbackString)\n      ))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouConversationServiceCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.candidate_pipeline.ConversationServiceResponseFeatureTransformer\nimport com.twitter.home_mixer.functional_component.decorator.HomeConversationServiceCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator.InNetworkFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.InvalidConversationModuleFilter\nimport com.twitter.home_mixer.functional_component.filter.InvalidSubscriptionTweetFilter\nimport com.twitter.home_mixer.functional_component.filter.PreviouslyServedTweetsFilter\nimport com.twitter.home_mixer.functional_component.filter.RetweetDeduplicationFilter\nimport com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedTweetDroppedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TimelineServiceTweetsFeature\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSource\nimport com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.ConversationServiceCandidateSourceRequest\nimport com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc.TweetWithConversationMetadata\nimport com.twitter.product_mixer.component_library.filter.FeatureFilter\nimport com.twitter.product_mixer.component_library.filter.PredicateFeatureFilter\nimport com.twitter.product_mixer.component_library.gate.NoCandidatesGate\nimport com.twitter.product_mixer.component_library.gate.NonEmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.DependentCandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Candidate Pipeline Config that fetches Tweet ancestors from Conversation Service Candidate Source\n */\n@Singleton\nclass ForYouConversationServiceCandidatePipelineConfig @Inject() (\n  forYouScoredTweetsCandidatePipelineConfig: ForYouScoredTweetsCandidatePipelineConfig,\n  forYouTimelineScorerCandidatePipelineConfig: ForYouTimelineScorerCandidatePipelineConfig,\n  conversationServiceCandidateSource: ConversationServiceCandidateSource,\n  tweetypieFeatureHydrator: TweetypieFeatureHydrator,\n  namesFeatureHydrator: NamesFeatureHydrator,\n  invalidSubscriptionTweetFilter: InvalidSubscriptionTweetFilter,\n  homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder)\n    extends DependentCandidatePipelineConfig[\n      ForYouQuery,\n      ConversationServiceCandidateSourceRequest,\n      TweetWithConversationMetadata,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouConversationService\")\n\n  override val gates: Seq[BaseGate[ForYouQuery]] = Seq(\n    NoCandidatesGate(\n      SpecificPipelines(\n        forYouTimelineScorerCandidatePipelineConfig.identifier,\n        forYouScoredTweetsCandidatePipelineConfig.identifier\n      )\n    ),\n    NonEmptySeqFeatureGate(TimelineServiceTweetsFeature)\n  )\n\n  override val candidateSource: BaseCandidateSource[\n    ConversationServiceCandidateSourceRequest,\n    TweetWithConversationMetadata\n  ] = conversationServiceCandidateSource\n\n  override val queryTransformer: DependentCandidatePipelineQueryTransformer[\n    ForYouQuery,\n    ConversationServiceCandidateSourceRequest\n  ] = { (query, candidates) =>\n    val timelineServiceTweets = query.features\n      .map(_.getOrElse(TimelineServiceTweetsFeature, Seq.empty)).getOrElse(Seq.empty)\n\n    val tweetsWithConversationMetadata = timelineServiceTweets.map { id =>\n      TweetWithConversationMetadata(\n        tweetId = id,\n        userId = None,\n        sourceTweetId = None,\n        sourceUserId = None,\n        inReplyToTweetId = None,\n        conversationId = None,\n        ancestors = Seq.empty\n      )\n    }\n    ConversationServiceCandidateSourceRequest(tweetsWithConversationMetadata)\n  }\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[TweetWithConversationMetadata]\n  ] = Seq(ConversationServiceResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    TweetWithConversationMetadata,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.tweetId) }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ForYouQuery, TweetCandidate, _]\n  ] = Seq(\n    InNetworkFeatureHydrator,\n    tweetypieFeatureHydrator\n  )\n\n  override def filters: Seq[Filter[ForYouQuery, TweetCandidate]] = Seq(\n    PreviouslyServedTweetsFilter,\n    RetweetDeduplicationFilter,\n    FeatureFilter.fromFeature(FilterIdentifier(\"TweetypieHydrated\"), IsHydratedFeature),\n    PredicateFeatureFilter.fromPredicate(\n      FilterIdentifier(\"QuotedTweetDropped\"),\n      shouldKeepCandidate = { features => !features.getOrElse(QuotedTweetDroppedFeature, false) }\n    ),\n    invalidSubscriptionTweetFilter,\n    InvalidConversationModuleFilter\n  )\n\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[ForYouQuery, TweetCandidate, _]\n  ] = Seq(namesFeatureHydrator)\n\n  override val decorator: Option[CandidateDecorator[ForYouQuery, TweetCandidate]] =\n    HomeConversationServiceCandidateDecorator(homeFeedbackActionInfoBuilder)\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),\n    HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouEntryPointPivotCandidatePipelineBuilder.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.EntryPointPivotModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.decorator.StrEntryPointPivotCategoryText\nimport com.twitter.home_mixer.product.for_you.gate.FollowingSportsUserGate\nimport com.twitter.product_mixer.component_library.candidate_source.entry_point_pivot.EntryPointPivotCandidateSource\nimport com.twitter.product_mixer.component_library.candidate_source.entry_point_pivot.GrokEntryPointPivotCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.pivot.PivotCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyView\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelines.render.{thriftscala => t}\nimport com.twitter.util.Duration\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject ForYouEntryPointPivotCandidatePipelineBuilder {\n  sealed abstract class EntryPointPivotType()\n\n  object EntryPointPivotType {\n    case object Events extends EntryPointPivotType()\n\n    case object Grok extends EntryPointPivotType()\n  }\n}\n@Singleton\nclass ForYouEntryPointPivotCandidatePipelineBuilder @Inject() (\n  entryPointPivotCandidateSource: EntryPointPivotCandidateSource,\n  grokEntryPointPivotCandidateSource: GrokEntryPointPivotCandidateSource) {\n\n  import ForYouEntryPointPivotCandidatePipelineBuilder._\n\n  def build(\n    entryPointPivotType: EntryPointPivotType,\n    supportedClientParam: FSParam[Boolean],\n    pivotMinInjectionIntervalParam: Param[Duration]\n  ): ForYouEntryPointPivotCandidatePipelineConfig =\n    new ForYouEntryPointPivotCandidatePipelineConfig(\n      identifier = getCandidateIdentifier(entryPointPivotType),\n      supportedClientParam = Some(supportedClientParam),\n      candidateSource = getCandidateSource(entryPointPivotType),\n      decorator = Some(getDecorator(entryPointPivotType)),\n      pivotMinInjectionIntervalParam = pivotMinInjectionIntervalParam,\n      extraGates = getPipelineGates(entryPointPivotType)\n    )\n\n  private def getCandidateSource(entryPointPivotType: EntryPointPivotType): CandidateSource[\n    StratoKeyView[String, Unit],\n    t.Pivot\n  ] =\n    entryPointPivotType match {\n      case EntryPointPivotType.Grok =>\n        grokEntryPointPivotCandidateSource\n      case EntryPointPivotType.Events =>\n        entryPointPivotCandidateSource\n    }\n\n  private def getDecorator(\n    entryPointPivotType: EntryPointPivotType\n  ): CandidateDecorator[PipelineQuery, PivotCandidate] =\n    entryPointPivotType match {\n      case EntryPointPivotType.Grok =>\n        EntryPointPivotModuleCandidateDecorator(\n          component = \"grok_pivot\",\n          headerText = StrEntryPointPivotCategoryText(\"\")\n        ).moduleDecorator\n      case EntryPointPivotType.Events =>\n        EntryPointPivotModuleCandidateDecorator(\n          component = \"sport_entry_point\",\n          headerText = StrEntryPointPivotCategoryText(\"Play now!\")\n        ).moduleDecorator\n    }\n\n  private def getPipelineGates(entryPointPivotType: EntryPointPivotType): Seq[Gate[PipelineQuery]] =\n    entryPointPivotType match {\n      case EntryPointPivotType.Grok =>\n        Seq.empty\n      case EntryPointPivotType.Events =>\n        Seq(FollowingSportsUserGate)\n    }\n\n  private def getCandidateIdentifier(\n    entryPointPivotType: EntryPointPivotType\n  ): CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(s\"ForYouEntryPointPivot${entryPointPivotType.toString}\")\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouEntryPointPivotCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.product_mixer.component_library.gate.DefinedUserIdGate\nimport com.twitter.product_mixer.component_library.model.candidate.pivot.PivotCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyView\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelines.render.{thriftscala => t}\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport com.twitter.util.Duration\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouEntryPointPivotCandidatePipelineConfig @Inject() (\n  override val identifier: CandidatePipelineIdentifier,\n  override val supportedClientParam: Option[FSParam[Boolean]],\n  override val candidateSource: CandidateSource[\n    StratoKeyView[String, Unit],\n    t.Pivot\n  ],\n  override val decorator: Option[CandidateDecorator[PipelineQuery, PivotCandidate]],\n  pivotMinInjectionIntervalParam: Param[Duration],\n  extraGates: Seq[Gate[PipelineQuery]])\n    extends CandidatePipelineConfig[\n      PipelineQuery with HasPipelineCursor[_],\n      StratoKeyView[String, Unit],\n      t.Pivot,\n      PivotCandidate\n    ] {\n\n  override val gates = Seq(\n    DefinedUserIdGate,\n    RateLimitGate,\n    TimelinesPersistenceStoreLastInjectionGate(\n      pivotMinInjectionIntervalParam,\n      EntityIdType.EntryPointPivot\n    )\n  ) ++ extraGates\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    PipelineQuery,\n    StratoKeyView[String, Unit]\n  ] = { query =>\n    StratoKeyView(\n      key = query.getCountryCode.getOrElse(\"US\"),\n      view = None\n    )\n  }\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    t.Pivot,\n    PivotCandidate\n  ] = { sourceResult =>\n    PivotCandidate(\n      // We only expect one item per \"messageprompt\" entryNamespace per URT response.\n      // As a result id=0L will always be unique in the entryNamespace.\n      id = identifier.name,\n      url = sourceResult.url,\n      displayType = sourceResult.displayType,\n      titleText = sourceResult.titleText,\n      detailText = sourceResult.detailText,\n      image = sourceResult.image,\n      badge = sourceResult.badge,\n      categoryText = sourceResult.categoryText,\n      detailTextImage = sourceResult.detailTextImage,\n      element = None\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouExplorationTweetsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.TweetHydrationFilter\nimport com.twitter.home_mixer.model.HomeFeatures.PersistenceEntriesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TLSOriginalTweetsWithConfirmedAuthorFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetAuthorFollowersFeature\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.TweetAuthorFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.TweetEngagementsFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.TweetAuthorFollowersFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.TweetEngagementCountsFeature\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.TweetEngagementCounts\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableExplorationTweetsCandidatePipelineParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.InNetworkExplorationTweetsMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ExplorationTweetsMaxFollowerCountParam\nimport com.twitter.home_mixer.product.for_you.response_transformer.ExplorationTweetResponseFeatureTransformer\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.PassthroughCandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.home_mixer.functional_component.gate.RecentlyServedByServedTypeGate\nimport com.twitter.home_mixer.product.for_you.gate.UserFollowingRangeGate\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.util.Random\n\n@Singleton\nclass ForYouExplorationTweetsCandidatePipelineConfig @Inject() (\n  tweetAuthorFeatureHydrator: TweetAuthorFeatureHydrator,\n  tweetEngagementsFeatureHydrator: TweetEngagementsFeatureHydrator,\n  tweetAuthorFollowersFeatureHydrator: TweetAuthorFollowersFeatureHydrator,\n  tweetypieFeatureHydrator: TweetypieFeatureHydrator,\n  homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder)\n    extends CandidatePipelineConfig[\n      ForYouQuery,\n      ForYouQuery,\n      Long,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouExplorationTweets\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] = Some(\n    EnableExplorationTweetsCandidatePipelineParam)\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ForYouQuery,\n    ForYouQuery\n  ] = identity\n\n  override val queryFeatureHydration: Seq[QueryFeatureHydrator[PipelineQuery]] = Seq(\n    tweetAuthorFeatureHydrator,\n    tweetEngagementsFeatureHydrator\n  )\n\n  override val queryFeatureHydrationPhase2: Seq[QueryFeatureHydrator[PipelineQuery]] = Seq(\n    tweetAuthorFollowersFeatureHydrator\n  )\n\n  override val gates: Seq[Gate[PipelineQuery]] = Seq(\n    RecentlyServedByServedTypeGate(\n      InNetworkExplorationTweetsMinInjectionIntervalParam,\n      hmt.ServedType.ForYouExploration\n    ),\n    UserFollowingRangeGate\n  )\n\n  override def candidateSource: CandidateSource[ForYouQuery, Long] =\n    PassthroughCandidateSource(\n      identifier = CandidateSourceIdentifier(\"ForYouExplorationTweets\"),\n      candidateExtractor = { query =>\n        val followedUserIds = query.features\n          .map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty))\n          .getOrElse(Seq.empty).toSet\n\n        val servedAuthorIds =\n          query.features\n            .map(_.getOrElse(PersistenceEntriesFeature, Seq.empty)).getOrElse(Seq.empty)\n            .flatMap(_.entries.flatMap(_.sourceAuthorIds))\n            .toSet\n\n        val explorationAuthors = followedUserIds -- servedAuthorIds\n\n        val engagementCounts = query.features\n          .map(_.getOrElse(TweetEngagementCountsFeature, Map.empty[Long, TweetEngagementCounts]))\n          .getOrElse(Map.empty)\n\n        val maxFollowerCount = query.params(ExplorationTweetsMaxFollowerCountParam)\n        val tweetAuthorFollowers = query.features\n          .map(_.getOrElse(TweetAuthorFollowersFeature, Map.empty[Long, Option[Long]]))\n          .getOrElse(Map.empty)\n\n        val groupedByAuthor = query.features\n          .map(_.getOrElse(TLSOriginalTweetsWithConfirmedAuthorFeature, Seq.empty))\n          .toSeq.flatten.filter {\n            case (tweetId, authorId) =>\n              explorationAuthors.contains(authorId) && {\n                val followerCount = tweetAuthorFollowers.get(tweetId).flatten.getOrElse(0L)\n                followerCount <= maxFollowerCount\n              }\n          }\n          .groupBy { case (_, authorId) => authorId }\n\n        val bestPostsByAuthor = groupedByAuthor.flatMap {\n          case (authorId, candidates) =>\n            val candidateWithEngagement = candidates.map {\n              case (tweetId, _) =>\n                val engagement = engagementCounts.get(tweetId)\n                val totalEngagement = engagement\n                  .map { counts =>\n                    counts.favoriteCount.getOrElse(0L) +\n                      counts.replyCount.getOrElse(0L) +\n                      counts.retweetCount.getOrElse(0L) +\n                      counts.quoteCount.getOrElse(0L) +\n                      counts.bookmarkCount.getOrElse(0L)\n                  }.getOrElse(0L)\n                (tweetId, totalEngagement)\n            }\n\n            if (candidateWithEngagement.nonEmpty) {\n              val bestPost = candidateWithEngagement.maxBy(_._2)\n              Some(bestPost._1)\n            } else None\n        }\n\n        Random.shuffle(bestPostsByAuthor.toSeq).take(1)\n      }\n    )\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[Long]\n  ] = Seq(ExplorationTweetResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[Long, TweetCandidate] = {\n    sourceResult => TweetCandidate(id = sourceResult)\n  }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ForYouQuery, TweetCandidate, _]\n  ] = Seq(tweetypieFeatureHydrator)\n\n  override val filters: Seq[Filter[ForYouQuery, TweetCandidate]] =\n    Seq(TweetHydrationFilter)\n\n  override val decorator: Option[CandidateDecorator[PipelineQuery, TweetCandidate]] = {\n    val clientEventInfoBuilder =\n      ClientEventInfoBuilder[PipelineQuery, TweetCandidate](\n        hmt.ServedType.ForYouExploration.originalName)\n\n    val tweetItemBuilder = TweetCandidateUrtItemBuilder[PipelineQuery, TweetCandidate](\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder),\n    )\n\n    Some(UrtItemCandidateDecorator(tweetItemBuilder))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouJetfuelFrameFrameCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.strato.columns.jetfuel.thriftscala.JetfuelRouteData\nimport com.twitter.home_mixer.product.for_you.candidate_source.JetfuelFrameCandidateSource\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableJetfuelFramePipelineParam\nimport com.twitter.home_mixer.product.for_you.gate.FollowingSportsUserGate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.jetfuel_entry_point.JetfuelCandidateFeatureTransformer\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.frame.FrameCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.gate.FirstPageGate\nimport com.twitter.product_mixer.component_library.model.candidate.FrameCandidate\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.frame.JetfuelPayloadFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyView\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.gate.ParamGate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.strato.generated.client.events.entryPoint.JetfuelEntryPointByCountryClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouJetfuelFrameCandidatePipelineConfig @Inject() (\n  jetfuelFrameCandidateSource: JetfuelFrameCandidateSource,\n  jetfuelPayloadFeatureHydrator: JetfuelPayloadFeatureHydrator)\n    extends CandidatePipelineConfig[\n      PipelineQuery with HasPipelineCursor[_],\n      StratoKeyView[\n        JetfuelEntryPointByCountryClientColumn.Key,\n        JetfuelEntryPointByCountryClientColumn.View\n      ],\n      JetfuelRouteData,\n      FrameCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouJetfuelFrame\")\n\n  private val FrameId = \"ForYouJetfuelFrame\"\n\n  override val candidateSource: JetfuelFrameCandidateSource = jetfuelFrameCandidateSource\n\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[PipelineQuery, FrameCandidate, _]\n  ] = Seq(jetfuelPayloadFeatureHydrator)\n\n  override val gates: Seq[Gate[PipelineQuery with HasPipelineCursor[_]]] = Seq(\n    FirstPageGate,\n    ParamGate(name = \"JetfuelFrameEnabled\", param = EnableJetfuelFramePipelineParam),\n    FollowingSportsUserGate\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    PipelineQuery,\n    StratoKeyView[\n      JetfuelEntryPointByCountryClientColumn.Key,\n      JetfuelEntryPointByCountryClientColumn.View\n    ]\n  ] = { query => StratoKeyView(key = query.getCountryCode.getOrElse(\"US\"), view = None) }\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[JetfuelRouteData]\n  ] = Seq(JetfuelCandidateFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    JetfuelRouteData,\n    FrameCandidate\n  ] = { sourceResult => FrameCandidate(id = sourceResult.route) }\n\n  override val decorator: Option[\n    CandidateDecorator[PipelineQuery, FrameCandidate]\n  ] = {\n    Some(\n      UrtItemCandidateDecorator(\n        FrameCandidateUrtItemBuilder(\n          frameId = FrameId,\n          clientEventInfoBuilder = None\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouKeywordTrendsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.events.recos.thriftscala.GetUnfiedCandidatesRequest\nimport com.twitter.events.recos.{thriftscala => t}\nimport com.twitter.home_mixer.functional_component.decorator.KeywordTrendsModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.product.for_you.candidate_source.TrendCandidate\nimport com.twitter.home_mixer.product.for_you.candidate_source.UnifiedTrendsCandidateSource\nimport com.twitter.home_mixer.product.for_you.filter.PromotedTrendFilter\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableKeywordTrendsParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.KeywordTrendsModuleMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.MaxNumberKeywordTrendsParam\nimport com.twitter.home_mixer.product.for_you.query_transformer.UnifiedCandidatesQueryTransformer\nimport com.twitter.home_mixer.product.for_you.response_transformer.KeywordTrendsFeatureTransformer\nimport com.twitter.product_mixer.component_library.gate.DefinedUserIdGate\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.UnifiedTrendCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouKeywordTrendsCandidatePipelineConfig @Inject() (\n  unifiedTrendsCandidateSource: UnifiedTrendsCandidateSource,\n  keywordTrendsModuleCandidateDecorator: KeywordTrendsModuleCandidateDecorator)\n    extends CandidatePipelineConfig[\n      PipelineQuery,\n      t.GetUnfiedCandidatesRequest,\n      TrendCandidate,\n      UnifiedTrendCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\n    \"ForYouKeywordTrends\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] = Some(EnableKeywordTrendsParam)\n\n  override val gates = Seq(\n    RateLimitGate,\n    DefinedUserIdGate,\n    TimelinesPersistenceStoreLastInjectionGate(\n      KeywordTrendsModuleMinInjectionIntervalParam,\n      EntityIdType.Trends\n    )\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    PipelineQuery,\n    GetUnfiedCandidatesRequest\n  ] = UnifiedCandidatesQueryTransformer(\n    maxResultsParam = MaxNumberKeywordTrendsParam,\n    candidatePipelineIdentifier = identifier)\n\n  override val candidateSource: CandidateSource[\n    t.GetUnfiedCandidatesRequest,\n    TrendCandidate\n  ] = unifiedTrendsCandidateSource\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[TrendCandidate]\n  ] = Seq(KeywordTrendsFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    TrendCandidate,\n    UnifiedTrendCandidate\n  ] = { result => UnifiedTrendCandidate(id = result.candidate.trendName) }\n\n  override val filters: Seq[Filter[PipelineQuery, UnifiedTrendCandidate]] = Seq(PromotedTrendFilter)\n\n  override val decorator: Option[CandidateDecorator[PipelineQuery, UnifiedTrendCandidate]] =\n    Some(keywordTrendsModuleCandidateDecorator.moduleDecorator)\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouMixerPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.clientapp.{thriftscala => ca}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.goldfinch.api.AdsInjectionSurfaceAreas\nimport com.twitter.home_mixer.candidate_pipeline.EditedTweetsCandidatePipelineConfig\nimport com.twitter.home_mixer.candidate_pipeline.NewTweetsPillCandidatePipelineConfig\nimport com.twitter.home_mixer.candidate_pipeline.VerifiedPromptCandidatePipelineConfig\nimport com.twitter.home_mixer.functional_component.feature_hydrator._\nimport com.twitter.home_mixer.functional_component.selector.UpdateConversationModuleId\nimport com.twitter.home_mixer.functional_component.selector.UpdateHomeClientEventDetails\nimport com.twitter.home_mixer.functional_component.selector.UpdateNewTweetsPillDecoration\nimport com.twitter.home_mixer.functional_component.side_effect._\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableSSPAdsBrandSafetySettingsFeatureHydratorParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableUserActionsShadowScribeParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.MaxNumberReplaceInstructionsParam\nimport com.twitter.home_mixer.param.HomeMixerFlagName.ScribeClientEventsFlag\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.DisplayedGrokTopicQueryFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.FollowingSportsAccountsQueryFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.TimelineServiceTweetsQueryFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.ViewerHasJobRecommendationsFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableAdsDebugParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableEntryPointPivotParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableFlipInjectionModuleCandidatePipelineParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableFollowedGrokTopicsHydrationParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableForYouAppUpsellParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableForYouTimelineAdsSurface\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableForYouTopicSelectorParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableGrokEntryPointPivotParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EntryPointPivotMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ExplorationTweetsTimelinePosition\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ForYouAppUpsellJetfuelRouteParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ForYouAppUpsellPosition\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ForYouTopicSelectorJetfuelRouteParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ForYouTopicSelectorPosition\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.GrokEntryPointPivotMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.GrokPivotModuleTimelinePosition\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.MaxNumberExplorationTweetsParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.RelevancePromptTweetPositionParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.StaticParamValueFive\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.StaticParamValueZero\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.SuperbowlModuleTimelinePosition\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.TuneFeedTimelinePosition\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.VideoCarouselNumCandidates\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.VideoCarouselNumTweetCandidatesToDedupeAgainstParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.VideoCarouselTimelinePosition\nimport com.twitter.home_mixer.product.for_you.selector.DebugUpdateSortAdsResult\nimport com.twitter.home_mixer.product.for_you.selector.RemoveDuplicateCandidatesOutsideModule\nimport com.twitter.home_mixer.product.for_you.side_effect.ServedCandidateFeatureKeysKafkaSideEffectBuilder\nimport com.twitter.home_mixer.product.for_you.side_effect.ServedStatsSideEffect\nimport com.twitter.home_mixer.product.for_you.side_effect.VideoServedStatsSideEffect\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.ads.SSPAdsBrandSafetySettingsFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.async.AsyncQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated.AsyncParamGatedQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated.ParamGatedQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.PreviewCreatorsQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.FlipPromptCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.selector.FlipPromptDynamicInsertionPosition\nimport com.twitter.product_mixer.component_library.pipeline.candidate.jetfuel_entry_point.JetfuelCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_subscribe_module.WhoToSubscribeCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.selector.DropDuplicateCandidates\nimport com.twitter.product_mixer.component_library.selector.DropMaxCandidates\nimport com.twitter.product_mixer.component_library.selector.DropMaxModuleItemCandidates\nimport com.twitter.product_mixer.component_library.selector.DropModuleTooFewModuleItemResults\nimport com.twitter.product_mixer.component_library.selector.DropOrthogonalCandidates\nimport com.twitter.product_mixer.component_library.selector.IdAndClassDuplicationKey\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.component_library.selector.InsertDynamicPositionResults\nimport com.twitter.product_mixer.component_library.selector.InsertFixedPositionResults\nimport com.twitter.product_mixer.component_library.selector.PickFirstCandidateMerger\nimport com.twitter.product_mixer.component_library.selector.SelectConditionally\nimport com.twitter.product_mixer.component_library.selector.UpdateSortCandidates\nimport com.twitter.product_mixer.component_library.selector.UpdateSortModuleItemCandidates\nimport com.twitter.product_mixer.component_library.selector.ads.AdsInjector\nimport com.twitter.product_mixer.component_library.selector.ads.InsertAdResults\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.UrtTransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.gate.ParamGate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.MixerPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineConfig\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouMixerPipelineConfig @Inject() (\n  forYouAdsCandidatePipelineBuilder: ForYouAdsCandidatePipelineBuilder,\n  forYouCommunitiesToJoinCandidatePipelineConfig: ForYouCommunitiesToJoinCandidatePipelineConfig,\n  forYouScoredTweetsCandidatePipelineConfig: ForYouScoredTweetsCandidatePipelineConfig,\n  forYouWhoToFollowCandidatePipelineConfigBuilder: ForYouWhoToFollowCandidatePipelineConfigBuilder,\n  forYouWhoToSubscribeCandidatePipelineConfigBuilder: ForYouWhoToSubscribeCandidatePipelineConfigBuilder,\n  forYouEntryPointPivotCandidatePipelineBuilder: ForYouEntryPointPivotCandidatePipelineBuilder,\n  forYouRecommendedJobsCandidatePipelineConfig: ForYouRecommendedJobsCandidatePipelineConfig,\n  forYouRecommendedRecruitingOrganizationsCandidatePipelineConfig: ForYouRecommendedRecruitingOrganizationsCandidatePipelineConfig,\n  forYouBookmarksCandidatePipelineConfig: ForYouBookmarksCandidatePipelineConfig,\n  forYouExplorationTweetsCandidatePipelineConfig: ForYouExplorationTweetsCandidatePipelineConfig,\n  forYouJetfuelFrameCandidatePipelineConfig: ForYouJetfuelFrameCandidatePipelineConfig,\n  forYouPinnedTweetsCandidatePipelineConfig: ForYouPinnedTweetsCandidatePipelineConfig,\n  forYouStoriesCandidatePipelineConfig: ForYouStoriesCandidatePipelineConfig,\n  flipPromptCandidatePipelineConfigBuilder: FlipPromptCandidatePipelineConfigBuilder,\n  forYouKeywordTrendsCandidatePipelineConfig: ForYouKeywordTrendsCandidatePipelineConfig,\n  forYouScoredVideoTweetsCandidatePipelineConfig: ForYouScoredVideoTweetsCandidatePipelineConfig,\n  forYouRelevancePromptCandidatePipelineConfig: ForYouRelevancePromptCandidatePipelineConfig,\n  editedTweetsCandidatePipelineConfig: EditedTweetsCandidatePipelineConfig,\n  forYouTuneFeedCandidatePipelineConfig: ForYouTuneFeedCandidatePipelineConfig,\n  newTweetsPillCandidatePipelineConfig: NewTweetsPillCandidatePipelineConfig[ForYouQuery],\n  forYouTweetPreviewsCandidatePipelineConfig: ForYouTweetPreviewsCandidatePipelineConfig,\n  verifiedPromptCandidatePipelineConfig: VerifiedPromptCandidatePipelineConfig,\n  jetfuelCandidatePipelineConfigBuilder: JetfuelCandidatePipelineConfigBuilder,\n  dismissInfoQueryFeatureHydrator: DismissInfoQueryFeatureHydrator,\n  followingSportsAccountsQueryFeatureHydrator: FollowingSportsAccountsQueryFeatureHydrator,\n  gizmoduckUserQueryFeatureHydrator: GizmoduckUserQueryFeatureHydrator,\n  impressionBloomFilterQueryFeatureHydrator: ImpressionBloomFilterQueryFeatureHydrator,\n  persistenceStoreQueryFeatureHydrator: PersistenceStoreQueryFeatureHydrator,\n  rateLimitQueryFeatureHydrator: RateLimitQueryFeatureHydrator,\n  requestQueryFeatureHydrator: RequestQueryFeatureHydrator[ForYouQuery],\n  timelineServiceTweetsQueryFeatureHydrator: TimelineServiceTweetsQueryFeatureHydrator,\n  previewCreatorsQueryFeatureHydrator: PreviewCreatorsQueryFeatureHydrator,\n  sgsFollowedUsersQueryFeatureHydrator: SGSFollowedUsersQueryFeatureHydrator,\n  userSubscriptionQueryFeatureHydrator: UserSubscriptionQueryFeatureHydrator,\n  displayedGrokTopicQueryFeatureHydrator: DisplayedGrokTopicQueryFeatureHydrator,\n  sspAdsBrandSafetySettingsFeatureHydrator: SSPAdsBrandSafetySettingsFeatureHydrator,\n  viewerHasJobRecommendationsFeatureHydrator: ViewerHasJobRecommendationsFeatureHydrator,\n  userActionsArrayByteQueryFeatureHydrator: UserActionsArrayByteQueryFeatureHydrator,\n  adsInjector: AdsInjector,\n  updateTimelinesPersistenceStoreSideEffect: UpdateTimelinesPersistenceStoreSideEffect,\n  truncateTimelinesPersistenceStoreSideEffect: TruncateTimelinesPersistenceStoreSideEffect,\n  homeScribeServedCandidatesSideEffect: HomeScribeServedCandidatesSideEffect,\n  servedCandidateFeatureKeysKafkaSideEffectBuilder: ServedCandidateFeatureKeysKafkaSideEffectBuilder,\n  clientEventsScribeEventPublisher: EventPublisher[ca.LogEvent],\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter],\n  urtTransportMarshaller: UrtTransportMarshaller,\n  @Flag(ScribeClientEventsFlag) enableScribeClientEvents: Boolean,\n  statsReceiver: StatsReceiver)\n    extends MixerPipelineConfig[ForYouQuery, Timeline, urt.TimelineResponse] {\n\n  override val identifier: MixerPipelineIdentifier = MixerPipelineIdentifier(\"ForYou\")\n\n  private val dependentCandidatesStep = MixerPipelineConfig.dependentCandidatePipelinesStep\n\n  override val fetchQueryFeatures: Seq[QueryFeatureHydrator[ForYouQuery]] = Seq(\n    rateLimitQueryFeatureHydrator,\n    requestQueryFeatureHydrator,\n    persistenceStoreQueryFeatureHydrator,\n    impressionBloomFilterQueryFeatureHydrator,\n    timelineServiceTweetsQueryFeatureHydrator,\n    previewCreatorsQueryFeatureHydrator,\n    sgsFollowedUsersQueryFeatureHydrator,\n    gizmoduckUserQueryFeatureHydrator,\n    viewerHasJobRecommendationsFeatureHydrator,\n    userSubscriptionQueryFeatureHydrator,\n    ParamGatedQueryFeatureHydrator(\n      EnableFollowedGrokTopicsHydrationParam,\n      displayedGrokTopicQueryFeatureHydrator\n    ),\n    ParamGatedQueryFeatureHydrator(\n      EnableSSPAdsBrandSafetySettingsFeatureHydratorParam,\n      sspAdsBrandSafetySettingsFeatureHydrator\n    ),\n    ParamGatedQueryFeatureHydrator(\n      EnableEntryPointPivotParam,\n      followingSportsAccountsQueryFeatureHydrator\n    ),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, dismissInfoQueryFeatureHydrator),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableUserActionsShadowScribeParam,\n      MixerPipelineConfig.resultSelectorsStep,\n      userActionsArrayByteQueryFeatureHydrator\n    )\n  )\n\n  private val forYouAdsCandidatePipelineConfig = forYouAdsCandidatePipelineBuilder.build()\n\n  private val forYouWhoToFollowCandidatePipelineConfig =\n    forYouWhoToFollowCandidatePipelineConfigBuilder.build()\n\n  private val forYouWhoToSubscribeCandidatePipelineConfig =\n    forYouWhoToSubscribeCandidatePipelineConfigBuilder.build()\n\n  private val flipPromptCandidatePipelineConfig =\n    flipPromptCandidatePipelineConfigBuilder.build[ForYouQuery](\n      supportedClientParam = Some(EnableFlipInjectionModuleCandidatePipelineParam)\n    )\n\n  private val forYouEventsEntryPointPivotCandidatePipelineConfig =\n    forYouEntryPointPivotCandidatePipelineBuilder.build(\n      entryPointPivotType =\n        ForYouEntryPointPivotCandidatePipelineBuilder.EntryPointPivotType.Events,\n      supportedClientParam = EnableEntryPointPivotParam,\n      pivotMinInjectionIntervalParam = EntryPointPivotMinInjectionIntervalParam\n    )\n\n  private val forYouGrokEntryPointPivotCandidatePipelineConfig =\n    forYouEntryPointPivotCandidatePipelineBuilder.build(\n      entryPointPivotType = ForYouEntryPointPivotCandidatePipelineBuilder.EntryPointPivotType.Grok,\n      supportedClientParam = EnableGrokEntryPointPivotParam,\n      pivotMinInjectionIntervalParam = GrokEntryPointPivotMinInjectionIntervalParam\n    )\n\n  private val forYouTopicSelectorCandidatePipelineConfig = {\n    jetfuelCandidatePipelineConfigBuilder.build[ForYouQuery](\n      frameId = \"ForYouTopicSelector\",\n      identifier = CandidatePipelineIdentifier(\"ForYouTopicSelector\"),\n      route = ForYouTopicSelectorJetfuelRouteParam,\n      gates = Seq(\n        ParamGate(name = \"ForYouTopicSelector\", param = EnableForYouTopicSelectorParam)\n      )\n    )\n  }\n\n  private val forYouAppUpsellCandidatePipelineConfig = {\n    jetfuelCandidatePipelineConfigBuilder.build[ForYouQuery](\n      frameId = \"ForYouAppUpsell\",\n      identifier = CandidatePipelineIdentifier(\"ForYouAppUpsell\"),\n      route = ForYouAppUpsellJetfuelRouteParam,\n      gates = Seq(\n        ParamGate(name = \"ForYouAppUpsell\", param = EnableForYouAppUpsellParam)\n      )\n    )\n  }\n\n  override val candidatePipelines: Seq[CandidatePipelineConfig[ForYouQuery, _, _, _]] = Seq(\n    forYouScoredTweetsCandidatePipelineConfig,\n    forYouAdsCandidatePipelineConfig,\n    forYouCommunitiesToJoinCandidatePipelineConfig,\n    forYouWhoToFollowCandidatePipelineConfig,\n    forYouWhoToSubscribeCandidatePipelineConfig,\n    forYouTweetPreviewsCandidatePipelineConfig,\n    forYouRecommendedJobsCandidatePipelineConfig,\n    forYouEventsEntryPointPivotCandidatePipelineConfig,\n    forYouGrokEntryPointPivotCandidatePipelineConfig,\n    forYouRecommendedRecruitingOrganizationsCandidatePipelineConfig,\n    forYouBookmarksCandidatePipelineConfig,\n    forYouExplorationTweetsCandidatePipelineConfig,\n    forYouPinnedTweetsCandidatePipelineConfig,\n    forYouStoriesCandidatePipelineConfig,\n    forYouScoredVideoTweetsCandidatePipelineConfig,\n    forYouTuneFeedCandidatePipelineConfig,\n    flipPromptCandidatePipelineConfig,\n    forYouKeywordTrendsCandidatePipelineConfig,\n    forYouJetfuelFrameCandidatePipelineConfig,\n    forYouRelevancePromptCandidatePipelineConfig,\n    forYouTopicSelectorCandidatePipelineConfig,\n    forYouAppUpsellCandidatePipelineConfig,\n  )\n\n  override val dependentCandidatePipelines: Seq[\n    DependentCandidatePipelineConfig[ForYouQuery, _, _, _]\n  ] = Seq(\n    editedTweetsCandidatePipelineConfig,\n    newTweetsPillCandidatePipelineConfig,\n    verifiedPromptCandidatePipelineConfig\n  )\n\n  override val failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy] = Map(\n    forYouScoredTweetsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouAdsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouCommunitiesToJoinCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouWhoToFollowCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouWhoToSubscribeCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouTweetPreviewsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouRecommendedJobsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouRecommendedRecruitingOrganizationsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouBookmarksCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouExplorationTweetsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouPinnedTweetsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouStoriesCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouScoredVideoTweetsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    flipPromptCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    editedTweetsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouTuneFeedCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    newTweetsPillCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouKeywordTrendsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouJetfuelFrameCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouEventsEntryPointPivotCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouGrokEntryPointPivotCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouRelevancePromptCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouTopicSelectorCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouAppUpsellCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n  )\n\n  override val resultSelectors: Seq[Selector[ForYouQuery]] = Seq(\n    UpdateSortCandidates(\n      ordering = CandidatesUtil.scoreOrdering,\n      candidatePipeline = forYouScoredTweetsCandidatePipelineConfig.identifier\n    ),\n    UpdateSortModuleItemCandidates(\n      candidatePipeline = forYouScoredTweetsCandidatePipelineConfig.identifier,\n      ordering = CandidatesUtil.conversationModuleTweetsOrdering\n    ),\n    UpdateSortCandidates(\n      ordering = CandidatesUtil.scoreOrdering,\n      candidatePipeline = forYouPinnedTweetsCandidatePipelineConfig.identifier\n    ),\n    UpdateSortModuleItemCandidates(\n      ordering = CandidatesUtil.scoreOrdering,\n      candidatePipeline = forYouPinnedTweetsCandidatePipelineConfig.identifier\n    ),\n    UpdateConversationModuleId(\n      pipelineScope = SpecificPipeline(forYouScoredTweetsCandidatePipelineConfig.identifier)\n    ),\n    RemoveDuplicateCandidatesOutsideModule(\n      pipelineScope = SpecificPipeline(forYouScoredVideoTweetsCandidatePipelineConfig.identifier),\n      candidatePipelinesOutsideModule = Set(forYouScoredTweetsCandidatePipelineConfig.identifier),\n      numCandidatesToCompareAgainst = VideoCarouselNumTweetCandidatesToDedupeAgainstParam\n    ),\n    RemoveDuplicateCandidatesOutsideModule(\n      pipelineScope = SpecificPipeline(forYouTuneFeedCandidatePipelineConfig.identifier),\n      candidatePipelinesOutsideModule = Set(forYouScoredTweetsCandidatePipelineConfig.identifier),\n      numCandidatesToCompareAgainst = StaticParam(10)\n    ),\n    DropMaxCandidates(\n      candidatePipeline = forYouScoredTweetsCandidatePipelineConfig.identifier,\n      maxSelectionsParam = ServerMaxResultsParam\n    ),\n    DropMaxCandidates(\n      candidatePipeline = editedTweetsCandidatePipelineConfig.identifier,\n      maxSelectionsParam = MaxNumberReplaceInstructionsParam\n    ),\n    DropMaxCandidates(\n      candidatePipeline = forYouExplorationTweetsCandidatePipelineConfig.identifier,\n      maxSelectionsParam = MaxNumberExplorationTweetsParam\n    ),\n    DropMaxModuleItemCandidates(\n      candidatePipeline = forYouWhoToFollowCandidatePipelineConfig.identifier,\n      maxModuleItemsParam = StaticParam(WhoToFollowCandidatePipelineConfig.MaxCandidatesSize)\n    ),\n    DropMaxModuleItemCandidates(\n      candidatePipeline = forYouWhoToSubscribeCandidatePipelineConfig.identifier,\n      maxModuleItemsParam = StaticParam(WhoToSubscribeCandidatePipelineConfig.MaxCandidatesSize)\n    ),\n    DropMaxModuleItemCandidates(\n      candidatePipeline = forYouBookmarksCandidatePipelineConfig.identifier,\n      maxModuleItemsParam = StaticParam(10)\n    ),\n    DropMaxModuleItemCandidates(\n      candidatePipeline = forYouScoredVideoTweetsCandidatePipelineConfig.identifier,\n      maxModuleItemsParam = VideoCarouselNumCandidates\n    ),\n    DropMaxModuleItemCandidates(\n      candidatePipeline = forYouPinnedTweetsCandidatePipelineConfig.identifier,\n      maxModuleItemsParam = StaticParam(10)\n    ),\n    DropMaxModuleItemCandidates(\n      candidatePipeline = forYouTuneFeedCandidatePipelineConfig.identifier,\n      maxModuleItemsParam = StaticParam(3)\n    ),\n    DropMaxCandidates(\n      candidatePipeline = forYouPinnedTweetsCandidatePipelineConfig.identifier,\n      maxSelectionsParam = StaticParam(1)\n    ),\n    DropDuplicateCandidates(\n      pipelineScope = SpecificPipelines(\n        Set(\n          forYouScoredTweetsCandidatePipelineConfig.identifier,\n          forYouExplorationTweetsCandidatePipelineConfig.identifier\n        )),\n      duplicationKey = IdAndClassDuplicationKey,\n      mergeStrategy = PickFirstCandidateMerger\n    ),\n    InsertAppendResults(\n      candidatePipeline = forYouScoredTweetsCandidatePipelineConfig.identifier\n    ),\n    DropOrthogonalCandidates(\n      orthogonalCandidatePipelines = Seq(\n        forYouKeywordTrendsCandidatePipelineConfig.identifier,\n        forYouStoriesCandidatePipelineConfig.identifier,\n        forYouWhoToFollowCandidatePipelineConfig.identifier,\n        forYouWhoToSubscribeCandidatePipelineConfig.identifier,\n        forYouTweetPreviewsCandidatePipelineConfig.identifier,\n        forYouRecommendedJobsCandidatePipelineConfig.identifier,\n        forYouRecommendedRecruitingOrganizationsCandidatePipelineConfig.identifier,\n        forYouCommunitiesToJoinCandidatePipelineConfig.identifier,\n        forYouBookmarksCandidatePipelineConfig.identifier,\n      )\n    ),\n    DropOrthogonalCandidates(\n      orthogonalCandidatePipelines = Seq(\n        forYouEventsEntryPointPivotCandidatePipelineConfig.identifier,\n        forYouGrokEntryPointPivotCandidatePipelineConfig.identifier,\n      )\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouAppUpsellCandidatePipelineConfig.identifier,\n      positionParam = ForYouAppUpsellPosition\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = verifiedPromptCandidatePipelineConfig.identifier,\n      positionParam = StaticParamValueZero\n    ),\n    InsertDynamicPositionResults(\n      candidatePipeline = flipPromptCandidatePipelineConfig.identifier,\n      dynamicInsertionPosition = FlipPromptDynamicInsertionPosition(StaticParamValueZero)\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouExplorationTweetsCandidatePipelineConfig.identifier,\n      positionParam = ExplorationTweetsTimelinePosition\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouJetfuelFrameCandidatePipelineConfig.identifier,\n      positionParam = SuperbowlModuleTimelinePosition\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouEventsEntryPointPivotCandidatePipelineConfig.identifier,\n      positionParam = SuperbowlModuleTimelinePosition\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouGrokEntryPointPivotCandidatePipelineConfig.identifier,\n      positionParam = GrokPivotModuleTimelinePosition\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouPinnedTweetsCandidatePipelineConfig.identifier,\n      positionParam = StaticParam(3)\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouScoredVideoTweetsCandidatePipelineConfig.identifier,\n      positionParam = VideoCarouselTimelinePosition\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouTuneFeedCandidatePipelineConfig.identifier,\n      positionParam = TuneFeedTimelinePosition\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouTopicSelectorCandidatePipelineConfig.identifier,\n      positionParam = ForYouTopicSelectorPosition\n    ),\n    InsertFixedPositionResults(\n      candidatePipelines = Set(\n        forYouKeywordTrendsCandidatePipelineConfig.identifier,\n        forYouStoriesCandidatePipelineConfig.identifier,\n        forYouWhoToFollowCandidatePipelineConfig.identifier,\n        forYouWhoToSubscribeCandidatePipelineConfig.identifier,\n        forYouTweetPreviewsCandidatePipelineConfig.identifier,\n        forYouRecommendedJobsCandidatePipelineConfig.identifier,\n        forYouRecommendedRecruitingOrganizationsCandidatePipelineConfig.identifier,\n        forYouCommunitiesToJoinCandidatePipelineConfig.identifier,\n        forYouBookmarksCandidatePipelineConfig.identifier,\n      ),\n      positionParam = StaticParamValueFive\n    ),\n    DropModuleTooFewModuleItemResults(\n      candidatePipeline = forYouWhoToFollowCandidatePipelineConfig.identifier,\n      minModuleItemsParam = StaticParam(WhoToFollowCandidatePipelineConfig.MinCandidatesSize)\n    ),\n    DropModuleTooFewModuleItemResults(\n      candidatePipeline = forYouWhoToSubscribeCandidatePipelineConfig.identifier,\n      minModuleItemsParam = StaticParam(WhoToSubscribeCandidatePipelineConfig.MinCandidatesSize)\n    ),\n    DropModuleTooFewModuleItemResults(\n      candidatePipeline = forYouBookmarksCandidatePipelineConfig.identifier,\n      minModuleItemsParam = StaticParam(2)\n    ),\n    DropModuleTooFewModuleItemResults(\n      candidatePipeline = forYouScoredVideoTweetsCandidatePipelineConfig.identifier,\n      minModuleItemsParam = StaticParam(3)\n    ),\n    DropModuleTooFewModuleItemResults(\n      candidatePipeline = forYouStoriesCandidatePipelineConfig.identifier,\n      minModuleItemsParam = StaticParam(3)\n    ),\n    DropModuleTooFewModuleItemResults(\n      candidatePipeline = forYouKeywordTrendsCandidatePipelineConfig.identifier,\n      minModuleItemsParam = StaticParam(3)\n    ),\n    SelectConditionally.paramNotGated(\n      InsertAdResults(\n        surfaceAreaName = AdsInjectionSurfaceAreas.HomeTimeline,\n        adsInjector = adsInjector.forSurfaceArea(AdsInjectionSurfaceAreas.HomeTimeline),\n        adsCandidatePipeline = forYouAdsCandidatePipelineConfig.identifier\n      ),\n      EnableForYouTimelineAdsSurface\n    ),\n    SelectConditionally.paramGated(\n      InsertAdResults(\n        surfaceAreaName = AdsInjectionSurfaceAreas.ForYouTimeline,\n        adsInjector = adsInjector.forSurfaceArea(AdsInjectionSurfaceAreas.ForYouTimeline),\n        adsCandidatePipeline = forYouAdsCandidatePipelineConfig.identifier\n      ),\n      EnableForYouTimelineAdsSurface\n    ),\n    SelectConditionally(\n      DebugUpdateSortAdsResult(forYouAdsCandidatePipelineConfig.identifier),\n      includeSelector = (query, _, _) => query.params(EnableAdsDebugParam)\n    ),\n    // This selector must come after the tweets are inserted into the results\n    UpdateNewTweetsPillDecoration(\n      pipelineScope = SpecificPipelines(\n        forYouScoredTweetsCandidatePipelineConfig.identifier,\n        newTweetsPillCandidatePipelineConfig.identifier\n      ),\n      stringCenter = stringCenterProvider.get(),\n      seeNewTweetsString = externalStrings.seeNewTweetsString,\n      tweetedString = externalStrings.tweetedString\n    ),\n    InsertAppendResults(candidatePipeline = editedTweetsCandidatePipelineConfig.identifier),\n    SelectConditionally(\n      selector =\n        InsertAppendResults(candidatePipeline = newTweetsPillCandidatePipelineConfig.identifier),\n      includeSelector = (_, _, results) => CandidatesUtil.containsType[TweetCandidate](results)\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouRelevancePromptCandidatePipelineConfig.identifier,\n      positionParam = RelevancePromptTweetPositionParam\n    ),\n    UpdateHomeClientEventDetails(\n      candidatePipelines = Set(\n        forYouScoredTweetsCandidatePipelineConfig.identifier,\n        forYouTweetPreviewsCandidatePipelineConfig.identifier,\n        forYouExplorationTweetsCandidatePipelineConfig.identifier,\n        forYouBookmarksCandidatePipelineConfig.identifier,\n        forYouPinnedTweetsCandidatePipelineConfig.identifier,\n        forYouScoredVideoTweetsCandidatePipelineConfig.identifier,\n        forYouTuneFeedCandidatePipelineConfig.identifier,\n      )\n    )\n  )\n\n  private val servedCandidateFeatureKeysKafkaSideEffect =\n    servedCandidateFeatureKeysKafkaSideEffectBuilder.build(\n      Set(forYouScoredTweetsCandidatePipelineConfig.identifier))\n\n  private val homeScribeClientEventSideEffect = HomeScribeClientEventSideEffect(\n    enableScribeClientEvents = enableScribeClientEvents,\n    logPipelinePublisher = clientEventsScribeEventPublisher,\n    injectedTweetsCandidatePipelineIdentifiers =\n      Seq(forYouScoredTweetsCandidatePipelineConfig.identifier),\n    adsCandidatePipelineIdentifier = Some(forYouAdsCandidatePipelineConfig.identifier),\n    whoToFollowCandidatePipelineIdentifier =\n      Some(forYouWhoToFollowCandidatePipelineConfig.identifier),\n    whoToSubscribeCandidatePipelineIdentifier =\n      Some(forYouWhoToSubscribeCandidatePipelineConfig.identifier),\n    forYouCommunitiesToJoinCandidatePipelineIdentifier =\n      Some(forYouCommunitiesToJoinCandidatePipelineConfig.identifier),\n    forYouRelevancePromptCandidatePipelineIdentifier =\n      Some(forYouRelevancePromptCandidatePipelineConfig.identifier)\n  )\n\n  override val resultSideEffects: Seq[PipelineResultSideEffect[ForYouQuery, Timeline]] = Seq(\n    updateTimelinesPersistenceStoreSideEffect,\n    truncateTimelinesPersistenceStoreSideEffect,\n    homeScribeClientEventSideEffect,\n    homeScribeServedCandidatesSideEffect,\n    servedCandidateFeatureKeysKafkaSideEffect,\n    ServedStatsSideEffect(\n      candidatePipelines = Set(forYouScoredTweetsCandidatePipelineConfig.identifier),\n      statsReceiver = statsReceiver\n    ),\n    VideoServedStatsSideEffect(\n      candidatePipelines = Set(forYouScoredTweetsCandidatePipelineConfig.identifier),\n      statsReceiver = statsReceiver\n    ),\n  )\n\n  override val domainMarshaller: DomainMarshaller[ForYouQuery, Timeline] =\n    ForYouResponseDomainMarshaller\n\n  override val transportMarshaller: TransportMarshaller[Timeline, urt.TimelineResponse] =\n    urtTransportMarshaller\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouPinnedTweetsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.PinnedTweetBroadcastCandidateDecorator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.GizmoduckAuthorFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.InNetworkFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.CurrentPinnedTweetFilter\nimport com.twitter.home_mixer.functional_component.filter.TweetHydrationFilter\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.product.for_you.candidate_source.BroadcastedPinnedTweetsCandidateSource\nimport com.twitter.home_mixer.product.for_you.candidate_source.PinnedTweetCandidate\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.CurrentPinnedTweetFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.filter.NotArticleFilter\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnablePinnedTweetsCandidatePipelineParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.PinnedTweetsModuleMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.response_transformer.PinnedTweetResponseFeatureTransformer\nimport com.twitter.home_mixer.product.for_you.scorer.PinnedTweetCandidateScorer\nimport com.twitter.product_mixer.component_library.filter.FeatureFilter\nimport com.twitter.product_mixer.component_library.gate.DefinedUserIdGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouPinnedTweetsCandidatePipelineConfig @Inject() (\n  broadcastedPinnedTweetsCandidateSource: BroadcastedPinnedTweetsCandidateSource,\n  currentPinnedTweetFeatureHydrator: CurrentPinnedTweetFeatureHydrator,\n  gizmoduckAuthorFeatureHydrator: GizmoduckAuthorFeatureHydrator,\n  tweetypieFeatureHydrator: TweetypieFeatureHydrator,\n  pinnedTweetBroadcastCandidateDecorator: PinnedTweetBroadcastCandidateDecorator,\n) extends CandidatePipelineConfig[\n      ForYouQuery,\n      ForYouQuery,\n      PinnedTweetCandidate,\n      TweetCandidate\n    ] {\n\n  private val InNetworkFilterId = \"InNetwork\"\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouPinnedTweets\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] = Some(\n    EnablePinnedTweetsCandidatePipelineParam)\n\n  override val gates: Seq[Gate[PipelineQuery]] = Seq(\n    DefinedUserIdGate,\n    RateLimitGate,\n    TimelinesPersistenceStoreLastInjectionGate(\n      PinnedTweetsModuleMinInjectionIntervalParam,\n      EntityIdType.PinnedTweetsModule\n    ),\n  )\n\n  override val queryFeatureHydration: Seq[QueryFeatureHydrator[PipelineQuery]] = Seq()\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ForYouQuery,\n    ForYouQuery\n  ] = identity\n\n  override def candidateSource: CandidateSource[ForYouQuery, PinnedTweetCandidate] =\n    broadcastedPinnedTweetsCandidateSource\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[PinnedTweetCandidate]\n  ] = Seq(PinnedTweetResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    PinnedTweetCandidate,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.tweetId) }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ForYouQuery, TweetCandidate, _]\n  ] = Seq(\n    tweetypieFeatureHydrator,\n    InNetworkFeatureHydrator,\n    currentPinnedTweetFeatureHydrator,\n    gizmoduckAuthorFeatureHydrator\n  )\n\n  override val filters: Seq[Filter[ForYouQuery, TweetCandidate]] = Seq(\n    TweetHydrationFilter,\n    NotArticleFilter,\n    FeatureFilter.fromFeature(FilterIdentifier(InNetworkFilterId), InNetworkFeature),\n    CurrentPinnedTweetFilter\n  )\n\n  override def scorers: Seq[Scorer[ForYouQuery, TweetCandidate]] = Seq(PinnedTweetCandidateScorer)\n\n  override val decorator: Option[\n    CandidateDecorator[PipelineQuery, TweetCandidate]\n  ] = Some(pinnedTweetBroadcastCandidateDecorator)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouProductPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.model.request.ForYouProductContext\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParamConfig\nimport com.twitter.home_mixer.service.HomeMixerAccessPolicy.DefaultHomeMixerAccessPolicy\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig.DefaultNotificationGroup\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.timelines.ChronologicalCursorUnmarshaller\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AccessPolicy\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.common.alert.EmptyResponseRateAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.LatencyAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.P99\nimport com.twitter.product_mixer.core.functional_component.common.alert.SuccessRateAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.ThroughputAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfAbove\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfBelow\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfLatencyAbove\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ProductPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineConfig\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.BadRequest\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineConfig\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.product_mixer.core.util.SortIndexBuilder\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport com.twitter.timelines.util.RequestCursorSerializer\nimport com.twitter.util.Time\nimport com.twitter.util.Try\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouProductPipelineConfig @Inject() (\n  forYouMixerPipelineConfig: ForYouMixerPipelineConfig,\n  forYouParamConfig: ForYouParamConfig)\n    extends ProductPipelineConfig[HomeMixerRequest, ForYouQuery, urt.TimelineResponse] {\n\n  override val identifier: ProductPipelineIdentifier = ProductPipelineIdentifier(\"ForYou\")\n\n  override val product: Product = ForYouProduct\n\n  override val paramConfig: ProductParamConfig = forYouParamConfig\n\n  override def pipelineQueryTransformer(\n    request: HomeMixerRequest,\n    params: Params\n  ): ForYouQuery = {\n    val context = request.productContext match {\n      case Some(context: ForYouProductContext) => context\n      case _ => throw PipelineFailure(BadRequest, \"ForYouProductContext not found\")\n    }\n\n    val debugOptions = request.debugParams.flatMap(_.debugOptions)\n\n    /**\n     * Unlike other clients, newly created tweets on Android have the sort index set to the current\n     * time instead of the top sort index + 1, so these tweets get stuck at the top of the timeline\n     * if subsequent timeline responses use the sort index from the previous response instead of\n     * the current time.\n     */\n    val pipelineCursor = request.serializedRequestCursor.flatMap { cursor =>\n      Try(UrtCursorSerializer.deserializeOrderedCursor(cursor))\n        .getOrElse(ChronologicalCursorUnmarshaller(RequestCursorSerializer.deserialize(cursor)))\n        .map {\n          case topCursor @ UrtOrderedCursor(_, _, Some(TopCursor), _) =>\n            val queryTime = debugOptions.flatMap(_.requestTimeOverride).getOrElse(Time.now)\n            topCursor.copy(initialSortIndex = SortIndexBuilder.timeToId(queryTime))\n          case cursor => cursor\n        }\n    }\n\n    ForYouQuery(\n      params = params,\n      clientContext = request.clientContext,\n      features = None,\n      pipelineCursor = pipelineCursor,\n      requestedMaxResults = Some(params(ServerMaxResultsParam)),\n      debugOptions = debugOptions,\n      deviceContext = context.deviceContext,\n      seenTweetIds = context.seenTweetIds,\n      dspClientContext = context.dspClientContext\n    )\n  }\n\n  override val pipelines: Seq[PipelineConfig] = Seq(forYouMixerPipelineConfig)\n\n  override def pipelineSelector(query: ForYouQuery): ComponentIdentifier =\n    forYouMixerPipelineConfig.identifier\n\n  override val alerts: Seq[Alert] = Seq(\n    SuccessRateAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfBelow(99.9, 20, 30),\n      criticalPredicate = TriggerIfBelow(99.9, 30, 30),\n    ),\n    LatencyAlert(\n      notificationGroup = DefaultNotificationGroup,\n      percentile = P99,\n      warnPredicate = TriggerIfLatencyAbove(2800.millis, 15, 30),\n      criticalPredicate = TriggerIfLatencyAbove(3000.millis, 15, 30)\n    ),\n    ThroughputAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfAbove(70000),\n      criticalPredicate = TriggerIfAbove(80000)\n    ),\n    EmptyResponseRateAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfAbove(2),\n      criticalPredicate = TriggerIfAbove(3)\n    )\n  )\n\n  override val debugAccessPolicies: Set[AccessPolicy] = DefaultHomeMixerAccessPolicy\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouPushToHomeMixerPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam\nimport com.twitter.product_mixer.component_library.premarshaller.urt.UrtDomainMarshaller\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.AddEntriesInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ClearCacheInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedBottomCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedTopCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ParamGatedIncludeInstruction\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.StaticTimelineScribeConfigBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtMetadataBuilder\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.UrtTransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.MixerPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineConfig\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouPushToHomeMixerPipelineConfig @Inject() (\n  forYouPushToHomeTweetCandidatePipelineConfig: ForYouPushToHomeTweetCandidatePipelineConfig,\n  urtTransportMarshaller: UrtTransportMarshaller)\n    extends MixerPipelineConfig[ForYouQuery, Timeline, urt.TimelineResponse] {\n\n  override val identifier: MixerPipelineIdentifier = MixerPipelineIdentifier(\"ForYouPushToHome\")\n\n  override val candidatePipelines: Seq[CandidatePipelineConfig[ForYouQuery, _, _, _]] =\n    Seq(forYouPushToHomeTweetCandidatePipelineConfig)\n\n  override val resultSelectors: Seq[Selector[ForYouQuery]] =\n    Seq(InsertAppendResults(forYouPushToHomeTweetCandidatePipelineConfig.identifier))\n\n  override val domainMarshaller: DomainMarshaller[ForYouQuery, Timeline] = {\n    val instructionBuilders = Seq(\n      ClearCacheInstructionBuilder(\n        ParamGatedIncludeInstruction(ForYouParam.EnableClearCacheOnPushToHome)),\n      AddEntriesInstructionBuilder())\n\n    val idSelector: PartialFunction[UniversalNoun[_], Long] = { case item: TweetItem => item.id }\n    val topCursorBuilder = OrderedTopCursorBuilder(idSelector)\n    val bottomCursorBuilder = OrderedBottomCursorBuilder(idSelector)\n\n    val metadataBuilder = UrtMetadataBuilder(\n      title = None,\n      scribeConfigBuilder = Some(\n        StaticTimelineScribeConfigBuilder(\n          TimelineScribeConfig(\n            page = Some(\"for_you_push_to_home\"),\n            section = None,\n            entityToken = None)\n        )\n      )\n    )\n\n    UrtDomainMarshaller(\n      instructionBuilders = instructionBuilders,\n      metadataBuilder = Some(metadataBuilder),\n      cursorBuilders = Seq(topCursorBuilder, bottomCursorBuilder)\n    )\n  }\n\n  override val transportMarshaller: TransportMarshaller[Timeline, urt.TimelineResponse] =\n    urtTransportMarshaller\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouPushToHomeTweetCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeClientEventInfoBuilder\nimport com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature\nimport com.twitter.home_mixer.product.for_you.functional_component.gate.PushToHomeRequestGate\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.PassthroughCandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouPushToHomeTweetCandidatePipelineConfig @Inject() ()\n    extends CandidatePipelineConfig[\n      ForYouQuery,\n      ForYouQuery,\n      TweetCandidate,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouPushToHomeTweet\")\n\n  override val gates: Seq[Gate[ForYouQuery]] = Seq(PushToHomeRequestGate)\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ForYouQuery,\n    ForYouQuery\n  ] = identity\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[TweetCandidate]\n  ] = Seq(new CandidateFeatureTransformer[TweetCandidate] {\n    override def features: Set[Feature[_, _]] = Set(SuggestTypeFeature)\n\n    override val identifier: TransformerIdentifier =\n      TransformerIdentifier(\"ForYouPushToHomeTweet\")\n\n    override def transform(input: TweetCandidate): FeatureMap =\n      FeatureMapBuilder().add(SuggestTypeFeature, Some(st.SuggestType.Magicrec)).build()\n  })\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    TweetCandidate,\n    TweetCandidate\n  ] = identity\n\n  override val candidateSource: CandidateSource[\n    ForYouQuery,\n    TweetCandidate\n  ] = PassthroughCandidateSource(\n    CandidateSourceIdentifier(\"PushToHomeTweet\"),\n    { query => query.pushToHomeTweetId.toSeq.map(TweetCandidate(_)) }\n  )\n\n  override val decorator: Option[\n    CandidateDecorator[ForYouQuery, TweetCandidate]\n  ] = {\n    val tweetItemBuilder = TweetCandidateUrtItemBuilder(\n      clientEventInfoBuilder = HomeClientEventInfoBuilder()\n    )\n    Some(UrtItemCandidateDecorator(tweetItemBuilder))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouRecommendedJobsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.candidateservice.core_io.FetchJobRecommendationsView\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.FeedbackStrings\nimport com.twitter.home_mixer.functional_component.gate.DismissFatigueGate\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.model.HomeFeatures.DismissInfoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerHasJobRecommendationsEnabled\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.product.for_you.candidate_source.RecommendedJobsCandidateSource\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableRecommendedJobsParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.MaxRecommendedJobCandidatesParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.RecommendedJobMinInjectionIntervalParam\nimport com.twitter.product_mixer.component_library.gate.DefinedUserIdGate\nimport com.twitter.product_mixer.component_library.gate.FeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.JobCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.job.RecommendedJobsCandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyView\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouRecommendedJobsCandidatePipelineConfig @Inject() (\n  recommendedJobsProductCandidateSource: RecommendedJobsCandidateSource,\n  @ProductScoped stringCenterProvider: Provider[StringCenter],\n  externalStrings: HomeMixerExternalStrings,\n  feedbackStrings: FeedbackStrings)\n    extends CandidatePipelineConfig[\n      PipelineQuery,\n      StratoKeyView[Long, FetchJobRecommendationsView],\n      Long,\n      JobCandidate\n    ] {\n\n  private val stringCenter = stringCenterProvider.get()\n\n  override val supportedClientParam: Option[FSParam[Boolean]] =\n    Some(EnableRecommendedJobsParam)\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouRecommendedJobs\")\n\n  override val gates = Seq(\n    DefinedUserIdGate,\n    FeatureGate.fromFeature(ViewerHasJobRecommendationsEnabled),\n    RateLimitGate,\n    TimelinesPersistenceStoreLastInjectionGate(\n      RecommendedJobMinInjectionIntervalParam,\n      EntityIdType.JobModule\n    ),\n    DismissFatigueGate(st.SuggestType.Job, DismissInfoFeature)\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    PipelineQuery,\n    StratoKeyView[Long, FetchJobRecommendationsView]\n  ] = { query =>\n    StratoKeyView(\n      query.getRequiredUserId,\n      FetchJobRecommendationsView(count = Some(query.params(MaxRecommendedJobCandidatesParam))))\n  }\n\n  override val candidateSource: BaseCandidateSource[\n    StratoKeyView[Long, FetchJobRecommendationsView],\n    Long\n  ] = recommendedJobsProductCandidateSource\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    Long,\n    JobCandidate\n  ] = { jobResult => JobCandidate(id = jobResult) }\n\n  override val decorator: Option[CandidateDecorator[PipelineQuery, JobCandidate]] = {\n    Some(\n      RecommendedJobsCandidateDecorator(\n        stringCenter = stringCenter,\n        headerString = externalStrings.RecommendedJobHeaderString,\n        footerString = externalStrings.RecommendedJobFooterString,\n        seeLessOftenString = feedbackStrings.seeLessOftenFeedbackString,\n        seeLessOftenConfirmationString = feedbackStrings.seeLessOftenConfirmationFeedbackString\n      ))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouRecommendedRecruitingOrganizationsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.FeedbackStrings\nimport com.twitter.home_mixer.functional_component.gate.DismissFatigueGate\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.model.HomeFeatures.DismissInfoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerHasRecruitingOrganizationRecommendationsEnabled\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.product.for_you.candidate_source.RecommendedRecruitingOrganizationsCandidateSource\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableRecommendedRecruitingOrganizationsParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.MaxRecommendedRecruitingOrganizationCandidatesParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.RecommendedRecruitingOrganizationMinInjectionIntervalParam\nimport com.twitter.product_mixer.component_library.gate.DefinedUserIdGate\nimport com.twitter.product_mixer.component_library.gate.FeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.RecruitingOrganizationCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.recruiting_organization.RecommendedRecruitingOrganizationsCandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyView\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.recruiting.organization.RecruitingOrganizationRecommendationsView\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouRecommendedRecruitingOrganizationsCandidatePipelineConfig @Inject() (\n  recommendedRecruitingOrganizationsProductCandidateSource: RecommendedRecruitingOrganizationsCandidateSource,\n  @ProductScoped stringCenterProvider: Provider[StringCenter],\n  externalStrings: HomeMixerExternalStrings,\n  feedbackStrings: FeedbackStrings)\n    extends CandidatePipelineConfig[\n      PipelineQuery,\n      StratoKeyView[Long, RecruitingOrganizationRecommendationsView],\n      Long,\n      RecruitingOrganizationCandidate\n    ] {\n\n  private val stringCenter = stringCenterProvider.get()\n\n  override val supportedClientParam: Option[FSParam[Boolean]] =\n    Some(EnableRecommendedRecruitingOrganizationsParam)\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouRecommendedRecruitingOrganizations\")\n\n  override val gates = Seq(\n    DefinedUserIdGate,\n    FeatureGate.fromFeature(ViewerHasRecruitingOrganizationRecommendationsEnabled),\n    RateLimitGate,\n    TimelinesPersistenceStoreLastInjectionGate(\n      RecommendedRecruitingOrganizationMinInjectionIntervalParam,\n      EntityIdType.RecruitingOrganizationModule\n    ),\n    DismissFatigueGate(st.SuggestType.RecruitingOrganization, DismissInfoFeature)\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    PipelineQuery,\n    StratoKeyView[Long, RecruitingOrganizationRecommendationsView]\n  ] = { query =>\n    StratoKeyView(\n      query.getRequiredUserId,\n      RecruitingOrganizationRecommendationsView(count =\n        Some(query.params(MaxRecommendedRecruitingOrganizationCandidatesParam)))\n    )\n  }\n\n  override val candidateSource: BaseCandidateSource[\n    StratoKeyView[Long, RecruitingOrganizationRecommendationsView],\n    Long\n  ] =\n    recommendedRecruitingOrganizationsProductCandidateSource\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    Long,\n    RecruitingOrganizationCandidate\n  ] = { orgResult => RecruitingOrganizationCandidate(id = orgResult) }\n\n  override val decorator: Option[\n    CandidateDecorator[PipelineQuery, RecruitingOrganizationCandidate]\n  ] = {\n    Some(\n      RecommendedRecruitingOrganizationsCandidateDecorator(\n        stringCenter = stringCenter,\n        headerString = externalStrings.RecommendedRecruitingOrganizationHeaderString,\n        footerString = externalStrings.RecommendedRecruitingOrganizationFooterString,\n        seeLessOftenString = feedbackStrings.seeLessOftenFeedbackString,\n        seeLessOftenConfirmationString = feedbackStrings.seeLessOftenConfirmationFeedbackString\n      ))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouRelevancePromptCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.RelevancePromptCandidateUrtItemBuilder\nimport com.twitter.home_mixer.functional_component.gate.PersistenceStoreDurationValidationGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.RelevancePromptEnableParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.RelevancePromptMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.RelevancePromptNegativeParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.RelevancePromptNeutralParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.RelevancePromptPositiveParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.RelevancePromptTitleParam\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.model.candidate.RelevancePromptCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.StaticCandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouRelevancePromptCandidatePipelineConfig @Inject() ()\n    extends CandidatePipelineConfig[ForYouQuery, Unit, Unit, RelevancePromptCandidate] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouRelevancePrompt\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] = Some(RelevancePromptEnableParam)\n\n  override val gates = Seq(\n    PersistenceStoreDurationValidationGate(),\n    TimelinesPersistenceStoreLastInjectionGate(\n      RelevancePromptMinInjectionIntervalParam,\n      EntityIdType.Annotation\n    )\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[ForYouQuery, Unit] = _ => Unit\n\n  override def candidateSource: CandidateSource[Unit, Unit] = StaticCandidateSource[Unit](\n    identifier = CandidateSourceIdentifier(identifier.name),\n    result = Seq(Unit)\n  )\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    Unit,\n    RelevancePromptCandidate\n  ] = _ => RelevancePromptCandidate(id = \"feed-survey-prompt\")\n\n  override val decorator: Option[CandidateDecorator[ForYouQuery, RelevancePromptCandidate]] = Some(\n    UrtItemCandidateDecorator(\n      RelevancePromptCandidateUrtItemBuilder(\n        RelevancePromptTitleParam,\n        RelevancePromptPositiveParam,\n        RelevancePromptNegativeParam,\n        RelevancePromptNeutralParam\n      ))\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouResponseDomainMarshaller.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder\nimport com.twitter.home_mixer.model.ClearCacheIncludeInstruction\nimport com.twitter.home_mixer.model.NavigationIncludeInstruction\nimport com.twitter.home_mixer.model.request.DeviceContext.RequestContext\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ClearCache\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.Navigation\nimport com.twitter.product_mixer.component_library.premarshaller.urt.UrtDomainMarshaller\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ClearCacheInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.NavigationInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedBottomCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedCursorIdSelector.TweetIdSelector\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedTopCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceAllEntries\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceEntryInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ShowAlertInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ShowCoverInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.StaticTimelineScribeConfigBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtMetadataBuilder\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.model.common.identifier.DomainMarshallerIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\n\nobject ForYouResponseDomainMarshaller extends DomainMarshaller[ForYouQuery, Timeline] {\n\n  override val identifier: DomainMarshallerIdentifier =\n    DomainMarshallerIdentifier(\"ForYouResponse\")\n\n  override def apply(\n    query: ForYouQuery,\n    selections: Seq[CandidateWithDetails]\n  ): Timeline = {\n    val coldStart =\n      query.deviceContext.flatMap(_.requestContextValue).contains(RequestContext.Launch)\n    val retainViewportItems =\n      if (coldStart && query.params(ClearCache.ColdStartRetainViewportParam)) Some(true) else None\n\n    val instructionBuilders = Seq(\n      ClearCacheInstructionBuilder(\n        includeInstruction = ClearCacheIncludeInstruction(\n          ClearCache.PtrEnableParam,\n          ClearCache.ColdStartEnableParam,\n          ClearCache.WarmStartEnableParam,\n          ClearCache.ManualRefreshEnableParam,\n          ClearCache.NavigateEnableParam,\n          ClearCache.MinEntriesParam\n        ),\n        retainViewportItems = retainViewportItems\n      ),\n      ReplaceEntryInstructionBuilder(ReplaceAllEntries),\n      // excludes alert, cover, and replace candidates\n      AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder(),\n      ShowAlertInstructionBuilder(),\n      ShowCoverInstructionBuilder(),\n      NavigationInstructionBuilder(\n        NavigationIncludeInstruction(\n          Navigation.PtrEnableParam,\n          Navigation.ColdStartEnableParam,\n          Navigation.WarmStartEnableParam,\n          Navigation.ManualRefreshEnableParam,\n          Navigation.NavigateEnableParam\n        ))\n    )\n\n    val topCursorBuilder = OrderedTopCursorBuilder(TweetIdSelector)\n    val bottomCursorBuilder = OrderedBottomCursorBuilder(TweetIdSelector)\n\n    val scribeConfigBuilder =\n      StaticTimelineScribeConfigBuilder(TimelineScribeConfig(page = Some(\"for_you\"), None, None))\n    val metadataBuilder = UrtMetadataBuilder(scribeConfigBuilder = Some(scribeConfigBuilder))\n\n    val domainMarshaller = UrtDomainMarshaller(\n      instructionBuilders = instructionBuilders,\n      metadataBuilder = Some(metadataBuilder),\n      cursorBuilders = Seq(topCursorBuilder, bottomCursorBuilder)\n    )\n\n    domainMarshaller(query, selections)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouScoredTweetsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.ForYouTweetCandidateDecorator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.BasketballContextFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.InNetworkFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.PostContextFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.InvalidSubscriptionTweetFilter\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableBasketballContextFeatureHydratorParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnablePostContextFeatureHydratorParam\nimport com.twitter.home_mixer.product.for_you.candidate_source.ScoredTweetWithConversationMetadata\nimport com.twitter.home_mixer.product.for_you.candidate_source.ScoredTweetsProductCandidateSource\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.FocalTweetFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.communities.CommunityNamesFeatureHydrator\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedBulkCandidateFeatureHydrator\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouScoredTweetsCandidatePipelineConfig @Inject() (\n  scoredTweetsProductCandidateSource: ScoredTweetsProductCandidateSource,\n  focalTweetFeatureHydrator: FocalTweetFeatureHydrator,\n  namesFeatureHydrator: NamesFeatureHydrator,\n  communityNamesFeatureHydrator: CommunityNamesFeatureHydrator,\n  basketballContextFeatureHydrator: BasketballContextFeatureHydrator,\n  postContextFeatureHydrator: PostContextFeatureHydrator,\n  invalidSubscriptionTweetFilter: InvalidSubscriptionTweetFilter,\n  forYouTweetCandidateDecorator: ForYouTweetCandidateDecorator)\n    extends CandidatePipelineConfig[\n      ForYouQuery,\n      ForYouQuery,\n      ScoredTweetWithConversationMetadata,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouScoredTweets\")\n\n  override val gates: Seq[Gate[ForYouQuery]] = Seq(RateLimitGate)\n\n  override val candidateSource: CandidateSource[ForYouQuery, ScoredTweetWithConversationMetadata] =\n    scoredTweetsProductCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[ForYouQuery, ForYouQuery] =\n    identity\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[ScoredTweetWithConversationMetadata]\n  ] = Seq(ForYouScoredTweetsResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    ScoredTweetWithConversationMetadata,\n    TweetCandidate\n  ] = { sourceResults => TweetCandidate(sourceResults.tweetId) }\n\n  override val filters: Seq[Filter[ForYouQuery, TweetCandidate]] = Seq(\n    invalidSubscriptionTweetFilter\n  )\n\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[ForYouQuery, TweetCandidate, _]\n  ] = Seq(\n    focalTweetFeatureHydrator,\n    InNetworkFeatureHydrator,\n    namesFeatureHydrator,\n    communityNamesFeatureHydrator,\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnableBasketballContextFeatureHydratorParam,\n      basketballContextFeatureHydrator\n    ),\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnablePostContextFeatureHydratorParam,\n      postContextFeatureHydrator\n    ),\n  )\n\n  override val decorator: Option[CandidateDecorator[ForYouQuery, TweetCandidate]] =\n    Some(forYouTweetCandidateDecorator.build())\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),\n    HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert(10, 20)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouScoredTweetsMixerPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.clientapp.{thriftscala => ca}\nimport com.twitter.goldfinch.api.AdsInjectionSurfaceAreas\nimport com.twitter.home_mixer.candidate_pipeline.EditedTweetsCandidatePipelineConfig\nimport com.twitter.home_mixer.candidate_pipeline.NewTweetsPillCandidatePipelineConfig\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator._\nimport com.twitter.home_mixer.functional_component.selector.DebunchCandidates\nimport com.twitter.home_mixer.functional_component.selector.UpdateConversationModuleId\nimport com.twitter.home_mixer.functional_component.selector.UpdateHomeClientEventDetails\nimport com.twitter.home_mixer.functional_component.selector.UpdateNewTweetsPillDecoration\nimport com.twitter.home_mixer.functional_component.side_effect._\nimport com.twitter.home_mixer.model.ClearCacheIncludeInstruction\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.MaxNumberReplaceInstructionsParam\nimport com.twitter.home_mixer.param.HomeMixerFlagName.ScribeClientEventsFlag\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.TimelineServiceTweetsQueryFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ClearCacheOnPtr\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableFlipInjectionModuleCandidatePipelineParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.FlipInlineInjectionModulePosition\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.TweetPreviewsPositionParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.WhoToFollowPositionParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.WhoToSubscribePositionParam\nimport com.twitter.home_mixer.product.for_you.side_effect.ServedCandidateFeatureKeysKafkaSideEffectBuilder\nimport com.twitter.home_mixer.product.for_you.side_effect.ServedCandidateKeysKafkaSideEffectBuilder\nimport com.twitter.home_mixer.product.for_you.side_effect.ServedStatsSideEffect\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.async.AsyncQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.PreviewCreatorsQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.FlipPromptCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_subscribe_module.WhoToSubscribeCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.premarshaller.urt.UrtDomainMarshaller\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ClearCacheInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedBottomCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedTopCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceAllEntries\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceEntryInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ShowAlertInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ShowCoverInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.StaticTimelineScribeConfigBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtMetadataBuilder\nimport com.twitter.product_mixer.component_library.selector.DropMaxCandidates\nimport com.twitter.product_mixer.component_library.selector.DropMaxModuleItemCandidates\nimport com.twitter.product_mixer.component_library.selector.DropModuleTooFewModuleItemResults\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.component_library.selector.InsertFixedPositionResults\nimport com.twitter.product_mixer.component_library.selector.SelectConditionally\nimport com.twitter.product_mixer.component_library.selector.UpdateSortCandidates\nimport com.twitter.product_mixer.component_library.selector.UpdateSortModuleItemCandidates\nimport com.twitter.product_mixer.component_library.selector.ads.AdsInjector\nimport com.twitter.product_mixer.component_library.selector.ads.InsertAdResults\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.UrtTransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.MixerPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineConfig\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouScoredTweetsMixerPipelineConfig @Inject() (\n  forYouAdsDependentCandidatePipelineBuilder: ForYouAdsDependentCandidatePipelineBuilder,\n  forYouConversationServiceCandidatePipelineConfig: ForYouConversationServiceCandidatePipelineConfig,\n  forYouPushToHomeTweetCandidatePipelineConfig: ForYouPushToHomeTweetCandidatePipelineConfig,\n  forYouScoredTweetsCandidatePipelineConfig: ForYouScoredTweetsCandidatePipelineConfig,\n  forYouWhoToFollowCandidatePipelineConfigBuilder: ForYouWhoToFollowCandidatePipelineConfigBuilder,\n  forYouWhoToSubscribeCandidatePipelineConfigBuilder: ForYouWhoToSubscribeCandidatePipelineConfigBuilder,\n  flipPromptCandidatePipelineConfigBuilder: FlipPromptCandidatePipelineConfigBuilder,\n  editedTweetsCandidatePipelineConfig: EditedTweetsCandidatePipelineConfig,\n  newTweetsPillCandidatePipelineConfig: NewTweetsPillCandidatePipelineConfig[ForYouQuery],\n  forYouTweetPreviewsCandidatePipelineConfig: ForYouTweetPreviewsCandidatePipelineConfig,\n  dismissInfoQueryFeatureHydrator: DismissInfoQueryFeatureHydrator,\n  gizmoduckUserQueryFeatureHydrator: GizmoduckUserQueryFeatureHydrator,\n  persistenceStoreQueryFeatureHydrator: PersistenceStoreQueryFeatureHydrator,\n  requestQueryFeatureHydrator: RequestQueryFeatureHydrator[ForYouQuery],\n  timelineServiceTweetsQueryFeatureHydrator: TimelineServiceTweetsQueryFeatureHydrator,\n  previewCreatorsQueryFeatureHydrator: PreviewCreatorsQueryFeatureHydrator,\n  sgsFollowedUsersQueryFeatureHydrator: SGSFollowedUsersQueryFeatureHydrator,\n  adsInjector: AdsInjector,\n  servedCandidateKeysKafkaSideEffectBuilder: ServedCandidateKeysKafkaSideEffectBuilder,\n  servedCandidateFeatureKeysKafkaSideEffectBuilder: ServedCandidateFeatureKeysKafkaSideEffectBuilder,\n  updateTimelinesPersistenceStoreSideEffect: UpdateTimelinesPersistenceStoreSideEffect,\n  truncateTimelinesPersistenceStoreSideEffect: TruncateTimelinesPersistenceStoreSideEffect,\n  homeScribeServedCandidatesSideEffect: HomeScribeServedCandidatesSideEffect,\n  servedStatsSideEffect: ServedStatsSideEffect,\n  clientEventsScribeEventPublisher: EventPublisher[ca.LogEvent],\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter],\n  urtTransportMarshaller: UrtTransportMarshaller,\n  @Flag(ScribeClientEventsFlag) enableScribeClientEvents: Boolean)\n    extends MixerPipelineConfig[ForYouQuery, Timeline, urt.TimelineResponse] {\n\n  override val identifier: MixerPipelineIdentifier = MixerPipelineIdentifier(\"ForYouScoredTweets\")\n\n  private val MaxConsecutiveOutOfNetworkCandidates = 2\n\n  private val PushToHomeTweetPosition = 0\n\n  private val dependentCandidatesStep = MixerPipelineConfig.dependentCandidatePipelinesStep\n\n  override val fetchQueryFeatures: Seq[QueryFeatureHydrator[ForYouQuery]] = Seq(\n    requestQueryFeatureHydrator,\n    persistenceStoreQueryFeatureHydrator,\n    timelineServiceTweetsQueryFeatureHydrator,\n    previewCreatorsQueryFeatureHydrator,\n    sgsFollowedUsersQueryFeatureHydrator,\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, dismissInfoQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, gizmoduckUserQueryFeatureHydrator),\n  )\n\n  private val scoredTweetsCandidatePipelineScope =\n    SpecificPipeline(forYouScoredTweetsCandidatePipelineConfig.identifier)\n\n  private val forYouAdsCandidatePipelineConfig = forYouAdsDependentCandidatePipelineBuilder\n    .build(scoredTweetsCandidatePipelineScope)\n\n  private val forYouWhoToFollowCandidatePipelineConfig =\n    forYouWhoToFollowCandidatePipelineConfigBuilder.build()\n\n  private val forYouWhoToSubscribeCandidatePipelineConfig =\n    forYouWhoToSubscribeCandidatePipelineConfigBuilder.build()\n\n  private val flipPromptCandidatePipelineConfig =\n    flipPromptCandidatePipelineConfigBuilder.build[ForYouQuery](\n      supportedClientParam = Some(EnableFlipInjectionModuleCandidatePipelineParam)\n    )\n\n  override val candidatePipelines: Seq[CandidatePipelineConfig[ForYouQuery, _, _, _]] = Seq(\n    forYouScoredTweetsCandidatePipelineConfig,\n    forYouPushToHomeTweetCandidatePipelineConfig,\n    forYouWhoToFollowCandidatePipelineConfig,\n    forYouWhoToSubscribeCandidatePipelineConfig,\n    forYouTweetPreviewsCandidatePipelineConfig,\n    flipPromptCandidatePipelineConfig\n  )\n\n  override val dependentCandidatePipelines: Seq[\n    DependentCandidatePipelineConfig[ForYouQuery, _, _, _]\n  ] = Seq(\n    forYouAdsCandidatePipelineConfig,\n    forYouConversationServiceCandidatePipelineConfig,\n    editedTweetsCandidatePipelineConfig,\n    newTweetsPillCandidatePipelineConfig\n  )\n\n  override val failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy] = Map(\n    forYouScoredTweetsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouAdsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouWhoToFollowCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouWhoToSubscribeCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouTweetPreviewsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    flipPromptCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    editedTweetsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    newTweetsPillCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n  )\n\n  override val resultSelectors: Seq[Selector[ForYouQuery]] = Seq(\n    UpdateSortCandidates(\n      ordering = CandidatesUtil.reverseChronTweetsOrdering,\n      candidatePipeline = forYouConversationServiceCandidatePipelineConfig.identifier\n    ),\n    UpdateSortCandidates(\n      ordering = CandidatesUtil.scoreOrdering,\n      candidatePipeline = forYouScoredTweetsCandidatePipelineConfig.identifier\n    ),\n    UpdateSortModuleItemCandidates(\n      candidatePipeline = forYouScoredTweetsCandidatePipelineConfig.identifier,\n      ordering = CandidatesUtil.conversationModuleTweetsOrdering\n    ),\n    DebunchCandidates(\n      pipelineScope = SpecificPipeline(forYouScoredTweetsCandidatePipelineConfig.identifier),\n      mustDebunch = {\n        case item: ItemCandidateWithDetails =>\n          !item.features.getOrElse(InNetworkFeature, false)\n        case module: ModuleCandidateWithDetails =>\n          !module.candidates.last.features.getOrElse(InNetworkFeature, false)\n      },\n      maxBunchSize = MaxConsecutiveOutOfNetworkCandidates\n    ),\n    UpdateConversationModuleId(\n      pipelineScope = SpecificPipeline(forYouScoredTweetsCandidatePipelineConfig.identifier)\n    ),\n    DropMaxCandidates(\n      candidatePipeline = forYouConversationServiceCandidatePipelineConfig.identifier,\n      maxSelectionsParam = ServerMaxResultsParam\n    ),\n    DropMaxCandidates(\n      candidatePipeline = forYouScoredTweetsCandidatePipelineConfig.identifier,\n      maxSelectionsParam = ServerMaxResultsParam\n    ),\n    DropMaxCandidates(\n      candidatePipeline = editedTweetsCandidatePipelineConfig.identifier,\n      maxSelectionsParam = MaxNumberReplaceInstructionsParam\n    ),\n    DropMaxModuleItemCandidates(\n      candidatePipeline = forYouWhoToFollowCandidatePipelineConfig.identifier,\n      maxModuleItemsParam = StaticParam(WhoToFollowCandidatePipelineConfig.MaxCandidatesSize)\n    ),\n    DropMaxModuleItemCandidates(\n      candidatePipeline = forYouWhoToSubscribeCandidatePipelineConfig.identifier,\n      maxModuleItemsParam = StaticParam(WhoToSubscribeCandidatePipelineConfig.MaxCandidatesSize)\n    ),\n    // The Conversation Service pipeline will only run if the Scored Tweets pipeline returned nothing\n    InsertAppendResults(\n      candidatePipeline = forYouConversationServiceCandidatePipelineConfig.identifier\n    ),\n    InsertAppendResults(\n      candidatePipeline = forYouScoredTweetsCandidatePipelineConfig.identifier\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouTweetPreviewsCandidatePipelineConfig.identifier,\n      positionParam = TweetPreviewsPositionParam\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouWhoToFollowCandidatePipelineConfig.identifier,\n      positionParam = WhoToFollowPositionParam\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouWhoToSubscribeCandidatePipelineConfig.identifier,\n      positionParam = WhoToSubscribePositionParam\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = flipPromptCandidatePipelineConfig.identifier,\n      positionParam = FlipInlineInjectionModulePosition\n    ),\n    // Insert Push To Home Tweet at top of Timeline\n    InsertFixedPositionResults(\n      candidatePipeline = forYouPushToHomeTweetCandidatePipelineConfig.identifier,\n      positionParam = StaticParam(PushToHomeTweetPosition)\n    ),\n    InsertAdResults(\n      surfaceAreaName = AdsInjectionSurfaceAreas.HomeTimeline,\n      adsInjector = adsInjector.forSurfaceArea(AdsInjectionSurfaceAreas.HomeTimeline),\n      adsCandidatePipeline = forYouAdsCandidatePipelineConfig.identifier\n    ),\n    // This selector must come after the tweets are inserted into the results\n    DropModuleTooFewModuleItemResults(\n      candidatePipeline = forYouWhoToFollowCandidatePipelineConfig.identifier,\n      minModuleItemsParam = StaticParam(WhoToFollowCandidatePipelineConfig.MinCandidatesSize)\n    ),\n    DropModuleTooFewModuleItemResults(\n      candidatePipeline = forYouWhoToSubscribeCandidatePipelineConfig.identifier,\n      minModuleItemsParam = StaticParam(WhoToSubscribeCandidatePipelineConfig.MinCandidatesSize)\n    ),\n    UpdateNewTweetsPillDecoration(\n      pipelineScope = SpecificPipelines(\n        forYouConversationServiceCandidatePipelineConfig.identifier,\n        forYouScoredTweetsCandidatePipelineConfig.identifier,\n        newTweetsPillCandidatePipelineConfig.identifier\n      ),\n      stringCenter = stringCenterProvider.get(),\n      seeNewTweetsString = externalStrings.seeNewTweetsString,\n      tweetedString = externalStrings.tweetedString\n    ),\n    InsertAppendResults(candidatePipeline = editedTweetsCandidatePipelineConfig.identifier),\n    SelectConditionally(\n      selector =\n        InsertAppendResults(candidatePipeline = newTweetsPillCandidatePipelineConfig.identifier),\n      includeSelector = (_, _, results) => CandidatesUtil.containsType[TweetCandidate](results)\n    ),\n    UpdateHomeClientEventDetails(\n      candidatePipelines = Set(\n        forYouConversationServiceCandidatePipelineConfig.identifier,\n        forYouScoredTweetsCandidatePipelineConfig.identifier\n      )\n    ),\n  )\n\n  private val servedCandidateKeysKafkaSideEffect =\n    servedCandidateKeysKafkaSideEffectBuilder.build(\n      Set(forYouScoredTweetsCandidatePipelineConfig.identifier))\n\n  private val servedCandidateFeatureKeysKafkaSideEffect =\n    servedCandidateFeatureKeysKafkaSideEffectBuilder.build(\n      Set(forYouScoredTweetsCandidatePipelineConfig.identifier))\n\n  private val homeScribeClientEventSideEffect = HomeScribeClientEventSideEffect(\n    enableScribeClientEvents = enableScribeClientEvents,\n    logPipelinePublisher = clientEventsScribeEventPublisher,\n    injectedTweetsCandidatePipelineIdentifiers = Seq(\n      forYouScoredTweetsCandidatePipelineConfig.identifier,\n      forYouConversationServiceCandidatePipelineConfig.identifier\n    ),\n    adsCandidatePipelineIdentifier = Some(forYouAdsCandidatePipelineConfig.identifier),\n    whoToFollowCandidatePipelineIdentifier = Some(\n      forYouWhoToFollowCandidatePipelineConfig.identifier\n    ),\n    whoToSubscribeCandidatePipelineIdentifier =\n      Some(forYouWhoToSubscribeCandidatePipelineConfig.identifier)\n  )\n\n  override val resultSideEffects: Seq[PipelineResultSideEffect[ForYouQuery, Timeline]] = Seq(\n    servedCandidateKeysKafkaSideEffect,\n    servedCandidateFeatureKeysKafkaSideEffect,\n    updateTimelinesPersistenceStoreSideEffect,\n    truncateTimelinesPersistenceStoreSideEffect,\n    homeScribeClientEventSideEffect,\n    homeScribeServedCandidatesSideEffect,\n    servedStatsSideEffect\n  )\n\n  override val domainMarshaller: DomainMarshaller[ForYouQuery, Timeline] = {\n    val instructionBuilders = Seq(\n      ClearCacheInstructionBuilder(\n        ClearCacheIncludeInstruction(\n          ClearCacheOnPtr.EnableParam,\n          ClearCacheOnPtr.MinEntriesParam,\n        )\n      ),\n      ReplaceEntryInstructionBuilder(ReplaceAllEntries),\n      // excludes alert, cover, and replace candidates\n      AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder(),\n      ShowAlertInstructionBuilder(),\n      ShowCoverInstructionBuilder(),\n    )\n\n    val idSelector: PartialFunction[UniversalNoun[_], Long] = {\n      // exclude ads while determining tweet cursor values\n      case item: TweetItem if item.promotedMetadata.isEmpty => item.id\n      case module: TimelineModule\n          if module.items.headOption.exists(_.item.isInstanceOf[TweetItem]) =>\n        module.items.last.item match { case item: TweetItem => item.id }\n    }\n    val topCursorBuilder = OrderedTopCursorBuilder(idSelector)\n    val bottomCursorBuilder = OrderedBottomCursorBuilder(idSelector)\n\n    val metadataBuilder = UrtMetadataBuilder(\n      title = None,\n      scribeConfigBuilder = Some(\n        StaticTimelineScribeConfigBuilder(\n          TimelineScribeConfig(\n            page = Some(\"for_you_scored_tweets\"),\n            section = None,\n            entityToken = None)))\n    )\n\n    UrtDomainMarshaller(\n      instructionBuilders = instructionBuilders,\n      metadataBuilder = Some(metadataBuilder),\n      cursorBuilders = Seq(topCursorBuilder, bottomCursorBuilder)\n    )\n  }\n\n  override val transportMarshaller: TransportMarshaller[Timeline, urt.TimelineResponse] =\n    urtTransportMarshaller\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouScoredTweetsResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.model.GrokTopics.GrokCategoryIdToNameMap\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.model.PhoenixPredictedBookmarkScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedDwellScoreFeature\nimport com.twitter.home_mixer.model.candidate_source.SourceSignal\nimport com.twitter.home_mixer.model.PhoenixPredictedFavoriteScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedGoodClickConvoDescUamGt2ScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedGoodProfileClickScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedNegativeFeedbackV2ScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedOpenLinkScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedReplyScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedRetweetScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedScreenshotScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedShareScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedVideoQualityViewScoreFeature\nimport com.twitter.home_mixer.model.PredictedDwellScoreFeature\nimport com.twitter.home_mixer.model.PredictedFavoriteScoreFeature\nimport com.twitter.home_mixer.model.PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature\nimport com.twitter.home_mixer.model.PredictedGoodClickConvoDescUamGt2ScoreFeature\nimport com.twitter.home_mixer.model.PredictedGoodProfileClickScoreFeature\nimport com.twitter.home_mixer.model.PredictedNegativeFeedbackV2ScoreFeature\nimport com.twitter.home_mixer.model.PredictedReplyEngagedByAuthorScoreFeature\nimport com.twitter.home_mixer.model.PredictedReplyScoreFeature\nimport com.twitter.home_mixer.model.PredictedRetweetScoreFeature\nimport com.twitter.home_mixer.model.PredictedShareScoreFeature\nimport com.twitter.home_mixer.model.PredictedVideoQualityViewScoreFeature\nimport com.twitter.home_mixer.product.for_you.candidate_source.ScoredTweetWithConversationMetadata\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityIdFeature\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityNameFeature\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationIdFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_is_nsfw.IsNsfw\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_visibility_reason.VisibilityReason\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BasicTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecWithEducationTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType\nimport com.twitter.timelines.render.{thriftscala => tl}\n\nobject ForYouScoredTweetsResponseFeatureTransformer\n    extends CandidateFeatureTransformer[ScoredTweetWithConversationMetadata] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ForYouScoredTweetsResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    AncestorsFeature,\n    AuthorIdFeature,\n    AuthorIsBlueVerifiedFeature,\n    AuthorIsCreatorFeature,\n    AuthorIsGoldVerifiedFeature,\n    AuthorIsGrayVerifiedFeature,\n    AuthorIsLegacyVerifiedFeature,\n    AuthorFollowersFeature,\n    ConversationModuleFocalTweetIdFeature,\n    ConversationModuleIdFeature,\n    DirectedAtUserIdFeature,\n    DebugStringFeature,\n    SourceSignalFeature,\n    ExclusiveConversationAuthorIdFeature,\n    FullScoringSucceededFeature,\n    FavoritedByUserIdsFeature,\n    FollowedByUserIdsFeature,\n    InNetworkFeature,\n    InReplyToTweetIdFeature,\n    InReplyToUserIdFeature,\n    IsAncestorCandidateFeature,\n    IsNsfw,\n    IsReadFromCacheFeature,\n    IsRetweetFeature,\n    CommunityIdFeature,\n    CommunityNameFeature,\n    ListIdFeature,\n    ListNameFeature,\n    LocationIdFeature,\n    PredictionRequestIdFeature,\n    QuotedTweetIdFeature,\n    QuotedUserIdFeature,\n    SGSValidFollowedByUserIdsFeature,\n    SGSValidLikedByUserIdsFeature,\n    ValidLikedByUserIdsFeature,\n    ScoreFeature,\n    ServedTypeFeature,\n    SourceTweetIdFeature,\n    SourceUserIdFeature,\n    TopicContextFunctionalityTypeFeature,\n    TopicIdSocialContextFeature,\n    TweetLanguageFeature,\n    TweetTextFeature,\n    TweetTypeMetricsFeature,\n    UserActionsSizeFeature,\n    UserActionsContainsExplicitSignalsFeature,\n    VisibilityReason,\n    ViralContentCreatorFeature,\n    GrokContentCreatorFeature,\n    GorkContentCreatorFeature,\n    HasVideoFeature,\n    VideoDurationMsFeature,\n    TweetMediaIdsFeature,\n    GrokAnnotationsFeature,\n    GrokTopCategoryFeature,\n    GrokIsGoreFeature,\n    GrokIsNsfwFeature,\n    GrokIsSpamFeature,\n    GrokIsViolentFeature,\n    GrokIsLowQualityFeature,\n    GrokIsOcrFeature,\n    PredictedDwellScoreFeature,\n    PredictedFavoriteScoreFeature,\n    PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature,\n    PredictedGoodClickConvoDescUamGt2ScoreFeature,\n    PredictedGoodProfileClickScoreFeature,\n    PredictedNegativeFeedbackV2ScoreFeature,\n    PredictedReplyEngagedByAuthorScoreFeature,\n    PredictedReplyScoreFeature,\n    PredictedRetweetScoreFeature,\n    PredictedShareScoreFeature,\n    PredictedVideoQualityViewScoreFeature,\n    PhoenixPredictedDwellScoreFeature,\n    PhoenixPredictedFavoriteScoreFeature,\n    PhoenixPredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature,\n    PhoenixPredictedGoodClickConvoDescUamGt2ScoreFeature,\n    PhoenixPredictedGoodProfileClickScoreFeature,\n    PhoenixPredictedNegativeFeedbackV2ScoreFeature,\n    PhoenixPredictedReplyScoreFeature,\n    PhoenixPredictedRetweetScoreFeature,\n    PhoenixPredictedShareScoreFeature,\n    PhoenixPredictedVideoQualityViewScoreFeature,\n    PhoenixPredictedOpenLinkScoreFeature,\n    PhoenixPredictedScreenshotScoreFeature,\n    PhoenixPredictedBookmarkScoreFeature\n  )\n\n  override def transform(input: ScoredTweetWithConversationMetadata): FeatureMap =\n    FeatureMapBuilder()\n      .add(AncestorsFeature, input.ancestors.getOrElse(Seq.empty))\n      .add(AuthorIdFeature, Some(input.authorId))\n      .add(AuthorIsBlueVerifiedFeature, input.authorIsBlueVerified.getOrElse(false))\n      .add(AuthorIsGoldVerifiedFeature, input.authorIsGoldVerified.getOrElse(false))\n      .add(AuthorIsGrayVerifiedFeature, input.authorIsGrayVerified.getOrElse(false))\n      .add(AuthorIsLegacyVerifiedFeature, input.authorIsLegacyVerified.getOrElse(false))\n      .add(AuthorIsCreatorFeature, input.authorIsCreator.getOrElse(false))\n      .add(AuthorFollowersFeature, input.authorFollowers)\n      .add(CommunityIdFeature, input.communityId)\n      .add(CommunityNameFeature, input.communityName)\n      .add(ConversationModuleIdFeature, input.conversationId)\n      .add(ConversationModuleFocalTweetIdFeature, input.conversationFocalTweetId)\n      .add(DirectedAtUserIdFeature, input.directedAtUserId)\n      .add(DebugStringFeature, input.debugString)\n      .add(\n        SourceSignalFeature,\n        input.sourceSignal.map { ss =>\n          SourceSignal(ss.id, ss.signalType, ss.signalEntity, ss.authorId)\n        }\n      )\n      .add(ExclusiveConversationAuthorIdFeature, input.exclusiveConversationAuthorId)\n      .add(SGSValidLikedByUserIdsFeature, input.sgsValidLikedByUserIds.getOrElse(Seq.empty))\n      .add(SGSValidFollowedByUserIdsFeature, input.sgsValidFollowedByUserIds.getOrElse(Seq.empty))\n      .add(ValidLikedByUserIdsFeature, input.validLikedByUserIds.getOrElse(Seq.empty))\n      .add(FavoritedByUserIdsFeature, input.sgsValidLikedByUserIds.getOrElse(Seq.empty))\n      .add(FollowedByUserIdsFeature, input.sgsValidFollowedByUserIds.getOrElse(Seq.empty))\n      .add(FullScoringSucceededFeature, true)\n      .add(InNetworkFeature, input.inNetwork.getOrElse(true))\n      .add(InReplyToTweetIdFeature, input.inReplyToTweetId)\n      .add(InReplyToUserIdFeature, input.inReplyToUserId)\n      .add(IsAncestorCandidateFeature, input.conversationFocalTweetId.exists(_ != input.tweetId))\n      .add(IsReadFromCacheFeature, input.isReadFromCache.getOrElse(false))\n      .add(IsRetweetFeature, input.sourceTweetId.isDefined)\n      .add(IsNsfw, input.isNsfw)\n      .add(ListIdFeature, input.listId)\n      .add(ListNameFeature, input.listName)\n      .add(LocationIdFeature, input.locationId)\n      .add(PredictionRequestIdFeature, input.predictionRequestId)\n      .add(QuotedTweetIdFeature, input.quotedTweetId)\n      .add(QuotedUserIdFeature, input.quotedUserId)\n      .add(ScoreFeature, input.score)\n      .add(SourceTweetIdFeature, input.sourceTweetId)\n      .add(SourceUserIdFeature, input.sourceUserId)\n      .add(ServedTypeFeature, input.servedType)\n      .add(\n        TopicContextFunctionalityTypeFeature,\n        input.topicFunctionalityType.collect {\n          case tl.TopicContextFunctionalityType.Basic => BasicTopicContextFunctionalityType\n          case tl.TopicContextFunctionalityType.Recommendation =>\n            RecommendationTopicContextFunctionalityType\n          case tl.TopicContextFunctionalityType.RecWithEducation =>\n            RecWithEducationTopicContextFunctionalityType\n        }\n      )\n      .add(TopicIdSocialContextFeature, input.topicId)\n      .add(TweetLanguageFeature, input.tweetLanguage)\n      .add(TweetTextFeature, input.tweetText)\n      .add(TweetTypeMetricsFeature, input.tweetTypeMetrics)\n      .add(UserActionsSizeFeature, input.userActionsSize)\n      .add(\n        UserActionsContainsExplicitSignalsFeature,\n        input.userActionsContainsExplicitSignals.getOrElse(false))\n      .add(VisibilityReason, input.visibilityReason)\n      .add(ViralContentCreatorFeature, input.viralContentCreatorFeature.getOrElse(false))\n      .add(GrokContentCreatorFeature, input.grokContentCreatorFeature.getOrElse(false))\n      .add(GorkContentCreatorFeature, input.gorkContentCreatorFeature.getOrElse(false))\n      .add(HasVideoFeature, input.hasVideoFeature.getOrElse(false))\n      .add(VideoDurationMsFeature, input.videoDurationMsFeature)\n      .add(TweetMediaIdsFeature, input.mediaIds.getOrElse(Seq.empty))\n      .add(GrokAnnotationsFeature, input.grokAnnotations)\n      .add(GrokIsGoreFeature, input.grokAnnotations.flatMap(_.metadata.map(_.isGore)))\n      .add(GrokIsNsfwFeature, input.grokAnnotations.flatMap(_.metadata.map(_.isNsfw)))\n      .add(GrokIsSpamFeature, input.grokAnnotations.flatMap(_.metadata.map(_.isSpam)))\n      .add(GrokIsViolentFeature, input.grokAnnotations.flatMap(_.metadata.map(_.isViolent)))\n      .add(GrokIsLowQualityFeature, input.grokAnnotations.flatMap(_.metadata.map(_.isLowQuality)))\n      .add(GrokIsOcrFeature, input.grokAnnotations.flatMap(_.metadata.map(_.isOcr)))\n      .add(\n        GrokTopCategoryFeature,\n        input.grokAnnotations.flatMap { annotations =>\n          annotations.categoryScores.flatMap { scores =>\n            val validCategories = scores.collect {\n              case (category, score)\n                  if category.forall(_.isDigit) &&\n                    GrokCategoryIdToNameMap.contains(category.toLong) =>\n                (category.toLong, score)\n            }\n            if (validCategories.nonEmpty) Some(validCategories.maxBy(_._2)._1) else None\n          }\n        }\n      )\n      .add(PredictedDwellScoreFeature, input.predictedScores.flatMap(_.dwellScore))\n      .add(PredictedFavoriteScoreFeature, input.predictedScores.flatMap(_.favoriteScore))\n      .add(\n        PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature,\n        input.predictedScores.flatMap(_.goodClickConvoDescFavoritedOrRepliedScore))\n      .add(\n        PredictedGoodClickConvoDescUamGt2ScoreFeature,\n        input.predictedScores.flatMap(_.goodClickConvoDescUamGt2Score))\n      .add(\n        PredictedGoodProfileClickScoreFeature,\n        input.predictedScores.flatMap(_.goodProfileClickScore))\n      .add(\n        PredictedNegativeFeedbackV2ScoreFeature,\n        input.predictedScores.flatMap(_.negativeFeedbackV2Score))\n      .add(\n        PredictedReplyEngagedByAuthorScoreFeature,\n        input.predictedScores.flatMap(_.replyEngagedByAuthorScore))\n      .add(PredictedReplyScoreFeature, input.predictedScores.flatMap(_.replyScore))\n      .add(PredictedRetweetScoreFeature, input.predictedScores.flatMap(_.retweetScore))\n      .add(PredictedShareScoreFeature, input.predictedScores.flatMap(_.shareScore))\n      .add(\n        PredictedVideoQualityViewScoreFeature,\n        input.predictedScores.flatMap(_.videoQualityViewScore))\n      .add(PhoenixPredictedDwellScoreFeature, input.phoenixPredictedScores.flatMap(_.dwellScore))\n      .add(\n        PhoenixPredictedFavoriteScoreFeature,\n        input.phoenixPredictedScores.flatMap(_.favoriteScore))\n      .add(\n        PhoenixPredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature,\n        input.phoenixPredictedScores.flatMap(_.goodClickConvoDescFavoritedOrRepliedScore))\n      .add(\n        PhoenixPredictedGoodClickConvoDescUamGt2ScoreFeature,\n        input.phoenixPredictedScores.flatMap(_.goodClickConvoDescUamGt2Score))\n      .add(\n        PhoenixPredictedGoodProfileClickScoreFeature,\n        input.phoenixPredictedScores.flatMap(_.goodProfileClickScore))\n      .add(\n        PhoenixPredictedNegativeFeedbackV2ScoreFeature,\n        input.phoenixPredictedScores.flatMap(_.negativeFeedbackV2Score))\n      .add(PhoenixPredictedReplyScoreFeature, input.phoenixPredictedScores.flatMap(_.replyScore))\n      .add(\n        PhoenixPredictedRetweetScoreFeature,\n        input.phoenixPredictedScores.flatMap(_.retweetScore))\n      .add(PhoenixPredictedShareScoreFeature, input.phoenixPredictedScores.flatMap(_.shareScore))\n      .add(\n        PhoenixPredictedVideoQualityViewScoreFeature,\n        input.phoenixPredictedScores.flatMap(_.videoQualityViewScore))\n      .add(\n        PhoenixPredictedOpenLinkScoreFeature,\n        input.phoenixPredictedScores.flatMap(_.openLinkScore))\n      .add(\n        PhoenixPredictedScreenshotScoreFeature,\n        input.phoenixPredictedScores.flatMap(_.screenshotScore))\n      .add(\n        PhoenixPredictedBookmarkScoreFeature,\n        input.phoenixPredictedScores.flatMap(_.bookmarkScore))\n      .build()\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouScoredVideoTweetsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.VideoCarouselModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.ConsistentAspectRatioFilter\nimport com.twitter.home_mixer.functional_component.filter.TweetHydrationFilter\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.product.for_you.candidate_source.ScoredVideoTweetCandidate\nimport com.twitter.home_mixer.product.for_you.candidate_source.ScoredVideoTweetsCategorizedProductCandidateSource\nimport com.twitter.home_mixer.product.for_you.candidate_source.ScoredVideoTweetsProductCandidateSource\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableScoredVideoTweetsCandidatePipelineParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.VideoCarouselAllowHorizontalVideos\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.VideoCarouselAllowVerticalVideos\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.VideoCarouselEnableFooterParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.VideoTweetsModuleMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.response_transformer.ScoredVideoTweetResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.gate.DefinedUserIdGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouScoredVideoTweetsCandidatePipelineConfig @Inject() (\n  scoredVideoTweetsProductCandidateSource: ScoredVideoTweetsProductCandidateSource,\n  scoredVideoTweetsCategorizedProductCandidateSource: ScoredVideoTweetsCategorizedProductCandidateSource,\n  tweetypieFeatureHydrator: TweetypieFeatureHydrator,\n  videoCarouselModuleCandidateDecorator: VideoCarouselModuleCandidateDecorator)\n    extends CandidatePipelineConfig[\n      ForYouQuery,\n      ForYouQuery,\n      ScoredVideoTweetCandidate,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouScoredVideoTweets\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] = Some(\n    EnableScoredVideoTweetsCandidatePipelineParam)\n\n  override val gates: Seq[Gate[PipelineQuery]] = Seq(\n    DefinedUserIdGate,\n    RateLimitGate,\n    TimelinesPersistenceStoreLastInjectionGate(\n      VideoTweetsModuleMinInjectionIntervalParam,\n      EntityIdType.VideoCarouselModule\n    ),\n  )\n\n  override val queryFeatureHydration: Seq[QueryFeatureHydrator[PipelineQuery]] = Seq()\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ForYouQuery,\n    ForYouQuery\n  ] = identity\n\n  override def candidateSource: CandidateSource[ForYouQuery, ScoredVideoTweetCandidate] =\n    scoredVideoTweetsCategorizedProductCandidateSource\n//    scoredVideoTweetsProductCandidateSource\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[ScoredVideoTweetCandidate]\n  ] = Seq(ScoredVideoTweetResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    ScoredVideoTweetCandidate,\n    TweetCandidate\n  ] = { sourceResult =>\n    TweetCandidate(id = sourceResult.tweetId)\n  }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ForYouQuery, TweetCandidate, _]\n  ] = Seq(tweetypieFeatureHydrator)\n\n  override val filters: Seq[Filter[ForYouQuery, TweetCandidate]] = Seq(\n    TweetHydrationFilter,\n    ConsistentAspectRatioFilter(\n      allowVerticalVideosParam = VideoCarouselAllowVerticalVideos,\n      allowHorizontalVideosParam = VideoCarouselAllowHorizontalVideos)\n  )\n\n  override val decorator: Option[\n    CandidateDecorator[PipelineQuery, TweetCandidate]\n  ] = Some(videoCarouselModuleCandidateDecorator.build(VideoCarouselEnableFooterParam))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouStoriesCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.StoriesModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.product.for_you.candidate_source.StoriesModuleCandidateSource\nimport com.twitter.home_mixer.product.for_you.candidate_source.StoryCandidate\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableTrendsParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.TrendsModuleMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.response_transformer.StoriesModuleResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.gate.DefinedUserIdGate\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.UnifiedTrendCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouStoriesCandidatePipelineConfig @Inject() (\n  storiesModule: StoriesModuleCandidateSource,\n  storiesModuleCandidateDecorator: StoriesModuleCandidateDecorator)\n    extends CandidatePipelineConfig[\n      PipelineQuery,\n      Long,\n      StoryCandidate,\n      UnifiedTrendCandidate\n    ] {\n\n  override val supportedClientParam: Option[FSParam[Boolean]] = Some(EnableTrendsParam)\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouStories\")\n\n  override val gates = Seq(\n    RateLimitGate,\n    DefinedUserIdGate,\n    TimelinesPersistenceStoreLastInjectionGate(\n      TrendsModuleMinInjectionIntervalParam,\n      EntityIdType.Trends\n    )\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    PipelineQuery,\n    Long\n  ] = { query => query.getRequiredUserId }\n\n  override def candidateSource: BaseCandidateSource[Long, StoryCandidate] = storiesModule\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[StoryCandidate]\n  ] = Seq(StoriesModuleResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    StoryCandidate,\n    UnifiedTrendCandidate\n  ] = { story => UnifiedTrendCandidate(id = story.id.toString) }\n\n  override val decorator: Option[CandidateDecorator[PipelineQuery, UnifiedTrendCandidate]] =\n    Some(storiesModuleCandidateDecorator.moduleDecorator)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouTimelineScorerCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeClientEventInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeConversationModuleMetadataBuilder\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeTimelinesScoreInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeTweetSocialContextBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator.InNetworkFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.NamesFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.PerspectiveFilteredSocialContextFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.SGSValidSocialContextFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.FeedbackFatigueFilter\nimport com.twitter.home_mixer.functional_component.filter.InvalidConversationModuleFilter\nimport com.twitter.home_mixer.functional_component.filter.InvalidSubscriptionTweetFilter\nimport com.twitter.home_mixer.functional_component.filter.RejectTweetFromViewerFilter\nimport com.twitter.home_mixer.functional_component.filter.RetweetDeduplicationFilter\nimport com.twitter.home_mixer.functional_component.scorer.FeedbackFatigueScorer\nimport com.twitter.home_mixer.functional_component.scorer.OONTweetScalingScorer\nimport com.twitter.home_mixer.marshaller.timelines.DeviceContextMarshaller\nimport com.twitter.home_mixer.marshaller.timelines.TimelineServiceCursorMarshaller\nimport com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsNsfwFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedTweetDroppedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TimelineServiceTweetsFeature\nimport com.twitter.home_mixer.model.request.DeviceContext\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.FocalTweetFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.filter.SocialContextFilter\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableTimelineScorerCandidatePipelineParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.component_library.candidate_source.timeline_scorer.ScoredTweetCandidateWithFocalTweet\nimport com.twitter.product_mixer.component_library.candidate_source.timeline_scorer.TimelineScorerCandidateSource\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ManualModuleId\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder\nimport com.twitter.product_mixer.component_library.filter.FeatureFilter\nimport com.twitter.product_mixer.component_library.filter.PredicateFeatureFilter\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.model.candidate.CandidateTweetSourceId\nimport com.twitter.timelines.service.{thriftscala => tst}\nimport com.twitter.timelinescorer.{thriftscala => t}\nimport com.twitter.timelineservice.{thriftscala => tlst}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Candidate Pipeline Config that fetches tweets from the Timeline Scorer Candidate Source\n */\n@Singleton\nclass ForYouTimelineScorerCandidatePipelineConfig @Inject() (\n  timelineScorerCandidateSource: TimelineScorerCandidateSource,\n  deviceContextMarshaller: DeviceContextMarshaller,\n  tweetypieFeatureHydrator: TweetypieFeatureHydrator,\n  sgsValidSocialContextFeatureHydrator: SGSValidSocialContextFeatureHydrator,\n  perspectiveFilteredSocialContextFeatureHydrator: PerspectiveFilteredSocialContextFeatureHydrator,\n  namesFeatureHydrator: NamesFeatureHydrator,\n  focalTweetFeatureHydrator: FocalTweetFeatureHydrator,\n  invalidSubscriptionTweetFilter: InvalidSubscriptionTweetFilter,\n  homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder,\n  homeTweetSocialContextBuilder: HomeTweetSocialContextBuilder)\n    extends CandidatePipelineConfig[\n      ForYouQuery,\n      t.ScoredTweetsRequest,\n      ScoredTweetCandidateWithFocalTweet,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouTimelineScorerTweets\")\n\n  private val TweetypieHydratedFilterId = \"TweetypieHydrated\"\n  private val QuotedTweetDroppedFilterId = \"QuotedTweetDropped\"\n  private val OutOfNetworkNSFWFilterId = \"OutOfNetworkNSFW\"\n  private val ConversationModuleNamespace = EntryNamespace(\"home-conversation\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] =\n    Some(EnableTimelineScorerCandidatePipelineParam)\n\n  override val candidateSource: BaseCandidateSource[\n    t.ScoredTweetsRequest,\n    ScoredTweetCandidateWithFocalTweet\n  ] = timelineScorerCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ForYouQuery,\n    t.ScoredTweetsRequest\n  ] = { query =>\n    val deviceContext = query.deviceContext.getOrElse(DeviceContext.Empty)\n\n    val scoredTweetsRequestContext = t.v1.ScoredTweetsRequestContext(\n      contextualUserId = query.clientContext.userId,\n      timelineId = query.clientContext.userId.map(tlst.TimelineId(tlst.TimelineType.Home, _, None)),\n      deviceContext = Some(deviceContextMarshaller(deviceContext, query.clientContext)),\n      seenTweetIds = query.seenTweetIds,\n      contextualUserContext = Some(tst.ContextualUserContext(query.clientContext.userRoles)),\n      timelineRequestCursor = query.pipelineCursor.flatMap(TimelineServiceCursorMarshaller(_))\n    )\n\n    val candidateTweetSourceIds = Seq(\n      CandidateTweetSourceId.RecycledTweet,\n      CandidateTweetSourceId.OrganicTweet,\n      CandidateTweetSourceId.AncestorsOnlyOrganicTweet,\n      CandidateTweetSourceId.BackfillOrganicTweet,\n      CandidateTweetSourceId.CroonTweet,\n      CandidateTweetSourceId.RecommendedTweet,\n      CandidateTweetSourceId.FrsTweet,\n      CandidateTweetSourceId.ListTweet,\n      CandidateTweetSourceId.RecommendedTrendTweet,\n      CandidateTweetSourceId.PopularTopicTweet\n    )\n\n    val timelineServiceTweets =\n      query.features.map(_.getOrElse(TimelineServiceTweetsFeature, Seq.empty)).getOrElse(Seq.empty)\n\n    val timelineEntries = timelineServiceTweets.map { id =>\n      tlst.TimelineEntry.Tweet(tlst.Tweet(statusId = id, sortIndex = id))\n    }\n\n    t.ScoredTweetsRequest.V1(\n      t.v1.ScoredTweetsRequest(\n        scoredTweetsRequestContext = Some(scoredTweetsRequestContext),\n        candidateTweetSourceIds =\n          Some(candidateTweetSourceIds.flatMap(CandidateTweetSourceId.toThrift)),\n        maxResultsCount = query.requestedMaxResults,\n        organicTimeline = Some(\n          tlst.Timeline(\n            timelineId = tlst.TimelineId(\n              timelineType = tlst.TimelineType.Home,\n              id = query.getRequiredUserId,\n              canonicalTimelineId = None),\n            entries = timelineEntries,\n            modules = tlst.TimelineModules()\n          ))\n      )\n    )\n  }\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[ScoredTweetCandidateWithFocalTweet]\n  ] = Seq(ForYouTimelineScorerResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    ScoredTweetCandidateWithFocalTweet,\n    TweetCandidate\n  ] = { candidateWithFocalTweetId =>\n    TweetCandidate(id = candidateWithFocalTweetId.candidate.tweetId)\n  }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ForYouQuery, TweetCandidate, _]\n  ] = Seq(\n    namesFeatureHydrator,\n    tweetypieFeatureHydrator,\n    InNetworkFeatureHydrator,\n    sgsValidSocialContextFeatureHydrator,\n    perspectiveFilteredSocialContextFeatureHydrator,\n  )\n\n  override def filters: Seq[Filter[ForYouQuery, TweetCandidate]] = Seq(\n    RetweetDeduplicationFilter,\n    FeatureFilter.fromFeature(FilterIdentifier(TweetypieHydratedFilterId), IsHydratedFeature),\n    PredicateFeatureFilter.fromPredicate(\n      FilterIdentifier(QuotedTweetDroppedFilterId),\n      shouldKeepCandidate = { features => !features.getOrElse(QuotedTweetDroppedFeature, false) }\n    ),\n    PredicateFeatureFilter.fromPredicate(\n      FilterIdentifier(OutOfNetworkNSFWFilterId),\n      shouldKeepCandidate = { features =>\n        features.getOrElse(InNetworkFeature, false) ||\n        !features.getOrElse(IsNsfwFeature, false)\n      }\n    ),\n    FeedbackFatigueFilter,\n    RejectTweetFromViewerFilter,\n    SocialContextFilter,\n    invalidSubscriptionTweetFilter,\n    InvalidConversationModuleFilter\n  )\n\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[ForYouQuery, TweetCandidate, _]\n  ] = Seq(focalTweetFeatureHydrator)\n\n  override val scorers: Seq[Scorer[ForYouQuery, TweetCandidate]] =\n    Seq(OONTweetScalingScorer, FeedbackFatigueScorer)\n\n  override val decorator: Option[CandidateDecorator[ForYouQuery, TweetCandidate]] = {\n    val clientEventInfoBuilder = HomeClientEventInfoBuilder()\n\n    val tweetItemBuilder = TweetCandidateUrtItemBuilder(\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      socialContextBuilder = Some(homeTweetSocialContextBuilder),\n      timelinesScoreInfoBuilder = Some(HomeTimelinesScoreInfoBuilder),\n      feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder)\n    )\n\n    val tweetDecorator = UrtItemCandidateDecorator(tweetItemBuilder)\n\n    val moduleBuilder = TimelineModuleBuilder(\n      entryNamespace = ConversationModuleNamespace,\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      moduleIdGeneration = ManualModuleId(0L),\n      displayTypeBuilder = StaticModuleDisplayTypeBuilder(VerticalConversation),\n      metadataBuilder = Some(HomeConversationModuleMetadataBuilder())\n    )\n\n    Some(\n      UrtMultipleModulesDecorator(\n        urtItemCandidateDecorator = tweetDecorator,\n        moduleBuilder = moduleBuilder,\n        groupByKey = (_, _, candidateFeatures) =>\n          candidateFeatures.getOrElse(ConversationModuleFocalTweetIdFeature, None)\n      ))\n  }\n\n  override val alerts: Seq[Alert] = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(),\n    HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert(10, 20)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouTimelineScorerMixerPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.clientapp.{thriftscala => ca}\nimport com.twitter.goldfinch.api.AdsInjectionSurfaceAreas\nimport com.twitter.home_mixer.candidate_pipeline.EditedTweetsCandidatePipelineConfig\nimport com.twitter.home_mixer.candidate_pipeline.NewTweetsPillCandidatePipelineConfig\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator.FeedbackHistoryQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator._\nimport com.twitter.home_mixer.functional_component.selector.DebunchCandidates\nimport com.twitter.home_mixer.functional_component.selector.UpdateConversationModuleId\nimport com.twitter.home_mixer.functional_component.selector.UpdateHomeClientEventDetails\nimport com.twitter.home_mixer.functional_component.selector.UpdateNewTweetsPillDecoration\nimport com.twitter.home_mixer.functional_component.side_effect._\nimport com.twitter.home_mixer.model.ClearCacheIncludeInstruction\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableImpressionBloomFilter\nimport com.twitter.home_mixer.param.HomeGlobalParams.MaxNumberReplaceInstructionsParam\nimport com.twitter.home_mixer.param.HomeMixerFlagName.ScribeClientEventsFlag\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.TimelineServiceTweetsQueryFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ClearCacheOnPtr\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableFlipInjectionModuleCandidatePipelineParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.FlipInlineInjectionModulePosition\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.TweetPreviewsPositionParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.WhoToFollowPositionParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.WhoToSubscribePositionParam\nimport com.twitter.home_mixer.product.for_you.side_effect.ServedCandidateFeatureKeysKafkaSideEffectBuilder\nimport com.twitter.home_mixer.product.for_you.side_effect.ServedCandidateKeysKafkaSideEffectBuilder\nimport com.twitter.home_mixer.product.for_you.side_effect.ServedStatsSideEffect\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.async.AsyncQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.impressed_tweets.ImpressedTweetsQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated.AsyncParamGatedQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.PreviewCreatorsQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.FlipPromptCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowArmCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_subscribe_module.WhoToSubscribeCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.premarshaller.urt.UrtDomainMarshaller\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ClearCacheInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedBottomCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedTopCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceAllEntries\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceEntryInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ShowAlertInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ShowCoverInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.StaticTimelineScribeConfigBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtMetadataBuilder\nimport com.twitter.product_mixer.component_library.selector.DropMaxCandidates\nimport com.twitter.product_mixer.component_library.selector.DropMaxModuleItemCandidates\nimport com.twitter.product_mixer.component_library.selector.DropModuleTooFewModuleItemResults\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.component_library.selector.InsertFixedPositionResults\nimport com.twitter.product_mixer.component_library.selector.SelectConditionally\nimport com.twitter.product_mixer.component_library.selector.UpdateSortCandidates\nimport com.twitter.product_mixer.component_library.selector.UpdateSortModuleItemCandidates\nimport com.twitter.product_mixer.component_library.selector.ads.AdsInjector\nimport com.twitter.product_mixer.component_library.selector.ads.InsertAdResults\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.UrtTransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.MixerPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineConfig\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouTimelineScorerMixerPipelineConfig @Inject() (\n  forYouTimelineScorerCandidatePipelineConfig: ForYouTimelineScorerCandidatePipelineConfig,\n  forYouPushToHomeTweetCandidatePipelineConfig: ForYouPushToHomeTweetCandidatePipelineConfig,\n  forYouConversationServiceCandidatePipelineConfig: ForYouConversationServiceCandidatePipelineConfig,\n  forYouAdsDependentCandidatePipelineBuilder: ForYouAdsDependentCandidatePipelineBuilder,\n  forYouWhoToFollowCandidatePipelineConfigBuilder: ForYouWhoToFollowCandidatePipelineConfigBuilder,\n  forYouWhoToSubscribeCandidatePipelineConfigBuilder: ForYouWhoToSubscribeCandidatePipelineConfigBuilder,\n  flipPromptCandidatePipelineConfigBuilder: FlipPromptCandidatePipelineConfigBuilder,\n  editedTweetsCandidatePipelineConfig: EditedTweetsCandidatePipelineConfig,\n  newTweetsPillCandidatePipelineConfig: NewTweetsPillCandidatePipelineConfig[ForYouQuery],\n  forYouTweetPreviewsCandidatePipelineConfig: ForYouTweetPreviewsCandidatePipelineConfig,\n  dismissInfoQueryFeatureHydrator: DismissInfoQueryFeatureHydrator,\n  gizmoduckUserQueryFeatureHydrator: GizmoduckUserQueryFeatureHydrator,\n  impressionBloomFilterQueryFeatureHydrator: ImpressionBloomFilterQueryFeatureHydrator[ForYouQuery],\n  manhattanTweetImpressionsQueryFeatureHydrator: TweetImpressionsQueryFeatureHydrator[ForYouQuery],\n  memcacheTweetImpressionsQueryFeatureHydrator: ImpressedTweetsQueryFeatureHydrator,\n  persistenceStoreQueryFeatureHydrator: PersistenceStoreQueryFeatureHydrator,\n  requestQueryFeatureHydrator: RequestQueryFeatureHydrator[ForYouQuery],\n  timelineServiceTweetsQueryFeatureHydrator: TimelineServiceTweetsQueryFeatureHydrator,\n  lastNonPollingTimeQueryFeatureHydrator: LastNonPollingTimeQueryFeatureHydrator,\n  feedbackHistoryQueryFeatureHydrator: FeedbackHistoryQueryFeatureHydrator,\n  previewCreatorsQueryFeatureHydrator: PreviewCreatorsQueryFeatureHydrator,\n  sgsFollowedUsersQueryFeatureHydrator: SGSFollowedUsersQueryFeatureHydrator,\n  adsInjector: AdsInjector,\n  servedCandidateKeysKafkaSideEffectBuilder: ServedCandidateKeysKafkaSideEffectBuilder,\n  servedCandidateFeatureKeysKafkaSideEffectBuilder: ServedCandidateFeatureKeysKafkaSideEffectBuilder,\n  updateLastNonPollingTimeSideEffect: UpdateLastNonPollingTimeSideEffect[ForYouQuery, Timeline],\n  publishClientSentImpressionsEventBusSideEffect: PublishClientSentImpressionsEventBusSideEffect,\n  publishClientSentImpressionsManhattanSideEffect: PublishClientSentImpressionsManhattanSideEffect,\n  publishImpressionBloomFilterSideEffect: PublishImpressionBloomFilterSideEffect,\n  updateTimelinesPersistenceStoreSideEffect: UpdateTimelinesPersistenceStoreSideEffect,\n  truncateTimelinesPersistenceStoreSideEffect: TruncateTimelinesPersistenceStoreSideEffect,\n  homeScribeServedCandidatesSideEffect: HomeScribeServedCandidatesSideEffect,\n  servedStatsSideEffect: ServedStatsSideEffect,\n  clientEventsScribeEventPublisher: EventPublisher[ca.LogEvent],\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter],\n  urtTransportMarshaller: UrtTransportMarshaller,\n  @Flag(ScribeClientEventsFlag) enableScribeClientEvents: Boolean)\n    extends MixerPipelineConfig[ForYouQuery, Timeline, urt.TimelineResponse] {\n\n  override val identifier: MixerPipelineIdentifier = MixerPipelineIdentifier(\"ForYouTimelineScorer\")\n\n  private val MaxConsecutiveOutOfNetworkCandidates = 2\n\n  private val PushToHomeTweetPosition = 0\n\n  private val dependentCandidatesStep = MixerPipelineConfig.dependentCandidatePipelinesStep\n  private val resultSelectorsStep = MixerPipelineConfig.resultSelectorsStep\n\n  override def fetchQueryFeatures: Seq[QueryFeatureHydrator[ForYouQuery]] = Seq(\n    requestQueryFeatureHydrator,\n    persistenceStoreQueryFeatureHydrator,\n    timelineServiceTweetsQueryFeatureHydrator,\n    feedbackHistoryQueryFeatureHydrator,\n    previewCreatorsQueryFeatureHydrator,\n    sgsFollowedUsersQueryFeatureHydrator,\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, dismissInfoQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, gizmoduckUserQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, lastNonPollingTimeQueryFeatureHydrator),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableImpressionBloomFilter,\n      resultSelectorsStep,\n      impressionBloomFilterQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(resultSelectorsStep, manhattanTweetImpressionsQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(resultSelectorsStep, memcacheTweetImpressionsQueryFeatureHydrator)\n  )\n\n  private val timelineScorerCandidatePipelineScope =\n    SpecificPipeline(forYouTimelineScorerCandidatePipelineConfig.identifier)\n\n  private val forYouAdsCandidatePipelineConfig = forYouAdsDependentCandidatePipelineBuilder\n    .build(timelineScorerCandidatePipelineScope)\n\n  private val forYouWhoToFollowCandidatePipelineConfig =\n    forYouWhoToFollowCandidatePipelineConfigBuilder.build()\n\n  private val forYouWhoToSubscribeCandidatePipelineConfig =\n    forYouWhoToSubscribeCandidatePipelineConfigBuilder.build()\n\n  private val flipPromptCandidatePipelineConfig =\n    flipPromptCandidatePipelineConfigBuilder.build[ForYouQuery](\n      supportedClientParam = Some(EnableFlipInjectionModuleCandidatePipelineParam)\n    )\n\n  override val candidatePipelines: Seq[CandidatePipelineConfig[ForYouQuery, _, _, _]] = Seq(\n    forYouTimelineScorerCandidatePipelineConfig,\n    forYouPushToHomeTweetCandidatePipelineConfig,\n    forYouWhoToFollowCandidatePipelineConfig,\n    forYouWhoToSubscribeCandidatePipelineConfig,\n    forYouTweetPreviewsCandidatePipelineConfig,\n    flipPromptCandidatePipelineConfig\n  )\n\n  override val dependentCandidatePipelines: Seq[\n    DependentCandidatePipelineConfig[ForYouQuery, _, _, _]\n  ] = Seq(\n    forYouAdsCandidatePipelineConfig,\n    forYouConversationServiceCandidatePipelineConfig,\n    editedTweetsCandidatePipelineConfig,\n    newTweetsPillCandidatePipelineConfig\n  )\n\n  override val failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy] = Map(\n    forYouTimelineScorerCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouAdsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouWhoToFollowCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouWhoToSubscribeCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    forYouTweetPreviewsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    flipPromptCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    editedTweetsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    newTweetsPillCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n  )\n\n  override val resultSelectors: Seq[Selector[ForYouQuery]] = Seq(\n    UpdateSortCandidates(\n      ordering = CandidatesUtil.reverseChronTweetsOrdering,\n      candidatePipeline = forYouConversationServiceCandidatePipelineConfig.identifier\n    ),\n    UpdateSortCandidates(\n      ordering = CandidatesUtil.scoreOrdering,\n      candidatePipeline = forYouTimelineScorerCandidatePipelineConfig.identifier\n    ),\n    UpdateSortModuleItemCandidates(\n      candidatePipeline = forYouTimelineScorerCandidatePipelineConfig.identifier,\n      ordering = CandidatesUtil.conversationModuleTweetsOrdering\n    ),\n    DebunchCandidates(\n      pipelineScope = SpecificPipeline(forYouTimelineScorerCandidatePipelineConfig.identifier),\n      mustDebunch = {\n        case item: ItemCandidateWithDetails =>\n          !item.features.getOrElse(InNetworkFeature, false)\n        case module: ModuleCandidateWithDetails =>\n          !module.candidates.last.features.getOrElse(InNetworkFeature, false)\n      },\n      maxBunchSize = MaxConsecutiveOutOfNetworkCandidates\n    ),\n    UpdateConversationModuleId(\n      pipelineScope = SpecificPipeline(forYouTimelineScorerCandidatePipelineConfig.identifier)\n    ),\n    DropMaxCandidates(\n      candidatePipeline = forYouConversationServiceCandidatePipelineConfig.identifier,\n      maxSelectionsParam = ServerMaxResultsParam\n    ),\n    DropMaxCandidates(\n      candidatePipeline = forYouTimelineScorerCandidatePipelineConfig.identifier,\n      maxSelectionsParam = ServerMaxResultsParam\n    ),\n    DropMaxCandidates(\n      candidatePipeline = editedTweetsCandidatePipelineConfig.identifier,\n      maxSelectionsParam = MaxNumberReplaceInstructionsParam\n    ),\n    DropMaxModuleItemCandidates(\n      candidatePipeline = forYouWhoToFollowCandidatePipelineConfig.identifier,\n      maxModuleItemsParam = StaticParam(WhoToFollowArmCandidatePipelineConfig.MaxCandidatesSize)\n    ),\n    DropModuleTooFewModuleItemResults(\n      candidatePipeline = forYouWhoToSubscribeCandidatePipelineConfig.identifier,\n      minModuleItemsParam = StaticParam(WhoToSubscribeCandidatePipelineConfig.MinCandidatesSize)\n    ),\n    DropMaxModuleItemCandidates(\n      candidatePipeline = forYouWhoToSubscribeCandidatePipelineConfig.identifier,\n      maxModuleItemsParam = StaticParam(WhoToSubscribeCandidatePipelineConfig.MaxCandidatesSize)\n    ),\n    // Add Conversation Service tweets to results only if the scored pipeline doesn't return any\n    SelectConditionally(\n      selector = InsertAppendResults(\n        candidatePipeline = forYouConversationServiceCandidatePipelineConfig.identifier),\n      includeSelector = (_, candidates, _) =>\n        !candidates.exists(candidate =>\n          forYouTimelineScorerCandidatePipelineConfig.identifier == candidate.source)\n    ),\n    InsertAppendResults(candidatePipeline = forYouTimelineScorerCandidatePipelineConfig.identifier),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouTweetPreviewsCandidatePipelineConfig.identifier,\n      positionParam = TweetPreviewsPositionParam\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouWhoToFollowCandidatePipelineConfig.identifier,\n      positionParam = WhoToFollowPositionParam\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = forYouWhoToSubscribeCandidatePipelineConfig.identifier,\n      positionParam = WhoToSubscribePositionParam\n    ),\n    InsertFixedPositionResults(\n      candidatePipeline = flipPromptCandidatePipelineConfig.identifier,\n      positionParam = FlipInlineInjectionModulePosition\n    ),\n    // Insert Push To Home Tweet at top of Timeline\n    InsertFixedPositionResults(\n      candidatePipeline = forYouPushToHomeTweetCandidatePipelineConfig.identifier,\n      positionParam = StaticParam(PushToHomeTweetPosition)\n    ),\n    InsertAdResults(\n      surfaceAreaName = AdsInjectionSurfaceAreas.HomeTimeline,\n      adsInjector = adsInjector.forSurfaceArea(AdsInjectionSurfaceAreas.HomeTimeline),\n      adsCandidatePipeline = forYouAdsCandidatePipelineConfig.identifier\n    ),\n    // This selector must come after the tweets are inserted into the results\n    DropModuleTooFewModuleItemResults(\n      candidatePipeline = forYouWhoToFollowCandidatePipelineConfig.identifier,\n      minModuleItemsParam = StaticParam(WhoToFollowArmCandidatePipelineConfig.MinCandidatesSize)\n    ),\n    UpdateNewTweetsPillDecoration(\n      pipelineScope = SpecificPipelines(\n        forYouConversationServiceCandidatePipelineConfig.identifier,\n        forYouTimelineScorerCandidatePipelineConfig.identifier,\n        newTweetsPillCandidatePipelineConfig.identifier\n      ),\n      stringCenter = stringCenterProvider.get(),\n      seeNewTweetsString = externalStrings.seeNewTweetsString,\n      tweetedString = externalStrings.tweetedString\n    ),\n    InsertAppendResults(candidatePipeline = editedTweetsCandidatePipelineConfig.identifier),\n    SelectConditionally(\n      selector =\n        InsertAppendResults(candidatePipeline = newTweetsPillCandidatePipelineConfig.identifier),\n      includeSelector = (_, _, results) => CandidatesUtil.containsType[TweetCandidate](results)\n    ),\n    UpdateHomeClientEventDetails(\n      candidatePipelines = Set(\n        forYouConversationServiceCandidatePipelineConfig.identifier,\n        forYouTimelineScorerCandidatePipelineConfig.identifier\n      )\n    ),\n  )\n\n  private val servedCandidateKeysKafkaSideEffect =\n    servedCandidateKeysKafkaSideEffectBuilder.build(\n      Set(forYouTimelineScorerCandidatePipelineConfig.identifier))\n\n  private val servedCandidateFeatureKeysKafkaSideEffect =\n    servedCandidateFeatureKeysKafkaSideEffectBuilder.build(\n      Set(forYouTimelineScorerCandidatePipelineConfig.identifier))\n\n  private val homeScribeClientEventSideEffect = HomeScribeClientEventSideEffect(\n    enableScribeClientEvents = enableScribeClientEvents,\n    logPipelinePublisher = clientEventsScribeEventPublisher,\n    injectedTweetsCandidatePipelineIdentifiers = Seq(\n      forYouTimelineScorerCandidatePipelineConfig.identifier,\n      forYouConversationServiceCandidatePipelineConfig.identifier\n    ),\n    adsCandidatePipelineIdentifier = Some(forYouAdsCandidatePipelineConfig.identifier),\n    whoToFollowCandidatePipelineIdentifier =\n      Some(forYouWhoToFollowCandidatePipelineConfig.identifier),\n    whoToSubscribeCandidatePipelineIdentifier =\n      Some(forYouWhoToSubscribeCandidatePipelineConfig.identifier)\n  )\n\n  override val resultSideEffects: Seq[PipelineResultSideEffect[ForYouQuery, Timeline]] = Seq(\n    homeScribeClientEventSideEffect,\n    homeScribeServedCandidatesSideEffect,\n    publishClientSentImpressionsEventBusSideEffect,\n    publishClientSentImpressionsManhattanSideEffect,\n    publishImpressionBloomFilterSideEffect,\n    servedCandidateKeysKafkaSideEffect,\n    servedCandidateFeatureKeysKafkaSideEffect,\n    servedStatsSideEffect,\n    truncateTimelinesPersistenceStoreSideEffect,\n    updateLastNonPollingTimeSideEffect,\n    updateTimelinesPersistenceStoreSideEffect\n  )\n\n  override val domainMarshaller: DomainMarshaller[ForYouQuery, Timeline] = {\n    val instructionBuilders = Seq(\n      ClearCacheInstructionBuilder(\n        ClearCacheIncludeInstruction(\n          ClearCacheOnPtr.EnableParam,\n          ClearCacheOnPtr.MinEntriesParam\n        )\n      ),\n      ReplaceEntryInstructionBuilder(ReplaceAllEntries),\n      // excludes alert, cover, and replace candidates\n      AddEntriesWithReplaceAndShowAlertAndCoverInstructionBuilder(),\n      ShowAlertInstructionBuilder(),\n      ShowCoverInstructionBuilder(),\n    )\n\n    val idSelector: PartialFunction[UniversalNoun[_], Long] = {\n      // exclude ads while determining tweet cursor values\n      case item: TweetItem if item.promotedMetadata.isEmpty => item.id\n      case module: TimelineModule\n          if module.items.headOption.exists(_.item.isInstanceOf[TweetItem]) =>\n        module.items.last.item match { case item: TweetItem => item.id }\n    }\n    val topCursorBuilder = OrderedTopCursorBuilder(idSelector)\n    val bottomCursorBuilder = OrderedBottomCursorBuilder(idSelector)\n\n    val metadataBuilder = UrtMetadataBuilder(\n      title = None,\n      scribeConfigBuilder = Some(\n        StaticTimelineScribeConfigBuilder(\n          TimelineScribeConfig(\n            page = Some(\"for_you_timeline_scorer\"),\n            section = None,\n            entityToken = None)))\n    )\n\n    UrtDomainMarshaller(\n      instructionBuilders = instructionBuilders,\n      metadataBuilder = Some(metadataBuilder),\n      cursorBuilders = Seq(topCursorBuilder, bottomCursorBuilder)\n    )\n  }\n\n  override val transportMarshaller: TransportMarshaller[Timeline, urt.TimelineResponse] =\n    urtTransportMarshaller\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouTimelineScorerResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.tweetconvosvc.tweet_ancestor.{thriftscala => ta}\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.mediaservices.commons.tweetmedia.{thriftscala => mt}\nimport com.twitter.product_mixer.component_library.candidate_source.timeline_scorer.ScoredTweetCandidateWithFocalTweet\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BasicTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecWithEducationTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage\nimport com.twitter.search.common.util.lang.ThriftLanguageUtil\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.timelinemixer.injection.model.candidate.AudioSpaceMetaData\nimport com.twitter.timelines.conversation_features.{thriftscala => cvt}\nimport com.twitter.timelinescorer.common.scoredtweetcandidate.{thriftscala => stc}\nimport com.twitter.timelineservice.suggests.{thriftscala => tls}\n\nobject ForYouTimelineScorerResponseFeatureTransformer\n    extends CandidateFeatureTransformer[ScoredTweetCandidateWithFocalTweet] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ForYouTimelineScorerResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    AncestorsFeature,\n    AudioSpaceMetaDataFeature,\n    AuthorIdFeature,\n    AuthorIsBlueVerifiedFeature,\n    AuthorIsCreatorFeature,\n    AuthorIsGoldVerifiedFeature,\n    AuthorIsGrayVerifiedFeature,\n    AuthorIsLegacyVerifiedFeature,\n    AuthoredByContextualUserFeature,\n    CandidateSourceIdFeature,\n    ConversationFeature,\n    ConversationModuleFocalTweetIdFeature,\n    ConversationModuleIdFeature,\n    DirectedAtUserIdFeature,\n    EarlybirdFeature,\n    EntityTokenFeature,\n    ExclusiveConversationAuthorIdFeature,\n    FavoritedByUserIdsFeature,\n    FollowedByUserIdsFeature,\n    TopicIdSocialContextFeature,\n    TopicContextFunctionalityTypeFeature,\n    FromInNetworkSourceFeature,\n    FullScoringSucceededFeature,\n    HasDisplayedTextFeature,\n    InNetworkFeature,\n    InReplyToTweetIdFeature,\n    IsAncestorCandidateFeature,\n    IsExtendedReplyFeature,\n    IsRandomTweetFeature,\n    IsReadFromCacheFeature,\n    IsRetweetFeature,\n    IsRetweetedReplyFeature,\n    NonSelfFavoritedByUserIdsFeature,\n    NumImagesFeature,\n    OriginalTweetCreationTimeFromSnowflakeFeature,\n    PredictionRequestIdFeature,\n    QuotedTweetIdFeature,\n    ScoreFeature,\n    SimclustersTweetTopKClustersWithScoresFeature,\n    SourceTweetIdFeature,\n    SourceUserIdFeature,\n    StreamToKafkaFeature,\n    SuggestTypeFeature,\n    TweetLanguageFeature,\n    VideoDurationMsFeature,\n  )\n\n  // Convert language code to ISO 639-3 format\n  private def getLanguageISOFormatByValue(languageCodeValue: Int): String =\n    ThriftLanguageUtil.getLanguageCodeOf(ThriftLanguage.findByValue(languageCodeValue))\n\n  override def transform(\n    candidateWithFocalTweet: ScoredTweetCandidateWithFocalTweet\n  ): FeatureMap = {\n    val candidate: stc.v1.ScoredTweetCandidate = candidateWithFocalTweet.candidate\n    val focalTweetId = candidateWithFocalTweet.focalTweetIdOpt\n\n    val originalTweetId = candidate.sourceTweetId.getOrElse(candidate.tweetId)\n    val tweetFeatures = candidate.tweetFeaturesMap.flatMap(_.get(originalTweetId))\n    val earlybirdFeatures = tweetFeatures.flatMap(_.recapFeatures.flatMap(_.tweetFeatures))\n    val directedAtUserIsInFirstDegree =\n      earlybirdFeatures.flatMap(_.directedAtUserIdIsInFirstDegree)\n    val isReply = candidate.inReplyToTweetId.nonEmpty\n    val isRetweet = candidate.isRetweet.getOrElse(false)\n    val isInNetwork = candidate.isInNetwork.getOrElse(true)\n    val conversationFeatures = candidate.conversationFeatures.flatMap {\n      case cvt.ConversationFeatures.V1(candidate) => Some(candidate)\n      case _ => None\n    }\n    val numImages = candidate.mediaMetaData\n      .map(\n        _.count(mediaEntity =>\n          mediaEntity.mediaInfo.exists(_.isInstanceOf[mt.MediaInfo.ImageInfo]) ||\n            mediaEntity.mediaInfo.isEmpty))\n    val hasImage = earlybirdFeatures.exists(_.hasImage)\n    val hasVideo = earlybirdFeatures.exists(_.hasVideo)\n    val hasCard = earlybirdFeatures.exists(_.hasCard)\n    val hasQuote = earlybirdFeatures.exists(_.hasQuote.contains(true))\n    val hasDisplayedText = earlybirdFeatures.exists(_.tweetLength.exists(length => {\n      val numMedia = Seq(hasVideo, (hasImage || hasCard), hasQuote).count(b => b)\n      val tcoLengthsPlusSpaces = 23 * numMedia + (if (numMedia > 0) numMedia - 1 else 0)\n      length > tcoLengthsPlusSpaces\n    }))\n    val suggestType = candidate.overrideSuggestType.orElse(Some(tls.SuggestType.Undefined))\n\n    val topicSocialProofMetadataOpt = candidate.entityData.flatMap(_.topicSocialProofMetadata)\n    val topicIdSocialContextOpt = topicSocialProofMetadataOpt.map(_.topicId)\n    val topicContextFunctionalityTypeOpt =\n      topicSocialProofMetadataOpt.map(_.topicContextFunctionalityType).collect {\n        case stc.v1.TopicContextFunctionalityType.Basic => BasicTopicContextFunctionalityType\n        case stc.v1.TopicContextFunctionalityType.Recommendation =>\n          RecommendationTopicContextFunctionalityType\n        case stc.v1.TopicContextFunctionalityType.RecWithEducation =>\n          RecWithEducationTopicContextFunctionalityType\n      }\n\n    FeatureMapBuilder()\n      .add(\n        AncestorsFeature,\n        candidate.ancestors\n          .getOrElse(Seq.empty)\n          .map(ancestor => ta.TweetAncestor(ancestor.tweetId, ancestor.userId.getOrElse(0L))))\n      .add(\n        AudioSpaceMetaDataFeature,\n        candidate.audioSpaceMetaDatalist.map(_.head).map(AudioSpaceMetaData.fromThrift))\n      .add(AuthorIdFeature, Some(candidate.authorId))\n      .add(AuthorIsBlueVerifiedFeature, candidate.authorIsBlueVerified.getOrElse(false))\n      .add(\n        AuthorIsCreatorFeature,\n        candidate.authorIsCreator.getOrElse(false)\n      )\n      .add(AuthorIsGoldVerifiedFeature, candidate.authorIsGoldVerified.getOrElse(false))\n      .add(AuthorIsGrayVerifiedFeature, candidate.authorIsGrayVerified.getOrElse(false))\n      .add(AuthorIsLegacyVerifiedFeature, candidate.authorIsLegacyVerified.getOrElse(false))\n      .add(\n        AuthoredByContextualUserFeature,\n        candidate.viewerId.contains(candidate.authorId) ||\n          candidate.viewerId.exists(candidate.sourceUserId.contains))\n      .add(CandidateSourceIdFeature, candidate.candidateTweetSourceId)\n      .add(ConversationFeature, conversationFeatures)\n      .add(ConversationModuleIdFeature, candidate.conversationId)\n      .add(ConversationModuleFocalTweetIdFeature, focalTweetId)\n      .add(DirectedAtUserIdFeature, candidate.directedAtUserId)\n      .add(EarlybirdFeature, earlybirdFeatures)\n      // This is temporary, will need to be updated with the encoded string.\n      .add(EntityTokenFeature, Some(\"test_EntityTokenForYou\"))\n      .add(ExclusiveConversationAuthorIdFeature, candidate.exclusiveConversationAuthorId)\n      .add(FavoritedByUserIdsFeature, candidate.favoritedByUserIds.getOrElse(Seq.empty))\n      .add(FollowedByUserIdsFeature, candidate.followedByUserIds.getOrElse(Seq.empty))\n      .add(TopicIdSocialContextFeature, topicIdSocialContextOpt)\n      .add(TopicContextFunctionalityTypeFeature, topicContextFunctionalityTypeOpt)\n      .add(FullScoringSucceededFeature, candidate.fullScoringSucceeded.getOrElse(false))\n      .add(HasDisplayedTextFeature, hasDisplayedText)\n      .add(InNetworkFeature, candidate.isInNetwork.getOrElse(true))\n      .add(InReplyToTweetIdFeature, candidate.inReplyToTweetId)\n      .add(IsAncestorCandidateFeature, candidate.isAncestorCandidate.getOrElse(false))\n      .add(\n        IsExtendedReplyFeature,\n        isInNetwork && isReply && !isRetweet && directedAtUserIsInFirstDegree.contains(false))\n      .add(FromInNetworkSourceFeature, candidate.isInNetwork.getOrElse(true))\n      .add(IsRandomTweetFeature, candidate.isRandomTweet.getOrElse(false))\n      .add(IsReadFromCacheFeature, candidate.isReadFromCache.getOrElse(false))\n      .add(IsRetweetFeature, candidate.isRetweet.getOrElse(false))\n      .add(IsRetweetedReplyFeature, isReply && isRetweet)\n      .add(\n        NonSelfFavoritedByUserIdsFeature,\n        candidate.favoritedByUserIds.getOrElse(Seq.empty).filterNot(_ == candidate.authorId))\n      .add(NumImagesFeature, numImages)\n      .add(\n        OriginalTweetCreationTimeFromSnowflakeFeature,\n        SnowflakeId.timeFromIdOpt(originalTweetId))\n      .add(PredictionRequestIdFeature, candidate.predictionRequestId)\n      .add(ScoreFeature, Some(candidate.score))\n      .add(\n        SimclustersTweetTopKClustersWithScoresFeature,\n        candidate.simclustersTweetTopKClustersWithScores.map(_.toMap).getOrElse(Map.empty))\n      .add(\n        StreamToKafkaFeature,\n        candidate.predictionRequestId.nonEmpty && candidate.fullScoringSucceeded.getOrElse(false))\n      .add(SourceTweetIdFeature, candidate.sourceTweetId)\n      .add(SourceUserIdFeature, candidate.sourceUserId)\n      .add(SuggestTypeFeature, suggestType)\n      .add(QuotedTweetIdFeature, candidate.quotedTweetId)\n      .add(\n        TweetLanguageFeature,\n        earlybirdFeatures.flatMap(_.language.map(_.value)).map(getLanguageISOFormatByValue))\n      .add(VideoDurationMsFeature, earlybirdFeatures.flatMap(_.videoDurationMs))\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouTuneFeedCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.geoduck.util.country.CountryInfo\nimport com.twitter.home_mixer.functional_component.decorator.TuneFeedModuleCandidateDecorator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.PreviouslySeenTweetsFilter\nimport com.twitter.home_mixer.functional_component.filter.TweetHydrationFilter\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.model.HomeFeatures.CurrentDisplayedGrokTopicFeature\nimport com.twitter.home_mixer.product.for_you.candidate_source.TuneFeedCandidateSource\nimport com.twitter.home_mixer.product.for_you.candidate_source.TuneFeedRequest\nimport com.twitter.home_mixer.product.for_you.gate.TuneFeedModuleGate\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableTuneFeedCandidatePipelineParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.TuneFeedModuleMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.response_transformer.TuneFeedFeatureTransformer\nimport com.twitter.product_mixer.component_library.gate.DefinedUserIdGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouTuneFeedCandidatePipelineConfig @Inject() (\n  tuneFeedCandidateSource: TuneFeedCandidateSource,\n  tweetypieFeatureHydrator: TweetypieFeatureHydrator,\n  tuneFeedModuleDecorator: TuneFeedModuleCandidateDecorator)\n    extends CandidatePipelineConfig[\n      ForYouQuery,\n      TuneFeedRequest,\n      TweetCandidate,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ForYouTuneFeed\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] = Some(\n    EnableTuneFeedCandidatePipelineParam)\n\n  override val gates: Seq[Gate[ForYouQuery]] = Seq(\n    DefinedUserIdGate,\n    RateLimitGate,\n    TuneFeedModuleGate,\n    TimelinesPersistenceStoreLastInjectionGate(\n      TuneFeedModuleMinInjectionIntervalParam,\n      EntityIdType.TuneFeedModule\n    ),\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ForYouQuery,\n    TuneFeedRequest\n  ] = { query =>\n    // Safe .get, enforced in TuneFeedModuleGate\n    val displayedGrokTopic =\n      query.features.get.getOrElse(CurrentDisplayedGrokTopicFeature, None).get\n\n    TuneFeedRequest(\n      grokTopic = displayedGrokTopic._1,\n      language = query.getLanguageCode,\n      placeId = query.getCountryCode.flatMap(CountryInfo.lookupByCode).map(_.placeIdLong)\n    )\n  }\n\n  override def candidateSource: BaseCandidateSource[TuneFeedRequest, TweetCandidate] =\n    tuneFeedCandidateSource\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[TweetCandidate]\n  ] = Seq(TuneFeedFeatureTransformer)\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ForYouQuery, TweetCandidate, _]\n  ] = Seq(tweetypieFeatureHydrator)\n\n  override val filters: Seq[Filter[ForYouQuery, TweetCandidate]] = Seq(\n    TweetHydrationFilter,\n    PreviouslySeenTweetsFilter\n  )\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    TweetCandidate,\n    TweetCandidate\n  ] = identity\n\n  override val decorator: Option[CandidateDecorator[ForYouQuery, TweetCandidate]] = Some(\n    tuneFeedModuleDecorator)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouTweetPreviewsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.functional_component.filter.DropMaxCandidatesFilter\nimport com.twitter.home_mixer.functional_component.filter.PreviouslyServedTweetPreviewsFilter\nimport com.twitter.home_mixer.functional_component.filter.TweetHydrationFilter\nimport com.twitter.home_mixer.functional_component.gate.PersistenceStoreDurationValidationGate\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorEnabledPreviewsFeature\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.ArticlePreviewTextFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.AuthorEnabledPreviewsFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.TweetPreviewTweetypieCandidateFeatureHydrator\nimport com.twitter.home_mixer.product.for_you.filter.TweetPreviewTextFilter\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableArticlePreviewTextHydrationParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableTweetPreviewsCandidatePipelineParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.TweetPreviewsMaxCandidatesParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.TweetPreviewsMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.query_transformer.TweetPreviewsQueryTransformer\nimport com.twitter.home_mixer.product.for_you.response_transformer.TweetPreviewResponseFeatureTransformer\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.candidate_source.earlybird.EarlybirdTweetCandidateSource\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref.ContextualTweetRefBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.PreviewCreatorsFeature\nimport com.twitter.product_mixer.component_library.filter.FeatureFilter\nimport com.twitter.product_mixer.component_library.gate.NonEmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.TimelineHomeTweetPreviewHydrationSafetyLevel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.TweetHydrationContext\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouTweetPreviewsCandidatePipelineConfig @Inject() (\n  earlybirdTweetCandidateSource: EarlybirdTweetCandidateSource,\n  authorEnabledPreviewsFeatureHydrator: AuthorEnabledPreviewsFeatureHydrator,\n  tweetPreviewsQueryTransformer: TweetPreviewsQueryTransformer,\n  tweetPreviewTweetypieCandidateFeatureHydrator: TweetPreviewTweetypieCandidateFeatureHydrator,\n  articlePreviewTextFeatureHydrator: ArticlePreviewTextFeatureHydrator,\n  homeFeedbackActionInfoBuilder: HomeFeedbackActionInfoBuilder)\n    extends CandidatePipelineConfig[\n      ForYouQuery,\n      eb.EarlybirdRequest,\n      eb.ThriftSearchResult,\n      TweetCandidate\n    ] {\n\n  val identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\"ForYouTweetPreviews\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] =\n    Some(EnableTweetPreviewsCandidatePipelineParam)\n\n  override val gates: Seq[Gate[ForYouQuery]] = {\n    Seq(\n      RateLimitGate,\n      PersistenceStoreDurationValidationGate(),\n      TimelinesPersistenceStoreLastInjectionGate(\n        TweetPreviewsMinInjectionIntervalParam,\n        EntityIdType.TweetPreview\n      ),\n      NonEmptySeqFeatureGate(PreviewCreatorsFeature)\n    )\n  }\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    PipelineQuery,\n    eb.EarlybirdRequest\n  ] = tweetPreviewsQueryTransformer\n\n  override val candidateSource: CandidateSourceWithExtractedFeatures[\n    eb.EarlybirdRequest,\n    eb.ThriftSearchResult\n  ] = earlybirdTweetCandidateSource\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[eb.ThriftSearchResult]\n  ] = Seq(TweetPreviewResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    eb.ThriftSearchResult,\n    TweetCandidate\n  ] = { tweet => TweetCandidate(tweet.id) }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ForYouQuery, TweetCandidate, _]\n  ] = Seq(\n    authorEnabledPreviewsFeatureHydrator,\n    tweetPreviewTweetypieCandidateFeatureHydrator,\n  )\n\n  override val preFilterFeatureHydrationPhase2: Seq[\n    BaseCandidateFeatureHydrator[ForYouQuery, TweetCandidate, _]\n  ] = Seq(\n    ParamGatedCandidateFeatureHydrator(\n      candidateFeatureHydrator = articlePreviewTextFeatureHydrator,\n      enabledParam = EnableArticlePreviewTextHydrationParam))\n\n  override val filters: Seq[\n    Filter[ForYouQuery, TweetCandidate]\n  ] = Seq(\n    PreviouslyServedTweetPreviewsFilter,\n    TweetHydrationFilter,\n    FeatureFilter\n      .fromFeature(FilterIdentifier(\"AuthorEnabledPreviews\"), AuthorEnabledPreviewsFeature),\n    TweetPreviewTextFilter,\n    // NOTE: since earlybird returns candidates sorted by score, this will filter\n    // for the top scoring candidates. Should be updated in the future to not assume order of\n    // candidates from earlybird.\n    DropMaxCandidatesFilter(TweetPreviewsMaxCandidatesParam)\n  )\n\n  override val decorator: Option[CandidateDecorator[PipelineQuery, TweetCandidate]] = {\n    val clientEventInfoBuilder =\n      ClientEventInfoBuilder[PipelineQuery, TweetCandidate](\n        hmt.ServedType.ForYouTweetPreview.originalName)\n\n    val tweetItemBuilder = TweetCandidateUrtItemBuilder[PipelineQuery, TweetCandidate](\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      contextualTweetRefBuilder = Some(\n        ContextualTweetRefBuilder(\n          TweetHydrationContext(\n            safetyLevelOverride = Some(TimelineHomeTweetPreviewHydrationSafetyLevel),\n            outerTweetContext = None\n          )\n        )\n      ),\n      feedbackActionInfoBuilder = Some(homeFeedbackActionInfoBuilder),\n    )\n\n    Some(UrtItemCandidateDecorator(tweetItemBuilder))\n  }\n\n  override val alerts: Seq[Alert] = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(95))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouWhoToFollowCandidatePipelineConfigBuilder.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeWhoToFollowFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.functional_component.gate.DismissFatigueGate\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TestUserProbabilisticGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.model.HomeFeatures.DismissInfoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.WhoToFollowExcludedUserIdsFeature\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableWhoToFollowCandidatePipelineParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.WhoToFollowDisplayLocationParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.WhoToFollowDisplayTypeIdParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.WhoToFollowMinInjectionIntervalParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.WhoToFollowUserDisplayTypeIdParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ParamWhoToFollowModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.ParamWhoToFollowUserDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowArmCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowArmCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport com.twitter.timelineservice.suggests.thriftscala.SuggestType\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouWhoToFollowCandidatePipelineConfigBuilder @Inject() (\n  whoToFollowArmCandidatePipelineConfigBuilder: WhoToFollowArmCandidatePipelineConfigBuilder,\n  homeWhoToFollowFeedbackActionInfoBuilder: HomeWhoToFollowFeedbackActionInfoBuilder,\n  testUserProbabilisticGate: TestUserProbabilisticGate) {\n\n  def build(): WhoToFollowArmCandidatePipelineConfig[ForYouQuery] = {\n    val gates: Seq[Gate[ForYouQuery]] = Seq(\n      RateLimitGate,\n      TimelinesPersistenceStoreLastInjectionGate(\n        WhoToFollowMinInjectionIntervalParam,\n        EntityIdType.WhoToFollow\n      ),\n      DismissFatigueGate(SuggestType.WhoToFollow, DismissInfoFeature),\n      testUserProbabilisticGate\n    )\n\n    whoToFollowArmCandidatePipelineConfigBuilder.build[ForYouQuery](\n      identifier = WhoToFollowArmCandidatePipelineConfig.identifier,\n      supportedClientParam = Some(EnableWhoToFollowCandidatePipelineParam),\n      alerts = alerts,\n      gates = gates,\n      moduleDisplayTypeBuilder =\n        ParamWhoToFollowModuleDisplayTypeBuilder(WhoToFollowDisplayTypeIdParam),\n      feedbackActionInfoBuilder = Some(homeWhoToFollowFeedbackActionInfoBuilder),\n      excludedUserIdsFeature = Some(WhoToFollowExcludedUserIdsFeature),\n      userDisplayTypeBuilder =\n        ParamWhoToFollowUserDisplayTypeBuilder(WhoToFollowUserDisplayTypeIdParam),\n      displayLocationParam = WhoToFollowDisplayLocationParam\n    )\n  }\n\n  private val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(70),\n    HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/ForYouWhoToSubscribeCandidatePipelineConfigBuilder.scala",
    "content": "package com.twitter.home_mixer.product.for_you\n\nimport com.twitter.home_mixer.functional_component.decorator.urt.builder.HomeWhoToSubscribeFeedbackActionInfoBuilder\nimport com.twitter.home_mixer.functional_component.gate.DismissFatigueGate\nimport com.twitter.home_mixer.functional_component.gate.RateLimitGate\nimport com.twitter.home_mixer.functional_component.gate.TimelinesPersistenceStoreLastInjectionGate\nimport com.twitter.home_mixer.model.HomeFeatures.DismissInfoFeature\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableWhoToSubscribeCandidatePipelineParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.WhoToSubscribeDisplayTypeIdParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.WhoToSubscribeMinInjectionIntervalParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ParamWhoToFollowModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_subscribe_module.WhoToSubscribeCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_subscribe_module.WhoToSubscribeCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.timelineservice.model.rich.EntityIdType\nimport com.twitter.timelineservice.suggests.thriftscala.SuggestType\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouWhoToSubscribeCandidatePipelineConfigBuilder @Inject() (\n  whoToSubscribeCandidatePipelineConfigBuilder: WhoToSubscribeCandidatePipelineConfigBuilder,\n  homeWhoToSubscribeFeedbackActionInfoBuilder: HomeWhoToSubscribeFeedbackActionInfoBuilder) {\n\n  def build(): WhoToSubscribeCandidatePipelineConfig[ForYouQuery] = {\n    val gates: Seq[Gate[ForYouQuery]] = Seq(\n      RateLimitGate,\n      TimelinesPersistenceStoreLastInjectionGate(\n        WhoToSubscribeMinInjectionIntervalParam,\n        EntityIdType.WhoToSubscribe\n      ),\n      DismissFatigueGate(SuggestType.WhoToSubscribe, DismissInfoFeature)\n    )\n\n    whoToSubscribeCandidatePipelineConfigBuilder.build[ForYouQuery](\n      identifier = WhoToSubscribeCandidatePipelineConfig.identifier,\n      supportedClientParam = Some(EnableWhoToSubscribeCandidatePipelineParam),\n      alerts = alerts,\n      gates = gates,\n      moduleDisplayTypeBuilder =\n        ParamWhoToFollowModuleDisplayTypeBuilder(WhoToSubscribeDisplayTypeIdParam),\n      feedbackActionInfoBuilder = Some(homeWhoToSubscribeFeedbackActionInfoBuilder)\n    )\n  }\n\n  private val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(70),\n    HomeMixerAlertConfig.BusinessHours.defaultEmptyResponseRateAlert()\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"events-recos/events-recos-service/src/main/thrift:events-recos-thrift-scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/model\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/product_pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"recruiting/candidate-service/src/main/thrift:thrift-scala\",\n        \"recruiting/src/main/thrift:thrift-scala\",\n        \"src/thrift/com/twitter/frigate/bookmarks:bookmarks-thrift-scala\",\n        \"src/thrift/com/twitter/timelines/impression_bloom_filter:thrift-scala\",\n        \"stitch/stitch-timelineservice/src/main/scala\",\n        \"strato/config/columns/events/entryPoint:entryPoint-strato-client\",\n        \"strato/config/columns/frigate/bookmarks:bookmarks-strato-client\",\n        \"strato/config/columns/recruiting/api/user:user-strato-client\",\n        \"strato/config/columns/recruiting/candidate_service:candidate_service-strato-client\",\n        \"strato/config/columns/trends/trip:trip-strato-client\",\n        \"strato/config/columns/trendsai/ordered:ordered-strato-client\",\n        \"strato/config/src/thrift/com/twitter/strato/columns/jetfuel:jetfuel-scala\",\n        \"timelines/src/main/scala/com/twitter/timelines/impressionstore/impressionbloomfilter\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source/BookmarksCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.for_you.candidate_source\n\nimport com.twitter.frigate.bookmarks.thriftscala.BookmarkedTweet\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.frigate.bookmarks.WeeklyBookmarksOnUserClientColumn\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.util.Random\n\n@Singleton\nclass BookmarksCandidateSource @Inject() (\n  weeklyBookmarksOnUserClientColumn: WeeklyBookmarksOnUserClientColumn)\n    extends CandidateSource[\n      Long,\n      BookmarkedTweet\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"Bookmarks\")\n\n  private val MaxCandidates = 20\n  private val fetcher = weeklyBookmarksOnUserClientColumn.fetcher\n\n  // Remove duplicates and unbookmarked tweets\n  private def filterValidBookmarks(bookmarks: Seq[BookmarkedTweet]): Seq[BookmarkedTweet] = {\n    bookmarks\n      .groupBy(_.tweetId)\n      .collect {\n        case (_, duplicatedBookmarks)\n            if !duplicatedBookmarks.exists(_.unbookmark.getOrElse(false)) =>\n          duplicatedBookmarks.head\n      }.toSeq\n  }\n\n  override def apply(userId: Long): Stitch[Seq[BookmarkedTweet]] = {\n    fetcher.fetch(userId, ()).map { res =>\n      res.v\n        .map { bookmarks =>\n          val filteredBookmarks = filterValidBookmarks(bookmarks.bookmarkedTweets)\n          Random.shuffle(filteredBookmarks).take(MaxCandidates)\n        }.getOrElse(Seq.empty)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source/BroadcastedPinnedTweetsCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.for_you.candidate_source\n\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.model.HomeFeatures.ImpressionBloomFilterFeature\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.timelineservice.TimelineService\nimport com.twitter.timelines.impressionstore.impressionbloomfilter.ImpressionBloomFilterItem\nimport com.twitter.timelineservice.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase class PinnedTweetCandidate(tweetId: Long, userId: Option[Long])\n\n@Singleton\nclass BroadcastedPinnedTweetsCandidateSource @Inject() (\n  timelineService: TimelineService)\n    extends CandidateSource[\n      ForYouQuery,\n      PinnedTweetCandidate\n    ] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"BroadcastedPinnedTweets\")\n\n  private val MaxTimelineServiceTweets = 100\n  private val MaxTweetsForHydration = 20\n\n  override def apply(query: ForYouQuery): Stitch[Seq[PinnedTweetCandidate]] = {\n    val userId = query.getRequiredUserId\n    val timelineQueryOptions = t.TimelineQueryOptions(\n      contextualUserId = Some(userId)\n    )\n\n    val timelineServiceQuery = t.TimelineQuery(\n      timelineType = t.TimelineType.PinnedTweets,\n      timelineId = userId,\n      maxCount = MaxTimelineServiceTweets.toShort,\n      cursor2 = None,\n      options = Some(timelineQueryOptions),\n      timelineId2 = Some(t.TimelineId(t.TimelineType.PinnedTweets, userId)),\n    )\n\n    timelineService\n      .getTimeline(timelineServiceQuery).map { timeline =>\n        timeline.entries.collect {\n          case t.TimelineEntry.Tweet(tweet) =>\n            PinnedTweetCandidate(tweet.statusId, tweet.userId)\n        }\n      }.map { candidates =>\n        val bloomFilterSeq = query.features.map(_.get(ImpressionBloomFilterFeature)).get\n        val bloomFilters =\n          bloomFilterSeq.entries.map(ImpressionBloomFilterItem.fromThrift(_).bloomFilter)\n\n        val seenTweetIds = query.seenTweetIds.getOrElse(Seq.empty).toSet\n\n        candidates\n          .filter { pinnedTweetCandidate =>\n            !seenTweetIds.contains(pinnedTweetCandidate.tweetId) &&\n            !bloomFilters.exists(filter => filter.mayContain(pinnedTweetCandidate.tweetId))\n          }\n          .take(MaxTweetsForHydration)\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source/JetfuelFrameCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.for_you.candidate_source\n\nimport com.twitter.strato.columns.jetfuel.thriftscala.JetfuelRouteData\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyViewFetcherSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.events.entryPoint.JetfuelEntryPointByCountryClientColumn\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Strato column to fetch entry point for sports to display in search results.\n */\n@Singleton\nclass JetfuelFrameCandidateSource @Inject() (\n  entryPointClientColumn: JetfuelEntryPointByCountryClientColumn)\n  extends StratoKeyViewFetcherSource[\n    JetfuelEntryPointByCountryClientColumn.Key,\n    JetfuelEntryPointByCountryClientColumn.View,\n    JetfuelEntryPointByCountryClientColumn.Value,\n    JetfuelRouteData\n  ] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"JetfuelFrame\")\n\n  override val fetcher: Fetcher[\n    JetfuelEntryPointByCountryClientColumn.Key,\n    JetfuelEntryPointByCountryClientColumn.View,\n    JetfuelEntryPointByCountryClientColumn.Value,\n  ] = entryPointClientColumn.fetcher\n\n  override def stratoResultTransformer(\n    stratoKey: JetfuelEntryPointByCountryClientColumn.Key,\n    stratoResult: JetfuelEntryPointByCountryClientColumn.Value\n  ): Seq[JetfuelRouteData] = stratoResult\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source/RecommendedJobsCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.for_you.candidate_source\n\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithModerateTimeout\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyViewFetcherSource\nimport com.twitter.strato.generated.client.recruiting.candidate_service.JobRecommendationCandidatesOnUserClientColumn\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.candidateservice.core_io.FetchJobRecommendationsView\nimport com.twitter.strato.client.Client\n\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass RecommendedJobsCandidateSource @Inject() (\n  @Named(BatchedStratoClientWithModerateTimeout) stratoClient: Client)\n    extends StratoKeyViewFetcherSource[\n      Long,\n      FetchJobRecommendationsView,\n      JobRecommendationCandidatesOnUserClientColumn.Value,\n      Long\n    ] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"RecommendedJobs\")\n\n  val fetcher = stratoClient.fetcher[\n    Long,\n    FetchJobRecommendationsView,\n    JobRecommendationCandidatesOnUserClientColumn.Value\n  ](JobRecommendationCandidatesOnUserClientColumn.Path)\n\n  override protected def stratoResultTransformer(\n    stratoKey: Long,\n    stratoResult: JobRecommendationCandidatesOnUserClientColumn.Value\n  ): Seq[Long] = stratoResult.map(_.apiJob)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source/RecommendedRecruitingOrganizationsCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.for_you.candidate_source\n\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithModerateTimeout\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyViewFetcherSource\nimport com.twitter.strato.generated.client.recruiting.api.user.RecruitingOrganizationRecommendationsOnUserClientColumn\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.recruiting.organization.RecruitingOrganizationRecommendationsView\nimport com.twitter.strato.client.Client\n\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass RecommendedRecruitingOrganizationsCandidateSource @Inject() (\n  @Named(BatchedStratoClientWithModerateTimeout) stratoClient: Client)\n    extends StratoKeyViewFetcherSource[\n      Long,\n      RecruitingOrganizationRecommendationsView,\n      RecruitingOrganizationRecommendationsOnUserClientColumn.Value,\n      Long\n    ] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"RecommendedRecruitingOrganizations\")\n\n  val fetcher = stratoClient.fetcher[\n    Long,\n    RecruitingOrganizationRecommendationsView,\n    RecruitingOrganizationRecommendationsOnUserClientColumn.Value\n  ](RecruitingOrganizationRecommendationsOnUserClientColumn.Path)\n\n  override protected def stratoResultTransformer(\n    stratoKey: Long,\n    stratoResult: RecruitingOrganizationRecommendationsOnUserClientColumn.Value\n  ): Seq[Long] = stratoResult.map(_.apiRecruitingOrganization)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source/ScoredTweetsProductCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.for_you.candidate_source\n\nimport com.google.inject.Provider\nimport com.twitter.home_mixer.model.HomeFeatures.ServedAuthorIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTweetIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SignupCountryFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SignupSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TimelineServiceTweetsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserFollowersCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerAllowsForYouRecommendationsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerHasPremiumTier\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.model.request.ScoredTweetsProduct\nimport com.twitter.home_mixer.model.request.ScoredTweetsProductContext\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.core.functional_component.candidate_source.product_pipeline.ProductPipelineCandidateSource\nimport com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.spam.rtf.{thriftscala => spam}\nimport com.twitter.timelines.render.{thriftscala => tl}\nimport com.twitter.tweetconvosvc.tweet_ancestor.{thriftscala => ta}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * [[ScoredTweetWithConversationMetadata]]\n **/\ncase class ScoredTweetWithConversationMetadata(\n  tweetId: Long,\n  authorId: Long,\n  score: Option[Double] = None,\n  servedType: t.ServedType,\n  sourceTweetId: Option[Long] = None,\n  sourceUserId: Option[Long] = None,\n  quotedTweetId: Option[Long] = None,\n  quotedUserId: Option[Long] = None,\n  inReplyToTweetId: Option[Long] = None,\n  inReplyToUserId: Option[Long] = None,\n  directedAtUserId: Option[Long] = None,\n  inNetwork: Option[Boolean] = None,\n  sgsValidLikedByUserIds: Option[Seq[Long]] = None,\n  sgsValidFollowedByUserIds: Option[Seq[Long]] = None,\n  validLikedByUserIds: Option[Seq[Long]] = None,\n  ancestors: Option[Seq[ta.TweetAncestor]] = None,\n  topicId: Option[Long] = None,\n  topicFunctionalityType: Option[tl.TopicContextFunctionalityType] = None,\n  conversationId: Option[Long] = None,\n  conversationFocalTweetId: Option[Long] = None,\n  isReadFromCache: Option[Boolean] = None,\n  exclusiveConversationAuthorId: Option[Long] = None,\n  authorIsBlueVerified: Option[Boolean] = None,\n  authorIsGoldVerified: Option[Boolean] = None,\n  authorIsGrayVerified: Option[Boolean] = None,\n  authorIsLegacyVerified: Option[Boolean] = None,\n  authorIsCreator: Option[Boolean] = None,\n  authorFollowers: Option[Long] = None,\n  locationId: Option[String] = None,\n  predictionRequestId: Option[Long] = None,\n  communityId: Option[Long] = None,\n  communityName: Option[String] = None,\n  listId: Option[Long] = None,\n  listName: Option[String] = None,\n  isNsfw: Option[Boolean] = None,\n  visibilityReason: Option[spam.FilteredReason] = None,\n  tweetLanguage: Option[String] = None,\n  tweetText: Option[String] = None,\n  tweetTypeMetrics: Option[Seq[Byte]] = None,\n  debugString: Option[String] = None,\n  viralContentCreatorFeature: Option[Boolean] = None,\n  grokContentCreatorFeature: Option[Boolean] = None,\n  gorkContentCreatorFeature: Option[Boolean] = None,\n  hasVideoFeature: Option[Boolean] = None,\n  videoDurationMsFeature: Option[Int] = None,\n  mediaIds: Option[Seq[Long]] = None,\n  grokAnnotations: Option[t.GrokAnnotations] = None,\n  predictedScores: Option[t.PredictedScores] = None,\n  phoenixPredictedScores: Option[t.PredictedScores] = None,\n  sourceSignal: Option[t.SourceSignal] = None,\n  userActionsSize: Option[Int] = None,\n  userActionsContainsExplicitSignals: Option[Boolean] = None)\n\n@Singleton\nclass ScoredTweetsProductCandidateSource @Inject() (\n  override val productPipelineRegistry: Provider[ProductPipelineRegistry],\n  override val paramsBuilder: Provider[ParamsBuilder])\n    extends ProductPipelineCandidateSource[\n      ForYouQuery,\n      HomeMixerRequest,\n      t.ScoredTweetsResponse,\n      ScoredTweetWithConversationMetadata\n    ] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"ScoredTweetsProduct\")\n\n  private val MaxModuleSize = 3\n  private val MaxAncestorsInConversation = 2\n\n  override def fsCustomMapInput(query: ForYouQuery): Map[String, Int] = {\n    val userAgeOpt = query.clientContext.userId.map { userId =>\n      SnowflakeId.timeFromIdOpt(userId).map(_.untilNow.inDays).getOrElse(Int.MaxValue)\n    }\n    val premium = query.features\n      .map(_.getOrElse(ViewerHasPremiumTier, false)).getOrElse(false)\n    Map(\n      \"account_age_in_days\" -> userAgeOpt.getOrElse(Int.MaxValue),\n      \"premium\" -> (if (premium) 1 else 0)\n    )\n  }\n\n  override def pipelineRequestTransformer(productPipelineQuery: ForYouQuery): HomeMixerRequest = {\n    HomeMixerRequest(\n      clientContext = productPipelineQuery.clientContext,\n      product = ScoredTweetsProduct,\n      productContext = Some(\n        ScoredTweetsProductContext(\n          productPipelineQuery.deviceContext,\n          productPipelineQuery.seenTweetIds,\n          productPipelineQuery.features.map(_.getOrElse(ServedTweetIdsFeature, Seq.empty)),\n          productPipelineQuery.features.map(_.getOrElse(TimelineServiceTweetsFeature, Seq.empty)),\n          productPipelineQuery.features.flatMap(_.getOrElse(SignupCountryFeature, None)),\n          productPipelineQuery.features\n            .flatMap(_.getOrElse(ViewerAllowsForYouRecommendationsFeature, None)),\n          productPipelineQuery.features.flatMap(_.getOrElse(SignupSourceFeature, None)),\n          productPipelineQuery.features.flatMap(_.getOrElse(UserFollowersCountFeature, None)),\n          productPipelineQuery.features\n            .map(_.getOrElse(ServedAuthorIdsFeature, Map.empty[Long, Seq[Long]]))\n        )\n      ),\n      serializedRequestCursor =\n        productPipelineQuery.pipelineCursor.map(UrtCursorSerializer.serializeCursor),\n      maxResults = None,\n      debugParams = None,\n      homeRequestParam = false\n    )\n  }\n\n  override def productPipelineResultTransformer(\n    productPipelineResult: t.ScoredTweetsResponse\n  ): Seq[ScoredTweetWithConversationMetadata] = {\n    val scoredTweets = productPipelineResult.scoredTweets.flatMap { focalTweet =>\n      val parentTweets = focalTweet.ancestors.getOrElse(Seq.empty).sortBy(-_.tweetId)\n      val (intermediates, root) = parentTweets.splitAt(parentTweets.size - 1)\n      val truncatedIntermediates =\n        intermediates.take(MaxModuleSize - MaxAncestorsInConversation).reverse\n      val rootScoredTweet: Seq[ScoredTweetWithConversationMetadata] = root.map { ancestor =>\n        ScoredTweetWithConversationMetadata(\n          tweetId = ancestor.tweetId,\n          authorId = ancestor.userId,\n          servedType = focalTweet.servedType,\n          conversationId = Some(ancestor.tweetId),\n          conversationFocalTweetId = Some(focalTweet.tweetId),\n          exclusiveConversationAuthorId = focalTweet.exclusiveConversationAuthorId\n        )\n      }\n      val conversationId = rootScoredTweet.headOption.map(_.tweetId)\n\n      val tweetsToParents =\n        if (parentTweets.nonEmpty) parentTweets.zip(parentTweets.tail).toMap\n        else Map.empty[ta.TweetAncestor, ta.TweetAncestor]\n\n      val intermediateScoredTweets = truncatedIntermediates.map { ancestor =>\n        ScoredTweetWithConversationMetadata(\n          tweetId = ancestor.tweetId,\n          authorId = ancestor.userId,\n          servedType = focalTweet.servedType,\n          inReplyToTweetId = tweetsToParents.get(ancestor).map(_.tweetId),\n          conversationId = conversationId,\n          conversationFocalTweetId = Some(focalTweet.tweetId),\n          exclusiveConversationAuthorId = focalTweet.exclusiveConversationAuthorId\n        )\n      }\n      val parentScoredTweets = rootScoredTweet ++ intermediateScoredTweets\n\n      val conversationFocalTweetId =\n        if (parentScoredTweets.nonEmpty) Some(focalTweet.tweetId) else None\n\n      val focalScoredTweet = ScoredTweetWithConversationMetadata(\n        tweetId = focalTweet.tweetId,\n        authorId = focalTweet.authorId,\n        score = focalTweet.score,\n        servedType = focalTweet.servedType,\n        sourceTweetId = focalTweet.sourceTweetId,\n        sourceUserId = focalTweet.sourceUserId,\n        quotedTweetId = focalTweet.quotedTweetId,\n        quotedUserId = focalTweet.quotedUserId,\n        inReplyToTweetId = parentScoredTweets.lastOption.map(_.tweetId),\n        inReplyToUserId = focalTweet.inReplyToUserId,\n        directedAtUserId = focalTweet.directedAtUserId,\n        inNetwork = focalTweet.inNetwork,\n        sgsValidLikedByUserIds = focalTweet.sgsValidLikedByUserIds,\n        sgsValidFollowedByUserIds = focalTweet.sgsValidFollowedByUserIds,\n        validLikedByUserIds = focalTweet.validLikedByUserIds,\n        topicId = focalTweet.topicId,\n        topicFunctionalityType = focalTweet.topicFunctionalityType,\n        ancestors = focalTweet.ancestors,\n        conversationId = conversationId,\n        conversationFocalTweetId = conversationFocalTweetId,\n        isReadFromCache = focalTweet.isReadFromCache,\n        exclusiveConversationAuthorId = focalTweet.exclusiveConversationAuthorId,\n        authorIsBlueVerified = focalTweet.authorMetadata.map(_.blueVerified),\n        authorIsGoldVerified = focalTweet.authorMetadata.map(_.goldVerified),\n        authorIsGrayVerified = focalTweet.authorMetadata.map(_.grayVerified),\n        authorIsLegacyVerified = focalTweet.authorMetadata.map(_.legacyVerified),\n        authorIsCreator = focalTweet.authorMetadata.map(_.creator),\n        authorFollowers = focalTweet.authorMetadata.flatMap(_.followers),\n        locationId = focalTweet.locationId,\n        predictionRequestId = focalTweet.predictionRequestId,\n        communityId = focalTweet.communityId,\n        communityName = focalTweet.communityName,\n        listId = focalTweet.listId,\n        listName = focalTweet.listName,\n        isNsfw = focalTweet.isNsfw,\n        visibilityReason = focalTweet.visibilityReason,\n        tweetLanguage = focalTweet.tweetLanguage,\n        tweetText = focalTweet.tweetText,\n        tweetTypeMetrics = focalTweet.tweetTypeMetrics,\n        debugString = focalTweet.debugString,\n        viralContentCreatorFeature = focalTweet.viralContentCreator,\n        hasVideoFeature = focalTweet.hasVideo,\n        videoDurationMsFeature = focalTweet.videoDurationMs,\n        mediaIds = focalTweet.mediaIds,\n        grokAnnotations = focalTweet.grokAnnotations,\n        predictedScores = focalTweet.predictedScores,\n        phoenixPredictedScores = focalTweet.phoenixPredictedScores,\n        sourceSignal = focalTweet.sourceSignal,\n        userActionsSize = focalTweet.userActionsSize,\n        userActionsContainsExplicitSignals = focalTweet.userActionsContainsExplicitSignals,\n        grokContentCreatorFeature = focalTweet.grokContentCreator,\n        gorkContentCreatorFeature = focalTweet.gorkContentCreator,\n      )\n\n      parentScoredTweets :+ focalScoredTweet\n    }\n\n    val dedupedTweets = scoredTweets.groupBy(_.tweetId).map {\n      case (_, duplicateAncestors) => duplicateAncestors.maxBy(_.score.getOrElse(0.0))\n    }\n\n    // Sort by tweet id to prevent issues with future assumptions of the root being the first\n    // tweet and the focal being the last tweet in a module. The tweets as a whole do not need\n    // to be sorted overall, only the relative order within modules must be kept.\n    dedupedTweets.toSeq.sortBy(_.tweetId)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source/ScoredVideoTweetsCategorizedProductCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.for_you.candidate_source\n\nimport com.google.inject.Provider\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.model.request.ScoredVideoTweetsProduct\nimport com.twitter.home_mixer.model.request.ScoredVideoTweetsProductContext\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.core.functional_component.candidate_source.product_pipeline.ProductPipelineCandidateSource\nimport com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry\nimport com.twitter.snowflake.id.SnowflakeId\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoredVideoTweetsCategorizedProductCandidateSource @Inject() (\n  override val productPipelineRegistry: Provider[ProductPipelineRegistry],\n  override val paramsBuilder: Provider[ParamsBuilder])\n    extends ProductPipelineCandidateSource[\n      ForYouQuery,\n      HomeMixerRequest,\n      t.ScoredTweetsResponse,\n      ScoredVideoTweetCandidate\n    ] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"ScoredVideoTweetsCategorizedProduct\")\n\n  private val MAX_RESULTS = 20\n\n  override def fsCustomMapInput(query: ForYouQuery): Map[String, Int] = {\n    val userAgeOpt = query.clientContext.userId.map { userId =>\n      SnowflakeId.timeFromIdOpt(userId).map(_.untilNow.inDays).getOrElse(Int.MaxValue)\n    }\n    userAgeOpt.map(\"account_age_in_days\" -> _).toMap\n  }\n\n  override def pipelineRequestTransformer(productPipelineQuery: ForYouQuery): HomeMixerRequest = {\n    HomeMixerRequest(\n      clientContext = productPipelineQuery.clientContext,\n      product = ScoredVideoTweetsProduct,\n      productContext = Some(\n        ScoredVideoTweetsProductContext(\n          productPipelineQuery.deviceContext,\n          productPipelineQuery.seenTweetIds,\n          Some(t.VideoType.Categorized),\n          None,\n          None,\n          None\n        )),\n      serializedRequestCursor = None,\n      maxResults = Some(MAX_RESULTS),\n      debugParams = None,\n      homeRequestParam = false\n    )\n  }\n\n  override def productPipelineResultTransformer(\n    productPipelineResult: t.ScoredTweetsResponse\n  ): Seq[ScoredVideoTweetCandidate] = {\n    productPipelineResult.scoredTweets.map { scoredTweet =>\n      ScoredVideoTweetCandidate(\n        scoredTweet.tweetId,\n        scoredTweet.authorId,\n        scoredTweet.score,\n        scoredTweet.servedType,\n        scoredTweet.aspectRatio\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source/ScoredVideoTweetsProductCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.for_you.candidate_source\n\nimport com.google.inject.Provider\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.model.request.ScoredVideoTweetsProduct\nimport com.twitter.home_mixer.model.request.ScoredVideoTweetsProductContext\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.core.functional_component.candidate_source.product_pipeline.ProductPipelineCandidateSource\nimport com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry\nimport com.twitter.snowflake.id.SnowflakeId\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * [[ScoredTweetWithConversationMetadata]]\n **/\ncase class ScoredVideoTweetCandidate(\n  tweetId: Long,\n  authorId: Long,\n  score: Option[Double],\n  servedType: hmt.ServedType,\n  aspectRatio: Option[Double])\n\n@Singleton\nclass ScoredVideoTweetsProductCandidateSource @Inject() (\n  override val productPipelineRegistry: Provider[ProductPipelineRegistry],\n  override val paramsBuilder: Provider[ParamsBuilder])\n    extends ProductPipelineCandidateSource[\n      ForYouQuery,\n      HomeMixerRequest,\n      t.ScoredTweetsResponse,\n      ScoredVideoTweetCandidate\n    ] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"ScoredVideoTweetsProduct\")\n\n  private val MAX_RESULTS = 20\n\n  override def fsCustomMapInput(query: ForYouQuery): Map[String, Int] = {\n    val userAgeOpt = query.clientContext.userId.map { userId =>\n      SnowflakeId.timeFromIdOpt(userId).map(_.untilNow.inDays).getOrElse(Int.MaxValue)\n    }\n    userAgeOpt.map(\"account_age_in_days\" -> _).toMap\n  }\n\n  override def pipelineRequestTransformer(productPipelineQuery: ForYouQuery): HomeMixerRequest = {\n    HomeMixerRequest(\n      clientContext = productPipelineQuery.clientContext,\n      product = ScoredVideoTweetsProduct,\n      productContext = Some(\n        ScoredVideoTweetsProductContext(\n          productPipelineQuery.deviceContext,\n          productPipelineQuery.seenTweetIds,\n          None,\n          None,\n          None,\n          None\n        )),\n      serializedRequestCursor = None,\n      maxResults = Some(MAX_RESULTS),\n      debugParams = None,\n      homeRequestParam = false\n    )\n  }\n\n  override def productPipelineResultTransformer(\n    productPipelineResult: t.ScoredTweetsResponse\n  ): Seq[ScoredVideoTweetCandidate] = {\n    productPipelineResult.scoredTweets.map { scoredTweet =>\n      ScoredVideoTweetCandidate(\n        scoredTweet.tweetId,\n        scoredTweet.authorId,\n        scoredTweet.score,\n        scoredTweet.servedType,\n        scoredTweet.aspectRatio\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source/StoriesModuleCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.for_you.candidate_source\n\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithModerateTimeout\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyFetcherSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.trendsai.ordered.TrendsModuleOnUserClientColumn\nimport com.twitter.strato.generated.client.trendsai.ordered.TrendsModuleOnUserClientColumn.Key\nimport com.twitter.strato.generated.client.trendsai.ordered.TrendsModuleOnUserClientColumn.Value\nimport com.twitter.strato.graphql.thriftscala.ApiImage\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\ncase class StoryCandidate(\n  id: Long,\n  title: String,\n  context: String,\n  hook: Option[String],\n  thumbnail: Option[ApiImage],\n  socialProof: Seq[String])\n\n@Singleton\nclass StoriesModuleCandidateSource @Inject() (\n  @Named(BatchedStratoClientWithModerateTimeout) stratoClient: Client)\n    extends StratoKeyFetcherSource[\n      TrendsModuleOnUserClientColumn.Key,\n      TrendsModuleOnUserClientColumn.Value,\n      StoryCandidate\n    ] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(name = \"StoriesModule\")\n\n  val fetcher: Fetcher[Key, Unit, Value] = stratoClient.fetcher[\n    TrendsModuleOnUserClientColumn.Key,\n    TrendsModuleOnUserClientColumn.View,\n    TrendsModuleOnUserClientColumn.Value\n  ](TrendsModuleOnUserClientColumn.Path)\n\n  override protected def stratoResultTransformer(stratoResult: Value): Seq[StoryCandidate] =\n    stratoResult.items.map { v =>\n      StoryCandidate(\n        v.id,\n        v.title,\n        v.context,\n        v.hook,\n        v.thumbnail.map { t => ApiImage(t.url, t.width, t.height) },\n        v.facepile\n      )\n    }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source/TuneFeedCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.for_you.candidate_source\n\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.trends.trip.GrokTopicTweetsClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase class TuneFeedRequest(\n  grokTopic: Long,\n  language: Option[String],\n  placeId: Option[Long])\n\n/*\n * Takes 3 grok topics the user is following at random, and fetches 1 post for each.\n */\n@Singleton\nclass TuneFeedCandidateSource @Inject() (\n  grokTopicTweetsStratoColumn: GrokTopicTweetsClientColumn)\n    extends CandidateSource[\n      TuneFeedRequest,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"TuneFeed\")\n\n  private val fetcher = grokTopicTweetsStratoColumn.fetcher\n\n  private val MaxModuleTweets = 20\n\n  override def apply(tuneFeedRequest: TuneFeedRequest): Stitch[Seq[TweetCandidate]] = {\n    val key = GrokTopicTweetsClientColumn.Key(\n      language = tuneFeedRequest.language,\n      placeId = tuneFeedRequest.placeId,\n      topicId = tuneFeedRequest.grokTopic\n    )\n\n    fetcher.fetch(key, {}).map { response =>\n      response.v\n        .map(_.map { tweetId =>\n          TweetCandidate(tweetId)\n        })\n        .getOrElse(Seq.empty).take(MaxModuleTweets)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source/UnifiedTrendsCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.for_you.candidate_source\n\nimport com.twitter.events.recos.{thriftscala => t}\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase class TrendCandidate(candidate: t.TrendCandidate, rank: Option[Int])\n\n@Singleton\nclass UnifiedTrendsCandidateSource @Inject() (\n  unifiedCandidatesService: t.EventsRecosService.MethodPerEndpoint)\n    extends CandidateSource[\n      t.GetUnfiedCandidatesRequest,\n      TrendCandidate\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"UnifiedTrends\")\n\n  override def apply(request: t.GetUnfiedCandidatesRequest): Stitch[Seq[TrendCandidate]] = {\n    Stitch.callFuture(unifiedCandidatesService.getUnifiedCandidates(request)).map {\n      unifiedCandidateResponse =>\n        val trends: Seq[t.TrendCandidate] = unifiedCandidateResponse.candidates.collect {\n          case t.UnifiedCandidate.Trend(trend) => trend\n        }\n\n        trends.zipWithIndex.map {\n          case (trend, index) => TrendCandidate(trend, Some(index + 1))\n        }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/ArticlePreviewTextFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.for_you.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.ArticleIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ArticlePreviewTextFeature\nimport com.twitter.home_mixer.product.for_you.feature_hydrator.ArticlePreviewTextFeatureHydrator.ArticlePreviewTextColumn\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ArticlePreviewTextFeatureHydrator @Inject() (\n  stratoClient: StratoClient)\n    extends CandidateFeatureHydrator[PipelineQuery, BaseTweetCandidate] {\n\n  private val fetcher =\n    stratoClient.fetcher[Long, Unit, String](ArticlePreviewTextColumn)\n\n  override val features: Set[Feature[_, _]] =\n    Set(ArticlePreviewTextFeature)\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"ArticlePreviewText\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: BaseTweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n\n    existingFeatures.get(ArticleIdFeature) match {\n      case Some(articleId) =>\n        fetcher.fetch(articleId).map { fetchResult =>\n          val articlePreviewText: Option[String] = fetchResult.v\n          FeatureMap(ArticlePreviewTextFeature, articlePreviewText)\n        }\n      case None =>\n        Stitch.value(FeatureMap(ArticlePreviewTextFeature, None))\n    }\n\n  }\n}\n\nobject ArticlePreviewTextFeatureHydrator {\n  final val ArticlePreviewTextColumn: String = \"article/fields/previewText.ArticleEntity\"\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/AuthorEnabledPreviewsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.for_you.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorEnabledPreviewsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.audiencerewards.audienceRewardsService.GetCreatorPreferencesOnUserClientColumn\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Hydrates the `AuthorEnabledPreviews` feature for tweets authored by creators by querying the\n * `GetCreatorPreferences` Strato column. This feature corresponds to the `previews_enabled` field of that column.\n * Given a tweet from a creator, this feature indicates whether that creator has enabled previews\n * on their profile.\n */\n@Singleton\nclass AuthorEnabledPreviewsFeatureHydrator @Inject() (\n  getCreatorPreferencesOnUserClientColumn: GetCreatorPreferencesOnUserClientColumn)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"AuthorEnabledPreviews\")\n\n  override val features: Set[Feature[_, _]] = Set(AuthorEnabledPreviewsFeature)\n\n  private val fetcher = getCreatorPreferencesOnUserClientColumn.fetcher\n\n  private val DefaultAuthorEnabledPreviewsValue = true\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val candidateAuthors = candidates\n      .map(_.features.getOrElse(AuthorIdFeature, None))\n      .toSet\n      .flatten\n\n    // Build a map of creator -> authorEnabledPreviews, then use it to populate candidate features\n    val authorIdToFeatureStitch = Stitch.collect {\n      candidateAuthors\n        .map { author =>\n          val isAuthorEnabledPreviews = fetcher.fetch(author).map {\n              _.v.map(_.previewsEnabled).getOrElse(DefaultAuthorEnabledPreviewsValue)\n          }\n          (author, isAuthorEnabledPreviews)\n        }.toMap\n    }\n\n    authorIdToFeatureStitch.map { authorIdToFeatureMap =>\n      candidates.map {\n        _.features.getOrElse(AuthorIdFeature, None) match {\n          case Some(authorId) => FeatureMapBuilder()\n            .add(AuthorEnabledPreviewsFeature, authorIdToFeatureMap(authorId))\n            .build()\n          case _ => FeatureMapBuilder()\n            .add(AuthorEnabledPreviewsFeature, DefaultAuthorEnabledPreviewsValue)\n            .build()\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"audience-rewards/thrift/src/main/thrift/common:thrift-scala\",\n        \"configapi/configapi-decider/src/main/scala/com/twitter/timelines/configapi/decider\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_visibility_reason\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"recruiting/candidate-service/src/main/thrift:thrift-scala\",\n        \"servo/repo/src/main/scala\",\n        \"stitch/stitch-timelineservice\",\n        \"strato/config/columns/audiencerewards/audienceRewardsService:audienceRewardsService-strato-client\",\n        \"strato/config/columns/interests:interests-strato-client\",\n        \"strato/config/columns/recruiting/api/user:user-strato-client\",\n        \"strato/config/columns/socialgraph/service:exists-strato-client\",\n        \"strato/config/columns/timelines/pintweet:pintweet-strato-client\",\n        \"strato/config/columns/tweetypie/federated:federated-strato-client\",\n        \"strato/config/columns/tweetypie/managed:managed-strato-client\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/CurrentPinnedTweetFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.for_you.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.CurrentPinnedTweetFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.timelines.pintweet.PinTweetFannedOutUserCacheClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CurrentPinnedTweetFeatureHydrator @Inject() (\n  pinTweetFannedOutUserCacheClientColumn: PinTweetFannedOutUserCacheClientColumn)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"CurrentPinnedTweet\")\n\n  override val features: Set[Feature[_, _]] = Set(CurrentPinnedTweetFeature)\n\n  private val fetcher: Fetcher[\n    PinTweetFannedOutUserCacheClientColumn.Key,\n    PinTweetFannedOutUserCacheClientColumn.View,\n    PinTweetFannedOutUserCacheClientColumn.Value\n  ] = pinTweetFannedOutUserCacheClientColumn.fetcher\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val authorIds = candidates.flatMap(_.features.getOrElse(AuthorIdFeature, None)).distinct\n\n    Stitch\n      .collectToTry {\n        authorIds.map { authorId =>\n          fetcher.fetch(authorId).map(_.v.map(data => authorId -> data.tweetId))\n        }\n      }.map { results =>\n        val authorPinTweetMap = results.flatMap(_.toOption).flatten.toMap\n        candidates.map { candidate =>\n          val authorId = candidate.features.getOrElse(AuthorIdFeature, None).getOrElse(0L)\n          val pinnedTweet = authorPinTweetMap.get(authorId)\n          FeatureMap(CurrentPinnedTweetFeature, pinnedTweet)\n        }\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/DisplayedGrokTopicQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.for_you.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.CurrentDisplayedGrokTopicFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.interests.FollowedGrokTopicsWithNameOnUserClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.util.Random\n\n@Singleton\nclass DisplayedGrokTopicQueryFeatureHydrator @Inject() (\n  followedGrokTopicsWithNameOnUserClientColumn: FollowedGrokTopicsWithNameOnUserClientColumn)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"FollowedGrokTopics\")\n\n  private val fetcher = followedGrokTopicsWithNameOnUserClientColumn.fetcher\n\n  override def features: Set[Feature[_, _]] =\n    Set(CurrentDisplayedGrokTopicFeature)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    fetcher\n      .fetch(query.getRequiredUserId)\n      .map { result =>\n        val topicResult = result.v.map(_.toSeq).getOrElse(Seq.empty)\n        val randomTopic = Random.shuffle(topicResult).take(1).headOption\n        FeatureMap(CurrentDisplayedGrokTopicFeature, randomTopic)\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/FocalTweetFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.for_you.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ConversationModuleIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FocalTweetAuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FocalTweetInNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FocalTweetRealNamesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FocalTweetScreenNamesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RealNamesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScreenNamesFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Social context for convo modules is hydrated on the root Tweet but needs info about the focal\n * Tweet (e.g. author) to render the banner. This hydrator copies focal Tweet data into the root.\n */\n@Singleton\nclass FocalTweetFeatureHydrator @Inject() ()\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"FocalTweet\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    FocalTweetAuthorIdFeature,\n    FocalTweetInNetworkFeature,\n    FocalTweetRealNamesFeature,\n    FocalTweetScreenNamesFeature\n  )\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(FocalTweetAuthorIdFeature, None)\n    .add(FocalTweetInNetworkFeature, None)\n    .add(FocalTweetRealNamesFeature, None)\n    .add(FocalTweetScreenNamesFeature, None)\n    .build()\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    // Build a map of all the focal tweets to their corresponding features\n    val focalTweetIdToFeatureMap = candidates.flatMap { candidate =>\n      val focalTweetId = candidate.features.getOrElse(ConversationModuleFocalTweetIdFeature, None)\n      if (focalTweetId.contains(candidate.candidate.id)) {\n        Some(candidate.candidate.id -> candidate.features)\n      } else None\n    }.toMap\n\n    val updatedFeatureMap = candidates.map { candidate =>\n      val focalTweetId = candidate.features.getOrElse(ConversationModuleFocalTweetIdFeature, None)\n      val conversationId = candidate.features.getOrElse(ConversationModuleIdFeature, None)\n\n      // Check if the candidate is a root tweet and ensure its focal tweet's features are available\n      if (conversationId.contains(candidate.candidate.id)\n        && focalTweetId.exists(focalTweetIdToFeatureMap.contains)) {\n        val featureMap = focalTweetIdToFeatureMap.get(focalTweetId.get).get\n        FeatureMapBuilder()\n          .add(FocalTweetAuthorIdFeature, featureMap.getOrElse(AuthorIdFeature, None))\n          .add(FocalTweetInNetworkFeature, Some(featureMap.getOrElse(InNetworkFeature, true)))\n          .add(\n            FocalTweetRealNamesFeature,\n            Some(featureMap.getOrElse(RealNamesFeature, Map.empty[Long, String])))\n          .add(\n            FocalTweetScreenNamesFeature,\n            Some(featureMap.getOrElse(ScreenNamesFeature, Map.empty[Long, String])))\n          .build()\n      } else DefaultFeatureMap\n    }\n\n    Stitch.value(updatedFeatureMap)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/FollowingSportsAccountQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.for_you.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.FollowsSportsAccountFeature\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.FollowingSportsGateUsersParam\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.socialgraph.{thriftscala => sg}\nimport com.twitter.strato.generated.client.socialgraph.service.ExistsClientColumn\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FollowingSportsAccountsQueryFeatureHydrator @Inject() (\n  existsClientColumn: ExistsClientColumn)\n  extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"FollowingSportsAccounts\")\n\n  override val features: Set[Feature[_, _]] = Set(FollowsSportsAccountFeature)\n\n  val lookupContext = sg.LookupContext(includeInactive = false)\n\n  private val fetcher: Fetcher[\n    ExistsClientColumn.Key,\n    ExistsClientColumn.View,\n    ExistsClientColumn.Value\n  ] = existsClientColumn.fetcher\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    query.getOptionalUserId match {\n      case Some(userId) =>\n        Stitch\n          .traverse(query.params(FollowingSportsGateUsersParam).toSeq) { sportsAccountId =>\n            val request =\n              (userId, Seq(sg.Relationship(sg.RelationshipType.Following)), sportsAccountId)\n            fetcher\n              .fetch(request, lookupContext).map(_.v.getOrElse(false))\n          }.map { followingAccounts: Seq[Boolean] =>\n            FeatureMap(FollowsSportsAccountFeature, followingAccounts.contains(true))\n          }\n      case None => Stitch.value(FeatureMap(FollowsSportsAccountFeature, false))\n    }\n\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/TimelineServiceTweetsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.for_you.feature_hydrator\n\nimport com.twitter.home_mixer.marshaller.timelines.DeviceContextMarshaller\nimport com.twitter.home_mixer.model.HomeFeatures.TLSOriginalTweetsWithAuthorFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TimelineServiceTweetsFeature\nimport com.twitter.home_mixer.model.request.DeviceContext\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.{EnableGetTweetsFromArchiveIndex, EnableTLSHydrationParam}\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.timelineservice.TimelineService\nimport com.twitter.timelineservice.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class TimelineServiceTweetsQueryFeatureHydrator @Inject() (\n  timelineService: TimelineService,\n  deviceContextMarshaller: DeviceContextMarshaller)\n    extends QueryFeatureHydrator[PipelineQuery with HasDeviceContext] \n        with Conditionally[PipelineQuery with HasDeviceContext] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TimelineServiceTweets\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(TimelineServiceTweetsFeature, TLSOriginalTweetsWithAuthorFeature)\n\n  override def onlyIf(query: PipelineQuery with HasDeviceContext): Boolean = {\n    query.params(EnableTLSHydrationParam)\n  }\n\n  private[this] def getTweetsFromArchiveIndexOpt(\n    query: PipelineQuery with HasDeviceContext\n  ): Option[Boolean] = {\n    if (query.params(EnableGetTweetsFromArchiveIndex)) {\n      // Fall back to the default in timelineservice, i.e. get tweets from archive index.\n      None\n    } else {\n      Some(false)\n    }\n  }\n\n  private val MaxTimelineServiceTweets = 200\n\n  override def hydrate(query: PipelineQuery with HasDeviceContext): Stitch[FeatureMap] = {\n    val deviceContext = query.deviceContext.getOrElse(DeviceContext.Empty)\n\n    val timelineQueryOptions = t.TimelineQueryOptions(\n      contextualUserId = query.clientContext.userId,\n      deviceContext = Some(deviceContextMarshaller(deviceContext, query.clientContext)),\n      getTweetsFromArchiveIndex = getTweetsFromArchiveIndexOpt(query)\n    )\n\n    val timelineServiceQuery = t.TimelineQuery(\n      timelineType = t.TimelineType.Home,\n      timelineId = query.getRequiredUserId,\n      maxCount = MaxTimelineServiceTweets.toShort,\n      cursor2 = None,\n      options = Some(timelineQueryOptions),\n      timelineId2 = query.clientContext.userId.map(t.TimelineId(t.TimelineType.Home, _, None)),\n    )\n\n    timelineService.getTimeline(timelineServiceQuery).map { timeline =>\n      val tweets = timeline.entries.collect {\n        case t.TimelineEntry.Tweet(tweet) => tweet.statusId\n      }\n\n      // Non replies and non retweets\n      val originalTweetsWithAuthor = timeline.entries.collect {\n        case t.TimelineEntry.Tweet(tweet)\n            if tweet.inReplyToStatusId.isEmpty\n              && tweet.sourceStatusId.isEmpty =>\n          (tweet.statusId, tweet.userId)\n      }\n\n      FeatureMapBuilder()\n        .add(TimelineServiceTweetsFeature, tweets)\n        .add(TLSOriginalTweetsWithAuthorFeature, originalTweetsWithAuthor)\n        .build()\n    }\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(95)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/TweetAuthorFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.for_you.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.{\n  TLSOriginalTweetsWithAuthorFeature,\n  TLSOriginalTweetsWithConfirmedAuthorFeature\n}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.{FeatureMap, FeatureMapBuilder}\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.spam.rtf.{thriftscala => rtf}\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.tweetypie.managed.HomeMixerTesOnTweetClientColumn\nimport com.twitter.strato.client.Client\nimport com.twitter.tweetypie.{thriftscala => tp}\nimport javax.inject.{Inject, Singleton}\nimport com.twitter.util.logging.Logging\nimport javax.inject.Named\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithLongTimeout\n\n@Singleton\nclass TweetAuthorFeatureHydrator @Inject() (\n  @Named(BatchedStratoClientWithLongTimeout) stratoClient: Client,\n  stats: StatsReceiver)\n  extends QueryFeatureHydrator[PipelineQuery]\n  with Logging {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetAuthor\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(TLSOriginalTweetsWithConfirmedAuthorFeature)\n\n  private def fetchTweetDataFromTES(tweetId: Long, tweetFieldsOptions: tp.GetTweetFieldsOptions): Stitch[Option[(Long, Long)]] = {\n    val fetcher = new HomeMixerTesOnTweetClientColumn(stratoClient).fetcher\n    fetcher\n      .fetch(tweetId, tweetFieldsOptions)\n      .map(_.v)\n      .map {\n        case Some(result) =>\n          result.tweetResult match {\n            case tp.TweetFieldsResultState.Found(tweetResult) =>\n              val isRetweet = tweetResult.retweetedTweet.isDefined\n              tweetResult.tweet.coreData.flatMap { coreData =>\n                val isReply = coreData.reply.isDefined\n                if (!isRetweet && !isReply) Some(tweetId -> coreData.userId) else None\n              }\n            case _ => None\n          }\n        case None => None\n      }\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val originals: Seq[(Long, Option[Long])] = query.features.map(_.getOrElse(TLSOriginalTweetsWithAuthorFeature, Seq())).getOrElse(Seq.empty)\n\n    if (originals.isEmpty)\n      return Stitch.value(\n        FeatureMapBuilder()\n          .add(TLSOriginalTweetsWithConfirmedAuthorFeature, Seq.empty[(Long, Long)])\n          .build()\n      )\n\n    val missingIds = originals.collect { case (tid, None) => tid }\n\n    val tweetFieldsOptions = tp.GetTweetFieldsOptions(\n      tweetIncludes = Set(tp.TweetInclude.TweetFieldId(tp.Tweet.CoreDataField.id), tp.TweetInclude.TweetFieldId(tp.Tweet.CountsField.id)),\n      safetyLevel = Some(rtf.SafetyLevel.TimelineHome),\n      forUserId = query.getOptionalUserId\n    )\n\n    val fetchedTweetData: Stitch[Map[Long, Long]] =\n      Stitch\n        .collect(missingIds.map { id =>\n          fetchTweetDataFromTES(id, tweetFieldsOptions)\n        })\n        .map(_.flatten.toMap)\n\n    fetchedTweetData.map { found =>\n      val rebuilt: Seq[(Long, Long)] =\n        originals.collect {\n          case (tid, authorOpt) =>\n            authorOpt.orElse(found.get(tid)).map(aid => (tid, aid))\n        }.flatten\n\n      FeatureMapBuilder()\n        .add(TLSOriginalTweetsWithConfirmedAuthorFeature, rebuilt)\n        .build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/TweetAuthorFollowersFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.for_you.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.gizmoduck.{thriftscala => gt}\nimport com.twitter.home_mixer.model.HomeFeatures.{\n  TLSOriginalTweetsWithConfirmedAuthorFeature,\n  TweetAuthorFollowersFeature\n}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.{FeatureMap, FeatureMapBuilder}\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport javax.inject.{Inject, Singleton}\nimport com.twitter.util.logging.Logging\n\n@Singleton\nclass TweetAuthorFollowersFeatureHydrator @Inject() (\n  gizmoduck: gt.UserService.MethodPerEndpoint,\n  stats: StatsReceiver)\n  extends QueryFeatureHydrator[PipelineQuery]\n  with Logging {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetAuthorFollowers\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(TweetAuthorFollowersFeature)\n\n  private val queryFields: Set[gt.QueryFields] = Set(\n    gt.QueryFields.Counts\n  )\n\n  private val lookupContext = gt.LookupContext(isRequestSheddable = Some(true))\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val originals: Seq[(Long, Long)] = query.features.map(_.getOrElse(TLSOriginalTweetsWithConfirmedAuthorFeature, Seq())).getOrElse(Seq.empty)\n\n    if (originals.isEmpty) {\n      return Stitch.value(\n        FeatureMapBuilder()\n          .add(TweetAuthorFollowersFeature, Map.empty[Long, Option[Long]])\n          .build()\n      )\n    }\n\n    val authorIds = originals.map(_._2).distinct\n\n    OffloadFuturePools.offloadFuture {\n      gizmoduck.get(lookupContext, authorIds, queryFields).map { gizmoduckResponse =>\n        val hydratedUsersMap = gizmoduckResponse.collect {\n          case userResult if userResult.user.isDefined =>\n            val user = userResult.user.get\n            user.id -> user\n        }.toMap.mapValues(user => user.counts.map(_.followers))\n\n        val tweetAuthorFollowersMap = originals.map { case (tweetId, authorId) =>\n          tweetId -> hydratedUsersMap.getOrElse(authorId, None)\n        }.toMap\n\n        FeatureMapBuilder()\n          .add(TweetAuthorFollowersFeature, tweetAuthorFollowersMap)\n          .build()\n      }\n    }\n  }\n} "
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/TweetEngagementsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.for_you.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.TimelineServiceTweetsFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.{FeatureMap, FeatureMapBuilder}\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.tweetypie.federated.ApiCountsOnTweetClientColumn\nimport com.twitter.strato.client.Client\nimport javax.inject.{Inject, Singleton}\nimport com.twitter.util.logging.Logging\nimport javax.inject.Named\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithLongTimeout\n\ncase class TweetEngagementCounts(\n  favoriteCount: Option[Long],\n  replyCount: Option[Long],\n  retweetCount: Option[Long],\n  quoteCount: Option[Long],\n  bookmarkCount: Option[Long]\n)\n\nobject TweetEngagementCountsFeature extends Feature[PipelineQuery, Map[Long, TweetEngagementCounts]]\n\n\n@Singleton\nclass TweetEngagementsFeatureHydrator @Inject() (\n  @Named(BatchedStratoClientWithLongTimeout) stratoClient: Client,\n  stats: StatsReceiver)\n  extends QueryFeatureHydrator[PipelineQuery]\n  with Logging {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetEngagements\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(TweetEngagementCountsFeature)\n\n  private val engagementCountsFoundCounter =\n    stats.counter(\"TweetEngagementsFound\")\n  private val engagementCountsNotFoundCounter =\n    stats.counter(\"TweetEngagementsNotFound\")\n\n  private def fetchEngagementCountsFromStrato(tweetId: Long): Stitch[Option[TweetEngagementCounts]] = {\n    val fetcher = new ApiCountsOnTweetClientColumn(stratoClient).fetcher\n    fetcher\n      .fetch(tweetId)\n      .map(_.v)\n      .map {\n        case Some(result) =>\n          engagementCountsFoundCounter.incr()\n          Some(TweetEngagementCounts(\n            favoriteCount = result.favoriteCount,\n            replyCount = result.replyCount,\n            retweetCount = result.retweetCount,\n            quoteCount = result.quoteCount,\n            bookmarkCount = result.bookmarkCount\n          ))\n        case None =>\n          engagementCountsNotFoundCounter.incr()\n          None\n      }\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val tweetIds: Seq[Long] = query.features.map(_.getOrElse(TimelineServiceTweetsFeature, Seq.empty[Long])).getOrElse(Seq.empty)\n\n    if (tweetIds.isEmpty)\n      return Stitch.value(\n        FeatureMapBuilder()\n          .add(TweetEngagementCountsFeature, Map.empty[Long, TweetEngagementCounts])\n          .build()\n      )\n\n    val fetchedEngagementCounts: Stitch[Map[Long, TweetEngagementCounts]] =\n      Stitch\n        .collect(tweetIds.map { id =>\n          fetchEngagementCountsFromStrato(id).map(_.map(counts => id -> counts))\n        })\n        .map(_.flatten.toMap)\n\n    fetchedEngagementCounts.map { found =>\n      FeatureMapBuilder()\n        .add(TweetEngagementCountsFeature, found)\n        .build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/TweetPreviewTweetypieCandidateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.for_you.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.ArticleIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTextFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableTweetEntityServiceMigrationParam\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithLongTimeout\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_visibility_reason.VisibilityReason\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.spam.rtf.{thriftscala => rtf}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.generated.client.tweetypie.managed.HomeMixerOnTweetClientColumn\nimport com.twitter.tweetypie.{thriftscala => TP}\nimport javax.inject.Named\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\n\n@Singleton\nclass TweetPreviewTweetypieCandidateFeatureHydrator @Inject() (\n  statsReceiver: StatsReceiver,\n  tweetypieStitchClient: TweetypieStitchClient,\n  @Named(BatchedStratoClientWithLongTimeout) stratoClient: Client)\n    extends CandidateFeatureHydrator[PipelineQuery, BaseTweetCandidate] {\n\n  private val tweetypieTweetsFoundCounter =\n    statsReceiver.counter(\"TweetPreviewTweetypieTweetsFound\")\n  private val tweetypieTweetsNotFoundCounter =\n    statsReceiver.counter(\"TweetPreviewTweetypieTweetsNotFound\")\n  private val tesTweetsFoundCounter =\n    statsReceiver.counter(\"TweetPreviewTesTweetsFound\")\n  private val tesTweetsNotFoundCounter =\n    statsReceiver.counter(\"TweetPreviewTesTweetsNotFound\")\n\n  private val CoreTweetFields: Set[TP.TweetInclude] = Set[TP.TweetInclude](\n    TP.TweetInclude.TweetFieldId(TP.Tweet.IdField.id),\n    TP.TweetInclude.TweetFieldId(TP.Tweet.CoreDataField.id),\n    TP.TweetInclude.TweetFieldId(TP.Tweet.ArticleField.id),\n  )\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(TweetTextFeature, None)\n    .add(IsHydratedFeature, false)\n    .add(AuthorIdFeature, None)\n    .add(VisibilityReason, None)\n    .add(ArticleIdFeature, None)\n    .build()\n\n  override val features: Set[Feature[_, _]] =\n    Set(TweetTextFeature, IsHydratedFeature, AuthorIdFeature, VisibilityReason, ArticleIdFeature)\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetPreviewTweetypie\")\n\n  private def buildFeatureMap(\n    gtfResult: Stitch[TP.GetTweetFieldsResult],\n    isTes: Boolean\n  ): Stitch[FeatureMap] = {\n    gtfResult.map {\n      case TP.GetTweetFieldsResult(_, TP.TweetFieldsResultState.Found(found), _, _) =>\n        if (isTes) tesTweetsFoundCounter.incr()\n        else tweetypieTweetsFoundCounter.incr()\n\n        val tweetText = found.tweet.coreData.map(_.text)\n        FeatureMapBuilder(sizeHint = 5)\n          .add(TweetTextFeature, tweetText)\n          .add(IsHydratedFeature, true)\n          .add(AuthorIdFeature, found.tweet.coreData.map(_.userId))\n          .add(VisibilityReason, found.suppressReason)\n          .add(ArticleIdFeature, found.tweet.article.map(_.id))\n          .build()\n      case _ =>\n        if (isTes) tesTweetsNotFoundCounter.incr()\n        else tweetypieTweetsNotFoundCounter.incr()\n        DefaultFeatureMap\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: BaseTweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    val getTweetFieldsOptions = TP.GetTweetFieldsOptions(\n      tweetIncludes = CoreTweetFields,\n      includeRetweetedTweet = false,\n      includeQuotedTweet = false,\n      visibilityPolicy = TP.TweetVisibilityPolicy.UserVisible,\n      safetyLevel = Some(rtf.SafetyLevel.TimelineHomeTweetPreview),\n      forUserId = query.getOptionalUserId\n    )\n\n    if (query.params(EnableTweetEntityServiceMigrationParam)) {\n      val fetcher = new HomeMixerOnTweetClientColumn(stratoClient).fetcher\n      fetcher\n        .fetch(\n          candidate.id,\n          getTweetFieldsOptions\n        ).map(_.v).flatMap {\n          case Some(result) => buildFeatureMap(Stitch.value(result), true)\n          case None =>\n            tesTweetsNotFoundCounter.incr()\n            Stitch.value(DefaultFeatureMap)\n        }\n    } else {\n      val gtfResult: Stitch[TP.GetTweetFieldsResult] = tweetypieStitchClient\n        .getTweetFields(\n          tweetId = candidate.id,\n          options = getTweetFieldsOptions\n        )\n      buildFeatureMap(gtfResult, false)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/feature_hydrator/ViewerHasJobRecommendationsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.for_you.feature_hydrator\n\nimport com.twitter.candidateservice.core_io.MatchingProfile\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerHasJobRecommendationsEnabled\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerHasRecruitingOrganizationRecommendationsEnabled\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithDefaultTimeout\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableViewerHasJobRecommendationsFeatureParam\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoErrCategorizer\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.generated.client.recruiting.api.user.MatchingProfileOnUserClientColumn\n\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass ViewerHasJobRecommendationsFeatureHydrator @Inject() (\n  @Named(BatchedStratoClientWithDefaultTimeout) stratoClient: Client)\n    extends QueryFeatureHydrator[PipelineQuery]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"ViewerHasJobRecommendations\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(ViewerHasJobRecommendationsEnabled, ViewerHasRecruitingOrganizationRecommendationsEnabled)\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableViewerHasJobRecommendationsFeatureParam)\n\n  private val fetcher = stratoClient.fetcher[\n    Long,\n    Unit,\n    MatchingProfile\n  ](MatchingProfileOnUserClientColumn.Path)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n    fetcher\n      .fetch(userId, ())\n      .map { result =>\n        val (orgsEnabled, jobsEnabled) = result.v match {\n          case None =>\n            // When no matching profile exists, the user cannot have consented\n            // to see job recommendations. Just show org recommendations.\n            (true, false)\n          case Some(profile) =>\n            // When a profile does exist, check for consent. Consent cannot be withdrawn.\n            // It indicates that at some point in the past, the user has opted in to see\n            // job recommendations. They can later toggle off recommendations. Possible\n            // states:\n            //\n            // 1. User consented and recommendations enabled\n            //    (a) there are recommendations --> show job recommendations\n            //    (b) there are no recommendations --> show org recommendations\n            // 2. User toggled off recommendations --> show nothing\n            // 3. User has not consented --> show org recommendations\n            if (!profile.recommendationsEnabled) {\n              (false, false)\n            } else {\n              val showJobRecommendations =\n                profile.consentedAt.nonEmpty && profile.hasJobRecommendations\n              (!showJobRecommendations, showJobRecommendations)\n            }\n        }\n\n        FeatureMapBuilder()\n          .add(ViewerHasRecruitingOrganizationRecommendationsEnabled, orgsEnabled)\n          .add(ViewerHasJobRecommendationsEnabled, jobsEnabled)\n          .build()\n      }\n      .rescue(StratoErrCategorizer.CategorizeStratoException)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/filter/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/trends_events\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/filter/NotArticleFilter.scala",
    "content": "package com.twitter.home_mixer.product.for_you.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.IsArticleFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Filter out tweets that contain articles\n */\nobject NotArticleFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"NotArticle\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (kept, removed) = candidates.partition { candidate =>\n      !candidate.features.getOrElse(IsArticleFeature, false)\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/filter/PromotedTrendFilter.scala",
    "content": "package com.twitter.home_mixer.product.for_you.filter\n\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.PromotedTrendNameFeature\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.UnifiedTrendCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * A filter that drops promoted trends\n */\nobject PromotedTrendFilter extends Filter[PipelineQuery, UnifiedTrendCandidate] {\n  override val identifier = FilterIdentifier(\"PromotedTrend\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[UnifiedTrendCandidate]]\n  ): Stitch[FilterResult[UnifiedTrendCandidate]] = {\n    val filterResult = {\n\n      val (removed, kept) = candidates.partition { candidate =>\n        candidate.features.get(PromotedTrendNameFeature).isDefined\n      }\n      FilterResult(kept.map(_.candidate), removed.map(_.candidate))\n    }\n    Stitch.value(filterResult)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/filter/SocialContextFilter.scala",
    "content": "package com.twitter.home_mixer.product.for_you.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\n\nobject SocialContextFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"SocialContext\")\n\n  // Tweets from candidate sources which don't need generic like/follow/topic proof\n  private val AllowedSources: Set[st.SuggestType] = Set(\n    st.SuggestType.RankedListTweet,\n    st.SuggestType.RecommendedTrendTweet,\n    st.SuggestType.MediaTweet\n  )\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val enableIsVerifiedAuthorFilter =\n      query.params(ForYouParam.EnableVerifiedAuthorSocialContextBypassParam)\n\n    val enableTopicSocialContextFilter =\n      query.params(ForYouParam.EnableTopicSocialContextFilterParam)\n\n    val validTweetIds = candidates\n      .filter { candidate =>\n        candidate.features.getOrElse(InNetworkFeature, true) ||\n        candidate.features.getOrElse(SuggestTypeFeature, None).exists(AllowedSources.contains) ||\n        candidate.features.getOrElse(ConversationModuleFocalTweetIdFeature, None).isDefined ||\n        (enableIsVerifiedAuthorFilter && isVerifiedAuthor(candidate.features)) ||\n        hasLikedBySocialContext(candidate.features) ||\n        hasFollowedBySocialContext(candidate.features) ||\n        (enableTopicSocialContextFilter && hasTopicSocialContext(candidate.features))\n      }.map(_.candidate.id).toSet\n\n    val (kept, removed) =\n      candidates.map(_.candidate).partition(candidate => validTweetIds.contains(candidate.id))\n\n    Stitch.value(FilterResult(kept = kept, removed = removed))\n  }\n\n  private def isVerifiedAuthor(candidateFeatures: FeatureMap): Boolean = {\n    candidateFeatures.getOrElse(AuthorIsBlueVerifiedFeature, false) ||\n    candidateFeatures.getOrElse(AuthorIsGoldVerifiedFeature, false) ||\n    candidateFeatures.getOrElse(AuthorIsGrayVerifiedFeature, false) ||\n    candidateFeatures.getOrElse(AuthorIsLegacyVerifiedFeature, false)\n  }\n\n  private def hasLikedBySocialContext(candidateFeatures: FeatureMap): Boolean =\n    candidateFeatures\n      .getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty)\n      .exists(\n        candidateFeatures\n          .getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Seq.empty)\n          .toSet.contains\n      )\n\n  private def hasFollowedBySocialContext(candidateFeatures: FeatureMap): Boolean =\n    candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Seq.empty).nonEmpty\n\n  private def hasTopicSocialContext(candidateFeatures: FeatureMap): Boolean = {\n    candidateFeatures.getOrElse(TopicIdSocialContextFeature, None).isDefined &&\n    candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None).isDefined\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/filter/TweetPreviewTextFilter.scala",
    "content": "package com.twitter.home_mixer.product.for_you.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.ArticlePreviewTextFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTextFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject TweetPreviewTextFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"TweetPreviewText\")\n\n  private val PreviewTextLength = 50\n  private val MinTweetLength = PreviewTextLength * 2\n  private val MaxNewlines = 2\n  private val HttpPrefix = \"http://\"\n  private val HttpsPrefix = \"https://\"\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val (kept, removed) = candidates\n      .partition { candidate =>\n        val text = candidate.features\n          .get(TweetTextFeature).orElse(\n            candidate.features.getOrElse(ArticlePreviewTextFeature, None)).getOrElse(\"\")\n\n        text.length > MinTweetLength &&\n        text.take(PreviewTextLength).count(_ == '\\n') <= MaxNewlines &&\n        !(text.startsWith(HttpPrefix) || text.startsWith(HttpsPrefix))\n      }\n\n    val filterResult = FilterResult(\n      kept = kept.map(_.candidate),\n      removed = removed.map(_.candidate)\n    )\n\n    Stitch.value(filterResult)\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/functional_component/gate/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/model\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/functional_component/gate/PushToHomeRequestGate.scala",
    "content": "package com.twitter.home_mixer.product.for_you.functional_component.gate\n\nimport com.twitter.home_mixer.product.for_you.model.ForYouQuery\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.stitch.Stitch\n\n/**\n * Continues when the request is a Push-To-Home notification request\n */\nobject PushToHomeRequestGate extends Gate[ForYouQuery] {\n  override val identifier: GateIdentifier = GateIdentifier(\"PushToHomeRequest\")\n\n  override def shouldContinue(query: ForYouQuery): Stitch[Boolean] =\n    Stitch.value(query.pushToHomeTweetId.isDefined)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/gate/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/param\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/gate/FollowingSportsUsersGate.scala",
    "content": "package com.twitter.home_mixer.product.for_you.gate\n\nimport com.twitter.home_mixer.model.HomeFeatures.FollowsSportsAccountFeature\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Continue for all users that follow a sports account\n */\nobject FollowingSportsUserGate extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"FollowingSportsUser\")\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val followsSportsAccount = query.features.map(_.getOrElse(FollowsSportsAccountFeature, false))\n    Stitch.value(followsSportsAccount.contains(true))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/gate/TuneFeedModuleGate.scala",
    "content": "package com.twitter.home_mixer.product.for_you.gate\n\nimport com.twitter.home_mixer.model.HomeFeatures.CurrentDisplayedGrokTopicFeature\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject TuneFeedModuleGate extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"TuneFeed\")\n\n  override def shouldContinue(\n    query: PipelineQuery\n  ): Stitch[Boolean] = {\n    val grokTopicToDisplay =\n      query.features.get.getOrElse(CurrentDisplayedGrokTopicFeature, None)\n\n    Stitch.value(grokTopicToDisplay.isDefined)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/gate/UserFollowingRangeGate.scala",
    "content": "package com.twitter.home_mixer.product.for_you.gate\n\nimport com.twitter.home_mixer.model.HomeFeatures.UserFollowingCountFeature\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.{MinFollowingCountParam, MaxFollowingCountParam}\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Gate that checks if the user's following count is within a valid range.\n * The range is controlled by configurable parameters for lower and upper bounds.\n */\nobject UserFollowingRangeGate extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"UserFollowingRange\")\n\n  override def shouldContinue(\n    query: PipelineQuery\n  ): Stitch[Boolean] = {\n    val userFollowingCount = query.features.get.getOrElse(UserFollowingCountFeature, None)\n    val lowerBound = query.params(MinFollowingCountParam)\n    val upperBound = query.params(MaxFollowingCountParam)\n\n    val isInRange = userFollowingCount match {\n      case Some(count) => count >= lowerBound && count <= upperBound\n      case None => false // If we don't have the following count, don't continue\n    }\n\n    Stitch.value(isInRange)\n  }\n} "
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/model/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/model/ForYouQuery.scala",
    "content": "package com.twitter.home_mixer.product.for_you.model\n\nimport com.twitter.adserver.thriftscala.HomeTimelineType\nimport com.twitter.adserver.thriftscala.TimelineRequestParams\nimport com.twitter.dspbidder.commons.{thriftscala => dsp}\nimport com.twitter.home_mixer.model.HomeAdsQuery\nimport com.twitter.home_mixer.model.request.DeviceContext\nimport com.twitter.home_mixer.model.request.ForYouProduct\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.model.request.HasSeenTweetIds\nimport com.twitter.onboarding.task.service.{thriftscala => ots}\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.HasFlipInjectionParams\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.request._\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Params\n\ncase class ForYouQuery(\n  override val params: Params,\n  override val clientContext: ClientContext,\n  override val pipelineCursor: Option[UrtOrderedCursor],\n  override val requestedMaxResults: Option[Int],\n  override val debugOptions: Option[DebugOptions],\n  override val features: Option[FeatureMap],\n  override val deviceContext: Option[DeviceContext],\n  override val seenTweetIds: Option[Seq[Long]],\n  override val dspClientContext: Option[dsp.DspClientContext])\n    extends PipelineQuery\n    with HasPipelineCursor[UrtOrderedCursor]\n    with HasDeviceContext\n    with HasSeenTweetIds\n    with HasFlipInjectionParams\n    with HomeAdsQuery {\n  override val product: Product = ForYouProduct\n\n  override def withFeatureMap(features: FeatureMap): ForYouQuery =\n    copy(features = Some(features))\n\n  override val timelineRequestParams: Option[TimelineRequestParams] =\n    Some(TimelineRequestParams(homeTimelineType = Some(HomeTimelineType.Home)))\n\n  // Fields below are used for FLIP Injection in Onboarding Task Service (OTS)\n  override val displayLocation: ots.DisplayLocation = ots.DisplayLocation.HomeTimeline\n  override val rankingDisablerWithLatestControlsAvailable: Option[Boolean] = None\n  override val isEmptyState: Option[Boolean] = None\n  override val isFirstRequestAfterSignup: Option[Boolean] = None\n  override val isEndOfTimeline: Option[Boolean] = None\n  override val timelineId: Option[Long] = None\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/model/ForYouTweetsResponse.scala",
    "content": "package com.twitter.home_mixer.product.for_you.model\n\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\n\ncase class ForYouTweetsResponse(tweetCandidates: Seq[Long]) extends HasMarshalling\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/param/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param/decider\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/communities_to_join\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/param/ForYouParam.scala",
    "content": "package com.twitter.home_mixer.product.for_you.param\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.param.decider.DeciderKey\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.WhoToFollowModuleDisplayType\nimport com.twitter.product_mixer.component_library.pipeline.candidate.communities_to_join.CommunityToJoinModuleDisplayType\nimport com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module.WhoToFollowUserDisplayType\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.decider.BooleanDeciderParam\nimport com.twitter.util.Duration\n\nobject ForYouParam {\n  val SupportedClientFSName = \"for_you_supported_client\"\n  val StaticParamValueZero = StaticParam(0)\n  val StaticParamValueFive = StaticParam(5)\n\n  object EnableWhoToFollowCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_who_to_follow\",\n        default = true\n      )\n\n  object EnableWhoToSubscribeCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_who_to_subscribe\",\n        default = false\n      )\n\n  object EnableTweetPreviewsCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_tweet_previews_candidate_pipeline\",\n        default = false\n      )\n\n  object ServerMaxResultsParam\n      extends FSBoundedParam[Int](\n        name = \"for_you_server_max_results\",\n        default = 35,\n        min = 1,\n        max = 500\n      )\n\n  object AdsNumOrganicItemsParam\n      extends FSBoundedParam[Int](\n        name = \"for_you_ads_num_organic_items\",\n        default = 35,\n        min = 1,\n        max = 100\n      )\n\n  object WhoToFollowMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_who_to_follow_min_injection_interval_in_minutes\",\n        default = 1800.minutes,\n        min = 0.minutes,\n        max = 6000.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object WhoToFollowDisplayTypeIdParam\n      extends FSEnumParam[WhoToFollowModuleDisplayType.type](\n        name = \"for_you_enable_who_to_follow_display_type_id\",\n        default = WhoToFollowModuleDisplayType.Vertical,\n        enum = WhoToFollowModuleDisplayType\n      )\n\n  object WhoToFollowUserDisplayTypeIdParam\n      extends FSEnumParam[WhoToFollowUserDisplayType.type](\n        name = \"for_you_enable_who_to_follow_user_display_type_id\",\n        default = WhoToFollowUserDisplayType.User,\n        enum = WhoToFollowUserDisplayType\n      )\n\n  object WhoToFollowDisplayLocationParam\n      extends FSParam[String](\n        name = \"for_you_who_to_follow_display_location\",\n        default = \"timeline\"\n      )\n\n  object WhoToSubscribeMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_who_to_subscribe_min_injection_interval_in_minutes\",\n        default = 1800.minutes,\n        min = 0.minutes,\n        max = 6000.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object WhoToSubscribeDisplayTypeIdParam\n      extends FSEnumParam[WhoToFollowModuleDisplayType.type](\n        name = \"for_you_enable_who_to_subscribe_display_type_id\",\n        default = WhoToFollowModuleDisplayType.Vertical,\n        enum = WhoToFollowModuleDisplayType\n      )\n\n  object TweetPreviewsMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_tweet_previews_min_injection_interval_in_minutes\",\n        default = 2.hours,\n        min = 0.minutes,\n        max = 600.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object TweetPreviewsMaxCandidatesParam\n      extends FSBoundedParam[Int](\n        name = \"for_you_tweet_previews_max_candidates\",\n        default = 1,\n        min = 0,\n        max = 1\n      )\n\n  object EnableFlipInjectionModuleCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_flip_inline_injection_module\",\n        default = true\n      )\n\n  object ClearCache {\n    object PtrEnableParam\n        extends FSParam[Boolean](\n          name = \"for_you_clear_cache_ptr_enable\",\n          default = false\n        )\n\n    object ColdStartEnableParam\n        extends FSParam[Boolean](\n          name = \"for_you_clear_cache_cold_start_enable\",\n          default = false\n        )\n\n    object WarmStartEnableParam\n        extends FSParam[Boolean](\n          name = \"for_you_clear_cache_warm_start_enable\",\n          default = false\n        )\n\n    object ManualRefreshEnableParam\n        extends FSParam[Boolean](\n          name = \"for_you_clear_cache_manual_refresh_enable\",\n          default = false\n        )\n\n    object NavigateEnableParam\n        extends FSParam[Boolean](\n          name = \"for_you_clear_cache_navigate_enable\",\n          default = false\n        )\n\n    object ColdStartRetainViewportParam\n        extends FSParam[Boolean](\n          name = \"for_you_clear_cache_retain_viewport\",\n          default = false\n        )\n\n    case object MinEntriesParam\n        extends FSBoundedParam[Int](\n          name = \"for_you_clear_cache_min_entries\",\n          default = 10,\n          min = 0,\n          max = 35\n        )\n  }\n\n  object Navigation {\n    object PtrEnableParam\n        extends FSParam[Boolean](\n          name = \"for_you_navigation_ptr_enable\",\n          default = false\n        )\n\n    object ColdStartEnableParam\n        extends FSParam[Boolean](\n          name = \"for_you_navigation_cold_start_enable\",\n          default = false\n        )\n\n    object WarmStartEnableParam\n        extends FSParam[Boolean](\n          name = \"for_you_navigation_warm_start_enable\",\n          default = false\n        )\n\n    object ManualRefreshEnableParam\n        extends FSParam[Boolean](\n          name = \"for_you_navigation_manual_refresh_enable\",\n          default = false\n        )\n\n    object NavigateEnableParam\n        extends FSParam[Boolean](\n          name = \"for_you_navigation_navigate_enable\",\n          default = false\n        )\n  }\n\n  /**\n   * This author ID list is used purely for realtime metrics collection around how often we\n   * are serving posts from these authors and which sources they are coming from.\n   */\n  object AuthorListForStatsParam\n      extends FSParam[Set[Long]](\n        name = \"for_you_author_list_for_stats\",\n        default = Set.empty\n      )\n\n  object ExperimentStatsParam\n      extends FSParam[String](\n        name = \"for_you_experiment_stats\",\n        default = \"\"\n      )\n\n  object CommunitiesToJoinMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_communities_to_join_min_injection_interval_in_minutes\",\n        default = 2100.minutes,\n        min = 0.minutes,\n        max = 6000.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object MaxCommunitiesToJoinCandidatesParam\n      extends FSBoundedParam[Int](\n        name = \"for_you_communities_to_join_max_candidates\",\n        default = 3,\n        min = 1,\n        max = 10)\n\n  object CommunitiesToJoinDisplayTypeIdParam\n      extends FSEnumParam[CommunityToJoinModuleDisplayType.type](\n        name = \"for_you_communities_to_join_display_type_id\",\n        default = CommunityToJoinModuleDisplayType.Carousel,\n        enum = CommunityToJoinModuleDisplayType\n      )\n\n  object EnableCommunitiesToJoinCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_communities_to_join\",\n        default = false\n      )\n\n  object RecommendedJobMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_recommended_job_min_injection_interval_in_minutes\",\n        default = 2100.minutes,\n        min = 0.minutes,\n        max = 6000.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object MaxRecommendedJobCandidatesParam\n      extends FSBoundedParam[Int](\n        name = \"for_you_recommended_job_max_candidates\",\n        default = 3,\n        min = 1,\n        max = 10)\n\n  object EnableRecommendedJobsParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_recommended_jobs\",\n        default = false\n      )\n\n  object EnableRecommendedRecruitingOrganizationsParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_recommended_recruiting_organizations\",\n        default = false\n      )\n\n  object RecommendedRecruitingOrganizationMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_recommended_recruiting_organization_min_injection_interval_in_minutes\",\n        default = 2100.minutes,\n        min = 0.minutes,\n        max = 6000.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object MaxRecommendedRecruitingOrganizationCandidatesParam\n      extends FSBoundedParam[Int](\n        name = \"for_you_recommended_recruiting_organization_max_candidates\",\n        default = 3,\n        min = 1,\n        max = 10\n      )\n\n  object EnableBookmarksCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_bookmarks_module\",\n        default = false\n      )\n\n  object EnablePinnedTweetsCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_pinned_tweets_pipeline\",\n        default = false\n      )\n\n  object EnableEntryPointPivotParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_entry_point_pivot\",\n        default = false\n      )\n\n  object EnableGrokEntryPointPivotParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_grok_entry_point_pivot\",\n        default = false\n      )\n\n  object PinnedTweetsModuleMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_pinned_tweets_module_min_injection_interval_in_minutes\",\n        default = 1440.minutes, // 1 day\n        min = 0.minutes,\n        max = 6000.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object EnableExplorationTweetsCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_exploration_tweets_pipeline\",\n        default = false\n      )\n\n  object EnableJetfuelFramePipelineParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_jetfuel_frame_pipeline\",\n        default = false\n      )\n\n  object FollowingSportsGateUsersParam\n      extends FSParam[Set[Long]](\n        name = \"for_you_following_sports_users_list\",\n        default = Set.empty\n      )\n\n  object ExplorationTweetsTimelinePosition\n      extends FSBoundedParam[Int](\n        name = \"for_you_exploration_tweets_position\",\n        default = 15,\n        min = 0,\n        max = 50\n      )\n\n  object SuperbowlModuleTimelinePosition\n      extends FSBoundedParam[Int](\n        name = \"for_you_superbowl_position\",\n        default = 3,\n        min = 0,\n        max = 50\n      )\n\n  object GrokPivotModuleTimelinePosition\n      extends FSBoundedParam[Int](\n        name = \"for_you_grok_pivot_position\",\n        default = 3,\n        min = 0,\n        max = 50\n      )\n\n  object EntryPointPivotMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_entry_point_pivot_min_injection_interval_in_minutes\",\n        default = 30.minutes,\n        min = 0.minutes,\n        max = 6000.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object GrokEntryPointPivotMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_grok_entry_point_pivot_min_injection_interval_in_minutes\",\n        default = 30.minutes,\n        min = 0.minutes,\n        max = 6000.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object MaxNumberExplorationTweetsParam\n      extends FSBoundedParam[Int](\n        name = \"for_you_exploration_tweets_max_number\",\n        default = 2,\n        min = 1,\n        max = 10\n      )\n\n  object EnableBookmarksModuleWeekendGate\n      extends FSParam[Boolean](\n        name = \"for_you_enable_bookmarks_module_weekend_gate\",\n        default = false\n      )\n\n  object BookmarksModuleMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_bookmarks_module_min_injection_interval_in_minutes\",\n        default = 2880.minutes, // 2 days\n        min = 0.minutes,\n        max = 6000.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object InNetworkExplorationTweetsMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        name = \"for_you_in_network_exploration_tweets_min_injection_interval_in_minutes\",\n        default = 30.minutes,\n        min = 0.minutes,\n        max = 60.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object ExplorationTweetsMaxFollowerCountParam\n      extends FSBoundedParam[Long](\n        name = \"for_you_exploration_tweets_max_follower_count\",\n        default = Long.MaxValue,\n        min = 0L,\n        max = Long.MaxValue\n      )\n\n  object EnableViewerHasJobRecommendationsFeatureParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_viewer_has_job_recommendations_feature\",\n        default = false\n      )\n\n  object EnableTrendsParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_trends\",\n        default = false\n      )\n\n  object EnableKeywordTrendsParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_keyword_trends\",\n        default = false\n      )\n\n  object MaxNumberKeywordTrendsParam\n      extends FSBoundedParam[Int](\n        name = \"for_you_keyword_trends_max_number\",\n        default = 5,\n        min = 1,\n        max = 10\n      )\n\n  object TrendsModuleMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_trends_min_injection_interval_in_minutes\",\n        default = 360.minutes,\n        min = 0.minutes,\n        max = 2880.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object KeywordTrendsModuleMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_keyword_trends_min_injection_interval_in_minutes\",\n        default = 360.minutes,\n        min = 0.minutes,\n        max = 2880.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object EnableScoredVideoTweetsCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_scored_video_tweets_pipeline\",\n        default = false\n      )\n\n  object VideoTweetsModuleMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_video_tweets_module_min_injection_interval_in_minutes\",\n        default = 1.hour,\n        min = 0.minutes,\n        max = 6000.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object VideoCarouselNumTweetCandidatesToDedupeAgainstParam\n      extends FSBoundedParam[Int](\n        name = \"for_you_video_carousel_num_tweet_candidates_to_dedupe_against\",\n        default = 10,\n        min = 0,\n        max = 50\n      )\n\n  object VideoCarouselTimelinePosition\n      extends FSBoundedParam[Int](\n        name = \"for_you_video_carousel_position\",\n        default = 5,\n        min = 0,\n        max = 50\n      )\n\n  object VideoCarouselNumCandidates\n      extends FSBoundedParam[Int](\n        name = \"for_you_video_carousel_num_candidates\",\n        default = 5,\n        min = 0,\n        max = 50\n      )\n\n  object VideoCarouselEnableFooterParam\n      extends FSParam[Boolean](\n        name = \"for_you_video_carousel_enable_footer\",\n        default = false\n      )\n\n  object VideoCarouselAllowVerticalVideos\n      extends FSParam[Boolean](\n        name = \"for_you_video_carousel_allow_vertical_videos\",\n        default = true\n      )\n\n  object VideoCarouselAllowHorizontalVideos\n      extends FSParam[Boolean](\n        name = \"for_you_video_carousel_allow_horizontal_videos\",\n        default = true\n      )\n\n  object EnableTLSHydrationParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_tls_hydration\",\n        default = false\n      )\n\n  object EnableGetTweetsFromArchiveIndex\n      extends BooleanDeciderParam(decider = DeciderKey.EnableGetTweetsFromArchiveIndex)\n\n  // Currently only supported by rweb\n  object EnableArticlePreviewTextHydrationParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_article_preview_hydration\",\n        default = false\n      )\n\n  object EnableAdsDebugParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_ads_debug\",\n        default = false\n      )\n\n  object RelevancePromptEnableParam\n      extends FSParam[Boolean](\n        name = \"for_you_relevance_prompt_enable\",\n        default = false\n      )\n\n  object RelevancePromptMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        name = \"for_you_relevance_prompt_min_injection_interval_minutes\",\n        default = 1440.minutes,\n        min = 0.minutes,\n        max = 144000.minutes\n      )\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object RelevancePromptTweetPositionParam\n      extends FSBoundedParam[Int](\n        name = \"for_you_relevance_prompt_position\",\n        default = 15,\n        min = 0,\n        max = 10000\n      )\n\n  object RelevancePromptTitleParam\n      extends FSParam[String](\n        name = \"for_you_relevance_prompt_title\",\n        default = \"\"\n      )\n\n  object RelevancePromptPositiveParam\n      extends FSParam[String](\n        name = \"for_you_relevance_prompt_positive\",\n        default = \"\"\n      )\n\n  object RelevancePromptNegativeParam\n      extends FSParam[String](\n        name = \"for_you_relevance_prompt_negative\",\n        default = \"\"\n      )\n\n  object RelevancePromptNeutralParam\n      extends FSParam[String](\n        name = \"for_you_relevance_prompt_neutral\",\n        default = \"\"\n      )\n\n  object EnableForYouTopicSelectorParam\n      extends FSParam[Boolean](\n        name = \"for_you_topic_selector_enabled\",\n        default = false\n      )\n\n  object ForYouTopicSelectorJetfuelRouteParam\n      extends FSParam[String](\n        name = \"for_you_topic_selector_jetfuel_route\",\n        default = \"\"\n      )\n\n  object ForYouTopicSelectorPosition\n      extends FSBoundedParam[Int](\n        name = \"for_you_topic_selector_position\",\n        default = 1,\n        min = 0,\n        max = 1000\n      )\n\n  object ForYouAppUpsellJetfuelRouteParam\n      extends FSParam[String](\n        name = \"for_you_app_upsell_jetfuel_route\",\n        default = \"/cards/pivots/appInstallPivot\"\n      )\n\n  object EnableForYouAppUpsellParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_app_upsell\",\n        default = false\n      )\n\n  object ForYouAppUpsellPosition\n      extends FSBoundedParam[Int](\n        name = \"for_you_app_upsell_position\",\n        default = 1,\n        min = 0,\n        max = 1000\n      )\n\n  object EnableTuneFeedCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_tune_feed_pipeline\",\n        default = false\n      )\n\n  object TuneFeedTimelinePosition\n      extends FSBoundedParam[Int](\n        name = \"for_you_tune_feed_position\",\n        default = 7,\n        min = 0,\n        max = 50\n      )\n\n  object TuneFeedModuleMinInjectionIntervalParam\n      extends FSBoundedParam[Duration](\n        \"for_you_tune_feed_module_min_injection_interval_in_minutes\",\n        default = 180.minutes, // 3 hours\n        min = 0.minutes,\n        max = 6000.minutes)\n      with HasDurationConversion {\n    override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n  }\n\n  object EnableFollowedGrokTopicsHydrationParam\n      extends FSParam[Boolean](\n        name = \"for_you_enable_followed_grok_topics_hydration\",\n        default = false\n      )\n\n  object EnableForYouTimelineAdsSurface\n      extends FSParam[Boolean](\n        name = \"for_you_enable_timeline_ads_surface\",\n        default = false\n      )\n\n  object MinFollowingCountParam\n      extends FSBoundedParam[Int](\n        name = \"for_you_user_following_range_gate_min_following_count\",\n        default = 0,\n        min = 0,\n        max = 1000000\n      )\n\n  object MaxFollowingCountParam\n      extends FSBoundedParam[Int](\n        name = \"for_you_user_following_range_gate_max_following_count\",\n        default = 10000,\n        min = 0,\n        max = 1000000\n      )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/param/ForYouParamConfig.scala",
    "content": "package com.twitter.home_mixer.product.for_you.param\n\nimport com.twitter.home_mixer.param.decider.DeciderKey\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam._\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.servo.decider.DeciderKeyName\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForYouParamConfig @Inject() () extends ProductParamConfig {\n  override val enabledDeciderKey: DeciderKeyName = DeciderKey.EnableForYouProduct\n  override val supportedClientFSName: String = SupportedClientFSName\n\n  override val booleanFSOverrides = Seq(\n    ClearCache.PtrEnableParam,\n    ClearCache.ColdStartEnableParam,\n    ClearCache.WarmStartEnableParam,\n    ClearCache.ManualRefreshEnableParam,\n    ClearCache.NavigateEnableParam,\n    ClearCache.ColdStartRetainViewportParam,\n    EnableCommunitiesToJoinCandidatePipelineParam,\n    EnableBookmarksCandidatePipelineParam,\n    EnableExplorationTweetsCandidatePipelineParam,\n    EnableJetfuelFramePipelineParam,\n    EnableBookmarksModuleWeekendGate,\n    EnablePinnedTweetsCandidatePipelineParam,\n    EnableEntryPointPivotParam,\n    EnableFlipInjectionModuleCandidatePipelineParam,\n    EnableGrokEntryPointPivotParam,\n    EnableRecommendedJobsParam,\n    EnableRecommendedRecruitingOrganizationsParam,\n    EnableTweetPreviewsCandidatePipelineParam,\n    EnableViewerHasJobRecommendationsFeatureParam,\n    EnableWhoToFollowCandidatePipelineParam,\n    EnableWhoToSubscribeCandidatePipelineParam,\n    EnableTrendsParam,\n    EnableKeywordTrendsParam,\n    EnableArticlePreviewTextHydrationParam,\n    EnableForYouTimelineAdsSurface,\n    EnableScoredVideoTweetsCandidatePipelineParam,\n    VideoCarouselEnableFooterParam,\n    VideoCarouselAllowVerticalVideos,\n    VideoCarouselAllowHorizontalVideos,\n    RelevancePromptEnableParam,\n    Navigation.PtrEnableParam,\n    Navigation.ColdStartEnableParam,\n    Navigation.WarmStartEnableParam,\n    Navigation.ManualRefreshEnableParam,\n    Navigation.NavigateEnableParam,\n    EnableAdsDebugParam,\n    EnableForYouAppUpsellParam,\n    EnableForYouTopicSelectorParam,\n    EnableTuneFeedCandidatePipelineParam,\n    EnableFollowedGrokTopicsHydrationParam,\n    EnableTLSHydrationParam\n  )\n\n  override val boundedLongFSOverrides = Seq(\n    ExplorationTweetsMaxFollowerCountParam\n  )\n\n  override val boundedIntFSOverrides = Seq(\n    AdsNumOrganicItemsParam,\n    ClearCache.MinEntriesParam,\n    ExplorationTweetsTimelinePosition,\n    GrokPivotModuleTimelinePosition,\n    SuperbowlModuleTimelinePosition,\n    VideoCarouselNumTweetCandidatesToDedupeAgainstParam,\n    VideoCarouselTimelinePosition,\n    VideoCarouselNumCandidates,\n    MaxCommunitiesToJoinCandidatesParam,\n    MaxNumberExplorationTweetsParam,\n    MaxNumberKeywordTrendsParam,\n    MaxRecommendedJobCandidatesParam,\n    MaxRecommendedRecruitingOrganizationCandidatesParam,\n    ServerMaxResultsParam,\n    TweetPreviewsMaxCandidatesParam,\n    RelevancePromptTweetPositionParam,\n    ForYouAppUpsellPosition,\n    ForYouTopicSelectorPosition,\n    TuneFeedTimelinePosition,\n    MinFollowingCountParam,\n    MaxFollowingCountParam\n  )\n\n  override val stringFSOverrides = Seq(\n    WhoToFollowDisplayLocationParam,\n    ExperimentStatsParam,\n    RelevancePromptTitleParam,\n    RelevancePromptPositiveParam,\n    RelevancePromptNegativeParam,\n    RelevancePromptNeutralParam,\n    ForYouAppUpsellJetfuelRouteParam,\n    ForYouTopicSelectorJetfuelRouteParam,\n  )\n\n  override val boundedDurationFSOverrides = Seq(\n    CommunitiesToJoinMinInjectionIntervalParam,\n    RecommendedJobMinInjectionIntervalParam,\n    RecommendedRecruitingOrganizationMinInjectionIntervalParam,\n    WhoToFollowMinInjectionIntervalParam,\n    WhoToSubscribeMinInjectionIntervalParam,\n    TweetPreviewsMinInjectionIntervalParam,\n    BookmarksModuleMinInjectionIntervalParam,\n    InNetworkExplorationTweetsMinInjectionIntervalParam,\n    PinnedTweetsModuleMinInjectionIntervalParam,\n    TrendsModuleMinInjectionIntervalParam,\n    KeywordTrendsModuleMinInjectionIntervalParam,\n    EntryPointPivotMinInjectionIntervalParam,\n    GrokEntryPointPivotMinInjectionIntervalParam,\n    VideoTweetsModuleMinInjectionIntervalParam,\n    RelevancePromptMinInjectionIntervalParam\n  )\n\n  override val enumFSOverrides = Seq(\n    WhoToFollowDisplayTypeIdParam,\n    WhoToSubscribeDisplayTypeIdParam,\n    WhoToFollowUserDisplayTypeIdParam,\n    CommunitiesToJoinDisplayTypeIdParam\n  )\n\n  override val booleanDeciderOverrides =\n    Seq(EnableGetTweetsFromArchiveIndex)\n\n  override val longSetFSOverrides = Seq(\n    AuthorListForStatsParam,\n    FollowingSportsGateUsersParam\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/query_transformer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"events-recos/events-recos-service/src/main/thrift:events-recos-thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/java/com/twitter/search/queryparser/query/search:search-query-nodes\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/query_transformer/TweetPreviewsQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.for_you.query_transformer\n\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.PreviewCreatorsFeature\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.search.common.ranking.{thriftscala => scr}\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant\nimport com.twitter.search.earlybird.{thriftscala => t}\nimport com.twitter.search.queryparser.query.Conjunction\nimport com.twitter.search.queryparser.query.Query\nimport com.twitter.search.queryparser.query.search.SearchOperator\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetPreviewsQueryTransformer @Inject() (clientId: ClientId)\n    extends CandidatePipelineQueryTransformer[PipelineQuery, t.EarlybirdRequest] {\n\n  private val MaxPreviewTweets = 200\n  private val EarlybirdRelevanceTensorflowModel = \"timelines_rectweet_replica\"\n  private val SinceDuration = 7.days\n\n  val MetadataOptions = t.ThriftSearchResultMetadataOptions(\n    getReferencedTweetAuthorId = true,\n    getFromUserId = true\n  )\n\n  override def transform(query: PipelineQuery): t.EarlybirdRequest = {\n    val candidatePreviewCreatorIds =\n      query.features.map(_.get(PreviewCreatorsFeature)).getOrElse(Seq.empty)\n\n    val searchQuery = new Conjunction(\n      // Include subscriber only (aka exclusive) Tweets\n      new SearchOperator.Builder()\n        .setType(SearchOperator.Type.FILTER)\n        .addOperand(EarlybirdFieldConstant.EXCLUSIVE_FILTER_TERM)\n        .build(),\n      // Include only original Tweets\n      new SearchOperator.Builder()\n        .setType(SearchOperator.Type.FILTER)\n        .addOperand(EarlybirdFieldConstant.NATIVE_RETWEETS_FILTER_TERM)\n        .setOccur(Query.Occur.MUST_NOT)\n        .build(),\n      new SearchOperator.Builder()\n        .setType(SearchOperator.Type.FILTER)\n        .addOperand(EarlybirdFieldConstant.REPLIES_FILTER_TERM)\n        .setOccur(Query.Occur.MUST_NOT)\n        .build(),\n      new SearchOperator.Builder()\n        .setType(SearchOperator.Type.FILTER)\n        .addOperand(EarlybirdFieldConstant.QUOTE_FILTER_TERM)\n        .setOccur(Query.Occur.MUST_NOT)\n        .build(),\n      new SearchOperator(SearchOperator.Type.SINCE_TIME, SinceDuration.ago.inSeconds.toString)\n    )\n\n    t.EarlybirdRequest(\n      searchQuery = t.ThriftSearchQuery(\n        serializedQuery = Some(searchQuery.serialize),\n        fromUserIDFilter64 = Some(candidatePreviewCreatorIds),\n        numResults = MaxPreviewTweets,\n        rankingMode = t.ThriftSearchRankingMode.Relevance,\n        relevanceOptions = Some(\n          t.ThriftSearchRelevanceOptions(\n            filterDups = true,\n            keepDupWithHigherScore = true,\n            proximityScoring = true,\n            maxConsecutiveSameUser = Some(5),\n            rankingParams = Some(\n              scr.ThriftRankingParams(\n                `type` = Some(scr.ThriftScoringFunctionType.TensorflowBased),\n                selectedTensorflowModel = Some(EarlybirdRelevanceTensorflowModel),\n                minScore = -1.0e100,\n                applyBoosts = false,\n              )\n            ),\n          ),\n        ),\n        resultMetadataOptions = Some(MetadataOptions),\n        searcherId = query.getOptionalUserId,\n      ),\n      getOlderResults = Some(true), // needed for archive access to older tweets\n      clientRequestID = Some(s\"${Trace.id.traceId}\"),\n      followedUserIds = Some(candidatePreviewCreatorIds.toSeq),\n      numResultsToReturnAtRoot = Some(MaxPreviewTweets),\n      clientId = Some(clientId.name),\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/query_transformer/UnifiedCandidatesQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.for_you.query_transformer\n\nimport com.twitter.events.recos.{thriftscala => t}\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\ncase class UnifiedCandidatesQueryTransformer(\n  maxResultsParam: Param[Int],\n  candidatePipelineIdentifier: CandidatePipelineIdentifier)\n    extends CandidatePipelineQueryTransformer[\n      PipelineQuery,\n      t.GetUnfiedCandidatesRequest\n    ] {\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"UnifiedCandidates\")\n\n  override def transform(\n    query: PipelineQuery\n  ): t.GetUnfiedCandidatesRequest = {\n\n    t.GetUnfiedCandidatesRequest(\n      displayLocation = t.DisplayLocation.Guide,\n      clientId = query.clientContext.appId,\n      userId = query.getOptionalUserId,\n      languageCode = query.getLanguageCode,\n      countryCode = query.getCountryCode,\n      maxResults = Some(query.params(maxResultsParam)),\n      userAgent = query.clientContext.userAgent,\n      isObjectiveTrendsRequest = Some(true)\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/response_transformer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"events-recos/events-recos-service/src/main/thrift:events-recos-thrift-scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/trends_events\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"src/thrift/com/twitter/frigate/bookmarks:bookmarks-thrift-scala\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/response_transformer/BookmarksResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.for_you.response_transformer\n\nimport com.twitter.frigate.bookmarks.thriftscala.BookmarkedTweet\nimport com.twitter.home_mixer.model.HomeFeatures.BookmarkedTweetTimestamp\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject BookmarksResponseFeatureTransformer extends CandidateFeatureTransformer[BookmarkedTweet] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"BookmarksResponse\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(BookmarkedTweetTimestamp, ServedTypeFeature)\n\n  def transform(input: BookmarkedTweet): FeatureMap = FeatureMapBuilder()\n    .add(BookmarkedTweetTimestamp, Some(input.timestamp))\n    .add(ServedTypeFeature, hmt.ServedType.ForYouResurfacedBookmark)\n    .build()\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/response_transformer/ExplorationTweetResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.for_you.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject ExplorationTweetResponseFeatureTransformer extends CandidateFeatureTransformer[Long] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ExplorationTweetResponse\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(ServedTypeFeature)\n\n  def transform(\n    input: Long\n  ): FeatureMap = {\n    FeatureMapBuilder()\n      .add(ServedTypeFeature, hmt.ServedType.ForYouExploration)\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/response_transformer/KeywordTrendsFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.for_you.response_transformer\n\nimport com.twitter.events.recos.{thriftscala => t}\nimport com.twitter.home_mixer.util.UrtUtil\nimport com.twitter.home_mixer.product.for_you.candidate_source.TrendCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events._\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.trend.GroupedTrend\n\nobject KeywordTrendsFeatureTransformer extends CandidateFeatureTransformer[TrendCandidate] {\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"KeywordTrends\")\n\n  override def features: Set[Feature[_, _]] = Set(\n    TrendNormalizedNameFeature,\n    TrendNameFeature,\n    TrendUrlFeature,\n    TrendDescriptionFeature,\n    TrendTweetCountFeature,\n    TrendDomainContextFeature,\n    TrendGroupedTrendsFeature,\n    PromotedTrendNameFeature,\n    TrendRankFeature\n  )\n\n  override def transform(input: TrendCandidate): FeatureMap = {\n    val trend: t.TrendCandidate = input.candidate\n\n    FeatureMapBuilder()\n      .add(TrendNameFeature, trend.trendName)\n      .add(TrendDescriptionFeature, trend.context.flatMap(_.description))\n      .add(TrendNormalizedNameFeature, trend.normalizedTrendName)\n      .add(TrendUrlFeature, UrtUtil.transformUrl(trend.url))\n      .add(TrendTweetCountFeature, trend.context.flatMap(_.tweetCount))\n      .add(TrendDomainContextFeature, trend.domainContext)\n      .add(TrendGroupedTrendsFeature, trend.relatedTrends.map(transformGroupedTrends))\n      .add(PromotedTrendNameFeature, trend.promotedMetadata.map(_.name))\n      .add(TrendRankFeature, input.rank)\n      .build()\n  }\n\n  private def transformGroupedTrends(groupedTrends: Seq[t.RelatedTrend]): Seq[GroupedTrend] = {\n    groupedTrends.map { trend =>\n      GroupedTrend(\n        trendName = trend.trendName,\n        url = UrtUtil.transformUrl(trend.url)\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/response_transformer/PinnedTweetResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.for_you.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.product.for_you.candidate_source.PinnedTweetCandidate\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject PinnedTweetResponseFeatureTransformer\n    extends CandidateFeatureTransformer[PinnedTweetCandidate] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"PinnedTweetResponse\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(AuthorIdFeature, ServedTypeFeature)\n\n  def transform(\n    input: PinnedTweetCandidate\n  ): FeatureMap = {\n    FeatureMapBuilder()\n      .add(AuthorIdFeature, input.userId)\n      .add(ServedTypeFeature, hmt.ServedType.ForYouPinned)\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/response_transformer/ScoredVideoTweetResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.for_you.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoAspectRatioFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoDisplayTypeFeature\nimport com.twitter.home_mixer.product.for_you.candidate_source.ScoredVideoTweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Carousel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.CompactCarousel\n\nobject ScoredVideoTweetResponseFeatureTransformer\n    extends CandidateFeatureTransformer[ScoredVideoTweetCandidate] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredVideoTweetResponse\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(AuthorIdFeature, ServedTypeFeature, VideoAspectRatioFeature, VideoDisplayTypeFeature)\n\n  def transform(\n    input: ScoredVideoTweetCandidate\n  ): FeatureMap = {\n    val displayType = input.aspectRatio.map { ratio =>\n      if (ratio > 1.0) Carousel else CompactCarousel\n    }\n    FeatureMapBuilder()\n      .add(AuthorIdFeature, Some(input.authorId))\n      .add(ServedTypeFeature, input.servedType)\n      .add(VideoAspectRatioFeature, input.aspectRatio.map(_.toFloat))\n      .add(VideoDisplayTypeFeature, displayType)\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/response_transformer/StoriesModuleResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.for_you.response_transformer\n\nimport com.twitter.home_mixer.product.for_you.candidate_source.StoryCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendDescriptionFeature\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendDomainContextFeature\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendNameFeature\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendSocialContextImages\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendThumbnail\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendUrlFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.graphql.ApiImage\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DeepLink\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\n\nobject StoriesModuleResponseFeatureTransformer extends CandidateFeatureTransformer[StoryCandidate] {\n\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"StoriesModuleResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    TrendNameFeature,\n    TrendDescriptionFeature,\n    TrendUrlFeature,\n    TrendDomainContextFeature,\n    TrendSocialContextImages,\n    TrendThumbnail\n  )\n\n  def transform(input: StoryCandidate): FeatureMap = {\n    val url = input.id\n    val thumbnail = input.thumbnail.map { img =>\n      ApiImage(\n        height = img.originalImgHeight,\n        width = img.originalImgWidth,\n        url = img.originalImgUrl\n      )\n    }\n\n    FeatureMapBuilder()\n      .add(TrendNameFeature, input.title)\n      .add(TrendDescriptionFeature, None)\n      .add(TrendUrlFeature, Url(DeepLink, url))\n      .add(TrendDomainContextFeature, Some(input.context))\n      .add(TrendSocialContextImages, Some(input.socialProof))\n      .add(TrendThumbnail, thumbnail)\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/response_transformer/TuneFeedFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.for_you.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject TuneFeedFeatureTransformer extends CandidateFeatureTransformer[TweetCandidate] {\n\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"TuneFeed\")\n\n  override val features: Set[Feature[_, _]] = Set(ServedTypeFeature)\n\n  def transform(\n    input: TweetCandidate\n  ): FeatureMap = FeatureMap(ServedTypeFeature, hmt.ServedType.ForYouPopularTopic)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/response_transformer/TweetPreviewResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.for_you.response_transformer\n\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsTweetPreviewFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.search.earlybird.{thriftscala => eb}\n\nobject TweetPreviewResponseFeatureTransformer\n    extends CandidateFeatureTransformer[eb.ThriftSearchResult] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"TweetPreviewResponse\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(AuthorIdFeature, IsTweetPreviewFeature, ServedTypeFeature)\n\n  def transform(\n    input: eb.ThriftSearchResult\n  ): FeatureMap = {\n    FeatureMapBuilder()\n      .add(IsTweetPreviewFeature, true)\n      .add(ServedTypeFeature, hmt.ServedType.ForYouTweetPreview)\n      .add(AuthorIdFeature, input.metadata.map(_.fromUserId))\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/scorer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/scorer/PinnedTweetCandidateScorer.scala",
    "content": "package com.twitter.home_mixer.product.for_you.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorFollowersFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject PinnedTweetCandidateScorer extends Scorer[PipelineQuery, TweetCandidate] {\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"PinnedTweetCandidate\")\n\n  override def features: Set[Feature[_, _]] = Set(ScoreFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n\n    val featureMaps = candidates.map { candidate =>\n      val score = candidate.features.getOrElse(AuthorFollowersFeature, None)\n      FeatureMap(ScoreFeature, score.map(_.toDouble))\n    }\n\n    Stitch.value(featureMaps)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/selector/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/param\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/selector/DebugUpdateSortAdsResult.scala",
    "content": "package com.twitter.home_mixer.product.for_you.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope.PartitionedCandidates\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n// This selector is used for debugging Ads. Tt will put all the Ads candidates to the top\ncase class DebugUpdateSortAdsResult(\n  adsCandidatePipeline: CandidatePipelineIdentifier)\n    extends Selector[PipelineQuery] {\n\n  override val pipelineScope: CandidateScope = SpecificPipeline(adsCandidatePipeline)\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val PartitionedCandidates(adCandidates, otherRemainingCandidates) =\n      pipelineScope.partition(result)\n\n    SelectorResult(\n      remainingCandidates = remainingCandidates,\n      result = adCandidates ++ otherRemainingCandidates)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/selector/DebunchCandidates.scala",
    "content": "package com.twitter.home_mixer.product.for_you.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope.PartitionedCandidates\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\ntrait MustDebunch {\n  def apply(candidate: CandidateWithDetails): Boolean\n}\n\n/**\n * This selector rearranges the candidates to only allow bunches of size [[maxBunchSize]],\n * where a bunch is a consecutive sequence of candidates that meet [[mustDebunch]].\n */\ncase class DebunchCandidates(\n  override val pipelineScope: CandidateScope,\n  mustDebunch: MustDebunch,\n  maxBunchSize: Param[Int])\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val PartitionedCandidates(selectedCandidates, otherCandidates) =\n      pipelineScope.partition(remainingCandidates)\n    val mutableCandidates = collection.mutable.ListBuffer(selectedCandidates: _*)\n\n    var candidatePointer = 0\n    var nonDebunchPointer = 0\n    var bunchSize = 0\n    var finalNonDebunch = -1\n\n    while (candidatePointer < mutableCandidates.size) {\n      if (mustDebunch(mutableCandidates(candidatePointer))) bunchSize += 1\n      else {\n        bunchSize = 0\n        finalNonDebunch = candidatePointer\n      }\n\n      if (bunchSize > query.params(maxBunchSize)) {\n        nonDebunchPointer = Math.max(candidatePointer, nonDebunchPointer)\n        while (nonDebunchPointer < mutableCandidates.size &&\n          mustDebunch(mutableCandidates(nonDebunchPointer))) {\n          nonDebunchPointer += 1\n        }\n        if (nonDebunchPointer == mutableCandidates.size)\n          candidatePointer = mutableCandidates.size\n        else {\n          val nextNonDebunch = mutableCandidates(nonDebunchPointer)\n          mutableCandidates.remove(nonDebunchPointer)\n          mutableCandidates.insert(candidatePointer, nextNonDebunch)\n          bunchSize = 0\n          finalNonDebunch = candidatePointer\n        }\n      }\n\n      candidatePointer += 1\n    }\n\n    val debunchedCandidates = mutableCandidates.toList\n    val updatedCandidates = otherCandidates ++ debunchedCandidates\n    SelectorResult(remainingCandidates = updatedCandidates, result = result)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/selector/RemoveDuplicateCandidatesOutsideModule.scala",
    "content": "package com.twitter.home_mixer.product.for_you.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\ncase class RemoveDuplicateCandidatesOutsideModule(\n  override val pipelineScope: CandidateScope,\n  candidatePipelinesOutsideModule: Set[CandidatePipelineIdentifier],\n  numCandidatesToCompareAgainst: Param[Int])\n    extends Selector[PipelineQuery] {\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val candidatesOutsideModule = getCandidateIdsOutsideModule(query, remainingCandidates)\n    val finalRemainingCandidates = remainingCandidates.map {\n      case module: ModuleCandidateWithDetails if pipelineScope.contains(module) =>\n        module.copy(candidates = module.candidates.filterNot { candidate =>\n          candidatesOutsideModule.contains(candidate.candidateIdLong)\n        })\n      case candidate => candidate\n    }\n\n    SelectorResult(remainingCandidates = finalRemainingCandidates, result = result)\n  }\n\n  private def getCandidateIdsOutsideModule(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails]\n  ): Set[Long] = {\n    remainingCandidates\n      .collect {\n        case candidate: ItemCandidateWithDetails\n            if candidatePipelinesOutsideModule.contains(candidate.source) =>\n          candidate.candidateIdLong\n      }.take(query.params(numCandidatesToCompareAgainst)).toSet\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/side_effect/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"kafka/finagle-kafka/finatra-kafka/src/main/scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters\",\n        \"src/thrift/com/twitter/timelines/served_candidates_logging:served_candidates_logging-scala\",\n        \"src/thrift/com/twitter/timelines/suggests/common:poly_data_record-java\",\n        \"timelines/ml:kafka\",\n        \"timelines/ml/cont_train/common/domain/src/main/scala/com/twitter/timelines/ml/cont_train/common/domain/non_scalding\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/side_effect/ServedCandidateFeatureKeysKafkaSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.for_you.side_effect\n\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsReadFromCacheFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PredictionRequestIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedIdFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableServedCandidateFeatureKeysKafkaPublishingParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.side_effect.KafkaPublishingSideEffect\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.ml.cont_train.common.domain.non_scalding.ServedCandidateFeatureKeysAdapter\nimport com.twitter.timelines.ml.cont_train.common.domain.non_scalding.ServedCandidateFeatureKeysFields\nimport com.twitter.timelines.ml.kafka.serde.CandidateFeatureKeySerde\nimport com.twitter.timelines.ml.kafka.serde.TBaseSerde\nimport com.twitter.timelines.served_candidates_logging.{thriftscala => sc}\nimport com.twitter.timelines.suggests.common.poly_data_record.{thriftjava => pldr}\nimport org.apache.kafka.clients.producer.ProducerRecord\nimport org.apache.kafka.common.serialization.Serializer\nimport scala.collection.JavaConverters._\n\n/**\n * Pipeline side-effect that publishes candidate keys to a Kafka topic.\n */\nclass ServedCandidateFeatureKeysKafkaSideEffect(\n  topic: String,\n  sourceIdentifiers: Set[identifier.CandidatePipelineIdentifier])\n    extends KafkaPublishingSideEffect[\n      sc.CandidateFeatureKey,\n      pldr.PolyDataRecord,\n      PipelineQuery,\n      Timeline\n    ]\n    with PipelineResultSideEffect.Conditionally[PipelineQuery, Timeline] {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"ServedCandidateFeatureKeys\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: Timeline\n  ): Boolean = query.params.getBoolean(EnableServedCandidateFeatureKeysKafkaPublishingParam)\n\n  override val bootstrapServer: String = \"/s/kafka/timeline:kafka-tls\"\n\n  override val keySerde: Serializer[sc.CandidateFeatureKey] =\n    CandidateFeatureKeySerde().serializer()\n\n  override val valueSerde: Serializer[pldr.PolyDataRecord] =\n    TBaseSerde.Thrift[pldr.PolyDataRecord]().serializer\n\n  override val clientId: String = \"home_mixer_served_candidate_feature_keys_producer\"\n\n  override def buildRecords(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: Timeline\n  ): Seq[ProducerRecord[sc.CandidateFeatureKey, pldr.PolyDataRecord]] = {\n    query.features\n      .flatMap(_.getOrElse(ServedIdFeature, None))\n      .fold(Seq.empty[ProducerRecord[sc.CandidateFeatureKey, pldr.PolyDataRecord]]) { servedId =>\n        CandidatesUtil\n          .getItemCandidates {\n            selectedCandidates.iterator\n              .filter(candidate => sourceIdentifiers.contains(candidate.source)).toSeq\n          }\n          .flatMap { candidate =>\n            candidate.features\n              .getOrElse(PredictionRequestIdFeature, None)\n              .map { predictionRequestId =>\n                val key = sc.CandidateFeatureKey(\n                  tweetId = candidate.candidateIdLong,\n                  viewerId = query.getRequiredUserId,\n                  servedId = -1L)\n\n                val record =\n                  ServedCandidateFeatureKeysAdapter\n                    .adaptToDataRecords(ServedCandidateFeatureKeysFields(\n                      viewerId = query.getRequiredUserId,\n                      tweetId = candidate.candidateIdLong,\n                      predictionRequestId = predictionRequestId,\n                      servedRequestIdOpt = None,\n                      servedId = servedId,\n                      injectionModuleName = candidate.getClass.getSimpleName,\n                      viewerFollowsOriginalAuthor =\n                        Some(candidate.features.getOrElse(InNetworkFeature, true)),\n                      finalPositionIndex = Some(candidate.sourcePosition),\n                      isReadFromCache = candidate.features.getOrElse(IsReadFromCacheFeature, false)\n                    )).asScala.head\n\n                new ProducerRecord(topic, key, pldr.PolyDataRecord.dataRecord(record))\n              }\n          }\n      }\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(98.5)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/side_effect/ServedCandidateFeatureKeysKafkaSideEffectBuilder.scala",
    "content": "package com.twitter.home_mixer.product.for_you.side_effect\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class ServedCandidateFeatureKeysKafkaSideEffectBuilder @Inject() (\n  injectedServiceIdentifier: ServiceIdentifier) {\n  def build(\n    sourceIdentifiers: Set[CandidatePipelineIdentifier]\n  ): ServedCandidateFeatureKeysKafkaSideEffect = {\n    val topic = injectedServiceIdentifier.environment.toLowerCase match {\n      case \"prod\" => \"tq_ct_served_candidate_feature_keys\"\n      case _ => \"tq_ct_served_candidate_feature_keys_staging\"\n    }\n    new ServedCandidateFeatureKeysKafkaSideEffect(topic, sourceIdentifiers)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/side_effect/ServedCandidateKafkaSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.for_you.side_effect\n\nimport com.twitter.home_mixer.model.HomeFeatures.IsReadFromCacheFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PredictionRequestIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedRequestIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.StreamToKafkaFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject ServedCandidateKafkaSideEffect {\n\n  def extractCandidates(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    sourceIdentifiers: Set[CandidatePipelineIdentifier]\n  ): Seq[ItemCandidateWithDetails] = {\n    val servedRequestIdOpt =\n      query.features.getOrElse(FeatureMap.empty).getOrElse(ServedRequestIdFeature, None)\n\n    selectedCandidates.iterator\n      .filter(candidate => sourceIdentifiers.contains(candidate.source))\n      .flatMap {\n        case item: ItemCandidateWithDetails => Seq(item)\n        case module: ModuleCandidateWithDetails => module.candidates\n      }\n      .filter(candidate => candidate.features.getOrElse(StreamToKafkaFeature, false))\n      .map { candidate =>\n        val servedId =\n          if (candidate.features.getOrElse(IsReadFromCacheFeature, false) &&\n            servedRequestIdOpt.nonEmpty)\n            servedRequestIdOpt\n          else\n            candidate.features.getOrElse(PredictionRequestIdFeature, None)\n\n        candidate.copy(features = candidate.features + (ServedIdFeature, servedId))\n      }.toSeq\n      // deduplicate by (tweetId, userId, servedId)\n      .groupBy { candidate =>\n        (\n          candidate.candidateIdLong,\n          query.getRequiredUserId,\n          candidate.features.getOrElse(ServedIdFeature, None))\n      }.values.map(_.head).toSeq\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/side_effect/ServedCandidateKeysKafkaSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.for_you.side_effect\n\nimport com.twitter.home_mixer.model.HomeFeatures.IsReadFromCacheFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PredictionRequestIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedRequestIdFeature\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.EnableServedCandidateKafkaPublishingParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.util.SRichDataRecord\nimport com.twitter.product_mixer.component_library.side_effect.KafkaPublishingSideEffect\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.ml.cont_train.common.domain.non_scalding.DataRecordLoggingRelatedFeatures.tlmServedKeysFeatureContext\nimport com.twitter.timelines.ml.kafka.serde.ServedCandidateKeySerde\nimport com.twitter.timelines.ml.kafka.serde.TBaseSerde\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport com.twitter.timelines.served_candidates_logging.{thriftscala => sc}\nimport com.twitter.timelines.suggests.common.poly_data_record.{thriftjava => pldr}\nimport com.twitter.util.Time\nimport org.apache.kafka.clients.producer.ProducerRecord\nimport org.apache.kafka.common.serialization.Serializer\n\n/**\n * Pipeline side-effect that publishes candidate keys to a Kafka topic.\n */\nclass ServedCandidateKeysKafkaSideEffect(\n  topic: String,\n  sourceIdentifiers: Set[CandidatePipelineIdentifier])\n    extends KafkaPublishingSideEffect[\n      sc.ServedCandidateKey,\n      pldr.PolyDataRecord,\n      PipelineQuery,\n      Timeline\n    ]\n    with PipelineResultSideEffect.Conditionally[PipelineQuery, Timeline] {\n\n  import ServedCandidateKafkaSideEffect._\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"ServedCandidateKeys\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: Timeline\n  ): Boolean = query.params.getBoolean(EnableServedCandidateKafkaPublishingParam)\n\n  override val bootstrapServer: String = \"/s/kafka/timeline:kafka-tls\"\n\n  override val keySerde: Serializer[sc.ServedCandidateKey] = ServedCandidateKeySerde.serializer()\n\n  override val valueSerde: Serializer[pldr.PolyDataRecord] =\n    TBaseSerde.Thrift[pldr.PolyDataRecord]().serializer\n\n  override val clientId: String = \"home_mixer_served_candidate_keys_producer\"\n\n  override def buildRecords(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: Timeline\n  ): Seq[ProducerRecord[sc.ServedCandidateKey, pldr.PolyDataRecord]] = {\n    val servedTimestamp = Time.now.inMilliseconds\n    val servedRequestIdOpt =\n      query.features.getOrElse(FeatureMap.empty).getOrElse(ServedRequestIdFeature, None)\n\n    extractCandidates(query, selectedCandidates, sourceIdentifiers).collect {\n      // Only publish non-cached tweets to the ServedCandidateKey topic\n      case candidate if !candidate.features.getOrElse(IsReadFromCacheFeature, false) =>\n        val key = sc.ServedCandidateKey(\n          tweetId = candidate.candidateIdLong,\n          viewerId = query.getRequiredUserId,\n          servedId = -1L\n        )\n\n        val record = SRichDataRecord(new DataRecord, tlmServedKeysFeatureContext)\n        record.setFeatureValueFromOption(\n          TimelinesSharedFeatures.PREDICTION_REQUEST_ID,\n          candidate.features.getOrElse(PredictionRequestIdFeature, None)\n        )\n        record\n          .setFeatureValueFromOption(TimelinesSharedFeatures.SERVED_REQUEST_ID, servedRequestIdOpt)\n        record.setFeatureValueFromOption(\n          TimelinesSharedFeatures.SERVED_ID,\n          candidate.features.getOrElse(ServedIdFeature, None)\n        )\n        record.setFeatureValueFromOption(\n          TimelinesSharedFeatures.INJECTION_TYPE,\n          record.getFeatureValueOpt(TimelinesSharedFeatures.INJECTION_TYPE))\n        record.setFeatureValue(\n          TimelinesSharedFeatures.SERVED_TIMESTAMP,\n          servedTimestamp\n        )\n        record.record.dropUnknownFeatures()\n\n        new ProducerRecord(topic, key, pldr.PolyDataRecord.dataRecord(record.getRecord))\n    }\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(98.5)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/side_effect/ServedCandidateKeysKafkaSideEffectBuilder.scala",
    "content": "package com.twitter.home_mixer.product.for_you.side_effect\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class ServedCandidateKeysKafkaSideEffectBuilder @Inject() (\n  injectedServiceIdentifier: ServiceIdentifier) {\n  def build(\n    sourceIdentifiers: Set[CandidatePipelineIdentifier]\n  ): ServedCandidateKeysKafkaSideEffect = {\n    val topic = injectedServiceIdentifier.environment.toLowerCase match {\n      case \"prod\" => \"tq_ct_served_candidate_keys\"\n      case _ => \"tq_ct_served_candidate_keys_staging\"\n    }\n    new ServedCandidateKeysKafkaSideEffect(topic, sourceIdentifiers)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/side_effect/ServedStatsSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.for_you.side_effect\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.AuthorListForDataCollectionParam\nimport com.twitter.home_mixer.product.for_you.param.ForYouParam.AuthorListForStatsParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.logging.Logging\n\ncase class ServedStatsSideEffect(\n  candidatePipelines: Set[CandidatePipelineIdentifier],\n  statsReceiver: StatsReceiver)\n    extends PipelineResultSideEffect[PipelineQuery, Timeline]\n    with Logging {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"ServedStats\")\n\n  private val baseStatsReceiver = statsReceiver.scope(identifier.toString)\n  private val authorStatsReceiver = baseStatsReceiver.scope(\"Author\")\n  private val tweetStatsReceiver = baseStatsReceiver.scope(\"Tweet\")\n  private val servedTypeStatsReceiver = baseStatsReceiver.scope(\"ServedType\")\n  private val followedUsersStatsReceiver = baseStatsReceiver.scope(\"FollowedUsers\")\n  private val responseSizeStatsReceiver = baseStatsReceiver.scope(\"ResponseSize\")\n  private val contentBalanceStatsReceiver = baseStatsReceiver.scope(\"ContentBalance\")\n\n  private val inNetworkStatsReceiver = contentBalanceStatsReceiver.scope(\"InNetwork\")\n  private val outOfNetworkStatsReceiver = contentBalanceStatsReceiver.scope(\"OutOfNetwork\")\n  private val replyStatsReceiver = contentBalanceStatsReceiver.scope(\"Reply\")\n  private val originalStatsReceiver = contentBalanceStatsReceiver.scope(\"Original\")\n\n  private val emptyStatsReceiver = responseSizeStatsReceiver.scope(\"Empty\")\n  private val lessThan5StatsReceiver = responseSizeStatsReceiver.scope(\"LessThan5\")\n  private val lessThan10StatsReceiver = responseSizeStatsReceiver.scope(\"LessThan10\")\n\n  override def apply(\n    inputs: PipelineResultSideEffect.Inputs[PipelineQuery, Timeline]\n  ): Stitch[Unit] = {\n    val tweetCandidates = CandidatesUtil\n      .getItemCandidates(inputs.selectedCandidates)\n      .filter(candidate => candidatePipelines.contains(candidate.source))\n\n    recordAuthorStats(\n      candidates = tweetCandidates,\n      authors = inputs.query.params(AuthorListForDataCollectionParam),\n      tweetLevelAuthors = inputs.query.params(AuthorListForStatsParam)\n    )\n    recordServedTypeStats(tweetCandidates)\n    recordFollowedUsersStats(\n      tweetCandidates,\n      inputs.query.features.map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty).size).getOrElse(0),\n    )\n    recordContentBalanceStats(tweetCandidates)\n    recordResponseSizeStats(tweetCandidates)\n    logColdContentData(tweetCandidates, inputs.query.getRequiredUserId)\n    Stitch.Unit\n  }\n\n  def recordAuthorStats(\n    candidates: Seq[CandidateWithDetails],\n    authors: Set[Long],\n    tweetLevelAuthors: Set[Long]\n  ): Unit = {\n    val filtered = candidates\n      .filter { candidate =>\n        candidate.features.getOrElse(AuthorIdFeature, None).exists(authors.contains) &&\n        // Only include original tweets\n        (!candidate.features.getOrElse(IsRetweetFeature, false)) &&\n        candidate.features.getOrElse(InReplyToTweetIdFeature, None).isEmpty\n      }\n\n    filtered\n      .groupBy { candidate =>\n        (getServedType(candidate), candidate.features.get(AuthorIdFeature).get)\n      }\n      .foreach {\n        case ((servedType, authorId), authorCandidates) =>\n          val authorStr = authorId.toString.takeRight(9)\n          authorStatsReceiver.scope(authorStr).counter(servedType).incr(authorCandidates.size)\n      }\n\n    filtered.map { candidate =>\n      val authorId = candidate.features.get(AuthorIdFeature).get\n      val authorStr = authorId.toString.takeRight(9)\n      authorStatsReceiver.scope(authorStr).counter(candidate.candidateIdLong.toString).incr()\n      if (tweetLevelAuthors.contains(authorId))\n        tweetStatsReceiver.counter(candidate.candidateIdLong.toString.takeRight(10)).incr()\n    }\n  }\n\n  def logColdContentData(candidates: Seq[CandidateWithDetails], viewerId: Long): Unit = {\n    candidates.foreach { candidate =>\n      val servedType = candidate.features.get(ServedTypeFeature)\n      if (servedType == hmt.ServedType.ForYouContentExplorationDeepRetrievalI2i) {\n        logger.info(\"Tier1: \" + candidate.candidateIdLong + \" viewerId: \" + viewerId)\n      } else if (servedType == hmt.ServedType.ForYouContentExplorationTier2DeepRetrievalI2i) {\n        logger.info(\"Tier2: \" + candidate.candidateIdLong + \" viewerId: \" + viewerId)\n      }\n    }\n  }\n\n  def recordServedTypeStats(\n    candidates: Seq[ItemCandidateWithDetails],\n  ): Unit = {\n    candidates.groupBy(getServedType).foreach {\n      case (servedType, servedTypeCandidates) =>\n        servedTypeStatsReceiver.counter(servedType).incr(servedTypeCandidates.size)\n    }\n  }\n\n  def recordFollowedUsersStats(\n    candidates: Seq[ItemCandidateWithDetails],\n    followedUsers: Int,\n  ): Unit = {\n    val followedUsersScope =\n      if (followedUsers < 10) \"0-9\"\n      else if (followedUsers < 100) \"10-99\"\n      else if (followedUsers < 1000) \"100-999\"\n      else if (followedUsers < 10000) \"1000-9999\"\n      else \">10000\"\n\n    candidates.groupBy(getServedType).foreach {\n      case (servedType, servedTypeCandidates) =>\n        followedUsersStatsReceiver\n          .scope(followedUsersScope).counter(servedType)\n          .incr(servedTypeCandidates.size)\n    }\n  }\n\n  def recordContentBalanceStats(\n    candidates: Seq[ItemCandidateWithDetails],\n  ): Unit = {\n    val (in, oon) = candidates.partition(_.features.getOrElse(InNetworkFeature, true))\n    inNetworkStatsReceiver.counter().incr(in.size)\n    outOfNetworkStatsReceiver.counter().incr(oon.size)\n\n    val (reply, original) =\n      candidates.partition(_.features.getOrElse(InReplyToTweetIdFeature, None).isDefined)\n    replyStatsReceiver.counter().incr(reply.size)\n    originalStatsReceiver.counter().incr(original.size)\n  }\n\n  def recordResponseSizeStats(\n    candidates: Seq[ItemCandidateWithDetails],\n  ): Unit = {\n    if (candidates.size == 0) emptyStatsReceiver.counter().incr()\n    if (candidates.size < 5) lessThan5StatsReceiver.counter().incr()\n    if (candidates.size < 10) lessThan10StatsReceiver.counter().incr()\n  }\n\n  private def getServedType(candidate: CandidateWithDetails): String =\n    candidate.features.get(ServedTypeFeature).name\n\n  override val alerts = Seq(HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert())\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/for_you/side_effect/VideoServedStatsSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.for_you.side_effect\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoDurationMsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ViralContentCreatorFeature\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\ncase class VideoServedStatsSideEffect(\n  candidatePipelines: Set[CandidatePipelineIdentifier],\n  statsReceiver: StatsReceiver)\n    extends PipelineResultSideEffect[PipelineQuery, Timeline] {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"VideoServedStats\")\n  private val baseStatsReceiver = statsReceiver.scope(identifier.toString)\n  private val videoStatsReceiver = baseStatsReceiver.scope(\"Video\")\n\n  override def apply(\n    inputs: PipelineResultSideEffect.Inputs[PipelineQuery, Timeline]\n  ): Stitch[Unit] = {\n    val tweetCandidates = CandidatesUtil\n      .getItemCandidates(inputs.selectedCandidates)\n      .filter(candidate => candidatePipelines.contains(candidate.source))\n\n    val clientId = inputs.query.clientContext.appId.getOrElse(0L).toString\n\n    // Filter candidates to include only those with video and valid duration\n    val videoCandidates = tweetCandidates.filter { candidate =>\n      candidate.features.getOrElse(HasVideoFeature, false) &&\n      candidate.features.get(VideoDurationMsFeature).exists(_ > 0)\n    }\n    statsReceiver\n      .scope(clientId).counter(\"HasVideo\")\n      .incr(videoCandidates.size)\n\n    recordVideoDurationStats(videoCandidates, videoStatsReceiver, clientId)\n    recordViralContentStats(videoCandidates, videoStatsReceiver, clientId)\n    Stitch.Unit\n  }\n\n  def recordViralContentStats(\n    candidates: Seq[ItemCandidateWithDetails],\n    statsReceiver: StatsReceiver,\n    clientId: String\n  ): Unit = {\n\n    val viralContentCount = candidates.count { candidate =>\n      candidate.features.getOrElse(ViralContentCreatorFeature, false)\n    }\n    val viralContentInNetworkCount = candidates.count { candidate =>\n      candidate.features.getOrElse(ViralContentCreatorFeature, false) &&\n      candidate.features.getOrElse(InNetworkFeature, true)\n    }\n    val viralContentOutOfNetworkCount = candidates.count { candidate =>\n      candidate.features.getOrElse(ViralContentCreatorFeature, false) &&\n      !candidate.features.getOrElse(InNetworkFeature, true)\n    }\n\n    statsReceiver\n      .scope(clientId).counter(\"ViralContent\")\n      .incr(viralContentCount)\n    statsReceiver\n      .scope(clientId)\n      .counter(\"ViralContentInNetwork\")\n      .incr(viralContentInNetworkCount)\n    statsReceiver\n      .scope(clientId)\n      .counter(\"ViralContentOutOfNetwork\")\n      .incr(viralContentOutOfNetworkCount)\n\n  }\n\n  def recordVideoDurationStats(\n    candidates: Seq[ItemCandidateWithDetails],\n    statsReceiver: StatsReceiver,\n    clientId: String\n  ): Unit = {\n\n    val lte10Sec = candidates.count(_.features.get(VideoDurationMsFeature).exists(_ <= 10000))\n    val gt60Sec = candidates.count(_.features.get(VideoDurationMsFeature).exists(_ > 60000))\n    val bt10And60Sec = candidates.count { candidate =>\n      candidate.features.get(VideoDurationMsFeature).exists { duration =>\n        duration > 10000 && duration <= 60000\n      }\n    }\n    val bt60And120Sec = candidates.count { candidate =>\n      candidate.features.get(VideoDurationMsFeature).exists { duration =>\n        duration > 60000 && duration <= 120000\n      }\n    }\n    val bt120And180Sec = candidates.count { candidate =>\n      candidate.features.get(VideoDurationMsFeature).exists { duration =>\n        duration > 120000 && duration <= 180000\n      }\n    }\n    val gt180Sec = candidates.count(_.features.get(VideoDurationMsFeature).exists(_ > 180000))\n\n    statsReceiver\n      .scope(clientId).counter(\"VideoLte10Sec\").incr(lte10Sec)\n    statsReceiver\n      .scope(clientId).counter(\"VideoGt60Sec\").incr(gt60Sec)\n    statsReceiver\n      .scope(clientId).counter(\"VideoBt10And60Sec\")\n      .incr(bt10And60Sec)\n    statsReceiver\n      .scope(clientId).counter(\"VideoBt60And120Sec\")\n      .incr(bt60And120Sec)\n    statsReceiver\n      .scope(clientId).counter(\"VideoBt120And180Sec\")\n      .incr(bt120And180Sec)\n    statsReceiver\n      .scope(clientId).counter(\"VideoGt180Sec\").incr(gt180Sec)\n  }\n  override val alerts = Seq(HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert())\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/filter\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector\",\n        \"product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice\",\n        \"src/thrift/com/twitter/hermit/candidate:hermit-candidate-scala\",\n        \"src/thrift/com/twitter/search:blender-scala\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/BlenderUsersCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users\n\nimport com.twitter.home_mixer.product.list_recommended_users.candidate_source.BlenderUsersCandidateSource\nimport com.twitter.home_mixer.product.list_recommended_users.feature_hydrator.IsGizmoduckValidUserFeatureHydrator\nimport com.twitter.home_mixer.product.list_recommended_users.feature_hydrator.IsSGSValidUserFeatureHydrator\nimport com.twitter.home_mixer.product.list_recommended_users.feature_hydrator.RecentListMembersFeature\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersQuery\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.user.UserCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.gate.EmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.search.blender.thriftscala.ThriftBlenderRequest\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass BlenderUsersCandidatePipelineConfig @Inject() (\n  blenderUsersCandidateSource: BlenderUsersCandidateSource,\n  isGizmoduckValidUserFeatureHydrator: IsGizmoduckValidUserFeatureHydrator,\n  isSGSValidUserFeatureHydrator: IsSGSValidUserFeatureHydrator)\n    extends CandidatePipelineConfig[\n      ListRecommendedUsersQuery,\n      ThriftBlenderRequest,\n      Long,\n      UserCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"BlenderUsers\")\n\n  override val gates: Seq[Gate[ListRecommendedUsersQuery]] =\n    Seq(EmptySeqFeatureGate(RecentListMembersFeature))\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ListRecommendedUsersQuery,\n    ThriftBlenderRequest\n  ] = BlenderUsersCandidatePipelineQueryTransformer\n\n  override val candidateSource: BaseCandidateSource[ThriftBlenderRequest, Long] =\n    blenderUsersCandidateSource\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    Long,\n    UserCandidate\n  ] = { candidate => UserCandidate(id = candidate) }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ListRecommendedUsersQuery, UserCandidate, _]\n  ] = Seq(\n    isGizmoduckValidUserFeatureHydrator,\n    isSGSValidUserFeatureHydrator\n  )\n\n  override val decorator: Option[CandidateDecorator[ListRecommendedUsersQuery, UserCandidate]] = {\n    val clientEventInfoBuilder = ClientEventInfoBuilder(\"user\")\n    val userItemBuilder = UserCandidateUrtItemBuilder(clientEventInfoBuilder)\n    Some(UrtItemCandidateDecorator(userItemBuilder))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/BlenderUsersCandidatePipelineQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users\n\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersQuery\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.search.adaptive.adaptive_results.thriftscala.ResultType\nimport com.twitter.search.blender.adaptive_search.thriftscala.AdaptiveSearchRequest\nimport com.twitter.search.blender.thriftscala.ThriftBlenderRequest\nimport com.twitter.search.blender.thriftscala.ThriftBlenderTweetypieOptions\nimport com.twitter.search.blender.thriftscala.ThriftBlenderWorkflowID\nimport com.twitter.search.common.constants.thriftscala.ThriftQuerySource\nimport com.twitter.spam.rtf.thriftscala.SafetyLevel\n\nobject BlenderUsersCandidatePipelineQueryTransformer\n    extends CandidatePipelineQueryTransformer[ListRecommendedUsersQuery, ThriftBlenderRequest] {\n\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"BlenderUsers\")\n\n  /**\n   * This is a user-defined descriptor used by Blender to track the source of traffic, and it\n   * is different from a client id, which is set during Finagle client construction.\n   */\n  private val ClientAppName = \"timelinemixer.list_recommended_users\"\n\n  override def transform(query: ListRecommendedUsersQuery): ThriftBlenderRequest = {\n\n    ThriftBlenderRequest(\n      workflowID = Some(ThriftBlenderWorkflowID.AdaptiveSearch),\n      userID = Some(query.getRequiredUserId), // perspectival\n      uiLang = query.clientContext.languageCode, // perspectival\n      clientAppName = Some(ClientAppName),\n      adaptiveSearchRequest = Some(\n        AdaptiveSearchRequest(\n          rawQuery = query.listName,\n          numResults = 40,\n          getPromotedContent = false,\n          resultFilter = Some(ResultType.User),\n        )\n      ),\n      querySource = Some(ThriftQuerySource.TypedQuery),\n      getCorrections = true,\n      tweetypieOptions = Some(\n        ThriftBlenderTweetypieOptions(\n          safetyLevel = Some(SafetyLevel.Recommendations)\n        )\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/ListMemberBasedUsersCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users\n\nimport com.twitter.hermit.candidate.{thriftscala => t}\nimport com.twitter.home_mixer.product.list_recommended_users.candidate_source.SimilarityBasedUsersCandidateSource\nimport com.twitter.home_mixer.product.list_recommended_users.feature_hydrator.IsGizmoduckValidUserFeatureHydrator\nimport com.twitter.home_mixer.product.list_recommended_users.feature_hydrator.IsListMemberFeatureHydrator\nimport com.twitter.home_mixer.product.list_recommended_users.feature_hydrator.IsSGSValidUserFeatureHydrator\nimport com.twitter.home_mixer.product.list_recommended_users.feature_hydrator.RecentListMembersFeature\nimport com.twitter.home_mixer.product.list_recommended_users.filter.DropMaxCandidatesByAggregatedScoreFilter\nimport com.twitter.home_mixer.product.list_recommended_users.filter.PreviouslyServedUsersFilter\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersFeatures.IsListMemberFeature\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersQuery\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.user.UserCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.filter.PredicateFeatureFilter\nimport com.twitter.product_mixer.component_library.gate.NonEmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ListMemberBasedUsersCandidatePipelineConfig @Inject() (\n  similarityBasedUsersCandidateSource: SimilarityBasedUsersCandidateSource,\n  isGizmoduckValidUserFeatureHydrator: IsGizmoduckValidUserFeatureHydrator,\n  isListMemberFeatureHydrator: IsListMemberFeatureHydrator,\n  isSGSValidUserFeatureHydrator: IsSGSValidUserFeatureHydrator)\n    extends CandidatePipelineConfig[\n      ListRecommendedUsersQuery,\n      Seq[Long],\n      t.Candidate,\n      UserCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ListMemberBasedUsers\")\n\n  override val gates: Seq[Gate[ListRecommendedUsersQuery]] =\n    Seq(NonEmptySeqFeatureGate(RecentListMembersFeature))\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[ListRecommendedUsersQuery, Seq[\n    Long\n  ]] = { query =>\n    query.features.map(_.getOrElse(RecentListMembersFeature, Seq.empty)).getOrElse(Seq.empty)\n  }\n\n  override val candidateSource: BaseCandidateSource[Seq[Long], t.Candidate] =\n    similarityBasedUsersCandidateSource\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[t.Candidate]\n  ] = Seq(ListMemberBasedUsersResponseFeatureTransfromer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    t.Candidate,\n    UserCandidate\n  ] = { candidate =>\n    UserCandidate(id = candidate.userId)\n  }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ListRecommendedUsersQuery, UserCandidate, _]\n  ] = Seq(isListMemberFeatureHydrator)\n\n  override val filters: Seq[Filter[ListRecommendedUsersQuery, UserCandidate]] =\n    Seq(\n      PreviouslyServedUsersFilter,\n      PredicateFeatureFilter.fromPredicate(\n        FilterIdentifier(\"IsListMember\"),\n        shouldKeepCandidate = { features => !features.getOrElse(IsListMemberFeature, false) }\n      ),\n      DropMaxCandidatesByAggregatedScoreFilter\n    )\n\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[ListRecommendedUsersQuery, UserCandidate, _]\n  ] = Seq(\n    isGizmoduckValidUserFeatureHydrator,\n    isSGSValidUserFeatureHydrator\n  )\n\n  override val decorator: Option[CandidateDecorator[ListRecommendedUsersQuery, UserCandidate]] = {\n    val clientEventInfoBuilder = ClientEventInfoBuilder(\"user\")\n    val userItemBuilder = UserCandidateUrtItemBuilder(clientEventInfoBuilder)\n    Some(UrtItemCandidateDecorator(userItemBuilder))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/ListMemberBasedUsersResponseFeatureTransfromer.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users\n\nimport com.twitter.hermit.candidate.{thriftscala => t}\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersFeatures.ScoreFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject ListMemberBasedUsersResponseFeatureTransfromer\n    extends CandidateFeatureTransformer[t.Candidate] {\n\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"ListMemberBasedUsers\")\n\n  override val features: Set[Feature[_, _]] = Set(ScoreFeature)\n\n  override def transform(candidate: t.Candidate): FeatureMap = FeatureMapBuilder()\n    .add(ScoreFeature, candidate.score)\n    .build()\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/ListRecommendedUsersMixerPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users\n\nimport com.twitter.home_mixer.product.list_recommended_users.feature_hydrator.RecentListMembersQueryFeatureHydrator\nimport com.twitter.home_mixer.product.list_recommended_users.gate.ViewerIsListOwnerGate\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersFeatures.IsGizmoduckValidUserFeature\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersFeatures.IsSGSValidUserFeature\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersQuery\nimport com.twitter.home_mixer.product.list_recommended_users.param.ListRecommendedUsersParam.ExcludedIdsMaxLengthParam\nimport com.twitter.home_mixer.product.list_recommended_users.param.ListRecommendedUsersParam.ServerMaxResultsParam\nimport com.twitter.product_mixer.component_library.premarshaller.urt.UrtDomainMarshaller\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.AddEntriesWithReplaceInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceAllEntries\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceEntryInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.StaticTimelineScribeConfigBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UnorderedExcludeIdsBottomCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtMetadataBuilder\nimport com.twitter.product_mixer.component_library.selector.DropFilteredCandidates\nimport com.twitter.product_mixer.component_library.selector.DropMaxCandidates\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.UrtTransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.MixerPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.UserItem\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineConfig\nimport com.twitter.timelines.render.{thriftscala => urt}\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ListRecommendedUsersMixerPipelineConfig @Inject() (\n  listMemberBasedUsersCandidatePipelineConfig: ListMemberBasedUsersCandidatePipelineConfig,\n  blenderUsersCandidatePipelineConfig: BlenderUsersCandidatePipelineConfig,\n  viewerIsListOwnerGate: ViewerIsListOwnerGate,\n  recentListMembersQueryFeatureHydrator: RecentListMembersQueryFeatureHydrator,\n  urtTransportMarshaller: UrtTransportMarshaller)\n    extends MixerPipelineConfig[ListRecommendedUsersQuery, Timeline, urt.TimelineResponse] {\n\n  override val identifier: MixerPipelineIdentifier = MixerPipelineIdentifier(\"ListRecommendedUsers\")\n\n  override val gates = Seq(viewerIsListOwnerGate)\n\n  override val fetchQueryFeatures: Seq[QueryFeatureHydrator[ListRecommendedUsersQuery]] =\n    Seq(recentListMembersQueryFeatureHydrator)\n\n  override val candidatePipelines: Seq[\n    CandidatePipelineConfig[ListRecommendedUsersQuery, _, _, _]\n  ] = Seq(\n    listMemberBasedUsersCandidatePipelineConfig,\n    blenderUsersCandidatePipelineConfig\n  )\n\n  private val candidatePipelineIdentifiers = Set(\n    listMemberBasedUsersCandidatePipelineConfig.identifier,\n    blenderUsersCandidatePipelineConfig.identifier\n  )\n\n  override val resultSelectors: Seq[Selector[ListRecommendedUsersQuery]] = Seq(\n    DropFilteredCandidates(\n      candidatePipelines = candidatePipelineIdentifiers,\n      filter = candidate =>\n        candidate.features.getOrElse(IsSGSValidUserFeature, false) &&\n          candidate.features.getOrElse(IsGizmoduckValidUserFeature, false)\n    ),\n    DropMaxCandidates(\n      candidatePipelines = candidatePipelineIdentifiers,\n      maxSelectionsParam = ServerMaxResultsParam),\n    InsertAppendResults(candidatePipelineIdentifiers)\n  )\n\n  override val domainMarshaller: DomainMarshaller[ListRecommendedUsersQuery, Timeline] = {\n    val instructionBuilders = Seq(\n      ReplaceEntryInstructionBuilder(ReplaceAllEntries),\n      AddEntriesWithReplaceInstructionBuilder()\n    )\n\n    val metadataBuilder = UrtMetadataBuilder(\n      title = None,\n      scribeConfigBuilder = Some(\n        StaticTimelineScribeConfigBuilder(\n          TimelineScribeConfig(\n            page = Some(\"list_recommended_users\"),\n            section = None,\n            entityToken = None)))\n    )\n\n    val excludeIdsSelector: PartialFunction[UniversalNoun[_], Long] = {\n      case item: UserItem => item.id\n    }\n\n    val cursorBuilder = UnorderedExcludeIdsBottomCursorBuilder(\n      excludedIdsMaxLengthParam = ExcludedIdsMaxLengthParam,\n      excludeIdsSelector = excludeIdsSelector)\n\n    UrtDomainMarshaller(\n      instructionBuilders = instructionBuilders,\n      metadataBuilder = Some(metadataBuilder),\n      cursorBuilders = Seq(cursorBuilder)\n    )\n  }\n\n  override val transportMarshaller: TransportMarshaller[Timeline, urt.TimelineResponse] =\n    urtTransportMarshaller\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/ListRecommendedUsersProductPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.marshaller.timelines.RecommendedUsersCursorUnmarshaller\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.model.request.ListRecommendedUsersProduct\nimport com.twitter.home_mixer.model.request.ListRecommendedUsersProductContext\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersQuery\nimport com.twitter.home_mixer.product.list_recommended_users.param.ListRecommendedUsersParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.product.list_recommended_users.param.ListRecommendedUsersParamConfig\nimport com.twitter.home_mixer.service.HomeMixerAccessPolicy.DefaultHomeMixerAccessPolicy\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig.DefaultNotificationGroup\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AccessPolicy\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfBelow\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfLatencyAbove\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.common.alert.LatencyAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.P99\nimport com.twitter.product_mixer.core.functional_component.common.alert.SuccessRateAlert\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ProductPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request\nimport com.twitter.product_mixer.core.pipeline.PipelineConfig\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.BadRequest\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineConfig\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport com.twitter.timelines.util.RequestCursorSerializer\nimport com.twitter.util.Try\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ListRecommendedUsersProductPipelineConfig @Inject() (\n  listRecommendedUsersMixerPipelineConfig: ListRecommendedUsersMixerPipelineConfig,\n  listRecommendedUsersParamConfig: ListRecommendedUsersParamConfig)\n    extends ProductPipelineConfig[\n      HomeMixerRequest,\n      ListRecommendedUsersQuery,\n      urt.TimelineResponse\n    ] {\n\n  override val identifier: ProductPipelineIdentifier =\n    ProductPipelineIdentifier(\"ListRecommendedUsers\")\n  override val product: request.Product = ListRecommendedUsersProduct\n  override val paramConfig: ProductParamConfig = listRecommendedUsersParamConfig\n\n  override def pipelineQueryTransformer(\n    request: HomeMixerRequest,\n    params: Params\n  ): ListRecommendedUsersQuery = {\n    val context = request.productContext match {\n      case Some(context: ListRecommendedUsersProductContext) => context\n      case _ => throw PipelineFailure(BadRequest, \"ListRecommendedUsersProductContext not found\")\n    }\n\n    val debugOptions = request.debugParams.flatMap(_.debugOptions)\n\n    val pipelineCursor = request.serializedRequestCursor.flatMap { cursor =>\n      Try(UrtCursorSerializer.deserializeUnorderedExcludeIdsCursor(cursor))\n        .getOrElse(RecommendedUsersCursorUnmarshaller(RequestCursorSerializer.deserialize(cursor)))\n    }\n\n    ListRecommendedUsersQuery(\n      listId = context.listId,\n      params = params,\n      clientContext = request.clientContext,\n      features = None,\n      pipelineCursor = pipelineCursor,\n      requestedMaxResults = Some(params(ServerMaxResultsParam)),\n      debugOptions = debugOptions,\n      selectedUserIds = context.selectedUserIds,\n      excludedUserIds = context.excludedUserIds,\n      listName = context.listName\n    )\n  }\n\n  override def pipelines: Seq[PipelineConfig] = Seq(listRecommendedUsersMixerPipelineConfig)\n\n  override def pipelineSelector(query: ListRecommendedUsersQuery): ComponentIdentifier =\n    listRecommendedUsersMixerPipelineConfig.identifier\n\n  override val alerts: Seq[Alert] = Seq(\n    SuccessRateAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfBelow(99.9, 20, 30),\n      criticalPredicate = TriggerIfBelow(99.9, 30, 30),\n    ),\n    LatencyAlert(\n      notificationGroup = DefaultNotificationGroup,\n      percentile = P99,\n      warnPredicate = TriggerIfLatencyAbove(1000.millis, 15, 30),\n      criticalPredicate = TriggerIfLatencyAbove(1500.millis, 15, 30)\n    )\n  )\n\n  override val debugAccessPolicies: Set[AccessPolicy] = DefaultHomeMixerAccessPolicy\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/candidate_source/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"src/thrift/com/twitter/hermit/candidate:hermit-candidate-scala\",\n        \"src/thrift/com/twitter/search:blender-scala\",\n        \"strato/config/columns/recommendations/similarity:similarity-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/candidate_source/BlenderUsersCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.candidate_source\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.search.adaptive.adaptive_results.thriftscala.AdaptiveSearchResultData\nimport com.twitter.search.adaptive.adaptive_results.thriftscala.Result\nimport com.twitter.search.adaptive.adaptive_results.thriftscala.ResultData\nimport com.twitter.search.blender.adaptive_search.thriftscala.AdaptiveSearchResponse\nimport com.twitter.search.blender.adaptive_search.thriftscala.Container\nimport com.twitter.search.blender.thriftscala.BlenderService\nimport com.twitter.search.blender.thriftscala.ThriftBlenderRequest\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass BlenderUsersCandidateSource @Inject() (\n  blenderClient: BlenderService.MethodPerEndpoint)\n    extends CandidateSource[ThriftBlenderRequest, Long] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"BlenderUsers\")\n\n  override def apply(request: ThriftBlenderRequest): Stitch[Seq[Long]] = {\n    Stitch.callFuture(\n      blenderClient.serveV2(request).map { response =>\n        val userIdsOpt =\n          response.adaptiveSearchResponse.map(extractUserIdsFromAdaptiveSearchResponse)\n        userIdsOpt.getOrElse(Seq.empty)\n      }\n    )\n  }\n\n  private def extractUserIdsFromAdaptiveSearchResponse(\n    response: AdaptiveSearchResponse\n  ): Seq[Long] = {\n    response match {\n      case AdaptiveSearchResponse(Some(Seq(Container(Some(results), _))), _, _) =>\n        results.map(_.data).collect {\n          case AdaptiveSearchResultData.Result(Result(ResultData.User(user), _)) =>\n            user.id\n        }\n      case _ => Seq.empty\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/candidate_source/SimilarityBasedUsersCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.candidate_source\n\nimport com.twitter.hermit.candidate.{thriftscala => t}\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.recommendations.similarity.SimilarUsersBySimsOnUserClientColumn\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SimilarityBasedUsersCandidateSource @Inject() (\n  similarUsersBySimsOnUserClientColumn: SimilarUsersBySimsOnUserClientColumn)\n    extends CandidateSource[Seq[Long], t.Candidate] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"SimilarityBasedUsers\")\n\n  private val fetcher: Fetcher[Long, Unit, t.Candidates] =\n    similarUsersBySimsOnUserClientColumn.fetcher\n\n  private val MaxCandidatesToKeep = 4000\n\n  override def apply(request: Seq[Long]): Stitch[Seq[t.Candidate]] = {\n    Stitch\n      .collect {\n        request.map { userId =>\n          fetcher\n            .fetch(userId, Unit).map { result =>\n              result.v.map(_.candidates).getOrElse(Seq.empty)\n            }.map { candidates =>\n              val sortedCandidates = candidates.sortBy(-_.score)\n              sortedCandidates.take(MaxCandidatesToKeep)\n            }\n        }\n      }.map(_.flatten)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/feature_hydrator/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/model\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/gizmoduck:thrift-scala\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"stitch/stitch-gizmoduck\",\n        \"stitch/stitch-socialgraph\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/feature_hydrator/IsGizmoduckValidUserFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.feature_hydrator\n\nimport com.twitter.gizmoduck.{thriftscala => gt}\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersFeatures.IsGizmoduckValidUserFeature\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.spam.rtf.{thriftscala => rtf}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.gizmoduck.Gizmoduck\nimport com.twitter.util.Return\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass IsGizmoduckValidUserFeatureHydrator @Inject() (gizmoduck: Gizmoduck)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, UserCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"IsGizmoduckValidUser\")\n\n  override val features: Set[Feature[_, _]] = Set(IsGizmoduckValidUserFeature)\n\n  private val queryFields: Set[gt.QueryFields] = Set(gt.QueryFields.Safety)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[UserCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val context = gt.LookupContext(\n      forUserId = query.getOptionalUserId,\n      includeProtected = true,\n      safetyLevel = Some(rtf.SafetyLevel.Recommendations)\n    )\n    val userIds = candidates.map(_.candidate.id)\n\n    Stitch\n      .collectToTry(\n        userIds.map(userId => gizmoduck.getUserById(userId, queryFields, context))).map {\n        userResults =>\n          val idToUserSafetyMap = userResults\n            .collect {\n              case Return(user) => user\n            }.map(user => user.id -> user.safety).toMap\n\n          candidates.map { candidate =>\n            val safety = idToUserSafetyMap.getOrElse(candidate.candidate.id, None)\n            val isValidUser = safety.isDefined &&\n              !safety.exists(_.deactivated) &&\n              !safety.exists(_.suspended) &&\n              !safety.exists(_.isProtected) &&\n              !safety.flatMap(_.offboarded).getOrElse(false)\n\n            FeatureMapBuilder()\n              .add(IsGizmoduckValidUserFeature, isValidUser)\n              .build()\n          }\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/feature_hydrator/IsListMemberFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.feature_hydrator\n\nimport com.twitter.home_mixer.model.request.HasListId\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersFeatures.IsListMemberFeature\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.socialgraph.{thriftscala => sg}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.SocialGraph\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass IsListMemberFeatureHydrator @Inject() (socialGraph: SocialGraph)\n    extends BulkCandidateFeatureHydrator[PipelineQuery with HasListId, UserCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"IsListMember\")\n\n  override val features: Set[Feature[_, _]] = Set(IsListMemberFeature)\n\n  override def apply(\n    query: PipelineQuery with HasListId,\n    candidates: Seq[CandidateWithFeatures[UserCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val userIds = candidates.map(_.candidate.id)\n    val request = sg.IdsRequest(\n      relationships = Seq(\n        sg.SrcRelationship(\n          source = query.listId,\n          relationshipType = sg.RelationshipType.ListHasMember,\n          hasRelationship = true,\n          targets = Some(userIds))),\n      pageRequest = Some(sg.PageRequest(selectAll = Some(true)))\n    )\n\n    socialGraph.ids(request).map(_.ids).map { listMembers =>\n      val listMembersSet = listMembers.toSet\n      candidates.map { candidate =>\n        FeatureMapBuilder()\n          .add(IsListMemberFeature, listMembersSet.contains(candidate.candidate.id))\n          .build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/feature_hydrator/IsSGSValidUserFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.feature_hydrator\n\nimport com.twitter.home_mixer.model.request.HasListId\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersFeatures.IsSGSValidUserFeature\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.socialgraph.{thriftscala => sg}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.SocialGraph\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass IsSGSValidUserFeatureHydrator @Inject() (socialGraph: SocialGraph)\n    extends BulkCandidateFeatureHydrator[PipelineQuery with HasListId, UserCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"IsSGSValidUser\")\n\n  override def features: Set[Feature[_, _]] = Set(IsSGSValidUserFeature)\n\n  override def apply(\n    query: PipelineQuery with HasListId,\n    candidates: Seq[CandidateWithFeatures[UserCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val sourceId = query.getRequiredUserId\n    val targetUserIds = candidates.map(_.candidate.id)\n    val request = sg.IdsRequest(\n      relationships = Seq(\n        sg.SrcRelationship(\n          source = sourceId,\n          relationshipType = sg.RelationshipType.Blocking,\n          hasRelationship = true,\n          targets = Some(targetUserIds)),\n        sg.SrcRelationship(\n          source = sourceId,\n          relationshipType = sg.RelationshipType.BlockedBy,\n          hasRelationship = true,\n          targets = Some(targetUserIds)),\n        sg.SrcRelationship(\n          source = sourceId,\n          relationshipType = sg.RelationshipType.Muting,\n          hasRelationship = true,\n          targets = Some(targetUserIds))\n      ),\n      pageRequest = Some(sg.PageRequest(selectAll = Some(true))),\n      context = Some(sg.LookupContext(performUnion = Some(true)))\n    )\n\n    socialGraph.ids(request).map(_.ids).map(_.toSet).map { hasRelationshipUserIds =>\n      candidates.map { candidate =>\n        FeatureMapBuilder()\n          .add(IsSGSValidUserFeature, !hasRelationshipUserIds.contains(candidate.candidate.id))\n          .build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/feature_hydrator/RecentListMembersQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.feature_hydrator\n\nimport com.twitter.home_mixer.model.request.HasListId\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.socialgraph.{thriftscala => sg}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.SocialGraph\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase object RecentListMembersFeature extends FeatureWithDefaultOnFailure[PipelineQuery, Seq[Long]] {\n  override val defaultValue: Seq[Long] = Seq.empty\n}\n\n@Singleton\nclass RecentListMembersQueryFeatureHydrator @Inject() (socialGraph: SocialGraph)\n    extends QueryFeatureHydrator[PipelineQuery with HasListId] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RecentListMembers\")\n\n  override val features: Set[Feature[_, _]] = Set(RecentListMembersFeature)\n\n  private val MaxRecentMembers = 10\n\n  override def hydrate(query: PipelineQuery with HasListId): Stitch[FeatureMap] = {\n    val request = sg.IdsRequest(\n      relationships = Seq(sg\n        .SrcRelationship(query.listId, sg.RelationshipType.ListHasMember, hasRelationship = true)),\n      pageRequest = Some(sg.PageRequest(selectAll = Some(true), count = Some(MaxRecentMembers)))\n    )\n    socialGraph.ids(request).map(_.ids).map { listMembers =>\n      FeatureMapBuilder().add(RecentListMembersFeature, listMembers).build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/filter/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/model\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n    ],\n    exports = [],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/filter/DropMaxCandidatesByAggregatedScoreFilter.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.filter\n\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersFeatures.ScoreFeature\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject DropMaxCandidatesByAggregatedScoreFilter extends Filter[PipelineQuery, UserCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"DropMaxCandidatesByAggregatedScore\")\n\n  private val MaxSimilarUserCandidates = 150\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[UserCandidate]]\n  ): Stitch[FilterResult[UserCandidate]] = {\n    val userIdToAggregatedScoreMap = candidates\n      .groupBy(_.candidate.id)\n      .map {\n        case (userId, candidates) =>\n          val aggregatedScore = candidates.map(_.features.getOrElse(ScoreFeature, 0.0)).sum\n          (userId, aggregatedScore)\n      }\n\n    val sortedCandidates = candidates.sortBy(candidate =>\n      -userIdToAggregatedScoreMap.getOrElse(candidate.candidate.id, 0.0))\n\n    val (kept, removed) = sortedCandidates.map(_.candidate).splitAt(MaxSimilarUserCandidates)\n\n    Stitch.value(FilterResult(kept, removed))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/filter/PreviouslyServedUsersFilter.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.filter\n\nimport com.twitter.home_mixer.product.list_recommended_users.feature_hydrator.RecentListMembersFeature\nimport com.twitter.home_mixer.product.list_recommended_users.model.ListRecommendedUsersQuery\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.stitch.Stitch\n\nobject PreviouslyServedUsersFilter extends Filter[ListRecommendedUsersQuery, UserCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"PreviouslyServedUsers\")\n\n  override def apply(\n    query: ListRecommendedUsersQuery,\n    candidates: Seq[CandidateWithFeatures[UserCandidate]]\n  ): Stitch[FilterResult[UserCandidate]] = {\n\n    val recentListMembers = query.features.map(_.getOrElse(RecentListMembersFeature, Seq.empty))\n\n    val servedUserIds = query.pipelineCursor.map(_.excludedIds)\n\n    val excludedUserIds = (recentListMembers.getOrElse(Seq.empty) ++\n      query.selectedUserIds.getOrElse(Seq.empty) ++\n      query.excludedUserIds.getOrElse(Seq.empty) ++\n      servedUserIds.getOrElse(Seq.empty)).toSet\n\n    val (removed, kept) =\n      candidates.map(_.candidate).partition(candidate => excludedUserIds.contains(candidate.id))\n\n    Stitch.value(FilterResult(kept = kept, removed = removed))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/gate/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"stitch/stitch-socialgraph\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/gate/ViewerIsListOwnerGate.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.gate\n\nimport com.twitter.home_mixer.model.request.HasListId\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.socialgraph.{thriftscala => sg}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.SocialGraph\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class ViewerIsListOwnerGate @Inject() (socialGraph: SocialGraph)\n    extends Gate[PipelineQuery with HasListId] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"ViewerIsListOwner\")\n\n  private val relationship = sg.Relationship(relationshipType = sg.RelationshipType.ListOwning)\n\n  override def shouldContinue(query: PipelineQuery with HasListId): Stitch[Boolean] = {\n    val request = sg.ExistsRequest(\n      source = query.getRequiredUserId,\n      target = query.listId,\n      relationships = Seq(relationship))\n    socialGraph.exists(request).map(_.exists)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/model/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/model/ListRecommendedUsersFeatures.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.model\n\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\n\nobject ListRecommendedUsersFeatures {\n  // Candidate features\n  object IsGizmoduckValidUserFeature extends Feature[UserCandidate, Boolean]\n  object IsListMemberFeature extends Feature[UserCandidate, Boolean]\n  object IsSGSValidUserFeature extends Feature[UserCandidate, Boolean]\n  object ScoreFeature extends Feature[UserCandidate, Double]\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/model/ListRecommendedUsersQuery.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.model\n\nimport com.twitter.home_mixer.model.request.HasListId\nimport com.twitter.home_mixer.model.request.ListRecommendedUsersProduct\nimport com.twitter.product_mixer.component_library.model.cursor.UrtUnorderedExcludeIdsCursor\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.request._\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Params\n\ncase class ListRecommendedUsersQuery(\n  override val listId: Long,\n  override val params: Params,\n  override val clientContext: ClientContext,\n  override val pipelineCursor: Option[UrtUnorderedExcludeIdsCursor],\n  override val requestedMaxResults: Option[Int],\n  override val debugOptions: Option[DebugOptions],\n  override val features: Option[FeatureMap],\n  selectedUserIds: Option[Seq[Long]],\n  excludedUserIds: Option[Seq[Long]],\n  listName: Option[String])\n    extends PipelineQuery\n    with HasPipelineCursor[UrtUnorderedExcludeIdsCursor]\n    with HasListId {\n\n  override val product: Product = ListRecommendedUsersProduct\n\n  override def withFeatureMap(features: FeatureMap): ListRecommendedUsersQuery =\n    copy(features = Some(features))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/param/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core/src/main/scala/com/twitter/timelines/configapi\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param/decider\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/param/ListRecommendedUsersParam.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.param\n\nimport com.twitter.timelines.configapi.FSBoundedParam\n\nobject ListRecommendedUsersParam {\n  val SupportedClientFSName = \"list_recommended_users_supported_client\"\n\n  object ServerMaxResultsParam\n      extends FSBoundedParam[Int](\n        name = \"list_recommended_users_server_max_results\",\n        default = 10,\n        min = 1,\n        max = 500\n      )\n\n  object ExcludedIdsMaxLengthParam\n      extends FSBoundedParam[Int](\n        name = \"list_recommended_users_excluded_ids_max_length\",\n        default = 2000,\n        min = 0,\n        max = 5000\n      )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_recommended_users/param/ListRecommendedUsersParamConfig.scala",
    "content": "package com.twitter.home_mixer.product.list_recommended_users.param\n\nimport com.twitter.home_mixer.param.decider.DeciderKey\nimport com.twitter.home_mixer.product.list_recommended_users.param.ListRecommendedUsersParam.ExcludedIdsMaxLengthParam\nimport com.twitter.home_mixer.product.list_recommended_users.param.ListRecommendedUsersParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.product.list_recommended_users.param.ListRecommendedUsersParam.SupportedClientFSName\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.servo.decider.DeciderKeyName\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ListRecommendedUsersParamConfig @Inject() () extends ProductParamConfig {\n  override val enabledDeciderKey: DeciderKeyName = DeciderKey.EnableListRecommendedUsersProduct\n  override val supportedClientFSName: String = SupportedClientFSName\n\n  override val boundedIntFSOverrides = Seq(\n    ServerMaxResultsParam,\n    ExcludedIdsMaxLengthParam\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/urt/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/decorator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_service\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/param_gated\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect\",\n        \"product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n        \"timelines/src/main/scala/com/twitter/timelines/injection/scribe\",\n    ],\n    exports = [\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/ListTweetsAdsCandidatePipelineBuilder.scala",
    "content": "package com.twitter.home_mixer.product.list_tweets\n\nimport com.twitter.adserver.{thriftscala => ads}\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeAdsClientEventDetailsBuilder\nimport com.twitter.home_mixer.functional_component.gate.ExcludeSoftUserGate\nimport com.twitter.home_mixer.param.HomeGlobalParams\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableAdvertiserBrandSafetySettingsFeatureHydratorParam\nimport com.twitter.home_mixer.product.list_tweets.model.ListTweetsQuery\nimport com.twitter.home_mixer.product.list_tweets.param.ListTweetsParam.EnableAdsCandidatePipelineParam\nimport com.twitter.product_mixer.component_library.candidate_source.ads.AdsProdThriftCandidateSource\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref.ContextualTweetRefBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.ad.AdsCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.ads.AdvertiserBrandSafetySettingsFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.gate.NonEmptyCandidatesGate\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.AdsDependentCandidatePipelineConfig\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.AdsDependentCandidatePipelineConfigBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.CountCandidatesFromPipelines\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.StaticAdsDisplayLocationBuilder\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.ValidAdImpressionIdFilter\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.gate.ParamNotGate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.TimelineHomePromotedHydrationSafetyLevel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.TweetHydrationContext\nimport com.twitter.timelines.injection.scribe.InjectionScribeUtil\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ListTweetsAdsCandidatePipelineBuilder @Inject() (\n  adsCandidatePipelineConfigBuilder: AdsDependentCandidatePipelineConfigBuilder,\n  adsCandidateSource: AdsProdThriftCandidateSource,\n  advertiserBrandSafetySettingsFeatureHydrator: AdvertiserBrandSafetySettingsFeatureHydrator[\n    ListTweetsQuery,\n    AdsCandidate\n  ]) {\n\n  private val identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\"ListTweetsAds\")\n\n  private val suggestType = st.SuggestType.Promoted\n\n  private val clientEventInfoBuilder = ClientEventInfoBuilder(\n    component = InjectionScribeUtil.scribeComponent(suggestType).get,\n    detailsBuilder = Some(HomeAdsClientEventDetailsBuilder(Some(suggestType.name)))\n  )\n\n  private val contextualTweetRefBuilder = ContextualTweetRefBuilder(\n    TweetHydrationContext(\n      safetyLevelOverride = Some(TimelineHomePromotedHydrationSafetyLevel),\n      outerTweetContext = None\n    ))\n\n  private val decorator = UrtItemCandidateDecorator(\n    AdsCandidateUrtItemBuilder(\n      tweetClientEventInfoBuilder = Some(clientEventInfoBuilder),\n      contextualTweetRefBuilder = Some(contextualTweetRefBuilder)\n    )\n  )\n\n  def build(\n    organicCandidatePipelines: CandidateScope\n  ): AdsDependentCandidatePipelineConfig[ListTweetsQuery] =\n    adsCandidatePipelineConfigBuilder.build[ListTweetsQuery](\n      adsCandidateSource = adsCandidateSource,\n      identifier = identifier,\n      adsDisplayLocationBuilder =\n        StaticAdsDisplayLocationBuilder(ads.DisplayLocation.TimelineHomeReverseChron),\n      countNumOrganicItems = CountCandidatesFromPipelines(organicCandidatePipelines),\n      supportedClientParam = Some(EnableAdsCandidatePipelineParam),\n      gates = Seq(\n        ParamNotGate(\n          name = \"AdsDisableInjectionBasedOnUserRole\",\n          param = HomeGlobalParams.AdsDisableInjectionBasedOnUserRoleParam\n        ),\n        ExcludeSoftUserGate,\n        NonEmptyCandidatesGate(organicCandidatePipelines)\n      ),\n      filters = Seq(ValidAdImpressionIdFilter),\n      postFilterFeatureHydration = Seq(\n        ParamGatedCandidateFeatureHydrator(\n          EnableAdvertiserBrandSafetySettingsFeatureHydratorParam,\n          advertiserBrandSafetySettingsFeatureHydrator\n        )\n      ),\n      decorator = Some(decorator),\n      urtRequest = Some(true),\n    )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/ListTweetsMixerPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.list_tweets\n\nimport com.twitter.clientapp.{thriftscala => ca}\nimport com.twitter.goldfinch.api.AdsInjectionSurfaceAreas\nimport com.twitter.home_mixer.candidate_pipeline.ConversationServiceCandidatePipelineConfigBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator.RequestQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.side_effect.HomeScribeClientEventSideEffect\nimport com.twitter.home_mixer.model.GapIncludeInstruction\nimport com.twitter.home_mixer.param.HomeMixerFlagName.ScribeClientEventsFlag\nimport com.twitter.home_mixer.product.list_tweets.decorator.ListConversationServiceCandidateDecorator\nimport com.twitter.home_mixer.product.list_tweets.model.ListTweetsQuery\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.gate.NonEmptyCandidatesGate\nimport com.twitter.product_mixer.component_library.premarshaller.urt.UrtDomainMarshaller\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.AddEntriesWithReplaceAndShowAlertInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedBottomCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedGapCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedTopCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceAllEntries\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceEntryInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ShowAlertInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.StaticTimelineScribeConfigBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtMetadataBuilder\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.component_library.selector.UpdateSortCandidates\nimport com.twitter.product_mixer.component_library.selector.ads.AdsInjector\nimport com.twitter.product_mixer.component_library.selector.ads.InsertAdResults\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.UrtTransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.MixerPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineConfig\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ListTweetsMixerPipelineConfig @Inject() (\n  listTweetsTimelineServiceCandidatePipelineConfig: ListTweetsTimelineServiceCandidatePipelineConfig,\n  conversationServiceCandidatePipelineConfigBuilder: ConversationServiceCandidatePipelineConfigBuilder[\n    ListTweetsQuery\n  ],\n  listTweetsAdsCandidatePipelineBuilder: ListTweetsAdsCandidatePipelineBuilder,\n  requestQueryFeatureHydrator: RequestQueryFeatureHydrator[ListTweetsQuery],\n  sgsFollowedUsersQueryFeatureHydrator: SGSFollowedUsersQueryFeatureHydrator,\n  adsInjector: AdsInjector,\n  clientEventsScribeEventPublisher: EventPublisher[ca.LogEvent],\n  urtTransportMarshaller: UrtTransportMarshaller,\n  @Flag(ScribeClientEventsFlag) enableScribeClientEvents: Boolean)\n    extends MixerPipelineConfig[ListTweetsQuery, Timeline, urt.TimelineResponse] {\n\n  override val identifier: MixerPipelineIdentifier = MixerPipelineIdentifier(\"ListTweets\")\n\n  private val conversationServiceCandidatePipelineConfig =\n    conversationServiceCandidatePipelineConfigBuilder.build(\n      Seq(\n        NonEmptyCandidatesGate(\n          SpecificPipelines(listTweetsTimelineServiceCandidatePipelineConfig.identifier))\n      ),\n      ListConversationServiceCandidateDecorator()\n    )\n\n  private val listTweetsAdsCandidatePipelineConfig = listTweetsAdsCandidatePipelineBuilder.build(\n    SpecificPipelines(listTweetsTimelineServiceCandidatePipelineConfig.identifier)\n  )\n\n  override val candidatePipelines: Seq[CandidatePipelineConfig[ListTweetsQuery, _, _, _]] =\n    Seq(listTweetsTimelineServiceCandidatePipelineConfig)\n\n  override val dependentCandidatePipelines: Seq[\n    DependentCandidatePipelineConfig[ListTweetsQuery, _, _, _]\n  ] =\n    Seq(conversationServiceCandidatePipelineConfig, listTweetsAdsCandidatePipelineConfig)\n\n  override val failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy] = Map(\n    conversationServiceCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    listTweetsAdsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always)\n\n  override val resultSelectors: Seq[Selector[ListTweetsQuery]] = Seq(\n    UpdateSortCandidates(\n      ordering = CandidatesUtil.reverseChronTweetsOrdering,\n      candidatePipeline = conversationServiceCandidatePipelineConfig.identifier\n    ),\n    InsertAppendResults(candidatePipeline = conversationServiceCandidatePipelineConfig.identifier),\n    InsertAdResults(\n      surfaceAreaName = AdsInjectionSurfaceAreas.HomeTimeline,\n      adsInjector = adsInjector.forSurfaceArea(AdsInjectionSurfaceAreas.HomeTimeline),\n      adsCandidatePipeline = listTweetsAdsCandidatePipelineConfig.identifier\n    ),\n  )\n\n  override val fetchQueryFeatures: Seq[QueryFeatureHydrator[ListTweetsQuery]] = Seq(\n    requestQueryFeatureHydrator,\n    sgsFollowedUsersQueryFeatureHydrator\n  )\n\n  private val homeScribeClientEventSideEffect = HomeScribeClientEventSideEffect(\n    enableScribeClientEvents = enableScribeClientEvents,\n    logPipelinePublisher = clientEventsScribeEventPublisher,\n    injectedTweetsCandidatePipelineIdentifiers =\n      Seq(conversationServiceCandidatePipelineConfig.identifier),\n    adsCandidatePipelineIdentifier = Some(listTweetsAdsCandidatePipelineConfig.identifier),\n  )\n\n  override val resultSideEffects: Seq[PipelineResultSideEffect[ListTweetsQuery, Timeline]] =\n    Seq(homeScribeClientEventSideEffect)\n\n  override val domainMarshaller: DomainMarshaller[ListTweetsQuery, Timeline] = {\n    val instructionBuilders = Seq(\n      ReplaceEntryInstructionBuilder(ReplaceAllEntries),\n      AddEntriesWithReplaceAndShowAlertInstructionBuilder(),\n      ShowAlertInstructionBuilder()\n    )\n\n    val idSelector: PartialFunction[UniversalNoun[_], Long] = {\n      // exclude ads while determining tweet cursor values\n      case item: TweetItem if item.promotedMetadata.isEmpty => item.id\n      case module: TimelineModule\n          if module.items.headOption.exists(_.item.isInstanceOf[TweetItem]) =>\n        module.items.last.item match {\n          case item: TweetItem => item.id\n        }\n    }\n\n    val topCursorBuilder = OrderedTopCursorBuilder(idSelector)\n    val bottomCursorBuilder =\n      OrderedBottomCursorBuilder(idSelector, GapIncludeInstruction.inverse())\n    val gapCursorBuilder = OrderedGapCursorBuilder(idSelector, GapIncludeInstruction)\n\n    val metadataBuilder = UrtMetadataBuilder(\n      title = None,\n      scribeConfigBuilder = Some(\n        StaticTimelineScribeConfigBuilder(\n          TimelineScribeConfig(page = Some(\"list_tweets\"), section = None, entityToken = None)))\n    )\n\n    UrtDomainMarshaller(\n      instructionBuilders = instructionBuilders,\n      metadataBuilder = Some(metadataBuilder),\n      cursorBuilders = Seq(topCursorBuilder, bottomCursorBuilder, gapCursorBuilder)\n    )\n  }\n\n  override val transportMarshaller: TransportMarshaller[Timeline, urt.TimelineResponse] =\n    urtTransportMarshaller\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/ListTweetsProductPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.list_tweets\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.marshaller.timelines.ChronologicalCursorUnmarshaller\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.model.request.ListTweetsProduct\nimport com.twitter.home_mixer.model.request.ListTweetsProductContext\nimport com.twitter.home_mixer.product.list_tweets.model.ListTweetsQuery\nimport com.twitter.home_mixer.product.list_tweets.param.ListTweetsParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.product.list_tweets.param.ListTweetsParamConfig\nimport com.twitter.home_mixer.service.HomeMixerAccessPolicy.DefaultHomeMixerAccessPolicy\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig.DefaultNotificationGroup\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AccessPolicy\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.common.alert.EmptyResponseRateAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.LatencyAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.P99\nimport com.twitter.product_mixer.core.functional_component.common.alert.SuccessRateAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.ThroughputAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfAbove\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfBelow\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfLatencyAbove\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ProductPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineConfig\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.BadRequest\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MalformedCursor\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineConfig\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.product_mixer.core.util.SortIndexBuilder\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport com.twitter.timelines.util.RequestCursorSerializer\nimport com.twitter.util.Time\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ListTweetsProductPipelineConfig @Inject() (\n  listTweetsMixerPipelineConfig: ListTweetsMixerPipelineConfig,\n  listTweetsParamConfig: ListTweetsParamConfig)\n    extends ProductPipelineConfig[HomeMixerRequest, ListTweetsQuery, urt.TimelineResponse] {\n\n  override val identifier: ProductPipelineIdentifier = ProductPipelineIdentifier(\"ListTweets\")\n  override val product: request.Product = ListTweetsProduct\n  override val paramConfig: ProductParamConfig = listTweetsParamConfig\n  override val denyLoggedOutUsers: Boolean = false\n\n  override def pipelineQueryTransformer(\n    request: HomeMixerRequest,\n    params: Params\n  ): ListTweetsQuery = {\n    val context = request.productContext match {\n      case Some(context: ListTweetsProductContext) => context\n      case _ => throw PipelineFailure(BadRequest, \"ListTweetsProductContext not found\")\n    }\n\n    val debugOptions = request.debugParams.flatMap(_.debugOptions)\n\n    /**\n     * Unlike other clients, newly created tweets on Android have the sort index set to the current\n     * time instead of the top sort index + 1, so these tweets get stuck at the top of the timeline\n     * if subsequent timeline responses use the sort index from the previous response instead of\n     * the current time.\n     */\n    val pipelineCursor = request.serializedRequestCursor.flatMap { cursor =>\n      Try(UrtCursorSerializer.deserializeOrderedCursor(cursor))\n        .getOrElse(ChronologicalCursorUnmarshaller(RequestCursorSerializer.deserialize(cursor)))\n        .map {\n          case UrtOrderedCursor(_, id, Some(GapCursor), gapBoundaryId)\n              if id.isEmpty || gapBoundaryId.isEmpty =>\n            throw PipelineFailure(MalformedCursor, \"Gap Cursor bounds not defined\")\n          case topCursor @ UrtOrderedCursor(_, _, Some(TopCursor), _) =>\n            val queryTime = debugOptions.flatMap(_.requestTimeOverride).getOrElse(Time.now)\n            topCursor.copy(initialSortIndex = SortIndexBuilder.timeToId(queryTime))\n          case cursor => cursor\n        }\n    }\n\n    ListTweetsQuery(\n      params = params,\n      clientContext = request.clientContext,\n      features = None,\n      pipelineCursor = pipelineCursor,\n      requestedMaxResults = Some(params(ServerMaxResultsParam)),\n      debugOptions = debugOptions,\n      listId = context.listId,\n      deviceContext = context.deviceContext,\n      dspClientContext = context.dspClientContext\n    )\n  }\n\n  override def pipelines: Seq[PipelineConfig] = Seq(listTweetsMixerPipelineConfig)\n\n  override def pipelineSelector(query: ListTweetsQuery): ComponentIdentifier =\n    listTweetsMixerPipelineConfig.identifier\n\n  override val alerts: Seq[Alert] = Seq(\n    SuccessRateAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfBelow(99.9, 20, 30),\n      criticalPredicate = TriggerIfBelow(99.9, 30, 30),\n    ),\n    LatencyAlert(\n      notificationGroup = DefaultNotificationGroup,\n      percentile = P99,\n      warnPredicate = TriggerIfLatencyAbove(300.millis, 15, 30),\n      criticalPredicate = TriggerIfLatencyAbove(400.millis, 15, 30)\n    ),\n    ThroughputAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfAbove(3000),\n      criticalPredicate = TriggerIfAbove(4000)\n    ),\n    EmptyResponseRateAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfAbove(65),\n      criticalPredicate = TriggerIfAbove(80)\n    )\n  )\n\n  override val debugAccessPolicies: Set[AccessPolicy] = DefaultHomeMixerAccessPolicy\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/ListTweetsTimelineServiceCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.list_tweets\n\nimport com.twitter.home_mixer.candidate_pipeline.TimelineServiceResponseFeatureTransformer\nimport com.twitter.home_mixer.marshaller.timelines.TimelineServiceCursorMarshaller\nimport com.twitter.home_mixer.product.list_tweets.model.ListTweetsQuery\nimport com.twitter.home_mixer.product.list_tweets.param.ListTweetsParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.component_library.candidate_source.timeline_service.TimelineServiceTweetCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelineservice.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ListTweetsTimelineServiceCandidatePipelineConfig @Inject() (\n  timelineServiceTweetCandidateSource: TimelineServiceTweetCandidateSource)\n    extends CandidatePipelineConfig[ListTweetsQuery, t.TimelineQuery, t.Tweet, TweetCandidate] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ListTweetsTimelineServiceTweets\")\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ListTweetsQuery,\n    t.TimelineQuery\n  ] = { query =>\n    val timelineQueryOptions = t.TimelineQueryOptions(\n      contextualUserId = query.clientContext.userId,\n    )\n\n    t.TimelineQuery(\n      timelineType = t.TimelineType.List,\n      timelineId = query.listId,\n      maxCount = query.maxResults(ServerMaxResultsParam).toShort,\n      cursor2 = query.pipelineCursor.flatMap(TimelineServiceCursorMarshaller(_)),\n      options = Some(timelineQueryOptions),\n      timelineId2 = Some(t.TimelineId(t.TimelineType.List, query.listId, None))\n    )\n  }\n\n  override def candidateSource: BaseCandidateSource[t.TimelineQuery, t.Tweet] =\n    timelineServiceTweetCandidateSource\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[t.Tweet, TweetCandidate] = {\n    sourceResult => TweetCandidate(id = sourceResult.statusId)\n  }\n\n  override val featuresFromCandidateSourceTransformers: Seq[CandidateFeatureTransformer[t.Tweet]] =\n    Seq(TimelineServiceResponseFeatureTransformer)\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.7)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/decorator/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/builder\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/decorator/builder\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"timelines/src/main/scala/com/twitter/timelines/injection/scribe\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/decorator/ListConversationServiceCandidateDecorator.scala",
    "content": "package com.twitter.home_mixer.product.list_tweets.decorator\n\nimport com.twitter.home_mixer.functional_component.decorator.builder.HomeConversationModuleMetadataBuilder\nimport com.twitter.home_mixer.functional_component.decorator.builder.ListClientEventDetailsBuilder\nimport com.twitter.home_mixer.model.HomeFeatures.ConversationModuleFocalTweetIdFeature\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.StaticModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.injection.scribe.InjectionScribeUtil\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\n\nobject ListConversationServiceCandidateDecorator {\n\n  private val ConversationModuleNamespace = EntryNamespace(\"list-conversation\")\n\n  def apply(): Some[UrtMultipleModulesDecorator[PipelineQuery, TweetCandidate, Long]] = {\n    val suggestType = st.SuggestType.OrganicListTweet\n    val component = InjectionScribeUtil.scribeComponent(suggestType).get\n    val clientEventInfoBuilder = ClientEventInfoBuilder(\n      component = component,\n      detailsBuilder = Some(ListClientEventDetailsBuilder(st.SuggestType.OrganicListTweet))\n    )\n    val tweetItemBuilder = TweetCandidateUrtItemBuilder(\n      clientEventInfoBuilder = clientEventInfoBuilder\n    )\n\n    val moduleBuilder = TimelineModuleBuilder(\n      entryNamespace = ConversationModuleNamespace,\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      displayTypeBuilder = StaticModuleDisplayTypeBuilder(VerticalConversation),\n      metadataBuilder = Some(HomeConversationModuleMetadataBuilder())\n    )\n\n    Some(\n      UrtMultipleModulesDecorator(\n        urtItemCandidateDecorator = UrtItemCandidateDecorator(tweetItemBuilder),\n        moduleBuilder = moduleBuilder,\n        groupByKey = (_, _, candidateFeatures) =>\n          candidateFeatures.getOrElse(ConversationModuleFocalTweetIdFeature, None)\n      ))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/decorator/builder/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/model/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/param\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/model/ListTweetsQuery.scala",
    "content": "package com.twitter.home_mixer.product.list_tweets.model\n\nimport com.twitter.adserver.thriftscala.HomeTimelineType\nimport com.twitter.adserver.thriftscala.TimelineRequestParams\nimport com.twitter.dspbidder.commons.{thriftscala => dsp}\nimport com.twitter.home_mixer.model.HomeAdsQuery\nimport com.twitter.home_mixer.model.request.DeviceContext\nimport com.twitter.home_mixer.model.request.HasListId\nimport com.twitter.home_mixer.model.request.ListTweetsProduct\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.request._\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Params\n\ncase class ListTweetsQuery(\n  override val params: Params,\n  override val clientContext: ClientContext,\n  override val pipelineCursor: Option[UrtOrderedCursor],\n  override val requestedMaxResults: Option[Int],\n  override val debugOptions: Option[DebugOptions],\n  override val features: Option[FeatureMap],\n  override val listId: Long,\n  override val deviceContext: Option[DeviceContext],\n  override val dspClientContext: Option[dsp.DspClientContext])\n    extends PipelineQuery\n    with HasPipelineCursor[UrtOrderedCursor]\n    with HasListId\n    with HomeAdsQuery {\n  override val product: Product = ListTweetsProduct\n\n  override def withFeatureMap(features: FeatureMap): ListTweetsQuery =\n    copy(features = Some(features))\n\n  override val timelineRequestParams: Option[TimelineRequestParams] =\n    Some(TimelineRequestParams(homeTimelineType = Some(HomeTimelineType.HomeLatest)))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/param/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core/src/main/scala/com/twitter/timelines/configapi\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param/decider\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/param/ListTweetsParam.scala",
    "content": "package com.twitter.home_mixer.product.list_tweets.param\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject ListTweetsParam {\n  val SupportedClientFSName = \"list_tweets_supported_client\"\n\n  object EnableAdsCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"list_tweets_enable_ads\",\n        default = false\n      )\n\n  object ServerMaxResultsParam\n      extends FSBoundedParam[Int](\n        name = \"list_tweets_server_max_results\",\n        default = 100,\n        min = 1,\n        max = 500\n      )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/list_tweets/param/ListTweetsParamConfig.scala",
    "content": "package com.twitter.home_mixer.product.list_tweets.param\n\nimport com.twitter.home_mixer.param.decider.DeciderKey\nimport com.twitter.home_mixer.product.list_tweets.param.ListTweetsParam.EnableAdsCandidatePipelineParam\nimport com.twitter.home_mixer.product.list_tweets.param.ListTweetsParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.product.list_tweets.param.ListTweetsParam.SupportedClientFSName\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.servo.decider.DeciderKeyName\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ListTweetsParamConfig @Inject() () extends ProductParamConfig {\n  override val enabledDeciderKey: DeciderKeyName = DeciderKey.EnableListTweetsProduct\n  override val supportedClientFSName: String = SupportedClientFSName\n\n  override val booleanFSOverrides =\n    Seq(EnableAdsCandidatePipelineParam)\n\n  override val boundedIntFSOverrides = Seq(\n    ServerMaxResultsParam\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer-features/thrift/src/main/thrift:thrift-scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/user_history\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/earlybird\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/marshaller\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scoring_pipeline\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/selector\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/offline_aggregates\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/param_gated\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/async\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/control_ai\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/impressed_tweets\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/location\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/param_gated\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/user_fav_avg_embeddings\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/recommendation\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/large_embeddings\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/ScoredTweetsProductPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedAuthorIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTweetIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SignupCountryFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SignupSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TimelineServiceTweetsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserFollowersCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ViewerAllowsForYouRecommendationsFeature\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.model.request.ScoredTweetsProduct\nimport com.twitter.home_mixer.model.request.ScoredTweetsProductContext\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParamConfig\nimport com.twitter.home_mixer.service.HomeMixerAccessPolicy.DefaultHomeMixerAccessPolicy\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AccessPolicy\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ProductPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport com.twitter.product_mixer.core.pipeline.PipelineConfig\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.BadRequest\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineConfig\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.timelines.configapi.Params\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoredTweetsProductPipelineConfig @Inject() (\n  scoredTweetsRecommendationPipelineConfig: ScoredTweetsRecommendationPipelineConfig,\n  scoredTweetsParamConfig: ScoredTweetsParamConfig)\n    extends ProductPipelineConfig[HomeMixerRequest, ScoredTweetsQuery, t.ScoredTweets] {\n\n  override val identifier: ProductPipelineIdentifier = ProductPipelineIdentifier(\"ScoredTweets\")\n\n  override val product: Product = ScoredTweetsProduct\n\n  override val paramConfig: ProductParamConfig = scoredTweetsParamConfig\n\n  override def pipelineQueryTransformer(\n    request: HomeMixerRequest,\n    params: Params\n  ): ScoredTweetsQuery = {\n    val context = request.productContext match {\n      case Some(context: ScoredTweetsProductContext) => context\n      case _ => throw PipelineFailure(BadRequest, \"ScoredTweetsProductContext not found\")\n    }\n\n    val featureMap = FeatureMapBuilder()\n      .add(ServedTweetIdsFeature, context.servedTweetIds.getOrElse(Seq.empty))\n      .add(TimelineServiceTweetsFeature, context.backfillTweetIds.getOrElse(Seq.empty))\n      .add(SignupCountryFeature, context.signupCountryCode)\n      .add(ViewerAllowsForYouRecommendationsFeature, context.allowForYouRecommendations)\n      .add(SignupSourceFeature, context.signupSource)\n      .add(ServedAuthorIdsFeature, context.servedAuthorIds.getOrElse(Map.empty[Long, Seq[Long]]))\n      .add(UserFollowersCountFeature, context.followerCount)\n      .build()\n\n    ScoredTweetsQuery(\n      params = params,\n      clientContext = request.clientContext,\n      pipelineCursor =\n        request.serializedRequestCursor.flatMap(UrtCursorSerializer.deserializeOrderedCursor),\n      requestedMaxResults = request.maxResults,\n      debugOptions = request.debugParams.flatMap(_.debugOptions),\n      features = Some(featureMap),\n      deviceContext = context.deviceContext,\n      seenTweetIds = context.seenTweetIds,\n      qualityFactorStatus = None,\n      product = product\n    )\n  }\n\n  override val pipelines: Seq[PipelineConfig] = Seq(scoredTweetsRecommendationPipelineConfig)\n\n  override def pipelineSelector(query: ScoredTweetsQuery): ComponentIdentifier =\n    scoredTweetsRecommendationPipelineConfig.identifier\n\n  override val debugAccessPolicies: Set[AccessPolicy] = DefaultHomeMixerAccessPolicy\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/ScoredTweetsRecommendationPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.functional_component.feature_hydrator.FollowableUttTopicsQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.CategoryDiversityRescoringFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.DiversityRescoringFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.FeedbackHistoryQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.GizmoduckUserQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.GrokTranslatedPostIsCachedFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.HeartbeatOptimizerParamsHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.HeavyRankerWeightsQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.ImpressedMediaClusterIdsQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.ImpressionBloomFilterQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.ListIdsQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.NaviClientConfigQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.OnPremRealGraphQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.OptimizerWeightsQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.PhoenixRescoringFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.RealGraphInNetworkScoresQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.RealGraphQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.RealTimeEntityRealGraphQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.RealTimeInteractionGraphUserVertexQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.RequestQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.RequestTimeQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.SimClustersUserSparseEmbeddingsQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetTypeMetricsFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TwhinRebuildUserEngagementQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TwhinRebuildUserPositiveQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TwhinUserEngagementQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TwhinUserFollowQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TwhinUserNegativeQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TwhinUserPositiveQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.UnifiedUserActionsUserIdentifierFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.UserActionsQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.UserEngagedGrokCategoriesFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.UserEngagedLanguagesFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.UserHistoryTransformerEmbeddingQueryFeatureHydratorBuilder\nimport com.twitter.home_mixer.functional_component.feature_hydrator.UserLanguagesFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.UserLargeEmbeddingsFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.UserStateQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.UserUnderstandableLanguagesFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.PartAAggregateQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.PartBAggregateQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.real_time_aggregates.UserEngagementRealTimeAggregatesFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.user_history.ScoredTweetsUserHistoryEventsQueryFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.ClipImageClusterDeduplicationFilter\nimport com.twitter.home_mixer.functional_component.filter.ClipVideoClusterDeduplicationFilter\nimport com.twitter.home_mixer.functional_component.filter.FeedbackFatigueFilter\nimport com.twitter.home_mixer.functional_component.filter.GrokGoreFilter\nimport com.twitter.home_mixer.functional_component.filter.GrokNsfwFilter\nimport com.twitter.home_mixer.functional_component.filter.GrokSpamFilter\nimport com.twitter.home_mixer.functional_component.filter.GrokViolentFilter\nimport com.twitter.home_mixer.functional_component.filter.HasAuthorFilter\nimport com.twitter.home_mixer.functional_component.filter.InvalidSubscriptionTweetFilter\nimport com.twitter.home_mixer.functional_component.filter.LocationFilter\nimport com.twitter.home_mixer.functional_component.filter.MediaIdDeduplicationFilter\nimport com.twitter.home_mixer.functional_component.filter.MinVideoDurationFilter\nimport com.twitter.home_mixer.functional_component.filter.PreviouslySeenTweetsFilter\nimport com.twitter.home_mixer.functional_component.filter.PreviouslyServedTweetsFilter\nimport com.twitter.home_mixer.functional_component.filter.QuoteDeduplicationFilter\nimport com.twitter.home_mixer.functional_component.filter.RejectTweetFromViewerFilter\nimport com.twitter.home_mixer.functional_component.filter.RetweetDeduplicationFilter\nimport com.twitter.home_mixer.functional_component.filter.SlopFilter\nimport com.twitter.home_mixer.functional_component.filter.TweetHydrationFilter\nimport com.twitter.home_mixer.functional_component.selector.SortFixedPositionContentExplorationMixedCandidates\nimport com.twitter.home_mixer.functional_component.selector.SortFixedPositionContentExplorationSimclusterColdPostsCandidates\nimport com.twitter.home_mixer.functional_component.selector.SortFixedPositionDeepRetrievalMixedCandidates\nimport com.twitter.home_mixer.functional_component.side_effect.PublishClientSentImpressionsEventBusSideEffect\nimport com.twitter.home_mixer.functional_component.side_effect.UpdateLastNonPollingTimeSideEffect\nimport com.twitter.home_mixer.model.HomeFeatures.ExclusiveConversationAuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsSupportAccountReplyFeature\nimport com.twitter.home_mixer.model.HomeFeatures.OonNsfwFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableLargeEmbeddingsFeatureHydrationParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableRealGraphQueryFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinRebuildUserEngagementFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinRebuildUserPositiveFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinUserNegativeFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinUserPositiveFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableUserFavAvgTextEmbeddingsQueryFeatureParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableUserHistoryTransformerJointBlueEmbeddingFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.CategoryDiversityRescoringParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.TwhinDiversityRescoringParam\nimport com.twitter.home_mixer.param.HomeMixerFlagName.TargetScoringLatency\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.CachedScoredTweetsCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsBackfillCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsContentExplorationCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsDirectUtegCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsListsCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsStaticCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsTweetMixerCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.earlybird.ScoredTweetsCommunitiesCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.earlybird.ScoredTweetsEarlybirdInNetworkCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.CachedScoredTweetsQueryFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.InvalidateCachedScoredTweetsQueryFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.IsColdStartPostFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.LowSignalUserQueryFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.SGSMutuallyFollowedUserHydrator\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.TweetypieVisibilityFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.ValidLikedByUserIdsFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.filter.ControlAiExcludeFilter\nimport com.twitter.home_mixer.product.scored_tweets.filter.ControlAiOnlyIncludeFilter\nimport com.twitter.home_mixer.product.scored_tweets.filter.CustomSnowflakeIdAgeFilter\nimport com.twitter.home_mixer.product.scored_tweets.filter.DuplicateConversationTweetsFilter\nimport com.twitter.home_mixer.product.scored_tweets.filter.GrokAutoTranslateLanguageFilter\nimport com.twitter.home_mixer.product.scored_tweets.filter.IsOutOfNetworkColdStartPostFilter\nimport com.twitter.home_mixer.product.scored_tweets.filter.LanguageFilter\nimport com.twitter.home_mixer.product.scored_tweets.filter.SGSAuthorFilter\nimport com.twitter.home_mixer.product.scored_tweets.marshaller.ScoredTweetsResponseDomainMarshaller\nimport com.twitter.home_mixer.product.scored_tweets.marshaller.ScoredTweetsResponseTransportMarshaller\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsResponse\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.DefaultRequestedMaxResultsParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableClipImageClusterDedupingParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableColdStartFilterParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableContentExplorationSimclusterColdPostsCandidateBoostingParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableContentExplorationCandidatePipelineParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableContentExplorationMixedCandidateBoostingParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableControlAiParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableDeepRetrievalMixedCandidateBoostingParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableHeartbeatOptimizerWeightsParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableMediaClusterDedupingParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableMediaDedupingParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnablePhoenixRescoreParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableRecentEngagementCacheRefreshParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableMediaClusterDecayParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableRealTimeEntityRealGraphFeaturesParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableTopicSocialProofFeaturesParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableUserActionsFeatureParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableUserEngagedLanguagesFeaturesParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableUserHistoryEventsFeaturesParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableUserIdentifierFeaturesParam\nimport com.twitter.home_mixer.product.scored_tweets.scoring_pipeline.ScoredTweetsHeuristicScoringPipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.scoring_pipeline.ScoredTweetsLowSignalScoringPipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.scoring_pipeline.ScoredTweetsModelScoringPipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.scoring_pipeline.ScoredTweetsRerankingScoringPipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.selector.KeepTopKCandidatesPerCommunity\nimport com.twitter.home_mixer.product.scored_tweets.side_effect.CacheCandidateFeaturesSideEffect\nimport com.twitter.home_mixer.product.scored_tweets.side_effect.CacheRequestInfoSideEffect\nimport com.twitter.home_mixer.product.scored_tweets.side_effect.CacheRetrievalSignalSideEffect\nimport com.twitter.home_mixer.product.scored_tweets.side_effect.CachedScoredTweetsSideEffect\nimport com.twitter.home_mixer.product.scored_tweets.side_effect.CommonFeaturesSideEffect\nimport com.twitter.home_mixer.product.scored_tweets.side_effect.ScoredCandidateFeatureKeysKafkaSideEffect\nimport com.twitter.home_mixer.product.scored_tweets.side_effect.ScoredContentExplorationCandidateScoreFeatureKafkaSideEffect\nimport com.twitter.home_mixer.product.scored_tweets.side_effect.ScoredPhoenixCandidatesKafkaSideEffect\nimport com.twitter.home_mixer.product.scored_tweets.side_effect.ScoredStatsSideEffect\nimport com.twitter.home_mixer.product.scored_tweets.side_effect.ScoredTweetsDiversityStatsSideEffect\nimport com.twitter.home_mixer.product.scored_tweets.side_effect.ScribeScoredCandidatesSideEffect\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedBulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.async.AsyncQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.communities.CommunityMembershipsQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.control_ai.ControlAiQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.impressed_tweets.ImpressedTweetsQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.location.UserLocationQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated.AsyncParamGatedQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated.ParamGatedQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.user_fav_avg_embeddings.UserFavAvgTextEmbeddingsQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.filter.ParamGatedFilter\nimport com.twitter.product_mixer.component_library.filter.PredicateFeatureFilter\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.selector.DropDuplicateCandidates\nimport com.twitter.product_mixer.component_library.selector.DropRequestedMaxResults\nimport com.twitter.product_mixer.component_library.selector.IdAndClassDuplicationKey\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.component_library.selector.PickFirstCandidateMerger\nimport com.twitter.product_mixer.component_library.selector.SelectConditionally\nimport com.twitter.product_mixer.component_library.selector.UpdateSortCandidates\nimport com.twitter.product_mixer.component_library.selector.sorter.FeatureValueSorter\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.RecommendationPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ScoringPipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.recommendation.RecommendationPipelineConfig\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipelineConfig\nimport com.twitter.product_mixer.core.quality_factor.BoundsWithDefault\nimport com.twitter.product_mixer.core.quality_factor.LinearLatencyQualityFactorConfig\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorConfig\nimport com.twitter.util.Duration\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoredTweetsRecommendationPipelineConfig @Inject() (\n  // Candidate pipelines\n  scoredTweetsTweetMixerCandidatePipelineConfig: ScoredTweetsTweetMixerCandidatePipelineConfig,\n  scoredTweetsStaticCandidatePipelineConfig: ScoredTweetsStaticCandidatePipelineConfig,\n  scoredTweetsListsCandidatePipelineConfig: ScoredTweetsListsCandidatePipelineConfig,\n  scoredTweetsBackfillCandidatePipelineConfig: ScoredTweetsBackfillCandidatePipelineConfig,\n  scoredTweetsEarlybirdInNetworkCandidatePipelineConfig: ScoredTweetsEarlybirdInNetworkCandidatePipelineConfig,\n  scoredTweetsCommunitiesCandidatePipelineConfig: ScoredTweetsCommunitiesCandidatePipelineConfig,\n  scoredTweetsDirectUtegCandidatePipelineConfig: ScoredTweetsDirectUtegCandidatePipelineConfig,\n  scoredTweetsContentExplorationCandidatePipelineConfig: ScoredTweetsContentExplorationCandidatePipelineConfig,\n  cachedScoredTweetsCandidatePipelineConfig: CachedScoredTweetsCandidatePipelineConfig,\n  // Query feature hydrators\n  followableUttTopicsQueryFeatureHydrator: FollowableUttTopicsQueryFeatureHydrator,\n  gizmoduckUserQueryFeatureHydrator: GizmoduckUserQueryFeatureHydrator,\n  controlAiQueryFeatureHydrator: ControlAiQueryFeatureHydrator,\n  lowSignalUserQueryFeatureHydrator: LowSignalUserQueryFeatureHydrator,\n  naviClientConfigQueryFeatureHydrator: NaviClientConfigQueryFeatureHydrator,\n  requestQueryFeatureHydrator: RequestQueryFeatureHydrator[ScoredTweetsQuery],\n  requestTimeQueryFeatureHydrator: RequestTimeQueryFeatureHydrator,\n  realTimeInteractionGraphUserVertexQueryFeatureHydrator: RealTimeInteractionGraphUserVertexQueryFeatureHydrator,\n  sgsMutuallyFollowedUserHydrator: SGSMutuallyFollowedUserHydrator,\n  simClustersUserSparseEmbeddingsQueryFeatureHydrator: SimClustersUserSparseEmbeddingsQueryFeatureHydrator,\n  userLocationQueryFeatureHydrator: UserLocationQueryFeatureHydrator,\n  userStateQueryFeatureHydrator: UserStateQueryFeatureHydrator,\n  userEngagementRealTimeAggregatesFeatureHydrator: UserEngagementRealTimeAggregatesFeatureHydrator,\n  twhinRebuildUserEngagementQueryFeatureHydrator: TwhinRebuildUserEngagementQueryFeatureHydrator,\n  twhinRebuildUserPositiveQueryFeatureHydrator: TwhinRebuildUserPositiveQueryFeatureHydrator,\n  twhinUserEngagementQueryFeatureHydrator: TwhinUserEngagementQueryFeatureHydrator,\n  twhinUserFollowQueryFeatureHydrator: TwhinUserFollowQueryFeatureHydrator,\n  twhinUserPositiveQueryFeatureHydrator: TwhinUserPositiveQueryFeatureHydrator,\n  twhinUserNegativeQueryFeatureHydrator: TwhinUserNegativeQueryFeatureHydrator,\n  userHistoryTransformerEmbeddingQueryFeatureHydratorBuilder: UserHistoryTransformerEmbeddingQueryFeatureHydratorBuilder,\n  cachedScoredTweetsQueryFeatureHydrator: CachedScoredTweetsQueryFeatureHydrator,\n  sgsFollowedUsersQueryFeatureHydrator: SGSFollowedUsersQueryFeatureHydrator,\n  scoredTweetsModelScoringPipelineConfig: ScoredTweetsModelScoringPipelineConfig,\n  scoredTweetsRerankingScoringPipelineConfig: ScoredTweetsRerankingScoringPipelineConfig,\n  impressionBloomFilterQueryFeatureHydrator: ImpressionBloomFilterQueryFeatureHydrator,\n  memcacheTweetImpressionsQueryFeatureHydrator: ImpressedTweetsQueryFeatureHydrator,\n  userEngagementsAvgTextEmbeddingsQueryFeatureHydrator: UserFavAvgTextEmbeddingsQueryFeatureHydrator,\n  listIdsQueryFeatureHydrator: ListIdsQueryFeatureHydrator,\n  feedbackHistoryQueryFeatureHydrator: FeedbackHistoryQueryFeatureHydrator,\n  communityMembershipsQueryFeatureHydrator: CommunityMembershipsQueryFeatureHydrator,\n  tweetypieVisibilityFeatureHydrator: TweetypieVisibilityFeatureHydrator,\n  realGraphInNetworkScoresQueryFeatureHydrator: RealGraphInNetworkScoresQueryFeatureHydrator,\n  realGraphQueryFeatureHydrator: RealGraphQueryFeatureHydrator,\n  onPremRealGraphQueryFeatureHydrator: OnPremRealGraphQueryFeatureHydrator,\n  entityRealGraphQueryFeatureHydrator: RealTimeEntityRealGraphQueryFeatureHydrator,\n  unifiedUserActionsUserIdentifierFeatureHydrator: UnifiedUserActionsUserIdentifierFeatureHydrator,\n  userEngagedLanguagesFeatureHydrator: UserEngagedLanguagesFeatureHydrator,\n  userLanguagesFeatureHydrator: UserLanguagesFeatureHydrator,\n  userUnderstandableLanguagesFeatureHydrator: UserUnderstandableLanguagesFeatureHydrator,\n  partAAggregateQueryFeatureHydrator: PartAAggregateQueryFeatureHydrator,\n  partBAggregateQueryFeatureHydrator: PartBAggregateQueryFeatureHydrator,\n  heavyRankerWeightsQueryFeatureHydrator: HeavyRankerWeightsQueryFeatureHydrator,\n  userLargeEmbeddingsFeatureHydrator: UserLargeEmbeddingsFeatureHydrator,\n  userHistoryEventsQueryFeatureHydrator: ScoredTweetsUserHistoryEventsQueryFeatureHydrator,\n  userActionsQueryFeatureHydrator: UserActionsQueryFeatureHydrator,\n  impressedMediaClusterIdsQueryFeatureHydrator: ImpressedMediaClusterIdsQueryFeatureHydrator,\n  heartbeatOptimizerParamsHydrator: HeartbeatOptimizerParamsHydrator,\n  optimizerWeightsQueryFeatureHydrator: OptimizerWeightsQueryFeatureHydrator,\n  userEngagedGrokCategoriesFeatureHydrator: UserEngagedGrokCategoriesFeatureHydrator,\n  grokTranslatedPostIsCachedFeatureHydrator: GrokTranslatedPostIsCachedFeatureHydrator,\n  isColdStartPostFeatureHydrator: IsColdStartPostFeatureHydrator,\n  // Filters\n  invalidSubscriptionTweetFilter: InvalidSubscriptionTweetFilter,\n  sgsAuthorFilter: SGSAuthorFilter,\n  // Side effects\n  cacheCandidateFeaturesSideEffect: CacheCandidateFeaturesSideEffect,\n  cachedScoredTweetsSideEffect: CachedScoredTweetsSideEffect,\n  cacheRetrievalSignalSideEffect: CacheRetrievalSignalSideEffect,\n  cacheRequestInfoSideEffect: CacheRequestInfoSideEffect,\n  scoredCandidateFeatureKeysKafkaSideEffect: ScoredCandidateFeatureKeysKafkaSideEffect,\n  scoredContentExplorationCandidateScoreFeatureKafkaSideEffect: ScoredContentExplorationCandidateScoreFeatureKafkaSideEffect,\n  publishClientSentImpressionsEventBusSideEffect: PublishClientSentImpressionsEventBusSideEffect,\n  scoredStatsSideEffect: ScoredStatsSideEffect,\n  scoredTweetsDiversityStatsSideEffect: ScoredTweetsDiversityStatsSideEffect,\n  scoredPhoenixCandidatesKafkaSideEffect: ScoredPhoenixCandidatesKafkaSideEffect,\n  scribeCommonFeaturesSideEffect: CommonFeaturesSideEffect,\n  scribeScoredCandidatesSideEffect: ScribeScoredCandidatesSideEffect,\n  updateLastNonPollingTimeSideEffect: UpdateLastNonPollingTimeSideEffect[\n    ScoredTweetsQuery,\n    ScoredTweetsResponse\n  ],\n  @Flag(TargetScoringLatency) targetScoringLatency: Duration)\n    extends RecommendationPipelineConfig[\n      ScoredTweetsQuery,\n      TweetCandidate,\n      ScoredTweetsResponse,\n      t.ScoredTweetsResponse\n    ] {\n\n  override val identifier: RecommendationPipelineIdentifier =\n    RecommendationPipelineIdentifier(\"ScoredTweets\")\n\n  private val SubscriptionReplyFilterId = \"SubscriptionReply\"\n  private val OutOfNetworkNSFWFilterId = \"OutOfNetworkNSFW\"\n\n  private val scoringStep = RecommendationPipelineConfig.scoringPipelinesStep\n\n  override val fetchQueryFeatures: Seq[QueryFeatureHydrator[ScoredTweetsQuery]] = Seq(\n    naviClientConfigQueryFeatureHydrator,\n    requestQueryFeatureHydrator,\n    realGraphInNetworkScoresQueryFeatureHydrator,\n    cachedScoredTweetsQueryFeatureHydrator,\n    sgsFollowedUsersQueryFeatureHydrator,\n    memcacheTweetImpressionsQueryFeatureHydrator,\n    listIdsQueryFeatureHydrator,\n    communityMembershipsQueryFeatureHydrator,\n    userStateQueryFeatureHydrator,\n    lowSignalUserQueryFeatureHydrator,\n    feedbackHistoryQueryFeatureHydrator,\n    requestTimeQueryFeatureHydrator,\n    userUnderstandableLanguagesFeatureHydrator,\n    ParamGatedQueryFeatureHydrator(\n      EnableContentExplorationCandidatePipelineParam,\n      userEngagedGrokCategoriesFeatureHydrator\n    ),\n    AsyncQueryFeatureHydrator(\n      RecommendationPipelineConfig.globalFiltersStep,\n      userLocationQueryFeatureHydrator\n    ),\n    AsyncQueryFeatureHydrator(\n      RecommendationPipelineConfig.globalFiltersStep,\n      impressionBloomFilterQueryFeatureHydrator\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableRealGraphQueryFeaturesParam,\n      scoringStep,\n      realGraphQueryFeatureHydrator\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableRealGraphQueryFeaturesParam,\n      scoringStep,\n      onPremRealGraphQueryFeatureHydrator\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableRealTimeEntityRealGraphFeaturesParam,\n      scoringStep,\n      entityRealGraphQueryFeatureHydrator\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableTwhinRebuildUserEngagementFeaturesParam,\n      scoringStep,\n      twhinRebuildUserEngagementQueryFeatureHydrator\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableTwhinRebuildUserPositiveFeaturesParam,\n      scoringStep,\n      twhinRebuildUserPositiveQueryFeatureHydrator\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableTwhinUserNegativeFeaturesParam,\n      scoringStep,\n      twhinUserNegativeQueryFeatureHydrator\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableTwhinUserPositiveFeaturesParam,\n      scoringStep,\n      twhinUserPositiveQueryFeatureHydrator\n    ),\n    AsyncQueryFeatureHydrator(\n      scoringStep,\n      userHistoryTransformerEmbeddingQueryFeatureHydratorBuilder.buildHomeBlueHydrator()\n    ),\n    AsyncQueryFeatureHydrator(\n      scoringStep,\n      userHistoryTransformerEmbeddingQueryFeatureHydratorBuilder.buildHomeGreenHydrator()\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableUserHistoryTransformerJointBlueEmbeddingFeaturesParam,\n      scoringStep,\n      userHistoryTransformerEmbeddingQueryFeatureHydratorBuilder.buildJointBlueHydrator()\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableTopicSocialProofFeaturesParam,\n      scoringStep,\n      followableUttTopicsQueryFeatureHydrator\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableUserEngagedLanguagesFeaturesParam,\n      scoringStep,\n      userEngagedLanguagesFeatureHydrator\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableLargeEmbeddingsFeatureHydrationParam,\n      scoringStep,\n      userLargeEmbeddingsFeatureHydrator,\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableUserHistoryEventsFeaturesParam,\n      scoringStep,\n      userHistoryEventsQueryFeatureHydrator,\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableUserActionsFeatureParam,\n      scoringStep,\n      userActionsQueryFeatureHydrator\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableMediaClusterDecayParam,\n      scoringStep,\n      impressedMediaClusterIdsQueryFeatureHydrator,\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableUserIdentifierFeaturesParam,\n      scoringStep,\n      unifiedUserActionsUserIdentifierFeatureHydrator\n    ),\n    AsyncQueryFeatureHydrator(scoringStep, userLanguagesFeatureHydrator),\n    AsyncQueryFeatureHydrator(scoringStep, sgsMutuallyFollowedUserHydrator),\n    AsyncQueryFeatureHydrator(scoringStep, userEngagementRealTimeAggregatesFeatureHydrator),\n    AsyncQueryFeatureHydrator(scoringStep, realTimeInteractionGraphUserVertexQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(scoringStep, twhinUserFollowQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(scoringStep, twhinUserEngagementQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(scoringStep, partAAggregateQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(scoringStep, partBAggregateQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(scoringStep, heavyRankerWeightsQueryFeatureHydrator),\n    ParamGatedQueryFeatureHydrator(\n      EnableHeartbeatOptimizerWeightsParam,\n      heartbeatOptimizerParamsHydrator\n    ),\n    AsyncQueryFeatureHydrator(scoringStep, simClustersUserSparseEmbeddingsQueryFeatureHydrator),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableControlAiParam,\n      scoringStep,\n      controlAiQueryFeatureHydrator\n    ),\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableUserFavAvgTextEmbeddingsQueryFeatureParam,\n      RecommendationPipelineConfig.resultSideEffectsStep,\n      userEngagementsAvgTextEmbeddingsQueryFeatureHydrator\n    ),\n  )\n\n  override val fetchQueryFeaturesPhase2: Seq[QueryFeatureHydrator[ScoredTweetsQuery]] = Seq(\n    AsyncParamGatedQueryFeatureHydrator(\n      EnableHeartbeatOptimizerWeightsParam,\n      scoringStep,\n      optimizerWeightsQueryFeatureHydrator\n    ),\n    ParamGatedQueryFeatureHydrator(\n      EnableRecentEngagementCacheRefreshParam,\n      InvalidateCachedScoredTweetsQueryFeatureHydrator\n    )\n  )\n\n  override val candidatePipelines: Seq[\n    CandidatePipelineConfig[ScoredTweetsQuery, _, _, TweetCandidate]\n  ] = Seq(\n    // Order matters. Duplicated candidates take the first occurrence in the pipelines when merger\n    // strategy is PickFirstCandidateMerger.\n    scoredTweetsStaticCandidatePipelineConfig,\n    cachedScoredTweetsCandidatePipelineConfig,\n    scoredTweetsEarlybirdInNetworkCandidatePipelineConfig,\n    scoredTweetsDirectUtegCandidatePipelineConfig,\n    scoredTweetsTweetMixerCandidatePipelineConfig,\n    scoredTweetsContentExplorationCandidatePipelineConfig,\n    scoredTweetsListsCandidatePipelineConfig,\n    scoredTweetsBackfillCandidatePipelineConfig,\n    scoredTweetsCommunitiesCandidatePipelineConfig\n  )\n\n  override val postCandidatePipelinesSelectors: Seq[Selector[ScoredTweetsQuery]] = Seq(\n    DropDuplicateCandidates(\n      pipelineScope = AllPipelines,\n      duplicationKey = IdAndClassDuplicationKey,\n      mergeStrategy = PickFirstCandidateMerger\n    ),\n    InsertAppendResults(AllPipelines)\n  )\n\n  override val globalFilters: Seq[Filter[ScoredTweetsQuery, TweetCandidate]] = Seq(\n    // sort these to have the \"cheaper\" filters run first\n    CustomSnowflakeIdAgeFilter(StaticParam(48.hours)),\n    HasAuthorFilter,\n    RejectTweetFromViewerFilter,\n    RetweetDeduplicationFilter,\n    LocationFilter,\n    PreviouslySeenTweetsFilter,\n    PreviouslyServedTweetsFilter,\n    PredicateFeatureFilter.fromPredicate(\n      FilterIdentifier(SubscriptionReplyFilterId),\n      shouldKeepCandidate = { features =>\n        features.getOrElse(InReplyToTweetIdFeature, None).isEmpty ||\n        features.getOrElse(ExclusiveConversationAuthorIdFeature, None).isEmpty\n      }\n    ),\n    FeedbackFatigueFilter,\n    invalidSubscriptionTweetFilter,\n    sgsAuthorFilter\n  )\n\n  override val candidatePipelineFailOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy] =\n    Map(\n      scoredTweetsStaticCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n      cachedScoredTweetsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n      scoredTweetsTweetMixerCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n      scoredTweetsListsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n      scoredTweetsBackfillCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n      scoredTweetsEarlybirdInNetworkCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n      scoredTweetsCommunitiesCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n      scoredTweetsDirectUtegCandidatePipelineConfig.identifier -> FailOpenPolicy.Always\n    )\n\n  override val scoringPipelineFailOpenPolicies: Map[ScoringPipelineIdentifier, FailOpenPolicy] =\n    Map(\n      ScoredTweetsHeuristicScoringPipelineConfig.identifier -> FailOpenPolicy.Always,\n      ScoredTweetsLowSignalScoringPipelineConfig.identifier -> FailOpenPolicy.Always\n    )\n\n  private val scoringPipelineQualityFactorConfig = LinearLatencyQualityFactorConfig(\n    qualityFactorBounds = BoundsWithDefault(minInclusive = 0.1, maxInclusive = 1.0, default = 0.95),\n    initialDelay = 60.seconds,\n    targetLatency = targetScoringLatency,\n    targetLatencyPercentile = 95.0,\n    delta = 0.00125\n  )\n\n  override val qualityFactorConfigs: Map[ComponentIdentifier, QualityFactorConfig] = Map(\n    scoredTweetsModelScoringPipelineConfig.identifier -> scoringPipelineQualityFactorConfig,\n  )\n\n  override val scoringPipelines: Seq[ScoringPipelineConfig[ScoredTweetsQuery, TweetCandidate]] =\n    Seq(\n      // scoring pipeline - run on non-cached candidates only since cached ones are already scored\n      scoredTweetsModelScoringPipelineConfig,\n      scoredTweetsRerankingScoringPipelineConfig,\n      // re-scoring pipeline - run on all candidates since these are request specific\n      ScoredTweetsHeuristicScoringPipelineConfig,\n      ScoredTweetsLowSignalScoringPipelineConfig,\n    )\n\n  override val postScoringFilters = Seq(\n    MinVideoDurationFilter,\n    ParamGatedFilter(\n      EnableControlAiParam,\n      ControlAiExcludeFilter\n    ),\n    ParamGatedFilter(\n      EnableControlAiParam,\n      ControlAiOnlyIncludeFilter\n    ),\n    SlopFilter,\n    GrokGoreFilter,\n    GrokNsfwFilter,\n    GrokSpamFilter,\n    GrokViolentFilter,\n    LanguageFilter,\n    GrokAutoTranslateLanguageFilter,\n    DuplicateConversationTweetsFilter,\n    PredicateFeatureFilter.fromPredicate(\n      FilterIdentifier(\"IsSupportAccountReply\"),\n      shouldKeepCandidate = { features =>\n        !features.getOrElse(IsSupportAccountReplyFeature, false)\n      }\n    ),\n  )\n\n  override val resultSelectors: Seq[Selector[ScoredTweetsQuery]] = Seq(\n    KeepTopKCandidatesPerCommunity(AllPipelines),\n    UpdateSortCandidates(AllPipelines, FeatureValueSorter.descending(ScoreFeature)),\n    SelectConditionally.paramGated(\n      SortFixedPositionContentExplorationMixedCandidates,\n      EnableContentExplorationMixedCandidateBoostingParam\n    ),\n    SelectConditionally.paramGated(\n      SortFixedPositionDeepRetrievalMixedCandidates,\n      EnableDeepRetrievalMixedCandidateBoostingParam\n    ),\n    SelectConditionally.paramGated(\n      SortFixedPositionContentExplorationSimclusterColdPostsCandidates,\n      EnableContentExplorationSimclusterColdPostsCandidateBoostingParam\n    ),\n    InsertAppendResults(AllPipelines),\n    DropRequestedMaxResults(\n      defaultRequestedMaxResultsParam = DefaultRequestedMaxResultsParam,\n      serverMaxResultsParam = ServerMaxResultsParam\n    )\n  )\n\n  override val postSelectionFeatureHydration = Seq(\n    grokTranslatedPostIsCachedFeatureHydrator,\n    tweetypieVisibilityFeatureHydrator,\n    TweetTypeMetricsFeatureHydrator,\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnablePhoenixRescoreParam,\n      PhoenixRescoringFeatureHydrator\n    ),\n    ParamGatedBulkCandidateFeatureHydrator(\n      TwhinDiversityRescoringParam,\n      DiversityRescoringFeatureHydrator\n    ),\n    ParamGatedBulkCandidateFeatureHydrator(\n      CategoryDiversityRescoringParam,\n      CategoryDiversityRescoringFeatureHydrator\n    ),\n    ParamGatedCandidateFeatureHydrator(\n      EnableColdStartFilterParam,\n      isColdStartPostFeatureHydrator\n    ),\n    ValidLikedByUserIdsFeatureHydrator\n  )\n\n  override val postSelectionFilters = Seq(\n    TweetHydrationFilter,\n    PredicateFeatureFilter.fromPredicate(\n      FilterIdentifier(OutOfNetworkNSFWFilterId),\n      shouldKeepCandidate = { features => !features.getOrElse(OonNsfwFeature, false) }\n    ),\n    QuoteDeduplicationFilter,\n    ParamGatedFilter(\n      EnableMediaDedupingParam,\n      MediaIdDeduplicationFilter\n    ),\n    ParamGatedFilter(\n      EnableMediaClusterDedupingParam,\n      ClipVideoClusterDeduplicationFilter\n    ),\n    ParamGatedFilter(\n      EnableClipImageClusterDedupingParam,\n      ClipImageClusterDeduplicationFilter\n    ),\n    ParamGatedFilter(\n      EnableColdStartFilterParam,\n      IsOutOfNetworkColdStartPostFilter\n    )\n  )\n\n  override val resultSideEffects: Seq[\n    PipelineResultSideEffect[ScoredTweetsQuery, ScoredTweetsResponse]\n  ] = Seq(\n    cacheCandidateFeaturesSideEffect,\n    cachedScoredTweetsSideEffect,\n    scoredCandidateFeatureKeysKafkaSideEffect,\n    scoredContentExplorationCandidateScoreFeatureKafkaSideEffect,\n    scoredPhoenixCandidatesKafkaSideEffect,\n    publishClientSentImpressionsEventBusSideEffect,\n    scoredStatsSideEffect,\n    scoredTweetsDiversityStatsSideEffect,\n    scribeCommonFeaturesSideEffect,\n    scribeScoredCandidatesSideEffect,\n    updateLastNonPollingTimeSideEffect,\n    cacheRetrievalSignalSideEffect,\n    cacheRequestInfoSideEffect\n  )\n\n  override val domainMarshaller: DomainMarshaller[\n    ScoredTweetsQuery,\n    ScoredTweetsResponse\n  ] = ScoredTweetsResponseDomainMarshaller\n\n  override val transportMarshaller: TransportMarshaller[\n    ScoredTweetsResponse,\n    t.ScoredTweetsResponse\n  ] = ScoredTweetsResponseTransportMarshaller\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/filter\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_ranker\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweet_mixer\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/uteg\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module:test-user-mapper\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"src/thrift/com/twitter/timelineranker:thrift-scala\",\n        \"tweet-mixer/thrift/src/main/thrift:thrift-scala\",\n    ],\n    exports = [\n        \"src/thrift/com/twitter/timelineranker:thrift-scala\",\n        \"tweet-mixer/thrift/src/main/thrift:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/CachedScoredTweetsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline\n\nimport com.twitter.home_mixer.model.HomeFeatures.CachedScoredTweetsFeature\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.CachedScoredTweetsCandidatePipelineConfig._\nimport com.twitter.home_mixer.product.scored_tweets.candidate_source.CachedScoredTweetsCandidateSource\nimport com.twitter.home_mixer.product.scored_tweets.gate.RecentFeedbackCheckGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.CachedScoredTweetsResponseFeatureTransformer\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.gate.NonEmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Candidate Pipeline Config that fetches tweets from Scored Tweets Cache.\n */\n@Singleton\nclass CachedScoredTweetsCandidatePipelineConfig @Inject() (\n  cachedScoredTweetsCandidateSource: CachedScoredTweetsCandidateSource)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      ScoredTweetsQuery,\n      hmt.ScoredTweet,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier = Identifier\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] = Seq(\n    NonEmptySeqFeatureGate(CachedScoredTweetsFeature),\n    RecentFeedbackCheckGate\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    ScoredTweetsQuery\n  ] = identity\n\n  override val candidateSource: BaseCandidateSource[ScoredTweetsQuery, hmt.ScoredTweet] =\n    cachedScoredTweetsCandidateSource\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[hmt.ScoredTweet]\n  ] = Seq(CachedScoredTweetsResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    hmt.ScoredTweet,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.tweetId) }\n}\n\nobject CachedScoredTweetsCandidatePipelineConfig {\n  val Identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\"CachedScoredTweets\")\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/ScoredTweetsBackfillCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetEntityServiceFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.HasAuthorFilter\nimport com.twitter.home_mixer.functional_component.filter.ReplyFilter\nimport com.twitter.home_mixer.model.HomeFeatures.CachedScoredTweetsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TimelineServiceTweetsFeature\nimport com.twitter.home_mixer.product.scored_tweets.gate.DenyLowSignalUserGate\nimport com.twitter.home_mixer.product.scored_tweets.gate.MinTimeSinceLastRequestGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableBackfillCandidatePipelineParam\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.ScoredTweetsBackfillResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.gate.EmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.gate.NonEmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.PassthroughCandidateSource\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoredTweetsBackfillCandidatePipelineConfig @Inject() (\n  tweetEntityServiceFeatureHydrator: TweetEntityServiceFeatureHydrator)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      ScoredTweetsQuery,\n      Long,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ScoredTweetsBackfill\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] =\n    Some(EnableBackfillCandidatePipelineParam)\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] =\n    Seq(\n      MinTimeSinceLastRequestGate,\n      DenyLowSignalUserGate,\n      NonEmptySeqFeatureGate(TimelineServiceTweetsFeature),\n      EmptySeqFeatureGate(CachedScoredTweetsFeature)\n    )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    ScoredTweetsQuery\n  ] = identity\n\n  override def candidateSource: CandidateSource[ScoredTweetsQuery, Long] =\n    PassthroughCandidateSource(\n      identifier = CandidateSourceIdentifier(\"ScoredTweetsBackfill\"),\n      candidateExtractor = { query =>\n        query.features.map(_.getOrElse(TimelineServiceTweetsFeature, Seq.empty)).toSeq.flatten\n      }\n    )\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[Long]\n  ] = Seq(ScoredTweetsBackfillResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[Long, TweetCandidate] = {\n    sourceResult => TweetCandidate(id = sourceResult)\n  }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ScoredTweetsQuery, TweetCandidate, _]\n  ] = Seq(tweetEntityServiceFeatureHydrator)\n\n  override val filters: Seq[Filter[ScoredTweetsQuery, TweetCandidate]] = Seq(\n    ReplyFilter,\n    HasAuthorFilter\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/ScoredTweetsContentExplorationCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetEntityServiceFeatureHydrator\nimport com.twitter.home_mixer.model.HomeFeatures.CachedScoredTweetsFeature\nimport com.twitter.home_mixer.product.scored_tweets.candidate_source.ContentExplorationCandidateResponse\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.candidate_source.ContentExplorationCandidateSource\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableContentExplorationCandidatePipelineParam\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.ContentExplorationQueryRequest\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.ContentExplorationQueryTransformer\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.ScoredTweetsContentExplorationResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.gate.EmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Candidate Pipeline Config that fetches tweets from the content exploration post source\n */\n@Singleton\nclass ScoredTweetsContentExplorationCandidatePipelineConfig @Inject() (\n  contentExplorationCandidateSource: ContentExplorationCandidateSource,\n  tweetEntityServiceFeatureHydrator: TweetEntityServiceFeatureHydrator)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      ContentExplorationQueryRequest,\n      ContentExplorationCandidateResponse,\n      TweetCandidate\n    ] {\n\n  override val supportedClientParam: Option[FSParam[Boolean]] = Some(\n    EnableContentExplorationCandidatePipelineParam)\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] = Seq(\n    EmptySeqFeatureGate(CachedScoredTweetsFeature))\n\n  override val identifier: CandidatePipelineIdentifier =\n    ScoredTweetsContentExplorationCandidatePipelineConfig.identifier\n\n  override val candidateSource: ContentExplorationCandidateSource =\n    contentExplorationCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    ContentExplorationQueryRequest\n  ] = ContentExplorationQueryTransformer\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[ContentExplorationCandidateResponse]\n  ] = Seq(ScoredTweetsContentExplorationResponseFeatureTransformer)\n\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[ScoredTweetsQuery, TweetCandidate, _]\n  ] = Seq(tweetEntityServiceFeatureHydrator)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    ContentExplorationCandidateResponse,\n    TweetCandidate\n  ] = { sourceResult =>\n    TweetCandidate(id = sourceResult.tweetId)\n  }\n}\n\nobject ScoredTweetsContentExplorationCandidatePipelineConfig {\n  val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ScoredTweetsContentExploration\")\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/ScoredTweetsDirectUtegCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.EarlybirdSearchResultFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetEntityServiceFeatureHydrator\nimport com.twitter.home_mixer.functional_component.gate.AllowForYouRecommendationsGate\nimport com.twitter.home_mixer.model.HomeFeatures.CachedScoredTweetsFeature\nimport com.twitter.home_mixer.product.scored_tweets.filter.FollowedAuthorFilter\nimport com.twitter.home_mixer.product.scored_tweets.filter.OONReplyFilter\nimport com.twitter.home_mixer.product.scored_tweets.filter.UtegMinFavCountFilter\nimport com.twitter.home_mixer.product.scored_tweets.filter.UtegTopKFilter\nimport com.twitter.home_mixer.product.scored_tweets.gate.DenyLowSignalUserGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CandidateSourceParams\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.UtegQueryTransformer\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.ScoredTweetsDirectUtegResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.candidate_source.uteg.UtegCandidateSource\nimport com.twitter.product_mixer.component_library.gate.EmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.recos.user_tweet_entity_graph.{thriftscala => uteg}\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoredTweetsDirectUtegCandidatePipelineConfig @Inject() (\n  utegCandidateSource: UtegCandidateSource,\n  tweetEntityServiceFeatureHydrator: TweetEntityServiceFeatureHydrator,\n  earlybirdSearchResultFeatureHydrator: EarlybirdSearchResultFeatureHydrator)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      uteg.RecommendTweetEntityRequest,\n      uteg.TweetRecommendation,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ScoredTweetsDirectUteg\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] = Some(\n    CandidateSourceParams.EnableUTEGCandidateSourceParam)\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] = Seq(\n    AllowForYouRecommendationsGate,\n    DenyLowSignalUserGate,\n    EmptySeqFeatureGate(CachedScoredTweetsFeature)\n  )\n\n  override val candidateSource: BaseCandidateSource[\n    uteg.RecommendTweetEntityRequest,\n    uteg.TweetRecommendation\n  ] = utegCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    uteg.RecommendTweetEntityRequest\n  ] = UtegQueryTransformer(identifier)\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ScoredTweetsQuery, TweetCandidate, _]\n  ] = Seq(\n    tweetEntityServiceFeatureHydrator,\n    earlybirdSearchResultFeatureHydrator\n  )\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[uteg.TweetRecommendation]\n  ] = Seq(ScoredTweetsDirectUtegResponseFeatureTransformer)\n\n  override val filters: Seq[Filter[ScoredTweetsQuery, TweetCandidate]] = Seq(\n    FollowedAuthorFilter,\n    UtegMinFavCountFilter,\n    OONReplyFilter,\n    UtegTopKFilter,\n  )\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    uteg.TweetRecommendation,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.tweetId) }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/ScoredTweetsFrsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline\n\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.FrsSeedUsersQueryFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.gate.MinCachedTweetsGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CachedScoredTweets\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CandidatePipeline\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.TimelineRankerFrsQueryTransformer\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.ScoredTweetsFrsResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.candidate_source.timeline_ranker.TimelineRankerRecapCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseQueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelineranker.{thriftscala => tlr}\nimport com.twitter.timelines.configapi.decider.DeciderParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Candidate Pipeline Config that takes user recommendations from Follow Recommendation Service (FRS)\n * and makes a TimelineRanker->Earlybird query for tweet candidates from those users.\n * Additionally, the candidate pipeline hydrates followedByUserIds so that followed-by social proof\n * can be used.\n */\n@Singleton\nclass ScoredTweetsFrsCandidatePipelineConfig @Inject() (\n  timelineRankerRecapCandidateSource: TimelineRankerRecapCandidateSource,\n  frsSeedUsersQueryFeatureHydrator: FrsSeedUsersQueryFeatureHydrator)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      tlr.RecapQuery,\n      tlr.CandidateTweet,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ScoredTweetsFrs\")\n\n  override val enabledDeciderParam: Option[DeciderParam[Boolean]] =\n    Some(CandidatePipeline.EnableFrsParam)\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] = Seq(\n    MinCachedTweetsGate(identifier, CachedScoredTweets.MinCachedTweetsParam)\n  )\n\n  override val queryFeatureHydration: Seq[\n    BaseQueryFeatureHydrator[ScoredTweetsQuery, _]\n  ] = Seq(frsSeedUsersQueryFeatureHydrator)\n\n  override val candidateSource: BaseCandidateSource[tlr.RecapQuery, tlr.CandidateTweet] =\n    timelineRankerRecapCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    tlr.RecapQuery\n  ] = TimelineRankerFrsQueryTransformer(identifier)\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[tlr.CandidateTweet]\n  ] = Seq(ScoredTweetsFrsResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    tlr.CandidateTweet,\n    TweetCandidate\n  ] = { candidate => TweetCandidate(candidate.tweet.get.id) }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/ScoredTweetsInNetworkCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline\n\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.IsExtendedReplyFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.ReplyFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.RetweetSourceTweetFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.filter.RetweetSourceTweetRemovingFilter\nimport com.twitter.home_mixer.product.scored_tweets.gate.MinCachedTweetsGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CachedScoredTweets\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CandidatePipeline\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.TimelineRankerInNetworkQueryTransformer\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.ScoredTweetsInNetworkResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.candidate_source.timeline_ranker.TimelineRankerInNetworkCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelineranker.{thriftscala => t}\nimport com.twitter.timelines.configapi.decider.DeciderParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Candidate Pipeline Config to fetch in-network tweets from Timeline Ranker's Recycled source\n */\n@Singleton\nclass ScoredTweetsInNetworkCandidatePipelineConfig @Inject() (\n  timelineRankerInNetworkCandidateSource: TimelineRankerInNetworkCandidateSource,\n  replyFeatureHydrator: ReplyFeatureHydrator)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      t.RecapQuery,\n      t.CandidateTweet,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ScoredTweetsInNetwork\")\n\n  override val enabledDeciderParam: Option[DeciderParam[Boolean]] =\n    Some(CandidatePipeline.EnableInNetworkParam)\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] = Seq(\n    MinCachedTweetsGate(identifier, CachedScoredTweets.MinCachedTweetsParam)\n  )\n\n  override val candidateSource: BaseCandidateSource[t.RecapQuery, t.CandidateTweet] =\n    timelineRankerInNetworkCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    t.RecapQuery\n  ] = TimelineRankerInNetworkQueryTransformer(identifier)\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[PipelineQuery, TweetCandidate, _]\n  ] = Seq(RetweetSourceTweetFeatureHydrator)\n\n  override def filters: Seq[Filter[ScoredTweetsQuery, TweetCandidate]] = Seq(\n    RetweetSourceTweetRemovingFilter\n  )\n\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[PipelineQuery, TweetCandidate, _]\n  ] = Seq(IsExtendedReplyFeatureHydrator, replyFeatureHydrator)\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[t.CandidateTweet]\n  ] = Seq(ScoredTweetsInNetworkResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    t.CandidateTweet,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.tweet.get.id) }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/ScoredTweetsListsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.ListIdsFeature\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetEntityServiceFeatureHydrator\nimport com.twitter.home_mixer.functional_component.filter.ReplyFilter\nimport com.twitter.home_mixer.functional_component.filter.RetweetFilter\nimport com.twitter.home_mixer.model.HomeFeatures.CachedScoredTweetsFeature\nimport com.twitter.home_mixer.product.scored_tweets.candidate_source.ListTweet\nimport com.twitter.home_mixer.product.scored_tweets.candidate_source.ListsCandidateSource\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.ListNameFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.ScoredTweetsListsResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.gate.EmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.gate.NonEmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelineservice.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoredTweetsListsCandidatePipelineConfig @Inject() (\n  listsCandidateSource: ListsCandidateSource,\n  listNameFeatureHydrator: ListNameFeatureHydrator,\n  tweetEntityServiceFeatureHydrator: TweetEntityServiceFeatureHydrator)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      Seq[t.TimelineQuery],\n      ListTweet,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ScoredTweetsLists\")\n\n  private val MaxTweetsToFetchPerList = 20\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] = Seq(\n    NonEmptySeqFeatureGate(ListIdsFeature),\n    EmptySeqFeatureGate(CachedScoredTweetsFeature)\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    Seq[t.TimelineQuery]\n  ] = { query =>\n    val listIds = query.features.map(_.get(ListIdsFeature)).get\n    listIds.map { listId =>\n      t.TimelineQuery(\n        timelineType = t.TimelineType.List,\n        timelineId = listId,\n        maxCount = MaxTweetsToFetchPerList.toShort,\n        options = Some(t.TimelineQueryOptions(query.clientContext.userId)),\n        timelineId2 = Some(t.TimelineId(t.TimelineType.List, listId, None))\n      )\n    }\n  }\n\n  override def candidateSource: CandidateSource[Seq[t.TimelineQuery], ListTweet] =\n    listsCandidateSource\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[ListTweet]\n  ] = Seq(ScoredTweetsListsResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[ListTweet, TweetCandidate] = {\n    sourceResult => TweetCandidate(id = sourceResult.tweet.statusId)\n  }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ScoredTweetsQuery, TweetCandidate, _]\n  ] = Seq(\n    listNameFeatureHydrator,\n    tweetEntityServiceFeatureHydrator,\n  )\n\n  override val filters: Seq[Filter[ScoredTweetsQuery, TweetCandidate]] =\n    Seq(ReplyFilter, RetweetFilter)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/ScoredTweetsPopularVideosCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline\n\nimport com.twitter.explore_ranker.{thriftscala => ert}\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.TweetypieStaticEntitiesFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.gate.MinCachedTweetsGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CachedScoredTweets\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CandidatePipeline\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.ScoredTweetsPopularVideosResponseFeatureTransformer\nimport com.twitter.home_mixer.util.CachedScoredTweetsHelper\nimport com.twitter.product_mixer.component_library.candidate_source.explore_ranker.ExploreRankerImmersiveRecsCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.marshaller.request.ClientContextMarshaller\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.decider.DeciderParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoredTweetsPopularVideosCandidatePipelineConfig @Inject() (\n  exploreRankerCandidateSource: ExploreRankerImmersiveRecsCandidateSource,\n  tweetypieStaticEntitiesFeatureHydrator: TweetypieStaticEntitiesFeatureHydrator)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      ert.ExploreRankerRequest,\n      ert.ExploreTweetRecommendation,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ScoredTweetsPopularVideos\")\n\n  private val MaxTweetsToFetch = 40\n\n  override val enabledDeciderParam: Option[DeciderParam[Boolean]] =\n    Some(CandidatePipeline.EnablePopularVideosParam)\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] = Seq(\n    MinCachedTweetsGate(identifier, CachedScoredTweets.MinCachedTweetsParam)\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    ert.ExploreRankerRequest\n  ] = { query =>\n    val excludedTweetIds = query.features.map(\n      CachedScoredTweetsHelper.tweetImpressionsAndCachedScoredTweets(_, identifier))\n\n    ert.ExploreRankerRequest(\n      clientContext = ClientContextMarshaller(query.clientContext),\n      product = ert.Product.HomeTimelineVideoInline,\n      productContext = Some(\n        ert.ProductContext.HomeTimelineVideoInline(ert.HomeTimelineVideoInline(excludedTweetIds))),\n      maxResults = Some(MaxTweetsToFetch)\n    )\n  }\n\n  override def candidateSource: BaseCandidateSource[\n    ert.ExploreRankerRequest,\n    ert.ExploreTweetRecommendation\n  ] = exploreRankerCandidateSource\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[ert.ExploreTweetRecommendation]\n  ] = Seq(ScoredTweetsPopularVideosResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    ert.ExploreTweetRecommendation,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.tweetId) }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ScoredTweetsQuery, TweetCandidate, _]\n  ] = Seq(tweetypieStaticEntitiesFeatureHydrator)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/ScoredTweetsStaticCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetEntityServiceFeatureHydrator\nimport com.twitter.home_mixer.functional_component.gate.AllowForYouRecommendationsGate\nimport com.twitter.home_mixer.model.HomeFeatures.SignupCountryFeature\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsStaticCandidatePipelineConfig.Identifier\nimport com.twitter.home_mixer.product.scored_tweets.candidate_source.StaticPostsCandidateSource\nimport com.twitter.home_mixer.product.scored_tweets.candidate_source.StaticSourceRequest\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CandidateSourceParams\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.ScoredTweetsStaticResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoredTweetsStaticCandidatePipelineConfig @Inject() (\n  staticPostsCandidateSource: StaticPostsCandidateSource,\n  tweetEntityServiceFeatureHydrator: TweetEntityServiceFeatureHydrator)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      StaticSourceRequest,\n      Long,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier = Identifier\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] =\n    Seq(AllowForYouRecommendationsGate)\n\n  override val supportedClientParam: Option[FSParam[Boolean]] =\n    Some(CandidateSourceParams.EnableStaticSourceParam)\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    StaticSourceRequest\n  ] = { query =>\n    val following =\n      query.features.map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty)).getOrElse(Seq.empty).toSet\n    val countryCode = query.clientContext.countryCode\n    val signupCountryCode = query.features.flatMap(_.getOrElse(SignupCountryFeature, None))\n    val countryCodes = Seq(countryCode, signupCountryCode).flatten\n    StaticSourceRequest(countryCodes = countryCodes, following = following)\n  }\n\n  override def candidateSource: CandidateSource[StaticSourceRequest, Long] =\n    staticPostsCandidateSource\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ScoredTweetsQuery, TweetCandidate, _]\n  ] = Seq(tweetEntityServiceFeatureHydrator)\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[Long]\n  ] = Seq(ScoredTweetsStaticResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    Long,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult) }\n}\n\nobject ScoredTweetsStaticCandidatePipelineConfig {\n  val SourceIdentifier = \"ScoredTweetsStatic\"\n  val Identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(SourceIdentifier)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/ScoredTweetsTweetMixerCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetEntityServiceFeatureHydrator\nimport com.twitter.home_mixer.functional_component.gate.AllowForYouRecommendationsGate\nimport com.twitter.home_mixer.product.scored_tweets.filter.ExtendedDirectedAtFilter\nimport com.twitter.home_mixer.product.scored_tweets.filter.QualifiedRepliesFilter\nimport com.twitter.tweet_mixer.{thriftscala => t}\nimport com.twitter.home_mixer.product.scored_tweets.gate.MinCachedTweetsGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CachedScoredTweets\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FetchParams\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.ScoredTweetsTweetMixerResponseFeatureTransformer\nimport com.twitter.home_mixer.util.CachedScoredTweetsHelper\nimport com.twitter.product_mixer.component_library.candidate_source.tweet_mixer.TweetMixerCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.module.TestUserMapper\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.marshaller.request.ClientContextMarshaller\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Candidate Pipeline Config that fetches tweets from TweetMixer.\n */\n@Singleton\nclass ScoredTweetsTweetMixerCandidatePipelineConfig @Inject() (\n  tweetMixerCandidateSource: TweetMixerCandidateSource,\n  tweetEntityServiceFeatureHydrator: TweetEntityServiceFeatureHydrator,\n  testUserMapper: TestUserMapper)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      t.TweetMixerRequest,\n      t.TweetResult,\n      TweetCandidate\n    ] {\n\n  import ScoredTweetsTweetMixerCandidatePipelineConfig._\n  override val identifier: CandidatePipelineIdentifier = Identifier\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] = Seq(\n    AllowForYouRecommendationsGate,\n    MinCachedTweetsGate(identifier, CachedScoredTweets.MinCachedTweetsParam)\n  )\n\n  override val candidateSource: BaseCandidateSource[t.TweetMixerRequest, t.TweetResult] =\n    tweetMixerCandidateSource\n\n  def getClientContext(\n    query: ScoredTweetsQuery\n  ): ClientContext = {\n    if (testUserMapper.isTestUser(query.clientContext)) testUserMapper(query.clientContext)\n    else query.clientContext\n  }\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    t.TweetMixerRequest\n  ] = { query =>\n    val excludedTweetIds = query.features.map(\n      CachedScoredTweetsHelper.tweetImpressionsAndCachedScoredTweets(_, identifier))\n\n    t.TweetMixerRequest(\n      clientContext = ClientContextMarshaller(getClientContext(query)),\n      product = t.Product.HomeRecommendedTweets,\n      productContext = Some(\n        t.ProductContext.HomeRecommendedTweetsProductContext(\n          t.HomeRecommendedTweetsProductContext(excludedTweetIds = excludedTweetIds.map(_.toSet)))),\n      maxResults = Some(query.params(FetchParams.TweetMixerMaxTweetsToFetchParam))\n    )\n  }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ScoredTweetsQuery, TweetCandidate, _]\n  ] = Seq(tweetEntityServiceFeatureHydrator)\n\n  override val filters: Seq[Filter[ScoredTweetsQuery, TweetCandidate]] = Seq(\n    QualifiedRepliesFilter,\n    ExtendedDirectedAtFilter\n  )\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[t.TweetResult]\n  ] = Seq(ScoredTweetsTweetMixerResponseFeatureTransformer())\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    t.TweetResult,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.tweetId) }\n}\n\nobject ScoredTweetsTweetMixerCandidatePipelineConfig {\n  val Identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ScoredTweetsTweetMixer\")\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/ScoredTweetsUtegCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline\n\nimport com.twitter.home_mixer.product.scored_tweets.gate.MinCachedTweetsGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CachedScoredTweets\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CandidatePipeline\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.TimelineRankerUtegQueryTransformer\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.ScoredTweetsUtegResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.candidate_source.timeline_ranker.TimelineRankerUtegCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelineranker.{thriftscala => t}\nimport com.twitter.timelines.configapi.decider.DeciderParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Candidate Pipeline Config that fetches tweets from the Timeline Ranker UTEG Candidate Source\n */\n@Singleton\nclass ScoredTweetsUtegCandidatePipelineConfig @Inject() (\n  timelineRankerUtegCandidateSource: TimelineRankerUtegCandidateSource)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      t.UtegLikedByTweetsQuery,\n      t.CandidateTweet,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ScoredTweetsUteg\")\n\n  override val enabledDeciderParam: Option[DeciderParam[Boolean]] =\n    Some(CandidatePipeline.EnableUtegParam)\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] = Seq(\n    MinCachedTweetsGate(identifier, CachedScoredTweets.MinCachedTweetsParam)\n  )\n\n  override val candidateSource: BaseCandidateSource[t.UtegLikedByTweetsQuery, t.CandidateTweet] =\n    timelineRankerUtegCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    t.UtegLikedByTweetsQuery\n  ] = TimelineRankerUtegQueryTransformer(identifier)\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[t.CandidateTweet]\n  ] = Seq(ScoredTweetsUtegResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    t.CandidateTweet,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.tweet.get.id) }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/earlybird/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/earlybird\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/earlybird\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/earlybird\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/param_gated\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/earlybird/ScoredTweetsCommunitiesCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.earlybird\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetEntityServiceFeatureHydrator\nimport com.twitter.home_mixer.model.HomeFeatures.CachedScoredTweetsFeature\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CandidateSourceParams\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.earlybird.CommunitiesEarlybirdQueryTransformer\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.earlybird.ScoredTweetsCommunitiesResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.candidate_source.earlybird.EarlybirdTweetCandidateSource\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.communities.CommunityNamesFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.communities.CommunityMembershipsFeature\nimport com.twitter.product_mixer.component_library.gate.EmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.gate.NonEmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.search.earlybird.{thriftscala => t}\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoredTweetsCommunitiesCandidatePipelineConfig @Inject() (\n  earlybirdTweetCandidateSource: EarlybirdTweetCandidateSource,\n  communityNamesFeatureHydrator: CommunityNamesFeatureHydrator,\n  communitiesEarlybirdQueryTransformer: CommunitiesEarlybirdQueryTransformer,\n  tweetEntityServiceFeatureHydrator: TweetEntityServiceFeatureHydrator)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      t.EarlybirdRequest,\n      t.ThriftSearchResult,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ScoredTweetsCommunities\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] =\n    Some(CandidateSourceParams.EnableCommunitiesCandidateSourceParam)\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] = Seq(\n    NonEmptySeqFeatureGate(CommunityMembershipsFeature),\n    EmptySeqFeatureGate(CachedScoredTweetsFeature)\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    t.EarlybirdRequest\n  ] = communitiesEarlybirdQueryTransformer\n\n  override def candidateSource: BaseCandidateSource[t.EarlybirdRequest, t.ThriftSearchResult] =\n    earlybirdTweetCandidateSource\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[t.ThriftSearchResult]\n  ] = Seq(ScoredTweetsCommunitiesResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    t.ThriftSearchResult,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.id) }\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ScoredTweetsQuery, TweetCandidate, _]\n  ] = Seq(tweetEntityServiceFeatureHydrator)\n\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[ScoredTweetsQuery, TweetCandidate, _]\n  ] = Seq(communityNamesFeatureHydrator)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/earlybird/ScoredTweetsEarlybirdFrsCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.earlybird\n\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.home_mixer.functional_component.candidate_source.EarlybirdCandidateSource\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.FrsSeedUsersQueryFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.gate.MinCachedTweetsGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CachedScoredTweets\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.earlybird.EarlybirdFrsQueryTransformer\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.earlybird.ScoredTweetsEarlybirdFrsResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseQueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Candidate Pipeline Config that fetches tweets from the earlybird FRS Candidate Source\n */\n@Singleton\nclass ScoredTweetsEarlybirdFrsCandidatePipelineConfig @Inject() (\n  earlybirdCandidateSource: EarlybirdCandidateSource,\n  frsSeedUsersQueryFeatureHydrator: FrsSeedUsersQueryFeatureHydrator,\n  clientId: ClientId)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      eb.EarlybirdRequest,\n      eb.ThriftSearchResult,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ScoredTweetsEarlybirdFrs\")\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] = Seq(\n    MinCachedTweetsGate(identifier, CachedScoredTweets.MinCachedTweetsParam)\n  )\n\n  override val queryFeatureHydration: Seq[\n    BaseQueryFeatureHydrator[ScoredTweetsQuery, _]\n  ] = Seq(frsSeedUsersQueryFeatureHydrator)\n\n  override val candidateSource: BaseCandidateSource[eb.EarlybirdRequest, eb.ThriftSearchResult] =\n    earlybirdCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    eb.EarlybirdRequest\n  ] = EarlybirdFrsQueryTransformer(identifier, clientId = Some(clientId.name))\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[eb.ThriftSearchResult]\n  ] = Seq(ScoredTweetsEarlybirdFrsResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    eb.ThriftSearchResult,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.id) }\n\n  override def filters: Seq[Filter[ScoredTweetsQuery, TweetCandidate]] = Seq.empty\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/earlybird/ScoredTweetsEarlybirdInNetworkCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.earlybird\n\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetEntityServiceFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.candidate_source.EarlybirdRealtimeCGTweetCandidateSource\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.FollowedUserScoresFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.IsExtendedReplyFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.filter.ExtendedDirectedAtFilter\nimport com.twitter.home_mixer.product.scored_tweets.filter.QualifiedRepliesFilter\nimport com.twitter.home_mixer.product.scored_tweets.gate.MinCachedTweetsGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CachedScoredTweets\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CandidateSourceParams\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.earlybird.EarlybirdInNetworkQueryTransformer\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.earlybird.ScoredTweetsEarlybirdInNetworkResponseFeatureTransformer\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated.ParamGatedQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseQueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport com.twitter.timelines.configapi.FSParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Candidate Pipeline Config that fetches tweets from the earlybird InNetwork Candidate Source\n */\n@Singleton\nclass ScoredTweetsEarlybirdInNetworkCandidatePipelineConfig @Inject() (\n  earlybirdTweetCandidateSource: EarlybirdRealtimeCGTweetCandidateSource,\n  followedUserScoresFeatureHydrator: FollowedUserScoresFeatureHydrator,\n  tweetEntityServiceFeatureHydrator: TweetEntityServiceFeatureHydrator,\n  clientId: ClientId)\n    extends CandidatePipelineConfig[\n      ScoredTweetsQuery,\n      eb.EarlybirdRequest,\n      eb.ThriftSearchResult,\n      TweetCandidate\n    ] {\n\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"ScoredTweetsEarlybirdInNetwork\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] = Some(\n    CandidateSourceParams.EnableInNetworkCandidateSourceParam)\n\n  override val gates: Seq[Gate[ScoredTweetsQuery]] = Seq(\n    MinCachedTweetsGate(identifier, CachedScoredTweets.MinCachedTweetsParam)\n  )\n\n  override val queryFeatureHydration: Seq[\n    BaseQueryFeatureHydrator[ScoredTweetsQuery, _]\n  ] = Seq(\n    ParamGatedQueryFeatureHydrator(\n      FeatureHydration.EnableFollowedUserScoreBackfillFeaturesParam,\n      followedUserScoresFeatureHydrator\n    )\n  )\n\n  override val candidateSource: BaseCandidateSource[eb.EarlybirdRequest, eb.ThriftSearchResult] =\n    earlybirdTweetCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    ScoredTweetsQuery,\n    eb.EarlybirdRequest\n  ] = EarlybirdInNetworkQueryTransformer(identifier, clientId = Some(clientId.name))\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[eb.ThriftSearchResult]\n  ] = Seq(ScoredTweetsEarlybirdInNetworkResponseFeatureTransformer)\n\n  override val preFilterFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ScoredTweetsQuery, TweetCandidate, _]\n  ] = Seq(tweetEntityServiceFeatureHydrator)\n\n  override val filters: Seq[Filter[ScoredTweetsQuery, TweetCandidate]] = Seq(\n    QualifiedRepliesFilter,\n    ExtendedDirectedAtFilter\n  )\n\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[ScoredTweetsQuery, TweetCandidate, _]\n  ] = Seq(IsExtendedReplyFeatureHydrator)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    eb.ThriftSearchResult,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.id) }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_source/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/earlybird\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"servo/repo/src/main/scala\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"src/thrift/com/twitter/timelines/pinned_timelines:thrift-scala\",\n        \"stitch/stitch-timelineservice\",\n        \"strato/config/columns/content_understanding:content_understanding-strato-client\",\n        \"strato/config/columns/timelines/pintweet:pintweet-strato-client\",\n        \"strato/config/src/thrift/com/twitter/strato/columns/content_understanding:content_understanding-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_source/CachedScoredTweetsCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_source\n\nimport com.twitter.home_mixer.util.CachedScoredTweetsHelper\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CachedScoredTweetsCandidateSource @Inject() ()\n    extends CandidateSource[PipelineQuery, hmt.ScoredTweet] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"CachedScoredTweets\")\n\n  override def apply(request: PipelineQuery): Stitch[Seq[hmt.ScoredTweet]] = {\n    Stitch.value(\n      request.features.map(CachedScoredTweetsHelper.unseenCachedScoredTweets).getOrElse(Seq.empty))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_source/ContentExplorationCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_source\n\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.ContentExplorationQueryRequest\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.Scan.Slice\nimport com.twitter.strato.generated.client.content_understanding.CategoryColdStartPostsMhClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.util.Random\n\ncase class ContentExplorationCandidateResponse(\n  tweetId: Long,\n  tier: String)\n\n@Singleton\nclass ContentExplorationCandidateSource @Inject() (\n  coldStartPostsColumn: CategoryColdStartPostsMhClientColumn)\n    extends CandidateSource[ContentExplorationQueryRequest, ContentExplorationCandidateResponse] {\n\n  private val MaxUserCategory = 5\n  private val MaxResultPerCategory = 100\n  private val MaxResult = 500\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"ContentExploration\")\n\n  private val scanner = coldStartPostsColumn.scanner\n\n  private def interweave(\n    candidates: Seq[Seq[ContentExplorationCandidateResponse]]\n  ): Seq[ContentExplorationCandidateResponse] = {\n    if (candidates.isEmpty) return Seq.empty\n    val maxLength = candidates.map(_.length).max\n    (0 until maxLength)\n      .flatMap { i =>\n        candidates.flatMap(seq => seq.lift(i))\n      }.take(MaxResult)\n  }\n\n  private def getSlice(): Slice[Long] = {\n    val now = System.currentTimeMillis()\n    val startId = SnowflakeId.firstIdFor(now - 6 * 60 * 60 * 1000L)\n    val endId = SnowflakeId.firstIdFor(now)\n\n    Slice[Long](\n      from = Some(startId),\n      to = Some(endId),\n      limit = Some(MaxResultPerCategory)\n    )\n  }\n\n  override def apply(\n    request: ContentExplorationQueryRequest\n  ): Stitch[Seq[ContentExplorationCandidateResponse]] = {\n    val tags = request.userCategories\n    val scan = getSlice()\n    val version = request.version\n\n    val scans: Seq[Stitch[Seq[ContentExplorationCandidateResponse]]] = tags.flatMap { tag =>\n      Seq(\n        scanner.scan((version + \"tier1_\" + tag._1, scan)).map { results =>\n          Random\n            .shuffle(results.map(_._1._2))\n            .map(id => ContentExplorationCandidateResponse(id, \"tier1\"))\n        },\n        scanner.scan((version + \"tier2_\" + tag._1, scan)).map { results =>\n          Random\n            .shuffle(results.map(_._1._2))\n            .map(id => ContentExplorationCandidateResponse(id, \"tier2\"))\n        }\n      )\n    }\n\n    Stitch.collect(scans).map(interweave)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_source/EarlybirdRealtimeCGTweetCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_source\n\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.EarlybirdRealtimCGEndpoint\nimport com.twitter.product_mixer.component_library.candidate_source.earlybird.EarlybirdTweetCandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.search.earlybird.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass EarlybirdRealtimeCGTweetCandidateSource @Inject() (\n  @Named(EarlybirdRealtimCGEndpoint) earlybirdService: t.EarlybirdService.MethodPerEndpoint)\n    extends EarlybirdTweetCandidateSource(earlybirdService) {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"EarlybirdRealtimeCGTweets\")\n}\n\n@Singleton\nclass EarlybirdNsfwCGTweetCandidateSource @Inject() (\n  @Named(EarlybirdRealtimCGEndpoint) earlybirdService: t.EarlybirdService.MethodPerEndpoint)\n    extends EarlybirdTweetCandidateSource(earlybirdService) {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"EarlybirdNsfwCGTweets\")\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_source/ListsCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_source\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.timelineservice.TimelineService\nimport com.twitter.timelineservice.{thriftscala => tls}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase class ListTweet(listId: Long, tweet: tls.Tweet)\n\n@Singleton\nclass ListsCandidateSource @Inject() (timelineService: TimelineService)\n    extends CandidateSource[Seq[tls.TimelineQuery], ListTweet] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"Lists\")\n\n  override def apply(requests: Seq[tls.TimelineQuery]): Stitch[Seq[ListTweet]] = Stitch\n    .traverse(requests) { request =>\n      timelineService.getTimeline(request).map { response =>\n        response.entries.collect {\n          case tls.TimelineEntry.Tweet(tweet) => ListTweet(response.timelineId.id, tweet)\n        }\n      }\n    }.map(_.flatten)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_source/StaticPostsCandidateSource.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.candidate_source\n\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.servo.cache.ExpiringLruInProcessCache\nimport com.twitter.servo.cache.InProcessCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.timelines.pintweet.PinnedPostsClientColumn\nimport com.twitter.timelines.pinned_timelines.{thriftscala => pt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.util.Random\n\ncase class StaticSourceRequest(countryCodes: Seq[String], following: Set[Long])\n\nobject StaticPostsCandidateSource {\n  private val BaseTTL = 30\n  private val StaticKey = 0\n  private val TTL = (BaseTTL + Random.nextInt(15)).minutes\n\n  val cache: InProcessCache[Int, Option[pt.PinnedPost]] =\n    new ExpiringLruInProcessCache(ttl = TTL, maximumSize = 100)\n}\n\n@Singleton\nclass StaticPostsCandidateSource @Inject() (pinnedPostsClientColumn: PinnedPostsClientColumn)\n    extends CandidateSource[StaticSourceRequest, Long] {\n\n  import StaticPostsCandidateSource._\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"StaticPosts\")\n\n  private val fetcher: Fetcher[Int, Unit, pt.PinnedPost] = pinnedPostsClientColumn.fetcher\n\n  override def apply(request: StaticSourceRequest): Stitch[Seq[Long]] = {\n    val postsStitch = cache.get(StaticKey).map(Stitch.value(_)).getOrElse {\n      fetcher.fetch(StaticKey).map { response =>\n        cache.set(StaticKey, response.v)\n        response.v\n      }\n    }\n\n    postsStitch.map {\n      _.map { post =>\n        val isInNetwork = request.following.contains(post.authorId)\n        val countryMatches = request.countryCodes.exists { code =>\n          post.countryCodes.forall(_.map(_.toLowerCase).contains(code.toLowerCase))\n        }\n        if (countryMatches && ((post.inNetwork && isInNetwork) || (post.outOfNetwork && !isInNetwork)))\n          Seq(post.postId)\n        else Seq.empty\n      }.getOrElse(Seq.empty)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/AncestorFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.tweetconvosvc.tweet_ancestor.{thriftscala => ta}\nimport com.twitter.tweetconvosvc.{thriftscala => tcs}\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AncestorFeatureHydrator @Inject() (\n  conversationServiceClient: tcs.ConversationService.MethodPerEndpoint)\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"Ancestor\")\n\n  override val features: Set[Feature[_, _]] = Set(AncestorsFeature)\n\n  private val DefaultFeatureMap = FeatureMapBuilder().add(AncestorsFeature, Seq.empty).build()\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = OffloadFuturePools.offloadFuture {\n    if (existingFeatures.getOrElse(InReplyToTweetIdFeature, None).isDefined) {\n      val ancestorsRequest = tcs.GetAncestorsRequest(Seq(candidate.id))\n      conversationServiceClient.getAncestors(ancestorsRequest).map { getAncestorsResponse =>\n        val ancestors = getAncestorsResponse.ancestors.headOption\n          .collect {\n            case tcs.TweetAncestorsResult.TweetAncestors(ancestorsResult)\n                if ancestorsResult.nonEmpty =>\n              ancestorsResult.head.ancestors ++ getTruncatedRootTweet(ancestorsResult.head)\n          }.getOrElse(Seq.empty)\n\n        FeatureMapBuilder().add(AncestorsFeature, ancestors).build()\n      }\n    } else Future.value(DefaultFeatureMap)\n  }\n\n  private def getTruncatedRootTweet(\n    ancestors: ta.TweetAncestors,\n  ): Option[ta.TweetAncestor] = {\n    ancestors.conversationRootAuthorId.collect {\n      case rootAuthorId\n          if ancestors.state == ta.ReplyState.Partial &&\n            ancestors.ancestors.last.tweetId != ancestors.conversationId =>\n        ta.TweetAncestor(ancestors.conversationId, rootAuthorId)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/AuthorFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.AuthorFeatureRepository\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.author_features.AuthorFeaturesAdapter\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.servo.repository.KeyValueResult\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.author_features.v1.{thriftjava => af}\nimport com.twitter.util.Future\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject AuthorFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass AuthorFeatureHydrator @Inject() (\n  @Named(AuthorFeatureRepository) client: KeyValueRepository[Seq[Long], Long, af.AuthorFeatures],\n  override val statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"AuthorFeature\")\n\n  override val features: Set[Feature[_, _]] = Set(AuthorFeature)\n\n  override val statScope: String = identifier.toString\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val possiblyAuthorIds = extractKeys(candidates)\n    val authorIds = possiblyAuthorIds.flatten\n\n    val response: Future[KeyValueResult[Long, af.AuthorFeatures]] =\n      if (authorIds.nonEmpty) client(authorIds)\n      else Future.value(KeyValueResult.empty)\n\n    response.map { result =>\n      possiblyAuthorIds.map { possiblyAuthorId =>\n        val value = observedGet(key = possiblyAuthorId, keyValueResult = result)\n        val transformedValue = postTransformer(value)\n\n        FeatureMapBuilder().add(AuthorFeature, transformedValue).build()\n      }\n    }\n  }\n\n  private def postTransformer(authorFeatures: Try[Option[af.AuthorFeatures]]): Try[DataRecord] = {\n    authorFeatures.map {\n      _.map { features => AuthorFeaturesAdapter.adaptToDataRecords(features).asScala.head }\n        .getOrElse(new DataRecord())\n    }\n  }\n\n  private def extractKeys(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] = {\n    candidates.map { candidate =>\n      CandidatesUtil.getOriginalAuthorId(candidate.features)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/AuthorIsCreatorFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIsCreatorFeature\nimport com.twitter.home_mixer.util.MissingKeyException\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.audiencerewards.audienceRewardsService.GetSuperFollowEligibilityOnUserClientColumn\nimport com.twitter.util.Throw\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AuthorIsCreatorFeatureHydrator @Inject() (\n  getSuperFollowEligibilityOnUserClientColumn: GetSuperFollowEligibilityOnUserClientColumn,\n  statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"AuthorIsCreator\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(AuthorIsCreatorFeature)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyFailureCounter = scopedStatsReceiver.counter(\"key/failure\")\n\n  private val MissingKeyFeatureMap = FeatureMapBuilder()\n    .add(AuthorIsCreatorFeature, Throw(MissingKeyException))\n    .build()\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(AuthorIsCreatorFeature, false)\n    .build()\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    OffloadFuturePools.offloadStitch {\n      val authorIds = candidates.flatMap(_.features.getOrElse(AuthorIdFeature, None)).distinct\n      Stitch\n        .collect {\n          authorIds.map { authorId =>\n            getSuperFollowEligibilityOnUserClientColumn.fetcher\n              .fetch(authorId)\n              .map { authorId -> _.v }\n          }\n        }.map { authorIdsToIsCreator =>\n          val authorIdsToIsCreatorMap = authorIdsToIsCreator.toMap\n          candidates.map { candidate =>\n            candidate.features.getOrElse(AuthorIdFeature, None) match {\n              case Some(authorId) =>\n                authorIdsToIsCreatorMap.get(authorId) match {\n                  case Some(response) =>\n                    keyFoundCounter.incr()\n                    FeatureMapBuilder()\n                      .add(AuthorIsCreatorFeature, response.getOrElse(false)).build()\n                  case _ =>\n                    keyFailureCounter.incr()\n                    DefaultFeatureMap\n                }\n              case _ => MissingKeyFeatureMap\n            }\n          }\n        }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/content\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/earlybird\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/earlybird\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_ranker\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_is_nsfw\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_visibility_reason\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"servo/repo/src/main/scala\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/conversation_features\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters\",\n        \"src/scala/com/twitter/timelines/prediction/features/semantic_core_features\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"strato/config/columns/content_understanding:content_understanding-strato-client\",\n        \"strato/config/columns/lists/reads:core-strato-client\",\n        \"strato/config/columns/recommendations/user-signal-service:user-signal-service-strato-client\",\n        \"strato/config/columns/tweetypie/managed:managed-strato-client\",\n        \"timelines/src/main/scala/com/twitter/timelines/earlybird/common/utils\",\n        \"timelines/src/main/scala/com/twitter/timelines/model/types\",\n        \"tweetsource/common/src/main/thrift:thrift-scala\",\n        \"user-signal-service/thrift/src/main/thrift:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/CachedScoredTweetsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.transport.Transport\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.ScoredTweetsCache\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CachedScoredTweets\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.TtlCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n/**\n * Fetch scored Tweets from cache and exclude the expired ones\n */\n@Singleton\ncase class CachedScoredTweetsQueryFeatureHydrator @Inject() (\n  @Named(ScoredTweetsCache)\n  scoredTweetsCache: TtlCache[Long, hmt.ScoredTweetsResponse])\n    extends QueryFeatureHydrator[PipelineQuery]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"CachedScoredTweets\")\n\n  override val features: Set[Feature[_, _]] = Set(CachedScoredTweetsFeature)\n\n  override def onlyIf(query: PipelineQuery): Boolean = {\n    val serviceIdentifier = ServiceIdentifier.fromCertificate(Transport.peerCertificate)\n    serviceIdentifier.role != \"explore-mixer\" && serviceIdentifier.role != \"video-mixer\"\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n    val tweetScoreTtl = query.params(CachedScoredTweets.TTLParam)\n\n    Stitch.callFuture(scoredTweetsCache.get(Seq(userId))).map { keyValueResult =>\n      keyValueResult(userId) match {\n        case Return(cachedCandidatesOpt) =>\n          val cachedScoredTweets = cachedCandidatesOpt.map(_.scoredTweets).getOrElse(Seq.empty)\n          val nonExpiredTweets = cachedScoredTweets.filter { tweet =>\n            tweet.lastScoredTimestampMs.exists(Time.fromMilliseconds(_).untilNow < tweetScoreTtl)\n          }\n          FeatureMap(CachedScoredTweetsFeature, nonExpiredTweets)\n        case Throw(exception) => throw exception\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/EarlybirdFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.UserLanguagesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DeviceLanguageFeature\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdSearchResultFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetUrlsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserScreenNameFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.EarlybirdRepository\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.earlybird.EarlybirdAdapter\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.home_mixer.util.earlybird.EarlybirdResponseUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.search.common.constants.{thriftscala => scc}\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport com.twitter.servo.keyvalue.KeyValueResult\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.earlybird.common.utils.InNetworkEngagement\nimport com.twitter.util.Future\nimport com.twitter.util.Return\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject EarlybirdDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass EarlybirdFeatureHydrator @Inject() (\n  @Named(EarlybirdRepository) client: KeyValueRepository[\n    (Seq[Long], Long),\n    Long,\n    eb.ThriftSearchResult\n  ],\n  override val statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"Earlybird\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    EarlybirdDataRecordFeature,\n    EarlybirdFeature,\n    EarlybirdSearchResultFeature,\n    SourceTweetEarlybirdFeature,\n    TweetUrlsFeature\n  )\n\n  override val statScope: String = identifier.toString\n\n  private val scopedStatsReceiver = statsReceiver.scope(statScope)\n  private val originalKeyFoundCounter = scopedStatsReceiver.counter(\"originalKey/found\")\n  private val originalKeyNotFoundCounter = scopedStatsReceiver.counter(\"originalKey/notFound\")\n\n  private val ebFeaturesNotExistPredicate: CandidateWithFeatures[TweetCandidate] => Boolean =\n    candidate => candidate.features.getOrElse(EarlybirdFeature, None).isEmpty\n\n  private val InternalUrlPattern = \"\"\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val candidatesWithoutExistingEBFeatures = candidates\n      .filter { candidate =>\n        val isEmpty = ebFeaturesNotExistPredicate(candidate)\n        if (isEmpty) originalKeyNotFoundCounter.incr() else originalKeyFoundCounter.incr()\n        isEmpty\n      }\n    val (candidatesWithSearchFeatures, candidatesWithoutSearchFeatures) =\n      candidatesWithoutExistingEBFeatures.partition { candidate =>\n        candidate.features.getOrElse(EarlybirdSearchResultFeature, None).nonEmpty\n      }\n    // hydrate both candidates and their sources\n    val candidateIdsToHydrate = (\n      candidatesWithSearchFeatures\n        .filter(_.features.getOrElse(IsRetweetFeature, false))\n        .map(CandidatesUtil.getOriginalTweetId) ++\n        candidatesWithoutSearchFeatures.map(_.candidate.id) ++\n        candidatesWithoutSearchFeatures.map(CandidatesUtil.getOriginalTweetId)\n    ).distinct\n\n    client((candidateIdsToHydrate, query.getRequiredUserId))\n      .flatMap(\n        handleResponse(query, candidates, _, candidateIdsToHydrate, candidatesWithSearchFeatures))\n  }\n\n  private[feature_hydrator] def getFeatureMap(\n    candidate: CandidateWithFeatures[TweetCandidate],\n    query: PipelineQuery,\n    idToSearchResults: Map[Long, eb.ThriftSearchResult],\n    screenName: Option[String],\n    userLanguages: Seq[scc.ThriftLanguage],\n    uiLanguage: Option[scc.ThriftLanguage],\n    tweetCountByAuthorId: Map[Long, Int],\n    followedUserIds: Set[Long],\n    mutuallyFollowedUserIds: Set[Long],\n    inNetworkEngagement: InNetworkEngagement,\n  ): FeatureMap = {\n    val candidateIsRetweet = candidate.features.getOrElse(IsRetweetFeature, false)\n    val existingEbFeatures = candidate.features.getOrElse(EarlybirdFeature, None)\n    val existingSourceTweetEbFeatures =\n      candidate.features.getOrElse(SourceTweetEarlybirdFeature, None)\n\n    val tweetEbFeatures =\n      if (existingEbFeatures.nonEmpty) existingEbFeatures\n      else {\n        idToSearchResults.get(candidate.candidate.id).map { searchResult =>\n          EarlybirdResponseUtil.getThriftTweetFeaturesFromSearchResult(\n            searcherUserId = query.getRequiredUserId,\n            screenName,\n            userLanguages,\n            uiLanguage,\n            tweetCountByAuthorId,\n            followedUserIds,\n            mutuallyFollowedUserIds,\n            idToSearchResults,\n            inNetworkEngagement,\n            searchResult\n          )\n        }\n      }\n\n    val sourceTweetEbFeatures = if (candidateIsRetweet) {\n      if (existingSourceTweetEbFeatures.nonEmpty) existingSourceTweetEbFeatures\n      else {\n        idToSearchResults.get(CandidatesUtil.getOriginalTweetId(candidate)).map { searchResult =>\n          EarlybirdResponseUtil.getThriftTweetFeaturesFromSearchResult(\n            searcherUserId = query.getRequiredUserId,\n            screenName,\n            userLanguages,\n            uiLanguage,\n            tweetCountByAuthorId,\n            followedUserIds,\n            mutuallyFollowedUserIds,\n            idToSearchResults,\n            inNetworkEngagement,\n            searchResult\n          )\n        }\n      }\n    } else None\n\n    val originalTweetEbFeatures =\n      if (sourceTweetEbFeatures.nonEmpty) sourceTweetEbFeatures else tweetEbFeatures\n    val earlybirdDataRecord =\n      EarlybirdAdapter.adaptToDataRecords(originalTweetEbFeatures).asScala.head\n\n    val tesUrls = candidate.features.getOrElse(TweetUrlsFeature, Seq.empty)\n    val urls =\n      if (tesUrls.isEmpty) tweetEbFeatures.flatMap(_.urlsList).getOrElse(Seq.empty) else tesUrls\n\n    val rdr = new RichDataRecord(earlybirdDataRecord)\n\n    FeatureMapBuilder(sizeHint = 5)\n      .add(EarlybirdFeature, tweetEbFeatures)\n      .add(EarlybirdDataRecordFeature, rdr.getRecord)\n      .add(EarlybirdSearchResultFeature, idToSearchResults.get(candidate.candidate.id))\n      .add(SourceTweetEarlybirdFeature, sourceTweetEbFeatures)\n      .add(TweetUrlsFeature, urls)\n      .build()\n  }\n\n  private def handleResponse(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n    results: KeyValueResult[Long, eb.ThriftSearchResult],\n    candidateIdsToHydrate: Seq[Long],\n    candidatesWithSearchFeatures: Seq[CandidateWithFeatures[TweetCandidate]],\n  ): Future[Seq[FeatureMap]] = {\n    val batchSize = 64\n    val queryFeatureMap = query.features.getOrElse(FeatureMap.empty)\n    val userLanguages = queryFeatureMap.getOrElse(UserLanguagesFeature, Seq.empty)\n    val uiLanguageCode = queryFeatureMap.getOrElse(DeviceLanguageFeature, None)\n    val screenName = queryFeatureMap.getOrElse(UserScreenNameFeature, None)\n    val followedUserIds = queryFeatureMap.getOrElse(SGSFollowedUsersFeature, Seq.empty).toSet\n    val mutuallyFollowedUserIds =\n      queryFeatureMap.getOrElse(SGSMutuallyFollowedUsersFeature, Seq.empty).toSet\n\n    val searchResults = candidateIdsToHydrate\n      .map { id =>\n        observedGet(Some(id), results)\n      }.collect {\n        case Return(Some(value)) => value\n      }\n\n    val allSearchResults = searchResults ++ candidatesWithSearchFeatures.flatMap { candidate =>\n      candidate.features.getOrElse(EarlybirdSearchResultFeature, None)\n    }\n\n    val idToSearchResults = allSearchResults.map { searchResults =>\n      searchResults.id -> searchResults\n    }.toMap\n\n    val uiLanguage = EarlybirdResponseUtil.getLanguage(uiLanguageCode)\n    val tweetCountByAuthorId = EarlybirdResponseUtil.getTweetCountByAuthorId(allSearchResults)\n    val inNetworkEngagement =\n      InNetworkEngagement(followedUserIds.toSeq, mutuallyFollowedUserIds, allSearchResults)\n\n    OffloadFuturePools\n      .offloadBatchElementToElement[CandidateWithFeatures[TweetCandidate], FeatureMap](\n        candidates,\n        candidate =>\n          getFeatureMap(\n            candidate = candidate,\n            query = query,\n            idToSearchResults = idToSearchResults,\n            screenName = screenName,\n            userLanguages = userLanguages,\n            uiLanguage = uiLanguage,\n            tweetCountByAuthorId = tweetCountByAuthorId,\n            followedUserIds = followedUserIds,\n            mutuallyFollowedUserIds = mutuallyFollowedUserIds,\n            inNetworkEngagement = inNetworkEngagement\n          ),\n        batchSize\n      )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/FollowedUserScoresFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject FollowedUserScoresFeature extends Feature[PipelineQuery, Map[Long, Double]]\n\n@Singleton\ncase class FollowedUserScoresFeatureHydrator @Inject() ()\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  private val UpdateFollowedUserThreshold = 1000\n  private val MaxFollowedUsers = 1500\n  private val DefaultAuthorScore = 2D\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"FollowedUserScores\")\n\n  override def features: Set[Feature[_, _]] = Set(FollowedUserScoresFeature)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val sgsFollowedUserIds =\n      query.features.map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty)).toSeq.flatten\n    val sgsFollowedUserIdSet = sgsFollowedUserIds.toSet\n    val authorScoreMap = query.features\n      .map(_.getOrElse(RealGraphInNetworkScoresFeature, Map.empty[Long, Double]))\n      .getOrElse(Map.empty)\n\n    val filteredAuthorScoreMap = authorScoreMap.filter { entry =>\n      sgsFollowedUserIdSet.contains(entry._1)\n    }\n\n    val updatedSgsFollowedUserIds = if (sgsFollowedUserIds.size >= UpdateFollowedUserThreshold) {\n      filteredAuthorScoreMap.keySet ++\n        sgsFollowedUserIds\n          .filter(!filteredAuthorScoreMap.keySet.contains(_))\n          .take(Math.max(0, MaxFollowedUsers - filteredAuthorScoreMap.size))\n    } else sgsFollowedUserIds\n\n    val existingAuthorIdScores = filteredAuthorScoreMap.values.toList.sorted\n    val imputedScoreIndex =\n      Math.min(existingAuthorIdScores.length - 1, (existingAuthorIdScores.length * 0.5f).toInt)\n    val imputedScore =\n      if (imputedScoreIndex >= 0) existingAuthorIdScores(imputedScoreIndex)\n      else DefaultAuthorScore\n\n    val updatedAuthorScoreMap = updatedSgsFollowedUserIds\n      .map(_ -> imputedScore).toMap ++ filteredAuthorScoreMap\n\n    Stitch.value {\n      FeatureMapBuilder()\n        .add(FollowedUserScoresFeature, updatedAuthorScoreMap)\n        .build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/FrsSeedUsersQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.follow_recommendations.{thriftscala => frs}\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.product_mixer.component_library.candidate_source.recommendations.UserFollowRecommendationsCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyView\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject FrsSeedUserIdsFeature extends Feature[TweetCandidate, Option[Seq[Long]]]\nobject FrsUserToFollowedByUserIdsFeature extends Feature[TweetCandidate, Map[Long, Seq[Long]]]\n\n@Singleton\ncase class FrsSeedUsersQueryFeatureHydrator @Inject() (\n  userFollowRecommendationsCandidateSource: UserFollowRecommendationsCandidateSource)\n    extends QueryFeatureHydrator[ScoredTweetsQuery] {\n\n  private val maxUsersToFetch = 100\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"FrsSeedUsers\")\n\n  override def features: Set[Feature[_, _]] = Set(\n    FrsSeedUserIdsFeature,\n    FrsUserToFollowedByUserIdsFeature\n  )\n\n  override def hydrate(query: ScoredTweetsQuery): Stitch[FeatureMap] = {\n    val frsRequest = frs.RecommendationRequest(\n      clientContext = frs.ClientContext(query.getOptionalUserId),\n      displayLocation = frs.DisplayLocation.HomeTimelineTweetRecs,\n      maxResults = Some(maxUsersToFetch)\n    )\n\n    userFollowRecommendationsCandidateSource(StratoKeyView(frsRequest, Unit))\n      .map { userRecommendations: Seq[frs.UserRecommendation] =>\n        val seedUserIds = userRecommendations.map(_.userId)\n        val seedUserIdsSet = seedUserIds.toSet\n\n        val userToFollowedByUserIds: Map[Long, Seq[Long]] = userRecommendations.flatMap {\n          userRecommendation =>\n            if (seedUserIdsSet.contains(userRecommendation.userId)) {\n              val followProof =\n                userRecommendation.reason.flatMap(_.accountProof).flatMap(_.followProof)\n              val followedByUserIds = followProof.map(_.userIds).getOrElse(Seq.empty)\n              Some(userRecommendation.userId -> followedByUserIds)\n            } else {\n              None\n            }\n        }.toMap\n\n        FeatureMapBuilder()\n          .add(FrsSeedUserIdsFeature, Some(seedUserIds))\n          .add(FrsUserToFollowedByUserIdsFeature, userToFollowedByUserIds)\n          .build()\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/GizmoduckAuthorFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.ads.entities.db.{thriftscala => ae}\nimport com.twitter.gizmoduck.{thriftscala => gt}\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIsBlueVerifiedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIsProtectedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsSupportAccountReplyFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.snowflake.id.SnowflakeId\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GizmoduckAuthorFeatureHydrator @Inject() (gizmoduck: gt.UserService.MethodPerEndpoint)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"GizmoduckAuthor\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(AuthorIsBlueVerifiedFeature, AuthorIsProtectedFeature, IsSupportAccountReplyFeature)\n\n  private val queryFields: Set[gt.QueryFields] =\n    Set(gt.QueryFields.AdvertiserAccount, gt.QueryFields.Profile, gt.QueryFields.Safety)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val authorIds = candidates.flatMap(_.features.getOrElse(AuthorIdFeature, None))\n\n    val response = gizmoduck.get(\n      userIds = authorIds.distinct,\n      queryFields = queryFields,\n      context = gt.LookupContext()\n    )\n\n    response.map { hydratedAuthors =>\n      val userMetadataMap = hydratedAuthors\n        .collect {\n          case userResult if userResult.user.isDefined =>\n            val user = userResult.user.get\n            val blueVerified = user.safety.flatMap(_.isBlueVerified).getOrElse(false)\n            val isProtected = user.safety.exists(_.isProtected)\n            (user.id, (blueVerified, isProtected))\n        }.toMap.withDefaultValue((false, false))\n\n      candidates.map { candidate =>\n        val authorId = candidate.features.get(AuthorIdFeature).get\n        val (isBlueVerified, isProtected) = userMetadataMap(authorId)\n\n        // Some accounts run promotions on Twitter and send replies automatically.\n        // We assume that a reply that took more than one minute is not an auto-reply.\n        // If time difference doesn't exist, this means that one of the tweets was\n        // not snowflake and therefore much older, and therefore OK as an extended reply.\n        val timeDifference = candidate.features.getOrElse(InReplyToTweetIdFeature, None).map {\n          SnowflakeId.timeFromId(candidate.candidate.id) - SnowflakeId.timeFromId(_)\n        }\n        val isAutoReply = timeDifference.exists(_ < 1.minute)\n\n        FeatureMapBuilder()\n          .add(AuthorIsBlueVerifiedFeature, isBlueVerified)\n          .add(AuthorIsProtectedFeature, isProtected)\n          .add(IsSupportAccountReplyFeature, isAutoReply)\n          .build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/GraphTwoHopFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.graph_feature_service.{thriftscala => gfs}\nimport com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.GraphTwoHopRepository\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.two_hop_features.TwoHopFeaturesAdapter\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject GraphTwoHopFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass GraphTwoHopFeatureHydrator @Inject() (\n  @Named(GraphTwoHopRepository) client: KeyValueRepository[(Seq[Long], Long), Long, Seq[\n    gfs.IntersectionValue\n  ]],\n  override val statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"GraphTwoHop\")\n\n  override val features: Set[Feature[_, _]] = Set(GraphTwoHopFeature, FollowedByUserIdsFeature)\n\n  override val statScope: String = identifier.toString\n\n  private val twoHopFeaturesAdapter = new TwoHopFeaturesAdapter\n\n  private val FollowFeatureType = gfs.FeatureType(gfs.EdgeType.Following, gfs.EdgeType.FollowedBy)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    // Apply filters to in network candidates for retweets only.\n    val (inNetworkCandidates, oonCandidates) = candidates.partition { candidate =>\n      candidate.features.getOrElse(FromInNetworkSourceFeature, false)\n    }\n\n    val inNetworkCandidatesToHydrate =\n      inNetworkCandidates.filter(_.features.getOrElse(IsRetweetFeature, false))\n\n    val candidatesToHydrate = (inNetworkCandidatesToHydrate ++ oonCandidates)\n      .flatMap(candidate => CandidatesUtil.getOriginalAuthorId(candidate.features)).distinct\n\n    val response = client((candidatesToHydrate, query.getRequiredUserId))\n\n    response.map { result =>\n      candidates.map { candidate =>\n        val originalAuthorId = CandidatesUtil.getOriginalAuthorId(candidate.features)\n\n        val value = observedGet(key = originalAuthorId, keyValueResult = result)\n        val transformedValue = postTransformer(value)\n        val followedByUserIds = value.toOption\n          .flatMap(getFollowedByUserIds(_))\n          .getOrElse(Seq.empty)\n\n        FeatureMapBuilder()\n          .add(GraphTwoHopFeature, transformedValue)\n          .add(FollowedByUserIdsFeature, followedByUserIds)\n          .build()\n      }\n    }\n  }\n\n  private def getFollowedByUserIds(input: Option[Seq[gfs.IntersectionValue]]): Option[Seq[Long]] =\n    input.map(_.filter(_.featureType == FollowFeatureType).flatMap(_.intersectionIds).flatten)\n\n  private def postTransformer(input: Try[Option[Seq[gfs.IntersectionValue]]]): Try[DataRecord] =\n    input.map(twoHopFeaturesAdapter.adaptToDataRecords(_).asScala.head)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/InvalidateCachedScoredTweetsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.CachedScoredTweetsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.LastNonPollingTimeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserLastExplicitSignalTimeFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Invalidates (flushes) cached ScoredTweets upon detecting explicit signal from user made since\n * previous request\n */\nobject InvalidateCachedScoredTweetsQueryFeatureHydrator\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"InvalidateCachedScoredTweets\")\n\n  override val features: Set[Feature[_, _]] = Set(CachedScoredTweetsFeature)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val lastRequestTime =\n      query.features.getOrElse(FeatureMap.empty).getOrElse(LastNonPollingTimeFeature, None)\n    val lastExplicitSignalTime =\n      query.features\n        .getOrElse(FeatureMap.empty).getOrElse(UserLastExplicitSignalTimeFeature, None)\n    val hasRecentExplicitSignal = lastExplicitSignalTime\n      .flatMap { engagementTime =>\n        lastRequestTime.map { requestTime => engagementTime > requestTime }\n      }.getOrElse(false)\n\n    val featureMap = if (hasRecentExplicitSignal) {\n      FeatureMap(CachedScoredTweetsFeature, Seq.empty)\n    } else {\n      val cachedTweets =\n        query.features.getOrElse(FeatureMap.empty).getOrElse(CachedScoredTweetsFeature, Seq.empty)\n      FeatureMap(CachedScoredTweetsFeature, cachedTweets)\n    }\n\n    Stitch.value(featureMap)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/IsColdStartPostFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.IsColdStartPostFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.IsColdStartPostInMemCache\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.InProcessCache\nimport com.twitter.strato.generated.client.content_understanding.ColdStartPostsMetadataMhClientColumn\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.columns.content_understanding.content_exploration.thriftscala.ColdStartPostStatus\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass IsColdStartPostFeatureHydrator @Inject() (\n  coldStartPostsMetadataMhClientColumn: ColdStartPostsMetadataMhClientColumn,\n  @Named(IsColdStartPostInMemCache) isColdStartPostInMemCache: InProcessCache[\n    Long,\n    Boolean\n  ],\n  statsReceiver: StatsReceiver)\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"IsColdStartPost\")\n\n  override val features: Set[Feature[_, _]] = Set(IsColdStartPostFeature)\n\n  private val DefaultFeatureMap = FeatureMap(IsColdStartPostFeature, false)\n\n  private val ineligibleStatusSet: Set[ColdStartPostStatus] =\n    Set(ColdStartPostStatus.Tier1Ineligible, ColdStartPostStatus.Tier1IneligibleHighQuality)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val cacheHitCounter = scopedStatsReceiver.counter(\"cache/hit\")\n  private val cacheMissCounter = scopedStatsReceiver.counter(\"cache/miss\")\n  private val storeHitCounter = scopedStatsReceiver.counter(\"store/hit\")\n  private val storeMissCounter = scopedStatsReceiver.counter(\"store/miss\")\n  private val fetchExceptionCounter = scopedStatsReceiver.counter(\"fetch/exception\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    val postId = candidate.id\n    isColdStartPostInMemCache\n      .get(postId)\n      .map { cachedValue =>\n        cacheHitCounter.incr()\n        Stitch.value(FeatureMap(IsColdStartPostFeature, cachedValue))\n      }.getOrElse {\n        cacheMissCounter.incr()\n        coldStartPostsMetadataMhClientColumn.fetcher\n          .fetch(postId)\n          .map { response =>\n            if (response.v.isDefined) {\n              storeHitCounter.incr()\n            } else {\n              storeMissCounter.incr()\n            }\n            val isColdStartPost = response.v.flatMap(_.status) match {\n              case Some(status) => !ineligibleStatusSet.contains(status)\n              case _ => false\n            }\n            isColdStartPostInMemCache.set(postId, isColdStartPost)\n            FeatureMap(IsColdStartPostFeature, isColdStartPost)\n          }.handle {\n            case _: Exception =>\n              fetchExceptionCounter.incr()\n              DefaultFeatureMap\n          }\n      }\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/IsExtendedReplyFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsExtendedReplyFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\n\nobject IsExtendedReplyFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"IsExtendedReply\")\n\n  override def features: Set[Feature[_, _]] = Set(IsExtendedReplyFeature)\n\n  private val TrueFeatureMap = FeatureMapBuilder().add(IsExtendedReplyFeature, true).build()\n  private val FalseFeatureMap = FeatureMapBuilder().add(IsExtendedReplyFeature, false).build()\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offload {\n    val followedUsers =\n      query.features.map(_.get(SGSFollowedUsersFeature)).getOrElse(Seq.empty).toSet\n\n    candidates.map { candidate =>\n      val features = candidate.features\n      val isExtendedReply = features.getOrElse(InReplyToTweetIdFeature, None).nonEmpty &&\n        !features.getOrElse(IsRetweetFeature, false) &&\n        features.getOrElse(InReplyToUserIdFeature, None).exists(!followedUsers.contains(_))\n\n      if (isExtendedReply) TrueFeatureMap else FalseFeatureMap\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/ListIdsQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.socialgraph.{thriftscala => sg}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.SocialGraph\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase object ListIdsFeature extends FeatureWithDefaultOnFailure[PipelineQuery, Seq[Long]] {\n  override val defaultValue: Seq[Long] = Seq.empty\n}\n\n@Singleton\nclass ListIdsQueryFeatureHydrator @Inject() (socialGraph: SocialGraph)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"ListIds\")\n\n  override val features: Set[Feature[_, _]] = Set(ListIdsFeature)\n\n  private val MaxListsToFetch = 20\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n\n    val ownedSubscribedRequest = sg.IdsRequest(\n      relationships = Seq(\n        sg.SrcRelationship(userId, sg.RelationshipType.ListIsSubscriber, hasRelationship = true),\n        sg.SrcRelationship(userId, sg.RelationshipType.ListOwning, hasRelationship = true)\n      ),\n      pageRequest = Some(sg.PageRequest(selectAll = Some(false), count = Some(MaxListsToFetch))),\n      context = Some(\n        sg.LookupContext(\n          includeInactive = false,\n          performUnion = Some(true),\n          includeAll = Some(false)\n        )\n      )\n    )\n\n    socialGraph.ids(ownedSubscribedRequest).map { response =>\n      FeatureMapBuilder().add(ListIdsFeature, response.ids).build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/ListNameFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.ListIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ListNameFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.socialgraph.{thriftscala => sg}\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.lists.reads.CoreOnListClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ListNameFeatureHydrator @Inject() (coreOnListClientColumn: CoreOnListClientColumn)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"ListName\")\n\n  override val features: Set[Feature[_, _]] = Set(ListNameFeature)\n\n  private val fetcher: Fetcher[Long, Unit, sg.SocialgraphList] = coreOnListClientColumn.fetcher\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val listIds = candidates.flatMap(_.features.getOrElse(ListIdFeature, None)).distinct\n\n    val listIdNameMapStitch = Stitch.collect {\n      listIds.map { listId => listId -> fetcher.fetch(listId).map(_.v.map(_.name)) }.toMap\n    }\n\n    listIdNameMapStitch.map { listIdNameMap =>\n      candidates.map { candidate =>\n        val listId = candidate.features.getOrElse(ListIdFeature, None)\n        val listName = listId.flatMap(listIdNameMap.get).flatten\n        FeatureMapBuilder().add(ListNameFeature, listName).build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/LowSignalUserQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.LowSignalUserMaxSignalCount\nimport com.twitter.home_mixer.util.SignalUtil\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.recommendations.user_signal_service.SignalsClientColumn\nimport com.twitter.usersignalservice.{thriftscala => uss}\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class LowSignalUserQueryFeatureHydrator @Inject() (\n  signalsClientColumn: SignalsClientColumn)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"LowSignalUser\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(\n      LowSignalUserFeature,\n      UserRecentEngagementTweetIdsFeature,\n      UserLastExplicitSignalTimeFeature)\n\n  val fetcher: Fetcher[SignalsClientColumn.Key, Unit, SignalsClientColumn.Value] =\n    signalsClientColumn.fetcher\n\n  val MaxFetch = 15L\n  val MinSignalFavCount = 0\n  val MaxSignalFavCount = 5000000\n  val LowSignalUserMaxSignalAge = 90.days\n\n  private def getTimestamp(signal: uss.Signal): Option[Time] = {\n    if (signal.timestamp == 0L) None else Some(Time.fromMilliseconds(signal.timestamp))\n  }\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val signalRequests = SignalUtil.ExplicitSignals.map { signal =>\n      uss.SignalRequest(\n        maxResults = Some(MaxFetch),\n        signalType = signal,\n        minFavCount = Some(MinSignalFavCount),\n        maxFavCount = Some(MaxSignalFavCount)\n      )\n    }\n\n    val batchSignalRequest = uss.BatchSignalRequest(\n      userId = query.getRequiredUserId,\n      signalRequest = signalRequests,\n      clientId = Some(uss.ClientIdentifier.CrMixerHome)\n    )\n\n    fetcher.fetch(batchSignalRequest).map { response =>\n      val signals = response.v.map(_.signalResponse.values.toSeq.flatten).getOrElse(Seq.empty)\n      val tweetIds = signals.collect {\n        case signal if signal.targetInternalId.isDefined =>\n          signal.targetInternalId.get match {\n            case id: com.twitter.simclusters_v2.thriftscala.InternalId.TweetId =>\n              Some(id.tweetId.toLong)\n            case _ => None\n          }\n      }.flatten\n\n      val timeFilteredSignals = signals.filter { signal =>\n        getTimestamp(signal).exists { signalTime =>\n          Time.now.since(signalTime) < LowSignalUserMaxSignalAge\n        }\n      }\n\n      val mostRecentTimestamp =\n        timeFilteredSignals.flatMap(getTimestamp).reduceOption((a, b) => if (a > b) a else b)\n\n      val lowSignalUser = timeFilteredSignals.size < query.params(LowSignalUserMaxSignalCount)\n      FeatureMapBuilder()\n        .add(LowSignalUserFeature, lowSignalUser)\n        .add(UserRecentEngagementTweetIdsFeature, tweetIds.toList)\n        .add(UserLastExplicitSignalTimeFeature, mostRecentTimestamp)\n        .build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/MetricCenterUserCountingFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MetricCenterUserCountingFeatureRepository\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.onboarding.relevance.features.{thriftjava => rf}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.keyvalue.KeyValueResult\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject MetricCenterUserCountingFeature\n    extends Feature[TweetCandidate, Option[rf.MCUserCountingFeatures]]\n\n@Singleton\nclass MetricCenterUserCountingFeatureHydrator @Inject() (\n  @Named(MetricCenterUserCountingFeatureRepository) client: KeyValueRepository[Seq[\n    Long\n  ], Long, rf.MCUserCountingFeatures],\n  override val statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"MetricCenterUserCounting\")\n\n  override val features: Set[Feature[_, _]] = Set(MetricCenterUserCountingFeature)\n\n  override val statScope: String = identifier.toString\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val possiblyAuthorIds = extractKeys(candidates)\n    val userIds = possiblyAuthorIds.flatten\n\n    val response: Future[KeyValueResult[Long, rf.MCUserCountingFeatures]] =\n      if (userIds.isEmpty) Future.value(KeyValueResult.empty) else client(userIds)\n\n    response.map { result =>\n      possiblyAuthorIds.map { possiblyAuthorId =>\n        val value = observedGet(key = possiblyAuthorId, keyValueResult = result)\n        FeatureMapBuilder().add(MetricCenterUserCountingFeature, value).build()\n      }\n    }\n  }\n\n  private def extractKeys(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] = {\n    candidates.map { candidate =>\n      candidate.features\n        .getTry(AuthorIdFeature)\n        .toOption\n        .flatten\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/RealGraphQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RealGraphFeatureRepository\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.repository.Repository\nimport com.twitter.timelines.real_graph.{thriftscala => rg}\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.model.UserId\nimport com.twitter.timelines.real_graph.v1.thriftscala.RealGraphEdgeFeatures\nimport com.twitter.user_session_store.{thriftscala => uss}\n\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject RealGraphFeatures extends Feature[PipelineQuery, Option[Map[UserId, RealGraphEdgeFeatures]]]\n\n@Singleton\nclass RealGraphQueryFeatureHydrator @Inject() (\n  @Named(RealGraphFeatureRepository) repository: Repository[Long, Option[uss.UserSession]])\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RealGraphFeatures\")\n\n  override val features: Set[Feature[_, _]] = Set(RealGraphFeatures)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    Stitch.callFuture {\n      repository(query.getRequiredUserId).map { userSession =>\n        val realGraphFeaturesMap = userSession.flatMap { userSession =>\n          userSession.realGraphFeatures.collect {\n            case rg.RealGraphFeatures.V1(realGraphFeatures) =>\n              val edgeFeatures = realGraphFeatures.edgeFeatures ++ realGraphFeatures.oonEdgeFeatures\n              edgeFeatures.map { edge => edge.destId -> edge }.toMap\n          }\n        }\n\n        FeatureMapBuilder().add(RealGraphFeatures, realGraphFeaturesMap).build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/RealGraphViewerAuthorFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.RealGraphViewerAuthorFeatureHydrator.getCombinedRealGraphFeatures\nimport com.twitter.home_mixer.util.MissingKeyException\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.real_graph.RealGraphEdgeFeaturesCombineAdapter\nimport com.twitter.timelines.prediction.adapters.real_graph.RealGraphFeaturesAdapter\nimport com.twitter.timelines.real_graph.v1.{thriftscala => v1}\nimport com.twitter.timelines.real_graph.{thriftscala => rg}\nimport com.twitter.util.Throw\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject RealGraphViewerAuthorDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject RealGraphViewerAuthorsDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass RealGraphViewerAuthorFeatureHydrator @Inject() ()\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RealGraphViewerAuthor\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(RealGraphViewerAuthorDataRecordFeature, RealGraphViewerAuthorsDataRecordFeature)\n\n  private val realGraphEdgeFeaturesAdapter = new RealGraphFeaturesAdapter\n  private val realGraphEdgeFeaturesCombineAdapter =\n    new RealGraphEdgeFeaturesCombineAdapter(prefix = \"authors.realgraph\")\n\n  private val MissingKeyFeatureMap = FeatureMapBuilder()\n    .add(RealGraphViewerAuthorDataRecordFeature, Throw(MissingKeyException))\n    .add(RealGraphViewerAuthorsDataRecordFeature, Throw(MissingKeyException))\n    .build()\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = OffloadFuturePools.offload {\n    val viewerId = query.getRequiredUserId\n    val realGraphFeatures = query.features\n      .flatMap(_.getOrElse(RealGraphFeatures, None))\n      .getOrElse(Map.empty[Long, v1.RealGraphEdgeFeatures])\n\n    existingFeatures.getOrElse(AuthorIdFeature, None) match {\n      case Some(authorId) =>\n        val realGraphAuthorFeatures =\n          getRealGraphViewerAuthorFeatures(viewerId, authorId, realGraphFeatures)\n        val realGraphAuthorDataRecord = realGraphEdgeFeaturesAdapter\n          .adaptToDataRecords(realGraphAuthorFeatures).asScala.headOption.getOrElse(new DataRecord)\n\n        val combinedRealGraphFeaturesDataRecord = for {\n          inReplyToAuthorId <- existingFeatures.getOrElse(InReplyToUserIdFeature, None)\n        } yield {\n          val combinedRealGraphFeatures =\n            getCombinedRealGraphFeatures(Seq(authorId, inReplyToAuthorId), realGraphFeatures)\n          realGraphEdgeFeaturesCombineAdapter\n            .adaptToDataRecords(Some(combinedRealGraphFeatures)).asScala.headOption\n            .getOrElse(new DataRecord)\n        }\n\n        FeatureMapBuilder()\n          .add(RealGraphViewerAuthorDataRecordFeature, realGraphAuthorDataRecord)\n          .add(\n            RealGraphViewerAuthorsDataRecordFeature,\n            combinedRealGraphFeaturesDataRecord.getOrElse(new DataRecord))\n          .build()\n      case _ => MissingKeyFeatureMap\n    }\n  }\n\n  private def getRealGraphViewerAuthorFeatures(\n    viewerId: Long,\n    authorId: Long,\n    realGraphEdgeFeaturesMap: Map[Long, v1.RealGraphEdgeFeatures]\n  ): rg.UserRealGraphFeatures = {\n    realGraphEdgeFeaturesMap.get(authorId) match {\n      case Some(realGraphEdgeFeatures) =>\n        rg.UserRealGraphFeatures(\n          srcId = viewerId,\n          features = rg.RealGraphFeatures.V1(\n            v1.RealGraphFeatures(edgeFeatures = Seq(realGraphEdgeFeatures))))\n      case _ =>\n        rg.UserRealGraphFeatures(\n          srcId = viewerId,\n          features = rg.RealGraphFeatures.V1(v1.RealGraphFeatures(edgeFeatures = Seq.empty)))\n    }\n  }\n}\n\nobject RealGraphViewerAuthorFeatureHydrator {\n  def getCombinedRealGraphFeatures(\n    userIds: Seq[Long],\n    realGraphEdgeFeaturesMap: Map[Long, v1.RealGraphEdgeFeatures]\n  ): rg.RealGraphFeatures = {\n    val edgeFeatures = userIds.flatMap(realGraphEdgeFeaturesMap.get)\n    rg.RealGraphFeatures.V1(v1.RealGraphFeatures(edgeFeatures = edgeFeatures))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/RealGraphViewerRelatedUsersFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MentionUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.real_graph.RealGraphEdgeFeaturesCombineAdapter\nimport com.twitter.timelines.real_graph.v1.{thriftscala => v1}\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject RealGraphViewerRelatedUsersDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass RealGraphViewerRelatedUsersFeatureHydrator @Inject() ()\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RealGraphViewerRelatedUsers\")\n\n  override val features: Set[Feature[_, _]] = Set(RealGraphViewerRelatedUsersDataRecordFeature)\n\n  private val RealGraphEdgeFeaturesCombineAdapter = new RealGraphEdgeFeaturesCombineAdapter\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = OffloadFuturePools.offload {\n    val realGraphQueryFeatures = query.features\n      .flatMap(_.getOrElse(RealGraphFeatures, None))\n      .getOrElse(Map.empty[Long, v1.RealGraphEdgeFeatures])\n\n    val allRelatedUserIds = getRelatedUserIds(existingFeatures)\n    val realGraphFeatures = RealGraphViewerAuthorFeatureHydrator.getCombinedRealGraphFeatures(\n      allRelatedUserIds,\n      realGraphQueryFeatures\n    )\n    val realGraphFeaturesDataRecord = RealGraphEdgeFeaturesCombineAdapter\n      .adaptToDataRecords(Some(realGraphFeatures)).asScala.headOption\n      .getOrElse(new DataRecord)\n\n    FeatureMapBuilder()\n      .add(RealGraphViewerRelatedUsersDataRecordFeature, realGraphFeaturesDataRecord)\n      .build()\n  }\n\n  private def getRelatedUserIds(features: FeatureMap): Seq[Long] = {\n    (CandidatesUtil.getEngagerUserIds(features) ++\n      features.getOrElse(AuthorIdFeature, None) ++\n      features.getOrElse(MentionUserIdFeature, Seq.empty) ++\n      features.getOrElse(SourceUserIdFeature, None) ++\n      features.getOrElse(DirectedAtUserIdFeature, None)).distinct\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/RealTimeInteractionGraphEdgeFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.adapters.realtime_interaction_graph.RealTimeInteractionGraphFeaturesAdapter\nimport com.twitter.timelines.prediction.features.realtime_interaction_graph.RealTimeInteractionGraphEdgeFeatures\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject RealTimeInteractionGraphEdgeFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass RealTimeInteractionGraphEdgeFeatureHydrator @Inject() ()\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RealTimeInteractionGraphEdge\")\n\n  override val features: Set[Feature[_, _]] = Set(RealTimeInteractionGraphEdgeFeature)\n\n  private val realTimeInteractionGraphFeaturesAdapter = new RealTimeInteractionGraphFeaturesAdapter\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offload {\n    val userVertex =\n      query.features.flatMap(_.getOrElse(RealTimeInteractionGraphUserVertexQueryFeature, None))\n    val realTimeInteractionGraphFeaturesMap =\n      userVertex.map(RealTimeInteractionGraphEdgeFeatures(_, Time.now))\n\n    candidates.map { candidate =>\n      val feature = candidate.features.getOrElse(AuthorIdFeature, None).flatMap { authorId =>\n        realTimeInteractionGraphFeaturesMap.flatMap(_.get(authorId))\n      }\n\n      val dataRecordFeature =\n        realTimeInteractionGraphFeaturesAdapter.adaptToDataRecords(feature).asScala.head\n\n      FeatureMapBuilder().add(RealTimeInteractionGraphEdgeFeature, dataRecordFeature).build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/RealTimeInteractionGraphUserVertexQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.RealTimeInteractionGraphUserVertexCache\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.stitch.Stitch\nimport com.twitter.wtf.real_time_interaction_graph.{thriftscala => ig}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject RealTimeInteractionGraphUserVertexQueryFeature\n    extends Feature[PipelineQuery, Option[ig.UserVertex]]\n\n@Singleton\nclass RealTimeInteractionGraphUserVertexQueryFeatureHydrator @Inject() (\n  @Named(RealTimeInteractionGraphUserVertexCache) client: ReadCache[Long, ig.UserVertex],\n  override val statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RealTimeInteractionGraphUserVertex\")\n\n  override val features: Set[Feature[_, _]] = Set(RealTimeInteractionGraphUserVertexQueryFeature)\n\n  override val statScope: String = identifier.toString\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n\n    Stitch.callFuture(\n      client.get(Seq(userId)).map { results =>\n        val feature = observedGet(key = Some(userId), keyValueResult = results)\n        FeatureMapBuilder()\n          .add(RealTimeInteractionGraphUserVertexQueryFeature, feature)\n          .build()\n      }\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/ReplyFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.TweetypieContentDataRecordFeature\nimport com.twitter.home_mixer.functional_component.feature_hydrator.WithDefaultFeatureMap\nimport com.twitter.home_mixer.model.ContentFeatures\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.content.InReplyToContentFeatureAdapter\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.earlybird.InReplyToEarlybirdAdapter\nimport com.twitter.home_mixer.util.ReplyRetweetUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.search.common.features.thriftscala.ThriftTweetFeatures\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.conversation_features.v1.thriftscala.ConversationFeatures\nimport com.twitter.timelines.conversation_features.{thriftscala => cf}\nimport com.twitter.timelines.prediction.adapters.conversation_features.ConversationFeaturesAdapter\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject InReplyToTweetHydratedEarlybirdFeature\n    extends Feature[TweetCandidate, Option[ThriftTweetFeatures]]\n\nobject ConversationDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject InReplyToEarlybirdDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject InReplyToTweetypieContentDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n/**\n * The purpose of this hydrator is to\n * 1) hydrate simple features into replies and their ancestor tweets\n * 2) keep both the normal replies and ancestor source candidates, but hydrate into the candidates\n * features useful for predicting the quality of the replies and source ancestor tweets.\n */\n@Singleton\nclass ReplyFeatureHydrator @Inject() (statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with WithDefaultFeatureMap {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"ReplyTweet\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    ConversationDataRecordFeature,\n    InReplyToTweetHydratedEarlybirdFeature,\n    InReplyToEarlybirdDataRecordFeature,\n    InReplyToTweetypieContentDataRecordFeature\n  )\n\n  private val defaultDataRecord: DataRecord = new DataRecord()\n\n  override val defaultFeatureMap = FeatureMap(\n    ConversationDataRecordFeature,\n    defaultDataRecord,\n    InReplyToTweetHydratedEarlybirdFeature,\n    None,\n    InReplyToEarlybirdDataRecordFeature,\n    defaultDataRecord,\n    InReplyToTweetypieContentDataRecordFeature,\n    defaultDataRecord\n  )\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val hydratedReplyCounter = scopedStatsReceiver.counter(\"hydratedReply\")\n  private val hydratedAncestorCounter = scopedStatsReceiver.counter(\"hydratedAncestor\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offload {\n    // only hydrate for IN candidates\n    val eligibleCandidates =\n      candidates.filter(_.features.getOrElse(FromInNetworkSourceFeature, false))\n    val replyToInReplyToTweetMap =\n      ReplyRetweetUtil.replyTweetIdToInReplyToTweetMap(eligibleCandidates)\n    val candidatesWithRepliesHydrated = candidates.map { candidate =>\n      replyToInReplyToTweetMap\n        .get(candidate.candidate.id).map { inReplyToTweet =>\n          hydratedReplyCounter.incr()\n          hydratedReplyCandidate(candidate, inReplyToTweet)\n        }.getOrElse((candidate, None, None, defaultDataRecord))\n    }\n\n    /**\n     * Update ancestor tweets with descendant replies and hydrate simple features from one of\n     * the descendants.\n     */\n    val ancestorTweetToDescendantRepliesMap =\n      ReplyRetweetUtil.ancestorTweetIdToDescendantRepliesMap(eligibleCandidates)\n    val candidatesWithRepliesAndAncestorTweetsHydrated = candidatesWithRepliesHydrated.map {\n      case (\n            maybeAncestorTweetCandidate,\n            updatedReplyConversationFeatures,\n            inReplyToTweetEarlyBirdFeature,\n            inReplyToTweetContentDataRecord\n          ) =>\n        ancestorTweetToDescendantRepliesMap\n          .get(maybeAncestorTweetCandidate.candidate.id)\n          .map { descendantReplies =>\n            hydratedAncestorCounter.incr()\n            val (ancestorTweetCandidate, updatedConversationFeatures): (\n              CandidateWithFeatures[TweetCandidate],\n              Option[ConversationFeatures]\n            ) =\n              hydrateAncestorTweetCandidate(\n                maybeAncestorTweetCandidate,\n                descendantReplies,\n                updatedReplyConversationFeatures)\n            (\n              ancestorTweetCandidate,\n              inReplyToTweetEarlyBirdFeature,\n              updatedConversationFeatures,\n              inReplyToTweetContentDataRecord)\n          }\n          .getOrElse(\n            (\n              maybeAncestorTweetCandidate,\n              inReplyToTweetEarlyBirdFeature,\n              updatedReplyConversationFeatures,\n              inReplyToTweetContentDataRecord\n            ))\n    }\n\n    candidatesWithRepliesAndAncestorTweetsHydrated.map {\n      case (\n            candidate,\n            inReplyToTweetEarlyBirdFeature,\n            updatedConversationFeatures,\n            inReplyToTweetContentDataRecord) =>\n        val conversationDataRecordFeature = updatedConversationFeatures\n          .map(f => ConversationFeaturesAdapter.adaptToDataRecord(cf.ConversationFeatures.V1(f)))\n          .getOrElse(defaultDataRecord)\n\n        val inReplyToEarlybirdDataRecord =\n          InReplyToEarlybirdAdapter\n            .adaptToDataRecords(inReplyToTweetEarlyBirdFeature).asScala.head\n        val inReplyToContentDataRecord = {\n          if (inReplyToTweetContentDataRecord.equals(defaultDataRecord)) {\n            InReplyToContentFeatureAdapter\n              .adaptToDataRecords(\n                inReplyToTweetEarlyBirdFeature.map(ContentFeatures.fromThrift)).asScala.head\n          } else\n            inReplyToTweetContentDataRecord\n        }\n\n        FeatureMap(\n          ConversationDataRecordFeature,\n          conversationDataRecordFeature,\n          InReplyToTweetHydratedEarlybirdFeature,\n          inReplyToTweetEarlyBirdFeature,\n          InReplyToEarlybirdDataRecordFeature,\n          inReplyToEarlybirdDataRecord,\n          InReplyToTweetypieContentDataRecordFeature,\n          inReplyToContentDataRecord\n        )\n      case _ => defaultFeatureMap\n    }\n  }\n\n  private def hydratedReplyCandidate(\n    replyCandidate: CandidateWithFeatures[TweetCandidate],\n    inReplyToTweetCandidate: CandidateWithFeatures[TweetCandidate]\n  ): (\n    CandidateWithFeatures[TweetCandidate],\n    Option[ConversationFeatures],\n    Option[ThriftTweetFeatures],\n    DataRecord\n  ) = {\n    val tweetedAfterInReplyToTweetInSecs =\n      (\n        originalTweetAgeFromSnowflake(inReplyToTweetCandidate),\n        originalTweetAgeFromSnowflake(replyCandidate)) match {\n        case (Some(inReplyToTweetAge), Some(replyTweetAge)) =>\n          Some((inReplyToTweetAge - replyTweetAge).inSeconds.toLong)\n        case _ => None\n      }\n\n    val existingConversationFeatures = Some(\n      replyCandidate.features\n        .getOrElse(ConversationFeature, None).getOrElse(ConversationFeatures()))\n\n    val updatedConversationFeatures = existingConversationFeatures match {\n      case Some(v1) =>\n        Some(\n          v1.copy(\n            tweetedAfterInReplyToTweetInSecs = tweetedAfterInReplyToTweetInSecs,\n            isSelfReply = Some(\n              replyCandidate.features.getOrElse(\n                AuthorIdFeature,\n                None) == inReplyToTweetCandidate.features.getOrElse(AuthorIdFeature, None))\n          )\n        )\n      case _ => None\n    }\n\n    // Note: if inReplyToTweet is a retweet, we need to read early bird feature from the merged\n    // early bird feature field from RetweetSourceTweetFeatureHydrator class.\n    // But if inReplyToTweet is a reply, we return its early bird feature directly\n    val sourceFeatures = inReplyToTweetCandidate.features\n      .getOrElse(SourceTweetEarlybirdFeature, None)\n    val inReplyToTweetThriftTweetFeaturesOpt = {\n      if (inReplyToTweetCandidate.features.getOrElse(IsRetweetFeature, false)\n        && sourceFeatures.nonEmpty) {\n        sourceFeatures\n      } else {\n        inReplyToTweetCandidate.features.getOrElse(EarlybirdFeature, None)\n      }\n    }\n    val inReplyToTweetContentDataRecord = inReplyToTweetCandidate.features\n      .getOrElse(TweetypieContentDataRecordFeature, defaultDataRecord)\n\n    (\n      replyCandidate,\n      updatedConversationFeatures,\n      inReplyToTweetThriftTweetFeaturesOpt,\n      inReplyToTweetContentDataRecord)\n  }\n\n  private def hydrateAncestorTweetCandidate(\n    ancestorTweetCandidate: CandidateWithFeatures[TweetCandidate],\n    descendantReplies: Seq[CandidateWithFeatures[TweetCandidate]],\n    updatedReplyConversationFeatures: Option[ConversationFeatures]\n  ): (CandidateWithFeatures[TweetCandidate], Option[ConversationFeatures]) = {\n    // Ancestor could be a reply. For example, in thread: tweetA -> tweetB -> tweetC,\n    // tweetB is a reply and ancestor at the same time. Hence, tweetB's conversation feature\n    // will be updated by hydratedReplyCandidate and hydrateAncestorTweetCandidate functions.\n    val existingConversationFeatures =\n      if (updatedReplyConversationFeatures.nonEmpty)\n        updatedReplyConversationFeatures\n      else\n        Some(\n          ancestorTweetCandidate.features\n            .getOrElse(ConversationFeature, None).getOrElse(ConversationFeatures()))\n\n    val updatedConversationFeatures = existingConversationFeatures match {\n      case Some(v1) =>\n        Some(\n          v1.copy(\n            hasDescendantReplyCandidate = Some(true),\n            hasInNetworkDescendantReply =\n              Some(descendantReplies.exists(_.features.getOrElse(InNetworkFeature, false)))\n          ))\n      case _ => None\n    }\n    (ancestorTweetCandidate, updatedConversationFeatures)\n  }\n\n  private def originalTweetAgeFromSnowflake(\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Option[Duration] = {\n    SnowflakeId\n      .timeFromIdOpt(\n        candidate.features\n          .getOrElse(SourceTweetIdFeature, None).getOrElse(candidate.candidate.id))\n      .map(Time.now - _)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/RequestTimeQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.HomeFeatures.FollowingLastNonPollingTimeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.LastNonPollingTimeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.NonPollingTimesFeature\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.util.FDsl._\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.features.time_features.AccountAgeInterval\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.ACCOUNT_AGE_INTERVAL\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.IS_12_MONTH_NEW_USER\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.IS_30_DAY_NEW_USER\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.TIME_BETWEEN_NON_POLLING_REQUESTS_AVG\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.TIME_SINCE_LAST_NON_POLLING_REQUEST\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.TIME_SINCE_VIEWER_ACCOUNT_CREATION_SECS\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures.USER_ID_IS_SNOWFLAKE_ID\nimport com.twitter.user_session_store.ReadRequest\nimport com.twitter.user_session_store.ReadWriteUserSessionStore\nimport com.twitter.user_session_store.UserSessionDataset\nimport com.twitter.user_session_store.UserSessionDataset.UserSessionDataset\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject RequestTimeDataRecordFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\ncase class RequestTimeQueryFeatureHydrator @Inject() (\n  userSessionStore: ReadWriteUserSessionStore)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"RequestTime\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    FollowingLastNonPollingTimeFeature,\n    LastNonPollingTimeFeature,\n    NonPollingTimesFeature,\n    RequestTimeDataRecordFeature\n  )\n\n  private val datasets: Set[UserSessionDataset] = Set(UserSessionDataset.NonPollingTimes)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    userSessionStore\n      .read(ReadRequest(query.getRequiredUserId, datasets))\n      .map { userSession =>\n        val nonPollingTimestamps = userSession.flatMap(_.nonPollingTimestamps)\n\n        val lastNonPollingTime = nonPollingTimestamps\n          .flatMap(_.nonPollingTimestampsMs.headOption)\n          .map(Time.fromMilliseconds)\n\n        val followingLastNonPollingTime = nonPollingTimestamps\n          .flatMap(_.mostRecentHomeLatestNonPollingTimestampMs)\n          .map(Time.fromMilliseconds)\n\n        val nonPollingTimes = nonPollingTimestamps\n          .map(_.nonPollingTimestampsMs)\n          .getOrElse(Seq.empty)\n\n        val requestTimeDataRecord = getRequestTimeDataRecord(query, nonPollingTimes)\n\n        FeatureMapBuilder()\n          .add(FollowingLastNonPollingTimeFeature, followingLastNonPollingTime)\n          .add(LastNonPollingTimeFeature, lastNonPollingTime)\n          .add(NonPollingTimesFeature, nonPollingTimes)\n          .add(RequestTimeDataRecordFeature, requestTimeDataRecord)\n          .build()\n      }\n  }\n\n  def getRequestTimeDataRecord(query: PipelineQuery, nonPollingTimes: Seq[Long]): DataRecord = {\n    val requestTimeMs = query.queryTime.inMillis\n    val accountAge = SnowflakeId.timeFromIdOpt(query.getRequiredUserId)\n    val timeSinceAccountCreation = accountAge.map(query.queryTime.since)\n    val timeSinceEarliestNonPollingRequest =\n      nonPollingTimes.lastOption.map(requestTimeMs - _)\n    val timeSinceLastNonPollingRequest =\n      nonPollingTimes.headOption.map(requestTimeMs - _)\n\n    new DataRecord()\n      .setFeatureValue(USER_ID_IS_SNOWFLAKE_ID, accountAge.isDefined)\n      .setFeatureValue(\n        IS_30_DAY_NEW_USER,\n        timeSinceAccountCreation.map(_ < 30.days).getOrElse(false)\n      )\n      .setFeatureValue(\n        IS_12_MONTH_NEW_USER,\n        timeSinceAccountCreation.map(_ < 365.days).getOrElse(false)\n      )\n      .setFeatureValueFromOption(\n        ACCOUNT_AGE_INTERVAL,\n        timeSinceAccountCreation.flatMap(AccountAgeInterval.fromDuration).map(_.id.toLong)\n      )\n      .setFeatureValueFromOption(\n        TIME_SINCE_VIEWER_ACCOUNT_CREATION_SECS,\n        timeSinceAccountCreation.map(_.inSeconds.toDouble)\n      )\n      .setFeatureValueFromOption(\n        TIME_BETWEEN_NON_POLLING_REQUESTS_AVG,\n        timeSinceEarliestNonPollingRequest.map(_.toDouble / math.max(1.0, nonPollingTimes.size))\n      )\n      .setFeatureValueFromOption(\n        TIME_SINCE_LAST_NON_POLLING_REQUEST,\n        timeSinceLastNonPollingRequest.map(_.toDouble)\n      )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/RetweetSourceTweetFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.product_mixer.component_library.candidate_source.timeline_ranker.TimelineRankerInNetworkSourceTweetsByTweetIdMapFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.search.common.features.thriftscala.ThriftTweetFeatures\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelineranker.thriftscala.CandidateTweet\n\nobject SourceTweetEarlybirdFeature extends Feature[TweetCandidate, Option[ThriftTweetFeatures]]\n\n/**\n * Feature Hydrator that bulk hydrates source tweets' features to retweet candidates\n */\nobject RetweetSourceTweetFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"RetweetSourceTweet\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    SourceTweetEarlybirdFeature,\n  )\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(SourceTweetEarlybirdFeature, None)\n    .build()\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val sourceTweetsByTweetId: Option[Map[Long, CandidateTweet]] = {\n      query.features.map(\n        _.getOrElse(\n          TimelineRankerInNetworkSourceTweetsByTweetIdMapFeature,\n          Map.empty[Long, CandidateTweet]))\n    }\n\n    /**\n     * Return DefaultFeatureMap (no-op to candidate) when it is unfeasible to hydrate the\n     * source tweet's feature to the current candidate: early bird does not return source\n     * tweets info / candidate is not a retweet / sourceTweetId is not found\n     */\n    Stitch.value {\n      if (sourceTweetsByTweetId.exists(_.nonEmpty)) {\n        candidates.map { candidate =>\n          val candidateIsRetweet = candidate.features.getOrElse(IsRetweetFeature, false)\n          val sourceTweetId = candidate.features.getOrElse(SourceTweetIdFeature, None)\n          if (!candidateIsRetweet || sourceTweetId.isEmpty) {\n            DefaultFeatureMap\n          } else {\n            val sourceTweet = sourceTweetsByTweetId.flatMap(_.get(sourceTweetId.get))\n            if (sourceTweet.nonEmpty) {\n              val source = sourceTweet.get\n              FeatureMapBuilder()\n                .add(SourceTweetEarlybirdFeature, source.features)\n                .build()\n            } else {\n              DefaultFeatureMap\n            }\n          }\n        }\n      } else {\n        candidates.map(_ => DefaultFeatureMap)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/SGSMutuallyFollowedUserHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.socialgraph.{thriftscala => sg}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.{SocialGraph => SocialGraphStitchClient}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject SGSMutuallyFollowedUsersFeature extends Feature[PipelineQuery, Seq[Long]]\n\n@Singleton\ncase class SGSMutuallyFollowedUserHydrator @Inject() (\n  socialGraphStitchClient: SocialGraphStitchClient)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"SGSMutuallyFollowedUsers\")\n\n  override val features: Set[Feature[_, _]] = Set(SGSMutuallyFollowedUsersFeature)\n\n  private val SocialGraphLimit = 14999\n  private val MaxFollowTargets = 1500\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(SGSMutuallyFollowedUsersFeature, Seq.empty)\n    .build()\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n\n    val sgsFollowedUserIds =\n      query.features.map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty)).toSeq.flatten\n\n    if (sgsFollowedUserIds.nonEmpty) {\n      val mutuallyFollowedRequest = sg.IdsRequest(\n        relationships = Seq(\n          sg.SrcRelationship(\n            query.getRequiredUserId,\n            sg.RelationshipType.FollowedBy,\n            hasRelationship = true,\n            targets = Some(sgsFollowedUserIds.take(MaxFollowTargets))\n          ),\n        ),\n        pageRequest = Some(sg.PageRequest(count = Some(SocialGraphLimit)))\n      )\n      socialGraphStitchClient.ids(mutuallyFollowedRequest).map(_.ids).map { mutuallyFollowedUsers =>\n        FeatureMapBuilder()\n          .add(SGSMutuallyFollowedUsersFeature, mutuallyFollowedUsers)\n          .build()\n      }\n    } else Stitch.value(DefaultFeatureMap)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/SemanticCoreFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.RealTimeEntityRealGraphFeatures\nimport com.twitter.home_mixer.functional_component.feature_hydrator.WithDefaultFeatureMap\nimport com.twitter.home_mixer.model.HomeFeatures.SemanticAnnotationIdsFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableRealTimeEntityRealGraphFeaturesParam\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.util.SRichDataRecord\nimport com.twitter.ml.api.{Feature => mlFeature}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.recos.entities.{thriftscala => ent}\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.features.semantic_core_features.SemanticCoreFeatures\nimport com.twitter.wtf.entity_real_graph.{thriftscala => erg}\n\nobject SemanticCoreDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject SemanticCoreFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery]\n    with WithDefaultFeatureMap {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"SemanticCore\")\n\n  override val features: Set[Feature[_, _]] = Set(SemanticCoreDataRecordFeature)\n\n  override val defaultFeatureMap: FeatureMap =\n    FeatureMap(SemanticCoreDataRecordFeature, SemanticCoreDataRecordFeature.defaultValue)\n\n  override def onlyIf(query: PipelineQuery): Boolean =\n    query.params(EnableRealTimeEntityRealGraphFeaturesParam)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offload {\n    val viewerErgFeatures =\n      query.features.flatMap(_.getOrElse(RealTimeEntityRealGraphFeatures, None))\n    candidates.map { candidate =>\n      val candidateEntities = candidate.features.getOrElse(SemanticAnnotationIdsFeature, Seq.empty)\n\n      val engagements: Seq[Map[erg.EngagementType, erg.Features]] = {\n        viewerErgFeatures\n          .map { engagementsByEntity =>\n            engagementsByEntity.collect {\n              case (ent.Entity.SemanticCore(sc), featuresByEngagementType)\n                  if candidateEntities.contains(sc.entityId) =>\n                featuresByEngagementType\n            }.toSeq\n          }.getOrElse(Seq.empty)\n      }\n\n      val rawFeaturesByEngagementType: Map[erg.EngagementType, Seq[erg.Features]] =\n        SemanticCoreFeatures.engagementTypes.map { engagementType =>\n          val features = engagements.flatMap(_.get(engagementType))\n          engagementType -> features\n        }.toMap\n\n      val decayedScoreFeatureValues: Map[mlFeature[_], List[Double]] =\n        SemanticCoreFeatures.decayedScoreFeatures.mapValues { engagementType =>\n          val valuesOpt: Option[Seq[Double]] =\n            rawFeaturesByEngagementType.get(engagementType).map(_.map(_.scores.oneDayHalfLife))\n          valuesOpt match {\n            case Some(values) => values.toList\n            case None => List.empty[Double]\n          }\n        }\n      val normalizedCountFeatureValues: Map[mlFeature[_], List[Double]] =\n        SemanticCoreFeatures.normalizedCountFeatures.mapValues { engagementType =>\n          val valuesOpt: Option[Seq[Double]] =\n            rawFeaturesByEngagementType.get(engagementType).map(_.flatMap(_.normalizedCount))\n          valuesOpt match {\n            case Some(values) => values.toList\n            case None => List.empty[Double]\n          }\n        }\n\n      buildFeatureMap(decayedScoreFeatureValues ++ normalizedCountFeatureValues)\n    }\n  }\n\n  private def buildFeatureMap(\n    featureValuesMap: Map[mlFeature[_], List[Double]]\n  ): FeatureMap = {\n    val featureContext = new FeatureContext(SemanticCoreFeatures.outputFeaturesPostMerge.toSeq: _*)\n    val richRecord = new SRichDataRecord(new DataRecord, featureContext)\n\n    SemanticCoreFeatures.hydrateCountFeatures(\n      richRecord = richRecord,\n      features = SemanticCoreFeatures.precomputedCountFeatures,\n      featureValuesMap = featureValuesMap\n    )\n\n    FeatureMap(SemanticCoreDataRecordFeature, richRecord.getRecord)\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/SimClustersEngagementSimilarityFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.clients.strato.twistly.SimClustersRecentEngagementSimilarityClient\nimport com.twitter.timelines.configapi.decider.BooleanDeciderParam\nimport com.twitter.timelines.prediction.adapters.twistly.SimClustersRecentEngagementSimilarityFeaturesAdapter\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject SimClustersEngagementSimilarityFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass SimClustersEngagementSimilarityFeatureHydrator @Inject() (\n  simClustersEngagementSimilarityClient: SimClustersRecentEngagementSimilarityClient)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"SimClustersEngagementSimilarity\")\n\n  override val features: Set[Feature[_, _]] = Set(SimClustersEngagementSimilarityFeature)\n\n  private val simClustersRecentEngagementSimilarityFeaturesAdapter =\n    new SimClustersRecentEngagementSimilarityFeaturesAdapter\n\n  override def onlyIf(query: PipelineQuery): Boolean = {\n    val param: BooleanDeciderParam =\n      ScoredTweetsParam.EnableSimClustersSimilarityFeatureHydrationDeciderParam\n    query.params.apply(param)\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val tweetToCandidates = candidates.map(candidate => candidate.candidate.id -> candidate).toMap\n    val tweetIds = tweetToCandidates.keySet.toSeq\n    val userId = query.getRequiredUserId\n    val userTweetEdges = tweetIds.map(tweetId => (userId, tweetId))\n    simClustersEngagementSimilarityClient\n      .getSimClustersRecentEngagementSimilarityScores(userTweetEdges).map {\n        simClustersRecentEngagementSimilarityScoresMap =>\n          candidates.map { candidate =>\n            val similarityFeatureOpt = simClustersRecentEngagementSimilarityScoresMap\n              .get(userId -> candidate.candidate.id).flatten\n            val dataRecordOpt = similarityFeatureOpt.map { similarityFeature =>\n              simClustersRecentEngagementSimilarityFeaturesAdapter\n                .adaptToDataRecords(similarityFeature)\n                .get(0)\n            }\n            FeatureMapBuilder()\n              .add(SimClustersEngagementSimilarityFeature, dataRecordOpt.getOrElse(new DataRecord))\n              .build()\n          }\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/SimClustersUserTweetScoresHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.dal.personal_data.{thriftjava => pd}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordOptionalFeature\nimport com.twitter.product_mixer.core.feature.datarecord.DoubleDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.Fetch\nimport com.twitter.strato.generated.client.ml.featureStore.SimClustersUserInterestedInTweetEmbeddingDotProduct20M145K2020OnUserTweetEdgeClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject SimClustersUserInterestedInTweetEmbeddingDataRecordFeature\n    extends DataRecordOptionalFeature[TweetCandidate, Double]\n    with DoubleDataRecordCompatible {\n  override val featureName: String =\n    \"user-tweet.recommendations.sim_clusters_scores.user_interested_in_tweet_embedding_dot_product_20m_145k_2020\"\n  override val personalDataTypes: Set[pd.PersonalDataType] =\n    Set(pd.PersonalDataType.InferredInterests)\n}\n\n@Singleton\nclass SimClustersUserTweetScoresHydrator @Inject() (\n  simClustersColumn: SimClustersUserInterestedInTweetEmbeddingDotProduct20M145K2020OnUserTweetEdgeClientColumn,\n  statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"SimClustersUserTweetScores\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    SimClustersUserInterestedInTweetEmbeddingDataRecordFeature)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyLossCounter = scopedStatsReceiver.counter(\"key/loss\")\n  private val keyFailureCounter = scopedStatsReceiver.counter(\"key/failure\")\n  private val keySkipCounter = scopedStatsReceiver.counter(\"key/skip\")\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(SimClustersUserInterestedInTweetEmbeddingDataRecordFeature, None)\n    .build()\n  private val MinFavToHydrate = 9\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    Stitch.run {\n      Stitch.collect {\n        candidates.map { candidate =>\n          val ebFeatures = candidate.features.getOrElse(EarlybirdFeature, None)\n          val favCount = ebFeatures.flatMap(_.favCountV2).getOrElse(0)\n          \n          if (ebFeatures.isEmpty || favCount >= MinFavToHydrate) {\n            simClustersColumn.fetcher\n              .fetch((query.getRequiredUserId, candidate.candidate.id), Unit)\n              .map {\n                case Fetch.Result(response, _) =>\n                  if (response.nonEmpty) keyFoundCounter.incr() else keyLossCounter.incr()\n                  FeatureMapBuilder()\n                    .add(SimClustersUserInterestedInTweetEmbeddingDataRecordFeature, response)\n                    .build()\n                case _ =>\n                  keyFailureCounter.incr()\n                  DefaultFeatureMap\n              }\n          } else {\n            keySkipCounter.incr()\n            Stitch.value(DefaultFeatureMap)\n          }\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/TSPInferredTopicFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.contentrecommender.{thriftscala => cr}\nimport com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TSPMetricTagFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.inferred_topic.InferredTopicAdapter\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BasicTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContextFunctionalityType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.clients.strato.topics.TopicSocialProofClient\nimport com.twitter.timelineservice.suggests.logging.candidate_tweet_source_id.{thriftscala => sid}\nimport com.twitter.topiclisting.TopicListingViewerContext\nimport com.twitter.tsp.{thriftscala => tsp}\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TSPInferredTopicFeature extends Feature[TweetCandidate, Map[Long, Double]]\nobject TSPInferredTopicDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TSPInferredTopicFeatureHydrator @Inject() (\n  topicSocialProofClient: TopicSocialProofClient)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TSPInferredTopic\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    TSPInferredTopicFeature,\n    TSPInferredTopicDataRecordFeature,\n    TopicIdSocialContextFeature,\n    TopicContextFunctionalityTypeFeature\n  )\n\n  private val topK = 3\n\n  private val SourcesToSetSocialProof: Set[sid.CandidateTweetSourceId] =\n    Set(sid.CandidateTweetSourceId.Simcluster)\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(TSPInferredTopicFeature, Map.empty[Long, Double])\n    .add(TSPInferredTopicDataRecordFeature, new DataRecord())\n    .add(TopicIdSocialContextFeature, None)\n    .add(TopicContextFunctionalityTypeFeature, None)\n    .build()\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val tags = candidates.collect {\n      case candidate if candidate.features.getTry(TSPMetricTagFeature).isReturn =>\n        candidate.candidate.id -> candidate.features\n          .getOrElse(TSPMetricTagFeature, Set.empty[tsp.MetricTag])\n    }.toMap\n\n    val topicSocialProofRequest = tsp.TopicSocialProofRequest(\n      userId = query.getRequiredUserId,\n      tweetIds = candidates.map(_.candidate.id).toSet,\n      displayLocation = cr.DisplayLocation.HomeTimeline,\n      topicListingSetting = tsp.TopicListingSetting.Followable,\n      context = TopicListingViewerContext.fromClientContext(query.clientContext).toThrift,\n      bypassModes = None,\n      // Only TweetMixer source has this data. Convert the TweetMixer metric tag to tsp metric tag.\n      tags = if (tags.isEmpty) None else Some(tags)\n    )\n\n    topicSocialProofClient\n      .getTopicTweetSocialProofResponse(topicSocialProofRequest)\n      .map {\n        case Some(response) =>\n          handleResponse(response, candidates)\n        case _ => candidates.map { _ => DefaultFeatureMap }\n      }\n  }\n\n  private def handleResponse(\n    response: tsp.TopicSocialProofResponse,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[FeatureMap] = {\n    candidates.map { candidate =>\n      val topicWithScores = response.socialProofs.getOrElse(candidate.candidate.id, Seq.empty)\n      if (topicWithScores.nonEmpty) {\n        val (socialProofId, socialProofFunctionalityType) =\n          if (candidate.features\n              .getOrElse(CandidateSourceIdFeature, None)\n              .exists(SourcesToSetSocialProof.contains)) {\n            getSocialProof(topicWithScores)\n          } else (None, None)\n\n        val inferredTopicFeatures =\n          topicWithScores.sortBy(-_.score).take(topK).map(a => (a.topicId, a.score)).toMap\n\n        val inferredTopicDataRecord =\n          InferredTopicAdapter.adaptToDataRecords(inferredTopicFeatures).asScala.head\n\n        FeatureMapBuilder()\n          .add(TSPInferredTopicFeature, inferredTopicFeatures)\n          .add(TSPInferredTopicDataRecordFeature, inferredTopicDataRecord)\n          .add(TopicIdSocialContextFeature, socialProofId)\n          .add(TopicContextFunctionalityTypeFeature, socialProofFunctionalityType)\n          .build()\n      } else DefaultFeatureMap\n    }\n  }\n\n  private def getSocialProof(\n    topicWithScores: Seq[tsp.TopicWithScore]\n  ): (Option[Long], Option[TopicContextFunctionalityType]) = {\n    val followingTopicId = topicWithScores.collectFirst {\n      case tsp.TopicWithScore(topicId, _, _, Some(tsp.TopicFollowType.Following)) => topicId\n    }\n\n    if (followingTopicId.nonEmpty) {\n      return (followingTopicId, Some(BasicTopicContextFunctionalityType))\n    }\n\n    val implicitFollowingId = topicWithScores.collectFirst {\n      case tsp.TopicWithScore(topicId, _, _, Some(tsp.TopicFollowType.ImplicitFollow)) =>\n        topicId\n    }\n\n    if (implicitFollowingId.nonEmpty) {\n      return (implicitFollowingId, Some(RecommendationTopicContextFunctionalityType))\n    }\n\n    (None, None)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/TweetMetaDataFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.constant.SharedFeatures\nimport com.twitter.ml.api.util.DataRecordConverters._\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport java.lang.{Long => JLong}\n\nobject TweetMetaDataDataRecord\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject TweetMetaDataFeatureHydrator\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TweetMetaData\")\n\n  override def features: Set[Feature[_, _]] = Set(TweetMetaDataDataRecord)\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = OffloadFuturePools.offload {\n    val richDataRecord = new RichDataRecord()\n    setFeatures(richDataRecord, candidate, existingFeatures)\n    FeatureMapBuilder().add(TweetMetaDataDataRecord, richDataRecord.getRecord).build()\n  }\n\n  private def setFeatures(\n    richDataRecord: RichDataRecord,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Unit = {\n    richDataRecord.setFeatureValue[JLong](SharedFeatures.TWEET_ID, candidate.id)\n\n    richDataRecord.setFeatureValueFromOption(\n      TimelinesSharedFeatures.ORIGINAL_AUTHOR_ID,\n      CandidatesUtil.getOriginalAuthorId(existingFeatures))\n\n    richDataRecord.setFeatureValueFromOption(\n      TimelinesSharedFeatures.CANDIDATE_TWEET_SOURCE_ID,\n      existingFeatures.getOrElse(CandidateSourceIdFeature, None).map(_.value.toLong))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/TweetTimeFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.NonPollingTimesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.util.FDsl._\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.features.time_features.TimeDataRecordFeatures._\nimport com.twitter.util.Duration\nimport scala.collection.Searching._\n\nobject TweetTimeDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject TweetTimeFeatureHydrator extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TweetTime\")\n\n  override val features: Set[Feature[_, _]] = Set(TweetTimeDataRecordFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    val tweetFeatures = existingFeatures.getOrElse(EarlybirdFeature, None)\n    val timeSinceTweetCreation = SnowflakeId.timeFromIdOpt(candidate.id).map(query.queryTime.since)\n    val timeSinceTweetCreationMs = timeSinceTweetCreation.map(_.inMillis)\n\n    val timeSinceSourceTweetCreationOpt = existingFeatures\n      .getOrElse(SourceTweetIdFeature, None)\n      .flatMap { sourceTweetId =>\n        SnowflakeId.timeFromIdOpt(sourceTweetId).map(query.queryTime.since)\n      }.orElse(timeSinceTweetCreation)\n\n    val lastFavSinceCreationHrs =\n      tweetFeatures.flatMap(_.lastFavSinceCreationHrs).map(_.toDouble)\n    val lastRetweetSinceCreationHrs =\n      tweetFeatures.flatMap(_.lastRetweetSinceCreationHrs).map(_.toDouble)\n    val lastReplySinceCreationHrs =\n      tweetFeatures.flatMap(_.lastReplySinceCreationHrs).map(_.toDouble)\n    val lastQuoteSinceCreationHrs =\n      tweetFeatures.flatMap(_.lastQuoteSinceCreationHrs).map(_.toDouble)\n    val timeSinceLastFavoriteHrs =\n      getTimeSinceLastEngagementHrs(lastFavSinceCreationHrs, timeSinceSourceTweetCreationOpt)\n    val timeSinceLastRetweetHrs =\n      getTimeSinceLastEngagementHrs(lastRetweetSinceCreationHrs, timeSinceSourceTweetCreationOpt)\n    val timeSinceLastReplyHrs =\n      getTimeSinceLastEngagementHrs(lastReplySinceCreationHrs, timeSinceSourceTweetCreationOpt)\n    val timeSinceLastQuoteHrs =\n      getTimeSinceLastEngagementHrs(lastQuoteSinceCreationHrs, timeSinceSourceTweetCreationOpt)\n\n    val nonPollingTimestampsMs = query.features.get.getOrElse(NonPollingTimesFeature, Seq.empty)\n    val timeSinceLastNonPollingRequest =\n      nonPollingTimestampsMs.headOption.map(query.queryTime.inMillis - _)\n\n    val nonPollingRequestsSinceTweetCreation =\n      if (nonPollingTimestampsMs.nonEmpty && timeSinceTweetCreationMs.isDefined) {\n        nonPollingTimestampsMs\n          .search(timeSinceTweetCreationMs.get)(Ordering[Long].reverse)\n          .insertionPoint\n      } else 0.0\n\n    val tweetAgeRatio =\n      if (timeSinceTweetCreationMs.exists(_ > 0.0) && timeSinceLastNonPollingRequest.isDefined) {\n        timeSinceLastNonPollingRequest.get / timeSinceTweetCreationMs.get.toDouble\n      } else 0.0\n\n    val dataRecord = new DataRecord()\n      .setFeatureValue(IS_TWEET_RECYCLED, false)\n      .setFeatureValue(TWEET_AGE_RATIO, tweetAgeRatio)\n      .setFeatureValueFromOption(\n        TIME_SINCE_TWEET_CREATION,\n        timeSinceTweetCreationMs.map(_.toDouble)\n      )\n      .setFeatureValue(\n        NON_POLLING_REQUESTS_SINCE_TWEET_CREATION,\n        nonPollingRequestsSinceTweetCreation\n      )\n      .setFeatureValueFromOption(LAST_FAVORITE_SINCE_CREATION_HRS, lastFavSinceCreationHrs)\n      .setFeatureValueFromOption(LAST_RETWEET_SINCE_CREATION_HRS, lastRetweetSinceCreationHrs)\n      .setFeatureValueFromOption(LAST_REPLY_SINCE_CREATION_HRS, lastReplySinceCreationHrs)\n      .setFeatureValueFromOption(LAST_QUOTE_SINCE_CREATION_HRS, lastQuoteSinceCreationHrs)\n      .setFeatureValueFromOption(TIME_SINCE_LAST_FAVORITE_HRS, timeSinceLastFavoriteHrs)\n      .setFeatureValueFromOption(TIME_SINCE_LAST_RETWEET_HRS, timeSinceLastRetweetHrs)\n      .setFeatureValueFromOption(TIME_SINCE_LAST_REPLY_HRS, timeSinceLastReplyHrs)\n      .setFeatureValueFromOption(TIME_SINCE_LAST_QUOTE_HRS, timeSinceLastQuoteHrs)\n\n    Stitch.value(FeatureMapBuilder().add(TweetTimeDataRecordFeature, dataRecord).build())\n  }\n\n  private def getTimeSinceLastEngagementHrs(\n    lastEngagementTimeSinceCreationHrsOpt: Option[Double],\n    timeSinceTweetCreation: Option[Duration]\n  ): Option[Double] = lastEngagementTimeSinceCreationHrsOpt.flatMap { lastEngagementTimeHrs =>\n    timeSinceTweetCreation.map(_.inHours - lastEngagementTimeHrs)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/TweetypieContentFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.escherbird.{thriftscala => esb}\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.MediaUnderstandingAnnotationIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetypieContentRepository\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.content.ContentFeatureAdapter\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.home_mixer.util.tweetypie.content.FeatureExtractionHelper\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.keyvalue.KeyValueResult\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.prediction.common.util.MediaUnderstandingAnnotations\nimport com.twitter.tweetypie.{thriftscala => tp}\nimport com.twitter.util.Future\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TweetypieContentDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TweetypieContentFeatureHydrator @Inject() (\n  @Named(TweetypieContentRepository) client: KeyValueRepository[Seq[Long], Long, tp.Tweet],\n  override val statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TweetypieContent\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    MediaUnderstandingAnnotationIdsFeature,\n    TweetypieContentDataRecordFeature\n  )\n\n  override val statScope: String = identifier.toString\n\n  private val bulkRequestLatencyStat =\n    statsReceiver.scope(statScope).scope(\"bulkRequest\").stat(\"latency_ms\")\n  private val postTransformerLatencyStat =\n    statsReceiver.scope(statScope).scope(\"postTransformer\").stat(\"latency_ms\")\n  private val bulkPostTransformerLatencyStat =\n    statsReceiver.scope(statScope).scope(\"bulkPostTransformer\").stat(\"latency_ms\")\n\n  private val DefaultDataRecord: DataRecord = new DataRecord()\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val tweetIdsToHydrate = candidates.map(getCandidateOriginalTweetId).distinct\n\n    val response: Future[KeyValueResult[Long, tp.Tweet]] = Stat.timeFuture(bulkRequestLatencyStat) {\n      if (tweetIdsToHydrate.isEmpty) Future.value(KeyValueResult.empty)\n      else client(tweetIdsToHydrate)\n    }\n\n    response.flatMap { result =>\n      Stat.timeFuture(bulkPostTransformerLatencyStat) {\n        OffloadFuturePools\n          .parallelize[CandidateWithFeatures[TweetCandidate], Try[(Seq[Long], DataRecord)]](\n            candidates,\n            parTransformer(result, _),\n            parallelism = 32,\n            default = Return((Seq.empty, DefaultDataRecord))\n          ).map {\n            _.map {\n              case Return(result) =>\n                FeatureMapBuilder()\n                  .add(MediaUnderstandingAnnotationIdsFeature, result._1)\n                  .add(TweetypieContentDataRecordFeature, result._2)\n                  .build()\n              case Throw(e) =>\n                FeatureMapBuilder()\n                  .add(MediaUnderstandingAnnotationIdsFeature, Throw(e))\n                  .add(TweetypieContentDataRecordFeature, Throw(e))\n                  .build()\n            }\n          }\n      }\n    }\n  }\n\n  private def parTransformer(\n    result: KeyValueResult[Long, tp.Tweet],\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Try[(Seq[Long], DataRecord)] = {\n    val originalTweetId = Some(getCandidateOriginalTweetId(candidate))\n    val value = observedGet(key = originalTweetId, keyValueResult = result)\n    Stat.time(postTransformerLatencyStat)(postTransformer(value))\n  }\n\n  private def postTransformer(\n    result: Try[Option[tp.Tweet]]\n  ): Try[(Seq[Long], DataRecord)] = {\n    result.map { tweet =>\n      val transformedValue = tweet.map(FeatureExtractionHelper.extractFeatures)\n      val semanticAnnotations = transformedValue\n        .flatMap { contentFeatures =>\n          contentFeatures.semanticCoreAnnotations.map {\n            getNonSensitiveHighRecallMediaUnderstandingAnnotationEntityIds\n          }\n        }.getOrElse(Seq.empty)\n      val dataRecord = ContentFeatureAdapter.adaptToDataRecords(transformedValue).asScala.head\n      (semanticAnnotations, dataRecord)\n    }\n  }\n\n  private def getCandidateOriginalTweetId(\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Long = {\n    candidate.features\n      .getOrElse(SourceTweetIdFeature, None).getOrElse(candidate.candidate.id)\n  }\n\n  private def getNonSensitiveHighRecallMediaUnderstandingAnnotationEntityIds(\n    semanticCoreAnnotations: Seq[esb.TweetEntityAnnotation]\n  ): Seq[Long] =\n    semanticCoreAnnotations\n      .filter(MediaUnderstandingAnnotations.isEligibleNonSensitiveHighRecallMUAnnotation)\n      .map(_.entityId)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/TweetypieStaticEntitiesFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.google.inject.name.Named\nimport com.twitter.conversions.DurationOps.RichDuration\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ExclusiveConversationAuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasImageFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MentionScreenNameFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MentionUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetypieStaticEntitiesCache\nimport com.twitter.home_mixer.util.tweetypie.RequestFields\nimport com.twitter.home_mixer.util.tweetypie.content.TweetMediaFeaturesExtractor\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.TtlCache\nimport com.twitter.spam.rtf.{thriftscala => sp}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}\nimport com.twitter.tweetypie.{thriftscala => tp}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetypieStaticEntitiesFeatureHydrator @Inject() (\n  tweetypieStitchClient: TweetypieStitchClient,\n  @Named(TweetypieStaticEntitiesCache) cacheClient: TtlCache[Long, tp.Tweet])\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetypieStaticEntities\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    AuthorIdFeature,\n    DirectedAtUserIdFeature,\n    ExclusiveConversationAuthorIdFeature,\n    HasImageFeature,\n    HasVideoFeature,\n    InReplyToTweetIdFeature,\n    InReplyToUserIdFeature,\n    IsRetweetFeature,\n    MentionScreenNameFeature,\n    MentionUserIdFeature,\n    QuotedTweetIdFeature,\n    QuotedUserIdFeature,\n    SourceTweetIdFeature,\n    SourceUserIdFeature\n  )\n\n  private val CacheTTL = 24.hours\n\n  private val DefaultFeatureMap = FeatureMapBuilder()\n    .add(AuthorIdFeature, None)\n    .add(DirectedAtUserIdFeature, None)\n    .add(ExclusiveConversationAuthorIdFeature, None)\n    .add(HasImageFeature, false)\n    .add(HasVideoFeature, false)\n    .add(InReplyToTweetIdFeature, None)\n    .add(InReplyToUserIdFeature, None)\n    .add(IsRetweetFeature, false)\n    .add(MentionScreenNameFeature, Seq.empty)\n    .add(MentionUserIdFeature, Seq.empty)\n    .add(QuotedTweetIdFeature, None)\n    .add(QuotedUserIdFeature, None)\n    .add(SourceTweetIdFeature, None)\n    .add(SourceUserIdFeature, None)\n    .build()\n\n  /**\n   * Steps:\n   *  1. query cache with all candidates\n   *  2. create a cached feature map\n   *  3. iterate candidates to hydrate features\n   *  3.a transform cached candidates\n   *  3.b hydrate non-cached candidates from Tweetypie and write to cache\n   */\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val tweetIds = candidates.map(_.candidate.id)\n    val cachedTweetsMapFu = cacheClient\n      .get(tweetIds)\n      .map(_.found)\n\n    Stitch.callFuture(cachedTweetsMapFu).flatMap { cachedTweets =>\n      Stitch.collect {\n        candidates.map { candidate =>\n          if (cachedTweets.contains(candidate.candidate.id))\n            Stitch.value(createFeatureMap(cachedTweets(candidate.candidate.id)))\n          else readFromTweetypie(query, candidate)\n        }\n      }\n    }\n  }\n\n  private def createFeatureMap(tweet: tp.Tweet): FeatureMap = {\n    val coreData = tweet.coreData\n    val quotedTweet = tweet.quotedTweet\n    val mentions = tweet.mentions.getOrElse(Seq.empty)\n    val share = coreData.flatMap(_.share)\n    val reply = coreData.flatMap(_.reply)\n\n    FeatureMapBuilder()\n      .add(AuthorIdFeature, coreData.map(_.userId))\n      .add(DirectedAtUserIdFeature, coreData.flatMap(_.directedAtUser.map(_.userId)))\n      .add(\n        ExclusiveConversationAuthorIdFeature,\n        tweet.exclusiveTweetControl.map(_.conversationAuthorId))\n      .add(HasImageFeature, TweetMediaFeaturesExtractor.hasImage(tweet))\n      .add(HasVideoFeature, TweetMediaFeaturesExtractor.hasVideo(tweet))\n      .add(InReplyToTweetIdFeature, reply.flatMap(_.inReplyToStatusId))\n      .add(InReplyToUserIdFeature, reply.map(_.inReplyToUserId))\n      .add(IsRetweetFeature, share.isDefined)\n      .add(MentionScreenNameFeature, mentions.map(_.screenName))\n      .add(MentionUserIdFeature, mentions.flatMap(_.userId))\n      .add(QuotedTweetIdFeature, quotedTweet.map(_.tweetId))\n      .add(QuotedUserIdFeature, quotedTweet.map(_.userId))\n      .add(SourceTweetIdFeature, share.map(_.sourceStatusId))\n      .add(SourceUserIdFeature, share.map(_.sourceUserId))\n      .build()\n  }\n\n  private def readFromTweetypie(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Stitch[FeatureMap] = {\n    tweetypieStitchClient\n      .getTweetFields(\n        tweetId = candidate.candidate.id,\n        options = tp.GetTweetFieldsOptions(\n          tweetIncludes = RequestFields.TweetStaticEntitiesFields,\n          includeRetweetedTweet = false,\n          includeQuotedTweet = false,\n          forUserId = query.getOptionalUserId, // Needed to get protected Tweets for certain users\n          visibilityPolicy = tp.TweetVisibilityPolicy.UserVisible,\n          safetyLevel = Some(sp.SafetyLevel.FilterNone) // VF is handled in the For You product\n        )\n      ).map {\n        case tp.GetTweetFieldsResult(_, tp.TweetFieldsResultState.Found(found), _, _) =>\n          cacheClient.set(candidate.candidate.id, found.tweet, CacheTTL)\n          createFeatureMap(found.tweet)\n        case _ =>\n          DefaultFeatureMap + (AuthorIdFeature, candidate.features.getOrElse(AuthorIdFeature, None))\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/TweetypieVisibilityFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsHydratedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.OonNsfwFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetLanguageFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTextFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnableTweetEntityServiceVisibilityMigrationParam\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithLongTimeout\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityIdFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_is_nsfw.IsNsfw\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_visibility_reason.VisibilityReason\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.spam.rtf.{thriftscala => rtf}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.generated.client.tweetypie.managed.HomeMixerOnTweetClientColumn\nimport com.twitter.tweetypie.{thriftscala => tp}\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\ncase class TweetypieVisibilityFeatures(\n  communityId: Option[Long],\n  isHydrated: Boolean,\n  isNsfw: Boolean,\n  oonNsfw: Boolean,\n  tweetText: Option[String],\n  tweetLanguage: Option[String],\n  visibilityReason: Option[rtf.FilteredReason])\n\n@Singleton\nclass TweetypieVisibilityFeatureHydrator @Inject() (\n  tweetypieStitchClient: TweetypieStitchClient,\n  statsReceiver: StatsReceiver,\n  @Named(BatchedStratoClientWithLongTimeout) stratoClient: Client)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Logging {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetypieVisibility\")\n\n  private val tweetypieTweetsVisibilityFoundCounter =\n    statsReceiver.counter(\"VisibilityFeatureTweetypieTweetsFound\")\n  private val tweetypieTweetsVisibilityNotFoundCounter =\n    statsReceiver.counter(\"VisibilityFeatureTweetypieTweetsNotFound\")\n  private val tesTweetsVisibilityFoundCounter =\n    statsReceiver.counter(\"VisibilityFeatureTesTweetsFound\")\n  private val tesTweetsVisibilityNotFoundCounter =\n    statsReceiver.counter(\"VisibilityFeatureTesTweetsNotFound\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    CommunityIdFeature,\n    IsHydratedFeature,\n    IsNsfw,\n    OonNsfwFeature,\n    TweetTextFeature,\n    TweetLanguageFeature,\n    VisibilityReason\n  )\n\n  private val HydrationFields: Set[tp.TweetInclude] = Set(\n    tp.TweetInclude.TweetFieldId(tp.Tweet.CommunitiesField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.CoreDataField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.IdField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.LanguageField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.QuotedTweetField.id)\n  )\n\n  private val DefaultTweetFieldsOptions = tp.GetTweetFieldsOptions(\n    tweetIncludes = HydrationFields,\n    includeRetweetedTweet = true,\n    includeQuotedTweet = true,\n    visibilityPolicy = tp.TweetVisibilityPolicy.UserVisible,\n    safetyLevel = Some(rtf.SafetyLevel.TimelineHome)\n  )\n\n  private val OutOfNetworkTweetFieldsOptions =\n    DefaultTweetFieldsOptions.copy(safetyLevel = Some(rtf.SafetyLevel.TimelineHomeRecommendations))\n\n  private val DefaultTweetypieVisibilityFeatures = TweetypieVisibilityFeatures(\n    communityId = None,\n    isHydrated = false,\n    isNsfw = false,\n    oonNsfw = false,\n    tweetLanguage = None,\n    tweetText = None,\n    visibilityReason = None\n  )\n\n  private def buildFeatureMap(\n    gtfResult: Stitch[tp.GetTweetFieldsResult],\n    inNetwork: Boolean,\n    fromTes: Boolean,\n    tweetId: Long\n  ): Stitch[(Long, TweetypieVisibilityFeatures)] = {\n    gtfResult.map {\n      case tp.GetTweetFieldsResult(_, tp.TweetFieldsResultState.Found(found), quote, _) =>\n        if (fromTes) tesTweetsVisibilityFoundCounter.incr()\n        else tweetypieTweetsVisibilityFoundCounter.incr()\n\n        val coreData = found.tweet.coreData\n\n        val isNsfwAdmin = coreData.exists(_.nsfwAdmin)\n        val isNsfwUser = coreData.exists(_.nsfwUser)\n        val sourceTweetIsNsfw =\n          found.retweetedTweet.exists(_.coreData.exists(data => data.nsfwAdmin || data.nsfwUser))\n\n        val quotedTweetDropped = quote.exists {\n          case _: tp.TweetFieldsResultState.Filtered => true\n          case _: tp.TweetFieldsResultState.NotFound => true\n          case _ => false\n        }\n        val quotedTweetIsNsfw = quote.exists {\n          case quoteTweet: tp.TweetFieldsResultState.Found =>\n            quoteTweet.found.tweet.coreData.exists(data => data.nsfwAdmin || data.nsfwUser)\n          case _ => false\n        }\n\n        val isNsfw = isNsfwAdmin || isNsfwUser || sourceTweetIsNsfw || quotedTweetIsNsfw\n\n        val communityId = found.tweet.communities.flatMap(_.communityIds.headOption)\n\n        tweetId -> TweetypieVisibilityFeatures(\n          communityId = communityId,\n          // Since this tweet was Found, it is not dropped, so we only need to check if there\n          // was a dropped quoted tweet.\n          isHydrated = !quotedTweetDropped,\n          isNsfw = isNsfw,\n          oonNsfw = !inNetwork && isNsfw,\n          tweetLanguage = found.tweet.language.map(_.language),\n          tweetText = coreData.map(_.text),\n          visibilityReason = found.suppressReason\n        )\n\n      case _ =>\n        if (fromTes) tesTweetsVisibilityNotFoundCounter.incr()\n        else tweetypieTweetsVisibilityNotFoundCounter.incr()\n        tweetId -> DefaultTweetypieVisibilityFeatures\n    }\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n\n    try {\n      val followedUserIds = query.features.get.getOrElse(SGSFollowedUsersFeature, Seq.empty).toSet\n      val inNetworkTweetFieldsOptions =\n        DefaultTweetFieldsOptions.copy(forUserId = Some(query.getRequiredUserId))\n\n      val outOfNetworkTweetFieldsOptions =\n        OutOfNetworkTweetFieldsOptions.copy(forUserId = Some(query.getRequiredUserId))\n\n      val resultsStitch = Stitch\n        .collect {\n          candidates\n            .flatMap { candidate =>\n              val ancestors = candidate.features.getOrElse(AncestorsFeature, Seq.empty).map {\n                ancestor =>\n                  val inNetwork =\n                    ancestor.userId == query.getRequiredUserId ||\n                      followedUserIds.contains(ancestor.userId)\n                  (ancestor.tweetId, ancestor.userId, inNetwork)\n              }\n\n              val authorId = candidate.features.get(AuthorIdFeature).get\n              val inNetwork =\n                authorId == query.getRequiredUserId || followedUserIds.contains(authorId)\n\n              Seq((candidate.candidate.id, authorId, inNetwork)) ++ ancestors.headOption ++\n                ancestors.lastOption\n            }.distinct.map {\n              case (tweetId, _, inNetwork) =>\n                val gtfOptions =\n                  if (inNetwork) inNetworkTweetFieldsOptions else outOfNetworkTweetFieldsOptions\n\n                try {\n                  if (query.params(EnableTweetEntityServiceVisibilityMigrationParam)) {\n                    val fetcher = new HomeMixerOnTweetClientColumn(stratoClient).fetcher\n                    val response = fetcher.fetch(tweetId, gtfOptions).map(_.v)\n                    response.flatMap {\n                      case Some(result) =>\n                        buildFeatureMap(Stitch.value(result), inNetwork, fromTes = true, tweetId)\n                      case None =>\n                        tesTweetsVisibilityNotFoundCounter.incr()\n                        Stitch.value(tweetId -> DefaultTweetypieVisibilityFeatures)\n                    }\n                  } else {\n                    buildFeatureMap(\n                      tweetypieStitchClient.getTweetFields(tweetId, gtfOptions),\n                      inNetwork,\n                      fromTes = false,\n                      tweetId\n                    )\n                  }\n                } catch {\n                  case e: Exception =>\n                    error(s\"2 - Error fetching tweetypie visibility: $e\")\n                    Stitch.value(tweetId -> DefaultTweetypieVisibilityFeatures)\n                }\n            }\n        }.onFailure {\n          case e: Exception =>\n            error(s\"1 - Error fetching tweetypie visibility: $e\")\n        }\n\n      resultsStitch.map { results =>\n        val resultsMap = results.toMap\n        candidates.map { candidate =>\n          val ancestors = candidate.features.getOrElse(AncestorsFeature, Seq.empty)\n          val ancestorTweetypieVisibilityFeatures =\n            (ancestors.headOption ++ ancestors.lastOption).toSeq.distinct.map { ancestor =>\n              resultsMap.getOrElse(ancestor.tweetId, DefaultTweetypieVisibilityFeatures)\n            }\n\n          val ancestorsHydrated =\n            ancestorTweetypieVisibilityFeatures.map(_.isHydrated).forall(identity)\n          val ancestorsOonNsfw = ancestorTweetypieVisibilityFeatures.map(_.oonNsfw).exists(identity)\n          val ancestorsNsfw = ancestorTweetypieVisibilityFeatures.map(_.isNsfw).exists(identity)\n\n          val tweetypieVisibilityFeatures =\n            resultsMap.getOrElse(candidate.candidate.id, DefaultTweetypieVisibilityFeatures)\n\n          FeatureMapBuilder()\n            .add(CommunityIdFeature, tweetypieVisibilityFeatures.communityId)\n            .add(IsHydratedFeature, tweetypieVisibilityFeatures.isHydrated && ancestorsHydrated)\n            .add(IsNsfw, Some(tweetypieVisibilityFeatures.isNsfw || ancestorsNsfw))\n            .add(OonNsfwFeature, tweetypieVisibilityFeatures.oonNsfw || ancestorsOonNsfw)\n            .add(TweetLanguageFeature, tweetypieVisibilityFeatures.tweetLanguage)\n            .add(TweetTextFeature, tweetypieVisibilityFeatures.tweetText)\n            .add(VisibilityReason, tweetypieVisibilityFeatures.visibilityReason)\n            .build()\n        }\n      }\n    } catch {\n      case e: Exception =>\n        error(s\"3 - Error fetching tweetypie visibility: $e\")\n        Stitch.value(Seq.empty)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/TwhinAuthorFollowFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinAuthorFollowFeatureRepository\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.twhin_embeddings.TwhinAuthorFollowEmbeddingsAdapter\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.servo.repository.KeyValueResult\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinAuthorFollowFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinAuthorFollowFeatureHydrator @Inject() (\n  @Named(TwhinAuthorFollowFeatureRepository)\n  client: KeyValueRepository[Seq[Long], Long, ml.FloatTensor],\n  override val statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TwhinAuthorFollow\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinAuthorFollowFeature)\n\n  override val statScope: String = identifier.toString\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val possiblyAuthorIds = extractKeys(candidates)\n    val authorIds = possiblyAuthorIds.flatten\n\n    val response: Future[KeyValueResult[Long, ml.FloatTensor]] =\n      if (authorIds.isEmpty) Future.value(KeyValueResult.empty) else client(authorIds)\n\n    response.map { result =>\n      possiblyAuthorIds.map { possiblyAuthorId =>\n        val value = observedGet(key = possiblyAuthorId, keyValueResult = result)\n        val transformedValue = postTransformer(value)\n\n        FeatureMapBuilder().add(TwhinAuthorFollowFeature, transformedValue).build()\n      }\n    }\n  }\n\n  private def postTransformer(embedding: Try[Option[ml.FloatTensor]]): Try[DataRecord] = {\n    embedding.map { floatTensor =>\n      TwhinAuthorFollowEmbeddingsAdapter.adaptToDataRecords(floatTensor).asScala.head\n    }\n  }\n\n  private def extractKeys(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] = {\n    candidates.map { candidate =>\n      CandidatesUtil.getOriginalAuthorId(candidate.features)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/TwhinUserEngagementQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinUserEngagementFeatureRepository\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.twhin_embeddings.TwhinUserEngagementEmbeddingsAdapter\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinUserEngagementFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinUserEngagementQueryFeatureHydrator @Inject() (\n  @Named(TwhinUserEngagementFeatureRepository)\n  client: KeyValueRepository[Seq[Long], Long, ml.FloatTensor],\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TwhinUserEngagement\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinUserEngagementFeature)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyLossCounter = scopedStatsReceiver.counter(\"key/loss\")\n  private val keyFailureCounter = scopedStatsReceiver.counter(\"key/failure\")\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n    Stitch.callFuture(client(Seq(userId))).map { results =>\n      val embedding: Option[ml.FloatTensor] = results(userId) match {\n        case Return(value) =>\n          if (value.exists(_.floats.nonEmpty)) keyFoundCounter.incr()\n          else keyLossCounter.incr()\n          value\n        case Throw(_) =>\n          keyFailureCounter.incr()\n          None\n        case _ =>\n          None\n      }\n\n      val dataRecord =\n        TwhinUserEngagementEmbeddingsAdapter.adaptToDataRecords(embedding).asScala.head\n\n      FeatureMapBuilder()\n        .add(TwhinUserEngagementFeature, dataRecord)\n        .build()\n    }\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/TwhinUserFollowQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwhinUserFollowFeatureRepository\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.twhin_embeddings.TwhinUserFollowEmbeddingsAdapter\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\nobject TwhinUserFollowFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwhinUserFollowQueryFeatureHydrator @Inject() (\n  @Named(TwhinUserFollowFeatureRepository)\n  client: KeyValueRepository[Seq[Long], Long, ml.FloatTensor],\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TwhinUserFollow\")\n\n  override val features: Set[Feature[_, _]] = Set(TwhinUserFollowFeature)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private val keyLossCounter = scopedStatsReceiver.counter(\"key/loss\")\n  private val keyFailureCounter = scopedStatsReceiver.counter(\"key/failure\")\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val userId = query.getRequiredUserId\n    Stitch.callFuture(client(Seq(userId))).map { results =>\n      val embedding: Option[ml.FloatTensor] = results(userId) match {\n        case Return(value) =>\n          if (value.exists(_.floats.nonEmpty)) keyFoundCounter.incr()\n          else keyLossCounter.incr()\n          value\n        case Throw(_) =>\n          keyFailureCounter.incr()\n          None\n        case _ =>\n          None\n      }\n\n      val dataRecord = TwhinUserFollowEmbeddingsAdapter.adaptToDataRecords(embedding).asScala.head\n\n      FeatureMapBuilder()\n        .add(TwhinUserFollowFeature, dataRecord)\n        .build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/UserFollowedTopicIdsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UserFollowedTopicIdsRepository\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.servo.keyvalue.KeyValueResult\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject UserFollowedTopicIdsFeature extends Feature[TweetCandidate, Seq[Long]]\n\n@Singleton\nclass UserFollowedTopicIdsFeatureHydrator @Inject() (\n  @Named(UserFollowedTopicIdsRepository)\n  client: KeyValueRepository[Seq[Long], Long, Seq[Long]],\n  override val statsReceiver: StatsReceiver)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserFollowedTopicIds\")\n\n  override val features: Set[Feature[_, _]] = Set(UserFollowedTopicIdsFeature)\n\n  override val statScope: String = identifier.toString\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val possiblyAuthorIds = extractKeys(candidates)\n    val authorIds = possiblyAuthorIds.flatten\n\n    val response: Future[KeyValueResult[Long, Seq[Long]]] =\n      if (authorIds.isEmpty) Future.value(KeyValueResult.empty) else client(authorIds)\n\n    response.map { result =>\n      possiblyAuthorIds.map { possiblyAuthorId =>\n        val value = observedGet(key = possiblyAuthorId, keyValueResult = result)\n        val transformedValue = postTransformer(value)\n\n        FeatureMapBuilder().add(UserFollowedTopicIdsFeature, transformedValue).build()\n      }\n    }\n  }\n\n  private def postTransformer(input: Try[Option[Seq[Long]]]): Try[Seq[Long]] = {\n    input.map(_.getOrElse(Seq.empty[Long]))\n  }\n\n  private def extractKeys(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] = {\n    candidates.map { candidate =>\n      candidate.features\n        .getTry(AuthorIdFeature)\n        .toOption\n        .flatten\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/UserLanguagesFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UserLanguagesRepository\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.search.common.constants.{thriftscala => scc}\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject UserLanguagesFeature extends Feature[PipelineQuery, Seq[scc.ThriftLanguage]]\n\n@Singleton\ncase class UserLanguagesFeatureHydrator @Inject() (\n  @Named(UserLanguagesRepository) client: KeyValueRepository[Seq[Long], Long, Seq[\n    scc.ThriftLanguage\n  ]],\n  statsReceiver: StatsReceiver)\n    extends QueryFeatureHydrator[PipelineQuery]\n    with ObservedKeyValueResultHandler {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"UserLanguages\")\n\n  override val features: Set[Feature[_, _]] = Set(UserLanguagesFeature)\n\n  override val statScope: String = identifier.toString\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val key = query.getRequiredUserId\n    Stitch.callFuture(client(Seq(key))).map { result =>\n      val feature =\n        observedGet(key = Some(key), keyValueResult = result).map(_.getOrElse(Seq.empty))\n      FeatureMapBuilder()\n        .add(UserLanguagesFeature, feature)\n        .build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/UserStateQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.UserStateFeature\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.user_health.v1.{thriftscala => uhv1}\nimport com.twitter.timelines.user_health.{thriftscala => uh}\nimport com.twitter.user_session_store.ReadOnlyUserSessionStore\nimport com.twitter.user_session_store.ReadRequest\nimport com.twitter.user_session_store.UserSessionDataset\nimport com.twitter.user_session_store.UserSessionDataset.UserSessionDataset\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class UserStateQueryFeatureHydrator @Inject() (\n  userSessionStore: ReadOnlyUserSessionStore)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserState\")\n\n  override val features: Set[Feature[_, _]] = Set(UserStateFeature)\n\n  private val datasets: Set[UserSessionDataset] = Set(UserSessionDataset.UserHealth)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    userSessionStore\n      .read(ReadRequest(query.getRequiredUserId, datasets))\n      .map { userSession =>\n        val userState = userSession.flatMap {\n          _.userHealth match {\n            case Some(uh.UserHealth.V1(uhv1.UserHealth(userState))) => userState\n            case _ => None\n          }\n        }\n\n        FeatureMapBuilder()\n          .add(UserStateFeature, userState)\n          .build()\n      }\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.9)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/UtegFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.FavoritedByCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RepliedByCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RepliedByEngagerIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RetweetedByCountFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RetweetedByEngagerIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UtegSocialProofRepository\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.recos.recos_common.{thriftscala => rc}\nimport com.twitter.recos.user_tweet_entity_graph.{thriftscala => uteg}\nimport com.twitter.servo.keyvalue.KeyValueResult\nimport com.twitter.servo.repository.KeyValueRepository\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass UtegFeatureHydrator @Inject() (\n  @Named(UtegSocialProofRepository) client: KeyValueRepository[\n    (Seq[Long], (Long, Map[Long, Double])),\n    Long,\n    uteg.TweetRecommendation\n  ]) extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with Conditionally[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"Uteg\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    FavoritedByUserIdsFeature,\n    RetweetedByEngagerIdsFeature,\n    RepliedByEngagerIdsFeature,\n    FavoritedByCountFeature,\n    RetweetedByCountFeature,\n    RepliedByCountFeature\n  )\n\n  override def onlyIf(query: PipelineQuery): Boolean = query.features\n    .exists(_.getOrElse(RealGraphInNetworkScoresFeature, Map.empty[Long, Double]).nonEmpty)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val seedUserWeights = query.features.map(_.get(RealGraphInNetworkScoresFeature)).get\n\n    val sourceTweetIds = candidates.flatMap(_.features.getOrElse(SourceTweetIdFeature, None))\n    val inReplyToTweetIds = candidates.flatMap(_.features.getOrElse(InReplyToTweetIdFeature, None))\n    val tweetIds = candidates.map(_.candidate.id)\n    val tweetIdsToSend = (tweetIds ++ sourceTweetIds ++ inReplyToTweetIds).distinct\n\n    val utegQuery = (tweetIdsToSend, (query.getRequiredUserId, seedUserWeights))\n\n    client(utegQuery).map(handleResponse(candidates, _))\n  }\n\n  private def handleResponse(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n    results: KeyValueResult[Long, uteg.TweetRecommendation],\n  ): Seq[FeatureMap] = {\n    candidates.map { candidate =>\n      val inNetwork = candidate.features.getOrElse(FromInNetworkSourceFeature, false)\n      val candidateProof = results(candidate.candidate.id).toOption.flatten\n      val sourceProof = candidate.features\n        .getOrElse(SourceTweetIdFeature, None).flatMap(results(_).toOption.flatten)\n      val proofs = Seq(candidateProof, sourceProof).flatten.map(_.socialProofByType)\n\n      val favoritedBy = proofs.flatMap(_.get(rc.SocialProofType.Favorite)).flatten\n      val retweetedBy = proofs.flatMap(_.get(rc.SocialProofType.Retweet)).flatten\n      val repliedBy = proofs.flatMap(_.get(rc.SocialProofType.Reply)).flatten\n\n      val (favoritedByCount, retweetedByCount, repliedByCount) =\n        if (!inNetwork) {\n          (favoritedBy.size.toDouble, retweetedBy.size.toDouble, repliedBy.size.toDouble)\n        } else { (0.0, 0.0, 0.0) }\n\n      FeatureMapBuilder()\n        .add(FavoritedByUserIdsFeature, favoritedBy)\n        .add(RetweetedByEngagerIdsFeature, retweetedBy)\n        .add(RepliedByEngagerIdsFeature, repliedBy)\n        .add(FavoritedByCountFeature, favoritedByCount)\n        .add(RetweetedByCountFeature, retweetedByCount)\n        .add(RepliedByCountFeature, repliedByCount)\n        .build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/ValidLikedByUserIdsFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator\n\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ValidLikedByUserIdsFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject ValidLikedByUserIdsFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"ValidLikedByUserIds\")\n\n  override val features: Set[Feature[_, _]] = Set(ValidLikedByUserIdsFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    Stitch.value {\n      candidates.map { candidate =>\n        val validLikers = candidate.features.getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty)\n        FeatureMapBuilder().add(ValidLikedByUserIdsFeature, validLikers).build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/author_features/AuthorFeaturesAdapter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.author_features\n\nimport com.twitter.home_mixer.util.DataRecordUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.util.CompactDataRecordConverter\nimport com.twitter.ml.api.util.FDsl._\nimport com.twitter.timelines.author_features.v1.{thriftjava => af}\nimport com.twitter.timelines.prediction.common.adapters.TimelinesAdapterBase\nimport com.twitter.timelines.prediction.common.aggregates.TimelinesAggregationConfig\nimport com.twitter.timelines.prediction.features.user_health.UserHealthFeatures\nimport scala.collection.JavaConverters._\n\nobject AuthorFeaturesAdapter extends TimelinesAdapterBase[af.AuthorFeatures] {\n\n  private val Prefix = \"original_author.timelines.original_author_aggregates.\"\n\n  private val typedAggregateGroups =\n    TimelinesAggregationConfig.originalAuthorAggregatesV1.buildTypedAggregateGroups()\n\n  private val aggregateFeaturesRenameMap: Map[Feature[_], Feature[_]] =\n    typedAggregateGroups.map(_.outputFeaturesToRenamedOutputFeatures(Prefix)).reduce(_ ++ _)\n\n  private val prefixedOriginalAuthorAggregateFeatures =\n    typedAggregateGroups.flatMap(_.allOutputFeatures).map { feature =>\n      aggregateFeaturesRenameMap.getOrElse(feature, feature)\n    }\n\n  private val authorFeatures = prefixedOriginalAuthorAggregateFeatures ++ Seq(\n    UserHealthFeatures.AuthorState,\n    UserHealthFeatures.NumAuthorFollowers,\n    UserHealthFeatures.NumAuthorConnectDays,\n    UserHealthFeatures.NumAuthorConnect\n  )\n\n  private val aggregateFeatureContext: FeatureContext =\n    new FeatureContext(typedAggregateGroups.flatMap(_.allOutputFeatures).asJava)\n\n  private lazy val prefixedAggregateFeatureContext: FeatureContext =\n    new FeatureContext(prefixedOriginalAuthorAggregateFeatures.asJava)\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(authorFeatures: _*)\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  private val compactDataRecordConverter = new CompactDataRecordConverter()\n\n  override def adaptToDataRecords(\n    authorFeatures: af.AuthorFeatures\n  ): java.util.List[DataRecord] = {\n    val dataRecord =\n      if (authorFeatures.aggregates != null) {\n        val originalAuthorAggregatesDataRecord =\n          compactDataRecordConverter.compactDataRecordToDataRecord(authorFeatures.aggregates)\n\n        DataRecordUtil.applyRename(\n          originalAuthorAggregatesDataRecord,\n          aggregateFeatureContext,\n          prefixedAggregateFeatureContext,\n          aggregateFeaturesRenameMap)\n      } else new DataRecord\n\n    if (authorFeatures.user_health != null) {\n      val userHealth = authorFeatures.user_health\n\n      if (userHealth.user_state != null) {\n        dataRecord.setFeatureValue(\n          UserHealthFeatures.AuthorState,\n          userHealth.user_state.getValue.toLong\n        )\n      }\n\n      dataRecord.setFeatureValue(\n        UserHealthFeatures.NumAuthorFollowers,\n        userHealth.num_followers.toDouble\n      )\n\n      dataRecord.setFeatureValue(\n        UserHealthFeatures.NumAuthorConnectDays,\n        userHealth.num_connect_days.toDouble\n      )\n\n      dataRecord.setFeatureValue(\n        UserHealthFeatures.NumAuthorConnect,\n        userHealth.num_connect.toDouble\n      )\n    }\n\n    List(dataRecord).asJava\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/author_features/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/scala/com/twitter/timelines/prediction/common/aggregates\",\n        \"src/scala/com/twitter/timelines/prediction/features/user_health\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/timelines/author_features:thrift-java\",\n        \"timelines/data_processing/ml_util/aggregation_framework\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/content/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters\",\n        \"src/scala/com/twitter/timelines/prediction/features/common\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/content/ContentFeatureAdapter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.content\n\nimport com.twitter.home_mixer.model.ContentFeatures\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.util.DataRecordConverters._\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.common.adapters.TweetLengthType\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport com.twitter.timelines.prediction.features.conversation_features.ConversationFeatures\nimport scala.collection.JavaConverters._\n\nobject ContentFeatureAdapter extends TimelinesMutatingAdapterBase[Option[ContentFeatures]] {\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(\n    ConversationFeatures.IS_SELF_THREAD_TWEET,\n    ConversationFeatures.IS_LEAF_IN_SELF_THREAD,\n    TimelinesSharedFeatures.ASPECT_RATIO_DEN,\n    TimelinesSharedFeatures.ASPECT_RATIO_NUM,\n    TimelinesSharedFeatures.BIT_RATE,\n    TimelinesSharedFeatures.CLASSIFICATION_LABELS,\n    TimelinesSharedFeatures.COLOR_1_BLUE,\n    TimelinesSharedFeatures.COLOR_1_GREEN,\n    TimelinesSharedFeatures.COLOR_1_PERCENTAGE,\n    TimelinesSharedFeatures.COLOR_1_RED,\n    TimelinesSharedFeatures.FACE_AREAS,\n    TimelinesSharedFeatures.HAS_APP_INSTALL_CALL_TO_ACTION,\n    TimelinesSharedFeatures.HAS_DESCRIPTION,\n    TimelinesSharedFeatures.HAS_QUESTION,\n    TimelinesSharedFeatures.HAS_SELECTED_PREVIEW_IMAGE,\n    TimelinesSharedFeatures.HAS_TITLE,\n    TimelinesSharedFeatures.HAS_VISIT_SITE_CALL_TO_ACTION,\n    TimelinesSharedFeatures.HAS_WATCH_NOW_CALL_TO_ACTION,\n    TimelinesSharedFeatures.HEIGHT_1,\n    TimelinesSharedFeatures.HEIGHT_2,\n    TimelinesSharedFeatures.HEIGHT_3,\n    TimelinesSharedFeatures.HEIGHT_4,\n    TimelinesSharedFeatures.IS_360,\n    TimelinesSharedFeatures.IS_EMBEDDABLE,\n    TimelinesSharedFeatures.IS_MANAGED,\n    TimelinesSharedFeatures.IS_MONETIZABLE,\n    TimelinesSharedFeatures.MEDIA_PROVIDERS,\n    TimelinesSharedFeatures.NUM_CAPS,\n    TimelinesSharedFeatures.NUM_COLOR_PALLETTE_ITEMS,\n    TimelinesSharedFeatures.NUM_FACES,\n    TimelinesSharedFeatures.NUM_MEDIA_TAGS,\n    TimelinesSharedFeatures.NUM_NEWLINES,\n    TimelinesSharedFeatures.NUM_STICKERS,\n    TimelinesSharedFeatures.NUM_WHITESPACES,\n    TimelinesSharedFeatures.RESIZE_METHOD_1,\n    TimelinesSharedFeatures.RESIZE_METHOD_2,\n    TimelinesSharedFeatures.RESIZE_METHOD_3,\n    TimelinesSharedFeatures.RESIZE_METHOD_4,\n    TimelinesSharedFeatures.TWEET_LENGTH,\n    TimelinesSharedFeatures.TWEET_LENGTH_TYPE,\n    TimelinesSharedFeatures.VIDEO_DURATION,\n    TimelinesSharedFeatures.VIEW_COUNT,\n    TimelinesSharedFeatures.WIDTH_1,\n    TimelinesSharedFeatures.WIDTH_2,\n    TimelinesSharedFeatures.WIDTH_3,\n    TimelinesSharedFeatures.WIDTH_4,\n  )\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  private def getTweetLengthType(tweetLength: Int): Long = {\n    tweetLength match {\n      case x if 0 > x || 280 < x => TweetLengthType.INVALID\n      case x if 0 <= x && x <= 30 => TweetLengthType.VERY_SHORT\n      case x if 30 < x && x <= 60 => TweetLengthType.SHORT\n      case x if 60 < x && x <= 90 => TweetLengthType.MEDIUM\n      case x if 90 < x && x <= 140 => TweetLengthType.LENGTHY\n      case x if 140 < x && x <= 210 => TweetLengthType.VERY_LENGTHY\n      case x if x > 210 => TweetLengthType.MAXIMUM_LENGTH\n    }\n  }\n\n  override def setFeatures(\n    contentFeatures: Option[ContentFeatures],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    if (contentFeatures.nonEmpty) {\n      val features = contentFeatures.get\n      // Conversation Features\n      richDataRecord.setFeatureValueFromOption(\n        ConversationFeatures.IS_SELF_THREAD_TWEET,\n        Some(features.selfThreadMetadata.nonEmpty)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        ConversationFeatures.IS_LEAF_IN_SELF_THREAD,\n        features.selfThreadMetadata.map(_.isLeaf)\n      )\n\n      // Media Features\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.ASPECT_RATIO_DEN,\n        features.aspectRatioDen.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.ASPECT_RATIO_NUM,\n        features.aspectRatioNum.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.BIT_RATE,\n        features.bitRate.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HEIGHT_1,\n        features.heights.flatMap(_.lift(0)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HEIGHT_2,\n        features.heights.flatMap(_.lift(1)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HEIGHT_3,\n        features.heights.flatMap(_.lift(2)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HEIGHT_4,\n        features.heights.flatMap(_.lift(3)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_MEDIA_TAGS,\n        features.numMediaTags.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.RESIZE_METHOD_1,\n        features.resizeMethods.flatMap(_.lift(0)).map(_.toLong)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.RESIZE_METHOD_2,\n        features.resizeMethods.flatMap(_.lift(1)).map(_.toLong)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.RESIZE_METHOD_3,\n        features.resizeMethods.flatMap(_.lift(2)).map(_.toLong)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.RESIZE_METHOD_4,\n        features.resizeMethods.flatMap(_.lift(3)).map(_.toLong)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.VIDEO_DURATION,\n        features.videoDurationMs.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.WIDTH_1,\n        features.widths.flatMap(_.lift(0)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.WIDTH_2,\n        features.widths.flatMap(_.lift(1)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.WIDTH_3,\n        features.widths.flatMap(_.lift(2)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.WIDTH_4,\n        features.widths.flatMap(_.lift(3)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_COLOR_PALLETTE_ITEMS,\n        features.numColors.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.COLOR_1_RED,\n        features.dominantColorRed.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.COLOR_1_BLUE,\n        features.dominantColorBlue.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.COLOR_1_GREEN,\n        features.dominantColorGreen.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.COLOR_1_PERCENTAGE,\n        features.dominantColorPercentage\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.MEDIA_PROVIDERS,\n        features.mediaOriginProviders.map(_.toSet.asJava)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.IS_360,\n        features.is360\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.VIEW_COUNT,\n        features.viewCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.IS_MANAGED,\n        features.isManaged\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.IS_MONETIZABLE,\n        features.isMonetizable\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.IS_EMBEDDABLE,\n        features.isEmbeddable\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_STICKERS,\n        features.stickerIds.map(_.length.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_FACES,\n        features.faceAreas.map(_.length.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.FACE_AREAS,\n        // guard for exception from max on empty seq\n        features.faceAreas.map(faceAreas =>\n          faceAreas.map(_.toDouble).reduceOption(_ max _).getOrElse(0.0))\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_SELECTED_PREVIEW_IMAGE,\n        features.hasSelectedPreviewImage\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_TITLE,\n        features.hasTitle\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_DESCRIPTION,\n        features.hasDescription\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_VISIT_SITE_CALL_TO_ACTION,\n        features.hasVisitSiteCallToAction\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_APP_INSTALL_CALL_TO_ACTION,\n        features.hasAppInstallCallToAction\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_WATCH_NOW_CALL_TO_ACTION,\n        features.hasWatchNowCallToAction\n      )\n      // text features\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_CAPS,\n        Some(features.numCaps.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.TWEET_LENGTH,\n        Some(features.length.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.TWEET_LENGTH_TYPE,\n        Some(getTweetLengthType(features.length.toInt))\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_WHITESPACES,\n        Some(features.numWhiteSpaces.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_QUESTION,\n        Some(features.hasQuestion)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.NUM_NEWLINES,\n        features.numNewlines.map(_.toDouble)\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/content/InReplyToContentFeatureAdapter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.content\n\nimport com.twitter.home_mixer.model.ContentFeatures\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.util.DataRecordConverters.RichDataRecordWrapper\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.common.InReplyToTweetTimelinesSharedFeatures\n\nobject InReplyToContentFeatureAdapter\n    extends TimelinesMutatingAdapterBase[Option[ContentFeatures]] {\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(\n    // Media Features\n    InReplyToTweetTimelinesSharedFeatures.ASPECT_RATIO_DEN,\n    InReplyToTweetTimelinesSharedFeatures.ASPECT_RATIO_NUM,\n    InReplyToTweetTimelinesSharedFeatures.HEIGHT_1,\n    InReplyToTweetTimelinesSharedFeatures.HEIGHT_2,\n    InReplyToTweetTimelinesSharedFeatures.VIDEO_DURATION,\n    // TextFeatures\n    InReplyToTweetTimelinesSharedFeatures.NUM_CAPS,\n    InReplyToTweetTimelinesSharedFeatures.TWEET_LENGTH,\n    InReplyToTweetTimelinesSharedFeatures.HAS_QUESTION,\n  )\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def setFeatures(\n    contentFeatures: Option[ContentFeatures],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    if (contentFeatures.nonEmpty) {\n      val features = contentFeatures.get\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.ASPECT_RATIO_DEN,\n        features.aspectRatioNum.map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.ASPECT_RATIO_NUM,\n        features.aspectRatioNum.map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.HEIGHT_1,\n        features.heights.flatMap(_.lift(0)).map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.HEIGHT_2,\n        features.heights.flatMap(_.lift(1)).map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.VIDEO_DURATION,\n        features.videoDurationMs.map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.NUM_CAPS,\n        Some(features.numCaps.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.TWEET_LENGTH,\n        Some(features.length.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.HAS_QUESTION,\n        Some(features.hasQuestion)\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/earlybird/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/scala/com/twitter/timelines/prediction/features/common\",\n        \"src/scala/com/twitter/timelines/prediction/features/recap\",\n        \"src/scala/com/twitter/timelines/util\",\n        \"src/thrift/com/twitter/search/common:features-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/earlybird/EarlybirdAdapter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.earlybird\n\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.util.DataRecordConverters._\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.search.common.features.{thriftscala => sc}\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport com.twitter.timelines.prediction.features.recap.RecapFeatures\nimport com.twitter.timelines.util.UrlExtractorUtil\nimport java.lang.{Boolean => JBoolean}\nimport java.lang.{Double => JDouble}\nimport java.util.{Map => JMap}\nimport scala.collection.JavaConverters._\n\nobject EarlybirdAdapter extends TimelinesMutatingAdapterBase[Option[sc.ThriftTweetFeatures]] {\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(\n    RecapFeatures.BIDIRECTIONAL_FAV_COUNT,\n    RecapFeatures.BIDIRECTIONAL_REPLY_COUNT,\n    RecapFeatures.BIDIRECTIONAL_RETWEET_COUNT,\n    RecapFeatures.BLENDER_SCORE,\n    RecapFeatures.CONTAINS_MEDIA,\n    RecapFeatures.CONVERSATIONAL_COUNT,\n    RecapFeatures.EMBEDS_IMPRESSION_COUNT,\n    RecapFeatures.EMBEDS_URL_COUNT,\n    RecapFeatures.FAV_COUNT,\n    RecapFeatures.FAV_COUNT_V2,\n    RecapFeatures.FROM_INACTIVE_USER,\n    RecapFeatures.FROM_MUTUAL_FOLLOW,\n    RecapFeatures.FROM_VERIFIED_ACCOUNT,\n    RecapFeatures.HAS_CARD,\n    RecapFeatures.HAS_CONSUMER_VIDEO,\n    RecapFeatures.HAS_HASHTAG,\n//    RecapFeatures.HAS_IMAGE,\n    RecapFeatures.HAS_LINK,\n    RecapFeatures.HAS_MENTION,\n    RecapFeatures.HAS_MULTIPLE_HASHTAGS_OR_TRENDS,\n    RecapFeatures.HAS_MULTIPLE_MEDIA,\n    RecapFeatures.HAS_NATIVE_IMAGE,\n    RecapFeatures.HAS_NATIVE_VIDEO,\n    RecapFeatures.HAS_NEWS,\n    RecapFeatures.HAS_PERISCOPE,\n    RecapFeatures.HAS_PRO_VIDEO,\n    RecapFeatures.HAS_TREND,\n//    RecapFeatures.HAS_VIDEO,\n    RecapFeatures.HAS_VINE,\n    RecapFeatures.HAS_VISIBLE_LINK,\n    RecapFeatures.IS_AUTHOR_BOT,\n    RecapFeatures.IS_AUTHOR_NEW,\n    RecapFeatures.IS_AUTHOR_NSFW,\n    RecapFeatures.IS_AUTHOR_PROFILE_EGG,\n    RecapFeatures.IS_AUTHOR_SPAM,\n    RecapFeatures.IS_BUSINESS_SCORE,\n    RecapFeatures.IS_OFFENSIVE,\n    RecapFeatures.IS_REPLY,\n    RecapFeatures.IS_RETWEET,\n    RecapFeatures.IS_RETWEETER_BOT,\n    RecapFeatures.IS_RETWEETER_NEW,\n    RecapFeatures.IS_RETWEETER_NSFW,\n    RecapFeatures.IS_RETWEETER_PROFILE_EGG,\n    RecapFeatures.IS_RETWEETER_SPAM,\n    RecapFeatures.IS_RETWEET_OF_REPLY,\n    RecapFeatures.IS_SENSITIVE,\n    RecapFeatures.LANGUAGE,\n    RecapFeatures.LINK_COUNT,\n    RecapFeatures.LINK_LANGUAGE,\n    RecapFeatures.MATCH_SEARCHER_LANGS,\n    RecapFeatures.MATCH_SEARCHER_MAIN_LANG,\n    RecapFeatures.MATCH_UI_LANG,\n    RecapFeatures.MENTIONED_SCREEN_NAMES,\n    RecapFeatures.MENTION_SEARCHER,\n    RecapFeatures.NUM_HASHTAGS,\n    RecapFeatures.NUM_MENTIONS,\n    RecapFeatures.PREV_USER_TWEET_ENGAGEMENT,\n    RecapFeatures.PROBABLY_FROM_FOLLOWED_AUTHOR,\n    RecapFeatures.REPLY_COUNT,\n    RecapFeatures.REPLY_COUNT_V2,\n    RecapFeatures.REPLY_OTHER,\n    RecapFeatures.REPLY_SEARCHER,\n    RecapFeatures.RETWEET_COUNT,\n    RecapFeatures.RETWEET_COUNT_V2,\n    RecapFeatures.RETWEET_DIRECTED_AT_USER_IN_FIRST_DEGREE,\n    RecapFeatures.RETWEET_OF_MUTUAL_FOLLOW,\n    RecapFeatures.RETWEET_OTHER,\n    RecapFeatures.RETWEET_SEARCHER,\n    RecapFeatures.SIGNATURE,\n    RecapFeatures.SOURCE_AUTHOR_REP,\n    RecapFeatures.TEXT_SCORE,\n    RecapFeatures.TWEET_COUNT_FROM_USER_IN_SNAPSHOT,\n    RecapFeatures.UNIDIRECTIONAL_FAV_COUNT,\n    RecapFeatures.UNIDIRECTIONAL_REPLY_COUNT,\n    RecapFeatures.UNIDIRECTIONAL_RETWEET_COUNT,\n    RecapFeatures.URL_DOMAINS,\n    RecapFeatures.USER_REP,\n    RecapFeatures.VIDEO_VIEW_COUNT,\n    // shared features\n    TimelinesSharedFeatures.WEIGHTED_FAV_COUNT,\n    TimelinesSharedFeatures.WEIGHTED_RETWEET_COUNT,\n    TimelinesSharedFeatures.WEIGHTED_REPLY_COUNT,\n    TimelinesSharedFeatures.WEIGHTED_QUOTE_COUNT,\n    TimelinesSharedFeatures.EMBEDS_IMPRESSION_COUNT_V2,\n    TimelinesSharedFeatures.EMBEDS_URL_COUNT_V2,\n    TimelinesSharedFeatures.DECAYED_FAVORITE_COUNT,\n    TimelinesSharedFeatures.DECAYED_RETWEET_COUNT,\n    TimelinesSharedFeatures.DECAYED_REPLY_COUNT,\n    TimelinesSharedFeatures.DECAYED_QUOTE_COUNT,\n    TimelinesSharedFeatures.FAKE_FAVORITE_COUNT,\n    TimelinesSharedFeatures.FAKE_RETWEET_COUNT,\n    TimelinesSharedFeatures.FAKE_REPLY_COUNT,\n    TimelinesSharedFeatures.FAKE_QUOTE_COUNT,\n    TimelinesSharedFeatures.QUOTE_COUNT,\n    TimelinesSharedFeatures.EARLYBIRD_SCORE,\n    // Safety features\n    TimelinesSharedFeatures.LABEL_ABUSIVE_FLAG,\n    TimelinesSharedFeatures.LABEL_ABUSIVE_HI_RCL_FLAG,\n    TimelinesSharedFeatures.LABEL_DUP_CONTENT_FLAG,\n    TimelinesSharedFeatures.LABEL_NSFW_HI_PRC_FLAG,\n    TimelinesSharedFeatures.LABEL_NSFW_HI_RCL_FLAG,\n    TimelinesSharedFeatures.LABEL_SPAM_FLAG,\n    TimelinesSharedFeatures.LABEL_SPAM_HI_RCL_FLAG,\n    // periscope features\n    TimelinesSharedFeatures.PERISCOPE_EXISTS,\n    TimelinesSharedFeatures.PERISCOPE_IS_LIVE,\n    TimelinesSharedFeatures.PERISCOPE_HAS_BEEN_FEATURED,\n    TimelinesSharedFeatures.PERISCOPE_IS_CURRENTLY_FEATURED,\n    TimelinesSharedFeatures.PERISCOPE_IS_FROM_QUALITY_SOURCE,\n    // VISIBLE_TOKEN_RATIO\n    TimelinesSharedFeatures.VISIBLE_TOKEN_RATIO,\n    TimelinesSharedFeatures.HAS_QUOTE,\n    TimelinesSharedFeatures.IS_COMPOSER_SOURCE_CAMERA,\n    // health features\n    TimelinesSharedFeatures.PREPORTED_TWEET_SCORE,\n    // media\n    TimelinesSharedFeatures.CLASSIFICATION_LABELS\n  )\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def setFeatures(\n    ebFeatures: Option[sc.ThriftTweetFeatures],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    if (ebFeatures.nonEmpty) {\n      val features = ebFeatures.get\n      richDataRecord.setFeatureValue[JDouble](\n        RecapFeatures.PREV_USER_TWEET_ENGAGEMENT,\n        features.prevUserTweetEngagement.toDouble\n      )\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.IS_SENSITIVE, features.isSensitiveContent)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.HAS_MULTIPLE_MEDIA, features.hasMultipleMedia)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.IS_AUTHOR_PROFILE_EGG, features.isAuthorProfileEgg)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.IS_AUTHOR_NEW, features.isAuthorNew)\n      richDataRecord\n        .setFeatureValue[JDouble](RecapFeatures.NUM_MENTIONS, features.numMentions.toDouble)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.HAS_MENTION, features.numMentions > 0)\n      richDataRecord\n        .setFeatureValue[JDouble](RecapFeatures.NUM_HASHTAGS, features.numHashtags.toDouble)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.HAS_HASHTAG, features.numHashtags > 0)\n      richDataRecord\n        .setFeatureValue[JDouble](RecapFeatures.LINK_LANGUAGE, features.linkLanguage.toDouble)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.IS_AUTHOR_NSFW, features.isAuthorNSFW)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.IS_AUTHOR_SPAM, features.isAuthorSpam)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.IS_AUTHOR_BOT, features.isAuthorBot)\n      richDataRecord.setFeatureValueFromOption(\n        RecapFeatures.LANGUAGE,\n        features.language.map(_.getValue.toLong))\n      richDataRecord.setFeatureValueFromOption(\n        RecapFeatures.SIGNATURE,\n        features.signature.map(_.toLong))\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.FROM_INACTIVE_USER, features.fromInActiveUser)\n      richDataRecord\n        .setFeatureValue[JBoolean](\n          RecapFeatures.PROBABLY_FROM_FOLLOWED_AUTHOR,\n          features.probablyFromFollowedAuthor)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.FROM_MUTUAL_FOLLOW, features.fromMutualFollow)\n      richDataRecord.setFeatureValue[JBoolean](\n        RecapFeatures.FROM_VERIFIED_ACCOUNT,\n        features.fromVerifiedAccount)\n      richDataRecord.setFeatureValue[JDouble](RecapFeatures.USER_REP, features.userRep)\n      richDataRecord\n        .setFeatureValue[JDouble](RecapFeatures.IS_BUSINESS_SCORE, features.isBusinessScore)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.HAS_CONSUMER_VIDEO, features.hasConsumerVideo)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.HAS_PRO_VIDEO, features.hasProVideo)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.HAS_VINE, features.hasVine)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.HAS_PERISCOPE, features.hasPeriscope)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.HAS_NATIVE_VIDEO, features.hasNativeVideo)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.HAS_NATIVE_IMAGE, features.hasNativeImage)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.HAS_CARD, features.hasCard)\n//      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.HAS_IMAGE, features.hasImage) //handled via TweetEntityServiceContentFeatureHydrator\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.HAS_NEWS, features.hasNews)\n//      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.HAS_VIDEO, features.hasVideo) //handled via TweetEntityServiceContentFeatureHydrator\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.CONTAINS_MEDIA, features.containsMedia)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.RETWEET_SEARCHER, features.retweetSearcher)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.REPLY_SEARCHER, features.replySearcher)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.MENTION_SEARCHER, features.mentionSearcher)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.REPLY_OTHER, features.replyOther)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.RETWEET_OTHER, features.retweetOther)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.IS_REPLY, features.isReply)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.IS_RETWEET, features.isRetweet)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.IS_OFFENSIVE, features.isOffensive)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.MATCH_UI_LANG, features.matchesUILang)\n      richDataRecord\n        .setFeatureValue[JBoolean](\n          RecapFeatures.MATCH_SEARCHER_MAIN_LANG,\n          features.matchesSearcherMainLang)\n      richDataRecord.setFeatureValue[JBoolean](\n        RecapFeatures.MATCH_SEARCHER_LANGS,\n        features.matchesSearcherLangs)\n      richDataRecord\n        .setFeatureValue[JDouble](\n          RecapFeatures.BIDIRECTIONAL_FAV_COUNT,\n          features.bidirectionalFavCount)\n      richDataRecord\n        .setFeatureValue[JDouble](\n          RecapFeatures.UNIDIRECTIONAL_FAV_COUNT,\n          features.unidirectionalFavCount)\n      richDataRecord\n        .setFeatureValue[JDouble](\n          RecapFeatures.BIDIRECTIONAL_REPLY_COUNT,\n          features.bidirectionalReplyCount)\n      richDataRecord\n        .setFeatureValue[JDouble](\n          RecapFeatures.UNIDIRECTIONAL_REPLY_COUNT,\n          features.unidirectionalReplyCount)\n      richDataRecord\n        .setFeatureValue[JDouble](\n          RecapFeatures.BIDIRECTIONAL_RETWEET_COUNT,\n          features.bidirectionalRetweetCount)\n      richDataRecord\n        .setFeatureValue[JDouble](\n          RecapFeatures.UNIDIRECTIONAL_RETWEET_COUNT,\n          features.unidirectionalRetweetCount)\n      richDataRecord\n        .setFeatureValue[JDouble](RecapFeatures.CONVERSATIONAL_COUNT, features.conversationCount)\n      richDataRecord.setFeatureValue[JDouble](\n        RecapFeatures.TWEET_COUNT_FROM_USER_IN_SNAPSHOT,\n        features.tweetCountFromUserInSnapshot\n      )\n      richDataRecord\n        .setFeatureValue[JBoolean](\n          RecapFeatures.IS_RETWEETER_PROFILE_EGG,\n          features.isRetweeterProfileEgg)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.IS_RETWEETER_NEW, features.isRetweeterNew)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.IS_RETWEETER_BOT, features.isRetweeterBot)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.IS_RETWEETER_NSFW, features.isRetweeterNSFW)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.IS_RETWEETER_SPAM, features.isRetweeterSpam)\n      richDataRecord\n        .setFeatureValue[JBoolean](\n          RecapFeatures.RETWEET_OF_MUTUAL_FOLLOW,\n          features.retweetOfMutualFollow)\n      richDataRecord\n        .setFeatureValue[JDouble](RecapFeatures.SOURCE_AUTHOR_REP, features.sourceAuthorRep)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.IS_RETWEET_OF_REPLY, features.isRetweetOfReply)\n      richDataRecord.setFeatureValueFromOption(\n        RecapFeatures.RETWEET_DIRECTED_AT_USER_IN_FIRST_DEGREE,\n        features.retweetDirectedAtUserInFirstDegree\n      )\n      richDataRecord\n        .setFeatureValue[JDouble](\n          RecapFeatures.EMBEDS_IMPRESSION_COUNT,\n          features.embedsImpressionCount.toDouble)\n      richDataRecord\n        .setFeatureValue[JDouble](RecapFeatures.EMBEDS_URL_COUNT, features.embedsUrlCount.toDouble)\n      richDataRecord\n        .setFeatureValue[JDouble](RecapFeatures.VIDEO_VIEW_COUNT, features.videoViewCount.toDouble)\n      richDataRecord\n        .setFeatureValue[JDouble](RecapFeatures.REPLY_COUNT, features.replyCount.toDouble)\n      richDataRecord\n        .setFeatureValue[JDouble](RecapFeatures.RETWEET_COUNT, features.retweetCount.toDouble)\n      richDataRecord.setFeatureValue[JDouble](RecapFeatures.FAV_COUNT, features.favCount.toDouble)\n      richDataRecord.setFeatureValue[JDouble](RecapFeatures.BLENDER_SCORE, features.blenderScore)\n      richDataRecord.setFeatureValue[JDouble](RecapFeatures.TEXT_SCORE, features.textScore)\n      richDataRecord\n        .setFeatureValue[JBoolean](RecapFeatures.HAS_VISIBLE_LINK, features.hasVisibleLink)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.HAS_LINK, features.hasLink)\n      richDataRecord.setFeatureValue[JBoolean](RecapFeatures.HAS_TREND, features.hasTrend)\n      richDataRecord.setFeatureValue[JBoolean](\n        RecapFeatures.HAS_MULTIPLE_HASHTAGS_OR_TRENDS,\n        features.hasMultipleHashtagsOrTrends\n      )\n      richDataRecord.setFeatureValueFromOption(\n        RecapFeatures.FAV_COUNT_V2,\n        features.favCountV2.map(_.toDouble))\n      richDataRecord.setFeatureValueFromOption(\n        RecapFeatures.RETWEET_COUNT_V2,\n        features.retweetCountV2.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        RecapFeatures.REPLY_COUNT_V2,\n        features.replyCountV2.map(_.toDouble))\n      richDataRecord.setFeatureValueFromOption(\n        RecapFeatures.MENTIONED_SCREEN_NAMES,\n        features.mentionsList.map(_.toSet.asJava)\n      )\n      val urls = features.urlsList.getOrElse(Seq.empty)\n      richDataRecord.setFeatureValue(\n        RecapFeatures.URL_DOMAINS,\n        urls.toSet.flatMap(UrlExtractorUtil.extractDomain).asJava)\n      richDataRecord.setFeatureValue[JDouble](RecapFeatures.LINK_COUNT, urls.size.toDouble)\n      // shared features\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.WEIGHTED_FAV_COUNT,\n        features.weightedFavoriteCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.WEIGHTED_RETWEET_COUNT,\n        features.weightedRetweetCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.WEIGHTED_REPLY_COUNT,\n        features.weightedReplyCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.WEIGHTED_QUOTE_COUNT,\n        features.weightedQuoteCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.EMBEDS_IMPRESSION_COUNT_V2,\n        features.embedsImpressionCountV2.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.EMBEDS_URL_COUNT_V2,\n        features.embedsUrlCountV2.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.DECAYED_FAVORITE_COUNT,\n        features.decayedFavoriteCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.DECAYED_RETWEET_COUNT,\n        features.decayedRetweetCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.DECAYED_REPLY_COUNT,\n        features.decayedReplyCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.DECAYED_QUOTE_COUNT,\n        features.decayedQuoteCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.FAKE_FAVORITE_COUNT,\n        features.fakeFavoriteCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.FAKE_RETWEET_COUNT,\n        features.fakeRetweetCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.FAKE_REPLY_COUNT,\n        features.fakeReplyCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.FAKE_QUOTE_COUNT,\n        features.fakeQuoteCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.QUOTE_COUNT,\n        features.quoteCount.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValue[JDouble](\n        TimelinesSharedFeatures.EARLYBIRD_SCORE,\n        features.earlybirdScore\n      )\n      // safety features\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.LABEL_ABUSIVE_FLAG,\n        features.labelAbusiveFlag\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.LABEL_ABUSIVE_HI_RCL_FLAG,\n        features.labelAbusiveHiRclFlag\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.LABEL_DUP_CONTENT_FLAG,\n        features.labelDupContentFlag\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.LABEL_NSFW_HI_PRC_FLAG,\n        features.labelNsfwHiPrcFlag\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.LABEL_NSFW_HI_RCL_FLAG,\n        features.labelNsfwHiRclFlag\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.LABEL_SPAM_FLAG,\n        features.labelSpamFlag\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.LABEL_SPAM_HI_RCL_FLAG,\n        features.labelSpamHiRclFlag\n      )\n      // periscope features\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.PERISCOPE_EXISTS,\n        features.periscopeExists\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.PERISCOPE_IS_LIVE,\n        features.periscopeIsLive\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.PERISCOPE_HAS_BEEN_FEATURED,\n        features.periscopeHasBeenFeatured\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.PERISCOPE_IS_CURRENTLY_FEATURED,\n        features.periscopeIsCurrentlyFeatured\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.PERISCOPE_IS_FROM_QUALITY_SOURCE,\n        features.periscopeIsFromQualitySource\n      )\n      // misc features\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.VISIBLE_TOKEN_RATIO,\n        features.visibleTokenRatio.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.HAS_QUOTE,\n        features.hasQuote\n      )\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.IS_COMPOSER_SOURCE_CAMERA,\n        features.isComposerSourceCamera\n      )\n      // health scores\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.PREPORTED_TWEET_SCORE,\n        features.pReportedTweetScore\n      )\n      // media\n      richDataRecord.setFeatureValueFromOption(\n        TimelinesSharedFeatures.CLASSIFICATION_LABELS,\n        features.mediaClassificationInfo.map(_.toMap.asJava.asInstanceOf[JMap[String, JDouble]])\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/earlybird/InReplyToEarlybirdAdapter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.earlybird\n\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.util.DataRecordConverters.RichDataRecordWrapper\nimport com.twitter.search.common.features.{thriftscala => sc}\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.common.InReplyToTweetTimelinesSharedFeatures\nimport com.twitter.timelines.prediction.features.recap.InReplyToRecapFeatures\nimport java.lang.{Boolean => JBoolean}\nimport java.lang.{Double => JDouble}\n\nobject InReplyToEarlybirdAdapter\n    extends TimelinesMutatingAdapterBase[Option[sc.ThriftTweetFeatures]] {\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(\n    // TextFeatures\n    InReplyToTweetTimelinesSharedFeatures.WEIGHTED_FAV_COUNT,\n    InReplyToTweetTimelinesSharedFeatures.WEIGHTED_RETWEET_COUNT,\n    InReplyToTweetTimelinesSharedFeatures.WEIGHTED_REPLY_COUNT,\n    InReplyToTweetTimelinesSharedFeatures.WEIGHTED_QUOTE_COUNT,\n    InReplyToTweetTimelinesSharedFeatures.DECAYED_FAVORITE_COUNT,\n    InReplyToTweetTimelinesSharedFeatures.DECAYED_RETWEET_COUNT,\n    InReplyToTweetTimelinesSharedFeatures.DECAYED_REPLY_COUNT,\n    InReplyToTweetTimelinesSharedFeatures.DECAYED_QUOTE_COUNT,\n    InReplyToTweetTimelinesSharedFeatures.QUOTE_COUNT,\n    InReplyToTweetTimelinesSharedFeatures.HAS_QUOTE,\n    InReplyToTweetTimelinesSharedFeatures.EARLYBIRD_SCORE,\n    InReplyToRecapFeatures.PREV_USER_TWEET_ENGAGEMENT,\n    InReplyToRecapFeatures.IS_SENSITIVE,\n    InReplyToRecapFeatures.IS_AUTHOR_NEW,\n    InReplyToRecapFeatures.NUM_MENTIONS,\n    InReplyToRecapFeatures.HAS_MENTION,\n    InReplyToRecapFeatures.HAS_HASHTAG,\n    InReplyToRecapFeatures.IS_AUTHOR_NSFW,\n    InReplyToRecapFeatures.IS_AUTHOR_SPAM,\n    InReplyToRecapFeatures.IS_AUTHOR_BOT,\n    InReplyToRecapFeatures.FROM_MUTUAL_FOLLOW,\n    InReplyToRecapFeatures.USER_REP,\n    InReplyToRecapFeatures.FROM_VERIFIED_ACCOUNT,\n    InReplyToRecapFeatures.HAS_IMAGE,\n    InReplyToRecapFeatures.HAS_NEWS,\n    InReplyToRecapFeatures.HAS_VIDEO,\n    InReplyToRecapFeatures.HAS_VISIBLE_LINK,\n    InReplyToRecapFeatures.IS_OFFENSIVE,\n    InReplyToRecapFeatures.IS_REPLY,\n    InReplyToRecapFeatures.BIDIRECTIONAL_REPLY_COUNT,\n    InReplyToRecapFeatures.UNIDIRECTIONAL_REPLY_COUNT,\n    InReplyToRecapFeatures.BIDIRECTIONAL_RETWEET_COUNT,\n    InReplyToRecapFeatures.UNIDIRECTIONAL_RETWEET_COUNT,\n    InReplyToRecapFeatures.BIDIRECTIONAL_FAV_COUNT,\n    InReplyToRecapFeatures.UNIDIRECTIONAL_FAV_COUNT,\n    InReplyToRecapFeatures.CONVERSATIONAL_COUNT,\n    InReplyToRecapFeatures.REPLY_COUNT,\n    InReplyToRecapFeatures.RETWEET_COUNT,\n    InReplyToRecapFeatures.FAV_COUNT,\n    InReplyToRecapFeatures.TEXT_SCORE,\n    InReplyToRecapFeatures.FAV_COUNT_V2,\n    InReplyToRecapFeatures.RETWEET_COUNT_V2,\n    InReplyToRecapFeatures.REPLY_COUNT_V2)\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def setFeatures(\n    ebFeatures: Option[sc.ThriftTweetFeatures],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    if (ebFeatures.nonEmpty) {\n      val features = ebFeatures.get\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.WEIGHTED_FAV_COUNT,\n        features.weightedFavoriteCount.map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.WEIGHTED_RETWEET_COUNT,\n        features.weightedRetweetCount.map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.WEIGHTED_REPLY_COUNT,\n        features.weightedReplyCount.map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.WEIGHTED_QUOTE_COUNT,\n        features.weightedQuoteCount.map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.DECAYED_FAVORITE_COUNT,\n        features.decayedFavoriteCount.map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.DECAYED_RETWEET_COUNT,\n        features.decayedRetweetCount.map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.DECAYED_REPLY_COUNT,\n        features.decayedReplyCount.map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.DECAYED_QUOTE_COUNT,\n        features.decayedQuoteCount.map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.QUOTE_COUNT,\n        features.quoteCount.map(_.toDouble)\n      )\n\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToTweetTimelinesSharedFeatures.HAS_QUOTE,\n        features.hasQuote\n      )\n\n      if (features.earlybirdScore > 0)\n        richDataRecord.setFeatureValue[JDouble](\n          InReplyToTweetTimelinesSharedFeatures.EARLYBIRD_SCORE,\n          features.earlybirdScore\n        )\n\n      richDataRecord.setFeatureValue[JDouble](\n        InReplyToRecapFeatures.PREV_USER_TWEET_ENGAGEMENT,\n        features.prevUserTweetEngagement.toDouble\n      )\n\n      richDataRecord\n        .setFeatureValue[JBoolean](InReplyToRecapFeatures.IS_SENSITIVE, features.isSensitiveContent)\n      richDataRecord\n        .setFeatureValue[JBoolean](InReplyToRecapFeatures.IS_AUTHOR_NEW, features.isAuthorNew)\n      richDataRecord.setFeatureValue[JDouble](\n        InReplyToRecapFeatures.NUM_MENTIONS,\n        features.numMentions.toDouble)\n      richDataRecord\n        .setFeatureValue[JBoolean](InReplyToRecapFeatures.HAS_MENTION, (features.numMentions > 0))\n      richDataRecord\n        .setFeatureValue[JBoolean](InReplyToRecapFeatures.HAS_HASHTAG, (features.numHashtags > 0))\n      richDataRecord\n        .setFeatureValue[JBoolean](InReplyToRecapFeatures.IS_AUTHOR_NSFW, features.isAuthorNSFW)\n      richDataRecord\n        .setFeatureValue[JBoolean](InReplyToRecapFeatures.IS_AUTHOR_SPAM, features.isAuthorSpam)\n      richDataRecord\n        .setFeatureValue[JBoolean](InReplyToRecapFeatures.IS_AUTHOR_BOT, features.isAuthorBot)\n      richDataRecord.setFeatureValue[JBoolean](\n        InReplyToRecapFeatures.FROM_MUTUAL_FOLLOW,\n        features.fromMutualFollow)\n      richDataRecord.setFeatureValue[JDouble](InReplyToRecapFeatures.USER_REP, features.userRep)\n      richDataRecord.setFeatureValue[JBoolean](\n        InReplyToRecapFeatures.FROM_VERIFIED_ACCOUNT,\n        features.fromVerifiedAccount)\n      richDataRecord.setFeatureValue[JBoolean](InReplyToRecapFeatures.HAS_IMAGE, features.hasImage)\n      richDataRecord.setFeatureValue[JBoolean](InReplyToRecapFeatures.HAS_NEWS, features.hasNews)\n      richDataRecord.setFeatureValue[JBoolean](InReplyToRecapFeatures.HAS_VIDEO, features.hasVideo)\n      richDataRecord\n        .setFeatureValue[JBoolean](InReplyToRecapFeatures.HAS_VISIBLE_LINK, features.hasVisibleLink)\n      richDataRecord\n        .setFeatureValue[JBoolean](InReplyToRecapFeatures.IS_OFFENSIVE, features.isOffensive)\n      richDataRecord.setFeatureValue[JBoolean](InReplyToRecapFeatures.IS_REPLY, features.isReply)\n      richDataRecord.setFeatureValue[JDouble](\n        InReplyToRecapFeatures.BIDIRECTIONAL_REPLY_COUNT,\n        features.bidirectionalReplyCount)\n      richDataRecord.setFeatureValue[JDouble](\n        InReplyToRecapFeatures.UNIDIRECTIONAL_REPLY_COUNT,\n        features.unidirectionalReplyCount)\n      richDataRecord.setFeatureValue[JDouble](\n        InReplyToRecapFeatures.BIDIRECTIONAL_RETWEET_COUNT,\n        features.bidirectionalRetweetCount)\n      richDataRecord.setFeatureValue[JDouble](\n        InReplyToRecapFeatures.UNIDIRECTIONAL_RETWEET_COUNT,\n        features.unidirectionalRetweetCount)\n      richDataRecord.setFeatureValue[JDouble](\n        InReplyToRecapFeatures.BIDIRECTIONAL_FAV_COUNT,\n        features.bidirectionalFavCount)\n      richDataRecord.setFeatureValue[JDouble](\n        InReplyToRecapFeatures.UNIDIRECTIONAL_FAV_COUNT,\n        features.unidirectionalFavCount)\n      richDataRecord.setFeatureValue[JDouble](\n        InReplyToRecapFeatures.CONVERSATIONAL_COUNT,\n        features.conversationCount)\n      richDataRecord\n        .setFeatureValue[JDouble](InReplyToRecapFeatures.REPLY_COUNT, features.replyCount.toDouble)\n      richDataRecord.setFeatureValue[JDouble](\n        InReplyToRecapFeatures.RETWEET_COUNT,\n        features.retweetCount.toDouble)\n      richDataRecord\n        .setFeatureValue[JDouble](InReplyToRecapFeatures.FAV_COUNT, features.favCount.toDouble)\n      richDataRecord.setFeatureValue[JDouble](InReplyToRecapFeatures.TEXT_SCORE, features.textScore)\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToRecapFeatures.FAV_COUNT_V2,\n        features.favCountV2.map(_.toDouble))\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToRecapFeatures.RETWEET_COUNT_V2,\n        features.retweetCountV2.map(_.toDouble)\n      )\n      richDataRecord.setFeatureValueFromOption(\n        InReplyToRecapFeatures.REPLY_COUNT_V2,\n        features.replyCountV2.map(_.toDouble))\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/inferred_topic/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/scala/com/twitter/timelines/prediction/features/common\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/inferred_topic/InferredTopicAdapter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.inferred_topic\n\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport scala.collection.JavaConverters._\n\nobject InferredTopicAdapter extends TimelinesMutatingAdapterBase[Map[Long, Double]] {\n\n  override val getFeatureContext: FeatureContext = new FeatureContext(\n    TimelinesSharedFeatures.INFERRED_TOPIC_IDS)\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def setFeatures(\n    inferredTopicFeatures: Map[Long, Double],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    richDataRecord.setFeatureValue(\n      TimelinesSharedFeatures.INFERRED_TOPIC_IDS,\n      inferredTopicFeatures.keys.map(_.toString).toSet.asJava)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/non_ml_features/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/ml/api/constant\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/scala/com/twitter/timelines/prediction/features/common\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/timelines/author_features:thrift-java\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/non_ml_features/NonMLCandidateFeaturesAdapter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.non_ml_features\n\nimport com.twitter.ml.api.constant.SharedFeatures\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport java.lang.{Long => JLong}\n\ncase class NonMLCandidateFeatures(\n  tweetId: Long,\n  sourceTweetId: Option[Long],\n  originalAuthorId: Option[Long],\n)\n\n/**\n * define non ml features adapter to create a data record which includes many non ml features\n * e.g. predictionRequestId, userId, tweetId to be used as joined key in batch pipeline.\n */\nobject NonMLCandidateFeaturesAdapter extends TimelinesMutatingAdapterBase[NonMLCandidateFeatures] {\n\n  private val featureContext = new FeatureContext(\n    SharedFeatures.TWEET_ID,\n    // For Secondary Engagement data generation\n    TimelinesSharedFeatures.SOURCE_TWEET_ID,\n    TimelinesSharedFeatures.ORIGINAL_AUTHOR_ID,\n  )\n\n  override def getFeatureContext: FeatureContext = featureContext\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n\n  override def setFeatures(\n    nonMLCandidateFeatures: NonMLCandidateFeatures,\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    richDataRecord.setFeatureValue[JLong](SharedFeatures.TWEET_ID, nonMLCandidateFeatures.tweetId)\n    nonMLCandidateFeatures.sourceTweetId.foreach(\n      richDataRecord.setFeatureValue[JLong](TimelinesSharedFeatures.SOURCE_TWEET_ID, _))\n    nonMLCandidateFeatures.originalAuthorId.foreach(\n      richDataRecord.setFeatureValue[JLong](TimelinesSharedFeatures.ORIGINAL_AUTHOR_ID, _))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/non_ml_features/NonMLCommonFeaturesAdapter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.non_ml_features\n\nimport com.twitter.ml.api.constant.SharedFeatures\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport java.lang.{Long => JLong}\n\ncase class NonMLCommonFeatures(\n  userId: Long,\n  predictionRequestId: Option[Long],\n  servedTimestamp: Long,\n)\n\n/**\n * define non ml features adapter to create a data record which includes many non ml features\n * e.g. predictionRequestId, userId, tweetId to be used as joined key in batch pipeline.\n */\nobject NonMLCommonFeaturesAdapter extends TimelinesMutatingAdapterBase[NonMLCommonFeatures] {\n\n  private val featureContext = new FeatureContext(\n    SharedFeatures.USER_ID,\n    TimelinesSharedFeatures.PREDICTION_REQUEST_ID,\n    TimelinesSharedFeatures.SERVED_TIMESTAMP,\n  )\n\n  override def getFeatureContext: FeatureContext = featureContext\n\n  override val commonFeatures: Set[Feature[_]] = Set(\n    SharedFeatures.USER_ID,\n    TimelinesSharedFeatures.PREDICTION_REQUEST_ID,\n    TimelinesSharedFeatures.SERVED_TIMESTAMP,\n  )\n\n  override def setFeatures(\n    nonMLCommonFeatures: NonMLCommonFeatures,\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    richDataRecord.setFeatureValue[JLong](SharedFeatures.USER_ID, nonMLCommonFeatures.userId)\n    nonMLCommonFeatures.predictionRequestId.foreach(\n      richDataRecord.setFeatureValue[JLong](TimelinesSharedFeatures.PREDICTION_REQUEST_ID, _))\n    richDataRecord.setFeatureValue[JLong](\n      TimelinesSharedFeatures.SERVED_TIMESTAMP,\n      nonMLCommonFeatures.servedTimestamp)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/offline_aggregates/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"timelines/data_processing/ml_util/aggregation_framework/conversion:for-timelines\",\n        \"timelineservice/common/src/main/scala/com/twitter/timelineservice/model\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/offline_aggregates/PassThroughAdapter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.offline_aggregates\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.IRecordOneToOneAdapter\n\nobject PassThroughAdapter extends IRecordOneToOneAdapter[Seq[DataRecord]] {\n  override def adaptToDataRecord(record: Seq[DataRecord]): DataRecord =\n    record.headOption.getOrElse(new DataRecord)\n\n  // This is not necessary and should not be used.\n  override def getFeatureContext = ???\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/offline_aggregates/SparseAggregatesToDenseAdapter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.offline_aggregates\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.conversion.CombineCountsPolicy\nimport com.twitter.timelines.prediction.common.adapters.TimelinesIRecordAdapter\n\nclass SparseAggregatesToDenseAdapter(policy: CombineCountsPolicy)\n    extends TimelinesIRecordAdapter[Seq[DataRecord]] {\n\n  override def setFeatures(input: Seq[DataRecord], mutableDataRecord: RichDataRecord): Unit =\n    policy.defaultMergeRecord(mutableDataRecord.getRecord, input.toList)\n\n  override val getFeatureContext: FeatureContext =\n    new FeatureContext(policy.outputFeaturesPostMerge.toSeq: _*)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/twhin_embeddings/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/ml/api:data-scala\",\n        \"src/thrift/com/twitter/ml/api:embedding-java\",\n        \"src/thrift/com/twitter/ml/api:embedding-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/twhin_embeddings/TwhinEmbeddingsAdapter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.twhin_embeddings\n\nimport com.twitter.ml.api.DataType\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.ml.api.util.ScalaToJavaDataRecordConversions\nimport com.twitter.ml.api.{thriftscala => ml}\nimport com.twitter.timelines.prediction.common.adapters.TimelinesMutatingAdapterBase\n\nsealed trait TwhinEmbeddingsAdapter extends TimelinesMutatingAdapterBase[Option[ml.FloatTensor]] {\n  def twhinEmbeddingsFeature: Feature.Tensor\n\n  override def getFeatureContext: FeatureContext = new FeatureContext(\n    twhinEmbeddingsFeature\n  )\n\n  override def setFeatures(\n    embedding: Option[ml.FloatTensor],\n    richDataRecord: RichDataRecord\n  ): Unit = {\n    embedding.foreach { floatTensor =>\n      richDataRecord.setFeatureValue(\n        twhinEmbeddingsFeature,\n        ScalaToJavaDataRecordConversions.scalaTensor2Java(\n          ml.GeneralTensor\n            .FloatTensor(floatTensor)))\n    }\n  }\n}\n\nobject TwhinEmbeddingsFeatures {\n  val TwhinAuthorFollowEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"original_author.twhin.tw_hi_n.author_follow_as_float_tensor\",\n    DataType.FLOAT\n  )\n\n  val TwhinUserEngagementEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.twhin.tw_hi_n.user_engagement_as_float_tensor\",\n    DataType.FLOAT\n  )\n\n  val TwhinUserFollowEmbeddingsFeature: Feature.Tensor = new Feature.Tensor(\n    \"user.twhin.tw_hi_n.user_follow_as_float_tensor\",\n    DataType.FLOAT\n  )\n}\n\nobject TwhinAuthorFollowEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinAuthorFollowEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set.empty\n}\n\nobject TwhinUserEngagementEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinUserEngagementEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set(twhinEmbeddingsFeature)\n}\n\nobject TwhinUserFollowEmbeddingsAdapter extends TwhinEmbeddingsAdapter {\n  override val twhinEmbeddingsFeature: Feature.Tensor =\n    TwhinEmbeddingsFeatures.TwhinUserFollowEmbeddingsFeature\n\n  override val commonFeatures: Set[Feature[_]] = Set(twhinEmbeddingsFeature)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/offline_aggregates/AggregateFeatureInfo.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates\n\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateType.AggregateType\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.TypedAggregateGroup\nimport scala.jdk.CollectionConverters.asJavaIterableConverter\n\n// A helper class deriving aggregate feature info from the given configuration parameters.\nclass AggregateFeatureInfo(\n  val aggregateGroups: Set[AggregateGroup],\n  val aggregateType: AggregateType) {\n\n  private val typedAggregateGroups = aggregateGroups.flatMap(_.buildTypedAggregateGroups()).toList\n\n  val featureContext: FeatureContext =\n    new FeatureContext(\n      (typedAggregateGroups.flatMap(_.allOutputFeatures) ++\n        typedAggregateGroups.flatMap(_.allOutputKeys) ++\n        Seq(TypedAggregateGroup.timestampFeature)).asJava)\n\n  val feature: BaseAggregateRootFeature =\n    AggregateFeatureInfo.pickFeature(aggregateType)\n}\n\nobject AggregateFeatureInfo {\n  val features: Set[BaseAggregateRootFeature] =\n    Set(PartAAggregateRootFeature, PartBAggregateRootFeature)\n\n  def pickFeature(aggregateType: AggregateType): BaseAggregateRootFeature = {\n    val filtered = features.filter(_.aggregateTypes.contains(aggregateType))\n    require(\n      filtered.size == 1,\n      \"requested AggregateType must be backed by exactly one physical store.\")\n    filtered.head\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/offline_aggregates/AggregateFeaturesToDecodeWithMetadata.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates\n\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.timelinemixer.injection.repository.uss.VersionedAggregateFeaturesDecoder\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.timelines.aggregate_interactions.thriftjava.UserAggregateInteractions\nimport com.twitter.timelines.aggregate_interactions.v17.thriftjava.{\n  UserAggregateInteractions => V17UserAggregateInteractions\n}\nimport com.twitter.timelines.aggregate_interactions.v1.thriftjava.{\n  UserAggregateInteractions => V1UserAggregateInteractions\n}\nimport com.twitter.timelines.suggests.common.dense_data_record.thriftjava.DenseCompactDataRecord\nimport com.twitter.timelines.suggests.common.dense_data_record.thriftscala.DenseFeatureMetadata\nimport java.lang.{Long => JLong}\nimport java.util.Collections\nimport java.util.{Map => JMap}\n\nprivate[offline_aggregates] case class AggregateFeaturesToDecodeWithMetadata(\n  metadataOpt: Option[DenseFeatureMetadata],\n  aggregates: UserAggregateInteractions) {\n  def toDataRecord(dr: DenseCompactDataRecord): DataRecord =\n    VersionedAggregateFeaturesDecoder.fromJDenseCompact(\n      metadataOpt,\n      dr.versionId,\n      NullStatsReceiver,\n      s\"V${dr.versionId}\"\n    )(dr)\n\n  def userAggregatesOpt: Option[DenseCompactDataRecord] = {\n    aggregates.getSetField match {\n      case UserAggregateInteractions._Fields.V17 =>\n        Option(aggregates.getV17.user_aggregates)\n      case _ =>\n        None\n    }\n  }\n\n  def userAuthorAggregates = extract(_.user_author_aggregates)\n  def userEngagerAggregates = extract(_.user_engager_aggregates)\n  def userMentionAggregates = extract(_.user_mention_aggregates)\n  def userOriginalAuthorAggregates = extract(_.user_original_author_aggregates)\n  def userRequestDowAggregates = extract(_.user_request_dow_aggregates)\n  def userRequestHourAggregates = extract(_.user_request_hour_aggregates)\n  def rectweetUserSimclustersTweetAggregates = extract(_.rectweet_user_simclusters_tweet_aggregates)\n  def userTwitterListAggregates = extract(_.user_list_aggregates)\n  def userTopicAggregates = extract(_.user_topic_aggregates)\n  def userInferredTopicAggregates = extract(_.user_inferred_topic_aggregates)\n  def userMediaUnderstandingAnnotationAggregates = extract(\n    _.user_media_understanding_annotation_aggregates)\n\n  private def extract[T](\n    v17Fn: V17UserAggregateInteractions => JMap[JLong, DenseCompactDataRecord]\n  ): JMap[JLong, DenseCompactDataRecord] = {\n    aggregates.getSetField match {\n      case UserAggregateInteractions._Fields.V17 =>\n        v17Fn(aggregates.getV17)\n      case _ =>\n        Collections.emptyMap()\n    }\n  }\n}\n\nobject AggregateFeaturesToDecodeWithMetadata {\n  val empty = new AggregateFeaturesToDecodeWithMetadata(\n    None,\n    UserAggregateInteractions.v1(new V1UserAggregateInteractions()))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/offline_aggregates/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/adapters/offline_aggregates\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"servo/repo/src/main/scala\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/request_context\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/scala/com/twitter/timelines/prediction/common/aggregates\",\n        \"src/scala/com/twitter/timelines/util\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/timelines/suggests/common:data_record_metadata-scala\",\n        \"src/thrift/com/twitter/timelines/suggests/common:dense_data_record-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n        \"src/thrift/com/twitter/user_session_store:thrift-java\",\n        \"stitch/stitch-core\",\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/repository/uss:versioned-aggregate-features-decoder\",\n        \"timelines/data_processing/jobs/timeline_ranking_user_features:mini\",\n        \"timelines/data_processing/ml_util/aggregation_framework:common_types\",\n        \"timelines/data_processing/ml_util/aggregation_framework/conversion:for-timelines\",\n        \"timelineservice/common/src/main/scala/com/twitter/timelineservice/model\",\n        \"user_session_store/src/main/scala/com/twitter/user_session_store\",\n        \"util/util-core\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/offline_aggregates/BaseAggregateQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.repository.Repository\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.aggregate_interactions.thriftjava.UserAggregateInteractions\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateType.AggregateType\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.StoreConfig\nimport com.twitter.timelines.suggests.common.dense_data_record.thriftscala.DenseFeatureMetadata\nimport com.twitter.user_session_store.thriftjava.UserSession\nimport com.twitter.util.Future\n\nabstract class BaseAggregateQueryFeatureHydrator(\n  featureRepository: Repository[Long, Option[UserSession]],\n  metadataRepository: Repository[Int, Option[DenseFeatureMetadata]],\n  feature: Feature[PipelineQuery, Option[AggregateFeaturesToDecodeWithMetadata]])\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val viewerId = query.getRequiredUserId\n\n    Stitch.callFuture(\n      featureRepository(viewerId)\n        .flatMap { userSession: Option[UserSession] =>\n          val featuresWithMetadata: Option[Future[AggregateFeaturesToDecodeWithMetadata]] =\n            userSession\n              .flatMap(decodeUserSession(_))\n\n          featuresWithMetadata\n            .map { fu: Future[AggregateFeaturesToDecodeWithMetadata] => fu.map(Some(_)) }\n            .getOrElse(Future.None)\n            .map { value =>\n              FeatureMapBuilder()\n                .add(feature, value)\n                .build()\n            }\n        }\n    )\n  }\n\n  private def decodeUserSession(\n    session: UserSession\n  ): Option[Future[AggregateFeaturesToDecodeWithMetadata]] = {\n    Option(session.user_aggregate_interactions).flatMap { aggregates =>\n      aggregates.getSetField match {\n        case UserAggregateInteractions._Fields.V17 =>\n          Some(\n            getAggregateFeaturesWithMetadata(\n              aggregates.getV17.user_aggregates.versionId,\n              UserAggregateInteractions.v17(aggregates.getV17))\n          )\n        case _ =>\n          None\n      }\n    }\n  }\n\n  private def getAggregateFeaturesWithMetadata(\n    versionId: Int,\n    userAggregateInteractions: UserAggregateInteractions,\n  ): Future[AggregateFeaturesToDecodeWithMetadata] = {\n    metadataRepository(versionId)\n      .map(AggregateFeaturesToDecodeWithMetadata(_, userAggregateInteractions))\n  }\n}\n\ntrait BaseAggregateRootFeature\n    extends Feature[PipelineQuery, Option[AggregateFeaturesToDecodeWithMetadata]] {\n  def aggregateStores: Set[StoreConfig[_]]\n\n  lazy val aggregateTypes: Set[AggregateType] = aggregateStores.map(_.aggregateType)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/offline_aggregates/BaseEdgeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.IRecordOneToOneAdapter\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateType.AggregateType\nimport com.twitter.timelines.suggests.common.dense_data_record.thriftjava.DenseCompactDataRecord\nimport java.lang.{Long => JLong}\nimport java.util.{Map => JMap}\n\nabstract case class BaseEdgeAggregateFeature(\n  aggregateGroups: Set[AggregateGroup],\n  aggregateType: AggregateType,\n  extractMapFn: AggregateFeaturesToDecodeWithMetadata => JMap[JLong, DenseCompactDataRecord],\n  adapter: IRecordOneToOneAdapter[Seq[DataRecord]],\n  getSecondaryKeysFn: CandidateWithFeatures[TweetCandidate] => Seq[Long])\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord\n\n  private val rootFeatureInfo = new AggregateFeatureInfo(aggregateGroups, aggregateType)\n  val featureContext: FeatureContext = rootFeatureInfo.featureContext\n  val rootFeature: BaseAggregateRootFeature = rootFeatureInfo.feature\n}\n\ntrait BaseEdgeAggregateFeatureHydrator\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  def aggregateFeatures: Set[BaseEdgeAggregateFeature]\n\n  override def features = aggregateFeatures.asInstanceOf[Set[Feature[_, _]]]\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offload {\n    val featureMapBuilders: Seq[FeatureMapBuilder] =\n      for (_ <- candidates) yield FeatureMapBuilder()\n\n    aggregateFeatures.foreach { feature =>\n      val featureValues = hydrateAggregateFeature(query, candidates, feature)\n      (featureMapBuilders zip featureValues).foreach {\n        case (featureMapBuilder, featureValue) => featureMapBuilder.add(feature, featureValue)\n      }\n    }\n\n    featureMapBuilders.map(_.build())\n  }\n\n  private def hydrateAggregateFeature(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n    feature: BaseEdgeAggregateFeature\n  ): Seq[DataRecord] = {\n    val rootFeature = feature.rootFeature\n    val extractMapFn = feature.extractMapFn\n    val featureContext = feature.featureContext\n    val secondaryIds: Seq[Seq[Long]] = candidates.map(feature.getSecondaryKeysFn)\n\n    val featuresToDecodeWithMetadata = query.features\n      .flatMap(_.getOrElse(rootFeature, None))\n      .getOrElse(AggregateFeaturesToDecodeWithMetadata.empty)\n\n    // Decode the DenseCompactDataRecords into DataRecords for each required secondary id.\n    val decoded: Map[Long, DataRecord] = Utils.selectAndTransform(\n      secondaryIds.flatten.distinct,\n      featuresToDecodeWithMetadata.toDataRecord,\n      extractMapFn(featuresToDecodeWithMetadata)\n    )\n\n    // Remove unnecessary features in-place. This is safe because the underlying DataRecords\n    // are unique and have just been generated in the previous step.\n    decoded.values.foreach(Utils.filterDataRecord(_, featureContext))\n\n    // Put features into the FeatureMapBuilders\n    secondaryIds.map { ids =>\n      val dataRecords = ids.flatMap(decoded.get)\n      feature.adapter.adaptToDataRecord(dataRecords)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/offline_aggregates/EdgeAggregateFeatures.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.offline_aggregates.PassThroughAdapter\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.offline_aggregates.SparseAggregatesToDenseAdapter\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MentionScreenNameFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.TSPInferredTopicFeature\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateType\nimport com.twitter.timelines.prediction.common.aggregates.TimelinesAggregationConfig\nimport com.twitter.timelines.prediction.common.aggregates.TimelinesAggregationConfig.CombineCountPolicies\n\nobject EdgeAggregateFeatures {\n\n  object UserAuthorAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        aggregateGroups = TimelinesAggregationConfig.userAuthorAggregatesV2 ++ Set(\n          TimelinesAggregationConfig.userAuthorAggregatesV5,\n          TimelinesAggregationConfig.tweetSourceUserAuthorAggregatesV1,\n          TimelinesAggregationConfig.twitterWideUserAuthorAggregates\n        ),\n        aggregateType = AggregateType.UserAuthor,\n        extractMapFn = _.userAuthorAggregates,\n        adapter = PassThroughAdapter,\n        getSecondaryKeysFn = _.features.getOrElse(AuthorIdFeature, None).toSeq\n      )\n\n  object UserOriginalAuthorAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        aggregateGroups = Set(TimelinesAggregationConfig.userOriginalAuthorAggregatesV1),\n        aggregateType = AggregateType.UserOriginalAuthor,\n        extractMapFn = _.userOriginalAuthorAggregates,\n        adapter = PassThroughAdapter,\n        getSecondaryKeysFn = candidate =>\n          CandidatesUtil.getOriginalAuthorId(candidate.features).toSeq\n      )\n\n  object UserTopicAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        aggregateGroups = Set(\n          TimelinesAggregationConfig.userTopicAggregates,\n          TimelinesAggregationConfig.userTopicAggregatesV2,\n        ),\n        aggregateType = AggregateType.UserTopic,\n        extractMapFn = _.userTopicAggregates,\n        adapter = PassThroughAdapter,\n        getSecondaryKeysFn = candidate =>\n          candidate.features.getOrElse(TopicIdSocialContextFeature, None).toSeq\n      )\n\n  object UserMentionAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        aggregateGroups = Set(TimelinesAggregationConfig.userMentionAggregates),\n        aggregateType = AggregateType.UserMention,\n        extractMapFn = _.userMentionAggregates,\n        adapter = new SparseAggregatesToDenseAdapter(CombineCountPolicies.MentionCountsPolicy),\n        getSecondaryKeysFn = candidate =>\n          candidate.features.getOrElse(MentionScreenNameFeature, Seq.empty).map(_.hashCode.toLong)\n      )\n\n  object UserInferredTopicAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        aggregateGroups = Set(\n          TimelinesAggregationConfig.userInferredTopicAggregates,\n        ),\n        aggregateType = AggregateType.UserInferredTopic,\n        extractMapFn = _.userInferredTopicAggregates,\n        adapter = new SparseAggregatesToDenseAdapter(\n          CombineCountPolicies.UserInferredTopicCountsPolicy),\n        getSecondaryKeysFn = candidate =>\n          candidate.features.getOrElse(TSPInferredTopicFeature, Map.empty[Long, Double]).keys.toSeq\n      )\n\n  object UserInferredTopicAggregateV2Feature\n      extends BaseEdgeAggregateFeature(\n        aggregateGroups = Set(\n          TimelinesAggregationConfig.userInferredTopicAggregatesV2\n        ),\n        aggregateType = AggregateType.UserInferredTopic,\n        extractMapFn = _.userInferredTopicAggregates,\n        adapter = new SparseAggregatesToDenseAdapter(\n          CombineCountPolicies.UserInferredTopicV2CountsPolicy),\n        getSecondaryKeysFn = candidate =>\n          candidate.features.getOrElse(TSPInferredTopicFeature, Map.empty[Long, Double]).keys.toSeq\n      )\n\n  object UserMediaUnderstandingAnnotationAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        aggregateGroups = Set(\n          TimelinesAggregationConfig.userMediaUnderstandingAnnotationAggregates),\n        aggregateType = AggregateType.UserMediaUnderstandingAnnotation,\n        extractMapFn = _.userMediaUnderstandingAnnotationAggregates,\n        adapter = new SparseAggregatesToDenseAdapter(\n          CombineCountPolicies.UserMediaUnderstandingAnnotationCountsPolicy),\n        getSecondaryKeysFn = candidate =>\n          CandidatesUtil.getMediaUnderstandingAnnotationIds(candidate.features)\n      )\n\n  object UserEngagerAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        aggregateGroups = Set(TimelinesAggregationConfig.userEngagerAggregates),\n        aggregateType = AggregateType.UserEngager,\n        extractMapFn = _.userEngagerAggregates,\n        adapter = new SparseAggregatesToDenseAdapter(CombineCountPolicies.EngagerCountsPolicy),\n        getSecondaryKeysFn = candidate => CandidatesUtil.getEngagerUserIds(candidate.features)\n      )\n\n  object UserEngagerGoodClickAggregateFeature\n      extends BaseEdgeAggregateFeature(\n        aggregateGroups = Set(TimelinesAggregationConfig.userEngagerGoodClickAggregates),\n        aggregateType = AggregateType.UserEngager,\n        extractMapFn = _.userEngagerAggregates,\n        adapter = new SparseAggregatesToDenseAdapter(\n          CombineCountPolicies.EngagerGoodClickCountsPolicy),\n        getSecondaryKeysFn = candidate => CandidatesUtil.getEngagerUserIds(candidate.features)\n      )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/offline_aggregates/PartAAggregateQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TimelineAggregateMetadataRepository\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TimelineAggregatePartARepository\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.servo.repository.Repository\nimport com.twitter.timelines.data_processing.jobs.timeline_ranking_user_features.TimelinesPartAStoreRegister\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.StoreConfig\nimport com.twitter.timelines.suggests.common.dense_data_record.thriftscala.DenseFeatureMetadata\nimport com.twitter.user_session_store.thriftjava.UserSession\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject PartAAggregateRootFeature extends BaseAggregateRootFeature {\n  override val aggregateStores: Set[StoreConfig[_]] = TimelinesPartAStoreRegister.allStores\n}\n\n@Singleton\nclass PartAAggregateQueryFeatureHydrator @Inject() (\n  @Named(TimelineAggregatePartARepository)\n  repository: Repository[Long, Option[UserSession]],\n  @Named(TimelineAggregateMetadataRepository)\n  metadataRepository: Repository[Int, Option[DenseFeatureMetadata]])\n    extends BaseAggregateQueryFeatureHydrator(\n      repository,\n      metadataRepository,\n      PartAAggregateRootFeature\n    ) {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"PartAAggregateQuery\")\n\n  override val features = Set(PartAAggregateRootFeature)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/offline_aggregates/PartBAggregateQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TimelineAggregateMetadataRepository\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TimelineAggregatePartBRepository\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.DataRecordMerger\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.repository.Repository\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.data_processing.jobs.timeline_ranking_user_features.TimelinesPartBStoreRegister\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateType\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.StoreConfig\nimport com.twitter.timelines.prediction.adapters.request_context.RequestContextAdapter\nimport com.twitter.timelines.prediction.common.aggregates.TimelinesAggregationConfig\nimport com.twitter.timelines.suggests.common.dense_data_record.thriftscala.DenseFeatureMetadata\nimport com.twitter.user_session_store.thriftjava.UserSession\nimport com.twitter.util.Time\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject PartBAggregateRootFeature extends BaseAggregateRootFeature {\n  override val aggregateStores: Set[StoreConfig[_]] = TimelinesPartBStoreRegister.allStores\n}\n\nobject UserAggregateFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass PartBAggregateQueryFeatureHydrator @Inject() (\n  @Named(TimelineAggregatePartBRepository)\n  repository: Repository[Long, Option[UserSession]],\n  @Named(TimelineAggregateMetadataRepository)\n  metadataRepository: Repository[Int, Option[DenseFeatureMetadata]])\n    extends BaseAggregateQueryFeatureHydrator(\n      repository,\n      metadataRepository,\n      PartBAggregateRootFeature\n    ) {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"PartBAggregateQuery\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(PartBAggregateRootFeature, UserAggregateFeature)\n\n  private val userAggregateFeatureInfo = new AggregateFeatureInfo(\n    aggregateGroups = Set(\n      TimelinesAggregationConfig.userAggregatesV2,\n      TimelinesAggregationConfig.userAggregatesV5Continuous,\n      TimelinesAggregationConfig.userAggregatesV6,\n      TimelinesAggregationConfig.twitterWideUserAggregates,\n    ),\n    aggregateType = AggregateType.User\n  )\n\n  private val userHourAggregateFeatureInfo = new AggregateFeatureInfo(\n    aggregateGroups = Set(\n      TimelinesAggregationConfig.userRequestHourAggregates,\n    ),\n    aggregateType = AggregateType.UserRequestHour\n  )\n\n  private val userDowAggregateFeatureInfo = new AggregateFeatureInfo(\n    aggregateGroups = Set(\n      TimelinesAggregationConfig.userRequestDowAggregates\n    ),\n    aggregateType = AggregateType.UserRequestDow\n  )\n\n  require(\n    userAggregateFeatureInfo.feature == PartBAggregateRootFeature,\n    \"UserAggregates feature must be provided by the PartB data source.\")\n  require(\n    userHourAggregateFeatureInfo.feature == PartBAggregateRootFeature,\n    \"UserRequstHourAggregates feature must be provided by the PartB data source.\")\n  require(\n    userDowAggregateFeatureInfo.feature == PartBAggregateRootFeature,\n    \"UserRequestDowAggregates feature must be provided by the PartB data source.\")\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    // Hydrate TimelineAggregatePartBFeature and UserAggregateFeature sequentially.\n    super.hydrate(query).map { featureMap =>\n      val time: Time = Time.now\n      val hourOfDay = RequestContextAdapter.hourFromTimestamp(time.inMilliseconds)\n      val dayOfWeek = RequestContextAdapter.dowFromTimestamp(time.inMilliseconds)\n\n      val dr = featureMap\n        .get(PartBAggregateRootFeature).map { featuresWithMetadata =>\n          val userAggregatesDr =\n            featuresWithMetadata.userAggregatesOpt\n              .map(featuresWithMetadata.toDataRecord)\n          val userRequestHourAggregatesDr =\n            Option(featuresWithMetadata.userRequestHourAggregates.get(hourOfDay))\n              .map(featuresWithMetadata.toDataRecord)\n          val userRequestDowAggregatesDr =\n            Option(featuresWithMetadata.userRequestDowAggregates.get(dayOfWeek))\n              .map(featuresWithMetadata.toDataRecord)\n\n          dropUnknownFeatures(userAggregatesDr, userAggregateFeatureInfo.featureContext)\n\n          dropUnknownFeatures(\n            userRequestHourAggregatesDr,\n            userHourAggregateFeatureInfo.featureContext)\n\n          dropUnknownFeatures(\n            userRequestDowAggregatesDr,\n            userDowAggregateFeatureInfo.featureContext)\n\n          mergeDataRecordOpts(\n            userAggregatesDr,\n            userRequestHourAggregatesDr,\n            userRequestDowAggregatesDr)\n\n        }.getOrElse(new DataRecord())\n\n      featureMap + (UserAggregateFeature, dr)\n    }\n  }\n\n  private val drMerger = new DataRecordMerger\n  private def mergeDataRecordOpts(dataRecordOpts: Option[DataRecord]*): DataRecord =\n    dataRecordOpts.flatten.foldLeft(new DataRecord) { (l, r) =>\n      drMerger.merge(l, r)\n      l\n    }\n\n  private def dropUnknownFeatures(\n    dataRecordOpt: Option[DataRecord],\n    featureContext: FeatureContext\n  ): Unit =\n    dataRecordOpt.foreach(new RichDataRecord(_, featureContext).dropUnknownFeatures())\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/offline_aggregates/Phase1EdgeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates.EdgeAggregateFeatures._\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass Phase1EdgeAggregateFeatureHydrator @Inject() extends BaseEdgeAggregateFeatureHydrator {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"Phase1EdgeAggregate\")\n\n  override val aggregateFeatures: Set[BaseEdgeAggregateFeature] = Set(\n    UserAuthorAggregateFeature,\n    UserOriginalAuthorAggregateFeature,\n    UserMentionAggregateFeature\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/offline_aggregates/Phase2EdgeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates\n\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserEngagerAggregateFeature\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserEngagerGoodClickAggregateFeature\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserInferredTopicAggregateFeature\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserInferredTopicAggregateV2Feature\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserMediaUnderstandingAnnotationAggregateFeature\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates.EdgeAggregateFeatures.UserTopicAggregateFeature\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass Phase2EdgeAggregateFeatureHydrator @Inject() extends BaseEdgeAggregateFeatureHydrator {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"Phase2EdgeAggregate\")\n\n  override val aggregateFeatures: Set[BaseEdgeAggregateFeature] =\n    Set(\n      UserEngagerAggregateFeature,\n      UserEngagerGoodClickAggregateFeature,\n      UserInferredTopicAggregateFeature,\n      UserInferredTopicAggregateV2Feature,\n      UserTopicAggregateFeature,\n      UserMediaUnderstandingAnnotationAggregateFeature\n    )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/offline_aggregates/Utils.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.offline_aggregates\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.RichDataRecord\nimport com.twitter.timelines.suggests.common.dense_data_record.thriftjava.DenseCompactDataRecord\n\nprivate[offline_aggregates] object Utils {\n\n  /**\n   * Selects only those values in map that correspond to the keys in ids and apply the provided\n   * transform to the selected values. This is a convenience method for use by Timelines Aggregation\n   * Framework based features.\n   *\n   * @param idsToSelect The set of ids to extract values for.\n   * @param transform A transform to apply to the selected values.\n   * @param map Map[Long, DenseCompactDataRecord]\n   */\n  def selectAndTransform(\n    idsToSelect: Seq[Long],\n    transform: DenseCompactDataRecord => DataRecord,\n    map: java.util.Map[java.lang.Long, DenseCompactDataRecord],\n  ): Map[Long, DataRecord] = {\n    val filtered: Seq[(Long, DataRecord)] =\n      for {\n        id <- idsToSelect if map.containsKey(id)\n      } yield {\n        id -> transform(map.get(id))\n      }\n    filtered.toMap\n  }\n\n  def filterDataRecord(dr: DataRecord, featureContext: FeatureContext): Unit = {\n    new RichDataRecord(dr, featureContext).dropUnknownFeatures()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"servo/repo/src/main/scala\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/ml/api/constant\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/scala/com/twitter/timelines/prediction/common/aggregates/real_time:base-config\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/wtf/real_time_interaction_graph:wtf-real_time_interaction_graph-thrift-java\",\n        \"stitch/stitch-core\",\n        \"timelines/data_processing/ml_util/aggregation_framework:common_types\",\n        \"timelines/data_processing/ml_util/aggregation_framework/heron\",\n        \"util/util-core\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/BaseRealTimeAggregateBulkCandidateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates\n\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\n\ntrait BaseRealTimeAggregateBulkCandidateFeatureHydrator[K]\n    extends BulkCandidateFeatureHydrator[PipelineQuery, TweetCandidate]\n    with BaseRealtimeAggregateHydrator[K] {\n\n  val outputFeature: DataRecordInAFeature[TweetCandidate]\n\n  override def features: Set[Feature[_, _]] = Set(outputFeature)\n\n  override lazy val statScope: String = identifier.toString\n\n  def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[K]]\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = OffloadFuturePools.offloadFuture {\n    val possiblyKeys = keysFromQueryAndCandidates(query, candidates)\n    fetchAndConstructDataRecords(possiblyKeys).map { dataRecords =>\n      dataRecords.map { dataRecord =>\n        FeatureMapBuilder().add(outputFeature, dataRecord).build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/BaseRealTimeAggregateQueryFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\n\ntrait BaseRealTimeAggregateQueryFeatureHydrator[K]\n    extends QueryFeatureHydrator[PipelineQuery]\n    with BaseRealtimeAggregateHydrator[K] {\n\n  val outputFeature: DataRecordInAFeature[PipelineQuery]\n\n  override def features: Set[Feature[_, _]] = Set(outputFeature)\n\n  override lazy val statScope: String = identifier.toString\n\n  def keysFromQueryAndCandidates(\n    query: PipelineQuery\n  ): Option[K]\n\n  override def hydrate(\n    query: PipelineQuery\n  ): Stitch[FeatureMap] = OffloadFuturePools.offloadFuture {\n    val possiblyKeys = keysFromQueryAndCandidates(query)\n    fetchAndConstructDataRecords(Seq(possiblyKeys)).map { dataRecords =>\n      FeatureMapBuilder()\n        .add(outputFeature, dataRecords.head)\n        .build()\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/BaseRealtimeAggregateHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates\n\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates.BaseRealtimeAggregateHydrator._\nimport com.twitter.home_mixer.util.DataRecordUtil\nimport com.twitter.home_mixer.util.ObservedKeyValueResultHandler\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.DataRecordMerger\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.constant.SharedFeatures\nimport com.twitter.ml.api.util.SRichDataRecord\nimport com.twitter.ml.api.{Feature => MLApiFeature}\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.servo.keyvalue.KeyValueResult\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport com.twitter.util.Try\nimport java.lang.{Double => JDouble}\nimport scala.collection.JavaConverters._\n\ntrait BaseRealtimeAggregateHydrator[K] extends ObservedKeyValueResultHandler {\n\n  val client: ReadCache[K, DataRecord]\n\n  val aggregateGroups: Seq[AggregateGroup]\n\n  val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map.empty\n\n  private lazy val typedAggregateGroupsList = aggregateGroups.map(_.buildTypedAggregateGroups())\n\n  private lazy val featureContexts: Seq[FeatureContext] = typedAggregateGroupsList.map {\n    typedAggregateGroups =>\n      new FeatureContext(\n        (SharedFeatures.TIMESTAMP +: typedAggregateGroups.flatMap(_.allOutputFeatures)).asJava\n      )\n  }\n\n  private lazy val aggregateFeaturesRenameMap: Map[MLApiFeature[_], MLApiFeature[_]] = {\n    val prefixes: Seq[Option[String]] = aggregateGroups.map(aggregateGroupToPrefix.get)\n\n    typedAggregateGroupsList\n      .zip(prefixes).map {\n        case (typedAggregateGroups, prefix) =>\n          if (prefix.nonEmpty)\n            typedAggregateGroups\n              .map {\n                _.outputFeaturesToRenamedOutputFeatures(prefix.get)\n              }.reduce(_ ++ _)\n          else\n            Map.empty[MLApiFeature[_], MLApiFeature[_]]\n      }.reduce(_ ++ _)\n  }\n\n  private lazy val renamedFeatureContexts: Seq[FeatureContext] =\n    typedAggregateGroupsList.map { typedAggregateGroups =>\n      val renamedAllOutputFeatures = typedAggregateGroups.flatMap(_.allOutputFeatures).map {\n        feature => aggregateFeaturesRenameMap.getOrElse(feature, feature)\n      }\n\n      new FeatureContext(renamedAllOutputFeatures.asJava)\n    }\n\n  private lazy val decays: Seq[TimeDecay] = typedAggregateGroupsList.map { typedAggregateGroups =>\n    RealTimeAggregateTimeDecay(\n      typedAggregateGroups.flatMap(_.continuousFeatureIdsToHalfLives).toMap)\n      .apply(_, _)\n  }\n\n  private val drMerger = new DataRecordMerger\n\n  private def postTransformer(dataRecord: Try[Option[DataRecord]]): Try[DataRecord] = {\n    dataRecord.map {\n      case Some(dr) =>\n        val newDr = new DataRecord()\n        featureContexts.zip(renamedFeatureContexts).zip(decays).foreach {\n          case ((featureContext, renamedFeatureContext), decay) =>\n            val decayedDr = applyDecay(dr, featureContext, decay)\n            val renamedDr = DataRecordUtil.applyRename(\n              dataRecord = decayedDr,\n              featureContext,\n              renamedFeatureContext,\n              aggregateFeaturesRenameMap)\n            drMerger.merge(newDr, renamedDr)\n        }\n        newDr\n      case _ => new DataRecord\n    }\n  }\n\n  def fetchAndConstructDataRecords(possiblyKeys: Seq[Option[K]]): Future[Seq[Try[DataRecord]]] = {\n    val keys = possiblyKeys.flatten\n\n    val response: Future[KeyValueResult[K, DataRecord]] =\n      if (keys.isEmpty) Future.value(KeyValueResult.empty)\n      else {\n        val batchResponses = keys\n          .grouped(RequestBatchSize)\n          .map(keyGroup => client.get(keyGroup))\n          .toSeq\n\n        Future.collect(batchResponses).map(_.reduce(_ ++ _))\n      }\n\n    response.map { result =>\n      possiblyKeys.map { possiblyKey =>\n        val value = observedGet(key = possiblyKey, keyValueResult = result)\n        postTransformer(value)\n      }\n    }\n  }\n}\n\nobject BaseRealtimeAggregateHydrator {\n  private val RequestBatchSize = 5\n\n  type TimeDecay = scala.Function2[com.twitter.ml.api.DataRecord, scala.Long, scala.Unit]\n\n  private def applyDecay(\n    dataRecord: DataRecord,\n    featureContext: FeatureContext,\n    decay: TimeDecay\n  ): DataRecord = {\n    def time: Long = Time.now.inMillis\n\n    val richFullDr = new SRichDataRecord(dataRecord, featureContext)\n    val richNewDr = new SRichDataRecord(new DataRecord, featureContext)\n    val featureIterator = featureContext.iterator()\n    featureIterator.forEachRemaining { feature =>\n      if (richFullDr.hasFeature(feature)) {\n        val typedFeature = feature.asInstanceOf[MLApiFeature[JDouble]]\n        richNewDr.setFeatureValue(typedFeature, richFullDr.getFeatureValue(typedFeature))\n      }\n    }\n    val resultDr = richNewDr.getRecord\n    decay(resultDr, time)\n    resultDr\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/EngagementsReceivedByAuthorRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.EngagementsReceivedByAuthorCache\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject EngagementsReceivedByAuthorRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass EngagementsReceivedByAuthorRealTimeAggregateFeatureHydrator @Inject() (\n  @Named(EngagementsReceivedByAuthorCache) override val client: ReadCache[Long, DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends BaseRealTimeAggregateBulkCandidateFeatureHydrator[Long] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"EngagementsReceivedByAuthorRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    EngagementsReceivedByAuthorRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    authorEngagementRealTimeAggregatesProd,\n    authorShareEngagementsRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    authorShareEngagementsRealTimeAggregates -> \"original_author.timelines.author_share_engagements_real_time_aggregates.\"\n  )\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] =\n    candidates.map(candidate => CandidatesUtil.getOriginalAuthorId(candidate.features))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/RealTimeAggregateTimeDecay.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.constant.SharedFeatures.TIMESTAMP\nimport com.twitter.util.Duration\n\n/**\n * The default TimeDecay implementation for real time aggregates.\n *\n * @param featureIdToHalfLife A precomputed map from aggregate feature ids to their half lives.\n * @param timestampFeatureId A discrete timestamp feature id.\n */\ncase class RealTimeAggregateTimeDecay(\n  featureIdToHalfLife: Map[Long, Duration],\n  timestampFeatureId: Long = TIMESTAMP.getFeatureId) {\n\n  /**\n   * Mutates the data record which is just a reference to the input.\n   *\n   * @param record    Data record to apply decay to (is mutated).\n   * @param timeNow   The current read time (in milliseconds) to decay counts forward to.\n   */\n  def apply(record: DataRecord, timeNow: Long): Unit = {\n    if (record.isSetDiscreteFeatures) {\n      val discreteFeatures = record.getDiscreteFeatures\n      if (discreteFeatures.containsKey(timestampFeatureId)) {\n        if (record.isSetContinuousFeatures) {\n          val ctsFeatures = record.getContinuousFeatures\n\n          val storedTimestamp: Long = discreteFeatures.get(timestampFeatureId)\n          val scaledDt = if (timeNow > storedTimestamp) {\n            (timeNow - storedTimestamp).toDouble * math.log(2)\n          } else 0.0\n          featureIdToHalfLife.foreach {\n            case (featureId, halfLife) =>\n              if (ctsFeatures.containsKey(featureId)) {\n                val storedValue = ctsFeatures.get(featureId)\n                val alpha =\n                  if (halfLife.inMilliseconds != 0) math.exp(-scaledDt / halfLife.inMilliseconds)\n                  else 0\n                val decayedValue: Double = alpha * storedValue\n                record.putToContinuousFeatures(featureId, decayedValue)\n              }\n          }\n        }\n        discreteFeatures.remove(timestampFeatureId)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/TopicCountryEngagementRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TopicCountryEngagementCache\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject TopicCountryEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TopicCountryEngagementRealTimeAggregateFeatureHydrator @Inject() (\n  @Named(TopicCountryEngagementCache) override val client: ReadCache[(Long, String), DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends BaseRealTimeAggregateBulkCandidateFeatureHydrator[(Long, String)] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TopicCountryEngagementRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    TopicCountryEngagementRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    topicCountryRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    topicCountryRealTimeAggregates -> \"topic-country_code.timelines.topic_country_engagement_real_time_aggregates.\"\n  )\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[(Long, String)]] = {\n    candidates.map { candidate =>\n      val maybeTopicId = candidate.features\n        .getTry(TopicIdSocialContextFeature)\n        .toOption\n        .flatten\n\n      val maybeCountryCode = query.clientContext.countryCode\n\n      for {\n        topicId <- maybeTopicId\n        countryCode <- maybeCountryCode\n      } yield (topicId, countryCode)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/TopicEngagementRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TopicEngagementCache\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject TopicEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TopicEngagementRealTimeAggregateFeatureHydrator @Inject() (\n  @Named(TopicEngagementCache) override val client: ReadCache[Long, DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends BaseRealTimeAggregateBulkCandidateFeatureHydrator[Long] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TopicEngagementRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    TopicEngagementRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    topicEngagementRealTimeAggregatesProd,\n    topicEngagement24HourRealTimeAggregatesProd,\n    topicShareEngagementsRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    topicEngagement24HourRealTimeAggregatesProd -> \"topic.timelines.topic_engagement_24_hour_real_time_aggregates.\",\n    topicShareEngagementsRealTimeAggregates -> \"topic.timelines.topic_share_engagements_real_time_aggregates.\"\n  )\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] = {\n    candidates.map { candidate =>\n      candidate.features\n        .getTry(TopicIdSocialContextFeature)\n        .toOption\n        .flatten\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/TweetCountryEngagementRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetCountryEngagementCache\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject TweetCountryEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TweetCountryEngagementRealTimeAggregateFeatureHydrator @Inject() (\n  @Named(TweetCountryEngagementCache) override val client: ReadCache[(Long, String), DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends BaseRealTimeAggregateBulkCandidateFeatureHydrator[(Long, String)] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetCountryEngagementRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    TweetCountryEngagementRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    tweetCountryRealTimeAggregates,\n    tweetCountryPrivateEngagementsRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    tweetCountryRealTimeAggregates -> \"tweet-country_code.timelines.tweet_country_engagement_real_time_aggregates.\",\n    tweetCountryPrivateEngagementsRealTimeAggregates -> \"tweet-country_code.timelines.tweet_country_private_engagement_real_time_aggregates.\"\n  )\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[(Long, String)]] = {\n    val countryCode = query.clientContext.countryCode\n    candidates.map { candidate =>\n      val originalTweetId = CandidatesUtil.getOriginalTweetId(candidate)\n      countryCode.map((originalTweetId, _))\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/TweetEngagementRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TweetEngagementCache\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject TweetEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TweetEngagementRealTimeAggregateFeatureHydrator @Inject() (\n  @Named(TweetEngagementCache) override val client: ReadCache[Long, DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends BaseRealTimeAggregateBulkCandidateFeatureHydrator[Long] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetEngagementRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    TweetEngagementRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    tweetEngagement30MinuteCountsProd,\n    tweetEngagementTotalCountsProd,\n    tweetEngagementUserStateRealTimeAggregatesProd,\n    tweetNegativeEngagementUserStateRealTimeAggregates,\n    tweetNegativeEngagement6HourCounts,\n    tweetNegativeEngagementTotalCounts,\n    tweetShareEngagementsRealTimeAggregates,\n    tweetBCEDwellEngagementsRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    tweetShareEngagementsRealTimeAggregates -> \"original_tweet.timelines.tweet_share_engagements_real_time_aggregates.\",\n    tweetBCEDwellEngagementsRealTimeAggregates -> \"original_tweet.timelines.tweet_bce_dwell_engagements_real_time_aggregates.\"\n  )\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] = {\n    candidates\n      .map(candidate => Some(CandidatesUtil.getOriginalTweetId(candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/TwitterListEngagementRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.TwitterListIdFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.TwitterListEngagementCache\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject TwitterListEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass TwitterListEngagementRealTimeAggregateFeatureHydrator @Inject() (\n  @Named(TwitterListEngagementCache) override val client: ReadCache[Long, DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends BaseRealTimeAggregateBulkCandidateFeatureHydrator[Long] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TwitterListEngagementRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    TwitterListEngagementRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    listEngagementRealTimeAggregatesProd\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    listEngagementRealTimeAggregatesProd -> \"twitter_list.timelines.twitter_list_engagement_real_time_aggregates.\"\n  )\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[Long]] = {\n    candidates.map { candidate =>\n      candidate.features\n        .getTry(TwitterListIdFeature)\n        .toOption\n        .flatten\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/UserAuthorEngagementRealTimeAggregateFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UserAuthorEngagementCache\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject UserAuthorEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass UserAuthorEngagementRealTimeAggregateFeatureHydrator @Inject() (\n  @Named(UserAuthorEngagementCache) override val client: ReadCache[(Long, Long), DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends BaseRealTimeAggregateBulkCandidateFeatureHydrator[(Long, Long)] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserAuthorEngagementRealTimeAggregate\")\n\n  override val outputFeature: DataRecordInAFeature[TweetCandidate] =\n    UserAuthorEngagementRealTimeAggregateFeature\n\n  override val aggregateGroups: Seq[AggregateGroup] = Seq(\n    userAuthorEngagementRealTimeAggregatesProd,\n    userAuthorShareEngagementsRealTimeAggregates\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    userAuthorEngagementRealTimeAggregatesProd -> \"user-author.timelines.user_author_engagement_real_time_aggregates.\",\n    userAuthorShareEngagementsRealTimeAggregates -> \"user-author.timelines.user_author_share_engagements_real_time_aggregates.\"\n  )\n\n  override def keysFromQueryAndCandidates(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[Option[(Long, Long)]] = {\n    val userId = query.getRequiredUserId\n    candidates.map { candidate =>\n      CandidatesUtil\n        .getOriginalAuthorId(candidate.features)\n        .map((userId, _))\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator/real_time_aggregates/UserEngagementRealTimeAggregatesFeatureHydrator.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.feature_hydrator.real_time_aggregates\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.UserEngagementCache\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.ReadCache\nimport com.twitter.timelines.data_processing.ml_util.aggregation_framework.AggregateGroup\nimport com.twitter.timelines.prediction.common.aggregates.real_time.TimelinesOnlineAggregationFeaturesOnlyConfig._\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject UserEngagementRealTimeAggregateFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\nclass UserEngagementRealTimeAggregatesFeatureHydrator @Inject() (\n  @Named(UserEngagementCache) override val client: ReadCache[Long, DataRecord],\n  override val statsReceiver: StatsReceiver)\n    extends BaseRealTimeAggregateQueryFeatureHydrator[Long] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"UserEngagementRealTimeAggregates\")\n\n  override val outputFeature: DataRecordInAFeature[PipelineQuery] =\n    UserEngagementRealTimeAggregateFeature\n\n  val aggregateGroups: Seq[AggregateGroup] = Seq(\n    userEngagementRealTimeAggregatesProd,\n    userShareEngagementsRealTimeAggregates,\n    userBCEDwellEngagementsRealTimeAggregates,\n    userEngagement48HourRealTimeAggregatesProd,\n    userNegativeEngagementAuthorUserState72HourRealTimeAggregates,\n    userNegativeEngagementAuthorUserStateRealTimeAggregates,\n    userProfileEngagementRealTimeAggregates,\n  )\n\n  override val aggregateGroupToPrefix: Map[AggregateGroup, String] = Map(\n    userShareEngagementsRealTimeAggregates -> \"user.timelines.user_share_engagements_real_time_aggregates.\",\n    userBCEDwellEngagementsRealTimeAggregates -> \"user.timelines.user_bce_dwell_engagements_real_time_aggregates.\",\n    userEngagement48HourRealTimeAggregatesProd -> \"user.timelines.user_engagement_48_hour_real_time_aggregates.\",\n    userNegativeEngagementAuthorUserState72HourRealTimeAggregates -> \"user.timelines.user_negative_engagement_author_user_state_72_hour_real_time_aggregates.\",\n    userProfileEngagementRealTimeAggregates -> \"user.timelines.user_profile_engagement_real_time_aggregates.\"\n  )\n\n  override def keysFromQueryAndCandidates(query: PipelineQuery): Option[Long] = {\n    Some(query.getRequiredUserId)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/util\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/control_ai\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"src/thrift/com/twitter/timelines/control_ai:timeline-control-ai-thrift-scala\",\n        \"stitch/stitch-socialgraph\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/ControlAiExcludeFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.ControlAiEmbeddingSimilarityThresholdParam\nimport com.twitter.home_mixer.product.scored_tweets.util.ControlAiUtil\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.control_ai.ControlAiTopicEmbeddingMapFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.control_ai.UserControlAiFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.control_ai.control.{thriftscala => ci}\n\nobject ControlAiExcludeFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"ControlAiExclude\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val actions = query.features\n      .flatMap(_.getOrElse(UserControlAiFeature, None))\n      .map(_.actions).getOrElse(Seq.empty).filter(_.actionType == ci.ActionType.Exclude)\n\n    val (removed, kept) = candidates.partition { candidate =>\n      actions.exists(\n        ControlAiUtil.conditionMatch(\n          _,\n          candidate,\n          query.features\n            .map(_.get(ControlAiTopicEmbeddingMapFeature)).getOrElse(Map.empty),\n          threshold = query.params(ControlAiEmbeddingSimilarityThresholdParam)\n        ))\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/ControlAiOnlyIncludeFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.ControlAiEmbeddingSimilarityThresholdParam\nimport com.twitter.home_mixer.product.scored_tweets.util.ControlAiUtil\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.control_ai.ControlAiTopicEmbeddingMapFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.control_ai.UserControlAiFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.control_ai.control.{thriftscala => ci}\n\nobject ControlAiOnlyIncludeFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"ControlAiOnlyInclude\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val actions = query.features\n      .flatMap(_.getOrElse(UserControlAiFeature, None))\n      .map(_.actions).getOrElse(Seq.empty).filter(_.actionType == ci.ActionType.Only)\n\n    val (kept, removed) = candidates.partition { candidate =>\n      actions.isEmpty || actions.exists(\n        ControlAiUtil.conditionMatch(\n          _,\n          candidate,\n          query.features\n            .map(_.get(ControlAiTopicEmbeddingMapFeature)).getOrElse(Map.empty),\n          threshold = query.params(ControlAiEmbeddingSimilarityThresholdParam)\n        ))\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/CustomSnowflakeIdAgeFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidatePipelines\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\n/**\n * Filters Snowflake ID-compatible content that are older than some configurable threshold.\n *\n * We derive the content age from the Snowflake ID (http://go/snowflake), and non-Snowflake IDs\n * are assumed to be too old.\n *\n * @param maxAgeParam Feature Switch configurable for convenience\n * @tparam Candidate The type of the candidates\n */\ncase class CustomSnowflakeIdAgeFilter[Candidate <: UniversalNoun[Long]](\n  maxAgeParam: Param[Duration])\n    extends Filter[PipelineQuery, Candidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"SnowflakeIdAge\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n    val maxAge = query.params(maxAgeParam)\n\n    val (keptCandidates, removedCandidates) = candidates\n      .map(c => (c.candidate, c.features.get(CandidatePipelines)))\n      .partition { filterCandidate =>\n        SnowflakeId.timeFromIdOpt(filterCandidate._1.id) match {\n          case Some(creationTime) =>\n            query.queryTime.since(creationTime) <= maxAge\n          case _ => false // Always deny if non-Snowflake\n        }\n      }\n\n    Stitch.value(\n      FilterResult(kept = keptCandidates.map(_._1), removed = removedCandidates.map(_._1)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/CustomSnowflakeIdAgeFilterWithIncludeRule.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\n/**\n * Filters Snowflake ID-compatible content that are older than some configurable threshold,\n * but only for those candidates that match the includeRule.\n *\n * Candidates not matching the includeRule will always be kept.\n *\n * @param maxAgeParam Feature Switch configurable for convenience\n * @tparam Candidate The type of the candidates\n */\ncase class CustomSnowflakeIdAgeFilterWithIncludeRule[Candidate <: UniversalNoun[Long]](\n  maxAgeParam: Param[Duration],\n  includeRule: Option[CandidateWithFeatures[Candidate] => Boolean] = None)\n    extends Filter[PipelineQuery, Candidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"SnowflakeIdAgeWithIncludeRule\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n    val maxAge = query.params(maxAgeParam)\n\n    val (keptCandidates, removedCandidates) = candidates.partition { filterCandidate =>\n      includeRule match {\n        case Some(rule) if rule(filterCandidate) =>\n          SnowflakeId.timeFromIdOpt(filterCandidate.candidate.id) match {\n            case Some(creationTime) =>\n              query.queryTime.since(creationTime) <= maxAge\n            case _ => false\n          }\n        case _ =>\n          true\n      }\n    }\n\n    Stitch.value(\n      FilterResult(\n        kept = keptCandidates.map(_.candidate),\n        removed = removedCandidates.map(_.candidate)\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/CustomSnowflakeIdAgeFilterWithSkipRule.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\n/**\n * Filters Snowflake ID-compatible content that are older than some configurable threshold.\n *\n * We derive the content age from the Snowflake ID (http://go/snowflake), and non-Snowflake IDs\n * are assumed to be too old.\n *\n * @param maxAgeParam Feature Switch configurable for convenience\n * @tparam Candidate The type of the candidates\n */\ncase class CustomSnowflakeIdAgeFilterWithSkipRule[Candidate <: UniversalNoun[Long]](\n  maxAgeParam: Param[Duration],\n  skipRule: Option[CandidateWithFeatures[Candidate] => Boolean] = None)\n    extends Filter[PipelineQuery, Candidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"SnowflakeIdAgeWithSkipRule\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n    val maxAge = query.params(maxAgeParam)\n\n    val (keptCandidates, removedCandidates) = candidates.partition { filterCandidate =>\n      skipRule match {\n        case Some(rule) if rule(filterCandidate) => true\n        case _ =>\n          SnowflakeId.timeFromIdOpt(filterCandidate.candidate.id) match {\n            case Some(creationTime) => query.queryTime.since(creationTime) <= maxAge\n            case _ => false\n          }\n      }\n    }\n\n    Stitch.value(\n      FilterResult(\n        kept = keptCandidates.map(_.candidate),\n        removed = removedCandidates.map(_.candidate)\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/DuplicateConversationTweetsFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Remove any candidate that is in the ancestor list of any reply, including retweets of ancestors.\n * Then deduplicate any replies with the same root tweet by score.\n *\n * E.g. if B replied to A and D was a retweet of A, we would prefer to drop D since otherwise\n * we may end up serving the same tweet twice in the timeline (e.g. serving both A->B and D).\n */\nobject DuplicateConversationTweetsFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"DuplicateConversationTweets\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val allAncestors = candidates\n      .flatMap(_.features.getOrElse(AncestorsFeature, Seq.empty))\n      .map(_.tweetId).toSet\n\n    val dedupedCandidates = candidates\n      .filter(candidate => !allAncestors.contains(CandidatesUtil.getOriginalTweetId(candidate)))\n      .groupBy(_.features.getOrElse(AncestorsFeature, Seq.empty).lastOption.map(_.tweetId))\n      .flatMap {\n        case (Some(_), conversationCandidates) =>\n          Seq(conversationCandidates.maxBy(_.features.getOrElse(ScoreFeature, None)).candidate.id)\n        case (None, nonConversationCandidates) => nonConversationCandidates.map(_.candidate.id)\n      }.toSet\n\n    val (kept, removed) =\n      candidates.partition(candidate => dedupedCandidates.contains(candidate.candidate.id))\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/ExtendedDirectedAtFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject ExtendedDirectedAtFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"ExtendedDirectedAt\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val sgsFollowedUsers =\n      query.features.map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty)).toSet.flatten\n\n    val (removed, kept) = candidates.partition { candidate =>\n      val authorId = candidate.features.getOrElse(AuthorIdFeature, None)\n      val inReplyToUser = candidate.features.getOrElse(InReplyToUserIdFeature, None)\n      val directedAtUser = candidate.features.getOrElse(DirectedAtUserIdFeature, None)\n\n      inReplyToUser.isEmpty && directedAtUser.exists(!sgsFollowedUsers.contains(_)) &&\n      authorId.exists(sgsFollowedUsers.contains)\n    }\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/FollowedAuthorFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * This filter removes tweets who's author is in the follow list\n */\nobject FollowedAuthorFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"FollowedAuthor\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val sgsFollowedUsers =\n      query.features.map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty)).toSet.flatten\n    val (removed, kept) = candidates.partition(\n      _.features.getOrElse(AuthorIdFeature, None).exists(sgsFollowedUsers.contains))\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/GrokAutoTranslateLanguageFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.GrokTranslatedPostIsCachedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetLanguageFromTweetypieFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserUnderstandableLanguagesFeature\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableGrokAutoTranslateLanguageFilter\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.stitch.Stitch\n\nobject GrokAutoTranslateLanguageFilter\n    extends Filter[ScoredTweetsQuery, TweetCandidate]\n    with Filter.Conditionally[ScoredTweetsQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"GrokAutoTranslateLanguage\")\n\n  override def onlyIf(\n    query: ScoredTweetsQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = query.params(EnableGrokAutoTranslateLanguageFilter)\n\n  override def apply(\n    query: ScoredTweetsQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n\n    val allUserUnderstandableLanguages: Seq[String] =\n      query.features\n        .getOrElse(FeatureMap.empty).getOrElse(\n          UserUnderstandableLanguagesFeature,\n          Seq.empty[String])\n\n    val (kept, removed) = {\n      if (!allUserUnderstandableLanguages.isEmpty) {\n        candidates.partition { candidate =>\n          val inNetwork =\n            candidate.features.getOrElse(InNetworkFeature, true)\n          val postLanguageOpt =\n            candidate.features.getOrElse(TweetLanguageFromTweetypieFeature, None).map(_.toLowerCase)\n          val grokTranslateCacheExists =\n            candidate.features.getOrElse(GrokTranslatedPostIsCachedFeature, true)\n\n          postLanguageOpt.forall { postLanguage =>\n            inNetwork || grokTranslateCacheExists ||\n            allUserUnderstandableLanguages.contains(postLanguage)\n          }\n        }\n      } else { (candidates, Seq.empty) }\n    }\n\n    val filterResult = FilterResult(\n      kept = kept.map(_.candidate),\n      removed = removed.map(_.candidate)\n    )\n\n    Stitch.value(filterResult)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/IsOutOfNetworkColdStartPostFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsColdStartPostFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject IsOutOfNetworkColdStartPostFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"IsOutOfNetworkColdStartPost\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (removed, kept) = candidates.partition { candidate =>\n      val isColdStartPost =\n        candidate.features.getOrElse(\n          IsColdStartPostFeature,\n          false\n        ) // Id none, assume its not cold start, don't filter out\n      val isOutOfNetwork =\n        !candidate.features\n          .getOrElse(InNetworkFeature, true) // If none, assume its in-network, don't filter out\n      isColdStartPost && isOutOfNetwork\n    }\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/LanguageFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.UserLanguagesFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DeviceLanguageFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetLanguageFromTweetypieFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserEngagedLanguagesFeature\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableLanguageFilter\nimport com.twitter.home_mixer.util.LanguageCode.AllowedLanguageCodes\nimport com.twitter.home_mixer.util.LanguageCode.languageToISO\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.stitch.Stitch\n\nobject LanguageFilter\n    extends Filter[ScoredTweetsQuery, TweetCandidate]\n    with Filter.Conditionally[ScoredTweetsQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"Language\")\n\n  override def onlyIf(\n    query: ScoredTweetsQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Boolean = query.params(EnableLanguageFilter)\n\n  override def apply(\n    query: ScoredTweetsQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val userLanguages = query.features.get\n      .getOrElse(UserLanguagesFeature, Seq.empty)\n      .flatMap(lang => languageToISO.get(lang.toString.toLowerCase))\n      .toSet\n\n    val userEngagedLanguages = query.features.get\n      .getOrElse(UserEngagedLanguagesFeature, Set.empty[String])\n      .map(_.toLowerCase)\n\n    val deviceLanguage =\n      query.features.get.getOrElse(DeviceLanguageFeature, None).map(_.toLowerCase).toSet\n\n    val allUserLanguages = userLanguages ++ userEngagedLanguages ++ deviceLanguage\n\n    val (kept, removed) = if (!allUserLanguages.isEmpty) {\n      candidates.partition { candidate =>\n        val inNetwork = candidate.features.getOrElse(InNetworkFeature, true)\n\n        val postLanguage = candidate.features\n          .getOrElse(TweetLanguageFromTweetypieFeature, None)\n          .map(_.toLowerCase)\n\n        postLanguage.forall { lang =>\n          inNetwork || allUserLanguages.contains(lang) || AllowedLanguageCodes.contains(lang)\n        }\n      }\n    } else { (candidates, Seq.empty) }\n\n    val filterResult = FilterResult(\n      kept = kept.map(_.candidate),\n      removed = removed.map(_.candidate)\n    )\n\n    Stitch.value(filterResult)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/OONReplyFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * This filter removes recommended replies to not followed users\n */\nobject OONReplyFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"OONReply\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val sgsFollowedUsers =\n      query.features.map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty)).toSet.flatten\n    val (removed, kept) = candidates.partition { candidate =>\n      val isValidRecommendedReply =\n        !candidate.features.getOrElse(IsRetweetFeature, false) &&\n          candidate.features\n            .getOrElse(InReplyToUserIdFeature, None).exists(sgsFollowedUsers.contains)\n\n      val isRecommendedReply =\n        candidate.features.getOrElse(InReplyToTweetIdFeature, None).nonEmpty &&\n          !candidate.features.getOrElse(AuthorIdFeature, None).exists(sgsFollowedUsers.contains)\n\n      isRecommendedReply && !isValidRecommendedReply\n    }\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/OutOfNetworkCompetitorFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CompetitorSetParam\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject OutOfNetworkCompetitorFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"OutOfNetworkCompetitor\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val competitorAuthors = query.params(CompetitorSetParam)\n    val (removed, kept) =\n      candidates.partition(isOutOfNetworkTweetFromCompetitor(_, competitorAuthors))\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n\n  def isOutOfNetworkTweetFromCompetitor(\n    candidate: CandidateWithFeatures[TweetCandidate],\n    competitorAuthors: Set[Long]\n  ): Boolean = {\n    !candidate.features.getOrElse(InNetworkFeature, true) &&\n    !candidate.features.getOrElse(IsRetweetFeature, false) &&\n    candidate.features.getOrElse(AuthorIdFeature, None).exists(competitorAuthors.contains)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/OutOfNetworkCompetitorURLFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetUrlsFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CompetitorURLSeqParam\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject OutOfNetworkCompetitorURLFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"OutOfNetworkCompetitorURL\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val competitorUrls = query.params(CompetitorURLSeqParam).toSet\n    val (removed, kept) = candidates.partition(hasOutOfNetworkUrlFromCompetitor(_, competitorUrls))\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n\n  def hasOutOfNetworkUrlFromCompetitor(\n    candidate: CandidateWithFeatures[TweetCandidate],\n    competitorUrls: Set[String]\n  ): Boolean = {\n    !candidate.features.getOrElse(InNetworkFeature, true) &&\n    !candidate.features.getOrElse(IsRetweetFeature, false) &&\n    candidate.features\n      .getOrElse(TweetUrlsFeature, Seq.empty).toSet.intersect(competitorUrls).nonEmpty\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/QualifiedRepliesFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsInReplyToReplyOrDirectedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsInReplyToRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject QualifiedRepliesFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"QualifiedReplies\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val sgsFollowedUsers =\n      query.features.map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty)).toSet.flatten\n\n    val (removed, kept) = candidates.partition { candidate =>\n      val isRetweet = candidate.features.getOrElse(IsRetweetFeature, false)\n      val authorId = candidate.features.getOrElse(AuthorIdFeature, None).getOrElse(0L)\n      val inReplyToUser = candidate.features.getOrElse(InReplyToUserIdFeature, None)\n      val replyToFollowed = inReplyToUser.exists(sgsFollowedUsers.contains)\n      val isValidReplyToUser = inReplyToUser.exists { user =>\n        user != query.getRequiredUserId && user != authorId\n      }\n      val inReplyToRetweetOrReplyOrDirected =\n        candidate.features.getOrElse(IsInReplyToReplyOrDirectedFeature, false) ||\n          candidate.features.getOrElse(IsInReplyToRetweetFeature, false)\n\n      inReplyToUser.nonEmpty && !replyToFollowed && sgsFollowedUsers.contains(authorId) &&\n      (isRetweet || !isValidReplyToUser || inReplyToRetweetOrReplyOrDirected)\n    }\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/RetweetSourceTweetRemovingFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.util.ReplyRetweetUtil\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * This filter removes source tweets of retweets, added via second EB call in TLR\n */\nobject RetweetSourceTweetRemovingFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"RetweetSourceTweetRemoving\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (kept, removed) =\n      candidates.partition(\n        _.features.getOrElse(EarlybirdFeature, None).exists(_.isSourceTweet)) match {\n        case (sourceTweets, nonSourceTweets) =>\n          val inReplyToTweetIds: Set[Long] =\n            nonSourceTweets\n              .filter(ReplyRetweetUtil.isEligibleReply(_)).flatMap(\n                _.features.getOrElse(InReplyToTweetIdFeature, None)).toSet\n          val (keptSourceTweets, removedSourceTweets) = sourceTweets\n            .map(_.candidate)\n            .partition(candidate => inReplyToTweetIds.contains(candidate.id))\n          (nonSourceTweets.map(_.candidate) ++ keptSourceTweets, removedSourceTweets)\n      }\n    Stitch.value(FilterResult(kept = kept, removed = removed))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/SGSAuthorFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.socialgraph.{thriftscala => sg}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.socialgraph.SocialGraph\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class SGSAuthorFilter @Inject() (socialGraph: SocialGraph)\n    extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"SGSAuthor\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val authorIds =\n      candidates.flatMap { candidate => candidate.features.get(AuthorIdFeature) }.distinct\n\n    val request = sg.IdsRequest(\n      relationships = Seq(\n        sg.SrcRelationship(\n          source = query.getRequiredUserId,\n          relationshipType = sg.RelationshipType.Blocking,\n          hasRelationship = true,\n          targets = Some(authorIds)),\n        sg.SrcRelationship(\n          source = query.getRequiredUserId,\n          relationshipType = sg.RelationshipType.BlockedBy,\n          hasRelationship = true,\n          targets = Some(authorIds)),\n        sg.SrcRelationship(\n          source = query.getRequiredUserId,\n          relationshipType = sg.RelationshipType.Muting,\n          hasRelationship = true,\n          targets = Some(authorIds))\n      ),\n      pageRequest = Some(sg.PageRequest(selectAll = Some(true))),\n      context = Some(sg.LookupContext(performUnion = Some(true)))\n    )\n\n    socialGraph.ids(request).map { result =>\n      val ids = result.ids.toSet\n      val (removed, kept) =\n        candidates.partition { candidate =>\n          ids.contains(candidate.features.get(AuthorIdFeature).get)\n        }\n      FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate))\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/ScoredTweetsSocialContextFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\n\nobject ScoredTweetsSocialContextFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"ScoredTweetsSocialContext\")\n\n  // Tweets from candidate sources which don't need generic like/follow/topic proof\n  private val AllowedSources: Set[st.SuggestType] = Set(\n    st.SuggestType.RankedListTweet,\n    st.SuggestType.RecommendedTrendTweet,\n    st.SuggestType.MediaTweet\n  )\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val validTweetIds = candidates\n      .filter { candidate =>\n        candidate.features.getOrElse(InNetworkFeature, true) ||\n        candidate.features.getOrElse(SuggestTypeFeature, None).exists(AllowedSources.contains) ||\n        candidate.features.getOrElse(InReplyToTweetIdFeature, None).isDefined ||\n        hasLikedBySocialContext(candidate.features) ||\n        hasFollowedBySocialContext(candidate.features) ||\n        hasTopicSocialContext(candidate.features)\n      }.map(_.candidate.id).toSet\n\n    val (kept, removed) =\n      candidates.map(_.candidate).partition(candidate => validTweetIds.contains(candidate.id))\n\n    Stitch.value(FilterResult(kept = kept, removed = removed))\n  }\n\n  private def hasLikedBySocialContext(candidateFeatures: FeatureMap): Boolean =\n    candidateFeatures\n      .getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty)\n      .exists(\n        candidateFeatures\n          .getOrElse(PerspectiveFilteredLikedByUserIdsFeature, Seq.empty)\n          .toSet.contains\n      )\n\n  private def hasFollowedBySocialContext(candidateFeatures: FeatureMap): Boolean =\n    candidateFeatures.getOrElse(SGSValidFollowedByUserIdsFeature, Seq.empty).nonEmpty\n\n  private def hasTopicSocialContext(candidateFeatures: FeatureMap): Boolean = {\n    candidateFeatures.getOrElse(TopicIdSocialContextFeature, None).isDefined &&\n    candidateFeatures.getOrElse(TopicContextFunctionalityTypeFeature, None).isDefined\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/TopKFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * This filter ranks tweets by a score feature and takes top k\n */\ncase class TopKFilter(scoreFeature: Feature[_, Double], topK: Int)\n    extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"TopK\")\n\n  private val DefaultScore = 0D\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val sorted = candidates\n      .sortBy(_.features.getOrElse(scoreFeature, DefaultScore))(Ordering[Double].reverse)\n    val (kept, removed) = sorted.splitAt(topK)\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/TopKOptionalFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * This filter ranks tweets by a score feature and takes top k\n */\ncase class TopKOptionalFilter(scoreFeature: Feature[_, Option[Double]], topK: Int)\n    extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"TopKOptional\")\n\n  private val DefaultScore = 0D\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val sorted = candidates\n      .sortBy(_.features.getOrElse(scoreFeature, None).getOrElse(DefaultScore))(\n        Ordering[Double].reverse)\n    val (kept, removed) = sorted.splitAt(topK)\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/UtegMinFavCountFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.UtegFavListFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject UtegMinFavCountFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"UtegMinFavCount\")\n\n  private val UtegMinFavCount = 1\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val (kept, removed) = candidates.partition { candidate =>\n      val nonAuthorFavList = candidate.features\n        .getOrElse(UtegFavListFeature, Seq.empty)\n        .filter(!candidate.features.getOrElse(AuthorIdFeature, None).contains(_))\n      nonAuthorFavList.size >= UtegMinFavCount\n    }\n\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/filter/UtegTopKFilter.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.filter\n\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdScoreFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FetchParams\nimport com.twitter.home_mixer.product.scored_tweets.response_transformer.UtegScoreFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject UtegTopKFilter extends Filter[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"UtegTopK\")\n\n  private val DefaultScore = 0D\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[FilterResult[TweetCandidate]] = {\n    val topK = query.params(FetchParams.UTEGMaxTweetsToFetchParam)\n    val sorted = candidates.sortBy { candidate =>\n      val utegScore = candidate.features.getOrElse(UtegScoreFeature, DefaultScore)\n      val ebScore = candidate.features\n        .getOrElse(EarlybirdScoreFeature, None)\n        .getOrElse(DefaultScore)\n      utegScore + ebScore * 2\n    }(Ordering[Double].reverse)\n    val (kept, removed) = sorted.splitAt(topK)\n    Stitch.value(FilterResult(kept = kept.map(_.candidate), removed = removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/gate/AllowLowSignalUserGate.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.gate\n\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableLowSignalUserCheck\nimport com.twitter.home_mixer.util.SignalUtil\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Continue for low signal users who also follow < 5 users\n *\n * Check gate param last to only evaluate for eligible users and avoid experimental dilution.\n */\nobject AllowLowSignalUserGate extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"AllowLowSignalUser\")\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val continue = \n      SignalUtil.isLowSignalUser(query) &&\n      query.params(EnableLowSignalUserCheck)\n\n    Stitch.value(continue)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/gate/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/gate/DenyLowSignalUserGate.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.gate\n\nimport com.twitter.home_mixer.model.HomeFeatures.SignupSourceFeature\nimport com.twitter.home_mixer.model.signup.MarchMadness\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableLowSignalUserCheck\nimport com.twitter.home_mixer.util.SignalUtil\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Continue for all users except low signal users who also follow < 25 users\n *\n * Check gate param last to only evaluate for eligible users and avoid experimental dilution.\n */\nobject DenyLowSignalUserGate extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"DenyLowSignalUser\")\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val signupSource = query.features.flatMap(_.getOrElse(SignupSourceFeature, None))\n\n    val stop = signupSource.contains(MarchMadness) &&\n      SignalUtil.isLowSignalUser(query) &&\n      query.params(EnableLowSignalUserCheck)\n\n    Stitch.value(!stop)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/gate/MatchesCountryGate.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.gate\n\nimport com.twitter.home_mixer.model.HomeFeatures.SignupCountryFeature\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * Check whether the signup country code feature or the input country code\n * exists within the input country codes param\n */\ncase class MatchesCountryGate(countryCodes: Param[Seq[String]]) extends Gate[PipelineQuery] {\n  override val identifier: GateIdentifier = GateIdentifier(\"MatchesSignupCountry\")\n\n  /**\n   * The main predicate that controls this gate. If this predicate returns true, the gate returns Continue.\n   */\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val countryCode = query.clientContext.countryCode\n    val signupCountryCode = query.features\n      .flatMap(_.getOrElse(SignupCountryFeature, None))\n    val codeMatchesInput =\n      Seq(countryCode, signupCountryCode).flatten.exists { code =>\n        query.params(countryCodes).map(_.toLowerCase).contains(code.toLowerCase)\n      }\n    Stitch.value(codeMatchesInput)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/gate/MinCachedTweetsGate.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.gate\n\nimport com.twitter.home_mixer.model.HomeFeatures.HasRecentFeedbackSinceCacheTtlFeature\nimport com.twitter.home_mixer.product.scored_tweets.gate.MinCachedTweetsGate.identifierSuffix\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableRecentFeedbackCheckParam\nimport com.twitter.home_mixer.util.CachedScoredTweetsHelper\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\ncase class MinCachedTweetsGate(\n  candidatePipelineIdentifier: CandidatePipelineIdentifier,\n  minCachedTweetsParam: Param[Int])\n    extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier =\n    GateIdentifier(candidatePipelineIdentifier + identifierSuffix)\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    val minCachedTweets = query.params(minCachedTweetsParam)\n    val cachedScoredTweets =\n      query.features.map(CachedScoredTweetsHelper.unseenCachedScoredTweets).getOrElse(Seq.empty)\n    val numCachedTweets = cachedScoredTweets.count { tweet =>\n      tweet.candidatePipelineIdentifier.exists(\n        CandidatePipelineIdentifier(_).equals(candidatePipelineIdentifier))\n    }\n\n    val hasMinCachedTweets = numCachedTweets < minCachedTweets\n    query.features.map(_.getOrElse(HasRecentFeedbackSinceCacheTtlFeature, false)) match {\n      case Some(true) =>\n        if (query.params(EnableRecentFeedbackCheckParam)) Stitch.True\n        else Stitch.value(hasMinCachedTweets)\n      case _ => Stitch.value(hasMinCachedTweets)\n    }\n  }\n}\n\nobject MinCachedTweetsGate {\n  val identifierSuffix = \"MinCachedTweets\"\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/gate/MinTimeSinceLastRequestGate.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.gate\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.HomeFeatures.LastNonPollingTimeFeature\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Gate continues if the amount of time passed since the previous request is greater\n * than the configured amount or if the previous request time in not available\n */\nobject MinTimeSinceLastRequestGate extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"TimeSinceLastRequest\")\n\n  private val MinTimeSinceLastRequest = 24.hours\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = Stitch.value {\n    query.features.exists { features =>\n      features\n        .getOrElse(LastNonPollingTimeFeature, None)\n        .forall(lnpt => (query.queryTime - lnpt) > MinTimeSinceLastRequest)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/gate/RecentFeedbackCheckGate.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.gate\n\nimport com.twitter.home_mixer.model.HomeFeatures.HasRecentFeedbackSinceCacheTtlFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableRecentFeedbackCheckParam\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Continue if a user has no don't like feedback within cached tweets ttl time (3 mins).\n * The reason is that if a user clicks don't like, tweets in the cache won't be affected\n * and it feels our system has slow response to user's negative feedback\n */\n\nobject RecentFeedbackCheckGate extends Gate[PipelineQuery] {\n  override val identifier: GateIdentifier = GateIdentifier(\"RecentFeedbackCheck\")\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    query.features.map(_.getOrElse(HasRecentFeedbackSinceCacheTtlFeature, false)) match {\n      case Some(true) =>\n        if (query.params(EnableRecentFeedbackCheckParam)) Stitch.False\n        else Stitch.True\n      case _ => Stitch.True\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/marshaller/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_is_nsfw\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_visibility_reason\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/marshaller/ScoredTweetsResponseDomainMarshaller.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.marshaller\n\nimport com.twitter.home_mixer.model.HomeFeatures.UserActionsContainsExplicitSignalsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserActionsSizeFeature\nimport com.twitter.home_mixer.product.scored_tweets.model.QueryMetadata\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsResponse\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.model.common.identifier.DomainMarshallerIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\n\n/**\n * Creates a domain model of the Scored Tweets product response from the set of candidates selected\n */\nobject ScoredTweetsResponseDomainMarshaller\n    extends DomainMarshaller[ScoredTweetsQuery, ScoredTweetsResponse] {\n\n  override val identifier: DomainMarshallerIdentifier =\n    DomainMarshallerIdentifier(\"ScoredTweetsResponse\")\n\n  override def apply(\n    query: ScoredTweetsQuery,\n    selections: Seq[CandidateWithDetails]\n  ): ScoredTweetsResponse = ScoredTweetsResponse(\n    scoredTweets = selections,\n    queryMetadata = Some(\n      QueryMetadata(\n        userActionsSize = query.features.get.getOrElse(UserActionsSizeFeature, None),\n        userActionsContainsExplicitSignals =\n          Some(query.features.get.getOrElse(UserActionsContainsExplicitSignalsFeature, false))\n      ))\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/marshaller/ScoredTweetsResponseTransportMarshaller.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.marshaller\n\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.model._\nimport com.twitter.home_mixer.product.scored_tweets.model.QueryMetadata\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsResponse\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityIdFeature\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityNameFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_is_nsfw.IsNsfw\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_visibility_reason.VisibilityReason\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.TopicContextFunctionalityTypeMarshaller\nimport com.twitter.product_mixer.core.model.common.identifier.TransportMarshallerIdentifier\n\n/**\n * Marshall the domain model into our transport (Thrift) model.\n */\nobject ScoredTweetsResponseTransportMarshaller\n    extends TransportMarshaller[ScoredTweetsResponse, hmt.ScoredTweetsResponse] {\n\n  override val identifier: TransportMarshallerIdentifier =\n    TransportMarshallerIdentifier(\"ScoredTweetsResponse\")\n\n  override def apply(input: ScoredTweetsResponse): hmt.ScoredTweetsResponse = {\n    val scoredTweets = input.scoredTweets.map { tweet =>\n      mkScoredTweet(tweet.candidateIdLong, tweet.features, input.queryMetadata)\n    }\n    hmt.ScoredTweetsResponse(scoredTweets)\n  }\n\n  private def mkScoredTweet(\n    tweetId: Long,\n    features: FeatureMap,\n    queryMetadata: Option[QueryMetadata]\n  ): hmt.ScoredTweet = {\n    val topicFunctionalityType = features\n      .getOrElse(TopicContextFunctionalityTypeFeature, None)\n      .map(TopicContextFunctionalityTypeMarshaller(_))\n\n    val predictedScores = hmt.PredictedScores(\n      favoriteScore = features.getOrElse(PredictedFavoriteScoreFeature, None),\n      replyScore = features.getOrElse(PredictedReplyScoreFeature, None),\n      retweetScore = features.getOrElse(PredictedRetweetScoreFeature, None),\n      replyEngagedByAuthorScore =\n        features.getOrElse(PredictedReplyEngagedByAuthorScoreFeature, None),\n      goodClickConvoDescFavoritedOrRepliedScore = features\n        .getOrElse(PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature, None),\n      goodClickConvoDescUamGt2Score =\n        features.getOrElse(PredictedGoodClickConvoDescUamGt2ScoreFeature, None),\n      goodProfileClickScore = features.getOrElse(PredictedGoodProfileClickScoreFeature, None),\n      videoQualityViewScore = features.getOrElse(PredictedVideoQualityViewScoreFeature, None),\n      shareScore = features.getOrElse(PredictedShareScoreFeature, None),\n      dwellScore = features.getOrElse(PredictedDwellScoreFeature, None),\n      negativeFeedbackV2Score = features.getOrElse(PredictedNegativeFeedbackV2ScoreFeature, None)\n    )\n\n    val phoenixPredictedScores = hmt.PredictedScores(\n      favoriteScore = features.getOrElse(PhoenixPredictedFavoriteScoreFeature, None),\n      replyScore = features.getOrElse(PhoenixPredictedReplyScoreFeature, None),\n      retweetScore = features.getOrElse(PhoenixPredictedRetweetScoreFeature, None),\n      replyEngagedByAuthorScore = None,\n      goodClickConvoDescFavoritedOrRepliedScore = features\n        .getOrElse(PhoenixPredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature, None),\n      goodClickConvoDescUamGt2Score =\n        features.getOrElse(PhoenixPredictedGoodClickConvoDescUamGt2ScoreFeature, None),\n      goodProfileClickScore =\n        features.getOrElse(PhoenixPredictedGoodProfileClickScoreFeature, None),\n      videoQualityViewScore =\n        features.getOrElse(PhoenixPredictedVideoQualityViewScoreFeature, None),\n      shareScore = features.getOrElse(PhoenixPredictedShareScoreFeature, None),\n      dwellScore = features.getOrElse(PhoenixPredictedDwellScoreFeature, None),\n      negativeFeedbackV2Score =\n        features.getOrElse(PhoenixPredictedNegativeFeedbackV2ScoreFeature, None),\n      openLinkScore = features.getOrElse(PhoenixPredictedOpenLinkScoreFeature, None),\n      screenshotScore = features.getOrElse(PhoenixPredictedScreenshotScoreFeature, None),\n      bookmarkScore = features.getOrElse(PhoenixPredictedBookmarkScoreFeature, None)\n    )\n\n    val sourceSignal: Option[hmt.SourceSignal] = features.getOrElse(SourceSignalFeature, None).map {\n      modelSignal =>\n        hmt.SourceSignal(\n          id = modelSignal.id,\n          signalType = modelSignal.signalType,\n          signalEntity = modelSignal.signalEntity,\n          authorId = modelSignal.authorId,\n        )\n    }\n    hmt.ScoredTweet(\n      tweetId = tweetId,\n      authorId = features.get(AuthorIdFeature).get,\n      score = features.get(ScoreFeature),\n      servedType = features.get(ServedTypeFeature),\n      sourceTweetId = features.getOrElse(SourceTweetIdFeature, None),\n      sourceUserId = features.getOrElse(SourceUserIdFeature, None),\n      quotedTweetId = features.getOrElse(QuotedTweetIdFeature, None),\n      quotedUserId = features.getOrElse(QuotedUserIdFeature, None),\n      inReplyToTweetId = features.getOrElse(InReplyToTweetIdFeature, None),\n      inReplyToUserId = features.getOrElse(InReplyToUserIdFeature, None),\n      directedAtUserId = features.getOrElse(DirectedAtUserIdFeature, None),\n      inNetwork = Some(features.getOrElse(InNetworkFeature, true)),\n      sgsValidLikedByUserIds = Some(features.getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty)),\n      sgsValidFollowedByUserIds =\n        Some(features.getOrElse(SGSValidFollowedByUserIdsFeature, Seq.empty)),\n      validLikedByUserIds = Some(features.getOrElse(ValidLikedByUserIdsFeature, Seq.empty)),\n      topicId = features.getOrElse(TopicIdSocialContextFeature, None),\n      topicFunctionalityType = topicFunctionalityType,\n      ancestors = Some(features.getOrElse(AncestorsFeature, Seq.empty)),\n      isReadFromCache = Some(features.getOrElse(IsReadFromCacheFeature, false)),\n      exclusiveConversationAuthorId =\n        features.getOrElse(ExclusiveConversationAuthorIdFeature, None),\n      authorMetadata = Some(\n        hmt.AuthorMetadata(\n          blueVerified = features.getOrElse(AuthorIsBlueVerifiedFeature, false),\n          goldVerified = features.getOrElse(AuthorIsGoldVerifiedFeature, false),\n          grayVerified = features.getOrElse(AuthorIsGrayVerifiedFeature, false),\n          legacyVerified = features.getOrElse(AuthorIsLegacyVerifiedFeature, false),\n          creator = features.getOrElse(AuthorIsCreatorFeature, false),\n          followers = features.getOrElse(AuthorFollowersFeature, None)\n        )),\n      lastScoredTimestampMs = None,\n      candidatePipelineIdentifier = None,\n      tweetUrls = None,\n      perspectiveFilteredLikedByUserIds = None,\n      predictionRequestId = features.getOrElse(PredictionRequestIdFeature, None),\n      communityId = features.getOrElse(CommunityIdFeature, None),\n      communityName = features.getOrElse(CommunityNameFeature, None),\n      listId = features.getOrElse(ListIdFeature, None),\n      listName = features.getOrElse(ListNameFeature, None),\n      isNsfw = features.getOrElse(IsNsfw, None),\n      visibilityReason = features.getOrElse(VisibilityReason, None),\n      tweetLanguage = features.getOrElse(TweetLanguageFeature, None),\n      tweetText = features.getOrElse(TweetTextFeature, None),\n      tweetTypeMetrics = features.getOrElse(TweetTypeMetricsFeature, None),\n      debugString = features.getOrElse(DebugStringFeature, None),\n      hasVideo = Some(features.getOrElse(HasVideoFeature, false)),\n      videoDurationMs = features.getOrElse(VideoDurationMsFeature, None),\n      mediaIds = Some(features.getOrElse(TweetMediaIdsFeature, Seq.empty)),\n      grokAnnotations = features.getOrElse(GrokAnnotationsFeature, None),\n      predictedScores = Some(predictedScores),\n      tweetMixerScore = features.getOrElse(TweetMixerScoreFeature, None),\n      phoenixPredictedScores = Some(phoenixPredictedScores),\n      sourceSignal = sourceSignal,\n      userActionsSize = queryMetadata.flatMap(_.userActionsSize),\n      userActionsContainsExplicitSignals =\n        queryMetadata.flatMap(_.userActionsContainsExplicitSignals)\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model/ScoredTweetsQuery.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.model\n\nimport com.twitter.home_mixer.model.request.DeviceContext\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.model.request.HasSeenTweetIds\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.request._\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorStatus\nimport com.twitter.timelines.configapi.Params\n\ncase class ScoredTweetsQuery(\n  override val params: Params,\n  override val clientContext: ClientContext,\n  override val pipelineCursor: Option[UrtOrderedCursor],\n  override val requestedMaxResults: Option[Int],\n  override val debugOptions: Option[DebugOptions],\n  override val features: Option[FeatureMap],\n  override val deviceContext: Option[DeviceContext],\n  override val seenTweetIds: Option[Seq[Long]],\n  override val qualityFactorStatus: Option[QualityFactorStatus],\n  override val product: Product,\n  videoType: Option[t.VideoType] = None,\n  pinnedRelatedTweetIds: Option[Seq[Long]] = None,\n  scorePinnedTweetsOnly: Option[Boolean] = None,\n  immersiveClientMetadata: Option[t.ImmersiveClientMetadata] = None)\n    extends PipelineQuery\n    with HasPipelineCursor[UrtOrderedCursor]\n    with HasDeviceContext\n    with HasSeenTweetIds\n    with HasQualityFactorStatus {\n\n  override def withFeatureMap(features: FeatureMap): ScoredTweetsQuery =\n    copy(features = Some(features))\n\n  override def withQualityFactorStatus(\n    qualityFactorStatus: QualityFactorStatus\n  ): ScoredTweetsQuery = copy(qualityFactorStatus = Some(qualityFactorStatus))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model/ScoredTweetsResponse.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.model\n\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.model.marshalling.HasLength\n\ncase class ScoredTweetsResponse(\n  scoredTweets: Seq[CandidateWithDetails],\n  queryMetadata: Option[QueryMetadata] = None)\n    extends HasMarshalling\n    with HasLength {\n  override def length: Int = scoredTweets.length\n}\n\ncase class QueryMetadata(\n  userActionsSize: Option[Int] = None,\n  userActionsContainsExplicitSignals: Option[Boolean] = None)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param/decider\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param/ScoredTweetsParam.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.param\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.param.decider.DeciderKey\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.decider.BooleanDeciderParam\nimport com.twitter.timelines.configapi.decider.DeciderBoundedParam\nimport com.twitter.util.Duration\n\nobject ScoredTweetsParam {\n  val SupportedClientFSName = \"scored_tweets_supported_client\"\n\n  object CandidateSourceParams {\n    object EnableCommunitiesCandidateSourceParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_enable_earlybird_communities_candidate_source\",\n          default = false\n        )\n\n    object EnableInNetworkCandidateSourceParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_enable_in_network_candidate_source\",\n          default = true\n        )\n\n    object EnableStaticSourceParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_enable_static_source\",\n          default = false\n        )\n\n    object EnableUTEGCandidateSourceParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_enable_uteg_candidate_source\",\n          default = true\n        )\n\n    object InNetworkIncludeRepliesParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_in_network_include_replies\",\n          default = true\n        )\n\n    object InNetworkIncludeRetweetsParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_in_network_include_retweets\",\n          default = true\n        )\n\n    object InNetworkIncludeExtendedRepliesParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_in_network_include_extended_replies\",\n          default = true\n        )\n  }\n\n  object EnableBackfillCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_backfill_candidate_pipeline\",\n        default = true\n      )\n\n  object EnableContentExplorationCandidatePipelineParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_content_exploration_candidate_pipeline\",\n        default = false\n      )\n\n  object ContentExplorationCandidateVersionParam\n      extends FSParam[String](\n        name = \"scored_tweets_enable_content_exploration_candidate_version\",\n        default = \"v1_\"\n      )\n\n  object EnableContentExplorationScoreScribingParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_content_exploration_score_scribing\",\n        default = false\n      )\n\n  object EnableContentExplorationCandidateMaxCountParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_content_exploration_candidate_max_count\",\n        default = false\n      )\n\n  object EnableContentExplorationSimclusterColdPostsCandidateBoostingParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_content_exploration_simcluster_cold_posts_candidate_boosting\",\n        default = false\n      )\n\n  object ContentExplorationBoostPosParam\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_content_exploration_boost_pos\",\n        default = 100,\n        min = 0,\n        max = 1000\n      )\n\n  object EnableDeepRetrievalMixedCandidateBoostingParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_deep_retrieval_mixed_candidate_boosting\",\n        default = false\n      )\n\n  object CategoryColdStartTierOneProbabilityParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_category_cold_start_tier_one_probability\",\n        default = 0,\n        min = 0,\n        max = 1\n      )\n\n  object CategoryColdStartProbabilisticReturnParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_category_cold_start_probabilistic_return\",\n        default = 0,\n        min = 0,\n        max = 1\n      )\n\n  object ContentExplorationViewerMaxFollowersParam\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_content_exploration_viewer_max_followers\",\n        default = 100000,\n        min = 0,\n        max = 1000000000\n      )\n\n  object EnableContentExplorationMixedCandidateBoostingParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_content_exploration_mixed_candidate_boosting\",\n        default = false\n      )\n\n  object DeepRetrievalBoostPosParam\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_deep_retrieval_boost_pos\",\n        default = 100,\n        min = 0,\n        max = 1000\n      )\n\n  object DeepRetrievalI2iProbabilityParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_deep_retrieval_i2i_probability\",\n        default = 0,\n        min = 0,\n        max = 1\n      )\n\n  object FetchParams {\n    object FRSMaxTweetsToFetchParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_frs_max_tweets_to_fetch\",\n          default = 100,\n          min = 0,\n          max = 10000\n        )\n\n    object InNetworkMaxTweetsToFetchParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_in_network_max_tweets_to_fetch\",\n          default = 600,\n          min = 0,\n          max = 10000\n        )\n\n    object TweetMixerMaxTweetsToFetchParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_tweet_mixer_max_tweets_to_fetch\",\n          default = 400,\n          min = 0,\n          max = 10000\n        )\n\n    object UTEGMaxTweetsToFetchParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_uteg_max_tweets_to_fetch\",\n          default = 300,\n          min = 0,\n          max = 10000\n        )\n  }\n\n  object QualityFactor {\n\n    object InNetworkMaxTweetsToScoreParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_quality_factor_earlybird_max_tweets_to_score\",\n          default = 600,\n          min = 0,\n          max = 10000\n        )\n\n    object UtegMaxTweetsToScoreParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_quality_factor_uteg_max_tweets_to_score\",\n          default = 300,\n          min = 0,\n          max = 10000\n        )\n\n    object TweetMixerMaxTweetsToScoreParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_quality_factor_tweet_mixer_max_tweets_to_score\",\n          default = 400,\n          min = 0,\n          max = 10000\n        )\n\n    object ListsMaxTweetsToScoreParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_quality_factor_lists_max_tweets_to_score\",\n          default = 100,\n          min = 0,\n          max = 100\n        )\n\n    object BackfillMaxTweetsToScoreParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_quality_factor_backfill_max_tweets_to_score\",\n          default = 200,\n          min = 0,\n          max = 10000\n        )\n\n    object CommunitiesMaxTweetsToScoreParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_quality_factor_communities_max_tweets_to_score\",\n          default = 100,\n          min = 0,\n          max = 10000\n        )\n  }\n\n  object ServerMaxResultsParam\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_server_max_results\",\n        default = 50,\n        min = 1,\n        max = 1500\n      )\n\n  object DefaultRequestedMaxResultsParam\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_default_requested_max_results\",\n        default = 50,\n        min = 1,\n        max = 1500\n      )\n\n  object CachedScoredTweets {\n    object TTLParam\n        extends FSBoundedParam[Duration](\n          name = \"scored_tweets_cached_scored_tweets_ttl_minutes\",\n          default = 3.minutes,\n          min = 0.minute,\n          max = 60.minutes\n        )\n        with HasDurationConversion {\n      override val durationConversion: DurationConversion = DurationConversion.FromMinutes\n    }\n\n    object MinCachedTweetsParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_cached_scored_tweets_min_cached_tweets\",\n          default = 30,\n          min = 0,\n          max = 1000\n        )\n  }\n\n  object FeatureHydration {\n\n    object EnableRealTimeEntityRealGraphFeaturesParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_real_time_entity_real_graph_features\",\n          default = false\n        )\n\n    object EnableFollowedUserScoreBackfillFeaturesParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_followed_user_score_backfill_features\",\n          default = false\n        )\n\n    object EnableSgsMutuallyFollowedUserFeaturesParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_sgs_mutually_followed_user_features\",\n          default = false\n        )\n\n    object EnableTopicSocialProofFeaturesParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_topic_social_proof_features\",\n          default = false\n        )\n\n    object EnableMediaClusterFeatureHydrationParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_media_cluster_feature\",\n          default = false\n        )\n\n    object EnableMediaCompletionRateFeatureHydrationParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_media_completion_rate_feature\",\n          default = false\n        )\n\n    object EnableImageClusterFeatureHydrationParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_image_cluster_feature\",\n          default = false\n        )\n\n    object EnableClipImagesClusterIdFeatureHydrationParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_clip_images_cluster_id_feature\",\n          default = false\n        )\n\n    object EnableMultiModalEmbeddingsFeatureHydratorParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_multi_modal_embeddings_feature_hydrator\",\n          default = false\n        )\n\n    object EnableTweetTextV8EmbeddingFeatureParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_tweet_text_v8_embedding_feature\",\n          default = false\n        )\n\n    object EnableUserEngagedLanguagesFeaturesParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_user_engaged_languages_features\",\n          default = false\n        )\n\n    object EnableUserIdentifierFeaturesParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_user_identifier_features\",\n          default = false\n        )\n\n    object EnableUserHistoryEventsFeaturesParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_user_history_events_features\",\n          default = false\n        )\n\n    object EnableUserActionsFeatureParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_user_actions_feature\",\n          default = false\n        )\n\n    object EnableDenseUserActionsHydrationParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_dense_user_actions_feature\",\n          default = false\n        )\n\n    object EnableMediaClusterDecayParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_media_cluster_decay\",\n          default = false\n        )\n\n    object EnableImageClusterDecayParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_feature_hydration_enable_image_cluster_decay\",\n          default = false\n        )\n\n    object UserHistoryEventsLengthParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_feature_hydration_user_history_events_length\",\n          default = 50,\n          min = 0,\n          max = 1000\n        )\n\n    object TwhinDiversityRescoringWeightParam\n        extends FSBoundedParam[Double](\n          name = \"scored_tweets_feature_hydration_twhin_diversity_rescoring_weight\",\n          default = 0.0,\n          min = -100.0,\n          max = 100.0\n        )\n\n    object TwhinDiversityRescoringRatioParam\n        extends FSBoundedParam[Double](\n          name = \"scored_tweets_feature_hydration_twhin_diversity_rescoring_ratio\",\n          default = 0.0,\n          min = 0.0,\n          max = 1.0\n        )\n\n    object CategoryDiversityRescoringWeightParam\n        extends FSBoundedParam[Double](\n          name = \"scored_tweets_feature_hydration_category_diversity_rescoring_weight\",\n          default = 0.0,\n          min = -1.0,\n          max = 1.0\n        )\n\n    object CategoryDiversityKParam\n        extends FSBoundedParam[Int](\n          name = \"scored_tweets_feature_hydration_category_diversity_k\",\n          default = 5,\n          min = 1,\n          max = 100\n        )\n  }\n\n  object ControlAiShowLessScaleFactorParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_control_ai_show_less_scale_factor\",\n        default = 0.05,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object ControlAiShowMoreScaleFactorParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_control_ai_show_more_scale_factor\",\n        default = 20.0,\n        min = 0.0,\n        max = 1000.0\n      )\n\n  object ControlAiEmbeddingSimilarityThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_control_ai_embedding_similarity_threshold\",\n        default = 0.67,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object CreatorInNetworkMultiplierParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_creator_in_network_multiplier\",\n        default = 1.0,\n        min = 0.0,\n        max = 100.0\n      )\n\n  object CreatorOutOfNetworkMultiplierParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_creator_out_of_network_multiplier\",\n        default = 1.0,\n        min = 0.0,\n        max = 100.0\n      )\n\n  object OutOfNetworkScaleFactorParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_out_of_network_scale_factor\",\n        default = 0.75,\n        min = 0.0,\n        max = 100.0\n      )\n\n  object ReplyScaleFactorParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_reply_scale_factor\",\n        default = 0.75,\n        min = 0.0,\n        max = 100.0\n      )\n\n  object EnableMediaDedupingParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_media_deduping\",\n        default = false\n      )\n\n  object EnableMediaClusterDedupingParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_media_cluster_deduping\",\n        default = false\n      )\n\n  object EnableClipImageClusterDedupingParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_clip_image_cluster_deduping\",\n        default = false\n      )\n\n  object EnableScribeScoredCandidatesParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_scribing\",\n        default = false\n      )\n\n  object EnableCacheRetrievalSignalParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_cache_retrieval_signal\",\n        default = false\n      )\n\n  object EnableCacheRequestInfoParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_cache_request_info_signal\",\n        default = false\n      )\n\n  object EnableScoredPhoenixCandidatesKafkaSideEffectParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_scored_phoenix_candidates_kafka_side_effect\",\n        default = false\n      )\n\n  object LiveContentScaleFactorParam\n      extends DeciderBoundedParam[Double](\n        DeciderKey.LiveSpacesFactor,\n        default = 1.0,\n        min = 0.1,\n        max = 10000.0\n      )\n\n  object EarlybirdTensorflowModel {\n    object InNetworkParam\n        extends FSParam[String](\n          name = \"scored_tweets_in_network_earlybird_tensorflow_model\",\n          default = \"timelines_recap_replica\"\n        )\n\n    object FrsParam\n        extends FSParam[String](\n          name = \"scored_tweets_frs_earlybird_tensorflow_model\",\n          default = \"timelines_rectweet_replica\"\n        )\n\n    object UtegParam\n        extends FSParam[String](\n          name = \"scored_tweets_uteg_earlybird_tensorflow_model\",\n          default = \"timelines_rectweet_replica\"\n        )\n  }\n\n  object MtlNormalization {\n\n    object EnableMtlNormalizationParam\n        extends FSParam[Boolean](\n          name = \"scored_tweets_enable_mtl_normalization\",\n          default = true\n        )\n\n    object AlphaParam\n        extends DeciderBoundedParam[Double](\n          decider = DeciderKey.MtlNormalizationAlpha,\n          default = 100.0,\n          min = 0.0,\n          max = 100.0\n        )\n\n    object BetaParam\n        extends FSBoundedParam[Long](\n          name = \"scored_tweets_mtl_normalization_beta\",\n          default = 100000000L,\n          min = 0L,\n          max = 1000000000L\n        )\n\n    object GammaParam\n        extends FSBoundedParam[Long](\n          name = \"scored_tweets_mtl_normalization_gamma\",\n          default = 5000000L,\n          min = 1L,\n          max = 100000000L\n        )\n  }\n\n  object EarlybirdMaxResultsPerPartitionParam\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_earlybird_max_results_per_partition\",\n        default = 300,\n        min = 0,\n        max = 1000\n      )\n\n  object TweetMixerRankingModeForStatsRecallAtKParam\n      extends FSParam[String](\n        name = \"scored_tweets_tweet_mixer_ranking_mode_for_stats_recall_at_k\",\n        default = \"Interleave\"\n      )\n\n  object EnablePublishCommonFeaturesKafkaDeciderParam\n      extends BooleanDeciderParam(decider = DeciderKey.EnablePublishCommonFeaturesKafka)\n\n  object AuthorDiversityDecayFactor\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_author_diversity_decay_factor\",\n        default = 0.5,\n        min = 0.0,\n        max = 1.0,\n      )\n\n  object AuthorDiversityOutNetworkDecayFactor\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_author_diversity_out_network_decay_factor\",\n        default = 0.5,\n        min = 0.0,\n        max = 1.0,\n      )\n  object AuthorDiversityInNetworkDecayFactor\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_author_diversity_in_network_decay_factor\",\n        default = 0.5,\n        min = 0.0,\n        max = 1.0,\n      )\n\n  object AuthorDiversityFloor\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_author_diversity_floor\",\n        default = 0.25,\n        min = 0.0,\n        max = 1.0,\n      )\n\n  object AuthorDiversityOutNetworkFloor\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_author_diversity_out_network_floor\",\n        default = 0.25,\n        min = 0.0,\n        max = 1.0,\n      )\n  object AuthorDiversityInNetworkFloor\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_author_diversity_in_network_floor\",\n        default = 0.25,\n        min = 0.0,\n        max = 1.0,\n      )\n\n  object SmallFollowGraphAuthorDiversityDecayFactor\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_small_follow_graph_author_diversity_decay_factor\",\n        default = 0.5,\n        min = 0.0,\n        max = 1.0,\n      )\n\n  object SmallFollowGraphAuthorDiversityFloor\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_small_follow_graph_author_diversity_floor\",\n        default = 0.25,\n        min = 0.0,\n        max = 1.0,\n      )\n\n  object EnableDeepRetrievalMaxCountParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_deep_retrieval_max_count\",\n        default = false\n      )\n\n  object DeepRetrievalMaxCountParam\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_deep_retrieval_max_count\",\n        default = 1,\n        min = 0,\n        max = 1000\n      )\n\n  object EnableEvergreenDeepRetrievalMaxCountParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_evergreen_deep_retrieval_max_count\",\n        default = false\n      )\n\n  object EvergreenDeepRetrievalMaxCountParam\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_evergreen_deep_retrieval_max_count\",\n        default = 1,\n        min = 0,\n        max = 1000\n      )\n\n  object EnableEvergreenDeepRetrievalCrossBorderMaxCountParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_evergreen_deep_retrieval_cross_border_max_count\",\n        default = false\n      )\n\n  object EvergreenDeepRetrievalCrossBorderMaxCountParam\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_evergreen_deep_retrieval_cross_border_max_count\",\n        default = 1,\n        min = 0,\n        max = 1000\n      )\n  object EnableControlAiParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_control_ai\",\n        default = false\n      )\n\n  object EnableHeartbeatOptimizerWeightsParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_heartbeat_optimizer_weights\",\n        default = false\n      )\n\n  object HeartbeatOptimizerParamsMHPkey\n      extends FSParam[String](\n        name = \"scored_tweets_heartbeat_optimizer_params_mh_pkey\",\n        default = \"0\"\n      )\n\n  object EnableHeuristicScoringPipeline\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_heuristic_scoring_pipeline\",\n        default = true\n      )\n\n  object EnablePhoenixScoreParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_phoenix_score\",\n        default = false\n      )\n\n  object EnablePhoenixRescoreParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_phoenix_rescore\",\n        default = false\n      )\n\n  object EnableColdStartFilterParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_cold_start_filter\",\n        default = false\n      )\n\n  object EnableImpressionBasedAuthorDecay\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_impression_based_author_decay\",\n        default = false\n      )\n\n  object EnableCandidateSourceDiversityDecay\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_candidate_source_diversity_decay\",\n        default = false\n      )\n\n  object CandidateSourceDiversityDecayFactor\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_candidate_source_diversity_decay_factor\",\n        default = 0.9,\n        min = 0.0,\n        max = 1.0,\n      )\n\n  object CandidateSourceDiversityFloor\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_candidate_source_diversity_floor\",\n        default = 0.8,\n        min = 0.0,\n        max = 1.0,\n      )\n\n  object EnableHomeMixerFeaturesService\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_home_mixer_features_service\",\n        default = false\n      )\n\n  object GrokSlopScoreDecayValueParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_grok_slop_score_decay_value\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object MultiModalEmbeddingRescorerGammaParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_multi_modal_embedding_rescorer_gamma\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object MultiModalEmbeddingRescorerMinScoreParam\n      extends FSBoundedParam[Double](\n        name = \"scored_tweets_multi_modal_embedding_rescorer_min_score\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object EnableContentFeatureFromTesService\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_home_mixer_feature_tweet_entity_service\",\n        default = false\n      )\n\n  object EnableLowSignalUserCheck\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_low_signal_user_check\",\n        default = false\n      )\n\n  object LowSignalUserMaxSignalCount\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_low_signal_user_max_signal_count\",\n        default = 10,\n        min = 0,\n        max = 100000\n      )\n\n  object MultiModalEmbeddingRescorerNumCandidatesParam\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_multi_modal_embedding_rescorer_num_candidates\",\n        default = 70,\n        min = 1,\n        max = 500\n      )\n\n  object EnableScoredCandidateFeatureKeysKafkaPublishingParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_scored_candidate_feature_keys_kafka_publishing\",\n        default = false\n      )\n\n  object EnableEarlybirdCommunitiesQueryLinearRankingParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_earlybird_communities_query_linear_ranking\",\n        default = false\n      )\n\n  object EarlyBirdCommunitiesMaxSearchResultsParam\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_earlybird_communities_max_search_results\",\n        default = 100,\n        min = 0,\n        max = 1000\n      )\n\n  object EnableRecentFeedbackCheckParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_recent_feedback_check\",\n        default = false\n      )\n\n  object ScribedScoredCandidateNumParam\n      extends FSBoundedParam[Int](\n        name = \"scored_tweets_scribed_scored_candidate_num\",\n        default = 2,\n        min = 0,\n        max = 2000\n      )\n\n  object EnableRecentEngagementCacheRefreshParam\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_recent_engagement_cache_refresh\",\n        default = false\n      )\n\n  object EnableLanguageFilter\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_language_filter\",\n        default = false\n      )\n\n  object EnableGrokAutoTranslateLanguageFilter\n      extends FSParam[Boolean](\n        name = \"scored_tweets_enable_grok_auto_translate_language_filter\",\n        default = false\n      )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param/ScoredTweetsParamConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.param\n\nimport com.twitter.home_mixer.param.decider.DeciderKey\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam._\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.servo.decider.DeciderKeyName\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.Param\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoredTweetsParamConfig @Inject() () extends ProductParamConfig {\n  override val enabledDeciderKey: DeciderKeyName = DeciderKey.EnableScoredTweetsProduct\n  override val supportedClientFSName: String = SupportedClientFSName\n\n  override val booleanDeciderOverrides = Seq(EnablePublishCommonFeaturesKafkaDeciderParam)\n\n  override val boundedDoubleDeciderOverrides = Seq(\n    LiveContentScaleFactorParam,\n    MtlNormalization.AlphaParam\n  )\n\n  override val booleanFSOverrides = Seq(\n    CandidateSourceParams.EnableCommunitiesCandidateSourceParam,\n    CandidateSourceParams.EnableInNetworkCandidateSourceParam,\n    CandidateSourceParams.InNetworkIncludeRepliesParam,\n    CandidateSourceParams.InNetworkIncludeRetweetsParam,\n    CandidateSourceParams.InNetworkIncludeExtendedRepliesParam,\n    CandidateSourceParams.EnableUTEGCandidateSourceParam,\n    CandidateSourceParams.EnableStaticSourceParam,\n    EnableBackfillCandidatePipelineParam,\n    EnableContentExplorationCandidatePipelineParam,\n    EnableContentExplorationSimclusterColdPostsCandidateBoostingParam,\n    EnableContentExplorationCandidateMaxCountParam,\n    EnableContentExplorationScoreScribingParam,\n    EnableContentExplorationMixedCandidateBoostingParam,\n    EnableDeepRetrievalMixedCandidateBoostingParam,\n    EnableScribeScoredCandidatesParam,\n    EnableCacheRetrievalSignalParam,\n    EnableCacheRequestInfoParam,\n    EnableScoredPhoenixCandidatesKafkaSideEffectParam,\n    FeatureHydration.EnableFollowedUserScoreBackfillFeaturesParam,\n    FeatureHydration.EnableRealTimeEntityRealGraphFeaturesParam,\n    FeatureHydration.EnableTopicSocialProofFeaturesParam,\n    FeatureHydration.EnableTweetTextV8EmbeddingFeatureParam,\n    FeatureHydration.EnableMediaClusterFeatureHydrationParam,\n    FeatureHydration.EnableMediaCompletionRateFeatureHydrationParam,\n    FeatureHydration.EnableClipImagesClusterIdFeatureHydrationParam,\n    FeatureHydration.EnableMultiModalEmbeddingsFeatureHydratorParam,\n    FeatureHydration.EnableUserEngagedLanguagesFeaturesParam,\n    FeatureHydration.EnableUserIdentifierFeaturesParam,\n    FeatureHydration.EnableUserHistoryEventsFeaturesParam,\n    FeatureHydration.EnableUserActionsFeatureParam,\n    FeatureHydration.EnableDenseUserActionsHydrationParam,\n    FeatureHydration.EnableMediaClusterDecayParam,\n    FeatureHydration.EnableImageClusterDecayParam,\n    EnableControlAiParam,\n    EnableHeartbeatOptimizerWeightsParam,\n    EnableHeuristicScoringPipeline,\n    EnablePhoenixScoreParam,\n    EnablePhoenixRescoreParam,\n    EnableColdStartFilterParam,\n    EnableImpressionBasedAuthorDecay,\n    EnableCandidateSourceDiversityDecay,\n    EnableHomeMixerFeaturesService,\n    EnableContentFeatureFromTesService,\n    EnableLowSignalUserCheck,\n    EnableDeepRetrievalMaxCountParam,\n    EnableEvergreenDeepRetrievalMaxCountParam,\n    EnableEvergreenDeepRetrievalCrossBorderMaxCountParam,\n    EnableScoredCandidateFeatureKeysKafkaPublishingParam,\n    EnableEarlybirdCommunitiesQueryLinearRankingParam,\n    EnableRecentFeedbackCheckParam,\n    EnableMediaDedupingParam,\n    EnableMediaClusterDedupingParam,\n    EnableClipImageClusterDedupingParam,\n    MtlNormalization.EnableMtlNormalizationParam,\n    EnableRecentEngagementCacheRefreshParam,\n    EnableLanguageFilter,\n    EnableGrokAutoTranslateLanguageFilter\n  )\n\n  override val boundedIntFSOverrides = Seq(\n    CachedScoredTweets.MinCachedTweetsParam,\n    EarlybirdMaxResultsPerPartitionParam,\n    ServerMaxResultsParam,\n    DefaultRequestedMaxResultsParam,\n    ContentExplorationBoostPosParam,\n    ContentExplorationViewerMaxFollowersParam,\n    DeepRetrievalBoostPosParam,\n    FeatureHydration.UserHistoryEventsLengthParam,\n    FetchParams.FRSMaxTweetsToFetchParam,\n    FetchParams.InNetworkMaxTweetsToFetchParam,\n    FetchParams.TweetMixerMaxTweetsToFetchParam,\n    FetchParams.UTEGMaxTweetsToFetchParam,\n    QualityFactor.BackfillMaxTweetsToScoreParam,\n    QualityFactor.TweetMixerMaxTweetsToScoreParam,\n    QualityFactor.InNetworkMaxTweetsToScoreParam,\n    QualityFactor.ListsMaxTweetsToScoreParam,\n    QualityFactor.UtegMaxTweetsToScoreParam,\n    QualityFactor.CommunitiesMaxTweetsToScoreParam,\n    DeepRetrievalMaxCountParam,\n    EvergreenDeepRetrievalMaxCountParam,\n    EvergreenDeepRetrievalCrossBorderMaxCountParam,\n    ScribedScoredCandidateNumParam,\n    LowSignalUserMaxSignalCount,\n    EarlyBirdCommunitiesMaxSearchResultsParam,\n    FeatureHydration.CategoryDiversityKParam,\n    MultiModalEmbeddingRescorerNumCandidatesParam\n  )\n\n  override val boundedLongFSOverrides = Seq(\n    MtlNormalization.BetaParam,\n    MtlNormalization.GammaParam,\n  )\n\n  override val boundedDurationFSOverrides = Seq(\n    CachedScoredTweets.TTLParam,\n  )\n\n  override val stringFSOverrides = Seq(\n    ContentExplorationCandidateVersionParam,\n    EarlybirdTensorflowModel.InNetworkParam,\n    EarlybirdTensorflowModel.FrsParam,\n    EarlybirdTensorflowModel.UtegParam,\n    HeartbeatOptimizerParamsMHPkey,\n    TweetMixerRankingModeForStatsRecallAtKParam,\n  )\n\n  override val boundedDoubleFSOverrides = Seq(\n    CategoryColdStartTierOneProbabilityParam,\n    CategoryColdStartProbabilisticReturnParam,\n    ControlAiShowLessScaleFactorParam,\n    ControlAiShowMoreScaleFactorParam,\n    ControlAiEmbeddingSimilarityThresholdParam,\n    CreatorInNetworkMultiplierParam,\n    CreatorOutOfNetworkMultiplierParam,\n    DeepRetrievalI2iProbabilityParam,\n    OutOfNetworkScaleFactorParam,\n    ReplyScaleFactorParam,\n    AuthorDiversityDecayFactor,\n    AuthorDiversityOutNetworkDecayFactor,\n    AuthorDiversityInNetworkDecayFactor,\n    SmallFollowGraphAuthorDiversityDecayFactor,\n    AuthorDiversityFloor,\n    AuthorDiversityOutNetworkFloor,\n    AuthorDiversityInNetworkFloor,\n    CandidateSourceDiversityDecayFactor,\n    CandidateSourceDiversityFloor,\n    SmallFollowGraphAuthorDiversityFloor,\n    FeatureHydration.TwhinDiversityRescoringWeightParam,\n    FeatureHydration.TwhinDiversityRescoringRatioParam,\n    FeatureHydration.CategoryDiversityRescoringWeightParam,\n    GrokSlopScoreDecayValueParam,\n    MultiModalEmbeddingRescorerGammaParam,\n    MultiModalEmbeddingRescorerMinScoreParam\n  )\n\n  override def longSetFSOverrides: Seq[Param[Set[Long]] with FSName] = Seq.empty\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/earlybird\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"src/thrift/com/twitter/timelineranker:thrift-scala\",\n        \"timelineranker/common/src/main/scala/com/twitter/timelineranker/model\",\n        \"timelines/src/main/scala/com/twitter/timelines/common/model\",\n        \"timelines/src/main/scala/com/twitter/timelines/earlybird/common/options\",\n        \"timelines/src/main/scala/com/twitter/timelines/earlybird/common/utils\",\n        \"timelines/src/main/scala/com/twitter/timelines/model/candidate\",\n        \"timelineservice/common:model\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/ContentExplorationQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.query_transformer\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.UserSubLevelCategoriesFeature\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.ContentExplorationCandidateVersionParam\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\n\ncase class ContentExplorationQueryRequest(\n  userCategories: Seq[(String, Double)],\n  version: String)\n\nobject ContentExplorationQueryTransformer\n    extends CandidatePipelineQueryTransformer[\n      ScoredTweetsQuery,\n      ContentExplorationQueryRequest\n    ] {\n\n  override def transform(query: ScoredTweetsQuery): ContentExplorationQueryRequest = {\n    ContentExplorationQueryRequest(\n      userCategories = query.features.get\n        .getOrElse(UserSubLevelCategoriesFeature, Seq.empty[(Long, Double)])\n        .map { case (id, score) => (id.toString, score) },\n      version = query.params(ContentExplorationCandidateVersionParam)\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/TimelineRankerFrsQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.query_transformer\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.core_workflows.user_model.{thriftscala => um}\nimport com.twitter.home_mixer.functional_component.feature_hydrator.FrsSeedUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserStateFeature\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FetchParams\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.TimelineRankerFrsQueryTransformer._\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.timelineranker.{thriftscala => t}\nimport com.twitter.timelines.common.model.TweetKindOption\nimport com.twitter.timelines.model.candidate.CandidateTweetSourceId\n\nobject TimelineRankerFrsQueryTransformer {\n  private val DefaultSinceDuration = 24.hours\n  private val ExpandedSinceDuration = 48.hours\n  private val tweetKindOptions: TweetKindOption.ValueSet =\n    TweetKindOption(includeOriginalTweetsAndQuotes = true)\n\n  private val UserStatesForExtendedSinceDuration: Set[um.UserState] = Set(\n    um.UserState.Light,\n    um.UserState.MediumNonTweeter,\n    um.UserState.MediumTweeter,\n    um.UserState.NearZero,\n    um.UserState.New,\n    um.UserState.VeryLight\n  )\n}\n\ncase class TimelineRankerFrsQueryTransformer[\n  Query <: PipelineQuery with HasQualityFactorStatus with HasDeviceContext\n](\n  override val candidatePipelineIdentifier: CandidatePipelineIdentifier)\n    extends CandidatePipelineQueryTransformer[Query, t.RecapQuery]\n    with TimelineRankerQueryTransformer[Query] {\n\n  override val candidateTweetSourceId = CandidateTweetSourceId.FrsTweet\n  override val options = tweetKindOptions\n  override def maxTweetsToFetch(query: Query): Int =\n    query.params(FetchParams.FRSMaxTweetsToFetchParam)\n\n  override def getTensorflowModel(query: Query): Option[String] = {\n    Some(query.params(ScoredTweetsParam.EarlybirdTensorflowModel.FrsParam))\n  }\n\n  override def seedAuthorIds(query: Query): Option[Seq[Long]] = {\n    query.features.flatMap(_.getOrElse(FrsSeedUserIdsFeature, None))\n  }\n\n  override def transform(input: Query): t.RecapQuery = {\n    val userState = input.features.get.getOrElse(UserStateFeature, None)\n\n    val sinceDuration =\n      if (userState.exists(UserStatesForExtendedSinceDuration.contains)) ExpandedSinceDuration\n      else DefaultSinceDuration\n\n    buildTimelineRankerQuery(input, sinceDuration).toThriftRecapQuery\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/TimelineRankerInNetworkQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.query_transformer\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.core_workflows.user_model.{thriftscala => um}\nimport com.twitter.home_mixer.model.HomeFeatures.UserStateFeature\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FetchParams\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.TimelineRankerInNetworkQueryTransformer._\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.timelineranker.{thriftscala => t}\nimport com.twitter.timelines.common.model.TweetKindOption\nimport com.twitter.timelines.model.candidate.CandidateTweetSourceId\n\nobject TimelineRankerInNetworkQueryTransformer {\n  private val DefaultSinceDuration = 24.hours\n  private val ExpandedSinceDuration = 48.hours\n\n  private val tweetKindOptions: TweetKindOption.ValueSet = TweetKindOption(\n    includeReplies = true,\n    includeRetweets = true,\n    includeOriginalTweetsAndQuotes = true,\n    includeExtendedReplies = true\n  )\n\n  private val UserStatesForExtendedSinceDuration: Set[um.UserState] = Set(\n    um.UserState.Light,\n    um.UserState.MediumNonTweeter,\n    um.UserState.MediumTweeter,\n    um.UserState.NearZero,\n    um.UserState.New,\n    um.UserState.VeryLight\n  )\n}\n\ncase class TimelineRankerInNetworkQueryTransformer[\n  Query <: PipelineQuery with HasQualityFactorStatus with HasDeviceContext\n](\n  override val candidatePipelineIdentifier: CandidatePipelineIdentifier)\n    extends CandidatePipelineQueryTransformer[Query, t.RecapQuery]\n    with TimelineRankerQueryTransformer[Query] {\n\n  override val candidateTweetSourceId = CandidateTweetSourceId.RecycledTweet\n  override val options = tweetKindOptions\n\n  override def maxTweetsToFetch(query: Query): Int =\n    query.params(FetchParams.InNetworkMaxTweetsToFetchParam)\n\n  override def getTensorflowModel(query: Query): Option[String] = {\n    Some(query.params(ScoredTweetsParam.EarlybirdTensorflowModel.InNetworkParam))\n  }\n\n  override def transform(input: Query): t.RecapQuery = {\n    val userState = input.features.get.getOrElse(UserStateFeature, None)\n\n    val sinceDuration =\n      if (userState.exists(UserStatesForExtendedSinceDuration.contains)) ExpandedSinceDuration\n      else DefaultSinceDuration\n\n    buildTimelineRankerQuery(input, sinceDuration).toThriftRecapQuery\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/TimelineRankerQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.query_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EarlybirdMaxResultsPerPartitionParam\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.TimelineRankerQueryTransformer._\nimport com.twitter.home_mixer.util.CachedScoredTweetsHelper\nimport com.twitter.home_mixer.util.earlybird.EarlybirdRequestUtil\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.timelineranker.{model => tlr}\nimport com.twitter.timelines.common.model.TweetKindOption\nimport com.twitter.timelines.earlybird.common.options.EarlybirdOptions\nimport com.twitter.timelines.earlybird.common.options.EarlybirdScoringModelConfig\nimport com.twitter.timelines.earlybird.common.utils.SearchOperator\nimport com.twitter.timelines.model.UserId\nimport com.twitter.timelines.model.candidate.CandidateTweetSourceId\nimport com.twitter.timelines.util.SnowflakeSortIndexHelper\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\n\nobject TimelineRankerQueryTransformer {\n\n  /**\n   * Specifies the maximum number of excluded tweet ids to include in the search index query.\n   * Earlybird's named multi term disjunction map feature supports up to 1500 tweet ids.\n   */\n  private val EarlybirdMaxExcludedTweets = 1500\n\n  /**\n   * Maximum number of query hits each earlybird shard is allowed to accumulate before\n   * early-terminating the query and reducing the hits to MaxNumEarlybirdResults.\n   */\n  private val EarlybirdMaxHits = 1000\n}\n\ntrait TimelineRankerQueryTransformer[\n  Query <: PipelineQuery with HasQualityFactorStatus with HasDeviceContext] {\n  def maxTweetsToFetch(query: Query): Int\n  def options: TweetKindOption.ValueSet = TweetKindOption.Default\n  def candidateTweetSourceId: CandidateTweetSourceId.Value\n  def utegLikedByTweetsOptions(query: Query): Option[tlr.UtegLikedByTweetsOptions] = None\n  def seedAuthorIds(query: Query): Option[Seq[Long]] = None\n  def candidatePipelineIdentifier: CandidatePipelineIdentifier\n  def earlybirdModels: Seq[EarlybirdScoringModelConfig] =\n    EarlybirdRequestUtil.EarlybirdScoringModels.UnifiedEngagementProd\n  def getTensorflowModel(query: Query): Option[String] = None\n\n  def buildTimelineRankerQuery(query: Query, sinceDuration: Duration): tlr.RecapQuery = {\n    val sinceTime: Time = sinceDuration.ago\n    val untilTime: Time = Time.now\n\n    val fromTweetIdExclusive = SnowflakeSortIndexHelper.timestampToFakeId(sinceTime)\n    val toTweetIdExclusive = SnowflakeSortIndexHelper.timestampToFakeId(untilTime)\n    val range = tlr.TweetIdRange(Some(fromTweetIdExclusive), Some(toTweetIdExclusive))\n\n    val excludedTweetIds = query.features.map { featureMap =>\n      CachedScoredTweetsHelper.tweetImpressionsAndCachedScoredTweetsInRange(\n        featureMap,\n        candidatePipelineIdentifier,\n        EarlybirdMaxExcludedTweets,\n        sinceTime,\n        untilTime)\n    }\n\n    val authorScoreMap = query.features\n      .map(_.getOrElse(RealGraphInNetworkScoresFeature, Map.empty[UserId, Double]))\n      .getOrElse(Map.empty)\n\n    val deviceContext =\n      query.deviceContext.map(_.toTimelineServiceDeviceContext(query.clientContext))\n\n    val tensorflowModel = getTensorflowModel(query)\n\n    val earlyBirdOptions = EarlybirdOptions(\n      maxNumHitsPerShard = EarlybirdMaxHits,\n      maxNumResultsPerShard = query.params(EarlybirdMaxResultsPerPartitionParam),\n      models = earlybirdModels,\n      authorScoreMap = authorScoreMap,\n      skipVeryRecentTweets = true,\n      tensorflowModel = tensorflowModel\n    )\n\n    tlr.RecapQuery(\n      userId = query.getRequiredUserId,\n      maxCount = Some(maxTweetsToFetch(query)),\n      range = Some(range),\n      options = options,\n      searchOperator = SearchOperator.Exclude,\n      earlybirdOptions = Some(earlyBirdOptions),\n      deviceContext = deviceContext,\n      authorIds = seedAuthorIds(query),\n      excludedTweetIds = excludedTweetIds,\n      utegLikedByTweetsOptions = utegLikedByTweetsOptions(query),\n      searchClientSubId = None,\n      candidateTweetSourceId = Some(candidateTweetSourceId),\n      hydratesContentFeatures = Some(false)\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/TimelineRankerUtegQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.query_transformer\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FetchParams\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.TimelineRankerUtegQueryTransformer._\nimport com.twitter.home_mixer.util.earlybird.EarlybirdRequestUtil\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.timelineranker.{model => tlr}\nimport com.twitter.timelineranker.{thriftscala => t}\nimport com.twitter.timelines.common.model.TweetKindOption\nimport com.twitter.timelines.earlybird.common.options.EarlybirdScoringModelConfig\nimport com.twitter.timelines.model.UserId\nimport com.twitter.timelines.model.candidate.CandidateTweetSourceId\n\nobject TimelineRankerUtegQueryTransformer {\n  private val SinceDuration = 24.hours\n  private val MaxUtegCandidates = 800\n\n  private val tweetKindOptions =\n    TweetKindOption(includeOriginalTweetsAndQuotes = true, includeReplies = true)\n\n  def utegEarlybirdModels: Seq[EarlybirdScoringModelConfig] =\n    EarlybirdRequestUtil.EarlybirdScoringModels.UnifiedEngagementRectweet\n}\n\ncase class TimelineRankerUtegQueryTransformer[\n  Query <: PipelineQuery with HasQualityFactorStatus with HasDeviceContext\n](\n  override val candidatePipelineIdentifier: CandidatePipelineIdentifier)\n    extends CandidatePipelineQueryTransformer[Query, t.UtegLikedByTweetsQuery]\n    with TimelineRankerQueryTransformer[Query] {\n\n  override val candidateTweetSourceId = CandidateTweetSourceId.RecommendedTweet\n  override val options = tweetKindOptions\n  override val earlybirdModels = utegEarlybirdModels\n\n  override def maxTweetsToFetch(query: Query): Int =\n    query.params(FetchParams.UTEGMaxTweetsToFetchParam)\n  override def getTensorflowModel(query: Query): Option[String] = {\n    Some(query.params(ScoredTweetsParam.EarlybirdTensorflowModel.UtegParam))\n  }\n\n  override def utegLikedByTweetsOptions(input: Query): Option[tlr.UtegLikedByTweetsOptions] = Some(\n    tlr.UtegLikedByTweetsOptions(\n      utegCount = MaxUtegCandidates,\n      isInNetwork = false,\n      weightedFollowings = input.features\n        .map(_.getOrElse(RealGraphInNetworkScoresFeature, Map.empty[UserId, Double]))\n        .getOrElse(Map.empty)\n    )\n  )\n\n  override def transform(input: Query): t.UtegLikedByTweetsQuery =\n    buildTimelineRankerQuery(input, SinceDuration).toThriftUtegLikedByTweetsQuery\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/UtegQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.query_transformer\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.UtegQueryTransformer._\nimport com.twitter.home_mixer.util.CachedScoredTweetsHelper\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.recos.recos_common.thriftscala.SocialProofType\nimport com.twitter.recos.user_tweet_entity_graph.thriftscala.RecommendationType\nimport com.twitter.recos.user_tweet_entity_graph.thriftscala.TweetEntityDisplayLocation\nimport com.twitter.recos.user_tweet_entity_graph.{thriftscala => uteg}\nimport com.twitter.util.Time\n\nobject UtegQueryTransformer {\n  private val MaxUserSocialProofSize = 10\n  private val MaxTweetSocialProofSize = 10\n  private val MinUserSocialProofSize = 1\n  private val DefaultSinceDuration = 24.hours\n  private val MaxTweetsToFetch = 800\n  private val MaxExcludedTweets = 1500\n}\n\ncase class UtegQueryTransformer[\n  Query <: PipelineQuery with HasQualityFactorStatus with HasDeviceContext\n](\n  candidatePipelineIdentifier: CandidatePipelineIdentifier)\n    extends CandidatePipelineQueryTransformer[Query, uteg.RecommendTweetEntityRequest] {\n\n  override def transform(query: Query): uteg.RecommendTweetEntityRequest = {\n\n    val weightedFollowings = query.features\n      .map(_.getOrElse(RealGraphInNetworkScoresFeature, Map.empty[Long, Double]))\n      .getOrElse(Map.empty)\n\n    val sinceTime: Time = DefaultSinceDuration.ago\n    val untilTime: Time = Time.now\n\n    val excludedTweetIds = query.features.map { featureMap =>\n      CachedScoredTweetsHelper.tweetImpressionsAndCachedScoredTweetsInRange(\n        featureMap,\n        candidatePipelineIdentifier,\n        MaxExcludedTweets,\n        sinceTime,\n        untilTime)\n    }\n\n    uteg.RecommendTweetEntityRequest(\n      requesterId = query.getRequiredUserId,\n      displayLocation = TweetEntityDisplayLocation.HomeTimeline,\n      recommendationTypes = Seq(RecommendationType.Tweet),\n      seedsWithWeights = weightedFollowings,\n      maxResultsByType = Some(Map(RecommendationType.Tweet -> MaxTweetsToFetch)),\n      maxTweetAgeInMillis = Some(sinceTime.untilNow.inMillis),\n      excludedTweetIds = excludedTweetIds,\n      maxUserSocialProofSize = Some(MaxUserSocialProofSize),\n      maxTweetSocialProofSize = Some(MaxTweetSocialProofSize),\n      minUserSocialProofSizes = Some(Map(RecommendationType.Tweet -> MinUserSocialProofSize)),\n      socialProofTypes = Some(Seq(SocialProofType.Favorite)),\n      tweetAuthors = None,\n      maxEngagementAgeInMillis = None,\n      tweetTypes = None\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/earlybird/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/earlybird\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/java/com/twitter/search/queryparser/query/search:search-query-nodes\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"timelines:util\",\n        \"timelines/src/main/scala/com/twitter/timelines/clients/relevance_search\",\n        \"timelines/src/main/scala/com/twitter/timelines/common/model\",\n        \"timelines/src/main/scala/com/twitter/timelines/earlybird/common/utils\",\n        \"timelines/src/main/scala/com/twitter/timelines/model/types\",\n        \"timelines/src/main/scala/com/twitter/timelines/util/stats\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/earlybird/CommunitiesEarlybirdQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.query_transformer.earlybird\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EarlyBirdCommunitiesMaxSearchResultsParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableEarlybirdCommunitiesQueryLinearRankingParam\nimport com.twitter.home_mixer.util.earlybird.RelevanceSearchUtil\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.communities.CommunityMembershipsFeature\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.search.common.query.thriftjava.thriftscala.CollectorParams\nimport com.twitter.search.common.query.thriftjava.thriftscala.CollectorTerminationParams\nimport com.twitter.search.common.ranking.{thriftscala => scr}\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport com.twitter.search.earlybird.{thriftscala => t}\nimport com.twitter.search.queryparser.query.Conjunction\nimport com.twitter.search.queryparser.query.Disjunction\nimport com.twitter.search.queryparser.query.search.SearchOperator\nimport com.twitter.search.queryparser.query.search.SearchOperatorConstants\nimport com.twitter.search.queryparser.query.{Query => SearchQuery}\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\n@Singleton\nclass CommunitiesEarlybirdQueryTransformer @Inject() (clientId: ClientId)\n    extends CandidatePipelineQueryTransformer[PipelineQuery, eb.EarlybirdRequest] {\n  private val SinceDuration = 48.hours\n  private val DefaultSearchProcessingTimeout = 200.milliseconds\n  private val TensorflowModel = \"timelines_unified_prod\"\n\n  override def transform(query: PipelineQuery): t.EarlybirdRequest = {\n\n    val maxSearchResults = query.params(EarlyBirdCommunitiesMaxSearchResultsParam)\n\n    val communityIds =\n      query.features.map(_.getOrElse(CommunityMembershipsFeature, Seq.empty)).toSeq.flatten.toSet\n    val entityIdsQuery = createEntityIdsQuery(communityIds)\n    val nullcastQuery =\n      new SearchOperator(SearchOperator.Type.INCLUDE, SearchOperatorConstants.NULLCAST)\n    val excludeRepliesQuery =\n      new SearchOperator(SearchOperator.Type.EXCLUDE, SearchOperatorConstants.REPLIES)\n    val sinceTimeQuery =\n      new SearchOperator(SearchOperator.Type.SINCE_TIME, SinceDuration.ago.inSeconds.toString)\n    val searchQuery =\n      new Conjunction(entityIdsQuery, excludeRepliesQuery, nullcastQuery, sinceTimeQuery)\n\n    val metadataOptions = t.ThriftSearchResultMetadataOptions(\n      getResultLocation = false,\n      getLuceneScore = false,\n      getInReplyToStatusId = true,\n      getReferencedTweetAuthorId = true,\n      getMediaBits = true,\n      getAllFeatures = true,\n      returnSearchResultFeatures = true,\n      getFromUserId = true\n    )\n\n    val ebRankingParams = Some(\n      scr.ThriftRankingParams(\n        `type` = Some(scr.ThriftScoringFunctionType.TensorflowBased),\n        selectedTensorflowModel = Some(TensorflowModel),\n        minScore = -1.0e100,\n        applyBoosts = false,\n      )\n    )\n\n    val linearRelevanceOptions = t.ThriftSearchRelevanceOptions(\n      rankingParams = Some(\n        scr.ThriftRankingParams(\n          `type` = Some(scr.ThriftScoringFunctionType.Linear),\n          applyBoosts = false,\n          favCountParams = Some(scr.ThriftLinearFeatureRankingParams(weight = 1000.0)),\n          replyCountParams = Some(scr.ThriftLinearFeatureRankingParams(weight = 10000.0)),\n          quotedCountParams = Some(scr.ThriftLinearFeatureRankingParams(weight = 1000.0))\n        )\n      ),\n      returnAllResults = Some(false)\n    )\n\n    val relOptions = RelevanceSearchUtil.RelevanceOptions.copy(\n      rankingParams = ebRankingParams,\n      returnAllResults = Some(false)\n    )\n\n    val relevanceOptions =\n      if (query.params(EnableEarlybirdCommunitiesQueryLinearRankingParam)) linearRelevanceOptions\n      else relOptions\n\n    val collectorParams = CollectorParams(\n      numResultsToReturn = maxSearchResults,\n      terminationParams = Some(\n        CollectorTerminationParams(\n          timeoutMs = DefaultSearchProcessingTimeout.inMilliseconds.toInt\n        )\n      )\n    )\n\n    t.EarlybirdRequest(\n      searchQuery = t.ThriftSearchQuery(\n        serializedQuery = Some(searchQuery.serialize),\n        rankingMode = t.ThriftSearchRankingMode.Relevance,\n        numResults = maxSearchResults,\n        resultMetadataOptions = Some(metadataOptions),\n        searcherId = query.getOptionalUserId,\n        relevanceOptions = Some(relevanceOptions),\n        maxHitsPerUser = -1,\n        collectorParams = Some(collectorParams)\n      ),\n      clientRequestID = Some(s\"${Trace.id.traceId}\"),\n      numResultsToReturnAtRoot = Some(maxSearchResults),\n      clientId = Some(clientId.name),\n    )\n  }\n\n  private def createEntityIdsQuery(entityIds: Set[Long]): Disjunction = {\n    val entityIdStrings = entityIds.map(_.toString)\n    val queryOps: Seq[SearchQuery] = entityIdStrings.map { entityId =>\n      new SearchOperator(SearchOperator.Type.ENTITY_ID, entityId)\n    }.toSeq\n    new Disjunction(queryOps.asJava)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/earlybird/EarlybirdFrsQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.query_transformer.earlybird\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.core_workflows.user_model.{thriftscala => um}\nimport com.twitter.home_mixer.functional_component.feature_hydrator.FrsSeedUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserStateFeature\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.earlybird.EarlybirdFrsQueryTransformer._\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport com.twitter.timelines.common.model.TweetKindOption\n\nobject EarlybirdFrsQueryTransformer {\n  private val DefaultSinceDuration = 24.hours\n  private val ExpandedSinceDuration = 48.hours\n  private val MaxTweetsToFetch = 500\n\n  private val TweetKindOptions: TweetKindOption.ValueSet =\n    TweetKindOption(includeOriginalTweetsAndQuotes = true)\n\n  private val UserStatesForExtendedSinceDuration: Set[um.UserState] = Set(\n    um.UserState.Light,\n    um.UserState.MediumNonTweeter,\n    um.UserState.MediumTweeter,\n    um.UserState.NearZero,\n    um.UserState.New,\n    um.UserState.VeryLight\n  )\n}\n\ncase class EarlybirdFrsQueryTransformer[\n  Query <: PipelineQuery with HasQualityFactorStatus with HasDeviceContext\n](\n  candidatePipelineIdentifier: CandidatePipelineIdentifier,\n  override val clientId: Option[String])\n    extends CandidatePipelineQueryTransformer[Query, eb.EarlybirdRequest]\n    with EarlybirdQueryTransformer[Query] {\n\n  override val tweetKindOptions: TweetKindOption.ValueSet = TweetKindOptions\n  override val maxTweetsToFetch: Int = MaxTweetsToFetch\n  override def getTensorflowModel(query: Query): Option[String] = {\n    Some(query.params(ScoredTweetsParam.EarlybirdTensorflowModel.FrsParam))\n  }\n\n  override def transform(query: Query): eb.EarlybirdRequest = {\n    val userState = query.features.flatMap(_.getOrElse(UserStateFeature, None))\n    val sinceDuration =\n      if (userState.exists(UserStatesForExtendedSinceDuration.contains)) ExpandedSinceDuration\n      else DefaultSinceDuration\n\n    val seedUserIds = query.features\n      .flatMap(_.getOrElse(FrsSeedUserIdsFeature, None))\n      .getOrElse(Seq.empty).toSet\n\n    buildEarlybirdQuery(query, sinceDuration, seedUserIds, None)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/earlybird/EarlybirdInNetworkQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.query_transformer.earlybird\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.core_workflows.user_model.{thriftscala => um}\nimport com.twitter.home_mixer.model.HomeFeatures.RealGraphInNetworkScoresFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserStateFeature\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.FollowedUserScoresFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam\nimport com.twitter.home_mixer.product.scored_tweets.query_transformer.earlybird.EarlybirdInNetworkQueryTransformer._\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport com.twitter.timelines.common.model.TweetKindOption\n\nobject EarlybirdInNetworkQueryTransformer {\n  private val DefaultSinceDuration = 24.hours\n  private val ExpandedSinceDuration = 48.hours\n  private val MaxTweetsToFetch = 660\n  private val MaxFollowUsers = 1500\n\n  private val DefaultTweetKindOptions: TweetKindOption.ValueSet = TweetKindOption(\n    includeReplies = true,\n    includeRetweets = true,\n    includeOriginalTweetsAndQuotes = true,\n    includeExtendedReplies = true\n  )\n\n  private val UserStatesForExtendedSinceDuration: Set[um.UserState] = Set(\n    um.UserState.Light,\n    um.UserState.MediumNonTweeter,\n    um.UserState.MediumTweeter,\n    um.UserState.NearZero,\n    um.UserState.New,\n    um.UserState.VeryLight\n  )\n}\n\ncase class EarlybirdInNetworkQueryTransformer[\n  Query <: PipelineQuery with HasQualityFactorStatus with HasDeviceContext\n](\n  candidatePipelineIdentifier: CandidatePipelineIdentifier,\n  override val clientId: Option[String])\n    extends CandidatePipelineQueryTransformer[Query, eb.EarlybirdRequest]\n    with EarlybirdQueryTransformer[Query] {\n\n  override def tweetKindOptions: TweetKindOption.ValueSet = DefaultTweetKindOptions\n  override val maxTweetsToFetch: Int = MaxTweetsToFetch\n  override val enableExcludeSourceTweetIdsQuery = true\n  override def getTensorflowModel(query: Query): Option[String] = {\n    Some(query.params(ScoredTweetsParam.EarlybirdTensorflowModel.InNetworkParam))\n  }\n\n  private def buildTweetKindOptions(query: Query): TweetKindOption.ValueSet = {\n    TweetKindOption(\n      includeReplies = query.params(ScoredTweetsParam.CandidateSourceParams.InNetworkIncludeRepliesParam),\n      includeRetweets = query.params(ScoredTweetsParam.CandidateSourceParams.InNetworkIncludeRetweetsParam),\n      includeOriginalTweetsAndQuotes = true, // Always include original tweets and quotes\n      includeExtendedReplies = query.params(ScoredTweetsParam.CandidateSourceParams.InNetworkIncludeExtendedRepliesParam)\n    )\n  }\n\n  override def transform(query: Query): eb.EarlybirdRequest = {\n\n    val userState = query.features.flatMap(_.getOrElse(UserStateFeature, None))\n\n    val sinceDuration =\n      if (userState.exists(UserStatesForExtendedSinceDuration.contains)) ExpandedSinceDuration\n      else DefaultSinceDuration\n\n    val updatedAuthorScoreMap =\n      query.features\n        .map(_.getOrElse(FollowedUserScoresFeature, Map.empty[Long, Double])).toSeq.flatten.toMap\n    val (authorScoreMap, followedUserIds) = if (updatedAuthorScoreMap.isEmpty) {\n      (\n        query.features.map(_.getOrElse(RealGraphInNetworkScoresFeature, Map.empty[Long, Double])),\n        query.features.map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty)).toSeq.flatten.toSet)\n    } else (Some(updatedAuthorScoreMap), updatedAuthorScoreMap.keySet)\n\n    buildEarlybirdQueryWithTweetKindOptions(\n      query,\n      sinceDuration,\n      followedUserIds.take(MaxFollowUsers) + query.getRequiredUserId,\n      authorScoreMap,\n      buildTweetKindOptions(query))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/query_transformer/earlybird/EarlybirdQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.query_transformer.earlybird\n\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.util.CachedScoredTweetsHelper\nimport com.twitter.home_mixer.util.earlybird.EarlybirdRequestUtil\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport com.twitter.timelines.clients.relevance_search.SearchClient.TweetTypes\nimport com.twitter.timelines.common.model.TweetKindOption\nimport com.twitter.timelines.util.SnowflakeSortIndexHelper\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\n\ntrait EarlybirdQueryTransformer[\n  Query <: PipelineQuery with HasQualityFactorStatus with HasDeviceContext] {\n\n  def candidatePipelineIdentifier: CandidatePipelineIdentifier\n  def clientId: Option[String] = None\n  def maxTweetsToFetch: Int = 100\n  def tweetKindOptions: TweetKindOption.ValueSet\n  def getTensorflowModel(query: Query): Option[String] = None\n  def enableExcludeSourceTweetIdsQuery: Boolean = false\n\n  private val EarlybirdMaxExcludedTweets = 1500\n\n  protected def getFollowedUsers(query: Query): Set[Long] = {\n    query.features\n      .map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty)).getOrElse(\n        Nil).toSet + query.getRequiredUserId\n  }\n\n  protected def buildEarlybirdQuery(\n    query: Query,\n    sinceDuration: Duration,\n    queryUserIds: Set[Long] = Set.empty,\n    authorScoreMap: Option[Map[Long, Double]] = None,\n    isVideoOnlyRequest: Boolean = false,\n    getOlderTweets: Boolean = false,\n    isRecency: Boolean = false,\n    until: Time = Time.now\n  ): eb.EarlybirdRequest = {\n    buildEarlybirdQueryWithTweetKindOptions(\n      query,\n      sinceDuration,\n      queryUserIds,\n      authorScoreMap,\n      tweetKindOptions,\n      isVideoOnlyRequest,\n      getOlderTweets,\n      isRecency,\n      until\n    )\n  }\n\n  protected def buildEarlybirdQueryWithTweetKindOptions(\n    query: Query,\n    sinceDuration: Duration,\n    queryUserIds: Set[Long] = Set.empty,\n    authorScoreMap: Option[Map[Long, Double]] = None,\n    tweetKindOptions: TweetKindOption.ValueSet,\n    isVideoOnlyRequest: Boolean = false,\n    getOlderTweets: Boolean = false,\n    isRecency: Boolean = false,\n    until: Time = Time.now\n  ): eb.EarlybirdRequest = {\n    val sinceTime: Time = sinceDuration.ago\n    val untilTime: Time = until\n\n    val fromTweetIdExclusive = SnowflakeSortIndexHelper.timestampToFakeId(sinceTime)\n    val toTweetIdExclusive = SnowflakeSortIndexHelper.timestampToFakeId(untilTime)\n\n    val excludedTweetIds = query.features.map { featureMap =>\n      CachedScoredTweetsHelper.tweetImpressionsAndCachedScoredTweetsInRange(\n        featureMap,\n        candidatePipelineIdentifier,\n        EarlybirdMaxExcludedTweets,\n        sinceTime,\n        untilTime)\n    }\n\n    EarlybirdRequestUtil.getTweetsRequest(\n      userId = Some(query.getRequiredUserId),\n      clientId = clientId,\n      skipVeryRecentTweets = true,\n      queryUserIds = queryUserIds,\n      retweetsMutedUserIds = Set.empty,\n      beforeTweetIdExclusive = Some(toTweetIdExclusive),\n      afterTweetIdExclusive = Some(fromTweetIdExclusive),\n      excludedTweetIds = excludedTweetIds.map(_.toSet),\n      maxCount = maxTweetsToFetch,\n      tweetTypes = TweetTypes.fromTweetKindOption(tweetKindOptions),\n      authorScoreMap = authorScoreMap,\n      tensorflowModel = getTensorflowModel(query),\n      enableExcludeSourceTweetIdsQuery = enableExcludeSourceTweetIdsQuery,\n      isVideoOnlyRequest = isVideoOnlyRequest,\n      getOlderTweets = getOlderTweets,\n      isRecency = isRecency,\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/marshaller/timelines\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie/content\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/location\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala\",\n        \"src/thrift/com/twitter/timelineranker:thrift-scala\",\n        \"topic-social-proof/server/src/main/thrift:thrift-scala\",\n        \"tweet-mixer/thrift/src/main/thrift:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/CachedScoredTweetsResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.marshaller.timelines.TopicContextFunctionalityTypeUnmarshaller\nimport com.twitter.home_mixer.model.candidate_source.SourceSignal\nimport com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorFollowersFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIsBlueVerifiedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIsCreatorFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIsGoldVerifiedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIsGrayVerifiedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIsLegacyVerifiedFeature\nimport com.twitter.home_mixer.model.HomeFeatures.CachedCandidatePipelineIdentifierFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ClipImageClusterIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DebugStringFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ExclusiveConversationAuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GorkContentCreatorFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokAnnotationsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokContentCreatorFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokSlopScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokTagsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokTopCategoryFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsArticleFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsReadFromCacheFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.LastScoredTimestampMsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ListIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ListNameFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MultiModalEmbeddingsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PredictionRequestIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidFollowedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceSignalFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TopicContextFunctionalityTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TopicIdSocialContextFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaClusterIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaCompletionRateFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMixerScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTextFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetTypeMetricsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetUrlsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoDurationMsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ViralContentCreatorFeature\nimport com.twitter.home_mixer.model.HomeFeatures.WeightedModelScoreFeature\nimport com.twitter.home_mixer.model.PredictedFavoriteScoreFeature\nimport com.twitter.home_mixer.model.PredictedReplyScoreFeature\nimport com.twitter.home_mixer.model.PredictedRetweetScoreFeature\nimport com.twitter.home_mixer.model.PredictedShareScoreFeature\nimport com.twitter.home_mixer.model.PredictedVideoQualityViewScoreFeature\nimport com.twitter.home_mixer.model.PredictedDwellScoreFeature\nimport com.twitter.home_mixer.model.PredictedNegativeFeedbackV2ScoreFeature\nimport com.twitter.home_mixer.model.PredictedGoodClickConvoDescUamGt2ScoreFeature\nimport com.twitter.home_mixer.model.PredictedGoodProfileClickScoreFeature\nimport com.twitter.home_mixer.model.PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature\nimport com.twitter.home_mixer.model.PredictedReplyEngagedByAuthorScoreFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityIdFeature\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityNameFeature\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationIdFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject CachedScoredTweetsResponseFeatureTransformer\n    extends CandidateFeatureTransformer[hmt.ScoredTweet] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"CachedScoredTweetsResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    AncestorsFeature,\n    AuthorIdFeature,\n    AuthorIsBlueVerifiedFeature,\n    AuthorIsCreatorFeature,\n    AuthorIsGoldVerifiedFeature,\n    AuthorIsGrayVerifiedFeature,\n    AuthorIsLegacyVerifiedFeature,\n    AuthorFollowersFeature,\n    CachedCandidatePipelineIdentifierFeature,\n    DirectedAtUserIdFeature,\n    DebugStringFeature,\n    ExclusiveConversationAuthorIdFeature,\n    InNetworkFeature,\n    InReplyToTweetIdFeature,\n    InReplyToUserIdFeature,\n    IsReadFromCacheFeature,\n    IsRetweetFeature,\n    LastScoredTimestampMsFeature,\n    PredictionRequestIdFeature,\n    QuotedTweetIdFeature,\n    QuotedUserIdFeature,\n    SGSValidFollowedByUserIdsFeature,\n    SGSValidLikedByUserIdsFeature,\n    ScoreFeature,\n    SourceTweetIdFeature,\n    SourceUserIdFeature,\n    ServedTypeFeature,\n    TopicContextFunctionalityTypeFeature,\n    TopicIdSocialContextFeature,\n    TweetTypeMetricsFeature,\n    TweetUrlsFeature,\n    ViralContentCreatorFeature,\n    GrokContentCreatorFeature,\n    GorkContentCreatorFeature,\n    WeightedModelScoreFeature,\n    CommunityIdFeature,\n    CommunityNameFeature,\n    ListIdFeature,\n    ListNameFeature,\n    LocationIdFeature,\n    IsArticleFeature,\n    HasVideoFeature,\n    VideoDurationMsFeature,\n    TweetMediaIdsFeature,\n    GrokAnnotationsFeature,\n    GrokTopCategoryFeature,\n    GrokTagsFeature,\n    PredictedFavoriteScoreFeature,\n    PredictedReplyScoreFeature,\n    PredictedRetweetScoreFeature,\n    PredictedReplyEngagedByAuthorScoreFeature,\n    PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature,\n    PredictedGoodClickConvoDescUamGt2ScoreFeature,\n    PredictedGoodProfileClickScoreFeature,\n    PredictedVideoQualityViewScoreFeature,\n    PredictedShareScoreFeature,\n    PredictedDwellScoreFeature,\n    PredictedNegativeFeedbackV2ScoreFeature,\n    TweetMixerScoreFeature,\n    SourceSignalFeature,\n    TweetMediaClusterIdsFeature,\n    ClipImageClusterIdsFeature,\n    GrokSlopScoreFeature,\n    TweetMediaCompletionRateFeature,\n    TweetTextFeature,\n    MultiModalEmbeddingsFeature\n  )\n\n  override def transform(candidate: hmt.ScoredTweet): FeatureMap = {\n    val grokTopCategory = candidate.grokAnnotations\n      .flatMap(_.categoryScores)\n      .flatMap { scores =>\n        val validCategories = scores.collect {\n          case (category, score) if category.forall(_.isDigit) && category.toLong % 10000 == 0 =>\n            (category.toLong, score)\n        }\n        if (validCategories.nonEmpty) Some(validCategories.maxBy(_._2)._1) else None\n      }\n\n    FeatureMapBuilder()\n      .add(AncestorsFeature, candidate.ancestors.getOrElse(Seq.empty))\n      .add(AuthorIdFeature, Some(candidate.authorId))\n      .add(AuthorIsBlueVerifiedFeature, candidate.authorMetadata.exists(_.blueVerified))\n      .add(AuthorIsGoldVerifiedFeature, candidate.authorMetadata.exists(_.goldVerified))\n      .add(AuthorIsGrayVerifiedFeature, candidate.authorMetadata.exists(_.grayVerified))\n      .add(AuthorIsLegacyVerifiedFeature, candidate.authorMetadata.exists(_.legacyVerified))\n      .add(AuthorIsCreatorFeature, candidate.authorMetadata.exists(_.creator))\n      .add(AuthorFollowersFeature, candidate.authorMetadata.flatMap(_.followers))\n      .add(CachedCandidatePipelineIdentifierFeature, candidate.candidatePipelineIdentifier)\n      .add(DirectedAtUserIdFeature, candidate.directedAtUserId)\n      .add(DebugStringFeature, candidate.debugString)\n      .add(ExclusiveConversationAuthorIdFeature, candidate.exclusiveConversationAuthorId)\n      .add(InNetworkFeature, candidate.inNetwork.getOrElse(true))\n      .add(InReplyToTweetIdFeature, candidate.inReplyToTweetId)\n      .add(InReplyToUserIdFeature, candidate.inReplyToUserId)\n      .add(IsReadFromCacheFeature, true)\n      .add(IsRetweetFeature, candidate.sourceTweetId.isDefined)\n      .add(LastScoredTimestampMsFeature, candidate.lastScoredTimestampMs)\n      .add(PredictionRequestIdFeature, candidate.predictionRequestId)\n      .add(QuotedTweetIdFeature, candidate.quotedTweetId)\n      .add(QuotedUserIdFeature, candidate.quotedUserId)\n      .add(ScoreFeature, candidate.score)\n      .add(SGSValidLikedByUserIdsFeature, candidate.sgsValidLikedByUserIds.getOrElse(Seq.empty))\n      .add(\n        SGSValidFollowedByUserIdsFeature,\n        candidate.sgsValidFollowedByUserIds.getOrElse(Seq.empty))\n      .add(SourceTweetIdFeature, candidate.sourceTweetId)\n      .add(SourceUserIdFeature, candidate.sourceUserId)\n      .add(ServedTypeFeature, candidate.servedType)\n      .add(\n        TopicContextFunctionalityTypeFeature,\n        candidate.topicFunctionalityType.map(TopicContextFunctionalityTypeUnmarshaller(_)))\n      .add(TopicIdSocialContextFeature, candidate.topicId)\n      .add(TweetTypeMetricsFeature, candidate.tweetTypeMetrics)\n      .add(TweetUrlsFeature, candidate.tweetUrls.getOrElse(Seq.empty))\n      .add(ViralContentCreatorFeature, candidate.viralContentCreator.contains(true))\n      .add(WeightedModelScoreFeature, candidate.score)\n      .add(CommunityIdFeature, candidate.communityId)\n      .add(CommunityNameFeature, candidate.communityName)\n      .add(ListIdFeature, candidate.listId)\n      .add(ListNameFeature, candidate.listName)\n      .add(LocationIdFeature, candidate.locationId)\n      .add(IsArticleFeature, candidate.isArticle.contains(true))\n      .add(HasVideoFeature, candidate.hasVideo.contains(true))\n      .add(VideoDurationMsFeature, candidate.videoDurationMs)\n      .add(TweetMediaIdsFeature, candidate.mediaIds.getOrElse(Seq.empty))\n      .add(GrokAnnotationsFeature, candidate.grokAnnotations)\n      .add(GrokTopCategoryFeature, grokTopCategory)\n      .add(\n        GrokTagsFeature,\n        candidate.grokAnnotations.map(_.tags.map(_.toLowerCase)).getOrElse(Seq.empty).toSet\n      )\n      .add(PredictedFavoriteScoreFeature, candidate.predictedScores.flatMap(_.favoriteScore))\n      .add(PredictedReplyScoreFeature, candidate.predictedScores.flatMap(_.replyScore))\n      .add(PredictedRetweetScoreFeature, candidate.predictedScores.flatMap(_.retweetScore))\n      .add(\n        PredictedReplyEngagedByAuthorScoreFeature,\n        candidate.predictedScores.flatMap(_.replyEngagedByAuthorScore))\n      .add(\n        PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature,\n        candidate.predictedScores.flatMap(_.goodClickConvoDescFavoritedOrRepliedScore))\n      .add(\n        PredictedGoodClickConvoDescUamGt2ScoreFeature,\n        candidate.predictedScores.flatMap(_.goodClickConvoDescUamGt2Score))\n      .add(\n        PredictedGoodProfileClickScoreFeature,\n        candidate.predictedScores.flatMap(_.goodProfileClickScore))\n      .add(\n        PredictedVideoQualityViewScoreFeature,\n        candidate.predictedScores.flatMap(_.videoQualityViewScore))\n      .add(PredictedShareScoreFeature, candidate.predictedScores.flatMap(_.shareScore))\n      .add(PredictedDwellScoreFeature, candidate.predictedScores.flatMap(_.dwellScore))\n      .add(\n        PredictedNegativeFeedbackV2ScoreFeature,\n        candidate.predictedScores.flatMap(_.negativeFeedbackV2Score))\n      .add(TweetMixerScoreFeature, candidate.tweetMixerScore)\n      .add(\n        SourceSignalFeature,\n        candidate.sourceSignal.map { signal =>\n          SourceSignal(\n            id = signal.id,\n            signalType = signal.signalType,\n            signalEntity = signal.signalEntity,\n            authorId = signal.authorId,\n          )\n        }\n      )\n      .add(\n        ClipImageClusterIdsFeature,\n        candidate.clipClusterIdsFeature\n          .flatMap(_.clipImageClusterIdsFeature).getOrElse(Map.empty[Long, Long]).toMap)\n      .add(\n        TweetMediaClusterIdsFeature,\n        candidate.clipClusterIdsFeature\n          .flatMap(_.tweetMediaClusterIdsFeature).getOrElse(Map.empty[Long, Long]).toMap)\n      .add(GrokSlopScoreFeature, candidate.grokSlopScoreFeature)\n      .add(TweetMediaCompletionRateFeature, candidate.mediaCompletionRate)\n      .add(TweetTextFeature, candidate.tweetText)\n      .add(GrokContentCreatorFeature, candidate.grokContentCreator.contains(true))\n      .add(GorkContentCreatorFeature, candidate.gorkContentCreator.contains(true))\n      .add(MultiModalEmbeddingsFeature, candidate.multiModalEmbedding)\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/ScoredTweetsBackfillResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.DebugStringFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject ScoredTweetsBackfillResponseFeatureTransformer extends CandidateFeatureTransformer[Long] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsBackfillResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    FromInNetworkSourceFeature,\n    ServedTypeFeature,\n    DebugStringFeature\n  )\n\n  override def transform(candidate: Long): FeatureMap = FeatureMapBuilder()\n    .add(FromInNetworkSourceFeature, true)\n    .add(ServedTypeFeature, hmt.ServedType.ForYouInNetwork)\n    .add(DebugStringFeature, Some(\"Backfill\"))\n    .build()\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/ScoredTweetsContentExplorationResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.DebugStringFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.product.scored_tweets.candidate_source.ContentExplorationCandidateResponse\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject ScoredTweetsContentExplorationResponseFeatureTransformer\n    extends CandidateFeatureTransformer[ContentExplorationCandidateResponse] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsContentExploration\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    ServedTypeFeature,\n    DebugStringFeature\n  )\n\n  override def transform(candidate: ContentExplorationCandidateResponse): FeatureMap = {\n    val servedType = candidate.tier match {\n      case \"tier1\" => hmt.ServedType.ForYouContentExploration\n      case \"tier2\" => hmt.ServedType.ForYouContentExplorationTier2\n      case _ => hmt.ServedType.ForYouContentExploration\n    }\n    FeatureMapBuilder()\n      .add(ServedTypeFeature, servedType)\n      .add(DebugStringFeature, Some(s\"Content Exploration: ${candidate.tier}\"))\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/ScoredTweetsDirectUtegResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.DebugStringFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.recos.recos_common.thriftscala.SocialProofType\nimport com.twitter.recos.user_tweet_entity_graph.{thriftscala => uteg}\n\nobject UtegFavListFeature extends Feature[TweetCandidate, Seq[Long]]\nobject UtegScoreFeature extends Feature[TweetCandidate, Double]\n\nobject ScoredTweetsDirectUtegResponseFeatureTransformer\n    extends CandidateFeatureTransformer[uteg.TweetRecommendation] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsDirectUtegResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    FromInNetworkSourceFeature,\n    ServedTypeFeature,\n    DebugStringFeature,\n    UtegFavListFeature,\n    UtegScoreFeature\n  )\n\n  override def transform(input: uteg.TweetRecommendation): FeatureMap =\n    FeatureMapBuilder()\n      .add(FromInNetworkSourceFeature, false)\n      .add(ServedTypeFeature, hmt.ServedType.ForYouUteg)\n      .add(DebugStringFeature, Some(\"Uteg\"))\n      .add(\n        UtegFavListFeature,\n        input.socialProofByType.getOrElse(SocialProofType.Favorite, Seq.empty)\n      )\n      .add(UtegScoreFeature, input.score)\n      .build()\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/ScoredTweetsFrsResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.timelineranker.{thriftscala => tlr}\n\nobject ScoredTweetsFrsResponseFeatureTransformer\n    extends CandidateFeatureTransformer[tlr.CandidateTweet] {\n\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"ScoredTweetsFrsResponse\")\n\n  override val features: Set[Feature[_, _]] = TimelineRankerResponseTransformer.features\n\n  override def transform(candidate: tlr.CandidateTweet): FeatureMap = {\n    val baseFeatures = TimelineRankerResponseTransformer.transform(candidate)\n\n    val features = FeatureMapBuilder()\n      .add(ServedTypeFeature, hmt.ServedType.ForYouFrs)\n      .build()\n\n    baseFeatures ++ features\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/ScoredTweetsInNetworkResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceSignalFeature\nimport com.twitter.home_mixer.model.candidate_source.SourceSignal\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.timelineranker.{thriftscala => tlr}\n\nobject ScoredTweetsInNetworkResponseFeatureTransformer\n    extends CandidateFeatureTransformer[tlr.CandidateTweet] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsInNetworkResponse\")\n\n  override val features: Set[Feature[_, _]] =\n    TimelineRankerResponseTransformer.features ++ Set(SourceSignalFeature)\n\n  override def transform(candidate: tlr.CandidateTweet): FeatureMap = {\n    val baseFeatures = TimelineRankerResponseTransformer.transform(candidate)\n\n    val features = FeatureMapBuilder()\n      .add(FromInNetworkSourceFeature, true)\n      .add(ServedTypeFeature, hmt.ServedType.ForYouInNetwork)\n      .add(\n        SourceSignalFeature,\n        Some(\n          SourceSignal(\n            id = candidate.tweet.flatMap(_.coreData.map(_.userId)).getOrElse(0L),\n            signalType = None,\n            signalEntity = None,\n            authorId = None,\n          ))\n      )\n      .build()\n\n    baseFeatures ++ features\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/ScoredTweetsListsResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ListIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.product.scored_tweets.candidate_source.ListTweet\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject ScoredTweetsListsResponseFeatureTransformer extends CandidateFeatureTransformer[ListTweet] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsListsResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    AuthorIdFeature,\n    FromInNetworkSourceFeature,\n    IsRetweetFeature,\n    ServedTypeFeature,\n    SourceTweetIdFeature,\n    SourceUserIdFeature,\n    ListIdFeature\n  )\n\n  override def transform(candidate: ListTweet): FeatureMap = FeatureMapBuilder()\n    .add(AuthorIdFeature, candidate.tweet.userId)\n    .add(FromInNetworkSourceFeature, false)\n    .add(IsRetweetFeature, candidate.tweet.sourceStatusId.isDefined)\n    .add(ServedTypeFeature, hmt.ServedType.ForYouList)\n    .add(SourceTweetIdFeature, candidate.tweet.sourceStatusId)\n    .add(SourceUserIdFeature, candidate.tweet.sourceUserId)\n    .add(ListIdFeature, Some(candidate.listId))\n    .build()\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/ScoredTweetsOfflineVideoRecoResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject ScoredTweetsOfflineVideoRecoResponseFeatureTransformer\n    extends CandidateFeatureTransformer[Long] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsOfflineVideoRecoResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    FromInNetworkSourceFeature,\n    ServedTypeFeature\n  )\n\n  override def transform(candidate: Long): FeatureMap = FeatureMapBuilder()\n    .add(FromInNetworkSourceFeature, false)\n    .add(ServedTypeFeature, hmt.ServedType.OfflineVideoReco)\n    .build()\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/ScoredTweetsPopularVideosResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.explore_ranker.{thriftscala => ert}\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRandomTweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.StreamToKafkaFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.timelineservice.suggests.logging.candidate_tweet_source_id.{thriftscala => cts}\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\n\nobject ScoredTweetsPopularVideosResponseFeatureTransformer\n    extends CandidateFeatureTransformer[ert.ExploreTweetRecommendation] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsPopularVideosResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    AuthorIdFeature,\n    CandidateSourceIdFeature,\n    FromInNetworkSourceFeature,\n    HasVideoFeature,\n    IsRandomTweetFeature,\n    StreamToKafkaFeature,\n    SuggestTypeFeature\n  )\n\n  override def transform(candidate: ert.ExploreTweetRecommendation): FeatureMap = {\n    FeatureMapBuilder()\n      .add(AuthorIdFeature, candidate.authorId)\n      .add(CandidateSourceIdFeature, Some(cts.CandidateTweetSourceId.MediaTweet))\n      .add(FromInNetworkSourceFeature, false)\n      .add(HasVideoFeature, candidate.mediaType.contains(ert.MediaType.Video))\n      .add(IsRandomTweetFeature, false)\n      .add(StreamToKafkaFeature, true)\n      .add(SuggestTypeFeature, Some(st.SuggestType.MediaTweet))\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/ScoredTweetsStaticResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject ScoredTweetsStaticResponseFeatureTransformer extends CandidateFeatureTransformer[Long] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsStatic\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    FromInNetworkSourceFeature,\n    ServedTypeFeature,\n    ScoreFeature,\n  )\n\n  private val StaticScore = 100000.0\n\n  override def transform(candidate: Long): FeatureMap = {\n    FeatureMapBuilder()\n      .add(FromInNetworkSourceFeature, false)\n      .add(ServedTypeFeature, hmt.ServedType.ForYouTweetMixer)\n      .add(ScoreFeature, Some(StaticScore))\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/ScoredTweetsTweetMixerResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.model.candidate_source._\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.tsp.{thriftscala => tsp}\nimport com.twitter.tweet_mixer.{thriftscala => tmt}\nimport com.twitter.usersignalservice.{thriftscala => uss}\n\ncase class ScoredTweetsTweetMixerResponseFeatureTransformer(debugPrefix: String = \"\")\n    extends CandidateFeatureTransformer[tmt.TweetResult] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsTweetMixerResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    FromInNetworkSourceFeature,\n    ServedTypeFeature,\n    TSPMetricTagFeature,\n    TweetMixerScoreFeature,\n    InReplyToTweetIdFeature,\n    DebugStringFeature,\n    SourceSignalFeature,\n  )\n\n  val FavoriteSignal = \"Fav\"\n  val RetweetSignal = \"Retweet\"\n  val ReplySignal = \"Reply\"\n  val BookmarkSignal = \"Bookmark\"\n  val ShareSignal = \"Share\"\n  val TweetSignal = \"Tweet\"\n  val VideoViewSignal = \"VideoView\"\n  val ImmersiveVideoViewSignal = \"ImmersiveVideoView\"\n  val SearcherRealtimeHistorySignal = \"SearcherRealtimeHistory\"\n  val FollowSignal = \"Follow\"\n  val ProfileVisitSignal = \"ProfileVisit\"\n  val NotificationsSignal = \"Notif\"\n\n  val UTEG = \"UTEG\"\n  val PopGeo = \"PopGeo\"\n  val PopTopic = \"PopTopic\"\n  val Simclusters = \"Simclusters\"\n  val Twhin = \"Twhin\"\n  val UTG = \"UTG\"\n  val UVG = \"UVG\"\n  val InNetwork = \"InNetwork\"\n  val DeepRetrieval = \"DeepRetrieval\"\n  val DeepRetrievalI2i = \"DeepRetrievalI2i\"\n  val ContentExploration = \"Tier1ContentExploration\"\n  val ContentExplorationTier2 = \"Tier2ContentExploration\"\n  val ContentExplorationDeepRetrievalI2i = \"Tier1DrContentExploration\"\n  val ContentExplorationTier2DeepRetrievalI2i = \"Tier2DrContentExploration\"\n  val ContentExplorationSimclusterColdPosts = \"ContentExplorationSimclusterColdPosts\"\n  val EvergreenDeepRetrievalHome = \"EvergreenDeepRetrievalHome\"\n  val EvergreenDeepRetrievalCrossBorderHome = \"EvergreenDeepRetrievalCrossBorderHome\"\n  val UserInterestSummary = \"UserInterestSummary\"\n  val ContentExplorationEvergreenDRI2i = \"ContentExplorationEvergreenDRI2i\"\n  val Local = \"Local\"\n  val Trends = \"Trends\"\n  val TwitterClipV0Short = \"TwitterClipV0Short\"\n  val TwitterClipV0Long = \"TwitterClipV0Long\"\n  val SemanticVideo = \"SemanticVideo\"\n  val RelatedCreator = \"RelatedCreator\"\n\n  override def transform(candidate: tmt.TweetResult): FeatureMap = {\n    val tweetMixerMetricTags: Seq[tmt.MetricTag] = candidate.metricTags.getOrElse(Seq.empty)\n    val tspMetricTag = tweetMixerMetricTags\n      .map(tweetMixerMetricTagToTspMetricTag)\n      .filter(_.nonEmpty).map(_.get).toSet\n\n    val servedType: hmt.ServedType = getServedType(candidate.tweetMetadata)\n    val fromInNetwork = servedType match {\n      case hmt.ServedType.ForYouInNetwork => true\n      case _ => false\n    }\n\n    FeatureMapBuilder()\n      .add(FromInNetworkSourceFeature, fromInNetwork)\n      .add(ServedTypeFeature, servedType)\n      .add(TSPMetricTagFeature, tspMetricTag)\n      .add(TweetMixerScoreFeature, candidate.score)\n      .add(DebugStringFeature, candidate.tweetMetadata.map(buildDebugString))\n      .add(SourceSignalFeature, candidate.tweetMetadata.map(buildSourceSignal))\n      .add(InReplyToTweetIdFeature, candidate.inReplyToTweetId)\n      .build()\n  }\n\n  private def tweetMixerMetricTagToTspMetricTag(\n    tweetMixerMetricTag: tmt.MetricTag\n  ): Option[tsp.MetricTag] = tweetMixerMetricTag match {\n    case tmt.MetricTag.TweetFavorite => Some(tsp.MetricTag.TweetFavorite)\n    case tmt.MetricTag.Retweet => Some(tsp.MetricTag.Retweet)\n    case tmt.MetricTag.PopGeo => None\n    case _ => None\n  }\n\n  private def buildSourceSignal(metadata: tmt.TweetMetadata): SourceSignal = {\n    SourceSignal(\n      id = metadata.sourceSignalId.getOrElse(0L),\n      signalType = metadata.signalType.flatMap(_.headOption.map(_.name)),\n      signalEntity = metadata.signalEntity,\n      authorId = metadata.authorId,\n    )\n  }\n\n  private def buildDebugString(metadata: tmt.TweetMetadata): String = {\n    val signalTypeStr = metadata.signalType\n      .map { signalTypes =>\n        signalTypes.map {\n          case uss.SignalType.TweetFavorite => FavoriteSignal\n          case uss.SignalType.Retweet => RetweetSignal\n          case uss.SignalType.Reply => ReplySignal\n          case uss.SignalType.TweetBookmarkV1 => BookmarkSignal\n          case uss.SignalType.TweetShareV1 => ShareSignal\n          case uss.SignalType.OriginalTweet => TweetSignal\n          case uss.SignalType.VideoView90dQualityV1 | uss.SignalType.VideoView90dPlayback50V1 |\n              uss.SignalType.VideoView90dQualityV1AllSurfaces =>\n            VideoViewSignal\n          case uss.SignalType.ImmersiveVideoQualityView => ImmersiveVideoViewSignal\n          case uss.SignalType.SearcherRealtimeHistory => SearcherRealtimeHistorySignal\n          case uss.SignalType.AccountFollow => FollowSignal\n          case uss.SignalType.RepeatedProfileVisit180dMinVisit6V1 |\n              uss.SignalType.RepeatedProfileVisit90dMinVisit6V1 |\n              uss.SignalType.RepeatedProfileVisit14dMinVisit2V1 |\n              uss.SignalType.RepeatedProfileVisit180dMinVisit6V1NoNegative |\n              uss.SignalType.RepeatedProfileVisit90dMinVisit6V1NoNegative |\n              uss.SignalType.RepeatedProfileVisit14dMinVisit2V1NoNegative =>\n            ProfileVisitSignal\n          case uss.SignalType.NotificationOpenAndClickV1 => NotificationsSignal\n          case other => other.name\n        }\n      }.getOrElse(Seq.empty).mkString(\",\")\n\n    val candidateTypeStr = metadata.servedType\n      .map {\n        case tmt.ServedType.Simclusters => Simclusters\n        case tmt.ServedType.Twhin => Twhin\n        case tmt.ServedType.Utg => UTG\n        case tmt.ServedType.Uvg => UVG\n        case tmt.ServedType.Uteg => UTEG\n        case tmt.ServedType.InNetwork => InNetwork\n        case tmt.ServedType.PopGeo => PopGeo\n        case tmt.ServedType.PopTopic => PopTopic\n        case tmt.ServedType.DeepRetrieval => DeepRetrieval\n        case tmt.ServedType.DeepRetrievalI2iEmb => DeepRetrievalI2i\n        case tmt.ServedType.ContentExploration => ContentExploration\n        case tmt.ServedType.ContentExplorationTier2 => ContentExplorationTier2\n        case tmt.ServedType.ContentExplorationDRI2i => ContentExplorationDeepRetrievalI2i\n        case tmt.ServedType.ContentExplorationDRI2iTier2 => ContentExplorationTier2DeepRetrievalI2i\n        case tmt.ServedType.ContentExplorationSimclusterColdPosts => ContentExplorationSimclusterColdPosts\n        case tmt.ServedType.EvergreenDRU2iHome => EvergreenDeepRetrievalHome\n        case tmt.ServedType.EvergreenDRCrossBorderU2iHome => EvergreenDeepRetrievalCrossBorderHome\n        case tmt.ServedType.UserInterestSummaryI2i => UserInterestSummary\n        case tmt.ServedType.ContentExplorationEvergreenDRI2i => ContentExplorationEvergreenDRI2i\n        case tmt.ServedType.Local => Local\n        case tmt.ServedType.Trends => Trends\n        case tmt.ServedType.TwitterClipV0Short => TwitterClipV0Short\n        case tmt.ServedType.TwitterClipV0Long => TwitterClipV0Long\n        case tmt.ServedType.SemanticVideo => SemanticVideo\n        case tmt.ServedType.RelatedCreator => RelatedCreator\n        case _ => \"\"\n      }.getOrElse(\"\")\n\n    s\"${metadata.sourceSignalId.getOrElse(0L)} $signalTypeStr $candidateTypeStr $debugPrefix\"\n  }\n\n  private def getServedType(metadata: Option[tmt.TweetMetadata]): hmt.ServedType = {\n    metadata\n      .flatMap {\n        _.servedType\n          .map {\n            case tmt.ServedType.Simclusters => hmt.ServedType.ForYouSimclusters\n            case tmt.ServedType.Twhin => hmt.ServedType.ForYouTwhin\n            case tmt.ServedType.Utg => hmt.ServedType.ForYouUtg\n            case tmt.ServedType.Uvg => hmt.ServedType.ForYouUvg\n            case tmt.ServedType.Uteg => hmt.ServedType.ForYouUteg\n            case tmt.ServedType.InNetwork => hmt.ServedType.ForYouInNetwork\n            case tmt.ServedType.PopGeo => hmt.ServedType.ForYouPopularGeo\n            case tmt.ServedType.PopTopic => hmt.ServedType.ForYouPopularTopic\n            case tmt.ServedType.DeepRetrieval => hmt.ServedType.ForYouDeepRetrieval\n            case tmt.ServedType.DeepRetrievalI2iEmb => hmt.ServedType.ForYouDeepRetrievalI2i\n            case tmt.ServedType.ContentExploration => hmt.ServedType.ForYouContentExploration\n            case tmt.ServedType.ContentExplorationTier2 =>\n              hmt.ServedType.ForYouContentExplorationTier2\n            case tmt.ServedType.ContentExplorationDRI2i =>\n              hmt.ServedType.ForYouContentExplorationDeepRetrievalI2i\n            case tmt.ServedType.ContentExplorationDRI2iTier2 =>\n              hmt.ServedType.ForYouContentExplorationTier2DeepRetrievalI2i\n            case tmt.ServedType.EvergreenDeepRetrieval =>\n              hmt.ServedType.ForYouEvergreenDeepRetrieval\n            case tmt.ServedType.EvergreenDRU2iHome =>\n              hmt.ServedType.ForYouEvergreenDeepRetrievalHome\n            case tmt.ServedType.EvergreenDRCrossBorderU2iHome =>\n              hmt.ServedType.ForYouEvergreenDeepRetrievalCrossBorderHome\n            case tmt.ServedType.UserInterestSummaryI2i =>\n              hmt.ServedType.ForYouUserInterestSummary\n            case tmt.ServedType.ContentExplorationEvergreenDRI2i =>\n              hmt.ServedType.ForYouContentExplorationEvergreenDeepRetrievalI2i\n            case tmt.ServedType.ContentExplorationSimclusterColdPosts =>\n              hmt.ServedType.ForYouContentExplorationSimclusterColdPosts\n            case tmt.ServedType.Local => hmt.ServedType.ForYouLocal\n            case tmt.ServedType.Trends => hmt.ServedType.ForYouTrends\n            case tmt.ServedType.TwitterClipV0Short => hmt.ServedType.ForYouTwitterClipV0Short\n            case tmt.ServedType.TwitterClipV0Long => hmt.ServedType.ForYouTwitterClipV0Long\n            case tmt.ServedType.SemanticVideo => hmt.ServedType.ForYouSemanticVideo\n            case tmt.ServedType.RelatedCreator => hmt.ServedType.ForYouRelatedCreator\n            case tmt.ServedType.PromotedCreator => hmt.ServedType.ForYouPromotedCreator\n            case tmt.ServedType.NsfwVideoContent => hmt.ServedType.ForYouNsfwVideoContent\n            case _ => hmt.ServedType.ForYouTweetMixer\n          }\n      }.getOrElse(hmt.ServedType.ForYouTweetMixer)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/ScoredTweetsUtegResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.CandidateSourceIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.timelineranker.{thriftscala => tlr}\nimport com.twitter.timelineservice.suggests.logging.candidate_tweet_source_id.{thriftscala => cts}\nimport com.twitter.timelineservice.suggests.{thriftscala => st}\n\nobject ScoredTweetsUtegResponseFeatureTransformer\n    extends CandidateFeatureTransformer[tlr.CandidateTweet] {\n\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"ScoredTweetsUtegResponse\")\n\n  override val features: Set[Feature[_, _]] = TimelineRankerResponseTransformer.features\n\n  override def transform(candidate: tlr.CandidateTweet): FeatureMap = {\n    val baseFeatures = TimelineRankerResponseTransformer.transform(candidate)\n\n    val features = FeatureMapBuilder()\n      .add(CandidateSourceIdFeature, Some(cts.CandidateTweetSourceId.RecommendedTweet))\n      .add(SuggestTypeFeature, Some(st.SuggestType.ActivityTweet))\n      .build()\n\n    baseFeatures ++ features\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/ScoredVideoTweetsPinnedTweetResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject ScoredVideoTweetsPinnedTweetResponseFeatureTransformer\n    extends CandidateFeatureTransformer[TweetCandidate] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredVideoTweetsPinnedTweetResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    ServedTypeFeature\n  )\n\n  override def transform(candidate: TweetCandidate): FeatureMap = FeatureMapBuilder()\n    .add(ServedTypeFeature, hmt.ServedType.ForYouPinned)\n    .build()\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/TimelineRankerResponseTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ExclusiveConversationAuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasImageFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MentionScreenNameFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MentionUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetUrlsFeature\nimport com.twitter.home_mixer.util.tweetypie.content.TweetMediaFeaturesExtractor\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityIdFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.timelineranker.{thriftscala => tlr}\n\nobject TimelineRankerResponseTransformer {\n\n  val features: Set[Feature[_, _]] = Set(\n    AuthorIdFeature,\n    CommunityIdFeature,\n    DirectedAtUserIdFeature,\n    EarlybirdFeature,\n    EarlybirdScoreFeature,\n    ExclusiveConversationAuthorIdFeature,\n    FromInNetworkSourceFeature,\n    HasImageFeature,\n    HasVideoFeature,\n    InReplyToTweetIdFeature,\n    InReplyToUserIdFeature,\n    IsRetweetFeature,\n    MentionScreenNameFeature,\n    MentionUserIdFeature,\n    QuotedTweetIdFeature,\n    QuotedUserIdFeature,\n    SourceTweetIdFeature,\n    SourceUserIdFeature,\n    ServedTypeFeature,\n    TweetUrlsFeature\n  )\n\n  def transform(candidate: tlr.CandidateTweet): FeatureMap = {\n    val tweet = candidate.tweet\n    val quotedTweet = tweet.filter(_.quotedTweet.exists(_.tweetId != 0)).flatMap(_.quotedTweet)\n    val mentions = tweet.flatMap(_.mentions).getOrElse(Seq.empty)\n    val coreData = tweet.flatMap(_.coreData)\n    val share = coreData.flatMap(_.share)\n    val reply = coreData.flatMap(_.reply)\n    val communityId = tweet.flatMap(_.communities).flatMap(_.communityIds.headOption)\n\n    FeatureMapBuilder()\n      .add(AuthorIdFeature, coreData.map(_.userId))\n      .add(CommunityIdFeature, communityId)\n      .add(DirectedAtUserIdFeature, coreData.flatMap(_.directedAtUser.map(_.userId)))\n      .add(EarlybirdFeature, candidate.features)\n      .add(EarlybirdScoreFeature, candidate.features.map(_.earlybirdScore))\n      .add(\n        ExclusiveConversationAuthorIdFeature,\n        tweet.flatMap(_.exclusiveTweetControl.map(_.conversationAuthorId)))\n      .add(FromInNetworkSourceFeature, false)\n      .add(HasImageFeature, tweet.exists(TweetMediaFeaturesExtractor.hasImage))\n      .add(HasVideoFeature, tweet.exists(TweetMediaFeaturesExtractor.hasVideo))\n      .add(InReplyToTweetIdFeature, reply.flatMap(_.inReplyToStatusId))\n      .add(InReplyToUserIdFeature, reply.map(_.inReplyToUserId))\n      .add(IsRetweetFeature, share.isDefined)\n      .add(MentionScreenNameFeature, mentions.map(_.screenName))\n      .add(MentionUserIdFeature, mentions.flatMap(_.userId))\n      .add(QuotedTweetIdFeature, quotedTweet.map(_.tweetId))\n      .add(QuotedUserIdFeature, quotedTweet.map(_.userId))\n      .add(SourceTweetIdFeature, share.map(_.sourceStatusId))\n      .add(SourceUserIdFeature, share.map(_.sourceUserId))\n      .add(TweetUrlsFeature, candidate.features.flatMap(_.urlsList).getOrElse(Seq.empty))\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/earlybird/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/communities\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/earlybird/EarlybirdResponseTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer.earlybird\n\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdSearchResultFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetUrlsFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.search.earlybird.{thriftscala => eb}\n\nobject EarlybirdResponseTransformer {\n\n  val features: Set[Feature[_, _]] = Set(\n    EarlybirdScoreFeature,\n    EarlybirdSearchResultFeature,\n    InReplyToTweetIdFeature,\n    InReplyToUserIdFeature,\n    IsRetweetFeature,\n    TweetUrlsFeature\n  )\n\n  def transform(candidate: eb.ThriftSearchResult): FeatureMap = {\n    val metadata = candidate.metadata\n    val isRetweet = metadata.flatMap(_.isRetweet).getOrElse(false)\n    val sharedStatusId = metadata.map(_.sharedStatusId).getOrElse(0L)\n    val referencedTweetAuthorId = metadata.map(_.referencedTweetAuthorId).getOrElse(0L)\n    val inReplyToTweetId = if (!isRetweet && sharedStatusId > 0) Some(sharedStatusId) else None\n    val inReplyToUserId =\n      if (!isRetweet && sharedStatusId > 0 && referencedTweetAuthorId > 0)\n        Some(referencedTweetAuthorId)\n      else None\n\n    FeatureMapBuilder()\n      .add(EarlybirdSearchResultFeature, Some(candidate))\n      .add(EarlybirdScoreFeature, candidate.metadata.flatMap(_.score))\n      .add(InReplyToTweetIdFeature, inReplyToTweetId)\n      .add(InReplyToUserIdFeature, inReplyToUserId)\n      .add(IsRetweetFeature, isRetweet)\n      .add(\n        TweetUrlsFeature,\n        candidate.metadata.flatMap(_.tweetUrls.map(_.map(_.originalUrl))).getOrElse(Seq.empty))\n      .build()\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/earlybird/ScoredTweetsCommunitiesResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer.earlybird\n\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityIdFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.search.earlybird.{thriftscala => t}\n\nobject ScoredTweetsCommunitiesResponseFeatureTransformer\n    extends CandidateFeatureTransformer[t.ThriftSearchResult] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsEarlybirdCommunitiesResponse\")\n\n  private val candidateSourceFeatures = Set(\n    FromInNetworkSourceFeature,\n    ServedTypeFeature\n  )\n\n  override val features: Set[Feature[_, _]] =\n    EarlybirdResponseTransformer.features ++ Set(CommunityIdFeature) ++ candidateSourceFeatures\n\n  override def transform(candidate: t.ThriftSearchResult): FeatureMap = {\n    val baseFeatures = EarlybirdResponseTransformer.transform(candidate)\n\n    val communityIdOpt =\n      candidate.tweetypieTweet.flatMap(_.communities.flatMap(_.communityIds.headOption))\n\n    val features = FeatureMapBuilder()\n      .add(FromInNetworkSourceFeature, false)\n      .add(ServedTypeFeature, hmt.ServedType.ForYouCommunity)\n      .add(CommunityIdFeature, communityIdOpt)\n      .build()\n\n    baseFeatures ++ features\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/earlybird/ScoredTweetsEarlybirdFrsResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer.earlybird\n\nimport com.twitter.home_mixer.model.HomeFeatures.DebugStringFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.search.earlybird.{thriftscala => eb}\n\nobject ScoredTweetsEarlybirdFrsResponseFeatureTransformer\n    extends CandidateFeatureTransformer[eb.ThriftSearchResult] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsEarlybirdFrsResponse\")\n\n  private val candidateSourceFeatures = Set(\n    FromInNetworkSourceFeature,\n    ServedTypeFeature,\n    DebugStringFeature\n  )\n\n  override val features: Set[Feature[_, _]] =\n    EarlybirdResponseTransformer.features ++ candidateSourceFeatures\n\n  override def transform(candidate: eb.ThriftSearchResult): FeatureMap = {\n    val baseFeatures = EarlybirdResponseTransformer.transform(candidate)\n\n    val features = FeatureMapBuilder()\n      .add(FromInNetworkSourceFeature, false)\n      .add(ServedTypeFeature, hmt.ServedType.ForYouFrs)\n      .add(DebugStringFeature, Some(\"FRS\"))\n      .build()\n\n    baseFeatures ++ features\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/earlybird/ScoredTweetsEarlybirdInNetworkResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer.earlybird\n\nimport com.twitter.home_mixer.model.HomeFeatures.DebugStringFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.search.earlybird.{thriftscala => eb}\n\nobject ScoredTweetsEarlybirdInNetworkResponseFeatureTransformer\n    extends CandidateFeatureTransformer[eb.ThriftSearchResult] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsEarlybirdInNetworkResponse\")\n\n  private val candidateSourceFeatures = Set(\n    FromInNetworkSourceFeature,\n    ServedTypeFeature,\n    DebugStringFeature\n  )\n\n  override val features: Set[Feature[_, _]] =\n    EarlybirdResponseTransformer.features ++ candidateSourceFeatures\n\n  override def transform(candidate: eb.ThriftSearchResult): FeatureMap = {\n    val baseFeatures = EarlybirdResponseTransformer.transform(candidate)\n\n    val features = FeatureMapBuilder()\n      .add(FromInNetworkSourceFeature, true)\n      .add(ServedTypeFeature, hmt.ServedType.ForYouInNetwork)\n      .add(DebugStringFeature, Some(\"In Network\"))\n      .build()\n\n    baseFeatures ++ features\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/response_transformer/earlybird/ScoredTweetsEarlybirdNsfwResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.response_transformer.earlybird\n\nimport com.twitter.home_mixer.model.HomeFeatures.DebugStringFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.search.earlybird.{thriftscala => eb}\n\nobject ScoredTweetsEarlybirdNsfwResponseFeatureTransformer\n    extends CandidateFeatureTransformer[eb.ThriftSearchResult] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ScoredTweetsEarlybirdNsfwResponse\")\n\n  private val candidateSourceFeatures = Set(\n    FromInNetworkSourceFeature,\n    ServedTypeFeature,\n    DebugStringFeature\n  )\n\n  override val features: Set[Feature[_, _]] =\n    EarlybirdResponseTransformer.features ++ candidateSourceFeatures\n\n  override def transform(candidate: eb.ThriftSearchResult): FeatureMap = {\n    val baseFeatures = EarlybirdResponseTransformer.transform(candidate)\n\n    val features = FeatureMapBuilder()\n      .add(FromInNetworkSourceFeature, false)\n      .add(ServedTypeFeature, hmt.ServedType.ForYouNsfwVideoContent)\n      .add(DebugStringFeature, Some(\"Nsfw Video\"))\n      .build()\n\n    baseFeatures ++ features\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/AuthorBasedListwiseRescoringProvider.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject AuthorBasedListwiseRescoringProvider\n    extends ListwiseRescoringProvider[CandidateWithFeatures[TweetCandidate], Long] {\n  private val MinFollowed = 50\n\n  /**\n   * Author-based, the groupBy key is the author id.\n   * Rescore the list of candidates that share the same author id.\n   */\n  override def groupByKey(candidate: CandidateWithFeatures[TweetCandidate]): Option[Long] =\n    candidate.features.getOrElse(AuthorIdFeature, None)\n\n  /**\n   * Defines the list of author-based candidate rescorers.\n   * Multiply the rescorers together for each candidate.\n   */\n  override def candidateRescoringFactor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate],\n    index: Int\n  ): Double = {\n    val isSmallFollowGraph =\n      query.features.get.getOrElse(SGSFollowedUsersFeature, Seq.empty).size <= MinFollowed\n\n    val decayFactor = if (isSmallFollowGraph) {\n      query.params(ScoredTweetsParam.SmallFollowGraphAuthorDiversityDecayFactor)\n    } else {\n      query.params(ScoredTweetsParam.AuthorDiversityDecayFactor)\n    }\n\n    val floor =\n      if (isSmallFollowGraph) query.params(ScoredTweetsParam.SmallFollowGraphAuthorDiversityFloor)\n      else query.params(ScoredTweetsParam.AuthorDiversityFloor)\n\n    authorDiversityBasedRescorer(index = index, decayFactor = decayFactor, floor = floor)\n  }\n\n  /**\n   * Re-scoring multiplier to apply to multiple tweets from the same author.\n   * Provides an exponential decay based discount by position (with a floor).\n   */\n  def authorDiversityBasedRescorer(\n    index: Int,\n    decayFactor: Double,\n    floor: Double\n  ): Double = (1 - floor) * Math.pow(decayFactor, index) + floor\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/user_history\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/scorer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/util\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/control_ai\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/impressed_tweets\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"src/thrift/com/twitter/timelines/control_ai:timeline-control-ai-thrift-scala\",\n        \"timelineservice/common:model\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/CandidateSourceDiversityListwiseRescoringProvider.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceSignalFeature\nimport com.twitter.home_mixer.model.candidate_source.SourceSignal\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject CandidateSourceDiversityListwiseRescoringProvider\n    extends ListwiseRescoringProvider[\n      CandidateWithFeatures[TweetCandidate],\n      (hmt.ServedType, Option[SourceSignal])\n    ] {\n\n  override def groupByKey(\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Option[(hmt.ServedType, Option[SourceSignal])] = {\n    val servedType = candidate.features.get(ServedTypeFeature)\n    val sourceSignalOpt = candidate.features.getOrElse(SourceSignalFeature, None)\n    Some((servedType, sourceSignalOpt))\n  }\n\n  override def candidateRescoringFactor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate],\n    index: Int\n  ): Double =\n    candidate.features.get(ServedTypeFeature) match {\n      case hmt.ServedType.ForYouInNetwork => 1.0\n      case _ =>\n        if (query.params(ScoredTweetsParam.EnableCandidateSourceDiversityDecay)) {\n          val decayFactor = query.params(ScoredTweetsParam.CandidateSourceDiversityDecayFactor)\n          val floor = query.params(ScoredTweetsParam.CandidateSourceDiversityFloor)\n          candidateSourceDiversityRescorer(index = index, decayFactor = decayFactor, floor = floor)\n        } else 1.0\n    }\n\n  /**\n   * Re-scoring multiplier to apply to multiple tweets from the same candidate source and reason.\n   * Provides an exponential decay based discount by position (with a floor).\n   */\n  def candidateSourceDiversityRescorer(\n    index: Int,\n    decayFactor: Double,\n    floor: Double\n  ): Double = (1 - floor) * Math.pow(decayFactor, index) + floor\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/ContentExplorationListwiseRescoringProvider.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableContentExplorationCandidateMaxCountParam\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject ContentExplorationListwiseRescoringProvider\n    extends ListwiseRescoringProvider[CandidateWithFeatures[TweetCandidate], hmt.ServedType] {\n\n  override def groupByKey(\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Option[hmt.ServedType] =\n    Some(candidate.features.get(ServedTypeFeature))\n\n  override def candidateRescoringFactor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate],\n    index: Int\n  ): Double = {\n    if (query.params(EnableContentExplorationCandidateMaxCountParam)) {\n      val servedType = candidate.features.get(ServedTypeFeature)\n      if (servedType == hmt.ServedType.ForYouUserInterestSummary ||\n        servedType == hmt.ServedType.ForYouContentExploration ||\n        servedType == hmt.ServedType.ForYouContentExplorationTier2 ||\n        servedType == hmt.ServedType.ForYouContentExplorationDeepRetrievalI2i ||\n        servedType == hmt.ServedType.ForYouContentExplorationTier2DeepRetrievalI2i) 0.0001\n      else 1.0\n    } else 1.0\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, Double] = {\n    candidates\n      .groupBy(groupByKey)\n      .flatMap {\n        case (Some(servedType), groupedCandidates) =>\n          groupedCandidates.zipWithIndex.map {\n            case (candidate, index) =>\n              candidate.candidate.id -> candidateRescoringFactor(query, candidate, index)\n          }\n        case _ => Map.empty\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/ControlAiRescorer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.ControlAiEmbeddingSimilarityThresholdParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.ControlAiShowLessScaleFactorParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.ControlAiShowMoreScaleFactorParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableControlAiParam\nimport com.twitter.home_mixer.product.scored_tweets.util.ControlAiUtil\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.control_ai.ControlAiTopicEmbeddingMapFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.control_ai.UserControlAiFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.control_ai.control.{thriftscala => ci}\n\nsealed trait ControlAiRescorer extends RescoringFactorProvider\n\nobject ControlAiRescorer {\n\n  private def buildControlAiRescorer(\n    actionType: ci.ActionType,\n    factorParam: FSBoundedParam[Double]\n  ): ControlAiRescorer = {\n    new ControlAiRescorer {\n      override def selector(\n        query: PipelineQuery,\n        candidate: CandidateWithFeatures[TweetCandidate]\n      ): Boolean = {\n        if (query.params(EnableControlAiParam)) {\n          val actions = query.features\n            .flatMap(_.getOrElse(UserControlAiFeature, None))\n            .map(_.actions).getOrElse(Seq.empty).filter(_.actionType == actionType)\n          actions.exists(\n            ControlAiUtil.conditionMatch(\n              _,\n              candidate,\n              query.features.map(_.get(ControlAiTopicEmbeddingMapFeature)).getOrElse(Map.empty),\n              threshold = query.params(ControlAiEmbeddingSimilarityThresholdParam)\n            )\n          )\n        } else false\n      }\n\n      override def factor(\n        query: PipelineQuery,\n        candidate: CandidateWithFeatures[TweetCandidate]\n      ): Double = query.params(factorParam)\n    }\n  }\n\n  val allRescorers = Seq(\n    buildControlAiRescorer(ci.ActionType.More, ControlAiShowMoreScaleFactorParam),\n    buildControlAiRescorer(ci.ActionType.Less, ControlAiShowLessScaleFactorParam)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/DeepRetrievalListwiseRescoringProvider.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.DeepRetrievalMaxCountParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableDeepRetrievalMaxCountParam\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject DeepRetrievalListwiseRescoringProvider\n    extends ListwiseRescoringProvider[CandidateWithFeatures[TweetCandidate], hmt.ServedType] {\n\n  override def groupByKey(\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Option[hmt.ServedType] =\n    Some(candidate.features.get(ServedTypeFeature))\n\n  override def candidateRescoringFactor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate],\n    index: Int\n  ): Double = {\n    if (query.params(EnableDeepRetrievalMaxCountParam)) {\n      val servedType = candidate.features.get(ServedTypeFeature)\n      val maxCount = query.params(DeepRetrievalMaxCountParam)\n      if (servedType == hmt.ServedType.ForYouContentExplorationDeepRetrievalI2i && index >= maxCount)\n        0.0001\n      else 1.0\n    } else 1.0\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, Double] = {\n    candidates\n      .groupBy(groupByKey)\n      .flatMap {\n        case (_, groupedCandidates) =>\n          groupedCandidates.zipWithIndex.map {\n            case (candidate, index) =>\n              candidate.candidate.id -> candidateRescoringFactor(query, candidate, index)\n          }\n        case _ => Map.empty\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/DiversityDiscountProvider.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\n\ntrait DiversityDiscountProvider {\n\n  /**\n   * Fetch the ID of the entity to diversify\n   */\n  def entityId(candidate: CandidateWithFeatures[TweetCandidate]): Option[Long]\n\n  /**\n   * Compute discount factor for each candidate based on position (zero-based)\n   * relative to other candidates associated with the same entity\n   */\n  def discount(position: Int): Double\n\n  /**\n   * Return candidate IDs sorted by score in descending order\n   */\n  def sort(candidates: Seq[CandidateWithFeatures[TweetCandidate]]): Seq[Long] = candidates\n    .map { candidate =>\n      (candidate.candidate.id, candidate.features.getOrElse(ScoreFeature, None).getOrElse(0.0))\n    }\n    .sortBy(_._2)(Ordering.Double.reverse)\n    .map(_._1)\n\n  /**\n   * Group by the specified entity ID (e.g. authors, likers, followers)\n   * Sort each group by score in descending order\n   * Determine the discount factor based on the position of each candidate\n   */\n  def apply(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, Double] = candidates\n    .groupBy(entityId)\n    .flatMap {\n      case (entityIdOpt, entityCandidates) =>\n        val sortedCandidateIds = sort(entityCandidates)\n\n        if (entityIdOpt.isDefined) {\n          sortedCandidateIds.zipWithIndex.map {\n            case (candidateId, index) =>\n              candidateId -> discount(index)\n          }\n        } else sortedCandidateIds.map(_ -> 1.0)\n    }\n}\n\nobject AuthorDiversityDiscountProvider extends DiversityDiscountProvider {\n  private val Decay = 0.5\n  private val Floor = 0.25\n\n  override def entityId(candidate: CandidateWithFeatures[TweetCandidate]): Option[Long] =\n    candidate.features.getOrElse(AuthorIdFeature, None)\n\n  // Provides an exponential decay based discount by position (with a floor)\n  override def discount(position: Int): Double =\n    (1 - Floor) * Math.pow(Decay, position) + Floor\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/EvergreenDeepRetrievalCrossBorderListwiseRescoringProvider.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EvergreenDeepRetrievalCrossBorderMaxCountParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableEvergreenDeepRetrievalCrossBorderMaxCountParam\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject EvergreenDeepRetrievalCrossBorderListwiseRescoringProvider\n    extends ListwiseRescoringProvider[CandidateWithFeatures[TweetCandidate], hmt.ServedType] {\n\n  override def groupByKey(\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Option[hmt.ServedType] =\n    Some(candidate.features.get(ServedTypeFeature))\n\n  override def candidateRescoringFactor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate],\n    index: Int\n  ): Double = {\n    if (query.params(EnableEvergreenDeepRetrievalCrossBorderMaxCountParam)) {\n      val servedType = candidate.features.get(ServedTypeFeature)\n      val maxCount = query.params(EvergreenDeepRetrievalCrossBorderMaxCountParam)\n      if (servedType == hmt.ServedType.ForYouEvergreenDeepRetrievalCrossBorderHome && index >= maxCount)\n        0.0001\n      else 1.0\n    } else 1.0\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, Double] = {\n    candidates\n      .groupBy(groupByKey)\n      .flatMap {\n        case (_, groupedCandidates) =>\n          groupedCandidates.zipWithIndex.map {\n            case (candidate, index) =>\n              candidate.candidate.id -> candidateRescoringFactor(query, candidate, index)\n          }\n        case _ => Map.empty\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/EvergreenDeepRetrievalListwiseRescoringProvider.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EvergreenDeepRetrievalMaxCountParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableEvergreenDeepRetrievalMaxCountParam\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject EvergreenDeepRetrievalListwiseRescoringProvider\n    extends ListwiseRescoringProvider[CandidateWithFeatures[TweetCandidate], hmt.ServedType] {\n\n  override def groupByKey(\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Option[hmt.ServedType] =\n    Some(candidate.features.get(ServedTypeFeature))\n\n  override def candidateRescoringFactor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate],\n    index: Int\n  ): Double = {\n    if (query.params(EnableEvergreenDeepRetrievalMaxCountParam)) {\n      val servedType = candidate.features.get(ServedTypeFeature)\n      val maxCount = query.params(EvergreenDeepRetrievalMaxCountParam)\n      if (servedType == hmt.ServedType.ForYouEvergreenDeepRetrievalHome && index >= maxCount)\n        0.0001\n      else 1.0\n    } else 1.0\n  }\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, Double] = {\n    candidates\n      .groupBy(groupByKey)\n      .flatMap {\n        case (_, groupedCandidates) =>\n          groupedCandidates.zipWithIndex.map {\n            case (candidate, index) =>\n              candidate.candidate.id -> candidateRescoringFactor(query, candidate, index)\n          }\n        case _ => Map.empty\n      }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/GrokSlopScoreRescorer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.finagle.stats.DefaultStatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.GrokSlopScoreFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.GrokSlopScoreDecayValueParam\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject GrokSlopScoreRescorer {\n\n  private val treatmentValue = 3L\n\n  private val statsReceiver = DefaultStatsReceiver.scope(\"GrokSlopScoreRescorer\")\n  private val numRescoredCandidatesCounter = statsReceiver.counter(\"rescored\")\n  private val totalCandidatesCounter = statsReceiver.counter(\"total\")\n\n  private def onlyIf(query: PipelineQuery): Boolean = {\n    query.params(GrokSlopScoreDecayValueParam) < 1.0\n  }\n\n  def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, Double] = {\n    if (!onlyIf(query)) {\n      return candidates.map { candidate => candidate.candidate.id -> 1.0 }.toMap\n    }\n\n    val decayValue = query.params(GrokSlopScoreDecayValueParam)\n\n    val rescoredCandidates = candidates.map { candidate =>\n      val featureValue = candidate.features.getOrElse(GrokSlopScoreFeature, None)\n      val rescoreFactor = if (featureValue.contains(treatmentValue)) decayValue else 1.0\n      candidate.candidate.id -> rescoreFactor\n    }\n\n    numRescoredCandidatesCounter.incr(rescoredCandidates.count(_._2 != 1.0))\n    totalCandidatesCounter.incr(candidates.size)\n\n    rescoredCandidates.toMap\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/HeuristicScorer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.PhoenixScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnablePhoenixScorerParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.EnableNoNegHeuristicParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnablePhoenixScoreParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.MtlNormalization\nimport com.twitter.home_mixer.util.RerankerUtil.Epsilon\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.util.MtlNormalizer\n\nobject HeuristicScorer extends Scorer[PipelineQuery, TweetCandidate] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"Heuristic\")\n\n  override val features: Set[Feature[_, _]] = Set(ScoreFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val noNegHeuristic = query.params(EnableNoNegHeuristicParam)\n    val rescorers = Seq(\n      RescoreOutOfNetwork,\n      RescoreReplies,\n      RescoreMTLNormalization(\n        MtlNormalizer(\n          alpha = query.params(MtlNormalization.AlphaParam) / 100.0,\n          beta = query.params(MtlNormalization.BetaParam),\n          gamma = query.params(MtlNormalization.GammaParam)\n        )\n      ),\n      RescoreListwise(ContentExplorationListwiseRescoringProvider(query, candidates)),\n      RescoreListwise(DeepRetrievalListwiseRescoringProvider(query, candidates)),\n      RescoreListwise(EvergreenDeepRetrievalListwiseRescoringProvider(query, candidates)),\n      RescoreListwise(\n        EvergreenDeepRetrievalCrossBorderListwiseRescoringProvider(query, candidates)\n      ),\n      RescoreListwise(AuthorBasedListwiseRescoringProvider(query, candidates)),\n      RescoreListwise(ImpressedAuthorDecayRescoringProvider(query, candidates)),\n      RescoreListwise(ImpressedMediaClusterBasedListwiseRescoringProvider(query, candidates)),\n      RescoreListwise(ImpressedImageClusterBasedListwiseRescoringProvider(query, candidates)),\n      RescoreListwise(CandidateSourceDiversityListwiseRescoringProvider(query, candidates)),\n      RescoreListwise(GrokSlopScoreRescorer(query, candidates)),\n      RescoreFeedbackFatigue(query),\n      RescoreListwise(MultimodalEmbeddingRescorer(query, candidates)),\n      RescoreLiveContent\n    ) ++ ControlAiRescorer.allRescorers\n\n    val usePhoenix = query.params(EnablePhoenixScorerParam) && query.params(EnablePhoenixScoreParam)\n\n    val updatedScores = candidates.map { candidate =>\n      val scoreOpt =\n        if (usePhoenix) candidate.features.getOrElse(PhoenixScoreFeature, None)\n        else candidate.features.getOrElse(ScoreFeature, None)\n\n      val scaleFactor = rescorers.map(_(query, candidate)).product\n      val updatedScore = scoreOpt.map { score =>\n        if (score < Epsilon && noNegHeuristic) score else score * scaleFactor\n      }\n      FeatureMap(ScoreFeature, updatedScore)\n    }\n\n    Stitch.value(updatedScores)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/ImpressedAuthorDecayRescoringProvider.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedAuthorIdsFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableImpressionBasedAuthorDecay\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.impressed_tweets.ImpressedTweets\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject ImpressedAuthorDecayRescoringProvider {\n\n  private def calculateAuthorImpressionFrequencies(query: PipelineQuery): Map[Long, Int] = {\n    val impressedTweetIds =\n      query.features.map(_.getOrElse(ImpressedTweets, Seq.empty)).getOrElse(Seq.empty).toSet\n\n    val servedAuthorMap = query.features.map(_.get(ServedAuthorIdsFeature)).getOrElse(Map.empty)\n\n    servedAuthorMap\n      .map {\n        case (authorId, tweetIds) =>\n          val impressedCount = tweetIds.count(impressedTweetIds.contains)\n          authorId -> impressedCount\n      }\n      .filter(_._2 > 0) // Only include authors with at least one impressed tweet\n  }\n\n  private def groupByKey(candidate: CandidateWithFeatures[TweetCandidate]): Option[Long] =\n    candidate.features.getOrElse(AuthorIdFeature, None)\n\n  private def onlyIf(query: PipelineQuery): Boolean = query.params(EnableImpressionBasedAuthorDecay)\n\n  def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, Double] = {\n    if (onlyIf(query)) {\n      val outNetworkDecayFactor =\n        query.params(ScoredTweetsParam.AuthorDiversityOutNetworkDecayFactor)\n      val outNetworkFloor = query.params(ScoredTweetsParam.AuthorDiversityOutNetworkFloor)\n\n      val inNetworkDecayFactor = query.params(ScoredTweetsParam.AuthorDiversityInNetworkDecayFactor)\n      val inNetworkFloor = query.params(ScoredTweetsParam.AuthorDiversityInNetworkFloor)\n\n      val authorFreq = calculateAuthorImpressionFrequencies(query)\n\n      candidates\n        .groupBy(groupByKey)\n        .flatMap {\n          case (Some(authorId), groupedCandidates) =>\n            val sortedCandidates = groupedCandidates\n              .sortBy(_.features.getOrElse(ScoreFeature, None).getOrElse(0.0))(\n                Ordering.Double.reverse)\n\n            sortedCandidates.zipWithIndex.map {\n              case (candidate, index) =>\n                candidate.candidate.id -> {\n                  val effectiveIndex = index + authorFreq.getOrElse(authorId, 0)\n                  val isInNetworkCandidate = candidate.features.getOrElse(InNetworkFeature, true)\n                  val decayFactor =\n                    if (isInNetworkCandidate) inNetworkDecayFactor else outNetworkDecayFactor\n                  val floor = if (isInNetworkCandidate) inNetworkFloor else outNetworkFloor\n                  authorDiversityBasedRescorer(effectiveIndex, decayFactor, floor)\n                }\n            }\n\n          case _ => Map.empty\n        }\n    } else Map.empty\n  }\n\n  /**\n   * Re-scoring multiplier to apply to multiple tweets from the same author.\n   * Provides an exponential decay based discount by position (with a floor).\n   */\n  private def authorDiversityBasedRescorer(\n    index: Int,\n    decayFactor: Double,\n    floor: Double\n  ): Double = (1 - floor) * Math.pow(decayFactor, index) + floor\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/ImpressedImageClusterBasedListwiseRescoringProvider.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.finagle.stats.DefaultStatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.ImpressedImageClusterIds\nimport com.twitter.home_mixer.model.HomeFeatures.ClipImageClusterIdsFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ImpressedImageClusterBasedRescoringParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableImageClusterDecayParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableImageClusterFeatureHydrationParam\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject ImpressedImageClusterBasedListwiseRescoringProvider {\n\n  /**\n   * Impressed Image cluster-Id based\n   * Rescore the list of candidates that have previously impressed imageCluster Ids.\n   */\n  private val impressedImageStats =\n    DefaultStatsReceiver.scope(\"ImpressedImageClusterBasedListwiseRescoringProvider\")\n  private val percentRescoredCandidatesStat =\n    impressedImageStats.stat(\"percent_rescored_candidates_x10\")\n\n  private def impressedImageClusterIds(query: PipelineQuery) =\n    query.features.map(_.getOrElse(ImpressedImageClusterIds, Seq[Long]())).getOrElse(Seq[Long]())\n\n  private def imageClusterId(\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ) = candidate.features.getOrElse(ClipImageClusterIdsFeature, Map[Long, Long]()).values.toSeq\n\n  private def rescoringFactor(\n    clusterIds: Seq[Long],\n    impressedImageClusterIdsToCountMap: Map[Long, Int],\n    decayFactor: Double\n  ): Double = {\n    val decayFactors = clusterIds.map { clusterId =>\n      impressedImageClusterIdsToCountMap.get(clusterId) match {\n        case Some(count) => math.pow(1.0 - decayFactor, count)\n        case None => 1.0\n      }\n    }\n    decayFactors.product\n  }\n\n  private def onlyIf(query: PipelineQuery): Boolean = {\n    (query.params(ImpressedImageClusterBasedRescoringParam) > 0.0) &&\n    query.params(EnableImageClusterDecayParam) &&\n    query.params(EnableImageClusterFeatureHydrationParam)\n  }\n\n  def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, Double] = {\n    if (!onlyIf(query)) {\n      return candidates.map { candidate =>\n        candidate.candidate.id -> 1.0\n      }.toMap\n    }\n\n    val impressedImageClusterIdsToCountMap = impressedImageClusterIds(query)\n      .groupBy(identity)\n      .mapValues(_.size)\n\n    val decayFactor = query.params(ImpressedImageClusterBasedRescoringParam)\n\n    val rescoringFactors = candidates.map { candidate =>\n      val imageClusterIds = imageClusterId(candidate)\n      val rescoreFactor =\n        rescoringFactor(imageClusterIds, impressedImageClusterIdsToCountMap, decayFactor)\n      candidate.candidate.id -> rescoreFactor\n    }.toMap\n\n    // Update Rescored candidates stats by number of candidates with rescoreFactor != 1\n    percentRescoredCandidatesStat.add(\n      (rescoringFactors.count(_._2 != 1.0) * 1000.0 / (candidates.size + 0.01)).toFloat)\n\n    rescoringFactors\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/ImpressedMediaClusterBasedListwiseRescoringProvider.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.finagle.stats.DefaultStatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.ImpressedMediaClusterIds\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaClusterIdsFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ImpressedMediaClusterBasedRescoringParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableMediaClusterDecayParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableMediaClusterFeatureHydrationParam\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject ImpressedMediaClusterBasedListwiseRescoringProvider {\n\n  /**\n   * Impressed Media cluster-Id based\n   * Rescore the list of candidates that have previously impressed mediaCluster Ids.\n   */\n  private val impressedMediaStats =\n    DefaultStatsReceiver.scope(\"ImpressedMediaClusterBasedListwiseRescoringProvider\")\n  private val percentRescoredCandidatesStat =\n    impressedMediaStats.stat(\"percent_rescored_candidates_x10\")\n\n  private def impressedMediaClusterIds(query: PipelineQuery) =\n    query.features.map(_.getOrElse(ImpressedMediaClusterIds, Seq[Long]())).getOrElse(Seq[Long]())\n\n  private def mediaClusterId(candidate: CandidateWithFeatures[TweetCandidate]) =\n    candidate.features.getOrElse(TweetMediaClusterIdsFeature, Map[Long, Long]()).values.toSeq\n\n  private def rescoringFactor(\n    clusterIds: Seq[Long],\n    impressedMediaClusterIdsToCountMap: Map[Long, Int],\n    decayFactor: Double\n  ): Double = {\n    val decayFactors = clusterIds.map { clusterId =>\n      impressedMediaClusterIdsToCountMap.get(clusterId) match {\n        case Some(count) => math.pow(1.0 - decayFactor, count)\n        case None => 1.0\n      }\n    }\n    decayFactors.product\n  }\n\n  private def onlyIf(query: PipelineQuery): Boolean = {\n    (query.params(ImpressedMediaClusterBasedRescoringParam) > 0.0) &&\n    query.params(EnableMediaClusterDecayParam) &&\n    query.params(EnableMediaClusterFeatureHydrationParam)\n  }\n\n  def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, Double] = {\n    if (!onlyIf(query)) {\n      return candidates.map { candidate =>\n        candidate.candidate.id -> 1.0\n      }.toMap\n    }\n\n    val impressedMediaClusterIdsToCountMap = impressedMediaClusterIds(query)\n      .groupBy(identity)\n      .mapValues(_.size)\n\n    val decayFactor = query.params(ImpressedMediaClusterBasedRescoringParam)\n\n    val rescoringFactors = candidates.map { candidate =>\n      val mediaClusterIds = mediaClusterId(candidate)\n      val rescoreFactor =\n        rescoringFactor(mediaClusterIds, impressedMediaClusterIdsToCountMap, decayFactor)\n      candidate.candidate.id -> rescoreFactor\n    }.toMap\n\n    // Update Rescored candidates stats by number of candidates with rescoreFactor != 1\n    percentRescoredCandidatesStat.add(\n      (rescoringFactors.count(_._2 != 1.0) * 1000.0 / (candidates.size + 0.01)).toFloat)\n\n    rescoringFactors\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/ListwiseRescoringProvider.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Defines a listwise rescoring provider for use of rescoring a candidate dependent on the other candidates that\n * co-exist with it.\n *\n * Requires:\n *  1) groupByKey: How to define the set of candidates that are dependent on each other for scoring.\n *  2) candidateRescoringFactor: Compute the final rescoring factor for each candidate in the group.\n *\n *  This rescorer will sort the grouped candidates by their current score, and then apply the rescoring factors to\n *  each candidate in their group.\n */\ntrait ListwiseRescoringProvider[C <: CandidateWithFeatures[TweetCandidate], K] {\n\n  /**\n   * Fetch the key used to create groups of candidates\n   */\n  def groupByKey(candidate: C): Option[K]\n\n  /**\n   * Compute the factor for each candidate based on position (zero-based)\n   * relative to other candidates associated with the same key\n   */\n  def candidateRescoringFactor(query: PipelineQuery, candidate: C, index: Int): Double\n\n  /**\n   * Group by the specified key (e.g. authors, likers, followers)\n   * Sort each group by score in descending order\n   * Determine the rescoring factor based on the position of each candidate\n   */\n  def apply(\n    query: PipelineQuery,\n    candidates: Seq[C]\n  ): Map[Long, Double] = candidates\n    .groupBy(groupByKey)\n    .flatMap {\n      case (Some(_), groupedCandidates) =>\n        val sortedCandidates = groupedCandidates\n          .sortBy(_.features.getOrElse(ScoreFeature, None).getOrElse(0.0))(Ordering.Double.reverse)\n\n        sortedCandidates.zipWithIndex.map {\n          case (candidate, index) =>\n            candidate.candidate.id -> candidateRescoringFactor(query, candidate, index)\n        }\n\n      case _ => Map.empty\n    }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/LowSignalScorer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidatePipelines\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateSourcePosition\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport scala.collection.immutable.ListSet\n\n/**\n * Assign scores based on candidate source positions, blending candidates from different sources\n */\nobject LowSignalScorer extends Scorer[PipelineQuery, TweetCandidate] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"LowSignal\")\n\n  override val features: Set[Feature[_, _]] = Set(ScoreFeature)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val candidatesByPipeline = candidates.groupBy {\n      _.features.getOrElse(CandidatePipelines, ListSet.empty[CandidatePipelineIdentifier]).head\n    }\n\n    val candidateScoreMap = candidatesByPipeline\n      .map {\n        case (_, pipelineCandidates) =>\n          val sortedCandidates = pipelineCandidates.sortBy(_.features.get(CandidateSourcePosition))\n          deduplicateAuthors(sortedCandidates).zipWithIndex.map {\n            case (candidate, index) => candidate.candidate.id -> index.toDouble\n          }\n      }.toSeq.flatten.toMap\n\n    val maxScore = candidates.size.toDouble\n    val updatedScores = candidates.map { candidate =>\n      val score = maxScore - candidateScoreMap.getOrElse(candidate.candidate.id, maxScore)\n      FeatureMap(ScoreFeature, Some(score))\n    }\n    Stitch.value(updatedScores)\n  }\n\n  def deduplicateAuthors(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Seq[CandidateWithFeatures[TweetCandidate]] = {\n    val seenAuthors = scala.collection.mutable.Set[Long]()\n    candidates.collect {\n      case c if seenAuthors.add(c.features.getOrElse(AuthorIdFeature, None).getOrElse(0L)) => c\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/MultimodalEmbeddingRescorer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.finagle.stats.DefaultStatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.MultiModalEmbeddingsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.MultiModalEmbeddingRescorerGammaParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.MultiModalEmbeddingRescorerMinScoreParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.MultiModalEmbeddingRescorerNumCandidatesParam\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport scala.collection.mutable.ArrayBuffer\n\nobject MultimodalEmbeddingRescorer {\n\n  private val statsReceiver = DefaultStatsReceiver.scope(\"MultimodalEmbeddingRescorer\")\n  private val rescoredCandidatesStat = statsReceiver.stat(\"rescored_candidates\")\n  private val rescoreFactorx100Stat = statsReceiver.stat(\"rescore_factor_x100\")\n\n  private def onlyIf(query: PipelineQuery): Boolean = {\n    (query.params(MultiModalEmbeddingRescorerGammaParam) > 0.0) &&\n    (query.params(MultiModalEmbeddingRescorerMinScoreParam) < 1.0)\n  }\n\n  private def dot(a: Array[Double], b: Array[Double]): Double = {\n    a.zip(b).map { case (x, y) => x * y }.sum\n  }\n\n  private def getSimilarityScoreFactors(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]],\n    gamma: Double,\n    minScore: Double\n  ): Map[Long, Double] = {\n    val embsBuilder = ArrayBuffer.empty[Array[Double]]\n    val factorsBuilder = Map.newBuilder[Long, Double]\n\n    candidates.foreach { candidate =>\n      val id = candidate.candidate.id\n      candidate.features.getOrElse(MultiModalEmbeddingsFeature, None) match {\n        case Some(embSeq: Seq[Double]) =>\n          val emb = embSeq.toArray\n          var similarCount = 0\n          var i = 0\n          while (i < embsBuilder.length) {\n            if (dot(embsBuilder(i), emb) > minScore)\n              similarCount += 1\n            i += 1\n          }\n          val factor = 1.0 / (1.0 + gamma * similarCount)\n          factorsBuilder += (id -> factor)\n          embsBuilder += emb\n        case _ =>\n      }\n    }\n\n    factorsBuilder.result()\n  }\n\n  def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, Double] = {\n    if (onlyIf(query)) {\n      // Get the top 100 candidates by score\n      val sortedCandidates =\n        candidates.sortBy(_.features.getOrElse(ScoreFeature, None).getOrElse(0.0))\n      // Get top 100 and rest of candidates\n      val numCandidatesToScore = query.params(MultiModalEmbeddingRescorerNumCandidatesParam)\n      val topCandidates = sortedCandidates.take(numCandidatesToScore)\n      val restCandidates = sortedCandidates.drop(numCandidatesToScore)\n\n      val candidateToFactorMap = getSimilarityScoreFactors(\n        topCandidates,\n        query.params(MultiModalEmbeddingRescorerGammaParam),\n        query.params(MultiModalEmbeddingRescorerMinScoreParam)\n      )\n      val rescoredCandidates = candidateToFactorMap.filter {\n        case (_, factor) => factor != 1.0\n      }\n\n      rescoredCandidatesStat.add(rescoredCandidates.size.toFloat)\n      rescoreFactorx100Stat.add(rescoredCandidates.values.map(_.toFloat).sum * 100)\n\n      // For the rest of the candidates, set factor to 1.0\n      restCandidates.map(candidate => candidate.candidate.id -> 1.0).toMap ++ candidateToFactorMap\n    } else candidates.map(candidate => candidate.candidate.id -> 1.0).toMap\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/NaviModelScorer.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.WeightedModelScoreFeature\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.scorer.PredictedScoreFeature.PredictedScoreFeatures\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.AllFeatures\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordConverter\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordExtractor\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.clients.predictionservice.PredictionGRPCService\nimport com.twitter.timelines.clients.predictionservice.PredictionServiceGRPCClient\nimport com.twitter.util.Future\nimport com.twitter.util.Return\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject CommonFeaturesDataRecordFeature\n    extends DataRecordInAFeature[PipelineQuery]\n    with FeatureWithDefaultOnFailure[PipelineQuery, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\nobject CandidateFeaturesDataRecordFeature\n    extends DataRecordInAFeature[TweetCandidate]\n    with FeatureWithDefaultOnFailure[TweetCandidate, DataRecord] {\n  override def defaultValue: DataRecord = new DataRecord()\n}\n\n@Singleton\ncase class NaviModelScorer @Inject() (\n  predictionGRPCService: PredictionGRPCService,\n  statsReceiver: StatsReceiver)\n    extends Scorer[ScoredTweetsQuery, TweetCandidate] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"NaviModel\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    CommonFeaturesDataRecordFeature,\n    CandidateFeaturesDataRecordFeature,\n    WeightedModelScoreFeature,\n    ScoreFeature\n  ) ++ PredictedScoreFeatures.asInstanceOf[Set[Feature[_, _]]]\n\n  private val queryDataRecordAdapter = new DataRecordConverter(AllFeatures())\n  private val candidatesDataRecordAdapter = new DataRecordConverter(AllFeatures())\n  private val resultDataRecordExtractor = new DataRecordExtractor(PredictedScoreFeatures)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val failuresStat = scopedStatsReceiver.stat(\"failures\")\n  private val responsesStat = scopedStatsReceiver.stat(\"responses\")\n  private val invalidResponsesCounter = scopedStatsReceiver.counter(\"invalidResponses\")\n  private val candidatesDataRecordAdapterLatencyStat =\n    scopedStatsReceiver.scope(\"candidatesDataRecordAdapter\").stat(\"latency_ms\")\n\n  private val StatsReadabilityMultiplier = 1000\n  private val Epsilon = 0.001\n  private val PredictedScoreStatName = f\"predictedScore${StatsReadabilityMultiplier}x\"\n  private val MissingScoreStatName = \"missingScore\"\n  private val scoreStat = scopedStatsReceiver.stat(f\"score${StatsReadabilityMultiplier}x\")\n\n  private val RequestBatchSize = 64\n  private val DataRecordConstructionParallelism = 32\n  private val ModelId = \"Home\"\n\n  private val modelClient = new PredictionServiceGRPCClient(\n    service = predictionGRPCService,\n    statsReceiver = statsReceiver,\n    requestBatchSize = RequestBatchSize,\n    useCompact = false\n  )\n\n  override def apply(\n    query: ScoredTweetsQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val commonRecord = query.features.map(queryDataRecordAdapter.toDataRecord)\n    val candidateRecords: Future[Seq[DataRecord]] =\n      Stat.time(candidatesDataRecordAdapterLatencyStat) {\n        OffloadFuturePools.parallelize[FeatureMap, DataRecord](\n          inputSeq = candidates.map(_.features),\n          transformer = candidatesDataRecordAdapter.toDataRecord(_),\n          parallelism = DataRecordConstructionParallelism,\n          default = new DataRecord\n        )\n      }\n\n    val scoreFeatureMaps = candidateRecords.flatMap { records =>\n      val predictionResponses =\n        modelClient.getPredictions(records, commonRecord, modelId = Some(ModelId))\n\n      predictionResponses.map { responses =>\n        failuresStat.add(responses.count(_.isThrow))\n        responsesStat.add(responses.size)\n\n        if (responses.size == candidates.size) {\n          val predictedScoreFeatureMaps = responses.map {\n            case Return(dataRecord) => resultDataRecordExtractor.fromDataRecord(dataRecord)\n            case _ => resultDataRecordExtractor.fromDataRecord(new DataRecord())\n          }\n\n          // Add Data Record to candidate Feature Map for logging in later stages\n          predictedScoreFeatureMaps.zip(records).map {\n            case (predictedScoreFeatureMap, candidateRecord) =>\n              val weightedModelScore = computeWeightedModelScore(query, predictedScoreFeatureMap)\n              scoreStat.add((weightedModelScore * StatsReadabilityMultiplier).toFloat)\n\n              predictedScoreFeatureMap +\n                (CandidateFeaturesDataRecordFeature, candidateRecord) +\n                (CommonFeaturesDataRecordFeature, commonRecord.getOrElse(new DataRecord())) +\n                (ScoreFeature, Some(weightedModelScore)) +\n                (WeightedModelScoreFeature, Some(weightedModelScore))\n          }\n        } else {\n          invalidResponsesCounter.incr()\n          throw PipelineFailure(IllegalStateFailure, \"Result size mismatched candidates size\")\n        }\n      }\n    }\n\n    Stitch.callFuture(scoreFeatureMaps)\n  }\n\n  /**\n   * Compute the weighted sum of predicted scores of all engagements\n   * Convert negative score to positive, if needed\n   */\n  private def computeWeightedModelScore(\n    query: PipelineQuery,\n    features: FeatureMap\n  ): Double = {\n    val weightedScoreAndModelWeightSeq = PredictedScoreFeatures.toSeq.map { predictedScoreFeature =>\n      val predictedScoreOpt = predictedScoreFeature.extractScore(features)\n\n      predictedScoreOpt match {\n        case Some(predictedScore) =>\n          scopedStatsReceiver\n            .stat(predictedScoreFeature.statName, PredictedScoreStatName)\n            .add((predictedScore * StatsReadabilityMultiplier).toFloat)\n        case None =>\n          scopedStatsReceiver.counter(predictedScoreFeature.statName, MissingScoreStatName).incr()\n      }\n\n      val weight = query.params(predictedScoreFeature.modelWeightParam)\n      val weightedScore = predictedScoreOpt.getOrElse(0.0) * weight\n      (weightedScore, weight)\n    }\n\n    val (weightedScores, modelWeights) = weightedScoreAndModelWeightSeq.unzip\n    val combinedScoreSum = weightedScores.sum\n\n    val positiveModelWeightsSum = modelWeights.filter(_ > 0.0).sum\n    val negativeModelWeightsSum = modelWeights.filter(_ < 0).sum.abs\n    val modelWeightsSum = positiveModelWeightsSum + negativeModelWeightsSum\n\n    val weightedScoresSum =\n      if (modelWeightsSum == 0) combinedScoreSum.max(0.0)\n      else if (combinedScoreSum < 0)\n        (combinedScoreSum + negativeModelWeightsSum) / modelWeightsSum * Epsilon\n      else combinedScoreSum + Epsilon\n\n    weightedScoresSum\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/PredictedScoreFeature.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.dal.personal_data.{thriftjava => pd}\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.Scoring.ModelWeights\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordOptionalFeature\nimport com.twitter.product_mixer.core.feature.datarecord.DoubleDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.prediction.features.recap.RecapFeatures\n\nsealed trait PredictedScoreFeature\n    extends DataRecordOptionalFeature[TweetCandidate, Double]\n    with DoubleDataRecordCompatible {\n\n  override val personalDataTypes: Set[pd.PersonalDataType] = Set.empty\n  def statName: String\n  def modelWeightParam: FSBoundedParam[Double]\n  def extractScore: FeatureMap => Option[Double] = _.getOrElse(this, None)\n}\n\nobject PredictedFavoriteScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_FAVORITED.getFeatureName\n  override val statName = \"fav\"\n  override val modelWeightParam = ModelWeights.FavParam\n}\n\nobject PredictedReplyScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_REPLIED.getFeatureName\n  override val statName = \"reply\"\n  override val modelWeightParam = ModelWeights.ReplyParam\n}\n\nobject PredictedRetweetScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_RETWEETED.getFeatureName\n  override val statName = \"retweet\"\n  override val modelWeightParam = ModelWeights.RetweetParam\n}\n\nobject PredictedReplyEngagedByAuthorScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_REPLIED_REPLY_ENGAGED_BY_AUTHOR.getFeatureName\n  override val statName = \"reply_engaged_by_author\"\n  override val modelWeightParam = ModelWeights.ReplyEngagedByAuthorParam\n}\n\nobject PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_GOOD_CLICKED_V1.getFeatureName\n  override val statName = \"good_click_convo_desc_favorited_or_replied\"\n  override val modelWeightParam = ModelWeights.GoodClickParam\n\n  override def extractScore: FeatureMap => Option[Double] = { featureMap =>\n    val goodClickV1Opt = featureMap.getOrElse(this, None)\n    val goodClickV2Opt = featureMap.getOrElse(PredictedGoodClickConvoDescUamGt2ScoreFeature, None)\n\n    (goodClickV1Opt, goodClickV2Opt) match {\n      case (Some(v1Score), Some(v2Score)) => Some(Math.max(v1Score, v2Score))\n      case _ => goodClickV1Opt.orElse(goodClickV2Opt)\n    }\n  }\n}\n\nobject PredictedGoodClickConvoDescUamGt2ScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_GOOD_CLICKED_V2.getFeatureName\n  override val statName = \"good_click_convo_desc_uam_gt_2\"\n  override val modelWeightParam = ModelWeights.GoodClickV2Param\n}\n\nobject PredictedGoodProfileClickScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_PROFILE_CLICKED_AND_PROFILE_ENGAGED.getFeatureName\n  override val statName = \"good_profile_click\"\n  override val modelWeightParam = ModelWeights.GoodProfileClickParam\n}\n\nobject PredictedVideoPlayback50ScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_VIDEO_PLAYBACK_50.getFeatureName\n  override val statName = \"video_playback_50\"\n  override val modelWeightParam = ModelWeights.VideoPlayback50Param\n}\n\nobject PredictedTweetDetailDwellScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_TWEET_DETAIL_DWELLED_15_SEC.getFeatureName\n  override val statName = \"tweet_detail_dwell\"\n  override val modelWeightParam = ModelWeights.TweetDetailDwellParam\n}\n\nobject PredictedProfileDwelledScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_PROFILE_DWELLED_20_SEC.getFeatureName\n  override val statName = \"profile_dwell\"\n  override val modelWeightParam = ModelWeights.ProfileDwelledParam\n}\n\nobject PredictedBookmarkScoreFeature extends PredictedScoreFeature {\n  override val featureName: String = RecapFeatures.PREDICTED_IS_BOOKMARKED.getFeatureName\n  override val statName = \"bookmark\"\n  override val modelWeightParam = ModelWeights.BookmarkParam\n}\n\nobject PredictedShareScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_SHARED.getFeatureName\n  override val statName = \"share\"\n  override val modelWeightParam = ModelWeights.ShareParam\n}\n\nobject PredictedShareMenuClickScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_SHARE_MENU_CLICKED.getFeatureName\n  override val statName = \"share_menu_click\"\n  override val modelWeightParam = ModelWeights.ShareMenuClickParam\n}\n\n// Negative Engagements\nobject PredictedNegativeFeedbackV2ScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_NEGATIVE_FEEDBACK_V2.getFeatureName\n  override val statName = \"negative_feedback_v2\"\n  override val modelWeightParam = ModelWeights.NegativeFeedbackV2Param\n}\n\nobject PredictedReportedScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_REPORT_TWEET_CLICKED.getFeatureName\n  override val statName = \"reported\"\n  override val modelWeightParam = ModelWeights.ReportParam\n}\n\nobject PredictedStrongNegativeFeedbackScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_STRONG_NEGATIVE_FEEDBACK.getFeatureName\n  override val statName = \"strong_negative_feedback\"\n  override val modelWeightParam = ModelWeights.StrongNegativeFeedbackParam\n}\n\nobject PredictedWeakNegativeFeedbackScoreFeature extends PredictedScoreFeature {\n  override val featureName: String =\n    RecapFeatures.PREDICTED_IS_WEAK_NEGATIVE_FEEDBACK.getFeatureName\n  override val statName = \"weak_negative_feedback\"\n  override val modelWeightParam = ModelWeights.WeakNegativeFeedbackParam\n}\n\nobject PredictedScoreFeature {\n  val PredictedScoreFeatures: Set[PredictedScoreFeature] = Set(\n    PredictedFavoriteScoreFeature,\n    PredictedReplyScoreFeature,\n    PredictedRetweetScoreFeature,\n    PredictedReplyEngagedByAuthorScoreFeature,\n    PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature,\n    PredictedGoodClickConvoDescUamGt2ScoreFeature,\n    PredictedGoodProfileClickScoreFeature,\n    PredictedVideoPlayback50ScoreFeature,\n    PredictedTweetDetailDwellScoreFeature,\n    PredictedProfileDwelledScoreFeature,\n    PredictedBookmarkScoreFeature,\n    PredictedShareScoreFeature,\n    PredictedShareMenuClickScoreFeature,\n    // Negative Engagements\n    PredictedNegativeFeedbackV2ScoreFeature,\n    PredictedReportedScoreFeature,\n    PredictedStrongNegativeFeedbackScoreFeature,\n    PredictedWeakNegativeFeedbackScoreFeature,\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer/RescoringFactorProvider.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scorer\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator.BroadcastStateFeature\nimport com.twitter.home_mixer.functional_component.feature_hydrator.SpaceStateFeature\nimport com.twitter.home_mixer.functional_component.scorer.FeedbackFatigueScorer\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam._\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.util.MtlNormalizer\nimport com.twitter.timelineservice.{thriftscala => tls}\nimport com.twitter.ubs.{thriftscala => ubs}\n\ntrait RescoringFactorProvider {\n\n  def selector(query: PipelineQuery, candidate: CandidateWithFeatures[TweetCandidate]): Boolean\n\n  def factor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Double\n\n  def apply(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate],\n  ): Double = if (selector(query, candidate)) factor(query, candidate) else 1.0\n}\n\ncase class RescoreListwise(listwiseRescoringMap: Map[Long, Double])\n    extends RescoringFactorProvider {\n\n  override def selector(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Boolean = listwiseRescoringMap.contains(candidate.candidate.id)\n\n  override def factor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Double = listwiseRescoringMap(candidate.candidate.id)\n}\n\n/**\n * Re-scoring multiplier to apply to out-of-network tweets\n */\nobject RescoreOutOfNetwork extends RescoringFactorProvider {\n\n  def selector(query: PipelineQuery, candidate: CandidateWithFeatures[TweetCandidate]): Boolean =\n    !candidate.features.getOrElse(InNetworkFeature, false)\n\n  def factor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Double = query.params(OutOfNetworkScaleFactorParam)\n}\n\n/**\n * Re-scoring multiplier to apply to reply candidates\n */\nobject RescoreReplies extends RescoringFactorProvider {\n\n  def selector(query: PipelineQuery, candidate: CandidateWithFeatures[TweetCandidate]): Boolean =\n    candidate.features.getOrElse(InReplyToTweetIdFeature, None).isDefined\n\n  def factor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Double = query.params(ReplyScaleFactorParam)\n}\n\n/**\n * Re-scoring multiplier to calibrate multi-tasks learning model prediction\n */\ncase class RescoreMTLNormalization(mtlNormalizer: MtlNormalizer) extends RescoringFactorProvider {\n\n  def selector(query: PipelineQuery, candidate: CandidateWithFeatures[TweetCandidate]): Boolean =\n    query.params(MtlNormalization.EnableMtlNormalizationParam)\n\n  def factor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Double = mtlNormalizer(\n    attribute = candidate.features.getOrElse(AuthorFollowersFeature, None),\n    retweet = candidate.features.getOrElse(SourceTweetIdFeature, None).isDefined,\n    reply = candidate.features.getOrElse(InReplyToTweetIdFeature, None).isDefined\n  )\n}\n\ncase class RescoreFeedbackFatigue(query: PipelineQuery) extends RescoringFactorProvider {\n\n  def selector(query: PipelineQuery, candidate: CandidateWithFeatures[TweetCandidate]): Boolean =\n    true\n\n  private val feedbackEntriesByEngagementType =\n    query.features\n      .getOrElse(FeatureMap.empty).getOrElse(FeedbackHistoryFeature, Seq.empty)\n      .filter { entry =>\n        val timeSinceFeedback = query.queryTime.minus(entry.timestamp)\n        timeSinceFeedback < FeedbackFatigueScorer.DurationForDiscounting &&\n        entry.feedbackType == tls.FeedbackType.SeeFewer\n      }.groupBy(_.engagementType)\n\n  private val authorsToDiscount =\n    FeedbackFatigueScorer.getUserDiscounts(\n      query.queryTime,\n      feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Tweet, Seq.empty))\n\n  private val likersToDiscount =\n    FeedbackFatigueScorer.getUserDiscounts(\n      query.queryTime,\n      feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Like, Seq.empty))\n\n  private val followersToDiscount =\n    FeedbackFatigueScorer.getUserDiscounts(\n      query.queryTime,\n      feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Follow, Seq.empty))\n\n  private val retweetersToDiscount =\n    FeedbackFatigueScorer.getUserDiscounts(\n      query.queryTime,\n      feedbackEntriesByEngagementType.getOrElse(tls.FeedbackEngagementType.Retweet, Seq.empty))\n\n  def factor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Double = {\n    FeedbackFatigueScorer.getScoreMultiplier(\n      candidate,\n      authorsToDiscount,\n      likersToDiscount,\n      followersToDiscount,\n      retweetersToDiscount\n    )\n  }\n}\n\n/**\n * Disabled in production\n */\nobject RescoreLiveContent extends RescoringFactorProvider {\n\n  private val MinFollowers = 1000000\n\n  def selector(query: PipelineQuery, candidate: CandidateWithFeatures[TweetCandidate]): Boolean = {\n    (\n      candidate.features.getOrElse(SpaceStateFeature, None).contains(ubs.BroadcastState.Running) ||\n      candidate.features.getOrElse(BroadcastStateFeature, None).contains(ubs.BroadcastState.Running)\n    ) &&\n    candidate.features.getOrElse(InNetworkFeature, false) &&\n    candidate.features.getOrElse(AuthorFollowersFeature, None).exists(_ > MinFollowers)\n  }\n\n  def factor(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate]\n  ): Double = query.params(LiveContentScaleFactorParam)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scoring_pipeline/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer-features/thrift/src/main/thrift:thrift-scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/offline_aggregates\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/real_time_aggregates\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/scorer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/earlybird\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/gate\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scorer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/embedding\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/param_gated\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/real_time_aggregates\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/param_gated\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/scoring\",\n        \"servo/repo/src/main/scala\",\n        \"src/scala/com/twitter/timelines/prediction/adapters/large_embeddings\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scoring_pipeline/ScoredTweetsHeuristicScoringPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scoring_pipeline\n\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsStaticCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.gate.DenyLowSignalUserGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam\nimport com.twitter.home_mixer.product.scored_tweets.scorer.HeuristicScorer\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.core.functional_component.common.AllExceptPipelines\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.model.common.identifier.ScoringPipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipelineConfig\nimport com.twitter.timelines.configapi.FSParam\n\nobject ScoredTweetsHeuristicScoringPipelineConfig\n    extends ScoringPipelineConfig[ScoredTweetsQuery, TweetCandidate] {\n\n  override val identifier: ScoringPipelineIdentifier =\n    ScoringPipelineIdentifier(\"ScoredTweetsHeuristic\")\n\n  override val supportedClientParam: Option[FSParam[Boolean]] =\n    Some(ScoredTweetsParam.EnableHeuristicScoringPipeline)\n\n  override val gates: Seq[BaseGate[ScoredTweetsQuery]] = Seq(DenyLowSignalUserGate)\n\n  private val allExcept = AllExceptPipelines(\n    pipelinesToExclude = Set(ScoredTweetsStaticCandidatePipelineConfig.Identifier)\n  )\n\n  override val selectors: Seq[Selector[ScoredTweetsQuery]] = Seq(InsertAppendResults(allExcept))\n\n  override val scorers: Seq[Scorer[ScoredTweetsQuery, TweetCandidate]] =\n    Seq(HeuristicScorer)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scoring_pipeline/ScoredTweetsLowSignalScoringPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scoring_pipeline\n\nimport com.twitter.home_mixer.product.scored_tweets.gate.AllowLowSignalUserGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.scorer.LowSignalScorer\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.model.common.identifier.ScoringPipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipelineConfig\n\nobject ScoredTweetsLowSignalScoringPipelineConfig\n    extends ScoringPipelineConfig[ScoredTweetsQuery, TweetCandidate] {\n\n  override val identifier: ScoringPipelineIdentifier =\n    ScoringPipelineIdentifier(\"ScoredTweetsLowSignal\")\n\n  override val gates: Seq[BaseGate[ScoredTweetsQuery]] = Seq(AllowLowSignalUserGate)\n\n  override val selectors: Seq[Selector[ScoredTweetsQuery]] = Seq(InsertAppendResults(AllPipelines))\n\n  override val scorers: Seq[Scorer[ScoredTweetsQuery, TweetCandidate]] = Seq(LowSignalScorer)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scoring_pipeline/ScoredTweetsModelScoringPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scoring_pipeline\n\nimport com.twitter.home_mixer.functional_component.feature_hydrator._\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.TopicEdgeAggregateFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.TopicEdgeTruncatedAggregateFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.TweetContentEdgeAggregateFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.UserEngagerEdgeAggregateFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.offline_aggregates.UserEntityAggregateFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.MediaClusterIdFeatureHydrator\nimport com.twitter.home_mixer.functional_component.feature_hydrator.real_time_aggregates._\nimport com.twitter.home_mixer.functional_component.scorer.NaviModelScorer\nimport com.twitter.home_mixer.functional_component.scorer.PhoenixScorer\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdScoreFeature\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnablePhoenixScorerParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableGeoduckAuthorLocationHydatorParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableSimclustersSparseTweetFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTransformerPostEmbeddingJointBlueFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTweetLanguageFeaturesParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinRebuildTweetFeaturesOnlineParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableTwhinTweetFeaturesOnlineParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.FeatureHydration.EnableViewCountFeaturesParam\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.CachedScoredTweetsCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsBackfillCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsDirectUtegCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsListsCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsStaticCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsTweetMixerCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.earlybird.ScoredTweetsCommunitiesCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.earlybird.ScoredTweetsEarlybirdInNetworkCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.EarlybirdFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.ReplyFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.SemanticCoreFeatureHydrator\nimport com.twitter.home_mixer.product.scored_tweets.gate.DenyLowSignalUserGate\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableClipImagesClusterIdFeatureHydrationParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableMediaCompletionRateFeatureHydrationParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableMultiModalEmbeddingsFeatureHydratorParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.EnableTweetTextV8EmbeddingFeatureParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.QualityFactor\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.embedding.TweetTextV8EmbeddingFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedBulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.component_library.gate.NonEmptyCandidatesGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.scorer.param_gated.ParamGatedScorer\nimport com.twitter.product_mixer.component_library.selector.DropMaxCandidates\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.component_library.selector.UpdateSortCandidates\nimport com.twitter.product_mixer.core.functional_component.common.AllExceptPipelines\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ScoringPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.UnexpectedCandidateResult\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipelineConfig\nimport com.twitter.timelines.configapi.Param\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoredTweetsModelScoringPipelineConfig @Inject() (\n  // Candidate sources\n  scoredTweetsTweetMixerCandidatePipelineConfig: ScoredTweetsTweetMixerCandidatePipelineConfig,\n  scoredTweetsListsCandidatePipelineConfig: ScoredTweetsListsCandidatePipelineConfig,\n  scoredTweetsBackfillCandidatePipelineConfig: ScoredTweetsBackfillCandidatePipelineConfig,\n  scoredTweetsEarlybirdInNetworkCandidatePipelineConfig: ScoredTweetsEarlybirdInNetworkCandidatePipelineConfig,\n  scoredTweetsCommunitiesCandidatePipelineConfig: ScoredTweetsCommunitiesCandidatePipelineConfig,\n  scoredTweetsDirectUtegCandidatePipelineConfig: ScoredTweetsDirectUtegCandidatePipelineConfig,\n  // Feature hydrators\n  ancestorFeatureHydrator: AncestorFeatureHydrator,\n  authorFeatureHydrator: AuthorFeatureHydrator,\n  broadcastStateFeatureHydrator: BroadcastStateFeatureHydrator,\n  earlybirdFeatureHydrator: EarlybirdFeatureHydrator,\n  gizmoduckAuthorFeatureHydrator: GizmoduckAuthorFeatureHydrator,\n  geoduckAuthorLocationHydrator: GeoduckAuthorLocationHydrator,\n  graphTwoHopFeatureHydrator: GraphTwoHopFeatureHydrator,\n  mediaClusterIdFeatureHydrator: MediaClusterIdFeatureHydrator,\n  mediaCompletionRateFeatureHydrator: MediaCompletionRateFeatureHydrator,\n  clipImagesClusterIdFeatureHydrator: ClipImageClusterIdFeatureHydrator,\n  viralContentCreatorMetricsFeatureHydrator: ViralContentCreatorMetricsFeatureHydrator,\n  multiModalEmbeddingsFeatureHydrator: MultiModalEmbeddingsFeatureHydrator,\n  realGraphViewerAuthorFeatureHydrator: RealGraphViewerAuthorFeatureHydrator,\n  realGraphViewerRelatedUsersFeatureHydrator: RealGraphViewerRelatedUsersFeatureHydrator,\n  realTimeInteractionGraphEdgeFeatureHydrator: RealTimeInteractionGraphEdgeFeatureHydrator,\n  replyFeatureHydrator: ReplyFeatureHydrator,\n  sgsValidSocialContextFeatureHydrator: SGSValidSocialContextFeatureHydrator,\n  simClustersEngagementSimilarityFeatureHydrator: SimClustersEngagementSimilarityFeatureHydrator,\n  simClustersUserTweetScoresHydrator: SimClustersUserTweetScoresHydrator,\n  spaceStateFeatureHydrator: SpaceStateFeatureHydrator,\n  tspInferredTopicFeatureHydrator: TSPInferredTopicFeatureHydrator,\n  tweetEntityServiceContentFeatureHydrator: TweetEntityServiceContentFeatureHydrator,\n  tweetTextV8EmbeddingFeatureHydrator: TweetTextV8EmbeddingFeatureHydrator,\n  twhinAuthorFollowFeatureHydrator: TwhinAuthorFollowFeatureHydrator,\n  twhinTweetFeatureHydrator: TwhinTweetFeatureHydrator,\n  twhinRebuildTweetFeatureHydrator: TwhinRebuildTweetFeatureHydrator,\n  utegFeatureHydrator: UtegFeatureHydrator,\n  slopAuthorFeatureHydrator: SlopAuthorFeatureHydrator,\n  grokAnnotationsFeatureHydrator: GrokAnnotationsFeatureHydrator,\n  // Real time aggregate feature hydrators\n  engagementsReceivedByAuthorRealTimeAggregateFeatureHydrator: EngagementsReceivedByAuthorRealTimeAggregateFeatureHydrator,\n  topicCountryEngagementRealTimeAggregateFeatureHydrator: TopicCountryEngagementRealTimeAggregateFeatureHydrator,\n  topicEngagementRealTimeAggregateFeatureHydrator: TopicEngagementRealTimeAggregateFeatureHydrator,\n  tweetCountryEngagementRealTimeAggregateFeatureHydrator: TweetCountryEngagementRealTimeAggregateFeatureHydrator,\n  tweetEngagementRealTimeAggregateFeatureHydrator: TweetEngagementRealTimeAggregateFeatureHydrator,\n  tweetLanguageFeatureHydrator: TweetLanguageFeatureHydrator,\n  twitterListEngagementRealTimeAggregateFeatureHydrator: TwitterListEngagementRealTimeAggregateFeatureHydrator,\n  userAuthorEngagementRealTimeAggregateFeatureHydrator: UserAuthorEngagementRealTimeAggregateFeatureHydrator,\n  viewCountsFeatureHydrator: ViewCountsFeatureHydrator,\n  // Large embeddings hydrators\n  authorLargeEmbeddingsFeatureHydrator: AuthorLargeEmbeddingsFeatureHydrator,\n  originalAuthorLargeEmbeddingsFeatureHydrator: OriginalAuthorLargeEmbeddingsFeatureHydrator,\n  tweetLargeEmbeddingsFeatureHydrator: TweetLargeEmbeddingsFeatureHydrator,\n  originalTweetLargeEmbeddingsFeatureHydrator: OriginalTweetLargeEmbeddingsFeatureHydrator,\n  // Transformer embeddings hydrators\n  transformerPostEmbeddingBlueFeatureHydrator: TransformerPostEmbeddingHomeBlueFeatureHydrator,\n  transformerPostEmbeddingGreenFeatureHydrator: TransformerPostEmbeddingHomeGreenFeatureHydrator,\n  transformerPostEmbeddingJointBlueFeatureHydrator: TransformerPostEmbeddingJointBlueFeatureHydrator,\n  simClustersLogFavBasedTweetFeatureHydrator: SimClustersLogFavBasedTweetFeatureHydrator,\n  // Scorers\n  naviModelScorer: NaviModelScorer,\n  phoenixScorer: PhoenixScorer)\n    extends ScoringPipelineConfig[ScoredTweetsQuery, TweetCandidate] {\n\n  override val identifier: ScoringPipelineIdentifier =\n    ScoringPipelineIdentifier(\"ScoredTweetsModel\")\n\n  private val nonCachedScoringPipelineScope = AllExceptPipelines(\n    pipelinesToExclude = Set(\n      CachedScoredTweetsCandidatePipelineConfig.Identifier,\n      ScoredTweetsStaticCandidatePipelineConfig.Identifier\n    )\n  )\n\n  override val gates: Seq[BaseGate[ScoredTweetsQuery]] = Seq(\n    DenyLowSignalUserGate,\n    NonEmptyCandidatesGate(nonCachedScoringPipelineScope)\n  )\n\n  private val earlybirdScorePipelineScope = Set(\n    scoredTweetsEarlybirdInNetworkCandidatePipelineConfig.identifier,\n    scoredTweetsDirectUtegCandidatePipelineConfig.identifier\n  )\n\n  private val earlybirdScoreOrdering: Ordering[CandidateWithDetails] =\n    Ordering.by[CandidateWithDetails, Double] {\n      case ItemCandidateWithDetails(_, _, features) =>\n        -features.getOrElse(EarlybirdScoreFeature, None).getOrElse(0.0)\n      case _ => throw PipelineFailure(UnexpectedCandidateResult, \"Invalid candidate type\")\n    }\n\n  private def qualityFactorDropMaxCandidates(\n    pipelineIdentifier: CandidatePipelineIdentifier,\n    qualityFactorParam: Param[Int]\n  ): DropMaxCandidates[ScoredTweetsQuery] = {\n    new DropMaxCandidates(\n      pipelineScope = SpecificPipelines(pipelineIdentifier),\n      maxSelector = (query, _, _) =>\n        (query.getQualityFactorCurrentValue(identifier) * query.params(qualityFactorParam)).toInt\n    )\n  }\n\n  override val selectors: Seq[Selector[ScoredTweetsQuery]] = Seq(\n    UpdateSortCandidates(SpecificPipelines(earlybirdScorePipelineScope), earlybirdScoreOrdering),\n    UpdateSortCandidates(\n      SpecificPipeline(scoredTweetsBackfillCandidatePipelineConfig.identifier),\n      CandidatesUtil.reverseChronTweetsOrdering\n    ),\n    qualityFactorDropMaxCandidates(\n      scoredTweetsTweetMixerCandidatePipelineConfig.identifier,\n      QualityFactor.TweetMixerMaxTweetsToScoreParam\n    ),\n    qualityFactorDropMaxCandidates(\n      scoredTweetsListsCandidatePipelineConfig.identifier,\n      QualityFactor.ListsMaxTweetsToScoreParam\n    ),\n    qualityFactorDropMaxCandidates(\n      scoredTweetsBackfillCandidatePipelineConfig.identifier,\n      QualityFactor.BackfillMaxTweetsToScoreParam\n    ),\n    qualityFactorDropMaxCandidates(\n      scoredTweetsEarlybirdInNetworkCandidatePipelineConfig.identifier,\n      QualityFactor.InNetworkMaxTweetsToScoreParam\n    ),\n    qualityFactorDropMaxCandidates(\n      scoredTweetsCommunitiesCandidatePipelineConfig.identifier,\n      QualityFactor.CommunitiesMaxTweetsToScoreParam\n    ),\n    qualityFactorDropMaxCandidates(\n      scoredTweetsDirectUtegCandidatePipelineConfig.identifier,\n      QualityFactor.UtegMaxTweetsToScoreParam\n    ),\n    // Select candidates for Heavy Ranker Feature Hydration and Scoring\n    InsertAppendResults(nonCachedScoringPipelineScope)\n  )\n\n  override val preScoringFeatureHydrationPhase1: Seq[\n    BaseCandidateFeatureHydrator[ScoredTweetsQuery, TweetCandidate, _]\n  ] = Seq(\n    DependentBulkCandidateFeatureHydrator(ancestorFeatureHydrator, Seq(replyFeatureHydrator)),\n    authorFeatureHydrator,\n    DependentBulkCandidateFeatureHydrator(\n      earlybirdFeatureHydrator,\n      Seq(spaceStateFeatureHydrator, broadcastStateFeatureHydrator, TweetTimeFeatureHydrator)),\n    gizmoduckAuthorFeatureHydrator,\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnableGeoduckAuthorLocationHydatorParam,\n      geoduckAuthorLocationHydrator,\n    ),\n    graphTwoHopFeatureHydrator,\n    InNetworkFeatureHydrator,\n    realGraphViewerAuthorFeatureHydrator,\n    realTimeInteractionGraphEdgeFeatureHydrator,\n    simClustersEngagementSimilarityFeatureHydrator,\n    simClustersUserTweetScoresHydrator,\n    DependentBulkCandidateFeatureHydrator(\n      tspInferredTopicFeatureHydrator,\n      Seq(\n        TopicEdgeAggregateFeatureHydrator,\n        TopicEdgeTruncatedAggregateFeatureHydrator,\n        topicCountryEngagementRealTimeAggregateFeatureHydrator,\n        topicEngagementRealTimeAggregateFeatureHydrator\n      )\n    ),\n    TweetMetaDataFeatureHydrator,\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnableTweetTextV8EmbeddingFeatureParam,\n      tweetTextV8EmbeddingFeatureHydrator\n    ),\n    DependentBulkCandidateFeatureHydrator(\n      tweetEntityServiceContentFeatureHydrator,\n      Seq(\n        SemanticCoreFeatureHydrator,\n        TweetContentEdgeAggregateFeatureHydrator,\n        mediaClusterIdFeatureHydrator)),\n    twhinAuthorFollowFeatureHydrator,\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnableTwhinTweetFeaturesOnlineParam,\n      twhinTweetFeatureHydrator\n    ),\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnableTwhinRebuildTweetFeaturesOnlineParam,\n      twhinRebuildTweetFeatureHydrator\n    ),\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnableViewCountFeaturesParam,\n      viewCountsFeatureHydrator\n    ),\n    DependentBulkCandidateFeatureHydrator(\n      utegFeatureHydrator,\n      Seq(\n        realGraphViewerRelatedUsersFeatureHydrator,\n        sgsValidSocialContextFeatureHydrator,\n        UserEngagerEdgeAggregateFeatureHydrator\n      )\n    ),\n    slopAuthorFeatureHydrator,\n    // Real time aggregates\n    engagementsReceivedByAuthorRealTimeAggregateFeatureHydrator,\n    tweetCountryEngagementRealTimeAggregateFeatureHydrator,\n    tweetEngagementRealTimeAggregateFeatureHydrator,\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnableTweetLanguageFeaturesParam,\n      tweetLanguageFeatureHydrator\n    ),\n    twitterListEngagementRealTimeAggregateFeatureHydrator,\n    userAuthorEngagementRealTimeAggregateFeatureHydrator,\n    // Offline aggregates\n    UserEntityAggregateFeatureHydrator,\n    viralContentCreatorMetricsFeatureHydrator,\n    GrokGorkContentCreatorFeatureHydrator,\n    // Large Embeddings\n    authorLargeEmbeddingsFeatureHydrator,\n    originalAuthorLargeEmbeddingsFeatureHydrator,\n    tweetLargeEmbeddingsFeatureHydrator,\n    originalTweetLargeEmbeddingsFeatureHydrator,\n    // Transformers\n    transformerPostEmbeddingBlueFeatureHydrator,\n    transformerPostEmbeddingGreenFeatureHydrator,\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnableTransformerPostEmbeddingJointBlueFeaturesParam,\n      transformerPostEmbeddingJointBlueFeatureHydrator\n    ),\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnableSimclustersSparseTweetFeaturesParam,\n      simClustersLogFavBasedTweetFeatureHydrator\n    ),\n    grokAnnotationsFeatureHydrator,\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnableMediaCompletionRateFeatureHydrationParam,\n      mediaCompletionRateFeatureHydrator,\n    ),\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnableClipImagesClusterIdFeatureHydrationParam,\n      clipImagesClusterIdFeatureHydrator\n    ),\n    ParamGatedBulkCandidateFeatureHydrator(\n      EnableMultiModalEmbeddingsFeatureHydratorParam,\n      multiModalEmbeddingsFeatureHydrator\n    )\n  )\n\n  override val scorers: Seq[Scorer[ScoredTweetsQuery, TweetCandidate]] = Seq(\n    naviModelScorer,\n    ParamGatedScorer(EnablePhoenixScorerParam, phoenixScorer)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/scoring_pipeline/ScoredTweetsRerankingScoringPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.scoring_pipeline\n\nimport com.twitter.home_mixer.functional_component.scorer.PhoenixModelRerankingScorer\nimport com.twitter.home_mixer.functional_component.scorer.WeighedModelRerankingScorer\nimport com.twitter.home_mixer.param.HomeGlobalParams.EnablePhoenixScorerParam\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.scorer.param_gated.ParamGatedScorer\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.model.common.identifier.ScoringPipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipelineConfig\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoredTweetsRerankingScoringPipelineConfig @Inject() (\n  // rerankers\n  weighedModelRerankingScorer: WeighedModelRerankingScorer,\n  phoenixModelRerankingScorer: PhoenixModelRerankingScorer,\n  // Base scoring pipeline config\n  scoredTweetsModelScoringPipelineConfig: ScoredTweetsModelScoringPipelineConfig)\n    extends ScoringPipelineConfig[ScoredTweetsQuery, TweetCandidate] {\n\n  override val identifier: ScoringPipelineIdentifier =\n    ScoringPipelineIdentifier(\"ScoredTweetsReranking\")\n\n  override val gates: Seq[BaseGate[ScoredTweetsQuery]] =\n    scoredTweetsModelScoringPipelineConfig.gates\n\n  override val selectors: Seq[Selector[ScoredTweetsQuery]] =\n    scoredTweetsModelScoringPipelineConfig.selectors\n\n  override val scorers: Seq[Scorer[ScoredTweetsQuery, TweetCandidate]] =\n    Seq(\n      weighedModelRerankingScorer,\n      ParamGatedScorer(EnablePhoenixScorerParam, phoenixModelRerankingScorer)\n    )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/selector/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/communities\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/selector/KeepBestOutOfNetworkCandidatePerAuthorPerSuggestType.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.selector\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SuggestTypeFeature\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class KeepBestOutOfNetworkCandidatePerAuthorPerSuggestType(\n  override val pipelineScope: CandidateScope)\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val (selectedCandidates, otherCandidates) =\n      remainingCandidates.partition(candidate =>\n        pipelineScope.contains(candidate) && !candidate.features.getOrElse(InNetworkFeature, true))\n\n    val filteredCandidates = selectedCandidates\n      .groupBy { candidate =>\n        (\n          candidate.features.getOrElse(AuthorIdFeature, None),\n          candidate.features.getOrElse(SuggestTypeFeature, None)\n        )\n      }\n      .values.map(_.maxBy(_.features.getOrElse(ScoreFeature, None)))\n      .toSeq\n\n    val updatedCandidates = otherCandidates ++ filteredCandidates\n    SelectorResult(remainingCandidates = updatedCandidates, result = result)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/selector/KeepTopKCandidatesPerCommunity.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.selector\n\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityIdFeature\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class KeepTopKCandidatesPerCommunity(override val pipelineScope: CandidateScope)\n    extends Selector[PipelineQuery] {\n\n  private val MaxCandidatesPerCommunity = 1\n  private val MaxCommunityCandidates = 3\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val (selectedCandidates, otherCandidates) = remainingCandidates.partition { candidate =>\n      pipelineScope.contains(candidate) &&\n      candidate.features.getOrElse(CommunityIdFeature, None).isDefined\n    }\n\n    val filteredCandidates = selectedCandidates\n      .groupBy { candidate => candidate.features.getOrElse(CommunityIdFeature, None) }\n      .values.flatMap {\n        _.sortBy(_.features.getOrElse(ScoreFeature, None)).reverse.take(MaxCandidatesPerCommunity)\n      }\n      .toSeq.sortBy(_.features.getOrElse(ScoreFeature, None)).reverse.take(MaxCommunityCandidates)\n\n    val updatedCandidates = otherCandidates ++ filteredCandidates\n    SelectorResult(remainingCandidates = updatedCandidates, result = result)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-mysql/src/main/scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/adapters/simclusters_features\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/scorer\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/candidate_source\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/candidate_pipeline/earlybird\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"kafka/finagle-kafka/finatra-kafka/src/main/scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweet_mixer\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/communities\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/location\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/impressed_tweets\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt\",\n        \"servo/repo/src/main/scala\",\n        \"src/scala/com/twitter/timelines/prediction/common/adapters:base\",\n        \"src/scala/com/twitter/timelines/prediction/features/common\",\n        \"src/thrift/com/twitter/timelines/served_candidates_logging:served_candidates_logging-scala\",\n        \"src/thrift/com/twitter/timelines/suggests/common:data_record_metadata-scala\",\n        \"src/thrift/com/twitter/timelines/suggests/common:poly_data_record-java\",\n        \"src/thrift/com/twitter/timelines/timeline_logging:thrift-scala\",\n        \"strato/config/columns/home-mixer:home-mixer-strato-client\",\n        \"strato/config/columns/videoRecommendations/twitterClip:twitterClip-strato-client\",\n        \"strato/config/src/thrift/com/twitter/strato/columns/content_understanding:content_understanding-scala\",\n        \"timelines/data_processing/jobs/light_ranking/light_ranking_features_prep:recap_partial_features_for_two_tower_models\",\n        \"timelines/ml:kafka\",\n        \"timelines/ml:pldr-client\",\n        \"timelines/ml:pldr-conversion\",\n        \"timelines/ml/cont_train/common/domain/src/main/scala/com/twitter/timelines/ml/cont_train/common/domain/non_scalding\",\n        \"user_history_transformer/service/src/main/java/com/x/user_action_sequence\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/CacheCandidateFeaturesSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.side_effect\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.side_effect.BaseCacheCandidateFeaturesSideEffect\nimport com.twitter.home_mixer.param.HomeMixerFlagName.DataRecordMetadataStoreConfigsYmlFlag\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames._\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.simclusters_v2.thriftscala.TwhinTweetEmbedding\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus.Store\nimport com.twitter.strato.generated.client.videoRecommendations.twitterClip.TwitterClipEmbeddingMhClientColumn\nimport com.twitter.timelines.served_candidates_logging.{thriftscala => sc}\nimport com.twitter.timelines.suggests.common.poly_data_record.{thriftjava => pdr}\nimport com.twitter.twistly.thriftscala.VideoViewEngagementType\nimport com.twitter.twistly.thriftscala.WatchTimeMetadata\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass CacheCandidateFeaturesSideEffect @Inject() (\n  @Flag(DataRecordMetadataStoreConfigsYmlFlag) dataRecordMetadataStoreConfigsYml: String,\n  @Named(MemcacheCandidateFeaturesStore) store: Store[\n    sc.CandidateFeatureKey,\n    pdr.PolyDataRecord\n  ],\n  @Named(TweetWatchTimeMetadataStore) tweetWatchTimeMetadataStore: ReadableStore[\n    (Long, VideoViewEngagementType),\n    WatchTimeMetadata\n  ],\n  twitterClipEmbeddingMhClientColumn: TwitterClipEmbeddingMhClientColumn,\n  @Named(TwhinVideoEmbeddingsStore) twhinVideoStore: ReadableStore[Long, TwhinTweetEmbedding],\n  statsReceiver: StatsReceiver)\n    extends BaseCacheCandidateFeaturesSideEffect(\n      dataRecordMetadataStoreConfigsYml,\n      store,\n      tweetWatchTimeMetadataStore,\n      twitterClipEmbeddingMhClientColumn,\n      twhinVideoStore,\n      statsReceiver) {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"CacheCandidateFeatures\")\n\n  override val statScope: String = getClass.getSimpleName\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/CacheRequestInfoSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.side_effect\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.home_mixer.model.HomeFeatures.IsReadFromCacheFeature\nimport com.twitter.home_mixer.model.PredictedFavoriteScoreFeature\nimport com.twitter.home_mixer.model.PredictedReplyScoreFeature\nimport com.twitter.home_mixer.model.PredictedRetweetScoreFeature\nimport com.twitter.home_mixer.model.PredictedShareScoreFeature\nimport com.twitter.home_mixer.model.PredictedVideoQualityViewScoreFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithLongTimeout\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableCacheRequestInfoParam\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.generated.client.home_mixer.StoreRequestInfoClientColumn\nimport com.twitter.strato.generated.client.home_mixer.StoreRequestInfoClientColumn.FavAndRetweetAndReplyAndShareAndVqv\nimport com.twitter.strato.generated.client.home_mixer.StoreRequestInfoClientColumn.PostIdAndHeavyRankerScores\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass CacheRequestInfoSideEffect @Inject() (\n  @Named(BatchedStratoClientWithLongTimeout) stratoClient: Client,\n  serviceIdentifier: ServiceIdentifier)\n    extends PipelineResultSideEffect[PipelineQuery, HasMarshalling]\n    with Conditionally[PipelineQuery, HasMarshalling] {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"CacheRequestInfo\")\n\n  private val isProdEnv = serviceIdentifier.environment == \"prod\"\n\n  private val retrievalSignalExecutor = new StoreRequestInfoClientColumn(stratoClient).executer\n\n  override def onlyIf(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Boolean = query.params(EnableCacheRequestInfoParam) && isProdEnv\n\n  override def apply(\n    inputs: PipelineResultSideEffect.Inputs[PipelineQuery, HasMarshalling]\n  ): Stitch[Unit] = {\n    val posts = inputs.selectedCandidates.collect {\n      case candidate if !candidate.features.getOrElse(IsReadFromCacheFeature, false) =>\n        PostIdAndHeavyRankerScores(\n          postId = candidate.candidateIdLong,\n          heavyRankerScores = Some(\n            FavAndRetweetAndReplyAndShareAndVqv(\n              fav = candidate.features.getOrElse(PredictedFavoriteScoreFeature, None),\n              retweet = candidate.features.getOrElse(PredictedRetweetScoreFeature, None),\n              reply = candidate.features.getOrElse(PredictedReplyScoreFeature, None),\n              share = candidate.features.getOrElse(PredictedShareScoreFeature, None),\n              vqv = candidate.features.getOrElse(PredictedVideoQualityViewScoreFeature, None)\n            )\n          )\n        )\n    }\n    val arg = StoreRequestInfoClientColumn.Arg(\n      userId = inputs.query.getRequiredUserId,\n      posts = posts,\n      requestTimestampMs = inputs.query.queryTime.inMilliseconds\n    )\n    retrievalSignalExecutor.execute(arg)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/CacheRetrievalSignalSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.side_effect\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SGSValidLikedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceSignalFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.BatchedStratoClientWithLongTimeout\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableCacheRetrievalSignalParam\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.client.Putter\nimport com.twitter.strato.generated.client.home_mixer.RetrievalSignalv2ClientColumn\nimport com.twitter.usersignalservice.{thriftscala => se}\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass CacheRetrievalSignalSideEffect @Inject() (\n  @Named(BatchedStratoClientWithLongTimeout) stratoClient: Client,\n  serviceIdentifier: ServiceIdentifier,\n  statsReceiver: StatsReceiver)\n    extends PipelineResultSideEffect[PipelineQuery, HasMarshalling]\n    with Conditionally[PipelineQuery, HasMarshalling] {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"CacheRetrievalSignal\")\n\n  private val scopedStats = statsReceiver.scope(getClass.getSimpleName)\n\n  private val isProdEnv = serviceIdentifier.environment == \"prod\"\n\n  private val retrievalSignalPutter: Putter[(Long, Long), hmt.RetrievalSignal] =\n    new RetrievalSignalv2ClientColumn(stratoClient).putter\n\n  override def onlyIf(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Boolean = query.params(EnableCacheRetrievalSignalParam) && isProdEnv\n\n  override def apply(\n    inputs: PipelineResultSideEffect.Inputs[PipelineQuery, HasMarshalling]\n  ): Stitch[Unit] = {\n    scopedStats.counter(\"side_effect_applied\").incr(1)\n\n    val sourceSignalWrites = inputs.selectedCandidates.map { candidate =>\n      scopedStats.counter(\"total_candidates_processed\").incr(1)\n\n      val userId = inputs.query.getRequiredUserId\n      val signalFeature = candidate.features.getOrElse(SourceSignalFeature, None)\n      val servedType = candidate.features.get(ServedTypeFeature)\n\n      val retrievalSignalOpt = (signalFeature, servedType) match {\n        case (Some(signal), _) if signal.id > 0L && signal.signalEntity.isDefined =>\n          scopedStats.counter(\"valid_existing_signal_feature\").incr(1)\n          signal.signalEntity.flatMap { entity =>\n            if (entity != se.SignalEntity.User || signal.id != userId) {\n              Some(\n                hmt.RetrievalSignal(\n                  signalId = signal.id,\n                  signalEntity = entity,\n                  authorId = signal.authorId.filter(_ > 0L)\n                ))\n            } else None\n          }\n\n        case (None, hmt.ServedType.ForYouInNetwork) =>\n          scopedStats.counter(\"valid_in_network_signal_feature\").incr(1)\n          candidate.features.get(AuthorIdFeature).map { authorId =>\n            hmt.RetrievalSignal(\n              signalId = authorId,\n              signalEntity = se.SignalEntity.User,\n              authorId = None\n            )\n          }\n\n        case (None, hmt.ServedType.ForYouUteg) =>\n          scopedStats.counter(\"valid_uteg_liked_by_signal_feature\").incr(1)\n          val validLikedByUserIds =\n            candidate.features.getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty)\n          validLikedByUserIds.headOption.map { lastLikedByUserId =>\n            hmt.RetrievalSignal(\n              signalId = lastLikedByUserId,\n              signalEntity = se.SignalEntity.User,\n              authorId = None\n            )\n          }\n\n        case _ =>\n          scopedStats.counter(\"missing_or_invalid_signal_feature\").incr(1)\n          None\n      }\n      retrievalSignalOpt\n        .map { retrievalSignal =>\n          val sourceTweetId: Long =\n            candidate.features\n              .getOrElse(SourceTweetIdFeature, None).getOrElse(candidate.candidateIdLong)\n          retrievalSignalPutter.put((userId, sourceTweetId), retrievalSignal)\n        }.getOrElse(Stitch.Unit)\n    }.toSeq\n\n    Stitch.collect(sourceSignalWrites).map(_ => ())\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/CachedScoredTweetsSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.side_effect\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.transport.Transport\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.model.PredictedFavoriteScoreFeature\nimport com.twitter.home_mixer.model.PredictedReplyScoreFeature\nimport com.twitter.home_mixer.model.PredictedRetweetScoreFeature\nimport com.twitter.home_mixer.model.PredictedShareScoreFeature\nimport com.twitter.home_mixer.model.PredictedVideoQualityViewScoreFeature\nimport com.twitter.home_mixer.model.PredictedDwellScoreFeature\nimport com.twitter.home_mixer.model.PredictedNegativeFeedbackV2ScoreFeature\nimport com.twitter.home_mixer.model.PredictedGoodClickConvoDescUamGt2ScoreFeature\nimport com.twitter.home_mixer.model.PredictedGoodProfileClickScoreFeature\nimport com.twitter.home_mixer.model.PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature\nimport com.twitter.home_mixer.model.PredictedReplyEngagedByAuthorScoreFeature\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.ScoredTweetsCache\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsResponse\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.CachedScoredTweets\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityIdFeature\nimport com.twitter.product_mixer.component_library.feature.communities.CommunitiesSharedFeatures.CommunityNameFeature\nimport com.twitter.product_mixer.component_library.feature.location.LocationSharedFeatures.LocationIdFeature\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.TopicContextFunctionalityTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.cache.TtlCache\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass CachedScoredTweetsSideEffect @Inject() (\n  @Named(ScoredTweetsCache)\n  scoredTweetsCache: TtlCache[Long, hmt.ScoredTweetsResponse])\n    extends PipelineResultSideEffect[PipelineQuery, ScoredTweetsResponse]\n    with Conditionally[PipelineQuery, ScoredTweetsResponse] {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"CachedScoredTweets\")\n\n  private val MaxTweetsToCache = 1000\n\n  override def onlyIf(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: ScoredTweetsResponse\n  ): Boolean = {\n    val serviceIdentifier = ServiceIdentifier.fromCertificate(Transport.peerCertificate)\n    serviceIdentifier.role != \"explore-mixer\" && serviceIdentifier.role != \"video-mixer\"\n  }\n\n  def buildCachedScoredTweets(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithDetails]\n  ): hmt.ScoredTweetsResponse = {\n    val tweets = candidates.map { candidate =>\n      val sgsValidLikedByUserIds =\n        candidate.features.getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty)\n      val validLikedByUserIds = candidate.features.getOrElse(ValidLikedByUserIdsFeature, Seq.empty)\n      val sgsValidFollowedByUserIds =\n        candidate.features.getOrElse(SGSValidFollowedByUserIdsFeature, Seq.empty)\n      val ancestors = candidate.features.getOrElse(AncestorsFeature, Seq.empty)\n      val mediaIds = candidate.features.getOrElse(TweetMediaIdsFeature, Seq.empty)\n      val sourceSignalOpt = candidate.features.getOrElse(SourceSignalFeature, None)\n      val sourceSignal = sourceSignalOpt.map { signal =>\n        hmt.SourceSignal(\n          id = signal.id,\n          signalType = signal.signalType,\n          signalEntity = signal.signalEntity,\n          authorId = signal.authorId,\n        )\n      }\n\n      val predictedScores = hmt.PredictedScores(\n        favoriteScore = candidate.features.getOrElse(PredictedFavoriteScoreFeature, None),\n        replyScore = candidate.features.getOrElse(PredictedReplyScoreFeature, None),\n        retweetScore = candidate.features.getOrElse(PredictedRetweetScoreFeature, None),\n        replyEngagedByAuthorScore =\n          candidate.features.getOrElse(PredictedReplyEngagedByAuthorScoreFeature, None),\n        goodClickConvoDescFavoritedOrRepliedScore = candidate.features\n          .getOrElse(PredictedGoodClickConvoDescFavoritedOrRepliedScoreFeature, None),\n        goodClickConvoDescUamGt2Score =\n          candidate.features.getOrElse(PredictedGoodClickConvoDescUamGt2ScoreFeature, None),\n        goodProfileClickScore =\n          candidate.features.getOrElse(PredictedGoodProfileClickScoreFeature, None),\n        videoQualityViewScore =\n          candidate.features.getOrElse(PredictedVideoQualityViewScoreFeature, None),\n        shareScore = candidate.features.getOrElse(PredictedShareScoreFeature, None),\n        dwellScore = candidate.features.getOrElse(PredictedDwellScoreFeature, None),\n        negativeFeedbackV2Score =\n          candidate.features.getOrElse(PredictedNegativeFeedbackV2ScoreFeature, None)\n      )\n\n      hmt.ScoredTweet(\n        tweetId = candidate.candidateIdLong,\n        authorId = candidate.features.get(AuthorIdFeature).get,\n        // Cache the model score instead of the final score because rescoring is per-request\n        score = candidate.features.getOrElse(WeightedModelScoreFeature, None),\n        servedType = candidate.features.get(ServedTypeFeature),\n        sourceTweetId = candidate.features.getOrElse(SourceTweetIdFeature, None),\n        sourceUserId = candidate.features.getOrElse(SourceUserIdFeature, None),\n        quotedTweetId = candidate.features.getOrElse(QuotedTweetIdFeature, None),\n        quotedUserId = candidate.features.getOrElse(QuotedUserIdFeature, None),\n        inReplyToTweetId = candidate.features.getOrElse(InReplyToTweetIdFeature, None),\n        inReplyToUserId = candidate.features.getOrElse(InReplyToUserIdFeature, None),\n        directedAtUserId = candidate.features.getOrElse(DirectedAtUserIdFeature, None),\n        inNetwork = Some(candidate.features.getOrElse(InNetworkFeature, true)),\n        sgsValidLikedByUserIds = Some(sgsValidLikedByUserIds),\n        validLikedByUserIds = Some(validLikedByUserIds),\n        sgsValidFollowedByUserIds = Some(sgsValidFollowedByUserIds),\n        topicId = candidate.features.getOrElse(TopicIdSocialContextFeature, None),\n        topicFunctionalityType = candidate.features\n          .getOrElse(TopicContextFunctionalityTypeFeature, None).map(\n            TopicContextFunctionalityTypeMarshaller(_)),\n        ancestors = if (ancestors.nonEmpty) Some(ancestors) else None,\n        isReadFromCache = Some(true),\n        exclusiveConversationAuthorId = candidate.features\n          .getOrElse(ExclusiveConversationAuthorIdFeature, None),\n        authorMetadata = Some(\n          hmt.AuthorMetadata(\n            blueVerified = candidate.features.getOrElse(AuthorIsBlueVerifiedFeature, false),\n            goldVerified = candidate.features.getOrElse(AuthorIsGoldVerifiedFeature, false),\n            grayVerified = candidate.features.getOrElse(AuthorIsGrayVerifiedFeature, false),\n            legacyVerified = candidate.features.getOrElse(AuthorIsLegacyVerifiedFeature, false),\n            creator = candidate.features.getOrElse(AuthorIsCreatorFeature, false),\n            followers = candidate.features.getOrElse(AuthorFollowersFeature, None)\n          )),\n        lastScoredTimestampMs = candidate.features\n          .getOrElse(LastScoredTimestampMsFeature, Some(query.queryTime.inMilliseconds)),\n        candidatePipelineIdentifier = candidate.features\n          .getOrElse(CachedCandidatePipelineIdentifierFeature, Some(candidate.source.name)),\n        tweetUrls = Some(candidate.features.getOrElse(TweetUrlsFeature, Seq.empty)),\n        perspectiveFilteredLikedByUserIds = None,\n        predictionRequestId = candidate.features.getOrElse(PredictionRequestIdFeature, None),\n        communityId = candidate.features.getOrElse(CommunityIdFeature, None),\n        communityName = candidate.features.getOrElse(CommunityNameFeature, None),\n        listId = candidate.features.getOrElse(ListIdFeature, None),\n        listName = candidate.features.getOrElse(ListNameFeature, None),\n        tweetTypeMetrics = candidate.features.getOrElse(TweetTypeMetricsFeature, None),\n        debugString = candidate.features.getOrElse(DebugStringFeature, None),\n        viralContentCreator = Some(candidate.features.getOrElse(ViralContentCreatorFeature, false)),\n        locationId = candidate.features.getOrElse(LocationIdFeature, None),\n        isArticle = Some(candidate.features.getOrElse(IsArticleFeature, false)),\n        hasVideo = Some(candidate.features.getOrElse(HasVideoFeature, false)),\n        videoDurationMs = candidate.features.getOrElse(VideoDurationMsFeature, None),\n        mediaIds = if (mediaIds.nonEmpty) Some(mediaIds) else None,\n        grokAnnotations = candidate.features.getOrElse(GrokAnnotationsFeature, None),\n        predictedScores = Some(predictedScores),\n        tweetMixerScore = candidate.features.getOrElse(TweetMixerScoreFeature, None),\n        clipClusterIdsFeature = Some(\n          hmt.ClipClusterIdsFeature(\n            tweetMediaClusterIdsFeature = Some(\n              candidate.features.getOrElse(TweetMediaClusterIdsFeature, Map.empty[Long, Long])),\n            clipImageClusterIdsFeature =\n              Some(candidate.features.getOrElse(ClipImageClusterIdsFeature, Map.empty[Long, Long]))\n          )),\n        grokSlopScoreFeature = candidate.features.getOrElse(GrokSlopScoreFeature, None),\n        mediaCompletionRate = candidate.features.getOrElse(TweetMediaCompletionRateFeature, None),\n        tweetText = candidate.features.getOrElse(TweetTextFeature, None),\n        sourceSignal = sourceSignal,\n        grokContentCreator = Some(candidate.features.getOrElse(GrokContentCreatorFeature, false)),\n        gorkContentCreator = Some(candidate.features.getOrElse(GorkContentCreatorFeature, false)),\n        // MultiModalEmbeddingsFeature are not cached because the value size is too large for memcache\n        multiModalEmbedding = None\n      )\n    }\n\n    hmt.ScoredTweetsResponse(tweets)\n  }\n\n  final override def apply(\n    inputs: PipelineResultSideEffect.Inputs[PipelineQuery, ScoredTweetsResponse]\n  ): Stitch[Unit] = {\n    val candidates =\n      (inputs.selectedCandidates ++ inputs.remainingCandidates ++ inputs.droppedCandidates)\n        .filter(_.features.getOrElse(ScoreFeature, None).exists(_ > 0.0))\n\n    val truncatedCandidates =\n      if (candidates.size > MaxTweetsToCache)\n        candidates\n          .sortBy(-_.features.getOrElse(ScoreFeature, None).getOrElse(0.0)).take(MaxTweetsToCache)\n      else candidates\n\n    if (truncatedCandidates.nonEmpty) {\n      val ttl = inputs.query.params(CachedScoredTweets.TTLParam)\n      val scoredTweets = buildCachedScoredTweets(inputs.query, truncatedCandidates)\n      Stitch.callFuture(scoredTweetsCache.set(inputs.query.getRequiredUserId, scoredTweets, ttl))\n    } else Stitch.Unit\n  }\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(99.4)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/CommonFeaturesSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.side_effect\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finatra.kafka.serde.ScalaSerdes\nimport com.twitter.home_mixer.functional_component.side_effect.CommonFeaturesPldrConverter\nimport com.twitter.home_mixer.param.HomeMixerFlagName.ScribeFeaturesFlag\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.CommonFeaturesScribeEventPublisher\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.side_effect.KafkaAndScribePublishingSideEffect\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.ml.kafka.serde.TBaseSerde\nimport com.twitter.timelines.suggests.common.poly_data_record.thriftjava.PolyDataRecord\nimport com.twitter.timelines.suggests.common.poly_data_record.{thriftjava => pldr}\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport org.apache.kafka.clients.producer.ProducerRecord\nimport org.apache.kafka.common.serialization.Serializer\n\n/**\n * Publish common features sent to prediction service + some other features as PLDR format\n * into both scribe logs and kafka\n */\n@Singleton\nclass CommonFeaturesSideEffect @Inject() (\n  serviceIdentifier: ServiceIdentifier,\n  commonFeaturesPldrConverter: CommonFeaturesPldrConverter,\n  @Flag(ScribeFeaturesFlag) enableScribeFeatures: Boolean,\n  @Named(CommonFeaturesScribeEventPublisher) override val logPipelinePublisher: EventPublisher[\n    pldr.PolyDataRecord\n  ]) extends KafkaAndScribePublishingSideEffect[\n      Long,\n      pldr.PolyDataRecord,\n      PipelineQuery,\n      HasMarshalling\n    ]\n    with Conditionally[PipelineQuery, HasMarshalling] {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"CommonFeaturesSideEffect\")\n\n  private val kafkaTopic: String = serviceIdentifier.environment.toLowerCase match {\n    case \"prod\" => \"tq_ct_common_features\"\n    case _ => \"tq_ct_common_features_staging\"\n  }\n\n  override val bootstrapServer: String = \"/s/kafka/timeline:kafka-tls\"\n  override val keySerde: Serializer[Long] = ScalaSerdes.Long.serializer()\n  override val valueSerde: Serializer[PolyDataRecord] =\n    TBaseSerde.Thrift[pldr.PolyDataRecord]().serializer\n  override val clientId: String = \"home_mixer_common_features_producer\"\n\n  override def enableScribePublishing(query: PipelineQuery): Boolean = enableScribeFeatures\n\n  /** @see [[common.Conditionally.onlyIf]] */\n  override def onlyIf(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Boolean = selectedCandidates.nonEmpty\n\n  /**\n   * Build the record to be published to Kafka from query, selections and response\n   *\n   * @param query               PipelineQuery\n   * @param selectedCandidates  Result after Selectors are executed\n   * @param remainingCandidates Candidates which were not selected\n   * @param droppedCandidates   Candidates dropped during selection\n   * @param response            Result after Unmarshalling\n   *\n   * @return A sequence of to-be-published ProducerRecords\n   */\n  override def buildRecords(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Seq[ProducerRecord[Long, PolyDataRecord]] = {\n    commonFeaturesPldrConverter\n      .getCommonFeaturesPldr(query, selectedCandidates).map {\n        case (predictionRequestId, pldr) =>\n          new ProducerRecord(kafkaTopic, predictionRequestId, pldr)\n      }.toSeq\n  }\n\n  override val alerts = Seq(HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(98.5))\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/ScoredCandidateFeatureKeysKafkaSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.side_effect\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mysql.Client\nimport com.twitter.finagle.mysql.Transactions\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.finatra.kafka.serde.ScalaSerdes\nimport com.twitter.home_mixer.functional_component.scorer.CandidateFeaturesDataRecordFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokAnnotationsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsReadFromCacheFeature\nimport com.twitter.home_mixer.model.HomeFeatures.PredictionRequestIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.WeightedModelScoreFeature\nimport com.twitter.home_mixer.model.PredictedScoreFeature.PredictedScoreFeatureSet\nimport com.twitter.home_mixer.param.HomeGlobalParams.IsSelectedByHeavyRankerCountParam\nimport com.twitter.home_mixer.param.HomeMixerFlagName.DataRecordMetadataStoreConfigsYmlFlag\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableScoredCandidateFeatureKeysKafkaPublishingParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.ScribedScoredCandidateNumParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.ml.api.DataRecordMerger\nimport com.twitter.product_mixer.component_library.side_effect.KafkaPublishingSideEffect\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordConverter\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.SpecificFeatures\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.ml.cont_train.common.domain.non_scalding.CandidateAndCommonFeaturesStreamingUtils\nimport com.twitter.timelines.data_processing.jobs.light_ranking.light_ranking_features_prep.RecapPartialFeaturesForTwoTowerModels\nimport com.twitter.timelines.ml.cont_train.common.domain.non_scalding.ScoredCandidateFeatureKeysAdapter\nimport com.twitter.timelines.ml.cont_train.common.domain.non_scalding.ScoredCandidateFeatureKeysFields\nimport com.twitter.timelines.ml.kafka.serde.TBaseSerde\nimport com.twitter.timelines.ml.pldr.client.MysqlClientUtils\nimport com.twitter.timelines.ml.pldr.client.VersionedMetadataCacheClient\nimport com.twitter.timelines.ml.pldr.conversion.VersionIdAndFeatures\nimport com.twitter.timelines.suggests.common.data_record_metadata.{thriftscala => drmd}\nimport com.twitter.timelines.suggests.common.poly_data_record.{thriftjava => pldr}\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport org.apache.kafka.clients.producer.ProducerRecord\nimport org.apache.kafka.common.serialization.Serializer\nimport scala.collection.JavaConverters._\n\n/**\n * Pipeline side-effect that publishes scored candidate feature keys to a Kafka topic.\n */\n@Singleton\nclass ScoredCandidateFeatureKeysKafkaSideEffect @Inject() (\n  serviceIdentifier: ServiceIdentifier,\n  @Flag(DataRecordMetadataStoreConfigsYmlFlag) dataRecordMetadataStoreConfigsYml: String,\n  statsReceiver: StatsReceiver)\n    extends KafkaPublishingSideEffect[\n      Long,\n      pldr.PolyDataRecord,\n      PipelineQuery,\n      HasMarshalling\n    ]\n    with Conditionally[PipelineQuery, HasMarshalling]\n    with Logging {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\n    \"ScoredCandidateFeatureKeysKafka\")\n  private val statScope: String = this.getClass.getSimpleName\n\n  private val scopedStatsReceiver = statsReceiver.scope(statScope)\n  private val metadataFetchFailedCounter = scopedStatsReceiver.counter(\"metadataFetchFailed\")\n  private val randomSelectedCandidatesStat =\n    scopedStatsReceiver.stat(\"randomSelectedCandidates\")\n  private val randomDroppedCandidatesStat =\n    scopedStatsReceiver.stat(\"randomDroppedCandidates\")\n\n  override def onlyIf(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Boolean = {\n    if (query.params.getBoolean(EnableScoredCandidateFeatureKeysKafkaPublishingParam)) {\n      val scribedCandidateNumber = query.params(ScribedScoredCandidateNumParam)\n      filterRankedCandidates(selectedCandidates).size >= scribedCandidateNumber &&\n      filterRankedCandidates(droppedCandidates).size >= scribedCandidateNumber && selectedCandidates\n        .forall(!_.features.getOrElse(IsReadFromCacheFeature, false)) && remainingCandidates\n        .forall(!_.features.getOrElse(IsReadFromCacheFeature, false)) && droppedCandidates\n        .forall(!_.features.getOrElse(IsReadFromCacheFeature, false))\n    } else false\n  }\n\n  private val kafkaTopic: String = serviceIdentifier.environment.toLowerCase match {\n    case \"prod\" => \"home_mixer_dropped_candidates_features\"\n    case _ => \"deep_retrieval_candidates_data_staging\"\n  }\n\n  override val bootstrapServer: String = \"/s/kafka/timeline:kafka-tls\"\n  override val keySerde: Serializer[Long] = ScalaSerdes.Long.serializer()\n  override val valueSerde: Serializer[pldr.PolyDataRecord] =\n    TBaseSerde.Thrift[pldr.PolyDataRecord]().serializer\n  override val clientId: String = \"home_mixer_dropped_candidate_feature_keys_producer\"\n\n  lazy private val dataRecordMetadataStoreClient: Option[Client with Transactions] = Try {\n    try {\n      val c = MysqlClientUtils.parseConfigFromYaml(dataRecordMetadataStoreConfigsYml)\n      logger.info(s\"pldr mysql config: ${c.host} ${c.port} ${c.user} ${c.database}\")\n    } catch {\n      case e: Throwable =>\n        logger.error(\"pldr mysql error: \" + e.toString)\n    }\n\n    MysqlClientUtils.mysqlClientProvider(\n      MysqlClientUtils.parseConfigFromYaml(dataRecordMetadataStoreConfigsYml)\n    )\n  }.toOption\n\n  lazy private val versionedMetadataCacheClientOpt: Option[\n    VersionedMetadataCacheClient[Map[drmd.FeaturesCategory, Option[VersionIdAndFeatures]]]\n  ] = dataRecordMetadataStoreClient.map { mysqlClient =>\n    new VersionedMetadataCacheClient[Map[drmd.FeaturesCategory, Option[VersionIdAndFeatures]]](\n      maximumSize = 1,\n      expireDurationOpt = None,\n      mysqlClient = mysqlClient,\n      transform = CandidateAndCommonFeaturesStreamingUtils.metadataTransformer,\n      statsReceiver = statsReceiver\n    )\n  }\n\n  versionedMetadataCacheClientOpt.foreach {\n    _.metadataFetchTimerTask(\n      CandidateAndCommonFeaturesStreamingUtils.metadataFetchKey,\n      metadataFetchTimer = DefaultTimer,\n      metadataFetchInterval = 90.seconds,\n      metadataFetchFailedCounter = metadataFetchFailedCounter\n    )\n  }\n\n  private val drMerger = new DataRecordMerger\n  private val predictedScoreFeaturesDataRecordAdapter =\n    new DataRecordConverter(SpecificFeatures(PredictedScoreFeatureSet))\n\n  override def buildRecords(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Seq[ProducerRecord[Long, pldr.PolyDataRecord]] = {\n    val rankedSelected = filterRankedCandidates(selectedCandidates)\n      .filterNot { candidate =>\n        val annotations = candidate.features.getOrElse(GrokAnnotationsFeature, None)\n        val isNsfw = annotations.flatMap(_.metadata.map(_.isNsfw)).getOrElse(false)\n        val isSoftNsfw = annotations.flatMap(_.metadata.map(_.isSoftNsfw)).getOrElse(false)\n        val isGore = annotations.flatMap(_.metadata.map(_.isGore)).getOrElse(false)\n        val isViolent = annotations.flatMap(_.metadata.map(_.isViolent)).getOrElse(false)\n        val isSpam = annotations.flatMap(_.metadata.map(_.isSpam)).getOrElse(false)\n        isNsfw || isSoftNsfw || isGore || isViolent || isSpam\n      }\n    val rankedDropped = filterRankedCandidates(droppedCandidates)\n\n    val candidates = rankedSelected ++ rankedDropped\n    val isSelectedCandidateIds: Set[Long] = selectedCandidates.map(_.candidateIdLong).toSet\n    val candidatesHeavyRankerScoreBasedRank: Map[Long, Int] = candidates\n      .sortBy(\n        -_.features\n          .getOrElse(WeightedModelScoreFeature, None).getOrElse(Double.NegativeInfinity)).map(\n        _.candidateIdLong).zipWithIndex.toMap\n    val isSelectedByHeavyRankerCount = query.params(IsSelectedByHeavyRankerCountParam)\n\n    val predictionRequestId = candidates.headOption.flatMap { candidate =>\n      candidate.features.getOrElse(PredictionRequestIdFeature, None)\n    }\n\n    val scribedCandidateNumber = query.params(ScribedScoredCandidateNumParam)\n\n    val randomRankedSelected = selectRandomCandidates(rankedSelected, scribedCandidateNumber)\n    val randomRankedDropped = selectRandomCandidates(rankedDropped, scribedCandidateNumber)\n\n    randomSelectedCandidatesStat.add(randomRankedSelected.size)\n    randomDroppedCandidatesStat.add(randomRankedDropped.size)\n\n    val randomCandidates = randomRankedSelected ++ randomRankedDropped\n\n    randomCandidates.flatMap { candidate =>\n      val key = candidate.candidateIdLong\n      try {\n        val scoredCandidateFeatureKeysDataRecord = ScoredCandidateFeatureKeysAdapter\n          .adaptToDataRecords(\n            ScoredCandidateFeatureKeysFields(\n              viewerId = query.getRequiredUserId,\n              tweetId = key,\n              isSelected = isSelectedCandidateIds.contains(candidate.candidateIdLong),\n              isSelectedByHeavyRanker = candidatesHeavyRankerScoreBasedRank\n                .getOrElse(\n                  candidate.candidateIdLong,\n                  candidates.size) < isSelectedByHeavyRankerCount,\n              predictionRequestId = predictionRequestId,\n              requestTimeMs = Some(query.queryTime.inMilliseconds),\n              scribedCandidateNumber = scribedCandidateNumber,\n              viewerFollowsOriginalAuthor =\n                Some(candidate.features.getOrElse(InNetworkFeature, false)),\n              rankByHeavyRanker =\n                Some(candidatesHeavyRankerScoreBasedRank.getOrElse(key, candidates.size))\n            )\n          ).asScala.head\n\n        val candidateFeaturesDataRecord = CandidateAndCommonFeaturesStreamingUtils\n          .extractFeaturesFromDrWithContext(\n            candidate.features.get(CandidateFeaturesDataRecordFeature),\n            RecapPartialFeaturesForTwoTowerModels.scoredCandidateFeatureContext\n          )\n        drMerger.merge(scoredCandidateFeatureKeysDataRecord, candidateFeaturesDataRecord)\n\n        val predictedScoreFeaturesDataRecord =\n          predictedScoreFeaturesDataRecordAdapter.toDataRecord(candidate.features)\n        drMerger.merge(scoredCandidateFeatureKeysDataRecord, predictedScoreFeaturesDataRecord)\n\n        val convertedPlDrOpt =\n          CandidateAndCommonFeaturesStreamingUtils.candidateFeaturesToPolyDataRecord(\n            versionedMetadataCacheClientOpt = versionedMetadataCacheClientOpt,\n            candidateFeatures = scoredCandidateFeatureKeysDataRecord,\n            valueFormat = pldr.PolyDataRecord._Fields.LITE_COMPACT_DATA_RECORD\n          )\n\n        convertedPlDrOpt.map { convertedPlDr =>\n          new ProducerRecord(kafkaTopic, key, convertedPlDr)\n        }\n      } catch {\n        case e: Exception =>\n          logger.error(\n            s\"Error while converting features to PolyDataRecord for candidateId: $key\",\n            e\n          )\n          None\n      }\n    }\n  }\n\n  private def filterRankedCandidates(\n    candidates: Seq[CandidateWithDetails]\n  ): Seq[CandidateWithDetails] =\n    candidates.filter(_.features.contains(CandidateFeaturesDataRecordFeature))\n\n  private def selectRandomCandidates(\n    candidates: Seq[CandidateWithDetails],\n    count: Int\n  ): Seq[CandidateWithDetails] =\n    scala.util.Random.shuffle(CandidatesUtil.getItemCandidates(candidates)).take(count)\n\n  override val alerts = Seq(\n    HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert(98.5)\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/ScoredContentExplorationCandidateScoreFeatureKafkaSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.side_effect\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.kafka.serde.ScalaSerdes\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.WeightedModelScoreFeature\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableContentExplorationScoreScribingParam\nimport com.twitter.product_mixer.component_library.side_effect.KafkaPublishingSideEffect\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.strato.columns.content_understanding.content_understanding.thriftscala.ContentExplorationServingResult\nimport com.twitter.strato.columns.content_understanding.content_understanding.thriftscala.ContentExplorationServingResultsAnalysis\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport org.apache.kafka.clients.producer.ProducerRecord\nimport org.apache.kafka.common.serialization.Serializer\n\n/**\n * Pipeline side-effect that publishes scores feature keys to a Kafka topic.\n */\n@Singleton\nclass ScoredContentExplorationCandidateScoreFeatureKafkaSideEffect @Inject() (\n  serviceIdentifier: ServiceIdentifier,\n  statsReceiver: StatsReceiver)\n    extends KafkaPublishingSideEffect[\n      Long,\n      ContentExplorationServingResultsAnalysis,\n      PipelineQuery,\n      HasMarshalling\n    ]\n    with Conditionally[PipelineQuery, HasMarshalling]\n    with Logging {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\n    \"ScoredContentExplorationCandidateScoreFeatureKafka\")\n  private val statScope: String = this.getClass.getSimpleName\n  private val scopedStatsReceiver = statsReceiver.scope(statScope)\n  private val failedScribingCount = scopedStatsReceiver.scope(\"failed\").counter()\n\n  override def onlyIf(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Boolean =\n    query.params(EnableContentExplorationScoreScribingParam)\n\n  private val eligibleServedType: Set[hmt.ServedType] = Set(\n    hmt.ServedType.ForYouContentExploration,\n    hmt.ServedType.ForYouContentExplorationTier2,\n    hmt.ServedType.ForYouContentExplorationDeepRetrievalI2i,\n    hmt.ServedType.ForYouContentExplorationTier2DeepRetrievalI2i,\n    hmt.ServedType.ForYouUserInterestSummary\n  )\n\n  private val kafkaTopic: String = serviceIdentifier.environment.toLowerCase match {\n    case \"prod\" => \"content_exploration_ranking_analysis\"\n    case _ => \"content_exploration_ranking_analysis_devel\"\n  }\n\n  override val bootstrapServer: String = \"\"\n  override val keySerde: Serializer[Long] = ScalaSerdes.Long.serializer()\n  override val valueSerde: Serializer[ContentExplorationServingResultsAnalysis] =\n    ScalaSerdes.Thrift[ContentExplorationServingResultsAnalysis].serializer\n  override val clientId: String = \"home_mixer_dropped_candidate_feature_keys_producer\"\n\n  override def buildRecords(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Seq[ProducerRecord[Long, ContentExplorationServingResultsAnalysis]] = {\n    try {\n      val userId: Long = query.getUserOrGuestId.getOrElse(-1L)\n\n      val allCandidates = selectedCandidates ++ remainingCandidates ++ droppedCandidates\n      val eligible = allCandidates.filter(selectEligibleCandidates)\n\n      val scoredResults: Seq[ContentExplorationServingResult] = eligible.map { cdd =>\n        ContentExplorationServingResult(\n          cdd.candidateIdLong,\n          cdd.features.get(ScoreFeature),\n          cdd.features.get(WeightedModelScoreFeature),\n          Some(cdd.features.getOrElse(ServedTypeFeature, hmt.ServedType.Undefined).toString)\n        )\n      }\n      val analysisResult = ContentExplorationServingResultsAnalysis(\n        userId,\n        scoredResults\n      )\n      Seq(\n        new ProducerRecord[Long, ContentExplorationServingResultsAnalysis](\n          kafkaTopic,\n          userId,\n          analysisResult\n        )\n      )\n    } catch {\n      case _: Throwable =>\n        failedScribingCount.incr()\n        Seq.empty\n    }\n  }\n\n  def selectEligibleCandidates(\n    candidate: CandidateWithDetails\n  ): Boolean = {\n    val servedType =\n      candidate.features\n        .getOrElse(ServedTypeFeature, hmt.ServedType.Undefined)\n    eligibleServedType.contains(servedType)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/ScoredPhoenixCandidatesKafkaSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.side_effect\n\nimport com.google.inject.name.Named\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.kafka.serde.ScalaSerdes\nimport com.twitter.finatra.kafka.serde.UnKeyed\nimport com.twitter.finatra.kafka.serde.UnKeyedSerde\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.home_mixer.model.PhoenixPredictedScoreFeature.PhoenixPredictedScoreFeatures\nimport com.twitter.home_mixer.model.PredictedScoreFeature.PredictedScoreFeatures\nimport com.twitter.home_mixer.param.HomeGlobalParams.PhoenixCluster\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableScoredPhoenixCandidatesKafkaSideEffectParam\nimport com.twitter.home_mixer.util.PhoenixUtils.createCandidateSets\nimport com.twitter.home_mixer.util.PhoenixUtils.getPredictionResponseMap\nimport com.twitter.home_mixer.util.PhoenixUtils.getTweetInfoFromCandidates\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.side_effect.KafkaPublishingSideEffect\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.util.MemoizingStatsReceiver\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.timeline_logging.thriftscala.PredictionScore\nimport com.twitter.timelines.timeline_logging.thriftscala.ScoredCandidate\nimport com.twitter.util.logging.Logging\nimport com.x.user_action_sequence.ActionName\nimport com.x.user_action_sequence.PredictNextActionsRequest\nimport io.grpc.ManagedChannel\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport org.apache.kafka.clients.producer.ProducerRecord\nimport org.apache.kafka.common.serialization.Serializer\n\n/**\n * Pipeline side-effect that publishes scored phoenix candidates to a Kafka topic.\n */\n@Singleton\nclass ScoredPhoenixCandidatesKafkaSideEffect @Inject() (\n  @Named(\"PhoenixClient\") channelsMap: Map[PhoenixCluster.Value, Seq[ManagedChannel]],\n  serviceIdentifier: ServiceIdentifier,\n  statsReceiver: StatsReceiver)\n    extends KafkaPublishingSideEffect[\n      UnKeyed,\n      ScoredCandidate,\n      PipelineQuery,\n      HasMarshalling\n    ]\n    with Conditionally[PipelineQuery, HasMarshalling]\n    with Logging {\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\n    \"ScoredPhoenixCandidatesKafka\")\n  private val statScope: String = this.getClass.getSimpleName\n\n  val memoizingStatsReceiver: MemoizingStatsReceiver = new MemoizingStatsReceiver(\n    statsReceiver.scope(statScope))\n\n  override def onlyIf(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Boolean = {\n    query.params(EnableScoredPhoenixCandidatesKafkaSideEffectParam) &&\n    query.features.flatMap(_.getOrElse(UserActionsFeature, None)).isDefined\n  }\n\n  private val kafkaTopic: String = serviceIdentifier.environment.toLowerCase match {\n    case \"prod\" => \"home_mixer_phoenix_scored_candidates\"\n    case _ => \"home_mixer_phoenix_scored_candidates_staging\"\n  }\n\n  override val bootstrapServer: String = \"\"\n  override val keySerde: Serializer[UnKeyed] = UnKeyedSerde.serializer()\n  override val valueSerde: Serializer[ScoredCandidate] =\n    ScalaSerdes.Thrift[ScoredCandidate].serializer\n  override val clientId: String = \"home_mixer_phoenix_scored_candidate_producer\"\n\n  // Returns Map from phoenixCluster -> Map [Tweets -> Map [Actions -> Score]]\n  private def getPredictionResponsesAllClusters(\n    request: PredictNextActionsRequest\n  ): Stitch[Map[String, Map[Long, Map[ActionName, Double]]]] = {\n    val phoenixClusters = channelsMap.keySet.toSeq\n    val timeoutMs = 1000\n    Stitch\n      .traverse(phoenixClusters) { phoenixCluster =>\n        val channels = channelsMap(phoenixCluster)\n        getPredictionResponseMap(\n          request,\n          channels,\n          phoenixCluster.toString,\n          timeoutMs,\n          memoizingStatsReceiver)\n          .handle {\n            case _ => Map.empty[Long, Map[ActionName, Double]]\n          }\n          .map(phoenixCluster.toString -> _)\n      }.map(_.toMap)\n  }\n\n  // Not defined buildRecords as we override apply to use buildRecordsStitch\n  override def buildRecords(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: HasMarshalling\n  ): Seq[ProducerRecord[UnKeyed, ScoredCandidate]] = {\n    ???\n  }\n\n  def buildRecordsStitch(\n    query: PipelineQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n  ): Stitch[Seq[ProducerRecord[UnKeyed, ScoredCandidate]]] = {\n    val nonCachedSelectedCandidates =\n      selectedCandidates.filterNot(_.features.getOrElse(IsReadFromCacheFeature, false))\n\n    val candidates = nonCachedSelectedCandidates.map(_.getCandidate[TweetCandidate]())\n    val featureMaps = nonCachedSelectedCandidates.map(_.features)\n    val tweetInfos = getTweetInfoFromCandidates(candidates, featureMaps)\n    val request = createCandidateSets(query, tweetInfos)\n    val predictionsMapStitch =\n      getPredictionResponsesAllClusters(request)\n\n    val scoredCandidatesStitch = Stitch.traverse(nonCachedSelectedCandidates) { candidate =>\n      val tweetId = candidate.candidateIdLong\n      val authorId = candidate.features.getOrElse(AuthorIdFeature, None)\n      val sourceTweetId =\n        candidate.features.getOrElse(SourceTweetIdFeature, None) match {\n          case Some(sourceTweetId) => sourceTweetId\n          case _ => tweetId\n        }\n      predictionsMapStitch.map { predictionsMap =>\n        val phoenixPredictionScores = predictionsMap.flatMap {\n          case (phoenixCluster, scoresMapPerTweet) =>\n            val actionPredictionsMap = scoresMapPerTweet.getOrElse(sourceTweetId, Map.empty)\n            PhoenixPredictedScoreFeatures.map { feature =>\n              val predictions = feature.actions.flatMap(actionPredictionsMap.get)\n              val score = if (predictions.nonEmpty) Some(predictions.max) else None\n              val featureString = f\"phoenix.$phoenixCluster.${feature.featureName}\"\n              PredictionScore(Some(featureString), score)\n            }\n        }.toSeq\n        val prodScores = PredictedScoreFeatures.map { feature =>\n          PredictionScore(Some(feature.statName), candidate.features.getOrElse(feature, None))\n        }\n        ScoredCandidate(\n          tweetId = tweetId,\n          authorId = authorId,\n          viewerId = query.getOptionalUserId,\n          sourceTweetId = Some(sourceTweetId),\n          servedRequestId = query.features.flatMap(_.getOrElse(PredictionRequestIdFeature, None)),\n          requestTimeMs = Some(query.queryTime.inMillis),\n          predictionScores = Some((phoenixPredictionScores ++ prodScores).toSet)\n        )\n      }\n    }\n\n    scoredCandidatesStitch.map { predictionScoresSeq =>\n      predictionScoresSeq.map {\n        new ProducerRecord(kafkaTopic, new UnKeyed, _)\n      }\n    }\n  }\n\n  override def apply(\n    inputs: PipelineResultSideEffect.Inputs[PipelineQuery, HasMarshalling]\n  ): Stitch[Unit] = {\n    val recordsStitch = buildRecordsStitch(\n      query = inputs.query,\n      selectedCandidates = inputs.selectedCandidates,\n    )\n\n    recordsStitch.flatMap { records =>\n      Stitch\n        .traverse(records) { record =>\n          Stitch.callFuture(kafkaProducer.send(record))\n        }\n    }.unit\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/ScoredStatsSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.side_effect\n\nimport com.twitter.core_workflows.user_model.thriftscala.UserState\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.NullStatsReceiver.NullCounter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokAnnotationsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasVideoFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InNetworkFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.LowSignalUserFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedAuthorIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SlopAuthorScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMediaIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserStateFeature\nimport com.twitter.home_mixer.model.HomeFeatures.WeightedModelScoreFeature\nimport com.twitter.home_mixer.model.PredictedScoreFeature\nimport com.twitter.home_mixer.model.PredictedScoreFeature.PredictedScoreFeatures\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.AuthorListForDataCollectionParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.RequestNormalizedScoresParam\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.CachedScoredTweetsCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.candidate_pipeline.ScoredTweetsTweetMixerCandidatePipelineConfig\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsResponse\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.impressed_tweets.ImpressedTweets\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidatePipelines\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateSourcePosition\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Memoize\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class ScoredStatsSideEffect @Inject() (\n  statsReceiver: StatsReceiver)\n    extends PipelineResultSideEffect[ScoredTweetsQuery, ScoredTweetsResponse]\n    with Logging {\n\n  import ScoredStatsSideEffect._\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"ScoredStats\")\n\n  private val baseStatsReceiver = statsReceiver.scope(identifier.toString)\n  private val authorStatsReceiver = baseStatsReceiver.scope(\"Author\")\n  private val authorScoreStatsReceiver = baseStatsReceiver.scope(\"AuthorScore10000x\")\n  private val servedTypeStatsReceiver = baseStatsReceiver.scope(\"ServedType\")\n  private val contentBalanceStatsReceiver = baseStatsReceiver.scope(\"ContentBalance\")\n  private val grokAnnotationsStatsReceiver = baseStatsReceiver.scope(\"GrokAnnotations\")\n\n  private val inNetworkStatsReceiver = contentBalanceStatsReceiver.scope(\"InNetwork\")\n  private val outOfNetworkStatsReceiver = contentBalanceStatsReceiver.scope(\"OutOfNetwork\")\n  private val replyStatsReceiver = contentBalanceStatsReceiver.scope(\"Reply\")\n  private val originalStatsReceiver = contentBalanceStatsReceiver.scope(\"Original\")\n  private val retweetStatsReceiver = contentBalanceStatsReceiver.scope(\"Retweet\")\n\n  private val tweetMixerRecallStatsReceiver = baseStatsReceiver.scope(\"TweetMixerRecall10000x\")\n  private val deepRetrievalRecallStatsReceiver =\n    baseStatsReceiver.scope(\"DeepRetrievalRecall10000x\")\n  private val tweetMixerTweetsLandOnTopKStatsReceiver = baseStatsReceiver.scope(\"TweetMixerTweets\")\n  private val deepRetrievalTweetsLandOnTopKStatsReceiver =\n    baseStatsReceiver.scope(\"DeepRetrievalTweets\")\n\n  private val grokAnnotationsInStatsReceiver = grokAnnotationsStatsReceiver.scope(\"InNetwork\")\n  private val grokAnnotationsOonStatsReceiver = grokAnnotationsStatsReceiver.scope(\"OutOfNetwork\")\n  private val grokAnnotationsReplyStatsReceiver = grokAnnotationsStatsReceiver.scope(\"Reply\")\n\n  private val signalStatsReceiver = baseStatsReceiver.scope(\"signal\")\n  private val authorDiversityStatsReceiver = baseStatsReceiver.scope(\"authorDiversity\")\n  private val impressedAuthorSizeStat = authorDiversityStatsReceiver.stat(\"impressedAuthorSize\")\n  private val meanImpressedAuthorCountStat =\n    authorDiversityStatsReceiver.stat(\"meanImpressedAuthorCount1000x\")\n\n  private val grokSlopScoreStatsReceiver = baseStatsReceiver.scope(\"GrokSlopScore\")\n\n  private val inNetAuthorStatsReceiver = baseStatsReceiver.scope(\"InNetAuthor\")\n\n  private val StatsReadabilityMultiplier = 1000\n  private val SmallFollowGraphSize = 25\n\n  private val StatsEligibleGrokTopics = Set(\n    \"News\",\n    \"Sports\",\n    \"Entertainment\",\n    \"Business & Finance\",\n    \"Technology\",\n    \"Politics\",\n    \"Fashion & Beauty\",\n    \"Gaming\",\n    \"Lifestyle\",\n    \"Movies & TV\",\n    \"Music\",\n    \"Travel\",\n    \"Food\",\n    \"Comedy\",\n    \"Health & Fitness\"\n  )\n\n  // Group of stats to collect per model\n  case class ModelStats(normalized: Boolean, rankingMode: String) {\n    val normalizedScope = if (normalized) \"normalized\" else \"weighted\"\n    val scopedStatsReceiver = baseStatsReceiver.scope(normalizedScope).scope(rankingMode)\n    val PredictedScoreCounterName = f\"predictedScore${StatsReadabilityMultiplier}xSum\"\n    val requestCounter = scopedStatsReceiver.counter(\"candidatesSum\")\n    private val predictedScoreCounters: Map[PredictedScoreFeature, Counter] =\n      PredictedScoreFeatures.map { scoreFeature =>\n        (\n          scoreFeature,\n          scopedStatsReceiver.counter(scoreFeature.statName, PredictedScoreCounterName))\n      }.toMap\n\n    def getPredictedScoreCounter(scoreFeature: PredictedScoreFeature): Counter =\n      predictedScoreCounters.getOrElse(scoreFeature, NullCounter)\n  }\n\n  // Memoize stats object so they are not created per request\n  private val statsPerModel = Memoize[(Boolean, String), ModelStats] {\n    case (normalized, rankingMode) => ModelStats(normalized, rankingMode)\n  }\n\n  override def apply(\n    inputs: PipelineResultSideEffect.Inputs[ScoredTweetsQuery, ScoredTweetsResponse]\n  ): Stitch[Unit] = {\n    val scopeCandidateMap = Map(\n      \"selected\" -> inputs.selectedCandidates,\n      \"dropped\" -> inputs.droppedCandidates,\n      \"remaining\" -> inputs.remainingCandidates\n    )\n    // We choose 50 because we don't have per head scores for cached candidates\n    val topSelectedCandidates = inputs.selectedCandidates.take(50)\n\n    val impressedTweetIds =\n      inputs.query.features.map(_.getOrElse(ImpressedTweets, Seq.empty)).getOrElse(Seq.empty).toSet\n    val servedAuthorMap: Map[Long, Seq[Long]] =\n      inputs.query.features\n        .map(_.getOrElse(ServedAuthorIdsFeature, Map.empty[Long, Seq[Long]]))\n        .getOrElse(Map.empty[Long, Seq[Long]])\n    recordAuthorDiversityStats(impressedTweetIds, servedAuthorMap)\n\n    recordScoreStats(\n      topSelectedCandidates,\n      inputs.query.params(RequestNormalizedScoresParam),\n      inputs.query.params(ScoredTweetsParam.TweetMixerRankingModeForStatsRecallAtKParam),\n      inputs.query\n    )\n\n    val allCandidates: Seq[CandidateWithDetails] =\n      inputs.selectedCandidates ++ inputs.droppedCandidates\n    recordInNetworkAuthorStats(allCandidates, inputs.query, \"retrieved\")\n    recordInNetworkAuthorStats(inputs.selectedCandidates, inputs.query, \"selected\")\n\n    scopeCandidateMap.map {\n      case (scope, candidates) =>\n        recordAuthorStats(candidates, scope, inputs.query.params(AuthorListForDataCollectionParam))\n        recordServedTypeStats(candidates, scope)\n        recordGrokAnnotationsStats(candidates, scope)\n        recordContentBalanceStats(\n          candidates,\n          scope,\n          inputs.query.params(ScoredTweetsParam.TweetMixerRankingModeForStatsRecallAtKParam)\n        )\n        recordHasVideoStats(candidates, scope)\n        recordHasMediaStats(candidates, scope)\n        recordSlopScoreStats(candidates, scope)\n    }\n\n    recordTweetMixerRecallTopKStats(\n      inputs.selectedCandidates.filter(\n        _.source != CachedScoredTweetsCandidatePipelineConfig.Identifier),\n      inputs.droppedCandidates.filter(\n        _.source != CachedScoredTweetsCandidatePipelineConfig.Identifier),\n      inputs.query.params(ScoredTweetsParam.TweetMixerRankingModeForStatsRecallAtKParam)\n    )\n\n    recordSignalStats(inputs.query)\n\n    Stitch.Unit\n  }\n\n  private def recordAuthorDiversityStats(\n    impressedTweetIds: Set[Long],\n    servedAuthorMap: Map[Long, Seq[Long]]\n  ): Unit = {\n    val authorFreq = servedAuthorMap\n      .collect {\n        case (authorId, tweetIds) =>\n          val impressedCount = tweetIds.count(impressedTweetIds.contains)\n          if (impressedCount > 0) Some(authorId -> impressedCount) else None\n      }.flatten.toMap\n\n    val impressedAuthorSize = authorFreq.size\n    val meanImpressedAuthorCount =\n      if (authorFreq.isEmpty) 0.0f else authorFreq.values.sum.toFloat / impressedAuthorSize\n\n    impressedAuthorSizeStat.add(impressedAuthorSize)\n    meanImpressedAuthorCountStat.add(meanImpressedAuthorCount * 1000)\n  }\n\n  private def recordScoreStats(\n    candidates: Seq[CandidateWithDetails],\n    normalized: Boolean,\n    rankingMode: String,\n    query: PipelineQuery\n  ): Unit = {\n    val modelStats = statsPerModel((normalized, rankingMode))\n    modelStats.requestCounter.incr()\n    PredictedScoreFeatures.map { predictedScoreFeature =>\n      val counter = modelStats.getPredictedScoreCounter(predictedScoreFeature)\n      val predictedScoreSum = candidates\n        .map { candidate =>\n          predictedScoreFeature\n            .extractScore(candidate.features, query)\n            .getOrElse(0.0) * StatsReadabilityMultiplier\n        }.sum.toInt\n      counter.incr(predictedScoreSum)\n    }\n  }\n\n  private def recordTweetMixerRecallTopKStats(\n    selectedCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    rankingMode: String\n  ): Unit = {\n    val allCandidates = selectedCandidates ++ droppedCandidates\n    val allCandidatesByHeavyRanker = allCandidates\n      .sortBy(\n        -_.features.getOrElse(WeightedModelScoreFeature, None).getOrElse(Double.NegativeInfinity))\n    val tweetMixerTweets = allCandidatesByHeavyRanker.filter(fromTweetMixer)\n    val tweetMixerTweetsSortByRank =\n      tweetMixerTweets.sortBy(candidate => candidate.features.get(CandidateSourcePosition))\n\n    val deepRetrievalTweets = tweetMixerTweets.filter(tweetMixerTweet =>\n      tweetMixerTweet.features.get(\n        ServedTypeFeature) == hmt.ServedType.ForYouDeepRetrieval || tweetMixerTweet.features.get(\n        ServedTypeFeature) == hmt.ServedType.ForYouEvergreenDeepRetrieval)\n    val deepRetrievalTweetsSortByRank =\n      deepRetrievalTweets.sortBy(_.features.get(CandidateSourcePosition))\n\n    if (deepRetrievalTweets.nonEmpty) {\n      HeavyRankerTopK.foreach { K =>\n        deepRetrievalTweetsLandOnTopKStatsReceiver\n          .scope(rankingMode)\n          .stat(s\"LandOnTop${K}\")\n          .add(allCandidatesByHeavyRanker\n            .take(K).count { candidate =>\n              fromServedType(candidate, hmt.ServedType.ForYouDeepRetrieval) || fromServedType(\n                candidate,\n                hmt.ServedType.ForYouEvergreenDeepRetrieval)\n            })\n      }\n\n      deepRetrievalTweetsLandOnTopKStatsReceiver\n        .scope(rankingMode)\n        .stat(\"LandInSelected\")\n        .add(selectedCandidates.count { candidate =>\n          fromServedType(candidate, hmt.ServedType.ForYouDeepRetrieval) || fromServedType(\n            candidate,\n            hmt.ServedType.ForYouEvergreenDeepRetrieval)\n        })\n\n      DeepRetrievalRecallTopK.foreach { K =>\n        val rankedByHeavyRankerScoreTopK = deepRetrievalTweets.take(K)\n        val rankedByTweetMixerRankTopK = deepRetrievalTweetsSortByRank.take(K)\n        val intersectTweets = rankedByHeavyRankerScoreTopK\n          .map(candidate => candidate.candidateIdLong)\n          .intersect(rankedByTweetMixerRankTopK.map(candidate => candidate.candidateIdLong))\n\n        deepRetrievalRecallStatsReceiver\n          .scope(rankingMode)\n          .stat(s\"RecallAt${K}\")\n          .add(intersectTweets.size * 10000 / Math.min(K, deepRetrievalTweets.size))\n      }\n    }\n\n    if (tweetMixerTweets.nonEmpty) {\n      TweetMixerRecallTopK.foreach { K =>\n        tweetMixerTweetsLandOnTopKStatsReceiver\n          .scope(rankingMode)\n          .stat(s\"LandOnTop${K}\")\n          .add(allCandidatesByHeavyRanker\n            .take(K).count(fromTweetMixer))\n      }\n\n      tweetMixerTweetsLandOnTopKStatsReceiver\n        .scope(rankingMode)\n        .stat(\"LandInSelected\")\n        .add(selectedCandidates.count(fromTweetMixer))\n\n      TweetMixerRecallTopK.foreach { K =>\n        val rankedByHeavyRankerScoreTopK = tweetMixerTweets.take(K)\n        val rankedByTweetMixerRankTopK = tweetMixerTweetsSortByRank.take(K)\n        val intersectTweets = rankedByHeavyRankerScoreTopK\n          .map(candidate => candidate.candidateIdLong)\n          .intersect(rankedByTweetMixerRankTopK.map(candidate => candidate.candidateIdLong))\n\n        tweetMixerRecallStatsReceiver\n          .scope(rankingMode)\n          .stat(s\"RecallAt${K}\")\n          .add(intersectTweets.size * 10000 / Math.min(K, tweetMixerTweets.size))\n      }\n    }\n  }\n\n  def recordAuthorStats(\n    candidates: Seq[CandidateWithDetails],\n    scope: String,\n    authors: Set[Long]\n  ): Unit = {\n    candidates\n      .filter { candidate =>\n        candidate.features.getOrElse(AuthorIdFeature, None).exists(authors.contains) &&\n        // Only include original tweets\n        (!candidate.features.getOrElse(IsRetweetFeature, false)) &&\n        candidate.features.getOrElse(InReplyToTweetIdFeature, None).isEmpty\n      }\n      .groupBy { candidate =>\n        val servedType = candidate.features.getOrElse(ServedTypeFeature, hmt.ServedType.Undefined)\n        (servedType.name, candidate.features.get(AuthorIdFeature).get)\n      }\n      .foreach {\n        case ((servedType, authorId), authorCandidates) =>\n          val authorStr = authorId.toString.takeRight(9)\n          authorStatsReceiver\n            .scope(scope).scope(authorStr).counter(servedType)\n            .incr(authorCandidates.size)\n\n          authorCandidates.map { candidate =>\n            val score =\n              candidate.features.getOrElse(ScoreFeature, None).getOrElse(0.0).toFloat * 10000\n            authorScoreStatsReceiver.scope(scope).scope(authorStr).stat(servedType).add(score)\n          }\n      }\n  }\n\n  def recordServedTypeStats(\n    candidates: Seq[CandidateWithDetails],\n    scope: String\n  ): Unit = {\n    candidates.groupBy(getServedType).foreach {\n      case (servedType, servedTypeCandidates) =>\n        servedTypeStatsReceiver.scope(scope).counter(servedType).incr(servedTypeCandidates.size)\n    }\n  }\n\n  private def recordHasVideoStats(\n    candidates: Seq[CandidateWithDetails],\n    scope: String\n  ): Unit = {\n    candidates.groupBy(_.features.getOrElse(HasVideoFeature, false)).foreach {\n      case (hasVideo, hasVideoCandidates) =>\n        servedTypeStatsReceiver\n          .scope(scope).counter(if (hasVideo) \"HasVideo\" else \"NoVideo\").incr(\n            hasVideoCandidates.size)\n    }\n  }\n\n  private def recordHasMediaStats(\n    candidates: Seq[CandidateWithDetails],\n    scope: String\n  ): Unit = {\n    candidates.groupBy(_.features.getOrElse(TweetMediaIdsFeature, Seq[Long]()).nonEmpty).foreach {\n      case (hasMedia, hasMediaCandidates) =>\n        servedTypeStatsReceiver\n          .scope(scope).counter(if (hasMedia) \"HasMedia\" else \"NoMedia\").incr(\n            hasMediaCandidates.size)\n    }\n  }\n\n  def recordGrokAnnotationsStats(\n    candidates: Seq[CandidateWithDetails],\n    scope: String\n  ): Unit = {\n    candidates.foreach { candidate =>\n      val annotations = candidate.features.getOrElse(GrokAnnotationsFeature, None)\n\n      val topics = annotations\n        .map(_.topics.filter(StatsEligibleGrokTopics.contains(_)))\n        .filter(_.nonEmpty)\n\n      val in = candidate.features.getOrElse(InNetworkFeature, true)\n      val reply = candidate.features.getOrElse(InReplyToTweetIdFeature, None).isDefined\n      val baseStat =\n        if (reply) grokAnnotationsReplyStatsReceiver\n        else if (in) grokAnnotationsInStatsReceiver\n        else grokAnnotationsOonStatsReceiver\n\n      if (topics.isDefined) baseStat.scope(scope).counter(\"Available\").incr()\n      else baseStat.scope(scope).counter(\"Unavailable\").incr()\n\n      topics.map(_.map(topic => baseStat.counter(topic).incr()))\n    }\n  }\n\n  def recordSignalStats(\n    query: ScoredTweetsQuery\n  ): Unit = {\n    val userState = query.features.flatMap(_.getOrElse(UserStateFeature, None))\n    val userStateStr = userState\n      .map {\n        case UserState.New => \"new\"\n        case UserState.NearZero => \"nearZero\"\n        case UserState.VeryLight => \"veryLight\"\n        case UserState.Light => \"light\"\n        case UserState.MediumTweeter => \"medium\"\n        case UserState.MediumNonTweeter => \"medium\"\n        case UserState.HeavyNonTweeter => \"heavy\"\n        case UserState.HeavyTweeter => \"heavy\"\n        case UserState.EnumUnknownUserState(_) => \"none\"\n      }.getOrElse(\"none\")\n\n    val followGraphSize = query.features.map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty).size)\n    val lowFollow = followGraphSize.exists(_ < SmallFollowGraphSize)\n    val lowSignalUser =\n      query.features.map(_.getOrElse(LowSignalUserFeature, false)).getOrElse(false)\n\n    val stats = signalStatsReceiver.scope(userStateStr)\n\n    if (lowSignalUser) {\n      if (lowFollow) stats.counter(\"lowSignal-lowFollow\").incr()\n      else stats.counter(\"lowSignal-okFollow\").incr()\n    } else {\n      if (lowFollow) stats.counter(\"okSignal-lowFollow\").incr()\n      else stats.counter(\"okSignal-okFollow\").incr()\n    }\n  }\n\n  def recordContentBalanceStats(\n    candidates: Seq[CandidateWithDetails],\n    scope: String,\n    rankingMode: String\n  ): Unit = {\n    // Split by network type\n    val (in, oon) = candidates.partition(_.features.getOrElse(InNetworkFeature, true))\n    inNetworkStatsReceiver.counter(scope).incr(in.size)\n    outOfNetworkStatsReceiver.counter(scope).incr(oon.size)\n\n    // Track tweet types for all candidates (maintaining backward compatibility)\n    val (reply, nonReply) =\n      candidates.partition(_.features.getOrElse(InReplyToTweetIdFeature, None).isDefined)\n    replyStatsReceiver.counter(scope).incr(reply.size)\n    replyStatsReceiver.scope(rankingMode).counter(scope).incr(reply.size)\n    originalStatsReceiver.scope(rankingMode).counter(scope).incr(nonReply.size)\n    originalStatsReceiver.counter(scope).incr(nonReply.size)\n\n    // Enhanced tracking: retweets, replies, and original posts for both in-network and out-of-network\n    val (retweets, nonRetweets) =\n      candidates.partition(_.features.getOrElse(IsRetweetFeature, false))\n    val (replies, originalPosts) =\n      nonRetweets.partition(_.features.getOrElse(InReplyToTweetIdFeature, None).isDefined)\n\n    // Record retweet stats\n    retweetStatsReceiver.counter(scope).incr(retweets.size)\n    retweetStatsReceiver.scope(rankingMode).counter(scope).incr(retweets.size)\n\n    // Track by network type\n    val (inRetweets, oonRetweets) = retweets.partition(_.features.getOrElse(InNetworkFeature, true))\n    val (inReplies, oonReplies) = replies.partition(_.features.getOrElse(InNetworkFeature, true))\n    val (inOriginalPosts, oonOriginalPosts) =\n      originalPosts.partition(_.features.getOrElse(InNetworkFeature, true))\n\n    // In-network tweet type stats\n    retweetStatsReceiver.scope(\"InNetwork\").counter(scope).incr(inRetweets.size)\n    retweetStatsReceiver.scope(\"InNetwork\").scope(rankingMode).counter(scope).incr(inRetweets.size)\n    replyStatsReceiver.scope(\"InNetwork\").counter(scope).incr(inReplies.size)\n    replyStatsReceiver.scope(\"InNetwork\").scope(rankingMode).counter(scope).incr(inReplies.size)\n    originalStatsReceiver.scope(\"InNetwork\").counter(scope).incr(inOriginalPosts.size)\n    originalStatsReceiver\n      .scope(\"InNetwork\").scope(rankingMode).counter(scope).incr(inOriginalPosts.size)\n\n    // Out-of-network tweet type stats\n    retweetStatsReceiver.scope(\"OutOfNetwork\").counter(scope).incr(oonRetweets.size)\n    retweetStatsReceiver\n      .scope(\"OutOfNetwork\").scope(rankingMode).counter(scope).incr(oonRetweets.size)\n    replyStatsReceiver.scope(\"OutOfNetwork\").counter(scope).incr(oonReplies.size)\n    replyStatsReceiver.scope(\"OutOfNetwork\").scope(rankingMode).counter(scope).incr(oonReplies.size)\n    originalStatsReceiver.scope(\"OutOfNetwork\").counter(scope).incr(oonOriginalPosts.size)\n    originalStatsReceiver\n      .scope(\"OutOfNetwork\").scope(rankingMode).counter(scope).incr(oonOriginalPosts.size)\n  }\n\n  private def getServedType(candidate: CandidateWithDetails): String =\n    candidate.features.get(ServedTypeFeature).name\n\n  private def fromTweetMixer(candidate: CandidateWithDetails): Boolean = {\n    candidate.features\n      .get(CandidatePipelines)\n      .contains(ScoredTweetsTweetMixerCandidatePipelineConfig.Identifier)\n  }\n\n  private def fromServedType(\n    candidate: CandidateWithDetails,\n    servedType: hmt.ServedType\n  ): Boolean = candidate.features.get(ServedTypeFeature) == servedType\n\n  private def recordSlopScoreStats(\n    candidates: Seq[CandidateWithDetails],\n    scope: String\n  ): Unit = {\n    val count1 = candidates.count(_.features.getOrElse(SlopAuthorScoreFeature, None).contains(1))\n    val count2 = candidates.count(_.features.getOrElse(SlopAuthorScoreFeature, None).contains(2))\n    val count3 = candidates.count(_.features.getOrElse(SlopAuthorScoreFeature, None).contains(3))\n    val noneScore = candidates.count(_.features.getOrElse(SlopAuthorScoreFeature, None).isEmpty)\n    val total = candidates.size\n    grokSlopScoreStatsReceiver.scope(scope).counter(\"count1\").incr(count1)\n    grokSlopScoreStatsReceiver.scope(scope).counter(\"count2\").incr(count2)\n    grokSlopScoreStatsReceiver.scope(scope).counter(\"count3\").incr(count3)\n    grokSlopScoreStatsReceiver.scope(scope).counter(\"none\").incr(noneScore)\n    grokSlopScoreStatsReceiver.scope(scope).counter(\"total\").incr(total)\n  }\n\n  def recordInNetworkAuthorStats(\n    candidates: Seq[CandidateWithDetails],\n    query: ScoredTweetsQuery,\n    scope: String\n  ): Unit = {\n    val inNet = candidates.filter { candidate =>\n      candidate.features.getOrElse(InNetworkFeature, true) &&\n      (candidate.features.getOrElse(FromInNetworkSourceFeature, false))\n    }\n    val total = inNet.size\n    if (total == 0) return\n\n    val authorCounts: Seq[Int] = inNet\n      .flatMap(_.features.getOrElse(AuthorIdFeature, None))\n      .groupBy(identity).map { case (k, v) => (k, v.size) }.values.toSeq\n\n    val followGraphSize =\n      query.features.map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty).size).getOrElse(0)\n    val sr = inNetAuthorStatsReceiver.scope(scope)\n    if (followGraphSize > 0) {\n      val recall1000xSGS = (authorCounts.size * 1000) / followGraphSize\n      sr.stat(\"Recall1000xSGS\").add(recall1000xSGS)\n    }\n\n    val sorted = authorCounts.sorted(Ordering[Int].reverse)\n    def share(k: Int): Int = (sorted.take(k).sum * 1000) / total\n\n    sr.stat(\"Top1Share1000x\").add(share(1))\n    sr.stat(\"Top3Share1000x\").add(share(3))\n    sr.stat(\"Top10Share1000x\").add(share(10))\n  }\n\n  override val alerts = Seq(HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert())\n}\n\nobject ScoredStatsSideEffect {\n  val TweetMixerRecallTopK = Seq(25, 50, 100, 200, 400)\n  val DeepRetrievalRecallTopK = Seq(50, 100, 200)\n  val HeavyRankerTopK = Seq(1, 5, 10, 25, 50, 100, 200, 400)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/ScoredTweetsDiversityStatsSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.side_effect\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.functional_component.feature_hydrator.SimClustersLogFavBasedTweetFeature\nimport com.twitter.home_mixer.functional_component.feature_hydrator.adapters.simclusters_features.SimclustersFeaturesAdapter.SimclustersSparseTweetEmbeddingsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsResponse\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.CategoryDiversityRescoringParam\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.FeatureHydration.CategoryDiversityKParam\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.ml.api.DataRecord\nimport scala.jdk.CollectionConverters._\n\n@Singleton\ncase class ScoredTweetsDiversityStatsSideEffect @Inject() (\n  statsReceiver: StatsReceiver)\n    extends PipelineResultSideEffect[ScoredTweetsQuery, ScoredTweetsResponse] {\n\n  import ScoredTweetsDiversityStatsSideEffect._\n\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\"ScoredTweetsDiversityStats\")\n  private val baseStatsReceiver = statsReceiver.scope(identifier.toString)\n  private val diversityStatsReceiverMap = diversityTopK.map { k =>\n    k.toString -> baseStatsReceiver.scope(\"numUniqCategories@k=\" + k.toString)\n  }.toMap\n\n  override def apply(\n    inputs: PipelineResultSideEffect.Inputs[ScoredTweetsQuery, ScoredTweetsResponse]\n  ): Stitch[Unit] = {\n    val query = inputs.query\n    val selectedCandidates = inputs.selectedCandidates\n\n    if (query.params(CategoryDiversityRescoringParam)) {\n      diversityTopK.foreach { topK =>\n        getNumUniqueCategoriesStats(\n          query,\n          selectedCandidates,\n          topK\n        )\n      }\n    }\n\n    Stitch.Unit\n  }\n\n  private def getNumUniqueCategoriesStats(\n    query: ScoredTweetsQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    topK: Int\n  ): Unit = {\n    val topKSelectedCandidates = selectedCandidates\n      .sortBy(_.features.getOrElse(ScoreFeature, None).getOrElse(0.0))\n      .take(topK)\n\n    val categories = topKSelectedCandidates.map { candidate =>\n      val topKClustersMap =\n        Option(\n          candidate.features\n            .getOrElse(SimClustersLogFavBasedTweetFeature, new DataRecord())\n            .getSparseContinuousFeatures\n        ).flatMap(mapOpt =>\n          Option(mapOpt.get(SimclustersSparseTweetEmbeddingsFeature.getFeatureId)))\n      topKClustersMap\n        .map(_.asScala.toSeq.sortBy(-_._2).take(query.params(CategoryDiversityKParam)).map(_._1))\n        .getOrElse(Seq.empty)\n    }\n    // assign the num of unique categories to the stats receiver\n    val uniqueCategories = categories.flatten.toSet\n    diversityStatsReceiverMap(topK.toString).stat(\"numUniqCategories\").add(uniqueCategories.size)\n  }\n}\n\nobject ScoredTweetsDiversityStatsSideEffect {\n  private val diversityTopK = Array(5, 15, 30)\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/ScribeScoredCandidatesSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.side_effect\n\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.home_mixer.model.HomeFeatures.AncestorsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.DirectedAtUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.EarlybirdScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FollowedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FromInNetworkSourceFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokIsGoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokIsLowQualityFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokIsNsfwFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokIsOcrFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokIsSpamFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokIsViolentFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokTopCategoryFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RequestJoinIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ServedTypeFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceSignalFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.TweetMixerScoreFeature\nimport com.twitter.home_mixer.model.PredictedScoreFeature\nimport com.twitter.home_mixer.param.HomeMixerFlagName.ScribeScoredCandidatesFlag\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsResponse\nimport com.twitter.home_mixer.product.scored_tweets.param.ScoredTweetsParam.EnableScribeScoredCandidatesParam\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.side_effect.ScribeLogEventSideEffect\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidatePipelines\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.timelines.prediction.features.common.TimelinesSharedFeatures\nimport com.twitter.timelines.timeline_logging.{thriftscala => t}\nimport com.twitter.util.logging.Logging\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Side effect that logs scored candidates from scoring pipelines\n */\n@Singleton\nclass ScribeScoredCandidatesSideEffect @Inject() (\n  @Flag(ScribeScoredCandidatesFlag) enableScribeScoredCandidates: Boolean,\n  eventBusPublisher: EventPublisher[t.ScoredCandidate])\n    extends ScribeLogEventSideEffect[\n      t.ScoredCandidate,\n      ScoredTweetsQuery,\n      ScoredTweetsResponse\n    ]\n    with PipelineResultSideEffect.Conditionally[\n      ScoredTweetsQuery,\n      ScoredTweetsResponse\n    ]\n    with Logging {\n\n  override val identifier: SideEffectIdentifier =\n    SideEffectIdentifier(\"ScribeScoredCandidates\")\n\n  override def onlyIf(\n    query: ScoredTweetsQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: ScoredTweetsResponse\n  ): Boolean = enableScribeScoredCandidates && query.params(EnableScribeScoredCandidatesParam)\n\n  /**\n   * Build the log events from query, selections and response\n   *\n   * @param query               PipelineQuery\n   * @param selectedCandidates  Result after Selectors are executed\n   * @param remainingCandidates Candidates which were not selected\n   * @param droppedCandidates   Candidates dropped during selection\n   * @param response            Result after Unmarshalling\n   *\n   * @return LogEvent in thrift\n   */\n  override def buildLogEvents(\n    query: ScoredTweetsQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: ScoredTweetsResponse\n  ): Seq[t.ScoredCandidate] = {\n    val returned = (selectedCandidates ++ remainingCandidates).map(toThrift(_, query, false))\n    val dropped = droppedCandidates.map(toThrift(_, query, true))\n    returned ++ dropped\n  }\n\n  private def toThrift(\n    candidate: CandidateWithDetails,\n    query: ScoredTweetsQuery,\n    isDropped: Boolean\n  ): t.ScoredCandidate = {\n    val grokMetadata = t.GrokMetadata(\n      isGore = candidate.features.getOrElse(GrokIsGoreFeature, None),\n      isNsfw = candidate.features.getOrElse(GrokIsNsfwFeature, None),\n      isSpam = candidate.features.getOrElse(GrokIsSpamFeature, None),\n      isViolent = candidate.features.getOrElse(GrokIsViolentFeature, None),\n      isLowQuality = candidate.features.getOrElse(GrokIsLowQualityFeature, None),\n      isOcr = candidate.features.getOrElse(GrokIsOcrFeature, None)\n    )\n    t.ScoredCandidate(\n      tweetId = candidate.candidateIdLong,\n      viewerId = query.getOptionalUserId,\n      authorId = candidate.features.getOrElse(AuthorIdFeature, None),\n      traceId = Some(Trace.id.traceId.toLong),\n      requestJoinId = query.features.flatMap(_.getOrElse(RequestJoinIdFeature, None)),\n      score = candidate.features.getOrElse(ScoreFeature, None),\n      suggestType = Some(candidate.features.get(ServedTypeFeature).name),\n      isInNetwork = candidate.features.getTry(FromInNetworkSourceFeature).toOption,\n      inReplyToTweetId = candidate.features.getOrElse(InReplyToTweetIdFeature, None),\n      inReplyToUserId = candidate.features.getOrElse(InReplyToUserIdFeature, None),\n      quotedTweetId = candidate.features.getOrElse(QuotedTweetIdFeature, None),\n      quotedUserId = candidate.features.getOrElse(QuotedUserIdFeature, None),\n      directedAtUserId = candidate.features.getOrElse(DirectedAtUserIdFeature, None),\n      favoritedByUserIds = convertSeqFeature(candidate, FavoritedByUserIdsFeature),\n      followedByUserIds = convertSeqFeature(candidate, FollowedByUserIdsFeature),\n      ancestors = convertSeqFeature(candidate, AncestorsFeature),\n      requestTimeMs = Some(query.queryTime.inMilliseconds),\n      candidatePipelineIdentifier =\n        candidate.features.getTry(CandidatePipelines).toOption.map(_.head.name),\n      earlybirdScore = candidate.features.getOrElse(EarlybirdScoreFeature, None),\n      isDropped = Some(isDropped),\n      servedRequestId = query.features.flatMap(_.getOrElse(ServedIdFeature, None)),\n      sourceTweetId = candidate.features.getOrElse(SourceTweetIdFeature, None),\n      predictionScores =\n        Some(extractPredictionScores(candidate) + extractCandidateSourceScore(candidate)),\n      grokAnnotation = Some(\n        t.GrokAnnotation(\n          category = candidate.features\n            .getOrElse(GrokTopCategoryFeature, None),\n          grokMetadata = Some(grokMetadata)\n        )),\n      sourceSignal = candidate.features.getOrElse(SourceSignalFeature, None).map { signal =>\n        t.SourceSignal(\n          id = Some(signal.id),\n          signalType = signal.signalType,\n          signalEntity = signal.signalEntity,\n          authorId = signal.authorId,\n        )\n      }\n    )\n  }\n\n  private def extractPredictionScores(candidate: CandidateWithDetails): Set[t.PredictionScore] =\n    PredictedScoreFeature.PredictedScoreFeatureSet\n      .map(feature =>\n        t.PredictionScore(Some(feature.featureName), candidate.features.getOrElse(feature, None)))\n      .filter(_.score.isDefined)\n\n  private def extractCandidateSourceScore(candidate: CandidateWithDetails): t.PredictionScore =\n    t.PredictionScore(\n      Some(TimelinesSharedFeatures.CANDIDATE_SOURCE_SCORE.getFeatureName),\n      candidate.features\n        .getOrElse(TweetMixerScoreFeature, None))\n\n  private def convertSeqFeature[T](\n    candidateWithDetails: CandidateWithDetails,\n    feature: Feature[_, Seq[T]]\n  ): Option[Seq[T]] =\n    Option(\n      candidateWithDetails.features\n        .getOrElse(feature, Seq.empty)).filter(_.nonEmpty)\n\n  override val logPipelinePublisher: EventPublisher[t.ScoredCandidate] = eventBusPublisher\n\n  override val alerts = Seq(HomeMixerAlertConfig.BusinessHours.defaultSuccessRateAlert())\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/side_effect/ScribeServedCommonFeaturesAndCandidateFeaturesSideEffect.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.side_effect\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mysql.Client\nimport com.twitter.finagle.mysql.Transactions\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.home_mixer.model.HomeFeatures.ServedRequestIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.param.HomeMixerFlagName.DataRecordMetadataStoreConfigsYmlFlag\nimport com.twitter.home_mixer.param.HomeMixerFlagName.ScribeServedCommonFeaturesAndCandidateFeaturesFlag\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.CandidateFeaturesScribeEventPublisher\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.CommonFeaturesScribeEventPublisher\nimport com.twitter.home_mixer.param.HomeMixerInjectionNames.MinimumFeaturesScribeEventPublisher\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.non_ml_features.NonMLCandidateFeatures\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.non_ml_features.NonMLCandidateFeaturesAdapter\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.non_ml_features.NonMLCommonFeatures\nimport com.twitter.home_mixer.product.scored_tweets.feature_hydrator.adapters.non_ml_features.NonMLCommonFeaturesAdapter\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsQuery\nimport com.twitter.home_mixer.product.scored_tweets.model.ScoredTweetsResponse\nimport com.twitter.home_mixer.product.scored_tweets.scorer.CandidateFeaturesDataRecordFeature\nimport com.twitter.home_mixer.product.scored_tweets.scorer.CommonFeaturesDataRecordFeature\nimport com.twitter.home_mixer.product.scored_tweets.scorer.PredictedScoreFeature.PredictedScoreFeatures\nimport com.twitter.home_mixer.util.CandidatesUtil.getOriginalAuthorId\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.ml.api.DataRecordMerger\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordConverter\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.SpecificFeatures\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.ml.cont_train.common.domain.non_scalding.CandidateAndCommonFeaturesStreamingUtils\nimport com.twitter.timelines.ml.pldr.client.MysqlClientUtils\nimport com.twitter.timelines.ml.pldr.client.VersionedMetadataCacheClient\nimport com.twitter.timelines.ml.pldr.conversion.VersionIdAndFeatures\nimport com.twitter.timelines.suggests.common.data_record_metadata.{thriftscala => drmd}\nimport com.twitter.timelines.suggests.common.poly_data_record.{thriftjava => pldr}\nimport com.twitter.timelines.util.stats.OptionObserver\nimport com.twitter.util.Time\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\n/**\n * (1) Scribe common features sent to prediction service + some other features as PLDR format into logs\n * (2) Scribe candidate features sent to prediction service + some other features as PLDR format into another logs\n */\n@Singleton\nclass ScribeServedCommonFeaturesAndCandidateFeaturesSideEffect @Inject() (\n  @Flag(DataRecordMetadataStoreConfigsYmlFlag) dataRecordMetadataStoreConfigsYml: String,\n  @Flag(ScribeServedCommonFeaturesAndCandidateFeaturesFlag) enableScribeServedCommonFeaturesAndCandidateFeatures: Boolean,\n  @Named(CommonFeaturesScribeEventPublisher) commonFeaturesScribeEventPublisher: EventPublisher[\n    pldr.PolyDataRecord\n  ],\n  @Named(CandidateFeaturesScribeEventPublisher) candidateFeaturesScribeEventPublisher: EventPublisher[\n    pldr.PolyDataRecord\n  ],\n  @Named(MinimumFeaturesScribeEventPublisher) minimumFeaturesScribeEventPublisher: EventPublisher[\n    pldr.PolyDataRecord\n  ],\n  statsReceiver: StatsReceiver,\n) extends PipelineResultSideEffect[ScoredTweetsQuery, ScoredTweetsResponse]\n    with PipelineResultSideEffect.Conditionally[ScoredTweetsQuery, ScoredTweetsResponse]\n    with Logging {\n\n  override val identifier: SideEffectIdentifier =\n    SideEffectIdentifier(\"ScribeServedCommonFeaturesAndCandidateFeatures\")\n\n  private val drMerger = new DataRecordMerger\n  private val postScoringCandidateFeatures = SpecificFeatures(PredictedScoreFeatures)\n  private val postScoringCandidateFeaturesDataRecordAdapter =\n    new DataRecordConverter(postScoringCandidateFeatures)\n\n  private val scopedStatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n  private val metadataFetchFailedCounter = scopedStatsReceiver.counter(\"metadataFetchFailed\")\n  private val commonFeaturesScribeCounter = scopedStatsReceiver.counter(\"commonFeaturesScribe\")\n  private val commonFeaturesPLDROptionObserver =\n    OptionObserver(scopedStatsReceiver.scope(\"commonFeaturesPLDR\"))\n  private val candidateFeaturesScribeCounter =\n    scopedStatsReceiver.counter(\"candidateFeaturesScribe\")\n  private val candidateFeaturesPLDROptionObserver =\n    OptionObserver(scopedStatsReceiver.scope(\"candidateFeaturesPLDR\"))\n  private val minimumFeaturesPLDROptionObserver =\n    OptionObserver(scopedStatsReceiver.scope(\"minimumFeaturesPLDR\"))\n  private val minimumFeaturesScribeCounter =\n    scopedStatsReceiver.counter(\"minimumFeaturesScribe\")\n\n  lazy private val dataRecordMetadataStoreClient: Option[Client with Transactions] =\n    Try {\n      MysqlClientUtils.mysqlClientProvider(\n        MysqlClientUtils.parseConfigFromYaml(dataRecordMetadataStoreConfigsYml))\n    }.onFailure { e => info(s\"Error building MySQL client: $e\") }.toOption\n\n  lazy private val versionedMetadataCacheClientOpt: Option[\n    VersionedMetadataCacheClient[Map[drmd.FeaturesCategory, Option[VersionIdAndFeatures]]]\n  ] =\n    dataRecordMetadataStoreClient.map { mysqlClient =>\n      new VersionedMetadataCacheClient[Map[drmd.FeaturesCategory, Option[VersionIdAndFeatures]]](\n        maximumSize = 1,\n        expireDurationOpt = None,\n        mysqlClient = mysqlClient,\n        transform = CandidateAndCommonFeaturesStreamingUtils.metadataTransformer,\n        statsReceiver = statsReceiver\n      )\n    }\n\n  versionedMetadataCacheClientOpt.foreach { versionedMetadataCacheClient =>\n    versionedMetadataCacheClient\n      .metadataFetchTimerTask(\n        CandidateAndCommonFeaturesStreamingUtils.metadataFetchKey,\n        metadataFetchTimer = DefaultTimer,\n        metadataFetchInterval = 90.seconds,\n        metadataFetchFailedCounter = metadataFetchFailedCounter\n      )\n  }\n\n  override def onlyIf(\n    query: ScoredTweetsQuery,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: ScoredTweetsResponse\n  ): Boolean = enableScribeServedCommonFeaturesAndCandidateFeatures\n\n  override def apply(\n    inputs: PipelineResultSideEffect.Inputs[ScoredTweetsQuery, ScoredTweetsResponse]\n  ): Stitch[Unit] = {\n    Stitch.value {\n      val servedTimestamp: Long = Time.now.inMilliseconds\n      val nonMLCommonFeatures = NonMLCommonFeatures(\n        userId = inputs.query.getRequiredUserId,\n        predictionRequestId =\n          inputs.query.features.flatMap(_.getOrElse(ServedRequestIdFeature, None)),\n        servedTimestamp = servedTimestamp\n      )\n      val nonMLCommonFeaturesDataRecord =\n        NonMLCommonFeaturesAdapter.adaptToDataRecords(nonMLCommonFeatures).asScala.head\n\n      /**\n       * Steps of scribing common features\n       * (1) fetch common features as data record\n       * (2) extract additional feature as data record, e.g. predictionRequestId which is used as join key in downstream jobs\n       * (3) merge two data records above and convert the merged data record to pldr\n       * (4) publish pldr\n       */\n      val commonFeaturesDataRecordOpt =\n        inputs.selectedCandidates.headOption.map(_.features.get(CommonFeaturesDataRecordFeature))\n      val commonFeaturesPLDROpt = commonFeaturesDataRecordOpt.flatMap { commonFeaturesDataRecord =>\n        drMerger.merge(commonFeaturesDataRecord, nonMLCommonFeaturesDataRecord)\n\n        CandidateAndCommonFeaturesStreamingUtils.commonFeaturesToPolyDataRecord(\n          versionedMetadataCacheClientOpt = versionedMetadataCacheClientOpt,\n          commonFeatures = commonFeaturesDataRecord,\n          valueFormat = pldr.PolyDataRecord._Fields.LITE_COMPACT_DATA_RECORD\n        )\n      }\n\n      commonFeaturesPLDROptionObserver(commonFeaturesPLDROpt).foreach { pldr =>\n        commonFeaturesScribeEventPublisher.publish(pldr)\n        commonFeaturesScribeCounter.incr()\n      }\n\n      /**\n       * steps of scribing candidate features\n       * (1) fetch candidate features as data record\n       * (2) extract additional features (mostly non ML features including predicted scores, predictionRequestId, userId, tweetId)\n       * (3) merge data records and convert the merged data record into pldr\n       * (4) publish pldr\n       */\n      inputs.selectedCandidates.foreach { candidate =>\n        val candidateFeaturesDataRecord = candidate.features.get(CandidateFeaturesDataRecordFeature)\n\n        /**\n         * extract predicted scores as data record and merge it into original data record\n         */\n        val postScoringCandidateFeaturesDataRecord =\n          postScoringCandidateFeaturesDataRecordAdapter.toDataRecord(candidate.features)\n        drMerger.merge(candidateFeaturesDataRecord, postScoringCandidateFeaturesDataRecord)\n\n        /**\n         * extract non ML common features as data record and merge it into original data record\n         */\n        drMerger.merge(candidateFeaturesDataRecord, nonMLCommonFeaturesDataRecord)\n\n        /**\n         * extract non ML candidate features as data record and merge it into original data record\n         */\n        val nonMLCandidateFeatures = NonMLCandidateFeatures(\n          tweetId = candidate.candidateIdLong,\n          sourceTweetId = candidate.features.getOrElse(SourceTweetIdFeature, None),\n          originalAuthorId = getOriginalAuthorId(candidate.features)\n        )\n        val nonMLCandidateFeaturesDataRecord =\n          NonMLCandidateFeaturesAdapter.adaptToDataRecords(nonMLCandidateFeatures).asScala.head\n        drMerger.merge(candidateFeaturesDataRecord, nonMLCandidateFeaturesDataRecord)\n\n        val candidateFeaturesPLDROpt =\n          CandidateAndCommonFeaturesStreamingUtils.candidateFeaturesToPolyDataRecord(\n            versionedMetadataCacheClientOpt = versionedMetadataCacheClientOpt,\n            candidateFeatures = candidateFeaturesDataRecord,\n            valueFormat = pldr.PolyDataRecord._Fields.LITE_COMPACT_DATA_RECORD\n          )\n\n        candidateFeaturesPLDROptionObserver(candidateFeaturesPLDROpt).foreach { pldr =>\n          candidateFeaturesScribeEventPublisher.publish(pldr)\n          candidateFeaturesScribeCounter.incr()\n        }\n\n        // scribe minimum features which are used to join labels from client events.\n        val minimumFeaturesPLDROpt = candidateFeaturesPLDROpt\n          .map(CandidateAndCommonFeaturesStreamingUtils.extractMinimumFeaturesFromPldr)\n          .map(pldr.PolyDataRecord.dataRecord)\n        minimumFeaturesPLDROptionObserver(minimumFeaturesPLDROpt).foreach { pldr =>\n          minimumFeaturesScribeEventPublisher.publish(pldr)\n          minimumFeaturesScribeCounter.incr()\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/util/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    dependencies = [\n        \"3rdparty/jvm/com/github/nscala_time\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/embedding\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"src/thrift/com/twitter/timelines/control_ai:timeline-control-ai-thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/scored_tweets/util/ControlAiUtil.scala",
    "content": "package com.twitter.home_mixer.product.scored_tweets.util\n\nimport com.github.nscala_time.time.Imports.LocalDate\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.embedding.TweetTextV8EmbeddingFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.timelines.control_ai.control.{thriftscala => ci}\n\nobject ControlAiUtil {\n  private def isDotProductClose(\n    vector1: Option[Seq[Double]],\n    vector2: Option[Seq[Double]],\n    threshold: Double\n  ): Boolean = {\n    (vector1, vector2) match {\n      case (Some(v1), Some(v2)) =>\n        require(v1.length == v2.length)\n        val dotProduct = v1.zip(v2).map { case (a, b) => a * b }.sum\n        dotProduct > threshold\n      case _ => false\n    }\n  }\n\n  def conditionMatch(\n    action: ci.Action,\n    candidate: CandidateWithFeatures[TweetCandidate],\n    topicMap: Map[String, Seq[Double]],\n    threshold: Double\n  ): Boolean = {\n    val now = LocalDate.now()\n    val currentTime = now.toDate.toInstant.getEpochSecond\n    val currentDayOfWeek = now.getDayOfWeek\n    val currentHourOfDay = now.toDateTimeAtCurrentTime.toDateTime.getHourOfDay\n\n    val conditions: Seq[ci.Condition => Boolean] =\n      Seq(\n        _.postTopic.forall { topic =>\n          isDotProductClose(\n            topicMap.get(topic),\n            candidate.features.getOrElse(TweetTextV8EmbeddingFeature, None),\n            threshold)\n        },\n        _.postLanguage.forall(candidate.features.getOrElse(TweetLanguageFeature, None).contains),\n        _.postHasVideo.forall(_ == candidate.features.getOrElse(HasVideoFeature, false)),\n        _.postHasImage.forall(_ == candidate.features.getOrElse(HasImageFeature, false)),\n        _.postIsReply.forall(\n          _ == candidate.features.getOrElse(InReplyToTweetIdFeature, None).isDefined),\n        _.postIsRetweet.forall(_ == candidate.features.getOrElse(IsRetweetFeature, false)),\n        _.postMaximumAge.forall(maxAge =>\n          SnowflakeId.timeFromIdOpt(candidate.candidate.id).exists(_.untilNow.inMinutes <= maxAge)),\n        _.userFollowsAuthor.forall(_ == candidate.features.get(InNetworkFeature)),\n        _.postLikesCountGreaterThan.forall(count =>\n          candidate.features\n            .getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= count))),\n        _.postLikesCountLessThan.forall(count =>\n          candidate.features\n            .getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ < count))),\n        _.postRetweetsCountGreaterThan.forall(count =>\n          candidate.features\n            .getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ >= count))),\n        _.postRetweetsCountLessThan.forall(count =>\n          candidate.features\n            .getOrElse(EarlybirdFeature, None).exists(_.retweetCountV2.exists(_ < count))),\n        _.postRepliesCountGreaterThan.forall(count =>\n          candidate.features\n            .getOrElse(EarlybirdFeature, None).exists(_.replyCountV2.exists(_ >= count))),\n        _.postRepliesCountLessThan.forall(count =>\n          candidate.features\n            .getOrElse(EarlybirdFeature, None).exists(_.replyCountV2.exists(_ < count))),\n        _.postQuotesCountGreaterThan.forall(count =>\n          candidate.features\n            .getOrElse(EarlybirdFeature, None).exists(_.quoteCount.exists(_ >= count))),\n        _.postQuotesCountLessThan.forall(count =>\n          candidate.features\n            .getOrElse(EarlybirdFeature, None).exists(_.quoteCount.exists(_ < count))),\n        _.postLikedByFollowings.forall(\n          _ == candidate.features.getOrElse(SGSValidLikedByUserIdsFeature, Seq.empty).nonEmpty),\n        _.authorId.forall(aid => candidate.features.getOrElse(AuthorIdFeature, None).contains(aid)),\n        _.authorLanguage.forall(lang =>\n          candidate.features.getOrElse(TweetLanguageFeature, None).contains(lang)),\n        _.authorAccountAgeGreaterThan.forall(count =>\n          candidate.features.getOrElse(AuthorAccountAge, None).exists(_.inDays / 365 >= count)),\n        _.authorAccountAgeLessThan.forall(count =>\n          candidate.features.getOrElse(AuthorAccountAge, None).exists(_.inDays / 365 < count)),\n        _.authorFollowerCountGreaterThan.forall(count =>\n          candidate.features.getOrElse(AuthorFollowersFeature, None).exists(_ >= count)),\n        _.authorFollowerCountLessThan.forall(count =>\n          candidate.features.getOrElse(AuthorFollowersFeature, None).exists(_ < count)),\n        _.authorFollowedByFollowings.forall(\n          _ == candidate.features.getOrElse(SGSValidFollowedByUserIdsFeature, Seq.empty).nonEmpty),\n        _.queryValidStartTime.forall(currentTime >= _),\n        _.queryValidEndTime.forall(currentTime < _),\n        _.queryValidDayOfWeek.forall(_.toInt == currentDayOfWeek),\n        _.queryValidHourOfDay.forall(_.toInt == currentHourOfDay)\n      )\n\n    conditions.forall(_(action.condition))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/candidate_pipeline\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/selector\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/side_effect\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/following/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/param\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/service\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/util\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/earlybird\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/async\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/impressed_tweets\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/location\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/cursor/timelines\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/earlybird\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/java/com/twitter/search/queryparser/query/search:search-query-nodes\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"stitch/stitch-tweetypie\",\n        \"timelinemixer/server/src/main/scala/com/twitter/timelinemixer/injection/model/candidate\",\n    ],\n    exports = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/param\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/SubscribedEarlybirdCandidatePipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.subscribed\n\nimport com.google.inject.Inject\nimport com.twitter.home_mixer.product.subscribed.model.SubscribedQuery\nimport com.twitter.product_mixer.component_library.candidate_source.earlybird.EarlybirdTweetCandidateSource\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSSubscribedUsersFeature\nimport com.twitter.product_mixer.component_library.filter.TweetVisibilityFilter\nimport com.twitter.product_mixer.component_library.gate.NonEmptySeqFeatureGate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.search.earlybird.{thriftscala => t}\nimport com.twitter.spam.rtf.thriftscala.SafetyLevel.TimelineHomeSubscribed\nimport com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}\nimport com.twitter.tweetypie.thriftscala.TweetVisibilityPolicy\n\nclass SubscribedEarlybirdCandidatePipelineConfig @Inject() (\n  earlybirdTweetCandidateSource: EarlybirdTweetCandidateSource,\n  tweetyPieStitchClient: TweetypieStitchClient,\n  subscribedEarlybirdQueryTransformer: SubscribedEarlybirdQueryTransformer)\n    extends CandidatePipelineConfig[\n      SubscribedQuery,\n      t.EarlybirdRequest,\n      t.ThriftSearchResult,\n      TweetCandidate\n    ] {\n  override val identifier: CandidatePipelineIdentifier =\n    CandidatePipelineIdentifier(\"SubscribedEarlybird\")\n\n  override val candidateSource: BaseCandidateSource[t.EarlybirdRequest, t.ThriftSearchResult] =\n    earlybirdTweetCandidateSource\n\n  override val gates: Seq[Gate[SubscribedQuery]] = Seq(\n    NonEmptySeqFeatureGate(SGSSubscribedUsersFeature)\n  )\n\n  override def filters: Seq[Filter[SubscribedQuery, TweetCandidate]] = Seq(\n    new TweetVisibilityFilter(\n      tweetyPieStitchClient,\n      TweetVisibilityPolicy.UserVisible,\n      TimelineHomeSubscribed\n    )\n  )\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    SubscribedQuery,\n    t.EarlybirdRequest\n  ] = subscribedEarlybirdQueryTransformer\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[t.ThriftSearchResult]\n  ] = Seq(SubscribedEarlybirdResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    t.ThriftSearchResult,\n    TweetCandidate\n  ] = { sourceResult => TweetCandidate(id = sourceResult.id) }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/SubscribedEarlybirdQueryTransformer.scala",
    "content": "package com.twitter.home_mixer.product.subscribed\n\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.home_mixer.product.subscribed.model.SubscribedQuery\nimport com.twitter.home_mixer.product.subscribed.param.SubscribedParam.ServerMaxResultsParam\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSSubscribedUsersFeature\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MalformedCursor\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant\nimport com.twitter.search.earlybird.{thriftscala => t}\nimport com.twitter.search.queryparser.query.Conjunction\nimport com.twitter.search.queryparser.query.search.SearchOperator\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class SubscribedEarlybirdQueryTransformer @Inject() (clientId: ClientId)\n    extends CandidatePipelineQueryTransformer[SubscribedQuery, t.EarlybirdRequest] {\n\n  override def transform(query: SubscribedQuery): t.EarlybirdRequest = {\n    val subscribedUserIds =\n      query.features.map(_.get(SGSSubscribedUsersFeature)).getOrElse(Seq.empty)\n\n    val subscribedUsersQuery = new SearchOperator.Builder()\n      .setType(SearchOperator.Type.FILTER)\n      .addOperand(EarlybirdFieldConstant.EXCLUSIVE_FILTER_TERM)\n      .build()\n\n    val searchQuery = query.pipelineCursor\n      .map { cursor =>\n        val sinceIdQuery =\n          (id: Long) => new SearchOperator(SearchOperator.Type.SINCE_ID, id.toString)\n        val maxIdQuery = // max ID is inclusive, so subtract 1\n          (id: Long) => new SearchOperator(SearchOperator.Type.MAX_ID, (id - 1).toString)\n\n        (cursor.cursorType, cursor.id, cursor.gapBoundaryId) match {\n          case (Some(TopCursor), Some(sinceId), _) =>\n            new Conjunction(sinceIdQuery(sinceId), subscribedUsersQuery)\n          case (Some(BottomCursor), Some(maxId), _) =>\n            new Conjunction(maxIdQuery(maxId), subscribedUsersQuery)\n          case (Some(GapCursor), Some(maxId), Some(sinceId)) =>\n            new Conjunction(sinceIdQuery(sinceId), maxIdQuery(maxId), subscribedUsersQuery)\n          case (Some(GapCursor), _, _) =>\n            throw PipelineFailure(MalformedCursor, \"Invalid cursor \" + cursor.toString)\n          case _ => subscribedUsersQuery\n        }\n      }.getOrElse(subscribedUsersQuery)\n\n    t.EarlybirdRequest(\n      searchQuery = t.ThriftSearchQuery(\n        serializedQuery = Some(searchQuery.serialize),\n        fromUserIDFilter64 = Some(subscribedUserIds),\n        numResults = query.requestedMaxResults.getOrElse(query.params(ServerMaxResultsParam)),\n        rankingMode = t.ThriftSearchRankingMode.Recency,\n      ),\n      getOlderResults = Some(true), // needed for archive access to older tweets\n      clientRequestID = Some(s\"${Trace.id.traceId}\"),\n      numResultsToReturnAtRoot = Some(query.params(ServerMaxResultsParam)),\n      clientId = Some(clientId.name),\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/SubscribedEarlybirdResponseFeatureTransformer.scala",
    "content": "package com.twitter.home_mixer.product.subscribed\n\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.search.earlybird.{thriftscala => t}\n\nobject SubscribedEarlybirdResponseFeatureTransformer\n    extends CandidateFeatureTransformer[t.ThriftSearchResult] {\n\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"SubscribedEarlybirdResponse\")\n\n  override val features: Set[Feature[_, _]] = Set(\n    AuthorIdFeature,\n    InReplyToTweetIdFeature,\n    IsRetweetFeature,\n    SourceTweetIdFeature,\n    SourceUserIdFeature,\n  )\n\n  override def transform(candidate: t.ThriftSearchResult): FeatureMap = FeatureMapBuilder()\n    .add(AuthorIdFeature, candidate.tweetypieTweet.flatMap(_.coreData.map(_.userId)))\n    .add(\n      InReplyToTweetIdFeature,\n      candidate.tweetypieTweet.flatMap(_.coreData.flatMap(_.reply.flatMap(_.inReplyToStatusId))))\n    .add(IsRetweetFeature, candidate.metadata.exists(_.isRetweet.contains(true)))\n    .add(SourceTweetIdFeature, candidate.sourceTweetypieTweet.map(_.id))\n    .add(SourceUserIdFeature, candidate.sourceTweetypieTweet.flatMap(_.coreData.map(_.userId)))\n    .build()\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/SubscribedMixerPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.subscribed\n\nimport com.twitter.clientapp.{thriftscala => ca}\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.home_mixer.candidate_pipeline.ConversationServiceCandidatePipelineConfigBuilder\nimport com.twitter.home_mixer.candidate_pipeline.EditedTweetsCandidatePipelineConfig\nimport com.twitter.home_mixer.candidate_pipeline.NewTweetsPillCandidatePipelineConfig\nimport com.twitter.home_mixer.functional_component.feature_hydrator._\nimport com.twitter.home_mixer.functional_component.selector.UpdateHomeClientEventDetails\nimport com.twitter.home_mixer.functional_component.selector.UpdateNewTweetsPillDecoration\nimport com.twitter.home_mixer.functional_component.side_effect._\nimport com.twitter.home_mixer.param.HomeGlobalParams.MaxNumberReplaceInstructionsParam\nimport com.twitter.home_mixer.param.HomeMixerFlagName.ScribeClientEventsFlag\nimport com.twitter.home_mixer.product.following.model.HomeMixerExternalStrings\nimport com.twitter.home_mixer.product.subscribed.model.SubscribedQuery\nimport com.twitter.home_mixer.product.subscribed.param.SubscribedParam.EnablePostContextFeatureHydratorParam\nimport com.twitter.home_mixer.product.subscribed.param.SubscribedParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.util.CandidatesUtil\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.async.AsyncQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.impressed_tweets.ImpressedTweetsQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.location.UserLocationQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSSubscribedUsersQueryFeatureHydrator\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.premarshaller.urt.UrtDomainMarshaller\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.AddEntriesWithReplaceAndShowAlertInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedBottomCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedCursorIdSelector.TweetIdSelector\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedGapCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedTopCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceAllEntries\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ReplaceEntryInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.ShowAlertInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.StaticTimelineScribeConfigBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtMetadataBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.earlybird.EarlybirdGapIncludeInstruction\nimport com.twitter.product_mixer.component_library.selector.DropMaxCandidates\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.component_library.selector.SelectConditionally\nimport com.twitter.product_mixer.component_library.selector.UpdateSortCandidates\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.UrtTransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.MixerPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineConfig\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Provider\nimport javax.inject.Singleton\n\n@Singleton\nclass SubscribedMixerPipelineConfig @Inject() (\n  subscribedEarlybirdCandidatePipelineConfig: SubscribedEarlybirdCandidatePipelineConfig,\n  conversationServiceCandidatePipelineConfigBuilder: ConversationServiceCandidatePipelineConfigBuilder[\n    SubscribedQuery\n  ],\n  editedTweetsCandidatePipelineConfig: EditedTweetsCandidatePipelineConfig,\n  newTweetsPillCandidatePipelineConfig: NewTweetsPillCandidatePipelineConfig[SubscribedQuery],\n  dismissInfoQueryFeatureHydrator: DismissInfoQueryFeatureHydrator,\n  gizmoduckUserQueryFeatureHydrator: GizmoduckUserQueryFeatureHydrator,\n  requestQueryFeatureHydrator: RequestQueryFeatureHydrator[SubscribedQuery],\n  sgsFollowedUsersQueryFeatureHydrator: SGSFollowedUsersQueryFeatureHydrator,\n  sgsSubscribedUsersQueryFeatureHydrator: SGSSubscribedUsersQueryFeatureHydrator,\n  memcacheTweetImpressionsQueryFeatureHydrator: ImpressedTweetsQueryFeatureHydrator,\n  userLocationQueryFeatureHydrator: UserLocationQueryFeatureHydrator,\n  publishClientSentImpressionsEventBusSideEffect: PublishClientSentImpressionsEventBusSideEffect,\n  homeTimelineServedCandidatesSideEffect: HomeScribeServedCandidatesSideEffect,\n  clientEventsScribeEventPublisher: EventPublisher[ca.LogEvent],\n  externalStrings: HomeMixerExternalStrings,\n  @ProductScoped stringCenterProvider: Provider[StringCenter],\n  urtTransportMarshaller: UrtTransportMarshaller,\n  @Flag(ScribeClientEventsFlag) enableScribeClientEvents: Boolean)\n    extends MixerPipelineConfig[SubscribedQuery, Timeline, urt.TimelineResponse] {\n\n  override val identifier: MixerPipelineIdentifier = MixerPipelineIdentifier(\"Subscribed\")\n\n  private val dependentCandidatesStep = MixerPipelineConfig.dependentCandidatePipelinesStep\n  private val resultSelectorsStep = MixerPipelineConfig.resultSelectorsStep\n\n  override val fetchQueryFeatures: Seq[QueryFeatureHydrator[SubscribedQuery]] = Seq(\n    requestQueryFeatureHydrator,\n    sgsFollowedUsersQueryFeatureHydrator,\n    sgsSubscribedUsersQueryFeatureHydrator,\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, dismissInfoQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, gizmoduckUserQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(dependentCandidatesStep, userLocationQueryFeatureHydrator),\n    AsyncQueryFeatureHydrator(resultSelectorsStep, memcacheTweetImpressionsQueryFeatureHydrator)\n  )\n\n  private val earlybirdCandidatePipelineScope =\n    SpecificPipeline(subscribedEarlybirdCandidatePipelineConfig.identifier)\n\n  private val conversationServiceCandidatePipelineConfig =\n    conversationServiceCandidatePipelineConfigBuilder.build(\n      earlybirdCandidatePipelineScope,\n      hmt.ServedType.Subscribed,\n      EnablePostContextFeatureHydratorParam\n    )\n\n  override val candidatePipelines: Seq[CandidatePipelineConfig[SubscribedQuery, _, _, _]] =\n    Seq(subscribedEarlybirdCandidatePipelineConfig)\n\n  override val dependentCandidatePipelines: Seq[\n    DependentCandidatePipelineConfig[SubscribedQuery, _, _, _]\n  ] = Seq(\n    conversationServiceCandidatePipelineConfig,\n    editedTweetsCandidatePipelineConfig,\n    newTweetsPillCandidatePipelineConfig\n  )\n\n  override val failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy] = Map(\n    editedTweetsCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n    newTweetsPillCandidatePipelineConfig.identifier -> FailOpenPolicy.Always,\n  )\n\n  override val resultSelectors: Seq[Selector[SubscribedQuery]] = Seq(\n    UpdateSortCandidates(\n      ordering = CandidatesUtil.reverseChronTweetsOrdering,\n      candidatePipeline = conversationServiceCandidatePipelineConfig.identifier\n    ),\n    DropMaxCandidates(\n      candidatePipeline = editedTweetsCandidatePipelineConfig.identifier,\n      maxSelectionsParam = MaxNumberReplaceInstructionsParam\n    ),\n    DropMaxCandidates(\n      candidatePipeline = conversationServiceCandidatePipelineConfig.identifier,\n      maxSelectionsParam = ServerMaxResultsParam\n    ),\n    InsertAppendResults(candidatePipeline = conversationServiceCandidatePipelineConfig.identifier),\n    // This selector must come after the tweets are inserted into the results\n    UpdateNewTweetsPillDecoration(\n      pipelineScope = SpecificPipelines(\n        conversationServiceCandidatePipelineConfig.identifier,\n        newTweetsPillCandidatePipelineConfig.identifier\n      ),\n      stringCenter = stringCenterProvider.get(),\n      seeNewTweetsString = externalStrings.seeNewTweetsString,\n      tweetedString = externalStrings.tweetedString\n    ),\n    InsertAppendResults(candidatePipeline = editedTweetsCandidatePipelineConfig.identifier),\n    SelectConditionally(\n      selector =\n        InsertAppendResults(candidatePipeline = newTweetsPillCandidatePipelineConfig.identifier),\n      includeSelector = (_, _, results) => CandidatesUtil.containsType[TweetCandidate](results)\n    ),\n    UpdateHomeClientEventDetails(\n      candidatePipelines = Set(conversationServiceCandidatePipelineConfig.identifier)\n    ),\n  )\n\n  private val homeScribeClientEventSideEffect = HomeScribeClientEventSideEffect(\n    enableScribeClientEvents = enableScribeClientEvents,\n    logPipelinePublisher = clientEventsScribeEventPublisher,\n    injectedTweetsCandidatePipelineIdentifiers =\n      Seq(conversationServiceCandidatePipelineConfig.identifier),\n  )\n\n  override val resultSideEffects: Seq[PipelineResultSideEffect[SubscribedQuery, Timeline]] = Seq(\n    homeScribeClientEventSideEffect,\n    homeTimelineServedCandidatesSideEffect,\n    publishClientSentImpressionsEventBusSideEffect\n  )\n\n  override val domainMarshaller: DomainMarshaller[SubscribedQuery, Timeline] = {\n    val instructionBuilders = Seq(\n      ReplaceEntryInstructionBuilder(ReplaceAllEntries),\n      AddEntriesWithReplaceAndShowAlertInstructionBuilder(),\n      ShowAlertInstructionBuilder(),\n    )\n\n    val topCursorBuilder = OrderedTopCursorBuilder(TweetIdSelector)\n    val bottomCursorBuilder =\n      OrderedBottomCursorBuilder(TweetIdSelector, EarlybirdGapIncludeInstruction.inverse())\n    val gapCursorBuilder = OrderedGapCursorBuilder(TweetIdSelector, EarlybirdGapIncludeInstruction)\n\n    val scribeConfigBuilder =\n      StaticTimelineScribeConfigBuilder(TimelineScribeConfig(Some(\"subscribed\"), None, None))\n    val metadataBuilder = UrtMetadataBuilder(scribeConfigBuilder = Some(scribeConfigBuilder))\n\n    UrtDomainMarshaller(\n      instructionBuilders = instructionBuilders,\n      metadataBuilder = Some(metadataBuilder),\n      cursorBuilders = Seq(topCursorBuilder, bottomCursorBuilder, gapCursorBuilder)\n    )\n  }\n\n  override val transportMarshaller: TransportMarshaller[Timeline, urt.TimelineResponse] =\n    urtTransportMarshaller\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/SubscribedProductPipelineConfig.scala",
    "content": "package com.twitter.home_mixer.product.subscribed\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.home_mixer.model.request.HomeMixerRequest\nimport com.twitter.home_mixer.model.request.SubscribedProduct\nimport com.twitter.home_mixer.model.request.SubscribedProductContext\nimport com.twitter.home_mixer.product.subscribed.model.SubscribedQuery\nimport com.twitter.home_mixer.product.subscribed.param.SubscribedParam.ServerMaxResultsParam\nimport com.twitter.home_mixer.service.HomeMixerAccessPolicy.DefaultHomeMixerAccessPolicy\nimport com.twitter.home_mixer.service.HomeMixerAlertConfig.DefaultNotificationGroup\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.timelines.ChronologicalCursorUnmarshaller\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AccessPolicy\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.common.alert.LatencyAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.P99\nimport com.twitter.product_mixer.core.functional_component.common.alert.SuccessRateAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.ThroughputAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfAbove\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfBelow\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfLatencyAbove\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ProductPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineConfig\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.BadRequest\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MalformedCursor\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineConfig\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.product_mixer.core.util.SortIndexBuilder\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport com.twitter.timelines.util.RequestCursorSerializer\nimport com.twitter.util.Time\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SubscribedProductPipelineConfig @Inject() (\n  subscribedMixerPipelineConfig: SubscribedMixerPipelineConfig,\n  subscribedParamConfig: param.SubscribedParamConfig)\n    extends ProductPipelineConfig[HomeMixerRequest, SubscribedQuery, urt.TimelineResponse] {\n\n  override val identifier: ProductPipelineIdentifier = ProductPipelineIdentifier(\"Subscribed\")\n\n  override val product: Product = SubscribedProduct\n  override val paramConfig: ProductParamConfig = subscribedParamConfig\n\n  override def pipelineQueryTransformer(\n    request: HomeMixerRequest,\n    params: Params\n  ): SubscribedQuery = {\n    val context = request.productContext match {\n      case Some(context: SubscribedProductContext) => context\n      case _ => throw PipelineFailure(BadRequest, \"SubscribedProductContext not found\")\n    }\n\n    val debugOptions = request.debugParams.flatMap(_.debugOptions)\n\n    /**\n     * Unlike other clients, newly created tweets on Android have the sort index set to the current\n     * time instead of the top sort index + 1, so these tweets get stuck at the top of the timeline\n     * if subsequent timeline responses use the sort index from the previous response instead of\n     * the current time.\n     */\n    val pipelineCursor = request.serializedRequestCursor.flatMap { cursor =>\n      Try(UrtCursorSerializer.deserializeOrderedCursor(cursor))\n        .getOrElse(ChronologicalCursorUnmarshaller(RequestCursorSerializer.deserialize(cursor)))\n        .map {\n          case UrtOrderedCursor(_, id, Some(GapCursor), gapBoundaryId)\n              if id.isEmpty || gapBoundaryId.isEmpty =>\n            throw PipelineFailure(MalformedCursor, \"Gap Cursor bounds not defined\")\n          case topCursor @ UrtOrderedCursor(_, _, Some(TopCursor), _) =>\n            val queryTime = debugOptions.flatMap(_.requestTimeOverride).getOrElse(Time.now)\n            topCursor.copy(initialSortIndex = SortIndexBuilder.timeToId(queryTime))\n          case cursor => cursor\n        }\n    }\n\n    SubscribedQuery(\n      params = params,\n      clientContext = request.clientContext,\n      features = None,\n      pipelineCursor = pipelineCursor,\n      requestedMaxResults = Some(params(ServerMaxResultsParam)),\n      debugOptions = debugOptions,\n      deviceContext = context.deviceContext,\n      seenTweetIds = context.seenTweetIds\n    )\n  }\n\n  override val pipelines: Seq[PipelineConfig] = Seq(subscribedMixerPipelineConfig)\n\n  override def pipelineSelector(\n    query: SubscribedQuery\n  ): ComponentIdentifier = subscribedMixerPipelineConfig.identifier\n\n  override val alerts: Seq[Alert] = Seq(\n    SuccessRateAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfBelow(99.9, 20, 30),\n      criticalPredicate = TriggerIfBelow(99.9, 30, 30),\n    ),\n    LatencyAlert(\n      notificationGroup = DefaultNotificationGroup,\n      percentile = P99,\n      warnPredicate = TriggerIfLatencyAbove(1100.millis, 15, 30),\n      criticalPredicate = TriggerIfLatencyAbove(1200.millis, 15, 30)\n    ),\n    ThroughputAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfAbove(18000),\n      criticalPredicate = TriggerIfAbove(20000)\n    )\n  )\n\n  override val debugAccessPolicies: Set[AccessPolicy] = DefaultHomeMixerAccessPolicy\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/model/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model/request\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/model/SubscribedQuery.scala",
    "content": "package com.twitter.home_mixer.product.subscribed.model\n\nimport com.twitter.home_mixer.model.request.DeviceContext\nimport com.twitter.home_mixer.model.request.HasDeviceContext\nimport com.twitter.home_mixer.model.request.HasSeenTweetIds\nimport com.twitter.home_mixer.model.request.SubscribedProduct\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.request._\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Params\n\ncase class SubscribedQuery(\n  override val params: Params,\n  override val clientContext: ClientContext,\n  override val pipelineCursor: Option[UrtOrderedCursor],\n  override val requestedMaxResults: Option[Int],\n  override val debugOptions: Option[DebugOptions],\n  override val features: Option[FeatureMap],\n  override val deviceContext: Option[DeviceContext],\n  override val seenTweetIds: Option[Seq[Long]])\n    extends PipelineQuery\n    with HasPipelineCursor[UrtOrderedCursor]\n    with HasDeviceContext\n    with HasSeenTweetIds {\n  override val product: Product = SubscribedProduct\n\n  override def withFeatureMap(features: FeatureMap): SubscribedQuery =\n    copy(features = Some(features))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/param/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param/decider\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/param/SubscribedParam.scala",
    "content": "package com.twitter.home_mixer.product.subscribed.param\n\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSParam\n\nobject SubscribedParam {\n  val SupportedClientFSName = \"subscribed_supported_client\"\n\n  object ServerMaxResultsParam\n      extends FSBoundedParam[Int](\n        name = \"subscribed_server_max_results\",\n        default = 100,\n        min = 1,\n        max = 500\n      )\n\n  object EnablePostContextFeatureHydratorParam\n      extends FSParam[Boolean](\n        name = \"subscribed_enable_post_context_feature_hydrator\",\n        default = false\n      )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/product/subscribed/param/SubscribedParamConfig.scala",
    "content": "package com.twitter.home_mixer.product.subscribed.param\n\nimport com.twitter.home_mixer.param.decider.DeciderKey\nimport com.twitter.home_mixer.product.subscribed.param.SubscribedParam._\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.servo.decider.DeciderKeyName\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SubscribedParamConfig @Inject() () extends ProductParamConfig {\n  override val enabledDeciderKey: DeciderKeyName = DeciderKey.EnableSubscribedProduct\n  override val supportedClientFSName: String = SupportedClientFSName\n\n  override val boundedIntFSOverrides = Seq(\n    ServerMaxResultsParam\n  )\n\n  override val booleanFSOverrides = Seq(\n    EnablePostContextFeatureHydratorParam\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/service/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/service/HomeMixerAccessPolicy.scala",
    "content": "package com.twitter.home_mixer.service\n\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AccessPolicy\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AllowedLdapGroups\n\nobject HomeMixerAccessPolicy {\n\n  /**\n   * Access policies can be configured on a product-by-product basis but you may also want products\n   * to have a common policy.\n   */\n  val DefaultHomeMixerAccessPolicy: Set[AccessPolicy] = Set(AllowedLdapGroups(Set.empty[String]))\n}"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/service/HomeMixerAlertConfig.scala",
    "content": "package com.twitter.home_mixer.service\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.product_mixer.core.functional_component.common.alert.Destination\nimport com.twitter.product_mixer.core.functional_component.common.alert.EmptyResponseRateAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.LatencyAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.NotificationGroup\nimport com.twitter.product_mixer.core.functional_component.common.alert.P99\nimport com.twitter.product_mixer.core.functional_component.common.alert.Percentile\nimport com.twitter.product_mixer.core.functional_component.common.alert.SuccessRateAlert\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfAbove\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfBelow\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfLatencyAbove\nimport com.twitter.util.Duration\n\n/**\n * Notifications (email, pagerduty, etc) can be specific per-alert but it is common for multiple\n * products to share notification configuration.\n */\nobject HomeMixerAlertConfig {\n  val DefaultNotificationGroup: NotificationGroup = NotificationGroup(\n    warn = Destination(emails = Seq(\"\")),\n    critical = Destination(emails = Seq(\"\"))\n  )\n\n  object BusinessHours {\n    val DefaultNotificationGroup: NotificationGroup = NotificationGroup(\n      warn = Destination(emails = Seq(\"\")),\n      critical = Destination(emails = Seq(\"\"))\n    )\n\n    def defaultEmptyResponseRateAlert(warnThreshold: Double = 50, criticalThreshold: Double = 80) =\n      EmptyResponseRateAlert(\n        notificationGroup = DefaultNotificationGroup,\n        warnPredicate = TriggerIfAbove(warnThreshold),\n        criticalPredicate = TriggerIfAbove(criticalThreshold)\n      )\n\n    def defaultSuccessRateAlert(\n      threshold: Double = 99.5,\n      warnDatapointsPastThreshold: Int = 20,\n      criticalDatapointsPastThreshold: Int = 30,\n      duration: Int = 30\n    ) = SuccessRateAlert(\n      notificationGroup = DefaultNotificationGroup,\n      warnPredicate = TriggerIfBelow(threshold, warnDatapointsPastThreshold, duration),\n      criticalPredicate = TriggerIfBelow(threshold, criticalDatapointsPastThreshold, duration),\n    )\n\n    def defaultLatencyAlert(\n      latencyThreshold: Duration = 200.millis,\n      warningDatapointsPastThreshold: Int = 15,\n      criticalDatapointsPastThreshold: Int = 30,\n      duration: Int = 30,\n      percentile: Percentile = P99\n    ): LatencyAlert = LatencyAlert(\n      notificationGroup = DefaultNotificationGroup,\n      percentile = percentile,\n      warnPredicate =\n        TriggerIfLatencyAbove(latencyThreshold, warningDatapointsPastThreshold, duration),\n      criticalPredicate =\n        TriggerIfLatencyAbove(latencyThreshold, criticalDatapointsPastThreshold, duration)\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/service/ScoredTweetsService.scala",
    "content": "package com.twitter.home_mixer.service\n\nimport com.twitter.home_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineRequest\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Params\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.reflect.runtime.universe._\n\n@Singleton\nclass ScoredTweetsService @Inject() (productPipelineRegistry: ProductPipelineRegistry) {\n\n  def getScoredTweetsResponse[RequestType <: Request](\n    request: RequestType,\n    params: Params\n  )(\n    implicit requestTypeTag: TypeTag[RequestType]\n  ): Stitch[t.ScoredTweetsResponse] = productPipelineRegistry\n    .getProductPipeline[RequestType, t.ScoredTweetsResponse](request.product)\n    .process(ProductPipelineRequest(request, params))\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/store/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/bijection:scrooge\",\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"finagle/finagle-memcached/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/offheap\",\n        \"media-understanding/video-summary/thrift/src/main/thrift:thrift-scala\",\n        \"src/java/com/twitter/ml\",\n        \"src/scala/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api/util:datarecord\",\n        \"src/scala/com/twitter/simclusters_v2/summingbird/stores\",\n        \"src/scala/com/twitter/storehaus_internal/manhattan\",\n        \"src/scala/com/twitter/storehaus_internal/memcache\",\n        \"src/scala/com/twitter/storehaus_internal/memcache/config\",\n        \"src/scala/com/twitter/storehaus_internal/offline\",\n        \"src/scala/com/twitter/storehaus_internal/util\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/timelines/realtime_aggregates:thrift-scala\",\n        \"src/thrift/com/twitter/twistly:twistly-scala\",\n        \"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala\",\n        \"stitch/stitch-core\",\n        \"util-internal/util-cache/src/main/java\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/store/MediaClusterId88Store.scala",
    "content": "package com.twitter.home_mixer.store\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MediaClusterId88Store @Inject() (\n  val statsReceiver: StatsReceiver,\n  val serviceIdentifier: ServiceIdentifier)\n    extends MediaClusterIdStoreTrait {\n\n  override val appId: String = \"media_understanding_embeddings_prod\"\n  override val dataset: String = \"media_embedding_twitter_clip_v0_cluster_id_88\"\n  override val memcacheDest: String = \"/s/cache/clip_cluster_id_88\"\n  override val keyPrefix: String = \"\" // Empty key prefix as per config\n  override val storeName: String = \"MediaClusterId88Store\"\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/store/MediaClusterId95Store.scala",
    "content": "package com.twitter.home_mixer.store\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MediaClusterId95Store @Inject() (\n  override val statsReceiver: StatsReceiver,\n  override val serviceIdentifier: ServiceIdentifier)\n    extends MediaClusterIdStoreTrait {\n\n  override def appId: String = \"media_understanding_embeddings_prod\"\n  override def dataset: String = \"media_embedding_twitter_clip_v0_cluster_id_95\"\n  override def memcacheDest: String = \"/s/cache/clip_cluster_id_95\"\n  override def keyPrefix: String = \"\" // Empty key prefix as per config\n  override def storeName: String = \"MediaClusterId95Store\"\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/store/MediaClusterIdStoreTrait.scala",
    "content": "package com.twitter.home_mixer.store\n\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.storage.client.manhattan.bijections.Bijections\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClient\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpointBuilder\nimport com.twitter.storage.client.manhattan.kv.impl.ReadOnlyKeyDescriptor\nimport com.twitter.storage.client.manhattan.kv.impl.ValueDescriptor\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.memcache.MemcacheStore\nimport com.twitter.storehaus_internal.util.ClientName\nimport com.twitter.storehaus_internal.util.ZkEndPoint\nimport com.twitter.bijection.Bijection\n\ntrait MediaClusterIdStoreTrait {\n  val statsReceiver: StatsReceiver\n  val serviceIdentifier: ServiceIdentifier\n\n  // Abstract methods that subclasses must implement\n  def appId: String\n  def dataset: String\n  def memcacheDest: String\n  def keyPrefix: String\n  def storeName: String // For logging\n\n  private val ManhattanDest = \"/s/manhattan/nash.native-thrift\"\n\n  lazy val clusterIdStore: ReadableStore[Long, Long] = {\n    val manhattanStore = createManhattanStore()\n    val cachedStore = createCachedStore(manhattanStore)\n    cachedStore\n  }\n\n  private def createCachedStore(\n    underlyingStore: ReadableStore[Long, Long]\n  ): ReadableStore[Long, Long] = {\n    val memcacheStats = statsReceiver.scope(s\"memcache_${keyPrefix}\")\n    val underlyingCacheClient = MemcacheStore.memcachedClient(\n      name = ClientName(serviceIdentifier.name),\n      dest = ZkEndPoint(memcacheDest),\n      statsReceiver = memcacheStats,\n      serviceIdentifier = serviceIdentifier,\n      timeout = 80.milliseconds\n    )\n\n    val memcacheStore = ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = underlyingStore,\n      cacheClient = underlyingCacheClient,\n      ttl = 3.days, // From config: foundTtl = 3.days\n    )(\n      valueInjection = Bijections.long2ByteArray,\n      statsReceiver = memcacheStats,\n      keyToString = { key: Long =>\n        val cacheKey = if (keyPrefix.isEmpty) key.toString else s\"${keyPrefix}_${key}\"\n        cacheKey\n      }\n    )\n\n    memcacheStore\n  }\n\n  private def createManhattanStore(): ReadableStore[Long, Long] = {\n\n    // Use proper key and value descriptors with correct encodings\n    // Sign flip for NativeEncoding (flip high bit to match server-side ordering)\n    def signFlip(a: Array[Byte]): Array[Byte] = { a(0) = (a(0) ^ 0x80.toByte).toByte; a }\n    val signBijection = Bijection.build[Array[Byte], Array[Byte]](signFlip _)(signFlip _)\n\n    val keyInjection =\n      Bijections.long2ByteArray.andThen(signBijection).andThen(Bijections.byteArray2Buf)\n    val keyDesc = ReadOnlyKeyDescriptor(keyInjection)\n    val datasetKey = keyDesc.withDataset(dataset)\n\n    val valueInjection = Bijections.long2ByteArray.andThen(Bijections.byteArray2Buf)\n    val valueDesc = ValueDescriptor(valueInjection)\n\n    val client = ManhattanKVClient(\n      appId = appId,\n      dest = ManhattanDest,\n      mtlsParams = ManhattanKVClientMtlsParams(serviceIdentifier),\n      label = storeName\n    )\n\n    val endpoint = ManhattanKVEndpointBuilder(client)\n      .maxRetryCount(3)\n      .defaultMaxTimeout(120.milliseconds)\n      .build()\n\n    new ReadableStore[Long, Long] {\n      override def get(key: Long) = {\n        import com.twitter.stitch.Stitch\n        val future = Stitch.run(endpoint.get(datasetKey.withPkey(key), valueDesc))\n        future.map(_.map(_.contents))\n      }\n    }\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/store/RTAMHStore.scala",
    "content": "package com.twitter.home_mixer.store\n\nimport com.twitter.bijection.Injection\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.home_mixer.store.RTAManhattanRealGraphKVDescriptor._\nimport com.twitter.io.Buf\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.stitch.Stitch\nimport com.twitter.storage.client.manhattan.bijections.Bijections\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpoint\nimport com.twitter.storage.client.manhattan.kv.impl.ReadOnlyKeyDescriptor\nimport com.twitter.storage.client.manhattan.kv.impl.ValueDescriptor\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport com.twitter.ml.api.{thriftscala => mlThrift}\nimport com.twitter.timelines.realtime_aggregates.{thriftscala => thrift}\nimport com.twitter.ml.api.util.ScalaToJavaDataRecordConversions._\n\nobject RTAManhattanRealGraphKVDescriptor {\n  val datasetName = \"timelines_real_time_aggregates_0\"\n\n  val keyInjection: Injection[thrift.AggregationKey, Buf] =\n    BinaryScalaCodec(thrift.AggregationKey).andThen(Bijections.byteArray2Buf)\n  val keyDesc = ReadOnlyKeyDescriptor(keyInjection)\n  val datasetKey = keyDesc.withDataset(datasetName)\n  val valueInjection = BinaryScalaCodec(mlThrift.DataRecord).andThen(Bijections.byteArray2Buf)\n  val valueDesc = ValueDescriptor(valueInjection)\n}\n\n/**\n *\n */\nclass RTAMHStore(manhattanKVEndpoint: ManhattanKVEndpoint)\n    extends ReadableStore[thrift.AggregationKey, DataRecord] {\n\n  override def get(key: thrift.AggregationKey): Future[Option[DataRecord]] = Stitch\n    .run(manhattanKVEndpoint.get(datasetKey.withPkey(key), valueDesc))\n    .map(_.map(mhResponse => mhResponse.contents).map(scalaDataRecord2JavaDataRecord))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/store/RealGraphInNetworkScoresStore.scala",
    "content": "package com.twitter.home_mixer.store\n\nimport com.twitter.bijection.Injection\nimport com.twitter.home_mixer.store.ManhattanRealGraphKVDescriptor._\nimport com.twitter.stitch.Stitch\nimport com.twitter.storage.client.manhattan.bijections.Bijections\nimport com.twitter.storage.client.manhattan.bijections.Bijections.BinaryScalaInjection\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpoint\nimport com.twitter.storage.client.manhattan.kv.impl.ReadOnlyKeyDescriptor\nimport com.twitter.storage.client.manhattan.kv.impl.ValueDescriptor\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport com.twitter.wtf.candidate.{thriftscala => wtf}\n\nobject ManhattanRealGraphKVDescriptor {\n  implicit val byteArray2Buf = Bijections.BytesBijection\n\n  val realGraphDatasetName = \"real_graph_scores_in_v1\"\n  val keyInjection = Injection.connect[Long, Array[Byte]].andThen(Bijections.BytesInjection)\n  val keyDesc = ReadOnlyKeyDescriptor(keyInjection)\n  val valueDesc = ValueDescriptor(BinaryScalaInjection(wtf.CandidateSeq))\n  val realGraphDatasetKey = keyDesc.withDataset(realGraphDatasetName)\n}\n\n/**\n * Hydrates real graph in network scores for a viewer\n */\nclass RealGraphInNetworkScoresStore(manhattanKVEndpoint: ManhattanKVEndpoint)\n    extends ReadableStore[Long, Seq[wtf.Candidate]] {\n\n  override def get(viewerId: Long): Future[Option[Seq[wtf.Candidate]]] = Stitch\n    .run(manhattanKVEndpoint.get(realGraphDatasetKey.withPkey(viewerId), valueDesc))\n    .map(_.map(mhResponse => mhResponse.contents.candidates))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/store/TweetWatchTimeMetadataStore.scala",
    "content": "package com.twitter.home_mixer.store\n\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.storage.client.manhattan.bijections.Bijections\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClient\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpointBuilder\nimport com.twitter.storage.client.manhattan.kv.impl.Component\nimport com.twitter.storage.client.manhattan.kv.impl.KeyDescriptor\nimport com.twitter.storage.client.manhattan.kv.impl.ValueDescriptor\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.memcache.MemcacheStore\nimport com.twitter.storehaus_internal.util.ClientName\nimport com.twitter.storehaus_internal.util.ZkEndPoint\nimport com.twitter.twistly.thriftscala.VideoViewEngagementType\nimport com.twitter.twistly.thriftscala.WatchTimeMetadata\nimport com.twitter.bijection.Injection\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.io.Buf\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.util.Try\n\n@Singleton\nclass TweetWatchTimeMetadataStore @Inject() (\n  statsReceiver: StatsReceiver,\n  serviceIdentifier: ServiceIdentifier) {\n\n  private val ManhattanDest = \"/s/manhattan/nash.native-thrift\"\n  private val AppId = \"uss_prod\"\n  private val Dataset = \"video_watch_time_metadata\"\n  private val MemcacheDest = \"/s/cache/tweet_aggregated_watch_time\"\n  private val KeyPrefix = \"\" // Empty key prefix as per config\n\n  lazy val tweetWatchTimeMetadataStore: ReadableStore[\n    (Long, VideoViewEngagementType),\n    WatchTimeMetadata\n  ] = {\n    val manhattanStore = createManhattanStore()\n    val cachedStore = createCachedStore(manhattanStore)\n    cachedStore\n  }\n\n  private def createCachedStore(\n    underlyingStore: ReadableStore[(Long, VideoViewEngagementType), WatchTimeMetadata]\n  ): ReadableStore[(Long, VideoViewEngagementType), WatchTimeMetadata] = {\n    val memcacheStats = statsReceiver.scope(s\"memcache_${KeyPrefix}\")\n    val underlyingCacheClient = MemcacheStore.memcachedClient(\n      name = ClientName(serviceIdentifier.name),\n      dest = ZkEndPoint(MemcacheDest),\n      statsReceiver = memcacheStats,\n      serviceIdentifier = serviceIdentifier,\n      timeout = 80.milliseconds\n    )\n\n    val memcacheStore = ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = underlyingStore,\n      cacheClient = underlyingCacheClient,\n      ttl = 10.minutes,\n    )(\n      valueInjection = BinaryScalaCodec(WatchTimeMetadata),\n      statsReceiver = memcacheStats,\n      keyToString = { key: (Long, VideoViewEngagementType) =>\n        val cacheKey =\n          if (KeyPrefix.isEmpty) s\"${key._1}_${key._2}\" else s\"${KeyPrefix}_${key._1}_${key._2}\"\n        cacheKey\n      }\n    )\n\n    memcacheStore\n  }\n\n  private def createManhattanStore(\n  ): ReadableStore[(Long, VideoViewEngagementType), WatchTimeMetadata] = {\n\n    val pkeyInjection: Injection[Long, Buf] =\n      Bijections.long2ByteArray.andThen(Bijections.BytesBijection)\n\n    val lkeyInjection: Injection[VideoViewEngagementType, Buf] =\n      Injection\n        .build[VideoViewEngagementType, Int](_.value)(i => Try(VideoViewEngagementType(i)))\n        .andThen(Bijections.int2ByteArray)\n        .andThen(Bijections.BytesBijection)\n\n    val keyDesc =\n      KeyDescriptor(Component(pkeyInjection), Component(lkeyInjection)).withDataset(Dataset)\n\n    val valueInjection: Injection[WatchTimeMetadata, Buf] =\n      Bijections.BinaryScalaInjection(WatchTimeMetadata)\n\n    val valueDesc = ValueDescriptor(valueInjection)\n\n    val client = ManhattanKVClient(\n      appId = AppId,\n      dest = ManhattanDest,\n      mtlsParams = ManhattanKVClientMtlsParams(serviceIdentifier),\n      label = \"TweetWatchTimeMetadata\"\n    )\n\n    val endpoint = ManhattanKVEndpointBuilder(client)\n      .maxRetryCount(3)\n      .defaultMaxTimeout(120.milliseconds)\n      .build()\n\n    new ReadableStore[(Long, VideoViewEngagementType), WatchTimeMetadata] {\n      override def get(key: (Long, VideoViewEngagementType)) = {\n        val (tweetId, engType) = key\n        val mhKey = keyDesc.withPkey(tweetId).withLkey(engType)\n        Stitch.run(endpoint.get(mhKey, valueDesc)).map { opt =>\n          opt.map { mv =>\n            mv.contents\n          }\n        }\n      }\n    }\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/store/TwhinEmbeddingsStore.scala",
    "content": "package com.twitter.home_mixer.store\n\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.hermit.store.offheap.Codecs\nimport com.twitter.hermit.store.offheap.OffheapCachedReadableStore\nimport com.twitter.scrooge.ThriftStruct\nimport com.twitter.simclusters_v2.summingbird.stores.ManhattanFromStratoStore\nimport com.twitter.simclusters_v2.thriftscala.PersistentTwhinTweetEmbedding\nimport com.twitter.simclusters_v2.thriftscala.PersistentTwhinUserEmbedding\nimport com.twitter.simclusters_v2.{thriftscala => sim}\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.memcache.MemcacheStore\nimport com.twitter.storehaus_internal.util._\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.offheap.KeyHashFunction\nimport com.twitter.offheap.MemoryWriter\nimport com.twitter.offheap.ObjectCodec\nimport com.twitter.offheap.OffheapReader\nimport com.twitter.simclusters_v2.thriftscala.TwhinTweetEmbedding\n\n@Singleton()\nclass TwhinEmbeddingsStore @Inject() (\n  statsReceiver: StatsReceiver,\n  serviceIdentifier: ServiceIdentifier) {\n\n  val ManhattanNashDest = \"/s/manhattan/nash.native-thrift\"\n  val TwhinEmbeddingsProdAppId = \"twhin_embeddings_prod\"\n  val UserPositiveDataset = \"twhin_user_positive_embeddings\"\n  val RebuildUserPositiveDataset = \"twhin_rebuild_user_rt_pos_emb\"\n  val UserNegativeDataset = \"twhin_user_negative_embeddings\"\n  val TweetDataset = \"twhin_tweet_embeddings\"\n  val TweetRebuildDataset = \"twhin_rebuild_tweet_rt_emb\"\n  val VideoDataset = \"twhin_video_embeddings\"\n\n  val MemcacheTweetDest = \"/s/cache/twhin_embeddings\"\n  val MemcacheVideoDest = \"/s/cache/twhin_video_embeddings\"\n  val KeyPrefixTweet = \"twhin_tweets\"\n  val KeyPrefixTweetRebuild = \"twhin_tweets_rebuild\"\n  val KeyPrefixVideo = \"twhin_videos\"\n  val InMemoryCachePrefix = \"in_memory_cache\"\n\n  val MinEngagementCount = 16\n  val IsProdEnv = serviceIdentifier.environment == \"prod\"\n\n  /**\n   * We do not generate the tweet or video embedding if the number of recent engagements\n   * is < `MinEngagementCount`. This is based on prior Simcluster embedding aggregation\n   * experience and in order to be consistent with the Strato column\n   * strato/config/columns/recommendations/twhin/CachedTwhinTweetEmbeddings.Tweet.strato\n   */\n  private def normalizeByCount(\n    persistentEmbedding: sim.PersistentTwhinTweetEmbedding\n  ): sim.TwhinTweetEmbedding = {\n    val embedding = persistentEmbedding.embedding.embedding\n    val updatedEmbedding =\n      if (persistentEmbedding.updatedCount < MinEngagementCount) embedding.map(_ => 0.0)\n      else embedding.map(_ / persistentEmbedding.updatedCount)\n    sim.TwhinTweetEmbedding(updatedEmbedding)\n  }\n\n  private def createManhattanStore[T <: ThriftStruct: Manifest](\n    dataset: String\n  ): ReadableStore[Long, T] = {\n    ManhattanFromStratoStore\n      .createPersistentTwhinStore[T](\n        dataset = dataset,\n        mhMtlsParams = ManhattanKVClientMtlsParams(serviceIdentifier),\n        statsReceiver = statsReceiver,\n        appId = TwhinEmbeddingsProdAppId,\n        dest = ManhattanNashDest\n      ).composeKeyMapping((_, 0L))\n  }\n\n  private def createManhattanVersionedStore[T <: ThriftStruct: Manifest](\n    dataset: String\n  ): ReadableStore[(Long, Long), T] = {\n    ManhattanFromStratoStore\n      .createPersistentTwhinStore[T](\n        dataset = dataset,\n        mhMtlsParams = ManhattanKVClientMtlsParams(serviceIdentifier),\n        statsReceiver = statsReceiver,\n        appId = TwhinEmbeddingsProdAppId,\n        dest = ManhattanNashDest\n      ).composeKeyMapping[(Long, Long)](key => key)\n  }\n\n  private def createCachedStore[K](\n    underlyingStore: ReadableStore[K, sim.TwhinTweetEmbedding],\n    cacheDest: String,\n    keyPrefix: String,\n    keyCodec: ObjectCodec[K],\n    keyHashFunction: KeyHashFunction[K]\n  ): ReadableStore[K, sim.TwhinTweetEmbedding] = {\n    val scopedStatsReceiver = statsReceiver.scope(keyPrefix)\n    val underlyingCacheClient = MemcacheStore.memcachedClient(\n      name = ClientName(keyPrefix),\n      dest = ZkEndPoint(cacheDest),\n      statsReceiver = scopedStatsReceiver,\n      serviceIdentifier = serviceIdentifier,\n      timeout = 80.milliseconds\n    )\n\n    val memcacheStore = ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = underlyingStore,\n      cacheClient = underlyingCacheClient,\n      ttl = 15.minutes,\n      asyncUpdate = IsProdEnv\n    )(\n      valueInjection = BinaryScalaCodec(sim.TwhinTweetEmbedding),\n      statsReceiver = scopedStatsReceiver,\n      keyToString = { key: K => s\"${keyPrefix}_${key}\" }\n    )\n\n    OffheapCachedReadableStore.fromCache(\n      memcacheStore,\n      tableSize = 256 * 1024,\n      capacity = 256 * 1024 * 2048,\n      keyHashFunction = keyHashFunction,\n      keyCodec = keyCodec,\n      valueCodec = TwhinEmbeddingsStore.TwhinTweetEmbeddingCodec,\n      ttl = 1.minutes,\n      statsReceiver = scopedStatsReceiver\n    )\n  }\n\n  val mhUserPositiveStore: ReadableStore[Long, sim.TwhinTweetEmbedding] =\n    createManhattanStore[PersistentTwhinUserEmbedding](UserPositiveDataset).mapValues(_.embedding)\n\n  val mhRebuildUserPositiveStore: ReadableStore[(Long, Long), sim.TwhinTweetEmbedding] =\n    createManhattanVersionedStore[PersistentTwhinUserEmbedding](RebuildUserPositiveDataset)\n      .mapValues(_.embedding)\n\n  val mhUserNegativeStore: ReadableStore[Long, sim.TwhinTweetEmbedding] =\n    createManhattanStore[PersistentTwhinUserEmbedding](UserNegativeDataset).mapValues(_.embedding)\n\n  val mhTweetStore: ReadableStore[Long, sim.TwhinTweetEmbedding] =\n    createManhattanStore[PersistentTwhinTweetEmbedding](TweetDataset).mapValues(normalizeByCount)\n\n  val mhVideoStore: ReadableStore[Long, sim.TwhinTweetEmbedding] =\n    createManhattanStore[PersistentTwhinTweetEmbedding](VideoDataset).mapValues(normalizeByCount)\n\n  val cachedTweetStore: ReadableStore[Long, TwhinTweetEmbedding] = createCachedStore(\n    mhTweetStore,\n    MemcacheTweetDest,\n    KeyPrefixTweet,\n    Codecs.LongKeyCodec,\n    key => java.lang.Long.hashCode(key)\n  )\n\n  val mhTweetRebuildStore: ReadableStore[(Long, Long), sim.TwhinTweetEmbedding] =\n    createManhattanVersionedStore[PersistentTwhinTweetEmbedding](TweetRebuildDataset)\n      .mapValues(normalizeByCount)\n\n  val cachedTweetRebuildStore: ReadableStore[(Long, Long), TwhinTweetEmbedding] =\n    createCachedStore[(Long, Long)](\n      mhTweetRebuildStore,\n      MemcacheTweetDest,\n      KeyPrefixTweetRebuild,\n      Codecs.LongTupleKeyCodec,\n      key => key.hashCode()\n    )\n\n  val cachedVideoStore: ReadableStore[Long, TwhinTweetEmbedding] = createCachedStore(\n    mhVideoStore,\n    MemcacheVideoDest,\n    KeyPrefixVideo,\n    Codecs.LongKeyCodec,\n    key => java.lang.Long.hashCode(key)\n  )\n}\n\nobject TwhinEmbeddingsStore {\n  object TwhinTweetEmbeddingCodec extends ObjectCodec[sim.TwhinTweetEmbedding] {\n    override def size(decoded: sim.TwhinTweetEmbedding): Int = {\n      Codecs.SeqDoubleCodec.size(decoded.embedding)\n    }\n\n    override def encode(decoded: sim.TwhinTweetEmbedding, writer: MemoryWriter): Unit = {\n      Codecs.SeqDoubleCodec.encode(decoded.embedding, writer)\n    }\n\n    override def decode(reader: OffheapReader): sim.TwhinTweetEmbedding = {\n\n      sim.TwhinTweetEmbedding(\n        embedding = Codecs.SeqDoubleCodec.decode(reader)\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/store/VideoEmbeddingMHStore.scala",
    "content": "package com.twitter.home_mixer.store\n\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.storage.client.manhattan.bijections.Bijections\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClient\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpointBuilder\nimport com.twitter.storage.client.manhattan.kv.impl.ReadOnlyKeyDescriptor\nimport com.twitter.storage.client.manhattan.kv.impl.ValueDescriptor\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.memcache.MemcacheStore\nimport com.twitter.storehaus_internal.util.ClientName\nimport com.twitter.storehaus_internal.util.ZkEndPoint\nimport com.twitter.media_understanding.video_summary.thriftscala.VideoEmbedding\nimport com.twitter.bijection.Injection\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.io.Buf\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass VideoEmbeddingMHStore @Inject() (\n  statsReceiver: StatsReceiver,\n  serviceIdentifier: ServiceIdentifier) {\n\n  private val ManhattanDest = \"/s/manhattan/nash.native-thrift\"\n  private val AppId = \"media_understanding_video_summary\"\n  private val Dataset = \"video_summary_embedding\"\n  private val MemcacheDest = \"/s/cache/video_summary_embedding\"\n  private val KeyPrefix = \"summary_embedding\"\n\n  lazy val videoEmbeddingMHStore: ReadableStore[Long, VideoEmbedding] = {\n    val manhattanStore = createManhattanStore()\n    val cachedStore = createCachedStore(manhattanStore)\n    cachedStore\n  }\n\n  private def createCachedStore(\n    underlyingStore: ReadableStore[Long, VideoEmbedding]\n  ): ReadableStore[Long, VideoEmbedding] = {\n    val memcacheStats = statsReceiver.scope(s\"memcache_${KeyPrefix}\")\n    val underlyingCacheClient = MemcacheStore.memcachedClient(\n      name = ClientName(serviceIdentifier.name),\n      dest = ZkEndPoint(MemcacheDest),\n      statsReceiver = memcacheStats,\n      serviceIdentifier = serviceIdentifier,\n      timeout = 80.milliseconds\n    )\n\n    val memcacheStore = ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = underlyingStore,\n      cacheClient = underlyingCacheClient,\n      ttl = 7.days, // foundTtl from config\n    )(\n      valueInjection = BinaryScalaCodec(VideoEmbedding),\n      statsReceiver = memcacheStats,\n      keyToString = { key: Long =>\n        s\"${KeyPrefix}_${key}\"\n      }\n    )\n\n    memcacheStore\n  }\n\n  private def createManhattanStore(): ReadableStore[Long, VideoEmbedding] = {\n\n    val keyInjection: Injection[Long, Buf] =\n      Bijections.long2ByteArray.andThen(Bijections.BytesBijection)\n\n    val keyDesc = ReadOnlyKeyDescriptor(keyInjection)\n    val datasetKey = keyDesc.withDataset(Dataset)\n\n    val valueInjection: Injection[VideoEmbedding, Buf] =\n      Bijections.BinaryScalaInjection(VideoEmbedding)\n\n    val valueDesc = ValueDescriptor(valueInjection)\n\n    val client = ManhattanKVClient(\n      appId = AppId,\n      dest = ManhattanDest,\n      mtlsParams = ManhattanKVClientMtlsParams(serviceIdentifier),\n      label = \"VideoEmbeddingMH\"\n    )\n\n    val endpoint = ManhattanKVEndpointBuilder(client)\n      .maxRetryCount(3)\n      .defaultMaxTimeout(120.milliseconds)\n      .build()\n\n    new ReadableStore[Long, VideoEmbedding] {\n      override def get(key: Long) = {\n        val mhKey = datasetKey.withPkey(key)\n        Stitch.run(endpoint.get(mhKey, valueDesc)).map { opt =>\n          opt.map { mv =>\n            mv.contents\n          }\n        }\n      }\n    }\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/io/grpc:grpc-protobuf\",\n        \"3rdparty/jvm/io/grpc:grpc-stub\",\n        \"finagle-internal/finagle-grpc/src/main/scala\",\n        \"finatra/inject/inject-utils/src/main/scala\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/param\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/impressed_tweets\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/social_graph\",\n        \"servo/repo/src/main/scala\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/search/common/util/lang\",\n        \"src/scala/com/twitter/ml/api/util\",\n        \"src/thrift/com/twitter/search/common:constants-java\",\n        \"src/thrift/com/twitter/service/metastore/gen:thrift-scala\",\n        \"storage/clients/manhattan/client/src/main/scala\",\n        \"strato/config/src/thrift/com/twitter/strato/columns/content_understanding:content_understanding-scala\",\n        \"user-signal-service/thrift/src/main/thrift:thrift-scala\",\n        \"user_history_transformer/service/src/main/java/com/x/user_action_sequence\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/CachedScoredTweetsHelper.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.home_mixer.model.HomeFeatures.CachedScoredTweetsFeature\nimport com.twitter.home_mixer.{thriftscala => hmt}\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.impressed_tweets.ImpressedTweets\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.util.Time\n\nobject CachedScoredTweetsHelper {\n\n  def tweetImpressionsAndCachedScoredTweets(\n    features: FeatureMap,\n    candidatePipelineIdentifier: CandidatePipelineIdentifier\n  ): Seq[Long] = {\n    val tweetImpressions = features.getOrElse(ImpressedTweets, Seq.empty).toSet\n    val cachedScoredTweets = features\n      .getOrElse(CachedScoredTweetsFeature, Seq.empty)\n      .filter { tweet =>\n        tweet.candidatePipelineIdentifier.exists(\n          CandidatePipelineIdentifier(_).equals(candidatePipelineIdentifier))\n      }.map(_.tweetId)\n\n    (tweetImpressions ++ cachedScoredTweets).toSeq\n  }\n\n  def tweetImpressionsAndCachedScoredTweetsInRange(\n    features: FeatureMap,\n    candidatePipelineIdentifier: CandidatePipelineIdentifier,\n    maxNumImpressions: Int,\n    sinceTime: Time,\n    untilTime: Time\n  ): Seq[Long] =\n    tweetImpressionsAndCachedScoredTweets(features, candidatePipelineIdentifier)\n      .filter { tweetId => SnowflakeId.isSnowflakeId(tweetId) }\n      .filter { tweetId =>\n        val creationTime = SnowflakeId.timeFromId(tweetId)\n        sinceTime <= creationTime && untilTime >= creationTime\n      }.take(maxNumImpressions)\n\n  def unseenCachedScoredTweets(\n    features: FeatureMap\n  ): Seq[hmt.ScoredTweet] = {\n    val seenTweetIds = features.getOrElse(ImpressedTweets, Seq.empty).toSet\n\n    features\n      .getOrElse(CachedScoredTweetsFeature, Seq.empty)\n      .filter(tweet => !seenTweetIds.contains(tweet.tweetId))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/CandidatesUtil.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.escherbird.common.thriftscala.QualifiedId\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.FavoritedByUserIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.GrokVideoMetadataFeature\nimport com.twitter.home_mixer.model.HomeFeatures.HasImageFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.IsRetweetFeature\nimport com.twitter.home_mixer.model.HomeFeatures.MediaUnderstandingAnnotationIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RepliedByEngagerIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.RetweetedByEngagerIdsFeature\nimport com.twitter.home_mixer.model.HomeFeatures.ScoreFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.VideoAspectRatioFeature\nimport com.twitter.product_mixer.component_library.model.candidate.CursorCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.UnexpectedCandidateResult\nimport scala.reflect.ClassTag\n\nobject CandidatesUtil {\n\n  def getItemCandidates(candidates: Seq[CandidateWithDetails]): Seq[ItemCandidateWithDetails] = {\n    candidates.collect {\n      case item: ItemCandidateWithDetails if !item.isCandidateType[CursorCandidate] => Seq(item)\n      case module: ModuleCandidateWithDetails => module.candidates\n    }.flatten\n  }\n\n  def getItemCandidatesWithOnlyModuleLast(\n    candidates: Seq[CandidateWithDetails]\n  ): Seq[ItemCandidateWithDetails] = {\n    candidates.collect {\n      case item: ItemCandidateWithDetails if !item.isCandidateType[CursorCandidate] => item\n      case module: ModuleCandidateWithDetails => module.candidates.last\n    }\n  }\n\n  def containsType[CandidateType <: UniversalNoun[_]](\n    candidates: Seq[CandidateWithDetails]\n  )(\n    implicit tag: ClassTag[CandidateType]\n  ): Boolean = candidates.exists {\n    case ItemCandidateWithDetails(_: CandidateType, _, _) => true\n    case module: ModuleCandidateWithDetails =>\n      module.candidates.head.isCandidateType[CandidateType]()\n    case _ => false\n  }\n\n  def getOriginalTweetId(candidate: CandidateWithFeatures[TweetCandidate]): Long = {\n    if (candidate.features.getOrElse(IsRetweetFeature, false))\n      candidate.features.getOrElse(SourceTweetIdFeature, None).getOrElse(candidate.candidate.id)\n    else candidate.candidate.id\n  }\n\n  def getOriginalTweetId(candidate: TweetCandidate, features: FeatureMap): Long = {\n    if (features.getOrElse(IsRetweetFeature, false))\n      features.getOrElse(SourceTweetIdFeature, None).getOrElse(candidate.id)\n    else candidate.id\n  }\n\n  def getOriginalTweetId(candidate: CandidateWithDetails): Long = {\n    candidate match {\n      case ItemCandidateWithDetails(candidate: TweetCandidate, _, feautres) =>\n        getOriginalTweetId(candidate, feautres)\n      case _ =>\n        throw PipelineFailure(UnexpectedCandidateResult, \"Invalid candidate type\")\n    }\n  }\n\n  def getOriginalTweetId(candidateId: Long, features: FeatureMap): Long = {\n    if (features.getOrElse(IsRetweetFeature, false))\n      features.getOrElse(SourceTweetIdFeature, None).getOrElse(candidateId)\n    else candidateId\n  }\n\n  def getOriginalAuthorId(candidateFeatures: FeatureMap): Option[Long] =\n    if (candidateFeatures.getOrElse(IsRetweetFeature, false))\n      candidateFeatures.getOrElse(SourceUserIdFeature, None)\n    else candidateFeatures.getOrElse(AuthorIdFeature, None)\n\n  def isOriginalTweet(candidate: CandidateWithFeatures[TweetCandidate]): Boolean =\n    !candidate.features.getOrElse(IsRetweetFeature, false) &&\n      candidate.features.getOrElse(InReplyToTweetIdFeature, None).isEmpty\n\n  def isOriginalTweet(candidate: CandidateWithDetails): Boolean =\n    !candidate.features.getOrElse(IsRetweetFeature, false) &&\n      candidate.features.getOrElse(InReplyToTweetIdFeature, None).isEmpty\n\n  def getEngagerUserIds(\n    candidateFeatures: FeatureMap\n  ): Seq[Long] = {\n    candidateFeatures.getOrElse(FavoritedByUserIdsFeature, Seq.empty) ++\n      candidateFeatures.getOrElse(RetweetedByEngagerIdsFeature, Seq.empty) ++\n      candidateFeatures.getOrElse(RepliedByEngagerIdsFeature, Seq.empty)\n  }\n\n  def getMediaUnderstandingAnnotationIds(\n    candidateFeatures: FeatureMap\n  ): Seq[Long] = {\n    if (candidateFeatures.get(HasImageFeature))\n      candidateFeatures.getOrElse(MediaUnderstandingAnnotationIdsFeature, Seq.empty)\n    else Seq.empty\n  }\n\n  def getTweetIdAndSourceId(candidate: CandidateWithFeatures[TweetCandidate]): Seq[Long] =\n    Seq(candidate.candidate.id) ++ candidate.features.getOrElse(SourceTweetIdFeature, None)\n\n  def isAuthoredByViewer(query: PipelineQuery, candidateFeatures: FeatureMap): Boolean =\n    candidateFeatures.getOrElse(AuthorIdFeature, None).contains(query.getRequiredUserId) ||\n      (candidateFeatures.getOrElse(IsRetweetFeature, false) &&\n        candidateFeatures.getOrElse(SourceUserIdFeature, None).contains(query.getRequiredUserId))\n\n  def getCandidateTopicAndAspectRatio(\n    candidate: CandidateWithDetails\n  ): (Option[QualifiedId], Boolean) = {\n    val video = candidate.features.getOrElse(GrokVideoMetadataFeature, None)\n    val optionalQualifiedId = video.flatMap {\n      _.entities.map { entities =>\n        entities.maxBy(entity => entity.score.getOrElse(0.0))._1\n      }\n    }\n    val aspectRatio = candidate.features.getOrElse(VideoAspectRatioFeature, None).exists(_ > 1.0)\n    (optionalQualifiedId, aspectRatio)\n  }\n\n  val reverseChronTweetsOrdering: Ordering[CandidateWithDetails] =\n    Ordering.by[CandidateWithDetails, Long] {\n      case ItemCandidateWithDetails(candidate: TweetCandidate, _, _) => -candidate.id\n      case ModuleCandidateWithDetails(candidates, _, _) if candidates.nonEmpty =>\n        -candidates.last.candidateIdLong\n      case _ => throw PipelineFailure(UnexpectedCandidateResult, \"Invalid candidate type\")\n    }\n\n  val scoreOrdering: Ordering[CandidateWithDetails] = Ordering.by[CandidateWithDetails, Double] {\n    case ItemCandidateWithDetails(_, _, features) =>\n      -features.getOrElse(ScoreFeature, None).getOrElse(0.0)\n    case ModuleCandidateWithDetails(candidates, _, _) =>\n      -candidates.last.features.getOrElse(ScoreFeature, None).getOrElse(0.0)\n    case _ => throw PipelineFailure(UnexpectedCandidateResult, \"Invalid candidate type\")\n  }\n\n  val conversationModuleTweetsOrdering: Ordering[CandidateWithDetails] =\n    Ordering.by[CandidateWithDetails, Long] {\n      case ItemCandidateWithDetails(candidate: TweetCandidate, _, _) => candidate.id\n      case _ => throw PipelineFailure(UnexpectedCandidateResult, \"Only Item candidate expected\")\n    }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/DataRecordUtil.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.util.SRichDataRecord\nimport com.twitter.ml.api.Feature\nimport java.lang.{Double => JDouble}\n\nobject DataRecordUtil {\n  def applyRename(\n    dataRecord: DataRecord,\n    featureContext: FeatureContext,\n    renamedFeatureContext: FeatureContext,\n    featureRenamingMap: Map[Feature[_], Feature[_]]\n  ): DataRecord = {\n    val richFullDr = new SRichDataRecord(dataRecord, featureContext)\n    val richNewDr = new SRichDataRecord(new DataRecord, renamedFeatureContext)\n    val featureIterator = featureContext.iterator()\n    featureIterator.forEachRemaining { feature =>\n      if (richFullDr.hasFeature(feature)) {\n        val renamedFeature = featureRenamingMap.getOrElse(feature, feature)\n\n        val typedFeature = feature.asInstanceOf[Feature[JDouble]]\n        val typedRenamedFeature = renamedFeature.asInstanceOf[Feature[JDouble]]\n\n        richNewDr.setFeatureValue(typedRenamedFeature, richFullDr.getFeatureValue(typedFeature))\n      }\n    }\n    richNewDr.getRecord\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/InjectionTransformer.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.bijection.Injection\nimport com.twitter.io.Buf\nimport com.twitter.servo.util.Transformer\nimport com.twitter.storage.client.manhattan.bijections.Bijections\nimport com.twitter.util.Return\nimport com.twitter.util.Try\nimport java.nio.ByteBuffer\n\nobject InjectionTransformerImplicits {\n  implicit class ByteArrayInjectionToByteBufferTransformer[A](baInj: Injection[A, Array[Byte]]) {\n\n    private val bbInj: Injection[A, ByteBuffer] = baInj\n      .andThen(Bijections.byteArray2Buf)\n      .andThen(Bijections.byteBuffer2Buf.inverse)\n\n    def toByteBufferTransformer(): Transformer[A, ByteBuffer] = new InjectionTransformer(bbInj)\n    def toByteArrayTransformer(): Transformer[A, Array[Byte]] = new InjectionTransformer(baInj)\n  }\n\n  implicit class BufInjectionToByteBufferTransformer[A](bufInj: Injection[A, Buf]) {\n\n    private val bbInj: Injection[A, ByteBuffer] = bufInj.andThen(Bijections.byteBuffer2Buf.inverse)\n    private val baInj: Injection[A, Array[Byte]] = bufInj.andThen(Bijections.byteArray2Buf.inverse)\n\n    def toByteBufferTransformer(): Transformer[A, ByteBuffer] = new InjectionTransformer(bbInj)\n    def toByteArrayTransformer(): Transformer[A, Array[Byte]] = new InjectionTransformer(baInj)\n  }\n\n  implicit class ByteBufferInjectionToByteBufferTransformer[A](bbInj: Injection[A, ByteBuffer]) {\n\n    private val baInj: Injection[A, Array[Byte]] = bbInj.andThen(Bijections.bb2ba)\n\n    def toByteBufferTransformer(): Transformer[A, ByteBuffer] = new InjectionTransformer(bbInj)\n    def toByteArrayTransformer(): Transformer[A, Array[Byte]] = new InjectionTransformer(baInj)\n  }\n}\n\nclass InjectionTransformer[A, B](inj: Injection[A, B]) extends Transformer[A, B] {\n  override def to(a: A): Try[B] = Return(inj(a))\n  override def from(b: B): Try[A] = Try.fromScala(inj.invert(b))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/LanguageCode.scala",
    "content": "package com.twitter.home_mixer.util\n\nobject LanguageCode {\n  final val Japanese = \"jp\"\n  final val English = \"en\"\n  final val Unknown = \"zxx\"\n\n  val AllowedLanguageCodes = Set(\n    \"art\", // Emojis\n    \"qam\", // Mentions\n    \"qct\", // Cashtag\n    \"qht\", // Hashtag\n    \"qme\", // Multiple entities\n    \"qst\", // Short text\n    \"und\", // Undefined\n    \"zxx\" // Links\n  )\n\n  val languageToISO: Map[String, String] = Map(\n    \"arabic\" -> \"ar\",\n    \"danish\" -> \"da\",\n    \"german\" -> \"de\",\n    \"greek\" -> \"el\",\n    \"english\" -> \"en\",\n    \"esperanto\" -> \"eo\",\n    \"spanish\" -> \"es\",\n    \"persian\" -> \"fa\",\n    \"finnish\" -> \"fi\",\n    \"french\" -> \"fr\",\n    \"hebrew\" -> \"he\",\n    \"hungarian\" -> \"hu\",\n    \"indonesian\" -> \"id\",\n    \"icelandic\" -> \"is\",\n    \"italian\" -> \"it\",\n    \"japanese\" -> \"ja\",\n    \"korean\" -> \"ko\",\n    \"lithuanian\" -> \"lt\",\n    \"dutch\" -> \"nl\",\n    \"norwegian\" -> \"no\",\n    \"polish\" -> \"pl\",\n    \"portuguese\" -> \"pt\",\n    \"russian\" -> \"ru\",\n    \"swedish\" -> \"sv\",\n    \"thai\" -> \"th\",\n    \"urdu\" -> \"ur\",\n    \"chinese\" -> \"zh\",\n    \"turkish\" -> \"tr\",\n    \"tagalog\" -> \"tl\",\n    \"hindi\" -> \"hi\",\n    \"malay\" -> \"ms\",\n    \"amharic\" -> \"am\",\n    \"bengali\" -> \"bn\",\n    \"tibetan\" -> \"bo\",\n    \"dhivehi\" -> \"dv\",\n    \"gujarati\" -> \"gu\",\n    \"armenian\" -> \"hy\",\n    \"inuktitut\" -> \"iu\",\n    \"georgian\" -> \"ka\",\n    \"khmer\" -> \"km\",\n    \"kannada\" -> \"kn\",\n    \"lao\" -> \"lo\",\n    \"malayalam\" -> \"ml\",\n    \"myanmar\" -> \"my\",\n    \"oriya\" -> \"or\",\n    \"panjabi\" -> \"pa\",\n    \"sinhala\" -> \"si\",\n    \"tamil\" -> \"ta\",\n    \"telugu\" -> \"te\",\n    \"vietnamese\" -> \"vi\",\n    \"bulgarian\" -> \"bg\",\n    \"nepali\" -> \"ne\",\n    \"estonian\" -> \"et\",\n    \"haitian\" -> \"ht\",\n    \"latvian\" -> \"lv\",\n    \"slovak\" -> \"sk\",\n    \"slovenian\" -> \"sl\",\n    \"ukrainian\" -> \"uk\",\n    \"basque\" -> \"eu\",\n    \"bosnian\" -> \"bs\",\n    \"catalan\" -> \"ca\",\n    \"croatian\" -> \"hr\",\n    \"czech\" -> \"cs\",\n    \"hindi latin\" -> \"hi\",\n    \"marathi\" -> \"mr\",\n    \"pashto\" -> \"ps\",\n    \"romanian\" -> \"ro\",\n    \"serbian\" -> \"sr\",\n    \"sindhi\" -> \"sd\",\n    \"welsh\" -> \"cy\",\n    \"uyghur\" -> \"ug\"\n  )\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/LanguageUtil.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.search.common.constants.{thriftscala => scc}\nimport com.twitter.search.common.util.lang.ThriftLanguageUtil\nimport com.twitter.service.metastore.gen.{thriftscala => smg}\n\nobject LanguageUtil {\n\n  private val DefaultMinProducedLanguageRatio = 0.05\n  private val DefaultMinConsumedLanguageConfidence = 0.8\n\n  /**\n   * Computes a list of languages based on UserLanguages information retrieved from Metastore.\n   *\n   * The list is sorted in descending order of confidence score associated with each language.\n   * That is, language with highest confidence value is in index 0.\n   */\n  def computeLanguages(\n    userLanguages: smg.UserLanguages,\n    minProducedLanguageRatio: Double = DefaultMinProducedLanguageRatio,\n    minConsumedLanguageConfidence: Double = DefaultMinConsumedLanguageConfidence\n  ): Seq[scc.ThriftLanguage] = {\n    val languageConfidenceMap = computeLanguageConfidenceMap(\n      userLanguages,\n      minProducedLanguageRatio,\n      minConsumedLanguageConfidence\n    )\n    languageConfidenceMap.toSeq.sortBy(-_._2).map(_._1) // _1 = language, _2 = score\n  }\n\n  /**\n   * Computes confidence map based on UserLanguages information retrieved from Metastore.\n   * where,\n   * key   = language code\n   * value = level of confidence that the language is applicable to a user.\n   */\n  private def computeLanguageConfidenceMap(\n    userLanguages: smg.UserLanguages,\n    minProducedLanguageRatio: Double,\n    minConsumedLanguageConfidence: Double\n  ): Map[scc.ThriftLanguage, Double] = {\n\n    val producedLanguages = getLanguageMap(userLanguages.produced)\n    val consumedLanguages = getLanguageMap(userLanguages.consumed)\n    val languages = (producedLanguages.keys ++ consumedLanguages.keys).toSet\n    var maxConfidence = 0.0\n\n    val confidenceMap = languages.map { language =>\n      val produceRatio = producedLanguages\n        .get(language)\n        .map { score => if (score < minProducedLanguageRatio) 0.0 else score }\n        .getOrElse(0.0)\n\n      val consumeConfidence = consumedLanguages\n        .get(language)\n        .map { score => if (score < minConsumedLanguageConfidence) 0.0 else score }\n        .getOrElse(0.0)\n\n      val overallConfidence = (0.3 + 4 * produceRatio) * (0.1 + consumeConfidence)\n      maxConfidence = Math.max(maxConfidence, overallConfidence)\n\n      (language -> overallConfidence)\n    }.toMap\n\n    val normalizedConfidenceMap = if (maxConfidence > 0) {\n      confidenceMap.map {\n        case (language, confidenceScore) =>\n          val normalizedScore = (confidenceScore / maxConfidence * 0.9) + 0.1\n          (language -> normalizedScore)\n      }\n    } else {\n      confidenceMap\n    }\n    normalizedConfidenceMap\n  }\n\n  private def getLanguageMap(\n    scoredLanguages: Seq[smg.ScoredString]\n  ): Map[scc.ThriftLanguage, Double] = {\n    scoredLanguages.flatMap { scoredLanguage =>\n      getThriftLanguage(scoredLanguage.item).map { language => (language -> scoredLanguage.weight) }\n    }.toMap\n  }\n\n  private def getThriftLanguage(languageName: String): Option[scc.ThriftLanguage] = {\n    val languageOrdinal = ThriftLanguageUtil.getThriftLanguageOf(languageName).ordinal\n    val language = scc.ThriftLanguage(languageOrdinal)\n    language match {\n      case scc.ThriftLanguage.Unknown => None\n      case _ => Some(language)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/MissingKeyException.scala",
    "content": "package com.twitter.home_mixer.util\n\nobject MissingKeyException extends Exception(\"Missing key\") {\n  override def toString: String = getMessage\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/NaviScorerStatsHandler.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.finagle.stats.BroadcastStatsReceiver\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.HomeFeatures.NaviClientConfigFeature\nimport com.twitter.home_mixer.model.PredictedScoreFeature\nimport com.twitter.home_mixer.model.PredictedScoreFeature.PredictedScoreFeatures\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelIdParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ModelNameParam\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.util.Memoize\n\ncase class ModelStats(broadcastStatsReceiver: StatsReceiver) {\n  private val StatsReadabilityMultiplier = 1000\n  private val PredictedScoreStatName = f\"predictedScore${StatsReadabilityMultiplier}x\"\n  private val MissingScoreStatName = \"missingScore\"\n  private val ValidScoreStatName = \"validScore\"\n  private val NullStat = NullStatsReceiver.stat(\"NullStat\")\n  private val NullCounter = NullStatsReceiver.counter(\"NullCounter\")\n\n  val failuresStat: Stat = broadcastStatsReceiver.stat(\"failures\")\n  val responsesStat: Stat = broadcastStatsReceiver.stat(\"responses\")\n  val invalidResponsesCounter: Counter = broadcastStatsReceiver.counter(\"invalidResponses\")\n  val scoreStat: Stat = broadcastStatsReceiver.stat(f\"score${StatsReadabilityMultiplier}x\")\n  val negativeFilterCounter: Counter = broadcastStatsReceiver.counter(\"negativeFiltered\")\n\n  private val predictedScoreStats: Map[PredictedScoreFeature, Stat] = PredictedScoreFeatures.map {\n    scoreFeature =>\n      (scoreFeature, broadcastStatsReceiver.stat(scoreFeature.statName, PredictedScoreStatName))\n  }.toMap\n\n  private val validScoreCounters: Map[PredictedScoreFeature, Counter] = PredictedScoreFeatures.map {\n    scoreFeature =>\n      (scoreFeature, broadcastStatsReceiver.counter(scoreFeature.statName, ValidScoreStatName))\n  }.toMap\n\n  private val missingScoreCounters: Map[PredictedScoreFeature, Counter] =\n    PredictedScoreFeatures.map { scoreFeature =>\n      (scoreFeature, broadcastStatsReceiver.counter(scoreFeature.statName, MissingScoreStatName))\n    }.toMap\n\n  def getPredictedScoreStat(scoreFeature: PredictedScoreFeature): Stat =\n    predictedScoreStats.getOrElse(scoreFeature, NullStat)\n\n  def getValidScoreCounter(scoreFeature: PredictedScoreFeature): Counter =\n    validScoreCounters.getOrElse(scoreFeature, NullCounter)\n\n  def getMissingScoreCounter(scoreFeature: PredictedScoreFeature): Counter =\n    missingScoreCounters.getOrElse(scoreFeature, NullCounter)\n\n  def trackPredictedScoreStats(\n    predictedScoreFeature: PredictedScoreFeature,\n    predictedScoreOpt: Option[Double]\n  ): Unit = {\n    predictedScoreOpt match {\n      case Some(predictedScore) =>\n        getPredictedScoreStat(predictedScoreFeature)\n          .add((predictedScore * StatsReadabilityMultiplier).toFloat)\n        getValidScoreCounter(predictedScoreFeature).incr()\n      case None =>\n        getMissingScoreCounter(predictedScoreFeature).incr()\n    }\n  }\n}\n\nclass NaviScorerStatsHandler(statsReceiver: StatsReceiver, scope: String) {\n  private val scopedStatsReceiver = statsReceiver.scope(scope)\n\n  // Memoize stats object so they are not created per request\n  private val statsPerModel = Memoize[(String, String, String, String), ModelStats] {\n    case (product, modelId, modelName, clientId) =>\n      // Collect stats overall, per product, per model, and per client\n      val broadcastStatsReceiver: StatsReceiver = BroadcastStatsReceiver(\n        Seq(\n          scopedStatsReceiver,\n          scopedStatsReceiver.scope(product),\n          scopedStatsReceiver.scope(modelId).scope(product),\n          scopedStatsReceiver.scope(modelName).scope(product),\n          scopedStatsReceiver.scope(clientId).scope(product)\n        )\n      )\n      ModelStats(broadcastStatsReceiver)\n  }\n\n  /** Retrieve ModelStats for the given query */\n  def getModelStats(\n    query: PipelineQuery\n  ): ModelStats = {\n    val modelId = query.params(ModelIdParam)\n    val modelName = query.params(ModelNameParam)\n    val modelNameStr = if (modelName.nonEmpty) modelName else \"EMPTY_MODEL_NAME\"\n\n    val naviClientConfig =\n      query.features.map(_.get(NaviClientConfigFeature)).get // Should always be present\n\n    val clientId = query.clientContext.appId.getOrElse(0L).toString\n    statsPerModel(\n      (\n        query.product.identifier.toString,\n        modelId + naviClientConfig.clusterStr,\n        modelNameStr + naviClientConfig.clusterStr,\n        clientId))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/ObservedKeyValueResultHandler.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.servo.keyvalue.KeyValueResult\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\n\ntrait ObservedKeyValueResultHandler {\n  val statsReceiver: StatsReceiver\n  val statScope: String\n\n  private lazy val scopedStatsReceiver = statsReceiver.scope(statScope)\n  private lazy val keyTotalCounter = scopedStatsReceiver.counter(\"key/total\")\n  private lazy val keyFoundCounter = scopedStatsReceiver.counter(\"key/found\")\n  private lazy val keyNotFoundCounter = scopedStatsReceiver.counter(\"key/notFound\")\n  private lazy val keyFailureCounter = scopedStatsReceiver.counter(\"key/failure\")\n\n  def observedGet[K, V](\n    key: Option[K],\n    keyValueResult: KeyValueResult[K, V],\n  ): Try[Option[V]] = {\n    if (key.nonEmpty) {\n      keyTotalCounter.incr()\n      keyValueResult(key.get) match {\n        case Return(Some(value)) =>\n          keyFoundCounter.incr()\n          Return(Some(value))\n        case Return(None) =>\n          keyNotFoundCounter.incr()\n          Return(None)\n        case Throw(exception) =>\n          keyFailureCounter.incr()\n          Throw(exception)\n        case _ =>\n          // never reaches here\n          Return(None)\n      }\n    } else {\n      Throw(MissingKeyException)\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/PhoenixScorerStatsHandler.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.finagle.stats.BroadcastStatsReceiver\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.home_mixer.model.PhoenixPredictedScoreFeature\nimport com.twitter.home_mixer.model.PhoenixPredictedScoreFeature.PhoenixPredictedScoreFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.util.Memoize\n\ncase class PhoenixModelStats(broadcastStatsReceiver: StatsReceiver) {\n  private val StatsReadabilityMultiplier = 1000\n  private val PredictedScoreStatName = f\"predictedScore${StatsReadabilityMultiplier}x\"\n  private val MissingScoreStatName = \"missingScore\"\n  private val ValidScoreStatName = \"validScore\"\n  private val NullStat = NullStatsReceiver.stat(\"NullStat\")\n  private val NullCounter = NullStatsReceiver.counter(\"NullCounter\")\n\n  val failuresStat: Stat = broadcastStatsReceiver.stat(\"failures\")\n  val responsesStat: Stat = broadcastStatsReceiver.stat(\"responses\")\n  val invalidResponsesCounter: Counter = broadcastStatsReceiver.counter(\"invalidResponses\")\n  val scoreStat: Stat = broadcastStatsReceiver.stat(f\"score${StatsReadabilityMultiplier}x\")\n  val negativeFilterCounter: Counter = broadcastStatsReceiver.counter(\"negativeFiltered\")\n\n  private val predictedScoreStats: Map[PhoenixPredictedScoreFeature, Stat] =\n    PhoenixPredictedScoreFeatures.map { scoreFeature =>\n      (scoreFeature, broadcastStatsReceiver.stat(scoreFeature.featureName, PredictedScoreStatName))\n    }.toMap\n\n  private val validScoreCounters: Map[PhoenixPredictedScoreFeature, Counter] =\n    PhoenixPredictedScoreFeatures.map { scoreFeature =>\n      (scoreFeature, broadcastStatsReceiver.counter(scoreFeature.featureName, ValidScoreStatName))\n    }.toMap\n\n  private val missingScoreCounters: Map[PhoenixPredictedScoreFeature, Counter] =\n    PhoenixPredictedScoreFeatures.map { scoreFeature =>\n      (scoreFeature, broadcastStatsReceiver.counter(scoreFeature.featureName, MissingScoreStatName))\n    }.toMap\n\n  def getPredictedScoreStat(scoreFeature: PhoenixPredictedScoreFeature): Stat =\n    predictedScoreStats.getOrElse(scoreFeature, NullStat)\n\n  def getValidScoreCounter(scoreFeature: PhoenixPredictedScoreFeature): Counter =\n    validScoreCounters.getOrElse(scoreFeature, NullCounter)\n\n  def getMissingScoreCounter(scoreFeature: PhoenixPredictedScoreFeature): Counter =\n    missingScoreCounters.getOrElse(scoreFeature, NullCounter)\n\n  def trackPredictedScoreStats(\n    predictedScoreFeature: PhoenixPredictedScoreFeature,\n    predictedScoreOpt: Option[Double]\n  ): Unit = {\n    predictedScoreOpt match {\n      case Some(predictedScore) =>\n        getPredictedScoreStat(predictedScoreFeature)\n          .add((predictedScore * StatsReadabilityMultiplier).toFloat)\n        getValidScoreCounter(predictedScoreFeature).incr()\n      case None =>\n        getMissingScoreCounter(predictedScoreFeature).incr()\n    }\n  }\n}\n\nclass PhoenixScorerStatsHandler(statsReceiver: StatsReceiver, scope: String) {\n  private val scopedStatsReceiver = statsReceiver.scope(scope)\n\n  // Memoize stats object so they are not created per request\n  private val statsPerModel = Memoize[(String, String, String), PhoenixModelStats] {\n    case (product, clientId, cluster) =>\n      // Collect stats overall, per product, and per client\n      val broadcastStatsReceiver: StatsReceiver = BroadcastStatsReceiver(\n        Seq(\n          scopedStatsReceiver,\n          scopedStatsReceiver.scope(product),\n          scopedStatsReceiver.scope(product).scope(cluster),\n          scopedStatsReceiver.scope(clientId).scope(product)\n        )\n      )\n      PhoenixModelStats(broadcastStatsReceiver)\n  }\n\n  def getModelStats(query: PipelineQuery, cluster: String): PhoenixModelStats = {\n    val clientId = query.clientContext.appId.getOrElse(0L).toString\n    statsPerModel((query.product.identifier.toString, clientId, cluster))\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/PhoenixUtils.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.finagle.grpc.FutureConverters\nimport com.twitter.finagle.service.RetryPolicy\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.home_mixer.model.HomeFeatures.AuthorIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.InReplyToTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.QuotedTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceTweetIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.SourceUserIdFeature\nimport com.twitter.home_mixer.model.HomeFeatures.UserActionsFeature\nimport com.twitter.inject.utils.RetryUtils\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.util.MemoizingStatsReceiver\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\nimport com.x.user_action_sequence.ActionName\nimport com.x.user_action_sequence.CandidateSet\nimport com.x.user_action_sequence.PredictNextActionsRequest\nimport com.x.user_action_sequence.PredictNextActionsResponse\nimport com.x.user_action_sequence.RecsysPredictorGrpc\nimport com.x.user_action_sequence.TweetBoolFeatures\nimport com.x.user_action_sequence.TweetInfo\nimport io.grpc.ManagedChannel\nimport java.util.concurrent.TimeUnit\nimport scala.collection.JavaConverters._\nimport scala.util.Random\n\nobject PhoenixUtils {\n  private val TopLogProbsNum = 50\n  private val MaxCandidates = 1400\n\n  def getTweetInfoFromCandidates(\n    candidateIds: Seq[TweetCandidate],\n    featureMaps: Seq[FeatureMap]\n  ): Seq[TweetInfo] = {\n    assert(\n      candidateIds.length == featureMaps.length,\n      \"FeatureMap doesn't match candidateIds in length\")\n    val candidateWithFeatureMaps = candidateIds.zip(featureMaps)\n    candidateWithFeatureMaps.take(MaxCandidates).map {\n      case (candidate, featureMap) =>\n        val tweetBooleanFeatureBuilder = TweetBoolFeatures.newBuilder()\n        val (sourceTweetId, sourceAuthorId) = (\n          featureMap.getOrElse(SourceTweetIdFeature, None),\n          featureMap.getOrElse(SourceUserIdFeature, None)) match {\n          case (Some(sourceTweetId), Some(sourceAuthorId)) =>\n            tweetBooleanFeatureBuilder.setIsRetweet(true)\n            (sourceTweetId, sourceAuthorId)\n          case _ =>\n            tweetBooleanFeatureBuilder.setIsRetweet(false)\n            (candidate.id, featureMap.get(AuthorIdFeature).get)\n        }\n        tweetBooleanFeatureBuilder\n          .setIsForYouPage(true)\n          .setIsPromotedTweet(false)\n          .setIsReply(featureMap.getOrElse(InReplyToTweetIdFeature, None).isDefined)\n          .setIsQuote(featureMap.getOrElse(QuotedTweetIdFeature, None).isDefined)\n        TweetInfo\n          .newBuilder()\n          .setTweetId(sourceTweetId)\n          .setAuthorId(sourceAuthorId)\n          .setTweetBoolFeatures(tweetBooleanFeatureBuilder.build())\n          .build()\n    }\n  }\n\n  def createCandidateSets(\n    query: PipelineQuery,\n    tweetInfos: Seq[TweetInfo]\n  ): PredictNextActionsRequest = {\n    val actionsSeqOpt = query.features.flatMap(_.getOrElse(UserActionsFeature, None))\n    val candidateSets = CandidateSet\n      .newBuilder()\n      .setUserId(query.getRequiredUserId)\n      .addAllCandidates(tweetInfos.asJava)\n      .build()\n\n    val predictionRequestBuilder = PredictNextActionsRequest\n      .newBuilder()\n      .addCandidateSets(candidateSets)\n      .setReturnLogprob(true)\n      .setTopLogprobsNum(TopLogProbsNum)\n\n    actionsSeqOpt.foreach { actionsSeq => predictionRequestBuilder.addSequences(actionsSeq) }\n    predictionRequestBuilder.build()\n  }\n\n  def predict(\n    request: PredictNextActionsRequest,\n    channels: Seq[ManagedChannel],\n    cluster: String,\n    timeoutMs: Int,\n    baseStat: MemoizingStatsReceiver,\n  ): Stitch[PredictNextActionsResponse] = {\n\n    val timeStat = baseStat.scope(\"rpcTime\")\n    val successStat = baseStat.scope(\"rpcSuccess\")\n    val failureStat = baseStat.scope(\"rpcFailure\")\n    val failureMsgStat = baseStat.scope(\"rpcFailureMsg\")\n\n    def attemptPredict(): Future[PredictNextActionsResponse] = {\n      // This is kept inside predict so that it doesn't use the same channel for retries\n      val channel = channels(Random.nextInt(channels.length))\n      val recsysPredictorFutureStub =\n        RecsysPredictorGrpc\n          .newFutureStub(channel).withDeadlineAfter(timeoutMs, TimeUnit.MILLISECONDS)\n\n      FutureConverters\n        .RichListenableFuture(recsysPredictorFutureStub.predictNextActions(request))\n        .toTwitter\n    }\n\n    // Retry configuration: 2 attempts, 500ms per attempt, total 1s max\n    val retryPolicy = RetryPolicy\n      .tries[Try[PredictNextActionsResponse]](2, { case Throw(_) => true; case Return(_) => false })\n\n    val attemptPredictRetriedFuture =\n      RetryUtils.retryFuture[PredictNextActionsResponse](retryPolicy)(attemptPredict)\n\n    Stitch\n      .callFuture(Stat.timeFuture(timeStat.stat(cluster))(attemptPredictRetriedFuture))\n      .onFailure { e =>\n        val errMsg = e.getMessage.replaceAll(\"\\\\W\", \"\").takeRight(100)\n        failureStat.counter(cluster).incr()\n        failureMsgStat.scope(cluster).counter(errMsg).incr()\n      }.onSuccess { _ =>\n        successStat.counter(cluster).incr()\n      }\n  }\n\n  def getPredictionResponseMap(\n    request: PredictNextActionsRequest,\n    channels: Seq[ManagedChannel],\n    cluster: String,\n    timeoutMs: Int,\n    memoizingStatsReceiver: MemoizingStatsReceiver\n  ): Stitch[Map[Long, Map[ActionName, Double]]] = {\n    val predictionsResponse = predict(request, channels, cluster, timeoutMs, memoizingStatsReceiver)\n    predictionsResponse.map { predictions =>\n      val distributionSetsList = predictions.getDistributionSetsList\n      val candidateDistributions = if (!distributionSetsList.isEmpty) {\n        distributionSetsList.get(0).getCandidateDistributionsList.asScala\n      } else Seq.empty\n\n      candidateDistributions.map { distribution =>\n        val candidateId = distribution.getCandidate.getTweetId\n        val probMap = distribution.getTopLogProbsList.asScala.zipWithIndex.map {\n          case (logProb, idx) =>\n            ActionName.forNumber(idx) -> math.exp(logProb.doubleValue())\n        }.toMap\n        candidateId -> probMap\n      }.toMap\n    }\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/ReplyRetweetUtil.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.home_mixer.model.HomeFeatures._\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\n\nobject ReplyRetweetUtil {\n\n  def isEligibleReply(candidate: CandidateWithFeatures[TweetCandidate]): Boolean = {\n    candidate.features.getOrElse(InReplyToTweetIdFeature, None).nonEmpty &&\n    !candidate.features.getOrElse(IsRetweetFeature, false)\n  }\n\n  /**\n   * Builds a map from reply tweet to all ancestors that are also hydrated candidates. If a reply\n   * does not have any ancestors which are also candidates, it will not add to the returned Map.\n   * Make sure ancestors are bottom-up ordered such that:\n   * (1) if parent tweet is a candidate, it should be the first item at the returned ancestors;\n   * (2) if root tweet is a candidate, it should be the last item at the returned ancestors.\n   * Retweets of replies or replies to retweets are not included.\n   */\n  def replyToAncestorTweetCandidatesMap(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, Seq[CandidateWithFeatures[TweetCandidate]]] = {\n    val replyToAncestorTweetIdsMap: Map[Long, Seq[Long]] =\n      candidates.flatMap { candidate =>\n        if (isEligibleReply(candidate)) {\n          val ancestorIds =\n            if (candidate.features.getOrElse(AncestorsFeature, Seq.empty).nonEmpty) {\n              candidate.features.getOrElse(AncestorsFeature, Seq.empty).map(_.tweetId)\n            } else {\n              Seq(\n                candidate.features.getOrElse(InReplyToTweetIdFeature, None),\n                candidate.features.getOrElse(ConversationModuleIdFeature, None)\n              ).flatten.distinct\n            }\n          Some(candidate.candidate.id -> ancestorIds)\n        } else {\n          None\n        }\n      }.toMap\n\n    val ancestorTweetIds = replyToAncestorTweetIdsMap.values.flatten.toSet\n    val ancestorTweetsMapById: Map[Long, CandidateWithFeatures[TweetCandidate]] = candidates\n      .filter { maybeAncestor =>\n        ancestorTweetIds.contains(maybeAncestor.candidate.id)\n      }.map { ancestor =>\n        ancestor.candidate.id -> ancestor\n      }.toMap\n\n    replyToAncestorTweetIdsMap\n      .mapValues { ancestorTweetIds =>\n        ancestorTweetIds.flatMap { ancestorTweetId =>\n          ancestorTweetsMapById.get(ancestorTweetId)\n        }\n      }.filter {\n        case (reply, ancestors) =>\n          ancestors.nonEmpty\n      }\n  }\n\n  /**\n   * This map is the opposite of [[replyToAncestorTweetCandidatesMap]].\n   * Builds a map from ancestor tweet to all descendant replies that are also hydrated candidates.\n   * Currently, we only return two ancestors at most: one is inReplyToTweetId and the other\n   * is conversationId.\n   * Retweets of replies are not included.\n   */\n  def ancestorTweetIdToDescendantRepliesMap(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, Seq[CandidateWithFeatures[TweetCandidate]]] = {\n    val tweetToCandidateMap = candidates.map(c => c.candidate.id -> c).toMap\n    replyToAncestorTweetCandidatesMap(candidates).toSeq\n      .flatMap {\n        case (reply, ancestorTweets) =>\n          ancestorTweets.map { ancestor =>\n            (ancestor.candidate.id, reply)\n          }\n      }.groupBy { case (ancestor, reply) => ancestor }\n      .mapValues { ancestorReplyPairs =>\n        ancestorReplyPairs.map(_._2).distinct\n      }.mapValues(tweetIds => tweetIds.map(tid => tweetToCandidateMap(tid)))\n  }\n\n  /**\n   * Builds a map from reply tweet to inReplyToTweet which is also a candidate.\n   * Retweets of replies or replies to retweets are not included\n   */\n  def replyTweetIdToInReplyToTweetMap(\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Map[Long, CandidateWithFeatures[TweetCandidate]] = {\n    val eligibleReplyCandidates = candidates.filter { candidate =>\n      isEligibleReply(candidate) && candidate.features\n        .getOrElse(InReplyToTweetIdFeature, None)\n        .nonEmpty\n    }\n\n    val inReplyToTweetIds = eligibleReplyCandidates\n      .flatMap(_.features.getOrElse(InReplyToTweetIdFeature, None))\n      .toSet\n\n    val inReplyToTweetIdToTweetMap: Map[Long, CandidateWithFeatures[TweetCandidate]] = candidates\n      .filter { maybeInReplyToTweet =>\n        inReplyToTweetIds.contains(maybeInReplyToTweet.candidate.id)\n      }.map { inReplyToTweet =>\n        inReplyToTweet.candidate.id -> inReplyToTweet\n      }.toMap\n\n    eligibleReplyCandidates.flatMap { reply =>\n      val inReplyToTweetId = reply.features.getOrElse(InReplyToTweetIdFeature, None)\n      if (inReplyToTweetId.nonEmpty) {\n        inReplyToTweetIdToTweetMap.get(inReplyToTweetId.get).map { inReplyToTweet =>\n          reply.candidate.id -> inReplyToTweet\n        }\n      } else {\n        None\n      }\n    }.toMap\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/RerankerUtil.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.home_mixer.model.PredictedScoreFeature.PredictedScoreFeatures\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.ConstantNegativeHead\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.EnableNegSectionRankingParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.NegativeScoreConstantFilterThresholdParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.NegativeScoreNormFilterThresholdParam\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.NormalizedNegativeHead\nimport com.twitter.home_mixer.param.HomeGlobalParams.Scoring.UseWeightForNegHeadParam\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject RerankerUtil {\n  val Epsilon = 0.001\n\n  def computeModelScores(\n    query: PipelineQuery,\n    candidate: CandidateWithFeatures[TweetCandidate],\n    modelStatsOpt: Option[ModelStats] = None\n  ): Seq[(Double, Double)] = {\n    PredictedScoreFeatures.map { predictedScoreFeature =>\n      val predictedScoreOpt = predictedScoreFeature.extractScore(candidate.features, query)\n\n      modelStatsOpt.foreach(_.trackPredictedScoreStats(predictedScoreFeature, predictedScoreOpt))\n\n      val weight =\n        query.features.flatMap(_.get(predictedScoreFeature.weightQueryFeature)).getOrElse(0.0)\n\n      val bias = predictedScoreFeature.biasQueryFeature\n        .flatMap(feature => query.features.flatMap(_.get(feature))).getOrElse(0.0)\n\n      val score =\n        if (predictedScoreFeature.isEligible(candidate.features, query))\n          predictedScoreOpt.getOrElse(0.0)\n        else bias\n\n      (score, weight)\n    }\n  }\n\n  def getScoresWithPerHeadMax(\n    scoresAndWeightsSeq: Seq[Seq[(Double, Double)]]\n  ): Seq[Seq[(Double, Double, Double)]] = {\n    if (scoresAndWeightsSeq.isEmpty) Seq.empty\n    else {\n      // Step 1: Transpose scores to group by heads\n      val headsScores: Seq[Seq[Double]] = scoresAndWeightsSeq.transpose.map { headScores =>\n        headScores.map { case (score, _) => score }\n      }\n\n      // Step 2: Get max scores per head\n      val headMaxScores: Seq[Double] = headsScores.map(_.max).toIndexedSeq\n\n      // Step 3: Use max scores to get per head transformed scores\n      scoresAndWeightsSeq.map { candidateScores =>\n        candidateScores.zipWithIndex.map {\n          case ((score, weight), headIdx) =>\n            val headMaxScore = headMaxScores(headIdx)\n            (score, headMaxScore, weight)\n        }\n      }\n    }\n  }\n\n  def computeDebugMetadata(\n    debugStr: String,\n    featureNames: Seq[String],\n    transformedScores: Seq[(Double, Double, Double)],\n    finalScore: Double\n  ): String = {\n    assert(\n      featureNames.size == transformedScores.size,\n      \"Feature names size doesn't matter scores size\")\n    val contributions: Seq[(String, Double)] = featureNames\n      .zip(transformedScores)\n      .collect {\n        case (feature, (score, _, weight)) if weight >= 0 =>\n          (feature, score * weight)\n      }\n\n    val topContributors: Seq[String] = contributions.collect {\n      case (name, contrib) if finalScore > 0 && (contrib / finalScore) > 0.3 =>\n        f\"$name:%%.2f\".format(contrib / finalScore)\n    }\n\n    debugStr + s\" [${topContributors.mkString(\", \")}]\"\n  }\n\n  def aggregateWeightedScores(\n    query: PipelineQuery,\n    scoresAndWeights: Seq[(Double, Double, Double)],\n    negativeFilterCounter: Counter\n  ): Double = {\n    val thresholdNegative = query.params(NegativeScoreConstantFilterThresholdParam)\n    val thresholdNegativeNormalized = query.params(NegativeScoreNormFilterThresholdParam)\n    val enableNegNormalized = query.params(NormalizedNegativeHead)\n    val enableNegConstant = query.params(ConstantNegativeHead)\n    val useWeightForNeg = query.params(UseWeightForNegHeadParam)\n    val negSectionRanking = query.params(EnableNegSectionRankingParam)\n    val (_, maxHeadScores, modelWeights) = scoresAndWeights.unzip3\n    val combinedScoreSum: Double = {\n      scoresAndWeights.foldLeft(0.0) {\n        case (combinedScore, (score, maxHeadScore, weight)) =>\n          if (weight >= 0.0 || useWeightForNeg) {\n            combinedScore + score * weight\n          } else {\n            // Apply filtering logic only for negative weights\n            val normScore = if (maxHeadScore == 0.0) 0.0 else score / maxHeadScore\n            val negFilterNorm = enableNegNormalized && normScore > thresholdNegativeNormalized\n            val negFilterConstant = enableNegConstant && score > thresholdNegative\n            if (negFilterNorm || negFilterConstant) {\n              negativeFilterCounter.incr()\n              // This should be shipped and cleaned as soon as possible\n              if (negSectionRanking) {\n                // Assumes negative scores are not greater than 0.9 and clipped to 1\n                combinedScore + weight * (1.0 min (score + 0.1))\n              } else\n                combinedScore + weight\n            } else combinedScore\n          }\n      }\n    }\n\n    val positiveModelWeightsSum = modelWeights.filter(_ > 0.0).sum\n    val negativeModelWeightsSum = modelWeights.filter(_ < 0).sum.abs\n    val modelWeightsSum = positiveModelWeightsSum + negativeModelWeightsSum\n\n    val weightedScoresSum =\n      if (modelWeightsSum == 0) combinedScoreSum.max(0.0)\n      else if (combinedScoreSum < 0)\n        (combinedScoreSum + negativeModelWeightsSum) / modelWeightsSum * Epsilon\n      else combinedScoreSum + Epsilon\n\n    weightedScoresSum\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/SignalUtil.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.home_mixer.model.HomeFeatures.LowSignalUserFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.social_graph.SGSFollowedUsersFeature\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.usersignalservice.{thriftscala => uss}\n\nobject SignalUtil {\n\n  val ExplicitSignals: Seq[uss.SignalType] = Seq(\n    uss.SignalType.TweetFavorite,\n    uss.SignalType.Retweet,\n    uss.SignalType.Reply,\n    uss.SignalType.TweetBookmarkV1,\n    uss.SignalType.TweetShareV1\n  )\n\n  private val SmallFollowGraphSize = 5\n\n  def isLowSignalUser(query: PipelineQuery): Boolean = {\n    val followGraphSize = query.features.map(_.getOrElse(SGSFollowedUsersFeature, Seq.empty).size)\n    val smallFollowGraph = followGraphSize.exists(_ < SmallFollowGraphSize)\n    val lowSignal = query.features.map(_.getOrElse(LowSignalUserFeature, false)).getOrElse(false)\n    lowSignal && smallFollowGraph\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/TensorFlowUtil.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.ml.api.thriftscala.FloatTensor\nimport com.twitter.ml.api.util.BufferToIterators.RichFloatBuffer\nimport java.nio.ByteBuffer\nimport java.nio.ByteOrder\n\n/**\n * Contains functionality to transform data records and Tensors\n */\n\nobject TensorFlowUtil {\n\n  private def skipEmbeddingBBHeader(bb: ByteBuffer): ByteBuffer = {\n    val bb_copy = bb.duplicate()\n    bb_copy.getLong()\n    bb_copy\n  }\n\n  private def byteBufferToFloatIterator(\n    bb: ByteBuffer\n  ): Iterator[Float] = {\n    bb.order(ByteOrder.LITTLE_ENDIAN).asFloatBuffer.iterator\n  }\n\n  def embeddingByteBufferToFloatTensor(\n    bb: ByteBuffer\n  ): FloatTensor = {\n    val bb_content = skipEmbeddingBBHeader(bb)\n    FloatTensor(byteBufferToFloatIterator(bb_content).map(_.toDouble).toList)\n  }\n\n  def embeddingNoHeaderByteBufferToFloatTensor(\n    bb: ByteBuffer\n  ): FloatTensor = {\n    FloatTensor(byteBufferToFloatIterator(bb).map(_.toDouble).toList)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/TweetImpressionsHelper.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.home_mixer.model.HomeFeatures.TweetImpressionsFeature\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.impressed_tweets.ImpressedTweets\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\n\nobject TweetImpressionsHelper {\n  def tweetImpressions(features: FeatureMap): Set[Long] = {\n    val manhattanImpressions =\n      features.getOrElse(TweetImpressionsFeature, Seq.empty).flatMap(_.tweetIds)\n    val memcacheImpressions = features.getOrElse(ImpressedTweets, Seq.empty)\n\n    (manhattanImpressions ++ memcacheImpressions).toSet\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/UrtUtil.scala",
    "content": "package com.twitter.home_mixer.util\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DeepLink\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ExternalUrl\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.UrtEndpoint\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.UrtEndpointOptions\nimport com.twitter.timelines.render.{thriftscala => t}\n\nobject UrtUtil {\n  def transformUrl(url: t.Url): Url = {\n    val endpointOptions = url.urtEndpointOptions.map { options =>\n      UrtEndpointOptions(\n        requestParams = options.requestParams.map(_.toMap),\n        title = options.title,\n        cacheId = options.cacheId,\n        subtitle = options.subtitle\n      )\n    }\n\n    val urlType = url.urlType match {\n      case t.UrlType.ExternalUrl => ExternalUrl\n      case t.UrlType.DeepLink => DeepLink\n      case t.UrlType.UrtEndpoint => UrtEndpoint\n      case t.UrlType.EnumUnknownUrlType(field) =>\n        throw new UnknownUrlTypeException(field)\n    }\n\n    Url(urlType = urlType, url = url.url, urtEndpointOptions = endpointOptions)\n  }\n}\nclass UnknownUrlTypeException(field: Int)\n    extends UnsupportedOperationException(s\"Unknown url type: $field\")\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/earlybird/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/common/util/lang\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"timelines/src/main/scala/com/twitter/timelines/clients/relevance_search\",\n        \"timelines/src/main/scala/com/twitter/timelines/earlybird/common/options\",\n        \"timelines/src/main/scala/com/twitter/timelines/earlybird/common/utils\",\n        \"timelines/src/main/scala/com/twitter/timelines/model/types\",\n        \"timelines/src/main/scala/com/twitter/timelines/util/stats\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/earlybird/EarlybirdRequestUtil.scala",
    "content": "package com.twitter.home_mixer.util.earlybird\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.search.common.query.thriftjava.{thriftscala => scq}\nimport com.twitter.search.common.ranking.{thriftscala => scr}\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport com.twitter.timelines.clients.relevance_search.SearchClient.TweetFeatures\nimport com.twitter.timelines.clients.relevance_search.SearchClient.TweetTypes\nimport com.twitter.timelines.clients.relevance_search.SearchQueryBuilder\nimport com.twitter.timelines.clients.relevance_search.SearchQueryBuilder.QueryWithNamedDisjunctions\nimport com.twitter.timelines.earlybird.common.options.EarlybirdScoringModelConfig\nimport com.twitter.timelines.earlybird.common.utils.SearchOperator\nimport com.twitter.util.Duration\n\nobject EarlybirdRequestUtil {\n\n  val DefaultMaxHitsToProcess = 1000\n  val DefaultSearchProcessingTimeout: Duration = 200.milliseconds\n  val DefaultFeatureHydrationTimeout: Duration = 50.milliseconds\n  val DefaultHydrationMaxNumResultsPerShard = 1000\n  val DefaultQueryMaxNumResultsPerShard = 300\n  val DefaultHydrationCollectorParams = mkCollectorParams(DefaultHydrationMaxNumResultsPerShard)\n\n  private val queryBuilder = new SearchQueryBuilder\n\n  object EarlybirdScoringModels {\n    val UnifiedEngagementProd: Seq[EarlybirdScoringModelConfig] = Seq(\n      EarlybirdScoringModelConfig(\"timelines_unified_engagement_prod.schema_based\", 1.0)\n    )\n\n    val UnifiedEngagementRectweet: Seq[EarlybirdScoringModelConfig] = Seq(\n      EarlybirdScoringModelConfig(\"timelines_unified_engagement_rectweet.schema_based\", 1.0)\n    )\n  }\n\n  private[earlybird] def mkCollectorParams(numResultsToReturn: Int): scq.CollectorParams = {\n    scq.CollectorParams(\n      // numResultsToReturn defines how many results each EB shard will return to search root\n      numResultsToReturn = numResultsToReturn,\n      // terminationParams.maxHitsToProcess is used for early terminating per shard results fetching.\n      terminationParams = Some(\n        scq.CollectorTerminationParams(\n          maxHitsToProcess = Some(DefaultMaxHitsToProcess),\n          timeoutMs = DefaultSearchProcessingTimeout.inMilliseconds.toInt\n        ))\n    )\n  }\n\n  private def getRankingParams(\n    authorScoreMap: Option[Map[Long, Double]],\n    tensorflowModel: Option[String],\n    ebModels: Seq[EarlybirdScoringModelConfig]\n  ): Option[scr.ThriftRankingParams] = {\n    if (tensorflowModel.nonEmpty) {\n      Some(\n        scr.ThriftRankingParams(\n          `type` = Some(scr.ThriftScoringFunctionType.TensorflowBased),\n          selectedTensorflowModel = tensorflowModel,\n          minScore = -1.0e100,\n          applyBoosts = false,\n          authorSpecificScoreAdjustments = authorScoreMap\n        )\n      )\n    } else if (ebModels.nonEmpty) {\n      Some(\n        scr.ThriftRankingParams(\n          `type` = Some(scr.ThriftScoringFunctionType.ModelBased),\n          selectedModels = Some(ebModels.map(m => m.name -> m.weight).toMap),\n          applyBoosts = false,\n          minScore = -1.0e100,\n          authorSpecificScoreAdjustments = authorScoreMap\n        )\n      )\n    } else None\n  }\n\n  def getTweetsRequest(\n    userId: Option[Long],\n    clientId: Option[String],\n    skipVeryRecentTweets: Boolean,\n    queryUserIds: Set[Long],\n    retweetsMutedUserIds: Set[Long],\n    beforeTweetIdExclusive: Option[Long],\n    afterTweetIdExclusive: Option[Long],\n    excludedTweetIds: Option[Set[Long]] = None,\n    maxCount: Int,\n    tweetTypes: TweetTypes.ValueSet,\n    authorScoreMap: Option[Map[Long, Double]] = None,\n    tensorflowModel: Option[String] = None,\n    ebModels: Seq[EarlybirdScoringModelConfig] = Seq.empty,\n    queryMaxNumResultsPerShard: Int = DefaultQueryMaxNumResultsPerShard,\n    enableExcludeSourceTweetIdsQuery: Boolean = false,\n    isVideoOnlyRequest: Boolean = false,\n    isRecency: Boolean = false,\n    getOlderTweets: Boolean = false,\n  ): eb.EarlybirdRequest = {\n\n    val QueryWithNamedDisjunctions(query, namedDisjunctionMap) = queryBuilder.create(\n      queryUserIds,\n      retweetsMutedUserIds,\n      beforeTweetIdExclusive,\n      afterTweetIdExclusive,\n      semanticCoreIds = None,\n      languages = None,\n      tweetTypes = tweetTypes,\n      searchOperator = SearchOperator.Exclude,\n      tweetFeatures = TweetFeatures.All,\n      excludedTweetIds = excludedTweetIds.getOrElse(Set.empty),\n      enableExcludeSourceTweetIdsQuery = enableExcludeSourceTweetIdsQuery,\n      isVideoOnlyRequest = isVideoOnlyRequest\n    )\n    val ebRankingParams = getRankingParams(authorScoreMap, tensorflowModel, ebModels)\n    val relOptions = RelevanceSearchUtil.RelevanceOptions.copy(\n      rankingParams = ebRankingParams,\n      returnAllResults = Some(false)\n    )\n\n    val metadataOptions =\n      if (isRecency)\n        eb.ThriftSearchResultMetadataOptions(\n          getTweetUrls = false,\n          getResultLocation = false,\n          deprecatedGetTopicIDs = false,\n          getInReplyToStatusId = true,\n          getReferencedTweetAuthorId = true,\n          getFromUserId = true\n        )\n      else RelevanceSearchUtil.MetadataOptions\n\n    val queryUserIdsSeq = queryUserIds.toSeq\n    val namedDisjunctionMapOpt =\n      if (namedDisjunctionMap.isEmpty) None\n      else Some(namedDisjunctionMap.mapValues(_.toSeq))\n\n    val rankingMode =\n      if (isRecency) eb.ThriftSearchRankingMode.Recency else eb.ThriftSearchRankingMode.Relevance\n\n    val thriftQuery = eb.ThriftSearchQuery(\n      serializedQuery = Some(query.serialize),\n      fromUserIDFilter64 = Some(queryUserIdsSeq),\n      numResults = maxCount,\n      collectConversationId = true,\n      rankingMode = rankingMode,\n      relevanceOptions = Some(relOptions),\n      collectorParams = Some(mkCollectorParams(queryMaxNumResultsPerShard)),\n      facetFieldNames = Some(RelevanceSearchUtil.FacetsToFetch),\n      resultMetadataOptions = Some(metadataOptions),\n      searcherId = userId,\n      searchStatusIds = None,\n      namedDisjunctionMap = namedDisjunctionMapOpt\n    )\n\n    eb.EarlybirdRequest(\n      searchQuery = thriftQuery,\n      clientId = clientId,\n      getOlderResults = Some(getOlderTweets),\n      followedUserIds = Some(queryUserIdsSeq),\n      getProtectedTweetsOnly = Some(false),\n      timeoutMs = DefaultSearchProcessingTimeout.inMilliseconds.toInt,\n      skipVeryRecentTweets = skipVeryRecentTweets,\n      numResultsToReturnAtRoot = Some(maxCount)\n    )\n  }\n\n  def getTweetsFeaturesRequest(\n    userId: Option[Long],\n    tweetIds: Option[Seq[Long]],\n    clientId: Option[String],\n    getOnlyProtectedTweets: Boolean = false,\n    authorScoreMap: Option[Map[Long, Double]] = None,\n    tensorflowModel: Option[String] = None,\n    ebModels: Seq[EarlybirdScoringModelConfig] = Seq.empty\n  ): eb.EarlybirdRequest = {\n\n    val candidateSize = tweetIds.getOrElse(Seq.empty).size\n    val ebRankingParams = getRankingParams(authorScoreMap, tensorflowModel, ebModels)\n    val relOptions = RelevanceSearchUtil.RelevanceOptions.copy(\n      rankingParams = ebRankingParams\n    )\n    val thriftQuery = eb.ThriftSearchQuery(\n      numResults = candidateSize,\n      collectConversationId = true,\n      rankingMode = eb.ThriftSearchRankingMode.Relevance,\n      relevanceOptions = Some(relOptions),\n      collectorParams = Some(DefaultHydrationCollectorParams),\n      facetFieldNames = Some(RelevanceSearchUtil.FacetsToFetch),\n      resultMetadataOptions = Some(RelevanceSearchUtil.MetadataOptions),\n      searcherId = userId,\n      searchStatusIds = tweetIds.map(_.toSet),\n    )\n\n    eb.EarlybirdRequest(\n      searchQuery = thriftQuery,\n      clientId = clientId,\n      getOlderResults = Some(false),\n      getProtectedTweetsOnly = Some(getOnlyProtectedTweets),\n      timeoutMs = DefaultSearchProcessingTimeout.inMilliseconds.toInt,\n      skipVeryRecentTweets = true,\n      // This param decides # of tweets to return from search superRoot and realtime/protected/Archive roots.\n      // It takes higher precedence than ThriftSearchQuery.numResults\n      numResultsToReturnAtRoot = Some(candidateSize),\n      partitionTimeoutMs = Some(DefaultFeatureHydrationTimeout.inMillis.toInt)\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/earlybird/EarlybirdResponseUtil.scala",
    "content": "package com.twitter.home_mixer.util.earlybird\n\nimport com.twitter.product_mixer.core.util.OffloadFuturePools\nimport com.twitter.search.common.constants.{thriftscala => scc}\nimport com.twitter.search.common.features.{thriftscala => sc}\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant._\nimport com.twitter.search.common.util.lang.ThriftLanguageUtil\nimport com.twitter.search.earlybird.{thriftscala => eb}\nimport com.twitter.timelines.earlybird.common.utils.InNetworkEngagement\nimport com.twitter.util.Future\n\nobject EarlybirdResponseUtil {\n\n  private[earlybird] val Mentions: String = \"mentions\"\n  private val CharsToRemoveFromMentions: Set[Char] = \"@\".toSet\n\n  // Default value of settings of ThriftTweetFeatures.\n  private[earlybird] val DefaultEarlybirdFeatures: sc.ThriftTweetFeatures = sc.ThriftTweetFeatures()\n  private[earlybird] val DefaultCount = 0\n  private[earlybird] val DefaultLanguage = 0\n  private[earlybird] val DefaultScore = 0.0\n  private[earlybird] val DefaultEBResponseProcessParallelism = 32\n\n  def getTweetCountByAuthorId(\n    searchResults: Seq[eb.ThriftSearchResult]\n  ): Map[Long, Int] = {\n    val tweetCounts = scala.collection.mutable.Map.empty[Long, Int]\n    searchResults.foreach { result =>\n      val authorId = result.metadata.map(_.fromUserId).getOrElse(0L)\n      tweetCounts(authorId) = tweetCounts.getOrElse(authorId, 0) + 1\n    }\n    tweetCounts.toMap.withDefaultValue(0)\n  }\n\n  def getLanguage(uiLanguageCode: Option[String]): Option[scc.ThriftLanguage] = {\n    uiLanguageCode.flatMap { languageCode =>\n      scc.ThriftLanguage.get(ThriftLanguageUtil.getThriftLanguageOf(languageCode).getValue)\n    }\n  }\n\n  private def getMentions(result: eb.ThriftSearchResult): Seq[String] = {\n    val facetLabels = result.metadata.flatMap(_.facetLabels).getOrElse(Seq.empty)\n    getFacets(facetLabels, Mentions, CharsToRemoveFromMentions)\n  }\n\n  private def getFacets(\n    facetLabels: Seq[eb.ThriftFacetLabel],\n    facetName: String,\n    charsToRemove: Set[Char]\n  ): Seq[String] = {\n    facetLabels.filter(_.fieldName == facetName).map(_.label.filterNot(charsToRemove))\n  }\n\n  private def isUserMentioned(\n    screenName: Option[String],\n    mentions: Seq[String],\n    mentionsInSourceTweet: Seq[String]\n  ): Boolean =\n    isUserMentioned(screenName, mentions) || isUserMentioned(screenName, mentionsInSourceTweet)\n\n  private def isUserMentioned(\n    screenName: Option[String],\n    mentions: Seq[String]\n  ): Boolean = {\n    screenName\n      .exists { screenName => mentions.exists(_.equalsIgnoreCase(screenName)) }\n  }\n\n  private[earlybird] def isUsersMainLanguage(\n    tweetLanguage: scc.ThriftLanguage,\n    userLanguages: Seq[scc.ThriftLanguage]\n  ): Boolean = {\n    (tweetLanguage != scc.ThriftLanguage.Unknown) && userLanguages.headOption.contains(\n      tweetLanguage)\n  }\n\n  private[earlybird] def isUsersLanguage(\n    tweetLanguage: scc.ThriftLanguage,\n    userLanguages: Seq[scc.ThriftLanguage]\n  ): Boolean = {\n    (tweetLanguage != scc.ThriftLanguage.Unknown) && userLanguages.contains(tweetLanguage)\n  }\n\n  private[earlybird] def isUILanguage(\n    tweetLanguage: scc.ThriftLanguage,\n    uiLanguage: Option[scc.ThriftLanguage]\n  ): Boolean = {\n    (tweetLanguage != scc.ThriftLanguage.Unknown) && uiLanguage.contains(tweetLanguage)\n  }\n\n  private def getBooleanOptFeature(\n    featureName: EarlybirdFieldConstant,\n    resultMapOpt: Option[scala.collection.Map[Int, Boolean]],\n    defaultValue: Boolean = false,\n  ): Option[Boolean] = {\n    resultMapOpt.map {\n      _.getOrElse(featureName.getFieldId, defaultValue)\n    }\n  }\n\n  private def getDoubleAsIntOptFeature(\n    featureName: EarlybirdFieldConstant,\n    resultMapOpt: Option[scala.collection.Map[Int, Double]]\n  ): Option[Int] = {\n    if (resultMapOpt.exists(_.contains(featureName.getFieldId)))\n      resultMapOpt\n        .map {\n          _.get(featureName.getFieldId)\n        }\n        .flatMap { doubleValue =>\n          doubleValue.map(_.toInt)\n        }\n    else\n      None\n  }\n\n  private def getIntOptFeature(\n    featureName: EarlybirdFieldConstant,\n    resultMapOpt: Option[scala.collection.Map[Int, Int]]\n  ): Option[Int] = {\n    if (resultMapOpt.exists(_.contains(featureName.getFieldId)))\n      resultMapOpt.flatMap {\n        _.get(featureName.getFieldId)\n      }\n    else\n      None\n  }\n\n  def getTweetThriftFeaturesByTweetId(\n    searcherUserId: Long,\n    screenName: Option[String],\n    userLanguages: Seq[scc.ThriftLanguage],\n    uiLanguageCode: Option[String] = None,\n    followedUserIds: Set[Long],\n    mutuallyFollowingUserIds: Set[Long],\n    idToSearchResults: Map[Long, eb.ThriftSearchResult]\n  ): Future[Map[Long, sc.ThriftTweetFeatures]] = {\n\n    val searchResults = idToSearchResults.values.toSeq\n    val inNetworkEngagement =\n      InNetworkEngagement(followedUserIds.toSeq, mutuallyFollowingUserIds, searchResults)\n    val tweetCountByAuthorId = getTweetCountByAuthorId(searchResults)\n    val uiLanguage = getLanguage(uiLanguageCode)\n    val idWithFeaturesSeqFu = OffloadFuturePools.parallelize[\n      eb.ThriftSearchResult,\n      (Long, sc.ThriftTweetFeatures)\n    ](\n      inputSeq = searchResults,\n      transformer = (searchResult: eb.ThriftSearchResult) =>\n        (\n          searchResult.id,\n          getThriftTweetFeaturesFromSearchResult(\n            searcherUserId,\n            screenName,\n            userLanguages,\n            uiLanguage,\n            tweetCountByAuthorId,\n            followedUserIds,\n            mutuallyFollowingUserIds,\n            idToSearchResults,\n            inNetworkEngagement,\n            searchResult\n          )),\n      parallelism = DefaultEBResponseProcessParallelism,\n    )\n    idWithFeaturesSeqFu.map(idWithFeaturesSeq =>\n      idWithFeaturesSeq.map(idWithFeatures => idWithFeatures._1 -> idWithFeatures._2).toMap)\n  }\n\n  def getThriftTweetFeaturesFromSearchResult(\n    searcherUserId: Long,\n    screenName: Option[String],\n    userLanguages: Seq[scc.ThriftLanguage],\n    uiLanguage: Option[scc.ThriftLanguage],\n    tweetCountByAuthorId: Map[Long, Int],\n    followedUserIds: Set[Long],\n    mutuallyFollowingUserIds: Set[Long],\n    idToSearchResults: Map[Long, eb.ThriftSearchResult],\n    inNetworkEngagement: InNetworkEngagement,\n    searchResult: eb.ThriftSearchResult,\n  ): sc.ThriftTweetFeatures = {\n    val applyFeatures = (applyUserIndependentFeatures(\n      searchResult\n    )(_)).andThen(\n      applyUserDependentFeatures(\n        searcherUserId,\n        screenName,\n        userLanguages,\n        uiLanguage,\n        tweetCountByAuthorId,\n        followedUserIds,\n        mutuallyFollowingUserIds,\n        idToSearchResults,\n        inNetworkEngagement,\n        searchResult\n      )(_)\n    )\n    val tweetFeatures = searchResult.tweetFeatures.getOrElse(DefaultEarlybirdFeatures)\n    applyFeatures(tweetFeatures)\n  }\n\n  private[earlybird] def applyUserIndependentFeatures(\n    result: eb.ThriftSearchResult\n  )(\n    thriftTweetFeatures: sc.ThriftTweetFeatures\n  ): sc.ThriftTweetFeatures = {\n\n    val features = result.metadata\n      .map { metadata =>\n        val isRetweet = metadata.isRetweet.getOrElse(false)\n        val isReply = metadata.isReply.getOrElse(false)\n\n        val searchResultSchemaFeatures = metadata.extraMetadata.flatMap(_.features)\n        val booleanSearchResultSchemaFeatures = searchResultSchemaFeatures.flatMap(_.boolValues)\n        val intSearchResultSchemaFeatures = searchResultSchemaFeatures.flatMap(_.intValues)\n        val doubleSearchResultSchemaFeatures = searchResultSchemaFeatures.flatMap(_.doubleValues)\n\n        thriftTweetFeatures.copy(\n          // Info about the Tweet.\n          isRetweet = isRetweet,\n          isOffensive = metadata.isOffensive.getOrElse(false),\n          isReply = isReply,\n          fromVerifiedAccount = metadata.fromVerifiedAccount.getOrElse(false),\n          cardType = metadata.cardType,\n          signature = metadata.signature,\n          language = metadata.language,\n          isAuthorNSFW = metadata.isUserNSFW.getOrElse(false),\n          isAuthorBot = metadata.isUserBot.getOrElse(false),\n          isAuthorSpam = metadata.isUserSpam.getOrElse(false),\n          isSensitiveContent =\n            metadata.extraMetadata.flatMap(_.isSensitiveContent).getOrElse(false),\n          isAuthorProfileEgg = metadata.extraMetadata.flatMap(_.profileIsEggFlag).getOrElse(false),\n          isAuthorNew = metadata.extraMetadata.flatMap(_.isUserNewFlag).getOrElse(false),\n          linkLanguage = metadata.extraMetadata.flatMap(_.linkLanguage).getOrElse(DefaultLanguage),\n          // Info about Tweet content/media.\n          hasCard = metadata.hasCard.getOrElse(false),\n          hasImage = metadata.hasImage.getOrElse(false),\n          hasNews = metadata.hasNews.getOrElse(false),\n          hasVideo = metadata.hasVideo.getOrElse(false),\n          hasConsumerVideo = metadata.hasConsumerVideo.getOrElse(false),\n          hasProVideo = metadata.hasProVideo.getOrElse(false),\n          hasVine = metadata.hasVine.getOrElse(false),\n          hasPeriscope = metadata.hasPeriscope.getOrElse(false),\n          hasNativeVideo = metadata.hasNativeVideo.getOrElse(false),\n          hasNativeImage = metadata.hasNativeImage.getOrElse(false),\n          hasLink = metadata.hasLink.getOrElse(false),\n          hasVisibleLink = metadata.hasVisibleLink.getOrElse(false),\n          hasTrend = metadata.hasTrend.getOrElse(false),\n          hasMultipleHashtagsOrTrends = metadata.hasMultipleHashtagsOrTrends.getOrElse(false),\n          hasQuote = metadata.extraMetadata.flatMap(_.hasQuote),\n          urlsList = metadata.tweetUrls.map {\n            _.map(_.originalUrl)\n          },\n          hasMultipleMedia =\n            metadata.extraMetadata.flatMap(_.hasMultipleMediaFlag).getOrElse(false),\n          visibleTokenRatio = getIntOptFeature(VISIBLE_TOKEN_RATIO, intSearchResultSchemaFeatures),\n          // Various counts.\n          favCount = metadata.favCount.getOrElse(DefaultCount),\n          replyCount = metadata.replyCount.getOrElse(DefaultCount),\n          retweetCount = metadata.retweetCount.getOrElse(DefaultCount),\n          quoteCount = metadata.extraMetadata.flatMap(_.quotedCount),\n          embedsImpressionCount = metadata.embedsImpressionCount.getOrElse(DefaultCount),\n          embedsUrlCount = metadata.embedsUrlCount.getOrElse(DefaultCount),\n          videoViewCount = metadata.videoViewCount.getOrElse(DefaultCount),\n          numMentions = metadata.extraMetadata.flatMap(_.numMentions).getOrElse(DefaultCount),\n          numHashtags = metadata.extraMetadata.flatMap(_.numHashtags).getOrElse(DefaultCount),\n          favCountV2 = metadata.extraMetadata.flatMap(_.favCountV2),\n          replyCountV2 = metadata.extraMetadata.flatMap(_.replyCountV2),\n          retweetCountV2 = metadata.extraMetadata.flatMap(_.retweetCountV2),\n          weightedFavoriteCount = metadata.extraMetadata.flatMap(_.weightedFavCount),\n          weightedReplyCount = metadata.extraMetadata.flatMap(_.weightedReplyCount),\n          weightedRetweetCount = metadata.extraMetadata.flatMap(_.weightedRetweetCount),\n          weightedQuoteCount = metadata.extraMetadata.flatMap(_.weightedQuoteCount),\n          embedsImpressionCountV2 =\n            getDoubleAsIntOptFeature(EMBEDS_IMPRESSION_COUNT_V2, doubleSearchResultSchemaFeatures),\n          embedsUrlCountV2 =\n            getDoubleAsIntOptFeature(EMBEDS_URL_COUNT_V2, doubleSearchResultSchemaFeatures),\n          decayedFavoriteCount =\n            getDoubleAsIntOptFeature(DECAYED_FAVORITE_COUNT, doubleSearchResultSchemaFeatures),\n          decayedRetweetCount =\n            getDoubleAsIntOptFeature(DECAYED_RETWEET_COUNT, doubleSearchResultSchemaFeatures),\n          decayedReplyCount =\n            getDoubleAsIntOptFeature(DECAYED_REPLY_COUNT, doubleSearchResultSchemaFeatures),\n          decayedQuoteCount =\n            getDoubleAsIntOptFeature(DECAYED_QUOTE_COUNT, doubleSearchResultSchemaFeatures),\n          fakeFavoriteCount =\n            getDoubleAsIntOptFeature(FAKE_FAVORITE_COUNT, doubleSearchResultSchemaFeatures),\n          fakeRetweetCount =\n            getDoubleAsIntOptFeature(FAKE_RETWEET_COUNT, doubleSearchResultSchemaFeatures),\n          fakeReplyCount =\n            getDoubleAsIntOptFeature(FAKE_REPLY_COUNT, doubleSearchResultSchemaFeatures),\n          fakeQuoteCount =\n            getDoubleAsIntOptFeature(FAKE_QUOTE_COUNT, doubleSearchResultSchemaFeatures),\n          // Scores.\n          textScore = metadata.textScore.getOrElse(DefaultScore),\n          earlybirdScore = metadata.score.getOrElse(DefaultScore),\n          parusScore = metadata.parusScore.getOrElse(DefaultScore),\n          userRep = metadata.userRep.getOrElse(DefaultScore),\n          pBlockScore = metadata.extraMetadata.flatMap(_.pBlockScore),\n          toxicityScore = metadata.extraMetadata.flatMap(_.toxicityScore),\n          pSpammyTweetScore = metadata.extraMetadata.flatMap(_.pSpammyTweetScore),\n          pReportedTweetScore = metadata.extraMetadata.flatMap(_.pReportedTweetScore),\n          pSpammyTweetContent = metadata.extraMetadata.flatMap(_.spammyTweetContentScore),\n          // Safety Signals\n          labelAbusiveFlag =\n            getBooleanOptFeature(LABEL_ABUSIVE_FLAG, booleanSearchResultSchemaFeatures),\n          labelAbusiveHiRclFlag =\n            getBooleanOptFeature(LABEL_ABUSIVE_HI_RCL_FLAG, booleanSearchResultSchemaFeatures),\n          labelDupContentFlag =\n            getBooleanOptFeature(LABEL_DUP_CONTENT_FLAG, booleanSearchResultSchemaFeatures),\n          labelNsfwHiPrcFlag =\n            getBooleanOptFeature(LABEL_NSFW_HI_PRC_FLAG, booleanSearchResultSchemaFeatures),\n          labelNsfwHiRclFlag =\n            getBooleanOptFeature(LABEL_NSFW_HI_RCL_FLAG, booleanSearchResultSchemaFeatures),\n          labelSpamFlag = getBooleanOptFeature(LABEL_SPAM_FLAG, booleanSearchResultSchemaFeatures),\n          labelSpamHiRclFlag =\n            getBooleanOptFeature(LABEL_SPAM_HI_RCL_FLAG, booleanSearchResultSchemaFeatures),\n          // Periscope Features\n          periscopeExists =\n            getBooleanOptFeature(PERISCOPE_EXISTS, booleanSearchResultSchemaFeatures),\n          periscopeHasBeenFeatured =\n            getBooleanOptFeature(PERISCOPE_HAS_BEEN_FEATURED, booleanSearchResultSchemaFeatures),\n          periscopeIsCurrentlyFeatured = getBooleanOptFeature(\n            PERISCOPE_IS_CURRENTLY_FEATURED,\n            booleanSearchResultSchemaFeatures),\n          periscopeIsFromQualitySource = getBooleanOptFeature(\n            PERISCOPE_IS_FROM_QUALITY_SOURCE,\n            booleanSearchResultSchemaFeatures),\n          periscopeIsLive =\n            getBooleanOptFeature(PERISCOPE_IS_LIVE, booleanSearchResultSchemaFeatures),\n          // Last Engagement Features\n          lastFavSinceCreationHrs =\n            getIntOptFeature(LAST_FAVORITE_SINCE_CREATION_HRS, intSearchResultSchemaFeatures),\n          lastRetweetSinceCreationHrs =\n            getIntOptFeature(LAST_RETWEET_SINCE_CREATION_HRS, intSearchResultSchemaFeatures),\n          lastReplySinceCreationHrs =\n            getIntOptFeature(LAST_REPLY_SINCE_CREATION_HRS, intSearchResultSchemaFeatures),\n          lastQuoteSinceCreationHrs =\n            getIntOptFeature(LAST_QUOTE_SINCE_CREATION_HRS, intSearchResultSchemaFeatures),\n          likedByUserIds = metadata.extraMetadata.flatMap(_.likedByUserIds),\n          isComposerSourceCamera =\n            getBooleanOptFeature(COMPOSER_SOURCE_IS_CAMERA_FLAG, booleanSearchResultSchemaFeatures),\n        )\n      }\n      .getOrElse(thriftTweetFeatures)\n\n    features\n  }\n\n  private def applyUserDependentFeatures(\n    searcherUserId: Long,\n    screenName: Option[String],\n    userLanguages: Seq[scc.ThriftLanguage],\n    uiLanguage: Option[scc.ThriftLanguage],\n    tweetCountByAuthorId: Map[Long, Int],\n    followedUserIds: Set[Long],\n    mutuallyFollowingUserIds: Set[Long],\n    idToSearchResults: Map[Long, eb.ThriftSearchResult],\n    inNetworkEngagement: InNetworkEngagement,\n    result: eb.ThriftSearchResult\n  )(\n    thriftTweetFeatures: sc.ThriftTweetFeatures\n  ): sc.ThriftTweetFeatures = {\n    result.metadata\n      .map { metadata =>\n        val isRetweet = metadata.isRetweet.getOrElse(false)\n        val sourceTweet =\n          if (isRetweet) idToSearchResults.get(metadata.sharedStatusId)\n          else None\n        val mentionsInSourceTweet = sourceTweet.map(getMentions).getOrElse(Seq.empty)\n\n        val isReply = metadata.isReply.getOrElse(false)\n        val replyToSearcher = isReply && (metadata.referencedTweetAuthorId == searcherUserId)\n        val replyOther = isReply && !replyToSearcher\n        val retweetOther = isRetweet && (metadata.referencedTweetAuthorId != searcherUserId)\n        val tweetLanguage = metadata.language.getOrElse(scc.ThriftLanguage.Unknown)\n\n        val referencedTweetAuthorId =\n          if (metadata.referencedTweetAuthorId > 0) Some(metadata.referencedTweetAuthorId) else None\n        val inReplyToUserId = if (!isRetweet) referencedTweetAuthorId else None\n\n        thriftTweetFeatures.copy(\n          // Info about the Tweet.\n          fromSearcher = metadata.fromUserId == searcherUserId,\n          probablyFromFollowedAuthor = followedUserIds.contains(metadata.fromUserId),\n          fromMutualFollow = mutuallyFollowingUserIds.contains(metadata.fromUserId),\n          replySearcher = replyToSearcher,\n          replyOther = replyOther,\n          retweetOther = retweetOther,\n          mentionSearcher = isUserMentioned(screenName, getMentions(result), mentionsInSourceTweet),\n          // Info about Tweet content/media.\n          matchesSearcherMainLang = isUsersMainLanguage(tweetLanguage, userLanguages),\n          matchesSearcherLangs = isUsersLanguage(tweetLanguage, userLanguages),\n          matchesUILang = isUILanguage(tweetLanguage, uiLanguage),\n          // Various counts.\n          prevUserTweetEngagement =\n            metadata.extraMetadata.flatMap(_.prevUserTweetEngagement).getOrElse(DefaultCount),\n          tweetCountFromUserInSnapshot = tweetCountByAuthorId(metadata.fromUserId),\n          bidirectionalReplyCount = inNetworkEngagement.biDirectionalReplyCounts(result.id),\n          unidirectionalReplyCount = inNetworkEngagement.uniDirectionalReplyCounts(result.id),\n          bidirectionalRetweetCount = inNetworkEngagement.biDirectionalRetweetCounts(result.id),\n          unidirectionalRetweetCount = inNetworkEngagement.uniDirectionalRetweetCounts(result.id),\n          conversationCount = inNetworkEngagement.descendantReplyCounts(result.id),\n          directedAtUserIdIsInFirstDegree =\n            if (isReply) inReplyToUserId.map(followedUserIds.contains) else None,\n        )\n      }\n      .getOrElse(thriftTweetFeatures)\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/earlybird/RelevanceSearchUtil.scala",
    "content": "package com.twitter.home_mixer.util.earlybird\n\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant\nimport com.twitter.search.earlybird.{thriftscala => eb}\n\nobject RelevanceSearchUtil {\n\n  val Mentions: String = EarlybirdFieldConstant.MENTIONS_FACET\n  val FacetsToFetch: Seq[String] = Seq(Mentions)\n  val MaxHitsToProcess: Int = 1500\n\n  val MetadataOptions: eb.ThriftSearchResultMetadataOptions = {\n    eb.ThriftSearchResultMetadataOptions(\n      getTweetUrls = true,\n      getResultLocation = false,\n      getLuceneScore = false,\n      getInReplyToStatusId = true,\n      getReferencedTweetAuthorId = true,\n      getMediaBits = true,\n      getAllFeatures = true,\n      returnSearchResultFeatures = true,\n      getExclusiveConversationAuthorId = true\n    )\n  }\n\n  val RelevanceOptions: eb.ThriftSearchRelevanceOptions = {\n    eb.ThriftSearchRelevanceOptions(\n      proximityScoring = true,\n      maxConsecutiveSameUser = Some(2),\n      rankingParams = None,\n      maxHitsToProcess = Some(MaxHitsToProcess),\n      maxUserBlendCount = Some(3),\n      proximityPhraseWeight = 9.0,\n      returnAllResults = Some(true)\n    )\n  }\n\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n        \"home-mixer/thrift/src/main/thrift:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie/RequestFields.scala",
    "content": "package com.twitter.home_mixer.util.tweetypie\n\nimport com.twitter.tweetypie.{thriftscala => tp}\n\nobject RequestFields {\n\n  val CoreTweetFields: Set[tp.TweetInclude] = Set[tp.TweetInclude](\n    tp.TweetInclude.TweetFieldId(tp.Tweet.IdField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.CoreDataField.id)\n  )\n  val MediaFields: Set[tp.TweetInclude] = Set[tp.TweetInclude](\n    tp.TweetInclude.TweetFieldId(tp.Tweet.MediaField.id),\n  )\n  val SelfThreadFields: Set[tp.TweetInclude] = Set[tp.TweetInclude](\n    tp.TweetInclude.TweetFieldId(tp.Tweet.SelfThreadMetadataField.id)\n  )\n  val MentionsTweetFields: Set[tp.TweetInclude] = Set[tp.TweetInclude](\n    tp.TweetInclude.TweetFieldId(tp.Tweet.MentionsField.id)\n  )\n  val SemanticAnnotationTweetFields: Set[tp.TweetInclude] = Set[tp.TweetInclude](\n    tp.TweetInclude.TweetFieldId(tp.Tweet.EscherbirdEntityAnnotationsField.id)\n  )\n  val NsfwLabelFields: Set[tp.TweetInclude] = Set[tp.TweetInclude](\n    // Tweet fields containing NSFW related attributes.\n    tp.TweetInclude.TweetFieldId(tp.Tweet.NsfwHighRecallLabelField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.NsfwHighPrecisionLabelField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.NsfaHighRecallLabelField.id)\n  )\n  val SafetyLabelFields: Set[tp.TweetInclude] = Set[tp.TweetInclude](\n    // Tweet fields containing RTF labels for abuse and spam.\n    tp.TweetInclude.TweetFieldId(tp.Tweet.SpamLabelField.id),\n    tp.TweetInclude.TweetFieldId(tp.Tweet.AbusiveLabelField.id)\n  )\n  val ConversationControlField: Set[tp.TweetInclude] =\n    Set[tp.TweetInclude](tp.TweetInclude.TweetFieldId(tp.Tweet.ConversationControlField.id))\n\n  val TweetTPHydrationFields: Set[tp.TweetInclude] = CoreTweetFields ++\n    NsfwLabelFields ++\n    SafetyLabelFields ++\n    SemanticAnnotationTweetFields ++\n    Set(\n      tp.TweetInclude.TweetFieldId(tp.Tweet.TakedownCountryCodesField.id),\n      // QTs imply a TweetyPie -> SGS request dependency\n      tp.TweetInclude.TweetFieldId(tp.Tweet.QuotedTweetField.id),\n      tp.TweetInclude.TweetFieldId(tp.Tweet.CommunitiesField.id),\n      // Field required for determining if a Tweet was created via News Camera.\n      tp.TweetInclude.TweetFieldId(tp.Tweet.ComposerSourceField.id),\n      tp.TweetInclude.TweetFieldId(tp.Tweet.LanguageField.id)\n    )\n\n  val TweetStaticEntitiesFields: Set[tp.TweetInclude] =\n    MentionsTweetFields ++ CoreTweetFields ++ SemanticAnnotationTweetFields ++ MediaFields\n\n  val ContentFields: Set[tp.TweetInclude] = CoreTweetFields ++ MediaFields ++ SelfThreadFields ++\n    ConversationControlField ++ SemanticAnnotationTweetFields ++\n    Set[tp.TweetInclude](\n      tp.TweetInclude.MediaEntityFieldId(tp.MediaEntity.AdditionalMetadataField.id))\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie/content/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"home-mixer/server/src/main/scala/com/twitter/home_mixer/model\",\n    ],\n)\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie/content/FeatureExtractionHelper.scala",
    "content": "package com.twitter.home_mixer.util.tweetypie.content\n\nimport com.twitter.home_mixer.model.ContentFeatures\nimport com.twitter.tweetypie.{thriftscala => tp}\n\nobject FeatureExtractionHelper {\n\n  def extractFeatures(\n    tweet: tp.Tweet,\n    isExtractMediaEntities: Boolean = true\n  ): ContentFeatures = {\n    val contentFeaturesFromTweet = ContentFeatures.Empty.copy(\n      selfThreadMetadata = tweet.selfThreadMetadata\n    )\n\n    val contentFeaturesWithText = TweetTextFeaturesExtractor.addTextFeaturesFromTweet(\n      contentFeaturesFromTweet,\n      tweet\n    )\n    val contentFeaturesWithMedia = TweetMediaFeaturesExtractor.addMediaFeaturesFromTweet(\n      contentFeaturesWithText,\n      tweet,\n      isExtractMediaEntities\n    )\n\n    contentFeaturesWithMedia.copy(\n      conversationControl = tweet.conversationControl,\n      semanticCoreAnnotations = tweet.escherbirdEntityAnnotations.map(_.entityAnnotations)\n    )\n  }\n}\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie/content/TweetMediaFeaturesExtractor.scala",
    "content": "package com.twitter.home_mixer.util.tweetypie.content\n\nimport com.twitter.home_mixer.model.ContentFeatures\nimport com.twitter.mediaservices.commons.mediainformation.{thriftscala => mi}\nimport com.twitter.mediaservices.commons.tweetmedia.{thriftscala => tm}\nimport com.twitter.mediaservices.commons.{thriftscala => ms}\nimport com.twitter.tweetypie.{thriftscala => tp}\nimport scala.collection.Map\n\nobject TweetMediaFeaturesExtractor {\n\n  private val ImageCategories = Set(\n    ms.MediaCategory.TweetImage.value,\n    ms.MediaCategory.TweetGif.value\n  )\n  private val VideoCategories = Set(\n    ms.MediaCategory.TweetVideo.value,\n    ms.MediaCategory.AmplifyVideo.value\n  )\n\n  def hasImage(tweet: tp.Tweet): Boolean = hasMediaByCategory(tweet, ImageCategories)\n\n  def hasVideo(tweet: tp.Tweet): Boolean = hasMediaByCategory(tweet, VideoCategories)\n\n  private def hasMediaByCategory(tweet: tp.Tweet, categories: Set[Int]): Boolean = {\n    tweet.media.exists { mediaEntities =>\n      mediaEntities.exists { mediaEntity =>\n        mediaEntity.mediaKey.map(_.mediaCategory).exists { mediaCategory =>\n          categories.contains(mediaCategory.value)\n        }\n      }\n    }\n  }\n\n  def getMediaIds(tweet: tp.Tweet): Seq[Long] = {\n    val tweetMediaIdsOpt = tweet.media.map { mediaEntities =>\n      mediaEntities.map { mediaEntity =>\n        mediaEntity.mediaId\n      }\n    }\n    tweetMediaIdsOpt.getOrElse(Seq.empty)\n  }\n\n  def addMediaFeaturesFromTweet(\n    inputFeatures: ContentFeatures,\n    tweet: tp.Tweet,\n    isExtractMediaEntities: Boolean = true\n  ): ContentFeatures = {\n    val featuresWithMediaEntity =\n      if (isExtractMediaEntities) {\n        tweet.media\n          .map { mediaEntities =>\n            val sizeFeatures = getSizeFeatures(mediaEntities)\n            val playbackFeatures = getPlaybackFeatures(mediaEntities)\n            val mediaWidths = sizeFeatures.map(_.width.toShort)\n            val mediaHeights = sizeFeatures.map(_.height.toShort)\n            val resizeMethods = sizeFeatures.map(_.resizeMethod.toShort)\n            val faceMapAreas = getFaceMapAreas(mediaEntities)\n            val sortedColorPalette = getSortedColorPalette(mediaEntities)\n            val stickerFeatures = getStickerFeatures(mediaEntities)\n            val mediaOriginProviders = getMediaOriginProviders(mediaEntities)\n            val isManaged = getIsManaged(mediaEntities)\n            val is360 = getIs360(mediaEntities)\n            val viewCount = getViewCount(mediaEntities)\n            val userDefinedProductMetadataFeatures =\n              getUserDefinedProductMetadataFeatures(mediaEntities)\n            val isMonetizable =\n              getOptBooleanFromSeqOpt(userDefinedProductMetadataFeatures.map(_.isMonetizable))\n            val isEmbeddable =\n              getOptBooleanFromSeqOpt(userDefinedProductMetadataFeatures.map(_.isEmbeddable))\n            val hasSelectedPreviewImage =\n              getOptBooleanFromSeqOpt(\n                userDefinedProductMetadataFeatures.map(_.hasSelectedPreviewImage))\n\n            val hasImage = Some(\n              mediaEntities.exists { entity =>\n                entity.mediaKey.exists { key =>\n                  ImageCategories.contains(key.mediaCategory.value)\n                }\n              }\n            )\n            val hasVideo = Some(\n              mediaEntities.exists { entity =>\n                entity.mediaKey.exists { key =>\n                  VideoCategories.contains(key.mediaCategory.value)\n                }\n              }\n            )\n\n            val hasTitle =\n              getOptBooleanFromSeqOpt(userDefinedProductMetadataFeatures.map(_.hasTitle))\n            val hasDescription =\n              getOptBooleanFromSeqOpt(userDefinedProductMetadataFeatures.map(_.hasDescription))\n            val hasVisitSiteCallToAction = getOptBooleanFromSeqOpt(\n              userDefinedProductMetadataFeatures.map(_.hasVisitSiteCallToAction))\n            val hasAppInstallCallToAction = getOptBooleanFromSeqOpt(\n              userDefinedProductMetadataFeatures.map(_.hasAppInstallCallToAction))\n            val hasWatchNowCallToAction =\n              getOptBooleanFromSeqOpt(\n                userDefinedProductMetadataFeatures.map(_.hasWatchNowCallToAction))\n\n            inputFeatures.copy(\n              videoDurationMs = playbackFeatures.durationMs,\n              bitRate = playbackFeatures.bitRate,\n              aspectRatioNum = playbackFeatures.aspectRatioNum,\n              aspectRatioDen = playbackFeatures.aspectRatioDen,\n              widths = Some(mediaWidths),\n              heights = Some(mediaHeights),\n              resizeMethods = Some(resizeMethods),\n              faceAreas = Some(faceMapAreas),\n              dominantColorRed = sortedColorPalette.headOption.map(_.rgb.red),\n              dominantColorBlue = sortedColorPalette.headOption.map(_.rgb.blue),\n              dominantColorGreen = sortedColorPalette.headOption.map(_.rgb.green),\n              dominantColorPercentage = sortedColorPalette.headOption.map(_.percentage),\n              numColors = Some(sortedColorPalette.size.toShort),\n              stickerIds = Some(stickerFeatures),\n              mediaOriginProviders = Some(mediaOriginProviders),\n              isManaged = Some(isManaged),\n              is360 = Some(is360),\n              viewCount = viewCount,\n              isMonetizable = isMonetizable,\n              isEmbeddable = isEmbeddable,\n              hasSelectedPreviewImage = hasSelectedPreviewImage,\n              hasTitle = hasTitle,\n              hasDescription = hasDescription,\n              hasVisitSiteCallToAction = hasVisitSiteCallToAction,\n              hasAppInstallCallToAction = hasAppInstallCallToAction,\n              hasWatchNowCallToAction = hasWatchNowCallToAction,\n              hasImage = hasImage,\n              hasVideo = hasVideo\n            )\n          }\n          .getOrElse(inputFeatures)\n      } else inputFeatures\n\n    val featuresWithMediaTags = tweet.mediaTags\n      .map { mediaTags =>\n        val mediaTagScreenNames = getMediaTagScreenNames(mediaTags.tagMap)\n        val numMediaTags = mediaTagScreenNames.size\n\n        featuresWithMediaEntity.copy(\n          mediaTagScreenNames = Some(mediaTagScreenNames),\n          numMediaTags = Some(numMediaTags.toShort)\n        )\n      }\n      .getOrElse(featuresWithMediaEntity)\n\n    featuresWithMediaTags\n      .copy(media = tweet.media)\n  }\n\n  private def getSizeFeatures(mediaEntities: Seq[tp.MediaEntity]): Seq[MediaSizeFeatures] = {\n    mediaEntities.map { mediaEntity =>\n      mediaEntity.sizes.foldLeft(MediaSizeFeatures(0, 0, 0))((accDimensions, dimensions) =>\n        MediaSizeFeatures(\n          width = math.max(dimensions.width, accDimensions.width),\n          height = math.max(dimensions.height, accDimensions.height),\n          resizeMethod = math.max(dimensions.resizeMethod.getValue, accDimensions.resizeMethod)\n        ))\n    }\n  }\n\n  private def getPlaybackFeatures(mediaEntities: Seq[tp.MediaEntity]): PlaybackFeatures = {\n    val allPlaybackFeatures = mediaEntities\n      .flatMap { mediaEntity =>\n        mediaEntity.mediaInfo map {\n          case videoEntity: tm.MediaInfo.VideoInfo =>\n            PlaybackFeatures(\n              durationMs = Some(videoEntity.videoInfo.durationMillis),\n              bitRate = videoEntity.videoInfo.variants.maxBy(_.bitRate).bitRate,\n              aspectRatioNum = Some(videoEntity.videoInfo.aspectRatio.numerator),\n              aspectRatioDen = Some(videoEntity.videoInfo.aspectRatio.denominator)\n            )\n          case gifEntity: tm.MediaInfo.AnimatedGifInfo =>\n            PlaybackFeatures(\n              durationMs = None,\n              bitRate = gifEntity.animatedGifInfo.variants.maxBy(_.bitRate).bitRate,\n              aspectRatioNum = Some(gifEntity.animatedGifInfo.aspectRatio.numerator),\n              aspectRatioDen = Some(gifEntity.animatedGifInfo.aspectRatio.denominator)\n            )\n          case _ => PlaybackFeatures(None, None, None, None)\n        }\n      }\n      .collect {\n        case playbackFeatures: PlaybackFeatures => playbackFeatures\n      }\n\n    if (allPlaybackFeatures.nonEmpty) allPlaybackFeatures.minBy(_.durationMs)\n    else PlaybackFeatures(None, None, None, None)\n  }\n\n  private def getMediaTagScreenNames(tagMap: Map[Long, Seq[tp.MediaTag]]): Seq[String] =\n    tagMap.values\n      .flatMap(seqMediaTag => seqMediaTag.flatMap(_.screenName))\n      .toSeq\n\n  // Areas of the faces identified in the media entities\n  private def getFaceMapAreas(mediaEntities: Seq[tp.MediaEntity]): Seq[Int] = {\n    for {\n      mediaEntity <- mediaEntities\n      metadata <- mediaEntity.additionalMetadata.toSeq\n      faceData <- metadata.faceData\n      faces <- faceData.faces\n    } yield {\n      faces\n        .getOrElse(\"orig\", Seq.empty[mi.Face])\n        .flatMap(f => f.boundingBox.map(bb => bb.width * bb.height))\n    }\n  }.flatten\n\n  // All ColorPalettes in the media sorted by the percentage in descending order\n  private def getSortedColorPalette(\n    mediaEntities: Seq[tp.MediaEntity]\n  ): Seq[mi.ColorPaletteItem] = {\n    for {\n      mediaEntity <- mediaEntities\n      metadata <- mediaEntity.additionalMetadata.toSeq\n      colorInfo <- metadata.colorInfo\n    } yield {\n      colorInfo.palette\n    }\n  }.flatten.sortBy(-_.percentage)\n\n  // Id's of stickers applied by the user\n  private def getStickerFeatures(mediaEntities: Seq[tp.MediaEntity]): Seq[Long] = {\n    for {\n      mediaEntity <- mediaEntities\n      metadata <- mediaEntity.additionalMetadata.toSeq\n      stickerInfo <- metadata.stickerInfo\n    } yield {\n      stickerInfo.stickers.map(_.id)\n    }\n  }.flatten\n\n  // 3rd party media providers. eg. giphy for gifs\n  private def getMediaOriginProviders(mediaEntities: Seq[tp.MediaEntity]): Seq[String] =\n    for {\n      mediaEntity <- mediaEntities\n      metadata <- mediaEntity.additionalMetadata.toSeq\n      mediaOrigin <- metadata.foundMediaOrigin\n    } yield {\n      mediaOrigin.provider\n    }\n\n  private def getIsManaged(mediaEntities: Seq[tp.MediaEntity]): Boolean = {\n    for {\n      mediaEntity <- mediaEntities\n      metadata <- mediaEntity.additionalMetadata.toSeq\n      managementInfo <- metadata.managementInfo\n    } yield {\n      managementInfo.managed\n    }\n  }.contains(true)\n\n  private def getIs360(mediaEntities: Seq[tp.MediaEntity]): Boolean = {\n    for {\n      mediaEntity <- mediaEntities\n      metadata <- mediaEntity.additionalMetadata.toSeq\n      info360 <- metadata.info360\n    } yield {\n      info360.is360\n    }\n  }.contains(Some(true))\n\n  private def getViewCount(mediaEntities: Seq[tp.MediaEntity]): Option[Long] = {\n    for {\n      mediaEntity <- mediaEntities\n      metadata <- mediaEntity.additionalMetadata.toSeq\n      engagementInfo <- metadata.engagementInfo\n      viewCounts <- engagementInfo.viewCount\n    } yield {\n      viewCounts\n    }\n  }.reduceOption(_ max _)\n\n  // metadata defined by the user when uploading the image\n  private def getUserDefinedProductMetadataFeatures(\n    mediaEntities: Seq[tp.MediaEntity]\n  ): Seq[UserDefinedProductMetadataFeatures] =\n    for {\n      mediaEntity <- mediaEntities\n      userDefinedMetadata <- mediaEntity.metadata\n    } yield {\n      UserDefinedProductMetadataFeatures(\n        isMonetizable = userDefinedMetadata.monetizable,\n        isEmbeddable = userDefinedMetadata.embeddable,\n        hasSelectedPreviewImage = Some(userDefinedMetadata.previewImage.nonEmpty),\n        hasTitle = userDefinedMetadata.title.map(_.nonEmpty),\n        hasDescription = userDefinedMetadata.description.map(_.nonEmpty),\n        hasVisitSiteCallToAction = userDefinedMetadata.callToActions.map(_.visitSite.nonEmpty),\n        hasAppInstallCallToAction = userDefinedMetadata.callToActions.map(_.appInstall.nonEmpty),\n        hasWatchNowCallToAction = userDefinedMetadata.callToActions.map(_.watchNow.nonEmpty)\n      )\n    }\n\n  private def getOptBooleanFromSeqOpt(\n    seqOpt: Seq[Option[Boolean]]\n  ): Option[Boolean] = Some(\n    seqOpt.exists(boolOpt => boolOpt.contains(true))\n  )\n}\n\ncase class MediaSizeFeatures(width: Int, height: Int, resizeMethod: Int)\n\ncase class PlaybackFeatures(\n  durationMs: Option[Int],\n  bitRate: Option[Int],\n  aspectRatioNum: Option[Short],\n  aspectRatioDen: Option[Short])\n\ncase class UserDefinedProductMetadataFeatures(\n  isMonetizable: Option[Boolean],\n  isEmbeddable: Option[Boolean],\n  hasSelectedPreviewImage: Option[Boolean],\n  hasTitle: Option[Boolean],\n  hasDescription: Option[Boolean],\n  hasVisitSiteCallToAction: Option[Boolean],\n  hasAppInstallCallToAction: Option[Boolean],\n  hasWatchNowCallToAction: Option[Boolean])\n"
  },
  {
    "path": "home-mixer/server/src/main/scala/com/twitter/home_mixer/util/tweetypie/content/TweetTextFeaturesExtractor.scala",
    "content": "package com.twitter.home_mixer.util.tweetypie.content\n\nimport com.twitter.home_mixer.model.ContentFeatures\nimport com.twitter.tweetypie.{thriftscala => tp}\n\nobject TweetTextFeaturesExtractor {\n\n  private val QUESTION_MARK_CHARS = Set(\n    '\\u003F', '\\u00BF', '\\u037E', '\\u055E', '\\u061F', '\\u1367', '\\u1945', '\\u2047', '\\u2048',\n    '\\u2049', '\\u2753', '\\u2754', '\\u2CFA', '\\u2CFB', '\\u2E2E', '\\uA60F', '\\uA6F7', '\\uFE16',\n    '\\uFE56', '\\uFF1F', '\\u1114', '\\u1E95'\n  )\n  private val NEW_LINE_REGEX = \"\\r\\n|\\r|\\n\".r\n\n  def addTextFeaturesFromTweet(\n    inputFeatures: ContentFeatures,\n    tweet: tp.Tweet\n  ): ContentFeatures = {\n    tweet.coreData\n      .map { coreData =>\n        val tweetText = coreData.text\n\n        inputFeatures.copy(\n          hasQuestion = hasQuestionCharacter(tweetText),\n          length = getLength(tweetText).toShort,\n          numCaps = getCaps(tweetText).toShort,\n          numWhiteSpaces = getSpaces(tweetText).toShort,\n          numNewlines = Some(getNumNewlines(tweetText)),\n        )\n      }\n      .getOrElse(inputFeatures)\n  }\n\n  def getLength(text: String): Int =\n    text.codePointCount(0, text.length())\n\n  def getCaps(text: String): Int = text.count(Character.isUpperCase)\n\n  def getSpaces(text: String): Int = text.count(Character.isWhitespace)\n\n  def hasQuestionCharacter(text: String): Boolean = text.exists(QUESTION_MARK_CHARS.contains)\n\n  def getNumNewlines(text: String): Short = NEW_LINE_REGEX.findAllIn(text).length.toShort\n}\n"
  },
  {
    "path": "navi/README.md",
    "content": "# Navi: High-Performance Machine Learning Serving Server in Rust\n\nNavi is a high-performance, versatile machine learning serving server implemented in Rust and tailored for production usage. It's designed to efficiently serve within the Twitter tech stack, offering top-notch performance while focusing on core features.\n\n## Key Features\n\n- **Minimalist Design Optimized for Production Use Cases**: Navi delivers ultra-high performance, stability, and availability, engineered to handle real-world application demands with a streamlined codebase.\n- **gRPC API Compatibility with TensorFlow Serving**: Seamless integration with existing TensorFlow Serving clients via its gRPC API, enabling easy integration, smooth deployment, and scaling in production environments.\n- **Plugin Architecture for Different Runtimes**: Navi's pluggable architecture supports various machine learning runtimes, providing adaptability and extensibility for diverse use cases. Out-of-the-box support is available for TensorFlow and Onnx Runtime, with PyTorch in an experimental state.\n\n## Current State\n\nWhile Navi's features may not be as comprehensive as its open-source counterparts, its performance-first mindset makes it highly efficient. \n- Navi for TensorFlow is currently the most feature-complete, supporting multiple input tensors of different types (float, int, string, etc.).\n- Navi for Onnx primarily supports one input tensor of type string, used in Twitter's home recommendation with a proprietary BatchPredictRequest format.\n- Navi for Pytorch is compilable and runnable but not yet production-ready in terms of performance and stability.\n\n## Directory Structure\n\n- `navi`: The main code repository for Navi\n- `dr_transform`: Twitter-specific converter that converts BatchPredictionRequest Thrift to ndarray\n- `segdense`: Twitter-specific config to specify how to retrieve feature values from BatchPredictionRequest\n- `thrift_bpr_adapter`: generated thrift code for BatchPredictionRequest\n\n## Content\nWe have included all *.rs source code files that make up the main Navi binaries for you to examine. However, we have not included the test and benchmark code, or various configuration files, due to data security concerns.\n\n## Run\nIn navi/navi, you can run the following commands:\n- `scripts/run_tf2.sh` for [TensorFlow](https://www.tensorflow.org/)\n- `scripts/run_onnx.sh` for [Onnx](https://onnx.ai/)\n\nDo note that you need to create a models directory and create some versions, preferably using epoch time, e.g., `1679693908377`.\nso the models structure looks like:\n  models/\n       -web_click\n        - 1809000\n        - 1809010\n\n## Build\nYou can adapt the above scripts to build using Cargo.\n"
  },
  {
    "path": "navi/dr_transform/Cargo.toml",
    "content": "[package]\nname = \"dr_transform\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[dependencies]\nserde = { version = \"1.0\", features = [\"derive\"] }\nserde_json = \"1.0\"\njson = \"0.12.4\"\nbpr_thrift = { path = \"../thrift_bpr_adapter/thrift/\"}\nsegdense = { path = \"../segdense/\"}\nthrift = \"0.17.0\"\nndarray = \"0.15\"\nbase64 = \"0.20.0\"\nnpyz = \"0.7.2\"\nlog = \"0.4.17\"\nenv_logger = \"0.9.0\"\nprometheus = \"0.13.1\"\nonce_cell = \"1.17.0\"\nrand = \"0.8.5\"\nitertools = \"0.10.5\"\nanyhow = \"1.0.70\"\n[target.'cfg(not(target_os=\"linux\"))'.dependencies]\nort = {git =\"https://github.com/pykeio/ort.git\", features=[\"profiling\"], tag=\"v1.14.6\"}\n[target.'cfg(target_os=\"linux\")'.dependencies]\nort = {git =\"https://github.com/pykeio/ort.git\", features=[\"profiling\", \"tensorrt\", \"cuda\", \"copy-dylibs\"], tag=\"v1.14.6\"}\n[dev-dependencies]\ncriterion = \"0.3.0\"\n\n[[bench]]\nname = \"bpr_benchmark\"\nharness = false\n"
  },
  {
    "path": "navi/dr_transform/src/all_config.rs",
    "content": "use serde::{Deserialize, Serialize};\n\nuse serde_json::Error;\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct AllConfig {\n    #[serde(rename = \"train_data\")]\n    pub train_data: TrainData,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct TrainData {\n    #[serde(rename = \"seg_dense_schema\")]\n    pub seg_dense_schema: SegDenseSchema,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct SegDenseSchema {\n    #[serde(rename = \"renamed_features\")]\n    pub renamed_features: RenamedFeatures,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct RenamedFeatures {\n    pub continuous: String,\n    pub binary: String,\n    pub discrete: String,\n    #[serde(rename = \"author_embedding\")]\n    pub author_embedding: String,\n    #[serde(rename = \"user_embedding\")]\n    pub user_embedding: String,\n    #[serde(rename = \"user_eng_embedding\")]\n    pub user_eng_embedding: String,\n    #[serde(rename = \"meta__author_id\")]\n    pub meta_author_id: String,\n    #[serde(rename = \"meta__user_id\")]\n    pub meta_user_id: String,\n    #[serde(rename = \"meta__tweet_id\")]\n    pub meta_tweet_id: String,\n}\n\npub fn parse(json_str: &str) -> Result<AllConfig, Error> {\n    let all_config: AllConfig = serde_json::from_str(json_str)?;\n    Ok(all_config)\n}\n"
  },
  {
    "path": "navi/dr_transform/src/converter.rs",
    "content": "use std::collections::BTreeSet;\nuse std::fmt::{self, Debug, Display};\nuse std::fs;\n\nuse crate::all_config;\nuse crate::all_config::AllConfig;\nuse anyhow::{bail, Context};\nuse bpr_thrift::data::DataRecord;\nuse bpr_thrift::prediction_service::BatchPredictionRequest;\nuse bpr_thrift::tensor::GeneralTensor;\nuse log::debug;\nuse ndarray::Array2;\nuse once_cell::sync::OnceCell;\nuse ort::tensor::InputTensor;\nuse prometheus::{HistogramOpts, HistogramVec};\nuse segdense::mapper::{FeatureMapper, MapReader};\nuse segdense::segdense_transform_spec_home_recap_2022::{DensificationTransformSpec, Root};\nuse segdense::util;\nuse thrift::protocol::{TBinaryInputProtocol, TSerializable};\nuse thrift::transport::TBufferChannel;\n\npub fn log_feature_match(\n    dr: &DataRecord,\n    seg_dense_config: &DensificationTransformSpec,\n    dr_type: String,\n) {\n    // Note the following algorithm matches features from config using linear search.\n    // Also the record source is MinDataRecord. This includes only binary and continous features for now.\n\n    for (feature_id, feature_value) in dr.continuous_features.as_ref().unwrap() {\n        debug!(\n            \"{} - Continous Datarecord => Feature ID: {}, Feature value: {}\",\n            dr_type, feature_id, feature_value\n        );\n        for input_feature in &seg_dense_config.cont.input_features {\n            if input_feature.feature_id == *feature_id {\n                debug!(\"Matching input feature: {:?}\", input_feature)\n            }\n        }\n    }\n\n    for feature_id in dr.binary_features.as_ref().unwrap() {\n        debug!(\n            \"{} - Binary Datarecord => Feature ID: {}\",\n            dr_type, feature_id\n        );\n        for input_feature in &seg_dense_config.binary.input_features {\n            if input_feature.feature_id == *feature_id {\n                debug!(\"Found input feature: {:?}\", input_feature)\n            }\n        }\n    }\n}\n\npub fn log_feature_matches(drs: &Vec<DataRecord>, seg_dense_config: &DensificationTransformSpec) {\n    for dr in drs {\n        log_feature_match(dr, seg_dense_config, String::from(\"individual\"));\n    }\n}\n\npub trait Converter: Send + Sync + Debug + 'static + Display {\n    fn convert(&self, input: Vec<Vec<u8>>) -> (Vec<InputTensor>, Vec<usize>);\n}\n\n#[derive(Debug)]\n#[allow(dead_code)]\npub struct BatchPredictionRequestToTorchTensorConverter {\n    all_config: AllConfig,\n    seg_dense_config: Root,\n    all_config_path: String,\n    seg_dense_config_path: String,\n    feature_mapper: FeatureMapper,\n    user_embedding_feature_id: i64,\n    user_eng_embedding_feature_id: i64,\n    author_embedding_feature_id: i64,\n    discrete_features_to_report: BTreeSet<i64>,\n    continuous_features_to_report: BTreeSet<i64>,\n    discrete_feature_metrics: &'static HistogramVec,\n    continuous_feature_metrics: &'static HistogramVec,\n}\n\nimpl Display for BatchPredictionRequestToTorchTensorConverter {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        write!(\n            f,\n            \"all_config_path: {}, seg_dense_config_path:{}\",\n            self.all_config_path, self.seg_dense_config_path\n        )\n    }\n}\n\nimpl BatchPredictionRequestToTorchTensorConverter {\n    pub fn new(\n        model_dir: &str,\n        model_version: &str,\n        reporting_feature_ids: Vec<(i64, &str)>,\n        register_metric_fn: Option<impl Fn(&HistogramVec)>,\n    ) -> anyhow::Result<BatchPredictionRequestToTorchTensorConverter> {\n        let all_config_path = format!(\"{}/{}/all_config.json\", model_dir, model_version);\n        let seg_dense_config_path = format!(\n            \"{}/{}/segdense_transform_spec_home_recap_2022.json\",\n            model_dir, model_version\n        );\n        let seg_dense_config = util::load_config(&seg_dense_config_path)?;\n        let all_config = all_config::parse(\n            &fs::read_to_string(&all_config_path)\n                .with_context(|| \"error loading all_config.json - \")?,\n        )?;\n\n        let feature_mapper = util::load_from_parsed_config(seg_dense_config.clone())?;\n\n        let user_embedding_feature_id = Self::get_feature_id(\n            &all_config\n                .train_data\n                .seg_dense_schema\n                .renamed_features\n                .user_embedding,\n            &seg_dense_config,\n        );\n        let user_eng_embedding_feature_id = Self::get_feature_id(\n            &all_config\n                .train_data\n                .seg_dense_schema\n                .renamed_features\n                .user_eng_embedding,\n            &seg_dense_config,\n        );\n        let author_embedding_feature_id = Self::get_feature_id(\n            &all_config\n                .train_data\n                .seg_dense_schema\n                .renamed_features\n                .author_embedding,\n            &seg_dense_config,\n        );\n        static METRICS: OnceCell<(HistogramVec, HistogramVec)> = OnceCell::new();\n        let (discrete_feature_metrics, continuous_feature_metrics) = METRICS.get_or_init(|| {\n            let discrete = HistogramVec::new(\n                HistogramOpts::new(\":navi:feature_id:discrete\", \"Discrete Feature ID values\")\n                    .buckets(Vec::from(&[\n                        0.0, 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0, 110.0,\n                        120.0, 130.0, 140.0, 150.0, 160.0, 170.0, 180.0, 190.0, 200.0, 250.0,\n                        300.0, 500.0, 1000.0, 10000.0, 100000.0,\n                    ] as &'static [f64])),\n                &[\"feature_id\"],\n            )\n            .expect(\"metric cannot be created\");\n            let continuous = HistogramVec::new(\n                HistogramOpts::new(\n                    \":navi:feature_id:continuous\",\n                    \"continuous Feature ID values\",\n                )\n                .buckets(Vec::from(&[\n                    0.0, 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0, 110.0, 120.0,\n                    130.0, 140.0, 150.0, 160.0, 170.0, 180.0, 190.0, 200.0, 250.0, 300.0, 500.0,\n                    1000.0, 10000.0, 100000.0,\n                ] as &'static [f64])),\n                &[\"feature_id\"],\n            )\n            .expect(\"metric cannot be created\");\n            register_metric_fn.map(|r| {\n                r(&discrete);\n                r(&continuous);\n            });\n            (discrete, continuous)\n        });\n\n        let mut discrete_features_to_report = BTreeSet::new();\n        let mut continuous_features_to_report = BTreeSet::new();\n\n        for (feature_id, feature_type) in reporting_feature_ids.iter() {\n            match *feature_type {\n                \"discrete\" => discrete_features_to_report.insert(feature_id.clone()),\n                \"continuous\" => continuous_features_to_report.insert(feature_id.clone()),\n                _ => bail!(\n                    \"Invalid feature type {} for reporting metrics!\",\n                    feature_type\n                ),\n            };\n        }\n\n        Ok(BatchPredictionRequestToTorchTensorConverter {\n            all_config,\n            seg_dense_config,\n            all_config_path,\n            seg_dense_config_path,\n            feature_mapper,\n            user_embedding_feature_id,\n            user_eng_embedding_feature_id,\n            author_embedding_feature_id,\n            discrete_features_to_report,\n            continuous_features_to_report,\n            discrete_feature_metrics,\n            continuous_feature_metrics,\n        })\n    }\n\n    fn get_feature_id(feature_name: &str, seg_dense_config: &Root) -> i64 {\n        // given a feature name, we get the complex feature type id\n        for feature in &seg_dense_config.complex_feature_type_transform_spec {\n            if feature.full_feature_name == feature_name {\n                return feature.feature_id;\n            }\n        }\n        -1\n    }\n\n    fn parse_batch_prediction_request(bytes: Vec<u8>) -> BatchPredictionRequest {\n        // parse batch prediction request into a struct from byte array repr.\n        let mut bc = TBufferChannel::with_capacity(bytes.len(), 0);\n        bc.set_readable_bytes(&bytes);\n        let mut protocol = TBinaryInputProtocol::new(bc, true);\n        BatchPredictionRequest::read_from_in_protocol(&mut protocol).unwrap()\n    }\n\n    fn get_embedding_tensors(\n        &self,\n        bprs: &[BatchPredictionRequest],\n        feature_id: i64,\n        batch_size: &[usize],\n    ) -> Array2<f32> {\n        // given an embedding feature id, extract the float tensor array into tensors.\n        let cols: usize = 200;\n        let rows: usize = batch_size[batch_size.len() - 1];\n        let total_size = rows * cols;\n\n        let mut working_set = vec![0 as f32; total_size];\n        let mut bpr_start = 0;\n        for (bpr, &bpr_end) in bprs.iter().zip(batch_size) {\n            if bpr.common_features.is_some() {\n                if bpr.common_features.as_ref().unwrap().tensors.is_some() {\n                    if bpr\n                        .common_features\n                        .as_ref()\n                        .unwrap()\n                        .tensors\n                        .as_ref()\n                        .unwrap()\n                        .contains_key(&feature_id)\n                    {\n                        let source_tensor = bpr\n                            .common_features\n                            .as_ref()\n                            .unwrap()\n                            .tensors\n                            .as_ref()\n                            .unwrap()\n                            .get(&feature_id)\n                            .unwrap();\n                        let tensor = match source_tensor {\n                            GeneralTensor::FloatTensor(float_tensor) =>\n                            //Tensor::of_slice(\n                            {\n                                float_tensor\n                                    .floats\n                                    .iter()\n                                    .map(|x| x.into_inner() as f32)\n                                    .collect::<Vec<_>>()\n                            }\n                            _ => vec![0 as f32; cols],\n                        };\n\n                        // since the tensor is found in common feature, add it in all batches\n                        for row in bpr_start..bpr_end {\n                            for col in 0..cols {\n                                working_set[row * cols + col] = tensor[col];\n                            }\n                        }\n                    }\n                }\n            }\n            // find the feature in individual feature list and add to corresponding batch.\n            for (index, datarecord) in bpr.individual_features_list.iter().enumerate() {\n                if datarecord.tensors.is_some()\n                    && datarecord\n                        .tensors\n                        .as_ref()\n                        .unwrap()\n                        .contains_key(&feature_id)\n                {\n                    let source_tensor = datarecord\n                        .tensors\n                        .as_ref()\n                        .unwrap()\n                        .get(&feature_id)\n                        .unwrap();\n                    let tensor = match source_tensor {\n                        GeneralTensor::FloatTensor(float_tensor) => float_tensor\n                            .floats\n                            .iter()\n                            .map(|x| x.into_inner() as f32)\n                            .collect::<Vec<_>>(),\n                        _ => vec![0 as f32; cols],\n                    };\n                    for col in 0..cols {\n                        working_set[(bpr_start + index) * cols + col] = tensor[col];\n                    }\n                }\n            }\n            bpr_start = bpr_end;\n        }\n        Array2::<f32>::from_shape_vec([rows, cols], working_set).unwrap()\n    }\n\n    // Todo : Refactor, create a generic version with different type and field accessors\n    //   Example paramterize and then instiantiate the following\n    //           (FLOAT --> FLOAT, DataRecord.continuous_feature)\n    //           (BOOL --> INT64, DataRecord.binary_feature)\n    //           (INT64 --> INT64, DataRecord.discrete_feature)\n    fn get_continuous(&self, bprs: &[BatchPredictionRequest], batch_ends: &[usize]) -> InputTensor {\n        // These need to be part of model schema\n        let rows: usize = batch_ends[batch_ends.len() - 1];\n        let cols: usize = 5293;\n        let full_size: usize = rows * cols;\n        let default_val = f32::NAN;\n\n        let mut tensor = vec![default_val; full_size];\n\n        let mut bpr_start = 0;\n        for (bpr, &bpr_end) in bprs.iter().zip(batch_ends) {\n            // Common features\n            if bpr.common_features.is_some()\n                && bpr\n                    .common_features\n                    .as_ref()\n                    .unwrap()\n                    .continuous_features\n                    .is_some()\n            {\n                let common_features = bpr\n                    .common_features\n                    .as_ref()\n                    .unwrap()\n                    .continuous_features\n                    .as_ref()\n                    .unwrap();\n\n                for feature in common_features {\n                    match self.feature_mapper.get(feature.0) {\n                        Some(f_info) => {\n                            let idx = f_info.index_within_tensor as usize;\n                            if idx < cols {\n                                // Set value in each row\n                                for r in bpr_start..bpr_end {\n                                    let flat_index: usize = r * cols + idx;\n                                    tensor[flat_index] = feature.1.into_inner() as f32;\n                                }\n                            }\n                        }\n                        None => (),\n                    }\n                    if self.continuous_features_to_report.contains(feature.0) {\n                        self.continuous_feature_metrics\n                            .with_label_values(&[feature.0.to_string().as_str()])\n                            .observe(feature.1.into_inner())\n                    } else if self.discrete_features_to_report.contains(feature.0) {\n                        self.discrete_feature_metrics\n                            .with_label_values(&[feature.0.to_string().as_str()])\n                            .observe(feature.1.into_inner())\n                    }\n                }\n            }\n\n            // Process the batch of datarecords\n            for r in bpr_start..bpr_end {\n                let dr: &DataRecord =\n                    &bpr.individual_features_list[usize::try_from(r - bpr_start).unwrap()];\n                if dr.continuous_features.is_some() {\n                    for feature in dr.continuous_features.as_ref().unwrap() {\n                        match self.feature_mapper.get(&feature.0) {\n                            Some(f_info) => {\n                                let idx = f_info.index_within_tensor as usize;\n                                let flat_index: usize = r * cols + idx;\n                                if flat_index < tensor.len() && idx < cols {\n                                    tensor[flat_index] = feature.1.into_inner() as f32;\n                                }\n                            }\n                            None => (),\n                        }\n                        if self.continuous_features_to_report.contains(feature.0) {\n                            self.continuous_feature_metrics\n                                .with_label_values(&[feature.0.to_string().as_str()])\n                                .observe(feature.1.into_inner() as f64)\n                        } else if self.discrete_features_to_report.contains(feature.0) {\n                            self.discrete_feature_metrics\n                                .with_label_values(&[feature.0.to_string().as_str()])\n                                .observe(feature.1.into_inner() as f64)\n                        }\n                    }\n                }\n            }\n            bpr_start = bpr_end;\n        }\n\n        InputTensor::FloatTensor(\n            Array2::<f32>::from_shape_vec([rows, cols], tensor)\n                .unwrap()\n                .into_dyn(),\n        )\n    }\n\n    fn get_binary(&self, bprs: &[BatchPredictionRequest], batch_ends: &[usize]) -> InputTensor {\n        // These need to be part of model schema\n        let rows: usize = batch_ends[batch_ends.len() - 1];\n        let cols: usize = 149;\n        let full_size: usize = rows * cols;\n        let default_val: i64 = 0;\n\n        let mut v = vec![default_val; full_size];\n\n        let mut bpr_start = 0;\n        for (bpr, &bpr_end) in bprs.iter().zip(batch_ends) {\n            // Common features\n            if bpr.common_features.is_some()\n                && bpr\n                    .common_features\n                    .as_ref()\n                    .unwrap()\n                    .binary_features\n                    .is_some()\n            {\n                let common_features = bpr\n                    .common_features\n                    .as_ref()\n                    .unwrap()\n                    .binary_features\n                    .as_ref()\n                    .unwrap();\n\n                for feature in common_features {\n                    match self.feature_mapper.get(feature) {\n                        Some(f_info) => {\n                            let idx = f_info.index_within_tensor as usize;\n                            if idx < cols {\n                                // Set value in each row\n                                for r in bpr_start..bpr_end {\n                                    let flat_index: usize = r * cols + idx;\n                                    v[flat_index] = 1;\n                                }\n                            }\n                        }\n                        None => (),\n                    }\n                }\n            }\n\n            // Process the batch of datarecords\n            for r in bpr_start..bpr_end {\n                let dr: &DataRecord = &bpr.individual_features_list[r - bpr_start];\n                if dr.binary_features.is_some() {\n                    for feature in dr.binary_features.as_ref().unwrap() {\n                        match self.feature_mapper.get(&feature) {\n                            Some(f_info) => {\n                                let idx = f_info.index_within_tensor as usize;\n                                let flat_index: usize = r * cols + idx;\n                                v[flat_index] = 1;\n                            }\n                            None => (),\n                        }\n                    }\n                }\n            }\n            bpr_start = bpr_end;\n        }\n        InputTensor::Int64Tensor(\n            Array2::<i64>::from_shape_vec([rows, cols], v)\n                .unwrap()\n                .into_dyn(),\n        )\n    }\n\n    #[allow(dead_code)]\n    fn get_discrete(&self, bprs: &[BatchPredictionRequest], batch_ends: &[usize]) -> InputTensor {\n        // These need to be part of model schema\n        let rows: usize = batch_ends[batch_ends.len() - 1];\n        let cols: usize = 320;\n        let full_size: usize = rows * cols;\n        let default_val: i64 = 0;\n\n        let mut v = vec![default_val; full_size];\n\n        let mut bpr_start = 0;\n        for (bpr, &bpr_end) in bprs.iter().zip(batch_ends) {\n            // Common features\n            if bpr.common_features.is_some()\n                && bpr\n                    .common_features\n                    .as_ref()\n                    .unwrap()\n                    .discrete_features\n                    .is_some()\n            {\n                let common_features = bpr\n                    .common_features\n                    .as_ref()\n                    .unwrap()\n                    .discrete_features\n                    .as_ref()\n                    .unwrap();\n\n                for feature in common_features {\n                    match self.feature_mapper.get(feature.0) {\n                        Some(f_info) => {\n                            let idx = f_info.index_within_tensor as usize;\n                            if idx < cols {\n                                // Set value in each row\n                                for r in bpr_start..bpr_end {\n                                    let flat_index: usize = r * cols + idx;\n                                    v[flat_index] = *feature.1;\n                                }\n                            }\n                        }\n                        None => (),\n                    }\n                    if self.discrete_features_to_report.contains(feature.0) {\n                        self.discrete_feature_metrics\n                            .with_label_values(&[feature.0.to_string().as_str()])\n                            .observe(*feature.1 as f64)\n                    }\n                }\n            }\n\n            // Process the batch of datarecords\n            for r in bpr_start..bpr_end {\n                let dr: &DataRecord = &bpr.individual_features_list[usize::try_from(r).unwrap()];\n                if dr.discrete_features.is_some() {\n                    for feature in dr.discrete_features.as_ref().unwrap() {\n                        match self.feature_mapper.get(&feature.0) {\n                            Some(f_info) => {\n                                let idx = f_info.index_within_tensor as usize;\n                                let flat_index: usize = r * cols + idx;\n                                if flat_index < v.len() && idx < cols {\n                                    v[flat_index] = *feature.1;\n                                }\n                            }\n                            None => (),\n                        }\n                        if self.discrete_features_to_report.contains(feature.0) {\n                            self.discrete_feature_metrics\n                                .with_label_values(&[feature.0.to_string().as_str()])\n                                .observe(*feature.1 as f64)\n                        }\n                    }\n                }\n            }\n            bpr_start = bpr_end;\n        }\n        InputTensor::Int64Tensor(\n            Array2::<i64>::from_shape_vec([rows, cols], v)\n                .unwrap()\n                .into_dyn(),\n        )\n    }\n\n    fn get_user_embedding(\n        &self,\n        bprs: &[BatchPredictionRequest],\n        batch_ends: &[usize],\n    ) -> InputTensor {\n        InputTensor::FloatTensor(\n            self.get_embedding_tensors(bprs, self.user_embedding_feature_id, batch_ends)\n                .into_dyn(),\n        )\n    }\n\n    fn get_eng_embedding(\n        &self,\n        bpr: &[BatchPredictionRequest],\n        batch_ends: &[usize],\n    ) -> InputTensor {\n        InputTensor::FloatTensor(\n            self.get_embedding_tensors(bpr, self.user_eng_embedding_feature_id, batch_ends)\n                .into_dyn(),\n        )\n    }\n\n    fn get_author_embedding(\n        &self,\n        bpr: &[BatchPredictionRequest],\n        batch_ends: &[usize],\n    ) -> InputTensor {\n        InputTensor::FloatTensor(\n            self.get_embedding_tensors(bpr, self.author_embedding_feature_id, batch_ends)\n                .into_dyn(),\n        )\n    }\n}\n\nimpl Converter for BatchPredictionRequestToTorchTensorConverter {\n    fn convert(&self, batched_bytes: Vec<Vec<u8>>) -> (Vec<InputTensor>, Vec<usize>) {\n        let bprs = batched_bytes\n            .into_iter()\n            .map(|bytes| {\n                BatchPredictionRequestToTorchTensorConverter::parse_batch_prediction_request(bytes)\n            })\n            .collect::<Vec<_>>();\n        let batch_ends = bprs\n            .iter()\n            .map(|bpr| bpr.individual_features_list.len())\n            .scan(0usize, |acc, e| {\n                //running total\n                *acc = *acc + e;\n                Some(*acc)\n            })\n            .collect::<Vec<_>>();\n\n        let t1 = self.get_continuous(&bprs, &batch_ends);\n        let t2 = self.get_binary(&bprs, &batch_ends);\n        //let _t3 = self.get_discrete(&bprs, &batch_ends);\n        let t4 = self.get_user_embedding(&bprs, &batch_ends);\n        let t5 = self.get_eng_embedding(&bprs, &batch_ends);\n        let t6 = self.get_author_embedding(&bprs, &batch_ends);\n\n        (vec![t1, t2, t4, t5, t6], batch_ends)\n    }\n}\n"
  },
  {
    "path": "navi/dr_transform/src/lib.rs",
    "content": "pub mod all_config;\npub mod converter;\n#[cfg(test)]\nmod test;\npub mod util;\npub extern crate ort;\n"
  },
  {
    "path": "navi/dr_transform/src/util.rs",
    "content": "use npyz::WriterBuilder;\nuse npyz::{AutoSerialize, WriteOptions};\nuse std::io::BufWriter;\nuse std::{\n    fs::File,\n    io::{self, BufRead},\n};\n\npub fn load_batch_prediction_request_base64(file_name: &str) -> Vec<Vec<u8>> {\n    let file = File::open(file_name).expect(\"could not read file\");\n    let mut result = vec![];\n    for (mut line_count, line) in io::BufReader::new(file).lines().enumerate() {\n        line_count += 1;\n        match base64::decode(line.unwrap().trim()) {\n            Ok(payload) => result.push(payload),\n            Err(err) => println!(\"error decoding line {file_name}:{line_count} - {err}\"),\n        }\n    }\n    println!(\"result len: {}\", result.len());\n    result\n}\n\npub fn save_to_npy<T: npyz::Serialize + AutoSerialize>(data: &[T], save_to: String) {\n    let mut writer = WriteOptions::new()\n        .default_dtype()\n        .shape(&[data.len() as u64, 1])\n        .writer(BufWriter::new(File::create(save_to).unwrap()))\n        .begin_nd()\n        .unwrap();\n    writer.extend(data.to_owned()).unwrap();\n    writer.finish().unwrap();\n}\n"
  },
  {
    "path": "navi/navi/Cargo.toml",
    "content": "[package]\nname = \"navi\"\nversion = \"2.0.45\"\nedition = \"2021\"\n\n[[bin]]\nname = \"navi\"\npath = \"src/bin/navi.rs\"\nrequired-features=[\"tf\"]\n[[bin]]\nname = \"navi_torch\"\npath = \"src/bin/navi_torch.rs\"\nrequired-features=[\"torch\"]\n[[bin]]\nname = \"navi_onnx\"\npath = \"src/bin/navi_onnx.rs\"\nrequired-features=[\"onnx\"]\n[[bin]]\nname = \"navi_onnx_test\"\npath = \"src/bin/bin_tests/navi_onnx_test.rs\"\n[[bin]]\nname = \"navi_torch_test\"\npath = \"src/bin/bin_tests/navi_torch_test.rs\"\nrequired-features=[\"torch\"]\n\n[features]\ndefault=[]\nnavi_console=[]\ntorch=[\"tch\"]\nonnx=[]\ntf=[\"tensorflow\"]\n[dependencies]\nitertools = \"0.10.5\"\nanyhow = \"1.0.57\"\narrayvec = \"0.7.2\"\nclap = { version = \"4.0.32\", features = [\"derive\"] }\nconsole-subscriber = \"0.1.6\"\ntime = { version = \"0.3.20\", features = [\"parsing\"] }\nenv_logger = \"0.10.0\"\nflamegraph = \"0.6.1\"\nfnv = \"1.0.7\"\nfutures = { version = \"0.3\", default-features = false }\nimage = \"0.24.5\"\nindexmap = \"1.8.1\"\nlazy_static = \"1.4\"\nlibloading = \"0.7\"\nlog = \"0.4.17\"\nndarray-rand = \"0.14.0\"\nprometheus = \"0.13.1\"\nprost = \"0.9\"\nprost-types = \"0.9\"\nparking_lot = \"0.12.1\"\nrand = \"0.8.5\"\nrand_pcg = \"0.3.1\"\nrandom = \"0.12.2\"\nx509-parser = \"0.15.0\"\nsha256 = \"1.0.3\"\ntonic = { version = \"0.6.2\", features=['compression', 'tls'] }\ntokio = { version = \"1.17.0\", features = [\"macros\", \"rt-multi-thread\", \"fs\", \"process\"] }\nwarp = \"0.3\"\nnpyz = \"0.7.3\"\nbase64 = \"0.21.0\"\nhistogram = \"0.6.9\"\ntch = {version = \"0.10.3\", optional = true}\ntensorflow = { version = \"0.18.0\", optional = true }\nonce_cell = {version = \"1.17.1\"}\nndarray = \"0.15\"\nserde = \"1.0.154\"\nserde_json = \"1.0.94\"\ndr_transform = { path = \"../dr_transform\"}\n[build-dependencies]\ntonic-build = {version = \"0.6.2\", features=['prost', \"compression\"] }\n[profile.release]\ndebug = true\n[dev-dependencies]\nndarray-rand = \"0.14.0\"\ntokio-test = \"*\"\nassert_cmd = \"2.0\"\ncriterion = \"0.4.0\"\n\n\n"
  },
  {
    "path": "navi/navi/build.rs",
    "content": "fn main() -> Result<(), Box<dyn std::error::Error>> {\n    //::compile_protos(\"proto/tensorflow_serving/apis/prediction_service.proto\")?;\n    tonic_build::configure().compile(\n        &[\n            \"proto/tensorflow_serving/apis/prediction_service.proto\",\n            \"proto/tensorflow/core/protobuf/config.proto\",\n            \"proto/tensorflow_serving/apis/prediction_log.proto\",\n            \"proto/kfserving/grpc_predict_v2.proto\",\n        ],\n        &[\"proto\"],\n    )?;\n    Ok(())\n}\n"
  },
  {
    "path": "navi/navi/proto/kfserving/grpc_predict_v2.proto",
    "content": "// Copyright 2020 kubeflow.org.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n//    http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.\n\nsyntax = \"proto3\";\npackage inference;\n\n// Inference Server GRPC endpoints.\nservice GRPCInferenceService\n{\n  // The ServerLive API indicates if the inference server is able to receive \n  // and respond to metadata and inference requests.\n  rpc ServerLive(ServerLiveRequest) returns (ServerLiveResponse) {}\n\n  // The ServerReady API indicates if the server is ready for inferencing.\n  rpc ServerReady(ServerReadyRequest) returns (ServerReadyResponse) {}\n\n  // The ModelReady API indicates if a specific model is ready for inferencing.\n  rpc ModelReady(ModelReadyRequest) returns (ModelReadyResponse) {}\n\n  // The ServerMetadata API provides information about the server. Errors are \n  // indicated by the google.rpc.Status returned for the request. The OK code \n  // indicates success and other codes indicate failure.\n  rpc ServerMetadata(ServerMetadataRequest) returns (ServerMetadataResponse) {}\n\n  // The per-model metadata API provides information about a model. Errors are \n  // indicated by the google.rpc.Status returned for the request. The OK code \n  // indicates success and other codes indicate failure.\n  rpc ModelMetadata(ModelMetadataRequest) returns (ModelMetadataResponse) {}\n\n  // The ModelInfer API performs inference using the specified model. Errors are\n  // indicated by the google.rpc.Status returned for the request. The OK code \n  // indicates success and other codes indicate failure.\n  rpc ModelInfer(ModelInferRequest) returns (ModelInferResponse) {}\n}\n\nmessage ServerLiveRequest {}\n\nmessage ServerLiveResponse\n{\n  // True if the inference server is live, false if not live.\n  bool live = 1;\n}\n\nmessage ServerReadyRequest {}\n\nmessage ServerReadyResponse\n{\n  // True if the inference server is ready, false if not ready.\n  bool ready = 1;\n}\n\nmessage ModelReadyRequest\n{\n  // The name of the model to check for readiness.\n  string name = 1;\n\n  // The version of the model to check for readiness. If not given the\n  // server will choose a version based on the model and internal policy.\n  string version = 2;\n}\n\nmessage ModelReadyResponse\n{\n  // True if the model is ready, false if not ready.\n  bool ready = 1;\n}\n\nmessage ServerMetadataRequest {}\n\nmessage ServerMetadataResponse\n{\n  // The server name.\n  string name = 1;\n\n  // The server version.\n  string version = 2;\n\n  // The extensions supported by the server.\n  repeated string extensions = 3;\n}\n\nmessage ModelMetadataRequest\n{\n  // The name of the model.\n  string name = 1;\n\n  // The version of the model to check for readiness. If not given the\n  // server will choose a version based on the model and internal policy.\n  string version = 2;\n}\n\nmessage ModelMetadataResponse\n{\n  // Metadata for a tensor.\n  message TensorMetadata\n  {\n    // The tensor name.\n    string name = 1;\n\n    // The tensor data type.\n    string datatype = 2;\n\n    // The tensor shape. A variable-size dimension is represented\n    // by a -1 value.\n    repeated int64 shape = 3;\n  }\n\n  // The model name.\n  string name = 1;\n\n  // The versions of the model available on the server.\n  repeated string versions = 2;\n\n  // The model's platform. See Platforms.\n  string platform = 3;\n\n  // The model's inputs.\n  repeated TensorMetadata inputs = 4;\n\n  // The model's outputs.\n  repeated TensorMetadata outputs = 5;\n}\n\nmessage ModelInferRequest\n{\n  // An input tensor for an inference request.\n  message InferInputTensor\n  {\n    // The tensor name.\n    string name = 1;\n\n    // The tensor data type.\n    string datatype = 2;\n\n    // The tensor shape.\n    repeated int64 shape = 3;\n\n    // Optional inference input tensor parameters.\n    map<string, InferParameter> parameters = 4;\n\n    // The tensor contents using a data-type format. This field must\n    // not be specified if \"raw\" tensor contents are being used for\n    // the inference request.\n    InferTensorContents contents = 5;\n  }\n\n  // An output tensor requested for an inference request.\n  message InferRequestedOutputTensor\n  {\n    // The tensor name.\n    string name = 1;\n\n    // Optional requested output tensor parameters.\n    map<string, InferParameter> parameters = 2;\n  }\n\n  // The name of the model to use for inferencing.\n  string model_name = 1;\n\n  // The version of the model to use for inference. If not given the\n  // server will choose a version based on the model and internal policy.\n  string model_version = 2;\n\n  // Optional identifier for the request. If specified will be\n  // returned in the response.\n  string id = 3;\n\n  // Optional inference parameters.\n  map<string, InferParameter> parameters = 4;\n\n  // The input tensors for the inference.\n  repeated InferInputTensor inputs = 5;\n\n  // The requested output tensors for the inference. Optional, if not\n  // specified all outputs produced by the model will be returned.\n  repeated InferRequestedOutputTensor outputs = 6;\n\n  // The data contained in an input tensor can be represented in \"raw\"\n  // bytes form or in the repeated type that matches the tensor's data\n  // type. To use the raw representation 'raw_input_contents' must be\n  // initialized with data for each tensor in the same order as\n  // 'inputs'. For each tensor, the size of this content must match\n  // what is expected by the tensor's shape and data type. The raw\n  // data must be the flattened, one-dimensional, row-major order of\n  // the tensor elements without any stride or padding between the\n  // elements. Note that the FP16 and BF16 data types must be represented as\n  // raw content as there is no specific data type for a 16-bit float type.\n  //\n  // If this field is specified then InferInputTensor::contents must\n  // not be specified for any input tensor.\n  repeated bytes raw_input_contents = 7;\n}\n\nmessage ModelInferResponse\n{\n  // An output tensor returned for an inference request.\n  message InferOutputTensor\n  {\n    // The tensor name.\n    string name = 1;\n\n    // The tensor data type.\n    string datatype = 2;\n\n    // The tensor shape.\n    repeated int64 shape = 3;\n\n    // Optional output tensor parameters.\n    map<string, InferParameter> parameters = 4;\n\n    // The tensor contents using a data-type format. This field must\n    // not be specified if \"raw\" tensor contents are being used for\n    // the inference response.\n    InferTensorContents contents = 5;\n  }\n\n  // The name of the model used for inference.\n  string model_name = 1;\n\n  // The version of the model used for inference.\n  string model_version = 2;\n\n  // The id of the inference request if one was specified.\n  string id = 3;\n\n  // Optional inference response parameters.\n  map<string, InferParameter> parameters = 4;\n\n  // The output tensors holding inference results.\n  repeated InferOutputTensor outputs = 5;\n\n  // The data contained in an output tensor can be represented in\n  // \"raw\" bytes form or in the repeated type that matches the\n  // tensor's data type. To use the raw representation 'raw_output_contents'\n  // must be initialized with data for each tensor in the same order as\n  // 'outputs'. For each tensor, the size of this content must match\n  // what is expected by the tensor's shape and data type. The raw\n  // data must be the flattened, one-dimensional, row-major order of\n  // the tensor elements without any stride or padding between the\n  // elements. Note that the FP16 and BF16 data types must be represented as\n  // raw content as there is no specific data type for a 16-bit float type.\n  //\n  // If this field is specified then InferOutputTensor::contents must\n  // not be specified for any output tensor.\n  repeated bytes raw_output_contents = 6;\n}\n\n// An inference parameter value. The Parameters message describes a \n// “name”/”value” pair, where the “name” is the name of the parameter\n// and the “value” is a boolean, integer, or string corresponding to \n// the parameter.\nmessage InferParameter\n{\n  // The parameter value can be a string, an int64, a boolean\n  // or a message specific to a predefined parameter.\n  oneof parameter_choice\n  {\n    // A boolean parameter value.\n    bool bool_param = 1;\n\n    // An int64 parameter value.\n    int64 int64_param = 2;\n\n    // A string parameter value.\n    string string_param = 3;\n  }\n}\n\n// The data contained in a tensor represented by the repeated type\n// that matches the tensor's data type. Protobuf oneof is not used\n// because oneofs cannot contain repeated fields.\nmessage InferTensorContents\n{\n  // Representation for BOOL data type. The size must match what is\n  // expected by the tensor's shape. The contents must be the flattened,\n  // one-dimensional, row-major order of the tensor elements.\n  repeated bool bool_contents = 1;\n\n  // Representation for INT8, INT16, and INT32 data types. The size\n  // must match what is expected by the tensor's shape. The contents\n  // must be the flattened, one-dimensional, row-major order of the\n  // tensor elements.\n  repeated int32 int_contents = 2;\n\n  // Representation for INT64 data types. The size must match what\n  // is expected by the tensor's shape. The contents must be the\n  // flattened, one-dimensional, row-major order of the tensor elements.\n  repeated int64 int64_contents = 3;\n\n  // Representation for UINT8, UINT16, and UINT32 data types. The size\n  // must match what is expected by the tensor's shape. The contents\n  // must be the flattened, one-dimensional, row-major order of the\n  // tensor elements.\n  repeated uint32 uint_contents = 4;\n\n  // Representation for UINT64 data types. The size must match what\n  // is expected by the tensor's shape. The contents must be the\n  // flattened, one-dimensional, row-major order of the tensor elements.\n  repeated uint64 uint64_contents = 5;\n\n  // Representation for FP32 data type. The size must match what is\n  // expected by the tensor's shape. The contents must be the flattened,\n  // one-dimensional, row-major order of the tensor elements.\n  repeated float fp32_contents = 6;\n\n  // Representation for FP64 data type. The size must match what is\n  // expected by the tensor's shape. The contents must be the flattened,\n  // one-dimensional, row-major order of the tensor elements.\n  repeated double fp64_contents = 7;\n\n  // Representation for BYTES data type. The size must match what is\n  // expected by the tensor's shape. The contents must be the flattened,\n  // one-dimensional, row-major order of the tensor elements.\n  repeated bytes bytes_contents = 8;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/example/example.proto",
    "content": "// Protocol messages for describing input data Examples for machine learning\n// model training or inference.\nsyntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/example/feature.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"ExampleProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.example\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/example\";\n\n// LINT.IfChange\n// An Example is a mostly-normalized data format for storing data for\n// training and inference.  It contains a key-value store (features); where\n// each key (string) maps to a Feature message (which is oneof packed BytesList,\n// FloatList, or Int64List).  This flexible and compact format allows the\n// storage of large amounts of typed data, but requires that the data shape\n// and use be determined by the configuration files and parsers that are used to\n// read and write this format.  That is, the Example is mostly *not* a\n// self-describing format.  In TensorFlow, Examples are read in row-major\n// format, so any configuration that describes data with rank-2 or above\n// should keep this in mind.  For example, to store an M x N matrix of Bytes,\n// the BytesList must contain M*N bytes, with M rows of N contiguous values\n// each.  That is, the BytesList value must store the matrix as:\n//     .... row 0 .... .... row 1 .... // ...........  // ... row M-1 ....\n//\n// An Example for a movie recommendation application:\n//   features {\n//     feature {\n//       key: \"age\"\n//       value { float_list {\n//         value: 29.0\n//       }}\n//     }\n//     feature {\n//       key: \"movie\"\n//       value { bytes_list {\n//         value: \"The Shawshank Redemption\"\n//         value: \"Fight Club\"\n//       }}\n//     }\n//     feature {\n//       key: \"movie_ratings\"\n//       value { float_list {\n//         value: 9.0\n//         value: 9.7\n//       }}\n//     }\n//     feature {\n//       key: \"suggestion\"\n//       value { bytes_list {\n//         value: \"Inception\"\n//       }}\n//     }\n//     # Note that this feature exists to be used as a label in training.\n//     # E.g., if training a logistic regression model to predict purchase\n//     # probability in our learning tool we would set the label feature to\n//     # \"suggestion_purchased\".\n//     feature {\n//       key: \"suggestion_purchased\"\n//       value { float_list {\n//         value: 1.0\n//       }}\n//     }\n//     # Similar to \"suggestion_purchased\" above this feature exists to be used\n//     # as a label in training.\n//     # E.g., if training a linear regression model to predict purchase\n//     # price in our learning tool we would set the label feature to\n//     # \"purchase_price\".\n//     feature {\n//       key: \"purchase_price\"\n//       value { float_list {\n//         value: 9.99\n//       }}\n//     }\n//  }\n//\n// A conformant Example data set obeys the following conventions:\n//   - If a Feature K exists in one example with data type T, it must be of\n//       type T in all other examples when present. It may be omitted.\n//   - The number of instances of Feature K list data may vary across examples,\n//       depending on the requirements of the model.\n//   - If a Feature K doesn't exist in an example, a K-specific default will be\n//       used, if configured.\n//   - If a Feature K exists in an example but contains no items, the intent\n//       is considered to be an empty tensor and no default will be used.\n\nmessage Example {\n  Features features = 1;\n}\n\n// A SequenceExample is an Example representing one or more sequences, and\n// some context.  The context contains features which apply to the entire\n// example. The feature_lists contain a key, value map where each key is\n// associated with a repeated set of Features (a FeatureList).\n// A FeatureList thus represents the values of a feature identified by its key\n// over time / frames.\n//\n// Below is a SequenceExample for a movie recommendation application recording a\n// sequence of ratings by a user. The time-independent features (\"locale\",\n// \"age\", \"favorites\") describing the user are part of the context. The sequence\n// of movies the user rated are part of the feature_lists. For each movie in the\n// sequence we have information on its name and actors and the user's rating.\n// This information is recorded in three separate feature_list(s).\n// In the example below there are only two movies. All three feature_list(s),\n// namely \"movie_ratings\", \"movie_names\", and \"actors\" have a feature value for\n// both movies. Note, that \"actors\" is itself a bytes_list with multiple\n// strings per movie.\n//\n// context: {\n//   feature: {\n//     key  : \"locale\"\n//     value: {\n//       bytes_list: {\n//         value: [ \"pt_BR\" ]\n//       }\n//     }\n//   }\n//   feature: {\n//     key  : \"age\"\n//     value: {\n//       float_list: {\n//         value: [ 19.0 ]\n//       }\n//     }\n//   }\n//   feature: {\n//     key  : \"favorites\"\n//     value: {\n//       bytes_list: {\n//         value: [ \"Majesty Rose\", \"Savannah Outen\", \"One Direction\" ]\n//       }\n//     }\n//   }\n// }\n// feature_lists: {\n//   feature_list: {\n//     key  : \"movie_ratings\"\n//     value: {\n//       feature: {\n//         float_list: {\n//           value: [ 4.5 ]\n//         }\n//       }\n//       feature: {\n//         float_list: {\n//           value: [ 5.0 ]\n//         }\n//       }\n//     }\n//   }\n//   feature_list: {\n//     key  : \"movie_names\"\n//     value: {\n//       feature: {\n//         bytes_list: {\n//           value: [ \"The Shawshank Redemption\" ]\n//         }\n//       }\n//       feature: {\n//         bytes_list: {\n//           value: [ \"Fight Club\" ]\n//         }\n//       }\n//     }\n//   }\n//   feature_list: {\n//     key  : \"actors\"\n//     value: {\n//       feature: {\n//         bytes_list: {\n//           value: [ \"Tim Robbins\", \"Morgan Freeman\" ]\n//         }\n//       }\n//       feature: {\n//         bytes_list: {\n//           value: [ \"Brad Pitt\", \"Edward Norton\", \"Helena Bonham Carter\" ]\n//         }\n//       }\n//     }\n//   }\n// }\n//\n// A conformant SequenceExample data set obeys the following conventions:\n//\n// Context:\n//   - All conformant context features K must obey the same conventions as\n//     a conformant Example's features (see above).\n// Feature lists:\n//   - A FeatureList L may be missing in an example; it is up to the\n//     parser configuration to determine if this is allowed or considered\n//     an empty list (zero length).\n//   - If a FeatureList L exists, it may be empty (zero length).\n//   - If a FeatureList L is non-empty, all features within the FeatureList\n//     must have the same data type T. Even across SequenceExamples, the type T\n//     of the FeatureList identified by the same key must be the same. An entry\n//     without any values may serve as an empty feature.\n//   - If a FeatureList L is non-empty, it is up to the parser configuration\n//     to determine if all features within the FeatureList must\n//     have the same size.  The same holds for this FeatureList across multiple\n//     examples.\n//   - For sequence modeling, e.g.:\n//        http://colah.github.io/posts/2015-08-Understanding-LSTMs/\n//        https://github.com/tensorflow/nmt\n//     the feature lists represent a sequence of frames.\n//     In this scenario, all FeatureLists in a SequenceExample have the same\n//     number of Feature messages, so that the ith element in each FeatureList\n//     is part of the ith frame (or time step).\n// Examples of conformant and non-conformant examples' FeatureLists:\n//\n// Conformant FeatureLists:\n//    feature_lists: { feature_list: {\n//      key: \"movie_ratings\"\n//      value: { feature: { float_list: { value: [ 4.5 ] } }\n//               feature: { float_list: { value: [ 5.0 ] } } }\n//    } }\n//\n// Non-conformant FeatureLists (mismatched types):\n//    feature_lists: { feature_list: {\n//      key: \"movie_ratings\"\n//      value: { feature: { float_list: { value: [ 4.5 ] } }\n//               feature: { int64_list: { value: [ 5 ] } } }\n//    } }\n//\n// Conditionally conformant FeatureLists, the parser configuration determines\n// if the feature sizes must match:\n//    feature_lists: { feature_list: {\n//      key: \"movie_ratings\"\n//      value: { feature: { float_list: { value: [ 4.5 ] } }\n//               feature: { float_list: { value: [ 5.0, 6.0 ] } } }\n//    } }\n//\n// Conformant pair of SequenceExample\n//    feature_lists: { feature_list: {\n//      key: \"movie_ratings\"\n//      value: { feature: { float_list: { value: [ 4.5 ] } }\n//               feature: { float_list: { value: [ 5.0 ] } } }\n//    } }\n// and:\n//    feature_lists: { feature_list: {\n//      key: \"movie_ratings\"\n//      value: { feature: { float_list: { value: [ 4.5 ] } }\n//               feature: { float_list: { value: [ 5.0 ] } }\n//               feature: { float_list: { value: [ 2.0 ] } } }\n//    } }\n//\n// Conformant pair of SequenceExample\n//    feature_lists: { feature_list: {\n//      key: \"movie_ratings\"\n//      value: { feature: { float_list: { value: [ 4.5 ] } }\n//               feature: { float_list: { value: [ 5.0 ] } } }\n//    } }\n// and:\n//    feature_lists: { feature_list: {\n//      key: \"movie_ratings\"\n//      value: { }\n//    } }\n//\n// Conditionally conformant pair of SequenceExample, the parser configuration\n// determines if the second feature_lists is consistent (zero-length) or\n// invalid (missing \"movie_ratings\"):\n//    feature_lists: { feature_list: {\n//      key: \"movie_ratings\"\n//      value: { feature: { float_list: { value: [ 4.5 ] } }\n//               feature: { float_list: { value: [ 5.0 ] } } }\n//    } }\n// and:\n//    feature_lists: { }\n//\n// Non-conformant pair of SequenceExample (mismatched types)\n//    feature_lists: { feature_list: {\n//      key: \"movie_ratings\"\n//      value: { feature: { float_list: { value: [ 4.5 ] } }\n//               feature: { float_list: { value: [ 5.0 ] } } }\n//    } }\n// and:\n//    feature_lists: { feature_list: {\n//      key: \"movie_ratings\"\n//      value: { feature: { int64_list: { value: [ 4 ] } }\n//               feature: { int64_list: { value: [ 5 ] } }\n//               feature: { int64_list: { value: [ 2 ] } } }\n//    } }\n//\n// Conditionally conformant pair of SequenceExample; the parser configuration\n// determines if the feature sizes must match:\n//    feature_lists: { feature_list: {\n//      key: \"movie_ratings\"\n//      value: { feature: { float_list: { value: [ 4.5 ] } }\n//               feature: { float_list: { value: [ 5.0 ] } } }\n//    } }\n// and:\n//    feature_lists: { feature_list: {\n//      key: \"movie_ratings\"\n//      value: { feature: { float_list: { value: [ 4.0 ] } }\n//               feature: { float_list: { value: [ 5.0, 3.0 ] } }\n//    } }\n\nmessage SequenceExample {\n  Features context = 1;\n  FeatureLists feature_lists = 2;\n}\n// LINT.ThenChange(\n//     https://www.tensorflow.org/code/tensorflow/python/training/training.py)\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/example/feature.proto",
    "content": "// Protocol messages for describing features for machine learning model\n// training or inference.\n//\n// There are three base Feature types:\n//   - bytes\n//   - float\n//   - int64\n//\n// A Feature contains Lists which may hold zero or more values.  These\n// lists are the base values BytesList, FloatList, Int64List.\n//\n// Features are organized into categories by name.  The Features message\n// contains the mapping from name to Feature.\n//\n// Example Features for a movie recommendation application:\n//   feature {\n//     key: \"age\"\n//     value { float_list {\n//       value: 29.0\n//     }}\n//   }\n//   feature {\n//     key: \"movie\"\n//     value { bytes_list {\n//       value: \"The Shawshank Redemption\"\n//       value: \"Fight Club\"\n//     }}\n//   }\n//   feature {\n//     key: \"movie_ratings\"\n//     value { float_list {\n//       value: 9.0\n//       value: 9.7\n//     }}\n//   }\n//   feature {\n//     key: \"suggestion\"\n//     value { bytes_list {\n//       value: \"Inception\"\n//     }}\n//   }\n//   feature {\n//     key: \"suggestion_purchased\"\n//     value { int64_list {\n//       value: 1\n//     }}\n//   }\n//   feature {\n//     key: \"purchase_price\"\n//     value { float_list {\n//       value: 9.99\n//     }}\n//   }\n//\n\nsyntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"FeatureProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.example\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/example\";\n\n// LINT.IfChange\n// Containers to hold repeated fundamental values.\nmessage BytesList {\n  repeated bytes value = 1;\n}\nmessage FloatList {\n  repeated float value = 1 [packed = true];\n}\nmessage Int64List {\n  repeated int64 value = 1 [packed = true];\n}\n\n// Containers for non-sequential data.\nmessage Feature {\n  // Each feature can be exactly one kind.\n  oneof kind {\n    BytesList bytes_list = 1;\n    FloatList float_list = 2;\n    Int64List int64_list = 3;\n  }\n}\n\nmessage Features {\n  // Map from feature name to feature.\n  map<string, Feature> feature = 1;\n}\n\n// Containers for sequential data.\n//\n// A FeatureList contains lists of Features.  These may hold zero or more\n// Feature values.\n//\n// FeatureLists are organized into categories by name.  The FeatureLists message\n// contains the mapping from name to FeatureList.\n//\nmessage FeatureList {\n  repeated Feature feature = 1;\n}\n\nmessage FeatureLists {\n  // Map from feature name to feature list.\n  map<string, FeatureList> feature_list = 1;\n}\n// LINT.ThenChange(\n//     https://www.tensorflow.org/code/tensorflow/python/training/training.py)\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/allocation_description.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"AllocationDescriptionProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/allocation_description_go_proto\";\n\nmessage AllocationDescription {\n  // Total number of bytes requested\n  int64 requested_bytes = 1;\n\n  // Total number of bytes allocated if known\n  int64 allocated_bytes = 2;\n\n  // Name of the allocator used\n  string allocator_name = 3;\n\n  // Identifier of the allocated buffer if known\n  int64 allocation_id = 4;\n\n  // Set if this tensor only has one remaining reference\n  bool has_single_reference = 5;\n\n  // Address of the allocation.\n  uint64 ptr = 6;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/api_def.proto",
    "content": "// Defines the text format for including per-op API definition and\n// overrides for client language op code generators.\n\nsyntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/attr_value.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"ApiDefProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/api_def_go_proto\";\n\n// Used to specify and override the default API & behavior in the\n// generated code for client languages, from what you would get from\n// the OpDef alone. There will be a set of ApiDefs that are common\n// to all client languages, and another set per client language.\n// The per-client-language ApiDefs will inherit values from the\n// common ApiDefs which it can either replace or modify.\n//\n// We separate the API definition from the OpDef so we can evolve the\n// API while remaining backwards compatible when interpreting old\n// graphs.  Overrides go in an \"api_def.pbtxt\" file with a text-format\n// ApiDefs message.\n//\n// WARNING: Be *very* careful changing the API for any existing op --\n// you can change the semantics of existing code.  These changes may\n// need to wait until a major release of TensorFlow to avoid breaking\n// our compatibility promises.\nmessage ApiDef {\n  // Name of the op (in the OpDef) to specify the API for.\n  string graph_op_name = 1;\n  // If this op is deprecated, set deprecation message to the message\n  // that should be logged when this op is used.\n  // The message should indicate alternative op to use, if any.\n  string deprecation_message = 12;\n  // Major version when the op will be deleted. For e.g. set this\n  // value to 2 if op API should be removed in TensorFlow 2.0 and\n  // deprecated in versions before that.\n  int32 deprecation_version = 13;\n\n  enum Visibility {\n    // Normally this is \"VISIBLE\" unless you are inheriting a\n    // different value from another ApiDef.\n    DEFAULT_VISIBILITY = 0;\n    // Publicly visible in the API.\n    VISIBLE = 1;\n    // Do not include this op in the generated API. If visibility is\n    // set to 'SKIP', other fields are ignored for this op.\n    SKIP = 2;\n    // Hide this op by putting it into an internal namespace (or whatever\n    // is appropriate in the target language).\n    HIDDEN = 3;\n  }\n  Visibility visibility = 2;\n\n  // If you specify any endpoint, this will replace all of the\n  // inherited endpoints.  The first endpoint should be the\n  // \"canonical\" endpoint, and should not be deprecated (unless all\n  // endpoints are deprecated).\n  message Endpoint {\n    // Name should be either like \"CamelCaseName\" or\n    // \"Package.CamelCaseName\". Client-language-specific ApiDefs may\n    // use a snake_case convention instead of CamelCase.\n    string name = 1;\n\n    // Set if this endpoint is deprecated. If set to true, a message suggesting\n    // to use a non-deprecated endpoint instead will be printed. If all\n    // endpoints are deprecated, set deprecation_message in ApiDef instead.\n    bool deprecated = 3;\n\n    // Major version when an endpoint will be deleted. For e.g. set this\n    // value to 2 if endpoint should be removed in TensorFlow 2.0 and\n    // deprecated in versions before that.\n    int32 deprecation_version = 4;\n  }\n  repeated Endpoint endpoint = 3;\n\n  message Arg {\n    string name = 1;\n\n    // Change the name used to access this arg in the API from what\n    // is used in the GraphDef.  Note that these names in `backticks`\n    // will also be replaced in the summary & description fields.\n    string rename_to = 2;\n\n    // Note: this will replace any inherited arg doc. There is no\n    // current way of modifying arg descriptions (other than replacing\n    // them entirely) as can be done with op descriptions.\n    string description = 3;\n  }\n  repeated Arg in_arg = 4;\n  repeated Arg out_arg = 5;\n  // List of original in_arg names to specify new argument order.\n  // Length of arg_order should be either empty to keep current order\n  // or match size of in_arg.\n  repeated string arg_order = 11;\n\n  // Description of the graph-construction-time configuration of this\n  // Op.  That is to say, this describes the attr fields that will\n  // be specified in the NodeDef.\n  message Attr {\n    string name = 1;\n\n    // Change the name used to access this attr in the API from what\n    // is used in the GraphDef.  Note that these names in `backticks`\n    // will also be replaced in the summary & description fields.\n    string rename_to = 2;\n\n    // Specify a new default value to use for this attr.  This default\n    // will be used when creating new graphs, as opposed to the\n    // default in the OpDef, which will be used when interpreting old\n    // GraphDefs.\n    AttrValue default_value = 3;\n\n    // Note: this will replace any inherited attr doc, there is no current\n    // way of modifying attr descriptions as can be done with op descriptions.\n    string description = 4;\n  }\n  repeated Attr attr = 6;\n\n  // One-line human-readable description of what the Op does.\n  string summary = 7;\n\n  // Additional, longer human-readable description of what the Op does.\n  string description = 8;\n\n  // Modify an existing/inherited description by adding text to the beginning\n  // or end.\n  string description_prefix = 9;\n  string description_suffix = 10;\n}\n\nmessage ApiDefs {\n  repeated ApiDef op = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/attr_value.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/tensor.proto\";\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/types.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"AttrValueProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/attr_value_go_proto\";\n\n// Protocol buffer representing the value for an attr used to configure an Op.\n// Comment indicates the corresponding attr type.  Only the field matching the\n// attr type may be filled.\nmessage AttrValue {\n  // LINT.IfChange\n  message ListValue {\n    repeated bytes s = 2;                        // \"list(string)\"\n    repeated int64 i = 3 [packed = true];        // \"list(int)\"\n    repeated float f = 4 [packed = true];        // \"list(float)\"\n    repeated bool b = 5 [packed = true];         // \"list(bool)\"\n    repeated DataType type = 6 [packed = true];  // \"list(type)\"\n    repeated TensorShapeProto shape = 7;         // \"list(shape)\"\n    repeated TensorProto tensor = 8;             // \"list(tensor)\"\n    repeated NameAttrList func = 9;              // \"list(attr)\"\n  }\n  // LINT.ThenChange(https://www.tensorflow.org/code/tensorflow/c/c_api.cc)\n\n  oneof value {\n    bytes s = 2;                 // \"string\"\n    int64 i = 3;                 // \"int\"\n    float f = 4;                 // \"float\"\n    bool b = 5;                  // \"bool\"\n    DataType type = 6;           // \"type\"\n    TensorShapeProto shape = 7;  // \"shape\"\n    TensorProto tensor = 8;      // \"tensor\"\n    ListValue list = 1;          // any \"list(...)\"\n\n    // \"func\" represents a function. func.name is a function's name or\n    // a primitive op's name. func.attr.first is the name of an attr\n    // defined for that function. func.attr.second is the value for\n    // that attr in the instantiation.\n    NameAttrList func = 10;\n\n    // This is a placeholder only used in nodes defined inside a\n    // function.  It indicates the attr value will be supplied when\n    // the function is instantiated.  For example, let us suppose a\n    // node \"N\" in function \"FN\". \"N\" has an attr \"A\" with value\n    // placeholder = \"foo\". When FN is instantiated with attr \"foo\"\n    // set to \"bar\", the instantiated node N's attr A will have been\n    // given the value \"bar\".\n    string placeholder = 9;\n  }\n}\n\n// A list of attr names and their values. The whole list is attached\n// with a string name.  E.g., MatMul[T=float].\nmessage NameAttrList {\n  string name = 1;\n  map<string, AttrValue> attr = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/cost_graph.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/types.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"CostGraphProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/cost_graph_go_proto\";\n\nmessage CostGraphDef {\n  message Node {\n    // The name of the node. Names are globally unique.\n    string name = 1;\n\n    // The device of the node. Can be empty if the node is mapped to the\n    // default partition or partitioning hasn't been run yet.\n    string device = 2;\n\n    // The id of the node. Node ids are only unique inside a partition.\n    int32 id = 3;\n\n    // Inputs of this node. They must be executed before this node can be\n    // executed. An input is a particular output of another node, specified\n    // by the node id and the output index.\n    message InputInfo {\n      int32 preceding_node = 1;\n      int32 preceding_port = 2;\n    }\n    repeated InputInfo input_info = 4;\n\n    // Outputs of this node.\n    message OutputInfo {\n      int64 size = 1;\n      // If >= 0, the output is an alias of an input. Note that an alias input\n      // may itself be an alias. The algorithm will therefore need to follow\n      // those pointers.\n      int64 alias_input_port = 2;\n      TensorShapeProto shape = 3;\n      DataType dtype = 4;\n    }\n    repeated OutputInfo output_info = 5;\n\n    // Temporary memory used by this node.\n    int64 temporary_memory_size = 6;\n\n    // Persistent memory used by this node.\n    int64 persistent_memory_size = 12;\n\n    int64 host_temp_memory_size = 10 [deprecated = true];\n    int64 device_temp_memory_size = 11 [deprecated = true];\n    int64 device_persistent_memory_size = 16 [deprecated = true];\n\n    // Estimate of the computational cost of this node, in microseconds.\n    int64 compute_cost = 9;\n\n    // Analytical estimate of the computational cost of this node, in\n    // microseconds.\n    int64 compute_time = 14;\n\n    // Analytical estimate of the memory access cost of this node, in\n    // microseconds.\n    int64 memory_time = 15;\n\n    // If true, the output is permanent: it can't be discarded, because this\n    // node is part of the \"final output\". Nodes may depend on final nodes.\n    bool is_final = 7;\n\n    // Ids of the control inputs for this node.\n    repeated int32 control_input = 8;\n\n    // Are the costs inaccurate?\n    bool inaccurate = 17;\n  }\n  repeated Node node = 1;\n\n  // Total cost of this graph, typically used for balancing decisions.\n  message AggregatedCost {\n    // Aggregated cost value.\n    float cost = 1;\n\n    // Aggregated cost dimension (e.g. 'memory', 'compute', 'network').\n    string dimension = 2;\n  }\n  repeated AggregatedCost cost = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/dataset_metadata.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.data;\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/dataset_metadata_go_proto\";\n\n// next: 2\nmessage Metadata {\n  bytes name = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/dataset_options.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.data;\n\nimport \"tensorflow/core/framework/model.proto\";\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/dataset_options_go_proto\";\n\n// Represents the type of auto-sharding we enable.\nenum AutoShardPolicy {\n  // AUTO: Attempts FILE-based sharding, falling back to DATA-based sharding.\n  AUTO = 0;\n  // FILE: Shards by input files (i.e. each worker will get a set of files to\n  // process). When this option is selected, make sure that there is at least as\n  // many files as workers. If there are fewer input files than workers, a\n  // runtime error will be raised.\n  FILE = 1;\n  // DATA: Shards by elements produced by the dataset. Each worker will process\n  // the whole dataset and discard the portion that is not for itself. Note that\n  // for this mode to correctly partitions the dataset elements, the dataset\n  // needs to produce elements in a deterministic order.\n  DATA = 2;\n  // HINT: Looks for the presence of `shard(SHARD_HINT, ...)` which is treated\n  // as a placeholder to replace with `shard(num_workers, worker_index)`.\n  HINT = 3;\n  // OFF: No sharding will be performed.\n  OFF = -1;\n}\n\n// next: 5\nmessage AutotuneOptions {\n  // Whether to automatically tune performance knobs.\n  oneof optional_enabled {\n    bool enabled = 1;\n  }\n  // When autotuning is enabled (through autotune), determines the CPU budget to\n  // use. Values greater than the number of schedulable CPU cores are allowed\n  // but may result in CPU contention.\n  oneof optional_cpu_budget {\n    int32 cpu_budget = 2;\n  }\n  // When autotuning is enabled (through autotune), determines the RAM budget to\n  // use. Values greater than the available RAM in bytes may result in OOM. If\n  // 0, defaults to half of the available RAM in bytes.\n  oneof optional_ram_budget {\n    int64 ram_budget = 3;\n  }\n\n  // When autotuning is enabled (through autotune), determines the algorithm to\n  // use. If not explicitly set by user, autotuning will follow HILL_CLIMB\n  // algorithm but has more flexibility to tune parameters more aggressively,\n  // in which case the behavior is implementation specific and may change over\n  // time.\n  oneof optional_autotune_algorithm {\n    model.AutotuneAlgorithm autotune_algorithm = 4;\n  }\n}\n\n// next: 2\nmessage CardinalityOptions {\n  enum ComputeLevel {\n    CARDINALITY_COMPUTE_UNSPECIFIED = 0;\n    // Cardinality will only be computed if it can be determined in a cheap\n    // manner (ie. without reading from file sources). If the cardinality would\n    // be nontrivial to compute, Cardinality() will return UNKNOWN_CARDINALITY.\n    CARDINALITY_COMPUTE_LOW = 1;\n    // Moderate effort will be made to determine cardinality, such as reading\n    // index data from source files. If significant work is needed to compute\n    // cardinality (e.g. reading entire source file contents or executing user\n    // defined functions), Cardinality() will return UNKNOWN_CARDINALITY.\n    CARDINALITY_COMPUTE_MODERATE = 2;\n  }\n  ComputeLevel compute_level = 1;\n}\n\n// next: 3\nmessage DistributeOptions {\n  AutoShardPolicy auto_shard_policy = 1;\n  // The number of devices attached to this input pipeline.\n  oneof optional_num_devices {\n    int32 num_devices = 2;\n  }\n}\n\n// next: 18\nmessage OptimizationOptions {\n  // Whether to apply default graph optimizations. If False, only graph\n  // optimizations that have been explicitly enabled will be applied.\n  oneof optional_apply_default_optimizations {\n    bool apply_default_optimizations = 1;\n  }\n  reserved 2;\n  reserved 3;\n  reserved 4;\n  reserved 5;\n  // Whether to fuse filter transformations.\n  oneof optional_filter_fusion {\n    bool filter_fusion = 6;\n  }\n  // NOTE: field id 7 deleted in June 2021.\n  reserved 7;\n  // NOTE: field id 8 deleted in June 2021.\n  reserved 8;\n  // Whether to fuse map and batch transformations.\n  oneof optional_map_and_batch_fusion {\n    bool map_and_batch_fusion = 9;\n  }\n  // Whether to fuse map and filter transformations.\n  oneof optional_map_and_filter_fusion {\n    bool map_and_filter_fusion = 10;\n  }\n  // Whether to fuse map transformations.\n  oneof optional_map_fusion {\n    bool map_fusion = 11;\n  }\n  // Whether to parallelize stateless map transformations.\n  oneof optional_map_parallelization {\n    bool map_parallelization = 12;\n  }\n\n  // NOTE: field id 13 deleted in June 2021.\n  reserved 13;\n\n  // Whether to eliminate no-op transformations.\n  oneof optional_noop_elimination {\n    bool noop_elimination = 14;\n  }\n  // Whether to parallelize copying of batch elements. This optimization is\n  // highly experimental and can cause performance degradation (e.g. when the\n  // parallelization overhead exceeds the benefits of performing the data copies\n  // in parallel). You should only enable this optimization if a) your input\n  // pipeline is bottlenecked on batching and b) you have validated that this\n  // optimization improves performance.\n  oneof optional_parallel_batch {\n    bool parallel_batch = 15;\n  }\n  // Field id 16 was removed in 06/2021.\n  reserved 16;\n  // Whether to fuse shuffle and repeat transformations.\n  oneof optional_shuffle_and_repeat_fusion {\n    bool shuffle_and_repeat_fusion = 17;\n  }\n}\n\n// next: 3\nmessage ThreadingOptions {\n  // If set, it overrides the maximum degree of intra-op parallelism.\n  oneof optional_max_intra_op_parallelism {\n    int32 max_intra_op_parallelism = 1;\n  }\n  // If set, the dataset will use a private threadpool of the given size.\n  oneof optional_private_threadpool_size {\n    int32 private_threadpool_size = 2;\n  }\n}\n\n// Represents how to handle external state during serialization.\nenum ExternalStatePolicy {\n  POLICY_WARN = 0;\n  POLICY_IGNORE = 1;\n  POLICY_FAIL = 2;\n}\n\n// Message stored with Dataset objects to control how datasets are processed and\n// optimized.\n//\n// next: 8\nmessage Options {\n  // Whether the outputs need to be produced in deterministic order.\n  oneof optional_deterministic {\n    bool deterministic = 1;\n  }\n  // The distribution strategy options associated with the dataset.\n  AutotuneOptions autotune_options = 7;\n  // The distribution strategy options associated with the dataset.\n  DistributeOptions distribute_options = 2;\n  // The optimization options associated with the dataset.\n  OptimizationOptions optimization_options = 3;\n  // Whether to introduce 'slack' in the last `prefetch` of the input pipeline,\n  // if it exists. This may reduce CPU contention with accelerator host-side\n  // activity at the start of a step. The slack frequency is determined by the\n  // number of devices attached to this input pipeline.\n  oneof optional_slack {\n    bool slack = 4;\n  }\n  // The threading options associated with the dataset.\n  ThreadingOptions threading_options = 5;\n  // This option can be used to override the default policy for how to handle\n  // external state when serializing a dataset or checkpointing its iterator.\n  // There are three settings available - IGNORE: External state is ignored\n  // without a warning; WARN: External state is ignored and a warning is logged;\n  // FAIL: External state results in an error.\n  oneof optional_external_state_policy {\n    ExternalStatePolicy external_state_policy = 6;\n  }\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/device_attributes.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"DeviceAttributesProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/device_attributes_go_proto\";\n\nmessage InterconnectLink {\n  int32 device_id = 1;\n  string type = 2;\n  int32 strength = 3;\n}\n\nmessage LocalLinks {\n  repeated InterconnectLink link = 1;\n}\n\nmessage DeviceLocality {\n  // Optional bus locality of device.  Default value of 0 means\n  // no specific locality.  Specific localities are indexed from 1.\n  int32 bus_id = 1;\n\n  // Optional NUMA locality of device.\n  int32 numa_node = 2;\n\n  // Optional local interconnect links to other devices.\n  LocalLinks links = 3;\n}\n\nmessage DeviceAttributes {\n  // Fully specified name of the device within a cluster.\n  string name = 1;\n\n  // String representation of device_type.\n  string device_type = 2;\n\n  // Memory capacity of device in bytes.\n  int64 memory_limit = 4;\n\n  // Platform-specific data about device that may be useful\n  // for supporting efficient data transfers.\n  DeviceLocality locality = 5;\n\n  // A device is assigned a global unique number each time it is\n  // initialized. \"incarnation\" should never be 0.\n  fixed64 incarnation = 6;\n\n  // String representation of the physical device that this device maps to.\n  string physical_device_desc = 7;\n\n  // A physical device ID for use in XLA DeviceAssignments, unique across\n  // clients in a multi-client setup. Set to -1 if unavailable, non-negative\n  // otherwise.\n  int64 xla_global_id = 8;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/full_type.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"FullTypeProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/full_type_go_proto\";\n\n// Experimental. Represents the complete type information of a TensorFlow value.\nenum FullTypeId {\n  // The default represents an uninitialized values.\n  TFT_UNSET = 0;\n\n  // Type symbols. Used to construct more complex type expressions like\n  // algebraic data types.\n\n  // Type variables may serve as placeholder for any other type ID in type\n  // templates.\n  //\n  // Examples:\n  //   TFT_DATASET[TFT_VAR[\"T\"]] is a Dataset returning a type indicated by \"T\".\n  //   TFT_TENSOR[TFT_VAR[\"T\"]] is a Tensor of n element type indicated by \"T\".\n  //   TFT_TENSOR[TFT_VAR[\"T\"]], TFT_TENSOR[TFT_VAR[\"T\"]] are two tensors of\n  //     identical element types.\n  //   TFT_TENSOR[TFT_VAR[\"P\"]], TFT_TENSOR[TFT_VAR[\"Q\"]] are two tensors of\n  //     independent element types.\n  //\n  TFT_VAR = 1;\n\n  // Wildcard type. Describes a parameter of unknown type. In TensorFlow, that\n  // can mean either a \"Top\" type (accepts any type), or a dynamically typed\n  // object whose type is unknown in context.\n  // Important: \"unknown\" does not necessarily mean undeterminable!\n  TFT_ANY = 2;\n\n  // The algebraic product type. This is an algebraic type that may be used just\n  // for logical grouping. Not to confused with TFT_TUPLE which describes a\n  // concrete object of several elements.\n  //\n  // Example:\n  //   TFT_DATASET[TFT_PRODUCT[TFT_TENSOR[TFT_INT32], TFT_TENSOR[TFT_FLOAT64]]]\n  //     is a Dataset producing two tensors, an integer one and a float one.\n  //\n  TFT_PRODUCT = 3;\n\n  // Represents a named field, with the name stored in the attribute.\n  //\n  // Parametrization:\n  //   TFT_NAMED[<type>]{<name>}\n  //   * <type> is the type of the field\n  //   * <name> is the field name, as string (thpugh can theoretically be an int\n  //     as well)\n  //\n  // Example:\n  //   TFT_RECORD[\n  //     TFT_NAMED[TFT_TENSOR[TFT_INT32]]{'foo'},\n  //     TFT_NAMED[TFT_TENSOR[TFT_FLOAT32]]{'bar'},\n  //   ]\n  //     is a structure with two fields, an int tensor \"foo\" and a float tensor\n  //     \"bar\".\n  TFT_NAMED = 4;\n\n  // Template definition. Expands the variables by repeating a template as\n  // arguments of container.\n  //\n  // Parametrization:\n  //   TFT_FOR_EACH[<container_type>, <template>, <expansions>]\n  //   * <container_type> is the type of the container that the template will be\n  //     expanded into\n  //   * <template> is any type definition that potentially contains type\n  //     variables\n  //   * <expansions> is a TFT_VAR and may include more types in the future\n  //\n  // Example:\n  //   TFT_FOR_EACH[\n  //         TFT_PRODUCT,\n  //         TFT_TENSOR[TFT_VAR[\"t\"]],\n  //         TFT_VAR[\"t\"]\n  //     ]\n  //     will substitute a T = TFT_INT32 to TFT_PRODUCT[TFT_TENSOR[TFT_INT32]]\n  //     and a T = (TFT_INT32, TFT_INT64) to\n  //     TFT_PRODUCT[TFT_TENSOR[TFT_INT32], TFT_TENSOR[TFT_INT64]].\n  TFT_FOR_EACH = 20;\n\n  // Callable types describe functions and ops.\n  //\n  // Parametrization:\n  //   TFT_CALLABLE[<arg type>, <return type>]\n  //   * <arg type> is the type of the arguments; TFT_PRODUCT represents\n  //   multiple\n  //     arguments.\n  //   * <return type> is the return type; TFT_PRODUCT represents multiple\n  //     return values (that means that callables returning multiple things\n  //     don't necessarily return a single tuple).\n  //\n  // Example:\n  //   TFT_CALLABLE[\n  //     TFT_ANY,\n  //     TFT_PRODUCT[TFT_TENSOR[TFT_INT32], TFT_TENSOR[TFT_FLOAT64]],\n  //   ]\n  //     is a callable with unspecified (for now) input arguments, and\n  //     two return values of type tensor.\n  //\n  TFT_CALLABLE = 100;\n\n  // Concrete type IDs, representing \"proper\" data types that can describe\n  // runtime TensorFlow objects.\n\n  // The usual Tensor. This is a parametric type.\n  //\n  // Parametrization:\n  //   TFT_TENSOR[<element type>, <shape type>]\n  //   * <element type> is currently limited to one of the element types\n  //     defined below.\n  //   * <shape type> is not yet defined, and may only be TFT_UNKNOWN for now.\n  //\n  // A TFT_SHAPE type will be defined in the future.\n  //\n  // Example:\n  //   TFT_TENSOR[TFT_INT32, TFT_UNKNOWN]\n  //     is a Tensor of int32 element type and unknown shape.\n  //\n  // TODO(mdan): Define TFT_SHAPE and add more examples.\n  TFT_TENSOR = 1000;\n\n  // Array (or tensorflow::TensorList in the variant type registry).\n  // Note: this is not to be confused with the deprecated `TensorArray*` ops\n  // which are not supported by FullType.\n  // This type represents a random-access list whose elements can be\n  // described by a single type. Although immutable, Array is expected to\n  // support efficient mutation semantics (i.e. element update) in the\n  // user-facing API.\n  // The element type may be generic or even TFT_ANY for a heterogenous list.\n  //\n  // Parametrization:\n  //   TFT_ARRAY[<element type>]\n  //   * <element type> may be any concrete type.\n  //\n  // Examples:\n  //   TFT_ARRAY[TFT_TENSOR[TFT_INT32]] is a TensorArray holding int32 Tensors\n  //     of any shape.\n  //   TFT_ARRAY[TFT_TENSOR[TFT_UNKNOWN]] is a TensorArray holding Tensors of\n  //     mixed element types.\n  //   TFT_ARRAY[TFT_UNKNOWN] is a TensorArray holding any element type.\n  //   TFT_ARRAY[] is equivalent to TFT_ARRAY[TFT_UNKNOWN].\n  //   TFT_ARRAY[TFT_ARRAY[]] is an array or arrays (of unknown types).\n  TFT_ARRAY = 1001;\n\n  // Optional (or tensorflow::OptionalVariant in the variant type registry).\n  // This type represents a value that may either hold an element of a single\n  // specified type, or nothing at all.\n  //\n  // Parametrization:\n  //   TFT_OPTIONAL[<element type>]\n  //   * <element type> may be any concrete type.\n  //\n  // Examples:\n  //   TFT_OPTIONAL[TFT_TENSOR[TFT_INT32]] is an Optional holding an int32\n  //     Tensor of any shape.\n  TFT_OPTIONAL = 1002;\n\n  // Literal types describe compile-time constant values.\n  // Literal types may also participate in dependent types.\n  //\n  // Parametrization:\n  //   TFT_LITERAL[<value type>]{<value>}\n  //   * <value type> may be any concrete type compatible that can hold <value>\n  //   * <value> is the type's attribute, and holds the actual literal value\n  //\n  // Examples:\n  //   TFT_LITERAL[TFT_INT32]{1} is the compile-time constant 1.\n  TFT_LITERAL = 1003;\n\n  // Type attributes. These always appear in the parametrization of a type,\n  // never alone. For example, there is no such thing as a \"bool\" TensorFlow\n  // object (for now).\n\n  // The bool element type.\n  // TODO(mdan): Quantized types, legacy representations (e.g. ref)\n  TFT_BOOL = 200;\n  // Integer element types.\n  TFT_UINT8 = 201;\n  TFT_UINT16 = 202;\n  TFT_UINT32 = 203;\n  TFT_UINT64 = 204;\n  TFT_INT8 = 205;\n  TFT_INT16 = 206;\n  TFT_INT32 = 207;\n  TFT_INT64 = 208;\n  // Floating-point element types.\n  TFT_HALF = 209;\n  TFT_FLOAT = 210;\n  TFT_DOUBLE = 211;\n  TFT_BFLOAT16 = 215;\n  // Complex element types.\n  // TODO(mdan): Represent as TFT_COMPLEX[TFT_DOUBLE] instead?\n  TFT_COMPLEX64 = 212;\n  TFT_COMPLEX128 = 213;\n  // The string element type.\n  TFT_STRING = 214;\n\n  // Other types that we don't know yet whether they will become part of the\n  // core type system or be consisdered third-party (and consequently moved to\n  // user-defined type mechanisms). Presently, they are effectively in the core\n  // type system, because key compilation passes like Placer account for their\n  // existence.\n\n  // Datasets created by tf.data ops and APIs. Datasets have generator/iterable\n  // semantics, that is, one can construct an iterator from them. Like\n  // Array, they are considered to return elements that can be described\n  // by a single type. Unlike Array, they do not support random access or\n  // mutation, and can potentially produce an infinite number of elements.\n  // A datasets can produce logical structures (e.g. multiple elements). This\n  // is expressed using TFT_PRODUCT.\n  //\n  //\n  // Parametrization: TFT_ARRAY[<element type>].\n  //   * <element type> may be a concrete type or a type symbol. It represents\n  //     the data type of the elements produced by the dataset.\n  //\n  // Examples:\n  //   TFT_DATSET[TFT_TENSOR[TFT_INT32]] is a Dataset producing single int32\n  //     Tensors of unknown shape.\n  //   TFT_DATSET[TFT_PRODUCT[TFT_TENSOR[TFT_INT32], TFT_TENSOR[TFT_FLOAT32]] is\n  //     a Dataset producing pairs of Tensors, one integer and one float.\n  // Note: The high ID number is to prepare for the eventuality that Datasets\n  // will be supported by user types in the future.\n  TFT_DATASET = 10102;\n\n  // A ragged tensor created by tf.ragged ops and APIs.\n  //\n  // Parametrization: TFT_RAGGED[<element_type>].\n  TFT_RAGGED = 10103;\n\n  // A mutex lock tensor, produced by tf.raw_ops.MutexLock.\n  // Unlike strict execution models, where ownership of a lock is denoted by\n  // \"running after the lock has been acquired\", in non-strict mode, lock\n  // ownership is in the true sense: \"the op argument representing the lock is\n  // available\".\n  // Mutex locks are the dynamic counterpart of control dependencies.\n  // TODO(mdan): Properly document this thing.\n  //\n  // Parametrization: TFT_MUTEX_LOCK[].\n  TFT_MUTEX_LOCK = 10202;\n\n  // The equivalent of a Tensor with DT_VARIANT dtype, kept here to simplify\n  // translation. This type should not normally appear after type inference.\n  // Note that LEGACY_VARIANT != ANY: TENSOR[INT32] is a subtype of ANY, but is\n  // not a subtype of LEGACY_VARIANT.\n  TFT_LEGACY_VARIANT = 10203;\n}\n\n// Highly experimental and very likely to change.\n// This encoding uses tags instead of dedicated messages for regularity. In\n// particular the encoding imposes no restrictions on what the parameters of any\n// type should be, which in particular needs to be true for type symbols.\nmessage FullTypeDef {\n  // The principal type represented by this object. This may be a concrete type\n  // (Tensor, Dataset) a type variable (used for dependent types) a type\n  // symbol (Any, Union). See FullTypeId for details.\n  FullTypeId type_id = 1;\n\n  repeated FullTypeDef args = 2;\n\n  // Literal values of this type object, if the the type admits one.\n  // For example, a type variable admits a string attribute - its name.\n  // Shape-related types may admit int attributes - their static shape values.\n  // Fields for more data types to be added as needed.\n  oneof attr {\n    string s = 3;\n    int64 i = 4;\n    // TODO(mdan): list/tensor, map? Need to reconcile with TFT_RECORD, etc.\n  }\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/function.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/attr_value.proto\";\nimport \"tensorflow/core/framework/node_def.proto\";\nimport \"tensorflow/core/framework/op_def.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"FunctionProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/function_go_proto\";\n\n// A library is a set of named functions.\nmessage FunctionDefLibrary {\n  repeated FunctionDef function = 1;\n  repeated GradientDef gradient = 2;\n  repeated RegisteredGradient registered_gradients = 3;\n}\n\n// A function can be instantiated when the runtime can bind every attr\n// with a value. When a GraphDef has a call to a function, it must\n// have binding for every attr defined in the signature.\n//\n// TODO(zhifengc):\n//   * device spec, etc.\nmessage FunctionDef {\n  // The definition of the function's name, arguments, return values,\n  // attrs etc.\n  OpDef signature = 1;\n\n  // Attributes specific to this function definition.\n  map<string, AttrValue> attr = 5;\n\n  // Attributes for function arguments. These attributes are the same set of\n  // valid attributes as to _Arg nodes.\n  message ArgAttrs {\n    map<string, AttrValue> attr = 1;\n  }\n  map<uint32, ArgAttrs> arg_attr = 7;\n\n  // Unique IDs for each resource argument, used to track aliasing resources. If\n  // Argument A and Argument B alias each other, then\n  // resource_arg_unique_ids[A.index] == resource_arg_unique_ids[B.index].\n  //\n  // If this field is empty, none of the arguments could alias; otherwise, every\n  // resource argument should have an entry in this field.\n  //\n  // When instantiated, the unique IDs will be attached to the _Arg nodes'\n  // \"_resource_arg_unique_id\" attribute.\n  map<uint32, uint32> resource_arg_unique_id = 8;\n\n  // NOTE: field id 2 deleted on Jan 11, 2017, GraphDef version 21.\n  reserved 2;\n\n  // In both of the following fields, there is the need to specify an\n  // output that is used as either the input to another node (in\n  // `node_def`) or as a return value of the function (in `ret`).\n  // Unlike the NodeDefs in GraphDef, we need to be able to specify a\n  // list in some cases (instead of just single outputs).  Also, we\n  // need to be able to deal with lists of unknown length (so the\n  // output index may not be known at function definition time).  So\n  // we use the following format instead:\n  // * \"fun_in\" where \"fun_in\" is the name of a function input arg in\n  //   the `signature` field above.  This represents that input, whether\n  //   it is a single tensor or a list.\n  // * \"fun_in:0\" gives the first element of a function input arg (a\n  //   non-list input is considered a list of length 1 for these\n  //   purposes).\n  // * \"node:out\" where \"node\" is the name of a node in `node_def` and\n  //   \"out\" is the name one of its op's output arguments (the name\n  //   comes from the OpDef of the node's op). This represents that\n  //   node's output, whether it is a single tensor or a list.\n  //   Note: We enforce that an op's output arguments are never\n  //   renamed in the backwards-compatibility test.\n  // * \"node:out:0\" gives the first element of a node output arg (a\n  //   non-list output is considered a list of length 1 for these\n  //   purposes).\n  //\n  // NOT CURRENTLY SUPPORTED (but may be in the future):\n  // * \"node:out:-1\" gives last element in a node output list\n  // * \"node:out:1:\" gives a list with all but the first element in a\n  //   node output list\n  // * \"node:out::-1\" gives a list with all but the last element in a\n  //   node output list\n\n  // The body of the function.  Unlike the NodeDefs in a GraphDef, attrs\n  // may have values of type `placeholder` and the `input` field uses\n  // the \"output\" format above.\n\n  // By convention, \"op\" in node_def is resolved by consulting with a\n  // user-defined library first. If not resolved, \"func\" is assumed to\n  // be a builtin op.\n  repeated NodeDef node_def = 3;\n\n  // A mapping from the output arg names from `signature` to the\n  // outputs from `node_def` that should be returned by the function.\n  map<string, string> ret = 4;\n\n  // A mapping from control output names from `signature` to node names in\n  // `node_def` which should be control outputs of this function.\n  map<string, string> control_ret = 6;\n}\n\n// GradientDef defines the gradient function of a function defined in\n// a function library.\n//\n// A gradient function g (specified by gradient_func) for a function f\n// (specified by function_name) must follow the following:\n//\n// The function 'f' must be a numerical function which takes N inputs\n// and produces M outputs. Its gradient function 'g', which is a\n// function taking N + M inputs and produces N outputs.\n//\n// I.e. if we have\n//    (y1, y2, ..., y_M) = f(x1, x2, ..., x_N),\n// then, g is\n//    (dL/dx1, dL/dx2, ..., dL/dx_N) = g(x1, x2, ..., x_N,\n//                                      dL/dy1, dL/dy2, ..., dL/dy_M),\n// where L is a scalar-value function of (x1, x2, ..., xN) (e.g., the\n// loss function). dL/dx_i is the partial derivative of L with respect\n// to x_i.\nmessage GradientDef {\n  string function_name = 1;  // The function name.\n  string gradient_func = 2;  // The gradient function's name.\n}\n\n// RegisteredGradient stores a gradient function that is registered in the\n// gradients library and used in the ops of a function in the function library.\n// Unlike GradientDef, these gradients are identified by op type, and not\n// directly linked to any function.\nmessage RegisteredGradient {\n  string gradient_func = 1;       // The gradient function's name.\n  string registered_op_type = 2;  // The gradient function's registered op type.\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/graph.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/function.proto\";\nimport \"tensorflow/core/framework/node_def.proto\";\nimport \"tensorflow/core/framework/versions.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"GraphProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/graph_go_proto\";\n\n// Represents the graph of operations\nmessage GraphDef {\n  repeated NodeDef node = 1;\n\n  // Compatibility versions of the graph.  See core/public/version.h for version\n  // history.  The GraphDef version is distinct from the TensorFlow version, and\n  // each release of TensorFlow will support a range of GraphDef versions.\n  VersionDef versions = 4;\n\n  // Deprecated single version field; use versions above instead.  Since all\n  // GraphDef changes before \"versions\" was introduced were forward\n  // compatible, this field is entirely ignored.\n  int32 version = 3 [deprecated = true];\n\n  // \"library\" provides user-defined functions.\n  //\n  // Naming:\n  //   * library.function.name are in a flat namespace.\n  //     NOTE: We may need to change it to be hierarchical to support\n  //     different orgs. E.g.,\n  //     { \"/google/nn\", { ... }},\n  //     { \"/google/vision\", { ... }}\n  //     { \"/org_foo/module_bar\", { ... }}\n  //     map<string, FunctionDefLib> named_lib;\n  //   * If node[i].op is the name of one function in \"library\",\n  //     node[i] is deemed as a function call. Otherwise, node[i].op\n  //     must be a primitive operation supported by the runtime.\n  //\n  //\n  // Function call semantics:\n  //\n  //   * The callee may start execution as soon as some of its inputs\n  //     are ready. The caller may want to use Tuple() mechanism to\n  //     ensure all inputs are ready in the same time.\n  //\n  //   * The consumer of return values may start executing as soon as\n  //     the return values the consumer depends on are ready.  The\n  //     consumer may want to use Tuple() mechanism to ensure the\n  //     consumer does not start until all return values of the callee\n  //     function are ready.\n  FunctionDefLibrary library = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/graph_transfer_info.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/types.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"GraphTransferInfoProto\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/graph_transfer_info_go_proto\";\n\nmessage GraphTransferNodeInput {\n  int32 node_id = 1;\n  int32 output_port = 2;\n}\nmessage GraphTransferNodeInfo {\n  string name = 1;\n  int32 node_id = 2;\n  string type_name = 3;\n  int32 soc_op_id = 4;\n  int32 padding_id = 5;\n  int32 input_count = 6;\n  int32 output_count = 7;\n}\nmessage GraphTransferConstNodeInfo {\n  string name = 1;\n  int32 node_id = 2;\n  repeated int64 shape = 3;\n  bytes data = 4;\n  DataType dtype = 5;\n}\nmessage GraphTransferNodeInputInfo {\n  int32 node_id = 1;\n  repeated GraphTransferNodeInput node_input = 2;\n}\nmessage GraphTransferNodeOutputInfo {\n  int32 node_id = 1;\n  repeated int32 max_byte_size = 2;\n}\nmessage GraphTransferGraphInputNodeInfo {\n  string name = 1;\n  repeated int64 shape = 2;\n  DataType dtype = 3;\n}\n\nmessage GraphTransferGraphOutputNodeInfo {\n  string name = 1;\n  repeated int64 shape = 2;\n  DataType dtype = 3;\n}\n\n// Protocol buffer representing a handle to a tensorflow resource. Handles are\n// not valid across executions, but can be serialized back and forth from within\n// a single run.\nmessage GraphTransferInfo {\n  enum Destination {\n    NOP = 0;\n    HEXAGON = 1;\n  }\n\n  repeated GraphTransferNodeInfo node_info = 1;\n  repeated GraphTransferConstNodeInfo const_node_info = 2;\n  repeated GraphTransferNodeInputInfo node_input_info = 3;\n  repeated GraphTransferNodeOutputInfo node_output_info = 4;\n  // Input Node parameters of transferred graph\n  repeated GraphTransferGraphInputNodeInfo graph_input_node_info = 5;\n  repeated GraphTransferGraphOutputNodeInfo graph_output_node_info = 6;\n  // Destination of graph transfer\n  Destination destination = 7;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/kernel_def.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/attr_value.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"KernelDefProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/kernel_def_go_proto\";\n\nmessage KernelDef {\n  // Must match the name of an Op.\n  string op = 1;\n\n  // Type of device this kernel runs on.\n  string device_type = 2;\n\n  message AttrConstraint {\n    // Name of an attr from the Op.\n    string name = 1;\n\n    // A list of values that this kernel supports for this attr.\n    // Like OpDef.AttrDef.allowed_values, except for kernels instead of Ops.\n    AttrValue allowed_values = 2;\n  }\n  repeated AttrConstraint constraint = 3;\n\n  // Names of the Op's input_/output_args that reside in host memory\n  // instead of device memory.\n  repeated string host_memory_arg = 4;\n\n  // This allows experimental kernels to be registered for an op that\n  // won't be used unless the user specifies a \"_kernel\" attr with\n  // value matching this.\n  string label = 5;\n\n  // Prioritization of kernel amongst different devices. By default we assume\n  // priority is 0. The higher the priority the better. By default (i.e. if\n  // this is not set), we prefer GPU kernels over CPU.\n  int32 priority = 6;\n}\n\n// A collection of KernelDefs\nmessage KernelList {\n  repeated KernelDef kernel = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/log_memory.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/tensor_description.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"LogMemoryProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/log_memory_go_proto\";\n\nmessage MemoryLogStep {\n  // Process-unique step id.\n  int64 step_id = 1;\n\n  // Handle describing the feeds and fetches of the step.\n  string handle = 2;\n}\n\nmessage MemoryLogTensorAllocation {\n  // Process-unique step id.\n  int64 step_id = 1;\n\n  // Name of the kernel making the allocation as set in GraphDef,\n  // e.g., \"affine2/weights/Assign\".\n  string kernel_name = 2;\n\n  // Allocated tensor details.\n  TensorDescription tensor = 3;\n}\n\nmessage MemoryLogTensorDeallocation {\n  // Id of the tensor buffer being deallocated, used to match to a\n  // corresponding allocation.\n  int64 allocation_id = 1;\n\n  // Name of the allocator used.\n  string allocator_name = 2;\n}\n\nmessage MemoryLogTensorOutput {\n  // Process-unique step id.\n  int64 step_id = 1;\n\n  // Name of the kernel producing an output as set in GraphDef, e.g.,\n  // \"affine2/weights/Assign\".\n  string kernel_name = 2;\n\n  // Index of the output being set.\n  int32 index = 3;\n\n  // Output tensor details.\n  TensorDescription tensor = 4;\n}\n\nmessage MemoryLogRawAllocation {\n  // Process-unique step id.\n  int64 step_id = 1;\n\n  // Name of the operation making the allocation.\n  string operation = 2;\n\n  // Number of bytes in the allocation.\n  int64 num_bytes = 3;\n\n  // Address of the allocation.\n  uint64 ptr = 4;\n\n  // Id of the tensor buffer being allocated, used to match to a\n  // corresponding deallocation.\n  int64 allocation_id = 5;\n\n  // Name of the allocator used.\n  string allocator_name = 6;\n}\n\nmessage MemoryLogRawDeallocation {\n  // Process-unique step id.\n  int64 step_id = 1;\n\n  // Name of the operation making the deallocation.\n  string operation = 2;\n\n  // Id of the tensor buffer being deallocated, used to match to a\n  // corresponding allocation.\n  int64 allocation_id = 3;\n\n  // Name of the allocator used.\n  string allocator_name = 4;\n\n  // True if the deallocation is queued and will be performed later,\n  // e.g. for GPU lazy freeing of buffers.\n  bool deferred = 5;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/model.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.data.model;\n\noption cc_enable_arenas = true;\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/model_go_proto\";\n\n// Class of a node in the performance model.\nenum NodeClass {\n  UNKNOWN = 0;\n  INTERLEAVE_MANY = 1;\n  ASYNC_INTERLEAVE_MANY = 2;\n  KNOWN_RATIO = 3;\n  ASYNC_KNOWN_RATIO = 4;\n  UNKNOWN_RATIO = 5;\n}\n\n// Algorithm used for model autotuning optimization.\nenum AutotuneAlgorithm {\n  DEFAULT = 0;\n  HILL_CLIMB = 1;\n  GRADIENT_DESCENT = 2;\n  MAX_PARALLELISM = 3;\n}\n\n// Protocol buffer representing the data used by the autotuning modeling\n// framework.\nmessage ModelProto {\n  // General representation of a node in the model.\n  message Node {\n    // Unique node ID.\n    int64 id = 1;\n\n    // Human-readable name of the node.\n    string name = 2;\n\n    // An indication whether autotuning is enabled for this node.\n    bool autotune = 3;\n\n    // The number of bytes stored in this node's buffer.\n    int64 buffered_bytes = 4;\n\n    // The number of elements stored in this node's buffer.\n    int64 buffered_elements = 5;\n\n    // The number of bytes consumed by the node.\n    int64 bytes_consumed = 6;\n\n    // The number of bytes produced by the node.\n    int64 bytes_produced = 7;\n\n    // The number of elements produced by the node.\n    int64 num_elements = 8;\n\n    // The aggregate processing time spent in this node.\n    int64 processing_time = 9;\n\n    // An indication whether this node records metrics about produced and\n    // consumed elements.\n    bool record_metrics = 10;\n\n    // Represents a node parameter.\n    message Parameter {\n      // Human-readable name of the parameter.\n      string name = 1;\n\n      // Identifies the model value of the parameter. This can be different from\n      // the actual value (e.g. during optimization search).\n      double value = 2;\n\n      // The actual value of the parameter.\n      double state_value = 3;\n\n      // Minimum value of the parameter.\n      double min = 4;\n\n      // Maximum value of the parameter.\n      double max = 5;\n\n      // Identifies whether the parameter should participate in autotuning.\n      bool tunable = 6;\n    }\n\n    // Parameters of this node.\n    repeated Parameter parameters = 11;\n\n    // Statistic of inputs processing time history.\n    double input_processing_time_sum = 12;\n    int64 input_processing_time_count = 13;\n\n    // IDs of inputs of this node.\n    repeated int64 inputs = 14;\n\n    // Class of this node.\n    NodeClass node_class = 15;\n\n    // Ratio of input to output elements. This is only used by KNOWN_RATIO and\n    // ASYNC_KNOWN_RATIO nodes.\n    double ratio = 16;\n\n    // Ratio identifies how many parallelism calls are introduced by one\n    // buffered element. This is only used by ASYNC_KNOWN_RATIO nodes.\n    double memory_ratio = 17;\n  }\n\n  // Map of node IDs to nodes of this model.\n  map<int64, Node> nodes = 1;\n\n  // ID of the output node of this model.\n  int64 output = 2;\n\n  // Counter for node IDs of this model.\n  int64 id_counter = 3;\n\n  reserved 4;\n\n  // Contains parameters of the model autotuning optimization.\n  message OptimizationParams {\n    // Algorithm used for autotuning optimization.\n    AutotuneAlgorithm algorithm = 1;\n\n    // Number of available logical threads.\n    int64 cpu_budget = 2;\n\n    // Amount of available memory in bytes.\n    int64 ram_budget = 3;\n\n    // Time between two consecutive `GetNext` calls to the iterator represented\n    // by the output node.\n    double model_input_time = 4;\n  }\n\n  OptimizationParams optimization_params = 5;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/node_def.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/attr_value.proto\";\nimport \"tensorflow/core/framework/full_type.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"NodeProto\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/node_def_go_proto\";\n\nmessage NodeDef {\n  // The name given to this operator. Used for naming inputs,\n  // logging, visualization, etc.  Unique within a single GraphDef.\n  // Must match the regexp \"[A-Za-z0-9.][A-Za-z0-9_>./]*\".\n  string name = 1;\n\n  // The operation name.  There may be custom parameters in attrs.\n  // Op names starting with an underscore are reserved for internal use.\n  string op = 2;\n\n  // Each input is \"node:src_output\" with \"node\" being a string name and\n  // \"src_output\" indicating which output tensor to use from \"node\". If\n  // \"src_output\" is 0 the \":0\" suffix can be omitted.  Regular inputs\n  // may optionally be followed by control inputs that have the format\n  // \"^node\".\n  repeated string input = 3;\n\n  // A (possibly partial) specification for the device on which this\n  // node should be placed.\n  // The expected syntax for this string is as follows:\n  //\n  // DEVICE_SPEC ::= PARTIAL_SPEC\n  //\n  // PARTIAL_SPEC ::= (\"/\" CONSTRAINT) *\n  // CONSTRAINT ::= (\"job:\" JOB_NAME)\n  //              | (\"replica:\" [1-9][0-9]*)\n  //              | (\"task:\" [1-9][0-9]*)\n  //              | (\"device:\" [A-Za-z]* \":\" ([1-9][0-9]* | \"*\") )\n  //\n  // Valid values for this string include:\n  // * \"/job:worker/replica:0/task:1/device:GPU:3\"  (full specification)\n  // * \"/job:worker/device:GPU:3\"                   (partial specification)\n  // * \"\"                                    (no specification)\n  //\n  // If the constraints do not resolve to a single device (or if this\n  // field is empty or not present), the runtime will attempt to\n  // choose a device automatically.\n  string device = 4;\n\n  // Operation-specific graph-construction-time configuration.\n  // Note that this should include all attrs defined in the\n  // corresponding OpDef, including those with a value matching\n  // the default -- this allows the default to change and makes\n  // NodeDefs easier to interpret on their own.  However, if\n  // an attr with a default is not specified in this list, the\n  // default will be used.\n  // The \"names\" (keys) must match the regexp \"[a-z][a-z0-9_]+\" (and\n  // one of the names from the corresponding OpDef's attr field).\n  // The values must have a type matching the corresponding OpDef\n  // attr's type field.\n  // TODO(josh11b): Add some examples here showing best practices.\n  map<string, AttrValue> attr = 5;\n\n  message ExperimentalDebugInfo {\n    // Opaque string inserted into error messages created by the runtime.\n    //\n    // This is intended to store the list of names of the nodes from the\n    // original graph that this node was derived. For example if this node, say\n    // C, was result of a fusion of 2 nodes A and B, then 'original_node' would\n    // be {A, B}. This information can be used to map errors originating at the\n    // current node to some top level source code.\n    repeated string original_node_names = 1;\n\n    // This is intended to store the list of names of the functions from the\n    // original graph that this node was derived. For example if this node, say\n    // C, was result of a fusion of node A in function FA and node B in function\n    // FB, then `original_funcs` would be {FA, FB}. If the node is in the top\n    // level graph, the `original_func` is empty. This information, with the\n    // `original_node_names` can be used to map errors originating at the\n    // current ndoe to some top level source code.\n    repeated string original_func_names = 2;\n  }\n\n  // This stores debug information associated with the node.\n  ExperimentalDebugInfo experimental_debug_info = 6;\n\n  // The complete type of this node. Experimental and subject to change.\n  // Currently, the field only contains the return types of the node. That will\n  // extend in the future to contain the entire signature of the node, as a\n  // function type.\n  FullTypeDef experimental_type = 7;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/op_def.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/attr_value.proto\";\nimport \"tensorflow/core/framework/full_type.proto\";\nimport \"tensorflow/core/framework/resource_handle.proto\";\nimport \"tensorflow/core/framework/types.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"OpDefProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/op_def_go_proto\";\n\n// Defines an operation. A NodeDef in a GraphDef specifies an Op by\n// using the \"op\" field which should match the name of a OpDef.\n// LINT.IfChange\nmessage OpDef {\n  // Op names starting with an underscore are reserved for internal use.\n  // Names should be CamelCase and match the regexp \"[A-Z][a-zA-Z0-9>_]*\".\n  string name = 1;\n\n  // For describing inputs and outputs.\n  message ArgDef {\n    // Name for the input/output.  Should match the regexp \"[a-z][a-z0-9_]*\".\n    string name = 1;\n\n    // Human readable description.\n    string description = 2;\n\n    // Describes the type of one or more tensors that are accepted/produced\n    // by this input/output arg.  The only legal combinations are:\n    // * For a single tensor: either the \"type\" field is set or the\n    //   \"type_attr\" field is set to the name of an attr with type \"type\".\n    // * For a sequence of tensors with the same type: the \"number_attr\"\n    //   field will be set to the name of an attr with type \"int\", and\n    //   either the \"type\" or \"type_attr\" field will be set as for\n    //   single tensors.\n    // * For a sequence of tensors, the \"type_list_attr\" field will be set\n    //   to the name of an attr with type \"list(type)\".\n    DataType type = 3;\n    string type_attr = 4;    // if specified, attr must have type \"type\"\n    string number_attr = 5;  // if specified, attr must have type \"int\"\n    // If specified, attr must have type \"list(type)\", and none of\n    // type, type_attr, and number_attr may be specified.\n    string type_list_attr = 6;\n\n    // The handle data for resource inputs.\n    repeated ResourceHandleProto.DtypeAndShape handle_data = 7;\n\n    // For inputs: if true, the inputs are required to be refs.\n    //   By default, inputs can be either refs or non-refs.\n    // For outputs: if true, outputs are refs, otherwise they are not.\n    bool is_ref = 16;\n\n    // Experimental. Full type declaration for this argument.\n    // The full type specification combines type, type_attr, type_list_attr,\n    // etc. into a unified representation.\n    // This declaration may contain non-concrete types (for example,\n    // Tensor<TypeVar<'T'>> is a valid type declaration.\n    //\n    // Note: this is a transient field. The long-term aim is to represent the\n    // entire OpDef as a single type: a callable. In that context, this field is\n    // just the type of a single argument.\n    FullTypeDef experimental_full_type = 17;\n  }\n\n  // Description of the input(s).\n  repeated ArgDef input_arg = 2;\n\n  // Description of the output(s).\n  repeated ArgDef output_arg = 3;\n\n  // Named control outputs for this operation. Useful only for composite\n  // operations (i.e. functions) which want to name different control outputs.\n  repeated string control_output = 20;\n\n  // Description of the graph-construction-time configuration of this\n  // Op.  That is to say, this describes the attr fields that will\n  // be specified in the NodeDef.\n  message AttrDef {\n    // A descriptive name for the argument.  May be used, e.g. by the\n    // Python client, as a keyword argument name, and so should match\n    // the regexp \"[a-z][a-z0-9_]+\".\n    string name = 1;\n\n    // One of the type names from attr_value.proto (\"string\", \"list(string)\",\n    // \"int\", etc.).\n    string type = 2;\n\n    // A reasonable default for this attribute if the user does not supply\n    // a value.  If not specified, the user must supply a value.\n    AttrValue default_value = 3;\n\n    // Human-readable description.\n    string description = 4;\n\n    // TODO(josh11b): bool is_optional?\n\n    // --- Constraints ---\n    // These constraints are only in effect if specified.  Default is no\n    // constraints.\n\n    // For type == \"int\", this is a minimum value.  For \"list(___)\"\n    // types, this is the minimum length.\n    bool has_minimum = 5;\n    int64 minimum = 6;\n\n    // The set of allowed values.  Has type that is the \"list\" version\n    // of the \"type\" field above (uses the \"list\" field of AttrValue).\n    // If type == \"type\" or \"list(type)\" above, then the \"type\" field\n    // of \"allowed_values.list\" has the set of allowed DataTypes.\n    // If type == \"string\" or \"list(string)\", then the \"s\" field of\n    // \"allowed_values.list\" has the set of allowed strings.\n    AttrValue allowed_values = 7;\n  }\n  repeated AttrDef attr = 4;\n\n  // Optional deprecation based on GraphDef versions.\n  OpDeprecation deprecation = 8;\n\n  // One-line human-readable description of what the Op does.\n  string summary = 5;\n\n  // Additional, longer human-readable description of what the Op does.\n  string description = 6;\n\n  // -------------------------------------------------------------------------\n  // Which optimizations this operation can participate in.\n\n  // True if the operation is commutative (\"op(a,b) == op(b,a)\" for all inputs)\n  bool is_commutative = 18;\n\n  // If is_aggregate is true, then this operation accepts N >= 2\n  // inputs and produces 1 output all of the same type.  Should be\n  // associative and commutative, and produce output with the same\n  // shape as the input.  The optimizer may replace an aggregate op\n  // taking input from multiple devices with a tree of aggregate ops\n  // that aggregate locally within each device (and possibly within\n  // groups of nearby devices) before communicating.\n  // TODO(josh11b): Implement that optimization.\n  bool is_aggregate = 16;  // for things like add\n\n  // Other optimizations go here, like\n  //   can_alias_input, rewrite_when_output_unused, partitioning_strategy, etc.\n\n  // -------------------------------------------------------------------------\n  // Optimization constraints.\n\n  // Ops are marked as stateful if their behavior depends on some state beyond\n  // their input tensors (e.g. variable reading op) or if they have\n  // a side-effect (e.g. printing or asserting ops). Equivalently, stateless ops\n  // must always produce the same output for the same input and have\n  // no side-effects.\n  //\n  // By default Ops may be moved between devices.  Stateful ops should\n  // either not be moved, or should only be moved if that state can also\n  // be moved (e.g. via some sort of save / restore).\n  // Stateful ops are guaranteed to never be optimized away by Common\n  // Subexpression Elimination (CSE).\n  bool is_stateful = 17;  // for things like variables, queue\n\n  // -------------------------------------------------------------------------\n  // Non-standard options.\n\n  // By default, all inputs to an Op must be initialized Tensors.  Ops\n  // that may initialize tensors for the first time should set this\n  // field to true, to allow the Op to take an uninitialized Tensor as\n  // input.\n  bool allows_uninitialized_input = 19;  // for Assign, etc.\n\n  // Indicates whether the op implementation uses distributed communication.\n  // If True, the op is allowed to return errors for network disconnection and\n  // trigger TF network failure handling logics.\n  bool is_distributed_communication = 21;\n}\n// LINT.ThenChange(\n//     https://www.tensorflow.org/code/tensorflow/core/framework/op_def_util.cc)\n\n// Information about version-dependent deprecation of an op\nmessage OpDeprecation {\n  // First GraphDef version at which the op is disallowed.\n  int32 version = 1;\n\n  // Explanation of why it was deprecated and what to use instead.\n  string explanation = 2;\n}\n\n// A collection of OpDefs\nmessage OpList {\n  repeated OpDef op = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/reader_base.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"ReaderBaseProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/reader_base_go_proto\";\n\n// For serializing and restoring the state of ReaderBase, see\n// reader_base.h for details.\nmessage ReaderBaseState {\n  int64 work_started = 1;\n  int64 work_finished = 2;\n  int64 num_records_produced = 3;\n  bytes current_work = 4;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/resource_handle.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/types.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"ResourceHandle\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/resource_handle_go_proto\";\n\n// Protocol buffer representing a handle to a tensorflow resource. Handles are\n// not valid across executions, but can be serialized back and forth from within\n// a single run.\nmessage ResourceHandleProto {\n  // Unique name for the device containing the resource.\n  string device = 1;\n\n  // Container in which this resource is placed.\n  string container = 2;\n\n  // Unique name of this resource.\n  string name = 3;\n\n  // Hash code for the type of the resource. Is only valid in the same device\n  // and in the same execution.\n  uint64 hash_code = 4;\n\n  // For debug-only, the name of the type pointed to by this handle, if\n  // available.\n  string maybe_type_name = 5;\n\n  // Protocol buffer representing a pair of (data type, tensor shape).\n  message DtypeAndShape {\n    DataType dtype = 1;\n    TensorShapeProto shape = 2;\n  }\n\n  // Data types and shapes for the underlying resource.\n  repeated DtypeAndShape dtypes_and_shapes = 6;\n\n  reserved 7;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/step_stats.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/allocation_description.proto\";\nimport \"tensorflow/core/framework/tensor_description.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"StepStatsProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/step_stats_go_proto\";\n\n// An allocation/de-allocation operation performed by the allocator.\nmessage AllocationRecord {\n  // The timestamp of the operation.\n  int64 alloc_micros = 1;\n  // Number of bytes allocated, or de-allocated if negative.\n  int64 alloc_bytes = 2;\n}\n\nmessage AllocatorMemoryUsed {\n  string allocator_name = 1;\n  // These are per-node allocator memory stats.\n  int64 total_bytes = 2;\n  int64 peak_bytes = 3;\n  // The bytes that are not deallocated.\n  int64 live_bytes = 4;\n  // The allocation and deallocation timeline.\n  repeated AllocationRecord allocation_records = 6;\n\n  // These are snapshots of the overall allocator memory stats.\n  // The number of live bytes currently allocated by the allocator.\n  int64 allocator_bytes_in_use = 5;\n}\n\n// Output sizes recorded for a single execution of a graph node.\nmessage NodeOutput {\n  int32 slot = 1;\n  TensorDescription tensor_description = 3;\n}\n\n// For memory tracking.\nmessage MemoryStats {\n  int64 temp_memory_size = 1;\n  int64 persistent_memory_size = 3;\n  repeated int64 persistent_tensor_alloc_ids = 5;\n\n  int64 device_temp_memory_size = 2 [deprecated = true];\n  int64 device_persistent_memory_size = 4 [deprecated = true];\n  repeated int64 device_persistent_tensor_alloc_ids = 6 [deprecated = true];\n}\n\n// Time/size stats recorded for a single execution of a graph node.\nmessage NodeExecStats {\n  // TODO(tucker): Use some more compact form of node identity than\n  // the full string name.  Either all processes should agree on a\n  // global id (cost_id?) for each node, or we should use a hash of\n  // the name.\n  string node_name = 1;\n  int64 all_start_micros = 2;\n  int64 op_start_rel_micros = 3;\n  int64 op_end_rel_micros = 4;\n  int64 all_end_rel_micros = 5;\n  repeated AllocatorMemoryUsed memory = 6;\n  repeated NodeOutput output = 7;\n  string timeline_label = 8;\n  int64 scheduled_micros = 9;\n  uint32 thread_id = 10;\n  repeated AllocationDescription referenced_tensor = 11;\n  MemoryStats memory_stats = 12;\n  int64 all_start_nanos = 13;\n  int64 op_start_rel_nanos = 14;\n  int64 op_end_rel_nanos = 15;\n  int64 all_end_rel_nanos = 16;\n  int64 scheduled_nanos = 17;\n}\n\nmessage DeviceStepStats {\n  string device = 1;\n  repeated NodeExecStats node_stats = 2;\n  // Its key is thread id.\n  map<uint32, string> thread_names = 3;\n}\n\nmessage StepStats {\n  repeated DeviceStepStats dev_stats = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/summary.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/tensor.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"SummaryProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/summary_go_proto\";\n\n// Metadata associated with a series of Summary data\nmessage SummaryDescription {\n  // Hint on how plugins should process the data in this series.\n  // Supported values include \"scalar\", \"histogram\", \"image\", \"audio\"\n  string type_hint = 1;\n}\n\n// Serialization format for histogram module in\n// core/lib/histogram/histogram.h\nmessage HistogramProto {\n  double min = 1;\n  double max = 2;\n  double num = 3;\n  double sum = 4;\n  double sum_squares = 5;\n\n  // Parallel arrays encoding the bucket boundaries and the bucket values.\n  // bucket(i) is the count for the bucket i.  The range for\n  // a bucket is:\n  //   i == 0:  -DBL_MAX .. bucket_limit(0)\n  //   i != 0:  bucket_limit(i-1) .. bucket_limit(i)\n  repeated double bucket_limit = 6 [packed = true];\n  repeated double bucket = 7 [packed = true];\n}\n\n// A SummaryMetadata encapsulates information on which plugins are able to make\n// use of a certain summary value.\nmessage SummaryMetadata {\n  message PluginData {\n    // The name of the plugin this data pertains to.\n    string plugin_name = 1;\n\n    // The content to store for the plugin. The best practice is for this to be\n    // a binary serialized protocol buffer.\n    bytes content = 2;\n  }\n\n  // Data that associates a summary with a certain plugin.\n  PluginData plugin_data = 1;\n\n  // Display name for viewing in TensorBoard.\n  string display_name = 2;\n\n  // Longform readable description of the summary sequence. Markdown supported.\n  string summary_description = 3;\n\n  // Class of data stored in this time series. Required for compatibility with\n  // TensorBoard's generic data facilities (`DataProvider`, et al.). This value\n  // imposes constraints on the dtype and shape of the corresponding tensor\n  // values. See `DataClass` docs for details.\n  DataClass data_class = 4;\n}\n\nenum DataClass {\n  // Unknown data class, used (implicitly) for legacy data. Will not be\n  // processed by data ingestion pipelines.\n  DATA_CLASS_UNKNOWN = 0;\n  // Scalar time series. Each `Value` for the corresponding tag must have\n  // `tensor` set to a rank-0 tensor of type `DT_FLOAT` (float32).\n  DATA_CLASS_SCALAR = 1;\n  // Tensor time series. Each `Value` for the corresponding tag must have\n  // `tensor` set. The tensor value is arbitrary, but should be small to\n  // accommodate direct storage in database backends: an upper bound of a few\n  // kilobytes is a reasonable rule of thumb.\n  DATA_CLASS_TENSOR = 2;\n  // Blob sequence time series. Each `Value` for the corresponding tag must\n  // have `tensor` set to a rank-1 tensor of bytestring dtype.\n  DATA_CLASS_BLOB_SEQUENCE = 3;\n}\n\n// A Summary is a set of named values to be displayed by the\n// visualizer.\n//\n// Summaries are produced regularly during training, as controlled by\n// the \"summary_interval_secs\" attribute of the training operation.\n// Summaries are also produced at the end of an evaluation.\nmessage Summary {\n  message Image {\n    // Dimensions of the image.\n    int32 height = 1;\n    int32 width = 2;\n    // Valid colorspace values are\n    //   1 - grayscale\n    //   2 - grayscale + alpha\n    //   3 - RGB\n    //   4 - RGBA\n    //   5 - DIGITAL_YUV\n    //   6 - BGRA\n    int32 colorspace = 3;\n    // Image data in encoded format.  All image formats supported by\n    // image_codec::CoderUtil can be stored here.\n    bytes encoded_image_string = 4;\n  }\n\n  message Audio {\n    // Sample rate of the audio in Hz.\n    float sample_rate = 1;\n    // Number of channels of audio.\n    int64 num_channels = 2;\n    // Length of the audio in frames (samples per channel).\n    int64 length_frames = 3;\n    // Encoded audio data and its associated RFC 2045 content type (e.g.\n    // \"audio/wav\").\n    bytes encoded_audio_string = 4;\n    string content_type = 5;\n  }\n\n  message Value {\n    // This field is deprecated and will not be set.\n    string node_name = 7;\n\n    // Tag name for the data. Used by TensorBoard plugins to organize data. Tags\n    // are often organized by scope (which contains slashes to convey\n    // hierarchy). For example: foo/bar/0\n    string tag = 1;\n\n    // Contains metadata on the summary value such as which plugins may use it.\n    // Take note that many summary values may lack a metadata field. This is\n    // because the FileWriter only keeps a metadata object on the first summary\n    // value with a certain tag for each tag. TensorBoard then remembers which\n    // tags are associated with which plugins. This saves space.\n    SummaryMetadata metadata = 9;\n\n    // Value associated with the tag.\n    oneof value {\n      float simple_value = 2;\n      bytes obsolete_old_style_histogram = 3;\n      Image image = 4;\n      HistogramProto histo = 5;\n      Audio audio = 6;\n      TensorProto tensor = 8;\n    }\n  }\n\n  // Set of values for the summary.\n  repeated Value value = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/tensor.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/resource_handle.proto\";\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/types.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"TensorProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/tensor_go_proto\";\n\n// Protocol buffer representing a tensor.\nmessage TensorProto {\n  DataType dtype = 1;\n\n  // Shape of the tensor.  TODO(touts): sort out the 0-rank issues.\n  TensorShapeProto tensor_shape = 2;\n\n  // Only one of the representations below is set, one of \"tensor_contents\" and\n  // the \"xxx_val\" attributes.  We are not using oneof because as oneofs cannot\n  // contain repeated fields it would require another extra set of messages.\n\n  // Version number.\n  //\n  // In version 0, if the \"repeated xxx\" representations contain only one\n  // element, that element is repeated to fill the shape.  This makes it easy\n  // to represent a constant Tensor with a single value.\n  int32 version_number = 3;\n\n  // Serialized raw tensor content from either Tensor::AsProtoTensorContent or\n  // memcpy in tensorflow::grpc::EncodeTensorToByteBuffer. This representation\n  // can be used for all tensor types. The purpose of this representation is to\n  // reduce serialization overhead during RPC call by avoiding serialization of\n  // many repeated small items.\n  bytes tensor_content = 4;\n\n  // Type specific representations that make it easy to create tensor protos in\n  // all languages.  Only the representation corresponding to \"dtype\" can\n  // be set.  The values hold the flattened representation of the tensor in\n  // row major order.\n\n  // DT_HALF, DT_BFLOAT16. Note that since protobuf has no int16 type, we'll\n  // have some pointless zero padding for each value here.\n  repeated int32 half_val = 13 [packed = true];\n\n  // DT_FLOAT.\n  repeated float float_val = 5 [packed = true];\n\n  // DT_DOUBLE.\n  repeated double double_val = 6 [packed = true];\n\n  // DT_INT32, DT_INT16, DT_UINT16, DT_INT8, DT_UINT8.\n  repeated int32 int_val = 7 [packed = true];\n\n  // DT_STRING\n  repeated bytes string_val = 8;\n\n  // DT_COMPLEX64. scomplex_val(2*i) and scomplex_val(2*i+1) are real\n  // and imaginary parts of i-th single precision complex.\n  repeated float scomplex_val = 9 [packed = true];\n\n  // DT_INT64\n  repeated int64 int64_val = 10 [packed = true];\n\n  // DT_BOOL\n  repeated bool bool_val = 11 [packed = true];\n\n  // DT_COMPLEX128. dcomplex_val(2*i) and dcomplex_val(2*i+1) are real\n  // and imaginary parts of i-th double precision complex.\n  repeated double dcomplex_val = 12 [packed = true];\n\n  // DT_RESOURCE\n  repeated ResourceHandleProto resource_handle_val = 14;\n\n  // DT_VARIANT\n  repeated VariantTensorDataProto variant_val = 15;\n\n  // DT_UINT32\n  repeated uint32 uint32_val = 16 [packed = true];\n\n  // DT_UINT64\n  repeated uint64 uint64_val = 17 [packed = true];\n}\n\n// Protocol buffer representing the serialization format of DT_VARIANT tensors.\nmessage VariantTensorDataProto {\n  // Name of the type of objects being serialized.\n  string type_name = 1;\n  // Portions of the object that are not Tensors.\n  bytes metadata = 2;\n  // Tensors contained within objects being serialized.\n  repeated TensorProto tensors = 3;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/tensor_description.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/allocation_description.proto\";\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/types.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"TensorDescriptionProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/tensor_description_go_proto\";\n\nmessage TensorDescription {\n  // Data type of tensor elements\n  DataType dtype = 1;\n\n  // Shape of the tensor.\n  TensorShapeProto shape = 2;\n\n  // Information about the size and allocator used for the data\n  AllocationDescription allocation_description = 4;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/tensor_shape.proto",
    "content": "// Protocol buffer representing the shape of tensors.\n\nsyntax = \"proto3\";\noption cc_enable_arenas = true;\noption java_outer_classname = \"TensorShapeProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/tensor_shape_go_proto\";\n\npackage tensorflow;\n\n// Dimensions of a tensor.\nmessage TensorShapeProto {\n  // One dimension of the tensor.\n  message Dim {\n    // Size of the tensor in that dimension.\n    // This value must be >= -1, but values of -1 are reserved for \"unknown\"\n    // shapes (values of -1 mean \"unknown\" dimension).  Certain wrappers\n    // that work with TensorShapeProto may fail at runtime when deserializing\n    // a TensorShapeProto containing a dim value of -1.\n    int64 size = 1;\n\n    // Optional name of the tensor dimension.\n    string name = 2;\n  };\n\n  // Dimensions of the tensor, such as {\"input\", 30}, {\"output\", 40}\n  // for a 30 x 40 2D tensor.  If an entry has size -1, this\n  // corresponds to a dimension of unknown size. The names are\n  // optional.\n  //\n  // The order of entries in \"dim\" matters: It indicates the layout of the\n  // values in the tensor in-memory representation.\n  //\n  // The first entry in \"dim\" is the outermost dimension used to layout the\n  // values, the last entry is the innermost dimension.  This matches the\n  // in-memory layout of RowMajor Eigen tensors.\n  //\n  // If \"dim.size()\" > 0, \"unknown_rank\" must be false.\n  repeated Dim dim = 2;\n\n  // If true, the number of dimensions in the shape is unknown.\n  //\n  // If true, \"dim.size()\" must be 0.\n  bool unknown_rank = 3;\n};\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/tensor_slice.proto",
    "content": "// Protocol buffer representing slices of a tensor\n\nsyntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"TensorSliceProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/tensor_slice_go_proto\";\n\n// Can only be interpreted if you know the corresponding TensorShape.\nmessage TensorSliceProto {\n  // Extent of the slice in one dimension.\n  message Extent {\n    // Either both or no attributes must be set.  When no attribute is set\n    // means: All data in that dimension.\n\n    // Start index of the slice, starting at 0.\n    int64 start = 1;\n\n    // Length of the slice: if the length is missing or -1 we will\n    // interpret this as \"everything in this dimension\".  We use\n    // \"oneof\" to preserve information about whether the length is\n    // present without changing the serialization format from the\n    // prior proto2 version of this proto.\n    oneof has_length {\n      int64 length = 2;\n    }\n  }\n\n  // Extent of the slice in all tensor dimensions.\n  //\n  // Must have one entry for each of the dimension of the tensor that this\n  // slice belongs to.  The order of sizes is the same as the order of\n  // dimensions in the TensorShape.\n  repeated Extent extent = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/types.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"TypesProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/types_go_proto\";\n\n// (== suppress_warning documentation-presence ==)\n// LINT.IfChange\nenum DataType {\n  // Not a legal value for DataType.  Used to indicate a DataType field\n  // has not been set.\n  DT_INVALID = 0;\n\n  // Data types that all computation devices are expected to be\n  // capable to support.\n  DT_FLOAT = 1;\n  DT_DOUBLE = 2;\n  DT_INT32 = 3;\n  DT_UINT8 = 4;\n  DT_INT16 = 5;\n  DT_INT8 = 6;\n  DT_STRING = 7;\n  DT_COMPLEX64 = 8;  // Single-precision complex\n  DT_INT64 = 9;\n  DT_BOOL = 10;\n  DT_QINT8 = 11;     // Quantized int8\n  DT_QUINT8 = 12;    // Quantized uint8\n  DT_QINT32 = 13;    // Quantized int32\n  DT_BFLOAT16 = 14;  // Float32 truncated to 16 bits.  Only for cast ops.\n  DT_QINT16 = 15;    // Quantized int16\n  DT_QUINT16 = 16;   // Quantized uint16\n  DT_UINT16 = 17;\n  DT_COMPLEX128 = 18;  // Double-precision complex\n  DT_HALF = 19;\n  DT_RESOURCE = 20;\n  DT_VARIANT = 21;  // Arbitrary C++ data types\n  DT_UINT32 = 22;\n  DT_UINT64 = 23;\n\n  // Do not use!  These are only for parameters.  Every enum above\n  // should have a corresponding value below (verified by types_test).\n  DT_FLOAT_REF = 101;\n  DT_DOUBLE_REF = 102;\n  DT_INT32_REF = 103;\n  DT_UINT8_REF = 104;\n  DT_INT16_REF = 105;\n  DT_INT8_REF = 106;\n  DT_STRING_REF = 107;\n  DT_COMPLEX64_REF = 108;\n  DT_INT64_REF = 109;\n  DT_BOOL_REF = 110;\n  DT_QINT8_REF = 111;\n  DT_QUINT8_REF = 112;\n  DT_QINT32_REF = 113;\n  DT_BFLOAT16_REF = 114;\n  DT_QINT16_REF = 115;\n  DT_QUINT16_REF = 116;\n  DT_UINT16_REF = 117;\n  DT_COMPLEX128_REF = 118;\n  DT_HALF_REF = 119;\n  DT_RESOURCE_REF = 120;\n  DT_VARIANT_REF = 121;\n  DT_UINT32_REF = 122;\n  DT_UINT64_REF = 123;\n}\n// LINT.ThenChange(\n//    https://www.tensorflow.org/code/tensorflow/c/tf_datatype.h,\n//    https://www.tensorflow.org/code/tensorflow/go/tensor.go,\n//    https://www.tensorflow.org/code/tensorflow/core/framework/tensor.cc,\n//    https://www.tensorflow.org/code/tensorflow/core/framework/types.h,\n//    https://www.tensorflow.org/code/tensorflow/core/framework/types.cc,\n//    https://www.tensorflow.org/code/tensorflow/python/framework/dtypes.py,\n//    https://www.tensorflow.org/code/tensorflow/python/framework/function.py)\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/variable.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"VariableProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/variable_go_proto\";\n\n// Indicates when a distributed variable will be synced.\nenum VariableSynchronization {\n  // `AUTO`: Indicates that the synchronization will be determined by the\n  // current `DistributionStrategy` (eg. With `MirroredStrategy` this would be\n  // `ON_WRITE`).\n  VARIABLE_SYNCHRONIZATION_AUTO = 0;\n  // `NONE`: Indicates that there will only be one copy of the variable, so\n  // there is no need to sync.\n  VARIABLE_SYNCHRONIZATION_NONE = 1;\n  // `ON_WRITE`: Indicates that the variable will be updated across devices\n  // every time it is written.\n  VARIABLE_SYNCHRONIZATION_ON_WRITE = 2;\n  // `ON_READ`: Indicates that the variable will be aggregated across devices\n  // when it is read (eg. when checkpointing or when evaluating an op that uses\n  // the variable).\n  VARIABLE_SYNCHRONIZATION_ON_READ = 3;\n}\n\n// Indicates how a distributed variable will be aggregated.\nenum VariableAggregation {\n  // `NONE`: This is the default, giving an error if you use a\n  // variable-update operation with multiple replicas.\n  VARIABLE_AGGREGATION_NONE = 0;\n  // `SUM`: Add the updates across replicas.\n  VARIABLE_AGGREGATION_SUM = 1;\n  // `MEAN`: Take the arithmetic mean (\"average\") of the updates across\n  // replicas.\n  VARIABLE_AGGREGATION_MEAN = 2;\n  // `ONLY_FIRST_REPLICA`: This is for when every replica is performing the same\n  // update, but we only want to perform the update once. Used, e.g., for the\n  // global step counter.\n  VARIABLE_AGGREGATION_ONLY_FIRST_REPLICA = 3;\n}\n\n// Protocol buffer representing a Variable.\nmessage VariableDef {\n  // Name of the variable tensor.\n  string variable_name = 1;\n\n  // Name of the tensor holding the variable's initial value.\n  string initial_value_name = 6;\n\n  // Name of the initializer op.\n  string initializer_name = 2;\n\n  // Name of the snapshot tensor.\n  string snapshot_name = 3;\n\n  // Support for saving variables as slices of a larger variable.\n  SaveSliceInfoDef save_slice_info_def = 4;\n\n  // Whether to represent this as a ResourceVariable.\n  bool is_resource = 5;\n\n  // Whether this variable should be trained.\n  bool trainable = 7;\n\n  // Indicates when a distributed variable will be synced.\n  VariableSynchronization synchronization = 8;\n\n  // Indicates how a distributed variable will be aggregated.\n  VariableAggregation aggregation = 9;\n}\n\nmessage SaveSliceInfoDef {\n  // Name of the full variable of which this is a slice.\n  string full_name = 1;\n  // Shape of the full variable.\n  repeated int64 full_shape = 2;\n  // Offset of this variable into the full variable.\n  repeated int64 var_offset = 3;\n  // Shape of this variable.\n  repeated int64 var_shape = 4;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/framework/versions.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"VersionsProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/framework/versions_go_proto\";\n\n// Version information for a piece of serialized data\n//\n// There are different types of versions for each type of data\n// (GraphDef, etc.), but they all have the same common shape\n// described here.\n//\n// Each consumer has \"consumer\" and \"min_producer\" versions (specified\n// elsewhere).  A consumer is allowed to consume this data if\n//\n//   producer >= min_producer\n//   consumer >= min_consumer\n//   consumer not in bad_consumers\n//\nmessage VersionDef {\n  // The version of the code that produced this data.\n  int32 producer = 1;\n\n  // Any consumer below this version is not allowed to consume this data.\n  int32 min_consumer = 2;\n\n  // Specific consumer versions which are disallowed (e.g. due to bugs).\n  repeated int32 bad_consumers = 3;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/autotuning.proto",
    "content": "// This file defines protos that store the results of autotuning various\n// operations.\n//\n// They are in proto format because we want to log them structured. They offer\n// tremendous statistical, testing, and debugging value.\nsyntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"google/protobuf/any.proto\";\nimport \"google/protobuf/duration.proto\";\nimport \"tensorflow/stream_executor/dnn.proto\";\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\nmessage CudnnVersion {\n  int32 major = 1;\n  int32 minor = 2;\n  int32 patch = 3;\n}\n\nmessage ComputeCapability {\n  int32 major = 1;\n  int32 minor = 2;\n}\n\nmessage AutotuneResult {\n  enum FailureKind {\n    UNKNOWN = 0;\n\n    // Algorithm wrote memory outside its output buffers.\n    REDZONE_MODIFIED = 1;\n\n    // Algorithm gave a different result from a reference algorithm.\n    WRONG_RESULT = 2;\n\n    // Algorithm was rejected for failing to run or for known bugs.\n    DISQUALIFIED = 3;\n  }\n\n  message FailureResult {\n    FailureKind kind = 1;\n    string msg = 2;\n\n    // For failure_kind == WRONG_RESULT, this field indicates the reference\n    // configuration that we compared against.\n    //\n    // Note that the reference algorithm isn't always correct.  However,\n    // empirically it's more correct, as it's \"algo 0\", less fancy than the\n    // compared one.\n    oneof key {\n      ConvKey reference_conv = 11;\n      GemmKey reference_gemm = 12;\n      CudaConvPlanKey reference_cuda_conv_plan = 14;\n      stream_executor.dnn.AlgorithmProto reference_algorithm = 15;\n    }\n\n    int64 buffer_address = 13;\n  }\n\n  // Legacy and unused in new data; superseded by AlgorithmProto.\n  message ConvKey {\n    int64 algorithm = 1;\n    bool tensor_ops_enabled = 2;\n  }\n\n  message GemmKey {\n    int64 algorithm = 1;\n  }\n\n  // Legacy and unused in new data; superseded by AlgorithmProto.\n  message CudaConvPlanKey {\n    string exec_plan_id = 1;\n  }\n\n  int64 scratch_bytes = 8;\n  google.protobuf.Duration run_time = 9;\n\n  FailureResult failure = 7;\n\n  oneof key {\n    ConvKey conv = 5;\n    GemmKey gemm = 6;\n    CudaConvPlanKey cuda_conv_plan = 15;\n    stream_executor.dnn.AlgorithmProto algorithm = 16;\n  }\n\n  // Next ID: 17\n}\n\nmessage AutotuningLog {\n  google.protobuf.Any instr = 1;\n\n  // Records all auto-tuning results per algorithm.\n  repeated AutotuneResult results = 2;\n\n  CudnnVersion cudnn_version = 3;\n  ComputeCapability compute_capability = 4;\n\n  // stream_executor::DeviceDescription::pci_bus_id.\n  string device_pci_bus_id = 5;\n\n  string blas_version = 6;\n\n  // Next ID: 7\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/bfc_memory_map.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Some of the data from AllocatorStats\nmessage MemAllocatorStats {\n  int64 num_allocs = 1;\n  int64 bytes_in_use = 2;\n  int64 peak_bytes_in_use = 3;\n  int64 largest_alloc_size = 4;\n  float fragmentation_metric = 5;\n}\n\nmessage MemChunk {\n  uint64 address = 1;\n  int64 size = 2;\n  int64 requested_size = 3;\n  int32 bin = 4;\n  string op_name = 5;\n  uint64 freed_at_count = 6;\n  uint64 action_count = 7;\n  bool in_use = 8;\n  uint64 step_id = 9;\n}\n\nmessage BinSummary {\n  int32 bin = 1;\n  int64 total_bytes_in_use = 2;\n  int64 total_bytes_in_bin = 3;\n  int64 total_chunks_in_use = 4;\n  int64 total_chunks_in_bin = 5;\n}\n\nmessage SnapShot {\n  uint64 action_count = 1;\n  int64 size = 2;\n}\n\nmessage MemoryDump {\n  string allocator_name = 1;\n  repeated BinSummary bin_summary = 2;\n  repeated MemChunk chunk = 3;\n  repeated SnapShot snap_shot = 4;\n  MemAllocatorStats stats = 5;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/cluster.proto",
    "content": "/* Copyright 2016 The TensorFlow Authors. All Rights Reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n==============================================================================*/\n\nsyntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"ClusterProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.distruntime\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// This file contains protos to be used when defining a TensorFlow\n// cluster.\n//\n// EXAMPLES\n// --------\n//\n// 1. A single-process cluster, containing \"/job:local/task:0\".\n//\n//    Cluster:\n//      job { name: 'local' tasks { key: 0 value: 'localhost:2222' } }\n//\n//    Server:\n//      cluster { $CLUSTER } job_name: 'local' task_index: 0\n//\n// 2. A two-process cluster, containing \"/job:local/task:{0,1}\".\n//\n//    Cluster:\n//      job { name: 'local' tasks { key: 0 value: 'localhost:2222' }\n//                          tasks { key: 1 value: 'localhost:2223' } }\n//\n//    Servers:\n//      cluster { $CLUSTER } job_name: 'local' task_index: 0\n//      cluster { $CLUSTER } job_name: 'local' task_index: 1\n//\n// 3. A two-job cluster, containing \"/job:worker/task:{0,1,2}\" and\n//    \"/job:ps/task:{0,1}\".\n//\n//    Cluster:\n//      job { name: 'worker' tasks { key: 0 value: 'worker1:2222' }\n//                           tasks { key: 1 value: 'worker2:2222' }\n//                           tasks { key: 2 value: 'worker3:2222' } }\n//      job { name: 'ps'     tasks { key: 0 value: 'ps0:2222' }\n//                           tasks { key: 1 value: 'ps1:2222' } }\n//\n//    Servers:\n//      cluster { $CLUSTER } job_name: 'worker' task_index: 0\n//      cluster { $CLUSTER } job_name: 'worker' task_index: 1\n//      cluster { $CLUSTER } job_name: 'worker' task_index: 2\n//      cluster { $CLUSTER } job_name: 'ps'     task_index: 0\n//      cluster { $CLUSTER } job_name: 'ps'     task_index: 1\n\n// Defines a single job in a TensorFlow cluster.\nmessage JobDef {\n  // The name of this job.\n  string name = 1;\n\n  // Mapping from task ID to \"hostname:port\" string.\n  //\n  // If the `name` field contains \"worker\", and the `tasks` map contains a\n  // mapping from 7 to \"example.org:2222\", then the device prefix\n  // \"/job:worker/task:7\" will be assigned to \"example.org:2222\".\n  map<int32, string> tasks = 2;\n}\n\n// Defines a TensorFlow cluster as a set of jobs.\nmessage ClusterDef {\n  // The jobs that comprise the cluster.\n  repeated JobDef job = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/composite_tensor_variant.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/protobuf/struct.proto\";\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Metadata for CompositeTensorVariant, used when serializing as Variant.\n//\n// We define a new message here (rather than directly using TypeSpecProto for\n// the metadata string) to retain flexibility to change the metadata encoding\n// to support additional features.\nmessage CompositeTensorVariantMetadata {\n  TypeSpecProto type_spec_proto = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/config.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/cost_graph.proto\";\nimport \"tensorflow/core/framework/graph.proto\";\nimport \"tensorflow/core/framework/step_stats.proto\";\nimport \"tensorflow/core/protobuf/cluster.proto\";\nimport \"tensorflow/core/protobuf/coordination_config.proto\";\nimport \"tensorflow/core/protobuf/debug.proto\";\nimport \"tensorflow/core/protobuf/rewriter_config.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"ConfigProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\nmessage GPUOptions {\n  // Fraction of the available GPU memory to allocate for each process.\n  // 1 means to allocate all of the GPU memory, 0.5 means the process\n  // allocates up to ~50% of the available GPU memory.\n  //\n  // GPU memory is pre-allocated unless the allow_growth option is enabled.\n  //\n  // If greater than 1.0, uses CUDA unified memory to potentially oversubscribe\n  // the amount of memory available on the GPU device by using host memory as a\n  // swap space. Accessing memory not available on the device will be\n  // significantly slower as that would require memory transfer between the host\n  // and the device. Options to reduce the memory requirement should be\n  // considered before enabling this option as this may come with a negative\n  // performance impact. Oversubscription using the unified memory requires\n  // Pascal class or newer GPUs and it is currently only supported on the Linux\n  // operating system. See\n  // https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#um-requirements\n  // for the detailed requirements.\n  double per_process_gpu_memory_fraction = 1;\n\n  // If true, the allocator does not pre-allocate the entire specified\n  // GPU memory region, instead starting small and growing as needed.\n  bool allow_growth = 4;\n\n  // The type of GPU allocation strategy to use.\n  //\n  // Allowed values:\n  // \"\": The empty string (default) uses a system-chosen default\n  //     which may change over time.\n  //\n  // \"BFC\": A \"Best-fit with coalescing\" algorithm, simplified from a\n  //        version of dlmalloc.\n  string allocator_type = 2;\n\n  // Delay deletion of up to this many bytes to reduce the number of\n  // interactions with gpu driver code.  If 0, the system chooses\n  // a reasonable default (several MBs).\n  int64 deferred_deletion_bytes = 3;\n\n  // A comma-separated list of GPU ids that determines the 'visible'\n  // to 'virtual' mapping of GPU devices.  For example, if TensorFlow\n  // can see 8 GPU devices in the process, and one wanted to map\n  // visible GPU devices 5 and 3 as \"/device:GPU:0\", and \"/device:GPU:1\",\n  // then one would specify this field as \"5,3\".  This field is similar in\n  // spirit to the CUDA_VISIBLE_DEVICES environment variable, except\n  // it applies to the visible GPU devices in the process.\n  //\n  // NOTE:\n  // 1. The GPU driver provides the process with the visible GPUs\n  //    in an order which is not guaranteed to have any correlation to\n  //    the *physical* GPU id in the machine.  This field is used for\n  //    remapping \"visible\" to \"virtual\", which means this operates only\n  //    after the process starts.  Users are required to use vendor\n  //    specific mechanisms (e.g., CUDA_VISIBLE_DEVICES) to control the\n  //    physical to visible device mapping prior to invoking TensorFlow.\n  // 2. In the code, the ids in this list are also called \"platform GPU id\"s,\n  //    and the 'virtual' ids of GPU devices (i.e. the ids in the device\n  //    name \"/device:GPU:<id>\") are also called \"TF GPU id\"s. Please\n  //    refer to third_party/tensorflow/core/common_runtime/gpu/gpu_id.h\n  //    for more information.\n  string visible_device_list = 5;\n\n  // In the event polling loop sleep this many microseconds between\n  // PollEvents calls, when the queue is not empty.  If value is not\n  // set or set to 0, gets set to a non-zero default.\n  int32 polling_active_delay_usecs = 6;\n\n  // This field is deprecated and ignored.\n  int32 polling_inactive_delay_msecs = 7;\n\n  // Force all tensors to be gpu_compatible. On a GPU-enabled TensorFlow,\n  // enabling this option forces all CPU tensors to be allocated with Cuda\n  // pinned memory. Normally, TensorFlow will infer which tensors should be\n  // allocated as the pinned memory. But in case where the inference is\n  // incomplete, this option can significantly speed up the cross-device memory\n  // copy performance as long as it fits the memory.\n  // Note that this option is not something that should be\n  // enabled by default for unknown or very large models, since all Cuda pinned\n  // memory is unpageable, having too much pinned memory might negatively impact\n  // the overall host system performance.\n  bool force_gpu_compatible = 8;\n\n  message Experimental {\n    // Configuration for breaking down a visible GPU into multiple \"virtual\"\n    // devices.\n    message VirtualDevices {\n      // Per \"virtual\" device memory limit, in MB. The number of elements in\n      // the list is the number of virtual devices to create on the\n      // corresponding visible GPU (see \"virtual_devices\" below).\n      // If empty, it will create single virtual device taking all available\n      // memory from the device.\n      //\n      // For the concept of \"visible\" and \"virtual\" GPU, see the comments for\n      // \"visible_device_list\" above for more information.\n      repeated float memory_limit_mb = 1;\n\n      // Priority values to use with the virtual devices. Use the cuda function\n      // cudaDeviceGetStreamPriorityRange to query for valid range of values for\n      // priority.\n      //\n      // On a P4000 GPU with cuda 10.1, the priority range reported was 0 for\n      // least priority and -1 for greatest priority.\n      //\n      // If this field is not specified, then the virtual devices will be\n      // created with the default. If this field has values set, then the size\n      // of this must match with the above memory_limit_mb.\n      repeated int32 priority = 2;\n    }\n\n    // The multi virtual device settings. If empty (not set), it will create\n    // single virtual device on each visible GPU, according to the settings\n    // in \"visible_device_list\" above. Otherwise, the number of elements in the\n    // list must be the same as the number of visible GPUs (after\n    // \"visible_device_list\" filtering if it is set), and the string represented\n    // device names (e.g. /device:GPU:<id>) will refer to the virtual\n    // devices and have the <id> field assigned sequentially starting from 0,\n    // according to the order they appear in this list and the \"memory_limit\"\n    // list inside each element. For example,\n    //   visible_device_list = \"1,0\"\n    //   virtual_devices { memory_limit: 1GB memory_limit: 2GB }\n    //   virtual_devices {}\n    // will create three virtual devices as:\n    //   /device:GPU:0 -> visible GPU 1 with 1GB memory\n    //   /device:GPU:1 -> visible GPU 1 with 2GB memory\n    //   /device:GPU:2 -> visible GPU 0 with all available memory\n    //\n    // NOTE:\n    // 1. It's invalid to set both this and \"per_process_gpu_memory_fraction\"\n    //    at the same time.\n    // 2. Currently this setting is per-process, not per-session. Using\n    //    different settings in different sessions within same process will\n    //    result in undefined behavior.\n    repeated VirtualDevices virtual_devices = 1;\n\n    // If true, uses CUDA unified memory for memory allocations. If\n    // per_process_gpu_memory_fraction option is greater than 1.0, then unified\n    // memory is used regardless of the value for this field. See comments for\n    // per_process_gpu_memory_fraction field for more details and requirements\n    // of the unified memory. This option is useful to oversubscribe memory if\n    // multiple processes are sharing a single GPU while individually using less\n    // than 1.0 per process memory fraction.\n    bool use_unified_memory = 2;\n\n    // If > 1, the number of device-to-device copy streams to create\n    // for each GPUDevice.  Default value is 0, which is automatically\n    // converted to 1.\n    int32 num_dev_to_dev_copy_streams = 3;\n\n    // If non-empty, defines a good GPU ring order on a single worker based on\n    // device interconnect.  This assumes that all workers have the same GPU\n    // topology.  Specify as a comma-separated string, e.g. \"3,2,1,0,7,6,5,4\".\n    // This ring order is used by the RingReducer implementation of\n    // CollectiveReduce, and serves as an override to automatic ring order\n    // generation in OrderTaskDeviceMap() during CollectiveParam resolution.\n    string collective_ring_order = 4;\n\n    // If true then extra work is done by GPUDevice and GPUBFCAllocator to\n    // keep track of when GPU memory is freed and when kernels actually\n    // complete so that we can know when a nominally free memory chunk\n    // is really not subject to pending use.\n    bool timestamped_allocator = 5;\n\n    // reserved id: 6\n\n    // Parameters for GPUKernelTracker.  By default no kernel tracking is done.\n    // Note that timestamped_allocator is only effective if some tracking is\n    // specified.\n    //\n    // If kernel_tracker_max_interval = n > 0, then a tracking event\n    // is inserted after every n kernels without an event.\n    int32 kernel_tracker_max_interval = 7;\n    // If kernel_tracker_max_bytes = n > 0, then a tracking event is\n    // inserted after every series of kernels allocating a sum of\n    // memory >= n.  If one kernel allocates b * n bytes, then one\n    // event will be inserted after it, but it will count as b against\n    // the pending limit.\n    int32 kernel_tracker_max_bytes = 8;\n    // If kernel_tracker_max_pending > 0 then no more than this many\n    // tracking events can be outstanding at a time.  An attempt to\n    // launch an additional kernel will stall until an event\n    // completes.\n    int32 kernel_tracker_max_pending = 9;\n\n    // BFC Allocator can return an allocated chunk of memory upto 2x the\n    // requested size. For virtual devices with tight memory constraints, and\n    // proportionately large allocation requests, this can lead to a significant\n    // reduction in available memory. The threshold below controls when a chunk\n    // should be split if the chunk size exceeds requested memory size. It is\n    // expressed as a fraction of total available memory for the tf device. For\n    // example setting it to 0.05 would imply a chunk needs to be split if its\n    // size exceeds the requested memory by 5% of the total virtual device/gpu\n    // memory size.\n    double internal_fragmentation_fraction = 10;\n\n    // When true, use CUDA cudaMallocAsync API instead of TF gpu allocator.\n    bool use_cuda_malloc_async = 11;\n\n    // By default, BFCAllocator may sleep when it runs out of memory, in the\n    // hopes that another thread will free up memory in the meantime.  Setting\n    // this to true disables the sleep; instead we'll OOM immediately.\n    bool disallow_retry_on_allocation_failure = 12;\n  }\n\n  // Everything inside experimental is subject to change and is not subject\n  // to API stability guarantees in\n  // https://www.tensorflow.org/guide/version_compat.\n  Experimental experimental = 9;\n}\n\n// Options passed to the graph optimizer\nmessage OptimizerOptions {\n  // If true, optimize the graph using common subexpression elimination.\n  // Note: the optimization Level L1 will override this setting to true. So in\n  // order to disable common subexpression elimination the opt_level has to be\n  // set to L0.\n  bool do_common_subexpression_elimination = 1;\n\n  // If true, perform constant folding optimization on the graph.\n  // Note: the optimization Level L1 will override this setting to true. So in\n  // order to disable constant folding the opt_level has to be set to L0.\n  bool do_constant_folding = 2;\n\n  // Constant folding optimization replaces tensors whose values can be\n  // predetermined, with constant nodes. To avoid inserting too large constants,\n  // the size of each constant created can be limited. If this value is zero, a\n  // default limit of 10 MiB will be applied. If constant folding optimization\n  // is disabled, this value is ignored.\n  int64 max_folded_constant_in_bytes = 6;\n\n  // If true, perform function inlining on the graph.\n  bool do_function_inlining = 4;\n\n  // Optimization level\n  enum Level {\n    // L1 is the default level.\n    // Optimization performed at L1 :\n    // 1. Common subexpression elimination\n    // 2. Constant folding\n    L1 = 0;\n\n    // No optimizations\n    L0 = -1;\n  }\n\n  // Overall optimization level. The actual optimizations applied will be the\n  // logical OR of the flags that this level implies and any flags already set.\n  Level opt_level = 3;\n\n  // Control the use of the compiler/jit.  Experimental.\n  enum GlobalJitLevel {\n    DEFAULT = 0;  // Default setting (\"off\" now, but later expected to be \"on\")\n    OFF = -1;\n    // The following settings turn on compilation, with higher values being\n    // more aggressive.  Higher values may reduce opportunities for parallelism\n    // and may use more memory.  (At present, there is no distinction, but this\n    // is expected to change.)\n    ON_1 = 1;\n    ON_2 = 2;\n  }\n  GlobalJitLevel global_jit_level = 5;\n\n  // CPU code will be autoclustered only if global_jit_level >= ON_1 and either:\n  //  - this flag is true, or\n  //  - TF_XLA_FLAGS contains --tf_xla_cpu_global_jit=true.\n  bool cpu_global_jit = 7;\n}\n\nmessage GraphOptions {\n  // Removed, use optimizer_options below.\n  reserved \"skip_common_subexpression_elimination\";\n  reserved 1;\n\n  // If true, use control flow to schedule the activation of Recv nodes.\n  // (Currently ignored.)\n  bool enable_recv_scheduling = 2;\n\n  // Options controlling how graph is optimized.\n  OptimizerOptions optimizer_options = 3;\n\n  // The number of steps to run before returning a cost model detailing\n  // the memory usage and performance of each node of the graph. 0 means\n  // no cost model.\n  int64 build_cost_model = 4;\n\n  // The number of steps to skip before collecting statistics for the\n  // cost model.\n  int64 build_cost_model_after = 9;\n\n  // Annotate each Node with Op output shape data, to the extent it can\n  // be statically inferred.\n  bool infer_shapes = 5;\n\n  // Only place the subgraphs that are run, rather than the entire graph.\n  //\n  // This is useful for interactive graph building, where one might\n  // produce graphs that cannot be placed during the debugging\n  // process.  In particular, it allows the client to continue work in\n  // a session after adding a node to a graph whose placement\n  // constraints are unsatisfiable.\n  bool place_pruned_graph = 6;\n\n  // If true, transfer float values between processes as bfloat16.\n  bool enable_bfloat16_sendrecv = 7;\n\n  // If > 0, record a timeline every this many steps.\n  // EXPERIMENTAL: This currently has no effect in MasterSession.\n  int32 timeline_step = 8;\n\n  // Options that control the type and amount of graph rewriting.\n  // Not currently configurable via the public Python API (i.e. there is no API\n  // stability guarantee if you import RewriterConfig explicitly).\n  RewriterConfig rewrite_options = 10;\n}\n\nmessage ThreadPoolOptionProto {\n  // The number of threads in the pool.\n  //\n  // 0 means the system picks a value based on where this option proto is used\n  // (see the declaration of the specific field for more info).\n  int32 num_threads = 1;\n\n  // The global name of the threadpool.\n  //\n  // If empty, then the threadpool is made and used according to the scope it's\n  // in - e.g., for a session threadpool, it is used by that session only.\n  //\n  // If non-empty, then:\n  // - a global threadpool associated with this name is looked\n  //   up or created. This allows, for example, sharing one threadpool across\n  //   many sessions (e.g., like the default behavior, if\n  //   inter_op_parallelism_threads is not configured), but still partitioning\n  //   into a large and small pool.\n  // - if the threadpool for this global_name already exists, then it is an\n  //   error if the existing pool was created using a different num_threads\n  //   value as is specified on this call.\n  // - threadpools created this way are never garbage collected.\n  string global_name = 2;\n}\n\nmessage RPCOptions {\n  // If true, always use RPC to contact the session target.\n  //\n  // If false (the default option), TensorFlow may use an optimized\n  // transport for client-master communication that avoids the RPC\n  // stack. This option is primarily for used testing the RPC stack.\n  bool use_rpc_for_inprocess_master = 1;\n\n  // The compression algorithm to be used. One of \"deflate\", \"gzip\".\n  string compression_algorithm = 2;\n\n  // If compression_algorithm is set, the compression level to be used.\n  // From 0 (no compression), up to 3.\n  int32 compression_level = 3;\n\n  // Setting cache_rpc_response to true will enable sender side caching of\n  // response for RecvTensorAsync and RecvBufAsync to allow receiver to retry\n  // requests . This is only necessary when the network fabric is experiencing a\n  // significant error rate.  Without it we'll fail a step on an network error,\n  // while with it we'll be able to complete long steps (like complex\n  // initializations) in the face of some network errors during RecvTensor.\n  bool cache_rpc_response = 4;\n\n  // Disables TCP connection sharing when opening a new RPC channel.\n  bool disable_session_connection_sharing = 5;\n\n  // Setting num_channels_per_target > 0 allows uses of multiple channels to\n  // communicate to the same target. This can be used to improve the aggregate\n  // throughput on high speed links (e.g 100G) where single connection is not\n  // sufficient to maximize link utilization. Note that a single RPC only goes\n  // on a single channel, this only helps in situations where there are multiple\n  // transfers to the same target overlapping in time.\n  int32 num_channels_per_target = 6;\n}\n\n// Metadata about the session.\n//\n// This can be used by the runtime and the Ops for debugging, monitoring, etc.\n//\n// The (name, version) tuple is expected to be a unique identifier for\n// sessions within the same process.\n//\n// NOTE: This is currently used and propagated only by the direct session.\nmessage SessionMetadata {\n  string name = 1;\n\n  // The version is optional. If set, needs to be >= 0.\n  int64 version = 2;\n}\n\n// Session configuration parameters.\n// The system picks appropriate values for fields that are not set.\nmessage ConfigProto {\n  // Map from device type name (e.g., \"CPU\" or \"GPU\" ) to maximum\n  // number of devices of that type to use.  If a particular device\n  // type is not found in the map, the system picks an appropriate\n  // number.\n  map<string, int32> device_count = 1;\n\n  // The execution of an individual op (for some op types) can be\n  // parallelized on a pool of intra_op_parallelism_threads.\n  // 0 means the system picks an appropriate number.\n  //\n  // If you create an ordinary session, e.g., from Python or C++,\n  // then there is exactly one intra op thread pool per process.\n  // The first session created determines the number of threads in this pool.\n  // All subsequent sessions reuse/share this one global pool.\n  //\n  // There are notable exceptions to the default behavior described above:\n  // 1. There is an environment variable  for overriding this thread pool,\n  //    named TF_OVERRIDE_GLOBAL_THREADPOOL.\n  // 2. When connecting to a server, such as a remote `tf.train.Server`\n  //    instance, then this option will be ignored altogether.\n  int32 intra_op_parallelism_threads = 2;\n\n  // Nodes that perform blocking operations are enqueued on a pool of\n  // inter_op_parallelism_threads available in each process.\n  //\n  // 0 means the system picks an appropriate number.\n  // Negative means all operations are performed in caller's thread.\n  //\n  // Note that the first Session created in the process sets the\n  // number of threads for all future sessions unless use_per_session_threads is\n  // true or session_inter_op_thread_pool is configured.\n  int32 inter_op_parallelism_threads = 5;\n\n  // If true, use a new set of threads for this session rather than the global\n  // pool of threads. Only supported by direct sessions.\n  //\n  // If false, use the global threads created by the first session, or the\n  // per-session thread pools configured by session_inter_op_thread_pool.\n  //\n  // This option is deprecated. The same effect can be achieved by setting\n  // session_inter_op_thread_pool to have one element, whose num_threads equals\n  // inter_op_parallelism_threads.\n  bool use_per_session_threads = 9;\n\n  // This option is experimental - it may be replaced with a different mechanism\n  // in the future.\n  //\n  // Configures session thread pools. If this is configured, then RunOptions for\n  // a Run call can select the thread pool to use.\n  //\n  // The intended use is for when some session invocations need to run in a\n  // background pool limited to a small number of threads:\n  // - For example, a session may be configured to have one large pool (for\n  // regular compute) and one small pool (for periodic, low priority work);\n  // using the small pool is currently the mechanism for limiting the inter-op\n  // parallelism of the low priority work.  Note that it does not limit the\n  // parallelism of work spawned by a single op kernel implementation.\n  // - Using this setting is normally not needed in training, but may help some\n  // serving use cases.\n  // - It is also generally recommended to set the global_name field of this\n  // proto, to avoid creating multiple large pools. It is typically better to\n  // run the non-low-priority work, even across sessions, in a single large\n  // pool.\n  repeated ThreadPoolOptionProto session_inter_op_thread_pool = 12;\n\n  // Assignment of Nodes to Devices is recomputed every placement_period\n  // steps until the system warms up (at which point the recomputation\n  // typically slows down automatically).\n  int32 placement_period = 3;\n\n  // When any filters are present sessions will ignore all devices which do not\n  // match the filters. Each filter can be partially specified, e.g. \"/job:ps\"\n  // \"/job:worker/replica:3\", etc.\n  repeated string device_filters = 4;\n\n  // Options that apply to all GPUs.\n  GPUOptions gpu_options = 6;\n\n  // Whether soft placement is allowed. If allow_soft_placement is true,\n  // an op will be placed on CPU if\n  //   1. there's no GPU implementation for the OP\n  // or\n  //   2. no GPU devices are known or registered\n  // or\n  //   3. need to co-locate with reftype input(s) which are from CPU.\n  bool allow_soft_placement = 7;\n\n  // Whether device placements should be logged.\n  bool log_device_placement = 8;\n\n  // Options that apply to all graphs.\n  GraphOptions graph_options = 10;\n\n  // Global timeout for all blocking operations in this session.  If non-zero,\n  // and not overridden on a per-operation basis, this value will be used as the\n  // deadline for all blocking operations.\n  int64 operation_timeout_in_ms = 11;\n\n  // Options that apply when this session uses the distributed runtime.\n  RPCOptions rpc_options = 13;\n\n  // Optional list of all workers to use in this session.\n  ClusterDef cluster_def = 14;\n\n  // If true, any resources such as Variables used in the session will not be\n  // shared with other sessions. However, when clusterspec propagation is\n  // enabled, this field is ignored and sessions are always isolated.\n  bool isolate_session_state = 15;\n\n  // When true, WorkerSessions are created with device attributes from the\n  // full cluster.\n  // This is helpful when a worker wants to partition a graph\n  // (for example during a PartitionedCallOp).\n  bool share_cluster_devices_in_session = 17;\n\n  // Everything inside Experimental is subject to change and is not subject\n  // to API stability guarantees in\n  // https://www.tensorflow.org/guide/version_compat.\n  message Experimental {\n    // Task name for group resolution.\n    string collective_group_leader = 1;\n\n    // We removed the flag client_handles_error_formatting. Marking the tag\n    // number as reserved.\n    // TODO(shikharagarwal): Should we just remove this tag so that it can be\n    // used in future for other purpose?\n    reserved 2;\n\n    // Which executor to use, the default executor will be used\n    // if it is an empty string or \"DEFAULT\"\n    string executor_type = 3;\n\n    // Guidance to formatting of large RecvBuf fields for transfer.\n    // Any positive value sets the max chunk size.  0 defaults to 4096.\n    // Any negative value indicates no max, i.e. one chunk only.\n    int32 recv_buf_max_chunk = 4;\n\n    // If true, and supported by the platform, the runtime will attempt to\n    // use NUMA affinity where applicable.  One consequence will be the\n    // existence of as many CPU devices as there are available NUMA nodes.\n    bool use_numa_affinity = 5;\n\n    // If true, make collective op execution order sequential and deterministic\n    // for potentially concurrent collective instances.\n    bool collective_deterministic_sequential_execution = 6;\n\n    // If true, use NCCL for CollectiveOps.  This feature is highly\n    // experimental.\n    bool collective_nccl = 7;\n\n    // In the following, session state means the value of a variable, elements\n    // in a hash table, or any other resource, accessible by worker sessions\n    // held by a TF server.\n    //\n    // When ClusterSpec propagation is enabled, the value of\n    // isolate_session_state is ignored when deciding whether to share session\n    // states in a TF server (for backwards compatibility reasons).\n    // - If share_session_state_in_clusterspec_propagation is true, the session\n    // states are shared.\n    // - If share_session_state_in_clusterspec_propagation is false, session\n    // states are isolated.\n    //\n    // When clusterspec propagation is not used, the value of\n    // share_session_state_in_clusterspec_propagation is ignored when deciding\n    // whether to share session states in a TF server.\n    // - If isolate_session_state is true, session states are isolated.\n    // - If isolate_session_state is false, session states are shared.\n    //\n    // TODO(b/129330037): Add a single API that consistently treats\n    // isolate_session_state and ClusterSpec propagation.\n    bool share_session_state_in_clusterspec_propagation = 8;\n\n    // If using a direct session, disable spinning while waiting for work in\n    // the thread pool. This may result in higher latency for completing ops,\n    // but in the case where there is a lot of spinning may result in lower\n    // CPU usage.\n    bool disable_thread_spinning = 9;\n\n    // This was promoted to a non-experimental API. Please use\n    // ConfigProto.share_cluster_devices_in_session instead.\n    bool share_cluster_devices_in_session = 10;\n\n    // Metadata about the session.\n    //\n    // If set, this can be used by the runtime and the Ops for debugging,\n    // monitoring, etc.\n    //\n    // NOTE: This is currently used and propagated only by the direct session.\n    SessionMetadata session_metadata = 11;\n\n    // If true, the session may treat the graph as being static for optimization\n    // purposes.\n    //\n    // If this option is set to true when a session is created, the full\n    // GraphDef must be passed in a single call to Session::Create(), and\n    // Session::Extend() may not be supported.\n    bool optimize_for_static_graph = 12;\n\n    // This field will eventually be deprecated and replaced by\n    // mlir_bridge_rollout (b/166038521).\n    //\n    // Whether to enable the MLIR-based TF->XLA bridge.\n    //\n    // This is a replacement to the existing bridge, and not ready for\n    // production usage yet.\n    // If this option is set to true when a session is created, MLIR is used to\n    // perform the set of graph transformations to put the graph in a form that\n    // can be executed with delegation of some computations to an accelerator.\n    // This builds on the model of XLA where a subset of the graph is\n    // encapsulated and attached to a \"compile\" operation, whose result is fed\n    // to an \"execute\" operation. The kernel for these operations is responsible\n    // to lower the encapsulated graph to a particular device.\n    bool enable_mlir_bridge = 13;\n\n    // An enum that describes the state of the MLIR bridge rollout.\n    enum MlirBridgeRollout {\n      // If this field is left unspecified, the MLIR bridge may be selectively\n      // enabled on a per graph basis.\n      MLIR_BRIDGE_ROLLOUT_UNSPECIFIED = 0;\n      // Enabling the MLIR bridge enables it for all graphs in this session.\n      MLIR_BRIDGE_ROLLOUT_ENABLED = 1;\n      // Disabling the MLIR bridge disables it for all graphs in this session.\n      MLIR_BRIDGE_ROLLOUT_DISABLED = 2;\n      // Enable the MLIR bridge on a per graph basis based on an analysis of\n      // the features used in the graph. If the features used by the graph are\n      // supported by the MLIR bridge, the MLIR bridge will be used to run the\n      // graph.\n      MLIR_BRIDGE_ROLLOUT_SAFE_MODE_ENABLED = 3;\n      // Enable the MLIR bridge in a fallback mode on a per graph basis based\n      // on an analysis of the features used in the graph.\n      // Running the MLIR bridge in the fallback mode means that it is\n      // executed and it commits all the changes to the TF graph in case\n      // of success. And it does not in case of failures and let the old bridge\n      // to process the TF graph.\n      MLIR_BRIDGE_ROLLOUT_SAFE_MODE_FALLBACK_ENABLED = 4;\n    }\n    // This field is underdevelopment, for now use enable_mlir_bridge\n    // (b/166038521).\n    //\n    // Whether to enable the MLIR-based TF->XLA bridge.\n    MlirBridgeRollout mlir_bridge_rollout = 17;\n\n    // Whether to enable the MLIR-based Graph optimizations.\n    //\n    // This will become a part of standard Tensorflow graph optimization\n    // pipeline, currently this is only used for gradual migration and testing\n    // new passes that are replacing existing optimizations in Grappler.\n    bool enable_mlir_graph_optimization = 16;\n\n    // If true, the session will not store an additional copy of the graph for\n    // each subgraph.\n    //\n    // If this option is set to true when a session is created, the\n    // `RunOptions.output_partition_graphs` options must not be set.\n    bool disable_output_partition_graphs = 14;\n\n    // Minimum number of batches run through the XLA graph before XLA fusion\n    // autotuner is enabled. Default value of zero disables the autotuner.\n    //\n    // The XLA fusion autotuner can improve performance by executing a heuristic\n    // search on the compiler parameters.\n    int64 xla_fusion_autotuner_thresh = 15;\n\n    // Whether runtime execution uses TFRT.\n    bool use_tfrt = 18;\n\n    // The field \"coordination_service was previously specified as a string;\n    // this has been replaced with a message below.\n    reserved 19;\n\n    // We removed the flag fetch_remote_devices_in_multi_client. Marking the tag\n    // number as reserved.\n    reserved 20;\n\n    // Whether functional control flow op lowering should be disabled. This is\n    // useful when executing within a portable runtime where control flow op\n    // kernels may not be loaded due to selective registration.\n    bool disable_functional_ops_lowering = 21;\n\n    // Provides a hint to XLA auto clustering to prefer forming a single large\n    // cluster that encompases most of the graph.\n    bool xla_prefer_single_graph_cluster = 22;\n\n    // Distributed coordination service configurations.\n    CoordinationServiceConfig coordination_config = 23;\n\n    // Next: 24\n  }\n\n  Experimental experimental = 16;\n\n  // Next: 18\n}\n\n// Options for a single Run() call.\nmessage RunOptions {\n  // TODO(pbar) Turn this into a TraceOptions proto which allows\n  // tracing to be controlled in a more orthogonal manner?\n  enum TraceLevel {\n    NO_TRACE = 0;\n    SOFTWARE_TRACE = 1;\n    HARDWARE_TRACE = 2;\n    FULL_TRACE = 3;\n  }\n  TraceLevel trace_level = 1;\n\n  // Time to wait for operation to complete in milliseconds.\n  int64 timeout_in_ms = 2;\n\n  // The thread pool to use, if session_inter_op_thread_pool is configured.\n  // To use the caller thread set this to -1 - this uses the caller thread\n  // to execute Session::Run() and thus avoids a context switch. Using the\n  // caller thread to execute Session::Run() should be done ONLY for simple\n  // graphs, where the overhead of an additional context switch is\n  // comparable with the overhead of Session::Run().\n  int32 inter_op_thread_pool = 3;\n\n  // Whether the partition graph(s) executed by the executor(s) should be\n  // outputted via RunMetadata.\n  bool output_partition_graphs = 5;\n\n  // EXPERIMENTAL.  Options used to initialize DebuggerState, if enabled.\n  DebugOptions debug_options = 6;\n\n  // When enabled, causes tensor allocation information to be included in\n  // the error message when the Run() call fails because the allocator ran\n  // out of memory (OOM).\n  //\n  // Enabling this option can slow down the Run() call.\n  bool report_tensor_allocations_upon_oom = 7;\n\n  // Everything inside Experimental is subject to change and is not subject\n  // to API stability guarantees in\n  // https://www.tensorflow.org/guide/version_compat.\n  message Experimental {\n    // If non-zero, declares that this graph is going to use collective\n    // ops and must synchronize step_ids with any other graph with this\n    // same group_key value (in a distributed computation where tasks\n    // run disjoint graphs).\n    int64 collective_graph_key = 1;\n    // If true, then operations (using the inter-op pool) across all\n    // session::run() calls will be centrally scheduled, optimizing for (median\n    // and tail) latency.\n    // Consider using this option for CPU-bound workloads like inference.\n    bool use_run_handler_pool = 2;\n    // Options for run handler thread pool.\n    message RunHandlerPoolOptions {\n      // Priority of the request. The run handler thread pool will schedule ops\n      // based on the priority number. The larger number means higher priority.\n      int64 priority = 1;\n    }\n    RunHandlerPoolOptions run_handler_pool_options = 3;\n  }\n\n  Experimental experimental = 8;\n\n  reserved 4;\n}\n\n// Metadata output (i.e., non-Tensor) for a single Run() call.\nmessage RunMetadata {\n  // Statistics traced for this step. Populated if tracing is turned on via the\n  // \"RunOptions\" proto.\n  // EXPERIMENTAL: The format and set of events may change in future versions.\n  StepStats step_stats = 1;\n\n  // The cost graph for the computation defined by the run call.\n  CostGraphDef cost_graph = 2;\n\n  // Graphs of the partitions executed by executors.\n  repeated GraphDef partition_graphs = 3;\n\n  message FunctionGraphs {\n    // TODO(nareshmodi): Include some sort of function/cache-key identifier?\n    repeated GraphDef partition_graphs = 1;\n\n    GraphDef pre_optimization_graph = 2;\n    GraphDef post_optimization_graph = 3;\n  }\n  // This is only populated for graphs that are run as functions in TensorFlow\n  // V2. There will be an entry below for each function that is traced.\n  // The main use cases of the post_optimization_graph and the partition_graphs\n  // is to give the caller insight into the graphs that were actually run by the\n  // runtime. Additional information (such as those in step_stats) will match\n  // these graphs.\n  // We also include the pre_optimization_graph since it is usually easier to\n  // read, and is helpful in situations where the caller wants to get a high\n  // level idea of what the built graph looks like (since the various graph\n  // optimization passes might change the structure of the graph significantly).\n  repeated FunctionGraphs function_graphs = 4;\n}\n\n// Defines a connection between two tensors in a `GraphDef`.\nmessage TensorConnection {\n  // A tensor name. The value of this tensor will be substituted for\n  // the tensor named in `to_tensor`.\n  string from_tensor = 1;\n\n  // A tensor name. The value of this tensor will be bound to the\n  // value of the tensor named in `from_tensor`.\n  string to_tensor = 2;\n}\n\n// Defines a subgraph in another `GraphDef` as a set of feed points and nodes\n// to be fetched or executed.\n//\n// Compare with the arguments to `Session::Run()`.\nmessage CallableOptions {\n  // Tensors to be fed in the callable. Each feed is the name of a tensor.\n  repeated string feed = 1;\n\n  // Fetches. A list of tensor names. The caller of the callable expects a\n  // tensor to be returned for each fetch[i] (see RunStepResponse.tensor). The\n  // order of specified fetches does not change the execution order.\n  repeated string fetch = 2;\n\n  // Target Nodes. A list of node names. The named nodes will be run by the\n  // callable but their outputs will not be returned.\n  repeated string target = 3;\n\n  // Options that will be applied to each run.\n  RunOptions run_options = 4;\n\n  // Tensors to be connected in the callable. Each TensorConnection denotes\n  // a pair of tensors in the graph, between which an edge will be created\n  // in the callable.\n  repeated TensorConnection tensor_connection = 5;\n\n  // The Tensor objects fed in the callable and fetched from the callable\n  // are expected to be backed by host (CPU) memory by default.\n  //\n  // The options below allow changing that - feeding tensors backed by\n  // device memory, or returning tensors that are backed by device memory.\n  //\n  // The maps below map the name of a feed/fetch tensor (which appears in\n  // 'feed' or 'fetch' fields above), to the fully qualified name of the device\n  // owning the memory backing the contents of the tensor.\n  //\n  // For example, creating a callable with the following options:\n  //\n  // CallableOptions {\n  //   feed: \"a:0\"\n  //   feed: \"b:0\"\n  //\n  //   fetch: \"x:0\"\n  //   fetch: \"y:0\"\n  //\n  //   feed_devices: {\n  //     \"a:0\": \"/job:localhost/replica:0/task:0/device:GPU:0\"\n  //   }\n  //\n  //   fetch_devices: {\n  //     \"y:0\": \"/job:localhost/replica:0/task:0/device:GPU:0\"\n  //  }\n  // }\n  //\n  // means that the Callable expects:\n  // - The first argument (\"a:0\") is a Tensor backed by GPU memory.\n  // - The second argument (\"b:0\") is a Tensor backed by host memory.\n  // and of its return values:\n  // - The first output (\"x:0\") will be backed by host memory.\n  // - The second output (\"y:0\") will be backed by GPU memory.\n  //\n  // FEEDS:\n  // It is the responsibility of the caller to ensure that the memory of the fed\n  // tensors will be correctly initialized and synchronized before it is\n  // accessed by operations executed during the call to Session::RunCallable().\n  //\n  // This is typically ensured by using the TensorFlow memory allocators\n  // (Device::GetAllocator()) to create the Tensor to be fed.\n  //\n  // Alternatively, for CUDA-enabled GPU devices, this typically means that the\n  // operation that produced the contents of the tensor has completed, i.e., the\n  // CUDA stream has been synchronized (e.g., via cuCtxSynchronize() or\n  // cuStreamSynchronize()).\n  map<string, string> feed_devices = 6;\n  map<string, string> fetch_devices = 7;\n\n  // By default, RunCallable() will synchronize the GPU stream before returning\n  // fetched tensors on a GPU device, to ensure that the values in those tensors\n  // have been produced. This simplifies interacting with the tensors, but\n  // potentially incurs a performance hit.\n  //\n  // If this options is set to true, the caller is responsible for ensuring\n  // that the values in the fetched tensors have been produced before they are\n  // used. The caller can do this by invoking `Device::Sync()` on the underlying\n  // device(s), or by feeding the tensors back to the same Session using\n  // `feed_devices` with the same corresponding device name.\n  bool fetch_skip_sync = 8;\n\n  // Next: 9\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/control_flow.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"ControlFlowProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Control flow context related protocol buffers.\n\n// Protocol buffer representing the values in ControlFlowContext.\nmessage ValuesDef {\n  // Value names that have been seen in this context.\n  repeated string values = 1;\n\n  // Value names referenced by but external to this context.\n  map<string, string> external_values = 2;\n}\n\n// Container for any kind of control flow context. Any other control flow\n// contexts that are added below should also be added here.\nmessage ControlFlowContextDef {\n  oneof ctxt {\n    CondContextDef cond_ctxt = 1;\n    WhileContextDef while_ctxt = 2;\n  }\n}\n\n// Protocol buffer representing a CondContext object.\nmessage CondContextDef {\n  // Name of the context.\n  string context_name = 1;\n\n  // Name of the pred tensor.\n  string pred_name = 2;\n\n  // Name of the pivot tensor.\n  string pivot_name = 3;\n\n  // Branch prediction. 0 or 1.\n  int32 branch = 4;\n\n  // Values and external values in control flow context.\n  ValuesDef values_def = 5;\n\n  // Contexts contained inside this context (e.g. nested conds).\n  repeated ControlFlowContextDef nested_contexts = 6;\n}\n\n// Protocol buffer representing a WhileContext object.\nmessage WhileContextDef {\n  // Name of the context.\n  string context_name = 1;\n\n  // The number of iterations allowed to run in parallel.\n  int32 parallel_iterations = 2;\n\n  // Whether backprop is enabled for this while loop.\n  bool back_prop = 3;\n\n  // Whether GPU-CPU memory swap is enabled for this loop.\n  bool swap_memory = 4;\n\n  // Name of the pivot tensor.\n  string pivot_name = 5;\n\n  // Name of the pivot_for_pred tensor.\n  string pivot_for_pred_name = 6;\n\n  // Name of the pivot_for_body tensor.\n  string pivot_for_body_name = 7;\n\n  // List of names for exit tensors.\n  repeated string loop_exit_names = 8;\n\n  // List of names for enter tensors.\n  repeated string loop_enter_names = 10;\n\n  // Values and external values in control flow context.\n  ValuesDef values_def = 9;\n\n  // Optional name of the maximum_iterations tensor.\n  string maximum_iterations_name = 11;\n\n  // Contexts contained inside this context (e.g. nested whiles).\n  repeated ControlFlowContextDef nested_contexts = 12;\n\n  // Next available id: 13.\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/conv_autotuning.proto",
    "content": "// This is used for convolution logging. Also see\n// tensorflow/core/protobuf/autotuing.h\nsyntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/stream_executor/dnn.proto\";\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// A convolution. Currently it's only used for logging. In the future, we may\n// want to use it in the API as well.\nmessage ConvolutionProto {\n  stream_executor.dnn.ConvolutionKind kind = 1;\n  stream_executor.dnn.TensorDescriptorProto input = 2;\n  stream_executor.dnn.TensorDescriptorProto filter = 3;\n  stream_executor.dnn.TensorDescriptorProto output = 4;\n  stream_executor.dnn.ConvolutionDescriptorProto conv_desc = 5;\n\n  // result = conv_scale * conv(...) + side_value_scale * side_value.\n  // side_value is an arbitrary buffer if activation is not none. Otherwise, it\n  // has to be the result buffer (using its old values).\n  double conv_scale = 6;\n  double side_value_scale = 7;\n\n  stream_executor.dnn.ActivationMode activation = 8;\n\n  int64 input_address = 9;\n  int64 filter_address = 10;\n  int64 output_address = 11;\n  int64 bias_address = 12;\n  int64 side_input_address = 13;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/coordination_config.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Coordination service configuration parameters.\n// The system picks appropriate values for fields that are not set.\nmessage CoordinationServiceConfig {\n  // Type of coordination service implementation to enable.\n  // For example, setting the service type as \"standalone\" starts a service\n  // instance on the leader task to provide the coordination services such as\n  // heartbeats and consistent key-value store.\n  string service_type = 1;\n\n  // Address where the coordination service instance is hosted.\n  string service_leader = 2;\n\n  // Whether to enable the health check mechanism.\n  bool enable_health_check = 3;\n\n  // Maximum wait time for all members in the cluster to be registered.\n  int64 cluster_register_timeout_in_ms = 4;\n\n  // Heartbeat timeout, if a worker does not record heartbeat in this time\n  // window, it will be considered disconnected.\n  int64 heartbeat_timeout_in_ms = 5;\n\n  // The list of jobs that partipate in the coordination service. If empty, all\n  // jobs will be included in the coordination service by default.\n  repeated string coordinated_jobs = 6;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/coordination_service.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/compiler/xla/pjrt/distributed/protocol.proto\";\nimport \"tensorflow/core/framework/device_attributes.proto\";\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Represents a remote worker task, specified by job name and task id.\nmessage CoordinatedTask {\n  string job_name = 1;\n  int32 task_id = 2;\n}\n\n// Status payload for all coordination service errors.\n// Note: an empty proto may be set if the error is triggered by the task's own\n// agent calls (i.e. not propagated by the service from another remote task).\nmessage CoordinationServiceError {\n  // Removed fields which used to specify the error origin.\n  reserved 1, 2;\n  // If true, error is reported via the agent API by the user (and not an\n  // internal service error).\n  bool is_reported_error = 3;\n  // Denotes which task hit the error. If unset, the error originated from the\n  // same task that is processing this error.\n  CoordinatedTask source_task = 4;\n}\n\n// Represent device information from different runtimes.\nmessage TfDeviceList {\n  repeated DeviceAttributes devices = 1;\n}\nmessage XlaDeviceList {\n  xla.GlobalTopologyProto devices = 1;\n}\nmessage CoordinationServiceDeviceInfo {\n  oneof type {\n    TfDeviceList tf = 1;\n    XlaDeviceList xla = 2;\n  }\n}\n\n// Request and response messages for registering a worker to the cluster leader.\n// Use `job` and `task` to represent the role of the worker, and use\n// `incarnation` to uniquely identify a worker process. Leader responds with its\n// `incarnation` to identify a leader process.\nmessage RegisterWorkerRequest {\n  // Removed fields which used to specify the task.\n  reserved 1, 2;\n  fixed64 incarnation = 3;\n  // Moved the field `local_device_attributes` from this request message to\n  // WaitForAllTasksRequest defined below.\n  reserved 4;\n  CoordinatedTask source_task = 5;\n}\n\nmessage RegisterWorkerResponse {\n  fixed64 leader_incarnation = 1;\n}\n\n// Request and response messages for sending heartbeats.\nmessage HeartbeatRequest {\n  // Removed fields which used to specify the remote task.\n  reserved 1, 2;\n  fixed64 incarnation = 3;\n  CoordinatedTask source_task = 4;\n}\n\nmessage HeartbeatResponse {\n  fixed64 leader_incarnation = 1;\n  // If there are failures in cluster, use additional metadata in response to\n  // broadcast error code and message to other workers.\n}\n\n// Request and response messages for waiting for all tasks.\nmessage WaitForAllTasksRequest {\n  // Removed fields which used to specify the remote task.\n  reserved 1, 2;\n  // Removed field that specifically used TF device info.\n  reserved 3;\n  // All local device attributes on the request sender.\n  CoordinationServiceDeviceInfo local_device_info = 4;\n  CoordinatedTask source_task = 5;\n}\n\nmessage WaitForAllTasksResponse {\n  fixed64 leader_incarnation = 1;\n  // Removed field that specifically used TF device info.\n  reserved 2;\n  // All devices in the cluster.\n  CoordinationServiceDeviceInfo cluster_device_info = 3;\n}\n\n// Request and response messages for reporting errors to task.\nmessage ReportErrorToAgentRequest {\n  int32 error_code = 1;\n  string error_message = 2;\n  // Removed fields that are embedded in payload.\n  reserved 3, 4;\n  CoordinationServiceError error_payload = 5;\n}\n\nmessage ReportErrorToAgentResponse {}\n\n// Request and response messages for reporting errors to service instance.\nmessage ReportErrorToServiceRequest {\n  int32 error_code = 1;\n  string error_message = 2;\n  // Removed fields which used to specify the error origin.\n  reserved 3, 4;\n  CoordinatedTask error_origin = 5;\n}\n\nmessage ReportErrorToServiceResponse {}\n\n// Message for configuration key value.\n// Key is structured like Unix file system, with multiple levels of directory\n// names separated by the slash ('/') characters.\nmessage KeyValueEntry {\n  string key = 1;\n  bytes value = 2;\n}\n\n// Request and response messages for inserting configuration key-value data.\nmessage InsertKeyValueRequest {\n  KeyValueEntry kv = 1;\n}\n\nmessage InsertKeyValueResponse {}\n\n// Request and response messages for getting configuration key-value data.\nmessage GetKeyValueRequest {\n  string key = 1;\n}\n\nmessage GetKeyValueResponse {\n  KeyValueEntry kv = 1;\n}\n\n// Request and response messages for deleting configuration key-value data.\n// When is_directory is true, delete key-values recursively under `key`.\nmessage DeleteKeyValueRequest {\n  string key = 1;\n  bool is_directory = 2;\n}\n\nmessage DeleteKeyValueResponse {}\n\n// Request and response messages for generic sync barriers.\nmessage BarrierRequest {\n  string barrier_id = 1;\n  int64 barrier_timeout_in_ms = 2;\n  // Denotes list of tasks that will wait for the barrier. If unspecified, it\n  // implies that the entire cluster is participating in the barrier.\n  repeated CoordinatedTask tasks = 3;\n  // Task that is making the request.\n  CoordinatedTask source_task = 4;\n}\n\nmessage BarrierResponse {}\n\n// Request and response messages for  cancelling generic sync barriers.\nmessage CancelBarrierRequest {\n  string barrier_id = 1;\n  // Task that is making the request.\n  CoordinatedTask source_task = 2;\n}\n\nmessage CancelBarrierResponse {}\n\n// Coordination Service defines a TensorFlow service that controls and\n// coordinates distributed execution in a cluster of multiple workers.\n//\n// The service keeps track of the cluster configuration and the state of cluster\n// members or the leader depending on the role of the current worker. The\n// distributed runtime leverages this service to coordinate and perform cluster\n// initialization, check the healthiness of workers, and propagate error\n// messages to the cluster.\nservice CoordinationService {\n  // Register task to coordination service so that the service starts to track\n  // liveness of the task. RPC blocks and returns only when it registers to\n  // the service successfully, or error happens in the registering process.\n  rpc RegisterWorker(RegisterWorkerRequest) returns (RegisterWorkerResponse);\n\n  // Heartbeat message from task to coordination service. Heartbeat is sent from\n  // a task to refresh its timestamp on leader to avoid it becoming stale.\n  // RPC responds immediately after refreshing the timestamp on leader.\n  rpc Heartbeat(HeartbeatRequest) returns (HeartbeatResponse);\n\n  // Wait for all tasks in the cluster to be up and running. The RPC request\n  // only gets responded when all workers are registered, or some error occurs.\n  rpc WaitForAllTasks(WaitForAllTasksRequest) returns (WaitForAllTasksResponse);\n\n  // Report error to the task. RPC sets the receiving instance of coordination\n  // service agent to error state permanently.\n  // TODO(b/195990880): Consider splitting this into a different RPC service.\n  rpc ReportErrorToAgent(ReportErrorToAgentRequest)\n      returns (ReportErrorToAgentResponse);\n\n  // Report task error to coordination service. RPC sets the service-side task\n  // state to error, and propagate the error to other tasks in the cluster.\n  rpc ReportErrorToService(ReportErrorToServiceRequest)\n      returns (ReportErrorToServiceResponse);\n\n  // Insert configuration key-value that will be accessible to all cluster\n  // workers. The key can be formatted as Unix file path with hierarchy. The\n  // coordination service key-value store should only be used for cluster\n  // configuration data.\n  rpc InsertKeyValue(InsertKeyValueRequest) returns (InsertKeyValueResponse);\n\n  // Get configuration key-value. The request blocks until the key-value data\n  // becomes available (i.e., set by a worker in the cluster).\n  rpc GetKeyValue(GetKeyValueRequest) returns (GetKeyValueResponse);\n\n  // Delete configuration key-value. If is_directory is set in request,\n  // recursively clean up all key-values under the path specified by `key`.\n  rpc DeleteKeyValue(DeleteKeyValueRequest) returns (DeleteKeyValueResponse);\n\n  // Blocks until all (or a subset of) tasks are at the barrier or the barrier\n  // fails.\n  //\n  // `barrier_id` should be unique across barriers. Once the barrier has passed\n  // or failed, subsequent calls will not block, and immediately respond with\n  // the previous response.\n  //\n  // The first WaitAtBarrier() call received by the service for a particular\n  // barrier id is special in that it determines the barrier deadline based on\n  // timeout duration.\n  // However, if subsequent calls by different agents specify a different set of\n  // `tasks` for the same `barrier_id`, the barrier will fail instantly.\n  //\n  // If no tasks are specified (default), the barrier will block for all the\n  // connected tasks.\n  //\n  // Possible service errors:\n  //   - DeadlineExceeded: Timed out waiting for specified tasks at the barrier.\n  //      Deadline is determined by the server timestamp when it receives the\n  //      first WaitAtBarrier() + timeout duration.\n  //   - Cancelled: One of the tasks called CancelBarrier().\n  //   - Aborted: Service is shutting down.\n  //   - Internal: Any participating task is in ERROR state.\n  //   - InvalidArgument: (1) Conflicting tasks specified by different agents\n  //       for the same barrier, (2) one of the participating tasks is not in\n  //       the cluster, or (3) task making the request is not included in the\n  //       list of participating tasks.\n  rpc Barrier(BarrierRequest) returns (BarrierResponse);\n\n  // Aborts the barrier if it is ongoing.\n  // Current and future WaitAtBarrier() calls with the same id will return a\n  // CANCELLED error status.\n  // Possible service errors:\n  //   - FailedPrecondition: Barrier has already been passed.\n  //   - NotFound: No barrier with the specified id is found.\n  rpc CancelBarrier(CancelBarrierRequest) returns (CancelBarrierResponse);\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/critical_section.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"CriticalSectionProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Protocol buffer representing a CriticalSection.\nmessage CriticalSectionDef {\n  // Name of the critical section handle.\n  string critical_section_name = 1;\n}\n\n// Protocol buffer representing a CriticalSection execution.\nmessage CriticalSectionExecutionDef {\n  // Name of the critical section handle.\n  string execute_in_critical_section_name = 1;\n  // Whether this operation requires exclusive access to its resources,\n  // (i.e., no other CriticalSections may request the same resources).\n  bool exclusive_resource_access = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/data_service.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.data;\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Next tag: 2\nmessage ProcessingModeDef {\n  // Specifies how data is sharded among tf.data service workers.\n  enum ShardingPolicy {\n    // No sharding will be performed. Each worker produces the entire dataset\n    // without any sharding. With this mode, the best practice is to shuffle the\n    // dataset nondeterministically so that workers process the dataset in\n    // different orders.\n    OFF = 0;\n\n    // The input dataset is dynamically split among workers at runtime. Each\n    // worker gets the next split when it reads data from the dispatcher. There\n    // is no fixed sharding with this mode.\n    DYNAMIC = 1;\n\n    // The following are static sharding policies. The semantics are similar to\n    // `tf.data.experimental.AutoShardPolicy`. These policies require:\n    // * The tf.data service cluster has a fixed size, and you need to specify\n    //   the workers in DispatcherConfig.\n    // * Each client only reads from the local tf.data service worker.\n    //\n    // Shards by input files (each worker will get a set of files to process).\n    // When this option is selected, make sure that there is at least as many\n    // files as workers. If there are fewer input files than workers, a runtime\n    // error will be raised.\n    FILE = 2;\n\n    // Shards by elements produced by the dataset. Each worker will process the\n    // whole dataset and discard the portion that is not for itself. Note that\n    // for this mode to correctly partitions the dataset elements, the dataset\n    // needs to produce elements in a deterministic order.\n    DATA = 3;\n\n    // Attempts FILE-based sharding, falling back to DATA-based sharding on\n    // failures.\n    FILE_OR_DATA = 4;\n\n    // Looks for the presence of `shard(SHARD_HINT, ...)` which is treated as a\n    // placeholder to replace with `shard(num_workers, worker_index)`.\n    HINT = 5;\n  }\n  ShardingPolicy sharding_policy = 1;\n}\n\n// tf.data service deployment mode.\nenum DeploymentMode {\n  DEPLOYMENT_MODE_UNSPECIFIED = 0;\n  // tf.data service workers colocate with TF workers.\n  DEPLOYMENT_MODE_COLOCATED = 1;\n  // tf.data service workers run in dedicated tf.data hosts.\n  DEPLOYMENT_MODE_REMOTE = 2;\n  // tf.data service workers run in colocated TF hosts and dedicated tf.data\n  // hosts.\n  DEPLOYMENT_MODE_HYBRID = 3;\n}\n\n// Metadata related to tf.data service datasets.\n// Next tag: 4\nmessage DataServiceMetadata {\n  oneof optional_element_spec {\n    // Serialized element spec.\n    bytes element_spec = 1;\n  }\n\n  enum Compression {\n    COMPRESSION_UNSPECIFIED = 0;\n    // No compression.\n    COMPRESSION_OFF = 1;\n    // Snappy compression as defined in tensorflow/core/platform/snappy.h.\n    COMPRESSION_SNAPPY = 2;\n  }\n  Compression compression = 2;\n\n  // Cardinality of the dataset.\n  int64 cardinality = 3;\n}\n\n// Data service config available to the client through GetDataServiceConfig RPC.\n// Next tag: 2\nmessage DataServiceConfig {\n  DeploymentMode deployment_mode = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/debug.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"DebugProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Option for watching a node in TensorFlow Debugger (tfdbg).\nmessage DebugTensorWatch {\n  // Name of the node to watch.\n  // Use \"*\" for wildcard. But note: currently, regex is not supported in\n  // general.\n  string node_name = 1;\n\n  // Output slot to watch.\n  // The semantics of output_slot == -1 is that all outputs of the node\n  // will be watched (i.e., a wildcard).\n  // Other negative values of output_slot are invalid and will lead to\n  // errors currently.\n  int32 output_slot = 2;\n\n  // Name(s) of the debugging op(s).\n  // One or more than one probes on a tensor.\n  // e.g., {\"DebugIdentity\", \"DebugNanCount\"}\n  repeated string debug_ops = 3;\n\n  // URL(s) for debug targets(s).\n  //\n  // Supported URL formats are:\n  //   - file:///foo/tfdbg_dump: Writes out Event content to file\n  //     /foo/tfdbg_dump.  Assumes all directories can be created if they don't\n  //     already exist.\n  //   - grpc://localhost:11011: Sends an RPC request to an EventListener\n  //     service running at localhost:11011 with the event.\n  //   - memcbk:///event_key: Routes tensors to clients using the\n  //     callback registered with the DebugCallbackRegistry for event_key.\n  //\n  // Each debug op listed in debug_ops will publish its output tensor (debug\n  // signal) to all URLs in debug_urls.\n  //\n  // N.B. Session::Run() supports concurrent invocations of the same inputs\n  // (feed keys), outputs and target nodes. If such concurrent invocations\n  // are to be debugged, the callers of Session::Run() must use distinct\n  // debug_urls to make sure that the streamed or dumped events do not overlap\n  // among the invocations.\n  // TODO(cais): More visible documentation of this in g3docs.\n  repeated string debug_urls = 4;\n\n  // Do not error out if debug op creation fails (e.g., due to dtype\n  // incompatibility). Instead, just log the failure.\n  bool tolerate_debug_op_creation_failures = 5;\n}\n\n// Options for initializing DebuggerState in TensorFlow Debugger (tfdbg).\nmessage DebugOptions {\n  // Debugging options\n  repeated DebugTensorWatch debug_tensor_watch_opts = 4;\n\n  // Caller-specified global step count.\n  // Note that this is distinct from the session run count and the executor\n  // step count.\n  int64 global_step = 10;\n\n  // Whether the total disk usage of tfdbg is to be reset to zero\n  // in this Session.run call. This is used by wrappers and hooks\n  // such as the local CLI ones to indicate that the dumped tensors\n  // are cleaned up from the disk after each Session.run.\n  bool reset_disk_byte_usage = 11;\n}\n\nmessage DebuggedSourceFile {\n  // The host name on which a source code file is located.\n  string host = 1;\n\n  // Path to the source code file.\n  string file_path = 2;\n\n  // The timestamp at which the source code file is last modified.\n  int64 last_modified = 3;\n\n  // Byte size of the file.\n  int64 bytes = 4;\n\n  // Line-by-line content of the source code file.\n  repeated string lines = 5;\n}\n\nmessage DebuggedSourceFiles {\n  // A collection of source code files.\n  repeated DebuggedSourceFile source_files = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/debug_event.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/tensor.proto\";\nimport \"tensorflow/core/protobuf/graph_debug_info.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"DebugEventProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.util\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Available modes for extracting debugging information from a Tensor.\n// TODO(cais): Document the detailed column names and semantics in a separate\n// markdown file once the implementation settles.\nenum TensorDebugMode {\n  UNSPECIFIED = 0;\n\n  // Only records what tensors are computed, eagerly or in graphs.\n  // No information regarding the value of the tensor is available.\n  NO_TENSOR = 1;\n\n  // A minimalist health summary for float-type tensors.\n  // Contains information only about the presence/absence of pathological\n  // values including Infinity and NaN.\n  // Applicable only to float dtypes.\n  CURT_HEALTH = 2;\n\n  // A concise health summary for float-type tensors.\n  // Contains more information that CURT_HEALTH.\n  // Infinity and NaN are treated differently.\n  // Applicable only to float and integer dtypes.\n  CONCISE_HEALTH = 3;\n\n  // A detailed health summary.\n  // Contains further detailed information than `CONCISE_HEALTH`.\n  // Information about device, dtype and shape are included.\n  // Counts for various types of values (Infinity, NaN, negative, zero,\n  // positive) are included.\n  // Applicable to float, integer and boolean dtypes.\n  FULL_HEALTH = 4;\n\n  // Provides full runtime shape information, up to a maximum rank, beyond\n  // which the dimension sizes are truncated.\n  SHAPE = 5;\n\n  // Full numeric summary.\n  // Including device, dtype, shape, counts of various types of values\n  // (Infinity, NaN, negative, zero, positive), and summary statistics\n  // (minimum, maximum, mean and variance).\n  // Applicable to float, integer and boolean dtypes.\n  FULL_NUMERICS = 6;\n\n  // Full tensor value.\n  FULL_TENSOR = 7;\n\n  // Reduce the elements of a tensor to a rank-1 tensor of shape [3], in which\n  // - the 1st element is -inf if any element of the tensor is -inf,\n  //   or zero otherwise.\n  // - the 2nd element is +inf if any element of the tensor is +inf,\n  //   or zero otherwise.\n  // - the 3rd element is nan if any element of the tensor is nan, or zero\n  //   otherwise.\n  REDUCE_INF_NAN_THREE_SLOTS = 8;\n}\n\n// An Event related to the debugging of a TensorFlow program.\nmessage DebugEvent {\n  // Timestamp in seconds (with microsecond precision).\n  double wall_time = 1;\n\n  // Step of training (if available).\n  int64 step = 2;\n\n  oneof what {\n    // Metadata related to this debugging data.\n    DebugMetadata debug_metadata = 3;\n\n    // The content of a source file.\n    SourceFile source_file = 4;\n\n    // A stack frame (filename, line number and column number, function name and\n    // code string) with ID.\n    StackFrameWithId stack_frame_with_id = 6;\n\n    // The creation of an op within a graph (e.g., a FuncGraph compiled from\n    // a Python function).\n    GraphOpCreation graph_op_creation = 7;\n\n    // Information about a debugged graph.\n    DebuggedGraph debugged_graph = 8;\n\n    // Execution of an op or a Graph (e.g., a tf.function).\n    Execution execution = 9;\n\n    // A graph execution trace: Contains information about the intermediate\n    // tensors computed during the graph execution.\n    GraphExecutionTrace graph_execution_trace = 10;\n\n    // The ID of the graph (i.e., FuncGraph) executed here: applicable only\n    // to the execution of a FuncGraph.\n    string graph_id = 11;\n\n    // A device on which debugger-instrumented ops and/or tensors reside.\n    DebuggedDevice debugged_device = 12;\n  }\n}\n\n// Metadata about the debugger and the debugged TensorFlow program.\nmessage DebugMetadata {\n  // Version of TensorFlow.\n  string tensorflow_version = 1;\n\n  // Version of the DebugEvent file format.\n  // Has a format of \"debug.Event:<number>\", e.g., \"debug.Event:1\".\n  string file_version = 2;\n\n  // A unique ID for the current run of tfdbg.\n  // A run of tfdbg is defined as a TensorFlow job instrumented by tfdbg.\n  // Multiple hosts in a distributed TensorFlow job instrumented by tfdbg\n  // have the same ID.\n  string tfdbg_run_id = 3;\n}\n\n// Content of a source file involved in the execution of the debugged TensorFlow\n// program.\nmessage SourceFile {\n  // Path to the file.\n  string file_path = 1;\n\n  // Name of the host on which the file is located.\n  string host_name = 2;\n\n  // Line-by-line content of the file.\n  repeated string lines = 3;\n}\n\n// A stack frame with ID.\nmessage StackFrameWithId {\n  // A unique ID for the stack frame: A UUID-like string.\n  string id = 1;\n\n  // Stack frame, i.e., a frame of a stack trace, containing information\n  // regarding the file name, line number, function name, code content\n  // of the line, and column number (if available).\n  GraphDebugInfo.FileLineCol file_line_col = 2;\n}\n\n// Code location information: A stack trace with host-name information.\n// Instead of encoding the detailed stack trace, this proto refers to IDs of\n// stack frames stored as `StackFrameWithId` protos.\nmessage CodeLocation {\n  // Host name on which the source files are located.\n  string host_name = 1;\n\n  // ID to a stack frame, each of which is pointed to\n  // by a unique ID. The ordering of the frames is consistent with Python's\n  // `traceback.extract_tb()`.\n  repeated string stack_frame_ids = 2;\n}\n\n// The creation of an op in a TensorFlow Graph (e.g., FuncGraph in TF2).\nmessage GraphOpCreation {\n  // Type of the op (e.g., \"MatMul\").\n  string op_type = 1;\n\n  // Name of the op (e.g., \"Dense/MatMul_1\").\n  string op_name = 2;\n\n  // Name of the graph that the op is a part of (if available).\n  string graph_name = 3;\n\n  // Unique ID of the graph (generated by debugger).\n  // This is the ID of the immediately-enclosing graph.\n  string graph_id = 4;\n\n  // Name of the device that the op is assigned to (if available).\n  string device_name = 5;\n\n  // Names of the input tensors to the op.\n  repeated string input_names = 6;\n\n  // Number of output tensors emitted by the op.\n  int32 num_outputs = 7;\n\n  // The unique ID for code location (stack trace) of the op's creation.\n  CodeLocation code_location = 8;\n\n  // Unique IDs for the output tensors of this op.\n  repeated int32 output_tensor_ids = 9;\n}\n\n// A debugger-instrumented graph.\nmessage DebuggedGraph {\n  // An ID for the graph.\n  // This can be used up to look up graph names. Generated by the debugger.\n  string graph_id = 1;\n\n  // Name of the graph (if available).\n  string graph_name = 2;\n\n  // Names of the instrumented ops. This can be used to look up op name\n  // based on the numeric-summary tensors (2nd column).\n  repeated string instrumented_ops = 3;\n\n  // Original (uninstrumented) GraphDef (if available).\n  bytes original_graph_def = 4;\n\n  // An encoded version of a GraphDef.\n  // This graph may include the debugger-inserted ops.\n  bytes instrumented_graph_def = 5;\n\n  // IDs of the immediate enclosing context (graph), if any.\n  string outer_context_id = 6;\n}\n\n// A device on which ops and/or tensors are instrumented by the debugger.\nmessage DebuggedDevice {\n  // Name of the device.\n  string device_name = 1;\n\n  // A debugger-generated ID for the device. Guaranteed to be unique within\n  // the scope of the debugged TensorFlow program, including single-host and\n  // multi-host settings.\n  // TODO(cais): Test the uniqueness guarantee in multi-host settings.\n  int32 device_id = 2;\n}\n\n// Data relating to the eager execution of an op or a Graph.\n// For a op that generates N output tensors (N >= 0), only one\n// Execution proto will be used to describe the execution event.\nmessage Execution {\n  // Op type (e.g., \"MatMul\").\n  // In the case of a Graph, this is the name of the Graph.\n  string op_type = 1;\n\n  // Number of output tensors.\n  int32 num_outputs = 2;\n\n  // The graph that's executed: applicable only to the eager\n  // execution of a FuncGraph.\n  string graph_id = 3;\n\n  // IDs of the input tensors (if available).\n  repeated int64 input_tensor_ids = 4;\n\n  // IDs of the output tensors (if availbable).\n  // If specified, must have the same length as tensor_protos.\n  repeated int64 output_tensor_ids = 5;\n\n  // Type of the tensor value encapsulated in this proto.\n  TensorDebugMode tensor_debug_mode = 6;\n\n  // Output Tensor values in the type described by `tensor_value_type`.\n  // The length of this should match `num_outputs`.\n  repeated TensorProto tensor_protos = 7;\n\n  // Stack trace of the eager execution.\n  CodeLocation code_location = 8;\n\n  // Debugged-generated IDs of the devices on which the output tensors reside.\n  // To look up details about the device (e.g., name), cross-reference this\n  // field with the DebuggedDevice messages.\n  repeated int32 output_tensor_device_ids = 9;\n\n  // TODO(cais): When backporting to V1 Session.run() support, add more fields\n  // such as fetches and feeds.\n}\n\n// Data relating to an execution of a Graph (e.g., an eager execution of a\n// FuncGraph).\n// The values of the intermediate tensors computed in the graph are recorded\n// in this proto. A graph execution may correspond to one or more pieces of\n// `GraphExecutionTrace`, depending on whether the instrumented tensor values\n// are summarized in an aggregated or separate fashion.\nmessage GraphExecutionTrace {\n  // Unique ID of the context that the executed op(s) belong to (e.g., a\n  // compiled concrete tf.function).\n  string tfdbg_context_id = 1;\n\n  // Name of the op (applicable only in the case of the `FULL_TENSOR` trace\n  // level).\n  string op_name = 2;\n\n  // Output slot of the tensor (applicable only in the case of the `FULL_TENSOR`\n  // trace level).\n  int32 output_slot = 3;\n\n  // Type of the tensor value encapsulated in this proto.\n  TensorDebugMode tensor_debug_mode = 4;\n\n  // Tensor value in the type described by `tensor_value_type`.\n  // This tensor may summarize the value of a single intermediate op of the\n  // graph, or those of multiple intermediate tensors.\n  TensorProto tensor_proto = 5;\n\n  // Name of the device that the op belongs to.\n  string device_name = 6;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/device_filters.proto",
    "content": "/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n==============================================================================*/\n\nsyntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"DeviceFiltersProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.distruntime\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// This file contains protos to be used when defining a TensorFlow\n// cluster.\n//\n// Configure device filters for remote tasks in the cluster. When associated\n// with a ClusterDef in setting up the cluster, a remote task will ignore all\n// devices which do not match any of its filters. Device filters must be\n// configured at the cluster startup, and cannot be updated once the cluster is\n// up and running.\n//\n// EXAMPLES\n// --------\n//\n// A two-job cluster with the following ClusterDef:\n//\n//  Cluster:\n//    job { name: 'worker' tasks { key: 0 value: 'worker1:2222' }\n//                         tasks { key: 1 value: 'worker2:2222' } }\n//    job { name: 'ps'     tasks { key: 0 value: 'ps0:2222' }\n//                         tasks { key: 1 value: 'ps1:2222' } }\n//\n// Set device filters to isolate worker tasks:\n//\n//  ClusterDeviceFilters:\n//    job { name: 'worker' tasks { key: 0\n//                                 value: device_filter '/job:ps'\n//                                        device_filter '/job:worker/task:0' }\n//                         tasks { key: 1\n//                                 value: device_filter '/job:ps'\n//                                        device_filter '/job:worker/task:1' } }\n\n// Defines the device filters for a remote task.\nmessage TaskDeviceFilters {\n  repeated string device_filters = 1;\n}\n\n// Defines the device filters for tasks in a job.\nmessage JobDeviceFilters {\n  // The name of this job.\n  string name = 1;\n\n  // Mapping from task ID to task device filters.\n  map<int32, TaskDeviceFilters> tasks = 2;\n}\n\n// Defines the device filters for jobs in a cluster.\nmessage ClusterDeviceFilters {\n  repeated JobDeviceFilters jobs = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/device_properties.proto",
    "content": "/* Copyright 2017 The TensorFlow Authors. All Rights Reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n   http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n==============================================================================*/\n\nsyntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"DevicePropertiesProtos\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\nmessage DeviceProperties {\n  // Device type (CPU, GPU, ...)\n  string type = 1;\n  // Vendor (Intel, nvidia, ...)\n  string vendor = 2;\n  // Model (Haswell, K40, ...)\n  string model = 3;\n  // Core Frequency in Mhz\n  int64 frequency = 4;\n  // Number of cores\n  int64 num_cores = 5;\n  // Version of the tools and libraries used with this device (e.g. gcc 4.9,\n  // cudnn 5.1)\n  map<string, string> environment = 6;\n  // Number of registers per core.\n  int64 num_registers = 7;\n  // L1 cache size in bytes\n  int64 l1_cache_size = 8;\n  // L2 cache size in bytes\n  int64 l2_cache_size = 9;\n  // L3 cache size in bytes\n  int64 l3_cache_size = 10;\n  // Shared memory size per multiprocessor in bytes. This field is\n  // applicable to GPUs only.\n  int64 shared_memory_size_per_multiprocessor = 11;\n  // Memory size in bytes\n  int64 memory_size = 12;\n  // Memory bandwidth in KB/s\n  int64 bandwidth = 13;\n}\n\nmessage NamedDevice {\n  string name = 1;\n  DeviceProperties properties = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/distributed_runtime_payloads.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.distributed_runtime;\n\noption cc_enable_arenas = true;\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Used to serialize and transmit tensorflow::Status payloads through\n// grpc::Status `error_details` since grpc::Status lacks payload API.\n// TODO(b/204231601): Use GRPC API once supported.\nmessage GrpcPayloadContainer {\n  map<string, bytes> payloads = 1;\n}\n\n// If included as a payload, this message flags the Status to have lost payloads\n// during the GRPC transmission.\n// URI: \"type.googleapis.com/tensorflow.distributed_runtime.GrpcPayloadsLost\"\nmessage GrpcPayloadsLost {}\n\n// If included as a payload, this message flags the Status to be a possible\n// outcome of a worker restart.\n// URI:\n// \"type.googleapis.com/tensorflow.distributed_runtime.WorkerPossiblyRestarted\"\nmessage WorkerPossiblyRestarted {}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/eager_service.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.eager;\n\nimport \"tensorflow/core/framework/attr_value.proto\";\nimport \"tensorflow/core/framework/device_attributes.proto\";\nimport \"tensorflow/core/framework/function.proto\";\nimport \"tensorflow/core/framework/tensor.proto\";\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/versions.proto\";\nimport \"tensorflow/core/protobuf/remote_tensor_handle.proto\";\nimport \"tensorflow/core/protobuf/tensorflow_server.proto\";\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// A proto representation of an eager operation.\nmessage Operation {\n  // A unique identifier for the operation. Set by the client so that the client\n  // can uniquely identify the outputs of the scheduled operation.\n  //\n  // In the initial implementation, sending duplicate IDs has undefined\n  // behaviour, but additional constraints may be placed upon this in the\n  // future.\n  int64 id = 1;\n  string name = 2;\n\n  message Input {\n    oneof item {\n      RemoteTensorHandle remote_handle = 1;\n      TensorProto tensor = 2;\n    }\n  }\n\n  repeated Input op_inputs = 10;\n\n  // Control Operation IDs that will be respected when ops are re-ordered by\n  // async execution. If async execution (+ op re-ordering) is not enabled, this\n  // should have no effect.\n  repeated int64 control_op_ids = 4;\n  map<string, AttrValue> attrs = 5;\n  string device = 6;\n\n  // Indicates whether the op is a component of a multi-device function.\n  bool is_component_function = 7;\n  // Set when is_component_function is true. It's initially generated\n  // when we create an FunctionLibraryRuntime::Options (negative value) and used\n  // to create Rendezvous for function execution. All components of a\n  // multi-device function should use the same step id to make sure that they\n  // can communicate through Send/Recv ops.\n  int64 func_step_id = 8;\n  // Indicates whether the op is a function.\n  bool is_function = 9;\n\n  reserved 3;\n}\n\nmessage QueueItem {\n  // The remote executor should be able to handle either executing ops directly,\n  // or releasing any unused tensor handles, since the tensor lifetime is\n  // maintained by the client.\n  oneof item {\n    RemoteTensorHandle handle_to_decref = 1;\n    Operation operation = 2;\n    SendTensorOp send_tensor = 3;\n    // Takes a FunctionDef and makes it enqueable on the remote worker.\n    RegisterFunctionOp register_function = 4;\n    CleanupFunctionOp cleanup_function = 5;\n    // A remote executor is created to execute ops/functions asynchronously\n    // enqueued in streaming call. Request with this item type waits for pending\n    // nodes to finish on the remote executor and report status.\n    SyncRemoteExecutorForStream sync_remote_executor_for_stream = 6;\n    SendPackedHandleOp send_packed_handle = 7;\n  }\n}\n\nmessage QueueResponse {\n  // `shape` and `tensor` cannot be set in the same response.\n  // Shapes of output tensors for creating remote TensorHandles.\n  repeated TensorShapeProto shape = 1;\n  // Optional. If set, represents the output devices of a function.\n  repeated string device = 3;\n\n  // Output tensors of a remote function. Set when Operation.id is invalid.\n  repeated TensorProto tensor = 2;\n}\n\nmessage CreateContextRequest {\n  // Identifies the full cluster, and this particular worker's position within.\n  ServerDef server_def = 1;\n\n  // Whether the ops on the worker should be executed synchronously or\n  // asynchronously. By default, ops are executed synchronously.\n  bool async = 2;\n\n  // Number of seconds to keep the context alive. If more than keep_alive_secs\n  // has passed since a particular context has been communicated with, it will\n  // be garbage collected.\n  int64 keep_alive_secs = 3;\n\n  // This is the version for all the ops that will be enqueued by the client.\n  VersionDef version_def = 4;\n\n  // Device attributes in the cluster\n  repeated DeviceAttributes cluster_device_attributes = 6;\n\n  // The ID of the created context. This is usually a randomly generated number,\n  // that will be used to identify the context in future requests to the\n  // service. Contexts are not persisted through server restarts.\n  // This ID will be used for all future communications as well. It is essential\n  // that both ends use this ID for selecting a rendezvous to get everything to\n  // match.\n  fixed64 context_id = 7;\n\n  // The view ID of the context.\n  fixed64 context_view_id = 8;\n\n  // For a multi device function, if false, eagerly copy all remote inputs to\n  // the default function device; if true, lazily copy remote inputs to their\n  // target devices after function instantiation to avoid redundant copies.\n  bool lazy_copy_remote_function_inputs = 9;\n\n  reserved 5;\n}\n\nmessage CreateContextResponse {\n  // List of devices that are locally accessible to the worker.\n  repeated DeviceAttributes device_attributes = 2;\n\n  reserved 1;\n}\n\nmessage UpdateContextRequest {\n  // Identifies the full cluster, and this particular worker's position within.\n  ServerDef server_def = 1;\n\n  // Device attributes in the cluster.\n  // If this field is empty, it indicates that this is a simple update request\n  // that only increments the cluster view ID and does not require changes to\n  // the workers it connects to.\n  repeated DeviceAttributes cluster_device_attributes = 2;\n\n  // The ID of the context to be updated. A context with the specified ID must\n  // already exist on the recepient server of this request.\n  fixed64 context_id = 3;\n\n  // The view ID of the context, which should be contiguously incremented when\n  // updating the same context.\n  fixed64 context_view_id = 4;\n}\n\nmessage UpdateContextResponse {\n  // List of devices that are locally accessible to the worker.\n  repeated DeviceAttributes device_attributes = 1;\n}\n\nmessage EnqueueRequest {\n  fixed64 context_id = 1;\n\n  repeated QueueItem queue = 3;\n}\n\nmessage EnqueueResponse {\n  // A single operation response for every item in the request.\n  repeated QueueResponse queue_response = 1;\n}\n\nmessage WaitQueueDoneRequest {\n  fixed64 context_id = 1;\n\n  // Ids to wait on. If empty, wait on everything currently pending.\n  repeated int64 op_id = 2;\n}\n\nmessage WaitQueueDoneResponse {\n  // TODO(nareshmodi): Consider adding NodeExecStats here to be able to\n  // propagate some stats.\n}\n\nmessage RunComponentFunctionRequest {\n  fixed64 context_id = 1;\n\n  Operation operation = 2;\n\n  // The output indices of its parent function.\n  repeated int32 output_num = 3;\n}\n\nmessage RunComponentFunctionResponse {\n  repeated TensorShapeProto shape = 1;\n\n  repeated TensorProto tensor = 2;\n}\n\nmessage KeepAliveRequest {\n  fixed64 context_id = 1;\n}\n\nmessage KeepAliveResponse {\n  // If the requested context_id is on the remote host, set the context view ID.\n  fixed64 context_view_id = 1;\n}\n\nmessage CloseContextRequest {\n  fixed64 context_id = 1;\n  fixed64 context_view_id = 2;\n}\n\nmessage CloseContextResponse {}\n\nmessage RegisterFunctionOp {\n  FunctionDef function_def = 1;\n\n  // If true, it means that function_def is produced by graph partition during\n  // multi-device function instantiation.\n  bool is_component_function = 2;\n\n  // All necessary FunctionDefs and GradientDefs to expand `function_def`.\n  // When is_component_function is true, `function_def` could be a nested\n  // function, since some nodes in its parent's function body could be\n  // replaced with a new function by the graph optimization passes. No need to\n  // add FunctionDefs here to the function cache in EagerContext since they\n  // won't be executed as KernelAndDevices.\n  FunctionDefLibrary library = 3;\n}\n\n// Cleanup the step state of a multi-device function (e.g. tensors buffered by\n// a `Send` op but not picked up by its corresponding `Recv` op).\nmessage CleanupFunctionOp {\n  int64 step_id = 1;\n}\n\nmessage SyncRemoteExecutorForStream {}\n\nmessage SendTensorOp {\n  // All remote tensors are identified by <Op ID, Output num>. To mimic this\n  // situation when directly sending tensors, we include an \"artificial\" op ID\n  // (which would have corresponded to the _Recv op when not using SendTensor).\n  int64 op_id = 1;\n  // The index within the repeated field is the output number that will help\n  // uniquely identify (along with the above op_id) the particular tensor.\n  repeated TensorProto tensors = 2;\n\n  // The device on which the tensors should be resident.\n  string device_name = 3;\n}\n\n// Send a packed TensorHandle to a remote worker.\nmessage SendPackedHandleOp {\n  // Op id of the remote packed TensorHandle.\n  int64 op_id = 1;\n\n  message LocalTensorHandle {\n    TensorProto tensor = 1;\n    // Device where the tensor is produced.\n    string device = 2;\n  }\n\n  message Handle {\n    oneof item {\n      LocalTensorHandle local_handle = 1;\n      RemoteTensorHandle remote_handle = 2;\n    }\n  }\n\n  repeated Handle handles = 2;\n\n  string device_name = 3;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// Eager Service defines a TensorFlow service that executes operations eagerly\n// on a set of local devices, on behalf of a remote Eager executor.\n//\n// The service impl will keep track of the various clients and devices it has\n// access to and allows the client to enqueue ops on any devices that it is able\n// to access and schedule data transfers from/to any of the peers.\n//\n// A client can generate multiple contexts to be able to independently execute\n// operations, but cannot share data between the two contexts.\n//\n// NOTE: Even though contexts generated by clients should be independent, the\n// lower level tensorflow execution engine is not, so they might share some data\n// (e.g. a Device's ResourceMgr).\n//\n////////////////////////////////////////////////////////////////////////////////\nservice EagerService {\n  // This initializes the worker, informing it about the other workers in the\n  // cluster and exchanging authentication tokens which will be used in all\n  // other RPCs to detect whether the worker has restarted.\n  rpc CreateContext(CreateContextRequest) returns (CreateContextResponse);\n\n  // This updates the eager context on an existing worker when updating the set\n  // of servers in a distributed eager cluster.\n  rpc UpdateContext(UpdateContextRequest) returns (UpdateContextResponse);\n\n  // This takes a list of Execute and DeleteTensorHandle operations and enqueues\n  // (in async mode) or executes (in sync mode) them on the remote server.\n  // All outputs of ops which were not explicitly deleted with\n  // DeleteTensorHandle entries will be assumed to be alive and are usable by\n  // future calls to Enqueue.\n  rpc Enqueue(EnqueueRequest) returns (EnqueueResponse);\n\n  // A streaming version of Enqueue.\n  // Current server implementation sends one response per received request.\n  // The benefit for using a streaming version is that subsequent requests\n  // can be sent without waiting for a response to the previous request. This\n  // synchronization is required in the regular Enqueue call because gRPC does\n  // not guarantee to preserve request order.\n  rpc StreamingEnqueue(stream EnqueueRequest) returns (stream EnqueueResponse);\n\n  // Takes a set of op IDs and waits until those ops are done. Returns any error\n  // in the stream so far.\n  rpc WaitQueueDone(WaitQueueDoneRequest) returns (WaitQueueDoneResponse);\n\n  // This takes an Eager operation and executes it in async mode on the remote\n  // server. Different from EnqueueRequest, ops/functions sent through this\n  // type of requests are allowed to execute in parallel and no ordering is\n  // preserved by RPC stream or executor.\n  // This request type should only be used for executing component functions.\n  // Ordering of component functions should be enforced by their corresponding\n  // main functions. The runtime ensures the following invarients for component\n  // functions (CFs) and their main functions (MFs):\n  // (1) MF1 -> MF2 ==> CF1 -> CF2 (\"->\" indicates order of execution);\n  // (2) MF1 || MF2 ==> CF1 || CF2 (\"||\" indicates possible parallel execution);\n  // (3) For CF1 and CF2 that come from the same MF, CF1 || CF2\n  // For executing ops/main functions, use Enqueue or StreamingEnqueue instead\n  // for correct ordering.\n  rpc RunComponentFunction(RunComponentFunctionRequest)\n      returns (RunComponentFunctionResponse);\n\n  // Contexts are always created with a deadline and no RPCs within a deadline\n  // will trigger a context garbage collection. KeepAlive calls can be used to\n  // delay this. It can also be used to validate the existence of a context ID\n  // on remote eager worker. If the context is on remote worker, return the same\n  // ID and the current context view ID. This is useful for checking if the\n  // remote worker (potentially with the same task name and hostname / port) is\n  // replaced with a new process.\n  rpc KeepAlive(KeepAliveRequest) returns (KeepAliveResponse);\n\n  // Closes the context. No calls to other methods using the existing context ID\n  // are valid after this.\n  rpc CloseContext(CloseContextRequest) returns (CloseContextResponse);\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/error_codes.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.error;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"ErrorCodesProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// The canonical error codes for TensorFlow APIs.\n//\n// Warnings:\n//\n// -   Do not change any numeric assignments.\n// -   Changes to this list should only be made if there is a compelling\n//     need that can't be satisfied in another way.  Such changes\n//     must be approved by at least two OWNERS.\n// -   These error codes must match gRPC and protobuf error codes (except for\n//     DO_NOT_USE_RESERVED_FOR_FUTURE_EXPANSION_USE_DEFAULT_IN_SWITCH_INSTEAD_).\n//\n// Sometimes multiple error codes may apply.  Services should return\n// the most specific error code that applies.  For example, prefer\n// OUT_OF_RANGE over FAILED_PRECONDITION if both codes apply.\n// Similarly prefer NOT_FOUND or ALREADY_EXISTS over FAILED_PRECONDITION.\nenum Code {\n  // Not an error; returned on success\n  OK = 0;\n\n  // The operation was cancelled (typically by the caller).\n  CANCELLED = 1;\n\n  // Unknown error.  An example of where this error may be returned is\n  // if a Status value received from another address space belongs to\n  // an error-space that is not known in this address space.  Also\n  // errors raised by APIs that do not return enough error information\n  // may be converted to this error.\n  UNKNOWN = 2;\n\n  // Client specified an invalid argument.  Note that this differs\n  // from FAILED_PRECONDITION.  INVALID_ARGUMENT indicates arguments\n  // that are problematic regardless of the state of the system\n  // (e.g., a malformed file name).\n  INVALID_ARGUMENT = 3;\n\n  // Deadline expired before operation could complete.  For operations\n  // that change the state of the system, this error may be returned\n  // even if the operation has completed successfully.  For example, a\n  // successful response from a server could have been delayed long\n  // enough for the deadline to expire.\n  DEADLINE_EXCEEDED = 4;\n\n  // Some requested entity (e.g., file or directory) was not found.\n  // For privacy reasons, this code *may* be returned when the client\n  // does not have the access right to the entity.\n  NOT_FOUND = 5;\n\n  // Some entity that we attempted to create (e.g., file or directory)\n  // already exists.\n  ALREADY_EXISTS = 6;\n\n  // The caller does not have permission to execute the specified\n  // operation.  PERMISSION_DENIED must not be used for rejections\n  // caused by exhausting some resource (use RESOURCE_EXHAUSTED\n  // instead for those errors).  PERMISSION_DENIED must not be\n  // used if the caller can not be identified (use UNAUTHENTICATED\n  // instead for those errors).\n  PERMISSION_DENIED = 7;\n\n  // The request does not have valid authentication credentials for the\n  // operation.\n  UNAUTHENTICATED = 16;\n\n  // Some resource has been exhausted, perhaps a per-user quota, or\n  // perhaps the entire file system is out of space.\n  RESOURCE_EXHAUSTED = 8;\n\n  // Operation was rejected because the system is not in a state\n  // required for the operation's execution.  For example, directory\n  // to be deleted may be non-empty, an rmdir operation is applied to\n  // a non-directory, etc.\n  //\n  // A litmus test that may help a service implementor in deciding\n  // between FAILED_PRECONDITION, ABORTED, and UNAVAILABLE:\n  //  (a) Use UNAVAILABLE if the client can retry just the failing call.\n  //  (b) Use ABORTED if the client should retry at a higher-level\n  //      (e.g., restarting a read-modify-write sequence).\n  //  (c) Use FAILED_PRECONDITION if the client should not retry until\n  //      the system state has been explicitly fixed.  E.g., if an \"rmdir\"\n  //      fails because the directory is non-empty, FAILED_PRECONDITION\n  //      should be returned since the client should not retry unless\n  //      they have first fixed up the directory by deleting files from it.\n  //  (d) Use FAILED_PRECONDITION if the client performs conditional\n  //      REST Get/Update/Delete on a resource and the resource on the\n  //      server does not match the condition. E.g., conflicting\n  //      read-modify-write on the same resource.\n  FAILED_PRECONDITION = 9;\n\n  // The operation was aborted, typically due to a concurrency issue\n  // like sequencer check failures, transaction aborts, etc.\n  //\n  // See litmus test above for deciding between FAILED_PRECONDITION,\n  // ABORTED, and UNAVAILABLE.\n  ABORTED = 10;\n\n  // Operation tried to iterate past the valid input range.  E.g., seeking or\n  // reading past end of file.\n  //\n  // Unlike INVALID_ARGUMENT, this error indicates a problem that may\n  // be fixed if the system state changes. For example, a 32-bit file\n  // system will generate INVALID_ARGUMENT if asked to read at an\n  // offset that is not in the range [0,2^32-1], but it will generate\n  // OUT_OF_RANGE if asked to read from an offset past the current\n  // file size.\n  //\n  // There is a fair bit of overlap between FAILED_PRECONDITION and\n  // OUT_OF_RANGE.  We recommend using OUT_OF_RANGE (the more specific\n  // error) when it applies so that callers who are iterating through\n  // a space can easily look for an OUT_OF_RANGE error to detect when\n  // they are done.\n  OUT_OF_RANGE = 11;\n\n  // Operation is not implemented or not supported/enabled in this service.\n  UNIMPLEMENTED = 12;\n\n  // Internal errors.  Means some invariant expected by the underlying\n  // system has been broken.  If you see one of these errors,\n  // something is very broken.\n  INTERNAL = 13;\n\n  // The service is currently unavailable.  This is a most likely a\n  // transient condition and may be corrected by retrying with\n  // a backoff.\n  //\n  // See litmus test above for deciding between FAILED_PRECONDITION,\n  // ABORTED, and UNAVAILABLE.\n  UNAVAILABLE = 14;\n\n  // Unrecoverable data loss or corruption.\n  DATA_LOSS = 15;\n\n  // An extra enum entry to prevent people from writing code that\n  // fails to compile when a new code is added.\n  //\n  // Nobody should ever reference this enumeration entry. In particular,\n  // if you write C++ code that switches on this enumeration, add a default:\n  // case instead of a case that mentions this enumeration entry.\n  //\n  // Nobody should rely on the value (currently 20) listed here.  It\n  // may change in the future.\n  DO_NOT_USE_RESERVED_FOR_FUTURE_EXPANSION_USE_DEFAULT_IN_SWITCH_INSTEAD_ = 20;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/graph_debug_info.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"GraphDebugInfoProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\nmessage GraphDebugInfo {\n  // This represents a file/line location in the source code.\n  message FileLineCol {\n    // File name index, which can be used to retrieve the file name string from\n    // `files`. The value should be between 0 and (len(files)-1)\n    int32 file_index = 1;\n\n    // Line number in the file.\n    int32 line = 2;\n\n    // Col number in the file line.\n    int32 col = 3;\n\n    // Name of function contains the file line.\n    string func = 4;\n\n    // Source code contained in this file line.\n    string code = 5;\n  }\n\n  // This represents a stack trace which is a ordered list of `FileLineCol`.\n  message StackTrace {\n    // Each line in the stack trace.\n    repeated FileLineCol file_line_cols = 1;\n  }\n\n  // This stores all the source code file names and can be indexed by the\n  // `file_index`.\n  repeated string files = 1;\n\n  // This maps a node name to a stack trace in the source code.\n  // The map key is a mangling of the containing function and op name with\n  // syntax:\n  //   op.name '@' func_name\n  // For ops in the top-level graph, the func_name is the empty string.\n  // Note that op names are restricted to a small number of characters which\n  // exclude '@', making it impossible to collide keys of this form. Function\n  // names accept a much wider set of characters.\n  // It would be preferable to avoid mangling and use a tuple key of (op.name,\n  // func_name), but this is not supported with protocol buffers.\n  map<string, StackTrace> traces = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/master.proto",
    "content": "/* Copyright 2016 The TensorFlow Authors. All Rights Reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n==============================================================================*/\n\nsyntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/device_attributes.proto\";\nimport \"tensorflow/core/framework/graph.proto\";\nimport \"tensorflow/core/framework/tensor.proto\";\nimport \"tensorflow/core/protobuf/config.proto\";\nimport \"tensorflow/core/protobuf/error_codes.proto\";\nimport \"tensorflow/core/protobuf/named_tensor.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"DistributedRuntimeProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.distruntime\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// CreateSession method request/response protos.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage CreateSessionRequest {\n  // The initial graph definition.\n  GraphDef graph_def = 1;\n\n  // Configuration options.\n  ConfigProto config = 2;\n\n  // The target string used from the client's perspective.\n  string target = 3;\n}\n\nmessage CreateSessionResponse {\n  // The session handle to be used in subsequent calls for the created session.\n  //\n  // The client must arrange to call CloseSession with this returned\n  // session handle to close the session.\n  string session_handle = 1;\n\n  // The initial version number for the graph, to be used in the next call\n  // to ExtendSession.\n  int64 graph_version = 2;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// ExtendSession method request/response protos.\n//\n// The \"graph_def\" specifies a set of nodes to be added to the session's graph.\n//\n// A typical \"graph_def\" will contain:\n//\n// * Zero or more new nodes with names that do not exist in the server-side\n//   graph. These will be added to the graph.\n//\n// PRECONDITION: The server-side current version is req.current_version.\n//   None of the names in req.graph_def appeared in previous successful calls to\n//   CreateSession or ExtendSession with the same session_handle.\n// POSTCONDITION: The server-side current version is resp.new_version.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage ExtendSessionRequest {\n  // REQUIRED: session_handle must be returned by a CreateSession call\n  // to the same master service.\n  string session_handle = 1;\n\n  // REQUIRED: The nodes to be added to the session's graph. If any node has\n  // the same name as an existing node, the operation will fail with\n  // ILLEGAL_ARGUMENT.\n  GraphDef graph_def = 2;\n\n  // REQUIRED: The version number of the graph to be extended. This will be\n  // tested against the current server-side version number, and the operation\n  // will fail with FAILED_PRECONDITION if they do not match.\n  int64 current_graph_version = 3;\n}\n\nmessage ExtendSessionResponse {\n  // TODO(mrry): Return something about the operation?\n\n  // The new version number for the extended graph, to be used in the next call\n  // to ExtendSession.\n  int64 new_graph_version = 4;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// RunStep method request/response protos.\n//\n// The caller should provide the feeds needed by the graph and specify\n// what nodes should be fetched.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage RunStepRequest {\n  // REQUIRED: session_handle must be returned by a CreateSession call\n  // to the same master service.\n  string session_handle = 1;\n\n  // Tensors to be fed in the step. Each feed is a named tensor.\n  repeated NamedTensorProto feed = 2;\n\n  // Fetches. A list of tensor names. The caller expects a tensor to\n  // be returned for each fetch[i] (see RunStepResponse.tensor). The\n  // order of specified fetches does not change the execution order.\n  repeated string fetch = 3;\n\n  // Target Nodes. A list of node names. The named nodes will be run\n  // to but their outputs will not be fetched.\n  repeated string target = 4;\n\n  // Options for the run call.\n  RunOptions options = 5;\n\n  // Partial run handle (optional). If specified, this will be a partial run\n  // execution, run up to the specified fetches.\n  string partial_run_handle = 6;\n\n  // If true then some errors, e.g., execution errors that have long\n  // error messages, may return an OK RunStepResponse with the actual\n  // error saved in the status_code/status_error_message fields of the\n  // response body. This is a workaround since the RPC subsystem may\n  // truncate long metadata messages.\n  bool store_errors_in_response_body = 7;\n\n  // Unique identifier for this request. Every RunStepRequest must\n  // have a unique request_id, and retried RunStepRequest must have\n  // the same request_id. If request_id is zero, retry detection is disabled.\n  int64 request_id = 8;\n}\n\nmessage RunStepResponse {\n  // NOTE: The order of the returned tensors may or may not match\n  // the fetch order specified in RunStepRequest.\n  repeated NamedTensorProto tensor = 1;\n\n  // Returned metadata if requested in the options.\n  RunMetadata metadata = 2;\n\n  // If store_errors_in_response_body is true in the request, then\n  // optionally the server may return an OK status for the RPC and\n  // fill the true status into the fields below, to allow for messages\n  // that are too long to fit in metadata.\n  error.Code status_code = 3;\n  string status_error_message = 4;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// PartialRunSetup method request/response protos.\n//\n// The caller should provide the future partial run feeds, fetches, and targets.\n// Then the caller can use RunStepRequest with is_partial set to make partial\n// run calls.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage PartialRunSetupRequest {\n  // REQUIRED: session_handle must be returned by a CreateSession call\n  // to the same master service.\n  string session_handle = 1;\n\n  // Tensors to be fed in future steps.\n  repeated string feed = 2;\n\n  // Fetches. A list of tensor names. The caller expects a tensor to be returned\n  // for each fetch[i] (see RunStepResponse.tensor), for corresponding partial\n  // RunStepRequests. The order of specified fetches does not change the\n  // execution order.\n  repeated string fetch = 3;\n\n  // Target Nodes. A list of node names. The named nodes will be run in future\n  // steps, but their outputs will not be fetched.\n  repeated string target = 4;\n\n  // Unique identifier for this request. Every PartialRunSetupRequest must\n  // have a unique request_id, and retried PartialRunSetupRequest must have\n  // the same request_id. If request_id is zero, retry detection is disabled.\n  int64 request_id = 5;\n}\n\nmessage PartialRunSetupResponse {\n  // The unique handle corresponding to the ongoing partial run call setup by\n  // the invocation to PartialRunSetup. This handle may be passed to\n  // RunStepRequest to send and receive tensors for this partial run.\n  string partial_run_handle = 1;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// CloseSession method request/response protos.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage CloseSessionRequest {\n  // REQUIRED: session_handle must be returned by a CreateSession call\n  // to the same master service.\n  string session_handle = 1;\n}\n\nmessage CloseSessionResponse {}\n\n// Reset() allows misbehaving or slow sessions to be aborted and closed, and\n// causes their resources eventually to be released.  Reset() does not wait\n// for the computations in old sessions to cease; it merely starts the\n// process of tearing them down.  However, if a new session is started after\n// a Reset(), the new session is isolated from changes that old sessions\n// (started prior to the Reset()) may continue to make to resources, provided\n// all those resources are in containers listed in \"containers\".\n//\n// Old sessions may continue to have side-effects on resources not in\n// containers listed in \"containers\", and thus may affect future\n// sessions' results in ways that are hard to predict.  Thus, if well-defined\n// behavior is desired, is it recommended that all containers be listed in\n// \"containers\".  Similarly, if a device_filter is specified, results may be\n// hard to predict.\nmessage ResetRequest {\n  // A list of container names, which may be empty.\n  //\n  // If 'container' is not empty, releases resources in the given\n  // containers in all devices.\n  //\n  // If 'container' is empty, releases resources in the default\n  // container in all devices.\n  repeated string container = 1;\n\n  // When any filters are present, only devices that match the filters\n  // will be reset. Each filter can be partially specified,\n  // e.g. \"/job:ps\" \"/job:worker/replica:3\", etc.\n  repeated string device_filters = 2;\n}\n\nmessage ResetResponse {}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// ListDevices method request/response protos.\n//\n// Returns information about the TensorFlow devices that are available\n// to this master.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage ListDevicesRequest {\n  // Optional: session_handle must be returned by a CreateSession call to the\n  // same master service.\n  //\n  // When session_handle is empty, the ClusterSpec provided when the master was\n  // started is used to compute the available devices. If the session_handle is\n  // provided but not recognized, an error is returned. Finally, if a valid\n  // session_handle is provided, the cluster configuration for that session is\n  // used when computing the response.\n  string session_handle = 1;\n}\n\nmessage ListDevicesResponse {\n  repeated DeviceAttributes local_device = 1;\n  repeated DeviceAttributes remote_device = 2;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// MakeCallable method request/response protos.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage MakeCallableRequest {\n  // REQUIRED: session_handle must be returned by a CreateSession call\n  // to the same master service.\n  string session_handle = 1;\n\n  // Options that define the behavior of the created callable.\n  CallableOptions options = 2;\n\n  // Unique identifier for this request. Every MakeCallableRequest must\n  // have a unique request_id, and retried MakeCallableRequest must have\n  // the same request_id. If request_id is zero, retry detection is disabled.\n  int64 request_id = 3;\n}\n\nmessage MakeCallableResponse {\n  // A handle to the created callable.\n  int64 handle = 1;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// RunCallable method request/response protos.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage RunCallableRequest {\n  // REQUIRED: session_handle must be returned by a CreateSession call\n  // to the same master service.\n  string session_handle = 1;\n  // REQUIRED: handle must be returned by a MakeCallable call to the same\n  // master service.\n  int64 handle = 2;\n\n  // Values of the tensors passed as arguments to the callable, in the order\n  // defined in the CallableOptions.feed field passed to MakeCallable.\n  repeated TensorProto feed = 3;\n\n  // Unique identifier for this request. Every RunCallableRequest must\n  // have a unique request_id, and retried RunCallableRequest must have\n  // the same request_id. If request_id is zero, retry detection is disabled.\n  int64 request_id = 4;\n}\n\nmessage RunCallableResponse {\n  // Values of the tensors returned by the callable, in the order defined in the\n  // CallableOptions.fetch field passed to MakeCallable.\n  repeated TensorProto fetch = 1;\n\n  // Returned metadata if requested in the options.\n  RunMetadata metadata = 2;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// ReleaseCallable method request/response protos.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage ReleaseCallableRequest {\n  // REQUIRED: session_handle must be returned by a CreateSession call\n  // to the same master service.\n  string session_handle = 1;\n\n  // REQUIRED: handle must be returned by a MakeCallable call to the same\n  // master service.\n  int64 handle = 2;\n}\n\nmessage ReleaseCallableResponse {}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/master_service.proto",
    "content": "/* Copyright 2016 The TensorFlow Authors. All Rights Reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n==============================================================================*/\n\nsyntax = \"proto3\";\n\npackage tensorflow.grpc;\n\nimport \"tensorflow/core/protobuf/master.proto\";\n\noption java_outer_classname = \"MasterServiceProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.distruntime\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// MasterService defines a TensorFlow service with which a client can\n// interact to execute a distributed TensorFlow computation.\n//\n// A master service keeps track of multiple \"master sessions\". Each\n// session encapsulates a computation graph and its associated state,\n// and typically corresponds to a single \"client session\" (e.g. a\n// `tensorflow::Session` instance).\n//\n// A session is responsible for the following:\n// * assigning each node to a device (locally or remotely) using a\n//   placement algorithm. This may make decisions based on collected\n//   statistics from the workers in the system (e.g., memory usage,\n//   bandwidth consumption, etc.)\n//\n// * inserting intermediate nodes and edges to support cross-device\n//   and cross-process data flows and resource management.\n//\n// * issuing commands to workers to execute the subgraphs associated\n//   with those workers.\n//\n// Typically, a client carries out an iterative computation\n// (e.g. training) by invoking RPCs against the master in a\n// client-side loop. The client first creates a client session that\n// connects to a particular master (using gRPC for example). The\n// master creates a corresponding master session that is hosted on\n// the master and caches state between the client's invocations.\n//\n// After the session is established, the master returns an opaque\n// handle to the client that can be used to associate the client and\n// master sessions.\n//\n// The client may send an initial graph to the master in the\n// CreateSession call, and add nodes to the graph using ExtendSession.\n//\n// The most frequent operation a master is \"RunStep\", which implements\n// the `Session::Run()` API. It supports feeding in arguments,\n// executing a dataflow computation, and fetching arguments.\n//\n// Finally, when the client no longer needs the session, it should\n// close the session by invoking CloseSession, which allows the master\n// to reclaim resources associated with the session. The master may\n// implement a garbage collection scheme that closes sessions that\n// have been inactive for some time.\n//\n// For example, the following pseudo-code illustrates how a client\n// interacts with a master:\n//\n// stub = NewStub(\"/job:mnist/replica:0/task:0\")\n// {handle} = stub->CreateSession({graph_def})\n// do {\n//   stub->RunStep({handle, {feeds}, {fetches}})\n//   // The client can evaluate a predicate locally, based on the\n//   // result of `fetches`, to determine whether to terminate. For\n//   // example, it might fetch the loss and evaluate whether it is less\n//   // than some threshold.\n// } while (!should_stop({fetches}));\n// stub->CloseSession({handle})\n//\n////////////////////////////////////////////////////////////////////////////////\n\nservice MasterService {\n  // Creates a session.\n  rpc CreateSession(CreateSessionRequest) returns (CreateSessionResponse);\n\n  // Extends a session.\n  rpc ExtendSession(ExtendSessionRequest) returns (ExtendSessionResponse);\n\n  // Prepares future partial run calls.\n  rpc PartialRunSetup(PartialRunSetupRequest) returns (PartialRunSetupResponse);\n\n  // Drives the graph computation.\n  rpc RunStep(RunStepRequest) returns (RunStepResponse);\n\n  // Closes a session.\n  rpc CloseSession(CloseSessionRequest) returns (CloseSessionResponse);\n\n  // List the devices usable by the master.\n  rpc ListDevices(ListDevicesRequest) returns (ListDevicesResponse);\n\n  // Close and abandon all existing sessions.  Ongoing computations\n  // will no longer affect fresh ones via the resources in containers listed in\n  // the ResetRequest.  See ResetRequest for more details.\n  rpc Reset(ResetRequest) returns (ResetResponse);\n\n  // Registers a callable for execution with RunCallable.\n  rpc MakeCallable(MakeCallableRequest) returns (MakeCallableResponse);\n\n  // Executes a callable registered with MakeCallable.\n  rpc RunCallable(RunCallableRequest) returns (RunCallableResponse);\n\n  // Frees resources associated with a callable registered with MakeCallable.\n  rpc ReleaseCallable(ReleaseCallableRequest) returns (ReleaseCallableResponse);\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/meta_graph.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"google/protobuf/any.proto\";\nimport \"tensorflow/core/framework/graph.proto\";\nimport \"tensorflow/core/framework/op_def.proto\";\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/types.proto\";\nimport \"tensorflow/core/protobuf/saved_object_graph.proto\";\nimport \"tensorflow/core/protobuf/saver.proto\";\nimport \"tensorflow/core/protobuf/struct.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"MetaGraphProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// NOTE: This protocol buffer is evolving, and will go through revisions in the\n// coming months.\n//\n// Protocol buffer containing the following which are necessary to restart\n// training, run inference. It can be used to serialize/de-serialize memory\n// objects necessary for running computation in a graph when crossing the\n// process boundary. It can be used for long term storage of graphs,\n// cross-language execution of graphs, etc.\n//   MetaInfoDef\n//   GraphDef\n//   SaverDef\n//   CollectionDef\n//   TensorInfo\n//   SignatureDef\nmessage MetaGraphDef {\n  // Meta information regarding the graph to be exported.  To be used by users\n  // of this protocol buffer to encode information regarding their meta graph.\n  message MetaInfoDef {\n    // User specified Version string. Can be the name of the model and revision,\n    // steps this model has been trained to, etc.\n    string meta_graph_version = 1;\n\n    // A copy of the OpDefs used by the producer of this graph_def.\n    // Descriptions and Ops not used in graph_def are stripped out.\n    OpList stripped_op_list = 2;\n\n    // A serialized protobuf. Can be the time this meta graph is created, or\n    // modified, or name of the model.\n    google.protobuf.Any any_info = 3;\n\n    // User supplied tag(s) on the meta_graph and included graph_def.\n    //\n    // MetaGraphDefs should be tagged with their capabilities or use-cases.\n    // Examples: \"train\", \"serve\", \"gpu\", \"tpu\", etc.\n    // These tags enable loaders to access the MetaGraph(s) appropriate for a\n    // specific use-case or runtime environment.\n    repeated string tags = 4;\n\n    // The __version__ string of the tensorflow build used to write this graph.\n    // This will be populated by the framework, which will overwrite any user\n    // supplied value.\n    string tensorflow_version = 5;\n\n    // The __git_version__ string of the tensorflow build used to write this\n    // graph. This will be populated by the framework, which will overwrite any\n    // user supplied value.\n    string tensorflow_git_version = 6;\n\n    // A flag to denote whether default-valued attrs have been stripped from\n    // the nodes in this graph_def.\n    bool stripped_default_attrs = 7;\n\n    // FunctionDef name to aliases mapping.\n    map<string, string> function_aliases = 8;\n  }\n  MetaInfoDef meta_info_def = 1;\n\n  // GraphDef.\n  GraphDef graph_def = 2;\n\n  // SaverDef.\n  SaverDef saver_def = 3;\n\n  // collection_def: Map from collection name to collections.\n  // See CollectionDef section for details.\n  map<string, CollectionDef> collection_def = 4;\n\n  // signature_def: Map from user supplied key for a signature to a single\n  // SignatureDef.\n  map<string, SignatureDef> signature_def = 5;\n\n  // Asset file def to be used with the defined graph.\n  repeated AssetFileDef asset_file_def = 6;\n\n  // Extra information about the structure of functions and stateful objects.\n  SavedObjectGraph object_graph_def = 7;\n}\n\n// CollectionDef should cover most collections.\n// To add a user-defined collection, do one of the following:\n// 1. For simple data types, such as string, int, float:\n//      tf.add_to_collection(\"your_collection_name\", your_simple_value)\n//    strings will be stored as bytes_list.\n//\n// 2. For Protobuf types, there are three ways to add them:\n//    1) tf.add_to_collection(\"your_collection_name\",\n//         your_proto.SerializeToString())\n//\n//       collection_def {\n//         key: \"user_defined_bytes_collection\"\n//         value {\n//           bytes_list {\n//             value: \"queue_name: \\\"test_queue\\\"\\n\"\n//           }\n//         }\n//       }\n//\n//  or\n//\n//    2) tf.add_to_collection(\"your_collection_name\", str(your_proto))\n//\n//       collection_def {\n//         key: \"user_defined_string_collection\"\n//         value {\n//          bytes_list {\n//             value: \"\\n\\ntest_queue\"\n//           }\n//         }\n//       }\n//\n//  or\n//\n//    3) any_buf = any_pb2.Any()\n//       tf.add_to_collection(\"your_collection_name\",\n//         any_buf.Pack(your_proto))\n//\n//       collection_def {\n//         key: \"user_defined_any_collection\"\n//         value {\n//           any_list {\n//             value {\n//               type_url: \"type.googleapis.com/tensorflow.QueueRunnerDef\"\n//               value: \"\\n\\ntest_queue\"\n//             }\n//           }\n//         }\n//       }\n//\n// 3. For Python objects, implement to_proto() and from_proto(), and register\n//    them in the following manner:\n//    ops.register_proto_function(\"your_collection_name\",\n//                                proto_type,\n//                                to_proto=YourPythonObject.to_proto,\n//                                from_proto=YourPythonObject.from_proto)\n//    These functions will be invoked to serialize and de-serialize the\n//    collection. For example,\n//    ops.register_proto_function(ops.GraphKeys.GLOBAL_VARIABLES,\n//                                proto_type=variable_pb2.VariableDef,\n//                                to_proto=Variable.to_proto,\n//                                from_proto=Variable.from_proto)\nmessage CollectionDef {\n  // NodeList is used for collecting nodes in graph. For example\n  // collection_def {\n  //   key: \"summaries\"\n  //   value {\n  //     node_list {\n  //       value: \"input_producer/ScalarSummary:0\"\n  //       value: \"shuffle_batch/ScalarSummary:0\"\n  //       value: \"ImageSummary:0\"\n  //     }\n  //   }\n  message NodeList {\n    repeated string value = 1;\n  }\n\n  // BytesList is used for collecting strings and serialized protobufs. For\n  // example:\n  // collection_def {\n  //   key: \"trainable_variables\"\n  //   value {\n  //     bytes_list {\n  //       value: \"\\n\\017conv1/weights:0\\022\\024conv1/weights/Assign\n  //              \\032\\024conv1/weights/read:0\"\n  //       value: \"\\n\\016conv1/biases:0\\022\\023conv1/biases/Assign\\032\n  //              \\023conv1/biases/read:0\"\n  //     }\n  //   }\n  // }\n  message BytesList {\n    repeated bytes value = 1;\n  }\n\n  // Int64List is used for collecting int, int64 and long values.\n  message Int64List {\n    repeated int64 value = 1 [packed = true];\n  }\n\n  // FloatList is used for collecting float values.\n  message FloatList {\n    repeated float value = 1 [packed = true];\n  }\n\n  // AnyList is used for collecting Any protos.\n  message AnyList {\n    repeated google.protobuf.Any value = 1;\n  }\n\n  oneof kind {\n    NodeList node_list = 1;\n    BytesList bytes_list = 2;\n    Int64List int64_list = 3;\n    FloatList float_list = 4;\n    AnyList any_list = 5;\n  }\n}\n\n// Information about a Tensor necessary for feeding or retrieval.\nmessage TensorInfo {\n  // For sparse tensors, The COO encoding stores a triple of values, indices,\n  // and shape.\n  message CooSparse {\n    // The shape of the values Tensor is [?].  Its dtype must be the dtype of\n    // the SparseTensor as a whole, given in the enclosing TensorInfo.\n    string values_tensor_name = 1;\n\n    // The indices Tensor must have dtype int64 and shape [?, ?].\n    string indices_tensor_name = 2;\n\n    // The dynamic logical shape represented by the SparseTensor is recorded in\n    // the Tensor referenced here.  It must have dtype int64 and shape [?].\n    string dense_shape_tensor_name = 3;\n  }\n\n  // Generic encoding for composite tensors.\n  message CompositeTensor {\n    // The serialized TypeSpec for the composite tensor.\n    TypeSpecProto type_spec = 1;\n\n    // A TensorInfo for each flattened component tensor.\n    repeated TensorInfo components = 2;\n  }\n\n  oneof encoding {\n    // For dense `Tensor`s, the name of the tensor in the graph.\n    string name = 1;\n    // There are many possible encodings of sparse matrices\n    // (https://en.wikipedia.org/wiki/Sparse_matrix).  Currently, TensorFlow\n    // uses only the COO encoding.  This is supported and documented in the\n    // SparseTensor Python class.\n    CooSparse coo_sparse = 4;\n    // Generic encoding for CompositeTensors.\n    CompositeTensor composite_tensor = 5;\n  }\n  DataType dtype = 2;\n  // The static shape should be recorded here, to the extent that it can\n  // be known in advance.  In the case of a SparseTensor, this field describes\n  // the logical shape of the represented tensor (aka dense_shape).\n  TensorShapeProto tensor_shape = 3;\n}\n\n// SignatureDef defines the signature of a computation supported by a TensorFlow\n// graph.\n//\n// For example, a model with two loss computations, sharing a single input,\n// might have the following signature_def map, in a MetaGraphDef message.\n//\n// Note that across the two SignatureDefs \"loss_A\" and \"loss_B\", the input key,\n// output key, and method_name are identical, and will be used by system(s) that\n// implement or rely upon this particular loss method. The output tensor names\n// differ, demonstrating how different outputs can exist for the same method.\n//\n// signature_def {\n//   key: \"loss_A\"\n//   value {\n//     inputs {\n//       key: \"input\"\n//       value {\n//         name: \"input:0\"\n//         dtype: DT_STRING\n//         tensor_shape: ...\n//       }\n//     }\n//     outputs {\n//       key: \"loss_output\"\n//       value {\n//         name: \"loss_output_A:0\"\n//         dtype: DT_FLOAT\n//         tensor_shape: ...\n//       }\n//     }\n//     method_name: \"some/package/compute_loss\"\n//   }\n//   ...\n// }\n// signature_def {\n//   key: \"loss_B\"\n//   value {\n//     inputs {\n//       key: \"input\"\n//       value {\n//         name: \"input:0\"\n//         dtype: DT_STRING\n//         tensor_shape: ...\n//       }\n//     }\n//     outputs {\n//       key: \"loss_output\"\n//       value {\n//         name: \"loss_output_B:0\"\n//         dtype: DT_FLOAT\n//         tensor_shape: ...\n//       }\n//     }\n//     method_name: \"some/package/compute_loss\"\n//   }\n//   ...\n// }\nmessage SignatureDef {\n  // Named input parameters.\n  map<string, TensorInfo> inputs = 1;\n  // Named output parameters.\n  map<string, TensorInfo> outputs = 2;\n  // Extensible method_name information enabling third-party users to mark a\n  // SignatureDef as supporting a particular method. This enables producers and\n  // consumers of SignatureDefs, e.g. a model definition library and a serving\n  // library to have a clear hand-off regarding the semantics of a computation.\n  //\n  // Note that multiple SignatureDefs in a single MetaGraphDef may have the same\n  // method_name. This is commonly used to support multi-headed computation,\n  // where a single graph computation may return multiple results.\n  string method_name = 3;\n}\n\n// An asset file def for a single file or a set of sharded files with the same\n// name.\nmessage AssetFileDef {\n  // The tensor to bind the asset filename to.\n  TensorInfo tensor_info = 1;\n  // The filename within an assets directory. Note: does not include the path\n  // prefix, i.e. directories. For an asset at /tmp/path/vocab.txt, the filename\n  // would be \"vocab.txt\".\n  string filename = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/named_tensor.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/tensor.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"NamedTensorProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// A pair of tensor name and tensor values.\nmessage NamedTensorProto {\n  // Name of the tensor.\n  string name = 1;\n\n  // The client can populate a TensorProto using a tensorflow::Tensor`, or\n  // directly using the protobuf field accessors.\n  //\n  // The client specifies whether the returned tensor values should be\n  // filled tensor fields (float_val, int_val, etc.) or encoded in a\n  // compact form in tensor.tensor_content.\n  TensorProto tensor = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/queue_runner.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/protobuf/error_codes.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"QueueRunnerProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Protocol buffer representing a QueueRunner.\nmessage QueueRunnerDef {\n  // Queue name.\n  string queue_name = 1;\n\n  // A list of enqueue operations.\n  repeated string enqueue_op_name = 2;\n\n  // The operation to run to close the queue.\n  string close_op_name = 3;\n\n  // The operation to run to cancel the queue.\n  string cancel_op_name = 4;\n\n  // A list of exception types considered to signal a safely closed queue\n  // if raised during enqueue operations.\n  repeated error.Code queue_closed_exception_types = 5;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/remote_tensor_handle.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.eager;\n\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/types.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"RemoteTensorHandleProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\nmessage ResourceDtypeAndShape {\n  DataType dtype = 1;\n  TensorShapeProto shape = 2;\n}\n\nmessage RemoteTensorHandle {\n  // The ID of the operation that produced this tensor.\n  int64 op_id = 1;\n  // The index into the outputs of the operation that produced this tensor.\n  int32 output_num = 2;\n  // Device where the tensor is located. Cannot be empty.\n  // For multi-device functions, it's the default device passed to placer.\n  string device = 3;\n  // Device of the operation producing this tensor. Can be empty if the\n  // operation producing this tensor is a multi-device function.\n  string op_device = 4;\n  // Tensor type.\n  DataType dtype = 5;\n  // Optional data types and shapes of a remote resource variable.\n  repeated ResourceDtypeAndShape resource_dtypes_and_shapes = 6;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/replay_log.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/protobuf/master.proto\";\n\noption cc_enable_arenas = true;\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Records the creation of a new replay session.  We record the device listing\n// here to capture the state of the cluster.\nmessage NewReplaySession {\n  ListDevicesResponse devices = 1;\n  string session_handle = 2;\n}\n\nmessage ReplayOp {\n  double start_time_us = 31;\n  double end_time_us = 32;\n\n  oneof op {\n    CreateSessionRequest create_session = 1;\n    ExtendSessionRequest extend_session = 2;\n    PartialRunSetupRequest partial_run_setup = 3;\n    RunStepRequest run_step = 4;\n    CloseSessionRequest close_session = 5;\n    ListDevicesRequest list_devices = 6;\n    ResetRequest reset_request = 7;\n    MakeCallableRequest make_callable = 8;\n    RunCallableRequest run_callable = 9;\n    ReleaseCallableRequest release_callable = 10;\n    NewReplaySession new_replay_session = 11;\n  }\n\n  oneof response {\n    CreateSessionResponse create_session_response = 21;\n    ExtendSessionResponse extend_session_response = 22;\n    PartialRunSetupResponse partial_run_setup_response = 23;\n    RunStepResponse run_step_response = 24;\n    CloseSessionResponse close_session_response = 25;\n    ListDevicesResponse list_devices_response = 26;\n    ResetResponse reset_request_response = 27;\n    MakeCallableResponse make_callable_response = 28;\n    RunCallableResponse run_callable_response = 29;\n    ReleaseCallableResponse release_callable_response = 30;\n  }\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/rewriter_config.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/attr_value.proto\";\nimport \"tensorflow/core/protobuf/verifier_config.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"RewriterConfigProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\nmessage AutoParallelOptions {\n  bool enable = 1;\n  int32 num_replicas = 2;\n}\n\nmessage ScopedAllocatorOptions {\n  // If present, only perform optimization for these ops.\n  repeated string enable_op = 1;\n}\n\nmessage RewriterConfig {\n  // Graph rewriting is experimental and subject to change, not covered by any\n  // API stability guarantees.\n\n  // Configuration options for the meta-optimizer. Unless otherwise noted, these\n  // configuration options do not apply to explicitly triggered optimization\n  // passes in the optimizers field.\n\n  enum Toggle {\n    DEFAULT = 0;\n    ON = 1;\n    OFF = 2;\n    // Enable some aggressive optimizations that use assumptions that TF graphs\n    // may break. For example, assume the shape of a placeholder matches its\n    // actual feed.\n    AGGRESSIVE = 3;\n  }\n\n  // Enum for layout conversion between NCHW and NHWC on CPU. Default is OFF.\n  enum CpuLayout {\n    NO_CONVERSION_ON_CPU = 0;\n    NCHW_TO_NHWC = 1;\n    NHWC_TO_NCHW = 2;\n  }\n\n  // Enum controlling the number of times to run optimizers. The default is to\n  // run them twice.\n  enum NumIterationsType {\n    DEFAULT_NUM_ITERS = 0;\n    ONE = 1;\n    TWO = 2;\n  }\n\n  // CPU Conversion settings between NHCW and NCHW.\n  CpuLayout cpu_layout_conversion = 50;\n\n  // Optimize tensor layouts (default is ON)\n  // e.g. This will try to use NCHW layout on GPU which is faster.\n  Toggle layout_optimizer = 1;\n  // Fold constants (default is ON)\n  // Statically infer the value of tensors when possible, and materialize the\n  // result using constants.\n  Toggle constant_folding = 3;\n  // Shape optimizations (default is ON)\n  // Simplify computations made on shapes.\n  Toggle shape_optimization = 13;\n  // Remapping (default is ON)\n  // Remap subgraphs onto more efficient implementations.\n  Toggle remapping = 14;\n  // Common subgraph elimination (default is ON)\n  // e.g. Simplify arithmetic ops; merge ops with same value (like constants).\n  Toggle common_subgraph_elimination = 24;\n  // Arithmetic optimizations (default is ON)\n  // e.g. Simplify arithmetic ops; merge ops with same value (like constants).\n  Toggle arithmetic_optimization = 7;\n  // Control dependency optimizations (default is ON).\n  // Remove redundant control dependencies, which may enable other optimization.\n  Toggle dependency_optimization = 8;\n  // Loop optimizations (default is ON).\n  Toggle loop_optimization = 9;\n  // Function optimizations (default is ON).\n  Toggle function_optimization = 10;\n  // Strips debug-related nodes from the graph (off by default).\n  Toggle debug_stripper = 11;\n  // If true, don't remove unnecessary ops from the graph\n  bool disable_model_pruning = 2;\n  // Try to allocate some independent Op outputs contiguously in order to\n  // merge or eliminate downstream Ops (off by default).\n  Toggle scoped_allocator_optimization = 15;\n  // Force small ops onto the CPU (default is OFF).\n  Toggle pin_to_host_optimization = 18;\n  // Enable the swap of kernel implementations based on the device placement\n  // (default is ON).\n  Toggle implementation_selector = 22;\n  // Optimize data types for CUDA (default is OFF).\n  // This will try to use float16 on GPU which is faster.\n  // Note that this can change the numerical stability of the graph and may\n  // require the use of loss scaling to maintain model convergence.\n  Toggle auto_mixed_precision = 23;\n  // Optimize data types for MKL (default is OFF).\n  // This will try to use bfloat16 on CPUs, which is faster.\n  // Note that this can change the numerical stability of the graph.\n  Toggle auto_mixed_precision_mkl = 25;\n  // Emulate a model using data type float16 on CPU (default is OFF).\n  // This will try to emulate the float16 inputs and outputs of an operator\n  // on CPU to have better correlation with float16 on GPU; however the\n  // computation in the operator is based on float32.\n  // Note that this can change the numerical stability of the graph.\n  Toggle auto_mixed_precision_cpu = 29;\n  // Disable the entire meta optimizer (off by default).\n  bool disable_meta_optimizer = 19;\n  // Optimizers registered by plugin (default is ON)\n  Toggle use_plugin_optimizers = 28;\n\n  // Controls how many times we run the optimizers in meta optimizer (default\n  // is once).\n  NumIterationsType meta_optimizer_iterations = 12;\n\n  // The minimum number of nodes in a graph to optimizer. For smaller graphs,\n  // optimization is skipped.\n  // 0 means the system picks an appropriate number.\n  // < 0 means do not skip optimization.\n  int32 min_graph_nodes = 17;\n\n  // Disable optimizations that assume compressed tensors. Note that this flag\n  // is experimental and may be removed in the future.\n  bool experimental_disable_compressed_tensor_optimization = 26;\n\n  // Disable folding quantization emulation ops such as FakeQuantWithMinMax* and\n  // QuantizeAndDequantize*. Some compilers (e.g. the TF-to-tflite converter)\n  // have to extract quantization configs (e.g. min/max range, number of bits,\n  // and per-channel) from the quantization emulation ops. Note that this flag\n  // is experimental and may be removed in the future. See b/174138564 for more\n  // details.\n  bool experimental_disable_folding_quantization_emulation = 27;\n\n  enum MemOptType {\n    // The default setting (SCHEDULING and SWAPPING HEURISTICS only)\n    DEFAULT_MEM_OPT = 0;\n    // Disabled in the meta-optimizer.\n    NO_MEM_OPT = 1;\n    // Driven by manual op-level annotations.\n    MANUAL = 2;\n\n    // Driven by heuristics. The behavior of these heuristics is subject to\n    // change. Currently includes an experimental recomputation and swapping\n    // heuristics. Manual annotations are respected, but additional nodes are\n    // selected automatically.\n\n    // Swapping heuristic will move a tensor from the GPU to the CPU and move\n    // it back when needed to reduce peak memory usage.\n    SWAPPING_HEURISTICS = 4;\n    // Recomputation heuristics will recompute ops (such as Relu activation)\n    // during backprop instead of storing them, reducing peak memory usage.\n    RECOMPUTATION_HEURISTICS = 5;\n    // Scheduling will split big ops such as AddN and try to enforce a schedule\n    // of the new computations that decreases peak memory usage.\n    SCHEDULING_HEURISTICS = 6;\n    // Use any combination of swapping and recomputation heuristics.\n    HEURISTICS = 3;\n  }\n  // Configures memory optimization passes through the meta-optimizer. Has no\n  // effect on manually requested memory optimization passes in the optimizers\n  // field.\n  MemOptType memory_optimization = 4;\n  // A node name scope for node names which are valid outputs of recomputations.\n  // Inputs to nodes that match this scope may be recomputed (subject either to\n  // manual annotation of those input nodes or to manual annotation and\n  // heuristics depending on memory_optimization), but the nodes themselves will\n  // not be recomputed. This matches any sub-scopes as well, meaning the scope\n  // can appear not just as a top-level scope. For example, if the value is\n  // \"gradients/\", the default, it will match node name \"gradients/foo\",\n  // \"foo/gradients/bar\", but not \"foo_gradients/\"\n  string memory_optimizer_target_node_name_scope = 6;\n  // Maximum number of milliseconds to spend optimizing a single graph before\n  // timing out. If less than or equal to 0 (default value) the optimizer will\n  // never time out.\n  int64 meta_optimizer_timeout_ms = 20;\n\n  // Configures AutoParallel optimization passes either through the\n  // meta-optimizer or when manually specified through the optimizers field.\n  AutoParallelOptions auto_parallel = 5;\n\n  // If true, any optimization pass failing will cause the MetaOptimizer to\n  // stop with an error. By default - or when set to false, failing passes are\n  // skipped silently.\n  bool fail_on_optimizer_errors = 21;\n\n  ScopedAllocatorOptions scoped_allocator_opts = 16;\n\n  // If non-empty, will use this as an alternative way to specify a list of\n  // optimizations to turn on and the order of the optimizations (replacing the\n  // meta-optimizer).\n  //\n  // Of the RewriterConfig options, only the AutoParallel configuration options\n  // (the auto_parallel field) apply to manually requested optimization passes\n  // (\"autoparallel\"). Memory optimization passes (\"memory\") invoked here are\n  // not configurable (in contrast to memory optimization passes through the\n  // meta-optimizer) and act only on manual op annotations.\n  //\n  // Custom optimizers (see custom_optimizers) that are not part of this\n  // schedule will be run after - in the order that they were specified.\n  repeated string optimizers = 100;\n\n  // Message to describe custom graph optimizer and its parameters\n  message CustomGraphOptimizer {\n    string name = 1;\n    map<string, AttrValue> parameter_map = 2;\n  }\n\n  // list of CustomGraphOptimizers to apply.\n  repeated CustomGraphOptimizer custom_optimizers = 200;\n\n  // VerifierConfig specifying the verifiers to be run after every optimizer.\n  VerifierConfig inter_optimizer_verifier_config = 300;\n\n  // VerifierConfig specifying the verifiers to be run at the end, after all\n  // optimizers have run.\n  VerifierConfig post_optimization_verifier_config = 301;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/saved_model.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/protobuf/meta_graph.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"SavedModelProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// SavedModel is the high level serialization format for TensorFlow Models.\n// See [todo: doc links, similar to session_bundle] for more information.\nmessage SavedModel {\n  // The schema version of the SavedModel instance. Used for versioning when\n  // making future changes to the specification/implementation. Initial value\n  // at release will be 1.\n  int64 saved_model_schema_version = 1;\n\n  // One or more MetaGraphs.\n  repeated MetaGraphDef meta_graphs = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/saved_object_graph.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"google/protobuf/any.proto\";\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/types.proto\";\nimport \"tensorflow/core/framework/variable.proto\";\nimport \"tensorflow/core/framework/versions.proto\";\nimport \"tensorflow/core/protobuf/struct.proto\";\nimport \"tensorflow/core/protobuf/trackable_object_graph.proto\";\n\noption cc_enable_arenas = true;\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// A SavedObjectGraph is part of object-based SavedModels in TF 2.0. It\n// describes the directed graph of Python objects (or equivalent in other\n// languages) that make up a model, with nodes[0] at the root.\n\n// SavedObjectGraph shares some structure with TrackableObjectGraph, but\n// SavedObjectGraph belongs to the MetaGraph and contains pointers to functions\n// and type information, while TrackableObjectGraph lives in the checkpoint\n// and contains pointers only to variable values.\n\nmessage SavedObjectGraph {\n  // Flattened list of objects in the object graph.\n  //\n  // The position of the object in this list indicates its id.\n  // Nodes[0] is considered the root node.\n  repeated SavedObject nodes = 1;\n\n  // Information about captures and output structures in concrete functions.\n  // Referenced from SavedBareConcreteFunction and SavedFunction.\n  map<string, SavedConcreteFunction> concrete_functions = 2;\n}\n\nmessage SavedObject {\n  // Objects which this object depends on: named edges in the dependency\n  // graph.\n  //\n  // Note: All kinds of SavedObject may have children, except\n  // \"constant\" and \"captured_tensor\".\n  repeated TrackableObjectGraph.TrackableObject.ObjectReference children = 1;\n\n  // Ordered list of dependencies that must be loaded before this object.\n  // SavedModel loads with the bottom-up approach, by first creating all objects\n  // (in the order defined by the dependencies), then connecting the edges.\n  repeated TrackableObjectGraph.TrackableObject.ObjectReference dependencies =\n      15;\n\n  // Removed when forking SavedObject from TrackableObjectGraph.\n  reserved \"attributes\";\n  reserved 2;\n\n  // Slot variables owned by this object. This describes the three-way\n  // (optimizer, variable, slot variable) relationship; none of the three\n  // depend on the others directly.\n  //\n  // Note: currently only valid if kind == \"user_object\".\n  repeated TrackableObjectGraph.TrackableObject.SlotVariableReference\n      slot_variables = 3;\n\n  oneof kind {\n    SavedUserObject user_object = 4;\n    SavedAsset asset = 5;\n    SavedFunction function = 6;\n    SavedVariable variable = 7;\n    SavedBareConcreteFunction bare_concrete_function = 8;\n    SavedConstant constant = 9;\n    SavedResource resource = 10;\n    CapturedTensor captured_tensor = 12;\n  }\n\n  // Stores the functions used to save and restore this object. At most one of\n  // `saveable_objects` or `registered_saver` is defined for each SavedObject.\n  // See the comment below for the difference between SaveableObject and\n  // registered savers.\n  map<string, SaveableObject> saveable_objects = 11;\n\n  // The fields below are filled when the user serializes a registered Trackable\n  // class or an object with a registered saver function.\n  //\n  // Registered classes may save additional metadata and supersede the\n  // default loading process where nodes are recreated from the proto.\n  // If the registered class cannot be found, then the object will load as one\n  // one of the default trackable objects: Autotrackable (a class similar to\n  // tf.Module), tf.function, or tf.Variable.\n  //\n  // Unlike SaveableObjects, which store the functions for saving and restoring\n  // from tensors, registered savers allow Trackables to write checkpoint shards\n  // directly (e.g. for performance or coordination reasons).\n  // *All registered savers must be available when loading the SavedModel.*\n\n  // The name of the registered class of the form \"{package}.{class_name}\".\n  // This field is used to search for the registered class at loading time.\n  string registered_name = 13;\n  // The user-generated proto storing metadata for this object, to be passed to\n  // the registered classes's _deserialize_from_proto method when this object is\n  // loaded from the SavedModel.\n  google.protobuf.Any serialized_user_proto = 14;\n\n  // String name of the registered saver. At most one of `saveable_objects` or\n  // `registered_saver` is defined for each SavedObject.\n  string registered_saver = 16;\n}\n\n// A SavedUserObject is an object (in the object-oriented language of the\n// TensorFlow program) of some user- or framework-defined class other than\n// those handled specifically by the other kinds of SavedObjects.\n//\n// This object cannot be evaluated as a tensor, and therefore cannot be bound\n// to an input of a function.\nmessage SavedUserObject {\n  // Corresponds to a registration of the type to use in the loading program.\n  string identifier = 1;\n  // Version information from the producer of this SavedUserObject.\n  VersionDef version = 2;\n  // Metadata for deserializing this object.\n  //\n  // Deprecated! At the time of deprecation, Keras was the only user of this\n  // field, and its saving and loading code will be updated shortly.\n  // Please save your application-specific metadata to a separate file.\n  string metadata = 3 [deprecated = true];\n}\n\n// A SavedAsset points to an asset in the MetaGraph.\n//\n// When bound to a function this object evaluates to a tensor with the absolute\n// filename. Users should not depend on a particular part of the filename to\n// remain stable (e.g. basename could be changed).\nmessage SavedAsset {\n  // Index into `MetaGraphDef.asset_file_def[]` that describes the Asset.\n  //\n  // Only the field `AssetFileDef.filename` is used. Other fields, such as\n  // `AssetFileDef.tensor_info`, MUST be ignored.\n  int32 asset_file_def_index = 1;\n}\n\n// A function with multiple signatures, possibly with non-Tensor arguments.\nmessage SavedFunction {\n  repeated string concrete_functions = 1;\n  FunctionSpec function_spec = 2;\n}\n\nmessage CapturedTensor {\n  // Name of captured tensor\n  string name = 1;\n\n  // Name of concrete function which contains the computed graph tensor.\n  string concrete_function = 2;\n}\n\n// Stores low-level information about a concrete function. Referenced in either\n// a SavedFunction or a SavedBareConcreteFunction.\nmessage SavedConcreteFunction {\n  repeated int32 bound_inputs = 2;\n\n  // Input in canonicalized form that was received to create this concrete\n  // function.\n  StructuredValue canonicalized_input_signature = 3;\n  // Output that was the return value of this function after replacing all\n  // Tensors with TensorSpecs. This can be an arbitrary nested function and will\n  // be used to reconstruct the full structure from pure tensors.\n  StructuredValue output_signature = 4;\n}\n\nmessage SavedBareConcreteFunction {\n  // Identifies a SavedConcreteFunction.\n  string concrete_function_name = 1;\n\n  // A sequence of unique strings, one per Tensor argument.\n  repeated string argument_keywords = 2;\n  // The prefix of `argument_keywords` which may be identified by position.\n  int64 allowed_positional_arguments = 3;\n  // The spec of the function that this ConcreteFunction is traced from. This\n  // allows the ConcreteFunction to be called with nest structure inputs. This\n  // field may not be populated. If this field is absent, the concrete function\n  // can only be called with flat inputs.\n  // TODO(b/169361281): support calling saved ConcreteFunction with structured\n  // inputs in C++ SavedModel API.\n  FunctionSpec function_spec = 4;\n}\n\nmessage SavedConstant {\n  // An Operation name for a ConstantOp in this SavedObjectGraph's MetaGraph.\n  string operation = 1;\n}\n\n// Represents a Variable that is initialized by loading the contents from the\n// checkpoint.\nmessage SavedVariable {\n  DataType dtype = 1;\n  TensorShapeProto shape = 2;\n  bool trainable = 3;\n  VariableSynchronization synchronization = 4;\n  VariableAggregation aggregation = 5;\n  string name = 6;\n  string device = 7;\n  // List of component variables for a distributed variable.\n  //\n  // When this field is non-empty, the SavedVariable will be assumed\n  // to be a distributed variable defined by the components listed here.\n  //\n  // This is only supported by experimental loaders at the moment.\n  repeated SavedVariable experimental_distributed_variable_components = 8;\n}\n\n// Represents `FunctionSpec` used in `Function`. This represents a\n// function that has been wrapped as a TensorFlow `Function`.\nmessage FunctionSpec {\n  // Full arg spec from inspect.getfullargspec().\n  StructuredValue fullargspec = 1;\n  // Whether this represents a class method.\n  bool is_method = 2;\n  // The input signature, if specified.\n  StructuredValue input_signature = 5;\n\n  // Whether the function should be compiled by XLA.\n  //\n  // The public interface to `tf.function` uses an optional boolean to\n  // represent three distinct states for this field.  Unfortunately, proto3\n  // removes the ability to explicitly check for the presence or absence of a\n  // field, so we instead map to an enum.\n  //\n  // See `tf.function` for details.\n  enum JitCompile {\n    DEFAULT = 0;\n    ON = 1;\n    OFF = 2;\n  }\n  JitCompile jit_compile = 6;\n\n  reserved 3, 4;\n}\n\n// A SavedResource represents a TF object that holds state during its lifetime.\n// An object of this type can have a reference to a:\n// create_resource() and an initialize() function.\nmessage SavedResource {\n  // A device specification indicating a required placement for the resource\n  // creation function, e.g. \"CPU\". An empty string allows the user to select a\n  // device.\n  string device = 1;\n}\n\nmessage SaveableObject {\n  // Node ids of concrete functions for saving and loading from a checkpoint.\n  // These functions save and restore directly from tensors.\n  int32 save_function = 2;\n  int32 restore_function = 3;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/saver.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"SaverProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.util\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Protocol buffer representing the configuration of a Saver.\nmessage SaverDef {\n  // The name of the tensor in which to specify the filename when saving or\n  // restoring a model checkpoint.\n  string filename_tensor_name = 1;\n\n  // The operation to run when saving a model checkpoint.\n  string save_tensor_name = 2;\n\n  // The operation to run when restoring a model checkpoint.\n  string restore_op_name = 3;\n\n  // Maximum number of checkpoints to keep.  If 0, no checkpoints are deleted.\n  int32 max_to_keep = 4;\n\n  // Shard the save files, one per device that has Variable nodes.\n  bool sharded = 5;\n\n  // How often to keep an additional checkpoint. If not specified, only the last\n  // \"max_to_keep\" checkpoints are kept; if specified, in addition to keeping\n  // the last \"max_to_keep\" checkpoints, an additional checkpoint will be kept\n  // for every n hours of training.\n  float keep_checkpoint_every_n_hours = 6;\n\n  // A version number that identifies a different on-disk checkpoint format.\n  // Usually, each subclass of BaseSaverBuilder works with a particular\n  // version/format.  However, it is possible that the same builder may be\n  // upgraded to support a newer checkpoint format in the future.\n  enum CheckpointFormatVersion {\n    // Internal legacy format.\n    LEGACY = 0;\n    // Deprecated format: tf.Saver() which works with tensorflow::table::Table.\n    V1 = 1;\n    // Current format: more efficient.\n    V2 = 2;\n  }\n  CheckpointFormatVersion version = 7;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/service_config.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.data.experimental;\n\nimport \"tensorflow/core/protobuf/data_service.proto\";\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Configuration for a tf.data service DispatchServer.\n// Next id: 10\nmessage DispatcherConfig {\n  // The port for the dispatcher to bind to. A value of 0 indicates that the\n  // dispatcher may bind to any available port.\n  int64 port = 1;\n  // The protocol for the dispatcher to use when connecting to workers.\n  string protocol = 2;\n  // A work directory to use for storing dispatcher state, and for recovering\n  // during restarts. The empty string indicates not to use any work directory.\n  string work_dir = 3;\n  // Whether to run in fault tolerant mode, where dispatcher state is saved\n  // across restarts. Requires that `work_dir` is nonempty.\n  bool fault_tolerant_mode = 4;\n  // (Optional.) If the job uses auto-sharding, it needs to specify a fixed list\n  // of worker addresses that will register with the dispatcher. The worker\n  // addresses should be in the format \"host\" or \"host:port\", where \"port\" is an\n  // integer, named port, or %port% to match any port.\n  repeated string worker_addresses = 7;\n  // (Optional.) tf.data service deployment mode. Supported values are \"REMOTE\",\n  // \"COLOCATED\", and \"HYBRID\". If unspecified, it is assumed to be \"REMOTE\".\n  DeploymentMode deployment_mode = 9;\n  // How often the dispatcher should scan through to delete old and unused\n  // jobs. A value of 0 indicates that the decision should be left up to the\n  // runtime.\n  int64 job_gc_check_interval_ms = 5;\n  // How long a job needs to be unused before it becomes a candidate for garbage\n  // collection. A value of -1 indicates that jobs should never be garbage\n  // collected. A value of 0 indicates that the decision should be left up to\n  // the runtime.\n  int64 job_gc_timeout_ms = 6;\n  // How long to wait before garbage-collecting a client that hasn't\n  // heartbeated to the dispatcher. A value of 0 indicates that the timeout\n  // should be left to the runtime.\n  int64 client_timeout_ms = 8;\n}\n\n// Configuration for a tf.data service WorkerServer.\n// Next id: 11\nmessage WorkerConfig {\n  // The port for the worker to bind to. A value of 0 indicates that the\n  // worker may bind to any available port.\n  int64 port = 1;\n  // The protocol for the worker to use when connecting to the dispatcher.\n  string protocol = 2;\n  // The address of the dispatcher to register with.\n  string dispatcher_address = 3;\n  // The address of the worker server. The substring \"%port%\", if specified,\n  // will be replaced with the worker's bound port. This is useful when the port\n  // is set to `0`.\n  string worker_address = 4;\n  // Tags attached to the worker. This allows reading from selected workers.\n  // For example, by applying a \"COLOCATED\" tag, tf.data service is able to read\n  // from the local tf.data worker if one exists, then from off-TF-host workers,\n  // to avoid cross-TF-host reads.\n  repeated string worker_tags = 10;\n  // How often the worker should heartbeat to the master. A value of 0 indicates\n  // that the decision should be left up to the runtime.\n  int64 heartbeat_interval_ms = 5;\n  // How long to retry requests to the dispatcher before giving up and reporting\n  // an error. A value of 0 indicates that the decision should be left up to the\n  // runtime.\n  int64 dispatcher_timeout_ms = 6;\n  // The protocol for the worker to use when transferring data to clients.\n  string data_transfer_protocol = 7;\n  // The data transfer address of the worker server. The substring \"%port%\", if\n  // specified, will be replaced with the worker's bound port. This is useful\n  // when the port is set to `0`.\n  string data_transfer_address = 8;\n  // When shutting down a worker, how long to wait for the gRPC server to\n  // process the final requests. This is used to achieve clean shutdown in unit\n  // tests.\n  int64 shutdown_quiet_period_ms = 9;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/snapshot.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.data.experimental;\n\nimport \"tensorflow/core/framework/tensor.proto\";\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/types.proto\";\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Each SnapshotRecord represents one batch of pre-processed input data. A batch\n// consists of a list of tensors that we encode as TensorProtos. This message\n// doesn't store the structure of the batch.\nmessage SnapshotRecord {\n  repeated .tensorflow.TensorProto tensor = 1;\n}\n\n// This stores the metadata information present in each snapshot record.\nmessage SnapshotMetadataRecord {\n  // Stores the fingerprint of the graph that describes the dataset that is\n  // snapshotted.\n  string graph_hash = 1;\n  // Run ID that this snapshot corresponds to.\n  string run_id = 2;\n  // Time when we started creating this snapshot.\n  int64 creation_timestamp = 3;\n  // Version of the snapshot data file format.\n  int64 version = 4;\n  // A list of tensor dtype corresponding to each element of the snapshot.\n  repeated .tensorflow.DataType dtype = 5;\n  // The number of elements in the snapshot.\n  int64 num_elements = 6;\n\n  bool finalized = 1000;\n}\n\n// Metadata for a single tensor in the Snapshot Record.\nmessage TensorMetadata {\n  .tensorflow.TensorShapeProto tensor_shape = 2;\n  // Number of uncompressed bytes used to store the tensor representation.\n  int64 tensor_size_bytes = 3;\n}\n\n// Metadata for all the tensors in a Snapshot Record.\nmessage SnapshotTensorMetadata {\n  repeated TensorMetadata tensor_metadata = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/status.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// If included as a payload, this message flags the Status to be a \"derived\"\n// Status. Used by StatusGroup to ignore certain Statuses when reporting\n// errors to end users.\nmessage DerivedStatus {}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/struct.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/tensor.proto\";\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/types.proto\";\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// `StructuredValue` represents a dynamically typed value representing various\n// data structures that are inspired by Python data structures typically used in\n// TensorFlow functions as inputs and outputs.\n//\n// For example when saving a Layer there may be a `training` argument. If the\n// user passes a boolean True/False, that switches between two concrete\n// TensorFlow functions. In order to switch between them in the same way after\n// loading the SavedModel, we need to represent \"True\" and \"False\".\n//\n// A more advanced example might be a function which takes a list of\n// dictionaries mapping from strings to Tensors. In order to map from\n// user-specified arguments `[{\"a\": tf.constant(1.)}, {\"q\": tf.constant(3.)}]`\n// after load to the right saved TensorFlow function, we need to represent the\n// nested structure and the strings, recording that we have a trace for anything\n// matching `[{\"a\": tf.TensorSpec(None, tf.float32)}, {\"q\": tf.TensorSpec([],\n// tf.float64)}]` as an example.\n//\n// Likewise functions may return nested structures of Tensors, for example\n// returning a dictionary mapping from strings to Tensors. In order for the\n// loaded function to return the same structure we need to serialize it.\n//\n// This is an ergonomic aid for working with loaded SavedModels, not a promise\n// to serialize all possible function signatures. For example we do not expect\n// to pickle generic Python objects, and ideally we'd stay language-agnostic.\nmessage StructuredValue {\n  // The kind of value.\n  oneof kind {\n    // Represents None.\n    NoneValue none_value = 1;\n\n    // Represents a double-precision floating-point value (a Python `float`).\n    double float64_value = 11;\n    // Represents a signed integer value, limited to 64 bits.\n    // Larger values from Python's arbitrary-precision integers are unsupported.\n    sint64 int64_value = 12;\n    // Represents a string of Unicode characters stored in a Python `str`.\n    // In Python 3, this is exactly what type `str` is.\n    // In Python 2, this is the UTF-8 encoding of the characters.\n    // For strings with ASCII characters only (as often used in TensorFlow code)\n    // there is effectively no difference between the language versions.\n    // The obsolescent `unicode` type of Python 2 is not supported here.\n    string string_value = 13;\n    // Represents a boolean value.\n    bool bool_value = 14;\n\n    // Represents a TensorShape.\n    tensorflow.TensorShapeProto tensor_shape_value = 31;\n    // Represents an enum value for dtype.\n    tensorflow.DataType tensor_dtype_value = 32;\n    // Represents a value for tf.TensorSpec.\n    TensorSpecProto tensor_spec_value = 33;\n    // Represents a value for tf.TypeSpec.\n    TypeSpecProto type_spec_value = 34;\n    // Represents a value for tf.BoundedTensorSpec.\n    BoundedTensorSpecProto bounded_tensor_spec_value = 35;\n\n    // Represents a list of `Value`.\n    ListValue list_value = 51;\n    // Represents a tuple of `Value`.\n    TupleValue tuple_value = 52;\n    // Represents a dict `Value`.\n    DictValue dict_value = 53;\n    // Represents Python's namedtuple.\n    NamedTupleValue named_tuple_value = 54;\n  }\n}\n\n// Represents None.\nmessage NoneValue {}\n\n// Represents a Python list.\nmessage ListValue {\n  repeated StructuredValue values = 1;\n}\n\n// Represents a Python tuple.\nmessage TupleValue {\n  repeated StructuredValue values = 1;\n}\n\n// Represents a Python dict keyed by `str`.\n// The comment on Unicode from Value.string_value applies analogously.\nmessage DictValue {\n  map<string, StructuredValue> fields = 1;\n}\n\n// Represents a (key, value) pair.\nmessage PairValue {\n  string key = 1;\n  StructuredValue value = 2;\n}\n\n// Represents Python's namedtuple.\nmessage NamedTupleValue {\n  string name = 1;\n  repeated PairValue values = 2;\n}\n\n// A protobuf to represent tf.TensorSpec.\nmessage TensorSpecProto {\n  string name = 1;\n  tensorflow.TensorShapeProto shape = 2;\n  tensorflow.DataType dtype = 3;\n}\n\n// A protobuf to represent tf.BoundedTensorSpec.\nmessage BoundedTensorSpecProto {\n  string name = 1;\n  tensorflow.TensorShapeProto shape = 2;\n  tensorflow.DataType dtype = 3;\n  tensorflow.TensorProto minimum = 4;\n  tensorflow.TensorProto maximum = 5;\n}\n\n// Represents a tf.TypeSpec\nmessage TypeSpecProto {\n  enum TypeSpecClass {\n    UNKNOWN = 0;\n    SPARSE_TENSOR_SPEC = 1;   // tf.SparseTensorSpec\n    INDEXED_SLICES_SPEC = 2;  // tf.IndexedSlicesSpec\n    RAGGED_TENSOR_SPEC = 3;   // tf.RaggedTensorSpec\n    TENSOR_ARRAY_SPEC = 4;    // tf.TensorArraySpec\n    DATA_DATASET_SPEC = 5;    // tf.data.DatasetSpec\n    DATA_ITERATOR_SPEC = 6;   // IteratorSpec from data/ops/iterator_ops.py\n    OPTIONAL_SPEC = 7;        // tf.OptionalSpec\n    PER_REPLICA_SPEC = 8;     // PerReplicaSpec from distribute/values.py\n    VARIABLE_SPEC = 9;        // tf.VariableSpec\n    ROW_PARTITION_SPEC = 10;  // RowPartitionSpec from ragged/row_partition.py\n    reserved 11;\n    REGISTERED_TYPE_SPEC = 12;  // The type registered as type_spec_class_name.\n    EXTENSION_TYPE_SPEC = 13;   // Subclasses of tf.ExtensionType\n  }\n  TypeSpecClass type_spec_class = 1;\n\n  // The value returned by TypeSpec._serialize().\n  StructuredValue type_state = 2;\n\n  // The name of the TypeSpec class.\n  //  * If type_spec_class == REGISTERED_TYPE_SPEC, the TypeSpec class is\n  //    the one registered under this name. For types registered outside\n  //    core TensorFlow by an add-on library, that library must be loaded\n  //    before this value can be deserialized by nested_structure_coder.\n  //  * If type_spec_class specifies a particular TypeSpec class, this field is\n  //    redundant with the type_spec_class enum, and is only used for error\n  //    reporting in older binaries that do not know the tupe_spec_class enum.\n  string type_spec_class_name = 3;\n\n  // The number of flat tensor components required by this TypeSpec.\n  int32 num_flat_components = 4;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/tensor_bundle.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/tensor_slice.proto\";\nimport \"tensorflow/core/framework/types.proto\";\nimport \"tensorflow/core/framework/versions.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"TensorBundleProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.util\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Protos used in the tensor bundle module (tf/core/util/tensor_bundle/).\n\n// Special header that is associated with a bundle.\n//\n// TODO(zongheng,zhifengc): maybe in the future, we can add information about\n// which binary produced this checkpoint, timestamp, etc. Sometime, these can be\n// valuable debugging information. And if needed, these can be used as defensive\n// information ensuring reader (binary version) of the checkpoint and the writer\n// (binary version) must match within certain range, etc.\nmessage BundleHeaderProto {\n  // Number of data files in the bundle.\n  int32 num_shards = 1;\n\n  // An enum indicating the endianness of the platform that produced this\n  // bundle.  A bundle can only be read by a platform with matching endianness.\n  // Defaults to LITTLE, as most modern platforms are little-endian.\n  //\n  // Affects the binary tensor data bytes only, not the metadata in protobufs.\n  enum Endianness {\n    LITTLE = 0;\n    BIG = 1;\n  }\n  Endianness endianness = 2;\n\n  // Versioning of the tensor bundle format.\n  VersionDef version = 3;\n}\n\n// Describes the metadata related to a checkpointed tensor.\nmessage BundleEntryProto {\n  // The tensor dtype and shape.\n  DataType dtype = 1;\n  TensorShapeProto shape = 2;\n  // The binary content of the tensor lies in:\n  //   File \"shard_id\": bytes [offset, offset + size).\n  int32 shard_id = 3;\n  int64 offset = 4;\n  int64 size = 5;\n\n  // The CRC32C checksum of the tensor bytes.\n  fixed32 crc32c = 6;\n\n  // Iff present, this entry represents a partitioned tensor.  The previous\n  // fields are interpreted as follows:\n  //\n  //   \"dtype\", \"shape\": describe the full tensor.\n  //   \"shard_id\", \"offset\", \"size\", \"crc32c\": all IGNORED.\n  //      These information for each slice can be looked up in their own\n  //      BundleEntryProto, keyed by each \"slice_name\".\n  repeated TensorSliceProto slices = 7;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/tensorflow_server.proto",
    "content": "/* Copyright 2016 The TensorFlow Authors. All Rights Reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n==============================================================================*/\n\nsyntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"tensorflow/core/protobuf/cluster.proto\";\nimport \"tensorflow/core/protobuf/config.proto\";\nimport \"tensorflow/core/protobuf/device_filters.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"ServerProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.distruntime\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Defines the configuration of a single TensorFlow server.\nmessage ServerDef {\n  // The cluster of which this server is a member.\n  ClusterDef cluster = 1;\n\n  // The name of the job of which this server is a member.\n  //\n  // NOTE(mrry): The `cluster` field must contain a `JobDef` with a `name` field\n  // that matches this name.\n  string job_name = 2;\n\n  // The task index of this server in its job.\n  //\n  // NOTE: The `cluster` field must contain a `JobDef` with a matching `name`\n  // and a mapping in its `tasks` field for this index.\n  int32 task_index = 3;\n\n  // The default configuration for sessions that run on this server.\n  ConfigProto default_session_config = 4;\n\n  // The protocol to be used by this server.\n  //\n  // Acceptable values include: \"grpc\", \"grpc+verbs\".\n  string protocol = 5;\n\n  // The server port. If not set, then we identify the port from the job_name.\n  int32 port = 6;\n\n  // Device filters for remote tasks in the cluster.\n  // NOTE: This is an experimental feature and only effective in TensorFlow 2.x.\n  ClusterDeviceFilters cluster_device_filters = 7;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/trackable_object_graph.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"google/protobuf/wrappers.proto\";\n\noption cc_enable_arenas = true;\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// A TensorBundle addition which saves extra information about the objects which\n// own variables, allowing for more robust checkpoint loading into modified\n// programs.\n\nmessage TrackableObjectGraph {\n  message TrackableObject {\n    message ObjectReference {\n      // An index into `TrackableObjectGraph.nodes`, indicating the object\n      // being referenced.\n      int32 node_id = 1;\n      // A user-provided name for the edge.\n      string local_name = 2;\n    }\n\n    message SerializedTensor {\n      // A name for the Tensor. Simple variables have only one\n      // `SerializedTensor` named \"VARIABLE_VALUE\" by convention. This value may\n      // be restored on object creation as an optimization.\n      string name = 1;\n      // The full name of the variable/tensor, if applicable. Used to allow\n      // name-based loading of checkpoints which were saved using an\n      // object-based API. Should match the checkpoint key which would have been\n      // assigned by tf.train.Saver.\n      string full_name = 2;\n      // The generated name of the Tensor in the checkpoint.\n      string checkpoint_key = 3;\n      // Deprecated bool field for optional restore. This field has never been\n      // set to True.\n      reserved \"optional_restore\";\n      reserved 4;\n    }\n\n    message SlotVariableReference {\n      // An index into `TrackableObjectGraph.nodes`, indicating the\n      // variable object this slot was created for.\n      int32 original_variable_node_id = 1;\n      // The name of the slot (e.g. \"m\"/\"v\").\n      string slot_name = 2;\n      // An index into `TrackableObjectGraph.nodes`, indicating the\n      // `Object` with the value of the slot variable.\n      int32 slot_variable_node_id = 3;\n    }\n\n    // Objects which this object depends on.\n    repeated ObjectReference children = 1;\n    // Serialized data specific to this object.\n    repeated SerializedTensor attributes = 2;\n    // Slot variables owned by this object.\n    repeated SlotVariableReference slot_variables = 3;\n\n    // The registered saver used to save this object. If this saver is not\n    // present when loading the checkpoint, then loading will fail.\n    RegisteredSaver registered_saver = 4;\n\n    // Whether this object has checkpoint values or descendants with checkpoint\n    // values. This is computed at save time to avoid traversing the entire\n    // object graph proto when restoring (which also has to traverse the live\n    // object graph).\n    google.protobuf.BoolValue has_checkpoint_values = 5;\n  }\n\n  repeated TrackableObject nodes = 1;\n}\n\nmessage RegisteredSaver {\n  // The name of the registered saver/restore function.\n  string name = 1;\n\n  // Unique auto-generated name of the object.\n  string object_name = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/transport_options.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// Extra data needed on a non-RDMA RecvBufResponse.\nmessage RecvBufRespExtra {\n  repeated bytes tensor_content = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/verifier_config.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow;\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"VerifierConfigProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.framework\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n// The config for graph verifiers.\nmessage VerifierConfig {\n  enum Toggle {\n    DEFAULT = 0;\n    ON = 1;\n    OFF = 2;\n  }\n\n  // Deadline for completion of all verification i.e. all the Toggle ON\n  // verifiers must complete execution within this time.\n  int64 verification_timeout_in_ms = 1;\n\n  // Perform structural validation on a tensorflow graph. Default is OFF.\n  Toggle structure_verifier = 2;\n\n  // Next tag: 3\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/worker.proto",
    "content": "/* Copyright 2016 The TensorFlow Authors. All Rights Reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n==============================================================================*/\n\nsyntax = \"proto3\";\n\npackage tensorflow;\n\nimport \"google/protobuf/any.proto\";\nimport \"tensorflow/core/framework/cost_graph.proto\";\nimport \"tensorflow/core/framework/device_attributes.proto\";\nimport \"tensorflow/core/framework/graph.proto\";\nimport \"tensorflow/core/framework/step_stats.proto\";\nimport \"tensorflow/core/framework/tensor.proto\";\nimport \"tensorflow/core/framework/tensor_shape.proto\";\nimport \"tensorflow/core/framework/types.proto\";\nimport \"tensorflow/core/protobuf/config.proto\";\nimport \"tensorflow/core/protobuf/coordination_config.proto\";\nimport \"tensorflow/core/protobuf/debug.proto\";\nimport \"tensorflow/core/protobuf/error_codes.proto\";\nimport \"tensorflow/core/protobuf/named_tensor.proto\";\nimport \"tensorflow/core/protobuf/tensorflow_server.proto\";\n\noption cc_enable_arenas = true;\noption java_outer_classname = \"WorkerProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.distruntime\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// GetStatus method request/response messages\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage GetStatusRequest {}\n\nmessage GetStatusResponse {\n  repeated DeviceAttributes device_attributes = 1;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// CreateSession method request/response messages\n//\n// For each session,\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage CreateWorkerSessionRequest {\n  // Sessions are identified by a given handle.\n  string session_handle = 1;\n\n  // Defines the configuration of a TensorFlow worker.\n  ServerDef server_def = 2;\n\n  // If true, any resources such as Variables used in the session will not be\n  // shared with other sessions.\n  bool isolate_session_state = 3;\n\n  // The device attributes of all the devices in the cluster.\n  repeated DeviceAttributes cluster_device_attributes = 4;\n\n  // The master task name from which the request is sent.\n  string master_task = 5;\n\n  // The incarnation ID of the master task local CPU device.\n  // If the target worker already has a WorkerSession created previously with\n  // the same master task name but a different incarnation, it usually indicates\n  // that the previous master failed before deleting the WorkerSession on the\n  // worker. To prevent memory leaks, the worker should garbage collect the old\n  // WorkerSessions.\n  int64 master_incarnation = 6;\n\n  // Configures coordination service within worker sessions.\n  CoordinationServiceConfig coordination_service_config = 7;\n}\n\nmessage CreateWorkerSessionResponse {}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// DeleteSession method request/response messages\n//\n// Deletes all worker-side state associated with the given session handle.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage DeleteWorkerSessionRequest {\n  // Sessions are identified by a given handle.\n  string session_handle = 1;\n}\n\nmessage DeleteWorkerSessionResponse {}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// RegisterGraph method request/response messages\n//\n// For each session, after the master placed every node on a device,\n// it partitions the whole graph into many subgraphs. All the nodes in\n// a subgraph were in the same worker, but potentially on many devices\n// owned by that worker (e.g. cpu0, plus gpu0, gpu1, ..., gpu7). The\n// master registers subgraphs for a worker before running any steps. A\n// successful registration returns a graph handle to be used in latter\n// RunGraph requests.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage RegisterGraphRequest {\n  // Subgraphs are scoped within one session.\n  string session_handle = 1;\n\n  // Set to true if `CreateWorkerSession` was called for `session_handle`.\n  bool create_worker_session_called = 6;\n\n  // \"graph_def\" has the subgraph of nodes for this worker, with each node\n  // having its device_name filled in.\n  GraphDef graph_def = 2;\n\n  // True iff the graph (before partitioning) contains control flow nodes.\n  //\n  // As of 01/11/2015, this is no longer set by clients.\n  bool has_control_flow = 3 [deprecated = true];\n\n  // Configuration options for the session in which this graph was created.\n  GraphOptions graph_options = 4;\n\n  // Field(s) used by TensorFlow Debugger (tfdbg).\n  DebugOptions debug_options = 5;\n\n  // If graph_def contains any collective ops this must be a positive\n  // integer used to coordinate execution with other graphs.  All\n  // graphs in a distributed execution with the same\n  // collective_graph_key will coordinate to use the same step_id\n  // concurrently so that BufRendezvous entries will make the correct\n  // values accessible.\n  int64 collective_graph_key = 7;\n\n  // ConfigProto from the session in which this graph was created.\n  // Contains additional parameters beyond graph_options, including\n  // the name of the requested executor.\n  ConfigProto config_proto = 8;\n}\n\nmessage RegisterGraphResponse {\n  // If the registration succeeds, returns an opaque graph_handle to\n  // the master. The master calls RunGraph with graph_handle to\n  // compute different steps.\n  string graph_handle = 1;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// DeregisterGraph method request/response messages\n//\n// The master deregisters the given graph_handle when the graph is no\n// longer needed (e.g., the overall graph is re-scheduled and nodes\n// are re-placed).\n//\n// The worker deregisters a graph_handle automatically according to on\n// a TTL-base policy in case of master restarts.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage DeregisterGraphRequest {\n  // The session_handle used when registering the graph. If session_handle is\n  // empty, a single global namespace is used.\n  string session_handle = 2;\n\n  // Set to true if `CreateWorkerSession` was called for `session_handle`.\n  bool create_worker_session_called = 3;\n\n  // REQUIRED: graph_handle must be returned by a RegisterGraph call\n  // to the same WorkerService.\n  string graph_handle = 1;\n}\n\nmessage DeregisterGraphResponse {\n  // TODO(mrry): Optionally add summary stats for the graph.\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// CleanupAll method request/response messages\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage CleanupAllRequest {\n  // A list of container names.\n  //\n  // If 'container' is not empty, releases resources in the given\n  // containers in all devices.\n  //\n  // If 'container' is empty, releases resources in the default\n  // container in all devices.\n  repeated string container = 1;\n}\n\nmessage CleanupAllResponse {}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// RunGraph request / response messages\n//\n// The worker executes all subgraphs registered under graph_handle.\n// RunGraph returns after the execution finishes or an error is\n// encountered.\n// A sequence of RunGraphRequests with is_partial may be sent to RunGraph for\n// partial graph execution.\n//\n////////////////////////////////////////////////////////////////////////////////\n\n// Options specific to the execution of a single step.\nmessage ExecutorOpts {\n  bool record_costs = 1;\n  bool record_timeline = 3;\n  bool record_partition_graphs = 4;\n  bool report_tensor_allocations_upon_oom = 5;\n}\n\nmessage RunGraphRequest {\n  // session_handle is the master-generated unique id for this session.\n  // If session_handle is non-empty, it must be the same as used when\n  // registering the graph. If it is empty, a single global namespace is used to\n  // search for the graph_handle.\n  string session_handle = 8;\n\n  // Set to true if `CreateWorkerSession` was called for `session_handle`.\n  bool create_worker_session_called = 10;\n\n  // REQUIRED: graph_handle must be returned by a RegisterGraph call\n  // to the same WorkerService.\n  string graph_handle = 1;\n\n  // A unique ID to distinguish different runs of the same graph.\n  //\n  // The master generates a global unique `step_id` to distinguish\n  // different runs of the graph computation. Subgraphs communicate\n  // (e.g., send/recv ops) with each other using `step_id` to\n  // distinguish tensors generated by different runs.\n  int64 step_id = 2;\n\n  // Options for this step.\n  ExecutorOpts exec_opts = 5;\n\n  // Runs the graph.\n  //\n  // Sends the tensors in \"send\" into the graph before the run and\n  // fetches the keys into `RunGraphResponse.recv` after the run.\n  repeated NamedTensorProto send = 3;\n  repeated string recv_key = 4;\n\n  // True if the RunGraphRequest is a partial run request.\n  bool is_partial = 6;\n  // True if this is the last partial run request in a sequence of requests.\n  bool is_last_partial_run = 7;\n\n  // If true then some errors, e.g., execution errors that have long\n  // error messages, may return an OK RunGraphResponse with the actual\n  // error saved in the status_code/status_error_message fields of the\n  // response body. This is a workaround since the RPC subsystem may\n  // truncate long metadata messages.\n  bool store_errors_in_response_body = 9;\n\n  // Unique identifier for this request. Every RunGraphRequest must have a\n  // unique request_id, and retried RunGraphRequests must have the same\n  // request_id. If request_id is zero, retry detection is disabled.\n  //\n  // Retried RunGraphRequests are problematic because they may issue a\n  // RecvTensor that will have no corresponding sender and will wait forever.\n  // Workers use request_ids to reject retried RunGraph requests instead of\n  // waiting forever.\n  int64 request_id = 11;\n\n  // Next: 12\n}\n\nmessage RunGraphResponse {\n  // A list of tensors corresponding to those requested by\n  // `RunGraphRequest.recv_key`.\n  repeated NamedTensorProto recv = 1;\n\n  // If the request asked for execution stats, the cost graph, or the partition\n  // graphs, these are returned here.\n  // TODO(suharshs): Package these in a RunMetadata instead.\n  StepStats step_stats = 2;\n  CostGraphDef cost_graph = 3;\n  repeated GraphDef partition_graph = 4;\n\n  // If store_errors_in_response_body is true in the request, then\n  // optionally the server may return an OK status for the RPC and\n  // fill the true status into the fields below, to allow for messages\n  // that are too long to fit in metadata.\n  error.Code status_code = 5;\n  string status_error_message = 6;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// CleanupGraph method request/response messages\n//\n// After the master receives RunGraph responses from all workers, the\n// master instructs every worker to cleanup any remaining state of a\n// step (e.g. tensors buffered by a `Send` op but not picked up by\n// other workers). The master does not necessarily need to wait for\n// completion of CleanupGraph calls.\n//\n// Workers should cleanup step states automatically according to a\n// TTL-based policy in case of master restarts.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage CleanupGraphRequest {\n  int64 step_id = 1;\n}\n\nmessage CleanupGraphResponse {}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// RecvTensor method request/response messages\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage RecvTensorRequest {\n  // The step in which the tensor will be produced.\n  //\n  // REQUIRED: This must eventually correspond to the `step_id` passed\n  // into a RunGraph call on the same WorkerService.\n  int64 step_id = 1;\n\n  // A key identifying the channel to receive tensors from. A RecvTensor request\n  // retrieves one tensor from the channel, but multiple tensors can be sent and\n  // received over the same channel with multiple RecvTensor requests. See\n  // rendezvous.h for details.\n  string rendezvous_key = 2;\n\n  // If true, use an out-of-band DMA mechanism to transfer the\n  // received tensor.\n  bool dma_ok = 3;\n\n  // Optional information on client-side device locality.\n  DeviceLocality client_locality = 4;\n\n  // Optional information on server-side device locality.\n  DeviceLocality server_locality = 5;\n\n  // Optional information needed by the RPC subsystem.\n  google.protobuf.Any transport_options = 6;\n\n  // Unique identifier for this request. Every RecvTensorRequest must have a\n  // unique request_id, and retried RecvTensorRequests must have the same\n  // request_id. If request_id is zero, retry detection and response cache\n  // are disabled.\n  //\n  // Retried RecvTensorRequests are problematic because a RecvTensor with no\n  // corresponding sender will wait forever, and the tensor may have been\n  // delivered to a previous retry. Workers use request_ids to reject retried\n  // RecvTensor requests instead of waiting forever.\n  int64 request_id = 7;\n}\n\nmessage RecvTensorResponse {\n  // The tensor as a proto.\n  TensorProto tensor = 1;\n\n  // If true, this tensor was the output of a dead node, and the\n  // content is invalid.\n  bool is_dead = 2;\n\n  // The time at which tensor was available and started to be returned.\n  int64 send_start_micros = 3;\n\n  // Optional additional information about how to receive the tensor,\n  // e.g. in the event that `RecvTensorRequest.dma_ok` was true.\n  google.protobuf.Any transport_options = 4;\n\n  // Whether the receiver should send a MarkRecvFinishedRequest to the sender\n  // to ack the message.\n  bool require_ack = 5;\n}\n\n// Message for managing the response cache maintained on the sender side.\n// Currently only used by the gRPC worker service.\nmessage MarkRecvFinishedRequest {\n  int64 request_id = 1;\n}\n\nmessage MarkRecvFinishedResponse {}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// Logging method request/response messages\n//\n// NOTE(mrry): This feature is not supported in the open-source\n// version, and these messages are expected to change.\n//\n////////////////////////////////////////////////////////////////////////////////\n\n// Out-of-band request to begin or end logging, or\n// to retrieve logs for particular steps.\nmessage LoggingRequest {\n  // If true, RPC logging will be enabled.\n  bool enable_rpc_logging = 1;\n\n  // If true, RPC logging will be disabled.\n  bool disable_rpc_logging = 4;\n\n  // If true, discard any saved logging data (for all steps).\n  bool clear = 2;\n\n  // When set, requests all saved log data pertaining to the step.\n  // Any log data retrieved is eliminated from the store and cannot be\n  // retrieved again.\n  repeated int64 fetch_step_id = 3;\n}\n\nmessage LabeledStepStats {\n  int64 step_id = 1;\n  StepStats step_stats = 2;\n}\n\nmessage LoggingResponse {\n  repeated LabeledStepStats step = 1;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// Tracing method request/response messages\n//\n// NOTE(mrry): This feature is not supported in the open-source\n// version, and these messages are expected to change.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage TraceOpts {\n  // Length of the trace to be taken, in seconds.\n  double duration = 1;\n  // If true, capture step profile locally in each worker. Currently\n  // unimplemented.\n  bool use_step_profiler = 2;\n  // If true, capture kernel events from each worker.\n  bool use_kernel_profiler = 3;\n  // If true, capture extended profiling events from TensorFlow process.\n  bool use_extended_profiler = 4;\n  // If true, capture GPU profiling events locally on each\n  // machine. Currently unimplemented.\n  bool use_gpu_profiler = 5;\n  // If true, collect sampled profile events. Currently unimplemented.\n  bool use_sample_profiler = 6;\n}\n\n// Out-of-band request to configure distributed tracing.\nmessage TracingRequest {\n  TraceOpts options = 1;\n}\n\nmessage TracingResponse {}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// Raw data transfers in support of Collective Ops.\n// These methods are experimental and subject to change.\n//\n// The intention is to allow collectives to take advantage of the most\n// efficient methods available on a platform, e.g. RDMA, and not be\n// constrained to use the RPC system in use by other methods.\n//\n////////////////////////////////////////////////////////////////////////////////\n\nmessage RecvBufRequest {\n  // Use of the fields below may vary by implementation.  For example\n  // the buf_ptr and num_bytes may be set only for local operations and\n  // not sent on the wire, or only sent on the wire in one direction.\n\n  // Used at server side to find the correct BufRendezvous.\n  int64 step_id = 1;\n\n  // Arbitrary string identifying a BufRendezvous entry.\n  string buf_rendezvous_key = 2;\n\n  // Size of value expected, must agree with BufRendezvous entry.\n  int64 num_bytes = 3;\n\n  // When RDMA is in use, address of destination field on client.\n  fixed64 buf_ptr = 4;\n\n  // Optional information on client-side device locality.\n  DeviceLocality client_locality = 5;\n\n  // Optional information on server-side device locality.\n  DeviceLocality server_locality = 6;\n\n  // Optional, implementation-specific data.\n  google.protobuf.Any transport_options = 7;\n  // For annotating timeline and device incarnation check.\n  string src_device = 8;\n  // Optional, for annotating the timeline.\n  string dst_device = 9;\n\n  // Depending on the RPC system in use, it may be necessary to set this\n  // id to detect resends of RPCs where the server is not aware that\n  // the prior RPC failed.\n  int64 request_id = 10;\n\n  // Incarnation number of the source device, used to detect worker failures.\n  uint64 src_incarnation = 11;\n}\n\nmessage RecvBufResponse {\n  // Use of the fields below may vary by implementation.  Comments give\n  // intended use.\n\n  fixed64 buf_ptr = 1;  // Address of source field on server.\n  int64 num_bytes = 2;  // Byte length of buf_ptr field, if set.\n  bool is_dead = 3;     // True if value is 'dead' like a tensor.\n  // Optional, implementation-specific data.\n  google.protobuf.Any transport_options = 4;\n  // Optional, for timeline.\n  int64 send_start_micros = 5;\n\n  // Whether the receiver should send a MarkRecvFinishedRequest to the sender\n  // to ack the message.\n  bool require_ack = 6;\n}\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// Collective Op dynamic group resolution messages.\n//\n////////////////////////////////////////////////////////////////////////////////\n\n// Supplies one or more device names as members of the group identified by\n// group_key.  Service will respond when all group_size devices become known.\n// All devices in group must have same type.\nmessage CompleteGroupRequest {\n  int32 group_key = 1;\n  int32 group_size = 2;\n  string device_type = 3;\n  int32 collective_type = 5;\n  DeviceAttributes device_attributes = 6;\n\n  reserved 4;\n}\n\n// Gives the complete membership of the group identified by group_key.\nmessage CompleteGroupResponse {\n  int32 group_key = 1;\n  int32 group_size = 2;\n  string device_type = 3;\n  int32 num_tasks = 4;  // number of distinct tasks hosting the devices\n  bytes communicator_key = 7;\n  repeated DeviceAttributes device_attributes = 8;\n\n  reserved 5, 6;\n}\n\n// Supplies data about one collective op belonging to the instance identified\n// by instance_key.  Service will respond when all group_size ops have\n// become known.  Most of the data being sent is for correctness checking,\n// to ensure that all ops in the instance share common attributes.\nmessage CompleteInstanceRequest {\n  string name = 1;\n  int32 type = 2;\n  DataType data_type = 3;\n  TensorShapeProto shape = 4;\n  int32 group_key = 5;\n  int32 group_size = 6;\n  int32 instance_key = 7;\n  string device_type = 8;\n  repeated int32 subdiv_offset = 9;\n  string device = 10;\n  bool is_source = 11;\n}\n\n// Confirms that every op in the instance has consistently declared itself.\n// Also gives the source_rank in case of broadcast.\nmessage CompleteInstanceResponse {\n  int32 instance_key = 1;\n  int32 source_rank = 2;\n  reserved 3;\n}\n\n// Request for next agreed-upon step_id for the specified graph_keys.\n// This is used to enable multiple graphs containing nodes from\n// a common collective instance to coordinate using the same step_ids.\nmessage GetStepSequenceRequest {\n  repeated int64 graph_key = 1;\n}\n\nmessage StepSequence {\n  int64 graph_key = 1;\n  int64 next_step_id = 2;\n}\n\n// Next valid step_ids for one or more graph_keys.\nmessage GetStepSequenceResponse {\n  repeated StepSequence step_sequence = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow/core/protobuf/worker_service.proto",
    "content": "/* Copyright 2016 The TensorFlow Authors. All Rights Reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n==============================================================================*/\n\nsyntax = \"proto3\";\n\npackage tensorflow.grpc;\n\nimport \"tensorflow/core/protobuf/worker.proto\";\n\noption java_outer_classname = \"WorkerServiceProtos\";\noption java_multiple_files = true;\noption java_package = \"org.tensorflow.distruntime\";\noption go_package = \"github.com/tensorflow/tensorflow/tensorflow/go/core/protobuf/for_core_protos_go_proto\";\n\n////////////////////////////////////////////////////////////////////////////////\n//\n// WorkerService defines a TensorFlow service that executes dataflow\n// graphs on a set of local devices, on behalf of a MasterService.\n//\n// A worker service keeps track of multiple \"registered graphs\". Each\n// registered graph is a subgraph of a client's graph, corresponding to\n// only the nodes that should execute on this worker (and any\n// additional nodes necessary for inter-process communication using\n// the `RecvTensor` method).\n//\n////////////////////////////////////////////////////////////////////////////////\n\nservice WorkerService {\n  // See worker.proto for details.\n  rpc GetStatus(GetStatusRequest) returns (GetStatusResponse);\n\n  // See worker.proto for details.\n  rpc CreateWorkerSession(CreateWorkerSessionRequest)\n      returns (CreateWorkerSessionResponse);\n\n  // See worker.proto for details.\n  rpc DeleteWorkerSession(DeleteWorkerSessionRequest)\n      returns (DeleteWorkerSessionResponse);\n\n  // See worker.proto for details.\n  rpc RegisterGraph(RegisterGraphRequest) returns (RegisterGraphResponse);\n\n  // See worker.proto for details.\n  rpc DeregisterGraph(DeregisterGraphRequest) returns (DeregisterGraphResponse);\n\n  // See worker.proto for details.\n  rpc RunGraph(RunGraphRequest) returns (RunGraphResponse);\n\n  // See worker.proto for details.\n  rpc CleanupGraph(CleanupGraphRequest) returns (CleanupGraphResponse);\n\n  // See worker.proto for details.\n  rpc CleanupAll(CleanupAllRequest) returns (CleanupAllResponse);\n\n  // See worker.proto for details.\n  rpc RecvTensor(RecvTensorRequest) returns (RecvTensorResponse) {\n    // RecvTensor Method\n  }\n\n  // See worker.proto for details.\n  rpc Logging(LoggingRequest) returns (LoggingResponse);\n\n  // See worker.proto for details.\n  rpc Tracing(TracingRequest) returns (TracingResponse);\n\n  // See worker.proto for details.\n  rpc RecvBuf(RecvBufRequest) returns (RecvBufResponse) {}\n\n  // See worker.proto for details.\n  rpc GetStepSequence(GetStepSequenceRequest) returns (GetStepSequenceResponse);\n\n  // See worker.proto for details.\n  rpc CompleteGroup(CompleteGroupRequest) returns (CompleteGroupResponse);\n\n  // See worker.proto for details.\n  rpc CompleteInstance(CompleteInstanceRequest)\n      returns (CompleteInstanceResponse);\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/classification.proto",
    "content": "syntax = \"proto3\";\n\noption cc_enable_arenas = true;\n\nimport \"tensorflow_serving/apis/input.proto\";\nimport \"tensorflow_serving/apis/model.proto\";\n\npackage tensorflow.serving;\n\n// A single class.\nmessage Class {\n  // Label or name of the class.\n  string label = 1;\n  // Score for this class (e.g., the probability the item belongs to this\n  // class). As per the proto3 default-value semantics, if the score is missing,\n  // it should be treated as 0.\n  float score = 2;\n}\n\n// List of classes for a single item (tensorflow.Example).\nmessage Classifications {\n  repeated Class classes = 1;\n}\n\n// Contains one result per input example, in the same order as the input in\n// ClassificationRequest.\nmessage ClassificationResult {\n  repeated Classifications classifications = 1;\n}\n\n// RPC Interfaces\n\nmessage ClassificationRequest {\n  // Model Specification. If version is not specified, will use the latest\n  // (numerical) version.\n  ModelSpec model_spec = 1;\n\n  // Input data.\n  tensorflow.serving.Input input = 2;\n}\n\nmessage ClassificationResponse {\n  // Effective Model Specification used for classification.\n  ModelSpec model_spec = 2;\n\n  // Result of the classification.\n  ClassificationResult result = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/get_model_metadata.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\noption cc_enable_arenas = true;\n\nimport \"google/protobuf/any.proto\";\nimport \"tensorflow/core/protobuf/meta_graph.proto\";\nimport \"tensorflow_serving/apis/model.proto\";\n\n// Message returned for \"signature_def\" field.\nmessage SignatureDefMap {\n  map<string, SignatureDef> signature_def = 1;\n};\n\nmessage GetModelMetadataRequest {\n  // Model Specification indicating which model we are querying for metadata.\n  // If version is not specified, will use the latest (numerical) version.\n  ModelSpec model_spec = 1;\n  // Metadata fields to get. Currently supported: \"signature_def\".\n  repeated string metadata_field = 2;\n}\n\nmessage GetModelMetadataResponse {\n  // Model Specification indicating which model this metadata belongs to.\n  ModelSpec model_spec = 1;\n  // Map of metadata field name to metadata field. The options for metadata\n  // field name are listed in GetModelMetadataRequest. Currently supported:\n  // \"signature_def\".\n  map<string, google.protobuf.Any> metadata = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/get_model_status.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\n\nimport \"tensorflow_serving/apis/model.proto\";\nimport \"tensorflow_serving/apis/status.proto\";\n\noption cc_enable_arenas = true;\n\n// GetModelStatusRequest contains a ModelSpec indicating the model for which\n// to get status.\nmessage GetModelStatusRequest {\n  // Model Specification. If version is not specified, information about all\n  // versions of the model will be returned. If a version is specified, the\n  // status of only that version will be returned.\n  ModelSpec model_spec = 1;\n}\n\n// Version number, state, and status for a single version of a model.\nmessage ModelVersionStatus {\n  // Model version.\n  int64 version = 1;\n\n  // States that map to ManagerState enum in\n  // tensorflow_serving/core/servable_state.h\n  enum State {\n    // Default value.\n    UNKNOWN = 0;\n\n    // The manager is tracking this servable, but has not initiated any action\n    // pertaining to it.\n    START = 10;\n\n    // The manager has decided to load this servable. In particular, checks\n    // around resource availability and other aspects have passed, and the\n    // manager is about to invoke the loader's Load() method.\n    LOADING = 20;\n\n    // The manager has successfully loaded this servable and made it available\n    // for serving (i.e. GetServableHandle(id) will succeed). To avoid races,\n    // this state is not reported until *after* the servable is made\n    // available.\n    AVAILABLE = 30;\n\n    // The manager has decided to make this servable unavailable, and unload\n    // it. To avoid races, this state is reported *before* the servable is\n    // made unavailable.\n    UNLOADING = 40;\n\n    // This servable has reached the end of its journey in the manager. Either\n    // it loaded and ultimately unloaded successfully, or it hit an error at\n    // some point in its lifecycle.\n    END = 50;\n  }\n\n  // Model state.\n  State state = 2;\n\n  // Model status.\n  StatusProto status = 3;\n}\n\n// Response for ModelStatusRequest on successful run.\nmessage GetModelStatusResponse {\n  // Version number and status information for applicable model version(s).\n  repeated ModelVersionStatus model_version_status = 1\n      [json_name = \"model_version_status\"];\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/inference.proto",
    "content": "// This file contains messages for various machine learning inferences\n// such as regression and classification.\n//\n// In many applications more than one type of inference is desired for a single\n// input.  For example, given meteorologic data an application may want to\n// perform a classification to determine if we should expect rain, snow or sun\n// and also perform a regression to predict the temperature.\n// Sharing the single input data between two inference tasks can be accomplished\n// using MultiInferenceRequest and MultiInferenceResponse.\n\nsyntax = \"proto3\";\n\noption cc_enable_arenas = true;\n\nimport \"tensorflow_serving/apis/classification.proto\";\nimport \"tensorflow_serving/apis/input.proto\";\nimport \"tensorflow_serving/apis/model.proto\";\nimport \"tensorflow_serving/apis/regression.proto\";\n\npackage tensorflow.serving;\n\n// Inference request such as classification, regression, etc...\nmessage InferenceTask {\n  // Model Specification. If version is not specified, will use the latest\n  // (numerical) version.\n  // All ModelSpecs in a MultiInferenceRequest must access the same model name.\n  ModelSpec model_spec = 1;\n\n  // Signature's method_name. Should be one of the method names defined in\n  // third_party/tensorflow/python/saved_model/signature_constants.py.\n  // e.g. \"tensorflow/serving/classify\".\n  string method_name = 2;\n}\n\n// Inference result, matches the type of request or is an error.\nmessage InferenceResult {\n  ModelSpec model_spec = 1;\n\n  oneof result {\n    ClassificationResult classification_result = 2;\n    RegressionResult regression_result = 3;\n  }\n}\n\n// Inference request containing one or more requests.\nmessage MultiInferenceRequest {\n  // Inference tasks.\n  repeated InferenceTask tasks = 1;\n\n  // Input data.\n  Input input = 2;\n}\n\n// Inference request containing one or more responses.\nmessage MultiInferenceResponse {\n  // List of results; one for each InferenceTask in the request, returned in the\n  // same order as the request.\n  repeated InferenceResult results = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/input.proto",
    "content": "// Input used in serving APIs.  Based on the tensorflow.Example family of\n// feature representations.\n\nsyntax = \"proto3\";\n\noption cc_enable_arenas = true;\n\nimport \"tensorflow/core/example/example.proto\";\n\npackage tensorflow.serving;\n\n// Specifies one or more fully independent input Examples.\n// See examples at:\n//     https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/example/example.proto\nmessage ExampleList {\n  repeated tensorflow.Example examples = 1;\n}\n\n// Specifies one or more independent input Examples, with a common context\n// Example.\n//\n// The common use case for context is to cleanly and optimally specify some\n// features that are common across multiple examples.\n//\n// See example below with a search query as the context and multiple restaurants\n// to perform some inference on.\n//\n// context: {\n//   features: {\n//     feature: {\n//       key  : \"query\"\n//       value: {\n//         bytes_list: {\n//           value: [ \"pizza\" ]\n//         }\n//       }\n//     }\n//   }\n// }\n// examples: {\n//   features: {\n//     feature: {\n//       key  : \"cuisine\"\n//       value: {\n//         bytes_list: {\n//           value: [ \"Pizzeria\" ]\n//         }\n//       }\n//     }\n//   }\n// }\n// examples: {\n//   features: {\n//     feature: {\n//       key  : \"cuisine\"\n//       value: {\n//         bytes_list: {\n//           value: [ \"Taqueria\" ]\n//         }\n//       }\n//     }\n//   }\n// }\n//\n// Implementations of ExampleListWithContext merge the context Example into each\n// of the Examples. Note that feature keys must not be duplicated between the\n// Examples and context Example, or the behavior is undefined.\n//\n// See also:\n//     tensorflow/core/example/example.proto\n//     https://developers.google.com/protocol-buffers/docs/proto3#maps\nmessage ExampleListWithContext {\n  repeated tensorflow.Example examples = 1;\n  tensorflow.Example context = 2;\n}\n\nmessage Input {\n  oneof kind {\n    ExampleList example_list = 1 [lazy = true];\n    ExampleListWithContext example_list_with_context = 2 [lazy = true];\n  }\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/logging.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\n\nimport \"tensorflow_serving/apis/model.proto\";\nimport \"tensorflow_serving/config/logging_config.proto\";\n\noption cc_enable_arenas = true;\n\n// Metadata logged along with the request logs.\nmessage LogMetadata {\n  ModelSpec model_spec = 1;\n  SamplingConfig sampling_config = 2;\n  // List of tags used to load the relevant MetaGraphDef from SavedModel.\n  repeated string saved_model_tags = 3;\n  // TODO(b/33279154): Add more metadata as mentioned in the bug.\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/model.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\noption cc_enable_arenas = true;\n\nimport \"google/protobuf/wrappers.proto\";\n\n// Metadata for an inference request such as the model name and version.\nmessage ModelSpec {\n  // Required servable name.\n  string name = 1;\n\n  // Optional choice of which version of the model to use.\n  //\n  // Recommended to be left unset in the common case. Should be specified only\n  // when there is a strong version consistency requirement.\n  //\n  // When left unspecified, the system will serve the best available version.\n  // This is typically the latest version, though during version transitions,\n  // notably when serving on a fleet of instances, may be either the previous or\n  // new version.\n  oneof version_choice {\n    // Use this specific version number.\n    google.protobuf.Int64Value version = 2;\n\n    // Use the version associated with the given label.\n    string version_label = 4;\n  }\n\n  // A named signature to evaluate. If unspecified, the default signature will\n  // be used.\n  string signature_name = 3;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/model_management.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\n\nimport \"tensorflow_serving/apis/status.proto\";\nimport \"tensorflow_serving/config/model_server_config.proto\";\n\noption cc_enable_arenas = true;\n\nmessage ReloadConfigRequest {\n  ModelServerConfig config = 1;\n}\n\nmessage ReloadConfigResponse {\n  StatusProto status = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/model_service.proto",
    "content": "syntax = \"proto3\";\n\noption cc_enable_arenas = true;\n\nimport \"tensorflow_serving/apis/get_model_status.proto\";\nimport \"tensorflow_serving/apis/model_management.proto\";\n\npackage tensorflow.serving;\n\n// ModelService provides methods to query and update the state of the server,\n// e.g. which models/versions are being served.\nservice ModelService {\n  // Gets status of model. If the ModelSpec in the request does not specify\n  // version, information about all versions of the model will be returned. If\n  // the ModelSpec in the request does specify a version, the status of only\n  // that version will be returned.\n  rpc GetModelStatus(GetModelStatusRequest) returns (GetModelStatusResponse);\n\n  // Reloads the set of served models. The new config supersedes the old one,\n  // so if a model is omitted from the new config it will be unloaded and no\n  // longer served.\n  rpc HandleReloadConfigRequest(ReloadConfigRequest)\n      returns (ReloadConfigResponse);\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/predict.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\noption cc_enable_arenas = true;\n\nimport \"tensorflow/core/framework/tensor.proto\";\nimport \"tensorflow_serving/apis/model.proto\";\n\n// PredictRequest specifies which TensorFlow model to run, as well as\n// how inputs are mapped to tensors and how outputs are filtered before\n// returning to user.\nmessage PredictRequest {\n  // Model Specification. If version is not specified, will use the latest\n  // (numerical) version.\n  ModelSpec model_spec = 1;\n\n  // Input tensors.\n  // Names of input tensor are alias names. The mapping from aliases to real\n  // input tensor names is stored in the SavedModel export as a prediction\n  // SignatureDef under the 'inputs' field.\n  map<string, TensorProto> inputs = 2;\n\n  // Output filter.\n  // Names specified are alias names. The mapping from aliases to real output\n  // tensor names is stored in the SavedModel export as a prediction\n  // SignatureDef under the 'outputs' field.\n  // Only tensors specified here will be run/fetched and returned, with the\n  // exception that when none is specified, all tensors specified in the\n  // named signature will be run/fetched and returned.\n  repeated string output_filter = 3;\n}\n\n// Response for PredictRequest on successful run.\nmessage PredictResponse {\n  // Effective Model Specification used to process PredictRequest.\n  ModelSpec model_spec = 2;\n\n  // Output tensors.\n  map<string, TensorProto> outputs = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/prediction_log.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\n\nimport \"tensorflow_serving/apis/classification.proto\";\nimport \"tensorflow_serving/apis/inference.proto\";\nimport \"tensorflow_serving/apis/logging.proto\";\nimport \"tensorflow_serving/apis/predict.proto\";\nimport \"tensorflow_serving/apis/regression.proto\";\nimport \"tensorflow_serving/apis/session_service.proto\";\n\noption cc_enable_arenas = true;\n\nmessage ClassifyLog {\n  ClassificationRequest request = 1;\n  ClassificationResponse response = 2;\n}\n\nmessage RegressLog {\n  RegressionRequest request = 1;\n  RegressionResponse response = 2;\n}\n\nmessage PredictLog {\n  PredictRequest request = 1;\n  PredictResponse response = 2;\n}\n\nmessage MultiInferenceLog {\n  MultiInferenceRequest request = 1;\n  MultiInferenceResponse response = 2;\n}\n\nmessage SessionRunLog {\n  SessionRunRequest request = 1;\n  SessionRunResponse response = 2;\n}\n\n// Logged model inference request.\nmessage PredictionLog {\n  LogMetadata log_metadata = 1;\n  oneof log_type {\n    ClassifyLog classify_log = 2;\n    RegressLog regress_log = 3;\n    PredictLog predict_log = 6;\n    MultiInferenceLog multi_inference_log = 4;\n    SessionRunLog session_run_log = 5;\n  }\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/prediction_service.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\noption cc_enable_arenas = true;\n\nimport \"tensorflow_serving/apis/classification.proto\";\nimport \"tensorflow_serving/apis/get_model_metadata.proto\";\nimport \"tensorflow_serving/apis/inference.proto\";\nimport \"tensorflow_serving/apis/predict.proto\";\nimport \"tensorflow_serving/apis/regression.proto\";\n\n// open source marker; do not remove\n// PredictionService provides access to machine-learned models loaded by\n// model_servers.\nservice PredictionService {\n  // Classify.\n  rpc Classify(ClassificationRequest) returns (ClassificationResponse);\n\n  // Regress.\n  rpc Regress(RegressionRequest) returns (RegressionResponse);\n\n  // Predict -- provides access to loaded TensorFlow model.\n  rpc Predict(PredictRequest) returns (PredictResponse);\n\n  // MultiInference API for multi-headed models.\n  rpc MultiInference(MultiInferenceRequest) returns (MultiInferenceResponse);\n\n  // GetModelMetadata - provides access to metadata for loaded models.\n  rpc GetModelMetadata(GetModelMetadataRequest)\n      returns (GetModelMetadataResponse);\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/regression.proto",
    "content": "syntax = \"proto3\";\n\noption cc_enable_arenas = true;\n\nimport \"tensorflow_serving/apis/input.proto\";\nimport \"tensorflow_serving/apis/model.proto\";\n\npackage tensorflow.serving;\n\n// Regression result for a single item (tensorflow.Example).\nmessage Regression {\n  float value = 1;\n}\n\n// Contains one result per input example, in the same order as the input in\n// RegressionRequest.\nmessage RegressionResult {\n  repeated Regression regressions = 1;\n}\n\n// RPC interfaces.\n\nmessage RegressionRequest {\n  // Model Specification. If version is not specified, will use the latest\n  // (numerical) version.\n  ModelSpec model_spec = 1;\n\n  // Input data.\n  tensorflow.serving.Input input = 2;\n}\n\nmessage RegressionResponse {\n  // Effective Model Specification used for regression.\n  ModelSpec model_spec = 2;\n\n  RegressionResult result = 1;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/session_service.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\n\nimport \"tensorflow/core/protobuf/config.proto\";\nimport \"tensorflow/core/protobuf/named_tensor.proto\";\nimport \"tensorflow_serving/apis/model.proto\";\n\noption cc_enable_arenas = true;\n\nmessage SessionRunRequest {\n  // Model Specification. If version is not specified, will use the latest\n  // (numerical) version.\n  ModelSpec model_spec = 1;\n\n  // Tensors to be fed in the step. Each feed is a named tensor.\n  repeated NamedTensorProto feed = 2;\n\n  // Fetches. A list of tensor names. The caller expects a tensor to\n  // be returned for each fetch[i] (see RunResponse.tensor). The\n  // order of specified fetches does not change the execution order.\n  repeated string fetch = 3;\n\n  // Target Nodes. A list of node names. The named nodes will be run\n  // to but their outputs will not be fetched.\n  repeated string target = 4;\n\n  // If true, treat names in feed/fetch/target as alias names than actual tensor\n  // names (that appear in the TF graph). Alias names are resolved to actual\n  // names using `SignatureDef` in SavedModel associated with the model.\n  bool tensor_name_is_alias = 6;\n\n  // Options for the run call. **Currently ignored.**\n  RunOptions options = 5;\n}\n\nmessage SessionRunResponse {\n  // Effective Model Specification used for session run.\n  ModelSpec model_spec = 3;\n\n  // NOTE: The order of the returned tensors may or may not match\n  // the fetch order specified in RunRequest.\n  repeated NamedTensorProto tensor = 1;\n\n  // Returned metadata if requested in the options.\n  RunMetadata metadata = 2;\n}\n\n// SessionService defines a service with which a client can interact to execute\n// Tensorflow model inference. The SessionService::SessionRun method is similar\n// to MasterService::RunStep of Tensorflow, except that all sessions are ready\n// to run, and you request a specific model/session with ModelSpec.\nservice SessionService {\n  // Runs inference of a given model.\n  rpc SessionRun(SessionRunRequest) returns (SessionRunResponse);\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/apis/status.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\n\nimport \"tensorflow/core/protobuf/error_codes.proto\";\n\noption cc_enable_arenas = true;\n\n// Status that corresponds to Status in\n// third_party/tensorflow/core/lib/core/status.h.\nmessage StatusProto {\n  // Error code.\n  error.Code error_code = 1 [json_name = \"error_code\"];\n\n  // Error message. Will only be set if an error was encountered.\n  string error_message = 2 [json_name = \"error_message\"];\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/config/file_system_storage_path_source.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\n\n// Config proto for FileSystemStoragePathSource.\nmessage FileSystemStoragePathSourceConfig {\n  // A policy that dictates which version(s) of a servable should be served.\n  message ServableVersionPolicy {\n    // Serve the latest versions (i.e. the ones with the highest version\n    // numbers), among those found on disk.\n    //\n    // This is the default policy, with the default number of versions as 1.\n    message Latest {\n      // Number of latest versions to serve. (The default is 1.)\n      uint32 num_versions = 1;\n    }\n\n    // Serve all versions found on disk.\n    message All {\n    }\n\n    // Serve a specific version (or set of versions).\n    //\n    // This policy is useful for rolling back to a specific version, or for\n    // canarying a specific version while still serving a separate stable\n    // version.\n    message Specific {\n      // The version numbers to serve.\n      repeated int64 versions = 1;\n    }\n\n    oneof policy_choice {\n      Latest latest = 100;\n      All all = 101;\n      Specific specific = 102;\n    }\n  }\n\n  // A servable name and base path to look for versions of the servable.\n  message ServableToMonitor {\n    // The servable name to supply in aspired-versions callback calls. Child\n    // paths of 'base_path' are considered to be versions of this servable.\n    string servable_name = 1;\n\n    // The path to monitor, i.e. look for child paths of the form base_path/123.\n    string base_path = 2;\n\n    // The policy to determines the number of versions of the servable to be\n    // served at the same time.\n    tensorflow.serving.FileSystemStoragePathSourceConfig.ServableVersionPolicy\n        servable_version_policy = 4;\n\n    reserved 3;  // Legacy version_policy definition.\n  }\n\n  // The servables to monitor for new versions, and aspire.\n  repeated ServableToMonitor servables = 5;\n\n  // A single servable name/base_path pair to monitor.\n  // DEPRECATED: Use 'servables' instead.\n  // TODO(b/30898016): Stop using these fields, and ultimately remove them here.\n  string servable_name = 1 [deprecated = true];\n  string base_path = 2 [deprecated = true];\n\n  // How long to wait between file-system polling to look for children of\n  // 'base_path', in seconds.\n  //\n  // If set to zero, filesystem will be polled exactly once. If set to a\n  // negative value (for testing use only), polling will be entirely disabled.\n  int64 file_system_poll_wait_seconds = 3;\n\n  // If true, then FileSystemStoragePathSource::Create() and ::UpdateConfig()\n  // fail if, for any configured servables, the file system doesn't currently\n  // contain at least one version under the base path.\n  // (Otherwise, it will emit a warning and keep pinging the file system to\n  // check for a version to appear later.)\n  // DEPRECATED: Use 'servable_versions_always_present' instead, which includes\n  // this behavior.\n  // TODO(b/30898016): Remove 2019-10-31 or later.\n  bool fail_if_zero_versions_at_startup = 4 [deprecated = true];\n\n  // If true, the servable is always expected to exist on the underlying\n  // filesystem. FileSystemStoragePathSource::Create() and ::UpdateConfig() will\n  // fail if, for any configured servables, the file system doesn't currently\n  // contain at least one version under the base path. In addition, if a polling\n  // loop find the base path empty, it will not unload existing servables.\n  bool servable_versions_always_present = 6;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/config/log_collector_config.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\noption cc_enable_arenas = true;\n\nmessage LogCollectorConfig {\n  // Identifies the type of the LogCollector we will use to collect these logs.\n  string type = 1;\n\n  // The prefix to use for the filenames of the logs.\n  string filename_prefix = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/config/logging_config.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\noption cc_enable_arenas = true;\n\nimport \"tensorflow_serving/config/log_collector_config.proto\";\n\nmessage SamplingConfig {\n  // Requests will be logged uniformly at random with this probability. Valid\n  // range: [0, 1.0].\n  double sampling_rate = 1;\n}\n\n// Configuration for logging query/responses.\nmessage LoggingConfig {\n  LogCollectorConfig log_collector_config = 1;\n  SamplingConfig sampling_config = 2;\n}\n"
  },
  {
    "path": "navi/navi/proto/tensorflow_serving/config/model_server_config.proto",
    "content": "syntax = \"proto3\";\n\npackage tensorflow.serving;\n\nimport \"google/protobuf/any.proto\";\nimport \"tensorflow_serving/config/file_system_storage_path_source.proto\";\nimport \"tensorflow_serving/config/logging_config.proto\";\n\noption cc_enable_arenas = true;\n\n// The type of model.\n// TODO(b/31336131): DEPRECATED.\nenum ModelType {\n  MODEL_TYPE_UNSPECIFIED = 0 [deprecated = true];\n  TENSORFLOW = 1 [deprecated = true];\n  OTHER = 2 [deprecated = true];\n}\n\n// Common configuration for loading a model being served.\nmessage ModelConfig {\n  // Name of the model.\n  string name = 1;\n\n  // Base path to the model, excluding the version directory.\n  // E.g> for a model at /foo/bar/my_model/123, where 123 is the version, the\n  // base path is /foo/bar/my_model.\n  //\n  // (This can be changed once a model is in serving, *if* the underlying data\n  // remains the same. Otherwise there are no guarantees about whether the old\n  // or new data will be used for model versions currently loaded.)\n  string base_path = 2;\n\n  // Type of model.\n  // TODO(b/31336131): DEPRECATED. Please use 'model_platform' instead.\n  ModelType model_type = 3 [deprecated = true];\n\n  // Type of model (e.g. \"tensorflow\").\n  //\n  // (This cannot be changed once a model is in serving.)\n  string model_platform = 4;\n\n  reserved 5, 9;\n\n  // Version policy for the model indicating which version(s) of the model to\n  // load and make available for serving simultaneously.\n  // The default option is to serve only the latest version of the model.\n  //\n  // (This can be changed once a model is in serving.)\n  FileSystemStoragePathSourceConfig.ServableVersionPolicy model_version_policy =\n      7;\n\n  // String labels to associate with versions of the model, allowing inference\n  // queries to refer to versions by label instead of number. Multiple labels\n  // can map to the same version, but not vice-versa.\n  //\n  // An envisioned use-case for these labels is canarying tentative versions.\n  // For example, one can assign labels \"stable\" and \"canary\" to two specific\n  // versions. Perhaps initially \"stable\" is assigned to version 0 and \"canary\"\n  // to version 1. Once version 1 passes canary, one can shift the \"stable\"\n  // label to refer to version 1 (at that point both labels map to the same\n  // version -- version 1 -- which is fine). Later once version 2 is ready to\n  // canary one can move the \"canary\" label to version 2. And so on.\n  map<string, int64> version_labels = 8;\n\n  // Configures logging requests and responses, to the model.\n  //\n  // (This can be changed once a model is in serving.)\n  LoggingConfig logging_config = 6;\n}\n\n// Static list of models to be loaded for serving.\nmessage ModelConfigList {\n  repeated ModelConfig config = 1;\n}\n\n// ModelServer config.\nmessage ModelServerConfig {\n  // ModelServer takes either a static file-based model config list or an Any\n  // proto representing custom model config that is fetched dynamically at\n  // runtime (through network RPC, custom service, etc.).\n  oneof config {\n    ModelConfigList model_config_list = 1;\n    google.protobuf.Any custom_model_config = 2;\n  }\n}\n"
  },
  {
    "path": "navi/navi/scripts/run_onnx.sh",
    "content": "#!/bin/sh\n#RUST_LOG=debug LD_LIBRARY_PATH=so/onnx/lib target/release/navi_onnx --port 30 --num-worker-threads 8 --intra-op-parallelism 8 --inter-op-parallelism 8 \\\nRUST_LOG=info LD_LIBRARY_PATH=so/onnx/lib cargo run --bin navi_onnx --features onnx -- \\\n    --port 8030 --num-worker-threads 8 \\\n    --model-check-interval-secs 30 \\\n    --modelsync-cli \"echo\" \\\n    --onnx-ep-options use_arena=true \\\n    --model-dir models/prod_home --output caligrated_probabilities  --input \"\" --intra-op-parallelism 8 --inter-op-parallelism 8 --max-batch-size 1 --batch-time-out-millis 1 \\\n    --model-dir models/prod_home1 --output caligrated_probabilities  --input \"\" --intra-op-parallelism 8 --inter-op-parallelism 8 --max-batch-size 1 --batch-time-out-millis 1 \\\n"
  },
  {
    "path": "navi/navi/scripts/run_tf2.sh",
    "content": "#!/bin/sh\nRUST_LOG=info LD_LIBRARY_PATH=so/tf2 cargo run --bin navi --features tf -- --port 30 --num-worker-threads 8 --intra-op-parallelism 8 --inter-op-parallelism 8 \\\n    --model-check-interval-secs 30 \\\n    --model-dir models/click/ \\\n    --input \"\" \\\n    --output output_0\n"
  },
  {
    "path": "navi/navi/src/batch.rs",
    "content": "use arrayvec::ArrayVec;\nuse itertools::Itertools;\nuse log::info;\nuse std::sync::Arc;\nuse tokio::sync::oneshot::Sender;\nuse tokio::time::Instant;\n\nuse crate::bootstrap::{TensorInput, TensorInputEnum};\nuse crate::cli_args::{ARGS, MODEL_SPECS};\nuse crate::{Callback, MAX_NUM_INPUTS, PredictResult};\nuse crate::metrics::{\n    BATCH_SIZE, BATCH_SIZE_BY_MODEL, BLOCKING_REQUEST_NUM, MODEL_INFERENCE_TIME_COLLECTOR,\n    NUM_BATCH_PREDICTION, NUM_BATCH_PREDICTION_BY_MODEL, NUM_BATCHES_DROPPED,\n    NUM_BATCHES_DROPPED_BY_MODEL, NUM_PREDICTION_BY_MODEL, NUM_REQUESTS_DROPPED,\n    NUM_REQUESTS_DROPPED_BY_MODEL,\n};\nuse crate::predict_service::Model;\nuse crate::tf_proto::tensorflow_serving::model_spec::VersionChoice;\nuse crate::tf_proto::tensorflow_serving::PredictRequest;\nuse crate::tf_proto::DataType;\n\n#[derive(Debug)]\npub struct BatchPredictor<T: Model> {\n    pub model: Arc<T>,\n    pub input_tensors: Vec<Vec<TensorInput>>,\n    pub callbacks: Vec<Callback>,\n    pub cur_batch_size: usize,\n    pub max_batch_size: usize,\n    pub batch_time_out_millis: u64,\n    pub queue_reset_ts: Instant,\n    pub queue_earliest_rq_ts: Instant,\n}\n\nimpl PredictRequest {\n    #[inline(always)]\n    pub fn take_input_vals(\n        &mut self,\n        inputs: &ArrayVec<String, MAX_NUM_INPUTS>,\n    ) -> Vec<TensorInput> {\n        let mut model_inputs = Vec::<TensorInput>::new();\n        for input_name in inputs.as_slice() {\n            let input_tensor = self\n                .inputs\n                .get_mut(input_name)\n                .unwrap_or_else(|| panic!(\"can't find {:?}\", input_name));\n            let dims = match &input_tensor.tensor_shape {\n                None => None,\n                Some(data) => Some(data.dim.iter().map(|d| d.size).collect_vec()),\n            };\n            match input_tensor.dtype() {\n                DataType::DtFloat => model_inputs.push(TensorInput::new(\n                    TensorInputEnum::Float(std::mem::take(&mut input_tensor.float_val)),\n                    input_name.to_string(),\n                    dims,\n                )),\n                DataType::DtDouble => model_inputs.push(TensorInput::new(\n                    TensorInputEnum::Double(std::mem::take(&mut input_tensor.double_val)),\n                    input_name.to_string(),\n                    dims,\n                )),\n                DataType::DtInt32 => model_inputs.push(TensorInput::new(\n                    TensorInputEnum::Int(std::mem::take(&mut input_tensor.int_val)),\n                    input_name.to_string(),\n                    dims,\n                )),\n                DataType::DtString => model_inputs.push(TensorInput::new(\n                    TensorInputEnum::String(std::mem::take(&mut input_tensor.string_val)),\n                    input_name.to_string(),\n                    dims,\n                )),\n                DataType::DtInt64 => model_inputs.push(TensorInput::new(\n                    TensorInputEnum::Int64(std::mem::take(&mut input_tensor.int64_val)),\n                    input_name.to_string(),\n                    dims,\n                )),\n                DataType::DtBool => model_inputs.push(TensorInput::new(\n                    TensorInputEnum::Boolean(std::mem::take(&mut input_tensor.bool_val)),\n                    input_name.to_string(),\n                    dims,\n                )),\n                _ => panic!(\"unsupport input tensor type {:?}\", input_tensor.dtype()),\n            }\n        }\n        model_inputs\n    }\n    #[inline(always)]\n    pub fn take_model_spec(&mut self) -> (String, Option<i64>) {\n        let model_spec = self.model_spec.as_mut().unwrap();\n        let version = model_spec\n            .version_choice\n            .as_ref()\n            .and_then(|choice| match choice {\n                VersionChoice::Version(version) => Some(*version),\n                _ => None,\n            });\n        (std::mem::take(&mut model_spec.name), version)\n    }\n}\n\nimpl<T: Model> Drop for BatchPredictor<T> {\n    fn drop(&mut self) {\n        info!(\n            \"drop old batch predictor for:{:}, queue:{}\",\n            self.model,\n            self.input_tensors.len()\n        );\n        if !self.input_tensors.is_empty() {\n            info!(\"now flush old predictor queue:{}\", self.input_tensors.len());\n            self.batch_predict();\n        }\n    }\n}\n\nimpl<T: Model> BatchPredictor<T> {\n    #[inline(always)]\n    pub fn push(&mut self, val: Vec<TensorInput>, resp: Sender<PredictResult>, ts: Instant) {\n        if self.input_tensors.is_empty() {\n            //only when queue is empty then we update ts to represent first request time\n            self.queue_reset_ts = Instant::now();\n            self.queue_earliest_rq_ts = ts;\n        }\n        self.cur_batch_size += 1;\n        self.input_tensors.push(val);\n        self.callbacks.push(Callback(resp, self.cur_batch_size));\n    }\n    #[inline(always)]\n    pub fn batch_predict(&mut self) {\n        BATCH_SIZE_BY_MODEL\n            .with_label_values(&[&MODEL_SPECS[self.model.model_idx()]])\n            .add(self.cur_batch_size as i64);\n        BATCH_SIZE.add(self.cur_batch_size as i64);\n        let mut batch_input_tensors = Vec::with_capacity(self.max_batch_size);\n        let mut batch_callbacks = Vec::with_capacity(self.max_batch_size);\n        let mut batch_size = 0;\n        //now we swap so we can take two queues to the blocking-send thread and reset current queues\n        std::mem::swap(&mut self.input_tensors, &mut batch_input_tensors);\n        std::mem::swap(&mut self.callbacks, &mut batch_callbacks);\n        std::mem::swap(&mut self.cur_batch_size, &mut batch_size);\n        let model = self.model.clone();\n        let batch_earliest_rq_ts = self.queue_earliest_rq_ts;\n        //info!(\"batch predict for model:{}, size:{}\", self.tf_model.export_dir, vals0.len());\n        BLOCKING_REQUEST_NUM.inc();\n        tokio::task::spawn_blocking(move || {\n            //proactively drop stale batches, we drop the entire batch\n            //as long as one request in that batch is stale. We may drop more than we could this way\n            //but this should work fairly decently well\n            if (batch_earliest_rq_ts.elapsed().as_millis() as u64) < ARGS.batch_drop_millis {\n                let model_inference_time_start = Instant::now();\n                let (tensor_outs, batch_ends) =\n                    model.do_predict(batch_input_tensors, batch_size as u64);\n                MODEL_INFERENCE_TIME_COLLECTOR\n                    .with_label_values(&[&MODEL_SPECS[model.model_idx()]])\n                    .observe(model_inference_time_start.elapsed().as_millis() as f64);\n                let mut batch_starts = vec![0; tensor_outs.len()];\n                for (i, Callback(resp, _)) in batch_callbacks.into_iter().enumerate() {\n                    let mut tensors_send_back = vec![];\n                    for (j, tensor_out) in tensor_outs.iter().enumerate() {\n                        tensors_send_back.push(tensor_out.slice(batch_starts[j], batch_ends[j][i]));\n                        batch_starts[j] = batch_ends[j][i];\n                    }\n                    if resp\n                        .send(PredictResult::Ok(tensors_send_back, model.version()))\n                        .is_err()\n                    {\n                        //use dropped metrics here as this is expected under high load\n                        NUM_REQUESTS_DROPPED.inc();\n                        NUM_REQUESTS_DROPPED_BY_MODEL\n                            .with_label_values(&[&MODEL_SPECS[model.model_idx()]])\n                            .inc();\n                    }\n                }\n            } else {\n                for Callback(resp, _) in batch_callbacks.into_iter() {\n                    if resp.send(PredictResult::DropDueToOverload).is_err() {\n                        NUM_REQUESTS_DROPPED.inc();\n                        NUM_REQUESTS_DROPPED_BY_MODEL\n                            .with_label_values(&[&MODEL_SPECS[model.model_idx()]])\n                            .inc();\n                    }\n                }\n                NUM_BATCHES_DROPPED.inc();\n                NUM_BATCHES_DROPPED_BY_MODEL\n                    .with_label_values(&[&MODEL_SPECS[model.model_idx()]])\n                    .inc();\n            }\n            BLOCKING_REQUEST_NUM.dec();\n        });\n        NUM_BATCH_PREDICTION.inc();\n        NUM_BATCH_PREDICTION_BY_MODEL\n            .with_label_values(&[&MODEL_SPECS[self.model.model_idx()]])\n            .inc();\n        // Note:\n        //  self.cur_batch_size is swapped with batch_size above\n        //  Use the local variable batch_size here\n        NUM_PREDICTION_BY_MODEL\n            .with_label_values(&[&MODEL_SPECS[self.model.model_idx()]])\n            .inc_by(batch_size as u64);\n    }\n    #[inline(always)]\n    pub fn duration_past(&self, millis: u64) -> bool {\n        self.queue_reset_ts.elapsed().as_millis() as u64 >= millis\n    }\n}\n"
  },
  {
    "path": "navi/navi/src/bin/navi.rs",
    "content": "use anyhow::Result;\nuse log::info;\nuse navi::cli_args::{ARGS, MODEL_SPECS};\nuse navi::cores::validator::validatior::cli_validator;\nuse navi::tf_model::tf::TFModel;\nuse navi::{bootstrap, metrics};\nuse sha256::digest;\n\nfn main() -> Result<()> {\n    env_logger::init();\n    cli_validator::validate_input_args();\n    //only validate in for tf as other models don't have this\n    assert_eq!(MODEL_SPECS.len(), ARGS.serving_sig.len());\n    metrics::register_custom_metrics();\n\n    //load all the custom ops - comma seperaed\n    if let Some(ref customops_lib) = ARGS.customops_lib {\n        for op_lib in customops_lib.split(\",\") {\n            load_custom_op(op_lib);\n        }\n    }\n\n    // versioning the customop so library\n    bootstrap::bootstrap(TFModel::new)\n}\n\nfn load_custom_op(lib_path: &str) -> () {\n    let res = tensorflow::Library::load(lib_path);\n    info!(\"{} load status:{:?}\", lib_path, res);\n    let customop_version_num = get_custom_op_version(lib_path);\n    // Last OP version is recorded\n    metrics::CUSTOMOP_VERSION.set(customop_version_num);\n}\n\n//fn get_custom_op_version(customops_lib: &String) -> i64 {\nfn get_custom_op_version(customops_lib: &str) -> i64 {\n    let customop_bytes = std::fs::read(customops_lib).unwrap(); // Vec<u8>\n    let customop_hash = digest(customop_bytes.as_slice());\n    //conver the last 4 hex digits to version number as prometheus metrics doesn't support string, the total space is 16^4 == 65536\n    let customop_version_num =\n        i64::from_str_radix(&customop_hash[customop_hash.len() - 4..], 16).unwrap();\n    info!(\n        \"customop hash: {}, version_number: {}\",\n        customop_hash, customop_version_num\n    );\n    customop_version_num\n}\n"
  },
  {
    "path": "navi/navi/src/bin/navi_onnx.rs",
    "content": "use anyhow::Result;\nuse log::info;\nuse navi::cli_args::{ARGS, MODEL_SPECS};\nuse navi::onnx_model::onnx::OnnxModel;\nuse navi::{bootstrap, metrics};\n\nfn main() -> Result<()> {\n    env_logger::init();\n    info!(\"global: {:?}\", ARGS.onnx_global_thread_pool_options);\n    let assert_session_params = if ARGS.onnx_global_thread_pool_options.is_empty() {\n        // std::env::set_var(\"OMP_NUM_THREADS\", \"1\");\n        info!(\"now we use per session thread pool\");\n        MODEL_SPECS.len()\n    }\n    else {\n       info!(\"now we use global thread pool\");\n        0\n    };\n    assert_eq!(assert_session_params, ARGS.inter_op_parallelism.len());\n    assert_eq!(assert_session_params, ARGS.inter_op_parallelism.len());\n\n    metrics::register_custom_metrics();\n    bootstrap::bootstrap(OnnxModel::new)\n}\n"
  },
  {
    "path": "navi/navi/src/bin/navi_torch.rs",
    "content": "use anyhow::Result;\nuse log::info;\nuse navi::cli_args::ARGS;\nuse navi::metrics;\nuse navi::torch_model::torch::TorchModel;\n\nfn main() -> Result<()> {\n    env_logger::init();\n    //torch only has global threadpool settings versus tf has per model threadpool settings\n    assert_eq!(1, ARGS.inter_op_parallelism.len());\n    assert_eq!(1, ARGS.intra_op_parallelism.len());\n    //TODO for now we, we assume each model's output has only 1 tensor.\n    //this will be lifted once torch_model properly implements mtl outputs\n    tch::set_num_interop_threads(ARGS.inter_op_parallelism[0].parse()?);\n    tch::set_num_threads(ARGS.intra_op_parallelism[0].parse()?);\n    info!(\"torch custom ops not used for now\");\n    metrics::register_custom_metrics();\n    navi::bootstrap::bootstrap(TorchModel::new)\n}\n"
  },
  {
    "path": "navi/navi/src/bootstrap.rs",
    "content": "use anyhow::Result;\nuse log::{info, warn};\nuse x509_parser::{prelude::{parse_x509_pem}, parse_x509_certificate};\nuse std::collections::HashMap;\nuse tokio::time::Instant;\nuse tonic::{\n    Request,\n    Response, Status, transport::{Certificate, Identity, Server, ServerTlsConfig},\n};\n\n// protobuf related\nuse crate::tf_proto::tensorflow_serving::{\n    ClassificationRequest, ClassificationResponse, GetModelMetadataRequest,\n    GetModelMetadataResponse, MultiInferenceRequest, MultiInferenceResponse, PredictRequest,\n    PredictResponse, RegressionRequest, RegressionResponse,\n};\nuse crate::{kf_serving::{\n    grpc_inference_service_server::GrpcInferenceService, ModelInferRequest, ModelInferResponse,\n    ModelMetadataRequest, ModelMetadataResponse, ModelReadyRequest, ModelReadyResponse,\n    ServerLiveRequest, ServerLiveResponse, ServerMetadataRequest, ServerMetadataResponse,\n    ServerReadyRequest, ServerReadyResponse,\n}, ModelFactory, tf_proto::tensorflow_serving::prediction_service_server::{\n    PredictionService, PredictionServiceServer,\n}, VERSION, NAME};\n\nuse crate::PredictResult;\nuse crate::cli_args::{ARGS, INPUTS, OUTPUTS};\nuse crate::metrics::{\n    NAVI_VERSION, NUM_PREDICTIONS, NUM_REQUESTS_FAILED, NUM_REQUESTS_FAILED_BY_MODEL,\n    NUM_REQUESTS_RECEIVED, NUM_REQUESTS_RECEIVED_BY_MODEL, RESPONSE_TIME_COLLECTOR,\n    CERT_EXPIRY_EPOCH\n};\nuse crate::predict_service::{Model, PredictService};\nuse crate::tf_proto::tensorflow_serving::model_spec::VersionChoice::Version;\nuse crate::tf_proto::tensorflow_serving::ModelSpec;\n\n#[derive(Debug)]\npub enum TensorInputEnum {\n    String(Vec<Vec<u8>>),\n    Int(Vec<i32>),\n    Int64(Vec<i64>),\n    Float(Vec<f32>),\n    Double(Vec<f64>),\n    Boolean(Vec<bool>),\n}\n\n#[derive(Debug)]\npub struct TensorInput {\n    pub tensor_data: TensorInputEnum,\n    pub name: String,\n    pub dims: Option<Vec<i64>>,\n}\n\nimpl TensorInput {\n    pub fn new(tensor_data: TensorInputEnum, name: String, dims: Option<Vec<i64>>) -> TensorInput {\n        TensorInput {\n            tensor_data,\n            name,\n            dims,\n        }\n    }\n}\n\nimpl TensorInputEnum {\n    #[inline(always)]\n    pub(crate) fn extend(&mut self, another: TensorInputEnum) {\n        match (self, another) {\n            (Self::String(input), Self::String(ex)) => input.extend(ex),\n            (Self::Int(input), Self::Int(ex)) => input.extend(ex),\n            (Self::Int64(input), Self::Int64(ex)) => input.extend(ex),\n            (Self::Float(input), Self::Float(ex)) => input.extend(ex),\n            (Self::Double(input), Self::Double(ex)) => input.extend(ex),\n            (Self::Boolean(input), Self::Boolean(ex)) => input.extend(ex),\n            x => panic!(\"input enum type not matched. input:{:?}, ex:{:?}\", x.0, x.1),\n        }\n    }\n    #[inline(always)]\n    pub(crate) fn merge_batch(input_tensors: Vec<Vec<TensorInput>>) -> Vec<TensorInput> {\n        input_tensors\n            .into_iter()\n            .reduce(|mut acc, e| {\n                for (i, ext) in acc.iter_mut().zip(e) {\n                    i.tensor_data.extend(ext.tensor_data);\n                }\n                acc\n            })\n            .unwrap() //invariant: we expect there's always rows in input_tensors\n    }\n}\n\n\n///entry point for tfServing gRPC\n#[tonic::async_trait]\nimpl<T: Model> GrpcInferenceService for PredictService<T> {\n    async fn server_live(\n        &self,\n        _request: Request<ServerLiveRequest>,\n    ) -> Result<Response<ServerLiveResponse>, Status> {\n        unimplemented!()\n    }\n    async fn server_ready(\n        &self,\n        _request: Request<ServerReadyRequest>,\n    ) -> Result<Response<ServerReadyResponse>, Status> {\n        unimplemented!()\n    }\n\n    async fn model_ready(\n        &self,\n        _request: Request<ModelReadyRequest>,\n    ) -> Result<Response<ModelReadyResponse>, Status> {\n        unimplemented!()\n    }\n\n    async fn server_metadata(\n        &self,\n        _request: Request<ServerMetadataRequest>,\n    ) -> Result<Response<ServerMetadataResponse>, Status> {\n        unimplemented!()\n    }\n\n    async fn model_metadata(\n        &self,\n        _request: Request<ModelMetadataRequest>,\n    ) -> Result<Response<ModelMetadataResponse>, Status> {\n        unimplemented!()\n    }\n\n    async fn model_infer(\n        &self,\n        _request: Request<ModelInferRequest>,\n    ) -> Result<Response<ModelInferResponse>, Status> {\n        unimplemented!()\n    }\n}\n\n#[tonic::async_trait]\nimpl<T: Model> PredictionService for PredictService<T> {\n    async fn classify(\n        &self,\n        _request: Request<ClassificationRequest>,\n    ) -> Result<Response<ClassificationResponse>, Status> {\n        unimplemented!()\n    }\n    async fn regress(\n        &self,\n        _request: Request<RegressionRequest>,\n    ) -> Result<Response<RegressionResponse>, Status> {\n        unimplemented!()\n    }\n    async fn predict(\n        &self,\n        request: Request<PredictRequest>,\n    ) -> Result<Response<PredictResponse>, Status> {\n        NUM_REQUESTS_RECEIVED.inc();\n        let start = Instant::now();\n        let mut req = request.into_inner();\n        let (model_spec, version) = req.take_model_spec();\n        NUM_REQUESTS_RECEIVED_BY_MODEL\n            .with_label_values(&[&model_spec])\n            .inc();\n        let idx = PredictService::<T>::get_model_index(&model_spec).ok_or_else(|| {\n            Status::failed_precondition(format!(\"model spec not found:{}\", model_spec))\n        })?;\n        let input_spec = match INPUTS[idx].get() {\n            Some(input) => input,\n            _ => return Err(Status::not_found(format!(\"model input spec {}\", idx))),\n        };\n        let input_val = req.take_input_vals(input_spec);\n        self.predict(idx, version, input_val, start)\n            .await\n            .map_or_else(\n                |e| {\n                    NUM_REQUESTS_FAILED.inc();\n                    NUM_REQUESTS_FAILED_BY_MODEL\n                        .with_label_values(&[&model_spec])\n                        .inc();\n                    Err(Status::internal(e.to_string()))\n                },\n                |res| {\n                    RESPONSE_TIME_COLLECTOR\n                        .with_label_values(&[&model_spec])\n                        .observe(start.elapsed().as_millis() as f64);\n\n                    match res {\n                        PredictResult::Ok(tensors, version) => {\n                            let mut outputs = HashMap::new();\n                            NUM_PREDICTIONS.with_label_values(&[&model_spec]).inc();\n                            //FIXME: uncomment when prediction scores are normal\n                            // PREDICTION_SCORE_SUM\n                            // .with_label_values(&[&model_spec])\n                            // .inc_by(tensors[0]as f64);\n                            for (tp, output_name) in tensors\n                                .into_iter()\n                                .map(|tensor| tensor.create_tensor_proto())\n                                .zip(OUTPUTS[idx].iter())\n                            {\n                                outputs.insert(output_name.to_owned(), tp);\n                            }\n                            let reply = PredictResponse {\n                                model_spec: Some(ModelSpec {\n                                    version_choice: Some(Version(version)),\n                                    ..Default::default()\n                                }),\n                                outputs,\n                            };\n                            Ok(Response::new(reply))\n                        }\n                        PredictResult::DropDueToOverload => Err(Status::resource_exhausted(\"\")),\n                        PredictResult::ModelNotFound(idx) => {\n                            Err(Status::not_found(format!(\"model index {}\", idx)))\n                        },\n                        PredictResult::ModelNotReady(idx) => {\n                            Err(Status::unavailable(format!(\"model index {}\", idx)))\n                        }\n                        PredictResult::ModelVersionNotFound(idx, version) => Err(\n                            Status::not_found(format!(\"model index:{}, version {}\", idx, version)),\n                        ),\n                    }\n                },\n            )\n    }\n\n    async fn multi_inference(\n        &self,\n        _request: Request<MultiInferenceRequest>,\n    ) -> Result<Response<MultiInferenceResponse>, Status> {\n        unimplemented!()\n    }\n    async fn get_model_metadata(\n        &self,\n        _request: Request<GetModelMetadataRequest>,\n    ) -> Result<Response<GetModelMetadataResponse>, Status> {\n        unimplemented!()\n    }\n}\n\n// A function that takes a timestamp as input and returns a ticker stream\nfn report_expiry(expiry_time: i64) {\n    info!(\"Certificate expires at epoch: {:?}\", expiry_time);\n    CERT_EXPIRY_EPOCH.set(expiry_time as i64);\n}\n\npub fn bootstrap<T: Model>(model_factory: ModelFactory<T>) -> Result<()> {\n    info!(\"package: {}, version: {}, args: {:?}\", NAME, VERSION, *ARGS);\n    //we follow SemVer. So here we assume MAJOR.MINOR.PATCH\n    let parts = VERSION\n        .split(\".\")\n        .map(|v| v.parse::<i64>())\n        .collect::<std::result::Result<Vec<_>, _>>()?;\n    if let [major, minor, patch] = &parts[..] {\n        NAVI_VERSION.set(major * 1000_000 + minor * 1000 + patch);\n    } else {\n        warn!(\n            \"version {} doesn't follow SemVer conversion of MAJOR.MINOR.PATCH\",\n            VERSION\n        );\n    }\n\n    \n    tokio::runtime::Builder::new_multi_thread()\n        .thread_name(\"async worker\")\n        .worker_threads(ARGS.num_worker_threads)\n        .max_blocking_threads(ARGS.max_blocking_threads)\n        .enable_all()\n        .build()\n        .unwrap()\n        .block_on(async {\n            #[cfg(feature = \"navi_console\")]\n            console_subscriber::init();\n            let addr = format!(\"0.0.0.0:{}\", ARGS.port).parse()?;\n\n            let ps = PredictService::init(model_factory).await;\n\n            let mut builder = if ARGS.ssl_dir.is_empty() {\n                Server::builder()\n            } else {\n                // Read the pem file as a string\n                let pem_str = std::fs::read_to_string(format!(\"{}/server.crt\", ARGS.ssl_dir)).unwrap();\n                let res = parse_x509_pem(&pem_str.as_bytes());\n                match res {\n                    Ok((rem, pem_2)) => {\n                        assert!(rem.is_empty());\n                        assert_eq!(pem_2.label, String::from(\"CERTIFICATE\"));\n                        let res_x509 = parse_x509_certificate(&pem_2.contents);\n                        info!(\"Certificate label: {}\", pem_2.label);\n                        assert!(res_x509.is_ok());\n                        report_expiry(res_x509.unwrap().1.validity().not_after.timestamp());\n                    },\n                    _ => panic!(\"PEM parsing failed: {:?}\", res),\n                }\n\n                let key = tokio::fs::read(format!(\"{}/server.key\", ARGS.ssl_dir))\n                    .await\n                    .expect(\"can't find key file\");\n                let crt = tokio::fs::read(format!(\"{}/server.crt\", ARGS.ssl_dir))\n                    .await\n                    .expect(\"can't find crt file\");\n                let chain = tokio::fs::read(format!(\"{}/server.chain\", ARGS.ssl_dir))\n                    .await\n                    .expect(\"can't find chain file\");\n                let mut pem = Vec::new();\n                pem.extend(crt);\n                pem.extend(chain);\n                let identity = Identity::from_pem(pem.clone(), key);\n                let client_ca_cert = Certificate::from_pem(pem.clone());\n                let tls = ServerTlsConfig::new()\n                    .identity(identity) \n                    .client_ca_root(client_ca_cert);\n                Server::builder()\n                    .tls_config(tls)\n                    .expect(\"fail to config SSL\")\n            };\n\n            info!(\n                \"Prometheus server started: 0.0.0.0: {}\",\n                ARGS.prometheus_port\n            );\n\n            let ps_server = builder\n                .add_service(PredictionServiceServer::new(ps).accept_gzip().send_gzip())\n                .serve(addr);\n            info!(\"Prediction server started: {}\", addr);\n            ps_server.await.map_err(anyhow::Error::msg)\n        })\n}\n"
  },
  {
    "path": "navi/navi/src/cli_args.rs",
    "content": "use crate::{MAX_NUM_INPUTS, MAX_NUM_MODELS, MAX_NUM_OUTPUTS};\nuse arrayvec::ArrayVec;\nuse clap::Parser;\nuse log::info;\nuse once_cell::sync::OnceCell;\nuse std::error::Error;\nuse time::OffsetDateTime;\nuse time::format_description::well_known::Rfc3339;\n#[derive(Parser, Debug, Clone)]\n///Navi is configured through CLI arguments(for now) defined below.\n//TODO: use clap_serde to make it config file driven\npub struct Args {\n    #[clap(short, long, help = \"gRPC port Navi runs ons\")]\n    pub port: i32,\n    #[clap(long, default_value_t = 9000, help = \"prometheus metrics port\")]\n    pub prometheus_port: u16,\n    #[clap(\n        short,\n        long,\n        default_value_t = 1,\n        help = \"number of worker threads for tokio async runtime\"\n    )]\n    pub num_worker_threads: usize,\n    #[clap(\n        long,\n        default_value_t = 14,\n        help = \"number of blocking threads in tokio blocking thread pool\"\n    )]\n    pub max_blocking_threads: usize,\n    #[clap(long, default_value = \"16\", help = \"maximum batch size for a batch\")]\n    pub max_batch_size: Vec<String>,\n    #[clap(\n        short,\n        long,\n        default_value = \"2\",\n        help = \"max wait time for accumulating a batch\"\n    )]\n    pub batch_time_out_millis: Vec<String>,\n    #[clap(\n        long,\n        default_value_t = 90,\n        help = \"threshold to start dropping batches under stress\"\n    )]\n    pub batch_drop_millis: u64,\n    #[clap(\n        long,\n        default_value_t = 300,\n        help = \"polling interval for new version of a model and META.json config\"\n    )]\n    pub model_check_interval_secs: u64,\n    #[clap(\n        short,\n        long,\n        default_value = \"models/pvideo/\",\n        help = \"root directory for models\"\n    )]\n    pub model_dir: Vec<String>,\n    #[clap(\n        long,\n        help = \"directory containing META.json config. separate from model_dir to facilitate remote config management\"\n    )]\n    pub meta_json_dir: Option<String>,\n    #[clap(short, long, default_value = \"\", help = \"directory for ssl certs\")]\n    pub ssl_dir: String,\n    #[clap(\n        long,\n        help = \"call out to external process to check model updates. custom logic can be written to pull from hdfs, gcs etc\"\n    )]\n    pub modelsync_cli: Option<String>,\n    #[clap(\n        long,\n        default_value_t = 1,\n        help = \"specify how many versions Navi retains in memory. good for cases of rolling model upgrade\"\n    )]\n    pub versions_per_model: usize,\n    #[clap(\n        short,\n        long,\n        help = \"most runtimes support loading ops custom writen. currently only implemented for TF\"\n    )]\n    pub customops_lib: Option<String>,\n    #[clap(\n        long,\n        default_value = \"8\",\n        help = \"number of threads to paralleling computations inside an op\"\n    )]\n    pub intra_op_parallelism: Vec<String>,\n    #[clap(\n        long,\n        help = \"number of threads to parallelize computations of the graph\"\n    )]\n    pub inter_op_parallelism: Vec<String>,\n    #[clap(\n        long,\n        help = \"signature of a serving. only TF\"\n    )]\n    pub serving_sig: Vec<String>,\n    #[clap(long, default_value = \"examples\", help = \"name of each input tensor\")]\n    pub input: Vec<String>,\n    #[clap(long, default_value = \"output_0\", help = \"name of each output tensor\")]\n    pub output: Vec<String>,\n    #[clap(\n        long,\n        default_value_t = 500,\n        help = \"max warmup records to use. warmup only implemented for TF\"\n    )]\n    pub max_warmup_records: usize,\n    #[clap(long, value_parser = Args::parse_key_val::<String, String>, value_delimiter=',')]\n    pub onnx_global_thread_pool_options: Vec<(String, String)>,\n    #[clap(\n    long,\n    default_value = \"true\",\n    help = \"when to use graph parallelization. only for ONNX\"\n    )]\n    pub onnx_use_parallel_mode: String,\n    // #[clap(long, default_value = \"false\")]\n    // pub onnx_use_onednn: String,\n    #[clap(\n        long,\n        default_value = \"true\",\n        help = \"trace internal memory allocation and generate bulk memory allocations. only for ONNX. turn if off if batch size dynamic\"\n    )]\n    pub onnx_use_memory_pattern: String,\n    #[clap(long, value_parser = Args::parse_key_val::<String, String>, value_delimiter=',')]\n    pub onnx_ep_options: Vec<(String, String)>,\n    #[clap(long, help = \"choice of gpu EPs for ONNX: cuda or tensorrt\")]\n    pub onnx_gpu_ep: Option<String>,\n    #[clap(\n        long,\n        default_value = \"home\",\n        help = \"converter for various input formats\"\n    )]\n    pub onnx_use_converter: Option<String>,\n    #[clap(\n        long,\n        help = \"whether to enable runtime profiling. only implemented for ONNX for now\"\n    )]\n    pub profiling: Option<String>,\n    #[clap(\n        long,\n        default_value = \"\",\n        help = \"metrics reporting for discrete features. only for Home converter for now\"\n    )]\n    pub onnx_report_discrete_feature_ids: Vec<String>,\n    #[clap(\n        long,\n        default_value = \"\",\n        help = \"metrics reporting for continuous features. only for Home converter for now\"\n    )]\n    pub onnx_report_continuous_feature_ids: Vec<String>,\n}\n\nimpl Args {\n    pub fn get_model_specs(model_dir: Vec<String>) -> Vec<String> {\n        let model_specs = model_dir\n            .iter()\n            //let it panic if some model_dir are wrong\n            .map(|dir| {\n                dir.trim_end_matches('/')\n                    .rsplit_once('/')\n                    .unwrap()\n                    .1\n                    .to_owned()\n            })\n            .collect();\n        info!(\"all model_specs: {:?}\", model_specs);\n        model_specs\n    }\n    pub fn version_str_to_epoch(dt_str: &str) -> Result<i64, anyhow::Error> {\n        dt_str\n            .parse::<i64>()\n            .or_else(|_| {\n                let ts = OffsetDateTime::parse(dt_str, &Rfc3339)\n                    .map(|d| (d.unix_timestamp_nanos()/1_000_000) as i64);\n                if ts.is_ok() {\n                    info!(\"original version {} -> {}\", dt_str, ts.unwrap());\n                }\n                ts\n            })\n            .map_err(anyhow::Error::msg)\n    }\n    /// Parse a single key-value pair\n    fn parse_key_val<T, U>(s: &str) -> Result<(T, U), Box<dyn Error + Send + Sync + 'static>>\n    where\n        T: std::str::FromStr,\n        T::Err: Error + Send + Sync + 'static,\n        U: std::str::FromStr,\n        U::Err: Error + Send + Sync + 'static,\n    {\n        let pos = s\n            .find('=')\n            .ok_or_else(|| format!(\"invalid KEY=value: no `=` found in `{}`\", s))?;\n        Ok((s[..pos].parse()?, s[pos + 1..].parse()?))\n    }\n}\n\nlazy_static! {\n    pub static ref ARGS: Args = Args::parse();\n    pub static ref MODEL_SPECS: ArrayVec<String, MAX_NUM_MODELS> = {\n        let mut specs = ArrayVec::<String, MAX_NUM_MODELS>::new();\n        Args::get_model_specs(ARGS.model_dir.clone())\n            .into_iter()\n            .for_each(|m| specs.push(m));\n        specs\n    };\n    pub static ref INPUTS: ArrayVec<OnceCell<ArrayVec<String, MAX_NUM_INPUTS>>, MAX_NUM_MODELS> = {\n        let mut inputs =\n            ArrayVec::<OnceCell<ArrayVec<String, MAX_NUM_INPUTS>>, MAX_NUM_MODELS>::new();\n        for (idx, o) in ARGS.input.iter().enumerate() {\n            if o.trim().is_empty() {\n                info!(\"input spec is empty for model {}, auto detect later\", idx);\n                inputs.push(OnceCell::new());\n            } else {\n                inputs.push(OnceCell::with_value(\n                    o.split(\",\")\n                        .map(|s| s.to_owned())\n                        .collect::<ArrayVec<String, MAX_NUM_INPUTS>>(),\n                ));\n            }\n        }\n        info!(\"all inputs:{:?}\", inputs);\n        inputs\n    };\n    pub static ref OUTPUTS: ArrayVec<ArrayVec<String, MAX_NUM_OUTPUTS>, MAX_NUM_MODELS> = {\n        let mut outputs = ArrayVec::<ArrayVec<String, MAX_NUM_OUTPUTS>, MAX_NUM_MODELS>::new();\n        for o in ARGS.output.iter() {\n            outputs.push(\n                o.split(\",\")\n                    .map(|s| s.to_owned())\n                    .collect::<ArrayVec<String, MAX_NUM_OUTPUTS>>(),\n            );\n        }\n        info!(\"all outputs:{:?}\", outputs);\n        outputs\n    };\n}\n"
  },
  {
    "path": "navi/navi/src/cores/validator.rs",
    "content": "pub mod validatior {\n    pub mod cli_validator {\n        use crate::cli_args::{ARGS, MODEL_SPECS};\n\n        pub fn validate_input_args() {\n            assert_eq!(MODEL_SPECS.len(), ARGS.inter_op_parallelism.len());\n            assert_eq!(MODEL_SPECS.len(), ARGS.intra_op_parallelism.len());\n            //TODO for now we, we assume each model's output has only 1 tensor.\n            //this will be lifted once tf_model properly implements mtl outputs\n            //assert_eq!(OUTPUTS.len(), OUTPUTS.iter().fold(0usize, |a, b| a+b.len()));\n        }\n\n        pub fn validate_ps_model_args() {\n            assert!(ARGS.versions_per_model <= 2);\n            assert!(ARGS.versions_per_model >= 1);\n            assert_eq!(MODEL_SPECS.len(), ARGS.input.len());\n            assert_eq!(MODEL_SPECS.len(), ARGS.model_dir.len());\n            assert_eq!(MODEL_SPECS.len(), ARGS.max_batch_size.len());\n            assert_eq!(MODEL_SPECS.len(), ARGS.batch_time_out_millis.len());\n        }\n    }\n}\n"
  },
  {
    "path": "navi/navi/src/lib.rs",
    "content": "#[macro_use]\nextern crate lazy_static;\nextern crate core;\n\nuse serde_json::Value;\nuse tokio::sync::oneshot::Sender;\nuse tokio::time::Instant;\nuse std::ops::Deref;\nuse itertools::Itertools;\nuse crate::bootstrap::TensorInput;\nuse crate::predict_service::Model;\nuse crate::tf_proto::{DataType, TensorProto};\n\npub mod batch;\npub mod bootstrap;\npub mod cli_args;\npub mod metrics;\npub mod onnx_model;\npub mod predict_service;\npub mod tf_model;\npub mod torch_model;\npub mod cores {\n    pub mod validator;\n}\n\npub mod tf_proto {\n    tonic::include_proto!(\"tensorflow\");\n    pub mod tensorflow_serving {\n        tonic::include_proto!(\"tensorflow.serving\");\n    }\n}\n\npub mod kf_serving {\n    tonic::include_proto!(\"inference\");\n}\n#[cfg(test)]\nmod tests {\n    use crate::cli_args::Args;\n    #[test]\n    fn test_version_string_to_epoch() {\n        assert_eq!(\n            Args::version_str_to_epoch(\"2022-12-20T10:18:53.000Z\").unwrap_or(-1),\n            1671531533000\n        );\n        assert_eq!(Args::version_str_to_epoch(\"1203444\").unwrap_or(-1), 1203444);\n    }\n}\n\nmod utils {\n    use crate::cli_args::{ARGS, MODEL_SPECS};\n    use anyhow::Result;\n    use log::info;\n    use serde_json::Value;\n\n    pub fn read_config(meta_file: &String) -> Result<Value> {\n        let json = std::fs::read_to_string(meta_file)?;\n        let v: Value = serde_json::from_str(&json)?;\n        Ok(v)\n    }\n    pub fn get_config_or_else<F>(model_config: &Value, key: &str, default: F) -> String\n    where\n        F: FnOnce() -> String,\n    {\n        match model_config[key] {\n            Value::String(ref v) => {\n                info!(\"from model_config: {}={}\", key, v);\n                v.to_string()\n            }\n            Value::Number(ref num) => {\n                info!(\n                    \"from model_config: {}={} (turn number into a string)\",\n                    key, num\n                );\n                num.to_string()\n            }\n            _ => {\n                let d = default();\n                info!(\"from default: {}={}\", key, d);\n                d\n            }\n        }\n    }\n    pub fn get_config_or(model_config: &Value, key: &str, default: &str) -> String {\n        get_config_or_else(model_config, key, || default.to_string())\n    }\n    pub fn get_meta_dir() -> &'static str {\n        ARGS.meta_json_dir\n            .as_ref()\n            .map(|s| s.as_str())\n            .unwrap_or_else(|| {\n                let model_dir = &ARGS.model_dir[0];\n                let meta_dir = &model_dir[0..model_dir.rfind(&MODEL_SPECS[0]).unwrap()];\n                info!(\n                    \"no meta_json_dir specified, hence derive from first model dir:{}->{}\",\n                    model_dir, meta_dir\n                );\n                meta_dir\n            })\n    }\n}\n\npub type SerializedInput = Vec<u8>;\npub const VERSION: &str = env!(\"CARGO_PKG_VERSION\");\npub const NAME: &str = env!(\"CARGO_PKG_NAME\");\npub type ModelFactory<T> = fn(usize, String, &Value) -> anyhow::Result<T>;\npub const MAX_NUM_MODELS: usize = 16;\npub const MAX_NUM_OUTPUTS: usize = 30;\npub const MAX_NUM_INPUTS: usize = 120;\npub const META_INFO: &str = \"META.json\";\n\n//use a heap allocated generic type here so that both\n//Tensorflow & Pytorch implementation can return their Tensor wrapped in a Box\n//without an extra memcopy to Vec\npub type TensorReturn<T> = Box<dyn Deref<Target = [T]>>;\n\n//returned tensor may be int64 i.e., a list of relevant ad ids\npub enum TensorReturnEnum {\n    FloatTensorReturn(TensorReturn<f32>),\n    StringTensorReturn(TensorReturn<String>),\n    Int64TensorReturn(TensorReturn<i64>),\n    Int32TensorReturn(TensorReturn<i32>),\n}\n\nimpl TensorReturnEnum {\n    #[inline(always)]\n    pub fn slice(&self, start: usize, end: usize) -> TensorScores {\n        match self {\n            TensorReturnEnum::FloatTensorReturn(f32_return) => {\n                TensorScores::Float32TensorScores(f32_return[start..end].to_vec())\n            }\n            TensorReturnEnum::Int64TensorReturn(i64_return) => {\n                TensorScores::Int64TensorScores(i64_return[start..end].to_vec())\n            }\n            TensorReturnEnum::Int32TensorReturn(i32_return) => {\n                TensorScores::Int32TensorScores(i32_return[start..end].to_vec())\n            }\n            TensorReturnEnum::StringTensorReturn(str_return) => {\n                TensorScores::StringTensorScores(str_return[start..end].to_vec())\n            }\n        }\n    }\n}\n\n#[derive(Debug)]\npub enum PredictResult {\n    Ok(Vec<TensorScores>, i64),\n    DropDueToOverload,\n    ModelNotFound(usize),\n    ModelNotReady(usize),\n    ModelVersionNotFound(usize, i64),\n}\n\n#[derive(Debug)]\npub enum TensorScores {\n    Float32TensorScores(Vec<f32>),\n    Int64TensorScores(Vec<i64>),\n    Int32TensorScores(Vec<i32>),\n    StringTensorScores(Vec<String>),\n}\n\nimpl TensorScores {\n    pub fn create_tensor_proto(self) -> TensorProto {\n        match self {\n            TensorScores::Float32TensorScores(f32_tensor) => TensorProto {\n                dtype: DataType::DtFloat as i32,\n                float_val: f32_tensor,\n                ..Default::default()\n            },\n            TensorScores::Int64TensorScores(i64_tensor) => TensorProto {\n                dtype: DataType::DtInt64 as i32,\n                int64_val: i64_tensor,\n                ..Default::default()\n            },\n            TensorScores::Int32TensorScores(i32_tensor) => TensorProto {\n                dtype: DataType::DtInt32 as i32,\n                int_val: i32_tensor,\n                ..Default::default()\n            },\n            TensorScores::StringTensorScores(str_tensor) => TensorProto {\n                dtype: DataType::DtString as i32,\n                string_val: str_tensor.into_iter().map(|s| s.into_bytes()).collect_vec(),\n                ..Default::default()\n            },\n        }\n    }\n    pub fn len(&self) -> usize {\n        match &self {\n            TensorScores::Float32TensorScores(t) => t.len(),\n            TensorScores::Int64TensorScores(t) => t.len(),\n            TensorScores::Int32TensorScores(t) => t.len(),\n            TensorScores::StringTensorScores(t) => t.len(),\n        }\n    }\n}\n\n#[derive(Debug)]\npub enum PredictMessage<T: Model> {\n    Predict(\n        usize,\n        Option<i64>,\n        Vec<TensorInput>,\n        Sender<PredictResult>,\n        Instant,\n    ),\n    UpsertModel(T),\n    /*\n    #[allow(dead_code)]\n    DeleteModel(usize),\n     */\n}\n\n#[derive(Debug)]\npub struct Callback(Sender<PredictResult>, usize);\n\npub const MAX_VERSIONS_PER_MODEL: usize = 2;\n"
  },
  {
    "path": "navi/navi/src/metrics.rs",
    "content": "use log::error;\nuse prometheus::{\n    CounterVec, HistogramOpts, HistogramVec, IntCounter, IntCounterVec, IntGauge, IntGaugeVec,\n    Opts, Registry,\n};\nuse warp::{Rejection, Reply};\nuse crate::{NAME, VERSION};\n\nlazy_static! {\n    pub static ref REGISTRY: Registry = Registry::new();\n    pub static ref NUM_REQUESTS_RECEIVED: IntCounter =\n        IntCounter::new(\":navi:num_requests\", \"Number of Requests Received\")\n            .expect(\"metric can be created\");\n    pub static ref NUM_REQUESTS_FAILED: IntCounter = IntCounter::new(\n        \":navi:num_requests_failed\",\n        \"Number of Request Inference Failed\"\n    )\n    .expect(\"metric can be created\");\n    pub static ref NUM_REQUESTS_DROPPED: IntCounter = IntCounter::new(\n        \":navi:num_requests_dropped\",\n        \"Number of Oneshot Receivers Dropped\"\n    )\n    .expect(\"metric can be created\");\n    pub static ref NUM_BATCHES_DROPPED: IntCounter = IntCounter::new(\n        \":navi:num_batches_dropped\",\n        \"Number of Batches Proactively Dropped\"\n    )\n    .expect(\"metric can be created\");\n    pub static ref NUM_BATCH_PREDICTION: IntCounter =\n        IntCounter::new(\":navi:num_batch_prediction\", \"Number of batch prediction\")\n            .expect(\"metric can be created\");\n    pub static ref BATCH_SIZE: IntGauge =\n        IntGauge::new(\":navi:batch_size\", \"Size of current batch\").expect(\"metric can be created\");\n    pub static ref NAVI_VERSION: IntGauge =\n        IntGauge::new(\":navi:navi_version\", \"navi's current version\")\n            .expect(\"metric can be created\");\n    pub static ref RESPONSE_TIME_COLLECTOR: HistogramVec = HistogramVec::new(\n        HistogramOpts::new(\":navi:response_time\", \"Response Time in ms\").buckets(Vec::from(&[\n            0.0, 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0, 110.0, 120.0, 130.0,\n            140.0, 150.0, 160.0, 170.0, 180.0, 190.0, 200.0, 250.0, 300.0, 500.0, 1000.0\n        ]\n            as &'static [f64])),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref NUM_PREDICTIONS: IntCounterVec = IntCounterVec::new(\n        Opts::new(\n            \":navi:num_predictions\",\n            \"Number of predictions made by model\"\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref PREDICTION_SCORE_SUM: CounterVec = CounterVec::new(\n        Opts::new(\n            \":navi:prediction_score_sum\",\n            \"Sum of prediction score made by model\"\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref NEW_MODEL_SNAPSHOT: IntCounterVec = IntCounterVec::new(\n        Opts::new(\n            \":navi:new_model_snapshot\",\n            \"Load a new version of model snapshot\"\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref MODEL_SNAPSHOT_VERSION: IntGaugeVec = IntGaugeVec::new(\n        Opts::new(\n            \":navi:model_snapshot_version\",\n            \"Record model snapshot version\"\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref NUM_REQUESTS_RECEIVED_BY_MODEL: IntCounterVec = IntCounterVec::new(\n        Opts::new(\n            \":navi:num_requests_by_model\",\n            \"Number of Requests Received by model\"\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref NUM_REQUESTS_FAILED_BY_MODEL: IntCounterVec = IntCounterVec::new(\n        Opts::new(\n            \":navi:num_requests_failed_by_model\",\n            \"Number of Request Inference Failed by model\"\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref NUM_REQUESTS_DROPPED_BY_MODEL: IntCounterVec = IntCounterVec::new(\n        Opts::new(\n            \":navi:num_requests_dropped_by_model\",\n            \"Number of Oneshot Receivers Dropped by model\"\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref NUM_BATCHES_DROPPED_BY_MODEL: IntCounterVec = IntCounterVec::new(\n        Opts::new(\n            \":navi:num_batches_dropped_by_model\",\n            \"Number of Batches Proactively Dropped by model\"\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref INFERENCE_FAILED_REQUESTS_BY_MODEL: IntCounterVec = IntCounterVec::new(\n        Opts::new(\n            \":navi:inference_failed_requests_by_model\",\n            \"Number of failed inference requests by model\"\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref NUM_PREDICTION_BY_MODEL: IntCounterVec = IntCounterVec::new(\n        Opts::new(\n            \":navi:num_prediction_by_model\",\n            \"Number of prediction by model\"\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref NUM_BATCH_PREDICTION_BY_MODEL: IntCounterVec = IntCounterVec::new(\n        Opts::new(\n            \":navi:num_batch_prediction_by_model\",\n            \"Number of batch prediction by model\"\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref BATCH_SIZE_BY_MODEL: IntGaugeVec = IntGaugeVec::new(\n        Opts::new(\n            \":navi:batch_size_by_model\",\n            \"Size of current batch by model\"\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref CUSTOMOP_VERSION: IntGauge =\n        IntGauge::new(\":navi:customop_version\", \"The hashed Custom OP Version\")\n            .expect(\"metric can be created\");\n    pub static ref MPSC_CHANNEL_SIZE: IntGauge =\n        IntGauge::new(\":navi:mpsc_channel_size\", \"The mpsc channel's request size\")\n            .expect(\"metric can be created\");\n    pub static ref BLOCKING_REQUEST_NUM: IntGauge = IntGauge::new(\n        \":navi:blocking_request_num\",\n        \"The (batch) request waiting or being executed\"\n    )\n    .expect(\"metric can be created\");\n    pub static ref MODEL_INFERENCE_TIME_COLLECTOR: HistogramVec = HistogramVec::new(\n        HistogramOpts::new(\":navi:model_inference_time\", \"Model inference time in ms\").buckets(\n            Vec::from(&[\n                0.0, 5.0, 10.0, 15.0, 20.0, 25.0, 30.0, 35.0, 40.0, 45.0, 50.0, 55.0, 60.0, 65.0,\n                70.0, 75.0, 80.0, 85.0, 90.0, 100.0, 110.0, 120.0, 130.0, 140.0, 150.0, 160.0,\n                170.0, 180.0, 190.0, 200.0, 250.0, 300.0, 500.0, 1000.0\n            ] as &'static [f64])\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref CONVERTER_TIME_COLLECTOR: HistogramVec = HistogramVec::new(\n        HistogramOpts::new(\":navi:converter_time\", \"converter time in microseconds\").buckets(\n            Vec::from(&[\n                0.0, 500.0, 1000.0, 1500.0, 2000.0, 2500.0, 3000.0, 3500.0, 4000.0, 4500.0, 5000.0,\n                5500.0, 6000.0, 6500.0, 7000.0, 20000.0\n            ] as &'static [f64])\n        ),\n        &[\"model_name\"]\n    )\n    .expect(\"metric can be created\");\n    pub static ref CERT_EXPIRY_EPOCH: IntGauge =\n        IntGauge::new(\":navi:cert_expiry_epoch\", \"Timestamp when the current cert expires\")\n            .expect(\"metric can be created\");\n}\n\npub fn register_custom_metrics() {\n    REGISTRY\n        .register(Box::new(NUM_REQUESTS_RECEIVED.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NUM_REQUESTS_FAILED.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NUM_REQUESTS_DROPPED.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(RESPONSE_TIME_COLLECTOR.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NAVI_VERSION.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(BATCH_SIZE.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NUM_BATCH_PREDICTION.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NUM_BATCHES_DROPPED.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NUM_PREDICTIONS.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(PREDICTION_SCORE_SUM.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NEW_MODEL_SNAPSHOT.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(MODEL_SNAPSHOT_VERSION.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NUM_REQUESTS_RECEIVED_BY_MODEL.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NUM_REQUESTS_FAILED_BY_MODEL.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NUM_REQUESTS_DROPPED_BY_MODEL.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NUM_BATCHES_DROPPED_BY_MODEL.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(INFERENCE_FAILED_REQUESTS_BY_MODEL.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NUM_PREDICTION_BY_MODEL.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(NUM_BATCH_PREDICTION_BY_MODEL.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(BATCH_SIZE_BY_MODEL.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(CUSTOMOP_VERSION.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(MPSC_CHANNEL_SIZE.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(BLOCKING_REQUEST_NUM.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(MODEL_INFERENCE_TIME_COLLECTOR.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n        .register(Box::new(CONVERTER_TIME_COLLECTOR.clone()))\n        .expect(\"collector can be registered\");\n    REGISTRY\n    .register(Box::new(CERT_EXPIRY_EPOCH.clone()))\n    .expect(\"collector can be registered\");\n\n}\n\npub fn register_dynamic_metrics(c: &HistogramVec) {\n    REGISTRY\n        .register(Box::new(c.clone()))\n        .expect(\"dynamic metric collector cannot be registered\");\n}\n\npub async fn metrics_handler() -> Result<impl Reply, Rejection> {\n    use prometheus::Encoder;\n    let encoder = prometheus::TextEncoder::new();\n\n    let mut buffer = Vec::new();\n    if let Err(e) = encoder.encode(&REGISTRY.gather(), &mut buffer) {\n        error!(\"could not encode custom metrics: {}\", e);\n    };\n    let mut res = match String::from_utf8(buffer) {\n        Ok(v) => format!(\"#{}:{}\\n{}\", NAME, VERSION, v),\n        Err(e) => {\n            error!(\"custom metrics could not be from_utf8'd: {}\", e);\n            String::default()\n        }\n    };\n\n    buffer = Vec::new();\n    if let Err(e) = encoder.encode(&prometheus::gather(), &mut buffer) {\n        error!(\"could not encode prometheus metrics: {}\", e);\n    };\n    let res_custom = match String::from_utf8(buffer) {\n        Ok(v) => v,\n        Err(e) => {\n            error!(\"prometheus metrics could not be from_utf8'd: {}\", e);\n            String::default()\n        }\n    };\n\n    res.push_str(&res_custom);\n    Ok(res)\n}\n"
  },
  {
    "path": "navi/navi/src/onnx_model.rs",
    "content": "#[cfg(feature = \"onnx\")]\npub mod onnx {\n    use crate::TensorReturnEnum;\n    use crate::bootstrap::{TensorInput, TensorInputEnum};\n    use crate::cli_args::{\n        Args, ARGS, INPUTS, MODEL_SPECS, OUTPUTS,\n    };\n    use crate::metrics::{self, CONVERTER_TIME_COLLECTOR};\n    use crate::predict_service::Model;\n    use crate::{MAX_NUM_INPUTS, MAX_NUM_OUTPUTS, META_INFO, utils};\n    use anyhow::Result;\n    use arrayvec::ArrayVec;\n    use dr_transform::converter::{BatchPredictionRequestToTorchTensorConverter, Converter};\n    use itertools::Itertools;\n    use log::{debug, info};\n    use dr_transform::ort::environment::Environment;\n    use dr_transform::ort::session::Session;\n    use dr_transform::ort::tensor::InputTensor;\n    use dr_transform::ort::{ExecutionProvider, GraphOptimizationLevel, SessionBuilder};\n    use dr_transform::ort::LoggingLevel;\n    use serde_json::Value;\n    use std::fmt::{Debug, Display};\n    use std::sync::Arc;\n    use std::{fmt, fs};\n    use tokio::time::Instant;\n    lazy_static! {\n        pub static ref ENVIRONMENT: Arc<Environment> = Arc::new(\n            Environment::builder()\n                .with_name(\"onnx home\")\n                .with_log_level(LoggingLevel::Error)\n                .with_global_thread_pool(ARGS.onnx_global_thread_pool_options.clone())\n                .build()\n                .unwrap()\n        );\n    }\n    #[derive(Debug)]\n    pub struct OnnxModel {\n        pub session: Session,\n        pub model_idx: usize,\n        pub version: i64,\n        pub export_dir: String,\n        pub output_filters: ArrayVec<usize, MAX_NUM_OUTPUTS>,\n        pub input_converter: Box<dyn Converter>,\n    }\n    impl Display for OnnxModel {\n        fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n            write!(\n                f,\n                \"idx: {}, onnx model_name:{}, version:{}, output_filters:{:?}, converter:{:}\",\n                self.model_idx,\n                MODEL_SPECS[self.model_idx],\n                self.version,\n                self.output_filters,\n                self.input_converter\n            )\n        }\n    }\n    impl Drop for OnnxModel {\n        fn drop(&mut self) {\n            if ARGS.profiling != None {\n                self.session.end_profiling().map_or_else(\n                    |e| {\n                        info!(\"end profiling with some error:{:?}\", e);\n                    },\n                    |f| {\n                        info!(\"profiling ended with file:{}\", f);\n                    },\n                );\n            }\n        }\n    }\n    impl OnnxModel {\n        fn get_output_filters(session: &Session, idx: usize) -> ArrayVec<usize, MAX_NUM_OUTPUTS> {\n            OUTPUTS[idx]\n                .iter()\n                .map(|output| session.outputs.iter().position(|o| o.name == *output))\n                .flatten()\n                .collect::<ArrayVec<usize, MAX_NUM_OUTPUTS>>()\n        }\n        #[cfg(target_os = \"linux\")]\n        fn ep_choices() -> Vec<ExecutionProvider> {\n            match ARGS.onnx_gpu_ep.as_ref().map(|e| e.as_str()) {\n                Some(\"onednn\") => vec![Self::ep_with_options(ExecutionProvider::onednn())],\n                Some(\"tensorrt\") => vec![Self::ep_with_options(ExecutionProvider::tensorrt())],\n                Some(\"cuda\") => vec![Self::ep_with_options(ExecutionProvider::cuda())],\n                _ => vec![Self::ep_with_options(ExecutionProvider::cpu())],\n            }\n        }\n        fn ep_with_options(mut ep: ExecutionProvider) -> ExecutionProvider {\n            for (ref k, ref v) in ARGS.onnx_ep_options.clone() {\n                ep = ep.with(k, v);\n                info!(\"setting option:{} -> {} and now ep is:{:?}\", k, v, ep);\n            }\n            ep\n        }\n        #[cfg(target_os = \"macos\")]\n        fn ep_choices() -> Vec<ExecutionProvider> {\n            vec![Self::ep_with_options(ExecutionProvider::cpu())]\n        }\n        pub fn new(idx: usize, version: String, model_config: &Value) -> Result<OnnxModel> {\n            let export_dir = format!(\"{}/{}/model.onnx\", ARGS.model_dir[idx], version);\n            let meta_info = format!(\"{}/{}/{}\", ARGS.model_dir[idx], version, META_INFO);\n            let mut builder = SessionBuilder::new(&ENVIRONMENT)?\n                .with_optimization_level(GraphOptimizationLevel::Level3)?\n                .with_parallel_execution(ARGS.onnx_use_parallel_mode == \"true\")?;\n            if ARGS.onnx_global_thread_pool_options.is_empty() {\n                builder = builder\n                    .with_inter_threads(\n                        utils::get_config_or(\n                            model_config,\n                            \"inter_op_parallelism\",\n                            &ARGS.inter_op_parallelism[idx],\n                        )\n                            .parse()?,\n                    )?\n                    .with_intra_threads(\n                        utils::get_config_or(\n                            model_config,\n                            \"intra_op_parallelism\",\n                            &ARGS.intra_op_parallelism[idx],\n                        )\n                            .parse()?,\n                    )?;\n            }\n            else {\n                builder = builder.with_disable_per_session_threads()?;\n            }\n            builder = builder\n                .with_memory_pattern(ARGS.onnx_use_memory_pattern == \"true\")?\n                .with_execution_providers(&OnnxModel::ep_choices())?;\n            match &ARGS.profiling {\n                Some(p) => {\n                    debug!(\"Enable profiling, writing to {}\", *p);\n                    builder = builder.with_profiling(p)?\n                }\n                _ => {}\n            }\n            let session = builder.with_model_from_file(&export_dir)?;\n\n            info!(\n                \"inputs: {:?}, outputs: {:?}\",\n                session.inputs.iter().format(\",\"),\n                session.outputs.iter().format(\",\")\n            );\n\n            fs::read_to_string(&meta_info)\n                .ok()\n                .map(|info| info!(\"meta info:{}\", info));\n            let output_filters = OnnxModel::get_output_filters(&session, idx);\n            let mut reporting_feature_ids: Vec<(i64, &str)> = vec![];\n\n            let input_spec_cell = &INPUTS[idx];\n            if input_spec_cell.get().is_none() {\n                let input_spec = session\n                    .inputs\n                    .iter()\n                    .map(|input| input.name.clone())\n                    .collect::<ArrayVec<String, MAX_NUM_INPUTS>>();\n                input_spec_cell.set(input_spec.clone()).map_or_else(\n                    |_| info!(\"unable to set the input_spec for model {}\", idx),\n                    |_| info!(\"auto detect and set the inputs: {:?}\", input_spec),\n                );\n            }\n            ARGS.onnx_report_discrete_feature_ids\n                .iter()\n                .for_each(|ids| {\n                    ids.split(\",\")\n                        .filter(|s| !s.is_empty())\n                        .map(|s| s.parse::<i64>().unwrap())\n                        .for_each(|id| reporting_feature_ids.push((id, \"discrete\")))\n                });\n            ARGS.onnx_report_continuous_feature_ids\n                .iter()\n                .for_each(|ids| {\n                    ids.split(\",\")\n                        .filter(|s| !s.is_empty())\n                        .map(|s| s.parse::<i64>().unwrap())\n                        .for_each(|id| reporting_feature_ids.push((id, \"continuous\")))\n                });\n\n            let onnx_model = OnnxModel {\n                session,\n                model_idx: idx,\n                version: Args::version_str_to_epoch(&version)?,\n                export_dir,\n                output_filters,\n                input_converter: Box::new(BatchPredictionRequestToTorchTensorConverter::new(\n                    &ARGS.model_dir[idx],\n                    &version,\n                    reporting_feature_ids,\n                    Some(metrics::register_dynamic_metrics),\n                )?),\n            };\n            onnx_model.warmup()?;\n            Ok(onnx_model)\n        }\n    }\n    ///Currently we only assume the input as just one string tensor.\n    ///The string tensor will be be converted to the actual raw tensors.\n    /// The converter we are using is very specific to home.\n    /// It reads a BatchDataRecord thrift and decode it to a batch of raw input tensors.\n    /// Navi will then do server side batching and feed it to ONNX runtime\n    impl Model for OnnxModel {\n        //TODO: implement a generic online warmup for all runtimes\n        fn warmup(&self) -> Result<()> {\n            Ok(())\n        }\n\n        #[inline(always)]\n        fn do_predict(\n            &self,\n            input_tensors: Vec<Vec<TensorInput>>,\n            _: u64,\n        ) -> (Vec<TensorReturnEnum>, Vec<Vec<usize>>) {\n            let batched_tensors = TensorInputEnum::merge_batch(input_tensors);\n            let (inputs, batch_ends): (Vec<Vec<InputTensor>>, Vec<Vec<usize>>) = batched_tensors\n                .into_iter()\n                .map(|batched_tensor| {\n                    match batched_tensor.tensor_data {\n                        TensorInputEnum::String(t) if ARGS.onnx_use_converter.is_some() => {\n                            let start = Instant::now();\n                            let (inputs, batch_ends) = self.input_converter.convert(t);\n                            // info!(\"batch_ends:{:?}\", batch_ends);\n                            CONVERTER_TIME_COLLECTOR\n                                .with_label_values(&[&MODEL_SPECS[self.model_idx()]])\n                                .observe(\n                                    start.elapsed().as_micros() as f64\n                                        / (*batch_ends.last().unwrap() as f64),\n                                );\n                            (inputs, batch_ends)\n                        }\n                        _ => unimplemented!(),\n                    }\n                })\n                .unzip();\n            //invariant we only support one input as string. will relax later\n            assert_eq!(inputs.len(), 1);\n            let output_tensors = self\n                .session\n                .run(inputs.into_iter().flatten().collect::<Vec<_>>())\n                .unwrap();\n            self.output_filters\n                .iter()\n                .map(|&idx| {\n                    let mut size = 1usize;\n                    let output = output_tensors[idx].try_extract::<f32>().unwrap();\n                    for &dim in self.session.outputs[idx].dimensions.iter().flatten() {\n                        size *= dim as usize;\n                    }\n                    let tensor_ends = batch_ends[0]\n                        .iter()\n                        .map(|&batch| batch * size)\n                        .collect::<Vec<_>>();\n\n                    (\n                        //only works for batch major\n                        //TODO: to_vec() obviously wasteful, especially for large batches(GPU) . Will refactor to\n                        //break up output and return Vec<Vec<TensorScore>> here\n                        TensorReturnEnum::FloatTensorReturn(Box::new(output.view().as_slice().unwrap().to_vec(),\n                        )),\n                        tensor_ends,\n                    )\n                })\n                .unzip()\n        }\n        #[inline(always)]\n        fn model_idx(&self) -> usize {\n            self.model_idx\n        }\n        #[inline(always)]\n        fn version(&self) -> i64 {\n            self.version\n        }\n    }\n}\n"
  },
  {
    "path": "navi/navi/src/predict_service.rs",
    "content": "use anyhow::{anyhow, Result};\nuse arrayvec::ArrayVec;\nuse itertools::Itertools;\nuse log::{error, info};\nuse std::fmt::{Debug, Display};\nuse std::string::String;\nuse std::sync::Arc;\nuse std::time::Duration;\nuse tokio::process::Command;\nuse tokio::sync::mpsc::error::TryRecvError;\nuse tokio::sync::mpsc::{Receiver, Sender};\nuse tokio::sync::{mpsc, oneshot};\nuse tokio::time::{Instant, sleep};\nuse warp::Filter;\n\nuse crate::batch::BatchPredictor;\nuse crate::bootstrap::TensorInput;\nuse crate::{MAX_NUM_MODELS, MAX_VERSIONS_PER_MODEL, META_INFO, metrics, ModelFactory, PredictMessage, PredictResult, TensorReturnEnum, utils};\n\nuse crate::cli_args::{ARGS, MODEL_SPECS};\nuse crate::cores::validator::validatior::cli_validator;\nuse crate::metrics::MPSC_CHANNEL_SIZE;\nuse serde_json::{self, Value};\n\npub trait Model: Send + Sync + Display + Debug + 'static {\n    fn warmup(&self) -> Result<()>;\n    //TODO: refactor this to return vec<vec<TensorScores>>, i.e.\n    //we have the underlying runtime impl to split the response to each client.\n    //It will eliminate some inefficient memory copy in onnx_model.rs as well as simplify code\n    fn do_predict(\n        &self,\n        input_tensors: Vec<Vec<TensorInput>>,\n        total_len: u64,\n    ) -> (Vec<TensorReturnEnum>, Vec<Vec<usize>>);\n    fn model_idx(&self) -> usize;\n    fn version(&self) -> i64;\n}\n\n#[derive(Debug)]\npub struct PredictService<T: Model> {\n    tx: Sender<PredictMessage<T>>,\n}\nimpl<T: Model> PredictService<T> {\n    pub async fn init(model_factory: ModelFactory<T>) -> Self {\n        cli_validator::validate_ps_model_args();\n        let (tx, rx) = mpsc::channel(32_000);\n        tokio::spawn(PredictService::tf_queue_manager(rx));\n        tokio::spawn(PredictService::model_watcher_latest(\n            model_factory,\n            tx.clone(),\n        ));\n        let metrics_route = warp::path!(\"metrics\").and_then(metrics::metrics_handler);\n        let metric_server = warp::serve(metrics_route).run(([0, 0, 0, 0], ARGS.prometheus_port));\n        tokio::spawn(metric_server);\n        PredictService { tx }\n    }\n    #[inline(always)]\n    pub async fn predict(\n        &self,\n        idx: usize,\n        version: Option<i64>,\n        val: Vec<TensorInput>,\n        ts: Instant,\n    ) -> Result<PredictResult> {\n        let (tx, rx) = oneshot::channel();\n        if let Err(e) = self\n            .tx\n            .clone()\n            .send(PredictMessage::Predict(idx, version, val, tx, ts))\n            .await\n        {\n            error!(\"mpsc send error:{}\", e);\n            Err(anyhow!(e))\n        } else {\n            MPSC_CHANNEL_SIZE.inc();\n            rx.await.map_err(anyhow::Error::msg)\n        }\n    }\n\n    async fn load_latest_model_from_model_dir(\n        model_factory: ModelFactory<T>,\n        model_config: &Value,\n        tx: Sender<PredictMessage<T>>,\n        idx: usize,\n        max_version: String,\n        latest_version: &mut String,\n    ) {\n        match model_factory(idx, max_version.clone(), model_config) {\n            Ok(tf_model) => tx\n                .send(PredictMessage::UpsertModel(tf_model))\n                .await\n                .map_or_else(\n                    |e| error!(\"send UpsertModel error: {}\", e),\n                    |_| *latest_version = max_version,\n                ),\n            Err(e) => {\n                error!(\"skip loading model due to failure: {:?}\", e);\n            }\n        }\n    }\n\n    async fn scan_load_latest_model_from_model_dir(\n        model_factory: ModelFactory<T>,\n        model_config: &Value,\n        tx: Sender<PredictMessage<T>>,\n        model_idx: usize,\n        cur_version: &mut String,\n    ) -> Result<()> {\n        let model_dir = &ARGS.model_dir[model_idx];\n        let next_version = utils::get_config_or_else(model_config, \"version\", || {\n            info!(\"no version found, hence use max version\");\n            std::fs::read_dir(model_dir)\n                .map_err(|e| format!(\"read dir error:{}\", e))\n                .and_then(|paths| {\n                    paths\n                        .into_iter()\n                        .flat_map(|p| {\n                            p.map_err(|e| error!(\"dir entry error: {}\", e))\n                                .and_then(|dir| {\n                                    dir.file_name()\n                                        .into_string()\n                                        .map_err(|e| error!(\"osstring error: {:?}\", e))\n                                })\n                                .ok()\n                        })\n                        .filter(|f| !f.to_lowercase().contains(&META_INFO.to_lowercase()))\n                        .max()\n                        .ok_or_else(|| \"no dir found hence no max\".to_owned())\n                })\n                .unwrap_or_else(|e| {\n                    error!(\n                        \"can't get the max version hence return cur_version, error is: {}\",\n                        e\n                    );\n                    cur_version.to_string()\n                })\n        });\n        //as long as next version doesn't match cur version maintained we reload\n        if next_version.ne(cur_version) {\n            info!(\"reload the version: {}->{}\", cur_version, next_version);\n            PredictService::load_latest_model_from_model_dir(\n                model_factory,\n                model_config,\n                tx,\n                model_idx,\n                next_version,\n                cur_version,\n            )\n            .await;\n        }\n        Ok(())\n    }\n\n    async fn model_watcher_latest(model_factory: ModelFactory<T>, tx: Sender<PredictMessage<T>>) {\n        async fn call_external_modelsync(cli: &str, cur_versions: &Vec<String>) -> Result<()> {\n            let mut args = cli.split_whitespace();\n\n            let mut cmd = Command::new(args.next().ok_or(anyhow!(\"model sync cli empty\"))?);\n            let extr_args = MODEL_SPECS\n                .iter()\n                .zip(cur_versions)\n                .flat_map(|(spec, version)| vec![\"--model-spec\", spec, \"--cur-version\", version])\n                .collect_vec();\n            info!(\"run model sync: {} with extra args: {:?}\", cli, extr_args);\n            let output = cmd.args(args).args(extr_args).output().await?;\n            info!(\"model sync stdout:{}\", String::from_utf8(output.stdout)?);\n            info!(\"model sync stderr:{}\", String::from_utf8(output.stderr)?);\n            if output.status.success() {\n                Ok(())\n            } else {\n                Err(anyhow!(\n                    \"model sync failed with status: {:?}!\",\n                    output.status\n                ))\n            }\n        }\n        let meta_dir = utils::get_meta_dir();\n        let meta_file = format!(\"{}{}\", meta_dir, META_INFO);\n        //initialize the latest version array\n        let mut cur_versions = vec![\"\".to_owned(); MODEL_SPECS.len()];\n        loop {\n            info!(\"***polling for models***\"); //nice deliminter\n            if let Some(ref cli) = ARGS.modelsync_cli {\n                if let Err(e) = call_external_modelsync(cli, &cur_versions).await {\n                    error!(\"model sync cli running error:{}\", e)\n                }\n            }\n            let config = utils::read_config(&meta_file).unwrap_or_else(|e| {\n                info!(\"config file {} not found due to: {}\", meta_file, e);\n                Value::Null\n            });\n            info!(\"config:{}\", config);\n            for (idx, cur_version) in cur_versions.iter_mut().enumerate() {\n                let model_dir = &ARGS.model_dir[idx];\n                PredictService::scan_load_latest_model_from_model_dir(\n                    model_factory,\n                    &config[&MODEL_SPECS[idx]],\n                    tx.clone(),\n                    idx,\n                    cur_version,\n                )\n                .await\n                .map_or_else(\n                    |e| error!(\"scanned {}, error {:?}\", model_dir, e),\n                    |_| info!(\"scanned {}, latest_version: {}\", model_dir, cur_version),\n                );\n            }\n            sleep(Duration::from_secs(ARGS.model_check_interval_secs)).await;\n        }\n    }\n    async fn tf_queue_manager(mut rx: Receiver<PredictMessage<T>>) {\n        // Start receiving messages\n        info!(\"setting up queue manager\");\n        let max_batch_size = ARGS\n            .max_batch_size\n            .iter()\n            .map(|b| b.parse().unwrap())\n            .collect::<Vec<usize>>();\n        let batch_time_out_millis = ARGS\n            .batch_time_out_millis\n            .iter()\n            .map(|b| b.parse().unwrap())\n            .collect::<Vec<u64>>();\n        let no_msg_wait_millis = *batch_time_out_millis.iter().min().unwrap();\n        let mut all_model_predictors: ArrayVec::<ArrayVec<BatchPredictor<T>, MAX_VERSIONS_PER_MODEL>, MAX_NUM_MODELS> =\n            (0 ..MAX_NUM_MODELS).map( |_| ArrayVec::<BatchPredictor<T>, MAX_VERSIONS_PER_MODEL>::new()).collect();\n        loop {\n            let msg = rx.try_recv();\n            let no_more_msg = match msg {\n                Ok(PredictMessage::Predict(model_spec_at, version, val, resp, ts)) => {\n                    if let Some(model_predictors) = all_model_predictors.get_mut(model_spec_at) {\n                        if model_predictors.is_empty() {\n                            resp.send(PredictResult::ModelNotReady(model_spec_at))\n                                .unwrap_or_else(|e| error!(\"cannot send back model not ready error: {:?}\", e));\n                        }\n                        else {\n                            match version {\n                                None => model_predictors[0].push(val, resp, ts),\n                                Some(the_version) => match model_predictors\n                                    .iter_mut()\n                                    .find(|x| x.model.version() == the_version)\n                                {\n                                    None => resp\n                                        .send(PredictResult::ModelVersionNotFound(\n                                            model_spec_at,\n                                            the_version,\n                                        ))\n                                        .unwrap_or_else(|e| {\n                                            error!(\"cannot send back version error: {:?}\", e)\n                                        }),\n                                    Some(predictor) => predictor.push(val, resp, ts),\n                                },\n                            }\n                        }\n                    } else {\n                        resp.send(PredictResult::ModelNotFound(model_spec_at))\n                            .unwrap_or_else(|e| error!(\"cannot send back model not found error: {:?}\", e))\n                    }\n                    MPSC_CHANNEL_SIZE.dec();\n                    false\n                }\n                Ok(PredictMessage::UpsertModel(tf_model)) => {\n                    let idx = tf_model.model_idx();\n                    let predictor = BatchPredictor {\n                        model: Arc::new(tf_model),\n                        input_tensors: Vec::with_capacity(max_batch_size[idx]),\n                        callbacks: Vec::with_capacity(max_batch_size[idx]),\n                        cur_batch_size: 0,\n                        max_batch_size: max_batch_size[idx],\n                        batch_time_out_millis: batch_time_out_millis[idx],\n                        //initialize to be current time\n                        queue_reset_ts: Instant::now(),\n                        queue_earliest_rq_ts: Instant::now(),\n                    };\n                    assert!(idx < all_model_predictors.len());\n                    metrics::NEW_MODEL_SNAPSHOT\n                        .with_label_values(&[&MODEL_SPECS[idx]])\n                        .inc();\n\n                    //we can do this since the vector is small\n                    let predictors = &mut all_model_predictors[idx];\n                    if predictors.len() == 0 {\n                        info!(\"now we serve new model: {}\", predictor.model);\n                    }\n                    else {\n                        info!(\"now we serve updated model: {}\", predictor.model);\n                    }\n                    if predictors.len() == ARGS.versions_per_model {\n                        predictors.remove(predictors.len() - 1);\n                    }\n                    predictors.insert(0, predictor);\n                    false\n                }\n                Err(TryRecvError::Empty) => true,\n                Err(TryRecvError::Disconnected) => true,\n            };\n            for predictor in all_model_predictors.iter_mut().flatten() {\n                //if predictor batch queue not empty and times out or no more msg in the queue, flush\n                if (!predictor.input_tensors.is_empty() && (predictor.duration_past(predictor.batch_time_out_millis) || no_more_msg))\n                    //if batch queue reaches limit, flush\n                    || predictor.cur_batch_size >= predictor.max_batch_size\n                {\n                    predictor.batch_predict();\n                }\n            }\n            if no_more_msg {\n                sleep(Duration::from_millis(no_msg_wait_millis)).await;\n            }\n        }\n    }\n    #[inline(always)]\n    pub fn get_model_index(model_spec: &str) -> Option<usize> {\n        MODEL_SPECS.iter().position(|m| m == model_spec)\n    }\n}\n"
  },
  {
    "path": "navi/navi/src/tf_model.rs",
    "content": "#[cfg(feature = \"tf\")]\npub mod tf {\n    use arrayvec::ArrayVec;\n    use itertools::Itertools;\n    use log::{debug, error, info, warn};\n    use prost::Message;\n    use std::fmt;\n    use std::fmt::Display;\n    use std::string::String;\n    use tensorflow::io::{RecordReader, RecordReadError};\n    use tensorflow::Operation;\n    use tensorflow::SavedModelBundle;\n    use tensorflow::SessionOptions;\n    use tensorflow::SessionRunArgs;\n    use tensorflow::Tensor;\n    use tensorflow::{DataType, FetchToken, Graph, TensorInfo, TensorType};\n\n    use std::thread::sleep;\n    use std::time::Duration;\n\n    use crate::cli_args::{Args, ARGS, INPUTS, MODEL_SPECS, OUTPUTS};\n    use crate::tf_proto::tensorflow_serving::prediction_log::LogType;\n    use crate::tf_proto::tensorflow_serving::{PredictionLog, PredictLog};\n    use crate::tf_proto::ConfigProto;\n    use anyhow::{Context, Result};\n    use serde_json::Value;\n\n    use crate::TensorReturnEnum;\n    use crate::bootstrap::{TensorInput, TensorInputEnum};\n    use crate::metrics::{\n        INFERENCE_FAILED_REQUESTS_BY_MODEL, NUM_REQUESTS_FAILED, NUM_REQUESTS_FAILED_BY_MODEL,\n    };\n    use crate::predict_service::Model;\n    use crate::{MAX_NUM_INPUTS, utils};\n\n    #[derive(Debug)]\n    pub enum TFTensorEnum {\n        String(Tensor<String>),\n        Int(Tensor<i32>),\n        Int64(Tensor<i64>),\n        Float(Tensor<f32>),\n        Double(Tensor<f64>),\n        Boolean(Tensor<bool>),\n    }\n\n    #[derive(Debug)]\n    pub struct TFModel {\n        pub model_idx: usize,\n        pub bundle: SavedModelBundle,\n        pub input_names: ArrayVec<String, MAX_NUM_INPUTS>,\n        pub input_info: Vec<TensorInfo>,\n        pub input_ops: Vec<Operation>,\n        pub output_names: Vec<String>,\n        pub output_info: Vec<TensorInfo>,\n        pub output_ops: Vec<Operation>,\n        pub export_dir: String,\n        pub version: i64,\n        pub inter_op: i32,\n        pub intra_op: i32,\n    }\n\n    impl Display for TFModel {\n        fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n            write!(\n                f,\n                \"idx: {}, tensorflow model_name:{}, export_dir:{}, version:{}, inter:{}, intra:{}\",\n                self.model_idx,\n                MODEL_SPECS[self.model_idx],\n                self.export_dir,\n                self.version,\n                self.inter_op,\n                self.intra_op\n            )\n        }\n    }\n\n    impl TFModel {\n        pub fn new(idx: usize, version: String, model_config: &Value) -> Result<TFModel> {\n            // Create input variables for our addition\n            let config = ConfigProto {\n                intra_op_parallelism_threads: utils::get_config_or(\n                    model_config,\n                    \"intra_op_parallelism\",\n                    &ARGS.intra_op_parallelism[idx],\n                )\n                .parse()?,\n                inter_op_parallelism_threads: utils::get_config_or(\n                    model_config,\n                    \"inter_op_parallelism\",\n                    &ARGS.inter_op_parallelism[idx],\n                )\n                .parse()?,\n                ..Default::default()\n            };\n            let mut buf = Vec::new();\n            buf.reserve(config.encoded_len());\n            config.encode(&mut buf).unwrap();\n            let mut opts = SessionOptions::new();\n            opts.set_config(&buf)?;\n            let export_dir = format!(\"{}/{}\", ARGS.model_dir[idx], version);\n            let mut graph = Graph::new();\n            let bundle = SavedModelBundle::load(&opts, [\"serve\"], &mut graph, &export_dir)\n                .context(\"error load model\")?;\n            let signature = bundle\n                .meta_graph_def()\n                .get_signature(&ARGS.serving_sig[idx])\n                .context(\"error finding signature\")?;\n            let input_names = INPUTS[idx]\n                .get_or_init(|| {\n                    let input_spec = signature\n                        .inputs()\n                        .iter()\n                        .map(|p| p.0.clone())\n                        .collect::<ArrayVec<String, MAX_NUM_INPUTS>>();\n                    info!(\n                        \"input not set from cli, now we set from model metadata:{:?}\",\n                        input_spec\n                    );\n                    input_spec\n                })\n                .clone();\n            let input_info = input_names\n                .iter()\n                .map(|i| {\n                    signature\n                        .get_input(i)\n                        .context(\"error finding input op info\")\n                        .unwrap()\n                        .clone()\n                })\n                .collect_vec();\n\n            let input_ops = input_info\n                .iter()\n                .map(|i| {\n                    graph\n                        .operation_by_name_required(&i.name().name)\n                        .context(\"error finding input op\")\n                        .unwrap()\n                })\n                .collect_vec();\n\n            info!(\"Model Input size: {}\", input_info.len());\n\n            let output_names = OUTPUTS[idx].to_vec().clone();\n\n            let output_info = output_names\n                .iter()\n                .map(|o| {\n                    signature\n                        .get_output(o)\n                        .context(\"error finding output op info\")\n                        .unwrap()\n                        .clone()\n                })\n                .collect_vec();\n\n            let output_ops = output_info\n                .iter()\n                .map(|o| {\n                    graph\n                        .operation_by_name_required(&o.name().name)\n                        .context(\"error finding output op\")\n                        .unwrap()\n                })\n                .collect_vec();\n\n            let tf_model = TFModel {\n                model_idx: idx,\n                bundle,\n                input_names,\n                input_info,\n                input_ops,\n                output_names,\n                output_info,\n                output_ops,\n                export_dir,\n                version: Args::version_str_to_epoch(&version)?,\n                inter_op: config.inter_op_parallelism_threads,\n                intra_op: config.intra_op_parallelism_threads,\n            };\n            tf_model.warmup()?;\n            Ok(tf_model)\n        }\n\n        #[inline(always)]\n        fn get_tftensor_dimensions<T>(\n            t: &[T],\n            input_size: u64,\n            batch_size: u64,\n            input_dims: Option<Vec<i64>>,\n        ) -> Vec<u64> {\n            // if input size is 1, we just specify a single dimension to outgoing tensor matching the\n            // size of the input tensor. This is for backwards compatiblity with existing Navi clients\n            // which specify input as a single string tensor (like tfexample) and use batching support.\n            let mut dims = vec![];\n            if input_size > 1 {\n                if batch_size == 1 && input_dims.is_some() {\n                    // client side batching is enabled?\n                    input_dims\n                        .unwrap()\n                        .iter()\n                        .for_each(|axis| dims.push(*axis as u64));\n                } else {\n                    dims.push(batch_size);\n                    dims.push(t.len() as u64 / batch_size);\n                }\n            } else {\n                dims.push(t.len() as u64);\n            }\n            dims\n        }\n\n        fn convert_to_tftensor_enum(\n            input: TensorInput,\n            input_size: u64,\n            batch_size: u64,\n        ) -> TFTensorEnum {\n            match input.tensor_data {\n                TensorInputEnum::String(t) => {\n                    let strings = t\n                        .into_iter()\n                        .map(|x| unsafe { String::from_utf8_unchecked(x) })\n                        .collect_vec();\n                    TFTensorEnum::String(\n                        Tensor::new(&TFModel::get_tftensor_dimensions(\n                            strings.as_slice(),\n                            input_size,\n                            batch_size,\n                            input.dims,\n                        ))\n                        .with_values(strings.as_slice())\n                        .unwrap(),\n                    )\n                }\n                TensorInputEnum::Int(t) => TFTensorEnum::Int(\n                    Tensor::new(&TFModel::get_tftensor_dimensions(\n                        t.as_slice(),\n                        input_size,\n                        batch_size,\n                        input.dims,\n                    ))\n                    .with_values(t.as_slice())\n                    .unwrap(),\n                ),\n                TensorInputEnum::Int64(t) => TFTensorEnum::Int64(\n                    Tensor::new(&TFModel::get_tftensor_dimensions(\n                        t.as_slice(),\n                        input_size,\n                        batch_size,\n                        input.dims,\n                    ))\n                    .with_values(t.as_slice())\n                    .unwrap(),\n                ),\n                TensorInputEnum::Float(t) => TFTensorEnum::Float(\n                    Tensor::new(&TFModel::get_tftensor_dimensions(\n                        t.as_slice(),\n                        input_size,\n                        batch_size,\n                        input.dims,\n                    ))\n                    .with_values(t.as_slice())\n                    .unwrap(),\n                ),\n                TensorInputEnum::Double(t) => TFTensorEnum::Double(\n                    Tensor::new(&TFModel::get_tftensor_dimensions(\n                        t.as_slice(),\n                        input_size,\n                        batch_size,\n                        input.dims,\n                    ))\n                    .with_values(t.as_slice())\n                    .unwrap(),\n                ),\n                TensorInputEnum::Boolean(t) => TFTensorEnum::Boolean(\n                    Tensor::new(&TFModel::get_tftensor_dimensions(\n                        t.as_slice(),\n                        input_size,\n                        batch_size,\n                        input.dims,\n                    ))\n                    .with_values(t.as_slice())\n                    .unwrap(),\n                ),\n            }\n        }\n        fn fetch_output<T: TensorType>(\n            args: &mut SessionRunArgs,\n            token_output: &FetchToken,\n            batch_size: u64,\n            output_size: u64,\n        ) -> (Tensor<T>, u64) {\n            let tensor_output = args.fetch::<T>(*token_output).expect(\"fetch output failed\");\n            let mut tensor_width = tensor_output.dims()[1];\n            if batch_size == 1 && output_size > 1 {\n                tensor_width = tensor_output.dims().iter().fold(1, |mut total, &val| {\n                    total *= val;\n                    total\n                });\n            }\n            (tensor_output, tensor_width)\n        }\n    }\n\n    impl Model for TFModel {\n        fn warmup(&self) -> Result<()> {\n            // warm up\n            let warmup_file = format!(\n                \"{}/assets.extra/tf_serving_warmup_requests\",\n                self.export_dir\n            );\n            if std::path::Path::new(&warmup_file).exists() {\n                use std::io::Cursor;\n                info!(\n                    \"found warmup assets in {}, now perform warming up\",\n                    warmup_file\n                );\n                let f = std::fs::File::open(warmup_file).context(\"cannot open warmup file\")?;\n                // let mut buf = Vec::new();\n                let read = std::io::BufReader::new(f);\n                let mut reader = RecordReader::new(read);\n                let mut warmup_cnt = 0;\n                loop {\n                    let next = reader.read_next_owned();\n                    match next {\n                        Ok(res) => match res {\n                            Some(vec) => {\n                                // info!(\"read one tfRecord\");\n                                match PredictionLog::decode(&mut Cursor::new(vec))\n                                    .context(\"can't parse PredictonLog\")?\n                                {\n                                    PredictionLog {\n                                        log_metadata: _,\n                                        log_type:\n                                            Some(LogType::PredictLog(PredictLog {\n                                                request: Some(mut req),\n                                                response: _,\n                                            })),\n                                    } => {\n                                        if warmup_cnt == ARGS.max_warmup_records {\n                                            //warm up to max_warmup_records  records\n                                            warn!(\n                                                \"reached max warmup {} records, exit warmup for {}\",\n                                                ARGS.max_warmup_records,\n                                                MODEL_SPECS[self.model_idx]\n                                            );\n                                            break;\n                                        }\n                                        self.do_predict(\n                                            vec![req.take_input_vals(&self.input_names)],\n                                            1,\n                                        );\n                                        sleep(Duration::from_millis(100));\n                                        warmup_cnt += 1;\n                                    }\n                                    _ => error!(\"some wrong record in warming up file\"),\n                                }\n                            }\n                            None => {\n                                info!(\"end of warmup file, warmed up with records: {}\", warmup_cnt);\n                                break;\n                            }\n                        },\n                        Err(RecordReadError::CorruptFile)\n                        | Err(RecordReadError::IoError { .. }) => {\n                            error!(\"read tfrecord error for warmup files, skip\");\n                        }\n                        _ => {}\n                    }\n                }\n            }\n            Ok(())\n        }\n\n        #[inline(always)]\n        fn do_predict(\n            &self,\n            input_tensors: Vec<Vec<TensorInput>>,\n            batch_size: u64,\n        ) -> (Vec<TensorReturnEnum>, Vec<Vec<usize>>) {\n            // let mut batch_ends = input_tensors.iter().map(|t| t.len()).collect::<Vec<usize>>();\n            let output_size = self.output_names.len() as u64;\n            let input_size = self.input_names.len() as u64;\n            debug!(\n                \"Request for Tensorflow with batch size: {} and input_size: {}\",\n                batch_size, input_size\n            );\n            // build a set of input TF tensors\n\n            let batch_end = (1usize..=input_tensors.len() as usize)\n                .into_iter()\n                .collect_vec();\n            let mut batch_ends = vec![batch_end; output_size as usize];\n\n            let batched_tensors = TensorInputEnum::merge_batch(input_tensors)\n                .into_iter()\n                .enumerate()\n                .map(|(_, i)| TFModel::convert_to_tftensor_enum(i, input_size, batch_size))\n                .collect_vec();\n\n            let mut args = SessionRunArgs::new();\n            for (index, tf_tensor) in batched_tensors.iter().enumerate() {\n                match tf_tensor {\n                    TFTensorEnum::String(inner) => args.add_feed(&self.input_ops[index], 0, inner),\n                    TFTensorEnum::Int(inner) => args.add_feed(&self.input_ops[index], 0, inner),\n                    TFTensorEnum::Int64(inner) => args.add_feed(&self.input_ops[index], 0, inner),\n                    TFTensorEnum::Float(inner) => args.add_feed(&self.input_ops[index], 0, inner),\n                    TFTensorEnum::Double(inner) => args.add_feed(&self.input_ops[index], 0, inner),\n                    TFTensorEnum::Boolean(inner) => args.add_feed(&self.input_ops[index], 0, inner),\n                }\n            }\n            // For output ops, we receive the same op object by name. Actual tensor tokens are available at different offsets.\n            // Since indices are ordered, its important to specify output flag to Navi in the same order.\n            let token_outputs = self\n                .output_ops\n                .iter()\n                .enumerate()\n                .map(|(idx, op)| args.request_fetch(op, idx as i32))\n                .collect_vec();\n            match self.bundle.session.run(&mut args) {\n                Ok(_) => (),\n                Err(e) => {\n                    NUM_REQUESTS_FAILED.inc_by(batch_size);\n                    NUM_REQUESTS_FAILED_BY_MODEL\n                        .with_label_values(&[&MODEL_SPECS[self.model_idx]])\n                        .inc_by(batch_size);\n                    INFERENCE_FAILED_REQUESTS_BY_MODEL\n                        .with_label_values(&[&MODEL_SPECS[self.model_idx]])\n                        .inc_by(batch_size);\n                    panic!(\"{model}: {e:?}\", model = MODEL_SPECS[self.model_idx], e = e);\n                }\n            }\n            let mut predict_return = vec![];\n            // Check the output.\n            for (index, token_output) in token_outputs.iter().enumerate() {\n                // same ops, with type info at different offsets.\n                let (res, width) = match self.output_ops[index].output_type(index) {\n                    DataType::Float => {\n                        let (tensor_output, tensor_width) =\n                            TFModel::fetch_output(&mut args, token_output, batch_size, output_size);\n                        (\n                            TensorReturnEnum::FloatTensorReturn(Box::new(tensor_output)),\n                            tensor_width,\n                        )\n                    }\n                    DataType::Int64 => {\n                        let (tensor_output, tensor_width) =\n                            TFModel::fetch_output(&mut args, token_output, batch_size, output_size);\n                        (\n                            TensorReturnEnum::Int64TensorReturn(Box::new(tensor_output)),\n                            tensor_width,\n                        )\n                    }\n                    DataType::Int32 => {\n                        let (tensor_output, tensor_width) =\n                            TFModel::fetch_output(&mut args, token_output, batch_size, output_size);\n                        (\n                            TensorReturnEnum::Int32TensorReturn(Box::new(tensor_output)),\n                            tensor_width,\n                        )\n                    }\n                    DataType::String => {\n                        let (tensor_output, tensor_width) =\n                            TFModel::fetch_output(&mut args, token_output, batch_size, output_size);\n                        (\n                            TensorReturnEnum::StringTensorReturn(Box::new(tensor_output)),\n                            tensor_width,\n                        )\n                    }\n                    _ => panic!(\"Unsupported return type!\"),\n                };\n                let width = width as usize;\n                for b in batch_ends[index].iter_mut() {\n                    *b *= width;\n                }\n                predict_return.push(res)\n            }\n            //TODO: remove in the future\n            //TODO: support actual mtl model outputs\n            (predict_return, batch_ends)\n        }\n        #[inline(always)]\n        fn model_idx(&self) -> usize {\n            self.model_idx\n        }\n        #[inline(always)]\n        fn version(&self) -> i64 {\n            self.version\n        }\n    }\n}\n"
  },
  {
    "path": "navi/navi/src/torch_model.rs",
    "content": "#[cfg(feature = \"torch\")]\npub mod torch {\n    use std::fmt;\n    use std::fmt::Display;\n    use std::string::String;\n\n    use crate::TensorReturnEnum;\n    use crate::SerializedInput;\n    use crate::bootstrap::TensorInput;\n    use crate::cli_args::{Args, ARGS, MODEL_SPECS};\n    use crate::metrics;\n    use crate::metrics::{\n        INFERENCE_FAILED_REQUESTS_BY_MODEL, NUM_REQUESTS_FAILED, NUM_REQUESTS_FAILED_BY_MODEL,\n    };\n    use crate::predict_service::Model;\n    use anyhow::Result;\n    use dr_transform::converter::BatchPredictionRequestToTorchTensorConverter;\n    use dr_transform::converter::Converter;\n    use serde_json::Value;\n    use tch::Tensor;\n    use tch::{kind, CModule, IValue};\n\n    #[derive(Debug)]\n    pub struct TorchModel {\n        pub model_idx: usize,\n        pub version: i64,\n        pub module: CModule,\n        pub export_dir: String,\n        // FIXME: make this Box<Option<..>> so input converter can be optional.\n        // Also consider adding output_converter.\n        pub input_converter: Box<dyn Converter>,\n    }\n\n    impl Display for TorchModel {\n        fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n            write!(\n                f,\n                \"idx: {}, torch model_name:{}, version:{}\",\n                self.model_idx, MODEL_SPECS[self.model_idx], self.version\n            )\n        }\n    }\n\n    impl TorchModel {\n        pub fn new(idx: usize, version: String, _model_config: &Value) -> Result<TorchModel> {\n            let export_dir = format!(\"{}/{}/model.pt\", ARGS.model_dir[idx], version);\n            let model = CModule::load(&export_dir).unwrap();\n            let torch_model = TorchModel {\n                model_idx: idx,\n                version: Args::version_str_to_epoch(&version)?,\n                module: model,\n                export_dir,\n                //TODO: move converter lookup in a registry.\n                input_converter: Box::new(BatchPredictionRequestToTorchTensorConverter::new(\n                    &ARGS.model_dir[idx].as_str(),\n                    version.as_str(),\n                    vec![],\n                    Some(&metrics::register_dynamic_metrics),\n                )),\n            };\n\n            torch_model.warmup()?;\n            Ok(torch_model)\n        }\n        #[inline(always)]\n        pub fn decode_to_inputs(bytes: SerializedInput) -> Vec<Tensor> {\n            //FIXME: for now we generate 4 random tensors as inputs to unblock end to end testing\n            //when Shajan's decoder is ready we will swap\n            let row = bytes.len() as i64;\n            let t1 = Tensor::randn(&[row, 5293], kind::FLOAT_CPU); //continuous\n            let t2 = Tensor::randint(10, &[row, 149], kind::INT64_CPU); //binary\n            let t3 = Tensor::randint(10, &[row, 320], kind::INT64_CPU); //discrete\n            let t4 = Tensor::randn(&[row, 200], kind::FLOAT_CPU); //user_embedding\n            let t5 = Tensor::randn(&[row, 200], kind::FLOAT_CPU); //user_eng_embedding\n            let t6 = Tensor::randn(&[row, 200], kind::FLOAT_CPU); //author_embedding\n\n            vec![t1, t2, t3, t4, t5, t6]\n        }\n        #[inline(always)]\n        pub fn output_to_vec(res: IValue, dst: &mut Vec<f32>) {\n            match res {\n                IValue::Tensor(tensor) => TorchModel::tensors_to_vec(&[tensor], dst),\n                IValue::Tuple(ivalues) => {\n                    TorchModel::tensors_to_vec(&TorchModel::ivalues_to_tensors(ivalues), dst)\n                }\n                _ => panic!(\"we only support output as a single tensor or a vec of tensors\"),\n            }\n        }\n        #[inline(always)]\n        pub fn tensor_flatten_size(t: &Tensor) -> usize {\n            t.size().into_iter().fold(1, |acc, x| acc * x) as usize\n        }\n        #[inline(always)]\n        pub fn tensor_to_vec<T: kind::Element>(res: &Tensor) -> Vec<T> {\n            let size = TorchModel::tensor_flatten_size(res);\n            let mut res_f32: Vec<T> = Vec::with_capacity(size);\n            unsafe {\n                res_f32.set_len(size);\n            }\n            res.copy_data(res_f32.as_mut_slice(), size);\n            // println!(\"Copied tensor:{}, {:?}\", res_f32.len(), res_f32);\n            res_f32\n        }\n        #[inline(always)]\n        pub fn tensors_to_vec(tensors: &[Tensor], dst: &mut Vec<f32>) {\n            let mut offset = dst.len();\n            tensors.iter().for_each(|t| {\n                let size = TorchModel::tensor_flatten_size(t);\n                let next_size = offset + size;\n                unsafe {\n                    dst.set_len(next_size);\n                }\n                t.copy_data(&mut dst[offset..], size);\n                offset = next_size;\n            });\n        }\n        pub fn ivalues_to_tensors(ivalues: Vec<IValue>) -> Vec<Tensor> {\n            ivalues\n                .into_iter()\n                .map(|t| {\n                    if let IValue::Tensor(vanilla_t) = t {\n                        vanilla_t\n                    } else {\n                        panic!(\"not a tensor\")\n                    }\n                })\n                .collect::<Vec<Tensor>>()\n        }\n    }\n\n    impl Model for TorchModel {\n        fn warmup(&self) -> Result<()> {\n            Ok(())\n        }\n        //TODO: torch runtime needs some refactor to make it a generic interface\n        #[inline(always)]\n        fn do_predict(\n            &self,\n            input_tensors: Vec<Vec<TensorInput>>,\n            total_len: u64,\n        ) -> (Vec<TensorReturnEnum>, Vec<Vec<usize>>) {\n            let mut buf: Vec<f32> = Vec::with_capacity(10_000);\n            let mut batch_ends = vec![0usize; input_tensors.len()];\n            for (i, batch_bytes_in_request) in input_tensors.into_iter().enumerate() {\n                for _ in batch_bytes_in_request.into_iter() {\n                    //FIXME: for now use some hack\n                    let model_input = TorchModel::decode_to_inputs(vec![0u8; 30]); //self.input_converter.convert(bytes);\n                    let input_batch_tensors = model_input\n                        .into_iter()\n                        .map(|t| IValue::Tensor(t))\n                        .collect::<Vec<IValue>>();\n                    // match self.module.forward_is(&input_batch_tensors) {\n                    match self.module.method_is(\"forward_serve\", &input_batch_tensors) {\n                        Ok(res) => TorchModel::output_to_vec(res, &mut buf),\n                        Err(e) => {\n                            NUM_REQUESTS_FAILED.inc_by(total_len);\n                            NUM_REQUESTS_FAILED_BY_MODEL\n                                .with_label_values(&[&MODEL_SPECS[self.model_idx]])\n                                .inc_by(total_len);\n                            INFERENCE_FAILED_REQUESTS_BY_MODEL\n                                .with_label_values(&[&MODEL_SPECS[self.model_idx]])\n                                .inc_by(total_len);\n                            panic!(\"{model}: {e:?}\", model = MODEL_SPECS[self.model_idx], e = e);\n                        }\n                    }\n                }\n                batch_ends[i] = buf.len();\n            }\n            (\n                vec![TensorReturnEnum::FloatTensorReturn(Box::new(buf))],\n                vec![batch_ends],\n            )\n        }\n        #[inline(always)]\n        fn model_idx(&self) -> usize {\n            self.model_idx\n        }\n        #[inline(always)]\n        fn version(&self) -> i64 {\n            self.version\n        }\n    }\n}\n"
  },
  {
    "path": "navi/segdense/Cargo.toml",
    "content": "[package]\nname = \"segdense\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n\n[dependencies]\nenv_logger = \"0.10.0\"\nserde = { version = \"1.0.104\", features = [\"derive\"] }\nserde_json = \"1.0.48\"\nlog = \"0.4.17\"\n"
  },
  {
    "path": "navi/segdense/src/error.rs",
    "content": "use std::fmt::Display;\n\n/**\n * Custom error\n */\n#[derive(Debug)]\npub enum SegDenseError {\n    IoError(std::io::Error),\n    Json(serde_json::Error),\n    JsonMissingRoot,\n    JsonMissingObject,\n    JsonMissingArray,\n    JsonArraySize,\n    JsonMissingInputFeature,\n}\n\nimpl Display for SegDenseError {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        match self {\n            SegDenseError::IoError(io_error) => write!(f, \"{}\", io_error),\n            SegDenseError::Json(serde_json) => write!(f, \"{}\", serde_json),\n            SegDenseError::JsonMissingRoot => {\n                write!(f, \"{}\", \"SegDense JSON: Root Node note found!\")\n            }\n            SegDenseError::JsonMissingObject => {\n                write!(f, \"{}\", \"SegDense JSON: Object note found!\")\n            }\n            SegDenseError::JsonMissingArray => {\n                write!(f, \"{}\", \"SegDense JSON: Array Node note found!\")\n            }\n            SegDenseError::JsonArraySize => {\n                write!(f, \"{}\", \"SegDense JSON: Array size not as expected!\")\n            }\n            SegDenseError::JsonMissingInputFeature => {\n                write!(f, \"{}\", \"SegDense JSON: Missing input feature!\")\n            }\n        }\n    }\n}\n\nimpl std::error::Error for SegDenseError {}\n\nimpl From<std::io::Error> for SegDenseError {\n    fn from(err: std::io::Error) -> Self {\n        SegDenseError::IoError(err)\n    }\n}\n\nimpl From<serde_json::Error> for SegDenseError {\n    fn from(err: serde_json::Error) -> Self {\n        SegDenseError::Json(err)\n    }\n}\n"
  },
  {
    "path": "navi/segdense/src/lib.rs",
    "content": "pub mod error;\npub mod mapper;\npub mod segdense_transform_spec_home_recap_2022;\npub mod util;\n"
  },
  {
    "path": "navi/segdense/src/main.rs",
    "content": "use std::env;\nuse std::fs;\n\nuse segdense::error::SegDenseError;\nuse segdense::util;\n\nfn main() -> Result<(), SegDenseError> {\n    env_logger::init();\n    let args: Vec<String> = env::args().collect();\n\n    let schema_file_name: &str = if args.len() == 1 {\n        \"json/compact.json\"\n    } else {\n        &args[1]\n    };\n\n    let json_str = fs::read_to_string(schema_file_name)?;\n\n    util::safe_load_config(&json_str)?;\n\n    Ok(())\n}\n"
  },
  {
    "path": "navi/segdense/src/mapper.rs",
    "content": "use std::collections::HashMap;\n\n#[derive(Debug)]\npub struct FeatureInfo {\n    pub tensor_index: i8,\n    pub index_within_tensor: i64,\n}\n\npub static NULL_INFO: FeatureInfo = FeatureInfo {\n    tensor_index: -1,\n    index_within_tensor: -1,\n};\n\n#[derive(Debug, Default)]\npub struct FeatureMapper {\n    map: HashMap<i64, FeatureInfo>,\n}\n\nimpl FeatureMapper {\n    pub fn new() -> FeatureMapper {\n        FeatureMapper {\n            map: HashMap::new(),\n        }\n    }\n}\n\npub trait MapWriter {\n    fn set(&mut self, feature_id: i64, info: FeatureInfo);\n}\n\npub trait MapReader {\n    fn get(&self, feature_id: &i64) -> Option<&FeatureInfo>;\n}\n\nimpl MapWriter for FeatureMapper {\n    fn set(&mut self, feature_id: i64, info: FeatureInfo) {\n        self.map.insert(feature_id, info);\n    }\n}\n\nimpl MapReader for FeatureMapper {\n    fn get(&self, feature_id: &i64) -> Option<&FeatureInfo> {\n        self.map.get(feature_id)\n    }\n}\n"
  },
  {
    "path": "navi/segdense/src/segdense_transform_spec_home_recap_2022.rs",
    "content": "use serde::{Deserialize, Serialize};\nuse serde_json::Value;\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct Root {\n    #[serde(rename = \"common_prefix\")]\n    pub common_prefix: String,\n    #[serde(rename = \"densification_transform_spec\")]\n    pub densification_transform_spec: DensificationTransformSpec,\n    #[serde(rename = \"identity_transform_spec\")]\n    pub identity_transform_spec: Vec<IdentityTransformSpec>,\n    #[serde(rename = \"complex_feature_type_transform_spec\")]\n    pub complex_feature_type_transform_spec: Vec<ComplexFeatureTypeTransformSpec>,\n    #[serde(rename = \"input_features_map\")]\n    pub input_features_map: Value,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct DensificationTransformSpec {\n    pub discrete: Discrete,\n    pub cont: Cont,\n    pub binary: Binary,\n    pub string: Value, // Use StringType\n    pub blob: Blob,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct Discrete {\n    pub tag: String,\n    #[serde(rename = \"generic_feature_type\")]\n    pub generic_feature_type: i64,\n    #[serde(rename = \"feature_identifier\")]\n    pub feature_identifier: String,\n    #[serde(rename = \"fixed_length\")]\n    pub fixed_length: i64,\n    #[serde(rename = \"default_value\")]\n    pub default_value: DefaultValue,\n    #[serde(rename = \"input_features\")]\n    pub input_features: Vec<InputFeature>,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct DefaultValue {\n    #[serde(rename = \"type\")]\n    pub type_field: String,\n    pub value: String,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct InputFeature {\n    #[serde(rename = \"feature_id\")]\n    pub feature_id: i64,\n    #[serde(rename = \"full_feature_name\")]\n    pub full_feature_name: String,\n    #[serde(rename = \"feature_type\")]\n    pub feature_type: i64,\n    pub index: i64,\n    #[serde(rename = \"maybe_exclude\")]\n    pub maybe_exclude: bool,\n    pub tag: String,\n    #[serde(rename = \"added_at\")]\n    pub added_at: i64,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct Cont {\n    pub tag: String,\n    #[serde(rename = \"generic_feature_type\")]\n    pub generic_feature_type: i64,\n    #[serde(rename = \"feature_identifier\")]\n    pub feature_identifier: String,\n    #[serde(rename = \"fixed_length\")]\n    pub fixed_length: i64,\n    #[serde(rename = \"default_value\")]\n    pub default_value: DefaultValue,\n    #[serde(rename = \"input_features\")]\n    pub input_features: Vec<InputFeature>,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct Binary {\n    pub tag: String,\n    #[serde(rename = \"generic_feature_type\")]\n    pub generic_feature_type: i64,\n    #[serde(rename = \"feature_identifier\")]\n    pub feature_identifier: String,\n    #[serde(rename = \"fixed_length\")]\n    pub fixed_length: i64,\n    #[serde(rename = \"default_value\")]\n    pub default_value: DefaultValue,\n    #[serde(rename = \"input_features\")]\n    pub input_features: Vec<InputFeature>,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct StringType {\n    pub tag: String,\n    #[serde(rename = \"generic_feature_type\")]\n    pub generic_feature_type: i64,\n    #[serde(rename = \"feature_identifier\")]\n    pub feature_identifier: String,\n    #[serde(rename = \"fixed_length\")]\n    pub fixed_length: i64,\n    #[serde(rename = \"default_value\")]\n    pub default_value: DefaultValue,\n    #[serde(rename = \"input_features\")]\n    pub input_features: Vec<InputFeature>,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct Blob {\n    pub tag: String,\n    #[serde(rename = \"generic_feature_type\")]\n    pub generic_feature_type: i64,\n    #[serde(rename = \"feature_identifier\")]\n    pub feature_identifier: String,\n    #[serde(rename = \"fixed_length\")]\n    pub fixed_length: i64,\n    #[serde(rename = \"default_value\")]\n    pub default_value: DefaultValue,\n    #[serde(rename = \"input_features\")]\n    pub input_features: Vec<Value>,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct IdentityTransformSpec {\n    #[serde(rename = \"feature_id\")]\n    pub feature_id: i64,\n    #[serde(rename = \"full_feature_name\")]\n    pub full_feature_name: String,\n    #[serde(rename = \"feature_type\")]\n    pub feature_type: i64,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct ComplexFeatureTypeTransformSpec {\n    #[serde(rename = \"feature_id\")]\n    pub feature_id: i64,\n    #[serde(rename = \"full_feature_name\")]\n    pub full_feature_name: String,\n    #[serde(rename = \"feature_type\")]\n    pub feature_type: i64,\n    pub index: i64,\n    #[serde(rename = \"maybe_exclude\")]\n    pub maybe_exclude: bool,\n    pub tag: String,\n    #[serde(rename = \"tensor_data_type\")]\n    pub tensor_data_type: Option<i64>,\n    #[serde(rename = \"added_at\")]\n    pub added_at: i64,\n    #[serde(rename = \"tensor_shape\")]\n    #[serde(default)]\n    pub tensor_shape: Vec<i64>,\n}\n\n#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"camelCase\")]\npub struct InputFeatureMapRecord {\n    #[serde(rename = \"feature_id\")]\n    pub feature_id: i64,\n    #[serde(rename = \"full_feature_name\")]\n    pub full_feature_name: String,\n    #[serde(rename = \"feature_type\")]\n    pub feature_type: i64,\n    pub index: i64,\n    #[serde(rename = \"maybe_exclude\")]\n    pub maybe_exclude: bool,\n    pub tag: String,\n    #[serde(rename = \"added_at\")]\n    pub added_at: i64,\n}\n"
  },
  {
    "path": "navi/segdense/src/util.rs",
    "content": "use log::debug;\nuse std::fs;\n\nuse serde_json::{Map, Value};\n\nuse crate::error::SegDenseError;\nuse crate::mapper::{FeatureInfo, FeatureMapper, MapWriter};\nuse crate::segdense_transform_spec_home_recap_2022::{self as seg_dense, InputFeature};\n\npub fn load_config(file_name: &str) -> Result<seg_dense::Root, SegDenseError> {\n    let json_str = fs::read_to_string(file_name)?;\n    // &format!(\"Unable to load segdense file {}\", file_name));\n    let seg_dense_config = parse(&json_str)?;\n    // &format!(\"Unable to parse segdense file {}\", file_name));\n    Ok(seg_dense_config)\n}\n\npub fn parse(json_str: &str) -> Result<seg_dense::Root, SegDenseError> {\n    let root: seg_dense::Root = serde_json::from_str(json_str)?;\n    Ok(root)\n}\n\n/**\n * Given a json string containing a seg dense schema create a feature mapper\n * which is essentially:\n *\n *   {feature-id -> (Tensor Index, Index of feature within the tensor)}\n *\n *   Feature id : 64 bit hash of the feature name used in DataRecords.\n *\n *   Tensor Index : A vector of tensors is passed to the model. Tensor\n *     index refers to the tensor this feature is part of.\n *\n *   Index of feature in tensor : The tensors are vectors, the index of\n *     feature is the position to put the feature value.\n *\n * There are many assumptions made in this function that is very model specific.\n * These assumptions are called out below and need to be schematized eventually.\n *\n * Call this once for each segdense schema and cache the FeatureMapper.\n */\npub fn safe_load_config(json_str: &str) -> Result<FeatureMapper, SegDenseError> {\n    let root = parse(json_str)?;\n    load_from_parsed_config(root)\n}\n\n// Perf note : make 'root' un-owned\npub fn load_from_parsed_config(root: seg_dense::Root) -> Result<FeatureMapper, SegDenseError> {\n    let v = root.input_features_map;\n\n    // Do error check\n    let map: Map<String, Value> = match v {\n        Value::Object(map) => map,\n        _ => return Err(SegDenseError::JsonMissingObject),\n    };\n\n    let mut fm: FeatureMapper = FeatureMapper::new();\n\n    let items = map.values();\n\n    // Perf : Consider a way to avoid clone here\n    for item in items.cloned() {\n        let mut vec = match item {\n            Value::Array(v) => v,\n            _ => return Err(SegDenseError::JsonMissingArray),\n        };\n\n        if vec.len() != 1 {\n            return Err(SegDenseError::JsonArraySize);\n        }\n\n        let val = vec.pop().unwrap();\n\n        let input_feature: seg_dense::InputFeature = serde_json::from_value(val)?;\n        let feature_id = input_feature.feature_id;\n        let feature_info = to_feature_info(&input_feature);\n\n        match feature_info {\n            Some(info) => {\n                debug!(\"{:?}\", info);\n                fm.set(feature_id, info)\n            }\n            None => (),\n        }\n    }\n\n    Ok(fm)\n}\n#[allow(dead_code)]\nfn add_feature_info_to_mapper(\n    feature_mapper: &mut FeatureMapper,\n    input_features: &Vec<InputFeature>,\n) {\n    for input_feature in input_features.iter() {\n        let feature_id = input_feature.feature_id;\n        let feature_info = to_feature_info(input_feature);\n\n        match feature_info {\n            Some(info) => {\n                debug!(\"{:?}\", info);\n                feature_mapper.set(feature_id, info)\n            }\n            None => (),\n        }\n    }\n}\n\npub fn to_feature_info(input_feature: &seg_dense::InputFeature) -> Option<FeatureInfo> {\n    if input_feature.maybe_exclude {\n        return None;\n    }\n\n    // This part needs to be schema driven\n    //\n    //   tensor index : Which of these tensors this feature is part of\n    //      [Continious, Binary, Discrete, User_embedding, user_eng_embedding, author_embedding]\n    //      Note that this order is fixed/hardcoded here, and need to be schematized\n    //\n    let tensor_idx: i8 = match input_feature.feature_id {\n        // user.timelines.twhin_user_follow_embeddings.twhin_user_follow_embeddings\n        // Feature name is mapped to a feature-id value. The hardcoded values below correspond to a specific feature name.\n        -2550691008059411095 => 3,\n\n        // user.timelines.twhin_user_engagement_embeddings.twhin_user_engagement_embeddings\n        5390650078733277231 => 4,\n\n        // original_author.timelines.twhin_author_follow_embeddings.twhin_author_follow_embeddings\n        3223956748566688423 => 5,\n\n        _ => match input_feature.feature_type {\n            //   feature_type : src/thrift/com/twitter/ml/api/data.thrift\n            //       BINARY = 1, CONTINUOUS = 2, DISCRETE = 3,\n            //    Map to slots in [Continious, Binary, Discrete, ..]\n            1 => 1,\n            2 => 0,\n            3 => 2,\n            _ => -1,\n        },\n    };\n\n    if input_feature.index < 0 {\n        return None;\n    }\n\n    // Handle this case later\n    if tensor_idx == -1 {\n        return None;\n    }\n\n    Some(FeatureInfo {\n        tensor_index: tensor_idx,\n        index_within_tensor: input_feature.index,\n    })\n}\n"
  },
  {
    "path": "navi/thrift_bpr_adapter/thrift/Cargo.toml",
    "content": "[package]\nname = \"bpr_thrift\"\ndescription = \"Thrift parser for Batch Prediction Request\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[dependencies]\nthrift = \"0.17.0\"\n"
  },
  {
    "path": "navi/thrift_bpr_adapter/thrift/src/data.rs",
    "content": "// Autogenerated by Thrift Compiler (0.17.0)\n// DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING\n\n#![allow(unused_imports)]\n#![allow(unused_extern_crates)]\n#![allow(clippy::too_many_arguments, clippy::type_complexity, clippy::vec_box)]\n#![cfg_attr(rustfmt, rustfmt_skip)]\n\nuse std::cell::RefCell;\nuse std::collections::{BTreeMap, BTreeSet};\nuse std::convert::{From, TryFrom};\nuse std::default::Default;\nuse std::error::Error;\nuse std::fmt;\nuse std::fmt::{Display, Formatter};\nuse std::rc::Rc;\n\nuse thrift::OrderedFloat;\nuse thrift::{ApplicationError, ApplicationErrorKind, ProtocolError, ProtocolErrorKind, TThriftClient};\nuse thrift::protocol::{TFieldIdentifier, TListIdentifier, TMapIdentifier, TMessageIdentifier, TMessageType, TInputProtocol, TOutputProtocol, TSerializable, TSetIdentifier, TStructIdentifier, TType};\nuse thrift::protocol::field_id;\nuse thrift::protocol::verify_expected_message_type;\nuse thrift::protocol::verify_expected_sequence_number;\nuse thrift::protocol::verify_expected_service_call;\nuse thrift::protocol::verify_required_field_exists;\nuse thrift::server::TProcessor;\n\nuse crate::tensor;\n\n#[derive(Copy, Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct FeatureType(pub i32);\n\nimpl FeatureType {\n  pub const BINARY: FeatureType = FeatureType(1);\n  pub const CONTINUOUS: FeatureType = FeatureType(2);\n  pub const DISCRETE: FeatureType = FeatureType(3);\n  pub const STRING: FeatureType = FeatureType(4);\n  pub const SPARSE_BINARY: FeatureType = FeatureType(5);\n  pub const SPARSE_CONTINUOUS: FeatureType = FeatureType(6);\n  pub const UNKNOWN: FeatureType = FeatureType(7);\n  pub const BLOB: FeatureType = FeatureType(8);\n  pub const TENSOR: FeatureType = FeatureType(9);\n  pub const SPARSE_TENSOR: FeatureType = FeatureType(10);\n  pub const FEATURE_TYPE11: FeatureType = FeatureType(11);\n  pub const FEATURE_TYPE12: FeatureType = FeatureType(12);\n  pub const ENUM_VALUES: &'static [Self] = &[\n    Self::BINARY,\n    Self::CONTINUOUS,\n    Self::DISCRETE,\n    Self::STRING,\n    Self::SPARSE_BINARY,\n    Self::SPARSE_CONTINUOUS,\n    Self::UNKNOWN,\n    Self::BLOB,\n    Self::TENSOR,\n    Self::SPARSE_TENSOR,\n    Self::FEATURE_TYPE11,\n    Self::FEATURE_TYPE12,\n  ];\n}\n\nimpl TSerializable for FeatureType {\n  #[allow(clippy::trivially_copy_pass_by_ref)]\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    o_prot.write_i32(self.0)\n  }\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<FeatureType> {\n    let enum_value = i_prot.read_i32()?;\n    Ok(FeatureType::from(enum_value))\n  }\n}\n\nimpl From<i32> for FeatureType {\n  fn from(i: i32) -> Self {\n    match i {\n      1 => FeatureType::BINARY,\n      2 => FeatureType::CONTINUOUS,\n      3 => FeatureType::DISCRETE,\n      4 => FeatureType::STRING,\n      5 => FeatureType::SPARSE_BINARY,\n      6 => FeatureType::SPARSE_CONTINUOUS,\n      7 => FeatureType::UNKNOWN,\n      8 => FeatureType::BLOB,\n      9 => FeatureType::TENSOR,\n      10 => FeatureType::SPARSE_TENSOR,\n      11 => FeatureType::FEATURE_TYPE11,\n      12 => FeatureType::FEATURE_TYPE12,\n      _ => FeatureType(i)\n    }\n  }\n}\n\nimpl From<&i32> for FeatureType {\n  fn from(i: &i32) -> Self {\n    FeatureType::from(*i)\n  }\n}\n\nimpl From<FeatureType> for i32 {\n  fn from(e: FeatureType) -> i32 {\n    e.0\n  }\n}\n\nimpl From<&FeatureType> for i32 {\n  fn from(e: &FeatureType) -> i32 {\n    e.0\n  }\n}\n\n//\n// DataRecord\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct DataRecord {\n  pub binary_features: Option<BTreeSet<i64>>,\n  pub continuous_features: Option<BTreeMap<i64, OrderedFloat<f64>>>,\n  pub discrete_features: Option<BTreeMap<i64, i64>>,\n  pub string_features: Option<BTreeMap<i64, String>>,\n  pub sparse_binary_features: Option<BTreeMap<i64, BTreeSet<String>>>,\n  pub sparse_continuous_features: Option<BTreeMap<i64, BTreeMap<String, OrderedFloat<f64>>>>,\n  pub blob_features: Option<BTreeMap<i64, Vec<u8>>>,\n  pub tensors: Option<BTreeMap<i64, tensor::GeneralTensor>>,\n  pub sparse_tensors: Option<BTreeMap<i64, tensor::SparseTensor>>,\n}\n\nimpl DataRecord {\n  pub fn new<F1, F2, F3, F4, F5, F6, F7, F8, F9>(binary_features: F1, continuous_features: F2, discrete_features: F3, string_features: F4, sparse_binary_features: F5, sparse_continuous_features: F6, blob_features: F7, tensors: F8, sparse_tensors: F9) -> DataRecord where F1: Into<Option<BTreeSet<i64>>>, F2: Into<Option<BTreeMap<i64, OrderedFloat<f64>>>>, F3: Into<Option<BTreeMap<i64, i64>>>, F4: Into<Option<BTreeMap<i64, String>>>, F5: Into<Option<BTreeMap<i64, BTreeSet<String>>>>, F6: Into<Option<BTreeMap<i64, BTreeMap<String, OrderedFloat<f64>>>>>, F7: Into<Option<BTreeMap<i64, Vec<u8>>>>, F8: Into<Option<BTreeMap<i64, tensor::GeneralTensor>>>, F9: Into<Option<BTreeMap<i64, tensor::SparseTensor>>> {\n    DataRecord {\n      binary_features: binary_features.into(),\n      continuous_features: continuous_features.into(),\n      discrete_features: discrete_features.into(),\n      string_features: string_features.into(),\n      sparse_binary_features: sparse_binary_features.into(),\n      sparse_continuous_features: sparse_continuous_features.into(),\n      blob_features: blob_features.into(),\n      tensors: tensors.into(),\n      sparse_tensors: sparse_tensors.into(),\n    }\n  }\n}\n\nimpl TSerializable for DataRecord {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<DataRecord> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<BTreeSet<i64>> = None;\n    let mut f_2: Option<BTreeMap<i64, OrderedFloat<f64>>> = None;\n    let mut f_3: Option<BTreeMap<i64, i64>> = None;\n    let mut f_4: Option<BTreeMap<i64, String>> = None;\n    let mut f_5: Option<BTreeMap<i64, BTreeSet<String>>> = None;\n    let mut f_6: Option<BTreeMap<i64, BTreeMap<String, OrderedFloat<f64>>>> = None;\n    let mut f_7: Option<BTreeMap<i64, Vec<u8>>> = None;\n    let mut f_8: Option<BTreeMap<i64, tensor::GeneralTensor>> = None;\n    let mut f_9: Option<BTreeMap<i64, tensor::SparseTensor>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let set_ident = i_prot.read_set_begin()?;\n          let mut val: BTreeSet<i64> = BTreeSet::new();\n          for _ in 0..set_ident.size {\n            let set_elem_0 = i_prot.read_i64()?;\n            val.insert(set_elem_0);\n          }\n          i_prot.read_set_end()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, OrderedFloat<f64>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_1 = i_prot.read_i64()?;\n            let map_val_2 = OrderedFloat::from(i_prot.read_double()?);\n            val.insert(map_key_1, map_val_2);\n          }\n          i_prot.read_map_end()?;\n          f_2 = Some(val);\n        },\n        3 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, i64> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_3 = i_prot.read_i64()?;\n            let map_val_4 = i_prot.read_i64()?;\n            val.insert(map_key_3, map_val_4);\n          }\n          i_prot.read_map_end()?;\n          f_3 = Some(val);\n        },\n        4 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, String> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_5 = i_prot.read_i64()?;\n            let map_val_6 = i_prot.read_string()?;\n            val.insert(map_key_5, map_val_6);\n          }\n          i_prot.read_map_end()?;\n          f_4 = Some(val);\n        },\n        5 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, BTreeSet<String>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_7 = i_prot.read_i64()?;\n            let set_ident = i_prot.read_set_begin()?;\n            let mut map_val_8: BTreeSet<String> = BTreeSet::new();\n            for _ in 0..set_ident.size {\n              let set_elem_9 = i_prot.read_string()?;\n              map_val_8.insert(set_elem_9);\n            }\n            i_prot.read_set_end()?;\n            val.insert(map_key_7, map_val_8);\n          }\n          i_prot.read_map_end()?;\n          f_5 = Some(val);\n        },\n        6 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, BTreeMap<String, OrderedFloat<f64>>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_10 = i_prot.read_i64()?;\n            let map_ident = i_prot.read_map_begin()?;\n            let mut map_val_11: BTreeMap<String, OrderedFloat<f64>> = BTreeMap::new();\n            for _ in 0..map_ident.size {\n              let map_key_12 = i_prot.read_string()?;\n              let map_val_13 = OrderedFloat::from(i_prot.read_double()?);\n              map_val_11.insert(map_key_12, map_val_13);\n            }\n            i_prot.read_map_end()?;\n            val.insert(map_key_10, map_val_11);\n          }\n          i_prot.read_map_end()?;\n          f_6 = Some(val);\n        },\n        7 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, Vec<u8>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_14 = i_prot.read_i64()?;\n            let map_val_15 = i_prot.read_bytes()?;\n            val.insert(map_key_14, map_val_15);\n          }\n          i_prot.read_map_end()?;\n          f_7 = Some(val);\n        },\n        8 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, tensor::GeneralTensor> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_16 = i_prot.read_i64()?;\n            let map_val_17 = tensor::GeneralTensor::read_from_in_protocol(i_prot)?;\n            val.insert(map_key_16, map_val_17);\n          }\n          i_prot.read_map_end()?;\n          f_8 = Some(val);\n        },\n        9 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, tensor::SparseTensor> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_18 = i_prot.read_i64()?;\n            let map_val_19 = tensor::SparseTensor::read_from_in_protocol(i_prot)?;\n            val.insert(map_key_18, map_val_19);\n          }\n          i_prot.read_map_end()?;\n          f_9 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    let ret = DataRecord {\n      binary_features: f_1,\n      continuous_features: f_2,\n      discrete_features: f_3,\n      string_features: f_4,\n      sparse_binary_features: f_5,\n      sparse_continuous_features: f_6,\n      blob_features: f_7,\n      tensors: f_8,\n      sparse_tensors: f_9,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"DataRecord\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    if let Some(ref fld_var) = self.binary_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"binaryFeatures\", TType::Set, 1))?;\n      o_prot.write_set_begin(&TSetIdentifier::new(TType::I64, fld_var.len() as i32))?;\n      for e in fld_var {\n        o_prot.write_i64(*e)?;\n      }\n      o_prot.write_set_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.continuous_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"continuousFeatures\", TType::Map, 2))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Double, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_double((*v).into())?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.discrete_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"discreteFeatures\", TType::Map, 3))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::I64, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_i64(*v)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.string_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"stringFeatures\", TType::Map, 4))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::String, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_string(v)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_binary_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseBinaryFeatures\", TType::Map, 5))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Set, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_set_begin(&TSetIdentifier::new(TType::String, v.len() as i32))?;\n        for e in v {\n          o_prot.write_string(e)?;\n        }\n        o_prot.write_set_end()?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_continuous_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseContinuousFeatures\", TType::Map, 6))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Map, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_map_begin(&TMapIdentifier::new(TType::String, TType::Double, v.len() as i32))?;\n        for (k, v) in v {\n          o_prot.write_string(k)?;\n          o_prot.write_double((*v).into())?;\n        }\n        o_prot.write_map_end()?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.blob_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"blobFeatures\", TType::Map, 7))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::String, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_bytes(v)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.tensors {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"tensors\", TType::Map, 8))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Struct, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        v.write_to_out_protocol(o_prot)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_tensors {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseTensors\", TType::Map, 9))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Struct, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        v.write_to_out_protocol(o_prot)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\nimpl Default for DataRecord {\n  fn default() -> Self {\n    DataRecord{\n      binary_features: Some(BTreeSet::new()),\n      continuous_features: Some(BTreeMap::new()),\n      discrete_features: Some(BTreeMap::new()),\n      string_features: Some(BTreeMap::new()),\n      sparse_binary_features: Some(BTreeMap::new()),\n      sparse_continuous_features: Some(BTreeMap::new()),\n      blob_features: Some(BTreeMap::new()),\n      tensors: Some(BTreeMap::new()),\n      sparse_tensors: Some(BTreeMap::new()),\n    }\n  }\n}\n\n//\n// CompactDataRecord\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct CompactDataRecord {\n  pub binary_features: Option<BTreeSet<i64>>,\n  pub continuous_features: Option<BTreeMap<i64, i32>>,\n  pub discrete_features: Option<BTreeMap<i64, i64>>,\n  pub string_features: Option<BTreeMap<i64, String>>,\n  pub sparse_binary_features: Option<BTreeMap<i64, BTreeSet<String>>>,\n  pub sparse_binary_features_with16b_sparse_key: Option<BTreeMap<i64, BTreeSet<i16>>>,\n  pub sparse_binary_features_with32b_sparse_key: Option<BTreeMap<i64, BTreeSet<i32>>>,\n  pub sparse_binary_features_with64b_sparse_key: Option<BTreeMap<i64, BTreeSet<i64>>>,\n  pub sparse_continuous_features: Option<BTreeMap<i64, BTreeMap<String, i32>>>,\n  pub sparse_continuous_features_with16b_sparse_key: Option<BTreeMap<i64, BTreeMap<i16, i32>>>,\n  pub sparse_continuous_features_with32b_sparse_key: Option<BTreeMap<i64, BTreeMap<i32, i32>>>,\n  pub sparse_continuous_features_with64b_sparse_key: Option<BTreeMap<i64, BTreeMap<i64, i32>>>,\n  pub blob_features: Option<BTreeMap<i64, Vec<u8>>>,\n  pub tensors: Option<BTreeMap<i64, tensor::GeneralTensor>>,\n  pub sparse_tensors: Option<BTreeMap<i64, tensor::SparseTensor>>,\n}\n\nimpl CompactDataRecord {\n  pub fn new<F1, F2, F3, F4, F5, F6, F7, F8, F9, F10, F11, F12, F13, F14, F15>(binary_features: F1, continuous_features: F2, discrete_features: F3, string_features: F4, sparse_binary_features: F5, sparse_binary_features_with16b_sparse_key: F6, sparse_binary_features_with32b_sparse_key: F7, sparse_binary_features_with64b_sparse_key: F8, sparse_continuous_features: F9, sparse_continuous_features_with16b_sparse_key: F10, sparse_continuous_features_with32b_sparse_key: F11, sparse_continuous_features_with64b_sparse_key: F12, blob_features: F13, tensors: F14, sparse_tensors: F15) -> CompactDataRecord where F1: Into<Option<BTreeSet<i64>>>, F2: Into<Option<BTreeMap<i64, i32>>>, F3: Into<Option<BTreeMap<i64, i64>>>, F4: Into<Option<BTreeMap<i64, String>>>, F5: Into<Option<BTreeMap<i64, BTreeSet<String>>>>, F6: Into<Option<BTreeMap<i64, BTreeSet<i16>>>>, F7: Into<Option<BTreeMap<i64, BTreeSet<i32>>>>, F8: Into<Option<BTreeMap<i64, BTreeSet<i64>>>>, F9: Into<Option<BTreeMap<i64, BTreeMap<String, i32>>>>, F10: Into<Option<BTreeMap<i64, BTreeMap<i16, i32>>>>, F11: Into<Option<BTreeMap<i64, BTreeMap<i32, i32>>>>, F12: Into<Option<BTreeMap<i64, BTreeMap<i64, i32>>>>, F13: Into<Option<BTreeMap<i64, Vec<u8>>>>, F14: Into<Option<BTreeMap<i64, tensor::GeneralTensor>>>, F15: Into<Option<BTreeMap<i64, tensor::SparseTensor>>> {\n    CompactDataRecord {\n      binary_features: binary_features.into(),\n      continuous_features: continuous_features.into(),\n      discrete_features: discrete_features.into(),\n      string_features: string_features.into(),\n      sparse_binary_features: sparse_binary_features.into(),\n      sparse_binary_features_with16b_sparse_key: sparse_binary_features_with16b_sparse_key.into(),\n      sparse_binary_features_with32b_sparse_key: sparse_binary_features_with32b_sparse_key.into(),\n      sparse_binary_features_with64b_sparse_key: sparse_binary_features_with64b_sparse_key.into(),\n      sparse_continuous_features: sparse_continuous_features.into(),\n      sparse_continuous_features_with16b_sparse_key: sparse_continuous_features_with16b_sparse_key.into(),\n      sparse_continuous_features_with32b_sparse_key: sparse_continuous_features_with32b_sparse_key.into(),\n      sparse_continuous_features_with64b_sparse_key: sparse_continuous_features_with64b_sparse_key.into(),\n      blob_features: blob_features.into(),\n      tensors: tensors.into(),\n      sparse_tensors: sparse_tensors.into(),\n    }\n  }\n}\n\nimpl TSerializable for CompactDataRecord {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<CompactDataRecord> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<BTreeSet<i64>> = None;\n    let mut f_2: Option<BTreeMap<i64, i32>> = None;\n    let mut f_3: Option<BTreeMap<i64, i64>> = None;\n    let mut f_4: Option<BTreeMap<i64, String>> = None;\n    let mut f_5: Option<BTreeMap<i64, BTreeSet<String>>> = None;\n    let mut f_6: Option<BTreeMap<i64, BTreeSet<i16>>> = None;\n    let mut f_7: Option<BTreeMap<i64, BTreeSet<i32>>> = None;\n    let mut f_8: Option<BTreeMap<i64, BTreeSet<i64>>> = None;\n    let mut f_9: Option<BTreeMap<i64, BTreeMap<String, i32>>> = None;\n    let mut f_10: Option<BTreeMap<i64, BTreeMap<i16, i32>>> = None;\n    let mut f_11: Option<BTreeMap<i64, BTreeMap<i32, i32>>> = None;\n    let mut f_12: Option<BTreeMap<i64, BTreeMap<i64, i32>>> = None;\n    let mut f_13: Option<BTreeMap<i64, Vec<u8>>> = None;\n    let mut f_14: Option<BTreeMap<i64, tensor::GeneralTensor>> = None;\n    let mut f_15: Option<BTreeMap<i64, tensor::SparseTensor>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let set_ident = i_prot.read_set_begin()?;\n          let mut val: BTreeSet<i64> = BTreeSet::new();\n          for _ in 0..set_ident.size {\n            let set_elem_20 = i_prot.read_i64()?;\n            val.insert(set_elem_20);\n          }\n          i_prot.read_set_end()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, i32> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_21 = i_prot.read_i64()?;\n            let map_val_22 = i_prot.read_i32()?;\n            val.insert(map_key_21, map_val_22);\n          }\n          i_prot.read_map_end()?;\n          f_2 = Some(val);\n        },\n        3 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, i64> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_23 = i_prot.read_i64()?;\n            let map_val_24 = i_prot.read_i64()?;\n            val.insert(map_key_23, map_val_24);\n          }\n          i_prot.read_map_end()?;\n          f_3 = Some(val);\n        },\n        4 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, String> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_25 = i_prot.read_i64()?;\n            let map_val_26 = i_prot.read_string()?;\n            val.insert(map_key_25, map_val_26);\n          }\n          i_prot.read_map_end()?;\n          f_4 = Some(val);\n        },\n        5 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, BTreeSet<String>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_27 = i_prot.read_i64()?;\n            let set_ident = i_prot.read_set_begin()?;\n            let mut map_val_28: BTreeSet<String> = BTreeSet::new();\n            for _ in 0..set_ident.size {\n              let set_elem_29 = i_prot.read_string()?;\n              map_val_28.insert(set_elem_29);\n            }\n            i_prot.read_set_end()?;\n            val.insert(map_key_27, map_val_28);\n          }\n          i_prot.read_map_end()?;\n          f_5 = Some(val);\n        },\n        6 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, BTreeSet<i16>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_30 = i_prot.read_i64()?;\n            let set_ident = i_prot.read_set_begin()?;\n            let mut map_val_31: BTreeSet<i16> = BTreeSet::new();\n            for _ in 0..set_ident.size {\n              let set_elem_32 = i_prot.read_i16()?;\n              map_val_31.insert(set_elem_32);\n            }\n            i_prot.read_set_end()?;\n            val.insert(map_key_30, map_val_31);\n          }\n          i_prot.read_map_end()?;\n          f_6 = Some(val);\n        },\n        7 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, BTreeSet<i32>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_33 = i_prot.read_i64()?;\n            let set_ident = i_prot.read_set_begin()?;\n            let mut map_val_34: BTreeSet<i32> = BTreeSet::new();\n            for _ in 0..set_ident.size {\n              let set_elem_35 = i_prot.read_i32()?;\n              map_val_34.insert(set_elem_35);\n            }\n            i_prot.read_set_end()?;\n            val.insert(map_key_33, map_val_34);\n          }\n          i_prot.read_map_end()?;\n          f_7 = Some(val);\n        },\n        8 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, BTreeSet<i64>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_36 = i_prot.read_i64()?;\n            let set_ident = i_prot.read_set_begin()?;\n            let mut map_val_37: BTreeSet<i64> = BTreeSet::new();\n            for _ in 0..set_ident.size {\n              let set_elem_38 = i_prot.read_i64()?;\n              map_val_37.insert(set_elem_38);\n            }\n            i_prot.read_set_end()?;\n            val.insert(map_key_36, map_val_37);\n          }\n          i_prot.read_map_end()?;\n          f_8 = Some(val);\n        },\n        9 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, BTreeMap<String, i32>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_39 = i_prot.read_i64()?;\n            let map_ident = i_prot.read_map_begin()?;\n            let mut map_val_40: BTreeMap<String, i32> = BTreeMap::new();\n            for _ in 0..map_ident.size {\n              let map_key_41 = i_prot.read_string()?;\n              let map_val_42 = i_prot.read_i32()?;\n              map_val_40.insert(map_key_41, map_val_42);\n            }\n            i_prot.read_map_end()?;\n            val.insert(map_key_39, map_val_40);\n          }\n          i_prot.read_map_end()?;\n          f_9 = Some(val);\n        },\n        10 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, BTreeMap<i16, i32>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_43 = i_prot.read_i64()?;\n            let map_ident = i_prot.read_map_begin()?;\n            let mut map_val_44: BTreeMap<i16, i32> = BTreeMap::new();\n            for _ in 0..map_ident.size {\n              let map_key_45 = i_prot.read_i16()?;\n              let map_val_46 = i_prot.read_i32()?;\n              map_val_44.insert(map_key_45, map_val_46);\n            }\n            i_prot.read_map_end()?;\n            val.insert(map_key_43, map_val_44);\n          }\n          i_prot.read_map_end()?;\n          f_10 = Some(val);\n        },\n        11 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, BTreeMap<i32, i32>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_47 = i_prot.read_i64()?;\n            let map_ident = i_prot.read_map_begin()?;\n            let mut map_val_48: BTreeMap<i32, i32> = BTreeMap::new();\n            for _ in 0..map_ident.size {\n              let map_key_49 = i_prot.read_i32()?;\n              let map_val_50 = i_prot.read_i32()?;\n              map_val_48.insert(map_key_49, map_val_50);\n            }\n            i_prot.read_map_end()?;\n            val.insert(map_key_47, map_val_48);\n          }\n          i_prot.read_map_end()?;\n          f_11 = Some(val);\n        },\n        12 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, BTreeMap<i64, i32>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_51 = i_prot.read_i64()?;\n            let map_ident = i_prot.read_map_begin()?;\n            let mut map_val_52: BTreeMap<i64, i32> = BTreeMap::new();\n            for _ in 0..map_ident.size {\n              let map_key_53 = i_prot.read_i64()?;\n              let map_val_54 = i_prot.read_i32()?;\n              map_val_52.insert(map_key_53, map_val_54);\n            }\n            i_prot.read_map_end()?;\n            val.insert(map_key_51, map_val_52);\n          }\n          i_prot.read_map_end()?;\n          f_12 = Some(val);\n        },\n        13 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, Vec<u8>> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_55 = i_prot.read_i64()?;\n            let map_val_56 = i_prot.read_bytes()?;\n            val.insert(map_key_55, map_val_56);\n          }\n          i_prot.read_map_end()?;\n          f_13 = Some(val);\n        },\n        14 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, tensor::GeneralTensor> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_57 = i_prot.read_i64()?;\n            let map_val_58 = tensor::GeneralTensor::read_from_in_protocol(i_prot)?;\n            val.insert(map_key_57, map_val_58);\n          }\n          i_prot.read_map_end()?;\n          f_14 = Some(val);\n        },\n        15 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, tensor::SparseTensor> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_59 = i_prot.read_i64()?;\n            let map_val_60 = tensor::SparseTensor::read_from_in_protocol(i_prot)?;\n            val.insert(map_key_59, map_val_60);\n          }\n          i_prot.read_map_end()?;\n          f_15 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    let ret = CompactDataRecord {\n      binary_features: f_1,\n      continuous_features: f_2,\n      discrete_features: f_3,\n      string_features: f_4,\n      sparse_binary_features: f_5,\n      sparse_binary_features_with16b_sparse_key: f_6,\n      sparse_binary_features_with32b_sparse_key: f_7,\n      sparse_binary_features_with64b_sparse_key: f_8,\n      sparse_continuous_features: f_9,\n      sparse_continuous_features_with16b_sparse_key: f_10,\n      sparse_continuous_features_with32b_sparse_key: f_11,\n      sparse_continuous_features_with64b_sparse_key: f_12,\n      blob_features: f_13,\n      tensors: f_14,\n      sparse_tensors: f_15,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"CompactDataRecord\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    if let Some(ref fld_var) = self.binary_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"binaryFeatures\", TType::Set, 1))?;\n      o_prot.write_set_begin(&TSetIdentifier::new(TType::I64, fld_var.len() as i32))?;\n      for e in fld_var {\n        o_prot.write_i64(*e)?;\n      }\n      o_prot.write_set_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.continuous_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"continuousFeatures\", TType::Map, 2))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::I32, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_i32(*v)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.discrete_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"discreteFeatures\", TType::Map, 3))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::I64, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_i64(*v)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.string_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"stringFeatures\", TType::Map, 4))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::String, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_string(v)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_binary_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseBinaryFeatures\", TType::Map, 5))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Set, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_set_begin(&TSetIdentifier::new(TType::String, v.len() as i32))?;\n        for e in v {\n          o_prot.write_string(e)?;\n        }\n        o_prot.write_set_end()?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_binary_features_with16b_sparse_key {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseBinaryFeaturesWith16bSparseKey\", TType::Map, 6))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Set, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_set_begin(&TSetIdentifier::new(TType::I16, v.len() as i32))?;\n        for e in v {\n          o_prot.write_i16(*e)?;\n        }\n        o_prot.write_set_end()?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_binary_features_with32b_sparse_key {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseBinaryFeaturesWith32bSparseKey\", TType::Map, 7))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Set, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_set_begin(&TSetIdentifier::new(TType::I32, v.len() as i32))?;\n        for e in v {\n          o_prot.write_i32(*e)?;\n        }\n        o_prot.write_set_end()?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_binary_features_with64b_sparse_key {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseBinaryFeaturesWith64bSparseKey\", TType::Map, 8))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Set, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_set_begin(&TSetIdentifier::new(TType::I64, v.len() as i32))?;\n        for e in v {\n          o_prot.write_i64(*e)?;\n        }\n        o_prot.write_set_end()?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_continuous_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseContinuousFeatures\", TType::Map, 9))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Map, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_map_begin(&TMapIdentifier::new(TType::String, TType::I32, v.len() as i32))?;\n        for (k, v) in v {\n          o_prot.write_string(k)?;\n          o_prot.write_i32(*v)?;\n        }\n        o_prot.write_map_end()?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_continuous_features_with16b_sparse_key {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseContinuousFeaturesWith16bSparseKey\", TType::Map, 10))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Map, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_map_begin(&TMapIdentifier::new(TType::I16, TType::I32, v.len() as i32))?;\n        for (k, v) in v {\n          o_prot.write_i16(*k)?;\n          o_prot.write_i32(*v)?;\n        }\n        o_prot.write_map_end()?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_continuous_features_with32b_sparse_key {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseContinuousFeaturesWith32bSparseKey\", TType::Map, 11))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Map, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_map_begin(&TMapIdentifier::new(TType::I32, TType::I32, v.len() as i32))?;\n        for (k, v) in v {\n          o_prot.write_i32(*k)?;\n          o_prot.write_i32(*v)?;\n        }\n        o_prot.write_map_end()?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_continuous_features_with64b_sparse_key {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseContinuousFeaturesWith64bSparseKey\", TType::Map, 12))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Map, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::I32, v.len() as i32))?;\n        for (k, v) in v {\n          o_prot.write_i64(*k)?;\n          o_prot.write_i32(*v)?;\n        }\n        o_prot.write_map_end()?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.blob_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"blobFeatures\", TType::Map, 13))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::String, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        o_prot.write_bytes(v)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.tensors {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"tensors\", TType::Map, 14))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Struct, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        v.write_to_out_protocol(o_prot)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_tensors {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseTensors\", TType::Map, 15))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Struct, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        v.write_to_out_protocol(o_prot)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\nimpl Default for CompactDataRecord {\n  fn default() -> Self {\n    CompactDataRecord{\n      binary_features: Some(BTreeSet::new()),\n      continuous_features: Some(BTreeMap::new()),\n      discrete_features: Some(BTreeMap::new()),\n      string_features: Some(BTreeMap::new()),\n      sparse_binary_features: Some(BTreeMap::new()),\n      sparse_binary_features_with16b_sparse_key: Some(BTreeMap::new()),\n      sparse_binary_features_with32b_sparse_key: Some(BTreeMap::new()),\n      sparse_binary_features_with64b_sparse_key: Some(BTreeMap::new()),\n      sparse_continuous_features: Some(BTreeMap::new()),\n      sparse_continuous_features_with16b_sparse_key: Some(BTreeMap::new()),\n      sparse_continuous_features_with32b_sparse_key: Some(BTreeMap::new()),\n      sparse_continuous_features_with64b_sparse_key: Some(BTreeMap::new()),\n      blob_features: Some(BTreeMap::new()),\n      tensors: Some(BTreeMap::new()),\n      sparse_tensors: Some(BTreeMap::new()),\n    }\n  }\n}\n\n//\n// TensorRecord\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct TensorRecord {\n  pub tensors: Option<BTreeMap<i64, tensor::GeneralTensor>>,\n  pub sparse_tensors: Option<BTreeMap<i64, tensor::SparseTensor>>,\n}\n\nimpl TensorRecord {\n  pub fn new<F1, F2>(tensors: F1, sparse_tensors: F2) -> TensorRecord where F1: Into<Option<BTreeMap<i64, tensor::GeneralTensor>>>, F2: Into<Option<BTreeMap<i64, tensor::SparseTensor>>> {\n    TensorRecord {\n      tensors: tensors.into(),\n      sparse_tensors: sparse_tensors.into(),\n    }\n  }\n}\n\nimpl TSerializable for TensorRecord {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<TensorRecord> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<BTreeMap<i64, tensor::GeneralTensor>> = None;\n    let mut f_2: Option<BTreeMap<i64, tensor::SparseTensor>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, tensor::GeneralTensor> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_61 = i_prot.read_i64()?;\n            let map_val_62 = tensor::GeneralTensor::read_from_in_protocol(i_prot)?;\n            val.insert(map_key_61, map_val_62);\n          }\n          i_prot.read_map_end()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let map_ident = i_prot.read_map_begin()?;\n          let mut val: BTreeMap<i64, tensor::SparseTensor> = BTreeMap::new();\n          for _ in 0..map_ident.size {\n            let map_key_63 = i_prot.read_i64()?;\n            let map_val_64 = tensor::SparseTensor::read_from_in_protocol(i_prot)?;\n            val.insert(map_key_63, map_val_64);\n          }\n          i_prot.read_map_end()?;\n          f_2 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    let ret = TensorRecord {\n      tensors: f_1,\n      sparse_tensors: f_2,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"TensorRecord\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    if let Some(ref fld_var) = self.tensors {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"tensors\", TType::Map, 1))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Struct, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        v.write_to_out_protocol(o_prot)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.sparse_tensors {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"sparseTensors\", TType::Map, 2))?;\n      o_prot.write_map_begin(&TMapIdentifier::new(TType::I64, TType::Struct, fld_var.len() as i32))?;\n      for (k, v) in fld_var {\n        o_prot.write_i64(*k)?;\n        v.write_to_out_protocol(o_prot)?;\n      }\n      o_prot.write_map_end()?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\nimpl Default for TensorRecord {\n  fn default() -> Self {\n    TensorRecord{\n      tensors: Some(BTreeMap::new()),\n      sparse_tensors: Some(BTreeMap::new()),\n    }\n  }\n}\n\n//\n// FeatureMetaInfo\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct FeatureMetaInfo {\n  pub feature_id: Option<i64>,\n  pub full_feature_name: Option<String>,\n  pub feature_type: Option<FeatureType>,\n}\n\nimpl FeatureMetaInfo {\n  pub fn new<F1, F2, F3>(feature_id: F1, full_feature_name: F2, feature_type: F3) -> FeatureMetaInfo where F1: Into<Option<i64>>, F2: Into<Option<String>>, F3: Into<Option<FeatureType>> {\n    FeatureMetaInfo {\n      feature_id: feature_id.into(),\n      full_feature_name: full_feature_name.into(),\n      feature_type: feature_type.into(),\n    }\n  }\n}\n\nimpl TSerializable for FeatureMetaInfo {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<FeatureMetaInfo> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<i64> = None;\n    let mut f_2: Option<String> = None;\n    let mut f_3: Option<FeatureType> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let val = i_prot.read_i64()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let val = i_prot.read_string()?;\n          f_2 = Some(val);\n        },\n        3 => {\n          let val = FeatureType::read_from_in_protocol(i_prot)?;\n          f_3 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    let ret = FeatureMetaInfo {\n      feature_id: f_1,\n      full_feature_name: f_2,\n      feature_type: f_3,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"FeatureMetaInfo\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    if let Some(fld_var) = self.feature_id {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"featureId\", TType::I64, 1))?;\n      o_prot.write_i64(fld_var)?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.full_feature_name {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"fullFeatureName\", TType::String, 2))?;\n      o_prot.write_string(fld_var)?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.feature_type {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"featureType\", TType::I32, 3))?;\n      fld_var.write_to_out_protocol(o_prot)?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\nimpl Default for FeatureMetaInfo {\n  fn default() -> Self {\n    FeatureMetaInfo{\n      feature_id: Some(0),\n      full_feature_name: Some(\"\".to_owned()),\n      feature_type: None,\n    }\n  }\n}\n\n//\n// FeatureMetaInfoList\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct FeatureMetaInfoList {\n  pub contents: Option<Vec<FeatureMetaInfo>>,\n}\n\nimpl FeatureMetaInfoList {\n  pub fn new<F1>(contents: F1) -> FeatureMetaInfoList where F1: Into<Option<Vec<FeatureMetaInfo>>> {\n    FeatureMetaInfoList {\n      contents: contents.into(),\n    }\n  }\n}\n\nimpl TSerializable for FeatureMetaInfoList {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<FeatureMetaInfoList> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<Vec<FeatureMetaInfo>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<FeatureMetaInfo> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_65 = FeatureMetaInfo::read_from_in_protocol(i_prot)?;\n            val.push(list_elem_65);\n          }\n          i_prot.read_list_end()?;\n          f_1 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    let ret = FeatureMetaInfoList {\n      contents: f_1,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"FeatureMetaInfoList\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    if let Some(ref fld_var) = self.contents {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"contents\", TType::List, 1))?;\n      o_prot.write_list_begin(&TListIdentifier::new(TType::Struct, fld_var.len() as i32))?;\n      for e in fld_var {\n        e.write_to_out_protocol(o_prot)?;\n      }\n      o_prot.write_list_end()?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\nimpl Default for FeatureMetaInfoList {\n  fn default() -> Self {\n    FeatureMetaInfoList{\n      contents: Some(Vec::new()),\n    }\n  }\n}\n\n"
  },
  {
    "path": "navi/thrift_bpr_adapter/thrift/src/decoder.rs",
    "content": "\n// A feature value can be one of these\nenum FeatureVal {\n  Empty,\n  U8Vector(Vec<u8>),\n  FloatVector(Vec<f32>),\n}\n\n// A Feture has a name and a value\n// The name for now is 'id' of type string\n// Eventually this needs to be flexible - example to accomodate feature-id\nstruct Feature {\n  id: String,\n  val: FeatureVal,\n}\n\nimpl Feature {\n  fn new() -> Feature {\n    Feature {\n      id: String::new(),\n      val: FeatureVal::Empty\n    }\n  }\n}\n\n// A single inference record will have multiple features\nstruct Record {\n  fields: Vec<Feature>,\n}\n\nimpl Record {\n  fn new() -> Record {\n    Record { fields: vec![] }\n  }\n}\n\n// This is the main API used by external components\n// Given a serialized input, decode it into Records\nfn decode(input: Vec<u8>) -> Vec<Record> {\n  // For helping define the interface\n  vec![get_random_record(), get_random_record()]\n}\n\n// Used for testing the API, will be eventually removed\nfn get_random_record() -> Record {\n  let mut record: Record = Record::new();\n\n  let f1: Feature = Feature {\n    id: String::from(\"continuous_features\"),\n    val: FeatureVal::FloatVector(vec![1.0f32; 2134]),\n  };\n\n  record.fields.push(f1);\n\n  let f2: Feature = Feature {\n    id: String::from(\"user_embedding\"),\n    val: FeatureVal::FloatVector(vec![2.0f32; 200]),\n  };\n\n  record.fields.push(f2);\n\n  let f3: Feature = Feature {\n    id: String::from(\"author_embedding\"),\n    val: FeatureVal::FloatVector(vec![3.0f32; 200]),\n  };\n\n  record.fields.push(f3);\n\n  let f4: Feature = Feature {\n    id: String::from(\"binary_features\"),\n    val: FeatureVal::U8Vector(vec![4u8; 43]),\n  };\n\n  record.fields.push(f4);\n\n  record\n}\n\n"
  },
  {
    "path": "navi/thrift_bpr_adapter/thrift/src/lib.rs",
    "content": "pub mod prediction_service;\npub mod data;\npub mod tensor;\n\n"
  },
  {
    "path": "navi/thrift_bpr_adapter/thrift/src/main.rs",
    "content": "use std::collections::BTreeSet;\nuse std::collections::BTreeMap;\n\nuse bpr_thrift::data::DataRecord;\nuse bpr_thrift::prediction_service::BatchPredictionRequest;\nuse thrift::OrderedFloat;\n\nuse thrift::protocol::TBinaryInputProtocol;\nuse thrift::protocol::TSerializable;\nuse thrift::transport::TBufferChannel;\nuse thrift::Result;\n\nfn main() {\n  let data_path = \"/tmp/current/timelines/output-1\";\n  let bin_data: Vec<u8> = std::fs::read(data_path).expect(\"Could not read file!\"); \n\n  println!(\"Length : {}\", bin_data.len());\n\n  let mut bc = TBufferChannel::with_capacity(bin_data.len(), 0);\n\n  bc.set_readable_bytes(&bin_data);\n\n  let mut protocol = TBinaryInputProtocol::new(bc, true); \n\n  let result: Result<BatchPredictionRequest> =\n    BatchPredictionRequest::read_from_in_protocol(&mut protocol);\n\n  match result {\n    Ok(bpr) => logBP(bpr),\n    Err(err) => println!(\"Error {}\", err),\n  }\n}\n\nfn logBP(bpr: BatchPredictionRequest) {\n  println!(\"-------[OUTPUT]---------------\");\n  println!(\"data {:?}\", bpr);\n  println!(\"------------------------------\");\n\n  /* \n  let common = bpr.common_features;\n  let recs = bpr.individual_features_list;\n\n  println!(\"--------[Len : {}]------------------\", recs.len());\n\n  println!(\"-------[COMMON]---------------\");\n  match common {\n    Some(dr) => logDR(dr),\n    None => println!(\"None\"),\n  }\n  println!(\"------------------------------\");\n  for rec in recs {\n    logDR(rec);\n  }\n  println!(\"------------------------------\");\n  */\n}\n\nfn logDR(dr: DataRecord) {\n  println!(\"--------[DR]------------------\");\n\n  match dr.binary_features {\n    Some(bf) => logBin(bf),\n    _ => (),\n  }\n\n  match dr.continuous_features {\n    Some(cf) => logCF(cf),\n    _ => (),\n  }\n  println!(\"------------------------------\");\n}\n\nfn logBin(bin: BTreeSet<i64>) {\n  println!(\"B: {:?}\", bin)\n}\n\nfn logCF(cf: BTreeMap<i64, OrderedFloat<f64>>) {\n  for (id, fs) in cf {\n    println!(\"C: {} -> [{}]\", id, fs);\n  }\n}\n"
  },
  {
    "path": "navi/thrift_bpr_adapter/thrift/src/prediction_service.rs",
    "content": "// Autogenerated by Thrift Compiler (0.17.0)\n// DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING\n\n#![allow(unused_imports)]\n#![allow(unused_extern_crates)]\n#![allow(clippy::too_many_arguments, clippy::type_complexity, clippy::vec_box)]\n#![cfg_attr(rustfmt, rustfmt_skip)]\n\nuse std::cell::RefCell;\nuse std::collections::{BTreeMap, BTreeSet};\nuse std::convert::{From, TryFrom};\nuse std::default::Default;\nuse std::error::Error;\nuse std::fmt;\nuse std::fmt::{Display, Formatter};\nuse std::rc::Rc;\n\nuse thrift::OrderedFloat;\nuse thrift::{ApplicationError, ApplicationErrorKind, ProtocolError, ProtocolErrorKind, TThriftClient};\nuse thrift::protocol::{TFieldIdentifier, TListIdentifier, TMapIdentifier, TMessageIdentifier, TMessageType, TInputProtocol, TOutputProtocol, TSerializable, TSetIdentifier, TStructIdentifier, TType};\nuse thrift::protocol::field_id;\nuse thrift::protocol::verify_expected_message_type;\nuse thrift::protocol::verify_expected_sequence_number;\nuse thrift::protocol::verify_expected_service_call;\nuse thrift::protocol::verify_required_field_exists;\nuse thrift::server::TProcessor;\n\nuse crate::data;\n\n//\n// PredictionServiceException\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct PredictionServiceException {\n  pub description: Option<String>,\n}\n\nimpl PredictionServiceException {\n  pub fn new<F1>(description: F1) -> PredictionServiceException where F1: Into<Option<String>> {\n    PredictionServiceException {\n      description: description.into(),\n    }\n  }\n}\n\nimpl TSerializable for PredictionServiceException {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<PredictionServiceException> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<String> = Some(\"\".to_owned());\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let val = i_prot.read_string()?;\n          f_1 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    let ret = PredictionServiceException {\n      description: f_1,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"PredictionServiceException\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    if let Some(ref fld_var) = self.description {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"description\", TType::String, 1))?;\n      o_prot.write_string(fld_var)?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\nimpl Default for PredictionServiceException {\n  fn default() -> Self {\n    PredictionServiceException{\n      description: Some(\"\".to_owned()),\n    }\n  }\n}\n\nimpl Error for PredictionServiceException {}\n\nimpl From<PredictionServiceException> for thrift::Error {\n  fn from(e: PredictionServiceException) -> Self {\n    thrift::Error::User(Box::new(e))\n  }\n}\n\nimpl Display for PredictionServiceException {\n  fn fmt(&self, f: &mut Formatter) -> fmt::Result {\n    write!(f, \"remote service threw PredictionServiceException\")\n  }\n}\n\n//\n// PredictionRequest\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct PredictionRequest {\n  pub features: data::DataRecord,\n}\n\nimpl PredictionRequest {\n  pub fn new(features: data::DataRecord) -> PredictionRequest {\n    PredictionRequest {\n      features,\n    }\n  }\n}\n\nimpl TSerializable for PredictionRequest {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<PredictionRequest> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<data::DataRecord> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let val = data::DataRecord::read_from_in_protocol(i_prot)?;\n          f_1 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"PredictionRequest.features\", &f_1)?;\n    let ret = PredictionRequest {\n      features: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"PredictionRequest\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"features\", TType::Struct, 1))?;\n    self.features.write_to_out_protocol(o_prot)?;\n    o_prot.write_field_end()?;\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// PredictionResponse\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct PredictionResponse {\n  pub prediction: data::DataRecord,\n}\n\nimpl PredictionResponse {\n  pub fn new(prediction: data::DataRecord) -> PredictionResponse {\n    PredictionResponse {\n      prediction,\n    }\n  }\n}\n\nimpl TSerializable for PredictionResponse {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<PredictionResponse> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<data::DataRecord> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let val = data::DataRecord::read_from_in_protocol(i_prot)?;\n          f_1 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"PredictionResponse.prediction\", &f_1)?;\n    let ret = PredictionResponse {\n      prediction: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"PredictionResponse\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"prediction\", TType::Struct, 1))?;\n    self.prediction.write_to_out_protocol(o_prot)?;\n    o_prot.write_field_end()?;\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// BatchPredictionRequest\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct BatchPredictionRequest {\n  pub individual_features_list: Vec<data::DataRecord>,\n  pub common_features: Option<data::DataRecord>,\n}\n\nimpl BatchPredictionRequest {\n  pub fn new<F2>(individual_features_list: Vec<data::DataRecord>, common_features: F2) -> BatchPredictionRequest where F2: Into<Option<data::DataRecord>> {\n    BatchPredictionRequest {\n      individual_features_list,\n      common_features: common_features.into(),\n    }\n  }\n}\n\nimpl TSerializable for BatchPredictionRequest {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<BatchPredictionRequest> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<Vec<data::DataRecord>> = None;\n    let mut f_2: Option<data::DataRecord> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<data::DataRecord> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_0 = data::DataRecord::read_from_in_protocol(i_prot)?;\n            val.push(list_elem_0);\n          }\n          i_prot.read_list_end()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let val = data::DataRecord::read_from_in_protocol(i_prot)?;\n          f_2 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"BatchPredictionRequest.individual_features_list\", &f_1)?;\n    let ret = BatchPredictionRequest {\n      individual_features_list: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n      common_features: f_2,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"BatchPredictionRequest\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"individualFeaturesList\", TType::List, 1))?;\n    o_prot.write_list_begin(&TListIdentifier::new(TType::Struct, self.individual_features_list.len() as i32))?;\n    for e in &self.individual_features_list {\n      e.write_to_out_protocol(o_prot)?;\n    }\n    o_prot.write_list_end()?;\n    o_prot.write_field_end()?;\n    if let Some(ref fld_var) = self.common_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"commonFeatures\", TType::Struct, 2))?;\n      fld_var.write_to_out_protocol(o_prot)?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// BatchPredictionResponse\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct BatchPredictionResponse {\n  pub predictions: Vec<data::DataRecord>,\n}\n\nimpl BatchPredictionResponse {\n  pub fn new(predictions: Vec<data::DataRecord>) -> BatchPredictionResponse {\n    BatchPredictionResponse {\n      predictions,\n    }\n  }\n}\n\nimpl TSerializable for BatchPredictionResponse {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<BatchPredictionResponse> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<Vec<data::DataRecord>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<data::DataRecord> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_1 = data::DataRecord::read_from_in_protocol(i_prot)?;\n            val.push(list_elem_1);\n          }\n          i_prot.read_list_end()?;\n          f_1 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"BatchPredictionResponse.predictions\", &f_1)?;\n    let ret = BatchPredictionResponse {\n      predictions: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"BatchPredictionResponse\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"predictions\", TType::List, 1))?;\n    o_prot.write_list_begin(&TListIdentifier::new(TType::Struct, self.predictions.len() as i32))?;\n    for e in &self.predictions {\n      e.write_to_out_protocol(o_prot)?;\n    }\n    o_prot.write_list_end()?;\n    o_prot.write_field_end()?;\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// DataRecordPair\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct DataRecordPair {\n  pub first: Option<data::DataRecord>,\n  pub second: Option<data::DataRecord>,\n}\n\nimpl DataRecordPair {\n  pub fn new<F1, F2>(first: F1, second: F2) -> DataRecordPair where F1: Into<Option<data::DataRecord>>, F2: Into<Option<data::DataRecord>> {\n    DataRecordPair {\n      first: first.into(),\n      second: second.into(),\n    }\n  }\n}\n\nimpl TSerializable for DataRecordPair {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<DataRecordPair> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<data::DataRecord> = None;\n    let mut f_2: Option<data::DataRecord> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let val = data::DataRecord::read_from_in_protocol(i_prot)?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let val = data::DataRecord::read_from_in_protocol(i_prot)?;\n          f_2 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    let ret = DataRecordPair {\n      first: f_1,\n      second: f_2,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"DataRecordPair\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    if let Some(ref fld_var) = self.first {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"first\", TType::Struct, 1))?;\n      fld_var.write_to_out_protocol(o_prot)?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.second {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"second\", TType::Struct, 2))?;\n      fld_var.write_to_out_protocol(o_prot)?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\nimpl Default for DataRecordPair {\n  fn default() -> Self {\n    DataRecordPair{\n      first: None,\n      second: None,\n    }\n  }\n}\n\n//\n// PredictionTrainingExample\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct PredictionTrainingExample {\n  pub features: Option<data::DataRecord>,\n  pub features_for_pairwise_learning: Option<DataRecordPair>,\n  pub compact_features: Option<data::CompactDataRecord>,\n  pub compressed_data_record: Option<Vec<u8>>,\n}\n\nimpl PredictionTrainingExample {\n  pub fn new<F1, F2, F3, F4>(features: F1, features_for_pairwise_learning: F2, compact_features: F3, compressed_data_record: F4) -> PredictionTrainingExample where F1: Into<Option<data::DataRecord>>, F2: Into<Option<DataRecordPair>>, F3: Into<Option<data::CompactDataRecord>>, F4: Into<Option<Vec<u8>>> {\n    PredictionTrainingExample {\n      features: features.into(),\n      features_for_pairwise_learning: features_for_pairwise_learning.into(),\n      compact_features: compact_features.into(),\n      compressed_data_record: compressed_data_record.into(),\n    }\n  }\n}\n\nimpl TSerializable for PredictionTrainingExample {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<PredictionTrainingExample> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<data::DataRecord> = None;\n    let mut f_2: Option<DataRecordPair> = None;\n    let mut f_3: Option<data::CompactDataRecord> = None;\n    let mut f_4: Option<Vec<u8>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let val = data::DataRecord::read_from_in_protocol(i_prot)?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let val = DataRecordPair::read_from_in_protocol(i_prot)?;\n          f_2 = Some(val);\n        },\n        3 => {\n          let val = data::CompactDataRecord::read_from_in_protocol(i_prot)?;\n          f_3 = Some(val);\n        },\n        4 => {\n          let val = i_prot.read_bytes()?;\n          f_4 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    let ret = PredictionTrainingExample {\n      features: f_1,\n      features_for_pairwise_learning: f_2,\n      compact_features: f_3,\n      compressed_data_record: f_4,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"PredictionTrainingExample\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    if let Some(ref fld_var) = self.features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"features\", TType::Struct, 1))?;\n      fld_var.write_to_out_protocol(o_prot)?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.features_for_pairwise_learning {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"featuresForPairwiseLearning\", TType::Struct, 2))?;\n      fld_var.write_to_out_protocol(o_prot)?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.compact_features {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"compactFeatures\", TType::Struct, 3))?;\n      fld_var.write_to_out_protocol(o_prot)?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.compressed_data_record {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"compressedDataRecord\", TType::String, 4))?;\n      o_prot.write_bytes(fld_var)?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\nimpl Default for PredictionTrainingExample {\n  fn default() -> Self {\n    PredictionTrainingExample{\n      features: None,\n      features_for_pairwise_learning: None,\n      compact_features: None,\n      compressed_data_record: Some(Vec::new()),\n    }\n  }\n}\n\n//\n// PredictionService service client\n//\n\npub trait TPredictionServiceSyncClient {\n  fn get_prediction(&mut self, request: PredictionRequest) -> thrift::Result<PredictionResponse>;\n  fn get_batch_prediction(&mut self, batch_request: BatchPredictionRequest) -> thrift::Result<BatchPredictionResponse>;\n}\n\npub trait TPredictionServiceSyncClientMarker {}\n\npub struct PredictionServiceSyncClient<IP, OP> where IP: TInputProtocol, OP: TOutputProtocol {\n  _i_prot: IP,\n  _o_prot: OP,\n  _sequence_number: i32,\n}\n\nimpl <IP, OP> PredictionServiceSyncClient<IP, OP> where IP: TInputProtocol, OP: TOutputProtocol {\n  pub fn new(input_protocol: IP, output_protocol: OP) -> PredictionServiceSyncClient<IP, OP> {\n    PredictionServiceSyncClient { _i_prot: input_protocol, _o_prot: output_protocol, _sequence_number: 0 }\n  }\n}\n\nimpl <IP, OP> TThriftClient for PredictionServiceSyncClient<IP, OP> where IP: TInputProtocol, OP: TOutputProtocol {\n  fn i_prot_mut(&mut self) -> &mut dyn TInputProtocol { &mut self._i_prot }\n  fn o_prot_mut(&mut self) -> &mut dyn TOutputProtocol { &mut self._o_prot }\n  fn sequence_number(&self) -> i32 { self._sequence_number }\n  fn increment_sequence_number(&mut self) -> i32 { self._sequence_number += 1; self._sequence_number }\n}\n\nimpl <IP, OP> TPredictionServiceSyncClientMarker for PredictionServiceSyncClient<IP, OP> where IP: TInputProtocol, OP: TOutputProtocol {}\n\nimpl <C: TThriftClient + TPredictionServiceSyncClientMarker> TPredictionServiceSyncClient for C {\n  fn get_prediction(&mut self, request: PredictionRequest) -> thrift::Result<PredictionResponse> {\n    (\n      {\n        self.increment_sequence_number();\n        let message_ident = TMessageIdentifier::new(\"getPrediction\", TMessageType::Call, self.sequence_number());\n        let call_args = PredictionServiceGetPredictionArgs { request };\n        self.o_prot_mut().write_message_begin(&message_ident)?;\n        call_args.write_to_out_protocol(self.o_prot_mut())?;\n        self.o_prot_mut().write_message_end()?;\n        self.o_prot_mut().flush()\n      }\n    )?;\n    {\n      let message_ident = self.i_prot_mut().read_message_begin()?;\n      verify_expected_sequence_number(self.sequence_number(), message_ident.sequence_number)?;\n      verify_expected_service_call(\"getPrediction\", &message_ident.name)?;\n      if message_ident.message_type == TMessageType::Exception {\n        let remote_error = thrift::Error::read_application_error_from_in_protocol(self.i_prot_mut())?;\n        self.i_prot_mut().read_message_end()?;\n        return Err(thrift::Error::Application(remote_error))\n      }\n      verify_expected_message_type(TMessageType::Reply, message_ident.message_type)?;\n      let result = PredictionServiceGetPredictionResult::read_from_in_protocol(self.i_prot_mut())?;\n      self.i_prot_mut().read_message_end()?;\n      result.ok_or()\n    }\n  }\n  fn get_batch_prediction(&mut self, batch_request: BatchPredictionRequest) -> thrift::Result<BatchPredictionResponse> {\n    (\n      {\n        self.increment_sequence_number();\n        let message_ident = TMessageIdentifier::new(\"getBatchPrediction\", TMessageType::Call, self.sequence_number());\n        let call_args = PredictionServiceGetBatchPredictionArgs { batch_request };\n        self.o_prot_mut().write_message_begin(&message_ident)?;\n        call_args.write_to_out_protocol(self.o_prot_mut())?;\n        self.o_prot_mut().write_message_end()?;\n        self.o_prot_mut().flush()\n      }\n    )?;\n    {\n      let message_ident = self.i_prot_mut().read_message_begin()?;\n      verify_expected_sequence_number(self.sequence_number(), message_ident.sequence_number)?;\n      verify_expected_service_call(\"getBatchPrediction\", &message_ident.name)?;\n      if message_ident.message_type == TMessageType::Exception {\n        let remote_error = thrift::Error::read_application_error_from_in_protocol(self.i_prot_mut())?;\n        self.i_prot_mut().read_message_end()?;\n        return Err(thrift::Error::Application(remote_error))\n      }\n      verify_expected_message_type(TMessageType::Reply, message_ident.message_type)?;\n      let result = PredictionServiceGetBatchPredictionResult::read_from_in_protocol(self.i_prot_mut())?;\n      self.i_prot_mut().read_message_end()?;\n      result.ok_or()\n    }\n  }\n}\n\n//\n// PredictionService service processor\n//\n\npub trait PredictionServiceSyncHandler {\n  fn handle_get_prediction(&self, request: PredictionRequest) -> thrift::Result<PredictionResponse>;\n  fn handle_get_batch_prediction(&self, batch_request: BatchPredictionRequest) -> thrift::Result<BatchPredictionResponse>;\n}\n\npub struct PredictionServiceSyncProcessor<H: PredictionServiceSyncHandler> {\n  handler: H,\n}\n\nimpl <H: PredictionServiceSyncHandler> PredictionServiceSyncProcessor<H> {\n  pub fn new(handler: H) -> PredictionServiceSyncProcessor<H> {\n    PredictionServiceSyncProcessor {\n      handler,\n    }\n  }\n  fn process_get_prediction(&self, incoming_sequence_number: i32, i_prot: &mut dyn TInputProtocol, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    TPredictionServiceProcessFunctions::process_get_prediction(&self.handler, incoming_sequence_number, i_prot, o_prot)\n  }\n  fn process_get_batch_prediction(&self, incoming_sequence_number: i32, i_prot: &mut dyn TInputProtocol, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    TPredictionServiceProcessFunctions::process_get_batch_prediction(&self.handler, incoming_sequence_number, i_prot, o_prot)\n  }\n}\n\npub struct TPredictionServiceProcessFunctions;\n\nimpl TPredictionServiceProcessFunctions {\n  pub fn process_get_prediction<H: PredictionServiceSyncHandler>(handler: &H, incoming_sequence_number: i32, i_prot: &mut dyn TInputProtocol, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let args = PredictionServiceGetPredictionArgs::read_from_in_protocol(i_prot)?;\n    match handler.handle_get_prediction(args.request) {\n      Ok(handler_return) => {\n        let message_ident = TMessageIdentifier::new(\"getPrediction\", TMessageType::Reply, incoming_sequence_number);\n        o_prot.write_message_begin(&message_ident)?;\n        let ret = PredictionServiceGetPredictionResult { result_value: Some(handler_return), prediction_service_exception: None };\n        ret.write_to_out_protocol(o_prot)?;\n        o_prot.write_message_end()?;\n        o_prot.flush()\n      },\n      Err(e) => {\n        match e {\n          thrift::Error::User(usr_err) => {\n            if usr_err.downcast_ref::<PredictionServiceException>().is_some() {\n              let err = usr_err.downcast::<PredictionServiceException>().expect(\"downcast already checked\");\n              let ret_err = PredictionServiceGetPredictionResult{ result_value: None, prediction_service_exception: Some(*err) };\n              let message_ident = TMessageIdentifier::new(\"getPrediction\", TMessageType::Reply, incoming_sequence_number);\n              o_prot.write_message_begin(&message_ident)?;\n              ret_err.write_to_out_protocol(o_prot)?;\n              o_prot.write_message_end()?;\n              o_prot.flush()\n            } else {\n              let ret_err = {\n                ApplicationError::new(\n                  ApplicationErrorKind::Unknown,\n                  usr_err.to_string()\n                )\n              };\n              let message_ident = TMessageIdentifier::new(\"getPrediction\", TMessageType::Exception, incoming_sequence_number);\n              o_prot.write_message_begin(&message_ident)?;\n              thrift::Error::write_application_error_to_out_protocol(&ret_err, o_prot)?;\n              o_prot.write_message_end()?;\n              o_prot.flush()\n            }\n          },\n          thrift::Error::Application(app_err) => {\n            let message_ident = TMessageIdentifier::new(\"getPrediction\", TMessageType::Exception, incoming_sequence_number);\n            o_prot.write_message_begin(&message_ident)?;\n            thrift::Error::write_application_error_to_out_protocol(&app_err, o_prot)?;\n            o_prot.write_message_end()?;\n            o_prot.flush()\n          },\n          _ => {\n            let ret_err = {\n              ApplicationError::new(\n                ApplicationErrorKind::Unknown,\n                e.to_string()\n              )\n            };\n            let message_ident = TMessageIdentifier::new(\"getPrediction\", TMessageType::Exception, incoming_sequence_number);\n            o_prot.write_message_begin(&message_ident)?;\n            thrift::Error::write_application_error_to_out_protocol(&ret_err, o_prot)?;\n            o_prot.write_message_end()?;\n            o_prot.flush()\n          },\n        }\n      },\n    }\n  }\n  pub fn process_get_batch_prediction<H: PredictionServiceSyncHandler>(handler: &H, incoming_sequence_number: i32, i_prot: &mut dyn TInputProtocol, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let args = PredictionServiceGetBatchPredictionArgs::read_from_in_protocol(i_prot)?;\n    match handler.handle_get_batch_prediction(args.batch_request) {\n      Ok(handler_return) => {\n        let message_ident = TMessageIdentifier::new(\"getBatchPrediction\", TMessageType::Reply, incoming_sequence_number);\n        o_prot.write_message_begin(&message_ident)?;\n        let ret = PredictionServiceGetBatchPredictionResult { result_value: Some(handler_return), prediction_service_exception: None };\n        ret.write_to_out_protocol(o_prot)?;\n        o_prot.write_message_end()?;\n        o_prot.flush()\n      },\n      Err(e) => {\n        match e {\n          thrift::Error::User(usr_err) => {\n            if usr_err.downcast_ref::<PredictionServiceException>().is_some() {\n              let err = usr_err.downcast::<PredictionServiceException>().expect(\"downcast already checked\");\n              let ret_err = PredictionServiceGetBatchPredictionResult{ result_value: None, prediction_service_exception: Some(*err) };\n              let message_ident = TMessageIdentifier::new(\"getBatchPrediction\", TMessageType::Reply, incoming_sequence_number);\n              o_prot.write_message_begin(&message_ident)?;\n              ret_err.write_to_out_protocol(o_prot)?;\n              o_prot.write_message_end()?;\n              o_prot.flush()\n            } else {\n              let ret_err = {\n                ApplicationError::new(\n                  ApplicationErrorKind::Unknown,\n                  usr_err.to_string()\n                )\n              };\n              let message_ident = TMessageIdentifier::new(\"getBatchPrediction\", TMessageType::Exception, incoming_sequence_number);\n              o_prot.write_message_begin(&message_ident)?;\n              thrift::Error::write_application_error_to_out_protocol(&ret_err, o_prot)?;\n              o_prot.write_message_end()?;\n              o_prot.flush()\n            }\n          },\n          thrift::Error::Application(app_err) => {\n            let message_ident = TMessageIdentifier::new(\"getBatchPrediction\", TMessageType::Exception, incoming_sequence_number);\n            o_prot.write_message_begin(&message_ident)?;\n            thrift::Error::write_application_error_to_out_protocol(&app_err, o_prot)?;\n            o_prot.write_message_end()?;\n            o_prot.flush()\n          },\n          _ => {\n            let ret_err = {\n              ApplicationError::new(\n                ApplicationErrorKind::Unknown,\n                e.to_string()\n              )\n            };\n            let message_ident = TMessageIdentifier::new(\"getBatchPrediction\", TMessageType::Exception, incoming_sequence_number);\n            o_prot.write_message_begin(&message_ident)?;\n            thrift::Error::write_application_error_to_out_protocol(&ret_err, o_prot)?;\n            o_prot.write_message_end()?;\n            o_prot.flush()\n          },\n        }\n      },\n    }\n  }\n}\n\nimpl <H: PredictionServiceSyncHandler> TProcessor for PredictionServiceSyncProcessor<H> {\n  fn process(&self, i_prot: &mut dyn TInputProtocol, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let message_ident = i_prot.read_message_begin()?;\n    let res = match &*message_ident.name {\n      \"getPrediction\" => {\n        self.process_get_prediction(message_ident.sequence_number, i_prot, o_prot)\n      },\n      \"getBatchPrediction\" => {\n        self.process_get_batch_prediction(message_ident.sequence_number, i_prot, o_prot)\n      },\n      method => {\n        Err(\n          thrift::Error::Application(\n            ApplicationError::new(\n              ApplicationErrorKind::UnknownMethod,\n              format!(\"unknown method {}\", method)\n            )\n          )\n        )\n      },\n    };\n    thrift::server::handle_process_result(&message_ident, res, o_prot)\n  }\n}\n\n//\n// PredictionServiceGetPredictionArgs\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\nstruct PredictionServiceGetPredictionArgs {\n  request: PredictionRequest,\n}\n\nimpl PredictionServiceGetPredictionArgs {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<PredictionServiceGetPredictionArgs> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<PredictionRequest> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let val = PredictionRequest::read_from_in_protocol(i_prot)?;\n          f_1 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"PredictionServiceGetPredictionArgs.request\", &f_1)?;\n    let ret = PredictionServiceGetPredictionArgs {\n      request: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"getPrediction_args\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"request\", TType::Struct, 1))?;\n    self.request.write_to_out_protocol(o_prot)?;\n    o_prot.write_field_end()?;\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// PredictionServiceGetPredictionResult\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\nstruct PredictionServiceGetPredictionResult {\n  result_value: Option<PredictionResponse>,\n  prediction_service_exception: Option<PredictionServiceException>,\n}\n\nimpl PredictionServiceGetPredictionResult {\n  fn ok_or(self) -> thrift::Result<PredictionResponse> {\n    if self.prediction_service_exception.is_some() {\n      Err(thrift::Error::User(Box::new(self.prediction_service_exception.unwrap())))\n    } else if self.result_value.is_some() {\n      Ok(self.result_value.unwrap())\n    } else {\n      Err(\n        thrift::Error::Application(\n          ApplicationError::new(\n            ApplicationErrorKind::MissingResult,\n            \"no result received for PredictionServiceGetPrediction\"\n          )\n        )\n      )\n    }\n  }\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<PredictionServiceGetPredictionResult> {\n    i_prot.read_struct_begin()?;\n    let mut f_0: Option<PredictionResponse> = None;\n    let mut f_1: Option<PredictionServiceException> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        0 => {\n          let val = PredictionResponse::read_from_in_protocol(i_prot)?;\n          f_0 = Some(val);\n        },\n        1 => {\n          let val = PredictionServiceException::read_from_in_protocol(i_prot)?;\n          f_1 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    let ret = PredictionServiceGetPredictionResult {\n      result_value: f_0,\n      prediction_service_exception: f_1,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"PredictionServiceGetPredictionResult\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    if let Some(ref fld_var) = self.result_value {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"result_value\", TType::Struct, 0))?;\n      fld_var.write_to_out_protocol(o_prot)?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.prediction_service_exception {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"predictionServiceException\", TType::Struct, 1))?;\n      fld_var.write_to_out_protocol(o_prot)?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// PredictionServiceGetBatchPredictionArgs\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\nstruct PredictionServiceGetBatchPredictionArgs {\n  batch_request: BatchPredictionRequest,\n}\n\nimpl PredictionServiceGetBatchPredictionArgs {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<PredictionServiceGetBatchPredictionArgs> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<BatchPredictionRequest> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let val = BatchPredictionRequest::read_from_in_protocol(i_prot)?;\n          f_1 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"PredictionServiceGetBatchPredictionArgs.batch_request\", &f_1)?;\n    let ret = PredictionServiceGetBatchPredictionArgs {\n      batch_request: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"getBatchPrediction_args\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"batchRequest\", TType::Struct, 1))?;\n    self.batch_request.write_to_out_protocol(o_prot)?;\n    o_prot.write_field_end()?;\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// PredictionServiceGetBatchPredictionResult\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\nstruct PredictionServiceGetBatchPredictionResult {\n  result_value: Option<BatchPredictionResponse>,\n  prediction_service_exception: Option<PredictionServiceException>,\n}\n\nimpl PredictionServiceGetBatchPredictionResult {\n  fn ok_or(self) -> thrift::Result<BatchPredictionResponse> {\n    if self.prediction_service_exception.is_some() {\n      Err(thrift::Error::User(Box::new(self.prediction_service_exception.unwrap())))\n    } else if self.result_value.is_some() {\n      Ok(self.result_value.unwrap())\n    } else {\n      Err(\n        thrift::Error::Application(\n          ApplicationError::new(\n            ApplicationErrorKind::MissingResult,\n            \"no result received for PredictionServiceGetBatchPrediction\"\n          )\n        )\n      )\n    }\n  }\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<PredictionServiceGetBatchPredictionResult> {\n    i_prot.read_struct_begin()?;\n    let mut f_0: Option<BatchPredictionResponse> = None;\n    let mut f_1: Option<PredictionServiceException> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        0 => {\n          let val = BatchPredictionResponse::read_from_in_protocol(i_prot)?;\n          f_0 = Some(val);\n        },\n        1 => {\n          let val = PredictionServiceException::read_from_in_protocol(i_prot)?;\n          f_1 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    let ret = PredictionServiceGetBatchPredictionResult {\n      result_value: f_0,\n      prediction_service_exception: f_1,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"PredictionServiceGetBatchPredictionResult\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    if let Some(ref fld_var) = self.result_value {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"result_value\", TType::Struct, 0))?;\n      fld_var.write_to_out_protocol(o_prot)?;\n      o_prot.write_field_end()?\n    }\n    if let Some(ref fld_var) = self.prediction_service_exception {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"predictionServiceException\", TType::Struct, 1))?;\n      fld_var.write_to_out_protocol(o_prot)?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n"
  },
  {
    "path": "navi/thrift_bpr_adapter/thrift/src/tensor.rs",
    "content": "// Autogenerated by Thrift Compiler (0.17.0)\n// DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING\n\n#![allow(unused_imports)]\n#![allow(unused_extern_crates)]\n#![allow(clippy::too_many_arguments, clippy::type_complexity, clippy::vec_box)]\n#![cfg_attr(rustfmt, rustfmt_skip)]\n\nuse std::cell::RefCell;\nuse std::collections::{BTreeMap, BTreeSet};\nuse std::convert::{From, TryFrom};\nuse std::default::Default;\nuse std::error::Error;\nuse std::fmt;\nuse std::fmt::{Display, Formatter};\nuse std::rc::Rc;\n\nuse thrift::OrderedFloat;\nuse thrift::{ApplicationError, ApplicationErrorKind, ProtocolError, ProtocolErrorKind, TThriftClient};\nuse thrift::protocol::{TFieldIdentifier, TListIdentifier, TMapIdentifier, TMessageIdentifier, TMessageType, TInputProtocol, TOutputProtocol, TSerializable, TSetIdentifier, TStructIdentifier, TType};\nuse thrift::protocol::field_id;\nuse thrift::protocol::verify_expected_message_type;\nuse thrift::protocol::verify_expected_sequence_number;\nuse thrift::protocol::verify_expected_service_call;\nuse thrift::protocol::verify_required_field_exists;\nuse thrift::server::TProcessor;\n\n#[derive(Copy, Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct DataType(pub i32);\n\nimpl DataType {\n  pub const FLOAT: DataType = DataType(0);\n  pub const DOUBLE: DataType = DataType(1);\n  pub const INT32: DataType = DataType(2);\n  pub const INT64: DataType = DataType(3);\n  pub const UINT8: DataType = DataType(4);\n  pub const STRING: DataType = DataType(5);\n  pub const BYTE: DataType = DataType(6);\n  pub const BOOL: DataType = DataType(7);\n  pub const RESERVED_1: DataType = DataType(8);\n  pub const RESERVED_2: DataType = DataType(9);\n  pub const RESERVED_3: DataType = DataType(10);\n  pub const ENUM_VALUES: &'static [Self] = &[\n    Self::FLOAT,\n    Self::DOUBLE,\n    Self::INT32,\n    Self::INT64,\n    Self::UINT8,\n    Self::STRING,\n    Self::BYTE,\n    Self::BOOL,\n    Self::RESERVED_1,\n    Self::RESERVED_2,\n    Self::RESERVED_3,\n  ];\n}\n\nimpl TSerializable for DataType {\n  #[allow(clippy::trivially_copy_pass_by_ref)]\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    o_prot.write_i32(self.0)\n  }\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<DataType> {\n    let enum_value = i_prot.read_i32()?;\n    Ok(DataType::from(enum_value))\n  }\n}\n\nimpl From<i32> for DataType {\n  fn from(i: i32) -> Self {\n    match i {\n      0 => DataType::FLOAT,\n      1 => DataType::DOUBLE,\n      2 => DataType::INT32,\n      3 => DataType::INT64,\n      4 => DataType::UINT8,\n      5 => DataType::STRING,\n      6 => DataType::BYTE,\n      7 => DataType::BOOL,\n      8 => DataType::RESERVED_1,\n      9 => DataType::RESERVED_2,\n      10 => DataType::RESERVED_3,\n      _ => DataType(i)\n    }\n  }\n}\n\nimpl From<&i32> for DataType {\n  fn from(i: &i32) -> Self {\n    DataType::from(*i)\n  }\n}\n\nimpl From<DataType> for i32 {\n  fn from(e: DataType) -> i32 {\n    e.0\n  }\n}\n\nimpl From<&DataType> for i32 {\n  fn from(e: &DataType) -> i32 {\n    e.0\n  }\n}\n\n//\n// StringTensor\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct StringTensor {\n  pub strings: Vec<String>,\n  pub shape: Option<Vec<i64>>,\n}\n\nimpl StringTensor {\n  pub fn new<F2>(strings: Vec<String>, shape: F2) -> StringTensor where F2: Into<Option<Vec<i64>>> {\n    StringTensor {\n      strings,\n      shape: shape.into(),\n    }\n  }\n}\n\nimpl TSerializable for StringTensor {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<StringTensor> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<Vec<String>> = None;\n    let mut f_2: Option<Vec<i64>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<String> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_0 = i_prot.read_string()?;\n            val.push(list_elem_0);\n          }\n          i_prot.read_list_end()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<i64> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_1 = i_prot.read_i64()?;\n            val.push(list_elem_1);\n          }\n          i_prot.read_list_end()?;\n          f_2 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"StringTensor.strings\", &f_1)?;\n    let ret = StringTensor {\n      strings: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n      shape: f_2,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"StringTensor\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"strings\", TType::List, 1))?;\n    o_prot.write_list_begin(&TListIdentifier::new(TType::String, self.strings.len() as i32))?;\n    for e in &self.strings {\n      o_prot.write_string(e)?;\n    }\n    o_prot.write_list_end()?;\n    o_prot.write_field_end()?;\n    if let Some(ref fld_var) = self.shape {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"shape\", TType::List, 2))?;\n      o_prot.write_list_begin(&TListIdentifier::new(TType::I64, fld_var.len() as i32))?;\n      for e in fld_var {\n        o_prot.write_i64(*e)?;\n      }\n      o_prot.write_list_end()?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// Int32Tensor\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct Int32Tensor {\n  pub ints: Vec<i32>,\n  pub shape: Option<Vec<i64>>,\n}\n\nimpl Int32Tensor {\n  pub fn new<F2>(ints: Vec<i32>, shape: F2) -> Int32Tensor where F2: Into<Option<Vec<i64>>> {\n    Int32Tensor {\n      ints,\n      shape: shape.into(),\n    }\n  }\n}\n\nimpl TSerializable for Int32Tensor {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<Int32Tensor> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<Vec<i32>> = None;\n    let mut f_2: Option<Vec<i64>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<i32> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_2 = i_prot.read_i32()?;\n            val.push(list_elem_2);\n          }\n          i_prot.read_list_end()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<i64> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_3 = i_prot.read_i64()?;\n            val.push(list_elem_3);\n          }\n          i_prot.read_list_end()?;\n          f_2 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"Int32Tensor.ints\", &f_1)?;\n    let ret = Int32Tensor {\n      ints: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n      shape: f_2,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"Int32Tensor\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"ints\", TType::List, 1))?;\n    o_prot.write_list_begin(&TListIdentifier::new(TType::I32, self.ints.len() as i32))?;\n    for e in &self.ints {\n      o_prot.write_i32(*e)?;\n    }\n    o_prot.write_list_end()?;\n    o_prot.write_field_end()?;\n    if let Some(ref fld_var) = self.shape {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"shape\", TType::List, 2))?;\n      o_prot.write_list_begin(&TListIdentifier::new(TType::I64, fld_var.len() as i32))?;\n      for e in fld_var {\n        o_prot.write_i64(*e)?;\n      }\n      o_prot.write_list_end()?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// Int64Tensor\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct Int64Tensor {\n  pub longs: Vec<i64>,\n  pub shape: Option<Vec<i64>>,\n}\n\nimpl Int64Tensor {\n  pub fn new<F2>(longs: Vec<i64>, shape: F2) -> Int64Tensor where F2: Into<Option<Vec<i64>>> {\n    Int64Tensor {\n      longs,\n      shape: shape.into(),\n    }\n  }\n}\n\nimpl TSerializable for Int64Tensor {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<Int64Tensor> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<Vec<i64>> = None;\n    let mut f_2: Option<Vec<i64>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<i64> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_4 = i_prot.read_i64()?;\n            val.push(list_elem_4);\n          }\n          i_prot.read_list_end()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<i64> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_5 = i_prot.read_i64()?;\n            val.push(list_elem_5);\n          }\n          i_prot.read_list_end()?;\n          f_2 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"Int64Tensor.longs\", &f_1)?;\n    let ret = Int64Tensor {\n      longs: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n      shape: f_2,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"Int64Tensor\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"longs\", TType::List, 1))?;\n    o_prot.write_list_begin(&TListIdentifier::new(TType::I64, self.longs.len() as i32))?;\n    for e in &self.longs {\n      o_prot.write_i64(*e)?;\n    }\n    o_prot.write_list_end()?;\n    o_prot.write_field_end()?;\n    if let Some(ref fld_var) = self.shape {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"shape\", TType::List, 2))?;\n      o_prot.write_list_begin(&TListIdentifier::new(TType::I64, fld_var.len() as i32))?;\n      for e in fld_var {\n        o_prot.write_i64(*e)?;\n      }\n      o_prot.write_list_end()?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// FloatTensor\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct FloatTensor {\n  pub floats: Vec<OrderedFloat<f64>>,\n  pub shape: Option<Vec<i64>>,\n}\n\nimpl FloatTensor {\n  pub fn new<F2>(floats: Vec<OrderedFloat<f64>>, shape: F2) -> FloatTensor where F2: Into<Option<Vec<i64>>> {\n    FloatTensor {\n      floats,\n      shape: shape.into(),\n    }\n  }\n}\n\nimpl TSerializable for FloatTensor {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<FloatTensor> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<Vec<OrderedFloat<f64>>> = None;\n    let mut f_2: Option<Vec<i64>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<OrderedFloat<f64>> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_6 = OrderedFloat::from(i_prot.read_double()?);\n            val.push(list_elem_6);\n          }\n          i_prot.read_list_end()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<i64> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_7 = i_prot.read_i64()?;\n            val.push(list_elem_7);\n          }\n          i_prot.read_list_end()?;\n          f_2 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"FloatTensor.floats\", &f_1)?;\n    let ret = FloatTensor {\n      floats: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n      shape: f_2,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"FloatTensor\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"floats\", TType::List, 1))?;\n    o_prot.write_list_begin(&TListIdentifier::new(TType::Double, self.floats.len() as i32))?;\n    for e in &self.floats {\n      o_prot.write_double((*e).into())?;\n    }\n    o_prot.write_list_end()?;\n    o_prot.write_field_end()?;\n    if let Some(ref fld_var) = self.shape {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"shape\", TType::List, 2))?;\n      o_prot.write_list_begin(&TListIdentifier::new(TType::I64, fld_var.len() as i32))?;\n      for e in fld_var {\n        o_prot.write_i64(*e)?;\n      }\n      o_prot.write_list_end()?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// DoubleTensor\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct DoubleTensor {\n  pub doubles: Vec<OrderedFloat<f64>>,\n  pub shape: Option<Vec<i64>>,\n}\n\nimpl DoubleTensor {\n  pub fn new<F2>(doubles: Vec<OrderedFloat<f64>>, shape: F2) -> DoubleTensor where F2: Into<Option<Vec<i64>>> {\n    DoubleTensor {\n      doubles,\n      shape: shape.into(),\n    }\n  }\n}\n\nimpl TSerializable for DoubleTensor {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<DoubleTensor> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<Vec<OrderedFloat<f64>>> = None;\n    let mut f_2: Option<Vec<i64>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<OrderedFloat<f64>> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_8 = OrderedFloat::from(i_prot.read_double()?);\n            val.push(list_elem_8);\n          }\n          i_prot.read_list_end()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<i64> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_9 = i_prot.read_i64()?;\n            val.push(list_elem_9);\n          }\n          i_prot.read_list_end()?;\n          f_2 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"DoubleTensor.doubles\", &f_1)?;\n    let ret = DoubleTensor {\n      doubles: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n      shape: f_2,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"DoubleTensor\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"doubles\", TType::List, 1))?;\n    o_prot.write_list_begin(&TListIdentifier::new(TType::Double, self.doubles.len() as i32))?;\n    for e in &self.doubles {\n      o_prot.write_double((*e).into())?;\n    }\n    o_prot.write_list_end()?;\n    o_prot.write_field_end()?;\n    if let Some(ref fld_var) = self.shape {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"shape\", TType::List, 2))?;\n      o_prot.write_list_begin(&TListIdentifier::new(TType::I64, fld_var.len() as i32))?;\n      for e in fld_var {\n        o_prot.write_i64(*e)?;\n      }\n      o_prot.write_list_end()?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// BoolTensor\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct BoolTensor {\n  pub booleans: Vec<bool>,\n  pub shape: Option<Vec<i64>>,\n}\n\nimpl BoolTensor {\n  pub fn new<F2>(booleans: Vec<bool>, shape: F2) -> BoolTensor where F2: Into<Option<Vec<i64>>> {\n    BoolTensor {\n      booleans,\n      shape: shape.into(),\n    }\n  }\n}\n\nimpl TSerializable for BoolTensor {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<BoolTensor> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<Vec<bool>> = None;\n    let mut f_2: Option<Vec<i64>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<bool> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_10 = i_prot.read_bool()?;\n            val.push(list_elem_10);\n          }\n          i_prot.read_list_end()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<i64> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_11 = i_prot.read_i64()?;\n            val.push(list_elem_11);\n          }\n          i_prot.read_list_end()?;\n          f_2 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"BoolTensor.booleans\", &f_1)?;\n    let ret = BoolTensor {\n      booleans: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n      shape: f_2,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"BoolTensor\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"booleans\", TType::List, 1))?;\n    o_prot.write_list_begin(&TListIdentifier::new(TType::Bool, self.booleans.len() as i32))?;\n    for e in &self.booleans {\n      o_prot.write_bool(*e)?;\n    }\n    o_prot.write_list_end()?;\n    o_prot.write_field_end()?;\n    if let Some(ref fld_var) = self.shape {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"shape\", TType::List, 2))?;\n      o_prot.write_list_begin(&TListIdentifier::new(TType::I64, fld_var.len() as i32))?;\n      for e in fld_var {\n        o_prot.write_i64(*e)?;\n      }\n      o_prot.write_list_end()?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// RawTypedTensor\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct RawTypedTensor {\n  pub data_type: DataType,\n  pub content: Vec<u8>,\n  pub shape: Option<Vec<i64>>,\n}\n\nimpl RawTypedTensor {\n  pub fn new<F3>(data_type: DataType, content: Vec<u8>, shape: F3) -> RawTypedTensor where F3: Into<Option<Vec<i64>>> {\n    RawTypedTensor {\n      data_type,\n      content,\n      shape: shape.into(),\n    }\n  }\n}\n\nimpl TSerializable for RawTypedTensor {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<RawTypedTensor> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<DataType> = None;\n    let mut f_2: Option<Vec<u8>> = None;\n    let mut f_3: Option<Vec<i64>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let val = DataType::read_from_in_protocol(i_prot)?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let val = i_prot.read_bytes()?;\n          f_2 = Some(val);\n        },\n        3 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<i64> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_12 = i_prot.read_i64()?;\n            val.push(list_elem_12);\n          }\n          i_prot.read_list_end()?;\n          f_3 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"RawTypedTensor.data_type\", &f_1)?;\n    verify_required_field_exists(\"RawTypedTensor.content\", &f_2)?;\n    let ret = RawTypedTensor {\n      data_type: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n      content: f_2.expect(\"auto-generated code should have checked for presence of required fields\"),\n      shape: f_3,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"RawTypedTensor\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"dataType\", TType::I32, 1))?;\n    self.data_type.write_to_out_protocol(o_prot)?;\n    o_prot.write_field_end()?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"content\", TType::String, 2))?;\n    o_prot.write_bytes(&self.content)?;\n    o_prot.write_field_end()?;\n    if let Some(ref fld_var) = self.shape {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"shape\", TType::List, 3))?;\n      o_prot.write_list_begin(&TListIdentifier::new(TType::I64, fld_var.len() as i32))?;\n      for e in fld_var {\n        o_prot.write_i64(*e)?;\n      }\n      o_prot.write_list_end()?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// BinaryTensor\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct BinaryTensor {\n  pub binaries: Vec<Vec<u8>>,\n  pub shape: Option<Vec<i64>>,\n}\n\nimpl BinaryTensor {\n  pub fn new<F2>(binaries: Vec<Vec<u8>>, shape: F2) -> BinaryTensor where F2: Into<Option<Vec<i64>>> {\n    BinaryTensor {\n      binaries,\n      shape: shape.into(),\n    }\n  }\n}\n\nimpl TSerializable for BinaryTensor {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<BinaryTensor> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<Vec<Vec<u8>>> = None;\n    let mut f_2: Option<Vec<i64>> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<Vec<u8>> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_13 = i_prot.read_bytes()?;\n            val.push(list_elem_13);\n          }\n          i_prot.read_list_end()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<i64> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_14 = i_prot.read_i64()?;\n            val.push(list_elem_14);\n          }\n          i_prot.read_list_end()?;\n          f_2 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"BinaryTensor.binaries\", &f_1)?;\n    let ret = BinaryTensor {\n      binaries: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n      shape: f_2,\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"BinaryTensor\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"binaries\", TType::List, 1))?;\n    o_prot.write_list_begin(&TListIdentifier::new(TType::String, self.binaries.len() as i32))?;\n    for e in &self.binaries {\n      o_prot.write_bytes(e)?;\n    }\n    o_prot.write_list_end()?;\n    o_prot.write_field_end()?;\n    if let Some(ref fld_var) = self.shape {\n      o_prot.write_field_begin(&TFieldIdentifier::new(\"shape\", TType::List, 2))?;\n      o_prot.write_list_begin(&TListIdentifier::new(TType::I64, fld_var.len() as i32))?;\n      for e in fld_var {\n        o_prot.write_i64(*e)?;\n      }\n      o_prot.write_list_end()?;\n      o_prot.write_field_end()?\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// GeneralTensor\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub enum GeneralTensor {\n  RawTypedTensor(RawTypedTensor),\n  StringTensor(StringTensor),\n  Int32Tensor(Int32Tensor),\n  Int64Tensor(Int64Tensor),\n  FloatTensor(FloatTensor),\n  DoubleTensor(DoubleTensor),\n  BoolTensor(BoolTensor),\n  BinaryTensor(BinaryTensor),\n}\n\nimpl TSerializable for GeneralTensor {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<GeneralTensor> {\n    let mut ret: Option<GeneralTensor> = None;\n    let mut received_field_count = 0;\n    i_prot.read_struct_begin()?;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let val = RawTypedTensor::read_from_in_protocol(i_prot)?;\n          if ret.is_none() {\n            ret = Some(GeneralTensor::RawTypedTensor(val));\n          }\n          received_field_count += 1;\n        },\n        2 => {\n          let val = StringTensor::read_from_in_protocol(i_prot)?;\n          if ret.is_none() {\n            ret = Some(GeneralTensor::StringTensor(val));\n          }\n          received_field_count += 1;\n        },\n        3 => {\n          let val = Int32Tensor::read_from_in_protocol(i_prot)?;\n          if ret.is_none() {\n            ret = Some(GeneralTensor::Int32Tensor(val));\n          }\n          received_field_count += 1;\n        },\n        4 => {\n          let val = Int64Tensor::read_from_in_protocol(i_prot)?;\n          if ret.is_none() {\n            ret = Some(GeneralTensor::Int64Tensor(val));\n          }\n          received_field_count += 1;\n        },\n        5 => {\n          let val = FloatTensor::read_from_in_protocol(i_prot)?;\n          if ret.is_none() {\n            ret = Some(GeneralTensor::FloatTensor(val));\n          }\n          received_field_count += 1;\n        },\n        6 => {\n          let val = DoubleTensor::read_from_in_protocol(i_prot)?;\n          if ret.is_none() {\n            ret = Some(GeneralTensor::DoubleTensor(val));\n          }\n          received_field_count += 1;\n        },\n        7 => {\n          let val = BoolTensor::read_from_in_protocol(i_prot)?;\n          if ret.is_none() {\n            ret = Some(GeneralTensor::BoolTensor(val));\n          }\n          received_field_count += 1;\n        },\n        8 => {\n          let val = BinaryTensor::read_from_in_protocol(i_prot)?;\n          if ret.is_none() {\n            ret = Some(GeneralTensor::BinaryTensor(val));\n          }\n          received_field_count += 1;\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n          received_field_count += 1;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    if received_field_count == 0 {\n      Err(\n        thrift::Error::Protocol(\n          ProtocolError::new(\n            ProtocolErrorKind::InvalidData,\n            \"received empty union from remote GeneralTensor\"\n          )\n        )\n      )\n    } else if received_field_count > 1 {\n      Err(\n        thrift::Error::Protocol(\n          ProtocolError::new(\n            ProtocolErrorKind::InvalidData,\n            \"received multiple fields for union from remote GeneralTensor\"\n          )\n        )\n      )\n    } else {\n      Ok(ret.expect(\"return value should have been constructed\"))\n    }\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"GeneralTensor\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    match *self {\n      GeneralTensor::RawTypedTensor(ref f) => {\n        o_prot.write_field_begin(&TFieldIdentifier::new(\"rawTypedTensor\", TType::Struct, 1))?;\n        f.write_to_out_protocol(o_prot)?;\n        o_prot.write_field_end()?;\n      },\n      GeneralTensor::StringTensor(ref f) => {\n        o_prot.write_field_begin(&TFieldIdentifier::new(\"stringTensor\", TType::Struct, 2))?;\n        f.write_to_out_protocol(o_prot)?;\n        o_prot.write_field_end()?;\n      },\n      GeneralTensor::Int32Tensor(ref f) => {\n        o_prot.write_field_begin(&TFieldIdentifier::new(\"int32Tensor\", TType::Struct, 3))?;\n        f.write_to_out_protocol(o_prot)?;\n        o_prot.write_field_end()?;\n      },\n      GeneralTensor::Int64Tensor(ref f) => {\n        o_prot.write_field_begin(&TFieldIdentifier::new(\"int64Tensor\", TType::Struct, 4))?;\n        f.write_to_out_protocol(o_prot)?;\n        o_prot.write_field_end()?;\n      },\n      GeneralTensor::FloatTensor(ref f) => {\n        o_prot.write_field_begin(&TFieldIdentifier::new(\"floatTensor\", TType::Struct, 5))?;\n        f.write_to_out_protocol(o_prot)?;\n        o_prot.write_field_end()?;\n      },\n      GeneralTensor::DoubleTensor(ref f) => {\n        o_prot.write_field_begin(&TFieldIdentifier::new(\"doubleTensor\", TType::Struct, 6))?;\n        f.write_to_out_protocol(o_prot)?;\n        o_prot.write_field_end()?;\n      },\n      GeneralTensor::BoolTensor(ref f) => {\n        o_prot.write_field_begin(&TFieldIdentifier::new(\"boolTensor\", TType::Struct, 7))?;\n        f.write_to_out_protocol(o_prot)?;\n        o_prot.write_field_end()?;\n      },\n      GeneralTensor::BinaryTensor(ref f) => {\n        o_prot.write_field_begin(&TFieldIdentifier::new(\"binaryTensor\", TType::Struct, 8))?;\n        f.write_to_out_protocol(o_prot)?;\n        o_prot.write_field_end()?;\n      },\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// COOSparseTensor\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub struct COOSparseTensor {\n  pub dense_shape: Vec<i64>,\n  pub indices: Int64Tensor,\n  pub values: GeneralTensor,\n}\n\nimpl COOSparseTensor {\n  pub fn new(dense_shape: Vec<i64>, indices: Int64Tensor, values: GeneralTensor) -> COOSparseTensor {\n    COOSparseTensor {\n      dense_shape,\n      indices,\n      values,\n    }\n  }\n}\n\nimpl TSerializable for COOSparseTensor {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<COOSparseTensor> {\n    i_prot.read_struct_begin()?;\n    let mut f_1: Option<Vec<i64>> = None;\n    let mut f_2: Option<Int64Tensor> = None;\n    let mut f_3: Option<GeneralTensor> = None;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let list_ident = i_prot.read_list_begin()?;\n          let mut val: Vec<i64> = Vec::with_capacity(list_ident.size as usize);\n          for _ in 0..list_ident.size {\n            let list_elem_15 = i_prot.read_i64()?;\n            val.push(list_elem_15);\n          }\n          i_prot.read_list_end()?;\n          f_1 = Some(val);\n        },\n        2 => {\n          let val = Int64Tensor::read_from_in_protocol(i_prot)?;\n          f_2 = Some(val);\n        },\n        3 => {\n          let val = GeneralTensor::read_from_in_protocol(i_prot)?;\n          f_3 = Some(val);\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    verify_required_field_exists(\"COOSparseTensor.dense_shape\", &f_1)?;\n    verify_required_field_exists(\"COOSparseTensor.indices\", &f_2)?;\n    verify_required_field_exists(\"COOSparseTensor.values\", &f_3)?;\n    let ret = COOSparseTensor {\n      dense_shape: f_1.expect(\"auto-generated code should have checked for presence of required fields\"),\n      indices: f_2.expect(\"auto-generated code should have checked for presence of required fields\"),\n      values: f_3.expect(\"auto-generated code should have checked for presence of required fields\"),\n    };\n    Ok(ret)\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"COOSparseTensor\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"denseShape\", TType::List, 1))?;\n    o_prot.write_list_begin(&TListIdentifier::new(TType::I64, self.dense_shape.len() as i32))?;\n    for e in &self.dense_shape {\n      o_prot.write_i64(*e)?;\n    }\n    o_prot.write_list_end()?;\n    o_prot.write_field_end()?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"indices\", TType::Struct, 2))?;\n    self.indices.write_to_out_protocol(o_prot)?;\n    o_prot.write_field_end()?;\n    o_prot.write_field_begin(&TFieldIdentifier::new(\"values\", TType::Struct, 3))?;\n    self.values.write_to_out_protocol(o_prot)?;\n    o_prot.write_field_end()?;\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n//\n// SparseTensor\n//\n\n#[derive(Clone, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]\npub enum SparseTensor {\n  CooSparseTensor(COOSparseTensor),\n}\n\nimpl TSerializable for SparseTensor {\n  fn read_from_in_protocol(i_prot: &mut dyn TInputProtocol) -> thrift::Result<SparseTensor> {\n    let mut ret: Option<SparseTensor> = None;\n    let mut received_field_count = 0;\n    i_prot.read_struct_begin()?;\n    loop {\n      let field_ident = i_prot.read_field_begin()?;\n      if field_ident.field_type == TType::Stop {\n        break;\n      }\n      let field_id = field_id(&field_ident)?;\n      match field_id {\n        1 => {\n          let val = COOSparseTensor::read_from_in_protocol(i_prot)?;\n          if ret.is_none() {\n            ret = Some(SparseTensor::CooSparseTensor(val));\n          }\n          received_field_count += 1;\n        },\n        _ => {\n          i_prot.skip(field_ident.field_type)?;\n          received_field_count += 1;\n        },\n      };\n      i_prot.read_field_end()?;\n    }\n    i_prot.read_struct_end()?;\n    if received_field_count == 0 {\n      Err(\n        thrift::Error::Protocol(\n          ProtocolError::new(\n            ProtocolErrorKind::InvalidData,\n            \"received empty union from remote SparseTensor\"\n          )\n        )\n      )\n    } else if received_field_count > 1 {\n      Err(\n        thrift::Error::Protocol(\n          ProtocolError::new(\n            ProtocolErrorKind::InvalidData,\n            \"received multiple fields for union from remote SparseTensor\"\n          )\n        )\n      )\n    } else {\n      Ok(ret.expect(\"return value should have been constructed\"))\n    }\n  }\n  fn write_to_out_protocol(&self, o_prot: &mut dyn TOutputProtocol) -> thrift::Result<()> {\n    let struct_ident = TStructIdentifier::new(\"SparseTensor\");\n    o_prot.write_struct_begin(&struct_ident)?;\n    match *self {\n      SparseTensor::CooSparseTensor(ref f) => {\n        o_prot.write_field_begin(&TFieldIdentifier::new(\"cooSparseTensor\", TType::Struct, 1))?;\n        f.write_to_out_protocol(o_prot)?;\n        o_prot.write_field_end()?;\n      },\n    }\n    o_prot.write_field_stop()?;\n    o_prot.write_struct_end()\n  }\n}\n\n"
  },
  {
    "path": "product-mixer/README.md",
    "content": "Product Mixer\n=============\n\n## Overview\n\nProduct Mixer is a common service framework and set of libraries that make it easy to build,\niterate on, and own product surface areas. It consists of:\n\n- **Core Libraries:** A set of libraries that enable you to build execution pipelines out of\n  reusable components. You define your logic in small, well-defined, reusable components and focus\n  on expressing the business logic you want to have. Then you can define easy to understand pipelines\n  that compose your components. Product Mixer handles the execution and monitoring of your pipelines\n  allowing you to focus on what really matters, your business logic.\n\n- **Service Framework:** A common service skeleton for teams to host their Product Mixer products.\n\n- **Component Library:** A shared library of components made by the Product Mixer Team, or\n  contributed by users. This enables you to both easily share the reusable components you make as well\n  as benefit from the work other teams have done by utilizing their shared components in the library.\n\n## Architecture\n\nThe bulk of a Product Mixer can be broken down into Pipelines and Components. Components allow you\nto break business logic into separate, standardized, reusable, testable, and easily composable\npieces, where each component has a well defined abstraction. Pipelines are essentially configuration\nfiles specifying which Components should be used and when. This makes it easy to understand how your\ncode will execute while keeping it organized and structured in a maintainable way.\n\nRequests first go to Product Pipelines, which are used to select which Mixer Pipeline or\nRecommendation Pipeline to run for a given request. Each Mixer or Recommendation\nPipeline may run multiple Candidate Pipelines to fetch candidates to include in the response.\n\nMixer Pipelines combine the results of multiple heterogeneous Candidate Pipelines together\n(e.g. ads, tweets, users) while Recommendation Pipelines are used to score (via Scoring Pipelines)\nand rank the results of homogenous Candidate Pipelines so that the top ranked ones can be returned.\nThese pipelines also marshall candidates into a domain object and then into a transport object\nto return to the caller.\n\nCandidate Pipelines fetch candidates from underlying Candidate Sources and perform some basic\noperations on the Candidates, such as filtering out unwanted candidates, applying decorations,\nand hydrating features.\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/account_recommendations_mixer/AccountRecommendationsMixerCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.account_recommendations_mixer\n\nimport com.twitter.account_recommendations_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject WhoToFollowModuleHeaderFeature extends Feature[UserCandidate, t.Header]\nobject WhoToFollowModuleFooterFeature extends Feature[UserCandidate, Option[t.Footer]]\nobject WhoToFollowModuleDisplayOptionsFeature\n    extends Feature[UserCandidate, Option[t.DisplayOptions]]\n\n@Singleton\nclass AccountRecommendationsMixerCandidateSource @Inject() (\n  accountRecommendationsMixer: t.AccountRecommendationsMixer.MethodPerEndpoint)\n    extends CandidateSourceWithExtractedFeatures[\n      t.AccountRecommendationsMixerRequest,\n      t.RecommendedUser\n    ] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(name = \"AccountRecommendationsMixer\")\n\n  override def apply(\n    request: t.AccountRecommendationsMixerRequest\n  ): Stitch[CandidatesWithSourceFeatures[t.RecommendedUser]] = {\n    Stitch\n      .callFuture(accountRecommendationsMixer.getWtfRecommendations(request))\n      .map { response: t.WhoToFollowResponse =>\n        responseToCandidatesWithSourceFeatures(\n          response.userRecommendations,\n          response.header,\n          response.footer,\n          response.displayOptions)\n      }\n  }\n\n  private def responseToCandidatesWithSourceFeatures(\n    userRecommendations: Seq[t.RecommendedUser],\n    header: t.Header,\n    footer: Option[t.Footer],\n    displayOptions: Option[t.DisplayOptions],\n  ): CandidatesWithSourceFeatures[t.RecommendedUser] = {\n    val features = FeatureMapBuilder()\n      .add(WhoToFollowModuleHeaderFeature, header)\n      .add(WhoToFollowModuleFooterFeature, footer)\n      .add(WhoToFollowModuleDisplayOptionsFeature, displayOptions)\n      .build()\n    CandidatesWithSourceFeatures(userRecommendations, features)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/account_recommendations_mixer/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"account-recommendations-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_common-scala\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"account-recommendations-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/ads/AdsProdStratoCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.ads\n\nimport com.twitter.adserver.thriftscala.AdImpression\nimport com.twitter.adserver.thriftscala.AdRequestParams\nimport com.twitter.adserver.thriftscala.AdRequestResponse\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyFetcherSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.ads.admixer.MakeAdRequestClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AdsProdStratoCandidateSource @Inject() (adsClient: MakeAdRequestClientColumn)\n    extends StratoKeyFetcherSource[\n      AdRequestParams,\n      AdRequestResponse,\n      AdImpression\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"AdsProdStrato\")\n\n  override val fetcher: Fetcher[AdRequestParams, Unit, AdRequestResponse] = adsClient.fetcher\n\n  override protected def stratoResultTransformer(\n    stratoResult: AdRequestResponse\n  ): Seq[AdImpression] =\n    stratoResult.impressions\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/ads/AdsProdThriftCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.ads\n\nimport com.twitter.adserver.thriftscala.AdImpression\nimport com.twitter.adserver.thriftscala.AdRequestParams\nimport com.twitter.adserver.thriftscala.NewAdServer\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AdsProdThriftCandidateSource @Inject() (\n  adServerClient: NewAdServer.MethodPerEndpoint)\n    extends CandidateSource[AdRequestParams, AdImpression] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"AdsProdThrift\")\n\n  override def apply(request: AdRequestParams): Stitch[Seq[AdImpression]] =\n    Stitch.callFuture(adServerClient.makeAdRequest(request)).map(_.impressions)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/ads/AdsStagingCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.ads\n\nimport com.twitter.adserver.thriftscala.AdImpression\nimport com.twitter.adserver.thriftscala.AdRequestParams\nimport com.twitter.adserver.thriftscala.AdRequestResponse\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyFetcherSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.ads.admixer.MakeAdRequestStagingClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AdsStagingCandidateSource @Inject() (adsClient: MakeAdRequestStagingClientColumn)\n    extends StratoKeyFetcherSource[\n      AdRequestParams,\n      AdRequestResponse,\n      AdImpression\n    ] {\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"AdsStaging\")\n\n  override val fetcher: Fetcher[AdRequestParams, Unit, AdRequestResponse] = adsClient.fetcher\n\n  override protected def stratoResultTransformer(\n    stratoResult: AdRequestResponse\n  ): Seq[AdImpression] =\n    stratoResult.impressions\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/ads/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_common-scala\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_rpc-scala\",\n        \"strato/config/columns/ads/admixer:admixer-strato-client\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_common-scala\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_rpc-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/ann/AnnCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.ann\n\nimport com.twitter.ann.common._\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.{Time => _, _}\nimport com.twitter.finagle.util.DefaultTimer\n\n/**\n * @param annQueryableById Ann Queryable by Id client that returns nearest neighbors for a sequence of queries\n * @param identifier Candidate Source Identifier\n * @tparam T1 type of the query.\n * @tparam T2 type of the result.\n * @tparam P  runtime parameters supported by the index.\n * @tparam D  distance function used in the index.\n */\nclass AnnCandidateSource[T1, T2, P <: RuntimeParams, D <: Distance[D]](\n  val annQueryableById: QueryableById[T1, T2, P, D],\n  val batchSize: Int,\n  val timeoutPerRequest: Duration,\n  override val identifier: CandidateSourceIdentifier)\n    extends CandidateSource[AnnIdQuery[T1, P], NeighborWithDistanceWithSeed[T1, T2, D]] {\n\n  implicit val timer = DefaultTimer\n\n  override def apply(\n    request: AnnIdQuery[T1, P]\n  ): Stitch[Seq[NeighborWithDistanceWithSeed[T1, T2, D]]] = {\n    val ids = request.ids\n    val numOfNeighbors = request.numOfNeighbors\n    val runtimeParams = request.runtimeParams\n    Stitch\n      .collect(\n        ids\n          .grouped(batchSize).map { batchedIds =>\n            annQueryableById\n              .batchQueryWithDistanceById(batchedIds, numOfNeighbors, runtimeParams).map {\n                annResult => annResult.toSeq\n              }.within(timeoutPerRequest).handle { case _ => Seq.empty }\n          }.toSeq).map(_.flatten)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/ann/AnnIdQuery.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.ann\n\nimport com.twitter.ann.common._\n\n/**\n * A [[AnnIdQuery]] is a query class which defines the ann entities with runtime params and number of neighbors requested\n *\n * @param ids Sequence of queries\n * @param numOfNeighbors Number of neighbors requested\n * @param runtimeParams ANN Runtime Params\n * @param batchSize Batch size to the stitch client\n * @tparam T type of  query.\n * @tparam P  runtime parameters supported by the index.\n */\ncase class AnnIdQuery[T, P <: RuntimeParams](\n  ids: Seq[T],\n  numOfNeighbors: Int,\n  runtimeParams: P)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/ann/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/scala/com/twitter/ann/hnsw\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"product-mixer/component-library/src/main/thrift/com/twitter/product_mixer/component_library:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"servo/manhattan/src/main/scala\",\n        \"servo/repo/src/main/scala\",\n        \"servo/util/src/main/scala\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/audiospace/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"src/thrift/com/twitter/periscope/audio_space:audio_space-scala\",\n        \"strato/config/columns/periscope:periscope-strato-client\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql:graphql-scala\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/audiospace/CreatedSpacesCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.audiospace\n\nimport com.twitter.periscope.audio_space.thriftscala.CreatedSpacesView\nimport com.twitter.periscope.audio_space.thriftscala.SpaceSlice\nimport com.twitter.product_mixer.component_library.model.cursor.NextCursorFeature\nimport com.twitter.product_mixer.component_library.model.cursor.PreviousCursorFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyViewFetcherWithSourceFeaturesSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.periscope.CreatedSpacesSliceOnUserClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CreatedSpacesCandidateSource @Inject() (\n  column: CreatedSpacesSliceOnUserClientColumn)\n    extends StratoKeyViewFetcherWithSourceFeaturesSource[\n      Long,\n      CreatedSpacesView,\n      SpaceSlice,\n      String\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"CreatedSpaces\")\n\n  override val fetcher: Fetcher[Long, CreatedSpacesView, SpaceSlice] = column.fetcher\n\n  override def stratoResultTransformer(\n    stratoKey: Long,\n    stratoResult: SpaceSlice\n  ): Seq[String] =\n    stratoResult.items\n\n  override protected def extractFeaturesFromStratoResult(\n    stratoKey: Long,\n    stratoResult: SpaceSlice\n  ): FeatureMap = {\n    val featureMapBuilder = FeatureMapBuilder()\n    stratoResult.sliceInfo.previousCursor.foreach { cursor =>\n      featureMapBuilder.add(PreviousCursorFeature, cursor)\n    }\n    stratoResult.sliceInfo.nextCursor.foreach { cursor =>\n      featureMapBuilder.add(NextCursorFeature, cursor)\n    }\n    featureMapBuilder.build()\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/business_profiles/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"strato/config/columns/consumer-identity/business-profiles:business-profiles-strato-client\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql:graphql-scala\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/business_profiles/TeamMembersCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.business_profiles\n\nimport com.twitter.product_mixer.component_library.model.cursor.NextCursorFeature\nimport com.twitter.product_mixer.component_library.model.cursor.PreviousCursorFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyViewFetcherWithSourceFeaturesSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.consumer_identity.business_profiles.BusinessProfileTeamMembersOnUserClientColumn\nimport com.twitter.strato.generated.client.consumer_identity.business_profiles.BusinessProfileTeamMembersOnUserClientColumn.{\n  Value => TeamMembersSlice\n}\nimport com.twitter.strato.generated.client.consumer_identity.business_profiles.BusinessProfileTeamMembersOnUserClientColumn.{\n  View => TeamMembersView\n}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TeamMembersCandidateSource @Inject() (\n  column: BusinessProfileTeamMembersOnUserClientColumn)\n    extends StratoKeyViewFetcherWithSourceFeaturesSource[\n      Long,\n      TeamMembersView,\n      TeamMembersSlice,\n      Long\n    ] {\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    \"BusinessProfileTeamMembers\")\n\n  override val fetcher: Fetcher[Long, TeamMembersView, TeamMembersSlice] = column.fetcher\n\n  override def stratoResultTransformer(\n    stratoKey: Long,\n    stratoResult: TeamMembersSlice\n  ): Seq[Long] =\n    stratoResult.members\n\n  override protected def extractFeaturesFromStratoResult(\n    stratoKey: Long,\n    stratoResult: TeamMembersSlice\n  ): FeatureMap = {\n    val featureMapBuilder = FeatureMapBuilder()\n    stratoResult.previousCursor.foreach { cursor =>\n      featureMapBuilder.add(PreviousCursorFeature, cursor.toString)\n    }\n    stratoResult.nextCursor.foreach { cursor =>\n      featureMapBuilder.add(NextCursorFeature, cursor.toString)\n    }\n    featureMapBuilder.build()\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/cr_mixer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/cr_mixer/CrMixerFrsBasedTweetRecommendationsCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.cr_mixer\n\nimport com.twitter.cr_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Returns out-of-network Tweet recommendations by using user recommendations\n * from FollowRecommendationService as an input seed-set to Earlybird\n */\n@Singleton\nclass CrMixerFrsBasedTweetRecommendationsCandidateSource @Inject() (\n  crMixerClient: t.CrMixer.MethodPerEndpoint)\n    extends CandidateSource[t.FrsTweetRequest, t.FrsTweet] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"CrMixerFrsBasedTweetRecommendations\")\n\n  override def apply(request: t.FrsTweetRequest): Stitch[Seq[t.FrsTweet]] = Stitch\n    .callFuture(crMixerClient.getFrsBasedTweetRecommendations(request))\n    .map(_.tweets)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/cr_mixer/CrMixerTweetRecommendationsCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.cr_mixer\n\nimport com.twitter.cr_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CrMixerTweetRecommendationsCandidateSource @Inject() (\n  crMixerClient: t.CrMixer.MethodPerEndpoint)\n    extends CandidateSource[t.CrMixerTweetRequest, t.TweetRecommendation] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"CrMixerTweetRecommendations\")\n\n  override def apply(request: t.CrMixerTweetRequest): Stitch[Seq[t.TweetRecommendation]] = Stitch\n    .callFuture(crMixerClient.getTweetRecommendations(request))\n    .map(_.tweets)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/earlybird/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/earlybird/EarlybirdTweetCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.earlybird\n\nimport com.twitter.search.earlybird.{thriftscala => t}\nimport com.twitter.inject.Logging\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass EarlybirdTweetCandidateSource @Inject() (\n  earlybirdService: t.EarlybirdService.MethodPerEndpoint)\n    extends CandidateSource[t.EarlybirdRequest, t.ThriftSearchResult]\n    with Logging {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"EarlybirdTweets\")\n\n  override def apply(request: t.EarlybirdRequest): Stitch[Seq[t.ThriftSearchResult]] = {\n    Stitch\n      .callFuture(earlybirdService.search(request))\n      .map { response: t.EarlybirdResponse =>\n        response.searchResults.map(_.results).getOrElse(Seq.empty)\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/explore_ranker/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"explore/explore-ranker/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/explore_ranker/ExploreRankerCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.explore_ranker\n\nimport com.twitter.explore_ranker.{thriftscala => t}\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ExploreRankerCandidateSource @Inject() (\n  exploreRankerService: t.ExploreRanker.MethodPerEndpoint)\n    extends CandidateSource[t.ExploreRankerRequest, t.ImmersiveRecsResult] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"ExploreRanker\")\n\n  override def apply(\n    request: t.ExploreRankerRequest\n  ): Stitch[Seq[t.ImmersiveRecsResult]] = {\n    Stitch\n      .callFuture(exploreRankerService.getRankedResults(request))\n      .map {\n        case t.ExploreRankerResponse(\n              t.ExploreRankerProductResponse\n                .ImmersiveRecsResponse(t.ImmersiveRecsResponse(immersiveRecsResults))) =>\n          immersiveRecsResults\n        case response =>\n          throw new UnsupportedOperationException(s\"Unknown response type: $response\")\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/flexible_injection_pipeline/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/flexible_injection_pipeline/PromptCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.flexible_injection_pipeline\n\nimport com.twitter.inject.Logging\nimport com.twitter.onboarding.injections.{thriftscala => injectionsthrift}\nimport com.twitter.onboarding.task.service.{thriftscala => servicethrift}\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Returns a list of prompts to insert into a user's timeline (inline prompt, cover modals, etc)\n * from go/flip (the prompting platform for Twitter).\n */\n@Singleton\nclass PromptCandidateSource @Inject() (taskService: servicethrift.TaskService.MethodPerEndpoint)\n    extends CandidateSource[servicethrift.GetInjectionsRequest, IntermediatePrompt]\n    with Logging {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    \"InjectionPipelinePrompts\")\n\n  override def apply(\n    request: servicethrift.GetInjectionsRequest\n  ): Stitch[Seq[IntermediatePrompt]] = {\n    Stitch\n      .callFuture(taskService.getInjections(request)).map {\n        _.injections.flatMap {\n          // The entire carousel is getting added to each IntermediatePrompt item with a\n          // corresponding index to be unpacked later on to populate its TimelineEntry counterpart.\n          case injection: injectionsthrift.Injection.TilesCarousel =>\n            injection.tilesCarousel.tiles.zipWithIndex.map {\n              case (tile: injectionsthrift.Tile, index: Int) =>\n                IntermediatePrompt(injection, Some(index), Some(tile))\n            }\n          case injection => Seq(IntermediatePrompt(injection, None, None))\n        }\n      }\n  }\n}\n\n/**\n * Gives an intermediate step to help 'explosion' of tile carousel tiles due to TimelineModule\n * not being an extension of TimelineItem\n */\ncase class IntermediatePrompt(\n  injection: injectionsthrift.Injection,\n  offsetInModule: Option[Int],\n  carouselTile: Option[injectionsthrift.Tile])\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/hermit/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"src/thrift/com/twitter/hermit:hermit-scala\",\n        \"strato/config/columns/onboarding:onboarding-strato-client\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"src/thrift/com/twitter/hermit:hermit-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/hermit/UsersSimilarToMeCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.hermit\n\nimport com.twitter.hermit.thriftscala.RecommendationRequest\nimport com.twitter.hermit.thriftscala.RecommendationResponse\nimport com.twitter.hermit.thriftscala.RelatedUser\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyViewFetcherSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.onboarding.HermitRecommendUsersClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UsersSimilarToMeCandidateSource @Inject() (\n  column: HermitRecommendUsersClientColumn)\n    extends StratoKeyViewFetcherSource[\n      Long,\n      RecommendationRequest,\n      RecommendationResponse,\n      RelatedUser\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"UsersSimilarToMe\")\n\n  override val fetcher: Fetcher[Long, RecommendationRequest, RecommendationResponse] =\n    column.fetcher\n\n  override def stratoResultTransformer(\n    stratoKey: Long,\n    result: RecommendationResponse\n  ): Seq[RelatedUser] = result.suggestions.getOrElse(Seq.empty).filter(_.id.isDefined)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/interest_discovery/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"interests-service/thrift/src/main/thrift:thrift-scala\",\n        \"interests_discovery/thrift/src/main/thrift:service-thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"interests-service/thrift/src/main/thrift:thrift-scala\",\n        \"interests_discovery/thrift/src/main/thrift:service-thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/interest_discovery/RelatedTopicsCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.interest_discovery\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.inject.Logging\nimport com.twitter.interests_discovery.{thriftscala => t}\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\n\n/**\n * Generate a list of related topics results from IDS getRelatedTopics (thrift) endpoint.\n * Returns related topics, given a topic, whereas [[RecommendedTopicsCandidateSource]] returns\n * recommended topics, given a user.\n */\n@Singleton\nclass RelatedTopicsCandidateSource @Inject() (\n  interestDiscoveryService: t.InterestsDiscoveryService.MethodPerEndpoint)\n    extends CandidateSource[t.RelatedTopicsRequest, t.RelatedTopic]\n    with Logging {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(name = \"RelatedTopics\")\n\n  override def apply(\n    request: t.RelatedTopicsRequest\n  ): Stitch[Seq[t.RelatedTopic]] = {\n    Stitch\n      .callFuture(interestDiscoveryService.getRelatedTopics(request))\n      .map { response: t.RelatedTopicsResponse =>\n        response.topics\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/lists/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"interests_discovery/thrift/src/main/thrift:service-thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"strato/config/columns/recommendations/interests_discovery/recommendations_mh:recommendations_mh-strato-client\",\n    ],\n    exports = [\n        \"strato/config/columns/recommendations/interests_discovery/recommendations_mh:recommendations_mh-strato-client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/lists/OrganicPopGeoListsCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.lists\n\nimport com.twitter.product_mixer.component_library.model.candidate.TwitterListCandidate\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyFetcherSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.recommendations.interests_discovery.recommendations_mh.OrganicPopgeoListsClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass OrganicPopGeoListsCandidateSource @Inject() (\n  organicPopgeoListsClientColumn: OrganicPopgeoListsClientColumn)\n    extends StratoKeyFetcherSource[\n      OrganicPopgeoListsClientColumn.Key,\n      OrganicPopgeoListsClientColumn.Value,\n      TwitterListCandidate\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    \"OrganicPopGeoLists\")\n\n  override val fetcher: Fetcher[\n    OrganicPopgeoListsClientColumn.Key,\n    Unit,\n    OrganicPopgeoListsClientColumn.Value\n  ] =\n    organicPopgeoListsClientColumn.fetcher\n\n  override def stratoResultTransformer(\n    stratoResult: OrganicPopgeoListsClientColumn.Value\n  ): Seq[TwitterListCandidate] = {\n    stratoResult.recommendedListsByAlgo.flatMap { topLists =>\n      topLists.lists.map { list =>\n        TwitterListCandidate(list.listId)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/people_discovery/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"people-discovery/api/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_common-scala\",\n        \"src/thrift/com/twitter/hermit:hermit-scala\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"people-discovery/api/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/people_discovery/PeopleDiscoveryCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.people_discovery\n\nimport com.twitter.peoplediscovery.api.{thriftscala => t}\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.UnexpectedCandidateResult\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject WhoToFollowModuleHeaderFeature extends Feature[UserCandidate, t.Header]\nobject WhoToFollowModuleDisplayOptionsFeature\n    extends Feature[UserCandidate, Option[t.DisplayOptions]]\nobject WhoToFollowModuleShowMoreFeature extends Feature[UserCandidate, Option[t.ShowMore]]\n\n@Singleton\nclass PeopleDiscoveryCandidateSource @Inject() (\n  peopleDiscoveryService: t.ThriftPeopleDiscoveryService.MethodPerEndpoint)\n    extends CandidateSourceWithExtractedFeatures[t.GetModuleRequest, t.RecommendedUser]\n    with Logging {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(name = \"PeopleDiscovery\")\n\n  override def apply(\n    request: t.GetModuleRequest\n  ): Stitch[CandidatesWithSourceFeatures[t.RecommendedUser]] = {\n    Stitch\n      .callFuture(peopleDiscoveryService.getModules(request))\n      .map { response: t.GetModuleResponse =>\n        // under the assumption getModules returns a maximum of one module\n        response.modules\n          .collectFirst { module =>\n            module.layout match {\n              case t.Layout.UserBioList(layout) =>\n                layoutToCandidatesWithSourceFeatures(\n                  layout.userRecommendations,\n                  layout.header,\n                  layout.displayOptions,\n                  layout.showMore)\n              case t.Layout.UserTweetCarousel(layout) =>\n                layoutToCandidatesWithSourceFeatures(\n                  layout.userRecommendations,\n                  layout.header,\n                  layout.displayOptions,\n                  layout.showMore)\n            }\n          }.getOrElse(throw PipelineFailure(UnexpectedCandidateResult, \"unexpected missing module\"))\n      }\n  }\n\n  private def layoutToCandidatesWithSourceFeatures(\n    userRecommendations: Seq[t.RecommendedUser],\n    header: t.Header,\n    displayOptions: Option[t.DisplayOptions],\n    showMore: Option[t.ShowMore],\n  ): CandidatesWithSourceFeatures[t.RecommendedUser] = {\n    val features = FeatureMapBuilder()\n      .add(WhoToFollowModuleHeaderFeature, header)\n      .add(WhoToFollowModuleDisplayOptionsFeature, displayOptions)\n      .add(WhoToFollowModuleShowMoreFeature, showMore)\n      .build()\n    CandidatesWithSourceFeatures(userRecommendations, features)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/recommendations/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"strato/config/columns/onboarding/follow-recommendations-service:follow-recommendations-service-strato-client\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/recommendations/UserFollowRecommendationsCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.recommendations\n\nimport com.twitter.follow_recommendations.{thriftscala => fr}\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyViewFetcherSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.onboarding.follow_recommendations_service.GetRecommendationsClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Returns a list of FollowRecommendations as [[fr.UserRecommendation]]s fetched from Strato\n */\n@Singleton\nclass UserFollowRecommendationsCandidateSource @Inject() (\n  getRecommendationsClientColumn: GetRecommendationsClientColumn)\n    extends StratoKeyViewFetcherSource[\n      fr.RecommendationRequest,\n      Unit,\n      fr.RecommendationResponse,\n      fr.UserRecommendation\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    \"FollowRecommendationsService\")\n\n  override val fetcher: Fetcher[fr.RecommendationRequest, Unit, fr.RecommendationResponse] =\n    getRecommendationsClientColumn.fetcher\n\n  override def stratoResultTransformer(\n    stratoKey: fr.RecommendationRequest,\n    stratoResult: fr.RecommendationResponse\n  ): Seq[fr.UserRecommendation] = {\n    stratoResult.recommendations.map {\n      case fr.Recommendation.User(userRec: fr.UserRecommendation) =>\n        userRec\n      case _ =>\n        throw new Exception(\"Invalid recommendation type returned from FRS\")\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/social_graph/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"socialgraph/server/src/main/scala/com/twitter/socialgraph/util\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/social_graph/SocialgraphCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.social_graph\n\nimport com.twitter.product_mixer.component_library.model.candidate.CursorType\nimport com.twitter.product_mixer.component_library.model.candidate.NextCursor\nimport com.twitter.product_mixer.component_library.model.candidate.PreviousCursor\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyViewFetcherSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.socialgraph.thriftscala\nimport com.twitter.socialgraph.thriftscala.IdsRequest\nimport com.twitter.socialgraph.thriftscala.IdsResult\nimport com.twitter.socialgraph.util.ByteBufferUtil\nimport com.twitter.strato.client.Fetcher\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nsealed trait SocialgraphResponse\ncase class SocialgraphResult(id: Long) extends SocialgraphResponse\ncase class SocialgraphCursor(cursor: Long, cursorType: CursorType) extends SocialgraphResponse\n\n@Singleton\nclass SocialgraphCandidateSource @Inject() (\n  override val fetcher: Fetcher[thriftscala.IdsRequest, Option[\n    thriftscala.RequestContext\n  ], thriftscala.IdsResult])\n    extends StratoKeyViewFetcherSource[\n      thriftscala.IdsRequest,\n      Option[thriftscala.RequestContext],\n      thriftscala.IdsResult,\n      SocialgraphResponse\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"Socialgraph\")\n\n  override def stratoResultTransformer(\n    stratoKey: IdsRequest,\n    stratoResult: IdsResult\n  ): Seq[SocialgraphResponse] = {\n    val prevCursor =\n      SocialgraphCursor(ByteBufferUtil.toLong(stratoResult.pageResult.prevCursor), PreviousCursor)\n    /* When an end cursor is passed to Socialgraph,\n     * Socialgraph returns the start cursor. To prevent\n     * clients from circularly fetching the timeline again,\n     * if we see a start cursor returned from Socialgraph,\n     * we replace it with an end cursor.\n     */\n    val nextCursor = ByteBufferUtil.toLong(stratoResult.pageResult.nextCursor) match {\n      case SocialgraphCursorConstants.StartCursor =>\n        SocialgraphCursor(SocialgraphCursorConstants.EndCursor, NextCursor)\n      case cursor => SocialgraphCursor(cursor, NextCursor)\n    }\n\n    stratoResult.ids\n      .map { id =>\n        SocialgraphResult(id)\n      } ++ Seq(nextCursor, prevCursor)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/social_graph/SocialgraphCursorConstants.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.social_graph\n\nobject SocialgraphCursorConstants {\n  val EndCursor: Long = 0L\n  val StartCursor: Long = -1L\n  val LastSortIndex: Long = 0L\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_ranker/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"src/thrift/com/twitter/timelineranker:thrift-scala\",\n        \"src/thrift/com/twitter/timelineranker/server/model:thrift-scala\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_ranker/TimelineRankerInNetworkCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.timeline_ranker\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelineranker.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Map of tweetId -> sourceTweet of retweets present in Timeline Ranker candidates list.\n * These tweets are used only for further ranking. They are not returned to the end user.\n */\nobject TimelineRankerInNetworkSourceTweetsByTweetIdMapFeature\n    extends Feature[PipelineQuery, Map[Long, t.CandidateTweet]]\n\n@Singleton\nclass TimelineRankerInNetworkCandidateSource @Inject() (\n  timelineRankerClient: t.TimelineRanker.MethodPerEndpoint)\n    extends CandidateSourceWithExtractedFeatures[t.RecapQuery, t.CandidateTweet] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"TimelineRankerInNetwork\")\n\n  override def apply(\n    request: t.RecapQuery\n  ): Stitch[CandidatesWithSourceFeatures[t.CandidateTweet]] = {\n    Stitch\n      .callFuture(timelineRankerClient.getRecycledTweetCandidates(Seq(request)))\n      .map { response: Seq[t.GetCandidateTweetsResponse] =>\n        val candidates =\n          response.headOption.flatMap(_.candidates).getOrElse(Seq.empty).filter(_.tweet.nonEmpty)\n        val sourceTweetsByTweetId =\n          response.headOption\n            .flatMap(_.sourceTweets).getOrElse(Seq.empty).filter(_.tweet.nonEmpty)\n            .map { candidate =>\n              (candidate.tweet.get.id, candidate)\n            }.toMap\n        val sourceTweetsByTweetIdMapFeature = FeatureMapBuilder()\n          .add(TimelineRankerInNetworkSourceTweetsByTweetIdMapFeature, sourceTweetsByTweetId)\n          .build()\n        CandidatesWithSourceFeatures(\n          candidates = candidates,\n          features = sourceTweetsByTweetIdMapFeature)\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_ranker/TimelineRankerRecapCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.timeline_ranker\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelineranker.{thriftscala => t}\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelineRankerRecapCandidateSource @Inject() (\n  timelineRankerClient: t.TimelineRanker.MethodPerEndpoint)\n    extends CandidateSource[t.RecapQuery, t.CandidateTweet] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"TimelineRankerRecap\")\n\n  override def apply(\n    request: t.RecapQuery\n  ): Stitch[Seq[t.CandidateTweet]] = {\n    Stitch\n      .callFuture(timelineRankerClient.getRecapCandidatesFromAuthors(Seq(request)))\n      .map { response: Seq[t.GetCandidateTweetsResponse] =>\n        response.headOption.flatMap(_.candidates).getOrElse(Seq.empty).filter(_.tweet.nonEmpty)\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_ranker/TimelineRankerUtegCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.timeline_ranker\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.UnexpectedCandidateResult\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelineranker.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Source tweets of retweets present in Timeline Ranker candidates list.\n * These tweets are used only for further ranking. They are not returned to the end user.\n */\ncase object TimelineRankerUtegSourceTweetsFeature\n    extends Feature[PipelineQuery, Seq[t.CandidateTweet]]\n\n@Singleton\nclass TimelineRankerUtegCandidateSource @Inject() (\n  timelineRankerClient: t.TimelineRanker.MethodPerEndpoint)\n    extends CandidateSourceWithExtractedFeatures[t.UtegLikedByTweetsQuery, t.CandidateTweet] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"TimelineRankerUteg\")\n\n  override def apply(\n    request: t.UtegLikedByTweetsQuery\n  ): Stitch[CandidatesWithSourceFeatures[t.CandidateTweet]] = {\n    Stitch\n      .callFuture(timelineRankerClient.getUtegLikedByTweetCandidates(Seq(request)))\n      .map { response =>\n        val result = response.headOption.getOrElse(\n          throw PipelineFailure(UnexpectedCandidateResult, \"Empty Timeline Ranker response\"))\n        val candidates = result.candidates.toSeq.flatten\n        val sourceTweets = result.sourceTweets.toSeq.flatten\n\n        val candidateSourceFeatures = FeatureMapBuilder()\n          .add(TimelineRankerUtegSourceTweetsFeature, sourceTweets)\n          .build()\n\n        CandidatesWithSourceFeatures(candidates = candidates, features = candidateSourceFeatures)\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_scorer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"src/thrift/com/twitter/timelinescorer:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/server/internal:thrift-scala\",\n        \"src/thrift/com/twitter/timelineservice/server/suggests/features:thrift-scala\",\n        \"src/thrift/com/twitter/timelineservice/server/suggests/logging:thrift-scala\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_scorer/TimelineScorerCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.timeline_scorer\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelinescorer.common.scoredtweetcandidate.thriftscala.v1\nimport com.twitter.timelinescorer.common.scoredtweetcandidate.thriftscala.v1.Ancestor\nimport com.twitter.timelinescorer.common.scoredtweetcandidate.{thriftscala => ct}\nimport com.twitter.timelinescorer.{thriftscala => t}\nimport com.twitter.timelineservice.suggests.logging.candidate_tweet_source_id.thriftscala.CandidateTweetSourceId\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase class ScoredTweetCandidateWithFocalTweet(\n  candidate: v1.ScoredTweetCandidate,\n  focalTweetIdOpt: Option[Long])\n\ncase object TimelineScorerCandidateSourceSucceededFeature extends Feature[PipelineQuery, Boolean]\n\n@Singleton\nclass TimelineScorerCandidateSource @Inject() (\n  timelineScorerClient: t.TimelineScorer.MethodPerEndpoint)\n    extends CandidateSourceWithExtractedFeatures[\n      t.ScoredTweetsRequest,\n      ScoredTweetCandidateWithFocalTweet\n    ] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"TimelineScorer\")\n\n  private val MaxConversationAncestors = 2\n\n  override def apply(\n    request: t.ScoredTweetsRequest\n  ): Stitch[CandidatesWithSourceFeatures[ScoredTweetCandidateWithFocalTweet]] = {\n    Stitch\n      .callFuture(timelineScorerClient.getScoredTweets(request))\n      .map { response =>\n        val scoredTweetsOpt = response match {\n          case t.ScoredTweetsResponse.V1(v1) => v1.scoredTweets\n          case t.ScoredTweetsResponse.UnknownUnionField(field) =>\n            throw new UnsupportedOperationException(s\"Unknown response type: ${field.field.name}\")\n        }\n        val scoredTweets = scoredTweetsOpt.getOrElse(Seq.empty)\n\n        val allAncestors = scoredTweets.flatMap {\n          case ct.ScoredTweetCandidate.V1(v1) if isEligibleReply(v1) =>\n            v1.ancestors.get.map(_.tweetId)\n          case _ => Seq.empty\n        }.toSet\n\n        // Remove tweets within ancestor list of other tweets to avoid serving duplicates\n        val keptTweets = scoredTweets.collect {\n          case ct.ScoredTweetCandidate.V1(v1) if !allAncestors.contains(originalTweetId(v1)) => v1\n        }\n\n        // Add parent and root tweet for eligible reply focal tweets\n        val candidates = keptTweets\n          .flatMap {\n            case v1 if isEligibleReply(v1) =>\n              val ancestors = v1.ancestors.get\n              val focalTweetId = v1.tweetId\n\n              // Include root tweet if the conversation has atleast 2 ancestors\n              val optionallyIncludedRootTweet = if (ancestors.size >= MaxConversationAncestors) {\n                val rootTweet = toScoredTweetCandidateFromAncestor(\n                  ancestor = ancestors.last,\n                  inReplyToTweetId = None,\n                  conversationId = v1.conversationId,\n                  ancestors = None,\n                  candidateTweetSourceId = v1.candidateTweetSourceId\n                )\n                Seq((rootTweet, Some(v1)))\n              } else Seq.empty\n\n              /**\n               * Setting the in-reply-to tweet id on the immediate parent, if one exists,\n               * helps ensure tweet type metrics correctly distinguish roots from non-roots.\n               */\n              val inReplyToTweetId = ancestors.tail.headOption.map(_.tweetId)\n              val parentAncestor = toScoredTweetCandidateFromAncestor(\n                ancestor = ancestors.head,\n                inReplyToTweetId = inReplyToTweetId,\n                conversationId = v1.conversationId,\n                ancestors = Some(ancestors.tail),\n                candidateTweetSourceId = v1.candidateTweetSourceId\n              )\n\n              optionallyIncludedRootTweet ++\n                Seq((parentAncestor, Some(v1)), (v1, Some(v1)))\n\n            case any => Seq((any, None)) // Set focalTweetId to None if not eligible for convo\n          }\n\n        /**\n         * Dedup each tweet keeping the one with highest scored Focal Tweet\n         * Focal Tweet ID != the Conversation ID, which is set to the root of the conversation\n         * Focal Tweet ID will be defined for tweets with ancestors that should be\n         * in conversation modules and None for standalone tweets.\n         */\n        val sortedDedupedCandidates = candidates\n          .groupBy { case (v1, _) => v1.tweetId }\n          .mapValues { group =>\n            val (candidate, focalTweetOpt) = group.maxBy {\n              case (_, Some(focal)) => focal.score\n              case (_, None) => 0\n            }\n            ScoredTweetCandidateWithFocalTweet(candidate, focalTweetOpt.map(focal => focal.tweetId))\n          }.values.toSeq.sortBy(_.candidate.tweetId)\n\n        CandidatesWithSourceFeatures(\n          candidates = sortedDedupedCandidates,\n          features = FeatureMapBuilder()\n            .add(TimelineScorerCandidateSourceSucceededFeature, true)\n            .build()\n        )\n      }\n  }\n\n  private def isEligibleReply(candidate: ct.ScoredTweetCandidateAliases.V1Alias): Boolean = {\n    candidate.inReplyToTweetId.nonEmpty &&\n    !candidate.isRetweet.getOrElse(false) &&\n    candidate.ancestors.exists(_.nonEmpty)\n  }\n\n  /**\n   * If we have a retweet, get the source tweet id.\n   * If it is not a retweet, get the regular tweet id.\n   */\n  private def originalTweetId(candidate: ct.ScoredTweetCandidateAliases.V1Alias): Long = {\n    candidate.sourceTweetId.getOrElse(candidate.tweetId)\n  }\n\n  private def toScoredTweetCandidateFromAncestor(\n    ancestor: Ancestor,\n    inReplyToTweetId: Option[Long],\n    conversationId: Option[Long],\n    ancestors: Option[Seq[Ancestor]],\n    candidateTweetSourceId: Option[CandidateTweetSourceId]\n  ): ct.ScoredTweetCandidateAliases.V1Alias = {\n    ct.v1.ScoredTweetCandidate(\n      tweetId = ancestor.tweetId,\n      authorId = ancestor.userId.getOrElse(0L),\n      score = 0.0,\n      isAncestorCandidate = Some(true),\n      inReplyToTweetId = inReplyToTweetId,\n      conversationId = conversationId,\n      ancestors = ancestors,\n      candidateTweetSourceId = candidateTweetSourceId\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_service/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala\",\n        \"stitch/stitch-timelineservice/src/main/scala\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"stitch/stitch-timelineservice/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timeline_service/TimelineServiceTweetCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.timeline_service\n\nimport com.twitter.product_mixer.component_library.model.cursor.NextCursorFeature\nimport com.twitter.product_mixer.component_library.model.cursor.PreviousCursorFeature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.timelineservice.TimelineService\nimport com.twitter.timelineservice.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase object TimelineServiceResponseWasTruncatedFeature\n    extends FeatureWithDefaultOnFailure[PipelineQuery, Boolean] {\n  override val defaultValue: Boolean = false\n}\n\n@Singleton\nclass TimelineServiceTweetCandidateSource @Inject() (\n  timelineService: TimelineService)\n    extends CandidateSourceWithExtractedFeatures[t.TimelineQuery, t.Tweet] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"TimelineServiceTweet\")\n\n  override def apply(request: t.TimelineQuery): Stitch[CandidatesWithSourceFeatures[t.Tweet]] = {\n    timelineService\n      .getTimeline(request).map { timeline =>\n        val candidates = timeline.entries.collect {\n          case t.TimelineEntry.Tweet(tweet) => tweet\n        }\n\n        val candidateSourceFeatures =\n          FeatureMapBuilder()\n            .add(TimelineServiceResponseWasTruncatedFeature, timeline.wasTruncated.getOrElse(false))\n            .add(PreviousCursorFeature, timeline.responseCursor.flatMap(_.top).getOrElse(\"\"))\n            .add(NextCursorFeature, timeline.responseCursor.flatMap(_.bottom).getOrElse(\"\"))\n            .build()\n\n        CandidatesWithSourceFeatures(candidates = candidates, features = candidateSourceFeatures)\n      }\n  }\n\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timelines_impression_store/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"src/thrift/com/twitter/timelines/impression:thrift-scala\",\n        \"strato/config/columns/timelines/impression-store:impression-store-strato-client\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"strato/config/columns/timelines/impression-store:impression-store-strato-client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/timelines_impression_store/TimelinesImpressionStoreCandidateSourceV2.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.timelines_impression_store\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyFetcherSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.timelines.impression_store.TweetImpressionStoreManhattanV2OnUserClientColumn\nimport com.twitter.timelines.impression.thriftscala.TweetImpressionsEntries\nimport com.twitter.timelines.impression.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelinesImpressionStoreCandidateSourceV2 @Inject() (\n  client: TweetImpressionStoreManhattanV2OnUserClientColumn)\n    extends StratoKeyFetcherSource[\n      Long,\n      t.TweetImpressionsEntries,\n      t.TweetImpressionsEntry\n    ] {\n\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\n    \"TimelinesImpressionStore\")\n\n  override val fetcher: Fetcher[Long, Unit, TweetImpressionsEntries] = client.fetcher\n\n  override def stratoResultTransformer(\n    stratoResult: t.TweetImpressionsEntries\n  ): Seq[t.TweetImpressionsEntry] =\n    stratoResult.entries\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/topics/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"strato/config/columns/interests:interests-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/topics/FollowedTopicsCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.topics\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.strato.StratoKeyViewFetcherSeqSource\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.strato.client.Fetcher\nimport com.twitter.strato.generated.client.interests.FollowedTopicsGetterClientColumn\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FollowedTopicsCandidateSource @Inject() (\n  column: FollowedTopicsGetterClientColumn)\n    extends StratoKeyViewFetcherSeqSource[\n      Long,\n      Unit,\n      Long\n    ] {\n  override val identifier: CandidateSourceIdentifier = CandidateSourceIdentifier(\"FollowedTopics\")\n\n  override val fetcher: Fetcher[Long, Unit, Seq[Long]] = column.fetcher\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala\",\n        \"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala\",\n        \"strato/config/columns/tweetconvosvc:tweetconvosvc-strato-client\",\n        \"tweetconvosvc/common/src/main/thrift/com/twitter/tweetconvosvc/tweet_ancestor:thrift-scala\",\n        \"tweetconvosvc/thrift/src/main/thrift:thrift-scala\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"strato/config/columns/tweetconvosvc:tweetconvosvc-strato-client\",\n        \"tweetconvosvc/thrift/src/main/thrift:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc/ConversationServiceCandidateSource.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\nimport com.twitter.tweetconvosvc.tweet_ancestor.{thriftscala => ta}\nimport com.twitter.tweetconvosvc.{thriftscala => tcs}\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\ncase class ConversationServiceCandidateSourceRequest(\n  tweetsWithConversationMetadata: Seq[TweetWithConversationMetadata])\n\ncase class TweetWithConversationMetadata(\n  tweetId: Long,\n  userId: Option[Long],\n  sourceTweetId: Option[Long],\n  sourceUserId: Option[Long],\n  inReplyToTweetId: Option[Long],\n  conversationId: Option[Long],\n  ancestors: Seq[ta.TweetAncestor])\n\n/**\n * Candidate source that fetches ancestors of input candidates from Tweetconvosvc and\n * returns a flattened list of input and ancestor candidates.\n */\n@Singleton\nclass ConversationServiceCandidateSource @Inject() (\n  conversationServiceClient: tcs.ConversationService.MethodPerEndpoint)\n    extends CandidateSourceWithExtractedFeatures[\n      ConversationServiceCandidateSourceRequest,\n      TweetWithConversationMetadata\n    ] {\n\n  override val identifier: CandidateSourceIdentifier =\n    CandidateSourceIdentifier(\"ConversationService\")\n\n  private val maxModuleSize = 3\n  private val maxAncestorsInConversation = 2\n  private val numberOfRootTweets = 1\n  private val maxTweetsInConversationWithSameId = 1\n\n  override def apply(\n    request: ConversationServiceCandidateSourceRequest\n  ): Stitch[CandidatesWithSourceFeatures[TweetWithConversationMetadata]] = {\n    val inputTweetsWithConversationMetadata: Seq[TweetWithConversationMetadata] =\n      request.tweetsWithConversationMetadata\n    val ancestorsRequest =\n      tcs.GetAncestorsRequest(inputTweetsWithConversationMetadata.map(_.tweetId))\n\n    // build the tweets with conversation metadata by calling the conversation service with reduced\n    // ancestors to limit to maxModuleSize\n    val tweetsWithConversationMetadataFromAncestors: Stitch[Seq[TweetWithConversationMetadata]] =\n      Stitch\n        .callFuture(conversationServiceClient.getAncestors(ancestorsRequest))\n        .map { getAncestorsResponse: tcs.GetAncestorsResponse =>\n          inputTweetsWithConversationMetadata\n            .zip(getAncestorsResponse.ancestors).collect {\n              case (focalTweet, tcs.TweetAncestorsResult.TweetAncestors(ancestorsResult))\n                  if ancestorsResult.nonEmpty =>\n                getTweetsInThread(focalTweet, ancestorsResult.head)\n            }.flatten\n        }\n\n    // dedupe the tweets in the list and transform the calling error to\n    // return the requested tweets with conversation metadata\n    val transformedTweetsWithConversationMetadata: Stitch[Seq[TweetWithConversationMetadata]] =\n      tweetsWithConversationMetadataFromAncestors.transform {\n        case Return(ancestors) =>\n          Stitch.value(dedupeCandidates(inputTweetsWithConversationMetadata, ancestors))\n        case Throw(_) =>\n          Stitch.value(inputTweetsWithConversationMetadata)\n      }\n\n    // return the candidates with empty source features from transformed tweetsWithConversationMetadata\n    transformedTweetsWithConversationMetadata.map {\n      responseTweetsWithConversationMetadata: Seq[TweetWithConversationMetadata] =>\n        CandidatesWithSourceFeatures(\n          responseTweetsWithConversationMetadata,\n          FeatureMap.empty\n        )\n    }\n  }\n\n  private def getTweetsInThread(\n    focalTweet: TweetWithConversationMetadata,\n    ancestors: ta.TweetAncestors\n  ): Seq[TweetWithConversationMetadata] = {\n    // Re-add the focal tweet so we can easily build modules and dedupe later.\n    // Note, TweetConvoSVC returns the bottom of the thread first, so we\n    // reverse them for easy rendering.\n    val focalTweetWithConversationMetadata = TweetWithConversationMetadata(\n      tweetId = focalTweet.tweetId,\n      userId = focalTweet.userId,\n      sourceTweetId = focalTweet.sourceTweetId,\n      sourceUserId = focalTweet.sourceUserId,\n      inReplyToTweetId = focalTweet.inReplyToTweetId,\n      conversationId = Some(focalTweet.tweetId),\n      ancestors = ancestors.ancestors\n    )\n\n    val parentTweets = ancestors.ancestors.map { ancestor =>\n      TweetWithConversationMetadata(\n        tweetId = ancestor.tweetId,\n        userId = Some(ancestor.userId),\n        sourceTweetId = None,\n        sourceUserId = None,\n        inReplyToTweetId = None,\n        conversationId = Some(focalTweet.tweetId),\n        ancestors = Seq.empty\n      )\n    } ++ getTruncatedRootTweet(ancestors, focalTweet.tweetId)\n\n    val (intermediates, root) = parentTweets.splitAt(parentTweets.size - numberOfRootTweets)\n    val truncatedIntermediates =\n      intermediates.take(maxModuleSize - maxAncestorsInConversation).reverse\n    root ++ truncatedIntermediates :+ focalTweetWithConversationMetadata\n  }\n\n  /**\n   * Ancestor store truncates at 256 ancestors. For very large reply threads, we try best effort\n   * to append the root tweet to the ancestor list based on the conversationId and\n   * conversationRootAuthorId. When rendering conversation modules, we can display the root tweet\n   * instead of the 256th highest ancestor.\n   */\n  private def getTruncatedRootTweet(\n    ancestors: ta.TweetAncestors,\n    focalTweetId: Long\n  ): Option[TweetWithConversationMetadata] = {\n    ancestors.conversationRootAuthorId.collect {\n      case rootAuthorId\n          if ancestors.state == ta.ReplyState.Partial &&\n            ancestors.ancestors.last.tweetId != ancestors.conversationId =>\n        TweetWithConversationMetadata(\n          tweetId = ancestors.conversationId,\n          userId = Some(rootAuthorId),\n          sourceTweetId = None,\n          sourceUserId = None,\n          inReplyToTweetId = None,\n          conversationId = Some(focalTweetId),\n          ancestors = Seq.empty\n        )\n    }\n  }\n\n  private def dedupeCandidates(\n    inputTweetsWithConversationMetadata: Seq[TweetWithConversationMetadata],\n    ancestors: Seq[TweetWithConversationMetadata]\n  ): Seq[TweetWithConversationMetadata] = {\n    val dedupedAncestors: Iterable[TweetWithConversationMetadata] = ancestors\n      .groupBy(_.tweetId).map {\n        case (_, duplicateAncestors)\n            if duplicateAncestors.size > maxTweetsInConversationWithSameId =>\n          duplicateAncestors.maxBy(_.conversationId.getOrElse(0L))\n        case (_, nonDuplicateAncestors) => nonDuplicateAncestors.head\n      }\n    // Sort by tweet id to prevent issues with future assumptions of the root being the first\n    // tweet and the focal being the last tweet in a module. The tweets as a whole do not need\n    // to be sorted overall, only the relative order within modules must be kept.\n    val sortedDedupedAncestors: Seq[TweetWithConversationMetadata] =\n      dedupedAncestors.toSeq.sortBy(_.tweetId)\n\n    val ancestorIds = sortedDedupedAncestors.map(_.tweetId).toSet\n    val updatedCandidates = inputTweetsWithConversationMetadata.filterNot { candidate =>\n      ancestorIds.contains(candidate.tweetId)\n    }\n    sortedDedupedAncestors ++ updatedCandidates\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc/ConversationServiceResponseFeatureTransformer.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc\n\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.timelineservice.suggests.thriftscala.SuggestType\n\nobject AuthorIdFeature extends Feature[TweetCandidate, Option[Long]]\nobject AncestorIdsFeature extends Feature[TweetCandidate, Seq[Long]]\nobject ConversationModuleFocalTweetIdFeature extends Feature[TweetCandidate, Option[Long]]\nobject InReplyToFeature extends Feature[TweetCandidate, Option[Long]]\nobject IsRetweetFeature extends Feature[TweetCandidate, Boolean]\nobject SourceTweetIdFeature extends Feature[TweetCandidate, Option[Long]]\nobject SourceUserIdFeature extends Feature[TweetCandidate, Option[Long]]\nobject SuggestTypeFeature extends Feature[TweetCandidate, Option[SuggestType]]\n\nobject ConversationServiceResponseFeatureTransformer\n    extends CandidateFeatureTransformer[TweetWithConversationMetadata] {\n  override val identifier: TransformerIdentifier =\n    TransformerIdentifier(\"ConversationServiceResponse\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(\n      AuthorIdFeature,\n      InReplyToFeature,\n      IsRetweetFeature,\n      SourceTweetIdFeature,\n      SourceUserIdFeature,\n      ConversationModuleFocalTweetIdFeature,\n      AncestorIdsFeature,\n      SuggestTypeFeature\n    )\n\n  override def transform(candidate: TweetWithConversationMetadata): FeatureMap = {\n    FeatureMapBuilder()\n      .add(AuthorIdFeature, candidate.userId)\n      .add(InReplyToFeature, candidate.inReplyToTweetId)\n      .add(IsRetweetFeature, candidate.sourceTweetId.isDefined)\n      .add(SourceTweetIdFeature, candidate.sourceTweetId)\n      .add(SourceUserIdFeature, candidate.sourceUserId)\n      .add(ConversationModuleFocalTweetIdFeature, candidate.conversationId)\n      .add(AncestorIdsFeature, candidate.ancestors.map(_.tweetId))\n      .add(SuggestTypeFeature, Some(SuggestType.OrganicConversation))\n      .build()\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/tweetconvosvc/DropMaxConversationModuleItemCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.candidate_source.tweetconvosvc\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Takes a conversation module item and truncates it to be at most the focal tweet, the focal tweet's\n * in reply to tweet and optionally, the root conversation tweet if desired.\n * @param pipelineScope What pipeline scopes to include in this.\n * @param includeRootTweet Whether to include the root tweet at the top of the conversation or not.\n * @tparam Query\n */\ncase class DropMaxConversationModuleItemCandidates[-Query <: PipelineQuery](\n  override val pipelineScope: CandidateScope,\n  includeRootTweet: Boolean)\n    extends Selector[Query] {\n  override def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val updatedCandidates = remainingCandidates.collect {\n      case moduleCandidate: ModuleCandidateWithDetails if pipelineScope.contains(moduleCandidate) =>\n        updateConversationModule(moduleCandidate, includeRootTweet)\n      case candidates => candidates\n    }\n    SelectorResult(remainingCandidates = updatedCandidates, result = result)\n  }\n\n  private def updateConversationModule(\n    module: ModuleCandidateWithDetails,\n    includeRootTweet: Boolean\n  ): ModuleCandidateWithDetails = {\n    // If the thread is only the root tweet & a focal tweet replying to it, no truncation can be done.\n    if (module.candidates.length <= 2) {\n      module\n    } else {\n      // If a thread is more 3 or more tweets, we optionally keep the root tweet if desired, and take\n      // the focal tweet tweet and its direct ancestor (the one it would have replied to) and return\n      // those.\n      val tweetCandidates = module.candidates\n      val replyAndFocalTweet = tweetCandidates.takeRight(2)\n      val updatedConversation = if (includeRootTweet) {\n        tweetCandidates.headOption ++ replyAndFocalTweet\n      } else {\n        replyAndFocalTweet\n      }\n      module.copy(candidates = updatedConversation.toSeq)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/slice/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/slice/builder\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/slice/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/slice/builder\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/slice/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/slice/SliceItemCandidateDecorator.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.slice\n\nimport com.twitter.product_mixer.component_library.model.candidate.CursorCandidate\nimport com.twitter.product_mixer.component_library.model.presentation.slice.SliceItemPresentation\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.DecoratorIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.functional_component.decorator.slice.builder.CandidateSliceItemBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.stitch.Stitch\n\n/**\n * Adds a [[Decoration]] for all `candidates` that are [[CursorCandidate]]s\n *\n * @note Only [[CursorCandidate]]s get decorated in [[SliceItemCandidateDecorator]]\n *       because the [[com.twitter.product_mixer.component_library.premarshaller.slice.SliceDomainMarshaller]]\n *       handles the undecorated non-[[CursorCandidate]] `candidates` directly.\n */\ncase class SliceItemCandidateDecorator[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n  cursorBuilder: CandidateSliceItemBuilder[Query, CursorCandidate, CursorItem],\n  override val identifier: DecoratorIdentifier = DecoratorIdentifier(\"SliceItemCandidate\"))\n    extends CandidateDecorator[Query, Candidate] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[Seq[Decoration]] = {\n    val cursorPresentations = candidates.collect {\n      case CandidateWithFeatures(candidate: CursorCandidate, features) =>\n        val cursorItem = cursorBuilder(query, candidate, features)\n        val presentation = SliceItemPresentation(sliceItem = cursorItem)\n\n        Decoration(candidate, presentation)\n    }\n\n    Stitch.value(cursorPresentations)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/slice/builder/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/slice/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/slice/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/slice/builder/CursorCandidateSliceItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.slice.builder\n\nimport com.twitter.product_mixer.component_library.model.candidate.CursorCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.{\n  NextCursor => CursorCandidateNextCursor\n}\nimport com.twitter.product_mixer.component_library.model.candidate.{\n  PreviousCursor => CursorCandidatePreviousCursor\n}\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorItem\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.NextCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.PreviousCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.functional_component.decorator.slice.builder.CandidateSliceItemBuilder\n\ncase class CursorCandidateSliceItemBuilder()\n    extends CandidateSliceItemBuilder[PipelineQuery, CursorCandidate, CursorItem] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: CursorCandidate,\n    featureMap: FeatureMap\n  ): CursorItem =\n    candidate.cursorType match {\n      case CursorCandidateNextCursor => CursorItem(candidate.value, NextCursor)\n      case CursorCandidatePreviousCursor => CursorItem(candidate.value, PreviousCursor)\n    }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/BUILD",
    "content": "scala_library(\n    name = \"urt\",\n    sources = [\"**/*.scala\"] + exclude_globs([\"builder/richtext/*.scala\"]),\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    scalac_plugins = [\"no-roomba\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":richtext\",\n        \"3rdparty/jvm/com/twitter/bijection:json\",\n        \"3rdparty/jvm/com/twitter/bijection:scrooge\",\n        \"explore/explore-mixer/server/src/main/scala/com/twitter/explore_mixer/model/request\",\n        \"interests-mixer/server/src/main/scala/com/twitter/interests_mixer/model/request\",\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/people_discovery\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/suggestion\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/trends_events\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/cover\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/icon\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/thrift/com/twitter/ads/adserver:ad_metadata_container-scala\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_common-scala\",\n        \"src/thrift/com/twitter/hermit:hermit-scala\",\n        \"src/thrift/com/twitter/suggests/controller_data:controller_data-scala\",\n        \"src/thrift/com/twitter/timelines/service:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala\",\n        \"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala\",\n        \"stringcenter/client\",\n        \"stringcenter/client/src/main/java\",\n        \"stringcenter/client/src/main/scala/com/twitter/stringcenter/client\",\n        \"timelines/src/main/scala/com/twitter/timelines/util\",\n        \"trends/trending_content/src/main/scala/com/twitter/trends/trending_content/util:compacting-number-localizer\",\n        \"tweetconvosvc/common/src/main/thrift/com/twitter/tweetconvosvc/tweet_ancestor:thrift-scala\",\n        \"twitter-text/lib/java/src/main/java/com/twitter/twittertext\",\n    ],\n    exports = [\n        \":richtext\",\n        \"3rdparty/jvm/com/twitter/bijection:json\",\n        \"3rdparty/jvm/com/twitter/bijection:scrooge\",\n        \"explore/explore-mixer/server/src/main/scala/com/twitter/explore_mixer/model/request\",\n        \"interests-mixer/server/src/main/scala/com/twitter/interests_mixer/model/request\",\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/suggestion\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/cover\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/icon\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/thrift/com/twitter/ads/adserver:ad_metadata_container-scala\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_common-scala\",\n        \"src/thrift/com/twitter/suggests/controller_data:controller_data-scala\",\n        \"src/thrift/com/twitter/timelines/service:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/common/scoredtweetcandidate:thrift-scala\",\n        \"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala\",\n        \"stringcenter/client\",\n        \"stringcenter/client/src/main/java\",\n        \"stringcenter/client/src/main/scala/com/twitter/stringcenter/client\",\n        \"timelines/src/main/scala/com/twitter/timelines/util\",\n        \"tweetconvosvc/common/src/main/thrift/com/twitter/tweetconvosvc/tweet_ancestor:thrift-scala\",\n        \"twitter-text/lib/java/src/main/java/com/twitter/twittertext\",\n    ],\n)\n\nscala_library(\n    name = \"richtext\",\n    sources = [\"builder/richtext/*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext\",\n        \"twitter-text/lib/java/src/main/java/com/twitter/twittertext\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext\",\n        \"twitter-text/lib/java/src/main/java/com/twitter/twittertext\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/UrtConversationItemCandidateDecorator.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.component_library.model.presentation.urt.ConversationModuleItem\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtItemPresentation\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.DecoratorIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ModuleItemTreeDisplay\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\ncase class UrtConversationItemCandidateDecorator[\n  Query <: PipelineQuery,\n  Candidate <: BaseTweetCandidate\n](\n  tweetCandidateUrtItemBuilder: TweetCandidateUrtItemBuilder[Query, Candidate],\n  override val identifier: DecoratorIdentifier = DecoratorIdentifier(\"UrtConversationItem\"))\n    extends CandidateDecorator[Query, Candidate] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[Seq[Decoration]] = {\n    val candidatePresentations = candidates.view.zipWithIndex.map {\n      case (candidate, index) =>\n        val itemPresentation = new UrtItemPresentation(\n          timelineItem = tweetCandidateUrtItemBuilder(\n            pipelineQuery = query,\n            tweetCandidate = candidate.candidate,\n            candidateFeatures = candidate.features)\n        ) with ConversationModuleItem {\n          override val treeDisplay: Option[ModuleItemTreeDisplay] = None\n          override val dispensable: Boolean = index < candidates.length - 1\n        }\n\n        Decoration(candidate.candidate, itemPresentation)\n    }\n\n    Stitch.value(candidatePresentations)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/UrtItemCandidateDecorator.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt\n\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtItemPresentation\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.DecoratorIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.stitch.Stitch\n\n/**\n * Decorator that will apply the provided [[CandidateUrtEntryBuilder]] to each candidate independently to make a [[TimelineItem]]\n */\ncase class UrtItemCandidateDecorator[\n  Query <: PipelineQuery,\n  BuilderInput <: UniversalNoun[Any],\n  BuilderOutput <: TimelineItem\n](\n  builder: CandidateUrtEntryBuilder[Query, BuilderInput, BuilderOutput],\n  override val identifier: DecoratorIdentifier = DecoratorIdentifier(\"UrtItemCandidate\"))\n    extends CandidateDecorator[Query, BuilderInput] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[BuilderInput]]\n  ): Stitch[Seq[Decoration]] = {\n    val candidatePresentations = candidates.map { candidate =>\n      val itemPresentation = UrtItemPresentation(\n        timelineItem = builder(query, candidate.candidate, candidate.features)\n      )\n\n      Decoration(candidate.candidate, itemPresentation)\n    }\n\n    Stitch.value(candidatePresentations)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/UrtItemInModuleDecorator.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt\n\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtItemPresentation\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtModulePresentation\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.DecoratorIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseTimelineModuleBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.stitch.Stitch\n\n/**\n * Decorator that will apply the provided [[urtItemCandidateDecorator]] to all the `candidates` and apply\n * the same [[UrtModulePresentation]] from [[moduleBuilder]] to each Candidate.\n */\ncase class UrtItemInModuleDecorator[\n  Query <: PipelineQuery,\n  BuilderInput <: UniversalNoun[Any],\n  BuilderOutput <: TimelineItem\n](\n  urtItemCandidateDecorator: CandidateDecorator[Query, BuilderInput],\n  moduleBuilder: BaseTimelineModuleBuilder[Query, BuilderInput],\n  override val identifier: DecoratorIdentifier = DecoratorIdentifier(\"UrtItemInModule\"))\n    extends CandidateDecorator[Query, BuilderInput] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[BuilderInput]]\n  ): Stitch[Seq[Decoration]] = {\n    if (candidates.nonEmpty) {\n      val urtItemCandidatesWithDecoration = urtItemCandidateDecorator(query, candidates)\n\n      // Pass candidates to support when the module is constructed dynamically based on the list\n      val modulePresentation =\n        UrtModulePresentation(moduleBuilder(query, candidates))\n\n      urtItemCandidatesWithDecoration.map { candidates =>\n        candidates.collect {\n          case Decoration(candidate, urtItemPresentation: UrtItemPresentation) =>\n            Decoration(\n              candidate,\n              urtItemPresentation.copy(modulePresentation = Some(modulePresentation)))\n        }\n      }\n    } else {\n      Stitch.Nil\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/UrtMultipleModulesDecorator.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt\n\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtItemPresentation\nimport com.twitter.product_mixer.component_library.model.presentation.urt.UrtModulePresentation\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.DecoratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.stitch.Stitch\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleIdGeneration\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.AutomaticUniqueModuleId\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseTimelineModuleBuilder\n\n/**\n * Given a [[CandidateWithFeatures]] return the corresponding group with which it should be\n * associated. Returning none will result in the candidate not being assigned to any module.\n */\ntrait GroupByKey[-Query <: PipelineQuery, -BuilderInput <: UniversalNoun[Any], Key] {\n  def apply(query: Query, candidate: BuilderInput, candidateFeatures: FeatureMap): Option[Key]\n}\n\n/**\n * Similar to [[UrtItemInModuleDecorator]] except that this decorator can assign items to different\n * modules based on the provided [[GroupByKey]].\n *\n * @param urtItemCandidateDecorator decorates individual item candidates\n * @param moduleBuilder builds a module from a particular candidate group\n * @param groupByKey assigns each candidate a module group. Returning [[None]] will result in the\n *                   candidate not being assigned to a module\n */\ncase class UrtMultipleModulesDecorator[\n  -Query <: PipelineQuery,\n  -BuilderInput <: UniversalNoun[Any],\n  GroupKey\n](\n  urtItemCandidateDecorator: CandidateDecorator[Query, BuilderInput],\n  moduleBuilder: BaseTimelineModuleBuilder[Query, BuilderInput],\n  groupByKey: GroupByKey[Query, BuilderInput, GroupKey],\n  override val identifier: DecoratorIdentifier = DecoratorIdentifier(\"UrtMultipleModules\"))\n    extends CandidateDecorator[Query, BuilderInput] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[BuilderInput]]\n  ): Stitch[Seq[Decoration]] = {\n    if (candidates.nonEmpty) {\n\n      /** Individual candidates with [[UrtItemPresentation]]s */\n      val decoratedCandidatesStitch: Stitch[\n        Seq[(CandidateWithFeatures[BuilderInput], Decoration)]\n      ] = urtItemCandidateDecorator(query, candidates).map(candidates.zip(_))\n\n      decoratedCandidatesStitch.map { decoratedCandidates =>\n        // Group candidates into modules\n        val candidatesByModule: Map[Option[GroupKey], Seq[\n          (CandidateWithFeatures[BuilderInput], Decoration)\n        ]] =\n          decoratedCandidates.groupBy {\n            case (CandidateWithFeatures(candidate, features), _) =>\n              groupByKey(query, candidate, features)\n          }\n\n        candidatesByModule.iterator.zipWithIndex.flatMap {\n\n          // A None group key indicates these candidates should not be put into a module. Return\n          // the decorated candidates.\n          case ((None, candidateGroup), _) =>\n            candidateGroup.map {\n              case (_, decoration) => decoration\n            }\n\n          // Build a UrtModulePresentation and add it to each candidate's decoration.\n          case ((_, candidateGroup), index) =>\n            val (candidatesWithFeatures, decorations) = candidateGroup.unzip\n\n            /**\n             * Build the module and update its ID if [[AutomaticUniqueModuleId]]s are being used.\n             * Forcing IDs to be different ensures that modules are never accidentally grouped\n             * together, since all other fields might otherwise be equal (candidates aren't added\n             * to modules until the domain marshalling phase).\n             */\n            val timelineModule = {\n              val module = moduleBuilder(query, candidatesWithFeatures)\n\n              ModuleIdGeneration(module.id) match {\n                case id: AutomaticUniqueModuleId => module.copy(id = id.withOffset(index).moduleId)\n                case _ => module\n              }\n            }\n\n            val modulePresentation = UrtModulePresentation(timelineModule)\n\n            decorations.collect {\n              case Decoration(candidate, urtItemPresentation: UrtItemPresentation) =>\n                Decoration(\n                  candidate,\n                  urtItemPresentation.copy(modulePresentation = Some(modulePresentation)))\n            }\n        }.toSeq\n      }\n    } else {\n      Stitch.Nil\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/contextual_ref/ContextualTweetRefBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.ContextualTweetRef\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.TweetHydrationContext\n\ncase class ContextualTweetRefBuilder[-Candidate <: BaseTweetCandidate](\n  tweetHydrationContext: TweetHydrationContext) {\n\n  def apply(candidate: Candidate): Option[ContextualTweetRef] =\n    Some(ContextualTweetRef(candidate.id, Some(tweetHydrationContext)))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/conversations/ConversationModuleMetadataBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.conversations\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleMetadataBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleConversationMetadata\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleMetadata\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class ConversationModuleMetadataBuilder[\n  Query <: PipelineQuery,\n  Candidate <: BaseTweetCandidate\n](\n  ancestorIdsFeature: Feature[_, Seq[Long]],\n  allIdsOrdering: Ordering[Long])\n    extends BaseModuleMetadataBuilder[Query, Candidate] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): ModuleMetadata = {\n\n    val ancestors = candidates.last.features.getOrElse(ancestorIdsFeature, Seq.empty)\n    val sortedAllTweetIds = (candidates.last.candidate.id +: ancestors).sorted(allIdsOrdering)\n\n    ModuleMetadata(\n      adsMetadata = None,\n      conversationMetadata = Some(\n        ModuleConversationMetadata(\n          allTweetIds = Some(sortedAllTweetIds),\n          socialContext = None,\n          enableDeduplication = Some(true)\n        )),\n      gridCarouselMetadata = None\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/flexible_injection_pipeline/FlipPromptCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline\n\nimport com.twitter.onboarding.injections.thriftscala.Injection\nimport com.twitter.onboarding.injections.{thriftscala => onboardingthrift}\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline.OnboardingInjectionConversions._\nimport com.twitter.product_mixer.component_library.model.candidate.BasePromptCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.FlipPromptCarouselTileFeature\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.FlipPromptInjectionsFeature\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.FlipPromptOffsetInModuleFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverFullCoverDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverHalfCoverDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.FullCover\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.FullCoverContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.HalfCover\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.HalfCoverContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.HeaderImagePromptMessageContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.InlinePromptMessageContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessagePromptItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.PromptItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject FlipPromptCandidateUrtItemBuilder {\n  val FlipPromptClientEventInfoElement: String = \"flip-prompt-message\"\n}\n\ncase class FlipPromptCandidateUrtItemBuilder[-Query <: PipelineQuery]()\n    extends CandidateUrtEntryBuilder[Query, BasePromptCandidate[Any], TimelineItem] {\n\n  override def apply(\n    query: Query,\n    promptCandidate: BasePromptCandidate[Any],\n    candidateFeatures: FeatureMap\n  ): TimelineItem = {\n    val injection = candidateFeatures.get(FlipPromptInjectionsFeature)\n\n    injection match {\n      case onboardingthrift.Injection.InlinePrompt(candidate) =>\n        MessagePromptItem(\n          id = promptCandidate.id.toString,\n          sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n          clientEventInfo = buildClientEventInfo(injection),\n          feedbackActionInfo = candidate.feedbackInfo.map(convertFeedbackInfo),\n          isPinned = Some(candidate.isPinnedEntry),\n          content = getInlinePromptMessageContent(candidate),\n          impressionCallbacks = candidate.impressionCallbacks.map(_.map(convertCallback).toList)\n        )\n      case onboardingthrift.Injection.FullCover(candidate) =>\n        FullCover(\n          id = promptCandidate.id.toString,\n          // Note that sort index is not used for Covers, as they are not TimelineEntry and do not have entryId\n          sortIndex = None,\n          clientEventInfo =\n            Some(OnboardingInjectionConversions.convertClientEventInfo(candidate.clientEventInfo)),\n          content = getFullCoverContent(candidate)\n        )\n      case onboardingthrift.Injection.HalfCover(candidate) =>\n        HalfCover(\n          id = promptCandidate.id.toString,\n          // Note that sort index is not used for Covers, as they are not TimelineEntry and do not have entryId\n          sortIndex = None,\n          clientEventInfo =\n            Some(OnboardingInjectionConversions.convertClientEventInfo(candidate.clientEventInfo)),\n          content = getHalfCoverContent(candidate)\n        )\n      case Injection.TilesCarousel(_) =>\n        val offsetInModuleOption =\n          candidateFeatures.get(FlipPromptOffsetInModuleFeature)\n        val offsetInModule =\n          offsetInModuleOption.getOrElse(throw FlipPromptOffsetInModuleMissing)\n        val tileOption =\n          candidateFeatures.get(FlipPromptCarouselTileFeature)\n        val tile = tileOption.getOrElse(throw FlipPromptCarouselTileMissing)\n        TilesCarouselConversions.convertTile(tile, offsetInModule)\n      case onboardingthrift.Injection.RelevancePrompt(candidate) =>\n        PromptItem(\n          id = promptCandidate.id.toString,\n          sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n          clientEventInfo = buildClientEventInfo(injection),\n          content = RelevancePromptConversions.convertContent(candidate),\n          impressionCallbacks = Some(candidate.impressionCallbacks.map(convertCallback).toList)\n        )\n      case _ => throw new UnsupportedFlipPromptException(injection)\n    }\n  }\n\n  private def getInlinePromptMessageContent(\n    candidate: onboardingthrift.InlinePrompt\n  ): MessageContent = {\n    candidate.image match {\n      case Some(image) =>\n        HeaderImagePromptMessageContent(\n          headerImage = convertImage(image),\n          headerText = Some(candidate.headerText.text),\n          bodyText = candidate.bodyText.map(_.text),\n          primaryButtonAction = candidate.primaryAction.map(convertButtonAction),\n          secondaryButtonAction = candidate.secondaryAction.map(convertButtonAction),\n          headerRichText = Some(convertRichText(candidate.headerText)),\n          bodyRichText = candidate.bodyText.map(convertRichText),\n          action =\n            None\n        )\n      case None =>\n        InlinePromptMessageContent(\n          headerText = candidate.headerText.text,\n          bodyText = candidate.bodyText.map(_.text),\n          primaryButtonAction = candidate.primaryAction.map(convertButtonAction),\n          secondaryButtonAction = candidate.secondaryAction.map(convertButtonAction),\n          headerRichText = Some(convertRichText(candidate.headerText)),\n          bodyRichText = candidate.bodyText.map(convertRichText),\n          socialContext = candidate.socialContext.map(convertSocialContext),\n          userFacepile = candidate.promptUserFacepile.map(convertUserFacePile)\n        )\n    }\n  }\n\n  private def getFullCoverContent(\n    candidate: onboardingthrift.FullCover\n  ): FullCoverContent =\n    FullCoverContent(\n      displayType = CoverFullCoverDisplayType,\n      primaryText = convertRichText(candidate.primaryText),\n      primaryCoverCta = convertCoverCta(candidate.primaryButtonAction),\n      secondaryCoverCta = candidate.secondaryButtonAction.map(convertCoverCta),\n      secondaryText = candidate.secondaryText.map(convertRichText),\n      imageVariant = candidate.image.map(img => convertImageVariant(img.image)),\n      details = candidate.detailText.map(convertRichText),\n      dismissInfo = candidate.dismissInfo.map(convertDismissInfo),\n      imageDisplayType = candidate.image.map(img => convertImageDisplayType(img.imageDisplayType)),\n      impressionCallbacks = candidate.impressionCallbacks.map(_.map(convertCallback).toList)\n    )\n\n  private def getHalfCoverContent(\n    candidate: onboardingthrift.HalfCover\n  ): HalfCoverContent =\n    HalfCoverContent(\n      displayType =\n        candidate.displayType.map(convertHalfCoverDisplayType).getOrElse(CoverHalfCoverDisplayType),\n      primaryText = convertRichText(candidate.primaryText),\n      primaryCoverCta = convertCoverCta(candidate.primaryButtonAction),\n      secondaryCoverCta = candidate.secondaryButtonAction.map(convertCoverCta),\n      secondaryText = candidate.secondaryText.map(convertRichText),\n      coverImage = candidate.image.map(convertCoverImage),\n      dismissible = candidate.dismissible,\n      dismissInfo = candidate.dismissInfo.map(convertDismissInfo),\n      impressionCallbacks = candidate.impressionCallbacks.map(_.map(convertCallback).toList)\n    )\n\n  private def buildClientEventInfo(\n    injection: Injection\n  ): Option[ClientEventInfo] = {\n    injection match {\n      //To keep parity between TimelineMixer and Product Mixer, inline prompt switches sets the prompt product identifier as the component and no element. Also includes clientEventDetails\n      case onboardingthrift.Injection.InlinePrompt(candidate) =>\n        val clientEventDetails: ClientEventDetails =\n          ClientEventDetails(\n            conversationDetails = None,\n            timelinesDetails = Some(TimelinesDetails(injectionType = Some(\"Message\"), None, None)),\n            articleDetails = None,\n            liveEventDetails = None,\n            commerceDetails = None\n          )\n        Some(\n          ClientEventInfo(\n            component = candidate.injectionIdentifier,\n            element = None,\n            details = Some(clientEventDetails),\n            action = None,\n            entityToken = None))\n      // To keep parity between TLM and PM we swap component and elements.\n      case onboardingthrift.Injection.RelevancePrompt(candidate) =>\n        Some(\n          ClientEventInfo(\n            // Identifier is prefixed with onboarding per TLM\n            component = Some(\"onboarding_\" + candidate.injectionIdentifier),\n            element = Some(\"relevance_prompt\"),\n            details = None,\n            action = None,\n            entityToken = None\n          ))\n\n      case _ => None\n    }\n  }\n\n}\n\nclass UnsupportedFlipPromptException(injection: onboardingthrift.Injection)\n    extends UnsupportedOperationException(\n      \"Unsupported timeline item \" + TransportMarshaller.getSimpleName(injection.getClass))\n\nobject FlipPromptOffsetInModuleMissing\n    extends NoSuchElementException(\n      \"FlipPromptOffsetInModuleFeature must be set for the TilesCarousel FLIP injection in PromptCandidateSource\")\n\nobject FlipPromptCarouselTileMissing\n    extends NoSuchElementException(\n      \"FlipPromptCarouselTileFeature must be set for the TilesCarousel FLIP injection in PromptCandidateSource\")\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/flexible_injection_pipeline/FlipPromptModuleGrouping.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline\n\nimport com.twitter.product_mixer.component_library.decorator.urt.GroupByKey\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.FlipPromptInjectionsFeature\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.FlipPromptOffsetInModuleFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject FlipPromptModuleGrouping extends GroupByKey[PipelineQuery, UniversalNoun[Any], Int] {\n  override def apply(\n    query: PipelineQuery,\n    candidate: UniversalNoun[Any],\n    candidateFeatures: FeatureMap\n  ): Option[Int] = {\n    val injection = candidateFeatures.get(FlipPromptInjectionsFeature)\n    val offsetInModule = candidateFeatures.getOrElse(FlipPromptOffsetInModuleFeature, None)\n\n    // We return None for any candidate that doesn't have an offsetInModule, so that they are left as independent items.\n    // Otherwise, we return a hash of the injection instance which will be used to aggregate candidates with matching values into a module.\n    offsetInModule.map(_ => injection.hashCode())\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/flexible_injection_pipeline/FlipPromptUrtModuleBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline\n\nimport com.twitter.onboarding.injections.thriftscala.Injection\nimport com.twitter.onboarding.injections.{thriftscala => onboardingthrift}\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.AutomaticUniqueModuleId\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleIdGeneration\nimport com.twitter.product_mixer.component_library.model.candidate.BasePromptCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.FlipPromptInjectionsFeature\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseTimelineModuleBuilder\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Carousel\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class FlipPromptUrtModuleBuilder[-Query <: PipelineQuery](\n  moduleIdGeneration: ModuleIdGeneration = AutomaticUniqueModuleId())\n    extends BaseTimelineModuleBuilder[Query, BasePromptCandidate[Any]] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[BasePromptCandidate[Any]]]\n  ): TimelineModule = {\n    val firstCandidate = candidates.head\n    val injection = firstCandidate.features.get(FlipPromptInjectionsFeature)\n    injection match {\n      case Injection.TilesCarousel(candidate) =>\n        TimelineModule(\n          id = moduleIdGeneration.moduleId,\n          sortIndex = None,\n          entryNamespace = EntryNamespace(\"flip-timeline-module\"),\n          clientEventInfo =\n            Some(OnboardingInjectionConversions.convertClientEventInfo(candidate.clientEventInfo)),\n          feedbackActionInfo =\n            candidate.feedbackInfo.map(OnboardingInjectionConversions.convertFeedbackInfo),\n          isPinned = Some(candidate.isPinnedEntry),\n          // Items are automatically set in the domain marshaller phase\n          items = Seq.empty,\n          displayType = Carousel,\n          header = candidate.header.map(TilesCarouselConversions.convertModuleHeader),\n          footer = None,\n          metadata = None,\n          showMoreBehavior = None\n        )\n      case _ => throw new UnsupportedFlipPromptInModuleException(injection)\n    }\n  }\n}\n\nclass UnsupportedFlipPromptInModuleException(injection: onboardingthrift.Injection)\n    extends UnsupportedOperationException(\n      \"Unsupported timeline item in a Flip prompt module \" + TransportMarshaller.getSimpleName(\n        injection.getClass))\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/flexible_injection_pipeline/OnboardingInjectionConversions.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline\n\nimport com.twitter.onboarding.injections.{thriftscala => onboardingthrift}\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CenterCoverHalfCoverDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverBehaviorDismiss\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverBehaviorNavigate\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverCta\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverCtaBehavior\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverHalfCoverDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverImage\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.HalfCoverDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon._\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.FollowAllMessageActionType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.LargeUserFacepileDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageActionType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageImage\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageTextAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.UserFacepile\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Bounce\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.ButtonStyle\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.Default\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.Primary\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.Secondary\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.Text\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.Destructive\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.Neutral\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.DestructiveSecondary\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.DestructiveText\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DeepLink\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Dismiss\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DismissInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ExternalUrl\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FollowGeneralContextType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.GeneralContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageAnimationType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageVariant\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FullWidth\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Icon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.IconSmall\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.UrtEndpoint\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.UrtEndpointOptions\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Center\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Natural\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Plain\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.ReferenceObject\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextAlignment\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextEntity\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextFormat\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextCashtag\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextHashtag\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextList\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextMention\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextUser\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Strong\n\n/***\n * Helper class to convert onboarding thrift to product-mixer models\n */\nobject OnboardingInjectionConversions {\n\n  def convertFeedbackInfo(\n    feedbackInfo: onboardingthrift.FeedbackInfo\n  ): FeedbackActionInfo = {\n    val actions = feedbackInfo.actions.map {\n      case onboardingthrift.FeedbackAction.DismissAction(dismissAction) =>\n        FeedbackAction(\n          Dismiss,\n          prompt = dismissAction.prompt,\n          confirmation = dismissAction.confirmation,\n          hasUndoAction = dismissAction.hasUndoAction,\n          feedbackUrl = dismissAction.feedbackUrl,\n          childFeedbackActions =\n            None, \n          confirmationDisplayType = None,\n          clientEventInfo = None,\n          icon = None,\n          richBehavior = None,\n          subprompt = None,\n          encodedFeedbackRequest = None\n        )\n      case onboardingthrift.FeedbackAction.UnknownUnionField(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n    }\n\n    FeedbackActionInfo(\n      feedbackActions = actions,\n      feedbackMetadata = None,\n      displayContext = None,\n      clientEventInfo = None)\n  }\n\n  def convertClientEventInfo(input: onboardingthrift.ClientEventInfo): ClientEventInfo =\n    ClientEventInfo(\n      component = input.component,\n      element = input.element,\n      details = None,\n      action = input.action,\n      entityToken = None)\n\n  def convertCallback(callback: onboardingthrift.Callback): Callback =\n    Callback(callback.endpoint)\n\n  def convertImage(image: onboardingthrift.Image): MessageImage =\n    MessageImage(\n      Set(convertImageVariant(image.image)),\n      backgroundColor =\n        None \n    )\n\n  def convertCoverImage(image: onboardingthrift.Image): CoverImage =\n    CoverImage(\n      convertImageVariant(image.image),\n      imageDisplayType = convertImageDisplayType(image.imageDisplayType),\n      imageAnimationType = image.imageAnimationType.map(convertImageAnimationType),\n    )\n\n  def convertImageDisplayType(\n    imageDisplayType: onboardingthrift.ImageDisplayType\n  ): ImageDisplayType =\n    imageDisplayType match {\n      case onboardingthrift.ImageDisplayType.Icon => Icon\n      case onboardingthrift.ImageDisplayType.FullWidth => FullWidth\n      case onboardingthrift.ImageDisplayType.IconSmall => IconSmall\n      case onboardingthrift.ImageDisplayType.EnumUnknownImageDisplayType(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n    }\n\n  private def convertImageAnimationType(\n    imageAnimationType: onboardingthrift.ImageAnimationType\n  ): ImageAnimationType =\n    imageAnimationType match {\n      case onboardingthrift.ImageAnimationType.Bounce => Bounce\n      case onboardingthrift.ImageAnimationType.EnumUnknownImageAnimationType(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n    }\n\n  def convertImageVariant(imageVariant: onboardingthrift.ImageVariant): ImageVariant =\n    ImageVariant(\n      url = imageVariant.url,\n      width = imageVariant.width,\n      height = imageVariant.height,\n      palette = None)\n\n  def convertButtonAction(\n    buttonAction: onboardingthrift.ButtonAction\n  ): MessageTextAction =\n    MessageTextAction(\n      buttonAction.text,\n      MessageAction(\n        dismissOnClick = buttonAction.dismissOnClick.getOrElse(true),\n        url = getActionUrl(buttonAction),\n        clientEventInfo = Some(convertClientEventInfo(buttonAction.clientEventInfo)),\n        onClickCallbacks = buttonAction.callbacks.map(_.map(convertCallback).toList)\n      )\n    )\n\n  private def getActionUrl(buttonAction: onboardingthrift.ButtonAction) =\n    buttonAction.buttonBehavior match {\n      case onboardingthrift.ButtonBehavior.Navigate(navigate) => Some(navigate.url.url)\n      case onboardingthrift.ButtonBehavior.Dismiss(_) => None\n      case onboardingthrift.ButtonBehavior.UnknownUnionField(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n    }\n\n  def convertRichText(\n    richText: com.twitter.onboarding.injections.thriftscala.RichText\n  ): RichText = {\n    val entities = richText.entities.map(entity =>\n      RichTextEntity(\n        entity.fromIndex,\n        entity.toIndex,\n        entity.ref.map(convertRef),\n        entity.format.map(convertFormat)))\n    RichText(\n      text = richText.text,\n      entities = entities.toList,\n      rtl = richText.rtl,\n      alignment = richText.alignment.map(convertAlignment))\n  }\n\n  private def convertAlignment(alignment: onboardingthrift.RichTextAlignment): RichTextAlignment =\n    alignment match {\n      case onboardingthrift.RichTextAlignment.Natural => Natural\n      case onboardingthrift.RichTextAlignment.Center => Center\n      case onboardingthrift.RichTextAlignment.EnumUnknownRichTextAlignment(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n    }\n\n  private def convertRef(ref: onboardingthrift.ReferenceObject): ReferenceObject =\n    ref match {\n      case onboardingthrift.ReferenceObject.User(user) => RichTextUser(user.id)\n      case onboardingthrift.ReferenceObject.Mention(mention) =>\n        RichTextMention(mention.id, mention.screenName)\n      case onboardingthrift.ReferenceObject.Hashtag(hashtag) => RichTextHashtag(hashtag.text)\n\n      case onboardingthrift.ReferenceObject.Cashtag(cashtag) => RichTextCashtag(cashtag.text)\n      case onboardingthrift.ReferenceObject.TwitterList(twList) =>\n        RichTextList(twList.id, twList.url)\n      case onboardingthrift.ReferenceObject.Url(url) => RichTextHashtag(url.url)\n      case onboardingthrift.ReferenceObject.UnknownUnionField(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n    }\n\n  private def convertFormat(format: onboardingthrift.RichTextFormat): RichTextFormat =\n    format match {\n      case onboardingthrift.RichTextFormat.Plain => Plain\n      case onboardingthrift.RichTextFormat.Strong => Strong\n      case onboardingthrift.RichTextFormat.EnumUnknownRichTextFormat(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n    }\n\n  // Specific to Message prompt\n  def convertSocialContext(socialContext: onboardingthrift.RichText): SocialContext =\n    GeneralContext(\n      contextType = FollowGeneralContextType,\n      text = socialContext.text,\n      url = None,\n      contextImageUrls = None,\n      landingUrl = None)\n\n  def convertUserFacePile(\n    userFacepile: onboardingthrift.PromptUserFacepile\n  ): UserFacepile =\n    UserFacepile(\n      userIds = userFacepile.userIds.toList,\n      featuredUserIds = userFacepile.featuredUserIds.toList,\n      action = userFacepile.action.map(convertButtonAction),\n      actionType = userFacepile.actionType.map(convertUserFacePileActionType),\n      displaysFeaturingText = userFacepile.displaysFeaturingText,\n      displayType = Some(LargeUserFacepileDisplayType)\n    )\n\n  private def convertUserFacePileActionType(\n    actionType: onboardingthrift.FacepileActionType\n  ): MessageActionType =\n    actionType match {\n      case onboardingthrift.FacepileActionType.FollowAll => FollowAllMessageActionType\n      case onboardingthrift.FacepileActionType.EnumUnknownFacepileActionType(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n    }\n\n  // Specific to Cover\n\n  def convertHalfCoverDisplayType(\n    displayType: onboardingthrift.HalfCoverDisplayType\n  ): HalfCoverDisplayType =\n    displayType match {\n      case onboardingthrift.HalfCoverDisplayType.Cover => CoverHalfCoverDisplayType\n      case onboardingthrift.HalfCoverDisplayType.CenterCover =>\n        CenterCoverHalfCoverDisplayType\n      case onboardingthrift.HalfCoverDisplayType.EnumUnknownHalfCoverDisplayType(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n    }\n\n  def convertDismissInfo(dismissInfo: onboardingthrift.DismissInfo): DismissInfo =\n    DismissInfo(dismissInfo.callbacks.map(_.map(convertCallback)))\n\n  def convertCoverCta(\n    buttonAction: onboardingthrift.ButtonAction\n  ): CoverCta =\n    CoverCta(\n      buttonAction.text,\n      ctaBehavior = convertCoverCtaBehavior(buttonAction.buttonBehavior),\n      callbacks = buttonAction.callbacks.map(_.map(convertCallback).toList),\n      clientEventInfo = Some(convertClientEventInfo(buttonAction.clientEventInfo)),\n      icon = buttonAction.icon.map(covertHorizonIcon),\n      buttonStyle = buttonAction.buttonStyle.map(covertButtonStyle)\n    )\n\n  private def convertCoverCtaBehavior(\n    behavior: onboardingthrift.ButtonBehavior\n  ): CoverCtaBehavior =\n    behavior match {\n      case onboardingthrift.ButtonBehavior.Navigate(navigate) =>\n        CoverBehaviorNavigate(convertUrl(navigate.url))\n      case onboardingthrift.ButtonBehavior.Dismiss(dismiss) =>\n        CoverBehaviorDismiss(dismiss.feedbackMessage.map(convertRichText))\n      case onboardingthrift.ButtonBehavior.UnknownUnionField(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n    }\n\n  private def covertButtonStyle(bStyle: onboardingthrift.CtaButtonStyle): ButtonStyle =\n    bStyle match {\n      case onboardingthrift.CtaButtonStyle.Default => Default\n      case onboardingthrift.CtaButtonStyle.Primary => Primary\n      case onboardingthrift.CtaButtonStyle.Secondary => Secondary\n      case onboardingthrift.CtaButtonStyle.Text => Text\n      case onboardingthrift.CtaButtonStyle.Destructive => Destructive\n      case onboardingthrift.CtaButtonStyle.Neutral => Neutral\n      case onboardingthrift.CtaButtonStyle.DestructiveSecondary => DestructiveSecondary\n      case onboardingthrift.CtaButtonStyle.DestructiveText => DestructiveText\n      case onboardingthrift.CtaButtonStyle.EnumUnknownCtaButtonStyle(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n    }\n\n  private def covertHorizonIcon(icon: onboardingthrift.HorizonIcon): HorizonIcon =\n    icon match {\n      case onboardingthrift.HorizonIcon.Bookmark => Bookmark\n      case onboardingthrift.HorizonIcon.Moment => Moment\n      case onboardingthrift.HorizonIcon.Debug => Debug\n      case onboardingthrift.HorizonIcon.Error => Error\n      case onboardingthrift.HorizonIcon.Follow => Follow\n      case onboardingthrift.HorizonIcon.Unfollow => Unfollow\n      case onboardingthrift.HorizonIcon.Smile => Smile\n      case onboardingthrift.HorizonIcon.Frown => Frown\n      case onboardingthrift.HorizonIcon.Help => Help\n      case onboardingthrift.HorizonIcon.Link => Link\n      case onboardingthrift.HorizonIcon.Message => Message\n      case onboardingthrift.HorizonIcon.No => No\n      case onboardingthrift.HorizonIcon.Outgoing => Outgoing\n      case onboardingthrift.HorizonIcon.Pin => Pin\n      case onboardingthrift.HorizonIcon.Retweet => Retweet\n      case onboardingthrift.HorizonIcon.Speaker => Speaker\n      case onboardingthrift.HorizonIcon.Trashcan => Trashcan\n      case onboardingthrift.HorizonIcon.Feedback => Feedback\n      case onboardingthrift.HorizonIcon.FeedbackClose => FeedbackClose\n      case onboardingthrift.HorizonIcon.EyeOff => EyeOff\n      case onboardingthrift.HorizonIcon.Moderation => Moderation\n      case onboardingthrift.HorizonIcon.Topic => Topic\n      case onboardingthrift.HorizonIcon.TopicClose => TopicClose\n      case onboardingthrift.HorizonIcon.Flag => Flag\n      case onboardingthrift.HorizonIcon.TopicFilled => TopicFilled\n      case onboardingthrift.HorizonIcon.NotificationsFollow => NotificationsFollow\n      case onboardingthrift.HorizonIcon.Person => Person\n      case onboardingthrift.HorizonIcon.Logo => Logo\n      case onboardingthrift.HorizonIcon.EnumUnknownHorizonIcon(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n\n    }\n\n  def convertUrl(url: onboardingthrift.Url): Url = {\n    val urlType = url.urlType match {\n      case onboardingthrift.UrlType.ExternalUrl => ExternalUrl\n      case onboardingthrift.UrlType.DeepLink => DeepLink\n      case onboardingthrift.UrlType.UrtEndpoint => UrtEndpoint\n      case onboardingthrift.UrlType.EnumUnknownUrlType(value) =>\n        throw new UnsupportedOperationException(s\"Unknown product: $value\")\n    }\n    Url(urlType, url.url, url.urtEndpointOptions.map(convertUrtEndpointOptions))\n  }\n\n  private def convertUrtEndpointOptions(\n    urtEndpointOptions: onboardingthrift.UrtEndpointOptions\n  ): UrtEndpointOptions =\n    UrtEndpointOptions(\n      requestParams = urtEndpointOptions.requestParams.map(_.toMap),\n      title = urtEndpointOptions.title,\n      cacheId = urtEndpointOptions.cacheId,\n      subtitle = urtEndpointOptions.subtitle\n    )\n\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/flexible_injection_pipeline/RelevancePromptConversions.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline\n\nimport com.twitter.onboarding.injections.{thriftscala => onboardingthrift}\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.Compact\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.Large\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.Normal\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.RelevancePromptContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.RelevancePromptDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\n\n/***\n * Helper class to convert Relevance Prompt related onboarding thrift to product-mixer models\n */\nobject RelevancePromptConversions {\n  def convertContent(\n    candidate: onboardingthrift.RelevancePrompt\n  ): RelevancePromptContent =\n    RelevancePromptContent(\n      displayType = convertDisplayType(candidate.displayType),\n      title = candidate.title.text,\n      confirmation = buildConfirmation(candidate),\n      isRelevantText = candidate.isRelevantButton.text,\n      notRelevantText = candidate.notRelevantButton.text,\n      isRelevantCallback = convertCallbacks(candidate.isRelevantButton.callbacks),\n      notRelevantCallback = convertCallbacks(candidate.notRelevantButton.callbacks),\n      isRelevantFollowUp = None, \n      notRelevantFollowUp = None \n    )\n\n  // Based on com.twitter.timelinemixer.injection.model.candidate.OnboardingRelevancePromptDisplayType#fromThrift\n  def convertDisplayType(\n    displayType: onboardingthrift.RelevancePromptDisplayType\n  ): RelevancePromptDisplayType =\n    displayType match {\n      case onboardingthrift.RelevancePromptDisplayType.Normal => Normal\n      case onboardingthrift.RelevancePromptDisplayType.Compact => Compact\n      case onboardingthrift.RelevancePromptDisplayType.Large => Large\n      case onboardingthrift.RelevancePromptDisplayType\n            .EnumUnknownRelevancePromptDisplayType(value) =>\n        throw new UnsupportedOperationException(s\"Unknown display type: $value\")\n    }\n\n  // Based on com.twitter.timelinemixer.injection.model.injection.OnboardingRelevancePromptInjection#buildConfirmation\n  def buildConfirmation(candidate: onboardingthrift.RelevancePrompt): String = {\n    val isRelevantTextConfirmation =\n      buttonToDismissFeedbackText(candidate.isRelevantButton).getOrElse(\"\")\n    val notRelevantTextConfirmation =\n      buttonToDismissFeedbackText(candidate.notRelevantButton).getOrElse(\"\")\n    if (isRelevantTextConfirmation != notRelevantTextConfirmation)\n      throw new IllegalArgumentException(\n        s\"\"\"confirmation text not consistent for two buttons, :\n          isRelevantConfirmation: ${isRelevantTextConfirmation}\n          notRelevantConfirmation: ${notRelevantTextConfirmation}\n        \"\"\"\n      )\n    isRelevantTextConfirmation\n  }\n\n  // Based on com.twitter.timelinemixer.injection.model.candidate.OnboardingInjectionAction#fromThrift\n  def buttonToDismissFeedbackText(button: onboardingthrift.ButtonAction): Option[String] = {\n    button.buttonBehavior match {\n      case onboardingthrift.ButtonBehavior.Dismiss(d) => d.feedbackMessage.map(_.text)\n      case _ => None\n    }\n  }\n\n  // Based on com.twitter.timelinemixer.injection.model.injection.OnboardingRelevancePromptInjection#buildCallback\n  def convertCallbacks(onboardingCallbacks: Option[Seq[onboardingthrift.Callback]]): Callback = {\n    OnboardingInjectionConversions.convertCallback(\n      onboardingCallbacks\n        .flatMap(_.headOption)\n        .getOrElse(\n          throw new NoSuchElementException(s\"Callback must be provided for the Relevance Prompt\")\n        ))\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/flexible_injection_pipeline/TilesCarouselConversions.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline\n\nimport com.twitter.onboarding.injections.{thriftscala => onboardingthrift}\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline.OnboardingInjectionConversions.convertImageVariant\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Classic\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.BlackRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.ClearRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.DeepBlueRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.DeepGrayRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.DeepGreenRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.DeepOrangeRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.DeepPurpleRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.DeepRedRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.DeepYellowRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.FadedBlueRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.FadedGrayRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.FadedGreenRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.FadedOrangeRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.FadedPurpleRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.FadedRedRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.FadedYellowRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.FaintBlueRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.FaintGrayRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.LightBlueRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.LightGrayRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.LightGreenRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.LightOrangeRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.LightPurpleRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.LightRedRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.LightYellowRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.MediumGrayRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.MediumGreenRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.MediumOrangeRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.MediumPurpleRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.MediumRedRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.MediumYellowRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.RosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.TextBlackRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.TextBlueRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.TwitterBlueRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.WhiteRosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile.CallToActionTileContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile.StandardTileContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile.TileItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Badge\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleHeader\n\nobject TilesCarouselConversions {\n  // Tiles Carousel Conversions\n  def convertTile(tile: onboardingthrift.Tile, id: Long): TileItem = {\n    tile.content match {\n      case standard: onboardingthrift.TileContent.Standard =>\n        TileItem(\n          id = id,\n          sortIndex = None,\n          clientEventInfo =\n            Some(OnboardingInjectionConversions.convertClientEventInfo(tile.clientEventInfo)),\n          feedbackActionInfo = None,\n          title = standard.standard.title,\n          supportingText = \"\",\n          url = tile.url.map(OnboardingInjectionConversions.convertUrl),\n          image = tile.image.map(img => convertImageVariant(img.image)),\n          content = StandardTileContent(\n            title = standard.standard.title,\n            supportingText = \"\",\n            badge = standard.standard.badge.map(convertTileBadge)\n          )\n        )\n      case cta: onboardingthrift.TileContent.CallToAction =>\n        TileItem(\n          id = id,\n          sortIndex = None,\n          clientEventInfo =\n            Some(OnboardingInjectionConversions.convertClientEventInfo(tile.clientEventInfo)),\n          feedbackActionInfo = None,\n          title = cta.callToAction.text,\n          supportingText = \"\",\n          url = tile.url.map(OnboardingInjectionConversions.convertUrl),\n          image = None,\n          content = CallToActionTileContent(\n            text = cta.callToAction.text,\n            richText = None,\n            ctaButton = None\n          )\n        )\n      case _ =>\n        throw new UnsupportedTileCarouselConversionException(s\"Tile Content: ${tile.content}\")\n    }\n  }\n\n  private def convertTileBadge(badge: onboardingthrift.Badge): Badge = {\n    Badge(\n      text = badge.text,\n      textColorName = badge.textColor.map(convertRosettaColor),\n      backgroundColorName = badge.backgroundColor.map(convertRosettaColor))\n  }\n\n  def convertModuleHeader(header: onboardingthrift.TilesCarouselHeader): ModuleHeader = {\n    ModuleHeader(header.header, None, None, None, None, Classic)\n  }\n\n  private def convertRosettaColor(color: onboardingthrift.RosettaColor): RosettaColor =\n    color match {\n      case onboardingthrift.RosettaColor.White => WhiteRosettaColor\n      case onboardingthrift.RosettaColor.Black => BlackRosettaColor\n      case onboardingthrift.RosettaColor.Clear => ClearRosettaColor\n      case onboardingthrift.RosettaColor.TextBlack => TextBlackRosettaColor\n      case onboardingthrift.RosettaColor.TextBlue => TextBlueRosettaColor\n\n      case onboardingthrift.RosettaColor.DeepGray => DeepGrayRosettaColor\n      case onboardingthrift.RosettaColor.MediumGray => MediumGrayRosettaColor\n      case onboardingthrift.RosettaColor.LightGray => LightGrayRosettaColor\n      case onboardingthrift.RosettaColor.FadedGray => FadedGrayRosettaColor\n      case onboardingthrift.RosettaColor.FaintGray => FaintGrayRosettaColor\n\n      case onboardingthrift.RosettaColor.DeepOrange => DeepOrangeRosettaColor\n      case onboardingthrift.RosettaColor.MediumOrange => MediumOrangeRosettaColor\n      case onboardingthrift.RosettaColor.LightOrange => LightOrangeRosettaColor\n      case onboardingthrift.RosettaColor.FadedOrange => FadedOrangeRosettaColor\n\n      case onboardingthrift.RosettaColor.DeepYellow => DeepYellowRosettaColor\n      case onboardingthrift.RosettaColor.MediumYellow => MediumYellowRosettaColor\n      case onboardingthrift.RosettaColor.LightYellow => LightYellowRosettaColor\n      case onboardingthrift.RosettaColor.FadedYellow => FadedYellowRosettaColor\n\n      case onboardingthrift.RosettaColor.DeepGreen => DeepGreenRosettaColor\n      case onboardingthrift.RosettaColor.MediumGreen => MediumGreenRosettaColor\n      case onboardingthrift.RosettaColor.LightGreen => LightGreenRosettaColor\n      case onboardingthrift.RosettaColor.FadedGreen => FadedGreenRosettaColor\n\n      case onboardingthrift.RosettaColor.DeepBlue => DeepBlueRosettaColor\n      case onboardingthrift.RosettaColor.TwitterBlue => TwitterBlueRosettaColor\n      case onboardingthrift.RosettaColor.LightBlue => LightBlueRosettaColor\n      case onboardingthrift.RosettaColor.FadedBlue => FadedBlueRosettaColor\n      case onboardingthrift.RosettaColor.FaintBlue => FaintBlueRosettaColor\n\n      case onboardingthrift.RosettaColor.DeepPurple => DeepPurpleRosettaColor\n      case onboardingthrift.RosettaColor.MediumPurple => MediumPurpleRosettaColor\n      case onboardingthrift.RosettaColor.LightPurple => LightPurpleRosettaColor\n      case onboardingthrift.RosettaColor.FadedPurple => FadedPurpleRosettaColor\n\n      case onboardingthrift.RosettaColor.DeepRed => DeepRedRosettaColor\n      case onboardingthrift.RosettaColor.MediumRed => MediumRedRosettaColor\n      case onboardingthrift.RosettaColor.LightRed => LightRedRosettaColor\n      case onboardingthrift.RosettaColor.FadedRed => FadedRedRosettaColor\n      case onboardingthrift.RosettaColor.EnumUnknownRosettaColor(i) =>\n        throw new UnknownThriftEnumException(\"RosettaColor\")\n    }\n  class UnknownThriftEnumException(enumName: String)\n      extends Exception(s\"Unknown Thrift Enum Found: ${enumName}\")\n\n  class UnsupportedTileCarouselConversionException(UnsupportedTileType: String)\n      extends Exception(s\"Unsupported Tile Type Found: ${UnsupportedTileType}\")\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/icon/HorizonIconBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.icon\n\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.icon.BaseHorizonIconBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.HorizonIcon\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class HorizonIconBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  icon: HorizonIcon)\n    extends BaseHorizonIconBuilder[Query, Candidate] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Option[HorizonIcon] = Some(icon)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/ad/AdsCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.ad\n\nimport com.twitter.ads.adserver.{thriftscala => ads}\nimport com.twitter.adserver.{thriftscala => adserver}\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref.ContextualTweetRefBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder.TweetClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsTweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.Tweet\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.AdMetadataContainer\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.Amplify\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.CallToAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.ClickTrackingInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DcmUrlOverrideType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DirectSponsorshipType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DisclaimerIssue\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DisclaimerPolitical\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DisclaimerType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DisclosureType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DynamicPrerollType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.Earned\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.IndirectSponsorshipType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.Issue\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.LiveTvEvent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.Marketplace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.MediaInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.NoDisclosure\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.NoSponsorshipSponsorshipType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.Political\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.Preroll\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PrerollMetadata\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PromotedMetadata\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.SkAdNetworkData\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.SponsorshipType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.UnknownUrlOverrideType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.VideoVariant\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.util.AdMetadataContainerSerializer\nimport com.twitter.timelines.util.PrerollMetadataSerializer\n\n/**\n * [[AdsCandidateUrtItemBuilder]] takes a [[AdsCandidate]] (with a [[Query]] as additional context)\n * and converts it into the Product Mixer URT representation, or throws an error.\n *\n * Currently, the only supported form for URT representation of the [[AdsCandidate]] is a [[Tweet]],\n * but in the future it could be expanded to handle other forms of ads.\n *\n * @param tweetClientEventInfoBuilder Optionally, provide a ClientEventInfoBuilder for Tweets\n *                                    that given an AdsTweetCandidate and element of \"tweet\".\n * @param tweetDisplayType Should be [[EmphasizedPromotedTweet]] on Profile timelines,\n *                         otherwise [[Tweet]]\n */\ncase class AdsCandidateUrtItemBuilder[Query <: PipelineQuery](\n  tweetClientEventInfoBuilder: Option[BaseClientEventInfoBuilder[Query, AdsTweetCandidate]] = None,\n  contextualTweetRefBuilder: Option[ContextualTweetRefBuilder[AdsTweetCandidate]] = None,\n  tweetDisplayType: TweetDisplayType = Tweet)\n    extends CandidateUrtEntryBuilder[Query, AdsCandidate, TimelineItem] {\n\n  override def apply(\n    pipelineQuery: Query,\n    candidate: AdsCandidate,\n    candidateFeatures: FeatureMap\n  ): TimelineItem = {\n    candidate match {\n      case tweetCandidate: AdsTweetCandidate =>\n        TweetItem(\n          id = tweetCandidate.id,\n          entryNamespace = TweetItem.PromotedTweetEntryNamespace,\n          sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n          clientEventInfo = tweetClientEventInfoBuilder.flatMap(\n            _.apply(\n              pipelineQuery,\n              tweetCandidate,\n              candidateFeatures,\n              Some(TweetClientEventInfoElement))),\n          feedbackActionInfo = None,\n          isPinned = None,\n          entryIdToReplace = None,\n          socialContext = None,\n          highlights = None,\n          innerTombstoneInfo = None,\n          timelinesScoreInfo = None,\n          hasModeratedReplies = None,\n          forwardPivot = None,\n          innerForwardPivot = None,\n          conversationAnnotation = None,\n          promotedMetadata = Some(promotedMetadata(tweetCandidate.adImpression)),\n          displayType = tweetDisplayType,\n          contextualTweetRef = contextualTweetRefBuilder.flatMap(_.apply(tweetCandidate)),\n          prerollMetadata = prerollMetadata(tweetCandidate.adImpression),\n          replyBadge = None,\n          destination = None\n        )\n    }\n  }\n\n  private def promotedMetadata(impression: adserver.AdImpression) = {\n    PromotedMetadata(\n      advertiserId = impression.advertiserId,\n      impressionString = impression.impressionString,\n      disclosureType = impression.disclosureType.map(convertDisclosureType),\n      experimentValues = impression.experimentValues.map(_.toMap),\n      promotedTrendId = impression.promotedTrendId.map(_.toLong),\n      promotedTrendName = impression.promotedTrendName,\n      promotedTrendQueryTerm = impression.promotedTrendQueryTerm,\n      promotedTrendDescription = impression.promotedTrendDescription,\n      clickTrackingInfo = impression.clickTrackingInfo.map(convertClickTrackingInfo),\n      adMetadataContainer = adMetadataContainer(impression)\n    )\n  }\n\n  private def convertDisclosureType(\n    disclosureType: adserver.DisclosureType\n  ): DisclosureType = disclosureType match {\n    case adserver.DisclosureType.None => NoDisclosure\n    case adserver.DisclosureType.Political => Political\n    case adserver.DisclosureType.Earned => Earned\n    case adserver.DisclosureType.Issue => Issue\n    case _ => throw new UnsupportedDisclosureTypeException(disclosureType)\n  }\n\n  private def convertClickTrackingInfo(\n    clickTracking: adserver.ClickTrackingInfo\n  ): ClickTrackingInfo = ClickTrackingInfo(\n    urlParams = clickTracking.urlParams.getOrElse(Map.empty),\n    urlOverride = clickTracking.urlOverride,\n    urlOverrideType = clickTracking.urlOverrideType.map {\n      case adserver.UrlOverrideType.Unknown => UnknownUrlOverrideType\n      case adserver.UrlOverrideType.Dcm => DcmUrlOverrideType\n      case _ => throw new UnsupportedClickTrackingInfoException(clickTracking)\n    }\n  )\n\n  private def prerollMetadata(adImpression: adserver.AdImpression): Option[PrerollMetadata] = {\n    adImpression.serializedPrerollMetadata\n      .flatMap(PrerollMetadataSerializer.deserialize).map { metadata =>\n        PrerollMetadata(\n          metadata.preroll.map(convertPreroll),\n          metadata.videoAnalyticsScribePassthrough\n        )\n      }\n  }\n\n  private def adMetadataContainer(\n    adImpression: adserver.AdImpression\n  ): Option[AdMetadataContainer] = {\n    adImpression.serializedAdMetadataContainer\n      .flatMap(AdMetadataContainerSerializer.deserialize).map { container =>\n        AdMetadataContainer(\n          removePromotedAttributionForPreroll = container.removePromotedAttributionForPreroll,\n          sponsorshipCandidate = container.sponsorshipCandidate,\n          sponsorshipOrganization = container.sponsorshipOrganization,\n          sponsorshipOrganizationWebsite = container.sponsorshipOrganizationWebsite,\n          sponsorshipType = container.sponsorshipType.map(convertSponsorshipType),\n          disclaimerType = container.disclaimerType.map(convertDisclaimerType),\n          skAdNetworkDataList = container.skAdNetworkDataList.map(convertSkAdNetworkDataList),\n          unifiedCardOverride = container.unifiedCardOverride\n        )\n      }\n  }\n\n  private def convertSponsorshipType(\n    sponsorshipType: ads.SponsorshipType\n  ): SponsorshipType = sponsorshipType match {\n    case ads.SponsorshipType.Direct => DirectSponsorshipType\n    case ads.SponsorshipType.Indirect => IndirectSponsorshipType\n    case ads.SponsorshipType.NoSponsorship => NoSponsorshipSponsorshipType\n    // Thrift has extras (e.g. Sponsorship4) that are not used in practice\n    case _ => throw new UnsupportedSponsorshipTypeException(sponsorshipType)\n  }\n\n  private def convertDisclaimerType(\n    disclaimerType: ads.DisclaimerType\n  ): DisclaimerType = disclaimerType match {\n    case ads.DisclaimerType.Political => DisclaimerPolitical\n    case ads.DisclaimerType.Issue => DisclaimerIssue\n    case _ => throw new UnsupportedDisclaimerTypeException(disclaimerType)\n  }\n\n  private def convertDynamicPrerollType(\n    dynamicPrerollType: ads.DynamicPrerollType\n  ): DynamicPrerollType =\n    dynamicPrerollType match {\n      case ads.DynamicPrerollType.Amplify => Amplify\n      case ads.DynamicPrerollType.Marketplace => Marketplace\n      case ads.DynamicPrerollType.LiveTvEvent => LiveTvEvent\n      case _ => throw new UnsupportedDynamicPrerollTypeException(dynamicPrerollType)\n    }\n\n  private def convertMediaInfo(mediaInfo: ads.MediaInfo): MediaInfo = {\n    MediaInfo(\n      uuid = mediaInfo.uuid,\n      publisherId = mediaInfo.publisherId,\n      callToAction = mediaInfo.callToAction.map(convertCallToAction),\n      durationMillis = mediaInfo.durationMillis,\n      videoVariants = mediaInfo.videoVariants.map(convertVideoVariants),\n      advertiserName = mediaInfo.advertiserName,\n      renderAdByAdvertiserName = mediaInfo.renderAdByAdvertiserName,\n      advertiserProfileImageUrl = mediaInfo.advertiserProfileImageUrl\n    )\n  }\n\n  private def convertVideoVariants(videoVariants: Seq[ads.VideoVariant]): Seq[VideoVariant] = {\n    videoVariants.map(videoVariant =>\n      VideoVariant(\n        url = videoVariant.url,\n        contentType = videoVariant.contentType,\n        bitrate = videoVariant.bitrate\n      ))\n  }\n\n  private def convertCallToAction(callToAction: ads.CallToAction): CallToAction = {\n    CallToAction(\n      callToActionType = callToAction.callToActionType,\n      url = callToAction.url\n    )\n  }\n\n  private def convertPreroll(\n    preroll: ads.Preroll\n  ): Preroll = {\n    Preroll(\n      preroll.prerollId,\n      preroll.dynamicPrerollType.map(convertDynamicPrerollType),\n      preroll.mediaInfo.map(convertMediaInfo)\n    )\n  }\n\n  private def convertSkAdNetworkDataList(\n    skAdNetworkDataList: Seq[ads.SkAdNetworkData]\n  ): Seq[SkAdNetworkData] = skAdNetworkDataList.map(sdAdNetwork =>\n    SkAdNetworkData(\n      version = sdAdNetwork.version,\n      srcAppId = sdAdNetwork.srcAppId,\n      dstAppId = sdAdNetwork.dstAppId,\n      adNetworkId = sdAdNetwork.adNetworkId,\n      campaignId = sdAdNetwork.campaignId,\n      impressionTimeInMillis = sdAdNetwork.impressionTimeInMillis,\n      nonce = sdAdNetwork.nonce,\n      signature = sdAdNetwork.signature,\n      fidelityType = sdAdNetwork.fidelityType\n    ))\n}\n\nclass UnsupportedClickTrackingInfoException(clickTrackingInfo: adserver.ClickTrackingInfo)\n    extends UnsupportedOperationException(\n      s\"Unsupported ClickTrackingInfo: $clickTrackingInfo\"\n    )\n\nclass UnsupportedDisclaimerTypeException(disclaimerType: ads.DisclaimerType)\n    extends UnsupportedOperationException(\n      s\"Unsupported DisclaimerType: $disclaimerType\"\n    )\n\nclass UnsupportedDisclosureTypeException(disclosureType: adserver.DisclosureType)\n    extends UnsupportedOperationException(\n      s\"Unsupported DisclosureType: $disclosureType\"\n    )\n\nclass UnsupportedDynamicPrerollTypeException(dynamicPrerollType: ads.DynamicPrerollType)\n    extends UnsupportedOperationException(\n      s\"Unsupported DynamicPrerollType: $dynamicPrerollType\"\n    )\n\nclass UnsupportedSponsorshipTypeException(sponsorshipType: ads.SponsorshipType)\n    extends UnsupportedOperationException(\n      s\"Unsupported SponsorshipType: $sponsorshipType\"\n    )\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/alert/DurationParamBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert\n\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseDurationBuilder\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\ncase class DurationParamBuilder(\n  durationParam: Param[Duration])\n    extends BaseDurationBuilder[PipelineQuery] {\n\n  def apply(\n    query: PipelineQuery,\n    candidate: ShowAlertCandidate,\n    features: FeatureMap\n  ): Option[Duration] =\n    Some(query.params(durationParam))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/alert/ShowAlertCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert.ShowAlertCandidateUrtItemBuilder.ShowAlertClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseDurationBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseShowAlertColorConfigurationBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseShowAlertDisplayLocationBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseShowAlertIconDisplayInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseShowAlertNavigationMetadataBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseShowAlertUserIdsBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ShowAlert\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.richtext.BaseRichTextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertType\n\nobject ShowAlertCandidateUrtItemBuilder {\n  val ShowAlertClientEventInfoElement: String = \"showAlert\"\n}\n\ncase class ShowAlertCandidateUrtItemBuilder[-Query <: PipelineQuery](\n  alertType: ShowAlertType,\n  displayLocationBuilder: BaseShowAlertDisplayLocationBuilder[Query],\n  colorConfigBuilder: BaseShowAlertColorConfigurationBuilder[Query],\n  triggerDelayBuilder: Option[BaseDurationBuilder[Query]] = None,\n  displayDurationBuilder: Option[BaseDurationBuilder[Query]] = None,\n  clientEventInfoBuilder: Option[BaseClientEventInfoBuilder[Query, ShowAlertCandidate]] = None,\n  collapseDelayBuilder: Option[BaseDurationBuilder[Query]] = None,\n  userIdsBuilder: Option[BaseShowAlertUserIdsBuilder[Query]] = None,\n  richTextBuilder: Option[BaseRichTextBuilder[Query, ShowAlertCandidate]] = None,\n  iconDisplayInfoBuilder: Option[BaseShowAlertIconDisplayInfoBuilder[Query]] = None,\n  navigationMetadataBuilder: Option[BaseShowAlertNavigationMetadataBuilder[Query]] = None)\n    extends CandidateUrtEntryBuilder[\n      Query,\n      ShowAlertCandidate,\n      ShowAlert\n    ] {\n\n  override def apply(\n    query: Query,\n    candidate: ShowAlertCandidate,\n    features: FeatureMap,\n  ): ShowAlert = ShowAlert(\n    id = candidate.id,\n    sortIndex = None,\n    alertType = alertType,\n    triggerDelay = triggerDelayBuilder.flatMap(_.apply(query, candidate, features)),\n    displayDuration = displayDurationBuilder.flatMap(_.apply(query, candidate, features)),\n    clientEventInfo = clientEventInfoBuilder.flatMap(\n      _.apply(query, candidate, features, Some(ShowAlertClientEventInfoElement))),\n    collapseDelay = collapseDelayBuilder.flatMap(_.apply(query, candidate, features)),\n    userIds = userIdsBuilder.flatMap(_.apply(query, candidate, features)),\n    richText = richTextBuilder.map(_.apply(query, candidate, features)),\n    iconDisplayInfo = iconDisplayInfoBuilder.flatMap(_.apply(query, candidate, features)),\n    displayLocation = displayLocationBuilder(query, candidate, features),\n    colorConfig = colorConfigBuilder(query, candidate, features),\n    navigationMetadata = navigationMetadataBuilder.flatMap(_.apply(query, candidate, features)),\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/alert/StaticShowAlertColorConfigurationBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert\n\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseShowAlertColorConfigurationBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertColorConfiguration\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class StaticShowAlertColorConfigurationBuilder[-Query <: PipelineQuery](\n  configuration: ShowAlertColorConfiguration)\n    extends BaseShowAlertColorConfigurationBuilder[Query] {\n\n  def apply(\n    query: Query,\n    candidate: ShowAlertCandidate,\n    features: FeatureMap\n  ): ShowAlertColorConfiguration = configuration\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/alert/StaticShowAlertDisplayLocationBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert\n\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseShowAlertDisplayLocationBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertDisplayLocation\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class StaticShowAlertDisplayLocationBuilder[-Query <: PipelineQuery](\n  location: ShowAlertDisplayLocation)\n    extends BaseShowAlertDisplayLocationBuilder[Query] {\n\n  def apply(\n    query: Query,\n    candidate: ShowAlertCandidate,\n    features: FeatureMap\n  ): ShowAlertDisplayLocation = location\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/alert/StaticShowAlertIconDisplayInfoBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.alert\n\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert.BaseShowAlertIconDisplayInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertIconDisplayInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class StaticShowAlertIconDisplayInfoBuilder[-Query <: PipelineQuery](\n  iconDisplayInfo: ShowAlertIconDisplayInfo)\n    extends BaseShowAlertIconDisplayInfoBuilder[Query] {\n\n  def apply(\n    query: Query,\n    candidate: ShowAlertCandidate,\n    features: FeatureMap\n  ): Option[ShowAlertIconDisplayInfo] = Some(iconDisplayInfo)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/article/ArticleCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.article\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.article.ArticleCandidateUrtItemBuilder.ArticleClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.BaseArticleCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.ArticleDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.ArticleItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.ArticleSeedType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.FollowingListSeed\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject ArticleCandidateUrtItemBuilder {\n  val ArticleClientEventInfoElement: String = \"article\"\n}\n\ncase class ArticleCandidateUrtItemBuilder[\n  -Query <: PipelineQuery,\n  Candidate <: BaseArticleCandidate\n](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, Candidate],\n  articleSeedType: ArticleSeedType = FollowingListSeed,\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, Candidate]\n  ] = None,\n  displayType: Option[ArticleDisplayType] = None,\n  socialContextBuilder: Option[BaseSocialContextBuilder[Query, Candidate]] = None,\n) extends CandidateUrtEntryBuilder[Query, Candidate, ArticleItem] {\n\n  override def apply(\n    query: Query,\n    articleCandidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): ArticleItem = ArticleItem(\n    id = articleCandidate.id,\n    sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n    clientEventInfo = clientEventInfoBuilder(\n      query,\n      articleCandidate,\n      candidateFeatures,\n      Some(ArticleClientEventInfoElement)),\n    feedbackActionInfo =\n      feedbackActionInfoBuilder.flatMap(_.apply(query, articleCandidate, candidateFeatures)),\n    displayType = displayType,\n    socialContext =\n      socialContextBuilder.flatMap(_.apply(query, articleCandidate, candidateFeatures)),\n    articleSeedType = articleSeedType\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/audio_space/AudioSpaceCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.audio_space\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.audio_space.AudioSpaceCandidateUrtItemBuilder.AudioSpaceClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.AudioSpaceCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.audio_space.AudioSpaceItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject AudioSpaceCandidateUrtItemBuilder {\n  val AudioSpaceClientEventInfoElement: String = \"audiospace\"\n}\n\ncase class AudioSpaceCandidateUrtItemBuilder[-Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, UniversalNoun[Any]],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, UniversalNoun[Any]]\n  ] = None)\n    extends CandidateUrtEntryBuilder[Query, AudioSpaceCandidate, AudioSpaceItem] {\n\n  override def apply(\n    query: Query,\n    audioSpaceCandidate: AudioSpaceCandidate,\n    candidateFeatures: FeatureMap\n  ): AudioSpaceItem = AudioSpaceItem(\n    id = audioSpaceCandidate.id,\n    sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n    clientEventInfo = clientEventInfoBuilder(\n      query,\n      audioSpaceCandidate,\n      candidateFeatures,\n      Some(AudioSpaceClientEventInfoElement)),\n    feedbackActionInfo =\n      feedbackActionInfoBuilder.flatMap(_.apply(query, audioSpaceCandidate, candidateFeatures))\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/card/CardCandidateUtrItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.card\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.card.CardCandidateUtrItemBuilder.CardClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.CardCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseStr\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseUrlBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.card.CardDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.card.CardItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject CardCandidateUtrItemBuilder {\n  val CardClientEventInfoElement: String = \"card\"\n}\n\ncase class CardCandidateUtrItemBuilder[-Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, CardCandidate],\n  cardUrlBuilder: BaseStr[Query, CardCandidate],\n  textBuilder: Option[BaseStr[Query, CardCandidate]],\n  subtextBuilder: Option[BaseStr[Query, CardCandidate]],\n  urlBuilder: Option[BaseUrlBuilder[Query, CardCandidate]],\n  cardDisplayType: Option[CardDisplayType],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, CardCandidate],\n  ] = None)\n    extends CandidateUrtEntryBuilder[Query, CardCandidate, CardItem] {\n\n  override def apply(\n    query: Query,\n    cardCandidate: CardCandidate,\n    candidateFeatures: FeatureMap\n  ): CardItem = CardItem(\n    id = cardCandidate.id,\n    sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n    clientEventInfo = clientEventInfoBuilder(\n      query,\n      cardCandidate,\n      candidateFeatures,\n      Some(CardClientEventInfoElement)),\n    feedbackActionInfo =\n      feedbackActionInfoBuilder.flatMap(_.apply(query, cardCandidate, candidateFeatures)),\n    cardUrl = cardUrlBuilder(query, cardCandidate, candidateFeatures),\n    text = textBuilder.map(_.apply(query, cardCandidate, candidateFeatures)),\n    subtext = textBuilder.map(_.apply(query, cardCandidate, candidateFeatures)),\n    url = urlBuilder.map(_.apply(query, cardCandidate, candidateFeatures)),\n    displayType = cardDisplayType\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/commerce/CommerceProductCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.commerce\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.commerce.CommerceProductCandidateUrtItemBuilder.CommerceProductClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.CommerceProductCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.commerce.CommerceProductItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject CommerceProductCandidateUrtItemBuilder {\n  val CommerceProductClientEventInfoElement: String = \"commerce-product\"\n}\n\ncase class CommerceProductCandidateUrtItemBuilder[-Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, CommerceProductCandidate],\n  feedbackActionInfoBuilder: Option[BaseFeedbackActionInfoBuilder[Query, CommerceProductCandidate]])\n    extends CandidateUrtEntryBuilder[\n      Query,\n      CommerceProductCandidate,\n      CommerceProductItem\n    ] {\n\n  override def apply(\n    query: Query,\n    candidate: CommerceProductCandidate,\n    candidateFeatures: FeatureMap\n  ): CommerceProductItem =\n    CommerceProductItem(\n      id = candidate.id,\n      sortIndex = None,\n      clientEventInfo = clientEventInfoBuilder(\n        query,\n        candidate,\n        candidateFeatures,\n        Some(CommerceProductClientEventInfoElement)),\n      feedbackActionInfo =\n        feedbackActionInfoBuilder.flatMap(_.apply(query, candidate, candidateFeatures))\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/commerce/CommerceProductGroupCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.commerce\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.commerce.CommerceProductGroupCandidateUrtItemBuilder.CommerceProductGroupClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.CommerceProductGroupCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.commerce.CommerceProductGroupItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject CommerceProductGroupCandidateUrtItemBuilder {\n  val CommerceProductGroupClientEventInfoElement: String = \"commerce-product-group\"\n}\n\ncase class CommerceProductGroupCandidateUrtItemBuilder[-Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, CommerceProductGroupCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, CommerceProductGroupCandidate]\n  ]) extends CandidateUrtEntryBuilder[\n      Query,\n      CommerceProductGroupCandidate,\n      CommerceProductGroupItem\n    ] {\n\n  override def apply(\n    query: Query,\n    candidate: CommerceProductGroupCandidate,\n    candidateFeatures: FeatureMap\n  ): CommerceProductGroupItem =\n    CommerceProductGroupItem(\n      id = candidate.id,\n      sortIndex = None,\n      clientEventInfo = clientEventInfoBuilder(\n        query,\n        candidate,\n        candidateFeatures,\n        Some(CommerceProductGroupClientEventInfoElement)),\n      feedbackActionInfo =\n        feedbackActionInfoBuilder.flatMap(_.apply(query, candidate, candidateFeatures))\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/event_summary/EventCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.event_summary\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.event_summary.EventCandidateUrtItemBuilder.EventClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.EventDisplayType\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.EventImage\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.EventTimeString\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.EventTitleFeature\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.EventUrl\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.UnifiedEventCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.event.EventSummaryItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject EventCandidateUrtItemBuilder {\n  val EventClientEventInfoElement = \"event\"\n}\n\ncase class EventCandidateUrtItemBuilder[Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, UnifiedEventCandidate],\n  feedbackActionInfoBuilder: Option[BaseFeedbackActionInfoBuilder[Query, UnifiedEventCandidate]] =\n    None)\n    extends CandidateUrtEntryBuilder[Query, UnifiedEventCandidate, TimelineItem] {\n\n  override def apply(\n    query: Query,\n    candidate: UnifiedEventCandidate,\n    candidateFeatures: FeatureMap\n  ): TimelineItem = {\n    EventSummaryItem(\n      id = candidate.id,\n      sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n      clientEventInfo = clientEventInfoBuilder(\n        query = query,\n        candidate = candidate,\n        candidateFeatures = candidateFeatures,\n        element = Some(EventClientEventInfoElement)\n      ),\n      feedbackActionInfo =\n        feedbackActionInfoBuilder.flatMap(_.apply(query, candidate, candidateFeatures)),\n      title = candidateFeatures.get(EventTitleFeature),\n      displayType = candidateFeatures.get(EventDisplayType),\n      url = candidateFeatures.get(EventUrl),\n      image = candidateFeatures.getOrElse(EventImage, None),\n      timeString = candidateFeatures.getOrElse(EventTimeString, None)\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/generic_summary/GenericSummaryActionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.generic_summary\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.generic_summary.GenericSummaryActionBuilder.GenericSummaryActionClientEventInfoElement\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseUrlBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary.GenericSummaryAction\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject GenericSummaryActionBuilder {\n  val GenericSummaryActionClientEventInfoElement: String = \"genericsummary-action\"\n}\n\ncase class GenericSummaryActionBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  urlBuilder: BaseUrlBuilder[Query, Candidate],\n  clientEventInfoBuilder: Option[BaseClientEventInfoBuilder[Query, Candidate]] = None) {\n\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): GenericSummaryAction = GenericSummaryAction(\n    url = urlBuilder.apply(query, candidate, candidateFeatures),\n    clientEventInfo = clientEventInfoBuilder.flatMap(\n      _.apply(\n        query,\n        candidate,\n        candidateFeatures,\n        Some(GenericSummaryActionClientEventInfoElement)))\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/generic_summary/GenericSummaryCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.generic_summary\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.generic_summary.GenericSummaryCandidateUrtItemBuilder.GenericSummaryClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.GenericSummaryCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.richtext.BaseRichTextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary.GenericSummaryItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary.GenericSummaryItemDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.media.Media\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PromotedMetadata\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.util.Time\n\nobject GenericSummaryCandidateUrtItemBuilder {\n  val GenericSummaryClientEventInfoElement: String = \"genericsummary\"\n}\n\ncase class GenericSummaryCandidateUrtItemBuilder[-Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, GenericSummaryCandidate],\n  headlineRichTextBuilder: BaseRichTextBuilder[Query, GenericSummaryCandidate],\n  displayType: GenericSummaryItemDisplayType,\n  genericSummaryContextCandidateUrtItemBuilder: Option[\n    GenericSummaryContextBuilder[Query, GenericSummaryCandidate]\n  ] = None,\n  genericSummaryActionCandidateUrtItemBuilder: Option[\n    GenericSummaryActionBuilder[Query, GenericSummaryCandidate]\n  ] = None,\n  timestamp: Option[Time] = None,\n  userAttributionIds: Option[Seq[Long]] = None,\n  media: Option[Media] = None,\n  promotedMetadata: Option[PromotedMetadata] = None,\n  feedbackActionInfoBuilder: Option[BaseFeedbackActionInfoBuilder[Query, GenericSummaryCandidate]] =\n    None)\n    extends CandidateUrtEntryBuilder[Query, GenericSummaryCandidate, GenericSummaryItem] {\n\n  override def apply(\n    query: Query,\n    genericSummaryCandidate: GenericSummaryCandidate,\n    candidateFeatures: FeatureMap\n  ): GenericSummaryItem = GenericSummaryItem(\n    id = genericSummaryCandidate.id,\n    sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n    clientEventInfo = clientEventInfoBuilder(\n      query,\n      genericSummaryCandidate,\n      candidateFeatures,\n      Some(GenericSummaryClientEventInfoElement)),\n    feedbackActionInfo =\n      feedbackActionInfoBuilder.flatMap(_.apply(query, genericSummaryCandidate, candidateFeatures)),\n    headline = headlineRichTextBuilder.apply(query, genericSummaryCandidate, candidateFeatures),\n    displayType = displayType,\n    userAttributionIds = userAttributionIds.getOrElse(Seq.empty),\n    media = media,\n    context = genericSummaryContextCandidateUrtItemBuilder.map(\n      _.apply(query, genericSummaryCandidate, candidateFeatures)),\n    timestamp = timestamp,\n    onClickAction = genericSummaryActionCandidateUrtItemBuilder.map(\n      _.apply(query, genericSummaryCandidate, candidateFeatures)),\n    promotedMetadata = promotedMetadata\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/generic_summary/GenericSummaryContextBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.generic_summary\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.richtext.BaseRichTextBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.HorizonIcon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary.GenericSummaryContext\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class GenericSummaryContextBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  richTextBuilder: BaseRichTextBuilder[Query, Candidate],\n  icon: Option[HorizonIcon] = None) {\n\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): GenericSummaryContext = GenericSummaryContext(\n    richTextBuilder.apply(query, candidate, candidateFeatures),\n    icon\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/icon_label/IconLabelCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.icon_label\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.icon_label.IconLabelCandidateUrtItemBuilder.IconLabelClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.LabelCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.richtext.BaseRichTextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.HorizonIcon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.icon_label.IconLabelItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextEntity\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject IconLabelCandidateUrtItemBuilder {\n  val IconLabelClientEventInfoElement: String = \"iconlabel\"\n}\n\ncase class IconLabelCandidateUrtItemBuilder[-Query <: PipelineQuery, Candidate <: LabelCandidate](\n  richTextBuilder: BaseRichTextBuilder[Query, Candidate],\n  icon: Option[HorizonIcon] = None,\n  entities: Option[List[RichTextEntity]] = None,\n  clientEventInfoBuilder: Option[BaseClientEventInfoBuilder[Query, Candidate]] = None,\n  feedbackActionInfoBuilder: Option[BaseFeedbackActionInfoBuilder[Query, Candidate]] = None)\n    extends CandidateUrtEntryBuilder[Query, Candidate, IconLabelItem] {\n\n  override def apply(\n    query: Query,\n    labelCandidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): IconLabelItem =\n    IconLabelItem(\n      id = labelCandidate.id.toString,\n      sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n      clientEventInfo = clientEventInfoBuilder.flatMap(\n        _.apply(query, labelCandidate, candidateFeatures, Some(IconLabelClientEventInfoElement))),\n      feedbackActionInfo =\n        feedbackActionInfoBuilder.flatMap(_.apply(query, labelCandidate, candidateFeatures)),\n      text = richTextBuilder(query, labelCandidate, candidateFeatures),\n      icon = icon,\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/message/CompactPromptCandidateUrtItemStringCenterBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.message\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.message.CompactPromptCandidateUrtItemStringCenterBuilder.CompactPromptClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.CompactPromptCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseStr\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.richtext.BaseRichTextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.CompactPromptMessageContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessagePromptItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject CompactPromptCandidateUrtItemStringCenterBuilder {\n  val CompactPromptClientEventInfoElement: String = \"message\"\n}\n\ncase class CompactPromptCandidateUrtItemStringCenterBuilder[-Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, CompactPromptCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, CompactPromptCandidate]\n  ] = None,\n  headerTextBuilder: BaseStr[Query, CompactPromptCandidate],\n  bodyTextBuilder: Option[BaseStr[Query, CompactPromptCandidate]] = None,\n  headerRichTextBuilder: Option[BaseRichTextBuilder[Query, CompactPromptCandidate]] = None,\n  bodyRichTextBuilder: Option[BaseRichTextBuilder[Query, CompactPromptCandidate]] = None)\n    extends CandidateUrtEntryBuilder[Query, CompactPromptCandidate, MessagePromptItem] {\n\n  override def apply(\n    query: Query,\n    compactPromptCandidate: CompactPromptCandidate,\n    candidateFeatures: FeatureMap\n  ): MessagePromptItem =\n    MessagePromptItem(\n      id = compactPromptCandidate.id.toString,\n      sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n      clientEventInfo = clientEventInfoBuilder(\n        query,\n        compactPromptCandidate,\n        candidateFeatures,\n        Some(CompactPromptClientEventInfoElement)),\n      feedbackActionInfo = feedbackActionInfoBuilder.flatMap(\n        _.apply(query, compactPromptCandidate, candidateFeatures)),\n      isPinned = None,\n      content = CompactPromptMessageContent(\n        headerText = headerTextBuilder.apply(query, compactPromptCandidate, candidateFeatures),\n        bodyText = bodyTextBuilder.map(_.apply(query, compactPromptCandidate, candidateFeatures)),\n        primaryButtonAction = None,\n        secondaryButtonAction = None,\n        action = None,\n        headerRichText =\n          headerRichTextBuilder.map(_.apply(query, compactPromptCandidate, candidateFeatures)),\n        bodyRichText =\n          bodyRichTextBuilder.map(_.apply(query, compactPromptCandidate, candidateFeatures))\n      ),\n      impressionCallbacks = None\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/message/InlinePromptCandidateUrtItemStringCenterBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.message\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.message.InlinePromptCandidateUrtItemStringCenterBuilder.InlinePromptClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.InlinePromptCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseStr\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.richtext.BaseRichTextBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.InlinePromptMessageContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessagePromptItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject InlinePromptCandidateUrtItemStringCenterBuilder {\n  val InlinePromptClientEventInfoElement: String = \"message\"\n}\n\ncase class InlinePromptCandidateUrtItemStringCenterBuilder[-Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, InlinePromptCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, InlinePromptCandidate]\n  ] = None,\n  headerTextBuilder: BaseStr[Query, InlinePromptCandidate],\n  bodyTextBuilder: Option[BaseStr[Query, InlinePromptCandidate]] = None,\n  headerRichTextBuilder: Option[BaseRichTextBuilder[Query, InlinePromptCandidate]] = None,\n  bodyRichTextBuilder: Option[BaseRichTextBuilder[Query, InlinePromptCandidate]] = None,\n  primaryMessageTextActionBuilder: Option[\n    MessageTextActionBuilder[Query, InlinePromptCandidate]\n  ] = None,\n  secondaryMessageTextActionBuilder: Option[\n    MessageTextActionBuilder[Query, InlinePromptCandidate]\n  ] = None,\n  socialContextBuilder: Option[BaseSocialContextBuilder[Query, InlinePromptCandidate]] = None,\n  userFacePileBuilder: Option[\n    UserFacePileBuilder\n  ] = None)\n    extends CandidateUrtEntryBuilder[Query, InlinePromptCandidate, MessagePromptItem] {\n\n  override def apply(\n    query: Query,\n    inlinePromptCandidate: InlinePromptCandidate,\n    candidateFeatures: FeatureMap\n  ): MessagePromptItem =\n    MessagePromptItem(\n      id = inlinePromptCandidate.id,\n      sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n      clientEventInfo = clientEventInfoBuilder(\n        query,\n        inlinePromptCandidate,\n        candidateFeatures,\n        Some(InlinePromptClientEventInfoElement)),\n      feedbackActionInfo =\n        feedbackActionInfoBuilder.flatMap(_.apply(query, inlinePromptCandidate, candidateFeatures)),\n      isPinned = None,\n      content = InlinePromptMessageContent(\n        headerText = headerTextBuilder.apply(query, inlinePromptCandidate, candidateFeatures),\n        bodyText = bodyTextBuilder.map(_.apply(query, inlinePromptCandidate, candidateFeatures)),\n        primaryButtonAction = primaryMessageTextActionBuilder.map(\n          _.apply(query, inlinePromptCandidate, candidateFeatures)),\n        secondaryButtonAction = secondaryMessageTextActionBuilder.map(\n          _.apply(query, inlinePromptCandidate, candidateFeatures)),\n        headerRichText =\n          headerRichTextBuilder.map(_.apply(query, inlinePromptCandidate, candidateFeatures)),\n        bodyRichText =\n          bodyRichTextBuilder.map(_.apply(query, inlinePromptCandidate, candidateFeatures)),\n        socialContext =\n          socialContextBuilder.flatMap(_.apply(query, inlinePromptCandidate, candidateFeatures)),\n        userFacepile = userFacePileBuilder.map(_.apply())\n      ),\n      impressionCallbacks = None\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/message/MessageTextActionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.message\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseStr\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageTextAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject MessageTextActionBuilder {\n  val MessageTextActionClientEventInfoElement: String = \"message-text-action\"\n}\n\ncase class MessageTextActionBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  textBuilder: BaseStr[Query, Candidate],\n  dismissOnClick: Boolean,\n  url: Option[String] = None,\n  clientEventInfo: Option[ClientEventInfo] = None,\n  onClickCallbacks: Option[List[Callback]] = None) {\n\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): MessageTextAction = MessageTextAction(\n    text = textBuilder(query, candidate, candidateFeatures),\n    action = MessageAction(\n      dismissOnClick,\n      url,\n      clientEventInfo,\n      onClickCallbacks\n    )\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/message/UserFacePileBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.message\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageActionType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageTextAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.UserFacepile\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.UserFacepileDisplayType\n\ncase class UserFacePileBuilder(\n  userIds: Seq[Long],\n  featuredUserIds: Seq[Long],\n  action: Option[MessageTextAction],\n  actionType: Option[MessageActionType],\n  displaysFeaturingText: Option[Boolean],\n  displayType: Option[UserFacepileDisplayType]) {\n\n  def apply(): UserFacepile = UserFacepile(\n    userIds = userIds,\n    featuredUserIds = featuredUserIds,\n    action = action,\n    actionType = actionType,\n    displaysFeaturingText = displaysFeaturingText,\n    displayType = displayType\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/moment/MomentAnnotationCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.moment\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.moment.MomentAnnotationCandidateUrtItemBuilder.MomentAnnotationItemClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.MomentAnnotationCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.richtext.BaseRichTextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.moment.MomentAnnotationItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject MomentAnnotationCandidateUrtItemBuilder {\n  val MomentAnnotationItemClientEventInfoElement = \"metadata\"\n}\n\ncase class MomentAnnotationCandidateUrtItemBuilder[Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, MomentAnnotationCandidate],\n  annotationTextRichTextBuilder: BaseRichTextBuilder[Query, MomentAnnotationCandidate],\n  annotationHeaderRichTextBuilder: BaseRichTextBuilder[Query, MomentAnnotationCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, MomentAnnotationCandidate]\n  ] = None,\n) extends CandidateUrtEntryBuilder[Query, MomentAnnotationCandidate, MomentAnnotationItem] {\n\n  override def apply(\n    query: Query,\n    candidate: MomentAnnotationCandidate,\n    candidateFeatures: FeatureMap\n  ): MomentAnnotationItem = MomentAnnotationItem(\n    id = candidate.id,\n    sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n    clientEventInfo = clientEventInfoBuilder(\n      query,\n      candidate,\n      candidateFeatures,\n      Some(MomentAnnotationItemClientEventInfoElement)),\n    feedbackActionInfo =\n      feedbackActionInfoBuilder.flatMap(_.apply(query, candidate, candidateFeatures)),\n    isPinned = None,\n    text =\n      candidate.text.map(_ => annotationTextRichTextBuilder(query, candidate, candidateFeatures)),\n    header = candidate.header.map(_ =>\n      annotationHeaderRichTextBuilder(query, candidate, candidateFeatures)),\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/relevance_prompt/RelevancePromptCandidateUrtItemStringCenterBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.relevance_prompt\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.relevance_prompt.RelevancePromptCandidateUrtItemStringCenterBuilder.RelevancePromptClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.RelevancePromptCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseStr\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.PromptItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.RelevancePromptContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.RelevancePromptDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.RelevancePromptFollowUpFeedbackType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject RelevancePromptCandidateUrtItemStringCenterBuilder {\n  val RelevancePromptClientEventInfoElement: String = \"relevance_prompt\"\n}\n\ncase class RelevancePromptCandidateUrtItemStringCenterBuilder[-Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, RelevancePromptCandidate],\n  titleTextBuilder: BaseStr[Query, RelevancePromptCandidate],\n  confirmationTextBuilder: BaseStr[Query, RelevancePromptCandidate],\n  isRelevantTextBuilder: BaseStr[Query, RelevancePromptCandidate],\n  notRelevantTextBuilder: BaseStr[Query, RelevancePromptCandidate],\n  displayType: RelevancePromptDisplayType,\n  isRelevantCallback: Callback,\n  notRelevantCallback: Callback,\n  isRelevantFollowUp: Option[RelevancePromptFollowUpFeedbackType] = None,\n  notRelevantFollowUp: Option[RelevancePromptFollowUpFeedbackType] = None,\n  impressionCallbacks: Option[List[Callback]] = None)\n    extends CandidateUrtEntryBuilder[Query, RelevancePromptCandidate, PromptItem] {\n\n  override def apply(\n    query: Query,\n    relevancePromptCandidate: RelevancePromptCandidate,\n    candidateFeatures: FeatureMap\n  ): PromptItem =\n    PromptItem(\n      id = relevancePromptCandidate.id,\n      sortIndex = None,\n      clientEventInfo = clientEventInfoBuilder(\n        query,\n        relevancePromptCandidate,\n        candidateFeatures,\n        Some(RelevancePromptClientEventInfoElement)),\n      feedbackActionInfo = None,\n      content = RelevancePromptContent(\n        title = titleTextBuilder(query, relevancePromptCandidate, candidateFeatures),\n        confirmation = confirmationTextBuilder(query, relevancePromptCandidate, candidateFeatures),\n        isRelevantText = isRelevantTextBuilder(query, relevancePromptCandidate, candidateFeatures),\n        notRelevantText =\n          notRelevantTextBuilder(query, relevancePromptCandidate, candidateFeatures),\n        isRelevantCallback = isRelevantCallback,\n        notRelevantCallback = notRelevantCallback,\n        displayType = displayType,\n        isRelevantFollowUp = isRelevantFollowUp,\n        notRelevantFollowUp = notRelevantFollowUp,\n      ),\n      impressionCallbacks = impressionCallbacks\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/suggestion/SpellingSuggestionCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.suggestion\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.suggestion.SpellingSuggestionCandidateUrtItemBuilder.SpellingItemClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.suggestion.SpellingSuggestionCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.suggestion.SpellingItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject SpellingSuggestionCandidateUrtItemBuilder {\n  val SpellingItemClientEventInfoElement: String = \"spelling\"\n}\n\ncase class SpellingSuggestionCandidateUrtItemBuilder[Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, SpellingSuggestionCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, SpellingSuggestionCandidate]\n  ] = None,\n) extends CandidateUrtEntryBuilder[Query, SpellingSuggestionCandidate, SpellingItem] {\n\n  override def apply(\n    query: Query,\n    candidate: SpellingSuggestionCandidate,\n    candidateFeatures: FeatureMap\n  ): SpellingItem = SpellingItem(\n    id = candidate.id,\n    sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n    clientEventInfo = clientEventInfoBuilder(\n      query,\n      candidate,\n      candidateFeatures,\n      Some(SpellingItemClientEventInfoElement)),\n    feedbackActionInfo =\n      feedbackActionInfoBuilder.flatMap(_.apply(query, candidate, candidateFeatures)),\n    textResult = candidate.textResult,\n    spellingActionType = candidate.spellingActionType,\n    originalQuery = candidate.originalQuery\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/tile/TileCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.tile\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tile.TileCandidateUrtItemBuilder.TopicTileClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.PromptCarouselTileCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile.StandardTileContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile.TileItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject TileCandidateUrtItemBuilder {\n  val TopicTileClientEventInfoElement: String = \"tile\"\n}\n\ncase class TileCandidateUrtItemBuilder[-Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, PromptCarouselTileCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, PromptCarouselTileCandidate]\n  ] = None)\n    extends CandidateUrtEntryBuilder[Query, PromptCarouselTileCandidate, TileItem] {\n\n  override def apply(\n    query: Query,\n    tileCandidate: PromptCarouselTileCandidate,\n    candidateFeatures: FeatureMap\n  ): TileItem = TileItem(\n    id = tileCandidate.id,\n    sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n    clientEventInfo = clientEventInfoBuilder(\n      query,\n      tileCandidate,\n      candidateFeatures,\n      Some(TopicTileClientEventInfoElement)),\n    title = \"\", //This data is ignored do\n    supportingText = \"\",\n    feedbackActionInfo =\n      feedbackActionInfoBuilder.flatMap(_.apply(query, tileCandidate, candidateFeatures)),\n    image = None,\n    url = None,\n    content = StandardTileContent(\n      title = \"\",\n      supportingText = \"\",\n      badge = None\n    )\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/topic/ParamTopicDisplayTypeBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.topic\n\nimport com.twitter.product_mixer.component_library.model.candidate.TopicCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.BasicTopicDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.PillTopicDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.NoIconTopicDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.PillWithoutActionIconDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicDisplayType\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.topic.BaseTopicDisplayTypeBuilder\n\nobject TopicCandidateDisplayType extends Enumeration {\n  type TopicDisplayType = Value\n\n  val Basic = Value\n  val Pill = Value\n  val NoIcon = Value\n  val PillWithoutActionIcon = Value\n}\n\ncase class ParamTopicDisplayTypeBuilder(\n  displayTypeParam: FSEnumParam[TopicCandidateDisplayType.type])\n    extends BaseTopicDisplayTypeBuilder[PipelineQuery, TopicCandidate] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TopicCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[TopicDisplayType] = {\n    val displayType = query.params(displayTypeParam)\n    displayType match {\n      case TopicCandidateDisplayType.Basic => Some(BasicTopicDisplayType)\n      case TopicCandidateDisplayType.Pill => Some(PillTopicDisplayType)\n      case TopicCandidateDisplayType.NoIcon =>\n        Some(NoIconTopicDisplayType)\n      case TopicCandidateDisplayType.PillWithoutActionIcon => Some(PillWithoutActionIconDisplayType)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/topic/ParamTopicFunctionalityTypeBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.topic\n\nimport com.twitter.product_mixer.component_library.model.candidate.TopicCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.BasicTopicFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.PivotTopicFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.RecommendationTopicFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicFunctionalityType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.topic.BaseTopicFunctionalityTypeBuilder\nimport com.twitter.timelines.configapi.FSEnumParam\n\nobject TopicFunctionalityTypeParamValue extends Enumeration {\n  type TopicFunctionalityType = Value\n\n  val Basic = Value\n  val Pivot = Value\n  val Recommendation = Value\n}\n\ncase class ParamTopicFunctionalityTypeBuilder(\n  functionalityTypeParam: FSEnumParam[TopicFunctionalityTypeParamValue.type])\n    extends BaseTopicFunctionalityTypeBuilder[PipelineQuery, TopicCandidate] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TopicCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[TopicFunctionalityType] = {\n    val functionalityType = query.params(functionalityTypeParam)\n    functionalityType match {\n      case TopicFunctionalityTypeParamValue.Basic => Some(BasicTopicFunctionalityType)\n      case TopicFunctionalityTypeParamValue.Pivot => Some(PivotTopicFunctionalityType)\n      case TopicFunctionalityTypeParamValue.Recommendation =>\n        Some(RecommendationTopicFunctionalityType)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/topic/StaticTopicDisplayTypeBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.topic\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTopicCandidate\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.topic.BaseTopicDisplayTypeBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicDisplayType\n\ncase class StaticTopicDisplayTypeBuilder(\n  displayType: TopicDisplayType)\n    extends BaseTopicDisplayTypeBuilder[PipelineQuery, BaseTopicCandidate] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: BaseTopicCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[TopicDisplayType] = Some(displayType)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/topic/StaticTopicFunctionalityTypeBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.topic\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTopicCandidate\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.topic.BaseTopicFunctionalityTypeBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicFunctionalityType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class StaticTopicFunctionalityTypeBuilder(\n  functionalityType: TopicFunctionalityType)\n    extends BaseTopicFunctionalityTypeBuilder[PipelineQuery, BaseTopicCandidate] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: BaseTopicCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[TopicFunctionalityType] = Some(functionalityType)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/topic/TopicCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.topic\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.topic.TopicCandidateUrtItemBuilder.TopicClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTopicCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.topic.BaseTopicDisplayTypeBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.topic.BaseTopicFunctionalityTypeBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject TopicCandidateUrtItemBuilder {\n  val TopicClientEventInfoElement: String = \"topic\"\n}\n\ncase class TopicCandidateUrtItemBuilder[-Query <: PipelineQuery, Candidate <: BaseTopicCandidate](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, Candidate],\n  topicFunctionalityTypeBuilder: Option[BaseTopicFunctionalityTypeBuilder[Query, Candidate]] = None,\n  topicDisplayTypeBuilder: Option[BaseTopicDisplayTypeBuilder[Query, Candidate]] = None,\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, Candidate]\n  ] = None)\n    extends CandidateUrtEntryBuilder[Query, Candidate, TopicItem] {\n\n  override def apply(\n    query: Query,\n    topicCandidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): TopicItem =\n    TopicItem(\n      id = topicCandidate.id,\n      sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n      clientEventInfo = clientEventInfoBuilder(\n        query,\n        topicCandidate,\n        candidateFeatures,\n        Some(TopicClientEventInfoElement)),\n      feedbackActionInfo =\n        feedbackActionInfoBuilder.flatMap(_.apply(query, topicCandidate, candidateFeatures)),\n      topicFunctionalityType =\n        topicFunctionalityTypeBuilder.flatMap(_.apply(query, topicCandidate, candidateFeatures)),\n      topicDisplayType =\n        topicDisplayTypeBuilder.flatMap(_.apply(query, topicCandidate, candidateFeatures))\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/topic/VerticalGridTopicCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.topic\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.topic.TopicCandidateUrtItemBuilder.TopicClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.TopicCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseUrlBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.VerticalGridItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.VerticalGridItemTileStyle\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.VerticalGridItemTopicFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.VerticalGridItemTopicTile\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class VerticalGridTopicCandidateUrtItemBuilder[-Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, TopicCandidate],\n  verticalGridItemTopicFunctionalityType: VerticalGridItemTopicFunctionalityType,\n  verticalGridItemTileStyle: VerticalGridItemTileStyle,\n  urlBuilder: Option[BaseUrlBuilder[Query, TopicCandidate]] = None,\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, TopicCandidate]\n  ] = None)\n    extends CandidateUrtEntryBuilder[Query, TopicCandidate, VerticalGridItem] {\n\n  override def apply(\n    query: Query,\n    topicCandidate: TopicCandidate,\n    candidateFeatures: FeatureMap\n  ): VerticalGridItem = {\n    VerticalGridItemTopicTile(\n      id = topicCandidate.id,\n      sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n      clientEventInfo = clientEventInfoBuilder(\n        query,\n        topicCandidate,\n        candidateFeatures,\n        Some(TopicClientEventInfoElement)),\n      feedbackActionInfo =\n        feedbackActionInfoBuilder.flatMap(_.apply(query, topicCandidate, candidateFeatures)),\n      style = Some(verticalGridItemTileStyle),\n      functionalityType = Some(verticalGridItemTopicFunctionalityType),\n      url = urlBuilder.map(_.apply(query, topicCandidate, candidateFeatures))\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/trend/TrendCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.trend\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.trend.TrendCandidateUrtItemBuilder.TrendsClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendDescription\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendDomainContext\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendGroupedTrends\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendNormalizedTrendName\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendTrendName\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendTweetCount\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendUrl\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.UnifiedTrendCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.promoted.BasePromotedMetadataBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.trend.TrendItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject TrendCandidateUrtItemBuilder {\n  final val TrendsClientEventInfoElement = \"trend\"\n}\n\ncase class TrendCandidateUrtItemBuilder[Query <: PipelineQuery](\n  trendMetaDescriptionBuilder: TrendMetaDescriptionBuilder[Query, UnifiedTrendCandidate],\n  promotedMetadataBuilder: BasePromotedMetadataBuilder[Query, UnifiedTrendCandidate],\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, UnifiedTrendCandidate],\n  feedbackActionInfoBuilder: Option[BaseFeedbackActionInfoBuilder[Query, UnifiedTrendCandidate]] =\n    None)\n    extends CandidateUrtEntryBuilder[Query, UnifiedTrendCandidate, TimelineItem] {\n\n  override def apply(\n    query: Query,\n    candidate: UnifiedTrendCandidate,\n    candidateFeatures: FeatureMap\n  ): TimelineItem = {\n    TrendItem(\n      id = candidate.id,\n      sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n      clientEventInfo = clientEventInfoBuilder(\n        query = query,\n        candidate = candidate,\n        candidateFeatures = candidateFeatures,\n        element = Some(TrendsClientEventInfoElement)\n      ),\n      feedbackActionInfo = None,\n      normalizedTrendName = candidateFeatures.get(TrendNormalizedTrendName),\n      trendName = candidateFeatures.get(TrendTrendName),\n      url = candidateFeatures.get(TrendUrl),\n      description = candidateFeatures.getOrElse(TrendDescription, None),\n      metaDescription = trendMetaDescriptionBuilder(query, candidate, candidateFeatures),\n      tweetCount = candidateFeatures.getOrElse(TrendTweetCount, None),\n      domainContext = candidateFeatures.getOrElse(TrendDomainContext, None),\n      promotedMetadata = promotedMetadataBuilder(\n        query = query,\n        candidate = candidate,\n        candidateFeatures = candidateFeatures\n      ),\n      groupedTrends = candidateFeatures.getOrElse(TrendGroupedTrends, None)\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/trend/TrendMetaDescriptionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.trend\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.Str\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.PromotedTrendAdvertiserNameFeature\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.TrendTweetCount\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.trends.trending_content.util.CompactingNumberLocalizer\n\ncase class TrendMetaDescriptionBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  promotedByMetaDescriptionStr: Str[PipelineQuery, UniversalNoun[Any]],\n  tweetCountMetaDescriptionStr: Str[PipelineQuery, UniversalNoun[Any]],\n  compactingNumberLocalizer: CompactingNumberLocalizer) {\n\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[String] = {\n    val promotedMetaDescription =\n      candidateFeatures.getOrElse(PromotedTrendAdvertiserNameFeature, None).map { advertiserName =>\n        promotedByMetaDescriptionStr(query, candidate, candidateFeatures).format(advertiserName)\n      }\n\n    val organicMetaDescription = candidateFeatures.getOrElse(TrendTweetCount, None).map {\n      tweetCount =>\n        val compactedTweetCount = compactingNumberLocalizer.localizeAndCompact(\n          query.getLanguageCode\n            .getOrElse(\"en\"),\n          tweetCount)\n        tweetCountMetaDescriptionStr(query, candidate, candidateFeatures).format(\n          compactedTweetCount)\n    }\n\n    promotedMetaDescription.orElse(organicMetaDescription)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/trend/TrendPromotedMetadataBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.trend\n\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.PromotedTrendDescriptionFeature\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.PromotedTrendDisclosureTypeFeature\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.PromotedTrendIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.PromotedTrendImpressionIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.PromotedTrendNameFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.promoted.BasePromotedMetadataBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PromotedMetadata\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject TrendPromotedMetadataBuilder\n    extends BasePromotedMetadataBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: UniversalNoun[Any],\n    candidateFeatures: FeatureMap\n  ): Option[PromotedMetadata] = {\n    // If a promoted trend name exists, then this is a promoted trend\n    candidateFeatures.getOrElse(PromotedTrendNameFeature, None).map { promotedTrendName =>\n      PromotedMetadata(\n        // This is the current product behavior that advertiserId is always set to 0L.\n        // Correct advertiser name comes from Trend's trendMetadata.metaDescription.\n        advertiserId = 0L,\n        disclosureType = candidateFeatures.getOrElse(PromotedTrendDisclosureTypeFeature, None),\n        experimentValues = None,\n        promotedTrendId = candidateFeatures.getOrElse(PromotedTrendIdFeature, None),\n        promotedTrendName = Some(promotedTrendName),\n        promotedTrendQueryTerm = None,\n        adMetadataContainer = None,\n        promotedTrendDescription =\n          candidateFeatures.getOrElse(PromotedTrendDescriptionFeature, None),\n        impressionString = candidateFeatures.getOrElse(PromotedTrendImpressionIdFeature, None),\n        clickTrackingInfo = None\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/tweet/TweetCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.contextual_ref.ContextualTweetRefBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder.TweetClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.IsPinnedFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.tweet.BaseEntryIdToReplaceBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.tweet.BaseTimelinesScoreInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.tweet.BaseTweetHighlightsBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseUrlBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.conversation_annotation.ConversationAnnotation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot.ForwardPivot\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone.TombstoneInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.Tweet\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Badge\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PrerollMetadata\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PromotedMetadata\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase object TweetCandidateUrtItemBuilder {\n  val TweetClientEventInfoElement = \"tweet\"\n}\n\ncase class TweetCandidateUrtItemBuilder[Query <: PipelineQuery, Candidate <: BaseTweetCandidate](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, Candidate],\n  displayType: TweetDisplayType = Tweet,\n  entryIdToReplaceBuilder: Option[BaseEntryIdToReplaceBuilder[Query, Candidate]] = None,\n  socialContextBuilder: Option[BaseSocialContextBuilder[Query, Candidate]] = None,\n  highlightsBuilder: Option[BaseTweetHighlightsBuilder[Query, Candidate]] = None,\n  innerTombstoneInfo: Option[TombstoneInfo] = None,\n  timelinesScoreInfoBuilder: Option[BaseTimelinesScoreInfoBuilder[Query, Candidate]] = None,\n  hasModeratedReplies: Option[Boolean] = None,\n  forwardPivot: Option[ForwardPivot] = None,\n  innerForwardPivot: Option[ForwardPivot] = None,\n  feedbackActionInfoBuilder: Option[BaseFeedbackActionInfoBuilder[Query, Candidate]] = None,\n  promotedMetadata: Option[PromotedMetadata] = None,\n  conversationAnnotation: Option[ConversationAnnotation] = None,\n  contextualTweetRefBuilder: Option[ContextualTweetRefBuilder[Candidate]] = None,\n  prerollMetadata: Option[PrerollMetadata] = None,\n  replyBadge: Option[Badge] = None,\n  destinationBuilder: Option[BaseUrlBuilder[Query, Candidate]] = None)\n    extends CandidateUrtEntryBuilder[Query, Candidate, TweetItem] {\n\n  override def apply(\n    pipelineQuery: Query,\n    tweetCandidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): TweetItem = {\n    val isPinned = candidateFeatures.getTry(IsPinnedFeature).toOption\n\n    TweetItem(\n      id = tweetCandidate.id,\n      entryNamespace = TweetItem.TweetEntryNamespace,\n      sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n      clientEventInfo = clientEventInfoBuilder(\n        pipelineQuery,\n        tweetCandidate,\n        candidateFeatures,\n        Some(TweetClientEventInfoElement)),\n      feedbackActionInfo = feedbackActionInfoBuilder.flatMap(\n        _.apply(pipelineQuery, tweetCandidate, candidateFeatures)),\n      isPinned = isPinned,\n      entryIdToReplace =\n        entryIdToReplaceBuilder.flatMap(_.apply(pipelineQuery, tweetCandidate, candidateFeatures)),\n      socialContext =\n        socialContextBuilder.flatMap(_.apply(pipelineQuery, tweetCandidate, candidateFeatures)),\n      highlights =\n        highlightsBuilder.flatMap(_.apply(pipelineQuery, tweetCandidate, candidateFeatures)),\n      displayType = displayType,\n      innerTombstoneInfo = innerTombstoneInfo,\n      timelinesScoreInfo = timelinesScoreInfoBuilder\n        .flatMap(_.apply(pipelineQuery, tweetCandidate, candidateFeatures)),\n      hasModeratedReplies = hasModeratedReplies,\n      forwardPivot = forwardPivot,\n      innerForwardPivot = innerForwardPivot,\n      promotedMetadata = promotedMetadata,\n      conversationAnnotation = conversationAnnotation,\n      contextualTweetRef = contextualTweetRefBuilder.flatMap(_.apply(tweetCandidate)),\n      prerollMetadata = prerollMetadata,\n      replyBadge = replyBadge,\n      destination =\n        destinationBuilder.map(_.apply(pipelineQuery, tweetCandidate, candidateFeatures))\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/twitter_list/TwitterListCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.twitter_list\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.twitter_list.TwitterListCandidateUrtItemBuilder.ListClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.TwitterListCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.twitter_list.TwitterListDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.twitter_list.TwitterListItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject TwitterListCandidateUrtItemBuilder {\n  val ListClientEventInfoElement: String = \"list\"\n}\n\ncase class TwitterListCandidateUrtItemBuilder[-Query <: PipelineQuery](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, TwitterListCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, TwitterListCandidate]\n  ] = None,\n  displayType: Option[TwitterListDisplayType] = None)\n    extends CandidateUrtEntryBuilder[Query, TwitterListCandidate, TwitterListItem] {\n\n  override def apply(\n    query: Query,\n    twitterListCandidate: TwitterListCandidate,\n    candidateFeatures: FeatureMap\n  ): TwitterListItem = TwitterListItem(\n    id = twitterListCandidate.id,\n    sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n    clientEventInfo = clientEventInfoBuilder(\n      query,\n      twitterListCandidate,\n      candidateFeatures,\n      Some(ListClientEventInfoElement)),\n    feedbackActionInfo =\n      feedbackActionInfoBuilder.flatMap(_.apply(query, twitterListCandidate, candidateFeatures)),\n    displayType = displayType\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/unified_trend_event/UnifiedTrendEventCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.unified_trend_event\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.event_summary.EventCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.trend.TrendCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.UnifiedEventCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.UnifiedTrendCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.trends_events.UnifiedTrendEventCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class UnifiedTrendEventCandidateUrtItemBuilder[Query <: PipelineQuery](\n  eventCandidateUrtItemBuilder: EventCandidateUrtItemBuilder[Query],\n  trendCandidateUrtItemBuilder: TrendCandidateUrtItemBuilder[Query])\n    extends CandidateUrtEntryBuilder[Query, UnifiedTrendEventCandidate[Any], TimelineItem] {\n\n  override def apply(\n    query: Query,\n    candidate: UnifiedTrendEventCandidate[Any],\n    candidateFeatures: FeatureMap\n  ): TimelineItem = {\n    candidate match {\n      case event: UnifiedEventCandidate =>\n        eventCandidateUrtItemBuilder(\n          query = query,\n          candidate = event,\n          candidateFeatures = candidateFeatures)\n      case trend: UnifiedTrendCandidate =>\n        trendCandidateUrtItemBuilder(\n          query = query,\n          candidate = trend,\n          candidateFeatures = candidateFeatures)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/item/user/UserCandidateUrtItemBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.item.user\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.user.UserCandidateUrtItemBuilder.UserClientEventInfoElement\nimport com.twitter.product_mixer.component_library.model.candidate.BaseUserCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.IsMarkUnreadFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.user.BaseUserReactiveTriggersBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.promoted.BasePromotedMetadataBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.User\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.UserDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.UserItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject UserCandidateUrtItemBuilder {\n  val UserClientEventInfoElement: String = \"user\"\n}\n\ncase class UserCandidateUrtItemBuilder[Query <: PipelineQuery, UserCandidate <: BaseUserCandidate](\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, UserCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, UserCandidate]\n  ] = None,\n  displayType: UserDisplayType = User,\n  promotedMetadataBuilder: Option[BasePromotedMetadataBuilder[Query, UserCandidate]] = None,\n  socialContextBuilder: Option[BaseSocialContextBuilder[Query, UserCandidate]] = None,\n  reactiveTriggersBuilder: Option[BaseUserReactiveTriggersBuilder[Query, UserCandidate]] = None,\n  enableReactiveBlending: Option[Boolean] = None)\n    extends CandidateUrtEntryBuilder[Query, UserCandidate, UserItem] {\n\n  override def apply(\n    query: Query,\n    userCandidate: UserCandidate,\n    candidateFeatures: FeatureMap\n  ): UserItem = {\n    val isMarkUnread = candidateFeatures.getTry(IsMarkUnreadFeature).toOption\n\n    UserItem(\n      id = userCandidate.id,\n      sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n      clientEventInfo = clientEventInfoBuilder(\n        query,\n        userCandidate,\n        candidateFeatures,\n        Some(UserClientEventInfoElement)),\n      feedbackActionInfo =\n        feedbackActionInfoBuilder.flatMap(_.apply(query, userCandidate, candidateFeatures)),\n      isMarkUnread = isMarkUnread,\n      displayType = displayType,\n      promotedMetadata =\n        promotedMetadataBuilder.flatMap(_.apply(query, userCandidate, candidateFeatures)),\n      socialContext =\n        socialContextBuilder.flatMap(_.apply(query, userCandidate, candidateFeatures)),\n      reactiveTriggers =\n        reactiveTriggersBuilder.flatMap(_.apply(query, userCandidate, candidateFeatures)),\n      enableReactiveBlending = enableReactiveBlending\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/metadata/ClientEventInfoBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.metadata\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder\n\n/**\n * Sets the [[ClientEventInfo]] with the `component` field set to [[component]]\n * @see  [[http://go/client-events]]\n */\ncase class ClientEventInfoBuilder[-Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n  component: String,\n  detailsBuilder: Option[BaseClientEventDetailsBuilder[Query, Candidate]] = None)\n    extends BaseClientEventInfoBuilder[Query, Candidate] {\n\n  override def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap,\n    element: Option[String]\n  ): Option[ClientEventInfo] =\n    Some(\n      ClientEventInfo(\n        component = Some(component),\n        element = element,\n        details = detailsBuilder.flatMap(_.apply(query, candidate, candidateFeatures)),\n        action = None,\n        entityToken = None)\n    )\n}\n\n/**\n * In rare cases you might not want to send client event info. For\n * example, this might be set already on the client for some legacy\n * timelines.\n */\nobject EmptyClientEventInfoBuilder\n    extends BaseClientEventInfoBuilder[PipelineQuery, UniversalNoun[Any]] {\n  override def apply(\n    query: PipelineQuery,\n    candidate: UniversalNoun[Any],\n    candidateFeatures: FeatureMap,\n    element: Option[String]\n  ): Option[ClientEventInfo] = None\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/metadata/ConversationTweetClientEventDetailsBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.metadata\n\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.bijection.Base64String\nimport com.twitter.bijection.{Injection => Serializer}\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ConversationTweetClientEventDetailsBuilder.ControllerDataSerializer\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.suggests.controller_data.home_tweets.thriftscala.HomeTweetsControllerData\nimport com.twitter.suggests.controller_data.thriftscala.ControllerData\nimport com.twitter.suggests.controller_data.home_tweets.v1.thriftscala.{\n  HomeTweetsControllerData => HomeTweetsControllerDataV1\n}\nimport com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}\n\nobject ConversationTweetClientEventDetailsBuilder {\n  implicit val ByteSerializer: Serializer[ControllerData, Array[Byte]] =\n    BinaryScalaCodec(ControllerData)\n\n  val ControllerDataSerializer: Serializer[ControllerData, String] =\n    Serializer.connect[ControllerData, Array[Byte], Base64String, String]\n}\n\ncase class ConversationTweetClientEventDetailsBuilder[-Query <: PipelineQuery](\n  injectionType: Option[String])\n    extends BaseClientEventDetailsBuilder[Query, BaseTweetCandidate] {\n\n  override def apply(\n    query: Query,\n    tweetCandidate: BaseTweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[ClientEventDetails] =\n    Some(\n      ClientEventDetails(\n        conversationDetails = None,\n        timelinesDetails = Some(\n          TimelinesDetails(\n            injectionType = injectionType,\n            controllerData = Some(buildControllerData(query.getUserOrGuestId)),\n            sourceData = None)),\n        articleDetails = None,\n        liveEventDetails = None,\n        commerceDetails = None\n      ))\n\n  private def buildControllerData(traceId: Option[Long]): String =\n    ControllerDataSerializer(\n      ControllerData.V2(\n        ControllerDataV2.HomeTweets(\n          HomeTweetsControllerData.V1(\n            HomeTweetsControllerDataV1(\n              tweetTypesBitmap = 0L,\n              traceId = traceId,\n            )\n          )\n        )\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/metadata/StaticUrlBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.metadata\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseUrlBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.UrlType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class StaticUrlBuilder(url: String, urlType: UrlType)\n    extends BaseUrlBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: UniversalNoun[Any],\n    candidateFeatures: FeatureMap\n  ): Url = Url(url = url, urlType = urlType)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/metadata/TopicClientEventDetailsBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.metadata\n\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.bijection.Base64String\nimport com.twitter.bijection.{Injection => Serializer}\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTopicCandidate\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.suggests.controller_data.thriftscala.ControllerData\nimport com.twitter.suggests.controller_data.timelines_topic.thriftscala.TimelinesTopicControllerData\nimport com.twitter.suggests.controller_data.timelines_topic.v1.thriftscala.{\n  TimelinesTopicControllerData => TimelinesTopicControllerDataV1\n}\nimport com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}\n\nobject TopicClientEventDetailsBuilder {\n  implicit val ByteSerializer: Serializer[ControllerData, Array[Byte]] =\n    BinaryScalaCodec(ControllerData)\n\n  val ControllerDataSerializer: Serializer[ControllerData, String] =\n    Serializer.connect[ControllerData, Array[Byte], Base64String, String]\n}\n\ncase class TopicClientEventDetailsBuilder[-Query <: PipelineQuery]()\n    extends BaseClientEventDetailsBuilder[Query, BaseTopicCandidate] {\n\n  import TopicClientEventDetailsBuilder._\n\n  override def apply(\n    query: Query,\n    topicCandidate: BaseTopicCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[ClientEventDetails] =\n    Some(\n      ClientEventDetails(\n        conversationDetails = None,\n        timelinesDetails = Some(\n          TimelinesDetails(\n            injectionType = None,\n            controllerData = buildControllerData(topicCandidate.id),\n            sourceData = None)),\n        articleDetails = None,\n        liveEventDetails = None,\n        commerceDetails = None\n      ))\n\n  private def buildControllerData(topicId: Long): Option[String] =\n    Some(\n      ControllerData\n        .V2(ControllerDataV2.TimelinesTopic(TimelinesTopicControllerData.V1(\n          TimelinesTopicControllerDataV1(topicTypesBitmap = 0L, topicId = topicId)))))\n      .map(ControllerDataSerializer)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/metadata/TopicNotInterestedFeedbackActionInfoBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.metadata\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTopicCandidate\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichBehavior\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorMarkNotInterestedTopic\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class TopicNotInterestedFeedbackActionInfoBuilder[-Query <: PipelineQuery]()\n    extends BaseFeedbackActionInfoBuilder[Query, BaseTopicCandidate] {\n\n  override def apply(\n    query: Query,\n    topicCandidate: BaseTopicCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackActionInfo] = {\n    Some(\n      FeedbackActionInfo(\n        feedbackActions = Seq(\n          FeedbackAction(\n            feedbackType = RichBehavior,\n            richBehavior = Some(\n              RichFeedbackBehaviorMarkNotInterestedTopic(topicCandidate.id.toString)\n            ),\n            hasUndoAction = Some(true),\n            prompt = None,\n            confirmation = None,\n            feedbackUrl = None,\n            clientEventInfo = None,\n            childFeedbackActions = None,\n            confirmationDisplayType = None,\n            icon = None,\n            subprompt = None,\n            encodedFeedbackRequest = None\n          )\n        ),\n        feedbackMetadata = None,\n        displayContext = None,\n        clientEventInfo = None\n      ))\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/metadata/TopicTweetClientEventDetailsBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.metadata\n\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.bijection.Base64String\nimport com.twitter.bijection.{Injection => Serializer}\nimport com.twitter.interests_mixer.model.request.{HasTopicId => InterestsMixerHasTopicId}\nimport com.twitter.explore_mixer.model.request.{HasTopicId => ExploreMixerHasTopicId}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.suggests.controller_data.home_tweets.thriftscala.HomeTweetsControllerData\nimport com.twitter.suggests.controller_data.home_tweets.v1.thriftscala.{\n  HomeTweetsControllerData => HomeTweetsControllerDataV1\n}\nimport com.twitter.suggests.controller_data.thriftscala.ControllerData\nimport com.twitter.suggests.controller_data.v2.thriftscala.{ControllerData => ControllerDataV2}\n\nobject TopicTweetClientEventDetailsBuilder {\n  implicit val ByteSerializer: Serializer[ControllerData, Array[Byte]] =\n    BinaryScalaCodec(ControllerData)\n\n  val ControllerDataSerializer: Serializer[ControllerData, String] =\n    Serializer.connect[ControllerData, Array[Byte], Base64String, String]\n}\n\ncase class TopicTweetClientEventDetailsBuilder[-Query <: PipelineQuery]()\n    extends BaseClientEventDetailsBuilder[Query, TweetCandidate] {\n\n  import TopicTweetClientEventDetailsBuilder._\n\n  override def apply(\n    query: Query,\n    topicTweetCandidate: TweetCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[ClientEventDetails] =\n    Some(\n      ClientEventDetails(\n        conversationDetails = None,\n        timelinesDetails = Some(\n          TimelinesDetails(\n            injectionType = None,\n            controllerData = buildControllerData(getTopicId(query)),\n            sourceData = None)),\n        articleDetails = None,\n        liveEventDetails = None,\n        commerceDetails = None\n      ))\n\n  private def getTopicId(query: Query): Option[Long] = {\n    query match {\n      case query: InterestsMixerHasTopicId => query.topicId\n      case query: ExploreMixerHasTopicId => query.topicId\n      case _ => None\n    }\n  }\n\n  private def buildControllerData(topicId: Option[Long]): Option[String] =\n    Some(\n      ControllerData\n        .V2(ControllerDataV2.HomeTweets(HomeTweetsControllerData.V1(\n          HomeTweetsControllerDataV1(tweetTypesBitmap = 0L, topicId = topicId)))))\n      .map(ControllerDataSerializer)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/metadata/TopicsToFollowModuleMetadataBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.metadata\n\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleMetadataBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.GridCarouselMetadata\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleMetadata\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\nobject TopicsToFollowModuleMetadataBuilder {\n\n  val TopicsPerRow = 7\n\n  /*\n   * rows = min(MAX_NUM_ROWS, # topics / TOPICS_PER_ROW)\n   * where TOPICS_PER_ROW = 7\n   */\n  def getCarouselRowCount(topicsCount: Int, maxCarouselRows: Int): Int =\n    Math.min(maxCarouselRows, (topicsCount / TopicsPerRow) + 1)\n}\n\ncase class TopicsToFollowModuleMetadataBuilder(maxCarouselRowsParam: Param[Int])\n    extends BaseModuleMetadataBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  import TopicsToFollowModuleMetadataBuilder._\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[UniversalNoun[Any]]]\n  ): ModuleMetadata = {\n    val rowCount = getCarouselRowCount(candidates.size, query.params(maxCarouselRowsParam))\n    ModuleMetadata(\n      adsMetadata = None,\n      conversationMetadata = None,\n      gridCarouselMetadata = Some(GridCarouselMetadata(numRows = Some(rowCount)))\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/metadata/WhoToFollowFeedbackActionInfoBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.metadata\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.Str\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.Frown\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SeeFewer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stringcenter.client.ExternalStringRegistry\nimport com.twitter.stringcenter.client.StringCenter\n\ncase class WhoToFollowFeedbackActionInfoBuilder[\n  -Query <: PipelineQuery,\n  -Candidate <: UniversalNoun[Any]\n](\n  externalStringRegistry: ExternalStringRegistry,\n  stringCenter: StringCenter,\n  encodedFeedbackRequest: Option[String])\n    extends BaseFeedbackActionInfoBuilder[Query, Candidate] {\n\n  private val seeLessOftenFeedback =\n    externalStringRegistry.createProdString(\"Feedback.seeLessOften\")\n  private val seeLessOftenConfirmationFeedback =\n    externalStringRegistry.createProdString(\"Feedback.seeLessOftenConfirmation\")\n\n  override def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackActionInfo] = Some(\n    FeedbackActionInfo(\n      feedbackActions = Seq(\n        FeedbackAction(\n          feedbackType = SeeFewer,\n          prompt = Some(\n            Str(seeLessOftenFeedback, stringCenter, None)\n              .apply(query, candidate, candidateFeatures)),\n          confirmation = Some(\n            Str(seeLessOftenConfirmationFeedback, stringCenter, None)\n              .apply(query, candidate, candidateFeatures)),\n          childFeedbackActions = None,\n          feedbackUrl = None,\n          confirmationDisplayType = None,\n          clientEventInfo = None,\n          richBehavior = None,\n          subprompt = None,\n          icon = Some(Frown), // ignored by unsupported clients\n          hasUndoAction = Some(true),\n          encodedFeedbackRequest = encodedFeedbackRequest\n        )\n      ),\n      feedbackMetadata = None,\n      displayContext = None,\n      clientEventInfo = None\n    )\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/operation/CursorCandidateUrtOperationBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.operation\n\nimport com.twitter.product_mixer.component_library.model.candidate.CursorCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.CandidateUrtEntryBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorDisplayTreatment\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorOperation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class CursorCandidateUrtOperationBuilder[-Query <: PipelineQuery](\n  cursorType: CursorType,\n  displayTreatment: Option[CursorDisplayTreatment] = None,\n  idToReplace: Option[Long] = None)\n    extends CandidateUrtEntryBuilder[Query, CursorCandidate, CursorOperation] {\n\n  override def apply(\n    query: Query,\n    cursorCandidate: CursorCandidate,\n    candidateFeatures: FeatureMap\n  ): CursorOperation = CursorOperation(\n    id = cursorCandidate.id,\n    sortIndex = None, // Sort indexes are automatically set in the domain marshaller phase\n    value = cursorCandidate.value,\n    cursorType = cursorType,\n    displayTreatment = displayTreatment,\n    idToReplace = idToReplace\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/promoted/FeaturePromotedMetadataBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.promoted\n\nimport com.twitter.ads.adserver.{thriftscala => ads}\nimport com.twitter.ads.common.base.{thriftscala => ac}\nimport com.twitter.adserver.{thriftscala => ad}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.promoted.BasePromotedMetadataBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.util.AdMetadataContainerSerializer\n\ncase class FeaturePromotedMetadataBuilder(adImpressionFeature: Feature[_, Option[ad.AdImpression]])\n    extends BasePromotedMetadataBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  def apply(\n    query: PipelineQuery,\n    candidate: UniversalNoun[Any],\n    candidateFeatures: FeatureMap\n  ): Option[PromotedMetadata] = {\n    candidateFeatures.getOrElse(adImpressionFeature, None).map { impression =>\n      PromotedMetadata(\n        advertiserId = impression.advertiserId,\n        disclosureType = impression.disclosureType.map(convertDisclosureType),\n        experimentValues = impression.experimentValues.map(_.toMap),\n        promotedTrendId = impression.promotedTrendId.map(_.toLong),\n        promotedTrendName = impression.promotedTrendName,\n        promotedTrendQueryTerm = impression.promotedTrendQueryTerm,\n        adMetadataContainer =\n          impression.serializedAdMetadataContainer.flatMap(convertAdMetadataContainer),\n        promotedTrendDescription = impression.promotedTrendDescription,\n        impressionString = impression.impressionString,\n        clickTrackingInfo = impression.clickTrackingInfo.map(convertClickTrackingInfo),\n      )\n    }\n  }\n\n  private def convertAdMetadataContainer(\n    serializedAdMetadataContainer: ac.SerializedThrift\n  ): Option[AdMetadataContainer] =\n    AdMetadataContainerSerializer.deserialize(serializedAdMetadataContainer).map { container =>\n      AdMetadataContainer(\n        removePromotedAttributionForPreroll = container.removePromotedAttributionForPreroll,\n        sponsorshipCandidate = container.sponsorshipCandidate,\n        sponsorshipOrganization = container.sponsorshipOrganization,\n        sponsorshipOrganizationWebsite = container.sponsorshipOrganizationWebsite,\n        sponsorshipType = container.sponsorshipType.map(convertSponsorshipType),\n        disclaimerType = container.disclaimerType.map(convertDisclaimerType),\n        skAdNetworkDataList = container.skAdNetworkDataList.map(convertSkAdNetworkDataList),\n        unifiedCardOverride = container.unifiedCardOverride\n      )\n    }\n\n  private def convertDisclosureType(disclosureType: ad.DisclosureType): DisclosureType =\n    disclosureType match {\n      case ad.DisclosureType.None => NoDisclosure\n      case ad.DisclosureType.Political => Political\n      case ad.DisclosureType.Earned => Earned\n      case ad.DisclosureType.Issue => Issue\n      case _ => throw new UnsupportedOperationException(s\"Unsupported: $disclosureType\")\n    }\n\n  private def convertSponsorshipType(sponsorshipType: ads.SponsorshipType): SponsorshipType =\n    sponsorshipType match {\n      case ads.SponsorshipType.Direct => DirectSponsorshipType\n      case ads.SponsorshipType.Indirect => IndirectSponsorshipType\n      case ads.SponsorshipType.NoSponsorship => NoSponsorshipSponsorshipType\n      case _ => throw new UnsupportedOperationException(s\"Unsupported: $sponsorshipType\")\n    }\n\n  private def convertDisclaimerType(disclaimerType: ads.DisclaimerType): DisclaimerType =\n    disclaimerType match {\n      case ads.DisclaimerType.Political => DisclaimerPolitical\n      case ads.DisclaimerType.Issue => DisclaimerIssue\n      case _ => throw new UnsupportedOperationException(s\"Unsupported: $disclaimerType\")\n    }\n\n  private def convertSkAdNetworkDataList(\n    skAdNetworkDataList: Seq[ads.SkAdNetworkData]\n  ): Seq[SkAdNetworkData] = skAdNetworkDataList.map { sdAdNetwork =>\n    SkAdNetworkData(\n      version = sdAdNetwork.version,\n      srcAppId = sdAdNetwork.srcAppId,\n      dstAppId = sdAdNetwork.dstAppId,\n      adNetworkId = sdAdNetwork.adNetworkId,\n      campaignId = sdAdNetwork.campaignId,\n      impressionTimeInMillis = sdAdNetwork.impressionTimeInMillis,\n      nonce = sdAdNetwork.nonce,\n      signature = sdAdNetwork.signature,\n      fidelityType = sdAdNetwork.fidelityType\n    )\n  }\n\n  private def convertClickTrackingInfo(clickTracking: ad.ClickTrackingInfo): ClickTrackingInfo =\n    ClickTrackingInfo(\n      urlParams = clickTracking.urlParams.getOrElse(Map.empty),\n      urlOverride = clickTracking.urlOverride,\n      urlOverrideType = clickTracking.urlOverrideType.map {\n        case ad.UrlOverrideType.Unknown => UnknownUrlOverrideType\n        case ad.UrlOverrideType.Dcm => DcmUrlOverrideType\n        case ad.UrlOverrideType.EnumUnknownUrlOverrideType(value) =>\n          throw new UnsupportedOperationException(s\"Unsupported: $value\")\n      }\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/richtext/RichTextBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.richtext\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseStr\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.richtext.BaseRichTextBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.UrlType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextAlignment\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class RichTextBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  textBuilder: BaseStr[Query, Candidate],\n  linkMap: Map[String, String],\n  rtl: Option[Boolean],\n  alignment: Option[RichTextAlignment],\n  linkTypeMap: Map[String, UrlType] = Map.empty)\n    extends BaseRichTextBuilder[Query, Candidate] {\n\n  def apply(query: Query, candidate: Candidate, candidateFeatures: FeatureMap): RichText = {\n    RichTextMarkupUtil.richTextFromMarkup(\n      text = textBuilder(query, candidate, candidateFeatures),\n      linkMap = linkMap,\n      rtl = rtl,\n      alignment = alignment,\n      linkTypeMap = linkTypeMap)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/richtext/RichTextMarkupUtil.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.richtext\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ExternalUrl\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.UrlType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextAlignment\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextEntity\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Strong\n\n/*\n * RichTextMarkupUtil facilitates building a Product Mixer URT RichText object out of\n * a string with inline XML markup.\n *\n * This allows us to use a string like \"Our system <a href=\"#promix\">Product Mixer</a> is the <b>best</b>\". Using\n * inline markup like this is advantageous since the string can go through translation/localization and the\n * translators will move the tags around in each language as appropriate.\n *\n * This class is derived from the OCF (onboarding/serve)'s RichTextUtil, but they diverge because:\n * - We generate ProMix URT structures, not OCF URT structures\n * - The OCF supports some internal OCF tags, like <data>\n * - The OCF has additional legacy support and processing that we don't need\n */\n\nobject RichTextMarkupUtil {\n\n  // Matches a anchor element, extracting the 'a' tag and the display text.\n  // First group is the tag\n  // Second group is the display text\n  // Allows any character in the display text, but matches reluctantly\n  private val LinkAnchorRegex = \"\"\"(?i)(?s)<a\\s+href\\s*=\\s*\"#([\\w-]*)\">(.*?)</a>\"\"\".r\n\n  // Matches a <b>bold text section</b>\n  private val BoldRegex = \"\"\"(?i)(?s)<b>(.*?)</b>\"\"\".r\n\n  def richTextFromMarkup(\n    text: String,\n    linkMap: Map[String, String],\n    rtl: Option[Boolean] = None,\n    alignment: Option[RichTextAlignment] = None,\n    linkTypeMap: Map[String, UrlType] = Map.empty\n  ): RichText = {\n\n    // Mutable!\n    var currentText = text\n    val entities = scala.collection.mutable.ArrayBuffer.empty[RichTextEntity]\n\n    // Using a while loop since we want to execute the regex after each iteration, so our indexes remain consistent\n\n    // Handle Links\n    var matchOpt = LinkAnchorRegex.findFirstMatchIn(currentText)\n    while (matchOpt.isDefined) {\n      matchOpt.foreach { linkMatch =>\n        val tag = linkMatch.group(1)\n        val displayText = linkMatch.group(2)\n\n        currentText = currentText.substring(0, linkMatch.start) + displayText + currentText\n          .substring(linkMatch.end)\n\n        adjustEntities(\n          entities,\n          linkMatch.start,\n          linkMatch.end - (linkMatch.start + displayText.length))\n\n        entities.append(\n          RichTextEntity(\n            fromIndex = linkMatch.start,\n            toIndex = linkMatch.start + displayText.length,\n            ref = linkMap.get(tag).map { url =>\n              Url(\n                urlType = linkTypeMap.getOrElse(tag, ExternalUrl),\n                url = url\n              )\n            },\n            format = None\n          )\n        )\n      }\n      matchOpt = LinkAnchorRegex.findFirstMatchIn(currentText)\n    }\n\n    // Handle Bold\n    matchOpt = BoldRegex.findFirstMatchIn(currentText)\n    while (matchOpt.isDefined) {\n      matchOpt.foreach { boldMatch =>\n        val text = boldMatch.group(1)\n\n        currentText =\n          currentText.substring(0, boldMatch.start) + text + currentText.substring(boldMatch.end)\n\n        adjustEntities(entities, boldMatch.start, boldMatch.end - (boldMatch.start + text.length))\n\n        entities.append(\n          RichTextEntity(\n            fromIndex = boldMatch.start,\n            toIndex = boldMatch.start + text.length,\n            ref = None,\n            format = Some(Strong),\n          )\n        )\n      }\n\n      matchOpt = BoldRegex.findFirstMatchIn(currentText)\n    }\n\n    RichText(\n      currentText,\n      entities.sortBy(_.fromIndex).toList, // always return immutable copies!\n      rtl,\n      alignment\n    )\n  }\n\n  /* When we create a new entity, we need to adjust\n   * any already existing entities that have been moved.\n   * Entities cannot overlap, so we can just compare start positions.\n   */\n  private def adjustEntities(\n    entities: scala.collection.mutable.ArrayBuffer[RichTextEntity],\n    start: Int,\n    length: Int\n  ): Unit = {\n    for (i <- entities.indices) {\n      if (entities(i).fromIndex > start) {\n        val old = entities(i)\n        entities.update(\n          i,\n          entities(i).copy(\n            fromIndex = old.fromIndex - length,\n            toIndex = old.toIndex - length\n          ))\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/richtext/RichTextReferenceObjectBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.richtext\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.ReferenceObject\nimport com.twitter.twittertext.Extractor\n\ntrait RichTextReferenceObjectBuilder {\n  def apply(entity: Extractor.Entity): Option[ReferenceObject]\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/richtext/RichTextRtlOptionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.richtext\n\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait RichTextRtlOptionBuilder[-Query <: PipelineQuery] {\n  def apply(query: Query): Option[Boolean]\n}\n\ncase class StaticRichTextRtlOptionBuilder[-Query <: PipelineQuery](rtlOption: Option[Boolean])\n    extends RichTextRtlOptionBuilder[Query] {\n  override def apply(query: Query): Option[Boolean] = rtlOption\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/richtext/StaticRichTextBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.richtext\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.richtext.BaseRichTextBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class StaticRichTextBuilder(richText: RichText)\n    extends BaseRichTextBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: UniversalNoun[Any],\n    candidateFeatures: FeatureMap\n  ): RichText = richText\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/richtext/twitter_text/TwitterTextEntityProcessor.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.richtext.twitter_text\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.richtext.RichTextReferenceObjectBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.richtext.twitter_text.TwitterTextEntityProcessor.DefaultReferenceObjectBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ExternalUrl\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.ReferenceObject\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextCashtag\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextHashtag\nimport com.twitter.twittertext.Extractor\nimport scala.collection.convert.ImplicitConversions._\n\nobject TwitterTextEntityProcessor {\n  object DefaultReferenceObjectBuilder extends RichTextReferenceObjectBuilder {\n    def apply(twitterEntity: Extractor.Entity): Option[ReferenceObject] = {\n      twitterEntity.getType match {\n        case Extractor.Entity.Type.URL =>\n          Some(Url(ExternalUrl, twitterEntity.getValue))\n        case Extractor.Entity.Type.HASHTAG =>\n          Some(RichTextHashtag(twitterEntity.getValue))\n        case Extractor.Entity.Type.CASHTAG =>\n          Some(RichTextCashtag(twitterEntity.getValue))\n        case _ => None\n      }\n    }\n  }\n}\n\n/**\n * Add the corresponding  [[RichTextEntity]] extraction logic into [[TwitterTextRenderer]].\n * The [[TwitterTextRenderer]] after being processed will extract the defined entities.\n */\ncase class TwitterTextEntityProcessor(\n  twitterTextReferenceObjectBuilder: RichTextReferenceObjectBuilder = DefaultReferenceObjectBuilder)\n    extends TwitterTextRendererProcessor {\n\n  private[this] val extractor = new Extractor()\n\n  def process(\n    twitterTextRenderer: TwitterTextRenderer\n  ): TwitterTextRenderer = {\n    val twitterEntities = extractor.extractEntitiesWithIndices(twitterTextRenderer.text)\n\n    twitterEntities.foreach { twitterEntity =>\n      twitterTextReferenceObjectBuilder(twitterEntity).foreach { refObject =>\n        twitterTextRenderer.setRefObject(twitterEntity.getStart, twitterEntity.getEnd, refObject)\n      }\n    }\n    twitterTextRenderer\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/richtext/twitter_text/TwitterTextFormatProcessor.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.richtext.twitter_text\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Plain\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextFormat\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Strong\nimport scala.collection.mutable\n\nobject TwitterTextFormatProcessor {\n  lazy val defaultFormatProcessor = TwitterTextFormatProcessor()\n}\n\n/**\n * Add the corresponding [[RichTextFormat]] extraction logic into [[TwitterTextRenderer]].\n * The [[TwitterTextRenderer]] after being processed will extract the defined entities. \n */\ncase class TwitterTextFormatProcessor(\n  formats: Set[RichTextFormat] = Set(Plain, Strong),\n) extends TwitterTextRendererProcessor {\n\n  private val formatMap = formats.map { format => format.name.toLowerCase -> format }.toMap\n\n  private[this] val formatMatcher = {\n    val formatNames = formatMap.keys.toSet\n    s\"<(/?)(${formatNames.mkString(\"|\")})>\".r\n  }\n\n  def renderText(text: String): RichText = {\n    process(TwitterTextRenderer(text)).build\n  }\n\n  def process(richTextBuilder: TwitterTextRenderer): TwitterTextRenderer = {\n    val text = richTextBuilder.text\n    val nodeStack = mutable.ArrayStack[(RichTextFormat, Int)]()\n    var offset = 0\n\n    formatMatcher.findAllMatchIn(text).foreach { m =>\n      formatMap.get(m.group(2)) match {\n        case Some(format) => {\n          if (m.group(1).nonEmpty) {\n            if (!nodeStack.headOption.exists {\n                case (formatFromStack, _) => formatFromStack == format\n              }) {\n              throw UnmatchedFormatTag(format)\n            }\n            val (_, startIndex) = nodeStack.pop\n            richTextBuilder.mergeFormat(startIndex, m.start + offset, format)\n          } else {\n            nodeStack.push((format, m.start + offset))\n          }\n          richTextBuilder.remove(m.start + offset, m.end + offset)\n          offset -= m.end - m.start\n        }\n        case _ => // if format is not found, skip this format\n      }\n    }\n\n    if (nodeStack.nonEmpty) {\n      throw UnmatchedFormatTag(nodeStack.head._1)\n    }\n\n    richTextBuilder\n  }\n}\n\ncase class UnmatchedFormatTag(format: RichTextFormat)\n    extends Exception(s\"Unmatched format start and end tags for $format\")\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/richtext/twitter_text/TwitterTextRenderer.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.richtext.twitter_text\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.ReferenceObject\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextAlignment\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextEntity\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextFormat\nimport scala.annotation.tailrec\nimport scala.collection.mutable\n\nobject TwitterTextRenderer {\n\n  /**\n   * Creates a new [[TwitterTextRenderer]] instance.\n   * @param text      The initial text representation\n   * @param rtl       Defines whether this text is in an RTL language\n   * @param alignment Assigns the [[RichTextAlignment]] of the given text for display\n   * @return          A new [[TwitterTextRenderer]] instance\n   */\n  def apply(\n    text: String,\n    rtl: Option[Boolean] = None,\n    alignment: Option[RichTextAlignment] = None\n  ): TwitterTextRenderer = {\n    TwitterTextRenderer(rtl, alignment).append(text)\n  }\n\n  /**\n   * Creates a new [[TwitterTextRenderer]] instance from a product-mixer [[RichText]] object.\n   * Converts Unicode entity indexes into JVM String indexes.\n   * @param richText  The product-mixer [[RichText]] representation\n   * @return          A new [[TwitterTextRenderer]] instance\n   */\n  def fromRichText(richText: RichText): TwitterTextRenderer = {\n    val builder = TwitterTextRenderer(richText.text, richText.rtl, richText.alignment)\n    richText.entities.foreach { e =>\n      val startIndex = richText.text.offsetByCodePoints(0, e.fromIndex)\n      val endIndex = richText.text.offsetByCodePoints(0, e.toIndex)\n      e.format.foreach { f =>\n        builder.setFormat(startIndex, endIndex, f)\n      }\n      e.ref.foreach { r =>\n        builder.setRefObject(startIndex, endIndex, r)\n      }\n    }\n    builder\n  }\n\n  private def buildRichTextEntity(\n    text: String,\n    entity: TwitterTextRendererEntity[_]\n  ): RichTextEntity = {\n    val fromIndex = text.codePointCount(0, entity.startIndex)\n    val toIndex = text.codePointCount(0, entity.endIndex)\n\n    entity.value match {\n      case format: RichTextFormat =>\n        RichTextEntity(fromIndex, toIndex, ref = None, format = Some(format))\n      case ref: ReferenceObject =>\n        RichTextEntity(fromIndex, toIndex, ref = Some(ref), format = None)\n    }\n  }\n}\n\ncase class TwitterTextRenderer(\n  rtl: Option[Boolean],\n  alignment: Option[RichTextAlignment],\n) {\n  private[this] val textBuilder = new mutable.StringBuilder()\n\n  private[richtext] val formatBuffer =\n    mutable.ArrayBuffer[TwitterTextRendererEntity[RichTextFormat]]()\n  private[richtext] val refObjectBuffer =\n    mutable.ArrayBuffer[TwitterTextRendererEntity[ReferenceObject]]()\n\n  /**\n   * Appends a string with attached [[RichTextFormat]] and [[ReferenceObject]] information.\n   * @param string    The text to append to the end of the existing text\n   * @param format    The [[RichTextFormat]] assigned to the new text\n   * @param refObject The [[ReferenceObject]] assigned to the new text\n   * @return          this\n   */\n  def append(\n    string: String,\n    format: Option[RichTextFormat] = None,\n    refObject: Option[ReferenceObject] = None\n  ): TwitterTextRenderer = {\n    if (string.nonEmpty) {\n      val start = textBuilder.length\n      val end = start + string.length\n      format.foreach { f =>\n        formatBuffer.append(TwitterTextRendererEntity(start, end, f))\n      }\n      refObject.foreach { r =>\n        refObjectBuffer.append(TwitterTextRendererEntity(start, end, r))\n      }\n      textBuilder.append(string)\n    }\n    this\n  }\n\n  /**\n   * Builds a new [[RichText]] thrift instance with Unicode entity ranges.\n   */\n  def build: RichText = {\n    val richTextString = this.text\n    val richTextEntities = this.entities\n      .map { e =>\n        TwitterTextRenderer.buildRichTextEntity(richTextString, e)\n      }\n\n    RichText(\n      text = richTextString,\n      rtl = rtl,\n      alignment = alignment,\n      entities = richTextEntities.toList\n    )\n  }\n\n  /**\n   * Modifies the TwitterTextRenderer with the provided [[TwitterTextRendererProcessor]]\n   */\n  def transform(twitterTextProcessor: TwitterTextRendererProcessor): TwitterTextRenderer = {\n    twitterTextProcessor.process(this)\n  }\n\n  /**\n   * Builds and returns a sorted list of [[TwitterTextRendererEntity]] with JVM String index entity ranges.\n   */\n  def entities: Seq[TwitterTextRendererEntity[_]] = {\n    buildEntities(formatBuffer.toList, refObjectBuffer.toList)\n  }\n\n  /**\n   * Assigns a [[RichTextFormat]] to the given range while keeping existing formatting information.\n   * New formatting will only be assigned to unformatted text ranges.\n   * @param start  Start index to apply formatting (inclusive)\n   * @param end    End index to apply formatting (exclusive)\n   * @param format The format to assign\n   * @return       this\n   */\n  def mergeFormat(start: Int, end: Int, format: RichTextFormat): TwitterTextRenderer = {\n    validateRange(start, end)\n    var injectionIndex: Option[Int] = None\n    var entity = TwitterTextRendererEntity(start, end, format)\n\n    val buffer = mutable.ArrayBuffer[TwitterTextRendererEntity[RichTextFormat]]()\n    val iterator = formatBuffer.zipWithIndex.reverseIterator\n\n    while (iterator.hasNext && injectionIndex.isEmpty) {\n      iterator.next match {\n        case (e, i) if e.startIndex >= end =>\n          buffer.append(e)\n\n        case (e, i) if e.enclosedIn(entity.startIndex, entity.endIndex) =>\n          val endEntity = entity.copy(startIndex = e.endIndex)\n          if (endEntity.nonEmpty) { buffer.append(endEntity) }\n          buffer.append(e)\n          entity = entity.copy(endIndex = e.startIndex)\n\n        case (e, i) if e.encloses(entity.startIndex, entity.endIndex) =>\n          buffer.append(e.copy(startIndex = entity.endIndex))\n          buffer.append(e.copy(endIndex = entity.startIndex))\n          injectionIndex = Some(i + 1)\n\n        case (e, i) if e.startsBetween(entity.startIndex, entity.endIndex) =>\n          buffer.append(e)\n          entity = entity.copy(endIndex = e.startIndex)\n\n        case (e, i) if e.endsBetween(entity.startIndex, entity.endIndex) =>\n          buffer.append(e)\n          entity = entity.copy(startIndex = e.endIndex)\n          injectionIndex = Some(i + 1)\n\n        case (e, i) if e.endIndex <= entity.startIndex =>\n          buffer.append(e)\n          injectionIndex = Some(i + 1)\n\n        case _ => // do nothing\n      }\n    }\n\n    val index = injectionIndex.map(_ - 1).getOrElse(0)\n    formatBuffer.remove(index, formatBuffer.length - index)\n    formatBuffer.appendAll(buffer.reverse)\n\n    if (entity.nonEmpty) {\n      formatBuffer.insert(injectionIndex.getOrElse(0), entity)\n    }\n\n    this\n  }\n\n  /**\n   * Removes text, formatting, and refObject information from the given range.\n   * @param start  Start index to apply formatting (inclusive)\n   * @param end    End index to apply formatting (exclusive)\n   * @return       this\n   */\n  def remove(start: Int, end: Int): TwitterTextRenderer = replace(start, end, \"\")\n\n  /**\n   * Replaces text, formatting, and refObject information in the given range.\n   * @param start     Start index to apply formatting (inclusive)\n   * @param end       End index to apply formatting (exclusive)\n   * @param string    The new text to insert\n   * @param format    The [[RichTextFormat]] assigned to the new text\n   * @param refObject The [[ReferenceObject]] assigned to the new text\n   * @return          this\n   */\n  def replace(\n    start: Int,\n    end: Int,\n    string: String,\n    format: Option[RichTextFormat] = None,\n    refObject: Option[ReferenceObject] = None\n  ): TwitterTextRenderer = {\n    validateRange(start, end)\n\n    val newEnd = start + string.length\n    val formatInjectIndex = removeAndOffsetFormats(start, end, string.length)\n    val refObjectInjectIndex = removeAndOffsetRefObjects(start, end, string.length)\n    format.foreach { f =>\n      formatBuffer.insert(formatInjectIndex, TwitterTextRendererEntity(start, newEnd, f))\n    }\n    refObject.foreach { r =>\n      refObjectBuffer.insert(refObjectInjectIndex, TwitterTextRendererEntity(start, newEnd, r))\n    }\n    textBuilder.replace(start, end, string)\n\n    this\n  }\n\n  /**\n   * Assigns a [[RichTextFormat]] to the given range. Trims existing format ranges that overlap the\n   * new format range. Removes format ranges that fall within the new range.\n   * @param start  Start index to apply formatting (inclusive)\n   * @param end    End index to apply formatting (exclusive)\n   * @param format The format to assign\n   * @return       this\n   */\n  def setFormat(start: Int, end: Int, format: RichTextFormat): TwitterTextRenderer = {\n    validateRange(start, end)\n    val bufferIndex = removeAndOffsetFormats(start, end, end - start)\n    formatBuffer.insert(bufferIndex, TwitterTextRendererEntity(start, end, format))\n\n    this\n  }\n\n  private[this] def removeAndOffsetFormats(start: Int, end: Int, newSize: Int): Int = {\n    val newEnd = start + newSize\n    val offset = newEnd - end\n    var injectionIndex: Option[Int] = None\n\n    val buffer = mutable.ArrayBuffer[TwitterTextRendererEntity[RichTextFormat]]()\n    val iterator = formatBuffer.zipWithIndex.reverseIterator\n\n    while (iterator.hasNext && injectionIndex.isEmpty) {\n      iterator.next match {\n        case (e, i) if e.startIndex >= end =>\n          buffer.append(e.offset(offset))\n        case (e, i) if e.encloses(start, end) =>\n          buffer.append(e.copy(startIndex = newEnd, endIndex = e.endIndex + offset))\n          buffer.append(e.copy(endIndex = e.endIndex + offset))\n          injectionIndex = Some(i + 1)\n        case (e, i) if e.endsBetween(start, end) =>\n          buffer.append(e.copy(endIndex = start))\n          injectionIndex = Some(i + 1)\n        case (e, i) if e.startsBetween(start, end) =>\n          buffer.append(e.copy(startIndex = newEnd, endIndex = e.endIndex + offset))\n        case (e, i) if e.endIndex <= start =>\n          buffer.append(e)\n          injectionIndex = Some(i + 1)\n        case _ => // do nothing\n      }\n    }\n    val index = injectionIndex.map(_ - 1).getOrElse(0)\n    formatBuffer.remove(index, formatBuffer.length - index)\n    formatBuffer.appendAll(buffer.reverse)\n\n    injectionIndex.getOrElse(0)\n  }\n\n  private[this] def validateRange(start: Int, end: Int): Unit = {\n    require(\n      start >= 0 && start < textBuilder.length && end > start && end <= textBuilder.length,\n      s\"The start ($start) and end ($end) indexes must be within the text range (0..${textBuilder.length})\"\n    )\n  }\n\n  /**\n   * Assigns a [[ReferenceObject]] to the given range. Since it makes little sense to trim object\n   * ranges, existing intersecting or overlapping ranges are removed entirely.\n   * @param start  Start index to apply formatting (inclusive)\n   * @param end       End index to apply formatting (exclusive)\n   * @param refObject The [[ReferenceObject]] to assign\n   * @return          this\n   */\n  def setRefObject(start: Int, end: Int, refObject: ReferenceObject): TwitterTextRenderer = {\n    validateRange(start, end)\n    val bufferIndex = removeAndOffsetRefObjects(start, end, end - start)\n    refObjectBuffer.insert(bufferIndex, TwitterTextRendererEntity(start, end, refObject))\n\n    this\n  }\n\n  private[this] def removeAndOffsetRefObjects(start: Int, end: Int, newSize: Int): Int = {\n    val newEnd = start + newSize\n    val offset = newEnd - end\n    var injectionIndex: Option[Int] = None\n\n    val buffer = mutable.ArrayBuffer[TwitterTextRendererEntity[ReferenceObject]]()\n    val iterator = refObjectBuffer.zipWithIndex.reverseIterator\n\n    while (iterator.hasNext && injectionIndex.isEmpty) {\n      iterator.next match {\n        case (e, i) if e.startIndex >= end => buffer.append(e.offset(offset))\n        case (e, i) if e.endIndex <= start => injectionIndex = Some(i + 1)\n        case _ => // do nothing\n      }\n    }\n    val index = injectionIndex.getOrElse(0)\n    refObjectBuffer.remove(index, refObjectBuffer.length - index)\n    refObjectBuffer.appendAll(buffer.reverse)\n\n    index\n  }\n\n  /**\n   * Builds and returns the full TwitterTextRenderer text with any changes applied to the builder instance.\n   */\n  def text: String = {\n    textBuilder.mkString\n  }\n\n  @tailrec\n  private def buildEntities(\n    formats: List[TwitterTextRendererEntity[RichTextFormat]],\n    refs: List[TwitterTextRendererEntity[ReferenceObject]],\n    acc: List[TwitterTextRendererEntity[_]] = List()\n  ): Seq[TwitterTextRendererEntity[_]] = {\n    (formats, refs) match {\n      case (Nil, Nil) => acc\n      case (remainingFormats, Nil) => acc ++ remainingFormats\n      case (Nil, remainingRefs) => acc ++ remainingRefs\n\n      case (format +: remainingFormats, ref +: remainingRefs)\n          if format.startIndex < ref.startIndex || (format.startIndex == ref.startIndex && format.endIndex < ref.endIndex) =>\n        buildEntities(remainingFormats, refs, acc :+ format)\n\n      case (format +: remainingFormats, ref +: remainingRefs)\n          if format.startIndex == ref.startIndex && format.endIndex == ref.endIndex =>\n        buildEntities(remainingFormats, remainingRefs, acc :+ format :+ ref)\n\n      case (_, ref +: remainingRefs) =>\n        buildEntities(formats, remainingRefs, acc :+ ref)\n    }\n  }\n}\n\ncase class TwitterTextRendererEntity[+T] private[richtext] (\n  startIndex: Int,\n  endIndex: Int,\n  value: T) {\n  require(startIndex <= endIndex, \"startIndex must be <= than endIndex\")\n\n  def nonEmpty: Boolean = !isEmpty\n\n  def isEmpty: Boolean = startIndex == endIndex\n\n  private[richtext] def enclosedIn(start: Int, end: Int): Boolean = {\n    start <= startIndex && endIndex <= end\n  }\n\n  private[richtext] def encloses(start: Int, end: Int): Boolean = {\n    startIndex < start && end < endIndex\n  }\n\n  private[richtext] def endsBetween(start: Int, end: Int): Boolean = {\n    start < endIndex && endIndex <= end && startIndex < start\n  }\n\n  private[richtext] def offset(num: Int): TwitterTextRendererEntity[T] = {\n    copy(startIndex = startIndex + num, endIndex = endIndex + num)\n  }\n\n  private[richtext] def startsBetween(start: Int, end: Int): Boolean = {\n    startIndex >= start && startIndex < end && endIndex > end\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/richtext/twitter_text/TwitterTextRendererProcessor.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.richtext.twitter_text\n\ntrait TwitterTextRendererProcessor {\n  def process(twitterTextRichTextBuilder: TwitterTextRenderer): TwitterTextRenderer\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/richtext/twitter_text/TwitterTextRichTextBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.richtext.twitter_text\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.richtext.RichTextReferenceObjectBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.richtext.RichTextRtlOptionBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.richtext.StaticRichTextRtlOptionBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.richtext.twitter_text.TwitterTextEntityProcessor.DefaultReferenceObjectBuilder\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseStr\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.richtext.BaseRichTextBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Plain\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextAlignment\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextFormat\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Strong\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class TwitterTextRichTextBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  stringBuilder: BaseStr[Query, Candidate],\n  alignment: Option[RichTextAlignment] = None,\n  formats: Set[RichTextFormat] = Set(Plain, Strong),\n  twitterTextRtlOptionBuilder: RichTextRtlOptionBuilder[Query] =\n    StaticRichTextRtlOptionBuilder[Query](None),\n  twitterTextReferenceObjectBuilder: RichTextReferenceObjectBuilder = DefaultReferenceObjectBuilder)\n    extends BaseRichTextBuilder[Query, Candidate] {\n  def apply(query: Query, candidate: Candidate, candidateFeatures: FeatureMap): RichText = {\n    val twitterTextRenderer = TwitterTextRenderer(\n      text = stringBuilder(query, candidate, candidateFeatures),\n      rtl = twitterTextRtlOptionBuilder(query),\n      alignment = alignment)\n\n    twitterTextRenderer\n      .transform(TwitterTextFormatProcessor(formats))\n      .transform(TwitterTextEntityProcessor(twitterTextReferenceObjectBuilder))\n      .build\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/social_context/FeatureSocialContextBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.social_context\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.service.{thriftscala => t}\n\n/**\n * Use this Builder to create Product Mixer [[SocialContext]] objects when you have a\n * Timeline Service Thrift [[SocialContext]] feature that you want to convert\n */\ncase class FeatureSocialContextBuilder(\n  socialContextFeature: Feature[_, Option[t.SocialContext]])\n    extends BaseSocialContextBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: UniversalNoun[Any],\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext] = {\n    candidateFeatures.getOrElse(socialContextFeature, None).map {\n      case t.SocialContext.GeneralContext(context) =>\n        val contextType = context.contextType match {\n          case t.ContextType.Like => LikeGeneralContextType\n          case t.ContextType.Follow => FollowGeneralContextType\n          case t.ContextType.Moment => MomentGeneralContextType\n          case t.ContextType.Reply => ReplyGeneralContextType\n          case t.ContextType.Conversation => ConversationGeneralContextType\n          case t.ContextType.Pin => PinGeneralContextType\n          case t.ContextType.TextOnly => TextOnlyGeneralContextType\n          case t.ContextType.Facepile => FacePileGeneralContextType\n          case t.ContextType.Megaphone => MegaPhoneGeneralContextType\n          case t.ContextType.Bird => BirdGeneralContextType\n          case t.ContextType.Feedback => FeedbackGeneralContextType\n          case t.ContextType.Topic => TopicGeneralContextType\n          case t.ContextType.List => ListGeneralContextType\n          case t.ContextType.Retweet => RetweetGeneralContextType\n          case t.ContextType.Location => LocationGeneralContextType\n          case t.ContextType.Community => CommunityGeneralContextType\n          case t.ContextType.SmartBlockExpiration => SmartblockExpirationGeneralContextType\n          case t.ContextType.Trending => TrendingGeneralContextType\n          case t.ContextType.Sparkle => SparkleGeneralContextType\n          case t.ContextType.Spaces => SpacesGeneralContextType\n          case t.ContextType.ReplyPin => ReplyPinGeneralContextType\n          case t.ContextType.NewUser => NewUserGeneralContextType\n          case t.ContextType.EnumUnknownContextType(field) =>\n            throw new UnsupportedOperationException(s\"Unknown context type: $field\")\n        }\n\n        val landingUrl = context.landingUrl.map { url =>\n          val endpointOptions = url.urtEndpointOptions.map { options =>\n            UrtEndpointOptions(\n              requestParams = options.requestParams.map(_.toMap),\n              title = options.title,\n              cacheId = options.cacheId,\n              subtitle = options.subtitle\n            )\n          }\n\n          val urlType = url.urlType match {\n            case t.UrlType.ExternalUrl => ExternalUrl\n            case t.UrlType.DeepLink => DeepLink\n            case t.UrlType.UrtEndpoint => UrtEndpoint\n            case t.UrlType.EnumUnknownUrlType(field) =>\n              throw new UnsupportedOperationException(s\"Unknown url type: $field\")\n          }\n\n          Url(urlType = urlType, url = url.url, urtEndpointOptions = endpointOptions)\n        }\n\n        GeneralContext(\n          text = context.text,\n          contextType = contextType,\n          url = context.url,\n          contextImageUrls = context.contextImageUrls.map(_.toList),\n          landingUrl = landingUrl\n        )\n      case t.SocialContext.TopicContext(context) =>\n        val functionalityType = context.functionalityType match {\n          case t.TopicContextFunctionalityType.Basic =>\n            BasicTopicContextFunctionalityType\n          case t.TopicContextFunctionalityType.Recommendation =>\n            RecommendationTopicContextFunctionalityType\n          case t.TopicContextFunctionalityType.RecWithEducation =>\n            RecWithEducationTopicContextFunctionalityType\n          case t.TopicContextFunctionalityType.EnumUnknownTopicContextFunctionalityType(field) =>\n            throw new UnsupportedOperationException(s\"Unknown functionality type: $field\")\n        }\n\n        TopicContext(\n          topicId = context.topicId,\n          functionalityType = Some(functionalityType)\n        )\n      case t.SocialContext.UnknownUnionField(field) =>\n        throw new UnsupportedOperationException(s\"Unknown social context: $field\")\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/social_context/GeneralModuleSocialContextBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.social_context\n\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseModuleStr\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseModuleSocialContextBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.GeneralContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.GeneralContextType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * This class works the same as [[GeneralSocialContextBuilder]] but passes a list of candidates\n * into [[BaseModuleStr]] when rendering the string.\n */\ncase class GeneralModuleSocialContextBuilder[\n  -Query <: PipelineQuery,\n  -Candidate <: UniversalNoun[Any]\n](\n  textBuilder: BaseModuleStr[Query, Candidate],\n  contextType: GeneralContextType,\n  url: Option[String] = None,\n  contextImageUrls: Option[List[String]] = None,\n  landingUrl: Option[Url] = None)\n    extends BaseModuleSocialContextBuilder[Query, Candidate] {\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Option[GeneralContext] =\n    Some(\n      GeneralContext(\n        text = textBuilder(query, candidates),\n        contextType = contextType,\n        url = url,\n        contextImageUrls = contextImageUrls,\n        landingUrl = landingUrl))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/social_context/GeneralSocialContextBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.social_context\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseStr\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.GeneralContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.GeneralContextType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class GeneralSocialContextBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  textBuilder: BaseStr[Query, Candidate],\n  contextType: GeneralContextType,\n  url: Option[String] = None,\n  contextImageUrls: Option[List[String]] = None,\n  landingUrl: Option[Url] = None)\n    extends BaseSocialContextBuilder[Query, Candidate] {\n\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[GeneralContext] =\n    Some(\n      GeneralContext(\n        text = textBuilder(query, candidate, candidateFeatures),\n        contextType = contextType,\n        url = url,\n        contextImageUrls = contextImageUrls,\n        landingUrl = landingUrl))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/social_context/WhoToFollowSocialContextBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.social_context\n\nimport com.twitter.hermit.{thriftscala => h}\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseSocialContextBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FollowGeneralContextType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.GeneralContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.GeneralContextType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.LocationGeneralContextType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.NewUserGeneralContextType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class WhoToFollowSocialContextBuilder(\n  socialTextFeature: Feature[_, Option[String]],\n  contextTypeFeature: Feature[_, Option[h.ContextType]])\n    extends BaseSocialContextBuilder[PipelineQuery, UserCandidate] {\n\n  def apply(\n    query: PipelineQuery,\n    candidate: UserCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[GeneralContext] = {\n    val socialTextOpt = candidateFeatures.getOrElse(socialTextFeature, None)\n    val contextTypeOpt = convertContextType(candidateFeatures.getOrElse(contextTypeFeature, None))\n\n    (socialTextOpt, contextTypeOpt) match {\n      case (Some(socialText), Some(contextType)) if socialText.nonEmpty =>\n        Some(\n          GeneralContext(\n            text = socialText,\n            contextType = contextType,\n            url = None,\n            contextImageUrls = None,\n            landingUrl = None))\n      case _ => None\n    }\n  }\n\n  private def convertContextType(contextType: Option[h.ContextType]): Option[GeneralContextType] =\n    contextType match {\n      case Some(h.ContextType.Geo) => Some(LocationGeneralContextType)\n      case Some(h.ContextType.Social) => Some(FollowGeneralContextType)\n      case Some(h.ContextType.NewUser) => Some(NewUserGeneralContextType)\n      case _ => None\n    }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/stringcenter/ModuleStr.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter\n\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseModuleStr\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.stringcenter.BaseModuleStringCenterPlaceholderBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.stringcenter.client.core.ExternalString\n\n/**\n * This class works the same as [[Str]] but passes in a list of candidates to the\n * [[BaseModuleStringCenterPlaceholderBuilder]] when building the placeholders.\n */\ncase class ModuleStr[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  text: ExternalString,\n  stringCenter: StringCenter,\n  stringCenterPlaceholderBuilder: Option[\n    BaseModuleStringCenterPlaceholderBuilder[Query, Candidate]\n  ] = None)\n    extends BaseModuleStr[Query, Candidate] {\n\n  def apply(query: Query, candidates: Seq[CandidateWithFeatures[Candidate]]): String = {\n    val placeholderMapOpt =\n      stringCenterPlaceholderBuilder.map(_.apply(query, candidates))\n    stringCenter.prepare(\n      externalString = text,\n      placeholders = placeholderMapOpt.getOrElse(Map.empty[String, Any])\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/stringcenter/Str.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseStr\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.stringcenter.BaseStringCenterPlaceholderBuilder\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.stringcenter.client.core.ExternalString\n\ncase class StrStatic(\n  text: String)\n    extends BaseStr[PipelineQuery, UniversalNoun[Any]] {\n  def apply(\n    query: PipelineQuery,\n    candidate: UniversalNoun[Any],\n    candidateFeatures: FeatureMap\n  ): String = text\n}\n\ncase class Str[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  text: ExternalString,\n  stringCenter: StringCenter,\n  stringCenterPlaceholderBuilder: Option[BaseStringCenterPlaceholderBuilder[Query, Candidate]] =\n    None)\n    extends BaseStr[Query, Candidate] {\n\n  def apply(query: Query, candidate: Candidate, candidateFeatures: FeatureMap): String = {\n    val placeholderMapOpt =\n      stringCenterPlaceholderBuilder.map(_.apply(query, candidate, candidateFeatures))\n    stringCenter.prepare(\n      externalString = text,\n      placeholders = placeholderMapOpt.getOrElse(Map.empty[String, Any])\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/timeline_module/FeatureModuleDisplayTypeBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class FeatureModuleDisplayTypeBuilder(\n  displayTypeFeature: Feature[_, Option[ModuleDisplayType]],\n  defaultDisplayType: ModuleDisplayType = VerticalConversation)\n    extends BaseModuleDisplayTypeBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[UniversalNoun[Any]]]\n  ): ModuleDisplayType = candidates.headOption\n    .flatMap(_.features.getOrElse(displayTypeFeature, None))\n    .getOrElse(defaultDisplayType)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/timeline_module/ModuleDynamicShowMoreBehaviorRevealByCountBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleShowMoreBehaviorBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleShowMoreBehavior\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleShowMoreBehaviorRevealByCount\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class ModuleDynamicShowMoreBehaviorRevealByCountBuilder(\n  initialItemsCount: Int,\n  showMoreItemsCount: Int)\n    extends BaseModuleShowMoreBehaviorBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: Seq[CandidateWithFeatures[UniversalNoun[Any]]]\n  ): ModuleShowMoreBehavior = ModuleShowMoreBehaviorRevealByCount(\n    initialItemsCount = initialItemsCount,\n    showMoreItemsCount = showMoreItemsCount\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/timeline_module/ModuleFooterBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleFooter\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseStr\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseUrlBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleFooterBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\n\ncase class ModuleFooterBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  textBuilder: BaseStr[Query, Candidate],\n  urlBuilder: Option[BaseUrlBuilder[Query, Candidate]])\n    extends BaseModuleFooterBuilder[Query, Candidate] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Option[ModuleFooter] = {\n    candidates.headOption.map { candidate =>\n      ModuleFooter(\n        text = textBuilder(query, candidate.candidate, candidate.features),\n        landingUrl = urlBuilder.map(_.apply(query, candidate.candidate, candidate.features))\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/timeline_module/ModuleHeaderBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.icon.BaseHorizonIconBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseStr\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context.BaseModuleSocialContextBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleHeaderBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleHeaderDisplayTypeBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Classic\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageVariant\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleHeader\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class ModuleHeaderBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  textBuilder: BaseStr[Query, Candidate],\n  isSticky: Option[Boolean] = None,\n  moduleHeaderIconBuilder: Option[BaseHorizonIconBuilder[Query, Candidate]] = None,\n  customIcon: Option[ImageVariant] = None,\n  moduleSocialContextBuilder: Option[BaseModuleSocialContextBuilder[Query, Candidate]] = None,\n  moduleHeaderDisplayTypeBuilder: BaseModuleHeaderDisplayTypeBuilder[Query, Candidate] =\n    ModuleHeaderDisplayTypeBuilder(Classic))\n    extends BaseModuleHeaderBuilder[Query, Candidate] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Option[ModuleHeader] = {\n    val firstCandidate = candidates.head\n    Some(\n      ModuleHeader(\n        text = textBuilder(query, firstCandidate.candidate, firstCandidate.features),\n        sticky = isSticky,\n        customIcon = customIcon,\n        socialContext = moduleSocialContextBuilder.flatMap(_.apply(query, candidates)),\n        icon = moduleHeaderIconBuilder.flatMap(_.apply(query, candidates)),\n        moduleHeaderDisplayType = moduleHeaderDisplayTypeBuilder(query, candidates),\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/timeline_module/ModuleHeaderDisplayTypeBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleHeaderDisplayTypeBuilder\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Classic\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleHeaderDisplayType\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class ModuleHeaderDisplayTypeBuilder[\n  -Query <: PipelineQuery,\n  -Candidate <: UniversalNoun[Any]\n](\n  moduleHeaderDisplayType: ModuleHeaderDisplayType = Classic)\n    extends BaseModuleHeaderDisplayTypeBuilder[Query, Candidate] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): ModuleHeaderDisplayType = moduleHeaderDisplayType\n\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/timeline_module/ModuleIdGeneration.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module\n\n/**\n *  This trait is used for Module ID generation. Clients are safe to ignore this code unless they\n *  have a specific use case that requires hard-coded, specific, module ids.  In that scenario,\n *  they can use the [[ManualModuleId]] case class.\n */\nsealed trait ModuleIdGeneration {\n  val moduleId: Long\n}\n\nobject ModuleIdGeneration {\n  def apply(moduleId: Long): ModuleIdGeneration = moduleId match {\n    case moduleId if AutomaticUniqueModuleId.isAutomaticUniqueModuleId(moduleId) =>\n      AutomaticUniqueModuleId(moduleId)\n    case moduleId => ManualModuleId(moduleId)\n  }\n}\n\n/**\n * Generate unique Ids for each module, which results in unique URT entryIds\n * for each module even if they share the same entryNamespace.\n * This is the default and recommended use case.\n * Note that the module Id value is just a placeholder\n */\ncase class AutomaticUniqueModuleId private (moduleId: Long = 0L) extends ModuleIdGeneration {\n  def withOffset(offset: Long): AutomaticUniqueModuleId = copy(\n    AutomaticUniqueModuleId.idRange.min + offset)\n}\n\nobject AutomaticUniqueModuleId {\n  // We use a specific numeric range to track whether IDs should be automatically generated.\n  val idRange: Range = Range(-10000, -1000)\n\n  def apply(): AutomaticUniqueModuleId = AutomaticUniqueModuleId(idRange.min)\n\n  def isAutomaticUniqueModuleId(moduleId: Long): Boolean = idRange.contains(moduleId)\n}\n\n/**\n * ManualModuleId should normally not be required, but is helpful if the\n * entryId of the module must be controlled. A scenario where this may be\n * required is if a single candidate source returns multiple modules, and\n * each module has the same presentation (e.g. Header, Footer). By setting\n * different IDs, we signal to the platform that each module should be separate\n * by using a different manual Id.\n */\ncase class ManualModuleId(override val moduleId: Long) extends ModuleIdGeneration {\n  // Negative module IDs are reserved for internal usage\n  if (moduleId < 0) throw new IllegalArgumentException(\"moduleId must be a positive number\")\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/timeline_module/ModuleShowMoreBehaviorRevealByCountBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleShowMoreBehavior\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleShowMoreBehaviorRevealByCount\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleShowMoreBehaviorBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.timelines.configapi.Param\n\ncase class ModuleShowMoreBehaviorRevealByCountBuilder(\n  initialItemsCountParam: Param[Int],\n  showMoreItemsCountParam: Param[Int])\n    extends BaseModuleShowMoreBehaviorBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: Seq[CandidateWithFeatures[UniversalNoun[Any]]]\n  ): ModuleShowMoreBehavior = {\n    ModuleShowMoreBehaviorRevealByCount(\n      initialItemsCount = query.params(initialItemsCountParam),\n      showMoreItemsCount = query.params(showMoreItemsCountParam)\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/timeline_module/ParamGatedModuleFooterBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleFooterBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleFooter\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\ncase class ParamGatedModuleFooterBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  enableParam: Param[Boolean],\n  enabledBuilder: BaseModuleFooterBuilder[Query, Candidate],\n  defaultBuilder: Option[BaseModuleFooterBuilder[Query, Candidate]] = None)\n    extends BaseModuleFooterBuilder[Query, Candidate] {\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Option[ModuleFooter] = {\n    if (query.params(enableParam)) {\n      enabledBuilder(query, candidates)\n    } else {\n      defaultBuilder.flatMap(_.apply(query, candidates))\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/timeline_module/ParamGatedModuleHeaderBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleHeaderBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleHeader\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\ncase class ParamGatedModuleHeaderBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  enableParam: Param[Boolean],\n  enabledBuilder: BaseModuleHeaderBuilder[Query, Candidate],\n  defaultBuilder: Option[BaseModuleHeaderBuilder[Query, Candidate]] = None)\n    extends BaseModuleHeaderBuilder[Query, Candidate] {\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Option[ModuleHeader] = {\n    if (query.params(enableParam)) {\n      enabledBuilder(query, candidates)\n    } else {\n      defaultBuilder.flatMap(_.apply(query, candidates))\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/timeline_module/ParamWhoToFollowModuleDisplayTypeBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Carousel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.CompactCarousel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ConversationTree\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.GridCarousel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Vertical\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalGrid\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalWithContextLine\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\nobject WhoToFollowModuleDisplayType extends Enumeration {\n  type ModuleDisplayType = Value\n\n  val Carousel = Value\n  val CompactCarousel = Value\n  val ConversationTree = Value\n  val GridCarousel = Value\n  val Vertical = Value\n  val VerticalConversation = Value\n  val VerticalGrid = Value\n  val VerticalWithContextLine = Value\n}\n\ncase class ParamWhoToFollowModuleDisplayTypeBuilder(\n  displayTypeParam: Param[WhoToFollowModuleDisplayType.Value] =\n    StaticParam(WhoToFollowModuleDisplayType.Vertical))\n    extends BaseModuleDisplayTypeBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[UniversalNoun[Any]]]\n  ): ModuleDisplayType = {\n    val displayType = query.params(displayTypeParam)\n    displayType match {\n      case WhoToFollowModuleDisplayType.Carousel => Carousel\n      case WhoToFollowModuleDisplayType.CompactCarousel => CompactCarousel\n      case WhoToFollowModuleDisplayType.ConversationTree => ConversationTree\n      case WhoToFollowModuleDisplayType.GridCarousel => GridCarousel\n      case WhoToFollowModuleDisplayType.Vertical => Vertical\n      case WhoToFollowModuleDisplayType.VerticalConversation => VerticalConversation\n      case WhoToFollowModuleDisplayType.VerticalGrid => VerticalGrid\n      case WhoToFollowModuleDisplayType.VerticalWithContextLine => VerticalWithContextLine\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/timeline_module/StaticModuleDisplayTypeBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleDisplayType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class StaticModuleDisplayTypeBuilder(displayType: ModuleDisplayType)\n    extends BaseModuleDisplayTypeBuilder[PipelineQuery, UniversalNoun[Any]] {\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[UniversalNoun[Any]]]\n  ): ModuleDisplayType = displayType\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt/builder/timeline_module/TimelineModuleBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleFooterBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleHeaderBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleMetadataBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleShowMoreBehaviorBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseTimelineModuleBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\n\ncase class TimelineModuleBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  entryNamespace: EntryNamespace,\n  displayTypeBuilder: BaseModuleDisplayTypeBuilder[Query, Candidate],\n  clientEventInfoBuilder: BaseClientEventInfoBuilder[Query, Candidate],\n  moduleIdGeneration: ModuleIdGeneration = AutomaticUniqueModuleId(),\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, Candidate]\n  ] = None,\n  headerBuilder: Option[BaseModuleHeaderBuilder[Query, Candidate]] = None,\n  footerBuilder: Option[BaseModuleFooterBuilder[Query, Candidate]] = None,\n  metadataBuilder: Option[BaseModuleMetadataBuilder[Query, Candidate]] = None,\n  showMoreBehaviorBuilder: Option[BaseModuleShowMoreBehaviorBuilder[Query, Candidate]] = None)\n    extends BaseTimelineModuleBuilder[Query, Candidate] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): TimelineModule = {\n    val firstCandidate = candidates.head\n    TimelineModule(\n      id = moduleIdGeneration.moduleId,\n      // Sort indexes are automatically set in the domain marshaller phase\n      sortIndex = None,\n      entryNamespace = entryNamespace,\n      // Modules should not need an element by default; only items should\n      clientEventInfo =\n        clientEventInfoBuilder(query, firstCandidate.candidate, firstCandidate.features, None),\n      feedbackActionInfo = feedbackActionInfoBuilder.flatMap(\n        _.apply(query, firstCandidate.candidate, firstCandidate.features)),\n      isPinned = None,\n      // Items are automatically set in the domain marshaller phase\n      items = Seq.empty,\n      displayType = displayTypeBuilder(query, candidates),\n      header = headerBuilder.flatMap(_.apply(query, candidates)),\n      footer = footerBuilder.flatMap(_.apply(query, candidates)),\n      metadata = metadataBuilder.map(_.apply(query, candidates)),\n      showMoreBehavior = showMoreBehaviorBuilder.map(_.apply(query, candidates))\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/experiments/metrics/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/github/scopt\",\n        \"util/util-core/src/main/java/com/twitter/io\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/experiments/metrics/MetricDefinitions.scala",
    "content": "package com.twitter.product_mixer.component_library.experiments.metrics\n\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\n\nobject MetricDefinition {\n  val SingleQuote = \"\"\"\"\"\"\"\n  val DoubleQuote = \"\"\"\"\"\"\"\"\n}\n\n/**\n * Base class for all metric definitions\n */\nsealed trait MetricDefinition {\n  def toCsvField: Seq[String]\n  val metricDefinitionType: String\n}\n\n/**\n * Pattern Metric Definition\n * @param pattern the regex pattern for this metric\n */\ncase class NamedPatternMetricDefinition(\n  pattern: Seq[String])\n    extends MetricDefinition {\n  override def toCsvField: Seq[String] = pattern\n  override val metricDefinitionType: String = \"NAMED_PATTERN\"\n}\n\n/**\n * Strainer Metric Definition\n * @param strainerExpression a filter on top of client events\n */\ncase class StrainerMetricDefinition(\n  strainerExpression: String)\n    extends MetricDefinition {\n  import MetricDefinition._\n  override def toCsvField: Seq[String] = {\n    Seq(strainerExpression.replaceAll(SingleQuote, DoubleQuote))\n  }\n  override val metricDefinitionType: String = \"STRAINER\"\n}\n\n/**\n * Lambda Metric Definition\n * @param lambdaExpression a scala function mapping client events to a double\n */\ncase class LambdaMetricDefinition(\n  lambdaExpression: String)\n    extends MetricDefinition {\n  import MetricDefinition._\n  override def toCsvField: Seq[String] = {\n    Seq(lambdaExpression.replaceAll(SingleQuote, DoubleQuote))\n  }\n  override val metricDefinitionType: String = \"LAMBDA\"\n}\n\ncase class BucketRatioMetricDefinition(\n  numerator: String,\n  denominator: String)\n    extends MetricDefinition {\n  override def toCsvField: Seq[String] = {\n    Seq(s\"(${numerator}) / (${denominator})\")\n  }\n  override val metricDefinitionType: String = \"BUCKET_RATIO\"\n}\n\nobject Metric {\n  val bucketRatioPattern = \"[(]+(.+)[)]+ / [(]+(.+)[)]+\".r\n\n  /**\n   * Creates a new Metric given a template line.\n   * @param line semicolon separated line string\n   * ignore line with comment, represented by hashtag at the beginning of the line\n   * @throws RuntimeException if the line is invalid\n   */\n  def fromLine(line: String): Metric = {\n    val splits = line.split(\";\")\n    // at least two parts separated by semicolon (third part is optional)\n    if (splits.lengthCompare(2) >= 0) {\n      val metricExpression = splits(0)\n      val metricName = splits(1)\n      val metricDefinition = Try(splits(2)) match {\n        case Return(\"NAMED_PATTERN\") => NamedPatternMetricDefinition(Seq(metricExpression))\n        case Return(\"STRAINER\") => StrainerMetricDefinition(metricExpression)\n        case Return(\"LAMBDA\") => LambdaMetricDefinition(metricExpression)\n        case Return(\"BUCKET_RATIO\") =>\n          metricExpression match {\n            case bucketRatioPattern(numerator, denominator) =>\n              BucketRatioMetricDefinition(numerator, denominator)\n            case _ =>\n              throw new RuntimeException(\n                s\"Invalid metric definition for Bucket Ratio. Expected format (numerator)<space>/<space>(denominator) but found $metricExpression\")\n          }\n        case Return(other) =>\n          throw new RuntimeException(s\"Invalid metric definition in line in template file: $line\")\n        // default to named pattern\n        case Throw(_) => NamedPatternMetricDefinition(List(metricExpression))\n      }\n\n      Metric(metricName, metricDefinition)\n    } else {\n      throw new RuntimeException(s\"Invalid line in template file: $line\")\n    }\n  }\n}\n\n/**\n *\n * @param name globally unique metric name (current DDG limitation)\n * @param definition the metric definition for this metric\n */\ncase class Metric(\n  name: String,\n  definition: MetricDefinition)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/experiments/metrics/MetricGroup.scala",
    "content": "package com.twitter.product_mixer.component_library.experiments.metrics\n\nimport scala.collection.immutable.ListSet\n\n/**\n *\n * @param id optional metric group id. If id is None, this means the group\n *           is being newly created and the id is not provisioned by go/ddg. Otherwise, the metric\n *           group is present in DDG and has a corresponding id.\n * @param name metric group name\n * @param description metric group description\n * @param metrics set of metrics that belong to this metric group\n */\ncase class MetricGroup(\n  id: Option[Long],\n  name: String,\n  description: String,\n  metrics: ListSet[Metric]) {\n\n  /*\n   * Returns a CSV representation of this metric group that can be imported via DDG's bulk import tool\n   * The bulk import tool consumes CSV data with the following columns:\n   * 1. group name\n   * 2. group description\n   * 3. metric name\n   * 4. metric description\n   * 5. metric pattern\n   * 6. group id -- numeric id\n   * 7. (optional) metric type -- `NAMED_PATTERN`, `STRAINER`, or `LAMBDA`.\n   */\n  def toCsv: String = {\n    val metricCsvLines: ListSet[String] = for {\n      metric <- metrics\n      definition <- metric.definition.toCsvField\n    } yield {\n      Seq(\n        name,\n        description,\n        metric.name,\n        metric.name,\n        // wrap in single quotes so that DDG bulk import tool correctly parses\n        s\"\"\"\"$definition\"\"\"\",\n        id.map(_.toString).getOrElse(\"\"),\n        metric.definition.metricDefinitionType\n      ).mkString(\",\")\n    }\n    println(s\"Generated metrics in CSV count: ${metricCsvLines.size}\")\n    metricCsvLines.mkString(\"\\n\")\n  }\n\n  // Unique metric names based on globally unique metric name\n  def uniqueMetricNames: Set[String] =\n    metrics.groupBy(_.name).keys.toSet\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/experiments/metrics/MetricTemplateCLIRunner.scala",
    "content": "package com.twitter.product_mixer.component_library.experiments.metrics\n\nimport com.twitter.product_mixer.component_library.experiments.metrics.PlaceholderConfig.PlaceholdersMap\nimport java.io.File\nimport java.io.PrintWriter\nimport scala.collection.immutable.ListSet\nimport scala.io.Source\nimport scopt.OptionParser\n\nprivate case class MetricTemplateCLIConfig(\n  // default values required for OptionParser\n  templateFileName: String = null,\n  outputFileName: String = null,\n  metricGroupName: String = null,\n  metricGroupDesc: String = null,\n  metricGroupId: Option[Long] = None,\n  absolutePath: Option[String] = None)\n\ntrait MetricTemplateCLIRunner {\n  def templateDir: String\n  def placeholders: PlaceholdersMap\n  private val ProgramName = \"Metric Template CLI\"\n  private val VersionNumber = \"1.0\"\n\n  private def mkPath(fileName: String, absolutePath: Option[String]): String = {\n    val relativeDir = s\"$templateDir/$fileName\"\n    absolutePath match {\n      case Some(path) => s\"$path/$relativeDir\"\n      case _ => relativeDir\n    }\n  }\n\n  def main(args: Array[String]): Unit = {\n    val parser = new OptionParser[MetricTemplateCLIConfig](ProgramName) {\n      head(ProgramName, VersionNumber)\n      // option invoked by -o or --output\n      opt[String]('o', \"output\")\n        .required()\n        .valueName(\"<file>\")\n        .action((value, config) => config.copy(outputFileName = value))\n        .text(\"output CSV file with interpolated lines\")\n      // option invoked by -t or --template\n      opt[String]('t', \"template\")\n        .required()\n        .valueName(\"<file>\")\n        .action((value, config) => config.copy(templateFileName = value))\n        .text(\n          s\"input template file (see README.md for template format). Path is relative to $templateDir.\")\n      // option invoked by -n or --name\n      opt[String]('n', \"name\")\n        .required()\n        .valueName(\"<groupName>\")\n        .action((value, config) => config.copy(metricGroupName = value))\n        .text(\"metric group name\")\n      // option invoked by -d or --description\n      opt[String]('d', \"description\")\n        .required()\n        .valueName(\"<groupDescription>\")\n        .action((value, config) => config.copy(metricGroupDesc = value))\n        .text(\"metric group description\")\n      // option invoked by --id\n      opt[Long](\"id\")\n        .optional()\n        .valueName(\"<groupId>\")\n        .action((value, config) => config.copy(metricGroupId = Some(value)))\n        .text(\"metric group ID (metric MUST be created in go/ddg)\")\n      // option invoked by -p or --path\n      opt[String]('p', \"path\")\n        .optional()\n        .valueName(\"<directory>\")\n        .action((value, config) => config.copy(absolutePath = Some(value)))\n        .text(s\"absolute path pointing to the $templateDir. Required by bazel\")\n    }\n\n    parser.parse(args, MetricTemplateCLIConfig()) match {\n      case Some(config) =>\n        val templateLines =\n          Source.fromFile(mkPath(config.templateFileName, config.absolutePath)).getLines.toList\n        val interpolatedLines = templateLines\n          .filter(!_.startsWith(\"#\")).flatMap(MetricTemplates.interpolate(_, placeholders))\n        val writer = new PrintWriter(new File(mkPath(config.outputFileName, config.absolutePath)))\n        val metrics = interpolatedLines.map(Metric.fromLine)\n        println(s\"${metrics.size} metric definitions found in template file.\")\n        val dupMetrics = metrics.groupBy(identity).collect {\n          case (dup, lst) if lst.lengthCompare(1) > 0 => dup\n        }\n        println(s\"\\nWARNING: ${dupMetrics.size} Duplicate metric definition(s)\\n$dupMetrics\\n\")\n        val metricGroup = MetricGroup(\n          config.metricGroupId,\n          config.metricGroupName,\n          config.metricGroupDesc,\n          metrics.to[ListSet])\n        println(s\"${metricGroup.uniqueMetricNames.size} unique DDG metrics with \" +\n          s\"${metricGroup.metrics.size} metric definitions in '${metricGroup.name}' metric group.\")\n        writer.write(metricGroup.toCsv)\n        writer.close()\n      case _ =>\n      // arguments are bad, error message will have been displayed\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/experiments/metrics/MetricTemplates.scala",
    "content": "package com.twitter.product_mixer.component_library.experiments.metrics\n\nimport com.twitter.product_mixer.component_library.experiments.metrics.PlaceholderConfig.PlaceholdersMap\nimport reflect.ClassTag\nimport scala.reflect.runtime.universe._\nimport scala.util.matching.Regex\n\ncase class MatchedPlaceholder(outerKey: String, innerKey: Option[String] = None)\n\nobject MetricTemplates {\n  // Matches \"${placeholder}\" where `placeholder` is in a matched group\n  val PlaceholderPattern: Regex = \"\\\\$\\\\{([^\\\\}]+)\\\\}\".r.unanchored\n  // Matches \"${placeholder[index]}\" where `placeholder` and `index` are in different matched groups\n  val IndexedPlaceholderPattern: Regex = \"\\\\$\\\\{([^\\\\[]+)\\\\[([^\\\\]]+)\\\\]\\\\}\".r.unanchored\n  val DefaultFieldName = \"name\"\n\n  def interpolate(inputTemplate: String, placeholders: PlaceholdersMap): Seq[String] = {\n    val matchedPlaceholders = getMatchedPlaceholders(inputTemplate)\n    val groupedPlaceholders = matchedPlaceholders.groupBy(_.outerKey)\n    val placeholderKeyValues = getPlaceholderKeyValues(groupedPlaceholders, placeholders)\n    val (keys, values) = (placeholderKeyValues.map(_._1), placeholderKeyValues.map(_._2))\n    val cross: Seq[List[Named]] = crossProduct(values)\n    val mirror = runtimeMirror(getClass.getClassLoader) // necessary for reflection\n    for {\n      interpolatables <- cross\n    } yield {\n      assert(\n        keys.length == interpolatables.length,\n        s\"Unexpected length mismatch between $keys and $interpolatables\")\n      var replacementStr = inputTemplate\n      keys.zip(interpolatables).foreach {\n        case (key, interpolatable) =>\n          val accessors = caseAccessors(mirror, interpolatable)\n          groupedPlaceholders(key).foreach { placeholder: MatchedPlaceholder =>\n            val templateKey = generateTemplateKey(placeholder)\n            val fieldName = placeholder.innerKey.getOrElse(DefaultFieldName)\n            val fieldValue = getFieldValue(mirror, interpolatable, accessors, fieldName)\n            replacementStr = replacementStr.replaceAll(templateKey, fieldValue)\n          }\n      }\n      replacementStr\n    }\n  }\n\n  def getMatchedPlaceholders(inputTemplate: String): Seq[MatchedPlaceholder] = {\n    for {\n      matched <- PlaceholderPattern.findAllIn(inputTemplate).toSeq\n    } yield {\n      val matchedWithIndexOpt = IndexedPlaceholderPattern.findFirstMatchIn(matched)\n      val (outer, inner) = matchedWithIndexOpt\n        .map { matchedWithIndex =>\n          (matchedWithIndex.group(1), Some(matchedWithIndex.group(2)))\n        }.getOrElse((matched, None))\n      val outerKey = unwrap(outer)\n      val innerKey = inner.map(unwrap(_))\n      MatchedPlaceholder(outerKey, innerKey)\n    }\n  }\n\n  def unwrap(str: String): String =\n    str.stripPrefix(\"${\").stripSuffix(\"}\")\n\n  def wrap(str: String): String =\n    \"\\\\$\\\\{\" + str + \"\\\\}\"\n\n  def getPlaceholderKeyValues(\n    groupedPlaceholders: Map[String, Seq[MatchedPlaceholder]],\n    placeholders: PlaceholdersMap\n  ): Seq[(String, Seq[Named])] = {\n    groupedPlaceholders.toSeq\n      .map {\n        case (outerKey, _) =>\n          val placeholderValues = placeholders.getOrElse(\n            outerKey,\n            throw new RuntimeException(s\"Failed to find values of $outerKey in placeholders\"))\n          outerKey -> placeholderValues\n      }\n  }\n\n  def crossProduct[T](seqOfSeqOfItems: Seq[Seq[T]]): Seq[List[T]] = {\n    if (seqOfSeqOfItems.isEmpty) {\n      List(Nil)\n    } else {\n      for {\n        // for every item in the head list\n        item <- seqOfSeqOfItems.head\n        // for every result (List) based on the cross-product of tail\n        resultList <- crossProduct(seqOfSeqOfItems.tail)\n      } yield {\n        item :: resultList\n      }\n    }\n  }\n\n  def generateTemplateKey(matched: MatchedPlaceholder): String = {\n    matched.innerKey match {\n      case None => wrap(matched.outerKey)\n      case Some(innerKeyString) => wrap(matched.outerKey + \"\\\\[\" + innerKeyString + \"\\\\]\")\n    }\n  }\n\n  // Given an instance and a field name, use reflection to get its value.\n  def getFieldValue[T: ClassTag](\n    mirror: Mirror,\n    cls: T,\n    accessors: Map[String, MethodSymbol],\n    fieldName: String\n  ): String = {\n    val instance: InstanceMirror = mirror.reflect(cls)\n    val accessor = accessors.getOrElse(\n      fieldName,\n      throw new RuntimeException(s\"Failed to find value of $fieldName for $cls\"))\n    instance.reflectField(accessor).get.toString // .get is safe due to check above\n  }\n\n  // Given an instance, use reflection to get a mapping for field name -> symbol\n  def caseAccessors[T: ClassTag](mirror: Mirror, cls: T): Map[String, MethodSymbol] = {\n    val classSymbol = mirror.classSymbol(cls.getClass)\n    classSymbol.toType.members.collect {\n      case m: MethodSymbol if m.isCaseAccessor => (m.name.toString -> m)\n    }.toMap\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/experiments/metrics/PlaceholderConfig.scala",
    "content": "package com.twitter.product_mixer.component_library.experiments.metrics\n\n// Base trait for all placeholder values\nsealed trait Named {\n  def name: String\n}\n\ncase class Const(override val name: String) extends Named\n\n// contains only client event patterns\ncase class CEPattern(\n  override val name: String,\n  client: String = \"\",\n  page: String = \"\",\n  section: String = \"\",\n  component: String = \"\",\n  element: String = \"\",\n  action: String = \"\",\n  strainer: String = \"\")\n    extends Named {\n\n  override def toString: String = {\n    \"\\\"\" + client + \":\" + page + \":\" + section + \":\" + component + \":\" + element + \":\" + action + \"\\\"\"\n  }\n\n}\n\ncase class Topic(\n  override val name: String,\n  topicId: String = \"\")\n    extends Named\n\nobject PlaceholderConfig {\n  type PlaceholderKey = String\n  type Placeholder = Seq[Named]\n  type PlaceholdersMap = Map[PlaceholderKey, Placeholder]\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/featurestorev1/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/featurestorev1/FeatureStoreV1QueryUserIdFeature.scala",
    "content": "package com.twitter.product_mixer.component_library.feature.featurestorev1\n\nimport com.twitter.ml.api.transform.FeatureRenameTransform\nimport com.twitter.ml.featurestore.catalog.entities\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.ml.featurestore.lib.entity.Entity\nimport com.twitter.ml.featurestore.lib.entity.EntityWithId\nimport com.twitter.ml.featurestore.lib.feature.TimelinesAggregationFrameworkFeatureGroup\nimport com.twitter.ml.featurestore.lib.feature.{Feature => FSv1Feature}\nimport com.twitter.product_mixer.core.feature.featurestorev1._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\nimport scala.reflect.ClassTag\nobject FeatureStoreV1QueryUserIdFeature {\n  def apply[Query <: PipelineQuery, Value](\n    feature: FSv1Feature[UserId, Value],\n    legacyName: Option[String] = None,\n    defaultValue: Option[Value] = None,\n    enabledParam: Option[FSParam[Boolean]] = None\n  ): FeatureStoreV1Feature[Query, Query, _ <: EntityId, Value]\n    with FeatureStoreV1QueryFeature[Query, _ <: EntityId, Value] =\n    FeatureStoreV1QueryFeature(feature, QueryUserIdEntity, legacyName, defaultValue, enabledParam)\n}\n\nobject FeatureStoreV1QueryUserIdAggregateFeature {\n  def apply[Query <: PipelineQuery](\n    featureGroup: TimelinesAggregationFrameworkFeatureGroup[UserId],\n    enabledParam: Option[FSParam[Boolean]] = None,\n    keepLegacyNames: Boolean = false,\n    featureNameTransform: Option[FeatureRenameTransform] = None\n  ): FeatureStoreV1QueryFeatureGroup[Query, _ <: EntityId] =\n    FeatureStoreV1QueryFeatureGroup(\n      featureGroup,\n      QueryUserIdEntity,\n      enabledParam,\n      keepLegacyNames,\n      featureNameTransform)((implicitly[ClassTag[UserId]]))\n}\n\nobject QueryUserIdEntity extends FeatureStoreV1QueryEntity[PipelineQuery, UserId] {\n  override val entity: Entity[UserId] = entities.core.User\n\n  override def entityWithId(query: PipelineQuery): EntityWithId[UserId] =\n    entity.withId(UserId(query.getUserIdLoggedOutSupport))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/featurestorev1/FeatureStoreV1QueryUserIdTweetCandidateAuthorIdFeature.scala",
    "content": "package com.twitter.product_mixer.component_library.feature.featurestorev1\n\nimport com.twitter.ml.api.transform.FeatureRenameTransform\nimport com.twitter.ml.featurestore.catalog.entities\nimport com.twitter.ml.featurestore.lib.EdgeEntityId\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.ml.featurestore.lib.entity.Entity\nimport com.twitter.ml.featurestore.lib.entity.EntityWithId\nimport com.twitter.ml.featurestore.lib.feature.TimelinesAggregationFrameworkFeatureGroup\nimport com.twitter.ml.featurestore.lib.feature.{Feature => FSv1Feature}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetAuthorIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featurestorev1._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\nimport scala.reflect.ClassTag\n\nobject FeatureStoreV1QueryUserIdTweetCandidateAuthorIdFeature {\n  def apply[Query <: PipelineQuery, Value](\n    feature: FSv1Feature[EdgeEntityId[UserId, UserId], Value],\n    legacyName: Option[String] = None,\n    defaultValue: Option[Value] = None,\n    enabledParam: Option[FSParam[Boolean]] = None\n  ): FeatureStoreV1CandidateFeature[Query, TweetCandidate, _ <: EntityId, Value] =\n    FeatureStoreV1CandidateFeature(\n      feature,\n      QueryUserIdTweetCandidateAuthorIdEntity,\n      legacyName,\n      defaultValue,\n      enabledParam)\n}\n\nobject FeatureStoreV1QueryUserIdTweetCandidateAuthorIdAggregateFeature {\n  def apply[Query <: PipelineQuery](\n    featureGroup: TimelinesAggregationFrameworkFeatureGroup[EdgeEntityId[UserId, UserId]],\n    enabledParam: Option[FSParam[Boolean]] = None,\n    keepLegacyNames: Boolean = false,\n    featureNameTransform: Option[FeatureRenameTransform] = None\n  ): FeatureStoreV1CandidateFeatureGroup[Query, TweetCandidate, _ <: EntityId] =\n    FeatureStoreV1CandidateFeatureGroup(\n      featureGroup,\n      QueryUserIdTweetCandidateAuthorIdEntity,\n      enabledParam,\n      keepLegacyNames,\n      featureNameTransform\n    )(implicitly[ClassTag[EdgeEntityId[UserId, UserId]]])\n}\n\nobject QueryUserIdTweetCandidateAuthorIdEntity\n    extends FeatureStoreV1CandidateEntity[\n      PipelineQuery,\n      TweetCandidate,\n      EdgeEntityId[UserId, UserId]\n    ] {\n  override val entity: Entity[EdgeEntityId[UserId, UserId]] = entities.core.UserAuthor\n\n  override def entityWithId(\n    query: PipelineQuery,\n    tweet: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): EntityWithId[EdgeEntityId[UserId, UserId]] =\n    entity.withId(\n      EdgeEntityId(\n        UserId(query.getUserIdLoggedOutSupport),\n        UserId(existingFeatures.get(TweetAuthorIdFeature))))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/featurestorev1/FeatureStoreV1QueryUserIdTweetCandidateTweetIdFeature.scala",
    "content": "package com.twitter.product_mixer.component_library.feature.featurestorev1\n\nimport com.twitter.ml.api.transform.FeatureRenameTransform\nimport com.twitter.ml.featurestore.catalog.entities\nimport com.twitter.ml.featurestore.lib.EdgeEntityId\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.TweetId\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.ml.featurestore.lib.entity.Entity\nimport com.twitter.ml.featurestore.lib.entity.EntityWithId\nimport com.twitter.ml.featurestore.lib.feature.TimelinesAggregationFrameworkFeatureGroup\nimport com.twitter.ml.featurestore.lib.feature.{Feature => FSv1Feature}\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featurestorev1._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\nimport scala.reflect.ClassTag\n\nobject FeatureStoreV1QueryUserIdTweetCandidateTweetIdFeature {\n  def apply[Query <: PipelineQuery, Candidate <: BaseTweetCandidate, Value](\n    feature: FSv1Feature[EdgeEntityId[UserId, TweetId], Value],\n    legacyName: Option[String] = None,\n    defaultValue: Option[Value] = None,\n    enabledParam: Option[FSParam[Boolean]] = None\n  ): FeatureStoreV1CandidateFeature[Query, Candidate, _ <: EntityId, Value] =\n    FeatureStoreV1CandidateFeature(\n      feature,\n      QueryUserIdTweetCandidateTweetIdEntity,\n      legacyName,\n      defaultValue,\n      enabledParam)\n}\n\nobject FeatureStoreV1QueryUserIdTweetCandidateTweetIdAggregateFeature {\n  def apply[Query <: PipelineQuery, Candidate <: BaseTweetCandidate](\n    featureGroup: TimelinesAggregationFrameworkFeatureGroup[EdgeEntityId[UserId, TweetId]],\n    enabledParam: Option[FSParam[Boolean]] = None,\n    keepLegacyNames: Boolean = false,\n    featureNameTransform: Option[FeatureRenameTransform] = None\n  ): FeatureStoreV1CandidateFeatureGroup[Query, TweetCandidate, _ <: EntityId] =\n    FeatureStoreV1CandidateFeatureGroup(\n      featureGroup,\n      QueryUserIdTweetCandidateTweetIdEntity,\n      enabledParam,\n      keepLegacyNames,\n      featureNameTransform\n    )(implicitly[ClassTag[EdgeEntityId[UserId, TweetId]]])\n}\n\nobject QueryUserIdTweetCandidateTweetIdEntity\n    extends FeatureStoreV1CandidateEntity[\n      PipelineQuery,\n      BaseTweetCandidate,\n      EdgeEntityId[UserId, TweetId]\n    ] {\n  override val entity: Entity[EdgeEntityId[UserId, TweetId]] = entities.core.UserTweet\n\n  override def entityWithId(\n    query: PipelineQuery,\n    tweet: BaseTweetCandidate,\n    existingFeatures: FeatureMap\n  ): EntityWithId[EdgeEntityId[UserId, TweetId]] =\n    entity.withId(EdgeEntityId(UserId(query.getUserIdLoggedOutSupport), TweetId(tweet.id)))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/featurestorev1/FeatureStoreV1TweetCandidateAuthorIdFeature.scala",
    "content": "package com.twitter.product_mixer.component_library.feature.featurestorev1\n\nimport com.twitter.ml.api.transform.FeatureRenameTransform\nimport com.twitter.ml.featurestore.catalog.entities\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.ml.featurestore.lib.entity.Entity\nimport com.twitter.ml.featurestore.lib.entity.EntityWithId\nimport com.twitter.ml.featurestore.lib.feature.TimelinesAggregationFrameworkFeatureGroup\nimport com.twitter.ml.featurestore.lib.feature.{Feature => FSv1Feature}\nimport com.twitter.product_mixer.component_library.model.candidate.TweetAuthorIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featurestorev1._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\nimport scala.reflect.ClassTag\n\nobject FeatureStoreV1TweetCandidateAuthorIdFeature {\n  def apply[Query <: PipelineQuery, Value](\n    feature: FSv1Feature[UserId, Value],\n    legacyName: Option[String] = None,\n    defaultValue: Option[Value] = None,\n    enabledParam: Option[FSParam[Boolean]] = None\n  ): FeatureStoreV1CandidateFeature[Query, TweetCandidate, _ <: EntityId, Value] =\n    FeatureStoreV1CandidateFeature(\n      feature,\n      TweetCandidateAuthorIdEntity,\n      legacyName,\n      defaultValue,\n      enabledParam)\n}\n\nobject FeatureStoreV1TweetCandidateAuthorIdAggregateFeature {\n  def apply[Query <: PipelineQuery](\n    featureGroup: TimelinesAggregationFrameworkFeatureGroup[UserId],\n    enabledParam: Option[FSParam[Boolean]] = None,\n    keepLegacyNames: Boolean = false,\n    featureNameTransform: Option[FeatureRenameTransform] = None\n  ): FeatureStoreV1CandidateFeatureGroup[Query, TweetCandidate, _ <: EntityId] =\n    FeatureStoreV1CandidateFeatureGroup(\n      featureGroup,\n      TweetCandidateAuthorIdEntity,\n      enabledParam,\n      keepLegacyNames,\n      featureNameTransform\n    )(implicitly[ClassTag[UserId]])\n}\n\nobject TweetCandidateAuthorIdEntity\n    extends FeatureStoreV1CandidateEntity[PipelineQuery, TweetCandidate, UserId] {\n  override val entity: Entity[UserId] = entities.core.Author\n\n  override def entityWithId(\n    query: PipelineQuery,\n    tweet: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): EntityWithId[UserId] =\n    entity.withId(UserId(existingFeatures.get(TweetAuthorIdFeature)))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/featurestorev1/FeatureStoreV1TweetCandidateTweetIdFeature.scala",
    "content": "package com.twitter.product_mixer.component_library.feature.featurestorev1\n\nimport com.twitter.ml.api.transform.FeatureRenameTransform\nimport com.twitter.ml.featurestore.catalog.entities\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.TweetId\nimport com.twitter.ml.featurestore.lib.entity.Entity\nimport com.twitter.ml.featurestore.lib.entity.EntityWithId\nimport com.twitter.ml.featurestore.lib.feature.TimelinesAggregationFrameworkFeatureGroup\nimport com.twitter.ml.featurestore.lib.feature.{Feature => FSv1Feature}\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featurestorev1._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\n\nobject FeatureStoreV1TweetCandidateTweetIdFeature {\n  def apply[Query <: PipelineQuery, Candidate <: BaseTweetCandidate, Value](\n    feature: FSv1Feature[TweetId, Value],\n    legacyName: Option[String] = None,\n    defaultValue: Option[Value] = None,\n    enabledParam: Option[FSParam[Boolean]] = None\n  ): FeatureStoreV1CandidateFeature[Query, Candidate, _ <: EntityId, Value] =\n    FeatureStoreV1CandidateFeature(\n      feature,\n      TweetCandidateTweetIdEntity,\n      legacyName,\n      defaultValue,\n      enabledParam)\n}\n\nobject FeatureStoreV1TweetCandidateTweetIdAggregateFeature {\n  def apply[Query <: PipelineQuery, Candidate <: BaseTweetCandidate](\n    featureGroup: TimelinesAggregationFrameworkFeatureGroup[TweetId],\n    enabledParam: Option[FSParam[Boolean]] = None,\n    keepLegacyNames: Boolean = false,\n    featureNameTransform: Option[FeatureRenameTransform] = None\n  ): FeatureStoreV1CandidateFeatureGroup[Query, Candidate, _ <: EntityId] =\n    FeatureStoreV1CandidateFeatureGroup(\n      featureGroup,\n      TweetCandidateTweetIdEntity,\n      enabledParam,\n      keepLegacyNames,\n      featureNameTransform\n    )\n}\n\nobject TweetCandidateTweetIdEntity\n    extends FeatureStoreV1CandidateEntity[PipelineQuery, BaseTweetCandidate, TweetId] {\n  override val entity: Entity[TweetId] = entities.core.Tweet\n\n  override def entityWithId(\n    query: PipelineQuery,\n    tweet: BaseTweetCandidate,\n    existingFeatures: FeatureMap\n  ): EntityWithId[TweetId] =\n    entity.withId(TweetId(tweet.id))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature/featurestorev1/FeatureStoreV1UserCandidateUserIdFeature.scala",
    "content": "package com.twitter.product_mixer.component_library.feature.featurestorev1\n\nimport com.twitter.ml.featurestore.catalog.entities\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.ml.featurestore.lib.entity.Entity\nimport com.twitter.ml.featurestore.lib.entity.EntityWithId\nimport com.twitter.ml.featurestore.lib.feature.{Feature => FSv1Feature}\nimport com.twitter.product_mixer.component_library.model.candidate.BaseUserCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featurestorev1._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\n\nobject FeatureStoreV1UserCandidateUserIdFeature {\n  def apply[Query <: PipelineQuery, Candidate <: BaseUserCandidate, Value](\n    feature: FSv1Feature[UserId, Value],\n    legacyName: Option[String] = None,\n    defaultValue: Option[Value] = None,\n    enabledParam: Option[FSParam[Boolean]] = None\n  ): FeatureStoreV1CandidateFeature[Query, Candidate, _ <: EntityId, Value] =\n    FeatureStoreV1CandidateFeature(\n      feature,\n      UserCandidateUserIdEntity,\n      legacyName,\n      defaultValue,\n      enabledParam)\n}\n\nobject UserCandidateUserIdEntity\n    extends FeatureStoreV1CandidateEntity[PipelineQuery, BaseUserCandidate, UserId] {\n  override val entity: Entity[UserId] = entities.core.User\n\n  override def entityWithId(\n    query: PipelineQuery,\n    user: BaseUserCandidate,\n    existingFeatures: FeatureMap\n  ): EntityWithId[UserId] =\n    entity.withId(UserId(user.id))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/spam/rtf:safety-result-scala\",\n        \"src/thrift/com/twitter/timelinescorer:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/server/internal:thrift-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n        \"stitch/stitch-core\",\n        \"stitch/stitch-tweetypie\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/timelinescorer:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/server/internal:thrift-scala\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/ads/AdvertiserBrandSafetySettingsFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.candidate.ads\n\nimport com.twitter.adserver.{thriftscala => ad}\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.model.query.ads.AdsQuery\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject AdvertiserBrandSafetySettingsFeature\n    extends FeatureWithDefaultOnFailure[AdsCandidate, Option[ad.AdvertiserBrandSafetySettings]] {\n  override val defaultValue = None\n}\n\n@Singleton\ncase class AdvertiserBrandSafetySettingsFeatureHydrator[\n  Query <: PipelineQuery with AdsQuery,\n  Candidate <: AdsCandidate] @Inject() (\n  advertiserBrandSafetySettingsStore: ReadableStore[Long, ad.AdvertiserBrandSafetySettings])\n    extends CandidateFeatureHydrator[Query, Candidate] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"AdvertiserBrandSafetySettings\")\n\n  override val features: Set[Feature[_, _]] = Set(AdvertiserBrandSafetySettingsFeature)\n\n  override def apply(\n    query: Query,\n    candidate: Candidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n\n    val featureMapFuture: Future[FeatureMap] = advertiserBrandSafetySettingsStore\n      .get(candidate.adImpression.advertiserId)\n      .map { advertiserBrandSafetySettingsOpt =>\n        FeatureMapBuilder()\n          .add(AdvertiserBrandSafetySettingsFeature, advertiserBrandSafetySettingsOpt).build()\n      }\n\n    Stitch.callFuture(featureMapFuture)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/ads/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"src/thrift/com/twitter/ads/adserver:ads_shared_types-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"src/thrift/com/twitter/ads/adserver:ads_shared_types-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/decay/BUILD.bazel",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/decay/DecayCandidateFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.candidate.decay\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\nobject DecayScore extends Feature[UniversalNoun[Long], Double]\n\n/**\n * Hydrates snowflake ID candidates with a decay score:\n *\n * It is using exponential decay formula to calculate the score\n * exp(k * age)\n * where k = ln(0.5) / half-life\n *\n * Here is an example for half-life = 1 day\n * For the brand new tweet it will be exp((ln(0.5)/1)*0) = 1\n * For the tweet which was created 1 day ago it will be exp((ln(0.5)/1)*1) = 0.5\n * For the tweet which was created 10 day ago it will be exp((ln(0.5)/1)*10) = 0.00097\n *\n * Reference: https://www.cuemath.com/exponential-decay-formula/\n *\n * @note This penalizes but does not filter out the candidate, so \"stale\" candidates can still appear.\n */\ncase class DecayCandidateFeatureHydrator[Candidate <: UniversalNoun[Long]](\n  halfLife: Param[Duration] = StaticParam[Duration](2.days),\n  resultFeature: Feature[UniversalNoun[Long], Double] = DecayScore)\n    extends CandidateFeatureHydrator[PipelineQuery, Candidate] {\n\n  override val features: Set[Feature[_, _]] = Set(resultFeature)\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"Decay\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: Candidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    val halfLifeInMillis = query.params(halfLife).inMillis\n\n    val creationTime = SnowflakeId.timeFromId(candidate.id)\n    val ageInMillis = creationTime.untilNow.inMilliseconds\n\n    // it is using a exponential decay formula:  e^(k * tweetAge)\n    // where k = ln(0.5) / half-life\n    val k = math.log(0.5D) / halfLifeInMillis\n    val decayScore = math.exp(k * ageInMillis)\n\n    Stitch.value(\n      FeatureMapBuilder()\n        .add(resultFeature, decayScore)\n        .build())\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/param_gated/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/param_gated/ParamGatedBulkCandidateFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated\n\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedBulkCandidateFeatureHydrator.IdentifierPrefix\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * A [[BulkCandidateFeatureHydrator]] with [[Conditionally]] based on a [[Param]]\n *\n * @param enabledParam the param to turn this [[BulkCandidateFeatureHydrator]] on and off\n * @param bulkCandidateFeatureHydrator the underlying [[BulkCandidateFeatureHydrator]] to run when `enabledParam` is true\n * @tparam Query The domain model for the query or request\n * @tparam Result The type of the candidates\n */\ncase class ParamGatedBulkCandidateFeatureHydrator[\n  -Query <: PipelineQuery,\n  Result <: UniversalNoun[Any]\n](\n  enabledParam: Param[Boolean],\n  bulkCandidateFeatureHydrator: BulkCandidateFeatureHydrator[Query, Result])\n    extends BulkCandidateFeatureHydrator[Query, Result]\n    with Conditionally[Query] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    IdentifierPrefix + bulkCandidateFeatureHydrator.identifier.name)\n\n  override val alerts: Seq[Alert] = bulkCandidateFeatureHydrator.alerts\n\n  override val features: Set[Feature[_, _]] = bulkCandidateFeatureHydrator.features\n\n  override def onlyIf(query: Query): Boolean =\n    Conditionally.and(query, bulkCandidateFeatureHydrator, query.params(enabledParam))\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Result]]\n  ): Stitch[Seq[FeatureMap]] = bulkCandidateFeatureHydrator(query, candidates)\n}\n\nobject ParamGatedBulkCandidateFeatureHydrator {\n  val IdentifierPrefix = \"ParamGated\"\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/param_gated/ParamGatedCandidateFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated\n\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedCandidateFeatureHydrator.IdentifierPrefix\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * A [[CandidateFeatureHydrator]] with [[Conditionally]] based on a [[Param]]\n *\n * @param enabledParam the param to turn this [[CandidateFeatureHydrator]] on and off\n * @param candidateFeatureHydrator the underlying [[CandidateFeatureHydrator]] to run when `enabledParam` is true\n * @tparam Query The domain model for the query or request\n * @tparam Result The type of the candidates\n */\ncase class ParamGatedCandidateFeatureHydrator[\n  -Query <: PipelineQuery,\n  -Result <: UniversalNoun[Any]\n](\n  enabledParam: Param[Boolean],\n  candidateFeatureHydrator: CandidateFeatureHydrator[Query, Result])\n    extends CandidateFeatureHydrator[Query, Result]\n    with Conditionally[Query] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    IdentifierPrefix + candidateFeatureHydrator.identifier.name)\n\n  override val alerts: Seq[Alert] = candidateFeatureHydrator.alerts\n\n  override val features: Set[Feature[_, _]] = candidateFeatureHydrator.features\n\n  override def onlyIf(query: Query): Boolean =\n    Conditionally.and(query, candidateFeatureHydrator, query.params(enabledParam))\n\n  override def apply(\n    query: Query,\n    candidate: Result,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = candidateFeatureHydrator.apply(query, candidate, existingFeatures)\n}\n\nobject ParamGatedCandidateFeatureHydrator {\n  val IdentifierPrefix = \"ParamGated\"\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/param_gated/featurestorev1/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/param_gated/featurestorev1/ParamGatedFeatureStoreV1CandidateFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.featurestorev1\n\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.featurestorev1.ParamGatedFeatureStoreV1CandidateFeatureHydrator.IdentifierPrefix\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featurestorev1.BaseFeatureStoreV1CandidateFeature\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1.FeatureStoreV1CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1.FeatureStoreV1DynamicClientBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * A [[FeatureStoreV1CandidateFeatureHydrator]] with [[Conditionally]] based on a [[Param]]\n *\n * @param enabledParam the param to turn this [[FeatureStoreV1CandidateFeatureHydrator]] on and off\n * @param candidateFeatureHydrator the underlying [[FeatureStoreV1CandidateFeatureHydrator]] to run when `enabledParam` is true\n * @tparam Query The domain model for the query or request\n * @tparam Candidate The type of the candidates\n */\ncase class ParamGatedFeatureStoreV1CandidateFeatureHydrator[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any]\n](\n  enabledParam: Param[Boolean],\n  candidateFeatureHydrator: FeatureStoreV1CandidateFeatureHydrator[Query, Candidate])\n    extends FeatureStoreV1CandidateFeatureHydrator[Query, Candidate]\n    with Conditionally[Query] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    IdentifierPrefix + candidateFeatureHydrator.identifier.name)\n\n  override val alerts: Seq[Alert] = candidateFeatureHydrator.alerts\n\n  override val features: Set[\n    BaseFeatureStoreV1CandidateFeature[Query, Candidate, _ <: EntityId, _]\n  ] = candidateFeatureHydrator.features\n\n  override val clientBuilder: FeatureStoreV1DynamicClientBuilder =\n    candidateFeatureHydrator.clientBuilder\n\n  override def onlyIf(query: Query): Boolean =\n    Conditionally.and(query, candidateFeatureHydrator, query.params(enabledParam))\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[Seq[FeatureMap]] = candidateFeatureHydrator(query, candidates)\n}\n\nobject ParamGatedFeatureStoreV1CandidateFeatureHydrator {\n  val IdentifierPrefix = \"ParamGated\"\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/qualityfactor_gated/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/qualityfactor_gated/QualityFactorGatedCandidateFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.candidate.qualityfactor_gated\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\nobject QualityFactorGatedCandidateFeatureHydrator {\n  val IdentifierPrefix = \"QfGated\"\n}\n\n/**\n * A [[CandidateFeatureHydrator]] with [[Conditionally]] based on a qualityFactor threshold.\n * @param pipelineIdentifier identifier of the pipeline that associated with observed quality factor\n * @param qualityFactorInclusiveThreshold the inclusive threshold of quality factor that value below it results in\n *                                        the underlying hydrator being turned off\n * @param candidateFeatureHydrator the underlying [[CandidateFeatureHydrator]] to run when quality factor value\n *                                 is above the given inclusive threshold\n * @tparam Query The domain model for the query or request\n * @tparam Result The type of the candidates\n */\ncase class QualityFactorGatedCandidateFeatureHydrator[\n  -Query <: PipelineQuery with HasQualityFactorStatus,\n  Result <: UniversalNoun[Any]\n](\n  pipelineIdentifier: ComponentIdentifier,\n  qualityFactorInclusiveThreshold: Param[Double],\n  candidateFeatureHydrator: CandidateFeatureHydrator[Query, Result])\n    extends CandidateFeatureHydrator[Query, Result]\n    with Conditionally[Query] {\n  import QualityFactorGatedCandidateFeatureHydrator._\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    IdentifierPrefix + candidateFeatureHydrator.identifier.name)\n\n  override val alerts: Seq[Alert] = candidateFeatureHydrator.alerts\n\n  override val features: Set[Feature[_, _]] = candidateFeatureHydrator.features\n\n  override def onlyIf(query: Query): Boolean = Conditionally.and(\n    query,\n    candidateFeatureHydrator,\n    query.getQualityFactorCurrentValue(pipelineIdentifier) >= query.params(\n      qualityFactorInclusiveThreshold))\n\n  override def apply(\n    query: Query,\n    candidate: Result,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = candidateFeatureHydrator.apply(query, candidate, existingFeatures)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_is_nsfw/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    scalac_plugins = [\"no-roomba\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/spam/rtf:safety-result-scala\",\n        \"src/thrift/com/twitter/timelinescorer:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/server/internal:thrift-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n        \"stitch/stitch-core\",\n        \"stitch/stitch-tweetypie\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/timelinescorer:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/server/internal:thrift-scala\",\n        \"stitch/stitch-core\",\n        \"stitch/stitch-tweetypie\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_is_nsfw/TweetIsNsfwCandidateFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_is_nsfw\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}\nimport com.twitter.tweetypie.{thriftscala => t}\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\n\n// The VF NsfwHighPrecisionLabel that powers the NSFW determination here has been deprecated and is no longer written to.\n@deprecated(\"Prefer VisibilityReason\")\nobject IsNsfw extends FeatureWithDefaultOnFailure[TweetCandidate, Option[Boolean]] {\n\n  /**\n   * Generic Logic to evaluate whether a tweet is nsfw\n   * @param hasNsfwHighPrecisionLabel flag for tweetypieTweet nsfwHighPrecision label\n   * @param isNsfwUser flag for tweetypieTweet coreData nsfwUser flag\n   * @param isNsfwAdmin flag for tweetypieTweet coreData nsfwAdmin flag\n   * @return isNsfw to true if any of the three flags is true\n   */\n  def apply(\n    hasNsfwHighPrecisionLabel: Option[Boolean],\n    isNsfwUser: Option[Boolean],\n    isNsfwAdmin: Option[Boolean]\n  ): Boolean = {\n    hasNsfwHighPrecisionLabel\n      .getOrElse(false) || (isNsfwUser.getOrElse(false) || isNsfwAdmin.getOrElse(false))\n  }\n\n  override val defaultValue = None\n}\n\n// The VF NsfwHighPrecisionLabel that powers the NSFW determination here has been deprecated and is no longer written to.\n// TODO: Remove after all dependencies have migrated to using TweetCandidateVisibilityReasonFeatureHydrator.\n@deprecated(\"Prefer TweetCandidateVisibilityReasonFeatureHydrator\")\ncase class TweetIsNsfwCandidateFeatureHydrator(\n  tweetypieStitchClient: TweetypieStitchClient,\n  tweetVisibilityPolicy: t.TweetVisibilityPolicy)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, BaseTweetCandidate]\n    with Logging {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TweetIsNsfw\")\n\n  override def features: Set[Feature[_, _]] = Set(IsNsfw)\n\n  private val NsfwLabelFields: Set[t.TweetInclude] = Set[t.TweetInclude](\n    // Tweet fields containing NSFW related attributes, in addition to what exists in coreData.\n    t.TweetInclude.TweetFieldId(t.Tweet.NsfwHighPrecisionLabelField.id),\n    t.TweetInclude.TweetFieldId(t.Tweet.CoreDataField.id)\n  )\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[BaseTweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    Stitch\n      .traverse(candidates.map(_.candidate.id)) { tweetId =>\n        tweetypieStitchClient\n          .getTweetFields(\n            tweetId = tweetId,\n            options = t.GetTweetFieldsOptions(\n              forUserId = query.getOptionalUserId,\n              tweetIncludes = NsfwLabelFields,\n              doNotCache = true,\n              visibilityPolicy = tweetVisibilityPolicy,\n              safetyLevel = None,\n            )\n          ).liftToTry\n      }.map { getTweetFieldsResults: Seq[Try[t.GetTweetFieldsResult]] =>\n        val tweets: Seq[Try[t.Tweet]] = getTweetFieldsResults.map {\n          case Return(t.GetTweetFieldsResult(_, t.TweetFieldsResultState.Found(found), _, _)) =>\n            Return(found.tweet)\n          case Return(t.GetTweetFieldsResult(_, resultState, _, _)) =>\n            Throw(IsNsfwFeatureHydrationFailure(s\"Unexpected tweet result state: ${resultState}\"))\n          case Throw(e) =>\n            Throw(e)\n        }\n\n        candidates.zip(tweets).map {\n          case (candidateWithFeatures, tweetTry) =>\n            val isNsfwFeature = tweetTry.map { tweet =>\n              IsNsfw(\n                hasNsfwHighPrecisionLabel = Some(tweet.nsfwHighPrecisionLabel.isDefined),\n                isNsfwUser = tweet.coreData.map(_.nsfwUser),\n                isNsfwAdmin = tweet.coreData.map(_.nsfwAdmin)\n              )\n            }\n\n            FeatureMapBuilder()\n              .add(IsNsfw, isNsfwFeature.map(Some(_)))\n              .build()\n        }\n      }\n  }\n}\n\ncase class IsNsfwFeatureHydrationFailure(message: String)\n    extends Exception(s\"IsNsfwFeatureHydrationFailure(${message})\")\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_tlx/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tweet_tlx\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/ml/featurestore/timelines:ml-features-timelines-scala\",\n        \"src/thrift/com/twitter/ml/featurestore/timelines:ml-features-timelines-strato\",\n        \"src/thrift/com/twitter/timelinescorer:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/server/internal:thrift-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n        \"stitch/stitch-core\",\n        \"strato/config/columns/ml/featureStore:featureStore-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tweet_tlx\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/ml/featurestore/timelines:ml-features-timelines-scala\",\n        \"src/thrift/com/twitter/ml/featurestore/timelines:ml-features-timelines-strato\",\n        \"src/thrift/com/twitter/timelinescorer:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/server/internal:thrift-scala\",\n        \"stitch/stitch-core\",\n        \"strato/config/columns/ml/featureStore:featureStore-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_tlx/TweetTLXScoreCandidateFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_tlx\n\nimport com.twitter.ml.featurestore.timelines.thriftscala.TimelineScorerScoreView\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.scorer.tweet_tlx.TLXScore\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.ml.featureStore.TimelineScorerTweetScoresV1ClientColumn\nimport com.twitter.timelinescorer.thriftscala.v1\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Hydrate Tweet Scores via Timeline Scorer (TLX)\n *\n * Note that this is the [[CandidateFeatureHydrator]] version of\n * [[com.twitter.product_mixer.component_library.scorer.tweet_tlx.TweetTLXStratoScorer]]\n */\n@Singleton\nclass TweetTLXScoreCandidateFeatureHydrator @Inject() (\n  column: TimelineScorerTweetScoresV1ClientColumn)\n    extends CandidateFeatureHydrator[PipelineQuery, TweetCandidate] {\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetTLXScore\")\n\n  override val features: Set[Feature[_, _]] = Set(TLXScore)\n\n  private val NoScoreMap = FeatureMapBuilder()\n    .add(TLXScore, None)\n    .build()\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: TweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    query.getOptionalUserId match {\n      case Some(userId) =>\n        column.fetcher\n          .fetch(candidate.id, TimelineScorerScoreView(Some(userId)))\n          .map(scoredTweet =>\n            scoredTweet.v match {\n              case Some(v1.ScoredTweet(Some(_), score, _, _)) =>\n                FeatureMapBuilder()\n                  .add(TLXScore, score)\n                  .build()\n              case _ => throw new Exception(s\"Invalid response from TLX: ${scoredTweet.v}\")\n            })\n      case _ =>\n        Stitch.value(NoScoreMap)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_tweetypie/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/spam/rtf:safety-result-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n        \"stitch/stitch-core\",\n        \"stitch/stitch-tweetypie\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/spam/rtf:safety-result-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n        \"stitch/stitch-core\",\n        \"stitch/stitch-tweetypie\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_tweetypie/TweetTweetypieCandidateFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_tweetypie\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.spam.rtf.thriftscala.SafetyLevel\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}\nimport com.twitter.tweetypie.thriftscala.TweetVisibilityPolicy\nimport com.twitter.tweetypie.{thriftscala => TP}\n\n// Candidate Features\nobject IsCommunityTweetFeature extends Feature[TweetCandidate, Boolean]\n\n// Tweetypie VF Features\nobject HasTakedownFeature extends Feature[TweetCandidate, Boolean]\nobject HasTakedownForLocaleFeature extends Feature[TweetCandidate, Boolean]\nobject IsHydratedFeature extends Feature[TweetCandidate, Boolean]\nobject IsNarrowcastFeature extends Feature[TweetCandidate, Boolean]\nobject IsNsfwAdminFeature extends Feature[TweetCandidate, Boolean]\nobject IsNsfwFeature extends Feature[TweetCandidate, Boolean]\nobject IsNsfwUserFeature extends Feature[TweetCandidate, Boolean]\nobject IsNullcastFeature extends Feature[TweetCandidate, Boolean]\nobject QuotedTweetDroppedFeature extends Feature[TweetCandidate, Boolean]\nobject QuotedTweetHasTakedownFeature extends Feature[TweetCandidate, Boolean]\nobject QuotedTweetHasTakedownForLocaleFeature extends Feature[TweetCandidate, Boolean]\nobject QuotedTweetIdFeature extends Feature[TweetCandidate, Option[Long]]\nobject SourceTweetHasTakedownFeature extends Feature[TweetCandidate, Boolean]\nobject SourceTweetHasTakedownForLocaleFeature extends Feature[TweetCandidate, Boolean]\nobject TakedownCountryCodesFeature extends Feature[TweetCandidate, Set[String]]\nobject IsReplyFeature extends Feature[TweetCandidate, Boolean]\nobject InReplyToFeature extends Feature[TweetCandidate, Option[Long]]\nobject IsRetweetFeature extends Feature[TweetCandidate, Boolean]\n\nobject TweetTweetypieCandidateFeatureHydrator {\n  val CoreTweetFields: Set[TP.TweetInclude] = Set[TP.TweetInclude](\n    TP.TweetInclude.TweetFieldId(TP.Tweet.IdField.id),\n    TP.TweetInclude.TweetFieldId(TP.Tweet.CoreDataField.id)\n  )\n\n  val NsfwLabelFields: Set[TP.TweetInclude] = Set[TP.TweetInclude](\n    // Tweet fields containing NSFW related attributes, in addition to what exists in coreData.\n    TP.TweetInclude.TweetFieldId(TP.Tweet.NsfwHighRecallLabelField.id),\n    TP.TweetInclude.TweetFieldId(TP.Tweet.NsfwHighPrecisionLabelField.id),\n    TP.TweetInclude.TweetFieldId(TP.Tweet.NsfaHighRecallLabelField.id)\n  )\n\n  val SafetyLabelFields: Set[TP.TweetInclude] = Set[TP.TweetInclude](\n    // Tweet fields containing RTF labels for abuse and spam.\n    TP.TweetInclude.TweetFieldId(TP.Tweet.SpamLabelField.id),\n    TP.TweetInclude.TweetFieldId(TP.Tweet.AbusiveLabelField.id)\n  )\n\n  val OrganicTweetTPHydrationFields: Set[TP.TweetInclude] = CoreTweetFields ++\n    NsfwLabelFields ++\n    SafetyLabelFields ++\n    Set(\n      TP.TweetInclude.TweetFieldId(TP.Tweet.TakedownCountryCodesField.id),\n      // QTs imply a TweetyPie -> SGS request dependency\n      TP.TweetInclude.TweetFieldId(TP.Tweet.QuotedTweetField.id),\n      TP.TweetInclude.TweetFieldId(TP.Tweet.EscherbirdEntityAnnotationsField.id),\n      TP.TweetInclude.TweetFieldId(TP.Tweet.CommunitiesField.id),\n      // Field required for determining if a Tweet was created via News Camera.\n      TP.TweetInclude.TweetFieldId(TP.Tweet.ComposerSourceField.id)\n    )\n\n  val InjectedTweetTPHydrationFields: Set[TP.TweetInclude] =\n    OrganicTweetTPHydrationFields ++ Set(\n      // Mentions imply a TweetyPie -> Gizmoduck request dependency\n      TP.TweetInclude.TweetFieldId(TP.Tweet.MentionsField.id),\n      TP.TweetInclude.TweetFieldId(TP.Tweet.HashtagsField.id)\n    )\n\n  val DefaultFeatureMap = FeatureMapBuilder()\n    .add(IsNsfwAdminFeature, false)\n    .add(IsNsfwUserFeature, false)\n    .add(IsNsfwFeature, false)\n    .add(IsNullcastFeature, false)\n    .add(IsNarrowcastFeature, false)\n    .add(HasTakedownFeature, false)\n    .add(IsCommunityTweetFeature, false)\n    .add(TakedownCountryCodesFeature, Set.empty: Set[String])\n    .add(IsHydratedFeature, false)\n    .add(HasTakedownForLocaleFeature, false)\n    .add(QuotedTweetDroppedFeature, false)\n    .add(SourceTweetHasTakedownFeature, false)\n    .add(QuotedTweetHasTakedownFeature, false)\n    .add(SourceTweetHasTakedownForLocaleFeature, false)\n    .add(QuotedTweetHasTakedownForLocaleFeature, false)\n    .add(IsReplyFeature, false)\n    .add(InReplyToFeature, None)\n    .add(IsRetweetFeature, false)\n    .build()\n}\n\nclass TweetTweetypieCandidateFeatureHydrator(\n  tweetypieStitchClient: TweetypieStitchClient,\n  safetyLevelPredicate: PipelineQuery => SafetyLevel)\n    extends CandidateFeatureHydrator[PipelineQuery, BaseTweetCandidate] {\n\n  import TweetTweetypieCandidateFeatureHydrator._\n\n  override val features: Set[Feature[_, _]] =\n    Set(\n      IsNsfwFeature,\n      IsNsfwAdminFeature,\n      IsNsfwUserFeature,\n      IsNullcastFeature,\n      IsNarrowcastFeature,\n      HasTakedownFeature,\n      IsCommunityTweetFeature,\n      TakedownCountryCodesFeature,\n      IsHydratedFeature,\n      HasTakedownForLocaleFeature,\n      QuotedTweetDroppedFeature,\n      SourceTweetHasTakedownFeature,\n      QuotedTweetHasTakedownFeature,\n      SourceTweetHasTakedownForLocaleFeature,\n      QuotedTweetHasTakedownForLocaleFeature,\n      IsReplyFeature,\n      InReplyToFeature,\n      IsRetweetFeature\n    )\n\n  override val identifier: FeatureHydratorIdentifier =\n    FeatureHydratorIdentifier(\"TweetTweetypie\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidate: BaseTweetCandidate,\n    existingFeatures: FeatureMap\n  ): Stitch[FeatureMap] = {\n    val countryCode = query.getCountryCode.getOrElse(\"\")\n\n    tweetypieStitchClient\n      .getTweetFields(\n        tweetId = candidate.id,\n        options = TP.GetTweetFieldsOptions(\n          tweetIncludes = OrganicTweetTPHydrationFields,\n          includeRetweetedTweet = true,\n          includeQuotedTweet = true,\n          visibilityPolicy = TweetVisibilityPolicy.UserVisible,\n          safetyLevel = Some(safetyLevelPredicate(query))\n        )\n      ).map {\n        case TP.GetTweetFieldsResult(_, TP.TweetFieldsResultState.Found(found), quoteOpt, _) =>\n          val coreData = found.tweet.coreData\n          val isNsfwAdmin = coreData.exists(_.nsfwAdmin)\n          val isNsfwUser = coreData.exists(_.nsfwUser)\n          val hasTakedown = coreData.exists(_.hasTakedown)\n          val isReply = coreData.exists(_.reply.nonEmpty)\n          val ancestorId = coreData.flatMap(_.reply).flatMap(_.inReplyToStatusId)\n          val isRetweet = coreData.exists(_.share.nonEmpty)\n          val takedownCountryCodes =\n            found.tweet.takedownCountryCodes.getOrElse(Seq.empty).map(_.toLowerCase).toSet\n\n          val quotedTweetDropped = quoteOpt.exists {\n            case _: TP.TweetFieldsResultState.Filtered =>\n              true\n            case _: TP.TweetFieldsResultState.NotFound =>\n              true\n            case _ => false\n          }\n          val quotedTweetIsNsfw = quoteOpt.exists {\n            case quoteTweet: TP.TweetFieldsResultState.Found =>\n              quoteTweet.found.tweet.coreData.exists(data => data.nsfwAdmin || data.nsfwUser)\n            case _ => false\n          }\n          val quotedTweetHasTakedown = quoteOpt.exists {\n            case quoteTweet: TP.TweetFieldsResultState.Found =>\n              quoteTweet.found.tweet.coreData.exists(_.hasTakedown)\n            case _ => false\n          }\n          val quotedTweetTakedownCountryCodes = quoteOpt\n            .collect {\n              case quoteTweet: TP.TweetFieldsResultState.Found =>\n                quoteTweet.found.tweet.takedownCountryCodes\n                  .getOrElse(Seq.empty).map(_.toLowerCase).toSet\n            }.getOrElse(Set.empty[String])\n\n          val sourceTweetIsNsfw =\n            found.retweetedTweet.exists(_.coreData.exists(data => data.nsfwAdmin || data.nsfwUser))\n          val sourceTweetHasTakedown =\n            found.retweetedTweet.exists(_.coreData.exists(_.hasTakedown))\n          val sourceTweetTakedownCountryCodes = found.retweetedTweet\n            .map { sourceTweet: TP.Tweet =>\n              sourceTweet.takedownCountryCodes.getOrElse(Seq.empty).map(_.toLowerCase).toSet\n            }.getOrElse(Set.empty)\n\n          FeatureMapBuilder()\n            .add(IsNsfwAdminFeature, isNsfwAdmin)\n            .add(IsNsfwUserFeature, isNsfwUser)\n            .add(IsNsfwFeature, isNsfwAdmin || isNsfwUser || sourceTweetIsNsfw || quotedTweetIsNsfw)\n            .add(IsNullcastFeature, coreData.exists(_.nullcast))\n            .add(IsNarrowcastFeature, coreData.exists(_.narrowcast.nonEmpty))\n            .add(HasTakedownFeature, hasTakedown)\n            .add(\n              HasTakedownForLocaleFeature,\n              hasTakedownForLocale(hasTakedown, countryCode, takedownCountryCodes))\n            .add(QuotedTweetDroppedFeature, quotedTweetDropped)\n            .add(SourceTweetHasTakedownFeature, sourceTweetHasTakedown)\n            .add(QuotedTweetHasTakedownFeature, quotedTweetHasTakedown)\n            .add(\n              SourceTweetHasTakedownForLocaleFeature,\n              hasTakedownForLocale(\n                sourceTweetHasTakedown,\n                countryCode,\n                sourceTweetTakedownCountryCodes))\n            .add(\n              QuotedTweetHasTakedownForLocaleFeature,\n              hasTakedownForLocale(\n                quotedTweetHasTakedown,\n                countryCode,\n                quotedTweetTakedownCountryCodes))\n            .add(IsCommunityTweetFeature, found.tweet.communities.exists(_.communityIds.nonEmpty))\n            .add(\n              TakedownCountryCodesFeature,\n              found.tweet.takedownCountryCodes.getOrElse(Seq.empty).map(_.toLowerCase).toSet)\n            .add(IsHydratedFeature, true)\n            .add(IsReplyFeature, isReply)\n            .add(InReplyToFeature, ancestorId)\n            .add(IsRetweetFeature, isRetweet)\n            .build()\n\n        // If no tweet result found, return default features\n        case _ =>\n          DefaultFeatureMap\n      }\n  }\n\n  private def hasTakedownForLocale(\n    hasTakedown: Boolean,\n    countryCode: String,\n    takedownCountryCodes: Set[String]\n  ) = hasTakedown && takedownCountryCodes.contains(countryCode)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_visibility_reason/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/spam/rtf:safety-result-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n        \"stitch/stitch-core\",\n        \"stitch/stitch-tweetypie\",\n        \"util/util-slf4j-api\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/spam/rtf:safety-result-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n        \"stitch/stitch-core\",\n        \"stitch/stitch-tweetypie\",\n        \"util/util-slf4j-api\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_visibility_reason/TweetVisibilityReasonBulkCandidateFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_visibility_reason\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.spam.rtf.{thriftscala => SPAM}\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}\nimport com.twitter.tweetypie.{thriftscala => TP}\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject VisibilityReason\n    extends FeatureWithDefaultOnFailure[TweetCandidate, Option[SPAM.FilteredReason]] {\n  override val defaultValue = None\n}\n\n/**\n * A [[BulkCandidateFeatureHydrator]] that hydrates TweetCandidates with VisibilityReason features\n * by [[SPAM.SafetyLevel]] when present. The [[VisibilityReason]] feature represents a VisibilityFiltering\n * [[SPAM.FilteredReason]], which contains safety filtering verdict information including action (e.g.\n * Drop, Avoid) and reason (e.g. Misinformation, Abuse). This feature can inform downstream services'\n * handling and presentation of Tweets (e.g. ad avoidance).\n *\n * @param tweetypieStitchClient used to retrieve Tweet fields for BaseTweetCandidates\n * @param safetyLevel specifies VisibilityFiltering SafetyLabel\n */\n\n@Singleton\ncase class TweetVisibilityReasonBulkCandidateFeatureHydrator @Inject() (\n  tweetypieStitchClient: TweetypieStitchClient,\n  safetyLevel: SPAM.SafetyLevel)\n    extends BulkCandidateFeatureHydrator[PipelineQuery, BaseTweetCandidate]\n    with Logging {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"TweetVisibilityReason\")\n\n  override def features: Set[Feature[_, _]] = Set(VisibilityReason)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[BaseTweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    Stitch\n      .traverse(candidates.map(_.candidate.id)) { tweetId =>\n        tweetypieStitchClient\n          .getTweetFields(\n            tweetId = tweetId,\n            options = TP.GetTweetFieldsOptions(\n              forUserId = query.getOptionalUserId,\n              tweetIncludes = Set.empty,\n              doNotCache = true,\n              visibilityPolicy = TP.TweetVisibilityPolicy.UserVisible,\n              safetyLevel = Some(safetyLevel)\n            )\n          ).liftToTry\n      }.map { getTweetFieldsResults: Seq[Try[TP.GetTweetFieldsResult]] =>\n        val tweetFields: Seq[Try[TP.TweetFieldsResultFound]] = getTweetFieldsResults.map {\n          case Return(TP.GetTweetFieldsResult(_, TP.TweetFieldsResultState.Found(found), _, _)) =>\n            Return(found)\n          case Return(TP.GetTweetFieldsResult(_, resultState, _, _)) =>\n            Throw(\n              VisibilityReasonFeatureHydrationFailure(\n                s\"Unexpected tweet result state: ${resultState}\"))\n          case Throw(e) =>\n            Throw(e)\n        }\n\n        tweetFields.map { tweetFieldTry =>\n          val tweetFilteredReason = tweetFieldTry.map { tweetField =>\n            tweetField.suppressReason match {\n              case Some(suppressReason) => Some(suppressReason)\n              case _ => None\n            }\n          }\n\n          FeatureMapBuilder()\n            .add(VisibilityReason, tweetFilteredReason)\n            .build()\n        }\n      }\n  }\n}\n\ncase class VisibilityReasonFeatureHydrationFailure(message: String)\n    extends Exception(s\"VisibilityReasonFeatureHydrationFailure($message)\")\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/async/AsyncQueryFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.query.async\n\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featurestorev1.BaseFeatureStoreV1QueryFeature\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.AsyncHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1.FeatureStoreV1DynamicClientBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1.FeatureStoreV1QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * A [[QueryFeatureHydrator]] with [[AsyncQueryFeatureHydrator]] that hydrated asynchronously for features\n * to be before the step identified in [[hydrateBefore]]\n *\n * @param hydrateBefore        the [[PipelineStepIdentifier]] step to make sure this feature is hydrated before.\n * @param queryFeatureHydrator the underlying [[QueryFeatureHydrator]] to run asynchronously\n * @tparam Query The domain model for the query or request\n */\ncase class AsyncQueryFeatureHydrator[-Query <: PipelineQuery] private[async] (\n  override val hydrateBefore: PipelineStepIdentifier,\n  queryFeatureHydrator: QueryFeatureHydrator[Query])\n    extends QueryFeatureHydrator[Query]\n    with AsyncHydrator {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"Async\" + queryFeatureHydrator.identifier.name)\n  override val alerts: Seq[Alert] = queryFeatureHydrator.alerts\n  override val features: Set[Feature[_, _]] = queryFeatureHydrator.features\n\n  override def hydrate(query: Query): Stitch[FeatureMap] = queryFeatureHydrator.hydrate(query)\n}\n\n/**\n * A [[FeatureStoreV1QueryFeatureHydrator]] with [[AsyncHydrator]] that hydrated asynchronously for features\n * to be before the step identified in [[hydrateBefore]]. We need a standalone class for feature store,\n * different from the above as FStore hydrators are exempt from validations at run time.\n *\n * @param hydrateBefore        the [[PipelineStepIdentifier]] step to make sure this feature is hydrated before.\n * @param queryFeatureHydrator the underlying [[QueryFeatureHydrator]] to run asynchronously\n * @tparam Query The domain model for the query or request\n */\ncase class AsyncFeatureStoreV1QueryFeatureHydrator[Query <: PipelineQuery] private[async] (\n  override val hydrateBefore: PipelineStepIdentifier,\n  featureStoreV1QueryFeatureHydrator: FeatureStoreV1QueryFeatureHydrator[Query])\n    extends FeatureStoreV1QueryFeatureHydrator[\n      Query\n    ]\n    with AsyncHydrator {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"Async\" + featureStoreV1QueryFeatureHydrator.identifier.name)\n  override val alerts: Seq[Alert] = featureStoreV1QueryFeatureHydrator.alerts\n\n  override val features: Set[BaseFeatureStoreV1QueryFeature[Query, _ <: EntityId, _]] =\n    featureStoreV1QueryFeatureHydrator.features\n\n  override val clientBuilder: FeatureStoreV1DynamicClientBuilder =\n    featureStoreV1QueryFeatureHydrator.clientBuilder\n}\n\nobject AsyncQueryFeatureHydrator {\n\n  /**\n   * A [[QueryFeatureHydrator]] with [[AsyncQueryFeatureHydrator]] that hydrated asynchronously for features\n   * to be before the step identified in [[hydrateBefore]]\n   *\n   * @param hydrateBefore        the [[PipelineStepIdentifier]] step to make sure this feature is hydrated before.\n   * @param queryFeatureHydrator the underlying [[QueryFeatureHydrator]] to run asynchronously\n   * @tparam Query The domain model for the query or request\n   */\n  def apply[Query <: PipelineQuery](\n    hydrateBefore: PipelineStepIdentifier,\n    queryFeatureHydrator: QueryFeatureHydrator[Query]\n  ): AsyncQueryFeatureHydrator[Query] =\n    new AsyncQueryFeatureHydrator(hydrateBefore, queryFeatureHydrator)\n\n  /**\n   * A [[FeatureStoreV1QueryFeatureHydrator]] with [[AsyncHydrator]] that hydrated asynchronously for features\n   * to be before the step identified in [[hydrateBefore]]. We need a standalone class for feature store,\n   * different from the above as FStore hydrators are exempt from validations at run time.\n   *\n   * @param hydrateBefore        the [[PipelineStepIdentifier]] step to make sure this feature is hydrated before.\n   * @param queryFeatureHydrator the underlying [[QueryFeatureHydrator]] to run asynchronously\n   * @tparam Query The domain model for the query or request\n   */\n  def apply[Query <: PipelineQuery](\n    hydrateBefore: PipelineStepIdentifier,\n    featureStoreV1QueryFeatureHydrator: FeatureStoreV1QueryFeatureHydrator[Query]\n  ): AsyncFeatureStoreV1QueryFeatureHydrator[Query] =\n    new AsyncFeatureStoreV1QueryFeatureHydrator(hydrateBefore, featureStoreV1QueryFeatureHydrator)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/async/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/cr_ml_ranker/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"cr-ml-ranker/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"cr-ml-ranker/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/cr_ml_ranker/CrMlRankerCommonQueryFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.query.cr_ml_ranker\n\nimport com.twitter.cr_ml_ranker.{thriftscala => t}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject CrMlRankerCommonFeatures extends Feature[PipelineQuery, t.CommonFeatures]\nobject CrMlRankerRankingConfig extends Feature[PipelineQuery, t.RankingConfig]\n\nprivate[cr_ml_ranker] class CrMlRankerCommonQueryFeatureHydrator(\n  crMlRanker: t.CrMLRanker.MethodPerEndpoint,\n  rankingConfigSelector: RankingConfigBuilder)\n    extends QueryFeatureHydrator[PipelineQuery] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"CrMlRanker\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(CrMlRankerCommonFeatures, CrMlRankerRankingConfig)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    val rankingConfig = rankingConfigSelector.apply(query)\n    Stitch\n      .callFuture(\n        crMlRanker.getCommonFeatures(\n          t.RankingRequestContext(query.getRequiredUserId, rankingConfig))).map { commonFeatures =>\n        FeatureMapBuilder()\n          .add(CrMlRankerRankingConfig, rankingConfig)\n          .add(CrMlRankerCommonFeatures, commonFeatures)\n          .build()\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/cr_ml_ranker/CrMlRankerCommonQueryFeatureHydratorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.query.cr_ml_ranker\n\nimport com.twitter.cr_ml_ranker.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Builds a query hydrator that hydrates Common Features for the given Query from CR ML Ranker\n * to be later used to call CR ML Ranker for scoring using the desired [[RankingConfigBuilder]]\n * for building the ranking config.\n */\n@Singleton\nclass CrMlRankerCommonQueryFeatureHydratorBuilder @Inject() (\n  crMlRanker: t.CrMLRanker.MethodPerEndpoint) {\n\n  def build(rankingConfigSelector: RankingConfigBuilder): CrMlRankerCommonQueryFeatureHydrator =\n    new CrMlRankerCommonQueryFeatureHydrator(crMlRanker, rankingConfigSelector)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/cr_ml_ranker/RankingConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.query.cr_ml_ranker\n\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.cr_ml_ranker.{thriftscala => t}\n\n/**\n * Builder for constructing a ranking config from a query\n */\ntrait RankingConfigBuilder {\n  def apply(query: PipelineQuery): t.RankingConfig\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/impressed_tweets/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/timelines/impression_store:thrift-scala\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"src/thrift/com/twitter/timelines/impression_store:thrift-scala\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/impressed_tweets/ImpressedTweetsQueryFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.query.impressed_tweets\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.impressionstore.thriftscala.ImpressionList\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Query Feature to store ids of the tweets impressed by the user.\n */\ncase object ImpressedTweets extends FeatureWithDefaultOnFailure[PipelineQuery, Seq[Long]] {\n  override val defaultValue: Seq[Long] = Seq.empty\n}\n\n/**\n * Enrich the query with a list of tweet ids that the user has already seen.\n */\n@Singleton\ncase class ImpressedTweetsQueryFeatureHydrator @Inject() (\n  tweetImpressionStore: ReadableStore[Long, ImpressionList])\n    extends QueryFeatureHydrator[PipelineQuery] {\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\"TweetsToExclude\")\n\n  override val features: Set[Feature[_, _]] = Set(ImpressedTweets)\n\n  override def hydrate(query: PipelineQuery): Stitch[FeatureMap] = {\n    query.getOptionalUserId match {\n      case Some(userId) =>\n        val featureMapResult: Future[FeatureMap] = tweetImpressionStore\n          .get(userId).map { impressionListOpt =>\n            val tweetIdsOpt = for {\n              impressionList <- impressionListOpt\n              impressions <- impressionList.impressions\n            } yield {\n              impressions.map(_.tweetId)\n            }\n            val tweetIds = tweetIdsOpt.getOrElse(Seq.empty)\n            FeatureMapBuilder().add(ImpressedTweets, tweetIds).build()\n          }\n        Stitch.callFuture(featureMapResult)\n      // Non-logged-in users do not have userId, returns empty feature\n\n      case None =>\n        val featureMapResult = FeatureMapBuilder().add(ImpressedTweets, Seq.empty).build()\n        Stitch.value(featureMapResult)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/logged_in_only/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/logged_in_only/LoggedInOnlyQueryFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.query.logged_in_only\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * A [[QueryFeatureHydrator]] with [[Conditionally]] to run only for logged in users\n *\n * @param queryFeatureHydrator the underlying [[QueryFeatureHydrator]] to run when query.isLoggedOut is false\n * @tparam Query The domain model for the query or request\n * @tparam Result The type of the candidates\n */\ncase class LoggedInOnlyQueryFeatureHydrator[-Query <: PipelineQuery, Result <: UniversalNoun[Any]](\n  queryFeatureHydrator: QueryFeatureHydrator[Query])\n    extends QueryFeatureHydrator[Query]\n    with Conditionally[Query] {\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"LoggedInOnly\" + queryFeatureHydrator.identifier.name)\n  override val alerts: Seq[Alert] = queryFeatureHydrator.alerts\n  override val features: Set[Feature[_, _]] = queryFeatureHydrator.features\n  override def onlyIf(query: Query): Boolean =\n    Conditionally.and(query, queryFeatureHydrator, !query.isLoggedOut)\n  override def hydrate(query: Query): Stitch[FeatureMap] = queryFeatureHydrator.hydrate(query)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/param_gated/AsyncParamGatedQueryFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.AsyncHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * A [[QueryFeatureHydrator]] with [[Conditionally]] based on a [[Param]] that hydrates asynchronously for features\n * to be before the step identified in [[hydrateBefore]]\n *\n * @param enabledParam the param to turn this [[QueryFeatureHydrator]] on and off\n * @param hydrateBefore the [[PipelineStepIdentifier]] step to make sure this feature is hydrated before.\n * @param queryFeatureHydrator the underlying [[QueryFeatureHydrator]] to run when `enabledParam` is true\n * @tparam Query The domain model for the query or request\n * @tparam Result The type of the candidates\n */\ncase class AsyncParamGatedQueryFeatureHydrator[\n  -Query <: PipelineQuery,\n  Result <: UniversalNoun[Any]\n](\n  enabledParam: Param[Boolean],\n  override val hydrateBefore: PipelineStepIdentifier,\n  queryFeatureHydrator: QueryFeatureHydrator[Query])\n    extends QueryFeatureHydrator[Query]\n    with Conditionally[Query]\n    with AsyncHydrator {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"AsyncParamGated\" + queryFeatureHydrator.identifier.name)\n\n  override val alerts: Seq[Alert] = queryFeatureHydrator.alerts\n\n  override val features: Set[Feature[_, _]] = queryFeatureHydrator.features\n\n  override def onlyIf(query: Query): Boolean =\n    Conditionally.and(query, queryFeatureHydrator, query.params(enabledParam))\n\n  override def hydrate(query: Query): Stitch[FeatureMap] = queryFeatureHydrator.hydrate(query)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/param_gated/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/param_gated/ParamGatedQueryFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * A [[QueryFeatureHydrator]] with [[Conditionally]] based on a [[Param]]\n *\n * @param enabledParam the param to turn this [[QueryFeatureHydrator]] on and off\n * @param queryFeatureHydrator the underlying [[QueryFeatureHydrator]] to run when `enabledParam` is true\n * @tparam Query The domain model for the query or request\n * @tparam Result The type of the candidates\n */\ncase class ParamGatedQueryFeatureHydrator[-Query <: PipelineQuery, Result <: UniversalNoun[Any]](\n  enabledParam: Param[Boolean],\n  queryFeatureHydrator: QueryFeatureHydrator[Query])\n    extends QueryFeatureHydrator[Query]\n    with Conditionally[Query] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"ParamGated\" + queryFeatureHydrator.identifier.name)\n\n  override val alerts: Seq[Alert] = queryFeatureHydrator.alerts\n\n  override val features: Set[Feature[_, _]] = queryFeatureHydrator.features\n\n  override def onlyIf(query: Query): Boolean =\n    Conditionally.and(query, queryFeatureHydrator, query.params(enabledParam))\n\n  override def hydrate(query: Query): Stitch[FeatureMap] = queryFeatureHydrator.hydrate(query)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/param_gated/featurestorev1/AsyncParamGatedFeatureStoreV1QueryFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated.featurestorev1\n\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featurestorev1.BaseFeatureStoreV1QueryFeature\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.AsyncHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1.FeatureStoreV1DynamicClientBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1.FeatureStoreV1QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * A [[QueryFeatureHydrator]] with [[Conditionally]] based on a [[Param]] that hydrates asynchronously for features\n * to be before the step identified in [[hydrateBefore]]\n *\n * @param enabledParam the param to turn this [[QueryFeatureHydrator]] on and off\n * @param hydrateBefore the [[PipelineStepIdentifier]] step to make sure this feature is hydrated before.\n * @param queryFeatureHydrator the underlying [[QueryFeatureHydrator]] to run when `enabledParam` is true\n * @tparam Query The domain model for the query or request\n * @tparam Result The type of the candidates\n */\ncase class AsyncParamGatedFeatureStoreV1QueryFeatureHydrator[\n  Query <: PipelineQuery,\n  Result <: UniversalNoun[Any]\n](\n  enabledParam: Param[Boolean],\n  override val hydrateBefore: PipelineStepIdentifier,\n  queryFeatureHydrator: FeatureStoreV1QueryFeatureHydrator[Query])\n    extends FeatureStoreV1QueryFeatureHydrator[Query]\n    with Conditionally[Query]\n    with AsyncHydrator {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"AsyncParamGated\" + queryFeatureHydrator.identifier.name)\n\n  override val alerts: Seq[Alert] = queryFeatureHydrator.alerts\n\n  override val features: Set[BaseFeatureStoreV1QueryFeature[Query, _ <: EntityId, _]] =\n    queryFeatureHydrator.features\n\n  override val clientBuilder: FeatureStoreV1DynamicClientBuilder =\n    queryFeatureHydrator.clientBuilder\n  override def onlyIf(query: Query): Boolean =\n    Conditionally.and(query, queryFeatureHydrator, query.params(enabledParam))\n\n  override def hydrate(query: Query): Stitch[FeatureMap] = queryFeatureHydrator.hydrate(query)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/param_gated/featurestorev1/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/param_gated/featurestorev1/ParamGatedFeatureStoreV1QueryFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated.featurestorev1\n\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featurestorev1.BaseFeatureStoreV1QueryFeature\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1.FeatureStoreV1DynamicClientBuilder\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1.FeatureStoreV1QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * A [[QueryFeatureHydrator]] with [[Conditionally]] based on a [[Param]]\n *\n * @param enabledParam the param to turn this [[QueryFeatureHydrator]] on and off\n * @param queryFeatureHydrator the underlying [[QueryFeatureHydrator]] to run when `enabledParam` is true\n * @tparam Query The domain model for the query or request\n * @tparam Result The type of the candidates\n */\ncase class ParamGatedFeatureStoreV1QueryFeatureHydrator[\n  Query <: PipelineQuery,\n  Result <: UniversalNoun[Any]\n](\n  enabledParam: Param[Boolean],\n  queryFeatureHydrator: FeatureStoreV1QueryFeatureHydrator[Query])\n    extends FeatureStoreV1QueryFeatureHydrator[Query]\n    with Conditionally[Query] {\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    \"ParamGated\" + queryFeatureHydrator.identifier.name)\n\n  override val alerts: Seq[Alert] = queryFeatureHydrator.alerts\n\n  override val features: Set[BaseFeatureStoreV1QueryFeature[Query, _ <: EntityId, _]] =\n    queryFeatureHydrator.features\n\n  override val clientBuilder: FeatureStoreV1DynamicClientBuilder =\n    queryFeatureHydrator.clientBuilder\n  override def onlyIf(query: Query): Boolean =\n    Conditionally.and(query, queryFeatureHydrator, query.params(enabledParam))\n\n  override def hydrate(query: Query): Stitch[FeatureMap] = queryFeatureHydrator.hydrate(query)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/qualityfactor_gated/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/qualityfactor_gated/QualityFactorGatedQueryFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.component_library.feature_hydrator.query.qualityfactor_gated\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\nobject QualityFactorGatedQueryFeatureHydrator {\n  val IdentifierPrefix = \"QfGated\"\n}\n\n/**\n * A [[QueryFeatureHydrator]] with [[Conditionally]] based on a qualityFactor threshold.\n * @param pipelineIdentifier identifier of the pipeline that associated with observed quality factor\n * @param qualityFactorInclusiveThreshold the threshold of the quality factor that results in the hydrator being turned off\n * @param queryFeatureHydrator the underlying [[QueryFeatureHydrator]] to run when quality factor value\n *                                 is above the given inclusive threshold\n * @tparam Query The domain model for the query or request\n * @tparam Result The type of the candidates\n */\ncase class QualityFactorGatedQueryFeatureHydrator[\n  -Query <: PipelineQuery with HasQualityFactorStatus,\n  Result <: UniversalNoun[Any]\n](\n  pipelineIdentifier: ComponentIdentifier,\n  qualityFactorInclusiveThreshold: Param[Double],\n  queryFeatureHydrator: QueryFeatureHydrator[Query])\n    extends QueryFeatureHydrator[Query]\n    with Conditionally[Query] {\n  import QualityFactorGatedQueryFeatureHydrator._\n\n  override val identifier: FeatureHydratorIdentifier = FeatureHydratorIdentifier(\n    IdentifierPrefix + queryFeatureHydrator.identifier.name)\n\n  override val alerts: Seq[Alert] = queryFeatureHydrator.alerts\n\n  override val features: Set[Feature[_, _]] = queryFeatureHydrator.features\n\n  override def onlyIf(query: Query): Boolean = Conditionally.and(\n    query,\n    queryFeatureHydrator,\n    query.getQualityFactorCurrentValue(pipelineIdentifier) >= query.params(\n      qualityFactorInclusiveThreshold))\n\n  override def hydrate(query: Query): Stitch[FeatureMap] = queryFeatureHydrator.hydrate(query)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/AdaptiveLongIntBloomFilterDedupFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.search.common.util.bloomfilter.AdaptiveLongIntBloomFilter\n\ntrait GetAdaptiveLongIntBloomFilter[Query <: PipelineQuery] {\n  def apply(query: Query): Option[AdaptiveLongIntBloomFilter]\n}\n\ncase class AdaptiveLongIntBloomFilterDedupFilter[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Long]\n](\n  getBloomFilter: GetAdaptiveLongIntBloomFilter[Query])\n    extends Filter[Query, Candidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\n    \"AdaptiveLongIntBloomFilterDedupFilter\")\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n\n    val filterResult = getBloomFilter(query)\n      .map { bloomFilter =>\n        val (kept, removed) =\n          candidates.map(_.candidate).partition(candidate => !bloomFilter.contains(candidate.id))\n        FilterResult(kept, removed)\n      }.getOrElse(FilterResult(candidates.map(_.candidate), Seq.empty))\n\n    Stitch.value(filterResult)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/candidate/tweet_tweetypie\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"src/java/com/twitter/search/common/util/bloomfilter\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n        \"stitch/stitch-core\",\n        \"stitch/stitch-tweetypie\",\n        \"stitch/stitch-tweetypie/src/main/scala\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"src/java/com/twitter/search/common/util/bloomfilter\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/ExcludedIdsFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.HasExcludedIds\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\ncase class ExcludedIdsFilter[\n  Query <: PipelineQuery with HasExcludedIds,\n  Candidate <: UniversalNoun[Long]\n]() extends Filter[Query, Candidate] {\n  override val identifier: FilterIdentifier = FilterIdentifier(\"ExcludedIds\")\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n    val (kept, removed) =\n      candidates.map(_.candidate).partition(candidate => !query.excludedIds.contains(candidate.id))\n\n    val filterResult = FilterResult(kept = kept, removed = removed)\n    Stitch.value(filterResult)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/FeatureFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject FeatureFilter {\n\n  /**\n   * Builds a Filter using the Feature name as the FilterIdentifier\n   *\n   * @see [[FeatureFilter.fromFeature(identifier, feature)]]\n   */\n  def fromFeature[Candidate <: UniversalNoun[Any]](\n    feature: Feature[Candidate, Boolean]\n  ): Filter[PipelineQuery, Candidate] =\n    FeatureFilter.fromFeature(FilterIdentifier(feature.toString), feature)\n\n  /**\n   * Builds a Filter that keeps candidates when the provided Boolean Feature is present and True.\n   * If the Feature is missing or False, the candidate is removed.\n   *\n   *  {{{\n   *  Filter.fromFeature(\n   *    FilterIdentifier(\"SomeFilter\"),\n   *    feature = SomeFeature\n   *  )\n   *  }}}\n   *\n   * @param identifier A FilterIdentifier for the new filter\n   * @param feature A feature of [Candidate, Boolean] type used to determine whether Candidates will be kept\n   *                            when this feature is present and true otherwise they will be removed.\n   */\n  def fromFeature[Candidate <: UniversalNoun[Any]](\n    identifier: FilterIdentifier,\n    feature: Feature[Candidate, Boolean]\n  ): Filter[PipelineQuery, Candidate] = {\n    val i = identifier\n\n    new Filter[PipelineQuery, Candidate] {\n      override val identifier: FilterIdentifier = i\n\n      override def apply(\n        query: PipelineQuery,\n        candidates: Seq[CandidateWithFeatures[Candidate]]\n      ): Stitch[FilterResult[Candidate]] = {\n        val (keptCandidates, removedCandidates) = candidates.partition { filterCandidate =>\n          filterCandidate.features.getOrElse(feature, false)\n        }\n\n        Stitch.value(\n          FilterResult(\n            kept = keptCandidates.map(_.candidate),\n            removed = removedCandidates.map(_.candidate)))\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/FeatureValueConditionalFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.product_mixer.component_library.filter.FeatureConditionalFilter.IdentifierInfix\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Predicate to apply to candidate feature, to determine whether to apply filter.\n * True indicates we will apply the filter. False indicates to keep candidate and not apply filter.\n * @tparam FeatureValue\n */\ntrait ShouldApplyFilter[FeatureValue] {\n  def apply(feature: FeatureValue): Boolean\n}\n\n/**\n * A filter that applies the [[filter]] for candidates for which [[shouldApplyFilter]] is true, and keeps the others\n * @param feature feature to determine whether to apply underyling filter\n * @param shouldApplyFilter function to determine whether to apply filter\n * @param filter the actual filter to apply if shouldApplyFilter is True\n * @tparam Query The domain model for the query or request\n * @tparam Candidate The type of the candidates\n * @tparam FeatureValueType\n */\ncase class FeatureValueConditionalFilter[\n  -Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  FeatureValueType\n](\n  feature: Feature[Candidate, FeatureValueType],\n  shouldApplyFilter: ShouldApplyFilter[FeatureValueType],\n  filter: Filter[Query, Candidate])\n    extends Filter[Query, Candidate] {\n  override val identifier: FilterIdentifier = FilterIdentifier(\n    feature.toString + IdentifierInfix + filter.identifier.name\n  )\n\n  override val alerts: Seq[Alert] = filter.alerts\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n    val (candidatesToFilter, candidatesToKeep) = candidates.partition { candidate =>\n      shouldApplyFilter(candidate.features.get(feature))\n    }\n    filter.apply(query, candidatesToFilter).map { filterResult =>\n      FilterResult(\n        kept = filterResult.kept ++ candidatesToKeep.map(_.candidate),\n        removed = filterResult.removed)\n    }\n  }\n}\n\nobject FeatureConditionalFilter {\n  val IdentifierInfix = \"FeatureConditional\"\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/HasAuthorIdFeatureFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.product_mixer.component_library.model.candidate.TweetAuthorIdFeature\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * A filter that checks for presence of a successfully hydrated [[TweetAuthorIdFeature]]\n */\ncase class HasAuthorIdFeatureFilter[Candidate <: TweetCandidate]()\n    extends Filter[PipelineQuery, Candidate] {\n\n  override val identifier = FilterIdentifier(\"HasAuthorIdFeature\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n    val (kept, removed) = candidates.partition(_.features.getTry(TweetAuthorIdFeature).isReturn)\n    Stitch.value(FilterResult(kept.map(_.candidate), removed.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/ParamGatedFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.product_mixer.component_library.filter.ParamGatedFilter.IdentifierPrefix\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * A [[Filter]] with [[Conditionally]] based on a [[Param]]\n *\n * @param enabledParam the param to turn this filter on and off\n * @param filter the underlying filter to run when `enabledParam` is true\n * @tparam Query The domain model for the query or request\n * @tparam Candidate The type of the candidates\n */\ncase class ParamGatedFilter[-Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n  enabledParam: Param[Boolean],\n  filter: Filter[Query, Candidate])\n    extends Filter[Query, Candidate]\n    with Filter.Conditionally[Query, Candidate] {\n  override val identifier: FilterIdentifier = FilterIdentifier(\n    IdentifierPrefix + filter.identifier.name)\n  override val alerts: Seq[Alert] = filter.alerts\n  override def onlyIf(query: Query, candidates: Seq[CandidateWithFeatures[Candidate]]): Boolean =\n    Conditionally.and(Filter.Input(query, candidates), filter, query.params(enabledParam))\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = filter.apply(query, candidates)\n}\n\nobject ParamGatedFilter {\n  val IdentifierPrefix = \"ParamGated\"\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/PredicateFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Predicate which will be applied to each candidate. True indicates that the candidate will be\n * @tparam Candidate - the type of the candidate\n */\ntrait ShouldKeepCandidate[Candidate] {\n  def apply(candidate: Candidate): Boolean\n}\n\nobject PredicateFilter {\n\n  /**\n   * Builds a simple Filter out of a predicate function from the candidate to a boolean. For clarity,\n   * we recommend including the name of the shouldKeepCandidate parameter.\n   *\n   *  {{{\n   *  Filter.fromPredicate(\n   *    FilterIdentifier(\"SomeFilter\"),\n   *    shouldKeepCandidate = { candidate: UserCandidate => candidate.id % 2 == 0L }\n   *  )\n   *  }}}\n   *\n   * @param identifier A FilterIdentifier for the new filter\n   * @param shouldKeepCandidate A predicate function from the candidate. Candidates will be kept\n   *                            when this function returns True.\n   */\n  def fromPredicate[Candidate <: UniversalNoun[Any]](\n    identifier: FilterIdentifier,\n    shouldKeepCandidate: ShouldKeepCandidate[Candidate]\n  ): Filter[PipelineQuery, Candidate] = {\n    val i = identifier\n\n    new Filter[PipelineQuery, Candidate] {\n      override val identifier: FilterIdentifier = i\n\n      /**\n       * Filter the list of candidates\n       *\n       * @return a FilterResult including both the list of kept candidate and the list of removed candidates\n       */\n      override def apply(\n        query: PipelineQuery,\n        candidates: Seq[CandidateWithFeatures[Candidate]]\n      ): Stitch[FilterResult[Candidate]] = {\n        val (keptCandidates, removedCandidates) = candidates.map(_.candidate).partition {\n          filterCandidate =>\n            shouldKeepCandidate(filterCandidate)\n        }\n\n        Stitch.value(FilterResult(kept = keptCandidates, removed = removedCandidates))\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/SnowflakeIdAgeFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\n/**\n * @param maxAgeParam Feature Switch configurable for convenience\n * @tparam Candidate The type of the candidates\n */\ncase class SnowflakeIdAgeFilter[Candidate <: UniversalNoun[Long]](\n  maxAgeParam: Param[Duration])\n    extends Filter[PipelineQuery, Candidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"SnowflakeIdAge\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n    val maxAge = query.params(maxAgeParam)\n\n    val (keptCandidates, removedCandidates) = candidates\n      .map(_.candidate)\n      .partition { filterCandidate =>\n        SnowflakeId.timeFromIdOpt(filterCandidate.id) match {\n          case Some(creationTime) =>\n            query.queryTime.since(creationTime) <= maxAge\n          case _ => false\n        }\n      }\n\n    Stitch.value(FilterResult(kept = keptCandidates, removed = removedCandidates))\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/TweetAuthorCountryFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * A [[filter]] that filters candidates based on a country code feature\n *\n * @param countryCodeFeature the feature to filter candidates on\n */\ncase class TweetAuthorCountryFilter[Candidate <: BaseTweetCandidate](\n  countryCodeFeature: Feature[Candidate, Option[String]])\n    extends Filter[PipelineQuery, Candidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"TweetAuthorCountry\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n\n    val userCountry = query.getCountryCode\n\n    val (keptCandidates, removedCandidates) = candidates.partition { filteredCandidate =>\n      val authorCountry = filteredCandidate.features.get(countryCodeFeature)\n\n      (authorCountry, userCountry) match {\n        case (Some(authorCountryCode), Some(userCountryCode)) =>\n          authorCountryCode.equalsIgnoreCase(userCountryCode)\n        case _ => true\n      }\n    }\n\n    Stitch.value(\n      FilterResult(\n        kept = keptCandidates.map(_.candidate),\n        removed = removedCandidates.map(_.candidate)\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/TweetAuthorIsSelfFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetAuthorIdFeature\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * A [[filter]] that filters based on whether query user is the author of the tweet. This will NOT filter empty user ids\n * @note It is recommended to apply [[HasAuthorIdFeatureFilter]] before this, as this will FAIL if feature is unavailable\n *\n * @tparam Candidate The type of the candidates\n */\ncase class TweetAuthorIsSelfFilter[Candidate <: BaseTweetCandidate]()\n    extends Filter[PipelineQuery, Candidate] {\n  override val identifier: FilterIdentifier = FilterIdentifier(\"TweetAuthorIsSelf\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n    val (kept, removed) = candidates.partition { candidate =>\n      val authorId = candidate.features.get(TweetAuthorIdFeature)\n      !query.getOptionalUserId.contains(authorId)\n    }\n\n    val filterResult = FilterResult(\n      kept = kept.map(_.candidate),\n      removed = removed.map(_.candidate)\n    )\n    Stitch.value(filterResult)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/TweetIsNotReplyFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\nimport com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_tweetypie.IsReplyFeature\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Filters out tweets that is a reply to a tweet\n */\ncase class TweetIsNotReplyFilter[Candidate <: BaseTweetCandidate]()\n    extends Filter[PipelineQuery, Candidate] {\n  override val identifier: FilterIdentifier = FilterIdentifier(\"TweetIsNotReply\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n\n    val (kept, removed) = candidates\n      .partition { candidate =>\n        !candidate.features.get(IsReplyFeature)\n      }\n\n    val filterResult = FilterResult(\n      kept = kept.map(_.candidate),\n      removed = removed.map(_.candidate)\n    )\n\n    Stitch.value(filterResult)\n  }\n\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/TweetLanguageFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\ncase class TweetLanguageFilter[Candidate <: BaseTweetCandidate](\n  languageCodeFeature: Feature[Candidate, Option[String]])\n    extends Filter[PipelineQuery, Candidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"TweetLanguage\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n\n    val userAppLanguage = query.getLanguageCode\n\n    val (keptCandidates, removedCandidates) = candidates.partition { filterCandidate =>\n      val tweetLanguage = filterCandidate.features.get(languageCodeFeature)\n\n      (tweetLanguage, userAppLanguage) match {\n        case (Some(tweetLanguageCode), Some(userAppLanguageCode)) =>\n          tweetLanguageCode.equalsIgnoreCase(userAppLanguageCode)\n        case _ => true\n      }\n    }\n\n    Stitch.value(\n      FilterResult(\n        kept = keptCandidates.map(_.candidate),\n        removed = removedCandidates.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/TweetVisibilityFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.util.logging.Logging\nimport com.twitter.product_mixer.component_library.filter.TweetVisibilityFilter._\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.spam.rtf.thriftscala.SafetyLevel\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.tweetypie.{TweetyPie => TweetypieStitchClient}\nimport com.twitter.tweetypie.{thriftscala => TP}\nimport com.twitter.util.Return\nimport com.twitter.util.Try\n\nobject TweetVisibilityFilter {\n  val DefaultTweetIncludes = Set(TP.TweetInclude.TweetFieldId(TP.Tweet.IdField.id))\n  private final val getTweetFieldsFailureMessage = \"TweetyPie.getTweetFields failed: \"\n}\n\ncase class TweetVisibilityFilter[Candidate <: BaseTweetCandidate](\n  tweetypieStitchClient: TweetypieStitchClient,\n  tweetVisibilityPolicy: TP.TweetVisibilityPolicy,\n  safetyLevel: SafetyLevel,\n  tweetIncludes: Set[TP.TweetInclude.TweetFieldId] = DefaultTweetIncludes)\n    extends Filter[PipelineQuery, Candidate]\n    with Logging {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"TweetVisibility\")\n\n  def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n    Stitch\n      .traverse(candidates.map(_.candidate.id)) { tweetId =>\n        tweetypieStitchClient\n          .getTweetFields(tweetId, getTweetFieldsOptions(query.getOptionalUserId))\n          .liftToTry\n      }\n      .map { getTweetFieldsResults: Seq[Try[TP.GetTweetFieldsResult]] =>\n        val (checkedSucceeded, checkFailed) = getTweetFieldsResults.partition(_.isReturn)\n        checkFailed.foreach(e => warn(() => getTweetFieldsFailureMessage, e.throwable))\n        if (checkFailed.nonEmpty) {\n          warn(() =>\n            s\"TweetVisibilityFilter dropped ${checkFailed.size} candidates due to tweetypie failure.\")\n        }\n\n        val allowedTweets = checkedSucceeded.collect {\n          case Return(TP.GetTweetFieldsResult(_, TP.TweetFieldsResultState.Found(found), _, _)) =>\n            found.tweet.id\n        }.toSet\n\n        val (kept, removed) =\n          candidates.map(_.candidate).partition(candidate => allowedTweets.contains(candidate.id))\n\n        FilterResult(kept = kept, removed = removed)\n      }\n  }\n\n  private def getTweetFieldsOptions(userId: Option[Long]) =\n    TP.GetTweetFieldsOptions(\n      forUserId = userId,\n      tweetIncludes = tweetIncludes.toSet,\n      doNotCache = true,\n      visibilityPolicy = tweetVisibilityPolicy,\n      safetyLevel = Some(safetyLevel)\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/UrtUnorderedExcludeIdsCursorFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtUnorderedExcludeIdsCursor\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\ncase class UrtUnorderedExcludeIdsCursorFilter[\n  Candidate <: UniversalNoun[Long],\n  Query <: PipelineQuery with HasPipelineCursor[UrtUnorderedExcludeIdsCursor]\n]() extends Filter[Query, Candidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"UnorderedExcludeIdsCursor\")\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n\n    val excludeIds = query.pipelineCursor.map(_.excludedIds.toSet).getOrElse(Set.empty)\n    val (kept, removed) =\n      candidates.map(_.candidate).partition(candidate => !excludeIds.contains(candidate.id))\n\n    val filterResult = FilterResult(kept = kept, removed = removed)\n    Stitch.value(filterResult)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/list_visibility/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"strato/config/columns/lists/reads:core-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"strato/config/columns/lists/reads:core-strato-client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/list_visibility/ListVisibilityFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter.list_visibility\n\nimport com.twitter.product_mixer.component_library.model.candidate.TwitterListCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.socialgraph.thriftscala.SocialgraphList\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.Fetch\nimport com.twitter.strato.generated.client.lists.reads.CoreOnListClientColumn\n\n/* This Filter queries the core.List.strato column\n * on Strato, and filters out any lists that are not\n * returned. core.List.strato performs an authorization\n * check, and does not return lists the viewer is not authorized\n * to have access to. */\nclass ListVisibilityFilter[Candidate <: UniversalNoun[Long]](\n  listsColumn: CoreOnListClientColumn)\n    extends Filter[PipelineQuery, Candidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"ListVisibility\")\n\n  def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n\n    val listCandidates = candidates.collect {\n      case CandidateWithFeatures(candidate: TwitterListCandidate, _) => candidate\n    }\n\n    Stitch\n      .traverse(\n        listCandidates.map(_.id)\n      ) { listId =>\n        listsColumn.fetcher.fetch(listId)\n      }.map { fetchResults =>\n        fetchResults.collect {\n          case Fetch.Result(Some(list: SocialgraphList), _) => list.id\n        }\n      }.map { allowedListIds =>\n        val (kept, excluded) = candidates.map(_.candidate).partition {\n          case candidate: TwitterListCandidate => allowedListIds.contains(candidate.id)\n          case _ => true\n        }\n        FilterResult(kept, excluded)\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/tweet_impression/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/impressed_tweets\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/filter/tweet_impression/TweetImpressionFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.filter.tweet_impression\n\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.impressed_tweets.ImpressedTweets\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Filters out tweets that the user has seen\n */\ncase class TweetImpressionFilter[Candidate <: BaseTweetCandidate](\n) extends Filter[PipelineQuery, Candidate] {\n\n  override val identifier: FilterIdentifier = FilterIdentifier(\"TweetImpression\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]] = {\n\n    // Set of Tweets that have impressed the user\n    val impressedTweetsSet: Set[Long] = query.features match {\n      case Some(featureMap) => featureMap.getOrElse(ImpressedTweets, Seq.empty).toSet\n      case None => Set.empty\n    }\n\n    val (keptCandidates, removedCandidates) = candidates.partition { filteredCandidate =>\n      !impressedTweetsSet.contains(filteredCandidate.candidate.id)\n    }\n\n    Stitch.value(FilterResult(keptCandidates.map(_.candidate), removedCandidates.map(_.candidate)))\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"src/scala/com/twitter/ml/featurestore/lib\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate/DefinedCountryCodeGate.scala",
    "content": "package com.twitter.product_mixer.component_library.gate\n\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject DefinedCountryCodeGate extends Gate[PipelineQuery] {\n  override val identifier: GateIdentifier = GateIdentifier(\"DefinedCountryCode\")\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] =\n    Stitch.value(query.getCountryCode.isDefined)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate/FeatureGate.scala",
    "content": "package com.twitter.product_mixer.component_library.gate\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.MissingFeatureException\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.gate.GateResult\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MisconfiguredFeatureMapFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\n\ntrait ShouldContinue[Value] {\n\n  /** Given the [[Feature]] value, returns whether the execution should continue */\n  def apply(featureValue: Value): Boolean\n\n  /** If the [[Feature]] is a failure, use this value */\n  def onFailedFeature(t: Throwable): GateResult = GateResult.Stop\n\n  /**\n   * If the [[Feature]], or [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]],\n   * is missing use this value\n   */\n  def onMissingFeature: GateResult = GateResult.Stop\n}\n\nobject FeatureGate {\n\n  def fromFeature(\n    feature: Feature[_, Boolean]\n  ): FeatureGate[Boolean] =\n    FeatureGate.fromFeature(GateIdentifier(feature.toString), feature)\n\n  def fromNegatedFeature(\n    feature: Feature[_, Boolean]\n  ): FeatureGate[Boolean] =\n    FeatureGate.fromNegatedFeature(GateIdentifier(feature.toString), feature)\n\n  def fromFeature(\n    gateIdentifier: GateIdentifier,\n    feature: Feature[_, Boolean]\n  ): FeatureGate[Boolean] =\n    FeatureGate[Boolean](gateIdentifier, feature, identity)\n\n  def fromNegatedFeature(\n    gateIdentifier: GateIdentifier,\n    feature: Feature[_, Boolean]\n  ): FeatureGate[Boolean] =\n    FeatureGate[Boolean](gateIdentifier, feature, !identity(_))\n\n}\n\n/**\n * A [[Gate]] that is actuated based upon the value of the provided feature\n */\ncase class FeatureGate[Value](\n  gateIdentifier: GateIdentifier,\n  feature: Feature[_, Value],\n  continue: ShouldContinue[Value])\n    extends Gate[PipelineQuery] {\n\n  override val identifier: GateIdentifier = gateIdentifier\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    Stitch\n      .value(\n        query.features.map(_.getTry(feature)) match {\n          case Some(Return(value)) => continue(value)\n          case Some(Throw(_: MissingFeatureException)) => continue.onMissingFeature.continue\n          case Some(Throw(t)) => continue.onFailedFeature(t).continue\n          case None =>\n            throw PipelineFailure(\n              MisconfiguredFeatureMapFailure,\n              \"Expected a FeatureMap to be present but none was found, ensure that your\" +\n                \"PipelineQuery has a FeatureMap configured before gating on Feature values\"\n            )\n        }\n      )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate/FirstPageGate.scala",
    "content": "package com.twitter.product_mixer.component_library.gate\n\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Gate used in first page. Use request cursor to determine if the gate should be open or closed.\n */\nobject FirstPageGate extends Gate[PipelineQuery with HasPipelineCursor[_]] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\"FirstPage\")\n\n  // If cursor is first page, then gate should return continue, otherwise return stop\n  override def shouldContinue(query: PipelineQuery with HasPipelineCursor[_]): Stitch[Boolean] =\n    Stitch.value(query.isFirstPage)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate/NoCandidatesGate.scala",
    "content": "package com.twitter.product_mixer.component_library.gate\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.gate.QueryAndCandidateGate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * A Gate that only continues if the previously returned candidates are empty. This is useful\n * for gating dependent candidate pipelines that are intedned to be used as a backfill when there\n * are no candidates available.\n */\ncase class NoCandidatesGate(scope: CandidateScope) extends QueryAndCandidateGate[PipelineQuery] {\n  override val identifier: GateIdentifier = GateIdentifier(\"NoCandidates\")\n  override def shouldContinue(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithDetails]\n  ): Stitch[Boolean] = Stitch.value(scope.partition(candidates).candidatesInScope.isEmpty)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate/NonEmptyAdsQueryStringGate.scala",
    "content": "package com.twitter.product_mixer.component_library.gate\n\nimport com.twitter.product_mixer.component_library.model.query.ads.AdsQuery\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject NonEmptyAdsQueryStringGate extends Gate[PipelineQuery with AdsQuery] {\n  override val identifier: GateIdentifier = GateIdentifier(\"NonEmptyAdsQueryString\")\n\n  override def shouldContinue(query: PipelineQuery with AdsQuery): Stitch[Boolean] = {\n    val queryString = query.searchRequestContext.flatMap(_.queryString)\n    Stitch.value(queryString.exists(_.trim.nonEmpty))\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate/NonEmptyCandidatesGate.scala",
    "content": "package com.twitter.product_mixer.component_library.gate\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.gate.QueryAndCandidateGate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * A Gate that only continues if the previously returned candidates are not empty. This is useful\n * for gating dependent candidate pipelines that are intended to only be used if a previous pipeline\n * completed successfully.\n */\ncase class NonEmptyCandidatesGate(scope: CandidateScope)\n    extends QueryAndCandidateGate[PipelineQuery] {\n  override val identifier: GateIdentifier = GateIdentifier(\"NonEmptyCandidates\")\n  override def shouldContinue(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithDetails]\n  ): Stitch[Boolean] = Stitch.value(scope.partition(candidates).candidatesInScope.nonEmpty)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate/QualityFactorGate.scala",
    "content": "package com.twitter.product_mixer.component_library.gate\n\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.stitch.Stitch\n\n/**\n * A Gate that only continues if the quality factor value of the pipeline is above the given\n * threshold. This is useful for disabling an expensive function when the pipeline is under pressure\n * (quality factor is low).\n */\ncase class QualityFactorGate(pipelineIdentifier: ComponentIdentifier, threshold: Double)\n    extends Gate[PipelineQuery with HasQualityFactorStatus] {\n\n  override val identifier: GateIdentifier = GateIdentifier(\n    s\"${pipelineIdentifier.name}QualityFactor\")\n\n  override def shouldContinue(\n    query: PipelineQuery with HasQualityFactorStatus\n  ): Stitch[Boolean] =\n    Stitch.value(query.getQualityFactorCurrentValue(pipelineIdentifier) >= threshold)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate/any_candidates_without_feature/AnyCandidatesWithoutFeatureGate.scala",
    "content": "package com.twitter.product_mixer.component_library.gate.any_candidates_without_feature\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.gate.QueryAndCandidateGate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * A gate that enables a component only if any candidates are missing a specific feature.\n * You can restrict which candidates to check with the scope parameter.\n * This is most commonly used to do backfill scoring, where you can have one Scoring Pipeline that\n * might return a score feature \"FeatureA\" and another sequential pipeline that you only want to run\n * if the previous scoring pipeline fails to hydrate for all candidates.\n * @param identifier Unique identifier for this gate. Typically, AnyCandidatesWithout{YourFeature}.\n * @param scope A [[CandidateScope]] to specify which candidates to check.\n * @param missingFeature The feature that should be missing for any of the candidates for this gate to continue\n */\ncase class AnyCandidatesWithoutFeatureGate(\n  override val identifier: GateIdentifier,\n  scope: CandidateScope,\n  missingFeature: Feature[_, _])\n    extends QueryAndCandidateGate[PipelineQuery] {\n\n  override def shouldContinue(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithDetails]\n  ): Stitch[Boolean] =\n    Stitch.value(scope.partition(candidates).candidatesInScope.exists { candidateWithDetails =>\n      !candidateWithDetails.features.getSuccessfulFeatures.contains(missingFeature)\n    })\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/gate/any_candidates_without_feature/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ArticleCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\ntrait BaseArticleCandidate extends UniversalNoun[Int]\n\n/**\n * Canonical ArticleCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class ArticleCandidate private (\n  override val id: Int)\n    extends BaseArticleCandidate {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[ArticleCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: ArticleCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject ArticleCandidate {\n  def apply(id: Int): ArticleCandidate = new ArticleCandidate(id)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/AudioSpaceCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical AudioSpaceCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class AudioSpaceCandidate private (\n  override val id: String)\n    extends UniversalNoun[String] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[AudioSpaceCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: AudioSpaceCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject AudioSpaceCandidate {\n  def apply(id: String): AudioSpaceCandidate = new AudioSpaceCandidate(id)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/CardCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical CardCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class CardCandidate private (\n  override val id: String)\n    extends UniversalNoun[String] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[CardCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: CardCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject CardCandidate {\n  def apply(id: String): CardCandidate = new CardCandidate(id)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/CommerceItemCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical CommerceProductCandidate model which encapsulates information about a specific Product.\n * Always prefer this version over all other variants. For example, iPhone 14, 128 GB, White. When a\n * user clicks on a CommerceProduct, they will be taken to the specific product page.\n *\n * @note Both CommerceProduct and CommerceProductGroups (below) can be shown in the same\n *       TimelineModule (i.e Carousel)\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class CommerceProductCandidate private (\n  override val id: Long)\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[CommerceProductCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: CommerceProductCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode) && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject CommerceProductCandidate {\n  def apply(id: Long): CommerceProductCandidate = new CommerceProductCandidate(id)\n}\n\n/**\n * Canonical CommerceProductGroupCandidate model which encapsulates information about a Single\n * Product Type and its corresponding versions. Always prefer this version over all other variants.\n * For example:\n * iPhone 14\n *   - 128 GB, White\n *   - 128 GB, Blue\n *   - 1TB, Grey\n * When a user clicks on a Product Group, they will be shown information about all of the possible\n * versions of the top level product.\n *\n * @note Both CommerceProduct (above) and CommerceProductGroups can be shown in the same\n *       TimelineModule (i.e Carousel)\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class CommerceProductGroupCandidate private (\n  override val id: Long)\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[CommerceProductGroupCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: CommerceProductGroupCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode) && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *         (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *         data structure), assuming stable hashCode implementations for these objects\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject CommerceProductGroupCandidate {\n  def apply(id: Long): CommerceProductGroupCandidate = new CommerceProductGroupCandidate(id)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/CursorCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\nsealed trait CursorType\ncase object PreviousCursor extends CursorType\ncase object NextCursor extends CursorType\n\n/**\n * Canonical CursorCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class CursorCandidate private (\n  override val id: Long,\n  val value: String,\n  val cursorType: CursorType)\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[CursorCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: CursorCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && value == candidate.value && cursorType == candidate.cursorType))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      31 * (\n        id.##\n      ) + value.##\n    ) + cursorType.##\n}\n\nobject CursorCandidate {\n  def apply(id: Long, value: String, cursorType: CursorType): CursorCandidate =\n    new CursorCandidate(id, value, cursorType)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/DMConvoCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\ntrait BaseDMConvoCandidate extends UniversalNoun[String] {\n  def lastReadableEventId: Option[Long]\n}\n\n/**\n * Canonical DMConvoCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class DMConvoCandidate private (\n  override val id: String,\n  override val lastReadableEventId: Option[Long])\n    extends BaseDMConvoCandidate {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[DMConvoCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: DMConvoCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && lastReadableEventId == candidate.lastReadableEventId))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      id.##\n    ) + lastReadableEventId.##\n}\n\nobject DMConvoCandidate {\n  def apply(id: String, lastReadableEventId: Option[Long]): DMConvoCandidate =\n    new DMConvoCandidate(id, lastReadableEventId)\n}\n\n/**\n * Canonical DMConvoSearchCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class DMConvoSearchCandidate private (\n  override val id: String,\n  override val lastReadableEventId: Option[Long])\n    extends BaseDMConvoCandidate {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[DMConvoSearchCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: DMConvoSearchCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && lastReadableEventId == candidate.lastReadableEventId))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *         (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *         data structure), assuming stable hashCode implementations for these objects\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      id.##\n    ) + lastReadableEventId.##\n}\n\nobject DMConvoSearchCandidate {\n  def apply(id: String, lastReadableEventId: Option[Long]): DMConvoSearchCandidate =\n    new DMConvoSearchCandidate(id, lastReadableEventId)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/DMEventCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical DM Events such as Message Create, Conversation Create, Join conversation, etc model.\n * Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class DMEventCandidate private (\n  override val id: Long)\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[DMEventCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: DMEventCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject DMEventCandidate {\n  def apply(id: Long): DMEventCandidate = new DMEventCandidate(id)\n}\n\n/**\n * Represent DM Events such as Message Create, Conversation Create, Join conversation, etc.\n *\n * @note historically this was used to represent events from Elastic\n *       Search rather than Strato. Now deprecated in favor of DMEvent.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n *\n */\n@deprecated(\"Prefer DMEvent\")\nfinal class DMMessageSearchCandidate private (\n  override val id: Long)\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[DMMessageSearchCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: DMMessageSearchCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *         (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *         data structure), assuming stable hashCode implementations for these objects\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject DMMessageSearchCandidate {\n  def apply(id: Long): DMMessageSearchCandidate = new DMMessageSearchCandidate(id)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/GenericSummaryCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical GenericSummaryCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class GenericSummaryCandidate private (\n  override val id: String)\n    extends UniversalNoun[String] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[GenericSummaryCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: GenericSummaryCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject GenericSummaryCandidate {\n  def apply(id: String): GenericSummaryCandidate = new GenericSummaryCandidate(id)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/LabelCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical LabelCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class LabelCandidate private (\n  override val id: Long)\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[LabelCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: LabelCandidate =>\n        ((this eq candidate)\n          || ((hashCode == candidate.hashCode) && (id == candidate.id)))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject LabelCandidate {\n  def apply(id: Long): LabelCandidate = new LabelCandidate(id)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/MomentAnnotationCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical MomentAnnotationCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class MomentAnnotationCandidate private (\n  override val id: Long,\n  val text: Option[String],\n  val header: Option[String])\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[MomentAnnotationCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: MomentAnnotationCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && text == candidate.text && header == candidate.header))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      31 * (\n        id.##\n      ) + text.##\n    ) + header.##\n}\n\nobject MomentAnnotationCandidate {\n  def apply(\n    id: Long,\n    text: Option[String],\n    header: Option[String]\n  ): MomentAnnotationCandidate = new MomentAnnotationCandidate(id, text, header)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/PromptCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\nsealed trait BasePromptCandidate[+T] extends UniversalNoun[T]\n\n/**\n * Canonical InlinePromptCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class InlinePromptCandidate private (\n  override val id: String)\n    extends BasePromptCandidate[String] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[InlinePromptCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: InlinePromptCandidate =>\n        ((this eq candidate)\n          || ((hashCode == candidate.hashCode) && (id == candidate.id)))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject InlinePromptCandidate {\n  def apply(id: String): InlinePromptCandidate = new InlinePromptCandidate(id)\n}\n\n/**\n * Canonical CompactPromptCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class CompactPromptCandidate private (\n  override val id: Long)\n    extends BasePromptCandidate[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[CompactPromptCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: CompactPromptCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode) && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *         (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *         data structure), assuming stable hashCode implementations for these objects\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject CompactPromptCandidate {\n  def apply(id: Long): CompactPromptCandidate = new CompactPromptCandidate(id)\n}\n\n/**\n * Canonical FullCoverPromptCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class FullCoverPromptCandidate private (\n  override val id: String)\n    extends BasePromptCandidate[String] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[FullCoverPromptCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: FullCoverPromptCandidate =>\n        ((this eq candidate)\n          || ((hashCode == candidate.hashCode) && (id == candidate.id)))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *         (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *         data structure), assuming stable hashCode implementations for these objects\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject FullCoverPromptCandidate {\n  def apply(id: String): FullCoverPromptCandidate = new FullCoverPromptCandidate(id)\n}\n\n/**\n * Canonical HalfCoverPromptCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class HalfCoverPromptCandidate private (\n  override val id: String)\n    extends BasePromptCandidate[String] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[HalfCoverPromptCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: HalfCoverPromptCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *         (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *         data structure), assuming stable hashCode implementations for these objects\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject HalfCoverPromptCandidate {\n  def apply(id: String): HalfCoverPromptCandidate = new HalfCoverPromptCandidate(id)\n}\n\n/**\n * Canonical PromptCarouselTileCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class PromptCarouselTileCandidate private (\n  override val id: Long)\n    extends BasePromptCandidate[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[PromptCarouselTileCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: PromptCarouselTileCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *         (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *         data structure), assuming stable hashCode implementations for these objects\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject PromptCarouselTileCandidate {\n  def apply(id: Long): PromptCarouselTileCandidate = new PromptCarouselTileCandidate(id)\n}\n\n/**\n * Canonical RelevancePromptCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class RelevancePromptCandidate private (\n  override val id: String,\n  val position: Option[Int])\n    extends BasePromptCandidate[String] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[RelevancePromptCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: RelevancePromptCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && position == candidate.position))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *         (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *         data structure), assuming stable hashCode implementations for these objects\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      id.##\n    ) + position.##\n}\n\nobject RelevancePromptCandidate {\n  def apply(\n    id: String,\n    position: Option[Int] = None\n  ): RelevancePromptCandidate =\n    new RelevancePromptCandidate(id, position)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ShowAlertCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical ShowAlertCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class ShowAlertCandidate private (\n  override val id: String,\n  val userIds: Seq[Long])\n    extends UniversalNoun[String] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[ShowAlertCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: ShowAlertCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && userIds == candidate.userIds))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      id.##\n    ) + userIds.##\n}\n\nobject ShowAlertCandidate {\n  def apply(id: String, userIds: Seq[Long]): ShowAlertCandidate =\n    new ShowAlertCandidate(id, userIds)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/TopicCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\ntrait BaseTopicCandidate extends UniversalNoun[Long]\n\n/**\n * Canonical TopicCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class TopicCandidate private (\n  override val id: Long)\n    extends BaseTopicCandidate {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[TopicCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: TopicCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode) && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject TopicCandidate {\n  def apply(id: Long): TopicCandidate = new TopicCandidate(id)\n}\n\n/**\n * Canonical CategorizedTopicCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\n@deprecated(\"Prefer TopicCandidate\")\nfinal class CategorizedTopicCandidate private (\n  override val id: Long,\n  val categoryId: Option[Long],\n  val categoryName: Option[String])\n    extends BaseTopicCandidate {\n\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[CategorizedTopicCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: CategorizedTopicCandidate =>\n        (\n          (this eq candidate)\n            || (\n              (hashCode == candidate.hashCode)\n                && (id == candidate.id && categoryId == candidate.categoryId && categoryName == candidate.categoryName)\n            )\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *         (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *         data structure), assuming stable hashCode implementations for these objects\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      31 * (\n        id.##\n      ) + categoryId.##\n    ) + categoryName.##\n}\n\nobject CategorizedTopicCandidate {\n  def apply(\n    id: Long,\n    categoryId: Option[Long] = None,\n    categoryName: Option[String] = None\n  ): CategorizedTopicCandidate =\n    new CategorizedTopicCandidate(id, categoryId, categoryName)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/TweetCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.fasterxml.jackson.annotation.JsonTypeName\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n// JSON type annotations are needed for identifying renderable entities to Turntable, most candidates\n// do not need them.\n@JsonTypeName(\"tweet\")\ntrait BaseTweetCandidate extends UniversalNoun[Long]\n\n/**\n * Canonical TweetCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class TweetCandidate private (\n  override val id: Long)\n    extends BaseTweetCandidate {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[TweetCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: TweetCandidate =>\n        ((this eq candidate)\n          || ((hashCode == candidate.hashCode) && (id == candidate.id)))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject TweetCandidate {\n  def apply(id: Long): TweetCandidate = new TweetCandidate(id)\n}\n\n/**\n * Tweet Author User ID of a given Tweet Candidate. This is typically needed when hydrating tweet\n * author extended features in Feature Store (e.g, [[TweetCandidateAuthorIdEntity]]). This feature\n * is typically extracted by hydrating it from Tweetypie, or extracting it in your candidate source\n * if it returns the Author ID alongside Tweet ID using a [[CandidatePipelineResultsTransformer]]\n */\nobject TweetAuthorIdFeature extends Feature[TweetCandidate, Long]\n\n/**\n * Whether the tweet should be pinned when marshalled to URT or not.\n * See [[com.twitter.product_mixer.component_library.decorator.urt.builder.item.tweet.TweetCandidateUrtItemBuilder]]\n */\nobject IsPinnedFeature extends Feature[TweetCandidate, Boolean]\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/TwitterListCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical TwitterListCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class TwitterListCandidate private (\n  override val id: Long)\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[TwitterListCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: TwitterListCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode) && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject TwitterListCandidate {\n  def apply(id: Long): TwitterListCandidate = new TwitterListCandidate(id)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/UserCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate\n\nimport com.fasterxml.jackson.annotation.JsonTypeName\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n// JSON type annotations are needed for identifying renderable entities to Turntable, most candidates\n// do not need them.\n@JsonTypeName(\"user\")\ntrait BaseUserCandidate extends UniversalNoun[Long]\n\n/**\n * Canonical UserCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class UserCandidate private (\n  override val id: Long)\n    extends BaseUserCandidate {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[UserCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: UserCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode) && (id == candidate.id))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject UserCandidate {\n  def apply(id: Long): UserCandidate = new UserCandidate(id)\n}\n\n/**\n * Feature to indicate whether a rendered user candidate should be marked unread in URT. Used in\n * [[UserCandidateUrtItemBuilder]] when decorating the candidate.r\n */\nobject IsMarkUnreadFeature extends Feature[BaseUserCandidate, Boolean]\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ads/AdsCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate.ads\n\nimport com.twitter.adserver.{thriftscala => adsthrift}\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * An [[AdsCandidate]] represents a piece of promoted content.\n *\n * This candidate class stores a reference to the adImpression, which is the common thrift structure\n * used by the Ads team to represent an ad.\n *\n * Goldfinch, the ads-injection library, consumes the [[AdImpression]].\n */\nsealed trait AdsCandidate extends UniversalNoun[Any] {\n  val adImpression: adsthrift.AdImpression\n}\n\n/**\n * Canonical AdsTweetCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class AdsTweetCandidate private (\n  override val id: Long,\n  override val adImpression: adsthrift.AdImpression)\n    extends AdsCandidate\n    with BaseTweetCandidate {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[AdsTweetCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: AdsTweetCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && adImpression == candidate.adImpression))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      id.##\n    ) + adImpression.##\n}\n\nobject AdsTweetCandidate {\n  def apply(id: Long, adImpression: adsthrift.AdImpression): AdsTweetCandidate =\n    new AdsTweetCandidate(id, adImpression)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ads/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    scalac_plugins = [\"no-roomba\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/model\",\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/model/promoted\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_common-scala\",\n    ],\n    exports = [\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/model\",\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/model/promoted\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_common-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/hubble/AdCreativeCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate.hubble\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.AdType\n\n/**\n * Canonical AdCreativeCandidate model which describes an Ad Creative from an ad management\n * perspective. It can be a tweet, or account, and has a 1:n relationship with ad units. Always\n * prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class AdCreativeCandidate private (\n  // This is the creativeId, but needs to be named ID to confirm to UniversalNoun\n  override val id: Long,\n  val adType: AdType,\n  val adAccountId: Long)\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[AdCreativeCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: AdCreativeCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && adType == candidate.adType && adAccountId == candidate.adAccountId))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      31 * (\n        id.##\n      ) + adType.##\n    ) + adAccountId.##\n}\n\nobject AdCreativeCandidate {\n  def apply(\n    id: Long,\n    adType: AdType,\n    adAccountId: Long\n  ): AdCreativeCandidate =\n    new AdCreativeCandidate(id, adType, adAccountId)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/hubble/AdGroupCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate.hubble\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical AdGroupCandidate model which describes an \"Ad Group\" from the the Ad Management\n * perspective. It is based on the LineItem table in Ads DB, and provides an ad group for\n * advertisers to manage and report different line items belonging to a single ad. Always prefer\n * this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class AdGroupCandidate private (\n  override val id: Long, // This is the ad_group_id, renamed to ID to conform to UniversalNoun\n  val adAccountId: Long)\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[AdGroupCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: AdGroupCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && adAccountId == candidate.adAccountId))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      id.##\n    ) + adAccountId.##\n}\n\nobject AdGroupCandidate {\n  def apply(\n    id: Long,\n    adAccountId: Long\n  ): AdGroupCandidate =\n    new AdGroupCandidate(id, adAccountId)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/hubble/AdUnitCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate.hubble\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical AdUnitCandidate model which describes an \"Ad\" from the Ad Management perspective. It is\n * based on the AdUnit table in Ads DB, and provides a candidate for advertisers to manage and\n * report on their advertising configurations.Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class AdUnitCandidate private (\n  // This is the adUnitId, but needs to be named ID to confirm to UniversalNoun\n  override val id: Long,\n  val adAccountId: Long)\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[AdUnitCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: AdUnitCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && adAccountId == candidate.adAccountId))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      id.##\n    ) + adAccountId.##\n}\n\nobject AdUnitCandidate {\n  def apply(\n    id: Long,\n    adAccountId: Long\n  ): AdUnitCandidate =\n    new AdUnitCandidate(id, adAccountId)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/hubble/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/hubble/CampaignCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate.hubble\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical CampaignCandidate model which describes a \"Campaign\" from the Ads Management\n * perspective. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class CampaignCandidate private (\n  // This is the campaignId, but needs to be named id to conform to UniversalNoun\n  override val id: Long,\n  val adAccountId: Long)\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[CampaignCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: CampaignCandidate =>\n        ((this eq candidate)\n          || ((hashCode == candidate.hashCode) && (id == candidate.id)))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = id.##\n}\n\nobject CampaignCandidate {\n  def apply(id: Long, adAccountId: Long): CampaignCandidate =\n    new CampaignCandidate(id, adAccountId)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/hubble/FundingSourceCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate.hubble\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/**\n * Canonical FundingSourceCandidate model which describes a \"Funding Instrument\" from the the Ad\n * Management perspective. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class FundingSourceCandidate private (\n  override val id: Long,\n  val adAccountId: Long)\n    extends UniversalNoun[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[FundingSourceCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: FundingSourceCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && adAccountId == candidate.adAccountId))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      id.##\n    ) + adAccountId.##\n}\n\nobject FundingSourceCandidate {\n  def apply(\n    id: Long,\n    adAccountId: Long\n  ): FundingSourceCandidate = new FundingSourceCandidate(id, adAccountId)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/suggestion/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/suggestion/QuerySuggestionCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate.suggestion\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.TypeaheadMetadata\n\n/**\n * Represents a query suggestion in typeahead\n */\nsealed trait BaseQuerySuggestionCandidate[+T] extends UniversalNoun[T]\n\n/**\n * Canonical QuerySuggestionCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class QuerySuggestionCandidate private (\n  override val id: String,\n  val metadata: Option[TypeaheadMetadata])\n    extends BaseQuerySuggestionCandidate[String] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[QuerySuggestionCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: QuerySuggestionCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && metadata == candidate.metadata))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      id.##\n    ) + metadata.##\n}\n\nobject QuerySuggestionCandidate {\n  def apply(\n    id: String,\n    metadata: Option[TypeaheadMetadata] = None\n  ): QuerySuggestionCandidate = new QuerySuggestionCandidate(id, metadata)\n}\n\n/**\n * Canonical TypeaheadQueryCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n *\n */\nfinal class TypeaheadQueryCandidate(\n  override val id: String,\n  val score: Option[Double])\n    extends BaseQuerySuggestionCandidate[String] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[TypeaheadQueryCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: TypeaheadQueryCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && score == candidate.score))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *         (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *         data structure), assuming stable hashCode implementations for these objects\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      id.##\n    ) + score.##\n}\n\nobject TypeaheadQueryCandidate {\n  def apply(id: String, score: Option[Double]): TypeaheadQueryCandidate =\n    new TypeaheadQueryCandidate(id, score)\n}\n\nfinal class TypeaheadEventCandidate private (\n  override val id: Long,\n  val metadata: Option[TypeaheadMetadata])\n    extends BaseQuerySuggestionCandidate[Long] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[TypeaheadQueryCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: TypeaheadEventCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && metadata == candidate.metadata))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *         (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *         data structure), assuming stable hashCode implementations for these objects\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      id.##\n    ) + metadata.##\n}\n\nobject TypeaheadEventCandidate {\n  def apply(\n    id: Long,\n    metadata: Option[TypeaheadMetadata] = None\n  ): TypeaheadEventCandidate = new TypeaheadEventCandidate(id, metadata)\n}\n\n/**\n * Canonical TweetAnnotationQueryCandidate model. Always prefer this version over all other variants.\n *\n * TODO Remove score from the candidate and use a Feature instead\n */\nfinal class TweetAnnotationQueryCandidate private (\n  override val id: String,\n  val score: Option[Double])\n    extends BaseQuerySuggestionCandidate[String] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[TweetAnnotationQueryCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: TweetAnnotationQueryCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && score == candidate.score))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *         (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *         data structure), assuming stable hashCode implementations for these objects\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      id.##\n    ) + score.##\n}\n\nobject TweetAnnotationQueryCandidate {\n  def apply(id: String, score: Option[Double]): TweetAnnotationQueryCandidate =\n    new TweetAnnotationQueryCandidate(id, score)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/suggestion/SpellingSuggestionCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate.suggestion\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.suggestion.SpellingActionType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.suggestion.TextResult\n\n/**\n * Canonical SpellingSuggestionCandidate model. Always prefer this version over all other variants.\n *\n * @note Any additional fields should be added as a [[com.twitter.product_mixer.core.feature.Feature]]\n *       on the candidate's [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]]. If the\n *       features come from the candidate source itself (as opposed to hydrated via a\n *       [[com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator]]),\n *       then [[com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig.featuresFromCandidateSourceTransformers]]\n *       can be used to extract features from the candidate source response.\n *\n * @note This class should always remain `final`. If for any reason the `final` modifier is removed,\n *       the equals() implementation must be updated in order to handle class inheritor equality\n *       (see note on the equals method below)\n */\nfinal class SpellingSuggestionCandidate private (\n  override val id: String,\n  val textResult: TextResult,\n  val spellingActionType: Option[SpellingActionType],\n  val originalQuery: Option[String])\n    extends UniversalNoun[String] {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[SpellingSuggestionCandidate]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case candidate: SpellingSuggestionCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id && textResult == candidate.textResult && spellingActionType == candidate.spellingActionType && originalQuery == candidate.originalQuery))\n        )\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated candidate\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int =\n    31 * (\n      31 * (\n        31 * (\n          id.##\n        ) + textResult.##\n      ) + spellingActionType.##\n    ) + originalQuery.##\n}\n\nobject SpellingSuggestionCandidate {\n  def apply(\n    id: String,\n    textResult: TextResult,\n    spellingActionType: Option[SpellingActionType],\n    originalQuery: Option[String]\n  ): SpellingSuggestionCandidate =\n    new SpellingSuggestionCandidate(id, textResult, spellingActionType, originalQuery)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/trends_events/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/trends_events/UnifiedTrendEventCandidate.scala",
    "content": "package com.twitter.product_mixer.component_library.model.candidate.trends_events\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.event.EventSummaryDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.trend.GroupedTrend\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageVariant\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DisclosureType\n\n/**\n * An [[UnifiedTrendEventCandidate]] represents a piece of Event or Trend content.\n * The Event and Trend candidate are represented by different types of keys that Event has a Long\n * eventId while Trend has a String trendName.\n */\nsealed trait UnifiedTrendEventCandidate[+T] extends UniversalNoun[T]\n\nfinal class UnifiedEventCandidate private (\n  override val id: Long)\n    extends UnifiedTrendEventCandidate[Long] {\n\n  override def canEqual(that: Any): Boolean = this.isInstanceOf[UnifiedEventCandidate]\n\n  override def equals(that: Any): Boolean = {\n    that match {\n      case candidate: UnifiedEventCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id))\n        )\n      case _ => false\n    }\n  }\n\n  override val hashCode: Int = id.##\n}\n\nobject UnifiedEventCandidate {\n  def apply(id: Long): UnifiedEventCandidate = new UnifiedEventCandidate(id)\n}\n\n/**\n * Text description of an Event. Usually this is extracted from curated Event metadata\n */\nobject EventTitleFeature extends Feature[UnifiedEventCandidate, String]\n\n/**\n * Display type of an Event. This will be used for client to differentiate if this Event will be\n * displayed as a normal cell, a hero, etc.\n */\nobject EventDisplayType extends Feature[UnifiedEventCandidate, EventSummaryDisplayType]\n\n/**\n * URL that servces as the landing page of an Event\n */\nobject EventUrl extends Feature[UnifiedEventCandidate, Url]\n\n/**\n * Use to render an Event cell's editorial image\n */\nobject EventImage extends Feature[UnifiedEventCandidate, Option[ImageVariant]]\n\n/**\n * Localized time string like \"LIVE\" or \"Last Night\" that is used to render the Event cell\n */\nobject EventTimeString extends Feature[UnifiedEventCandidate, Option[String]]\n\nfinal class UnifiedTrendCandidate private (\n  override val id: String)\n    extends UnifiedTrendEventCandidate[String] {\n\n  override def canEqual(that: Any): Boolean = this.isInstanceOf[UnifiedTrendCandidate]\n\n  override def equals(that: Any): Boolean = {\n    that match {\n      case candidate: UnifiedTrendCandidate =>\n        (\n          (this eq candidate)\n            || ((hashCode == candidate.hashCode)\n              && (id == candidate.id))\n        )\n      case _ => false\n    }\n  }\n\n  override val hashCode: Int = id.##\n}\n\nobject UnifiedTrendCandidate {\n  def apply(id: String): UnifiedTrendCandidate = new UnifiedTrendCandidate(id)\n}\n\nobject TrendNormalizedTrendName extends Feature[UnifiedTrendCandidate, String]\n\nobject TrendTrendName extends Feature[UnifiedTrendCandidate, String]\n\nobject TrendUrl extends Feature[UnifiedTrendCandidate, Url]\n\nobject TrendDescription extends Feature[UnifiedTrendCandidate, Option[String]]\n\nobject TrendTweetCount extends Feature[UnifiedTrendCandidate, Option[Int]]\n\nobject TrendDomainContext extends Feature[UnifiedTrendCandidate, Option[String]]\n\nobject TrendGroupedTrends extends Feature[UnifiedTrendCandidate, Option[Seq[GroupedTrend]]]\n\nobject PromotedTrendNameFeature extends Feature[UnifiedTrendCandidate, Option[String]]\n\nobject PromotedTrendDescriptionFeature extends Feature[UnifiedTrendCandidate, Option[String]]\n\nobject PromotedTrendAdvertiserNameFeature extends Feature[UnifiedTrendCandidate, Option[String]]\n\nobject PromotedTrendIdFeature extends Feature[UnifiedTrendCandidate, Option[Long]]\n\nobject PromotedTrendDisclosureTypeFeature\n    extends Feature[UnifiedTrendCandidate, Option[DisclosureType]]\n\nobject PromotedTrendImpressionIdFeature extends Feature[UnifiedTrendCandidate, Option[String]]\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/scala/com/twitter/search/common/util/bloomfilter\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/scala/com/twitter/search/common/util/bloomfilter\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor/OrderedCursor.scala",
    "content": "package com.twitter.product_mixer.component_library.model.cursor\n\nimport com.twitter.product_mixer.core.pipeline.PipelineCursor\nimport com.twitter.product_mixer.core.pipeline.UrtPipelineCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.{\n  CursorType => UrtCursorType\n}\n\n/**\n * Cursor model that may be used when cursoring over an ordered candidate source.\n *\n * @param initialSortIndex See [[UrtPipelineCursor]]\n * @param id represents the ID of the element, typically the top element for a top cursor or the\n *           bottom element for a bottom cursor, in an ordered candidate list\n * @param gapBoundaryId represents the ID of the gap boundary element, which in gap cursors is the\n *                      opposite bound of the gap to be filled with the cursor\n */\ncase class UrtOrderedCursor(\n  override val initialSortIndex: Long,\n  id: Option[Long],\n  cursorType: Option[UrtCursorType],\n  gapBoundaryId: Option[Long] = None)\n    extends UrtPipelineCursor\n\ncase class OrderedCursor(\n  id: Option[Long],\n  cursorType: Option[CursorType],\n  gapBoundaryId: Option[Long] = None)\n    extends PipelineCursor\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor/PassThroughCursor.scala",
    "content": "package com.twitter.product_mixer.component_library.model.cursor\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorType\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.UrtPipelineCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.{\n  CursorType => UrtCursorType\n}\n\ncase object PreviousCursorFeature\n    extends Feature[PipelineQuery with HasPipelineCursor[UrtPassThroughCursor], String]\n\ncase object NextCursorFeature\n    extends Feature[PipelineQuery with HasPipelineCursor[UrtPassThroughCursor], String]\n\n/**\n * Cursor model that may be used when we want to pass through the cursor value from and back to\n * a downstream as-is.\n *\n * @param initialSortIndex See [[UrtPipelineCursor]]\n * @param cursorValue the pass through cursor\n */\ncase class UrtPassThroughCursor(\n  override val initialSortIndex: Long,\n  cursorValue: String,\n  cursorType: Option[UrtCursorType] = None)\n    extends UrtPipelineCursor\n\ncase class PassThroughCursor(\n  cursorValue: String,\n  cursorType: Option[CursorType] = None)\n    extends PipelineCursor\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor/UnorderedBloomFilterCursor.scala",
    "content": "package com.twitter.product_mixer.component_library.model.cursor\n\nimport com.twitter.product_mixer.core.pipeline.PipelineCursor\nimport com.twitter.product_mixer.core.pipeline.UrtPipelineCursor\nimport com.twitter.search.common.util.bloomfilter.AdaptiveLongIntBloomFilter\n\n/**\n * Cursor model that may be used when cursoring over a unordered candidate source. On each server\n * round-trip, the server will add the IDs of the candidates into a space efficient bloom filter.\n * Then on subsequent requests the client will return the cursor, and the bloom filter can be sent to\n * the downstream's bloom filter parameter in serialized form, or exclude candidates locally via a\n * filter on the candidate source pipeline.\n *\n * @param initialSortIndex See [[UrtPipelineCursor]]\n * @param longIntBloomFilter the bloom filter to use to dedup candidate from the candidate list\n */\ncase class UrtUnorderedBloomFilterCursor(\n  override val initialSortIndex: Long,\n  // space-efficient and mutable variant of the BloomFilter class used for storing long integers.\n  longIntBloomFilter: AdaptiveLongIntBloomFilter)\n    extends UrtPipelineCursor\n\ncase class UnorderedBloomFilterCursor(\n  longIntBloomFilter: AdaptiveLongIntBloomFilter)\n    extends PipelineCursor\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor/UnorderedExcludeIdsCursor.scala",
    "content": "package com.twitter.product_mixer.component_library.model.cursor\n\nimport com.twitter.product_mixer.core.pipeline.PipelineCursor\nimport com.twitter.product_mixer.core.pipeline.UrtPipelineCursor\n\n/**\n * URT Cursor model that may be used when cursoring over a unordered candidate source. On each server\n * round-trip, the server will append the IDs of the elements in the response to the cursor. Then\n * on subsequent requests the client will return the cursor, and the excludedIds list can be sent to\n * the downstream's excludeIds parameter, or excluded locally via a filter on the candidate source\n * pipeline.\n *\n * Note that the cursor is bounded, as the excludedIds list cannot be appended to indefinitely due\n * to payload size constraints. As such, this strategy is typically used for bounded (limited page\n * size) products, or for unbounded (unlimited page size) products in conjunction with an\n * impression store. In the latter case, the cursor excludedIds list would be limited to a max size\n * via a circular buffer implementation, which would be unioned with the impression store IDs when\n * filtering. This usage allows the impression store to \"catch up\", as there is often latency\n * between when an impression client event is sent by the client and storage in the impression\n * store.\n *\n * @param initialSortIndex See [[UrtPipelineCursor]]\n * @param excludedIds the list of IDs to exclude from the candidate list\n */\ncase class UrtUnorderedExcludeIdsCursor(\n  override val initialSortIndex: Long,\n  excludedIds: Seq[Long])\n    extends UrtPipelineCursor\n\ncase class UnorderedExcludeIdsCursor(excludedIds: Seq[Long]) extends PipelineCursor\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor/UrtPlaceholderCursor.scala",
    "content": "package com.twitter.product_mixer.component_library.model.cursor\n\nimport com.twitter.product_mixer.core.pipeline.UrtPipelineCursor\n\n/**\n * Cursor model that may be used when we just need a placeholder but no real cursor value. Since URT\n * requires that top and bottom cursors are always present, placeholders are often used when up\n * scrolling (PTR) is not supported on a timeline. While placeholder cursors generally should not be\n * submitted back by the client, they sometimes are like in the case of client-side background\n * auto-refresh. If submitted, the backend will treat any request with a placeholder cursor like no\n * cursor was submitted, which will behave the same way as an initial page load.\n */\ncase class UrtPlaceholderCursor() extends UrtPipelineCursor {\n  // This value is unused, in that it is not serialized into the final cursor value\n  override def initialSortIndex: Long = throw new UnsupportedOperationException(\n    \"initialSortIndex is not defined for placeholder cursors\")\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/feature/flexible_injection_pipeline/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"strato/config/columns/ml/featureStore:featureStore-strato-client\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"strato/config/columns/ml/featureStore:featureStore-strato-client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/slice/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/slice/SliceItemPresentation.scala",
    "content": "package com.twitter.product_mixer.component_library.model.presentation.slice\n\nimport com.twitter.product_mixer.core.model.common.presentation.slice.BaseSliceItemPresentation\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.SliceItem\n\ncase class SliceItemPresentation(override val sliceItem: SliceItem)\n    extends BaseSliceItemPresentation\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/urt/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/urt\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/urt\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/urt/ConversationModuleItem.scala",
    "content": "package com.twitter.product_mixer.component_library.model.presentation.urt\n\nimport com.twitter.product_mixer.core.model.common.presentation.urt.BaseUrtItemPresentation\nimport com.twitter.product_mixer.core.model.common.presentation.urt.IsDispensable\nimport com.twitter.product_mixer.core.model.common.presentation.urt.WithItemTreeDisplay\n\ntrait ConversationModuleItem\n    extends BaseUrtItemPresentation\n    with IsDispensable\n    with WithItemTreeDisplay\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/urt/UrtItemPresentation.scala",
    "content": "package com.twitter.product_mixer.component_library.model.presentation.urt\n\nimport com.twitter.product_mixer.core.model.common.presentation.urt.BaseUrtItemPresentation\nimport com.twitter.product_mixer.core.model.common.presentation.urt.BaseUrtModulePresentation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\ncase class UrtItemPresentation(\n  override val timelineItem: TimelineItem,\n  override val modulePresentation: Option[BaseUrtModulePresentation] = None)\n    extends BaseUrtItemPresentation\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/urt/UrtModulePresentation.scala",
    "content": "package com.twitter.product_mixer.component_library.model.presentation.urt\n\nimport com.twitter.product_mixer.core.model.common.presentation.urt.BaseUrtModulePresentation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\n\nfinal case class UrtModulePresentation(\n  override val timelineModule: TimelineModule)\n    extends BaseUrtModulePresentation\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/presentation/urt/UrtOperationPresentation.scala",
    "content": "package com.twitter.product_mixer.component_library.model.presentation.urt\n\nimport com.twitter.product_mixer.core.model.common.presentation.urt.BaseUrtOperationPresentation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineOperation\n\nfinal case class UrtOperationPresentation(\n  override val timelineOperation: TimelineOperation)\n    extends BaseUrtOperationPresentation\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads/AdsQuery.scala",
    "content": "package com.twitter.product_mixer.component_library.model.query.ads\n\nimport com.twitter.adserver.{thriftscala => ads}\nimport com.twitter.dspbidder.commons.{thriftscala => dsp}\n\n/**\n * AdsQuery holds request-time fields required by our ads candidate pipelines\n */\ntrait AdsQuery {\n\n  /**\n   * Timelines-specific context.\n   *\n   * @note used in Home Timelines\n   */\n  def timelineRequestParams: Option[ads.TimelineRequestParams] = None\n\n  /**\n   * Navigation action trigger-type\n   *\n   * @note used in Home Timelines\n   */\n  def requestTriggerType: Option[ads.RequestTriggerType] = None\n\n  /**\n   * Autoplay setting\n   *\n   * @note used in Home Timelines\n   */\n  def autoplayEnabled: Option[Boolean] = None\n\n  /**\n   * Disable NSFW avoidance for ads mixing\n   *\n   * @note used in Home Timelines\n   */\n  def disableNsfwAvoidance: Option[Boolean] = None\n\n  /**\n   * DSP context for adwords\n   *\n   * @note used in Home Timelines\n   */\n  def dspClientContext: Option[dsp.DspClientContext] = None\n\n  /**\n   * User ID for the User Profile being viewed.\n   *\n   * @note used in Profile Timelines\n   */\n  def userProfileViewedUserId: Option[Long] = None\n\n  /**\n   * Search-specific context.\n   *\n   * @note used in Search Timelines\n   */\n  def searchRequestContext: Option[ads.SearchRequestContext] = None\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"dspbidder/thrift/src/main/thrift/com/twitter/dspbidder/commons:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_rpc-scala\",\n    ],\n    exports = [\n        \"dspbidder/thrift/src/main/thrift/com/twitter/dspbidder/commons:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_rpc-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/AccountRecommendationsMixerModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.annotations.Flags\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.account_recommendations_mixer.thriftscala.AccountRecommendationsMixer\nimport com.twitter.util.Duration\n\n/**\n * Implementation with reasonable defaults for an idempotent Account Recommendations Mixer Thrift client.\n *\n * Note that the per request and total timeouts configured in this module are meant to represent a\n * reasonable starting point only. These were selected based on common practice, and should not be\n * assumed to be optimal for any particular use case. If you are interested in further tuning the\n * settings in this module, it is recommended to create local copy for your service.\n */\nobject AccountRecommendationsMixerModule\n    extends ThriftMethodBuilderClientModule[\n      AccountRecommendationsMixer.ServicePerEndpoint,\n      AccountRecommendationsMixer.MethodPerEndpoint\n    ]\n    with MtlsClient {\n  final val AccountRecommendationsMixerTimeoutPerRequest =\n    \"account_recommendations_mixer.timeout_per_request\"\n  final val AccountRecommendationsMixerTimeoutTotal = \"account_recommendations_mixer.timeout_total\"\n\n  flag[Duration](\n    name = AccountRecommendationsMixerTimeoutPerRequest,\n    default = 800.milliseconds,\n    help = \"Timeout per request for AccountRecommendationsMixer\")\n\n  flag[Duration](\n    name = AccountRecommendationsMixerTimeoutTotal,\n    default = 1200.milliseconds,\n    help = \"Timeout total for AccountRecommendationsMixer\")\n\n  override val label: String = \"account-recs-mixer\"\n\n  override val dest: String = \"/s/account-recs-mixer/account-recs-mixer:thrift\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    val timeOutPerRequest: Duration = injector\n      .instance[Duration](Flags.named(AccountRecommendationsMixerTimeoutPerRequest))\n    val timeOutTotal: Duration =\n      injector.instance[Duration](Flags.named(AccountRecommendationsMixerTimeoutTotal))\n    methodBuilder\n      .withTimeoutPerRequest(timeOutPerRequest)\n      .withTimeoutTotal(timeOutTotal)\n      .idempotent(5.percent)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"account-recommendations-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"content-recommender/thrift/src/main/thrift:thrift-scala\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"decider/src/main/scala\",\n        \"explore/explore-ranker/thrift/src/main/thrift:thrift-scala\",\n        \"finagle/finagle-memcached/src/main/scala\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"finatra/utils/src/main/java/com/twitter/finatra/annotations\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"home-scorer/thrift/src/main/thrift:thrift-scala\",\n        \"interests_discovery/thrift/src/main/thrift:service-thrift-scala\",\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"people-discovery/api/thrift/src/main/thrift:thrift-scala\",\n        \"src/scala/com/twitter/summingbird_internal/runner/store_config\",\n        \"src/thrift/com/twitter/gizmoduck:thrift-scala\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"src/thrift/com/twitter/timelinemixer:thrift-scala\",\n        \"src/thrift/com/twitter/timelineranker:thrift-scala\",\n        \"src/thrift/com/twitter/timelines/impression_store:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer:thrift-scala\",\n        \"src/thrift/com/twitter/timelineservice:thrift-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"stitch/stitch-gizmoduck\",\n        \"stitch/stitch-socialgraph\",\n        \"stitch/stitch-timelineservice/src/main/scala\",\n        \"stitch/stitch-tweetypie\",\n        \"timelines/src/main/scala/com/twitter/timelines/impressionstore/store\",\n        \"tweetconvosvc/client/src/main/scala/com/twitter/tweetconvosvc/client/builder\",\n        \"user_session_store/src/main/scala/com/twitter/user_session_store/config\",\n        \"user_session_store/src/main/scala/com/twitter/user_session_store/impl/manhattan/readonly\",\n        \"user_session_store/src/main/scala/com/twitter/user_session_store/impl/manhattan/readwrite\",\n    ],\n    exports = [\n        \"3rdparty/src/jvm/com/twitter/storehaus:core\",\n        \"account-recommendations-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"content-recommender/thrift/src/main/thrift:thrift-scala\",\n        \"finagle/finagle-memcached/src/main/scala\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"finatra/utils/src/main/java/com/twitter/finatra/annotations\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"interests_discovery/thrift/src/main/thrift:service-thrift-scala\",\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"src/scala/com/twitter/summingbird_internal/runner/store_config\",\n        \"src/thrift/com/twitter/gizmoduck:thrift-scala\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"src/thrift/com/twitter/timelineranker:thrift-scala\",\n        \"src/thrift/com/twitter/timelines/impression_store:thrift-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"stitch/stitch-gizmoduck\",\n        \"stitch/stitch-socialgraph\",\n        \"stitch/stitch-tweetypie\",\n        \"timelines/src/main/scala/com/twitter/timelines/impressionstore/store\",\n        \"user_session_store/src/main/scala/com/twitter/user_session_store/config\",\n        \"user_session_store/src/main/scala/com/twitter/user_session_store/impl/manhattan/readonly\",\n        \"user_session_store/src/main/scala/com/twitter/user_session_store/impl/manhattan/readwrite\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/ConversationServiceModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.tweetconvosvc.thriftscala.ConversationService\nimport com.twitter.util.Duration\nimport org.apache.thrift.protocol.TCompactProtocol\n\nobject ConversationServiceModule\n    extends ThriftMethodBuilderClientModule[\n      ConversationService.ServicePerEndpoint,\n      ConversationService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label: String = \"tweetconvosvc\"\n  override val dest: String = \"/s/tweetconvosvc/tweetconvosvc\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder =\n    methodBuilder\n      .withTimeoutTotal(200.milliseconds)\n      .withTimeoutPerRequest(100.milliseconds)\n      .idempotent(1.percent)\n\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client =\n    super\n      .configureThriftMuxClient(injector, client)\n      .withProtocolFactory(new TCompactProtocol.Factory())\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/CrMixerClientModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.cr_mixer.{thriftscala => t}\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.util.Duration\n\nobject CrMixerClientModule\n    extends ThriftMethodBuilderClientModule[\n      t.CrMixer.ServicePerEndpoint,\n      t.CrMixer.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"cr-mixer\"\n  override val dest = \"/s/cr-mixer/cr-mixer\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(500.millis)\n      .withTimeoutTotal(750.millis)\n      .idempotent(1.percent)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/DarkTrafficFilterModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.decider.Decider\nimport com.twitter.decider.RandomRecipient\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.thrift.service.Filterable\nimport com.twitter.finagle.thrift.service.ReqRepServicePerEndpointBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.annotations.Flags\nimport com.twitter.inject.thrift.modules.ReqRepDarkTrafficFilterModule\nimport scala.reflect.ClassTag\n\nclass DarkTrafficFilterModule[MethodIface <: Filterable[MethodIface]: ClassTag](\n  implicit serviceBuilder: ReqRepServicePerEndpointBuilder[MethodIface])\n    extends ReqRepDarkTrafficFilterModule\n    with MtlsClient {\n\n  override protected def enableSampling(injector: Injector): Any => Boolean = _ => {\n    val decider = injector.instance[Decider]\n    val deciderKey =\n      injector.instance[String](Flags.named(\"thrift.dark.traffic.filter.decider_key\"))\n    val fromProxy = ClientId.current\n      .map(_.name).exists(name => name.contains(\"diffy\") || name.contains(\"darktraffic\"))\n    !fromProxy && decider.isAvailable(deciderKey, recipient = Some(RandomRecipient))\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/EarlybirdModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.annotations.Flags\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.search.earlybird.{thriftscala => t}\nimport com.twitter.util.Duration\nimport org.apache.thrift.protocol.TCompactProtocol\n\nobject EarlybirdModule\n    extends ThriftMethodBuilderClientModule[\n      t.EarlybirdService.ServicePerEndpoint,\n      t.EarlybirdService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n  final val EarlybirdTimeoutPerRequest = \"earlybird.timeout_per_request\"\n  final val EarlybirdTimeoutTotal = \"earlybird.timeout_total\"\n\n  flag[Duration](\n    name = EarlybirdTimeoutPerRequest,\n    default = 200.milliseconds,\n    help = \"Timeout per request for Earlybird\")\n\n  flag[Duration](\n    name = EarlybirdTimeoutTotal,\n    default = 400.milliseconds,\n    help = \"Timeout total for Earlybird\")\n\n  override val dest = \"/s/earlybird-root-superroot/root-superroot\"\n  override val label = \"earlybird\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    val timeOutPerRequest: Duration = injector\n      .instance[Duration](Flags.named(EarlybirdTimeoutPerRequest))\n    val timeOutTotal: Duration = injector.instance[Duration](Flags.named(EarlybirdTimeoutTotal))\n    methodBuilder\n    // See TL-14313 for load testing details that led to 200ms being selected as request timeout\n      .withTimeoutPerRequest(timeOutPerRequest)\n      .withTimeoutTotal(timeOutTotal)\n      .idempotent(5.percent)\n  }\n\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client =\n    super\n      .configureThriftMuxClient(injector, client)\n      .withProtocolFactory(new TCompactProtocol.Factory())\n\n  override protected def sessionAcquisitionTimeout: Duration = 1.seconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/ExploreRankerClientModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.explore_ranker.thriftscala.ExploreRanker\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.annotations.Flags\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.util.Duration\n\nobject ExploreRankerClientModule\n    extends ThriftMethodBuilderClientModule[\n      ExploreRanker.ServicePerEndpoint,\n      ExploreRanker.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label: String = \"explore-ranker\"\n  override val dest: String = \"/s/explore-ranker/explore-ranker\"\n\n  private final val ExploreRankerTimeoutTotal = \"explore_ranker.timeout_total\"\n\n  flag[Duration](\n    name = ExploreRankerTimeoutTotal,\n    default = 800.milliseconds,\n    help = \"Timeout total for ExploreRanker\")\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    val timeoutTotal: Duration = injector.instance[Duration](Flags.named(ExploreRankerTimeoutTotal))\n    methodBuilder\n      .withTimeoutTotal(timeoutTotal)\n      .nonIdempotent\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/FollowRecommenderServiceModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.follow_recommendations.thriftscala.FollowRecommendationsThriftService\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.util.Duration\n\nobject FollowRecommenderServiceModule\n    extends ThriftMethodBuilderClientModule[\n      FollowRecommendationsThriftService.ServicePerEndpoint,\n      FollowRecommendationsThriftService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label: String = \"follow-recommendations-service\"\n\n  override val dest: String = \"/s/follow-recommendations/follow-recos-service\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(400.millis)\n      .withTimeoutTotal(800.millis)\n      .idempotent(5.percent)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/GizmoduckClientModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.gizmoduck.thriftscala.UserService\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.stitch.gizmoduck.Gizmoduck\nimport com.twitter.util.Duration\nimport javax.inject.Singleton\n\n/**\n * Implementation with reasonable defaults for an idempotent Gizmoduck Thrift and Stitch client.\n *\n * Note that the per request and total timeouts configured in this module are meant to represent a\n * reasonable starting point only. These were selected based on common practice, and should not be\n * assumed to be optimal for any particular use case. If you are interested in further tuning the\n * settings in this module, it is recommended to create local copy for your service.\n */\nobject GizmoduckClientModule\n    extends ThriftMethodBuilderClientModule[\n      UserService.ServicePerEndpoint,\n      UserService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n  override val label: String = \"gizmoduck\"\n  override val dest: String = \"/s/gizmoduck/gizmoduck\"\n\n  @Singleton\n  @Provides\n  def provideGizmoduckStitchClient(userService: UserService.MethodPerEndpoint): Gizmoduck =\n    new Gizmoduck(userService)\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder =\n    methodBuilder\n      .withTimeoutPerRequest(200.milliseconds)\n      .withTimeoutTotal(400.milliseconds)\n      .idempotent(1.percent)\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/HomeScorerClientModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.home_scorer.{thriftscala => t}\nimport com.twitter.util.Duration\n\nobject HomeScorerClientModule\n    extends ThriftMethodBuilderClientModule[\n      t.HomeScorer.ServicePerEndpoint,\n      t.HomeScorer.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"home-scorer\"\n  override val dest = \"/s/home-scorer/home-scorer\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(1200.millis)\n      .withTimeoutTotal(2400.millis)\n      .idempotent(1.percent)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/InterestsDiscoveryServiceModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.interests_discovery.thriftscala.InterestsDiscoveryService\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.util.Duration\n\nobject InterestsDiscoveryServiceModule\n    extends ThriftMethodBuilderClientModule[\n      InterestsDiscoveryService.ServicePerEndpoint,\n      InterestsDiscoveryService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label: String = \"interests-discovery-service\"\n\n  override val dest: String = \"/s/interests_discovery/interests_discovery\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(500.millis)\n      .withTimeoutTotal(1000.millis)\n      .idempotent(5.percent)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/OnboardingTaskServiceModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.onboarding.task.service.thriftscala.TaskService\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.util.Duration\nimport com.twitter.conversions.DurationOps._\n\nobject OnboardingTaskServiceModule\n    extends ThriftMethodBuilderClientModule[\n      TaskService.ServicePerEndpoint,\n      TaskService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n  override val label: String = \"onboarding-task-service\"\n  override val dest: String = \"/s/onboarding-task-service/onboarding-task-service\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(500.millis)\n      .withTimeoutTotal(1000.millis)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/PeopleDiscoveryServiceModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.peoplediscovery.api.thriftscala.ThriftPeopleDiscoveryService\nimport com.twitter.util.Duration\n\n/**\n * Implementation with reasonable defaults for an idempotent People Discovery Thrift client.\n *\n * Note that the per request and total timeouts configured in this module are meant to represent a\n * reasonable starting point only. These were selected based on common practice, and should not be\n * assumed to be optimal for any particular use case. If you are interested in further tuning the\n * settings in this module, it is recommended to create local copy for your service.\n */\nobject PeopleDiscoveryServiceModule\n    extends ThriftMethodBuilderClientModule[\n      ThriftPeopleDiscoveryService.ServicePerEndpoint,\n      ThriftPeopleDiscoveryService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label: String = \"people-discovery-api\"\n\n  override val dest: String = \"/s/people-discovery-api/people-discovery-api:thrift\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(800.millis)\n      .withTimeoutTotal(1200.millis)\n      .idempotent(5.percent)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/SocialGraphServiceModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.socialgraph.thriftscala.SocialGraphService\nimport com.twitter.stitch.socialgraph.SocialGraph\nimport javax.inject.Singleton\n\nobject SocialGraphServiceModule\n    extends ThriftMethodBuilderClientModule[\n      SocialGraphService.ServicePerEndpoint,\n      SocialGraphService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  val label: String = \"socialgraphservice\"\n  val dest: String = \"/s/socialgraph/socialgraph\"\n\n  @Singleton\n  @Provides\n  def provideGizmoduckStitchClient(\n    socialGraphService: SocialGraphService.MethodPerEndpoint\n  ): SocialGraph =\n    new SocialGraph(socialGraphService)\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder.withTimeoutPerRequest(400.millis)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/TimelineMixerClientModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.timelinemixer.{thriftscala => t}\nimport com.twitter.util.Duration\n\nobject TimelineMixerClientModule\n    extends ThriftMethodBuilderClientModule[\n      t.TimelineMixer.ServicePerEndpoint,\n      t.TimelineMixer.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"timeline-mixer\"\n  override val dest = \"/s/timelinemixer/timelinemixer\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(2000.millis)\n      .withTimeoutTotal(4000.millis)\n      .idempotent(1.percent)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/TimelineRankerClientModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.timelineranker.{thriftscala => t}\nimport com.twitter.util.Duration\nimport org.apache.thrift.protocol.TCompactProtocol\n\nobject TimelineRankerClientModule\n    extends ThriftMethodBuilderClientModule[\n      t.TimelineRanker.ServicePerEndpoint,\n      t.TimelineRanker.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"timeline-ranker\"\n  override val dest = \"/s/timelineranker/timelineranker:compactthrift\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(750.millis)\n      .withTimeoutTotal(750.millis)\n  }\n\n  override def configureThriftMuxClient(\n    injector: Injector,\n    client: ThriftMux.Client\n  ): ThriftMux.Client = {\n    val serviceIdentifier = injector.instance[ServiceIdentifier]\n    super\n      .configureThriftMuxClient(injector, client)\n      .withProtocolFactory(new TCompactProtocol.Factory())\n      .withMutualTls(serviceIdentifier)\n      .withPerEndpointStats\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/TimelineScorerClientModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.timelinescorer.{thriftscala => t}\nimport com.twitter.util.Duration\n\nobject TimelineScorerClientModule\n    extends ThriftMethodBuilderClientModule[\n      t.TimelineScorer.ServicePerEndpoint,\n      t.TimelineScorer.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"timeline-scorer\"\n  override val dest = \"/s/timelinescorer/timelinescorer\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(2000.millis)\n      .withTimeoutTotal(4000.millis)\n      .idempotent(1.percent)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/TimelineServiceClientModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.stitch.timelineservice.TimelineService\nimport com.twitter.timelineservice.{thriftscala => t}\nimport com.twitter.util.Duration\nimport javax.inject.Singleton\n\nobject TimelineServiceClientModule\n    extends ThriftMethodBuilderClientModule[\n      t.TimelineService.ServicePerEndpoint,\n      t.TimelineService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n\n  override val label = \"timelineservice\"\n  override val dest = \"/s/timelineservice/timelineservice\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutPerRequest(1200.millis)\n      .withTimeoutTotal(2400.millis)\n      .idempotent(1.percent)\n  }\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n\n  @Singleton\n  @Provides\n  def providesTimelineServiceStitchClient(\n    client: t.TimelineService.MethodPerEndpoint\n  ): TimelineService = {\n    new TimelineService(client)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/TweetImpressionStoreModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.Memcached\nimport com.twitter.finagle.Resolver\nimport com.twitter.finagle.memcached.protocol.Command\nimport com.twitter.finagle.memcached.protocol.Response\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient._\nimport com.twitter.finagle.param.HighResTimer\nimport com.twitter.finagle.service.RetryExceptionsFilter\nimport com.twitter.finagle.service.RetryPolicy\nimport com.twitter.finagle.service.StatsFilter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.impressionstore.store.TweetImpressionsStore\nimport com.twitter.timelines.impressionstore.thriftscala.ImpressionList\nimport javax.inject.Singleton\n\nobject TweetImpressionStoreModule extends TwitterModule {\n  private val TweetImpressionMemcacheWilyPath = \"/s/cache/timelines_impressionstore:twemcaches\"\n  private val tweetImpressionLabel = \"timelinesTweetImpressions\"\n\n  @Provides\n  @Singleton\n  def provideTimelineTweetImpressionStore(\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): ReadableStore[Long, ImpressionList] = {\n    val scopedStatsReceiver = statsReceiver.scope(\"timelinesTweetImpressions\")\n\n    // the below values for configuring the Memcached client\n    // are set to be the same as Home timeline's read path to the impression store.\n    val acquisitionTimeoutMillis = 200.milliseconds\n    val requestTimeoutMillis = 300.milliseconds\n    val numTries = 2\n\n    val statsFilter = new StatsFilter[Command, Response](scopedStatsReceiver)\n    val retryFilter = new RetryExceptionsFilter[Command, Response](\n      retryPolicy = RetryPolicy.tries(\n        numTries,\n        RetryPolicy.TimeoutAndWriteExceptionsOnly\n          .orElse(RetryPolicy.ChannelClosedExceptionsOnly)\n      ),\n      timer = HighResTimer.Default,\n      statsReceiver = scopedStatsReceiver\n    )\n\n    val client = Memcached.client\n      .withMutualTls(serviceIdentifier)\n      .withSession\n      .acquisitionTimeout(acquisitionTimeoutMillis)\n      .withRequestTimeout(requestTimeoutMillis)\n      .withStatsReceiver(scopedStatsReceiver)\n      .filtered(statsFilter.andThen(retryFilter))\n      .newRichClient(\n        dest = Resolver.eval(TweetImpressionMemcacheWilyPath),\n        label = tweetImpressionLabel\n      )\n\n    TweetImpressionsStore.tweetImpressionsStore(client)\n  }\n\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/TweetyPieClientModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.PercentOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.stitch.tweetypie.TweetyPie\nimport com.twitter.tweetypie.thriftscala.TweetService\nimport com.twitter.util.Duration\nimport javax.inject.Singleton\n\n/**\n * Implementation with reasonable defaults for an idempotent TweetyPie Thrift and Stitch client.\n *\n * Note that the per request and total timeouts are meant to represent a reasonable starting point\n * only. These were selected based on common practice, and should not be assumed to be optimal for\n * any particular use case. If you are interested in further tuning the settings in this module,\n * it is recommended to create local copy for your service.\n */\nobject TweetyPieClientModule\n    extends ThriftMethodBuilderClientModule[\n      TweetService.ServicePerEndpoint,\n      TweetService.MethodPerEndpoint\n    ]\n    with MtlsClient {\n  override val label: String = \"tweetypie\"\n  override val dest: String = \"/s/tweetypie/tweetypie\"\n\n  @Singleton\n  @Provides\n  def providesTweetypieStitchClient(tweetService: TweetService.MethodPerEndpoint): TweetyPie =\n    new TweetyPie(tweetService)\n\n  /**\n   * TweetyPie client id must be in the form of {service.env} or it will not be treated as an\n   * unauthorized client\n   */\n  override protected def clientId(injector: Injector): ClientId = {\n    val serviceIdentifier = injector.instance[ServiceIdentifier]\n    ClientId(s\"${serviceIdentifier.service}.${serviceIdentifier.environment}\")\n  }\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder =\n    methodBuilder\n      .withTimeoutPerRequest(200.milliseconds)\n      .withTimeoutTotal(400.milliseconds)\n      .idempotent(1.percent)\n\n  override protected def sessionAcquisitionTimeout: Duration = 500.milliseconds\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/UserSessionStoreModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.user_session_store.ReadOnlyUserSessionStore\nimport com.twitter.user_session_store.ReadWriteUserSessionStore\nimport com.twitter.user_session_store.UserSessionDataset\nimport com.twitter.user_session_store.UserSessionDataset.UserSessionDataset\nimport com.twitter.user_session_store.config.manhattan.UserSessionStoreManhattanConfig\nimport com.twitter.user_session_store.impl.manhattan.readonly.ReadOnlyManhattanUserSessionStoreBuilder\nimport com.twitter.user_session_store.impl.manhattan.readwrite.ReadWriteManhattanUserSessionStoreBuilder\n\nimport javax.inject.Singleton\n\nobject UserSessionStoreModule extends TwitterModule {\n  private val ReadWriteAppId = \"timelineservice_user_session_store\"\n  private val ReadWriteStagingDataset = \"tls_user_session_store_nonprod\"\n  private val ReadWriteProdDataset = \"tls_user_session_store\"\n  private val ReadOnlyAppId = \"user_session_store\"\n  private val ReadOnlyDataset = \"user_session_fields\"\n\n  @Provides\n  @Singleton\n  def providesReadWriteUserSessionStore(\n    injectedServiceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): ReadWriteUserSessionStore = {\n    val scopedStatsReceiver = statsReceiver.scope(injectedServiceIdentifier.service)\n\n    val dataset = injectedServiceIdentifier.environment.toLowerCase match {\n      case \"prod\" => ReadWriteProdDataset\n      case _ => ReadWriteStagingDataset\n    }\n\n    val clientReadWriteConfig = new UserSessionStoreManhattanConfig.Prod.ReadWrite.Omega {\n      override val appId = ReadWriteAppId\n      override val defaultMaxTimeout = 400.milliseconds\n      override val maxRetryCount = 1\n      override val serviceIdentifier = injectedServiceIdentifier\n      override val datasetNamesById = Map[UserSessionDataset, String](\n        UserSessionDataset.ActiveDaysInfo -> dataset,\n        UserSessionDataset.NonPollingTimes -> dataset\n      )\n    }\n\n    ReadWriteManhattanUserSessionStoreBuilder\n      .buildReadWriteUserSessionStore(clientReadWriteConfig, statsReceiver, scopedStatsReceiver)\n  }\n\n  @Provides\n  @Singleton\n  def providesReadOnlyUserSessionStore(\n    injectedServiceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): ReadOnlyUserSessionStore = {\n    val scopedStatsReceiver = statsReceiver.scope(injectedServiceIdentifier.service)\n\n    val clientReadOnlyConfig = new UserSessionStoreManhattanConfig.Prod.ReadOnly.Athena {\n      override val appId = ReadOnlyAppId\n      override val defaultMaxTimeout = 400.milliseconds\n      override val maxRetryCount = 1\n      override val serviceIdentifier = injectedServiceIdentifier\n      override val datasetNamesById = Map[UserSessionDataset, String](\n        UserSessionDataset.UserHealth -> ReadOnlyDataset\n      )\n    }\n\n    ReadOnlyManhattanUserSessionStoreBuilder\n      .buildReadOnlyUserSessionStore(clientReadOnlyConfig, statsReceiver, scopedStatsReceiver)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/cr_ml_ranker/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"cr-ml-ranker/thrift/src/main/thrift:thrift-scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/cr_ml_ranker\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/cr_ml_ranker/CrMlRankerModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module.cr_ml_ranker\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cr_ml_ranker.thriftscala.CrMLRanker\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule\nimport com.twitter.product_mixer.component_library.scorer.cr_ml_ranker.CrMlRankerScoreStitchClient\nimport com.twitter.util.Duration\nimport javax.inject.Singleton\n\ncase class CrMLRankerModule(totalTimeout: Duration = 100.milliseconds, batchSize: Int = 50)\n    extends ThriftMethodBuilderClientModule[\n      CrMLRanker.ServicePerEndpoint,\n      CrMLRanker.MethodPerEndpoint\n    ]\n    with MtlsClient {\n  override val label = \"cr-ml-ranker\"\n  override val dest = \"/s/cr-ml-ranker/cr-ml-ranker\"\n\n  override protected def configureMethodBuilder(\n    injector: Injector,\n    methodBuilder: MethodBuilder\n  ): MethodBuilder = {\n    methodBuilder\n      .withTimeoutTotal(totalTimeout)\n  }\n\n  @Provides\n  @Singleton\n  def providesStitchClient(\n    crMlRankerThriftClient: CrMLRanker.MethodPerEndpoint\n  ): CrMlRankerScoreStitchClient = new CrMlRankerScoreStitchClient(\n    crMlRankerThriftClient,\n    maxBatchSize = batchSize\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/http/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-http/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/http-client/src/main/scala\",\n        \"finatra/inject/inject-app/src/main/java/com/twitter/inject/annotations\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/product_mixer_flags\",\n        \"product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/http_client\",\n        \"util/util-core:scala\",\n        \"util/util-jackson/src/main/scala/com/twitter/util/jackson\",\n        \"util/util-security/src/main/scala/com/twitter/util/security\",\n    ],\n    exports = [\n        \"finagle/finagle-http/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/http-client/src/main/scala\",\n        \"finatra/inject/inject-app/src/main/java/com/twitter/inject/annotations\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/product_mixer_flags\",\n        \"product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/http_client\",\n        \"util/util-core:scala\",\n        \"util/util-jackson/src/main/scala/com/twitter/util/jackson\",\n        \"util/util-security/src/main/scala/com/twitter/util/security\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/http/FinagleHttpClientModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module.http\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.Http\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.shared_library.http_client.FinagleHttpClientBuilder.buildFinagleHttpClientMutualTls\nimport com.twitter.util.Duration\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject FinagleHttpClientModule extends TwitterModule {\n\n  final val HttpClientRequestTimeout = \"http_client.request_timeout\"\n  final val HttpClientConnectTimeout = \"http_client.connect_timeout\"\n  final val HttpClientAcquisitionTimeout = \"http_client.acquisition_timeout\"\n\n  flag[Duration](\n    name = HttpClientRequestTimeout,\n    default = 200.milliseconds,\n    help = \"HTTP client request timeout\")\n\n  flag[Duration](\n    name = HttpClientConnectTimeout,\n    default = 500.milliseconds,\n    help = \"HTTP client transport connect timeout\")\n\n  flag[Duration](\n    name = HttpClientAcquisitionTimeout,\n    default = 500.milliseconds,\n    help = \"HTTP client session acquisition timeout\")\n\n  final val FinagleHttpClientModule = \"FinagleHttpClientModule\"\n\n  /**\n   * Provides a Finagle HTTP client with S2S Auth / Mutual TLS\n   *\n   * Note that the timeouts configured in this module are meant to be a reasonable starting point\n   * only. To further tuning the settings, either override the flags or create local copy of the module.\n   *\n   * @param requestTimeout     HTTP client request timeout\n   * @param connectTimeout     HTTP client transport connect timeout\n   * @param acquisitionTimeout HTTP client session acquisition timeout\n   * @param serviceIdentifier  Service ID used to S2S Auth\n   * @param statsReceiver      Stats\n   *\n   * @return Finagle HTTP Client with S2S Auth / Mutual TLS\n   */\n  @Provides\n  @Singleton\n  @Named(FinagleHttpClientModule)\n  def providesFinagleHttpClient(\n    @Flag(HttpClientRequestTimeout) requestTimeout: Duration,\n    @Flag(HttpClientConnectTimeout) connectTimeout: Duration,\n    @Flag(HttpClientAcquisitionTimeout) acquisitionTimeout: Duration,\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Http.Client =\n    buildFinagleHttpClientMutualTls(\n      requestTimeout = requestTimeout,\n      connectTimeout = connectTimeout,\n      acquisitionTimeout = acquisitionTimeout,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/http/FinagleHttpClientWithCredentialProxyModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module.http\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.Http\nimport com.twitter.finagle.http.ProxyCredentials\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientAcquisitionTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientConnectTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientRequestTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientWithProxyModule.HttpClientWithProxyRemoteHost\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientWithProxyModule.HttpClientWithProxyRemotePort\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientWithProxyModule.HttpClientWithProxyTwitterHost\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientWithProxyModule.HttpClientWithProxyTwitterPort\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ServiceLocal\nimport com.twitter.product_mixer.shared_library.http_client.FinagleHttpClientWithProxyBuilder.buildFinagleHttpClientWithCredentialProxy\nimport com.twitter.product_mixer.shared_library.http_client.HttpHostPort\nimport com.twitter.util.Duration\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject FinagleHttpClientWithCredentialProxyModule extends TwitterModule {\n\n  final val FinagleHttpClientWithCredentialProxy = \"FinagleHttpClientWithCredentialProxy\"\n\n  /**\n   * Provide a Finagle HTTP client with Egress Proxy support using Credentials\n   *\n   * Note that the timeouts configured in this module are meant to be a reasonable starting point\n   * only. To further tuning the settings, either override the flags or create local copy of the module.\n   *\n   * @param proxyTwitterHost       Twitter egress proxy host\n   * @param proxyTwitterPort       Twitter egress proxy port\n   * @param proxyRemoteHost        Remote proxy host\n   * @param proxyRemotePort        Remote proxy port\n   * @param requestTimeout         HTTP client request timeout\n   * @param connectTimeout         HTTP client transport connect timeout\n   * @param acquisitionTimeout     HTTP client session acquisition timeout\n   * @param isServiceLocal         If this is a Local deployment for testing\n   * @param proxyCredentials       Proxy credentials\n   * @param statsReceiver          Stats\n   *\n   * @return Finagle HTTP client with Egress Proxy support using Credentials\n   */\n  @Provides\n  @Singleton\n  @Named(FinagleHttpClientWithCredentialProxy)\n  def providesFinagleHttpClientWithCredentialProxy(\n    @Flag(HttpClientWithProxyTwitterHost) proxyTwitterHost: String,\n    @Flag(HttpClientWithProxyTwitterPort) proxyTwitterPort: Int,\n    @Flag(HttpClientWithProxyRemoteHost) proxyRemoteHost: String,\n    @Flag(HttpClientWithProxyRemotePort) proxyRemotePort: Int,\n    @Flag(HttpClientRequestTimeout) requestTimeout: Duration,\n    @Flag(HttpClientConnectTimeout) connectTimeout: Duration,\n    @Flag(HttpClientAcquisitionTimeout) acquisitionTimeout: Duration,\n    @Flag(ServiceLocal) isServiceLocal: Boolean,\n    proxyCredentials: ProxyCredentials,\n    statsReceiver: StatsReceiver\n  ): Http.Client = {\n\n    val twitterProxyHostPort = HttpHostPort(proxyTwitterHost, proxyTwitterPort)\n    val remoteProxyHostPort = HttpHostPort(proxyRemoteHost, proxyRemotePort)\n\n    buildFinagleHttpClientWithCredentialProxy(\n      twitterProxyHostPort = twitterProxyHostPort,\n      remoteProxyHostPort = remoteProxyHostPort,\n      requestTimeout = requestTimeout,\n      connectTimeout = connectTimeout,\n      acquisitionTimeout = acquisitionTimeout,\n      proxyCredentials = proxyCredentials,\n      statsReceiver = statsReceiver\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/http/FinagleHttpClientWithProxyModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module.http\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.Http\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientAcquisitionTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientConnectTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientRequestTimeout\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ServiceLocal\nimport com.twitter.product_mixer.shared_library.http_client.FinagleHttpClientWithProxyBuilder.buildFinagleHttpClientWithProxy\nimport com.twitter.product_mixer.shared_library.http_client.HttpHostPort\nimport com.twitter.util.Duration\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject FinagleHttpClientWithProxyModule extends TwitterModule {\n  final val HttpClientWithProxyTwitterHost = \"http_client.proxy.twitter_host\"\n  final val HttpClientWithProxyTwitterPort = \"http_client.proxy.twitter_port\"\n  final val HttpClientWithProxyRemoteHost = \"http_client.proxy.remote_host\"\n  final val HttpClientWithProxyRemotePort = \"http_client.proxy.remote_port\"\n\n  flag[String](\n    HttpClientWithProxyTwitterHost,\n    \"httpproxy.local.twitter.com\",\n    \"Twitter egress proxy host\")\n\n  flag[Int](HttpClientWithProxyTwitterPort, 3128, \"Twitter egress proxy port\")\n\n  flag[String](HttpClientWithProxyRemoteHost, \"Host that the proxy will connect to\")\n\n  flag[Int](HttpClientWithProxyRemotePort, 443, \"Port that the proxy will connect to\")\n\n  final val FinagleHttpClientWithProxy = \"FinagleHttpClientWithProxy\"\n\n  /**\n   * Provide a Finagle HTTP client with Egress Proxy support\n   *\n   * Note that the timeouts configured in this module are meant to be a reasonable starting point\n   * only. To further tuning the settings, either override the flags or create local copy of the module.\n   *\n   * @param proxyTwitterHost       Twitter egress proxy host\n   * @param proxyTwitterPort       Twitter egress proxy port\n   * @param proxyRemoteHost        Remote proxy host\n   * @param proxyRemotePort        Remote proxy port\n   * @param requestTimeout         HTTP client request timeout\n   * @param connectTimeout         HTTP client transport connect timeout\n   * @param acquisitionTimeout     HTTP client session acquisition timeout\n   * @param isServiceLocal         If this is a Local deployment for testing\n   * @param statsReceiver          Stats\n   *\n   * @return Finagle HTTP client with Egress Proxy support\n   */\n  @Provides\n  @Singleton\n  @Named(FinagleHttpClientWithProxy)\n  def providesFinagleHttpClientWithProxy(\n    @Flag(HttpClientWithProxyTwitterHost) proxyTwitterHost: String,\n    @Flag(HttpClientWithProxyTwitterPort) proxyTwitterPort: Int,\n    @Flag(HttpClientWithProxyRemoteHost) proxyRemoteHost: String,\n    @Flag(HttpClientWithProxyRemotePort) proxyRemotePort: Int,\n    @Flag(HttpClientRequestTimeout) requestTimeout: Duration,\n    @Flag(HttpClientConnectTimeout) connectTimeout: Duration,\n    @Flag(HttpClientAcquisitionTimeout) acquisitionTimeout: Duration,\n    @Flag(ServiceLocal) isServiceLocal: Boolean,\n    statsReceiver: StatsReceiver\n  ): Http.Client = {\n    val twitterProxyHostPort = HttpHostPort(proxyTwitterHost, proxyTwitterPort)\n    val remoteProxyHostPort = HttpHostPort(proxyRemoteHost, proxyRemotePort)\n\n    buildFinagleHttpClientWithProxy(\n      twitterProxyHostPort = twitterProxyHostPort,\n      remoteProxyHostPort = remoteProxyHostPort,\n      requestTimeout = requestTimeout,\n      connectTimeout = connectTimeout,\n      acquisitionTimeout = acquisitionTimeout,\n      statsReceiver = statsReceiver\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/http/FinatraHttpClientModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module.http\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.httpclient.HttpClient\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientAcquisitionTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientConnectTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientRequestTimeout\nimport com.twitter.product_mixer.shared_library.http_client.FinagleHttpClientBuilder.buildFinagleHttpClientMutualTls\nimport com.twitter.product_mixer.shared_library.http_client.HttpHostPort\nimport com.twitter.util.Duration\nimport com.twitter.util.jackson.ScalaObjectMapper\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject FinatraHttpClientModule extends TwitterModule {\n\n  final val HttpClientHost = \"http_client.host\"\n  final val HttpClientPort = \"http_client.port\"\n\n  flag[String](HttpClientHost, \"Host that the client will connect to\")\n\n  flag[Int](HttpClientPort, 443, \"Port that the client will connect to\")\n\n  final val FinatraHttpClient = \"FinatraHttpClient\"\n\n  /**\n   * Build a Finatra HTTP client for a host. The Finatra HTTP client can be helpful (as opposed to\n   * the base Finagle HTTP Client), as it provides built-in JSON response parsing and other\n   * convenience methods\n   *\n   * Note that the timeouts configured in this module are meant to be a reasonable starting point\n   * only. To further tuning the settings, either override the flags or create local copy of the module.\n   *\n   * @param requestTimeout     HTTP client request timeout\n   * @param connectTimeout     HTTP client transport connect timeout\n   * @param acquisitionTimeout HTTP client session acquisition timeout\n   * @param host               Host to build Finatra client\n   * @param port               Port to build Finatra client\n   * @param scalaObjectMapper  Object mapper used by the built-in JSON response parsing\n   * @param serviceIdentifier  Service ID used to S2S Auth\n   * @param statsReceiver      Stats\n   *\n   * @return Finatra HTTP client\n   */\n  @Provides\n  @Singleton\n  @Named(FinatraHttpClient)\n  def providesFinatraHttpClient(\n    @Flag(HttpClientRequestTimeout) requestTimeout: Duration,\n    @Flag(HttpClientConnectTimeout) connectTimeout: Duration,\n    @Flag(HttpClientAcquisitionTimeout) acquisitionTimeout: Duration,\n    @Flag(HttpClientHost) host: String,\n    @Flag(HttpClientPort) port: Int,\n    scalaObjectMapper: ScalaObjectMapper,\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): HttpClient = {\n    val finagleHttpClient = buildFinagleHttpClientMutualTls(\n      requestTimeout = requestTimeout,\n      connectTimeout = connectTimeout,\n      acquisitionTimeout = acquisitionTimeout,\n      serviceIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver\n    )\n\n    val hostPort = HttpHostPort(host, port)\n    val finagleHttpService = finagleHttpClient.newService(hostPort.toString)\n\n    new HttpClient(\n      hostname = hostPort.host,\n      httpService = finagleHttpService,\n      mapper = scalaObjectMapper\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/http/FinatraHttpClientWithCredentialProxyModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module.http\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.http.ProxyCredentials\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.httpclient.HttpClient\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientAcquisitionTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientConnectTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientRequestTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientWithProxyModule.HttpClientWithProxyRemoteHost\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientWithProxyModule.HttpClientWithProxyRemotePort\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientWithProxyModule.HttpClientWithProxyTwitterHost\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientWithProxyModule.HttpClientWithProxyTwitterPort\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ServiceLocal\nimport com.twitter.product_mixer.shared_library.http_client.FinagleHttpClientWithProxyBuilder.buildFinagleHttpClientWithCredentialProxy\nimport com.twitter.product_mixer.shared_library.http_client.FinagleHttpClientWithProxyBuilder.buildFinagleHttpServiceWithProxy\nimport com.twitter.product_mixer.shared_library.http_client.HttpHostPort\nimport com.twitter.util.Duration\nimport com.twitter.util.jackson.ScalaObjectMapper\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject FinatraHttpClientWithCredentialProxyModule extends TwitterModule {\n\n  final val FinatraHttpClientWithCredentialProxy = \"FinagleHttpClientWithCredentialProxy\"\n\n  /**\n   * Build a Finatra HTTP client with Egress Proxy support with Credentials for a host. The Finatra\n   * HTTP client can be helpful (as opposed to the base Finagle HTTP Client), as it provides\n   * built-in JSON response zparsing and other convenience methods\n   *\n   * Note that the timeouts configured in this module are meant to be a reasonable starting point\n   * only. To further tuning the settings, either override the flags or create local copy of the module.\n   *\n   * @param proxyTwitterHost       Twitter egress proxy host\n   * @param proxyTwitterPort       Twitter egress proxy port\n   * @param proxyRemoteHost        Remote proxy host\n   * @param proxyRemotePort        Remote proxy port\n   * @param requestTimeout         HTTP client request timeout\n   * @param connectTimeout         HTTP client transport connect timeout\n   * @param acquisitionTimeout     HTTP client session acquisition timeout\n   * @param isServiceLocal         Local deployment for testing\n   * @param proxyCredentials       Proxy credentials\n   * @param scalaObjectMapper      Object mapper used by the built-in JSON response parsing\n   * @param statsReceiver          Stats\n   *\n   * @return Finatra HTTP client with Egress Proxy support for a host\n   */\n  @Provides\n  @Singleton\n  @Named(FinatraHttpClientWithCredentialProxy)\n  def providesFinatraHttpClientWithCredentialProxy(\n    @Flag(HttpClientWithProxyTwitterHost) proxyTwitterHost: String,\n    @Flag(HttpClientWithProxyTwitterPort) proxyTwitterPort: Int,\n    @Flag(HttpClientWithProxyRemoteHost) proxyRemoteHost: String,\n    @Flag(HttpClientWithProxyRemotePort) proxyRemotePort: Int,\n    @Flag(HttpClientRequestTimeout) requestTimeout: Duration,\n    @Flag(HttpClientConnectTimeout) connectTimeout: Duration,\n    @Flag(HttpClientAcquisitionTimeout) acquisitionTimeout: Duration,\n    @Flag(ServiceLocal) isServiceLocal: Boolean,\n    proxyCredentials: ProxyCredentials,\n    scalaObjectMapper: ScalaObjectMapper,\n    statsReceiver: StatsReceiver\n  ): HttpClient = {\n    val twitterProxyHostPort = HttpHostPort(proxyTwitterHost, proxyTwitterPort)\n    val proxyRemoteHostPort = HttpHostPort(proxyRemoteHost, proxyRemotePort)\n\n    val finagleHttpClientWithCredentialProxy =\n      buildFinagleHttpClientWithCredentialProxy(\n        twitterProxyHostPort = twitterProxyHostPort,\n        remoteProxyHostPort = proxyRemoteHostPort,\n        requestTimeout = requestTimeout,\n        connectTimeout = connectTimeout,\n        acquisitionTimeout = acquisitionTimeout,\n        proxyCredentials = proxyCredentials,\n        statsReceiver = statsReceiver\n      )\n\n    val finagleHttpServiceWithCredentialProxy =\n      buildFinagleHttpServiceWithProxy(\n        finagleHttpClientWithProxy = finagleHttpClientWithCredentialProxy,\n        twitterProxyHostPort = twitterProxyHostPort\n      )\n\n    new HttpClient(\n      hostname = twitterProxyHostPort.host,\n      httpService = finagleHttpServiceWithCredentialProxy,\n      mapper = scalaObjectMapper\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/http/FinatraHttpClientWithProxyModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module.http\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.httpclient.HttpClient\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientAcquisitionTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientConnectTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.HttpClientRequestTimeout\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientWithProxyModule.HttpClientWithProxyRemoteHost\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientWithProxyModule.HttpClientWithProxyRemotePort\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientWithProxyModule.HttpClientWithProxyTwitterHost\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientWithProxyModule.HttpClientWithProxyTwitterPort\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ServiceLocal\nimport com.twitter.product_mixer.shared_library.http_client.FinagleHttpClientWithProxyBuilder.buildFinagleHttpClientWithProxy\nimport com.twitter.product_mixer.shared_library.http_client.FinagleHttpClientWithProxyBuilder.buildFinagleHttpServiceWithProxy\nimport com.twitter.product_mixer.shared_library.http_client.HttpHostPort\nimport com.twitter.util.Duration\nimport com.twitter.util.jackson.ScalaObjectMapper\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject FinatraHttpClientWithProxyModule extends TwitterModule {\n\n  final val FinatraHttpClientWithProxy = \"FinagleHttpClientWithProxy\"\n\n  /**\n   * Build a Finatra HTTP client with Egress Proxy support for a host. The Finatra HTTP client can\n   * be helpful (as opposed to the base Finagle HTTP Client), as it provides built-in JSON response\n   * parsing and other convenience methods\n   *\n   * Note that the timeouts configured in this module are meant to be a reasonable starting point\n   * only. To further tuning the settings, either override the flags or create local copy of the module.\n   *\n   * @param proxyTwitterHost       Twitter egress proxy host\n   * @param proxyTwitterPort       Twitter egress proxy port\n   * @param proxyRemoteHost        Remote proxy host\n   * @param proxyRemotePort        Remote proxy port\n   * @param requestTimeout         HTTP client request timeout\n   * @param connectTimeout         HTTP client transport connect timeout\n   * @param acquisitionTimeout     HTTP client session acquisition timeout\n   * @param isServiceLocal         Local deployment for testing\n   * @param scalaObjectMapper      Object mapper used by the built-in JSON response parsing\n   * @param statsReceiver          Stats\n   *\n   * @return Finatra HTTP client with Egress Proxy support for a host\n   */\n  @Provides\n  @Singleton\n  @Named(FinatraHttpClientWithProxy)\n  def providesFinatraHttpClientWithProxy(\n    @Flag(HttpClientWithProxyTwitterHost) proxyTwitterHost: String,\n    @Flag(HttpClientWithProxyTwitterPort) proxyTwitterPort: Int,\n    @Flag(HttpClientWithProxyRemoteHost) proxyRemoteHost: String,\n    @Flag(HttpClientWithProxyRemotePort) proxyRemotePort: Int,\n    @Flag(HttpClientRequestTimeout) requestTimeout: Duration,\n    @Flag(HttpClientConnectTimeout) connectTimeout: Duration,\n    @Flag(HttpClientAcquisitionTimeout) acquisitionTimeout: Duration,\n    @Flag(ServiceLocal) isServiceLocal: Boolean,\n    scalaObjectMapper: ScalaObjectMapper,\n    statsReceiver: StatsReceiver\n  ): HttpClient = {\n    val twitterProxyHostPort = HttpHostPort(proxyTwitterHost, proxyTwitterPort)\n    val proxyRemoteHostPort = HttpHostPort(proxyRemoteHost, proxyRemotePort)\n\n    val finagleHttpClientWithProxy =\n      buildFinagleHttpClientWithProxy(\n        twitterProxyHostPort = twitterProxyHostPort,\n        remoteProxyHostPort = proxyRemoteHostPort,\n        requestTimeout = requestTimeout,\n        connectTimeout = connectTimeout,\n        acquisitionTimeout = acquisitionTimeout,\n        statsReceiver = statsReceiver\n      )\n\n    val finagleHttpServiceWithProxy =\n      buildFinagleHttpServiceWithProxy(\n        finagleHttpClientWithProxy = finagleHttpClientWithProxy,\n        twitterProxyHostPort = twitterProxyHostPort\n      )\n\n    new HttpClient(\n      hostname = twitterProxyHostPort.host,\n      httpService = finagleHttpServiceWithProxy,\n      mapper = scalaObjectMapper\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/http/ProxyCredentialsModule.scala",
    "content": "package com.twitter.product_mixer.component_library.module.http\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.http.ProxyCredentials\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.util.security.{Credentials => CredentialsUtil}\nimport java.io.File\nimport javax.inject.Singleton\n\nobject ProxyCredentialsModule extends TwitterModule {\n  final val HttpClientWithProxyCredentialsPath = \"http_client.proxy.proxy_credentials_path\"\n\n  flag[String](HttpClientWithProxyCredentialsPath, \"\", \"Path the load the proxy credentials\")\n\n  @Provides\n  @Singleton\n  def providesProxyCredentials(\n    @Flag(HttpClientWithProxyCredentialsPath) proxyCredentialsPath: String,\n  ): ProxyCredentials = {\n    val credentialsFile = new File(proxyCredentialsPath)\n    ProxyCredentials(CredentialsUtil(credentialsFile))\n      .getOrElse(throw MissingProxyCredentialsException)\n  }\n\n  object MissingProxyCredentialsException extends Exception(\"Proxy Credentials not found\")\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/AdsCandidatePipelineConfig.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.ads\n\nimport com.twitter.adserver.{thriftscala => ads}\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.model.query.ads.AdsQuery\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.decider.DeciderParam\nimport javax.inject.Inject\n\nclass AdsCandidatePipelineConfig[Query <: PipelineQuery with AdsQuery] @Inject() (\n  override val identifier: CandidatePipelineIdentifier,\n  override val enabledDeciderParam: Option[DeciderParam[Boolean]],\n  override val supportedClientParam: Option[FSParam[Boolean]],\n  override val gates: Seq[Gate[Query]],\n  override val candidateSource: CandidateSource[\n    ads.AdRequestParams,\n    ads.AdImpression\n  ],\n  override val filters: Seq[Filter[Query, AdsCandidate]],\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[Query, AdsCandidate, _]\n  ],\n  override val decorator: Option[CandidateDecorator[Query, AdsCandidate]],\n  override val alerts: Seq[Alert],\n  adsDisplayLocationBuilder: AdsDisplayLocationBuilder[Query],\n  estimateNumOrganicItems: EstimateNumOrganicItems[Query],\n  urtRequest: Option[Boolean],\n) extends CandidatePipelineConfig[\n      Query,\n      ads.AdRequestParams,\n      ads.AdImpression,\n      AdsCandidate\n    ] {\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[Query, ads.AdRequestParams] =\n    AdsCandidatePipelineQueryTransformer(\n      adsDisplayLocationBuilder = adsDisplayLocationBuilder,\n      estimatedNumOrganicItems = estimateNumOrganicItems,\n      urtRequest = urtRequest)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    ads.AdImpression,\n    AdsCandidate\n  ] = AdsCandidatePipelineResultsTransformer\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/AdsCandidatePipelineConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.ads\n\nimport com.twitter.adserver.thriftscala.AdImpression\nimport com.twitter.adserver.thriftscala.AdRequestParams\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.ad.AdsCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.model.query.ads.AdsQuery\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.decider.DeciderParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AdsCandidatePipelineConfigBuilder @Inject() () {\n\n  def build[Query <: PipelineQuery with AdsQuery](\n    adsCandidateSource: CandidateSource[AdRequestParams, AdImpression],\n    adsDisplayLocationBuilder: AdsDisplayLocationBuilder[Query],\n    estimateNumOrganicItems: EstimateNumOrganicItems[Query],\n    identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\"Ads\"),\n    enabledDeciderParam: Option[DeciderParam[Boolean]] = None,\n    supportedClientParam: Option[FSParam[Boolean]] = None,\n    gates: Seq[Gate[Query]] = Seq.empty,\n    filters: Seq[Filter[Query, AdsCandidate]] = Seq.empty,\n    postFilterFeatureHydration: Seq[BaseCandidateFeatureHydrator[Query, AdsCandidate, _]] =\n      Seq.empty,\n    decorator: Option[CandidateDecorator[Query, AdsCandidate]] =\n      Some(UrtItemCandidateDecorator(AdsCandidateUrtItemBuilder())),\n    alerts: Seq[Alert] = Seq.empty,\n    urtRequest: Option[Boolean] = None,\n  ): AdsCandidatePipelineConfig[Query] = {\n    new AdsCandidatePipelineConfig(\n      identifier = identifier,\n      enabledDeciderParam = enabledDeciderParam,\n      supportedClientParam = supportedClientParam,\n      gates = gates,\n      candidateSource = adsCandidateSource,\n      filters = filters,\n      postFilterFeatureHydration = postFilterFeatureHydration,\n      decorator = decorator,\n      alerts = alerts,\n      adsDisplayLocationBuilder = adsDisplayLocationBuilder,\n      estimateNumOrganicItems = estimateNumOrganicItems,\n      urtRequest = urtRequest,\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/AdsCandidatePipelineQueryTransformer.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.ads\n\nimport com.twitter.adserver.{thriftscala => ads}\nimport com.twitter.product_mixer.component_library.model.query.ads.AdsQuery\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.AdsCandidatePipelineQueryTransformer.buildAdRequestParams\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Transform a PipelineQuery with AdsQuery into an AdsRequestParams\n *\n * @param adsDisplayLocationBuilder Builder that determines the display location for the ads\n * @param estimatedNumOrganicItems  Estimate for the number of organic items that will be served\n *                                  alongside inorganic items such as ads. \n */\ncase class AdsCandidatePipelineQueryTransformer[Query <: PipelineQuery with AdsQuery](\n  adsDisplayLocationBuilder: AdsDisplayLocationBuilder[Query],\n  estimatedNumOrganicItems: EstimateNumOrganicItems[Query],\n  urtRequest: Option[Boolean],\n) extends CandidatePipelineQueryTransformer[Query, ads.AdRequestParams] {\n\n  override def transform(query: Query): ads.AdRequestParams =\n    buildAdRequestParams(\n      query = query,\n      adsDisplayLocation = adsDisplayLocationBuilder(query),\n      organicItemIds = None,\n      numOrganicItems = Some(estimatedNumOrganicItems(query)),\n      urtRequest = urtRequest\n    )\n}\n\nobject AdsCandidatePipelineQueryTransformer {\n\n  def buildAdRequestParams(\n    query: PipelineQuery with AdsQuery,\n    adsDisplayLocation: ads.DisplayLocation,\n    organicItemIds: Option[Seq[Long]],\n    numOrganicItems: Option[Short],\n    urtRequest: Option[Boolean],\n  ): ads.AdRequestParams = {\n    val searchRequestContext = query.searchRequestContext\n    val queryString = query.searchRequestContext.flatMap(_.queryString)\n\n    val adRequest = ads.AdRequest(\n      queryString = queryString, \n      displayLocation = adsDisplayLocation,\n      searchRequestContext = searchRequestContext,\n      organicItemIds = organicItemIds,\n      numOrganicItems = numOrganicItems,\n      profileUserId = query.userProfileViewedUserId,\n      isDebug = Some(false),\n      isTest = Some(false),\n      requestTriggerType = query.requestTriggerType,\n      disableNsfwAvoidance = query.disableNsfwAvoidance,\n      timelineRequestParams = query.timelineRequestParams,\n    )\n\n    val context = query.clientContext\n\n    val clientInfo = ads.ClientInfo(\n      clientId = context.appId.map(_.toInt),\n      userId64 = context.userId,\n      userIp = context.ipAddress,\n      guestId = context.guestIdAds,\n      userAgent = context.userAgent,\n      deviceId = context.deviceId,\n      languageCode = context.languageCode,\n      countryCode = context.countryCode,\n      mobileDeviceId = context.mobileDeviceId,\n      mobileDeviceAdId = context.mobileDeviceAdId,\n      limitAdTracking = context.limitAdTracking,\n      autoplayEnabled = query.autoplayEnabled,\n      urtRequest = urtRequest,\n      dspClientContext = query.dspClientContext\n    )\n\n    ads.AdRequestParams(adRequest, clientInfo)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/AdsCandidatePipelineResultsTransformer.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.ads\n\nimport com.twitter.adserver.thriftscala.AdImpression\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsTweetCandidate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.UnexpectedCandidateResult\n\nobject AdsCandidatePipelineResultsTransformer\n    extends CandidatePipelineResultsTransformer[AdImpression, AdsCandidate] {\n\n  override def transform(sourceResult: AdImpression): AdsCandidate =\n    (sourceResult.nativeRtbCreative, sourceResult.promotedTweetId) match {\n      case (None, Some(promotedTweetId)) =>\n        AdsTweetCandidate(\n          id = promotedTweetId,\n          adImpression = sourceResult\n        )\n      case (Some(_), None) =>\n        throw unsupportedAdImpressionPipelineFailure(\n          impression = sourceResult,\n          reason = \"Received ad impression with rtbCreative\")\n      case (Some(_), Some(_)) =>\n        throw unsupportedAdImpressionPipelineFailure(\n          impression = sourceResult,\n          reason = \"Received ad impression with both rtbCreative and promoted tweetId\")\n      case (None, None) =>\n        throw unsupportedAdImpressionPipelineFailure(\n          impression = sourceResult,\n          reason = \"Received ad impression with neither rtbCreative nor promoted tweetId\")\n    }\n\n  private def unsupportedAdImpressionPipelineFailure(impression: AdImpression, reason: String) =\n    PipelineFailure(\n      UnexpectedCandidateResult,\n      reason =\n        s\"Unsupported AdImpression ($reason). impressionString: ${impression.impressionString}\")\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/AdsDependentCandidatePipelineConfig.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.ads\n\nimport com.twitter.adserver.{thriftscala => ads}\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.model.query.ads.AdsQuery\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.DependentCandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.decider.DeciderParam\n\nclass AdsDependentCandidatePipelineConfig[Query <: PipelineQuery with AdsQuery](\n  override val identifier: CandidatePipelineIdentifier,\n  override val enabledDeciderParam: Option[DeciderParam[Boolean]],\n  override val supportedClientParam: Option[FSParam[Boolean]],\n  override val gates: Seq[BaseGate[Query]],\n  override val candidateSource: CandidateSource[\n    ads.AdRequestParams,\n    ads.AdImpression\n  ],\n  override val filters: Seq[Filter[Query, AdsCandidate]],\n  override val postFilterFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[Query, AdsCandidate, _]\n  ],\n  override val decorator: Option[CandidateDecorator[Query, AdsCandidate]],\n  override val alerts: Seq[Alert],\n  adsDisplayLocationBuilder: AdsDisplayLocationBuilder[Query],\n  urtRequest: Option[Boolean],\n  getOrganicItemIds: GetOrganicItemIds,\n  countNumOrganicItems: CountNumOrganicItems[Query],\n) extends DependentCandidatePipelineConfig[\n      Query,\n      ads.AdRequestParams,\n      ads.AdImpression,\n      AdsCandidate\n    ] {\n\n  override def queryTransformer: DependentCandidatePipelineQueryTransformer[\n    Query,\n    ads.AdRequestParams\n  ] = AdsDependentCandidatePipelineQueryTransformer(\n    adsDisplayLocationBuilder = adsDisplayLocationBuilder,\n    getOrganicItemIds = getOrganicItemIds,\n    countNumOrganicItems = countNumOrganicItems,\n    urtRequest = urtRequest\n  )\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    ads.AdImpression,\n    AdsCandidate\n  ] = AdsCandidatePipelineResultsTransformer\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/AdsDependentCandidatePipelineConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.ads\n\nimport com.twitter.adserver.thriftscala.AdImpression\nimport com.twitter.adserver.thriftscala.AdRequestParams\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.ad.AdsCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.model.query.ads.AdsQuery\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.decider.DeciderParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AdsDependentCandidatePipelineConfigBuilder @Inject() () {\n\n  /**\n   * Build a AdsDependentCandidatePipelineConfig\n   */\n  def build[Query <: PipelineQuery with AdsQuery](\n    adsCandidateSource: CandidateSource[AdRequestParams, AdImpression],\n    identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\"Ads\"),\n    adsDisplayLocationBuilder: AdsDisplayLocationBuilder[Query],\n    getOrganicItemIds: GetOrganicItemIds = EmptyOrganicItemIds,\n    countNumOrganicItems: CountNumOrganicItems[Query] = CountAllCandidates,\n    enabledDeciderParam: Option[DeciderParam[Boolean]] = None,\n    supportedClientParam: Option[FSParam[Boolean]] = None,\n    gates: Seq[BaseGate[Query]] = Seq.empty,\n    filters: Seq[Filter[Query, AdsCandidate]] = Seq.empty,\n    postFilterFeatureHydration: Seq[BaseCandidateFeatureHydrator[Query, AdsCandidate, _]] =\n      Seq.empty,\n    decorator: Option[CandidateDecorator[Query, AdsCandidate]] =\n      Some(UrtItemCandidateDecorator(AdsCandidateUrtItemBuilder())),\n    alerts: Seq[Alert] = Seq.empty,\n    urtRequest: Option[Boolean] = None,\n  ): AdsDependentCandidatePipelineConfig[Query] = new AdsDependentCandidatePipelineConfig[Query](\n    identifier = identifier,\n    enabledDeciderParam = enabledDeciderParam,\n    supportedClientParam = supportedClientParam,\n    gates = gates,\n    candidateSource = adsCandidateSource,\n    filters = filters,\n    postFilterFeatureHydration = postFilterFeatureHydration,\n    decorator = decorator,\n    alerts = alerts,\n    adsDisplayLocationBuilder = adsDisplayLocationBuilder,\n    getOrganicItemIds = getOrganicItemIds,\n    countNumOrganicItems = countNumOrganicItems,\n    urtRequest = urtRequest)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/AdsDependentCandidatePipelineQueryTransformer.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.ads\n\nimport com.twitter.adserver.{thriftscala => ads}\nimport com.twitter.product_mixer.component_library.model.query.ads.AdsQuery\nimport com.twitter.product_mixer.component_library.pipeline.candidate.ads.AdsCandidatePipelineQueryTransformer.buildAdRequestParams\nimport com.twitter.product_mixer.core.functional_component.transformer.DependentCandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Transform a PipelineQuery with AdsQuery into an AdsRequestParams\n *\n * @param adsDisplayLocationBuilder Builder that determines the display location for the ads\n * @param countNumOrganicItems      Count organic items from the response \n */\ncase class AdsDependentCandidatePipelineQueryTransformer[Query <: PipelineQuery with AdsQuery](\n  adsDisplayLocationBuilder: AdsDisplayLocationBuilder[Query],\n  getOrganicItemIds: GetOrganicItemIds,\n  countNumOrganicItems: CountNumOrganicItems[Query],\n  urtRequest: Option[Boolean],\n) extends DependentCandidatePipelineQueryTransformer[Query, ads.AdRequestParams] {\n\n  override def transform(\n    query: Query,\n    previousCandidates: Seq[CandidateWithDetails]\n  ): ads.AdRequestParams = buildAdRequestParams(\n    query = query,\n    adsDisplayLocation = adsDisplayLocationBuilder(query),\n    organicItemIds = getOrganicItemIds.apply(previousCandidates),\n    numOrganicItems = Some(countNumOrganicItems.apply(query, previousCandidates)),\n    urtRequest = urtRequest\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/AdsDisplayLocationBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.ads\n\nimport com.twitter.adserver.{thriftscala => ads}\nimport com.twitter.product_mixer.component_library.model.query.ads.AdsQuery\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait AdsDisplayLocationBuilder[-Query <: PipelineQuery with AdsQuery] {\n\n  def apply(query: Query): ads.DisplayLocation\n}\n\ncase class StaticAdsDisplayLocationBuilder(displayLocation: ads.DisplayLocation)\n    extends AdsDisplayLocationBuilder[PipelineQuery with AdsQuery] {\n\n  def apply(query: PipelineQuery with AdsQuery): ads.DisplayLocation = displayLocation\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    scalac_plugins = [\"no-roomba\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/CountNumOrganicItems.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.ads\n\nimport com.twitter.product_mixer.component_library.model.query.ads.AdsQuery\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Derive an estimate of the number of organic items from the query. If you need a more precise number,\n * consider switching to [[AdsDependentCandidatePipelineConfig]]\n */\ntrait EstimateNumOrganicItems[Query <: PipelineQuery with AdsQuery] {\n\n  def apply(query: Query): Short\n}\n\n/**\n * Compute the number of organic items from the query and set of previous candidates.\n *\n * @note the key difference between [[CountNumOrganicItems]] and [[EstimateNumOrganicItems]] is\n *       that for [[EstimateNumOrganicItems]] we don't have any candidates returned yet, so we can\n *       only guess as to the number of organic items in the result set. In contrast,\n *       [[CountNumOrganicItems]] is used on dependant candidate pipelines where we can scan over\n *       the candidate pipelines result set to count the number of organic items.\n */\ntrait CountNumOrganicItems[-Query <: PipelineQuery with AdsQuery] {\n\n  def apply(query: Query, previousCandidates: Seq[CandidateWithDetails]): Short\n}\n\n/**\n * Treat all previously retrieved candidates as organic\n */\ncase object CountAllCandidates extends CountNumOrganicItems[PipelineQuery with AdsQuery] {\n\n  def apply(\n    query: PipelineQuery with AdsQuery,\n    previousCandidates: Seq[CandidateWithDetails]\n  ): Short =\n    previousCandidates.length.toShort\n}\n\n/**\n * Only count candidates from a specific subset of pipelines as organic\n */\ncase class CountCandidatesFromPipelines(pipelines: CandidateScope)\n    extends CountNumOrganicItems[PipelineQuery with AdsQuery] {\n\n  def apply(\n    query: PipelineQuery with AdsQuery,\n    previousCandidates: Seq[CandidateWithDetails]\n  ): Short =\n    previousCandidates.count(pipelines.contains).toShort\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/GetOrganicItemIds.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.ads\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\n\n/**\n * Get organic item candidates from the set of previous candidates\n */\ntrait GetOrganicItemIds {\n\n  def apply(previousCandidates: Seq[CandidateWithDetails]): Option[Seq[Long]]\n}\n\n/**\n * Get organic items from specified pipelines\n */\ncase class PipelineScopedOrganicItemIds(pipelines: CandidateScope) extends GetOrganicItemIds {\n\n  def apply(previousCandidates: Seq[CandidateWithDetails]): Option[Seq[Long]] =\n    Some(previousCandidates.filter(pipelines.contains).map(_.candidateIdLong))\n}\n\n/**\n * Get an empty list of organic item candidates\n */\ncase object EmptyOrganicItemIds extends GetOrganicItemIds {\n\n  def apply(previousCandidates: Seq[CandidateWithDetails]): Option[Seq[Long]] = None\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/PromotedTweetsOnlyFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.ads\n\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsTweetCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\ncase class PromotedTweetsOnlyFilter[Query <: PipelineQuery](\n  underlyingFilter: Filter[Query, AdsTweetCandidate])\n    extends Filter[Query, AdsCandidate] {\n\n  override val identifier: FilterIdentifier =\n    FilterIdentifier(s\"PromotedTweets${underlyingFilter.identifier.name}\")\n\n  override def apply(\n    query: Query,\n    candidatesWithFeatures: Seq[CandidateWithFeatures[AdsCandidate]]\n  ): Stitch[FilterResult[AdsCandidate]] = {\n\n    val adsTweetCandidates: Seq[CandidateWithFeatures[AdsTweetCandidate]] =\n      candidatesWithFeatures.flatMap {\n        case tweetCandidateWithFeatures @ CandidateWithFeatures(_: AdsTweetCandidate, _) =>\n          Some(tweetCandidateWithFeatures.asInstanceOf[CandidateWithFeatures[AdsTweetCandidate]])\n        case _ => None\n      }\n\n    underlyingFilter\n      .apply(query, adsTweetCandidates)\n      .map { filterResult =>\n        val removedSet = filterResult.removed.toSet[AdsCandidate]\n        val (removed, kept) = candidatesWithFeatures.map(_.candidate).partition(removedSet.contains)\n        FilterResult(kept, removed)\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/ads/ValidAdImpressionIdFilter.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.ads\n\nimport com.twitter.product_mixer.component_library.model.candidate.ads.AdsCandidate\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.filter.FilterResult\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject ValidAdImpressionIdFilter extends Filter[PipelineQuery, AdsCandidate] {\n  override val identifier: FilterIdentifier = FilterIdentifier(\"ValidAdImpressionId\")\n\n  override def apply(\n    query: PipelineQuery,\n    candidatesWithFeatures: Seq[CandidateWithFeatures[AdsCandidate]]\n  ): Stitch[FilterResult[AdsCandidate]] = {\n    val (kept, removed) = candidatesWithFeatures\n      .map(_.candidate)\n      .partition(candidate => candidate.adImpression.impressionString.exists(_.nonEmpty))\n\n    Stitch.value(FilterResult(kept, removed))\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/flexible_injection_pipeline\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/flexible_injection_pipeline\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/FlipPromptCandidatePipelineConfig.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline\n\nimport com.twitter.onboarding.task.service.thriftscala.GetInjectionsRequest\nimport com.twitter.onboarding.task.service.{thriftscala => servicethrift}\nimport com.twitter.product_mixer.component_library.candidate_source.flexible_injection_pipeline.IntermediatePrompt\nimport com.twitter.product_mixer.component_library.candidate_source.flexible_injection_pipeline.PromptCandidateSource\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline.FlipPromptCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline.FlipPromptModuleGrouping\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline.FlipPromptUrtModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.BasePromptCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.FlipCandidateFeatureTransformer\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.FlipQueryTransformer\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.HasFlipInjectionParams\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.PromptResultsTransformer\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.decider.DeciderParam\n\n/**\n * A candidate pipeline for Flexible Injection Pipeline Candidates.\n * Fetches prompts from FLIP (inside onboarding-task-service).\n */\nclass FlipPromptCandidatePipelineConfig[Query <: PipelineQuery with HasFlipInjectionParams](\n  override val identifier: CandidatePipelineIdentifier,\n  override val enabledDeciderParam: Option[DeciderParam[Boolean]],\n  override val supportedClientParam: Option[FSParam[Boolean]],\n  promptCandidateSource: PromptCandidateSource)\n    extends CandidatePipelineConfig[\n      Query,\n      servicethrift.GetInjectionsRequest,\n      IntermediatePrompt,\n      BasePromptCandidate[Any]\n    ] {\n\n  override val candidateSource: CandidateSource[GetInjectionsRequest, IntermediatePrompt] =\n    promptCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[Query, GetInjectionsRequest] =\n    FlipQueryTransformer\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    IntermediatePrompt,\n    BasePromptCandidate[Any]\n  ] = PromptResultsTransformer\n\n  override val decorator: Option[\n    CandidateDecorator[Query, BasePromptCandidate[Any]]\n  ] = Some(\n    UrtMultipleModulesDecorator(\n      urtItemCandidateDecorator = UrtItemCandidateDecorator(FlipPromptCandidateUrtItemBuilder()),\n      moduleBuilder = FlipPromptUrtModuleBuilder(),\n      groupByKey = FlipPromptModuleGrouping\n    ))\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[IntermediatePrompt]\n  ] = Seq(FlipCandidateFeatureTransformer)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/FlipPromptCandidatePipelineConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline\n\nimport com.twitter.product_mixer.component_library.candidate_source.flexible_injection_pipeline.PromptCandidateSource\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.HasFlipInjectionParams\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.decider.DeciderParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FlipPromptCandidatePipelineConfigBuilder @Inject() (\n  promptCandidateSource: PromptCandidateSource) {\n\n  /**\n   * Build a FlipPromptCandidatePipelineConfig\n   *\n   * @note If injected classes are needed to populate parameters in this method, consider creating a\n   *       ProductFlipPromptCandidatePipelineConfigBuilder with a single `def build()` method.\n   *       That product-specific builder class can then inject everything it needs (including this\n   *       class), and delegate to this class's build() method within its own build() method.\n   */\n  def build[Query <: PipelineQuery with HasFlipInjectionParams](\n    identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\"FlipPrompt\"),\n    enabledDeciderParam: Option[DeciderParam[Boolean]] = None,\n    supportedClientParam: Option[FSParam[Boolean]] = None,\n  ): FlipPromptCandidatePipelineConfig[Query] = {\n    new FlipPromptCandidatePipelineConfig(\n      identifier = identifier,\n      enabledDeciderParam = enabledDeciderParam,\n      supportedClientParam = supportedClientParam,\n      promptCandidateSource = promptCandidateSource)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/FlipPromptDependentCandidatePipelineConfig.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline\n\nimport com.twitter.onboarding.task.service.thriftscala.GetInjectionsRequest\nimport com.twitter.onboarding.task.service.{thriftscala => servicethrift}\nimport com.twitter.product_mixer.component_library.candidate_source.flexible_injection_pipeline.IntermediatePrompt\nimport com.twitter.product_mixer.component_library.candidate_source.flexible_injection_pipeline.PromptCandidateSource\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtMultipleModulesDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline.FlipPromptCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline.FlipPromptModuleGrouping\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.flexible_injection_pipeline.FlipPromptUrtModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.BasePromptCandidate\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.FlipCandidateFeatureTransformer\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.FlipQueryTransformer\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.HasFlipInjectionParams\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.PromptResultsTransformer\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.decider.DeciderParam\n\n/**\n * A dependent candidate pipeline for Flexible Injection Pipeline Candidates.\n * Fetches prompts from FLIP (inside onboarding-task-service).\n */\nclass FlipPromptDependentCandidatePipelineConfig[\n  Query <: PipelineQuery with HasFlipInjectionParams\n](\n  override val identifier: CandidatePipelineIdentifier,\n  override val enabledDeciderParam: Option[DeciderParam[Boolean]],\n  override val supportedClientParam: Option[FSParam[Boolean]],\n  promptCandidateSource: PromptCandidateSource)\n    extends DependentCandidatePipelineConfig[\n      Query,\n      servicethrift.GetInjectionsRequest,\n      IntermediatePrompt,\n      BasePromptCandidate[Any]\n    ] {\n\n  override val candidateSource: CandidateSource[GetInjectionsRequest, IntermediatePrompt] =\n    promptCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[Query, GetInjectionsRequest] =\n    FlipQueryTransformer\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    IntermediatePrompt,\n    BasePromptCandidate[Any]\n  ] = PromptResultsTransformer\n\n  override val decorator: Option[\n    CandidateDecorator[Query, BasePromptCandidate[Any]]\n  ] = Some(\n    UrtMultipleModulesDecorator(\n      urtItemCandidateDecorator = UrtItemCandidateDecorator(FlipPromptCandidateUrtItemBuilder()),\n      moduleBuilder = FlipPromptUrtModuleBuilder(),\n      groupByKey = FlipPromptModuleGrouping\n    ))\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[IntermediatePrompt]\n  ] = Seq(FlipCandidateFeatureTransformer)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/FlipPromptDependentCandidatePipelineConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline\n\nimport com.twitter.product_mixer.component_library.candidate_source.flexible_injection_pipeline.PromptCandidateSource\nimport com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer.HasFlipInjectionParams\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.decider.DeciderParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FlipPromptDependentCandidatePipelineConfigBuilder @Inject() (\n  promptCandidateSource: PromptCandidateSource) {\n\n  /**\n   * Build a FlipPromptDependentCandidatePipelineConfig\n   *\n   * @note If injected classes are needed to populate parameters in this method, consider creating a\n   *       ProductFlipPromptDependentCandidatePipelineConfigBuilder with a single `def build()` method.\n   *       That product-specific builder class can then inject everything it needs (including this\n   *       class), and delegate to this class's build() method within its own build() method.\n   */\n  def build[Query <: PipelineQuery with HasFlipInjectionParams](\n    identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\"FlipPrompt\"),\n    enabledDeciderParam: Option[DeciderParam[Boolean]] = None,\n    supportedClientParam: Option[FSParam[Boolean]] = None,\n  ): FlipPromptDependentCandidatePipelineConfig[Query] = {\n    new FlipPromptDependentCandidatePipelineConfig(\n      identifier = identifier,\n      enabledDeciderParam = enabledDeciderParam,\n      supportedClientParam = supportedClientParam,\n      promptCandidateSource = promptCandidateSource)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/transformer/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    scalac_plugins = [\"no-roomba\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/flexible_injection_pipeline\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/flexible_injection_pipeline\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/transformer/FlipCandidateFeatureTransformer.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer\n\nimport com.twitter.onboarding.injections.{thriftscala => onboardingthrift}\nimport com.twitter.product_mixer.component_library.candidate_source.flexible_injection_pipeline.IntermediatePrompt\nimport com.twitter.product_mixer.component_library.model.candidate.BasePromptCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.PromptCarouselTileCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\ncase object FlipPromptCarouselTileFeature\n    extends Feature[PromptCarouselTileCandidate, Option[onboardingthrift.Tile]]\n\ncase object FlipPromptInjectionsFeature\n    extends Feature[BasePromptCandidate[String], onboardingthrift.Injection]\n\ncase object FlipPromptOffsetInModuleFeature\n    extends Feature[PromptCarouselTileCandidate, Option[Int]]\n\nobject FlipCandidateFeatureTransformer extends CandidateFeatureTransformer[IntermediatePrompt] {\n\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"FlipCandidateFeature\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(FlipPromptInjectionsFeature, FlipPromptOffsetInModuleFeature, FlipPromptCarouselTileFeature)\n\n  /** Hydrates a [[FeatureMap]] for a given [[Inputs]] */\n  override def transform(input: IntermediatePrompt): FeatureMap = {\n    FeatureMapBuilder()\n      .add(FlipPromptInjectionsFeature, input.injection)\n      .add(FlipPromptOffsetInModuleFeature, input.offsetInModule)\n      .add(FlipPromptCarouselTileFeature, input.carouselTile)\n      .build()\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/transformer/FlipInjectionParams.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer\n\nimport com.twitter.onboarding.task.service.{thriftscala => flip}\n\ntrait HasFlipInjectionParams {\n  def displayLocation: flip.DisplayLocation\n  def rankingDisablerWithLatestControlsAvailable: Option[Boolean]\n  def isEmptyState: Option[Boolean]\n  def isFirstRequestAfterSignup: Option[Boolean]\n  def isEndOfTimeline: Option[Boolean]\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/transformer/FlipQueryTransformer.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer\n\nimport com.twitter.onboarding.task.service.thriftscala.PromptType\nimport com.twitter.onboarding.task.service.{thriftscala => flip}\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject FlipQueryTransformer\n    extends CandidatePipelineQueryTransformer[\n      PipelineQuery with HasFlipInjectionParams,\n      flip.GetInjectionsRequest\n    ] {\n\n  val SUPPORTED_PROMPT_TYPES: Set[PromptType] = Set(\n    PromptType.InlinePrompt,\n    PromptType.FullCover,\n    PromptType.HalfCover,\n    PromptType.TileCarousel,\n    PromptType.RelevancePrompt)\n\n  override def transform(\n    query: PipelineQuery with HasFlipInjectionParams\n  ): flip.GetInjectionsRequest = {\n    val clientContext = flip.ClientContext(\n      userId = query.clientContext.userId,\n      guestId = query.clientContext.guestId,\n      clientApplicationId = query.clientContext.appId,\n      deviceId = query.clientContext.deviceId,\n      countryCode = query.clientContext.countryCode,\n      languageCode = query.clientContext.languageCode,\n      userAgent = query.clientContext.userAgent,\n      guestIdMarketing = query.clientContext.guestIdMarketing,\n      guestIdAds = query.clientContext.guestIdAds,\n      isInternalOrTwoffice = query.clientContext.isTwoffice,\n      ipAddress = query.clientContext.ipAddress\n    )\n    val displayContext: flip.DisplayContext =\n      flip.DisplayContext(\n        displayLocation = query.displayLocation,\n        timelineId = query.clientContext.userId\n      )\n\n    val requestTargetingContext: flip.RequestTargetingContext =\n      flip.RequestTargetingContext(\n        rankingDisablerWithLatestControlsAvaliable =\n          query.rankingDisablerWithLatestControlsAvailable,\n        reactivePromptContext = None,\n        isEmptyState = query.isEmptyState,\n        isFirstRequestAfterSignup = query.isFirstRequestAfterSignup,\n        isEndOfTimeline = query.isEndOfTimeline\n      )\n\n    flip.GetInjectionsRequest(\n      clientContext = clientContext,\n      displayContext = displayContext,\n      requestTargetingContext = Some(requestTargetingContext),\n      userRoles = query.clientContext.userRoles,\n      timelineContext = None,\n      supportedPromptTypes = Some(SUPPORTED_PROMPT_TYPES)\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/flexible_injection_pipeline/transformer/PromptResultsTransformer.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.flexible_injection_pipeline.transformer\n\nimport com.twitter.onboarding.injections.{thriftscala => flipinjection}\nimport com.twitter.product_mixer.component_library.candidate_source.flexible_injection_pipeline.IntermediatePrompt\nimport com.twitter.product_mixer.component_library.model.candidate.BasePromptCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.FullCoverPromptCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.HalfCoverPromptCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.InlinePromptCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.PromptCarouselTileCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.RelevancePromptCandidate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\n\nobject PromptResultsTransformer\n    extends CandidatePipelineResultsTransformer[\n      IntermediatePrompt,\n      BasePromptCandidate[Any]\n    ] {\n\n  /**\n   * Transforms a Flip Injection to a Product Mixer domain object deriving from BasePromptCandidate.\n   * Supported injection types have to match those declared in com.twitter.product_mixer.component_library.transformer.flexible_injection_pipeline.FlipQueryTransformer#supportedPromptFormats\n   */\n  override def transform(input: IntermediatePrompt): BasePromptCandidate[Any] =\n    input.injection match {\n      case inlinePrompt: flipinjection.Injection.InlinePrompt =>\n        InlinePromptCandidate(id = inlinePrompt.inlinePrompt.injectionIdentifier\n          .getOrElse(throw new MissingInjectionId(input.injection)))\n      case _: flipinjection.Injection.FullCover =>\n        FullCoverPromptCandidate(id = \"0\")\n      case _: flipinjection.Injection.HalfCover =>\n        HalfCoverPromptCandidate(id = \"0\")\n      case _: flipinjection.Injection.TilesCarousel =>\n        PromptCarouselTileCandidate(id =\n          input.offsetInModule.getOrElse(throw FlipPromptOffsetInModuleMissing))\n      case relevancePrompt: flipinjection.Injection.RelevancePrompt =>\n        RelevancePromptCandidate(\n          id = relevancePrompt.relevancePrompt.injectionIdentifier,\n          position = relevancePrompt.relevancePrompt.requestedPosition.map(_.toInt))\n      case injection => throw new UnsupportedInjectionType(injection)\n    }\n}\n\nclass MissingInjectionId(injection: flipinjection.Injection)\n    extends IllegalArgumentException(\n      s\"Injection identifier is missing ${TransportMarshaller.getSimpleName(injection.getClass)}\")\n\nclass UnsupportedInjectionType(injection: flipinjection.Injection)\n    extends UnsupportedOperationException(\n      s\"Unsupported FLIP injection Type : ${TransportMarshaller.getSimpleName(injection.getClass)}\")\n\nobject FlipPromptOffsetInModuleMissing\n    extends NoSuchElementException(\n      \"FlipPromptOffsetInModuleFeature must be set for the TilesCarousel FLIP injection in PromptCandidateSource\")\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/account_recommendations_mixer\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/candidate_source/people_discovery\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"servo/repo/src/main/scala\",\n        \"src/thrift/com/twitter/hermit/internal:hermit-internal-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowArmCandidateDecorator.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.product_mixer.component_library.candidate_source.account_recommendations_mixer.WhoToFollowModuleFooterFeature\nimport com.twitter.product_mixer.component_library.candidate_source.account_recommendations_mixer.WhoToFollowModuleHeaderFeature\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemInModuleDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.user.UserCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.StaticUrlBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.promoted.FeaturePromotedMetadataBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.social_context.WhoToFollowSocialContextBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.StrStatic\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleFooterBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleHeaderBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DeepLink\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject WhoToFollowArmCandidateDecorator {\n  val ClientEventComponent = \"suggest_who_to_follow\"\n  val EntryNamespaceString = \"who-to-follow\"\n}\n\ncase class WhoToFollowArmCandidateDecorator[-Query <: PipelineQuery](\n  moduleDisplayTypeBuilder: BaseModuleDisplayTypeBuilder[Query, UserCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, UserCandidate]\n  ]) extends CandidateDecorator[Query, UserCandidate] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[UserCandidate]]\n  ): Stitch[Seq[Decoration]] = {\n    val clientEventDetailsBuilder = WhoToFollowClientEventDetailsBuilder(TrackingTokenFeature)\n    val clientEventInfoBuilder = ClientEventInfoBuilder[Query, UserCandidate](\n      WhoToFollowArmCandidateDecorator.ClientEventComponent,\n      Some(clientEventDetailsBuilder))\n    val promotedMetadataBuilder = FeaturePromotedMetadataBuilder(AdImpressionFeature)\n    val socialContextBuilder =\n      WhoToFollowSocialContextBuilder(SocialTextFeature, HermitContextTypeFeature)\n    val userItemBuilder = UserCandidateUrtItemBuilder(\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      promotedMetadataBuilder = Some(promotedMetadataBuilder),\n      socialContextBuilder = Some(socialContextBuilder))\n    val userItemDecorator = UrtItemCandidateDecorator(userItemBuilder)\n\n    val whoToFollowModuleBuilder = {\n      val whoToFollowHeaderOpt = query.features.map(_.get(WhoToFollowModuleHeaderFeature))\n      val whoToFollowFooterOpt = query.features.flatMap(_.get(WhoToFollowModuleFooterFeature))\n      val whoToFollowModuleHeaderBuilder = whoToFollowHeaderOpt.flatMap(_.title).map { title =>\n        ModuleHeaderBuilder(textBuilder = StrStatic(title), isSticky = Some(true))\n      }\n      val whoToFollowModuleFooterBuilder = whoToFollowFooterOpt.flatMap(_.action).map { action =>\n        ModuleFooterBuilder(\n          textBuilder = StrStatic(action.title),\n          urlBuilder = Some(StaticUrlBuilder(action.actionUrl, DeepLink)))\n      }\n\n      TimelineModuleBuilder(\n        entryNamespace = EntryNamespace(WhoToFollowArmCandidateDecorator.EntryNamespaceString),\n        clientEventInfoBuilder = clientEventInfoBuilder,\n        displayTypeBuilder = moduleDisplayTypeBuilder,\n        headerBuilder = whoToFollowModuleHeaderBuilder,\n        footerBuilder = whoToFollowModuleFooterBuilder,\n        feedbackActionInfoBuilder = feedbackActionInfoBuilder,\n      )\n    }\n\n    UrtItemInModuleDecorator(\n      userItemDecorator,\n      whoToFollowModuleBuilder\n    ).apply(query, candidates)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowArmCandidatePipelineConfig.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\n\nobject WhoToFollowArmCandidatePipelineConfig {\n  val MinCandidatesSize = 3\n  val MaxCandidatesSize = 20\n\n  val identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\"WhoToFollowArm\")\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowArmCandidatePipelineQueryTransformer.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.account_recommendations_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.marshaller.request.ClientContextMarshaller\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.BadRequest\nimport com.twitter.timelines.configapi.Param\n\nobject WhoToFollowArmCandidatePipelineQueryTransformer {\n  val HomeDisplayLocation = \"timeline\"\n  val HomeReverseChronDisplayLocation = \"timeline_reverse_chron\"\n  val ProfileDisplayLocation = \"profile_timeline\"\n}\n\ncase class WhoToFollowArmCandidatePipelineQueryTransformer[-Query <: PipelineQuery](\n  displayLocationParam: Param[String],\n  excludedUserIdsFeature: Option[Feature[PipelineQuery, Seq[Long]]],\n  profileUserIdFeature: Option[Feature[PipelineQuery, Long]])\n    extends CandidatePipelineQueryTransformer[Query, t.AccountRecommendationsMixerRequest] {\n\n  override def transform(input: Query): t.AccountRecommendationsMixerRequest = {\n    input.params(displayLocationParam) match {\n      case WhoToFollowArmCandidatePipelineQueryTransformer.HomeReverseChronDisplayLocation =>\n        t.AccountRecommendationsMixerRequest(\n          clientContext = ClientContextMarshaller(input.clientContext),\n          product = t.Product.HomeReverseChronWhoToFollow,\n          productContext = Some(\n            t.ProductContext.HomeReverseChronWhoToFollowProductContext(\n              t.HomeReverseChronWhoToFollowProductContext(\n                wtfReactiveContext = Some(getWhoToFollowReactiveContext(input))\n              )))\n        )\n      case WhoToFollowArmCandidatePipelineQueryTransformer.HomeDisplayLocation =>\n        t.AccountRecommendationsMixerRequest(\n          clientContext = ClientContextMarshaller(input.clientContext),\n          product = t.Product.HomeWhoToFollow,\n          productContext = Some(\n            t.ProductContext.HomeWhoToFollowProductContext(\n              t.HomeWhoToFollowProductContext(\n                wtfReactiveContext = Some(getWhoToFollowReactiveContext(input))\n              )))\n        )\n      case WhoToFollowArmCandidatePipelineQueryTransformer.ProfileDisplayLocation =>\n        t.AccountRecommendationsMixerRequest(\n          clientContext = ClientContextMarshaller(input.clientContext),\n          product = t.Product.ProfileWhoToFollow,\n          productContext = Some(\n            t.ProductContext.ProfileWhoToFollowProductContext(t.ProfileWhoToFollowProductContext(\n              wtfReactiveContext = Some(getWhoToFollowReactiveContext(input)),\n              profileUserId = profileUserIdFeature\n                .flatMap(feature => input.features.map(_.get(feature)))\n                .getOrElse(throw PipelineFailure(BadRequest, \"profileUserId not provided\")),\n            )))\n        )\n      case displayLocation =>\n        throw PipelineFailure(BadRequest, s\"display location $displayLocation not supported\")\n    }\n  }\n\n  private def getWhoToFollowReactiveContext(\n    input: Query\n  ): t.WhoToFollowReactiveContext = {\n    t.WhoToFollowReactiveContext(\n      excludedUserIds = excludedUserIdsFeature.flatMap(feature =>\n        input.features\n          .map(_.get(feature))),\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowArmDependentCandidatePipelineConfig.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.account_recommendations_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.component_library.candidate_source.account_recommendations_mixer.AccountRecommendationsMixerCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelines.configapi.decider.DeciderParam\n\nclass WhoToFollowArmDependentCandidatePipelineConfig[Query <: PipelineQuery](\n  override val identifier: CandidatePipelineIdentifier,\n  override val enabledDeciderParam: Option[DeciderParam[Boolean]],\n  override val supportedClientParam: Option[FSParam[Boolean]],\n  override val alerts: Seq[Alert],\n  override val gates: Seq[BaseGate[Query]],\n  accountRecommendationsMixerCandidateSource: AccountRecommendationsMixerCandidateSource,\n  override val filters: Seq[Filter[Query, UserCandidate]],\n  moduleDisplayTypeBuilder: BaseModuleDisplayTypeBuilder[Query, UserCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[PipelineQuery, UserCandidate]\n  ],\n  displayLocationParam: Param[String],\n  excludedUserIdsFeature: Option[Feature[PipelineQuery, Seq[Long]]],\n  profileUserIdFeature: Option[Feature[PipelineQuery, Long]])\n    extends DependentCandidatePipelineConfig[\n      Query,\n      t.AccountRecommendationsMixerRequest,\n      t.RecommendedUser,\n      UserCandidate\n    ] {\n\n  override val candidateSource: BaseCandidateSource[\n    t.AccountRecommendationsMixerRequest,\n    t.RecommendedUser\n  ] =\n    accountRecommendationsMixerCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    PipelineQuery,\n    t.AccountRecommendationsMixerRequest\n  ] = WhoToFollowArmCandidatePipelineQueryTransformer(\n    displayLocationParam = displayLocationParam,\n    excludedUserIdsFeature = excludedUserIdsFeature,\n    profileUserIdFeature = profileUserIdFeature\n  )\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[t.RecommendedUser]\n  ] = Seq(WhoToFollowArmResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    t.RecommendedUser,\n    UserCandidate\n  ] = { user => UserCandidate(user.userId) }\n\n  override val decorator: Option[CandidateDecorator[Query, UserCandidate]] =\n    Some(\n      WhoToFollowArmCandidateDecorator(\n        moduleDisplayTypeBuilder,\n        feedbackActionInfoBuilder\n      ))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowArmDependentCandidatePipelineConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.product_mixer.component_library.candidate_source.account_recommendations_mixer.AccountRecommendationsMixerCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelines.configapi.decider.DeciderParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass WhoToFollowArmDependentCandidatePipelineConfigBuilder @Inject() (\n  accountRecommendationsMixerCandidateSource: AccountRecommendationsMixerCandidateSource) {\n\n  /**\n   * Build a WhoToFollowArmDependentCandidatePipelineConfig\n   *\n   *\n   * To create a regular CandidatePipelineConfig instead see [[WhoToFollowArmCandidatePipelineConfigBuilder]].\n   *\n   * @note If injected classes are needed to populate parameters in this method, consider creating a\n   *       ProductWhoToFollowCandidatePipelineConfigBuilder with a single `def build()` method. That\n   *       product-specific builder class can then inject everything it needs (including this class),\n   *       and delegate to this class's build() method within its own build() method.\n   */\n  def build[Query <: PipelineQuery](\n    moduleDisplayTypeBuilder: BaseModuleDisplayTypeBuilder[Query, UserCandidate],\n    identifier: CandidatePipelineIdentifier = WhoToFollowArmCandidatePipelineConfig.identifier,\n    enabledDeciderParam: Option[DeciderParam[Boolean]] = None,\n    supportedClientParam: Option[FSParam[Boolean]] = None,\n    alerts: Seq[Alert] = Seq.empty,\n    gates: Seq[BaseGate[Query]] = Seq.empty,\n    filters: Seq[Filter[Query, UserCandidate]] = Seq.empty,\n    feedbackActionInfoBuilder: Option[BaseFeedbackActionInfoBuilder[\n      PipelineQuery,\n      UserCandidate\n    ]] = None,\n    displayLocationParam: Param[String] =\n      StaticParam(WhoToFollowArmCandidatePipelineQueryTransformer.HomeDisplayLocation),\n    excludedUserIdsFeature: Option[Feature[PipelineQuery, Seq[Long]]],\n    profileUserIdFeature: Option[Feature[PipelineQuery, Long]]\n  ): WhoToFollowArmDependentCandidatePipelineConfig[Query] =\n    new WhoToFollowArmDependentCandidatePipelineConfig(\n      identifier = identifier,\n      enabledDeciderParam = enabledDeciderParam,\n      supportedClientParam = supportedClientParam,\n      alerts = alerts,\n      gates = gates,\n      accountRecommendationsMixerCandidateSource = accountRecommendationsMixerCandidateSource,\n      filters = filters,\n      moduleDisplayTypeBuilder = moduleDisplayTypeBuilder,\n      feedbackActionInfoBuilder = feedbackActionInfoBuilder,\n      displayLocationParam = displayLocationParam,\n      excludedUserIdsFeature = excludedUserIdsFeature,\n      profileUserIdFeature = profileUserIdFeature\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowArmResponseFeatureTransformer.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.hermit.{thriftscala => h}\nimport com.twitter.account_recommendations_mixer.{thriftscala => t}\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject ContextTypeFeature extends Feature[UserCandidate, Option[t.ContextType]]\n\nobject WhoToFollowArmResponseFeatureTransformer\n    extends CandidateFeatureTransformer[t.RecommendedUser] {\n\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"WhoToFollowArmResponse\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(\n      AdImpressionFeature,\n      ContextTypeFeature,\n      HermitContextTypeFeature,\n      SocialTextFeature,\n      TrackingTokenFeature,\n      ScoreFeature)\n\n  override def transform(input: t.RecommendedUser): FeatureMap = FeatureMapBuilder()\n    .add(AdImpressionFeature, input.adImpression)\n    .add(ContextTypeFeature, input.contextType)\n    .add(\n      HermitContextTypeFeature,\n      input.contextType.map(contextType => h.ContextType(contextType.value)))\n    .add(SocialTextFeature, input.socialText)\n    .add(TrackingTokenFeature, input.trackingToken)\n    .add(ScoreFeature, input.mlPredictionScore)\n    .build()\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowCandidateDecorator.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.product_mixer.component_library.candidate_source.people_discovery.WhoToFollowModuleHeaderFeature\nimport com.twitter.product_mixer.component_library.candidate_source.people_discovery.WhoToFollowModuleShowMoreFeature\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemCandidateDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.UrtItemInModuleDecorator\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.item.user.UserCandidateUrtItemBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.ClientEventInfoBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.metadata.StaticUrlBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.promoted.FeaturePromotedMetadataBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.social_context.WhoToFollowSocialContextBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.stringcenter.StrStatic\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleDynamicShowMoreBehaviorRevealByCountBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleFooterBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleHeaderBuilder\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.TimelineModuleBuilder\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DeepLink\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\nobject WhoToFollowCandidateDecorator {\n  val ClientEventComponent = \"suggest_who_to_follow\"\n  val EntryNamespaceString = \"who-to-follow\"\n}\n\ncase class WhoToFollowCandidateDecorator[-Query <: PipelineQuery](\n  moduleDisplayTypeBuilder: BaseModuleDisplayTypeBuilder[Query, UserCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[Query, UserCandidate]\n  ]) extends CandidateDecorator[Query, UserCandidate] {\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[UserCandidate]]\n  ): Stitch[Seq[Decoration]] = {\n    val clientEventDetailsBuilder = WhoToFollowClientEventDetailsBuilder(TrackingTokenFeature)\n    val clientEventInfoBuilder = ClientEventInfoBuilder[Query, UserCandidate](\n      WhoToFollowCandidateDecorator.ClientEventComponent,\n      Some(clientEventDetailsBuilder))\n    val promotedMetadataBuilder = FeaturePromotedMetadataBuilder(AdImpressionFeature)\n    val socialContextBuilder =\n      WhoToFollowSocialContextBuilder(SocialTextFeature, HermitContextTypeFeature)\n    val userItemBuilder = UserCandidateUrtItemBuilder(\n      clientEventInfoBuilder = clientEventInfoBuilder,\n      promotedMetadataBuilder = Some(promotedMetadataBuilder),\n      socialContextBuilder = Some(socialContextBuilder))\n    val userItemDecorator = UrtItemCandidateDecorator(userItemBuilder)\n\n    val whoToFollowModuleBuilder = {\n      val whoToFollowHeaderOpt = query.features.map(_.get(WhoToFollowModuleHeaderFeature))\n      val whoToFollowModuleHeaderBuilder = whoToFollowHeaderOpt.flatMap(_.title).map { title =>\n        ModuleHeaderBuilder(textBuilder = StrStatic(title.text), isSticky = Some(true))\n      }\n      val whoToFollowModuleFooterBuilder = whoToFollowHeaderOpt.flatMap(_.action).map { action =>\n        ModuleFooterBuilder(\n          textBuilder = StrStatic(action.title),\n          urlBuilder = Some(StaticUrlBuilder(action.actionUrl, DeepLink)))\n      }\n      val showMoreBehaviorBuilder =\n        query.features.flatMap(_.get(WhoToFollowModuleShowMoreFeature)).map { showMore =>\n          ModuleDynamicShowMoreBehaviorRevealByCountBuilder(\n            showMore.initialToShow,\n            showMore.extraToShow)\n        }\n\n      TimelineModuleBuilder(\n        entryNamespace = EntryNamespace(WhoToFollowCandidateDecorator.EntryNamespaceString),\n        clientEventInfoBuilder = clientEventInfoBuilder,\n        displayTypeBuilder = moduleDisplayTypeBuilder,\n        headerBuilder = whoToFollowModuleHeaderBuilder,\n        footerBuilder = whoToFollowModuleFooterBuilder,\n        feedbackActionInfoBuilder = feedbackActionInfoBuilder,\n        showMoreBehaviorBuilder = showMoreBehaviorBuilder\n      )\n    }\n\n    UrtItemInModuleDecorator(\n      userItemDecorator,\n      whoToFollowModuleBuilder\n    ).apply(query, candidates)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowCandidatePipelineConfig.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.peoplediscovery.api.{thriftscala => t}\nimport com.twitter.product_mixer.component_library.candidate_source.people_discovery.PeopleDiscoveryCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelines.configapi.decider.DeciderParam\n\nobject WhoToFollowCandidatePipelineConfig {\n  val MinCandidatesSize = 3\n  val MaxCandidatesSize = 20\n\n  val identifier: CandidatePipelineIdentifier = CandidatePipelineIdentifier(\"WhoToFollow\")\n}\n\nclass WhoToFollowCandidatePipelineConfig[Query <: PipelineQuery](\n  override val identifier: CandidatePipelineIdentifier,\n  override val enabledDeciderParam: Option[DeciderParam[Boolean]],\n  override val supportedClientParam: Option[FSParam[Boolean]],\n  override val alerts: Seq[Alert],\n  override val gates: Seq[Gate[Query]],\n  whoToFollowCandidateSource: PeopleDiscoveryCandidateSource,\n  override val filters: Seq[Filter[Query, UserCandidate]],\n  moduleDisplayTypeBuilder: BaseModuleDisplayTypeBuilder[Query, UserCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[PipelineQuery, UserCandidate]\n  ],\n  displayLocationParam: Param[String],\n  supportedLayoutsParam: Param[Seq[String]],\n  layoutVersionParam: Param[Int],\n  excludedUserIdsFeature: Option[Feature[PipelineQuery, Seq[Long]]],\n) extends CandidatePipelineConfig[\n      Query,\n      t.GetModuleRequest,\n      t.RecommendedUser,\n      UserCandidate\n    ] {\n\n  override val candidateSource: BaseCandidateSource[t.GetModuleRequest, t.RecommendedUser] =\n    whoToFollowCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    PipelineQuery,\n    t.GetModuleRequest\n  ] = WhoToFollowCandidatePipelineQueryTransformer(\n    displayLocationParam,\n    supportedLayoutsParam,\n    layoutVersionParam,\n    excludedUserIdsFeature)\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[t.RecommendedUser]\n  ] = Seq(WhoToFollowResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    t.RecommendedUser,\n    UserCandidate\n  ] = { user => UserCandidate(user.userId) }\n\n  override val decorator: Option[CandidateDecorator[Query, UserCandidate]] =\n    Some(WhoToFollowCandidateDecorator(moduleDisplayTypeBuilder, feedbackActionInfoBuilder))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowCandidatePipelineConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.product_mixer.component_library.candidate_source.people_discovery.PeopleDiscoveryCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelines.configapi.decider.DeciderParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass WhoToFollowCandidatePipelineConfigBuilder @Inject() (\n  whoToFollowCandidateSource: PeopleDiscoveryCandidateSource) {\n\n  /**\n   * Build a WhoToFollowCandidatePipelineConfig\n   *\n   * To create a DependentCandidatePipelineConfig instead see [[WhoToFollowDependentCandidatePipelineConfigBuilder]].\n   *\n   * @note If injected classes are needed to populate parameters in this method, consider creating a\n   *       ProductWhoToFollowCandidatePipelineConfigBuilder with a single `def build()` method. That\n   *       product-specific builder class can then inject everything it needs (including this class),\n   *       and delegate to this class's build() method within its own build() method.\n   */\n  def build[Query <: PipelineQuery](\n    moduleDisplayTypeBuilder: BaseModuleDisplayTypeBuilder[Query, UserCandidate],\n    identifier: CandidatePipelineIdentifier = WhoToFollowCandidatePipelineConfig.identifier,\n    enabledDeciderParam: Option[DeciderParam[Boolean]] = None,\n    supportedClientParam: Option[FSParam[Boolean]] = None,\n    alerts: Seq[Alert] = Seq.empty,\n    gates: Seq[Gate[Query]] = Seq.empty,\n    filters: Seq[Filter[Query, UserCandidate]] = Seq.empty,\n    feedbackActionInfoBuilder: Option[BaseFeedbackActionInfoBuilder[\n      PipelineQuery,\n      UserCandidate\n    ]] = None,\n    displayLocationParam: Param[String] =\n      StaticParam(WhoToFollowCandidatePipelineQueryTransformer.DisplayLocation),\n    supportedLayoutsParam: Param[Seq[String]] =\n      StaticParam(WhoToFollowCandidatePipelineQueryTransformer.SupportedLayouts),\n    layoutVersionParam: Param[Int] =\n      StaticParam(WhoToFollowCandidatePipelineQueryTransformer.LayoutVersion),\n    excludedUserIdsFeature: Option[Feature[PipelineQuery, Seq[Long]]] = None,\n  ): WhoToFollowCandidatePipelineConfig[Query] =\n    new WhoToFollowCandidatePipelineConfig(\n      identifier = identifier,\n      enabledDeciderParam = enabledDeciderParam,\n      supportedClientParam = supportedClientParam,\n      alerts = alerts,\n      gates = gates,\n      moduleDisplayTypeBuilder = moduleDisplayTypeBuilder,\n      whoToFollowCandidateSource = whoToFollowCandidateSource,\n      filters = filters,\n      feedbackActionInfoBuilder = feedbackActionInfoBuilder,\n      displayLocationParam = displayLocationParam,\n      supportedLayoutsParam = supportedLayoutsParam,\n      layoutVersionParam = layoutVersionParam,\n      excludedUserIdsFeature = excludedUserIdsFeature\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowCandidatePipelineQueryTransformer.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.peoplediscovery.api.thriftscala.ClientContext\nimport com.twitter.peoplediscovery.api.thriftscala.GetModuleRequest\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\nobject WhoToFollowCandidatePipelineQueryTransformer {\n  val DisplayLocation = \"timeline\"\n  val SupportedLayouts = Seq(\"user-bio-list\")\n  val LayoutVersion = 2\n}\n\ncase class WhoToFollowCandidatePipelineQueryTransformer[-Query <: PipelineQuery](\n  displayLocationParam: Param[String],\n  supportedLayoutsParam: Param[Seq[String]],\n  layoutVersionParam: Param[Int],\n  excludedUserIdsFeature: Option[Feature[PipelineQuery, Seq[Long]]],\n) extends CandidatePipelineQueryTransformer[Query, GetModuleRequest] {\n\n  override def transform(input: Query): GetModuleRequest =\n    GetModuleRequest(\n      clientContext = ClientContext(\n        userId = input.getRequiredUserId,\n        deviceId = input.clientContext.deviceId,\n        userAgent = input.clientContext.userAgent,\n        countryCode = input.clientContext.countryCode,\n        languageCode = input.clientContext.languageCode,\n      ),\n      displayLocation = input.params(displayLocationParam),\n      supportedLayouts = input.params(supportedLayoutsParam),\n      layoutVersion = input.params(layoutVersionParam),\n      excludedUserIds =\n        excludedUserIdsFeature.flatMap(feature => input.features.map(_.get(feature))),\n      includePromoted = Some(true),\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowClientEventDetailsBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.bijection.Base64String\nimport com.twitter.bijection.{Injection => Serializer}\nimport com.twitter.hermit.internal.thriftscala.HermitTrackingToken\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseClientEventDetailsBuilder\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails\nimport com.twitter.servo.cache.ThriftSerializer\nimport com.twitter.suggests.controller_data.thriftscala.ControllerData\nimport com.twitter.util.Try\nimport org.apache.thrift.protocol.TBinaryProtocol\n\nobject WhoToFollowClientEventDetailsBuilder {\n\n  val InjectionType = \"WhoToFollow\"\n\n  private implicit val ByteSerializer: Serializer[ControllerData, Array[Byte]] =\n    BinaryScalaCodec(ControllerData)\n\n  private val TrackingTokenSerializer =\n    new ThriftSerializer[HermitTrackingToken](HermitTrackingToken, new TBinaryProtocol.Factory())\n\n  val ControllerDataSerializer: Serializer[ControllerData, String] =\n    Serializer.connect[ControllerData, Array[Byte], Base64String, String]\n\n  def deserializeTrackingToken(token: Option[String]): Option[HermitTrackingToken] =\n    token.flatMap(t => Try(TrackingTokenSerializer.fromString(t)).toOption)\n\n  def serializeControllerData(cd: ControllerData): String = ControllerDataSerializer(cd)\n}\n\ncase class WhoToFollowClientEventDetailsBuilder[-Query <: PipelineQuery](\n  trackingTokenFeature: Feature[_, Option[String]],\n) extends BaseClientEventDetailsBuilder[Query, UserCandidate] {\n\n  override def apply(\n    query: Query,\n    candidate: UserCandidate,\n    candidateFeatures: FeatureMap\n  ): Option[ClientEventDetails] = {\n    val serializedTrackingToken = candidateFeatures.getOrElse(trackingTokenFeature, None)\n\n    val controllerData = WhoToFollowClientEventDetailsBuilder\n      .deserializeTrackingToken(serializedTrackingToken)\n      .flatMap(_.controllerData)\n      .map(WhoToFollowClientEventDetailsBuilder.serializeControllerData)\n\n    Some(\n      ClientEventDetails(\n        conversationDetails = None,\n        timelinesDetails = Some(\n          TimelinesDetails(\n            injectionType = Some(WhoToFollowClientEventDetailsBuilder.InjectionType),\n            controllerData = controllerData,\n            sourceData = serializedTrackingToken)),\n        articleDetails = None,\n        liveEventDetails = None,\n        commerceDetails = None\n      ))\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowDependentCandidatePipelineConfig.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.peoplediscovery.api.{thriftscala => t}\nimport com.twitter.product_mixer.component_library.candidate_source.people_discovery.PeopleDiscoveryCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelines.configapi.decider.DeciderParam\n\nclass WhoToFollowDependentCandidatePipelineConfig[Query <: PipelineQuery](\n  override val identifier: CandidatePipelineIdentifier,\n  override val enabledDeciderParam: Option[DeciderParam[Boolean]],\n  override val supportedClientParam: Option[FSParam[Boolean]],\n  override val alerts: Seq[Alert],\n  override val gates: Seq[BaseGate[Query]],\n  whoToFollowCandidateSource: PeopleDiscoveryCandidateSource,\n  override val filters: Seq[Filter[Query, UserCandidate]],\n  moduleDisplayTypeBuilder: BaseModuleDisplayTypeBuilder[Query, UserCandidate],\n  feedbackActionInfoBuilder: Option[\n    BaseFeedbackActionInfoBuilder[PipelineQuery, UserCandidate]\n  ],\n  displayLocationParam: Param[String],\n  supportedLayoutsParam: Param[Seq[String]],\n  layoutVersionParam: Param[Int],\n  excludedUserIdsFeature: Option[Feature[PipelineQuery, Seq[Long]]])\n    extends DependentCandidatePipelineConfig[\n      Query,\n      t.GetModuleRequest,\n      t.RecommendedUser,\n      UserCandidate\n    ] {\n\n  override val candidateSource: BaseCandidateSource[t.GetModuleRequest, t.RecommendedUser] =\n    whoToFollowCandidateSource\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    PipelineQuery,\n    t.GetModuleRequest\n  ] = WhoToFollowCandidatePipelineQueryTransformer(\n    displayLocationParam = displayLocationParam,\n    supportedLayoutsParam = supportedLayoutsParam,\n    layoutVersionParam = layoutVersionParam,\n    excludedUserIdsFeature = excludedUserIdsFeature\n  )\n\n  override val featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[t.RecommendedUser]\n  ] = Seq(WhoToFollowResponseFeatureTransformer)\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[\n    t.RecommendedUser,\n    UserCandidate\n  ] = { user => UserCandidate(user.userId) }\n\n  override val decorator: Option[CandidateDecorator[Query, UserCandidate]] =\n    Some(\n      WhoToFollowCandidateDecorator(\n        moduleDisplayTypeBuilder,\n        feedbackActionInfoBuilder\n      ))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowDependentCandidatePipelineConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.product_mixer.component_library.candidate_source.people_discovery.PeopleDiscoveryCandidateSource\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata.BaseFeedbackActionInfoBuilder\nimport com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module.BaseModuleDisplayTypeBuilder\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelines.configapi.decider.DeciderParam\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass WhoToFollowDependentCandidatePipelineConfigBuilder @Inject() (\n  whoToFollowCandidateSource: PeopleDiscoveryCandidateSource) {\n\n  /**\n   * Build a WhoToFollowDependentCandidatePipelineConfig\n   *\n   *\n   * To create a regular CandidatePipelineConfig instead see [[WhoToFollowCandidatePipelineConfigBuilder]].\n   *\n   * @note If injected classes are needed to populate parameters in this method, consider creating a\n   *       ProductWhoToFollowCandidatePipelineConfigBuilder with a single `def build()` method. That\n   *       product-specific builder class can then inject everything it needs (including this class),\n   *       and delegate to this class's build() method within its own build() method.\n   */\n  def build[Query <: PipelineQuery](\n    moduleDisplayTypeBuilder: BaseModuleDisplayTypeBuilder[Query, UserCandidate],\n    identifier: CandidatePipelineIdentifier = WhoToFollowCandidatePipelineConfig.identifier,\n    enabledDeciderParam: Option[DeciderParam[Boolean]] = None,\n    supportedClientParam: Option[FSParam[Boolean]] = None,\n    alerts: Seq[Alert] = Seq.empty,\n    gates: Seq[BaseGate[Query]] = Seq.empty,\n    filters: Seq[Filter[Query, UserCandidate]] = Seq.empty,\n    feedbackActionInfoBuilder: Option[BaseFeedbackActionInfoBuilder[\n      PipelineQuery,\n      UserCandidate\n    ]] = None,\n    displayLocationParam: Param[String] =\n      StaticParam(WhoToFollowCandidatePipelineQueryTransformer.DisplayLocation),\n    supportedLayoutsParam: Param[Seq[String]] =\n      StaticParam(WhoToFollowCandidatePipelineQueryTransformer.SupportedLayouts),\n    layoutVersionParam: Param[Int] =\n      StaticParam(WhoToFollowCandidatePipelineQueryTransformer.LayoutVersion),\n    excludedUserIdsFeature: Option[Feature[PipelineQuery, Seq[Long]]] = None,\n  ): WhoToFollowDependentCandidatePipelineConfig[Query] =\n    new WhoToFollowDependentCandidatePipelineConfig(\n      identifier = identifier,\n      enabledDeciderParam = enabledDeciderParam,\n      supportedClientParam = supportedClientParam,\n      alerts = alerts,\n      gates = gates,\n      whoToFollowCandidateSource = whoToFollowCandidateSource,\n      filters = filters,\n      moduleDisplayTypeBuilder = moduleDisplayTypeBuilder,\n      feedbackActionInfoBuilder = feedbackActionInfoBuilder,\n      displayLocationParam = displayLocationParam,\n      supportedLayoutsParam = supportedLayoutsParam,\n      layoutVersionParam = layoutVersionParam,\n      excludedUserIdsFeature = excludedUserIdsFeature\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/pipeline/candidate/who_to_follow_module/WhoToFollowResponseFeatureTransformer.scala",
    "content": "package com.twitter.product_mixer.component_library.pipeline.candidate.who_to_follow_module\n\nimport com.twitter.adserver.{thriftscala => ad}\nimport com.twitter.hermit.{thriftscala => h}\nimport com.twitter.peoplediscovery.api.{thriftscala => t}\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\nobject AdImpressionFeature extends Feature[UserCandidate, Option[ad.AdImpression]]\nobject HermitContextTypeFeature extends Feature[UserCandidate, Option[h.ContextType]]\nobject SocialTextFeature extends Feature[UserCandidate, Option[String]]\nobject TrackingTokenFeature extends Feature[UserCandidate, Option[String]]\nobject ScoreFeature extends Feature[UserCandidate, Option[Double]]\n\nobject WhoToFollowResponseFeatureTransformer\n    extends CandidateFeatureTransformer[t.RecommendedUser] {\n\n  override val identifier: TransformerIdentifier = TransformerIdentifier(\"WhoToFollowResponse\")\n\n  override val features: Set[Feature[_, _]] =\n    Set(\n      AdImpressionFeature,\n      HermitContextTypeFeature,\n      SocialTextFeature,\n      TrackingTokenFeature,\n      ScoreFeature)\n\n  override def transform(input: t.RecommendedUser): FeatureMap = FeatureMapBuilder()\n    .add(AdImpressionFeature, input.adImpression)\n    .add(HermitContextTypeFeature, input.reason.flatMap(_.contextType))\n    .add(SocialTextFeature, input.socialText)\n    .add(TrackingTokenFeature, input.trackingToken)\n    .add(ScoreFeature, input.mlPredictionScore)\n    .build()\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/cursor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/component-library/src/main/thrift/com/twitter/product_mixer/component_library:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"scrooge/scrooge-serializer/src/main/scala\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/component-library/src/main/thrift/com/twitter/product_mixer/component_library:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"scrooge/scrooge-serializer/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/cursor/CursorSerializer.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.cursor\n\nimport com.twitter.product_mixer.component_library.model.cursor.OrderedCursor\nimport com.twitter.product_mixer.component_library.model.cursor.PassThroughCursor\nimport com.twitter.product_mixer.component_library.model.cursor.UnorderedBloomFilterCursor\nimport com.twitter.product_mixer.component_library.model.cursor.UnorderedExcludeIdsCursor\nimport com.twitter.product_mixer.component_library.{thriftscala => t}\nimport com.twitter.product_mixer.core.pipeline.PipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MalformedCursor\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.scrooge.BinaryThriftStructSerializer\nimport com.twitter.scrooge.ThriftStructCodec\nimport com.twitter.search.common.util.bloomfilter.AdaptiveLongIntBloomFilterSerializer\nimport com.twitter.util.Base64UrlSafeStringEncoder\nimport com.twitter.util.StringEncoder\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.slice.CursorTypeMarshaller\n\n/**\n * Handles serialization and deserialization for all supported generic cursors. Note that generic\n * cursors may be used for Slices or any bespoke marshalling format.\n */\nobject CursorSerializer extends PipelineCursorSerializer[PipelineCursor] {\n\n  private[cursor] val CursorThriftSerializer: BinaryThriftStructSerializer[\n    t.ProductMixerRequestCursor\n  ] =\n    new BinaryThriftStructSerializer[t.ProductMixerRequestCursor] {\n      override def codec: ThriftStructCodec[t.ProductMixerRequestCursor] =\n        t.ProductMixerRequestCursor\n      override def encoder: StringEncoder = Base64UrlSafeStringEncoder\n    }\n\n  override def serializeCursor(cursor: PipelineCursor): String =\n    cursor match {\n      case OrderedCursor(id, cursorType, gapBoundaryId) =>\n        val cursorTypeMarshaller = new CursorTypeMarshaller()\n        val thriftCursor = t.ProductMixerRequestCursor.OrderedCursor(\n          t.OrderedCursor(\n            id = id,\n            cursorType = cursorType.map(cursorTypeMarshaller.apply),\n            gapBoundaryId))\n\n        CursorThriftSerializer.toString(thriftCursor)\n      case UnorderedExcludeIdsCursor(excludedIds) =>\n        val thriftCursor = t.ProductMixerRequestCursor.UnorderedExcludeIdsCursor(\n          t.UnorderedExcludeIdsCursor(excludedIds = Some(excludedIds)))\n\n        CursorThriftSerializer.toString(thriftCursor)\n      case UnorderedBloomFilterCursor(longIntBloomFilter) =>\n        val thriftCursor = t.ProductMixerRequestCursor.UnorderedBloomFilterCursor(\n          t.UnorderedBloomFilterCursor(\n            serializedLongIntBloomFilter =\n              AdaptiveLongIntBloomFilterSerializer.serialize(longIntBloomFilter)\n          ))\n\n        CursorThriftSerializer.toString(thriftCursor)\n      case PassThroughCursor(cursorValue, cursorType) =>\n        val cursorTypeMarshaller = new CursorTypeMarshaller()\n        val thriftCursor = t.ProductMixerRequestCursor.PassThroughCursor(\n          t.PassThroughCursor(\n            cursorValue = cursorValue,\n            cursorType = cursorType.map(cursorTypeMarshaller.apply)\n          ))\n\n        CursorThriftSerializer.toString(thriftCursor)\n      case _ =>\n        throw PipelineFailure(IllegalStateFailure, \"Unknown cursor type\")\n    }\n\n  def deserializeOrderedCursor(cursorString: String): Option[OrderedCursor] =\n    deserializeCursor(\n      cursorString,\n      {\n        case Some(\n              t.ProductMixerRequestCursor\n                .OrderedCursor(t.OrderedCursor(id, cursorType, gapBoundaryId))) =>\n          val cursorTypeMarshaller = new CursorTypeMarshaller()\n          Some(\n            OrderedCursor(\n              id = id,\n              cursorType = cursorType.map(cursorTypeMarshaller.unmarshall),\n              gapBoundaryId))\n      }\n    )\n\n  def deserializeUnorderedExcludeIdsCursor(\n    cursorString: String\n  ): Option[UnorderedExcludeIdsCursor] = {\n    deserializeCursor(\n      cursorString,\n      {\n        case Some(\n              t.ProductMixerRequestCursor\n                .UnorderedExcludeIdsCursor(t.UnorderedExcludeIdsCursor(excludedIdsOpt))) =>\n          Some(UnorderedExcludeIdsCursor(excludedIds = excludedIdsOpt.getOrElse(Seq.empty)))\n      }\n    )\n  }\n\n  def deserializeUnorderedBloomFilterCursor(\n    cursorString: String\n  ): Option[UnorderedBloomFilterCursor] =\n    deserializeCursor(\n      cursorString,\n      {\n        case Some(\n              t.ProductMixerRequestCursor.UnorderedBloomFilterCursor(\n                t.UnorderedBloomFilterCursor(serializedLongIntBloomFilter))) =>\n          val bloomFilter = AdaptiveLongIntBloomFilterSerializer\n            .deserialize(serializedLongIntBloomFilter).getOrElse(\n              throw PipelineFailure(\n                MalformedCursor,\n                s\"Failed to deserialize UnorderedBloomFilterCursor from cursor string: $cursorString\")\n            )\n\n          Some(UnorderedBloomFilterCursor(longIntBloomFilter = bloomFilter))\n      }\n    )\n\n  def deserializePassThroughCursor(cursorString: String): Option[PassThroughCursor] =\n    deserializeCursor(\n      cursorString,\n      {\n        case Some(\n              t.ProductMixerRequestCursor\n                .PassThroughCursor(t.PassThroughCursor(cursorValue, cursorType))) =>\n          val cursorTypeMarshaller = new CursorTypeMarshaller()\n          Some(\n            PassThroughCursor(\n              cursorValue = cursorValue,\n              cursorType = cursorType.map(cursorTypeMarshaller.unmarshall)))\n      }\n    )\n\n  // Note that the \"A\" type of the PartialFunction cannot be inferred due to the thrift type not\n  // being present on the PipelineCursorSerializer trait. By using this private def with the\n  // deserializePf type declared, it can be inferred.\n  private def deserializeCursor[Cursor <: PipelineCursor](\n    cursorString: String,\n    deserializePf: PartialFunction[Option[t.ProductMixerRequestCursor], Option[Cursor]]\n  ): Option[Cursor] =\n    PipelineCursorSerializer.deserializeCursor(\n      cursorString,\n      CursorThriftSerializer,\n      deserializePf\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/cursor/UrtCursorSerializer.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.cursor\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.component_library.model.cursor.UrtPassThroughCursor\nimport com.twitter.product_mixer.component_library.model.cursor.UrtPlaceholderCursor\nimport com.twitter.product_mixer.component_library.model.cursor.UrtUnorderedBloomFilterCursor\nimport com.twitter.product_mixer.component_library.model.cursor.UrtUnorderedExcludeIdsCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.CursorSerializer.CursorThriftSerializer\nimport com.twitter.product_mixer.component_library.{thriftscala => t}\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer.deserializeCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.product_mixer.core.pipeline.UrtPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MalformedCursor\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.search.common.util.bloomfilter.AdaptiveLongIntBloomFilterSerializer\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.operation.CursorTypeMarshaller\n\n/**\n * Handles serialization and deserialization for all supported URT cursors\n */\nobject UrtCursorSerializer extends PipelineCursorSerializer[UrtPipelineCursor] {\n\n  val SerializedUrtPlaceholderCursor = CursorThriftSerializer.toString(\n    t.ProductMixerRequestCursor.UrtPlaceholderCursor(t.UrtPlaceholderCursor()))\n\n  val cursorTypeMarshaller = new CursorTypeMarshaller()\n\n  override def serializeCursor(cursor: UrtPipelineCursor): String =\n    cursor match {\n      case UrtOrderedCursor(initialSortIndex, id, cursorType, gapBoundaryId) =>\n        val thriftCursor = t.ProductMixerRequestCursor.UrtOrderedCursor(\n          t.UrtOrderedCursor(\n            initialSortIndex = initialSortIndex,\n            id = id,\n            cursorType.map(cursorTypeMarshaller.apply),\n            gapBoundaryId = gapBoundaryId))\n\n        CursorThriftSerializer.toString(thriftCursor)\n      case UrtUnorderedExcludeIdsCursor(initialSortIndex, excludedIds) =>\n        val thriftCursor = t.ProductMixerRequestCursor.UrtUnorderedExcludeIdsCursor(\n          t.UrtUnorderedExcludeIdsCursor(\n            initialSortIndex = initialSortIndex,\n            excludedIds = Some(excludedIds)))\n\n        CursorThriftSerializer.toString(thriftCursor)\n      case UrtUnorderedBloomFilterCursor(initialSortIndex, longIntBloomFilter) =>\n        val thriftCursor = t.ProductMixerRequestCursor.UrtUnorderedBloomFilterCursor(\n          t.UrtUnorderedBloomFilterCursor(\n            initialSortIndex = initialSortIndex,\n            serializedLongIntBloomFilter =\n              AdaptiveLongIntBloomFilterSerializer.serialize(longIntBloomFilter)\n          ))\n\n        CursorThriftSerializer.toString(thriftCursor)\n      case UrtPassThroughCursor(initialSortIndex, cursorValue, cursorType) =>\n        val thriftCursor = t.ProductMixerRequestCursor.UrtPassThroughCursor(\n          t.UrtPassThroughCursor(\n            initialSortIndex = initialSortIndex,\n            cursorValue = cursorValue,\n            cursorType = cursorType.map(cursorTypeMarshaller.apply)\n          ))\n\n        CursorThriftSerializer.toString(thriftCursor)\n      case UrtPlaceholderCursor() =>\n        SerializedUrtPlaceholderCursor\n      case _ =>\n        throw PipelineFailure(IllegalStateFailure, \"Unknown cursor type\")\n    }\n\n  def deserializeOrderedCursor(cursorString: String): Option[UrtOrderedCursor] = {\n    deserializeUrtCursor(\n      cursorString,\n      {\n        case Some(\n              t.ProductMixerRequestCursor.UrtOrderedCursor(\n                t.UrtOrderedCursor(initialSortIndex, id, cursorType, gapBoundaryId))) =>\n          Some(\n            UrtOrderedCursor(\n              initialSortIndex = initialSortIndex,\n              id = id,\n              cursorType = cursorType.map(cursorTypeMarshaller.unmarshall),\n              gapBoundaryId))\n      }\n    )\n  }\n\n  def deserializeUnorderedExcludeIdsCursor(\n    cursorString: String\n  ): Option[UrtUnorderedExcludeIdsCursor] = {\n    deserializeUrtCursor(\n      cursorString,\n      {\n        case Some(\n              t.ProductMixerRequestCursor.UrtUnorderedExcludeIdsCursor(\n                t.UrtUnorderedExcludeIdsCursor(initialSortIndex, excludedIdsOpt))) =>\n          Some(\n            UrtUnorderedExcludeIdsCursor(\n              initialSortIndex = initialSortIndex,\n              excludedIds = excludedIdsOpt.getOrElse(Seq.empty)))\n      }\n    )\n  }\n\n  def deserializeUnorderedBloomFilterCursor(\n    cursorString: String\n  ): Option[UrtUnorderedBloomFilterCursor] = {\n    deserializeUrtCursor(\n      cursorString,\n      {\n        case Some(\n              t.ProductMixerRequestCursor.UrtUnorderedBloomFilterCursor(\n                t.UrtUnorderedBloomFilterCursor(initialSortIndex, serializedLongIntBloomFilter))) =>\n          val longIntBloomFilter = AdaptiveLongIntBloomFilterSerializer\n            .deserialize(serializedLongIntBloomFilter).getOrElse(\n              throw PipelineFailure(\n                MalformedCursor,\n                s\"Failed to deserialize UrtUnorderedBloomFilterCursor from cursor string: $cursorString\")\n            )\n\n          Some(\n            UrtUnorderedBloomFilterCursor(\n              initialSortIndex = initialSortIndex,\n              longIntBloomFilter = longIntBloomFilter))\n      }\n    )\n  }\n\n  def deserializePassThroughCursor(cursorString: String): Option[UrtPassThroughCursor] = {\n    deserializeUrtCursor(\n      cursorString,\n      {\n        case Some(\n              t.ProductMixerRequestCursor\n                .UrtPassThroughCursor(\n                  t.UrtPassThroughCursor(initialSortIndex, cursorValue, cursorType))) =>\n          Some(\n            UrtPassThroughCursor(\n              initialSortIndex = initialSortIndex,\n              cursorValue = cursorValue,\n              cursorType = cursorType.map(cursorTypeMarshaller.unmarshall)))\n      }\n    )\n  }\n\n  private def deserializeUrtCursor[Cursor <: PipelineCursor](\n    cursorString: String,\n    deserializePf: PartialFunction[Option[t.ProductMixerRequestCursor], Option[Cursor]]\n  ): Option[Cursor] = {\n    deserializeCursor[t.ProductMixerRequestCursor, Cursor](\n      cursorString,\n      CursorThriftSerializer,\n      deserializePf orElse {\n        case Some(t.ProductMixerRequestCursor.UrtPlaceholderCursor(t.UrtPlaceholderCursor())) =>\n          // Treat submitted placeholder cursor like an initial page load\n          None\n      },\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/hubble\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/suggestion\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/hubble\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/suggestion\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/SliceDomainMarshaller.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.slice\n\nimport com.twitter.product_mixer.component_library.model.candidate._\nimport com.twitter.product_mixer.component_library.model.candidate.hubble.AdCreativeCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.hubble.AdGroupCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.hubble.AdUnitCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.hubble.CampaignCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.hubble.FundingSourceCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.suggestion.QuerySuggestionCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.suggestion.TypeaheadEventCandidate\nimport com.twitter.product_mixer.component_library.premarshaller.slice.builder.SliceBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.slice.builder.SliceCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.slice.builder.SliceCursorUpdater\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.UndecoratedCandidateDomainMarshallerException\nimport com.twitter.product_mixer.core.functional_component.premarshaller.UnsupportedCandidateDomainMarshallerException\nimport com.twitter.product_mixer.core.functional_component.premarshaller.UnsupportedModuleDomainMarshallerException\nimport com.twitter.product_mixer.core.functional_component.premarshaller.UnsupportedPresentationDomainMarshallerException\nimport com.twitter.product_mixer.core.model.common.identifier.DomainMarshallerIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.slice.BaseSliceItemPresentation\nimport com.twitter.product_mixer.core.model.marshalling.response.slice._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Domain marshaller that generates Slices automatically for most candidates but a different\n * presentation can be provided by decorators that implement [[BaseSliceItemPresentation]]. This will\n * only be necessary in the rare case that a candidate contains more than an id. For example,\n * cursors require a value/type rather than an id.\n */\ncase class SliceDomainMarshaller[-Query <: PipelineQuery](\n  override val cursorBuilders: Seq[SliceCursorBuilder[Query]] = Seq.empty,\n  override val cursorUpdaters: Seq[SliceCursorUpdater[Query]] = Seq.empty,\n  override val identifier: DomainMarshallerIdentifier = DomainMarshallerIdentifier(\"Slice\"))\n    extends DomainMarshaller[Query, Slice]\n    with SliceBuilder[Query] {\n\n  override def apply(\n    query: Query,\n    selections: Seq[CandidateWithDetails]\n  ): Slice = {\n    val entries = selections.map {\n      case ItemCandidateWithDetails(_, Some(presentation: BaseSliceItemPresentation), _) =>\n        presentation.sliceItem\n      case candidateWithDetails @ ItemCandidateWithDetails(candidate, None, _) =>\n        val source = candidateWithDetails.source\n        candidate match {\n          case candidate: BaseTopicCandidate => TopicItem(candidate.id)\n          case candidate: BaseTweetCandidate => TweetItem(candidate.id)\n          case candidate: BaseUserCandidate => UserItem(candidate.id)\n          case candidate: TwitterListCandidate => TwitterListItem(candidate.id)\n          case candidate: DMConvoSearchCandidate =>\n            DMConvoSearchItem(candidate.id, candidate.lastReadableEventId)\n          case candidate: DMEventCandidate =>\n            DMEventItem(candidate.id)\n          case candidate: DMConvoCandidate =>\n            DMConvoItem(candidate.id, candidate.lastReadableEventId)\n          case candidate: DMMessageSearchCandidate => DMMessageSearchItem(candidate.id)\n          case candidate: QuerySuggestionCandidate =>\n            TypeaheadQuerySuggestionItem(candidate.id, candidate.metadata)\n          case candidate: TypeaheadEventCandidate =>\n            TypeaheadEventItem(candidate.id, candidate.metadata)\n          case candidate: AdUnitCandidate =>\n            AdItem(candidate.id, candidate.adAccountId)\n          case candidate: AdCreativeCandidate =>\n            AdCreativeItem(candidate.id, candidate.adType, candidate.adAccountId)\n          case candidate: AdGroupCandidate =>\n            AdGroupItem(candidate.id, candidate.adAccountId)\n          case candidate: CampaignCandidate =>\n            CampaignItem(candidate.id, candidate.adAccountId)\n          case candidate: FundingSourceCandidate =>\n            FundingSourceItem(candidate.id, candidate.adAccountId)\n          case candidate: CursorCandidate =>\n            // Cursors must contain a cursor type which is defined by the presentation. As a result,\n            // cursors are expected to be handled by the Some(presentation) case above, and must not\n            // fall into this case.\n            throw new UndecoratedCandidateDomainMarshallerException(candidate, source)\n          case candidate =>\n            throw new UnsupportedCandidateDomainMarshallerException(candidate, source)\n        }\n      case itemCandidateWithDetails @ ItemCandidateWithDetails(candidate, Some(presentation), _) =>\n        throw new UnsupportedPresentationDomainMarshallerException(\n          candidate,\n          presentation,\n          itemCandidateWithDetails.source)\n      case moduleCandidateWithDetails @ ModuleCandidateWithDetails(_, presentation, _) =>\n        throw new UnsupportedModuleDomainMarshallerException(\n          presentation,\n          moduleCandidateWithDetails.source)\n    }\n\n    buildSlice(query, entries)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/builder/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/builder/OrderedNextCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.slice.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.OrderedCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.CursorSerializer\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorType\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.NextCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.SliceItem\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Builds [[OrderedCursor]] in the Next position\n *\n * @param idSelector Specifies the entry from which to derive the `id` field\n * @param includeOperation Specifies whether to include the builder operation in the response\n * @param serializer Converts the cursor to an encoded string\n */\ncase class OrderedNextCursorBuilder[Query <: PipelineQuery with HasPipelineCursor[OrderedCursor]](\n  idSelector: PartialFunction[SliceItem, Long],\n  override val includeOperation: ShouldInclude[Query] = AlwaysInclude,\n  serializer: PipelineCursorSerializer[OrderedCursor] = CursorSerializer)\n    extends SliceCursorBuilder[Query] {\n  override val cursorType: CursorType = NextCursor\n\n  override def cursorValue(\n    query: Query,\n    entries: Seq[SliceItem]\n  ): String = {\n    val bottomId = entries.reverseIterator.collectFirst(idSelector)\n\n    val id = bottomId.orElse(query.pipelineCursor.flatMap(_.id))\n\n    val cursor = OrderedCursor(id = id, cursorType = Some(cursorType))\n\n    serializer.serializeCursor(cursor)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/builder/OrderedNextCursorUpdater.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.slice.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.OrderedCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.CursorSerializer\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorType\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.NextCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.SliceItem\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Updates an [[OrderedCursor]] in the Next position\n *\n * @param idSelector Specifies the entry from which to derive the `id` field\n * @param includeOperation Specifies whether to include the builder operation in the response\n * @param serializer Converts the cursor to an encoded string\n */\ncase class OrderedNextCursorUpdater[Query <: PipelineQuery with HasPipelineCursor[OrderedCursor]](\n  idSelector: PartialFunction[SliceItem, Long],\n  override val includeOperation: ShouldInclude[Query] = AlwaysInclude,\n  serializer: PipelineCursorSerializer[OrderedCursor] = CursorSerializer)\n    extends SliceCursorUpdaterFromUnderlyingBuilder[Query] {\n  override val cursorType: CursorType = NextCursor\n\n  override val underlying: OrderedNextCursorBuilder[Query] =\n    OrderedNextCursorBuilder(idSelector, includeOperation, serializer)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/builder/OrderedPreviousCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.slice.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.OrderedCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.CursorSerializer\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorType\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.PreviousCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.SliceItem\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Builds [[OrderedCursor]] in the Previous position\n *\n * @param idSelector Specifies the entry from which to derive the `id` field\n * @param includeOperation Specifies whether to include the builder operation in the response\n * @param serializer Converts the cursor to an encoded string\n */\ncase class OrderedPreviousCursorBuilder[\n  Query <: PipelineQuery with HasPipelineCursor[OrderedCursor]\n](\n  idSelector: PartialFunction[SliceItem, Long],\n  override val includeOperation: ShouldInclude[Query] = AlwaysInclude,\n  serializer: PipelineCursorSerializer[OrderedCursor] = CursorSerializer)\n    extends SliceCursorBuilder[Query] {\n  override val cursorType: CursorType = PreviousCursor\n\n  override def cursorValue(\n    query: Query,\n    entries: Seq[SliceItem]\n  ): String = {\n    val topId = entries.collectFirst(idSelector)\n\n    val id = topId.orElse(query.pipelineCursor.flatMap(_.id))\n\n    val cursor = OrderedCursor(id = id, cursorType = Some(cursorType))\n\n    serializer.serializeCursor(cursor)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/builder/OrderedPreviousCursorUpdater.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.slice.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.OrderedCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.CursorSerializer\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorType\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.PreviousCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.SliceItem\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Updates an [[OrderedCursor]] in the Previous position\n *\n * @param idSelector Specifies the entry from which to derive the `id` field\n * @param includeOperation Specifies whether to include the builder operation in the response\n * @param serializer Converts the cursor to an encoded string\n */\ncase class OrderedPreviousCursorUpdater[\n  Query <: PipelineQuery with HasPipelineCursor[OrderedCursor]\n](\n  idSelector: PartialFunction[SliceItem, Long],\n  override val includeOperation: ShouldInclude[Query] = AlwaysInclude,\n  serializer: PipelineCursorSerializer[OrderedCursor] = CursorSerializer)\n    extends SliceCursorUpdaterFromUnderlyingBuilder[Query] {\n  override val cursorType: CursorType = PreviousCursor\n\n  override val underlying: OrderedPreviousCursorBuilder[Query] =\n    OrderedPreviousCursorBuilder(idSelector, includeOperation, serializer)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/builder/ShouldInclude.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.slice.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.SliceItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait ShouldInclude[-Query <: PipelineQuery] {\n  def apply(query: Query, items: Seq[SliceItem]): Boolean\n}\n\nobject AlwaysInclude extends ShouldInclude[PipelineQuery] {\n  override def apply(query: PipelineQuery, entries: Seq[SliceItem]): Boolean = true\n}\n\nobject IncludeOnNonEmpty extends ShouldInclude[PipelineQuery] {\n  override def apply(query: PipelineQuery, entries: Seq[SliceItem]): Boolean = entries.nonEmpty\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/builder/SliceBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.slice.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorItem\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.NextCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.GapCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.PreviousCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.Slice\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.SliceInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.SliceItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.UnexpectedCandidateInMarshaller\n\ntrait SliceBuilder[-Query <: PipelineQuery] {\n  def cursorBuilders: Seq[SliceCursorBuilder[Query]]\n  def cursorUpdaters: Seq[SliceCursorUpdater[Query]]\n\n  private def containsGapCursor(items: Seq[SliceItem]): Boolean =\n    items.collectFirst { case CursorItem(_, GapCursor) => () }.nonEmpty\n\n  final def buildSlice(query: Query, items: Seq[SliceItem]): Slice = {\n    val builtCursors = cursorBuilders.flatMap(_.build(query, items))\n\n    // Iterate over the cursorUpdaters in the order they were defined. Note that each updater will\n    // be passed the items updated by the previous cursorUpdater.\n    val updatedItems = cursorUpdaters.foldLeft(items) { (items, cursorUpdater) =>\n      cursorUpdater.update(query, items)\n    } ++ builtCursors\n\n    val (cursors, nonCursorItems) = updatedItems.partition(_.isInstanceOf[CursorItem])\n    val nextCursor = cursors.collectFirst {\n      case cursor @ CursorItem(_, NextCursor) => cursor.value\n    }\n    val previousCursor = cursors.collectFirst {\n      case cursor @ CursorItem(_, PreviousCursor) => cursor.value\n    }\n\n    /**\n     * Identify whether a [[GapCursor]] is present and give as much detail to point to where it came from\n     * Since this is already a fatal error case for the request, its okay to be a little expensive to get\n     * the best error message possible for debug purposes.\n     */\n    if (containsGapCursor(cursors)) {\n      val errorDetails =\n        if (containsGapCursor(builtCursors)) {\n          \"This means one of your `cursorBuilders` returned a GapCursor.\"\n        } else if (containsGapCursor(items)) {\n          \"This means one of your `CandidateDecorator`s decorated a Candidate with a GapCursor.\"\n        } else {\n          \"This means one of your `cursorUpdaters` returned a GapCursor.\"\n        }\n      throw PipelineFailure(\n        UnexpectedCandidateInMarshaller,\n        s\"SliceBuilder does not support GapCursors but one was given. $errorDetails\"\n      )\n    }\n\n    Slice(\n      items = nonCursorItems,\n      sliceInfo = SliceInfo(previousCursor = previousCursor, nextCursor = nextCursor))\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/builder/SliceCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.slice.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorItem\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorType\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.SliceItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait SliceCursorBuilder[-Query <: PipelineQuery] {\n\n  val includeOperation: ShouldInclude[Query] = AlwaysInclude\n\n  def cursorValue(query: Query, items: Seq[SliceItem]): String\n  def cursorType: CursorType\n\n  def build(query: Query, entries: Seq[SliceItem]): Option[CursorItem] = {\n    if (includeOperation(query, entries)) {\n      Some(\n        CursorItem(\n          cursorType = cursorType,\n          value = cursorValue(query, entries)\n        ))\n    } else None\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/slice/builder/SliceCursorUpdater.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.slice.builder\n\nimport com.twitter.product_mixer.component_library.premarshaller.slice.builder.SliceCursorUpdater.getCursorByType\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorItem\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorType\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.SliceItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject SliceCursorUpdater {\n\n  def getCursorByType(\n    items: Seq[SliceItem],\n    cursorType: CursorType\n  ): Option[CursorItem] = {\n    items.collectFirst {\n      case cursor: CursorItem if cursor.cursorType == cursorType => cursor\n    }\n  }\n}\n\n/**\n * If [[SliceCursorBuilder.includeOperation]] is true and a cursor does exist in the `items`,\n * this will run the the underlying [[SliceCursorBuilder]] with the full `items`\n * (including all cursors which may be present) then filter out only the originally\n * found [[CursorItem]] from the results). Then append the new cursor to the end of the results.\n *\n * If you have multiple cursors that need to be updated, you will need to have multiple updaters.\n *\n * If a CursorCandidate is returned by a Candidate Source, use this trait to update the Cursor\n * (if necessary) and add it to the end of the candidates list.\n */\ntrait SliceCursorUpdater[-Query <: PipelineQuery] extends SliceCursorBuilder[Query] { self =>\n\n  def getExistingCursor(items: Seq[SliceItem]): Option[CursorItem] = {\n    getCursorByType(items, self.cursorType)\n  }\n\n  def update(query: Query, items: Seq[SliceItem]): Seq[SliceItem] = {\n    if (includeOperation(query, items)) {\n      getExistingCursor(items)\n        .map { existingCursor =>\n          // Safe get because includeOperation() is shared in this context\n          val newCursor = build(query, items).get\n\n          items.filterNot(_ == existingCursor) :+ newCursor\n        }.getOrElse(items)\n    } else items\n  }\n}\n\ntrait SliceCursorUpdaterFromUnderlyingBuilder[-Query <: PipelineQuery]\n    extends SliceCursorUpdater[Query] {\n  def underlying: SliceCursorBuilder[Query]\n  override def cursorValue(\n    query: Query,\n    entries: Seq[SliceItem]\n  ): String = underlying.cursorValue(query, entries)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urp/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urp/builder\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urp/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urp/UrpDomainMarshaller.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urp\n\nimport com.twitter.product_mixer.component_library.premarshaller.urp.builder.PageBodyBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urp.builder.PageHeaderBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urp.builder.PageNavBarBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urp.builder.TimelineScribeConfigBuilder\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.model.common.identifier.DomainMarshallerIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urp._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject UrpDomainMarshaller {\n  val PageIdSuffix = \"-Page\"\n}\n\n/**\n * Domain marshaller that given the builders for the body, header and navbar will generate a URP Page\n *\n * @param pageBodyBuilder     PageBody builder that generates a PageBody with the query and selections\n * @param scribeConfigBuilder Scribe Config builder that generates the configuration for scribing of the page\n * @param pageHeaderBuilder   PageHeader builder that generates a PageHeader with the query and selections\n * @param pageNavBarBuilder   PageNavBar builder that generates a PageNavBar with the query and selections\n * @tparam Query The type of Query that this Marshaller operates with\n */\ncase class UrpDomainMarshaller[-Query <: PipelineQuery](\n  pageBodyBuilder: PageBodyBuilder[Query],\n  pageHeaderBuilder: Option[PageHeaderBuilder[Query]] = None,\n  pageNavBarBuilder: Option[PageNavBarBuilder[Query]] = None,\n  scribeConfigBuilder: Option[TimelineScribeConfigBuilder[Query]] = None,\n  override val identifier: DomainMarshallerIdentifier =\n    DomainMarshallerIdentifier(\"UnifiedRichPage\"))\n    extends DomainMarshaller[Query, Page] {\n\n  override def apply(\n    query: Query,\n    selections: Seq[CandidateWithDetails]\n  ): Page = {\n    val pageBody = pageBodyBuilder.build(query, selections)\n    val pageHeader = pageHeaderBuilder.flatMap(_.build(query, selections))\n    val pageNavBar = pageNavBarBuilder.flatMap(_.build(query, selections))\n    val scribeConfig = scribeConfigBuilder.flatMap(_.build(query, pageBody, pageHeader, pageNavBar))\n\n    Page(\n      id = query.product.identifier.toString + UrpDomainMarshaller.PageIdSuffix,\n      pageBody = pageBody,\n      scribeConfig = scribeConfig,\n      pageHeader = pageHeader,\n      pageNavBar = pageNavBar\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urp/builder/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stringcenter/client\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stringcenter/client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urp/builder/PageBodyBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urp.builder\n\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PageBody\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Trait for our builder which given a query and selections will return a `PageBody`\n *\n * @tparam Query\n */\ntrait PageBodyBuilder[-Query <: PipelineQuery] {\n\n  def build(\n    query: Query,\n    selections: Seq[CandidateWithDetails]\n  ): PageBody\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urp/builder/PageHeaderBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urp.builder\n\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PageHeader\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Trait for our builder which given a query and selections will return an `Option[PageHeader]`\n *\n * @tparam Query\n */\ntrait PageHeaderBuilder[-Query <: PipelineQuery] {\n\n  def build(\n    query: Query,\n    selections: Seq[CandidateWithDetails]\n  ): Option[PageHeader]\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urp/builder/PageNavBarBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urp.builder\n\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PageNavBar\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Trait for our builder which given a query and selections will return an `Option[PageNavBar]`\n *\n * @tparam Query\n */\ntrait PageNavBarBuilder[-Query <: PipelineQuery] {\n\n  def build(\n    query: Query,\n    selections: Seq[CandidateWithDetails]\n  ): Option[PageNavBar]\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urp/builder/StaticTimelineScribeConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urp.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PageBody\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PageHeader\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PageNavBar\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class StaticTimelineScribeConfigBuilder(\n  timelineScribeConfig: TimelineScribeConfig)\n    extends TimelineScribeConfigBuilder[PipelineQuery] {\n\n  override def build(\n    query: PipelineQuery,\n    pageBody: PageBody,\n    pageHeader: Option[PageHeader],\n    pageNavBar: Option[PageNavBar]\n  ): Option[TimelineScribeConfig] = Some(timelineScribeConfig)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urp/builder/TimelineScribeConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urp.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PageBody\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PageHeader\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PageNavBar\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Trait for our builder which given a query and page info will return an `Option[TimelineScribeConfig]`\n *\n * @tparam Query\n */\ntrait TimelineScribeConfigBuilder[-Query <: PipelineQuery] {\n\n  def build(\n    query: Query,\n    pageBody: PageBody,\n    pageHeader: Option[PageHeader],\n    pageNavBar: Option[PageNavBar]\n  ): Option[TimelineScribeConfig]\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    scalac_plugins = [\"no-roomba\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/decorator/urt\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/UndecoratedUrtDomainMarshaller.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt\n\nimport com.twitter.product_mixer.component_library.model.candidate.ArticleCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.AudioSpaceCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.TopicCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.TwitterListCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.UserCandidate\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.AddEntriesInstructionBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.BaseUrtMetadataBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtCursorBuilder\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtCursorUpdater\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtInstructionBuilder\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.UnsupportedCandidateDomainMarshallerException\nimport com.twitter.product_mixer.core.functional_component.premarshaller.UnsupportedModuleDomainMarshallerException\nimport com.twitter.product_mixer.core.functional_component.premarshaller.UnsupportedPresentationDomainMarshallerException\nimport com.twitter.product_mixer.core.model.common.identifier.DomainMarshallerIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.ArticleItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.FollowingListSeed\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.audio_space.AudioSpaceItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.Tweet\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.twitter_list.TwitterListItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.User\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.UserItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Decorator that is useful for fast prototyping, as it will generate URT entries from only\n * candidate IDs (no ItemPresentations or ModulePresentations from candidate pipeline decorators\n * are required).\n */\ncase class UndecoratedUrtDomainMarshaller[Query <: PipelineQuery](\n  override val instructionBuilders: Seq[UrtInstructionBuilder[Query, TimelineInstruction]] =\n    Seq(AddEntriesInstructionBuilder()),\n  override val cursorBuilders: Seq[UrtCursorBuilder[Query]] = Seq.empty,\n  override val cursorUpdaters: Seq[UrtCursorUpdater[Query]] = Seq.empty,\n  override val metadataBuilder: Option[BaseUrtMetadataBuilder[Query]] = None,\n  override val sortIndexStep: Int = 1,\n  override val identifier: DomainMarshallerIdentifier =\n    DomainMarshallerIdentifier(\"UndecoratedUnifiedRichTimeline\"))\n    extends DomainMarshaller[Query, Timeline]\n    with UrtBuilder[Query, TimelineInstruction] {\n\n  override def apply(\n    query: Query,\n    selections: Seq[CandidateWithDetails]\n  ): Timeline = {\n    val entries = selections.map {\n      case itemCandidateWithDetails @ ItemCandidateWithDetails(candidate, None, _) =>\n        candidate match {\n          case candidate: ArticleCandidate =>\n            ArticleItem(\n              id = candidate.id,\n              articleSeedType = FollowingListSeed,\n              sortIndex = None,\n              clientEventInfo = None,\n              feedbackActionInfo = None,\n              displayType = None,\n              socialContext = None,\n            )\n          case candidate: AudioSpaceCandidate =>\n            AudioSpaceItem(\n              id = candidate.id,\n              sortIndex = None,\n              clientEventInfo = None,\n              feedbackActionInfo = None)\n          case candidate: TopicCandidate =>\n            TopicItem(\n              id = candidate.id,\n              sortIndex = None,\n              clientEventInfo = None,\n              feedbackActionInfo = None,\n              topicFunctionalityType = None,\n              topicDisplayType = None\n            )\n          case candidate: TweetCandidate =>\n            TweetItem(\n              id = candidate.id,\n              entryNamespace = TweetItem.TweetEntryNamespace,\n              sortIndex = None,\n              clientEventInfo = None,\n              feedbackActionInfo = None,\n              isPinned = None,\n              entryIdToReplace = None,\n              socialContext = None,\n              highlights = None,\n              displayType = Tweet,\n              innerTombstoneInfo = None,\n              timelinesScoreInfo = None,\n              hasModeratedReplies = None,\n              forwardPivot = None,\n              innerForwardPivot = None,\n              promotedMetadata = None,\n              conversationAnnotation = None,\n              contextualTweetRef = None,\n              prerollMetadata = None,\n              replyBadge = None,\n              destination = None\n            )\n          case candidate: TwitterListCandidate =>\n            TwitterListItem(\n              id = candidate.id,\n              sortIndex = None,\n              clientEventInfo = None,\n              feedbackActionInfo = None,\n              displayType = None\n            )\n          case candidate: UserCandidate =>\n            UserItem(\n              id = candidate.id,\n              sortIndex = None,\n              clientEventInfo = None,\n              feedbackActionInfo = None,\n              isMarkUnread = None,\n              displayType = User,\n              promotedMetadata = None,\n              socialContext = None,\n              reactiveTriggers = None,\n              enableReactiveBlending = None\n            )\n          case candidate =>\n            throw new UnsupportedCandidateDomainMarshallerException(\n              candidate,\n              itemCandidateWithDetails.source)\n        }\n      case itemCandidateWithDetails @ ItemCandidateWithDetails(candidate, Some(presentation), _) =>\n        throw new UnsupportedPresentationDomainMarshallerException(\n          candidate,\n          presentation,\n          itemCandidateWithDetails.source)\n      case moduleCandidateWithDetails @ ModuleCandidateWithDetails(_, presentation, _) =>\n        throw new UnsupportedModuleDomainMarshallerException(\n          presentation,\n          moduleCandidateWithDetails.source)\n    }\n\n    buildTimeline(query, entries)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/UrtDomainMarshaller.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt\n\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ManualModuleId\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.ModuleIdGeneration\nimport com.twitter.product_mixer.component_library.decorator.urt.builder.timeline_module.AutomaticUniqueModuleId\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder._\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.premarshaller.UndecoratedCandidateDomainMarshallerException\nimport com.twitter.product_mixer.core.functional_component.premarshaller.UndecoratedModuleDomainMarshallerException\nimport com.twitter.product_mixer.core.functional_component.premarshaller.UnsupportedModuleDomainMarshallerException\nimport com.twitter.product_mixer.core.functional_component.premarshaller.UnsupportedPresentationDomainMarshallerException\nimport com.twitter.product_mixer.core.model.common.identifier.DomainMarshallerIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.urt.BaseUrtItemPresentation\nimport com.twitter.product_mixer.core.model.common.presentation.urt.BaseUrtModulePresentation\nimport com.twitter.product_mixer.core.model.common.presentation.urt.BaseUrtOperationPresentation\nimport com.twitter.product_mixer.core.model.common.presentation.urt.IsDispensable\nimport com.twitter.product_mixer.core.model.common.presentation.urt.WithItemTreeDisplay\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ModuleItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineInstruction\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Domain marshaller that generates URT timelines automatically if the candidate pipeline decorators\n * use item and module presentations types that implement [[BaseUrtItemPresentation]] and\n * [[BaseUrtModulePresentation]], respectively to hold URT presentation data.\n */\ncase class UrtDomainMarshaller[-Query <: PipelineQuery](\n  override val instructionBuilders: Seq[UrtInstructionBuilder[Query, TimelineInstruction]] =\n    Seq(AddEntriesInstructionBuilder()),\n  override val cursorBuilders: Seq[UrtCursorBuilder[Query]] = Seq.empty,\n  override val cursorUpdaters: Seq[UrtCursorUpdater[Query]] = Seq.empty,\n  override val metadataBuilder: Option[BaseUrtMetadataBuilder[Query]] = None,\n  override val sortIndexStep: Int = 1,\n  override val identifier: DomainMarshallerIdentifier =\n    DomainMarshallerIdentifier(\"UnifiedRichTimeline\"))\n    extends DomainMarshaller[Query, Timeline]\n    with UrtBuilder[Query, TimelineInstruction] {\n\n  override def apply(\n    query: Query,\n    selections: Seq[CandidateWithDetails]\n  ): Timeline = {\n    val initialSortIndex = getInitialSortIndex(query)\n\n    val entries = selections.zipWithIndex.map {\n      case (ItemCandidateWithDetails(_, Some(presentation: BaseUrtItemPresentation), _), _) =>\n        presentation.timelineItem\n      case (ItemCandidateWithDetails(_, Some(presentation: BaseUrtOperationPresentation), _), _) =>\n        presentation.timelineOperation\n      case (\n            ModuleCandidateWithDetails(\n              candidates,\n              Some(presentation: BaseUrtModulePresentation),\n              _),\n            index) =>\n        val moduleItems = candidates.collect {\n          case ItemCandidateWithDetails(_, Some(itemPresentation: BaseUrtItemPresentation), _) =>\n            buildModuleItem(itemPresentation)\n        }\n\n        ModuleIdGeneration(presentation.timelineModule.id) match {\n          case _: AutomaticUniqueModuleId =>\n            //  Module IDs are unique using this method since initialSortIndex is based on time of request combined\n            //  with each timeline module index\n            presentation.timelineModule.copy(id = initialSortIndex + index, items = moduleItems)\n          case ManualModuleId(moduleId) =>\n            presentation.timelineModule.copy(id = moduleId, items = moduleItems)\n        }\n      case (\n            itemCandidateWithDetails @ ItemCandidateWithDetails(candidate, Some(presentation), _),\n            _) =>\n        throw new UnsupportedPresentationDomainMarshallerException(\n          candidate,\n          presentation,\n          itemCandidateWithDetails.source)\n      case (itemCandidateWithDetails @ ItemCandidateWithDetails(candidate, None, _), _) =>\n        throw new UndecoratedCandidateDomainMarshallerException(\n          candidate,\n          itemCandidateWithDetails.source)\n      case (\n            moduleCandidateWithDetails @ ModuleCandidateWithDetails(_, presentation @ Some(_), _),\n            _) =>\n        // handles given a non `BaseUrtModulePresentation` presentation type\n        throw new UnsupportedModuleDomainMarshallerException(\n          presentation,\n          moduleCandidateWithDetails.source)\n      case (moduleCandidateWithDetails @ ModuleCandidateWithDetails(_, None, _), _) =>\n        throw new UndecoratedModuleDomainMarshallerException(moduleCandidateWithDetails.source)\n    }\n\n    buildTimeline(query, entries)\n  }\n\n  private def buildModuleItem(itemPresentation: BaseUrtItemPresentation): ModuleItem = {\n    val isDispensable = itemPresentation match {\n      case isDispensable: IsDispensable => Some(isDispensable.dispensable)\n      case _ => None\n    }\n    val treeDisplay = itemPresentation match {\n      case withItemTreeDisplay: WithItemTreeDisplay => withItemTreeDisplay.treeDisplay\n      case _ => None\n    }\n    ModuleItem(\n      itemPresentation.timelineItem,\n      dispensable = isDispensable,\n      treeDisplay = treeDisplay)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/AddEntriesInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class AddEntriesInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, AddEntriesTimelineInstruction] {\n\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[AddEntriesTimelineInstruction] = {\n    if (entries.nonEmpty && includeInstruction(query, entries))\n      Seq(AddEntriesTimelineInstruction(entries))\n    else Seq.empty\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/AddEntriesWithAddToModuleInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Build the AddEntries instruction with special handling for AddToModule entries.\n *\n * Entries which are going to be added to a module are going to be added via\n * AddToModuleInstructionBuilder, for other entries in the same response (like cursor entries) we\n * still need an AddEntriesTimelineInstruction which is going to be created by this builder.\n */\ncase class AddEntriesWithAddToModuleInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, AddEntriesTimelineInstruction] {\n\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[AddEntriesTimelineInstruction] = {\n    if (includeInstruction(query, entries)) {\n      val entriesToAdd = entries.filter {\n        case _: TimelineModule => false\n        case _ => true\n      }\n      if (entriesToAdd.nonEmpty) Seq(AddEntriesTimelineInstruction(entriesToAdd))\n      else Seq.empty\n    } else\n      Seq.empty\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/AddEntriesWithPinnedAndReplaceInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Iterates over all the [[TimelineEntry]] passed it and creates `addEntry` entries in the URT for\n * any entries which are not pinned and not replaceable(cursors are replaceable)\n *\n * This is because pinned entries always show up in the `pinEntry` section, and replaceable entries\n * will show up in the `replaceEntry` section.\n */\ncase class AddEntriesWithPinnedAndReplaceInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, AddEntriesTimelineInstruction] {\n\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[AddEntriesTimelineInstruction] = {\n    if (includeInstruction(query, entries)) {\n      val entriesToAdd = entries\n        .filterNot(_.isPinned.getOrElse(false))\n        .filter(_.entryIdToReplace.isEmpty)\n      if (entriesToAdd.nonEmpty) Seq(AddEntriesTimelineInstruction(entriesToAdd))\n      else Seq.empty\n    } else\n      Seq.empty\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/AddEntriesWithReplaceAndShowAlertInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ShowAlert\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class AddEntriesWithReplaceAndShowAlertInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, AddEntriesTimelineInstruction] {\n\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[AddEntriesTimelineInstruction] = {\n    if (includeInstruction(query, entries)) {\n      val entriesToAdd = entries\n        .filterNot(_.isInstanceOf[ShowAlert])\n        .filter(_.entryIdToReplace.isEmpty)\n      if (entriesToAdd.nonEmpty) Seq(AddEntriesTimelineInstruction(entriesToAdd))\n      else Seq.empty\n    } else\n      Seq.empty\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/AddEntriesWithReplaceInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Build the AddEntries instruction with special handling for replaceable entries.\n *\n * Entries (though almost always a single entry) with a non-empty entryIdToReplace field should be\n * collected and transformed into ReplaceEntry instructions. These should be filtered out of the\n * AddEntries instruction. We avoid doing this as part of the regular AddEntriesInstructionBuilder\n * because replacement is rare and detecting replaceable entries takes linear time.\n */\ncase class AddEntriesWithReplaceInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, AddEntriesTimelineInstruction] {\n\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[AddEntriesTimelineInstruction] = {\n    if (includeInstruction(query, entries)) {\n      val entriesToAdd = entries.filter(_.entryIdToReplace.isEmpty)\n      if (entriesToAdd.nonEmpty) Seq(AddEntriesTimelineInstruction(entriesToAdd))\n      else Seq.empty\n    } else {\n      Seq.empty\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/AddEntriesWithShowCoverInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Cover\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Build AddEntries instruction with special handling for Covers.\n *\n * Cover Entries should be collected and transformed into ShowCover instructions. These should be\n * filtered out of the AddEntries instruction. We avoid doing this as part of the regular\n * AddEntriesInstructionBuilder because covers are used only used when using a Flip Pipeline and\n * detecting cover entries takes linear time.\n */\ncase class AddEntriesWithShowCoverInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, AddEntriesTimelineInstruction] {\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[AddEntriesTimelineInstruction] = {\n    if (includeInstruction(query, entries)) {\n      val entriesToAdd = entries.filterNot(_.isInstanceOf[Cover])\n      if (entriesToAdd.nonEmpty) Seq(AddEntriesTimelineInstruction(entriesToAdd)) else Seq.empty\n    } else\n      Seq.empty\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/AddToModuleInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddToModuleTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class AddToModuleInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, AddToModuleTimelineInstruction] {\n\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[AddToModuleTimelineInstruction] = {\n    if (includeInstruction(query, entries)) {\n      val moduleEntries = entries.collect {\n        case module: TimelineModule => module\n      }\n      if (moduleEntries.nonEmpty) {\n        assert(moduleEntries.size == 1, \"Currently we only support appending to one module\")\n        moduleEntries.headOption.map { moduleEntry =>\n          AddToModuleTimelineInstruction(\n            moduleItems = moduleEntry.items,\n            moduleEntryId = moduleEntry.entryIdentifier,\n            // Currently configuring moduleItemEntryId and prepend fields are not supported.\n            moduleItemEntryId = None,\n            prepend = None\n          )\n        }\n      }.toSeq\n      else Seq.empty\n    } else {\n      Seq.empty\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"src/scala/com/twitter/search/common/util/bloomfilter\",\n        \"stringcenter/client\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/cursor\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/cursor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"src/scala/com/twitter/search/common/util/bloomfilter\",\n        \"stringcenter/client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/BaseUnorderedExcludeIdsBottomCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtUnorderedExcludeIdsCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorType\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\ntrait BaseUnorderedExcludeIdsBottomCursorBuilder\n    extends UrtCursorBuilder[\n      PipelineQuery with HasPipelineCursor[UrtUnorderedExcludeIdsCursor]\n    ] {\n\n  def excludedIdsMaxLengthParam: Param[Int]\n\n  def excludeEntriesCollector(entries: Seq[TimelineEntry]): Seq[Long]\n\n  def serializer: PipelineCursorSerializer[UrtUnorderedExcludeIdsCursor]\n\n  override val cursorType: CursorType = BottomCursor\n\n  override def cursorValue(\n    query: PipelineQuery with HasPipelineCursor[UrtUnorderedExcludeIdsCursor],\n    entries: Seq[TimelineEntry]\n  ): String = {\n    val excludedIdsMaxLength = query.params(excludedIdsMaxLengthParam)\n    assert(excludedIdsMaxLength > 0, \"Excluded IDs max length must be greater than zero\")\n\n    val newEntryIds = excludeEntriesCollector(entries)\n    assert(\n      newEntryIds.length < excludedIdsMaxLength,\n      \"New entry IDs length must be smaller than excluded IDs max length\")\n\n    val excludedIds = query.pipelineCursor\n      .map(_.excludedIds ++ newEntryIds)\n      .getOrElse(newEntryIds)\n      .takeRight(excludedIdsMaxLength)\n\n    val cursor = UrtUnorderedExcludeIdsCursor(\n      initialSortIndex = nextBottomInitialSortIndex(query, entries),\n      excludedIds = excludedIds\n    )\n\n    serializer.serializeCursor(cursor)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/ClearCacheInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ClearCacheTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class ClearCacheInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, ClearCacheTimelineInstruction] {\n\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[ClearCacheTimelineInstruction] =\n    if (includeInstruction(query, entries)) Seq(ClearCacheTimelineInstruction()) else Seq.empty\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/FeaturePassThroughCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtPassThroughCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorType\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class PassThroughCursorBuilder[\n  -Query <: PipelineQuery with HasPipelineCursor[UrtPassThroughCursor]\n](\n  cursorFeature: Feature[Query, String],\n  override val cursorType: CursorType)\n    extends UrtCursorBuilder[Query] {\n\n  override val includeOperation: IncludeInstruction[Query] = { (query, _) =>\n    query.features.exists(_.getOrElse(cursorFeature, \"\").nonEmpty)\n  }\n\n  override def cursorValue(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): String =\n    UrtCursorSerializer.serializeCursor(\n      UrtPassThroughCursor(\n        cursorSortIndex(query, entries),\n        query.features.map(_.get(cursorFeature)).getOrElse(\"\"),\n        cursorType = Some(cursorType)\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/IncludeInstruction.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait IncludeInstruction[-Query <: PipelineQuery] { self =>\n  def apply(query: Query, entries: Seq[TimelineEntry]): Boolean\n\n  def inverse(): IncludeInstruction[Query] = new IncludeInstruction[Query] {\n    def apply(query: Query, entries: Seq[TimelineEntry]): Boolean = !self.apply(query, entries)\n  }\n}\n\nobject AlwaysInclude extends IncludeInstruction[PipelineQuery] {\n  override def apply(query: PipelineQuery, entries: Seq[TimelineEntry]): Boolean = true\n}\n\nobject IncludeOnFirstPage extends IncludeInstruction[PipelineQuery with HasPipelineCursor[_]] {\n  override def apply(\n    query: PipelineQuery with HasPipelineCursor[_],\n    entries: Seq[TimelineEntry]\n  ): Boolean = query.isFirstPage\n}\n\nobject IncludeAfterFirstPage extends IncludeInstruction[PipelineQuery with HasPipelineCursor[_]] {\n  override def apply(\n    query: PipelineQuery with HasPipelineCursor[_],\n    entries: Seq[TimelineEntry]\n  ): Boolean = !query.isFirstPage\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/MarkUnreadInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.MarkEntriesUnreadInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.MarkUnreadableEntry\n\n/**\n * Build a MarkUnreadEntries instruction\n *\n * Note that this implementation currently supports top-level entries, but not module item entries.\n */\ncase class MarkUnreadInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, MarkEntriesUnreadInstruction] {\n\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[MarkEntriesUnreadInstruction] = {\n    if (includeInstruction(query, entries)) {\n      val filteredEntries = entries.collect {\n        case entry: MarkUnreadableEntry if entry.isMarkUnread.contains(true) =>\n          entry.entryIdentifier\n      }\n      if (filteredEntries.nonEmpty) Seq(MarkEntriesUnreadInstruction(filteredEntries))\n      else Seq.empty\n    } else {\n      Seq.empty\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/OrderedBottomCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorType\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Builds [[UrtOrderedCursor]] in the Bottom position\n *\n * @param idSelector Specifies the entry from which to derive the `id` field\n * @param includeOperation Logic to determine whether or not to build the bottom cursor, which only\n *                         applies if gap cursors are required (e.g. Home Latest). When applicable,\n *                         this logic should always be the inverse of the logic used to decide\n *                         whether or not to build the gap cursor via [[OrderedGapCursorBuilder]],\n *                         since either the gap or the bottom cursor must always be returned.\n * @param serializer Converts the cursor to an encoded string\n */\ncase class OrderedBottomCursorBuilder[\n  -Query <: PipelineQuery with HasPipelineCursor[UrtOrderedCursor]\n](\n  idSelector: PartialFunction[TimelineEntry, Long],\n  override val includeOperation: IncludeInstruction[Query] = AlwaysInclude,\n  serializer: PipelineCursorSerializer[UrtOrderedCursor] = UrtCursorSerializer)\n    extends UrtCursorBuilder[Query] {\n  override val cursorType: CursorType = BottomCursor\n\n  override def cursorValue(query: Query, timelineEntries: Seq[TimelineEntry]): String = {\n    val bottomId = timelineEntries.reverseIterator.collectFirst(idSelector)\n\n    val id = bottomId.orElse(query.pipelineCursor.flatMap(_.id))\n\n    val cursor = UrtOrderedCursor(\n      initialSortIndex = nextBottomInitialSortIndex(query, timelineEntries),\n      id = id,\n      cursorType = Some(cursorType)\n    )\n\n    serializer.serializeCursor(cursor)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/OrderedGapCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Builds [[UrtOrderedCursor]] in the Bottom position as a Gap cursor.\n *\n * @param idSelector Specifies the entry from which to derive the `id` field\n * @param includeOperation Logic to determine whether or not to build the gap cursor, which should\n *                         always be the inverse of the logic used to decide whether or not to build\n *                         the bottom cursor via [[OrderedBottomCursorBuilder]], since either the\n *                         gap or the bottom cursor must always be returned.\n * @param serializer Converts the cursor to an encoded string\n */\ncase class OrderedGapCursorBuilder[\n  -Query <: PipelineQuery with HasPipelineCursor[UrtOrderedCursor]\n](\n  idSelector: PartialFunction[TimelineEntry, Long],\n  override val includeOperation: IncludeInstruction[Query],\n  serializer: PipelineCursorSerializer[UrtOrderedCursor] = UrtCursorSerializer)\n    extends UrtCursorBuilder[Query] {\n  override val cursorType: CursorType = GapCursor\n\n  override def cursorValue(\n    query: Query,\n    timelineEntries: Seq[TimelineEntry]\n  ): String = {\n    // To determine the gap boundary, use any existing cursor gap boundary id (i.e. if submitted\n    // from a previous gap cursor, else use the existing cursor id (i.e. from a previous top cursor)\n    val gapBoundaryId = query.pipelineCursor.flatMap(_.gapBoundaryId).orElse {\n      query.pipelineCursor.flatMap(_.id)\n    }\n\n    val bottomId = timelineEntries.reverseIterator.collectFirst(idSelector)\n\n    val id = bottomId.orElse(gapBoundaryId)\n\n    val cursor = UrtOrderedCursor(\n      initialSortIndex = nextBottomInitialSortIndex(query, timelineEntries),\n      id = id,\n      cursorType = Some(cursorType),\n      gapBoundaryId = gapBoundaryId\n    )\n\n    serializer.serializeCursor(cursor)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/OrderedTopCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtOrderedCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.OrderedTopCursorBuilder.TopCursorOffset\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase object OrderedTopCursorBuilder {\n  // Ensure that the next initial sort index is at least 10000 entries away from top cursor's\n  // current sort index. This is to ensure that the contents of the next page can be populated\n  // without being assigned sort indices which conflict with that of the current page. This assumes\n  // that each page will have fewer than 10000 entries.\n  val TopCursorOffset = 10000L\n}\n\n/**\n * Builds [[UrtOrderedCursor]] in the Top position\n *\n * @param idSelector Specifies the entry from which to derive the `id` field\n * @param serializer Converts the cursor to an encoded string\n */\ncase class OrderedTopCursorBuilder(\n  idSelector: PartialFunction[UniversalNoun[_], Long],\n  serializer: PipelineCursorSerializer[UrtOrderedCursor] = UrtCursorSerializer)\n    extends UrtCursorBuilder[\n      PipelineQuery with HasPipelineCursor[UrtOrderedCursor]\n    ] {\n  override val cursorType: CursorType = TopCursor\n\n  override def cursorValue(\n    query: PipelineQuery with HasPipelineCursor[UrtOrderedCursor],\n    timelineEntries: Seq[TimelineEntry]\n  ): String = {\n    val topId = timelineEntries.collectFirst(idSelector)\n\n    val id = topId.orElse(query.pipelineCursor.flatMap(_.id))\n\n    val cursor = UrtOrderedCursor(\n      initialSortIndex = cursorSortIndex(query, timelineEntries) + TopCursorOffset,\n      id = id,\n      cursorType = Some(cursorType)\n    )\n\n    serializer.serializeCursor(cursor)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/PinEntryInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.PinEntryTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.PinnableEntry\n\ncase class PinEntryInstructionBuilder()\n    extends UrtInstructionBuilder[PipelineQuery, PinEntryTimelineInstruction] {\n\n  override def build(\n    query: PipelineQuery,\n    entries: Seq[TimelineEntry]\n  ): Seq[PinEntryTimelineInstruction] = {\n    // Only one entry can be pinned and the desirable behavior is to pick the entry with the highest\n    // sort index in the event that multiple pinned items exist. Since the entries are already\n    // sorted we can accomplish this by picking the first one.\n    entries.collectFirst {\n      case entry: PinnableEntry if entry.isPinned.getOrElse(false) =>\n        PinEntryTimelineInstruction(entry)\n    }.toSeq\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/PlaceholderTopCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtPlaceholderCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.PlaceholderTopCursorBuilder.DefaultPlaceholderCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.UrtPipelineCursor\n\nobject PlaceholderTopCursorBuilder {\n  val DefaultPlaceholderCursor = UrtPlaceholderCursor()\n}\n\n/**\n * Top cursor builder that can be used when the Product does not support paging up. The URT spec\n * requires that both bottom and top cursors always be present on each page. Therefore, if the\n * product does not support paging up, then we can use a cursor value that is not deserializable.\n * This way if the client submits a TopCursor, the backend will treat the the request as if no\n * cursor was submitted.\n */\ncase class PlaceholderTopCursorBuilder(\n  serializer: PipelineCursorSerializer[UrtPipelineCursor] = UrtCursorSerializer)\n    extends UrtCursorBuilder[PipelineQuery with HasPipelineCursor[UrtPipelineCursor]] {\n  override val cursorType: CursorType = TopCursor\n\n  override def cursorValue(\n    query: PipelineQuery with HasPipelineCursor[UrtPipelineCursor],\n    timelineEntries: Seq[TimelineEntry]\n  ): String = serializer.serializeCursor(DefaultPlaceholderCursor)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/ReplaceEntryInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ReplaceEntryTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorOperation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Selects one or more [[TimelineEntry]] instance from the input timeline entries.\n *\n * @tparam Query The domain model for the [[PipelineQuery]] used as input.\n */\ntrait EntriesToReplace[-Query <: PipelineQuery] {\n  def apply(query: Query, entries: Seq[TimelineEntry]): Seq[TimelineEntry]\n}\n\n/**\n * Selects all entries with a non-empty valid entryIdToReplace.\n *\n * @note this will result in multiple [[ReplaceEntryTimelineInstruction]]s\n */\ncase object ReplaceAllEntries extends EntriesToReplace[PipelineQuery] {\n  def apply(query: PipelineQuery, entries: Seq[TimelineEntry]): Seq[TimelineEntry] =\n    entries.filter(_.entryIdToReplace.isDefined)\n}\n\n/**\n * Selects a replaceable URT [[CursorOperation]] from the timeline entries, that matches the\n * input cursorType.\n */\ncase class ReplaceUrtCursor(cursorType: CursorType) extends EntriesToReplace[PipelineQuery] {\n  override def apply(query: PipelineQuery, entries: Seq[TimelineEntry]): Seq[TimelineEntry] =\n    entries.collectFirst {\n      case cursorOperation: CursorOperation\n          if cursorOperation.cursorType == cursorType && cursorOperation.entryIdToReplace.isDefined =>\n        cursorOperation\n    }.toSeq\n}\n\n/**\n * Create a ReplaceEntry instruction\n *\n * @param entriesToReplace   each replace instruction can contain only one entry. Users specify which\n *                           entry to replace using [[EntriesToReplace]]. If multiple entries are\n *                           specified, multiple [[ReplaceEntryTimelineInstruction]]s will be created.\n * @param includeInstruction whether the instruction should be included in the response\n */\ncase class ReplaceEntryInstructionBuilder[Query <: PipelineQuery](\n  entriesToReplace: EntriesToReplace[Query],\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, ReplaceEntryTimelineInstruction] {\n\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[ReplaceEntryTimelineInstruction] = {\n    if (includeInstruction(query, entries))\n      entriesToReplace(query, entries).map(ReplaceEntryTimelineInstruction)\n    else\n      Seq.empty\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/ShowAlertInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ShowAlert\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ShowAlertInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class ShowAlertInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, ShowAlertInstruction] {\n\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[ShowAlertInstruction] = {\n    if (includeInstruction(query, entries)) {\n      // Currently only one Alert is supported per response\n      entries.collectFirst {\n        case alertEntry: ShowAlert => ShowAlertInstruction(alertEntry)\n      }.toSeq\n    } else Seq.empty\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/ShowCoverInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ShowCoverInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Cover\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class ShowCoverInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends UrtInstructionBuilder[Query, ShowCoverInstruction] {\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[ShowCoverInstruction] = {\n    if (includeInstruction(query, entries)) {\n      // Currently only one cover is supported per response\n      entries.collectFirst {\n        case coverEntry: Cover => ShowCoverInstruction(coverEntry)\n      }.toSeq\n    } else {\n      Seq.empty\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/StaticTimelineScribeConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class StaticTimelineScribeConfigBuilder(\n  timelineScribeConfig: TimelineScribeConfig)\n    extends TimelineScribeConfigBuilder[PipelineQuery] {\n\n  def build(\n    query: PipelineQuery,\n    entries: Seq[TimelineEntry]\n  ): Option[TimelineScribeConfig] = Some(timelineScribeConfig)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/TerminateInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.BottomTermination\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TerminateTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineTerminationDirection\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TopAndBottomTermination\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TopTermination\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nsealed trait TerminateInstructionBuilder[Query <: PipelineQuery]\n    extends UrtInstructionBuilder[Query, TerminateTimelineInstruction] {\n\n  def direction: TimelineTerminationDirection\n\n  override def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[TerminateTimelineInstruction] =\n    if (includeInstruction(query, entries))\n      Seq(TerminateTimelineInstruction(terminateTimelineDirection = direction))\n    else Seq.empty\n}\n\ncase class TerminateTopInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends TerminateInstructionBuilder[Query] {\n\n  override val direction = TopTermination\n}\n\ncase class TerminateBottomInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends TerminateInstructionBuilder[Query] {\n\n  override val direction = BottomTermination\n}\n\ncase class TerminateTopAndBottomInstructionBuilder[Query <: PipelineQuery](\n  override val includeInstruction: IncludeInstruction[Query] = AlwaysInclude)\n    extends TerminateInstructionBuilder[Query] {\n\n  override val direction = TopAndBottomTermination\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/TimelineScribeConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Trait for our builder which given a query and entries will return an `Option[TimelineScribeConfig]`\n *\n * @tparam Query\n */\ntrait TimelineScribeConfigBuilder[-Query <: PipelineQuery] {\n\n  def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Option[TimelineScribeConfig]\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/UnorderedBloomFilterBottomCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtUnorderedBloomFilterCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorType\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.search.common.util.bloomfilter.AdaptiveLongIntBloomFilterBuilder\n\n/**\n * Builds [[UrtUnorderedBloomFilterCursor]] in the Bottom position\n *\n * @param idSelector Specifies the entry from which to derive the `id` field\n * @param serializer Converts the cursor to an encoded string\n */\ncase class UnorderedBloomFilterBottomCursorBuilder(\n  idSelector: PartialFunction[UniversalNoun[_], Long],\n  serializer: PipelineCursorSerializer[UrtUnorderedBloomFilterCursor] = UrtCursorSerializer)\n    extends UrtCursorBuilder[\n      PipelineQuery with HasPipelineCursor[UrtUnorderedBloomFilterCursor]\n    ] {\n\n  override val cursorType: CursorType = BottomCursor\n\n  override def cursorValue(\n    query: PipelineQuery with HasPipelineCursor[UrtUnorderedBloomFilterCursor],\n    entries: Seq[TimelineEntry]\n  ): String = {\n    val bloomFilter = query.pipelineCursor.map(_.longIntBloomFilter)\n    val ids = entries.collect(idSelector)\n\n    val cursor = UrtUnorderedBloomFilterCursor(\n      initialSortIndex = nextBottomInitialSortIndex(query, entries),\n      longIntBloomFilter = AdaptiveLongIntBloomFilterBuilder.build(ids, bloomFilter)\n    )\n\n    serializer.serializeCursor(cursor)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/UnorderedExcludeIdsBottomCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtUnorderedExcludeIdsCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.timelines.configapi.Param\n\n/**\n * Builds [[UrtUnorderedExcludeIdsCursor]] in the Bottom position\n *\n * @param excludedIdsMaxLengthParam The maximum length of the cursor\n * @param excludeIdsSelector Specifies the entry Ids to populate on the `excludedIds` field\n * @param serializer Converts the cursor to an encoded string\n */\ncase class UnorderedExcludeIdsBottomCursorBuilder(\n  override val excludedIdsMaxLengthParam: Param[Int],\n  excludeIdsSelector: PartialFunction[UniversalNoun[_], Long],\n  override val serializer: PipelineCursorSerializer[UrtUnorderedExcludeIdsCursor] =\n    UrtCursorSerializer)\n    extends BaseUnorderedExcludeIdsBottomCursorBuilder {\n\n  override def excludeEntriesCollector(entries: Seq[TimelineEntry]): Seq[Long] =\n    entries.collect(excludeIdsSelector)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/UnorderedExcludeIdsSeqBottomCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.component_library.model.cursor.UrtUnorderedExcludeIdsCursor\nimport com.twitter.product_mixer.component_library.premarshaller.cursor.UrtCursorSerializer\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineCursorSerializer\nimport com.twitter.timelines.configapi.Param\n\n/**\n * Builds [[UrtUnorderedExcludeIdsCursor]] in the Bottom position when we want to also exclude ids\n * of items inside a module. The reason we cannot use [[UnorderedExcludeIdsBottomCursorBuilder]] in\n * such case is that the excludeIdsSelector of [[UnorderedExcludeIdsBottomCursorBuilder]] is doing a\n * one to one mapping between entries and excluded ids, but in case of having a module, a module\n * entry can result in excluding a sequence of entries.\n *\n * @param excludedIdsMaxLengthParam The maximum length of the cursor\n * @param excludeIdsSelector Specifies the entry Ids to populate on the `excludedIds` field\n * @param serializer Converts the cursor to an encoded string\n */\ncase class UnorderedExcludeIdsSeqBottomCursorBuilder(\n  override val excludedIdsMaxLengthParam: Param[Int],\n  excludeIdsSelector: PartialFunction[UniversalNoun[_], Seq[Long]],\n  override val serializer: PipelineCursorSerializer[UrtUnorderedExcludeIdsCursor] =\n    UrtCursorSerializer)\n    extends BaseUnorderedExcludeIdsBottomCursorBuilder {\n\n  override def excludeEntriesCollector(entries: Seq[TimelineEntry]): Seq[Long] =\n    entries.collect(excludeIdsSelector).flatten\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/UrtBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorOperation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineInstruction\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.UrtPipelineCursor\nimport com.twitter.product_mixer.core.util.SortIndexBuilder\n\ntrait UrtBuilder[-Query <: PipelineQuery, +Instruction <: TimelineInstruction] {\n  private val TimelineIdSuffix = \"-Timeline\"\n\n  def instructionBuilders: Seq[UrtInstructionBuilder[Query, Instruction]]\n\n  def cursorBuilders: Seq[UrtCursorBuilder[Query]]\n  def cursorUpdaters: Seq[UrtCursorUpdater[Query]]\n\n  def metadataBuilder: Option[BaseUrtMetadataBuilder[Query]]\n\n  // Timeline entry sort indexes will count down by this value. Values higher than 1 are useful to\n  // leave room in the sequence for dynamically injecting content in between existing entries.\n  def sortIndexStep: Int = 1\n\n  final def buildTimeline(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Timeline = {\n    val initialSortIndex = getInitialSortIndex(query)\n\n    // Set the sort indexes of the entries before we pass them to the cursor builders, since many\n    // cursor implementations use the sort index of the first/last entry as part of the cursor value\n    val sortIndexedEntries = updateSortIndexes(initialSortIndex, entries)\n\n    // Iterate over the cursorUpdaters in the order they were defined. Note that each updater will\n    // be passed the timelineEntries updated by the previous cursorUpdater.\n    val updatedCursorEntries: Seq[TimelineEntry] =\n      cursorUpdaters.foldLeft(sortIndexedEntries) { (timelineEntries, cursorUpdater) =>\n        cursorUpdater.update(query, timelineEntries)\n      }\n\n    val allCursoredEntries =\n      updatedCursorEntries ++ cursorBuilders.flatMap(_.build(query, updatedCursorEntries))\n\n    val instructions: Seq[Instruction] =\n      instructionBuilders.flatMap(_.build(query, allCursoredEntries))\n\n    val metadata = metadataBuilder.map(_.build(query, allCursoredEntries))\n\n    Timeline(\n      id = query.product.identifier.toString + TimelineIdSuffix,\n      instructions = instructions,\n      metadata = metadata\n    )\n  }\n\n  final def getInitialSortIndex(query: Query): Long =\n    query match {\n      case cursorQuery: HasPipelineCursor[_] =>\n        UrtPipelineCursor\n          .getCursorInitialSortIndex(cursorQuery)\n          .getOrElse(SortIndexBuilder.timeToId(query.queryTime))\n      case _ => SortIndexBuilder.timeToId(query.queryTime)\n    }\n\n  /**\n   * Updates the sort indexes in the timeline entries starting from the given initial sort index\n   * value and decreasing by the value defined in the sort index step field\n   *\n   * @param initialSortIndex The initial value of the sort index\n   * @param timelineEntries Timeline entries to update\n   */\n  final def updateSortIndexes(\n    initialSortIndex: Long,\n    timelineEntries: Seq[TimelineEntry]\n  ): Seq[TimelineEntry] = {\n    val indexRange =\n      initialSortIndex to (initialSortIndex - (timelineEntries.size * sortIndexStep)) by -sortIndexStep\n\n    // Skip any existing cursors because their sort indexes will be managed by their cursor updater.\n    // If the cursors are not removed first, then the remaining entries would have a gap everywhere\n    // an existing cursor was present.\n    val (cursorEntries, nonCursorEntries) = timelineEntries.partition {\n      case _: CursorOperation => true\n      case _ => false\n    }\n\n    nonCursorEntries.zip(indexRange).map {\n      case (entry, index) =>\n        entry.withSortIndex(index)\n    } ++ cursorEntries\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/UrtCursorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtCursorBuilder.DefaultSortIndex\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtCursorBuilder.NextPageTopCursorEntryOffset\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtCursorBuilder.UrtEntryOffset\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.BottomCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorOperation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.GapCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.TopCursor\nimport com.twitter.product_mixer.core.pipeline.HasPipelineCursor\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.UrtPipelineCursor\nimport com.twitter.product_mixer.core.util.SortIndexBuilder\n\nobject UrtCursorBuilder {\n  val NextPageTopCursorEntryOffset = 1L\n  val UrtEntryOffset = 1L\n  val DefaultSortIndex = (query: PipelineQuery) => SortIndexBuilder.timeToId(query.queryTime)\n}\n\ntrait UrtCursorBuilder[-Query <: PipelineQuery] {\n\n  val includeOperation: IncludeInstruction[Query] = AlwaysInclude\n\n  def cursorType: CursorType\n  def cursorValue(query: Query, entries: Seq[TimelineEntry]): String\n\n  /**\n   * Identifier of an *existing* timeline cursor that this new cursor would replace, if this cursor\n   * is returned in a `ReplaceEntry` timeline instruction.\n   *\n   * Note:\n   *   - This id is used to populate the `entryIdToReplace` field on the URT TimelineEntry\n   *     generated. More details at [[CursorOperation.entryIdToReplace]].\n   *   - As a convention, we use the sortIndex of the cursor for its id/entryId fields. So the\n   *     `idToReplace` should represent the sortIndex of the existing cursor to be replaced.\n   */\n  def idToReplace(query: Query): Option[Long] = None\n\n  def cursorSortIndex(query: Query, entries: Seq[TimelineEntry]): Long =\n    (query, cursorType) match {\n      case (query: PipelineQuery with HasPipelineCursor[_], TopCursor) =>\n        topCursorSortIndex(query, entries)\n      case (query: PipelineQuery with HasPipelineCursor[_], BottomCursor | GapCursor) =>\n        bottomCursorSortIndex(query, entries)\n      case _ =>\n        throw new UnsupportedOperationException(\n          \"Automatic sort index support limited to top and bottom cursors\")\n    }\n\n  def build(query: Query, entries: Seq[TimelineEntry]): Option[CursorOperation] = {\n    if (includeOperation(query, entries)) {\n      val sortIndex = cursorSortIndex(query, entries)\n\n      val cursorOperation = CursorOperation(\n        id = sortIndex,\n        sortIndex = Some(sortIndex),\n        value = cursorValue(query, entries),\n        cursorType = cursorType,\n        displayTreatment = None,\n        idToReplace = idToReplace(query),\n      )\n\n      Some(cursorOperation)\n    } else None\n  }\n\n  /**\n   * Build the top cursor sort index which handles the following cases:\n   * 1. When there is at least one non-cursor entry, use the first entry's sort index + UrtEntryOffset\n   * 2. When there are no non-cursor entries, and initialSortIndex is not set which indicates that\n   *    it is the first page, use DefaultSortIndex + UrtEntryOffset\n   * 3. When there are no non-cursor entries, and initialSortIndex is set which indicates that it is\n   *    not the first page, use the query.initialSortIndex from the passed-in cursor + UrtEntryOffset\n   */\n  protected def topCursorSortIndex(\n    query: PipelineQuery with HasPipelineCursor[_],\n    entries: Seq[TimelineEntry]\n  ): Long = {\n    val nonCursorEntries = entries.filter {\n      case _: CursorOperation => false\n      case _: CursorItem => false\n      case _ => true\n    }\n\n    lazy val initialSortIndex =\n      UrtPipelineCursor.getCursorInitialSortIndex(query).getOrElse(DefaultSortIndex(query))\n\n    nonCursorEntries.headOption.flatMap(_.sortIndex).getOrElse(initialSortIndex) + UrtEntryOffset\n  }\n\n  /**\n   * Specifies the point at which the next page's entries' sort indices will start counting.\n   *\n   * Note that in the case of URT, the next page's entries' does not include the top cursor. As\n   * such, the value of initialSortIndex passed back in the cursor is typically the bottom cursor's\n   * sort index - 2. Subtracting 2 leaves room for the next page's top cursor, which will have a\n   * sort index of top entry + 1.\n   */\n  protected def nextBottomInitialSortIndex(\n    query: PipelineQuery with HasPipelineCursor[_],\n    entries: Seq[TimelineEntry]\n  ): Long = {\n    bottomCursorSortIndex(query, entries) - NextPageTopCursorEntryOffset - UrtEntryOffset\n  }\n\n  /**\n   * Build the bottom cursor sort index which handles the following cases:\n   * 1. When there is at least one non-cursor entry, use the last entry's sort index - UrtEntryOffset\n   * 2. When there are no non-cursor entries, and initialSortIndex is not set which indicates that\n   *    it is the first page, use DefaultSortIndex\n   * 3. When there are no non-cursor entries, and initialSortIndex is set which indicates that it is\n   *    not the first page, use the query.initialSortIndex from the passed-in cursor\n   */\n  protected def bottomCursorSortIndex(\n    query: PipelineQuery with HasPipelineCursor[_],\n    entries: Seq[TimelineEntry]\n  ): Long = {\n    val nonCursorEntries = entries.filter {\n      case _: CursorOperation => false\n      case _: CursorItem => false\n      case _ => true\n    }\n\n    lazy val initialSortIndex =\n      UrtPipelineCursor.getCursorInitialSortIndex(query).getOrElse(DefaultSortIndex(query))\n\n    nonCursorEntries.lastOption\n      .flatMap(_.sortIndex).map(_ - UrtEntryOffset).getOrElse(initialSortIndex)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/UrtCursorUpdater.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.component_library.premarshaller.urt.builder.UrtCursorUpdater.getCursorByType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorOperation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject UrtCursorUpdater {\n\n  def getCursorByType(\n    entries: Seq[TimelineEntry],\n    cursorType: CursorType\n  ): Option[CursorOperation] = {\n    entries.collectFirst {\n      case cursor: CursorOperation if cursor.cursorType == cursorType => cursor\n    }\n  }\n}\n\n// If a CursorCandidate is returned by a Candidate Source, use this trait to update that Cursor as\n// necessary (as opposed to building a new cursor which is done with the UrtCursorBuilder)\ntrait UrtCursorUpdater[-Query <: PipelineQuery] extends UrtCursorBuilder[Query] { self =>\n\n  def getExistingCursor(entries: Seq[TimelineEntry]): Option[CursorOperation] = {\n    getCursorByType(entries, self.cursorType)\n  }\n\n  def update(query: Query, entries: Seq[TimelineEntry]): Seq[TimelineEntry] = {\n    if (includeOperation(query, entries)) {\n      getExistingCursor(entries)\n        .map { existingCursor =>\n          // Safe .get because includeOperation() is shared in this context\n          // build() method creates a new CursorOperation. We copy over the `idToReplace`\n          // from the existing cursor.\n          val newCursor =\n            build(query, entries).get\n              .copy(idToReplace = existingCursor.idToReplace)\n\n          entries.filterNot(_ == existingCursor) :+ newCursor\n        }.getOrElse(entries)\n    } else entries\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/UrtInstructionBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineInstruction\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait UrtInstructionBuilder[-Query <: PipelineQuery, +Instruction <: TimelineInstruction] {\n\n  def includeInstruction: IncludeInstruction[Query] = AlwaysInclude\n\n  def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): Seq[Instruction]\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/premarshaller/urt/builder/UrtMetadataBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.premarshaller.urt.builder\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineMetadata\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.stringcenter.client.core.ExternalString\n\ntrait BaseUrtMetadataBuilder[-Query <: PipelineQuery] {\n  def build(\n    query: Query,\n    entries: Seq[TimelineEntry]\n  ): TimelineMetadata\n}\n\ncase class UrtMetadataBuilder(\n  title: Option[String] = None,\n  scribeConfigBuilder: Option[TimelineScribeConfigBuilder[PipelineQuery]])\n    extends BaseUrtMetadataBuilder[PipelineQuery] {\n\n  override def build(\n    query: PipelineQuery,\n    entries: Seq[TimelineEntry]\n  ): TimelineMetadata = TimelineMetadata(\n    title = title,\n    scribeConfig = scribeConfigBuilder.flatMap(_.build(query, entries))\n  )\n}\n\ncase class UrtMetadataStringCenterBuilder(\n  titleKey: ExternalString,\n  scribeConfigBuilder: Option[TimelineScribeConfigBuilder[PipelineQuery]],\n  stringCenter: StringCenter)\n    extends BaseUrtMetadataBuilder[PipelineQuery] {\n\n  override def build(\n    query: PipelineQuery,\n    entries: Seq[TimelineEntry]\n  ): TimelineMetadata = TimelineMetadata(\n    title = Some(stringCenter.prepare(titleKey)),\n    scribeConfig = scribeConfigBuilder.flatMap(_.build(query, entries))\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/common/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/io/grpc:grpc-netty\",\n        \"3rdparty/jvm/io/netty:netty4-tcnative-boringssl-static\",\n        \"3rdparty/jvm/io/opil:tensorflow-serving-client\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/jvm/triton/inference:triton-grpc\",\n        \"finagle-internal/finagle-grpc/src/main/scala\",\n        \"finagle-internal/finagle-grpc/src/test/java\",\n        \"finagle-internal/finagle-grpc/src/test/proto\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/client\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"finatra-internal/mtls/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"ml-serving/scala:kfserving-tfserving-converter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/triton/inference:triton-grpc\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/common/MLModelInferenceClient.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.common\n\nimport com.twitter.stitch.Stitch\nimport inference.GrpcService.ModelInferRequest\nimport inference.GrpcService.ModelInferResponse\n\n/**\n * MLModelInferenceClient for calling different Inference Service such as ManagedModelClient or NaviModelClient.\n */\ntrait MLModelInferenceClient {\n  def score(request: ModelInferRequest): Stitch[ModelInferResponse]\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/common/ManagedModelClient.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.common\n\nimport com.twitter.finagle.Http\nimport com.twitter.finagle.grpc.FinagleChannelBuilder\nimport com.twitter.finagle.grpc.FutureConverters\nimport com.twitter.stitch.Stitch\nimport inference.GRPCInferenceServiceGrpc\nimport inference.GrpcService.ModelInferRequest\nimport inference.GrpcService.ModelInferResponse\nimport io.grpc.ManagedChannel\n\n/**\n * Client wrapper for calling a Cortex Managed Inference Service (go/cmis) ML Model using GRPC.\n * @param httpClient Finagle HTTP Client to use for connection.\n * @param modelPath Wily path to the ML Model service (e.g. /cluster/local/role/service/instance).\n */\ncase class ManagedModelClient(\n  httpClient: Http.Client,\n  modelPath: String)\n    extends MLModelInferenceClient {\n\n  private val channel: ManagedChannel =\n    FinagleChannelBuilder.forTarget(modelPath).httpClient(httpClient).build()\n\n  private val inferenceServiceStub = GRPCInferenceServiceGrpc.newFutureStub(channel)\n\n  def score(request: ModelInferRequest): Stitch[ModelInferResponse] = {\n    Stitch\n      .callFuture(\n        FutureConverters\n          .RichListenableFuture(inferenceServiceStub.modelInfer(request)).toTwitter)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/common/ModelSelector.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.common\n\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\n/**\n * Selector for choosing which Model ID/Name to use when calling an underlying ML Model Service.\n */\ntrait ModelSelector[-Query <: PipelineQuery] {\n  def apply(query: Query): Option[String]\n}\n\n/**\n * Simple Model ID Selector that chooses model based off of a Param object.\n * @param param ConfigAPI Param that decides the model id.\n */\ncase class ParamModelSelector[Query <: PipelineQuery](param: Param[String])\n    extends ModelSelector[Query] {\n  override def apply(query: Query): Option[String] = Some(query.params(param))\n}\n\n/**\n * Static Selector that chooses the same model name always\n * @param modelName The model name to use.\n */\ncase class StaticModelSelector(modelName: String) extends ModelSelector[PipelineQuery] {\n  override def apply(query: PipelineQuery): Option[String] = Some(modelName)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/common/NaviModelClient.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.common\n\nimport com.twitter.finagle.Http\nimport com.twitter.finagle.grpc.FinagleChannelBuilder\nimport com.twitter.finagle.grpc.FutureConverters\nimport com.twitter.mlserving.frontend.TFServingInferenceServiceImpl\nimport com.twitter.stitch.Stitch\nimport tensorflow.serving.PredictionServiceGrpc\nimport inference.GrpcService.ModelInferRequest\nimport inference.GrpcService.ModelInferResponse\nimport io.grpc.ManagedChannel\nimport io.grpc.Status\n\n/**\n * Client wrapper for calling a Navi Inference Service (go/navi).\n * @param httpClient Finagle HTTP Client to use for connection.\n * @param modelPath Wily path to the ML Model service (e.g. /s/role/service).\n */\ncase class NaviModelClient(\n  httpClient: Http.Client,\n  modelPath: String)\n    extends MLModelInferenceClient {\n\n  private val channel: ManagedChannel =\n    FinagleChannelBuilder\n      .forTarget(modelPath)\n      .httpClient(httpClient)\n      // Navi enforces an authority name.\n      .overrideAuthority(\"rustserving\")\n      // certain GRPC errors need to be retried.\n      .enableRetryForStatus(Status.UNKNOWN)\n      .enableRetryForStatus(Status.RESOURCE_EXHAUSTED)\n      // this is required at channel level as mTLS is enabled at httpClient level\n      .usePlaintext()\n      .build()\n\n  private val inferenceServiceStub = PredictionServiceGrpc.newFutureStub(channel)\n\n  def score(request: ModelInferRequest): Stitch[ModelInferResponse] = {\n    val tfServingRequest = TFServingInferenceServiceImpl.adaptModelInferRequest(request)\n    Stitch\n      .callFuture(\n        FutureConverters\n          .RichListenableFuture(inferenceServiceStub.predict(tfServingRequest)).toTwitter\n          .map { response =>\n            TFServingInferenceServiceImpl.adaptModelInferResponse(response)\n          }\n      )\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/cortex/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-http/src/main/scala\",\n        \"finatra-internal/mtls/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/http\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/common\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"src/scala/com/twitter/ml/featurestore/lib\",\n        \"src/thrift/com/twitter/ml/prediction_service:prediction_service-java\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/module/http\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/common\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/cortex/CortexManagedInferenceServiceDataRecordScorer.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.cortex\n\nimport com.google.protobuf.ByteString\nimport com.twitter.ml.prediction_service.BatchPredictionRequest\nimport com.twitter.ml.prediction_service.BatchPredictionResponse\nimport com.twitter.product_mixer.component_library.scorer.common.ManagedModelClient\nimport com.twitter.product_mixer.component_library.scorer.common.ModelSelector\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.datarecord.BaseDataRecordFeature\nimport com.twitter.product_mixer.core.feature.datarecord.TensorDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordConverter\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordExtractor\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.FeaturesScope\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport inference.GrpcService\nimport inference.GrpcService.ModelInferRequest\nimport inference.GrpcService.ModelInferResponse\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.stitch.Stitch\nimport org.apache.thrift.TDeserializer\nimport org.apache.thrift.TSerializer\nimport scala.collection.JavaConverters._\n\nprivate[cortex] class CortexManagedDataRecordScorer[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  QueryFeatures <: BaseDataRecordFeature[Query, _],\n  CandidateFeatures <: BaseDataRecordFeature[Candidate, _],\n  ResultFeatures <: BaseDataRecordFeature[Candidate, _] with TensorDataRecordCompatible[_]\n](\n  override val identifier: ScorerIdentifier,\n  modelSignature: String,\n  modelSelector: ModelSelector[Query],\n  modelClient: ManagedModelClient,\n  queryFeatures: FeaturesScope[QueryFeatures],\n  candidateFeatures: FeaturesScope[CandidateFeatures],\n  resultFeatures: Set[ResultFeatures])\n    extends Scorer[Query, Candidate] {\n\n  require(resultFeatures.nonEmpty, \"Result features cannot be empty\")\n  override val features: Set[Feature[_, _]] = resultFeatures.asInstanceOf[Set[Feature[_, _]]]\n\n  private val queryDataRecordAdapter = new DataRecordConverter(queryFeatures)\n  private val candidatesDataRecordAdapter = new DataRecordConverter(candidateFeatures)\n  private val resultDataRecordExtractor = new DataRecordExtractor(resultFeatures)\n\n  private val localTSerializer = new ThreadLocal[TSerializer] {\n    override protected def initialValue: TSerializer = new TSerializer()\n  }\n\n  private val localTDeserializer = new ThreadLocal[TDeserializer] {\n    override protected def initialValue: TDeserializer = new TDeserializer()\n  }\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    modelClient.score(buildRequest(query, candidates)).map(buildResponse(candidates, _))\n  }\n\n  /**\n   * Takes candidates to be scored and converts it to a ModelInferRequest that can be passed to the\n   * managed ML service\n   */\n  private def buildRequest(\n    query: Query,\n    scorerCandidates: Seq[CandidateWithFeatures[Candidate]]\n  ): ModelInferRequest = {\n    // Convert the feature maps to thrift data records and construct thrift request.\n    val thriftDataRecords = scorerCandidates.map { candidate =>\n      candidatesDataRecordAdapter.toDataRecord(candidate.features)\n    }\n    val batchRequest = new BatchPredictionRequest(thriftDataRecords.asJava)\n    query.features.foreach { featureMap =>\n      batchRequest.setCommonFeatures(queryDataRecordAdapter.toDataRecord(featureMap))\n    }\n    val serializedBatchRequest = localTSerializer.get().serialize(batchRequest)\n\n    // Build Tensor Request\n    val requestBuilder = ModelInferRequest\n      .newBuilder()\n\n    modelSelector.apply(query).foreach { modelName =>\n      requestBuilder.setModelName(modelName) // model name in the model config\n    }\n\n    val inputTensorBuilder = ModelInferRequest.InferInputTensor\n      .newBuilder()\n      .setName(\"request\")\n      .setDatatype(\"UINT8\")\n      .addShape(serializedBatchRequest.length)\n\n    val inferParameter = GrpcService.InferParameter\n      .newBuilder()\n      .setStringParam(modelSignature) // signature of exported tf function\n      .build()\n\n    requestBuilder\n      .addInputs(inputTensorBuilder)\n      .addRawInputContents(ByteString.copyFrom(serializedBatchRequest))\n      .putParameters(\"signature_name\", inferParameter)\n      .build()\n  }\n\n  private def buildResponse(\n    scorerCandidates: Seq[CandidateWithFeatures[Candidate]],\n    response: ModelInferResponse\n  ): Seq[FeatureMap] = {\n\n    val responseByteString = if (response.getRawOutputContentsList.isEmpty()) {\n      throw PipelineFailure(\n        IllegalStateFailure,\n        \"Model inference response has empty raw outputContents\")\n    } else {\n      response.getRawOutputContents(0)\n    }\n    val batchPredictionResponse: BatchPredictionResponse = new BatchPredictionResponse()\n    localTDeserializer.get().deserialize(batchPredictionResponse, responseByteString.toByteArray)\n\n    // get the prediction values from the batch prediction response\n    val resultScoreMaps =\n      batchPredictionResponse.predictions.asScala.map(resultDataRecordExtractor.fromDataRecord)\n\n    if (resultScoreMaps.size != scorerCandidates.size) {\n      throw PipelineFailure(IllegalStateFailure, \"Result Size mismatched candidates size\")\n    }\n\n    resultScoreMaps\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/cortex/CortexManagedInferenceServiceDataRecordScorerBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.cortex\n\nimport com.twitter.finagle.Http\nimport com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule.FinagleHttpClientModule\nimport com.twitter.product_mixer.component_library.scorer.common.ManagedModelClient\nimport com.twitter.product_mixer.component_library.scorer.common.ModelSelector\nimport com.twitter.product_mixer.core.feature.datarecord.BaseDataRecordFeature\nimport com.twitter.product_mixer.core.feature.datarecord.TensorDataRecordCompatible\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.FeaturesScope\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport javax.inject.Inject\nimport javax.inject.Named\nimport javax.inject.Singleton\n\n@Singleton\nclass CortexManagedInferenceServiceDataRecordScorerBuilder @Inject() (\n  @Named(FinagleHttpClientModule) httpClient: Http.Client) {\n\n  /**\n   * Builds a configurable Scorer to call into your desired DataRecord-backed Cortex Managed ML Model Service.\n   *\n   * If your service does not bind an Http.Client implementation, add\n   * [[com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule]]\n   * to your server module list\n   *\n   * @param scorerIdentifier  Unique identifier for the scorer\n   * @param modelPath         MLS path to model\n   * @param modelSignature    Model Signature Key\n   * @param modelSelector [[ModelSelector]] for choosing the model name, can be an anon function.\n   * @param candidateFeatures Desired candidate level feature store features to pass to the model.\n   * @param resultFeatures Desired candidate level feature store features to extract from the model.\n   *                       Since the Cortex Managed Platform always returns tensor values, the\n   *                       feature must use a [[TensorDataRecordCompatible]].\n   * @tparam Query Type of pipeline query.\n   * @tparam Candidate Type of candidates to score.\n   * @tparam QueryFeatures type of the query level features consumed by the scorer.\n   * @tparam CandidateFeatures type of the candidate level features consumed by the scorer.\n   * @tparam ResultFeatures type of the candidate level features returned by the scorer.\n   */\n  def build[\n    Query <: PipelineQuery,\n    Candidate <: UniversalNoun[Any],\n    QueryFeatures <: BaseDataRecordFeature[Query, _],\n    CandidateFeatures <: BaseDataRecordFeature[Candidate, _],\n    ResultFeatures <: BaseDataRecordFeature[Candidate, _] with TensorDataRecordCompatible[_]\n  ](\n    scorerIdentifier: ScorerIdentifier,\n    modelPath: String,\n    modelSignature: String,\n    modelSelector: ModelSelector[Query],\n    queryFeatures: FeaturesScope[QueryFeatures],\n    candidateFeatures: FeaturesScope[CandidateFeatures],\n    resultFeatures: Set[ResultFeatures]\n  ): Scorer[Query, Candidate] =\n    new CortexManagedDataRecordScorer(\n      identifier = scorerIdentifier,\n      modelSignature = modelSignature,\n      modelSelector = modelSelector,\n      modelClient = ManagedModelClient(httpClient, modelPath),\n      queryFeatures = queryFeatures,\n      candidateFeatures = candidateFeatures,\n      resultFeatures = resultFeatures\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/cortex/CortexManagedInferenceServiceTensorScorer.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.cortex\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.component_library.scorer.common.MLModelInferenceClient\nimport com.twitter.product_mixer.component_library.scorer.tensorbuilder.ModelInferRequestBuilder\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.logging.Logging\nimport inference.GrpcService.ModelInferRequest\nimport inference.GrpcService.ModelInferResponse.InferOutputTensor\nimport scala.collection.convert.ImplicitConversions.`collection AsScalaIterable`\n\nprivate[scorer] class CortexManagedInferenceServiceTensorScorer[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any]\n](\n  override val identifier: ScorerIdentifier,\n  modelInferRequestBuilder: ModelInferRequestBuilder[\n    Query,\n    Candidate\n  ],\n  resultFeatureExtractors: Seq[FeatureWithExtractor[Query, Candidate, _]],\n  client: MLModelInferenceClient,\n  statsReceiver: StatsReceiver)\n    extends Scorer[Query, Candidate]\n    with Logging {\n\n  require(resultFeatureExtractors.nonEmpty, \"Result Extractors cannot be empty\")\n\n  private val managedServiceRequestFailures = statsReceiver.counter(\"managedServiceRequestFailures\")\n  override val features: Set[Feature[_, _]] =\n    resultFeatureExtractors.map(_.feature).toSet.asInstanceOf[Set[Feature[_, _]]]\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val batchInferRequest: ModelInferRequest = modelInferRequestBuilder(query, candidates)\n\n    val managedServiceResponse: Stitch[Seq[InferOutputTensor]] =\n      client.score(batchInferRequest).map(_.getOutputsList.toSeq).onFailure { e =>\n        error(s\"request to ML Managed Service Failed: $e\")\n        managedServiceRequestFailures.incr()\n      }\n\n    managedServiceResponse.map { responses =>\n      extractResponse(query, candidates.map(_.candidate), responses)\n    }\n  }\n\n  def extractResponse(\n    query: Query,\n    candidates: Seq[Candidate],\n    tensorOutput: Seq[InferOutputTensor]\n  ): Seq[FeatureMap] = {\n    val featureMapBuilders = candidates.map { _ => FeatureMapBuilder.apply() }\n    // Extract the feature for each candidate from the tensor outputs\n    resultFeatureExtractors.foreach {\n      case FeatureWithExtractor(feature, extractor) =>\n        val extractedValues = extractor.apply(query, tensorOutput)\n        if (candidates.size != extractedValues.size) {\n          throw PipelineFailure(\n            IllegalStateFailure,\n            s\"Managed Service returned a different number of $feature than the number of candidates.\" +\n              s\"Returned ${extractedValues.size} scores but there were ${candidates.size} candidates.\"\n          )\n        }\n        // Go through the extracted features list one by one and update the feature map result for each candidate.\n        featureMapBuilders.zip(extractedValues).foreach {\n          case (builder, value) =>\n            builder.add(feature, Some(value))\n        }\n    }\n\n    featureMapBuilders.map(_.build())\n  }\n}\n\ncase class FeatureWithExtractor[\n  -Query <: PipelineQuery,\n  -Candidate <: UniversalNoun[Any],\n  ResultType\n](\n  feature: Feature[Candidate, Option[ResultType]],\n  featureExtractor: ModelFeatureExtractor[Query, ResultType])\n\nclass UnexpectedFeatureTypeException(feature: Feature[_, _])\n    extends UnsupportedOperationException(s\"Unsupported Feature type passed in $feature\")\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/cortex/CortexManagedInferenceServiceTensorScorerBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.cortex\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.component_library.scorer.common.MLModelInferenceClient\nimport com.twitter.product_mixer.component_library.scorer.tensorbuilder.ModelInferRequestBuilder\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CortexManagedInferenceServiceTensorScorerBuilder @Inject() (\n  statsReceiver: StatsReceiver) {\n\n  /**\n   * Builds a configurable Scorer to call into your desired Cortex Managed ML Model Service.\n   *\n   * If your service does not bind an Http.Client implementation, add\n   * [[com.twitter.product_mixer.component_library.module.http.FinagleHttpClientModule]]\n   * to your server module list\n   *\n   * @param scorerIdentifier        Unique identifier for the scorer\n   * @param resultFeatureExtractors The result features an their tensor extractors for each candidate.\n   * @tparam Query Type of pipeline query.\n   * @tparam Candidate Type of candidates to score.\n   * @tparam QueryFeatures type of the query level features consumed by the scorer.\n   * @tparam CandidateFeatures type of the candidate level features consumed by the scorer.\n   */\n  def build[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n    scorerIdentifier: ScorerIdentifier,\n    modelInferRequestBuilder: ModelInferRequestBuilder[\n      Query,\n      Candidate\n    ],\n    resultFeatureExtractors: Seq[FeatureWithExtractor[Query, Candidate, _]],\n    client: MLModelInferenceClient\n  ): Scorer[Query, Candidate] =\n    new CortexManagedInferenceServiceTensorScorer(\n      scorerIdentifier,\n      modelInferRequestBuilder,\n      resultFeatureExtractors,\n      client,\n      statsReceiver.scope(scorerIdentifier.name)\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/cortex/ModelFeatureExtractor.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.cortex\n\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport inference.GrpcService.ModelInferResponse.InferOutputTensor\n\n/**\n * Extractor defining how a Scorer should go from outputted tensors to the individual results\n * for each candidate being scored.\n *\n * @tparam Result the type of the Value being returned.\n * Users can pass in an anonymous function\n */\ntrait ModelFeatureExtractor[-Query <: PipelineQuery, Result] {\n  def apply(query: Query, tensorOutput: Seq[InferOutputTensor]): Seq[Result]\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/cr_ml_ranker/BUILD.bazel",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"cr-ml-ranker/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/cr_ml_ranker\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n    ],\n    exports = [\n        \"cr-ml-ranker/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/feature_hydrator/query/cr_ml_ranker\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/cr_ml_ranker/CrMlRankerScorer.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.cr_ml_ranker\n\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.cr_ml_ranker.CrMlRankerCommonFeatures\nimport com.twitter.product_mixer.component_library.feature_hydrator.query.cr_ml_ranker.CrMlRankerRankingConfig\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject CrMlRankerScore extends Feature[TweetCandidate, Double]\n\n/**\n * Scorer that scores tweets using the Content Recommender ML Light Ranker: http://go/cr-ml-ranker\n */\n@Singleton\nclass CrMlRankerScorer @Inject() (crMlRanker: CrMlRankerScoreStitchClient)\n    extends Scorer[PipelineQuery, TweetCandidate] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"CrMlRanker\")\n\n  override val features: Set[Feature[_, _]] = Set(CrMlRankerScore)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val queryFeatureMap = query.features.getOrElse(FeatureMap.empty)\n    val rankingConfig = queryFeatureMap.get(CrMlRankerRankingConfig)\n    val commonFeatures = queryFeatureMap.get(CrMlRankerCommonFeatures)\n    val userId = query.getRequiredUserId\n\n    val scoresStitch = Stitch.collect(candidates.map { candidateWithFeatures =>\n      crMlRanker\n        .getScore(userId, candidateWithFeatures.candidate, rankingConfig, commonFeatures).map(\n          _.score)\n    })\n    scoresStitch.map { scores =>\n      scores.map { score =>\n        FeatureMapBuilder()\n          .add(CrMlRankerScore, score)\n          .build()\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/cr_ml_ranker/CrMlRankerStitchClient.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.cr_ml_ranker\n\nimport com.twitter.cr_ml_ranker.{thriftscala => t}\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.stitch.SeqGroup\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport com.twitter.util.Return\nimport com.twitter.util.Try\n\ncase class CrMlRankerResult(\n  tweetId: Long,\n  score: Double)\n\nclass CrMlRankerScoreStitchClient(\n  crMLRanker: t.CrMLRanker.MethodPerEndpoint,\n  maxBatchSize: Int) {\n\n  def getScore(\n    userId: Long,\n    tweetCandidate: BaseTweetCandidate,\n    rankingConfig: t.RankingConfig,\n    commonFeatures: t.CommonFeatures\n  ): Stitch[CrMlRankerResult] = {\n    Stitch.call(\n      tweetCandidate,\n      CrMlRankerGroup(\n        userId = userId,\n        rankingConfig = rankingConfig,\n        commonFeatures = commonFeatures\n      )\n    )\n  }\n\n  private case class CrMlRankerGroup(\n    userId: Long,\n    rankingConfig: t.RankingConfig,\n    commonFeatures: t.CommonFeatures)\n      extends SeqGroup[BaseTweetCandidate, CrMlRankerResult] {\n\n    override val maxSize: Int = maxBatchSize\n\n    override protected def run(\n      tweetCandidates: Seq[BaseTweetCandidate]\n    ): Future[Seq[Try[CrMlRankerResult]]] = {\n      val crMlRankerCandidates =\n        tweetCandidates.map { tweetCandidate =>\n          t.RankingCandidate(\n            tweetId = tweetCandidate.id,\n            hydrationContext = Some(\n              t.FeatureHydrationContext.HomeHydrationContext(t\n                .HomeFeatureHydrationContext(tweetAuthor = None)))\n          )\n        }\n\n      val thriftResults = crMLRanker.getRankedResults(\n        t.RankingRequest(\n          requestContext = t.RankingRequestContext(\n            userId = userId,\n            config = rankingConfig\n          ),\n          candidates = crMlRankerCandidates,\n          commonFeatures = commonFeatures.commonFeatures\n        )\n      )\n\n      thriftResults.map { response =>\n        response.scoredTweets.map { scoredTweet =>\n          Return(\n            CrMlRankerResult(\n              tweetId = scoredTweet.tweetId,\n              score = scoredTweet.score\n            )\n          )\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/deepbird/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"cortex-deepbird/thrift/src/main/thrift:thrift-java\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"finatra-internal/mtls/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"scrooge/scrooge-serializer\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/ml/common/base\",\n        \"src/java/com/twitter/ml/prediction/core\",\n        \"src/thrift/com/twitter/ml/prediction_service:prediction_service-java\",\n        \"src/thrift/com/twitter/ml/prediction_service:prediction_service-scala\",\n        \"twml/runtime/src/main/scala/com/twitter/deepbird/runtime/prediction_engine\",\n    ],\n    exports = [\n        \"cortex-deepbird/thrift/src/main/thrift:thrift-java\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"src/java/com/twitter/ml/prediction/core\",\n        \"src/thrift/com/twitter/ml/prediction_service:prediction_service-java\",\n        \"twml/runtime/src/main/scala/com/twitter/deepbird/runtime/prediction_engine\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/deepbird/BaseDeepbirdV2Scorer.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.deepbird\n\nimport com.twitter.product_mixer.core.feature.datarecord.BaseDataRecordFeature\nimport com.twitter.ml.prediction_service.BatchPredictionRequest\nimport com.twitter.ml.prediction_service.BatchPredictionResponse\nimport com.twitter.cortex.deepbird.thriftjava.{ModelSelector => TModelSelector}\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.component_library.scorer.common.ModelSelector\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordConverter\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordExtractor\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.FeaturesScope\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport scala.collection.JavaConverters._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\n\nabstract class BaseDeepbirdV2Scorer[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  QueryFeatures <: BaseDataRecordFeature[Query, _],\n  CandidateFeatures <: BaseDataRecordFeature[Candidate, _],\n  ResultFeatures <: BaseDataRecordFeature[Candidate, _]\n](\n  override val identifier: ScorerIdentifier,\n  modelIdSelector: ModelSelector[Query],\n  queryFeatures: FeaturesScope[QueryFeatures],\n  candidateFeatures: FeaturesScope[CandidateFeatures],\n  resultFeatures: Set[ResultFeatures])\n    extends Scorer[Query, Candidate] {\n\n  private val queryDataRecordConverter = new DataRecordConverter(queryFeatures)\n  private val candidateDataRecordConverter = new DataRecordConverter(candidateFeatures)\n  private val resultDataRecordExtractor = new DataRecordExtractor(resultFeatures)\n\n  require(resultFeatures.nonEmpty, \"Result features cannot be empty\")\n  override val features: Set[Feature[_, _]] = resultFeatures.asInstanceOf[Set[Feature[_, _]]]\n  def getBatchPredictions(\n    request: BatchPredictionRequest,\n    modelSelector: TModelSelector\n  ): Future[BatchPredictionResponse]\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    // Convert all candidate feature maps to java datarecords then to scala datarecords.\n    val thriftCandidateDataRecords = candidates.map { candidate =>\n      candidateDataRecordConverter.toDataRecord(candidate.features)\n    }\n\n    val request = new BatchPredictionRequest(thriftCandidateDataRecords.asJava)\n\n    // Convert the query feature map to data record if available.\n    query.features.foreach { featureMap =>\n      request.setCommonFeatures(queryDataRecordConverter.toDataRecord(featureMap))\n    }\n\n    val modelSelector = modelIdSelector\n      .apply(query).map { id =>\n        val selector = new TModelSelector()\n        selector.setId(id)\n        selector\n      }.orNull\n\n    Stitch.callFuture(getBatchPredictions(request, modelSelector)).map { response =>\n      val dataRecords = Option(response.predictions).map(_.asScala).getOrElse(Seq.empty)\n      buildResults(candidates, dataRecords)\n    }\n  }\n\n  private def buildResults(\n    candidates: Seq[CandidateWithFeatures[Candidate]],\n    dataRecords: Seq[DataRecord]\n  ): Seq[FeatureMap] = {\n    if (dataRecords.size != candidates.size) {\n      throw PipelineFailure(IllegalStateFailure, \"Result Size mismatched candidates size\")\n    }\n\n    dataRecords.map { resultDataRecord =>\n      resultDataRecordExtractor.fromDataRecord(resultDataRecord)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/deepbird/DeepbirdV2PredictionServerScorer.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.deepbird\n\nimport com.twitter.cortex.deepbird.{thriftjava => t}\nimport com.twitter.ml.prediction_service.BatchPredictionRequest\nimport com.twitter.ml.prediction_service.BatchPredictionResponse\nimport com.twitter.product_mixer.component_library.scorer.common.ModelSelector\nimport com.twitter.product_mixer.core.feature.datarecord.BaseDataRecordFeature\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.FeaturesScope\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.util.Future\n\n/**\n * Configurable Scorer that calls any Deepbird Prediction Service thrift.\n * @param identifier Unique identifier for the scorer\n * @param predictionService The Prediction Thrift Service\n * @param modelSelector Model ID Selector to decide which model to select, can also be represented\n *                        as an anonymous function: { query: Query => Some(\"Ex\") }\n * @param queryFeatures The Query Features to convert and pass to the deepbird model.\n * @param candidateFeatures The Candidate Features to convert and pass to the deepbird model.\n * @param resultFeatures The Candidate features returned by the model.\n * @tparam Query Type of pipeline query.\n * @tparam Candidate Type of candidates to score.\n * @tparam QueryFeatures type of the query level features consumed by the scorer.\n * @tparam CandidateFeatures type of the candidate level features consumed by the scorer.\n * @tparam ResultFeatures type of the candidate level features returned by the scorer.\n */\ncase class DeepbirdV2PredictionServerScorer[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  QueryFeatures <: BaseDataRecordFeature[Query, _],\n  CandidateFeatures <: BaseDataRecordFeature[Candidate, _],\n  ResultFeatures <: BaseDataRecordFeature[Candidate, _]\n](\n  override val identifier: ScorerIdentifier,\n  predictionService: t.DeepbirdPredictionService.ServiceToClient,\n  modelSelector: ModelSelector[Query],\n  queryFeatures: FeaturesScope[QueryFeatures],\n  candidateFeatures: FeaturesScope[CandidateFeatures],\n  resultFeatures: Set[ResultFeatures])\n    extends BaseDeepbirdV2Scorer[\n      Query,\n      Candidate,\n      QueryFeatures,\n      CandidateFeatures,\n      ResultFeatures\n    ](identifier, modelSelector, queryFeatures, candidateFeatures, resultFeatures) {\n\n  override def getBatchPredictions(\n    request: BatchPredictionRequest,\n    modelSelector: t.ModelSelector\n  ): Future[BatchPredictionResponse] =\n    predictionService.batchPredictFromModel(request, modelSelector)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/deepbird/LollyPredictionEngineScorer.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.deepbird\n\nimport com.twitter.ml.prediction.core.PredictionEngine\nimport com.twitter.ml.prediction_service.PredictionRequest\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.datarecord.BaseDataRecordFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordConverter\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.DataRecordExtractor\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.FeaturesScope\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Scorer that locally loads a Deepbird model.\n * @param identifier Unique identifier for the scorer\n * @param predictionEngine Prediction Engine hosting the Deepbird model.\n * @param candidateFeatures The Candidate Features to convert and pass to the deepbird model.\n * @param resultFeatures The Candidate features returned by the model.\n * @tparam Query Type of pipeline query.\n * @tparam Candidate Type of candidates to score.\n * @tparam QueryFeatures type of the query level features consumed by the scorer.\n * @tparam CandidateFeatures type of the candidate level features consumed by the scorer.\n * @tparam ResultFeatures type of the candidate level features returned by the scorer.\n */\nclass LollyPredictionEngineScorer[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  QueryFeatures <: BaseDataRecordFeature[Query, _],\n  CandidateFeatures <: BaseDataRecordFeature[Candidate, _],\n  ResultFeatures <: BaseDataRecordFeature[Candidate, _]\n](\n  override val identifier: ScorerIdentifier,\n  predictionEngine: PredictionEngine,\n  candidateFeatures: FeaturesScope[CandidateFeatures],\n  resultFeatures: Set[ResultFeatures])\n    extends Scorer[Query, Candidate] {\n\n  private val dataRecordAdapter = new DataRecordConverter(candidateFeatures)\n\n  require(resultFeatures.nonEmpty, \"Result features cannot be empty\")\n  override val features: Set[Feature[_, _]] = resultFeatures.asInstanceOf[Set[Feature[_, _]]]\n\n  private val resultsDataRecordExtractor = new DataRecordExtractor(resultFeatures)\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val featureMaps = candidates.map { candidateWithFeatures =>\n      val dataRecord = dataRecordAdapter.toDataRecord(candidateWithFeatures.features)\n      val predictionResponse = predictionEngine.apply(new PredictionRequest(dataRecord), true)\n      resultsDataRecordExtractor.fromDataRecord(predictionResponse.getPrediction)\n    }\n    Stitch.value(featureMaps)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/deepbird/TensorflowPredictionEngineScorer.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.deepbird\n\nimport com.twitter.cortex.deepbird.runtime.prediction_engine.TensorflowPredictionEngine\nimport com.twitter.cortex.deepbird.thriftjava.ModelSelector\nimport com.twitter.ml.prediction_service.BatchPredictionRequest\nimport com.twitter.ml.prediction_service.BatchPredictionResponse\nimport com.twitter.product_mixer.core.feature.datarecord.BaseDataRecordFeature\nimport com.twitter.product_mixer.core.feature.featuremap.datarecord.FeaturesScope\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.util.Future\n\n/**\n * Configurable Scorer that calls a TensorflowPredictionEngine.\n * @param identifier Unique identifier for the scorer\n * @param tensorflowPredictionEngine The TensorFlow Prediction Engine\n * @param queryFeatures The Query Features to convert and pass to the deepbird model.\n * @param candidateFeatures The Candidate Features to convert and pass to the deepbird model.\n * @param resultFeatures The Candidate features returned by the model.\n * @tparam Query Type of pipeline query.\n * @tparam Candidate Type of candidates to score.\n * @tparam QueryFeatures type of the query level features consumed by the scorer.\n * @tparam CandidateFeatures type of the candidate level features consumed by the scorer.\n * @tparam ResultFeatures type of the candidate level features returned by the scorer.\n */\nclass TensorflowPredictionEngineScorer[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  QueryFeatures <: BaseDataRecordFeature[Query, _],\n  CandidateFeatures <: BaseDataRecordFeature[Candidate, _],\n  ResultFeatures <: BaseDataRecordFeature[Candidate, _]\n](\n  override val identifier: ScorerIdentifier,\n  tensorflowPredictionEngine: TensorflowPredictionEngine,\n  queryFeatures: FeaturesScope[QueryFeatures],\n  candidateFeatures: FeaturesScope[CandidateFeatures],\n  resultFeatures: Set[ResultFeatures])\n    extends BaseDeepbirdV2Scorer[\n      Query,\n      Candidate,\n      QueryFeatures,\n      CandidateFeatures,\n      ResultFeatures\n    ](\n      identifier,\n      { _: Query =>\n        None\n      },\n      queryFeatures,\n      candidateFeatures,\n      resultFeatures) {\n\n  override def getBatchPredictions(\n    request: BatchPredictionRequest,\n    modelSelector: ModelSelector\n  ): Future[BatchPredictionResponse] = tensorflowPredictionEngine.getBatchPrediction(request)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/param_gated/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/param_gated/ParamGatedScorer.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.param_gated\n\nimport com.twitter.product_mixer.component_library.scorer.param_gated.ParamGatedScorer.IdentifierPrefix\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * A [[scorer]] with [[Conditionally]] based on a [[Param]]\n *\n * @param enabledParam the param to turn this [[scorer]] on and off\n * @param scorer the underlying [[scorer]] to run when `enabledParam` is true\n * @tparam Query The domain model for the query or request\n * @tparam Result The type of the candidates\n */\ncase class ParamGatedScorer[-Query <: PipelineQuery, Result <: UniversalNoun[Any]](\n  enabledParam: Param[Boolean],\n  scorer: Scorer[Query, Result])\n    extends Scorer[Query, Result]\n    with Conditionally[Query] {\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\n    IdentifierPrefix + scorer.identifier.name)\n  override val alerts: Seq[Alert] = scorer.alerts\n  override val features: Set[Feature[_, _]] = scorer.features\n  override def onlyIf(query: Query): Boolean =\n    Conditionally.and(query, scorer, query.params(enabledParam))\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Result]]\n  ): Stitch[Seq[FeatureMap]] = scorer(query, candidates)\n}\n\nobject ParamGatedScorer {\n  val IdentifierPrefix = \"ParamGated\"\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/qualityfactor_gated/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/qualityfactor_gated/QualityFactorGatedScorer.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.qualityfactor_gated\n\nimport com.twitter.product_mixer.component_library.scorer.qualityfactor_gated.QualityFactorGatedScorer.IdentifierPrefix\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * A [[scorer]] with [[Conditionally]] based on quality factor value and threshold\n *\n * @param qualityFactorThreshold quliaty factor threshold that turn off the scorer\n * @param pipelineIdentifier identifier of the pipeline that quality factor is based on\n * @param scorer the underlying [[scorer]] to run when `enabledParam` is true\n * @tparam Query The domain model for the query or request\n * @tparam Result The type of the candidates\n */\ncase class QualityFactorGatedScorer[\n  -Query <: PipelineQuery with HasQualityFactorStatus,\n  Result <: UniversalNoun[Any]\n](\n  pipelineIdentifier: ComponentIdentifier,\n  qualityFactorThresholdParam: Param[Double],\n  scorer: Scorer[Query, Result])\n    extends Scorer[Query, Result]\n    with Conditionally[Query] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\n    IdentifierPrefix + scorer.identifier.name)\n\n  override val alerts: Seq[Alert] = scorer.alerts\n\n  override val features: Set[Feature[_, _]] = scorer.features\n\n  override def onlyIf(query: Query): Boolean =\n    Conditionally.and(\n      query,\n      scorer,\n      query.getQualityFactorCurrentValue(pipelineIdentifier) >= query.params(\n        qualityFactorThresholdParam))\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Result]]\n  ): Stitch[Seq[FeatureMap]] = scorer(query, candidates)\n}\n\nobject QualityFactorGatedScorer {\n  val IdentifierPrefix = \"QualityFactorGated\"\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/thrift/com/twitter/ml/api:embedding-scala\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder/BooleanInferInputTensorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.tensorbuilder\n\nimport inference.GrpcService.ModelInferRequest.InferInputTensor\n\ncase object BooleanInferInputTensorBuilder extends InferInputTensorBuilder[Boolean] {\n  def apply(\n    featureName: String,\n    featureValues: Seq[Boolean]\n  ): Seq[InferInputTensor] = {\n    val tensorShape = Seq(featureValues.size, 1)\n    InferInputTensorBuilder.buildBoolInferInputTensor(featureName, featureValues, tensorShape)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder/BytesInferInputTensorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.tensorbuilder\n\nimport inference.GrpcService.ModelInferRequest.InferInputTensor\n\ncase object BytesInferInputTensorBuilder extends InferInputTensorBuilder[String] {\n  def apply(\n    featureName: String,\n    featureValues: Seq[String]\n  ): Seq[InferInputTensor] = {\n    val tensorShape = Seq(featureValues.size, 1)\n    InferInputTensorBuilder.buildBytesInferInputTensor(featureName, featureValues, tensorShape)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder/CandidateInferInputTensorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.tensorbuilder\n\nimport com.twitter.ml.api.thriftscala.FloatTensor\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.ModelFeatureName\nimport com.twitter.product_mixer.core.feature.featuremap.featurestorev1.FeatureStoreV1FeatureMap._\nimport com.twitter.product_mixer.core.feature.featurestorev1.FeatureStoreV1CandidateFeature\nimport com.twitter.product_mixer.core.feature.featurestorev1.FeatureStoreV1QueryFeature\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport inference.GrpcService.ModelInferRequest.InferInputTensor\n\nclass CandidateInferInputTensorBuilder[-Candidate <: UniversalNoun[Any], +Value](\n  builder: InferInputTensorBuilder[Value],\n  features: Set[_ <: Feature[Candidate, _] with ModelFeatureName]) {\n  def apply(\n    candidates: Seq[CandidateWithFeatures[Candidate]],\n  ): Seq[InferInputTensor] = {\n    features.flatMap { feature =>\n      val featureValues: Seq[Value] = feature match {\n        case feature: FeatureStoreV1CandidateFeature[_, Candidate, _, Value] =>\n          candidates.map(_.features.getFeatureStoreV1CandidateFeature(feature))\n        case feature: FeatureStoreV1QueryFeature[_, _, _] =>\n          throw new UnexpectedFeatureTypeException(feature)\n        case feature: FeatureWithDefaultOnFailure[Candidate, Value] =>\n          candidates.map(_.features.getTry(feature).toOption.getOrElse(feature.defaultValue))\n        case feature: Feature[Candidate, Value] =>\n          candidates.map(_.features.get(feature))\n      }\n      builder.apply(feature.featureName, featureValues)\n    }.toSeq\n  }\n}\n\ncase class CandidateBooleanInferInputTensorBuilder[-Candidate <: UniversalNoun[Any]](\n  features: Set[_ <: Feature[Candidate, Boolean] with ModelFeatureName])\n    extends CandidateInferInputTensorBuilder[Candidate, Boolean](\n      BooleanInferInputTensorBuilder,\n      features)\n\ncase class CandidateBytesInferInputTensorBuilder[-Candidate <: UniversalNoun[Any]](\n  features: Set[_ <: Feature[Candidate, String] with ModelFeatureName])\n    extends CandidateInferInputTensorBuilder[Candidate, String](\n      BytesInferInputTensorBuilder,\n      features)\n\ncase class CandidateFloat32InferInputTensorBuilder[-Candidate <: UniversalNoun[Any]](\n  features: Set[_ <: Feature[Candidate, _ <: AnyVal] with ModelFeatureName])\n    extends CandidateInferInputTensorBuilder[Candidate, AnyVal](\n      Float32InferInputTensorBuilder,\n      features)\n\ncase class CandidateFloatTensorInferInputTensorBuilder[-Candidate <: UniversalNoun[Any]](\n  features: Set[_ <: Feature[Candidate, FloatTensor] with ModelFeatureName])\n    extends CandidateInferInputTensorBuilder[Candidate, FloatTensor](\n      FloatTensorInferInputTensorBuilder,\n      features)\n\ncase class CandidateInt64InferInputTensorBuilder[-Candidate <: UniversalNoun[Any]](\n  features: Set[_ <: Feature[Candidate, _ <: AnyVal] with ModelFeatureName])\n    extends CandidateInferInputTensorBuilder[Candidate, AnyVal](\n      Int64InferInputTensorBuilder,\n      features)\n\ncase class CandidateSparseMapInferInputTensorBuilder[-Candidate <: UniversalNoun[Any]](\n  features: Set[_ <: Feature[Candidate, Option[Map[Int, Double]]] with ModelFeatureName])\n    extends CandidateInferInputTensorBuilder[Candidate, Option[Map[Int, Double]]](\n      SparseMapInferInputTensorBuilder,\n      features)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder/Float32InferInputTensorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.tensorbuilder\n\nimport inference.GrpcService.ModelInferRequest.InferInputTensor\n\ncase object Float32InferInputTensorBuilder extends InferInputTensorBuilder[AnyVal] {\n\n  private def toFloat(x: AnyVal): Float = {\n    x match {\n      case y: Float => y\n      case y: Int => y.toFloat\n      case y: Long => y.toFloat\n      case y: Double => y.toFloat\n      case y => throw new UnexpectedDataTypeException(y, this)\n    }\n  }\n\n  def apply(\n    featureName: String,\n    featureValues: Seq[AnyVal]\n  ): Seq[InferInputTensor] = {\n    val tensorShape = Seq(featureValues.size, 1)\n    InferInputTensorBuilder.buildFloat32InferInputTensor(\n      featureName,\n      featureValues.map(toFloat),\n      tensorShape)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder/FloatTensorInferInputTensorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.tensorbuilder\n\nimport com.twitter.ml.api.thriftscala.FloatTensor\nimport inference.GrpcService.ModelInferRequest.InferInputTensor\n\ncase object FloatTensorInferInputTensorBuilder extends InferInputTensorBuilder[FloatTensor] {\n\n  private[tensorbuilder] def extractTensorShape(featureValues: Seq[FloatTensor]): Seq[Int] = {\n    val headFloatTensor = featureValues.head\n    if (headFloatTensor.shape.isEmpty) {\n      Seq(\n        featureValues.size,\n        featureValues.head.floats.size\n      )\n    } else {\n      Seq(featureValues.size) ++ headFloatTensor.shape.get.map(_.toInt)\n    }\n  }\n\n  def apply(\n    featureName: String,\n    featureValues: Seq[FloatTensor]\n  ): Seq[InferInputTensor] = {\n    if (featureValues.isEmpty) throw new EmptyFloatTensorException(featureName)\n    val tensorShape = extractTensorShape(featureValues)\n    val floatValues = featureValues.flatMap { featureValue =>\n      featureValue.floats.map(_.toFloat)\n    }\n    InferInputTensorBuilder.buildFloat32InferInputTensor(featureName, floatValues, tensorShape)\n  }\n}\nclass EmptyFloatTensorException(featureName: String)\n    extends RuntimeException(s\"FloatTensor in feature $featureName is empty!\")\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder/InferInputTensorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.tensorbuilder\n\nimport com.google.protobuf.ByteString\nimport com.twitter.product_mixer.core.feature.Feature\nimport inference.GrpcService.InferTensorContents\nimport inference.GrpcService.ModelInferRequest.InferInputTensor\n\n// This class contains most of common versions at Twitter, but in the future we can add more:\n// https://github.com/kserve/kserve/blob/master/docs/predict-api/v2/required_api.md#tensor-data-1\n\ntrait InferInputTensorBuilder[Value] {\n\n  def apply(\n    featureName: String,\n    featureValues: Seq[Value]\n  ): Seq[InferInputTensor]\n\n}\n\nobject InferInputTensorBuilder {\n\n  def checkTensorShapeMatchesValueLength(\n    featureName: String,\n    featureValues: Seq[Any],\n    tensorShape: Seq[Int]\n  ): Unit = {\n    val featureValuesSize = featureValues.size\n    val tensorShapeSize = tensorShape.product\n    if (featureValuesSize != tensorShapeSize) {\n      throw new FeatureValuesAndShapeMismatchException(\n        featureName,\n        featureValuesSize,\n        tensorShapeSize)\n    }\n  }\n\n  def buildBoolInferInputTensor(\n    featureName: String,\n    featureValues: Seq[Boolean],\n    tensorShape: Seq[Int]\n  ): Seq[InferInputTensor] = {\n\n    checkTensorShapeMatchesValueLength(featureName, featureValues, tensorShape)\n\n    val inputTensorBuilder = InferInputTensor.newBuilder().setName(featureName)\n    tensorShape.foreach { shape =>\n      inputTensorBuilder.addShape(shape)\n    }\n    val inputTensor = inputTensorBuilder\n      .setDatatype(\"BOOL\")\n      .setContents {\n        val contents = InferTensorContents.newBuilder()\n        featureValues.foreach { featureValue =>\n          contents.addBoolContents(featureValue)\n        }\n        contents\n      }\n      .build()\n    Seq(inputTensor)\n  }\n\n  def buildBytesInferInputTensor(\n    featureName: String,\n    featureValues: Seq[String],\n    tensorShape: Seq[Int]\n  ): Seq[InferInputTensor] = {\n\n    checkTensorShapeMatchesValueLength(featureName, featureValues, tensorShape)\n\n    val inputTensorBuilder = InferInputTensor.newBuilder().setName(featureName)\n    tensorShape.foreach { shape =>\n      inputTensorBuilder.addShape(shape)\n    }\n    val inputTensor = inputTensorBuilder\n      .setDatatype(\"BYTES\")\n      .setContents {\n        val contents = InferTensorContents.newBuilder()\n        featureValues.foreach { featureValue =>\n          val featureValueBytes = ByteString.copyFromUtf8(featureValue)\n          contents.addByteContents(featureValueBytes)\n        }\n        contents\n      }\n      .build()\n    Seq(inputTensor)\n  }\n\n  def buildFloat32InferInputTensor(\n    featureName: String,\n    featureValues: Seq[Float],\n    tensorShape: Seq[Int]\n  ): Seq[InferInputTensor] = {\n\n    checkTensorShapeMatchesValueLength(featureName, featureValues, tensorShape)\n\n    val inputTensorBuilder = InferInputTensor.newBuilder().setName(featureName)\n    tensorShape.foreach { shape =>\n      inputTensorBuilder.addShape(shape)\n    }\n    val inputTensor = inputTensorBuilder\n      .setDatatype(\"FP32\")\n      .setContents {\n        val contents = InferTensorContents.newBuilder()\n        featureValues.foreach { featureValue =>\n          contents.addFp32Contents(featureValue.floatValue)\n        }\n        contents\n      }\n      .build()\n    Seq(inputTensor)\n  }\n\n  def buildInt64InferInputTensor(\n    featureName: String,\n    featureValues: Seq[Long],\n    tensorShape: Seq[Int]\n  ): Seq[InferInputTensor] = {\n\n    checkTensorShapeMatchesValueLength(featureName, featureValues, tensorShape)\n\n    val inputTensorBuilder = InferInputTensor.newBuilder().setName(featureName)\n    tensorShape.foreach { shape =>\n      inputTensorBuilder.addShape(shape)\n    }\n    val inputTensor = inputTensorBuilder\n      .setDatatype(\"INT64\")\n      .setContents {\n        val contents = InferTensorContents.newBuilder()\n        featureValues.foreach { featureValue =>\n          contents.addInt64Contents(featureValue)\n        }\n        contents\n      }\n      .build()\n    Seq(inputTensor)\n  }\n}\n\nclass UnexpectedFeatureTypeException(feature: Feature[_, _])\n    extends UnsupportedOperationException(s\"Unsupported Feature type passed in $feature\")\n\nclass FeatureValuesAndShapeMismatchException(\n  featureName: String,\n  featureValuesSize: Int,\n  tensorShapeSize: Int)\n    extends UnsupportedOperationException(\n      s\"Feature $featureName has mismatching FeatureValues (size: $featureValuesSize) and TensorShape (size: $tensorShapeSize)!\")\n\nclass UnexpectedDataTypeException[T](value: T, builder: InferInputTensorBuilder[_])\n    extends UnsupportedOperationException(\n      s\"Unsupported data type ${value} passed in at ${builder.getClass.toString}\")\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder/Int64InferInputTensorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.tensorbuilder\n\nimport com.twitter.ml.featurestore.lib.Discrete\nimport inference.GrpcService.ModelInferRequest.InferInputTensor\n\ncase object Int64InferInputTensorBuilder extends InferInputTensorBuilder[AnyVal] {\n\n  private def toLong(x: AnyVal): Long = {\n    x match {\n      case y: Int => y.toLong\n      case y: Long => y\n      case y: Discrete => y.value\n      case y => throw new UnexpectedDataTypeException(y, this)\n    }\n  }\n  def apply(\n    featureName: String,\n    featureValues: Seq[AnyVal]\n  ): Seq[InferInputTensor] = {\n    val tensorShape = Seq(featureValues.size, 1)\n    InferInputTensorBuilder.buildInt64InferInputTensor(\n      featureName,\n      featureValues.map(toLong),\n      tensorShape)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder/ModelInferRequestBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.tensorbuilder\n\nimport com.twitter.product_mixer.component_library.scorer.common.ModelSelector\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport inference.GrpcService.InferParameter\nimport inference.GrpcService.ModelInferRequest\nimport scala.collection.JavaConverters._\n\nclass ModelInferRequestBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]](\n  queryInferInputTensorBuilders: Seq[QueryInferInputTensorBuilder[Query, Any]],\n  candidateInferInputTensorBuilders: Seq[\n    CandidateInferInputTensorBuilder[Candidate, Any]\n  ],\n  modelSignatureName: String,\n  modelSelector: ModelSelector[Query]) {\n\n  private val modelSignature: InferParameter =\n    InferParameter.newBuilder().setStringParam(modelSignatureName).build()\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]],\n  ): ModelInferRequest = {\n    val inferRequest = ModelInferRequest\n      .newBuilder()\n      .putParameters(\"signature_name\", modelSignature)\n    modelSelector.apply(query).foreach { modelName =>\n      inferRequest.setModelName(modelName)\n    }\n    queryInferInputTensorBuilders.foreach { builder =>\n      inferRequest.addAllInputs(builder(query).asJava)\n    }\n    candidateInferInputTensorBuilders.foreach { builder =>\n      inferRequest.addAllInputs(builder(candidates).asJava)\n    }\n    inferRequest.build()\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder/QueryInferInputTensorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.tensorbuilder\n\nimport com.twitter.ml.api.thriftscala.FloatTensor\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.ModelFeatureName\nimport com.twitter.product_mixer.core.feature.featuremap.featurestorev1.FeatureStoreV1FeatureMap._\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featurestorev1.FeatureStoreV1CandidateFeature\nimport com.twitter.product_mixer.core.feature.featurestorev1.FeatureStoreV1QueryFeature\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport inference.GrpcService.ModelInferRequest.InferInputTensor\n\nclass QueryInferInputTensorBuilder[-Query <: PipelineQuery, +Value](\n  builder: InferInputTensorBuilder[Value],\n  features: Set[_ <: Feature[Query, _] with ModelFeatureName]) {\n  def apply(query: Query): Seq[InferInputTensor] = {\n    val featureMap = query.features.getOrElse(FeatureMap.empty)\n    features.flatMap { feature =>\n      val queryFeatureValue: Value = feature match {\n        case feature: FeatureStoreV1QueryFeature[Query, _, Value] =>\n          featureMap.getFeatureStoreV1QueryFeature(feature)\n        case feature: FeatureStoreV1CandidateFeature[Query, _, _, Value] =>\n          throw new UnexpectedFeatureTypeException(feature)\n        case feature: FeatureWithDefaultOnFailure[Query, Value] =>\n          featureMap.getTry(feature).toOption.getOrElse(feature.defaultValue)\n        case feature: Feature[Query, Value] =>\n          featureMap.get(feature)\n      }\n      builder.apply(feature.featureName, Seq(queryFeatureValue))\n    }.toSeq\n  }\n}\n\ncase class QueryBooleanInferInputTensorBuilder[-Query <: PipelineQuery](\n  features: Set[_ <: Feature[Query, Boolean] with ModelFeatureName])\n    extends QueryInferInputTensorBuilder[Query, Boolean](BooleanInferInputTensorBuilder, features)\n\ncase class QueryBytesInferInputTensorBuilder[-Query <: PipelineQuery](\n  features: Set[_ <: Feature[Query, String] with ModelFeatureName])\n    extends QueryInferInputTensorBuilder[Query, String](BytesInferInputTensorBuilder, features)\n\ncase class QueryFloat32InferInputTensorBuilder[-Query <: PipelineQuery](\n  features: Set[_ <: Feature[Query, _ <: AnyVal] with ModelFeatureName])\n    extends QueryInferInputTensorBuilder[Query, AnyVal](Float32InferInputTensorBuilder, features)\n\ncase class QueryFloatTensorInferInputTensorBuilder[-Query <: PipelineQuery](\n  features: Set[_ <: Feature[Query, FloatTensor] with ModelFeatureName])\n    extends QueryInferInputTensorBuilder[Query, FloatTensor](\n      FloatTensorInferInputTensorBuilder,\n      features)\n\ncase class QueryInt64InferInputTensorBuilder[-Query <: PipelineQuery](\n  features: Set[_ <: Feature[Query, _ <: AnyVal] with ModelFeatureName])\n    extends QueryInferInputTensorBuilder[Query, AnyVal](Int64InferInputTensorBuilder, features)\n\ncase class QuerySparseMapInferInputTensorBuilder[-Query <: PipelineQuery](\n  features: Set[_ <: Feature[Query, Option[Map[Int, Double]]] with ModelFeatureName])\n    extends QueryInferInputTensorBuilder[Query, Option[Map[Int, Double]]](\n      SparseMapInferInputTensorBuilder,\n      features)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tensorbuilder/SparseMapInferInputTensorBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.tensorbuilder\n\nimport inference.GrpcService.InferTensorContents\nimport inference.GrpcService.ModelInferRequest.InferInputTensor\n\ncase object SparseMapInferInputTensorBuilder\n    extends InferInputTensorBuilder[Option[Map[Int, Double]]] {\n\n  private final val batchFeatureNameSuffix: String = \"batch\"\n  private final val keyFeatureNameSuffix: String = \"key\"\n  private final val valueFeatureNameSuffix: String = \"value\"\n\n  def apply(\n    featureName: String,\n    featureValues: Seq[Option[Map[Int, Double]]]\n  ): Seq[InferInputTensor] = {\n    val batchIdsTensorContents = InferTensorContents.newBuilder()\n    val sparseKeysTensorContents = InferTensorContents.newBuilder()\n    val sparseValuesTensorContents = InferTensorContents.newBuilder()\n    featureValues.zipWithIndex.foreach {\n      case (featureValueOption, batchIndex) =>\n        featureValueOption.foreach { featureValue =>\n          featureValue.foreach {\n            case (sparseKey, sparseValue) =>\n              batchIdsTensorContents.addInt64Contents(batchIndex.toLong)\n              sparseKeysTensorContents.addInt64Contents(sparseKey.toLong)\n              sparseValuesTensorContents.addFp32Contents(sparseValue.floatValue)\n          }\n        }\n    }\n\n    val batchIdsInputTensor = InferInputTensor\n      .newBuilder()\n      .setName(Seq(featureName, batchFeatureNameSuffix).mkString(\"_\"))\n      .addShape(batchIdsTensorContents.getInt64ContentsCount)\n      .addShape(1)\n      .setDatatype(\"INT64\")\n      .setContents(batchIdsTensorContents)\n      .build()\n\n    val sparseKeysInputTensor = InferInputTensor\n      .newBuilder()\n      .setName(Seq(featureName, keyFeatureNameSuffix).mkString(\"_\"))\n      .addShape(sparseKeysTensorContents.getInt64ContentsCount)\n      .addShape(1)\n      .setDatatype(\"INT64\")\n      .setContents(sparseKeysTensorContents)\n      .build()\n\n    val sparseValuesInputTensor = InferInputTensor\n      .newBuilder()\n      .setName(Seq(featureName, valueFeatureNameSuffix).mkString(\"_\"))\n      .addShape(sparseValuesTensorContents.getFp32ContentsCount)\n      .addShape(1)\n      .setDatatype(\"FP32\")\n      .setContents(sparseValuesTensorContents)\n      .build()\n\n    Seq(batchIdsInputTensor, sparseKeysInputTensor, sparseValuesInputTensor)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tweet_tlx/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/thrift/com/twitter/ml/featurestore/timelines:ml-features-timelines-scala\",\n        \"src/thrift/com/twitter/timelinescorer:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/server/internal:thrift-scala\",\n        \"strato/config/columns/ml/featureStore:featureStore-strato-client\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/thrift/com/twitter/ml/featurestore/timelines:ml-features-timelines-scala\",\n        \"src/thrift/com/twitter/timelinescorer:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/server/internal:thrift-scala\",\n        \"strato/config/columns/ml/featureStore:featureStore-strato-client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tweet_tlx/TweetTLXStratoScorer.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.tweet_tlx\n\nimport com.twitter.ml.featurestore.timelines.thriftscala.TimelineScorerScoreView\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.Fetch.Result\nimport com.twitter.strato.generated.client.ml.featureStore.TimelineScorerTweetScoresV1ClientColumn\nimport com.twitter.timelinescorer.thriftscala.v1\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Score Tweets via Timeline Scorer (TLX) Strato API\n *\n * @note This results in an additional hop through Strato Server\n * @note This is the [[Scorer]] version of\n * [[com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_tlx.TweetTLXScoreCandidateFeatureHydrator]]\n */\n@Singleton\nclass TweetTLXStratoScorer @Inject() (column: TimelineScorerTweetScoresV1ClientColumn)\n    extends Scorer[PipelineQuery, TweetCandidate] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"TweetTLX\")\n\n  override val features: Set[Feature[_, _]] = Set(TLXScore)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = query.getOptionalUserId match {\n    case Some(userId) => getScoredTweetsFromTLX(userId, candidates.map(_.candidate))\n    case _ =>\n      val defaultFeatureMap = FeatureMapBuilder().add(TLXScore, None).build()\n      Stitch.value(candidates.map(_ => defaultFeatureMap))\n  }\n\n  def getScoredTweetsFromTLX(\n    userId: Long,\n    tweetCandidates: Seq[TweetCandidate]\n  ): Stitch[Seq[FeatureMap]] = Stitch.collect(tweetCandidates.map { candidate =>\n    column.fetcher\n      .fetch(candidate.id, TimelineScorerScoreView(Some(userId)))\n      .map {\n        case Result(Some(v1.ScoredTweet(_, score, _, _)), _) =>\n          FeatureMapBuilder()\n            .add(TLXScore, score)\n            .build()\n        case fetchResult => throw new Exception(s\"Invalid response from TLX: ${fetchResult.v}\")\n      }\n  })\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/scorer/tweet_tlx/TweetTLXThriftScorer.scala",
    "content": "package com.twitter.product_mixer.component_library.scorer.tweet_tlx\n\nimport com.twitter.product_mixer.component_library.model.candidate.TweetCandidate\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelinescorer.thriftscala.v1\nimport com.twitter.timelinescorer.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * @note This Feature is shared with\n * [[com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_tlx.TweetTLXScoreCandidateFeatureHydrator]]\n * and\n * [[com.twitter.product_mixer.component_library.scorer.tweet_tlx.TweetTLXStratoScorer]]\n * as the these components should not be used at the same time by the same Product\n */\nobject TLXScore extends FeatureWithDefaultOnFailure[TweetCandidate, Option[Double]] {\n  override val defaultValue = None\n}\n\n/**\n * Score Tweets via Timeline Scorer (TLX) Thrift API\n *\n * @note This is the [[Scorer]] version of\n * [[com.twitter.product_mixer.component_library.feature_hydrator.candidate.tweet_tlx.TweetTLXScoreCandidateFeatureHydrator]]\n */\n@Singleton\nclass TweetTLXThriftScorer @Inject() (timelineScorerClient: t.TimelineScorer.MethodPerEndpoint)\n    extends Scorer[PipelineQuery, TweetCandidate] {\n\n  override val identifier: ScorerIdentifier = ScorerIdentifier(\"TLX\")\n\n  override val features: Set[Feature[_, _]] = Set(TLXScore)\n\n  override def apply(\n    query: PipelineQuery,\n    candidates: Seq[CandidateWithFeatures[TweetCandidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    val userId = query.getOptionalUserId\n    val tweetScoringQuery = v1.TweetScoringQuery(\n      predictionPipeline = v1.PredictionPipeline.Recap,\n      tweetIds = candidates.map(_.candidate.id))\n\n    val tweetScoringRequest = t.TweetScoringRequest.V1(\n      v1.TweetScoringRequest(\n        tweetScoringRequestContext = Some(v1.TweetScoringRequestContext(userId = userId)),\n        tweetScoringQueries = Some(Seq(tweetScoringQuery)),\n        retrieveFeatures = Some(false)\n      ))\n\n    Stitch.callFuture(timelineScorerClient.getTweetScores(tweetScoringRequest)).map {\n      case t.TweetScoringResponse.V1(response) =>\n        val tweetIdScoreMap = response.tweetScoringResults\n          .flatMap {\n            _.headOption.map {\n              _.scoredTweets.flatMap(tweet => tweet.tweetId.map(_ -> tweet.score))\n            }\n          }.getOrElse(Seq.empty).toMap\n\n        candidates.map { candidateWithFeatures =>\n          val score = tweetIdScoreMap.getOrElse(candidateWithFeatures.candidate.id, None)\n          FeatureMapBuilder()\n            .add(TLXScore, score)\n            .build()\n\n        }\n      case t.TweetScoringResponse.UnknownUnionField(field) =>\n        throw new UnsupportedOperationException(s\"Unknown response type: ${field.field.name}\")\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/BUILD",
    "content": "scala_library(\n    name = \"insert_append_results\",\n    sources = [\"InsertAppendResults.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n    ],\n)\n\nscala_library(\n    sources = [\n        \"!InsertAppendResults.scala\",\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    scalac_plugins = [\"no-roomba\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":insert_append_results\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"finatra/inject/inject-slf4j/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n    ],\n    exports = [\n        \":insert_append_results\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/Bucketer.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\n\n/**\n * Given a [[CandidateWithDetails]] return the corresponding [[Bucket]]\n * it should be associated with when used in a `pattern` or `ratio`\n * in [[InsertAppendPatternResults]] or [[InsertAppendRatioResults]]\n */\ntrait Bucketer[Bucket] {\n  def apply(candidateWithDetails: CandidateWithDetails): Bucket\n}\n\nobject Bucketer {\n\n  /** A [[Bucketer]] that buckets by [[CandidateWithDetails.source]] */\n  val ByCandidateSource: Bucketer[CandidatePipelineIdentifier] =\n    (candidateWithDetails: CandidateWithDetails) => candidateWithDetails.source\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/CandidateMergeStrategy.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.IsPinnedFeature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.CandidatePipelines\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateSources\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateSourcePosition\n\n/**\n * Once a pair of duplicate candidates has been found we need to someone 'resolve' the duplication.\n * This may be as simple as picking whichever candidate came first (see [[PickFirstCandidateMerger]]\n * but this strategy could mean losing important candidate information. Candidates might, for\n * example, have different features. [[CandidateMergeStrategy]] lets you define a custom behavior\n * for resolving duplication to help support these more nuanced situations.\n */\ntrait CandidateMergeStrategy {\n  def apply(\n    existingCandidate: ItemCandidateWithDetails,\n    newCandidate: ItemCandidateWithDetails\n  ): ItemCandidateWithDetails\n}\n\n/**\n * Keep whichever candidate was encountered first.\n */\nobject PickFirstCandidateMerger extends CandidateMergeStrategy {\n  override def apply(\n    existingCandidate: ItemCandidateWithDetails,\n    newCandidate: ItemCandidateWithDetails\n  ): ItemCandidateWithDetails = existingCandidate\n}\n\n/**\n * Keep the candidate encountered first but combine all candidate feature maps.\n */\nobject CombineFeatureMapsCandidateMerger extends CandidateMergeStrategy {\n  override def apply(\n    existingCandidate: ItemCandidateWithDetails,\n    newCandidate: ItemCandidateWithDetails\n  ): ItemCandidateWithDetails = {\n    // Prepend new because list set keeps insertion order, and last operations in ListSet are O(1)\n    val mergedCandidateSourceIdentifiers =\n      newCandidate.features.get(CandidateSources) ++ existingCandidate.features\n        .get(CandidateSources)\n    val mergedCandidatePipelineIdentifiers =\n      newCandidate.features.get(CandidatePipelines) ++ existingCandidate.features\n        .get(CandidatePipelines)\n\n    // the unitary features are pulled from the existing candidate as explained above, while\n    // Set Features are merged/accumulated.\n    val mergedCommonFeatureMap = FeatureMapBuilder()\n      .add(CandidatePipelines, mergedCandidatePipelineIdentifiers)\n      .add(CandidateSources, mergedCandidateSourceIdentifiers)\n      .add(CandidateSourcePosition, existingCandidate.sourcePosition)\n      .build()\n\n    existingCandidate.copy(features =\n      existingCandidate.features ++ newCandidate.features ++ mergedCommonFeatureMap)\n  }\n}\n\n/**\n * Keep the pinnable candidate. For cases where we are dealing with duplicate entries across\n * different candidate types, such as different sub-classes of\n * [[com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate]], we will\n * prioritize the candidate with [[IsPinnedFeature]] because it contains additional information\n * needed for the positioning of a pinned entry on a timeline.\n */\nobject PickPinnedCandidateMerger extends CandidateMergeStrategy {\n  override def apply(\n    existingCandidate: ItemCandidateWithDetails,\n    newCandidate: ItemCandidateWithDetails\n  ): ItemCandidateWithDetails =\n    Seq(existingCandidate, newCandidate)\n      .collectFirst {\n        case candidate @ ItemCandidateWithDetails(_: BaseTweetCandidate, _, features)\n            if features.getTry(IsPinnedFeature).toOption.contains(true) =>\n          candidate\n      }.getOrElse(existingCandidate)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/CandidatePositionInResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\nimport com.twitter.product_mixer.component_library.model.candidate.RelevancePromptCandidate\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Compute a position for inserting a specific candidate into the result sequence originally provided to the Selector.\n * If a `None` is returned, the Selector using this would not insert that candidate into the result.\n */\ntrait CandidatePositionInResults[-Query <: PipelineQuery] {\n  def apply(\n    query: Query,\n    candidate: CandidateWithDetails,\n    result: Seq[CandidateWithDetails]\n  ): Option[Int]\n}\n\nobject PromptCandidatePositionInResults extends CandidatePositionInResults[PipelineQuery] {\n  override def apply(\n    query: PipelineQuery,\n    candidate: CandidateWithDetails,\n    result: Seq[CandidateWithDetails]\n  ): Option[Int] = candidate match {\n    case ItemCandidateWithDetails(candidate, _, _) =>\n      candidate match {\n        case relevancePromptCandidate: RelevancePromptCandidate => relevancePromptCandidate.position\n        case _ => None\n      }\n    // not supporting ModuleCandidateWithDetails right now as RelevancePromptCandidate shouldn't be in a module\n    case _ => None\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DeduplicationKey.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\n\n/**\n * [[DropSelector]] detects duplicates by looking for candidates with the same key. A key can be\n * anything but is typically derived from a candidate's id and class. This approach is not always\n * appropriate. For example, two candidate sources might both return different sub-classes of\n * [[com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate]] resulting in\n * them not being treated as duplicates.\n */\ntrait DeduplicationKey[Key] {\n  def apply(candidate: ItemCandidateWithDetails): Key\n}\n\n/**\n * Use candidate id and class to determine duplicates.\n */\nobject IdAndClassDuplicationKey extends DeduplicationKey[(String, Class[_ <: UniversalNoun[Any]])] {\n  def apply(item: ItemCandidateWithDetails): (String, Class[_ <: UniversalNoun[Any]]) =\n    (item.candidate.id.toString, item.candidate.getClass)\n}\n\n/**\n * Use candidate id to determine duplicates.\n * This should be used instead of [[IdAndClassDuplicationKey]] in order to deduplicate across\n * different candidate types, such as different implementations of\n * [[com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate]].\n */\nobject IdDuplicationKey extends DeduplicationKey[String] {\n  def apply(item: ItemCandidateWithDetails): String = item.candidate.id.toString\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropAllCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope.PartitionedCandidates\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Drops all Candidates on the `remainingCandidates` side which are in the [[pipelineScope]]\n *\n * This is typically used as a placeholder when templating out a new pipeline or\n * as a simple filter to drop candidates based only on the [[CandidateScope]]\n */\ncase class DropAllCandidates(override val pipelineScope: CandidateScope = AllPipelines)\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val PartitionedCandidates(inScope, outOfScope) = pipelineScope.partition(remainingCandidates)\n\n    SelectorResult(remainingCandidates = outOfScope, result = result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropDuplicateCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.selector.DropSelector.dropDuplicates\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Keep only the first instance of a candidate in the `remainingCandidates` as determined by comparing\n * the contained candidate ID and class type. Subsequent matching instances will be dropped. For\n * more details, see DropSelector#dropDuplicates\n *\n * @param duplicationKey how to generate the key used to identify duplicate candidates (by default use id and class name)\n * @param mergeStrategy how to merge two candidates with the same key (by default pick the first one)\n *\n * @note [[com.twitter.product_mixer.component_library.model.candidate.CursorCandidate]] are ignored.\n * @note [[com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails]] are ignored.\n *\n * @example if `remainingCandidates`\n * `Seq(sourceA_Id1, sourceA_Id1, sourceA_Id2, sourceB_id1, sourceB_id2, sourceB_id3, sourceC_id4)`\n * then the output candidates will be `Seq(sourceA_Id1, sourceA_Id2, sourceB_id3, sourceC_id4)`\n */\ncase class DropDuplicateCandidates(\n  override val pipelineScope: CandidateScope = AllPipelines,\n  duplicationKey: DeduplicationKey[_] = IdAndClassDuplicationKey,\n  mergeStrategy: CandidateMergeStrategy = PickFirstCandidateMerger)\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val dedupedCandidates = dropDuplicates(\n      pipelineScope = pipelineScope,\n      candidates = remainingCandidates,\n      duplicationKey = duplicationKey,\n      mergeStrategy = mergeStrategy)\n\n    SelectorResult(remainingCandidates = dedupedCandidates, result = result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropDuplicateModuleItemCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.selector.DropSelector.dropDuplicates\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector._\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject DropDuplicateModuleItemCandidates {\n\n  /**\n   * Limit the number of module item candidates (for 1 or more modules) from a certain candidate\n   * source. See [[DropDuplicateModuleItemCandidates]] for more details.\n   *\n   * @param candidatePipeline pipelines on which to run the selector\n   *\n   * @note Scala doesn't allow overloaded methods with default arguments. Users wanting to customize\n   *       the de-dupe logic should use the default constructor. We could provide multiple\n   *       constructors but that seemed more confusing (five ways to instantiate the selector) or not\n   *       necessarily less verbose (if we picked specific use-cases rather than trying to support\n   *       everything).\n   */\n  def apply(candidatePipeline: CandidatePipelineIdentifier) = new DropDuplicateModuleItemCandidates(\n    SpecificPipeline(candidatePipeline),\n    IdAndClassDuplicationKey,\n    PickFirstCandidateMerger)\n\n  def apply(candidatePipelines: Set[CandidatePipelineIdentifier]) =\n    new DropDuplicateModuleItemCandidates(\n      SpecificPipelines(candidatePipelines),\n      IdAndClassDuplicationKey,\n      PickFirstCandidateMerger)\n}\n\n/**\n * Limit the number of module item candidates (for 1 or more modules) from certain candidate\n * pipelines.\n *\n * This acts like a [[DropDuplicateCandidates]] but for modules in `remainingCandidates`\n * from any of the provided [[candidatePipelines]]. Similar to [[DropDuplicateCandidates]], it\n * keeps only the first instance of a candidate within a module as determined by comparing\n * the contained candidate ID and class type.\n *\n * @param pipelineScope pipeline scope on which to run the selector\n * @param duplicationKey how to generate the key used to identify duplicate candidates (by default use id and class name)\n * @param mergeStrategy how to merge two candidates with the same key (by default pick the first one)\n *\n * For example, if a candidatePipeline returned 5 modules each\n * containing duplicate items in the candidate pool, then the module items in each of the\n * 5 modules will be filtered to the unique items within each module.\n *\n * Another example is if you have 2 modules each with the same items as the other,\n * it won't deduplicate across modules.\n *\n * @note this updates the module in the `remainingCandidates`\n */\ncase class DropDuplicateModuleItemCandidates(\n  override val pipelineScope: CandidateScope,\n  duplicationKey: DeduplicationKey[_] = IdAndClassDuplicationKey,\n  mergeStrategy: CandidateMergeStrategy = PickFirstCandidateMerger)\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n\n    val remainingCandidatesLimited = remainingCandidates.map {\n      case module: ModuleCandidateWithDetails if pipelineScope.contains(module) =>\n        // this applies to all candidates in a module, even if they are from a different\n        // candidate source, which can happen if items are added to a module during selection\n        module.copy(candidates = dropDuplicates(\n          pipelineScope = AllPipelines,\n          candidates = module.candidates,\n          duplicationKey = duplicationKey,\n          mergeStrategy = mergeStrategy))\n      case candidate => candidate\n    }\n\n    SelectorResult(remainingCandidates = remainingCandidatesLimited, result = result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropDuplicateResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.selector.DropSelector.dropDuplicates\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Keep only the first instance of a candidate in the `result` as determined by comparing\n * the contained candidate ID and class type. Subsequent matching instances will be dropped. For\n * more details, see DropSelector#dropDuplicates\n *\n * @param duplicationKey how to generate the key used to identify duplicate candidates (by default use id and class name)\n * @param mergeStrategy how to merge two candidates with the same key (by default pick the first one)\n *\n * @note [[com.twitter.product_mixer.component_library.model.candidate.CursorCandidate]] are ignored.\n * @note [[com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails]] are ignored.\n *\n * @example if `result`\n * `Seq(sourceA_Id1, sourceA_Id1, sourceA_Id2, sourceB_id1, sourceB_id2, sourceB_id3, sourceC_id4)`\n * then the output result will be `Seq(sourceA_Id1, sourceA_Id2, sourceB_id3, sourceC_id4)`\n */\ncase class DropDuplicateResults(\n  duplicationKey: DeduplicationKey[_] = IdAndClassDuplicationKey,\n  mergeStrategy: CandidateMergeStrategy = PickFirstCandidateMerger)\n    extends Selector[PipelineQuery] {\n\n  override val pipelineScope: CandidateScope = AllPipelines\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val dedupedResults = dropDuplicates(\n      pipelineScope = pipelineScope,\n      candidates = result,\n      duplicationKey = duplicationKey,\n      mergeStrategy = mergeStrategy)\n\n    SelectorResult(remainingCandidates = remainingCandidates, result = dedupedResults)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropFilteredCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Predicate which will be applied to each candidate. True indicates that the candidate will be\n * kept.\n */\ntrait ShouldKeepCandidate {\n  def apply(candidateWithDetails: CandidateWithDetails): Boolean\n}\n\nobject DropFilteredCandidates {\n  def apply(candidatePipeline: CandidatePipelineIdentifier, filter: ShouldKeepCandidate) =\n    new DropFilteredCandidates(SpecificPipeline(candidatePipeline), filter)\n\n  def apply(candidatePipelines: Set[CandidatePipelineIdentifier], filter: ShouldKeepCandidate) =\n    new DropFilteredCandidates(SpecificPipelines(candidatePipelines), filter)\n}\n\n/**\n * Limit candidates from certain candidates sources to those which satisfy the provided predicate.\n */\ncase class DropFilteredCandidates(\n  override val pipelineScope: CandidateScope,\n  filter: ShouldKeepCandidate)\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val candidatesUpdated = remainingCandidates.filter { candidate =>\n      if (pipelineScope.contains(candidate)) filter.apply(candidate)\n      else true\n    }\n\n    SelectorResult(remainingCandidates = candidatesUpdated, result = result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropFilteredModuleItemCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject DropFilteredModuleItemCandidates {\n  def apply(candidatePipeline: CandidatePipelineIdentifier, filter: ShouldKeepCandidate) =\n    new DropFilteredModuleItemCandidates(SpecificPipeline(candidatePipeline), filter)\n\n  def apply(candidatePipelines: Set[CandidatePipelineIdentifier], filter: ShouldKeepCandidate) =\n    new DropFilteredModuleItemCandidates(SpecificPipelines(candidatePipelines), filter)\n}\n\n/**\n * Limit candidates in modules from certain candidates sources to those which satisfy\n * the provided predicate.\n *\n * This acts like a [[DropFilteredCandidates]] but for modules in `remainingCandidates`\n * from any of the provided [[candidatePipelines]].\n *\n * @note this updates the module in the `remainingCandidates`\n */\ncase class DropFilteredModuleItemCandidates(\n  override val pipelineScope: CandidateScope,\n  filter: ShouldKeepCandidate)\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val candidatesUpdated = remainingCandidates.map {\n      case module: ModuleCandidateWithDetails if pipelineScope.contains(module) =>\n        // this applies to all candidates in a module, even if they are from a different\n        // candidate source, which can happen if items are added to a module during selection\n        module.copy(candidates = module.candidates.filter(filter.apply))\n      case candidate => candidate\n    }\n\n    SelectorResult(remainingCandidates = candidatesUpdated, result = result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropMaxCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\ntrait MaxSelector[-Query <: PipelineQuery] {\n  def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): Int\n}\n\nobject DropMaxCandidates {\n\n  /**\n   * A [[DropMaxCandidates]] Selector based on a [[Param]] applied to a single candidate pipeline\n   */\n  def apply[Query <: PipelineQuery](\n    candidatePipeline: CandidatePipelineIdentifier,\n    maxSelectionsParam: Param[Int]\n  ) = new DropMaxCandidates[Query](\n    SpecificPipeline(candidatePipeline),\n    (query, _, _) => query.params(maxSelectionsParam))\n\n  /**\n   * A [[DropMaxCandidates]] Selector based on a [[Param]] with multiple candidate pipelines\n   */\n  def apply[Query <: PipelineQuery](\n    candidatePipelines: Set[CandidatePipelineIdentifier],\n    maxSelectionsParam: Param[Int]\n  ) = new DropMaxCandidates[Query](\n    SpecificPipelines(candidatePipelines),\n    (query, _, _) => query.params(maxSelectionsParam))\n\n  /**\n   * A [[DropMaxCandidates]] Selector based on a [[Param]] that applies to a [[CandidateScope]]\n   */\n  def apply[Query <: PipelineQuery](\n    pipelineScope: CandidateScope,\n    maxSelectionsParam: Param[Int]\n  ) = new DropMaxCandidates[Query](pipelineScope, (query, _, _) => query.params(maxSelectionsParam))\n}\n\n/**\n * Limit the number of item and module (not items inside modules) candidates from the\n * specified pipelines based on the value provided by the [[MaxSelector]]\n *\n * For example, if value from the [[MaxSelector]] is 3, and a candidatePipeline returned 10 items\n * in the candidate pool, then these items will be reduced to the first 3 items. Note that to\n * update the ordering of the candidates, an UpdateCandidateOrderingSelector may be used prior to\n * using this Selector.\n *\n * Another example, if the [[MaxSelector]] value is 3, and a candidatePipeline returned 10 modules\n * in the candidate pool, then these will be reduced to the first 3 modules. The items inside the\n * modeles will not be affected by this selector. To control the number of items inside modules see\n * [[DropMaxModuleItemCandidates]].\n */\ncase class DropMaxCandidates[-Query <: PipelineQuery](\n  override val pipelineScope: CandidateScope,\n  maxSelector: MaxSelector[Query])\n    extends Selector[Query] {\n\n  override def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val maxSelections = maxSelector(query, remainingCandidates, result)\n    assert(maxSelections > 0, \"Max selections must be greater than zero\")\n\n    val remainingCandidatesLimited =\n      DropSelector.takeUntil(maxSelections, remainingCandidates, pipelineScope)\n\n    SelectorResult(remainingCandidates = remainingCandidatesLimited, result = result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropMaxModuleItemCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\n/**\n * Limit the number of module item candidates (for 1 or more modules) from a certain candidate\n * source.\n *\n * For example, if maxModuleItemsParam is 3, and a candidatePipeline returned 1 module containing 10\n * items in the candidate pool, then these module items will be reduced to the first 3 module items.\n * Note that to update the ordering of the candidates, an UpdateModuleItemsCandidateOrderingSelector\n * may be used prior to using this selector.\n *\n * Another example, if maxModuleItemsParam is 3, and a candidatePipeline returned 5 modules each\n * containing 10 items in the candidate pool, then the module items in each of the 5 modules will be\n * reduced to the first 3 module items.\n *\n * @note this updates the module in the `remainingCandidates`\n */\ncase class DropMaxModuleItemCandidates(\n  candidatePipeline: CandidatePipelineIdentifier,\n  maxModuleItemsParam: Param[Int])\n    extends Selector[PipelineQuery] {\n\n  override val pipelineScope: CandidateScope = SpecificPipelines(candidatePipeline)\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n\n    val maxModuleItemSelections = query.params(maxModuleItemsParam)\n    assert(maxModuleItemSelections > 0, \"Max module item selections must be greater than zero\")\n\n    val remainingCandidatesLimited = remainingCandidates.map {\n      case module: ModuleCandidateWithDetails if pipelineScope.contains(module) =>\n        // this applies to all candidates in a module, even if they are from a different\n        // candidate source which can happen if items are added to a module during selection\n        module.copy(candidates = DropSelector.takeUntil(maxModuleItemSelections, module.candidates))\n      case candidate => candidate\n    }\n\n    SelectorResult(remainingCandidates = remainingCandidatesLimited, result = result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropMaxResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\n/**\n * Limit the number of results\n *\n * For example, if maxResultsParam is 3, and the results contain 10 items, then these items will be\n * reduced to the first 3 selected items. Note that the ordering of results is determined by the\n * selector configuration.\n *\n * Another example, if maxResultsParam is 3, and the results contain 10 modules, then these will be\n * reduced to the first 3 modules. The items inside the modules will not be affected by this\n * selector.\n */\ncase class DropMaxResults(\n  maxResultsParam: Param[Int])\n    extends Selector[PipelineQuery] {\n\n  override val pipelineScope: CandidateScope = AllPipelines\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val maxResults = query.params(maxResultsParam)\n    assert(maxResults > 0, \"Max results must be greater than zero\")\n\n    val resultUpdated = DropSelector.takeUntil(maxResults, result)\n\n    SelectorResult(remainingCandidates = remainingCandidates, result = resultUpdated)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropModuleTooFewModuleItemResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.model.candidate.CursorCandidate\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\n/**\n * Drop the module from the `result` if it doesn't contain enough item candidates.\n *\n * For example, for a given module, if minResultsParam is 3, and the results contain 2 items,\n * then that module will be entirely dropped from the results.\n */\ncase class DropModuleTooFewModuleItemResults(\n  candidatePipeline: CandidatePipelineIdentifier,\n  minModuleItemsParam: Param[Int])\n    extends Selector[PipelineQuery] {\n\n  override val pipelineScope: CandidateScope = SpecificPipelines(candidatePipeline)\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val minModuleItemSelections = query.params(minModuleItemsParam)\n    assert(minModuleItemSelections > 0, \"Min results must be greater than zero\")\n\n    val updatedResults = result.filter {\n      case module: ModuleCandidateWithDetails\n          if pipelineScope.contains(module) && module.candidates.count { candidateWithDetails =>\n            !candidateWithDetails.candidate.isInstanceOf[CursorCandidate]\n          } < minModuleItemSelections =>\n        false\n      case _ => true\n    }\n\n    SelectorResult(remainingCandidates = remainingCandidates, result = updatedResults)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropNonDuplicateCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.model.candidate.CursorCandidate\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Keep only the candidates in `remainingCandidates` that appear multiple times.\n * This ignores modules and cursors from being removed.\n *\n * @param duplicationKey how to generate the key used to identify duplicate candidates\n *\n * @note [[com.twitter.product_mixer.component_library.model.candidate.CursorCandidate]] are ignored.\n * @note [[com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails]] are ignored.\n *\n * @example if `remainingCandidates`\n * `Seq(sourceA_Id1, sourceA_Id1, sourceA_Id2, sourceB_id1, sourceB_id2, sourceB_id3, sourceC_id4)`\n * then the output result will be `Seq(sourceA_Id1, sourceA_Id2)`\n */\ncase class DropNonDuplicateCandidates(\n  override val pipelineScope: CandidateScope,\n  duplicationKey: DeduplicationKey[_] = IdAndClassDuplicationKey)\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val duplicateCandidates = dropNonDuplicates(\n      pipelineScope = pipelineScope,\n      candidates = remainingCandidates,\n      duplicationKey = duplicationKey)\n\n    SelectorResult(remainingCandidates = duplicateCandidates, result = result)\n  }\n\n  /**\n   * Identify and keep candidates using the supplied key extraction and merger functions. By default\n   * this will keep only candidates that appear multiple times as determined by comparing\n   * the contained candidate ID and class type. Candidates appearing only once will be dropped.\n   *\n   * @note [[com.twitter.product_mixer.component_library.model.candidate.CursorCandidate]] are ignored.\n   * @note [[com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails]] are ignored.\n   *\n   * @param candidates which may have elements to drop\n   * @param duplicationKey how to generate a key for a candidate for identifying duplicates\n   */\n  private[this] def dropNonDuplicates[Candidate <: CandidateWithDetails, Key](\n    pipelineScope: CandidateScope,\n    candidates: Seq[Candidate],\n    duplicationKey: DeduplicationKey[Key],\n  ): Seq[Candidate] = {\n    // Here we are checking if each candidate has multiple appearances or not\n    val isCandidateADuplicate: Map[Key, Boolean] = candidates\n      .collect {\n        case item: ItemCandidateWithDetails\n            if pipelineScope.contains(item) && !item.candidate.isInstanceOf[CursorCandidate] =>\n          item\n      }.groupBy(duplicationKey(_))\n      .mapValues(_.length > 1)\n\n    candidates.filter {\n      case item: ItemCandidateWithDetails =>\n        isCandidateADuplicate.getOrElse(duplicationKey(item), true)\n      case _: ModuleCandidateWithDetails => true\n      case _ => false\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropOrthogonalCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Limit candidates to the first candidate source in the provided orthogonalCandidatePipelines\n * seq that has candidates in the candidate pool. For the subsequent candidate sources in the seq,\n * remove their candidates from the candidate pool.\n *\n * @example if [[orthogonalCandidatePipelines]] is `Seq(D, A, C)`, and the remaining candidates\n * component identifiers are `Seq(A, A, A, B, B, C, C, D, D, D)`, then `Seq(B, B, D, D, D)` will remain\n * in the candidate pool.\n *\n * @example if [[orthogonalCandidatePipelines]] is `Seq(D, A, C)`, and the remaining candidates\n * component identifiers are `Seq(A, A, A, B, B, C, C)`, then `Seq(A, A, A, B, B)` will remain\n * in the candidate pool.\n */\ncase class DropOrthogonalCandidates(\n  orthogonalCandidatePipelines: Seq[CandidatePipelineIdentifier])\n    extends Selector[PipelineQuery] {\n\n  override val pipelineScope: CandidateScope =\n    SpecificPipelines(orthogonalCandidatePipelines.toSet)\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val firstMatchingOrthogonalSourceOpt = orthogonalCandidatePipelines\n      .find { orthogonalCandidatePipeline =>\n        remainingCandidates.exists(_.source == orthogonalCandidatePipeline)\n      }\n\n    val remainingCandidatesLimited = firstMatchingOrthogonalSourceOpt match {\n      case Some(firstMatchingOrthogonalSource) =>\n        val subsequentOrthogonalSources =\n          orthogonalCandidatePipelines.toSet - firstMatchingOrthogonalSource\n\n        remainingCandidates.filterNot { candidate =>\n          subsequentOrthogonalSources.contains(candidate.source)\n        }\n      case None => remainingCandidates\n    }\n\n    SelectorResult(remainingCandidates = remainingCandidatesLimited, result = result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropRequestedMaxModuleItemCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\n/**\n * Limit the number of results (for 1 or more modules) from a certain candidate\n * source to PipelineQuery.requestedMaxResults.\n *\n * PipelineQuery.requestedMaxResults is optionally set in the pipelineQuery.\n * If it is not set, then the default value of DefaultRequestedMaxModuleItemsParam is used.\n *\n * For example, if PipelineQuery.requestedMaxResults is 3, and a candidatePipeline returned 1 module\n * containing 10 items in the candidate pool, then these module items will be reduced to the first 3\n * module items. Note that to update the ordering of the candidates, an\n * UpdateModuleItemsCandidateOrderingSelector may be used prior to using this selector.\n *\n * Another example, if PipelineQuery.requestedMaxResults is 3, and a candidatePipeline returned 5\n * modules each containing 10 items in the candidate pool, then the module items in each of the 5\n * modules will be reduced to the first 3 module items.\n *\n * @note this updates the module in the `remainingCandidates`\n */\ncase class DropRequestedMaxModuleItemCandidates(\n  override val pipelineScope: CandidateScope,\n  defaultRequestedMaxModuleItemResultsParam: Param[Int])\n    extends Selector[PipelineQuery] {\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n\n    val requestedMaxModuleItemSelections =\n      query.maxResults(defaultRequestedMaxModuleItemResultsParam)\n    assert(\n      requestedMaxModuleItemSelections > 0,\n      \"Requested Max module item selections must be greater than zero\")\n\n    val resultUpdated = result.map {\n      case module: ModuleCandidateWithDetails if pipelineScope.contains(module) =>\n        // this applies to all candidates in a module, even if they are from a different\n        // candidate source which can happen if items are added to a module during selection\n        module.copy(candidates =\n          DropSelector.takeUntil(requestedMaxModuleItemSelections, module.candidates))\n      case candidate => candidate\n    }\n\n    SelectorResult(remainingCandidates = remainingCandidates, result = resultUpdated)\n  }\n}\n\nobject DropRequestedMaxModuleItemCandidates {\n  def apply(\n    candidatePipeline: CandidatePipelineIdentifier,\n    defaultRequestedMaxModuleItemResultsParam: Param[Int]\n  ) =\n    new DropRequestedMaxModuleItemCandidates(\n      SpecificPipeline(candidatePipeline),\n      defaultRequestedMaxModuleItemResultsParam)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropRequestedMaxResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.BoundedParam\n\n/**\n * Limit the number of results to min(PipelineQuery.requestedMaxResults, ServerMaxResultsParam)\n *\n * PipelineQuery.requestedMaxResults is optionally set in the pipelineQuery.\n * If it is not set, then the default value of DefaultRequestedMaxResultsParam is used.\n *\n * ServerMaxResultsParam specifies the maximum number of results supported, irrespective of what is\n * specified by the client in PipelineQuery.requestedMaxResults\n * (or the DefaultRequestedMaxResultsParam default if not specified)\n *\n * For example, if ServerMaxResultsParam is 5, PipelineQuery.requestedMaxResults is 3,\n * and the results contain 10 items, then these items will be reduced to the first 3 selected items.\n *\n * If PipelineQuery.requestedMaxResults is not set, DefaultRequestedMaxResultsParam is 3,\n * ServerMaxResultsParam is 5 and the results contain 10 items,\n * then these items will be reduced to the first 3 selected items.\n *\n * Another example, if ServerMaxResultsParam is 5, PipelineQuery.requestedMaxResults is 8,\n * and the results contain 10 items, then these will be reduced to the first 5 selected items.\n *\n * The items inside the modules will not be affected by this selector.\n */\ncase class DropRequestedMaxResults(\n  defaultRequestedMaxResultsParam: BoundedParam[Int],\n  serverMaxResultsParam: BoundedParam[Int])\n    extends Selector[PipelineQuery] {\n\n  override val pipelineScope: CandidateScope = AllPipelines\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val requestedMaxResults = query.maxResults(defaultRequestedMaxResultsParam)\n    val serverMaxResults = query.params(serverMaxResultsParam)\n    assert(requestedMaxResults > 0, \"Requested max results must be greater than zero\")\n    assert(serverMaxResults > 0, \"Server max results must be greater than zero\")\n\n    val appliedMaxResults = Math.min(requestedMaxResults, serverMaxResults)\n    val resultUpdated = DropSelector.takeUntil(appliedMaxResults, result)\n\n    SelectorResult(remainingCandidates = remainingCandidates, result = resultUpdated)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropSelector.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.model.candidate.CursorCandidate\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport scala.collection.mutable\n\nprivate[selector] object DropSelector {\n\n  /**\n   * Identify and merge duplicates using the supplied key extraction and merger functions. By default\n   * this will keep only the first instance of a candidate in the `candidate` as determined by comparing\n   * the contained candidate ID and class type. Subsequent matching instances will be dropped. For\n   * more details, see DropSelector#dropDuplicates.\n   *\n   * @note [[com.twitter.product_mixer.component_library.model.candidate.CursorCandidate]] are ignored.\n   * @note [[com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails]] are ignored.\n   *\n   * @param candidates which may have elements to drop\n   * @param duplicationKey how to generate a key for a candidate for identifying duplicates\n   * @param mergeStrategy how to merge two candidates with the same key (by default pick the first one)\n   */\n  def dropDuplicates[Candidate <: CandidateWithDetails, Key](\n    pipelineScope: CandidateScope,\n    candidates: Seq[Candidate],\n    duplicationKey: DeduplicationKey[Key],\n    mergeStrategy: CandidateMergeStrategy\n  ): Seq[Candidate] = {\n    val seenCandidatePositions = mutable.HashMap[Key, Int]()\n    // We assume that, most of the time, most candidates aren't duplicates so the result Seq will be\n    // approximately the size of the candidates Seq.\n    val deduplicatedCandidates = new mutable.ArrayBuffer[Candidate](candidates.length)\n\n    for (candidate <- candidates) {\n      candidate match {\n\n        // candidate is from one of the Pipelines the selector applies to and is not a CursorCandidate\n        case item: ItemCandidateWithDetails\n            if pipelineScope.contains(item) &&\n              !item.candidate.isInstanceOf[CursorCandidate] =>\n          val key = duplicationKey(item)\n\n          // Perform a merge if the candidate has been seen already\n          if (seenCandidatePositions.contains(key)) {\n            val candidateIndex = seenCandidatePositions(key)\n\n            // Safe because only ItemCandidateWithDetails are added to seenCandidatePositions so\n            // seenCandidatePositions(key) *must* point to an ItemCandidateWithDetails\n            val originalCandidate =\n              deduplicatedCandidates(candidateIndex).asInstanceOf[ItemCandidateWithDetails]\n\n            deduplicatedCandidates.update(\n              candidateIndex,\n              mergeStrategy(originalCandidate, item).asInstanceOf[Candidate])\n          } else {\n            // Otherwise add a new entry to the list of kept candidates and update our map to track\n            // the new index\n            deduplicatedCandidates.append(item.asInstanceOf[Candidate])\n            seenCandidatePositions.update(key, deduplicatedCandidates.length - 1)\n          }\n        case item => deduplicatedCandidates.append(item)\n      }\n    }\n\n    deduplicatedCandidates\n  }\n\n  /**\n   * Takes `candidates` from all [[CandidateWithDetails.source]]s but only `candidates` in the provided\n   * `pipelineScope` are counted towards the `max` non-cursor candidates are included.\n   *\n   * @param max the maximum number of non-cursor candidates from the provided `pipelineScope` to return\n   * @param candidates a sequence of candidates which may have elements dropped\n   * @param pipelineScope the scope of which `candidates` should count towards the `max`\n   */\n  def takeUntil[Candidate <: CandidateWithDetails](\n    max: Int,\n    candidates: Seq[Candidate],\n    pipelineScope: CandidateScope = AllPipelines\n  ): Seq[Candidate] = {\n    val resultsBuilder = Seq.newBuilder[Candidate]\n    resultsBuilder.sizeHint(candidates)\n\n    candidates.foldLeft(0) {\n      case (\n            count,\n            candidate @ ItemCandidateWithDetails(_: CursorCandidate, _, _)\n          ) =>\n        // keep cursors, not included in the `count`\n        resultsBuilder += candidate.asInstanceOf[Candidate]\n        count\n\n      case (count, candidate) if !pipelineScope.contains(candidate) =>\n        // keep candidates that don't match the provided `pipelineScope`, not included in the `count`\n        resultsBuilder += candidate\n        count\n\n      case (count, candidate) if count < max =>\n        // keep candidates if theres space and increment the `count`\n        resultsBuilder += candidate\n        count + 1\n\n      case (dropCurrentCandidate, _) =>\n        // drop non-cursor candidate because theres no space left\n        dropCurrentCandidate\n    }\n    resultsBuilder.result()\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DropTooFewResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.PipelineResult\nimport com.twitter.timelines.configapi.Param\n\n/**\n * Drop all results if the minimum item threshold is not met. Some products would rather return\n * nothing than, for example, a single tweet. This lets us leverage existing client logic for\n * handling no results such as logic to not render the product at all.\n */\ncase class DropTooFewResults(minResultsParam: Param[Int]) extends Selector[PipelineQuery] {\n\n  override val pipelineScope: CandidateScope = AllPipelines\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val minResults = query.params(minResultsParam)\n    assert(minResults > 0, \"Min results must be greater than zero\")\n\n    if (PipelineResult.resultSize(result) < minResults) {\n      SelectorResult(remainingCandidates = remainingCandidates, result = Seq.empty)\n    } else {\n      SelectorResult(remainingCandidates = remainingCandidates, result = result)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/DynamicPositionSelector.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nprivate[selector] object DynamicPositionSelector {\n\n  sealed trait IndexType\n  case object RelativeIndices extends IndexType\n  case object AbsoluteIndices extends IndexType\n\n  /**\n   * Given an existing `result` seq, inserts candidates from `candidatesToInsertByIndex` into the `result` 1-by-1 with\n   * the provided index being the index relative to the `result` if given [[RelativeIndices]] or\n   * absolute index if given [[AbsoluteIndices]] (excluding duplicate insertions at an index, see below).\n   *\n   * Indices below 0 are added to the front and indices > the length are added to the end\n   *\n   * @note if multiple candidates exist with the same index, they are inserted in the order which they appear and only count\n   *       as a single element with regards to the absolute index values, see the example below\n   *\n   * @example when using [[RelativeIndices]] {{{\n   *          mergeByIndexIntoResult(\n   *          Seq(\n   *            0 -> \"a\",\n   *            0 -> \"b\",\n   *            0 -> \"c\",\n   *            1 -> \"e\",\n   *            2 -> \"g\",\n   *            2 -> \"h\"),\n   *          Seq(\n   *            \"D\",\n   *            \"F\"\n   *          ),\n   *          RelativeIndices) == Seq(\n   *            \"a\",\n   *            \"b\",\n   *            \"c\",\n   *            \"D\",\n   *            \"e\",\n   *            \"F\",\n   *            \"g\",\n   *            \"h\"\n   *          )\n   * }}}\n   *\n   * @example when using [[AbsoluteIndices]] {{{\n   *          mergeByIndexIntoResult(\n   *          Seq(\n   *            0 -> \"a\",\n   *            0 -> \"b\",\n   *            1 -> \"c\",\n   *            3 -> \"e\",\n   *            5 -> \"g\",\n   *            6 -> \"h\"),\n   *          Seq(\n   *            \"D\",\n   *            \"F\"\n   *          ),\n   *          AbsoluteIndices) == Seq(\n   *            \"a\", // index 0, \"a\" and \"b\" together only count as 1 element with regards to indexes because they have duplicate insertion points\n   *            \"b\", // index 0\n   *            \"c\", // index 1\n   *            \"D\", // index 2\n   *            \"e\", // index 3\n   *            \"F\", // index 4\n   *            \"g\", // index 5\n   *            \"h\" // index 6\n   *          )\n   * }}}\n   */\n  def mergeByIndexIntoResult[T]( // generic on `T` to simplify unit testing\n    candidatesToInsertByIndex: Seq[(Int, T)],\n    result: Seq[T],\n    indexType: IndexType\n  ): Seq[T] = {\n    val positionAndCandidateList = candidatesToInsertByIndex.sortWith {\n      case ((indexLeft: Int, _), (indexRight: Int, _)) =>\n        indexLeft < indexRight // order by desired absolute index ascending\n    }\n\n    // Merge result and positionAndCandidateList into resultUpdated while making sure that the entries\n    // from the positionAndCandidateList are inserted at the right index.\n    val resultUpdated = Seq.newBuilder[T]\n    resultUpdated.sizeHint(result.size + positionAndCandidateList.size)\n\n    var currentResultIndex = 0\n    val inputResultIterator = result.iterator\n    val positionAndCandidateIterator = positionAndCandidateList.iterator.buffered\n    var previousInsertPosition: Option[Int] = None\n\n    while (inputResultIterator.nonEmpty && positionAndCandidateIterator.nonEmpty) {\n      positionAndCandidateIterator.head match {\n        case (nextInsertionPosition, nextCandidateToInsert)\n            if previousInsertPosition.contains(nextInsertionPosition) =>\n          // inserting multiple candidates at the same index\n          resultUpdated += nextCandidateToInsert\n          // do not increment any indices, but insert the candidate and advance to the next candidate\n          positionAndCandidateIterator.next()\n\n        case (nextInsertionPosition, nextCandidateToInsert)\n            if currentResultIndex >= nextInsertionPosition =>\n          // inserting a candidate at a new index\n          // add candidate to the results\n          resultUpdated += nextCandidateToInsert\n          // save the position of the inserted element to handle duplicate index insertions\n          previousInsertPosition = Some(nextInsertionPosition)\n          // advance to next candidate\n          positionAndCandidateIterator.next()\n          if (indexType == AbsoluteIndices) {\n            // if the indices are absolute, instead of relative to the original `result` we need to\n            // count the insertions of candidates into the results towards the `currentResultIndex`\n            currentResultIndex += 1\n          }\n        case _ =>\n          // no candidate to insert by index so use the candidates from the result and increment the index\n          resultUpdated += inputResultIterator.next()\n          currentResultIndex += 1\n      }\n    }\n    // one of the iterators is empty, so append the remaining candidates in order to the end\n    resultUpdated ++= positionAndCandidateIterator.map { case (_, candidate) => candidate }\n    resultUpdated ++= inputResultIterator\n\n    resultUpdated.result()\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertAppendIntoModuleCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.selector.InsertIntoModule.ModuleAndIndex\nimport com.twitter.product_mixer.component_library.selector.InsertIntoModule.ModuleWithItemsToAddAndOtherCandidates\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Append all candidates from [[candidatePipeline]] into a module from [[targetModuleCandidatePipeline]].\n * If the results contain multiple modules from the target candidate pipeline,\n * then the candidates will be inserted into the first module.\n *\n * @note this will throw an [[UnsupportedOperationException]] if the [[candidatePipeline]] contains any modules.\n *\n * @note this updates the module in the `remainingCandidates`\n */\ncase class InsertAppendIntoModuleCandidates(\n  candidatePipeline: CandidatePipelineIdentifier,\n  targetModuleCandidatePipeline: CandidatePipelineIdentifier)\n    extends Selector[PipelineQuery] {\n\n  override val pipelineScope: CandidateScope =\n    SpecificPipelines(candidatePipeline, targetModuleCandidatePipeline)\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n\n    val ModuleWithItemsToAddAndOtherCandidates(\n      moduleToUpdateAndIndex,\n      itemsToInsertIntoModule,\n      otherCandidates) =\n      InsertIntoModule.moduleToUpdate(\n        candidatePipeline,\n        targetModuleCandidatePipeline,\n        remainingCandidates)\n\n    val updatedRemainingCandidates = moduleToUpdateAndIndex match {\n      case None => remainingCandidates\n      case _ if itemsToInsertIntoModule.isEmpty => remainingCandidates\n      case Some(ModuleAndIndex(moduleToUpdate, indexOfModuleInOtherCandidates)) =>\n        val updatedModuleItems = moduleToUpdate.candidates ++ itemsToInsertIntoModule\n        val updatedModule = moduleToUpdate.copy(candidates = updatedModuleItems)\n        otherCandidates.updated(indexOfModuleInOtherCandidates, updatedModule)\n    }\n\n    SelectorResult(remainingCandidates = updatedRemainingCandidates, result = result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertAppendPatternResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport scala.collection.mutable\n\n/**\n * Select candidates and add them according to the `pattern`.\n * The pattern is repeated until all candidates contained in the pattern are added to the `result`.\n * If the candidates for a specific [[Bucket]] in the pattern are exhausted, that [[Bucket]] will be\n * skipped on subsequent iterations.\n * If a candidate has a [[Bucket]] that isn't in the pattern it is added to the end of the `result`.\n * The end result is all candidates from all [[candidatePipelines]]s provided will end up in the result.\n *\n * @example If there are no more candidates from a given `CandidatePipeline` then it is skipped, so\n *          with the pattern `Seq(A, A, B, C)`, if there are no more candidates from `B` then it is\n *          effectively the same as `Seq(A, A, C)`. The `result` will contain all candidates from all\n *          `CandidatePipeline`s who's `Bucket` is in the `pattern`.\n *\n * @example If the pattern is `Seq(A, A, B, C)` and the remaining candidates\n *          from the provided `candidatePipelines` are:\n *          - 5 `A`s\n *          - 2 `B`s\n *          - 1 `C`\n *          - 1 `D`s\n *\n *          then the resulting output for each iteration over the pattern is\n *          - `Seq(A, A, B, C)`\n *          - `Seq(A, A, B)` since there's no more `C`s\n *          - `Seq(A)` since there are no more `B`s or `C`s\n *          - `Seq(D)` since it wasn't in the pattern but is from one of the provided\n *            `candidatePipelines`, it's appended at the end\n *\n *          so the `result` that's returned would be `Seq(A, A, B, C, A, A, B, A, D)`\n */\ncase class InsertAppendPatternResults[-Query <: PipelineQuery, Bucket](\n  candidatePipelines: Set[CandidatePipelineIdentifier],\n  bucketer: Bucketer[Bucket],\n  pattern: Seq[Bucket])\n    extends Selector[Query] {\n\n  require(pattern.nonEmpty, \"`pattern` must be non-empty\")\n\n  override val pipelineScope: CandidateScope = SpecificPipelines(candidatePipelines)\n\n  private sealed trait PatternResult\n  private case object NotASelectedCandidatePipeline extends PatternResult\n  private case object NotABucketInThePattern extends PatternResult\n  private case class Bucketed(bucket: Bucket) extends PatternResult\n\n  private val allBucketsInPattern = pattern.toSet\n\n  override def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val groupedCandidates: Map[PatternResult, Seq[CandidateWithDetails]] =\n      remainingCandidates.groupBy { candidateWithDetails =>\n        if (pipelineScope.contains(candidateWithDetails)) {\n          // if a candidate's Bucket doesnt appear in the pattern it's backfilled at the end\n          val bucket = bucketer(candidateWithDetails)\n          if (allBucketsInPattern.contains(bucket)) {\n            Bucketed(bucket)\n          } else {\n            NotABucketInThePattern\n          }\n        } else {\n          NotASelectedCandidatePipeline\n        }\n      }\n\n    val otherCandidates =\n      groupedCandidates.getOrElse(NotASelectedCandidatePipeline, Seq.empty)\n\n    val notABucketInThePattern =\n      groupedCandidates.getOrElse(NotABucketInThePattern, Seq.empty)\n\n    // mutable so we can remove finished iterators to optimize when looping for large patterns\n    val groupedBucketsIterators = mutable.HashMap(groupedCandidates.collect {\n      case (Bucketed(bucket), candidatesWithDetails) => (bucket, candidatesWithDetails.iterator)\n    }.toSeq: _*)\n\n    val patternIterator = Iterator.continually(pattern).flatten\n\n    val newResult = new mutable.ArrayBuffer[CandidateWithDetails]()\n    while (groupedBucketsIterators.nonEmpty) {\n      val bucket = patternIterator.next()\n      groupedBucketsIterators.get(bucket) match {\n        case Some(iterator) if iterator.nonEmpty => newResult += iterator.next()\n        case Some(_) => groupedBucketsIterators.remove(bucket)\n        case None =>\n      }\n    }\n\n    SelectorResult(\n      remainingCandidates = otherCandidates,\n      result = result ++ newResult ++ notABucketInThePattern)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertAppendRatioResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\nimport scala.annotation.tailrec\nimport scala.collection.mutable\nimport scala.util.Random\n\n/**\n * Select candidates and add them according to the ratio assigned for each [[Bucket]]\n * For instance, if given `Set((A, 0.8), (B, 0.2))` then candidates will randomly be added to the\n * results with an 80% chance of any candidate being from `A` and 20% from`B`. If there are no more\n * candidates from a given `CandidatePipeline` then it's simply skipped, so if we run out of `A`\n * candidates the rest will be `B`. The end result is all candidates from all [[candidatePipelines]]s\n * provided will end up in the result.\n *\n * For example, an output may look like `Seq(A, A, B, A, A)`, `Seq(A, A, A, A, B)`. If we eventually\n * run out of `A` candidates then we would end up with the remaining candidates at the end,\n * `Seq(A, A, B, A, A, A, B, A, A, A [run out of A], B, B, B, B, B, B)`\n *\n * @note the ratios provided are proportional to the sum of all ratios, so if you give 0.3 and 0.7,\n *       they will be function as to 30% and 70%, and the same for if you provided 3000 and 7000 for\n *       ratios.\n *\n * @note Its important to be sure to update all [[Param]]s when changing the ratio for 1 of them\n *       otherwise you may get unexpected results. For instance, of you have 0.3 and 0.7 which\n *       correspond to 30% and 70%, and you change `0.7 -> 0.9`, then the total sum of the ratios is\n *       now 1.2, so you have 25% and 75% when you intended to have 10% and 90%. To prevent this,\n *       be sure to update all [[Param]]s together, so `0.3 -> 0.1` and `0.7 -> 0.9` so the total\n *       remains the same.\n */\ncase class InsertAppendRatioResults[-Query <: PipelineQuery, Bucket](\n  candidatePipelines: Set[CandidatePipelineIdentifier],\n  bucketer: Bucketer[Bucket],\n  ratios: Map[Bucket, Param[Double]],\n  random: Random = new Random(0))\n    extends Selector[Query] {\n\n  require(ratios.nonEmpty, \"bucketRatios must be non-empty\")\n\n  override val pipelineScope: CandidateScope = SpecificPipelines(candidatePipelines)\n\n  private sealed trait PatternResult\n  private case object NotASelectedCandidatePipeline extends PatternResult\n  private case object NotABucketInThePattern extends PatternResult\n  private case class Bucketed(bucket: Bucket) extends PatternResult\n\n  override def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n\n    val groupedCandidates: Map[PatternResult, Seq[CandidateWithDetails]] =\n      remainingCandidates.groupBy { candidateWithDetails =>\n        if (pipelineScope.contains(candidateWithDetails)) {\n          // if a candidate's Bucket doesnt appear in the pattern it's backfilled at the end\n          val bucket = bucketer(candidateWithDetails)\n          if (ratios.contains(bucket)) {\n            Bucketed(bucket)\n          } else {\n            NotABucketInThePattern\n          }\n        } else {\n          NotASelectedCandidatePipeline\n        }\n      }\n\n    val otherCandidates =\n      groupedCandidates.getOrElse(NotASelectedCandidatePipeline, Seq.empty)\n\n    val notABucketInThePattern =\n      groupedCandidates.getOrElse(NotABucketInThePattern, Seq.empty)\n\n    val groupedCandidatesIterators = groupedCandidates.collect {\n      case (Bucketed(bucket), candidatesWithDetails) => (bucket, candidatesWithDetails.iterator)\n    }\n\n    // using a LinkedHashMap and sorting by descending ratio\n    // the highest ratios will always be checked first when iterating\n    // mutable so we can remove finished ratios when they are finished to optimize looping for large numbers of ratios\n    val currentBucketRatios: mutable.Map[Bucket, Double] = {\n      val bucketsAndRatiosSortedByRatio =\n        ratios.iterator\n          .map {\n            case (bucket, param) =>\n              val ratio = query.params(param)\n              require(\n                ratio >= 0,\n                \"The ratio for an InsertAppendRatioResults selector can not be negative\")\n              (bucket, ratio)\n          }.toSeq\n          .sortBy { case (_, ratio) => ratio }(Ordering.Double.reverse)\n      mutable.LinkedHashMap(bucketsAndRatiosSortedByRatio: _*)\n    }\n\n    // keep track of the sum of all ratios so we can look only at random values between 0 and that\n    var ratioSum = currentBucketRatios.valuesIterator.sum\n\n    // add candidates to `newResults` until all remaining candidates are for a single bucket\n    val newResult = new mutable.ArrayBuffer[CandidateWithDetails]()\n    while (currentBucketRatios.size > 1) {\n      // random number between 0 and the sum of the ratios of all params\n      val randomValue = random.nextDouble() * ratioSum\n\n      val currentBucketRatiosIterator: Iterator[(Bucket, Double)] =\n        currentBucketRatios.iterator\n      val (currentBucket, ratio) = currentBucketRatiosIterator.next()\n\n      val componentToTakeFrom = findBucketToTakeFrom(\n        randomValue = randomValue,\n        cumulativeSumOfRatios = ratio,\n        bucket = currentBucket,\n        bucketRatiosIterator = currentBucketRatiosIterator\n      )\n\n      groupedCandidatesIterators.get(componentToTakeFrom) match {\n        case Some(iteratorForBucket) if iteratorForBucket.nonEmpty =>\n          newResult += iteratorForBucket.next()\n        case _ =>\n          ratioSum -= currentBucketRatios(componentToTakeFrom)\n          currentBucketRatios.remove(componentToTakeFrom)\n      }\n    }\n    // with only have 1 source remaining, we can skip all the above work and insert them in bulk\n    val remainingBucketInRatio =\n      currentBucketRatios.keysIterator.flatMap(groupedCandidatesIterators.get).flatten\n\n    SelectorResult(\n      remainingCandidates = otherCandidates,\n      result = result ++ newResult ++ remainingBucketInRatio ++ notABucketInThePattern)\n  }\n\n  /**\n   * iterates through the `bucketRatiosIterator` until it finds a the\n   * [[Bucket]] that corresponds with the current `randomValue`.\n   *\n   * This method expects that `0 <= randomValue <= sum of all ratios`\n   *\n   * @example If the given ratios are `Seq(A -> 0.2, B -> 0.35, C -> 0.45)`\n   *          check if the given `randomValue` is\n   *          - `< 0.45`, if not then check\n   *          - `< 0.8` (0.45 + 0.35), if not then check\n   *          - `< 1.0` (0.45 + 0.35 + 0.2)\n   *\n   *          and return the corresponding [[Bucket]]\n   */\n  @tailrec private def findBucketToTakeFrom(\n    randomValue: Double,\n    cumulativeSumOfRatios: Double,\n    bucket: Bucket,\n    bucketRatiosIterator: Iterator[(Bucket, Double)]\n  ): Bucket = {\n    if (randomValue < cumulativeSumOfRatios || bucketRatiosIterator.isEmpty) {\n      bucket\n    } else {\n      val (nextBucket, ratio) = bucketRatiosIterator.next()\n      findBucketToTakeFrom(\n        randomValue,\n        cumulativeSumOfRatios + ratio,\n        nextBucket,\n        bucketRatiosIterator)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertAppendResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport CandidateScope.PartitionedCandidates\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject InsertAppendResults {\n  def apply(candidatePipeline: CandidatePipelineIdentifier): InsertAppendResults[PipelineQuery] =\n    new InsertAppendResults(SpecificPipeline(candidatePipeline))\n\n  def apply(\n    candidatePipelines: Set[CandidatePipelineIdentifier]\n  ): InsertAppendResults[PipelineQuery] = new InsertAppendResults(\n    SpecificPipelines(candidatePipelines))\n}\n\n/**\n * Select all candidates from candidate pipeline(s) and append to the end of the result.\n *\n * @note that if multiple candidate pipelines are specified, their candidates will be added\n *       to the result in the order in which they appear in the candidate pool. This ordering often\n *       reflects the order in which the candidate pipelines were listed in the mixer/recommendations\n *       pipeline, unless for example an UpdateSortCandidates selector was run prior to running\n *       this selector which could change this ordering.\n *\n * @note if inserting results from multiple candidate pipelines (see note above related to ordering),\n *       it is more performant to include all (or a subset) of the candidate pipelines in a single\n *       InsertAppendResults, as opposed to calling InsertAppendResults individually for each\n *       candidate pipeline because each selector does an O(n) pass on the candidate pool.\n */\ncase class InsertAppendResults[-Query <: PipelineQuery](\n  override val pipelineScope: CandidateScope)\n    extends Selector[Query] {\n\n  override def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val PartitionedCandidates(selectedCandidates, otherCandidates) =\n      pipelineScope.partition(remainingCandidates)\n\n    val resultUpdated = result ++ selectedCandidates\n\n    SelectorResult(remainingCandidates = otherCandidates, result = resultUpdated)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertAppendWeaveResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport scala.collection.mutable\n\nobject InsertAppendWeaveResults {\n  def apply[Query <: PipelineQuery, Bucket](\n    candidatePipelines: Set[CandidatePipelineIdentifier],\n    bucketer: Bucketer[Bucket],\n  ): InsertAppendWeaveResults[Query, Bucket] =\n    new InsertAppendWeaveResults(SpecificPipelines(candidatePipelines), bucketer)\n\n  def apply[Query <: PipelineQuery, Bucket](\n    candidatePipeline: CandidatePipelineIdentifier,\n    bucketer: Bucketer[Bucket],\n  ): InsertAppendWeaveResults[Query, Bucket] =\n    new InsertAppendWeaveResults(SpecificPipeline(candidatePipeline), bucketer)\n}\n\n/**\n * Select candidates weave them together according to their [[Bucket]].\n *\n * Candidates are grouped according to [[Bucket]] and one candidate is added from each group until\n * no candidates belonging to any group are left.\n *\n * Functionally similar to [[InsertAppendPatternResults]]. [[InsertAppendPatternResults]] is useful\n * if you have more complex ordering requirements but it requires you to know all the buckets in\n * advance.\n *\n * @note The order in which candidates are weaved together depends on the order in which the buckets\n *       were first seen on candidates.\n *\n * @example If the candidates are Seq(Tweet(10), Tweet(8), Tweet(3), Tweet(13)) and they are bucketed\n *          using an IsEven bucketing function, then the resulting buckets would be:\n *\n *          - Seq(Tweet(10), Tweet(8))\n *          - Seq(Tweet(3), Tweet(13))\n *\n *          The selector would then loop through these buckets and produce:\n *\n *          - Tweet(10)\n *          - Tweet(3)\n *          - Tweet(8)\n *          - Tweet(13)\n *\n *          Note that first bucket encountered was the 'even' bucket so weaving proceeds first with\n *          the even bucket then the odd bucket. Tweet(3) had been first then the opposite would be\n *          true.\n */\ncase class InsertAppendWeaveResults[-Query <: PipelineQuery, Bucket](\n  override val pipelineScope: CandidateScope,\n  bucketer: Bucketer[Bucket])\n    extends Selector[Query] {\n\n  override def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val (bucketableCandidates, otherCandidates) =\n      remainingCandidates.partition(pipelineScope.contains)\n\n    val groupedCandidates = groupByBucket(bucketableCandidates)\n\n    val candidateBucketQueues: mutable.Queue[mutable.Queue[CandidateWithDetails]] =\n      mutable.Queue() ++= groupedCandidates\n    val newResult = mutable.ArrayBuffer[CandidateWithDetails]()\n\n    // Take the next group of candidates from the queue and attempt to add the first candidate from\n    // that group into the result. The loop will terminate when every queue is empty.\n    while (candidateBucketQueues.nonEmpty) {\n      val nextCandidateQueue = candidateBucketQueues.dequeue()\n\n      if (nextCandidateQueue.nonEmpty) {\n        newResult += nextCandidateQueue.dequeue()\n\n        // Re-queue this bucket of candidates if it's still non-empty\n        if (nextCandidateQueue.nonEmpty) {\n          candidateBucketQueues.enqueue(nextCandidateQueue)\n        }\n      }\n    }\n\n    SelectorResult(remainingCandidates = otherCandidates, result = result ++ newResult)\n  }\n\n  /**\n   * Similar to `groupBy` but respect the order in which individual bucket values are first seen.\n   * This is useful when the candidates have already been sorted prior to the selector running.\n   */\n  private def groupByBucket(\n    candidates: Seq[CandidateWithDetails]\n  ): mutable.ArrayBuffer[mutable.Queue[CandidateWithDetails]] = {\n    val bucketToCandidateGroupIndex = mutable.Map.empty[Bucket, Int]\n    val candidateGroups = mutable.ArrayBuffer[mutable.Queue[CandidateWithDetails]]()\n\n    candidates.foreach { candidate =>\n      val bucket = bucketer(candidate)\n\n      // Index points to the specific sub-group in candidateGroups where we want to insert the next\n      // candidate. If a bucket has already been seen then this value is known, otherwise we need\n      // to add a new entry for it.\n      if (!bucketToCandidateGroupIndex.contains(bucket)) {\n        candidateGroups.append(mutable.Queue())\n        bucketToCandidateGroupIndex.put(bucket, candidateGroups.length - 1)\n      }\n\n      candidateGroups(bucketToCandidateGroupIndex(bucket)).enqueue(candidate)\n    }\n\n    candidateGroups\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertAppendWithoutFeatureResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * A selector that appends all candidates missing a specific feature to the results pool and keeps\n * the rest in the remaining candidates. This is useful for backfill scoring candidates without\n * a score from a previous scorer.\n * @param pipelineScope The pipeline scope to check\n * @param missingFeature The missing feature to check for.\n */\ncase class InsertAppendWithoutFeatureResults(\n  override val pipelineScope: CandidateScope,\n  missingFeature: Feature[_, _])\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val (candidatesWithMissingFeature, candidatesWithFeature) = remainingCandidates.partition {\n      candidate =>\n        pipelineScope.contains(candidate) && !candidate.features.getSuccessfulFeatures\n          .contains(missingFeature)\n    }\n    val updatedResults = result ++ candidatesWithMissingFeature\n    SelectorResult(remainingCandidates = candidatesWithFeature, result = updatedResults)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertDynamicPositionResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector._\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject InsertDynamicPositionResults {\n  def apply[Query <: PipelineQuery](\n    candidatePipeline: CandidatePipelineIdentifier,\n    dynamicInsertionPosition: DynamicInsertionPosition[Query],\n  ): InsertDynamicPositionResults[Query] =\n    new InsertDynamicPositionResults(SpecificPipeline(candidatePipeline), dynamicInsertionPosition)\n\n  def apply[Query <: PipelineQuery](\n    candidatePipelines: Set[CandidatePipelineIdentifier],\n    dynamicInsertionPosition: DynamicInsertionPosition[Query]\n  ): InsertDynamicPositionResults[Query] =\n    new InsertDynamicPositionResults(\n      SpecificPipelines(candidatePipelines),\n      dynamicInsertionPosition)\n}\n\n/**\n * Compute a position for inserting the candidates into result. If a `None` is returned, the\n * Selector using this would not insert the candidates into the result.\n */\ntrait DynamicInsertionPosition[-Query <: PipelineQuery] {\n  def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): Option[Int]\n}\n\n/**\n * Insert all candidates in a pipeline scope at a 0-indexed dynamic position computed\n * using the provided [[DynamicInsertionPosition]] instance. If the current results are a shorter\n * length than the computed position, then the candidates will be appended to the results.\n * If the [[DynamicInsertionPosition]] returns a `None`, the candidates are not\n * added to the result.\n */\ncase class InsertDynamicPositionResults[-Query <: PipelineQuery](\n  override val pipelineScope: CandidateScope,\n  dynamicInsertionPosition: DynamicInsertionPosition[Query])\n    extends Selector[Query] {\n\n  override def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    dynamicInsertionPosition(query, remainingCandidates, result) match {\n      case Some(position) =>\n        InsertSelector.insertIntoResultsAtPosition(\n          position = position,\n          pipelineScope = pipelineScope,\n          remainingCandidates = remainingCandidates,\n          result = result)\n      case None =>\n        // When a valid position is not provided, do not insert the candidates.\n        // Both the remainingCandidates and result are unchanged.\n        SelectorResult(remainingCandidates = remainingCandidates, result = result)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertFixedPositionIntoModuleCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.selector.InsertIntoModule.ModuleAndIndex\nimport com.twitter.product_mixer.component_library.selector.InsertIntoModule.ModuleWithItemsToAddAndOtherCandidates\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\n/**\n * Insert all candidates from [[candidatePipeline]] at a 0-indexed fixed position into a module from\n * [[targetModuleCandidatePipeline]]. If the results contain multiple modules from the target candidate\n * pipeline, then the candidates will be inserted into the first module. If the target module's\n * items are a shorter length than the requested position, then the candidates will be appended\n * to the results.\n *\n * @note this will throw an [[UnsupportedOperationException]] if the [[candidatePipeline]] contains any modules.\n *\n * @note this updates the module in the `remainingCandidates`\n */\ncase class InsertFixedPositionIntoModuleCandidates(\n  candidatePipeline: CandidatePipelineIdentifier,\n  targetModuleCandidatePipeline: CandidatePipelineIdentifier,\n  positionParam: Param[Int])\n    extends Selector[PipelineQuery] {\n\n  override val pipelineScope: CandidateScope =\n    SpecificPipelines(candidatePipeline, targetModuleCandidatePipeline)\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n\n    val position = query.params(positionParam)\n    assert(position >= 0, \"Position must be equal to or greater than zero\")\n\n    val ModuleWithItemsToAddAndOtherCandidates(\n      moduleToUpdateAndIndex,\n      itemsToInsertIntoModule,\n      otherCandidates) =\n      InsertIntoModule.moduleToUpdate(\n        candidatePipeline,\n        targetModuleCandidatePipeline,\n        remainingCandidates)\n\n    val updatedRemainingCandidates = moduleToUpdateAndIndex match {\n      case None => remainingCandidates\n      case _ if itemsToInsertIntoModule.isEmpty => remainingCandidates\n      case Some(ModuleAndIndex(moduleToUpdate, indexOfModuleInOtherCandidates)) =>\n        val updatedModuleItems =\n          if (position < moduleToUpdate.candidates.length) {\n            val (left, right) = moduleToUpdate.candidates.splitAt(position)\n            left ++ itemsToInsertIntoModule ++ right\n          } else {\n            moduleToUpdate.candidates ++ itemsToInsertIntoModule\n          }\n        val updatedModule = moduleToUpdate.copy(candidates = updatedModuleItems)\n        otherCandidates.updated(indexOfModuleInOtherCandidates, updatedModule)\n    }\n\n    SelectorResult(remainingCandidates = updatedRemainingCandidates, result = result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertFixedPositionResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\nobject InsertFixedPositionResults {\n  def apply(\n    candidatePipeline: CandidatePipelineIdentifier,\n    positionParam: Param[Int],\n  ): InsertFixedPositionResults =\n    new InsertFixedPositionResults(SpecificPipeline(candidatePipeline), positionParam)\n\n  def apply(\n    candidatePipelines: Set[CandidatePipelineIdentifier],\n    positionParam: Param[Int]\n  ): InsertFixedPositionResults =\n    new InsertFixedPositionResults(SpecificPipelines(candidatePipelines), positionParam)\n}\n\n/**\n * Insert all candidates in a pipeline scope at a 0-indexed fixed position. If the current\n * results are a shorter length than the requested position, then the candidates will be appended\n * to the results.\n */\ncase class InsertFixedPositionResults(\n  override val pipelineScope: CandidateScope,\n  positionParam: Param[Int])\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = InsertSelector.insertIntoResultsAtPosition(\n    position = query.params(positionParam),\n    pipelineScope = pipelineScope,\n    remainingCandidates = remainingCandidates,\n    result = result)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertIntoModule.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport scala.collection.immutable.Queue\n\nprivate[selector] object InsertIntoModule {\n  case class ModuleAndIndex(\n    moduleToInsertInto: ModuleCandidateWithDetails,\n    indexOfModuleInOtherCandidates: Int)\n\n  case class ModuleWithItemsToAddAndOtherCandidates(\n    moduleToUpdateAndIndex: Option[ModuleAndIndex],\n    itemsToInsertIntoModule: Queue[ItemCandidateWithDetails],\n    otherCandidates: Queue[CandidateWithDetails])\n\n  /**\n   * Given a Seq of `candidates`, returns the first module with it's index that matches the\n   * `targetModuleCandidatePipeline` with all the [[ItemCandidateWithDetails]] that match the\n   * `candidatePipeline` added to the `itemsToInsert` and the remaining candidates, including the\n   * module, in the `otherCandidates`\n   */\n  def moduleToUpdate(\n    candidatePipeline: CandidatePipelineIdentifier,\n    targetModuleCandidatePipeline: CandidatePipelineIdentifier,\n    candidates: Seq[CandidateWithDetails]\n  ): ModuleWithItemsToAddAndOtherCandidates = {\n    candidates.foldLeft[ModuleWithItemsToAddAndOtherCandidates](\n      ModuleWithItemsToAddAndOtherCandidates(None, Queue.empty, Queue.empty)) {\n      case (\n            state @ ModuleWithItemsToAddAndOtherCandidates(_, itemsToInsertIntoModule, _),\n            selectedItem: ItemCandidateWithDetails) if selectedItem.source == candidatePipeline =>\n        state.copy(itemsToInsertIntoModule = itemsToInsertIntoModule :+ selectedItem)\n\n      case (\n            state @ ModuleWithItemsToAddAndOtherCandidates(None, _, otherCandidates),\n            module: ModuleCandidateWithDetails) if module.source == targetModuleCandidatePipeline =>\n        val insertionIndex = otherCandidates.length\n        val moduleAndIndex = Some(\n          ModuleAndIndex(\n            moduleToInsertInto = module,\n            indexOfModuleInOtherCandidates = insertionIndex))\n        val otherCandidatesWithModuleAppended = otherCandidates :+ module\n        state.copy(\n          moduleToUpdateAndIndex = moduleAndIndex,\n          otherCandidates = otherCandidatesWithModuleAppended)\n\n      case (_, invalidModule: ModuleCandidateWithDetails)\n          if invalidModule.source == candidatePipeline =>\n        /**\n         * while not exactly an illegal state, its most likely an incorrectly configured candidate pipeline\n         * that returned a module instead of returning the candidates the module contains. Since you can't\n         * nest a module inside of a module, we can either throw or ignore it and we choose to ignore it\n         * to catch a potential bug a customer may do accidentally.\n         */\n        throw new UnsupportedOperationException(\n          s\"Expected the candidatePipeline $candidatePipeline to contain items to put into the module from the targetModuleCandidatePipeline $targetModuleCandidatePipeline, but not contain modules itself. \" +\n            s\"This can occur if your $candidatePipeline was incorrectly configured and returns a module when you intended to return the candidates the module contained.\"\n        )\n\n      case (\n            state @ ModuleWithItemsToAddAndOtherCandidates(_, _, otherCandidates),\n            unselectedCandidate) =>\n        state.copy(otherCandidates = otherCandidates :+ unselectedCandidate)\n    }\n  }\n\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertPerCandidateDynamicPositionResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject InsertPerCandidateDynamicPositionResults {\n  def apply[Query <: PipelineQuery](\n    candidatePipeline: CandidatePipelineIdentifier,\n    candidatePositionInResults: CandidatePositionInResults[Query]\n  ): InsertPerCandidateDynamicPositionResults[Query] =\n    InsertPerCandidateDynamicPositionResults[Query](\n      SpecificPipeline(candidatePipeline),\n      candidatePositionInResults)\n\n  def apply[Query <: PipelineQuery](\n    candidatePipelines: Set[CandidatePipelineIdentifier],\n    candidatePositionInResults: CandidatePositionInResults[Query]\n  ): InsertPerCandidateDynamicPositionResults[Query] =\n    InsertPerCandidateDynamicPositionResults[Query](\n      SpecificPipelines(candidatePipelines),\n      candidatePositionInResults)\n}\n\n/**\n * Insert each candidate in the [[CandidateScope]] at the index relative to the original candidate in the `result`\n * at that index using the provided [[CandidatePositionInResults]] instance. If the current results are shorter\n * length than the computed position, then the candidate will be appended to the results.\n *\n * When the [[CandidatePositionInResults]] returns a `None`, that candidate is not\n * added to the result. Negative position values are treated as 0 (front of the results).\n *\n * @example if [[CandidatePositionInResults]] results in a candidate mapping from index to candidate of\n *          `{0 -> a, 0 -> b, 0 -> c, 1 -> e, 2 -> g, 2 -> h} ` with  original `results` = `[D, F]`,\n *          then the resulting output would look like `[a, b, c, D, e, F, g, h]`\n */\ncase class InsertPerCandidateDynamicPositionResults[-Query <: PipelineQuery](\n  pipelineScope: CandidateScope,\n  candidatePositionInResults: CandidatePositionInResults[Query])\n    extends Selector[Query] {\n\n  override def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val (candidatesToInsert, otherRemainingCandidatesTuples) = remainingCandidates\n      .map { candidate: CandidateWithDetails =>\n        val position =\n          if (pipelineScope.contains(candidate))\n            candidatePositionInResults(query, candidate, result)\n          else\n            None\n        (position, candidate)\n      }.partition { case (index, _) => index.isDefined }\n\n    val otherRemainingCandidates = otherRemainingCandidatesTuples.map {\n      case (_, candidate) => candidate\n    }\n\n    val positionAndCandidateList = candidatesToInsert.collect {\n      case (Some(position), candidate) => (position, candidate)\n    }\n\n    val mergedResult = DynamicPositionSelector.mergeByIndexIntoResult(\n      positionAndCandidateList,\n      result,\n      DynamicPositionSelector.RelativeIndices\n    )\n\n    SelectorResult(remainingCandidates = otherRemainingCandidates, result = mergedResult)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertRandomPositionResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.selector.InsertRandomPositionResults.randomIndices\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope.PartitionedCandidates\nimport com.twitter.product_mixer.core.functional_component.configapi.StaticParam\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\nimport scala.collection.mutable\nimport scala.util.Random\n\nobject InsertRandomPositionResults {\n\n  /**\n   * Iterator containing random index between `startIndex` and `endIndex` + `n`\n   * where `n` is the number of times `next` has been called on the iterator\n   * without duplication\n   */\n  private[selector] def randomIndices(\n    resultLength: Int,\n    startIndex: Int,\n    endIndex: Int,\n    random: Random\n  ): Iterator[Int] = {\n\n    /** exclusive because [[Random.nextInt]]'s bound is exclusive */\n    val indexUpperBound = Math.min(endIndex, resultLength)\n\n    /**\n     * keep track of the available indices, `O(n)` space where `n` is `min(endIndex, resultLength) - max(startIndex, 0)`\n     * this ensures fairness which duplicate indices could otherwise skew\n     */\n    val values = mutable.ArrayBuffer(Math.max(0, startIndex) to indexUpperBound: _*)\n\n    /**\n     * Iterator that starts at 1 above the last valid index, [[indexUpperBound]] + 1, and increments monotonically\n     * representing the new highest index possible in the results for the next call\n     */\n    Iterator\n      .from(indexUpperBound + 1)\n      .map { indexUpperBound =>\n        /**\n         * pick a random index-to-insert-candidate-into-results from [[values]] replacing the value at\n         * the chosen index with the new highest index from [[indexUpperBound]], this results in\n         * constant time for picking the random index and adding the new highest valid index instead\n         * of removing the item from the middle and appending the new, which would be `O(n)` to shift\n         * all indices after the removal point\n         */\n        val i = random.nextInt(values.length)\n        val randomIndexToUse = values(i)\n        // override the value at i with the new `upperBoundExclusive` to account for the new index value in the next iteration\n        values(i) = indexUpperBound\n\n        randomIndexToUse\n      }\n  }\n}\n\nsealed trait InsertedCandidateOrder\n\n/**\n * Candidates from the `remainingCandidates` side will be inserted in a random order into the `result`\n *\n * @example if inserting `[ x, y, z ]` into the `result` then the relative positions of `x`, `y` and `z`\n *          to each other is random, e.g. `y` could come before `x` in the result.\n */\ncase object UnstableOrderingOfInsertedCandidates extends InsertedCandidateOrder\n\n/**\n * Candidates from the `remainingCandidates` side will be inserted in their original order into the `result`\n *\n * @example if inserting `[ x, y, z ]` into the `result` then the relative positions of `x`, `y` and `z`\n *          to each other will remain the same, e.g. `x` is always before `y` is always before `z` in the final result\n */\ncase object StableOrderingOfInsertedCandidates extends InsertedCandidateOrder\n\n/**\n * Insert `remainingCandidates` into a random position between the specified indices (inclusive)\n *\n * @example let `result` = `[ a, b, c, d ]` and we want to insert randomly `[ x, y, z ]`\n *          with `startIndex` =  1, `endIndex` = 2, and [[UnstableOrderingOfInsertedCandidates]].\n *          We can expect a result that looks like `[ a, ... , d ]` where `...` is\n *          a random insertion of `x`, `y`, and `z` into  `[ b, c ]`. So this could look like\n *          `[ a, y, b, x, c, z, d ]`, note that the inserted elements are randomly distributed\n *          among the elements that were originally between the specified indices.\n *          This functions like taking a slice of the original `result` between the indices,\n *          e.g. `[ b, c ]`, then randomly inserting into the slice, e.g. `[ y, b, x, c, z ]`,\n *          before reassembling the `result`, e.g. `[ a ] ++ [ y, b, x, c, z ] ++ [ d ]`.\n *\n * @example let `result` = `[ a, b, c, d ]` and we want to insert randomly `[ x, y, z ]`\n *          with `startIndex` =  1, `endIndex` = 2, and [[StableOrderingOfInsertedCandidates]].\n *          We can expect a result that looks something like `[ a, x, b, y, c, z, d ]`,\n *          where `x` is before `y` which is before `z`\n *\n * @param startIndex an inclusive index which starts the range where the candidates will be inserted\n * @param endIndex an inclusive index which ends the range where the candidates will be inserted\n */\ncase class InsertRandomPositionResults[-Query <: PipelineQuery](\n  pipelineScope: CandidateScope,\n  remainingCandidateOrder: InsertedCandidateOrder,\n  startIndex: Param[Int] = StaticParam(0),\n  endIndex: Param[Int] = StaticParam(Int.MaxValue),\n  random: Random = new Random(0))\n    extends Selector[Query] {\n\n  override def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n\n    val PartitionedCandidates(candidatesInScope, candidatesOutOfScope) =\n      pipelineScope.partition(remainingCandidates)\n\n    val randomIndexIterator = {\n      val randomIndexIterator =\n        randomIndices(result.length, query.params(startIndex), query.params(endIndex), random)\n\n      remainingCandidateOrder match {\n        case StableOrderingOfInsertedCandidates =>\n          randomIndexIterator.take(candidatesInScope.length).toSeq.sorted.iterator\n        case UnstableOrderingOfInsertedCandidates =>\n          randomIndexIterator\n      }\n    }\n\n    val mergedResult = DynamicPositionSelector.mergeByIndexIntoResult(\n      candidatesToInsertByIndex = randomIndexIterator.zip(candidatesInScope.iterator).toSeq,\n      result = result,\n      DynamicPositionSelector.AbsoluteIndices\n    )\n\n    SelectorResult(remainingCandidates = candidatesOutOfScope, result = mergedResult)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertRelativePositionResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport CandidateScope.PartitionedCandidates\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\n/**\n * Insert all candidates from a candidate pipeline at a position below, relative to the last\n * selection of the relative to candidate pipeline. If the relative to candidate pipeline does not\n * contain candidates, then the candidates will be inserted with padding relative to position zero.\n * If the current results are a shorter length than the requested padding, then the candidates will\n * be appended to the results.\n */\ncase class InsertRelativePositionResults(\n  candidatePipeline: CandidatePipelineIdentifier,\n  relativeToCandidatePipeline: CandidatePipelineIdentifier,\n  paddingAboveParam: Param[Int])\n    extends Selector[PipelineQuery] {\n\n  override val pipelineScope: CandidateScope = SpecificPipelines(candidatePipeline)\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n\n    val paddingAbove = query.params(paddingAboveParam)\n    assert(paddingAbove >= 0, \"Padding above must be equal to or greater than zero\")\n\n    val PartitionedCandidates(selectedCandidates, otherCandidates) =\n      pipelineScope.partition(remainingCandidates)\n\n    val resultUpdated = if (selectedCandidates.nonEmpty) {\n      // If the relativeToCandidatePipeline has zero candidates, lastIndexWhere will return -1 which\n      // will start padding from the zero position\n      val relativePosition = result.lastIndexWhere(_.source == relativeToCandidatePipeline) + 1\n      val position = relativePosition + paddingAbove\n\n      if (position < result.length) {\n        val (left, right) = result.splitAt(position)\n        left ++ selectedCandidates ++ right\n      } else {\n        result ++ selectedCandidates\n      }\n    } else {\n      result\n    }\n\n    SelectorResult(remainingCandidates = otherCandidates, result = resultUpdated)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/InsertSelector.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport CandidateScope.PartitionedCandidates\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\n\nprivate[selector] object InsertSelector {\n\n  /**\n   * Insert all candidates from a candidate pipeline at a 0-indexed fixed position. If the current\n   * results are a shorter length than the requested position, then the candidates will be appended\n   * to the results.\n   */\n  def insertIntoResultsAtPosition(\n    position: Int,\n    pipelineScope: CandidateScope,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    assert(position >= 0, \"Position must be equal to or greater than zero\")\n\n    val PartitionedCandidates(selectedCandidates, otherCandidates) =\n      pipelineScope.partition(remainingCandidates)\n\n    val resultUpdated = if (selectedCandidates.nonEmpty) {\n      if (position < result.length) {\n        val (left, right) = result.splitAt(position)\n        left ++ selectedCandidates ++ right\n      } else {\n        result ++ selectedCandidates\n      }\n    } else {\n      result\n    }\n\n    SelectorResult(remainingCandidates = otherCandidates, result = resultUpdated)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/SelectConditionally.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.timelines.configapi.Param\n\ntrait IncludeSelector[-Query <: PipelineQuery] {\n  def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): Boolean\n}\n\n/**\n * Run [[selector]] if [[includeSelector]] resolves to true, else no-op the selector\n */\ncase class SelectConditionally[-Query <: PipelineQuery](\n  selector: Selector[Query],\n  includeSelector: IncludeSelector[Query])\n    extends Selector[Query] {\n\n  override val pipelineScope: CandidateScope = selector.pipelineScope\n\n  override def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    if (includeSelector(query, remainingCandidates, result)) {\n      selector(query, remainingCandidates, result)\n    } else SelectorResult(remainingCandidates = remainingCandidates, result = result)\n  }\n}\n\nobject SelectConditionally {\n\n  /**\n   * Wrap each [[Selector]] in `selectors` in an [[IncludeSelector]] with `includeSelector` as the [[SelectConditionally.includeSelector]]\n   */\n  def apply[Query <: PipelineQuery](\n    selectors: Seq[Selector[Query]],\n    includeSelector: IncludeSelector[Query]\n  ): Seq[Selector[Query]] =\n    selectors.map(SelectConditionally(_, includeSelector))\n\n  /**\n   * A [[SelectConditionally]] based on a [[Param]]\n   */\n  def paramGated[Query <: PipelineQuery](\n    selector: Selector[Query],\n    enabledParam: Param[Boolean],\n  ): SelectConditionally[Query] =\n    SelectConditionally(selector, (query, _, _) => query.params(enabledParam))\n\n  /**\n   * Wrap each [[Selector]] in `selectors` in a [[SelectConditionally]] based on a [[Param]]\n   */\n  def paramGated[Query <: PipelineQuery](\n    selectors: Seq[Selector[Query]],\n    enabledParam: Param[Boolean],\n  ): Seq[Selector[Query]] =\n    selectors.map(SelectConditionally.paramGated(_, enabledParam))\n\n  /**\n   * A [[SelectConditionally]] based on an inverted [[Param]]\n   */\n  def paramNotGated[Query <: PipelineQuery](\n    selector: Selector[Query],\n    enabledParamToInvert: Param[Boolean],\n  ): SelectConditionally[Query] =\n    SelectConditionally(selector, (query, _, _) => !query.params(enabledParamToInvert))\n\n  /**\n   * Wrap each [[Selector]] in `selectors` in a [[SelectConditionally]] based on an inverted [[Param]]\n   */\n  def paramNotGated[Query <: PipelineQuery](\n    selectors: Seq[Selector[Query]],\n    enabledParamToInvert: Param[Boolean],\n  ): Seq[Selector[Query]] =\n    selectors.map(SelectConditionally.paramNotGated(_, enabledParamToInvert))\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/SelectFromSubpoolCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport scala.reflect.ClassTag\n\nsealed trait SubpoolIncludeTypes\n\ntrait IncludeInSubpool[-Query <: PipelineQuery] extends SubpoolIncludeTypes {\n\n  /**\n   * Given the `query`, current `remainingCandidate`, and the `result`,\n   * returns whether the specific `remainingCandidate` should be passed into the\n   * [[SelectFromSubpoolCandidates]]'s [[SelectFromSubpoolCandidates.selector]]\n   *\n   * @note the `result` contains the [[SelectorResult.result]] that was passed into this selector,\n   *       so each `remainingCandidate` will get the same `result` Seq.\n   */\n  def apply(\n    query: Query,\n    remainingCandidate: CandidateWithDetails,\n    result: Seq[CandidateWithDetails]\n  ): Boolean\n}\n\ncase class IncludeCandidateTypeInSubpool[CandidateType <: UniversalNoun[_]](\n)(\n  implicit tag: ClassTag[CandidateType])\n    extends IncludeInSubpool[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidate: CandidateWithDetails,\n    result: Seq[CandidateWithDetails]\n  ): Boolean = remainingCandidate.isCandidateType[CandidateType]()\n}\n\ntrait IncludeSetInSubpool[-Query <: PipelineQuery] extends SubpoolIncludeTypes {\n\n  /**\n   * Given the `query`, all `remainingCandidates`` and `results`,\n   * returns a Set of which candidates should be included in the subpool.\n   *\n   * @note the returned set is only used to determine subpool membership. Mutating the candidates\n   *       is invalid and won't work. The order of the candidates will be preserved from the current\n   *       order of the remaining candidates sequence.\n   */\n  def apply(\n    query: Query,\n    remainingCandidate: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): Set[CandidateWithDetails]\n}\n\nsealed trait SubpoolRemainingCandidatesHandler\n\n/**\n * Candidates remaining in the subpool after running the selector will be\n * prepended to the beginning of the [[SelectorResult.remainingCandidates]]\n */\ncase object PrependToBeginningOfRemainingCandidates extends SubpoolRemainingCandidatesHandler\n\n/**\n * Candidates remaining in the subpool after running the selector will be\n * appended to the end of the [[SelectorResult.remainingCandidates]]\n */\ncase object AppendToEndOfRemainingCandidates extends SubpoolRemainingCandidatesHandler\n\n/**\n * Creates a subpool of all `remainingCandidates` for which [[subpoolInclude]] resolves to true\n * (in the same order as the original `remainingCandidates`) and runs the [[selector]] with the\n * subpool passed in as the `remainingCandidates`.\n *\n * Most customers want to use a IncludeInSubpool that chooses if each candidate should be included\n * in the subpool.\n * Where necessary, IncludeSetInSubpool allows you to define them in bulk w/ a Set.\n *\n * @note any candidates in the subpool which are not added to the [[SelectorResult.result]]\n *       will be treated according to the [[SubpoolRemainingCandidatesHandler]]\n */\nclass SelectFromSubpoolCandidates[-Query <: PipelineQuery] private[selector] (\n  val selector: Selector[Query],\n  subpoolInclude: SubpoolIncludeTypes,\n  subpoolRemainingCandidatesHandler: SubpoolRemainingCandidatesHandler =\n    AppendToEndOfRemainingCandidates)\n    extends Selector[Query] {\n\n  override val pipelineScope: CandidateScope = selector.pipelineScope\n\n  override def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n\n    val (selectedCandidates, otherCandidates) = subpoolInclude match {\n      case includeInSubpool: IncludeInSubpool[Query] =>\n        remainingCandidates.partition(candidate =>\n          pipelineScope.contains(candidate) && includeInSubpool(query, candidate, result))\n      case includeSetInSubpool: IncludeSetInSubpool[Query] =>\n        val includeSet =\n          includeSetInSubpool(query, remainingCandidates.filter(pipelineScope.contains), result)\n        remainingCandidates.partition(candidate => includeSet.contains(candidate))\n    }\n\n    val underlyingSelectorResult = selector.apply(query, selectedCandidates, result)\n    val remainingCandidatesWithSubpoolRemainingCandidates =\n      subpoolRemainingCandidatesHandler match {\n        case AppendToEndOfRemainingCandidates =>\n          otherCandidates ++ underlyingSelectorResult.remainingCandidates\n        case PrependToBeginningOfRemainingCandidates =>\n          underlyingSelectorResult.remainingCandidates ++ otherCandidates\n      }\n    underlyingSelectorResult.copy(remainingCandidates =\n      remainingCandidatesWithSubpoolRemainingCandidates)\n  }\n\n  override def toString: String = s\"SelectFromSubpoolCandidates(${selector.toString}))\"\n}\n\nobject SelectFromSubpoolCandidates {\n  def apply[Query <: PipelineQuery](\n    selector: Selector[Query],\n    includeInSubpool: IncludeInSubpool[Query],\n    subpoolRemainingCandidatesHandler: SubpoolRemainingCandidatesHandler =\n      AppendToEndOfRemainingCandidates\n  ) = new SelectFromSubpoolCandidates[Query](\n    selector,\n    includeInSubpool,\n    subpoolRemainingCandidatesHandler\n  )\n\n  def includeSet[Query <: PipelineQuery](\n    selector: Selector[Query],\n    includeSetInSubpool: IncludeSetInSubpool[Query],\n    subpoolRemainingCandidatesHandler: SubpoolRemainingCandidatesHandler =\n      AppendToEndOfRemainingCandidates\n  ) = new SelectFromSubpoolCandidates[Query](\n    selector,\n    includeSetInSubpool,\n    subpoolRemainingCandidatesHandler\n  )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/UpdateSortCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.selector.sorter.SorterFromOrdering\nimport com.twitter.product_mixer.component_library.selector.sorter.SorterProvider\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope.PartitionedCandidates\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector._\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject UpdateSortCandidates {\n  def apply(\n    candidatePipeline: CandidatePipelineIdentifier,\n    sorterProvider: SorterProvider,\n  ) = new UpdateSortCandidates(SpecificPipeline(candidatePipeline), sorterProvider)\n\n  def apply(\n    candidatePipeline: CandidatePipelineIdentifier,\n    ordering: Ordering[CandidateWithDetails]\n  ) =\n    new UpdateSortCandidates(SpecificPipeline(candidatePipeline), SorterFromOrdering(ordering))\n\n  def apply(\n    candidatePipelines: Set[CandidatePipelineIdentifier],\n    ordering: Ordering[CandidateWithDetails]\n  ) =\n    new UpdateSortCandidates(SpecificPipelines(candidatePipelines), SorterFromOrdering(ordering))\n\n  def apply(\n    candidatePipelines: Set[CandidatePipelineIdentifier],\n    sorterProvider: SorterProvider,\n  ) = new UpdateSortCandidates(SpecificPipelines(candidatePipelines), sorterProvider)\n\n  def apply(\n    pipelineScope: CandidateScope,\n    ordering: Ordering[CandidateWithDetails]\n  ) = new UpdateSortCandidates(pipelineScope, SorterFromOrdering(ordering))\n}\n\n/**\n * Sort item and module (not items inside modules) candidates in a pipeline scope.\n * Note that if sorting across multiple candidate sources, the candidates will be grouped together\n * in sorted order, starting from the position of the first candidate.\n *\n * For example, we could specify the following ordering to sort by score descending:\n * Ordering\n *   .by[CandidateWithDetails, Double](_.features.get(ScoreFeature) match {\n *     case Scored(score) => score\n *     case _ => Double.MinValue\n *   }).reverse\n */\ncase class UpdateSortCandidates(\n  override val pipelineScope: CandidateScope,\n  sorterProvider: SorterProvider)\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val PartitionedCandidates(selectedCandidates, otherCandidates) =\n      pipelineScope.partition(remainingCandidates)\n\n    val updatedRemainingCandidates = if (selectedCandidates.nonEmpty) {\n      // Safe .head due to nonEmpty check\n      val position = remainingCandidates.indexOf(selectedCandidates.head)\n      val orderedSelectedCandidates =\n        sorterProvider.sorter(query, remainingCandidates, result).sort(selectedCandidates)\n\n      if (position < otherCandidates.length) {\n        val (left, right) = otherCandidates.splitAt(position)\n        left ++ orderedSelectedCandidates ++ right\n      } else {\n        otherCandidates ++ orderedSelectedCandidates\n      }\n    } else {\n      remainingCandidates\n    }\n\n    SelectorResult(remainingCandidates = updatedRemainingCandidates, result = result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/UpdateSortModuleItemCandidates.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.selector.sorter.SorterFromOrdering\nimport com.twitter.product_mixer.component_library.selector.sorter.SorterProvider\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipelines\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject UpdateSortModuleItemCandidates {\n  def apply(\n    candidatePipeline: CandidatePipelineIdentifier,\n    ordering: Ordering[CandidateWithDetails]\n  ): UpdateSortModuleItemCandidates =\n    UpdateSortModuleItemCandidates(\n      SpecificPipeline(candidatePipeline),\n      SorterFromOrdering(ordering))\n\n  def apply(\n    candidatePipeline: CandidatePipelineIdentifier,\n    sorterProvider: SorterProvider\n  ): UpdateSortModuleItemCandidates =\n    UpdateSortModuleItemCandidates(SpecificPipeline(candidatePipeline), sorterProvider)\n\n  def apply(\n    candidatePipelines: Set[CandidatePipelineIdentifier],\n    ordering: Ordering[CandidateWithDetails]\n  ): UpdateSortModuleItemCandidates =\n    UpdateSortModuleItemCandidates(\n      SpecificPipelines(candidatePipelines),\n      SorterFromOrdering(ordering))\n\n  def apply(\n    candidatePipelines: Set[CandidatePipelineIdentifier],\n    sorterProvider: SorterProvider\n  ): UpdateSortModuleItemCandidates =\n    UpdateSortModuleItemCandidates(SpecificPipelines(candidatePipelines), sorterProvider)\n}\n\n/**\n * Sort items inside a module from a candidate source and update the remainingCandidates.\n *\n * For example, we could specify the following ordering to sort by score descending:\n *\n * {{{\n * Ordering\n *   .by[CandidateWithDetails, Double](_.features.get(ScoreFeature) match {\n *     case Scored(score) => score\n *     case _ => Double.MinValue\n *   }).reverse\n *\n * // Before sorting:\n * ModuleCandidateWithDetails(\n *  Seq(\n *    ItemCandidateWithLowScore,\n *    ItemCandidateWithMidScore,\n *    ItemCandidateWithHighScore),\n *  ... other params\n * )\n *\n * // After sorting:\n * ModuleCandidateWithDetails(\n *  Seq(\n *    ItemCandidateWithHighScore,\n *    ItemCandidateWithMidScore,\n *    ItemCandidateWithLowScore),\n *  ... other params\n * )\n * }}}\n *\n * @note this updates the modules in the `remainingCandidates`\n */\ncase class UpdateSortModuleItemCandidates(\n  override val pipelineScope: CandidateScope,\n  sorterProvider: SorterProvider)\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val updatedCandidates = remainingCandidates.map {\n      case module: ModuleCandidateWithDetails if pipelineScope.contains(module) =>\n        module.copy(candidates =\n          sorterProvider.sorter(query, remainingCandidates, result).sort(module.candidates))\n      case candidate => candidate\n    }\n    SelectorResult(remainingCandidates = updatedCandidates, result = result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/UpdateSortResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector\n\nimport com.twitter.product_mixer.component_library.selector.sorter.SorterFromOrdering\nimport com.twitter.product_mixer.component_library.selector.sorter.SorterProvider\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject UpdateSortResults {\n  def apply(\n    ordering: Ordering[CandidateWithDetails]\n  ) =\n    new UpdateSortResults((_, _, _) => SorterFromOrdering(ordering))\n}\n\n/**\n * Sort item and module (not items inside modules) results.\n *\n * For example, we could specify the following ordering to sort by score descending:\n * Ordering\n *   .by[CandidateWithDetails, Double](_.features.get(ScoreFeature) match {\n *     case Scored(score) => score\n *     case _ => Double.MinValue\n *   }).reverse\n */\ncase class UpdateSortResults(\n  sorterProvider: SorterProvider,\n  override val pipelineScope: CandidateScope = AllPipelines)\n    extends Selector[PipelineQuery] {\n\n  override def apply(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    val updatedResult = sorterProvider.sorter(query, remainingCandidates, result).sort(result)\n\n    SelectorResult(remainingCandidates = remainingCandidates, result = updatedResult)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/ads/AdsInjector.scala",
    "content": "package com.twitter.product_mixer.component_library.selector.ads\n\nimport com.google.inject.Inject\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.goldfinch.adaptors.ads.productmixer.ProductMixerPromotedEntriesAdaptor\nimport com.twitter.goldfinch.adaptors.productmixer.ProductMixerNonPromotedEntriesAdaptor\nimport com.twitter.goldfinch.adaptors.productmixer.ProductMixerQueryConverter\nimport com.twitter.goldfinch.api.AdsInjectionRequestContextConverter\nimport com.twitter.goldfinch.api.AdsInjectionSurfaceAreas.SurfaceAreaName\nimport com.twitter.goldfinch.api.{AdsInjector => GoldfinchAdsInjector}\nimport com.twitter.goldfinch.api.NonPromotedEntriesAdaptor\nimport com.twitter.goldfinch.api.PromotedEntriesAdaptor\nimport com.twitter.goldfinch.impl.injector.AdsInjectorBuilder\nimport com.twitter.goldfinch.impl.injector.product_mixer.AdsInjectionSurfaceAreaAdjustersMap\nimport com.twitter.goldfinch.impl.injector.product_mixer.VerticalSizeAdjustmentConfigMap\nimport com.twitter.inject.Logging\nimport com.twitter.product_mixer.component_library.model.query.ads._\nimport com.twitter.product_mixer.core.model.common.presentation._\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport javax.inject.Singleton\nimport com.twitter.goldfinch.impl.core.DefaultFeatureSwitchResultsFactory\nimport com.twitter.goldfinch.impl.core.LocalDevelopmentFeatureSwitchResultsFactory\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ConfigRepoLocalPath\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ServiceLocal\n\n@Singleton\nclass AdsInjector @Inject() (\n  statsReceiver: StatsReceiver,\n  @Flag(ConfigRepoLocalPath) localConfigRepoPath: String,\n  @Flag(ServiceLocal) isServiceLocal: Boolean)\n    extends Logging {\n  private val adsQueryRequestConverter: AdsInjectionRequestContextConverter[\n    PipelineQuery with AdsQuery\n  ] = ProductMixerQueryConverter\n\n  def forSurfaceArea(\n    surfaceAreaName: SurfaceAreaName\n  ): GoldfinchAdsInjector[\n    PipelineQuery with AdsQuery,\n    CandidateWithDetails,\n    CandidateWithDetails\n  ] = {\n\n    val scopedStatsReceiver: StatsReceiver =\n      statsReceiver.scope(\"goldfinch\", surfaceAreaName.toString)\n\n    val nonAdsAdaptor: NonPromotedEntriesAdaptor[CandidateWithDetails] =\n      ProductMixerNonPromotedEntriesAdaptor(\n        VerticalSizeAdjustmentConfigMap.configsBySurfaceArea(surfaceAreaName),\n        scopedStatsReceiver)\n\n    val adsAdaptor: PromotedEntriesAdaptor[CandidateWithDetails] =\n      new ProductMixerPromotedEntriesAdaptor(scopedStatsReceiver)\n\n    val featureSwitchFactory = if (isServiceLocal) {\n      new LocalDevelopmentFeatureSwitchResultsFactory(\n        surfaceAreaName.toString,\n        configRepoAbsPath = localConfigRepoPath)\n    } else new DefaultFeatureSwitchResultsFactory(surfaceAreaName.toString)\n\n    new AdsInjectorBuilder[PipelineQuery with AdsQuery, CandidateWithDetails, CandidateWithDetails](\n      requestAdapter = adsQueryRequestConverter,\n      nonPromotedEntriesAdaptor = nonAdsAdaptor,\n      promotedEntriesAdaptor = adsAdaptor,\n      adjusters =\n        AdsInjectionSurfaceAreaAdjustersMap.getAdjusters(surfaceAreaName, scopedStatsReceiver),\n      featureSwitchFactory = featureSwitchFactory,\n      statsReceiver = scopedStatsReceiver,\n      logger = logger\n    ).build()\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/ads/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    scalac_plugins = [\"no-roomba\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/adaptors/ads/productmixer\",\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/adaptors/ads/productmixer/util\",\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/adaptors/productmixer\",\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/api\",\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/impl/core\",\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/impl/injector\",\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/impl/injector/product_mixer\",\n        \"finatra/inject/inject-slf4j/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate/ads\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/query/ads\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n    ],\n    exports = [\n        \"ads-injection/lib/src/main/scala/com/twitter/goldfinch/impl/injector\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/ads/InsertAdResults.scala",
    "content": "package com.twitter.product_mixer.component_library.selector.ads\n\nimport com.twitter.goldfinch.api.AdsInjectionSurfaceAreas.SurfaceAreaName\nimport com.twitter.goldfinch.api.AdsInjectorAdditionalRequestParams\nimport com.twitter.goldfinch.api.AdsInjectorOutput\nimport com.twitter.goldfinch.api.{AdsInjector => GoldfinchAdsInjector}\nimport com.twitter.product_mixer.component_library.model.query.ads._\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport CandidateScope.PartitionedCandidates\nimport com.twitter.product_mixer.core.functional_component.common.SpecificPipeline\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Injects the sequence of AdCandidates in the `result` in the\n * sequence of the Other Candidates(which are not ads).\n *\n * Every SurfaceArea or DisplayLocation runs their own desired set of adjusters(set in pipeline)\n * to inject ads and reposition the ads in the sequence of other candidates of `result` :\n * which are fetched by AdsInjectionSurfaceAreaAdjustersMap\n * Note: The original sequence of non_promoted entries(non-ads) is retained and the ads\n * are inserted in the sequence using `goldfinch` library based on the 'insertion-position'\n * hydrated in AdsCandidate by Adserver/Admixer.\n *\n * ***** Goldfinch recommends to run this selector as close to the marshalling of candidates to have\n * more realistic view of served-timeline in Goldfinch-BQ-Logs and avoid any further updates on the\n * timeline(sequence of entries) created. ****\n *\n * Any surface area like `search_tweets(surface_area)` can call\n * InsertAdResults(surfaceArea = \"TweetSearch\", candidatePipeline = adsCandidatePipeline.identifier,\n * ProductMixerAdsInjector = productMixerAdsInjector)\n * where the pipeline config can call\n * productMixerAdsInjector.forSurfaceArea(\"TweetSearch\") to get AdsInjector Object\n *\n * @example\n * `Seq(source1NonAd_Id1, source1NonAd_Id2, source2NonAd_Id1, source2NonAd_Id2,source1NonAd_Id3, source3NonAd_Id3,source3Ad_Id1_InsertionPos1, source3Ad_Id2_InsertionPos4)`\n * then the output result can be\n * `Seq(source1NonAd_Id1, source3Ad_Id1_InsertionPos1, source1NonAd_Id2, source2NonAd_Id1, source3Ad_Id2_InsertionPos4,source2NonAd_Id2, source1NonAd_Id3, source3NonAd_Id3)`\n * depending on the insertion position of Ads and other adjusters shifting the ads\n */\ncase class InsertAdResults(\n  surfaceAreaName: SurfaceAreaName,\n  adsInjector: GoldfinchAdsInjector[\n    PipelineQuery with AdsQuery,\n    CandidateWithDetails,\n    CandidateWithDetails\n  ],\n  adsCandidatePipeline: CandidatePipelineIdentifier)\n    extends Selector[PipelineQuery with AdsQuery] {\n\n  override val pipelineScope: CandidateScope = SpecificPipeline(adsCandidatePipeline)\n\n  override def apply(\n    query: PipelineQuery with AdsQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult = {\n    // Read into ads and non-ads candidates.\n    val PartitionedCandidates(adCandidates, otherRemainingCandidates) =\n      pipelineScope.partition(remainingCandidates)\n\n    // Create this param from Query/AdsCandidate based on surface_area, if required.\n    val adsInjectorAdditionalRequestParams =\n      AdsInjectorAdditionalRequestParams(budgetAwareExperimentId = None)\n\n    val adsInjectorOutput: AdsInjectorOutput[CandidateWithDetails, CandidateWithDetails] =\n      adsInjector.applyForAllEntries(\n        query = query,\n        nonPromotedEntries = result,\n        promotedEntries = adCandidates,\n        adsInjectorAdditionalRequestParams = adsInjectorAdditionalRequestParams)\n\n    val updatedRemainingCandidates = otherRemainingCandidates ++\n      GoldfinchResults(adsInjectorOutput.unusedEntries).adapt\n    val mergedResults = GoldfinchResults(adsInjectorOutput.mergedEntries).adapt\n    SelectorResult(remainingCandidates = updatedRemainingCandidates, result = mergedResults)\n  }\n\n  /**\n   * Goldfinch separates NonPromotedEntryType and PromotedEntryType models, while in ProMix both\n   * non-promoted and promoted entries are CandidateWithDetails. As such, we need to flatten the\n   * result back into a single Seq of CandidateWithDetails. See [[AdsInjectorOutput]]\n   */\n  case class GoldfinchResults(results: Seq[Either[CandidateWithDetails, CandidateWithDetails]]) {\n    def adapt: Seq[CandidateWithDetails] = {\n      results.collect {\n        case Right(value) => value\n        case Left(value) => value\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    scalac_plugins = [\"no-roomba\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter/FeatureValueSorter.scala",
    "content": "package com.twitter.product_mixer.component_library.selector.sorter\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport scala.reflect.runtime.universe._\n\nobject FeatureValueSorter {\n\n  /**\n   * Sort by a feature value ascending. If the feature failed or is missing, use an inferred default\n   * based on the type of [[FeatureValue]]. For Numeric values this is the MinValue\n   * (e.g. Long.MinValue, Double.MinValue).\n   *\n   * @param feature feature with value to sort by\n   * @param dummyImplicit due to type erasure, implicit used to disambiguate `def ascending()`\n   *                      between def with param `feature: Feature[Candidate, FeatureValue]`\n   *                      from def with param `feature: Feature[Candidate, Option[FeatureValue]]`\n   * @param typeTag allows for inferring default value from the FeatureValue type.\n   *                See [[featureValueSortDefaultValue]]\n   * @tparam Candidate candidate for the feature\n   * @tparam FeatureValue feature value with an [[Ordering]] context bound\n   */\n  def ascending[Candidate <: UniversalNoun[Any], FeatureValue: Ordering](\n    feature: Feature[Candidate, FeatureValue]\n  )(\n    implicit dummyImplicit: DummyImplicit,\n    typeTag: TypeTag[FeatureValue]\n  ): SorterProvider = {\n    val defaultFeatureValue: FeatureValue = featureValueSortDefaultValue(feature, Ascending)\n\n    ascending(feature, defaultFeatureValue)\n  }\n\n  /**\n   * Sort by a feature value ascending. If the feature failed or is missing, use the provided\n   * default.\n   *\n   * @param feature feature with value to sort by\n   * @param dummyImplicit due to type erasure, implicit used to disambiguate `def ascending()`\n   *                      between def with param `feature: Feature[Candidate, FeatureValue]`\n   *                      from def with param `feature: Feature[Candidate, Option[FeatureValue]]`\n   * @tparam Candidate candidate for the feature\n   * @tparam FeatureValue feature value with an [[Ordering]] context bound\n   */\n  def ascending[Candidate <: UniversalNoun[Any], FeatureValue: Ordering](\n    feature: Feature[Candidate, FeatureValue],\n    defaultFeatureValue: FeatureValue\n  )(\n    implicit dummyImplicit: DummyImplicit\n  ): SorterProvider = {\n    val ordering = Ordering.by[CandidateWithDetails, FeatureValue](\n      _.features.getOrElse(feature, defaultFeatureValue))\n\n    SorterFromOrdering(ordering, Ascending)\n  }\n\n  /**\n   * Sort by an optional feature value ascending. If the feature failed or is missing, use an\n   * inferred default based on the type of [[FeatureValue]]. For Numeric values this is the MinValue\n   * (e.g. Long.MinValue, Double.MinValue).\n   *\n   * @param feature feature with value to sort by\n   * @param typeTag allows for inferring default value from the FeatureValue type.\n   *                See [[featureOptionalValueSortDefaultValue]]\n   * @tparam Candidate candidate for the feature\n   * @tparam FeatureValue feature value with an [[Ordering]] context bound\n   */\n  def ascending[Candidate <: UniversalNoun[Any], FeatureValue: Ordering](\n    feature: Feature[Candidate, Option[FeatureValue]]\n  )(\n    implicit typeTag: TypeTag[FeatureValue]\n  ): SorterProvider = {\n    val defaultFeatureValue: FeatureValue = featureOptionalValueSortDefaultValue(feature, Ascending)\n\n    ascending(feature, defaultFeatureValue)\n  }\n\n  /**\n   * Sort by an optional feature value ascending. If the feature failed or is missing, use the\n   * provided default.\n   *\n   * @param feature feature with value to sort by\n   * @tparam Candidate candidate for the feature\n   * @tparam FeatureValue feature value with an [[Ordering]] context bound\n   */\n  def ascending[Candidate <: UniversalNoun[Any], FeatureValue: Ordering](\n    feature: Feature[Candidate, Option[FeatureValue]],\n    defaultFeatureValue: FeatureValue\n  ): SorterProvider = {\n    val ordering = Ordering.by[CandidateWithDetails, FeatureValue](\n      _.features.getOrElse(feature, None).getOrElse(defaultFeatureValue))\n\n    SorterFromOrdering(ordering, Ascending)\n  }\n\n  /**\n   * Sort by a feature value descending. If the feature failed or is missing, use an inferred\n   * default based on the type of [[FeatureValue]]. For Numeric values this is the MaxValue\n   * (e.g. Long.MaxValue, Double.MaxValue).\n   *\n   * @param feature feature with value to sort by\n   * @param dummyImplicit due to type erasure, implicit used to disambiguate `def descending()`\n   *                      between def with param `feature: Feature[Candidate, FeatureValue]`\n   *                      from def with param `feature: Feature[Candidate, Option[FeatureValue]]`\n   * @param typeTag allows for inferring default value from the FeatureValue type.\n   *                See [[featureValueSortDefaultValue]]\n   * @tparam Candidate candidate for the feature\n   * @tparam FeatureValue feature value with an [[Ordering]] context bound\n   */\n  def descending[Candidate <: UniversalNoun[Any], FeatureValue: Ordering](\n    feature: Feature[Candidate, FeatureValue]\n  )(\n    implicit dummyImplicit: DummyImplicit,\n    typeTag: TypeTag[FeatureValue]\n  ): SorterProvider = {\n    val defaultFeatureValue: FeatureValue = featureValueSortDefaultValue(feature, Descending)\n\n    descending(feature, defaultFeatureValue)\n  }\n\n  /**\n   * Sort by a feature value descending. If the feature failed or is missing, use the provided\n   * default.\n   *\n   * @param feature feature with value to sort by\n   * @param dummyImplicit due to type erasure, implicit used to disambiguate `def descending()`\n   *                      between def with param `feature: Feature[Candidate, FeatureValue]`\n   *                      from def with param `feature: Feature[Candidate, Option[FeatureValue]]`\n   * @tparam Candidate candidate for the feature\n   * @tparam FeatureValue feature value with an [[Ordering]] context bound\n   */\n  def descending[Candidate <: UniversalNoun[Any], FeatureValue: Ordering](\n    feature: Feature[Candidate, FeatureValue],\n    defaultFeatureValue: FeatureValue\n  )(\n    implicit dummyImplicit: DummyImplicit\n  ): SorterProvider = {\n    val ordering = Ordering.by[CandidateWithDetails, FeatureValue](\n      _.features.getOrElse(feature, defaultFeatureValue))\n\n    SorterFromOrdering(ordering, Descending)\n  }\n\n  /**\n   * Sort by an optional feature value descending. If the feature failed or is missing, use an\n   * inferred default based on the type of [[FeatureValue]]. For Numeric values this is the MaxValue\n   * (e.g. Long.MaxValue, Double.MaxValue).\n   *\n   * @param feature feature with value to sort by\n   * @param typeTag allows for inferring default value from the FeatureValue type.\n   *                See [[featureOptionalValueSortDefaultValue]]\n   * @tparam Candidate candidate for the feature\n   * @tparam FeatureValue feature value with an [[Ordering]] context bound\n   */\n  def descending[Candidate <: UniversalNoun[Any], FeatureValue: Ordering](\n    feature: Feature[Candidate, Option[FeatureValue]]\n  )(\n    implicit typeTag: TypeTag[FeatureValue]\n  ): SorterProvider = {\n    val defaultFeatureValue: FeatureValue =\n      featureOptionalValueSortDefaultValue(feature, Descending)\n\n    descending(feature, defaultFeatureValue)\n  }\n\n  /**\n   * Sort by an optional feature value descending. If the feature failed or is missing, use the\n   * provided default.\n   *\n   * @param feature feature with value to sort by\n   * @tparam Candidate candidate for the feature\n   * @tparam FeatureValue feature value with an [[Ordering]] context bound\n   */\n  def descending[Candidate <: UniversalNoun[Any], FeatureValue: Ordering](\n    feature: Feature[Candidate, Option[FeatureValue]],\n    defaultFeatureValue: FeatureValue\n  ): SorterProvider = {\n    val ordering = Ordering.by[CandidateWithDetails, FeatureValue](\n      _.features.getOrElse(feature, None).getOrElse(defaultFeatureValue))\n\n    SorterFromOrdering(ordering, Descending)\n  }\n\n  private[sorter] def featureValueSortDefaultValue[FeatureValue: Ordering](\n    feature: Feature[_, FeatureValue],\n    sortOrder: SortOrder\n  )(\n    implicit typeTag: TypeTag[FeatureValue]\n  ): FeatureValue = {\n    val defaultValue = sortOrder match {\n      case Descending =>\n        typeOf[FeatureValue] match {\n          case t if t <:< typeOf[Short] => Short.MinValue\n          case t if t <:< typeOf[Int] => Int.MinValue\n          case t if t <:< typeOf[Long] => Long.MinValue\n          case t if t <:< typeOf[Double] => Double.MinValue\n          case t if t <:< typeOf[Float] => Float.MinValue\n          case _ =>\n            throw new UnsupportedOperationException(s\"Default value not supported for $feature\")\n        }\n      case Ascending =>\n        typeOf[FeatureValue] match {\n          case t if t <:< typeOf[Short] => Short.MaxValue\n          case t if t <:< typeOf[Int] => Int.MaxValue\n          case t if t <:< typeOf[Long] => Long.MaxValue\n          case t if t <:< typeOf[Double] => Double.MaxValue\n          case t if t <:< typeOf[Float] => Float.MaxValue\n          case _ =>\n            throw new UnsupportedOperationException(s\"Default value not supported for $feature\")\n        }\n    }\n\n    defaultValue.asInstanceOf[FeatureValue]\n  }\n\n  private[sorter] def featureOptionalValueSortDefaultValue[FeatureValue: Ordering](\n    feature: Feature[_, Option[FeatureValue]],\n    sortOrder: SortOrder\n  )(\n    implicit typeTag: TypeTag[FeatureValue]\n  ): FeatureValue = {\n    val defaultValue = sortOrder match {\n      case Descending =>\n        typeOf[Option[FeatureValue]] match {\n          case t if t <:< typeOf[Option[Short]] => Short.MinValue\n          case t if t <:< typeOf[Option[Int]] => Int.MinValue\n          case t if t <:< typeOf[Option[Long]] => Long.MinValue\n          case t if t <:< typeOf[Option[Double]] => Double.MinValue\n          case t if t <:< typeOf[Option[Float]] => Float.MinValue\n          case _ =>\n            throw new UnsupportedOperationException(s\"Default value not supported for $feature\")\n        }\n      case Ascending =>\n        typeOf[Option[FeatureValue]] match {\n          case t if t <:< typeOf[Option[Short]] => Short.MaxValue\n          case t if t <:< typeOf[Option[Int]] => Int.MaxValue\n          case t if t <:< typeOf[Option[Long]] => Long.MaxValue\n          case t if t <:< typeOf[Option[Double]] => Double.MaxValue\n          case t if t <:< typeOf[Option[Float]] => Float.MaxValue\n          case _ =>\n            throw new UnsupportedOperationException(s\"Default value not supported for $feature\")\n        }\n    }\n\n    defaultValue.asInstanceOf[FeatureValue]\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter/RandomShuffleSorter.scala",
    "content": "package com.twitter.product_mixer.component_library.selector.sorter\n\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport scala.util.Random\n\n/**\n * Randomly shuffles candidates using the provided [[random]]\n *\n * @example `UpdateSortResults(RandomShuffleSorter())`\n * @param random used to set the seed and for ease of testing, in most cases leaving it as the default is fine.\n */\ncase class RandomShuffleSorter(random: Random = new Random(0)) extends SorterProvider with Sorter {\n\n  override def sort[Candidate <: CandidateWithDetails](candidates: Seq[Candidate]): Seq[Candidate] =\n    random.shuffle(candidates)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter/ReverseSorter.scala",
    "content": "package com.twitter.product_mixer.component_library.selector.sorter\n\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\n\n/**\n * Reverse candidates.\n *\n * @example `UpdateSortResults(ReverseSorter())`\n */\nobject ReverseSorter extends SorterProvider with Sorter {\n\n  override def sort[Candidate <: CandidateWithDetails](candidates: Seq[Candidate]): Seq[Candidate] =\n    candidates.reverse\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter/SortOrder.scala",
    "content": "package com.twitter.product_mixer.component_library.selector.sorter\n\nsealed trait SortOrder\ncase object Ascending extends SortOrder\ncase object Descending extends SortOrder\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter/SorterFromOrdering.scala",
    "content": "package com.twitter.product_mixer.component_library.selector.sorter\n\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\n\nobject SorterFromOrdering {\n  def apply(ordering: Ordering[CandidateWithDetails], sortOrder: SortOrder): SorterFromOrdering =\n    SorterFromOrdering(if (sortOrder == Descending) ordering.reverse else ordering)\n}\n\n/**\n * Sorts candidates based on the provided [[ordering]]\n *\n * @note the [[Ordering]] must be transitive, so if `A < B` and `B < C` then `A < C`.\n * @note sorting randomly via `Ordering.by[CandidateWithDetails, Double](_ => Random.nextDouble())`\n *       is not safe and can fail at runtime since TimSort depends on stable sort values for\n *       pivoting. To sort randomly, use [[RandomShuffleSorter]] instead.\n */\ncase class SorterFromOrdering(\n  ordering: Ordering[CandidateWithDetails])\n    extends SorterProvider\n    with Sorter {\n\n  override def sort[Candidate <: CandidateWithDetails](candidates: Seq[Candidate]): Seq[Candidate] =\n    candidates.sorted(ordering)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter/SorterProvider.scala",
    "content": "package com.twitter.product_mixer.component_library.selector.sorter\n\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Makes a [[Sorter]] to run for the given input based on the\n * [[PipelineQuery]], the `remainingCandidates`, and the `result`.\n *\n * @note this should be used to choose between different [[Sorter]]s,\n *       if you want to conditionally sort wrap your [[Sorter]] with\n *       [[com.twitter.product_mixer.component_library.selector.SelectConditionally]] instead.\n */\ntrait SorterProvider {\n\n  /** Makes a [[Sorter]] for the given inputs */\n  def sorter(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): Sorter\n}\n\n/**\n * Sorts the candidates\n *\n * All [[Sorter]]s also implement [[SorterProvider]] to provide themselves for convenience.\n */\ntrait Sorter { self: SorterProvider =>\n\n  /** Sorts the `candidates` */\n  def sort[Candidate <: CandidateWithDetails](candidates: Seq[Candidate]): Seq[Candidate]\n\n  /** Any [[Sorter]] can be used in place of a [[SorterProvider]] to provide itself */\n  override final def sorter(\n    query: PipelineQuery,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): Sorter = self\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter/featurestorev1/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    scalac_plugins = [\"no-roomba\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector/sorter/featurestorev1/FeatureStoreV1FeatureValueSorter.scala",
    "content": "package com.twitter.product_mixer.component_library.selector.sorter.featurestorev1\n\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.product_mixer.component_library.selector.sorter.Ascending\nimport com.twitter.product_mixer.component_library.selector.sorter.Descending\nimport com.twitter.product_mixer.component_library.selector.sorter.FeatureValueSorter.featureValueSortDefaultValue\nimport com.twitter.product_mixer.component_library.selector.sorter.SorterFromOrdering\nimport com.twitter.product_mixer.component_library.selector.sorter.SorterProvider\nimport com.twitter.product_mixer.core.feature.featuremap.featurestorev1.FeatureStoreV1FeatureMap._\nimport com.twitter.product_mixer.core.feature.featurestorev1.FeatureStoreV1CandidateFeature\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport scala.reflect.runtime.universe._\n\n/**\n * Feature Store v1 version of [[com.twitter.product_mixer.component_library.selector.sorter.FeatureValueSorter]]\n */\nobject FeatureStoreV1FeatureValueSorter {\n\n  /**\n   * Sort by a Feature Store v1 feature value ascending. If the feature failed or is missing, use an\n   * inferred default based on the type of [[FeatureValue]]. For Numeric values this is the MinValue\n   * (e.g. Long.MinValue, Double.MinValue).\n   *\n   * @param feature Feature Store v1 feature with value to sort by\n   * @param typeTag allows for inferring default value from the FeatureValue type.\n   *                See [[com.twitter.product_mixer.component_library.selector.sorter.FeatureValueSorter.featureValueSortDefaultValue]]\n   * @tparam Candidate candidate for the feature\n   * @tparam FeatureValue feature value with an [[Ordering]] context bound\n   */\n  def ascending[Candidate <: UniversalNoun[Any], FeatureValue: Ordering](\n    feature: FeatureStoreV1CandidateFeature[PipelineQuery, Candidate, _ <: EntityId, FeatureValue]\n  )(\n    implicit typeTag: TypeTag[FeatureValue]\n  ): SorterProvider = {\n    val defaultFeatureValue: FeatureValue = featureValueSortDefaultValue(feature, Ascending)\n\n    ascending(feature, defaultFeatureValue)\n  }\n\n  /**\n   * Sort by a Feature Store v1 feature value ascending. If the feature failed or is missing, use\n   * the provided default.\n   *\n   * @param feature Feature Store v1 feature with value to sort by\n   * @tparam Candidate candidate for the feature\n   * @tparam FeatureValue feature value with an [[Ordering]] context bound\n   */\n  def ascending[Candidate <: UniversalNoun[Any], FeatureValue: Ordering](\n    feature: FeatureStoreV1CandidateFeature[PipelineQuery, Candidate, _ <: EntityId, FeatureValue],\n    defaultFeatureValue: FeatureValue\n  ): SorterProvider = {\n    val ordering = Ordering.by[CandidateWithDetails, FeatureValue](\n      _.features.getOrElseFeatureStoreV1CandidateFeature(feature, defaultFeatureValue))\n\n    SorterFromOrdering(ordering, Ascending)\n  }\n\n  /**\n   * Sort by a Feature Store v1 feature value descending. If the feature failed or is missing, use\n   * an inferred default based on the type of [[FeatureValue]]. For Numeric values this is the\n   * MaxValue (e.g. Long.MaxValue, Double.MaxValue).\n   *\n   * @param feature Feature Store v1 feature with value to sort by\n   * @param typeTag allows for inferring default value from the FeatureValue type.\n   *                See [[com.twitter.product_mixer.component_library.selector.sorter.FeatureValueSorter.featureValueSortDefaultValue]]\n   * @tparam Candidate candidate for the feature\n   * @tparam FeatureValue feature value with an [[Ordering]] context bound\n   */\n  def descending[Candidate <: UniversalNoun[Any], FeatureValue: Ordering](\n    feature: FeatureStoreV1CandidateFeature[PipelineQuery, Candidate, _ <: EntityId, FeatureValue]\n  )(\n    implicit typeTag: TypeTag[FeatureValue]\n  ): SorterProvider = {\n    val defaultFeatureValue: FeatureValue = featureValueSortDefaultValue(feature, Descending)\n\n    descending(feature, defaultFeatureValue)\n  }\n\n  /**\n   * Sort by a Feature Store v1 feature value descending. If the feature failed or is missing, use\n   * the provided default.\n   *\n   * @param feature Feature Store v1 feature with value to sort by\n   * @tparam Candidate candidate for the feature\n   * @tparam FeatureValue feature value with an [[Ordering]] context bound\n   */\n  def descending[Candidate <: UniversalNoun[Any], FeatureValue: Ordering](\n    feature: FeatureStoreV1CandidateFeature[PipelineQuery, Candidate, _ <: EntityId, FeatureValue],\n    defaultFeatureValue: FeatureValue\n  ): SorterProvider = {\n    val ordering = Ordering.by[CandidateWithDetails, FeatureValue](\n      _.features.getOrElseFeatureStoreV1CandidateFeature(feature, defaultFeatureValue))\n\n    SorterFromOrdering(ordering, Descending)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/src/java/com/twitter/logpipeline/client:logpipeline-event-publisher-thin\",\n        \"abdecider/src/main/scala:abdeciderutils\",\n        \"decider/src/main/scala\",\n        \"finatra-internal/messaging/kafka/src/main/scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect\",\n        \"scribelib/marshallers/src/main/scala/com/twitter/scribelib/marshallers\",\n        \"scribelib/validators/src/main/scala/com/twitter/scribelib/validators\",\n        \"scrooge/scrooge-serializer/src/main/scala\",\n        \"src/thrift/com/twitter/clientapp/gen:clientapp-scala\",\n        \"stitch/stitch-core\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"user_session_store/src/main/scala/com/twitter/user_session_store\",\n        \"util/util-core:util-core-util\",\n    ],\n    exports = [\n        \"3rdparty/jvm/com/twitter/src/java/com/twitter/logpipeline/client:logpipeline-event-publisher-thin\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect\",\n        \"scrooge/scrooge-serializer/src/main/scala\",\n        \"src/thrift/com/twitter/clientapp/gen:clientapp-scala\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect/KafkaPublishingSideEffect.scala",
    "content": "package com.twitter.product_mixer.component_library.side_effect\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.conversions.StorageUnitOps._\nimport com.twitter.finatra.kafka.producers.FinagleKafkaProducerBuilder\nimport com.twitter.finatra.kafka.producers.KafkaProducerBase\nimport com.twitter.finatra.kafka.producers.TwitterKafkaProducerConfig\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Duration\nimport com.twitter.util.StorageUnit\nimport org.apache.kafka.clients.producer.ProducerRecord\nimport org.apache.kafka.common.serialization.Serializer\nimport org.apache.kafka.common.record.CompressionType\n\n/**\n * The Kafka publishing side effect.\n * This class creates a Kafka producer with provided and default parameters.\n * Note that callers may not provide arbitrary params as this class will do validity check on some\n * params, e.g. maxBlock, to make sure it is safe for online services.\n *\n * PLEASE NOTE: caller needs to add the following to the Aurora file to successfully enable the TLS\n * '-com.twitter.finatra.kafka.producers.principal={{role}}',\n *\n * @tparam K type of the key\n * @tparam V type of the value\n * @tparam Query pipeline query\n */\ntrait KafkaPublishingSideEffect[K, V, Query <: PipelineQuery, ResponseType <: HasMarshalling]\n    extends PipelineResultSideEffect[Query, ResponseType] {\n\n  /**\n   * Kafka servers list. It is usually a WilyNs name at Twitter\n   */\n  val bootstrapServer: String\n\n  /**\n   * The serde of the key\n   */\n  val keySerde: Serializer[K]\n\n  /**\n   * The serde of the value\n   */\n  val valueSerde: Serializer[V]\n\n  /**\n   * An id string to pass to the server when making requests.\n   * The purpose of this is to be able to track the source of requests beyond just ip/port by\n   * allowing a logical application name to be included in server-side request logging.\n   */\n  val clientId: String\n\n  /**\n   * The configuration controls how long <code>KafkaProducer.send()</code> and\n   * <code>KafkaProducer.partitionsFor()</code> will block.\n   * These methods can be blocked either because the buffer is full or metadata unavailable.\n   * Blocking in the user-supplied serializers or partitioner will not be counted against this timeout.\n   *\n   * Set 200ms by default to not blocking the thread too long which is critical to most ProMixer\n   * powered services. Please note that there is a hard limit check of not greater than 1 second.\n   *\n   */\n  val maxBlock: Duration = 200.milliseconds\n\n  /**\n   * Retries due to broker failures, etc., may write duplicates of the retried message in the\n   * stream. Note that enabling idempotence requires\n   * <code> MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION </code> to be less than or equal to 5,\n   * <code> RETRIES_CONFIG </code> to be greater than 0 and <code> ACKS_CONFIG </code>\n   * must be 'all'. If these values are not explicitly set by the user, suitable values will be\n   * chosen. If incompatible values are set, a <code>ConfigException</code> will be thrown.\n   *\n   * false by default, setting to true may introduce issues to brokers since brokers will keep\n   * tracking all requests which is resource expensive.\n   */\n  val idempotence: Boolean = false\n\n  /**\n   * The producer will attempt to batch records together into fewer requests whenever multiple\n   * records are being sent to the same partition. This helps performance on both the client and\n   * the server. This configuration controls the default batch size in bytes.\n   * No attempt will be made to batch records larger than this size.\n   * Requests sent to brokers will contain multiple batches, one for each partition with data\n   * available to be sent. A small batch size will make batching less common and may reduce\n   * throughput (a batch size of zero will disable batching entirely).\n   * A very large batch size may use memory a bit more wastefully as we will always allocate a\n   * buffer of the specified batch size in anticipation of additional records.\n   *\n   * Default 16KB which comes from Kafka's default\n   */\n  val batchSize: StorageUnit = 16.kilobytes\n\n  /**\n   * The producer groups together any records that arrive in between request transmissions into\n   * a single batched request. \"Normally this occurs only under load when records arrive faster\n   * than they can be sent out. However in some circumstances the client may want to reduce the\n   * number of requests even under moderate load. This setting accomplishes this by adding a\n   * small amount of artificial delay&mdash;that is, rather than immediately sending out a record\n   * the producer will wait for up to the given delay to allow other records to be sent so that\n   * the sends can be batched together. This can be thought of as analogous to Nagle's algorithm\n   * in TCP. This setting gives the upper bound on the delay for batching: once we get\n   * BATCH_SIZE_CONFIG worth of records for a partition it will be sent immediately regardless\n   * of this setting, however if we have fewer than this many bytes accumulated for this\n   * partition we will 'linger' for the specified time waiting for more records to show up.\n   * This setting defaults to 0 (i.e. no delay). Setting LINGER_MS_CONFIG=5, for example,\n   * would have the effect of reducing the number of requests sent but would add up to 5ms of\n   * latency to records sent in the absence of load.\n   *\n   * Default 0ms, which is Kafka's default. If the record size is much larger than the batchSize,\n   * you may consider to enlarge both batchSize and linger to have better compression (only when\n   * compression is enabled.)\n   */\n  val linger: Duration = 0.milliseconds\n\n  /**\n   * The total bytes of memory the producer can use to buffer records waiting to be sent to the\n   * server. If records are sent faster than they can be delivered to the server the producer\n   * will block for MAX_BLOCK_MS_CONFIG after which it will throw an exception.\n   * This setting should correspond roughly to the total memory the producer will use, but is not\n   * a hard bound since not all memory the producer uses is used for buffering.\n   * Some additional memory will be used for compression (if compression is enabled) as well as\n   * for maintaining in-flight requests.\n   *\n   * Default 32MB which is Kafka's default. Please consider to enlarge this value if the EPS and\n   * the per-record size is large (millions EPS with >1KB per-record size) in case the broker has\n   * issues (which fills the buffer pretty quickly.)\n   */\n  val bufferMemorySize: StorageUnit = 32.megabytes\n\n  /**\n   * Producer compression type\n   *\n   * Default LZ4 which is a good tradeoff between compression and efficiency.\n   * Please be careful of choosing ZSTD, which the compression rate is better it might introduce\n   * huge burden to brokers once the topic is consumed, which needs decompression at the broker side.\n   */\n  val compressionType: CompressionType = CompressionType.LZ4\n\n  /**\n   * Setting a value greater than zero will cause the client to resend any request that fails\n   * with a potentially transient error\n   *\n   * Default set to 3, to intentionally reduce the retries.\n   */\n  val retries: Int = 3\n\n  /**\n   * The amount of time to wait before attempting to retry a failed request to a given topic\n   * partition. This avoids repeatedly sending requests in a tight loop under some failure\n   * scenarios\n   */\n  val retryBackoff: Duration = 1.second\n\n  /**\n   * The configuration controls the maximum amount of time the client will wait\n   * for the response of a request. If the response is not received before the timeout\n   * elapses the client will resend the request if necessary or fail the request if\n   * retries are exhausted.\n   *\n   * Default 5 seconds which is intentionally low but not too low.\n   * Since Kafka's publishing is async this is in general safe (as long as the bufferMem is not full.)\n   */\n  val requestTimeout: Duration = 5.seconds\n\n  require(\n    maxBlock.inMilliseconds <= 1000,\n    \"We intentionally set the maxBlock to be smaller than 1 second to not block the thread for too long!\")\n\n  lazy val kafkaProducer: KafkaProducerBase[K, V] = {\n    val jaasConfig = TwitterKafkaProducerConfig().configMap\n    val builder = FinagleKafkaProducerBuilder[K, V]()\n      .keySerializer(keySerde)\n      .valueSerializer(valueSerde)\n      .dest(bootstrapServer, 1.second) // NOTE: this method blocks!\n      .clientId(clientId)\n      .maxBlock(maxBlock)\n      .batchSize(batchSize)\n      .linger(linger)\n      .bufferMemorySize(bufferMemorySize)\n      .maxRequestSize(4.megabytes)\n      .compressionType(compressionType)\n      .enableIdempotence(idempotence)\n      .maxInFlightRequestsPerConnection(5)\n      .retries(retries)\n      .retryBackoff(retryBackoff)\n      .requestTimeout(requestTimeout)\n      .withConfig(\"acks\", \"all\")\n      .withConfig(\"delivery.timeout.ms\", requestTimeout + linger)\n\n    builder.withConfig(jaasConfig).build()\n  }\n\n  /**\n   * Build the record to be published to Kafka from query, selections and response\n   * @param query PipelineQuery\n   * @param selectedCandidates Result after Selectors are executed\n   * @param remainingCandidates Candidates which were not selected\n   * @param droppedCandidates Candidates dropped during selection\n   * @param response Result after Unmarshalling\n   * @return A sequence of to-be-published ProducerRecords\n   */\n  def buildRecords(\n    query: Query,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: ResponseType\n  ): Seq[ProducerRecord[K, V]]\n\n  final override def apply(\n    inputs: PipelineResultSideEffect.Inputs[Query, ResponseType]\n  ): Stitch[Unit] = {\n    val records = buildRecords(\n      query = inputs.query,\n      selectedCandidates = inputs.selectedCandidates,\n      remainingCandidates = inputs.remainingCandidates,\n      droppedCandidates = inputs.droppedCandidates,\n      response = inputs.response\n    )\n\n    Stitch\n      .collect(\n        records\n          .map { record =>\n            Stitch.callFuture(kafkaProducer.send(record))\n          }\n      ).unit\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect/ParamGatedPipelineResultSideEffect.scala",
    "content": "package com.twitter.product_mixer.component_library.side_effect\n\nimport com.twitter.product_mixer.component_library.side_effect.ParamGatedPipelineResultSideEffect.IdentifierPrefix\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.side_effect.ExecuteSynchronously\nimport com.twitter.product_mixer.core.functional_component.side_effect.FailOpen\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\n/**\n * A [[PipelineResultSideEffect]] with [[Conditionally]] based on a [[Param]]\n *\n * @param enabledParam the param to turn this filter on and off\n * @param sideEffect the underlying side effect to run when `enabledParam` is true\n * @tparam Query The domain model for the query or request\n */\nsealed case class ParamGatedPipelineResultSideEffect[\n  -Query <: PipelineQuery,\n  ResultType <: HasMarshalling\n] private (\n  enabledParam: Param[Boolean],\n  sideEffect: PipelineResultSideEffect[Query, ResultType])\n    extends PipelineResultSideEffect[Query, ResultType]\n    with PipelineResultSideEffect.Conditionally[Query, ResultType] {\n  override val identifier: SideEffectIdentifier = SideEffectIdentifier(\n    IdentifierPrefix + sideEffect.identifier.name)\n  override val alerts: Seq[Alert] = sideEffect.alerts\n  override def onlyIf(\n    query: Query,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: ResultType\n  ): Boolean =\n    Conditionally.and(\n      PipelineResultSideEffect\n        .Inputs(query, selectedCandidates, remainingCandidates, droppedCandidates, response),\n      sideEffect,\n      query.params(enabledParam))\n  override def apply(inputs: PipelineResultSideEffect.Inputs[Query, ResultType]): Stitch[Unit] =\n    sideEffect.apply(inputs)\n}\n\nobject ParamGatedPipelineResultSideEffect {\n\n  val IdentifierPrefix = \"ParamGated\"\n\n  /**\n   * A [[PipelineResultSideEffect]] with [[Conditionally]] based on a [[Param]]\n   *\n   * @param enabledParam the param to turn this filter on and off\n   * @param sideEffect the underlying side effect to run when `enabledParam` is true\n   * @tparam Query The domain model for the query or request\n   */\n  def apply[Query <: PipelineQuery, ResultType <: HasMarshalling](\n    enabledParam: Param[Boolean],\n    sideEffect: PipelineResultSideEffect[Query, ResultType]\n  ): ParamGatedPipelineResultSideEffect[Query, ResultType] = {\n    sideEffect match {\n      case _: FailOpen =>\n        new ParamGatedPipelineResultSideEffect(enabledParam, sideEffect)\n          with ExecuteSynchronously\n          with FailOpen\n      case _: ExecuteSynchronously =>\n        new ParamGatedPipelineResultSideEffect(enabledParam, sideEffect) with ExecuteSynchronously\n      case _ =>\n        new ParamGatedPipelineResultSideEffect(enabledParam, sideEffect)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect/ScribeClientEventSideEffect.scala",
    "content": "package com.twitter.product_mixer.component_library.side_effect\n\nimport com.twitter.abdecider.ScribingABDeciderUtil\nimport com.twitter.clientapp.thriftscala.LogEvent\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.scribelib.marshallers\nimport com.twitter.scribelib.marshallers.ClientDataProvider\nimport com.twitter.scribelib.marshallers.LogEventMarshaller\n\n/**\n * Side effect to log client events server-side. Create an implementation of this trait by\n * defining the `buildClientEvents` method, and the `page` val.\n * The ClientEvent will be automatically converted into a [[LogEvent]] and scribed.\n */\ntrait ScribeClientEventSideEffect[\n  Query <: PipelineQuery,\n  UnmarshalledResponseType <: HasMarshalling]\n    extends ScribeLogEventSideEffect[LogEvent, Query, UnmarshalledResponseType] {\n\n  /**\n   * The page which will be defined in the namespace. This is typically the service name that's scribing\n   */\n  val page: String\n\n  /**\n   * Build the client events from query, selections and response\n   *\n   * @param query PipelineQuery\n   * @param selectedCandidates Result after Selectors are executed\n   * @param remainingCandidates Candidates which were not selected\n   * @param droppedCandidates Candidates dropped during selection\n   * @param response Result after Unmarshalling\n   */\n  def buildClientEvents(\n    query: Query,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: UnmarshalledResponseType\n  ): Seq[ScribeClientEventSideEffect.ClientEvent]\n\n  final override def buildLogEvents(\n    query: Query,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: UnmarshalledResponseType\n  ): Seq[LogEvent] = {\n    buildClientEvents(\n      query = query,\n      selectedCandidates = selectedCandidates,\n      remainingCandidates = remainingCandidates,\n      droppedCandidates = droppedCandidates,\n      response = response).flatMap { event =>\n      val clientData = clientContextToClientDataProvider(query)\n\n      val clientName = ScribingABDeciderUtil.clientForAppId(clientData.clientApplicationId)\n\n      val namespaceMap: Map[String, String] = Map(\n        \"client\" -> Some(clientName),\n        \"page\" -> Some(page),\n        \"section\" -> event.namespace.section,\n        \"component\" -> event.namespace.component,\n        \"element\" -> event.namespace.element,\n        \"action\" -> event.namespace.action\n      ).collect { case (k, Some(v)) => k -> v }\n\n      val data: Map[Any, Any] = Seq(\n        event.eventValue.map(\"event_value\" -> _),\n        event.latencyMs.map(\"latency_ms\" -> _)\n      ).flatten.toMap\n\n      val clientEventData = data +\n        (\"event_namespace\" -> namespaceMap) +\n        (marshallers.CategoryKey -> \"client_event\")\n\n      LogEventMarshaller.marshal(\n        data = clientEventData,\n        clientData = clientData\n      )\n    }\n  }\n\n  /**\n   * Makes a [[ClientDataProvider]] from the [[PipelineQuery.clientContext]] from the [[query]]\n   */\n  private def clientContextToClientDataProvider(query: Query): ClientDataProvider = {\n    new ClientDataProvider {\n      override val userId = query.clientContext.userId\n      override val guestId = query.clientContext.guestId\n      override val personalizationId = None\n      override val deviceId = query.clientContext.deviceId\n      override val clientApplicationId = query.clientContext.appId\n      override val parentApplicationId = None\n      override val countryCode = query.clientContext.countryCode\n      override val languageCode = query.clientContext.languageCode\n      override val userAgent = query.clientContext.userAgent\n      override val isSsl = None\n      override val referer = None\n      override val externalReferer = None\n    }\n  }\n}\n\nobject ScribeClientEventSideEffect {\n  case class EventNamespace(\n    section: Option[String] = None,\n    component: Option[String] = None,\n    element: Option[String] = None,\n    action: Option[String] = None)\n\n  case class ClientEvent(\n    namespace: EventNamespace,\n    eventValue: Option[Long] = None,\n    latencyMs: Option[Long] = None)\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect/ScribeLogEventAsyncSideEffect.scala",
    "content": "package com.twitter.product_mixer.component_library.side_effect\n\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.scrooge.ThriftStruct\nimport com.twitter.stitch.Stitch\n\n/**\n * A [[PipelineResultSideEffect]] that logs [[Thrift]] data may not already be available to Scribe\n */\ntrait ScribeLogEventAsyncSideEffect[\n  Thrift <: ThriftStruct,\n  Query <: PipelineQuery,\n  ResponseType <: HasMarshalling]\n    extends PipelineResultSideEffect[Query, ResponseType] {\n\n  /**\n   * Build the log events from query, selections and response\n   * @param query PipelineQuery\n   * @param selectedCandidates Result after Selectors are executed\n   * @param remainingCandidates Candidates which were not selected\n   * @param droppedCandidates Candidates dropped during selection\n   * @param response Result after Unmarshalling\n   * @return LogEvent in thrift\n   */\n  def buildLogEvents(\n    query: Query,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: ResponseType\n  ): Stitch[Seq[Thrift]]\n\n  val logPipelinePublisher: EventPublisher[Thrift]\n\n  final override def apply(\n    inputs: PipelineResultSideEffect.Inputs[Query, ResponseType]\n  ): Stitch[Unit] = {\n    val logEvents = buildLogEvents(\n      query = inputs.query,\n      selectedCandidates = inputs.selectedCandidates,\n      remainingCandidates = inputs.remainingCandidates,\n      droppedCandidates = inputs.droppedCandidates,\n      response = inputs.response\n    )\n\n    logEvents.flatMap { logEvents: Seq[Thrift] =>\n      Stitch.collect {\n        logEvents.map { logEvent =>\n          Stitch.callFuture(logPipelinePublisher.publish(logEvent))\n        }\n      }.unit\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect/ScribeLogEventSideEffect.scala",
    "content": "package com.twitter.product_mixer.component_library.side_effect\n\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.scrooge.ThriftStruct\nimport com.twitter.stitch.Stitch\n\n/**\n * A [[PipelineResultSideEffect]] that logs [[Thrift]] data that's already available to Scribe\n */\ntrait ScribeLogEventSideEffect[\n  Thrift <: ThriftStruct,\n  Query <: PipelineQuery,\n  ResponseType <: HasMarshalling]\n    extends PipelineResultSideEffect[Query, ResponseType] {\n\n  /**\n   * Build the log events from query, selections and response\n   * @param query PipelineQuery\n   * @param selectedCandidates Result after Selectors are executed\n   * @param remainingCandidates Candidates which were not selected\n   * @param droppedCandidates Candidates dropped during selection\n   * @param response Result after Unmarshalling\n   * @return LogEvent in thrift\n   */\n  def buildLogEvents(\n    query: Query,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: ResponseType\n  ): Seq[Thrift]\n\n  val logPipelinePublisher: EventPublisher[Thrift]\n\n  final override def apply(\n    inputs: PipelineResultSideEffect.Inputs[Query, ResponseType]\n  ): Stitch[Unit] = {\n    val logEvents = buildLogEvents(\n      query = inputs.query,\n      selectedCandidates = inputs.selectedCandidates,\n      remainingCandidates = inputs.remainingCandidates,\n      droppedCandidates = inputs.droppedCandidates,\n      response = inputs.response\n    )\n\n    Stitch\n      .collect(\n        logEvents\n          .map { logEvent =>\n            Stitch.callFuture(logPipelinePublisher.publish(logEvent))\n          }\n      ).unit\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect/StratoInsertSideEffect.scala",
    "content": "package com.twitter.product_mixer.component_library.side_effect\n\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Inserter\n\n/**\n * Side effect that writes to Strato column's Insert Op. Create an implementation of this trait by\n * defining the `buildEvents` method and providing a Strato Column inserter of type\n * (StratoKeyarg, StratoValue) -> Any.\n * See https://docbird.twitter.biz/strato/ColumnCatalog.html#insert for information about\n * the Insert operation in Strato.\n *\n * @tparam StratoKeyarg Argument used as a key for Strato column. Could be Unit for common use-cases.\n * @tparam StratoValue Value that is inserted at the Strato column.\n * @tparam Query PipelineQuery\n * @tparam DomainResponseType Timeline response that is marshalled to domain model (e.g. URT, Slice etc).\n */\ntrait StratoInsertSideEffect[\n  StratoKeyarg,\n  StratoValue,\n  Query <: PipelineQuery,\n  DomainResponseType <: HasMarshalling]\n    extends PipelineResultSideEffect[Query, DomainResponseType] {\n\n  /**\n   * Inserter for the InsertOp on a StratoColumn. In Strato, the InsertOp is represented as\n   * (Keyarg, Value) => Key, where Key represents the result returned by the Insert operation.\n   * For the side-effect behavior, we do not need the return value and use Any instead.\n   */\n  val stratoInserter: Inserter[StratoKeyarg, StratoValue, Any]\n\n  /**\n   * Builds the events that are inserted to the Strato column. This method supports generating\n   * multiple events for a single side-effect invocation.\n   *\n   * @param query PipelineQuery\n   * @param selectedCandidates Result after Selectors are executed\n   * @param remainingCandidates Candidates which were not selected\n   * @param droppedCandidates Candidates dropped during selection\n   * @param response Timeline response that is marshalled to domain model (e.g. URT, Slice etc).\n   * @return Tuples of (StratoKeyArg, StratoValue) that are used to call the stratoInserter.\n   */\n  def buildEvents(\n    query: Query,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: DomainResponseType\n  ): Seq[(StratoKeyarg, StratoValue)]\n\n  final override def apply(\n    inputs: PipelineResultSideEffect.Inputs[Query, DomainResponseType]\n  ): Stitch[Unit] = {\n    val events = buildEvents(\n      query = inputs.query,\n      selectedCandidates = inputs.selectedCandidates,\n      remainingCandidates = inputs.remainingCandidates,\n      droppedCandidates = inputs.droppedCandidates,\n      response = inputs.response\n    )\n\n    Stitch\n      .traverse(events) { case (keyarg, value) => stratoInserter.insert(keyarg, value) }\n      .unit\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect/UserSessionStoreUpdateSideEffect.scala",
    "content": "package com.twitter.product_mixer.component_library.side_effect\n\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.user_session_store.ReadWriteUserSessionStore\nimport com.twitter.user_session_store.WriteRequest\n\n/**\n * A [[PipelineResultSideEffect]] that writes to a [[ReadWriteUserSessionStore]]\n */\ntrait UserSessionStoreUpdateSideEffect[\n  Request <: WriteRequest,\n  Query <: PipelineQuery,\n  ResponseType <: HasMarshalling]\n    extends PipelineResultSideEffect[Query, ResponseType] {\n\n  /**\n   * Build the write request from the query\n   * @param query PipelineQuery\n   * @return WriteRequest\n   */\n  def buildWriteRequest(query: Query): Option[Request]\n\n  val userSessionStore: ReadWriteUserSessionStore\n\n  final override def apply(\n    inputs: PipelineResultSideEffect.Inputs[Query, ResponseType]\n  ): Stitch[Unit] = {\n    buildWriteRequest(inputs.query)\n      .map(userSessionStore.write)\n      .getOrElse(Stitch.Unit)\n  }\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect/metrics/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"util/util-core:util-core-util\",\n        \"util/util-slf4j-api/src/main/scala/com/twitter/util/logging\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect/metrics/CandidateMetricFunction.scala",
    "content": "package com.twitter.product_mixer.component_library.side_effect.metrics\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.component_library.model.candidate.BaseUserCandidate\nimport com.twitter.product_mixer.component_library.side_effect.metrics.CandidateMetricFunction.getCountForType\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\n\n/**\n * Function to extract numerical metric value from [[CandidateWithDetails]].\n * This CandidateMetricFunction will be applied on all [[CandidateWithDetails]] instances in the\n * candidateSelection from the RecommendationPipeline.\n */\ntrait CandidateMetricFunction {\n  def apply(candidateWithDetails: CandidateWithDetails): Long\n}\n\nobject CandidateMetricFunction {\n\n  private val defaultCountOnePf: PartialFunction[CandidateWithDetails, Long] = {\n    case _ => 0L\n  }\n\n  /**\n   * Count the occurrences of a certain candidate type from [[CandidateWithDetails]].\n   */\n  def getCountForType(\n    candidateWithDetails: CandidateWithDetails,\n    countOnePf: PartialFunction[CandidateWithDetails, Long]\n  ): Long = {\n    (countOnePf orElse defaultCountOnePf)(candidateWithDetails)\n  }\n}\n\nobject DefaultServedTweetsSumFunction extends CandidateMetricFunction {\n  override def apply(candidateWithDetails: CandidateWithDetails): Long =\n    getCountForType(\n      candidateWithDetails,\n      {\n        case item: ItemCandidateWithDetails =>\n          item.candidate match {\n            case _: BaseTweetCandidate => 1L\n            case _ => 0L\n          }\n      })\n}\n\nobject DefaultServedUsersSumFunction extends CandidateMetricFunction {\n  override def apply(candidateWithDetails: CandidateWithDetails): Long =\n    getCountForType(\n      candidateWithDetails,\n      {\n        case item: ItemCandidateWithDetails =>\n          item.candidate match {\n            case _: BaseUserCandidate => 1L\n            case _ => 0L\n          }\n      })\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect/metrics/ScribeClientEventMetricsSideEffect.scala",
    "content": "package com.twitter.product_mixer.component_library.side_effect.metrics\n\nimport com.twitter.clientapp.thriftscala.LogEvent\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.side_effect.ScribeClientEventSideEffect\nimport com.twitter.product_mixer.component_library.side_effect.ScribeClientEventSideEffect.EventNamespace\nimport com.twitter.product_mixer.component_library.side_effect.ScribeClientEventSideEffect.ClientEvent\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Config of a client event to be scribed under certain namespace.\n * @param eventNamespaceOverride overrides the default eventNamespace in the side effect.\n *                               Note that its fields (section/component/element/action) will\n *                               override the default namespace fields only if the fields are not\n *                               None. i.e. if you want to override the \"section\" field in the\n *                               default namespace with an empty section, you must specify\n *                                  section = Some(\"\")\n *                               in the override instead of\n *                                  section = None\n *\n * @param metricFunction the function that extracts the metric value from a candidate.\n */\ncase class EventConfig(\n  eventNamespaceOverride: EventNamespace,\n  metricFunction: CandidateMetricFunction)\n\n/**\n * Side effect to log client events server-side and to build metrics in the metric center.\n * By default will return \"requests\" event config.\n */\nclass ScribeClientEventMetricsSideEffect[\n  Query <: PipelineQuery,\n  UnmarshalledResponseType <: HasMarshalling\n](\n  override val identifier: SideEffectIdentifier,\n  override val logPipelinePublisher: EventPublisher[LogEvent],\n  override val page: String,\n  defaultEventNamespace: EventNamespace,\n  eventConfigs: Seq[EventConfig])\n    extends ScribeClientEventSideEffect[Query, UnmarshalledResponseType] {\n\n  override def buildClientEvents(\n    query: Query,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: UnmarshalledResponseType\n  ): Seq[ScribeClientEventSideEffect.ClientEvent] = {\n    // count the number of client events of type \"requests\"\n    val requestClientEvent = ClientEvent(\n      namespace = buildEventNamespace(EventNamespace(action = Some(\"requests\")))\n    )\n\n    eventConfigs\n      .map { config =>\n        ClientEvent(\n          namespace = buildEventNamespace(config.eventNamespaceOverride),\n          eventValue = Some(selectedCandidates.map(config.metricFunction(_)).sum))\n      }\n      // scribe client event only when the metric sum is non-zero\n      .filter(clientEvent => clientEvent.eventValue.exists(_ > 0L)) :+ requestClientEvent\n  }\n\n  private def buildEventNamespace(eventNamespaceOverride: EventNamespace): EventNamespace =\n    EventNamespace(\n      section = eventNamespaceOverride.section.orElse(defaultEventNamespace.section),\n      component = eventNamespaceOverride.component.orElse(defaultEventNamespace.component),\n      element = eventNamespaceOverride.element.orElse(defaultEventNamespace.element),\n      action = eventNamespaceOverride.action.orElse(defaultEventNamespace.action)\n    )\n}\n"
  },
  {
    "path": "product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/side_effect/metrics/ScribeClientEventMetricsSideEffectBuilder.scala",
    "content": "package com.twitter.product_mixer.component_library.side_effect.metrics\n\nimport com.twitter.clientapp.thriftscala.LogEvent\nimport com.twitter.logpipeline.client.common.EventPublisher\nimport com.twitter.product_mixer.component_library.side_effect.ScribeClientEventSideEffect.EventNamespace\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Build [[ScribeClientEventMetricsSideEffect]] with extra [[EventConfig]]\n */\ncase class ScribeClientEventMetricsSideEffectBuilder(\n  eventConfigs: Seq[EventConfig] = Seq.empty) {\n\n  /**\n   * Append extra [[EventConfig]] to [[ScribeClientEventMetricsSideEffectBuilder]]\n   */\n  def withEventConfig(\n    eventConfig: EventConfig\n  ): ScribeClientEventMetricsSideEffectBuilder =\n    this.copy(eventConfigs = this.eventConfigs :+ eventConfig)\n\n  /**\n   * Build [[EventConfig]] with customized [[EventNamespace]] and customized [[CandidateMetricFunction]]\n   * @param eventNamespaceOverride Override the default event namespace in [[ScribeClientEventMetricsSideEffect]]\n   * @param metricFunction [[CandidateMetricFunction]]\n   */\n  def withEventConfig(\n    eventNamespaceOverride: EventNamespace,\n    metricFunction: CandidateMetricFunction\n  ): ScribeClientEventMetricsSideEffectBuilder =\n    withEventConfig(EventConfig(eventNamespaceOverride, metricFunction))\n\n  /**\n   * Log served tweets events server side and build metrics in the metric center.\n   * Default event name space action is \"served_tweets\", default metric function is [[DefaultServedTweetsSumFunction]]\n   * @param eventNamespaceOverride Override the default event namespace in [[ScribeClientEventMetricsSideEffect]]\n   * @param metricFunction [[CandidateMetricFunction]]\n   */\n  def withServedTweets(\n    eventNamespaceOverride: EventNamespace = EventNamespace(action = Some(\"served_tweets\")),\n    metricFunction: CandidateMetricFunction = DefaultServedTweetsSumFunction\n  ): ScribeClientEventMetricsSideEffectBuilder = withEventConfig(\n    eventNamespaceOverride = eventNamespaceOverride,\n    metricFunction = metricFunction)\n\n  /**\n   * Log served users events server side and build metrics in the metric center.\n   * Default event name space action is \"served_users\", default metric function is [[DefaultServedUsersSumFunction]]\n   * @param eventNamespaceOverride Override the default event namespace in [[ScribeClientEventMetricsSideEffect]]\n   * @param metricFunction [[CandidateMetricFunction]]\n   */\n  def withServedUsers(\n    eventNamespaceOverride: EventNamespace = EventNamespace(action = Some(\"served_users\")),\n    metricFunction: CandidateMetricFunction = DefaultServedUsersSumFunction\n  ): ScribeClientEventMetricsSideEffectBuilder = withEventConfig(\n    eventNamespaceOverride = eventNamespaceOverride,\n    metricFunction = metricFunction)\n\n  /**\n   * Build [[ScribeClientEventMetricsSideEffect]]\n   * @param identifier unique identifier of the side effect\n   * @param defaultEventNamespace default event namespace to log\n   * @param logPipelinePublisher [[EventPublisher]] to publish events\n   * @param page The page which will be defined in the namespace. This is typically the service name that's scribing\n   * @tparam Query [[PipelineQuery]]\n   * @tparam UnmarshalledResponseType [[HasMarshalling]]\n   * @return [[ScribeClientEventMetricsSideEffect]]\n   */\n  def build[Query <: PipelineQuery, UnmarshalledResponseType <: HasMarshalling](\n    identifier: SideEffectIdentifier,\n    defaultEventNamespace: EventNamespace,\n    logPipelinePublisher: EventPublisher[LogEvent],\n    page: String\n  ): ScribeClientEventMetricsSideEffect[Query, UnmarshalledResponseType] = {\n    new ScribeClientEventMetricsSideEffect[Query, UnmarshalledResponseType](\n      identifier = identifier,\n      logPipelinePublisher = logPipelinePublisher,\n      defaultEventNamespace = defaultEventNamespace,\n      page = page,\n      eventConfigs = eventConfigs)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope/ProductScoped.java",
    "content": "package com.twitter.product_mixer.core.product.guice.scope;\n\nimport static java.lang.annotation.ElementType.METHOD;\nimport static java.lang.annotation.ElementType.TYPE;\nimport java.lang.annotation.Retention;\nimport static java.lang.annotation.RetentionPolicy.RUNTIME;\nimport java.lang.annotation.Target;\n\nimport com.google.inject.ScopeAnnotation;\n\n@Target({ TYPE, METHOD })\n@Retention(RUNTIME)\n@ScopeAnnotation\npublic @interface ProductScoped {}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers/AlertConfig.scala",
    "content": "package com.twitter.product_mixer.core.controllers\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.common.alert.NotificationGroup\nimport com.twitter.product_mixer.core.functional_component.common.alert.Source\n\n/**\n * Simple representation for an [[Alert]] used for Product Mixer's JSON API, which in turn is\n * consumed by our monitoring script generation job and Turntable.\n *\n * @note not all mixers will upgrade at the same time so new fields should be added with backwards\n *       compatibility in mind.\n */\n@JsonIgnoreProperties(ignoreUnknown = true)\nprivate[core] case class AlertConfig(\n  source: Source,\n  metricType: String,\n  notificationGroup: NotificationGroup,\n  warnPredicate: PredicateConfig,\n  criticalPredicate: PredicateConfig,\n  runbookLink: Option[String],\n  metricSuffix: Option[String])\n\nprivate[core] object AlertConfig {\n\n  /** Represent this [[Alert]] as an [[AlertConfig]] case class */\n  private[core] def apply(alert: Alert): AlertConfig =\n    AlertConfig(\n      alert.source,\n      alert.alertType.metricType,\n      alert.notificationGroup,\n      PredicateConfig(alert.warnPredicate),\n      PredicateConfig(alert.criticalPredicate),\n      alert.runbookLink,\n      alert.metricSuffix\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/bijection:core\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/http-server/src/main/scala/com/twitter/finatra/http\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/access_policy\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/product_mixer_flags\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/recommendation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/component_registry\",\n        \"scrooge-internal/scrooge-schema/src/main/scala/com/twitter/scrooge/schema\",\n        \"scrooge-internal/scrooge-schema/src/main/scala/com/twitter/scrooge/schema/json\",\n        \"scrooge-internal/scrooge-schema/src/main/scala/com/twitter/scrooge/schema/scrooge/scala\",\n        \"scrooge-internal/scrooge-schema/src/main/scala/com/twitter/scrooge/schema/serialization/thrift\",\n        \"scrooge-internal/scrooge-schema/src/main/scala/com/twitter/scrooge/schema/tree\",\n        \"scrooge/scrooge-core/src/main/scala\",\n        \"scrooge/scrooge-generator/src/main/scala\",\n        \"scrooge/scrooge-serializer\",\n        \"src/thrift/com/twitter/context:twitter-context-scala\",\n        \"src/thrift/com/twitter/scrooge_internal/schema:thrift-scala\",\n        \"twitter-context/src/main/scala\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/access_policy\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"src/thrift/com/twitter/context:twitter-context-scala\",\n        \"twitter-context/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers/DebugTwitterContext.scala",
    "content": "package com.twitter.product_mixer.core.controllers\n\nimport com.twitter.context.TwitterContext\nimport com.twitter.context.thriftscala.Viewer\nimport com.twitter.product_mixer.TwitterContextPermit\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\n\n/**\n * Mixes in support to forge the UserIds in TwitterContext for debug purposes.\n *\n * A thrift controller can extend DebugTwitterContext and wrap it's execution logic:\n *\n * {{{\n * withDebugTwitterContext(request.clientContext) {\n *   Stitch.run(...)\n * }\n * }}}\n */\ntrait DebugTwitterContext {\n\n  private val ctx = TwitterContext(TwitterContextPermit)\n\n  /**\n   * Wrap some function in a debug TwitterContext with hardcoded userIds\n   * to the ClientContext.userId.\n   *\n   * @param clientContext - A product mixer request client context\n   * @param f The function to wrap\n   */\n  def withDebugTwitterContext[T](clientContext: ClientContext)(f: => T): T = {\n    ctx.let(\n      forgeTwitterContext(\n        clientContext.userId\n          .getOrElse(throw new IllegalArgumentException(\"missing required field: user id\")))\n    )(f)\n  }\n\n  // Generate a fake Twitter Context for debug usage.\n  // Generally the TwitterContext is created by the API service, and Strato uses it for permission control.\n  // When we use our debug endpoint, we instead create our own context so that Strato finds something useful.\n  // We enforce ACLs directly via Thrift Web Forms' permission system.\n  private def forgeTwitterContext(userId: Long): Viewer = {\n    Viewer(\n      auditIp = None,\n      ipTags = Set.empty,\n      userId = Some(userId),\n      guestId = None,\n      clientApplicationId = None,\n      userAgent = None,\n      locationToken = None,\n      authenticatedUserId = Some(userId),\n      guestToken = None\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers/GetComponentRegistryHandler.scala",
    "content": "package com.twitter.product_mixer.core.controllers\n\nimport com.twitter.finagle.http.Request\nimport com.twitter.inject.Injector\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AccessPolicy\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.WithDebugAccessPolicies\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.pipeline.Pipeline\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineConfig\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineConfig\nimport com.twitter.product_mixer.core.pipeline.recommendation.RecommendationPipelineConfig\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorConfig\nimport com.twitter.product_mixer.core.service.component_registry\nimport com.twitter.product_mixer.core.service.component_registry.ComponentRegistry\nimport com.twitter.product_mixer.core.service.component_registry.ComponentRegistrySnapshot\nimport com.twitter.util.Future\n\ncase class GetComponentRegistryHandler(injector: Injector) {\n  lazy val componentRegistry: ComponentRegistry = injector.instance[ComponentRegistry]\n\n  def apply(request: Request): Future[ComponentRegistryResponse] = {\n    componentRegistry.get.map { currentComponentRegistry: ComponentRegistrySnapshot =>\n      val registeredComponents = currentComponentRegistry.getAllRegisteredComponents.map {\n        registeredComponent =>\n          val componentIdentifier = registeredComponent.identifier\n          val childComponents = currentComponentRegistry\n            .getChildComponents(componentIdentifier)\n            .map { childComponent =>\n              ChildComponent(\n                componentType = childComponent.componentType,\n                name = childComponent.name,\n                relativeScopes = componentIdentifier.toScopes ++ childComponent.toScopes,\n                qualityFactorMonitoringConfig =\n                  buildQualityFactoringMonitoringConfig(registeredComponent, childComponent)\n              )\n            }\n\n          RegisteredComponent(\n            componentType = componentIdentifier.componentType,\n            name = componentIdentifier.name,\n            scopes = componentIdentifier.toScopes,\n            children = childComponents,\n            alertConfig = Some(registeredComponent.component.alerts.map(AlertConfig.apply)),\n            sourceFile = Some(registeredComponent.sourceFile),\n            debugAccessPolicies = Some(registeredComponent.component match {\n              case withDebugAccessPolicies: WithDebugAccessPolicies =>\n                withDebugAccessPolicies.debugAccessPolicies\n              case _ => Set.empty\n            })\n          )\n      }\n\n      ComponentRegistryResponse(registeredComponents)\n    }\n  }\n\n  private def buildQualityFactoringMonitoringConfig(\n    parent: component_registry.RegisteredComponent,\n    child: ComponentIdentifier\n  ): Option[QualityFactorMonitoringConfig] = {\n    val qualityFactorConfigs: Option[Map[ComponentIdentifier, QualityFactorConfig]] =\n      parent.component match {\n        case pipeline: Pipeline[_, _] =>\n          pipeline.config match {\n            case config: RecommendationPipelineConfig[_, _, _, _] =>\n              Some(config.qualityFactorConfigs)\n            case config: MixerPipelineConfig[_, _, _] =>\n              Some(\n                config.qualityFactorConfigs\n                  .asInstanceOf[Map[ComponentIdentifier, QualityFactorConfig]])\n            case config: ProductPipelineConfig[_, _, _] =>\n              Some(config.qualityFactorConfigs)\n            case _ => None\n          }\n        case _ => None\n      }\n\n    val qfConfigForChild: Option[QualityFactorConfig] = qualityFactorConfigs.flatMap(_.get(child))\n\n    qfConfigForChild.map { qfConfig =>\n      QualityFactorMonitoringConfig(\n        boundMin = qfConfig.qualityFactorBounds.bounds.minInclusive,\n        boundMax = qfConfig.qualityFactorBounds.bounds.maxInclusive\n      )\n    }\n  }\n}\n\ncase class RegisteredComponent(\n  componentType: String,\n  name: String,\n  scopes: Seq[String],\n  children: Seq[ChildComponent],\n  alertConfig: Option[Seq[AlertConfig]],\n  sourceFile: Option[String],\n  debugAccessPolicies: Option[Set[AccessPolicy]])\n\ncase class ChildComponent(\n  componentType: String,\n  name: String,\n  relativeScopes: Seq[String],\n  qualityFactorMonitoringConfig: Option[QualityFactorMonitoringConfig])\n\n/**\n * The shape of the data returned to callers after hitting the `component-registry` endpoint\n *\n * @note changes to [[ComponentRegistryResponse]] or contained types should be reflected\n *       in dashboard generation code in the `monitoring-configs/product_mixer` directory.\n */\ncase class ComponentRegistryResponse(\n  registeredComponents: Seq[RegisteredComponent])\n\ncase class ProductPipeline(identifier: String)\ncase class ProductPipelinesResponse(productPipelines: Seq[ProductPipeline])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers/GetDebugConfigurationHandler.scala",
    "content": "package com.twitter.product_mixer.core.controllers\n\nimport com.twitter.finagle.http.Request\nimport com.twitter.scrooge.BinaryThriftStructSerializer\nimport com.twitter.scrooge.ThriftMethod\nimport com.twitter.scrooge.schema.ThriftDefinitions\nimport com.twitter.scrooge.schema.scrooge.scala.CompiledScroogeDefBuilder\nimport com.twitter.scrooge.schema.serialization.thrift.ReferenceResolver\nimport com.twitter.scrooge.schema.serialization.thrift.ThriftDefinitionsSerializer\nimport com.twitter.scrooge.schema.{thriftscala => THRIFT}\n\n/**\n * Endpoint to expose a Mixer's expected query configuration, including the request schema.\n *\n * @param debugEndpoint the debug Thrift endpoint. Passing [[None]] disables the query debugging\n *                      feature.\n * @tparam ServiceIface a thrift service containing the [[debugEndpoint]]\n */\ncase class GetDebugConfigurationHandler[ServiceIface](\n  thriftMethod: ThriftMethod\n)(\n  implicit val serviceIFace: Manifest[ServiceIface]) {\n\n  // We need to binary encode the service def because the underlying Thrift isn't sufficiently\n  // annotated to be serialized/deserialized by Jackson\n  private val serviceDef = {\n    val fullServiceDefinition: ThriftDefinitions.ServiceDef = CompiledScroogeDefBuilder\n      .build(serviceIFace).asInstanceOf[ThriftDefinitions.ServiceDef]\n\n    val endpointDefinition: ThriftDefinitions.ServiceEndpointDef =\n      fullServiceDefinition.endpointsByName(thriftMethod.name)\n\n    // Create a service definition which just contains the debug endpoint. At a bare minimum, we need\n    // to give callers a way to identify the debug endpoint. Sending back all the endpoints is\n    // redundant.\n    val serviceDefinition: ThriftDefinitions.ServiceDef =\n      fullServiceDefinition.copy(endpoints = Seq(endpointDefinition))\n\n    val thriftDefinitionsSerializer = {\n      // We don't make use of references but a reference resolver is required by the Scrooge API\n      val noopReferenceResolver: ReferenceResolver =\n        (_: THRIFT.ReferenceDef) => throw new Exception(\"no references\")\n\n      new ThriftDefinitionsSerializer(noopReferenceResolver, enableReferences = false)\n    }\n\n    val thriftBinarySerializer = BinaryThriftStructSerializer.apply(THRIFT.Definition)\n\n    val serializedServiceDef = thriftDefinitionsSerializer(serviceDefinition)\n\n    thriftBinarySerializer.toBytes(serializedServiceDef)\n  }\n\n  def apply(request: Request): DebugConfigurationResponse =\n    DebugConfigurationResponse(thriftMethod.name, serviceDef)\n}\n\ncase class DebugConfigurationResponse(\n  debugEndpointName: String,\n  serviceDefinition: Array[Byte])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers/PredicateConfig.scala",
    "content": "package com.twitter.product_mixer.core.controllers\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.Predicate\n\n/** Simple representation for a [[Predicate]] used for dashboard generation */\nprivate[core] case class PredicateConfig(\n  operator: String,\n  threshold: Double,\n  datapointsPastThreshold: Int,\n  duration: Int,\n  metricGranularity: String)\n\nprivate[core] object PredicateConfig {\n\n  /** Convert this [[Predicate]] into a [[PredicateConfig]] */\n  def apply(predicate: Predicate): PredicateConfig = PredicateConfig(\n    predicate.operator.toString,\n    predicate.threshold,\n    predicate.datapointsPastThreshold,\n    predicate.duration,\n    predicate.metricGranularity.unit)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers/ProductMixerController.scala",
    "content": "package com.twitter.product_mixer.core.controllers\n\nimport com.twitter.finagle.http.Request\nimport com.twitter.finagle.http.Response\nimport com.twitter.finagle.http.Status\nimport com.twitter.finagle.http.RouteIndex\nimport com.twitter.finatra.http.Controller\nimport com.twitter.scrooge.ThriftMethod\nimport com.twitter.inject.Injector\nimport com.twitter.inject.annotations.Flags\nimport com.twitter.product_mixer.core.model.common.identifier.ProductIdentifier\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ServiceLocal\nimport com.twitter.product_mixer.core.service.component_registry.ComponentRegistry\nimport com.twitter.product_mixer.core.service.component_registry.{\n  RegisteredComponent => ComponentRegistryRegisteredComponent\n}\nimport com.twitter.util.Future\nimport java.net.URLEncoder\n\n/**\n * Register endpoints necessary for enabling Product Mixer tooling such as alerts, dashboard\n * generation and Turntable.\n *\n * @param debugEndpoint a debug endpoint to run queries against. This feature is experimental and we\n *                      do not recommend that teams use it yet. Providing [[None]] will disable\n *                      debug queries.\n * @tparam ServiceIface a thrift service containing the [[debugEndpoint]]\n */\ncase class ProductMixerController[ServiceIface](\n  injector: Injector,\n  debugEndpoint: ThriftMethod,\n)(\n  implicit val serviceIFace: Manifest[ServiceIface])\n    extends Controller {\n\n  val isLocal: Boolean = injector.instance[Boolean](Flags.named(ServiceLocal))\n\n  if (!isLocal) {\n    prefix(\"/admin/product-mixer\") {\n      val productNamesFut: Future[Seq[String]] =\n        injector.instance[ComponentRegistry].get.map { componentRegistry =>\n          componentRegistry.getAllRegisteredComponents.collect {\n            case ComponentRegistryRegisteredComponent(identifier: ProductIdentifier, _, _) =>\n              identifier.name\n          }\n        }\n\n      productNamesFut.map { productNames =>\n        productNames.foreach { productName =>\n          get(\n            route = \"/debug-query/\" + productName,\n            admin = true,\n            index = Some(RouteIndex(alias = \"Query \" + productName, group = \"Feeds/Debug Query\"))\n          ) { _: Request =>\n            val auroraPath =\n              URLEncoder.encode(System.getProperty(\"aurora.instanceKey\", \"\"), \"UTF-8\")\n\n            // Extract service name from clientId since there isn't a specific flag for that\n            val serviceName = injector\n              .instance[String](Flags.named(\"thrift.clientId\"))\n              .split(\"\\\\.\")(0)\n\n            val redirectUrl =\n              s\"https://feeds.twitter.biz/dtab/$serviceName/$productName?servicePath=$auroraPath\"\n\n            val response = Response().status(Status.Found)\n            response.location = redirectUrl\n            response\n          }\n        }\n      }\n    }\n  }\n\n  prefix(\"/product-mixer\") {\n    get(route = \"/component-registry\")(GetComponentRegistryHandler(injector).apply)\n    get(route = \"/debug-configuration\")(GetDebugConfigurationHandler(debugEndpoint).apply)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/controllers/QualityFactorMonitoringConfig.scala",
    "content": "package com.twitter.product_mixer.core.controllers\n\n// Bounds here are inclusive\ncase class QualityFactorMonitoringConfig(\n  boundMin: Double,\n  boundMax: Double)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"util/util-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/Feature.scala",
    "content": "package com.twitter.product_mixer.core.feature\n\n/**\n * A [[Feature]] is a single measurable or computable property of an entity.\n *\n * @note If a [[Feature]] is optional then the [[Value]] should be `Option[Value]`\n *\n * @note If a [[Feature]] is populated with a [[com.twitter.product_mixer.core.functional_component.feature_hydrator.FeatureHydrator]]\n *       and the hydration fails, a failure will be stored for the [[Feature]].\n *       If that [[Feature]] is accessed with [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap.get]]\n *       then the stored exception will be thrown, essentially failing-closed.\n *       You can use [[FeatureWithDefaultOnFailure]] or [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap.getOrElse]]\n *       instead to avoid these issues and instead fail-open.\n *       If correctly hydrating a Feature's value is essential to the request being correct,\n *       then you should fail-closed on failure to hydrate it by extending [[Feature]] directly.\n *\n *       This does not apply to [[Feature]]s from [[com.twitter.product_mixer.core.functional_component.transformer.FeatureTransformer]]\n *       which throw in the calling Pipeline instead of storing their failures.\n *\n * @tparam Entity The type of entity that this feature works with. This could be a User, Tweet,\n *                Query, etc.\n * @tparam Value The type of the value of this feature.\n */\ntrait Feature[-Entity, Value] { self =>\n  override def toString: String = {\n    Feature.getSimpleName(self.getClass)\n  }\n}\n\n/**\n * With a [[Feature]], if the [[com.twitter.product_mixer.core.functional_component.feature_hydrator.FeatureHydrator]] fails,\n * the failure will be caught by the platform and stored in the [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]].\n * Accessing a failed feature via [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap.get()]]\n * will throw the exception that was caught while attempting to hydrate the feature. If there's a\n * reasonable default for a [[Feature]] to fail-open with, then throwing the exception at read time\n * can be prevented by defining a `defaultValue` via [[FeatureWithDefaultOnFailure]]. When accessing\n * a failed feature via [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap.get()]]\n * for a [[FeatureWithDefaultOnFailure]], the `defaultValue` will be returned.\n *\n *\n * @note [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap.getOrElse()]] can also be used\n *       to access a failed feature without throwing the exception, by defining the default via the\n *       [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap.getOrElse()]] method call\n *       instead of as part of the feature declaration.\n * @note This does not apply to [[FeatureWithDefaultOnFailure]]s from [[com.twitter.product_mixer.core.functional_component.transformer.FeatureTransformer]]\n *       which throw in the calling Pipeline instead of storing their failures.\n *\n * @tparam Entity The type of entity that this feature works with. This could be a User, Tweet,\n *                Query, etc.\n * @tparam Value The type of the value of this feature.\n */\ntrait FeatureWithDefaultOnFailure[Entity, Value] extends Feature[Entity, Value] {\n\n  /** The default value a feature should return should it fail to be hydrated */\n  def defaultValue: Value\n}\n\ntrait ModelFeatureName { self: Feature[_, _] =>\n  def featureName: String\n}\n\nobject Feature {\n\n  /**\n   * Avoid `malformed class name` exceptions due to the presence of the `$` character\n   * Also strip off trailing $ signs for readability\n   */\n  def getSimpleName[T](c: Class[T]): String = {\n    c.getName.stripSuffix(\"$\").lastIndexOf(\"$\") match {\n      case -1 => c.getSimpleName.stripSuffix(\"$\")\n      case index => c.getName.substring(index + 1).stripSuffix(\"$\")\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api/util:datarecord\",\n        \"src/thrift/com/twitter/dal/personal_data:personal_data-java\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/ml/api:data-scala\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"src/thrift/com/twitter/dal/personal_data:personal_data-java\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/ml/api:data-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord/DataRecordCompatible.scala",
    "content": "package com.twitter.product_mixer.core.feature.datarecord\n\nimport com.twitter.dal.personal_data.thriftjava.PersonalDataType\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.DataType\nimport com.twitter.ml.api.thriftscala.GeneralTensor\nimport com.twitter.ml.api.thriftscala.StringTensor\nimport com.twitter.ml.api.util.ScalaToJavaDataRecordConversions\nimport com.twitter.ml.api.{GeneralTensor => JGeneralTensor}\nimport com.twitter.ml.api.{RawTypedTensor => JRawTypedTensor}\nimport com.twitter.ml.api.{Feature => MlFeature}\nimport java.nio.ByteBuffer\nimport java.nio.ByteOrder\nimport java.util.{Map => JMap}\nimport java.util.{Set => JSet}\nimport java.lang.{Long => JLong}\nimport java.lang.{Boolean => JBoolean}\nimport java.lang.{Double => JDouble}\nimport scala.collection.JavaConverters._\n\n/**\n * Defines a conversion function for customers to mix-in when constructing a DataRecord supported\n * feature. We do this because the ML Feature representation is written in Java and uses Java types.\n * Furthermore, allowing customers to construct their own ML Feature directly can leave room\n * for mistyping errors, such as using a Double ML Feature on a String Product Mixer feature.\n * This mix in enforces that the customer only uses the right types, while making it easier\n * to setup a DataRecord Feature with nothing but a feature name and personal data types.\n * @tparam FeatureValueType The type of the underlying Product Mixer feature value.\n */\nsealed trait DataRecordCompatible[FeatureValueType] {\n  // The feature value type in ProMix.\n  final type FeatureType = FeatureValueType\n  // The underlying DataRecord value type, sometimes this differs from the Feature Store and ProMix type.\n  type DataRecordType\n\n  def featureName: String\n  def personalDataTypes: Set[PersonalDataType]\n\n  private[product_mixer] def mlFeature: MlFeature[DataRecordType]\n\n  /**\n   * To & from Data Record value converters. In most cases, this is one to one when the types match\n   * but in some cases, certain features are modeled as different types in Data Record. For example,\n   * some features that are Long (e.g, such as TweepCred) are sometimes stored as Doubles.\n   */\n  private[product_mixer] def toDataRecordFeatureValue(featureValue: FeatureType): DataRecordType\n  private[product_mixer] def fromDataRecordFeatureValue(featureValue: DataRecordType): FeatureType\n\n}\n\n/**\n * Converter for going from String feature value to String ML Feature.\n */\ntrait StringDataRecordCompatible extends DataRecordCompatible[String] {\n  override type DataRecordType = String\n\n  final override lazy val mlFeature: MlFeature[String] =\n    new MlFeature.Text(featureName, personalDataTypes.asJava)\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: String\n  ): String = featureValue\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: String\n  ): String = featureValue\n}\n\n/**\n * Converter for going from Long feature value to Discrete/Long ML Feature.\n */\ntrait LongDiscreteDataRecordCompatible extends DataRecordCompatible[Long] {\n  override type DataRecordType = JLong\n\n  final override lazy val mlFeature: MlFeature[JLong] =\n    new Feature.Discrete(featureName, personalDataTypes.asJava)\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: JLong\n  ): Long = featureValue\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: Long\n  ): JLong = featureValue\n}\n\n/**\n * Converter for going from Long feature value to Continuous/Double ML Feature.\n */\ntrait LongContinuousDataRecordCompatible extends DataRecordCompatible[Long] {\n  override type DataRecordType = JDouble\n\n  final override lazy val mlFeature: MlFeature[JDouble] =\n    new Feature.Continuous(featureName, personalDataTypes.asJava)\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: FeatureType\n  ): JDouble = featureValue.toDouble\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: JDouble\n  ): Long = featureValue.longValue()\n}\n\n/**\n * Converter for going from an Integer feature value to Long/Discrete ML Feature.\n */\ntrait IntDiscreteDataRecordCompatible extends DataRecordCompatible[Int] {\n  override type DataRecordType = JLong\n\n  final override lazy val mlFeature: MlFeature[JLong] =\n    new MlFeature.Discrete(featureName, personalDataTypes.asJava)\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: JLong\n  ): Int = featureValue.toInt\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: Int\n  ): JLong = featureValue.toLong\n}\n\n/**\n * Converter for going from Integer feature value to Continuous/Double ML Feature.\n */\ntrait IntContinuousDataRecordCompatible extends DataRecordCompatible[Int] {\n  override type DataRecordType = JDouble\n\n  final override lazy val mlFeature: MlFeature[JDouble] =\n    new Feature.Continuous(featureName, personalDataTypes.asJava)\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: Int\n  ): JDouble = featureValue.toDouble\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: JDouble\n  ): Int = featureValue.toInt\n}\n\n/**\n * Converter for going from Double feature value to Continuous/Double ML Feature.\n */\ntrait DoubleDataRecordCompatible extends DataRecordCompatible[Double] {\n  override type DataRecordType = JDouble\n\n  final override lazy val mlFeature: MlFeature[JDouble] =\n    new MlFeature.Continuous(featureName, personalDataTypes.asJava)\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: JDouble\n  ): Double = featureValue\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: Double\n  ): JDouble = featureValue\n}\n\n/**\n * Converter for going from Boolean feature value to Boolean ML Feature.\n */\ntrait BoolDataRecordCompatible extends DataRecordCompatible[Boolean] {\n  override type DataRecordType = JBoolean\n\n  final override lazy val mlFeature: MlFeature[JBoolean] =\n    new MlFeature.Binary(featureName, personalDataTypes.asJava)\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: JBoolean\n  ): Boolean = featureValue\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: Boolean\n  ): JBoolean = featureValue\n}\n\n/**\n * Converter for going from a ByteBuffer feature value to ByteBuffer ML Feature.\n */\ntrait BlobDataRecordCompatible extends DataRecordCompatible[ByteBuffer] {\n  override type DataRecordType = ByteBuffer\n\n  final override lazy val mlFeature: MlFeature[ByteBuffer] =\n    new Feature.Blob(featureName, personalDataTypes.asJava)\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: ByteBuffer\n  ): ByteBuffer = featureValue\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: ByteBuffer\n  ): ByteBuffer = featureValue\n}\n\n/**\n * Converter for going from a Map[String, Double] feature value to Sparse Double/Continious ML Feature.\n */\ntrait SparseContinuousDataRecordCompatible extends DataRecordCompatible[Map[String, Double]] {\n  override type DataRecordType = JMap[String, JDouble]\n\n  final override lazy val mlFeature: MlFeature[JMap[String, JDouble]] =\n    new Feature.SparseContinuous(featureName, personalDataTypes.asJava)\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: Map[String, Double]\n  ): JMap[String, JDouble] =\n    featureValue.mapValues(_.asInstanceOf[JDouble]).asJava\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: JMap[String, JDouble]\n  ) = featureValue.asScala.toMap.mapValues(_.doubleValue())\n}\n\n/**\n * Converter for going from a Set[String] feature value to SparseBinary/String Set ML Feature.\n */\ntrait SparseBinaryDataRecordCompatible extends DataRecordCompatible[Set[String]] {\n  override type DataRecordType = JSet[String]\n\n  final override lazy val mlFeature: MlFeature[JSet[String]] =\n    new Feature.SparseBinary(featureName, personalDataTypes.asJava)\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: JSet[String]\n  ) = featureValue.asScala.toSet\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: FeatureType\n  ): JSet[String] = featureValue.asJava\n}\n\n/**\n * Marker trait for any feature value to Tensor ML Feature. Not directly usable.\n */\nsealed trait TensorDataRecordCompatible[FeatureV] extends DataRecordCompatible[FeatureV] {\n  override type DataRecordType = JGeneralTensor\n  override def mlFeature: MlFeature[JGeneralTensor]\n}\n\n/**\n * Converter for a double to a Tensor feature encoded as float encoded RawTypedTensor\n */\ntrait RawTensorFloatDoubleDataRecordCompatible extends TensorDataRecordCompatible[Double] {\n  final override lazy val mlFeature: MlFeature[JGeneralTensor] =\n    new Feature.Tensor(\n      featureName,\n      DataType.FLOAT,\n      List.empty[JLong].asJava,\n      personalDataTypes.asJava)\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: FeatureType\n  ) = {\n    val byteBuffer: ByteBuffer =\n      ByteBuffer\n        .allocate(4).order(ByteOrder.LITTLE_ENDIAN).putFloat(featureValue.toFloat)\n    byteBuffer.flip()\n    val tensor = new JGeneralTensor()\n    tensor.setRawTypedTensor(new JRawTypedTensor(DataType.FLOAT, byteBuffer))\n    tensor\n  }\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: JGeneralTensor\n  ) = {\n    val tensor = Option(featureValue.getRawTypedTensor)\n      .getOrElse(throw new UnexpectedTensorException(featureValue))\n    tensor.content.order(ByteOrder.LITTLE_ENDIAN).getFloat().toDouble\n  }\n}\n\n/**\n *  Converter for a scala general tensor to java general tensor ML feature.\n */\ntrait GeneralTensorDataRecordCompatible extends TensorDataRecordCompatible[GeneralTensor] {\n\n  def dataType: DataType\n  final override lazy val mlFeature: MlFeature[JGeneralTensor] =\n    new Feature.Tensor(featureName, dataType, List.empty[JLong].asJava, personalDataTypes.asJava)\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: FeatureType\n  ) = ScalaToJavaDataRecordConversions.scalaTensor2Java(featureValue)\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: JGeneralTensor\n  ) = ScalaToJavaDataRecordConversions.javaTensor2Scala(featureValue)\n}\n\n/**\n *  Converter for a scala string tensor to java general tensor ML feature.\n */\ntrait StringTensorDataRecordCompatible extends TensorDataRecordCompatible[StringTensor] {\n  final override lazy val mlFeature: MlFeature[JGeneralTensor] =\n    new Feature.Tensor(\n      featureName,\n      DataType.STRING,\n      List.empty[JLong].asJava,\n      personalDataTypes.asJava)\n\n  override private[product_mixer] def fromDataRecordFeatureValue(\n    featureValue: JGeneralTensor\n  ) = {\n    ScalaToJavaDataRecordConversions.javaTensor2Scala(featureValue) match {\n      case GeneralTensor.StringTensor(stringTensor) => stringTensor\n      case _ => throw new UnexpectedTensorException(featureValue)\n    }\n  }\n\n  override private[product_mixer] def toDataRecordFeatureValue(\n    featureValue: FeatureType\n  ) = ScalaToJavaDataRecordConversions.scalaTensor2Java(GeneralTensor.StringTensor(featureValue))\n}\n\nclass UnexpectedTensorException(tensor: JGeneralTensor)\n    extends Exception(s\"Unexpected Tensor: $tensor\")\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord/DataRecordFeature.scala",
    "content": "package com.twitter.product_mixer.core.feature.datarecord\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.product_mixer.core.feature.Feature\n\n/**\n * A DataRecord supported feature mixin for enabling conversions from Product Mixer Features\n * to DataRecords. When using Feature Store features, this is pre-configured for the customer\n * under the hood. For non-Feature Store features, customers must mix in either [[DataRecordFeature]]\n * for required features, or [[DataRecordOptionalFeature]] for optional features, as well as mixing\n * in a corresponding [[DataRecordCompatible]] for their feature type.\n * @tparam Entity The type of entity that this feature works with. This could be a User, Tweet,\n *                Query, etc.\n * @tparam Value The type of the value of this feature.\n */\nsealed trait BaseDataRecordFeature[-Entity, Value] extends Feature[Entity, Value]\n\nprivate[product_mixer] abstract class FeatureStoreDataRecordFeature[-Entity, Value]\n    extends BaseDataRecordFeature[Entity, Value]\n\n/**\n * Feature in a DataRecord for a required feature value; the corresponding feature will always be\n * available in the built DataRecord.\n */\ntrait DataRecordFeature[-Entity, Value] extends BaseDataRecordFeature[Entity, Value] {\n  self: DataRecordCompatible[Value] =>\n}\n\n/**\n * Feature in a DataRecord for an optional feature value; the corresponding feature will only\n * ever be set in a DataRecord if the value in the feature map is defined (Some(V)).\n */\ntrait DataRecordOptionalFeature[-Entity, Value]\n    extends BaseDataRecordFeature[Entity, Option[Value]] {\n  self: DataRecordCompatible[Value] =>\n}\n\n/**\n * An entire DataRecord as a feature. This is useful when there is an existing DataRecord that\n * should be used as a whole instead of as individual [[DataRecordFeature]]s for example.\n */\ntrait DataRecordInAFeature[-Entity] extends BaseDataRecordFeature[Entity, DataRecord]\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1/featurevalue\",\n        \"util/util-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"util/util-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/FeatureMap.scala",
    "content": "package com.twitter.product_mixer.core.feature.featuremap\n\nimport com.fasterxml.jackson.databind.annotation.JsonSerialize\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure\nimport com.twitter.product_mixer.core.feature.featurestorev1.featurevalue.FeatureStoreV1Response\nimport com.twitter.product_mixer.core.feature.featurestorev1.featurevalue.{\n  FeatureStoreV1ResponseFeature => FSv1Feature\n}\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\n\n/**\n * A set of features and their values. Associated with a specific instance of an Entity, though\n * that association is maintained by the framework.\n *\n * [[FeatureMapBuilder]] is used to build new FeatureMap instances\n */\n@JsonSerialize(using = classOf[FeatureMapSerializer])\ncase class FeatureMap private[feature] (\n  private[core] val underlyingMap: Map[Feature[_, _], Try[_]]) {\n\n  /**\n   * Returns the [[Value]] associated with the Feature\n   *\n   * If the Feature is missing from the feature map, it throws a [[MissingFeatureException]].\n   * If the Feature failed and isn't a [[FeatureWithDefaultOnFailure]] this will throw the underlying exception\n   * that the feature failed with during hydration.\n   */\n  def get[Value](feature: Feature[_, Value]): Value =\n    getOrElse(feature, throw MissingFeatureException(feature), None)\n\n  /**\n   * Returns the [[Value]] associated with the Feature with the same semantics as\n   * [[FeatureMap.get()]], but the underlying [[Try]] is returned to allow for checking the success\n   * or error state of a feature hydration. This is helpful for implementing fall-back behavior in\n   * case the feature is missing or hydration failed without a [[FeatureWithDefaultOnFailure]] set.\n   *\n   * @note The [[FeatureMap.getOrElse()]] logic is duplicated here to avoid unpacking and repacking\n   *       the [[Try]] that is already available in the [[underlyingMap]]\n   */\n  def getTry[Value](feature: Feature[_, Value]): Try[Value] =\n    underlyingMap.get(feature) match {\n      case None => Throw(MissingFeatureException(feature))\n      case Some(value @ Return(_)) => value.asInstanceOf[Return[Value]]\n      case Some(value @ Throw(_)) =>\n        feature match {\n          case f: FeatureWithDefaultOnFailure[_, Value] @unchecked => Return(f.defaultValue)\n          case _ => value.asInstanceOf[Throw[Value]]\n        }\n    }\n\n  /**\n   * Returns the [[Value]] associated with the feature or a default if the key is not contained in the map\n   * `default` can also be used to throw an exception.\n   *\n   *  e.g. `.getOrElse(feature, throw new MyCustomException())`\n   *\n   * @note for [[FeatureWithDefaultOnFailure]], the [[FeatureWithDefaultOnFailure.defaultValue]]\n   *       will be returned if the [[Feature]] failed, but if it is missing/never hydrated,\n   *       then the `default` provided here will be used.\n   */\n  def getOrElse[Value](feature: Feature[_, Value], default: => Value): Value = {\n    getOrElse(feature, default, Some(default))\n  }\n\n  /**\n   * Private helper for getting features from the feature map, allowing us to define a default\n   * when the feature is missing from the feature map, vs when its in the feature map but failed.\n   * In the case of failed features, if the feature is a [FeatureWithDefaultOnFailure], it will\n   * prioritize that default.\n   * @param feature The feature to retrieve\n   * @param missingDefault The default value to use when the feature is missing from the map.\n   * @param failureDefault The default value to use when the feature is present but failed.\n   * @tparam Value The value type of the feature.\n   * @return The value stored in the map.\n   */\n  private def getOrElse[Value](\n    feature: Feature[_, Value],\n    missingDefault: => Value,\n    failureDefault: => Option[Value]\n  ): Value =\n    underlyingMap.get(feature) match {\n      case None => missingDefault\n      case Some(Return(value)) => value.asInstanceOf[Value]\n      case Some(Throw(err)) =>\n        feature match {\n          case f: FeatureWithDefaultOnFailure[_, Value] @unchecked => f.defaultValue\n          case _ => failureDefault.getOrElse(throw err)\n        }\n    }\n\n  /**\n   * returns a new FeatureMap with\n   * - the new Feature and Value pair if the Feature was not present\n   * - overriding the previous Value if that Feature was previously present\n   */\n  def +[V](key: Feature[_, V], value: V): FeatureMap =\n    new FeatureMap(underlyingMap + (key -> Return(value)))\n\n  /**\n   * returns a new FeatureMap with all the elements of current FeatureMap and `other`.\n   *\n   * @note if a [[Feature]] exists in both maps, the Value from `other` takes precedence\n   */\n  def ++(other: FeatureMap): FeatureMap = {\n    if (other.isEmpty) {\n      this\n    } else if (isEmpty) {\n      other\n    } else if (this.getFeatures.contains(FSv1Feature) && other.getFeatures.contains(FSv1Feature)) {\n      val mergedResponse =\n        FeatureStoreV1Response.merge(this.get(FSv1Feature), other.get(FSv1Feature))\n      val mergedResponseFeatureMap = FeatureMapBuilder().add(FSv1Feature, mergedResponse).build()\n      new FeatureMap(underlyingMap ++ other.underlyingMap ++ mergedResponseFeatureMap.underlyingMap)\n    } else {\n      new FeatureMap(underlyingMap ++ other.underlyingMap)\n    }\n  }\n\n  /** returns the keySet of Features in the map */\n  def getFeatures: Set[Feature[_, _]] = underlyingMap.keySet\n\n  /**\n   * The Set of Features in the FeatureMap that have a successfully returned value. Failed features\n   * will obviously not be here.\n   */\n  def getSuccessfulFeatures: Set[Feature[_, _]] = underlyingMap.collect {\n    case (feature, Return(_)) => feature\n  }.toSet\n\n  def isEmpty: Boolean = underlyingMap.isEmpty\n\n  override def toString: String = s\"FeatureMap(${underlyingMap.toString})\"\n}\n\nobject FeatureMap {\n  // Restrict access to the apply method.\n  // This shouldn't be required after scala 2.13.2 (https://github.com/scala/scala/pull/7702)\n  private[feature] def apply(underlyingMap: Map[Feature[_, _], Try[_]]): FeatureMap =\n    FeatureMap(underlyingMap)\n\n  /** Merges an arbitrary number of [[FeatureMap]]s from left-to-right */\n  def merge(featureMaps: TraversableOnce[FeatureMap]): FeatureMap = {\n    val builder = FeatureMapBuilder()\n\n    /**\n     * merge the current [[FSv1Feature]] with the existing accumulated one\n     * and add the rest of the [[FeatureMap]]'s [[Feature]]s to the `builder`\n     */\n    def mergeInternal(\n      featureMap: FeatureMap,\n      accumulatedFsV1Response: Option[FeatureStoreV1Response]\n    ): Option[FeatureStoreV1Response] = {\n      if (featureMap.isEmpty) {\n        accumulatedFsV1Response\n      } else {\n\n        val currentFsV1Response =\n          if (featureMap.getFeatures.contains(FSv1Feature))\n            Some(featureMap.get(FSv1Feature))\n          else\n            None\n\n        val mergedFsV1Response = (accumulatedFsV1Response, currentFsV1Response) match {\n          case (Some(merged), Some(current)) =>\n            // both present so merge them\n            Some(FeatureStoreV1Response.merge(merged, current))\n          case (merged, current) =>\n            // one or both are missing so use whichever is available\n            merged.orElse(current)\n        }\n\n        featureMap.underlyingMap.foreach {\n          case (FSv1Feature, _) => // FSv1Feature is only added once at the very end\n          case (feature, value) => builder.addWithoutValidation(feature, value)\n        }\n        mergedFsV1Response\n      }\n    }\n\n    featureMaps\n      .foldLeft[Option[FeatureStoreV1Response]](None) {\n        case (fsV1Response, featureMap) => mergeInternal(featureMap, fsV1Response)\n      }.foreach(\n        // add merged `FSv1Feature` to the `builder` at the end\n        builder.add(FSv1Feature, _)\n      )\n\n    builder.build()\n  }\n\n  val empty = new FeatureMap(Map.empty)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/FeatureMapBuilder.scala",
    "content": "package com.twitter.product_mixer.core.feature.featuremap\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\nimport scala.collection.mutable\n\n/**\n * [[FeatureMapBuilder]] is a typesafe way (it checks types vs the [[Feature]]s on `.add`) to build a [[FeatureMap]].\n *\n * Throws a [[DuplicateFeatureException]] if you try to add the same [[Feature]] more than once.\n *\n * These builders are __not__ reusable.\n */\n\nclass FeatureMapBuilder {\n  private val underlying = Map.newBuilder[Feature[_, _], Try[Any]]\n  private val keys = mutable.HashSet.empty[Feature[_, _]]\n  private var built = false\n\n  /**\n   * Add a [[Try]] of a [[Feature]] `value` to the map,\n   * handling both the [[Return]] and [[Throw]] cases.\n   *\n   * Throws a [[DuplicateFeatureException]] if it's already present.\n   *\n   * @note If you have a [[Feature]] with a non-optional value type `Feature[_, V]`\n   *       but have an `Option[V]` you can use [[Try.orThrow]] to convert the [[Option]]\n   *       to a [[Try]], which will store the successful or failed [[Feature]] in the map.\n   */\n  def add[V](feature: Feature[_, V], value: Try[V]): FeatureMapBuilder = addTry(feature, value)\n\n  /**\n   * Add a successful [[Feature]] `value` to the map\n   *\n   * Throws a [[DuplicateFeatureException]] if it's already present.\n   *\n   * @note If you have a [[Feature]] with a non-optional value type `Feature[_, V]`\n   *       but have an `Option[V]` you can use [[Option.get]] or [[Option.getOrElse]]\n   *       to convert the [[Option]] to extract the underlying value,\n   *       which will throw immediately if it's [[None]] or add the successful [[Feature]] in the map.\n   */\n  def add[V](feature: Feature[_, V], value: V): FeatureMapBuilder =\n    addTry(feature, Return(value))\n\n  /**\n   * Add a failed [[Feature]] `value` to the map\n   *\n   * Throws a [[DuplicateFeatureException]] if it's already present.\n   */\n  def addFailure(feature: Feature[_, _], throwable: Throwable): FeatureMapBuilder =\n    addTry(feature, Throw(throwable))\n\n  /**\n   * [[add]] but for when the [[Feature]] types aren't known\n   *\n   * Add a [[Try]] of a [[Feature]] `value` to the map,\n   * handling both the [[Return]] and [[Throw]] cases.\n   *\n   * Throws a [[DuplicateFeatureException]] if it's already present.\n   *\n   * @note If you have a [[Feature]] with a non-optional value type `Feature[_, V]`\n   *       but have an `Option[V]` you can use [[Try.orThrow]] to convert the [[Option]]\n   *       to a [[Try]], which will store the successful or failed [[Feature]] in the map.\n   */\n  def addTry(feature: Feature[_, _], value: Try[_]): FeatureMapBuilder = {\n    if (keys.contains(feature)) {\n      throw new DuplicateFeatureException(feature)\n    }\n    addWithoutValidation(feature, value)\n  }\n\n  /**\n   * [[addTry]] but without a [[DuplicateFeatureException]] check\n   *\n   * @note Only for use internally within [[FeatureMap.merge]]\n   */\n  private[featuremap] def addWithoutValidation(\n    feature: Feature[_, _],\n    value: Try[_]\n  ): FeatureMapBuilder = {\n    keys += feature\n    underlying += ((feature, value))\n    this\n  }\n\n  /** Builds the FeatureMap */\n  def build(): FeatureMap = {\n    if (built) {\n      throw ReusedFeatureMapBuilderException\n    }\n\n    built = true\n    new FeatureMap(underlying.result())\n  }\n}\n\nobject FeatureMapBuilder {\n\n  /** Returns a new [[FeatureMapBuilder]] for making [[FeatureMap]]s */\n  def apply(): FeatureMapBuilder = new FeatureMapBuilder\n}\n\nclass DuplicateFeatureException(feature: Feature[_, _])\n    extends UnsupportedOperationException(s\"Feature $feature already exists in FeatureMap\")\n\nobject ReusedFeatureMapBuilderException\n    extends UnsupportedOperationException(\n      \"build() cannot be called more than once since FeatureMapBuilders are not reusable\")\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/FeatureMapException.scala",
    "content": "package com.twitter.product_mixer.core.feature.featuremap\n\nimport com.twitter.product_mixer.core.feature.Feature\n\ncase class MissingFeatureException(feature: Feature[_, _])\n    extends Exception(\"Missing value for \" + feature) {\n  override def toString: String = getMessage\n}\n\nclass InvalidPredictionRecordMergeException\n    extends Exception(\n      \"Use FeatureMap.plusPlusOptimized instead of FeatureMap.++ when the FeatureMaps on both sides of the merge contain PredictionRecords\")\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/FeatureMapSerializer.scala",
    "content": "package com.twitter.product_mixer.core.feature.featuremap\n\nimport com.fasterxml.jackson.core.JsonGenerator\nimport com.fasterxml.jackson.databind.JsonSerializer\nimport com.fasterxml.jackson.databind.SerializerProvider\nimport com.twitter.product_mixer.core.feature.featurestorev1.featurevalue.FeatureStoreV1Response\nimport com.twitter.product_mixer.core.feature.featurestorev1.featurevalue.FeatureStoreV1ResponseFeature\nimport com.twitter.util.Return\n\n/**\n * Rendering feature maps is dangerous because we don't control all the data that's stored in them.\n * This can result failed requests, as we might try to render a recursive structure, very large\n * structure, etc. Create a simple map using toString, this mostly works and is better than failing\n * the request.\n *\n * @note changes to serialization logic can have serious performance implications given how hot the\n *       serialization path is. Consider benchmarking changes with [[com.twitter.product_mixer.core.benchmark.CandidatePipelineResultSerializationBenchmark]]\n */\nprivate[featuremap] class FeatureMapSerializer() extends JsonSerializer[FeatureMap] {\n  override def serialize(\n    featureMap: FeatureMap,\n    gen: JsonGenerator,\n    serializers: SerializerProvider\n  ): Unit = {\n    gen.writeStartObject()\n\n    featureMap.underlyingMap.foreach {\n      case (FeatureStoreV1ResponseFeature, Return(value)) =>\n        // We know that value has to be [[FeatureStoreV1Response]] but its type has been erased,\n        // preventing us from pattern-matching.\n        val featureStoreResponse = value.asInstanceOf[FeatureStoreV1Response]\n\n        val featuresIterator = featureStoreResponse.richDataRecord.allFeaturesIterator()\n        while (featuresIterator.moveNext()) {\n          gen.writeStringField(\n            featuresIterator.getFeature.getFeatureName,\n            s\"${featuresIterator.getFeatureType.name}(${truncateString(\n              featuresIterator.getFeatureValue.toString)})\")\n        }\n\n        featureStoreResponse.failedFeatures.foreach {\n          case (failedFeature, failureReasons) =>\n            gen.writeStringField(\n              failedFeature.toString,\n              s\"Failed(${truncateString(failureReasons.toString)})\")\n        }\n      case (name, Return(value)) =>\n        gen.writeStringField(name.toString, truncateString(value.toString))\n      case (name, error) =>\n        // Note: we don't match on Throw(error) because we want to keep it for the toString\n        gen.writeStringField(name.toString, truncateString(error.toString))\n\n    }\n\n    gen.writeEndObject()\n  }\n\n  // Some features can be very large when stringified, for example when a dependant candidate\n  // pipeline is used, the entire previous candidate pipeline result is serialized into a feature.\n  // This causes significant performance issues when the result is later sent over the wire.\n  private def truncateString(input: String): String =\n    if (input.length > 1000) input.take(1000) + \"...\" else input\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/asyncfeaturemap/AsyncFeatureMap.scala",
    "content": "package com.twitter.product_mixer.core.feature.featuremap.asyncfeaturemap\n\nimport com.fasterxml.jackson.databind.annotation.JsonSerialize\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.stitch.Stitch\n\nimport scala.collection.immutable.Queue\n\n/**\n * An internal representation of an async [[FeatureMap]] containing [[Stitch]]s of [[FeatureMap]]s\n * which are already running in the background.\n *\n * Async features are added by providing the [[PipelineStepIdentifier]] of the [[com.twitter.product_mixer.core.pipeline.PipelineBuilder.Step Step]]\n * before which the async [[Feature]]s are needed, and a [[Stitch]] of the async [[FeatureMap]].\n * It's expected that the [[Stitch]] has already been started and is running in the background.\n *\n * While not essential to it's core behavior, [[AsyncFeatureMap]] also keeps track of the [[FeatureHydratorIdentifier]]\n * and the Set of [[Feature]]s which will be hydrated for each [[Stitch]] of a [[FeatureMap]] it's given.\n *\n * @param asyncFeatureMaps the [[FeatureMap]]s for [[PipelineStepIdentifier]]s which have not been reached yet\n *\n * @note [[PipelineStepIdentifier]]s must only refer to [[com.twitter.product_mixer.core.pipeline.PipelineBuilder.Step Step]]s\n *       in the current [[com.twitter.product_mixer.core.pipeline.Pipeline Pipeline]].\n *       Only plain [[FeatureMap]]s are passed into underlying [[com.twitter.product_mixer.core.model.common.Component Component]]s and\n *       [[com.twitter.product_mixer.core.pipeline.Pipeline Pipeline]]s so [[AsyncFeatureMap]]s are scoped\n *       for a specific [[com.twitter.product_mixer.core.pipeline.Pipeline Pipeline]] only.\n */\n@JsonSerialize(using = classOf[AsyncFeatureMapSerializer])\nprivate[core] case class AsyncFeatureMap(\n  asyncFeatureMaps: Map[PipelineStepIdentifier, Queue[\n    (FeatureHydratorIdentifier, Set[Feature[_, _]], Stitch[FeatureMap])\n  ]]) {\n\n  def ++(right: AsyncFeatureMap): AsyncFeatureMap = {\n    val map = Map.newBuilder[\n      PipelineStepIdentifier,\n      Queue[(FeatureHydratorIdentifier, Set[Feature[_, _]], Stitch[FeatureMap])]]\n    (asyncFeatureMaps.keysIterator ++ right.asyncFeatureMaps.keysIterator).foreach { key =>\n      val currentThenRightAsyncFeatureMaps =\n        asyncFeatureMaps.getOrElse(key, Queue.empty) ++\n          right.asyncFeatureMaps.getOrElse(key, Queue.empty)\n      map += (key -> currentThenRightAsyncFeatureMaps)\n    }\n    AsyncFeatureMap(map.result())\n  }\n\n  /**\n   * Returns a new [[AsyncFeatureMap]] which now keeps track of the provided `features`\n   * and will make them available when calling [[hydrate]] with `hydrateBefore`.\n   *\n   * @param featureHydratorIdentifier the [[FeatureHydratorIdentifier]] of the [[com.twitter.product_mixer.core.functional_component.feature_hydrator.FeatureHydrator FeatureHydrator]]\n   *                                  which these [[Feature]]s are from\n   * @param hydrateBefore             the [[PipelineStepIdentifier]] before which the [[Feature]]s need to be hydrated\n   * @param featuresToHydrate         a Set of the [[Feature]]s which will be hydrated\n   * @param features                  a [[Stitch]] of the [[FeatureMap]]\n   */\n  def addAsyncFeatures(\n    featureHydratorIdentifier: FeatureHydratorIdentifier,\n    hydrateBefore: PipelineStepIdentifier,\n    featuresToHydrate: Set[Feature[_, _]],\n    features: Stitch[FeatureMap]\n  ): AsyncFeatureMap = {\n    val featureMapList =\n      asyncFeatureMaps.getOrElse(hydrateBefore, Queue.empty) :+\n        ((featureHydratorIdentifier, featuresToHydrate, features))\n    AsyncFeatureMap(asyncFeatureMaps + (hydrateBefore -> featureMapList))\n  }\n\n  /**\n   * The current state of the [[AsyncFeatureMap]] excluding the [[Stitch]]s.\n   */\n  def features: Map[PipelineStepIdentifier, Seq[(FeatureHydratorIdentifier, Set[Feature[_, _]])]] =\n    asyncFeatureMaps.mapValues(_.map {\n      case (featureHydratorIdentifier, features, _) => (featureHydratorIdentifier, features)\n    })\n\n  /**\n   * Returns a [[Some]] containing a [[Stitch]] with a [[FeatureMap]] holding the [[Feature]]s that are\n   * supposed to be hydrated at `identifier` if there are [[Feature]]s to hydrate at `identifier`\n   *\n   * Returns [[None]] if there are no [[Feature]]s to hydrate at the provided `identifier`,\n   * this allows for determining if there is work to do without running a [[Stitch]].\n   *\n   * @note this only hydrates the [[Feature]]s for the specific `identifier`, it does not hydrate\n   *       [[Feature]]s for earlier Steps.\n   * @param identifier the [[PipelineStepIdentifier]] to hydrate [[Feature]]s for\n   */\n  def hydrate(\n    identifier: PipelineStepIdentifier\n  ): Option[Stitch[FeatureMap]] =\n    asyncFeatureMaps.get(identifier) match {\n      case Some(Queue((_, _, featureMap))) =>\n        // if there is only 1 `FeatureMap` we dont need to do a collect so just return that Stitch\n        Some(featureMap)\n      case Some(featureMapList) =>\n        // if there are multiple `FeatureMap`s we need to collect and merge them together\n        Some(\n          Stitch\n            .collect(featureMapList.map { case (_, _, featureMap) => featureMap })\n            .map { featureMapList => FeatureMap.merge(featureMapList) })\n      case None =>\n        // No results for the provided `identifier` so return `None`\n        None\n    }\n}\n\nprivate[core] object AsyncFeatureMap {\n  val empty: AsyncFeatureMap = AsyncFeatureMap(Map.empty)\n\n  /**\n   * Builds the an [[AsyncFeatureMap]] from a Seq of [[Stitch]] of [[FeatureMap]]\n   * tupled with the relevant metadata we use to build the necessary state.\n   *\n   * This is primarily for convenience, since in most cases an [[AsyncFeatureMap]]\n   * will be built from the result of individual [[com.twitter.product_mixer.core.functional_component.feature_hydrator.FeatureHydrator FeatureHydrator]]s\n   * and combining them into the correct internal state.\n   */\n  def fromFeatureMaps(\n    asyncFeatureMaps: Seq[\n      (FeatureHydratorIdentifier, PipelineStepIdentifier, Set[Feature[_, _]], Stitch[FeatureMap])\n    ]\n  ): AsyncFeatureMap =\n    AsyncFeatureMap(\n      asyncFeatureMaps\n        .groupBy { case (_, hydrateBefore, _, _) => hydrateBefore }\n        .mapValues(featureMaps =>\n          Queue(featureMaps.map {\n            case (hydratorIdentifier, _, featuresToHydrate, stitch) =>\n              (hydratorIdentifier, featuresToHydrate, stitch)\n          }: _*)))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/asyncfeaturemap/AsyncFeatureMapSerializer.scala",
    "content": "package com.twitter.product_mixer.core.feature.featuremap.asyncfeaturemap\n\nimport com.fasterxml.jackson.core.JsonGenerator\nimport com.fasterxml.jackson.databind.JsonSerializer\nimport com.fasterxml.jackson.databind.SerializerProvider\n\n/**\n * Since an [[AsyncFeatureMap]] is typically incomplete, and by the time it's serialized, all the [[com.twitter.product_mixer.core.feature.Feature]]s\n * it will typically be completed and part of the Query or Candidate's individual [[com.twitter.product_mixer.core.feature.Feature]]s\n * we instead opt to provide a summary of the Features which would be hydrated using [[AsyncFeatureMap.features]]\n *\n * This indicates which [[com.twitter.product_mixer.core.feature.Feature]]s will be ready at which Steps\n * and which [[com.twitter.product_mixer.core.functional_component.feature_hydrator.FeatureHydrator]]\n * are responsible for those [[com.twitter.product_mixer.core.feature.Feature]]\n *\n * @note changes to serialization logic can have serious performance implications given how hot the\n *       serialization path is. Consider benchmarking changes with [[com.twitter.product_mixer.core.benchmark.AsyncQueryFeatureMapSerializationBenchmark]]\n */\nprivate[asyncfeaturemap] class AsyncFeatureMapSerializer() extends JsonSerializer[AsyncFeatureMap] {\n  override def serialize(\n    asyncFeatureMap: AsyncFeatureMap,\n    gen: JsonGenerator,\n    serializers: SerializerProvider\n  ): Unit = {\n    gen.writeStartObject()\n\n    asyncFeatureMap.features.foreach {\n      case (stepIdentifier, featureHydrators) =>\n        gen.writeObjectFieldStart(stepIdentifier.toString)\n\n        featureHydrators.foreach {\n          case (hydratorIdentifier, featuresFromHydrator) =>\n            gen.writeArrayFieldStart(hydratorIdentifier.toString)\n\n            featuresFromHydrator.foreach(feature => gen.writeString(feature.toString))\n\n            gen.writeEndArray()\n        }\n\n        gen.writeEndObject()\n    }\n\n    gen.writeEndObject()\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/asyncfeaturemap/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"stitch/stitch-core\",\n        \"util/util-jackson/src/main/scala/com/twitter/util/jackson\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1/featurevalue\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/scala/com/twitter/ml/api/util:datarecord\",\n        \"util/util-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"util/util-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord/DataRecordConverter.scala",
    "content": "package com.twitter.product_mixer.core.feature.featuremap.datarecord\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.DataRecordMerger\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.datarecord._\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\n\nobject DataRecordConverter {\n  val merger = new DataRecordMerger\n}\n\n/**\n * Constructs a FeatureMap from a DataRecord, given a predefined set of features from a FeaturesScope.\n *\n * @param featuresScope scope of predefined set of BaseDataRecordFeatures that should be included in the output FeatureMap.\n */\nclass DataRecordConverter[DRFeature <: BaseDataRecordFeature[_, _]](\n  featuresScope: FeaturesScope[DRFeature]) {\n  import DataRecordConverter._\n\n  def toDataRecord(featureMap: FeatureMap): DataRecord = {\n    // Initialize a DataRecord with the Feature Store features in it and then add all the\n    // non-Feature Store features that support DataRecords to DataRecord. We don't\n    // need to add Feature Store features because they're already in the initial DataRecord.\n    // If there are any pre-built DataRecords, we merge those in.\n    val richDataRecord = featuresScope.getFeatureStoreFeaturesDataRecord(featureMap)\n    val features = featuresScope.getNonFeatureStoreDataRecordFeatures(featureMap)\n    features.foreach {\n      case _: FeatureStoreDataRecordFeature[_, _] =>\n      case requiredFeature: DataRecordFeature[_, _] with DataRecordCompatible[_] =>\n        richDataRecord.setFeatureValue(\n          requiredFeature.mlFeature,\n          requiredFeature.toDataRecordFeatureValue(\n            featureMap.get(requiredFeature).asInstanceOf[requiredFeature.FeatureType]))\n      case optionalFeature: DataRecordOptionalFeature[_, _] with DataRecordCompatible[_] =>\n        featureMap\n          .get(\n            optionalFeature.asInstanceOf[Feature[_, Option[optionalFeature.FeatureType]]]).foreach {\n            value =>\n              richDataRecord\n                .setFeatureValue(\n                  optionalFeature.mlFeature,\n                  optionalFeature.toDataRecordFeatureValue(value))\n          }\n      case dataRecordInAFeature: DataRecordInAFeature[_] =>\n        merger.merge(richDataRecord.getRecord, featureMap.get(dataRecordInAFeature))\n    }\n    richDataRecord.getRecord\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord/DataRecordExtractor.scala",
    "content": "package com.twitter.product_mixer.core.feature.featuremap.datarecord\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.util.SRichDataRecord\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.datarecord._\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport scala.collection.JavaConverters._\n\n/**\n * Constructs a DataRecord from a FeatureMap, given a predefined set of features.\n *\n * @param features predefined set of BaseDataRecordFeatures that should be included in the output DataRecord.\n */\nclass DataRecordExtractor[DRFeature <: BaseDataRecordFeature[_, _]](\n  features: Set[DRFeature]) {\n\n  private val featureContext = new FeatureContext(features.collect {\n    case dataRecordCompatible: DataRecordCompatible[_] => dataRecordCompatible.mlFeature\n  }.asJava)\n\n  def fromDataRecord(dataRecord: DataRecord): FeatureMap = {\n    val featureMapBuilder = FeatureMapBuilder()\n    val richDataRecord = SRichDataRecord(dataRecord, featureContext)\n    features.foreach {\n      // FeatureStoreDataRecordFeature is currently not supported\n      case _: FeatureStoreDataRecordFeature[_, _] =>\n        throw new UnsupportedOperationException(\n          \"FeatureStoreDataRecordFeature cannot be extracted from a DataRecord\")\n      case feature: DataRecordFeature[_, _] with DataRecordCompatible[_] =>\n        // Java API will return null, so use Option to convert it to Scala Option which is None when null.\n        richDataRecord.getFeatureValueOpt(feature.mlFeature)(\n          feature.fromDataRecordFeatureValue) match {\n          case Some(value) =>\n            featureMapBuilder.add(feature.asInstanceOf[Feature[_, feature.FeatureType]], value)\n          case None =>\n            featureMapBuilder.addFailure(\n              feature,\n              PipelineFailure(\n                IllegalStateFailure,\n                s\"Required DataRecord feature is missing: ${feature.mlFeature.getFeatureName}\")\n            )\n        }\n      case feature: DataRecordOptionalFeature[_, _] with DataRecordCompatible[_] =>\n        val featureValue =\n          richDataRecord.getFeatureValueOpt(feature.mlFeature)(feature.fromDataRecordFeatureValue)\n        featureMapBuilder\n          .add(feature.asInstanceOf[Feature[_, Option[feature.FeatureType]]], featureValue)\n      // DataRecordInAFeature is currently not supported\n      case _: DataRecordInAFeature[_] =>\n        throw new UnsupportedOperationException(\n          \"DataRecordInAFeature cannot be extracted from a DataRecord\")\n    }\n    featureMapBuilder.build()\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/datarecord/FeaturesScope.scala",
    "content": "package com.twitter.product_mixer.core.feature.featuremap.datarecord\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.FeatureContext\nimport com.twitter.ml.api.util.SRichDataRecord\nimport scala.collection.JavaConverters._\nimport com.twitter.product_mixer.core.feature.datarecord.BaseDataRecordFeature\nimport com.twitter.product_mixer.core.feature.datarecord.DataRecordCompatible\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featurestorev1.FeatureStoreV1Feature\nimport com.twitter.product_mixer.core.feature.featurestorev1.featurevalue.FeatureStoreV1ResponseFeature\n\n/**\n * FeaturesScope for defining what features should be included in a DataRecord from a FeatureMap.\n * Where possible, prefer [[SpecificFeatures]]. It fails loudly on missing features which can help\n * identify programmer error, but can be complex to manage for multi-phase hydrators.\n */\nsealed trait FeaturesScope[+DRFeature <: BaseDataRecordFeature[_, _]] {\n  def getNonFeatureStoreDataRecordFeatures(featureMap: FeatureMap): Seq[DRFeature]\n\n  /**\n   * Because Feature Store features aren't direct features in the FeatureMap and instead live\n   * aggregated in a DataRecord in our Feature Map, we need to interface with the underlying Data\n   * Record instead. e.g. for the `AllFeatures` case, we won't know what all FStore ProMix Features\n   * we have in a FeatureMap just by looping through features & need to just return the DataRecord.\n   */\n  def getFeatureStoreFeaturesDataRecord(featureMap: FeatureMap): SRichDataRecord\n}\n\n/**\n * Use all DataRecord features on a FeatureMap to output a DataRecord.\n */\ncase class AllFeatures[-Entity]() extends FeaturesScope[BaseDataRecordFeature[Entity, _]] {\n  override def getNonFeatureStoreDataRecordFeatures(\n    featureMap: FeatureMap\n  ): Seq[BaseDataRecordFeature[Entity, _]] = {\n\n    /**\n     * See [[com.twitter.product_mixer.core.benchmark.FeatureMapBenchmark]]\n     *\n     * `toSeq`` is a no-op, `view`` makes later compositions lazy. Currently we only perform a `forEach`\n     * on the result but `view` here has no performance impact but protects us if we accidentally add\n     * more compositions in the middle.\n     *\n     * Feature Store features aren't in the FeatureMap so this will only ever return the non-FStore Features.\n     */\n    featureMap.getFeatures.toSeq.view.collect {\n      case feature: BaseDataRecordFeature[Entity, _] => feature\n    }\n  }\n\n  // Get the entire underlying DataRecord if available.\n  override def getFeatureStoreFeaturesDataRecord(\n    featureMap: FeatureMap\n  ): SRichDataRecord = if (featureMap.getFeatures.contains(FeatureStoreV1ResponseFeature)) {\n    // Note, we do not copy over the feature context because JRichDataRecord will enforce that\n    // all features are in the FeatureContext which we do not know at init time, and it's pricey\n    // to compute at run time.\n    SRichDataRecord(featureMap.get(FeatureStoreV1ResponseFeature).richDataRecord.getRecord)\n  } else {\n    SRichDataRecord(new DataRecord())\n  }\n}\n\n/**\n * Build a DataRecord with only the given features from the FeatureMap used. Missing features\n * will fail loudly.\n * @param features the specific features to include in the DataRecord.\n */\ncase class SpecificFeatures[DRFeature <: BaseDataRecordFeature[_, _]](\n  features: Set[DRFeature])\n    extends FeaturesScope[DRFeature] {\n\n  private val featuresForContext = features.collect {\n    case featureStoreFeatures: FeatureStoreV1Feature[_, _, _, _] =>\n      featureStoreFeatures.boundFeature.mlApiFeature\n    case dataRecordCompatible: DataRecordCompatible[_] => dataRecordCompatible.mlFeature\n  }.asJava\n\n  private val featureContext = new FeatureContext(featuresForContext)\n\n  private val fsFeatures = features\n    .collect {\n      case featureStoreV1Feature: FeatureStoreV1Feature[_, _, _, _] =>\n        featureStoreV1Feature\n    }\n\n  // Since it's possible a customer will pass feature store features in the DR Feature list, let's\n  // partition them out to only return non-FS ones in getFeatures. See [[FeaturesScope]] comment.\n  private val nonFsFeatures: Seq[DRFeature] = features.flatMap {\n    case _: FeatureStoreV1Feature[_, _, _, _] =>\n      None\n    case otherFeature => Some(otherFeature)\n  }.toSeq\n\n  override def getNonFeatureStoreDataRecordFeatures(\n    featureMap: FeatureMap\n  ): Seq[DRFeature] = nonFsFeatures\n\n  override def getFeatureStoreFeaturesDataRecord(\n    featureMap: FeatureMap\n  ): SRichDataRecord =\n    if (fsFeatures.nonEmpty && featureMap.getFeatures.contains(FeatureStoreV1ResponseFeature)) {\n      // Return a DataRecord only with the explicitly requested features set.\n      val richDataRecord = SRichDataRecord(new DataRecord(), featureContext)\n      val existingDataRecord = featureMap.get(FeatureStoreV1ResponseFeature).richDataRecord\n      fsFeatures.foreach { feature =>\n        richDataRecord.setFeatureValue(\n          feature.boundFeature.mlApiFeature,\n          existingDataRecord.getFeatureValue(feature.boundFeature.mlApiFeature))\n      }\n      richDataRecord\n    } else {\n      SRichDataRecord(new DataRecord())\n    }\n}\n\n/**\n * Build a DataRecord with every feature available in a FeatureMap except for the ones provided.\n * @param featuresToExclude the features to be excluded in the DataRecord.\n */\ncase class AllExceptFeatures(\n  featuresToExclude: Set[BaseDataRecordFeature[_, _]])\n    extends FeaturesScope[BaseDataRecordFeature[_, _]] {\n\n  private val fsFeatures = featuresToExclude\n    .collect {\n      case featureStoreV1Feature: FeatureStoreV1Feature[_, _, _, _] =>\n        featureStoreV1Feature\n    }\n\n  override def getNonFeatureStoreDataRecordFeatures(\n    featureMap: FeatureMap\n  ): Seq[BaseDataRecordFeature[_, _]] =\n    featureMap.getFeatures\n      .collect {\n        case feature: BaseDataRecordFeature[_, _] => feature\n      }.filterNot(featuresToExclude.contains).toSeq\n\n  override def getFeatureStoreFeaturesDataRecord(\n    featureMap: FeatureMap\n  ): SRichDataRecord = if (featureMap.getFeatures.contains(FeatureStoreV1ResponseFeature)) {\n    // Return a data record only with the explicitly requested features set. Do this by copying\n    // the existing one and removing the features in the denylist.\n    // Note, we do not copy over the feature context because JRichDataRecord will enforce that\n    // all features are in the FeatureContext which we do not know at init time, and it's pricey\n    // to compute at run time.\n    val richDataRecord = SRichDataRecord(\n      featureMap.get(FeatureStoreV1ResponseFeature).richDataRecord.getRecord.deepCopy())\n    fsFeatures.foreach { feature =>\n      richDataRecord.clearFeature(feature.boundFeature.mlApiFeature)\n    }\n    richDataRecord\n  } else {\n    SRichDataRecord(new DataRecord())\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/featurestorev1/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1/featurevalue\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1/featurevalue\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/featurestorev1/FeatureStoreV1FeatureMap.scala",
    "content": "package com.twitter.product_mixer.core.feature.featuremap.featurestorev1\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.MissingFeatureException\nimport com.twitter.product_mixer.core.feature.featurestorev1.FeatureStoreV1CandidateFeature\nimport com.twitter.product_mixer.core.feature.featurestorev1.FeatureStoreV1CandidateFeatureGroup\nimport com.twitter.product_mixer.core.feature.featurestorev1.FeatureStoreV1QueryFeature\nimport com.twitter.product_mixer.core.feature.featurestorev1.FeatureStoreV1QueryFeatureGroup\nimport com.twitter.product_mixer.core.feature.featurestorev1.featurevalue.FeatureStoreV1ResponseFeature\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.util.Try\n\nobject FeatureStoreV1FeatureMap {\n\n  /**\n   * Implicitly add convenience accessors for FeatureStoreV1 features in [[FeatureMap]]. Note that\n   * we cannot add these methods directly to [[FeatureMap]] because it would introduce a circular\n   * dependency ([[PipelineQuery]] depends on [[FeatureMap]], and the methods below depend on\n   * [[PipelineQuery]])\n   *\n   * @param featureMap the featureMap we are wrapping\n   * @note The FeatureStoreV1Feature::defaultValue set on the BoundFeature is only used and set\n   *       during PredictionRecord to DataRecord conversion. Therefore, the default will not be set\n   *       on the PredictionRecord value if reading from it directly, and as such for convenience\n   *       the defaultValue is manually returned during retrieval from PredictionRecord.\n   * @note the Value generic type on the methods below cannot be passed to\n   *       FeatureStoreV1QueryFeature's Value generic type. While this is actually the same type,\n   *       (note the explicit type cast back to Value), we must instead use an existential on\n   *       FeatureStoreV1QueryFeature since it is constructed with an existential for the Value\n   *       generic (see [[FeatureStoreV1QueryFeature]] and [[FeatureStoreV1CandidateFeature]])\n   */\n  implicit class FeatureStoreV1FeatureMapAccessors(private val featureMap: FeatureMap) {\n\n    def getFeatureStoreV1QueryFeature[Query <: PipelineQuery, Value](\n      feature: FeatureStoreV1QueryFeature[Query, _ <: EntityId, Value]\n    ): Value =\n      getOrElseFeatureStoreV1QueryFeature(\n        feature,\n        feature.defaultValue.getOrElse {\n          throw MissingFeatureException(feature)\n        })\n\n    def getFeatureStoreV1QueryFeatureTry[Query <: PipelineQuery, Value](\n      feature: FeatureStoreV1QueryFeature[Query, _ <: EntityId, Value]\n    ): Try[Value] =\n      Try(getFeatureStoreV1QueryFeature(feature))\n\n    def getOrElseFeatureStoreV1QueryFeature[Query <: PipelineQuery, Value](\n      feature: FeatureStoreV1QueryFeature[Query, _ <: EntityId, Value],\n      default: => Value\n    ): Value = {\n\n      /**\n       * FeatureStoreV1ResponseFeature should never be missing from the FeatureMap as FSv1 is\n       * guaranteed to return a prediction record per feature store request. However, this may be\n       * called on candidates that never hydrated FSv1 features. For example by\n       * [[com.twitter.product_mixer.component_library.selector.sorter.featurestorev1.FeatureStoreV1FeatureValueSorter]]\n       */\n      val featureStoreV1FeatureValueOpt = featureMap.getTry(FeatureStoreV1ResponseFeature).toOption\n\n      val dataRecordValue: Option[Value] = featureStoreV1FeatureValueOpt.flatMap {\n        featureStoreV1FeatureValue =>\n          featureStoreV1FeatureValue.richDataRecord.getFeatureValueOpt(\n            feature.boundFeature.mlApiFeature)(feature.fromDataRecordValue)\n      }\n\n      dataRecordValue.getOrElse(default)\n    }\n\n    def getFeatureStoreV1CandidateFeature[\n      Query <: PipelineQuery,\n      Candidate <: UniversalNoun[Any],\n      Value\n    ](\n      feature: FeatureStoreV1CandidateFeature[Query, Candidate, _ <: EntityId, Value]\n    ): Value =\n      getOrElseFeatureStoreV1CandidateFeature(\n        feature,\n        feature.defaultValue.getOrElse {\n          throw MissingFeatureException(feature)\n        })\n\n    def getFeatureStoreV1CandidateFeatureTry[\n      Query <: PipelineQuery,\n      Candidate <: UniversalNoun[Any],\n      Value\n    ](\n      feature: FeatureStoreV1CandidateFeature[Query, Candidate, _ <: EntityId, Value]\n    ): Try[Value] =\n      Try(getFeatureStoreV1CandidateFeature(feature))\n\n    def getOrElseFeatureStoreV1CandidateFeature[\n      Query <: PipelineQuery,\n      Candidate <: UniversalNoun[Any],\n      Value\n    ](\n      feature: FeatureStoreV1CandidateFeature[Query, Candidate, _ <: EntityId, Value],\n      default: => Value\n    ): Value = {\n\n      /**\n       * FeatureStoreV1ResponseFeature should never be missing from the FeatureMap as FSv1 is\n       * guaranteed to return a prediction record per feature store request. However, this may be\n       * called on candidates that never hydrated FSv1 features. For example by\n       * [[com.twitter.product_mixer.component_library.selector.sorter.featurestorev1.FeatureStoreV1FeatureValueSorter]]\n       */\n      val featureStoreV1FeatureValueOpt = featureMap.getTry(FeatureStoreV1ResponseFeature).toOption\n\n      val dataRecordValue: Option[Value] = featureStoreV1FeatureValueOpt.flatMap {\n        featureStoreV1FeatureValue =>\n          featureStoreV1FeatureValue.richDataRecord.getFeatureValueOpt(\n            feature.boundFeature.mlApiFeature)(feature.fromDataRecordValue)\n      }\n\n      dataRecordValue.getOrElse(default)\n    }\n\n    /**\n     * Get queryFeatureGroup, which is store in the featureMap as a DataRecordInAFeature\n     * It doesn't have the mlApiFeature as other regular FeatureStoreV1 features\n     * Please refer to [[com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature]] scaladoc for more details\n     */\n    def getFeatureStoreV1QueryFeatureGroup[Query <: PipelineQuery](\n      featureGroup: FeatureStoreV1QueryFeatureGroup[Query, _ <: EntityId]\n    ): DataRecord =\n      getOrElseFeatureStoreV1QueryFeatureGroup(\n        featureGroup,\n        throw MissingFeatureException(featureGroup)\n      )\n\n    def getFeatureStoreV1CandidateFeatureGroupTry[Query <: PipelineQuery](\n      featureGroup: FeatureStoreV1QueryFeatureGroup[Query, _ <: EntityId]\n    ): Try[DataRecord] =\n      Try(getFeatureStoreV1QueryFeatureGroup(featureGroup))\n\n    def getOrElseFeatureStoreV1QueryFeatureGroup[Query <: PipelineQuery](\n      featureGroup: FeatureStoreV1QueryFeatureGroup[Query, _ <: EntityId],\n      default: => DataRecord\n    ): DataRecord = {\n      featureMap.getTry(featureGroup).toOption.getOrElse(default)\n    }\n\n    /**\n     * Get candidateFeatureGroup, which is store in the featureMap as a DataRecordInAFeature\n     * It doesn't have the mlApiFeature as other regular FeatureStoreV1 features\n     * Please refer to [[com.twitter.product_mixer.core.feature.datarecord.DataRecordInAFeature]] scaladoc for more details\n     */\n    def getFeatureStoreV1CandidateFeatureGroup[\n      Query <: PipelineQuery,\n      Candidate <: UniversalNoun[Any]\n    ](\n      featureGroup: FeatureStoreV1CandidateFeatureGroup[Query, Candidate, _ <: EntityId]\n    ): DataRecord =\n      getOrElseFeatureStoreV1CandidateFeatureGroup(\n        featureGroup,\n        throw MissingFeatureException(featureGroup)\n      )\n\n    def getFeatureStoreV1CandidateFeatureGroupTry[\n      Query <: PipelineQuery,\n      Candidate <: UniversalNoun[Any]\n    ](\n      featureGroup: FeatureStoreV1CandidateFeatureGroup[Query, Candidate, _ <: EntityId]\n    ): Try[DataRecord] =\n      Try(getFeatureStoreV1CandidateFeatureGroup(featureGroup))\n\n    def getOrElseFeatureStoreV1CandidateFeatureGroup[\n      Query <: PipelineQuery,\n      Candidate <: UniversalNoun[Any]\n    ](\n      featureGroup: FeatureStoreV1CandidateFeatureGroup[Query, Candidate, _ <: EntityId],\n      default: => DataRecord\n    ): DataRecord = {\n      featureMap.getTry(featureGroup).toOption.getOrElse(default)\n    }\n\n    def getOrElseFeatureStoreV1FeatureDataRecord(\n      default: => DataRecord\n    ) = {\n      val featureStoreV1FeatureValueOpt = featureMap.getTry(FeatureStoreV1ResponseFeature).toOption\n\n      featureStoreV1FeatureValueOpt\n        .map { featureStoreV1FeatureValue =>\n          featureStoreV1FeatureValue.richDataRecord.getRecord\n        }.getOrElse(default)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/entities/core\",\n        \"src/scala/com/twitter/ml/featurestore/lib/data\",\n        \"src/scala/com/twitter/ml/featurestore/lib/dynamic\",\n        \"src/scala/com/twitter/ml/featurestore/lib/feature\",\n        \"src/scala/com/twitter/ml/featurestore/lib/feature:aggregates-feature-group\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/entities/core\",\n        \"src/scala/com/twitter/ml/featurestore/lib/data\",\n        \"src/scala/com/twitter/ml/featurestore/lib/dynamic\",\n        \"src/scala/com/twitter/ml/featurestore/lib/feature\",\n        \"src/scala/com/twitter/ml/featurestore/lib/feature:aggregates-feature-group\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1/FeatureStoreV1Entity.scala",
    "content": "package com.twitter.product_mixer.core.feature.featurestorev1\n\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.entity.Entity\nimport com.twitter.ml.featurestore.lib.entity.EntityWithId\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nsealed trait FeatureStoreV1Entity[\n  -Query <: PipelineQuery,\n  -Input,\n  FeatureStoreEntityId <: EntityId] {\n\n  val entity: Entity[FeatureStoreEntityId]\n}\n\ntrait FeatureStoreV1QueryEntity[-Query <: PipelineQuery, FeatureStoreEntityId <: EntityId]\n    extends FeatureStoreV1Entity[Query, Query, FeatureStoreEntityId] {\n\n  def entityWithId(query: Query): EntityWithId[FeatureStoreEntityId]\n}\n\ntrait FeatureStoreV1CandidateEntity[\n  -Query <: PipelineQuery,\n  -Input <: UniversalNoun[Any],\n  FeatureStoreEntityId <: EntityId]\n    extends FeatureStoreV1Entity[Query, Input, FeatureStoreEntityId] {\n\n  def entityWithId(\n    query: Query,\n    input: Input,\n    existingFeatures: FeatureMap\n  ): EntityWithId[FeatureStoreEntityId]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1/FeatureStoreV1Feature.scala",
    "content": "package com.twitter.product_mixer.core.feature.featurestorev1\n\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.transform.FeatureRenameTransform\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.dynamic.BaseGatedFeatures\nimport com.twitter.ml.featurestore.lib.feature.BoundFeature\nimport com.twitter.ml.featurestore.lib.feature.BoundFeatureSet\nimport com.twitter.ml.featurestore.lib.feature.TimelinesAggregationFrameworkFeatureGroup\nimport com.twitter.ml.featurestore.lib.feature.{Feature => FSv1Feature}\nimport com.twitter.product_mixer.core.feature.ModelFeatureName\nimport com.twitter.product_mixer.core.feature.datarecord.FeatureStoreDataRecordFeature\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.servo.util.{Gate => ServoGate}\nimport com.twitter.timelines.configapi.FSParam\nimport scala.reflect.ClassTag\n\n/**\n * The base trait for all feature store features on ProMix. This should not be constructed directly\n * and should instead be used through the other implementations below\n * @tparam Query Product Mixer Query Type\n * @tparam Input The input type the feature should be keyed on, this is same as Query for query\n *               features and\n * @tparam FeatureStoreEntityId Feature Store Entity Type\n * @tparam Value The type of the value of this feature.\n */\nsealed trait BaseFeatureStoreV1Feature[\n  -Query <: PipelineQuery,\n  -Input,\n  FeatureStoreEntityId <: EntityId,\n  Value]\n    extends FeatureStoreDataRecordFeature[Input, Value]\n    with BaseGatedFeatures[Query] {\n  val fsv1Feature: FSv1Feature[FeatureStoreEntityId, Value]\n\n  val entity: FeatureStoreV1Entity[Query, Input, FeatureStoreEntityId]\n\n  val enabledParam: Option[FSParam[Boolean]]\n\n  override final lazy val gate: ServoGate[Query] = enabledParam\n    .map { param =>\n      new ServoGate[PipelineQuery] {\n        override def apply[U](query: U)(implicit asT: <:<[U, PipelineQuery]): Boolean = {\n          query.params(param)\n        }\n      }\n    }.getOrElse(ServoGate.True)\n\n  override final lazy val boundFeatureSet: BoundFeatureSet = new BoundFeatureSet(Set(boundFeature))\n\n  val boundFeature: BoundFeature[FeatureStoreEntityId, Value]\n\n  /**\n   * Since this trait is normally constructed inline, avoid the anonymous toString and use the bounded feature name.\n   */\n  override lazy val toString: String = boundFeature.name\n}\n\n/**\n * A unitary (non-aggregate group) feature store feature in ProMix. This should be constructed using\n * [[FeatureStoreV1CandidateFeature]] or [[FeatureStoreV1QueryFeature]].\n * @tparam Query Product Mixer Query Type\n * @tparam Input The input type the feature should be keyed on, this is same as Query for query\n *               features and\n * @tparam FeatureStoreEntityId Feature Store Entity Type\n * @tparam Value The type of the value of this feature.\n */\nsealed trait FeatureStoreV1Feature[\n  -Query <: PipelineQuery,\n  -Input,\n  FeatureStoreEntityId <: EntityId,\n  Value]\n    extends BaseFeatureStoreV1Feature[Query, Input, FeatureStoreEntityId, Value]\n    with ModelFeatureName {\n\n  val legacyName: Option[String]\n  val defaultValue: Option[Value]\n\n  override lazy val featureName: String = boundFeature.name\n\n  override final lazy val boundFeature = (legacyName, defaultValue) match {\n    case (Some(legacyName), Some(defaultValue)) =>\n      fsv1Feature.bind(entity.entity).withLegacyName(legacyName).withDefault(defaultValue)\n    case (Some(legacyName), _) =>\n      fsv1Feature.bind(entity.entity).withLegacyName(legacyName)\n    case (_, Some(defaultValue)) =>\n      fsv1Feature.bind(entity.entity).withDefault(defaultValue)\n    case _ =>\n      fsv1Feature.bind(entity.entity)\n  }\n\n  def fromDataRecordValue(recordValue: boundFeature.feature.mfc.V): Value =\n    boundFeature.feature.mfc.fromDataRecordValue(recordValue)\n}\n\n/**\n * A feature store aggregated group feature in ProMix. This should be constructed using\n * [[FeatureStoreV1CandidateFeatureGroup]] or [[FeatureStoreV1QueryFeatureGroup]].\n *\n * @tparam Query Product Mixer Query Type\n * @tparam Input The input type the feature should be keyed on, this is same as Query for query\n *               features and\n * @tparam FeatureStoreEntityId Feature Store Entity Type\n */\nabstract class FeatureStoreV1FeatureGroup[\n  -Query <: PipelineQuery,\n  -Input,\n  FeatureStoreEntityId <: EntityId: ClassTag]\n    extends BaseFeatureStoreV1Feature[Query, Input, FeatureStoreEntityId, DataRecord] {\n  val keepLegacyNames: Boolean\n  val featureNameTransform: Option[FeatureRenameTransform]\n\n  val featureGroup: TimelinesAggregationFrameworkFeatureGroup[FeatureStoreEntityId]\n\n  override lazy val fsv1Feature: FSv1Feature[FeatureStoreEntityId, DataRecord] =\n    featureGroup.FeaturesAsDataRecord\n\n  override final lazy val boundFeature = (keepLegacyNames, featureNameTransform) match {\n    case (_, Some(transform)) =>\n      fsv1Feature.bind(entity.entity).withLegacyIndividualFeatureNames(transform)\n    case (true, _) =>\n      fsv1Feature.bind(entity.entity).keepLegacyNames\n    case _ =>\n      fsv1Feature.bind(entity.entity)\n  }\n}\n\nsealed trait BaseFeatureStoreV1QueryFeature[\n  -Query <: PipelineQuery,\n  FeatureStoreEntityId <: EntityId,\n  Value]\n    extends BaseFeatureStoreV1Feature[Query, Query, FeatureStoreEntityId, Value] {\n\n  override val entity: FeatureStoreV1QueryEntity[Query, FeatureStoreEntityId]\n}\n\ntrait FeatureStoreV1QueryFeature[-Query <: PipelineQuery, FeatureStoreEntityId <: EntityId, Value]\n    extends FeatureStoreV1Feature[Query, Query, FeatureStoreEntityId, Value]\n    with BaseFeatureStoreV1QueryFeature[Query, FeatureStoreEntityId, Value]\n\ntrait FeatureStoreV1QueryFeatureGroup[-Query <: PipelineQuery, FeatureStoreEntityId <: EntityId]\n    extends FeatureStoreV1FeatureGroup[Query, Query, FeatureStoreEntityId]\n    with BaseFeatureStoreV1QueryFeature[Query, FeatureStoreEntityId, DataRecord]\n\nobject FeatureStoreV1QueryFeature {\n\n  /**\n   * Query-based Feature Store backed feature\n   * @param feature The underling feature store feature this represents.\n   * @param _entity The entity for binding the Feature Store features\n   * @param _legacyName Feature Store legacy name if required\n   * @param _defaultValue The default value to return for this feature if not hydrated.\n   * @param _enabledParam The Feature Switch Param to gate this feature, always enabled if none.\n   * @tparam Query The Product Mixer query type this feature is keyed on.\n   * @tparam FeatureStoreEntityId Feature Store Entity ID\n   * @tparam Value The type of the value this feature contains.\n   * @return Product Mixer Feature\n   */\n  def apply[Query <: PipelineQuery, FeatureStoreEntityId <: EntityId, Value](\n    feature: FSv1Feature[FeatureStoreEntityId, Value],\n    _entity: FeatureStoreV1QueryEntity[Query, FeatureStoreEntityId],\n    _legacyName: Option[String] = None,\n    _defaultValue: Option[Value] = None,\n    _enabledParam: Option[FSParam[Boolean]] = None\n  ): FeatureStoreV1QueryFeature[Query, FeatureStoreEntityId, Value] =\n    new FeatureStoreV1QueryFeature[Query, FeatureStoreEntityId, Value] {\n      override val fsv1Feature: FSv1Feature[FeatureStoreEntityId, Value] = feature\n      override val entity: FeatureStoreV1QueryEntity[Query, FeatureStoreEntityId] = _entity\n      override val legacyName: Option[String] = _legacyName\n      override val defaultValue: Option[Value] = _defaultValue\n      override val enabledParam: Option[FSParam[Boolean]] = _enabledParam\n    }\n}\n\nobject FeatureStoreV1QueryFeatureGroup {\n\n  /**\n   * Query-based Feature Store Aggregated group backed feature\n   *\n   * @param featureGroup  The underling aggregation group feature this represents.\n   * @param _entity       The entity for binding the Feature Store features\n   * @param _enabledParam The Feature Switch Param to gate this feature, always enabled if none.\n   * @param _keepLegacyNames Whether to keep the legacy names as is for the entire group\n   * @param _featureNameTransform Rename the entire group's legacy names using the [[FeatureRenameTransform]]\n   * @tparam Query                The Product Mixer query type this feature is keyed on.\n   * @tparam FeatureStoreEntityId Feature Store Entity ID\n   *\n   * @return Product Mixer Feature\n   */\n  def apply[Query <: PipelineQuery, FeatureStoreEntityId <: EntityId: ClassTag](\n    _featureGroup: TimelinesAggregationFrameworkFeatureGroup[FeatureStoreEntityId],\n    _entity: FeatureStoreV1QueryEntity[Query, FeatureStoreEntityId],\n    _enabledParam: Option[FSParam[Boolean]] = None,\n    _keepLegacyNames: Boolean = false,\n    _featureNameTransform: Option[FeatureRenameTransform] = None\n  ): FeatureStoreV1QueryFeatureGroup[Query, FeatureStoreEntityId] =\n    new FeatureStoreV1QueryFeatureGroup[Query, FeatureStoreEntityId] {\n      override val entity: FeatureStoreV1QueryEntity[Query, FeatureStoreEntityId] = _entity\n      override val featureGroup: TimelinesAggregationFrameworkFeatureGroup[\n        FeatureStoreEntityId\n      ] = _featureGroup\n\n      override val enabledParam: Option[FSParam[Boolean]] = _enabledParam\n\n      override val keepLegacyNames: Boolean = _keepLegacyNames\n      override val featureNameTransform: Option[FeatureRenameTransform] = _featureNameTransform\n    }\n}\n\nsealed trait BaseFeatureStoreV1CandidateFeature[\n  -Query <: PipelineQuery,\n  -Input <: UniversalNoun[Any],\n  FeatureStoreEntityId <: EntityId,\n  Value]\n    extends BaseFeatureStoreV1Feature[Query, Input, FeatureStoreEntityId, Value] {\n\n  override val entity: FeatureStoreV1CandidateEntity[Query, Input, FeatureStoreEntityId]\n}\n\ntrait FeatureStoreV1CandidateFeature[\n  -Query <: PipelineQuery,\n  -Input <: UniversalNoun[Any],\n  FeatureStoreEntityId <: EntityId,\n  Value]\n    extends FeatureStoreV1Feature[Query, Input, FeatureStoreEntityId, Value]\n    with BaseFeatureStoreV1CandidateFeature[Query, Input, FeatureStoreEntityId, Value]\n\ntrait FeatureStoreV1CandidateFeatureGroup[\n  -Query <: PipelineQuery,\n  -Input <: UniversalNoun[Any],\n  FeatureStoreEntityId <: EntityId]\n    extends FeatureStoreV1FeatureGroup[Query, Input, FeatureStoreEntityId]\n    with BaseFeatureStoreV1CandidateFeature[Query, Input, FeatureStoreEntityId, DataRecord]\n\nobject FeatureStoreV1CandidateFeature {\n\n  /**\n   * Candidate-based Feature Store backed feature\n   * @param feature The underling feature store feature this represents.\n   * @param _entity The entity for binding the Feature Store features\n   * @param _legacyName Feature Store legacy name if required\n   * @param _defaultValue The default value to return for this feature if not hydrated.\n   * @param _enabledParam The Feature Switch Param to gate this feature, always enabled if none.\n   * @tparam Query The Product Mixer query type this feature is keyed on.\n   * @tparam FeatureStoreEntityId The feature store entity type\n   * @tparam Input The type of the candidate this feature is keyed on\n   * @tparam Value The type of value this feature contains.\n   * @return Product Mixer Feature\n   */\n  def apply[\n    Query <: PipelineQuery,\n    Input <: UniversalNoun[Any],\n    FeatureStoreEntityId <: EntityId,\n    Value\n  ](\n    feature: FSv1Feature[FeatureStoreEntityId, Value],\n    _entity: FeatureStoreV1CandidateEntity[Query, Input, FeatureStoreEntityId],\n    _legacyName: Option[String] = None,\n    _defaultValue: Option[Value] = None,\n    _enabledParam: Option[FSParam[Boolean]] = None\n  ): FeatureStoreV1CandidateFeature[Query, Input, FeatureStoreEntityId, Value] =\n    new FeatureStoreV1CandidateFeature[Query, Input, FeatureStoreEntityId, Value] {\n      override val fsv1Feature: FSv1Feature[FeatureStoreEntityId, Value] = feature\n      override val entity: FeatureStoreV1CandidateEntity[Query, Input, FeatureStoreEntityId] =\n        _entity\n      override val legacyName: Option[String] = _legacyName\n      override val defaultValue: Option[Value] = _defaultValue\n      override val enabledParam: Option[FSParam[Boolean]] = _enabledParam\n    }\n}\n\nobject FeatureStoreV1CandidateFeatureGroup {\n\n  /**\n   * Candidate-based Feature Store Aggregated group backed feature\n   *\n   * @param featureGroup          The underling aggregation group feature this represents.\n   * @param _entity               The entity for binding the Feature Store features\n   * @param _enabledParam         The Feature Switch Param to gate this feature, always enabled if none.\n   * @param _keepLegacyNames      Whether to keep the legacy names as is for the entire group\n   * @param _featureNameTransform Rename the entire group's legacy names using the [[FeatureRenameTransform]]\n   * @tparam Query                The Product Mixer query type this feature is keyed on.\n   * @tparam Input The type of the candidate this feature is keyed on\n   * @tparam FeatureStoreEntityId Feature Store Entity ID\n   *\n   * @return Product Mixer Feature\n   */\n  def apply[\n    Query <: PipelineQuery,\n    Input <: UniversalNoun[Any],\n    FeatureStoreEntityId <: EntityId: ClassTag,\n  ](\n    _featureGroup: TimelinesAggregationFrameworkFeatureGroup[FeatureStoreEntityId],\n    _entity: FeatureStoreV1CandidateEntity[Query, Input, FeatureStoreEntityId],\n    _enabledParam: Option[FSParam[Boolean]] = None,\n    _keepLegacyNames: Boolean = false,\n    _featureNameTransform: Option[FeatureRenameTransform] = None\n  ): FeatureStoreV1CandidateFeatureGroup[Query, Input, FeatureStoreEntityId] =\n    new FeatureStoreV1CandidateFeatureGroup[Query, Input, FeatureStoreEntityId] {\n      override val entity: FeatureStoreV1CandidateEntity[Query, Input, FeatureStoreEntityId] =\n        _entity\n      override val featureGroup: TimelinesAggregationFrameworkFeatureGroup[\n        FeatureStoreEntityId\n      ] = _featureGroup\n\n      override val enabledParam: Option[FSParam[Boolean]] = _enabledParam\n\n      override val keepLegacyNames: Boolean = _keepLegacyNames\n      override val featureNameTransform: Option[FeatureRenameTransform] = _featureNameTransform\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1/featurevalue/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"src/scala/com/twitter/ml/api/util:datarecord\",\n        \"src/scala/com/twitter/ml/featurestore/lib/data\",\n        \"src/scala/com/twitter/ml/featurestore/lib/dynamic\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"src/scala/com/twitter/ml/api/util:datarecord\",\n        \"src/scala/com/twitter/ml/featurestore/lib/data\",\n        \"src/scala/com/twitter/ml/featurestore/lib/dynamic\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1/featurevalue/FeatureStoreV1Response.scala",
    "content": "package com.twitter.product_mixer.core.feature.featurestorev1.featurevalue\n\nimport com.fasterxml.jackson.annotation.JsonIgnoreProperties\nimport com.fasterxml.jackson.annotation.JsonProperty\nimport com.twitter.ml.api.DataRecordMerger\nimport com.twitter.ml.api.util.SRichDataRecord\nimport com.twitter.ml.featurestore.lib.data.HydrationError\nimport com.twitter.product_mixer.core.feature.Feature\n\nprivate[product_mixer] object FeatureStoreV1ResponseFeature\n    extends Feature[Any, FeatureStoreV1Response]\n\n@JsonIgnoreProperties(Array(\"richDataRecord\", \"failedFeatures\"))\nprivate[product_mixer] case class FeatureStoreV1Response(\n  @JsonProperty(\"richDataRecord\") richDataRecord: SRichDataRecord,\n  @JsonProperty(\"failedFeatures\") failedFeatures: Map[_ <: Feature[_, _], Set[HydrationError]]) {\n  // Since RichDataRecord is Java, we need to override this.\n  override def equals(obj: Any): Boolean = obj match {\n    case that: FeatureStoreV1Response =>\n      failedFeatures == that.failedFeatures && richDataRecord.getRecord.equals(\n        that.richDataRecord.getRecord)\n    case _ => false\n  }\n}\n\nprivate[product_mixer] object FeatureStoreV1Response {\n  val dataRecordMerger = new DataRecordMerger\n  def merge(\n    left: FeatureStoreV1Response,\n    right: FeatureStoreV1Response\n  ): FeatureStoreV1Response = {\n    val newDataRecord = left.richDataRecord.getRecord.deepCopy()\n    dataRecordMerger.merge(newDataRecord, right.richDataRecord.getRecord)\n    FeatureStoreV1Response(\n      richDataRecord = SRichDataRecord(newDataRecord),\n      left.failedFeatures ++ right.failedFeatures\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/access_policy/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/fasterxml/jackson/core:jackson-annotations\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/CandidateSource.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source\n\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\n\nsealed trait BaseCandidateSource[-Request, +Candidate] extends Component {\n\n  /** @see [[CandidateSourceIdentifier]] */\n  val identifier: CandidateSourceIdentifier\n}\n\n/**\n * A [[CandidateSource]] returns a Seq of ''potential'' content\n *\n * @note [[CandidateSource]]s that return a single value need to transform\n *       it into a Seq, either by doing `Seq(value)` or extracting\n *       candidates from the value.\n *\n * @tparam Request arguments to get the potential content\n * @tparam Candidate the potential content\n */\ntrait CandidateSource[-Request, +Candidate] extends BaseCandidateSource[Request, Candidate] {\n\n  /** returns a Seq of ''potential'' content */\n  def apply(request: Request): Stitch[Seq[Candidate]]\n}\n\n/**\n * A [[CandidateSourceWithExtractedFeatures]] returns a result containing both a Seq of\n * ''potential'' candidates as well as an extracted feature map that will later be appended\n * to the pipeline's [[com.twitter.product_mixer.core.pipeline.PipelineQuery]] feature map. This is\n * useful for candidate sources that return features that might be useful later on without needing\n * to re-hydrate them.\n *\n * @note [[CandidateSource]]s that return a single value need to transform\n *       it into a Seq, either by doing `Seq(value)` or extracting\n *       candidates from the value.\n *\n * @tparam Request arguments to get the potential content\n * @tparam Candidate the potential content\n */\ntrait CandidateSourceWithExtractedFeatures[-Request, +Candidate]\n    extends BaseCandidateSource[Request, Candidate] {\n\n  /** returns a result containing a seq of ''potential'' content and extracted features\n   * from the candidate source.\n   * */\n  def apply(request: Request): Stitch[CandidatesWithSourceFeatures[Candidate]]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/CandidatesWithSourceFeatures.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\n\n/**\n * Results from a candidate source, optionally carrying extracted query level features to add\n * to the query's feature map (e.g, extracting reusable features from the thrift response of thrift\n * call).\n * @param candidates The candidates returned from the underlying CandidateSoure\n * @param features [[FeatureMap]] containing the features from the candidate source\n *                                    to merge back into the PipelineQuery FeatureMap.\n * @tparam Candidate The type of result\n */\ncase class CandidatesWithSourceFeatures[+Candidate](\n  candidates: Seq[Candidate],\n  features: FeatureMap)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/PassthroughCandidateSource.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Retrieve Candidates from the Query\n */\ntrait CandidateExtractor[-Request, +Candidate] {\n\n  def apply(query: Request): Seq[Candidate]\n}\n\n/**\n * Identity extractor for returning the Request as a Seq of candidates\n */\ncase class IdentityCandidateExtractor[Request]() extends CandidateExtractor[Request, Request] {\n\n  def apply(candidate: Request): Seq[Request] = Seq(candidate)\n}\n\n/**\n * Retrieve Candidates from a [[Feature]] on the [[PipelineQuery]]'s FeatureMap. This extractor\n * supports a transform if the Feature value and the Seq of [[Candidate]] types do not match\n */\ntrait QueryFeatureCandidateExtractor[-Query <: PipelineQuery, FeatureValue, +Candidate]\n    extends CandidateExtractor[Query, Candidate] {\n\n  def feature: Feature[Query, FeatureValue]\n\n  override def apply(query: Query): Seq[Candidate] =\n    query.features.map(featureMap => transform(featureMap.get(feature))).getOrElse(Seq.empty)\n\n  def transform(featureValue: FeatureValue): Seq[Candidate]\n}\n\n/**\n * Retrieve Candidates from a [[Feature]] on the [[PipelineQuery]]'s FeatureMap. This extractor can\n * be used with a single [[Feature]] if the Feature value and the Seq of [[Candidate]] types match.\n */\ncase class CandidateQueryFeatureCandidateExtractor[-Query <: PipelineQuery, Candidate](\n  override val feature: Feature[Query, Seq[Candidate]])\n    extends QueryFeatureCandidateExtractor[Query, Seq[Candidate], Candidate] {\n\n  override def transform(featureValue: Seq[Candidate]): Seq[Candidate] = featureValue\n}\n\n/**\n * A [[CandidateSource]] that retrieves candidates from the Request via a [[CandidateExtractor]]\n */\ncase class PassthroughCandidateSource[-Request, +Candidate](\n  override val identifier: CandidateSourceIdentifier,\n  candidateExtractor: CandidateExtractor[Request, Candidate])\n    extends CandidateSource[Request, Candidate] {\n\n  def apply(query: Request): Stitch[Seq[Candidate]] = Stitch.value(candidateExtractor(query))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/StaticCandidateSource.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source\n\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.stitch.Stitch\n\n/**\n * A [[CandidateSource]] that always returns [[result]] regardless of the input\n */\ncase class StaticCandidateSource[Candidate](\n  override val identifier: CandidateSourceIdentifier,\n  result: Seq[Candidate])\n    extends CandidateSource[Any, Candidate] {\n\n  def apply(request: Any): Stitch[Seq[Candidate]] = Stitch.value(result)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/product_pipeline/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/product_pipeline/ProductPipelineCandidateSource.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source.product_pipeline\n\nimport com.google.inject.Provider\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineRequest\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry\nimport com.twitter.stitch.Stitch\nimport scala.reflect.runtime.universe._\n\n/**\n * A [[CandidateSource]] for getting candidates from a different\n * [[com.twitter.product_mixer.core.model.marshalling.request.Product]] within the same Product\n * Mixer-based service. This is useful when calling a RecommendationPipeline-based Product from a\n * MixerPipeline-based Product. In this scenario, the two Products can remain\n * independent and encapsulated within the Product Mixer service, which provides future optionality\n * for migrating one of the two products into a new Product Mixer-based service based on the\n * scaling needs.\n *\n * @tparam Query [[PipelineQuery]] from the originating Product\n * @tparam MixerRequest the [[Request]] domain model for the Product Mixer service. Adds a Context\n *                      bound (syntactic sugar) to add TypeTag to implicit scope for\n *                      [[ProductPipelineRegistry.getProductPipeline()]]. Note that `trait` does not\n *                      support context bounds, so this abstraction is expressed as an\n *                      `abstract class`.\n * @tparam ProductPipelineResult the return type of the candidate source Product. Adds a Context\n *                               bound (syntactic sugar) to add TypeTag to implicit scope for\n *                               [[ProductPipelineRegistry.getProductPipeline()]]\n * @tparam Candidate the type of candidate returned by this candidate source, which is typically\n *                   extracted from within the ProductPipelineResult type\n */\nabstract class ProductPipelineCandidateSource[\n  -Query <: PipelineQuery,\n  MixerRequest <: Request: TypeTag,\n  ProductPipelineResult: TypeTag,\n  +Candidate]\n    extends CandidateSource[Query, Candidate] {\n\n  /**\n   * @note Define as a Guice [[Provider]] in order to break the circular injection dependency\n   */\n  val productPipelineRegistry: Provider[ProductPipelineRegistry]\n\n  /**\n   * @note Define as a Guice [[Provider]] in order to break the circular injection dependency\n   */\n  val paramsBuilder: Provider[ParamsBuilder]\n\n  def pipelineRequestTransformer(currentPipelineQuery: Query): MixerRequest\n\n  def productPipelineResultTransformer(productPipelineResult: ProductPipelineResult): Seq[Candidate]\n\n  override def apply(query: Query): Stitch[Seq[Candidate]] = {\n    val request = pipelineRequestTransformer(query)\n\n    val params = paramsBuilder\n      .get().build(\n        clientContext = request.clientContext,\n        product = request.product,\n        featureOverrides = request.debugParams.flatMap(_.featureOverrides).getOrElse(Map.empty)\n      )\n\n    productPipelineRegistry\n      .get()\n      .getProductPipeline[MixerRequest, ProductPipelineResult](request.product)\n      .process(ProductPipelineRequest(request, params))\n      .map(productPipelineResultTransformer)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"stitch/stitch-core\",\n        \"strato/src/main/scala/com/twitter/strato/catalog\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"strato/src/main/scala/com/twitter/strato/response\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"stitch/stitch-core\",\n        \"strato/src/main/scala/com/twitter/strato/catalog\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato/StratoErrCategorizer.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source.strato\n\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.Unauthorized\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.response.Err\n\n/**\n * Categorize Strato's Err messages to our PipelineFailures.\n *\n * This should be used by all strato-based candidate source, and we can\n * add more cases here as they're useful.\n */\nobject StratoErrCategorizer {\n  val CategorizeStratoException: PartialFunction[Throwable, Stitch[Nothing]] = {\n    case err @ Err(Err.Authorization, reason, context) =>\n      Stitch.exception(\n        PipelineFailure(Unauthorized, s\"$reason [${context.toString}]\", underlying = Some(err))\n      )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato/StratoKeyFetcherSeqSource.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source.strato\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\n\n/**\n * A [[CandidateSource]] for getting Candidates from Strato where the\n * Strato column's View is [[Unit]] and the Value is a Seq of [[StratoResult]]\n *\n * @tparam StratoKey the column's Key type\n * @tparam StratoResult the column's Value's Seq type\n */\ntrait StratoKeyFetcherSeqSource[StratoKey, StratoResult]\n    extends CandidateSource[StratoKey, StratoResult] {\n\n  val fetcher: Fetcher[StratoKey, Unit, Seq[StratoResult]]\n\n  override def apply(key: StratoKey): Stitch[Seq[StratoResult]] = {\n    fetcher\n      .fetch(key)\n      .map { result =>\n        result.v\n          .getOrElse(Seq.empty)\n      }.rescue(StratoErrCategorizer.CategorizeStratoException)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato/StratoKeyFetcherSource.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source.strato\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\n\n/**\n * A [[CandidateSource]] for getting Candidates from Strato where the\n * Strato column's View is [[Unit]] and the Value is a [[StratoValue]]\n *\n * A `stratoResultTransformer` must be defined to convert the [[StratoValue]] into a Seq of [[Candidate]]\n *\n * If you need to extract features from the [[StratoValue]] (like a cursor),\n * use [[StratoKeyFetcherWithSourceFeaturesSource]] instead.\n *\n * @tparam StratoKey the column's Key type\n * @tparam StratoValue the column's Value type\n */\ntrait StratoKeyFetcherSource[StratoKey, StratoValue, Candidate]\n    extends CandidateSource[StratoKey, Candidate] {\n\n  val fetcher: Fetcher[StratoKey, Unit, StratoValue]\n\n  /**\n   * Transforms the value type returned by Strato into a Seq[Candidate].\n   *\n   * This might be as simple as `Seq(stratoResult)` if you're always returning a single candidate.\n   *\n   * Often, it just extracts a Seq from within a larger wrapper object.\n   *\n   * If there is global metadata that you need to include, you can zip it with the candidates,\n   * returning something like Seq((candiate, metadata), (candidate, metadata)) etc.\n   */\n  protected def stratoResultTransformer(stratoResult: StratoValue): Seq[Candidate]\n\n  override def apply(key: StratoKey): Stitch[Seq[Candidate]] = {\n    fetcher\n      .fetch(key)\n      .map { result =>\n        result.v\n          .map(stratoResultTransformer)\n          .getOrElse(Seq.empty)\n      }.rescue(StratoErrCategorizer.CategorizeStratoException)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato/StratoKeyFetcherWithSourceFeaturesSource.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source.strato\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\n\n/**\n * A [[CandidateSource]] for getting Candidates from Strato where the\n * Strato column's View is [[Unit]] and the Value is a [[StratoValue]]\n *\n * A [[stratoResultTransformer]] must be defined to convert the\n * [[StratoValue]] into a Seq of [[Candidate]]\n *\n * A [[extractFeaturesFromStratoResult]] must be defined to extract a\n * [[FeatureMap]] from the [[StratoValue]]. If you don't need to do that,\n * use a [[StratoKeyFetcherSource]] instead.\n *\n * @tparam StratoKey the column's Key type\n * @tparam StratoValue the column's Value type\n */\ntrait StratoKeyFetcherWithSourceFeaturesSource[StratoKey, StratoValue, Candidate]\n    extends CandidateSourceWithExtractedFeatures[StratoKey, Candidate] {\n\n  val fetcher: Fetcher[StratoKey, Unit, StratoValue]\n\n  /**\n   * Transforms the value type returned by Strato into a Seq[Candidate].\n   *\n   * This might be as simple as `Seq(stratoResult)` if you're always returning a single candidate.\n   *\n   * Often, it just extracts a Seq from within a larger wrapper object.\n   *\n   * If there is global metadata that you need to include, see [[extractFeaturesFromStratoResult]]\n   * below to put that into a Feature.\n   */\n  protected def stratoResultTransformer(stratoResult: StratoValue): Seq[Candidate]\n\n  /***\n   * Transforms the value type returned by Strato into a FeatureMap.\n   *\n   * Override this to extract global metadata like cursors and place the results\n   * into a Feature.\n   *\n   * For example, a cursor.\n   */\n  protected def extractFeaturesFromStratoResult(stratoResult: StratoValue): FeatureMap\n\n  override def apply(key: StratoKey): Stitch[CandidatesWithSourceFeatures[Candidate]] = {\n    fetcher\n      .fetch(key)\n      .map { result =>\n        val candidates = result.v\n          .map(stratoResultTransformer)\n          .getOrElse(Seq.empty)\n\n        val features = result.v\n          .map(extractFeaturesFromStratoResult)\n          .getOrElse(FeatureMap.empty)\n\n        CandidatesWithSourceFeatures(candidates, features)\n      }.rescue(StratoErrCategorizer.CategorizeStratoException)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato/StratoKeyView.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source.strato\n\n/** Represents a Strato column's Key and View arguments */\ncase class StratoKeyView[StratoKey, StratoView](key: StratoKey, view: StratoView)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato/StratoKeyViewFetcherSeqSource.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source.strato\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\n\n/**\n * A [[CandidateSource]] for getting Candidates from Strato where the\n * Strato column's View is [[StratoView]] and the Value is a Seq of [[StratoResult]]\n *\n * @tparam StratoKey the column's Key type\n * @tparam StratoView the column's View type\n * @tparam StratoResult the column's Value's Seq type\n */\ntrait StratoKeyViewFetcherSeqSource[StratoKey, StratoView, StratoResult]\n    extends CandidateSource[StratoKeyView[StratoKey, StratoView], StratoResult] {\n\n  val fetcher: Fetcher[StratoKey, StratoView, Seq[StratoResult]]\n\n  override def apply(\n    request: StratoKeyView[StratoKey, StratoView]\n  ): Stitch[Seq[StratoResult]] = {\n    fetcher\n      .fetch(request.key, request.view)\n      .map { result =>\n        result.v\n          .getOrElse(Seq.empty)\n      }.rescue(StratoErrCategorizer.CategorizeStratoException)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato/StratoKeyViewFetcherSource.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source.strato\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\n\n/**\n * A [[CandidateSource]] for getting Candidates from Strato where the\n * Strato column's View is [[StratoView]] and the Value is a [[StratoValue]]\n *\n * A `stratoResultTransformer` must be defined to convert the [[StratoValue]] into a Seq of [[Candidate]]\n *\n * If you need to extract features from the [[StratoValue]] (like a cursor),\n * use [[StratoKeyViewFetcherWithSourceFeaturesSource]] instead.\n *\n * @tparam StratoKey the column's Key type\n * @tparam StratoView the column's View type\n * @tparam StratoValue the column's Value type\n */\ntrait StratoKeyViewFetcherSource[StratoKey, StratoView, StratoValue, Candidate]\n    extends CandidateSource[StratoKeyView[StratoKey, StratoView], Candidate] {\n\n  val fetcher: Fetcher[StratoKey, StratoView, StratoValue]\n\n  /**\n   * Transforms the value type returned by Strato into a Seq[Candidate].\n   *\n   * This might be as simple as `Seq(stratoResult)` if you're always returning a single candidate.\n   *\n   * Often, it just extracts a Seq from within a larger wrapper object.\n   *\n   * If there is global metadata that you need to include, you can zip it with the candidates,\n   * returning something like Seq((candiate, metadata), (candidate, metadata)) etc.\n   */\n  protected def stratoResultTransformer(\n    stratoKey: StratoKey,\n    stratoResult: StratoValue\n  ): Seq[Candidate]\n\n  override def apply(\n    request: StratoKeyView[StratoKey, StratoView]\n  ): Stitch[Seq[Candidate]] = {\n    fetcher\n      .fetch(request.key, request.view)\n      .map { result =>\n        result.v\n          .map((stratoResult: StratoValue) => stratoResultTransformer(request.key, stratoResult))\n          .getOrElse(Seq.empty)\n      }.rescue(StratoErrCategorizer.CategorizeStratoException)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source/strato/StratoKeyViewFetcherWithSourceFeaturesSource.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.candidate_source.strato\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.client.Fetcher\n\n/**\n * A [[CandidateSource]] for getting Candidates from Strato where the\n * Strato column's View is [[StratoView]] and the Value is a [[StratoValue]]\n *\n * A [[stratoResultTransformer]] must be defined to convert the\n * [[StratoValue]] into a Seq of [[Candidate]]\n *\n * [[extractFeaturesFromStratoResult]] must be defined to extract a\n * [[FeatureMap]] from the [[StratoValue]]. If you don't need to do that,\n * use a [[StratoKeyViewFetcherSource]] instead.\n *\n * @tparam StratoKey the column's Key type\n * @tparam StratoView the column's View type\n * @tparam StratoValue the column's Value type\n */\ntrait StratoKeyViewFetcherWithSourceFeaturesSource[StratoKey, StratoView, StratoValue, Candidate]\n    extends CandidateSourceWithExtractedFeatures[StratoKeyView[StratoKey, StratoView], Candidate] {\n\n  val fetcher: Fetcher[StratoKey, StratoView, StratoValue]\n\n  /**\n   * Transforms the value type returned by Strato into a Seq[Candidate].\n   *\n   * This might be as simple as `Seq(stratoResult)` if you're always returning a single candidate.\n   *\n   * Often, it just extracts a Seq from within a larger wrapper object.\n   *\n   * If there is global metadata that you need to include, see [[extractFeaturesFromStratoResult]]\n   * below to put that into a Feature.\n   */\n  protected def stratoResultTransformer(\n    stratoKey: StratoKey,\n    stratoResult: StratoValue\n  ): Seq[Candidate]\n\n  /**\n   * Transforms the value type returned by Strato into a FeatureMap.\n   *\n   * Override this to extract global metadata like cursors and place the results\n   * into a Feature.\n   *\n   * For example, a cursor.\n   */\n  protected def extractFeaturesFromStratoResult(\n    stratoKey: StratoKey,\n    stratoResult: StratoValue\n  ): FeatureMap\n\n  override def apply(\n    request: StratoKeyView[StratoKey, StratoView]\n  ): Stitch[CandidatesWithSourceFeatures[Candidate]] = {\n    fetcher\n      .fetch(request.key, request.view)\n      .map { result =>\n        val candidates = result.v\n          .map((stratoResult: StratoValue) => stratoResultTransformer(request.key, stratoResult))\n          .getOrElse(Seq.empty)\n\n        val features = result.v\n          .map((stratoResult: StratoValue) =>\n            extractFeaturesFromStratoResult(request.key, stratoResult))\n          .getOrElse(FeatureMap.empty)\n\n        CandidatesWithSourceFeatures(candidates, features)\n      }.rescue(StratoErrCategorizer.CategorizeStratoException)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/CandidateScope.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common\n\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.CandidatePipelines\n\n/**\n * Specifies whether a function component (e.g, [[Gate]] or [[Selector]])\n * should apply to a given [[CandidateWithDetails]]\n */\nsealed trait CandidateScope {\n\n  /**\n   * returns True if the provided `candidate` is in scope\n   */\n  def contains(candidate: CandidateWithDetails): Boolean\n\n  /** partitions `candidates` into those that this scope [[contains]] and those it does not */\n  final def partition(\n    candidates: Seq[CandidateWithDetails]\n  ): CandidateScope.PartitionedCandidates = {\n    val (candidatesInScope, candidatesOutOfScope) = candidates.partition(contains)\n    CandidateScope.PartitionedCandidates(candidatesInScope, candidatesOutOfScope)\n  }\n}\n\nobject CandidateScope {\n  case class PartitionedCandidates(\n    candidatesInScope: Seq[CandidateWithDetails],\n    candidatesOutOfScope: Seq[CandidateWithDetails])\n}\n\n/**\n * A [[CandidateScope]] that applies the given functional component\n * to all candidates regardless of which pipeline is their [[com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails.source]].\n */\ncase object AllPipelines extends CandidateScope {\n  override def contains(candidate: CandidateWithDetails): Boolean = true\n}\n\n/**\n * A [[CandidateScope]] that applies the given [[com.twitter.product_mixer.core.functional_component.selector.Selector]]\n * only to candidates whose [[com.twitter.product_mixer.core.model.common.presentation.CandidatePipelines]]\n * has an Identifier in the [[pipelines]] Set.\n * In most cases where candidates are not pre-merged, the Set contains the candidate pipeline identifier the candidate\n * came from. In the case where a candidate's feature maps were merged using [[CombineFeatureMapsCandidateMerger]], the\n * set contains all candidate pipelines the merged candidate came from and this scope will include the candidate if any\n * of the pipelines match.\n */\ncase class SpecificPipelines(pipelines: Set[CandidatePipelineIdentifier]) extends CandidateScope {\n\n  require(\n    pipelines.nonEmpty,\n    \"Expected `SpecificPipelines` have a non-empty Set of CandidatePipelineIdentifiers.\")\n\n  override def contains(candidate: CandidateWithDetails): Boolean = {\n    candidate.features.get(CandidatePipelines).exists(pipelines.contains)\n  }\n}\n\n/**\n * A [[CandidateScope]] that applies the given [[com.twitter.product_mixer.core.functional_component.selector.Selector]]\n * only to candidates whose [[com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails.source]]\n * is [[pipeline]].\n */\ncase class SpecificPipeline(pipeline: CandidatePipelineIdentifier) extends CandidateScope {\n\n  override def contains(candidate: CandidateWithDetails): Boolean = candidate.features\n    .get(CandidatePipelines).contains(pipeline)\n}\n\nobject SpecificPipelines {\n  def apply(\n    pipeline: CandidatePipelineIdentifier,\n    pipelines: CandidatePipelineIdentifier*\n  ): CandidateScope = {\n    if (pipelines.isEmpty)\n      SpecificPipeline(pipeline)\n    else\n      SpecificPipelines((pipeline +: pipelines).toSet)\n  }\n}\n\n/**\n * A [[CandidateScope]] that applies the given [[com.twitter.product_mixer.core.functional_component.selector.Selector]]\n * to all candidates except for the candidates whose [[com.twitter.product_mixer.core.model.common.presentation.CandidatePipelines]]\n * has an Identifier in the [[pipelinesToExclude]] Set.\n * In most cases where candidates are not pre-merged, the Set contains the candidate pipeline identifier the candidate\n * came from. In the case where a candidate's feature maps were merged using [[CombineFeatureMapsCandidateMerger]], the\n * set contains all candidate pipelines the merged candidate came from and this scope will include the candidate if any\n * of the pipelines match.\n */\ncase class AllExceptPipelines(\n  pipelinesToExclude: Set[CandidatePipelineIdentifier])\n    extends CandidateScope {\n  override def contains(candidate: CandidateWithDetails): Boolean = !candidate.features\n    .get(CandidatePipelines).exists(pipelinesToExclude.contains)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/access_policy/AccessPolicy.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.access_policy\n\nimport com.fasterxml.jackson.annotation.JsonSubTypes\nimport com.fasterxml.jackson.annotation.JsonTypeInfo\n\n/**\n * The Access Policy to set for gating querying in the turntable tool.\n *\n * @note implementations must be simple case classes with unique structures for serialization\n */\n@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, include = JsonTypeInfo.As.PROPERTY)\n@JsonSubTypes(\n  Array(\n    new JsonSubTypes.Type(value = classOf[AllowedLdapGroups], name = \"AllowedLdapGroups\"),\n    new JsonSubTypes.Type(value = classOf[BlockEverything], name = \"BlockEverything\")\n  )\n)\nsealed trait AccessPolicy\n\n/**\n * Users must be in *at least* one of the provided LDAP groups in order to make a query.\n *\n * @param groups LDAP groups allowed to access this product\n */\ncase class AllowedLdapGroups(groups: Set[String]) extends AccessPolicy\n\nobject AllowedLdapGroups {\n  def apply(group: String): AllowedLdapGroups = AllowedLdapGroups(Set(group))\n}\n\n/**\n * Block all requests to a product.\n *\n * @note this needs to be a case class rather than an object because classOf doesn't work on objects\n *       and JsonSubTypes requires the annotation argument to be a constant (ruling out .getClass).\n *       This issue may be resolved in Scala 2.13: https://github.com/scala/scala/pull/9279\n */\ncase class BlockEverything() extends AccessPolicy\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/access_policy/AccessPolicyEvaluator.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.access_policy\n\n/**\n * Controls how access policies are applied to allow/reject a request\n */\nobject AccessPolicyEvaluator {\n  def evaluate(productAccessPolicies: Set[AccessPolicy], userLdapGroups: Set[String]): Boolean =\n    productAccessPolicies.exists {\n      case AllowedLdapGroups(allowedGroups) => allowedGroups.exists(userLdapGroups.contains)\n      case _: BlockEverything => false\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/access_policy/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/fasterxml/jackson/core:jackson-annotations\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n    ],\n    exports = [\n        \"3rdparty/jvm/com/fasterxml/jackson/core:jackson-annotations\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/access_policy/WithDebugAccessPolicies.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.access_policy\n\nimport com.twitter.product_mixer.core.model.common.Component\n\nprivate[core] trait WithDebugAccessPolicies { self: Component =>\n\n  /** The [[AccessPolicy]]s that will be used for this component in turntable & other debug tooling\n   * to execute arbitrary queries against the component */\n  val debugAccessPolicies: Set[AccessPolicy] = Set.empty\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/Alert.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.Predicate\n\n/**\n * [[Alert]]s will trigger notifications to their [[NotificationGroup]]\n * when the [[Predicate]]s are triggered.\n */\ntrait Alert {\n\n  /** A group of alert levels and where the alerts for those levels should be sent */\n  val notificationGroup: NotificationGroup\n\n  /** Predicate indicating that the component is in a degraded state */\n  val warnPredicate: Predicate\n\n  /** Predicate indicating that the component is not functioning correctly */\n  val criticalPredicate: Predicate\n\n  /** An optional link to the runbook detailing how to respond to this alert */\n  val runbookLink: Option[String]\n\n  /** Indicates which metrics this [[Alert]] is for */\n  val alertType: AlertType\n\n  /** Where the metrics are from, @see [[Source]] */\n  val source: Source = Server()\n\n  /** A suffix to add to the end of the metric, this is often a [[Percentile]] */\n  val metricSuffix: Option[String] = None\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/AlertType.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\n/**\n * [[AlertType]] is used to indicate which metric an alert is for\n *\n * @note adding new [[AlertType]]s requires updating the dashboard generation code\n */\nsealed trait AlertType { val metricType: String }\n\n/** Monitors the latency */\ncase object Latency extends AlertType { override val metricType: String = \"Latency\" }\n\n/** Monitors the success rate __excluding__ client failures */\ncase object SuccessRate extends AlertType { override val metricType: String = \"SuccessRate\" }\n\n/** Monitors the throughput */\ncase object Throughput extends AlertType { override val metricType: String = \"Throughput\" }\n\n/** Monitors the empty response rate */\ncase object EmptyResponseRate extends AlertType {\n  override val metricType: String = \"EmptyResponseRate\"\n}\n\n/** Monitors the empty response size */\ncase object ResponseSize extends AlertType { override val metricType: String = \"ResponseSize\" }\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/fasterxml/jackson/core:jackson-annotations\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/jvm/javax/mail\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/predicate\",\n        \"strato/src/main/scala/com/twitter/strato/catalog\",\n        \"util/util-core:scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/predicate\",\n        \"strato/src/main/scala/com/twitter/strato/catalog\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/EmptyResponseRateAlert.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfAbove\n\n/**\n * [[EmptyResponseRateAlert]] triggers when the percentage of requests with empty responses (defined\n * as the number of items returned excluding cursors) rises above the [[TriggerIfAbove]] threshold\n * for a configured amount of time.\n *\n * @note EmptyResponseRate thresholds must be between 0 and 100%\n */\ncase class EmptyResponseRateAlert(\n  override val notificationGroup: NotificationGroup,\n  override val warnPredicate: TriggerIfAbove,\n  override val criticalPredicate: TriggerIfAbove,\n  override val runbookLink: Option[String] = None)\n    extends Alert {\n  override val alertType: AlertType = EmptyResponseRate\n  require(\n    warnPredicate.threshold > 0 && warnPredicate.threshold <= 100,\n    s\"EmptyResponseRateAlert predicates must be between 0 and 100 but got warnPredicate = ${warnPredicate.threshold}\"\n  )\n  require(\n    criticalPredicate.threshold > 0 && criticalPredicate.threshold <= 100,\n    s\"EmptyResponseRateAlert predicates must be between 0 and 100 but got criticalPredicate = ${criticalPredicate.threshold}\"\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/GenericClientLatencyAlert.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfLatencyAbove\n\n/**\n * Similar to [[LatencyAlert]] but intended for use with an external client calling Product Mixer.\n *\n * [[GenericClientLatencyAlert]] triggers when the Latency for the specified client\n * rises above the [[TriggerIfLatencyAbove]] threshold for the configured amount of time.\n */\ncase class GenericClientLatencyAlert(\n  override val source: GenericClient,\n  override val notificationGroup: NotificationGroup,\n  override val warnPredicate: TriggerIfLatencyAbove,\n  override val criticalPredicate: TriggerIfLatencyAbove,\n  override val runbookLink: Option[String] = None)\n    extends Alert {\n  override val alertType: AlertType = Latency\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/GenericClientSuccessRateAlert.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfBelow\n\n/**\n * Similar to [[SuccessRateAlert]] but intended for use with an external client calling Product Mixer\n *\n * [[GenericClientSuccessRateAlert]] triggers when the Success Rate for the external client\n * drops below the [[TriggerIfBelow]] threshold for the configured amount of time\n *\n * @note SuccessRate thresholds must be between 0 and 100%\n */\ncase class GenericClientSuccessRateAlert(\n  override val source: GenericClient,\n  override val notificationGroup: NotificationGroup,\n  override val warnPredicate: TriggerIfBelow,\n  override val criticalPredicate: TriggerIfBelow,\n  override val runbookLink: Option[String] = None)\n    extends Alert {\n  override val alertType: AlertType = SuccessRate\n  require(\n    warnPredicate.threshold > 0 && warnPredicate.threshold <= 100,\n    s\"SuccessRateAlert predicates must be between 0 and 100 but got warnPredicate = ${warnPredicate.threshold}\"\n  )\n  require(\n    criticalPredicate.threshold > 0 && criticalPredicate.threshold <= 100,\n    s\"SuccessRateAlert predicates must be between 0 and 100 but got criticalPredicate = ${criticalPredicate.threshold}\"\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/GenericClientThroughputAlert.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.ThroughputPredicate\n\n/**\n * Similar to [[ThroughputAlert]] but intended for an external client calling Product Mixer.\n *\n * [[GenericClientThroughputAlert]] triggers when the requests/sec for the external client\n * is outside of the predicate set by a [[ThroughputPredicate]] for the configured amount of time\n */\ncase class GenericClientThroughputAlert(\n  override val source: GenericClient,\n  override val notificationGroup: NotificationGroup,\n  override val warnPredicate: ThroughputPredicate,\n  override val criticalPredicate: ThroughputPredicate,\n  override val runbookLink: Option[String] = None)\n    extends Alert {\n  override val alertType: AlertType = Throughput\n  require(\n    warnPredicate.threshold >= 0,\n    s\"ThroughputAlert predicates must be >= 0 but got warnPredicate = ${warnPredicate.threshold}\")\n  require(\n    criticalPredicate.threshold >= 0,\n    s\"ThroughputAlert predicates must be >= 0 but got criticalPredicate = ${criticalPredicate.threshold}\")\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/IsObservableFromStrato.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\n/**\n * Indicates that an [[Alert]] can be passed to [[StratoColumnAlert]]. Not all [[Alert]]s can be\n * Strato alerts since our ability to observe from Strato's perspective is limited by the available\n * metrics.\n *\n * @note can only be used in conjunction with [[Alert]]\n */\ntrait IsObservableFromStrato { _: Alert => }\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/LatencyAlert.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfLatencyAbove\n\n/**\n * [[GenericClientLatencyAlert]] triggers when the Latency for the component this is used with\n * rises above the [[TriggerIfLatencyAbove]] threshold for the configured amount of time\n */\ncase class LatencyAlert(\n  override val notificationGroup: NotificationGroup,\n  percentile: Percentile,\n  override val warnPredicate: TriggerIfLatencyAbove,\n  override val criticalPredicate: TriggerIfLatencyAbove,\n  override val runbookLink: Option[String] = None)\n    extends Alert\n    with IsObservableFromStrato {\n  override val alertType: AlertType = Latency\n\n  override val metricSuffix: Option[String] = Some(percentile.metricSuffix)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/NotificationGroup.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\nimport com.twitter.util.Try\nimport javax.mail.internet.InternetAddress\n\n/**\n * Destination represents the place to which alerts will be sent. Often you will only need one field\n * populated (either a Pager Duty key or a list of emails).\n *\n * See the Monitoring 2.0 docs for more information on [[https://docbird.twitter.biz/mon/how-to-guides.html?highlight=notificationgroup#set-up-email-pagerduty-and-slack-notifications setting up a Pager Duty rotation]]\n */\ncase class Destination(\n  pagerDutyKey: Option[String] = None,\n  emails: Seq[String] = Seq.empty) {\n\n  require(\n    pagerDutyKey.forall(_.length == 32),\n    s\"Expected `pagerDutyKey` to be 32 characters long but got `$pagerDutyKey`\")\n  emails.foreach { email =>\n    require(\n      Try(new InternetAddress(email).validate()).isReturn,\n      s\"Expected only valid email addresses but got an invalid email address: `$email`\")\n  }\n  require(\n    pagerDutyKey.nonEmpty || emails.nonEmpty,\n    s\"Expected a `pagerDutyKey` or at least 1 email address but got neither\")\n}\n\n/**\n * NotificationGroup maps alert levels to destinations. Having different destinations based on the\n * urgency of the alert can sometimes be useful. For example, you could have a daytime on-call for\n * warn alerts and a 24 on-call for critical alerts.\n */\ncase class NotificationGroup(\n  critical: Destination,\n  warn: Destination)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/Percentile.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\n/**\n * [[Percentile]] is the specific metric that should be monitored.\n * Some metrics such as Latency are recorded using [[https://twitter.github.io/util/docs/com/twitter/finagle/stats/Stat.html Stats]]\n * the stats are recorded as various percentiles such as `my/stat.p95` or `my/stat.min`.\n */\nsealed trait Percentile { val metricSuffix: String }\ncase object Min extends Percentile { override val metricSuffix: String = \".min\" }\ncase object Avg extends Percentile { override val metricSuffix: String = \".avg\" }\ncase object P50 extends Percentile { override val metricSuffix: String = \".p50\" }\ncase object P90 extends Percentile { override val metricSuffix: String = \".p90\" }\ncase object P95 extends Percentile { override val metricSuffix: String = \".p95\" }\ncase object P99 extends Percentile { override val metricSuffix: String = \".p99\" }\ncase object P999 extends Percentile { override val metricSuffix: String = \".p9990\" }\ncase object P9999 extends Percentile { override val metricSuffix: String = \".p9999\" }\ncase object Max extends Percentile { override val metricSuffix: String = \".max\" }\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/ResponseSizeAlert.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.ThroughputPredicate\n\n/**\n * [[ResponseSizeAlert]] triggers when the specified percentile of requests with empty responses (defined\n * as the number of items returned excluding cursors) is beyond the [[ThroughputPredicate]] threshold\n * for a configured amount of time.\n */\ncase class ResponseSizeAlert(\n  override val notificationGroup: NotificationGroup,\n  percentile: Percentile,\n  override val warnPredicate: ThroughputPredicate,\n  override val criticalPredicate: ThroughputPredicate,\n  override val runbookLink: Option[String] = None)\n    extends Alert {\n  override val metricSuffix: Option[String] = Some(percentile.metricSuffix)\n  override val alertType: AlertType = ResponseSize\n  require(\n    warnPredicate.threshold >= 0,\n    s\"ResponseSizeAlert predicates must be >= 0 but got warnPredicate = ${warnPredicate.threshold}\"\n  )\n  require(\n    criticalPredicate.threshold >= 0,\n    s\"ResponseSizeAlert predicates must be >= 0 but got criticalPredicate = ${criticalPredicate.threshold}\"\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/Source.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\nimport com.fasterxml.jackson.annotation.JsonSubTypes\nimport com.fasterxml.jackson.annotation.JsonTypeInfo\n\n/**\n * where the metric originates from, such as from the server or from a client\n *\n * @note implementations must be simple case classes with unique structures for serialization\n */\n@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, include = JsonTypeInfo.As.PROPERTY)\n@JsonSubTypes(\n  Array(\n    new JsonSubTypes.Type(value = classOf[Server], name = \"Server\"),\n    new JsonSubTypes.Type(value = classOf[Strato], name = \"Strato\"),\n    new JsonSubTypes.Type(value = classOf[GenericClient], name = \"GenericClient\")\n  )\n)\nsealed trait Source\n\n/** metrics for the Product Mixer server */\ncase class Server() extends Source\n\n/** metrics from the perspective of a Strato column */\ncase class Strato(stratoColumnPath: String, stratoColumnOp: String) extends Source\n\n/**\n * metrics from the perspective of a generic client\n *\n * @param displayName human readable name for the client\n * @param service service referenced in the query, of the form <role>.<env>.<job>\n * @param metricSource the source of the metric query, usually of the form sd.<role>.<env>.<job>\n * @param failureMetric the name of the metric indicating a client failure\n * @param requestMetric the name of the metric indicating a request has been made\n * @param latencyMetric the name of the metric measuring a request's latency\n *\n * @note We strongly recommend the use of [[Strato]] where possible. [[GenericClient]] is provided as a\n *       catch-all source for teams that have unusual legacy call paths (such as Macaw).\n */\ncase class GenericClient(\n  displayName: String,\n  service: String,\n  metricSource: String,\n  failureMetric: String,\n  requestMetric: String,\n  latencyMetric: String)\n    extends Source\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/StratoColumnAlert.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.Predicate\nimport com.twitter.strato.catalog.OpTag\n\n/**\n * triggers when the a Strato column's is outside of the predicate set by the provided [[Alert]]\n *\n * @note the [[Alert]] passed into a [[StratoColumnAlert]]\n *       can not be a [[StratoColumnAlert]]\n */\ncase class StratoColumnAlert(column: String, op: OpTag, alert: Alert with IsObservableFromStrato)\n    extends Alert {\n\n  override val source: Source = Strato(column, op.tag)\n  override val notificationGroup: NotificationGroup = alert.notificationGroup\n  override val warnPredicate: Predicate = alert.warnPredicate\n  override val criticalPredicate: Predicate = alert.criticalPredicate\n  override val runbookLink: Option[String] = alert.runbookLink\n  override val alertType: AlertType = alert.alertType\n  override val metricSuffix: Option[String] = alert.metricSuffix\n}\n\nobject StratoColumnAlerts {\n\n  /** Make a seq of Alerts for the provided Strato column */\n  def apply(\n    column: String,\n    op: OpTag,\n    alerts: Seq[Alert with IsObservableFromStrato]\n  ): Seq[Alert] = {\n    alerts.map(StratoColumnAlert(column, op, _))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/SuccessRateAlert.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.TriggerIfBelow\n\n/**\n * [[SuccessRateAlert]] triggers when the Success Rate for the component this is used\n * with drops below the [[TriggerIfBelow]] threshold for the configured amount of time\n *\n * @note SuccessRate thresholds must be between 0 and 100%\n */\ncase class SuccessRateAlert(\n  override val notificationGroup: NotificationGroup,\n  override val warnPredicate: TriggerIfBelow,\n  override val criticalPredicate: TriggerIfBelow,\n  override val runbookLink: Option[String] = None)\n    extends Alert\n    with IsObservableFromStrato {\n  override val alertType: AlertType = SuccessRate\n  require(\n    warnPredicate.threshold > 0 && warnPredicate.threshold <= 100,\n    s\"SuccessRateAlert predicates must be between 0 and 100 but got warnPredicate = ${warnPredicate.threshold}\"\n  )\n  require(\n    criticalPredicate.threshold > 0 && criticalPredicate.threshold <= 100,\n    s\"SuccessRateAlert predicates must be between 0 and 100 but got criticalPredicate = ${criticalPredicate.threshold}\"\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/ThroughputAlert.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.predicate.ThroughputPredicate\n\n/**\n * [[ThroughputAlert]] triggers when the requests/sec for the component this is used\n * with is outside of the predicate set by a [[ThroughputPredicate]] for\n * the configured amount of time\n */\ncase class ThroughputAlert(\n  override val notificationGroup: NotificationGroup,\n  override val warnPredicate: ThroughputPredicate,\n  override val criticalPredicate: ThroughputPredicate,\n  override val runbookLink: Option[String] = None)\n    extends Alert\n    with IsObservableFromStrato {\n  override val alertType: AlertType = Throughput\n  require(\n    warnPredicate.threshold >= 0,\n    s\"ThroughputAlert predicates must be >= 0 but got warnPredicate = ${warnPredicate.threshold}\")\n  require(\n    criticalPredicate.threshold >= 0,\n    s\"ThroughputAlert predicates must be >= 0 but got criticalPredicate = ${criticalPredicate.threshold}\")\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/predicate/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"util/util-core:scala\",\n    ],\n    exports = [\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/predicate/MetricGranularity.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert.predicate\n\n/**\n * Specifies the metric granularity\n *\n * @see [[https://docbird.twitter.biz/mon/reference.html#predicate DURATION]]\n */\nsealed trait MetricGranularity { val unit: String }\n\n/**\n * Use minutely metrics and have alert durations in terms of minutes\n *\n * i.e. for a [[Predicate]] if [[Predicate.datapointsPastThreshold]] = 5 and [[Predicate.duration]] = 10\n * then the alert will trigger if there are at least 5 '''minutely''' metric points that are past the threshold\n * in any 10 '''minute''' period\n */\ncase object Minutes extends MetricGranularity { override val unit: String = \"m\" }\n\n/**\n * Use hourly metrics and have alert durations in terms of hours\n *\n * i.e. for a [[Predicate]] if [[Predicate.datapointsPastThreshold]] = 5 and [[Predicate.duration]] = 10\n * then the alert will trigger if there are at least 5 '''hourly''' metric points that are past the threshold\n * in any 10 '''hour''' period\n */\ncase object Hours extends MetricGranularity { override val unit: String = \"h\" }\n\n/**\n * Use daily metrics and have alert durations in terms of days\n *\n * i.e. for a [[Predicate]] if [[Predicate.datapointsPastThreshold]] = 5 and [[Predicate.duration]] = 10\n * then the alert will trigger if there are at least 5 '''daily''' metric points that are past the threshold\n * in any 10 '''day''' period\n */\ncase object Days extends MetricGranularity { override val unit: String = \"d\" }\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/predicate/Operator.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert.predicate\n\n/**\n * Used for building [[Predicate]]s\n *\n * @see [[https://docbird.twitter.biz/mon/reference.html#predicate OPERATOR]]\n */\nprivate[alert] sealed trait Operator\nprivate[alert] case object `>` extends Operator\nprivate[alert] case object `>=` extends Operator\nprivate[alert] case object `<` extends Operator\nprivate[alert] case object `<=` extends Operator\nprivate[alert] case object `!=` extends Operator\nprivate[alert] case object `=` extends Operator\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/predicate/Predicate.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert.predicate\n\n/**\n * [[Predicate]]s will trigger if the metric's value is past the\n * `threshold` for `datapointsPastThreshold` or more datapoints\n * in a given `duration`\n *\n * @see [[https://docbird.twitter.biz/mon/reference.html#predicate Predicate]]\n */\ntrait Predicate {\n\n  /** @see [[https://docbird.twitter.biz/mon/reference.html#predicate OPERATOR]] */\n  val operator: Operator\n\n  /** @see [[https://docbird.twitter.biz/mon/reference.html#predicate THRESHOLD]] */\n  val threshold: Double\n\n  /**\n   * The number of datapoints in a given duration beyond the threshold that will trigger an alert\n   * @see [[https://docbird.twitter.biz/mon/reference.html#predicate DATAPOINTS]]\n   */\n  val datapointsPastThreshold: Int\n\n  /**\n   * @note if using a [[metricGranularity]] of [[Minutes]] then this must be >= 3\n   * @see [[https://docbird.twitter.biz/mon/reference.html#predicate DURATION]]\n   */\n  val duration: Int\n\n  /**\n   * Specifies the metric granularity\n   * @see [[https://docbird.twitter.biz/mon/reference.html#predicate DURATION]]\n   */\n  val metricGranularity: MetricGranularity\n\n  require(\n    datapointsPastThreshold > 0,\n    s\"`datapointsPastThreshold` must be > 0 but got `datapointsPastThreshold` = $datapointsPastThreshold\"\n  )\n\n  require(\n    datapointsPastThreshold <= duration,\n    s\"`datapointsPastThreshold` must be <= than `duration.inMinutes` but got `datapointsPastThreshold` = $datapointsPastThreshold `duration` = $duration\"\n  )\n  require(\n    metricGranularity != Minutes || duration >= 3,\n    s\"Predicate durations must be at least 3 minutes but got $duration\"\n  )\n}\n\n/** [[ThroughputPredicate]]s are predicates that can trigger when the throughput is too low or high */\ntrait ThroughputPredicate extends Predicate\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/predicate/TriggerIfAbove.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert.predicate\n\n/**\n * A [[Predicate]] that triggers if the metric this is used with rises above\n * the [[threshold]] for [[datapointsPastThreshold]] per [[duration]]\n */\ncase class TriggerIfAbove(\n  override val threshold: Double,\n  override val datapointsPastThreshold: Int = 10,\n  override val duration: Int = 15,\n  override val metricGranularity: MetricGranularity = Minutes)\n    extends Predicate\n    with ThroughputPredicate {\n  override val operator: Operator = `>`\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/predicate/TriggerIfBelow.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert.predicate\n\n/**\n * A [[Predicate]] that triggers if the metric this is used with lowers below\n * the [[threshold]] for [[datapointsPastThreshold]] per [[duration]]\n */\ncase class TriggerIfBelow(\n  override val threshold: Double,\n  override val datapointsPastThreshold: Int = 10,\n  override val duration: Int = 15,\n  override val metricGranularity: MetricGranularity = Minutes)\n    extends Predicate\n    with ThroughputPredicate {\n  override val operator: Operator = `<`\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert/predicate/TriggerIfLatencyAbove.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.common.alert.predicate\n\nimport com.twitter.util.Duration\n\n/**\n * A [[Predicate]] that triggers if the metric this is used with rises above the\n * [[latencyThreshold]] for [[datapointsPastThreshold]] per [[duration]]\n *\n * @note [[latencyThreshold]] must be > 0\n */\ncase class TriggerIfLatencyAbove(\n  latencyThreshold: Duration,\n  override val datapointsPastThreshold: Int = 10,\n  override val duration: Int = 15,\n  override val metricGranularity: MetricGranularity = Minutes)\n    extends Predicate {\n  override val threshold: Double = latencyThreshold.inMillis\n  override val operator: Operator = `>`\n  require(\n    latencyThreshold > Duration.Zero,\n    s\"TriggerIfLatencyAbove thresholds must be greater than 0 but got $latencyThreshold\")\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-featureswitches:v2\",\n        \"featureswitches/featureswitches-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/registry\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-featureswitches:v2\",\n        \"featureswitches/featureswitches-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/registry\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/ConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.configapi\n\nimport com.twitter.product_mixer.core.functional_component.configapi.registry.GlobalParamRegistry\nimport com.twitter.product_mixer.core.product.registry.ProductParamRegistry\nimport com.twitter.timelines.configapi.CompositeConfig\nimport com.twitter.timelines.configapi.Config\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ConfigBuilder @Inject() (\n  productParamRegistry: ProductParamRegistry,\n  globalParamRegistry: GlobalParamRegistry) {\n\n  def build(): Config =\n    new CompositeConfig(productParamRegistry.build() ++ Seq(globalParamRegistry.build()))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/ParamsBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.configapi\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport com.twitter.servo.util.MemoizingStatsReceiver\nimport com.twitter.timelines.configapi.Config\nimport com.twitter.timelines.configapi.FeatureValue\nimport com.twitter.timelines.configapi.Params\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/** Singleton object for building [[Params]] to override */\n@Singleton\nclass ParamsBuilder @Inject() (\n  config: Config,\n  requestContextBuilder: RequestContextBuilder,\n  statsReceiver: StatsReceiver) {\n\n  private[this] val scopedStatsReceiver =\n    new MemoizingStatsReceiver(statsReceiver.scope(\"configapi\"))\n\n  def build(\n    clientContext: ClientContext,\n    product: Product,\n    featureOverrides: Map[String, FeatureValue],\n    fsCustomMapInput: Map[String, Any] = Map.empty\n  ): Params = {\n    val requestContext =\n      requestContextBuilder.build(clientContext, product, featureOverrides, fsCustomMapInput)\n\n    config(requestContext, scopedStatsReceiver)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/RequestContext.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.configapi\n\nimport com.twitter.timelines.configapi.BaseRequestContext\nimport com.twitter.timelines.configapi.FeatureContext\nimport com.twitter.timelines.configapi.GuestId\nimport com.twitter.timelines.configapi.UserId\nimport com.twitter.timelines.configapi.WithFeatureContext\nimport com.twitter.timelines.configapi.WithGuestId\nimport com.twitter.timelines.configapi.WithUserId\n\n/** Represents [[com.twitter.timelines.configapi]]'s context information */\ncase class RequestContext(\n  userId: Option[UserId],\n  guestId: Option[GuestId],\n  featureContext: FeatureContext)\n    extends BaseRequestContext\n    with WithUserId\n    with WithGuestId\n    with WithFeatureContext\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/RequestContextBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.configapi\n\nimport com.twitter.featureswitches.v2.FeatureSwitches\nimport com.twitter.featureswitches.UserAgent\nimport com.twitter.featureswitches.{Recipient => FeatureSwitchRecipient}\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport com.twitter.timelines.configapi.featureswitches.v2.FeatureSwitchResultsFeatureContext\nimport com.twitter.timelines.configapi.FeatureContext\nimport com.twitter.timelines.configapi.FeatureValue\nimport com.twitter.timelines.configapi.ForcedFeatureContext\nimport com.twitter.timelines.configapi.OrElseFeatureContext\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Request Context Factory is used to build RequestContext objects which are used\n * by the config api to determine the param overrides to apply to the request.\n * The param overrides are determined per request by configs which specify which\n * FS/Deciders/AB translate to what param overrides.\n */\n@Singleton\nclass RequestContextBuilder @Inject() (featureSwitches: FeatureSwitches) {\n\n  /**\n   * @param `fsCustomMapInput` allows you to set custom fields on your feature switches.\n   * This feature isn't directly supported by product mixer yet, so using this argument\n   * will likely result in future cleanup work.\n   *\n   */\n  def build(\n    clientContext: ClientContext,\n    product: Product,\n    featureOverrides: Map[String, FeatureValue],\n    fsCustomMapInput: Map[String, Any]\n  ): RequestContext = {\n    val featureContext =\n      getFeatureContext(clientContext, product, featureOverrides, fsCustomMapInput)\n\n    RequestContext(clientContext.userId, clientContext.guestId, featureContext)\n  }\n\n  private[configapi] def getFeatureContext(\n    clientContext: ClientContext,\n    product: Product,\n    featureOverrides: Map[String, FeatureValue],\n    fsCustomMapInput: Map[String, Any]\n  ): FeatureContext = {\n    val recipient = getFeatureSwitchRecipient(clientContext)\n      .withCustomFields(\"product\" -> product.identifier.toString, fsCustomMapInput.toSeq: _*)\n\n    val results = featureSwitches.matchRecipient(recipient)\n    OrElseFeatureContext(\n      ForcedFeatureContext(featureOverrides),\n      new FeatureSwitchResultsFeatureContext(results))\n  }\n\n  private[configapi] def getFeatureSwitchRecipient(\n    clientContext: ClientContext\n  ): FeatureSwitchRecipient = FeatureSwitchRecipient(\n    userId = clientContext.userId,\n    userRoles = clientContext.userRoles,\n    deviceId = clientContext.deviceId,\n    guestId = clientContext.guestId,\n    languageCode = clientContext.languageCode,\n    countryCode = clientContext.countryCode,\n    userAgent = clientContext.userAgent.flatMap(UserAgent.apply),\n    isVerified = None,\n    clientApplicationId = clientContext.appId,\n    isTwoffice = clientContext.isTwoffice\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/StaticParam.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.configapi\n\nimport com.twitter.timelines.configapi.Param\n\n/** A [[Param]] used for constant values where we do not require backing by feature switches or deciders */\ncase class StaticParam[ValueType](value: ValueType) extends Param[ValueType](value)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/registry/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-decider\",\n        \"servo/decider\",\n        \"util/util-core:util-core-util\",\n        \"util/util-logging\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-decider\",\n        \"servo/decider\",\n        \"util/util-core:util-core-util\",\n        \"util/util-logging\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/registry/GlobalParamConfig.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.configapi.registry\n\n/**\n * Register Params that do not relate to a specific product.\n * See [[ParamConfig]] for hooks to register Params based on type.\n */\ntrait GlobalParamConfig extends ParamConfig with ParamConfigBuilder\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/registry/GlobalParamRegistry.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.configapi.registry\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.servo.decider.DeciderGateBuilder\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.Config\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GlobalParamRegistry @Inject() (\n  globalParamConfig: GlobalParamConfig,\n  deciderGateBuilder: DeciderGateBuilder,\n  statsReceiver: StatsReceiver) {\n\n  def build(): Config = {\n    val globalConfigs = globalParamConfig.build(deciderGateBuilder, statsReceiver)\n\n    BaseConfigBuilder(globalConfigs).build(\"GlobalParamRegistry\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/registry/ParamConfig.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.configapi.registry\n\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil.DefinedFeatureName\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil.EnumParamWithFeatureName\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil.EnumSeqParamWithFeatureName\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil.ValueFeatureName\nimport com.twitter.timelines.configapi.decider.HasDecider\nimport com.twitter.timelines.configapi.Bounded\nimport com.twitter.timelines.configapi.FSName\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.timelines.configapi.OptionalOverride\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\n/** ParamConfig is used to configure overrides for [[Param]]s of various types */\ntrait ParamConfig {\n\n  def booleanDeciderOverrides: Seq[Param[Boolean] with HasDecider] = Seq.empty\n\n  def booleanFSOverrides: Seq[Param[Boolean] with FSName] = Seq.empty\n\n  def optionalBooleanOverrides: Seq[\n    (Param[Option[Boolean]], DefinedFeatureName, ValueFeatureName)\n  ] = Seq.empty\n\n  def enumFSOverrides: Seq[EnumParamWithFeatureName[_ <: Enumeration]] = Seq.empty\n\n  def enumSeqFSOverrides: Seq[EnumSeqParamWithFeatureName[_ <: Enumeration]] = Seq.empty\n\n  /**\n   * Support for non-Duration supplied FS overrides (e.g. `timeFromStringFSOverrides`,\n   * `timeFromNumberFSOverrides`, `getBoundedOptionalDurationFromMillisOverrides`) is not provided\n   * as Duration is preferred\n   */\n  def boundedDurationFSOverrides: Seq[\n    Param[Duration] with Bounded[Duration] with FSName with HasDurationConversion\n  ] = Seq.empty\n\n  /** Support for unbounded numeric FS overrides is not provided as bounded is preferred */\n  def boundedIntFSOverrides: Seq[Param[Int] with Bounded[Int] with FSName] = Seq.empty\n\n  def boundedOptionalIntOverrides: Seq[\n    (Param[Option[Int]] with Bounded[Option[Int]], DefinedFeatureName, ValueFeatureName)\n  ] = Seq.empty\n\n  def intSeqFSOverrides: Seq[Param[Seq[Int]] with FSName] = Seq.empty\n\n  def boundedLongFSOverrides: Seq[Param[Long] with Bounded[Long] with FSName] = Seq.empty\n\n  def boundedOptionalLongOverrides: Seq[\n    (Param[Option[Long]] with Bounded[Option[Long]], DefinedFeatureName, ValueFeatureName)\n  ] = Seq.empty\n\n  def longSeqFSOverrides: Seq[Param[Seq[Long]] with FSName] = Seq.empty\n\n  def longSetFSOverrides: Seq[Param[Set[Long]] with FSName] = Seq.empty\n\n  def boundedDoubleFSOverrides: Seq[Param[Double] with Bounded[Double] with FSName] = Seq.empty\n\n  def boundedOptionalDoubleOverrides: Seq[\n    (Param[Option[Double]] with Bounded[Option[Double]], DefinedFeatureName, ValueFeatureName)\n  ] = Seq.empty\n\n  def doubleSeqFSOverrides: Seq[Param[Seq[Double]] with FSName] = Seq.empty\n\n  def stringFSOverrides: Seq[Param[String] with FSName] = Seq.empty\n\n  def stringSeqFSOverrides: Seq[Param[Seq[String]] with FSName] = Seq.empty\n\n  def optionalStringOverrides: Seq[(Param[Option[String]], DefinedFeatureName, ValueFeatureName)] =\n    Seq.empty\n\n  def gatedOverrides: Map[String, Seq[OptionalOverride[_]]] = Map.empty\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/registry/ParamConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.configapi.registry\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.servo.decider.DeciderGateBuilder\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.OptionalOverride\nimport com.twitter.timelines.configapi.decider.DeciderUtils\n\ntrait ParamConfigBuilder { paramConfig: ParamConfig =>\n\n  /** Builds a Seq of [[OptionalOverride]]s based on the [[paramConfig]] */\n  def build(\n    deciderGateBuilder: DeciderGateBuilder,\n    statsReceiver: StatsReceiver\n  ): Seq[OptionalOverride[_]] = {\n    val logger = Logger(classOf[ParamConfig])\n\n    DeciderUtils.getBooleanDeciderOverrides(deciderGateBuilder, booleanDeciderOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getBooleanFSOverrides(booleanFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getOptionalBooleanOverrides(optionalBooleanOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getEnumFSOverrides(\n        statsReceiver.scope(\"enumConversion\"),\n        logger,\n        enumFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getEnumSeqFSOverrides(\n        statsReceiver.scope(\"enumSeqConversion\"),\n        logger,\n        enumSeqFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getBoundedDurationFSOverrides(boundedDurationFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(boundedIntFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getBoundedOptionalIntOverrides(boundedOptionalIntOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getIntSeqFSOverrides(intSeqFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getBoundedLongFSOverrides(boundedLongFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getBoundedOptionalLongOverrides(boundedOptionalLongOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getLongSeqFSOverrides(longSeqFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getLongSetFSOverrides(longSetFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(boundedDoubleFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getBoundedOptionalDoubleOverrides(\n        boundedOptionalDoubleOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getDoubleSeqFSOverrides(doubleSeqFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getStringFSOverrides(stringFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getStringSeqFSOverrides(stringSeqFSOverrides: _*) ++\n      FeatureSwitchOverrideUtil.getOptionalStringOverrides(optionalStringOverrides: _*) ++\n      gatedOverrides.flatMap {\n        case (fsName, overrides) => FeatureSwitchOverrideUtil.gatedOverrides(fsName, overrides: _*)\n      }.toSeq\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/BUILD",
    "content": "scala_library(\n    name = \"decoration\",\n    sources = [\"Decoration.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n    ],\n)\n\nscala_library(\n    sources = [\n        \"!Decoration.scala\",\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":decoration\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \":decoration\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/CandidateDecorator.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.DecoratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * [[CandidateDecorator]] generates a [[com.twitter.product_mixer.core.model.common.presentation.UniversalPresentation]]\n * for Candidates, which encapsulate information about how to present the candidate\n *\n * @see [[https://docbird.twitter.biz/product-mixer/functional-components.html#candidate-decorator]]\n * @see [[com.twitter.product_mixer.core.model.common.presentation.UniversalPresentation]]\n */\ntrait CandidateDecorator[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]]\n    extends Component {\n\n  override val identifier: DecoratorIdentifier = CandidateDecorator.DefaultCandidateDecoratorId\n\n  /**\n   * Given a Seq of `Candidate`, returns a [[Decoration]] for candidates which should be decorated\n   *\n   * `Candidate`s which aren't decorated can be omitted from the results\n   */\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[Seq[Decoration]]\n}\n\nobject CandidateDecorator {\n  private[core] val DefaultCandidateDecoratorId: DecoratorIdentifier =\n    DecoratorIdentifier(ComponentIdentifier.BasedOnParentComponent)\n\n  /**\n   * For use when building a [[CandidateDecorator]] in a [[com.twitter.product_mixer.core.pipeline.PipelineBuilder]]\n   * to ensure that the identifier is updated with the parent [[com.twitter.product_mixer.core.pipeline.Pipeline.identifier]]\n   */\n  private[core] def copyWithUpdatedIdentifier[\n    Query <: PipelineQuery,\n    Candidate <: UniversalNoun[Any]\n  ](\n    decorator: CandidateDecorator[Query, Candidate],\n    parentIdentifier: ComponentIdentifier\n  ): CandidateDecorator[Query, Candidate] = {\n    if (decorator.identifier == DefaultCandidateDecoratorId) {\n      new CandidateDecorator[Query, Candidate] {\n        override val identifier: DecoratorIdentifier = DecoratorIdentifier(parentIdentifier.name)\n        override def apply(\n          query: Query,\n          candidates: Seq[CandidateWithFeatures[Candidate]]\n        ): Stitch[Seq[Decoration]] = decorator.apply(query, candidates)\n      }\n    } else {\n      decorator\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/Decoration.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.presentation.UniversalPresentation\n\n/**\n * Decoration associates specific [[UniversalPresentation]] with a candidate\n */\ncase class Decoration(\n  candidate: UniversalNoun[Any],\n  presentation: UniversalPresentation)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/slice/builder/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/slice/builder/CandidateSliceItemBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.slice.builder\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.SliceItem\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait CandidateSliceItemBuilder[\n  -Query <: PipelineQuery,\n  -BuilderInput <: UniversalNoun[Any],\n  BuilderOutput <: SliceItem] {\n\n  def apply(query: Query, candidate: BuilderInput, featureMap: FeatureMap): BuilderOutput\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/BUILD",
    "content": "scala_library(\n    sources = [\"**/*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/CandidateUrtEntryBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait CandidateUrtEntryBuilder[\n  -Query <: PipelineQuery,\n  -BuilderInput <: UniversalNoun[Any],\n  BuilderOutput <: TimelineEntry] {\n\n  def apply(query: Query, candidate: BuilderInput, candidateFeatures: FeatureMap): BuilderOutput\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/icon/BaseHorizonIconBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.icon\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.HorizonIcon\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseHorizonIconBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Option[HorizonIcon]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/item/alert/BaseDurationBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert\n\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.util.Duration\n\ntrait BaseDurationBuilder[-Query <: PipelineQuery] {\n\n  def apply(query: Query, candidate: ShowAlertCandidate, features: FeatureMap): Option[Duration]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/item/alert/BaseShowAlertColorConfigurationBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert\n\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertColorConfiguration\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseShowAlertColorConfigurationBuilder[-Query <: PipelineQuery] {\n\n  def apply(\n    query: Query,\n    candidate: ShowAlertCandidate,\n    features: FeatureMap\n  ): ShowAlertColorConfiguration\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/item/alert/BaseShowAlertDisplayLocationBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert\n\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertDisplayLocation\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseShowAlertDisplayLocationBuilder[-Query <: PipelineQuery] {\n\n  def apply(\n    query: Query,\n    candidate: ShowAlertCandidate,\n    features: FeatureMap\n  ): ShowAlertDisplayLocation\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/item/alert/BaseShowAlertIconDisplayInfoBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert\n\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertIconDisplayInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseShowAlertIconDisplayInfoBuilder[-Query <: PipelineQuery] {\n\n  def apply(\n    query: Query,\n    candidate: ShowAlertCandidate,\n    features: FeatureMap\n  ): Option[ShowAlertIconDisplayInfo]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/item/alert/BaseShowAlertNavigationMetadataBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert\n\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertNavigationMetadata\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseShowAlertNavigationMetadataBuilder[-Query <: PipelineQuery] {\n\n  def apply(\n    query: Query,\n    candidate: ShowAlertCandidate,\n    features: FeatureMap\n  ): Option[ShowAlertNavigationMetadata]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/item/alert/BaseShowAlertUserIdsBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.alert\n\nimport com.twitter.product_mixer.component_library.model.candidate.ShowAlertCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseShowAlertUserIdsBuilder[-Query <: PipelineQuery] {\n\n  def apply(query: Query, candidate: ShowAlertCandidate, features: FeatureMap): Option[Seq[Long]]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/item/topic/BaseTopicDisplayTypeBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.topic\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicDisplayType\n\ntrait BaseTopicDisplayTypeBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: PipelineQuery,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[TopicDisplayType]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/item/topic/BaseTopicFunctionalityTypeBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.topic\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicFunctionalityType\n\ntrait BaseTopicFunctionalityTypeBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: PipelineQuery,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[TopicFunctionalityType]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/item/tweet/BaseEntryIdToReplaceBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.tweet\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseEntryIdToReplaceBuilder[-Query <: PipelineQuery, -Candidate <: BaseTweetCandidate] {\n\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[String]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/item/tweet/BaseTimelinesScoreInfoBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.tweet\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TimelinesScoreInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseTimelinesScoreInfoBuilder[-Query <: PipelineQuery, -Candidate <: BaseTweetCandidate] {\n\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[TimelinesScoreInfo]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/item/tweet/BaseTweetHighlightsBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.tweet\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseTweetCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetHighlights\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseTweetHighlightsBuilder[-Query <: PipelineQuery, -Candidate <: BaseTweetCandidate] {\n\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[TweetHighlights]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/item/user/BaseUserReactiveTriggersBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.item.user\n\nimport com.twitter.product_mixer.component_library.model.candidate.BaseUserCandidate\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.UserReactiveTriggers\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseUserReactiveTriggersBuilder[-Query <: PipelineQuery, -Candidate <: BaseUserCandidate] {\n\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[UserReactiveTriggers]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/metadata/BaseClientEventDetailsBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseClientEventDetailsBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  /**\n   * @return a [[ClientEventDetails]] for the provided [[Candidate]]\n   * @see [[ClientEventDetails]]\n   */\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[ClientEventDetails]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/metadata/BaseClientEventInfoBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseClientEventInfoBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  /**\n   * @return a [[ClientEventInfo]] for the provided [[Candidate]]\n   * @see [[ClientEventInfo]]\n   */\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap,\n    element: Option[String]\n  ): Option[ClientEventInfo]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/metadata/BaseFeedbackActionInfoBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseFeedbackActionInfoBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[FeedbackActionInfo]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/metadata/BaseModuleStr.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseModuleStr[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(query: Query, candidates: Seq[CandidateWithFeatures[Candidate]]): String\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/metadata/BaseStr.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseStr[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(query: Query, candidate: Candidate, candidateFeatures: FeatureMap): String\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/metadata/BaseUrlBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.metadata\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\n\ntrait BaseUrlBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(query: Query, candidate: Candidate, candidateFeatures: FeatureMap): Url\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/promoted/BasePromotedMetadataBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.promoted\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PromotedMetadata\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BasePromotedMetadataBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[PromotedMetadata]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/richtext/BaseRichTextBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.richtext\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseRichTextBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(query: Query, candidate: Candidate, candidateFeatures: FeatureMap): RichText\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/social_context/BaseModuleSocialContextBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseModuleSocialContextBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Option[SocialContext]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/social_context/BaseSocialContextBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.social_context\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseSocialContextBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: Query,\n    candidate: Candidate,\n    candidateFeatures: FeatureMap\n  ): Option[SocialContext]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/stringcenter/BaseModuleStringCenterPlaceholderBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.stringcenter\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseModuleStringCenterPlaceholderBuilder[\n  -Query <: PipelineQuery,\n  -Candidate <: UniversalNoun[Any]] {\n\n  def apply(query: Query, candidates: Seq[CandidateWithFeatures[Candidate]]): Map[String, Any]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/stringcenter/BaseStringCenterPlaceholderBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.stringcenter\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseStringCenterPlaceholderBuilder[\n  -Query <: PipelineQuery,\n  -Candidate <: UniversalNoun[Any]] {\n\n  def apply(query: Query, candidate: Candidate, candidateFeatures: FeatureMap): Map[String, Any]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/timeline_module/BaseModuleDisplayTypeBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleDisplayType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseModuleDisplayTypeBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): ModuleDisplayType\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/timeline_module/BaseModuleFooterBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleFooter\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseModuleFooterBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Option[ModuleFooter]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/timeline_module/BaseModuleHeaderBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleHeader\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseModuleHeaderBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Option[ModuleHeader]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/timeline_module/BaseModuleHeaderDisplayTypeBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleHeaderDisplayType\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseModuleHeaderDisplayTypeBuilder[\n  -Query <: PipelineQuery,\n  -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): ModuleHeaderDisplayType\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/timeline_module/BaseModuleMetadataBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleMetadata\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseModuleMetadataBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): ModuleMetadata\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/timeline_module/BaseModuleShowMoreBehaviorBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleShowMoreBehavior\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseModuleShowMoreBehaviorBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): ModuleShowMoreBehavior\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator/urt/builder/timeline_module/BaseTimelineModuleBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.decorator.urt.builder.timeline_module\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait BaseTimelineModuleBuilder[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]] {\n\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): TimelineModule\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/CandidateFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.feature_hydrator\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.SupportsConditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Hydrate features for a specific candidate\n * e.g. if the candidate is a Tweet then a feature could be whether it's is marked as sensitive\n *\n * @note if you want to conditionally run a [[BaseCandidateFeatureHydrator]] you can use the mixin [[com.twitter.product_mixer.core.model.common.Conditionally]]\n *       or to gate on a [[com.twitter.timelines.configapi.Param]] you can use\n *       [[com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedCandidateFeatureHydrator]] or\n *       [[com.twitter.product_mixer.component_library.feature_hydrator.candidate.param_gated.ParamGatedBulkCandidateFeatureHydrator]]\n */\nsealed trait BaseCandidateFeatureHydrator[\n  -Query <: PipelineQuery,\n  -Result <: UniversalNoun[Any],\n  FeatureType <: Feature[_, _]]\n    extends FeatureHydrator[FeatureType]\n    with SupportsConditionally[Query]\n\n/**\n * A candidate feature hydrator that provides an implementation for hydrating a single candidate\n * at the time. Product Mixer core takes care of hydrating all your candidates for you by\n * calling this for each candidate. This is useful for Stitch-powered downstream APIs (such\n * as Strato, Gizmoduck, etc) where the API takes a single candidate/key and Stitch handles\n * batching for you.\n *\n * @note Any exceptions that are thrown or returned as [[Stitch.exception]] will be added to the\n *       [[FeatureMap]] for *all* [[Feature]]s intended to be hydrated.\n *       Accessing a failed Feature will throw if using [[FeatureMap.get]] for Features that aren't\n *       [[com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure]]\n *\n * @tparam Query The query type\n * @tparam Result The Candidate type\n */\ntrait CandidateFeatureHydrator[-Query <: PipelineQuery, -Result <: UniversalNoun[Any]]\n    extends BaseCandidateFeatureHydrator[Query, Result, Feature[_, _]] {\n\n  override val identifier: FeatureHydratorIdentifier\n\n  /** Hydrates a [[FeatureMap]] for a single candidate */\n  def apply(query: Query, candidate: Result, existingFeatures: FeatureMap): Stitch[FeatureMap]\n}\n\n/**\n * Hydrate features for a list of candidates\n * e.g. for a list of Tweet candidates, a feature could be the visibility reason whether to show or not show each Tweet\n */\ntrait BaseBulkCandidateFeatureHydrator[\n  -Query <: PipelineQuery,\n  -Result <: UniversalNoun[Any],\n  FeatureType <: Feature[_, _]]\n    extends BaseCandidateFeatureHydrator[Query, Result, FeatureType] {\n\n  /**\n   * Hydrates a set of [[FeatureMap]]s for the bulk list of candidates. Every input candidate must\n   * have corresponding entry in the returned seq with a feature map.\n   */\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Result]]\n  ): Stitch[Seq[FeatureMap]]\n}\n\n/**\n * A candidate feature hydrator that allows a user to bulk hydrate features for all candidates\n * at once. This is useful for downstream APIs that take a list of candidates in one go such\n * as feature store or scorers.\n *\n * @note Any exceptions that are thrown or returned as [[Stitch.exception]] will be added to the\n *       [[FeatureMap]] for *all* [[Feature]]s of *all* candidates intended to be hydrated.\n *       An alternative to throwing an exception is per-candidate failure handling (e.g. adding\n *       a failed [[Feature]] with `addFailure`, a Try with `add`, or an optional value with `add`\n *       using [[FeatureMapBuilder]]).\n *       Accessing a failed Feature will throw if using [[FeatureMap.get]] for Features that aren't\n *       [[com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure]].\n *\n * @tparam Query The query type\n * @tparam Result The Candidate type\n */\ntrait BulkCandidateFeatureHydrator[-Query <: PipelineQuery, Candidate <: UniversalNoun[Any]]\n    extends BaseBulkCandidateFeatureHydrator[Query, Candidate, Feature[_, _]] {\n  override val identifier: FeatureHydratorIdentifier\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[Seq[FeatureMap]]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/FeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.feature_hydrator\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.model.common.Component\n\n/** Hydrates a [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]] for a given input */\ntrait FeatureHydrator[FeatureType <: Feature[_, _]] extends Component {\n  def features: Set[FeatureType]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/HydratorCandidateResult.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.feature_hydrator\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\ncase class HydratorCandidateResult[+Candidate <: UniversalNoun[Any]](\n  override val candidate: Candidate,\n  override val features: FeatureMap)\n    extends CandidateWithFeatures[Candidate]\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/QueryFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.feature_hydrator\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.SupportsConditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Hydrate features about the query itself (not about the candidates)\n * e.g. features about the user who is making the request, what country the request originated from, etc.\n *\n * @note [[BaseQueryFeatureHydrator]]s populate [[Feature]]s with last-write-wins semantics for\n *       duplicate [[Feature]]s, where the last hydrator to run that populates a [[Feature]] will\n *       override any previously run [[BaseQueryFeatureHydrator]]s values for that [[Feature]].\n *       In a [[com.twitter.product_mixer.core.pipeline.PipelineConfig PipelineConfig]] this means\n *       that the right-most [[BaseQueryFeatureHydrator]] to populate a given [[Feature]] will be\n *       the value that is available to use.\n *\n * @note if you want to conditionally run a [[BaseQueryFeatureHydrator]] you can use the mixin [[com.twitter.product_mixer.core.model.common.Conditionally]]\n *       or to gate on a [[com.twitter.timelines.configapi.Param]] you can use [[com.twitter.product_mixer.component_library.feature_hydrator.query.param_gated.ParamGatedQueryFeatureHydrator]]\n *\n * @note Any exceptions that are thrown or returned as [[Stitch.exception]] will be added to the\n *       [[FeatureMap]] for the [[Feature]]s that were supposed to be hydrated.\n *       Accessing a failed Feature will throw if using [[FeatureMap.get]] for Features that aren't\n *       [[com.twitter.product_mixer.core.feature.FeatureWithDefaultOnFailure]]\n */\ntrait BaseQueryFeatureHydrator[-Query <: PipelineQuery, FeatureType <: Feature[_, _]]\n    extends FeatureHydrator[FeatureType]\n    with SupportsConditionally[Query] {\n\n  override val identifier: FeatureHydratorIdentifier\n\n  /** Hydrates a [[FeatureMap]] for a given [[Query]] */\n  def hydrate(query: Query): Stitch[FeatureMap]\n}\n\ntrait QueryFeatureHydrator[-Query <: PipelineQuery]\n    extends BaseQueryFeatureHydrator[Query, Feature[_, _]]\n\n/**\n * When an [[AsyncHydrator]] is run it will hydrate features in the background\n * and will make them available starting at the specified point in execution.\n *\n * When `hydrateBefore` is reached, any duplicate [[Feature]]s that were already hydrated will be\n * overridden with the new value from the [[AsyncHydrator]]\n *\n * @note [[AsyncHydrator]]s have the same last-write-wins semantics for duplicate [[Feature]]s\n *       as [[BaseQueryFeatureHydrator]] but with some nuance. If [[AsyncHydrator]]s for the\n *       same [[Feature]] have the same `hydrateBefore` then the right-most [[AsyncHydrator]]s\n *       value takes precedence. Similarly, [[AsyncHydrator]]s always hydrate after any other\n *       [[BaseQueryFeatureHydrator]]. See the examples for more detail.\n * @example if [[QueryFeatureHydrator]]s that populate the same [[Feature]] are defined in a `PipelineConfig`\n *          such as `[ asyncHydratorForFeatureA, normalHydratorForFeatureA ]`, where `asyncHydratorForFeatureA`\n *          is an [[AsyncHydrator]], when `asyncHydratorForFeatureA` reaches it's `hydrateBefore`\n *          Step in the Pipeline, the value for `FeatureA` from the `asyncHydratorForFeatureA` will override\n *          the existing value from `normalHydratorForFeatureA`, even though in the initial `PipelineConfig`\n *          they are ordered differently.\n * @example if [[AsyncHydrator]]s that populate the same [[Feature]] are defined in a `PipelineConfig`\n *          such as `[ asyncHydratorForFeatureA1, asyncHydratorForFeatureA2 ]`, where both [[AsyncHydrator]]s\n *          have the same `hydrateBefore`, when `hydrateBefore` is reached, the value for `FeatureA` from\n *          `asyncHydratorForFeatureA2` will override the value from `asyncHydratorForFeatureA1`.\n */\ntrait AsyncHydrator {\n  _: BaseQueryFeatureHydrator[_, _] =>\n\n  /**\n   * A [[PipelineStepIdentifier]] from the [[com.twitter.product_mixer.core.pipeline.PipelineConfig]] this is used in\n   * by which the [[FeatureMap]] returned by this [[AsyncHydrator]] will be completed.\n   *\n   * Access to the [[Feature]]s from this [[AsyncHydrator]] prior to reaching the provided\n   * [[PipelineStepIdentifier]]s will result in a [[com.twitter.product_mixer.core.feature.featuremap.MissingFeatureException]].\n   *\n   * @note If [[PipelineStepIdentifier]] is a Step which is run in parallel, the [[Feature]]s will be available for all the parallel Steps.\n   */\n  def hydrateBefore: PipelineStepIdentifier\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/datarecord\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1/featurevalue\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"src/scala/com/twitter/ml/featurestore/lib/online\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1/featurevalue\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"src/scala/com/twitter/ml/featurestore/lib/online\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1/FeatureStoreDatasetErrorHandler.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1\n\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.data.DatasetErrorsById\nimport com.twitter.ml.featurestore.lib.data.HydrationError\nimport com.twitter.ml.featurestore.lib.dataset.DatasetId\nimport com.twitter.product_mixer.core.feature.featurestorev1.BaseFeatureStoreV1Feature\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject FeatureStoreDatasetErrorHandler {\n\n  /**\n   * This function takes a set of feature store features and constructs a mapping from the underlying\n   * feature store dataset back to the features. This is useful for looking up what ProMix features\n   * failed based off of a failed feature store dataset at request time. A ProMix feature can be\n   * powered by multiple feature store datasets, and conversely, a dataset can be used by many features.\n   */\n  def datasetToFeaturesMapping[\n    Query <: PipelineQuery,\n    Input,\n    FeatureType <: BaseFeatureStoreV1Feature[Query, Input, _ <: EntityId, _]\n  ](\n    features: Set[FeatureType]\n  ): Map[DatasetId, Set[FeatureType]] = {\n    val datasetsAndFeatures: Set[(DatasetId, FeatureType)] = features\n      .flatMap { feature: FeatureType =>\n        feature.boundFeatureSet.sourceDatasets.map(_.id).map { datasetId: DatasetId =>\n          datasetId -> feature\n        }\n      }\n\n    datasetsAndFeatures\n      .groupBy { case (datasetId, _) => datasetId }.mapValues(_.map {\n        case (_, feature) => feature\n      })\n  }\n\n  /**\n   * This takes a mapping of Feature Store Dataset => ProMix Features, as well as the dataset errors\n   * from PredictionRecord and computing a final, deduped mapping from ProMix Feature to Exceptions.\n   */\n  def featureToHydrationErrors[\n    Query <: PipelineQuery,\n    Input,\n    FeatureType <: BaseFeatureStoreV1Feature[Query, Input, _ <: EntityId, _]\n  ](\n    datasetToFeatures: Map[DatasetId, Set[\n      FeatureType\n    ]],\n    errorsByDatasetId: DatasetErrorsById\n  ): Map[FeatureType, Set[HydrationError]] = {\n    val hasError = errorsByDatasetId.datasets.nonEmpty\n    if (hasError) {\n      val featuresAndErrors: Set[(FeatureType, Set[HydrationError])] = errorsByDatasetId.datasets\n        .flatMap { id: DatasetId =>\n          val errors: Set[HydrationError] = errorsByDatasetId.get(id).values.toSet\n          if (errors.nonEmpty) {\n            val datasetFeatures: Set[FeatureType] = datasetToFeatures.getOrElse(id, Set.empty)\n            datasetFeatures.map { feature =>\n              feature -> errors\n            }.toSeq\n          } else {\n            Seq.empty\n          }\n        }\n      featuresAndErrors\n        .groupBy { case (feature, _) => feature }.mapValues(_.flatMap {\n          case (_, errors: Set[HydrationError]) => errors\n        })\n    } else {\n      Map.empty\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1/FeatureStoreV1CandidateFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1\n\nimport com.twitter.ml.api.util.SRichDataRecord\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.data.PredictionRecordAdapter\nimport com.twitter.ml.featurestore.lib.entity.EntityWithId\nimport com.twitter.ml.featurestore.lib.online.FeatureStoreRequest\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.feature.featurestorev1.BaseFeatureStoreV1CandidateFeature\nimport com.twitter.product_mixer.core.feature.featurestorev1.FeatureStoreV1CandidateEntity\nimport com.twitter.product_mixer.core.feature.featurestorev1.featurevalue.FeatureStoreV1Response\nimport com.twitter.product_mixer.core.feature.featurestorev1.featurevalue.FeatureStoreV1ResponseFeature\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseBulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.FeatureHydrationFailed\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.logging.Logging\n\ntrait FeatureStoreV1CandidateFeatureHydrator[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any]]\n    extends BaseBulkCandidateFeatureHydrator[\n      Query,\n      Candidate,\n      BaseFeatureStoreV1CandidateFeature[Query, Candidate, _ <: EntityId, _]\n    ]\n    with Logging {\n\n  override def features: Set[BaseFeatureStoreV1CandidateFeature[Query, Candidate, _ <: EntityId, _]]\n\n  def clientBuilder: FeatureStoreV1DynamicClientBuilder\n\n  private lazy val hydrationConfig = FeatureStoreV1CandidateFeatureHydrationConfig(features)\n\n  private lazy val client = clientBuilder.build(hydrationConfig)\n\n  private lazy val datasetToFeatures =\n    FeatureStoreDatasetErrorHandler.datasetToFeaturesMapping(features)\n\n  private lazy val dataRecordAdapter =\n    PredictionRecordAdapter.oneToOne(hydrationConfig.allBoundFeatures)\n\n  private lazy val featureContext = hydrationConfig.allBoundFeatures.toFeatureContext\n\n  override def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[Seq[FeatureMap]] = {\n    // Duplicate entities are expected across features, so de-dupe via the Set before converting to Seq\n    val entities: Seq[FeatureStoreV1CandidateEntity[Query, Candidate, _ <: EntityId]] =\n      features.map(_.entity).toSeq\n\n    val featureStoreRequests = candidates.map { candidate =>\n      val candidateEntityIds: Seq[EntityWithId[_ <: EntityId]] =\n        entities.map(_.entityWithId(query, candidate.candidate, candidate.features))\n\n      FeatureStoreRequest(entityIds = candidateEntityIds)\n    }\n\n    val featureMaps = client(featureStoreRequests, query).map { predictionRecords =>\n      if (predictionRecords.size == candidates.size)\n        predictionRecords\n          .zip(candidates).map {\n            case (predictionRecord, candidate) =>\n              val datasetErrors = predictionRecord.getDatasetHydrationErrors\n              val errorMap =\n                FeatureStoreDatasetErrorHandler.featureToHydrationErrors(\n                  datasetToFeatures,\n                  datasetErrors)\n\n              if (errorMap.nonEmpty) {\n                logger.debug(() =>\n                  s\"$identifier hydration errors for candidate ${candidate.candidate.id}: $errorMap\")\n              }\n              val dataRecord =\n                new SRichDataRecord(\n                  dataRecordAdapter.adaptToDataRecord(predictionRecord),\n                  featureContext)\n              val featureStoreResponse =\n                FeatureStoreV1Response(dataRecord, errorMap)\n              FeatureMapBuilder()\n                .add(FeatureStoreV1ResponseFeature, featureStoreResponse).build()\n          }\n      else\n        // Should not happen as FSv1 is guaranteed to return a prediction record per feature store request\n        throw PipelineFailure(\n          FeatureHydrationFailed,\n          \"Unexpected response length from Feature Store V1 while hydrating candidate features\")\n    }\n\n    Stitch.callFuture(featureMaps)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1/FeatureStoreV1DynamicClientBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1\n\nimport com.twitter.ml.featurestore.lib.dynamic.BaseDynamicHydrationConfig\nimport com.twitter.ml.featurestore.lib.dynamic.BaseGatedFeatures\nimport com.twitter.ml.featurestore.lib.dynamic.DynamicFeatureStoreClient\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait FeatureStoreV1DynamicClientBuilder {\n  def build[Query <: PipelineQuery](\n    dynamicHydrationConfig: BaseDynamicHydrationConfig[Query, _ <: BaseGatedFeatures[Query]]\n  ): DynamicFeatureStoreClient[Query]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1/FeatureStoreV1HydrationConfig.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1\n\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.dynamic.BaseDynamicHydrationConfig\nimport com.twitter.product_mixer.core.feature.featurestorev1.BaseFeatureStoreV1CandidateFeature\nimport com.twitter.product_mixer.core.feature.featurestorev1.BaseFeatureStoreV1QueryFeature\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ncase class FeatureStoreV1QueryFeatureHydrationConfig[Query <: PipelineQuery](\n  features: Set[BaseFeatureStoreV1QueryFeature[Query, _ <: EntityId, _]])\n    extends BaseDynamicHydrationConfig[\n      Query,\n      BaseFeatureStoreV1QueryFeature[Query, _ <: EntityId, _]\n    ](features)\n\ncase class FeatureStoreV1CandidateFeatureHydrationConfig[\n  Query <: PipelineQuery,\n  Input <: UniversalNoun[Any]\n](\n  features: Set[BaseFeatureStoreV1CandidateFeature[Query, Input, _ <: EntityId, _]])\n    extends BaseDynamicHydrationConfig[\n      Query,\n      BaseFeatureStoreV1CandidateFeature[Query, Input, _ <: EntityId, _]\n    ](features)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1/FeatureStoreV1QueryFeatureHydrator.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1\n\nimport com.twitter.ml.api.util.SRichDataRecord\nimport com.twitter.ml.featurestore.lib.EntityId\nimport com.twitter.ml.featurestore.lib.data.PredictionRecordAdapter\nimport com.twitter.ml.featurestore.lib.entity.EntityWithId\nimport com.twitter.ml.featurestore.lib.online.FeatureStoreRequest\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.feature.featurestorev1.BaseFeatureStoreV1QueryFeature\nimport com.twitter.product_mixer.core.feature.featurestorev1.FeatureStoreV1QueryEntity\nimport com.twitter.product_mixer.core.feature.featurestorev1.featurevalue.FeatureStoreV1Response\nimport com.twitter.product_mixer.core.feature.featurestorev1.featurevalue.FeatureStoreV1ResponseFeature\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseQueryFeatureHydrator\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.FeatureHydrationFailed\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.logging.Logging\n\ntrait FeatureStoreV1QueryFeatureHydrator[Query <: PipelineQuery]\n    extends BaseQueryFeatureHydrator[\n      Query,\n      BaseFeatureStoreV1QueryFeature[Query, _ <: EntityId, _]\n    ]\n    with Logging {\n\n  def features: Set[BaseFeatureStoreV1QueryFeature[Query, _ <: EntityId, _]]\n\n  def clientBuilder: FeatureStoreV1DynamicClientBuilder\n\n  private lazy val hydrationConfig = FeatureStoreV1QueryFeatureHydrationConfig(features)\n\n  private lazy val client = clientBuilder.build(hydrationConfig)\n\n  private lazy val datasetToFeatures =\n    FeatureStoreDatasetErrorHandler.datasetToFeaturesMapping(features)\n\n  private lazy val dataRecordAdapter =\n    PredictionRecordAdapter.oneToOne(hydrationConfig.allBoundFeatures)\n\n  private lazy val featureContext = hydrationConfig.allBoundFeatures.toFeatureContext\n\n  override def hydrate(\n    query: Query\n  ): Stitch[FeatureMap] = {\n    // Duplicate entities are expected across features, so de-dupe via the Set before converting to Seq\n    val entities: Seq[FeatureStoreV1QueryEntity[Query, _ <: EntityId]] =\n      features.map(_.entity).toSeq\n    val entityIds: Seq[EntityWithId[_ <: EntityId]] = entities.map(_.entityWithId(query))\n\n    val featureStoreRequest = Seq(FeatureStoreRequest(entityIds = entityIds))\n\n    val featureMap = client(featureStoreRequest, query).map { predictionRecords =>\n      // Should not happen as FSv1 is guaranteed to return a prediction record per feature store request\n      val predictionRecord = predictionRecords.headOption.getOrElse {\n        throw PipelineFailure(\n          FeatureHydrationFailed,\n          \"Unexpected empty response from Feature Store V1 while hydrating query features\")\n      }\n\n      val datasetErrors = predictionRecord.getDatasetHydrationErrors\n      val errorMap =\n        FeatureStoreDatasetErrorHandler.featureToHydrationErrors(datasetToFeatures, datasetErrors)\n\n      if (errorMap.nonEmpty) {\n        logger.debug(() => s\"$identifier hydration errors for query: $errorMap\")\n      }\n\n      val richDataRecord =\n        SRichDataRecord(dataRecordAdapter.adaptToDataRecord(predictionRecord), featureContext)\n      val featureStoreResponse =\n        FeatureStoreV1Response(richDataRecord, errorMap)\n      FeatureMapBuilder().add(FeatureStoreV1ResponseFeature, featureStoreResponse).build()\n    }\n\n    Stitch.callFuture(featureMap)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter/Filter.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.filter\n\nimport com.twitter.product_mixer.core.functional_component.filter.Filter.SupportsConditionally\nimport com.twitter.product_mixer.core.model.common\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/**\n * Takes a sequence of candidates and can filter some out\n *\n * @note if you want to conditionally run a [[Filter]] you can use the mixin [[Filter.Conditionally]]\n *       or to gate on a [[com.twitter.timelines.configapi.Param]] you can use [[com.twitter.product_mixer.component_library.filter.ParamGatedFilter]]\n *\n * @tparam Query The domain model for the query or request\n * @tparam Candidate The type of the candidates\n */\ntrait Filter[-Query <: PipelineQuery, Candidate <: UniversalNoun[Any]]\n    extends Component\n    with SupportsConditionally[Query, Candidate] {\n\n  /** @see [[FilterIdentifier]] */\n  override val identifier: FilterIdentifier\n\n  /**\n   * Filter the list of candidates\n   *\n   * @return a FilterResult including both the list of kept candidate and the list of removed candidates\n   */\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[FilterResult[Candidate]]\n}\n\nobject Filter {\n\n  /**\n   * Mixin for when you want to conditionally run a [[Filter]]\n   *\n   * This is a thin wrapper around [[common.Conditionally]] exposing a nicer API for the [[Filter]] specific use-case.\n   */\n  trait Conditionally[-Query <: PipelineQuery, Candidate <: UniversalNoun[Any]]\n      extends common.Conditionally[Input[Query, Candidate]] { _: Filter[Query, Candidate] =>\n\n    /** @see [[common.Conditionally.onlyIf]] */\n    def onlyIf(\n      query: Query,\n      candidates: Seq[CandidateWithFeatures[Candidate]]\n    ): Boolean\n\n    override final def onlyIf(input: Input[Query, Candidate]): Boolean =\n      onlyIf(input.query, input.candidates)\n  }\n\n  /** Type alias to obscure [[Filter.Input]] from customers */\n  type SupportsConditionally[-Query <: PipelineQuery, Candidate <: UniversalNoun[Any]] =\n    common.SupportsConditionally[Input[Query, Candidate]]\n\n  /** A case class representing the input arguments to a [[Filter]], mostly for internal use */\n  case class Input[+Query <: PipelineQuery, +Candidate <: UniversalNoun[Any]](\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]])\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter/FilterResult.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.filter\n\n/** `Candidate`s were `kept` and `removed` by a [[Filter]] */\ncase class FilterResult[+Candidate](kept: Seq[Candidate], removed: Seq[Candidate])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate/Gate.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.gate\n\nimport com.twitter.product_mixer.core.functional_component.gate.Gate.SkippedResult\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.CandidatePipelineResults\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Stitch\n\n/**\n * A gate controls if a pipeline or other component is executed\n *\n * A gate is mostly controlled by it's `shouldContinue` function - when this function\n * returns true, execution Continues.\n *\n * Gates also have a optional `shouldSkip`- When it returns\n * true, then we Continue without executing `main`.\n *\n * @tparam Query The query type that the gate will receive as input\n *\n * @return A GateResult includes both the boolean `continue` and a specific reason. See [[GateResult]] for more\n *         information.\n */\n\nsealed trait BaseGate[-Query <: PipelineQuery] extends Component {\n  override val identifier: GateIdentifier\n\n  /**\n   * If a shouldSkip returns true, the gate returns a Skip(continue=true) without executing\n   * the main predicate. We expect this to be useful for debugging, dogfooding, etc.\n   */\n  def shouldSkip(query: Query): Stitch[Boolean] = Stitch.False\n\n  /**\n   * The main predicate that controls this gate. If this predicate returns true, the gate returns Continue.\n   */\n  def shouldContinue(query: Query): Stitch[Boolean]\n\n  /** returns a [[GateResult]] to determine whether a pipeline should be executed based on `t` */\n  final def apply(t: Query): Stitch[GateResult] = {\n    shouldSkip(t).flatMap { skipResult =>\n      if (skipResult) {\n        SkippedResult\n      } else {\n        shouldContinue(t).map { mainResult =>\n          if (mainResult) GateResult.Continue else GateResult.Stop\n        }\n      }\n    }\n  }\n\n  /** Arrow representation of `this` [[Gate]] */\n  final def arrow: Arrow[Query, GateResult] = Arrow(apply)\n}\n\n/**\n * A regular Gate which only has access to the Query typed PipelineQuery. This can be used anywhere\n * Gates are available.\n *\n * A gate is mostly controlled by it's `shouldContinue` function - when this function\n * returns true, execution Continues.\n *\n * Gates also have a optional `shouldSkip`- When it returns\n * true, then we Continue without executing `main`.\n * @tparam Query The query type that the gate will receive as input\n *\n * @return A GateResult includes both the boolean `continue` and a specific reason. See [[GateResult]] for more\n *         information.\n */\ntrait Gate[-Query <: PipelineQuery] extends BaseGate[Query]\n\n/**\n * A Query And Candidate Gate which only has access both to the Query typed PipelineQuery and the\n * list of previously fetched candidates. This can be used on dependent candidate pipelines to\n * make a decision on whether to enable/disable them based on previous candidates.\n *\n * A gate is mostly controlled by it's `shouldContinue` function - when this function\n * returns true, execution Continues.\n *\n * Gates also have a optional `shouldSkip`- When it returns\n * true, then we Continue without executing `main`.\n *\n * @tparam Query The query type that the gate will receive as input\n *\n * @return A GateResult includes both the boolean `continue` and a specific reason. See [[GateResult]] for more\n *         information.\n */\ntrait QueryAndCandidateGate[-Query <: PipelineQuery] extends BaseGate[Query] {\n\n  /**\n   * If a shouldSkip returns true, the gate returns a Skip(continue=true) without executing\n   * the main predicate. We expect this to be useful for debugging, dogfooding, etc.\n   */\n  def shouldSkip(query: Query, candidates: Seq[CandidateWithDetails]): Stitch[Boolean] =\n    Stitch.False\n\n  /**\n   * The main predicate that controls this gate. If this predicate returns true, the gate returns Continue.\n   */\n  def shouldContinue(query: Query, candidates: Seq[CandidateWithDetails]): Stitch[Boolean]\n\n  final override def shouldSkip(query: Query): Stitch[Boolean] = {\n    val candidates = query.features\n      .map(_.get(CandidatePipelineResults)).getOrElse(\n        throw PipelineFailure(\n          IllegalStateFailure,\n          \"Candidate Pipeline Results Feature missing from query features\"))\n    shouldSkip(query, candidates)\n  }\n\n  final override def shouldContinue(query: Query): Stitch[Boolean] = {\n    val candidates = query.features\n      .map(_.get(CandidatePipelineResults)).getOrElse(\n        throw PipelineFailure(\n          IllegalStateFailure,\n          \"Candidate Pipeline Results Feature missing from query features\"))\n    shouldContinue(query, candidates)\n  }\n}\n\nobject Gate {\n  val SkippedResult: Stitch[GateResult] = Stitch.value(GateResult.Skipped)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate/GateResult.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.gate\n\n/**\n * A [[Gate]] controls if a pipeline or other component is executed.\n *\n * Application logic should usually use `GateResult.continue: Boolean` to interpret a GateResult. `continue` will be\n * true if we should continue with execution, and false if we should stop.\n *\n * You can case match against the `GateResult` to understand how exactly execution happened. See `object GateResult`\n * below, but this is useful if you want to know if we are continuing due to the skip or main predicates.\n */\nsealed trait GateResult {\n\n  /** Should we continue? */\n  val continue: Boolean\n}\n\nobject GateResult {\n\n  /**\n   * Continue Execution\n   *\n   * the Skip predicate evaluated to true,\n   * so we Skipped execution of the Main predicate and should continue\n   */\n  case object Skipped extends GateResult {\n    override val continue = true\n  }\n\n  /**\n   * Continue Execution\n   *\n   * the main predicate evaluated to true\n   */\n  case object Continue extends GateResult {\n    override val continue = true\n  }\n\n  /**\n   * Stop execution\n   *\n   * the main predicate evaluated to false\n   */\n  case object Stop extends GateResult {\n    override val continue = false\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate/ShouldContinue.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.gate\n\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait ShouldContinue[Query <: PipelineQuery] {\n  def apply(query: Query): Boolean\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/TransportMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller\n\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.identifier.TransportMarshallerIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\n\nobject TransportMarshaller {\n\n  /** Avoid `malformed class name` exceptions due to the presence of the `$` character */\n  def getSimpleName[T](c: Class[T]): String = {\n    c.getName.lastIndexOf(\"$\") match {\n      case -1 => c.getSimpleName\n      case index => c.getName.substring(index + 1)\n    }\n  }\n}\n\n/**\n * Marshals a [[MarshallerInput]] into a type that can be sent over the wire\n *\n * This transformation should be mechanical and not contain business logic\n *\n * @note this is different from `com.twitter.product_mixer.core.functional_component.premarshaller`\n *       which can contain business logic.\n */\ntrait TransportMarshaller[-MarshallerInput <: HasMarshalling, +MarshallerOutput] extends Component {\n\n  override val identifier: TransportMarshallerIdentifier\n\n  def apply(input: MarshallerInput): MarshallerOutput\n}\n\n/**\n * No op marshalling that passes through a [[HasMarshalling]] into any type. This is useful if\n * the response does not need to be sent over the wire, such as with a\n * [[com.twitter.product_mixer.core.functional_component.candidate_source.product_pipeline.ProductPipelineCandidateSource]]\n */\nobject NoOpTransportMarshaller extends TransportMarshaller[HasMarshalling, Any] {\n  override val identifier: TransportMarshallerIdentifier = TransportMarshallerIdentifier(\"NoOp\")\n\n  override def apply(input: HasMarshalling): Any = input\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/request/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/request/ClientContextMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.request\n\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.product_mixer.core.{thriftscala => t}\n\nobject ClientContextMarshaller {\n\n  def apply(clientContext: ClientContext): t.ClientContext = {\n    t.ClientContext(\n      userId = clientContext.userId,\n      guestId = clientContext.guestId,\n      appId = clientContext.appId,\n      ipAddress = clientContext.ipAddress,\n      userAgent = clientContext.userAgent,\n      countryCode = clientContext.countryCode,\n      languageCode = clientContext.languageCode,\n      isTwoffice = clientContext.isTwoffice,\n      userRoles = clientContext.userRoles,\n      deviceId = clientContext.deviceId,\n      mobileDeviceId = clientContext.mobileDeviceId,\n      mobileDeviceAdId = clientContext.mobileDeviceAdId,\n      limitAdTracking = clientContext.limitAdTracking,\n      guestIdAds = clientContext.guestIdAds,\n      guestIdMarketing = clientContext.guestIdMarketing\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/request/ClientContextUnmarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.request\n\nimport com.twitter.product_mixer.core.model.marshalling.request.ClientContext\nimport com.twitter.product_mixer.core.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ClientContextUnmarshaller @Inject() () {\n\n  def apply(clientContext: t.ClientContext): ClientContext = {\n    ClientContext(\n      userId = clientContext.userId,\n      guestId = clientContext.guestId,\n      guestIdAds = clientContext.guestIdAds,\n      guestIdMarketing = clientContext.guestIdMarketing,\n      appId = clientContext.appId,\n      ipAddress = clientContext.ipAddress,\n      userAgent = clientContext.userAgent,\n      countryCode = clientContext.countryCode,\n      languageCode = clientContext.languageCode,\n      isTwoffice = clientContext.isTwoffice,\n      userRoles = clientContext.userRoles.map(_.toSet),\n      deviceId = clientContext.deviceId,\n      mobileDeviceId = clientContext.mobileDeviceId,\n      mobileDeviceAdId = clientContext.mobileDeviceAdId,\n      limitAdTracking = clientContext.limitAdTracking\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/request/FeatureValueUnmarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.request\n\nimport com.twitter.product_mixer.core.{thriftscala => t}\nimport com.twitter.timelines.configapi.BooleanFeatureValue\nimport com.twitter.timelines.configapi.FeatureValue\nimport com.twitter.timelines.configapi.NumberFeatureValue\nimport com.twitter.timelines.configapi.StringFeatureValue\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FeatureValueUnmarshaller @Inject() () {\n\n  def apply(featureValue: t.FeatureValue): FeatureValue = featureValue match {\n    case t.FeatureValue.PrimitiveValue(t.PrimitiveFeatureValue.BoolValue(bool)) =>\n      BooleanFeatureValue(bool)\n    case t.FeatureValue.PrimitiveValue(t.PrimitiveFeatureValue.StrValue(string)) =>\n      StringFeatureValue(string)\n    case t.FeatureValue.PrimitiveValue(t.PrimitiveFeatureValue.IntValue(int)) =>\n      NumberFeatureValue(int)\n    case t.FeatureValue.PrimitiveValue(t.PrimitiveFeatureValue.LongValue(long)) =>\n      NumberFeatureValue(long)\n    case t.FeatureValue.PrimitiveValue(t.PrimitiveFeatureValue.DoubleValue(double)) =>\n      NumberFeatureValue(double)\n    case t.FeatureValue.PrimitiveValue(t.PrimitiveFeatureValue.UnknownUnionField(field)) =>\n      throw new UnsupportedOperationException(\n        s\"Unknown feature value primitive: ${field.field.name}\")\n    case t.FeatureValue.UnknownUnionField(field) =>\n      throw new UnsupportedOperationException(s\"Unknown feature value: ${field.field.name}\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/graphql/contextual_ref/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/rtf/safety_level\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/rtf/safety_level\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/contextual_ref\",\n        \"src/thrift/com/twitter/spam/rtf:safety-level-scala\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql/contextual_refs:graphql-refs-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/rtf/safety_level\",\n        \"src/thrift/com/twitter/spam/rtf:safety-level-scala\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql/contextual_refs:graphql-refs-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/graphql/contextual_ref/ContextualTweetRefMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.graphql.contextual_ref\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.ContextualTweetRef\nimport com.twitter.strato.graphql.contextual_refs.{thriftscala => thrift}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ContextualTweetRefMarshaller @Inject() (\n  tweetHydrationContextMarshaller: TweetHydrationContextMarshaller) {\n\n  def apply(contextualTweetRef: ContextualTweetRef): thrift.ContextualTweetRef =\n    thrift.ContextualTweetRef(\n      id = contextualTweetRef.id,\n      hydrationContext =\n        contextualTweetRef.hydrationContext.map(tweetHydrationContextMarshaller(_)))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/graphql/contextual_ref/OuterTweetContextMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.graphql.contextual_ref\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.OuterTweetContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.QuoteTweetId\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.RetweetId\nimport com.twitter.strato.graphql.contextual_refs.{thriftscala => thrift}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass OuterTweetContextMarshaller @Inject() () {\n\n  def apply(outerTweetContext: OuterTweetContext): thrift.OuterTweetContext =\n    outerTweetContext match {\n      case QuoteTweetId(id) => thrift.OuterTweetContext.QuoteTweetId(id)\n      case RetweetId(id) => thrift.OuterTweetContext.RetweetId(id)\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/graphql/contextual_ref/TweetHydrationContextMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.graphql.contextual_ref\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.rtf.safety_level.SafetyLevelMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.TweetHydrationContext\nimport com.twitter.strato.graphql.contextual_refs.{thriftscala => thrift}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetHydrationContextMarshaller @Inject() (\n  safetyLevelMarshaller: SafetyLevelMarshaller,\n  outerTweetContextMarshaller: OuterTweetContextMarshaller) {\n\n  def apply(tweetHydrationContext: TweetHydrationContext): thrift.TweetHydrationContext =\n    thrift.TweetHydrationContext(\n      safetyLevelOverride = tweetHydrationContext.safetyLevelOverride.map(safetyLevelMarshaller(_)),\n      outerTweetContext =\n        tweetHydrationContext.outerTweetContext.map(outerTweetContextMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/rtf/safety_level/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/rtf/safety_level\",\n        \"src/thrift/com/twitter/spam/rtf:safety-level-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/rtf/safety_level\",\n        \"src/thrift/com/twitter/spam/rtf:safety-level-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/rtf/safety_level/SafetyLevelMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.rtf.safety_level\n\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.ConversationFocalTweetSafetyLevel\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.ConversationInjectedTweetSafetyLevel\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.ConversationReplySafetyLevel\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.SafetyLevel\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.TimelineFocalTweetSafetyLevel\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.TimelineHomePromotedHydrationSafetyLevel\nimport com.twitter.spam.rtf.{thriftscala => thrift}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SafetyLevelMarshaller @Inject() () {\n\n  def apply(safetyLevel: SafetyLevel): thrift.SafetyLevel = safetyLevel match {\n    case ConversationFocalTweetSafetyLevel => thrift.SafetyLevel.ConversationFocalTweet\n    case ConversationReplySafetyLevel => thrift.SafetyLevel.ConversationReply\n    case ConversationInjectedTweetSafetyLevel => thrift.SafetyLevel.ConversationInjectedTweet\n    case TimelineFocalTweetSafetyLevel => thrift.SafetyLevel.TimelineFocalTweet\n    case TimelineHomePromotedHydrationSafetyLevel =>\n      thrift.SafetyLevel.TimelineHomePromotedHydration\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/slice/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/thrift/com/twitter/product_mixer/component_library:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql:api-media-graphql-scala\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql:graphql-scala\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql:topics-graphql-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/slice/CursorTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.slice\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.product_mixer.component_library.{thriftscala => t}\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.NextCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.PreviousCursor\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.CursorType\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.GapCursor\n\n@Singleton\nclass CursorTypeMarshaller @Inject() () {\n\n  def apply(cursorType: CursorType): t.CursorType = cursorType match {\n    case NextCursor => t.CursorType.Next\n    case PreviousCursor => t.CursorType.Previous\n    case GapCursor => t.CursorType.Gap\n  }\n\n  def unmarshall(cursorType: t.CursorType): CursorType = cursorType match {\n    case t.CursorType.Next => NextCursor\n    case t.CursorType.Previous => PreviousCursor\n    case t.CursorType.Gap => GapCursor\n    case t.CursorType.EnumUnknownCursorType(id) =>\n      throw new UnsupportedOperationException(\n        s\"Attempted to unmarshall unrecognized cursor type: $id\")\n  }\n\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/slice/SliceItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.slice\n\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.AdType\nimport com.twitter.product_mixer.core.model.marshalling.response.slice\nimport com.twitter.strato.graphql.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SliceItemMarshaller @Inject() () {\n  def apply(item: slice.SliceItem): t.SliceItem = {\n    item match {\n      case item: slice.TweetItem =>\n        t.SliceItem.TweetItem(t.TweetItem(id = item.id))\n      case item: slice.UserItem =>\n        t.SliceItem.UserItem(t.UserItem(id = item.id))\n      case item: slice.TwitterListItem =>\n        t.SliceItem.TwitterListItem(t.TwitterListItem(id = item.id))\n      case item: slice.DMConvoSearchItem =>\n        t.SliceItem.DmConvoSearchItem(t.DMConvoSearchItem(id = item.id))\n      case item: slice.DMConvoItem =>\n        t.SliceItem.DmConvoItem(t.DMConvoItem(id = item.id))\n      case item: slice.DMEventItem =>\n        t.SliceItem.DmEventItem(t.DMEventItem(id = item.id))\n      case item: slice.DMMessageSearchItem =>\n        t.SliceItem.DmMessageSearchItem(t.DMMessageSearchItem(id = item.id))\n      case item: slice.TopicItem =>\n        t.SliceItem.TopicItem(t.TopicItem(id = item.id.toString))\n      case item: slice.TypeaheadEventItem =>\n        t.SliceItem.TypeaheadEventItem(\n          t.TypeaheadEventItem(\n            eventId = item.eventId,\n            metadata = item.metadata.map(marshalTypeaheadMetadata)\n          )\n        )\n      case item: slice.TypeaheadQuerySuggestionItem =>\n        t.SliceItem.TypeaheadQuerySuggestionItem(\n          t.TypeaheadQuerySuggestionItem(\n            query = item.query,\n            metadata = item.metadata.map(marshalTypeaheadMetadata)\n          )\n        )\n      case item: slice.TypeaheadUserItem =>\n        t.SliceItem.TypeaheadUserItem(\n          t.TypeaheadUserItem(\n            userId = item.userId,\n            metadata = item.metadata.map(marshalTypeaheadMetadata),\n            badges = Some(item.badges.map { badge =>\n              t.UserBadge(\n                badgeUrl = badge.badgeUrl,\n                description = Some(badge.description),\n                badgeType = Some(badge.badgeType))\n            })\n          )\n        )\n      case item: slice.AdItem =>\n        t.SliceItem.AdItem(\n          t.AdItem(\n            adKey = t.AdKey(\n              adAccountId = item.adAccountId,\n              adUnitId = item.adUnitId,\n            )\n          )\n        )\n      case item: slice.AdCreativeItem =>\n        t.SliceItem.AdCreativeItem(\n          t.AdCreativeItem(\n            adCreativeKey = t.AdCreativeKey(\n              adAccountId = item.adAccountId,\n              adType = marshalAdType(item.adType),\n              creativeId = item.creativeId\n            )\n          )\n        )\n      case item: slice.AdGroupItem =>\n        t.SliceItem.AdGroupItem(\n          t.AdGroupItem(\n            adGroupKey = t.AdGroupKey(\n              adAccountId = item.adAccountId,\n              adGroupId = item.adGroupId\n            )\n          )\n        )\n      case item: slice.CampaignItem =>\n        t.SliceItem.CampaignItem(\n          t.CampaignItem(\n            campaignKey = t.CampaignKey(\n              adAccountId = item.adAccountId,\n              campaignId = item.campaignId\n            )\n          )\n        )\n      case item: slice.FundingSourceItem =>\n        t.SliceItem.FundingSourceItem(\n          t.FundingSourceItem(\n            fundingSourceKey = t.FundingSourceKey(\n              adAccountId = item.adAccountId,\n              fundingSourceId = item.fundingSourceId\n            )\n          )\n        )\n    }\n  }\n\n  private def marshalTypeaheadMetadata(metadata: slice.TypeaheadMetadata) = {\n    t.TypeaheadMetadata(\n      score = metadata.score,\n      source = metadata.source,\n      resultContext = metadata.context.map(context =>\n        t.TypeaheadResultContext(\n          displayString = context.displayString,\n          contextType = marshalRequestContextType(context.contextType),\n          iconUrl = context.iconUrl\n        ))\n    )\n  }\n\n  private def marshalRequestContextType(\n    context: slice.TypeaheadResultContextType\n  ): t.TypeaheadResultContextType = {\n    context match {\n      case slice.You => t.TypeaheadResultContextType.You\n      case slice.Location => t.TypeaheadResultContextType.Location\n      case slice.NumFollowers => t.TypeaheadResultContextType.NumFollowers\n      case slice.FollowRelationship => t.TypeaheadResultContextType.FollowRelationship\n      case slice.Bio => t.TypeaheadResultContextType.Bio\n      case slice.NumTweets => t.TypeaheadResultContextType.NumTweets\n      case slice.Trending => t.TypeaheadResultContextType.Trending\n      case slice.HighlightedLabel => t.TypeaheadResultContextType.HighlightedLabel\n      case _ => t.TypeaheadResultContextType.Undefined\n    }\n  }\n\n  private def marshalAdType(\n    adType: AdType\n  ): t.AdType = {\n    adType match {\n      case AdType.Tweet => t.AdType.Tweet\n      case AdType.Account => t.AdType.Account\n      case AdType.InStreamVideo => t.AdType.InStreamVideo\n      case AdType.DisplayCreative => t.AdType.DisplayCreative\n      case AdType.Trend => t.AdType.Trend\n      case AdType.Spotlight => t.AdType.Spotlight\n      case AdType.Takeover => t.AdType.Takeover\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/slice/SliceTransportMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.slice\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.model.common.identifier.TransportMarshallerIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.Slice\nimport com.twitter.strato.graphql.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SliceTransportMarshaller @Inject() (sliceItemMarshaller: SliceItemMarshaller)\n    extends TransportMarshaller[Slice, t.SliceResult] {\n\n  override val identifier: TransportMarshallerIdentifier = TransportMarshallerIdentifier(\"Slice\")\n\n  override def apply(slice: Slice): t.SliceResult = {\n    t.SliceResult.Slice(\n      t.Slice(\n        items = slice.items.map(sliceItemMarshaller(_)),\n        sliceInfo = t.SliceInfo(\n          previousCursor = slice.sliceInfo.previousCursor,\n          nextCursor = slice.sliceInfo.nextCursor\n        )\n      ))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp\",\n        \"src/thrift/com/twitter/pages/render:thrift-scala\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql/timelines:graphql-timelines-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp\",\n        \"src/thrift/com/twitter/pages/render:thrift-scala\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql/timelines:graphql-timelines-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/PageBodyMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.pages.render.{thriftscala => urp}\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PageBody\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.SegmentedTimelinesPageBody\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TimelineKeyPageBody\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PageBodyMarshaller @Inject() (\n  timelineKeyMarshaller: TimelineKeyMarshaller,\n  segmentedTimelinesMarshaller: SegmentedTimelinesMarshaller) {\n\n  def apply(pageBody: PageBody): urp.PageBody = pageBody match {\n    case pageBody: TimelineKeyPageBody =>\n      urp.PageBody.Timeline(timelineKeyMarshaller(pageBody.timeline))\n    case pageBody: SegmentedTimelinesPageBody =>\n      urp.PageBody.SegmentedTimelines(segmentedTimelinesMarshaller(pageBody))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/PageHeaderMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.pages.render.{thriftscala => urp}\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PageHeader\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TopicPageHeader\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PageHeaderMarshaller @Inject() (\n  topicPageHeaderMarshaller: TopicPageHeaderMarshaller) {\n\n  def apply(pageHeader: PageHeader): urp.PageHeader = pageHeader match {\n    case pageHeader: TopicPageHeader =>\n      urp.PageHeader.TopicPageHeader(topicPageHeaderMarshaller(pageHeader))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/PageNavBarMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.pages.render.{thriftscala => urp}\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PageNavBar\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TopicPageNavBar\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TitleNavBar\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PageNavBarMarshaller @Inject() (\n  topicPageNavBarMarshaller: TopicPageNavBarMarshaller,\n  titleNavBarMarshaller: TitleNavBarMarshaller) {\n\n  def apply(pageNavBar: PageNavBar): urp.PageNavBar = pageNavBar match {\n    case pageNavBar: TopicPageNavBar =>\n      urp.PageNavBar.TopicPageNavBar(topicPageNavBarMarshaller(pageNavBar))\n    case pageNavBar: TitleNavBar =>\n      urp.PageNavBar.TitleNavBar(titleNavBarMarshaller(pageNavBar))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/SegmentedTimelineMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.pages.render.{thriftscala => urp}\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.TimelineScribeConfigMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.SegmentedTimeline\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SegmentedTimelineMarshaller @Inject() (\n  timelineKeyMarshaller: TimelineKeyMarshaller,\n  timelineScribeConfigMarshaller: TimelineScribeConfigMarshaller) {\n\n  def apply(segmentedTimeline: SegmentedTimeline): urp.SegmentedTimeline = urp.SegmentedTimeline(\n    id = segmentedTimeline.id,\n    labelText = segmentedTimeline.labelText,\n    timeline = timelineKeyMarshaller(segmentedTimeline.timeline),\n    scribeConfig = segmentedTimeline.scribeConfig.map(timelineScribeConfigMarshaller(_)),\n    refreshIntervalSec = segmentedTimeline.refreshIntervalSec\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/SegmentedTimelinesMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.pages.render.{thriftscala => urp}\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.SegmentedTimelinesPageBody\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SegmentedTimelinesMarshaller @Inject() (\n  segmentedTimelineMarshaller: SegmentedTimelineMarshaller) {\n\n  def apply(segmentedTimelinesPageBody: SegmentedTimelinesPageBody): urp.SegmentedTimelines =\n    urp.SegmentedTimelines(\n      initialTimeline = segmentedTimelineMarshaller(segmentedTimelinesPageBody.initialTimeline),\n      timelines = segmentedTimelinesPageBody.timelines.map(segmentedTimelineMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/TimelineKeyMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.FollowedTopicsMeTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.FollowedTopicsOtherTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.ForYouExploreMixerTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.NoteworthyAccountsTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.NotInterestedTopicsMeTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.NuxForYouCategoryUserRecommendationsTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.NuxGeoCategoryUserRecommendationsTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.NuxPymkCategoryUserRecommendationsTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.NuxSingleInterestCategoryUserRecommendationsTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.NuxUserRecommendationsTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.ShoppingHomeTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TimelineKey\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TopicsLandingTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TopicsPickerTimeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TrendingExploreMixerTimeline\nimport com.twitter.strato.graphql.timelines.{thriftscala => graphql}\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelineKeyMarshaller {\n\n  def apply(timelineKey: TimelineKey): graphql.TimelineKey = timelineKey match {\n    case TopicsLandingTimeline(topicId) =>\n      graphql.TimelineKey.TopicTimeline(graphql.TopicId(topicId))\n    case NoteworthyAccountsTimeline(topicId) =>\n      graphql.TimelineKey.NoteworthyAccountsTimeline(graphql.TopicId(topicId))\n    case TopicsPickerTimeline(topicId) =>\n      graphql.TimelineKey.TopicsPickerTimeline(graphql.TopicId(topicId))\n    case FollowedTopicsMeTimeline() =>\n      graphql.TimelineKey.FollowedTopicsMeTimeline(graphql.Void())\n    case NotInterestedTopicsMeTimeline() =>\n      graphql.TimelineKey.NotInterestedTopicsMeTimeline(graphql.Void())\n    case FollowedTopicsOtherTimeline(userId) =>\n      graphql.TimelineKey.FollowedTopicsOtherTimeline(userId)\n    case NuxUserRecommendationsTimeline() =>\n      graphql.TimelineKey.NuxUserRecommendationsTimeline(graphql.Void())\n    case NuxForYouCategoryUserRecommendationsTimeline() =>\n      graphql.TimelineKey.NuxForYouCategoryUserRecommendationsTimeline(graphql.Void())\n    case NuxPymkCategoryUserRecommendationsTimeline() =>\n      graphql.TimelineKey.NuxPymkCategoryUserRecommendationsTimeline(graphql.Void())\n    case NuxGeoCategoryUserRecommendationsTimeline() =>\n      graphql.TimelineKey.NuxGeoCategoryUserRecommendationsTimeline(graphql.Void())\n    case NuxSingleInterestCategoryUserRecommendationsTimeline(topicId) =>\n      graphql.TimelineKey.NuxSingleInterestCategoryUserRecommendationsTimeline(\n        graphql.TopicId(topicId))\n    case ShoppingHomeTimeline() => graphql.TimelineKey.ShoppingHome(graphql.Void())\n    case ForYouExploreMixerTimeline() =>\n      graphql.TimelineKey.ForYouExploreMixerTimeline(graphql.Void())\n    case TrendingExploreMixerTimeline() =>\n      graphql.TimelineKey.TrendingExploreMixerTimeline(graphql.Void())\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/TitleNavBarMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.pages.render.{thriftscala => urp}\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventInfoMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TitleNavBar\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TitleNavBarMarshaller @Inject() (\n  clientEventInfoMarshaller: ClientEventInfoMarshaller) {\n\n  def apply(titleNavBar: TitleNavBar): urp.TitleNavBar =\n    urp.TitleNavBar(\n      title = titleNavBar.title,\n      subtitle = titleNavBar.subtitle,\n      clientEventInfo = titleNavBar.clientEventInfo.map(clientEventInfoMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/TopicPageHeaderDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.BasicTopicPageHeaderDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.PersonalizedTopicPageHeaderDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TopicPageHeaderDisplayType\nimport com.twitter.pages.render.{thriftscala => urp}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TopicPageHeaderDisplayTypeMarshaller @Inject() () {\n\n  def apply(\n    topicPageHeaderDisplayType: TopicPageHeaderDisplayType\n  ): urp.TopicPageHeaderDisplayType = topicPageHeaderDisplayType match {\n    case BasicTopicPageHeaderDisplayType => urp.TopicPageHeaderDisplayType.Basic\n    case PersonalizedTopicPageHeaderDisplayType => urp.TopicPageHeaderDisplayType.Personalized\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/TopicPageHeaderFacepileMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.pages.render.{thriftscala => urp}\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TopicPageHeaderFacepile\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TopicPageHeaderFacepileMarshaller @Inject() (\n  urlMarshaller: UrlMarshaller) {\n\n  def apply(topicPageHeaderFacepile: TopicPageHeaderFacepile): urp.TopicPageHeaderFacepile =\n    urp.TopicPageHeaderFacepile(\n      userIds = topicPageHeaderFacepile.userIds,\n      facepileUrl = topicPageHeaderFacepile.facepileUrl.map(urlMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/TopicPageHeaderMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.pages.render.{thriftscala => urp}\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventInfoMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TopicPageHeader\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TopicPageHeaderMarshaller @Inject() (\n  topicPageHeaderFacepileMarshaller: TopicPageHeaderFacepileMarshaller,\n  clientEventInfoMarshaller: ClientEventInfoMarshaller,\n  topicPageHeaderDisplayTypeMarshaller: TopicPageHeaderDisplayTypeMarshaller) {\n\n  def apply(topicPageHeader: TopicPageHeader): urp.TopicPageHeader =\n    urp.TopicPageHeader(\n      topicId = topicPageHeader.topicId,\n      facepile = topicPageHeader.facepile.map(topicPageHeaderFacepileMarshaller(_)),\n      clientEventInfo = topicPageHeader.clientEventInfo.map(clientEventInfoMarshaller(_)),\n      landingContext = topicPageHeader.landingContext,\n      displayType = topicPageHeader.displayType\n        .map(topicPageHeaderDisplayTypeMarshaller(_)).getOrElse(\n          urp.TopicPageHeaderDisplayType.Basic)\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/TopicPageNavBarMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.pages.render.{thriftscala => urp}\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventInfoMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.TopicPageNavBar\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TopicPageNavBarMarshaller @Inject() (\n  clientEventInfoMarshaller: ClientEventInfoMarshaller) {\n\n  def apply(topicPageNavBar: TopicPageNavBar): urp.TopicPageNavBar =\n    urp.TopicPageNavBar(\n      topicId = topicPageNavBar.topicId,\n      clientEventInfo = topicPageNavBar.clientEventInfo.map(clientEventInfoMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/UrpTransportMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.pages.render.{thriftscala => urp}\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.TimelineScribeConfigMarshaller\nimport com.twitter.product_mixer.core.model.common.identifier.TransportMarshallerIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urp.Page\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UrpTransportMarshaller @Inject() (\n  pageBodyMarshaller: PageBodyMarshaller,\n  timelineScribeConfigMarshaller: TimelineScribeConfigMarshaller,\n  pageHeaderMarshaller: PageHeaderMarshaller,\n  pageNavBarMarshaller: PageNavBarMarshaller)\n    extends TransportMarshaller[Page, urp.Page] {\n\n  override val identifier: TransportMarshallerIdentifier =\n    TransportMarshallerIdentifier(\"UnifiedRichPage\")\n\n  override def apply(page: Page): urp.Page = urp.Page(\n    id = page.id,\n    pageBody = pageBodyMarshaller(page.pageBody),\n    scribeConfig = page.scribeConfig.map(timelineScribeConfigMarshaller(_)),\n    pageHeader = page.pageHeader.map(pageHeaderMarshaller(_)),\n    pageNavBar = page.pageNavBar.map(pageNavBarMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp/UrpTransportMarshallerBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urp\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.TimelineScribeConfigMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ArticleDetailsMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventDetailsMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.CommerceDetailsMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ConversationDetailsMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ConversationSectionMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.LiveEventDetailsMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.TimelinesDetailsMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrtEndpointOptionsMarshaller\n\nobject UrpTransportMarshallerBuilder {\n  // Convenience constructor for services not using dependency injection and unit tests. If using\n  // dependency injection, instead @Inject an instance of UrpTransportMarshaller to construct.\n\n  val timelineKeyMarshaller = new TimelineKeyMarshaller\n  val timelineScribeConfigMarshaller = new TimelineScribeConfigMarshaller\n  val urlMarshaller = new UrlMarshaller(new UrlTypeMarshaller, new UrtEndpointOptionsMarshaller)\n  val clientEventInfoMarshaller = new ClientEventInfoMarshaller(\n    new ClientEventDetailsMarshaller(\n      new ConversationDetailsMarshaller(new ConversationSectionMarshaller),\n      new TimelinesDetailsMarshaller,\n      new ArticleDetailsMarshaller,\n      new LiveEventDetailsMarshaller,\n      new CommerceDetailsMarshaller)\n  )\n\n  val segmentedTimelineMarshaller =\n    new SegmentedTimelineMarshaller(timelineKeyMarshaller, timelineScribeConfigMarshaller)\n  val segmentedTimelinesMarshaller = new SegmentedTimelinesMarshaller(segmentedTimelineMarshaller)\n\n  val pageBodyMarshaller: PageBodyMarshaller = new PageBodyMarshaller(\n    timelineKeyMarshaller,\n    segmentedTimelinesMarshaller\n  )\n\n  val topicPageHeaderFacepileMarshaller = new TopicPageHeaderFacepileMarshaller(urlMarshaller)\n  val topicPageHeaderDisplayTypeMarshaller = new TopicPageHeaderDisplayTypeMarshaller\n  val topicPageHeaderMarshaller = new TopicPageHeaderMarshaller(\n    topicPageHeaderFacepileMarshaller,\n    clientEventInfoMarshaller,\n    topicPageHeaderDisplayTypeMarshaller\n  )\n  val pageHeaderMarshaller: PageHeaderMarshaller = new PageHeaderMarshaller(\n    topicPageHeaderMarshaller)\n\n  val topicPageNavBarMarshaller = new TopicPageNavBarMarshaller(clientEventInfoMarshaller)\n  val titleNavBarMarshaller = new TitleNavBarMarshaller(clientEventInfoMarshaller)\n  val pageNavBarMarshaller: PageNavBarMarshaller = new PageNavBarMarshaller(\n    topicPageNavBarMarshaller,\n    titleNavBarMarshaller\n  )\n\n  val marshaller: UrpTransportMarshaller =\n    new UrpTransportMarshaller(\n      pageBodyMarshaller = pageBodyMarshaller,\n      timelineScribeConfigMarshaller = timelineScribeConfigMarshaller,\n      pageHeaderMarshaller = pageHeaderMarshaller,\n      pageNavBarMarshaller = pageNavBarMarshaller\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/AddEntriesInstructionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AddEntriesInstructionMarshaller @Inject() (\n  timelineEntryMarshaller: TimelineEntryMarshaller) {\n\n  def apply(instruction: AddEntriesTimelineInstruction): urt.AddEntries = urt.AddEntries(\n    entries = instruction.entries.map(timelineEntryMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/AddToModuleInstructionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddToModuleTimelineInstruction\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AddToModuleInstructionMarshaller @Inject() (moduleItemMarshaller: ModuleItemMarshaller) {\n\n  def apply(instruction: AddToModuleTimelineInstruction): urt.AddToModule = urt.AddToModule(\n    moduleItems = instruction.moduleItems.map(moduleItemMarshaller(_, instruction.moduleEntryId)),\n    moduleEntryId = instruction.moduleEntryId,\n    moduleItemEntryId = instruction.moduleItemEntryId,\n    prepend = instruction.prepend\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/graphql/contextual_ref\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/button\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/color\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/cover\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/media\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/operation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"src/thrift/com/twitter/spam/rtf:safety-level-scala\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql/contextual_refs:graphql-refs-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/color\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/cover\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/media\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/operation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"src/thrift/com/twitter/spam/rtf:safety-level-scala\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql/contextual_refs:graphql-refs-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/CoverMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover.CoverContentMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventInfoMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Cover\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.FullCover\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.HalfCover\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CoverMarshaller @Inject() (\n  coverContentMarshaller: CoverContentMarshaller,\n  clientEventInfoMarshaller: ClientEventInfoMarshaller) {\n\n  def apply(cover: Cover): urt.ShowCover = cover match {\n    case halfCover: HalfCover =>\n      urt.ShowCover(\n        cover = coverContentMarshaller(halfCover.content),\n        clientEventInfo = cover.clientEventInfo.map(clientEventInfoMarshaller(_)))\n    case fullCover: FullCover =>\n      urt.ShowCover(\n        cover = coverContentMarshaller(fullCover.content),\n        clientEventInfo = cover.clientEventInfo.map(clientEventInfoMarshaller(_)))\n  }\n}\n\nclass UnsupportedTimelineCoverException(cover: Cover)\n    extends UnsupportedOperationException(\n      \"Unsupported timeline cover \" + TransportMarshaller.getSimpleName(cover.getClass))\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/MarkEntriesUnreadInstructionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.MarkEntriesUnreadInstruction\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.timelines.render.{thriftscala => urt}\n\n@Singleton\nclass MarkEntriesUnreadInstructionMarshaller @Inject() () {\n\n  def apply(instruction: MarkEntriesUnreadInstruction): urt.MarkEntriesUnread =\n    urt.MarkEntriesUnread(entryIds = instruction.entryIds)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/ModuleItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ModuleItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ModuleItemMarshaller @Inject() (\n  timelineItemMarshaller: TimelineItemMarshaller,\n  moduleItemTreeDisplayMarshaller: ModuleItemTreeDisplayMarshaller) {\n\n  def apply(moduleItem: ModuleItem, moduleEntryId: String): urt.ModuleItem = urt.ModuleItem(\n    /* Module items have an identifier comprising both the module entry id and the module item id.\n    Some URT clients will deduplicate globally across different modules.\n    Using the entry id as a prefix ensures that deduplication only happens within a single module,\n    which we believe better aligns with engineers' intentions. */\n    entryId = moduleEntryId + \"-\" + moduleItem.item.entryIdentifier,\n    item = timelineItemMarshaller(moduleItem.item),\n    dispensable = moduleItem.dispensable,\n    treeDisplay = moduleItem.treeDisplay.map(moduleItemTreeDisplayMarshaller.apply)\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/ModuleItemTreeDisplayMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module.ModuleDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ModuleItemTreeDisplay\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ModuleItemTreeDisplayMarshaller @Inject() (\n  moduleDisplayTypeMarshaller: ModuleDisplayTypeMarshaller) {\n\n  def apply(moduleItemTreeDisplay: ModuleItemTreeDisplay): urt.ModuleItemTreeDisplay =\n    urt.ModuleItemTreeDisplay(\n      parentModuleItemEntryId = moduleItemTreeDisplay.parentModuleEntryItemId,\n      indentFromParent = moduleItemTreeDisplay.indentFromParent,\n      displayType = moduleItemTreeDisplay.displayType.map(moduleDisplayTypeMarshaller(_)),\n      isAnchorChild = moduleItemTreeDisplay.isAnchorChild\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/PinEntryInstructionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.PinEntryTimelineInstruction\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PinEntryInstructionMarshaller @Inject() (\n  timelineEntryMarshaller: TimelineEntryMarshaller) {\n\n  def apply(instruction: PinEntryTimelineInstruction): urt.PinEntry = {\n    urt.PinEntry(entry = timelineEntryMarshaller(instruction.entry))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/ReaderModeConfigMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ReaderModeConfig\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ReaderModeConfigMarshaller @Inject() (urlMarshaller: UrlMarshaller) {\n\n  def apply(readerModeConfig: ReaderModeConfig): urt.ReaderModeConfig = urt.ReaderModeConfig(\n    isReaderModeAvailable = readerModeConfig.isReaderModeAvailable,\n    landingUrl = urlMarshaller(readerModeConfig.landingUrl)\n  )\n\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/ReplaceEntryInstructionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ReplaceEntryTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ReplaceEntryInstructionMarshaller @Inject() (\n  timelineEntryMarshaller: TimelineEntryMarshaller) {\n\n  def apply(instruction: ReplaceEntryTimelineInstruction): urt.ReplaceEntry = {\n    val instructionEntry = instruction.entry\n    urt.ReplaceEntry(\n      entryIdToReplace = instructionEntry.entryIdToReplace\n        .getOrElse(throw new MissingEntryToReplaceException(instructionEntry)),\n      entry = timelineEntryMarshaller(instructionEntry)\n    )\n  }\n}\n\nclass MissingEntryToReplaceException(entry: TimelineEntry)\n    extends IllegalArgumentException(\n      s\"Missing entry ID to replace ${TransportMarshaller.getSimpleName(entry.getClass)}\")\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/ShowAlertInstructionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert.ShowAlertColorConfigurationMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert.ShowAlertDisplayLocationMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert.ShowAlertIconDisplayInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert.ShowAlertNavigationMetadataMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert.ShowAlertTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ShowAlertInstruction\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ShowAlertInstructionMarshaller @Inject() (\n  showAlertTypeMarshaller: ShowAlertTypeMarshaller,\n  clientEventInfoMarshaller: ClientEventInfoMarshaller,\n  richTextMarshaller: RichTextMarshaller,\n  showAlertIconDisplayInfoMarshaller: ShowAlertIconDisplayInfoMarshaller,\n  showAlertColorConfigurationMarshaller: ShowAlertColorConfigurationMarshaller,\n  showAlertDisplayLocationMarshaller: ShowAlertDisplayLocationMarshaller,\n  showAlertNavigationMetadataMarshaller: ShowAlertNavigationMetadataMarshaller,\n) {\n\n  def apply(instruction: ShowAlertInstruction): urt.ShowAlert = urt.ShowAlert(\n    alertType = showAlertTypeMarshaller(instruction.showAlert.alertType),\n    triggerDelayMs = instruction.showAlert.triggerDelay.map(_.inMillis.toInt),\n    displayDurationMs = instruction.showAlert.displayDuration.map(_.inMillis.toInt),\n    clientEventInfo = instruction.showAlert.clientEventInfo.map(clientEventInfoMarshaller(_)),\n    collapseDelayMs = instruction.showAlert.collapseDelay.map(_.inMillis.toInt),\n    userIds = instruction.showAlert.userIds,\n    richText = instruction.showAlert.richText.map(richTextMarshaller(_)),\n    iconDisplayInfo =\n      instruction.showAlert.iconDisplayInfo.map(showAlertIconDisplayInfoMarshaller(_)),\n    colorConfig = showAlertColorConfigurationMarshaller(instruction.showAlert.colorConfig),\n    displayLocation = showAlertDisplayLocationMarshaller(instruction.showAlert.displayLocation),\n    navigationMetadata =\n      instruction.showAlert.navigationMetadata.map(showAlertNavigationMetadataMarshaller(_)),\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/TerminateTimelineInstructionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.BottomTermination\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TerminateTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TopTermination\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TopAndBottomTermination\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TerminateTimelineInstructionMarshaller @Inject() () {\n\n  def apply(instruction: TerminateTimelineInstruction): urt.TerminateTimeline =\n    urt.TerminateTimeline(\n      direction = instruction.terminateTimelineDirection match {\n        case TopTermination => urt.TimelineTerminationDirection.Top\n        case BottomTermination => urt.TimelineTerminationDirection.Bottom\n        case TopAndBottomTermination => urt.TimelineTerminationDirection.TopAndBottom\n      }\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/TimelineEntryContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineOperation\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelineEntryContentMarshaller @Inject() (\n  timelineItemMarshaller: TimelineItemMarshaller,\n  timelineModuleMarshaller: TimelineModuleMarshaller,\n  timelineOperationMarshaller: TimelineOperationMarshaller) {\n\n  def apply(entry: TimelineEntry): urt.TimelineEntryContent = entry match {\n    case item: TimelineItem =>\n      urt.TimelineEntryContent.Item(timelineItemMarshaller(item))\n    case module: TimelineModule =>\n      urt.TimelineEntryContent.TimelineModule(timelineModuleMarshaller(module))\n    case operation: TimelineOperation =>\n      urt.TimelineEntryContent.Operation(timelineOperationMarshaller(operation))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/TimelineEntryMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelineEntryMarshaller @Inject() (\n  timelineEntryContentMarshaller: TimelineEntryContentMarshaller) {\n\n  def apply(entry: TimelineEntry): urt.TimelineEntry =\n    urt.TimelineEntry(\n      entryId = entry.entryIdentifier,\n      sortIndex = entry.sortIndex.getOrElse(throw new TimelineEntryMissingSortIndexException),\n      content = timelineEntryContentMarshaller(entry),\n      expiryTime = entry.expirationTimeInMillis\n    )\n}\n\nclass TimelineEntryMissingSortIndexException\n    extends UnsupportedOperationException(\"Timeline entry missing sort index\")\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/TimelineInstructionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddEntriesTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.AddToModuleTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ClearCacheTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.MarkEntriesUnreadInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.PinEntryTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ReplaceEntryTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ShowAlertInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ShowCoverInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TerminateTimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineInstruction\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelineInstructionMarshaller @Inject() (\n  addEntriesInstructionMarshaller: AddEntriesInstructionMarshaller,\n  addToModuleInstructionMarshaller: AddToModuleInstructionMarshaller,\n  markEntriesUnreadInstructionMarshaller: MarkEntriesUnreadInstructionMarshaller,\n  pinEntryInstructionMarshaller: PinEntryInstructionMarshaller,\n  replaceEntryInstructionMarshaller: ReplaceEntryInstructionMarshaller,\n  showAlertInstructionMarshaller: ShowAlertInstructionMarshaller,\n  terminateTimelineInstructionMarshaller: TerminateTimelineInstructionMarshaller,\n  coverMarshaller: CoverMarshaller) {\n\n  def apply(instruction: TimelineInstruction): urt.TimelineInstruction = instruction match {\n    case instruction: AddEntriesTimelineInstruction =>\n      urt.TimelineInstruction.AddEntries(addEntriesInstructionMarshaller(instruction))\n    case instruction: AddToModuleTimelineInstruction =>\n      urt.TimelineInstruction.AddToModule(addToModuleInstructionMarshaller(instruction))\n    case _: ClearCacheTimelineInstruction =>\n      urt.TimelineInstruction.ClearCache(urt.ClearCache())\n    case instruction: MarkEntriesUnreadInstruction =>\n      urt.TimelineInstruction.MarkEntriesUnread(\n        markEntriesUnreadInstructionMarshaller(instruction)\n      )\n    case instruction: PinEntryTimelineInstruction =>\n      urt.TimelineInstruction.PinEntry(pinEntryInstructionMarshaller(instruction))\n    case instruction: ReplaceEntryTimelineInstruction =>\n      urt.TimelineInstruction.ReplaceEntry(replaceEntryInstructionMarshaller(instruction))\n    case instruction: ShowCoverInstruction =>\n      urt.TimelineInstruction.ShowCover(coverMarshaller(instruction.cover))\n    case instruction: ShowAlertInstruction =>\n      urt.TimelineInstruction.ShowAlert(showAlertInstructionMarshaller(instruction))\n    case instruction: TerminateTimelineInstruction =>\n      urt.TimelineInstruction.TerminateTimeline(terminateTimelineInstructionMarshaller(instruction))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/TimelineItemContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.commerce.CommerceProductGroupItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.commerce.CommerceProductItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.article.ArticleItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.audio_space.AudioSpaceItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.card.CardItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.event.EventSummaryItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.generic_summary_item.GenericSummaryItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.icon_label.IconLabelItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.label.LabelItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message.MessagePromptItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.moment.MomentAnnotationItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.prompt.PromptItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.suggestion.SpellingItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.thread.ThreadHeaderItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tile.TileItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tombstone.TombstoneItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.topic.TopicFollowPromptItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.topic.TopicItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.trend.TrendItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet.TweetItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet_composer.TweetComposerItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.twitter_list.TwitterListItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.user.UserItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.vertical_grid_item.VerticalGridItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.operation.CursorItemMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Cover\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.ArticleItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.audio_space.AudioSpaceItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.card.CardItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.commerce.CommerceProductGroupItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.commerce.CommerceProductItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.event.EventSummaryItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary.GenericSummaryItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.icon_label.IconLabelItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.label.LabelItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessagePromptItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.moment.MomentAnnotationItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.PromptItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.suggestion.SpellingItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.thread.ThreadHeaderItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile.TileItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone.TombstoneItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicFollowPromptItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.trend.TrendItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet_composer.TweetComposerItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.twitter_list.TwitterListItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.UserItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.VerticalGridItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelineItemContentMarshaller @Inject() (\n  articleItemMarshaller: ArticleItemMarshaller,\n  audioSpaceItemMarshaller: AudioSpaceItemMarshaller,\n  cardItemMarshaller: CardItemMarshaller,\n  cursorItemMarshaller: CursorItemMarshaller,\n  eventSummaryItemMarshaller: EventSummaryItemMarshaller,\n  iconLabelItemMarshaller: IconLabelItemMarshaller,\n  labelItemMarshaller: LabelItemMarshaller,\n  messagePromptItemMarshaller: MessagePromptItemMarshaller,\n  tileItemMarshaller: TileItemMarshaller,\n  tombstoneItemMarshaller: TombstoneItemMarshaller,\n  topicFollowPromptItemMarshaller: TopicFollowPromptItemMarshaller,\n  topicItemMarshaller: TopicItemMarshaller,\n  tweetComposerItemMarshaller: TweetComposerItemMarshaller,\n  tweetItemMarshaller: TweetItemMarshaller,\n  twitterListItemMarshaller: TwitterListItemMarshaller,\n  userItemMarshaller: UserItemMarshaller,\n  verticalGridItemMarshaller: VerticalGridItemMarshaller,\n  threadHeaderItemMarshaller: ThreadHeaderItemMarshaller,\n  promptItemMarshaller: PromptItemMarshaller,\n  spellingItemMarshaller: SpellingItemMarshaller,\n  momentAnnotationItemMarshaller: MomentAnnotationItemMarshaller,\n  genericSummaryItemMarshaller: GenericSummaryItemMarshaller,\n  commerceProductItemMarshaller: CommerceProductItemMarshaller,\n  commerceProductGroupItemMarshaller: CommerceProductGroupItemMarshaller,\n  trendItemMarshaller: TrendItemMarshaller) {\n\n  def apply(item: TimelineItem): urt.TimelineItemContent = item match {\n    case articleItem: ArticleItem => articleItemMarshaller(articleItem)\n    case audioSpaceItem: AudioSpaceItem => audioSpaceItemMarshaller(audioSpaceItem)\n    case cardItem: CardItem => cardItemMarshaller(cardItem)\n    case cursorItem: CursorItem => cursorItemMarshaller(cursorItem)\n    case eventSummaryItem: EventSummaryItem => eventSummaryItemMarshaller(eventSummaryItem)\n    case genericSummaryItem: GenericSummaryItem => genericSummaryItemMarshaller(genericSummaryItem)\n    case iconLabelItem: IconLabelItem => iconLabelItemMarshaller(iconLabelItem)\n    case labelItem: LabelItem => labelItemMarshaller(labelItem)\n    case messagePromptItem: MessagePromptItem => messagePromptItemMarshaller(messagePromptItem)\n    case tileItem: TileItem => tileItemMarshaller(tileItem)\n    case tombstoneItem: TombstoneItem => tombstoneItemMarshaller(tombstoneItem)\n    case topicFollowPromptItem: TopicFollowPromptItem =>\n      topicFollowPromptItemMarshaller(topicFollowPromptItem)\n    case topicItem: TopicItem => topicItemMarshaller(topicItem)\n    case tweetComposerItem: TweetComposerItem => tweetComposerItemMarshaller(tweetComposerItem)\n    case tweetItem: TweetItem => tweetItemMarshaller(tweetItem)\n    case twitterListItem: TwitterListItem => twitterListItemMarshaller(twitterListItem)\n    case userItem: UserItem => userItemMarshaller(userItem)\n    case verticalGridItem: VerticalGridItem => verticalGridItemMarshaller(verticalGridItem)\n    case threadHeaderItem: ThreadHeaderItem => threadHeaderItemMarshaller(threadHeaderItem)\n    case promptItem: PromptItem => promptItemMarshaller(promptItem)\n    case spellingItem: SpellingItem => spellingItemMarshaller(spellingItem)\n    case momentAnnotationItem: MomentAnnotationItem =>\n      momentAnnotationItemMarshaller(momentAnnotationItem)\n    case commerceProductItem: CommerceProductItem =>\n      commerceProductItemMarshaller(commerceProductItem)\n    case commerceProductGroupItem: CommerceProductGroupItem =>\n      commerceProductGroupItemMarshaller(commerceProductGroupItem)\n    case trendItem: TrendItem => trendItemMarshaller(trendItem)\n    case _: Cover => throw TimelineCoverNotFilteredException\n    case _ => throw new UnsupportedTimelineItemException(item)\n  }\n}\n\nclass UnsupportedTimelineItemException(timelineItem: TimelineItem)\n    extends UnsupportedOperationException(\n      \"Unsupported timeline item \" + TransportMarshaller.getSimpleName(timelineItem.getClass))\n\nobject TimelineCoverNotFilteredException\n    extends UnsupportedOperationException(\"AddEntriesInstructionBuilder does not support Cover. \" +\n      \"ShowCoverInstructionBuilder should be used with AddEntriesWithShowCoverInstructionBuilder \" +\n      \"in order to filter out the Cover.\")\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/TimelineItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.FeedbackInfoMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelineItemMarshaller @Inject() (\n  timelineItemContentMarshaller: TimelineItemContentMarshaller,\n  clientEventInfoMarshaller: ClientEventInfoMarshaller,\n  feedbackInfoMarshaller: FeedbackInfoMarshaller) {\n\n  def apply(item: TimelineItem): urt.TimelineItem = urt.TimelineItem(\n    content = timelineItemContentMarshaller(item),\n    clientEventInfo = item.clientEventInfo.map(clientEventInfoMarshaller(_)),\n    feedbackInfo = item.feedbackActionInfo.map(feedbackInfoMarshaller(_)),\n    prompt = None\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/TimelineMetadataMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineMetadata\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelineMetadataMarshaller @Inject() (\n  timelineScribeConfigMarshaller: TimelineScribeConfigMarshaller,\n  readerModeConfigMarshaller: ReaderModeConfigMarshaller) {\n\n  def apply(timelineMetadata: TimelineMetadata): urt.TimelineMetadata = urt.TimelineMetadata(\n    title = timelineMetadata.title,\n    scribeConfig = timelineMetadata.scribeConfig.map(timelineScribeConfigMarshaller(_)),\n    readerModeConfig = timelineMetadata.readerModeConfig.map(readerModeConfigMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/TimelineModuleMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.FeedbackInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module.ModuleDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module.ModuleFooterMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module.ModuleHeaderMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module.ModuleMetadataMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module.ModuleShowMoreBehaviorMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelineModuleMarshaller @Inject() (\n  moduleItemMarshaller: ModuleItemMarshaller,\n  moduleDisplayTypeMarshaller: ModuleDisplayTypeMarshaller,\n  moduleHeaderMarshaller: ModuleHeaderMarshaller,\n  moduleFooterMarshaller: ModuleFooterMarshaller,\n  clientEventInfoMarshaller: ClientEventInfoMarshaller,\n  feedbackInfoMarshaller: FeedbackInfoMarshaller,\n  moduleMetadataMarshaller: ModuleMetadataMarshaller,\n  moduleShowMoreBehaviorMarshaller: ModuleShowMoreBehaviorMarshaller) {\n\n  def apply(timelineModule: TimelineModule): urt.TimelineModule = urt.TimelineModule(\n    items = timelineModule.items.map(moduleItemMarshaller(_, timelineModule.entryIdentifier)),\n    displayType = moduleDisplayTypeMarshaller(timelineModule.displayType),\n    header = timelineModule.header.map(moduleHeaderMarshaller(_)),\n    footer = timelineModule.footer.map(moduleFooterMarshaller(_)),\n    clientEventInfo = timelineModule.clientEventInfo.map(clientEventInfoMarshaller(_)),\n    feedbackInfo = timelineModule.feedbackActionInfo.map(feedbackInfoMarshaller(_)),\n    metadata = timelineModule.metadata.map(moduleMetadataMarshaller(_)),\n    showMoreBehavior = timelineModule.showMoreBehavior.map(moduleShowMoreBehaviorMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/TimelineOperationMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.operation.CursorOperationMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineOperation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorOperation\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelineOperationMarshaller @Inject() (\n  cursorOperationMarshaller: CursorOperationMarshaller) {\n\n  def apply(operation: TimelineOperation): urt.TimelineOperation = operation match {\n    case cursorOperation: CursorOperation => cursorOperationMarshaller(cursorOperation)\n    case _ =>\n      throw new UnsupportedTimelineOperationException(operation)\n  }\n}\n\nclass UnsupportedTimelineOperationException(operation: TimelineOperation)\n    extends UnsupportedOperationException(\n      \"Unsupported timeline operation \" + TransportMarshaller.getSimpleName(operation.getClass))\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/TimelineScribeConfigMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelineScribeConfigMarshaller @Inject() () {\n\n  def apply(timelineScribeConfig: TimelineScribeConfig): urt.TimelineScribeConfig =\n    urt.TimelineScribeConfig(\n      page = timelineScribeConfig.page,\n      section = timelineScribeConfig.section,\n      entityToken = timelineScribeConfig.entityToken\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/UrtTransportMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ChildFeedbackActionMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.FeedbackActionMarshaller\nimport com.twitter.product_mixer.core.model.common.identifier.TransportMarshallerIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Timeline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineInstruction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ContainsFeedbackActionInfos\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction\nimport com.twitter.timelines.render.thriftscala.TimelineResponse\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * [[TransportMarshaller]] for URT types\n *\n * @note to make an instance of a [[UrtTransportMarshaller]] you can use [[UrtTransportMarshallerBuilder.marshaller]]\n */\n@Singleton\nclass UrtTransportMarshaller @Inject() (\n  timelineInstructionMarshaller: TimelineInstructionMarshaller,\n  feedbackActionMarshaller: FeedbackActionMarshaller,\n  childFeedbackActionMarshaller: ChildFeedbackActionMarshaller,\n  timelineMetadataMarshaller: TimelineMetadataMarshaller)\n    extends TransportMarshaller[Timeline, urt.TimelineResponse] {\n\n  override val identifier: TransportMarshallerIdentifier =\n    TransportMarshallerIdentifier(\"UnifiedRichTimeline\")\n\n  override def apply(timeline: Timeline): urt.TimelineResponse = {\n    val feedbackActions: Option[Map[String, urt.FeedbackAction]] = {\n      collectAndMarshallFeedbackActions(timeline.instructions)\n    }\n    urt.TimelineResponse(\n      state = urt.TimelineState.Ok,\n      timeline = urt.Timeline(\n        id = timeline.id,\n        instructions = timeline.instructions.map(timelineInstructionMarshaller(_)),\n        responseObjects =\n          feedbackActions.map(actions => urt.ResponseObjects(feedbackActions = Some(actions))),\n        metadata = timeline.metadata.map(timelineMetadataMarshaller(_))\n      )\n    )\n  }\n\n  // Currently, feedbackActionInfo at the URT TimelineItem level is supported, which covers almost all\n  // existing use cases. However, if additional feedbackActionInfos are defined on the URT\n  // TimelineItemContent level for \"compound\" URT types (see deprecated TopicCollection /\n  // TopicCollectionData), this is not supported. If \"compound\" URT types are added in the future,\n  // support must be added within that type (see ModuleItem) to handle the collection and marshalling\n  // of these feedbackActionInfos.\n\n  private[this] def collectAndMarshallFeedbackActions(\n    instructions: Seq[TimelineInstruction]\n  ): Option[Map[String, urt.FeedbackAction]] = {\n    val feedbackActions: Seq[FeedbackAction] = for {\n      feedbackActionInfos <- instructions.collect {\n        case c: ContainsFeedbackActionInfos => c.feedbackActionInfos\n      }\n      feedbackInfoOpt <- feedbackActionInfos\n      feedbackInfo <- feedbackInfoOpt.toSeq\n      feedbackAction <- feedbackInfo.feedbackActions\n    } yield feedbackAction\n\n    if (feedbackActions.nonEmpty) {\n      val urtFeedbackActions = feedbackActions.map(feedbackActionMarshaller(_))\n\n      val urtChildFeedbackActions: Seq[urt.FeedbackAction] = for {\n        feedbackAction <- feedbackActions\n        childFeedbackActions <- feedbackAction.childFeedbackActions.toSeq\n        childFeedbackAction <- childFeedbackActions\n      } yield childFeedbackActionMarshaller(childFeedbackAction)\n\n      val allUrtFeedbackActions = urtFeedbackActions ++ urtChildFeedbackActions\n\n      Some(\n        allUrtFeedbackActions.map { urtAction =>\n          FeedbackActionMarshaller.generateKey(urtAction) -> urtAction\n        }.toMap\n      )\n    } else {\n      None\n    }\n  }\n}\n\nobject UrtTransportMarshaller {\n  def unavailable(timelineId: String): TimelineResponse = {\n    urt.TimelineResponse(\n      state = urt.TimelineState.Unavailable,\n      timeline = urt.Timeline(\n        id = timelineId,\n        instructions = Seq.empty,\n        responseObjects = None,\n        metadata = None\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/UrtTransportMarshallerBuilder.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.graphql.contextual_ref.ContextualTweetRefMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.graphql.contextual_ref.OuterTweetContextMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.graphql.contextual_ref.TweetHydrationContextMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.rtf.safety_level.SafetyLevelMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert.ShowAlertColorConfigurationMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert.ShowAlertDisplayLocationMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert.ShowAlertIconDisplayInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert.ShowAlertIconMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert.ShowAlertNavigationMetadataMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert.ShowAlertTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.button.ButtonStyleMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.button.CtaButtonMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.button.IconCtaButtonMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.button.TextCtaButtonMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.color.ColorMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.color.ColorPaletteMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.color.RosettaColorMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.commerce.CommerceProductGroupItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.commerce.CommerceProductItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover.CoverContentMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover.CoverCtaBehaviorMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover.CoverCtaMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover.CoverImageMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover.FullCoverContentMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover.FullCoverDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover.HalfCoverContentMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover.HalfCoverDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.icon.HorizonIconMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.article.ArticleDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.article.ArticleItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.article.ArticleSeedTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.audio_space.AudioSpaceItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.card.CardDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.card.CardItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.conversation_annotation.ConversationAnnotationMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.conversation_annotation.ConversationAnnotationTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.event.EventSummaryDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.event.EventSummaryItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.forward_pivot.ForwardPivotDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.forward_pivot.ForwardPivotMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.forward_pivot.SoftInterventionDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.generic_summary_item.GenericSummaryActionMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.generic_summary_item.GenericSummaryContextMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.generic_summary_item.GenericSummaryDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.generic_summary_item.GenericSummaryItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.highlight.HighlightedSectionMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.icon_label.IconLabelItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.label.LabelDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.label.LabelItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message._\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.moment.MomentAnnotationItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.prompt._\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.suggestion.SpellingActionTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.suggestion.SpellingItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.suggestion.TextResultMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.thread.ThreadHeaderContentMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.thread.ThreadHeaderItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tile.CallToActionTileContentMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tile.StandardTileContentMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tile.TileContentMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tile.TileItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tombstone.TombstoneDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tombstone.TombstoneInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tombstone.TombstoneItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.topic.TopicDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.topic.TopicFollowPromptDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.topic.TopicFollowPromptItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.topic.TopicFunctionalityTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.topic.TopicItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.trend.TrendItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet.TimelinesScoreInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet.TweetDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet.TweetHighlightsMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet.TweetItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet_composer.TweetComposerDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet_composer.TweetComposerItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.twitter_list.TwitterListDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.twitter_list.TwitterListItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.user.UserDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.user.UserItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.user.UserReactiveTriggersMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.vertical_grid_item.VerticalGridItemContentMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.vertical_grid_item.VerticalGridItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.vertical_grid_item.VerticalGridItemTileStyleMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.vertical_grid_item.VerticalGridItemTopicFunctionalityTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.vertical_grid_item.VerticalGridItemTopicTileMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media.AspectRatioMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media.BroadcastIdMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media.MediaEntityMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media.MediaKeyMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media.MediaMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media.RectMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media.TweetMediaMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata._\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.operation.CursorDisplayTreatmentMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.operation.CursorItemMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.operation.CursorOperationMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.operation.CursorTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.AdMetadataContainerMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.CallToActionMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.ClickTrackingInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.DisclaimerTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.DisclosureTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.DynamicPrerollTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.MediaInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.PrerollMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.PrerollMetadataMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.PromotedMetadataMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.SkAdNetworkDataMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.SponsorshipTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.UrlOverrideTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.VideoVariantsMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.reaction.TimelineReactionMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.ReferenceObjectMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextAlignmentMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextEntityMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextFormatMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module._\n\n/**\n * Convenience constructor for services not using dependency injection and unit tests. If using\n * dependency injection, instead `@Inject` an instance of [[UrtTransportMarshaller]] to construct.\n */\nobject UrtTransportMarshallerBuilder {\n  val conversationSectionMarshaller = new ConversationSectionMarshaller\n  val conversationDetailsMarshaller = new ConversationDetailsMarshaller(\n    conversationSectionMarshaller)\n  val timelinesDetailsMarshaller = new TimelinesDetailsMarshaller\n  val articleDetailsMarshaller = new ArticleDetailsMarshaller\n  val liveEventDetailsMarshaller = new LiveEventDetailsMarshaller\n  val commerceDetailsMarshaller = new CommerceDetailsMarshaller\n  val clientEventDetailsMarshaller =\n    new ClientEventDetailsMarshaller(\n      conversationDetailsMarshaller,\n      timelinesDetailsMarshaller,\n      articleDetailsMarshaller,\n      liveEventDetailsMarshaller,\n      commerceDetailsMarshaller)\n  val clientEventInfoMarshaller = new ClientEventInfoMarshaller(clientEventDetailsMarshaller)\n\n  val feedbackTypeMarshaller = new FeedbackTypeMarshaller\n  val confirmationDisplayTypeMarshaller = new ConfirmationDisplayTypeMarshaller\n  val horizonIconMarshaller = new HorizonIconMarshaller\n  val richFeedbackBehaviorMarshaller = new RichFeedbackBehaviorMarshaller\n  val childFeedbackActionMarshaller = new ChildFeedbackActionMarshaller(\n    feedbackTypeMarshaller = feedbackTypeMarshaller,\n    confirmationDisplayTypeMarshaller = confirmationDisplayTypeMarshaller,\n    clientEventInfoMarshaller = clientEventInfoMarshaller,\n    horizonIconMarshaller = horizonIconMarshaller,\n    richFeedbackBehaviorMarshaller = richFeedbackBehaviorMarshaller\n  )\n  val feedbackActionMarshaller = new FeedbackActionMarshaller(\n    childFeedbackActionMarshaller = childFeedbackActionMarshaller,\n    feedbackTypeMarshaller = feedbackTypeMarshaller,\n    confirmationDisplayTypeMarshaller = confirmationDisplayTypeMarshaller,\n    clientEventInfoMarshaller = clientEventInfoMarshaller,\n    horizonIconMarshaller = horizonIconMarshaller,\n    richFeedbackBehaviorMarshaller = richFeedbackBehaviorMarshaller\n  )\n  val feedbackDisplayContextMarshaller = new FeedbackDisplayContextMarshaller\n  val feedbackInfoMarshaller = new FeedbackInfoMarshaller(\n    feedbackActionMarshaller = feedbackActionMarshaller,\n    feedbackDisplayContextMarshaller = feedbackDisplayContextMarshaller,\n    clientEventInfoMarshaller = clientEventInfoMarshaller\n  )\n\n  val urlTypeMarshaller = new UrlTypeMarshaller\n  val urtEndpointOptionsMarshaller = new UrtEndpointOptionsMarshaller\n  val urlMarshaller = new UrlMarshaller(\n    urlTypeMarshaller = urlTypeMarshaller,\n    urtEndpointOptionsMarshaller = urtEndpointOptionsMarshaller\n  )\n  val referenceObjectMarshaller = new ReferenceObjectMarshaller(urlMarshaller)\n  val richTextFormatMarshaller = new RichTextFormatMarshaller\n  val richTextEntityMarshaller =\n    new RichTextEntityMarshaller(referenceObjectMarshaller, richTextFormatMarshaller)\n  val richTextAlignmentMarshaller = new RichTextAlignmentMarshaller\n  val richTextMarshaller =\n    new RichTextMarshaller(richTextEntityMarshaller, richTextAlignmentMarshaller)\n\n  val tombstoneInfoMarshaller = new TombstoneInfoMarshaller(richTextMarshaller = richTextMarshaller)\n\n  val generalContextTypeMarshaller = new GeneralContextTypeMarshaller\n  val generalContextMarshaller = new GeneralContextMarshaller(\n    generalContextTypeMarshaller = generalContextTypeMarshaller,\n    urlMarshaller = urlMarshaller\n  )\n\n  val timelineReactionMarshaller = new TimelineReactionMarshaller\n\n  val topicContextMarshaller = new TopicContextMarshaller()\n\n  val socialContextMarshaller = new SocialContextMarshaller(\n    generalContextMarshaller = generalContextMarshaller,\n    topicContextMarshaller = topicContextMarshaller\n  )\n\n  val highlightedSectionMarshaller = new HighlightedSectionMarshaller()\n  val tweetHighlightsMarshaller = new TweetHighlightsMarshaller(highlightedSectionMarshaller)\n\n  val topicDisplayTypeMarshaller = new TopicDisplayTypeMarshaller\n  val topicFunctionalityTypeMarshaller = new TopicFunctionalityTypeMarshaller\n  val topicItemMarshaller = new TopicItemMarshaller(\n    displayTypeMarshaller = topicDisplayTypeMarshaller,\n    functionalityTypeMarshaller = topicFunctionalityTypeMarshaller\n  )\n\n  val topicFollowPromptDisplayTypeMarshaller = new TopicFollowPromptDisplayTypeMarshaller\n  val topicFollowPromptItemMarshaller = new TopicFollowPromptItemMarshaller(\n    displayTypeMarshaller = topicFollowPromptDisplayTypeMarshaller\n  )\n\n  val rosettaColorMarshaller = new RosettaColorMarshaller()\n  val badgeMarshaller = new BadgeMarshaller(\n    rosettaColorMarshaller = rosettaColorMarshaller\n  )\n  val iconCtaButtonMarshaller = new IconCtaButtonMarshaller(horizonIconMarshaller, urlMarshaller)\n  val textCtaButtonMarshaller = new TextCtaButtonMarshaller(urlMarshaller)\n  val ctaButtonMarshaller =\n    new CtaButtonMarshaller(iconCtaButtonMarshaller, textCtaButtonMarshaller)\n\n  val standardTileContentMarshaller = new StandardTileContentMarshaller(\n    badgeMarshaller = badgeMarshaller\n  )\n  val callToActionTileContentMarshaller = new CallToActionTileContentMarshaller(\n    ctaButtonMarshaller = ctaButtonMarshaller,\n    richTextMarshaller = richTextMarshaller\n  )\n\n  val tileContentMarshaller = new TileContentMarshaller(\n    standardTileContentMarshaller = standardTileContentMarshaller,\n    callToActionTileContentMarshaller = callToActionTileContentMarshaller\n  )\n  val colorMarshaller = new ColorMarshaller()\n  val colorPaletteMarshaller = new ColorPaletteMarshaller(\n    colorMarshaller = colorMarshaller\n  )\n  val imageVariantMarshaller = new ImageVariantMarshaller(\n    colorPaletteMarshaller = colorPaletteMarshaller\n  )\n  val imageDisplayTypeMarshaller = new ImageDisplayTypeMarshaller()\n  val imageAnimationTypeMarshaller = new ImageAnimationTypeMarshaller()\n\n  val softInterventionDisplayTypeMarshaller = new SoftInterventionDisplayTypeMarshaller\n  val forwardPivotDisplayTypeMarshaller = new ForwardPivotDisplayTypeMarshaller\n  val forwardPivotMarshaller = new ForwardPivotMarshaller(\n    urlMarshaller = urlMarshaller,\n    richTextMarshaller = richTextMarshaller,\n    forwardPivotDisplayTypeMarshaller = forwardPivotDisplayTypeMarshaller,\n    imageVariantMarshaller = imageVariantMarshaller,\n    badgeMarshaller = badgeMarshaller,\n    rosettaColorMarshaller = rosettaColorMarshaller,\n    softInterventionDisplayTypeMarshaller = softInterventionDisplayTypeMarshaller\n  )\n\n  val tweetDisplayTypeMarshaller = new TweetDisplayTypeMarshaller\n  val timelinesScoreInfoMarshaller = new TimelinesScoreInfoMarshaller\n  val disclosureTypeMarshaller = new DisclosureTypeMarshaller\n  val dynamicPrerollTypeMarshaller = new DynamicPrerollTypeMarshaller\n  val callToActionMarshaller = new CallToActionMarshaller\n  val videoVariantsMarshaller = new VideoVariantsMarshaller\n  val mediaInfoMarshaller = new MediaInfoMarshaller(\n    callToActionMarshaller = callToActionMarshaller,\n    videoVariantsMarshaller = videoVariantsMarshaller\n  )\n  val prerollMarshaller = new PrerollMarshaller(\n    dynamicPrerollTypeMarshaller = dynamicPrerollTypeMarshaller,\n    mediaInfoMarshaller = mediaInfoMarshaller\n  )\n  val sponsorshipTypeMarshaller = new SponsorshipTypeMarshaller\n  val disclaimerTypeMarshaller = new DisclaimerTypeMarshaller\n  val skAdNetworkDataMarshaller = new SkAdNetworkDataMarshaller\n  val adMetadataContainerMarshaller = new AdMetadataContainerMarshaller(\n    sponsorshipTypeMarshaller = sponsorshipTypeMarshaller,\n    disclaimerTypeMarshaller = disclaimerTypeMarshaller,\n    skAdNetworkDataMarshaller = skAdNetworkDataMarshaller\n  )\n  val urlOverrideTypeMarshaller = new UrlOverrideTypeMarshaller\n  val clickTrackingInfoMarshaller = new ClickTrackingInfoMarshaller(\n    urlOverrideTypeMarshaller = urlOverrideTypeMarshaller\n  )\n  val promotedMetadataMarshaller = new PromotedMetadataMarshaller(\n    disclosureTypeMarshaller = disclosureTypeMarshaller,\n    adMetadataContainerMarshaller = adMetadataContainerMarshaller,\n    clickTrackingInfoMarshaller = clickTrackingInfoMarshaller\n  )\n\n  val conversationAnnotationTypeMarshaller = new ConversationAnnotationTypeMarshaller\n  val conversationAnnotationMarshaller = new ConversationAnnotationMarshaller(\n    conversationAnnotationTypeMarshaller = conversationAnnotationTypeMarshaller,\n    richTextMarshaller = richTextMarshaller\n  )\n\n  val safetyLevelMarshaller = new SafetyLevelMarshaller\n  val outerTweetContextMarshaller = new OuterTweetContextMarshaller\n  val tweetHydrationContextMarshaller = new TweetHydrationContextMarshaller(\n    safetyLevelMarshaller = safetyLevelMarshaller,\n    outerTweetContextMarshaller = outerTweetContextMarshaller\n  )\n  val contextualTweetRefMarshaller = new ContextualTweetRefMarshaller(\n    tweetHydrationContextMarshaller = tweetHydrationContextMarshaller\n  )\n  val prerollMetadataMarshaller = new PrerollMetadataMarshaller(\n    prerollMarshaller = prerollMarshaller\n  )\n\n  val rectMarshaller = new RectMarshaller\n  val mediaKeyMarshaller = new MediaKeyMarshaller\n  val broadcastIdMarshaller = new BroadcastIdMarshaller\n  val tweetMediaMarshaller = new TweetMediaMarshaller\n  val mediaEntityMarshaller = new MediaEntityMarshaller(\n    tweetMediaMarshaller = tweetMediaMarshaller,\n    broadcastIdMarshaller = broadcastIdMarshaller,\n    imageVariantMarshaller = imageVariantMarshaller)\n  val aspectRatioMarshaller = new AspectRatioMarshaller\n  val mediaMarshaller = new MediaMarshaller(\n    mediaEntityMarshaller = mediaEntityMarshaller,\n    mediaKeyMarshaller = mediaKeyMarshaller,\n    rectMarshaller = rectMarshaller,\n    aspectRatioMarshaller = aspectRatioMarshaller)\n\n  val tweetItemMarshaller = new TweetItemMarshaller(\n    tweetDisplayTypeMarshaller = tweetDisplayTypeMarshaller,\n    socialContextMarshaller = socialContextMarshaller,\n    tweetHighlightsMarshaller = tweetHighlightsMarshaller,\n    tombstoneInfoMarshaller = tombstoneInfoMarshaller,\n    timelinesScoreInfoMarshaller = timelinesScoreInfoMarshaller,\n    forwardPivotMarshaller = forwardPivotMarshaller,\n    promotedMetadataMarshaller = promotedMetadataMarshaller,\n    conversationAnnotationMarshaller = conversationAnnotationMarshaller,\n    contextualTweetRefMarshaller = contextualTweetRefMarshaller,\n    prerollMetadataMarshaller = prerollMetadataMarshaller,\n    badgeMarshaller = badgeMarshaller,\n    urlMarshaller = urlMarshaller\n  )\n\n  val eventSummaryDisplayTypeMarshaller = new EventSummaryDisplayTypeMarshaller\n  val eventSummaryItemMarshaller = new EventSummaryItemMarshaller(\n    eventSummaryDisplayTypeMarshaller = eventSummaryDisplayTypeMarshaller,\n    imageVariantMarshaller = imageVariantMarshaller,\n    urlMarshaller = urlMarshaller\n  )\n\n  val trendItemMarshaller = new TrendItemMarshaller(\n    promotedMetadataMarshaller = promotedMetadataMarshaller,\n    urlMarshaller = urlMarshaller\n  )\n\n  val userDisplayTypeMarshaller = new UserDisplayTypeMarshaller\n  val userReactiveTriggersMarshaller = new UserReactiveTriggersMarshaller(\n    timelineReactionMarshaller)\n  val userItemMarshaller = new UserItemMarshaller(\n    userDisplayTypeMarshaller = userDisplayTypeMarshaller,\n    promotedMetadataMarshaller = promotedMetadataMarshaller,\n    socialContextMarshaller = socialContextMarshaller,\n    userReactiveTriggersMarshaller = userReactiveTriggersMarshaller,\n  )\n\n  val verticalGridItemTileStyleMarshaller = new VerticalGridItemTileStyleMarshaller\n  val verticalGridItemTopicFunctionalityTypeMarshaller =\n    new VerticalGridItemTopicFunctionalityTypeMarshaller\n\n  val verticalGridItemTopicTileMarshaller = new VerticalGridItemTopicTileMarshaller(\n    styleMarshaller = verticalGridItemTileStyleMarshaller,\n    functionalityTypeMarshaller = verticalGridItemTopicFunctionalityTypeMarshaller,\n    urlMarshaller = urlMarshaller\n  )\n\n  val verticalGridItemContentMarshaller = new VerticalGridItemContentMarshaller(\n    verticalGridItemTopicTileMarshaller)\n\n  val verticalGridItemMarshaller = new VerticalGridItemMarshaller(verticalGridItemContentMarshaller)\n\n  val tombstoneDisplayTypeMarshaller = new TombstoneDisplayTypeMarshaller\n  val tombstoneItemMarshaller = new TombstoneItemMarshaller(\n    displayTypeMarshaller = tombstoneDisplayTypeMarshaller,\n    tombstoneInfoMarshaller = tombstoneInfoMarshaller,\n    tweetItemMarshaller = tweetItemMarshaller)\n\n  val iconLabelItemMarshaller = new IconLabelItemMarshaller(\n    richTextMarshaller,\n    horizonIconMarshaller\n  )\n\n  val labelDisplayTypeMarshaller = new LabelDisplayTypeMarshaller\n  val labelItemMarshaller = new LabelItemMarshaller(\n    displayTypeMarshaller = labelDisplayTypeMarshaller,\n    urlMarshaller = urlMarshaller\n  )\n\n  val tileItemMarshaller = new TileItemMarshaller(\n    tileContentMarshaller = tileContentMarshaller,\n    urlMarshaller = urlMarshaller,\n    imageVariantMarshaller = imageVariantMarshaller\n  )\n\n  val callbackMarshaller = new CallbackMarshaller\n  val messageActionMarshaller = new MessageActionMarshaller(\n    callbackMarshaller,\n    clientEventInfoMarshaller\n  )\n  val messageTextActionMarshaller = new MessageTextActionMarshaller(messageActionMarshaller)\n  val messageImageMarshaller = new MessageImageMarshaller(\n    imageVariantMarshaller\n  )\n  val userFacepileDisplayTypeMarshaller = new UserFacepileDisplayTypeMarshaller()\n  val messageActionTypeMarshaller = new MessageActionTypeMarshaller()\n  val userFacepileMarshaller = new UserFacepileMarshaller(\n    messageActionTypeMarshaller,\n    messageTextActionMarshaller,\n    userFacepileDisplayTypeMarshaller\n  )\n  val inlinePromptMessageContentMarshaller = new InlinePromptMessageContentMarshaller(\n    messageTextActionMarshaller = messageTextActionMarshaller,\n    richTextMarshaller = richTextMarshaller,\n    socialContextMarshaller = socialContextMarshaller,\n    userFacepileMarshaller = userFacepileMarshaller\n  )\n  val headerImagePromptMessageContentMarshaller = new HeaderImagePromptMessageContentMarshaller(\n    messageImageMarshaller = messageImageMarshaller,\n    messageTextActionMarshaller = messageTextActionMarshaller,\n    messageActionMarshaller = messageActionMarshaller,\n    richTextMarshaller = richTextMarshaller\n  )\n  val compactPromptMessageContentMarshaller = new CompactPromptMessageContentMarshaller(\n    messageTextActionMarshaller = messageTextActionMarshaller,\n    messageActionMarshaller = messageActionMarshaller,\n    richTextMarshaller = richTextMarshaller\n  )\n  val messageContentMarshaller = new MessageContentMarshaller(\n    inlinePromptMessageContentMarshaller = inlinePromptMessageContentMarshaller,\n    headerImagePromptMessageContentMarshaller = headerImagePromptMessageContentMarshaller,\n    compactPromptMessageContentMarshaller = compactPromptMessageContentMarshaller\n  )\n  val messagePromptItemMarshaller = new MessagePromptItemMarshaller(\n    messageContentMarshaller = messageContentMarshaller,\n    callbackMarshaller = callbackMarshaller\n  )\n\n  val tweetComposerDisplayTypeMarshaller = new TweetComposerDisplayTypeMarshaller\n  val tweetComposerItemMarshaller = new TweetComposerItemMarshaller(\n    tweetComposerDisplayTypeMarshaller = tweetComposerDisplayTypeMarshaller,\n    urlMarshaller = urlMarshaller\n  )\n\n  val cursorTypeMarshaller = new CursorTypeMarshaller\n  val cursorDisplayTreatmentMarshaller = new CursorDisplayTreatmentMarshaller\n  val cursorItemMarshaller = new CursorItemMarshaller(\n    cursorTypeMarshaller = cursorTypeMarshaller,\n    cursorDisplayTreatmentMarshaller = cursorDisplayTreatmentMarshaller)\n  val articleDisplayTypeMarshaller = new ArticleDisplayTypeMarshaller\n  val articleSeedTypeMarshaller = new ArticleSeedTypeMarshaller\n  val articleItemMarshaller =\n    new ArticleItemMarshaller(\n      articleDisplayTypeMarshaller,\n      socialContextMarshaller,\n      articleSeedTypeMarshaller)\n  val audioSpaceItemMarshaller = new AudioSpaceItemMarshaller\n  val cardDisplayTypeMarshaller = new CardDisplayTypeMarshaller\n  val cardItemMarshaller = new CardItemMarshaller(\n    cardDisplayTypeMarshaller = cardDisplayTypeMarshaller,\n    urlMarshaller = urlMarshaller\n  )\n\n  val twitterListDisplayTypeMarshaller = new TwitterListDisplayTypeMarshaller\n  val twitterListItemMarshaller = new TwitterListItemMarshaller(\n    twitterListDisplayTypeMarshaller = twitterListDisplayTypeMarshaller)\n\n  val threadHeaderItemMarshaller = new ThreadHeaderItemMarshaller(\n    threadHeaderContentMarshaller = new ThreadHeaderContentMarshaller\n  )\n\n  val relevancePromptFollowUpTextInputMarshaller = new RelevancePromptFollowUpTextInputMarshaller(\n    callbackMarshaller = callbackMarshaller\n  )\n  val relevancePromptFollowUpFeedbackTypeMarshaller =\n    new RelevancePromptFollowUpFeedbackTypeMarshaller(\n      relevancePromptFollowUpTextInputMarshaller = relevancePromptFollowUpTextInputMarshaller\n    )\n  val relevancePromptDisplayTypeMarshaller = new RelevancePromptDisplayTypeMarshaller\n  val relevancePromptContentMarshaller = new RelevancePromptContentMarshaller(\n    callbackMarshaller = callbackMarshaller,\n    relevancePromptDisplayTypeMarshaller = relevancePromptDisplayTypeMarshaller,\n    relevancePromptFollowUpFeedbackTypeMarshaller = relevancePromptFollowUpFeedbackTypeMarshaller\n  )\n  val promptContentMarshaller = new PromptContentMarshaller(\n    relevancePromptContentMarshaller = relevancePromptContentMarshaller\n  )\n  val promptItemMarshaller = new PromptItemMarshaller(\n    promptContentMarshaller = promptContentMarshaller,\n    clientEventInfoMarshaller = clientEventInfoMarshaller,\n    callbackMarshaller = callbackMarshaller\n  )\n\n  val textResultMarshaller = new TextResultMarshaller(highlightedSectionMarshaller)\n  val spellingActionTypeMarshaller = new SpellingActionTypeMarshaller()\n  val spellingItemMarshaller = new SpellingItemMarshaller(\n    textResultMarshaller = textResultMarshaller,\n    spellingActionTypeMarshaller = spellingActionTypeMarshaller)\n\n  val momentAnnotationItemMarshaller = new MomentAnnotationItemMarshaller(richTextMarshaller)\n\n  val genericSummaryDisplayTypeMarshaller = new GenericSummaryDisplayTypeMarshaller\n  val genericSummaryActionMarshaller = new GenericSummaryActionMarshaller(\n    urlMarshaller = urlMarshaller,\n    clientEventInfoMarshaller = clientEventInfoMarshaller)\n  val genericSummaryContextMarshaller = new GenericSummaryContextMarshaller(\n    richTextMarshaller = richTextMarshaller,\n    horizonIconMarshaller = horizonIconMarshaller\n  )\n  val genericSummaryItemMarshaller = new GenericSummaryItemMarshaller(\n    genericSummaryDisplayTypeMarshaller = genericSummaryDisplayTypeMarshaller,\n    genericSummaryContextMarshaller = genericSummaryContextMarshaller,\n    genericSummaryActionMarshaller = genericSummaryActionMarshaller,\n    mediaMarshaller = mediaMarshaller,\n    promotedMetadataMarshaller = promotedMetadataMarshaller,\n    richTextMarshaller = richTextMarshaller\n  )\n\n  val commerceProductItemMarshaller = new CommerceProductItemMarshaller\n  val commerceProductGroupItemMarshaller = new CommerceProductGroupItemMarshaller\n\n  val timelineItemMarshaller = new TimelineItemMarshaller(\n    timelineItemContentMarshaller = new TimelineItemContentMarshaller(\n      articleItemMarshaller = articleItemMarshaller,\n      audioSpaceItemMarshaller = audioSpaceItemMarshaller,\n      cardItemMarshaller = cardItemMarshaller,\n      cursorItemMarshaller = cursorItemMarshaller,\n      eventSummaryItemMarshaller = eventSummaryItemMarshaller,\n      iconLabelItemMarshaller = iconLabelItemMarshaller,\n      labelItemMarshaller = labelItemMarshaller,\n      messagePromptItemMarshaller = messagePromptItemMarshaller,\n      tileItemMarshaller = tileItemMarshaller,\n      tombstoneItemMarshaller = tombstoneItemMarshaller,\n      topicFollowPromptItemMarshaller = topicFollowPromptItemMarshaller,\n      topicItemMarshaller = topicItemMarshaller,\n      tweetComposerItemMarshaller = tweetComposerItemMarshaller,\n      tweetItemMarshaller = tweetItemMarshaller,\n      twitterListItemMarshaller = twitterListItemMarshaller,\n      userItemMarshaller = userItemMarshaller,\n      verticalGridItemMarshaller = verticalGridItemMarshaller,\n      threadHeaderItemMarshaller = threadHeaderItemMarshaller,\n      promptItemMarshaller = promptItemMarshaller,\n      spellingItemMarshaller = spellingItemMarshaller,\n      momentAnnotationItemMarshaller = momentAnnotationItemMarshaller,\n      genericSummaryItemMarshaller = genericSummaryItemMarshaller,\n      commerceProductItemMarshaller = commerceProductItemMarshaller,\n      commerceProductGroupItemMarshaller = commerceProductGroupItemMarshaller,\n      trendItemMarshaller = trendItemMarshaller\n    ),\n    clientEventInfoMarshaller = clientEventInfoMarshaller,\n    feedbackInfoMarshaller = feedbackInfoMarshaller\n  )\n\n  val moduleDisplayTypeMarshaller = new ModuleDisplayTypeMarshaller\n  val moduleItemTreeDisplayMarshaller =\n    new ModuleItemTreeDisplayMarshaller(moduleDisplayTypeMarshaller)\n\n  val moduleItemMarshaller = new ModuleItemMarshaller(\n    timelineItemMarshaller = timelineItemMarshaller,\n    moduleItemTreeDisplayMarshaller = moduleItemTreeDisplayMarshaller)\n\n  val moduleHeaderDisplayTypeMarshaller = new ModuleHeaderDisplayTypeMarshaller\n  val moduleHeaderMarshaller = new ModuleHeaderMarshaller(\n    horizonIconMarshaller = horizonIconMarshaller,\n    imageVariantMarshaller = imageVariantMarshaller,\n    socialContextMarshaller = socialContextMarshaller,\n    moduleHeaderDisplayTypeMarshaller = moduleHeaderDisplayTypeMarshaller\n  )\n  val moduleFooterMarshaller = new ModuleFooterMarshaller(urlMarshaller = urlMarshaller)\n  val adsMetadataMarshaller = new AdsMetadataMarshaller\n  val moduleConversationMetadataMarshaller = new ModuleConversationMetadataMarshaller(\n    socialContextMarshaller = socialContextMarshaller)\n  val gridCarouselMetadataMarshaller = new GridCarouselMetadataMarshaller\n  val moduleMetadataMarshaller = new ModuleMetadataMarshaller(\n    adsMetadataMarshaller = adsMetadataMarshaller,\n    moduleConversationMetadataMarshaller = moduleConversationMetadataMarshaller,\n    gridCarouselMetadataMarshaller = gridCarouselMetadataMarshaller\n  )\n  val moduleShowMoreBehaviorRevealByCountMarshaller =\n    new ModuleShowMoreBehaviorRevealByCountMarshaller\n  val moduleShowMoreBehaviorMarshaller = new ModuleShowMoreBehaviorMarshaller(\n    moduleShowMoreBehaviorRevealByCountMarshaller = moduleShowMoreBehaviorRevealByCountMarshaller\n  )\n  val timelineModuleMarshaller = new TimelineModuleMarshaller(\n    moduleItemMarshaller = moduleItemMarshaller,\n    moduleDisplayTypeMarshaller = moduleDisplayTypeMarshaller,\n    moduleHeaderMarshaller = moduleHeaderMarshaller,\n    moduleFooterMarshaller = moduleFooterMarshaller,\n    clientEventInfoMarshaller = clientEventInfoMarshaller,\n    feedbackInfoMarshaller = feedbackInfoMarshaller,\n    moduleMetadataMarshaller = moduleMetadataMarshaller,\n    moduleShowMoreBehaviorMarshaller = moduleShowMoreBehaviorMarshaller\n  )\n\n  val halfCoverDisplayTypeMarshaller = new HalfCoverDisplayTypeMarshaller()\n  val fullCoverDisplayTypeMarshaller = new FullCoverDisplayTypeMarshaller()\n  val coverCtaBehaviorMarshaller = new CoverCtaBehaviorMarshaller(richTextMarshaller, urlMarshaller)\n  val buttonStyleMarshaller = new ButtonStyleMarshaller()\n  val coverCtaMarshaller = new CoverCtaMarshaller(\n    coverCtaBehaviorMarshaller,\n    callbackMarshaller,\n    clientEventInfoMarshaller,\n    horizonIconMarshaller,\n    buttonStyleMarshaller)\n  val coverImageMarshaller =\n    new CoverImageMarshaller(\n      imageVariantMarshaller,\n      imageDisplayTypeMarshaller,\n      imageAnimationTypeMarshaller)\n  val dismissInfoMarshaller = new DismissInfoMarshaller(callbackMarshaller)\n\n  val halfCoverContentMarshaller = new HalfCoverContentMarshaller(\n    halfCoverDisplayTypeMarshaller,\n    coverCtaMarshaller,\n    richTextMarshaller,\n    coverImageMarshaller,\n    dismissInfoMarshaller,\n    callbackMarshaller)\n  val fullCoverContentMarshaller = new FullCoverContentMarshaller(\n    fullCoverDisplayTypeMarshaller,\n    coverCtaMarshaller,\n    richTextMarshaller,\n    imageVariantMarshaller,\n    dismissInfoMarshaller,\n    imageDisplayTypeMarshaller,\n    callbackMarshaller)\n  val coverContentMarshaller =\n    new CoverContentMarshaller(fullCoverContentMarshaller, halfCoverContentMarshaller)\n  val coverMarshaller = new CoverMarshaller(coverContentMarshaller, clientEventInfoMarshaller)\n\n  val cursorOperationMarshaller = new CursorOperationMarshaller(\n    cursorTypeMarshaller = cursorTypeMarshaller,\n    cursorDisplayTreatmentMarshaller = cursorDisplayTreatmentMarshaller)\n  val timelineOperationMarshaller = new TimelineOperationMarshaller(\n    cursorOperationMarshaller = cursorOperationMarshaller)\n\n  val timelineEntryMarshaller = new TimelineEntryMarshaller(\n    timelineEntryContentMarshaller = new TimelineEntryContentMarshaller(\n      timelineItemMarshaller = timelineItemMarshaller,\n      timelineModuleMarshaller = timelineModuleMarshaller,\n      timelineOperationMarshaller = timelineOperationMarshaller))\n\n  val addEntriesInstructionMarshaller = new AddEntriesInstructionMarshaller(\n    timelineEntryMarshaller = timelineEntryMarshaller)\n\n  val markEntriesUnreadInstructionMarshaller = new MarkEntriesUnreadInstructionMarshaller()\n\n  val addToModuleInstructionMarshaller = new AddToModuleInstructionMarshaller(\n    moduleItemMarshaller = moduleItemMarshaller)\n\n  val replaceEntryInstructionMarshaller = new ReplaceEntryInstructionMarshaller(\n    timelineEntryMarshaller = timelineEntryMarshaller\n  )\n\n  val pinEntryInstructionMarshaller = new PinEntryInstructionMarshaller(\n    timelineEntryMarshaller = timelineEntryMarshaller\n  )\n\n  val showAlertTypeMarshaller = new ShowAlertTypeMarshaller()\n  val showAlertIconMarshaller = new ShowAlertIconMarshaller()\n  val showAlertIconDisplayInfoMarshaller = new ShowAlertIconDisplayInfoMarshaller(\n    showAlertIconMarshaller = showAlertIconMarshaller,\n    rosettaColorMarshaller = rosettaColorMarshaller\n  )\n  val showAlertColorConfigurationMarshaller = new ShowAlertColorConfigurationMarshaller(\n    rosettaColorMarshaller = rosettaColorMarshaller\n  )\n  val showAlertDisplayLocationMarshaller = new ShowAlertDisplayLocationMarshaller()\n  val showAlertNavigationMetadataMarshaller = new ShowAlertNavigationMetadataMarshaller()\n  val showAlertInstructionMarshaller = new ShowAlertInstructionMarshaller(\n    showAlertTypeMarshaller = new ShowAlertTypeMarshaller(),\n    clientEventInfoMarshaller = clientEventInfoMarshaller,\n    richTextMarshaller = richTextMarshaller,\n    showAlertIconDisplayInfoMarshaller = showAlertIconDisplayInfoMarshaller,\n    showAlertColorConfigurationMarshaller = showAlertColorConfigurationMarshaller,\n    showAlertDisplayLocationMarshaller = showAlertDisplayLocationMarshaller,\n    showAlertNavigationMetadataMarshaller = showAlertNavigationMetadataMarshaller\n  )\n\n  val timelineInstructionMarshaller = new TimelineInstructionMarshaller(\n    addEntriesInstructionMarshaller = addEntriesInstructionMarshaller,\n    addToModuleInstructionMarshaller = addToModuleInstructionMarshaller,\n    markEntriesUnreadInstructionMarshaller = markEntriesUnreadInstructionMarshaller,\n    pinEntryInstructionMarshaller = pinEntryInstructionMarshaller,\n    replaceEntryInstructionMarshaller = replaceEntryInstructionMarshaller,\n    showAlertInstructionMarshaller = showAlertInstructionMarshaller,\n    terminateTimelineInstructionMarshaller = new TerminateTimelineInstructionMarshaller,\n    coverMarshaller = coverMarshaller,\n  )\n\n  val timelineScribeConfigMarshaller = new TimelineScribeConfigMarshaller\n\n  val readerModeConfigMarshaller = new ReaderModeConfigMarshaller(urlMarshaller)\n\n  val timelineMetadataMarshaller = new TimelineMetadataMarshaller(\n    timelineScribeConfigMarshaller = timelineScribeConfigMarshaller,\n    readerModeConfigMarshaller = readerModeConfigMarshaller\n  )\n\n  val marshaller: UrtTransportMarshaller =\n    new UrtTransportMarshaller(\n      timelineInstructionMarshaller = timelineInstructionMarshaller,\n      feedbackActionMarshaller = feedbackActionMarshaller,\n      childFeedbackActionMarshaller = childFeedbackActionMarshaller,\n      timelineMetadataMarshaller = timelineMetadataMarshaller\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/alert/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/color\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n        \"util/util-core:util-core-util\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/color\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n        \"util/util-core:util-core-util\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/alert/ShowAlertColorConfigurationMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.color.RosettaColorMarshaller\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertColorConfiguration\nimport com.twitter.timelines.render.{thriftscala => urt}\n\n@Singleton\nclass ShowAlertColorConfigurationMarshaller @Inject() (\n  rosettaColorMarshaller: RosettaColorMarshaller) {\n\n  def apply(colorConfiguration: ShowAlertColorConfiguration): urt.ShowAlertColorConfiguration =\n    urt.ShowAlertColorConfiguration(\n      background = rosettaColorMarshaller(colorConfiguration.background),\n      text = rosettaColorMarshaller(colorConfiguration.text),\n      border = colorConfiguration.border.map(rosettaColorMarshaller(_)),\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/alert/ShowAlertDisplayLocationMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertDisplayLocation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.Top\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.Bottom\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.timelines.render.{thriftscala => urt}\n\n@Singleton\nclass ShowAlertDisplayLocationMarshaller @Inject() () {\n\n  def apply(alertDisplayLocation: ShowAlertDisplayLocation): urt.ShowAlertDisplayLocation =\n    alertDisplayLocation match {\n      case Top => urt.ShowAlertDisplayLocation.Top\n      case Bottom => urt.ShowAlertDisplayLocation.Bottom\n    }\n\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/alert/ShowAlertIconDisplayInfoMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.color.RosettaColorMarshaller\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertIconDisplayInfo\n\n@Singleton\nclass ShowAlertIconDisplayInfoMarshaller @Inject() (\n  showAlertIconMarshaller: ShowAlertIconMarshaller,\n  rosettaColorMarshaller: RosettaColorMarshaller,\n) {\n\n  def apply(alertIconDisplayInfo: ShowAlertIconDisplayInfo): urt.ShowAlertIconDisplayInfo =\n    urt.ShowAlertIconDisplayInfo(\n      icon = showAlertIconMarshaller(alertIconDisplayInfo.icon),\n      tint = rosettaColorMarshaller(alertIconDisplayInfo.tint),\n    )\n\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/alert/ShowAlertIconMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.DownArrow\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertIcon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.UpArrow\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.timelines.render.{thriftscala => urt}\n\n@Singleton\nclass ShowAlertIconMarshaller @Inject() () {\n\n  def apply(alertIcon: ShowAlertIcon): urt.ShowAlertIcon = alertIcon match {\n    case UpArrow => urt.ShowAlertIcon.UpArrow\n    case DownArrow => urt.ShowAlertIcon.DownArrow\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/alert/ShowAlertNavigationMetadataMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertNavigationMetadata\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ShowAlertNavigationMetadataMarshaller @Inject() () {\n\n  def apply(alertNavigationMetadata: ShowAlertNavigationMetadata): urt.ShowAlertNavigationMetadata =\n    urt.ShowAlertNavigationMetadata(navigateToEntryId =\n      Some(alertNavigationMetadata.navigateToEntryId))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/alert/ShowAlertTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.alert\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.Navigate\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.NewTweets\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ShowAlertTypeMarshaller @Inject() () {\n\n  def apply(alertType: ShowAlertType): urt.AlertType = alertType match {\n    case NewTweets => urt.AlertType.NewTweets\n    case Navigate => urt.AlertType.Navigate\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/button/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/icon\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/button\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/icon\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/button\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/button/ButtonStyleMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.button\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.ButtonStyle\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.Default\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.Primary\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.Secondary\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.Text\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.Destructive\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.Neutral\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.DestructiveSecondary\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.DestructiveText\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ButtonStyleMarshaller @Inject() () {\n  def apply(buttonStyle: ButtonStyle): urt.ButtonStyle =\n    buttonStyle match {\n      case Default => urt.ButtonStyle.Default\n      case Primary => urt.ButtonStyle.Primary\n      case Secondary => urt.ButtonStyle.Secondary\n      case Text => urt.ButtonStyle.Text\n      case Destructive => urt.ButtonStyle.Destructive\n      case Neutral => urt.ButtonStyle.Neutral\n      case DestructiveSecondary => urt.ButtonStyle.DestructiveSecondary\n      case DestructiveText => urt.ButtonStyle.DestructiveText\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/button/CtaButtonMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.button\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.CtaButton\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.IconCtaButton\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.TextCtaButton\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CtaButtonMarshaller @Inject() (\n  iconCtaButtonMarshaller: IconCtaButtonMarshaller,\n  textCtaButtonMarshaller: TextCtaButtonMarshaller) {\n\n  def apply(ctaButton: CtaButton): urt.CtaButton = ctaButton match {\n    case button: TextCtaButton => urt.CtaButton.Text(textCtaButtonMarshaller(button))\n    case button: IconCtaButton => urt.CtaButton.Icon(iconCtaButtonMarshaller(button))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/button/IconCtaButtonMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.button\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.icon.HorizonIconMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.IconCtaButton\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass IconCtaButtonMarshaller @Inject() (\n  horizonIconMarshaller: HorizonIconMarshaller,\n  urlMarshaller: UrlMarshaller) {\n\n  def apply(iconCtaButton: IconCtaButton): urt.IconCtaButton =\n    urt.IconCtaButton(\n      buttonIcon = horizonIconMarshaller(iconCtaButton.buttonIcon),\n      accessibilityLabel = iconCtaButton.accessibilityLabel,\n      url = urlMarshaller(iconCtaButton.url)\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/button/TextCtaButtonMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.button\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.TextCtaButton\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TextCtaButtonMarshaller @Inject() (\n  urlMarshaller: UrlMarshaller) {\n\n  def apply(textCtaButton: TextCtaButton): urt.TextCtaButton =\n    urt.TextCtaButton(\n      buttonText = textCtaButton.buttonText,\n      url = urlMarshaller(textCtaButton.url)\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/color/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/color\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/color\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/color/ColorMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.color\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.Color\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Singleton\n\n@Singleton\nclass ColorMarshaller {\n\n  def apply(color: Color): urt.Color = urt.Color(\n    red = color.red,\n    green = color.green,\n    blue = color.blue,\n    opacity = color.opacity\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/color/ColorPaletteMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.color\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.ColorPalette\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ColorPaletteMarshaller @Inject() (\n  colorMarshaller: ColorMarshaller) {\n\n  def apply(colorPalette: ColorPalette): urt.ColorPaletteItem = urt.ColorPaletteItem(\n    rgb = colorMarshaller(colorPalette.rgb),\n    percentage = colorPalette.percentage\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/color/RosettaColorMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.color\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color._\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RosettaColorMarshaller @Inject() () {\n\n  def apply(rosettaColor: RosettaColor): urt.RosettaColor = rosettaColor match {\n    case WhiteRosettaColor => urt.RosettaColor.White\n    case BlackRosettaColor => urt.RosettaColor.Black\n    case ClearRosettaColor => urt.RosettaColor.Clear\n    case TextBlackRosettaColor => urt.RosettaColor.TextBlack\n    case TextBlueRosettaColor => urt.RosettaColor.TextBlue\n    case DeepGrayRosettaColor => urt.RosettaColor.DeepGray\n    case MediumGrayRosettaColor => urt.RosettaColor.MediumGray\n    case LightGrayRosettaColor => urt.RosettaColor.LightGray\n    case FadedGrayRosettaColor => urt.RosettaColor.FadedGray\n    case FaintGrayRosettaColor => urt.RosettaColor.FaintGray\n    case DeepOrangeRosettaColor => urt.RosettaColor.DeepOrange\n    case MediumOrangeRosettaColor => urt.RosettaColor.MediumOrange\n    case LightOrangeRosettaColor => urt.RosettaColor.LightOrange\n    case FadedOrangeRosettaColor => urt.RosettaColor.FadedOrange\n    case DeepYellowRosettaColor => urt.RosettaColor.DeepYellow\n    case MediumYellowRosettaColor => urt.RosettaColor.MediumYellow\n    case LightYellowRosettaColor => urt.RosettaColor.LightYellow\n    case FadedYellowRosettaColor => urt.RosettaColor.FadedYellow\n    case DeepGreenRosettaColor => urt.RosettaColor.DeepGreen\n    case MediumGreenRosettaColor => urt.RosettaColor.MediumGreen\n    case LightGreenRosettaColor => urt.RosettaColor.LightGreen\n    case FadedGreenRosettaColor => urt.RosettaColor.FadedGreen\n    case DeepBlueRosettaColor => urt.RosettaColor.DeepBlue\n    case TwitterBlueRosettaColor => urt.RosettaColor.TwitterBlue\n    case LightBlueRosettaColor => urt.RosettaColor.LightBlue\n    case FadedBlueRosettaColor => urt.RosettaColor.FadedBlue\n    case FaintBlueRosettaColor => urt.RosettaColor.FaintBlue\n    case DeepPurpleRosettaColor => urt.RosettaColor.DeepPurple\n    case MediumPurpleRosettaColor => urt.RosettaColor.MediumPurple\n    case LightPurpleRosettaColor => urt.RosettaColor.LightPurple\n    case FadedPurpleRosettaColor => urt.RosettaColor.FadedPurple\n    case DeepRedRosettaColor => urt.RosettaColor.DeepRed\n    case MediumRedRosettaColor => urt.RosettaColor.MediumRed\n    case LightRedRosettaColor => urt.RosettaColor.LightRed\n    case FadedRedRosettaColor => urt.RosettaColor.FadedRed\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/cover/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/button\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/icon\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/richtext\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/cover\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/icon\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/richtext\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/cover\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/cover/CoverContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.FullCoverContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.HalfCoverContent\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CoverContentMarshaller @Inject() (\n  fullCoverContentMarshaller: FullCoverContentMarshaller,\n  halfCoverContentMarshaller: HalfCoverContentMarshaller) {\n\n  def apply(coverContent: CoverContent): urt.Cover = coverContent match {\n    case fullCover: FullCoverContent => fullCoverContentMarshaller(fullCover)\n    case halfCover: HalfCoverContent => halfCoverContentMarshaller(halfCover)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/cover/CoverCtaBehaviorMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverCtaBehavior\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverBehaviorDismiss\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverBehaviorNavigate\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\n\n@Singleton\nclass CoverCtaBehaviorMarshaller @Inject() (\n  richTextMarshaller: RichTextMarshaller,\n  urlMarshaller: UrlMarshaller) {\n\n  def apply(coverCtaBehavior: CoverCtaBehavior): urt.CoverCtaBehavior =\n    coverCtaBehavior match {\n      case dismiss: CoverBehaviorDismiss =>\n        urt.CoverCtaBehavior.Dismiss(\n          urt.CoverBehaviorDismiss(dismiss.feedbackMessage.map(richTextMarshaller(_))))\n      case nav: CoverBehaviorNavigate =>\n        urt.CoverCtaBehavior.Navigate(urt.CoverBehaviorNavigate(urlMarshaller(nav.url)))\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/cover/CoverCtaMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.button.ButtonStyleMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.icon.HorizonIconMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.CallbackMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventInfoMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverCta\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CoverCtaMarshaller @Inject() (\n  coverCtaBehaviorMarshaller: CoverCtaBehaviorMarshaller,\n  callbackMarshaller: CallbackMarshaller,\n  clientEventInfoMarshaller: ClientEventInfoMarshaller,\n  horizonIconMarshaller: HorizonIconMarshaller,\n  buttonStyleMarshaller: ButtonStyleMarshaller) {\n\n  def apply(coverCta: CoverCta): urt.CoverCta = urt.CoverCta(\n    text = coverCta.text,\n    ctaBehavior = coverCtaBehaviorMarshaller(coverCta.ctaBehavior),\n    callbacks = coverCta.callbacks.map(_.map(callbackMarshaller(_))),\n    clientEventInfo = coverCta.clientEventInfo.map(clientEventInfoMarshaller(_)),\n    icon = coverCta.icon.map(horizonIconMarshaller(_)),\n    buttonStyle = coverCta.buttonStyle.map(buttonStyleMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/cover/CoverImageMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ImageAnimationTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ImageDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ImageVariantMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverImage\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CoverImageMarshaller @Inject() (\n  imageVariantMarshaller: ImageVariantMarshaller,\n  imageDisplayTypeMarshaller: ImageDisplayTypeMarshaller,\n  imageAnimationTypeMarshaller: ImageAnimationTypeMarshaller) {\n\n  def apply(coverImage: CoverImage): urt.CoverImage =\n    urt.CoverImage(\n      image = imageVariantMarshaller(coverImage.imageVariant),\n      imageDisplayType = imageDisplayTypeMarshaller(coverImage.imageDisplayType),\n      imageAnimationType = coverImage.imageAnimationType.map(imageAnimationTypeMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/cover/FullCoverContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.CallbackMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.DismissInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ImageDisplayTypeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ImageVariantMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.FullCoverContent\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FullCoverContentMarshaller @Inject() (\n  fullCoverDisplayTypeMarshaller: FullCoverDisplayTypeMarshaller,\n  coverCtaMarshaller: CoverCtaMarshaller,\n  richTextMarshaller: RichTextMarshaller,\n  imageVariantMarshaller: ImageVariantMarshaller,\n  dismissInfoMarshaller: DismissInfoMarshaller,\n  imageDisplayTypeMarshaller: ImageDisplayTypeMarshaller,\n  callbackMarshaller: CallbackMarshaller) {\n\n  def apply(fullCover: FullCoverContent): urt.Cover =\n    urt.Cover.FullCover(\n      urt.FullCover(\n        displayType = fullCoverDisplayTypeMarshaller(fullCover.displayType),\n        primaryText = richTextMarshaller(fullCover.primaryText),\n        primaryCoverCta = coverCtaMarshaller(fullCover.primaryCoverCta),\n        secondaryCoverCta = fullCover.secondaryCoverCta.map(coverCtaMarshaller(_)),\n        secondaryText = fullCover.secondaryText.map(richTextMarshaller(_)),\n        image = fullCover.imageVariant.map(imageVariantMarshaller(_)),\n        details = fullCover.details.map(richTextMarshaller(_)),\n        dismissInfo = fullCover.dismissInfo.map(dismissInfoMarshaller(_)),\n        imageDisplayType = fullCover.imageDisplayType.map(imageDisplayTypeMarshaller(_)),\n        impressionCallbacks = fullCover.impressionCallbacks.map(_.map(callbackMarshaller(_)))\n      ))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/cover/FullCoverDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverFullCoverDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.FullCoverDisplayType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FullCoverDisplayTypeMarshaller @Inject() () {\n\n  def apply(halfCoverDisplayType: FullCoverDisplayType): urt.FullCoverDisplayType =\n    halfCoverDisplayType match {\n      case CoverFullCoverDisplayType => urt.FullCoverDisplayType.Cover\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/cover/HalfCoverContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.CallbackMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.DismissInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.HalfCoverContent\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HalfCoverContentMarshaller @Inject() (\n  halfCoverDisplayTypeMarshaller: HalfCoverDisplayTypeMarshaller,\n  coverCtaMarshaller: CoverCtaMarshaller,\n  richTextMarshaller: RichTextMarshaller,\n  coverImageMarshaller: CoverImageMarshaller,\n  dismissInfoMarshaller: DismissInfoMarshaller,\n  callbackMarshaller: CallbackMarshaller) {\n\n  def apply(halfCover: HalfCoverContent): urt.Cover =\n    urt.Cover.HalfCover(\n      urt.HalfCover(\n        displayType = halfCoverDisplayTypeMarshaller(halfCover.displayType),\n        primaryText = richTextMarshaller(halfCover.primaryText),\n        primaryCoverCta = coverCtaMarshaller(halfCover.primaryCoverCta),\n        secondaryCoverCta = halfCover.secondaryCoverCta.map(coverCtaMarshaller(_)),\n        secondaryText = halfCover.secondaryText.map(richTextMarshaller(_)),\n        impressionCallbacks = halfCover.impressionCallbacks.map(_.map(callbackMarshaller(_))),\n        dismissible = halfCover.dismissible,\n        coverImage = halfCover.coverImage.map(coverImageMarshaller(_)),\n        dismissInfo = halfCover.dismissInfo.map(dismissInfoMarshaller(_))\n      ))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/cover/HalfCoverDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.cover\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CenterCoverHalfCoverDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.CoverHalfCoverDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.HalfCoverDisplayType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HalfCoverDisplayTypeMarshaller @Inject() () {\n\n  def apply(halfCoverDisplayType: HalfCoverDisplayType): urt.HalfCoverDisplayType =\n    halfCoverDisplayType match {\n      case CenterCoverHalfCoverDisplayType => urt.HalfCoverDisplayType.CenterCover\n      case CoverHalfCoverDisplayType => urt.HalfCoverDisplayType.Cover\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/icon/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/icon\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/icon\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/icon/HorizonIconMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.icon\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon._\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HorizonIconMarshaller @Inject() () {\n\n  def apply(icon: HorizonIcon): urt.HorizonIcon = icon match {\n    case Bookmark => urt.HorizonIcon.Bookmark\n    case Moment => urt.HorizonIcon.Moment\n    case Debug => urt.HorizonIcon.Debug\n    case Error => urt.HorizonIcon.Error\n    case Follow => urt.HorizonIcon.Follow\n    case Unfollow => urt.HorizonIcon.Unfollow\n    case Smile => urt.HorizonIcon.Smile\n    case Frown => urt.HorizonIcon.Frown\n    case Help => urt.HorizonIcon.Help\n    case Link => urt.HorizonIcon.Link\n    case Message => urt.HorizonIcon.Message\n    case No => urt.HorizonIcon.No\n    case Outgoing => urt.HorizonIcon.Outgoing\n    case Pin => urt.HorizonIcon.Pin\n    case Retweet => urt.HorizonIcon.Retweet\n    case Speaker => urt.HorizonIcon.Speaker\n    case Trashcan => urt.HorizonIcon.Trashcan\n    case Feedback => urt.HorizonIcon.Feedback\n    case FeedbackClose => urt.HorizonIcon.FeedbackClose\n    case EyeOff => urt.HorizonIcon.EyeOff\n    case Moderation => urt.HorizonIcon.Moderation\n    case Topic => urt.HorizonIcon.Topic\n    case TopicClose => urt.HorizonIcon.TopicClose\n    case Flag => urt.HorizonIcon.Flag\n    case TopicFilled => urt.HorizonIcon.TopicFilled\n    case NotificationsFollow => urt.HorizonIcon.NotificationsFollow\n    case Person => urt.HorizonIcon.Person\n    case BalloonStroke => urt.HorizonIcon.BalloonStroke\n    case Calendar => urt.HorizonIcon.Calendar\n    case LocationStroke => urt.HorizonIcon.LocationStroke\n    case PersonStroke => urt.HorizonIcon.PersonStroke\n    case Safety => urt.HorizonIcon.Safety\n    case Logo => urt.HorizonIcon.Logo\n    case SparkleOn => urt.HorizonIcon.SparkleOn\n    case StarRising => urt.HorizonIcon.StarRising\n    case CameraVideo => urt.HorizonIcon.CameraVideo\n    case ShoppingClock => urt.HorizonIcon.ShoppingClock\n    case ArrowRight => urt.HorizonIcon.ArrowRight\n    case SpeakerOff => urt.HorizonIcon.SpeakerOff\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/BUILD",
    "content": "scala_library(\n    sources = [\"**/*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/graphql/contextual_ref\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/button\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/media\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/reaction\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/richtext\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/button\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/media\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/reaction\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/richtext\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/article/ArticleDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.article\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.ArticleDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.Default\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ArticleDisplayTypeMarshaller @Inject() () {\n  def apply(articleDisplayType: ArticleDisplayType): urt.ArticleDisplayType =\n    articleDisplayType match {\n      case Default => urt.ArticleDisplayType.Default\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/article/ArticleItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.article\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.SocialContextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.ArticleItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ArticleItemMarshaller @Inject() (\n  articleDisplayTypeMarshaller: ArticleDisplayTypeMarshaller,\n  socialContextMarshaller: SocialContextMarshaller,\n  articleSeedTypeMarshaller: ArticleSeedTypeMarshaller) {\n  def apply(articleItem: ArticleItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.Article(\n      urt.Article(\n        id = articleItem.id,\n        displayType = articleItem.displayType.map(articleDisplayTypeMarshaller(_)),\n        socialContext = articleItem.socialContext.map(socialContextMarshaller(_)),\n        articleSeedType = Some(articleSeedTypeMarshaller(articleItem.articleSeedType))\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/article/ArticleSeedTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.article\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.ArticleSeedType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.FollowingListSeed\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.FriendsOfFriendsSeed\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.article.ListIdSeed\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ArticleSeedTypeMarshaller @Inject() () {\n\n  def apply(articleSeedType: ArticleSeedType): urt.ArticleSeedType =\n    articleSeedType match {\n      case FollowingListSeed => urt.ArticleSeedType.FollowingList\n      case FriendsOfFriendsSeed => urt.ArticleSeedType.FriendsOfFriends\n      case ListIdSeed => urt.ArticleSeedType.ListId\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/audio_space/AudioSpaceItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.audio_space\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.audio_space.AudioSpaceItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AudioSpaceItemMarshaller @Inject() () {\n\n  def apply(audioSpaceItem: AudioSpaceItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.AudioSpace(\n      urt.AudioSpace(\n        id = audioSpaceItem.id\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/card/CardDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.card\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.card._\nimport com.twitter.timelines.render.{thriftscala => urt}\n\n@Singleton\nclass CardDisplayTypeMarshaller @Inject() () {\n\n  def apply(cardDisplayType: CardDisplayType): urt.CardDisplayType = cardDisplayType match {\n    case HeroDisplayType => urt.CardDisplayType.Hero\n    case CellDisplayType => urt.CardDisplayType.Cell\n    case TweetCardDisplayType => urt.CardDisplayType.TweetCard\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/card/CardItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.card\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.card.CardItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CardItemMarshaller @Inject() (\n  cardDisplayTypeMarshaller: CardDisplayTypeMarshaller,\n  urlMarshaller: UrlMarshaller) {\n\n  def apply(cardItem: CardItem): urt.TimelineItemContent = {\n    urt.TimelineItemContent.Card(\n      urt.Card(\n        cardUrl = cardItem.cardUrl,\n        text = cardItem.text,\n        subtext = cardItem.subtext,\n        url = cardItem.url.map(urlMarshaller(_)),\n        cardDisplayType = cardItem.displayType.map(cardDisplayTypeMarshaller(_))\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/commerce/CommerceProductGroupItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.commerce\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.commerce.CommerceProductGroupItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CommerceProductGroupItemMarshaller @Inject() () {\n\n  def apply(commerceProductGroupItem: CommerceProductGroupItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.CommerceProductGroup(\n      urt.CommerceProductGroup(commerceProductGroupItem.id))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/commerce/CommerceProductItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.commerce\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.commerce.CommerceProductItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CommerceProductItemMarshaller @Inject() () {\n\n  def apply(commerceProductItem: CommerceProductItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.CommerceProduct(urt.CommerceProduct(commerceProductItem.id))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/conversation_annotation/ConversationAnnotationMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.conversation_annotation\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.conversation_annotation.ConversationAnnotation\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ConversationAnnotationMarshaller @Inject() (\n  conversationAnnotationTypeMarshaller: ConversationAnnotationTypeMarshaller,\n  richTextMarshaller: RichTextMarshaller) {\n\n  def apply(conversationAnnotation: ConversationAnnotation): urt.ConversationAnnotation = {\n    urt.ConversationAnnotation(\n      conversationAnnotationType =\n        conversationAnnotationTypeMarshaller(conversationAnnotation.conversationAnnotationType),\n      header = conversationAnnotation.header.map(richTextMarshaller(_)),\n      description = conversationAnnotation.description.map(richTextMarshaller(_))\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/conversation_annotation/ConversationAnnotationTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.conversation_annotation\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.conversation_annotation.ConversationAnnotationType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.conversation_annotation.Large\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.conversation_annotation.Political\n\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ConversationAnnotationTypeMarshaller @Inject() () {\n\n  def apply(\n    conversationAnnotationType: ConversationAnnotationType\n  ): urt.ConversationAnnotationType = conversationAnnotationType match {\n    case Large => urt.ConversationAnnotationType.Large\n    case Political => urt.ConversationAnnotationType.Political\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/event/EventSummaryDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.event\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.event.EventSummaryDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.event.CellEventSummaryDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.event.HeroEventSummaryDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.event.CellWithProminentSocialContextEventSummaryDisplayType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass EventSummaryDisplayTypeMarshaller @Inject() () {\n\n  def apply(\n    eventSummaryDisplayType: EventSummaryDisplayType\n  ): urt.EventSummaryDisplayType = eventSummaryDisplayType match {\n    case CellEventSummaryDisplayType =>\n      urt.EventSummaryDisplayType.Cell\n    case HeroEventSummaryDisplayType =>\n      urt.EventSummaryDisplayType.Hero\n    case CellWithProminentSocialContextEventSummaryDisplayType =>\n      urt.EventSummaryDisplayType.CellWithProminentSocialContext\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/event/EventSummaryItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.event\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ImageVariantMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.event.EventSummaryItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass EventSummaryItemMarshaller @Inject() (\n  eventSummaryDisplayTypeMarshaller: EventSummaryDisplayTypeMarshaller,\n  imageVariantMarshaller: ImageVariantMarshaller,\n  urlMarshaller: UrlMarshaller) {\n\n  def apply(eventSummary: EventSummaryItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.EventSummary(\n      urt.EventSummary(\n        id = eventSummary.id,\n        title = eventSummary.title,\n        displayType = eventSummaryDisplayTypeMarshaller(eventSummary.displayType),\n        url = urlMarshaller(eventSummary.url),\n        image = eventSummary.image.map(imageVariantMarshaller(_)),\n        timeString = eventSummary.timeString\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/forward_pivot/ForwardPivotDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.forward_pivot\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot.CommunityNotes\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot.ForwardPivotDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot.LiveEvent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot.SoftIntervention\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForwardPivotDisplayTypeMarshaller @Inject() () {\n\n  def apply(forwardPivotDisplayType: ForwardPivotDisplayType): urt.ForwardPivotDisplayType =\n    forwardPivotDisplayType match {\n      case LiveEvent => urt.ForwardPivotDisplayType.LiveEvent\n      case SoftIntervention => urt.ForwardPivotDisplayType.SoftIntervention\n      case CommunityNotes => urt.ForwardPivotDisplayType.CommunityNotes\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/forward_pivot/ForwardPivotMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.forward_pivot\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.color.RosettaColorMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.BadgeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ImageVariantMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot.ForwardPivot\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ForwardPivotMarshaller @Inject() (\n  urlMarshaller: UrlMarshaller,\n  richTextMarshaller: RichTextMarshaller,\n  forwardPivotDisplayTypeMarshaller: ForwardPivotDisplayTypeMarshaller,\n  softInterventionDisplayTypeMarshaller: SoftInterventionDisplayTypeMarshaller,\n  imageVariantMarshaller: ImageVariantMarshaller,\n  badgeMarshaller: BadgeMarshaller,\n  rosettaColorMarshaller: RosettaColorMarshaller) {\n\n  def apply(forwardPivot: ForwardPivot): urt.ForwardPivot = urt.ForwardPivot(\n    text = richTextMarshaller(forwardPivot.text),\n    landingUrl = urlMarshaller(forwardPivot.landingUrl),\n    displayType = forwardPivotDisplayTypeMarshaller(forwardPivot.displayType),\n    iconImageVariant = forwardPivot.iconImageVariant.map(imageVariantMarshaller(_)),\n    stateBadge = forwardPivot.stateBadge.map(badgeMarshaller(_)),\n    subtext = forwardPivot.subtext.map(richTextMarshaller(_)),\n    backgroundColorName = forwardPivot.backgroundColorName.map(rosettaColorMarshaller(_)),\n    engagementNudge = forwardPivot.engagementNudge,\n    softInterventionDisplayType =\n      forwardPivot.softInterventionDisplayType.map(softInterventionDisplayTypeMarshaller(_)),\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/forward_pivot/SoftInterventionDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.forward_pivot\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot.GetTheLatest\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot.GovernmentRequested\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot.Misleading\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot.SoftInterventionDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot.StayInformed\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SoftInterventionDisplayTypeMarshaller @Inject() () {\n\n  def apply(\n    softInterventionDisplayType: SoftInterventionDisplayType\n  ): urt.SoftInterventionDisplayType =\n    softInterventionDisplayType match {\n      case GetTheLatest => urt.SoftInterventionDisplayType.GetTheLatest\n      case StayInformed => urt.SoftInterventionDisplayType.StayInformed\n      case Misleading => urt.SoftInterventionDisplayType.Misleading\n      case GovernmentRequested => urt.SoftInterventionDisplayType.GovernmentRequested\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/generic_summary_item/GenericSummaryActionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.generic_summary_item\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary.GenericSummaryAction\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GenericSummaryActionMarshaller @Inject() (\n  urlMarshaller: UrlMarshaller,\n  clientEventInfoMarshaller: ClientEventInfoMarshaller) {\n\n  def apply(genericSummaryItemAction: GenericSummaryAction): urt.GenericSummaryAction =\n    urt.GenericSummaryAction(\n      url = urlMarshaller(genericSummaryItemAction.url),\n      clientEventInfo = genericSummaryItemAction.clientEventInfo.map(clientEventInfoMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/generic_summary_item/GenericSummaryContextMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.generic_summary_item\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.icon.HorizonIconMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary.GenericSummaryContext\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GenericSummaryContextMarshaller @Inject() (\n  richTextMarshaller: RichTextMarshaller,\n  horizonIconMarshaller: HorizonIconMarshaller) {\n\n  def apply(genericSummaryItemContext: GenericSummaryContext): urt.GenericSummaryContext =\n    urt.GenericSummaryContext(\n      text = richTextMarshaller(genericSummaryItemContext.text),\n      icon = genericSummaryItemContext.icon.map(horizonIconMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/generic_summary_item/GenericSummaryDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.generic_summary_item\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary.GenericSummaryItemDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary.HeroDisplayType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GenericSummaryDisplayTypeMarshaller @Inject() () {\n\n  def apply(\n    genericSummaryItemDisplayType: GenericSummaryItemDisplayType\n  ): urt.GenericSummaryDisplayType =\n    genericSummaryItemDisplayType match {\n      case HeroDisplayType => urt.GenericSummaryDisplayType.Hero\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/generic_summary_item/GenericSummaryItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.generic_summary_item\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media.MediaMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.PromotedMetadataMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary.GenericSummaryItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GenericSummaryItemMarshaller @Inject() (\n  genericSummaryDisplayTypeMarshaller: GenericSummaryDisplayTypeMarshaller,\n  genericSummaryContextMarshaller: GenericSummaryContextMarshaller,\n  genericSummaryActionMarshaller: GenericSummaryActionMarshaller,\n  mediaMarshaller: MediaMarshaller,\n  promotedMetadataMarshaller: PromotedMetadataMarshaller,\n  richTextMarshaller: RichTextMarshaller) {\n\n  def apply(genericSummaryItem: GenericSummaryItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.GenericSummary(\n      urt.GenericSummary(\n        headline = richTextMarshaller(genericSummaryItem.headline),\n        displayType = genericSummaryDisplayTypeMarshaller(genericSummaryItem.displayType),\n        userAttributionIds = genericSummaryItem.userAttributionIds,\n        media = genericSummaryItem.media.map(mediaMarshaller(_)),\n        context = genericSummaryItem.context.map(genericSummaryContextMarshaller(_)),\n        timestamp = genericSummaryItem.timestamp.map(_.inMilliseconds),\n        onClickAction = genericSummaryItem.onClickAction.map(genericSummaryActionMarshaller(_)),\n        promotedMetadata = genericSummaryItem.promotedMetadata.map(promotedMetadataMarshaller(_))\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/highlight/HighlightedSectionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.highlight\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.highlight.HighlightedSection\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HighlightedSectionMarshaller @Inject() () {\n\n  def apply(highlightedSection: HighlightedSection): urt.HighlightedSection =\n    urt.HighlightedSection(\n      startIndex = highlightedSection.startIndex,\n      endIndex = highlightedSection.endIndex\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/icon_label/IconLabelItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.icon_label\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.icon.HorizonIconMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.icon_label.IconLabelItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass IconLabelItemMarshaller @Inject() (\n  richTextMarshaller: RichTextMarshaller,\n  horizonIconMarshaller: HorizonIconMarshaller) {\n\n  def apply(iconLabelItem: IconLabelItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.IconLabel(\n      urt.IconLabel(\n        text = richTextMarshaller(iconLabelItem.text),\n        icon = iconLabelItem.icon.map(horizonIconMarshaller(_))\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/label/LabelDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.label\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.label._\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass LabelDisplayTypeMarshaller @Inject() () {\n\n  def apply(labelDisplayType: LabelDisplayType): urt.LabelDisplayType = labelDisplayType match {\n    case InlineHeaderLabelDisplayType => urt.LabelDisplayType.InlineHeader\n    case OtherRepliesSectionHeaderLabelDisplayType => urt.LabelDisplayType.OtherRepliesSectionHeader\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/label/LabelItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.label\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.label.LabelItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass LabelItemMarshaller @Inject() (\n  displayTypeMarshaller: LabelDisplayTypeMarshaller,\n  urlMarshaller: UrlMarshaller) {\n\n  def apply(labelItem: LabelItem): urt.TimelineItemContent = {\n    urt.TimelineItemContent.Label(\n      urt.Label(\n        text = labelItem.text,\n        subtext = labelItem.subtext,\n        disclosureIndicator = labelItem.disclosureIndicator,\n        url = labelItem.url.map(urlMarshaller(_)),\n        displayType = labelItem.displayType.map(displayTypeMarshaller(_))\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/message/CompactPromptMessageContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.CompactPromptMessageContent\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CompactPromptMessageContentMarshaller @Inject() (\n  messageTextActionMarshaller: MessageTextActionMarshaller,\n  messageActionMarshaller: MessageActionMarshaller,\n  richTextMarshaller: RichTextMarshaller) {\n\n  def apply(compactPromptMessageContent: CompactPromptMessageContent): urt.MessageContent =\n    urt.MessageContent.CompactPrompt(\n      urt.CompactPrompt(\n        headerText = compactPromptMessageContent.headerText,\n        bodyText = compactPromptMessageContent.bodyText,\n        primaryButtonAction =\n          compactPromptMessageContent.primaryButtonAction.map(messageTextActionMarshaller(_)),\n        secondaryButtonAction =\n          compactPromptMessageContent.secondaryButtonAction.map(messageTextActionMarshaller(_)),\n        action = compactPromptMessageContent.action.map(messageActionMarshaller(_)),\n        headerRichText = compactPromptMessageContent.headerRichText.map(richTextMarshaller(_)),\n        bodyRichText = compactPromptMessageContent.bodyRichText.map(richTextMarshaller(_))\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/message/HeaderImagePromptMessageContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.HeaderImagePromptMessageContent\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass HeaderImagePromptMessageContentMarshaller @Inject() (\n  messageImageMarshaller: MessageImageMarshaller,\n  messageTextActionMarshaller: MessageTextActionMarshaller,\n  messageActionMarshaller: MessageActionMarshaller,\n  richTextMarshaller: RichTextMarshaller) {\n\n  def apply(\n    headerImagePromptMessageContent: HeaderImagePromptMessageContent\n  ): urt.MessageContent =\n    urt.MessageContent.HeaderImagePrompt(\n      urt.HeaderImagePrompt(\n        headerImage = messageImageMarshaller(headerImagePromptMessageContent.headerImage),\n        headerText = headerImagePromptMessageContent.headerText,\n        bodyText = headerImagePromptMessageContent.bodyText,\n        primaryButtonAction =\n          headerImagePromptMessageContent.primaryButtonAction.map(messageTextActionMarshaller(_)),\n        secondaryButtonAction =\n          headerImagePromptMessageContent.secondaryButtonAction.map(messageTextActionMarshaller(_)),\n        action = headerImagePromptMessageContent.action.map(messageActionMarshaller(_)),\n        headerRichText = headerImagePromptMessageContent.headerRichText.map(richTextMarshaller(_)),\n        bodyRichText = headerImagePromptMessageContent.bodyRichText.map(richTextMarshaller(_))\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/message/InlinePromptMessageContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.SocialContextMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.InlinePromptMessageContent\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass InlinePromptMessageContentMarshaller @Inject() (\n  messageTextActionMarshaller: MessageTextActionMarshaller,\n  richTextMarshaller: RichTextMarshaller,\n  socialContextMarshaller: SocialContextMarshaller,\n  userFacepileMarshaller: UserFacepileMarshaller) {\n\n  def apply(inlinePromptMessageContent: InlinePromptMessageContent): urt.MessageContent =\n    urt.MessageContent.InlinePrompt(\n      urt.InlinePrompt(\n        headerText = inlinePromptMessageContent.headerText,\n        bodyText = inlinePromptMessageContent.bodyText,\n        primaryButtonAction =\n          inlinePromptMessageContent.primaryButtonAction.map(messageTextActionMarshaller(_)),\n        secondaryButtonAction =\n          inlinePromptMessageContent.secondaryButtonAction.map(messageTextActionMarshaller(_)),\n        headerRichText = inlinePromptMessageContent.headerRichText.map(richTextMarshaller(_)),\n        bodyRichText = inlinePromptMessageContent.bodyRichText.map(richTextMarshaller(_)),\n        socialContext = inlinePromptMessageContent.socialContext.map(socialContextMarshaller(_)),\n        userFacepile = inlinePromptMessageContent.userFacepile.map(userFacepileMarshaller(_))\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/message/MessageActionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.CallbackMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventInfoMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageAction\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MessageActionMarshaller @Inject() (\n  callbackMarshaller: CallbackMarshaller,\n  clientEventInfoMarshaller: ClientEventInfoMarshaller) {\n\n  def apply(messageAction: MessageAction): urt.MessageAction = {\n\n    urt.MessageAction(\n      dismissOnClick = messageAction.dismissOnClick,\n      url = messageAction.url,\n      clientEventInfo = messageAction.clientEventInfo.map(clientEventInfoMarshaller(_)),\n      onClickCallbacks =\n        messageAction.onClickCallbacks.map(callbackList => callbackList.map(callbackMarshaller(_)))\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/message/MessageActionTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.FollowAllMessageActionType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageActionType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MessageActionTypeMarshaller @Inject() () {\n\n  def apply(messageActionType: MessageActionType): urt.MessageActionType = messageActionType match {\n    case FollowAllMessageActionType => urt.MessageActionType.FollowAll\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/message/MessageContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.CompactPromptMessageContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.InlinePromptMessageContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.HeaderImagePromptMessageContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageContent\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MessageContentMarshaller @Inject() (\n  inlinePromptMessageContentMarshaller: InlinePromptMessageContentMarshaller,\n  headerImagePromptMessageContentMarshaller: HeaderImagePromptMessageContentMarshaller,\n  compactPromptMessageContentMarshaller: CompactPromptMessageContentMarshaller) {\n\n  def apply(messageContent: MessageContent): urt.MessageContent = messageContent match {\n    case inlinePromptMessageContent: InlinePromptMessageContent =>\n      inlinePromptMessageContentMarshaller(inlinePromptMessageContent)\n    case headerImagePromptMessageContent: HeaderImagePromptMessageContent =>\n      headerImagePromptMessageContentMarshaller(headerImagePromptMessageContent)\n    case compactPromptMessageContent: CompactPromptMessageContent =>\n      compactPromptMessageContentMarshaller(compactPromptMessageContent)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/message/MessageImageMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ImageVariantMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageImage\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MessageImageMarshaller @Inject() (\n  imageVariantMarshaller: ImageVariantMarshaller) {\n\n  def apply(messageImage: MessageImage): urt.MessageImage = {\n    urt.MessageImage(\n      imageVariants = messageImage.imageVariants.map(imageVariantMarshaller(_)),\n      backgroundColor = messageImage.backgroundColor\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/message/MessagePromptItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.CallbackMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessagePromptItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MessagePromptItemMarshaller @Inject() (\n  messageContentMarshaller: MessageContentMarshaller,\n  callbackMarshaller: CallbackMarshaller) {\n\n  def apply(messagePromptItem: MessagePromptItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.Message(\n      urt.MessagePrompt(\n        content = messageContentMarshaller(messagePromptItem.content),\n        impressionCallbacks = messagePromptItem.impressionCallbacks.map { callbackList =>\n          callbackList.map(callbackMarshaller(_))\n        }\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/message/MessageTextActionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.MessageTextAction\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MessageTextActionMarshaller @Inject() (\n  messageActionMarshaller: MessageActionMarshaller) {\n\n  def apply(messageTextAction: MessageTextAction): urt.MessageTextAction =\n    urt.MessageTextAction(\n      text = messageTextAction.text,\n      action = messageActionMarshaller(messageTextAction.action)\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/message/UserFacepileDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.LargeUserFacepileDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.CompactUserFacepileDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.UserFacepileDisplayType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserFacepileDisplayTypeMarshaller @Inject() () {\n\n  def apply(userFacepileDisplayType: UserFacepileDisplayType): urt.UserFacepileDisplayType =\n    userFacepileDisplayType match {\n      case LargeUserFacepileDisplayType => urt.UserFacepileDisplayType.Large\n      case CompactUserFacepileDisplayType => urt.UserFacepileDisplayType.Compact\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/message/UserFacepileMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.message\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.message.UserFacepile\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserFacepileMarshaller @Inject() (\n  messageActionTypeMarshaller: MessageActionTypeMarshaller,\n  messageTextActionMarshaller: MessageTextActionMarshaller,\n  userFacepileDisplayTypeMarshaller: UserFacepileDisplayTypeMarshaller) {\n\n  def apply(userFacepile: UserFacepile): urt.UserFacepile =\n    urt.UserFacepile(\n      userIds = userFacepile.userIds,\n      featuredUserIds = userFacepile.featuredUserIds,\n      action = userFacepile.action.map(messageTextActionMarshaller(_)),\n      actionType = userFacepile.actionType.map(messageActionTypeMarshaller(_)),\n      displaysFeaturingText = userFacepile.displaysFeaturingText,\n      displayType = userFacepile.displayType.map(userFacepileDisplayTypeMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/moment/MomentAnnotationItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.moment\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.moment.MomentAnnotationItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MomentAnnotationItemMarshaller @Inject() (richTextMarshaller: RichTextMarshaller) {\n  def apply(momentAnnotationItem: MomentAnnotationItem): urt.TimelineItemContent = {\n    urt.TimelineItemContent.MomentAnnotation(\n      urt.MomentAnnotation(\n        annotationId = momentAnnotationItem.id,\n        text = momentAnnotationItem.text.map(richTextMarshaller(_)),\n        header = momentAnnotationItem.header.map(richTextMarshaller(_)),\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/prompt/PromptContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.prompt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.PromptContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.RelevancePromptContent\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PromptContentMarshaller @Inject() (\n  relevancePromptContentMarshaller: RelevancePromptContentMarshaller) {\n\n  def apply(promptContent: PromptContent): urt.PromptContent = promptContent match {\n    case relevancePromptContent: RelevancePromptContent =>\n      urt.PromptContent.RelevancePrompt(relevancePromptContentMarshaller(relevancePromptContent))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/prompt/PromptItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.prompt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.CallbackMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ClientEventInfoMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.PromptItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PromptItemMarshaller @Inject() (\n  promptContentMarshaller: PromptContentMarshaller,\n  clientEventInfoMarshaller: ClientEventInfoMarshaller,\n  callbackMarshaller: CallbackMarshaller) {\n\n  def apply(relevancePromptItem: PromptItem): urt.TimelineItemContent = {\n    urt.TimelineItemContent.Prompt(\n      urt.Prompt(\n        content = promptContentMarshaller(relevancePromptItem.content),\n        clientEventInfo = relevancePromptItem.clientEventInfo.map(clientEventInfoMarshaller(_)),\n        impressionCallbacks = relevancePromptItem.impressionCallbacks.map { callbackList =>\n          callbackList.map(callbackMarshaller(_))\n        }\n      ))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/prompt/RelevancePromptContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.prompt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.CallbackMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.RelevancePromptContent\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RelevancePromptContentMarshaller @Inject() (\n  callbackMarshaller: CallbackMarshaller,\n  relevancePromptDisplayTypeMarshaller: RelevancePromptDisplayTypeMarshaller,\n  relevancePromptFollowUpFeedbackTypeMarshaller: RelevancePromptFollowUpFeedbackTypeMarshaller) {\n\n  def apply(relevancePromptContent: RelevancePromptContent): urt.RelevancePrompt =\n    urt.RelevancePrompt(\n      title = relevancePromptContent.title,\n      confirmation = relevancePromptContent.confirmation,\n      isRelevantText = relevancePromptContent.isRelevantText,\n      notRelevantText = relevancePromptContent.notRelevantText,\n      isRelevantCallback = callbackMarshaller(relevancePromptContent.isRelevantCallback),\n      notRelevantCallback = callbackMarshaller(relevancePromptContent.notRelevantCallback),\n      displayType = relevancePromptDisplayTypeMarshaller(relevancePromptContent.displayType),\n      isRelevantFollowUp = relevancePromptContent.isRelevantFollowUp.map(\n        relevancePromptFollowUpFeedbackTypeMarshaller(_)),\n      notRelevantFollowUp = relevancePromptContent.notRelevantFollowUp.map(\n        relevancePromptFollowUpFeedbackTypeMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/prompt/RelevancePromptDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.prompt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt._\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Singleton\n\n@Singleton\nclass RelevancePromptDisplayTypeMarshaller {\n\n  def apply(\n    relevancePromptDisplayType: RelevancePromptDisplayType\n  ): urt.RelevancePromptDisplayType = relevancePromptDisplayType match {\n    case Normal => urt.RelevancePromptDisplayType.Normal\n    case Compact => urt.RelevancePromptDisplayType.Compact\n    case Large => urt.RelevancePromptDisplayType.Large\n    case ThumbsUpAndDown => urt.RelevancePromptDisplayType.ThumbsUpAndDown\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/prompt/RelevancePromptFollowUpFeedbackTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.prompt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt._\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RelevancePromptFollowUpFeedbackTypeMarshaller @Inject() (\n  relevancePromptFollowUpTextInputMarshaller: RelevancePromptFollowUpTextInputMarshaller) {\n\n  def apply(\n    relevancePromptFollowUpFeedbackType: RelevancePromptFollowUpFeedbackType\n  ): urt.RelevancePromptFollowUpFeedbackType = relevancePromptFollowUpFeedbackType match {\n    case relevancePromptFollowUpTextInput: RelevancePromptFollowUpTextInput =>\n      urt.RelevancePromptFollowUpFeedbackType.FollowUpTextInput(\n        relevancePromptFollowUpTextInputMarshaller(relevancePromptFollowUpTextInput))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/prompt/RelevancePromptFollowUpTextInputMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.prompt\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.CallbackMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt.RelevancePromptFollowUpTextInput\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RelevancePromptFollowUpTextInputMarshaller @Inject() (\n  callbackMarshaller: CallbackMarshaller) {\n\n  def apply(\n    relevancePromptFollowUpTextInput: RelevancePromptFollowUpTextInput\n  ): urt.RelevancePromptFollowUpTextInput = urt.RelevancePromptFollowUpTextInput(\n    context = relevancePromptFollowUpTextInput.context,\n    textFieldPlaceholder = relevancePromptFollowUpTextInput.textFieldPlaceholder,\n    sendTextCallback = callbackMarshaller(relevancePromptFollowUpTextInput.sendTextCallback)\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/suggestion/SpellingActionTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.suggestion\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.suggestion._\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Singleton\n\n@Singleton\nclass SpellingActionTypeMarshaller {\n\n  def apply(spellingActionType: SpellingActionType): urt.SpellingActionType =\n    spellingActionType match {\n      case ReplaceSpellingActionType => urt.SpellingActionType.Replace\n      case ExpandSpellingActionType => urt.SpellingActionType.Expand\n      case SuggestSpellingActionType => urt.SpellingActionType.Suggest\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/suggestion/SpellingItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.suggestion\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.suggestion.SpellingItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SpellingItemMarshaller @Inject() (\n  textResultMarshaller: TextResultMarshaller,\n  spellingActionTypeMarshaller: SpellingActionTypeMarshaller) {\n\n  def apply(spellingItem: SpellingItem): urt.TimelineItemContent = {\n    urt.TimelineItemContent.Spelling(\n      urt.Spelling(\n        spellingResult = textResultMarshaller(spellingItem.textResult),\n        spellingAction = spellingItem.spellingActionType.map(spellingActionTypeMarshaller(_)),\n        originalQuery = spellingItem.originalQuery\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/suggestion/TextResultMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.suggestion\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.highlight.HighlightedSectionMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.highlight.HighlightedSection\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.suggestion.TextResult\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TextResultMarshaller @Inject() (highlightedSectionMarshaller: HighlightedSectionMarshaller) {\n\n  def apply(textResult: TextResult): urt.TextResult = {\n    val hitHighlights = textResult.hitHighlights.map {\n      highlightedSections: Seq[HighlightedSection] =>\n        highlightedSections.map(highlightedSectionMarshaller(_))\n    }\n\n    urt.TextResult(\n      text = textResult.text,\n      hitHighlights = hitHighlights,\n      score = textResult.score,\n      querySource = textResult.querySource)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/thread/ThreadHeaderContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.thread\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.thread._\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ThreadHeaderContentMarshaller @Inject() () {\n  def apply(content: ThreadHeaderContent): urt.ThreadHeaderContent = content match {\n    case UserThreadHeader(userId) =>\n      urt.ThreadHeaderContent.UserThreadHeader(urt.UserThreadHeader(userId))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/thread/ThreadHeaderItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.thread\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.thread.ThreadHeaderItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ThreadHeaderItemMarshaller @Inject() (\n  threadHeaderContentMarshaller: ThreadHeaderContentMarshaller) {\n\n  def apply(threadHeaderItem: ThreadHeaderItem): urt.TimelineItemContent.ThreadHeader =\n    urt.TimelineItemContent.ThreadHeader(\n      urt.ThreadHeaderItem(\n        content = threadHeaderContentMarshaller(threadHeaderItem.content)\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tile/CallToActionTileContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tile\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.button.CtaButtonMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile.CallToActionTileContent\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CallToActionTileContentMarshaller @Inject() (\n  ctaButtonMarshaller: CtaButtonMarshaller,\n  richTextMarshaller: RichTextMarshaller) {\n\n  def apply(callToActionTileContent: CallToActionTileContent): urt.TileContentCallToAction =\n    urt.TileContentCallToAction(\n      text = callToActionTileContent.text,\n      richText = callToActionTileContent.richText.map(richTextMarshaller(_)),\n      ctaButton = callToActionTileContent.ctaButton.map(ctaButtonMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tile/StandardTileContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tile\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.BadgeMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile.StandardTileContent\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass StandardTileContentMarshaller @Inject() (\n  badgeMarshaller: BadgeMarshaller) {\n\n  def apply(standardTileContent: StandardTileContent): urt.TileContentStandard =\n    urt.TileContentStandard(\n      title = standardTileContent.title,\n      supportingText = standardTileContent.supportingText,\n      badge = standardTileContent.badge.map(badgeMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tile/TileContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tile\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile.CallToActionTileContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile.StandardTileContent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile.TileContent\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TileContentMarshaller @Inject() (\n  standardTileContentMarshaller: StandardTileContentMarshaller,\n  callToActionTileContentMarshaller: CallToActionTileContentMarshaller) {\n\n  def apply(tileContent: TileContent): urt.TileContent = tileContent match {\n    case tileCont: StandardTileContent =>\n      urt.TileContent.Standard(standardTileContentMarshaller(tileCont))\n    case tileCont: CallToActionTileContent =>\n      urt.TileContent.CallToAction(callToActionTileContentMarshaller(tileCont))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tile/TileItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tile\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ImageVariantMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile.TileItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TileItemMarshaller @Inject() (\n  tileContentMarshaller: TileContentMarshaller,\n  urlMarshaller: UrlMarshaller,\n  imageVariantMarshaller: ImageVariantMarshaller) {\n\n  def apply(tileItem: TileItem): urt.TimelineItemContent = {\n    urt.TimelineItemContent.Tile(\n      urt.Tile(\n        title = tileItem.title,\n        supportingText = tileItem.supportingText,\n        url = tileItem.url.map(urlMarshaller(_)),\n        image = tileItem.image.map(imageVariantMarshaller(_)),\n        badge = None,\n        content = tileContentMarshaller(tileItem.content)\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tombstone/TombstoneDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tombstone\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone.DisconnectedRepliesAncestor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone.DisconnectedRepliesDescendant\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone.Inline\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone.NonCompliant\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone.TombstoneDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone.TweetUnavailable\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TombstoneDisplayTypeMarshaller @Inject() () {\n\n  def apply(tombstoneDisplayType: TombstoneDisplayType): urt.TombstoneDisplayType =\n    tombstoneDisplayType match {\n      case TweetUnavailable => urt.TombstoneDisplayType.TweetUnavailable\n      case DisconnectedRepliesAncestor => urt.TombstoneDisplayType.DisconnectedRepliesAncestor\n      case DisconnectedRepliesDescendant => urt.TombstoneDisplayType.DisconnectedRepliesDescendant\n      case Inline => urt.TombstoneDisplayType.Inline\n      case NonCompliant => urt.TombstoneDisplayType.NonCompliant\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tombstone/TombstoneInfoMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tombstone\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext.RichTextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone.TombstoneInfo\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TombstoneInfoMarshaller @Inject() (\n  richTextMarshaller: RichTextMarshaller) {\n\n  def apply(tombstoneInfo: TombstoneInfo): urt.TombstoneInfo = urt.TombstoneInfo(\n    text = tombstoneInfo.text,\n    richText = tombstoneInfo.richText.map(richTextMarshaller(_)),\n    richRevealText = tombstoneInfo.richRevealText.map(richTextMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tombstone/TombstoneItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tombstone\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet.TweetItemMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone.TombstoneItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TombstoneItemMarshaller @Inject() (\n  displayTypeMarshaller: TombstoneDisplayTypeMarshaller,\n  tombstoneInfoMarshaller: TombstoneInfoMarshaller,\n  tweetItemMarshaller: TweetItemMarshaller) {\n\n  def apply(tombstoneItem: TombstoneItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.Tombstone(\n      urt.Tombstone(\n        displayType = displayTypeMarshaller(tombstoneItem.tombstoneDisplayType),\n        tombstoneInfo = tombstoneItem.tombstoneInfo.map(tombstoneInfoMarshaller(_)),\n        tweet = tombstoneItem.tweet.map(tweetItemMarshaller(_).tweet)\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/topic/TopicDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.topic\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.BasicTopicDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.NoIconTopicDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.PillTopicDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.PillWithoutActionIconDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicDisplayType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TopicDisplayTypeMarshaller @Inject() () {\n\n  def apply(topicDisplayType: TopicDisplayType): urt.TopicDisplayType = topicDisplayType match {\n    case BasicTopicDisplayType => urt.TopicDisplayType.Basic\n    case PillTopicDisplayType => urt.TopicDisplayType.Pill\n    case NoIconTopicDisplayType => urt.TopicDisplayType.NoIcon\n    case PillWithoutActionIconDisplayType => urt.TopicDisplayType.PillWithoutActionIcon\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/topic/TopicFollowPromptDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.topic\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.IncentiveFocusTopicFollowPromptDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicFocusTopicFollowPromptDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicFollowPromptDisplayType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TopicFollowPromptDisplayTypeMarshaller @Inject() () {\n\n  def apply(\n    topicFollowPromptDisplayType: TopicFollowPromptDisplayType\n  ): urt.TopicFollowPromptDisplayType =\n    topicFollowPromptDisplayType match {\n      case IncentiveFocusTopicFollowPromptDisplayType =>\n        urt.TopicFollowPromptDisplayType.IncentiveFocus\n      case TopicFocusTopicFollowPromptDisplayType => urt.TopicFollowPromptDisplayType.TopicFocus\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/topic/TopicFollowPromptItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.topic\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicFollowPromptItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TopicFollowPromptItemMarshaller @Inject() (\n  displayTypeMarshaller: TopicFollowPromptDisplayTypeMarshaller) {\n\n  def apply(topicFollowPromptItem: TopicFollowPromptItem): urt.TimelineItemContent = {\n    urt.TimelineItemContent.TopicFollowPrompt(\n      urt.TopicFollowPrompt(\n        topicId = topicFollowPromptItem.id.toString,\n        displayType = displayTypeMarshaller(topicFollowPromptItem.topicFollowPromptDisplayType),\n        followIncentiveTitle = topicFollowPromptItem.followIncentiveTitle,\n        followIncentiveText = topicFollowPromptItem.followIncentiveText\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/topic/TopicFunctionalityTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.topic\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.BasicTopicFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.PivotTopicFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.RecommendationTopicFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicFunctionalityType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TopicFunctionalityTypeMarshaller @Inject() () {\n\n  def apply(topicFunctionalityType: TopicFunctionalityType): urt.TopicFunctionalityType =\n    topicFunctionalityType match {\n      case BasicTopicFunctionalityType => urt.TopicFunctionalityType.Basic\n      case RecommendationTopicFunctionalityType => urt.TopicFunctionalityType.Recommendation\n      case PivotTopicFunctionalityType => urt.TopicFunctionalityType.Pivot\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/topic/TopicItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.topic\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic.TopicItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TopicItemMarshaller @Inject() (\n  displayTypeMarshaller: TopicDisplayTypeMarshaller,\n  functionalityTypeMarshaller: TopicFunctionalityTypeMarshaller) {\n\n  def apply(topicItem: TopicItem): urt.TimelineItemContent = {\n    urt.TimelineItemContent.Topic(\n      urt.Topic(\n        topicId = topicItem.id.toString,\n        topicDisplayType = topicItem.topicDisplayType\n          .map(displayTypeMarshaller(_)).getOrElse(urt.TopicDisplayType.Basic),\n        topicFunctionalityType = topicItem.topicFunctionalityType\n          .map(functionalityTypeMarshaller(_)).getOrElse(urt.TopicFunctionalityType.Basic),\n        // This is currently not required by users of this library\n        reactiveTriggers = None\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/trend/TrendItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.trend\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.PromotedMetadataMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.trend.TrendItem\nimport com.twitter.timelines.render.thriftscala.GroupedTrend\nimport com.twitter.timelines.render.thriftscala.TrendMetadata\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.timelines.render.{thriftscala => urt}\n\n@Singleton\nclass TrendItemMarshaller @Inject() (\n  promotedMetadataMarshaller: PromotedMetadataMarshaller,\n  urlMarshaller: UrlMarshaller) {\n\n  def apply(trendItem: TrendItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.Trend(\n      urt.Trend(\n        name = trendItem.trendName,\n        url = urlMarshaller(trendItem.url),\n        promotedMetadata = trendItem.promotedMetadata.map(promotedMetadataMarshaller(_)),\n        description = trendItem.description,\n        trendMetadata = Some(\n          TrendMetadata(\n            metaDescription = trendItem.metaDescription,\n            url = Some(urlMarshaller(trendItem.url)),\n            domainContext = trendItem.domainContext\n          )),\n        groupedTrends = trendItem.groupedTrends.map { trends =>\n          trends.map { trend =>\n            GroupedTrend(name = trend.trendName, url = urlMarshaller(trend.url))\n          }\n        }\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tweet/TimelinesScoreInfoMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TimelinesScoreInfo\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelinesScoreInfoMarshaller @Inject() () {\n\n  def apply(timelinesScoreInfo: TimelinesScoreInfo): urt.TimelinesScoreInfo =\n    urt.TimelinesScoreInfo(score = timelinesScoreInfo.score)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tweet/TweetDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet._\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetDisplayTypeMarshaller @Inject() () {\n\n  def apply(tweetDisplayType: TweetDisplayType): urt.TweetDisplayType = tweetDisplayType match {\n    case Tweet => urt.TweetDisplayType.Tweet\n    case TweetFollowOnly => urt.TweetDisplayType.TweetFollowOnly\n    case Media => urt.TweetDisplayType.Media\n    case MomentTimelineTweet => urt.TweetDisplayType.MomentTimelineTweet\n    case EmphasizedPromotedTweet => urt.TweetDisplayType.EmphasizedPromotedTweet\n    case QuotedTweet => urt.TweetDisplayType.QuotedTweet\n    case SelfThread => urt.TweetDisplayType.SelfThread\n    case CompactPromotedTweet => urt.TweetDisplayType.CompactPromotedTweet\n    case TweetWithoutCard => urt.TweetDisplayType.TweetWithoutCard\n    case ReaderModeRoot => urt.TweetDisplayType.ReaderModeRoot\n    case ReaderMode => urt.TweetDisplayType.ReaderMode\n    case CondensedTweet => urt.TweetDisplayType.CondensedTweet\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tweet/TweetHighlightsMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.highlight.HighlightedSectionMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetHighlights\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetHighlightsMarshaller @Inject() (\n  highlightedSectionMarshaller: HighlightedSectionMarshaller) {\n\n  def apply(tweetHighlights: TweetHighlights): urt.TweetHighlights =\n    urt.TweetHighlights(\n      textHighlights = tweetHighlights.textHighlights\n        .map(_.map(highlightedSectionMarshaller(_))),\n      cardTitleHighlights = tweetHighlights.cardTitleHighlights\n        .map(_.map(highlightedSectionMarshaller(_))),\n      cardDescriptionHighlights = tweetHighlights.cardDescriptionHighlights\n        .map(_.map(highlightedSectionMarshaller(_)))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tweet/TweetItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.graphql.contextual_ref.ContextualTweetRefMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.conversation_annotation.ConversationAnnotationMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.forward_pivot.ForwardPivotMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tombstone.TombstoneInfoMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.SocialContextMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.PrerollMetadataMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.PromotedMetadataMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.BadgeMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetItemMarshaller @Inject() (\n  tweetDisplayTypeMarshaller: TweetDisplayTypeMarshaller,\n  socialContextMarshaller: SocialContextMarshaller,\n  tweetHighlightsMarshaller: TweetHighlightsMarshaller,\n  tombstoneInfoMarshaller: TombstoneInfoMarshaller,\n  timelinesScoreInfoMarshaller: TimelinesScoreInfoMarshaller,\n  forwardPivotMarshaller: ForwardPivotMarshaller,\n  promotedMetadataMarshaller: PromotedMetadataMarshaller,\n  conversationAnnotationMarshaller: ConversationAnnotationMarshaller,\n  contextualTweetRefMarshaller: ContextualTweetRefMarshaller,\n  prerollMetadataMarshaller: PrerollMetadataMarshaller,\n  badgeMarshaller: BadgeMarshaller,\n  urlMarshaller: UrlMarshaller) {\n\n  def apply(tweetItem: TweetItem): urt.TimelineItemContent.Tweet = urt.TimelineItemContent.Tweet(\n    urt.Tweet(\n      id = tweetItem.id,\n      displayType = tweetDisplayTypeMarshaller(tweetItem.displayType),\n      socialContext = tweetItem.socialContext.map(socialContextMarshaller(_)),\n      highlights = tweetItem.highlights.map(tweetHighlightsMarshaller(_)),\n      innerTombstoneInfo = tweetItem.innerTombstoneInfo.map(tombstoneInfoMarshaller(_)),\n      timelinesScoreInfo = tweetItem.timelinesScoreInfo.map(timelinesScoreInfoMarshaller(_)),\n      hasModeratedReplies = tweetItem.hasModeratedReplies,\n      forwardPivot = tweetItem.forwardPivot.map(forwardPivotMarshaller(_)),\n      innerForwardPivot = tweetItem.innerForwardPivot.map(forwardPivotMarshaller(_)),\n      promotedMetadata = tweetItem.promotedMetadata.map(promotedMetadataMarshaller(_)),\n      conversationAnnotation =\n        tweetItem.conversationAnnotation.map(conversationAnnotationMarshaller(_)),\n      contextualTweetRef = tweetItem.contextualTweetRef.map(contextualTweetRefMarshaller(_)),\n      prerollMetadata = tweetItem.prerollMetadata.map(prerollMetadataMarshaller(_)),\n      replyBadge = tweetItem.replyBadge.map(badgeMarshaller(_)),\n      destination = tweetItem.destination.map(urlMarshaller(_))\n    )\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tweet_composer/TweetComposerDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet_composer\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet_composer.Reply\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet_composer.TweetComposerDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet_composer.TweetComposerSelfThread\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetComposerDisplayTypeMarshaller @Inject() () {\n\n  def apply(displayType: TweetComposerDisplayType): urt.TweetComposerDisplayType =\n    displayType match {\n      case TweetComposerSelfThread => urt.TweetComposerDisplayType.SelfThread\n      case Reply => urt.TweetComposerDisplayType.Reply\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/tweet_composer/TweetComposerItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.tweet_composer\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet_composer.TweetComposerItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetComposerItemMarshaller @Inject() (\n  tweetComposerDisplayTypeMarshaller: TweetComposerDisplayTypeMarshaller,\n  urlMarshaller: UrlMarshaller) {\n\n  def apply(tweetComposer: TweetComposerItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.TweetComposer(\n      urt.TweetComposer(\n        displayType = tweetComposerDisplayTypeMarshaller(tweetComposer.displayType),\n        text = tweetComposer.text,\n        url = urlMarshaller(tweetComposer.url)\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/twitter_list/TwitterListDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.twitter_list\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.twitter_list.List\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.twitter_list.ListTile\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.twitter_list.ListWithPin\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.twitter_list.ListWithSubscribe\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.twitter_list.TwitterListDisplayType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TwitterListDisplayTypeMarshaller @Inject() () {\n\n  def apply(twitterListDisplayType: TwitterListDisplayType): urt.TwitterListDisplayType =\n    twitterListDisplayType match {\n      case List => urt.TwitterListDisplayType.List\n      case ListTile => urt.TwitterListDisplayType.ListTile\n      case ListWithPin => urt.TwitterListDisplayType.ListWithPin\n      case ListWithSubscribe => urt.TwitterListDisplayType.ListWithSubscribe\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/twitter_list/TwitterListItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.twitter_list\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.twitter_list.TwitterListItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TwitterListItemMarshaller @Inject() (\n  twitterListDisplayTypeMarshaller: TwitterListDisplayTypeMarshaller) {\n\n  def apply(twitterListItem: TwitterListItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.TwitterList(\n      urt.TwitterList(\n        id = twitterListItem.id,\n        displayType = twitterListItem.displayType.map(twitterListDisplayTypeMarshaller(_))\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/user/UserDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.user\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.PendingFollowUser\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.User\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.UserDetailed\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.UserDisplayType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserDisplayTypeMarshaller @Inject() () {\n\n  def apply(userDisplayType: UserDisplayType): urt.UserDisplayType =\n    userDisplayType match {\n      case User => urt.UserDisplayType.User\n      case UserDetailed => urt.UserDisplayType.UserDetailed\n      case PendingFollowUser => urt.UserDisplayType.PendingFollowUser\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/user/UserItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.user\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.SocialContextMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted.PromotedMetadataMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.UserItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserItemMarshaller @Inject() (\n  userDisplayTypeMarshaller: UserDisplayTypeMarshaller,\n  promotedMetadataMarshaller: PromotedMetadataMarshaller,\n  socialContextMarshaller: SocialContextMarshaller,\n  userReactiveTriggersMarshaller: UserReactiveTriggersMarshaller) {\n\n  def apply(userItem: UserItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.User(\n      urt.User(\n        id = userItem.id,\n        displayType = userDisplayTypeMarshaller(userItem.displayType),\n        promotedMetadata = userItem.promotedMetadata.map(promotedMetadataMarshaller(_)),\n        socialContext = userItem.socialContext.map(socialContextMarshaller(_)),\n        enableReactiveBlending = userItem.enableReactiveBlending,\n        reactiveTriggers = userItem.reactiveTriggers.map(userReactiveTriggersMarshaller(_))\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/user/UserReactiveTriggersMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.user\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.reaction.TimelineReactionMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.user.UserReactiveTriggers\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UserReactiveTriggersMarshaller @Inject() (\n  timelineReactionMarshaller: TimelineReactionMarshaller) {\n\n  def apply(userReactiveTriggers: UserReactiveTriggers): urt.UserReactiveTriggers = {\n    urt.UserReactiveTriggers(\n      onFollow = userReactiveTriggers.onFollow.map(timelineReactionMarshaller(_)))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/vertical_grid_item/VerticalGridItemContentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.vertical_grid_item\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.VerticalGridItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.VerticalGridItemTopicTile\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass VerticalGridItemContentMarshaller @Inject() (\n  verticalGridItemTopicTileMarshaller: VerticalGridItemTopicTileMarshaller) {\n\n  def apply(item: VerticalGridItem): urt.VerticalGridItemContent = item match {\n    case verticalGridItemTopicTile: VerticalGridItemTopicTile =>\n      verticalGridItemTopicTileMarshaller(verticalGridItemTopicTile)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/vertical_grid_item/VerticalGridItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.vertical_grid_item\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.VerticalGridItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass VerticalGridItemMarshaller @Inject() (\n  verticalGridItemContentMarshaller: VerticalGridItemContentMarshaller) {\n\n  def apply(verticalGridItem: VerticalGridItem): urt.TimelineItemContent =\n    urt.TimelineItemContent.VerticalGridItem(\n      urt.VerticalGridItem(\n        content = verticalGridItemContentMarshaller(verticalGridItem)\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/vertical_grid_item/VerticalGridItemTileStyleMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.vertical_grid_item\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.SingleStateDefaultVerticalGridItemTileStyle\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.DoubleStateDefaultVerticalGridItemTileStyle\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.VerticalGridItemTileStyle\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass VerticalGridItemTileStyleMarshaller @Inject() () {\n\n  def apply(verticalGridItemTileStyle: VerticalGridItemTileStyle): urt.VerticalGridItemTileStyle =\n    verticalGridItemTileStyle match {\n      case SingleStateDefaultVerticalGridItemTileStyle =>\n        urt.VerticalGridItemTileStyle.SingleStateDefault\n      case DoubleStateDefaultVerticalGridItemTileStyle =>\n        urt.VerticalGridItemTileStyle.DoubleStateDefault\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/vertical_grid_item/VerticalGridItemTopicFunctionalityTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.vertical_grid_item\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.PivotVerticalGridItemTopicFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.RecommendationVerticalGridItemTopicFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.VerticalGridItemTopicFunctionalityType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass VerticalGridItemTopicFunctionalityTypeMarshaller @Inject() () {\n\n  def apply(\n    verticalGridItemTopicFunctionalityType: VerticalGridItemTopicFunctionalityType\n  ): urt.VerticalGridItemTopicFunctionalityType = verticalGridItemTopicFunctionalityType match {\n    case PivotVerticalGridItemTopicFunctionalityType =>\n      urt.VerticalGridItemTopicFunctionalityType.Pivot\n    case RecommendationVerticalGridItemTopicFunctionalityType =>\n      urt.VerticalGridItemTopicFunctionalityType.Recommendation\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/item/vertical_grid_item/VerticalGridItemTopicTileMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.item.vertical_grid_item\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item.VerticalGridItemTopicTile\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass VerticalGridItemTopicTileMarshaller @Inject() (\n  styleMarshaller: VerticalGridItemTileStyleMarshaller,\n  functionalityTypeMarshaller: VerticalGridItemTopicFunctionalityTypeMarshaller,\n  urlMarshaller: UrlMarshaller) {\n\n  def apply(verticalGridItemTopicTile: VerticalGridItemTopicTile): urt.VerticalGridItemContent =\n    urt.VerticalGridItemContent.TopicTile(\n      urt.VerticalGridItemTopicTile(\n        topicId = verticalGridItemTopicTile.id.toString,\n        style = verticalGridItemTopicTile.style\n          .map(styleMarshaller(_)).getOrElse(urt.VerticalGridItemTileStyle.SingleStateDefault),\n        functionalityType = verticalGridItemTopicTile.functionalityType\n          .map(functionalityTypeMarshaller(_)).getOrElse(\n            urt.VerticalGridItemTopicFunctionalityType.Pivot),\n        url = verticalGridItemTopicTile.url.map(urlMarshaller(_))\n      )\n    )\n\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/media/AspectRatioMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.media.AspectRatio\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.timelines.render.{thriftscala => urt}\n\n@Singleton\nclass AspectRatioMarshaller @Inject() () {\n\n  def apply(aspectRatio: AspectRatio): urt.AspectRatio = urt.AspectRatio(\n    numerator = aspectRatio.numerator,\n    denominator = aspectRatio.denominator\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/media/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/media\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/media\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/media/BroadcastIdMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.media.BroadcastId\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass BroadcastIdMarshaller @Inject() () {\n\n  def apply(broadcastId: BroadcastId): urt.BroadcastId = urt.BroadcastId(\n    id = broadcastId.id\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/media/MediaEntityMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ImageVariantMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.media.BroadcastId\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.media.Image\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.media.MediaEntity\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.media.TweetMedia\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MediaEntityMarshaller @Inject() (\n  tweetMediaMarshaller: TweetMediaMarshaller,\n  broadcastIdMarshaller: BroadcastIdMarshaller,\n  imageVariantMarshaller: ImageVariantMarshaller) {\n\n  def apply(mediaEntity: MediaEntity): urt.MediaEntity = mediaEntity match {\n    case tweetMedia: TweetMedia => urt.MediaEntity.TweetMedia(tweetMediaMarshaller(tweetMedia))\n    case broadcastId: BroadcastId => urt.MediaEntity.BroadcastId(broadcastIdMarshaller(broadcastId))\n    case image: Image => urt.MediaEntity.Image(imageVariantMarshaller(image.image))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/media/MediaKeyMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.media.MediaKey\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MediaKeyMarshaller @Inject() () {\n\n  def apply(mediaKey: MediaKey): urt.MediaKey = urt.MediaKey(\n    id = mediaKey.id,\n    category = mediaKey.category\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/media/MediaMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.media.Media\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MediaMarshaller @Inject() (\n  mediaEntityMarshaller: MediaEntityMarshaller,\n  mediaKeyMarshaller: MediaKeyMarshaller,\n  rectMarshaller: RectMarshaller,\n  aspectRatioMarshaller: AspectRatioMarshaller) {\n\n  def apply(media: Media): urt.Media = urt.Media(\n    mediaEntity = media.mediaEntity.map(mediaEntityMarshaller(_)),\n    mediaKey = media.mediaKey.map(mediaKeyMarshaller(_)),\n    imagePossibleCropping = media.imagePossibleCropping.map { rects =>\n      rects.map(rectMarshaller(_))\n    },\n    aspectRatio = media.aspectRatio.map(aspectRatioMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/media/RectMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.media.Rect\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.timelines.render.{thriftscala => urt}\n\n@Singleton\nclass RectMarshaller @Inject() () {\n\n  def apply(rect: Rect): urt.Rect = urt.Rect(\n    left = rect.left,\n    top = rect.top,\n    width = rect.width,\n    height = rect.height\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/media/TweetMediaMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.media\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.media.TweetMedia\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TweetMediaMarshaller @Inject() () {\n\n  def apply(tweetMedia: TweetMedia): urt.TweetMedia = urt.TweetMedia(\n    tweetId = tweetMedia.tweetId,\n    momentId = tweetMedia.momentId\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/ArticleDetailsMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ArticleDetails\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ArticleDetailsMarshaller @Inject() () {\n\n  def apply(articleDetails: ArticleDetails): urt.ArticleDetails = urt.ArticleDetails(\n    articlePosition = articleDetails.articlePosition,\n    shareCount = articleDetails.shareCount\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/color\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/icon\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/color\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/icon\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/BadgeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.color.RosettaColorMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Badge\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass BadgeMarshaller @Inject() (\n  rosettaColorMarshaller: RosettaColorMarshaller) {\n\n  def apply(badge: Badge): urt.Badge = urt.Badge(\n    text = badge.text,\n    textColorName = badge.textColorName.map(rosettaColorMarshaller(_)),\n    backgroundColorName = badge.backgroundColorName.map(rosettaColorMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/CallbackMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CallbackMarshaller @Inject() () {\n\n  def apply(callback: Callback): urt.Callback = urt.Callback(\n    endpoint = callback.endpoint\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/ChildFeedbackActionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.icon.HorizonIconMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ChildFeedbackAction\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ChildFeedbackActionMarshaller @Inject() (\n  feedbackTypeMarshaller: FeedbackTypeMarshaller,\n  confirmationDisplayTypeMarshaller: ConfirmationDisplayTypeMarshaller,\n  clientEventInfoMarshaller: ClientEventInfoMarshaller,\n  horizonIconMarshaller: HorizonIconMarshaller,\n  richFeedbackBehaviorMarshaller: RichFeedbackBehaviorMarshaller) {\n\n  def apply(feedbackAction: ChildFeedbackAction): urt.FeedbackAction = {\n    urt.FeedbackAction(\n      feedbackType = feedbackTypeMarshaller(feedbackAction.feedbackType),\n      prompt = feedbackAction.prompt,\n      confirmation = feedbackAction.confirmation,\n      childKeys = None,\n      feedbackUrl = feedbackAction.feedbackUrl,\n      hasUndoAction = feedbackAction.hasUndoAction,\n      confirmationDisplayType =\n        feedbackAction.confirmationDisplayType.map(confirmationDisplayTypeMarshaller(_)),\n      clientEventInfo = feedbackAction.clientEventInfo.map(clientEventInfoMarshaller(_)),\n      icon = feedbackAction.icon.map(horizonIconMarshaller(_)),\n      richBehavior = feedbackAction.richBehavior.map(richFeedbackBehaviorMarshaller(_)),\n      subprompt = feedbackAction.subprompt\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/ClientEventDetailsMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventDetails\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ClientEventDetailsMarshaller @Inject() (\n  conversationDetailsMarshaller: ConversationDetailsMarshaller,\n  timelinesDetailsMarshaller: TimelinesDetailsMarshaller,\n  articleDetailsMarshaller: ArticleDetailsMarshaller,\n  liveEventDetailsMarshaller: LiveEventDetailsMarshaller,\n  commerceDetailsMarshaller: CommerceDetailsMarshaller) {\n\n  def apply(clientEventDetails: ClientEventDetails): urt.ClientEventDetails = {\n    urt.ClientEventDetails(\n      conversationDetails =\n        clientEventDetails.conversationDetails.map(conversationDetailsMarshaller(_)),\n      timelinesDetails = clientEventDetails.timelinesDetails.map(timelinesDetailsMarshaller(_)),\n      articleDetails = clientEventDetails.articleDetails.map(articleDetailsMarshaller(_)),\n      liveEventDetails = clientEventDetails.liveEventDetails.map(liveEventDetailsMarshaller(_)),\n      commerceDetails = clientEventDetails.commerceDetails.map(commerceDetailsMarshaller(_))\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/ClientEventInfoMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ClientEventInfoMarshaller @Inject() (\n  clientEventDetailsMarshaller: ClientEventDetailsMarshaller) {\n\n  def apply(clientEventInfo: ClientEventInfo): urt.ClientEventInfo = {\n    urt.ClientEventInfo(\n      component = clientEventInfo.component,\n      element = clientEventInfo.element,\n      details = clientEventInfo.details.map(clientEventDetailsMarshaller(_)),\n      action = clientEventInfo.action,\n      entityToken = clientEventInfo.entityToken\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/CommerceDetailsMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.CommerceDetails\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CommerceDetailsMarshaller @Inject() () {\n\n  def apply(commerceDetails: CommerceDetails): urt.CommerceDetails = urt.CommerceDetails(\n    dropId = commerceDetails.dropId,\n    shopV2Id = commerceDetails.shopV2Id,\n    productKey = commerceDetails.productKey,\n    merchantId = commerceDetails.merchantId,\n    productIndex = commerceDetails.productIndex,\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/ConfirmationDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BottomSheet\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ConfirmationDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Inline\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ConfirmationDisplayTypeMarshaller @Inject() () {\n\n  def apply(confirmationDisplayType: ConfirmationDisplayType): urt.ConfirmationDisplayType =\n    confirmationDisplayType match {\n      case Inline => urt.ConfirmationDisplayType.Inline\n      case BottomSheet => urt.ConfirmationDisplayType.BottomSheet\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/ConversationDetailsMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ConversationDetails\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ConversationDetailsMarshaller @Inject() (sectionMarshaller: ConversationSectionMarshaller) {\n\n  def apply(conversationDetails: ConversationDetails): urt.ConversationDetails =\n    urt.ConversationDetails(\n      conversationSection = conversationDetails.conversationSection.map(sectionMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/ConversationSectionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.AbusiveQuality\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ConversationSection\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.HighQuality\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.LowQuality\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RelatedTweet\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ConversationSectionMarshaller @Inject() () {\n\n  def apply(section: ConversationSection): urt.ConversationSection = section match {\n    case HighQuality => urt.ConversationSection.HighQuality\n    case LowQuality => urt.ConversationSection.LowQuality\n    case AbusiveQuality => urt.ConversationSection.AbusiveQuality\n    case RelatedTweet => urt.ConversationSection.RelatedTweet\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/DismissInfoMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DismissInfo\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass DismissInfoMarshaller @Inject() (callbackMarshaller: CallbackMarshaller) {\n\n  def apply(dismissInfo: DismissInfo): urt.DismissInfo =\n    urt.DismissInfo(dismissInfo.callbacks.map(_.map(callbackMarshaller(_))))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/FeedbackActionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.icon.HorizonIconMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.FeedbackActionMarshaller.generateKey\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackAction\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\nobject FeedbackActionMarshaller {\n  def generateKey(feedbackAction: urt.FeedbackAction): String = {\n    feedbackAction.hashCode.toString\n  }\n}\n\n@Singleton\nclass FeedbackActionMarshaller @Inject() (\n  childFeedbackActionMarshaller: ChildFeedbackActionMarshaller,\n  feedbackTypeMarshaller: FeedbackTypeMarshaller,\n  confirmationDisplayTypeMarshaller: ConfirmationDisplayTypeMarshaller,\n  clientEventInfoMarshaller: ClientEventInfoMarshaller,\n  horizonIconMarshaller: HorizonIconMarshaller,\n  richFeedbackBehaviorMarshaller: RichFeedbackBehaviorMarshaller) {\n\n  def apply(feedbackAction: FeedbackAction): urt.FeedbackAction = {\n    val childKeys = feedbackAction.childFeedbackActions\n      .map(_.map { childFeedbackAction =>\n        val urtChildFeedbackAction = childFeedbackActionMarshaller(childFeedbackAction)\n        generateKey(urtChildFeedbackAction)\n      })\n\n    urt.FeedbackAction(\n      feedbackType = feedbackTypeMarshaller(feedbackAction.feedbackType),\n      prompt = feedbackAction.prompt,\n      confirmation = feedbackAction.confirmation,\n      childKeys = childKeys,\n      feedbackUrl = feedbackAction.feedbackUrl,\n      hasUndoAction = feedbackAction.hasUndoAction,\n      confirmationDisplayType =\n        feedbackAction.confirmationDisplayType.map(confirmationDisplayTypeMarshaller(_)),\n      clientEventInfo = feedbackAction.clientEventInfo.map(clientEventInfoMarshaller(_)),\n      icon = feedbackAction.icon.map(horizonIconMarshaller(_)),\n      richBehavior = feedbackAction.richBehavior.map(richFeedbackBehaviorMarshaller(_)),\n      subprompt = feedbackAction.subprompt,\n      encodedFeedbackRequest = feedbackAction.encodedFeedbackRequest\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/FeedbackDisplayContextMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackDisplayContext\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FeedbackDisplayContextMarshaller @Inject() () {\n\n  def apply(displayContext: FeedbackDisplayContext): urt.FeedbackDisplayContext =\n    urt.FeedbackDisplayContext(\n      reason = displayContext.reason\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/FeedbackInfoMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FeedbackInfoMarshaller @Inject() (\n  feedbackActionMarshaller: FeedbackActionMarshaller,\n  feedbackDisplayContextMarshaller: FeedbackDisplayContextMarshaller,\n  clientEventInfoMarshaller: ClientEventInfoMarshaller) {\n\n  def apply(feedbackActionInfo: FeedbackActionInfo): urt.FeedbackInfo = urt.FeedbackInfo(\n    // Generate key from the hashcode of the marshalled feedback action URT\n    feedbackKeys = feedbackActionInfo.feedbackActions\n      .map(feedbackActionMarshaller(_)).map(FeedbackActionMarshaller.generateKey),\n    feedbackMetadata = feedbackActionInfo.feedbackMetadata,\n    displayContext = feedbackActionInfo.displayContext.map(feedbackDisplayContextMarshaller(_)),\n    clientEventInfo = feedbackActionInfo.clientEventInfo.map(clientEventInfoMarshaller(_)),\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/FeedbackTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass FeedbackTypeMarshaller @Inject() () {\n\n  def apply(feedbackType: FeedbackType): urt.FeedbackType = feedbackType match {\n    case Dismiss => urt.FeedbackType.Dismiss\n    case SeeFewer => urt.FeedbackType.SeeFewer\n    case DontLike => urt.FeedbackType.DontLike\n    case NotRelevant => urt.FeedbackType.NotRelevant\n    case SeeMore => urt.FeedbackType.SeeMore\n    case NotCredible => urt.FeedbackType.NotCredible\n    case GiveFeedback => urt.FeedbackType.GiveFeedback\n    case NotRecent => urt.FeedbackType.NotRecent\n    case UnfollowEntity => urt.FeedbackType.UnfollowEntity\n    case Relevant => urt.FeedbackType.Relevant\n    case Moderate => urt.FeedbackType.Moderate\n    case RichBehavior => urt.FeedbackType.RichBehavior\n    case NotAboutTopic => urt.FeedbackType.NotAboutTopic\n    case Generic => urt.FeedbackType.Generic\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/GeneralContextMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.GeneralContext\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GeneralContextMarshaller @Inject() (\n  generalContextTypeMarshaller: GeneralContextTypeMarshaller,\n  urlMarshaller: UrlMarshaller) {\n\n  def apply(generalContext: GeneralContext): urt.SocialContext = {\n    urt.SocialContext.GeneralContext(\n      urt.GeneralContext(\n        contextType = generalContextTypeMarshaller(generalContext.contextType),\n        text = generalContext.text,\n        url = generalContext.url,\n        contextImageUrls = generalContext.contextImageUrls,\n        landingUrl = generalContext.landingUrl.map(urlMarshaller(_))\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/GeneralContextTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata._\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GeneralContextTypeMarshaller @Inject() () {\n\n  def apply(generalContextType: GeneralContextType): urt.ContextType = generalContextType match {\n    case LikeGeneralContextType => urt.ContextType.Like\n    case FollowGeneralContextType => urt.ContextType.Follow\n    case MomentGeneralContextType => urt.ContextType.Moment\n    case ReplyGeneralContextType => urt.ContextType.Reply\n    case ConversationGeneralContextType => urt.ContextType.Conversation\n    case PinGeneralContextType => urt.ContextType.Pin\n    case TextOnlyGeneralContextType => urt.ContextType.TextOnly\n    case FacePileGeneralContextType => urt.ContextType.Facepile\n    case MegaPhoneGeneralContextType => urt.ContextType.Megaphone\n    case BirdGeneralContextType => urt.ContextType.Bird\n    case FeedbackGeneralContextType => urt.ContextType.Feedback\n    case TopicGeneralContextType => urt.ContextType.Topic\n    case ListGeneralContextType => urt.ContextType.List\n    case RetweetGeneralContextType => urt.ContextType.Retweet\n    case LocationGeneralContextType => urt.ContextType.Location\n    case CommunityGeneralContextType => urt.ContextType.Community\n    case NewUserGeneralContextType => urt.ContextType.NewUser\n    case SmartblockExpirationGeneralContextType => urt.ContextType.SmartBlockExpiration\n    case TrendingGeneralContextType => urt.ContextType.Trending\n    case SparkleGeneralContextType => urt.ContextType.Sparkle\n    case SpacesGeneralContextType => urt.ContextType.Spaces\n    case ReplyPinGeneralContextType => urt.ContextType.ReplyPin\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/ImageAnimationTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageAnimationType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Bounce\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ImageAnimationTypeMarshaller @Inject() () {\n\n  def apply(imageAnimationType: ImageAnimationType): urt.ImageAnimationType =\n    imageAnimationType match {\n      case Bounce => urt.ImageAnimationType.Bounce\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/ImageDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Icon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FullWidth\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.IconSmall\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ImageDisplayTypeMarshaller @Inject() () {\n\n  def apply(imageDisplayType: ImageDisplayType): urt.ImageDisplayType =\n    imageDisplayType match {\n      case Icon => urt.ImageDisplayType.Icon\n      case FullWidth => urt.ImageDisplayType.FullWidth\n      case IconSmall => urt.ImageDisplayType.IconSmall\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/ImageVariantMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.color.ColorPaletteMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageVariant\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ImageVariantMarshaller @Inject() (\n  colorPaletteMarshaller: ColorPaletteMarshaller) {\n\n  def apply(imageVariant: ImageVariant): urt.ImageVariant = urt.ImageVariant(\n    url = imageVariant.url,\n    width = imageVariant.width,\n    height = imageVariant.height,\n    palette = imageVariant.palette.map { paletteList => paletteList.map(colorPaletteMarshaller(_)) }\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/LiveEventDetailsMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.LiveEventDetails\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass LiveEventDetailsMarshaller @Inject() () {\n\n  def apply(liveEventDetails: LiveEventDetails): urt.LiveEventDetails = urt.LiveEventDetails(\n    eventId = liveEventDetails.eventId\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/RichFeedbackBehaviorMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.NotPinnableReplyPinState\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.PinnableReplyPinState\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.PinnedReplyPinState\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehavior\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorBlockUser\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorMarkNotInterestedTopic\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorReplyPinState\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorReportList\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorReportTweet\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleFollowTopic\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleFollowTopicV2\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleFollowUser\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleMuteList\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RichFeedbackBehaviorToggleMuteUser\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RichFeedbackBehaviorMarshaller @Inject() () {\n\n  def apply(richFeedbackBehavior: RichFeedbackBehavior): urt.RichFeedbackBehavior =\n    richFeedbackBehavior match {\n      case RichFeedbackBehaviorReportList(listId, userId) =>\n        urt.RichFeedbackBehavior.ReportList(urt.RichFeedbackBehaviorReportList(listId, userId))\n      case RichFeedbackBehaviorBlockUser(userId) =>\n        urt.RichFeedbackBehavior.BlockUser(urt.RichFeedbackBehaviorBlockUser(userId))\n      case RichFeedbackBehaviorToggleFollowTopic(topicId) =>\n        urt.RichFeedbackBehavior.ToggleFollowTopic(\n          urt.RichFeedbackBehaviorToggleFollowTopic(topicId))\n      case RichFeedbackBehaviorToggleFollowTopicV2(topicId) =>\n        urt.RichFeedbackBehavior.ToggleFollowTopicV2(\n          urt.RichFeedbackBehaviorToggleFollowTopicV2(topicId))\n      case RichFeedbackBehaviorToggleMuteList(listId) =>\n        urt.RichFeedbackBehavior.ToggleMuteList(urt.RichFeedbackBehaviorToggleMuteList(listId))\n      case RichFeedbackBehaviorMarkNotInterestedTopic(topicId) =>\n        urt.RichFeedbackBehavior.MarkNotInterestedTopic(\n          urt.RichFeedbackBehaviorMarkNotInterestedTopic(topicId))\n      case RichFeedbackBehaviorReplyPinState(replyPinState) =>\n        val pinState: urt.ReplyPinState = replyPinState match {\n          case PinnedReplyPinState => urt.ReplyPinState.Pinned\n          case PinnableReplyPinState => urt.ReplyPinState.Pinnable\n          case NotPinnableReplyPinState => urt.ReplyPinState.NotPinnable\n        }\n        urt.RichFeedbackBehavior.ReplyPinState(urt.RichFeedbackBehaviorReplyPinState(pinState))\n      case RichFeedbackBehaviorToggleMuteUser(userId) =>\n        urt.RichFeedbackBehavior.ToggleMuteUser(urt.RichFeedbackBehaviorToggleMuteUser(userId))\n      case RichFeedbackBehaviorToggleFollowUser(userId) =>\n        urt.RichFeedbackBehavior.ToggleFollowUser(urt.RichFeedbackBehaviorToggleFollowUser(userId))\n      case RichFeedbackBehaviorReportTweet(entryId) =>\n        urt.RichFeedbackBehavior.ReportTweet(urt.RichFeedbackBehaviorReportTweet(entryId))\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/SocialContextMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.GeneralContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContext\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SocialContextMarshaller @Inject() (\n  generalContextMarshaller: GeneralContextMarshaller,\n  topicContextMarshaller: TopicContextMarshaller) {\n\n  def apply(socialContext: SocialContext): urt.SocialContext =\n    socialContext match {\n      case generalContextBanner: GeneralContext =>\n        generalContextMarshaller(generalContextBanner)\n      case topicContextBanner: TopicContext =>\n        topicContextMarshaller(topicContextBanner)\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/TimelinesDetailsMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TimelinesDetails\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelinesDetailsMarshaller @Inject() () {\n\n  def apply(timelinesDetails: TimelinesDetails): urt.TimelinesDetails = urt.TimelinesDetails(\n    injectionType = timelinesDetails.injectionType,\n    controllerData = timelinesDetails.controllerData,\n    sourceData = timelinesDetails.sourceData\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/TopicContextFunctionalityTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BasicTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecWithEducationTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.RecommendationTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContextFunctionalityType\nimport com.twitter.timelines.render.{thriftscala => urt}\n\nobject TopicContextFunctionalityTypeMarshaller {\n\n  def apply(\n    topicContextFunctionalityType: TopicContextFunctionalityType\n  ): urt.TopicContextFunctionalityType = topicContextFunctionalityType match {\n    case BasicTopicContextFunctionalityType => urt.TopicContextFunctionalityType.Basic\n    case RecommendationTopicContextFunctionalityType =>\n      urt.TopicContextFunctionalityType.Recommendation\n    case RecWithEducationTopicContextFunctionalityType =>\n      urt.TopicContextFunctionalityType.RecWithEducation\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/TopicContextMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.BasicTopicContextFunctionalityType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.TopicContext\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TopicContextMarshaller @Inject() () {\n\n  def apply(topicContext: TopicContext): urt.SocialContext = {\n    urt.SocialContext.TopicContext(\n      urt.TopicContext(\n        topicId = topicContext.topicId,\n        functionalityType = TopicContextFunctionalityTypeMarshaller(\n          topicContext.functionalityType.getOrElse(BasicTopicContextFunctionalityType))\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/UrlMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UrlMarshaller @Inject() (\n  urlTypeMarshaller: UrlTypeMarshaller,\n  urtEndpointOptionsMarshaller: UrtEndpointOptionsMarshaller) {\n\n  def apply(url: Url): urt.Url = urt.Url(\n    urlType = urlTypeMarshaller(url.urlType),\n    url = url.url,\n    urtEndpointOptions = url.urtEndpointOptions.map(urtEndpointOptionsMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/UrlTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DeepLink\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ExternalUrl\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.UrlType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.UrtEndpoint\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UrlTypeMarshaller @Inject() () {\n\n  def apply(urlType: UrlType): urt.UrlType = urlType match {\n    case ExternalUrl => urt.UrlType.ExternalUrl\n    case DeepLink => urt.UrlType.DeepLink\n    case UrtEndpoint => urt.UrlType.UrtEndpoint\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata/UrtEndpointOptionsMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.UrtEndpointOptions\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UrtEndpointOptionsMarshaller @Inject() () {\n\n  def apply(urtEndpointOptions: UrtEndpointOptions): urt.UrtEndpointOptions =\n    urt.UrtEndpointOptions(\n      requestParams = urtEndpointOptions.requestParams,\n      title = urtEndpointOptions.title,\n      cacheId = urtEndpointOptions.cacheId,\n      subtitle = urtEndpointOptions.subtitle\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/operation/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/operation/CursorDisplayTreatmentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.operation\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorDisplayTreatment\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CursorDisplayTreatmentMarshaller @Inject() () {\n\n  def apply(treatment: CursorDisplayTreatment): urt.CursorDisplayTreatment =\n    urt.CursorDisplayTreatment(\n      actionText = treatment.actionText,\n      labelText = treatment.labelText\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/operation/CursorItemMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.operation\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorItem\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CursorItemMarshaller @Inject() (\n  cursorTypeMarshaller: CursorTypeMarshaller,\n  cursorDisplayTreatmentMarshaller: CursorDisplayTreatmentMarshaller) {\n\n  def apply(cursorItem: CursorItem): urt.TimelineItemContent.TimelineCursor =\n    urt.TimelineItemContent.TimelineCursor(\n      urt.TimelineCursor(\n        value = cursorItem.value,\n        cursorType = cursorTypeMarshaller(cursorItem.cursorType),\n        displayTreatment = cursorItem.displayTreatment.map(cursorDisplayTreatmentMarshaller(_))\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/operation/CursorOperationMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.operation\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorOperation\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CursorOperationMarshaller @Inject() (\n  cursorTypeMarshaller: CursorTypeMarshaller,\n  cursorDisplayTreatmentMarshaller: CursorDisplayTreatmentMarshaller) {\n\n  def apply(cursorOperation: CursorOperation): urt.TimelineOperation.Cursor =\n    urt.TimelineOperation.Cursor(\n      urt.TimelineCursor(\n        value = cursorOperation.value,\n        cursorType = cursorTypeMarshaller(cursorOperation.cursorType),\n        displayTreatment = cursorOperation.displayTreatment.map(cursorDisplayTreatmentMarshaller(_))\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/operation/CursorTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.operation\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation._\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CursorTypeMarshaller @Inject() () {\n\n  def apply(cursorType: CursorType): urt.CursorType = cursorType match {\n    case TopCursor => urt.CursorType.Top\n    case BottomCursor => urt.CursorType.Bottom\n    case GapCursor => urt.CursorType.Gap\n    case PivotCursor => urt.CursorType.Pivot\n    case SubBranchCursor => urt.CursorType.Subbranch\n    case ShowMoreCursor => urt.CursorType.ShowMore\n    case ShowMoreThreadsCursor => urt.CursorType.ShowMoreThreads\n    case ShowMoreThreadsPromptCursor => urt.CursorType.ShowMoreThreadsPrompt\n    case SecondRepliesSectionCursor => urt.CursorType.SecondRepliesSection\n    case ThirdRepliesSectionCursor => urt.CursorType.ThirdRepliesSection\n  }\n\n  def unmarshall(cursorType: urt.CursorType): CursorType = cursorType match {\n    case urt.CursorType.Top => TopCursor\n    case urt.CursorType.Bottom => BottomCursor\n    case urt.CursorType.Gap => GapCursor\n    case urt.CursorType.Pivot => PivotCursor\n    case urt.CursorType.Subbranch => SubBranchCursor\n    case urt.CursorType.ShowMore => ShowMoreCursor\n    case urt.CursorType.ShowMoreThreads => ShowMoreThreadsCursor\n    case urt.CursorType.ShowMoreThreadsPrompt => ShowMoreThreadsPromptCursor\n    case urt.CursorType.SecondRepliesSection => SecondRepliesSectionCursor\n    case urt.CursorType.ThirdRepliesSection => ThirdRepliesSectionCursor\n    case urt.CursorType.EnumUnknownCursorType(id) =>\n      throw new UnsupportedOperationException(s\"Unexpected cursor enum field: $id\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/AdMetadataContainerMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.AdMetadataContainer\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AdMetadataContainerMarshaller @Inject() (\n  sponsorshipTypeMarshaller: SponsorshipTypeMarshaller,\n  disclaimerTypeMarshaller: DisclaimerTypeMarshaller,\n  skAdNetworkDataMarshaller: SkAdNetworkDataMarshaller) {\n\n  def apply(adMetadataContainer: AdMetadataContainer): urt.AdMetadataContainer =\n    urt.AdMetadataContainer(\n      removePromotedAttributionForPreroll = adMetadataContainer.removePromotedAttributionForPreroll,\n      sponsorshipCandidate = adMetadataContainer.sponsorshipCandidate,\n      sponsorshipOrganization = adMetadataContainer.sponsorshipOrganization,\n      sponsorshipOrganizationWebsite = adMetadataContainer.sponsorshipOrganizationWebsite,\n      sponsorshipType = adMetadataContainer.sponsorshipType.map(sponsorshipTypeMarshaller(_)),\n      disclaimerType = adMetadataContainer.disclaimerType.map(disclaimerTypeMarshaller(_)),\n      skAdNetworkDataList =\n        adMetadataContainer.skAdNetworkDataList.map(_.map(skAdNetworkDataMarshaller(_))),\n      unifiedCardOverride = adMetadataContainer.unifiedCardOverride\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/CallToActionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.CallToAction\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Singleton\n\n@Singleton\nclass CallToActionMarshaller {\n  def apply(callToAction: CallToAction): urt.CallToAction = {\n    urt.CallToAction(\n      callToActionType = callToAction.callToActionType,\n      url = callToAction.url\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/ClickTrackingInfoMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.ClickTrackingInfo\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ClickTrackingInfoMarshaller @Inject() (\n  urlOverrideTypeMarshaller: UrlOverrideTypeMarshaller) {\n\n  def apply(clickTrackingInfo: ClickTrackingInfo): urt.ClickTrackingInfo =\n    urt.ClickTrackingInfo(\n      urlParams = clickTrackingInfo.urlParams,\n      urlOverride = clickTrackingInfo.urlOverride,\n      urlOverrideType = clickTrackingInfo.urlOverrideType.map(urlOverrideTypeMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/DisclaimerTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DisclaimerIssue\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DisclaimerPolitical\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DisclaimerType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass DisclaimerTypeMarshaller @Inject() () {\n\n  def apply(disclaimerType: DisclaimerType): urt.DisclaimerType = disclaimerType match {\n    case DisclaimerPolitical => urt.DisclaimerType.Political\n    case DisclaimerIssue => urt.DisclaimerType.Issue\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/DisclosureTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DisclosureType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.Earned\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.Issue\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.NoDisclosure\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.Political\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass DisclosureTypeMarshaller @Inject() () {\n\n  def apply(disclosureType: DisclosureType): urt.DisclosureType = disclosureType match {\n    case NoDisclosure => urt.DisclosureType.NoDisclosure\n    case Political => urt.DisclosureType.Political\n    case Earned => urt.DisclosureType.Earned\n    case Issue => urt.DisclosureType.Issue\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/DynamicPrerollTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.Amplify\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DynamicPrerollType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.LiveTvEvent\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.Marketplace\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass DynamicPrerollTypeMarshaller @Inject() () {\n\n  def apply(dynamicPrerollType: DynamicPrerollType): urt.DynamicPrerollType =\n    dynamicPrerollType match {\n      case Amplify => urt.DynamicPrerollType.Amplify\n      case Marketplace => urt.DynamicPrerollType.Marketplace\n      case LiveTvEvent => urt.DynamicPrerollType.LiveTvEvent\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/MediaInfoMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.MediaInfo\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MediaInfoMarshaller @Inject() (\n  callToActionMarshaller: CallToActionMarshaller,\n  videoVariantsMarshaller: VideoVariantsMarshaller) {\n  def apply(mediaInfo: MediaInfo): urt.MediaInfo = {\n    urt.MediaInfo(\n      uuid = mediaInfo.uuid,\n      publisherId = mediaInfo.publisherId,\n      callToAction = mediaInfo.callToAction.map(callToActionMarshaller(_)),\n      durationMillis = mediaInfo.durationMillis,\n      videoVariants = mediaInfo.videoVariants.map(videoVariantsMarshaller(_)),\n      advertiserName = mediaInfo.advertiserName,\n      renderAdByAdvertiserName = mediaInfo.renderAdByAdvertiserName,\n      advertiserProfileImageUrl = mediaInfo.advertiserProfileImageUrl\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/PrerollMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.Preroll\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PrerollMarshaller @Inject() (\n  dynamicPrerollTypeMarshaller: DynamicPrerollTypeMarshaller,\n  mediaInfoMarshaller: MediaInfoMarshaller) {\n\n  def apply(preroll: Preroll): urt.Preroll =\n    urt.Preroll(\n      prerollId = preroll.prerollId,\n      dynamicPrerollType = preroll.dynamicPrerollType.map(dynamicPrerollTypeMarshaller(_)),\n      mediaInfo = preroll.mediaInfo.map(mediaInfoMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/PrerollMetadataMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PrerollMetadata\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PrerollMetadataMarshaller @Inject() (\n  prerollMarshaller: PrerollMarshaller) {\n  def apply(prerollMetadata: PrerollMetadata): urt.PrerollMetadata =\n    urt.PrerollMetadata(\n      preroll = prerollMetadata.preroll.map(prerollMarshaller(_)),\n      videoAnalyticsScribePassthrough = prerollMetadata.videoAnalyticsScribePassthrough\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/PromotedMetadataMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PromotedMetadata\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PromotedMetadataMarshaller @Inject() (\n  disclosureTypeMarshaller: DisclosureTypeMarshaller,\n  adMetadataContainerMarshaller: AdMetadataContainerMarshaller,\n  clickTrackingInfoMarshaller: ClickTrackingInfoMarshaller) {\n\n  /** See comments on [[com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PromotedMetadata]]\n   * regarding impressionId and impressionString\n   *\n   * TL;DR the domain model only has impressionString (the newer version) an this marshaller sets both\n   * impressionId (the older) and impressionString based on it for compatibility.\n   * */\n  def apply(promotedMetadata: PromotedMetadata): urt.PromotedMetadata =\n    urt.PromotedMetadata(\n      advertiserId = promotedMetadata.advertiserId,\n      impressionId = promotedMetadata.impressionString,\n      disclosureType = promotedMetadata.disclosureType.map(disclosureTypeMarshaller(_)),\n      experimentValues = promotedMetadata.experimentValues,\n      promotedTrendId = promotedMetadata.promotedTrendId,\n      promotedTrendName = promotedMetadata.promotedTrendName,\n      promotedTrendQueryTerm = promotedMetadata.promotedTrendQueryTerm,\n      adMetadataContainer =\n        promotedMetadata.adMetadataContainer.map(adMetadataContainerMarshaller(_)),\n      promotedTrendDescription = promotedMetadata.promotedTrendDescription,\n      impressionString = promotedMetadata.impressionString,\n      clickTrackingInfo = promotedMetadata.clickTrackingInfo.map(clickTrackingInfoMarshaller(_))\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/SkAdNetworkDataMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.SkAdNetworkData\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.timelines.render.{thriftscala => urt}\n\n@Singleton\nclass SkAdNetworkDataMarshaller @Inject() () {\n\n  def apply(skAdNetworkData: SkAdNetworkData): urt.SkAdNetworkData =\n    urt.SkAdNetworkData(\n      version = skAdNetworkData.version,\n      srcAppId = skAdNetworkData.srcAppId,\n      dstAppId = skAdNetworkData.dstAppId,\n      adNetworkId = skAdNetworkData.adNetworkId,\n      campaignId = skAdNetworkData.campaignId,\n      impressionTimeInMillis = skAdNetworkData.impressionTimeInMillis,\n      nonce = skAdNetworkData.nonce,\n      signature = skAdNetworkData.signature,\n      fidelityType = skAdNetworkData.fidelityType\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/SponsorshipTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DirectSponsorshipType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.IndirectSponsorshipType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.NoSponsorshipSponsorshipType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.SponsorshipType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SponsorshipTypeMarshaller @Inject() () {\n\n  def apply(sponsorshipType: SponsorshipType): urt.SponsorshipType = sponsorshipType match {\n    case DirectSponsorshipType => urt.SponsorshipType.Direct\n    case IndirectSponsorshipType => urt.SponsorshipType.Indirect\n    case NoSponsorshipSponsorshipType => urt.SponsorshipType.NoSponsorship\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/UrlOverrideTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.DcmUrlOverrideType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.UnknownUrlOverrideType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.UrlOverrideType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass UrlOverrideTypeMarshaller @Inject() () {\n\n  def apply(urlOverrideType: UrlOverrideType): urt.UrlOverrideType = urlOverrideType match {\n    case UnknownUrlOverrideType => urt.UrlOverrideType.Unknown\n    case DcmUrlOverrideType => urt.UrlOverrideType.Dcm\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/promoted/VideoVariantsMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.promoted\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.VideoVariant\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Singleton\n\n@Singleton\nclass VideoVariantsMarshaller {\n  def apply(videoVariants: Seq[VideoVariant]): Seq[urt.VideoVariant] = {\n    videoVariants.map(videoVariant =>\n      urt.VideoVariant(\n        url = videoVariant.url,\n        contentType = videoVariant.contentType,\n        bitrate = videoVariant.bitrate\n      ))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/reaction/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/reaction\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/reaction\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/reaction/TimelineReactionMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.reaction\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.reaction.ImmediateTimelineReaction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.reaction.RemoteTimelineReaction\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.reaction.TimelineReaction\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TimelineReactionMarshaller @Inject() () {\n  def apply(timelineReaction: TimelineReaction): urt.TimelineReaction = {\n    val execution = timelineReaction.execution match {\n      case ImmediateTimelineReaction(key) =>\n        urt.TimelineReactionExecution.Immediate(urt.ImmediateTimelineReaction(key))\n      case RemoteTimelineReaction(requestParams, timeoutInSeconds) =>\n        urt.TimelineReactionExecution.Remote(\n          urt.RemoteTimelineReaction(\n            requestParams,\n            timeoutInSeconds\n          ))\n    }\n    urt.TimelineReaction(\n      execution = execution,\n      maxExecutionCount = timelineReaction.maxExecutionCount\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/richtext/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/richtext/ReferenceObjectMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.ReferenceObject\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextCashtag\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextHashtag\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextList\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextMention\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextUser\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ReferenceObjectMarshaller @Inject() (urlMarshaller: UrlMarshaller) {\n\n  def apply(ref: ReferenceObject): urt.ReferenceObject = ref match {\n    case url: Url => urt.ReferenceObject.Url(urlMarshaller(url))\n    case user: RichTextUser => urt.ReferenceObject.User(urt.RichTextUser(id = user.id))\n    case mention: RichTextMention =>\n      urt.ReferenceObject.Mention(\n        urt.RichTextMention(id = mention.id, screenName = mention.screenName))\n    case hashtag: RichTextHashtag =>\n      urt.ReferenceObject.Hashtag(urt.RichTextHashtag(text = hashtag.text))\n    case cashtag: RichTextCashtag =>\n      urt.ReferenceObject.Cashtag(urt.RichTextCashtag(text = cashtag.text))\n    case twitterList: RichTextList =>\n      urt.ReferenceObject.TwitterList(urt.RichTextList(id = twitterList.id, url = twitterList.url))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/richtext/RichTextAlignmentMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Center\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Natural\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextAlignment\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RichTextAlignmentMarshaller @Inject() () {\n\n  def apply(alignment: RichTextAlignment): urt.RichTextAlignment = alignment match {\n    case Natural => urt.RichTextAlignment.Natural\n    case Center => urt.RichTextAlignment.Center\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/richtext/RichTextEntityMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextEntity\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RichTextEntityMarshaller @Inject() (\n  referenceObjectMarshaller: ReferenceObjectMarshaller,\n  richTextFormatMarshaller: RichTextFormatMarshaller) {\n\n  def apply(entity: RichTextEntity): urt.RichTextEntity = urt.RichTextEntity(\n    fromIndex = entity.fromIndex,\n    toIndex = entity.toIndex,\n    ref = entity.ref.map(referenceObjectMarshaller(_)),\n    format = entity.format.map(richTextFormatMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/richtext/RichTextFormatMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Plain\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichTextFormat\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.Strong\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RichTextFormatMarshaller @Inject() () {\n\n  def apply(format: RichTextFormat): urt.RichTextFormat = format match {\n    case Plain => urt.RichTextFormat.Plain\n    case Strong => urt.RichTextFormat.Strong\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/richtext/RichTextMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.richtext\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RichTextMarshaller @Inject() (\n  richTextEntityMarshaller: RichTextEntityMarshaller,\n  richTextAlignmentMarshaller: RichTextAlignmentMarshaller) {\n\n  def apply(richText: RichText): urt.RichText = urt.RichText(\n    text = richText.text,\n    entities = richText.entities.map(richTextEntityMarshaller(_)),\n    rtl = richText.rtl,\n    alignment = richText.alignment.map(richTextAlignmentMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module/AdsMetadataMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.AdsMetadata\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AdsMetadataMarshaller @Inject() () {\n\n  def apply(adsMetadata: AdsMetadata): urt.AdsMetadata =\n    urt.AdsMetadata(carouselId = adsMetadata.carouselId)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module\",\n        \"src/thrift/com/twitter/timelines/render:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module/GridCarouselMetadataMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.GridCarouselMetadata\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GridCarouselMetadataMarshaller @Inject() () {\n\n  def apply(gridCarouselMetadata: GridCarouselMetadata): urt.GridCarouselMetadata =\n    urt.GridCarouselMetadata(numRows = gridCarouselMetadata.numRows)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module/ModuleConversationMetadataMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.SocialContextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleConversationMetadata\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ModuleConversationMetadataMarshaller @Inject() (\n  socialContextMarshaller: SocialContextMarshaller) {\n\n  def apply(\n    moduleConversationMetadata: ModuleConversationMetadata\n  ): urt.ModuleConversationMetadata = urt.ModuleConversationMetadata(\n    allTweetIds = moduleConversationMetadata.allTweetIds,\n    socialContext = moduleConversationMetadata.socialContext.map(socialContextMarshaller(_)),\n    enableDeduplication = moduleConversationMetadata.enableDeduplication\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module/ModuleDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Carousel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.CompactCarousel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ConversationTree\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.GridCarousel\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Vertical\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalConversation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalWithContextLine\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.VerticalGrid\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.timelines.render.{thriftscala => urt}\n\n@Singleton\nclass ModuleDisplayTypeMarshaller @Inject() () {\n\n  def apply(displayType: ModuleDisplayType): urt.ModuleDisplayType = displayType match {\n    case Vertical => urt.ModuleDisplayType.Vertical\n    case Carousel => urt.ModuleDisplayType.Carousel\n    case VerticalWithContextLine => urt.ModuleDisplayType.VerticalWithContextLine\n    case VerticalConversation => urt.ModuleDisplayType.VerticalConversation\n    case ConversationTree => urt.ModuleDisplayType.ConversationTree\n    case GridCarousel => urt.ModuleDisplayType.GridCarousel\n    case CompactCarousel => urt.ModuleDisplayType.CompactCarousel\n    case VerticalGrid => urt.ModuleDisplayType.VerticalGrid\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module/ModuleFooterMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.UrlMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleFooter\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport com.twitter.timelines.render.{thriftscala => urt}\n\n@Singleton\nclass ModuleFooterMarshaller @Inject() (urlMarshaller: UrlMarshaller) {\n\n  def apply(footer: ModuleFooter): urt.ModuleFooter = urt.ModuleFooter(\n    text = footer.text,\n    landingUrl = footer.landingUrl.map(urlMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module/ModuleHeaderDisplayTypeMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.Classic\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ClassicNoDivider\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ContextEmphasis\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleHeaderDisplayType\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ModuleHeaderDisplayTypeMarshaller @Inject() () {\n\n  def apply(displayType: ModuleHeaderDisplayType): urt.ModuleHeaderDisplayType =\n    displayType match {\n      case Classic => urt.ModuleHeaderDisplayType.Classic\n      case ContextEmphasis => urt.ModuleHeaderDisplayType.ContextEmphasis\n      case ClassicNoDivider => urt.ModuleHeaderDisplayType.ClassicNoDivider\n    }\n\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module/ModuleHeaderMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.icon.HorizonIconMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.ImageVariantMarshaller\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.metadata.SocialContextMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleHeader\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ModuleHeaderMarshaller @Inject() (\n  horizonIconMarshaller: HorizonIconMarshaller,\n  imageVariantMarshaller: ImageVariantMarshaller,\n  socialContextMarshaller: SocialContextMarshaller,\n  moduleHeaderDisplayTypeMarshaller: ModuleHeaderDisplayTypeMarshaller) {\n\n  def apply(header: ModuleHeader): urt.ModuleHeader = urt.ModuleHeader(\n    text = header.text,\n    sticky = header.sticky,\n    icon = header.icon.map(horizonIconMarshaller(_)),\n    customIcon = header.customIcon.map(imageVariantMarshaller(_)),\n    socialContext = header.socialContext.map(socialContextMarshaller(_)),\n    displayType = moduleHeaderDisplayTypeMarshaller(header.moduleHeaderDisplayType)\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module/ModuleMetadataMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleMetadata\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ModuleMetadataMarshaller @Inject() (\n  adsMetadataMarshaller: AdsMetadataMarshaller,\n  moduleConversationMetadataMarshaller: ModuleConversationMetadataMarshaller,\n  gridCarouselMetadataMarshaller: GridCarouselMetadataMarshaller) {\n\n  def apply(moduleMetadata: ModuleMetadata): urt.ModuleMetadata = urt.ModuleMetadata(\n    adsMetadata = moduleMetadata.adsMetadata.map(adsMetadataMarshaller(_)),\n    conversationMetadata =\n      moduleMetadata.conversationMetadata.map(moduleConversationMetadataMarshaller(_)),\n    gridCarouselMetadata =\n      moduleMetadata.gridCarouselMetadata.map(gridCarouselMetadataMarshaller(_))\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module/ModuleShowMoreBehaviorMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleShowMoreBehavior\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleShowMoreBehaviorRevealByCount\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ModuleShowMoreBehaviorMarshaller @Inject() (\n  moduleShowMoreBehaviorRevealByCountMarshaller: ModuleShowMoreBehaviorRevealByCountMarshaller) {\n\n  def apply(\n    moduleShowMoreBehavior: ModuleShowMoreBehavior\n  ): urt.ModuleShowMoreBehavior = moduleShowMoreBehavior match {\n    case moduleShowMoreBehaviorRevealByCount: ModuleShowMoreBehaviorRevealByCount =>\n      moduleShowMoreBehaviorRevealByCountMarshaller(moduleShowMoreBehaviorRevealByCount)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/timeline_module/ModuleShowMoreBehaviorRevealByCountMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.marshaller.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleShowMoreBehaviorRevealByCount\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ModuleShowMoreBehaviorRevealByCountMarshaller @Inject() () {\n\n  def apply(\n    moduleShowMoreBehaviorRevealByCount: ModuleShowMoreBehaviorRevealByCount\n  ): urt.ModuleShowMoreBehavior =\n    urt.ModuleShowMoreBehavior.RevealByCount(\n      urt.ModuleShowMoreBehaviorRevealByCount(\n        initialItemsCount = moduleShowMoreBehaviorRevealByCount.initialItemsCount,\n        showMoreItemsCount = moduleShowMoreBehaviorRevealByCount.showMoreItemsCount\n      )\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller/DomainMarshaller.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.premarshaller\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.DomainMarshallerIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModulePresentation\nimport com.twitter.product_mixer.core.model.common.presentation.UniversalPresentation\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * Transforms the `selections` into a [[DomainResponseType]] object (often URT, Slice, etc)\n *\n * [[DomainMarshaller]]s may contain business logic\n *\n * @note This is different from `com.twitter.product_mixer.core.marshaller`s\n *       which transforms into a wire-compatible type\n */\ntrait DomainMarshaller[-Query <: PipelineQuery, DomainResponseType] extends Component {\n\n  override val identifier: DomainMarshallerIdentifier\n\n  /** Transforms the `selections` into a [[DomainResponseType]] object */\n  def apply(\n    query: Query,\n    selections: Seq[CandidateWithDetails]\n  ): DomainResponseType\n}\n\nclass UnsupportedCandidateDomainMarshallerException(\n  candidate: Any,\n  candidateSource: ComponentIdentifier)\n    extends UnsupportedOperationException(\n      s\"Domain marshaller does not support candidate ${TransportMarshaller.getSimpleName(\n        candidate.getClass)} from source $candidateSource\")\n\nclass UndecoratedCandidateDomainMarshallerException(\n  candidate: Any,\n  candidateSource: ComponentIdentifier)\n    extends UnsupportedOperationException(\n      s\"Domain marshaller does not support undecorated candidate ${TransportMarshaller\n        .getSimpleName(candidate.getClass)} from source $candidateSource\")\n\nclass UnsupportedPresentationDomainMarshallerException(\n  candidate: Any,\n  presentation: UniversalPresentation,\n  candidateSource: ComponentIdentifier)\n    extends UnsupportedOperationException(\n      s\"Domain marshaller does not support decorator presentation ${TransportMarshaller\n        .getSimpleName(presentation.getClass)} for candidate ${TransportMarshaller.getSimpleName(\n        candidate.getClass)} from source $candidateSource\")\n\nclass UnsupportedModuleDomainMarshallerException(\n  presentation: Option[ModulePresentation],\n  candidateSource: ComponentIdentifier)\n    extends UnsupportedOperationException(\n      s\"Domain marshaller does not support module presentation ${presentation\n        .map(p =>\n          TransportMarshaller\n            .getSimpleName(presentation.getClass)).getOrElse(\"\")} but was given a module from source $candidateSource\")\n\nclass UndecoratedModuleDomainMarshallerException(\n  candidateSource: ComponentIdentifier)\n    extends UnsupportedOperationException(\n      s\"Domain marshaller does not support undecorated module from source $candidateSource\")\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer/ScoredCandidateResult.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.scorer\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\n/** A [[Candidate]] and it's [[FeatureMap]] after being processed by a [[Scorer]] */\ncase class ScoredCandidateResult[Candidate <: UniversalNoun[Any]](\n  candidate: Candidate,\n  scorerResult: FeatureMap)\n    extends CandidateWithFeatures[Candidate] {\n  override val features: FeatureMap = scorerResult\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer/Scorer.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.scorer\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseBulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.SupportsConditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ScorerIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\n\n/** Scores the provided `candidates` */\ntrait Scorer[-Query <: PipelineQuery, -Candidate <: UniversalNoun[Any]]\n    extends BaseBulkCandidateFeatureHydrator[Query, Candidate, Feature[_, _]]\n    with SupportsConditionally[Query] {\n\n  /** @see [[ScorerIdentifier]] */\n  override val identifier: ScorerIdentifier\n\n  /**\n   * Features returned by the Scorer\n   */\n  def features: Set[Feature[_, _]]\n\n  /**\n   * Scores the provided `candidates`\n   *\n   * @note the returned Seq of [[FeatureMap]] must contain all the input 'candidates'\n   * and be in the same order as the input 'candidates'\n   **/\n  def apply(\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]]\n  ): Stitch[Seq[FeatureMap]]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector/Selector.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.selector\n\nimport com.twitter.product_mixer.core.functional_component.common.CandidateScope\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/** Selects some `remainingCandidates` and add them to the `result` */\ntrait Selector[-Query <: PipelineQuery] {\n\n  /**\n   * Specifies which [[com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails.source]]s\n   * this [[Selector]] will apply to.\n   *\n   * @note it is up to each [[Selector]] implementation to correctly handle this behavior\n   */\n  def pipelineScope: CandidateScope\n\n  /** Selects some `remainingCandidates` and add them to the `result` */\n  def apply(\n    query: Query,\n    remainingCandidates: Seq[CandidateWithDetails],\n    result: Seq[CandidateWithDetails]\n  ): SelectorResult\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector/SelectorResult.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.selector\n\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\n\n/**\n * The result of a [[Selector]] where items that were added\n * to the [[result]] are removed from the [[remainingCandidates]]\n */\ncase class SelectorResult(\n  remainingCandidates: Seq[CandidateWithDetails],\n  result: Seq[CandidateWithDetails])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect/ExecuteSynchronously.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.side_effect\n\n/**\n * A modifier for any [[SideEffect]] so that the request waits for it to complete before being returned\n *\n * @note this will make the [[SideEffect]]'s latency impact the overall request's latency\n *\n * @example {{{\n * class MySideEffect extends PipelineResultSideEffect[T] with ExecuteSynchronously {...}\n * }}}\n *\n * @example {{{\n * class MySideEffect extends ScribeLogEventSideEffect[T] with ExecuteSynchronously {...}\n * }}}\n */\ntrait ExecuteSynchronously { _: SideEffect[_] => }\n\n/**\n * A modifier for any [[ExecuteSynchronously]] [[SideEffect]] that makes it so failures will be\n * reported in the results but wont cause the request as a whole to fail.\n */\ntrait FailOpen { _: ExecuteSynchronously => }\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect/PipelineResultSideEffect.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.side_effect\n\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect.Inputs\nimport com.twitter.product_mixer.core.model.common\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\n/**\n * A side-effect that can be run with a pipeline result before transport marshalling\n *\n * @see SideEffect\n *\n * @tparam Query pipeline query\n * @tparam ResultType response after domain marshalling\n */\ntrait PipelineResultSideEffect[-Query <: PipelineQuery, -ResultType <: HasMarshalling]\n    extends SideEffect[Inputs[Query, ResultType]]\n    with PipelineResultSideEffect.SupportsConditionally[Query, ResultType]\n\nobject PipelineResultSideEffect {\n\n  /**\n   * Mixin for when you want to conditionally run a [[PipelineResultSideEffect]]\n   *\n   * This is a thin wrapper around [[common.Conditionally]] exposing a nicer API for the [[PipelineResultSideEffect]] specific use-case.\n   */\n  trait Conditionally[-Query <: PipelineQuery, -ResultType <: HasMarshalling]\n      extends common.Conditionally[Inputs[Query, ResultType]] {\n    _: PipelineResultSideEffect[Query, ResultType] =>\n\n    /** @see [[common.Conditionally.onlyIf]] */\n    def onlyIf(\n      query: Query,\n      selectedCandidates: Seq[CandidateWithDetails],\n      remainingCandidates: Seq[CandidateWithDetails],\n      droppedCandidates: Seq[CandidateWithDetails],\n      response: ResultType\n    ): Boolean\n\n    override final def onlyIf(input: Inputs[Query, ResultType]): Boolean =\n      onlyIf(\n        input.query,\n        input.selectedCandidates,\n        input.remainingCandidates,\n        input.droppedCandidates,\n        input.response)\n\n  }\n\n  type SupportsConditionally[-Query <: PipelineQuery, -ResultType <: HasMarshalling] =\n    common.SupportsConditionally[Inputs[Query, ResultType]]\n\n  case class Inputs[+Query <: PipelineQuery, +ResultType <: HasMarshalling](\n    query: Query,\n    selectedCandidates: Seq[CandidateWithDetails],\n    remainingCandidates: Seq[CandidateWithDetails],\n    droppedCandidates: Seq[CandidateWithDetails],\n    response: ResultType)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect/SideEffect.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.side_effect\n\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.stitch.Stitch\n\n/**\n * A side-effect is a ancillary action that doesn't affect the result of execution directly.\n *\n * For example: Logging, history stores\n *\n * Implementing components can express failures by throwing an exception. These exceptions\n * will be caught and not affect the request processing.\n *\n * @note Side effects execute asynchronously in a fire-and-forget way, it's important to add alerts\n *       to the [[SideEffect]] component itself since a failures wont show up in metrics\n *       that just monitor your pipeline as a whole.\n *\n * @see [[ExecuteSynchronously]] for modifying a [[SideEffect]] to execute with synchronously with\n *      the request waiting on the side effect to complete, this will impact the overall request's latency\n **/\ntrait SideEffect[-Inputs] extends Component {\n\n  /** @see [[SideEffectIdentifier]] */\n  override val identifier: SideEffectIdentifier\n\n  def apply(inputs: Inputs): Stitch[Unit]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer/CandidateFeatureTransformer.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.transformer\n\n/**\n * Populates a [[com.twitter.product_mixer.core.feature.featuremap.FeatureMap]] with Features\n * that are available in the [[CandidateSourceResult]]\n */\ntrait CandidateFeatureTransformer[-CandidateSourceResult]\n    extends FeatureTransformer[CandidateSourceResult]\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer/CandidatePipelineQueryTransformer.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.transformer\n\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.CandidatePipelineResults\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\n\n/**\n * A transformer for transforming a mixer or recommendation pipeline's query type into a candidate\n * pipeline's query type.\n * @tparam Query The parent pipeline's query type\n * @tparam CandidateSourceQuery The Candidate Source's query type that the Query should be converted to\n */\nprotected[core] sealed trait BaseCandidatePipelineQueryTransformer[\n  -Query <: PipelineQuery,\n  +CandidateSourceQuery]\n    extends Transformer[Query, CandidateSourceQuery] {\n\n  override val identifier: TransformerIdentifier =\n    BaseCandidatePipelineQueryTransformer.DefaultTransformerId\n}\n\ntrait CandidatePipelineQueryTransformer[-Query <: PipelineQuery, CandidateSourceQuery]\n    extends BaseCandidatePipelineQueryTransformer[Query, CandidateSourceQuery]\n\ntrait DependentCandidatePipelineQueryTransformer[-Query <: PipelineQuery, CandidateSourceQuery]\n    extends BaseCandidatePipelineQueryTransformer[Query, CandidateSourceQuery] {\n  def transform(query: Query, candidates: Seq[CandidateWithDetails]): CandidateSourceQuery\n\n  final override def transform(query: Query): CandidateSourceQuery = {\n    val candidates = query.features\n      .map(_.get(CandidatePipelineResults)).getOrElse(\n        throw PipelineFailure(\n          IllegalStateFailure,\n          \"Candidate Pipeline Results Feature missing from query features\"))\n    transform(query, candidates)\n  }\n}\n\nobject BaseCandidatePipelineQueryTransformer {\n  private[core] val DefaultTransformerId: TransformerIdentifier =\n    TransformerIdentifier(ComponentIdentifier.BasedOnParentComponent)\n  private[core] val TransformerIdSuffix = \"Query\"\n\n  /**\n   * For use when building a [[BaseCandidatePipelineQueryTransformer]] in a [[com.twitter.product_mixer.core.pipeline.PipelineBuilder]]\n   * to ensure that the identifier is updated with the parent [[com.twitter.product_mixer.core.pipeline.Pipeline.identifier]]\n   */\n  private[core] def copyWithUpdatedIdentifier[Query <: PipelineQuery, CandidateSourceQuery](\n    queryTransformer: BaseCandidatePipelineQueryTransformer[Query, CandidateSourceQuery],\n    parentIdentifier: ComponentIdentifier\n  ): BaseCandidatePipelineQueryTransformer[Query, CandidateSourceQuery] = {\n    if (queryTransformer.identifier == DefaultTransformerId) {\n      val transformerIdentifierFromParentName = TransformerIdentifier(\n        s\"${parentIdentifier.name}$TransformerIdSuffix\")\n      queryTransformer match {\n        case queryTransformer: CandidatePipelineQueryTransformer[Query, CandidateSourceQuery] =>\n          new CandidatePipelineQueryTransformer[Query, CandidateSourceQuery] {\n            override val identifier: TransformerIdentifier = transformerIdentifierFromParentName\n\n            override def transform(input: Query): CandidateSourceQuery =\n              queryTransformer.transform(input)\n          }\n        case queryTransformer: DependentCandidatePipelineQueryTransformer[\n              Query,\n              CandidateSourceQuery\n            ] =>\n          new DependentCandidatePipelineQueryTransformer[Query, CandidateSourceQuery] {\n            override val identifier: TransformerIdentifier = transformerIdentifierFromParentName\n\n            override def transform(\n              input: Query,\n              candidates: Seq[CandidateWithDetails]\n            ): CandidateSourceQuery =\n              queryTransformer.transform(input, candidates)\n          }\n      }\n    } else {\n      queryTransformer\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer/CandidatePipelineResultsTransformer.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.transformer\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\n/**\n * A transformer for transforming a candidate pipeline's source result type into the parent's\n * mixer ore recommendation pipeline's type.\n * @tparam SourceResult The type of the result of the candidate source being used.\n * @tparam PipelineResult The type of the parent pipeline's expected\n */\ntrait CandidatePipelineResultsTransformer[SourceResult, PipelineResult <: UniversalNoun[Any]]\n    extends Transformer[SourceResult, PipelineResult] {\n\n  override val identifier: TransformerIdentifier =\n    CandidatePipelineResultsTransformer.DefaultTransformerId\n}\n\nobject CandidatePipelineResultsTransformer {\n  private[core] val DefaultTransformerId: TransformerIdentifier =\n    TransformerIdentifier(ComponentIdentifier.BasedOnParentComponent)\n  private[core] val TransformerIdSuffix = \"Results\"\n\n  /**\n   * For use when building a [[CandidatePipelineResultsTransformer]] in a [[com.twitter.product_mixer.core.pipeline.PipelineBuilder]]\n   * to ensure that the identifier is updated with the parent [[com.twitter.product_mixer.core.pipeline.Pipeline.identifier]]\n   */\n  private[core] def copyWithUpdatedIdentifier[SourceResult, PipelineResult <: UniversalNoun[Any]](\n    resultTransformer: CandidatePipelineResultsTransformer[SourceResult, PipelineResult],\n    parentIdentifier: ComponentIdentifier\n  ): CandidatePipelineResultsTransformer[SourceResult, PipelineResult] = {\n    if (resultTransformer.identifier == DefaultTransformerId) {\n      new CandidatePipelineResultsTransformer[SourceResult, PipelineResult] {\n        override val identifier: TransformerIdentifier = TransformerIdentifier(\n          s\"${parentIdentifier.name}$TransformerIdSuffix\")\n\n        override def transform(input: SourceResult): PipelineResult =\n          resultTransformer.transform(input)\n      }\n    } else {\n      resultTransformer\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer/FeatureTransformer.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.transformer\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\n/**\n * [[FeatureTransformer]] allow you to populate a [[com.twitter.product_mixer.core.feature.Feature]]s\n * value which is already available or can be derived without making an RPC.\n *\n * A [[FeatureTransformer]] transforms a given [[Inputs]] into a [[FeatureMap]].\n * The transformer must specify which [[com.twitter.product_mixer.core.feature.Feature]]s it will populate using the `features` field\n * and the returned [[FeatureMap]] must always have the specified [[com.twitter.product_mixer.core.feature.Feature]]s populated.\n *\n * @note Unlike [[com.twitter.product_mixer.core.functional_component.feature_hydrator.FeatureHydrator]] implementations,\n *       an exception thrown in a [[FeatureTransformer]] will not be added to the [[FeatureMap]] and will instead be\n *       bubble up to the calling pipeline's [[com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureClassifier]].\n */\ntrait FeatureTransformer[-Inputs] extends Transformer[Inputs, FeatureMap] {\n\n  def features: Set[Feature[_, _]]\n\n  override val identifier: TransformerIdentifier\n\n  /** Hydrates a [[FeatureMap]] for a given [[Inputs]] */\n  override def transform(input: Inputs): FeatureMap\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer/Transformer.scala",
    "content": "package com.twitter.product_mixer.core.functional_component.transformer\n\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\n/**\n * A transformer is a synchronous transformation that takes the provided [[Input]] and returns some\n * defined [[Output]]. For example, extracting a score from from a scored candidates.\n */\ntrait Transformer[-Inputs, +Output] extends Component {\n  override val identifier: TransformerIdentifier\n\n  /** Takes [[Inputs]] and transformers them into some [[Output]] of your choosing. */\n  def transform(input: Inputs): Output\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/gate/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/gate/DenyLoggedOutUsersGate.scala",
    "content": "package com.twitter.product_mixer.core.gate\n\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.gate.DenyLoggedOutUsersGate.Suffix\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.Authentication\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.stitch.Stitch\n\ncase class DenyLoggedOutUsersGate(pipelineIdentifier: ComponentIdentifier)\n    extends Gate[PipelineQuery] {\n  override val identifier: GateIdentifier = GateIdentifier(pipelineIdentifier + Suffix)\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] = {\n    if (query.getUserOrGuestId.nonEmpty) {\n      Stitch.value(!query.isLoggedOut)\n    } else {\n      Stitch.exception(\n        PipelineFailure(\n          Authentication,\n          \"Expected either a `userId` (for logged in users) or `guestId` (for logged out users) but found neither\"\n        ))\n    }\n  }\n}\n\nobject DenyLoggedOutUsersGate {\n  val Suffix = \"DenyLoggedOutUsers\"\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/gate/ParamGate.scala",
    "content": "package com.twitter.product_mixer.core.gate\n\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\ncase class ParamGate(name: String, param: Param[Boolean])(implicit file: sourcecode.File)\n    extends Gate[PipelineQuery] {\n\n  // From a customer-perspective, it's more useful to see the file that created the ParamGate\n  override val identifier: GateIdentifier = GateIdentifier(name)(file)\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] =\n    Stitch.value(query.params(param))\n}\n\nobject ParamGate {\n  val EnabledGateSuffix = \"Enabled\"\n  val SupportedClientGateSuffix = \"SupportedClient\"\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/gate/ParamNotGate.scala",
    "content": "package com.twitter.product_mixer.core.gate\n\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Param\n\ncase class ParamNotGate(name: String, param: Param[Boolean]) extends Gate[PipelineQuery] {\n  override val identifier: GateIdentifier = GateIdentifier(name)\n\n  override def shouldContinue(query: PipelineQuery): Stitch[Boolean] =\n    Stitch.value(!query.params(param))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/fasterxml/jackson/core:jackson-annotations\",\n        \"3rdparty/jvm/com/fasterxml/jackson/core:jackson-core\",\n        \"3rdparty/jvm/com/fasterxml/jackson/core:jackson-databind\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n    ],\n    exports = [\n        \"3rdparty/jvm/com/fasterxml/jackson/core:jackson-annotations\",\n        \"3rdparty/jvm/com/fasterxml/jackson/core:jackson-core\",\n        \"3rdparty/jvm/com/fasterxml/jackson/core:jackson-databind\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/CandidateWithFeatures.scala",
    "content": "package com.twitter.product_mixer.core.model.common\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\n\n/** [[Candidate]] and it's FeatureMap */\ntrait CandidateWithFeatures[+Candidate <: UniversalNoun[Any]] {\n  val candidate: Candidate\n  val features: FeatureMap\n}\n\nobject CandidateWithFeatures {\n  def unapply[Candidate <: UniversalNoun[Any]](\n    candidateWithFeatures: CandidateWithFeatures[Candidate]\n  ): Option[(Candidate, FeatureMap)] =\n    Some(\n      (candidateWithFeatures.candidate, candidateWithFeatures.features)\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/Component.scala",
    "content": "package com.twitter.product_mixer.core.model.common\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.HasComponentIdentifier\n\n/**\n * Components are very generically reusable composable pieces\n * Components are uniquely identifiable and centrally registered\n */\ntrait Component extends HasComponentIdentifier {\n\n  /** @see [[ComponentIdentifier]] */\n  override val identifier: ComponentIdentifier\n\n  /** the [[Alert]]s that will be used for this component. */\n  val alerts: Seq[Alert] = Seq.empty\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/Conditionally.scala",
    "content": "package com.twitter.product_mixer.core.model.common\n\n/**\n * A mixin trait that can be added to a [[Component]] that's marked with [[SupportsConditionally]]\n * A [[Component]] with [[SupportsConditionally]] and [[Conditionally]] will only be run when `onlyIf` returns true\n * if `onlyIf` returns false, the [[Component]] is skipped and no stats are recorded for it.\n *\n * @note if an exception is thrown when evaluating `onlyIf`, it will bubble up to the containing `Pipeline`,\n *       however the [[Component]]'s stats will not be incremented. Because of this `onlyIf` should never throw.\n *\n * @note each [[Component]] that [[SupportsConditionally]] has an implementation with in the\n *       component library that will conditionally run the component based on a [[com.twitter.timelines.configapi.Param]]\n *\n * @note [[Conditionally]] functionality is wired into the Component's Executor.\n *\n * @tparam Input the input that is used to gate a component on or off\n */\ntrait Conditionally[-Input] { _: SupportsConditionally[Input] =>\n\n  /**\n   * if `onlyIf` returns true, the underling [[Component]] is run, otherwise it's skipped\n   * @note must not throw\n   */\n  def onlyIf(query: Input): Boolean\n}\n\n/**\n * Marker trait added  to the base type for each [[Component]] which supports the [[Conditionally]] mixin\n *\n * @note this is `private[core]` because it can only be added to the base implementation of components by the Product Mixer team\n *\n * @tparam Input the input that is used to gate a component on or off if [[Conditionally]] is mixed in\n */\nprivate[core] trait SupportsConditionally[-Input] { _: Component => }\n\nobject Conditionally {\n\n  /**\n   * Helper method for combining the [[Conditionally.onlyIf]] of an underlying [[Component]] with an additional predicate\n   */\n  def and[ComponentType <: Component, Input](\n    query: Input,\n    component: ComponentType with SupportsConditionally[Input],\n    onlyIf: Boolean\n  ): Boolean =\n    onlyIf && {\n      component match {\n        // @unchecked is safe here because the type parameter is guaranteed by\n        // the `SupportsConditionally[Input]` type parameter\n        case underlying: Conditionally[Input @unchecked] =>\n          underlying.onlyIf(query)\n        case _ =>\n          true\n      }\n    }\n\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/UniversalNoun.scala",
    "content": "package com.twitter.product_mixer.core.model.common\n\nimport com.fasterxml.jackson.annotation.JsonTypeInfo\n\n@JsonTypeInfo(include = JsonTypeInfo.As.PROPERTY, use = JsonTypeInfo.Id.NAME)\ntrait UniversalNoun[+T] extends Equals {\n  def id: T\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/lihaoyi:sourcecode\",\n        \"util/util-core\",\n        \"util/util-jackson/src/main/scala/com/twitter/util/jackson\",\n    ],\n    exports = [\n        \"3rdparty/jvm/com/lihaoyi:sourcecode\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/CandidatePipelineIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Candidate Pipeline identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class CandidatePipelineIdentifier(override val name: String)\n    extends ComponentIdentifier(\"CandidatePipeline\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[CandidatePipelineIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: CandidatePipelineIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject CandidatePipelineIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): CandidatePipelineIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new CandidatePipelineIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal CandidatePipelineIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/CandidateSourceIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Candidate Source identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class CandidateSourceIdentifier(override val name: String)\n    extends ComponentIdentifier(\"CandidateSource\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[CandidateSourceIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: CandidateSourceIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject CandidateSourceIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): CandidateSourceIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new CandidateSourceIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal CandidateSourceIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/ComponentIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\nimport com.fasterxml.jackson.databind.annotation.JsonSerialize\nimport com.twitter.conversions.StringOps\nimport scala.util.matching.Regex\n\n/**\n * Component Identifiers are a type of identifier used in product mixer to identify\n * unique components - products, pipelines, candidate sources.\n *\n * Each identifier has two parts - a type and a name. Subclasses of [[ComponentIdentifier]]\n * should hardcode the `componentType`, and be declared in this file.\n *\n * For example, a [[ProductPipelineIdentifier]] has the type \"ProductPipeline\".\n *\n * Component identifiers are used in:\n *   - Logs\n *   - Tooling\n *   - Metrics\n *   - Feature Switches\n *\n  * A component identifier name is restricted to:\n *   - 3 to 80 characters to ensure reasonable length\n *   - A-Z, a-z, and Digits\n *   - Must start with A-Z\n *   - Digits only on the ends of \"words\"\n *   - Examples include \"AlphaSample\" and \"UsersLikeMe\"\n *   - and \"SimsV2\" or \"Test6\"\n *\n * Avoid including types like \"Pipeline\", \"MixerPipeline\" etc in your identifier. these\n * can be implied by the type itself, and will automatically be used where appropriate (logs etc).\n */\n@JsonSerialize(using = classOf[ComponentIdentifierSerializer])\nabstract class ComponentIdentifier(\n  val componentType: String,\n  val name: String)\n    extends Equals {\n\n  val file: sourcecode.File = \"\"\n\n  override val toString: String = s\"$name$componentType\"\n\n  val snakeCase: String = StringOps.toSnakeCase(toString)\n\n  val toScopes: Seq[String] = Seq(componentType, name)\n}\n\nobject ComponentIdentifier {\n  // Allows for CamelCase and CamelCaseVer3 styles\n  val AllowedCharacters: Regex = \"([A-Z][A-Za-z]*[0-9]*)+\".r\n  val MinLength = 3\n  val MaxLength = 80\n\n  /**\n   * When a [[ComponentIdentifier.name]] is [[BasedOnParentComponent]]\n   * then when operations that depend on the [[ComponentIdentifier]]\n   * are performed, like registering and stats, we will perform that\n   * operation by substituting the [[ComponentIdentifier.name]] with\n   * the parent component's [[ComponentIdentifier.name]].\n   */\n  private[core] val BasedOnParentComponent = \"BasedOnParentComponent\"\n\n  def isValidName(name: String): Boolean = {\n    name match {\n      case n if n.length < MinLength =>\n        false\n      case n if n.length > MaxLength =>\n        false\n      case AllowedCharacters(_*) =>\n        true\n      case _ =>\n        false\n    }\n  }\n\n  implicit val ordering: Ordering[ComponentIdentifier] =\n    Ordering.by { component =>\n      val componentTypeRank = component match {\n        case _: ProductIdentifier => 0\n        case _: ProductPipelineIdentifier => 1\n        case _: MixerPipelineIdentifier => 2\n        case _: RecommendationPipelineIdentifier => 3\n        case _: ScoringPipelineIdentifier => 4\n        case _: CandidatePipelineIdentifier => 5\n        case _: PipelineStepIdentifier => 6\n        case _: CandidateSourceIdentifier => 7\n        case _: FeatureHydratorIdentifier => 8\n        case _: GateIdentifier => 9\n        case _: FilterIdentifier => 10\n        case _: TransformerIdentifier => 11\n        case _: ScorerIdentifier => 12\n        case _: DecoratorIdentifier => 13\n        case _: DomainMarshallerIdentifier => 14\n        case _: TransportMarshallerIdentifier => 15\n        case _: SideEffectIdentifier => 16\n        case _: PlatformIdentifier => 17\n        case _: SelectorIdentifier => 18\n        case _ => Int.MaxValue\n      }\n\n      // First rank by type, then by name for equivalent types for overall order stability\n      (componentTypeRank, component.name)\n    }\n}\n\n/**\n * HasComponentIdentifier indicates that component has a [[ComponentIdentifier]]\n */\ntrait HasComponentIdentifier {\n  val identifier: ComponentIdentifier\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/ComponentIdentifierSerializer.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\nimport com.fasterxml.jackson.core.JsonGenerator\nimport com.fasterxml.jackson.databind.JsonSerializer\nimport com.fasterxml.jackson.databind.SerializerProvider\n\nprivate[identifier] class ComponentIdentifierSerializer()\n    extends JsonSerializer[ComponentIdentifier] {\n\n  private case class SerializableComponentIdentifier(\n    identifier: String,\n    sourceFile: String)\n\n  override def serialize(\n    componentIdentifier: ComponentIdentifier,\n    gen: JsonGenerator,\n    serializers: SerializerProvider\n  ): Unit = serializers.defaultSerializeValue(\n    SerializableComponentIdentifier(componentIdentifier.toString, componentIdentifier.file.value),\n    gen)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/ComponentIdentifierStack.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\nimport com.fasterxml.jackson.databind.annotation.JsonSerialize\n\n/**\n * A non-empty immutable stack of [[ComponentIdentifier]]s\n *\n * [[ComponentIdentifierStack]] does not support removing [[ComponentIdentifier]]s,\n * instead a [[ComponentIdentifierStack]] should be used by adding new [[ComponentIdentifier]]s\n * as processing enters a given `Component`, then discarded after.\n * Think of this as similar to a let-scoped variable, where the let-scope is the given component.\n */\n@JsonSerialize(using = classOf[ComponentIdentifierStackSerializer])\nclass ComponentIdentifierStack private (val componentIdentifiers: List[ComponentIdentifier]) {\n\n  /** Make a new [[ComponentIdentifierStack]] with the [[ComponentIdentifier]] added at the top */\n  def push(newComponentIdentifier: ComponentIdentifier): ComponentIdentifierStack =\n    new ComponentIdentifierStack(newComponentIdentifier :: componentIdentifiers)\n\n  /** Make a new [[ComponentIdentifierStack]] with the [[ComponentIdentifier]]s added at the top */\n  def push(newComponentIdentifiers: ComponentIdentifierStack): ComponentIdentifierStack =\n    new ComponentIdentifierStack(\n      newComponentIdentifiers.componentIdentifiers ::: componentIdentifiers)\n\n  /** Make a new [[ComponentIdentifierStack]] with the [[ComponentIdentifier]]s added at the top */\n  def push(newComponentIdentifiers: Option[ComponentIdentifierStack]): ComponentIdentifierStack = {\n    newComponentIdentifiers match {\n      case Some(newComponentIdentifiers) => push(newComponentIdentifiers)\n      case None => this\n    }\n  }\n\n  /** Return the top element of the [[ComponentIdentifierStack]] */\n  val peek: ComponentIdentifier = componentIdentifiers.head\n\n  /** Return the size of the [[ComponentIdentifierStack]] */\n  def size: Int = componentIdentifiers.length\n\n  override def toString: String =\n    s\"ComponentIdentifierStack(componentIdentifiers = $componentIdentifiers)\"\n\n  override def equals(obj: Any): Boolean = {\n    obj match {\n      case componentIdentifierStack: ComponentIdentifierStack\n          if componentIdentifierStack.eq(this) ||\n            componentIdentifierStack.componentIdentifiers == componentIdentifiers =>\n        true\n      case _ => false\n    }\n  }\n}\n\nobject ComponentIdentifierStack {\n\n  /**\n   * Returns a [[ComponentIdentifierStack]] from the given [[ComponentIdentifier]]s,\n   * where the top of the stack is the left-most [[ComponentIdentifier]]\n   */\n  def apply(\n    componentIdentifier: ComponentIdentifier,\n    componentIdentifierStack: ComponentIdentifier*\n  ) =\n    new ComponentIdentifierStack(componentIdentifier :: componentIdentifierStack.toList)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/ComponentIdentifierStackSerializer.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\nimport com.fasterxml.jackson.core.JsonGenerator\nimport com.fasterxml.jackson.databind.JsonSerializer\nimport com.fasterxml.jackson.databind.SerializerProvider\n\nprivate[identifier] class ComponentIdentifierStackSerializer()\n    extends JsonSerializer[ComponentIdentifierStack] {\n  override def serialize(\n    componentIdentifierStack: ComponentIdentifierStack,\n    gen: JsonGenerator,\n    serializers: SerializerProvider\n  ): Unit = serializers.defaultSerializeValue(componentIdentifierStack.componentIdentifiers, gen)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/DecoratorIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Decorator identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class DecoratorIdentifier(override val name: String)\n    extends ComponentIdentifier(\"Decorator\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[DecoratorIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: DecoratorIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject DecoratorIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): DecoratorIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new DecoratorIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal DecoratorIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/DomainMarshallerIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Domain Marshaller identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class DomainMarshallerIdentifier(override val name: String)\n    extends ComponentIdentifier(\"DomainMarshaller\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[DomainMarshallerIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: DomainMarshallerIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject DomainMarshallerIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): DomainMarshallerIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new DomainMarshallerIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal DomainMarshallerIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/FeatureHydratorIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Feature Hydrator identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class FeatureHydratorIdentifier(override val name: String)\n    extends ComponentIdentifier(\"FeatureHydrator\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[FeatureHydratorIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: FeatureHydratorIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject FeatureHydratorIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): FeatureHydratorIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new FeatureHydratorIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal FeatureHydratorIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/FilterIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Filter identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class FilterIdentifier(override val name: String)\n    extends ComponentIdentifier(\"Filter\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[FilterIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: FilterIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject FilterIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): FilterIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new FilterIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal FilterIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/GateIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Gate identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class GateIdentifier(override val name: String)\n    extends ComponentIdentifier(\"Gate\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[GateIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: GateIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject GateIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): GateIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new GateIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal GateIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/MixerPipelineIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Mixer Pipeline identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class MixerPipelineIdentifier(override val name: String)\n    extends ComponentIdentifier(\"MixerPipeline\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[MixerPipelineIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: MixerPipelineIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject MixerPipelineIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): MixerPipelineIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new MixerPipelineIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal MixerPipelineIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/PipelineStepIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Pipeline Step identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class PipelineStepIdentifier(\n  override val name: String)\n    extends ComponentIdentifier(\"Step\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[PipelineStepIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: PipelineStepIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nclass Person(val name: String, val age: Int) extends Equals {\n  override def canEqual(that: Any): Boolean =\n    that.isInstanceOf[Person]\n\n  //Intentionally avoiding the call to super.equals because no ancestor has overridden equals (see note 7 below)\n  override def equals(that: Any): Boolean =\n    that match {\n      case person: Person =>\n        (this eq person) || (hashCode == person.hashCode) && ((name == person.name) && (age == person.age))\n\n      case _ =>\n        false\n    }\n\n  //Intentionally avoiding the call to super.hashCode because no ancestor has overridden hashCode (see note 7 below)\n  override def hashCode(): Int =\n    31 * (\n      name.##\n    ) + age.##\n}\n\nobject PipelineStepIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): PipelineStepIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new PipelineStepIdentifier(name) { override val file: sourcecode.File = sourceFile }\n    else\n      throw new IllegalArgumentException(s\"Illegal StepIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/PlatformIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * [[ComponentIdentifier]] type used by internal parts of Product Mixer that need to be identified\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class PlatformIdentifier(override val name: String)\n    extends ComponentIdentifier(\"Platform\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[PlatformIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: PlatformIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject PlatformIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): PlatformIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new PlatformIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal PlatformIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/ProductIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Product identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class ProductIdentifier(override val name: String)\n    extends ComponentIdentifier(\"Product\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[ProductIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: ProductIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject ProductIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): ProductIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new ProductIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal ProductIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/ProductPipelineIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Product Pipeline identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class ProductPipelineIdentifier(override val name: String)\n    extends ComponentIdentifier(\"ProductPipeline\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[ProductPipelineIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: ProductPipelineIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject ProductPipelineIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): ProductPipelineIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new ProductPipelineIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal ProductPipelineIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/RecommendationPipelineIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Recommendation Pipeline identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class RecommendationPipelineIdentifier(override val name: String)\n    extends ComponentIdentifier(\"RecommendationPipeline\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[RecommendationPipelineIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: RecommendationPipelineIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject RecommendationPipelineIdentifier {\n  def apply(\n    name: String\n  )(\n    implicit sourceFile: sourcecode.File\n  ): RecommendationPipelineIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new RecommendationPipelineIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal RecommendationPipelineIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/RootIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Root identifier used as the root identifier for products during component registration\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class RootIdentifier extends ComponentIdentifier(\"Root\", \"\") {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[RootIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: RootIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject RootIdentifier {\n  def apply()(implicit sourceFile: sourcecode.File): RootIdentifier = {\n    new RootIdentifier() {\n      override val file: sourcecode.File = sourceFile\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/ScorerIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Scorer identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class ScorerIdentifier(override val name: String)\n    extends ComponentIdentifier(\"Scorer\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[ScorerIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: ScorerIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject ScorerIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): ScorerIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new ScorerIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal ScorerIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/ScoringPipelineIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Scoring Pipeline identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class ScoringPipelineIdentifier(override val name: String)\n    extends ComponentIdentifier(\"ScoringPipeline\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[ScoringPipelineIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: ScoringPipelineIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject ScoringPipelineIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): ScoringPipelineIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new ScoringPipelineIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal ScoringPipelineIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/SelectorIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Selector identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class SelectorIdentifier(override val name: String)\n    extends ComponentIdentifier(\"Selector\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[SelectorIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: SelectorIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\n/**\n * Used in `SelectorExecutor` to give an ID to each selector in the `ComponentIdentifierStack`\n *\n * These are often generated automatically in the executor and are dependent on the class names\n * so we skip validation to avoid issues. Since we dont record stats for Selectors this is okay.\n */\nprivate[core] object SelectorIdentifier {\n  def apply(\n    name: String,\n    index: Int\n  )(\n    implicit sourceFile: sourcecode.File\n  ): SelectorIdentifier = {\n    val capitalizedWithoutSpecialCharacters = name.replace(\"$\", \"\").capitalize\n    new SelectorIdentifier(index.toString + capitalizedWithoutSpecialCharacters) {\n      override val file: sourcecode.File = sourceFile\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/SideEffectIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Side Effect identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class SideEffectIdentifier(override val name: String)\n    extends ComponentIdentifier(\"SideEffect\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[SideEffectIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: SideEffectIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject SideEffectIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): SideEffectIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new SideEffectIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal SideEffectIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/TransformerIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Transformer identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class TransformerIdentifier(override val name: String)\n    extends ComponentIdentifier(\"Transformer\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[TransformerIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: TransformerIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject TransformerIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): TransformerIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new TransformerIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal TransformerIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier/TransportMarshallerIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.common.identifier\n\n/**\n * Transport Marshaller identifier\n *\n * @note This class should always remain effectively `final`. If for any reason the `sealed`\n *       modifier is removed, the equals() implementation must be updated in order to handle class\n *       inheritor equality (see note on the equals method below)\n */\nsealed abstract class TransportMarshallerIdentifier(override val name: String)\n    extends ComponentIdentifier(\"TransportMarshaller\", name) {\n\n  /**\n   * @inheritdoc\n   */\n  override def canEqual(that: Any): Boolean = that.isInstanceOf[TransportMarshallerIdentifier]\n\n  /**\n   * High performance implementation of equals method that leverages:\n   *  - Referential equality short circuit\n   *  - Cached hashcode equality short circuit\n   *  - Field values are only checked if the hashCodes are equal to handle the unlikely case\n   *    of a hashCode collision\n   *  - Removal of check for `that` being an equals-compatible descendant since this class is final\n   *\n   * @note `candidate.canEqual(this)` is not necessary because this class is final\n   * @see [[http://www.artima.com/pins1ed/object-equality.html Programming in Scala,\n   *      Chapter 28]] for discussion and design.\n   */\n  override def equals(that: Any): Boolean =\n    that match {\n      case identifier: TransportMarshallerIdentifier =>\n        // Note identifier.canEqual(this) is not necessary because this class is effectively final\n        ((this eq identifier)\n          || ((hashCode == identifier.hashCode) && ((componentType == identifier.componentType) && (name == identifier.name))))\n      case _ =>\n        false\n    }\n\n  /**\n   * Leverage domain-specific constraints (see notes below) to safely construct and cache the\n   * hashCode as a val, such that it is instantiated once on object construction. This prevents the\n   * need to recompute the hashCode on each hashCode() invocation, which is the behavior of the\n   * Scala compiler case class-generated hashCode() since it cannot make assumptions regarding field\n   * object mutability and hashCode implementations.\n   *\n   * @note Caching the hashCode is only safe if all of the fields used to construct the hashCode\n   *       are immutable. This includes:\n   *       - Inability to mutate the object reference on for an existing instantiated identifier\n   *       (i.e. each field is a val)\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       - Inability to mutate the field object instance itself (i.e. each field is an immutable\n   *       data structure), assuming stable hashCode implementations for these objects\n   *\n   * @note In order for the hashCode to be consistent with object equality, `##` must be used for\n   *       boxed numeric types and null. As such, always prefer `.##` over `.hashCode()`.\n   */\n  override val hashCode: Int = 31 * componentType.## + name.##\n}\n\nobject TransportMarshallerIdentifier {\n  def apply(name: String)(implicit sourceFile: sourcecode.File): TransportMarshallerIdentifier = {\n    if (ComponentIdentifier.isValidName(name))\n      new TransportMarshallerIdentifier(name) {\n        override val file: sourcecode.File = sourceFile\n      }\n    else\n      throw new IllegalArgumentException(s\"Illegal TransportMarshallerIdentifier: $name\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/CandidateFeatures.scala",
    "content": "package com.twitter.product_mixer.core.model.common.presentation\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport scala.collection.immutable.ListSet\n\n/**\n * A list set of all the candidate pipelines a candidate originated from. This is typically a\n * single element set, but merging candidates across pipelines using\n * [[com.twitter.product_mixer.component_library.selector.CombineFeatureMapsCandidateMerger]]\n * will merge sets for the candidate. The last element of the set is the first pipeline identifier\n * as we prepend new ones since we want O(1) access for the last element.\n */\nobject CandidatePipelines extends Feature[UniversalNoun[Any], ListSet[CandidatePipelineIdentifier]]\n\n/**\n * A list set of all the candidate sources a candidate originated from. This is typically a\n * single element set, but merging candidates across pipelines using\n * [[com.twitter.product_mixer.component_library.selector.CombineFeatureMapsCandidateMerger]]\n * will merge sets for the candidate. The last element of the set is the first source identifier\n * as we prepend new ones since we want O(1) access for the last element.\n */\nobject CandidateSources extends Feature[UniversalNoun[Any], ListSet[CandidateSourceIdentifier]]\n\n/**\n * The source position relative to all candidates the originating candidate source a candidate\n * came from. When merged with other candidates, the position from the first candidate source\n * takes priority.\n */\nobject CandidateSourcePosition extends Feature[UniversalNoun[Any], Int]\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/CandidateWithDetails.scala",
    "content": "package com.twitter.product_mixer.core.model.common.presentation\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.UnexpectedCandidateResult\nimport scala.collection.immutable.ListSet\nimport scala.reflect.ClassTag\n\nsealed trait CandidateWithDetails { self =>\n  def presentation: Option[UniversalPresentation]\n  def features: FeatureMap\n\n  // last of the set because in ListSet, the last element is the first inserted one with O(1)\n  // access\n  lazy val source: CandidatePipelineIdentifier = features.get(CandidatePipelines).last\n  lazy val sourcePosition: Int = features.get(CandidateSourcePosition)\n\n  /**\n   * @see [[getCandidateId]]\n   */\n  def candidateIdLong: Long = getCandidateId[Long]\n\n  /**\n   * @see [[getCandidateId]]\n   */\n  def candidateIdString: String = getCandidateId[String]\n\n  /**\n   * Convenience method for retrieving a candidate ID off of the base [[CandidateWithDetails]] trait\n   * without manually pattern matching.\n   *\n   * @throws PipelineFailure if CandidateIdType does not match the expected Item Candidate Id type,\n   *                         or if invoked on a Module Candidate\n   */\n  def getCandidateId[CandidateIdType](\n  )(\n    implicit tag: ClassTag[CandidateIdType]\n  ): CandidateIdType =\n    self match {\n      case item: ItemCandidateWithDetails =>\n        item.candidate.id match {\n          case id: CandidateIdType => id\n          case _ =>\n            throw PipelineFailure(\n              UnexpectedCandidateResult,\n              s\"Invalid Item Candidate ID type expected $tag for Item Candidate type ${item.candidate.getClass}\")\n        }\n      case _: ModuleCandidateWithDetails =>\n        throw PipelineFailure(\n          UnexpectedCandidateResult,\n          \"Cannot retrieve Item Candidate ID for a Module\")\n    }\n\n  /**\n   * Convenience method for retrieving a candidate off of the base [[CandidateWithDetails]] trait\n   * without manually pattern matching.\n   *\n   * @throws PipelineFailure if CandidateType does not match the expected Item Candidate type, or\n   *                         if invoked on a Module Candidate\n   */\n  def getCandidate[CandidateType <: UniversalNoun[_]](\n  )(\n    implicit tag: ClassTag[CandidateType]\n  ): CandidateType =\n    self match {\n      case ItemCandidateWithDetails(candidate: CandidateType, _, _) => candidate\n      case item: ItemCandidateWithDetails =>\n        throw PipelineFailure(\n          UnexpectedCandidateResult,\n          s\"Invalid Item Candidate type expected $tag for Item Candidate type ${item.candidate.getClass}\")\n      case _: ModuleCandidateWithDetails =>\n        throw PipelineFailure(\n          UnexpectedCandidateResult,\n          \"Cannot retrieve Item Candidate for a Module\")\n    }\n\n  /**\n   * Convenience method for checking if this contains a certain candidate type\n   *\n   * @throws PipelineFailure if CandidateType does not match the expected Item Candidate type, or\n   *                         if invoked on a Module Candidate\n   */\n  def isCandidateType[CandidateType <: UniversalNoun[_]](\n  )(\n    implicit tag: ClassTag[CandidateType]\n  ): Boolean = self match {\n    case ItemCandidateWithDetails(_: CandidateType, _, _) => true\n    case _ => false\n  }\n}\n\ncase class ItemCandidateWithDetails(\n  override val candidate: UniversalNoun[Any],\n  presentation: Option[UniversalPresentation],\n  override val features: FeatureMap)\n    extends CandidateWithDetails\n    with CandidateWithFeatures[UniversalNoun[Any]]\n\ncase class ModuleCandidateWithDetails(\n  candidates: Seq[ItemCandidateWithDetails],\n  presentation: Option[ModulePresentation],\n  override val features: FeatureMap)\n    extends CandidateWithDetails\n\nobject ItemCandidateWithDetails {\n  def apply(\n    candidate: UniversalNoun[Any],\n    presentation: Option[UniversalPresentation],\n    source: CandidatePipelineIdentifier,\n    sourcePosition: Int,\n    features: FeatureMap\n  ): ItemCandidateWithDetails = {\n    val newFeatureMap =\n      FeatureMapBuilder()\n        .add(CandidateSourcePosition, sourcePosition)\n        .add(CandidatePipelines, ListSet.empty + source).build() ++ features\n    ItemCandidateWithDetails(candidate, presentation, newFeatureMap)\n  }\n}\n\nobject ModuleCandidateWithDetails {\n  def apply(\n    candidates: Seq[ItemCandidateWithDetails],\n    presentation: Option[ModulePresentation],\n    source: CandidatePipelineIdentifier,\n    sourcePosition: Int,\n    features: FeatureMap\n  ): ModuleCandidateWithDetails = {\n    val newFeatureMap =\n      FeatureMapBuilder()\n        .add(CandidateSourcePosition, sourcePosition)\n        .add(CandidatePipelines, ListSet.empty + source).build() ++ features\n\n    ModuleCandidateWithDetails(candidates, presentation, newFeatureMap)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/ItemPresentation.scala",
    "content": "package com.twitter.product_mixer.core.model.common.presentation\n\ntrait ItemPresentation extends UniversalPresentation {\n  // Optional field which if populated, will group the items into the specified module\n  def modulePresentation: Option[ModulePresentation] = None\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/ModulePresentation.scala",
    "content": "package com.twitter.product_mixer.core.model.common.presentation\n\ntrait ModulePresentation extends UniversalPresentation\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/UniversalPresentation.scala",
    "content": "package com.twitter.product_mixer.core.model.common.presentation\n\n/**\n * Encapsulates information about how to present a Candidate\n *\n * Implementations of a [[UniversalPresentation]] contain information about how to present the Candidate.\n * This extra information can be in fields in the implementations or in their types.\n *\n * For instance, a Tweet candidate that will be displayed as a URT Tweet Item will be decorated with a\n * [[UniversalPresentation]] implementation that reflects the presentation such as\n * [[com.twitter.product_mixer.component_library.model.presentation.urt.UrtItemPresentation]]\n *\n * @see [[com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator]] for associating a\n *      [[UniversalPresentation]] with a Candidate.\n */\ntrait UniversalPresentation\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/slice/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/slice/BaseSliceItemPresentation.scala",
    "content": "package com.twitter.product_mixer.core.model.common.presentation.slice\n\nimport com.twitter.product_mixer.core.model.common.presentation.ItemPresentation\nimport com.twitter.product_mixer.core.model.marshalling.response.slice.SliceItem\n\ntrait BaseSliceItemPresentation extends ItemPresentation {\n  def sliceItem: SliceItem\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/urt/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/urt/BaseUrtItemPresentation.scala",
    "content": "package com.twitter.product_mixer.core.model.common.presentation.urt\n\nimport com.twitter.product_mixer.core.model.common.presentation.ItemPresentation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\ntrait BaseUrtItemPresentation extends ItemPresentation {\n\n  def timelineItem: TimelineItem\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/urt/BaseUrtModulePresentation.scala",
    "content": "package com.twitter.product_mixer.core.model.common.presentation.urt\n\nimport com.twitter.product_mixer.core.model.common.presentation.ModulePresentation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineModule\n\ntrait BaseUrtModulePresentation extends ModulePresentation {\n  def timelineModule: TimelineModule\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/urt/BaseUrtOperationPresentation.scala",
    "content": "package com.twitter.product_mixer.core.model.common.presentation.urt\n\nimport com.twitter.product_mixer.core.model.common.presentation.ItemPresentation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineOperation\n\ntrait BaseUrtOperationPresentation extends ItemPresentation {\n\n  def timelineOperation: TimelineOperation\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/urt/IsDispensable.scala",
    "content": "package com.twitter.product_mixer.core.model.common.presentation.urt\n\n/**\n * Whether an item is considered dispensable within a module.\n * Dispensable module items should never be left as the final remaining\n * items within a module. Whenever a module would be left with only\n * dispensable contents (through removal or dismissal of other items) the\n * entire module should be discarded as if contained 0 items.\n *\n * @see http://go/urtDispensableModuleItems\n */\ntrait IsDispensable { self: BaseUrtItemPresentation =>\n  def dispensable: Boolean\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation/urt/WithItemTreeDisplay.scala",
    "content": "package com.twitter.product_mixer.core.model.common.presentation.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ModuleItemTreeDisplay\n\n/*\n * Tree state declaring item’s parent relationship with any other items in\n * the module, any display indentation information, and/or collapsed display state.\n */\ntrait WithItemTreeDisplay { self: BaseUrtItemPresentation =>\n  def treeDisplay: Option[ModuleItemTreeDisplay]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/HasMarshalling.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling\n\ntrait HasMarshalling\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/fasterxml/jackson:jackson-module-scala\",\n        \"3rdparty/jvm/com/fasterxml/jackson/core:jackson-annotations\",\n        \"configapi/configapi-core/src/main/scala/com/twitter/timelines/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"util/util-core:util-core-util\",\n    ],\n    exports = [\n        \"3rdparty/jvm/com/fasterxml/jackson:jackson-module-scala\",\n        \"3rdparty/jvm/com/fasterxml/jackson/core:jackson-annotations\",\n        \"configapi/configapi-core/src/main/scala/com/twitter/timelines/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n        \"util/util-core:util-core-util\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request/ClientContext.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.request\n\nimport com.fasterxml.jackson.annotation.JsonIgnore\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.BadRequest\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\n\n/**\n * ClientContext contains fields related to the client making the request.\n */\ncase class ClientContext(\n  userId: Option[Long],\n  guestId: Option[Long],\n  guestIdAds: Option[Long],\n  guestIdMarketing: Option[Long],\n  appId: Option[Long],\n  ipAddress: Option[String],\n  userAgent: Option[String],\n  countryCode: Option[String],\n  languageCode: Option[String],\n  isTwoffice: Option[Boolean],\n  userRoles: Option[Set[String]],\n  deviceId: Option[String],\n  mobileDeviceId: Option[String],\n  mobileDeviceAdId: Option[String],\n  limitAdTracking: Option[Boolean])\n\nobject ClientContext {\n  val empty: ClientContext = ClientContext(\n    userId = None,\n    guestId = None,\n    guestIdAds = None,\n    guestIdMarketing = None,\n    appId = None,\n    ipAddress = None,\n    userAgent = None,\n    countryCode = None,\n    languageCode = None,\n    isTwoffice = None,\n    userRoles = None,\n    deviceId = None,\n    mobileDeviceId = None,\n    mobileDeviceAdId = None,\n    limitAdTracking = None\n  )\n}\n\n/**\n * HasClientContext indicates that a request has [[ClientContext]] and adds helper functions for\n * accessing [[ClientContext]] fields.\n */\ntrait HasClientContext {\n  def clientContext: ClientContext\n\n  /**\n   * getRequiredUserId returns a userId and throw if it's missing.\n   *\n   * @note logged out requests are disabled by default so this is safe for most products\n   */\n  @JsonIgnore /** Jackson tries to serialize this method, throwing an exception for guest products */\n  def getRequiredUserId: Long = clientContext.userId.getOrElse(\n    throw PipelineFailure(BadRequest, \"Missing required field: userId\"))\n\n  /**\n   * getOptionalUserId returns a userId if one is set\n   */\n  def getOptionalUserId: Option[Long] = clientContext.userId\n\n  /**\n   * getUserIdLoggedOutSupport returns a userId and falls back to 0 if none is set\n   */\n  def getUserIdLoggedOutSupport: Long = clientContext.userId.getOrElse(0L)\n\n  /**\n   * getUserOrGuestId returns a userId or a guestId if no userId has been set\n   */\n  def getUserOrGuestId: Option[Long] = clientContext.userId.orElse(clientContext.guestId)\n\n  /**\n   * getCountryCode returns a country code if one is set\n   */\n  def getCountryCode: Option[String] = clientContext.countryCode\n\n  /**\n   * getLanguageCode returns a language code if one is set\n   */\n  def getLanguageCode: Option[String] = clientContext.languageCode\n\n  /**\n   * isLoggedOut returns true if the user is logged out (no userId present).\n   *\n   * @note this can be useful in conjunction with [[getUserIdLoggedOutSupport]]\n   */\n  def isLoggedOut: Boolean = clientContext.userId.isEmpty\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request/DebugOptions.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.request\n\nimport com.twitter.util.Time\n\ntrait DebugOptions {\n  // Manually override the request time which is useful for writing deterministic Feature tests,\n  // since Feature tests do not support mocking Time. For example, URT sort indexes start with a\n  // Snowflake ID based on request time if no initialSortIndex is set on the request cursor, so to\n  // write a Feature test for this scenario, we can manually set the request time to use here.\n  def requestTimeOverride: Option[Time] = None\n}\n\ntrait HasDebugOptions {\n  def debugOptions: Option[DebugOptions]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request/DebugParams.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.request\n\nimport com.twitter.timelines.configapi.{FeatureValue => ConfigApiFeatureValue}\n\ncase class DebugParams(\n  featureOverrides: Option[Map[String, ConfigApiFeatureValue]],\n  override val debugOptions: Option[DebugOptions])\n    extends HasDebugOptions\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request/HasExcludedIds.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.request\n\n/**\n * Allow clients to pass in a set of IDs that would be excluded from the results.\n */\ntrait HasExcludedIds {\n  val excludedIds: Set[Long] = Set.empty\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request/HasSerializedRequestCursor.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.request\n\n/**\n * serializedRequestCursor is any serialized representation of a cursor.\n *\n * The serialized representation is implementation-specific but will often be a base 64\n * representation of a Thrift struct. Cursors should not be deserialized in the unmarshaller.\n */\ntrait HasSerializedRequestCursor {\n  def serializedRequestCursor: Option[String]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request/Product.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.request\n\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.identifier.ProductIdentifier\n\ntrait Product extends Component {\n\n  /**\n   * Identifier names on products can be used to create Feature Switch rules by product,\n   * which useful if bucketing occurs in a component shared by multiple products.\n   *\n   * @see [[com.twitter.product_mixer.core.product.ProductParamConfig.supportedClientFSName]]\n   */\n  override val identifier: ProductIdentifier\n\n  /**\n   * To support StringCenter, override this val to `Some(\"name-of-string-center-project\")` and\n   * include the `ProductScopeStringCenterModule` in the server's modules list\n   */\n  val stringCenterProject: Option[String] = None\n}\n\ntrait HasProduct {\n  def product: Product\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request/ProductContext.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.request\n\ntrait ProductContext\n\ntrait HasProductContext {\n  def productContext: Option[ProductContext]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request/Request.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.request\n\ntrait Request\n    extends HasClientContext\n    with HasProduct\n    with HasProductContext\n    with HasSerializedRequestCursor {\n  def maxResults: Option[Int]\n  def debugParams: Option[DebugParams]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/rtf/safety_level/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/rtf/safety_level/SafetyLevel.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level\n\n/*\n  These are model objects for the thrift enum - src/thrift/com/twitter/spam/rtf/safety_level.thrift\n  Please add new objects as needed for the marhallers\n */\nsealed trait SafetyLevel\n\ncase object ConversationFocalTweetSafetyLevel extends SafetyLevel\ncase object ConversationReplySafetyLevel extends SafetyLevel\ncase object ConversationInjectedTweetSafetyLevel extends SafetyLevel\ncase object TimelineFocalTweetSafetyLevel extends SafetyLevel\ncase object TimelineHomePromotedHydrationSafetyLevel extends SafetyLevel\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/slice/SliceItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.slice\n\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\n\n/**\n * These are the Ad Types exposed on AdUnits\n *\n * They are to be kept in sync with strato/config/src/thrift/com/twitter/strato/graphql/hubble.thrift\n */\nsealed trait AdType\nobject AdType {\n  case object Tweet extends AdType\n  case object Account extends AdType\n  case object InStreamVideo extends AdType\n  case object DisplayCreative extends AdType\n  case object Trend extends AdType\n  case object Spotlight extends AdType\n  case object Takeover extends AdType\n}\n\ntrait SliceItem\ncase class TweetItem(id: Long) extends SliceItem\ncase class UserItem(id: Long) extends SliceItem\ncase class TwitterListItem(id: Long) extends SliceItem\ncase class DMConvoSearchItem(id: String, lastReadableEventId: Option[Long]) extends SliceItem\ncase class DMEventItem(id: Long) extends SliceItem\ncase class DMConvoItem(id: String, lastReadableEventId: Option[Long]) extends SliceItem\ncase class DMMessageSearchItem(id: Long) extends SliceItem\ncase class TopicItem(id: Long) extends SliceItem\ncase class TypeaheadEventItem(eventId: Long, metadata: Option[TypeaheadMetadata]) extends SliceItem\ncase class TypeaheadQuerySuggestionItem(query: String, metadata: Option[TypeaheadMetadata])\n    extends SliceItem\ncase class TypeaheadUserItem(\n  userId: Long,\n  metadata: Option[TypeaheadMetadata],\n  badges: Seq[UserBadge])\n    extends SliceItem\ncase class AdItem(adUnitId: Long, adAccountId: Long) extends SliceItem\ncase class AdCreativeItem(creativeId: Long, adType: AdType, adAccountId: Long) extends SliceItem\ncase class AdGroupItem(adGroupId: Long, adAccountId: Long) extends SliceItem\ncase class CampaignItem(campaignId: Long, adAccountId: Long) extends SliceItem\ncase class FundingSourceItem(fundingSourceId: Long, adAccountId: Long) extends SliceItem\n\nsealed trait CursorType\ncase object PreviousCursor extends CursorType\ncase object NextCursor extends CursorType\n@deprecated(\n  \"GapCursors are not supported by Product Mixer Slice marshallers, if you need support for these reach out to #product-mixer\")\ncase object GapCursor extends CursorType\n\n// CursorItem extends SliceItem to enable support for GapCursors\ncase class CursorItem(value: String, cursorType: CursorType) extends SliceItem\n\ncase class SliceInfo(\n  previousCursor: Option[String],\n  nextCursor: Option[String])\n\ncase class Slice(\n  items: Seq[SliceItem],\n  sliceInfo: SliceInfo)\n    extends HasMarshalling\n\nsealed trait TypeaheadResultContextType\ncase object You extends TypeaheadResultContextType\ncase object Location extends TypeaheadResultContextType\ncase object NumFollowers extends TypeaheadResultContextType\ncase object FollowRelationship extends TypeaheadResultContextType\ncase object Bio extends TypeaheadResultContextType\ncase object NumTweets extends TypeaheadResultContextType\ncase object Trending extends TypeaheadResultContextType\ncase object HighlightedLabel extends TypeaheadResultContextType\n\ncase class TypeaheadResultContext(\n  contextType: TypeaheadResultContextType,\n  displayString: String,\n  iconUrl: Option[String])\n\ncase class TypeaheadMetadata(\n  score: Double,\n  source: Option[String],\n  context: Option[TypeaheadResultContext])\n\n// Used to render badges in Typeahead, such as Business-affiliated badges\ncase class UserBadge(badgeType: String, badgeUrl: String, description: String)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp/Page.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urp\n\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\n\ncase class Page(\n  id: String,\n  pageBody: PageBody,\n  scribeConfig: Option[TimelineScribeConfig] = None,\n  pageHeader: Option[PageHeader] = None,\n  pageNavBar: Option[PageNavBar] = None)\n    extends HasMarshalling\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp/PageBody.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urp\n\nsealed trait PageBody\n\ncase class TimelineKeyPageBody(timeline: TimelineKey) extends PageBody\n\ncase class SegmentedTimelinesPageBody(\n  initialTimeline: SegmentedTimeline,\n  timelines: Seq[SegmentedTimeline])\n    extends PageBody\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp/PageHeader.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urp\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.HasClientEventInfo\n\nsealed trait PageHeader\n\ncase class TopicPageHeader(\n  topicId: String,\n  facepile: Option[TopicPageHeaderFacepile] = None,\n  override val clientEventInfo: Option[ClientEventInfo] = None,\n  landingContext: Option[String] = None,\n  displayType: Option[TopicPageHeaderDisplayType] = Some(BasicTopicPageHeaderDisplayType))\n    extends PageHeader\n    with HasClientEventInfo\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp/PageNavBar.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urp\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.HasClientEventInfo\n\nsealed trait PageNavBar\n\ncase class TopicPageNavBar(\n  topicId: String,\n  override val clientEventInfo: Option[ClientEventInfo] = None)\n    extends PageNavBar\n    with HasClientEventInfo\n\ncase class TitleNavBar(\n  title: String,\n  subtitle: Option[String] = None,\n  override val clientEventInfo: Option[ClientEventInfo] = None)\n    extends PageNavBar\n    with HasClientEventInfo\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp/SegmentedTimeline.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urp\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineScribeConfig\n\ncase class SegmentedTimeline(\n  id: String,\n  labelText: String,\n  timeline: TimelineKey,\n  scribeConfig: Option[TimelineScribeConfig] = None,\n  refreshIntervalSec: Option[Long] = None)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp/TimelineKey.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urp\n\nsealed trait TimelineKey\n\ncase class TopicsLandingTimeline(topicId: Option[String]) extends TimelineKey\n\ncase class NoteworthyAccountsTimeline(topicId: Option[String]) extends TimelineKey\n\ncase class TopicsPickerTimeline(topicId: Option[String]) extends TimelineKey\n\ncase class NotInterestedTopicsMeTimeline() extends TimelineKey\n\ncase class FollowedTopicsMeTimeline() extends TimelineKey\n\ncase class FollowedTopicsOtherTimeline(userId: Long) extends TimelineKey\n\ncase class NuxUserRecommendationsTimeline() extends TimelineKey\n\ncase class NuxForYouCategoryUserRecommendationsTimeline() extends TimelineKey\n\ncase class NuxPymkCategoryUserRecommendationsTimeline() extends TimelineKey\n\ncase class NuxGeoCategoryUserRecommendationsTimeline() extends TimelineKey\n\ncase class NuxSingleInterestCategoryUserRecommendationsTimeline(topicId: Option[String])\n    extends TimelineKey\n\ncase class ShoppingHomeTimeline() extends TimelineKey\n\ncase class ForYouExploreMixerTimeline() extends TimelineKey\n\ncase class TrendingExploreMixerTimeline() extends TimelineKey\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp/TopicPageHeaderDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urp\n\nsealed trait TopicPageHeaderDisplayType\n\ncase object BasicTopicPageHeaderDisplayType extends TopicPageHeaderDisplayType\ncase object PersonalizedTopicPageHeaderDisplayType extends TopicPageHeaderDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp/TopicPageHeaderFacepile.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urp\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\n\ncase class TopicPageHeaderFacepile(\n  userIds: Seq[Long],\n  facepileUrl: Option[Url] = None)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/button\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module\",\n        \"util/util-core:util-core-util\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module\",\n        \"util/util-core:util-core-util\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/Cover.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\ntrait Cover extends TimelineItem\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/EntryNamespace.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\nimport scala.util.matching.Regex\n\n/**\n * Entry Identifiers (commonly entry ids) are a type of identifier used in URT to identify\n * unique timeline entries - tweets, users, modules, etc.\n *\n * Entry Identifiers are formed from two parts - a namespace (EntryNamespace) and an underlying\n * id.\n *\n * A Entry Namespace is restricted to:\n * - 3 to 60 characters to ensure reasonable length\n * - a-z and dashes (kebab-case)\n * - Examples include \"user\" and \"tweet\"\n *\n * When specific entries identifiers are created, they will be appended with a dash and their\n * own id, like user-12 or tweet-20\n */\n\ntrait HasEntryNamespace {\n  val entryNamespace: EntryNamespace\n}\n\n// sealed abstract case class is basically a scala 2.12 opaque type -\n// you can only create them via the factory method on the companion\n// allowing us to enforce validation\nsealed abstract case class EntryNamespace(override val toString: String)\n\nobject EntryNamespace {\n  val AllowedCharacters: Regex = \"[a-z-]+\".r // Allows for kebab-case\n\n  def apply(str: String): EntryNamespace = {\n    val isValid = str match {\n      case n if n.length < 3 =>\n        false\n      case n if n.length > 60 =>\n        false\n      case AllowedCharacters() =>\n        true\n      case _ =>\n        false\n    }\n\n    if (isValid)\n      new EntryNamespace(str) {}\n    else\n      throw new IllegalArgumentException(s\"Illegal EntryNamespace: $str\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/HasEntryIdentifier.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\ntrait HasEntryIdentifier extends UniversalNoun[Any] with HasEntryNamespace {\n  // Distinctly identifies this entry and must be unique relative to other entries within a response\n  lazy val entryIdentifier: String = s\"$entryNamespace-$id\"\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/HasExpirationTime.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\nimport com.twitter.util.Time\n\ntrait HasExpirationTime {\n  def expirationTime: Option[Time] = None\n\n  final def expirationTimeInMillis: Option[Long] = expirationTime.map(_.inMillis)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/HasSortIndex.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\ntrait HasSortIndex { timelineEntry: TimelineEntry =>\n  def sortIndex: Option[Long]\n\n  def withSortIndex(sortIndex: Long): TimelineEntry\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/ModuleItemTreeDisplay.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleDisplayType\n\ncase class ModuleItemTreeDisplay(\n  parentModuleEntryItemId: Option[String],\n  indentFromParent: Option[Boolean],\n  displayType: Option[ModuleDisplayType],\n  isAnchorChild: Option[Boolean])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/ReaderModeConfig.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\nimport metadata.Url\n\ncase class ReaderModeConfig(isReaderModeAvailable: Boolean, landingUrl: Url)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/ShowAlert.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.ShowAlert.ShowAlertEntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertColorConfiguration\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertDisplayLocation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertIconDisplayInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertNavigationMetadata\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.alert.ShowAlertType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.util.Duration\n\n/**\n * Domain model for the URT ShowAlert [[https://docbird.twitter.biz/unified_rich_timelines_urt/gen/com/twitter/timelines/render/thriftscala/ShowAlert.html]]\n *\n * @note the text field (id: 2) has been deliberately excluded as it's been deprecated since 2018. Use RichText instead.\n */\ncase class ShowAlert(\n  override val id: String,\n  override val sortIndex: Option[Long],\n  alertType: ShowAlertType,\n  triggerDelay: Option[Duration],\n  displayDuration: Option[Duration],\n  clientEventInfo: Option[ClientEventInfo],\n  collapseDelay: Option[Duration],\n  userIds: Option[Seq[Long]],\n  richText: Option[RichText],\n  iconDisplayInfo: Option[ShowAlertIconDisplayInfo],\n  colorConfig: ShowAlertColorConfiguration,\n  displayLocation: ShowAlertDisplayLocation,\n  navigationMetadata: Option[ShowAlertNavigationMetadata],\n) extends TimelineItem {\n  override val entryNamespace: EntryNamespace = ShowAlertEntryNamespace\n\n  // Note that sort index is not used for ShowAlerts, as they are not TimelineEntry and do not have entryId\n  override def withSortIndex(newSortIndex: Long): TimelineEntry =\n    copy(sortIndex = Some(newSortIndex))\n\n  // Not used for ShowAlerts\n  override def feedbackActionInfo: Option[FeedbackActionInfo] = None\n}\n\nobject ShowAlert {\n  val ShowAlertEntryNamespace: EntryNamespace = EntryNamespace(\"show-alert\")\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/Timeline.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\n\ncase class Timeline(\n  id: String,\n  instructions: Seq[TimelineInstruction],\n  // responseObjects::feedbackActions actions are populated implicitly, see UrtTransportMarshaller\n  metadata: Option[TimelineMetadata] = None)\n    extends HasMarshalling\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/TimelineEntry.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ContainsFeedbackActionInfos\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.HasClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.HasFeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.PinnableEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ReplaceableEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.MarkUnreadableEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleFooter\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleHeader\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleMetadata\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module.ModuleShowMoreBehavior\n\nsealed trait TimelineEntry\n    extends HasEntryIdentifier\n    with HasSortIndex\n    with HasExpirationTime\n    with PinnableEntry\n    with ReplaceableEntry\n    with MarkUnreadableEntry\n\ntrait TimelineItem extends TimelineEntry with HasClientEventInfo with HasFeedbackActionInfo\n\ncase class ModuleItem(\n  item: TimelineItem,\n  dispensable: Option[Boolean],\n  treeDisplay: Option[ModuleItemTreeDisplay])\n\ncase class TimelineModule(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val entryNamespace: EntryNamespace,\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  override val isPinned: Option[Boolean],\n  items: Seq[ModuleItem],\n  displayType: ModuleDisplayType,\n  header: Option[ModuleHeader],\n  footer: Option[ModuleFooter],\n  metadata: Option[ModuleMetadata],\n  showMoreBehavior: Option[ModuleShowMoreBehavior])\n    extends TimelineEntry\n    with HasClientEventInfo\n    with HasFeedbackActionInfo\n    with ContainsFeedbackActionInfos {\n  override def feedbackActionInfos: Seq[Option[FeedbackActionInfo]] = {\n    items.map(_.item.feedbackActionInfo) :+ feedbackActionInfo\n  }\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n\ntrait TimelineOperation extends TimelineEntry\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/TimelineInstruction.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ContainsFeedbackActionInfos\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.HasFeedbackActionInfo\n\nsealed trait TimelineInstruction\n\ncase class AddEntriesTimelineInstruction(entries: Seq[TimelineEntry])\n    extends TimelineInstruction\n    with ContainsFeedbackActionInfos {\n  override def feedbackActionInfos: Seq[Option[FeedbackActionInfo]] =\n    entries.flatMap {\n      // Order is important, as entries that implement both ContainsFeedbackActionInfos and\n      // HasFeedbackActionInfo are expected to include both when implementing ContainsFeedbackActionInfos\n      case containsFeedbackActionInfos: ContainsFeedbackActionInfos =>\n        containsFeedbackActionInfos.feedbackActionInfos\n      case hasFeedbackActionInfo: HasFeedbackActionInfo =>\n        Seq(hasFeedbackActionInfo.feedbackActionInfo)\n      case _ => Seq.empty\n    }\n}\n\ncase class ReplaceEntryTimelineInstruction(entry: TimelineEntry)\n    extends TimelineInstruction\n    with ContainsFeedbackActionInfos {\n  override def feedbackActionInfos: Seq[Option[FeedbackActionInfo]] =\n    entry match {\n      // Order is important, as entries that implement both ContainsFeedbackActionInfos and\n      // HasFeedbackActionInfo are expected to include both when implementing ContainsFeedbackActionInfos\n      case containsFeedbackActionInfos: ContainsFeedbackActionInfos =>\n        containsFeedbackActionInfos.feedbackActionInfos\n      case hasFeedbackActionInfo: HasFeedbackActionInfo =>\n        Seq(hasFeedbackActionInfo.feedbackActionInfo)\n      case _ => Seq.empty\n    }\n}\n\ncase class AddToModuleTimelineInstruction(\n  moduleItems: Seq[ModuleItem],\n  moduleEntryId: String,\n  moduleItemEntryId: Option[String],\n  prepend: Option[Boolean])\n    extends TimelineInstruction\n    with ContainsFeedbackActionInfos {\n  override def feedbackActionInfos: Seq[Option[FeedbackActionInfo]] =\n    moduleItems.map(_.item.feedbackActionInfo)\n}\n\ncase class PinEntryTimelineInstruction(entry: TimelineEntry) extends TimelineInstruction\n\ncase class MarkEntriesUnreadInstruction(entryIds: Seq[String]) extends TimelineInstruction\n\ncase class ClearCacheTimelineInstruction() extends TimelineInstruction\n\nsealed trait TimelineTerminationDirection\ncase object TopTermination extends TimelineTerminationDirection\ncase object BottomTermination extends TimelineTerminationDirection\ncase object TopAndBottomTermination extends TimelineTerminationDirection\ncase class TerminateTimelineInstruction(terminateTimelineDirection: TimelineTerminationDirection)\n    extends TimelineInstruction\n\ncase class ShowCoverInstruction(cover: Cover) extends TimelineInstruction\n\ncase class ShowAlertInstruction(showAlert: ShowAlert) extends TimelineInstruction\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/TimelineMetadata.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\ncase class TimelineMetadata(\n  title: Option[String],\n  scribeConfig: Option[TimelineScribeConfig],\n  readerModeConfig: Option[ReaderModeConfig] = None)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/TimelineScribeConfig.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt\n\ncase class TimelineScribeConfig(\n  page: Option[String],\n  section: Option[String],\n  entityToken: Option[String])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/color\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext\",\n        \"util/util-core:scala\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/color\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert/ShowAlertColorConfiguration.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.alert\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.RosettaColor\n\ncase class ShowAlertColorConfiguration(\n  background: RosettaColor,\n  text: RosettaColor,\n  border: Option[RosettaColor],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert/ShowAlertDisplayLocation.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.alert\n\nsealed trait ShowAlertDisplayLocation\ncase object Top extends ShowAlertDisplayLocation\ncase object Bottom extends ShowAlertDisplayLocation\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert/ShowAlertIcon.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.alert\n\nsealed trait ShowAlertIcon\ncase object UpArrow extends ShowAlertIcon\ncase object DownArrow extends ShowAlertIcon\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert/ShowAlertIconDisplayInfo.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.alert\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.RosettaColor\n\ncase class ShowAlertIconDisplayInfo(icon: ShowAlertIcon, tint: RosettaColor)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert/ShowAlertNavigationMetadata.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.alert\n\ncase class ShowAlertNavigationMetadata(navigateToEntryId: String)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/alert/ShowAlertType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.alert\n\nsealed trait ShowAlertType\ncase object NewTweets extends ShowAlertType\ncase object Navigate extends ShowAlertType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/button/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/icon\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/icon\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/button/ButtonStyle.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.button\n\nsealed trait ButtonStyle\n\ncase object Default extends ButtonStyle\ncase object Primary extends ButtonStyle\ncase object Secondary extends ButtonStyle\ncase object Text extends ButtonStyle\ncase object Destructive extends ButtonStyle\ncase object Neutral extends ButtonStyle\ncase object DestructiveSecondary extends ButtonStyle\ncase object DestructiveText extends ButtonStyle\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/button/CtaButton.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.button\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.HorizonIcon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\n\nsealed trait CtaButton\n\ncase class TextCtaButton(buttonText: String, url: Url) extends CtaButton\n\ncase class IconCtaButton(buttonIcon: HorizonIcon, accessibilityLabel: String, url: Url)\n    extends CtaButton\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/color/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/color/Color.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.color\n\ncase class Color(\n  red: Short,\n  green: Short,\n  blue: Short,\n  opacity: Option[Short])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/color/ColorPalette.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.color\n\ncase class ColorPalette(\n  rgb: Color,\n  percentage: Double)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/color/RosettaColor.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.color\n\nsealed trait RosettaColor\n\ncase object WhiteRosettaColor extends RosettaColor\ncase object BlackRosettaColor extends RosettaColor\ncase object ClearRosettaColor extends RosettaColor\n\ncase object TextBlackRosettaColor extends RosettaColor\ncase object TextBlueRosettaColor extends RosettaColor\n\ncase object DeepGrayRosettaColor extends RosettaColor\ncase object MediumGrayRosettaColor extends RosettaColor\ncase object LightGrayRosettaColor extends RosettaColor\ncase object FadedGrayRosettaColor extends RosettaColor\ncase object FaintGrayRosettaColor extends RosettaColor\n\ncase object DeepOrangeRosettaColor extends RosettaColor\ncase object MediumOrangeRosettaColor extends RosettaColor\ncase object LightOrangeRosettaColor extends RosettaColor\ncase object FadedOrangeRosettaColor extends RosettaColor\n\ncase object DeepYellowRosettaColor extends RosettaColor\ncase object MediumYellowRosettaColor extends RosettaColor\ncase object LightYellowRosettaColor extends RosettaColor\ncase object FadedYellowRosettaColor extends RosettaColor\n\ncase object DeepGreenRosettaColor extends RosettaColor\ncase object MediumGreenRosettaColor extends RosettaColor\ncase object LightGreenRosettaColor extends RosettaColor\ncase object FadedGreenRosettaColor extends RosettaColor\n\ncase object DeepBlueRosettaColor extends RosettaColor\ncase object TwitterBlueRosettaColor extends RosettaColor\ncase object LightBlueRosettaColor extends RosettaColor\ncase object FadedBlueRosettaColor extends RosettaColor\ncase object FaintBlueRosettaColor extends RosettaColor\n\ncase object DeepPurpleRosettaColor extends RosettaColor\ncase object MediumPurpleRosettaColor extends RosettaColor\ncase object LightPurpleRosettaColor extends RosettaColor\ncase object FadedPurpleRosettaColor extends RosettaColor\n\ncase object DeepRedRosettaColor extends RosettaColor\ncase object MediumRedRosettaColor extends RosettaColor\ncase object LightRedRosettaColor extends RosettaColor\ncase object FadedRedRosettaColor extends RosettaColor\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/contextual_ref/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/rtf/safety_level\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/rtf/safety_level\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/contextual_ref/ContextualTweetRef.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref\n\ncase class ContextualTweetRef(\n  id: Long,\n  hydrationContext: Option[TweetHydrationContext])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/contextual_ref/OuterTweetContext.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref\n\nsealed trait OuterTweetContext\n\ncase class QuoteTweetId(id: Long) extends OuterTweetContext\ncase class RetweetId(id: Long) extends OuterTweetContext\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/contextual_ref/TweetHydrationContext.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref\n\nimport com.twitter.product_mixer.core.model.marshalling.response.rtf.safety_level.SafetyLevel\n\ncase class TweetHydrationContext(\n  safetyLevelOverride: Option[SafetyLevel],\n  outerTweetContext: Option[OuterTweetContext])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/cover/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/button\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/button\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/cover/CoverContent.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.cover\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.DismissInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageVariant\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\n\nsealed trait CoverContent\n\ncase class FullCoverContent(\n  displayType: FullCoverDisplayType,\n  primaryText: RichText,\n  primaryCoverCta: CoverCta,\n  secondaryCoverCta: Option[CoverCta],\n  secondaryText: Option[RichText],\n  imageVariant: Option[ImageVariant],\n  details: Option[RichText],\n  dismissInfo: Option[DismissInfo],\n  imageDisplayType: Option[ImageDisplayType],\n  impressionCallbacks: Option[List[Callback]])\n    extends CoverContent\n\ncase class HalfCoverContent(\n  displayType: HalfCoverDisplayType,\n  primaryText: RichText,\n  primaryCoverCta: CoverCta,\n  secondaryCoverCta: Option[CoverCta],\n  secondaryText: Option[RichText],\n  impressionCallbacks: Option[List[Callback]],\n  dismissible: Option[Boolean],\n  coverImage: Option[CoverImage],\n  dismissInfo: Option[DismissInfo])\n    extends CoverContent\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/cover/CoverCta.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.cover\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.ButtonStyle\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.HorizonIcon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\n\ncase class CoverCta(\n  text: String,\n  ctaBehavior: CoverCtaBehavior,\n  callbacks: Option[List[Callback]],\n  clientEventInfo: Option[ClientEventInfo],\n  icon: Option[HorizonIcon],\n  buttonStyle: Option[ButtonStyle])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/cover/CoverCtaBehavior.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.cover\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\n\nsealed trait CoverCtaBehavior\n\ncase class CoverBehaviorNavigate(url: Url) extends CoverCtaBehavior\ncase class CoverBehaviorDismiss(feedbackMessage: Option[RichText]) extends CoverCtaBehavior\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/cover/CoverImage.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.cover\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageAnimationType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageDisplayType\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageVariant\n\ncase class CoverImage(\n  imageVariant: ImageVariant,\n  imageDisplayType: ImageDisplayType,\n  imageAnimationType: Option[ImageAnimationType])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/cover/FullCoverDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.cover\n\nsealed trait FullCoverDisplayType\n\ncase object CoverFullCoverDisplayType extends FullCoverDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/cover/HalfCoverDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.cover\n\nsealed trait HalfCoverDisplayType\n\ncase object CoverHalfCoverDisplayType extends HalfCoverDisplayType\ncase object CenterCoverHalfCoverDisplayType extends HalfCoverDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/cover/ShowCover.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.cover\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.Cover\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.FullCover.FullCoverEntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.cover.HalfCover.HalfCoverEntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\n\nobject HalfCover {\n  val HalfCoverEntryNamespace = EntryNamespace(\"half-cover\")\n}\ncase class HalfCover(\n  override val id: String,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  content: HalfCoverContent)\n    extends Cover {\n\n  override val entryNamespace: EntryNamespace = HalfCoverEntryNamespace\n\n  // Note that sort index is not used for Covers, as they are not TimelineEntry and do not have entryId\n  override def withSortIndex(newSortIndex: Long): TimelineEntry =\n    copy(sortIndex = Some(newSortIndex))\n\n  // Not used for covers\n  override def feedbackActionInfo: Option[FeedbackActionInfo] = None\n}\n\nobject FullCover {\n  val FullCoverEntryNamespace = EntryNamespace(\"full-cover\")\n}\ncase class FullCover(\n  override val id: String,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  content: FullCoverContent)\n    extends Cover {\n\n  override val entryNamespace: EntryNamespace = FullCoverEntryNamespace\n\n  // Note that sort index is not used for Covers, as they are not TimelineEntry and do not have entryId\n  override def withSortIndex(newSortIndex: Long): TimelineEntry =\n    copy(sortIndex = Some(newSortIndex))\n\n  // Not used for covers\n  override def feedbackActionInfo: Option[FeedbackActionInfo] = None\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/icon/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [],\n    exports = [],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/icon/HorizonIcon.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.icon\n\nsealed trait HorizonIcon\n\ncase object Bookmark extends HorizonIcon\ncase object Moment extends HorizonIcon\ncase object Debug extends HorizonIcon\ncase object Error extends HorizonIcon\ncase object Follow extends HorizonIcon\ncase object Unfollow extends HorizonIcon\ncase object Smile extends HorizonIcon\ncase object Frown extends HorizonIcon\ncase object Help extends HorizonIcon\ncase object Link extends HorizonIcon\ncase object Message extends HorizonIcon\ncase object No extends HorizonIcon\ncase object Outgoing extends HorizonIcon\ncase object Pin extends HorizonIcon\ncase object Retweet extends HorizonIcon\ncase object Speaker extends HorizonIcon\ncase object Trashcan extends HorizonIcon\ncase object Feedback extends HorizonIcon\ncase object FeedbackClose extends HorizonIcon\ncase object EyeOff extends HorizonIcon\ncase object Moderation extends HorizonIcon\ncase object Topic extends HorizonIcon\ncase object TopicClose extends HorizonIcon\ncase object Flag extends HorizonIcon\ncase object TopicFilled extends HorizonIcon\ncase object NotificationsFollow extends HorizonIcon\ncase object Person extends HorizonIcon\ncase object BalloonStroke extends HorizonIcon\ncase object Calendar extends HorizonIcon\ncase object LocationStroke extends HorizonIcon\ncase object PersonStroke extends HorizonIcon\ncase object Safety extends HorizonIcon\ncase object Logo extends HorizonIcon\ncase object SparkleOn extends HorizonIcon\ncase object StarRising extends HorizonIcon\ncase object CameraVideo extends HorizonIcon\ncase object ShoppingClock extends HorizonIcon\ncase object ArrowRight extends HorizonIcon\ncase object SpeakerOff extends HorizonIcon\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/BUILD",
    "content": "scala_library(\n    sources = [\"**/*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/button\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/contextual_ref\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/media\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/reaction\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/button\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/contextual_ref\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/media\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/reaction\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/article/ArticleDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.article\n\nsealed trait ArticleDisplayType\n\ncase object Default extends ArticleDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/article/ArticleItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.article\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nobject ArticleItem {\n  val ArticleEntryNamespace = EntryNamespace(\"article\")\n}\n\ncase class ArticleItem(\n  override val id: Int,\n  articleSeedType: ArticleSeedType,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  displayType: Option[ArticleDisplayType],\n  socialContext: Option[SocialContext])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = ArticleItem.ArticleEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/article/ArticleSeedType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.article\n\nsealed trait ArticleSeedType\n\n/**\n * Seed UTEG with a user's following list (1st degree network)\n */\ncase object FollowingListSeed extends ArticleSeedType\n\n/**\n * Seed UTEG with a user's friends of friends (follow graph + 1) list\n */\ncase object FriendsOfFriendsSeed extends ArticleSeedType\n\n/**\n * Seed UTEG with a given lists' members\n */\ncase object ListIdSeed extends ArticleSeedType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/audio_space/AudioSpaceItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.audio_space\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nobject AudioSpaceItem {\n  val SpaceEntryNamespace = EntryNamespace(\"audiospace\")\n}\n\ncase class AudioSpaceItem(\n  override val id: String,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = AudioSpaceItem.SpaceEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/card/CardDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.card\n\nsealed trait CardDisplayType\n\ncase object HeroDisplayType extends CardDisplayType\ncase object CellDisplayType extends CardDisplayType\ncase object TweetCardDisplayType extends CardDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/card/CardItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.card\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\n\nobject CardItem {\n  val CardEntryNamespace = EntryNamespace(\"card\")\n}\n\ncase class CardItem(\n  override val id: String,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  cardUrl: String,\n  text: Option[String],\n  subtext: Option[String],\n  url: Option[Url],\n  displayType: Option[CardDisplayType])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = CardItem.CardEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/commerce/CommerceProductGroupItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.commerce\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.commerce.CommerceProductGroupItem.CommerceProductGroupEntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\n\nobject CommerceProductGroupItem {\n  val CommerceProductGroupEntryNamespace: EntryNamespace = EntryNamespace(\"commerce-product-group\")\n}\n\ncase class CommerceProductGroupItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo])\n    extends TimelineItem {\n\n  val entryNamespace: EntryNamespace = CommerceProductGroupEntryNamespace\n  def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/commerce/CommerceProductItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.commerce\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.commerce.CommerceProductItem.CommerceProductEntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\n\nobject CommerceProductItem {\n  val CommerceProductEntryNamespace: EntryNamespace = EntryNamespace(\"commerce-product\")\n}\n\ncase class CommerceProductItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo])\n    extends TimelineItem {\n\n  val entryNamespace: EntryNamespace = CommerceProductEntryNamespace\n  def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/conversation_annotation/ConversationAnnotation.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.conversation_annotation\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\n\ncase class ConversationAnnotation(\n  conversationAnnotationType: ConversationAnnotationType,\n  header: Option[RichText],\n  description: Option[RichText])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/conversation_annotation/ConversationAnnotationType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.conversation_annotation\n\nsealed trait ConversationAnnotationType\n\ncase object Political extends ConversationAnnotationType\ncase object Large extends ConversationAnnotationType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/event/EventSummaryDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.event\n\nsealed trait EventSummaryDisplayType\n\ncase object CellEventSummaryDisplayType extends EventSummaryDisplayType\ncase object HeroEventSummaryDisplayType extends EventSummaryDisplayType\ncase object CellWithProminentSocialContextEventSummaryDisplayType extends EventSummaryDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/event/EventSummaryItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.event\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageVariant\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nobject EventSummaryItem {\n  val EventSummaryItemEntryNamespace = EntryNamespace(\"eventsummary\")\n}\n\ncase class EventSummaryItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  title: String,\n  displayType: EventSummaryDisplayType,\n  url: Url,\n  image: Option[ImageVariant],\n  timeString: Option[String])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace =\n    EventSummaryItem.EventSummaryItemEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/forward_pivot/ForwardPivot.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.RosettaColor\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Badge\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageVariant\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\n\ncase class ForwardPivot(\n  text: RichText,\n  landingUrl: Url,\n  displayType: ForwardPivotDisplayType,\n  iconImageVariant: Option[ImageVariant],\n  stateBadge: Option[Badge],\n  subtext: Option[RichText],\n  backgroundColorName: Option[RosettaColor],\n  engagementNudge: Option[Boolean],\n  softInterventionDisplayType: Option[SoftInterventionDisplayType])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/forward_pivot/ForwardPivotDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot\n\nsealed trait ForwardPivotDisplayType\n\ncase object LiveEvent extends ForwardPivotDisplayType\ncase object SoftIntervention extends ForwardPivotDisplayType\ncase object CommunityNotes extends ForwardPivotDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/forward_pivot/SoftInterventionDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot\n\nsealed trait SoftInterventionDisplayType\n\ncase object GetTheLatest extends SoftInterventionDisplayType\ncase object StayInformed extends SoftInterventionDisplayType\ncase object Misleading extends SoftInterventionDisplayType\ncase object GovernmentRequested extends SoftInterventionDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/generic_summary/GenericSummaryAction.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\n\ncase class GenericSummaryAction(\n  url: Url,\n  clientEventInfo: Option[ClientEventInfo])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/generic_summary/GenericSummaryContext.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.HorizonIcon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\n\ncase class GenericSummaryContext(\n  text: RichText,\n  icon: Option[HorizonIcon])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/generic_summary/GenericSummaryDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary\n\nsealed trait GenericSummaryItemDisplayType\n\ncase object HeroDisplayType extends GenericSummaryItemDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/generic_summary/GenericSummaryItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.generic_summary\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.media.Media\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PromotedMetadata\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.util.Time\n\nobject GenericSummaryItem {\n  val GenericSummaryItemNamespace: EntryNamespace = EntryNamespace(\"genericsummary\")\n}\n\ncase class GenericSummaryItem(\n  override val id: String,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  headline: RichText,\n  displayType: GenericSummaryItemDisplayType,\n  userAttributionIds: Seq[Long],\n  media: Option[Media],\n  context: Option[GenericSummaryContext],\n  timestamp: Option[Time],\n  onClickAction: Option[GenericSummaryAction],\n  promotedMetadata: Option[PromotedMetadata])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = GenericSummaryItem.GenericSummaryItemNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/highlight/HighlightedSection.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.highlight\n\ncase class HighlightedSection(startIndex: Int, endIndex: Int)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/icon_label/IconLabelItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.icon_label\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.HorizonIcon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nobject IconLabelItem {\n  val IconLabelEntryNamespace = EntryNamespace(\"iconlabel\")\n}\n\ncase class IconLabelItem(\n  override val id: String,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  text: RichText,\n  icon: Option[HorizonIcon])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = IconLabelItem.IconLabelEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/label/LabelDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.label\n\nsealed trait LabelDisplayType\n\ncase object InlineHeaderLabelDisplayType extends LabelDisplayType\ncase object OtherRepliesSectionHeaderLabelDisplayType extends LabelDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/label/LabelItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.label\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nobject LabelItem {\n  val LabelEntryNamespace = EntryNamespace(\"label\")\n}\n\ncase class LabelItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  text: String,\n  subtext: Option[String],\n  disclosureIndicator: Option[Boolean],\n  url: Option[Url],\n  displayType: Option[LabelDisplayType])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = LabelItem.LabelEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/message/MessageAction.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.message\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\n\ncase class MessageAction(\n  dismissOnClick: Boolean,\n  url: Option[String],\n  clientEventInfo: Option[ClientEventInfo],\n  onClickCallbacks: Option[Seq[Callback]])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/message/MessageActionType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.message\n\nsealed trait MessageActionType\n\ncase object FollowAllMessageActionType extends MessageActionType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/message/MessageContent.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.message\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\n\nsealed trait MessageContent\n\ncase class InlinePromptMessageContent(\n  headerText: String,\n  bodyText: Option[String],\n  primaryButtonAction: Option[MessageTextAction],\n  secondaryButtonAction: Option[MessageTextAction],\n  headerRichText: Option[RichText],\n  bodyRichText: Option[RichText],\n  socialContext: Option[SocialContext],\n  userFacepile: Option[UserFacepile])\n    extends MessageContent\n\ncase class HeaderImagePromptMessageContent(\n  headerImage: MessageImage,\n  headerText: Option[String],\n  bodyText: Option[String],\n  primaryButtonAction: Option[MessageTextAction],\n  secondaryButtonAction: Option[MessageTextAction],\n  action: Option[MessageAction],\n  headerRichText: Option[RichText],\n  bodyRichText: Option[RichText])\n    extends MessageContent\n\ncase class CompactPromptMessageContent(\n  headerText: String,\n  bodyText: Option[String],\n  primaryButtonAction: Option[MessageTextAction],\n  secondaryButtonAction: Option[MessageTextAction],\n  action: Option[MessageAction],\n  headerRichText: Option[RichText],\n  bodyRichText: Option[RichText])\n    extends MessageContent\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/message/MessageImage.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.message\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageVariant\n\ncase class MessageImage(\n  imageVariants: Set[ImageVariant],\n  backgroundColor: Option[String])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/message/MessagePromptItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.message\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nobject MessagePromptItem {\n  val MessagePromptEntryNamespace = EntryNamespace(\"messageprompt\")\n}\n\ncase class MessagePromptItem(\n  override val id: String,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  override val isPinned: Option[Boolean],\n  content: MessageContent,\n  impressionCallbacks: Option[List[Callback]])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace =\n    MessagePromptItem.MessagePromptEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/message/MessageTextAction.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.message\n\ncase class MessageTextAction(\n  text: String,\n  action: MessageAction)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/message/UserFacepile.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.message\n\ncase class UserFacepile(\n  userIds: Seq[Long],\n  featuredUserIds: Seq[Long],\n  action: Option[MessageTextAction],\n  actionType: Option[MessageActionType],\n  displaysFeaturingText: Option[Boolean],\n  displayType: Option[UserFacepileDisplayType])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/message/UserFacepileDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.message\n\nsealed trait UserFacepileDisplayType\n\ncase object LargeUserFacepileDisplayType extends UserFacepileDisplayType\ncase object CompactUserFacepileDisplayType extends UserFacepileDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/moment/MomentAnnotationItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.moment\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\n\nobject MomentAnnotationItem {\n  val MomentAnnotationEntryNamespace = EntryNamespace(\"momentannotation\")\n}\n\n/**\n * Represents a MomentAnnotation URT item.\n * This is primarily used by Trends Searth Result Page for displaying Trends Title or Description\n * URT API Reference: https://docbird.twitter.biz/unified_rich_timelines_urt/gen/com/twitter/timelines/render/thriftscala/MomentAnnotation.html\n */\ncase class MomentAnnotationItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  override val isPinned: Option[Boolean],\n  text: Option[RichText],\n  header: Option[RichText],\n) extends TimelineItem {\n\n  override val entryNamespace: EntryNamespace =\n    MomentAnnotationItem.MomentAnnotationEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/prompt/PromptContent.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\n\n/**\n * Represents different types of URT Prompts supported such as the Relevance Prompt.\n *\n * URT API Reference: https://docbird.twitter.biz/unified_rich_timelines_urt/gen/com/twitter/timelines/render/thriftscala/PromptContent.html\n */\nsealed trait PromptContent\n\n/**\n * Relevance Prompt is a Yes-No style prompt that can be used for collecting feedback from a User\n * about a part of their timeline.\n *\n * URT API Reference: https://docbird.twitter.biz/unified_rich_timelines_urt/gen/com/twitter/timelines/render/thriftscala/RelevancePrompt.html\n */\ncase class RelevancePromptContent(\n  title: String,\n  confirmation: String,\n  isRelevantText: String,\n  notRelevantText: String,\n  isRelevantCallback: Callback,\n  notRelevantCallback: Callback,\n  displayType: RelevancePromptDisplayType,\n  isRelevantFollowUp: Option[RelevancePromptFollowUpFeedbackType],\n  notRelevantFollowUp: Option[RelevancePromptFollowUpFeedbackType])\n    extends PromptContent\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/prompt/PromptItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\n\nobject PromptItem {\n  val PromptEntryNamespace = EntryNamespace(\"relevanceprompt\")\n}\n\ncase class PromptItem(\n  override val id: String,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo] = None,\n  content: PromptContent,\n  impressionCallbacks: Option[List[Callback]])\n    extends TimelineItem {\n\n  override val entryNamespace: EntryNamespace = PromptItem.PromptEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/prompt/RelevancePromptDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt\n\n/**\n * Represents the different ways to display the Relevance Prompt in a timeline.\n *\n * URT API Reference: https://docbird.twitter.biz/unified_rich_timelines_urt/gen/com/twitter/timelines/render/thriftscala/RelevancePromptDisplayType.html\n */\nsealed trait RelevancePromptDisplayType\n\ncase object Normal extends RelevancePromptDisplayType\ncase object Compact extends RelevancePromptDisplayType\ncase object Large extends RelevancePromptDisplayType\ncase object ThumbsUpAndDown extends RelevancePromptDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/prompt/RelevancePromptFollowUpFeedbackType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.prompt\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Callback\n\n/**\n * Different kinds of follow-ups after a positive-negative feedback on a prompt button.\n *\n * URT API Reference: https://docbird.twitter.biz/unified_rich_timelines_urt/gen/com/twitter/timelines/render/thriftscala/RelevancePromptFollowUpFeedbackType.html\n */\nsealed trait RelevancePromptFollowUpFeedbackType\n\ncase class RelevancePromptFollowUpTextInput(\n  context: String,\n  textFieldPlaceholder: String,\n  sendTextCallback: Callback)\n    extends RelevancePromptFollowUpFeedbackType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/suggestion/SpellingActionType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.suggestion\n\n/**\n * Represents the different types of Spelling Suggestion items.\n *\n * URT API Reference: https://docbird.twitter.biz/unified_rich_timelines_urt/gen/com/twitter/timelines/render/thriftscala/SpellingActionType.html\n */\nsealed trait SpellingActionType\n\n/**\n * Used when the original query is replaced completed by another query in the backend.\n * Clients use the text 'Searching instead for …' to display this suggestion.\n */\ncase object ReplaceSpellingActionType extends SpellingActionType\n\n/**\n * Used when the original query is expanded by a suggestion when performing the search.\n * Clients use the text 'Including results for …' to display this suggestion.\n */\ncase object ExpandSpellingActionType extends SpellingActionType\n\n/**\n * Used when the search query is not changed and a suggestion is displayed as an alternative query.\n * Clients use the text 'Did you mean … ?' to display the suggestion.\n */\ncase object SuggestSpellingActionType extends SpellingActionType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/suggestion/SpellingItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.suggestion\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\n\nobject SpellingItem {\n  val SpellingEntryNamespace = EntryNamespace(\"spelling\")\n}\n\n/**\n * Represents a Spelling Suggestion URT item. This is primary used by Search timelines for\n * displaying Spelling correction information.\n *\n * URT API Reference: https://docbird.twitter.biz/unified_rich_timelines_urt/gen/com/twitter/timelines/render/thriftscala/Spelling.html\n */\ncase class SpellingItem(\n  override val id: String,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  textResult: TextResult,\n  spellingActionType: Option[SpellingActionType],\n  originalQuery: Option[String])\n    extends TimelineItem {\n\n  override val entryNamespace: EntryNamespace = SpellingItem.SpellingEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/suggestion/TextResult.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.suggestion\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.highlight.HighlightedSection\n\n/**\n * Represents text with hit-highlights used for returning search query suggestions.\n *\n * URT API Reference: https://docbird.twitter.biz/unified_rich_timelines_urt/gen/com/twitter/timelines/render/thriftscala/TextResult.html\n */\ncase class TextResult(\n  text: String,\n  hitHighlights: Option[Seq[HighlightedSection]],\n  score: Option[Double],\n  querySource: Option[String])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/thread/ThreadHeaderContent.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.thread\n\nsealed trait ThreadHeaderContent\n\ncase class UserThreadHeader(userId: Long) extends ThreadHeaderContent\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/thread/ThreadHeaderItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.thread\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\n\nobject ThreadHeaderItem {\n  val ThreadHeaderEntryNamespace = EntryNamespace(\"threadheader\")\n}\n\ncase class ThreadHeaderItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  override val isPinned: Option[Boolean],\n  content: ThreadHeaderContent)\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = ThreadHeaderItem.ThreadHeaderEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/tile/TileContent.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.button.CtaButton\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Badge\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\n\nsealed trait TileContent\n\ncase class StandardTileContent(\n  title: String,\n  supportingText: String,\n  badge: Option[Badge])\n    extends TileContent\n\ncase class CallToActionTileContent(\n  text: String,\n  richText: Option[RichText],\n  ctaButton: Option[CtaButton])\n    extends TileContent\n\n//todo: Add other TileContent types later\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/tile/TileItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.tile\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageVariant\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nobject TileItem {\n  val TileEntryNamespace = EntryNamespace(\"tile\")\n}\n\ncase class TileItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  title: String,\n  supportingText: String,\n  url: Option[Url],\n  image: Option[ImageVariant],\n  content: TileContent)\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = TileItem.TileEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/tombstone/TombstoneDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone\n\nsealed trait TombstoneDisplayType\n\ncase object TweetUnavailable extends TombstoneDisplayType\ncase object DisconnectedRepliesAncestor extends TombstoneDisplayType\ncase object DisconnectedRepliesDescendant extends TombstoneDisplayType\ncase object Inline extends TombstoneDisplayType\ncase object NonCompliant extends TombstoneDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/tombstone/TombstoneInfo.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.RichText\n\ncase class TombstoneInfo(\n  text: String,\n  richText: Option[RichText],\n  richRevealText: Option[RichText])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/tombstone/TombstoneItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet.TweetItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nobject TombstoneItem {\n  val TombstoneEntryNamespace = EntryNamespace(\"tombstone\")\n}\n\ncase class TombstoneItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  tombstoneDisplayType: TombstoneDisplayType,\n  tombstoneInfo: Option[TombstoneInfo],\n  tweet: Option[TweetItem])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = TombstoneItem.TombstoneEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/topic/TopicDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic\n\nsealed trait TopicDisplayType\n\ncase object BasicTopicDisplayType extends TopicDisplayType\ncase object PillTopicDisplayType extends TopicDisplayType\ncase object NoIconTopicDisplayType extends TopicDisplayType\ncase object PillWithoutActionIconDisplayType extends TopicDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/topic/TopicFollowPromptDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic\n\nsealed trait TopicFollowPromptDisplayType\n\ncase object IncentiveFocusTopicFollowPromptDisplayType extends TopicFollowPromptDisplayType\ncase object TopicFocusTopicFollowPromptDisplayType extends TopicFollowPromptDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/topic/TopicFollowPromptItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nobject TopicFollowPromptItem {\n  val TopicFollowPromptEntryNamespace = EntryNamespace(\"topicfollowprompt\")\n}\n\ncase class TopicFollowPromptItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  topicFollowPromptDisplayType: TopicFollowPromptDisplayType,\n  followIncentiveTitle: Option[String],\n  followIncentiveText: Option[String])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace =\n    TopicFollowPromptItem.TopicFollowPromptEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/topic/TopicFunctionalityType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic\n\nsealed trait TopicFunctionalityType\n\ncase object BasicTopicFunctionalityType extends TopicFunctionalityType\ncase object PivotTopicFunctionalityType extends TopicFunctionalityType\ncase object RecommendationTopicFunctionalityType extends TopicFunctionalityType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/topic/TopicItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.topic\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nobject TopicItem {\n  val TopicEntryNamespace = EntryNamespace(\"topic\")\n}\n\ncase class TopicItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  topicFunctionalityType: Option[TopicFunctionalityType],\n  topicDisplayType: Option[TopicDisplayType])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = TopicItem.TopicEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/trend/TrendItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.trend\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PromotedMetadata\n\nobject TrendItem {\n  val TrendItemEntryNamespace = EntryNamespace(\"trend\")\n}\n\ncase class GroupedTrend(trendName: String, url: Url)\n\ncase class TrendItem(\n  override val id: String,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  normalizedTrendName: String,\n  trendName: String,\n  url: Url,\n  description: Option[String],\n  metaDescription: Option[String],\n  tweetCount: Option[Int],\n  domainContext: Option[String],\n  promotedMetadata: Option[PromotedMetadata],\n  groupedTrends: Option[Seq[GroupedTrend]])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = TrendItem.TrendItemEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/tweet/TimelinesScoreInfo.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet\n\ncase class TimelinesScoreInfo(score: Double)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/tweet/TweetDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet\n\nsealed trait TweetDisplayType\n\ncase object Tweet extends TweetDisplayType\ncase object TweetFollowOnly extends TweetDisplayType\ncase object Media extends TweetDisplayType\ncase object MomentTimelineTweet extends TweetDisplayType\ncase object EmphasizedPromotedTweet extends TweetDisplayType\ncase object QuotedTweet extends TweetDisplayType\ncase object SelfThread extends TweetDisplayType\ncase object CompactPromotedTweet extends TweetDisplayType\ncase object TweetWithoutCard extends TweetDisplayType\ncase object ReaderModeRoot extends TweetDisplayType\ncase object ReaderMode extends TweetDisplayType\ncase object CondensedTweet extends TweetDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/tweet/TweetHighlights.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.highlight.HighlightedSection\n\ncase class TweetHighlights(\n  textHighlights: Option[List[HighlightedSection]],\n  cardTitleHighlights: Option[List[HighlightedSection]],\n  cardDescriptionHighlights: Option[List[HighlightedSection]])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/tweet/TweetItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.contextual_ref.ContextualTweetRef\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.conversation_annotation.ConversationAnnotation\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.forward_pivot.ForwardPivot\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.item.tombstone.TombstoneInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Badge\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PrerollMetadata\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PromotedMetadata\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\n\nobject TweetItem {\n  val TweetEntryNamespace = EntryNamespace(\"tweet\")\n  val PromotedTweetEntryNamespace = EntryNamespace(\"promoted-tweet\")\n}\n\ncase class TweetItem(\n  override val id: Long,\n  override val entryNamespace: EntryNamespace,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  override val isPinned: Option[Boolean],\n  override val entryIdToReplace: Option[String],\n  socialContext: Option[SocialContext],\n  highlights: Option[TweetHighlights],\n  displayType: TweetDisplayType,\n  innerTombstoneInfo: Option[TombstoneInfo],\n  timelinesScoreInfo: Option[TimelinesScoreInfo],\n  hasModeratedReplies: Option[Boolean],\n  forwardPivot: Option[ForwardPivot],\n  innerForwardPivot: Option[ForwardPivot],\n  promotedMetadata: Option[PromotedMetadata],\n  conversationAnnotation: Option[ConversationAnnotation],\n  contextualTweetRef: Option[ContextualTweetRef],\n  prerollMetadata: Option[PrerollMetadata],\n  replyBadge: Option[Badge],\n  destination: Option[Url])\n    extends TimelineItem {\n\n  /**\n   * Promoted tweets need to include the impression ID in the entry ID since some clients have\n   * client-side logic that deduplicates ads impression callbacks based on a combination of the\n   * tweet and impression IDs. Not including the impression ID will lead to over deduplication.\n   */\n  override lazy val entryIdentifier: String = promotedMetadata\n    .map { metadata =>\n      val impressionId = metadata.impressionString match {\n        case Some(impressionString) if impressionString.nonEmpty => impressionString\n        case _ => throw new IllegalStateException(s\"Promoted Tweet $id missing impression ID\")\n      }\n      s\"$entryNamespace-$id-$impressionId\"\n    }.getOrElse(s\"$entryNamespace-$id\")\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/tweet_composer/TweetComposerDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet_composer\n\nsealed trait TweetComposerDisplayType\n\ncase object TweetComposerSelfThread extends TweetComposerDisplayType\ncase object Reply extends TweetComposerDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/tweet_composer/TweetComposerItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.tweet_composer\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nobject TweetComposerItem {\n  val TweetComposerEntryNameSpace = EntryNamespace(\"tweetcomposer\")\n}\n\ncase class TweetComposerItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  displayType: TweetComposerDisplayType,\n  text: String,\n  url: Url)\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = TweetComposerItem.TweetComposerEntryNameSpace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/twitter_list/TwitterListDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.twitter_list\n\nsealed trait TwitterListDisplayType\n\ncase object List extends TwitterListDisplayType\ncase object ListTile extends TwitterListDisplayType\ncase object ListWithPin extends TwitterListDisplayType\ncase object ListWithSubscribe extends TwitterListDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/twitter_list/TwitterListItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.twitter_list\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nobject TwitterListItem {\n  val ListEntryNamespace = EntryNamespace(\"list\")\n}\n\ncase class TwitterListItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  displayType: Option[TwitterListDisplayType])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = TwitterListItem.ListEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/user/UserDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.user\n\nsealed trait UserDisplayType\n\ncase object User extends UserDisplayType\ncase object UserDetailed extends UserDisplayType\ncase object PendingFollowUser extends UserDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/user/UserItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.user\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.promoted.PromotedMetadata\n\nobject UserItem {\n  val UserEntryNamespace: EntryNamespace = EntryNamespace(\"user\")\n}\n\ncase class UserItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  override val isMarkUnread: Option[Boolean],\n  displayType: UserDisplayType,\n  promotedMetadata: Option[PromotedMetadata],\n  socialContext: Option[SocialContext],\n  reactiveTriggers: Option[UserReactiveTriggers],\n  enableReactiveBlending: Option[Boolean])\n    extends TimelineItem {\n  override val entryNamespace: EntryNamespace = UserItem.UserEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/user/UserReactiveTriggers.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.user\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.reaction.TimelineReaction\n\ncase class UserReactiveTriggers(onFollow: Option[TimelineReaction])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/vertical_grid_item/VerticalGridItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\nsealed trait VerticalGridItem extends TimelineItem\n\nobject VerticalGridItemTopicTile {\n  val VerticalGridItemTopicTileEntryNamespace = EntryNamespace(\"verticalgriditemtopictile\")\n}\n\ncase class VerticalGridItemTopicTile(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  style: Option[VerticalGridItemTileStyle],\n  functionalityType: Option[VerticalGridItemTopicFunctionalityType],\n  url: Option[Url])\n    extends VerticalGridItem {\n  override val entryNamespace: EntryNamespace =\n    VerticalGridItemTopicTile.VerticalGridItemTopicTileEntryNamespace\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/vertical_grid_item/VerticalGridItemTileStyle.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item\n\nsealed trait VerticalGridItemTileStyle\n\ncase object SingleStateDefaultVerticalGridItemTileStyle extends VerticalGridItemTileStyle\ncase object DoubleStateDefaultVerticalGridItemTileStyle extends VerticalGridItemTileStyle\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/item/vertical_grid_item/VerticalGridItemTopicFunctionalityType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.item.vertical_grid_item\n\nsealed trait VerticalGridItemTopicFunctionalityType\n\ncase object PivotVerticalGridItemTopicFunctionalityType\n    extends VerticalGridItemTopicFunctionalityType\ncase object RecommendationVerticalGridItemTopicFunctionalityType\n    extends VerticalGridItemTopicFunctionalityType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/media/AspectRatio.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.media\n\ncase class AspectRatio(\n  numerator: Short,\n  denominator: Short)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/media/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt/metadata\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/media/Media.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.media\n\ncase class Media(\n  mediaEntity: Option[MediaEntity],\n  mediaKey: Option[MediaKey],\n  imagePossibleCropping: Option[List[Rect]],\n  aspectRatio: Option[AspectRatio])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/media/MediaEntity.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.media\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageVariant\n\nsealed trait MediaEntity\n\ncase class TweetMedia(\n  tweetId: Long,\n  momentId: Option[Long])\n    extends MediaEntity\n\ncase class BroadcastId(id: String) extends MediaEntity\n\ncase class Image(image: ImageVariant) extends MediaEntity\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/media/MediaKey.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.media\n\ncase class MediaKey(\n  id: Long,\n  category: Int)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/media/Rect.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.media\n\ncase class Rect(\n  left: Int,\n  top: Int,\n  width: Int,\n  height: Int)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/ArticleDetails.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\ncase class ArticleDetails(\n  articlePosition: Int,\n  shareCount: Int)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/color\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/icon\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/color\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/icon\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/Badge.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.RosettaColor\n\ncase class Badge(\n  text: Option[String],\n  textColorName: Option[RosettaColor],\n  backgroundColorName: Option[RosettaColor])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/Callback.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\ncase class Callback(endpoint: String)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/ClientEventInfo.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\ntrait HasClientEventInfo {\n  def clientEventInfo: Option[ClientEventInfo]\n}\n\n/**\n * Information used to build Client Events\n * @see [[http://go/client-events]]\n */\ncase class ClientEventInfo(\n  component: Option[String],\n  element: Option[String],\n  details: Option[ClientEventDetails],\n  action: Option[String],\n  entityToken: Option[String])\n\n/**\n * Additional client events fields\n *\n * @note if a field from [[http://go/client_app.thrift]] is needed but is not here\n *       contact the `#product-mixer` team to have it added.\n */\ncase class ClientEventDetails(\n  conversationDetails: Option[ConversationDetails],\n  timelinesDetails: Option[TimelinesDetails],\n  articleDetails: Option[ArticleDetails],\n  liveEventDetails: Option[LiveEventDetails],\n  commerceDetails: Option[CommerceDetails])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/CommerceDetails.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\ncase class CommerceDetails(\n  dropId: Option[Long],\n  shopV2Id: Option[Long],\n  productKey: Option[Long],\n  merchantId: Option[Long],\n  productIndex: Option[Int])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/ConfirmationDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\nsealed trait ConfirmationDisplayType\n\ncase object Inline extends ConfirmationDisplayType\ncase object BottomSheet extends ConfirmationDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/ConversationDetails.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\ncase class ConversationDetails(conversationSection: Option[ConversationSection])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/ConversationSection.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\nsealed trait ConversationSection\n\ncase object HighQuality extends ConversationSection\ncase object LowQuality extends ConversationSection\ncase object AbusiveQuality extends ConversationSection\ncase object RelatedTweet extends ConversationSection\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/DismissInfo.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\ncase class DismissInfo(callbacks: Option[Seq[Callback]])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/FeedbackAction.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.HorizonIcon\n\ncase class FeedbackAction(\n  feedbackType: FeedbackType,\n  prompt: Option[String],\n  confirmation: Option[String],\n  childFeedbackActions: Option[Seq[ChildFeedbackAction]],\n  feedbackUrl: Option[String],\n  hasUndoAction: Option[Boolean],\n  confirmationDisplayType: Option[ConfirmationDisplayType],\n  clientEventInfo: Option[ClientEventInfo],\n  icon: Option[HorizonIcon],\n  richBehavior: Option[RichFeedbackBehavior],\n  subprompt: Option[String],\n  encodedFeedbackRequest: Option[String])\n\ncase class ChildFeedbackAction(\n  feedbackType: FeedbackType,\n  prompt: Option[String],\n  confirmation: Option[String],\n  feedbackUrl: Option[String],\n  hasUndoAction: Option[Boolean],\n  confirmationDisplayType: Option[ConfirmationDisplayType],\n  clientEventInfo: Option[ClientEventInfo],\n  icon: Option[HorizonIcon],\n  richBehavior: Option[RichFeedbackBehavior],\n  subprompt: Option[String])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/FeedbackActionInfo.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\ntrait HasFeedbackActionInfo {\n  def feedbackActionInfo: Option[FeedbackActionInfo]\n}\n\ntrait ContainsFeedbackActionInfos {\n  def feedbackActionInfos: Seq[Option[FeedbackActionInfo]]\n}\n\ncase class FeedbackActionInfo(\n  feedbackActions: Seq[FeedbackAction],\n  feedbackMetadata: Option[String],\n  displayContext: Option[FeedbackDisplayContext],\n  clientEventInfo: Option[ClientEventInfo])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/FeedbackInfo.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\ntrait HasFeedbackInfo {\n  def feedbackInfo: Option[FeedbackInfo]\n}\n\ncase class FeedbackDisplayContext(reason: String)\n\ncase class FeedbackInfo(\n  feedbackKeys: Seq[String],\n  feedbackMetadata: Option[String],\n  displayContext: Option[FeedbackDisplayContext],\n  clientEventInfo: Option[ClientEventInfo])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/FeedbackType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\nsealed trait FeedbackType\n\ncase object Dismiss extends FeedbackType\ncase object SeeFewer extends FeedbackType\ncase object DontLike extends FeedbackType\ncase object NotRelevant extends FeedbackType\ncase object SeeMore extends FeedbackType\ncase object NotCredible extends FeedbackType\ncase object GiveFeedback extends FeedbackType\ncase object NotRecent extends FeedbackType\ncase object UnfollowEntity extends FeedbackType\ncase object Relevant extends FeedbackType\ncase object Moderate extends FeedbackType\ncase object RichBehavior extends FeedbackType\ncase object NotAboutTopic extends FeedbackType\ncase object Generic extends FeedbackType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/ImageAnimationType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\nsealed trait ImageAnimationType\n\ncase object Bounce extends ImageAnimationType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/ImageDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\nsealed trait ImageDisplayType\n\ncase object Icon extends ImageDisplayType\ncase object FullWidth extends ImageDisplayType\ncase object IconSmall extends ImageDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/ImageVariant.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.color.ColorPalette\n\ncase class ImageVariant(\n  url: String,\n  width: Int,\n  height: Int,\n  palette: Option[List[ColorPalette]])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/LiveEventDetails.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\ncase class LiveEventDetails(eventId: Option[Long])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/MarkUnreadableEntry.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\n// Track unread entries for the MarkUnread URT instruction.\ntrait MarkUnreadableEntry {\n  def isMarkUnread: Option[Boolean] = None\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/PinnableEntry.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\ntrait PinnableEntry {\n  def isPinned: Option[Boolean] = None\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/ReplaceableEntry.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\ntrait ReplaceableEntry {\n  def entryIdToReplace: Option[String] = None\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/ReplyPinState.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\nsealed trait ReplyPinState\n\nobject PinnedReplyPinState extends ReplyPinState\nobject PinnableReplyPinState extends ReplyPinState\nobject NotPinnableReplyPinState extends ReplyPinState\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/RichFeedbackBehavior.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\nsealed trait RichFeedbackBehavior\n\ncase class RichFeedbackBehaviorReportList(listId: Long, userId: Long) extends RichFeedbackBehavior\ncase class RichFeedbackBehaviorBlockUser(userId: Long) extends RichFeedbackBehavior\ncase class RichFeedbackBehaviorToggleFollowTopic(topicId: Long) extends RichFeedbackBehavior\ncase class RichFeedbackBehaviorToggleFollowTopicV2(topicId: String) extends RichFeedbackBehavior\ncase class RichFeedbackBehaviorToggleMuteList(listId: Long) extends RichFeedbackBehavior\ncase class RichFeedbackBehaviorMarkNotInterestedTopic(topicId: String) extends RichFeedbackBehavior\ncase class RichFeedbackBehaviorReplyPinState(replyPinState: ReplyPinState)\n    extends RichFeedbackBehavior\ncase class RichFeedbackBehaviorToggleMuteUser(userId: Long) extends RichFeedbackBehavior\ncase class RichFeedbackBehaviorToggleFollowUser(userId: Long) extends RichFeedbackBehavior\ncase class RichFeedbackBehaviorReportTweet(entryId: Long) extends RichFeedbackBehavior\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/SocialContext.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\nsealed trait SocialContext\n\ntrait HasSocialContext {\n  def socialContext: Option[SocialContext]\n}\n\nsealed trait GeneralContextType\ncase object LikeGeneralContextType extends GeneralContextType\ncase object FollowGeneralContextType extends GeneralContextType\ncase object MomentGeneralContextType extends GeneralContextType\ncase object ReplyGeneralContextType extends GeneralContextType\ncase object ConversationGeneralContextType extends GeneralContextType\ncase object PinGeneralContextType extends GeneralContextType\ncase object TextOnlyGeneralContextType extends GeneralContextType\ncase object FacePileGeneralContextType extends GeneralContextType\ncase object MegaPhoneGeneralContextType extends GeneralContextType\ncase object BirdGeneralContextType extends GeneralContextType\ncase object FeedbackGeneralContextType extends GeneralContextType\ncase object TopicGeneralContextType extends GeneralContextType\ncase object ListGeneralContextType extends GeneralContextType\ncase object RetweetGeneralContextType extends GeneralContextType\ncase object LocationGeneralContextType extends GeneralContextType\ncase object CommunityGeneralContextType extends GeneralContextType\ncase object NewUserGeneralContextType extends GeneralContextType\ncase object SmartblockExpirationGeneralContextType extends GeneralContextType\ncase object TrendingGeneralContextType extends GeneralContextType\ncase object SparkleGeneralContextType extends GeneralContextType\ncase object SpacesGeneralContextType extends GeneralContextType\ncase object ReplyPinGeneralContextType extends GeneralContextType\n\ncase class GeneralContext(\n  contextType: GeneralContextType,\n  text: String,\n  url: Option[String],\n  contextImageUrls: Option[List[String]],\n  landingUrl: Option[Url])\n    extends SocialContext\n\nsealed trait TopicContextFunctionalityType\ncase object BasicTopicContextFunctionalityType extends TopicContextFunctionalityType\ncase object RecommendationTopicContextFunctionalityType extends TopicContextFunctionalityType\ncase object RecWithEducationTopicContextFunctionalityType extends TopicContextFunctionalityType\n\ncase class TopicContext(\n  topicId: String,\n  functionalityType: Option[TopicContextFunctionalityType])\n    extends SocialContext\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/TimelinesDetails.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\ncase class TimelinesDetails(\n  injectionType: Option[String],\n  controllerData: Option[String],\n  sourceData: Option[String])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata/Url.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.metadata\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.richtext.ReferenceObject\n\nsealed trait UrlType\ncase object ExternalUrl extends UrlType\ncase object DeepLink extends UrlType\ncase object UrtEndpoint extends UrlType\n\ncase class UrtEndpointOptions(\n  requestParams: Option[Map[String, String]],\n  title: Option[String],\n  cacheId: Option[String],\n  subtitle: Option[String])\n\ncase class Url(urlType: UrlType, url: String, urtEndpointOptions: Option[UrtEndpointOptions] = None)\n    extends ReferenceObject\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation/CursorDisplayTreatment.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.operation\n\ncase class CursorDisplayTreatment(\n  actionText: Option[String],\n  labelText: Option[String])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation/CursorItem.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.operation\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ClientEventInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.FeedbackActionInfo\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorOperation.CursorEntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineItem\n\n/**\n * CursorItem should only be used for Module cursors\n * For timeline cursors, see\n * [[com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorOperation]]\n */\ncase class CursorItem(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  override val clientEventInfo: Option[ClientEventInfo],\n  override val feedbackActionInfo: Option[FeedbackActionInfo],\n  value: String,\n  cursorType: CursorType,\n  displayTreatment: Option[CursorDisplayTreatment])\n    extends TimelineItem {\n\n  override val entryNamespace: EntryNamespace = CursorEntryNamespace\n\n  override lazy val entryIdentifier: String =\n    s\"$entryNamespace-${cursorType.entryNamespace}-$id\"\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation/CursorOperation.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.operation\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.operation.CursorOperation.CursorEntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineEntry\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.TimelineOperation\n\nobject CursorOperation {\n  val CursorEntryNamespace = EntryNamespace(\"cursor\")\n\n  private def entryIdentifier(cursorType: CursorType, identifier: Long): String =\n    s\"$CursorEntryNamespace-${cursorType.entryNamespace.toString}-$identifier\"\n}\n\ncase class CursorOperation(\n  override val id: Long,\n  override val sortIndex: Option[Long],\n  value: String,\n  cursorType: CursorType,\n  displayTreatment: Option[CursorDisplayTreatment],\n  idToReplace: Option[Long])\n    extends TimelineOperation {\n  override val entryNamespace: EntryNamespace = CursorEntryNamespace\n\n  override lazy val entryIdentifier: String = CursorOperation.entryIdentifier(cursorType, id)\n\n  override def entryIdToReplace: Option[String] =\n    idToReplace.map(CursorOperation.entryIdentifier(cursorType, _))\n\n  override def withSortIndex(sortIndex: Long): TimelineEntry = copy(sortIndex = Some(sortIndex))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/operation/CursorType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.operation\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.EntryNamespace\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.HasEntryNamespace\n\nsealed trait CursorType extends HasEntryNamespace\n\ncase object TopCursor extends CursorType {\n  override val entryNamespace: EntryNamespace = EntryNamespace(\"top\")\n}\ncase object BottomCursor extends CursorType {\n  override val entryNamespace: EntryNamespace = EntryNamespace(\"bottom\")\n}\ncase object GapCursor extends CursorType {\n  override val entryNamespace: EntryNamespace = EntryNamespace(\"gap\")\n}\ncase object PivotCursor extends CursorType {\n  override val entryNamespace: EntryNamespace = EntryNamespace(\"pivot\")\n}\ncase object SubBranchCursor extends CursorType {\n  override val entryNamespace: EntryNamespace = EntryNamespace(\"subbranch\")\n}\ncase object ShowMoreCursor extends CursorType {\n  override val entryNamespace: EntryNamespace = EntryNamespace(\"showmore\")\n}\ncase object ShowMoreThreadsCursor extends CursorType {\n  override val entryNamespace: EntryNamespace = EntryNamespace(\"showmorethreads\")\n}\ncase object ShowMoreThreadsPromptCursor extends CursorType {\n  override val entryNamespace: EntryNamespace = EntryNamespace(\"showmorethreadsprompt\")\n}\ncase object SecondRepliesSectionCursor extends CursorType {\n  override val entryNamespace: EntryNamespace = EntryNamespace(\"secondrepliessection\")\n}\ncase object ThirdRepliesSectionCursor extends CursorType {\n  override val entryNamespace: EntryNamespace = EntryNamespace(\"thirdrepliessection\")\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/AdMetadataContainer.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\ncase class AdMetadataContainer(\n  removePromotedAttributionForPreroll: Option[Boolean],\n  sponsorshipCandidate: Option[String],\n  sponsorshipOrganization: Option[String],\n  sponsorshipOrganizationWebsite: Option[String],\n  sponsorshipType: Option[SponsorshipType],\n  disclaimerType: Option[DisclaimerType],\n  skAdNetworkDataList: Option[Seq[SkAdNetworkData]],\n  unifiedCardOverride: Option[String])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n    ],\n    exports = [\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/CallToAction.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\ncase class CallToAction(\n  callToActionType: Option[String],\n  url: Option[String])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/ClickTrackingInfo.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\nimport scala.collection.Map\n\ncase class ClickTrackingInfo(\n  urlParams: Map[String, String],\n  urlOverride: Option[String],\n  urlOverrideType: Option[UrlOverrideType])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/DisclaimerType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\nsealed trait DisclaimerType\n\nobject DisclaimerPolitical extends DisclaimerType\nobject DisclaimerIssue extends DisclaimerType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/DisclosureType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\nsealed trait DisclosureType\n\ncase object NoDisclosure extends DisclosureType\ncase object Political extends DisclosureType\ncase object Earned extends DisclosureType\ncase object Issue extends DisclosureType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/DynamicPrerollType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\nsealed trait DynamicPrerollType\n\nobject Amplify extends DynamicPrerollType\nobject Marketplace extends DynamicPrerollType\nobject LiveTvEvent extends DynamicPrerollType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/MediaInfo.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\ncase class MediaInfo(\n  uuid: Option[String],\n  publisherId: Option[Long],\n  callToAction: Option[CallToAction],\n  durationMillis: Option[Int],\n  videoVariants: Option[Seq[VideoVariant]],\n  advertiserName: Option[String],\n  renderAdByAdvertiserName: Option[Boolean],\n  advertiserProfileImageUrl: Option[String])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/Preroll.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\ncase class Preroll(\n  prerollId: Option[String],\n  dynamicPrerollType: Option[DynamicPrerollType],\n  mediaInfo: Option[MediaInfo])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/PrerollMetadata.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\ncase class PrerollMetadata(\n  preroll: Option[Preroll],\n  videoAnalyticsScribePassthrough: Option[String])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/PromotedMetadata.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\n/*\n * As per discussion with #revenue-serving on 9/22/2017, `impressionId` should be set from `impressionString`.\n * impressionId often returns None from adserver, as it's been replaced with impressionString.\n *\n * However, Android (at least) crashes without impressionId filled out in the response.\n *\n * So, we've removed `impressionId` from this case class, and our marshaller will set both `impressionId`\n * and `impressionString` in the render thrift from `impressionString`.\n */\n\ncase class PromotedMetadata(\n  advertiserId: Long,\n  disclosureType: Option[DisclosureType],\n  experimentValues: Option[Map[String, String]],\n  promotedTrendId: Option[Long],\n  promotedTrendName: Option[String],\n  promotedTrendQueryTerm: Option[String],\n  adMetadataContainer: Option[AdMetadataContainer],\n  promotedTrendDescription: Option[String],\n  impressionString: Option[String],\n  clickTrackingInfo: Option[ClickTrackingInfo])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/SkAdNetworkData.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\ncase class SkAdNetworkData(\n  version: Option[String], // version of the SKAdNetwork protocol\n  srcAppId: Option[String], // app showing the ad (Twitter app or app promoting through MOPUB)\n  dstAppId: Option[String], // app being promoted\n  adNetworkId: Option[String], // the ad-network-id being used\n  campaignId: Option[Long], // the sk-campaign-id - different from the Twitter campaign id\n  impressionTimeInMillis: Option[Long], // the timestamp of the impression\n  nonce: Option[String], // nonce used to generate the signature\n  signature: Option[String], // the signed payload\n  fidelityType: Option[Long] // th\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/SponsorshipType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\nsealed trait SponsorshipType\n\ncase object DirectSponsorshipType extends SponsorshipType\ncase object IndirectSponsorshipType extends SponsorshipType\ncase object NoSponsorshipSponsorshipType extends SponsorshipType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/UrlOverrideType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\nsealed trait UrlOverrideType\n\nobject UnknownUrlOverrideType extends UrlOverrideType\nobject DcmUrlOverrideType extends UrlOverrideType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/promoted/VideoVariant.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.promoted\n\ncase class VideoVariant(\n  url: Option[String],\n  contentType: Option[String],\n  bitrate: Option[Int])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/reaction/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n    ],\n    exports = [\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/reaction/TimelineReaction.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.reaction\n\ncase class TimelineReaction(\n  execution: TimelineReactionExecution,\n  maxExecutionCount: Option[Short])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/reaction/TimelineReactionExecution.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.reaction\n\nsealed abstract class TimelineReactionExecution\n\ncase class ImmediateTimelineReaction(key: String) extends TimelineReactionExecution\n\ncase class RemoteTimelineReaction(\n  requestParams: Map[String, String],\n  timeoutInSeconds: Option[Short])\n    extends TimelineReactionExecution\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n    ],\n    exports = [\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext/ReferenceObject.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.richtext\n\ntrait ReferenceObject\n\ncase class RichTextUser(id: Long) extends ReferenceObject\ncase class RichTextMention(id: Long, screenName: String) extends ReferenceObject\ncase class RichTextHashtag(text: String) extends ReferenceObject\ncase class RichTextCashtag(text: String) extends ReferenceObject\ncase class RichTextList(id: Long, url: String) extends ReferenceObject\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext/RichText.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.richtext\n\ncase class RichText(\n  text: String,\n  entities: List[RichTextEntity],\n  rtl: Option[Boolean],\n  alignment: Option[RichTextAlignment])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext/RichTextAlignment.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.richtext\n\nsealed trait RichTextAlignment\n\ncase object Natural extends RichTextAlignment\ncase object Center extends RichTextAlignment\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext/RichTextEntity.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.richtext\n\ncase class RichTextEntity(\n  fromIndex: Int,\n  toIndex: Int,\n  ref: Option[ReferenceObject],\n  format: Option[RichTextFormat])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/richtext/RichTextFormat.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.richtext\n\nsealed trait RichTextFormat {\n  def name: String\n}\n\ncase object Plain extends RichTextFormat {\n  override val name: String = \"Plain\"\n}\n\ncase object Strong extends RichTextFormat {\n  override val name: String = \"Strong\"\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module/AdsMetadata.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module\n\ncase class AdsMetadata(carouselId: Option[Long])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/metadata\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module/GridCarouselMetadata.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module\n\ncase class GridCarouselMetadata(numRows: Option[Int])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module/ModuleConversationMetadata.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\n\ncase class ModuleConversationMetadata(\n  allTweetIds: Option[Seq[Long]],\n  socialContext: Option[SocialContext],\n  enableDeduplication: Option[Boolean])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module/ModuleDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module\n\nsealed trait ModuleDisplayType\n\ncase object Carousel extends ModuleDisplayType\ncase object CompactCarousel extends ModuleDisplayType\ncase object ConversationTree extends ModuleDisplayType\ncase object GridCarousel extends ModuleDisplayType\ncase object Vertical extends ModuleDisplayType\ncase object VerticalConversation extends ModuleDisplayType\ncase object VerticalGrid extends ModuleDisplayType\ncase object VerticalWithContextLine extends ModuleDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module/ModuleFooter.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.Url\n\ncase class ModuleFooter(\n  text: String,\n  landingUrl: Option[Url])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module/ModuleHeader.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module\n\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.icon.HorizonIcon\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.ImageVariant\nimport com.twitter.product_mixer.core.model.marshalling.response.urt.metadata.SocialContext\n\ncase class ModuleHeader(\n  text: String,\n  sticky: Option[Boolean],\n  icon: Option[HorizonIcon],\n  customIcon: Option[ImageVariant],\n  socialContext: Option[SocialContext],\n  moduleHeaderDisplayType: ModuleHeaderDisplayType)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module/ModuleHeaderDisplayType.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module\n\nsealed trait ModuleHeaderDisplayType\n\ncase object Classic extends ModuleHeaderDisplayType\ncase object ContextEmphasis extends ModuleHeaderDisplayType\ncase object ClassicNoDivider extends ModuleHeaderDisplayType\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module/ModuleMetadata.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module\n\nobject ModuleMetadata {\n  def isConversationModule(moduleMetadata: Option[ModuleMetadata]): Boolean =\n    moduleMetadata.map(_.conversationMetadata).isDefined\n}\n\ncase class ModuleMetadata(\n  adsMetadata: Option[AdsMetadata],\n  conversationMetadata: Option[ModuleConversationMetadata],\n  gridCarouselMetadata: Option[GridCarouselMetadata])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt/timeline_module/ModuleShowMoreBehavior.scala",
    "content": "package com.twitter.product_mixer.core.model.marshalling.response.urt.timeline_module\n\nsealed trait ModuleShowMoreBehavior\n\ncase class ModuleShowMoreBehaviorRevealByCount(\n  initialItemsCount: Int,\n  showMoreItemsCount: Int)\n    extends ModuleShowMoreBehavior\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/ABDeciderModule.scala",
    "content": "package com.twitter.product_mixer.core.module\n\nimport com.google.inject.Provides\nimport com.twitter.abdecider.ABDeciderFactory\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.logging._\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ScribeABImpressions\nimport javax.inject.Singleton\n\nobject ABDeciderModule extends TwitterModule {\n  private val YmlPath = \"/usr/local/config/abdecider/abdecider.yml\"\n\n  @Provides\n  @Singleton\n  def provideLoggingABDecider(\n    @Flag(ScribeABImpressions) isScribeAbImpressions: Boolean,\n    stats: StatsReceiver\n  ): LoggingABDecider = {\n    val clientEventsHandler: HandlerFactory =\n      if (isScribeAbImpressions) {\n        QueueingHandler(\n          maxQueueSize = 10000,\n          handler = ScribeHandler(\n            category = \"client_event\",\n            formatter = BareFormatter,\n            level = Some(Level.INFO),\n            statsReceiver = stats.scope(\"abdecider\"))\n        )\n      } else { () =>\n        NullHandler\n      }\n\n    val factory = LoggerFactory(\n      node = \"abdecider\",\n      level = Some(Level.INFO),\n      useParents = false,\n      handlers = clientEventsHandler :: Nil\n    )\n\n    val abDeciderFactory = ABDeciderFactory(\n      abDeciderYmlPath = YmlPath,\n      scribeLogger = Some(factory()),\n      environment = Some(\"production\")\n    )\n\n    abDeciderFactory.buildWithLogging()\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"abdecider\",\n        \"decider\",\n        \"featureswitches/featureswitches-core/src/main/scala/com/twitter/featureswitches/v2/builder\",\n        \"finatra-internal/decider/src/main/scala\",\n        \"finatra-internal/international/src/main/scala/com/twitter/finatra/international/modules\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"finatra/jackson/src/main/scala/com/twitter/finatra/jackson/modules\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/product_mixer_flags\",\n        \"servo/decider\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"timelines/src/main/scala/com/twitter/timelines/features/app\",\n        \"util-internal/scribe\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"abdecider\",\n        \"decider\",\n        \"finatra-internal/decider/src/main/scala\",\n        \"finatra-internal/international/src/main/scala/com/twitter/finatra/international/modules\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"finatra/jackson/src/main/scala/com/twitter/finatra/jackson/modules\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/product_mixer_flags\",\n        \"servo/decider\",\n        \"timelines/src/main/scala/com/twitter/timelines/features/app\",\n        \"util-internal/scribe\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/ConfigApiModule.scala",
    "content": "package com.twitter.product_mixer.core.module\n\nimport com.google.inject.Provides\nimport com.twitter.decider.Decider\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.core.functional_component.configapi.ConfigBuilder\nimport com.twitter.servo.decider.DeciderGateBuilder\nimport com.twitter.timelines.configapi.Config\nimport javax.inject.Singleton\n\nobject ConfigApiModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def providesDeciderGateBuilder(decider: Decider): DeciderGateBuilder =\n    new DeciderGateBuilder(decider)\n\n  @Provides\n  @Singleton\n  def providesConfig(configBuilder: ConfigBuilder): Config = configBuilder.build()\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/FeatureSwitchesModule.scala",
    "content": "package com.twitter.product_mixer.core.module\n\nimport com.google.inject.Provides\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.featureswitches.v2.FeatureSwitches\nimport com.twitter.featureswitches.v2.builder.FeatureSwitchesBuilder\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ConfigRepoLocalPath\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.FeatureSwitchesPath\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ServiceLocal\nimport com.twitter.timelines.features.app.ForcibleFeatureValuesModule\nimport javax.inject.Singleton\n\nobject FeatureSwitchesModule extends TwitterModule with ForcibleFeatureValuesModule {\n  private val DefaultConfigRepoPath = \"/usr/local/config\"\n\n  @Provides\n  @Singleton\n  def providesFeatureSwitches(\n    abDecider: LoggingABDecider,\n    statsReceiver: StatsReceiver,\n    @Flag(ServiceLocal) isServiceLocal: Boolean,\n    @Flag(ConfigRepoLocalPath) localConfigRepoPath: String,\n    @Flag(FeatureSwitchesPath) featuresPath: String\n  ): FeatureSwitches = {\n    val configRepoPath = if (isServiceLocal) {\n      localConfigRepoPath\n    } else {\n      DefaultConfigRepoPath\n    }\n\n    val baseBuilder = FeatureSwitchesBuilder\n      .createDefault(featuresPath, abDecider, Some(statsReceiver))\n      .configRepoAbsPath(configRepoPath)\n      .forcedValues(getFeatureSwitchOverrides)\n      // Track stats when an experiment impression is made. For example:\n      // \"experiment_impressions/test_experiment_1234/\"\n      // \"experiment_impressions/test_experiment_1234/control\"\n      // \"experiment_impressions/test_experiment_1234/treatment\"\n      .experimentImpressionStatsEnabled(true)\n      .unitsOfDiversionEnable(true)\n\n    val finalBuilder = if (isServiceLocal) {\n      baseBuilder\n    } else {\n      baseBuilder.serviceDetailsFromAurora()\n    }\n\n    finalBuilder.build()\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/LoggingThrowableExceptionMapper.scala",
    "content": "package com.twitter.product_mixer.core.module\n\nimport com.twitter.finatra.thrift.exceptions.ExceptionMapper\nimport com.twitter.inject.Logging\nimport com.twitter.util.Future\nimport javax.inject.Singleton\nimport scala.util.control.NonFatal\n\n/**\n * Similar to [[com.twitter.finatra.thrift.internal.exceptions.ThrowableExceptionMapper]]\n *\n * But this one also logs the exceptions.\n */\n@Singleton\nclass LoggingThrowableExceptionMapper extends ExceptionMapper[Throwable, Nothing] with Logging {\n\n  override def handleException(throwable: Throwable): Future[Nothing] = {\n    error(\"Unhandled Exception\", throwable)\n\n    throwable match {\n      case NonFatal(e) => Future.exception(e)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/PipelineExecutionLoggerModule.scala",
    "content": "package com.twitter.product_mixer.core.module\n\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.core.service.pipeline_execution_logger.AllowListedPipelineExecutionLogger\nimport com.twitter.product_mixer.core.service.pipeline_execution_logger.PipelineExecutionLogger\n\nobject PipelineExecutionLoggerModule extends TwitterModule {\n\n  override protected def configure(): Unit = {\n    bind[PipelineExecutionLogger].to[AllowListedPipelineExecutionLogger]\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/ProductMixerModule.scala",
    "content": "package com.twitter.product_mixer.core.module\n\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule\nimport com.twitter.finatra.decider.modules.DeciderModule\nimport com.twitter.finatra.international.modules.LanguagesModule\nimport com.twitter.product_mixer.core.product.guice.ProductScopeModule\nimport com.twitter.finatra.jackson.modules.ScalaObjectMapperModule\nimport com.twitter.inject.thrift.modules.ThriftClientIdModule\n\n/**\n * ProductMixerModule provides modules required by all Product Mixer services.\n *\n * @note if your service calls Strato you will need to add the [[StratoClientModule]] yourself.\n */\nobject ProductMixerModule extends TwitterModule {\n\n  override val modules = Seq(\n    ABDeciderModule,\n    ConfigApiModule,\n    DeciderModule,\n    FeatureSwitchesModule,\n    LanguagesModule,\n    PipelineExecutionLoggerModule,\n    ProductMixerFlagModule,\n    new ProductScopeModule(),\n    ScalaObjectMapperModule,\n    ThriftClientIdModule,\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/StratoClientModule.scala",
    "content": "package com.twitter.product_mixer.core.module\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.ssl.OpportunisticTls\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ServiceLocal\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.StratoLocalRequestTimeout\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.client.Strato\nimport com.twitter.util.Duration\nimport javax.inject.Singleton\n\n/**\n * Product Mixer prefers to use a single strato client module over having a variety with different\n * timeouts. Latency Budgets in Product Mixer systems should be defined at the application layer.\n */\nobject StratoClientModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def providesStratoClient(\n    serviceIdentifier: ServiceIdentifier,\n    @Flag(ServiceLocal) isServiceLocal: Boolean,\n    @Flag(StratoLocalRequestTimeout) timeout: Option[Duration]\n  ): Client = {\n    val stratoClient = Strato.client.withMutualTls(serviceIdentifier, OpportunisticTls.Required)\n\n    // For local development it can be useful to have a larger timeout than the Strato default of\n    // 280ms. We strongly discourage setting client-level timeouts outside of this use-case. We\n    // recommend setting an overall timeout for your pipeline's end-to-end running time.\n    if (isServiceLocal && timeout.isDefined)\n      stratoClient.withRequestTimeout(timeout.get).build()\n    else {\n      stratoClient.build()\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/product_mixer_flags/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-app/src/main/java/com/twitter/inject/annotations\",\n        \"finatra/inject/inject-core/src/main/scala\",\n    ],\n    exports = [\n        \"finatra/inject/inject-app/src/main/java/com/twitter/inject/annotations\",\n        \"finatra/inject/inject-core/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/product_mixer_flags/ProductMixerFlagModule.scala",
    "content": "package com.twitter.product_mixer.core.module.product_mixer_flags\n\nimport com.twitter.inject.annotations.Flags\nimport com.twitter.inject.Injector\nimport com.twitter.inject.TwitterModule\nimport com.twitter.util.Duration\n\nobject ProductMixerFlagModule extends TwitterModule {\n  final val ServiceLocal = \"service.local\"\n  final val ConfigRepoLocalPath = \"configrepo.local_path\"\n  final val FeatureSwitchesPath = \"feature_switches.path\"\n  final val StratoLocalRequestTimeout = \"strato.local.request_timeout\"\n  final val ScribeABImpressions = \"scribe.ab_impressions\"\n  final val PipelineExecutionLoggerAllowList = \"pipeline_execution_logger.allow_list\"\n\n  flag[Boolean](\n    name = ServiceLocal,\n    default = false,\n    help = \"Is the server running locally or in a DC\")\n\n  flag[String](\n    name = ConfigRepoLocalPath,\n    default = System.getProperty(\"user.home\") + \"/workspace/config\",\n    help = \"Path to your local config repo\"\n  )\n\n  flag[Boolean](\n    name = ScribeABImpressions,\n    help = \"Enable scribing of AB impressions\"\n  )\n\n  flag[String](\n    name = FeatureSwitchesPath,\n    help = \"Path to your local config repo\"\n  )\n\n  flag[Duration](\n    name = StratoLocalRequestTimeout,\n    help = \"Override the request timeout for Strato when running locally\"\n  )\n\n  flag[Seq[String]](\n    name = PipelineExecutionLoggerAllowList,\n    default = Seq.empty,\n    help =\n      \"Specify user role(s) for which detailed log messages (containing PII) can be made. Accepts a single role or a comma separated list 'a,b,c'\"\n  )\n\n  /**\n   * Invoked at the end of server startup.\n   *\n   * If we're running locally, we display a nice message and a link to the admin page\n   */\n  override def singletonPostWarmupComplete(injector: Injector): Unit = {\n    val isLocalService = injector.instance[Boolean](Flags.named(ServiceLocal))\n    if (isLocalService) {\n      // Extract service name from clientId since there isn't a specific flag for that\n      val clientId = injector.instance[String](Flags.named(\"thrift.clientId\"))\n      val name = clientId.split(\"\\\\.\")(0)\n\n      val adminPort = injector.instance[String](Flags.named(\"admin.port\"))\n      val url = s\"http://localhost$adminPort/\"\n\n      // Print instead of log, so it goes on stdout and doesn't get the logging decorations.\n      // Update our local development recipe (local-development.rst) if making changes to this\n      // message.\n      println(\"===============================================================================\")\n      println(s\"Welcome to a Product Mixer Service, $name\")\n      println(s\"You can view the admin endpoint and thrift web forms at $url\")\n      println(\"Looking for support? Have questions? #product-mixer on Slack.\")\n      println(\"===============================================================================\")\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/stringcenter/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-app/src/main/java/com/twitter/inject/annotations\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"stringcenter/client/src/main/scala/com/twitter/stringcenter/client\",\n        \"stringcenter/client/src/main/scala/com/twitter/stringcenter/client/sources\",\n        \"stringcenter/client/src/main/scala/com/twitter/stringcenter/client/sources/refreshing\",\n        \"util/util-jackson/src/main/scala/com/twitter/util/jackson\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope\",\n        \"stringcenter/client/src/main/scala/com/twitter/stringcenter/client\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/stringcenter/ProductScopeStringCenterModule.scala",
    "content": "package com.twitter.product_mixer.core.module.stringcenter\n\nimport com.google.inject.Provides\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.jackson.ScalaObjectMapper\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport com.twitter.stringcenter.client.ExternalStringRegistry\nimport com.twitter.stringcenter.client.StringCenter\nimport com.twitter.stringcenter.client.StringCenterClientConfig\nimport com.twitter.stringcenter.client.sources.RefreshingStringSource\nimport com.twitter.stringcenter.client.sources.RefreshingStringSourceConfig\nimport com.twitter.stringcenter.client.sources.StringSource\nimport com.twitter.translation.Languages\nimport javax.inject.Singleton\nimport scala.collection.concurrent\n\n/*\n * Fun trivia - this has to be a Class not an Object, otherwise when you ./bazel test blah/...\n * and glob multiple feature tests together it'll reuse the concurrentMaps below across\n * executions / different server objects.\n */\nclass ProductScopeStringCenterModule extends TwitterModule {\n\n  private val loadNothing =\n    flag[Boolean](name = \"stringcenter.dontload\", default = false, help = \"Avoid loading any files\")\n\n  flag[Boolean](\n    name = \"stringcenter.handle.language.fallback\",\n    default = true,\n    help = \"Handle language fallback for services that don't already handle it\")\n\n  flag[String](\n    name = \"stringcenter.default_bundle_path\",\n    default = \"stringcenter\",\n    help = \"The path on disk to the default bundle available at startup time\")\n\n  private val refreshingInterval = flag[Int](\n    name = \"stringcenter.refresh_interval_minutes\",\n    default = 3,\n    help = \"How often to poll the refreshing bundle path to check for new bundles\")\n\n  /* The Guice injector is single threaded, but out of a preponderance of caution we use a concurrent Map.\n   *\n   * We need to ensure that we only build one StringSource, StringCenter client, and External String\n   * Registry for each String Center Project. @ProductScoped doesn't ensure this on it's own as\n   * two products can have the same String Center Project set.\n   */\n  val stringSources: concurrent.Map[String, StringSource] = concurrent.TrieMap.empty\n  val stringCenterClients: concurrent.Map[String, StringCenter] = concurrent.TrieMap.empty\n  val externalStringRegistries: concurrent.Map[String, ExternalStringRegistry] =\n    concurrent.TrieMap.empty\n\n  @ProductScoped\n  @Provides\n  def providesStringCenterClients(\n    abDecider: LoggingABDecider,\n    stringSource: StringSource,\n    languages: Languages,\n    statsReceiver: StatsReceiver,\n    clientConfig: StringCenterClientConfig,\n    product: Product\n  ): StringCenter = {\n    stringCenterClients.getOrElseUpdate(\n      stringCenterForProduct(product), {\n        new StringCenter(\n          abDecider,\n          stringSource,\n          languages,\n          statsReceiver,\n          clientConfig\n        )\n      })\n  }\n\n  @ProductScoped\n  @Provides\n  def providesExternalStringRegistries(\n    product: Product\n  ): ExternalStringRegistry = {\n    externalStringRegistries.getOrElseUpdate(\n      stringCenterForProduct(product), {\n        new ExternalStringRegistry()\n      })\n  }\n\n  @ProductScoped\n  @Provides\n  def providesStringCenterSources(\n    mapper: ScalaObjectMapper,\n    statsReceiver: StatsReceiver,\n    product: Product,\n    @Flag(\"stringcenter.default_bundle_path\") defaultBundlePath: String\n  ): StringSource = {\n    if (loadNothing()) {\n      StringSource.Empty\n    } else {\n      val stringCenterProduct = stringCenterForProduct(product)\n\n      stringSources.getOrElseUpdate(\n        stringCenterProduct, {\n          val config = RefreshingStringSourceConfig(\n            stringCenterProduct,\n            defaultBundlePath,\n            \"stringcenter/downloaded/current/stringcenter\",\n            refreshingInterval().minutes\n          )\n          new RefreshingStringSource(\n            config,\n            mapper,\n            statsReceiver\n              .scope(\"StringCenter\", \"refreshing\", \"project\", stringCenterProduct))\n        }\n      )\n    }\n  }\n\n  private def stringCenterForProduct(product: Product): String =\n    product.stringCenterProject.getOrElse {\n      throw new UnsupportedOperationException(\n        s\"No StringCenter project defined for Product ${product.identifier}\")\n    }\n\n  @Singleton\n  @Provides\n  def providesStringCenterClientConfig(\n    @Flag(\"stringcenter.handle.language.fallback\") handleLanguageFallback: Boolean\n  ): StringCenterClientConfig = {\n    StringCenterClientConfig(handleLanguageFallback = handleLanguageFallback)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/BUILD",
    "content": "scala_library(\n    name = \"query\",\n    sources = [\"PipelineQuery.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n    ],\n)\n\nscala_library(\n    name = \"executor\",\n    sources = [\n        \"FailOpenPolicy.scala\",\n        \"PipelineResult.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"util/util-core\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"util/util-core\",\n    ],\n)\n\nscala_library(\n    sources = [\n        \"!FailOpenPolicy.scala\",\n        \"!PipelineQuery.scala\",\n        \"!PipelineResult.scala\",\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":executor\",\n        \":query\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"scrooge/scrooge-serializer/src/main/scala\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \":executor\",\n        \":query\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"scrooge/scrooge-serializer/src/main/scala\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/CandidatePipelineFeatures.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\n\nprivate[core] object CandidatePipelineResults\n    extends Feature[PipelineQuery, Seq[CandidateWithDetails]]\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/FailOpenPolicy.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MisconfiguredFeatureMapFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureCategory\n\n/**\n * [[FailOpenPolicy]] determines what should happen in the event that a candidate pipeline fails\n * to execute successfully.\n *\n * Exercise caution when creating new fail open policies. Product Mixer will fail open by default in\n * certain error cases (e.g. closed gate on a candidate pipeline) but these might inadvertently be\n * excluded by a new policy.\n */\ntrait FailOpenPolicy {\n  def apply(failureCategory: PipelineFailureCategory): Boolean\n}\n\nobject FailOpenPolicy {\n\n  /**\n   * Always fail open on candidate pipeline failures except\n   * for [[MisconfiguredFeatureMapFailure]]s because it's a programmer error\n   * and should always fail loudly, even with an [[Always]] p[[FailOpenPolicy]]\n   */\n  val Always: FailOpenPolicy = (category: PipelineFailureCategory) => {\n    category != MisconfiguredFeatureMapFailure\n  }\n\n  /**\n   * Never fail open on candidate pipeline failures.\n   *\n   * @note this is more restrictive than the default behavior which is to allow gate closed\n   *       failures.\n   */\n  val Never: FailOpenPolicy = (_: PipelineFailureCategory) => false\n\n  // Build a policy that will fail open for a given set of categories\n  def apply(categories: Set[PipelineFailureCategory]): FailOpenPolicy =\n    (failureCategory: PipelineFailureCategory) =>\n      categories\n        .contains(failureCategory)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/InvalidStepStateException.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\n\ncase class InvalidStepStateException(step: PipelineStepIdentifier, missingData: String)\n    extends Exception(\n      s\"Invalid Step State: Step $step requires $missingData\"\n    )\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/NewPipelineArrowBuilder.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.state.HasExecutorResults\nimport com.twitter.product_mixer.core.pipeline.state.HasResult\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorStatus\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.Executor.Context\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Arrow.Iso\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\n\n/**\n * Pipeline Arrow Builder used for constructing a final arrow for a pipeline after adding necessary\n * steps.\n *\n * @param steps The kept non-empty Pipeline Steps\n * @param addedSteps Steps that have been added, but not necessarily kept.\n * @param statsReceiver Stats Receiver for metric book keeping\n * @tparam Result sThe expected final result type of the pipeline.\n * @tparam State The input state type, which should implement [[HasResult]].\n */\ncase class NewPipelineArrowBuilder[\n  Result,\n  State <: HasExecutorResults[State] with HasResult[Result]\n] private (\n  private val steps: Seq[PipelineStep[State, _, _, _]],\n  override val statsReceiver: StatsReceiver)\n    extends Executor {\n\n  def add[Config, ExecutorInput, ExResult <: ExecutorResult](\n    pipelineStepIdentifier: PipelineStepIdentifier,\n    step: Step[State, Config, ExecutorInput, ExResult],\n    executorConfig: Config\n  ): NewPipelineArrowBuilder[Result, State] = {\n    require(\n      !steps.contains(pipelineStepIdentifier),\n      s\"Found duplicate step $pipelineStepIdentifier when building pipeline arrow\")\n\n    // If the step has nothing to execute, drop it for simplification but still added it to the\n    // \"addedSteps\" field for build time validation\n    if (step.isEmpty(executorConfig)) {\n      this\n    } else {\n      val newPipelineStep =\n        PipelineStep(pipelineStepIdentifier, executorConfig, step)\n      val newSteps = steps :+ newPipelineStep\n      this.copy(steps = newSteps)\n    }\n  }\n\n  def buildArrow(\n    context: Executor.Context\n  ): Arrow[State, NewPipelineResult[Result]] = {\n    val initialArrow = Arrow\n      .map { input: State => NewStepData[State](input) }\n    val allStepArrows = steps.map { step =>\n      Iso.onlyIf[NewStepData[State]] { stepData => !stepData.stopExecuting } {\n        wrapStepWithExecutorBookkeeping(step, context)\n      }\n    }\n    val combinedArrow = isoArrowsSequentially(allStepArrows)\n    val resultArrow = Arrow.map { stepData: NewStepData[State] =>\n      stepData.pipelineFailure match {\n        case Some(failure) =>\n          NewPipelineResult.Failure(failure, stepData.pipelineState.executorResultsByPipelineStep)\n        case None =>\n          NewPipelineResult.Success(\n            stepData.pipelineState.buildResult,\n            stepData.pipelineState.executorResultsByPipelineStep)\n      }\n    }\n    initialArrow.andThen(combinedArrow).andThen(resultArrow)\n  }\n\n  private[this] def wrapStepWithExecutorBookkeeping(\n    step: PipelineStep[State, _, _, _],\n    context: Context\n  ): Arrow.Iso[NewStepData[State]] = {\n    val wrapped = wrapStepWithExecutorBookkeeping[NewStepData[State], NewStepData[State]](\n      context = context,\n      identifier = step.stepIdentifier,\n      arrow = step.arrow(context),\n      // extract the failure only if it's present. Not sure if this is needed???\n      transformer = _.pipelineFailure.map(Throw(_)).getOrElse(Return.Unit)\n    )\n\n    Arrow\n      .zipWithArg(wrapped.liftToTry)\n      .map {\n        case (_: NewStepData[State], Return(result)) =>\n          // if Step was successful, return the result\n          result\n        case (previous: NewStepData[State], Throw(pipelineFailure: PipelineFailure)) =>\n          // if the Step failed in such a way that the failure was NOT captured\n          // in the result object, then update the State with the failure\n          previous.withFailure(pipelineFailure)\n        case (_, Throw(ex)) =>\n          // an exception was thrown which was not handled by the failure classifier\n          // this only happens with cancellation exceptions which are re-thrown\n          throw ex\n      }\n  }\n\n  /**\n   * Sets up stats [[com.twitter.finagle.stats.Gauge]]s for any [[QualityFactorStatus]]\n   *\n   * @note We use provideGauge so these gauges live forever even without a reference.\n   */\n  private[pipeline] def buildGaugesForQualityFactor(\n    pipelineIdentifier: ComponentIdentifier,\n    qualityFactorStatus: QualityFactorStatus,\n    statsReceiver: StatsReceiver\n  ): Unit = {\n    qualityFactorStatus.qualityFactorByPipeline.foreach {\n      case (identifier, qualityFactor) =>\n        // QF is a relative stat (since the parent pipeline is monitoring a child pipeline)\n        val scopes = pipelineIdentifier.toScopes ++ identifier.toScopes :+ \"QualityFactor\"\n        statsReceiver.provideGauge(scopes: _*) { qualityFactor.currentValue.toFloat }\n    }\n  }\n}\n\nobject NewPipelineArrowBuilder {\n  def apply[Result, InputState <: HasExecutorResults[InputState] with HasResult[Result]](\n    statsReceiver: StatsReceiver\n  ): NewPipelineArrowBuilder[Result, InputState] = {\n    NewPipelineArrowBuilder(\n      Seq.empty,\n      statsReceiver\n    )\n  }\n}\n\n/**\n * This is a pipeline specific instance of a step, i.e, a generic step with the step identifier\n * within the pipeline and its executor configs.\n * @param stepIdentifier Step identifier of the step within a pipeline\n * @param executorConfig Config to execute the step with\n * @param step The underlying step to be used\n * @tparam InputState The input state object\n * @tparam ExecutorConfig The config expected for the given step\n * @tparam ExecutorInput Input for the underlying executor\n * @tparam ExecResult The result type\n */\ncase class PipelineStep[\n  State <: HasExecutorResults[State],\n  PipelineStepConfig,\n  ExecutorInput,\n  ExecResult <: ExecutorResult\n](\n  stepIdentifier: PipelineStepIdentifier,\n  executorConfig: PipelineStepConfig,\n  step: Step[State, PipelineStepConfig, ExecutorInput, ExecResult]) {\n\n  def arrow(\n    context: Executor.Context\n  ): Arrow.Iso[NewStepData[State]] = {\n    val inputArrow = Arrow.map { stepData: NewStepData[State] =>\n      step.adaptInput(stepData.pipelineState, executorConfig)\n    }\n\n    Arrow\n      .zipWithArg(inputArrow.andThen(step.arrow(executorConfig, context))).map {\n        case (stepData: NewStepData[State], executorResult: ExecResult @unchecked) =>\n          val updatedResultsByPipelineStep =\n            stepData.pipelineState.executorResultsByPipelineStep + (stepIdentifier -> executorResult)\n          val updatedPipelineState = step\n            .updateState(stepData.pipelineState, executorResult, executorConfig).setExecutorResults(\n              updatedResultsByPipelineStep)\n\n          NewStepData(updatedPipelineState)\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/NewPipelineBuilder.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.pipeline.state.HasExecutorResults\nimport com.twitter.product_mixer.core.pipeline.state.HasResult\n\n/**\n * A pipeline builder that is responsible for taking a PipelineConfig and creating a final pipeline\n * from it. It provides an [[NewPipelineArrowBuilder]] for composing the pipeline's underlying arrow\n * from [[Step]]s.\n *\n * @tparam Config The Pipeline Config\n * @tparam PipelineArrowResult The expected final result\n * @tparam PipelineArrowState State object for maintaining state across the pipeline.\n * @tparam OutputPipeline The final pipeline\n */\ntrait NewPipelineBuilder[\n  Config <: PipelineConfig,\n  PipelineArrowResult,\n  PipelineArrowState <: HasExecutorResults[PipelineArrowState] with HasResult[PipelineArrowResult],\n  OutputPipeline <: Pipeline[_, _]] {\n\n  type ArrowResult = PipelineArrowResult\n  type ArrowState = PipelineArrowState\n\n  def build(\n    parentComponentIdentifierStack: ComponentIdentifierStack,\n    arrowBuilder: NewPipelineArrowBuilder[ArrowResult, ArrowState],\n    config: Config\n  ): OutputPipeline\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/NewPipelineResult.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport scala.collection.immutable.ListMap\n\nsealed trait NewPipelineResult[-Result] {\n  def executorResultsByPipelineStep: ListMap[PipelineStepIdentifier, ExecutorResult]\n}\n\nobject NewPipelineResult {\n  case class Failure(\n    failure: PipelineFailure,\n    override val executorResultsByPipelineStep: ListMap[PipelineStepIdentifier, ExecutorResult])\n      extends NewPipelineResult[Any]\n\n  case class Success[Result](\n    result: Result,\n    override val executorResultsByPipelineStep: ListMap[PipelineStepIdentifier, ExecutorResult])\n      extends NewPipelineResult[Result]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/NewStepData.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.state.HasExecutorResults\n\ncase class NewStepData[State <: HasExecutorResults[State]](\n  pipelineState: State,\n  pipelineFailure: Option[PipelineFailure] = None) {\n\n  val stopExecuting = pipelineFailure.isDefined\n  def withFailure(failure: PipelineFailure): NewStepData[State] =\n    this.copy(pipelineFailure = Some(failure))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/Pipeline.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Stitch\n\n/** Base trait for all `pipeline` implementations */\ntrait Pipeline[-Query, Result] extends Component {\n\n  /** The [[PipelineConfig]] that was used to create this [[Pipeline]] */\n  private[core] val config: PipelineConfig\n\n  /** Returns the underlying arrow that represents the pipeline. This is a val because we want to ensure\n   * that the arrow is long-lived and consistent, not generated per-request.\n   *\n   * Directly using this arrow allows you to combine it with other arrows efficiently.\n   */\n  val arrow: Arrow[Query, PipelineResult[Result]]\n\n  /** all child [[Component]]s that this [[Pipeline]] contains which will be registered and monitored */\n  val children: Seq[Component]\n\n  /**\n   * A helper for executing a single query.\n   *\n   * toResultTry and lowerFromTry has the end result of adapting PipelineResult into either a\n   * successful result or a Stitch exception, which is a common use-case for callers,\n   * particularly in the case of [[com.twitter.product_mixer.core.pipeline.product.ProductPipeline]].\n   */\n  def process(query: Query): Stitch[Result] = arrow(query).map(_.toResultTry).lowerFromTry\n\n  final override def toString = s\"Pipeline(identifier=$identifier)\"\n\n  /**\n   * [[Pipeline]]s are equal to one another if they were generated from the same [[PipelineConfig]],\n   * we check this by doing a reference checks first then comparing the [[PipelineConfig]] instances.\n   *\n   * We can skip additional checks because the other fields (e.g. [[identifier]] and [[children]])\n   * are derived from the [[PipelineConfig]].\n   */\n  final override def equals(obj: Any): Boolean = obj match {\n    case pipeline: Pipeline[_, _] =>\n      pipeline.eq(this) || pipeline.config.eq(config) || pipeline.config == config\n    case _ => false\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/PipelineBuilder.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorStatus\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.Executor.Context\nimport com.twitter.product_mixer.core.service\nimport com.twitter.stitch.Arrow\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\n\ntrait PipelineBuilder[Query] extends Executor {\n\n  /**\n   * When a step is mostly the same, but only the result update changes,\n   * you can pass in a [[ResultUpdater]] to the [[Step]] to perform the update\n   * such as with multi-step hydration.\n   */\n  trait ResultUpdater[R <: PipelineResult[_], ER <: service.ExecutorResult] {\n    def apply(existingResult: R, executorResult: ER): R\n  }\n\n  type UnderlyingResultType\n  type PipelineResultType <: PipelineResult[UnderlyingResultType]\n\n  /** the data that every step has as input and output - the query, and the in-progress result */\n  case class StepData(query: Query, result: PipelineResultType)\n\n  /** An [[Arrow.Iso]] [[Arrow]] is an arrow with the same input and output types. */\n  type StepArrow = Arrow.Iso[StepData]\n\n  /**\n   * We break pipeline execution into a linear sequence of [[Step]]s. The execution logic of each\n   * step is represented as an [[Executor]] (which is reusable between pipelines).\n   *\n   * Each step has access to the [[PipelineResult]] generated by previous steps, and can update it\n   * with some new data.\n   *\n   * We define a pipeline Step as having three parts:\n   *\n   *   - An underlying [[Executor]] [[Arrow]], from the underlying executor\n   *   - An input adaptor to extract the right data from the previous [[PipelineResult]]\n   *   - A result updater to update the [[PipelineResult]]\n   *\n   * This keeps knowledge of [[PipelineResult]] out of the executors, so they're reusable.\n   *\n   * @tparam ExecutorInput The input type used by the executor\n   * @tparam ExecutorResult The output/result type used by the executor\n   */\n  trait Step[ExecutorInput, ExecutorResult] {\n    def identifier: PipelineStepIdentifier\n    def executorArrow: Arrow[ExecutorInput, ExecutorResult]\n    def inputAdaptor(query: Query, previousResult: PipelineResultType): ExecutorInput\n    def resultUpdater(\n      previousPipelineResult: PipelineResultType,\n      executorResult: ExecutorResult\n    ): PipelineResultType\n\n    /**\n     * Optionally, steps can define a function to update the Query\n     */\n    def queryUpdater(query: Query, executorResult: ExecutorResult): Query = query\n\n    /**\n     * Arrow that adapts the input, runs the underlying Executor, adapts the output, and updates the state\n     */\n    val stepArrow: StepArrow = {\n      val inputAdaptorArrow: Arrow[StepData, ExecutorInput] = Arrow.map { stepData: StepData =>\n        inputAdaptor(stepData.query, stepData.result)\n      }\n      val outputAdaptorArrow: Arrow[(StepData, ExecutorResult), StepData] = Arrow.map {\n        // abstract type pattern ExecutorResult is unchecked since it is eliminated by erasure\n        case (previousStepData: StepData, executorResult: ExecutorResult @unchecked) =>\n          StepData(\n            query = queryUpdater(previousStepData.query, executorResult),\n            result = resultUpdater(previousStepData.result, executorResult)\n          )\n      }\n\n      Arrow\n        .zipWithArg(inputAdaptorArrow.andThen(executorArrow))\n        .andThen(outputAdaptorArrow)\n    }\n  }\n\n  /**\n   * Wraps a step with [[wrapStepWithExecutorBookkeeping]]\n   *\n   * When an error is encountered in execution, we update the [[PipelineResult.failure]] field,\n   * and we return the partial results from all previously executed steps.\n   */\n  def wrapStepWithExecutorBookkeeping(\n    context: Context,\n    step: Step[_, _]\n  ): Arrow.Iso[StepData] = {\n    val wrapped = wrapStepWithExecutorBookkeeping[StepData, StepData](\n      context = context,\n      identifier = step.identifier,\n      arrow = step.stepArrow,\n      // extract the failure only if it's present\n      transformer = _.result.failure match {\n        case Some(pipelineFailure) => Throw(pipelineFailure)\n        case None => Return.Unit\n      }\n    )\n\n    Arrow\n      .zipWithArg(wrapped.liftToTry)\n      .map {\n        case (_: StepData, Return(result)) =>\n          // if Step was successful, return the result\n          result\n        case (StepData(query, previousResult), Throw(pipelineFailure: PipelineFailure)) =>\n          // if the Step failed in such a way that the failure was NOT captured\n          // in the result object, then update the State with the failure\n          StepData(\n            query,\n            previousResult.withFailure(pipelineFailure).asInstanceOf[PipelineResultType])\n        case (_, Throw(ex)) =>\n          // an exception was thrown which was not handled by the failure classifier\n          // this only happens with cancellation exceptions which are re-thrown\n          throw ex\n      }\n  }\n\n  /**\n   * Builds a combined arrow out of steps.\n   *\n   * Wraps them in error handling, and only executes each step if the previous step is successful.\n   */\n  def buildCombinedArrowFromSteps(\n    steps: Seq[Step[_, _]],\n    context: Executor.Context,\n    initialEmptyResult: PipelineResultType,\n    stepsInOrderFromConfig: Seq[PipelineStepIdentifier]\n  ): Arrow[Query, PipelineResultType] = {\n\n    validateConfigAndBuilderAreInSync(steps, stepsInOrderFromConfig)\n\n    /**\n     * Prepare the step arrows.\n     *   1. Wrap them in executor bookkeeping\n     *   2. Wrap them in Iso.onlyIf - so we only execute them if we don't have a result or failure yet\n     *   3. Combine them using [[isoArrowsSequentially]]\n     *\n     * @note this results in no Executor bookkeeping actions for [[Step]]s after\n     *       we reach a [[PipelineResult.stopExecuting]].\n     */\n    val stepArrows = isoArrowsSequentially(steps.map { step =>\n      Arrow.Iso.onlyIf[StepData](stepData => !stepData.result.stopExecuting)(\n        wrapStepWithExecutorBookkeeping(context, step))\n    })\n\n    Arrow\n      .identity[Query]\n      .map { query => StepData(query, initialEmptyResult) }\n      .andThen(stepArrows)\n      .map { case StepData(_, result) => result }\n  }\n\n  /**\n   * Sets up stats [[com.twitter.finagle.stats.Gauge]]s for any [[QualityFactorStatus]]\n   *\n   * @note We use provideGauge so these gauges live forever even without a reference.\n   */\n  private[pipeline] def buildGaugesForQualityFactor(\n    pipelineIdentifier: ComponentIdentifier,\n    qualityFactorStatus: QualityFactorStatus,\n    statsReceiver: StatsReceiver\n  ): Unit = {\n    qualityFactorStatus.qualityFactorByPipeline.foreach {\n      case (identifier, qualityFactor) =>\n        // QF is a relative stat (since the parent pipeline is monitoring a child pipeline)\n        val scopes = pipelineIdentifier.toScopes ++ identifier.toScopes :+ \"QualityFactor\"\n        statsReceiver.provideGauge(scopes: _*) { qualityFactor.currentValue.toFloat }\n    }\n  }\n\n  /** Validates that the [[PipelineConfigCompanion]] is in sync with the [[Step]]s a [[PipelineBuilder]] produces */\n  private[this] def validateConfigAndBuilderAreInSync(\n    builtSteps: Seq[Step[_, _]],\n    stepsInOrder: Seq[PipelineStepIdentifier]\n  ): Unit = {\n    require(\n      builtSteps.map(_.identifier) == stepsInOrder,\n      s\"Builder and Config are out of sync, bug in Product Mixer Core, `PipelineCompanion` and `PipelineBuilder` \" +\n        s\"have different definitions of what Steps are run in this Pipeline \\n\" +\n        s\"${builtSteps.map(_.identifier).zip(stepsInOrder).mkString(\"\\n\")}\"\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/PipelineConfig.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.product_mixer.core.model.common.identifier.HasComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\n\ntrait PipelineConfig extends HasComponentIdentifier\n\ntrait PipelineConfigCompanion {\n\n  /** used to generate `AsyncFeaturesFor` [[PipelineStepIdentifier]]s for the internal Async Features Step */\n  private[core] def asyncFeaturesStep(\n    stepToHydrateFor: PipelineStepIdentifier\n  ): PipelineStepIdentifier =\n    PipelineStepIdentifier(\"AsyncFeaturesFor\" + stepToHydrateFor.name)\n\n  /** All the Steps which are executed by a [[Pipeline]] in the order in which they are run */\n  val stepsInOrder: Seq[PipelineStepIdentifier]\n\n  val stepsAsyncFeatureHydrationCanBeCompletedBy: Set[PipelineStepIdentifier] = Set.empty\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/PipelineCursor.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\n/**\n * PipelineCursor represents any product-specific cursor model. Typically the PipelineCursor will be\n * a de-serialized base 64 thrift struct from initial request.\n */\ntrait PipelineCursor\n\n/**\n * HasPipelineCursor indicates that a [[PipelineQuery]] has a cursor\n */\ntrait HasPipelineCursor[+Cursor <: PipelineCursor] {\n  def pipelineCursor: Option[Cursor]\n\n  /**\n   * If the cursor is not present, this typically means that we are on the first page\n   */\n  def isFirstPage: Boolean = pipelineCursor.isEmpty\n}\n\n/**\n * UrtPipelineCursor represents a URT product-specific cursor model. Typically the UrtPipelineCursor\n * will be a de-serialized base 64 thrift struct from initial request.\n */\ntrait UrtPipelineCursor extends PipelineCursor {\n\n  /** See [[UrtCursorBuilder]] for background on building initialSortIndex */\n  def initialSortIndex: Long\n}\n\nobject UrtPipelineCursor {\n  def getCursorInitialSortIndex(query: PipelineQuery with HasPipelineCursor[_]): Option[Long] = {\n    query.pipelineCursor match {\n      case Some(cursor: UrtPipelineCursor) => Some(cursor.initialSortIndex)\n      case _ => None\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/PipelineCursorSerializer.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MalformedCursor\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.scrooge.BinaryThriftStructSerializer\nimport com.twitter.scrooge.ThriftStruct\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\n\n/**\n * Serializes a [[PipelineCursor]] into thrift and then into a base64 encoded string\n */\ntrait PipelineCursorSerializer[-Cursor <: PipelineCursor] {\n  def serializeCursor(cursor: Cursor): String\n}\n\nobject PipelineCursorSerializer {\n\n  /**\n   * Deserializes a cursor string into thrift and then into a [[PipelineCursor]]\n   *\n   * @param cursorString to deserialize, which is base64 encoded thrift\n   * @param cursorThriftSerializer to deserialize the cursor string into thrift\n   * @param deserializePf specifies how to transform the serialized thrift into a [[PipelineCursor]]\n   * @return optional [[PipelineCursor]]. `None` may or may not be a failure depending on the\n   *         implementation of deserializePf.\n   *\n   * @note The \"A\" type of deserializePf cannot be inferred due to the thrift type not being present\n   *       on the PipelineCursorSerializer trait. Therefore invokers must often add an explicit type\n   *       on the deserializeCursor call to help out the compiler when passing deserializePf inline.\n   *       Alternatively, deserializePf can be declared as a val with a type annotation before it is\n   *       passed into this method.\n   */\n  def deserializeCursor[Thrift <: ThriftStruct, Cursor <: PipelineCursor](\n    cursorString: String,\n    cursorThriftSerializer: BinaryThriftStructSerializer[Thrift],\n    deserializePf: PartialFunction[Option[Thrift], Option[Cursor]]\n  ): Option[Cursor] = {\n    val thriftCursor: Option[Thrift] =\n      Try {\n        cursorThriftSerializer.fromString(cursorString)\n      } match {\n        case Return(thriftCursor) => Some(thriftCursor)\n        case Throw(_) => None\n      }\n\n    // Add type annotation to help out the compiler since the type is lost due to the _ match\n    val defaultDeserializePf: PartialFunction[Option[Thrift], Option[Cursor]] = {\n      case _ =>\n        // This case is the result of the client submitting a cursor we do not expect\n        throw PipelineFailure(MalformedCursor, s\"Unknown request cursor: $cursorString\")\n    }\n\n    (deserializePf orElse defaultDeserializePf)(thriftCursor)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/PipelineQuery.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.request.HasClientContext\nimport com.twitter.product_mixer.core.model.marshalling.request.HasDebugOptions\nimport com.twitter.product_mixer.core.model.marshalling.request.HasProduct\nimport com.twitter.timelines.configapi.HasParams\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Time\n\ntrait PipelineQuery extends HasParams with HasClientContext with HasProduct with HasDebugOptions {\n  self =>\n\n  /** Set a query time val that is constant for the duration of the query lifecycle */\n  val queryTime: Time = self.debugOptions.flatMap(_.requestTimeOverride).getOrElse(Time.now)\n\n  /** The requested max results is specified, or not specified, by the thrift client */\n  def requestedMaxResults: Option[Int]\n\n  /** Retrieves the max results with a default Param, if not specified by the thrift client */\n  def maxResults(defaultRequestedMaxResultParam: Param[Int]): Int =\n    requestedMaxResults.getOrElse(params(defaultRequestedMaxResultParam))\n\n  /** Optional [[FeatureMap]], this may be updated later using [[withFeatureMap]] */\n  def features: Option[FeatureMap]\n\n  /**\n   * Since Query-Level features can be hydrated later, we need this method to update the PipelineQuery\n   * usually this will be implemented via `copy(features = Some(features))`\n   */\n  def withFeatureMap(features: FeatureMap): PipelineQuery\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/PipelineResult.scala",
    "content": "package com.twitter.product_mixer.core.pipeline\n\nimport com.twitter.product_mixer.component_library.model.candidate.CursorCandidate\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ExecutionFailed\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\n\n/**\n * Pipelines return a PipelineResult.\n *\n * This allows us to return a single main result (optionally, incase the pipeline didn't execute successfully), but\n * still have a detailed response object to show how that result was produced.\n */\ntrait PipelineResult[ResultType] {\n  val failure: Option[PipelineFailure]\n  val result: Option[ResultType]\n\n  def withFailure(failure: PipelineFailure): PipelineResult[ResultType]\n  def withResult(result: ResultType): PipelineResult[ResultType]\n\n  def resultSize(): Int\n\n  private[pipeline] def stopExecuting: Boolean = failure.isDefined || result.isDefined\n\n  final def toTry: Try[this.type] = (result, failure) match {\n    case (_, Some(failure)) =>\n      Throw(failure)\n    case (_: Some[ResultType], _) =>\n      Return(this)\n    // Pipelines should always finish with either a result or a failure\n    case _ => Throw(PipelineFailure(ExecutionFailed, \"Pipeline did not execute\"))\n  }\n\n  final def toResultTry: Try[ResultType] = {\n    // `.get` is safe here because `toTry` guarantees a value in the `Return` case\n    toTry.map(_.result.get)\n  }\n}\n\nobject PipelineResult {\n\n  /**\n   * Track number of candidates returned by a Pipeline. Cursors are excluded from this\n   * count and modules are counted as the sum of their candidates.\n   *\n   * @note this is a somewhat subjective measure of 'size' and it is spread across pipeline\n   *       definitions as well as selectors.\n   */\n  def resultSize(results: Seq[CandidateWithDetails]): Int = results.map {\n    case module: ModuleCandidateWithDetails => resultSize(module.candidates)\n    case ItemCandidateWithDetails(_: CursorCandidate, _, _) => 0\n    case _ => 1\n  }.sum\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-decider\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/async_feature_map_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_decorator_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_hydrator_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_source_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/filter_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/group_results_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/query_feature_hydrator_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n        \"util/util-core:scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-decider\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/async_feature_map_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_decorator_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_hydrator_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_source_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/filter_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/group_results_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/query_feature_hydrator_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate/CandidatePipeline.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.candidate\n\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.Pipeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Arrow\n\n/**\n * A Candidate Pipeline\n *\n * This is an abstract class, as we only construct these via the [[CandidatePipelineBuilder]].\n *\n * A [[CandidatePipeline]] is capable of processing requests (queries) and returning candidates\n * in the form of a [[CandidatePipelineResult]]\n *\n * @tparam Query the domain model for the query or request\n */\nabstract class CandidatePipeline[-Query <: PipelineQuery] private[candidate]\n    extends Pipeline[CandidatePipeline.Inputs[Query], Seq[CandidateWithDetails]] {\n  override private[core] val config: BaseCandidatePipelineConfig[Query, _, _, _]\n  override val arrow: Arrow[CandidatePipeline.Inputs[Query], CandidatePipelineResult]\n  override val identifier: CandidatePipelineIdentifier\n}\n\nobject CandidatePipeline {\n  case class Inputs[+Query <: PipelineQuery](\n    query: Query,\n    existingCandidates: Seq[CandidateWithDetails])\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate/CandidatePipelineBuilder.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.candidate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.asyncfeaturemap.AsyncFeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseQueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.transformer.BaseCandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.gate.ParamGate\nimport com.twitter.product_mixer.core.gate.ParamGate.EnabledGateSuffix\nimport com.twitter.product_mixer.core.gate.ParamGate.SupportedClientGateSuffix\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.InvalidStepStateException\nimport com.twitter.product_mixer.core.pipeline.PipelineBuilder\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ClosedGate\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureClassifier\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutor\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutorResults\nimport com.twitter.product_mixer.core.service.candidate_decorator_executor.CandidateDecoratorExecutor\nimport com.twitter.product_mixer.core.service.candidate_decorator_executor.CandidateDecoratorExecutorResult\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutorResult\nimport com.twitter.product_mixer.core.service.candidate_source_executor.CandidateSourceExecutor\nimport com.twitter.product_mixer.core.service.candidate_source_executor.CandidateSourceExecutorResult\nimport com.twitter.product_mixer.core.service.candidate_source_executor.FetchedCandidateWithFeatures\nimport com.twitter.product_mixer.core.service.filter_executor.FilterExecutor\nimport com.twitter.product_mixer.core.service.filter_executor.FilterExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.StoppedGateException\nimport com.twitter.product_mixer.core.service.group_results_executor.GroupResultsExecutor\nimport com.twitter.product_mixer.core.service.group_results_executor.GroupResultsExecutorInput\nimport com.twitter.product_mixer.core.service.group_results_executor.GroupResultsExecutorResult\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor\nimport com.twitter.stitch.Arrow\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\n\nclass CandidatePipelineBuilder[\n  Query <: PipelineQuery,\n  CandidateSourceQuery,\n  CandidateSourceResult,\n  Result <: UniversalNoun[Any]] @Inject() (\n  queryFeatureHydratorExecutor: QueryFeatureHydratorExecutor,\n  asyncFeatureMapExecutor: AsyncFeatureMapExecutor,\n  candidateDecoratorExecutor: CandidateDecoratorExecutor,\n  candidateFeatureHydratorExecutor: CandidateFeatureHydratorExecutor,\n  candidateSourceExecutor: CandidateSourceExecutor,\n  groupResultsExecutor: GroupResultsExecutor,\n  filterExecutor: FilterExecutor,\n  gateExecutor: GateExecutor,\n  override val statsReceiver: StatsReceiver)\n    extends PipelineBuilder[CandidatePipeline.Inputs[Query]]\n    with Logging {\n\n  override type UnderlyingResultType = Seq[CandidateWithDetails]\n  override type PipelineResultType = IntermediateCandidatePipelineResult[Result]\n\n  def build(\n    parentComponentIdentifierStack: ComponentIdentifierStack,\n    config: BaseCandidatePipelineConfig[\n      Query,\n      CandidateSourceQuery,\n      CandidateSourceResult,\n      Result\n    ]\n  ): CandidatePipeline[Query] = {\n\n    val pipelineIdentifier = config.identifier\n    val candidateSourceIdentifier = config.candidateSource.identifier\n\n    val context = Executor.Context(\n      PipelineFailureClassifier(\n        config.failureClassifier.orElse(StoppedGateException.classifier(ClosedGate))),\n      parentComponentIdentifierStack.push(pipelineIdentifier)\n    )\n\n    val enabledGateOpt = config.enabledDeciderParam.map { deciderParam =>\n      ParamGate(pipelineIdentifier + EnabledGateSuffix, deciderParam)\n    }\n    val supportedClientGateOpt = config.supportedClientParam.map { param =>\n      ParamGate(pipelineIdentifier + SupportedClientGateSuffix, param)\n    }\n\n    /**\n     * Evaluate enabled decider gate first since if it's off, there is no reason to proceed\n     * Next evaluate supported client feature switch gate, followed by customer configured gates\n     */\n    val allGates = enabledGateOpt.toSeq ++ supportedClientGateOpt.toSeq ++ config.gates\n\n    // Dynamically replace the identifier of both transformers if config used the inline constructor\n    // which sets a default identifier. We need to do this to ensure uniqueness of identifiers.\n    val queryTransformer = BaseCandidatePipelineQueryTransformer.copyWithUpdatedIdentifier(\n      config.queryTransformer,\n      pipelineIdentifier)\n\n    val resultsTransformer = CandidatePipelineResultsTransformer.copyWithUpdatedIdentifier(\n      config.resultTransformer,\n      pipelineIdentifier)\n\n    val decorator = config.decorator.map(decorator =>\n      CandidateDecorator.copyWithUpdatedIdentifier(decorator, pipelineIdentifier))\n\n    val GatesStep = new Step[Query, GateExecutorResult] {\n      override def identifier: PipelineStepIdentifier = CandidatePipelineConfig.gatesStep\n\n      override def executorArrow: Arrow[Query, GateExecutorResult] = {\n        gateExecutor.arrow(allGates, context)\n      }\n\n      override def inputAdaptor(\n        query: CandidatePipeline.Inputs[Query],\n        previousResult: IntermediateCandidatePipelineResult[Result]\n      ): Query =\n        query.query\n\n      override def resultUpdater(\n        previousPipelineResult: IntermediateCandidatePipelineResult[Result],\n        executorResult: GateExecutorResult\n      ): IntermediateCandidatePipelineResult[Result] =\n        previousPipelineResult.copy(underlyingResult =\n          previousPipelineResult.underlyingResult.copy(gateResult = Some(executorResult)))\n    }\n\n    def queryFeatureHydrationStep(\n      queryFeatureHydrators: Seq[BaseQueryFeatureHydrator[Query, _]],\n      stepIdentifier: PipelineStepIdentifier,\n      updater: ResultUpdater[CandidatePipelineResult, QueryFeatureHydratorExecutor.Result]\n    ): Step[Query, QueryFeatureHydratorExecutor.Result] =\n      new Step[Query, QueryFeatureHydratorExecutor.Result] {\n        override def identifier: PipelineStepIdentifier = stepIdentifier\n\n        override def executorArrow: Arrow[Query, QueryFeatureHydratorExecutor.Result] =\n          queryFeatureHydratorExecutor.arrow(\n            queryFeatureHydrators,\n            CandidatePipelineConfig.stepsAsyncFeatureHydrationCanBeCompletedBy,\n            context)\n\n        override def inputAdaptor(\n          query: CandidatePipeline.Inputs[Query],\n          previousResult: IntermediateCandidatePipelineResult[Result]\n        ): Query = query.query\n\n        override def resultUpdater(\n          previousPipelineResult: IntermediateCandidatePipelineResult[Result],\n          executorResult: QueryFeatureHydratorExecutor.Result\n        ): IntermediateCandidatePipelineResult[Result] =\n          previousPipelineResult.copy(\n            underlyingResult = updater(previousPipelineResult.underlyingResult, executorResult))\n\n        override def queryUpdater(\n          query: CandidatePipeline.Inputs[Query],\n          executorResult: QueryFeatureHydratorExecutor.Result\n        ): CandidatePipeline.Inputs[Query] =\n          CandidatePipeline.Inputs(\n            query.query\n              .withFeatureMap(\n                query.query.features.getOrElse(\n                  FeatureMap.empty) ++ executorResult.featureMap).asInstanceOf[Query],\n            query.existingCandidates)\n      }\n\n    def asyncFeaturesStep(\n      stepToHydrateFor: PipelineStepIdentifier,\n      context: Executor.Context\n    ): Step[AsyncFeatureMap, AsyncFeatureMapExecutorResults] =\n      new Step[AsyncFeatureMap, AsyncFeatureMapExecutorResults] {\n        override def identifier: PipelineStepIdentifier =\n          CandidatePipelineConfig.asyncFeaturesStep(stepToHydrateFor)\n\n        override def executorArrow: Arrow[AsyncFeatureMap, AsyncFeatureMapExecutorResults] =\n          asyncFeatureMapExecutor.arrow(stepToHydrateFor, identifier, context)\n\n        override def inputAdaptor(\n          query: CandidatePipeline.Inputs[Query],\n          previousResult: IntermediateCandidatePipelineResult[Result]\n        ): AsyncFeatureMap =\n          previousResult.underlyingResult.mergedAsyncQueryFeatures\n            .getOrElse(\n              throw InvalidStepStateException(identifier, \"MergedAsyncQueryFeatures\")\n            )\n\n        override def resultUpdater(\n          previousPipelineResult: IntermediateCandidatePipelineResult[Result],\n          executorResult: AsyncFeatureMapExecutorResults\n        ): IntermediateCandidatePipelineResult[Result] =\n          previousPipelineResult.copy(\n            underlyingResult =\n              previousPipelineResult.underlyingResult.copy(asyncFeatureHydrationResults =\n                previousPipelineResult.underlyingResult.asyncFeatureHydrationResults match {\n                  case Some(existingResults) => Some(existingResults ++ executorResult)\n                  case None => Some(executorResult)\n                }))\n\n        override def queryUpdater(\n          query: CandidatePipeline.Inputs[Query],\n          executorResult: AsyncFeatureMapExecutorResults\n        ): CandidatePipeline.Inputs[Query] =\n          if (executorResult.featureMapsByStep\n              .getOrElse(stepToHydrateFor, FeatureMap.empty).isEmpty) {\n            query\n          } else {\n            val updatedQuery = query.query\n              .withFeatureMap(\n                query.query.features\n                  .getOrElse(FeatureMap.empty) ++ executorResult.featureMapsByStep(\n                  stepToHydrateFor)).asInstanceOf[Query]\n            CandidatePipeline.Inputs(updatedQuery, query.existingCandidates)\n          }\n      }\n\n    val CandidateSourceStep =\n      new Step[Query, CandidateSourceExecutorResult[Result]] {\n        override def identifier: PipelineStepIdentifier =\n          CandidatePipelineConfig.candidateSourceStep\n\n        override def executorArrow: Arrow[\n          Query,\n          CandidateSourceExecutorResult[Result]\n        ] =\n          candidateSourceExecutor\n            .arrow(\n              config.candidateSource,\n              queryTransformer,\n              resultsTransformer,\n              config.featuresFromCandidateSourceTransformers,\n              context\n            )\n\n        override def inputAdaptor(\n          query: CandidatePipeline.Inputs[Query],\n          previousResult: IntermediateCandidatePipelineResult[Result]\n        ): Query =\n          query.query\n\n        override def resultUpdater(\n          previousPipelineResult: IntermediateCandidatePipelineResult[Result],\n          executorResult: CandidateSourceExecutorResult[Result]\n        ): IntermediateCandidatePipelineResult[Result] =\n          previousPipelineResult.copy(underlyingResult =\n            previousPipelineResult.underlyingResult.copy(\n              candidateSourceResult =\n                Some(executorResult.asInstanceOf[CandidateSourceExecutorResult[UniversalNoun[Any]]])\n            ))\n\n        override def queryUpdater(\n          query: CandidatePipeline.Inputs[Query],\n          executorResult: CandidateSourceExecutorResult[Result]\n        ): CandidatePipeline.Inputs[Query] = {\n          val updatedFeatureMap =\n            query.query.features\n              .getOrElse(FeatureMap.empty) ++ executorResult.candidateSourceFeatureMap\n          val updatedQuery = query.query\n            .withFeatureMap(updatedFeatureMap).asInstanceOf[Query]\n          CandidatePipeline.Inputs(updatedQuery, query.existingCandidates)\n        }\n      }\n\n    val PreFilterFeatureHydrationPhase1Step =\n      new Step[\n        CandidateFeatureHydratorExecutor.Inputs[Query, Result],\n        CandidateFeatureHydratorExecutorResult[Result]\n      ] {\n        override def identifier: PipelineStepIdentifier =\n          CandidatePipelineConfig.preFilterFeatureHydrationPhase1Step\n\n        override def executorArrow: Arrow[\n          CandidateFeatureHydratorExecutor.Inputs[Query, Result],\n          CandidateFeatureHydratorExecutorResult[Result]\n        ] =\n          candidateFeatureHydratorExecutor.arrow(config.preFilterFeatureHydrationPhase1, context)\n\n        override def inputAdaptor(\n          query: CandidatePipeline.Inputs[Query],\n          previousResult: IntermediateCandidatePipelineResult[Result]\n        ): CandidateFeatureHydratorExecutor.Inputs[Query, Result] = {\n          val candidateSourceExecutorResult =\n            previousResult.underlyingResult.candidateSourceResult.getOrElse {\n              throw InvalidStepStateException(identifier, \"CandidateSourceResult\")\n            }\n          CandidateFeatureHydratorExecutor.Inputs(\n            query.query,\n            candidateSourceExecutorResult.candidates\n              .asInstanceOf[Seq[CandidateWithFeatures[Result]]])\n        }\n\n        override def resultUpdater(\n          previousPipelineResult: IntermediateCandidatePipelineResult[Result],\n          executorResult: CandidateFeatureHydratorExecutorResult[Result]\n        ): IntermediateCandidatePipelineResult[Result] = {\n          val candidateSourceExecutorResult =\n            previousPipelineResult.underlyingResult.candidateSourceResult.getOrElse {\n              throw InvalidStepStateException(identifier, \"CandidateSourceResult\")\n            }\n\n          val featureMapsFromPreFilter = executorResult.results.map { result =>\n            result.candidate -> result.features\n          }.toMap\n\n          val mergedFeatureMaps = candidateSourceExecutorResult.candidates.map { candidate =>\n            val candidateFeatureMap = candidate.features\n            val preFilterFeatureMap =\n              featureMapsFromPreFilter.getOrElse(\n                candidate.candidate.asInstanceOf[Result],\n                FeatureMap.empty)\n\n            candidate.candidate.asInstanceOf[Result] -> (candidateFeatureMap ++ preFilterFeatureMap)\n          }.toMap\n\n          previousPipelineResult.copy(\n            underlyingResult = previousPipelineResult.underlyingResult.copy(\n              preFilterHydrationResult = Some(\n                executorResult\n                  .asInstanceOf[CandidateFeatureHydratorExecutorResult[UniversalNoun[Any]]])\n            ),\n            featureMaps = Some(mergedFeatureMaps)\n          )\n        }\n      }\n\n    val PreFilterFeatureHydrationPhase2Step =\n      new Step[\n        CandidateFeatureHydratorExecutor.Inputs[Query, Result],\n        CandidateFeatureHydratorExecutorResult[Result]\n      ] {\n        override def identifier: PipelineStepIdentifier =\n          CandidatePipelineConfig.preFilterFeatureHydrationPhase2Step\n\n        override def executorArrow: Arrow[\n          CandidateFeatureHydratorExecutor.Inputs[Query, Result],\n          CandidateFeatureHydratorExecutorResult[Result]\n        ] =\n          candidateFeatureHydratorExecutor.arrow(config.preFilterFeatureHydrationPhase2, context)\n\n        override def inputAdaptor(\n          query: CandidatePipeline.Inputs[Query],\n          previousResult: IntermediateCandidatePipelineResult[Result]\n        ): CandidateFeatureHydratorExecutor.Inputs[Query, Result] = {\n          val candidates = previousResult.underlyingResult.preFilterHydrationResult.getOrElse {\n            throw InvalidStepStateException(identifier, \"PreFilterHydrationResult\")\n          }.results\n          CandidateFeatureHydratorExecutor.Inputs(\n            query.query,\n            candidates.asInstanceOf[Seq[CandidateWithFeatures[Result]]]\n          )\n        }\n\n        override def resultUpdater(\n          previousPipelineResult: IntermediateCandidatePipelineResult[Result],\n          executorResult: CandidateFeatureHydratorExecutorResult[Result]\n        ): IntermediateCandidatePipelineResult[Result] = {\n\n          val featureMapsFromPreFilterPhase2 = executorResult.results.map { result =>\n            result.candidate -> result.features\n          }.toMap\n\n          val mergedFeatureMaps = previousPipelineResult.featureMaps\n            .getOrElse(throw InvalidStepStateException(identifier, \"FeatureMaps\"))\n            .map {\n              case (candidate, featureMap) =>\n                val preFilterPhase2FeatureMap =\n                  featureMapsFromPreFilterPhase2.getOrElse(candidate, FeatureMap.empty)\n\n                candidate -> (featureMap ++ preFilterPhase2FeatureMap)\n            }\n\n          previousPipelineResult.copy(\n            underlyingResult = previousPipelineResult.underlyingResult.copy(\n              preFilterHydrationResultPhase2 = Some(\n                executorResult\n                  .asInstanceOf[CandidateFeatureHydratorExecutorResult[UniversalNoun[Any]]])\n            ),\n            featureMaps = Some(mergedFeatureMaps)\n          )\n        }\n      }\n\n    val FiltersStep =\n      new Step[(Query, Seq[CandidateWithFeatures[Result]]), FilterExecutorResult[Result]] {\n        override def identifier: PipelineStepIdentifier = CandidatePipelineConfig.filtersStep\n\n        override def executorArrow: Arrow[\n          (Query, Seq[CandidateWithFeatures[Result]]),\n          FilterExecutorResult[\n            Result\n          ]\n        ] =\n          filterExecutor.arrow(config.filters, context)\n\n        override def inputAdaptor(\n          query: CandidatePipeline.Inputs[Query],\n          previousResult: IntermediateCandidatePipelineResult[Result]\n        ): (Query, Seq[CandidateWithFeatures[Result]]) = {\n          val candidates =\n            previousResult.underlyingResult.candidateSourceResult\n              .getOrElse {\n                throw InvalidStepStateException(identifier, \"CandidateSourceResult\")\n              }.candidates.map(_.candidate).asInstanceOf[Seq[Result]]\n\n          val featureMaps = previousResult.featureMaps\n            .getOrElse(throw InvalidStepStateException(identifier, \"FeatureMaps\"))\n\n          (\n            query.query,\n            candidates.map(candidate =>\n              CandidateWithFeaturesImpl(\n                candidate,\n                featureMaps.getOrElse(candidate, FeatureMap.empty))))\n        }\n\n        override def resultUpdater(\n          previousPipelineResult: IntermediateCandidatePipelineResult[Result],\n          executorResult: FilterExecutorResult[Result]\n        ): IntermediateCandidatePipelineResult[Result] =\n          previousPipelineResult.copy(underlyingResult =\n            previousPipelineResult.underlyingResult.copy(\n              filterResult =\n                Some(executorResult.asInstanceOf[FilterExecutorResult[UniversalNoun[Any]]])\n            ))\n      }\n\n    val PostFilterFeatureHydrationStep =\n      new Step[\n        CandidateFeatureHydratorExecutor.Inputs[Query, Result],\n        CandidateFeatureHydratorExecutorResult[Result]\n      ] {\n        override def identifier: PipelineStepIdentifier =\n          CandidatePipelineConfig.postFilterFeatureHydrationStep\n\n        override def executorArrow: Arrow[\n          CandidateFeatureHydratorExecutor.Inputs[Query, Result],\n          CandidateFeatureHydratorExecutorResult[Result]\n        ] =\n          candidateFeatureHydratorExecutor.arrow(config.postFilterFeatureHydration, context)\n\n        override def inputAdaptor(\n          query: CandidatePipeline.Inputs[Query],\n          previousResult: IntermediateCandidatePipelineResult[Result]\n        ): CandidateFeatureHydratorExecutor.Inputs[Query, Result] = {\n          val filterResult = previousResult.underlyingResult.filterResult\n            .getOrElse(\n              throw InvalidStepStateException(identifier, \"FilterResult\")\n            ).result.asInstanceOf[Seq[Result]]\n\n          val featureMaps = previousResult.featureMaps.getOrElse(\n            throw InvalidStepStateException(identifier, \"FeatureMaps\")\n          )\n\n          val filteredCandidates = filterResult.map { candidate =>\n            CandidateWithFeaturesImpl(candidate, featureMaps.getOrElse(candidate, FeatureMap.empty))\n          }\n          CandidateFeatureHydratorExecutor.Inputs(query.query, filteredCandidates)\n        }\n\n        override def resultUpdater(\n          previousPipelineResult: IntermediateCandidatePipelineResult[Result],\n          executorResult: CandidateFeatureHydratorExecutorResult[Result]\n        ): IntermediateCandidatePipelineResult[Result] = {\n          val filterResult = previousPipelineResult.underlyingResult.filterResult\n            .getOrElse(\n              throw InvalidStepStateException(identifier, \"FilterResult\")\n            ).result.asInstanceOf[Seq[Result]]\n\n          val featureMaps = previousPipelineResult.featureMaps.getOrElse(\n            throw InvalidStepStateException(identifier, \"FeatureMaps\")\n          )\n\n          val postFilterFeatureMaps = executorResult.results.map { result =>\n            result.candidate -> result.features\n          }.toMap\n\n          val mergedFeatureMaps = filterResult.map { candidate =>\n            candidate ->\n              (featureMaps\n                .getOrElse(candidate, FeatureMap.empty) ++ postFilterFeatureMaps.getOrElse(\n                candidate,\n                FeatureMap.empty))\n          }.toMap\n\n          previousPipelineResult.copy(\n            underlyingResult = previousPipelineResult.underlyingResult.copy(\n              postFilterHydrationResult = Some(\n                executorResult\n                  .asInstanceOf[CandidateFeatureHydratorExecutorResult[UniversalNoun[Any]]])\n            ),\n            featureMaps = Some(mergedFeatureMaps)\n          )\n        }\n      }\n\n    val ScorersStep =\n      new Step[\n        CandidateFeatureHydratorExecutor.Inputs[Query, Result],\n        CandidateFeatureHydratorExecutorResult[Result]\n      ] {\n        override def identifier: PipelineStepIdentifier = CandidatePipelineConfig.scorersStep\n\n        override def executorArrow: Arrow[\n          CandidateFeatureHydratorExecutor.Inputs[Query, Result],\n          CandidateFeatureHydratorExecutorResult[Result]\n        ] =\n          candidateFeatureHydratorExecutor.arrow(config.scorers, context)\n\n        override def inputAdaptor(\n          query: CandidatePipeline.Inputs[Query],\n          previousResult: IntermediateCandidatePipelineResult[Result]\n        ): CandidateFeatureHydratorExecutor.Inputs[Query, Result] = {\n          val filterResult = previousResult.underlyingResult.filterResult\n            .getOrElse(\n              throw InvalidStepStateException(identifier, \"FilterResult\")\n            ).result.asInstanceOf[Seq[Result]]\n\n          val featureMaps = previousResult.featureMaps.getOrElse(\n            throw InvalidStepStateException(identifier, \"FeatureMaps\")\n          )\n\n          val filteredCandidates = filterResult.map { candidate =>\n            CandidateWithFeaturesImpl(candidate, featureMaps.getOrElse(candidate, FeatureMap.empty))\n          }\n          CandidateFeatureHydratorExecutor.Inputs(query.query, filteredCandidates)\n        }\n\n        override def resultUpdater(\n          previousPipelineResult: IntermediateCandidatePipelineResult[Result],\n          executorResult: CandidateFeatureHydratorExecutorResult[Result]\n        ): IntermediateCandidatePipelineResult[Result] = {\n          val filterResult = previousPipelineResult.underlyingResult.filterResult\n            .getOrElse(\n              throw InvalidStepStateException(identifier, \"FilterResult\")\n            ).result.asInstanceOf[Seq[Result]]\n\n          val featureMaps = previousPipelineResult.featureMaps.getOrElse(\n            throw InvalidStepStateException(identifier, \"FeatureMaps\")\n          )\n\n          val scoringFeatureMaps = executorResult.results.map { result =>\n            result.candidate -> result.features\n          }.toMap\n\n          val mergedFeatureMaps = filterResult.map { candidate =>\n            candidate ->\n              (featureMaps\n                .getOrElse(candidate, FeatureMap.empty) ++ scoringFeatureMaps.getOrElse(\n                candidate,\n                FeatureMap.empty))\n          }.toMap\n\n          previousPipelineResult.copy(\n            underlyingResult = previousPipelineResult.underlyingResult.copy(\n              scorersResult = Some(\n                executorResult\n                  .asInstanceOf[CandidateFeatureHydratorExecutorResult[UniversalNoun[Any]]])\n            ),\n            featureMaps = Some(mergedFeatureMaps)\n          )\n        }\n      }\n\n    val DecorationStep =\n      new Step[(Query, Seq[CandidateWithFeatures[Result]]), CandidateDecoratorExecutorResult] {\n        override def identifier: PipelineStepIdentifier = CandidatePipelineConfig.decoratorStep\n\n        override def executorArrow: Arrow[\n          (Query, Seq[CandidateWithFeatures[Result]]),\n          CandidateDecoratorExecutorResult\n        ] =\n          candidateDecoratorExecutor.arrow(decorator, context)\n\n        override def inputAdaptor(\n          query: CandidatePipeline.Inputs[Query],\n          previousResult: IntermediateCandidatePipelineResult[Result]\n        ): (Query, Seq[CandidateWithFeatures[Result]]) = {\n          val keptCandidates = previousResult.underlyingResult.filterResult\n            .getOrElse {\n              throw InvalidStepStateException(identifier, \"FilterResult\")\n            }.result.asInstanceOf[Seq[Result]]\n\n          val featureMaps = previousResult.featureMaps.getOrElse {\n            throw InvalidStepStateException(identifier, \"FeatureMaps\")\n          }\n\n          (\n            query.query,\n            keptCandidates.map(candidate =>\n              CandidateWithFeaturesImpl(\n                candidate,\n                featureMaps.getOrElse(candidate, FeatureMap.empty))))\n        }\n\n        override def resultUpdater(\n          previousPipelineResult: IntermediateCandidatePipelineResult[Result],\n          executorResult: CandidateDecoratorExecutorResult\n        ): IntermediateCandidatePipelineResult[Result] =\n          previousPipelineResult.copy(underlyingResult =\n            previousPipelineResult.underlyingResult.copy(\n              candidateDecoratorResult = Some(executorResult)\n            ))\n      }\n\n    /**\n     * ResultStep is a synchronous step that basically takes the outputs from the other steps, groups modules,\n     * and puts things into the final result object\n     */\n    val ResultStep = new Step[GroupResultsExecutorInput[Result], GroupResultsExecutorResult] {\n      override def identifier: PipelineStepIdentifier = CandidatePipelineConfig.resultStep\n\n      override def executorArrow: Arrow[\n        GroupResultsExecutorInput[Result],\n        GroupResultsExecutorResult\n      ] = groupResultsExecutor.arrow(pipelineIdentifier, candidateSourceIdentifier, context)\n\n      override def inputAdaptor(\n        query: CandidatePipeline.Inputs[Query],\n        previousResult: IntermediateCandidatePipelineResult[Result]\n      ): GroupResultsExecutorInput[Result] = {\n\n        val underlying = previousResult.underlyingResult\n\n        val keptCandidates = underlying.filterResult\n          .getOrElse(\n            throw InvalidStepStateException(identifier, \"FilterResult\")\n          ).result.asInstanceOf[Seq[Result]]\n\n        val decorations = underlying.candidateDecoratorResult\n          .getOrElse(\n            throw InvalidStepStateException(identifier, \"DecorationResult\")\n          ).result.map(decoration => decoration.candidate -> decoration.presentation).toMap\n\n        val combinedFeatureMaps: Map[Result, FeatureMap] = previousResult.featureMaps.getOrElse(\n          throw InvalidStepStateException(identifier, \"FeatureMaps\"))\n\n        val filteredCandidates = keptCandidates.map { candidate =>\n          val updatedMap = combinedFeatureMaps\n            .get(candidate).getOrElse(FeatureMap.empty)\n          FetchedCandidateWithFeatures(candidate, updatedMap)\n        }\n\n        GroupResultsExecutorInput(\n          candidates = filteredCandidates,\n          decorations = decorations\n        )\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: IntermediateCandidatePipelineResult[Result],\n        executorResult: GroupResultsExecutorResult\n      ): IntermediateCandidatePipelineResult[Result] =\n        previousPipelineResult.copy(underlyingResult = previousPipelineResult.underlyingResult\n          .copy(result = Some(executorResult.candidatesWithDetails)))\n    }\n\n    val builtSteps = Seq(\n      GatesStep,\n      queryFeatureHydrationStep(\n        config.queryFeatureHydration,\n        CandidatePipelineConfig.fetchQueryFeaturesStep,\n        (pipelineResult, executorResult) =>\n          pipelineResult.copy(queryFeatures = Some(executorResult))\n      ),\n      queryFeatureHydrationStep(\n        config.queryFeatureHydrationPhase2,\n        CandidatePipelineConfig.fetchQueryFeaturesPhase2Step,\n        (pipelineResult, executorResult) =>\n          pipelineResult.copy(\n            queryFeaturesPhase2 = Some(executorResult),\n            mergedAsyncQueryFeatures = Some(\n              pipelineResult.queryFeatures\n                .getOrElse(\n                  throw InvalidStepStateException(\n                    CandidatePipelineConfig.fetchQueryFeaturesPhase2Step,\n                    \"QueryFeatures\")\n                ).asyncFeatureMap ++ executorResult.asyncFeatureMap)\n          )\n      ),\n      asyncFeaturesStep(CandidatePipelineConfig.candidateSourceStep, context),\n      CandidateSourceStep,\n      asyncFeaturesStep(CandidatePipelineConfig.preFilterFeatureHydrationPhase1Step, context),\n      PreFilterFeatureHydrationPhase1Step,\n      asyncFeaturesStep(CandidatePipelineConfig.preFilterFeatureHydrationPhase2Step, context),\n      PreFilterFeatureHydrationPhase2Step,\n      asyncFeaturesStep(CandidatePipelineConfig.filtersStep, context),\n      FiltersStep,\n      asyncFeaturesStep(CandidatePipelineConfig.postFilterFeatureHydrationStep, context),\n      PostFilterFeatureHydrationStep,\n      asyncFeaturesStep(CandidatePipelineConfig.scorersStep, context),\n      ScorersStep,\n      asyncFeaturesStep(CandidatePipelineConfig.decoratorStep, context),\n      DecorationStep,\n      ResultStep\n    )\n\n    /** The main execution logic for this Candidate Pipeline. */\n    val finalArrow: Arrow[CandidatePipeline.Inputs[Query], CandidatePipelineResult] =\n      buildCombinedArrowFromSteps(\n        steps = builtSteps,\n        context = context,\n        initialEmptyResult =\n          IntermediateCandidatePipelineResult.empty[Result](config.candidateSource.identifier),\n        stepsInOrderFromConfig = CandidatePipelineConfig.stepsInOrder\n      ).map(_.underlyingResult)\n\n    val configFromBuilder = config\n    new CandidatePipeline[Query] {\n      override private[core] val config: BaseCandidatePipelineConfig[Query, _, _, _] =\n        configFromBuilder\n      override val arrow: Arrow[CandidatePipeline.Inputs[Query], CandidatePipelineResult] =\n        finalArrow\n      override val identifier: CandidatePipelineIdentifier = pipelineIdentifier\n      override val alerts: Seq[Alert] = config.alerts\n      override val children: Seq[Component] =\n        allGates ++\n          config.queryFeatureHydration ++\n          Seq(queryTransformer, config.candidateSource, resultsTransformer) ++\n          config.featuresFromCandidateSourceTransformers ++\n          decorator.toSeq ++\n          config.preFilterFeatureHydrationPhase1 ++\n          config.filters ++\n          config.postFilterFeatureHydration ++\n          config.scorers\n    }\n  }\n\n  private case class CandidateWithFeaturesImpl(candidate: Result, features: FeatureMap)\n      extends CandidateWithFeatures[Result]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate/CandidatePipelineBuilderFactory.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.candidate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutor\nimport com.twitter.product_mixer.core.service.candidate_decorator_executor.CandidateDecoratorExecutor\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.candidate_source_executor.CandidateSourceExecutor\nimport com.twitter.product_mixer.core.service.filter_executor.FilterExecutor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutor\nimport com.twitter.product_mixer.core.service.group_results_executor.GroupResultsExecutor\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CandidatePipelineBuilderFactory @Inject() (\n  queryFeatureHydratorExecutor: QueryFeatureHydratorExecutor,\n  asyncFeatureMapExecutor: AsyncFeatureMapExecutor,\n  candidateDecoratorExecutor: CandidateDecoratorExecutor,\n  candidateFeatureHydratorExecutor: CandidateFeatureHydratorExecutor,\n  candidateSourceExecutor: CandidateSourceExecutor,\n  groupResultsExecutor: GroupResultsExecutor,\n  filterExecutor: FilterExecutor,\n  gateExecutor: GateExecutor,\n  statsReceiver: StatsReceiver) {\n  def get[\n    Query <: PipelineQuery,\n    CandidateSourceQuery,\n    CandidateSourceResult,\n    Result <: UniversalNoun[Any]\n  ]: CandidatePipelineBuilder[\n    Query,\n    CandidateSourceQuery,\n    CandidateSourceResult,\n    Result\n  ] = {\n    new CandidatePipelineBuilder[\n      Query,\n      CandidateSourceQuery,\n      CandidateSourceResult,\n      Result\n    ](\n      queryFeatureHydratorExecutor,\n      asyncFeatureMapExecutor,\n      candidateDecoratorExecutor,\n      candidateFeatureHydratorExecutor,\n      candidateSourceExecutor,\n      groupResultsExecutor,\n      filterExecutor,\n      gateExecutor,\n      statsReceiver\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate/CandidatePipelineConfig.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.candidate\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseQueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer._\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineConfig\nimport com.twitter.product_mixer.core.pipeline.PipelineConfigCompanion\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.decider.DeciderParam\n\nsealed trait BaseCandidatePipelineConfig[\n  -Query <: PipelineQuery,\n  CandidateSourceQuery,\n  CandidateSourceResult,\n  Result <: UniversalNoun[Any]]\n    extends PipelineConfig {\n\n  val identifier: CandidatePipelineIdentifier\n\n  /**\n   * A candidate pipeline can fetch query-level features for use within the candidate source. It's\n   * generally recommended to set a hydrator in the parent recos or mixer pipeline if multiple\n   * candidate pipelines share the same feature but if a specific query feature hydrator is used\n   * by one pipeline and you don't want to block the others, you could explicitly set it here.\n   * If a feature is hydrated both in the parent pipeline or here, this one takes priority.\n   */\n  def queryFeatureHydration: Seq[BaseQueryFeatureHydrator[Query, _]] = Seq.empty\n\n  /**\n   * For query-level features that are dependent on query-level features from [[queryFeatureHydration]]\n   */\n  def queryFeatureHydrationPhase2: Seq[BaseQueryFeatureHydrator[Query, _]] = Seq.empty\n\n  /**\n   * When these Params are defined, they will automatically be added as Gates in the pipeline\n   * by the CandidatePipelineBuilder\n   *\n   * The enabled decider param can to be used to quickly disable a Candidate Pipeline via Decider\n   */\n  val enabledDeciderParam: Option[DeciderParam[Boolean]] = None\n\n  /**\n   * This supported client feature switch param can be used with a Feature Switch to control the\n   * rollout of a new Candidate Pipeline from dogfood to experiment to production\n   */\n  val supportedClientParam: Option[FSParam[Boolean]] = None\n\n  /** [[Gate]]s that are applied sequentially, the pipeline will only run if all the Gates are open */\n  def gates: Seq[BaseGate[Query]] = Seq.empty\n\n  /**\n   * A pair of transforms to adapt the underlying candidate source to the pipeline's query and result types\n   * Complex use cases such as those that need access to features should construct their own transformer, but\n   * for simple use cases, you can pass in an anonymous function.\n   * @example\n   * {{{ override val queryTransformer: CandidatePipelineQueryTransformer[Query, CandidateSourceQuery] = { query =>\n   *   query.toExampleThrift\n   *  }\n   * }}}\n   */\n  def queryTransformer: BaseCandidatePipelineQueryTransformer[\n    Query,\n    CandidateSourceQuery\n  ]\n\n  /** Source for Candidates for this Pipeline */\n  def candidateSource: BaseCandidateSource[CandidateSourceQuery, CandidateSourceResult]\n\n  /**\n   * [[CandidateFeatureTransformer]] allow you to define [[com.twitter.product_mixer.core.feature.Feature]] extraction logic from your [[CandidateSource]] results.\n   * If your candidate sources return [[com.twitter.product_mixer.core.feature.Feature]]s alongside the candidate that might be useful later on,\n   * add transformers for constructing feature maps.\n   *\n   * @note If multiple transformers extract the same feature, the last one takes priority and is kept.\n   */\n  def featuresFromCandidateSourceTransformers: Seq[\n    CandidateFeatureTransformer[CandidateSourceResult]\n  ] = Seq.empty\n\n  /**\n   * a result Transformer may throw PipelineFailure for candidates that are malformed and\n   * should be removed. This should be exceptional behavior, and not a replacement for adding a Filter.\n   * Complex use cases such as those that need access to features should construct their own transformer, but\n   * for simple use cases, you can pass in an anonymous function.\n   * @example\n   * {{{ override val queryTransformer: CandidatePipelineResultsTransformer[CandidateSourceResult, Result] = { sourceResult =>\n   *   ExampleCandidate(sourceResult.id)\n   *  }\n   * }}}\n   *\n   */\n  val resultTransformer: CandidatePipelineResultsTransformer[CandidateSourceResult, Result]\n\n  /**\n   * Before filters are run, you can fetch features for each candidate.\n   *\n   * Uses Stitch, so you're encouraged to use a working Stitch Adaptor to batch between candidates.\n   *\n   * The existing features (from the candidate source) are passed in as an input. You are not expected\n   * to put them into the resulting feature map yourself - they will be merged for you by the platform.\n   *\n   * This API is likely to change when Product Mixer does managed feature hydration\n   */\n  val preFilterFeatureHydrationPhase1: Seq[BaseCandidateFeatureHydrator[Query, Result, _]] =\n    Seq.empty\n\n  /**\n   * A second phase of feature hydration that can be run before filtering and after the first phase\n   * of [[preFilterFeatureHydrationPhase1]]. You are not expected to put them into the resulting\n   * feature map yourself - they will be merged for you by the platform.\n   */\n  val preFilterFeatureHydrationPhase2: Seq[BaseCandidateFeatureHydrator[Query, Result, _]] =\n    Seq.empty\n\n  /** A list of filters to apply. Filters will be applied in sequential order. */\n  def filters: Seq[Filter[Query, Result]] = Seq.empty\n\n  /**\n   * After filters are run, you can fetch features for each candidate.\n   *\n   * Uses Stitch, so you're encouraged to use a working Stitch Adaptor to batch between candidates.\n   *\n   * The existing features (from the candidate source) & pre-filtering are passed in as an input.\n   * You are not expected to put them into the resulting feature map yourself -\n   * they will be merged for you by the platform.\n   *\n   * This API is likely to change when Product Mixer does managed feature hydration\n   */\n  val postFilterFeatureHydration: Seq[BaseCandidateFeatureHydrator[Query, Result, _]] = Seq.empty\n\n  /**\n   * Decorators allow for adding Presentations to candidates. While the Presentation can contain any\n   * arbitrary data, Decorators are often used to add a UrtItemPresentation for URT item support, or\n   * a UrtModulePresentation for grouping the candidates in a URT module.\n   */\n  val decorator: Option[CandidateDecorator[Query, Result]] = None\n\n  /**\n   * A candidate pipeline can define a partial function to rescue failures here. They will be treated as failures\n   * from a monitoring standpoint, and cancellation exceptions will always be propagated (they cannot be caught here).\n   */\n  def failureClassifier: PartialFunction[Throwable, PipelineFailure] = PartialFunction.empty\n\n  /**\n   * Scorers for candidates. Scorers are executed in parallel. Order does not matter.\n   */\n  def scorers: Seq[Scorer[Query, Result]] = Seq.empty\n\n  /**\n   * Alerts can be used to indicate the pipeline's service level objectives. Alerts and\n   * dashboards will be automatically created based on this information.\n   */\n  val alerts: Seq[Alert] = Seq.empty\n\n  /**\n   * This method is used by the product mixer framework to build the pipeline.\n   */\n  private[core] final def build(\n    parentComponentIdentifierStack: ComponentIdentifierStack,\n    factory: CandidatePipelineBuilderFactory\n  ): CandidatePipeline[Query] = {\n    factory.get.build(parentComponentIdentifierStack, this)\n  }\n}\n\ntrait CandidatePipelineConfig[\n  -Query <: PipelineQuery,\n  CandidateSourceQuery,\n  CandidateSourceResult,\n  Result <: UniversalNoun[Any]]\n    extends BaseCandidatePipelineConfig[\n      Query,\n      CandidateSourceQuery,\n      CandidateSourceResult,\n      Result\n    ] {\n  override val gates: Seq[Gate[Query]] = Seq.empty\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[\n    Query,\n    CandidateSourceQuery\n  ]\n}\n\ntrait DependentCandidatePipelineConfig[\n  -Query <: PipelineQuery,\n  CandidateSourceQuery,\n  CandidateSourceResult,\n  Result <: UniversalNoun[Any]]\n    extends BaseCandidatePipelineConfig[\n      Query,\n      CandidateSourceQuery,\n      CandidateSourceResult,\n      Result\n    ]\n\n/**\n * Contains [[PipelineStepIdentifier]]s for the Steps that are available for all [[BaseCandidatePipelineConfig]]s\n */\nobject CandidatePipelineConfig extends PipelineConfigCompanion {\n  val gatesStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Gates\")\n  val fetchQueryFeaturesStep: PipelineStepIdentifier = PipelineStepIdentifier(\"FetchQueryFeatures\")\n  val fetchQueryFeaturesPhase2Step: PipelineStepIdentifier = PipelineStepIdentifier(\n    \"FetchQueryFeaturesPhase2\")\n  val candidateSourceStep: PipelineStepIdentifier = PipelineStepIdentifier(\"CandidateSource\")\n  val preFilterFeatureHydrationPhase1Step: PipelineStepIdentifier =\n    PipelineStepIdentifier(\"PreFilterFeatureHydration\")\n  val preFilterFeatureHydrationPhase2Step: PipelineStepIdentifier =\n    PipelineStepIdentifier(\"PreFilterFeatureHydrationPhase2\")\n  val filtersStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Filters\")\n  val postFilterFeatureHydrationStep: PipelineStepIdentifier =\n    PipelineStepIdentifier(\"PostFilterFeatureHydration\")\n  val scorersStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Scorer\")\n  val decoratorStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Decorator\")\n  val resultStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Result\")\n\n  /** All the steps which are executed by a [[CandidatePipeline]] in the order in which they are run */\n  override val stepsInOrder: Seq[PipelineStepIdentifier] = Seq(\n    gatesStep,\n    fetchQueryFeaturesStep,\n    fetchQueryFeaturesPhase2Step,\n    asyncFeaturesStep(candidateSourceStep),\n    candidateSourceStep,\n    asyncFeaturesStep(preFilterFeatureHydrationPhase1Step),\n    preFilterFeatureHydrationPhase1Step,\n    asyncFeaturesStep(preFilterFeatureHydrationPhase2Step),\n    preFilterFeatureHydrationPhase2Step,\n    asyncFeaturesStep(filtersStep),\n    filtersStep,\n    asyncFeaturesStep(postFilterFeatureHydrationStep),\n    postFilterFeatureHydrationStep,\n    asyncFeaturesStep(scorersStep),\n    scorersStep,\n    asyncFeaturesStep(decoratorStep),\n    decoratorStep,\n    resultStep\n  )\n\n  override val stepsAsyncFeatureHydrationCanBeCompletedBy: Set[PipelineStepIdentifier] = Set(\n    candidateSourceStep,\n    preFilterFeatureHydrationPhase1Step,\n    preFilterFeatureHydrationPhase2Step,\n    filtersStep,\n    postFilterFeatureHydrationStep,\n    scorersStep,\n    decoratorStep\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate/CandidatePipelineResult.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.candidate\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.asyncfeaturemap.AsyncFeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineResult\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutorResults\nimport com.twitter.product_mixer.core.service.candidate_decorator_executor.CandidateDecoratorExecutorResult\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutorResult\nimport com.twitter.product_mixer.core.service.candidate_source_executor.CandidateSourceExecutorResult\nimport com.twitter.product_mixer.core.service.filter_executor.FilterExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutorResult\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor\n\ncase class CandidatePipelineResult(\n  candidateSourceIdentifier: CandidateSourceIdentifier,\n  gateResult: Option[GateExecutorResult],\n  queryFeatures: Option[QueryFeatureHydratorExecutor.Result],\n  queryFeaturesPhase2: Option[QueryFeatureHydratorExecutor.Result],\n  mergedAsyncQueryFeatures: Option[AsyncFeatureMap],\n  candidateSourceResult: Option[CandidateSourceExecutorResult[UniversalNoun[Any]]],\n  preFilterHydrationResult: Option[CandidateFeatureHydratorExecutorResult[UniversalNoun[Any]]],\n  preFilterHydrationResultPhase2: Option[\n    CandidateFeatureHydratorExecutorResult[UniversalNoun[Any]]\n  ],\n  filterResult: Option[FilterExecutorResult[UniversalNoun[Any]]],\n  postFilterHydrationResult: Option[CandidateFeatureHydratorExecutorResult[UniversalNoun[Any]]],\n  candidateDecoratorResult: Option[CandidateDecoratorExecutorResult],\n  scorersResult: Option[CandidateFeatureHydratorExecutorResult[UniversalNoun[Any]]],\n  asyncFeatureHydrationResults: Option[AsyncFeatureMapExecutorResults],\n  failure: Option[PipelineFailure],\n  result: Option[Seq[CandidateWithDetails]])\n    extends PipelineResult[Seq[CandidateWithDetails]] {\n\n  override def withFailure(failure: PipelineFailure): CandidatePipelineResult =\n    copy(failure = Some(failure))\n\n  override def withResult(\n    result: Seq[CandidateWithDetails]\n  ): CandidatePipelineResult = copy(result = Some(result))\n\n  override val resultSize: Int = result.map(PipelineResult.resultSize).getOrElse(0)\n}\n\nprivate[candidate] object IntermediateCandidatePipelineResult {\n  def empty[Candidate <: UniversalNoun[Any]](\n    candidateSourceIdentifier: CandidateSourceIdentifier\n  ): IntermediateCandidatePipelineResult[Candidate] = {\n    IntermediateCandidatePipelineResult(\n      CandidatePipelineResult(\n        candidateSourceIdentifier = candidateSourceIdentifier,\n        None,\n        None,\n        None,\n        None,\n        None,\n        None,\n        None,\n        None,\n        None,\n        None,\n        None,\n        None,\n        None,\n        None\n      ),\n      None\n    )\n  }\n}\n\nprivate[candidate] case class IntermediateCandidatePipelineResult[Candidate <: UniversalNoun[Any]](\n  underlyingResult: CandidatePipelineResult,\n  featureMaps: Option[Map[Candidate, FeatureMap]])\n    extends PipelineResult[Seq[CandidateWithDetails]] {\n  override val failure: Option[PipelineFailure] = underlyingResult.failure\n  override val result: Option[Seq[CandidateWithDetails]] = underlyingResult.result\n\n  override def withFailure(\n    failure: PipelineFailure\n  ): IntermediateCandidatePipelineResult[Candidate] =\n    copy(underlyingResult = underlyingResult.withFailure(failure))\n\n  override def withResult(\n    result: Seq[CandidateWithDetails]\n  ): IntermediateCandidatePipelineResult[Candidate] =\n    copy(underlyingResult = underlyingResult.withResult(result))\n\n  override def resultSize(): Int = underlyingResult.resultSize\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate/PassthroughCandidatePipelineConfig.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.candidate\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.PassthroughCandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateExtractor\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\nobject PassthroughCandidatePipelineConfig {\n\n  /**\n   * Build a [[PassthroughCandidatePipelineConfig]] with a [[PassthroughCandidateSource]] built from\n   * a [[CandidateExtractor]]\n   */\n  def apply[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n    identifier: CandidatePipelineIdentifier,\n    extractor: CandidateExtractor[Query, Candidate],\n    decorator: Option[CandidateDecorator[Query, Candidate]] = None\n  ): PassthroughCandidatePipelineConfig[Query, Candidate] = {\n\n    // Renaming variables to keep the interface clean, but avoid naming collisions when creating\n    // the anonymous class.\n    val _identifier = identifier\n    val _extractor = extractor\n    val _decorator = decorator\n\n    new PassthroughCandidatePipelineConfig[Query, Candidate] {\n      override val identifier = _identifier\n      override val candidateSource =\n        PassthroughCandidateSource(CandidateSourceIdentifier(_identifier.name), _extractor)\n      override val decorator = _decorator\n    }\n  }\n}\n\ntrait PassthroughCandidatePipelineConfig[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]]\n    extends CandidatePipelineConfig[Query, Query, Candidate, Candidate] {\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[Query, Query] = identity\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[Candidate, Candidate] =\n    identity\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate/StaticCandidatePipelineConfig.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.candidate\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.StaticCandidateSource\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\n\nobject StaticCandidatePipelineConfig {\n\n  /**\n   * Build a [[StaticCandidatePipelineConfig]] with a [[CandidateSource]] that returns the [[candidate]]\n   */\n  def apply[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n    identifier: CandidatePipelineIdentifier,\n    candidate: Candidate,\n    decorator: Option[CandidateDecorator[Query, Candidate]] = None\n  ): StaticCandidatePipelineConfig[Query, Candidate] = {\n\n    // Renaming variables to keep the interface clean, but avoid naming collisions when creating\n    // the anonymous class.\n    val _identifier = identifier\n    val _candidate = candidate\n    val _decorator = decorator\n\n    new StaticCandidatePipelineConfig[Query, Candidate] {\n      override val identifier = _identifier\n      override val candidate = _candidate\n      override val decorator = _decorator\n    }\n  }\n}\n\ntrait StaticCandidatePipelineConfig[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]]\n    extends CandidatePipelineConfig[Query, Unit, Unit, Candidate] {\n\n  val candidate: Candidate\n\n  override def candidateSource: CandidateSource[Unit, Unit] = StaticCandidateSource[Unit](\n    identifier = CandidateSourceIdentifier(identifier.name),\n    result = Seq(()))\n\n  override val queryTransformer: CandidatePipelineQueryTransformer[Query, Unit] = _ => Unit\n\n  override val resultTransformer: CandidatePipelineResultsTransformer[Unit, Candidate] = _ =>\n    candidate\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/async_feature_map_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_pipeline_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/domain_marshaller_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_result_side_effect_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/quality_factor_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/query_feature_hydrator_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/selector_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transport_marshaller_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_pipeline_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/domain_marshaller_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_result_side_effect_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/quality_factor_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/query_feature_hydrator_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/selector_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transport_marshaller_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer/MixerPipeline.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.mixer\n\nimport com.twitter.product_mixer.core.model.common.identifier.MixerPipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.Pipeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Arrow\n\n/**\n * A Mixer Pipeline\n *\n * This is an abstract class, as we only construct these via the [[MixerPipelineBuilder]].\n *\n * A [[MixerPipeline]] is capable of processing requests (queries) and returning responses (results)\n * in the correct format to directly send to users.\n *\n * @tparam Query the domain model for the query or request\n * @tparam Result the final marshalled result type\n */\nabstract class MixerPipeline[Query <: PipelineQuery, Result] private[mixer]\n    extends Pipeline[Query, Result] {\n  override private[core] val config: MixerPipelineConfig[Query, _, Result]\n  override val arrow: Arrow[Query, MixerPipelineResult[Result]]\n  override val identifier: MixerPipelineIdentifier\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer/MixerPipelineBuilder.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.mixer\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.asyncfeaturemap.AsyncFeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.common.identifier.MixerPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.InvalidStepStateException\nimport com.twitter.product_mixer.core.pipeline.PipelineBuilder\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipeline\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineBuilderFactory\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureClassifier\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ProductDisabled\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorObserver\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorStatus\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutor\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutorResults\nimport com.twitter.product_mixer.core.service.candidate_pipeline_executor.CandidatePipelineExecutor\nimport com.twitter.product_mixer.core.service.candidate_pipeline_executor.CandidatePipelineExecutorResult\nimport com.twitter.product_mixer.core.service.domain_marshaller_executor.DomainMarshallerExecutor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.StoppedGateException\nimport com.twitter.product_mixer.core.service.pipeline_result_side_effect_executor.PipelineResultSideEffectExecutor\nimport com.twitter.product_mixer.core.service.quality_factor_executor.QualityFactorExecutorResult\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutor\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutorResult\nimport com.twitter.product_mixer.core.service.transport_marshaller_executor.TransportMarshallerExecutor\nimport com.twitter.stitch.Arrow\nimport com.twitter.util.logging.Logging\n\n/**\n * MixerPipelineBuilder builds [[MixerPipeline]]s from [[MixerPipelineConfig]]s.\n *\n * You should inject a [[MixerPipelineBuilderFactory]] and call `.get` to build these.\n *\n * @see [[MixerPipelineConfig]] for the description of the type parameters\n */\nclass MixerPipelineBuilder[Query <: PipelineQuery, DomainResultType <: HasMarshalling, Result](\n  candidatePipelineExecutor: CandidatePipelineExecutor,\n  gateExecutor: GateExecutor,\n  selectorExecutor: SelectorExecutor,\n  queryFeatureHydratorExecutor: QueryFeatureHydratorExecutor,\n  asyncFeatureMapExecutor: AsyncFeatureMapExecutor,\n  domainMarshallerExecutor: DomainMarshallerExecutor,\n  transportMarshallerExecutor: TransportMarshallerExecutor,\n  pipelineResultSideEffectExecutor: PipelineResultSideEffectExecutor,\n  candidatePipelineBuilderFactory: CandidatePipelineBuilderFactory,\n  override val statsReceiver: StatsReceiver)\n    extends PipelineBuilder[Query]\n    with Logging {\n\n  override type UnderlyingResultType = Result\n  override type PipelineResultType = MixerPipelineResult[Result]\n\n  def qualityFactorStep(\n    qualityFactorStatus: QualityFactorStatus\n  ): Step[Query, QualityFactorExecutorResult] =\n    new Step[Query, QualityFactorExecutorResult] {\n      override def identifier: PipelineStepIdentifier = MixerPipelineConfig.qualityFactorStep\n\n      override def executorArrow: Arrow[Query, QualityFactorExecutorResult] =\n        Arrow\n          .map[Query, QualityFactorExecutorResult] { _ =>\n            QualityFactorExecutorResult(\n              pipelineQualityFactors =\n                qualityFactorStatus.qualityFactorByPipeline.mapValues(_.currentValue)\n            )\n          }\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: MixerPipelineResult[Result]\n      ): Query = query\n\n      override def resultUpdater(\n        previousPipelineResult: MixerPipelineResult[Result],\n        executorResult: QualityFactorExecutorResult\n      ): MixerPipelineResult[Result] =\n        previousPipelineResult.copy(qualityFactorResult = Some(executorResult))\n\n      override def queryUpdater(\n        query: Query,\n        executorResult: QualityFactorExecutorResult\n      ): Query = {\n        query match {\n          case queryWithQualityFactor: HasQualityFactorStatus =>\n            queryWithQualityFactor\n              .withQualityFactorStatus(\n                queryWithQualityFactor.qualityFactorStatus.getOrElse(QualityFactorStatus.empty) ++\n                  qualityFactorStatus\n              ).asInstanceOf[Query]\n          case _ =>\n            query\n        }\n      }\n    }\n\n  def gatesStep(\n    gates: Seq[Gate[Query]],\n    context: Executor.Context\n  ): Step[Query, GateExecutorResult] = new Step[Query, GateExecutorResult] {\n    override def identifier: PipelineStepIdentifier = MixerPipelineConfig.gatesStep\n\n    override def executorArrow: Arrow[Query, GateExecutorResult] =\n      gateExecutor.arrow(gates, context)\n\n    override def inputAdaptor(query: Query, previousResult: MixerPipelineResult[Result]): Query =\n      query\n\n    override def resultUpdater(\n      previousPipelineResult: MixerPipelineResult[Result],\n      executorResult: GateExecutorResult\n    ): MixerPipelineResult[Result] =\n      previousPipelineResult.copy(gateResult = Some(executorResult))\n  }\n\n  def fetchQueryFeaturesStep(\n    queryFeatureHydrators: Seq[QueryFeatureHydrator[Query]],\n    stepIdentifier: PipelineStepIdentifier,\n    updater: ResultUpdater[MixerPipelineResult[Result], QueryFeatureHydratorExecutor.Result],\n    context: Executor.Context\n  ): Step[Query, QueryFeatureHydratorExecutor.Result] =\n    new Step[Query, QueryFeatureHydratorExecutor.Result] {\n      override def identifier: PipelineStepIdentifier = stepIdentifier\n\n      override def executorArrow: Arrow[Query, QueryFeatureHydratorExecutor.Result] =\n        queryFeatureHydratorExecutor.arrow(\n          queryFeatureHydrators,\n          MixerPipelineConfig.stepsAsyncFeatureHydrationCanBeCompletedBy,\n          context)\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: MixerPipelineResult[Result]\n      ): Query = query\n\n      override def resultUpdater(\n        previousPipelineResult: MixerPipelineResult[Result],\n        executorResult: QueryFeatureHydratorExecutor.Result\n      ): MixerPipelineResult[Result] =\n        updater(previousPipelineResult, executorResult)\n\n      override def queryUpdater(\n        query: Query,\n        executorResult: QueryFeatureHydratorExecutor.Result\n      ): Query =\n        query\n          .withFeatureMap(\n            query.features\n              .getOrElse(FeatureMap.empty) ++ executorResult.featureMap).asInstanceOf[Query]\n    }\n\n  def asyncFeaturesStep(\n    stepToHydrateFor: PipelineStepIdentifier,\n    context: Executor.Context\n  ): Step[AsyncFeatureMap, AsyncFeatureMapExecutorResults] =\n    new Step[AsyncFeatureMap, AsyncFeatureMapExecutorResults] {\n      override def identifier: PipelineStepIdentifier =\n        MixerPipelineConfig.asyncFeaturesStep(stepToHydrateFor)\n\n      override def executorArrow: Arrow[AsyncFeatureMap, AsyncFeatureMapExecutorResults] =\n        asyncFeatureMapExecutor.arrow(\n          stepToHydrateFor,\n          identifier,\n          context\n        )\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: MixerPipelineResult[Result]\n      ): AsyncFeatureMap =\n        previousResult.mergedAsyncQueryFeatures\n          .getOrElse(\n            throw InvalidStepStateException(identifier, \"MergedAsyncQueryFeatures\")\n          )\n\n      override def resultUpdater(\n        previousPipelineResult: MixerPipelineResult[Result],\n        executorResult: AsyncFeatureMapExecutorResults\n      ): MixerPipelineResult[Result] = previousPipelineResult.copy(\n        asyncFeatureHydrationResults = previousPipelineResult.asyncFeatureHydrationResults match {\n          case Some(existingResults) => Some(existingResults ++ executorResult)\n          case None => Some(executorResult)\n        })\n\n      override def queryUpdater(\n        query: Query,\n        executorResult: AsyncFeatureMapExecutorResults\n      ): Query =\n        if (executorResult.featureMapsByStep\n            .getOrElse(stepToHydrateFor, FeatureMap.empty).isEmpty) {\n          query\n        } else {\n          query\n            .withFeatureMap(\n              query.features\n                .getOrElse(FeatureMap.empty) ++ executorResult.featureMapsByStep(\n                stepToHydrateFor)).asInstanceOf[Query]\n        }\n    }\n\n  def candidatePipelinesStep(\n    candidatePipelines: Seq[CandidatePipeline[Query]],\n    defaultFailOpenPolicy: FailOpenPolicy,\n    failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy],\n    qualityFactorObserverByPipeline: Map[ComponentIdentifier, QualityFactorObserver],\n    context: Executor.Context\n  ): Step[CandidatePipeline.Inputs[Query], CandidatePipelineExecutorResult] =\n    new Step[CandidatePipeline.Inputs[Query], CandidatePipelineExecutorResult] {\n      override def identifier: PipelineStepIdentifier = MixerPipelineConfig.candidatePipelinesStep\n\n      override def executorArrow: Arrow[CandidatePipeline.Inputs[\n        Query\n      ], CandidatePipelineExecutorResult] =\n        candidatePipelineExecutor\n          .arrow(\n            candidatePipelines,\n            defaultFailOpenPolicy,\n            failOpenPolicies,\n            qualityFactorObserverByPipeline,\n            context\n          )\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: MixerPipelineResult[Result]\n      ): CandidatePipeline.Inputs[Query] = CandidatePipeline.Inputs[Query](query, Seq.empty)\n\n      override def resultUpdater(\n        previousPipelineResult: MixerPipelineResult[Result],\n        executorResult: CandidatePipelineExecutorResult\n      ): MixerPipelineResult[Result] =\n        previousPipelineResult.copy(candidatePipelineResults = Some(executorResult))\n\n      override def queryUpdater(\n        query: Query,\n        executorResult: CandidatePipelineExecutorResult\n      ): Query = {\n        val updatedFeatureMap = query.features\n          .getOrElse(FeatureMap.empty) ++ executorResult.queryFeatureMap\n        query\n          .withFeatureMap(updatedFeatureMap).asInstanceOf[Query]\n      }\n    }\n\n  def dependentCandidatePipelinesStep(\n    candidatePipelines: Seq[CandidatePipeline[Query]],\n    defaultFailOpenPolicy: FailOpenPolicy,\n    failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy],\n    qualityFactorObserverByPipeline: Map[ComponentIdentifier, QualityFactorObserver],\n    context: Executor.Context\n  ): Step[CandidatePipeline.Inputs[Query], CandidatePipelineExecutorResult] =\n    new Step[CandidatePipeline.Inputs[Query], CandidatePipelineExecutorResult] {\n      override def identifier: PipelineStepIdentifier =\n        MixerPipelineConfig.dependentCandidatePipelinesStep\n\n      override def executorArrow: Arrow[CandidatePipeline.Inputs[\n        Query\n      ], CandidatePipelineExecutorResult] =\n        candidatePipelineExecutor\n          .arrow(\n            candidatePipelines,\n            defaultFailOpenPolicy,\n            failOpenPolicies,\n            qualityFactorObserverByPipeline,\n            context\n          )\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: MixerPipelineResult[Result]\n      ): CandidatePipeline.Inputs[Query] = {\n        val previousCandidates = previousResult.candidatePipelineResults\n          .getOrElse {\n            throw InvalidStepStateException(identifier, \"Candidates\")\n          }.candidatePipelineResults.flatMap(_.result.getOrElse(Seq.empty))\n        CandidatePipeline.Inputs[Query](query, previousCandidates)\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: MixerPipelineResult[Result],\n        executorResult: CandidatePipelineExecutorResult\n      ): MixerPipelineResult[Result] =\n        previousPipelineResult.copy(dependentCandidatePipelineResults = Some(executorResult))\n\n      override def queryUpdater(\n        query: Query,\n        executorResult: CandidatePipelineExecutorResult\n      ): Query = {\n        val updatedFeatureMap = query.features\n          .getOrElse(FeatureMap.empty) ++ executorResult.queryFeatureMap\n        query\n          .withFeatureMap(updatedFeatureMap).asInstanceOf[Query]\n      }\n    }\n\n  def resultSelectorsStep(\n    selectors: Seq[Selector[Query]],\n    context: Executor.Context\n  ): Step[SelectorExecutor.Inputs[Query], SelectorExecutorResult] =\n    new Step[SelectorExecutor.Inputs[Query], SelectorExecutorResult] {\n      override def identifier: PipelineStepIdentifier = MixerPipelineConfig.resultSelectorsStep\n\n      override def executorArrow: Arrow[SelectorExecutor.Inputs[Query], SelectorExecutorResult] =\n        selectorExecutor.arrow(selectors, context)\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: MixerPipelineResult[Result]\n      ): SelectorExecutor.Inputs[Query] = {\n        val candidates = previousResult.candidatePipelineResults\n          .getOrElse {\n            throw InvalidStepStateException(identifier, \"Candidates\")\n          }.candidatePipelineResults.flatMap(_.result.getOrElse(Seq.empty))\n\n        val dependentCandidates =\n          previousResult.dependentCandidatePipelineResults\n            .getOrElse {\n              throw InvalidStepStateException(identifier, \"DependentCandidates\")\n            }.candidatePipelineResults.flatMap(_.result.getOrElse(Seq.empty))\n\n        SelectorExecutor.Inputs(\n          query = query,\n          candidatesWithDetails = candidates ++ dependentCandidates\n        )\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: MixerPipelineResult[Result],\n        executorResult: SelectorExecutorResult\n      ): MixerPipelineResult[Result] =\n        previousPipelineResult.copy(resultSelectorResults = Some(executorResult))\n    }\n\n  def domainMarshallingStep(\n    domainMarshaller: DomainMarshaller[Query, DomainResultType],\n    context: Executor.Context\n  ): Step[DomainMarshallerExecutor.Inputs[Query], DomainMarshallerExecutor.Result[\n    DomainResultType\n  ]] =\n    new Step[DomainMarshallerExecutor.Inputs[Query], DomainMarshallerExecutor.Result[\n      DomainResultType\n    ]] {\n      override def identifier: PipelineStepIdentifier = MixerPipelineConfig.domainMarshallerStep\n\n      override def executorArrow: Arrow[\n        DomainMarshallerExecutor.Inputs[Query],\n        DomainMarshallerExecutor.Result[DomainResultType]\n      ] =\n        domainMarshallerExecutor.arrow(domainMarshaller, context)\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: MixerPipelineResult[Result]\n      ): DomainMarshallerExecutor.Inputs[Query] = {\n        val selectorResults = previousResult.resultSelectorResults.getOrElse {\n          throw InvalidStepStateException(identifier, \"SelectorResults\")\n        }\n\n        DomainMarshallerExecutor.Inputs(\n          query = query,\n          candidatesWithDetails = selectorResults.selectedCandidates\n        )\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: MixerPipelineResult[Result],\n        executorResult: DomainMarshallerExecutor.Result[DomainResultType]\n      ): MixerPipelineResult[Result] = previousPipelineResult.copy(\n        domainMarshallerResults = Some(executorResult)\n      )\n    }\n\n  def resultSideEffectsStep(\n    sideEffects: Seq[PipelineResultSideEffect[Query, DomainResultType]],\n    context: Executor.Context\n  ): Step[\n    PipelineResultSideEffect.Inputs[Query, DomainResultType],\n    PipelineResultSideEffectExecutor.Result\n  ] = new Step[\n    PipelineResultSideEffect.Inputs[Query, DomainResultType],\n    PipelineResultSideEffectExecutor.Result\n  ] {\n    override def identifier: PipelineStepIdentifier = MixerPipelineConfig.resultSideEffectsStep\n\n    override def executorArrow: Arrow[\n      PipelineResultSideEffect.Inputs[Query, DomainResultType],\n      PipelineResultSideEffectExecutor.Result\n    ] = pipelineResultSideEffectExecutor.arrow(sideEffects, context)\n\n    override def inputAdaptor(\n      query: Query,\n      previousResult: MixerPipelineResult[Result]\n    ): PipelineResultSideEffect.Inputs[Query, DomainResultType] = {\n\n      val selectorResults = previousResult.resultSelectorResults.getOrElse {\n        throw InvalidStepStateException(identifier, \"SelectorResults\")\n      }\n\n      val domainMarshallerResults = previousResult.domainMarshallerResults.getOrElse {\n        throw InvalidStepStateException(identifier, \"DomainMarshallerResults\")\n      }\n\n      PipelineResultSideEffect.Inputs[Query, DomainResultType](\n        query = query,\n        selectedCandidates = selectorResults.selectedCandidates,\n        remainingCandidates = selectorResults.remainingCandidates,\n        droppedCandidates = selectorResults.droppedCandidates,\n        response = domainMarshallerResults.result.asInstanceOf[DomainResultType]\n      )\n    }\n\n    override def resultUpdater(\n      previousPipelineResult: MixerPipelineResult[Result],\n      executorResult: PipelineResultSideEffectExecutor.Result\n    ): MixerPipelineResult[Result] =\n      previousPipelineResult.copy(resultSideEffectResults = Some(executorResult))\n  }\n\n  def transportMarshallingStep(\n    transportMarshaller: TransportMarshaller[DomainResultType, Result],\n    context: Executor.Context\n  ): Step[\n    TransportMarshallerExecutor.Inputs[DomainResultType],\n    TransportMarshallerExecutor.Result[Result]\n  ] = new Step[TransportMarshallerExecutor.Inputs[\n    DomainResultType\n  ], TransportMarshallerExecutor.Result[Result]] {\n    override def identifier: PipelineStepIdentifier = MixerPipelineConfig.transportMarshallerStep\n\n    override def executorArrow: Arrow[TransportMarshallerExecutor.Inputs[\n      DomainResultType\n    ], TransportMarshallerExecutor.Result[Result]] =\n      transportMarshallerExecutor.arrow(transportMarshaller, context)\n\n    override def inputAdaptor(\n      query: Query,\n      previousResult: MixerPipelineResult[Result]\n    ): TransportMarshallerExecutor.Inputs[DomainResultType] = {\n      val domainMarshallingResults = previousResult.domainMarshallerResults.getOrElse {\n        throw InvalidStepStateException(identifier, \"DomainMarshallerResults\")\n      }\n\n      // Since the PipelineResult just uses HasMarshalling\n      val domainResult = domainMarshallingResults.result.asInstanceOf[DomainResultType]\n\n      TransportMarshallerExecutor.Inputs(domainResult)\n    }\n\n    override def resultUpdater(\n      previousPipelineResult: MixerPipelineResult[Result],\n      executorResult: TransportMarshallerExecutor.Result[Result]\n    ): MixerPipelineResult[Result] = previousPipelineResult.copy(\n      transportMarshallerResults = Some(executorResult),\n      result = Some(executorResult.result)\n    )\n  }\n\n  def build(\n    parentComponentIdentifierStack: ComponentIdentifierStack,\n    config: MixerPipelineConfig[Query, DomainResultType, Result]\n  ): MixerPipeline[Query, Result] = {\n\n    val pipelineIdentifier = config.identifier\n\n    val context = Executor.Context(\n      PipelineFailureClassifier(\n        config.failureClassifier.orElse(StoppedGateException.classifier(ProductDisabled))),\n      parentComponentIdentifierStack.push(pipelineIdentifier)\n    )\n\n    val qualityFactorStatus: QualityFactorStatus =\n      QualityFactorStatus.build(config.qualityFactorConfigs)\n\n    val qualityFactorObserverByPipeline =\n      qualityFactorStatus.qualityFactorByPipeline.mapValues { qualityFactor =>\n        qualityFactor.buildObserver()\n      }\n\n    buildGaugesForQualityFactor(pipelineIdentifier, qualityFactorStatus, statsReceiver)\n\n    val candidatePipelines: Seq[CandidatePipeline[Query]] = config.candidatePipelines.map {\n      pipelineConfig: CandidatePipelineConfig[Query, _, _, _] =>\n        pipelineConfig.build(context.componentStack, candidatePipelineBuilderFactory)\n    }\n\n    val dependentCandidatePipelines: Seq[CandidatePipeline[Query]] =\n      config.dependentCandidatePipelines.map {\n        pipelineConfig: DependentCandidatePipelineConfig[Query, _, _, _] =>\n          pipelineConfig.build(context.componentStack, candidatePipelineBuilderFactory)\n      }\n\n    val builtSteps = Seq(\n      qualityFactorStep(qualityFactorStatus),\n      gatesStep(config.gates, context),\n      fetchQueryFeaturesStep(\n        config.fetchQueryFeatures,\n        MixerPipelineConfig.fetchQueryFeaturesStep,\n        (previousPipelineResult, executorResult) =>\n          previousPipelineResult.copy(queryFeatures = Some(executorResult)),\n        context\n      ),\n      fetchQueryFeaturesStep(\n        config.fetchQueryFeaturesPhase2,\n        MixerPipelineConfig.fetchQueryFeaturesPhase2Step,\n        (previousPipelineResult, executorResult) =>\n          previousPipelineResult.copy(\n            queryFeaturesPhase2 = Some(executorResult),\n            mergedAsyncQueryFeatures = Some(\n              previousPipelineResult.queryFeatures\n                .getOrElse(throw InvalidStepStateException(\n                  MixerPipelineConfig.fetchQueryFeaturesPhase2Step,\n                  \"QueryFeatures\"))\n                .asyncFeatureMap ++ executorResult.asyncFeatureMap)\n          ),\n        context\n      ),\n      asyncFeaturesStep(MixerPipelineConfig.candidatePipelinesStep, context),\n      candidatePipelinesStep(\n        candidatePipelines,\n        config.defaultFailOpenPolicy,\n        config.failOpenPolicies,\n        qualityFactorObserverByPipeline,\n        context),\n      asyncFeaturesStep(MixerPipelineConfig.dependentCandidatePipelinesStep, context),\n      dependentCandidatePipelinesStep(\n        dependentCandidatePipelines,\n        config.defaultFailOpenPolicy,\n        config.failOpenPolicies,\n        qualityFactorObserverByPipeline,\n        context),\n      asyncFeaturesStep(MixerPipelineConfig.resultSelectorsStep, context),\n      resultSelectorsStep(config.resultSelectors, context),\n      domainMarshallingStep(config.domainMarshaller, context),\n      asyncFeaturesStep(MixerPipelineConfig.resultSideEffectsStep, context),\n      resultSideEffectsStep(config.resultSideEffects, context),\n      transportMarshallingStep(config.transportMarshaller, context)\n    )\n\n    val finalArrow = buildCombinedArrowFromSteps(\n      steps = builtSteps,\n      context = context,\n      initialEmptyResult = MixerPipelineResult.empty,\n      stepsInOrderFromConfig = MixerPipelineConfig.stepsInOrder\n    )\n\n    val configFromBuilder = config\n    new MixerPipeline[Query, Result] {\n      override private[core] val config: MixerPipelineConfig[Query, _, Result] = configFromBuilder\n      override val arrow: Arrow[Query, MixerPipelineResult[Result]] = finalArrow\n      override val identifier: MixerPipelineIdentifier = pipelineIdentifier\n      override val alerts: Seq[Alert] = config.alerts\n      override val children: Seq[Component] =\n        config.gates ++\n          config.fetchQueryFeatures ++\n          candidatePipelines ++\n          dependentCandidatePipelines ++\n          config.resultSideEffects ++\n          Seq(config.domainMarshaller, config.transportMarshaller)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer/MixerPipelineBuilderFactory.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.mixer\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineBuilderFactory\nimport com.twitter.product_mixer.core.service.candidate_pipeline_executor.CandidatePipelineExecutor\nimport com.twitter.product_mixer.core.service.domain_marshaller_executor.DomainMarshallerExecutor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutor\nimport com.twitter.product_mixer.core.service.pipeline_result_side_effect_executor.PipelineResultSideEffectExecutor\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutor\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutor\nimport com.twitter.product_mixer.core.service.transport_marshaller_executor.TransportMarshallerExecutor\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass MixerPipelineBuilderFactory @Inject() (\n  candidatePipelineExecutor: CandidatePipelineExecutor,\n  gateExecutor: GateExecutor,\n  selectorExecutor: SelectorExecutor,\n  queryFeatureHydratorExecutor: QueryFeatureHydratorExecutor,\n  asyncFeatureMapExecutor: AsyncFeatureMapExecutor,\n  domainMarshallerExecutor: DomainMarshallerExecutor,\n  transportMarshallerExecutor: TransportMarshallerExecutor,\n  pipelineResultSideEffectExecutor: PipelineResultSideEffectExecutor,\n  candidatePipelineBuilderFactory: CandidatePipelineBuilderFactory,\n  statsReceiver: StatsReceiver) {\n  def get[\n    Query <: PipelineQuery,\n    DomainResultType <: HasMarshalling,\n    Result\n  ]: MixerPipelineBuilder[Query, DomainResultType, Result] = {\n    new MixerPipelineBuilder[Query, DomainResultType, Result](\n      candidatePipelineExecutor,\n      gateExecutor,\n      selectorExecutor,\n      queryFeatureHydratorExecutor,\n      asyncFeatureMapExecutor,\n      domainMarshallerExecutor,\n      transportMarshallerExecutor,\n      pipelineResultSideEffectExecutor,\n      candidatePipelineBuilderFactory,\n      statsReceiver\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer/MixerPipelineConfig.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.mixer\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.QueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.common.identifier.MixerPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.PipelineConfig\nimport com.twitter.product_mixer.core.pipeline.PipelineConfigCompanion\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ClosedGate\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorConfig\n\n/**\n *  This is the configuration necessary to generate a Mixer Pipeline. Product code should create a\n *  MixerPipelineConfig, and then use a MixerPipelineBuilder to get the final MixerPipeline which can\n *  process requests.\n *\n * @tparam Query - The domain model for the query or request\n * @tparam UnmarshalledResultType - The result type of the pipeline, but before marshalling to a wire protocol like URT\n * @tparam Result - The final result that will be served to users\n */\ntrait MixerPipelineConfig[Query <: PipelineQuery, UnmarshalledResultType <: HasMarshalling, Result]\n    extends PipelineConfig {\n\n  override val identifier: MixerPipelineIdentifier\n\n  /**\n   * Mixer Pipeline Gates will be executed before any other step (including retrieval from candidate\n   * pipelines). They're executed sequentially, and any \"Stop\" result will prevent pipeline execution.\n   */\n  def gates: Seq[Gate[Query]] = Seq.empty\n\n  /**\n   * A mixer pipeline can fetch query-level features before candidate pipelines are executed.\n   */\n  def fetchQueryFeatures: Seq[QueryFeatureHydrator[Query]] = Seq.empty\n\n  /**\n   * For query-level features that are dependent on query-level features from [[fetchQueryFeatures]]\n   */\n  def fetchQueryFeaturesPhase2: Seq[QueryFeatureHydrator[Query]] = Seq.empty\n\n  /**\n   * Candidate pipelines retrieve candidates for possible inclusion in the result\n   */\n  def candidatePipelines: Seq[CandidatePipelineConfig[Query, _, _, _]]\n\n  /**\n   * Dependent candidate pipelines to retrieve candidates that depend on the result of [[candidatePipelines]]\n   * [[DependentCandidatePipelineConfig]] have access to the list of previously retrieved & decorated\n   * candidates for use in constructing the query object.\n   */\n  def dependentCandidatePipelines: Seq[DependentCandidatePipelineConfig[Query, _, _, _]] = Seq.empty\n\n  /**\n   * [[defaultFailOpenPolicy]] is the [[FailOpenPolicy]] that will be applied to any candidate\n   * pipeline that isn't in the [[failOpenPolicies]] map. By default Candidate Pipelines will fail\n   * open for Closed Gates only.\n   */\n  def defaultFailOpenPolicy: FailOpenPolicy = FailOpenPolicy(Set(ClosedGate))\n\n  /**\n   * [[failOpenPolicies]] associates [[FailOpenPolicy]]s to specific candidate pipelines using\n   * [[CandidatePipelineIdentifier]].\n   *\n   * @note these [[FailOpenPolicy]]s override the [[defaultFailOpenPolicy]] for a mapped\n   *       Candidate Pipeline.\n   */\n  def failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy] = Map.empty\n\n  /**\n   ** [[qualityFactorConfigs]] associates [[QualityFactorConfig]]s to specific candidate pipelines\n   * using [[CandidatePipelineIdentifier]].\n   */\n  def qualityFactorConfigs: Map[CandidatePipelineIdentifier, QualityFactorConfig] =\n    Map.empty\n\n  /**\n   * Selectors are executed in sequential order to combine the candidates into a result\n   */\n  def resultSelectors: Seq[Selector[Query]]\n\n  /**\n   * Mixer result side effects that are executed after selection and domain marshalling\n   */\n  def resultSideEffects: Seq[PipelineResultSideEffect[Query, UnmarshalledResultType]] = Seq()\n\n  /**\n   * Domain marshaller transforms the selections into the model expected by the marshaller\n   */\n  def domainMarshaller: DomainMarshaller[Query, UnmarshalledResultType]\n\n  /**\n   * Transport marshaller transforms the model into our line-level API like URT or JSON\n   */\n  def transportMarshaller: TransportMarshaller[UnmarshalledResultType, Result]\n\n  /**\n   * A pipeline can define a partial function to rescue failures here. They will be treated as failures\n   * from a monitoring standpoint, and cancellation exceptions will always be propagated (they cannot be caught here).\n   */\n  def failureClassifier: PartialFunction[Throwable, PipelineFailure] = PartialFunction.empty\n\n  /**\n   * Alert can be used to indicate the pipeline's service level objectives. Alerts and\n   * dashboards will be automatically created based on this information.\n   */\n  val alerts: Seq[Alert] = Seq.empty\n\n  /**\n   * This method is used by the product mixer framework to build the pipeline.\n   */\n  private[core] final def build(\n    parentComponentIdentifierStack: ComponentIdentifierStack,\n    builder: MixerPipelineBuilderFactory\n  ): MixerPipeline[Query, Result] =\n    builder.get.build(parentComponentIdentifierStack, this)\n}\n\nobject MixerPipelineConfig extends PipelineConfigCompanion {\n  val qualityFactorStep: PipelineStepIdentifier = PipelineStepIdentifier(\"QualityFactor\")\n  val gatesStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Gates\")\n  val fetchQueryFeaturesStep: PipelineStepIdentifier = PipelineStepIdentifier(\"FetchQueryFeatures\")\n  val fetchQueryFeaturesPhase2Step: PipelineStepIdentifier =\n    PipelineStepIdentifier(\"FetchQueryFeaturesPhase2\")\n  val candidatePipelinesStep: PipelineStepIdentifier = PipelineStepIdentifier(\"CandidatePipelines\")\n  val dependentCandidatePipelinesStep: PipelineStepIdentifier =\n    PipelineStepIdentifier(\"DependentCandidatePipelines\")\n  val resultSelectorsStep: PipelineStepIdentifier = PipelineStepIdentifier(\"ResultSelectors\")\n  val domainMarshallerStep: PipelineStepIdentifier = PipelineStepIdentifier(\"DomainMarshaller\")\n  val resultSideEffectsStep: PipelineStepIdentifier = PipelineStepIdentifier(\"ResultSideEffects\")\n  val transportMarshallerStep: PipelineStepIdentifier = PipelineStepIdentifier(\n    \"TransportMarshaller\")\n\n  /** All the Steps which are executed by a [[MixerPipeline]] in the order in which they are run */\n  override val stepsInOrder: Seq[PipelineStepIdentifier] = Seq(\n    qualityFactorStep,\n    gatesStep,\n    fetchQueryFeaturesStep,\n    fetchQueryFeaturesPhase2Step,\n    asyncFeaturesStep(candidatePipelinesStep),\n    candidatePipelinesStep,\n    asyncFeaturesStep(dependentCandidatePipelinesStep),\n    dependentCandidatePipelinesStep,\n    asyncFeaturesStep(resultSelectorsStep),\n    resultSelectorsStep,\n    domainMarshallerStep,\n    asyncFeaturesStep(resultSideEffectsStep),\n    resultSideEffectsStep,\n    transportMarshallerStep\n  )\n\n  /**\n   * All the Steps which an [[com.twitter.product_mixer.core.functional_component.feature_hydrator.AsyncHydrator AsyncHydrator]]\n   * can be configured to [[com.twitter.product_mixer.core.functional_component.feature_hydrator.AsyncHydrator.hydrateBefore hydrateBefore]]\n   */\n  override val stepsAsyncFeatureHydrationCanBeCompletedBy: Set[PipelineStepIdentifier] = Set(\n    candidatePipelinesStep,\n    dependentCandidatePipelinesStep,\n    resultSelectorsStep,\n    resultSideEffectsStep\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer/MixerPipelineResult.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.mixer\n\nimport com.twitter.product_mixer.core.feature.featuremap.asyncfeaturemap.AsyncFeatureMap\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineResult\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutorResults\nimport com.twitter.product_mixer.core.service.candidate_pipeline_executor.CandidatePipelineExecutorResult\nimport com.twitter.product_mixer.core.service.domain_marshaller_executor.DomainMarshallerExecutor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutorResult\nimport com.twitter.product_mixer.core.service.pipeline_result_side_effect_executor.PipelineResultSideEffectExecutor\nimport com.twitter.product_mixer.core.service.quality_factor_executor.QualityFactorExecutorResult\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutorResult\nimport com.twitter.product_mixer.core.service.transport_marshaller_executor.TransportMarshallerExecutor\n\n/**\n * A [[MixerPipelineResult]] includes both the user-visible [[PipelineResult]] and all the\n * Execution details possible - intermediate results, what components did, etc.\n */\ncase class MixerPipelineResult[Result](\n  qualityFactorResult: Option[QualityFactorExecutorResult],\n  gateResult: Option[GateExecutorResult],\n  queryFeatures: Option[QueryFeatureHydratorExecutor.Result],\n  queryFeaturesPhase2: Option[QueryFeatureHydratorExecutor.Result],\n  mergedAsyncQueryFeatures: Option[AsyncFeatureMap],\n  candidatePipelineResults: Option[CandidatePipelineExecutorResult],\n  dependentCandidatePipelineResults: Option[CandidatePipelineExecutorResult],\n  resultSelectorResults: Option[SelectorExecutorResult],\n  domainMarshallerResults: Option[DomainMarshallerExecutor.Result[HasMarshalling]],\n  resultSideEffectResults: Option[PipelineResultSideEffectExecutor.Result],\n  asyncFeatureHydrationResults: Option[AsyncFeatureMapExecutorResults],\n  transportMarshallerResults: Option[TransportMarshallerExecutor.Result[Result]],\n  failure: Option[PipelineFailure],\n  result: Option[Result])\n    extends PipelineResult[Result] {\n\n  override def withFailure(failure: PipelineFailure): PipelineResult[Result] =\n    copy(failure = Some(failure))\n\n  override def withResult(result: Result): PipelineResult[Result] = copy(result = Some(result))\n\n  /**\n   * resultSize is calculated based on the selector results rather than the marshalled results. The\n   * structure of the marshalled format is unknown, making operating on selector results more\n   * convenient. This will implicitly excluded cursors built during marshalling but cursors don't\n   * contribute to the result size anyway.\n   */\n  override val resultSize: Int =\n    resultSelectorResults.map(_.selectedCandidates).map(PipelineResult.resultSize).getOrElse(0)\n}\n\nobject MixerPipelineResult {\n  def empty[A]: MixerPipelineResult[A] = MixerPipelineResult(\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"util/util-jackson/src/main/scala/com/twitter/util/jackson\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure/PipelineFailure.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.pipeline_failure\n\nimport com.fasterxml.jackson.databind.annotation.JsonSerialize\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport scala.util.control.NoStackTrace\n\n/**\n * Pipeline Failures represent pipeline requests that were not able to complete.\n *\n * A pipeline result will always define either a result or a failure.\n *\n * The reason field should not be displayed to end-users, and is free to change over time.\n * It should always be free of private user data such that we can log it.\n *\n * The pipeline can classify it's own failures into categories (timeouts, invalid arguments,\n * rate limited, etc) such that the caller can choose how to handle it.\n *\n * @note [[componentStack]] should only be set by the product mixer framework,\n *       it should **NOT** be set when making a [[PipelineFailure]]\n */\n@JsonSerialize(using = classOf[PipelineFailureSerializer])\ncase class PipelineFailure(\n  category: PipelineFailureCategory,\n  reason: String,\n  underlying: Option[Throwable] = None,\n  componentStack: Option[ComponentIdentifierStack] = None)\n    extends Exception(\n      \"PipelineFailure(\" +\n        s\"category = $category, \" +\n        s\"reason = $reason, \" +\n        s\"underlying = $underlying, \" +\n        s\"componentStack = $componentStack)\",\n      underlying.orNull\n    ) {\n  override def toString: String = getMessage\n\n  /** Returns an updated copy of this [[PipelineFailure]] with the same exception stacktrace */\n  def copy(\n    category: PipelineFailureCategory = this.category,\n    reason: String = this.reason,\n    underlying: Option[Throwable] = this.underlying,\n    componentStack: Option[ComponentIdentifierStack] = this.componentStack\n  ): PipelineFailure = {\n    val newPipelineFailure =\n      new PipelineFailure(category, reason, underlying, componentStack) with NoStackTrace\n    newPipelineFailure.setStackTrace(this.getStackTrace)\n    newPipelineFailure\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure/PipelineFailureCategory.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.pipeline_failure\n\n/**\n * Failures are grouped into categories based on which party is 'responsible' for the issue. This\n * is important for generating accurate SLOs and ensuring that the correct team is alerted.\n */\nsealed trait PipelineFailureCategory {\n  val categoryName: String\n  val failureName: String\n}\n\n/**\n * Client Failures are failures where the client is deemed responsible for the issue. Such as by\n * issuing an invalid request or not having the right permissions.\n *\n * A failure might belong in this category if it relates to behaviour on the client and is not\n * actionable by the team which owns the product.\n */\ntrait ClientFailure extends PipelineFailureCategory {\n  override val categoryName: String = \"ClientFailure\"\n}\n\n/**\n * The requested product is disabled so the request cannot be served.\n */\ncase object ProductDisabled extends ClientFailure {\n  override val failureName: String = \"ProductDisabled\"\n}\n\n/**\n * The request was deemed invalid by this or a backing service.\n */\ncase object BadRequest extends ClientFailure {\n  override val failureName: String = \"BadRequest\"\n}\n\n/**\n * Credentials proving the identity of the caller were missing, not trusted, or expired.\n * For example, an auth cookie might be expired and in need of refreshing.\n *\n * Do not confuse this with Authorization, where the credentials are believed but not allowed to perform the operation.\n */\ncase object Authentication extends ClientFailure {\n  override val failureName: String = \"Authentication\"\n}\n\n/**\n * The operation was forbidden (often, but not always, by a Strato access control policy).\n *\n * Do not confuse this with Authentication, where the given credentials were missing, not trusted, or expired.\n */\ncase object Unauthorized extends ClientFailure {\n  override val failureName: String = \"Unauthorized\"\n}\n\n/**\n * The operation returned a Not Found response.\n */\ncase object NotFound extends ClientFailure {\n  override val failureName: String = \"NotFound\"\n}\n\n/**\n * An invalid input is included in a cursor field.\n */\ncase object MalformedCursor extends ClientFailure {\n  override val failureName: String = \"MalformedCursor\"\n}\n\n/**\n * The operation did not succeed due to a closed gate\n */\ncase object ClosedGate extends ClientFailure {\n  override val failureName: String = \"ClosedGate\"\n}\n\n/**\n * Server Failures are failures for which the owner of the product is responsible. Typically this\n * means the request was valid but an issue within Product Mixer or a dependent service prevented\n * it from succeeding.\n *\n * Server Failures contribute to the success rate SLO for the product.\n */\ntrait ServerFailure extends PipelineFailureCategory {\n  override val categoryName: String = \"ServerFailure\"\n}\n\n/**\n * Unclassified failures occur when product code throws an exception that Product Mixer does not\n * know how to classify.\n *\n * They can be used in failOpen policies, etc - but it's always preferred to instead add additional\n * classification logic and to keep Unclassified failures at 0.\n */\ncase object UncategorizedServerFailure extends ServerFailure {\n  override val failureName: String = \"UncategorizedServerFailure\"\n}\n\n/**\n * A hydrator or transformer returned a misconfigured feature map, this indicates a customer\n * configuration error. The owner of the component should make sure the hydrator always returns a\n * [[FeatureMap]] with the all features defined in the hydrator also set in the map, it should not have\n * any unregistered features nor should registered features be absent.\n */\ncase object MisconfiguredFeatureMapFailure extends ServerFailure {\n  override val failureName: String = \"MisconfiguredFeatureMapFailure\"\n}\n\n/**\n * A PipelineSelector returned an invalid ComponentIdentifier.\n *\n * A pipeline selector should choose the identifier of a pipeline that is contained by the 'pipelines'\n * sequence of the ProductPipelineConfig.\n */\ncase object InvalidPipelineSelected extends ServerFailure {\n  override val failureName: String = \"InvalidPipelineSelected\"\n}\n\n/**\n * Failures that occur when product code reaches an unexpected or otherwise illegal state.\n */\ncase object IllegalStateFailure extends ServerFailure {\n  override val failureName: String = \"IllegalStateFailure\"\n}\n\n/**\n * An unexpected candidate was returned in a candidate source that was unable to be transformed.\n */\ncase object UnexpectedCandidateResult extends ServerFailure {\n  override val failureName: String = \"UnexpectedCandidateResult\"\n}\n\n/**\n * An unexpected Candidate was returned in a marshaller\n */\ncase object UnexpectedCandidateInMarshaller extends ServerFailure {\n  override val failureName: String = \"UnexpectedCandidateInMarshaller\"\n}\n\n/**\n * Pipeline execution failed due to an incorrectly configured quality factor (e.g, accessing\n * quality factor state for a pipeline that does not have quality factor configured)\n */\ncase object MisconfiguredQualityFactor extends ServerFailure {\n  override val failureName: String = \"MisconfiguredQualityFactor\"\n}\n\n/**\n * Pipeline execution failed due to an incorrectly configured decorator (e.g, decorator\n * returned the wrong type or tried to decorate an already decorated candidate)\n */\ncase object MisconfiguredDecorator extends ServerFailure {\n  override val failureName: String = \"MisconfiguredDecorator\"\n}\n\n/**\n * Candidate Source Pipeline execution failed due to a timeout.\n */\ncase object CandidateSourceTimeout extends ServerFailure {\n  override val failureName: String = \"CandidateSourceTimeout\"\n}\n\n/**\n * Platform Failures are issues in the core Product Mixer logic itself which prevent a pipeline from\n * properly executing. These failures are the responsibility of the Product Mixer team.\n */\ntrait PlatformFailure extends PipelineFailureCategory {\n  override val categoryName: String = \"PlatformFailure\"\n}\n\n/**\n * Pipeline execution failed due to an unexpected error in Product Mixer.\n *\n * ExecutionFailed indicates a bug with the core Product Mixer execution logic rather than with a\n * specific product. For example, a bug in PipelineBuilder leading to us returning a\n * ProductPipelineResult that neither succeeded nor failed.\n */\ncase object ExecutionFailed extends PlatformFailure {\n  override val failureName: String = \"ExecutionFailed\"\n}\n\n/**\n * Pipeline execution failed due to a feature hydration failure.\n *\n * FeatureHydrationFailed indicates that the underlying hydration for a feature defined in a hydrator\n * failed (e.g, typically from a RPC call failing).\n */\ncase object FeatureHydrationFailed extends PlatformFailure {\n  override val failureName: String = \"FeatureHydrationFailed\"\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure/PipelineFailureClassifier.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.pipeline_failure\n\n/** Represents a way to classify a given [[Throwable]] to a [[PipelineFailure]] */\ncase class PipelineFailureClassifier(\n  classifier: PartialFunction[Throwable, PipelineFailure])\n    extends PartialFunction[Throwable, PipelineFailure] {\n  override def isDefinedAt(throwable: Throwable): Boolean = classifier.isDefinedAt(throwable)\n  override def apply(throwable: Throwable): PipelineFailure = classifier.apply(throwable)\n}\n\nprivate[core] object PipelineFailureClassifier {\n  val Empty: PipelineFailureClassifier = PipelineFailureClassifier(PartialFunction.empty)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure/PipelineFailureSerializer.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.pipeline_failure\n\nimport com.fasterxml.jackson.core.JsonGenerator\nimport com.fasterxml.jackson.databind.JsonSerializer\nimport com.fasterxml.jackson.databind.SerializerProvider\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\n\nprivate[pipeline_failure] class PipelineFailureSerializer()\n    extends JsonSerializer[PipelineFailure] {\n\n  private sealed trait BaseSerializableException\n\n  private case class SerializableException(\n    `class`: String,\n    message: String,\n    stackTrace: Seq[String],\n    cause: Option[BaseSerializableException])\n      extends BaseSerializableException\n\n  private case class SerializablePipelineFailure(\n    category: String,\n    reason: String,\n    underlying: Option[BaseSerializableException],\n    componentStack: Option[ComponentIdentifierStack],\n    stackTrace: Seq[String])\n      extends BaseSerializableException\n\n  private def serializeStackTrace(stackTrace: Array[StackTraceElement]): Seq[String] =\n    stackTrace.map(stackTraceElement => \"at \" + stackTraceElement.toString)\n\n  private def mkSerializableException(\n    t: Throwable,\n    recursionDepth: Int = 0\n  ): Option[BaseSerializableException] = {\n    t match {\n      case _ if recursionDepth > 4 =>\n        // in the unfortunate case of a super deep chain of exceptions, stop if we get too deep\n        None\n      case pipelineFailure: PipelineFailure =>\n        Some(\n          SerializablePipelineFailure(\n            category =\n              pipelineFailure.category.categoryName + \"/\" + pipelineFailure.category.failureName,\n            reason = pipelineFailure.reason,\n            underlying =\n              pipelineFailure.underlying.flatMap(mkSerializableException(_, recursionDepth + 1)),\n            componentStack = pipelineFailure.componentStack,\n            stackTrace = serializeStackTrace(pipelineFailure.getStackTrace)\n          ))\n      case t =>\n        Some(\n          SerializableException(\n            `class` = t.getClass.getName,\n            message = t.getMessage,\n            stackTrace = serializeStackTrace(t.getStackTrace),\n            cause = Option(t.getCause).flatMap(mkSerializableException(_, recursionDepth + 1))\n          )\n        )\n    }\n  }\n\n  override def serialize(\n    pipelineFailure: PipelineFailure,\n    gen: JsonGenerator,\n    serializers: SerializerProvider\n  ): Unit = serializers.defaultSerializeValue(mkSerializableException(pipelineFailure), gen)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/access_policy\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/recommendation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_execution_logger\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_selector_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/quality_factor_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n        \"stringcenter/client/src/main/scala/com/twitter/stringcenter/client\",\n        \"stringcenter/client/src/main/scala/com/twitter/stringcenter/client/stitch\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/access_policy\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_execution_logger\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_selector_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/quality_factor_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product/ProductPipeline.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.product\n\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.WithDebugAccessPolicies\nimport com.twitter.product_mixer.core.model.common.identifier.ProductPipelineIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.Pipeline\nimport com.twitter.stitch.Arrow\n\n/**\n * A Product Pipeline\n *\n * This is an abstract class, as we only construct these via the [[ProductPipelineBuilder]].\n *\n * A [[ProductPipeline]] is capable of processing a [[Request]] and returning a response.\n *\n * @tparam RequestType the domain model for the query or request\n * @tparam ResponseType the final marshalled result type\n */\nabstract class ProductPipeline[RequestType <: Request, ResponseType] private[product]\n    extends Pipeline[ProductPipelineRequest[RequestType], ResponseType]\n    with WithDebugAccessPolicies {\n  override private[core] val config: ProductPipelineConfig[RequestType, _, ResponseType]\n  override val arrow: Arrow[\n    ProductPipelineRequest[RequestType],\n    ProductPipelineResult[ResponseType]\n  ]\n  override val identifier: ProductPipelineIdentifier\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product/ProductPipelineBuilder.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.product\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.finagle.transport.Transport\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AccessPolicy\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.gate.DenyLoggedOutUsersGate\nimport com.twitter.product_mixer.core.gate.ParamGate\nimport com.twitter.product_mixer.core.gate.ParamGate.EnabledGateSuffix\nimport com.twitter.product_mixer.core.gate.ParamGate.SupportedClientGateSuffix\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.common.identifier.ProductPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.InvalidStepStateException\nimport com.twitter.product_mixer.core.pipeline.Pipeline\nimport com.twitter.product_mixer.core.pipeline.PipelineBuilder\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineBuilderFactory\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineConfig\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineResult\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureClassifier\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ProductDisabled\nimport com.twitter.product_mixer.core.pipeline.recommendation.RecommendationPipelineBuilderFactory\nimport com.twitter.product_mixer.core.pipeline.recommendation.RecommendationPipelineConfig\nimport com.twitter.product_mixer.core.pipeline.recommendation.RecommendationPipelineResult\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorObserver\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorStatus\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.StoppedGateException\nimport com.twitter.product_mixer.core.service.pipeline_execution_logger.PipelineExecutionLogger\nimport com.twitter.product_mixer.core.service.pipeline_executor.PipelineExecutor\nimport com.twitter.product_mixer.core.service.pipeline_executor.PipelineExecutorRequest\nimport com.twitter.product_mixer.core.service.pipeline_executor.PipelineExecutorResult\nimport com.twitter.product_mixer.core.service.pipeline_selector_executor.PipelineSelectorExecutor\nimport com.twitter.product_mixer.core.service.pipeline_selector_executor.PipelineSelectorExecutorResult\nimport com.twitter.product_mixer.core.service.quality_factor_executor.QualityFactorExecutorResult\nimport com.twitter.stitch.Arrow\nimport com.twitter.stringcenter.client.StringCenterRequestContext\nimport com.twitter.stringcenter.client.stitch.StringCenterRequestContextLetter\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.util.logging.Logging\nimport org.slf4j.MDC\n\nclass ProductPipelineBuilder[TRequest <: Request, Query <: PipelineQuery, Response](\n  gateExecutor: GateExecutor,\n  pipelineSelectorExecutor: PipelineSelectorExecutor,\n  pipelineExecutor: PipelineExecutor,\n  mixerPipelineBuilderFactory: MixerPipelineBuilderFactory,\n  recommendationPipelineBuilderFactory: RecommendationPipelineBuilderFactory,\n  override val statsReceiver: StatsReceiver,\n  pipelineExecutionLogger: PipelineExecutionLogger)\n    extends PipelineBuilder[ProductPipelineRequest[TRequest]]\n    with Logging { builder =>\n\n  override type UnderlyingResultType = Response\n  override type PipelineResultType = ProductPipelineResult[Response]\n\n  /**\n   * Query Transformer Step is implemented inline instead of using an executor.\n   *\n   * It's a simple, synchronous step that executes the query transformer.\n   *\n   * Since the output of the transformer is used in multiple other steps (Gate, Pipeline Execution),\n   * we've promoted the transformer to a step so that it's outputs can be reused easily.\n   */\n  def pipelineQueryTransformerStep(\n    queryTransformer: (TRequest, Params) => Query,\n    context: Executor.Context\n  ): Step[ProductPipelineRequest[TRequest], Query] =\n    new Step[ProductPipelineRequest[TRequest], Query] {\n\n      override def identifier: PipelineStepIdentifier =\n        ProductPipelineConfig.pipelineQueryTransformerStep\n\n      override def executorArrow: Arrow[ProductPipelineRequest[TRequest], Query] = {\n        wrapWithErrorHandling(context, identifier)(\n          Arrow.map[ProductPipelineRequest[TRequest], Query] {\n            case ProductPipelineRequest(request, params) => queryTransformer(request, params)\n          }\n        )\n      }\n\n      override def inputAdaptor(\n        query: ProductPipelineRequest[TRequest],\n        previousResult: ProductPipelineResult[Response]\n      ): ProductPipelineRequest[TRequest] = query\n\n      override def resultUpdater(\n        previousPipelineResult: ProductPipelineResult[Response],\n        executorResult: Query\n      ): ProductPipelineResult[Response] =\n        previousPipelineResult.copy(transformedQuery = Some(executorResult))\n    }\n\n  def qualityFactorStep(\n    qualityFactorStatus: QualityFactorStatus\n  ): Step[Query, QualityFactorExecutorResult] = {\n    new Step[Query, QualityFactorExecutorResult] {\n      override def identifier: PipelineStepIdentifier = ProductPipelineConfig.qualityFactorStep\n\n      override def executorArrow: Arrow[Query, QualityFactorExecutorResult] =\n        Arrow\n          .map[Query, QualityFactorExecutorResult] { _ =>\n            QualityFactorExecutorResult(\n              pipelineQualityFactors =\n                qualityFactorStatus.qualityFactorByPipeline.mapValues(_.currentValue)\n            )\n          }\n\n      override def inputAdaptor(\n        query: ProductPipelineRequest[TRequest],\n        previousResult: ProductPipelineResult[Response]\n      ): Query = previousResult.transformedQuery\n        .getOrElse {\n          throw InvalidStepStateException(identifier, \"TransformedQuery\")\n        }.asInstanceOf[Query]\n\n      override def resultUpdater(\n        previousPipelineResult: ProductPipelineResult[Response],\n        executorResult: QualityFactorExecutorResult\n      ): ProductPipelineResult[Response] = {\n        previousPipelineResult.copy(\n          transformedQuery = previousPipelineResult.transformedQuery.map {\n            case queryWithQualityFactor: HasQualityFactorStatus =>\n              queryWithQualityFactor\n                .withQualityFactorStatus(qualityFactorStatus).asInstanceOf[Query]\n            case query =>\n              query\n          },\n          qualityFactorResult = Some(executorResult)\n        )\n      }\n    }\n  }\n\n  def gatesStep(\n    gates: Seq[Gate[Query]],\n    context: Executor.Context\n  ): Step[Query, GateExecutorResult] = new Step[Query, GateExecutorResult] {\n    override def identifier: PipelineStepIdentifier = ProductPipelineConfig.gatesStep\n\n    override def executorArrow: Arrow[Query, GateExecutorResult] = {\n      gateExecutor.arrow(gates, context)\n    }\n\n    override def inputAdaptor(\n      query: ProductPipelineRequest[TRequest],\n      previousResult: ProductPipelineResult[Response]\n    ): Query = previousResult.transformedQuery\n      .getOrElse {\n        throw InvalidStepStateException(identifier, \"TransformedQuery\")\n      }.asInstanceOf[Query]\n\n    override def resultUpdater(\n      previousPipelineResult: ProductPipelineResult[Response],\n      executorResult: GateExecutorResult\n    ): ProductPipelineResult[Response] =\n      previousPipelineResult.copy(gateResult = Some(executorResult))\n  }\n\n  def pipelineSelectorStep(\n    pipelineByIdentifer: Map[ComponentIdentifier, Pipeline[Query, Response]],\n    pipelineSelector: Query => ComponentIdentifier,\n    context: Executor.Context\n  ): Step[Query, PipelineSelectorExecutorResult] =\n    new Step[Query, PipelineSelectorExecutorResult] {\n      override def identifier: PipelineStepIdentifier = ProductPipelineConfig.pipelineSelectorStep\n\n      override def executorArrow: Arrow[\n        Query,\n        PipelineSelectorExecutorResult\n      ] = pipelineSelectorExecutor.arrow(pipelineByIdentifer, pipelineSelector, context)\n\n      override def inputAdaptor(\n        query: ProductPipelineRequest[TRequest],\n        previousResult: ProductPipelineResult[Response]\n      ): Query =\n        previousResult.transformedQuery\n          .getOrElse(throw InvalidStepStateException(identifier, \"TransformedQuery\")).asInstanceOf[\n            Query]\n\n      override def resultUpdater(\n        previousPipelineResult: ProductPipelineResult[Response],\n        executorResult: PipelineSelectorExecutorResult\n      ): ProductPipelineResult[Response] =\n        previousPipelineResult.copy(pipelineSelectorResult = Some(executorResult))\n    }\n\n  def pipelineExecutionStep(\n    pipelineByIdentifier: Map[ComponentIdentifier, Pipeline[Query, Response]],\n    qualityFactorObserverByPipeline: Map[ComponentIdentifier, QualityFactorObserver],\n    context: Executor.Context\n  ): Step[PipelineExecutorRequest[Query], PipelineExecutorResult[Response]] =\n    new Step[PipelineExecutorRequest[Query], PipelineExecutorResult[Response]] {\n      override def identifier: PipelineStepIdentifier = ProductPipelineConfig.pipelineExecutionStep\n\n      override def executorArrow: Arrow[\n        PipelineExecutorRequest[Query],\n        PipelineExecutorResult[Response]\n      ] = {\n        pipelineExecutor.arrow(pipelineByIdentifier, qualityFactorObserverByPipeline, context)\n      }\n\n      override def inputAdaptor(\n        request: ProductPipelineRequest[TRequest],\n        previousResult: ProductPipelineResult[Response]\n      ): PipelineExecutorRequest[Query] = {\n        val query = previousResult.transformedQuery\n          .getOrElse {\n            throw InvalidStepStateException(identifier, \"TransformedQuery\")\n          }.asInstanceOf[Query]\n\n        val pipelineIdentifier = previousResult.pipelineSelectorResult\n          .map(_.pipelineIdentifier).getOrElse {\n            throw InvalidStepStateException(identifier, \"PipelineSelectorResult\")\n          }\n\n        PipelineExecutorRequest(query, pipelineIdentifier)\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: ProductPipelineResult[Response],\n        executorResult: PipelineExecutorResult[Response]\n      ): ProductPipelineResult[Response] = {\n\n        val mixerPipelineResult = executorResult.pipelineResult match {\n          case mixerPipelineResult: MixerPipelineResult[Response] @unchecked =>\n            Some(mixerPipelineResult)\n          case _ =>\n            None\n        }\n\n        val recommendationPipelineResult = executorResult.pipelineResult match {\n          case recommendationPipelineResult: RecommendationPipelineResult[\n                _,\n                Response\n              ] @unchecked =>\n            Some(recommendationPipelineResult)\n          case _ =>\n            None\n        }\n\n        previousPipelineResult.copy(\n          mixerPipelineResult = mixerPipelineResult,\n          recommendationPipelineResult = recommendationPipelineResult,\n          traceId = Trace.idOption.map(_.traceId.toString()),\n          result = executorResult.pipelineResult.result\n        )\n      }\n    }\n\n  def build(\n    parentComponentIdentifierStack: ComponentIdentifierStack,\n    config: ProductPipelineConfig[TRequest, Query, Response]\n  ): ProductPipeline[TRequest, Response] = {\n\n    val pipelineIdentifier = config.identifier\n\n    val context = Executor.Context(\n      PipelineFailureClassifier(\n        config.failureClassifier.orElse(StoppedGateException.classifier(ProductDisabled))),\n      parentComponentIdentifierStack.push(pipelineIdentifier)\n    )\n\n    val denyLoggedOutUsersGate = if (config.denyLoggedOutUsers) {\n      Some(DenyLoggedOutUsersGate(pipelineIdentifier))\n    } else {\n      None\n    }\n    val enabledGate: ParamGate =\n      ParamGate(pipelineIdentifier + EnabledGateSuffix, config.paramConfig.EnabledDeciderParam)\n    val supportedClientGate =\n      ParamGate(\n        pipelineIdentifier + SupportedClientGateSuffix,\n        config.paramConfig.SupportedClientParam)\n\n    /**\n     * Evaluate enabled decider gate first since if it's off, there is no reason to proceed\n     * Next evaluate supported client feature switch gate, followed by customer configured gates\n     */\n    val allGates =\n      denyLoggedOutUsersGate.toSeq ++: enabledGate +: supportedClientGate +: config.gates\n\n    val childPipelines: Seq[Pipeline[Query, Response]] =\n      config.pipelines.map {\n        case mixerConfig: MixerPipelineConfig[Query, _, Response] =>\n          mixerConfig.build(context.componentStack, mixerPipelineBuilderFactory)\n        case recommendationConfig: RecommendationPipelineConfig[Query, _, _, Response] =>\n          recommendationConfig.build(context.componentStack, recommendationPipelineBuilderFactory)\n        case other =>\n          throw new IllegalArgumentException(\n            s\"Product Pipelines only support Mixer and Recommendation pipelines, not $other\")\n      }\n\n    val pipelineByIdentifier: Map[ComponentIdentifier, Pipeline[Query, Response]] =\n      childPipelines.map { pipeline =>\n        (pipeline.identifier, pipeline)\n      }.toMap\n\n    val qualityFactorStatus: QualityFactorStatus =\n      QualityFactorStatus.build(config.qualityFactorConfigs)\n\n    val qualityFactorObserverByPipeline = qualityFactorStatus.qualityFactorByPipeline.mapValues {\n      qualityFactor =>\n        qualityFactor.buildObserver()\n    }\n\n    buildGaugesForQualityFactor(pipelineIdentifier, qualityFactorStatus, statsReceiver)\n\n    /**\n     * Initialize MDC with access logging with everything we have at request time. We can put\n     * more stuff into MDC later down the pipeline, but at risk of exceptions/errors preventing\n     * them from being added\n     */\n    val mdcInitArrow =\n      Arrow.map[ProductPipelineRequest[TRequest], ProductPipelineRequest[TRequest]] { request =>\n        val serviceIdentifier = ServiceIdentifier.fromCertificate(Transport.peerCertificate)\n        MDC.put(\"product\", config.product.identifier.name)\n        MDC.put(\"serviceIdentifier\", ServiceIdentifier.asString(serviceIdentifier))\n        request\n      }\n\n    val builtSteps = Seq(\n      pipelineQueryTransformerStep(config.pipelineQueryTransformer, context),\n      qualityFactorStep(qualityFactorStatus),\n      gatesStep(allGates, context),\n      pipelineSelectorStep(pipelineByIdentifier, config.pipelineSelector, context),\n      pipelineExecutionStep(pipelineByIdentifier, qualityFactorObserverByPipeline, context)\n    )\n\n    val underlying: Arrow[ProductPipelineRequest[TRequest], ProductPipelineResult[Response]] =\n      buildCombinedArrowFromSteps(\n        steps = builtSteps,\n        context = context,\n        initialEmptyResult = ProductPipelineResult.empty,\n        stepsInOrderFromConfig = ProductPipelineConfig.stepsInOrder\n      )\n\n    /**\n     * Unlike other components and pipelines, [[ProductPipeline]] must be observed in the\n     * [[ProductPipelineBuilder]] directly because the resulting [[ProductPipeline.arrow]]\n     * is run directly without an executor so must contain all stats.\n     */\n    val observed =\n      wrapProductPipelineWithExecutorBookkeeping[\n        ProductPipelineRequest[TRequest],\n        ProductPipelineResult[Response]\n      ](context, pipelineIdentifier)(underlying)\n\n    val finalArrow: Arrow[ProductPipelineRequest[TRequest], ProductPipelineResult[Response]] =\n      Arrow\n        .letWithArg[\n          ProductPipelineRequest[TRequest],\n          ProductPipelineResult[Response],\n          StringCenterRequestContext](StringCenterRequestContextLetter)(request =>\n          StringCenterRequestContext(\n            request.request.clientContext.languageCode,\n            request.request.clientContext.countryCode\n          ))(\n          mdcInitArrow\n            .andThen(observed)\n            .onSuccess(result => result.transformedQuery.map(pipelineExecutionLogger(_, result))))\n\n    val configFromBuilder = config\n    new ProductPipeline[TRequest, Response] {\n      override private[core] val config: ProductPipelineConfig[TRequest, _, Response] =\n        configFromBuilder\n      override val arrow: Arrow[ProductPipelineRequest[TRequest], ProductPipelineResult[Response]] =\n        finalArrow\n      override val identifier: ProductPipelineIdentifier = pipelineIdentifier\n      override val alerts: Seq[Alert] = config.alerts\n      override val debugAccessPolicies: Set[AccessPolicy] = config.debugAccessPolicies\n      override val children: Seq[Component] = allGates ++ childPipelines\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product/ProductPipelineBuilderFactory.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.product\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineBuilderFactory\nimport com.twitter.product_mixer.core.pipeline.recommendation.RecommendationPipelineBuilderFactory\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutor\nimport com.twitter.product_mixer.core.service.pipeline_execution_logger.PipelineExecutionLogger\nimport com.twitter.product_mixer.core.service.pipeline_executor.PipelineExecutor\nimport com.twitter.product_mixer.core.service.pipeline_selector_executor.PipelineSelectorExecutor\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ProductPipelineBuilderFactory @Inject() (\n  gateExecutor: GateExecutor,\n  pipelineSelectorExecutor: PipelineSelectorExecutor,\n  pipelineExecutor: PipelineExecutor,\n  mixerPipelineBuilderFactory: MixerPipelineBuilderFactory,\n  recommendationPipelineBuilderFactory: RecommendationPipelineBuilderFactory,\n  statsReceiver: StatsReceiver,\n  pipelineExecutionLogger: PipelineExecutionLogger) {\n  def get[\n    TRequest <: Request,\n    Query <: PipelineQuery,\n    Response\n  ]: ProductPipelineBuilder[TRequest, Query, Response] = {\n    new ProductPipelineBuilder[TRequest, Query, Response](\n      gateExecutor,\n      pipelineSelectorExecutor,\n      pipelineExecutor,\n      mixerPipelineBuilderFactory,\n      recommendationPipelineBuilderFactory,\n      statsReceiver,\n      pipelineExecutionLogger\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product/ProductPipelineConfig.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.product\n\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AccessPolicy\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ProductPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.PipelineConfig\nimport com.twitter.product_mixer.core.pipeline.PipelineConfigCompanion\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.product.ProductParamConfig\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorConfig\nimport com.twitter.timelines.configapi.Params\n\ntrait ProductPipelineConfig[TRequest <: Request, Query <: PipelineQuery, Response]\n    extends PipelineConfig {\n\n  override val identifier: ProductPipelineIdentifier\n\n  val product: Product\n  val paramConfig: ProductParamConfig\n\n  /**\n   * Product Pipeline Gates will be executed before any other step (including retrieval from mixer\n   * pipelines). They're executed sequentially, and any \"Stop\" result will prevent pipeline execution.\n   */\n  def gates: Seq[Gate[Query]] = Seq.empty\n\n  def pipelineQueryTransformer(request: TRequest, params: Params): Query\n\n  /**\n   * A list of all pipelines that power this product directly (there is no need to include pipelines\n   * called by those pipelines).\n   *\n   * Only pipeline from this list should referenced from the pipelineSelector\n   */\n  def pipelines: Seq[PipelineConfig]\n\n  /**\n   * A pipeline selector selects a pipeline (from the list in `def pipelines`) to handle the\n   * current request.\n   */\n  def pipelineSelector(query: Query): ComponentIdentifier\n\n  /**\n   ** [[qualityFactorConfigs]] associates [[QualityFactorConfig]]s to specific pipelines\n   * using [[ComponentIdentifier]].\n   */\n  def qualityFactorConfigs: Map[ComponentIdentifier, QualityFactorConfig] =\n    Map.empty\n\n  /**\n   * By default (for safety), product mixer pipelines do not allow logged out requests.\n   * A \"DenyLoggedOutUsersGate\" will be generated and added to the pipeline.\n   *\n   * You can disable this behavior by overriding `denyLoggedOutUsers` with False.\n   */\n  val denyLoggedOutUsers: Boolean = true\n\n  /**\n   * A pipeline can define a partial function to rescue failures here. They will be treated as failures\n   * from a monitoring standpoint, and cancellation exceptions will always be propagated (they cannot be caught here).\n   */\n  def failureClassifier: PartialFunction[Throwable, PipelineFailure] = PartialFunction.empty\n\n  /**\n   * Alerts can be used to indicate the pipeline's service level objectives. Alerts and\n   * dashboards will be automatically created based on this information.\n   */\n  val alerts: Seq[Alert] = Seq.empty\n\n  /**\n   * Access Policies can be used to gate who can query a product from Product Mixer's query tool\n   * (go/turntable).\n   *\n   * This will typically be gated by an LDAP group associated with your team. For example:\n   *\n   * {{{\n   *   override val debugAccessPolicies: Set[AccessPolicy] = Set(AllowedLdapGroups(\"NAME\"))\n   * }}}\n   *\n   * You can disable all queries by using the [[com.twitter.product_mixer.core.functional_component.common.access_policy.BlockEverything]] policy.\n   */\n  val debugAccessPolicies: Set[AccessPolicy]\n}\n\nobject ProductPipelineConfig extends PipelineConfigCompanion {\n  val pipelineQueryTransformerStep: PipelineStepIdentifier = PipelineStepIdentifier(\n    \"PipelineQueryTransformer\")\n  val qualityFactorStep: PipelineStepIdentifier = PipelineStepIdentifier(\"QualityFactor\")\n  val gatesStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Gates\")\n  val pipelineSelectorStep: PipelineStepIdentifier = PipelineStepIdentifier(\"PipelineSelector\")\n  val pipelineExecutionStep: PipelineStepIdentifier = PipelineStepIdentifier(\"PipelineExecution\")\n\n  /** All the Steps which are executed by a [[ProductPipeline]] in the order in which they are run */\n  override val stepsInOrder: Seq[PipelineStepIdentifier] = Seq(\n    pipelineQueryTransformerStep,\n    qualityFactorStep,\n    gatesStep,\n    pipelineSelectorStep,\n    pipelineExecutionStep\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product/ProductPipelineRequest.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.product\n\nimport com.twitter.timelines.configapi.Params\n\ncase class ProductPipelineRequest[RequestType](request: RequestType, params: Params)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product/ProductPipelineResult.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.product\n\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.PipelineResult\nimport com.twitter.product_mixer.core.pipeline.mixer.MixerPipelineResult\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.recommendation.RecommendationPipelineResult\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutorResult\nimport com.twitter.product_mixer.core.service.pipeline_selector_executor.PipelineSelectorExecutorResult\nimport com.twitter.product_mixer.core.service.quality_factor_executor.QualityFactorExecutorResult\n\ncase class ProductPipelineResult[Result](\n  transformedQuery: Option[PipelineQuery],\n  qualityFactorResult: Option[QualityFactorExecutorResult],\n  gateResult: Option[GateExecutorResult],\n  pipelineSelectorResult: Option[PipelineSelectorExecutorResult],\n  mixerPipelineResult: Option[MixerPipelineResult[Result]],\n  recommendationPipelineResult: Option[RecommendationPipelineResult[_, Result]],\n  traceId: Option[String],\n  failure: Option[PipelineFailure],\n  result: Option[Result])\n    extends PipelineResult[Result] {\n\n  override val resultSize: Int = {\n    if (mixerPipelineResult.isDefined) {\n      mixerPipelineResult.map(_.resultSize).getOrElse(0)\n    } else {\n      recommendationPipelineResult.map(_.resultSize).getOrElse(0)\n    }\n  }\n\n  override def withFailure(failure: PipelineFailure): PipelineResult[Result] =\n    copy(failure = Some(failure))\n\n  override def withResult(result: Result): PipelineResult[Result] = copy(result = Some(result))\n}\n\nobject ProductPipelineResult {\n  def empty[A]: ProductPipelineResult[A] = ProductPipelineResult(\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None\n  )\n\n  def fromResult[A](result: A): ProductPipelineResult[A] = ProductPipelineResult(\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    None,\n    Some(result)\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/recommendation/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector:insert_append_results\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/scoring\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/async_feature_map_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_pipeline_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/filter_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/quality_factor_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/query_feature_hydrator_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/scoring_pipeline_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/selector_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/common/alert\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/async_feature_map_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_pipeline_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/filter_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/quality_factor_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/query_feature_hydrator_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/scoring_pipeline_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/selector_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/recommendation/RecommendationPipeline.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.recommendation\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.RecommendationPipelineIdentifier\nimport com.twitter.product_mixer.core.pipeline.Pipeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Arrow\n\n/**\n * A Recommendation Pipeline\n *\n * This is an abstract class, as we only construct these via the [[RecommendationPipelineBuilder]].\n *\n * A [[RecommendationPipeline]] is capable of processing requests (queries) and returning responses (results)\n * in the correct format to directly send to users.\n *\n * @tparam Query the domain model for the query or request\n * @tparam Candidate the type of the candidates\n * @tparam Result the final marshalled result type\n */\nabstract class RecommendationPipeline[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  Result]\n    extends Pipeline[Query, Result] {\n  override private[core] val config: RecommendationPipelineConfig[Query, Candidate, _, Result]\n  override val arrow: Arrow[Query, RecommendationPipelineResult[Candidate, Result]]\n  override val identifier: RecommendationPipelineIdentifier\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/recommendation/RecommendationPipelineBuilder.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.recommendation\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.logging.Logging\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.asyncfeaturemap.AsyncFeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseQueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.common.identifier.RecommendationPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ScoringPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemPresentation\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.InvalidStepStateException\nimport com.twitter.product_mixer.core.pipeline.PipelineBuilder\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipeline\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineBuilderFactory\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MisconfiguredDecorator\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureClassifier\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ProductDisabled\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipeline\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipelineBuilderFactory\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipelineConfig\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorObserver\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorStatus\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutor\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutorResults\nimport com.twitter.product_mixer.core.service.candidate_decorator_executor.CandidateDecoratorExecutor\nimport com.twitter.product_mixer.core.service.candidate_decorator_executor.CandidateDecoratorExecutorResult\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutorResult\nimport com.twitter.product_mixer.core.service.candidate_pipeline_executor.CandidatePipelineExecutor\nimport com.twitter.product_mixer.core.service.candidate_pipeline_executor.CandidatePipelineExecutorResult\nimport com.twitter.product_mixer.core.service.domain_marshaller_executor.DomainMarshallerExecutor\nimport com.twitter.product_mixer.core.service.filter_executor.FilterExecutor\nimport com.twitter.product_mixer.core.service.filter_executor.FilterExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.StoppedGateException\nimport com.twitter.product_mixer.core.service.pipeline_result_side_effect_executor.PipelineResultSideEffectExecutor\nimport com.twitter.product_mixer.core.service.quality_factor_executor.QualityFactorExecutorResult\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.scoring_pipeline_executor.ScoringPipelineExecutor\nimport com.twitter.product_mixer.core.service.scoring_pipeline_executor.ScoringPipelineExecutorResult\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutor\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutorResult\nimport com.twitter.product_mixer.core.service.transport_marshaller_executor.TransportMarshallerExecutor\nimport com.twitter.stitch.Arrow\n\n/**\n * RecommendationPipelineBuilder builds [[RecommendationPipeline]]s from [[RecommendationPipelineConfig]]s.\n *\n * You should inject a [[RecommendationPipelineBuilderFactory]] and call `.get` to build these.\n *\n * @see [[RecommendationPipelineConfig]] for the description of the type parameters.\n *\n * @note Almost a mirror of MixerPipelineBuilder\n */\n\nclass RecommendationPipelineBuilder[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  DomainResultType <: HasMarshalling,\n  Result\n](\n  candidatePipelineExecutor: CandidatePipelineExecutor,\n  gateExecutor: GateExecutor,\n  selectorExecutor: SelectorExecutor,\n  queryFeatureHydratorExecutor: QueryFeatureHydratorExecutor,\n  asyncFeatureMapExecutor: AsyncFeatureMapExecutor,\n  candidateFeatureHydratorExecutor: CandidateFeatureHydratorExecutor,\n  filterExecutor: FilterExecutor,\n  scoringPipelineExecutor: ScoringPipelineExecutor,\n  candidateDecoratorExecutor: CandidateDecoratorExecutor,\n  domainMarshallerExecutor: DomainMarshallerExecutor,\n  transportMarshallerExecutor: TransportMarshallerExecutor,\n  pipelineResultSideEffectExecutor: PipelineResultSideEffectExecutor,\n  candidatePipelineBuilderFactory: CandidatePipelineBuilderFactory,\n  scoringPipelineBuilderFactory: ScoringPipelineBuilderFactory,\n  override val statsReceiver: StatsReceiver)\n    extends PipelineBuilder[Query]\n    with Logging {\n\n  override type UnderlyingResultType = Result\n  override type PipelineResultType = RecommendationPipelineResult[Candidate, Result]\n\n  def qualityFactorStep(\n    qualityFactorStatus: QualityFactorStatus\n  ): Step[Query, QualityFactorExecutorResult] =\n    new Step[Query, QualityFactorExecutorResult] {\n      override def identifier: PipelineStepIdentifier =\n        RecommendationPipelineConfig.qualityFactorStep\n\n      override def executorArrow: Arrow[Query, QualityFactorExecutorResult] =\n        Arrow\n          .map[Query, QualityFactorExecutorResult] { _ =>\n            QualityFactorExecutorResult(\n              pipelineQualityFactors =\n                qualityFactorStatus.qualityFactorByPipeline.mapValues(_.currentValue)\n            )\n          }\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: RecommendationPipelineResult[Candidate, Result]\n      ): Query = query\n\n      override def resultUpdater(\n        previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n        executorResult: QualityFactorExecutorResult\n      ): RecommendationPipelineResult[Candidate, Result] =\n        previousPipelineResult.copy(qualityFactorResult = Some(executorResult))\n\n      override def queryUpdater(\n        query: Query,\n        executorResult: QualityFactorExecutorResult\n      ): Query = {\n        query match {\n          case queryWithQualityFactor: HasQualityFactorStatus =>\n            queryWithQualityFactor\n              .withQualityFactorStatus(\n                queryWithQualityFactor.qualityFactorStatus.getOrElse(QualityFactorStatus.empty) ++\n                  qualityFactorStatus\n              ).asInstanceOf[Query]\n          case _ =>\n            query\n        }\n      }\n    }\n\n  def gatesStep(\n    gates: Seq[Gate[Query]],\n    context: Executor.Context\n  ): Step[Query, GateExecutorResult] = new Step[Query, GateExecutorResult] {\n    override def identifier: PipelineStepIdentifier = RecommendationPipelineConfig.gatesStep\n\n    override def executorArrow: Arrow[Query, GateExecutorResult] =\n      gateExecutor.arrow(gates, context)\n\n    override def inputAdaptor(\n      query: Query,\n      previousResult: RecommendationPipelineResult[Candidate, Result]\n    ): Query =\n      query\n\n    override def resultUpdater(\n      previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n      executorResult: GateExecutorResult\n    ): RecommendationPipelineResult[Candidate, Result] =\n      previousPipelineResult.copy(gateResult = Some(executorResult))\n  }\n\n  def fetchQueryFeaturesStep(\n    queryFeatureHydrators: Seq[BaseQueryFeatureHydrator[Query, _]],\n    stepIdentifier: PipelineStepIdentifier,\n    updater: ResultUpdater[\n      RecommendationPipelineResult[Candidate, Result],\n      QueryFeatureHydratorExecutor.Result\n    ],\n    context: Executor.Context\n  ): Step[Query, QueryFeatureHydratorExecutor.Result] =\n    new Step[Query, QueryFeatureHydratorExecutor.Result] {\n      override def identifier: PipelineStepIdentifier = stepIdentifier\n\n      override def executorArrow: Arrow[Query, QueryFeatureHydratorExecutor.Result] =\n        queryFeatureHydratorExecutor.arrow(\n          queryFeatureHydrators,\n          RecommendationPipelineConfig.stepsAsyncFeatureHydrationCanBeCompletedBy,\n          context)\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: RecommendationPipelineResult[Candidate, Result]\n      ): Query = query\n\n      override def resultUpdater(\n        previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n        executorResult: QueryFeatureHydratorExecutor.Result\n      ): RecommendationPipelineResult[Candidate, Result] =\n        updater(previousPipelineResult, executorResult)\n\n      override def queryUpdater(\n        query: Query,\n        executorResult: QueryFeatureHydratorExecutor.Result\n      ): Query =\n        query\n          .withFeatureMap(\n            query.features\n              .getOrElse(FeatureMap.empty) ++ executorResult.featureMap).asInstanceOf[Query]\n    }\n\n  def asyncFeaturesStep(\n    stepToHydrateFor: PipelineStepIdentifier,\n    context: Executor.Context\n  ): Step[AsyncFeatureMap, AsyncFeatureMapExecutorResults] =\n    new Step[AsyncFeatureMap, AsyncFeatureMapExecutorResults] {\n      override def identifier: PipelineStepIdentifier =\n        RecommendationPipelineConfig.asyncFeaturesStep(stepToHydrateFor)\n\n      override def executorArrow: Arrow[AsyncFeatureMap, AsyncFeatureMapExecutorResults] =\n        asyncFeatureMapExecutor.arrow(\n          stepToHydrateFor,\n          identifier,\n          context\n        )\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: RecommendationPipelineResult[Candidate, Result]\n      ): AsyncFeatureMap =\n        previousResult.mergedAsyncQueryFeatures\n          .getOrElse(\n            throw InvalidStepStateException(identifier, \"MergedAsyncQueryFeatures\")\n          )\n\n      override def resultUpdater(\n        previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n        executorResult: AsyncFeatureMapExecutorResults\n      ): RecommendationPipelineResult[Candidate, Result] =\n        previousPipelineResult.copy(\n          asyncFeatureHydrationResults = previousPipelineResult.asyncFeatureHydrationResults match {\n            case Some(existingResults) => Some(existingResults ++ executorResult)\n            case None => Some(executorResult)\n          })\n\n      override def queryUpdater(\n        query: Query,\n        executorResult: AsyncFeatureMapExecutorResults\n      ): Query =\n        if (executorResult.featureMapsByStep\n            .getOrElse(stepToHydrateFor, FeatureMap.empty).isEmpty) {\n          query\n        } else {\n          query\n            .withFeatureMap(\n              query.features\n                .getOrElse(FeatureMap.empty) ++ executorResult.featureMapsByStep(\n                stepToHydrateFor)).asInstanceOf[Query]\n        }\n    }\n\n  def candidatePipelinesStep(\n    candidatePipelines: Seq[CandidatePipeline[Query]],\n    defaultFailOpenPolicy: FailOpenPolicy,\n    failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy],\n    qualityFactorObserverByPipeline: Map[ComponentIdentifier, QualityFactorObserver],\n    context: Executor.Context\n  ): Step[CandidatePipeline.Inputs[Query], CandidatePipelineExecutorResult] =\n    new Step[CandidatePipeline.Inputs[Query], CandidatePipelineExecutorResult] {\n      override def identifier: PipelineStepIdentifier =\n        RecommendationPipelineConfig.candidatePipelinesStep\n\n      override def executorArrow: Arrow[CandidatePipeline.Inputs[\n        Query\n      ], CandidatePipelineExecutorResult] =\n        candidatePipelineExecutor\n          .arrow(\n            candidatePipelines,\n            defaultFailOpenPolicy,\n            failOpenPolicies,\n            qualityFactorObserverByPipeline,\n            context\n          )\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: RecommendationPipelineResult[Candidate, Result]\n      ): CandidatePipeline.Inputs[\n        Query\n      ] = CandidatePipeline.Inputs(query, Seq.empty)\n\n      override def resultUpdater(\n        previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n        executorResult: CandidatePipelineExecutorResult\n      ): RecommendationPipelineResult[Candidate, Result] =\n        previousPipelineResult.copy(candidatePipelineResults = Some(executorResult))\n\n      override def queryUpdater(\n        query: Query,\n        executorResult: CandidatePipelineExecutorResult\n      ): Query = {\n        val updatedFeatureMap = query.features\n          .getOrElse(FeatureMap.empty) ++ executorResult.queryFeatureMap\n        query\n          .withFeatureMap(updatedFeatureMap).asInstanceOf[Query]\n      }\n    }\n\n  def dependentCandidatePipelinesStep(\n    candidatePipelines: Seq[CandidatePipeline[Query]],\n    defaultFailOpenPolicy: FailOpenPolicy,\n    failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy],\n    qualityFactorObserverByPipeline: Map[ComponentIdentifier, QualityFactorObserver],\n    context: Executor.Context\n  ): Step[CandidatePipeline.Inputs[Query], CandidatePipelineExecutorResult] =\n    new Step[CandidatePipeline.Inputs[Query], CandidatePipelineExecutorResult] {\n      override def identifier: PipelineStepIdentifier =\n        RecommendationPipelineConfig.dependentCandidatePipelinesStep\n\n      override def executorArrow: Arrow[CandidatePipeline.Inputs[\n        Query\n      ], CandidatePipelineExecutorResult] =\n        candidatePipelineExecutor\n          .arrow(\n            candidatePipelines,\n            defaultFailOpenPolicy,\n            failOpenPolicies,\n            qualityFactorObserverByPipeline,\n            context\n          )\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: RecommendationPipelineResult[Candidate, Result]\n      ): CandidatePipeline.Inputs[\n        Query\n      ] = {\n        val previousCandidates = previousResult.candidatePipelineResults\n          .getOrElse {\n            throw InvalidStepStateException(identifier, \"Candidates\")\n          }.candidatePipelineResults.flatMap(_.result.getOrElse(Seq.empty))\n\n        CandidatePipeline.Inputs(query, previousCandidates)\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n        executorResult: CandidatePipelineExecutorResult\n      ): RecommendationPipelineResult[Candidate, Result] =\n        previousPipelineResult.copy(dependentCandidatePipelineResults = Some(executorResult))\n\n      override def queryUpdater(\n        query: Query,\n        executorResult: CandidatePipelineExecutorResult\n      ): Query = {\n        val updatedFeatureMap = query.features\n          .getOrElse(FeatureMap.empty) ++ executorResult.queryFeatureMap\n        query\n          .withFeatureMap(updatedFeatureMap).asInstanceOf[Query]\n      }\n    }\n\n  abstract class FilterStep(\n    filters: Seq[Filter[Query, Candidate]],\n    context: Executor.Context,\n    override val identifier: PipelineStepIdentifier)\n      extends Step[\n        (Query, Seq[CandidateWithFeatures[Candidate]]),\n        FilterExecutorResult[Candidate]\n      ] {\n\n    def itemCandidates(\n      previousResult: RecommendationPipelineResult[Candidate, Result]\n    ): Seq[CandidateWithDetails]\n\n    override def executorArrow: Arrow[\n      (Query, Seq[CandidateWithFeatures[Candidate]]),\n      FilterExecutorResult[Candidate]\n    ] =\n      filterExecutor.arrow(filters, context)\n\n    override def inputAdaptor(\n      query: Query,\n      previousResult: RecommendationPipelineResult[Candidate, Result]\n    ): (Query, Seq[CandidateWithFeatures[Candidate]]) = {\n\n      val extractedItemCandidates = itemCandidates(previousResult).collect {\n        case itemCandidate: ItemCandidateWithDetails => itemCandidate\n      }\n\n      (query, extractedItemCandidates.asInstanceOf[Seq[CandidateWithFeatures[Candidate]]])\n    }\n  }\n\n  def postCandidatePipelinesSelectorStep(\n    selectors: Seq[Selector[Query]],\n    context: Executor.Context\n  ): Step[SelectorExecutor.Inputs[Query], SelectorExecutorResult] =\n    new Step[SelectorExecutor.Inputs[Query], SelectorExecutorResult] {\n      override def identifier: PipelineStepIdentifier =\n        RecommendationPipelineConfig.postCandidatePipelinesSelectorsStep\n\n      override def executorArrow: Arrow[SelectorExecutor.Inputs[\n        Query\n      ], SelectorExecutorResult] =\n        selectorExecutor.arrow(selectors, context)\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: RecommendationPipelineResult[Candidate, Result]\n      ): SelectorExecutor.Inputs[Query] = {\n        val candidatePipelineResults = previousResult.candidatePipelineResults\n          .getOrElse {\n            throw InvalidStepStateException(identifier, \"CandidatePipelineResults\")\n          }.candidatePipelineResults.flatMap(_.result.getOrElse(Seq.empty))\n        val dependentCandidatePipelineResults = previousResult.dependentCandidatePipelineResults\n          .getOrElse {\n            throw InvalidStepStateException(identifier, \"DependentCandidatePipelineResults\")\n          }.candidatePipelineResults.flatMap(_.result.getOrElse(Seq.empty))\n\n        SelectorExecutor.Inputs(\n          query = query,\n          candidatesWithDetails = candidatePipelineResults ++ dependentCandidatePipelineResults\n        )\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n        executorResult: SelectorExecutorResult\n      ): RecommendationPipelineResult[Candidate, Result] =\n        previousPipelineResult.copy(postCandidatePipelinesSelectorResults = Some(executorResult))\n    }\n\n  def postCandidatePipelinesFeatureHydrationStep(\n    hydrators: Seq[BaseCandidateFeatureHydrator[Query, Candidate, _]],\n    context: Executor.Context\n  ): Step[\n    CandidateFeatureHydratorExecutor.Inputs[Query, Candidate],\n    CandidateFeatureHydratorExecutorResult[Candidate]\n  ] = new Step[\n    CandidateFeatureHydratorExecutor.Inputs[Query, Candidate],\n    CandidateFeatureHydratorExecutorResult[Candidate]\n  ] {\n    override def identifier: PipelineStepIdentifier =\n      RecommendationPipelineConfig.postCandidatePipelinesFeatureHydrationStep\n\n    override def executorArrow: Arrow[\n      CandidateFeatureHydratorExecutor.Inputs[Query, Candidate],\n      CandidateFeatureHydratorExecutorResult[Candidate]\n    ] =\n      candidateFeatureHydratorExecutor.arrow(hydrators, context)\n\n    override def inputAdaptor(\n      query: Query,\n      previousResult: RecommendationPipelineResult[Candidate, Result]\n    ): CandidateFeatureHydratorExecutor.Inputs[Query, Candidate] = {\n      val selectedCandidatesResult =\n        previousResult.postCandidatePipelinesSelectorResults.getOrElse {\n          throw InvalidStepStateException(identifier, \"PostCandidatePipelinesSelectorResults\")\n        }.selectedCandidates\n\n      CandidateFeatureHydratorExecutor.Inputs(\n        query,\n        selectedCandidatesResult.asInstanceOf[Seq[CandidateWithFeatures[Candidate]]])\n    }\n\n    override def resultUpdater(\n      previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n      executorResult: CandidateFeatureHydratorExecutorResult[Candidate]\n    ): RecommendationPipelineResult[Candidate, Result] = previousPipelineResult.copy(\n      postCandidatePipelinesFeatureHydrationResults = Some(executorResult)\n    )\n  }\n\n  def globalFiltersStep(\n    filters: Seq[Filter[Query, Candidate]],\n    context: Executor.Context\n  ): Step[(Query, Seq[CandidateWithFeatures[Candidate]]), FilterExecutorResult[Candidate]] =\n    new FilterStep(filters, context, RecommendationPipelineConfig.globalFiltersStep) {\n      override def itemCandidates(\n        previousResult: RecommendationPipelineResult[Candidate, Result]\n      ): Seq[CandidateWithDetails] = {\n        val candidates = previousResult.postCandidatePipelinesSelectorResults\n          .getOrElse {\n            throw InvalidStepStateException(identifier, \"PostCandidatePipelineSelectorResults\")\n          }.selectedCandidates.collect {\n            case itemCandidate: ItemCandidateWithDetails => itemCandidate\n          }\n\n        val featureMaps = previousResult.postCandidatePipelinesFeatureHydrationResults\n          .getOrElse {\n            throw InvalidStepStateException(\n              identifier,\n              \"PostCandidatePipelineFeatureHydrationResults\")\n          }.results.map(_.features)\n        // If no hydrators were run, this list would be empty. Otherwise, order and cardinality is\n        // always ensured to match.\n        if (featureMaps.isEmpty) {\n          candidates\n        } else {\n          candidates.zip(featureMaps).map {\n            case (candidate, featureMap) =>\n              candidate.copy(features = candidate.features ++ featureMap)\n          }\n        }\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n        executorResult: FilterExecutorResult[Candidate]\n      ): RecommendationPipelineResult[Candidate, Result] = previousPipelineResult.copy(\n        globalFilterResults = Some(executorResult)\n      )\n    }\n\n  def scoringPipelinesStep(\n    scoringPipelines: Seq[ScoringPipeline[Query, Candidate]],\n    context: Executor.Context,\n    defaultFailOpenPolicy: FailOpenPolicy,\n    failOpenPolicies: Map[ScoringPipelineIdentifier, FailOpenPolicy],\n    qualityFactorObserverByPipeline: Map[ComponentIdentifier, QualityFactorObserver]\n  ): Step[ScoringPipelineExecutor.Inputs[Query], ScoringPipelineExecutorResult[\n    Candidate\n  ]] =\n    new Step[ScoringPipelineExecutor.Inputs[Query], ScoringPipelineExecutorResult[\n      Candidate\n    ]] {\n      override def identifier: PipelineStepIdentifier =\n        RecommendationPipelineConfig.scoringPipelinesStep\n\n      override def executorArrow: Arrow[\n        ScoringPipelineExecutor.Inputs[Query],\n        ScoringPipelineExecutorResult[Candidate]\n      ] = scoringPipelineExecutor.arrow(\n        scoringPipelines,\n        context,\n        defaultFailOpenPolicy,\n        failOpenPolicies,\n        qualityFactorObserverByPipeline\n      )\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: RecommendationPipelineResult[Candidate, Result]\n      ): ScoringPipelineExecutor.Inputs[Query] = {\n        val selectedCandidates =\n          previousResult.postCandidatePipelinesSelectorResults.getOrElse {\n            throw InvalidStepStateException(identifier, \"PostCandidatePipelinesSelectorResults\")\n          }.selectedCandidates\n\n        val itemCandidates = selectedCandidates.collect {\n          case itemCandidate: ItemCandidateWithDetails => itemCandidate\n        }\n\n        val featureMaps = previousResult.postCandidatePipelinesFeatureHydrationResults\n          .getOrElse {\n            throw InvalidStepStateException(\n              identifier,\n              \"PostCandidatePipelineFeatureHydrationResults\")\n          }.results.map(_.features)\n        // If no hydrators were run, this list would be empty. Otherwise, order and cardinality is\n        // always ensured to match.\n        val updatedCandidates = if (featureMaps.isEmpty) {\n          itemCandidates\n        } else {\n          itemCandidates.zip(featureMaps).map {\n            case (candidate, featureMap) =>\n              candidate.copy(features = candidate.features ++ featureMap)\n          }\n        }\n\n        // Filter the original list of candidates to keep only the ones that were kept from\n        // filtering\n        val filterResults: Set[Candidate] = previousResult.globalFilterResults\n          .getOrElse {\n            throw InvalidStepStateException(identifier, \"FilterResults\")\n          }.result.toSet\n\n        val filteredItemCandidates = updatedCandidates.filter { itemCandidate =>\n          filterResults.contains(itemCandidate.candidate.asInstanceOf[Candidate])\n        }\n\n        ScoringPipelineExecutor.Inputs(\n          query,\n          filteredItemCandidates\n        )\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n        executorResult: ScoringPipelineExecutorResult[Candidate]\n      ): RecommendationPipelineResult[Candidate, Result] = previousPipelineResult\n        .copy(scoringPipelineResults = Some(executorResult))\n    }\n\n  def resultSelectorsStep(\n    selectors: Seq[Selector[Query]],\n    context: Executor.Context\n  ): Step[SelectorExecutor.Inputs[Query], SelectorExecutorResult] =\n    new Step[SelectorExecutor.Inputs[Query], SelectorExecutorResult] {\n      override def identifier: PipelineStepIdentifier =\n        RecommendationPipelineConfig.resultSelectorsStep\n\n      override def executorArrow: Arrow[SelectorExecutor.Inputs[\n        Query\n      ], SelectorExecutorResult] =\n        selectorExecutor.arrow(selectors, context)\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: RecommendationPipelineResult[Candidate, Result]\n      ): SelectorExecutor.Inputs[Query] = {\n\n        /**\n         * See [[ScoringPipelineExecutor]], scoringPipelineResults contains the fully re-merged\n         * and updated FeatureMap so there's no need to do any recomposition. Scoring Pipeline Results\n         * has only candidates that were kept in previous filtering, with their final merged feature\n         * map.\n         */\n        val scorerResults = previousResult.scoringPipelineResults.getOrElse {\n          throw InvalidStepStateException(identifier, \"Scores\")\n        }\n\n        SelectorExecutor.Inputs(\n          query = query,\n          candidatesWithDetails = scorerResults.result\n        )\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n        executorResult: SelectorExecutorResult\n      ): RecommendationPipelineResult[Candidate, Result] =\n        previousPipelineResult.copy(resultSelectorResults = Some(executorResult))\n    }\n\n  def postSelectionFiltersStep(\n    filters: Seq[Filter[Query, Candidate]],\n    context: Executor.Context\n  ): Step[(Query, Seq[CandidateWithFeatures[Candidate]]), FilterExecutorResult[Candidate]] =\n    new FilterStep(filters, context, RecommendationPipelineConfig.postSelectionFiltersStep) {\n\n      override def itemCandidates(\n        previousResult: RecommendationPipelineResult[Candidate, Result]\n      ): Seq[CandidateWithDetails] = {\n        previousResult.resultSelectorResults.getOrElse {\n          throw InvalidStepStateException(identifier, \"Candidates\")\n        }.selectedCandidates\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n        executorResult: FilterExecutorResult[Candidate]\n      ): RecommendationPipelineResult[Candidate, Result] = {\n        previousPipelineResult.copy(postSelectionFilterResults = Some(executorResult))\n      }\n    }\n\n  def decoratorStep(\n    decorator: Option[CandidateDecorator[Query, Candidate]],\n    context: Executor.Context\n  ): Step[(Query, Seq[CandidateWithFeatures[Candidate]]), CandidateDecoratorExecutorResult] =\n    new Step[(Query, Seq[CandidateWithFeatures[Candidate]]), CandidateDecoratorExecutorResult] {\n      override def identifier: PipelineStepIdentifier = RecommendationPipelineConfig.decoratorStep\n\n      override lazy val executorArrow: Arrow[\n        (Query, Seq[CandidateWithFeatures[Candidate]]),\n        CandidateDecoratorExecutorResult\n      ] =\n        candidateDecoratorExecutor.arrow(decorator, context)\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: RecommendationPipelineResult[Candidate, Result]\n      ): (Query, Seq[CandidateWithFeatures[Candidate]]) = {\n\n        val selectorResults = previousResult.resultSelectorResults\n          .getOrElse {\n            throw InvalidStepStateException(identifier, \"SelectorResults\")\n          }.selectedCandidates\n          .collect { case candidate: ItemCandidateWithDetails => candidate }\n\n        val filterResults = previousResult.postSelectionFilterResults\n          .getOrElse {\n            throw InvalidStepStateException(identifier, \"PostSelectionFilterResults\")\n          }.result.toSet\n\n        val itemCandidateWithDetailsPostFiltering =\n          selectorResults\n            .filter(candidateWithDetails =>\n              filterResults.contains(\n                candidateWithDetails.candidate\n                  .asInstanceOf[Candidate]))\n            .asInstanceOf[Seq[CandidateWithFeatures[Candidate]]]\n\n        (query, itemCandidateWithDetailsPostFiltering)\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n        executorResult: CandidateDecoratorExecutorResult\n      ): RecommendationPipelineResult[Candidate, Result] =\n        previousPipelineResult.copy(\n          candidateDecoratorResult = Some(executorResult)\n        )\n    }\n\n  def domainMarshallingStep(\n    domainMarshaller: DomainMarshaller[Query, DomainResultType],\n    context: Executor.Context\n  ): Step[DomainMarshallerExecutor.Inputs[Query], DomainMarshallerExecutor.Result[\n    DomainResultType\n  ]] =\n    new Step[DomainMarshallerExecutor.Inputs[Query], DomainMarshallerExecutor.Result[\n      DomainResultType\n    ]] {\n      override def identifier: PipelineStepIdentifier =\n        RecommendationPipelineConfig.domainMarshallerStep\n\n      override def executorArrow: Arrow[\n        DomainMarshallerExecutor.Inputs[Query],\n        DomainMarshallerExecutor.Result[DomainResultType]\n      ] =\n        domainMarshallerExecutor.arrow(domainMarshaller, context)\n\n      override def inputAdaptor(\n        query: Query,\n        previousResult: RecommendationPipelineResult[Candidate, Result]\n      ): DomainMarshallerExecutor.Inputs[Query] = {\n        val selectorResults = previousResult.resultSelectorResults.getOrElse {\n          throw InvalidStepStateException(identifier, \"SelectorResults\")\n        }\n\n        val filterResults = previousResult.postSelectionFilterResults\n          .getOrElse {\n            throw InvalidStepStateException(identifier, \"PostSelectionFilterResults\")\n          }.result.toSet\n\n        val filteredResults = selectorResults.selectedCandidates.collect {\n          case candidate: ItemCandidateWithDetails\n              if filterResults.contains(candidate.candidate.asInstanceOf[Candidate]) =>\n            candidate\n        }\n\n        val decoratorResults = previousResult.candidateDecoratorResult\n          .getOrElse(throw InvalidStepStateException(identifier, \"DecoratorStep\")).result.map {\n            decoration =>\n              decoration.candidate -> decoration.presentation\n          }.toMap\n\n        val finalResults = filteredResults.map { itemWithDetails =>\n          decoratorResults.get(itemWithDetails.candidate) match {\n            case Some(presentation: ItemPresentation) =>\n              if (itemWithDetails.presentation.isDefined) {\n                throw PipelineFailure(\n                  category = MisconfiguredDecorator,\n                  reason = \"Item Candidate already decorated\",\n                  componentStack = Some(context.componentStack))\n              } else {\n                itemWithDetails.copy(presentation = Some(presentation))\n              }\n            case Some(_) =>\n              throw PipelineFailure(\n                category = MisconfiguredDecorator,\n                reason = \"Item Candidate got back a non ItemPresentation from decorator\",\n                componentStack = Some(context.componentStack))\n            case None => itemWithDetails\n          }\n        }\n        DomainMarshallerExecutor.Inputs(\n          query = query,\n          candidatesWithDetails = finalResults\n        )\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n        executorResult: DomainMarshallerExecutor.Result[DomainResultType]\n      ): RecommendationPipelineResult[Candidate, Result] = previousPipelineResult.copy(\n        domainMarshallerResults = Some(executorResult)\n      )\n    }\n\n  def resultSideEffectsStep(\n    sideEffects: Seq[PipelineResultSideEffect[Query, DomainResultType]],\n    context: Executor.Context\n  ): Step[\n    PipelineResultSideEffect.Inputs[Query, DomainResultType],\n    PipelineResultSideEffectExecutor.Result\n  ] = new Step[\n    PipelineResultSideEffect.Inputs[Query, DomainResultType],\n    PipelineResultSideEffectExecutor.Result\n  ] {\n    override def identifier: PipelineStepIdentifier =\n      RecommendationPipelineConfig.resultSideEffectsStep\n\n    override def executorArrow: Arrow[\n      PipelineResultSideEffect.Inputs[Query, DomainResultType],\n      PipelineResultSideEffectExecutor.Result\n    ] = pipelineResultSideEffectExecutor.arrow(sideEffects, context)\n\n    override def inputAdaptor(\n      query: Query,\n      previousResult: RecommendationPipelineResult[Candidate, Result]\n    ): PipelineResultSideEffect.Inputs[Query, DomainResultType] = {\n\n      // Re-apply decorations to the selected results\n      val resultSelectorResults = {\n        val decoratorResults = previousResult.candidateDecoratorResult\n          .getOrElse(throw InvalidStepStateException(identifier, \"DecoratorStep\")).result.map {\n            decoration =>\n              decoration.candidate -> decoration.presentation\n          }.toMap\n\n        val previousSelectorResults = previousResult.resultSelectorResults.getOrElse {\n          throw InvalidStepStateException(identifier, \"SelectorResults\")\n        }\n\n        val filterResults = previousResult.postSelectionFilterResults\n          .getOrElse {\n            throw InvalidStepStateException(identifier, \"PostSelectionFilterResults\")\n          }.result.toSet\n\n        val filteredSelectorResults = previousSelectorResults.selectedCandidates.collect {\n          case candidate: ItemCandidateWithDetails\n              if filterResults.contains(candidate.candidate.asInstanceOf[Candidate]) =>\n            candidate\n        }\n\n        val decoratedSelectedResults = filteredSelectorResults.map {\n          case itemWithDetails: ItemCandidateWithDetails =>\n            decoratorResults.get(itemWithDetails.candidate) match {\n              case Some(presentation: ItemPresentation) =>\n                if (itemWithDetails.presentation.isDefined) {\n                  throw PipelineFailure(\n                    category = MisconfiguredDecorator,\n                    reason = \"Item Candidate already decorated\",\n                    componentStack = Some(context.componentStack))\n                } else {\n                  itemWithDetails.copy(presentation = Some(presentation))\n                }\n              case Some(_) =>\n                throw PipelineFailure(\n                  category = MisconfiguredDecorator,\n                  reason = \"Item Candidate got back a non ItemPresentation from decorator\",\n                  componentStack = Some(context.componentStack))\n              case None => itemWithDetails\n            }\n          case item =>\n            // This branch should be impossible to hit since we do a .collect on ItemCandidateWithDetails\n            // as part of executing the candidate pipelines.\n            throw PipelineFailure(\n              category = IllegalStateFailure,\n              reason =\n                s\"Only ItemCandidateWithDetails expected in pipeline, found: ${item.toString}\",\n              componentStack = Some(context.componentStack)\n            )\n        }\n\n        previousSelectorResults.copy(selectedCandidates = decoratedSelectedResults)\n      }\n\n      val domainMarshallerResults = previousResult.domainMarshallerResults.getOrElse {\n        throw InvalidStepStateException(identifier, \"DomainMarshallerResults\")\n      }\n\n      PipelineResultSideEffect.Inputs[Query, DomainResultType](\n        query = query,\n        selectedCandidates = resultSelectorResults.selectedCandidates,\n        remainingCandidates = resultSelectorResults.remainingCandidates,\n        droppedCandidates = resultSelectorResults.droppedCandidates,\n        response = domainMarshallerResults.result.asInstanceOf[DomainResultType]\n      )\n    }\n\n    override def resultUpdater(\n      previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n      executorResult: PipelineResultSideEffectExecutor.Result\n    ): RecommendationPipelineResult[Candidate, Result] =\n      previousPipelineResult.copy(resultSideEffectResults = Some(executorResult))\n  }\n\n  def transportMarshallingStep(\n    transportMarshaller: TransportMarshaller[DomainResultType, Result],\n    context: Executor.Context\n  ): Step[\n    TransportMarshallerExecutor.Inputs[DomainResultType],\n    TransportMarshallerExecutor.Result[Result]\n  ] = new Step[TransportMarshallerExecutor.Inputs[\n    DomainResultType\n  ], TransportMarshallerExecutor.Result[Result]] {\n    override def identifier: PipelineStepIdentifier =\n      RecommendationPipelineConfig.transportMarshallerStep\n\n    override def executorArrow: Arrow[TransportMarshallerExecutor.Inputs[\n      DomainResultType\n    ], TransportMarshallerExecutor.Result[Result]] =\n      transportMarshallerExecutor.arrow(transportMarshaller, context)\n\n    override def inputAdaptor(\n      query: Query,\n      previousResult: RecommendationPipelineResult[Candidate, Result]\n    ): TransportMarshallerExecutor.Inputs[DomainResultType] = {\n      val domainMarshallingResults = previousResult.domainMarshallerResults.getOrElse {\n        throw InvalidStepStateException(identifier, \"DomainMarshallerResults\")\n      }\n\n      // Since the PipelineResult just uses HasMarshalling\n      val domainResult = domainMarshallingResults.result.asInstanceOf[DomainResultType]\n\n      TransportMarshallerExecutor.Inputs(domainResult)\n    }\n\n    override def resultUpdater(\n      previousPipelineResult: RecommendationPipelineResult[Candidate, Result],\n      executorResult: TransportMarshallerExecutor.Result[Result]\n    ): RecommendationPipelineResult[Candidate, Result] = previousPipelineResult.copy(\n      transportMarshallerResults = Some(executorResult),\n      result = Some(executorResult.result)\n    )\n  }\n\n  def build(\n    parentComponentIdentifierStack: ComponentIdentifierStack,\n    config: RecommendationPipelineConfig[\n      Query,\n      Candidate,\n      DomainResultType,\n      Result\n    ]\n  ): RecommendationPipeline[Query, Candidate, Result] = {\n    val pipelineIdentifier = config.identifier\n\n    val context = Executor.Context(\n      PipelineFailureClassifier(\n        config.failureClassifier.orElse(StoppedGateException.classifier(ProductDisabled))),\n      parentComponentIdentifierStack.push(pipelineIdentifier)\n    )\n\n    val decorator = config.decorator.map(decorator =>\n      CandidateDecorator.copyWithUpdatedIdentifier(decorator, pipelineIdentifier))\n\n    val qualityFactorStatus: QualityFactorStatus =\n      QualityFactorStatus.build(config.qualityFactorConfigs)\n\n    val qualityFactorObserverByPipeline =\n      qualityFactorStatus.qualityFactorByPipeline.mapValues { qualityFactor =>\n        qualityFactor.buildObserver()\n      }\n\n    buildGaugesForQualityFactor(pipelineIdentifier, qualityFactorStatus, statsReceiver)\n\n    val candidatePipelines: Seq[CandidatePipeline[Query]] = config.candidatePipelines.map {\n      pipelineConfig: CandidatePipelineConfig[Query, _, _, _] =>\n        pipelineConfig.build(context.componentStack, candidatePipelineBuilderFactory)\n    }\n\n    val dependentCandidatePipelines: Seq[CandidatePipeline[Query]] =\n      config.dependentCandidatePipelines.map {\n        pipelineConfig: DependentCandidatePipelineConfig[Query, _, _, _] =>\n          pipelineConfig.build(context.componentStack, candidatePipelineBuilderFactory)\n      }\n\n    val scoringPipelines: Seq[ScoringPipeline[Query, Candidate]] = config.scoringPipelines.map {\n      pipelineConfig: ScoringPipelineConfig[Query, Candidate] =>\n        pipelineConfig.build(context.componentStack, scoringPipelineBuilderFactory)\n    }\n\n    val builtSteps = Seq(\n      qualityFactorStep(qualityFactorStatus),\n      gatesStep(config.gates, context),\n      fetchQueryFeaturesStep(\n        config.fetchQueryFeatures,\n        RecommendationPipelineConfig.fetchQueryFeaturesStep,\n        (previousPipelineResult, executorResult) =>\n          previousPipelineResult.copy(queryFeatures = Some(executorResult)),\n        context\n      ),\n      fetchQueryFeaturesStep(\n        config.fetchQueryFeaturesPhase2,\n        RecommendationPipelineConfig.fetchQueryFeaturesPhase2Step,\n        (previousPipelineResult, executorResult) =>\n          previousPipelineResult.copy(\n            queryFeaturesPhase2 = Some(executorResult),\n            mergedAsyncQueryFeatures = Some(\n              previousPipelineResult.queryFeatures\n                .getOrElse(throw InvalidStepStateException(\n                  RecommendationPipelineConfig.fetchQueryFeaturesPhase2Step,\n                  \"QueryFeatures\"))\n                .asyncFeatureMap ++ executorResult.asyncFeatureMap)\n          ),\n        context\n      ),\n      asyncFeaturesStep(RecommendationPipelineConfig.candidatePipelinesStep, context),\n      candidatePipelinesStep(\n        candidatePipelines,\n        config.defaultFailOpenPolicy,\n        config.candidatePipelineFailOpenPolicies,\n        qualityFactorObserverByPipeline,\n        context),\n      asyncFeaturesStep(RecommendationPipelineConfig.dependentCandidatePipelinesStep, context),\n      dependentCandidatePipelinesStep(\n        dependentCandidatePipelines,\n        config.defaultFailOpenPolicy,\n        config.candidatePipelineFailOpenPolicies,\n        qualityFactorObserverByPipeline,\n        context),\n      asyncFeaturesStep(RecommendationPipelineConfig.postCandidatePipelinesSelectorsStep, context),\n      postCandidatePipelinesSelectorStep(config.postCandidatePipelinesSelectors, context),\n      asyncFeaturesStep(\n        RecommendationPipelineConfig.postCandidatePipelinesFeatureHydrationStep,\n        context),\n      postCandidatePipelinesFeatureHydrationStep(\n        config.postCandidatePipelinesFeatureHydration,\n        context),\n      asyncFeaturesStep(RecommendationPipelineConfig.globalFiltersStep, context),\n      globalFiltersStep(config.globalFilters, context),\n      asyncFeaturesStep(RecommendationPipelineConfig.scoringPipelinesStep, context),\n      scoringPipelinesStep(\n        scoringPipelines,\n        context,\n        config.defaultFailOpenPolicy,\n        config.scoringPipelineFailOpenPolicies,\n        qualityFactorObserverByPipeline\n      ),\n      asyncFeaturesStep(RecommendationPipelineConfig.resultSelectorsStep, context),\n      resultSelectorsStep(config.resultSelectors, context),\n      asyncFeaturesStep(RecommendationPipelineConfig.postSelectionFiltersStep, context),\n      postSelectionFiltersStep(config.postSelectionFilters, context),\n      asyncFeaturesStep(RecommendationPipelineConfig.decoratorStep, context),\n      decoratorStep(decorator, context),\n      domainMarshallingStep(config.domainMarshaller, context),\n      asyncFeaturesStep(RecommendationPipelineConfig.resultSideEffectsStep, context),\n      resultSideEffectsStep(config.resultSideEffects, context),\n      transportMarshallingStep(config.transportMarshaller, context)\n    )\n\n    val finalArrow = buildCombinedArrowFromSteps(\n      steps = builtSteps,\n      context = context,\n      initialEmptyResult = RecommendationPipelineResult.empty,\n      stepsInOrderFromConfig = RecommendationPipelineConfig.stepsInOrder\n    )\n\n    val configFromBuilder = config\n    new RecommendationPipeline[Query, Candidate, Result] {\n      override private[core] val config: RecommendationPipelineConfig[\n        Query,\n        Candidate,\n        _,\n        Result\n      ] =\n        configFromBuilder\n      override val arrow: Arrow[Query, RecommendationPipelineResult[Candidate, Result]] =\n        finalArrow\n      override val identifier: RecommendationPipelineIdentifier = pipelineIdentifier\n      override val alerts: Seq[Alert] = config.alerts\n      override val children: Seq[Component] =\n        config.gates ++\n          config.fetchQueryFeatures ++\n          candidatePipelines ++\n          dependentCandidatePipelines ++\n          config.postCandidatePipelinesFeatureHydration ++\n          config.globalFilters ++\n          scoringPipelines ++\n          config.postSelectionFilters ++\n          config.resultSideEffects ++\n          decorator.toSeq ++\n          Seq(config.domainMarshaller, config.transportMarshaller)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/recommendation/RecommendationPipelineBuilderFactory.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.recommendation\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineBuilderFactory\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipelineBuilderFactory\nimport com.twitter.product_mixer.core.service.candidate_decorator_executor.CandidateDecoratorExecutor\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.candidate_pipeline_executor.CandidatePipelineExecutor\nimport com.twitter.product_mixer.core.service.domain_marshaller_executor.DomainMarshallerExecutor\nimport com.twitter.product_mixer.core.service.filter_executor.FilterExecutor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutor\nimport com.twitter.product_mixer.core.service.pipeline_result_side_effect_executor.PipelineResultSideEffectExecutor\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutor\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.scoring_pipeline_executor.ScoringPipelineExecutor\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutor\nimport com.twitter.product_mixer.core.service.transport_marshaller_executor.TransportMarshallerExecutor\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass RecommendationPipelineBuilderFactory @Inject() (\n  candidatePipelineExecutor: CandidatePipelineExecutor,\n  gateExecutor: GateExecutor,\n  selectorExecutor: SelectorExecutor,\n  queryFeatureHydratorExecutor: QueryFeatureHydratorExecutor,\n  asyncFeatureMapExecutor: AsyncFeatureMapExecutor,\n  candidateFeatureHydratorExecutor: CandidateFeatureHydratorExecutor,\n  filterExecutor: FilterExecutor,\n  scoringPipelineExecutor: ScoringPipelineExecutor,\n  candidateDecoratorExecutor: CandidateDecoratorExecutor,\n  domainMarshallerExecutor: DomainMarshallerExecutor,\n  transportMarshallerExecutor: TransportMarshallerExecutor,\n  pipelineResultSideEffectExecutor: PipelineResultSideEffectExecutor,\n  candidatePipelineBuilderFactory: CandidatePipelineBuilderFactory,\n  scoringPipelineBuilderFactory: ScoringPipelineBuilderFactory,\n  statsReceiver: StatsReceiver) {\n\n  def get[\n    Query <: PipelineQuery,\n    Candidate <: UniversalNoun[Any],\n    DomainResultType <: HasMarshalling,\n    Result\n  ]: RecommendationPipelineBuilder[Query, Candidate, DomainResultType, Result] = {\n    new RecommendationPipelineBuilder[Query, Candidate, DomainResultType, Result](\n      candidatePipelineExecutor,\n      gateExecutor,\n      selectorExecutor,\n      queryFeatureHydratorExecutor,\n      asyncFeatureMapExecutor,\n      candidateFeatureHydratorExecutor,\n      filterExecutor,\n      scoringPipelineExecutor,\n      candidateDecoratorExecutor,\n      domainMarshallerExecutor,\n      transportMarshallerExecutor,\n      pipelineResultSideEffectExecutor,\n      candidatePipelineBuilderFactory,\n      scoringPipelineBuilderFactory,\n      statsReceiver\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/recommendation/RecommendationPipelineConfig.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.recommendation\n\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseQueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.functional_component.gate.Gate\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.common.identifier.RecommendationPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ScoringPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.PipelineConfig\nimport com.twitter.product_mixer.core.pipeline.PipelineConfigCompanion\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.candidate.DependentCandidatePipelineConfig\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ClosedGate\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipelineConfig\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorConfig\n\n/**\n *  This is the configuration necessary to generate a Recommendation Pipeline. Product code should create a\n *  RecommendationPipelineConfig, and then use a RecommendationPipelineBuilder to get the final RecommendationPipeline which can\n *  process requests.\n *\n * @tparam Query - The domain model for the query or request\n * @tparam Candidate - The type of the candidates that the Candidate Pipelines are generating\n * @tparam UnmarshalledResultType - The result type of the pipeline, but before marshalling to a wire protocol like URT\n * @tparam Result - The final result that will be served to users\n */\ntrait RecommendationPipelineConfig[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  UnmarshalledResultType <: HasMarshalling,\n  Result]\n    extends PipelineConfig {\n\n  override val identifier: RecommendationPipelineIdentifier\n\n  /**\n   * Recommendation Pipeline Gates will be executed before any other step (including retrieval from candidate\n   * pipelines). They're executed sequentially, and any \"Stop\" result will prevent pipeline execution.\n   */\n  def gates: Seq[Gate[Query]] = Seq.empty\n\n  /**\n   * A recommendation pipeline can fetch query-level features before candidate pipelines are executed.\n   */\n  def fetchQueryFeatures: Seq[BaseQueryFeatureHydrator[Query, _]] = Seq.empty\n\n  /**\n   * Candidate pipelines retrieve candidates for possible inclusion in the result\n   */\n  def fetchQueryFeaturesPhase2: Seq[BaseQueryFeatureHydrator[Query, _]] = Seq.empty\n\n  /**\n   * What candidate pipelines should this Recommendations Pipeline get candidate from?\n   */\n  def candidatePipelines: Seq[CandidatePipelineConfig[Query, _, _, _]]\n\n  /**\n   * Dependent candidate pipelines to retrieve candidates that depend on the result of [[candidatePipelines]]\n   * [[DependentCandidatePipelineConfig]] have access to the list of previously retrieved & decorated\n   * candidates for use in constructing the query object.\n   */\n  def dependentCandidatePipelines: Seq[DependentCandidatePipelineConfig[Query, _, _, _]] = Seq.empty\n\n  /**\n   * Takes final ranked list of candidates & apply any business logic (e.g, deduplicating and merging\n   * candidates before scoring).\n   */\n  def postCandidatePipelinesSelectors: Seq[Selector[Query]] = Seq(InsertAppendResults(AllPipelines))\n\n  /**\n   * After selectors are run, you can fetch features for each candidate.\n   * The existing features from previous hydrations are passed in as inputs. You are not expected to\n   * put them into the resulting feature map yourself - they will be merged for you by the platform.\n   */\n  def postCandidatePipelinesFeatureHydration: Seq[\n    BaseCandidateFeatureHydrator[Query, Candidate, _]\n  ] =\n    Seq.empty\n\n  /**\n   * Global filters to run on all candidates.\n   */\n  def globalFilters: Seq[Filter[Query, Candidate]] = Seq.empty\n\n  /**\n   * By default, a Recommendation Pipeline will fail closed - if any candidate or scoring\n   * pipeline fails to return a result, then the Recommendation Pipeline will not return a result.\n   * You can adjust this default policy, or provide specific policies to specific pipelines.\n   * Those specific policies will take priority.\n   *\n   * FailOpenPolicy.All will always fail open (the RecommendationPipeline will continue without that pipeline)\n   * FailOpenPolicy.Never will always fail closed (the RecommendationPipeline will fail if that pipeline fails)\n   *\n   * There's a default policy, and a specific Map of policies that takes precedence.\n   */\n  def defaultFailOpenPolicy: FailOpenPolicy = FailOpenPolicy(Set(ClosedGate))\n  def candidatePipelineFailOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy] =\n    Map.empty\n  def scoringPipelineFailOpenPolicies: Map[ScoringPipelineIdentifier, FailOpenPolicy] = Map.empty\n\n  /**\n   ** [[qualityFactorConfigs]] associates [[QualityFactorConfig]]s to specific candidate pipelines\n   * using [[ComponentIdentifier]].\n   */\n  def qualityFactorConfigs: Map[ComponentIdentifier, QualityFactorConfig] =\n    Map.empty\n\n  /**\n   * Scoring pipelines for scoring candidates.\n   * @note These do not drop or re-order candidates, you should do those in the sub-sequent selectors\n   * step based off of the scores on candidates set in those [[ScoringPipeline]]s.\n   */\n  def scoringPipelines: Seq[ScoringPipelineConfig[Query, Candidate]]\n\n  /**\n   * Takes final ranked list of candidates & apply any business logic (e.g, capping number\n   * of ad accounts or pacing ad accounts).\n   */\n  def resultSelectors: Seq[Selector[Query]]\n\n  /**\n   * Takes the final selected list of candidates and applies a final list of filters.\n   * Useful for doing very expensive filtering at the end of your pipeline.\n   */\n  def postSelectionFilters: Seq[Filter[Query, Candidate]] = Seq.empty\n\n  /**\n   * Decorators allow for adding Presentations to candidates. While the Presentation can contain any\n   * arbitrary data, Decorators are often used to add a UrtItemPresentation for URT item support. Most\n   * customers will prefer to set a decorator in their respective candidate pipeline, however, a final\n   * global one is available for those that do global decoration as late possible to avoid unnecessary hydrations.\n   * @note This decorator can only return an ItemPresentation.\n   * @note This decorator cannot decorate an already decorated candidate from the prior decorator\n   *       step in candidate pipelines.\n   */\n  def decorator: Option[CandidateDecorator[Query, Candidate]] = None\n\n  /**\n   * Domain marshaller transforms the selections into the model expected by the marshaller\n   */\n  def domainMarshaller: DomainMarshaller[Query, UnmarshalledResultType]\n\n  /**\n   * Mixer result side effects that are executed after selection and domain marshalling\n   */\n  def resultSideEffects: Seq[PipelineResultSideEffect[Query, UnmarshalledResultType]] = Seq()\n\n  /**\n   * Transport marshaller transforms the model into our line-level API like URT or JSON\n   */\n  def transportMarshaller: TransportMarshaller[UnmarshalledResultType, Result]\n\n  /**\n   * A pipeline can define a partial function to rescue failures here. They will be treated as failures\n   * from a monitoring standpoint, and cancellation exceptions will always be propagated (they cannot be caught here).\n   */\n  def failureClassifier: PartialFunction[Throwable, PipelineFailure] = PartialFunction.empty\n\n  /**\n   * Alerts can be used to indicate the pipeline's service level objectives. Alerts and\n   * dashboards will be automatically created based on this information.\n   */\n  val alerts: Seq[Alert] = Seq.empty\n\n  /**\n   * This method is used by the product mixer framework to build the pipeline.\n   */\n  private[core] final def build(\n    parentComponentIdentifierStack: ComponentIdentifierStack,\n    builder: RecommendationPipelineBuilderFactory\n  ): RecommendationPipeline[Query, Candidate, Result] =\n    builder.get.build(parentComponentIdentifierStack, this)\n}\n\nobject RecommendationPipelineConfig extends PipelineConfigCompanion {\n  val qualityFactorStep: PipelineStepIdentifier = PipelineStepIdentifier(\"QualityFactor\")\n  val gatesStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Gates\")\n  val fetchQueryFeaturesStep: PipelineStepIdentifier = PipelineStepIdentifier(\"FetchQueryFeatures\")\n  val fetchQueryFeaturesPhase2Step: PipelineStepIdentifier = PipelineStepIdentifier(\n    \"FetchQueryFeaturesPhase2\")\n  val candidatePipelinesStep: PipelineStepIdentifier = PipelineStepIdentifier(\"CandidatePipelines\")\n  val dependentCandidatePipelinesStep: PipelineStepIdentifier =\n    PipelineStepIdentifier(\"DependentCandidatePipelines\")\n  val postCandidatePipelinesSelectorsStep: PipelineStepIdentifier =\n    PipelineStepIdentifier(\"PostCandidatePipelinesSelectors\")\n  val postCandidatePipelinesFeatureHydrationStep: PipelineStepIdentifier =\n    PipelineStepIdentifier(\"PostCandidatePipelinesFeatureHydration\")\n  val globalFiltersStep: PipelineStepIdentifier = PipelineStepIdentifier(\"GlobalFilters\")\n  val scoringPipelinesStep: PipelineStepIdentifier = PipelineStepIdentifier(\"ScoringPipelines\")\n  val resultSelectorsStep: PipelineStepIdentifier = PipelineStepIdentifier(\"ResultSelectors\")\n  val postSelectionFiltersStep: PipelineStepIdentifier = PipelineStepIdentifier(\n    \"PostSelectionFilters\")\n  val decoratorStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Decorator\")\n  val domainMarshallerStep: PipelineStepIdentifier = PipelineStepIdentifier(\"DomainMarshaller\")\n  val resultSideEffectsStep: PipelineStepIdentifier = PipelineStepIdentifier(\"ResultSideEffects\")\n  val transportMarshallerStep: PipelineStepIdentifier = PipelineStepIdentifier(\n    \"TransportMarshaller\")\n\n  /** All the Steps which are executed by a [[RecommendationPipeline]] in the order in which they are run */\n  override val stepsInOrder: Seq[PipelineStepIdentifier] = Seq(\n    qualityFactorStep,\n    gatesStep,\n    fetchQueryFeaturesStep,\n    fetchQueryFeaturesPhase2Step,\n    asyncFeaturesStep(candidatePipelinesStep),\n    candidatePipelinesStep,\n    asyncFeaturesStep(dependentCandidatePipelinesStep),\n    dependentCandidatePipelinesStep,\n    asyncFeaturesStep(postCandidatePipelinesSelectorsStep),\n    postCandidatePipelinesSelectorsStep,\n    asyncFeaturesStep(postCandidatePipelinesFeatureHydrationStep),\n    postCandidatePipelinesFeatureHydrationStep,\n    asyncFeaturesStep(globalFiltersStep),\n    globalFiltersStep,\n    asyncFeaturesStep(scoringPipelinesStep),\n    scoringPipelinesStep,\n    asyncFeaturesStep(resultSelectorsStep),\n    resultSelectorsStep,\n    asyncFeaturesStep(postSelectionFiltersStep),\n    postSelectionFiltersStep,\n    asyncFeaturesStep(decoratorStep),\n    decoratorStep,\n    domainMarshallerStep,\n    asyncFeaturesStep(resultSideEffectsStep),\n    resultSideEffectsStep,\n    transportMarshallerStep\n  )\n\n  /**\n   * All the Steps which an [[com.twitter.product_mixer.core.functional_component.feature_hydrator.AsyncHydrator AsyncHydrator]]\n   * can be configured to [[com.twitter.product_mixer.core.functional_component.feature_hydrator.AsyncHydrator.hydrateBefore hydrateBefore]]\n   */\n  override val stepsAsyncFeatureHydrationCanBeCompletedBy: Set[PipelineStepIdentifier] = Set(\n    candidatePipelinesStep,\n    dependentCandidatePipelinesStep,\n    postCandidatePipelinesSelectorsStep,\n    postCandidatePipelinesFeatureHydrationStep,\n    globalFiltersStep,\n    scoringPipelinesStep,\n    resultSelectorsStep,\n    postSelectionFiltersStep,\n    decoratorStep,\n    resultSideEffectsStep,\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/recommendation/RecommendationPipelineResult.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.recommendation\n\nimport com.twitter.product_mixer.core.feature.featuremap.asyncfeaturemap.AsyncFeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineResult\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutorResults\nimport com.twitter.product_mixer.core.service.candidate_decorator_executor.CandidateDecoratorExecutorResult\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutorResult\nimport com.twitter.product_mixer.core.service.candidate_pipeline_executor.CandidatePipelineExecutorResult\nimport com.twitter.product_mixer.core.service.domain_marshaller_executor.DomainMarshallerExecutor\nimport com.twitter.product_mixer.core.service.filter_executor.FilterExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutorResult\nimport com.twitter.product_mixer.core.service.pipeline_result_side_effect_executor.PipelineResultSideEffectExecutor\nimport com.twitter.product_mixer.core.service.quality_factor_executor.QualityFactorExecutorResult\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.scoring_pipeline_executor.ScoringPipelineExecutorResult\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutorResult\nimport com.twitter.product_mixer.core.service.transport_marshaller_executor.TransportMarshallerExecutor\n\ncase class RecommendationPipelineResult[Candidate <: UniversalNoun[Any], ResultType](\n  qualityFactorResult: Option[QualityFactorExecutorResult],\n  gateResult: Option[GateExecutorResult],\n  queryFeatures: Option[QueryFeatureHydratorExecutor.Result],\n  queryFeaturesPhase2: Option[QueryFeatureHydratorExecutor.Result],\n  mergedAsyncQueryFeatures: Option[AsyncFeatureMap],\n  candidatePipelineResults: Option[CandidatePipelineExecutorResult],\n  dependentCandidatePipelineResults: Option[CandidatePipelineExecutorResult],\n  postCandidatePipelinesSelectorResults: Option[SelectorExecutorResult],\n  postCandidatePipelinesFeatureHydrationResults: Option[\n    CandidateFeatureHydratorExecutorResult[Candidate]\n  ],\n  globalFilterResults: Option[FilterExecutorResult[Candidate]],\n  scoringPipelineResults: Option[ScoringPipelineExecutorResult[Candidate]],\n  resultSelectorResults: Option[SelectorExecutorResult],\n  postSelectionFilterResults: Option[FilterExecutorResult[Candidate]],\n  candidateDecoratorResult: Option[CandidateDecoratorExecutorResult],\n  domainMarshallerResults: Option[DomainMarshallerExecutor.Result[HasMarshalling]],\n  resultSideEffectResults: Option[PipelineResultSideEffectExecutor.Result],\n  asyncFeatureHydrationResults: Option[AsyncFeatureMapExecutorResults],\n  transportMarshallerResults: Option[TransportMarshallerExecutor.Result[ResultType]],\n  failure: Option[PipelineFailure],\n  result: Option[ResultType])\n    extends PipelineResult[ResultType] {\n  override val resultSize: Int = result match {\n    case Some(seqResult @ Seq(_)) => seqResult.length\n    case Some(_) => 1\n    case None => 0\n  }\n\n  override def withFailure(\n    failure: PipelineFailure\n  ): RecommendationPipelineResult[Candidate, ResultType] =\n    copy(failure = Some(failure))\n  override def withResult(result: ResultType): RecommendationPipelineResult[Candidate, ResultType] =\n    copy(result = Some(result))\n}\n\nobject RecommendationPipelineResult {\n  def empty[A <: UniversalNoun[Any], B]: RecommendationPipelineResult[A, B] =\n    RecommendationPipelineResult(\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/scoring/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/selector:insert_append_results\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/candidate_feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n        \"util/util-core:scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/candidate_feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/scoring/NewScoringPipelineBuilder.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.scoring\n\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.functional_component.scorer.ScoredCandidateResult\nimport com.twitter.product_mixer.core.gate.ParamGate\nimport com.twitter.product_mixer.core.gate.ParamGate._\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ScoringPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.NewPipelineBuilder\nimport com.twitter.product_mixer.core.pipeline.NewPipelineArrowBuilder\nimport com.twitter.product_mixer.core.pipeline.NewPipelineResult\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ClosedGate\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureClassifier\nimport com.twitter.product_mixer.core.pipeline.state.HasCandidatesWithDetails\nimport com.twitter.product_mixer.core.pipeline.state.HasCandidatesWithFeatures\nimport com.twitter.product_mixer.core.pipeline.state.HasExecutorResults\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasResult\nimport com.twitter.product_mixer.core.pipeline.step.candidate_feature_hydrator.CandidateFeatureHydratorStep\nimport com.twitter.product_mixer.core.pipeline.step.gate.GateStep\nimport com.twitter.product_mixer.core.pipeline.step.scorer.ScorerStep\nimport com.twitter.product_mixer.core.pipeline.step.selector.SelectorStep\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.StoppedGateException\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\nimport scala.collection.immutable.ListMap\n\n/**\n * NewScoringPipelineBuilder builds [[ScoringPipeline]]s from [[ScoringPipelineConfig]]s.\n * New because it's meant to eventually replace [[ScoringPipelineBuilder]]\n * You should inject a [[ScoringPipelineBuilderFactory]] and call `.get` to build these.\n *\n * @see [[ScoringPipelineConfig]] for the description of the type parameters\n * @tparam Query the type of query these accept.\n * @tparam Candidate the domain model for the candidate being scored\n */\nclass NewScoringPipelineBuilder[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]] @Inject() (\n  selectionStep: SelectorStep[Query, ScoringPipelineState[Query, Candidate]],\n  gateStep: GateStep[Query, ScoringPipelineState[Query, Candidate]],\n  candidateFeatureHydrationStep: CandidateFeatureHydratorStep[\n    Query,\n    Candidate,\n    ScoringPipelineState[Query, Candidate]\n  ],\n  scorerStep: ScorerStep[Query, Candidate, ScoringPipelineState[Query, Candidate]])\n    extends NewPipelineBuilder[ScoringPipelineConfig[Query, Candidate], Seq[\n      CandidateWithFeatures[Candidate]\n    ], ScoringPipelineState[Query, Candidate], ScoringPipeline[Query, Candidate]] {\n\n  override def build(\n    parentComponentIdentifierStack: ComponentIdentifierStack,\n    arrowBuilder: NewPipelineArrowBuilder[ArrowResult, ArrowState],\n    scoringPipelineConfig: ScoringPipelineConfig[Query, Candidate]\n  ): ScoringPipeline[Query, Candidate] = {\n    val pipelineIdentifier = scoringPipelineConfig.identifier\n\n    val context = Executor.Context(\n      PipelineFailureClassifier(\n        scoringPipelineConfig.failureClassifier.orElse(\n          StoppedGateException.classifier(ClosedGate))),\n      parentComponentIdentifierStack.push(pipelineIdentifier)\n    )\n\n    val enabledGateOpt = scoringPipelineConfig.enabledDeciderParam.map { deciderParam =>\n      ParamGate(pipelineIdentifier + EnabledGateSuffix, deciderParam)\n    }\n    val supportedClientGateOpt = scoringPipelineConfig.supportedClientParam.map { param =>\n      ParamGate(pipelineIdentifier + SupportedClientGateSuffix, param)\n    }\n\n    /**\n     * Evaluate enabled decider gate first since if it's off, there is no reason to proceed\n     * Next evaluate supported client feature switch gate, followed by customer configured gates\n     */\n    val allGates =\n      enabledGateOpt.toSeq ++ supportedClientGateOpt.toSeq ++ scoringPipelineConfig.gates\n\n    val underlyingArrow = arrowBuilder\n      .add(ScoringPipelineConfig.gatesStep, gateStep, allGates)\n      .add(ScoringPipelineConfig.selectorsStep, selectionStep, scoringPipelineConfig.selectors)\n      .add(\n        ScoringPipelineConfig.preScoringFeatureHydrationPhase1Step,\n        candidateFeatureHydrationStep,\n        scoringPipelineConfig.preScoringFeatureHydrationPhase1)\n      .add(\n        ScoringPipelineConfig.preScoringFeatureHydrationPhase2Step,\n        candidateFeatureHydrationStep,\n        scoringPipelineConfig.preScoringFeatureHydrationPhase2)\n      .add(ScoringPipelineConfig.scorersStep, scorerStep, scoringPipelineConfig.scorers).buildArrow(\n        context)\n\n    val finalArrow = Arrow\n      .map { inputs: ScoringPipeline.Inputs[Query] =>\n        ScoringPipelineState[Query, Candidate](inputs.query, inputs.candidates, ListMap.empty)\n      }.andThen(underlyingArrow).map { pipelineResult =>\n        ScoringPipelineResult(\n          gateResults = pipelineResult.executorResultsByPipelineStep\n            .get(ScoringPipelineConfig.gatesStep)\n            .map(_.asInstanceOf[GateExecutorResult]),\n          selectorResults = pipelineResult.executorResultsByPipelineStep\n            .get(ScoringPipelineConfig.selectorsStep)\n            .map(_.asInstanceOf[SelectorExecutorResult]),\n          preScoringHydrationPhase1Result = pipelineResult.executorResultsByPipelineStep\n            .get(ScoringPipelineConfig.preScoringFeatureHydrationPhase1Step)\n            .map(_.asInstanceOf[CandidateFeatureHydratorExecutorResult[Candidate]]),\n          preScoringHydrationPhase2Result = pipelineResult.executorResultsByPipelineStep\n            .get(ScoringPipelineConfig.preScoringFeatureHydrationPhase2Step)\n            .map(_.asInstanceOf[CandidateFeatureHydratorExecutorResult[Candidate]]),\n          scorerResults = pipelineResult.executorResultsByPipelineStep\n            .get(ScoringPipelineConfig.scorersStep)\n            .map(_.asInstanceOf[CandidateFeatureHydratorExecutorResult[Candidate]]),\n          failure = pipelineResult match {\n            case failure: NewPipelineResult.Failure =>\n              Some(failure.failure)\n            case _ => None\n          },\n          result = pipelineResult match {\n            case result: NewPipelineResult.Success[Seq[CandidateWithFeatures[Candidate]]] =>\n              Some(result.result.map { candidateWithFeatures =>\n                ScoredCandidateResult(\n                  candidateWithFeatures.candidate,\n                  candidateWithFeatures.features)\n              })\n            case _ => None\n          }\n        )\n      }\n\n    new ScoringPipeline[Query, Candidate] {\n      override val arrow: Arrow[ScoringPipeline.Inputs[Query], ScoringPipelineResult[Candidate]] =\n        finalArrow\n\n      override val identifier: ScoringPipelineIdentifier = scoringPipelineConfig.identifier\n\n      override val alerts: Seq[Alert] = scoringPipelineConfig.alerts\n\n      override val children: Seq[Component] =\n        allGates ++ scoringPipelineConfig.preScoringFeatureHydrationPhase1 ++ scoringPipelineConfig.preScoringFeatureHydrationPhase2 ++ scoringPipelineConfig.scorers\n\n      override private[core] val config = scoringPipelineConfig\n    }\n  }\n}\n\ncase class ScoringPipelineState[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n  override val query: Query,\n  candidates: Seq[ItemCandidateWithDetails],\n  override val executorResultsByPipelineStep: ListMap[PipelineStepIdentifier, ExecutorResult])\n    extends HasQuery[Query, ScoringPipelineState[Query, Candidate]]\n    with HasCandidatesWithDetails[ScoringPipelineState[Query, Candidate]]\n    with HasCandidatesWithFeatures[Candidate, ScoringPipelineState[Query, Candidate]]\n    with HasExecutorResults[ScoringPipelineState[Query, Candidate]]\n    with HasResult[Seq[CandidateWithFeatures[Candidate]]] {\n\n  override val candidatesWithDetails: Seq[CandidateWithDetails] = candidates\n\n  override val candidatesWithFeatures: Seq[CandidateWithFeatures[Candidate]] =\n    candidates.asInstanceOf[Seq[CandidateWithFeatures[Candidate]]]\n\n  override val buildResult: Seq[CandidateWithFeatures[Candidate]] = candidatesWithFeatures\n\n  override def updateCandidatesWithDetails(\n    newCandidates: Seq[CandidateWithDetails]\n  ): ScoringPipelineState[Query, Candidate] = {\n    this.copy(candidates = newCandidates.asInstanceOf[Seq[ItemCandidateWithDetails]])\n  }\n\n  override def updateQuery(newQuery: Query): ScoringPipelineState[Query, Candidate] =\n    this.copy(query = newQuery)\n\n  override def updateDecorations(\n    decoration: Seq[Decoration]\n  ): ScoringPipelineState[Query, Candidate] = ???\n\n  override def updateCandidatesWithFeatures(\n    newCandidates: Seq[CandidateWithFeatures[Candidate]]\n  ): ScoringPipelineState[Query, Candidate] = {\n    val updatedCandidates = candidates.zip(newCandidates).map {\n      case (itemCandidateWithDetails, newCandidate) =>\n        itemCandidateWithDetails.copy(features =\n          itemCandidateWithDetails.features ++ newCandidate.features)\n    }\n    this.copy(query, updatedCandidates)\n  }\n\n  override private[pipeline] def setExecutorResults(\n    newMap: ListMap[PipelineStepIdentifier, ExecutorResult]\n  ) = this.copy(executorResultsByPipelineStep = newMap)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/scoring/ScoringPipeline.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.scoring\n\nimport com.twitter.product_mixer.core.functional_component.scorer.ScoredCandidateResult\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ScoringPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.Pipeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.stitch.Arrow\n\n/**\n * A Scoring Pipeline\n *\n * This is an abstract class, as we only construct these via the [[ScoringPipelineBuilder]].\n *\n * A [[ScoringPipeline]] is capable of pre-filtering candidates for scoring, performing the scoring\n * then running selection heuristics (ranking, dropping, etc) based off of the score.\n * @tparam Query the domain model for the query or request\n * @tparam Candidate the domain model for the candidate being scored\n */\nabstract class ScoringPipeline[-Query <: PipelineQuery, Candidate <: UniversalNoun[Any]]\n    extends Pipeline[ScoringPipeline.Inputs[Query], Seq[ScoredCandidateResult[Candidate]]] {\n  override private[core] val config: ScoringPipelineConfig[Query, Candidate]\n  override val arrow: Arrow[ScoringPipeline.Inputs[Query], ScoringPipelineResult[Candidate]]\n  override val identifier: ScoringPipelineIdentifier\n}\n\nobject ScoringPipeline {\n  case class Inputs[+Query <: PipelineQuery](\n    query: Query,\n    candidates: Seq[ItemCandidateWithDetails])\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/scoring/ScoringPipelineBuilder.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.scoring\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.scorer.ScoredCandidateResult\nimport com.twitter.product_mixer.core.gate.ParamGate\nimport com.twitter.product_mixer.core.gate.ParamGate.EnabledGateSuffix\nimport com.twitter.product_mixer.core.gate.ParamGate.SupportedClientGateSuffix\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.common.identifier.ScoringPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.InvalidStepStateException\nimport com.twitter.product_mixer.core.pipeline.PipelineBuilder\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ClosedGate\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureClassifier\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipeline.Inputs\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.StoppedGateException\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutor\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * ScoringPipelineBuilder builds [[ScoringPipeline]]s from [[ScoringPipelineConfig]]s.\n *\n * You should inject a [[ScoringPipelineBuilderFactory]] and call `.get` to build these.\n *\n * @see [[ScoringPipelineConfig]] for the description of the type parameters\n * @tparam Query the type of query these accept.\n * @tparam Candidate the domain model for the candidate being scored\n */\nclass ScoringPipelineBuilder[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]] @Inject() (\n  gateExecutor: GateExecutor,\n  selectorExecutor: SelectorExecutor,\n  candidateFeatureHydratorExecutor: CandidateFeatureHydratorExecutor,\n  override val statsReceiver: StatsReceiver)\n    extends PipelineBuilder[Inputs[Query]] {\n\n  override type UnderlyingResultType = Seq[ScoredCandidateResult[Candidate]]\n  override type PipelineResultType = ScoringPipelineResult[Candidate]\n\n  def build(\n    parentComponentIdentifierStack: ComponentIdentifierStack,\n    config: ScoringPipelineConfig[Query, Candidate]\n  ): ScoringPipeline[Query, Candidate] = {\n\n    val pipelineIdentifier = config.identifier\n\n    val context = Executor.Context(\n      PipelineFailureClassifier(\n        config.failureClassifier.orElse(StoppedGateException.classifier(ClosedGate))),\n      parentComponentIdentifierStack.push(pipelineIdentifier)\n    )\n\n    val enabledGateOpt = config.enabledDeciderParam.map { deciderParam =>\n      ParamGate(pipelineIdentifier + EnabledGateSuffix, deciderParam)\n    }\n    val supportedClientGateOpt = config.supportedClientParam.map { param =>\n      ParamGate(pipelineIdentifier + SupportedClientGateSuffix, param)\n    }\n\n    /**\n     * Evaluate enabled decider gate first since if it's off, there is no reason to proceed\n     * Next evaluate supported client feature switch gate, followed by customer configured gates\n     */\n    val allGates = enabledGateOpt.toSeq ++ supportedClientGateOpt.toSeq ++ config.gates\n\n    val GatesStep = new Step[Query, GateExecutorResult] {\n      override def identifier: PipelineStepIdentifier = ScoringPipelineConfig.gatesStep\n\n      override lazy val executorArrow: Arrow[Query, GateExecutorResult] =\n        gateExecutor.arrow(allGates, context)\n\n      override def inputAdaptor(\n        query: ScoringPipeline.Inputs[Query],\n        previousResult: ScoringPipelineResult[Candidate]\n      ): Query = {\n        query.query\n      }\n\n      override def resultUpdater(\n        previousPipelineResult: ScoringPipelineResult[Candidate],\n        executorResult: GateExecutorResult\n      ): ScoringPipelineResult[Candidate] =\n        previousPipelineResult.copy(gateResults = Some(executorResult))\n    }\n\n    val SelectorsStep = new Step[SelectorExecutor.Inputs[Query], SelectorExecutorResult] {\n      override def identifier: PipelineStepIdentifier = ScoringPipelineConfig.selectorsStep\n\n      override def executorArrow: Arrow[SelectorExecutor.Inputs[Query], SelectorExecutorResult] =\n        selectorExecutor.arrow(config.selectors, context)\n\n      override def inputAdaptor(\n        query: ScoringPipeline.Inputs[Query],\n        previousResult: ScoringPipelineResult[Candidate]\n      ): SelectorExecutor.Inputs[Query] = SelectorExecutor.Inputs(query.query, query.candidates)\n\n      override def resultUpdater(\n        previousPipelineResult: ScoringPipelineResult[Candidate],\n        executorResult: SelectorExecutorResult\n      ): ScoringPipelineResult[Candidate] =\n        previousPipelineResult.copy(selectorResults = Some(executorResult))\n    }\n\n    val PreScoringFeatureHydrationPhase1Step =\n      new Step[\n        CandidateFeatureHydratorExecutor.Inputs[Query, Candidate],\n        CandidateFeatureHydratorExecutorResult[Candidate]\n      ] {\n        override def identifier: PipelineStepIdentifier =\n          ScoringPipelineConfig.preScoringFeatureHydrationPhase1Step\n\n        override def executorArrow: Arrow[\n          CandidateFeatureHydratorExecutor.Inputs[Query, Candidate],\n          CandidateFeatureHydratorExecutorResult[Candidate]\n        ] =\n          candidateFeatureHydratorExecutor.arrow(config.preScoringFeatureHydrationPhase1, context)\n\n        override def inputAdaptor(\n          query: ScoringPipeline.Inputs[Query],\n          previousResult: ScoringPipelineResult[Candidate]\n        ): CandidateFeatureHydratorExecutor.Inputs[Query, Candidate] = {\n          val selectedCandidatesResult = previousResult.selectorResults.getOrElse {\n            throw InvalidStepStateException(identifier, \"SelectorResults\")\n          }.selectedCandidates\n\n          CandidateFeatureHydratorExecutor.Inputs(\n            query.query,\n            selectedCandidatesResult.asInstanceOf[Seq[CandidateWithFeatures[Candidate]]])\n        }\n\n        override def resultUpdater(\n          previousPipelineResult: ScoringPipelineResult[Candidate],\n          executorResult: CandidateFeatureHydratorExecutorResult[Candidate]\n        ): ScoringPipelineResult[Candidate] = previousPipelineResult.copy(\n          preScoringHydrationPhase1Result = Some(executorResult)\n        )\n      }\n\n    val PreScoringFeatureHydrationPhase2Step =\n      new Step[\n        CandidateFeatureHydratorExecutor.Inputs[Query, Candidate],\n        CandidateFeatureHydratorExecutorResult[Candidate]\n      ] {\n        override def identifier: PipelineStepIdentifier =\n          ScoringPipelineConfig.preScoringFeatureHydrationPhase2Step\n\n        override def executorArrow: Arrow[\n          CandidateFeatureHydratorExecutor.Inputs[Query, Candidate],\n          CandidateFeatureHydratorExecutorResult[Candidate]\n        ] =\n          candidateFeatureHydratorExecutor.arrow(config.preScoringFeatureHydrationPhase2, context)\n\n        override def inputAdaptor(\n          query: ScoringPipeline.Inputs[Query],\n          previousResult: ScoringPipelineResult[Candidate]\n        ): CandidateFeatureHydratorExecutor.Inputs[Query, Candidate] = {\n          val preScoringHydrationPhase1FeatureMaps: Seq[FeatureMap] =\n            previousResult.preScoringHydrationPhase1Result\n              .getOrElse(\n                throw InvalidStepStateException(identifier, \"PreScoringHydrationPhase1Result\"))\n              .results.map(_.features)\n\n          val itemCandidates = previousResult.selectorResults\n            .getOrElse(throw InvalidStepStateException(identifier, \"SelectionResults\"))\n            .selectedCandidates.collect {\n              case itemCandidate: ItemCandidateWithDetails => itemCandidate\n            }\n          // If there is no feature hydration (empty results), no need to attempt merging.\n          val candidates = if (preScoringHydrationPhase1FeatureMaps.isEmpty) {\n            itemCandidates\n          } else {\n            itemCandidates.zip(preScoringHydrationPhase1FeatureMaps).map {\n              case (itemCandidate, featureMap) =>\n                itemCandidate.copy(features = itemCandidate.features ++ featureMap)\n            }\n          }\n\n          CandidateFeatureHydratorExecutor.Inputs(\n            query.query,\n            candidates.asInstanceOf[Seq[CandidateWithFeatures[Candidate]]])\n        }\n\n        override def resultUpdater(\n          previousPipelineResult: ScoringPipelineResult[Candidate],\n          executorResult: CandidateFeatureHydratorExecutorResult[Candidate]\n        ): ScoringPipelineResult[Candidate] = previousPipelineResult.copy(\n          preScoringHydrationPhase2Result = Some(executorResult)\n        )\n      }\n\n    def getMergedPreScoringFeatureMap(\n      stepIdentifier: PipelineStepIdentifier,\n      previousResult: ScoringPipelineResult[Candidate]\n    ): Seq[FeatureMap] = {\n      val preScoringHydrationPhase1FeatureMaps: Seq[FeatureMap] =\n        previousResult.preScoringHydrationPhase1Result\n          .getOrElse(\n            throw InvalidStepStateException(\n              stepIdentifier,\n              \"PreScoringHydrationPhase1Result\")).results.map(_.features)\n\n      val preScoringHydrationPhase2FeatureMaps: Seq[FeatureMap] =\n        previousResult.preScoringHydrationPhase2Result\n          .getOrElse(\n            throw InvalidStepStateException(\n              stepIdentifier,\n              \"PreScoringHydrationPhase2Result\")).results.map(_.features)\n      /*\n       * If either pre-scoring hydration phase feature map is empty, no need to merge them,\n       * we can just take all non-empty ones.\n       */\n      if (preScoringHydrationPhase1FeatureMaps.isEmpty) {\n        preScoringHydrationPhase2FeatureMaps\n      } else if (preScoringHydrationPhase2FeatureMaps.isEmpty) {\n        preScoringHydrationPhase1FeatureMaps\n      } else {\n        // No need to check the size in both, since the inputs to both hydration phases are the\n        // same and each phase ensures the number of candidates and ordering matches the input.\n        preScoringHydrationPhase1FeatureMaps.zip(preScoringHydrationPhase2FeatureMaps).map {\n          case (preScoringHydrationPhase1FeatureMap, preScoringHydrationPhasesFeatureMap) =>\n            preScoringHydrationPhase1FeatureMap ++ preScoringHydrationPhasesFeatureMap\n        }\n      }\n    }\n\n    val ScorersStep =\n      new Step[\n        CandidateFeatureHydratorExecutor.Inputs[Query, Candidate],\n        CandidateFeatureHydratorExecutorResult[Candidate]\n      ] {\n        override def identifier: PipelineStepIdentifier = ScoringPipelineConfig.scorersStep\n\n        override def inputAdaptor(\n          query: ScoringPipeline.Inputs[Query],\n          previousResult: ScoringPipelineResult[Candidate]\n        ): CandidateFeatureHydratorExecutor.Inputs[Query, Candidate] = {\n\n          val mergedPreScoringFeatureHydrationFeatures: Seq[FeatureMap] =\n            getMergedPreScoringFeatureMap(ScoringPipelineConfig.scorersStep, previousResult)\n\n          val itemCandidates = previousResult.selectorResults\n            .getOrElse(throw InvalidStepStateException(identifier, \"SelectionResults\"))\n            .selectedCandidates.collect {\n              case itemCandidate: ItemCandidateWithDetails => itemCandidate\n            }\n\n          // If there was no pre-scoring features hydration, no need to re-merge feature maps\n          // and construct a new item candidate\n          val updatedCandidates = if (mergedPreScoringFeatureHydrationFeatures.isEmpty) {\n            itemCandidates\n          } else {\n            itemCandidates.zip(mergedPreScoringFeatureHydrationFeatures).map {\n              case (itemCandidate, preScoringFeatureMap) =>\n                itemCandidate.copy(features = itemCandidate.features ++ preScoringFeatureMap)\n            }\n          }\n          CandidateFeatureHydratorExecutor.Inputs(\n            query.query,\n            updatedCandidates.asInstanceOf[Seq[CandidateWithFeatures[Candidate]]])\n        }\n\n        override lazy val executorArrow: Arrow[\n          CandidateFeatureHydratorExecutor.Inputs[Query, Candidate],\n          CandidateFeatureHydratorExecutorResult[\n            Candidate\n          ]\n        ] = candidateFeatureHydratorExecutor.arrow(config.scorers.toSeq, context)\n\n        override def resultUpdater(\n          previousPipelineResult: ScoringPipelineResult[Candidate],\n          executorResult: CandidateFeatureHydratorExecutorResult[Candidate]\n        ): ScoringPipelineResult[Candidate] =\n          previousPipelineResult.copy(scorerResults = Some(executorResult))\n      }\n\n    val ResultStep =\n      new Step[Seq[CandidateWithFeatures[UniversalNoun[Any]]], Seq[\n        CandidateWithFeatures[UniversalNoun[Any]]\n      ]] {\n        override def identifier: PipelineStepIdentifier = ScoringPipelineConfig.resultStep\n\n        override def executorArrow: Arrow[Seq[CandidateWithFeatures[UniversalNoun[Any]]], Seq[\n          CandidateWithFeatures[UniversalNoun[Any]]\n        ]] = Arrow.identity\n\n        override def inputAdaptor(\n          query: Inputs[Query],\n          previousResult: ScoringPipelineResult[Candidate]\n        ): Seq[CandidateWithFeatures[UniversalNoun[Any]]] = previousResult.selectorResults\n          .getOrElse(throw InvalidStepStateException(identifier, \"SelectionResults\"))\n          .selectedCandidates.collect {\n            case itemCandidate: ItemCandidateWithDetails => itemCandidate\n          }\n\n        override def resultUpdater(\n          previousPipelineResult: ScoringPipelineResult[Candidate],\n          executorResult: Seq[CandidateWithFeatures[UniversalNoun[Any]]]\n        ): ScoringPipelineResult[Candidate] = {\n          val scorerResults: Seq[FeatureMap] = previousPipelineResult.scorerResults\n            .getOrElse(throw InvalidStepStateException(identifier, \"ScorerResult\")).results.map(\n              _.features)\n\n          val mergedPreScoringFeatureHydrationFeatureMaps: Seq[FeatureMap] =\n            getMergedPreScoringFeatureMap(ScoringPipelineConfig.resultStep, previousPipelineResult)\n\n          val itemCandidates = executorResult.asInstanceOf[Seq[ItemCandidateWithDetails]]\n          val finalFeatureMap = if (mergedPreScoringFeatureHydrationFeatureMaps.isEmpty) {\n            scorerResults\n          } else {\n            scorerResults\n              .zip(mergedPreScoringFeatureHydrationFeatureMaps).map {\n                case (preScoringFeatureMap, scoringFeatureMap) =>\n                  preScoringFeatureMap ++ scoringFeatureMap\n              }\n          }\n\n          val finalResults = itemCandidates.zip(finalFeatureMap).map {\n            case (itemCandidate, featureMap) =>\n              ScoredCandidateResult(itemCandidate.candidate.asInstanceOf[Candidate], featureMap)\n          }\n          previousPipelineResult.withResult(finalResults)\n        }\n      }\n\n    val builtSteps = Seq(\n      GatesStep,\n      SelectorsStep,\n      PreScoringFeatureHydrationPhase1Step,\n      PreScoringFeatureHydrationPhase2Step,\n      ScorersStep,\n      ResultStep\n    )\n\n    /** The main execution logic for this Candidate Pipeline. */\n    val finalArrow: Arrow[ScoringPipeline.Inputs[Query], ScoringPipelineResult[Candidate]] =\n      buildCombinedArrowFromSteps(\n        steps = builtSteps,\n        context = context,\n        initialEmptyResult = ScoringPipelineResult.empty,\n        stepsInOrderFromConfig = ScoringPipelineConfig.stepsInOrder\n      )\n\n    val configFromBuilder = config\n    new ScoringPipeline[Query, Candidate] {\n      override private[core] val config: ScoringPipelineConfig[Query, Candidate] = configFromBuilder\n      override val arrow: Arrow[ScoringPipeline.Inputs[Query], ScoringPipelineResult[Candidate]] =\n        finalArrow\n      override val identifier: ScoringPipelineIdentifier = pipelineIdentifier\n      override val alerts: Seq[Alert] = config.alerts\n      override val children: Seq[Component] =\n        allGates ++ config.preScoringFeatureHydrationPhase1 ++ config.preScoringFeatureHydrationPhase2 ++ config.scorers\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/scoring/ScoringPipelineBuilderFactory.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.scoring\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutor\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutor\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ScoringPipelineBuilderFactory @Inject() (\n  gateExecutor: GateExecutor,\n  selectorExecutor: SelectorExecutor,\n  candidateFeatureHydratorExecutor: CandidateFeatureHydratorExecutor,\n  statsReceiver: StatsReceiver) {\n\n  def get[\n    Query <: PipelineQuery,\n    Candidate <: UniversalNoun[Any]\n  ]: ScoringPipelineBuilder[Query, Candidate] = {\n    new ScoringPipelineBuilder[Query, Candidate](\n      gateExecutor,\n      selectorExecutor,\n      candidateFeatureHydratorExecutor,\n      statsReceiver\n    )\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/scoring/ScoringPipelineConfig.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.scoring\n\nimport com.twitter.product_mixer.component_library.selector.InsertAppendResults\nimport com.twitter.product_mixer.core.functional_component.common.AllPipelines\nimport com.twitter.product_mixer.core.functional_component.common.alert.Alert\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.common.identifier.ScoringPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineConfig\nimport com.twitter.product_mixer.core.pipeline.PipelineConfigCompanion\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.decider.DeciderParam\n\n/**\n *  This is the configuration necessary to generate a Scoring Pipeline. Product code should create a\n *  ScoringPipelineConfig, and then use a ScoringPipelineBuilder to get the final ScoringPipeline which can\n *  process requests.\n *\n * @tparam Query - The domain model for the query or request\n * @tparam Candidate the domain model for the candidate being scored\n */\ntrait ScoringPipelineConfig[-Query <: PipelineQuery, Candidate <: UniversalNoun[Any]]\n    extends PipelineConfig {\n\n  override val identifier: ScoringPipelineIdentifier\n\n  /**\n   * When these Params are defined, they will automatically be added as Gates in the pipeline\n   * by the CandidatePipelineBuilder\n   *\n   * The enabled decider param can to be used to quickly disable a Candidate Pipeline via Decider\n   */\n  val enabledDeciderParam: Option[DeciderParam[Boolean]] = None\n\n  /**\n   * This supported client feature switch param can be used with a Feature Switch to control the\n   * rollout of a new Candidate Pipeline from dogfood to experiment to production\n   */\n  val supportedClientParam: Option[FSParam[Boolean]] = None\n\n  /** [[BaseGate]]s that are applied sequentially, the pipeline will only run if all the Gates are open */\n  def gates: Seq[BaseGate[Query]] = Seq.empty\n\n  /**\n   * Logic for selecting which candidates to score. Note, this doesn't drop the candidates from\n   * the final result, just whether to score it in this pipeline or not.\n   */\n  def selectors: Seq[Selector[Query]]\n\n  /**\n   * After selectors are run, you can fetch features for each candidate.\n   * The existing features from previous hydrations are passed in as inputs. You are not expected to\n   * put them into the resulting feature map yourself - they will be merged for you by the platform.\n   */\n  def preScoringFeatureHydrationPhase1: Seq[BaseCandidateFeatureHydrator[Query, Candidate, _]] =\n    Seq.empty\n\n  /**\n   * A second phase of feature hydration that can be run after selection and after the first phase\n   * of pre-scoring feature hydration. You are not expected to put them into the resulting\n   * feature map yourself - they will be merged for you by the platform.\n   */\n  def preScoringFeatureHydrationPhase2: Seq[BaseCandidateFeatureHydrator[Query, Candidate, _]] =\n    Seq.empty\n\n  /**\n   * Ranker Function for candidates. Scorers are executed in parallel.\n   * Note: Order does not matter, this could be a Set if Set was covariant over it's type.\n   */\n  def scorers: Seq[Scorer[Query, Candidate]]\n\n  /**\n   * A pipeline can define a partial function to rescue failures here. They will be treated as failures\n   * from a monitoring standpoint, and cancellation exceptions will always be propagated (they cannot be caught here).\n   */\n  def failureClassifier: PartialFunction[Throwable, PipelineFailure] = PartialFunction.empty\n\n  /**\n   * Alerts can be used to indicate the pipeline's service level objectives. Alerts and\n   * dashboards will be automatically created based on this information.\n   */\n  val alerts: Seq[Alert] = Seq.empty\n\n  /**\n   * This method is used by the product mixer framework to build the pipeline.\n   */\n  private[core] final def build(\n    parentComponentIdentifierStack: ComponentIdentifierStack,\n    builder: ScoringPipelineBuilderFactory\n  ): ScoringPipeline[Query, Candidate] =\n    builder.get.build(parentComponentIdentifierStack, this)\n}\n\nobject ScoringPipelineConfig extends PipelineConfigCompanion {\n  def apply[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n    scorer: Scorer[Query, Candidate]\n  ): ScoringPipelineConfig[Query, Candidate] = new ScoringPipelineConfig[Query, Candidate] {\n    override val identifier: ScoringPipelineIdentifier = ScoringPipelineIdentifier(\n      s\"ScoreAll${scorer.identifier.name}\")\n\n    override val selectors: Seq[Selector[Query]] = Seq(InsertAppendResults(AllPipelines))\n\n    override val scorers: Seq[Scorer[Query, Candidate]] = Seq(scorer)\n  }\n\n  val gatesStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Gates\")\n  val selectorsStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Selectors\")\n  val preScoringFeatureHydrationPhase1Step: PipelineStepIdentifier =\n    PipelineStepIdentifier(\"PreScoringFeatureHydrationPhase1\")\n  val preScoringFeatureHydrationPhase2Step: PipelineStepIdentifier =\n    PipelineStepIdentifier(\"PreScoringFeatureHydrationPhase2\")\n  val scorersStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Scorers\")\n  val resultStep: PipelineStepIdentifier = PipelineStepIdentifier(\"Result\")\n\n  /** All the Steps which are executed by a [[ScoringPipeline]] in the order in which they are run */\n  override val stepsInOrder: Seq[PipelineStepIdentifier] = Seq(\n    gatesStep,\n    selectorsStep,\n    preScoringFeatureHydrationPhase1Step,\n    preScoringFeatureHydrationPhase2Step,\n    scorersStep,\n    resultStep\n  )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/scoring/ScoringPipelineResult.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.scoring\n\nimport com.twitter.product_mixer.core.functional_component.scorer.ScoredCandidateResult\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineResult\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutorResult\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutorResult\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutorResult\n\n/**\n * The Results of every step during the ScoringPipeline process. The end result contains\n * only the candidates that were actually scored (e.g, not dropped by a filter) with an updated,\n * combined feature map of all features that were passed in with the candidate plus all features\n * returned as part of scoring.\n */\ncase class ScoringPipelineResult[Candidate <: UniversalNoun[Any]](\n  gateResults: Option[GateExecutorResult],\n  selectorResults: Option[SelectorExecutorResult],\n  preScoringHydrationPhase1Result: Option[CandidateFeatureHydratorExecutorResult[Candidate]],\n  preScoringHydrationPhase2Result: Option[CandidateFeatureHydratorExecutorResult[Candidate]],\n  scorerResults: Option[CandidateFeatureHydratorExecutorResult[\n    Candidate\n  ]],\n  failure: Option[PipelineFailure],\n  result: Option[Seq[ScoredCandidateResult[Candidate]]])\n    extends PipelineResult[Seq[ScoredCandidateResult[Candidate]]] {\n  override val resultSize: Int = result.map(_.size).getOrElse(0)\n\n  override def withFailure(\n    failure: PipelineFailure\n  ): ScoringPipelineResult[Candidate] =\n    copy(failure = Some(failure))\n  override def withResult(\n    result: Seq[ScoredCandidateResult[Candidate]]\n  ): ScoringPipelineResult[Candidate] =\n    copy(result = Some(result))\n}\n\nobject ScoringPipelineResult {\n  def empty[Candidate <: UniversalNoun[Any]]: ScoringPipelineResult[Candidate] =\n    ScoringPipelineResult(\n      None,\n      None,\n      None,\n      None,\n      None,\n      None,\n      None\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state/BUILD",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/asyncfeaturemap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator:decoration\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/asyncfeaturemap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator:decoration\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state/HasAsyncFeatureMap.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.state\n\nimport com.twitter.product_mixer.core.feature.featuremap.asyncfeaturemap.AsyncFeatureMap\n\ntrait HasAsyncFeatureMap[State] {\n  def asyncFeatureMap: AsyncFeatureMap\n\n  private[core] def addAsyncFeatureMap(newFeatureMap: AsyncFeatureMap): State\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state/HasCandidates.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.state\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\ntrait HasCandidates[Candidate <: UniversalNoun[Any], T] {\n  def candidates: Seq[Candidate]\n  def updateCandidates(newCandidates: Seq[Candidate]): T\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state/HasCandidatesWithDetails.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.state\n\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\n\ntrait HasCandidatesWithDetails[T] {\n  def candidatesWithDetails: Seq[CandidateWithDetails]\n  def updateCandidatesWithDetails(newCandidates: Seq[CandidateWithDetails]): T\n\n  def updateDecorations(decoration: Seq[Decoration]): T\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state/HasCandidatesWithFeatures.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.state\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\n\ntrait HasCandidatesWithFeatures[Candidate <: UniversalNoun[Any], T] {\n  def candidatesWithFeatures: Seq[CandidateWithFeatures[Candidate]]\n  def updateCandidatesWithFeatures(newCandidates: Seq[CandidateWithFeatures[Candidate]]): T\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state/HasExecutorResults.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.state\n\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport scala.collection.immutable.ListMap\n\ntrait HasExecutorResults[State] {\n  // We use a list map to maintain the insertion order\n  val executorResultsByPipelineStep: ListMap[PipelineStepIdentifier, ExecutorResult]\n  private[pipeline] def setExecutorResults(\n    newMap: ListMap[PipelineStepIdentifier, ExecutorResult]\n  ): State\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state/HasParams.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.state\n\nimport com.twitter.timelines.configapi.Params\n\ntrait HasParams {\n  def params: Params\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state/HasQuery.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.state\n\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait HasQuery[Query <: PipelineQuery, T] {\n  def query: Query\n  def updateQuery(query: Query): T\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state/HasRequest.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.state\n\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\n\ntrait HasRequest[TRequest <: Request] {\n  def request: TRequest\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state/HasResult.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.state\n\n/**\n * Defines how to build a result from a pipeline state. Pipeline States should extend this and\n * implement [[buildResult]] which computes the final result from their current state.\n * @tparam Result Type of result\n */\ntrait HasResult[+Result] {\n  def buildResult(): Result\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/BUILD",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/Step.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step\n\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport com.twitter.stitch.Arrow\n\n/**\n * A Step within a Pipeline, a Step is a unitary phase within an entire chain that makes up a pipeline.\n * @tparam State The request domain model.\n * @tparam ExecutorConfig The configs that should be passed into the executor at build time.\n * @tparam ExecutorInput The input type that an executor takes at request time.\n * @tparam ExResult The result that a step's executor will return.\n * @tparam OutputState The final/updated state a step would output, this is typically taking the ExResult\n *                     and mutating/transforming the State.\n */\ntrait Step[State, -Config, ExecutorInput, ExResult <: ExecutorResult] {\n\n  /**\n   * Adapt the state into the expected input for the Step's Arrow.\n   *\n   * @param state State object passed into the Step.\n   * @param config The config object used to build the executor arrow or input.\n   * @return ExecutorInput that is used in the arrow of the underlying executor.\n   */\n  def adaptInput(state: State, config: Config): ExecutorInput\n\n  /**\n   * The actual arrow to be executed for the step, taking in the adapted input from [[adaptInput]]\n   * and returning the expected result.\n   * @param config Runtime configurations to configure the arrow with.\n   * @param context Context of Executor.\n   */\n  def arrow(\n    config: Config,\n    context: Executor.Context\n  ): Arrow[ExecutorInput, ExResult]\n\n  /**\n   * Whether the step is considered a noop/empty based off of input being passed in. Empty\n   * steps are skipped when being executed.\n   */\n  def isEmpty(config: Config): Boolean\n\n  /**\n   * Update the passed in state based off of the result from [[arrow]]\n   * @param state State object passed into the Step.\n   * @param executorResult Executor result returned from [[arrow]]\n   * @param config The config object used to build the executor arrow or input.\n   * @return Updated state object passed.\n   */\n  def updateState(state: State, executorResult: ExResult, config: Config): State\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/async_feature_map/AsyncFeatureMapStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.async_feature_map\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.asyncfeaturemap.AsyncFeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasAsyncFeatureMap\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutor\nimport com.twitter.product_mixer.core.service.async_feature_map_executor.AsyncFeatureMapExecutorResults\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * Async Feature Hydrator Step, it takes an existing asyn feature map and executes any hydration\n * needed before the next step. The state object is responsible for keeping the updated query\n * with the updated feature map.\n *\n * @param asyncFeatureMapExecutor Async feature map executor\n *\n * @tparam Query Type of PipelineQuery domain model\n * @tparam State The pipeline state domain model.\n */\ncase class AsyncFeatureMapStep[\n  Query <: PipelineQuery,\n  State <: HasQuery[Query, State] with HasAsyncFeatureMap[State]] @Inject() (\n  asyncFeatureMapExecutor: AsyncFeatureMapExecutor)\n    extends Step[\n      State,\n      AsyncFeatureMapStepConfig,\n      AsyncFeatureMap,\n      AsyncFeatureMapExecutorResults\n    ] {\n  override def isEmpty(config: AsyncFeatureMapStepConfig): Boolean = false\n\n  override def adaptInput(\n    state: State,\n    config: AsyncFeatureMapStepConfig\n  ): AsyncFeatureMap = state.asyncFeatureMap\n\n  override def arrow(\n    config: AsyncFeatureMapStepConfig,\n    context: Executor.Context\n  ): Arrow[AsyncFeatureMap, AsyncFeatureMapExecutorResults] =\n    asyncFeatureMapExecutor.arrow(config.stepToHydrateFor, config.currentStep, context)\n\n  override def updateState(\n    state: State,\n    executorResult: AsyncFeatureMapExecutorResults,\n    config: AsyncFeatureMapStepConfig\n  ): State = {\n    val hydratedFeatureMap =\n      executorResult.featureMapsByStep.getOrElse(config.stepToHydrateFor, FeatureMap.empty)\n    if (hydratedFeatureMap.isEmpty) {\n      state\n    } else {\n      val updatedFeatureMap = state.query.features\n        .getOrElse(FeatureMap.empty) ++ hydratedFeatureMap\n      state.updateQuery(\n        state.query\n          .withFeatureMap(updatedFeatureMap).asInstanceOf[Query])\n    }\n  }\n}\n\ncase class AsyncFeatureMapStepConfig(\n  stepToHydrateFor: PipelineStepIdentifier,\n  currentStep: PipelineStepIdentifier)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/async_feature_map/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/asyncfeaturemap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/async_feature_map_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/asyncfeaturemap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/async_feature_map_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/candidate_feature_hydrator/BUILD",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_hydrator_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_hydrator_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/candidate_feature_hydrator/CandidateFeatureHydratorStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.candidate_feature_hydrator\n\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasCandidatesWithFeatures\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * A candidate level feature hydration step, it takes the input list of candidates and the given\n * hydrators and executes them. The [[State]] object is responsible for merging the resulting\n * feature maps with the hydrated ones in its updateCandidatesWithFeatures.\n *\n * @param candidateFeatureHydratorExecutor Hydrator Executor\n * @tparam Query Type of PipelineQuery domain model\n * @tparam Candidate Type of Candidates to hydrate features for.\n * @tparam State The pipeline state domain model.\n */\ncase class CandidateFeatureHydratorStep[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  State <: HasQuery[Query, State] with HasCandidatesWithFeatures[\n    Candidate,\n    State\n  ]] @Inject() (\n  candidateFeatureHydratorExecutor: CandidateFeatureHydratorExecutor)\n    extends Step[State, Seq[\n      BaseCandidateFeatureHydrator[Query, Candidate, _]\n    ], CandidateFeatureHydratorExecutor.Inputs[\n      Query,\n      Candidate\n    ], CandidateFeatureHydratorExecutorResult[Candidate]] {\n\n  override def adaptInput(\n    state: State,\n    config: Seq[BaseCandidateFeatureHydrator[Query, Candidate, _]]\n  ): CandidateFeatureHydratorExecutor.Inputs[Query, Candidate] =\n    CandidateFeatureHydratorExecutor.Inputs(state.query, state.candidatesWithFeatures)\n\n  override def arrow(\n    config: Seq[BaseCandidateFeatureHydrator[Query, Candidate, _]],\n    context: Executor.Context\n  ): Arrow[\n    CandidateFeatureHydratorExecutor.Inputs[Query, Candidate],\n    CandidateFeatureHydratorExecutorResult[Candidate]\n  ] = candidateFeatureHydratorExecutor.arrow(config, context)\n\n  override def updateState(\n    input: State,\n    executorResult: CandidateFeatureHydratorExecutorResult[Candidate],\n    config: Seq[BaseCandidateFeatureHydrator[Query, Candidate, _]]\n  ): State = {\n    val candidatesWithHydratedFeatures = executorResult.results\n    if (candidatesWithHydratedFeatures.isEmpty) {\n      input\n    } else {\n      input.updateCandidatesWithFeatures(candidatesWithHydratedFeatures)\n    }\n  }\n\n  override def isEmpty(\n    config: Seq[BaseCandidateFeatureHydrator[Query, Candidate, _]]\n  ): Boolean =\n    config.isEmpty\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/candidate_source/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_source_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_source_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/candidate_source/CandidateSourceStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.candidate_source\n\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.transformer.BaseCandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasCandidatesWithFeatures\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.candidate_source_executor.CandidateSourceExecutor\nimport com.twitter.product_mixer.core.service.candidate_source_executor.CandidateSourceExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * A candidate source step, which takes the query and gets csandidates from the candidate source.\n *\n * @param candidateSourceExecutor Candidate Source Executor\n * @tparam Query Type of PipelineQuery domain model\n * @tparam Candidate Type of Candidates to filter\n * @tparam State The pipeline state domain model.\n */\ncase class CandidateSourceStep[\n  Query <: PipelineQuery,\n  CandidateSourceQuery,\n  CandidateSourceResult,\n  Candidate <: UniversalNoun[Any],\n  State <: HasQuery[Query, State] with HasCandidatesWithFeatures[Candidate, State]] @Inject() (\n  candidateSourceExecutor: CandidateSourceExecutor)\n    extends Step[\n      State,\n      CandidateSourceConfig[Query, CandidateSourceQuery, CandidateSourceResult, Candidate],\n      Query,\n      CandidateSourceExecutorResult[\n        Candidate\n      ]\n    ] {\n  override def isEmpty(\n    config: CandidateSourceConfig[Query, CandidateSourceQuery, CandidateSourceResult, Candidate]\n  ): Boolean = false\n\n  override def adaptInput(\n    state: State,\n    config: CandidateSourceConfig[Query, CandidateSourceQuery, CandidateSourceResult, Candidate]\n  ): Query = state.query\n\n  override def arrow(\n    config: CandidateSourceConfig[Query, CandidateSourceQuery, CandidateSourceResult, Candidate],\n    context: Executor.Context\n  ): Arrow[Query, CandidateSourceExecutorResult[Candidate]] = candidateSourceExecutor.arrow(\n    config.candidateSource,\n    config.queryTransformer,\n    config.resultTransformer,\n    config.resultFeaturesTransformers,\n    context\n  )\n\n  override def updateState(\n    state: State,\n    executorResult: CandidateSourceExecutorResult[Candidate],\n    config: CandidateSourceConfig[Query, CandidateSourceQuery, CandidateSourceResult, Candidate]\n  ): State = state\n    .updateQuery(\n      state.query\n        .withFeatureMap(executorResult.candidateSourceFeatureMap).asInstanceOf[\n          Query]).updateCandidatesWithFeatures(executorResult.candidates)\n}\n\ncase class CandidateSourceConfig[\n  Query <: PipelineQuery,\n  CandidateSourceQuery,\n  CandidateSourceResult,\n  Candidate <: UniversalNoun[Any]\n](\n  candidateSource: BaseCandidateSource[CandidateSourceQuery, CandidateSourceResult],\n  queryTransformer: BaseCandidatePipelineQueryTransformer[\n    Query,\n    CandidateSourceQuery\n  ],\n  resultTransformer: CandidatePipelineResultsTransformer[CandidateSourceResult, Candidate],\n  resultFeaturesTransformers: Seq[CandidateFeatureTransformer[CandidateSourceResult]])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/decorator/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_decorator_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_decorator_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/decorator/DecoratorStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.decorator\n\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasCandidatesWithDetails\nimport com.twitter.product_mixer.core.pipeline.state.HasCandidatesWithFeatures\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.candidate_decorator_executor.CandidateDecoratorExecutor\nimport com.twitter.product_mixer.core.service.candidate_decorator_executor.CandidateDecoratorExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * A candidate decoration step, which takes the query and candidates and outputs decorations for them\n *\n * @param candidateDecoratorExecutor Candidate Source Executor\n * @tparam Query Type of PipelineQuery domain model\n * @tparam Candidate Type of Candidates to filter\n * @tparam State The pipeline state domain model.\n */\ncase class DecoratorStep[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  State <: HasQuery[Query, State] with HasCandidatesWithDetails[\n    State\n  ] with HasCandidatesWithFeatures[\n    Candidate,\n    State\n  ]] @Inject() (candidateDecoratorExecutor: CandidateDecoratorExecutor)\n    extends Step[\n      State,\n      Option[CandidateDecorator[Query, Candidate]],\n      (Query, Seq[CandidateWithFeatures[Candidate]]),\n      CandidateDecoratorExecutorResult\n    ] {\n\n  override def isEmpty(config: Option[CandidateDecorator[Query, Candidate]]): Boolean =\n    config.isEmpty\n\n  override def adaptInput(\n    state: State,\n    config: Option[CandidateDecorator[Query, Candidate]]\n  ): (Query, Seq[CandidateWithFeatures[Candidate]]) =\n    (state.query, state.candidatesWithFeatures)\n\n  override def arrow(\n    config: Option[CandidateDecorator[Query, Candidate]],\n    context: Executor.Context\n  ): Arrow[(Query, Seq[CandidateWithFeatures[Candidate]]), CandidateDecoratorExecutorResult] =\n    candidateDecoratorExecutor.arrow(config, context)\n\n  override def updateState(\n    state: State,\n    executorResult: CandidateDecoratorExecutorResult,\n    config: Option[CandidateDecorator[Query, Candidate]]\n  ): State = {\n    state.updateDecorations(executorResult.result)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/domain_marshaller/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/domain_marshaller_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/domain_marshaller_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/domain_marshaller/DomainMarshallerStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.domain_marshaller\n\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasCandidatesWithDetails\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.domain_marshaller_executor.DomainMarshallerExecutor\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * A domain marshaller step, it takes the input list of candidates and the given\n * domain marshaller and executes its to return a marshalled result. The [[State]] object is\n * responsible for keeping a reference of the built Response.\n *\n * @param domainMarshallerExecutor Domain Marshaller executor.\n * @tparam Query Type of PipelineQuery domain model\n * @tparam ResponseType the domain marshalling type expected to be returned.\n * @tparam State The pipeline state domain model.\n */\ncase class DomainMarshallerStep[\n  Query <: PipelineQuery,\n  ResponseType <: HasMarshalling,\n  State <: HasQuery[Query, State] with HasCandidatesWithDetails[State]] @Inject() (\n  domainMarshallerExecutor: DomainMarshallerExecutor)\n    extends Step[State, DomainMarshaller[Query, ResponseType], DomainMarshallerExecutor.Inputs[\n      Query\n    ], DomainMarshallerExecutor.Result[ResponseType]] {\n\n  override def isEmpty(config: DomainMarshaller[Query, ResponseType]): Boolean = false\n\n  override def adaptInput(\n    state: State,\n    config: DomainMarshaller[Query, ResponseType]\n  ): DomainMarshallerExecutor.Inputs[Query] =\n    DomainMarshallerExecutor.Inputs(state.query, state.candidatesWithDetails)\n\n  override def arrow(\n    config: DomainMarshaller[Query, ResponseType],\n    context: Executor.Context\n  ): Arrow[DomainMarshallerExecutor.Inputs[Query], DomainMarshallerExecutor.Result[ResponseType]] =\n    domainMarshallerExecutor.arrow(config, context)\n\n  // Noop since the pipeline updates the executor results for us\n  override def updateState(\n    state: State,\n    executorResult: DomainMarshallerExecutor.Result[ResponseType],\n    config: DomainMarshaller[Query, ResponseType]\n  ): State = state\n\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/filter/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/filter_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/filter_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/filter/FilterStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.filter\n\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasCandidatesWithFeatures\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.filter_executor.FilterExecutor\nimport com.twitter.product_mixer.core.service.filter_executor.FilterExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * A candidate filter step, it takes the input list of candidates and the given filter and applies\n * the filters on the candidates in sequence, returning the final kept candidates list to State.\n *\n * @param filterExecutor Filter Executor\n * @tparam Query Type of PipelineQuery domain model\n * @tparam Candidate Type of Candidates to filter\n * @tparam State The pipeline state domain model.\n */\ncase class FilterStep[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  State <: HasQuery[Query, State] with HasCandidatesWithFeatures[\n    Candidate,\n    State\n  ]] @Inject() (filterExecutor: FilterExecutor)\n    extends Step[State, Seq[\n      Filter[Query, Candidate]\n    ], (Query, Seq[CandidateWithFeatures[Candidate]]), FilterExecutorResult[Candidate]] {\n\n  override def isEmpty(config: Seq[Filter[Query, Candidate]]): Boolean = config.isEmpty\n\n  override def adaptInput(\n    state: State,\n    config: Seq[Filter[Query, Candidate]]\n  ): (Query, Seq[CandidateWithFeatures[Candidate]]) =\n    (state.query, state.candidatesWithFeatures)\n\n  override def arrow(\n    config: Seq[Filter[Query, Candidate]],\n    context: Executor.Context\n  ): Arrow[(Query, Seq[CandidateWithFeatures[Candidate]]), FilterExecutorResult[Candidate]] =\n    filterExecutor.arrow(config, context)\n\n  override def updateState(\n    state: State,\n    executorResult: FilterExecutorResult[Candidate],\n    config: Seq[Filter[Query, Candidate]]\n  ): State = {\n    val keptCandidates = executorResult.result\n    val candidatesMap = state.candidatesWithFeatures.map { candidatesWithFeatures =>\n      candidatesWithFeatures.candidate -> candidatesWithFeatures\n    }.toMap\n    val newCandidates = keptCandidates.flatMap { candidate =>\n      candidatesMap.get(candidate)\n    }\n    state.updateCandidatesWithFeatures(newCandidates)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/gate/BUILD",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/gate/GateStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.gate\n\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutor\nimport com.twitter.product_mixer.core.service.gate_executor.GateExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * A gate step, it takes the query and the given gates and executes them. Gates do not update state\n * if they return continue, and throw an exception if any gate says stopped, thus no state changes\n * are expected in this step. The [[NewPipelineArrowBuilder]] and [[PipelineStep]] handle short\n * circuiting the pipeline's execution if this throws.\n *\n * @param gateExecutor Gate Executor for executing the gates\n * @tparam Query Type of PipelineQuery domain model\n * @tparam State The pipeline state domain model.\n */\ncase class GateStep[Query <: PipelineQuery, State <: HasQuery[Query, State]] @Inject() (\n  gateExecutor: GateExecutor)\n    extends Step[State, Seq[BaseGate[Query]], Query, GateExecutorResult] {\n\n  override def adaptInput(state: State, config: Seq[BaseGate[Query]]): Query = state.query\n\n  override def arrow(\n    config: Seq[BaseGate[Query]],\n    context: Executor.Context\n  ): Arrow[Query, GateExecutorResult] = gateExecutor.arrow(config, context)\n\n  // Gate Executor is a noop, if it continues, the state isn't changed. If it stops the world,\n  // an exception gets thrown.\n  override def updateState(\n    input: State,\n    executorResult: GateExecutorResult,\n    config: Seq[BaseGate[Query]]\n  ): State = input\n\n  override def isEmpty(config: Seq[BaseGate[Query]]): Boolean = config.isEmpty\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/group_results/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/group_results_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/group_results_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/group_results/GroupResultsStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.group_results\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.pipeline.state.HasCandidatesWithDetails\nimport com.twitter.product_mixer.core.pipeline.state.HasCandidatesWithFeatures\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.group_results_executor.GroupResultsExecutor\nimport com.twitter.product_mixer.core.service.group_results_executor.GroupResultsExecutorInput\nimport com.twitter.product_mixer.core.service.group_results_executor.GroupResultsExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * A group results step, it takes the input list of candidates and decorations, and assembles\n * properly decorated candidates with details.\n *\n * @param groupResultsExecutor Group results executor\n * @tparam Candidate Type of candidates\n * @tparam State The pipeline state domain model.\n */\ncase class GroupResultsStep[\n  Candidate <: UniversalNoun[Any],\n  State <: HasCandidatesWithDetails[State] with HasCandidatesWithFeatures[\n    Candidate,\n    State\n  ]] @Inject() (\n  groupResultsExecutor: GroupResultsExecutor)\n    extends Step[State, CandidatePipelineContext, GroupResultsExecutorInput[\n      Candidate\n    ], GroupResultsExecutorResult] {\n\n  override def isEmpty(config: CandidatePipelineContext): Boolean = false\n  override def adaptInput(\n    state: State,\n    config: CandidatePipelineContext\n  ): GroupResultsExecutorInput[Candidate] = {\n    val presentationMap = state.candidatesWithDetails.flatMap { candidateWithDetails =>\n      candidateWithDetails.presentation\n        .map { presentation =>\n          candidateWithDetails.getCandidate[UniversalNoun[Any]] -> presentation\n        }\n    }.toMap\n    GroupResultsExecutorInput(state.candidatesWithFeatures, presentationMap)\n  }\n\n  override def arrow(\n    config: CandidatePipelineContext,\n    context: Executor.Context\n  ): Arrow[GroupResultsExecutorInput[Candidate], GroupResultsExecutorResult] =\n    groupResultsExecutor.arrow(\n      config.candidatePipelineIdentifier,\n      config.candidateSourceIdentifier,\n      context)\n\n  override def updateState(\n    state: State,\n    executorResult: GroupResultsExecutorResult,\n    config: CandidatePipelineContext\n  ): State = state.updateCandidatesWithDetails(executorResult.candidatesWithDetails)\n}\n\ncase class CandidatePipelineContext(\n  candidatePipelineIdentifier: CandidatePipelineIdentifier,\n  candidateSourceIdentifier: CandidateSourceIdentifier)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/pipeline_executor/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/pipeline_selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_executor\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_executor\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/pipeline_executor/PipelineExecutorStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.pipeline_executor\n\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.Pipeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.state.HasExecutorResults\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasResult\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.pipeline.step.pipeline_selector.PipelineSelectorResult\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorObserver\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.pipeline_executor.PipelineExecutor\nimport com.twitter.product_mixer.core.service.pipeline_executor.PipelineExecutorRequest\nimport com.twitter.product_mixer.core.service.pipeline_executor.PipelineExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * Pipeline Execution step that takes a selected pipeline and executes it.\n *\n * @param pipelineExecutor Pipeline executor that executes the selected pipeline\n *\n * @tparam Query Pipeline query model with quality factor status\n * @tparam Result The expected result type\n * @tparam State The pipeline state domain model.\n */\ncase class PipelineExecutorStep[\n  Query <: PipelineQuery,\n  Result,\n  State <: HasQuery[Query, State] with HasExecutorResults[State] with HasResult[Result]] @Inject() (\n  pipelineExecutor: PipelineExecutor)\n    extends Step[\n      State,\n      PipelineExecutorStepConfig[Query, Result],\n      PipelineExecutorRequest[Query],\n      PipelineExecutorResult[Result]\n    ] {\n\n  override def isEmpty(config: PipelineExecutorStepConfig[Query, Result]): Boolean =\n    false\n\n  override def adaptInput(\n    state: State,\n    config: PipelineExecutorStepConfig[Query, Result]\n  ): PipelineExecutorRequest[Query] = {\n    val pipelineSelectorResult = state.executorResultsByPipelineStep\n      .getOrElse(\n        config.selectedPipelineResultIdentifier,\n        throw PipelineFailure(\n          IllegalStateFailure,\n          \"Missing Selected Pipeline in Pipeline Executor Step\")).asInstanceOf[\n        PipelineSelectorResult]\n    PipelineExecutorRequest(state.query, pipelineSelectorResult.pipelineIdentifier)\n  }\n\n  override def arrow(\n    config: PipelineExecutorStepConfig[Query, Result],\n    context: Executor.Context\n  ): Arrow[PipelineExecutorRequest[Query], PipelineExecutorResult[Result]] = pipelineExecutor.arrow(\n    config.pipelinesByIdentifier,\n    config.qualityFactorObserversByIdentifier,\n    context\n  )\n\n  // Noop since the platform will add the final result to the executor result map then state\n  // is responsible for reading it in [[WithResult]]\n  override def updateState(\n    state: State,\n    executorResult: PipelineExecutorResult[Result],\n    config: PipelineExecutorStepConfig[Query, Result]\n  ): State = state\n}\n\ncase class PipelineExecutorStepConfig[Query <: PipelineQuery, Result](\n  pipelinesByIdentifier: Map[ComponentIdentifier, Pipeline[Query, Result]],\n  selectedPipelineResultIdentifier: PipelineStepIdentifier,\n  qualityFactorObserversByIdentifier: Map[ComponentIdentifier, QualityFactorObserver])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/pipeline_selector/BUILD",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/pipeline_selector/PipelineSelectorStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.pipeline_selector\n\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * Pipeline Selection step to decide which pipeline to execute. This step doesn't update state, as\n * the selected pipeline identifier is added to the executor results list map for later retrieval\n *\n * @tparam Query Pipeline query model\n * @tparam State The pipeline state domain model.\n */\ncase class PipelineSelectorStep[Query <: PipelineQuery, State <: HasQuery[Query, State]] @Inject() (\n) extends Step[State, Query => ComponentIdentifier, Query, PipelineSelectorResult] {\n  override def isEmpty(config: Query => ComponentIdentifier): Boolean = false\n\n  override def adaptInput(\n    state: State,\n    config: Query => ComponentIdentifier\n  ): Query = state.query\n\n  override def arrow(\n    config: Query => ComponentIdentifier,\n    context: Executor.Context\n  ): Arrow[Query, PipelineSelectorResult] = Arrow.map { query: Query =>\n    PipelineSelectorResult(config(query))\n  }\n\n  // Noop since we keep the identifier in the executor results\n  override def updateState(\n    state: State,\n    executorResult: PipelineSelectorResult,\n    config: Query => ComponentIdentifier\n  ): State = state\n}\n\ncase class PipelineSelectorResult(pipelineIdentifier: ComponentIdentifier) extends ExecutorResult\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/quality_factor/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/quality_factor/QualityFactorStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.quality_factor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.quality_factor.HasQualityFactorStatus\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorStatus\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * Quality Factor building step that builds up the state snapshot for a map of configs.\n *\n * @param statsReceiver Stats Receiver used to build finagle gauges for QF State\n *\n * @tparam Query Pipeline query model with quality factor status\n * @tparam State The pipeline state domain model.\n */\ncase class QualityFactorStep[\n  Query <: PipelineQuery with HasQualityFactorStatus,\n  State <: HasQuery[Query, State]] @Inject() (\n  statsReceiver: StatsReceiver)\n    extends Step[\n      State,\n      QualityFactorStepConfig,\n      Any,\n      QualityFactorStepResult\n    ] {\n  override def isEmpty(config: QualityFactorStepConfig): Boolean =\n    config.qualityFactorStatus.qualityFactorByPipeline.isEmpty\n\n  override def adaptInput(\n    state: State,\n    config: QualityFactorStepConfig\n  ): Any = ()\n\n  override def arrow(\n    config: QualityFactorStepConfig,\n    context: Executor.Context\n  ): Arrow[Any, QualityFactorStepResult] = {\n    // We use provideGauge so these gauges live forever even without a reference.\n    val currentValues = config.qualityFactorStatus.qualityFactorByPipeline.map {\n      case (identifier, qualityFactor) =>\n        // QF is a relative stat (since the parent pipeline is monitoring a child pipeline)\n        val scopes = config.pipelineIdentifier.toScopes ++ identifier.toScopes :+ \"QualityFactor\"\n        val currentValue = qualityFactor.currentValue.toFloat\n        statsReceiver.provideGauge(scopes: _*) {\n          currentValue\n        }\n        identifier -> currentValue\n    }\n    Arrow.value(QualityFactorStepResult(currentValues))\n  }\n\n  override def updateState(\n    state: State,\n    executorResult: QualityFactorStepResult,\n    config: QualityFactorStepConfig\n  ): State = state.updateQuery(\n    state.query.withQualityFactorStatus(config.qualityFactorStatus).asInstanceOf[Query])\n}\n\ncase class QualityFactorStepConfig(\n  pipelineIdentifier: ComponentIdentifier,\n  qualityFactorStatus: QualityFactorStatus)\n\ncase class QualityFactorStepResult(currentValues: Map[ComponentIdentifier, Float])\n    extends ExecutorResult\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/query_feature_hydrator/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/asyncfeaturemap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/query_feature_hydrator_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/query_feature_hydrator_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/query_feature_hydrator/QueryFeatureHydratorStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.query_feature_hydrator\n\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseQueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasAsyncFeatureMap\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * A query level feature hydration step, it takes the input list of candidates and the given\n * hydrators and executes them. The [[State]] object is responsible for merging the resulting\n * feature maps with the hydrated ones in its updateCandidatesWithFeatures.\n *\n * @param queryFeatureHydratorExecutor Hydrator Executor\n * @tparam Query Type of PipelineQuery domain model\n * @tparam State The pipeline state domain model.\n */\ncase class QueryFeatureHydratorStep[\n  Query <: PipelineQuery,\n  State <: HasQuery[Query, State] with HasAsyncFeatureMap[State]] @Inject() (\n  queryFeatureHydratorExecutor: QueryFeatureHydratorExecutor)\n    extends Step[State, QueryFeatureHydratorStepConfig[\n      Query\n    ], Query, QueryFeatureHydratorExecutor.Result] {\n  override def isEmpty(config: QueryFeatureHydratorStepConfig[Query]): Boolean =\n    config.hydrators.isEmpty\n\n  override def adaptInput(state: State, config: QueryFeatureHydratorStepConfig[Query]): Query =\n    state.query\n\n  override def arrow(\n    config: QueryFeatureHydratorStepConfig[Query],\n    context: Executor.Context\n  ): Arrow[Query, QueryFeatureHydratorExecutor.Result] =\n    queryFeatureHydratorExecutor.arrow(\n      config.hydrators,\n      config.validPipelineStepIdentifiers,\n      context)\n\n  override def updateState(\n    state: State,\n    executorResult: QueryFeatureHydratorExecutor.Result,\n    config: QueryFeatureHydratorStepConfig[Query]\n  ): State = {\n    val updatedQuery = state.query\n      .withFeatureMap(executorResult.featureMap).asInstanceOf[Query]\n    state\n      .updateQuery(updatedQuery).addAsyncFeatureMap(executorResult.asyncFeatureMap)\n  }\n}\n\ncase class QueryFeatureHydratorStepConfig[Query <: PipelineQuery](\n  hydrators: Seq[BaseQueryFeatureHydrator[Query, _]],\n  validPipelineStepIdentifiers: Set[PipelineStepIdentifier])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/query_transformer/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/query_transformer/QueryTransformerStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.query_transformer\n\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasParams\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasRequest\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport com.twitter.stitch.Arrow\nimport com.twitter.timelines.configapi.Params\n\n/**\n * Query Transformation Step that takes an incoming thrift request model object and returns a\n * pipeline query. The pipeline state is responsible for keeping the updated query.\n *\n * @tparam TRequest Thrift request domain model\n * @tparam Query PipelineQuery type to transform to h\n * @tparam State The request domain model\n */\ncase class QueryTransformerStep[\n  TRequest <: Request,\n  Query <: PipelineQuery,\n  State <: HasQuery[Query, State] with HasRequest[TRequest] with HasParams\n]() extends Step[State, (TRequest, Params) => Query, (TRequest, Params), QueryTransformerResult[\n      Query\n    ]] {\n\n  override def isEmpty(config: (TRequest, Params) => Query): Boolean = false\n\n  override def arrow(\n    config: (TRequest, Params) => Query,\n    context: Executor.Context\n  ): Arrow[(TRequest, Params), QueryTransformerResult[Query]] = Arrow.map {\n    case (request: TRequest @unchecked, params: Params) =>\n      QueryTransformerResult(config(request, params))\n  }\n\n  override def updateState(\n    state: State,\n    executorResult: QueryTransformerResult[Query],\n    config: (TRequest, Params) => Query\n  ): State = state.updateQuery(executorResult.query)\n\n  override def adaptInput(\n    state: State,\n    config: (TRequest, Params) => Query\n  ): (TRequest, Params) = (state.request, state.params)\n}\n\ncase class QueryTransformerResult[Query <: PipelineQuery](query: Query) extends ExecutorResult\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/scorer/BUILD",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_hydrator_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/scorer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_hydrator_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/scorer/ScorerStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.scorer\n\nimport com.twitter.product_mixer.core.functional_component.scorer.Scorer\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasCandidatesWithFeatures\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutor\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * A scoring step, it takes the input list of candidates and the given\n * scorers and executes them. The [[State]] object is responsible for merging the resulting\n * feature maps with the scored ones in its updateCandidatesWithFeatures.\n *\n * @param candidateFeatureHydratorExecutor Hydrator Executor\n * @tparam Query Type of PipelineQuery domain model\n * @tparam Candidate Type of Candidates to hydrate features for.\n * @tparam State The pipeline state domain model.\n */\ncase class ScorerStep[\n  Query <: PipelineQuery,\n  Candidate <: UniversalNoun[Any],\n  State <: HasQuery[Query, State] with HasCandidatesWithFeatures[\n    Candidate,\n    State\n  ]] @Inject() (\n  candidateFeatureHydratorExecutor: CandidateFeatureHydratorExecutor)\n    extends Step[State, Seq[\n      Scorer[Query, Candidate]\n    ], CandidateFeatureHydratorExecutor.Inputs[\n      Query,\n      Candidate\n    ], CandidateFeatureHydratorExecutorResult[Candidate]] {\n\n  override def adaptInput(\n    state: State,\n    config: Seq[Scorer[Query, Candidate]]\n  ): CandidateFeatureHydratorExecutor.Inputs[Query, Candidate] =\n    CandidateFeatureHydratorExecutor.Inputs(state.query, state.candidatesWithFeatures)\n\n  override def arrow(\n    config: Seq[Scorer[Query, Candidate]],\n    context: Executor.Context\n  ): Arrow[\n    CandidateFeatureHydratorExecutor.Inputs[Query, Candidate],\n    CandidateFeatureHydratorExecutorResult[Candidate]\n  ] = candidateFeatureHydratorExecutor.arrow(config, context)\n\n  override def updateState(\n    input: State,\n    executorResult: CandidateFeatureHydratorExecutorResult[Candidate],\n    config: Seq[Scorer[Query, Candidate]]\n  ): State = {\n    val resultCandidates = executorResult.results\n    if (resultCandidates.isEmpty) {\n      input\n    } else {\n      input.updateCandidatesWithFeatures(resultCandidates)\n    }\n  }\n\n  override def isEmpty(config: Seq[Scorer[Query, Candidate]]): Boolean =\n    config.isEmpty\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/selector/BUILD",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/selector_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/component-library/src/main/scala/com/twitter/product_mixer/component_library/model/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/selector_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/selector/SelectorStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.selector\n\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.state.HasCandidatesWithDetails\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutor\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * A selection step, it takes the input list of candidates with details and the given\n * selectors and executes them to decide which candidates should be selected.\n *\n * @param selectorExecutor Selector Executor\n * @tparam Query Type of PipelineQuery domain model\n * @tparam State The pipeline state domain model.\n */\ncase class SelectorStep[\n  Query <: PipelineQuery,\n  State <: HasQuery[Query, State] with HasCandidatesWithDetails[State]] @Inject() (\n  selectorExecutor: SelectorExecutor)\n    extends Step[State, Seq[\n      Selector[Query]\n    ], SelectorExecutor.Inputs[\n      Query\n    ], SelectorExecutorResult] {\n\n  override def adaptInput(\n    state: State,\n    config: Seq[Selector[Query]]\n  ): SelectorExecutor.Inputs[Query] =\n    SelectorExecutor.Inputs(state.query, state.candidatesWithDetails)\n\n  override def arrow(\n    config: Seq[Selector[Query]],\n    context: Executor.Context\n  ): Arrow[SelectorExecutor.Inputs[Query], SelectorExecutorResult] =\n    selectorExecutor.arrow(config, context)\n\n  override def updateState(\n    input: State,\n    executorResult: SelectorExecutorResult,\n    config: Seq[Selector[Query]]\n  ): State = input.updateCandidatesWithDetails(executorResult.selectedCandidates)\n\n  // Selection is a bit different to other steps (i.e, other steps, empty means don't change anything)\n  // where an empty selection list drops all candidates.\n  override def isEmpty(config: Seq[Selector[Query]]): Boolean = false\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/side_effect/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/domain_marshaller_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_result_side_effect_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/selector_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_result_side_effect_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/side_effect/SideEffectStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.side_effect\n\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.state.HasExecutorResults\nimport com.twitter.product_mixer.core.pipeline.state.HasQuery\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.domain_marshaller_executor.DomainMarshallerExecutor\nimport com.twitter.product_mixer.core.service.pipeline_result_side_effect_executor.PipelineResultSideEffectExecutor\nimport com.twitter.product_mixer.core.service.selector_executor.SelectorExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * A side effect step, it takes the input list of side effects and and executes them.\n *\n * @param sideEffectExecutor Side Effect Executor\n *\n * @tparam Query Type of PipelineQuery domain model\n * @tparam DomainResultType Domain Marshaller result type\n * @tparam State The pipeline state domain model.\n */\ncase class SideEffectStep[\n  Query <: PipelineQuery,\n  DomainResultType <: HasMarshalling,\n  State <: HasQuery[Query, State] with HasExecutorResults[State]] @Inject() (\n  sideEffectExecutor: PipelineResultSideEffectExecutor)\n    extends Step[\n      State,\n      PipelineStepConfig[Query, DomainResultType],\n      PipelineResultSideEffect.Inputs[\n        Query,\n        DomainResultType\n      ],\n      PipelineResultSideEffectExecutor.Result\n    ] {\n  override def isEmpty(config: PipelineStepConfig[Query, DomainResultType]): Boolean =\n    config.sideEffects.isEmpty\n\n  override def adaptInput(\n    state: State,\n    config: PipelineStepConfig[Query, DomainResultType]\n  ): PipelineResultSideEffect.Inputs[Query, DomainResultType] = {\n    val selectorResults = state.executorResultsByPipelineStep\n      .getOrElse(\n        config.selectorStepIdentifier,\n        throw PipelineFailure(\n          IllegalStateFailure,\n          \"Missing Selector Result in Side Effect Step\")).asInstanceOf[SelectorExecutorResult]\n\n    val domainMarshallerResult = state.executorResultsByPipelineStep\n      .getOrElse(\n        config.domainMarshallerStepIdentifier,\n        throw PipelineFailure(\n          IllegalStateFailure,\n          \"Missing Domain Marshaller Result in Side Effect Step\")).asInstanceOf[\n        DomainMarshallerExecutor.Result[DomainResultType]]\n\n    PipelineResultSideEffect.Inputs(\n      query = state.query,\n      selectedCandidates = selectorResults.selectedCandidates,\n      remainingCandidates = selectorResults.remainingCandidates,\n      droppedCandidates = selectorResults.droppedCandidates,\n      response = domainMarshallerResult.result\n    )\n  }\n\n  override def arrow(\n    config: PipelineStepConfig[Query, DomainResultType],\n    context: Executor.Context\n  ): Arrow[\n    PipelineResultSideEffect.Inputs[Query, DomainResultType],\n    PipelineResultSideEffectExecutor.Result\n  ] = sideEffectExecutor.arrow(config.sideEffects, context)\n\n  override def updateState(\n    state: State,\n    executorResult: PipelineResultSideEffectExecutor.Result,\n    config: PipelineStepConfig[Query, DomainResultType]\n  ): State = state\n}\n\n/**\n * Wrapper case class containing side effects to be executed and other information needed to execute\n * @param sideEffects The side effects to execute.\n * @param selectorStepIdentifier The identifier of the selector step in the parent\n *                               pipeline to get selection results from.\n * @param domainMarshallerStepIdentifier The identifier of the domain marshaller step in the parent\n *                                       pipeline to get domain marshalled results from.\n *\n * @tparam Query Type of PipelineQuery domain model\n * @tparam DomainResultType Domain Marshaller result type\n */\ncase class PipelineStepConfig[Query <: PipelineQuery, DomainResultType <: HasMarshalling](\n  sideEffects: Seq[PipelineResultSideEffect[Query, DomainResultType]],\n  selectorStepIdentifier: PipelineStepIdentifier,\n  domainMarshallerStepIdentifier: PipelineStepIdentifier)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/transport_marshaller/BUILD.bazel",
    "content": "scala_library(\n    sources = [\n        \"*.scala\",\n    ],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/domain_marshaller_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transport_marshaller_executor\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/state\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transport_marshaller_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/step/transport_marshaller/TransportMarshallerStep.scala",
    "content": "package com.twitter.product_mixer.core.pipeline.step.transport_marshaller\n\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.state.HasExecutorResults\nimport com.twitter.product_mixer.core.pipeline.step.Step\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.domain_marshaller_executor.DomainMarshallerExecutor\nimport com.twitter.product_mixer.core.service.transport_marshaller_executor.TransportMarshallerExecutor\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\n\n/**\n * A transport marshaller step, it takes domain marshalled result as input and returns trasnport\n * ready marshalled object.\n * The [[State]] object is responsible for keeping a reference of the built marshalled response.\n *\n * @param transportMarshallerExecutor Domain Marshaller executor.\n * @tparam Query Type of PipelineQuery domain model\n * @tparam DomainResponseType the domain marshalling type used as input\n * @tparam TransportResponseType the expected returned transport type\n * @tparam State The pipeline state domain model.\n */\ncase class TransportMarshallerStep[\n  DomainResponseType <: HasMarshalling,\n  TransportResponseType,\n  State <: HasExecutorResults[State]] @Inject() (\n  transportMarshallerExecutor: TransportMarshallerExecutor)\n    extends Step[\n      State,\n      TransportMarshallerConfig[DomainResponseType, TransportResponseType],\n      TransportMarshallerExecutor.Inputs[DomainResponseType],\n      TransportMarshallerExecutor.Result[TransportResponseType]\n    ] {\n\n  override def isEmpty(\n    config: TransportMarshallerConfig[DomainResponseType, TransportResponseType]\n  ): Boolean = false\n\n  override def adaptInput(\n    state: State,\n    config: TransportMarshallerConfig[DomainResponseType, TransportResponseType]\n  ): TransportMarshallerExecutor.Inputs[DomainResponseType] = {\n    val domainMarshallerResult = state.executorResultsByPipelineStep\n      .getOrElse(\n        config.domainMarshallerStepIdentifier,\n        throw PipelineFailure(\n          IllegalStateFailure,\n          \"Missing Domain Marshaller in Transport Marshaller Step\")).asInstanceOf[\n        DomainMarshallerExecutor.Result[DomainResponseType]]\n    TransportMarshallerExecutor.Inputs(domainMarshallerResult.result)\n  }\n\n  // Noop as platform updates executor result\n  override def updateState(\n    state: State,\n    executorResult: TransportMarshallerExecutor.Result[TransportResponseType],\n    config: TransportMarshallerConfig[DomainResponseType, TransportResponseType]\n  ): State = state\n\n  override def arrow(\n    config: TransportMarshallerConfig[DomainResponseType, TransportResponseType],\n    context: Executor.Context\n  ): Arrow[TransportMarshallerExecutor.Inputs[\n    DomainResponseType\n  ], TransportMarshallerExecutor.Result[TransportResponseType]] =\n    transportMarshallerExecutor.arrow(config.transportMarshaller, context)\n\n}\n\ncase class TransportMarshallerConfig[DomainResponseType <: HasMarshalling, TransportResponseType](\n  transportMarshaller: TransportMarshaller[DomainResponseType, TransportResponseType],\n  domainMarshallerStepIdentifier: PipelineStepIdentifier)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/registry\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/registry\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/mixer\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/ProductParamConfig.scala",
    "content": "package com.twitter.product_mixer.core.product\n\nimport com.twitter.product_mixer.core.functional_component.configapi.registry.ParamConfig\nimport com.twitter.servo.decider.DeciderKeyName\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.decider.BooleanDeciderParam\n\ntrait ProductParamConfig extends ParamConfig with ProductParamConfigBuilder {\n\n  /**\n   * This enabled decider param can to be used to quickly disable a Product via Decider\n   *\n   * This value must correspond to the deciders configured in the `resources/config/decider.yml` file\n   */\n  val enabledDeciderKey: DeciderKeyName\n\n  /**\n   * This supported client feature switch param can be used with a Feature Switch to control the\n   * rollout of a new Product from dogfood to experiment to production\n   *\n   * FeatureSwitches are configured by defining both a [[com.twitter.timelines.configapi.Param]] in code\n   * and in an associated `.yml` file in the __config repo__.\n   *\n   * The `.yml` file path is determined by the `feature_switches_path` in your aurora file and tge Product name\n   * so the resulting path in the __config repo__ is essentially `s\"{feature_switches_path}/{snakeCase(Product.identifier)}\"`\n   */\n  val supportedClientFSName: String\n\n  object EnabledDeciderParam extends BooleanDeciderParam(enabledDeciderKey)\n\n  object SupportedClientParam\n      extends FSParam(\n        name = supportedClientFSName,\n        default = false\n      )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/ProductParamConfigBuilder.scala",
    "content": "package com.twitter.product_mixer.core.product\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.configapi.registry.ParamConfigBuilder\nimport com.twitter.servo.decider.DeciderGateBuilder\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.OptionalOverride\nimport com.twitter.timelines.configapi.decider.DeciderUtils\n\ntrait ProductParamConfigBuilder extends ParamConfigBuilder {\n  productParamConfig: ProductParamConfig =>\n\n  override def build(\n    deciderGateBuilder: DeciderGateBuilder,\n    statsReceiver: StatsReceiver\n  ): Seq[OptionalOverride[_]] = {\n    DeciderUtils.getBooleanDeciderOverrides(deciderGateBuilder, EnabledDeciderParam) ++\n      FeatureSwitchOverrideUtil.getBooleanFSOverrides(SupportedClientParam) ++\n      super.build(deciderGateBuilder, statsReceiver)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice/ProductScope.scala",
    "content": "package com.twitter.product_mixer.core.product.guice\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport com.google.inject.Key\n\n/**\n * A specialization of SimpleScope - a simple Guice Scope that takes an initial Product Mixer Product as a key\n */\nclass ProductScope extends SimpleScope {\n  def let[T](product: Product)(f: => T): T = super.let(Map(Key.get(classOf[Product]) -> product))(f)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice/ProductScopeModule.scala",
    "content": "package com.twitter.product_mixer.core.product.guice\n\nimport com.google.inject.Provides\nimport com.twitter.inject.TwitterModule\nimport com.twitter.product_mixer.core.product.guice.scope.ProductScoped\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport javax.inject.Singleton\n\n/**\n * Registers the @ProductScoped scope.\n *\n * See https://github.com/google/guice/wiki/CustomScopes#registering-the-scope\n */\n@Singleton\nclass ProductScopeModule extends TwitterModule {\n\n  val productScope: ProductScope = new ProductScope\n\n  override def configure(): Unit = {\n    bindScope(classOf[ProductScoped], productScope)\n\n    bind[Product].toProvider(SimpleScope.SEEDED_KEY_PROVIDER).in(classOf[ProductScoped])\n  }\n\n  @Provides\n  def providesProductScope(): ProductScope = productScope\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/guice/SimpleScope.scala",
    "content": "package com.twitter.product_mixer.core.product.guice\n\nimport com.google.inject.Key\nimport com.google.inject.OutOfScopeException\nimport com.google.inject.Provider\nimport com.google.inject.Scope\nimport com.google.inject.Scopes\nimport com.twitter.util.Local\nimport scala.collection.concurrent\nimport scala.collection.mutable\n\n/**\n * A scala-esque implementation of SimpleScope: https://github.com/google/guice/wiki/CustomScopes#implementing-scope\n *\n * Scopes the execution of a single block of code via `let`\n */\nclass SimpleScope extends Scope {\n\n  private val values = new Local[concurrent.Map[Key[_], Any]]()\n\n  /**\n   * Execute a block with a fresh scope.\n   *\n   * You can optionally supply a map of initialObjects to 'seed' the new scope.\n   */\n  def let[T](initialObjects: Map[Key[_], Any] = Map.empty)(f: => T): T = {\n    val newMap: concurrent.Map[Key[_], Any] = concurrent.TrieMap.empty\n\n    initialObjects.foreach { case (key, value) => newMap.put(key, value) }\n\n    values.let(newMap)(f)\n  }\n\n  override def scope[T](\n    key: Key[T],\n    unscoped: Provider[T]\n  ): Provider[T] = () => {\n    val scopedObjects: mutable.Map[Key[T], Any] = getScopedObjectMap(key)\n\n    scopedObjects\n      .get(key).map(_.asInstanceOf[T]).getOrElse {\n        val objectFromUnscoped: T = unscoped.get()\n\n        if (Scopes.isCircularProxy(objectFromUnscoped)) {\n          objectFromUnscoped // Don't remember proxies\n        } else {\n          scopedObjects.put(key, objectFromUnscoped)\n          objectFromUnscoped\n        }\n      }\n  }\n\n  def getScopedObjectMap[T](key: Key[T]): concurrent.Map[Key[T], Any] = {\n    values()\n      .getOrElse(\n        throw new OutOfScopeException(s\"Cannot access $key outside of a scoping block\")\n      ).asInstanceOf[concurrent.Map[Key[T], Any]]\n  }\n}\n\nobject SimpleScope {\n\n  val SEEDED_KEY_PROVIDER: Provider[Nothing] = () =>\n    throw new IllegalStateException(\n      \"\"\"If you got here then it means that your code asked for scoped object which should have\n      | been explicitly seeded in this scope by calling SimpleScope.seed(),\n      | but was not.\"\"\".stripMargin)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/java/com/twitter/product_mixer/core/product/guice/scope\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/registry\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/component_registry\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi/registry\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/request\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/component_registry\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry/ProductParamRegistry.scala",
    "content": "package com.twitter.product_mixer.core.product.registry\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.servo.decider.DeciderGateBuilder\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.Config\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass ProductParamRegistry @Inject() (\n  productPipelineRegistryConfig: ProductPipelineRegistryConfig,\n  deciderGateBuilder: DeciderGateBuilder,\n  statsReceiver: StatsReceiver) {\n\n  def build(): Seq[Config] = {\n    val productConfigs = productPipelineRegistryConfig.productPipelineConfigs.map {\n      productPipelineConfig =>\n        BaseConfigBuilder(\n          productPipelineConfig.paramConfig.build(deciderGateBuilder, statsReceiver))\n          .build(productPipelineConfig.paramConfig.getClass.getSimpleName)\n    }\n\n    productConfigs\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry/ProductPipelineRegistry.scala",
    "content": "package com.twitter.product_mixer.core.product.registry\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.common.identifier.ProductIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ProductPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.RootIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.Pipeline\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipeline\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineBuilderFactory\nimport com.twitter.product_mixer.core.service.component_registry.ComponentRegistry\nimport com.twitter.product_mixer.core.service.component_registry.ComponentRegistrySnapshot\nimport com.twitter.product_mixer.shared_library.observer.Observer\nimport com.twitter.util.Try\nimport com.twitter.util.Var\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.reflect.runtime.universe._\n\n@Singleton\nclass ProductPipelineRegistry @Inject() (\n  componentRegistry: ComponentRegistry,\n  productPipelineRegistryConfig: ProductPipelineRegistryConfig,\n  productPipelineBuilderFactory: ProductPipelineBuilderFactory,\n  statsReceiver: StatsReceiver)\n    extends Logging {\n\n  private val rootIdentifierStack = ComponentIdentifierStack(RootIdentifier())\n\n  private val rebuildObserver =\n    Observer.function[Unit](statsReceiver, \"ProductPipelineRegistry\", \"rebuild\")\n\n  /**\n   * Internal state of ProductPipelineRegistry.\n   *\n   * Build once on startup, and later whenever `rebuild()` is called.\n   */\n  private[this] val productPipelineByProduct =\n    Var[Map[Product, ProductPipeline[_ <: Request, _]]](buildProductPipelineByProduct())\n\n  /**\n   * Triggers a rebuild of the ProductPipelineRegistry and also the ComponentRegistry\n   *\n   * Failed rebuilds will throw an exception - likely one of the listed ones - and the product\n   * registry and component registry will not be modified.\n   *\n   * @throws MultipleProductPipelinesForAProductException\n   * @throws ComponentIdentifierCollisionException\n   * @throws ChildComponentCollisionException\n   */\n  private[core] def rebuild(): Unit = {\n    Try {\n      rebuildObserver {\n        productPipelineByProduct.update(buildProductPipelineByProduct())\n      }\n    }.onFailure { ex =>\n        error(\"Failed to rebuild ProductPipelineRegistry\", ex)\n      }.get()\n  }\n\n  /**\n   * register the provided pipeline recursively register all of it's children components\n   * that are added to the [[Pipeline]]'s [[Pipeline.children]]\n   */\n  private def registerPipelineAndChildren(\n    componentRegistrySnapshot: ComponentRegistrySnapshot,\n    pipeline: Pipeline[_, _],\n    parentIdentifierStack: ComponentIdentifierStack\n  ): Unit = {\n    val identifierStackString =\n      s\"${parentIdentifierStack.componentIdentifiers.reverse.mkString(\"\\t->\\t\")}\\t->\\t${pipeline.identifier}\"\n    info(identifierStackString)\n\n    componentRegistrySnapshot.register(\n      component = pipeline,\n      parentIdentifierStack = parentIdentifierStack)\n\n    val identifierStackWithCurrentPipeline = parentIdentifierStack.push(pipeline.identifier)\n    pipeline.children.foreach {\n      case childPipeline: Pipeline[_, _] =>\n        info(s\"$identifierStackString\\t->\\t${childPipeline.identifier}\")\n        registerPipelineAndChildren(\n          componentRegistrySnapshot,\n          childPipeline,\n          identifierStackWithCurrentPipeline)\n      case component =>\n        info(s\"$identifierStackString\\t->\\t${component.identifier}\")\n        componentRegistrySnapshot.register(\n          component = component,\n          parentIdentifierStack = identifierStackWithCurrentPipeline)\n    }\n  }\n\n  /*\n   * Internal method (not for callers outside of this class, see rebuild() for those)\n   *\n   * Produces an updated Map[Product, ProductPipeline] and also refreshes the global component registry\n   */\n  private[this] def buildProductPipelineByProduct(\n  ): Map[Product, ProductPipeline[_ <: Request, _]] = {\n\n    // Build a new component registry snapshot.\n    val newComponentRegistry = new ComponentRegistrySnapshot()\n\n    info(\n      \"Registering all products, pipelines, and components (this may be helpful if you encounter dependency injection errors)\")\n    info(\"debug details are in the form of `parent -> child`\")\n\n    // handle the case of multiple ProductPipelines having the same product\n    checkForAndThrowMultipleProductPipelinesForAProduct()\n\n    // Build a Map[Product, ProductPipeline], registering everything in the new component registry recursively\n    val pipelinesByProduct: Map[Product, ProductPipeline[_ <: Request, _]] =\n      productPipelineRegistryConfig.productPipelineConfigs.map { productPipelineConfig =>\n        val product = productPipelineConfig.product\n        info(s\"Recursively registering ${product.identifier}\")\n\n        // gets the ComponentIdentifierStack without the RootIdentifier since\n        // we don't want RootIdentifier to show up in stats or errors\n        val productPipeline =\n          productPipelineBuilderFactory.get.build(\n            ComponentIdentifierStack(product.identifier),\n            productPipelineConfig)\n\n        // gets RootIdentifier so we can register Products under the correct hierarchy\n        newComponentRegistry.register(product, rootIdentifierStack)\n        registerPipelineAndChildren(\n          newComponentRegistry,\n          productPipeline,\n          rootIdentifierStack.push(product.identifier))\n\n        // In addition to registering the component in the main registry, we want to maintain a map of\n        // product to the product pipeline to allow for O(1) lookup by product on the request hot path\n        product -> productPipeline\n      }.toMap\n\n    info(\n      s\"Successfully registered ${newComponentRegistry.getAllRegisteredComponents\n        .count(_.identifier.isInstanceOf[ProductIdentifier])} products and \" +\n        s\"${newComponentRegistry.getAllRegisteredComponents.length} \" +\n        s\"components total, query the component registry endpoint for details\")\n\n    componentRegistry.set(newComponentRegistry)\n\n    pipelinesByProduct\n  }\n\n  // handle the case of multiple ProductPipelines having the same product\n  private def checkForAndThrowMultipleProductPipelinesForAProduct(): Unit = {\n    productPipelineRegistryConfig.productPipelineConfigs.groupBy(_.product.identifier).foreach {\n      case (product, productPipelines) if productPipelines.length != 1 =>\n        throw new MultipleProductPipelinesForAProductException(\n          product,\n          productPipelines.map(_.identifier))\n      case _ =>\n    }\n  }\n\n  def getProductPipeline[MixerRequest <: Request: TypeTag, ResponseType: TypeTag](\n    product: Product\n  ): ProductPipeline[MixerRequest, ResponseType] = {\n    // Check and cast the bounded existential types to the concrete types\n    (typeOf[MixerRequest], typeOf[ResponseType]) match {\n      case (req, res) if req =:= typeOf[MixerRequest] && res =:= typeOf[ResponseType] =>\n        productPipelineByProduct.sample\n          .getOrElse(product, throw new ProductNotFoundException(product))\n          .asInstanceOf[ProductPipeline[MixerRequest, ResponseType]]\n      case _ =>\n        throw new UnknownPipelineResponseException(product)\n    }\n  }\n}\n\nclass ProductNotFoundException(product: Product)\n    extends RuntimeException(s\"No Product found for $product\")\n\nclass UnknownPipelineResponseException(product: Product)\n    extends RuntimeException(s\"Unknown pipeline response for $product\")\n\nclass MultipleProductPipelinesForAProductException(\n  product: ProductIdentifier,\n  pipelineIdentifiers: Seq[ProductPipelineIdentifier])\n    extends IllegalStateException(s\"Multiple ProductPipelines found for $product, found \" +\n      s\"${pipelineIdentifiers\n        .map(productPipelineIdentifier => s\"$productPipelineIdentifier from ${productPipelineIdentifier.file}\")\n        .mkString(\", \")} \")\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry/ProductPipelineRegistryConfig.scala",
    "content": "package com.twitter.product_mixer.core.product.registry\n\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineConfig\n\ntrait ProductPipelineRegistryConfig {\n  def productPipelineConfigs: Seq[ProductPipelineConfig[_ <: Request, _ <: PipelineQuery, _]]\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finagle/finagle-mux/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:query\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"servo/util\",\n        \"stitch/stitch-core\",\n        \"util/util-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor/Bounds.scala",
    "content": "package com.twitter.product_mixer.core.quality_factor\n\n/**\n * Provides a way to apply inclusive min/max bounds to a given value.\n */\ncase class Bounds[T](minInclusive: T, maxInclusive: T)(implicit ordering: Ordering[T]) {\n\n  def apply(value: T): T = ordering.min(maxInclusive, ordering.max(minInclusive, value))\n\n  def isWithin(value: T): Boolean =\n    ordering.gteq(value, minInclusive) && ordering.lteq(value, maxInclusive)\n\n  def throwIfOutOfBounds(value: T, messagePrefix: String): Unit =\n    require(isWithin(value), s\"$messagePrefix: value must be within $toString\")\n\n  override def toString: String = s\"[$minInclusive, $maxInclusive]\"\n}\n\nobject BoundsWithDefault {\n  def apply[T](\n    minInclusive: T,\n    maxInclusive: T,\n    default: T\n  )(\n    implicit ordering: Ordering[T]\n  ): BoundsWithDefault[T] = BoundsWithDefault(Bounds(minInclusive, maxInclusive), default)\n}\n\ncase class BoundsWithDefault[T](bounds: Bounds[T], default: T)(implicit ordering: Ordering[T]) {\n  bounds.throwIfOutOfBounds(default, \"default\")\n\n  def apply(valueOpt: Option[T]): T = valueOpt.map(bounds.apply).getOrElse(default)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor/LinearLatencyQualityFactor.scala",
    "content": "package com.twitter.product_mixer.core.quality_factor\n\nimport com.twitter.util.Duration\nimport com.twitter.util.Stopwatch\n\ncase class LinearLatencyQualityFactor(\n  override val config: LinearLatencyQualityFactorConfig)\n    extends QualityFactor[Duration] {\n\n  private val delayedUntilInMillis = Stopwatch.timeMillis() + config.initialDelay.inMillis\n\n  private var state: Double = config.qualityFactorBounds.default\n\n  override def currentValue: Double = state\n\n  override def update(latency: Duration): Unit = {\n    if (Stopwatch.timeMillis() >= delayedUntilInMillis) {\n      if (latency > config.targetLatency) {\n        adjustState(getNegativeDelta)\n      } else {\n        adjustState(config.delta)\n      }\n    }\n  }\n\n  override def buildObserver(): QualityFactorObserver = LinearLatencyQualityFactorObserver(this)\n\n  private def getNegativeDelta: Double =\n    -config.delta * config.targetLatencyPercentile / (100.0 - config.targetLatencyPercentile)\n\n  private def adjustState(delta: Double): Unit = {\n    state = config.qualityFactorBounds.bounds(state + delta)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor/LinearLatencyQualityFactorObserver.scala",
    "content": "package com.twitter.product_mixer.core.quality_factor\n\nimport com.twitter.util.Duration\nimport com.twitter.util.Try\n\ncase class LinearLatencyQualityFactorObserver(\n  override val qualityFactor: LinearLatencyQualityFactor)\n    extends QualityFactorObserver {\n\n  override def apply(result: Try[_], latency: Duration): Unit = {\n    result\n      .onSuccess(_ => qualityFactor.update(latency))\n      .onFailure {\n        case t if qualityFactor.config.ignorableFailures.isDefinedAt(t) => ()\n        case _ => qualityFactor.update(Duration.Top)\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor/QualityFactor.scala",
    "content": "package com.twitter.product_mixer.core.quality_factor\n\n/**\n * [[QualityFactor]] is an abstract number that enables a feedback loop to control operation costs and ultimately\n * maintain the operation success rate. Abstractly, if operations/calls are too expensive (such as high\n * latencies), the quality factor should go down, which helps future calls to ease their demand/load (such as\n * reducing request width); if ops/calls are fast, the quality factor should go up, so we can incur more load.\n *\n * @note to avoid overhead the underlying state may sometimes not be synchronized.\n *       If a part of an application is unhealthy, it will likely be unhealthy for all threads,\n *       it will eventually result in a close-enough quality factor value for all thread's view of the state.\n *\n *       In extremely low volume scenarios such as manual testing in a development environment,\n *       it's possible that different threads will have vastly different views of the underling state,\n *       but in practice, in production systems, they will be close-enough.\n */\ntrait QualityFactor[Input] { self =>\n\n  /** get the current [[QualityFactor]]'s value */\n  def currentValue: Double\n\n  def config: QualityFactorConfig\n\n  /** update of the current `factor` value */\n  def update(input: Input): Unit\n\n  /** a [[QualityFactorObserver]] for this [[QualityFactor]] */\n  def buildObserver(): QualityFactorObserver\n\n  override def toString: String = {\n    self.getClass.getSimpleName.stripSuffix(\"$\")\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor/QualityFactorConfig.scala",
    "content": "package com.twitter.product_mixer.core.quality_factor\n\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ClientFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorConfig.defaultIgnorableFailures\nimport com.twitter.servo.util.CancelledExceptionExtractor\nimport com.twitter.util.Duration\nimport com.twitter.conversions.DurationOps.RichDuration\n\n/**\n * Quality factor is an abstract number that enables a feedback loop to control operation costs and ultimately\n * maintain the operation success rate. Abstractly, if operations/calls are too expensive (such as high\n * latencies), the quality factor should go down, which helps future calls to ease their demand/load (such as\n * reducing request width); if ops/calls are fast, the quality factor should go up, so we can incur more load.\n */\nsealed trait QualityFactorConfig {\n\n  /**\n   * specifies the quality factor min and max bounds and default value.\n   */\n  def qualityFactorBounds: BoundsWithDefault[Double]\n\n  /**\n   * initialDelay Specifies how much delay we should have before the quality factor calculation start to kick in. This is\n   * mostly to ease the load during the initial warmup/startup.\n   */\n  def initialDelay: Duration\n\n  /**\n   * [[Throwable]]s that should be ignored when calculating\n   * the [[QualityFactor]] if this is [[PartialFunction.isDefinedAt]]\n   */\n  def ignorableFailures: PartialFunction[Throwable, Unit] = defaultIgnorableFailures\n}\n\nobject QualityFactorConfig {\n\n  /**\n   * Default value for [[QualityFactorConfig.ignorableFailures]] that ignores any\n   * Cancelled requests and [[ClientFailure]]\n   */\n  val defaultIgnorableFailures: PartialFunction[Throwable, Unit] = {\n    case PipelineFailure(_: ClientFailure, _, _, _) => ()\n    case CancelledExceptionExtractor(_) => ()\n  }\n}\n\n/**\n * This is a linear quality factor implementation, aimed to achieve and maintain a percentile latency target.\n *\n * If we call quality factor q, target latency t and target percentile p,\n *   then the q (quality factor) formula should be:\n *   q += delta                      for each request with latency <= t\n *   q -= delta * p / (100 - p)      for each request with latency > t ms or a timeout.\n *\n *   When percentile p latency stays at target latency t, then based on the formula above, q will\n *   stay constant (fluctuates around a constant value).\n *\n *   For example, assume t = 100ms, p = p99, and q = 0.5\n *   let's say, p99 latency stays at 100ms when q = 0.5. p99 means that out of every 100 latencies,\n *   99 times the latency is below 100ms and 1 time it is above 100ms. So based on the formula above,\n *   q will increase by \"delta\" 99 times and it will decrease by delta * p / (100 - p) = delta * 99 once,\n *   which results in the same q = 0.5.\n *\n * @param targetLatency This is the latency target, calls with latencies above which will cause quality\n * factor to go down, and vice versa. e.g. 500ms.\n * @param targetLatencyPercentile This the percentile where the target latency is aimed at. e.g. 95.0.\n * @param delta the step for adjusting quality factor. It should be a positive double. If delta is\n *              too large, then quality factor will fluctuate more, and if it is too small, the\n *              responsiveness will be reduced.\n */\ncase class LinearLatencyQualityFactorConfig(\n  override val qualityFactorBounds: BoundsWithDefault[Double],\n  override val initialDelay: Duration,\n  targetLatency: Duration,\n  targetLatencyPercentile: Double,\n  delta: Double,\n  override val ignorableFailures: PartialFunction[Throwable, Unit] =\n    QualityFactorConfig.defaultIgnorableFailures)\n    extends QualityFactorConfig {\n  require(\n    targetLatencyPercentile >= 50.0 && targetLatencyPercentile < 100.0,\n    s\"Invalid targetLatencyPercentile value: ${targetLatencyPercentile}.\\n\" +\n      s\"Correct sample values: 95.0, 99.9. Incorrect sample values: 0.95, 0.999.\"\n  )\n}\n\n/**\n * A quality factor provides component capacity state based on sampling component\n * Queries Per Second (qps) at local host level.\n *\n * If we call quality factor q, max qps R:\n *   then the q (quality factor) formula should be:\n *   q = Math.min([[qualityFactorBounds.bounds.maxInclusive]], q + delta)      for each request that observed qps <= R on local host\n *   q -= delta                                      for each request that observed qps > R on local host\n *\n *   When qps r stays below R, q will stay as constant (value at [[qualityFactorBounds.bounds.maxInclusive]]).\n *   When qps r starts to increase above R, q will decrease by delta per request,\n *   with delta being an additive factor that controls how sensitive q is when max qps R is exceeded.\n *\n *   @param initialDelay Specifies an initial delay time to allow query rate counter warm up to start reflecting actual traffic load.\n *                       Qf value would only start to update after this initial delay.\n *   @param maxQueriesPerSecond The max qps the underlying component can take. Requests go above this qps threshold will cause quality factor to go down.\n *   @param queriesPerSecondSampleWindow The window of underlying query rate counter counting with and calculate an average qps over the window,\n *                                 default to count with 10 seconds time window (i.e. qps = total requests over last 10 secs / 10).\n *                                 Note: underlying query rate counter has a sliding window with 10 fixed slices. Therefore a larger\n *                                 window would lead to a coarser qps calculation. (e.g. with 60 secs time window, it sliding over 6 seconds slice (60 / 10 = 6 secs)).\n *                                 A larger time window also lead to a slower reaction to sudden qps burst, but more robust to flaky qps pattern.\n *   @param delta The step for adjusting quality factor. It should be a positive double. If the delta is large, the quality factor\n *                will fluctuate more and be more responsive to exceeding max qps, and if it is small, the quality factor will be less responsive.\n */\ncase class QueriesPerSecondBasedQualityFactorConfig(\n  override val qualityFactorBounds: BoundsWithDefault[Double],\n  override val initialDelay: Duration,\n  maxQueriesPerSecond: Int,\n  queriesPerSecondSampleWindow: Duration = 10.seconds,\n  delta: Double = 0.001,\n  override val ignorableFailures: PartialFunction[Throwable, Unit] =\n    QualityFactorConfig.defaultIgnorableFailures)\n    extends QualityFactorConfig\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor/QualityFactorObserver.scala",
    "content": "package com.twitter.product_mixer.core.quality_factor\n\nimport com.twitter.util.Duration\nimport com.twitter.util.Try\n\n/** Updates the [[QualityFactor]] */\ntrait QualityFactorObserver {\n\n  /** The [[QualityFactor]] to update when observing */\n  def qualityFactor: QualityFactor[_]\n\n  /**\n   * updates the [[qualityFactor]] given the result [[Try]] and the latency\n   * @note implementations must be sure to correctly ignore\n   *       [[QualityFactor.config]]'s [[QualityFactorConfig.ignorableFailures]]\n   */\n  def apply(result: Try[_], latency: Duration): Unit\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor/QualityFactorStatus.scala",
    "content": "package com.twitter.product_mixer.core.quality_factor\n\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MisconfiguredQualityFactor\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\n\ncase class QualityFactorStatus(\n  qualityFactorByPipeline: Map[ComponentIdentifier, QualityFactor[_]]) {\n\n  /**\n   * returns a new [[QualityFactorStatus]] with all the elements of current QualityFactorStatus and `other`.\n   * If a [[ComponentIdentifier]] exists in both maps, the Value from `other` takes precedence\n   */\n  def ++(other: QualityFactorStatus): QualityFactorStatus = {\n    if (other.qualityFactorByPipeline.isEmpty) {\n      this\n    } else if (qualityFactorByPipeline.isEmpty) {\n      other\n    } else {\n      QualityFactorStatus(qualityFactorByPipeline ++ other.qualityFactorByPipeline)\n    }\n  }\n}\n\nobject QualityFactorStatus {\n  def build[Identifier <: ComponentIdentifier](\n    qualityFactorConfigs: Map[Identifier, QualityFactorConfig]\n  ): QualityFactorStatus = {\n    QualityFactorStatus(\n      qualityFactorConfigs.map {\n        case (key, config: LinearLatencyQualityFactorConfig) =>\n          key -> LinearLatencyQualityFactor(config)\n        case (key, config: QueriesPerSecondBasedQualityFactorConfig) =>\n          key -> QueriesPerSecondBasedQualityFactor(config)\n      }\n    )\n  }\n\n  val empty: QualityFactorStatus = QualityFactorStatus(Map.empty)\n}\n\ntrait HasQualityFactorStatus {\n  def qualityFactorStatus: Option[QualityFactorStatus] = None\n  def withQualityFactorStatus(qualityFactorStatus: QualityFactorStatus): PipelineQuery\n\n  def getQualityFactorCurrentValue(\n    identifier: ComponentIdentifier\n  ): Double = getQualityFactor(identifier).currentValue\n\n  def getQualityFactor(\n    identifier: ComponentIdentifier\n  ): QualityFactor[_] = qualityFactorStatus\n    .flatMap(_.qualityFactorByPipeline.get(identifier))\n    .getOrElse {\n      throw PipelineFailure(\n        MisconfiguredQualityFactor,\n        s\"Quality factor not configured for $identifier\")\n    }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor/QueriesPerSecondBasedQualityFactor.scala",
    "content": "package com.twitter.product_mixer.core.quality_factor\n\nimport com.google.common.annotations.VisibleForTesting\nimport com.twitter.util.Stopwatch\n\ncase class QueriesPerSecondBasedQualityFactor(\n  override val config: QueriesPerSecondBasedQualityFactorConfig)\n    extends QualityFactor[Int] {\n\n  @VisibleForTesting\n  private[quality_factor] val queryRateCounter: QueryRateCounter = QueryRateCounter(\n    config.queriesPerSecondSampleWindow)\n\n  private val delayedUntilInMillis = Stopwatch.timeMillis() + config.initialDelay.inMillis\n\n  private var state: Double = config.qualityFactorBounds.default\n\n  override def currentValue: Double = state\n\n  override def update(count: Int = 1): Unit = {\n    val queryRate = incrementAndGetQueryRateCount(count)\n\n    // Only update quality factor until the initial delay past.\n    // This allows query rate counter get warm up to reflect\n    // actual traffic load by the time initial delay expires.\n    if (Stopwatch.timeMillis() >= delayedUntilInMillis) {\n      if (queryRate > config.maxQueriesPerSecond) {\n        state = config.qualityFactorBounds.bounds(state - config.delta)\n      } else {\n        state = config.qualityFactorBounds.bounds(state + config.delta)\n      }\n    }\n  }\n\n  private def incrementAndGetQueryRateCount(count: Int): Double = {\n    // Int.MaxValue is used as a special signal from [[QueriesPerSecondBasedQualityFactorObserver]]\n    // to indicate a component failure is observed.\n    // In this case, we do not update queryRateCounter and instead return Int.MaxValue.\n    // As the largest Int value, this should result in the threshold qps for quality factor being\n    // exceeded and directly decrementing quality factor.\n    if (count == Int.MaxValue) {\n      Int.MaxValue.toDouble\n    } else {\n      queryRateCounter.increment(count)\n      queryRateCounter.getRate()\n    }\n  }\n\n  override def buildObserver(): QualityFactorObserver =\n    QueriesPerSecondBasedQualityFactorObserver(this)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor/QueriesPerSecondBasedQualityFactorObserver.scala",
    "content": "package com.twitter.product_mixer.core.quality_factor\n\nimport com.twitter.util.Duration\nimport com.twitter.util.Try\n\ncase class QueriesPerSecondBasedQualityFactorObserver(\n  override val qualityFactor: QueriesPerSecondBasedQualityFactor)\n    extends QualityFactorObserver {\n  override def apply(\n    result: Try[_],\n    latency: Duration\n  ): Unit = {\n    result\n      .onSuccess(_ => qualityFactor.update())\n      .onFailure {\n        case t if qualityFactor.config.ignorableFailures.isDefinedAt(t) => ()\n        // Degrade qf as a proactive mitigation for any non ignorable failures.\n        case _ => qualityFactor.update(Int.MaxValue)\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor/QueryRateCounter.scala",
    "content": "package com.twitter.product_mixer.core.quality_factor\n\nimport com.twitter.util.Duration\nimport com.twitter.util.Stopwatch\nimport com.twitter.util.TokenBucket\n\n/**\n * Query rate counter based on a leaky bucket. For more, see [[com.twitter.util.TokenBucket]].\n */\ncase class QueryRateCounter private[quality_factor] (\n  queryRateWindow: Duration) {\n\n  private val queryRateWindowInSeconds = queryRateWindow.inSeconds\n\n  private val leakyBucket: TokenBucket =\n    TokenBucket.newLeakyBucket(ttl = queryRateWindow, reserve = 0, nowMs = Stopwatch.timeMillis)\n\n  def increment(count: Int): Unit = leakyBucket.put(count)\n\n  def getRate(): Double = leakyBucket.count / queryRateWindowInSeconds\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/observer\",\n        \"servo/util\",\n        \"stitch/stitch-core\",\n        \"util/util-core\",\n        \"util/util-stats\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline:executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/observer\",\n        \"stitch/stitch-core\",\n        \"util/util-stats\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/Executor.scala",
    "content": "package com.twitter.product_mixer.core.service\n\nimport com.twitter.finagle.stats.BroadcastStatsReceiver\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.DefaultStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.tracing.Annotation\nimport com.twitter.finagle.tracing.Record\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.finagle.tracing.TraceId\nimport com.twitter.finagle.tracing.TraceServiceName\nimport com.twitter.finagle.tracing.Tracing.LocalBeginAnnotation\nimport com.twitter.finagle.tracing.Tracing.LocalEndAnnotation\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.common.identifier.ProductPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.PipelineResult\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.FeatureHydrationFailed\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MisconfiguredFeatureMapFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureClassifier\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.UncategorizedServerFailure\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorObserver\nimport com.twitter.product_mixer.core.service.Executor.AlwaysFailOpenIncludingProgrammerErrors\nimport com.twitter.product_mixer.core.service.Executor.Context\nimport com.twitter.product_mixer.core.service.Executor.TracingConfig\nimport com.twitter.product_mixer.core.service.Executor.toPipelineFailureWithComponentIdentifierStack\nimport com.twitter.servo.util.CancelledExceptionExtractor\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.Stitch.Letter\nimport com.twitter.util.Duration\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Time\nimport com.twitter.util.Try\n\n/**\n * Base trait that all executors implement\n *\n * All executors should:\n *   - implement a `def arrow` or `def apply` with the relevant types for their use case\n *     and take in an implicit [[PipelineFailureClassifier]] and [[ComponentIdentifierStack]].\n *   - add a `@singleton` annotation to the class and `@inject` annotation to the argument list\n *   - take in a [[StatsReceiver]]\n *\n * @example {{{\n *   @Singleton class MyExecutor @Inject() (\n *     override val statsReceiver: StatsReceiver\n *   ) extends Executor {\n *     def arrow(\n *       arg: MyArg,\n *       ...,\n *       context: Context\n *     ): Arrow[In,Out] = ???\n *   }\n * }}}\n */\nprivate[core] trait Executor {\n  val statsReceiver: StatsReceiver\n\n  /**\n   * Applies the `pipelineFailureClassifier` to the output of the `arrow`\n   * and adds the `componentStack` to the [[PipelineFailure]]\n   */\n  def wrapWithErrorHandling[In, Out](\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier\n  )(\n    arrow: Arrow[In, Out]\n  ): Arrow[In, Out] = {\n    arrow.mapFailure(\n      toPipelineFailureWithComponentIdentifierStack(context, currentComponentIdentifier))\n  }\n\n  /**\n   * Chain a `Seq` of [[Arrow.Iso]], only passing successful results to the next [[Arrow.Iso]]\n   *\n   * @note the resulting [[Arrow]] runs the passed in [[Arrow]]s one after the other,\n   *       as an ordered execution, this means that each [[Arrow]] is dependent\n   *       on all previous [[Arrow]]s in the `Seq` so no `Stitch` batching can occur\n   *       between them.\n   */\n  def isoArrowsSequentially[T](arrows: Seq[Arrow.Iso[T]]): Arrow.Iso[T] = {\n    // avoid excess Arrow complexity when there is only a single Arrow\n    arrows match {\n      case Seq() => Arrow.identity\n      case Seq(onlyOneArrow) => onlyOneArrow\n      case Seq(head, tail @ _*) =>\n        tail.foldLeft(head) {\n          case (combinedArrow, nextArrow) => combinedArrow.flatMapArrow(nextArrow)\n        }\n    }\n  }\n\n  /**\n   * Start running the [[Arrow]] in the background returning a [[Stitch.Ref]] which will complete\n   * when the background task is finished\n   */\n  def startArrowAsync[In, Out](arrow: Arrow[In, Out]): Arrow[In, Stitch[Out]] = {\n    Arrow\n      .map { arg: In =>\n        // wrap in a `ref` so we only compute it's value once\n        Stitch.ref(arrow(arg))\n      }\n      .andThen(\n        Arrow.zipWithArg(\n          // satisfy the `ref` async\n          Arrow.async(Arrow.flatMap[Stitch[Out], Out](identity))))\n      .map { case (ref, _) => ref }\n  }\n\n  /**\n   * for [[com.twitter.product_mixer.core.model.common.Component]]s which\n   * are executed per-candidate or which we don't want to record stats for.\n   * This performs Tracing but does not record Stats\n   *\n   * @note This should be used around the computation that includes the execution of the\n   *       underlying Component over all the Candidates, not around each execution\n   *        of the component around each candidate for per-candidate Components.\n   *\n   * @note when using this you should only use [[wrapPerCandidateComponentWithExecutorBookkeepingWithoutTracing]]\n   *       for handling Stats.\n   */\n  def wrapComponentsWithTracingOnly[In, Out](\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier\n  )(\n    arrow: Arrow[In, Out]\n  ): Arrow[In, Out] = {\n    Executor.wrapArrowWithLocalTracingSpan(\n      Arrow\n        .time(arrow)\n        .map {\n          case (result, latency) =>\n            Executor.recordTraceData(\n              componentStack = context.componentStack,\n              componentIdentifier = currentComponentIdentifier,\n              result = result,\n              latency = latency,\n              size = None)\n            result\n        }.lowerFromTry)\n  }\n\n  /**\n   * for [[com.twitter.product_mixer.core.model.common.Component]]s which\n   * are executed per-candidate. Records Stats but does not do Tracing.\n   *\n   * @note when using this you should only use [[wrapPerCandidateComponentsWithTracingOnly]]\n   *       for handling Tracing\n   */\n  def wrapPerCandidateComponentWithExecutorBookkeepingWithoutTracing[In, Out](\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier\n  )(\n    arrow: Arrow[In, Out]\n  ): Arrow[In, Out] = {\n    val observerSideEffect =\n      ExecutorObserver.executorObserver[Out](context, currentComponentIdentifier, statsReceiver)\n\n    Executor.wrapWithExecutorBookkeeping[In, Out, Out](\n      context = context,\n      currentComponentIdentifier = currentComponentIdentifier,\n      executorResultSideEffect = observerSideEffect,\n      transformer = Return(_),\n      tracingConfig = TracingConfig.NoTracing\n    )(arrow)\n  }\n\n  /** for [[com.twitter.product_mixer.core.model.common.Component]]s */\n  def wrapComponentWithExecutorBookkeeping[In, Out](\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier\n  )(\n    arrow: Arrow[In, Out]\n  ): Arrow[In, Out] = {\n    val observerSideEffect =\n      ExecutorObserver.executorObserver[Out](context, currentComponentIdentifier, statsReceiver)\n\n    Executor.wrapWithExecutorBookkeeping[In, Out, Out](\n      context = context,\n      currentComponentIdentifier = currentComponentIdentifier,\n      executorResultSideEffect = observerSideEffect,\n      transformer = Return(_)\n    )(arrow)\n  }\n\n  /**\n   * for [[com.twitter.product_mixer.core.model.common.Component]]s which an `onSuccess`\n   * to add custom stats or logging of results\n   */\n  def wrapComponentWithExecutorBookkeeping[In, Out](\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier,\n    onSuccess: Out => Unit\n  )(\n    arrow: Arrow[In, Out]\n  ): Arrow[In, Out] = {\n    val observerSideEffect =\n      ExecutorObserver.executorObserver[Out](context, currentComponentIdentifier, statsReceiver)\n\n    Executor.wrapWithExecutorBookkeeping[In, Out, Out](\n      context = context,\n      currentComponentIdentifier = currentComponentIdentifier,\n      executorResultSideEffect = observerSideEffect,\n      transformer = Return(_),\n      onComplete = (transformed: Try[Out]) => transformed.onSuccess(onSuccess)\n    )(arrow)\n  }\n\n  /** for [[com.twitter.product_mixer.core.pipeline.Pipeline]]s */\n  def wrapPipelineWithExecutorBookkeeping[In, Out <: PipelineResult[_]](\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier,\n    qualityFactorObserver: Option[QualityFactorObserver],\n    failOpenPolicy: FailOpenPolicy = FailOpenPolicy.Never\n  )(\n    arrow: Arrow[In, Out]\n  ): Arrow[In, Out] = {\n    val observerSideEffect =\n      ExecutorObserver\n        .pipelineExecutorObserver[Out](context, currentComponentIdentifier, statsReceiver)\n\n    Executor.wrapWithExecutorBookkeeping[In, Out, Out](\n      context = context,\n      currentComponentIdentifier = currentComponentIdentifier,\n      executorResultSideEffect = observerSideEffect,\n      transformer = (result: Out) => result.toTry,\n      size = Some(_.resultSize()),\n      failOpenPolicy = failOpenPolicy,\n      qualityFactorObserver = qualityFactorObserver\n    )(arrow)\n  }\n\n  /** for [[com.twitter.product_mixer.core.pipeline.product.ProductPipeline]]s */\n  def wrapProductPipelineWithExecutorBookkeeping[In, Out <: PipelineResult[_]](\n    context: Context,\n    currentComponentIdentifier: ProductPipelineIdentifier\n  )(\n    arrow: Arrow[In, Out]\n  ): Arrow[In, Out] = {\n\n    val observerSideEffect =\n      ExecutorObserver\n        .productPipelineExecutorObserver[Out](currentComponentIdentifier, statsReceiver)\n\n    Executor.wrapWithExecutorBookkeeping[In, Out, Out](\n      context = context,\n      currentComponentIdentifier = currentComponentIdentifier,\n      executorResultSideEffect = observerSideEffect,\n      transformer = _.toTry,\n      size = Some(_.resultSize()),\n      failOpenPolicy =\n        // always save Failures in the Result object instead of failing the request\n        AlwaysFailOpenIncludingProgrammerErrors\n    )(arrow)\n  }\n\n  /** for [[com.twitter.product_mixer.core.model.common.Component]]s which need a result size stat */\n  def wrapComponentWithExecutorBookkeepingWithSize[In, Out](\n    context: Context,\n    currentComponentIdentifier: CandidateSourceIdentifier,\n    size: Out => Int\n  )(\n    arrow: Arrow[In, Out]\n  ): Arrow[In, Out] = {\n    val observerSideEffect =\n      ExecutorObserver.executorObserverWithSize(context, currentComponentIdentifier, statsReceiver)\n\n    Executor.wrapWithExecutorBookkeeping[In, Out, Int](\n      context = context,\n      currentComponentIdentifier = currentComponentIdentifier,\n      executorResultSideEffect = observerSideEffect,\n      transformer = (out: Out) => Try(size(out)),\n      size = Some(identity)\n    )(arrow)\n  }\n\n  /** for [[com.twitter.product_mixer.core.pipeline.PipelineBuilder.Step]]s */\n  def wrapStepWithExecutorBookkeeping[In, Out](\n    context: Context,\n    identifier: PipelineStepIdentifier,\n    arrow: Arrow[In, Out],\n    transformer: Out => Try[Unit]\n  ): Arrow[In, Out] = {\n    val observerSideEffect =\n      ExecutorObserver.stepExecutorObserver(context, identifier, statsReceiver)\n\n    Executor.wrapWithExecutorBookkeeping[In, Out, Unit](\n      context = context,\n      currentComponentIdentifier = identifier,\n      executorResultSideEffect = observerSideEffect,\n      transformer = transformer,\n      failOpenPolicy = AlwaysFailOpenIncludingProgrammerErrors\n    )(arrow)\n  }\n}\n\nprivate[core] object Executor {\n\n  private[service] object TracingConfig {\n\n    /** Used to specify whether a wrapped Arrow should be Traced in [[wrapWithExecutorBookkeeping]] */\n    sealed trait TracingConfig\n    case object NoTracing extends TracingConfig\n    case object WrapWithSpanAndTracingData extends TracingConfig\n  }\n\n  /**\n   * Always fail-open and return the [[com.twitter.product_mixer.core.pipeline.product.ProductPipelineResult]]\n   * containing the exception, this differs from [[FailOpenPolicy.Always]] in that this will still\n   * fail-open and return the overall result object even if the underlying failure is the result\n   * of programmer error.\n   */\n  private val AlwaysFailOpenIncludingProgrammerErrors: FailOpenPolicy = _ => true\n\n  /**\n   * Wraps an [[Arrow]] so that bookkeeping around the execution occurs uniformly.\n   *\n   * @note should __never__ be called directly!\n   *\n   *   - For successful results, apply the `transformer`\n   *   - convert any exceptions to PipelineFailures\n   *   - record stats and update [[QualityFactorObserver]]\n   *   - wraps the execution in a Trace span and record Trace data (can be turned off by [[TracingConfig]])\n   *   - applies a trace span and records metadata to the provided `arrow`\n   *   - determine whether to fail-open based on `result.flatMap(transformer)`\n   *     - if failing-open, always return the original result\n   *     - if failing-closed and successful, return the original result\n   *     - otherwise, return the failure (from `result.flatMap(transformer)`)\n   *\n   * @param context                    the [[Executor.Context]]\n   * @param currentComponentIdentifier the current component's [[ComponentIdentifier]]\n   * @param executorResultSideEffect   the [[ExecutorObserver]] used to record stats\n   * @param transformer                function to convert a successful result into possibly a failed result\n   * @param failOpenPolicy             [[FailOpenPolicy]] to apply to the results of `result.flatMap(transformer)`\n   * @param qualityFactorObserver      [[QualityFactorObserver]] to update based on the results of `result.flatMap(transformer)`\n   * @param tracingConfig              indicates whether the [[Arrow]] should be wrapped with Tracing\n   * @param onComplete                 runs the function for its side effects with the result of `result.flatMap(transformer)`\n   * @param arrow                      an input [[Arrow]] to wrap so that after it's execution, we perform all these operations\n   *\n   * @return the wrapped [[Arrow]]\n   */\n  private[service] def wrapWithExecutorBookkeeping[In, Out, Transformed](\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier,\n    executorResultSideEffect: ExecutorObserver[Transformed],\n    transformer: Out => Try[Transformed],\n    size: Option[Transformed => Int] = None,\n    failOpenPolicy: FailOpenPolicy = FailOpenPolicy.Never,\n    qualityFactorObserver: Option[QualityFactorObserver] = None,\n    tracingConfig: TracingConfig.TracingConfig = TracingConfig.WrapWithSpanAndTracingData,\n    onComplete: Try[Transformed] => Unit = { _: Try[Transformed] => () }\n  )(\n    arrow: Arrow[In, Out]\n  ): Arrow[In, Out] = {\n\n    val failureClassifier =\n      toPipelineFailureWithComponentIdentifierStack(context, currentComponentIdentifier)\n\n    /** transform the results, mapping all exceptions to [[PipelineFailure]]s, and tuple with original result */\n    val transformResultAndClassifyFailures: Arrow[Out, (Out, Try[Transformed])] =\n      Arrow.join(\n        Arrow.mapFailure(failureClassifier),\n        Arrow\n          .transformTry[Out, Transformed](result =>\n            result\n              .flatMap(transformer)\n              .rescue { case t => Throw(failureClassifier(t)) })\n          .liftToTry\n      )\n\n    /** Only record tracing data if [[TracingConfig.WrapWithSpanAndTracingData]] */\n    val maybeRecordTracingData: (Try[Transformed], Duration) => Unit = tracingConfig match {\n      case TracingConfig.NoTracing => (_, _) => ()\n      case TracingConfig.WrapWithSpanAndTracingData =>\n        (transformedAndClassifiedResult, latency) =>\n          recordTraceData(\n            context.componentStack,\n            currentComponentIdentifier,\n            transformedAndClassifiedResult,\n            latency,\n            transformedAndClassifiedResult.toOption.flatMap(result => size.map(_.apply(result)))\n          )\n    }\n\n    /** Will never be in a failed state so we can do a simple [[Arrow.map]] */\n    val recordStatsAndUpdateQualityFactor =\n      Arrow\n        .map[(Try[(Out, Try[Transformed])], Duration), Unit] {\n          case (tryResultAndTryTransformed, latency) =>\n            val transformedAndClassifiedResult = tryResultAndTryTransformed.flatMap {\n              case (_, transformed) => transformed\n            }\n            executorResultSideEffect(transformedAndClassifiedResult, latency)\n            qualityFactorObserver.foreach(_.apply(transformedAndClassifiedResult, latency))\n            onComplete(transformedAndClassifiedResult)\n            maybeRecordTracingData(transformedAndClassifiedResult, latency)\n        }.unit\n\n    /**\n     * Applies the provided [[FailOpenPolicy]] based on the [[transformer]]'s results,\n     * returning the original result or an exception\n     */\n    val applyFailOpenPolicyBasedOnTransformedResult: Arrow[\n      (Try[(Out, Try[Transformed])], Duration),\n      Out\n    ] =\n      Arrow\n        .map[(Try[(Out, Try[Transformed])], Duration), Try[(Out, Try[Transformed])]] {\n          case (tryResultAndTryTransformed, _) => tryResultAndTryTransformed\n        }\n        .lowerFromTry\n        .map {\n          case (result, Throw(pipelineFailure: PipelineFailure))\n              if failOpenPolicy(pipelineFailure.category) =>\n            Return(result)\n          case (_, t: Throw[_]) => t.asInstanceOf[Throw[Out]]\n          case (result, _) => Return(result)\n        }.lowerFromTry\n\n    /** The complete Arrow minus a Local span wrapping */\n    val arrowWithTimingExecutorSideEffects = Arrow\n      .time(arrow.andThen(transformResultAndClassifyFailures))\n      .applyEffect(recordStatsAndUpdateQualityFactor)\n      .andThen(applyFailOpenPolicyBasedOnTransformedResult)\n\n    /** Dont wrap with a span if we are not tracing */\n    tracingConfig match {\n      case TracingConfig.WrapWithSpanAndTracingData =>\n        wrapArrowWithLocalTracingSpan(arrowWithTimingExecutorSideEffects)\n      case TracingConfig.NoTracing =>\n        arrowWithTimingExecutorSideEffects\n    }\n  }\n\n  /** Let-scopes a [[TraceId]] around a computation */\n  private[this] object TracingLetter extends Letter[TraceId] {\n    override def let[S](traceId: TraceId)(s: => S): S = Trace.letId(traceId)(s)\n  }\n\n  /**\n   * Wraps the Arrow's execution in a new trace span as a child of the current parent span\n   *\n   * @note Should __never__ be called directly!\n   *\n   * It's expected that the contained `arrow` will invoke [[recordTraceData]] exactly ONCE\n   * during it's execution.\n   *\n   * @note this does not record any data about the trace, it only sets the [[Trace]] Span\n   *       for the execution of `arrow`\n   */\n  private[service] def wrapArrowWithLocalTracingSpan[In, Out](\n    arrow: Arrow[In, Out]\n  ): Arrow[In, Out] =\n    Arrow.ifelse(\n      _ => Trace.isActivelyTracing,\n      Arrow.let(TracingLetter)(Trace.nextId)(arrow),\n      arrow\n    )\n\n  private[this] object Tracing {\n\n    /**\n     * Duplicate of [[com.twitter.finagle.tracing.Tracing]]'s `localSpans` which\n     * uses an un-scoped [[StatsReceiver]]\n     *\n     * Since we needed to roll-our-own latency measurement we are unable to increment the\n     * `local_spans` metric automatically, this is important in the event a service is\n     * unexpectedly not recording spans or unexpectedly recording too many, so we manually\n     * increment it\n     */\n    val localSpans: Counter = DefaultStatsReceiver.counter(\"tracing\", \"local_spans\")\n\n    /** Local Component field of a span in the UI */\n    val localComponentTag = \"lc\"\n    val sizeTag = \"product_mixer.result.size\"\n    val successTag = \"product_mixer.result.success\"\n    val successValue = \"success\"\n    val cancelledTag = \"product_mixer.result.cancelled\"\n    val failureTag = \"product_mixer.result.failure\"\n  }\n\n  /**\n   * Records metadata onto the current [[Trace]] Span\n   *\n   * @note Should __never__ be called directly!\n   *\n   * This should be called exactly ONCE in the Arrow passed into [[wrapArrowWithLocalTracingSpan]]\n   * to record data for the Span.\n   */\n  private[service] def recordTraceData[T](\n    componentStack: ComponentIdentifierStack,\n    componentIdentifier: ComponentIdentifier,\n    result: Try[T],\n    latency: Duration,\n    size: Option[Int] = None\n  ): Unit = {\n    if (Trace.isActivelyTracing) {\n      Tracing.localSpans.incr()\n      val traceId = Trace.id\n      val endTime = Time.nowNanoPrecision\n\n      // These annotations are needed for the Zipkin UI to display the span properly\n      TraceServiceName().foreach(Trace.recordServiceName)\n      Trace.recordRpc(componentIdentifier.snakeCase) // name of the span in the UI\n      Trace.recordBinary(\n        Tracing.localComponentTag,\n        componentStack.peek.toString + \"/\" + componentIdentifier.toString)\n      Trace.record(Record(traceId, endTime - latency, Annotation.Message(LocalBeginAnnotation)))\n      Trace.record(Record(traceId, endTime, Annotation.Message(LocalEndAnnotation)))\n\n      // product mixer specific zipkin data\n      size.foreach(size => Trace.recordBinary(Tracing.sizeTag, size))\n      result match {\n        case Return(_) =>\n          Trace.recordBinary(Tracing.successTag, Tracing.successValue)\n        case Throw(CancelledExceptionExtractor(e)) =>\n          Trace.recordBinary(Tracing.cancelledTag, e)\n        case Throw(e) =>\n          Trace.recordBinary(Tracing.failureTag, e)\n      }\n    }\n  }\n\n  /**\n   * Returns a tuple of the stats scopes for the current component and the relative scope for\n   * the parent component and the current component together\n   *\n   * This is useful when recording stats for a component by itself as well as stats for calls to that component from it's parent.\n   *\n   * @example if the current component has a scope of \"currentComponent\" and the parent component has a scope of \"parentComponent\"\n   *          then this will return `(Seq(\"currentComponent\"), Seq(\"parentComponent\", \"currentComponent\"))`\n   */\n  def buildScopes(\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier\n  ): Executor.Scopes = {\n    val parentScopes = context.componentStack.peek.toScopes\n    val componentScopes = currentComponentIdentifier.toScopes\n    val relativeScopes = parentScopes ++ componentScopes\n    Executor.Scopes(componentScopes, relativeScopes)\n  }\n\n  /**\n   * Makes a [[BroadcastStatsReceiver]] that will broadcast stats to the correct\n   * current component's scope and to the scope relative to the parent.\n   */\n  def broadcastStatsReceiver(\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier,\n    statsReceiver: StatsReceiver\n  ): StatsReceiver = {\n    val Executor.Scopes(componentScopes, relativeScopes) =\n      Executor.buildScopes(context, currentComponentIdentifier)\n\n    BroadcastStatsReceiver(\n      Seq(statsReceiver.scope(relativeScopes: _*), statsReceiver.scope(componentScopes: _*)))\n  }\n\n  /**\n   * Returns a feature map containing all the [[com.twitter.product_mixer.core.feature.Feature]]s\n   * stored as failures using the exception provided with as the reason wrapped in a PipelineFailure.\n   * e.g, for features A & B that threw an ExampleException b, this will return:\n   * { A -> Throw(PipelineFailure(...)), B -> Throw(PipelineFailure(...)) }\n   */\n  def featureMapWithFailuresForFeatures(\n    features: Set[Feature[_, _]],\n    error: Throwable,\n    context: Executor.Context\n  ): FeatureMap = {\n    val builder = FeatureMapBuilder()\n    features.foreach { feature =>\n      val pipelineFailure = PipelineFailure(\n        FeatureHydrationFailed,\n        s\"Feature hydration failed for ${feature.toString}\",\n        Some(error),\n        Some(context.componentStack))\n      builder.addFailure(feature, pipelineFailure)\n    }\n    builder.build()\n  }\n\n  /**\n   * Validates and returns back the passed feature map if it passes validation. A feature map\n   * is considered valid if it contains only the passed `registeredFeatures` features in it,\n   * nothing else and nothing missing.\n   */\n  @throws(classOf[PipelineFailure])\n  def validateFeatureMap(\n    registeredFeatures: Set[Feature[_, _]],\n    featureMap: FeatureMap,\n    context: Executor.Context\n  ): FeatureMap = {\n    val hydratedFeatures = featureMap.getFeatures\n    if (hydratedFeatures == registeredFeatures) {\n      featureMap\n    } else {\n      val missingFeatures = registeredFeatures -- hydratedFeatures\n      val unregisteredFeatures = hydratedFeatures -- registeredFeatures\n      throw PipelineFailure(\n        MisconfiguredFeatureMapFailure,\n        s\"Unregistered features $unregisteredFeatures and missing features $missingFeatures\",\n        None,\n        Some(context.componentStack)\n      )\n    }\n  }\n\n  object NotAMisconfiguredFeatureMapFailure {\n\n    /**\n     * Will return any exception that isn't a [[MisconfiguredFeatureMapFailure]] [[PipelineFailure]]\n     * Allows for easy [[Arrow.handle]]ing all exceptions that aren't [[MisconfiguredFeatureMapFailure]]s\n     */\n    def unapply(e: Throwable): Option[Throwable] = e match {\n      case pipelineFailure: PipelineFailure\n          if pipelineFailure.category == MisconfiguredFeatureMapFailure =>\n        None\n      case e => Some(e)\n    }\n  }\n\n  /**\n   * contains the scopes for recording metrics for the component by itself and\n   * the relative scope of that component within it's parent component scope\n   *\n   * @see [[Executor.buildScopes]]\n   */\n  case class Scopes(componentScopes: Seq[String], relativeScope: Seq[String])\n\n  /**\n   * Wrap the [[Throwable]] in a [[UncategorizedServerFailure]] [[PipelineFailure]] with the original\n   * [[Throwable]] as the cause, even if it's already a [[PipelineFailure]].\n   *\n   * This ensures that any access to the stored feature will result in a meaningful [[UncategorizedServerFailure]]\n   * [[com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureCategory]] in stats which is more useful\n   * for customers components which access a failed [[Feature]] than the original [[com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureCategory]].\n   */\n  def uncategorizedServerFailure(\n    componentStack: ComponentIdentifierStack,\n    throwable: Throwable\n  ): PipelineFailure = {\n    PipelineFailure(\n      UncategorizedServerFailure,\n      reason = \"Unclassified Failure in Pipeline\",\n      Some(throwable),\n      Some(componentStack)\n    )\n  }\n\n  /**\n   * [[PartialFunction]] that converts any [[Throwable]] into a\n   * [[PipelineFailure]] based on the provided `failureClassifier`\n   */\n  def toPipelineFailureWithComponentIdentifierStack(\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier\n  ): PipelineFailureClassifier = {\n    // if given a `currentComponentIdentifier` then ensure we correctly handle `BasedOnParentComponent` identifier types\n    val contextWithCurrentComponentIdentifier =\n      context.pushToComponentStack(currentComponentIdentifier)\n    PipelineFailureClassifier(\n      contextWithCurrentComponentIdentifier.pipelineFailureClassifier\n        .orElse[Throwable, PipelineFailure] {\n          case CancelledExceptionExtractor(throwable) => throw throwable\n          case pipelineFailure: PipelineFailure => pipelineFailure\n          case throwable =>\n            uncategorizedServerFailure(\n              contextWithCurrentComponentIdentifier.componentStack,\n              throwable)\n        }.andThen { pipelineFailure =>\n          pipelineFailure.componentStack match {\n            case _: Some[_] => pipelineFailure\n            case None =>\n              pipelineFailure.copy(componentStack =\n                Some(contextWithCurrentComponentIdentifier.componentStack))\n          }\n        }\n    )\n  }\n\n  /**\n   * information used by an [[Executor]] that provides context around execution\n   */\n  case class Context(\n    pipelineFailureClassifier: PipelineFailureClassifier,\n    componentStack: ComponentIdentifierStack) {\n\n    def pushToComponentStack(newComponentIdentifier: ComponentIdentifier): Context =\n      copy(componentStack = componentStack.push(newComponentIdentifier))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/ExecutorObserver.scala",
    "content": "package com.twitter.product_mixer.core.service\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ProductPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineResult\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.service.Executor.Context\nimport com.twitter.product_mixer.shared_library.observer.Observer\nimport com.twitter.product_mixer.shared_library.observer.Observer.Observer\nimport com.twitter.product_mixer.shared_library.observer.ResultsStatsObserver.ResultsStatsObserver\nimport com.twitter.util.Duration\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\n\nprivate[core] object ExecutorObserver {\n\n  /** Make a [[ExecutorObserver]] with stats for the [[ComponentIdentifier]] and relative to the parent in the [[Context.componentStack]] */\n  def executorObserver[T](\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier,\n    statsReceiver: StatsReceiver\n  ): ExecutorObserver[T] = new ExecutorObserver[T](\n    Executor.broadcastStatsReceiver(context, currentComponentIdentifier, statsReceiver))\n\n  /** Make a [[ExecutorObserverWithSize]] with stats for the [[ComponentIdentifier]] and relative to the parent in the [[Context.componentStack]] */\n  def executorObserverWithSize(\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier,\n    statsReceiver: StatsReceiver\n  ): ExecutorObserverWithSize = new ExecutorObserverWithSize(\n    Executor.broadcastStatsReceiver(context, currentComponentIdentifier, statsReceiver))\n\n  /** Make a [[PipelineExecutorObserver]] with stats for the [[ComponentIdentifier]] and relative to the parent in the [[Context.componentStack]] */\n  def pipelineExecutorObserver[T <: PipelineResult[_]](\n    context: Context,\n    currentComponentIdentifier: ComponentIdentifier,\n    statsReceiver: StatsReceiver\n  ): PipelineExecutorObserver[T] = new PipelineExecutorObserver[T](\n    Executor.broadcastStatsReceiver(context, currentComponentIdentifier, statsReceiver))\n\n  /**\n   * Make a [[PipelineExecutorObserver]] specifically for a [[com.twitter.product_mixer.core.pipeline.product.ProductPipeline]]\n   * with no relative stats\n   */\n  def productPipelineExecutorObserver[T <: PipelineResult[_]](\n    currentComponentIdentifier: ProductPipelineIdentifier,\n    statsReceiver: StatsReceiver\n  ): PipelineExecutorObserver[T] =\n    new PipelineExecutorObserver[T](statsReceiver.scope(currentComponentIdentifier.toScopes: _*))\n\n  /**\n   * Make a [[PipelineExecutorObserver]] with only stats relative to the parent pipeline\n   * for [[com.twitter.product_mixer.core.pipeline.PipelineBuilder.Step]]s\n   */\n  def stepExecutorObserver(\n    context: Context,\n    currentComponentIdentifier: PipelineStepIdentifier,\n    statsReceiver: StatsReceiver\n  ): ExecutorObserver[Unit] = {\n    new ExecutorObserver[Unit](\n      statsReceiver.scope(\n        Executor.buildScopes(context, currentComponentIdentifier).relativeScope: _*))\n  }\n}\n\n/**\n * An [[Observer]] which is called as a side effect. Unlike the other observers which wrap a computation,\n * this [[Observer]] expects the caller to provide the latency value and wire it in\n */\nprivate[core] sealed class ExecutorObserver[T](\n  override val statsReceiver: StatsReceiver)\n    extends {\n\n  /**\n   * always empty because we expect an already scoped [[com.twitter.finagle.stats.BroadcastStatsReceiver]] to be passed in\n   * @note uses early definitions [[https://docs.scala-lang.org/tutorials/FAQ/initialization-order.html]] to avoid null values for `scopes` in [[Observer]]\n   */\n  override val scopes: Seq[String] = Seq.empty\n} with Observer[T] {\n\n  /**\n   * Serialize the provided [[Throwable]], prefixing [[PipelineFailure]]s with their\n   * [[com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureCategory.categoryName]] and\n   * [[com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureCategory.failureName]]\n   */\n  override def serializeThrowable(throwable: Throwable): Seq[String] = {\n    throwable match {\n      case PipelineFailure(category, _, None, _) =>\n        Seq(category.categoryName, category.failureName)\n      case PipelineFailure(category, _, Some(underlying), _) =>\n        Seq(category.categoryName, category.failureName) ++ serializeThrowable(underlying)\n      case throwable: Throwable => super.serializeThrowable(throwable)\n    }\n  }\n\n  /** record success, failure, and latency stats based on `t` and `latency` */\n  def apply(t: Try[T], latency: Duration): Unit = observe(t, latency)\n}\n\n/**\n * Same as [[ExecutorObserver]] but records a size stat for [[PipelineResult]]s and\n * records a failure counter for the cause of the failure under `failures/$pipelineFailureCategory/$componentType/$componentName`.\n *\n * @example if `GateIdentifier(\"MyGate\")` is at the top of the [[PipelineFailure.componentStack]] then\n *          the resulting metric `failures/ClientFailure/Gate/MyGate` will be incremented.\n */\nprivate[core] final class PipelineExecutorObserver[T <: PipelineResult[_]](\n  override val statsReceiver: StatsReceiver)\n    extends ExecutorObserver[T](statsReceiver)\n    with ResultsStatsObserver[T] {\n  override val size: T => Int = _.resultSize()\n\n  override def apply(t: Try[T], latency: Duration): Unit = {\n    super.apply(t, latency)\n    t match {\n      case Return(result) => observeResults(result)\n      case Throw(PipelineFailure(category, _, _, Some(componentIdentifierStack))) =>\n        statsReceiver\n          .counter(\n            Seq(\n              Observer.Failures,\n              category.categoryName,\n              category.failureName) ++ componentIdentifierStack.peek.toScopes: _*).incr()\n      case _ =>\n    }\n  }\n}\n\n/** Same as [[ExecutorObserver]] but records a size stat */\nprivate[core] final class ExecutorObserverWithSize(\n  override val statsReceiver: StatsReceiver)\n    extends ExecutorObserver[Int](statsReceiver)\n    with ResultsStatsObserver[Int] {\n  override val size: Int => Int = identity\n\n  override def apply(t: Try[Int], latency: Duration): Unit = {\n    super.apply(t, latency)\n    t match {\n      case Return(result) => observeResults(result)\n      case _ =>\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/ExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service\n\ntrait ExecutorResult\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/async_feature_map_executor/AsyncFeatureMapExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.async_feature_map_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.asyncfeaturemap.AsyncFeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.Executor._\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass AsyncFeatureMapExecutor @Inject() (\n  override val statsReceiver: StatsReceiver)\n    extends Executor {\n\n  /**\n   * Forces an [[AsyncFeatureMap]] to hydrate and resolve into a [[FeatureMap]]\n   * containing all [[com.twitter.product_mixer.core.feature.Feature]]s that are\n   * supposed to be hydrated before `stepToHydrateBefore`.\n   */\n  def arrow(\n    stepToHydrateFor: PipelineStepIdentifier,\n    currentStep: PipelineStepIdentifier,\n    context: Context\n  ): Arrow[AsyncFeatureMap, AsyncFeatureMapExecutorResults] = {\n    Arrow\n      .map[AsyncFeatureMap, Option[Stitch[FeatureMap]]](_.hydrate(stepToHydrateFor))\n      .andThen(\n        Arrow.choose(\n          Arrow.Choice.ifDefinedAt(\n            { case Some(stitchOfFeatureMap) => stitchOfFeatureMap },\n            // only stat if there's something to hydrate\n            wrapComponentWithExecutorBookkeeping(context, currentStep)(\n              Arrow\n                .flatMap[Stitch[FeatureMap], FeatureMap](identity)\n                .map(featureMap =>\n                  AsyncFeatureMapExecutorResults(Map(stepToHydrateFor -> featureMap)))\n            )\n          ),\n          Arrow.Choice.otherwise(Arrow.value(AsyncFeatureMapExecutorResults(Map.empty)))\n        )\n      )\n  }\n}\n\ncase class AsyncFeatureMapExecutorResults(\n  featureMapsByStep: Map[PipelineStepIdentifier, FeatureMap])\n    extends ExecutorResult {\n  def ++(\n    asyncFeatureMapExecutorResults: AsyncFeatureMapExecutorResults\n  ): AsyncFeatureMapExecutorResults =\n    AsyncFeatureMapExecutorResults(\n      featureMapsByStep ++ asyncFeatureMapExecutorResults.featureMapsByStep)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/async_feature_map_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/asyncfeaturemap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/asyncfeaturemap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_decorator_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/decorator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/presentation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_decorator_executor/CandidateDecoratorExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.candidate_decorator_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.decorator.CandidateDecorator\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.stitch.Arrow\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CandidateDecoratorExecutor @Inject() (override val statsReceiver: StatsReceiver)\n    extends Executor {\n  def arrow[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n    decoratorOpt: Option[CandidateDecorator[Query, Candidate]],\n    context: Executor.Context\n  ): Arrow[(Query, Seq[CandidateWithFeatures[Candidate]]), CandidateDecoratorExecutorResult] = {\n    val decoratorArrow =\n      decoratorOpt match {\n        case Some(decorator) =>\n          val candidateDecoratorArrow =\n            Arrow.flatMap[(Query, Seq[CandidateWithFeatures[Candidate]]), Seq[Decoration]] {\n              case (query, candidatesWithFeatures) => decorator.apply(query, candidatesWithFeatures)\n            }\n\n          wrapComponentWithExecutorBookkeeping(context, decorator.identifier)(\n            candidateDecoratorArrow)\n\n        case _ => Arrow.value(Seq.empty[Decoration])\n      }\n\n    decoratorArrow.map(CandidateDecoratorExecutorResult)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_decorator_executor/CandidateDecoratorExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service.candidate_decorator_executor\n\nimport com.twitter.product_mixer.core.functional_component.decorator.Decoration\nimport com.twitter.product_mixer.core.service.ExecutorResult\n\ncase class CandidateDecoratorExecutorResult(result: Seq[Decoration]) extends ExecutorResult\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_hydrator_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/feature_hydrator_observer\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_hydrator_executor/CandidateFeatureHydratorExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseBulkCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseCandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.HydratorCandidateResult\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1.FeatureStoreV1CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.MisconfiguredFeatureMapFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.Executor._\nimport com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor.CandidateFeatureHydratorExecutor.Inputs\nimport com.twitter.product_mixer.core.service.feature_hydrator_observer.FeatureHydratorObserver\nimport com.twitter.stitch.Arrow\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CandidateFeatureHydratorExecutor @Inject() (override val statsReceiver: StatsReceiver)\n    extends Executor {\n  def arrow[Query <: PipelineQuery, Result <: UniversalNoun[Any]](\n    hydrators: Seq[BaseCandidateFeatureHydrator[Query, Result, _]],\n    context: Executor.Context\n  ): Arrow[\n    Inputs[Query, Result],\n    CandidateFeatureHydratorExecutorResult[\n      Result\n    ]\n  ] = {\n\n    val observer = new FeatureHydratorObserver(statsReceiver, hydrators, context)\n\n    val candidateFeatureHydratorExecutorResults: Seq[Arrow[\n      Inputs[Query, Result],\n      CandidateFeatureHydratorExecutorResult[Result]\n    ]] = hydrators.map(getCandidateHydratorArrow(_, context, observer))\n\n    val runHydrators = Arrow.collect(candidateFeatureHydratorExecutorResults).map {\n      candidateFeatureHydratorExecutorResult: Seq[CandidateFeatureHydratorExecutorResult[Result]] =>\n        candidateFeatureHydratorExecutorResult.foldLeft(\n          CandidateFeatureHydratorExecutorResult[Result](\n            Seq.empty,\n            Map.empty\n          )\n        ) { (accumulator, additionalResult) =>\n          // accumulator.results and additionalResults.results are either the same length or one may be empty\n          // checks in each Hydrator's Arrow implementation ensure the ordering and length are correct\n          val mergedFeatureMaps =\n            if (accumulator.results.length == additionalResult.results.length) {\n              // merge if there are results for both and they are the same size\n              // also handles both being empty\n              accumulator.results.zip(additionalResult.results).map {\n                case (accumulatedScoredCandidate, resultScoredCandidate) =>\n                  val updatedFeatureMap =\n                    accumulatedScoredCandidate.features ++ resultScoredCandidate.features\n                  HydratorCandidateResult(resultScoredCandidate.candidate, updatedFeatureMap)\n              }\n            } else if (accumulator.results.isEmpty) {\n              // accumulator is empty (the initial case) so use additional results\n              additionalResult.results\n            } else {\n              // empty results but non-empty accumulator due to Hydrator being turned off so use accumulator results\n              accumulator.results\n            }\n\n          CandidateFeatureHydratorExecutorResult(\n            mergedFeatureMaps,\n            accumulator.individualFeatureHydratorResults ++ additionalResult.individualFeatureHydratorResults\n          )\n        }\n    }\n\n    Arrow.ifelse[Inputs[Query, Result], CandidateFeatureHydratorExecutorResult[Result]](\n      _.candidates.nonEmpty,\n      runHydrators,\n      Arrow.value(CandidateFeatureHydratorExecutorResult(Seq.empty, Map.empty)))\n  }\n\n  /** @note the returned [[Arrow]] must have a result for every candidate passed into it in the same order OR a completely empty result */\n  private def getCandidateHydratorArrow[Query <: PipelineQuery, Result <: UniversalNoun[Any]](\n    hydrator: BaseCandidateFeatureHydrator[Query, Result, _],\n    context: Executor.Context,\n    candidateFeatureHydratorObserver: FeatureHydratorObserver\n  ): Arrow[\n    Inputs[Query, Result],\n    CandidateFeatureHydratorExecutorResult[Result]\n  ] = {\n    val componentExecutorContext = context.pushToComponentStack(hydrator.identifier)\n\n    val validateFeatureMapFn: FeatureMap => FeatureMap =\n      hydrator match {\n        // Feature store candidate hydrators store the resulting PredictionRecords and\n        // not the features, so we cannot validate the same way\n        case _: FeatureStoreV1CandidateFeatureHydrator[Query, Result] =>\n          identity\n        case _ =>\n          validateFeatureMap(\n            hydrator.features.asInstanceOf[Set[Feature[_, _]]],\n            _,\n            componentExecutorContext)\n      }\n\n    val hydratorBaseArrow = hydrator match {\n      case hydrator: CandidateFeatureHydrator[Query, Result] =>\n        singleCandidateHydratorArrow(\n          hydrator,\n          validateFeatureMapFn,\n          componentExecutorContext,\n          parentContext = context)\n\n      case hydrator: BaseBulkCandidateFeatureHydrator[Query, Result, _] =>\n        bulkCandidateHydratorArrow(\n          hydrator,\n          validateFeatureMapFn,\n          componentExecutorContext,\n          parentContext = context)\n    }\n\n    val candidateFeatureHydratorArrow =\n      Arrow\n        .zipWithArg(hydratorBaseArrow)\n        .map {\n          case (\n                arg: CandidateFeatureHydratorExecutor.Inputs[Query, Result],\n                featureMapSeq: Seq[FeatureMap]) =>\n            val candidates = arg.candidates.map(_.candidate)\n\n            candidateFeatureHydratorObserver.observeFeatureSuccessAndFailures(\n              hydrator,\n              featureMapSeq)\n\n            // Build a map from candidate to FeatureMap\n            val candidateAndFeatureMaps = if (candidates.size == featureMapSeq.size) {\n              candidates.zip(featureMapSeq).map {\n                case (candidate, featureMap) => HydratorCandidateResult(candidate, featureMap)\n              }\n            } else {\n              throw PipelineFailure(\n                MisconfiguredFeatureMapFailure,\n                s\"Unexpected response length from ${hydrator.identifier}, ensure hydrator returns feature map for all candidates\")\n            }\n            val individualFeatureHydratorFeatureMaps =\n              Map(hydrator.identifier -> IndividualFeatureHydratorResult(candidateAndFeatureMaps))\n            CandidateFeatureHydratorExecutorResult(\n              candidateAndFeatureMaps,\n              individualFeatureHydratorFeatureMaps)\n        }\n\n    val conditionallyRunArrow = hydrator match {\n      case hydrator: BaseCandidateFeatureHydrator[Query, Result, _] with Conditionally[\n            Query @unchecked\n          ] =>\n        Arrow.ifelse[Inputs[Query, Result], CandidateFeatureHydratorExecutorResult[Result]](\n          { case Inputs(query: Query @unchecked, _) => hydrator.onlyIf(query) },\n          candidateFeatureHydratorArrow,\n          Arrow.value(\n            CandidateFeatureHydratorExecutorResult(\n              Seq.empty,\n              Map(hydrator.identifier -> FeatureHydratorDisabled[Result]())\n            ))\n        )\n      case _ => candidateFeatureHydratorArrow\n    }\n\n    wrapWithErrorHandling(context, hydrator.identifier)(conditionallyRunArrow)\n  }\n\n  private def singleCandidateHydratorArrow[Query <: PipelineQuery, Result <: UniversalNoun[Any]](\n    hydrator: CandidateFeatureHydrator[Query, Result],\n    validateFeatureMap: FeatureMap => FeatureMap,\n    componentContext: Context,\n    parentContext: Context\n  ): Arrow[Inputs[Query, Result], Seq[FeatureMap]] = {\n    val inputTransformer = Arrow\n      .map { inputs: Inputs[Query, Result] =>\n        inputs.candidates.map { candidate =>\n          (inputs.query, candidate.candidate, candidate.features)\n        }\n      }\n\n    val hydratorArrow = Arrow\n      .flatMap[(Query, Result, FeatureMap), FeatureMap] {\n        case (query, candidate, featureMap) =>\n          hydrator.apply(query, candidate, featureMap)\n      }\n\n    // validate before observing so validation failures are caught in the metrics\n    val hydratorArrowWithValidation = hydratorArrow.map(validateFeatureMap)\n\n    // no tracing here since per-Component spans is overkill\n    val observedArrow =\n      wrapPerCandidateComponentWithExecutorBookkeepingWithoutTracing(\n        parentContext,\n        hydrator.identifier\n      )(hydratorArrowWithValidation)\n\n    // only handle non-validation failures\n    val liftNonValidationFailuresToFailedFeatures = Arrow.handle[FeatureMap, FeatureMap] {\n      case NotAMisconfiguredFeatureMapFailure(e) =>\n        featureMapWithFailuresForFeatures(hydrator.features, e, componentContext)\n    }\n\n    wrapComponentsWithTracingOnly(parentContext, hydrator.identifier)(\n      inputTransformer.andThen(\n        Arrow.sequence(observedArrow.andThen(liftNonValidationFailuresToFailedFeatures))\n      )\n    )\n  }\n\n  private def bulkCandidateHydratorArrow[Query <: PipelineQuery, Result <: UniversalNoun[Any]](\n    hydrator: BaseBulkCandidateFeatureHydrator[Query, Result, _],\n    validateFeatureMap: FeatureMap => FeatureMap,\n    componentContext: Context,\n    parentContext: Context\n  ): Arrow[Inputs[Query, Result], Seq[FeatureMap]] = {\n    val hydratorArrow: Arrow[Inputs[Query, Result], Seq[FeatureMap]] =\n      Arrow.flatMap { inputs =>\n        hydrator.apply(inputs.query, inputs.candidates)\n      }\n\n    val validationArrow: Arrow[(Inputs[Query, Result], Seq[FeatureMap]), Seq[FeatureMap]] = Arrow\n      .map[(Inputs[Query, Result], Seq[FeatureMap]), Seq[FeatureMap]] {\n        case (inputs, results) =>\n          // For bulk APIs, this ensures no candidates are omitted and also ensures the order is preserved.\n          if (inputs.candidates.length != results.length) {\n            throw PipelineFailure(\n              MisconfiguredFeatureMapFailure,\n              s\"Unexpected response from ${hydrator.identifier}, ensure hydrator returns features for all candidates. Missing results for ${inputs.candidates.length - results.length} candidates\"\n            )\n          }\n\n          results.map(validateFeatureMap)\n      }\n\n    // validate before observing so validation failures are caught in the metrics\n    val hydratorArrowWithValidation: Arrow[Inputs[Query, Result], Seq[FeatureMap]] =\n      Arrow.zipWithArg(hydratorArrow).andThen(validationArrow)\n\n    val observedArrow =\n      wrapComponentWithExecutorBookkeeping(parentContext, hydrator.identifier)(\n        hydratorArrowWithValidation)\n\n    // only handle non-validation failures\n    val liftNonValidationFailuresToFailedFeatures =\n      Arrow.map[(Inputs[Query, Result], Try[Seq[FeatureMap]]), Try[Seq[FeatureMap]]] {\n        case (inputs, resultTry) =>\n          resultTry.handle {\n            case NotAMisconfiguredFeatureMapFailure(e) =>\n              val errorFeatureMap =\n                featureMapWithFailuresForFeatures(\n                  hydrator.features.asInstanceOf[Set[Feature[_, _]]],\n                  e,\n                  componentContext)\n              inputs.candidates.map(_ => errorFeatureMap)\n          }\n      }\n\n    Arrow\n      .zipWithArg(observedArrow.liftToTry)\n      .andThen(liftNonValidationFailuresToFailedFeatures)\n      .lowerFromTry\n  }\n}\n\nobject CandidateFeatureHydratorExecutor {\n  case class Inputs[+Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n    query: Query,\n    candidates: Seq[CandidateWithFeatures[Candidate]])\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_hydrator_executor/CandidateFeatureHydratorExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service.candidate_feature_hydrator_executor\n\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.service.ExecutorResult\n\ncase class CandidateFeatureHydratorExecutorResult[+Result <: UniversalNoun[Any]](\n  results: Seq[CandidateWithFeatures[Result]],\n  individualFeatureHydratorResults: Map[\n    _ <: ComponentIdentifier,\n    BaseIndividualFeatureHydratorResult[Result]\n  ]) extends ExecutorResult\n\nsealed trait BaseIndividualFeatureHydratorResult[+Result <: UniversalNoun[Any]]\ncase class FeatureHydratorDisabled[+Result <: UniversalNoun[Any]]()\n    extends BaseIndividualFeatureHydratorResult[Result]\ncase class IndividualFeatureHydratorResult[+Result <: UniversalNoun[Any]](\n  result: Seq[CandidateWithFeatures[Result]])\n    extends BaseIndividualFeatureHydratorResult[Result]\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_transformer_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_transformer_executor/CandidateFeatureTransformerExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.candidate_feature_transformer_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.Executor._\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CandidateFeatureTransformerExecutor @Inject() (override val statsReceiver: StatsReceiver)\n    extends Executor {\n  def arrow[Result](\n    transformers: Seq[CandidateFeatureTransformer[Result]],\n    context: Executor.Context\n  ): Arrow[Seq[Result], CandidateFeatureTransformerExecutorResult] = {\n    if (transformers.isEmpty) {\n      // must always return a Seq of FeatureMaps, even if there are no Transformers\n      Arrow.map[Seq[Result], CandidateFeatureTransformerExecutorResult] { candidates =>\n        CandidateFeatureTransformerExecutorResult(candidates.map(_ => FeatureMap.empty), Seq.empty)\n      }\n    } else {\n      val transformerArrows: Seq[Arrow[Seq[Result], Seq[(TransformerIdentifier, FeatureMap)]]] =\n        transformers.map { transformer =>\n          val transformerContext = context.pushToComponentStack(transformer.identifier)\n\n          val liftNonValidationFailuresToFailedFeatures =\n            Arrow.handle[FeatureMap, FeatureMap] {\n              case NotAMisconfiguredFeatureMapFailure(e) =>\n                featureMapWithFailuresForFeatures(transformer.features, e, transformerContext)\n            }\n\n          val underlyingArrow = Arrow\n            .map(transformer.transform)\n            .map(validateFeatureMap(transformer.features, _, transformerContext))\n\n          val observedArrowWithoutTracing =\n            wrapPerCandidateComponentWithExecutorBookkeepingWithoutTracing(\n              context,\n              transformer.identifier)(underlyingArrow)\n\n          val seqArrow =\n            Arrow.sequence(\n              observedArrowWithoutTracing\n                .andThen(liftNonValidationFailuresToFailedFeatures)\n                .map(transformer.identifier -> _)\n            )\n\n          wrapComponentsWithTracingOnly(context, transformer.identifier)(seqArrow)\n        }\n\n      Arrow.collect(transformerArrows).map { results =>\n        /**\n         * Inner Seqs are a given Transformer applied to all the candidates\n         *\n         * We want to merge the FeatureMaps for each candidate\n         * from all the Transformers. We do this by merging all the FeatureMaps at\n         * each index `i` of each Seq in `results` by `transpose`-ing the `results`\n         * so the inner Seq becomes all the FeatureMaps for Candidate\n         * at index `i` in the input Seq.\n         *\n         * {{{\n         *  Seq(\n         *    Seq(transformer1FeatureMapCandidate1, ..., transformer1FeatureMapCandidateN),\n         *    ...,\n         *    Seq(transformerMFeatureMapCandidate1, ..., transformerMFeatureMapCandidateN)\n         *  ).transpose == Seq(\n         *    Seq(transformer1FeatureMapCandidate1, ..., transformerMFeatureMapCandidate1),\n         *    ...,\n         *    Seq(transformer1FeatureMapCandidateN, ..., transformerMFeatureMapCandidateN)\n         *  )\n         * }}}\n         *\n         * we could avoid the transpose if we ran each candidate through all the transformers\n         * one-after-the-other, but then we couldn't have a single tracing span for all applications\n         * of a Transformer, so instead we apply each transformer to all candidates together, then\n         * move onto the next transformer.\n         *\n         * It's worth noting that the outer Seq is bounded by the number of Transformers that are\n         * applied which will typically be small.\n         */\n        val transposed = results.transpose\n        val combinedMaps = transposed.map(featureMapsForSingleCandidate =>\n          FeatureMap.merge(featureMapsForSingleCandidate.map { case (_, maps) => maps }))\n\n        CandidateFeatureTransformerExecutorResult(combinedMaps, transposed.map(_.toMap))\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_transformer_executor/CandidateFeatureTransformerExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service.candidate_feature_transformer_executor\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.identifier.TransformerIdentifier\n\ncase class CandidateFeatureTransformerExecutorResult(\n  featureMaps: Seq[FeatureMap],\n  individualFeatureMaps: Seq[Map[TransformerIdentifier, FeatureMap]])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_pipeline_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-slf4j/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-slf4j/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_pipeline_executor/CandidatePipelineExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.candidate_pipeline_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.pipeline.CandidatePipelineResults\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipeline\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineResult\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorObserver\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.stitch.Arrow\nimport com.twitter.util.logging.Logging\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass CandidatePipelineExecutor @Inject() (override val statsReceiver: StatsReceiver)\n    extends Executor\n    with Logging {\n\n  def arrow[Query <: PipelineQuery](\n    candidatePipelines: Seq[CandidatePipeline[Query]],\n    defaultFailOpenPolicy: FailOpenPolicy,\n    failOpenPolicies: Map[CandidatePipelineIdentifier, FailOpenPolicy],\n    qualityFactorObserverByPipeline: Map[ComponentIdentifier, QualityFactorObserver],\n    context: Executor.Context\n  ): Arrow[CandidatePipeline.Inputs[Query], CandidatePipelineExecutorResult] = {\n\n    // Get the `.arrow` of each Candidate Pipeline, and wrap it in a ResultObserver\n    val observedArrows: Seq[Arrow[CandidatePipeline.Inputs[Query], CandidatePipelineResult]] =\n      candidatePipelines.map { pipeline =>\n        wrapPipelineWithExecutorBookkeeping(\n          context = context,\n          currentComponentIdentifier = pipeline.identifier,\n          qualityFactorObserver = qualityFactorObserverByPipeline.get(pipeline.identifier),\n          failOpenPolicy = failOpenPolicies.getOrElse(pipeline.identifier, defaultFailOpenPolicy)\n        )(pipeline.arrow)\n      }\n\n    // Collect the results from all the candidate pipelines together\n    Arrow.zipWithArg(Arrow.collect(observedArrows)).map {\n      case (input: CandidatePipeline.Inputs[Query], results: Seq[CandidatePipelineResult]) =>\n        val candidateWithDetails = results.flatMap(_.result.getOrElse(Seq.empty))\n        val previousCandidateWithDetails = input.query.features\n          .map(_.getOrElse(CandidatePipelineResults, Seq.empty))\n          .getOrElse(Seq.empty)\n\n        val featureMapWithCandidates = FeatureMapBuilder()\n          .add(CandidatePipelineResults, previousCandidateWithDetails ++ candidateWithDetails)\n          .build()\n\n        // Merge the query feature hydrator and candidate source query features back in. While this\n        // is done internally in the pipeline, we have to pass it back since we don't expose the\n        // updated pipeline query today.\n        val queryFeatureHydratorFeatureMaps =\n          results\n            .flatMap(result => Seq(result.queryFeatures, result.queryFeaturesPhase2))\n            .collect { case Some(result) => result.featureMap }\n        val asyncFeatureHydratorFeatureMaps =\n          results\n            .flatMap(_.asyncFeatureHydrationResults)\n            .flatMap(_.featureMapsByStep.values)\n\n        val candidateSourceFeatureMaps =\n          results\n            .flatMap(_.candidateSourceResult)\n            .map(_.candidateSourceFeatureMap)\n\n        val featureMaps =\n          (featureMapWithCandidates +: queryFeatureHydratorFeatureMaps) ++ asyncFeatureHydratorFeatureMaps ++ candidateSourceFeatureMaps\n        val mergedFeatureMap = FeatureMap.merge(featureMaps)\n        CandidatePipelineExecutorResult(\n          candidatePipelineResults = results,\n          queryFeatureMap = mergedFeatureMap)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_pipeline_executor/CandidatePipelineExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service.candidate_pipeline_executor\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.pipeline.candidate.CandidatePipelineResult\n\ncase class CandidatePipelineExecutorResult(\n  candidatePipelineResults: Seq[CandidatePipelineResult],\n  queryFeatureMap: FeatureMap)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_source_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_transformer_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transformer_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n        \"util/util-core:scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_transformer_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transformer_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_source_executor/CandidateSourceExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.candidate_source_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.functional_component.candidate_source.BaseCandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSource\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidateSourceWithExtractedFeatures\nimport com.twitter.product_mixer.core.functional_component.candidate_source.CandidatesWithSourceFeatures\nimport com.twitter.product_mixer.core.functional_component.transformer.BaseCandidatePipelineQueryTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidateFeatureTransformer\nimport com.twitter.product_mixer.core.functional_component.transformer.CandidatePipelineResultsTransformer\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateSourcePosition\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateSources\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ExecutionFailed\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.UnexpectedCandidateResult\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.candidate_feature_transformer_executor.CandidateFeatureTransformerExecutor\nimport com.twitter.product_mixer.core.service.transformer_executor.PerCandidateTransformerExecutor\nimport com.twitter.product_mixer.core.service.transformer_executor.TransformerExecutor\nimport com.twitter.stitch.Arrow\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.immutable.ListSet\n\n/**\n * [[CandidateSourceExecutor]]:\n *   - Executes a [[BaseCandidateSource]], using a [[BaseCandidatePipelineQueryTransformer]] and a [[CandidatePipelineResultsTransformer]]\n *   - in parallel, uses a [[CandidateFeatureTransformer]] to optionally extract [[com.twitter.product_mixer.core.feature.Feature]]s from the result\n *   - Handles [[UnexpectedCandidateResult]] [[PipelineFailure]]s returned from [[CandidatePipelineResultsTransformer]] failures by removing those candidates from the result\n */\n@Singleton\nclass CandidateSourceExecutor @Inject() (\n  override val statsReceiver: StatsReceiver,\n  candidateFeatureTransformerExecutor: CandidateFeatureTransformerExecutor,\n  transformerExecutor: TransformerExecutor,\n  perCandidateTransformerExecutor: PerCandidateTransformerExecutor)\n    extends Executor\n    with Logging {\n\n  def arrow[\n    Query <: PipelineQuery,\n    CandidateSourceQuery,\n    CandidateSourceResult,\n    Candidate <: UniversalNoun[Any]\n  ](\n    candidateSource: BaseCandidateSource[CandidateSourceQuery, CandidateSourceResult],\n    queryTransformer: BaseCandidatePipelineQueryTransformer[\n      Query,\n      CandidateSourceQuery\n    ],\n    resultTransformer: CandidatePipelineResultsTransformer[CandidateSourceResult, Candidate],\n    resultFeaturesTransformers: Seq[CandidateFeatureTransformer[CandidateSourceResult]],\n    context: Executor.Context\n  ): Arrow[Query, CandidateSourceExecutorResult[Candidate]] = {\n\n    val candidateSourceArrow: Arrow[CandidateSourceQuery, CandidatesWithSourceFeatures[\n      CandidateSourceResult\n    ]] =\n      candidateSource match {\n        case regularCandidateSource: CandidateSource[CandidateSourceQuery, CandidateSourceResult] =>\n          Arrow.flatMap(regularCandidateSource.apply).map { candidates =>\n            CandidatesWithSourceFeatures(candidates, FeatureMap.empty)\n          }\n        case candidateSourceWithExtractedFeatures: CandidateSourceWithExtractedFeatures[\n              CandidateSourceQuery,\n              CandidateSourceResult\n            ] =>\n          Arrow.flatMap(candidateSourceWithExtractedFeatures.apply)\n      }\n\n    val resultsTransformerArrow: Arrow[Seq[CandidateSourceResult], Seq[Try[Candidate]]] =\n      perCandidateTransformerExecutor.arrow(resultTransformer, context)\n\n    val featureMapTransformersArrow: Arrow[\n      Seq[CandidateSourceResult],\n      Seq[FeatureMap]\n    ] =\n      candidateFeatureTransformerExecutor\n        .arrow(resultFeaturesTransformers, context).map(_.featureMaps)\n\n    val candidatesResultArrow: Arrow[CandidatesWithSourceFeatures[CandidateSourceResult], Seq[\n      (Candidate, FeatureMap)\n    ]] = Arrow\n      .map[CandidatesWithSourceFeatures[CandidateSourceResult], Seq[CandidateSourceResult]](\n        _.candidates)\n      .andThen(Arrow\n        .joinMap(resultsTransformerArrow, featureMapTransformersArrow) {\n          case (transformed, features) =>\n            if (transformed.length != features.length)\n              throw PipelineFailure(\n                ExecutionFailed,\n                s\"Found ${transformed.length} candidates and ${features.length} FeatureMaps, expected their lengths to be equal\")\n            transformed.iterator\n              .zip(features.iterator)\n              .collect { case ErrorHandling(result) => result }\n              .toSeq\n        })\n\n    // Build the final CandidateSourceExecutorResult\n    val executorResultArrow: Arrow[\n      (FeatureMap, Seq[(Candidate, FeatureMap)]),\n      CandidateSourceExecutorResult[\n        Candidate\n      ]\n    ] = Arrow.map {\n      case (queryFeatures: FeatureMap, results: Seq[(Candidate, FeatureMap)]) =>\n        val candidatesWithFeatures: Seq[FetchedCandidateWithFeatures[Candidate]] =\n          results.zipWithIndex.map {\n            case ((candidate, featureMap), index) =>\n              FetchedCandidateWithFeatures(\n                candidate,\n                featureMap + (CandidateSourcePosition, index) + (CandidateSources, ListSet(\n                  candidateSource.identifier))\n              )\n          }\n        CandidateSourceExecutorResult(\n          candidates = candidatesWithFeatures,\n          candidateSourceFeatureMap = queryFeatures\n        )\n    }\n\n    val queryTransformerArrow =\n      transformerExecutor.arrow[Query, CandidateSourceQuery](queryTransformer, context)\n\n    val combinedArrow =\n      queryTransformerArrow\n        .andThen(candidateSourceArrow)\n        .andThen(\n          Arrow\n            .join(\n              Arrow.map[CandidatesWithSourceFeatures[CandidateSourceResult], FeatureMap](\n                _.features),\n              candidatesResultArrow\n            ))\n        .andThen(executorResultArrow)\n\n    wrapComponentWithExecutorBookkeepingWithSize[Query, CandidateSourceExecutorResult[Candidate]](\n      context,\n      candidateSource.identifier,\n      result => result.candidates.size\n    )(combinedArrow)\n  }\n\n  object ErrorHandling {\n\n    /** Silently drop [[UnexpectedCandidateResult]] */\n    def unapply[Candidate](\n      candidateTryAndFeatureMap: (Try[Candidate], FeatureMap)\n    ): Option[(Candidate, FeatureMap)] = {\n      val (candidateTry, featureMap) = candidateTryAndFeatureMap\n      val candidateOpt = candidateTry match {\n        case Throw(PipelineFailure(UnexpectedCandidateResult, _, _, _)) => None\n        case Throw(ex) => throw ex\n        case Return(r) => Some(r)\n      }\n\n      candidateOpt.map { candidate => (candidate, featureMap) }\n    }\n  }\n}\n\ncase class FetchedCandidateWithFeatures[Candidate <: UniversalNoun[Any]](\n  candidate: Candidate,\n  features: FeatureMap)\n    extends CandidateWithFeatures[Candidate]\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_source_executor/CandidateSourceExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service.candidate_source_executor\n\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.service.ExecutorResult\n\ncase class CandidateSourceExecutorResult[Candidate <: UniversalNoun[Any]](\n  candidates: Seq[FetchedCandidateWithFeatures[Candidate]],\n  candidateSourceFeatureMap: FeatureMap)\n    extends ExecutorResult\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/component_registry/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"util/util-slf4j-api\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/component_registry/ComponentRegistry.scala",
    "content": "package com.twitter.product_mixer.core.service.component_registry\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.util.Activity\nimport com.twitter.util.Future\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\nimport java.util.concurrent.ConcurrentHashMap\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.JavaConverters._\n\n/**\n * The [[ComponentRegistry]] works closely with [[ComponentIdentifier]]s and the [[com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry]]\n * to provide the Product Mixer framework information about the [[com.twitter.product_mixer.core.pipeline.Pipeline]]s and [[Component]]s\n * that make up an application.\n *\n * This registration allows us to configure alerts and dashboards,\n * to query your application structure letting us display the graph of the execution and the results of queries,\n * and to garner insight into usages.\n *\n * The registry is a snapshot of the state of the world when pipelines were last built successfully.\n * For most services, this only happens once on startup. However, some services may rebuild their\n * pipelines dynamically later on.\n */\n\n@Singleton\nclass ComponentRegistry @Inject() (statsReceiver: StatsReceiver) {\n  // Initially pending until the first snapshot is built by [[ProductPipelineRegistry]]\n  private val (snapshotActivity, snapshotWitness) = Activity[ComponentRegistrySnapshot]()\n  private val snapshotCount = statsReceiver.counter(\"ComponentRegistry\", \"SnapshotCount\")\n\n  def get: Future[ComponentRegistrySnapshot] = snapshotActivity.values.toFuture.lowerFromTry\n  private[core] def set(snapshot: ComponentRegistrySnapshot): Unit = {\n    snapshotCount.incr()\n    snapshotWitness.notify(Try(snapshot))\n  }\n}\n\nclass ComponentRegistrySnapshot() extends Logging {\n\n  /** for storing the [[RegisteredComponent]]s */\n  private[this] val componentRegistry =\n    new ConcurrentHashMap[ComponentIdentifier, RegisteredComponent]\n\n  /** for determining the children of a [[ComponentIdentifier]] */\n  private[this] val componentChildren =\n    new ConcurrentHashMap[ComponentIdentifier, Set[ComponentIdentifier]]\n\n  /** for determining [[ComponentIdentifier]] uniqueness within a given [[ComponentIdentifierStack]] */\n  private[this] val componentHierarchy =\n    new ConcurrentHashMap[ComponentIdentifierStack, Set[ComponentIdentifier]]\n\n  /**\n   * Register the given [[Component]] at the end of path provided by `parentIdentifierStack`\n   * or throws an exception if adding the component results in an invalid configuration.\n   *\n   * @throws ChildComponentCollisionException if a [[Component]] with the same [[ComponentIdentifier]] is registered\n   *                                          more than once under the same parent.\n   *                                          e.g. if you register `ComponentA` under `ProductA -> PipelineA` twice,\n   *                                          this exception will be thrown when registering `ComponentA` the second\n   *                                          time. This is pretty much always a configuration error due to copy-pasting\n   *                                          and forgetting to update the identifier, or accidentally using the same\n   *                                          component twice under the same parent. If this didn't throw, stats from\n   *                                          these 2 components would be indistinguishable.\n   *\n   * @throws ComponentIdentifierCollisionException if a [[Component]] with the same [[ComponentIdentifier]] is registered\n   *                                               but it's type is not the same as a previously registered [[Component]]\n   *                                               with the same [[ComponentIdentifier]]\n   *                                               e.g. if you register 2 [[Component]]s with the same [[ComponentIdentifier]]\n   *                                               such as `new Component` and an instance of\n   *                                               `class MyComponent extends Component` the `new Component` will have a\n   *                                               type of `Component` and the other one will have a type of `MyComponent`\n   *                                               which will throw. This is usually due to copy-pasting a component as\n   *                                               a starting point and forgetting to update the identifier. If this\n   *                                               didn't throw, absolute stats from these 2 components would be\n   *                                               indistinguishable.\n   *\n   *\n   * @note this will log details of component identifier reuse if the underling components are not equal, but otherwise are of the same class.\n   *       Their stats will be merged and indistinguishable but since they are the same name and same class, we assume the differences are\n   *       minor enough that this is okay, but make a note in the log at startup in case someone sees unexpected metrics, we can look\n   *       back at the logs and see the details.\n   *\n   * @param component the component to register\n   * @param parentIdentifierStack the complete [[ComponentIdentifierStack]] excluding the current [[Component]]'s [[ComponentIdentifier]]\n   */\n  def register(\n    component: Component,\n    parentIdentifierStack: ComponentIdentifierStack\n  ): Unit = synchronized {\n    val identifier = component.identifier\n    val parentIdentifier = parentIdentifierStack.peek\n\n    val registeredComponent =\n      RegisteredComponent(identifier, component, component.identifier.file.value)\n\n    componentRegistry.asScala\n      .get(identifier)\n      .filter(_.component != component) // only do the foreach if the components aren't equal\n      .foreach {\n        case existingComponent if existingComponent.component.getClass != component.getClass =>\n          /**\n           * The same component may be registered under different parent components.\n           * However, different component types cannot use the same component identifier.\n           *\n           * This catches some copy-pasting of a config or component and forgetting to update the identifier.\n           */\n          throw new ComponentIdentifierCollisionException(\n            componentIdentifier = identifier,\n            component = registeredComponent,\n            existingComponent = componentRegistry.get(identifier),\n            parentIdentifierStack = parentIdentifierStack,\n            existingIdentifierStack = componentHierarchy.search[ComponentIdentifierStack](\n              1,\n              (stack, identifiers) => if (identifiers.contains(identifier)) stack else null)\n          )\n        case existingComponent =>\n          /**\n           * The same component may be registered under different parent components.\n           * However, if the components are not equal it __may be__ a configuration error\n           * so we log a detailed description of the issue in case they need to debug.\n           *\n           * This warns customers of some copy-pasting of a config or component and forgetting to update the\n           * identifier and of reusing components with hard-coded values which are configured differently.\n           */\n          val existingIdentifierStack = componentHierarchy.search[ComponentIdentifierStack](\n            1,\n            (stack, identifiers) => if (identifiers.contains(identifier)) stack else null)\n          logger.info(\n            s\"Found duplicate identifiers for non-equal components, $identifier from ${registeredComponent.sourceFile} \" +\n              s\"under ${parentIdentifierStack.componentIdentifiers.reverse.mkString(\" -> \")} \" +\n              s\"was already defined and is unequal to ${existingComponent.sourceFile} \" +\n              s\"under ${existingIdentifierStack.componentIdentifiers.reverse.mkString(\" -> \")}. \" +\n              s\"Merging these components in the registry, this will result in their metrics being merged. \" +\n              s\"If these components should have separate metrics, consider providing unique identifiers for them instead.\"\n          )\n      }\n\n    /** The same component may not be registered multiple times under the same parent */\n    if (componentHierarchy.getOrDefault(parentIdentifierStack, Set.empty).contains(identifier))\n      throw new ChildComponentCollisionException(identifier, parentIdentifierStack)\n\n    // add component to registry\n    componentRegistry.putIfAbsent(identifier, registeredComponent)\n    // add component to parent's `children` set for easy lookup\n    componentChildren.merge(parentIdentifier, Set(identifier), _ ++ _)\n    // add the component to the hierarchy under it's parent's identifier stack\n    componentHierarchy.merge(parentIdentifierStack, Set(identifier), _ ++ _)\n  }\n\n  def getAllRegisteredComponents: Seq[RegisteredComponent] =\n    componentRegistry.values.asScala.toSeq.sorted\n\n  def getChildComponents(component: ComponentIdentifier): Seq[ComponentIdentifier] =\n    Option(componentChildren.get(component)) match {\n      case Some(components) => components.toSeq.sorted(ComponentIdentifier.ordering)\n      case None => Seq.empty\n    }\n}\n\nclass ComponentIdentifierCollisionException(\n  componentIdentifier: ComponentIdentifier,\n  component: RegisteredComponent,\n  existingComponent: RegisteredComponent,\n  parentIdentifierStack: ComponentIdentifierStack,\n  existingIdentifierStack: ComponentIdentifierStack)\n    extends IllegalArgumentException(\n      s\"Tried to register component $componentIdentifier: of type ${component.component.getClass} from ${component.sourceFile} \" +\n        s\"under ${parentIdentifierStack.componentIdentifiers.reverse.mkString(\" -> \")} \" +\n        s\"but it was already defined with a different type ${existingComponent.component.getClass} from ${existingComponent.sourceFile} \" +\n        s\"under ${existingIdentifierStack.componentIdentifiers.reverse.mkString(\" -> \")}. \" +\n        s\"Ensure you aren't reusing a component identifier which can happen when copy-pasting existing component code by accident\")\n\nclass ChildComponentCollisionException(\n  componentIdentifier: ComponentIdentifier,\n  parentIdentifierStack: ComponentIdentifierStack)\n    extends IllegalArgumentException(\n      s\"Component $componentIdentifier already defined under parent component $parentIdentifierStack\")\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/component_registry/RegisteredComponent.scala",
    "content": "package com.twitter.product_mixer.core.service.component_registry\n\nimport com.twitter.product_mixer.core.model.common.Component\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\n\nobject RegisteredComponent {\n  // Sort by ComponentIdentifier which has its own implicit ordering defined\n  implicit val ordering: Ordering[RegisteredComponent] =\n    Ordering.by[RegisteredComponent, ComponentIdentifier](_.component.identifier)\n}\n\ncase class RegisteredComponent(\n  identifier: ComponentIdentifier,\n  component: Component,\n  sourceFile: String)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/debug_query/AuthorizationService.scala",
    "content": "package com.twitter.product_mixer.core.service.debug_query\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AccessPolicy\nimport com.twitter.product_mixer.core.functional_component.common.access_policy.AccessPolicyEvaluator\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ServiceLocal\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.Authentication\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.BadRequest\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.turntable.{thriftscala => t}\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Basic class that provides a verification method for checking if a call to our debugging\n * features is allowed/authorized to make said call.\n * @param isServiceLocal Whether the service is being run locally.\n */\n@Singleton\nclass AuthorizationService @Inject() (@Flag(ServiceLocal) isServiceLocal: Boolean) {\n  import AuthorizationService._\n\n  /**\n   * Check whether a call to a given product is authorized. Throws an [[UnauthorizedServiceCallException]]\n   * if not.\n   * @param requestingServiceIdentifier The Service Identifier of the calling service\n   * @param productAccessPolicies The access policies of the product being called.\n   * @param requestContext The request context of the caller.\n   */\n  def verifyRequestAuthorization(\n    componentIdentifierStack: ComponentIdentifierStack,\n    requestingServiceIdentifier: ServiceIdentifier,\n    productAccessPolicies: Set[AccessPolicy],\n    requestContext: t.TurntableRequestContext\n  ): Unit = {\n    val isServiceCallAuthorized =\n      requestingServiceIdentifier.role == AllowedServiceIdentifierRole && requestingServiceIdentifier.service == AllowedServiceIdentifierName\n    val userLdapGroups = requestContext.ldapGroups.map(_.toSet)\n\n    val accessPolicyAllowed = AccessPolicyEvaluator.evaluate(\n      productAccessPolicies = productAccessPolicies,\n      userLdapGroups = userLdapGroups.getOrElse(Set.empty)\n    )\n\n    if (!isServiceLocal && !isServiceCallAuthorized) {\n      throw new UnauthorizedServiceCallException(\n        requestingServiceIdentifier,\n        componentIdentifierStack)\n    }\n\n    if (!isServiceLocal && !accessPolicyAllowed) {\n      throw new InsufficientAccessException(\n        userLdapGroups,\n        productAccessPolicies,\n        componentIdentifierStack)\n    }\n  }\n}\n\nobject AuthorizationService {\n  final val AllowedServiceIdentifierRole = \"turntable\"\n  final val AllowedServiceIdentifierName = \"turntable\"\n}\n\nclass UnauthorizedServiceCallException(\n  serviceIdentifier: ServiceIdentifier,\n  componentIdentifierStack: ComponentIdentifierStack)\n    extends PipelineFailure(\n      BadRequest,\n      s\"Unexpected Service tried to call Turntable Debug endpoint: ${ServiceIdentifier.asString(serviceIdentifier)}\",\n      componentStack = Some(componentIdentifierStack))\n\nclass InsufficientAccessException(\n  ldapGroups: Option[Set[String]],\n  desiredAccessPolicies: Set[AccessPolicy],\n  componentIdentifierStack: ComponentIdentifierStack)\n    extends PipelineFailure(\n      Authentication,\n      s\"Request did not satisfy access policies: $desiredAccessPolicies with ldapGroups = $ldapGroups\",\n      componentStack = Some(componentIdentifierStack))\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/debug_query/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/configapi\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/recommendation\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"product-mixer/turntable/service/src/main/scala/com/twitter/turntable/context:key\",\n        \"stitch/stitch-core\",\n        \"util/util-core:scala\",\n        \"util/util-jackson/src/main/scala/com/twitter/util/jackson\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/debug_query/DebugQueryNotSupportedService.scala",
    "content": "package com.twitter.product_mixer.core.service.debug_query\n\nimport com.twitter.finagle.Service\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ProductDisabled\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineResult\nimport com.twitter.scrooge.{Request => ScroogeRequest}\nimport com.twitter.scrooge.{Response => ScroogeResponse}\nimport com.twitter.util.Future\nimport com.twitter.product_mixer.core.{thriftscala => t}\nimport com.twitter.util.jackson.ScalaObjectMapper\n\n/**\n * All Mixers must implement a debug query interface. This can be a problem for in-place migrations\n * where a service may only partially implement Product Mixer patterns. This service can be used as\n * a noop implementation of [[DebugQueryService]] until the Mixer service is fully migrated.\n */\nobject DebugQueryNotSupportedService\n    extends Service[ScroogeRequest[_], ScroogeResponse[t.PipelineExecutionResult]] {\n\n  val failureJson: String = {\n    val message = \"This service does not support debug queries, this is usually due to an active \" +\n      \"in-place migration to Product Mixer. Please reach out in #product-mixer if you have any questions.\"\n\n    ScalaObjectMapper().writeValueAsString(\n      ProductPipelineResult(\n        transformedQuery = None,\n        qualityFactorResult = None,\n        gateResult = None,\n        pipelineSelectorResult = None,\n        mixerPipelineResult = None,\n        recommendationPipelineResult = None,\n        traceId = None,\n        failure = Some(PipelineFailure(ProductDisabled, message)),\n        result = None,\n      ))\n  }\n\n  override def apply(\n    thriftRequest: ScroogeRequest[_]\n  ): Future[ScroogeResponse[t.PipelineExecutionResult]] =\n    Future.value(ScroogeResponse(t.PipelineExecutionResult(failureJson)))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/debug_query/DebugQueryService.scala",
    "content": "package com.twitter.product_mixer.core.service.debug_query\n\nimport com.fasterxml.jackson.databind.SerializationFeature\nimport com.twitter.finagle.Service\nimport com.twitter.finagle.context.Contexts\nimport com.twitter.finagle.tracing.Trace.traceLocal\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.transport.Transport\nimport com.twitter.product_mixer.core.functional_component.configapi.ParamsBuilder\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifierStack\nimport com.twitter.product_mixer.core.model.marshalling.request.Product\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipeline\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineRequest\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry\nimport com.twitter.product_mixer.core.{thriftscala => t}\nimport com.twitter.scrooge.ThriftStruct\nimport com.twitter.scrooge.{Request => ScroogeRequest}\nimport com.twitter.scrooge.{Response => ScroogeResponse}\nimport com.twitter.stitch.Stitch\nimport com.twitter.turntable.context.TurntableRequestContextKey\nimport com.twitter.util.jackson.ScalaObjectMapper\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.reflect.runtime.universe.TypeTag\n\n/**\n * Returns the complete execution log for a pipeline query. These endpoints are intended for\n * debugging (primarily through Turntable).\n */\n@Singleton\nclass DebugQueryService @Inject() (\n  productPipelineRegistry: ProductPipelineRegistry,\n  paramsBuilder: ParamsBuilder,\n  authorizationService: AuthorizationService) {\n\n  private val mapper =\n    ScalaObjectMapper.builder\n      .withAdditionalJacksonModules(Seq(ParamsSerializerModule))\n      .withSerializationConfig(\n        Map(\n          // These are copied from the default serialization config.\n          SerializationFeature.WRITE_DATES_AS_TIMESTAMPS -> false,\n          SerializationFeature.WRITE_ENUMS_USING_TO_STRING -> true,\n          // Generally we want to be defensive when serializing since we don't control everything that's\n          // serialized. This issue also came up when trying to serialize Unit as part of sync side effects.\n          SerializationFeature.FAIL_ON_EMPTY_BEANS -> false,\n        ))\n      // The default implementation represents numbers as JSON Numbers (i.e. Double with 53 bit precision\n      // which leads to Snowflake IDs being cropped in the case of tweets.\n      .withNumbersAsStrings(true)\n      .objectMapper\n\n  def apply[\n    ThriftRequest <: ThriftStruct with Product1[MixerServiceRequest],\n    MixerServiceRequest <: ThriftStruct,\n    MixerRequest <: Request\n  ](\n    unmarshaller: MixerServiceRequest => MixerRequest\n  )(\n    implicit requestTypeTag: TypeTag[MixerRequest]\n  ): Service[ScroogeRequest[ThriftRequest], ScroogeResponse[t.PipelineExecutionResult]] = {\n    (thriftRequest: ScroogeRequest[ThriftRequest]) =>\n      {\n\n        val request = unmarshaller(thriftRequest.args._1)\n        val params = paramsBuilder.build(\n          clientContext = request.clientContext,\n          product = request.product,\n          featureOverrides = request.debugParams.flatMap(_.featureOverrides).getOrElse(Map.empty)\n        )\n\n        val productPipeline = productPipelineRegistry\n          .getProductPipeline[MixerRequest, Any](request.product)\n        verifyRequestAuthorization(request.product, productPipeline)\n        Contexts.broadcast.letClear(TurntableRequestContextKey) {\n          Stitch\n            .run(productPipeline\n              .arrow(ProductPipelineRequest(request, params)).map { detailedResult =>\n                // Serialization can be slow so a trace is useful both for optimization by the Promix\n                // team and to give visibility to customers.\n                val serializedJSON =\n                  traceLocal(\"serialize_debug_response\")(mapper.writeValueAsString(detailedResult))\n                t.PipelineExecutionResult(serializedJSON)\n              })\n            .map(ScroogeResponse(_))\n        }\n      }\n  }\n\n  private def verifyRequestAuthorization(\n    product: Product,\n    productPipeline: ProductPipeline[_, _]\n  ): Unit = {\n    val serviceIdentifier = ServiceIdentifier.fromCertificate(Transport.peerCertificate)\n    val requestContext = Contexts.broadcast\n      .get(TurntableRequestContextKey).getOrElse(throw MissingTurntableRequestContextException)\n\n    val componentStack = ComponentIdentifierStack(productPipeline.identifier, product.identifier)\n    authorizationService.verifyRequestAuthorization(\n      componentStack,\n      serviceIdentifier,\n      productPipeline.debugAccessPolicies,\n      requestContext)\n  }\n}\n\nobject MissingTurntableRequestContextException\n    extends Exception(\"Request is missing turntable request context\")\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/debug_query/ParamsSerializerModule.scala",
    "content": "package com.twitter.product_mixer.core.service.debug_query\n\nimport com.fasterxml.jackson.core.JsonGenerator\nimport com.fasterxml.jackson.databind.SerializerProvider\nimport com.fasterxml.jackson.databind.ser.std.StdSerializer\nimport com.twitter.timelines.configapi.Params\nimport com.fasterxml.jackson.databind.module.SimpleModule\nimport com.twitter.timelines.configapi.Config\n\nobject ParamsSerializerModule extends SimpleModule {\n  addSerializer(ParamsConfigSerializer)\n  addSerializer(ParamsStdSerializer)\n}\n\nobject ParamsStdSerializer extends StdSerializer[Params](classOf[Params]) {\n  override def serialize(\n    value: Params,\n    gen: JsonGenerator,\n    provider: SerializerProvider\n  ): Unit = {\n    gen.writeStartObject()\n    gen.writeObjectField(\"applied_params\", value.allAppliedValues)\n    gen.writeEndObject()\n  }\n}\n\nobject ParamsConfigSerializer extends StdSerializer[Config](classOf[Config]) {\n  override def serialize(\n    value: Config,\n    gen: JsonGenerator,\n    provider: SerializerProvider\n  ): Unit = {\n    gen.writeString(value.simpleName)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/domain_marshaller_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/premarshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/domain_marshaller_executor/DomainMarshallerExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.domain_marshaller_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.premarshaller.DomainMarshaller\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport com.twitter.product_mixer.core.service.domain_marshaller_executor.DomainMarshallerExecutor.Inputs\nimport com.twitter.product_mixer.core.service.domain_marshaller_executor.DomainMarshallerExecutor.Result\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Executes a [[DomainMarshaller]].\n *\n * @note This is a synchronous transform, so we don't observe it directly. Failures and such\n *       can be observed at the parent pipeline.\n */\n@Singleton\nclass DomainMarshallerExecutor @Inject() (override val statsReceiver: StatsReceiver)\n    extends Executor {\n  def arrow[Query <: PipelineQuery, DomainResponseType <: HasMarshalling](\n    marshaller: DomainMarshaller[Query, DomainResponseType],\n    context: Executor.Context\n  ): Arrow[Inputs[Query], Result[DomainResponseType]] = {\n    val arrow = Arrow\n      .map[Inputs[Query], DomainMarshallerExecutor.Result[DomainResponseType]] {\n        case Inputs(query, candidates) =>\n          DomainMarshallerExecutor.Result(marshaller(query, candidates))\n      }\n\n    wrapComponentWithExecutorBookkeeping(context, marshaller.identifier)(arrow)\n  }\n}\n\nobject DomainMarshallerExecutor {\n  case class Inputs[Query <: PipelineQuery](\n    query: Query,\n    candidatesWithDetails: Seq[CandidateWithDetails])\n  case class Result[+DomainResponseType](result: DomainResponseType) extends ExecutorResult\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/feature_hydrator_observer/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/feature_hydrator_observer/FeatureHydratorObserver.scala",
    "content": "package com.twitter.product_mixer.core.service.feature_hydrator_observer\n\nimport com.twitter.finagle.stats.BroadcastStatsReceiver\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.RollupStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.ml.featurestore.lib.data.HydrationError\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featurestorev1.featurevalue.FeatureStoreV1ResponseFeature\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.FeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1.FeatureStoreV1CandidateFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1.FeatureStoreV1QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.shared_library.observer.Observer\nimport com.twitter.servo.util.CancelledExceptionExtractor\nimport com.twitter.util.Throw\nimport com.twitter.util.Throwables\n\nclass FeatureHydratorObserver(\n  statsReceiver: StatsReceiver,\n  hydrators: Seq[FeatureHydrator[_]],\n  context: Executor.Context) {\n\n  private val hydratorAndFeatureToStats: Map[\n    ComponentIdentifier,\n    Map[Feature[_, _], FeatureCounters]\n  ] =\n    hydrators.map { hydrator =>\n      val hydratorScope = Executor.buildScopes(context, hydrator.identifier)\n      val featureToCounterMap: Map[Feature[_, _], FeatureCounters] = hydrator.features\n        .asInstanceOf[Set[Feature[_, _]]].map { feature =>\n          val scopedStats = scopedBroadcastStats(hydratorScope, feature)\n          // Initialize so we have them registered\n          val requestsCounter = scopedStats.counter(Observer.Requests)\n          val successCounter = scopedStats.counter(Observer.Success)\n          // These are dynamic so we can't really cache them\n          scopedStats.counter(Observer.Failures)\n          scopedStats.counter(Observer.Cancelled)\n          feature -> FeatureCounters(requestsCounter, successCounter, scopedStats)\n        }.toMap\n      hydrator.identifier -> featureToCounterMap\n    }.toMap\n\n  def observeFeatureSuccessAndFailures(\n    hydrator: FeatureHydrator[_],\n    featureMaps: Seq[FeatureMap]\n  ): Unit = {\n\n    val features = hydrator.features.asInstanceOf[Set[Feature[_, _]]]\n\n    val failedFeaturesWithErrorNames: Map[Feature[_, _], Seq[Seq[String]]] = hydrator match {\n      case _: FeatureStoreV1QueryFeatureHydrator[_] |\n          _: FeatureStoreV1CandidateFeatureHydrator[_, _] =>\n        featureMaps.toIterator\n          .flatMap(_.getTry(FeatureStoreV1ResponseFeature).toOption.map(_.failedFeatures)).flatMap {\n            failureMap: Map[_ <: Feature[_, _], Set[HydrationError]] =>\n              failureMap.flatMap {\n                case (feature, errors: Set[HydrationError]) =>\n                  errors.headOption.map { error =>\n                    feature -> Seq(Observer.Failures, error.errorType)\n                  }\n              }.toIterator\n          }.toSeq.groupBy { case (feature, _) => feature }.mapValues { seqOfTuples =>\n            seqOfTuples.map { case (_, error) => error }\n          }\n\n      case _: FeatureHydrator[_] =>\n        features.toIterator\n          .flatMap { feature =>\n            featureMaps\n              .flatMap(_.underlyingMap\n                .get(feature).collect {\n                  case Throw(CancelledExceptionExtractor(throwable)) =>\n                    (feature, Observer.Cancelled +: Throwables.mkString(throwable))\n                  case Throw(throwable) =>\n                    (feature, Observer.Failures +: Throwables.mkString(throwable))\n                })\n          }.toSeq.groupBy { case (feature, _) => feature }.mapValues { seqOfTuples =>\n            seqOfTuples.map { case (_, error) => error }\n          }\n    }\n\n    val failedFeaturesWithErrorCountsMap: Map[Feature[_, _], Map[Seq[String], Int]] =\n      failedFeaturesWithErrorNames.mapValues(_.groupBy { statKey => statKey }.mapValues(_.size))\n\n    val featuresToCounterMap = hydratorAndFeatureToStats.getOrElse(\n      hydrator.identifier,\n      throw new MissingHydratorException(hydrator.identifier))\n    features.foreach { feature =>\n      val hydratorFeatureCounters: FeatureCounters = featuresToCounterMap.getOrElse(\n        feature,\n        throw new MissingFeatureException(hydrator.identifier, feature))\n      val failedMapsCount = failedFeaturesWithErrorNames.getOrElse(feature, Seq.empty).size\n      val failedFeatureErrorCounts = failedFeaturesWithErrorCountsMap.getOrElse(feature, Map.empty)\n\n      hydratorFeatureCounters.requestsCounter.incr(featureMaps.size)\n      hydratorFeatureCounters.successCounter.incr(featureMaps.size - failedMapsCount)\n      failedFeatureErrorCounts.foreach {\n        case (failure, count) =>\n          hydratorFeatureCounters.scopedStats.counter(failure: _*).incr(count)\n      }\n    }\n  }\n\n  private def scopedBroadcastStats(\n    hydratorScope: Executor.Scopes,\n    feature: Feature[_, _],\n  ): StatsReceiver = {\n    val suffix = Seq(\"Feature\", feature.toString)\n    val localScope = hydratorScope.componentScopes ++ suffix\n    val relativeScope = hydratorScope.relativeScope ++ suffix\n    new RollupStatsReceiver(\n      BroadcastStatsReceiver(\n        Seq(\n          statsReceiver.scope(localScope: _*),\n          statsReceiver.scope(relativeScope: _*),\n        )\n      ))\n  }\n}\n\ncase class FeatureCounters(\n  requestsCounter: Counter,\n  successCounter: Counter,\n  scopedStats: StatsReceiver)\n\nclass MissingHydratorException(featureHydratorIdentifier: ComponentIdentifier)\n    extends Exception(s\"Missing Feature Hydrator in Stats Map: ${featureHydratorIdentifier.name}\")\n\nclass MissingFeatureException(\n  featureHydratorIdentifier: ComponentIdentifier,\n  feature: Feature[_, _])\n    extends Exception(\n      s\"Missing Feature in Stats Map: ${feature.toString} for ${featureHydratorIdentifier.name}\")\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/filter_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/filter\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/filter_executor/FilterExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.filter_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.filter.Filter\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.filter_executor.FilterExecutor.FilterState\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Arrow.Iso\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.immutable.Queue\n\n/**\n * Applies a `Seq[Filter]` in sequential order.\n * Returns the results and a detailed Seq of each filter's results (for debugging / coherence).\n *\n * Note that each successive filter is only passed the 'kept' Seq from the previous filter, not the full\n * set of candidates.\n */\n@Singleton\nclass FilterExecutor @Inject() (override val statsReceiver: StatsReceiver) extends Executor {\n\n  private val Kept = \"kept\"\n  private val Removed = \"removed\"\n\n  def arrow[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n    filters: Seq[Filter[Query, Candidate]],\n    context: Executor.Context\n  ): Arrow[(Query, Seq[CandidateWithFeatures[Candidate]]), FilterExecutorResult[Candidate]] = {\n\n    val filterArrows = filters.map(getIsoArrowForFilter(_, context))\n    val combinedArrow = isoArrowsSequentially(filterArrows)\n\n    Arrow\n      .map[(Query, Seq[CandidateWithFeatures[Candidate]]), FilterState[Query, Candidate]] {\n        case (query, filterCandidates) =>\n          // transform the input to the initial state of a `FilterExecutorResult`\n          val initialFilterExecutorResult =\n            FilterExecutorResult(filterCandidates.map(_.candidate), Queue.empty)\n          val allCandidates: Map[Candidate, CandidateWithFeatures[Candidate]] =\n            filterCandidates.map { fc =>\n              (fc.candidate, fc)\n            }.toMap\n\n          FilterState(query, allCandidates, initialFilterExecutorResult)\n      }\n      .flatMapArrow(combinedArrow)\n      .map {\n        case FilterState(_, _, filterExecutorResult) =>\n          filterExecutorResult.copy(individualFilterResults =\n            // materialize the Queue into a List for faster future iterations\n            filterExecutorResult.individualFilterResults.toList)\n      }\n\n  }\n\n  /**\n   * Adds filter specific stats, generic [[wrapComponentWithExecutorBookkeeping]] stats, and wraps with failure handling\n   *\n   * If the filter is a [[Conditionally]] ensures that we dont record stats if its turned off\n   *\n   * @note For performance, the [[FilterExecutorResult.individualFilterResults]] is build backwards - the head being the most recent result.\n   * @param filter the filter to make an [[Arrow]] out of\n   * @param context the [[Executor.Context]] for the pipeline this is a part of\n   */\n  private def getIsoArrowForFilter[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n    filter: Filter[Query, Candidate],\n    context: Executor.Context\n  ): Iso[FilterState[Query, Candidate]] = {\n    val broadcastStatsReceiver =\n      Executor.broadcastStatsReceiver(context, filter.identifier, statsReceiver)\n\n    val keptCounter = broadcastStatsReceiver.counter(Kept)\n    val removedCounter = broadcastStatsReceiver.counter(Removed)\n\n    val filterArrow = Arrow.flatMap[\n      (Query, Seq[CandidateWithFeatures[Candidate]]),\n      FilterExecutorIndividualResult[Candidate]\n    ] {\n      case (query, candidates) =>\n        filter.apply(query, candidates).map { result =>\n          FilterExecutorIndividualResult(\n            identifier = filter.identifier,\n            kept = result.kept,\n            removed = result.removed)\n        }\n    }\n\n    val observedArrow: Arrow[\n      (Query, Seq[CandidateWithFeatures[Candidate]]),\n      FilterExecutorIndividualResult[\n        Candidate\n      ]\n    ] = wrapComponentWithExecutorBookkeeping(\n      context = context,\n      currentComponentIdentifier = filter.identifier,\n      onSuccess = { result: FilterExecutorIndividualResult[Candidate] =>\n        keptCounter.incr(result.kept.size)\n        removedCounter.incr(result.removed.size)\n      }\n    )(\n      filterArrow\n    )\n\n    val conditionallyRunArrow: Arrow[\n      (Query, Seq[CandidateWithFeatures[Candidate]]),\n      IndividualFilterResults[\n        Candidate\n      ]\n    ] =\n      filter match {\n        case filter: Filter[Query, Candidate] with Conditionally[\n              Filter.Input[Query, Candidate] @unchecked\n            ] =>\n          Arrow.ifelse(\n            {\n              case (query, candidates) =>\n                filter.onlyIf(Filter.Input(query, candidates))\n            },\n            observedArrow,\n            Arrow.value(ConditionalFilterDisabled(filter.identifier))\n          )\n        case _ => observedArrow\n      }\n\n    // return an `Iso` arrow for easier composition later\n    Arrow\n      .zipWithArg(\n        Arrow\n          .map[FilterState[Query, Candidate], (Query, Seq[CandidateWithFeatures[Candidate]])] {\n            case FilterState(query, candidateToFeaturesMap, FilterExecutorResult(result, _)) =>\n              (query, result.flatMap(candidateToFeaturesMap.get))\n          }.andThen(conditionallyRunArrow))\n      .map {\n        case (\n              FilterState(query, allCandidates, filterExecutorResult),\n              filterResult: FilterExecutorIndividualResult[Candidate]) =>\n          val resultWithCurrentPrepended =\n            filterExecutorResult.individualFilterResults :+ filterResult\n          val newFilterExecutorResult = FilterExecutorResult(\n            result = filterResult.kept,\n            individualFilterResults = resultWithCurrentPrepended)\n          FilterState(query, allCandidates, newFilterExecutorResult)\n\n        case (filterState, filterDisabledResult: ConditionalFilterDisabled) =>\n          filterState.copy(\n            executorResult = filterState.executorResult.copy(\n              individualFilterResults =\n                filterState.executorResult.individualFilterResults :+ filterDisabledResult\n            ))\n      }\n  }\n}\n\nobject FilterExecutor {\n\n  /**\n   * FilterState is an internal representation of the state that is passed between each individual filter arrow.\n   *\n   * @param query: The query\n   * @param candidateToFeaturesMap: A lookup mapping from Candidate -> FilterCandidate, to rebuild the inputs quickly for the next filter\n   * @param executorResult: The in-progress executor result\n   */\n  private case class FilterState[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n    query: Query,\n    candidateToFeaturesMap: Map[Candidate, CandidateWithFeatures[Candidate]],\n    executorResult: FilterExecutorResult[Candidate])\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/filter_executor/FilterExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service.filter_executor\n\nimport com.twitter.product_mixer.core.model.common.identifier.FilterIdentifier\nimport com.twitter.product_mixer.core.service.ExecutorResult\n\ncase class FilterExecutorResult[Candidate](\n  result: Seq[Candidate],\n  individualFilterResults: Seq[IndividualFilterResults[Candidate]])\n    extends ExecutorResult\n\nsealed trait IndividualFilterResults[+Candidate]\ncase class ConditionalFilterDisabled(identifier: FilterIdentifier)\n    extends IndividualFilterResults[Nothing]\ncase class FilterExecutorIndividualResult[+Candidate](\n  identifier: FilterIdentifier,\n  kept: Seq[Candidate],\n  removed: Seq[Candidate])\n    extends IndividualFilterResults[Candidate]\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n        \"util/util-core:scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/gate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor/ExecutedGateResult.scala",
    "content": "package com.twitter.product_mixer.core.service.gate_executor\n\nimport com.twitter.product_mixer.core.functional_component.gate.GateResult\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\n\ncase class ExecutedGateResult(identifier: GateIdentifier, result: GateResult)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor/GateExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.gate_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.gate.BaseGate\nimport com.twitter.product_mixer.core.functional_component.gate.GateResult\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Arrow.Iso\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.immutable.Queue\n\n/**\n * A GateExecutor takes a Seq[Gate], executes them all sequentially, and\n * determines a final Continue or Stop decision.\n */\n@Singleton\nclass GateExecutor @Inject() (override val statsReceiver: StatsReceiver) extends Executor {\n\n  private val Continue = \"continue\"\n  private val Skipped = \"skipped\"\n  private val Stop = \"stop\"\n\n  def arrow[Query <: PipelineQuery](\n    gates: Seq[BaseGate[Query]],\n    context: Executor.Context\n  ): Arrow[Query, GateExecutorResult] = {\n\n    val gateArrows = gates.map(getIsoArrowForGate(_, context))\n    val combinedArrow = isoArrowsSequentially(gateArrows)\n\n    Arrow\n      .map { query: Query => (query, GateExecutorResult(Queue.empty)) }\n      .andThen(combinedArrow)\n      .map {\n        case (_, gateExecutorResult) =>\n          // materialize the Queue into a List for faster future iterations\n          GateExecutorResult(gateExecutorResult.individualGateResults.toList)\n      }\n  }\n\n  /**\n   * Each gate is transformed into a Iso Arrow over (Quest, List[GatewayResult]).\n   *\n   * This arrow:\n   * - Adapts the input and output types of the underlying Gate arrow (an [[Iso[(Query, QueryResult)]])\n   * - throws a [[StoppedGateException]] if [[GateResult.continue]] is false\n   * - if its not false, prepends the current results to the [[GateExecutorResult.individualGateResults]] list\n   */\n  private def getIsoArrowForGate[Query <: PipelineQuery](\n    gate: BaseGate[Query],\n    context: Executor.Context\n  ): Iso[(Query, GateExecutorResult)] = {\n    val broadcastStatsReceiver =\n      Executor.broadcastStatsReceiver(context, gate.identifier, statsReceiver)\n\n    val continueCounter = broadcastStatsReceiver.counter(Continue)\n    val skippedCounter = broadcastStatsReceiver.counter(Skipped)\n    val stopCounter = broadcastStatsReceiver.counter(Stop)\n\n    val observedArrow = wrapComponentWithExecutorBookkeeping(\n      context,\n      gate.identifier,\n      onSuccess = { gateResult: GateResult =>\n        gateResult match {\n          case GateResult.Continue => continueCounter.incr()\n          case GateResult.Skipped => skippedCounter.incr()\n          case GateResult.Stop => stopCounter.incr()\n        }\n      }\n    )(gate.arrow)\n\n    val inputAdapted: Arrow[(Query, GateExecutorResult), GateResult] =\n      Arrow\n        .map[(Query, GateExecutorResult), Query] { case (query, _) => query }\n        .andThen(observedArrow)\n\n    val zipped = Arrow.zipWithArg(inputAdapted)\n\n    // at each step, the current `GateExecutorResult.continue` value is correct for all already run gates\n    val withStoppedGatesAsExceptions = zipped.map {\n      case ((query, previousResults), currentResult) if currentResult.continue =>\n        Return(\n          (\n            query,\n            GateExecutorResult(\n              previousResults.individualGateResults :+ ExecutedGateResult(\n                gate.identifier,\n                currentResult))\n          ))\n      case _ => Throw(StoppedGateException(gate.identifier))\n    }.lowerFromTry\n\n    /**\n     * we gather stats before converting closed gates to exceptions because a closed gate\n     * isn't a failure for the gate, its a normal behavior\n     * but we do want to remap the the [[StoppedGateException]] created because the [[BaseGate]] is closed\n     * to the correct [[com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure]],\n     * so we remap with [[wrapWithErrorHandling]]\n     */\n    wrapWithErrorHandling(context, gate.identifier)(withStoppedGatesAsExceptions)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor/GateExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service.gate_executor\n\nimport com.twitter.product_mixer.core.service.ExecutorResult\n\ncase class GateExecutorResult(\n  individualGateResults: Seq[ExecutedGateResult])\n    extends ExecutorResult\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/gate_executor/StoppedGateException.scala",
    "content": "package com.twitter.product_mixer.core.service.gate_executor\n\nimport com.twitter.product_mixer.core.model.common.identifier.GateIdentifier\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureCategory\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailureClassifier\n\nimport scala.util.control.NoStackTrace\n\ncase class StoppedGateException(identifier: GateIdentifier)\n    extends Exception(\"Closed gate stopped execution of the pipeline\")\n    with NoStackTrace {\n  override def toString: String = s\"StoppedGateException($identifier)\"\n}\n\nobject StoppedGateException {\n\n  /**\n   * Creates a [[PipelineFailureClassifier]] that is used as the default for classifying failures\n   * in a pipeline by mapping [[StoppedGateException]] to a [[PipelineFailure]] with the provided\n   * [[PipelineFailureCategory]]\n   */\n  def classifier(\n    category: PipelineFailureCategory\n  ): PipelineFailureClassifier = PipelineFailureClassifier {\n    case stoppedGateException: StoppedGateException =>\n      PipelineFailure(category, stoppedGateException.getMessage, Some(stoppedGateException))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/group_results_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_transformer_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transformer_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n        \"util/util-core:scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/candidate_source\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/candidate_feature_transformer_executor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transformer_executor\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/group_results_executor/GroupResultsExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.group_results_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMapBuilder\nimport com.twitter.product_mixer.core.model.common.CandidateWithFeatures\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.CandidatePipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.CandidateSourceIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PlatformIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidatePipelines\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateSourcePosition\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateSources\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ItemPresentation\nimport com.twitter.product_mixer.core.model.common.presentation.ModuleCandidateWithDetails\nimport com.twitter.product_mixer.core.model.common.presentation.ModulePresentation\nimport com.twitter.product_mixer.core.model.common.presentation.UniversalPresentation\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.immutable.ListSet\n\n// Most executors are in the core.service package, but this one is pipeline specific\n@Singleton\nclass GroupResultsExecutor @Inject() (override val statsReceiver: StatsReceiver) extends Executor {\n\n  val identifier: ComponentIdentifier = PlatformIdentifier(\"GroupResults\")\n\n  def arrow[Candidate <: UniversalNoun[Any]](\n    pipelineIdentifier: CandidatePipelineIdentifier,\n    sourceIdentifier: CandidateSourceIdentifier,\n    context: Executor.Context\n  ): Arrow[GroupResultsExecutorInput[Candidate], GroupResultsExecutorResult] = {\n\n    val groupArrow = Arrow.map { input: GroupResultsExecutorInput[Candidate] =>\n      val modules: Map[Option[ModulePresentation], Seq[CandidateWithFeatures[Candidate]]] =\n        input.candidates\n          .map { candidate: CandidateWithFeatures[Candidate] =>\n            val modulePresentationOpt: Option[ModulePresentation] =\n              input.decorations.get(candidate.candidate).collect {\n                case itemPresentation: ItemPresentation\n                    if itemPresentation.modulePresentation.isDefined =>\n                  itemPresentation.modulePresentation.get\n              }\n\n            (candidate, modulePresentationOpt)\n          }.groupBy {\n            case (_, modulePresentationOpt) => modulePresentationOpt\n          }.mapValues {\n            resultModuleOptTuples: Seq[\n              (CandidateWithFeatures[Candidate], Option[ModulePresentation])\n            ] =>\n              resultModuleOptTuples.map {\n                case (result, _) => result\n              }\n          }\n\n      // Modules should be in their original order, but the groupBy above isn't stable.\n      // Sort them by the sourcePosition, using the sourcePosition of their first contained\n      // candidate.\n      val sortedModules: Seq[(Option[ModulePresentation], Seq[CandidateWithFeatures[Candidate]])] =\n        modules.toSeq\n          .sortBy {\n            case (_, results) =>\n              results.headOption.map(_.features.get(CandidateSourcePosition))\n          }\n\n      val candidatesWithDetails: Seq[CandidateWithDetails] = sortedModules.flatMap {\n        case (modulePresentationOpt, resultsSeq) =>\n          val itemsWithDetails = resultsSeq.map { result =>\n            val presentationOpt = input.decorations.get(result.candidate) match {\n              case itemPresentation @ Some(_: ItemPresentation) => itemPresentation\n              case _ => None\n            }\n\n            val baseFeatureMap = FeatureMapBuilder()\n              .add(CandidatePipelines, ListSet.empty + pipelineIdentifier)\n              .build()\n\n            ItemCandidateWithDetails(\n              candidate = result.candidate,\n              presentation = presentationOpt,\n              features = baseFeatureMap ++ result.features\n            )\n          }\n\n          modulePresentationOpt\n            .map { modulePresentation =>\n              val moduleSourcePosition =\n                resultsSeq.head.features.get(CandidateSourcePosition)\n              val baseFeatureMap = FeatureMapBuilder()\n                .add(CandidatePipelines, ListSet.empty + pipelineIdentifier)\n                .add(CandidateSourcePosition, moduleSourcePosition)\n                .add(CandidateSources, ListSet.empty + sourceIdentifier)\n                .build()\n\n              Seq(\n                ModuleCandidateWithDetails(\n                  candidates = itemsWithDetails,\n                  presentation = Some(modulePresentation),\n                  features = baseFeatureMap\n                ))\n            }.getOrElse(itemsWithDetails)\n      }\n\n      GroupResultsExecutorResult(candidatesWithDetails)\n    }\n\n    wrapWithErrorHandling(context, identifier)(groupArrow)\n  }\n}\n\ncase class GroupResultsExecutorInput[Candidate <: UniversalNoun[Any]](\n  candidates: Seq[CandidateWithFeatures[Candidate]],\n  decorations: Map[UniversalNoun[Any], UniversalPresentation])\n\ncase class GroupResultsExecutorResult(candidatesWithDetails: Seq[CandidateWithDetails])\n    extends ExecutorResult\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_execution_logger/AllowListedPipelineExecutionLogger.scala",
    "content": "package com.twitter.product_mixer.core.service.pipeline_execution_logger\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.PipelineExecutionLoggerAllowList\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ServiceLocal\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.util.FuturePools\nimport com.twitter.product_mixer.shared_library.observer.Observer.FutureObserver\nimport com.twitter.util.Try\nimport com.twitter.util.logging.Logging\nimport pprint.PPrinter\nimport pprint.Tree\nimport pprint.Util\nimport pprint.tuplePrefix\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Initial implementation from:\n * https://stackoverflow.com/questions/15718506/scala-how-to-print-case-classes-like-pretty-printed-tree/57080463#57080463\n */\nobject AllowListedPipelineExecutionLogger {\n\n  /**\n   * Given a case class who's arguments are all declared fields on the class,\n   * returns an iterator of the field name and values\n   */\n  private[pipeline_execution_logger] def caseClassFields(\n    caseClass: Product\n  ): Iterator[(String, Any)] = {\n    val fieldValues = caseClass.productIterator.toSet\n    val fields = caseClass.getClass.getDeclaredFields.toSeq\n      .filterNot(f => f.isSynthetic || java.lang.reflect.Modifier.isStatic(f.getModifiers))\n    fields.iterator\n      .map { f =>\n        f.setAccessible(true)\n        f.getName -> f.get(caseClass)\n      }.filter { case (_, v) => fieldValues.contains(v) }\n  }\n\n  /**\n   * Returns whether a given [[Product]] is a case class which we can render nicely which:\n   * - has a [[Product.productArity]] <= the number of declared fields\n   * - isn't a built in binary operator\n   * - isn't a tuple\n   * - who's arguments are fields (not methods)\n   * - every [[Product.productElement]] has a corresponding field\n   *\n   * This will return false for some case classes where we can not reliably determine which field names correspond to\n   * each value in the case class (this can happen if a case class implements an abstract class resulting in val fields\n   * becoming methods.\n   */\n  private[pipeline_execution_logger] def isRenderableCaseClass(caseClass: Product): Boolean = {\n    val possibleToBeRenderableCaseClass =\n      caseClass.getClass.getDeclaredFields.length >= caseClass.productArity\n    val isntBuiltInOperator =\n      !(caseClass.productArity == 2 && Util.isOperator(caseClass.productPrefix))\n    val isntTuple = !caseClass.getClass.getName.startsWith(tuplePrefix)\n    val declaredFieldsMatchesCaseClassFields = {\n      val caseClassFields = caseClass.productIterator.toSet\n      caseClass.getClass.getDeclaredFields.iterator\n        .filterNot(f => f.isSynthetic || java.lang.reflect.Modifier.isStatic(f.getModifiers))\n        .count { f =>\n          f.setAccessible(true)\n          caseClassFields.contains(f.get(caseClass))\n        } >= caseClass.productArity\n    }\n\n    possibleToBeRenderableCaseClass && isntBuiltInOperator && isntTuple && declaredFieldsMatchesCaseClassFields\n  }\n\n  /** Makes a [[Tree]] which will render as `key = value` */\n  private def keyValuePair(key: String, value: Tree): Tree = {\n    Tree.Infix(Tree.Literal(key), \"=\", value)\n  }\n\n  /**\n   * Special handling for case classes who's field names can be easily paired with their values.\n   * This will make the [[PPrinter]] render them as\n   * {{{\n   *   CaseClassName(\n   *     field1 = value1,\n   *     field2 = value2\n   *   )\n   * }}}\n   * instead of\n   * {{{\n   *   CaseClassName(\n   *     value1,\n   *     value2\n   *   )\n   * }}}\n   *\n   * For case classes who's fields end up being compiled as methods, this will fall back\n   * to the built in handling of case classes without their field names.\n   */\n  private[pipeline_execution_logger] def additionalHandlers: PartialFunction[Any, Tree] = {\n    case caseClass: Product if isRenderableCaseClass(caseClass) =>\n      Tree.Apply(\n        caseClass.productPrefix,\n        caseClassFields(caseClass).flatMap {\n          case (key, value) =>\n            val valueTree = printer.treeify(value, false, true)\n            Seq(keyValuePair(key, valueTree))\n        }\n      )\n  }\n\n  /**\n   * [[PPrinter]] instance to use when rendering scala objects\n   * uses BlackAndWhite because colors mangle the output when looking at the logs in plain text\n   */\n  private val printer: PPrinter = PPrinter.BlackWhite.copy(\n    // arbitrary high value to turn off truncation\n    defaultHeight = Int.MaxValue,\n    // the relatively high width will cause some wrapping but many of the pretty printed objects\n    // will be sparse (e.g. None,\\n None,\\n, None,\\n)\n    defaultWidth = 300,\n    // use reflection to print field names (can be deleted in Scala 2.13)\n    additionalHandlers = additionalHandlers\n  )\n\n  /** Given any scala object, return a String representation of it */\n  private[pipeline_execution_logger] def objectAsString(o: Any): String =\n    printer.tokenize(o).mkString\n}\n\n@Singleton\nclass AllowListedPipelineExecutionLogger @Inject() (\n  @Flag(ServiceLocal) isServiceLocal: Boolean,\n  @Flag(PipelineExecutionLoggerAllowList) allowList: Seq[String],\n  statsReceiver: StatsReceiver)\n    extends PipelineExecutionLogger\n    with Logging {\n\n  private val scopedStats = statsReceiver.scope(\"AllowListedPipelineExecutionLogger\")\n\n  val allowListRoles: Set[String] = allowList.toSet\n\n  private val futurePool =\n    FuturePools.boundedFixedThreadPool(\n      \"AllowListedPipelineExecutionLogger\",\n      // single thread, may need to be adjusted higher if it cant keep up with the work queue\n      fixedThreadCount = 1,\n      // arbitrarily large enough to handle spikes without causing large allocations or retaining past multiple GC cycles\n      workQueueSize = 100,\n      scopedStats\n    )\n\n  private val futureObserver = new FutureObserver[Unit](scopedStats, Seq.empty)\n\n  private val loggerOutputPath = Try(System.getProperty(\"log.allow_listed_execution_logger.output\"))\n\n  override def apply(pipelineQuery: PipelineQuery, message: Any): Unit = {\n\n    val userRoles: Set[String] = pipelineQuery.clientContext.userRoles.getOrElse(Set.empty)\n\n    // Check if this request is in the allowlist via a cleverly optimized set intersection\n    val allowListed =\n      if (allowListRoles.size > userRoles.size)\n        userRoles.exists(allowListRoles.contains)\n      else\n        allowListRoles.exists(userRoles.contains)\n\n    if (isServiceLocal || allowListed) {\n      futureObserver(\n        /**\n         * failure to enqueue the work will result with a failed [[com.twitter.util.Future]]\n         * containing a [[java.util.concurrent.RejectedExecutionException]] which the wrapping [[futureObserver]]\n         * will record metrics for.\n         */\n        futurePool {\n          logger.info(AllowListedPipelineExecutionLogger.objectAsString(message))\n\n          if (isServiceLocal && loggerOutputPath.isReturn) {\n            println(s\"Logged request to: ${loggerOutputPath.get()}\")\n          }\n        }\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_execution_logger/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/com/lihaoyi:pprint\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"finatra/inject/inject-app/src/main/java/com/twitter/inject/annotations\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala/com/twitter/inject\",\n        \"finatra/inject/inject-slf4j/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/product_mixer_flags\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n    ],\n    exports = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/google/inject/extensions:guice-assistedinject\",\n        \"3rdparty/jvm/com/lihaoyi:pprint\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"finatra/inject/inject-app/src/main/java/com/twitter/inject/annotations\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-slf4j/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/product_mixer_flags\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_execution_logger/PipelineExecutionLogger.scala",
    "content": "package com.twitter.product_mixer.core.service.pipeline_execution_logger\n\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\n\ntrait PipelineExecutionLogger {\n  def apply(pipelineQuery: PipelineQuery, message: Any): Unit\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_executor/PipelineExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.pipeline_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.pipeline.Pipeline\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.InvalidPipelineSelected\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorObserver\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.stitch.Arrow\nimport com.twitter.util.logging.Logging\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * PipelineExecutor executes a single pipeline (of any type)\n * It does not currently support fail open/closed policies like CandidatePipelineExecutor does\n * In the future, maybe they can be merged.\n */\n\ncase class PipelineExecutorRequest[Query](query: Query, pipelineIdentifier: ComponentIdentifier)\n\n@Singleton\nclass PipelineExecutor @Inject() (override val statsReceiver: StatsReceiver)\n    extends Executor\n    with Logging {\n\n  def arrow[Query, ResultType](\n    pipelineByIdentifier: Map[ComponentIdentifier, Pipeline[Query, ResultType]],\n    qualityFactorObserverByPipeline: Map[ComponentIdentifier, QualityFactorObserver],\n    context: Executor.Context\n  ): Arrow[PipelineExecutorRequest[Query], PipelineExecutorResult[ResultType]] = {\n\n    val wrappedPipelineArrowsByIdentifier = pipelineByIdentifier.mapValues { pipeline =>\n      wrapPipelineWithExecutorBookkeeping(\n        context,\n        pipeline.identifier,\n        qualityFactorObserverByPipeline.get(pipeline.identifier))(pipeline.arrow)\n    }\n\n    val appliedPipelineArrow = Arrow\n      .identity[PipelineExecutorRequest[Query]]\n      .map {\n        case PipelineExecutorRequest(query, pipelineIdentifier) =>\n          val pipeline = wrappedPipelineArrowsByIdentifier.getOrElse(\n            pipelineIdentifier,\n            // throwing instead of returning a `Throw(_)` and then `.lowerFromTry` because this is an exceptional case and we want to emphasize that by explicitly throwing\n            // this case should never happen since this is checked in the `PipelineSelectorExecutor` but we check it anyway\n            throw PipelineFailure(\n              InvalidPipelineSelected,\n              s\"${context.componentStack.peek} attempted to execute $pipelineIdentifier\",\n              // the `componentStack` includes the missing pipeline so it can show up in metrics easier\n              componentStack = Some(context.componentStack.push(pipelineIdentifier))\n            )\n          )\n          (pipeline, query)\n      }\n      // less efficient than an `andThen` but since we dispatch this dynamically we need to use either `applyArrow` or `flatMap` and this is the better of those options\n      .applyArrow\n      .map(PipelineExecutorResult(_))\n\n    // no additional error handling needed since we populate the component stack above already\n    appliedPipelineArrow\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_executor/PipelineExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service.pipeline_executor\n\nimport com.twitter.product_mixer.core.pipeline.PipelineResult\nimport com.twitter.product_mixer.core.service.ExecutorResult\n\ncase class PipelineExecutorResult[ResultType](\n  pipelineResult: PipelineResult[ResultType])\n    extends ExecutorResult\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_result_side_effect_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/side_effect\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_result_side_effect_executor/PipelineResultSideEffectExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.pipeline_result_side_effect_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.side_effect.ExecuteSynchronously\nimport com.twitter.product_mixer.core.functional_component.side_effect.FailOpen\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect\nimport com.twitter.product_mixer.core.functional_component.side_effect.PipelineResultSideEffect.Inputs\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.SideEffectIdentifier\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport com.twitter.product_mixer.core.service.pipeline_result_side_effect_executor.PipelineResultSideEffectExecutor._\nimport com.twitter.stitch.Arrow\nimport com.twitter.util.Return\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PipelineResultSideEffectExecutor @Inject() (override val statsReceiver: StatsReceiver)\n    extends Executor {\n  def arrow[Query <: PipelineQuery, MixerDomainResultType <: HasMarshalling](\n    sideEffects: Seq[PipelineResultSideEffect[Query, MixerDomainResultType]],\n    context: Executor.Context\n  ): Arrow[Inputs[Query, MixerDomainResultType], PipelineResultSideEffectExecutor.Result] = {\n\n    val individualArrows: Seq[\n      Arrow[Inputs[Query, MixerDomainResultType], (SideEffectIdentifier, SideEffectResultType)]\n    ] = sideEffects.map {\n      case synchronousSideEffect: ExecuteSynchronously =>\n        val failsRequestIfThrows = {\n          wrapComponentWithExecutorBookkeeping(context, synchronousSideEffect.identifier)(\n            Arrow.flatMap(synchronousSideEffect.apply))\n        }\n        synchronousSideEffect match {\n          case failOpen: FailOpen =>\n            // lift the failure\n            failsRequestIfThrows.liftToTry.map(t =>\n              (failOpen.identifier, SynchronousSideEffectResult(t)))\n          case _ =>\n            // don't encapsulate the failure\n            failsRequestIfThrows.map(_ =>\n              (synchronousSideEffect.identifier, SynchronousSideEffectResult(Return.Unit)))\n        }\n\n      case sideEffect =>\n        Arrow\n          .async(\n            wrapComponentWithExecutorBookkeeping(context, sideEffect.identifier)(\n              Arrow.flatMap(sideEffect.apply)))\n          .andThen(Arrow.value((sideEffect.identifier, SideEffectResult)))\n    }\n\n    val conditionallyRunArrows = sideEffects.zip(individualArrows).map {\n      case (\n            sideEffect: Conditionally[\n              PipelineResultSideEffect.Inputs[Query, MixerDomainResultType] @unchecked\n            ],\n            arrow) =>\n        Arrow.ifelse[\n          Inputs[Query, MixerDomainResultType],\n          (SideEffectIdentifier, SideEffectResultType)](\n          input => sideEffect.onlyIf(input),\n          arrow,\n          Arrow.value((sideEffect.identifier, TurnedOffByConditionally)))\n      case (_, arrow) => arrow\n    }\n\n    Arrow\n      .collect(conditionallyRunArrows)\n      .map(results => Result(results))\n  }\n}\n\nobject PipelineResultSideEffectExecutor {\n  case class Result(sideEffects: Seq[(SideEffectIdentifier, SideEffectResultType)])\n      extends ExecutorResult\n\n  sealed trait SideEffectResultType\n\n  /** The [[PipelineResultSideEffect]] was executed asynchronously in a fire-and-forget way so no result will be available */\n  case object SideEffectResult extends SideEffectResultType\n\n  /** The result of the [[PipelineResultSideEffect]] that was executed with [[ExecuteSynchronously]] */\n  case class SynchronousSideEffectResult(result: Try[Unit]) extends SideEffectResultType\n\n  /** The result for when a [[PipelineResultSideEffect]] is turned off by [[Conditionally]]'s [[Conditionally.onlyIf]] */\n  case object TurnedOffByConditionally extends SideEffectResultType\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_selector_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"finatra/inject/inject-slf4j/src/main/scala/com/twitter/inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_selector_executor/PipelineSelectorExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.pipeline_selector_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.logging.Logging\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PlatformIdentifier\nimport com.twitter.product_mixer.core.pipeline.Pipeline\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.InvalidPipelineSelected\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.stitch.Arrow\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass PipelineSelectorExecutor @Inject() (override val statsReceiver: StatsReceiver)\n    extends Executor\n    with Logging {\n\n  val identifier: ComponentIdentifier = PlatformIdentifier(\"PipelineSelector\")\n\n  def arrow[Query <: PipelineQuery, Response](\n    pipelineByIdentifier: Map[ComponentIdentifier, Pipeline[Query, Response]],\n    pipelineSelector: Query => ComponentIdentifier,\n    context: Executor.Context\n  ): Arrow[Query, PipelineSelectorExecutorResult] = {\n\n    val validateSelectedPipelineExists = Arrow\n      .map(pipelineSelector)\n      .map { chosenIdentifier =>\n        if (pipelineByIdentifier.contains(chosenIdentifier)) {\n          PipelineSelectorExecutorResult(chosenIdentifier)\n        } else {\n          // throwing instead of returning a `Throw(_)` and then `.lowerFromTry` because this is an exceptional case and we want to emphasize that by explicitly throwing\n          throw PipelineFailure(\n            InvalidPipelineSelected,\n            s\"${context.componentStack.peek} attempted to select $chosenIdentifier\",\n            // the `componentStack` includes the missing pipeline so it can show up in metrics easier\n            componentStack = Some(context.componentStack.push(chosenIdentifier))\n          )\n        }\n      }\n\n    wrapWithErrorHandling(context, identifier)(validateSelectedPipelineExists)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/pipeline_selector_executor/PipelineSelectorExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service.pipeline_selector_executor\n\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\n\ncase class PipelineSelectorExecutorResult(pipelineIdentifier: ComponentIdentifier)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/quality_factor_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/quality_factor\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/quality_factor_executor/QualityFactorExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service.quality_factor_executor\n\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\n\ncase class QualityFactorExecutorResult(\n  pipelineQualityFactors: Map[ComponentIdentifier, Double])\n\nobject QualityFactorExecutorResult {\n  val empty: QualityFactorExecutorResult = QualityFactorExecutorResult(Map.empty)\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/query_feature_hydrator_executor/AsyncIndividualFeatureHydratorResultSerializer.scala",
    "content": "package com.twitter.product_mixer.core.service.query_feature_hydrator_executor\n\nimport com.fasterxml.jackson.core.JsonGenerator\nimport com.fasterxml.jackson.databind.JsonSerializer\nimport com.fasterxml.jackson.databind.SerializerProvider\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor.AsyncIndividualFeatureHydratorResult\n\n/** A [[JsonSerializer]] that skips the `Stitch` values */\nprivate[query_feature_hydrator_executor] class AsyncIndividualFeatureHydratorResultSerializer()\n    extends JsonSerializer[AsyncIndividualFeatureHydratorResult] {\n\n  override def serialize(\n    asyncIndividualFeatureHydratorResult: AsyncIndividualFeatureHydratorResult,\n    gen: JsonGenerator,\n    serializers: SerializerProvider\n  ): Unit =\n    serializers.defaultSerializeValue(\n      // implicitly calls `toString` on the identifier because they are keys in the Map\n      Map(\n        asyncIndividualFeatureHydratorResult.hydrateBefore ->\n          asyncIndividualFeatureHydratorResult.features.map(_.toString)),\n      gen\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/query_feature_hydrator_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/asyncfeaturemap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/feature_hydrator_observer\",\n        \"util/util-jackson/src/main/scala/com/twitter/util/jackson\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/feature/featuremap/asyncfeaturemap\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/feature_hydrator/featurestorev1\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/query_feature_hydrator_executor/QueryFeatureHydratorExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.query_feature_hydrator_executor\n\nimport com.fasterxml.jackson.databind.annotation.JsonSerialize\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.feature.Feature\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.feature.featuremap.asyncfeaturemap.AsyncFeatureMap\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.AsyncHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.BaseQueryFeatureHydrator\nimport com.twitter.product_mixer.core.functional_component.feature_hydrator.featurestorev1.FeatureStoreV1QueryFeatureHydrator\nimport com.twitter.product_mixer.core.model.common.Conditionally\nimport com.twitter.product_mixer.core.model.common.identifier.FeatureHydratorIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.PipelineStepIdentifier\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.Executor._\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport com.twitter.product_mixer.core.service.feature_hydrator_observer.FeatureHydratorObserver\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor.AsyncIndividualFeatureHydratorResult\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor.BaseIndividualFeatureHydratorResult\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor.FeatureHydratorDisabled\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor.IndividualFeatureHydratorResult\nimport com.twitter.product_mixer.core.service.query_feature_hydrator_executor.QueryFeatureHydratorExecutor.validateAsyncQueryFeatureHydrator\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass QueryFeatureHydratorExecutor @Inject() (override val statsReceiver: StatsReceiver)\n    extends Executor {\n\n  def arrow[Query <: PipelineQuery](\n    hydrators: Seq[BaseQueryFeatureHydrator[Query, _]],\n    validPipelineSteps: Set[PipelineStepIdentifier],\n    context: Executor.Context\n  ): Arrow[Query, QueryFeatureHydratorExecutor.Result] = {\n\n    val observer = new FeatureHydratorObserver(statsReceiver, hydrators, context)\n    val hydratorsWithErrorHandling =\n      hydrators.map { hydrator =>\n        val queryFeatureHydratorArrow =\n          getQueryHydratorArrow(hydrator, context, observer)\n        val wrappedWithAsyncHandling =\n          handleAsyncHydrator(hydrator, validPipelineSteps, queryFeatureHydratorArrow)\n        handleConditionally(hydrator, wrappedWithAsyncHandling)\n      }\n\n    Arrow\n      .collect(hydratorsWithErrorHandling)\n      .map {\n        results: Seq[\n          (FeatureHydratorIdentifier, BaseIndividualFeatureHydratorResult)\n        ] =>\n          val combinedFeatureMap = FeatureMap.merge(results.collect {\n            case (_, IndividualFeatureHydratorResult(featureMap)) => featureMap\n          })\n\n          val asyncFeatureMaps = results.collect {\n            case (\n                  hydratorIdentifier,\n                  AsyncIndividualFeatureHydratorResult(hydrateBefore, featuresToHydrate, ref)) =>\n              (hydratorIdentifier, hydrateBefore, featuresToHydrate, ref)\n          }\n\n          QueryFeatureHydratorExecutor.Result(\n            individualFeatureMaps = results.toMap,\n            featureMap = combinedFeatureMap,\n            asyncFeatureMap = AsyncFeatureMap.fromFeatureMaps(asyncFeatureMaps)\n          )\n      }\n  }\n\n  def handleConditionally[Query <: PipelineQuery](\n    hydrator: BaseQueryFeatureHydrator[Query, _],\n    arrow: Arrow[\n      Query,\n      BaseIndividualFeatureHydratorResult\n    ]\n  ): Arrow[\n    Query,\n    (FeatureHydratorIdentifier, BaseIndividualFeatureHydratorResult)\n  ] = {\n    val conditionallyRunArrow = hydrator match {\n      case hydrator: BaseQueryFeatureHydrator[Query, _] with Conditionally[Query @unchecked] =>\n        Arrow.ifelse[Query, BaseIndividualFeatureHydratorResult](\n          hydrator.onlyIf,\n          arrow,\n          Arrow.value(FeatureHydratorDisabled)\n        )\n      case _ => arrow\n    }\n\n    Arrow.join(\n      Arrow.value(hydrator.identifier),\n      conditionallyRunArrow\n    )\n  }\n\n  def handleAsyncHydrator[Query <: PipelineQuery](\n    hydrator: BaseQueryFeatureHydrator[Query, _],\n    validPipelineSteps: Set[PipelineStepIdentifier],\n    arrow: Arrow[\n      Query,\n      IndividualFeatureHydratorResult\n    ]\n  ): Arrow[Query, BaseIndividualFeatureHydratorResult] = {\n    hydrator match {\n      case hydrator: BaseQueryFeatureHydrator[\n            Query,\n            _\n          ] with AsyncHydrator =>\n        validateAsyncQueryFeatureHydrator(hydrator, validPipelineSteps)\n\n        startArrowAsync(arrow.map(_.featureMap))\n          .map { ref =>\n            AsyncIndividualFeatureHydratorResult(\n              hydrator.hydrateBefore,\n              hydrator.features.asInstanceOf[Set[Feature[_, _]]],\n              ref\n            )\n          }\n\n      case _ => arrow\n    }\n  }\n\n  def getQueryHydratorArrow[Query <: PipelineQuery](\n    hydrator: BaseQueryFeatureHydrator[Query, _],\n    context: Executor.Context,\n    queryFeatureHydratorObserver: FeatureHydratorObserver\n  ): Arrow[Query, IndividualFeatureHydratorResult] = {\n\n    val componentExecutorContext = context.pushToComponentStack(hydrator.identifier)\n    val hydratorArrow: Arrow[Query, FeatureMap] =\n      Arrow.flatMap { query: Query => hydrator.hydrate(query) }\n\n    val validationFn: FeatureMap => FeatureMap = hydrator match {\n      // Feature store query hydrators store the resulting PredictionRecord and not\n      // the features, so we cannot validate the same way\n      case _: FeatureStoreV1QueryFeatureHydrator[Query] =>\n        identity\n      case _ =>\n        validateFeatureMap(\n          hydrator.features.asInstanceOf[Set[Feature[_, _]]],\n          _,\n          componentExecutorContext)\n    }\n\n    // record the component-level stats\n    val observedArrow =\n      wrapComponentWithExecutorBookkeeping[Query, FeatureMap](\n        context,\n        hydrator.identifier\n      )(hydratorArrow.map(validationFn))\n\n    // store non-configuration errors in the FeatureMap\n    val liftNonValidationFailuresToFailedFeatures = Arrow.handle[FeatureMap, FeatureMap] {\n      case NotAMisconfiguredFeatureMapFailure(e) =>\n        featureMapWithFailuresForFeatures(\n          hydrator.features.asInstanceOf[Set[Feature[_, _]]],\n          e,\n          componentExecutorContext)\n    }\n\n    val observedLiftedAndWrapped = observedArrow\n      .andThen(liftNonValidationFailuresToFailedFeatures)\n      .applyEffect(Arrow.map[FeatureMap, Unit](featureMap =>\n        // record per-feature stats, this is separate from the component stats handled by `wrapWithExecutorBookkeeping`\n        queryFeatureHydratorObserver.observeFeatureSuccessAndFailures(hydrator, Seq(featureMap))))\n      .map(IndividualFeatureHydratorResult)\n\n    observedLiftedAndWrapped\n  }\n}\n\nobject QueryFeatureHydratorExecutor {\n  case class Result(\n    individualFeatureMaps: Map[\n      FeatureHydratorIdentifier,\n      BaseIndividualFeatureHydratorResult\n    ],\n    featureMap: FeatureMap,\n    asyncFeatureMap: AsyncFeatureMap)\n      extends ExecutorResult\n\n  sealed trait BaseIndividualFeatureHydratorResult\n\n  case object FeatureHydratorDisabled extends BaseIndividualFeatureHydratorResult\n  case class IndividualFeatureHydratorResult(featureMap: FeatureMap)\n      extends BaseIndividualFeatureHydratorResult\n\n  /** Async result, serializes without the [[Stitch]] field since it's not serializable */\n  @JsonSerialize(using = classOf[AsyncIndividualFeatureHydratorResultSerializer])\n  case class AsyncIndividualFeatureHydratorResult(\n    hydrateBefore: PipelineStepIdentifier,\n    features: Set[Feature[_, _]],\n    ref: Stitch[FeatureMap])\n      extends BaseIndividualFeatureHydratorResult\n\n  /**\n   * Validates whether the [[AsyncHydrator.hydrateBefore]] [[PipelineStepIdentifier]] is valid\n   *\n   * @param asyncQueryFeatureHydrator the hydrator to validate\n   * @param validPipelineSteps        a Set of [[PipelineStepIdentifier]]s which are valid places to populate async\n   *                                  [[Feature]]s in a [[com.twitter.product_mixer.core.pipeline.Pipeline]]\n   */\n  def validateAsyncQueryFeatureHydrator(\n    asyncQueryFeatureHydrator: AsyncHydrator,\n    validPipelineSteps: Set[PipelineStepIdentifier]\n  ): Unit =\n    require(\n      validPipelineSteps.contains(asyncQueryFeatureHydrator.hydrateBefore),\n      s\"`AsyncHydrator.hydrateBefore` contained ${asyncQueryFeatureHydrator.hydrateBefore} which was not in the parent pipeline's \" +\n        s\"`PipelineConfig` Companion object field `stepsAsyncFeatureHydrationCanBeCompletedBy = $validPipelineSteps`.\"\n    )\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/scoring_pipeline_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/scoring\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/scoring\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/scoring_pipeline_executor/ScoringPipelineExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.scoring_pipeline_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.feature.featuremap.FeatureMap\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.identifier.ComponentIdentifier\nimport com.twitter.product_mixer.core.model.common.identifier.ScoringPipelineIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.FailOpenPolicy\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipeline\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipelineResult\nimport com.twitter.product_mixer.core.quality_factor.QualityFactorObserver\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.scoring_pipeline_executor.ScoringPipelineExecutor.ScoringPipelineState\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Arrow.Iso\nimport com.twitter.util.logging.Logging\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.collection.immutable.Queue\n\n@Singleton\nclass ScoringPipelineExecutor @Inject() (override val statsReceiver: StatsReceiver)\n    extends Executor\n    with Logging {\n  def arrow[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n    pipelines: Seq[ScoringPipeline[Query, Candidate]],\n    context: Executor.Context,\n    defaultFailOpenPolicy: FailOpenPolicy,\n    failOpenPolicies: Map[ScoringPipelineIdentifier, FailOpenPolicy],\n    qualityFactorObserverByPipeline: Map[ComponentIdentifier, QualityFactorObserver],\n  ): Arrow[ScoringPipelineExecutor.Inputs[Query], ScoringPipelineExecutorResult[Candidate]] = {\n    val scoringPipelineArrows = pipelines.map { pipeline =>\n      val failOpenPolicy = failOpenPolicies.getOrElse(pipeline.identifier, defaultFailOpenPolicy)\n      val qualityFactorObserver = qualityFactorObserverByPipeline.get(pipeline.identifier)\n\n      getIsoArrowForScoringPipeline(\n        pipeline,\n        context,\n        failOpenPolicy,\n        qualityFactorObserver\n      )\n    }\n    val combinedArrow = isoArrowsSequentially(scoringPipelineArrows)\n    Arrow\n      .map[ScoringPipelineExecutor.Inputs[Query], ScoringPipelineState[Query, Candidate]] {\n        case input =>\n          ScoringPipelineState(\n            input.query,\n            input.itemCandidatesWithDetails,\n            ScoringPipelineExecutorResult(input.itemCandidatesWithDetails, Queue.empty))\n      }.flatMapArrow(combinedArrow).map { state =>\n        state.executorResult.copy(individualPipelineResults =\n          // materialize the Queue into a List for faster future iterations\n          state.executorResult.individualPipelineResults.toList)\n      }\n  }\n\n  private def getIsoArrowForScoringPipeline[\n    Query <: PipelineQuery,\n    Candidate <: UniversalNoun[Any]\n  ](\n    pipeline: ScoringPipeline[Query, Candidate],\n    context: Executor.Context,\n    failOpenPolicy: FailOpenPolicy,\n    qualityFactorObserver: Option[QualityFactorObserver]\n  ): Iso[ScoringPipelineState[Query, Candidate]] = {\n    val pipelineArrow = Arrow\n      .map[ScoringPipelineState[Query, Candidate], ScoringPipeline.Inputs[Query]] { state =>\n        ScoringPipeline.Inputs(state.query, state.allCandidates)\n      }.flatMapArrow(pipeline.arrow)\n\n    val observedArrow = wrapPipelineWithExecutorBookkeeping(\n      context,\n      pipeline.identifier,\n      qualityFactorObserver,\n      failOpenPolicy)(pipelineArrow)\n\n    Arrow\n      .zipWithArg(\n        observedArrow\n      ).map {\n        case (\n              scoringPipelinesState: ScoringPipelineState[Query, Candidate],\n              scoringPipelineResult: ScoringPipelineResult[Candidate]) =>\n          val updatedCandidates: Seq[ItemCandidateWithDetails] =\n            mkUpdatedCandidates(pipeline.identifier, scoringPipelinesState, scoringPipelineResult)\n          ScoringPipelineState(\n            scoringPipelinesState.query,\n            updatedCandidates,\n            scoringPipelinesState.executorResult\n              .copy(\n                updatedCandidates,\n                scoringPipelinesState.executorResult.individualPipelineResults :+ scoringPipelineResult)\n          )\n      }\n  }\n\n  private def mkUpdatedCandidates[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n    scoringPipelineIdentifier: ScoringPipelineIdentifier,\n    scoringPipelinesState: ScoringPipelineState[Query, Candidate],\n    scoringPipelineResult: ScoringPipelineResult[Candidate]\n  ): Seq[ItemCandidateWithDetails] = {\n    if (scoringPipelineResult.failure.isEmpty) {\n\n      /**\n       * It's important that we map back from which actual item candidate was scored by looking\n       * at the selector results. This is to defend against the same candidate being selected\n       * from two different candidate pipelines. If one is selected and the other isn't, we\n       * should only score the selected one. If both are selected and each is scored differently\n       * we should get the right score for each.\n       */\n      val selectedItemCandidates: Seq[ItemCandidateWithDetails] =\n        scoringPipelineResult.selectorResults\n          .getOrElse(throw PipelineFailure(\n            IllegalStateFailure,\n            s\"Missing Selector Results in Scoring Pipeline $scoringPipelineIdentifier\")).selectedCandidates.collect {\n            case itemCandidateWithDetails: ItemCandidateWithDetails =>\n              itemCandidateWithDetails\n          }\n      val scoredFeatureMaps: Seq[FeatureMap] = scoringPipelineResult.result\n        .getOrElse(Seq.empty).map(_.features)\n\n      if (scoredFeatureMaps.isEmpty) {\n        // It's possible that all Scorers are [[Conditionally]] off. In this case, we return empty\n        // and don't validate the list size since this is done in the hydrator/scorer executor.\n        scoringPipelinesState.allCandidates\n      } else if (selectedItemCandidates.length != scoredFeatureMaps.length) {\n        // The length of the inputted candidates should always match the returned feature map, unless\n        throw PipelineFailure(\n          IllegalStateFailure,\n          s\"Missing configured scorer result, length of scorer results does not match the length of selected candidates\")\n      } else {\n        /* Zip the selected item candidate seq back to the scored feature maps, this works\n         * because the scored results will always have the same number of elements returned\n         * and it should match the same order. We then loop through all candidates because the\n         * expectation is to always keep the result since a subsequent scoring pipeline can score a\n         * candidate that the current one did not. We only update the feature map of the candidate\n         *  if it was selected and scored.\n         */\n        val selectedItemCandidateToScorerMap: Map[ItemCandidateWithDetails, FeatureMap] =\n          selectedItemCandidates.zip(scoredFeatureMaps).toMap\n        scoringPipelinesState.allCandidates.map { itemCandidateWithDetails =>\n          selectedItemCandidateToScorerMap.get(itemCandidateWithDetails) match {\n            case Some(scorerResult) =>\n              itemCandidateWithDetails.copy(features =\n                itemCandidateWithDetails.features ++ scorerResult)\n            case None => itemCandidateWithDetails\n          }\n        }\n      }\n    } else {\n      // If the underlying scoring pipeline has failed open, just keep the existing candidates\n      scoringPipelinesState.allCandidates\n    }\n  }\n}\n\nobject ScoringPipelineExecutor {\n  private case class ScoringPipelineState[Query <: PipelineQuery, Candidate <: UniversalNoun[Any]](\n    query: Query,\n    allCandidates: Seq[ItemCandidateWithDetails],\n    executorResult: ScoringPipelineExecutorResult[Candidate])\n\n  case class Inputs[Query <: PipelineQuery](\n    query: Query,\n    itemCandidatesWithDetails: Seq[ItemCandidateWithDetails])\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/scoring_pipeline_executor/ScoringPipelineExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service.scoring_pipeline_executor\n\nimport com.twitter.product_mixer.core.model.common.UniversalNoun\nimport com.twitter.product_mixer.core.model.common.presentation.ItemCandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.scoring.ScoringPipelineResult\n\ncase class ScoringPipelineExecutorResult[Candidate <: UniversalNoun[Any]](\n  result: Seq[ItemCandidateWithDetails],\n  individualPipelineResults: Seq[ScoringPipelineResult[Candidate]])\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/selector_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n        \"util/util-core:scala\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/selector\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/candidate\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/selector_executor/SelectorExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.selector_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.selector.Selector\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.identifier.SelectorIdentifier\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.pipeline.PipelineQuery\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.IllegalStateFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.stitch.Arrow\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Applies a `Seq[Selector]` in sequential order.\n * Returns the results, and also a detailed list each selector's results (for debugging / understandability).\n */\n@Singleton\nclass SelectorExecutor @Inject() (override val statsReceiver: StatsReceiver) extends Executor {\n  def arrow[Query <: PipelineQuery](\n    selectors: Seq[Selector[Query]],\n    context: Executor.Context\n  ): Arrow[SelectorExecutor.Inputs[Query], SelectorExecutorResult] = {\n\n    if (selectors.isEmpty) {\n      throw PipelineFailure(\n        IllegalStateFailure,\n        \"Must provide a non-empty Seq of Selectors. Check the config indicated by the componentStack and ensure that a non-empty Selector Seq is provided.\",\n        componentStack = Some(context.componentStack)\n      )\n    }\n\n    val selectorArrows =\n      selectors.zipWithIndex.foldLeft(Arrow.identity[(Query, IndexedSeq[SelectorResult])]) {\n        case (previousSelectorArrows, (selector, index)) =>\n          val selectorResult = getIndividualSelectorIsoArrow(selector, index, context)\n          previousSelectorArrows.andThen(selectorResult)\n      }\n\n    Arrow\n      .zipWithArg(\n        Arrow\n          .map[SelectorExecutor.Inputs[Query], (Query, IndexedSeq[SelectorResult])] {\n            case SelectorExecutor.Inputs(query, candidates) =>\n              (query, IndexedSeq(SelectorResult(candidates, Seq.empty)))\n          }.andThen(selectorArrows)).map {\n        case (inputs, (_, selectorResults)) =>\n          // the last results, safe because it's always non-empty since it starts with 1 element in it\n          val SelectorResult(remainingCandidates, result) = selectorResults.last\n\n          val resultsAndRemainingCandidates =\n            (result.iterator ++ remainingCandidates.iterator).toSet\n\n          // the droppedCandidates are all the candidates which are in neither the result or remainingCandidates\n          val droppedCandidates = inputs.candidatesWithDetails.iterator\n            .filterNot(resultsAndRemainingCandidates.contains)\n            .toIndexedSeq\n\n          SelectorExecutorResult(\n            selectedCandidates = result,\n            remainingCandidates = remainingCandidates,\n            droppedCandidates = droppedCandidates,\n            individualSelectorResults =\n              selectorResults.tail // `.tail` to remove the initial state we had\n          )\n      }\n  }\n\n  private def getIndividualSelectorIsoArrow[Query <: PipelineQuery](\n    selector: Selector[Query],\n    index: Int,\n    context: Executor.Context\n  ): Arrow.Iso[(Query, IndexedSeq[SelectorResult])] = {\n    val identifier = SelectorIdentifier(selector.getClass.getSimpleName, index)\n\n    val arrow = Arrow\n      .identity[(Query, IndexedSeq[SelectorResult])]\n      .map {\n        case (query, previousResults) =>\n          // last is safe here because we pass in a non-empty IndexedSeq\n          val previousResult = previousResults.last\n          val currentResult = selector.apply(\n            query,\n            previousResult.remainingCandidates,\n            previousResult.result\n          )\n          (query, previousResults :+ currentResult)\n      }\n\n    wrapComponentsWithTracingOnly(context, identifier)(\n      wrapWithErrorHandling(context, identifier)(\n        arrow\n      )\n    )\n  }\n}\n\nobject SelectorExecutor {\n  case class Inputs[Query <: PipelineQuery](\n    query: Query,\n    candidatesWithDetails: Seq[CandidateWithDetails])\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/selector_executor/SelectorExecutorResult.scala",
    "content": "package com.twitter.product_mixer.core.service.selector_executor\n\nimport com.twitter.product_mixer.core.functional_component.selector.SelectorResult\nimport com.twitter.product_mixer.core.model.common.presentation.CandidateWithDetails\nimport com.twitter.product_mixer.core.service.ExecutorResult\n\ncase class SelectorExecutorResult(\n  selectedCandidates: Seq[CandidateWithDetails],\n  remainingCandidates: Seq[CandidateWithDetails],\n  droppedCandidates: Seq[CandidateWithDetails],\n  individualSelectorResults: Seq[SelectorResult])\n    extends ExecutorResult\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/slice/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry\",\n        \"stitch/stitch-core\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql:api-media-graphql-scala\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql:graphql-scala\",\n        \"strato/config/src/thrift/com/twitter/strato/graphql:topics-graphql-scala\",\n        \"util/util-core:scala\",\n    ],\n    exports = [\n        \"strato/config/src/thrift/com/twitter/strato/graphql:graphql-scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/slice/SliceService.scala",
    "content": "package com.twitter.product_mixer.core.service.slice\n\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineRequest\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.graphql.thriftscala.SliceResult\nimport com.twitter.timelines.configapi.Params\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.reflect.runtime.universe.TypeTag\n\n/**\n * Look up and execute Slice products in the [[ProductPipelineRegistry]]\n */\n@Singleton\nclass SliceService @Inject() (productPipelineRegistry: ProductPipelineRegistry) {\n\n  def getSliceResponse[RequestType <: Request](\n    request: RequestType,\n    params: Params\n  )(\n    implicit requestTypeTag: TypeTag[RequestType]\n  ): Stitch[SliceResult] =\n    productPipelineRegistry\n      .getProductPipeline[RequestType, SliceResult](request.product)\n      .process(ProductPipelineRequest(request, params))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transformer_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/transformer\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transformer_executor/PerCandidateTransformerExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.transformer_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.transformer.Transformer\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.stitch.Arrow\nimport com.twitter.util.Try\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * For wrapping [[Transformer]]s that are applied per-candidate\n *\n * Records a single span for running all the components,\n * but stats per-component.\n */\n@Singleton\nclass PerCandidateTransformerExecutor @Inject() (override val statsReceiver: StatsReceiver)\n    extends Executor {\n\n  def arrow[In, Out](\n    transformer: Transformer[In, Out],\n    context: Executor.Context,\n  ): Arrow[Seq[In], Seq[Try[Out]]] = {\n    val perCandidateArrow = wrapPerCandidateComponentWithExecutorBookkeepingWithoutTracing(\n      context,\n      transformer.identifier\n    )(Arrow.map(transformer.transform)).liftToTry\n\n    wrapComponentsWithTracingOnly(\n      context,\n      transformer.identifier\n    )(Arrow.sequence(perCandidateArrow))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transformer_executor/TransformerExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.transformer_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.transformer.Transformer\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.stitch.Arrow\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass TransformerExecutor @Inject() (override val statsReceiver: StatsReceiver) extends Executor {\n  def arrow[In, Out](\n    transformer: Transformer[In, Out],\n    context: Executor.Context\n  ): Arrow[In, Out] = {\n    wrapComponentWithExecutorBookkeeping(\n      context,\n      transformer.identifier\n    )(Arrow.map(transformer.transform))\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transport_marshaller_executor/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n    exports = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/common/identifier\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service\",\n        \"stitch/stitch-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/transport_marshaller_executor/TransportMarshallerExecutor.scala",
    "content": "package com.twitter.product_mixer.core.service.transport_marshaller_executor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.core.functional_component.marshaller.TransportMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.HasMarshalling\nimport com.twitter.product_mixer.core.service.Executor\nimport com.twitter.product_mixer.core.service.ExecutorResult\nimport com.twitter.product_mixer.core.service.transport_marshaller_executor.TransportMarshallerExecutor.Inputs\nimport com.twitter.product_mixer.core.service.transport_marshaller_executor.TransportMarshallerExecutor.Result\nimport com.twitter.stitch.Arrow\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n/**\n * Executes a [[TransportMarshaller]].\n *\n * @note This is a synchronous transform, so we don't observe it directly. Failures and such\n *       can be observed at the parent pipeline.\n */\n@Singleton\nclass TransportMarshallerExecutor @Inject() (override val statsReceiver: StatsReceiver)\n    extends Executor {\n\n  def arrow[DomainResponseType <: HasMarshalling, TransportResponseType](\n    marshaller: TransportMarshaller[DomainResponseType, TransportResponseType],\n    context: Executor.Context\n  ): Arrow[Inputs[DomainResponseType], Result[TransportResponseType]] = {\n    val arrow =\n      Arrow.map[Inputs[DomainResponseType], Result[TransportResponseType]] {\n        case Inputs(domainResponse) => Result(marshaller(domainResponse))\n      }\n\n    wrapComponentWithExecutorBookkeeping(context, marshaller.identifier)(arrow)\n  }\n}\n\nobject TransportMarshallerExecutor {\n  case class Inputs[DomainResponseType <: HasMarshalling](domainResponse: DomainResponseType)\n  case class Result[TransportResponseType](result: TransportResponseType) extends ExecutorResult\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/urp/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/lihaoyi:pprint\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urp\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urp\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry\",\n        \"stitch/stitch-core\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/urp/UrpService.scala",
    "content": "package com.twitter.product_mixer.core.service.urp\n\nimport com.twitter.pages.render.{thriftscala => urp}\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineRequest\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Params\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.reflect.runtime.universe.TypeTag\n\n@Singleton\nclass UrpService @Inject() (productPipelineRegistry: ProductPipelineRegistry) {\n\n  def getUrpResponse[RequestType <: Request](\n    request: RequestType,\n    params: Params\n  )(\n    implicit requestTypeTag: TypeTag[RequestType]\n  ): Stitch[urp.Page] =\n    productPipelineRegistry\n      .getProductPipeline[RequestType, urp.Page](request.product)\n      .process(ProductPipelineRequest(request, params))\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/urt/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"configapi/configapi-core\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/functional_component/marshaller/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/model/marshalling/response/urt\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/product/registry\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"stitch/stitch-core\",\n        \"util/util-core:scala\",\n        \"util/util-jackson/src/main/scala/com/twitter/util/jackson\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/service/urt/UrtService.scala",
    "content": "package com.twitter.product_mixer.core.service.urt\n\nimport com.fasterxml.jackson.databind.SerializationFeature\nimport com.twitter.product_mixer.core.functional_component.marshaller.response.urt.UrtTransportMarshaller\nimport com.twitter.product_mixer.core.model.marshalling.request.Request\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.PipelineFailure\nimport com.twitter.product_mixer.core.pipeline.pipeline_failure.ProductDisabled\nimport com.twitter.product_mixer.core.pipeline.product.ProductPipelineRequest\nimport com.twitter.product_mixer.core.product.registry.ProductPipelineRegistry\nimport com.twitter.product_mixer.core.{thriftscala => t}\nimport com.twitter.stitch.Stitch\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.timelines.render.{thriftscala => urt}\nimport com.twitter.util.jackson.ScalaObjectMapper\n\nimport javax.inject.Inject\nimport javax.inject.Singleton\nimport scala.reflect.runtime.universe.TypeTag\n\n/**\n * Look up and execute products in the [[ProductPipelineRegistry]]\n */\n@Singleton\nclass UrtService @Inject() (productPipelineRegistry: ProductPipelineRegistry) {\n\n  def getUrtResponse[RequestType <: Request](\n    request: RequestType,\n    params: Params\n  )(\n    implicit requestTypeTag: TypeTag[RequestType]\n  ): Stitch[urt.TimelineResponse] =\n    productPipelineRegistry\n      .getProductPipeline[RequestType, urt.TimelineResponse](request.product)\n      .process(ProductPipelineRequest(request, params))\n      .handle {\n        // Detect ProductDisabled and convert it to TimelineUnavailable\n        case pipelineFailure: PipelineFailure if pipelineFailure.category == ProductDisabled =>\n          UrtTransportMarshaller.unavailable(\"\")\n      }\n\n  /**\n   * Get detailed pipeline execution as a serialized JSON String\n   */\n  def getPipelineExecutionResult[RequestType <: Request](\n    request: RequestType,\n    params: Params\n  )(\n    implicit requestTypeTag: TypeTag[RequestType]\n  ): Stitch[t.PipelineExecutionResult] =\n    productPipelineRegistry\n      .getProductPipeline[RequestType, urt.TimelineResponse](request.product)\n      .arrow(ProductPipelineRequest(request, params)).map { detailedResult =>\n        val mapper = ScalaObjectMapper()\n        // configure so that exception is not thrown whenever case class is not serializable\n        mapper.underlying.configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false)\n        val serializedJSON = mapper.writePrettyString(detailedResult)\n        t.PipelineExecutionResult(serializedJSON)\n      }\n\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-core/src/main\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"servo/util\",\n        \"snowflake:id\",\n        \"stitch/stitch-core\",\n        \"util/util-core:util-core-util\",\n        \"util/util-stats\",\n    ],\n    exports = [\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/pipeline/pipeline_failure\",\n        \"servo/util\",\n        \"snowflake:id\",\n        \"stitch/stitch-core\",\n        \"util/util-core:util-core-util\",\n        \"util/util-stats\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util/FuturePools.scala",
    "content": "package com.twitter.product_mixer.core.util\n\nimport com.twitter.concurrent.NamedPoolThreadFactory\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.Duration\nimport com.twitter.util.FuturePool\n\nimport java.util.concurrent.ArrayBlockingQueue\nimport java.util.concurrent.BlockingQueue\nimport java.util.concurrent.LinkedBlockingQueue\nimport java.util.concurrent.ThreadPoolExecutor\nimport java.util.concurrent.TimeUnit\n\n/**\n * Utility for making [[FuturePool]] with finite thread counts and different work queue options\n * while also monitoring the size of the work queue that's used.\n *\n * This only handles the cases where the number of threads are bounded.\n * For unbounded numbers of threads in a [[FuturePool]] use [[FuturePool.interruptibleUnboundedPool]] instead.\n */\nobject FuturePools {\n\n  /**\n   * Makes a [[FuturePool]] with a fixed number of threads and a work queue\n   * with a maximum size of `maxWorkQueueSize`.\n   *\n   * @note the [[FuturePool]] returns a failed [[com.twitter.util.Future]]s containing\n   *       [[java.util.concurrent.RejectedExecutionException]] when trying to enqueue\n   *       work when the work queue is full.\n   */\n  def boundedFixedThreadPool(\n    name: String,\n    fixedThreadCount: Int,\n    workQueueSize: Int,\n    statsReceiver: StatsReceiver\n  ): FuturePool =\n    futurePool(\n      name = name,\n      minThreads = fixedThreadCount,\n      maxThreads = fixedThreadCount,\n      keepIdleThreadsAlive = Duration.Zero,\n      workQueue = new ArrayBlockingQueue[Runnable](workQueueSize),\n      statsReceiver = statsReceiver\n    )\n\n  /**\n   * Makes a [[FuturePool]] with a fix number of threads and an unbounded work queue\n   *\n   * @note Since the work queue is unbounded, this will fill up memory if the available worker threads can't keep up\n   */\n  def unboundedFixedThreadPool(\n    name: String,\n    fixedThreadCount: Int,\n    statsReceiver: StatsReceiver\n  ): FuturePool =\n    futurePool(\n      name = name,\n      minThreads = fixedThreadCount,\n      maxThreads = fixedThreadCount,\n      keepIdleThreadsAlive = Duration.Zero,\n      workQueue = new LinkedBlockingQueue[Runnable],\n      statsReceiver = statsReceiver\n    )\n\n  /**\n   * Makes a [[FuturePool]] with the provided thread configuration and\n   * who's `workQueue` is monitored by a [[com.twitter.finagle.stats.Gauge]]\n   *\n   * @note if `minThreads` == `maxThreads` then the threadpool has a fixed size\n   *\n   * @param name name of the threadpool\n   * @param minThreads minimum number of threads in the pool\n   * @param maxThreads maximum number of threads in the pool\n   * @param keepIdleThreadsAlive threads that are idle for this long will be removed from\n   *                             the pool if there are more than `minThreads` in the pool.\n   *                             If the pool size is fixed this is ignored.\n   */\n  private def futurePool(\n    name: String,\n    minThreads: Int,\n    maxThreads: Int,\n    keepIdleThreadsAlive: Duration,\n    workQueue: BlockingQueue[Runnable],\n    statsReceiver: StatsReceiver\n  ): FuturePool = {\n    val gaugeReference = statsReceiver.addGauge(\"workQueueSize\")(workQueue.size())\n\n    val threadFactory = new NamedPoolThreadFactory(name, makeDaemons = true)\n\n    val executorService =\n      new ThreadPoolExecutor(\n        minThreads,\n        maxThreads, // ignored by ThreadPoolExecutor when an unbounded queue is provided\n        keepIdleThreadsAlive.inMillis,\n        TimeUnit.MILLISECONDS,\n        workQueue,\n        threadFactory)\n\n    FuturePool.interruptible(executorService)\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util/OffloadFuturePools.scala",
    "content": "package com.twitter.product_mixer.core.util\n\nimport com.twitter.finagle.offload.OffloadFuturePool\nimport com.twitter.util.Future\n\nobject OffloadFuturePools {\n\n  def parallelize[In, Out](\n    inputSeq: Seq[In],\n    transformer: In => Out,\n    parallelism: Int\n  ): Future[Seq[Out]] = {\n    parallelize(inputSeq, transformer.andThen(Some(_)), parallelism, None).map(_.flatten)\n  }\n\n  def parallelize[In, Out](\n    inputSeq: Seq[In],\n    transformer: In => Out,\n    parallelism: Int,\n    default: Out\n  ): Future[Seq[Out]] = {\n    val threadProcessFutures = (0 until parallelism).map { i =>\n      OffloadFuturePool.getPool(partitionAndProcessInput(inputSeq, transformer, i, parallelism))\n    }\n\n    val resultMap = Future.collect(threadProcessFutures).map(_.flatten.toMap)\n\n    Future.collect {\n      inputSeq.indices.map { idx =>\n        resultMap.map(_.getOrElse(idx, default))\n      }\n    }\n  }\n\n  private def partitionAndProcessInput[In, Out](\n    inputSeq: Seq[In],\n    transformer: In => Out,\n    threadId: Int,\n    parallelism: Int\n  ): Seq[(Int, Out)] = {\n    partitionInputForThread(inputSeq, threadId, parallelism)\n      .map {\n        case (inputRecord, idx) =>\n          (idx, transformer(inputRecord))\n      }\n  }\n\n  private def partitionInputForThread[In](\n    inputSeq: Seq[In],\n    threadId: Int,\n    parallelism: Int\n  ): Seq[(In, Int)] = {\n    inputSeq.zipWithIndex\n      .filter {\n        case (_, idx) => idx % parallelism == threadId\n        case _ => false\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/core/src/main/scala/com/twitter/product_mixer/core/util/SortIndexBuilder.scala",
    "content": "package com.twitter.product_mixer.core.util\n\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.util.Time\n\nobject SortIndexBuilder {\n\n  /** the [[Time]] from a [[SnowflakeId]] */\n  def idToTime(id: Long): Time =\n    Time.fromMilliseconds(SnowflakeId.unixTimeMillisOrFloorFromId(id))\n\n  /** the first [[SnowflakeId]] possible for a given [[Time]]  */\n  def timeToId(time: Time): Long = SnowflakeId.firstIdFor(time)\n\n  /** the first [[SnowflakeId]] possible for a given unix epoch millis  */\n  def timeToId(timeMillis: Long): Long = SnowflakeId.firstIdFor(timeMillis)\n}\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/http_client/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"finatra-internal/mtls-http/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"util/util-core\",\n    ],\n    exports = [\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"finatra-internal/mtls-http/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"util/util-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/http_client/FinagleHttpClientBuilder.scala",
    "content": "package com.twitter.product_mixer.shared_library.http_client\n\nimport com.twitter.finagle.Http\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.Duration\n\nobject FinagleHttpClientBuilder {\n\n  /**\n   * Build a Finagle HTTP client with S2S Auth / Mutual TLS\n   *\n   * @param requestTimeout     HTTP client request timeout\n   * @param connectTimeout     HTTP client transport connect timeout\n   * @param acquisitionTimeout HTTP client session acquisition timeout\n   * @param serviceIdentifier  Service ID used to S2S Auth\n   * @param statsReceiver      Stats\n   *\n   * @return Finagle HTTP Client with S2S Auth / Mutual TLS\n   */\n  def buildFinagleHttpClientMutualTls(\n    requestTimeout: Duration,\n    connectTimeout: Duration,\n    acquisitionTimeout: Duration,\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): Http.Client =\n    buildFinagleHttpClient(\n      requestTimeout = requestTimeout,\n      connectTimeout = connectTimeout,\n      acquisitionTimeout = acquisitionTimeout,\n      statsReceiver = statsReceiver\n    ).withMutualTls(serviceIdentifier)\n\n  /**\n   * Build a Finagle HTTP client\n   *\n   * @param requestTimeout     HTTP client request timeout\n   * @param connectTimeout     HTTP client transport connect timeout\n   * @param acquisitionTimeout HTTP client session acquisition timeout\n   * @param statsReceiver      stats\n   *\n   * @return Finagle HTTP Client\n   */\n  def buildFinagleHttpClient(\n    requestTimeout: Duration,\n    connectTimeout: Duration,\n    acquisitionTimeout: Duration,\n    statsReceiver: StatsReceiver,\n  ): Http.Client =\n    Http.client\n      .withStatsReceiver(statsReceiver)\n      .withRequestTimeout(requestTimeout)\n      .withTransport.connectTimeout(connectTimeout)\n      .withSession.acquisitionTimeout(acquisitionTimeout)\n}\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/http_client/FinagleHttpClientWithProxyBuilder.scala",
    "content": "package com.twitter.product_mixer.shared_library.http_client\n\nimport com.twitter.finagle.Http\nimport com.twitter.finagle.Service\nimport com.twitter.finagle.client.Transporter\nimport com.twitter.finagle.http.ProxyCredentials\nimport com.twitter.finagle.http.Request\nimport com.twitter.finagle.http.Response\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.shared_library.http_client.FinagleHttpClientBuilder.buildFinagleHttpClient\nimport com.twitter.util.Duration\n\nobject FinagleHttpClientWithProxyBuilder {\n\n  /**\n   * Build a Finagle HTTP client with Egress Proxy support using Credentials\n   *\n   * @param twitterProxyHostPort    Twitter egress proxy host port\n   * @param remoteProxyHostPort     Remote proxy host port\n   * @param requestTimeout          HTTP client request timeout\n   * @param connectTimeout          HTTP client transport connect timeout\n   * @param acquisitionTimeout      HTTP client session acquisition timeout\n   * @param proxyCredentials        Proxy credentials\n   * @param statsReceiver           Stats\n   *\n   * @return Finagle HTTP client with Egress Proxy support using Credentials\n   */\n  def buildFinagleHttpClientWithCredentialProxy(\n    twitterProxyHostPort: HttpHostPort,\n    remoteProxyHostPort: HttpHostPort,\n    requestTimeout: Duration,\n    connectTimeout: Duration,\n    acquisitionTimeout: Duration,\n    proxyCredentials: ProxyCredentials,\n    statsReceiver: StatsReceiver,\n  ): Http.Client = {\n    val httpClient = buildFinagleHttpClient(\n      requestTimeout = requestTimeout,\n      connectTimeout = connectTimeout,\n      acquisitionTimeout = acquisitionTimeout,\n      statsReceiver = statsReceiver\n    )\n\n    httpClient.withTransport\n      .httpProxyTo(\n        host = remoteProxyHostPort.toString,\n        credentials = Transporter.Credentials(proxyCredentials.username, proxyCredentials.password))\n      .withTls(remoteProxyHostPort.host)\n  }\n\n  /**\n   * Build a Finagle HTTP client with Egress Proxy support\n   *\n   * @param twitterProxyHostPort   Twitter egress proxy host port\n   * @param remoteProxyHostPort    Remote proxy host port\n   * @param requestTimeout         HTTP client request timeout\n   * @param connectTimeout         HTTP client transport connect timeout\n   * @param acquisitionTimeout     HTTP client session acquisition timeout\n   * @param statsReceiver          Stats\n   *\n   * @return Finagle HTTP client with Egress Proxy support\n   */\n  def buildFinagleHttpClientWithProxy(\n    twitterProxyHostPort: HttpHostPort,\n    remoteProxyHostPort: HttpHostPort,\n    requestTimeout: Duration,\n    connectTimeout: Duration,\n    acquisitionTimeout: Duration,\n    statsReceiver: StatsReceiver,\n  ): Http.Client = {\n    val httpClient = buildFinagleHttpClient(\n      requestTimeout = requestTimeout,\n      connectTimeout = connectTimeout,\n      acquisitionTimeout = acquisitionTimeout,\n      statsReceiver = statsReceiver\n    )\n\n    httpClient.withTransport\n      .httpProxyTo(remoteProxyHostPort.toString)\n      .withTls(remoteProxyHostPort.host)\n  }\n\n  /**\n   * Build a Finagle HTTP service with Egress Proxy support\n   *\n   * @param finagleHttpClientWithProxy Finagle HTTP client from which to build the service\n   * @param twitterProxyHostPort       Twitter egress proxy host port\n   *\n   * @return Finagle HTTP service with Egress Proxy support\n   */\n  def buildFinagleHttpServiceWithProxy(\n    finagleHttpClientWithProxy: Http.Client,\n    twitterProxyHostPort: HttpHostPort\n  ): Service[Request, Response] = {\n    finagleHttpClientWithProxy.newService(twitterProxyHostPort.toString)\n  }\n}\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/http_client/HttpHostPort.scala",
    "content": "package com.twitter.product_mixer.shared_library.http_client\n\ncase class HttpHostPort(host: String, port: Int) {\n  override val toString: String = s\"$host:$port\"\n}\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/manhattan_client/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authorization\",\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"src/scala/com/twitter/storehaus_internal/manhattan/config\",\n        \"src/thrift/com/twitter/manhattan:v1-scala\",\n        \"storage/clients/manhattan\",\n        \"util/util-core\",\n    ],\n    exports = [\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authorization\",\n        \"finagle/finagle-core/src/main\",\n        \"src/scala/com/twitter/storehaus_internal/manhattan/config\",\n        \"storage/clients/manhattan\",\n        \"util/util-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/manhattan_client/ManhattanClientBuilder.scala",
    "content": "package com.twitter.product_mixer.shared_library.manhattan_client\n\nimport com.twitter.finagle.mtls.authentication.EmptyServiceIdentifier\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.ssl.OpportunisticTls\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.manhattan.v1.{thriftscala => mh}\nimport com.twitter.storage.client.manhattan.kv.Experiments\nimport com.twitter.storage.client.manhattan.kv.Experiments.Experiment\nimport com.twitter.storage.client.manhattan.kv.Guarantee\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClient\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpoint\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpointBuilder\nimport com.twitter.storage.client.manhattan.kv.NoMtlsParams\nimport com.twitter.storehaus_internal.manhattan.ManhattanCluster\nimport com.twitter.util.Duration\n\nobject ManhattanClientBuilder {\n\n  /**\n   * Build a ManhattanKVClient/Endpoint [[ManhattanKVEndpoint]] / [[ManhattanKVClient]]\n   *\n   * @param cluster Manhattan cluster\n   * @param appId Manhattan appid\n   * @param numTries Max number of times to try\n   * @param maxTimeout Max request timeout\n   * @param maxItemsPerRequest Max items per request\n   * @param guarantee Consistency guarantee\n   * @param serviceIdentifier Service ID used to S2S Auth\n   * @param statsReceiver Stats\n   * @param experiments MH client experiments to include\n   * @return ManhattanKVEndpoint\n   */\n  def buildManhattanEndpoint(\n    cluster: ManhattanCluster,\n    appId: String,\n    numTries: Int,\n    maxTimeout: Duration,\n    guarantee: Guarantee,\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver,\n    maxItemsPerRequest: Int = 100,\n    experiments: Seq[Experiment] = Seq(Experiments.ApertureLoadBalancer)\n  ): ManhattanKVEndpoint = {\n    val client = buildManhattanClient(\n      cluster,\n      appId,\n      serviceIdentifier,\n      experiments\n    )\n\n    ManhattanKVEndpointBuilder(client)\n      .defaultGuarantee(guarantee)\n      .defaultMaxTimeout(maxTimeout)\n      .maxRetryCount(numTries)\n      .maxItemsPerRequest(maxItemsPerRequest)\n      .statsReceiver(statsReceiver)\n      .build()\n  }\n\n  /**\n   *  Build a ManhattanKVClient\n   *\n   * @param cluster Manhattan cluster\n   * @param appId   Manhattan appid\n   * @param serviceIdentifier Service ID used to S2S Auth\n   * @param experiments MH client experiments to include\n   *\n   * @return ManhattanKVClient\n   */\n  def buildManhattanClient(\n    cluster: ManhattanCluster,\n    appId: String,\n    serviceIdentifier: ServiceIdentifier,\n    experiments: Seq[Experiment] = Seq(Experiments.ApertureLoadBalancer)\n  ): ManhattanKVClient = {\n    val mtlsParams = serviceIdentifier match {\n      case EmptyServiceIdentifier => NoMtlsParams\n      case serviceIdentifier =>\n        ManhattanKVClientMtlsParams(\n          serviceIdentifier = serviceIdentifier,\n          opportunisticTls = OpportunisticTls.Required)\n    }\n\n    val label = s\"manhattan/${cluster.prefix}\"\n\n    new ManhattanKVClient(\n      appId = appId,\n      dest = cluster.wilyName,\n      mtlsParams = mtlsParams,\n      label = label,\n      experiments = experiments\n    )\n  }\n\n  def buildManhattanV1FinagleClient(\n    cluster: ManhattanCluster,\n    serviceIdentifier: ServiceIdentifier,\n    experiments: Seq[Experiment] = Seq(Experiments.ApertureLoadBalancer)\n  ): mh.ManhattanCoordinator.MethodPerEndpoint = {\n    val mtlsParams = serviceIdentifier match {\n      case EmptyServiceIdentifier => NoMtlsParams\n      case serviceIdentifier =>\n        ManhattanKVClientMtlsParams(\n          serviceIdentifier = serviceIdentifier,\n          opportunisticTls = OpportunisticTls.Required)\n    }\n\n    val label = s\"manhattan/${cluster.prefix}\"\n\n    Experiments\n      .clientWithExperiments(experiments, mtlsParams)\n      .build[mh.ManhattanCoordinator.MethodPerEndpoint](cluster.wilyName, label)\n  }\n}\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/memcached_client/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-memcached/src/main/scala\",\n        \"finatra-internal/mtls-http/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"servo/repo/src/main/scala\",\n    ],\n    exports = [\n        \"finagle/finagle-memcached/src/main/scala\",\n        \"finatra-internal/mtls-http/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"servo/repo/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/memcached_client/MemcachedClientBuilder.scala",
    "content": "package com.twitter.product_mixer.shared_library.memcached_client\n\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.memcached.protocol.Command\nimport com.twitter.finagle.memcached.protocol.Response\nimport com.twitter.finagle.mtls.client.MtlsStackClient._\nimport com.twitter.finagle.service.RetryExceptionsFilter\nimport com.twitter.finagle.service.RetryPolicy\nimport com.twitter.finagle.service.TimeoutFilter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.finagle.GlobalRequestTimeoutException\nimport com.twitter.finagle.Memcached\nimport com.twitter.finagle.liveness.FailureAccrualFactory\nimport com.twitter.finagle.liveness.FailureAccrualPolicy\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.util.Duration\n\nobject MemcachedClientBuilder {\n\n  /**\n   * Build a Finagle Memcached [[Client]].\n   *\n   * @param destName             Destination as a Wily path e.g. \"/s/sample/sample\".\n   * @param numTries             Maximum number of times to try.\n   * @param requestTimeout       Thrift client timeout per request. The Finagle default\n   *                             is unbounded which is almost never optimal.\n   * @param globalTimeout        Thrift client total timeout. The Finagle default is\n   *                             unbounded which is almost never optimal.\n   * @param connectTimeout       Thrift client transport connect timeout. The Finagle\n   *                             default of one second is reasonable but we lower this\n   *                             to match acquisitionTimeout for consistency.\n   * @param acquisitionTimeout   Thrift client session acquisition timeout. The Finagle\n   *                             default is unbounded which is almost never optimal.\n   * @param serviceIdentifier    Service ID used to S2S Auth.\n   * @param statsReceiver        Stats.\n   * @param failureAccrualPolicy Policy to determine when to mark a cache server as dead.\n   *                             Memcached client will use default failure accrual policy\n   *                             if it is not set.\n   * @param keyHasher            Hash algorithm that hashes a key into a 32-bit or 64-bit\n   *                             number. Memcached client will use default hash algorithm\n   *                             if it is not set.\n   *\n   * @see [[https://confluence.twitter.biz/display/CACHE/Finagle-memcached+User+Guide user guide]]\n   * @return Finagle Memcached [[Client]]\n   */\n  def buildMemcachedClient(\n    destName: String,\n    numTries: Int,\n    requestTimeout: Duration,\n    globalTimeout: Duration,\n    connectTimeout: Duration,\n    acquisitionTimeout: Duration,\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver,\n    failureAccrualPolicy: Option[FailureAccrualPolicy] = None,\n    keyHasher: Option[KeyHasher] = None\n  ): Client = {\n    buildRawMemcachedClient(\n      numTries,\n      requestTimeout,\n      globalTimeout,\n      connectTimeout,\n      acquisitionTimeout,\n      serviceIdentifier,\n      statsReceiver,\n      failureAccrualPolicy,\n      keyHasher\n    ).newRichClient(destName)\n  }\n\n  def buildRawMemcachedClient(\n    numTries: Int,\n    requestTimeout: Duration,\n    globalTimeout: Duration,\n    connectTimeout: Duration,\n    acquisitionTimeout: Duration,\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver,\n    failureAccrualPolicy: Option[FailureAccrualPolicy] = None,\n    keyHasher: Option[KeyHasher] = None\n  ): Memcached.Client = {\n    val globalTimeoutFilter = new TimeoutFilter[Command, Response](\n      timeout = globalTimeout,\n      exception = new GlobalRequestTimeoutException(globalTimeout),\n      timer = DefaultTimer)\n    val retryFilter = new RetryExceptionsFilter[Command, Response](\n      RetryPolicy.tries(numTries),\n      DefaultTimer,\n      statsReceiver)\n\n    val client = Memcached.client.withTransport\n      .connectTimeout(connectTimeout)\n      .withMutualTls(serviceIdentifier)\n      .withSession\n      .acquisitionTimeout(acquisitionTimeout)\n      .withRequestTimeout(requestTimeout)\n      .withStatsReceiver(statsReceiver)\n      .filtered(globalTimeoutFilter.andThen(retryFilter))\n\n    (keyHasher, failureAccrualPolicy) match {\n      case (Some(hasher), Some(policy)) =>\n        client\n          .withKeyHasher(hasher)\n          .configured(FailureAccrualFactory.Param(() => policy))\n      case (Some(hasher), None) =>\n        client\n          .withKeyHasher(hasher)\n      case (None, Some(policy)) =>\n        client\n          .configured(FailureAccrualFactory.Param(() => policy))\n      case _ =>\n        client\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/observer/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"servo/util\",\n        \"stitch/stitch-core\",\n        \"util/util-core:util-core-util\",\n        \"util/util-stats\",\n    ],\n    exports = [\n        \"servo/util\",\n        \"stitch/stitch-core\",\n        \"util/util-core:util-core-util\",\n        \"util/util-stats\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/observer/Observer.scala",
    "content": "package com.twitter.product_mixer.shared_library.observer\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.RollupStatsReceiver\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.servo.util.CancelledExceptionExtractor\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Throwables\nimport com.twitter.util.Try\n\n/**\n * Helper functions to observe requests, success, failures, cancellations, exceptions, and latency.\n * Supports native functions and asynchronous operations.\n */\nobject Observer {\n  val Requests = \"requests\"\n  val Success = \"success\"\n  val Failures = \"failures\"\n  val Cancelled = \"cancelled\"\n  val Latency = \"latency_ms\"\n\n  /**\n   * Helper function to observe a stitch\n   *\n   * @see [[StitchObserver]]\n   */\n  def stitch[T](statsReceiver: StatsReceiver, scopes: String*): StitchObserver[T] =\n    new StitchObserver[T](statsReceiver, scopes)\n\n  /**\n   * Helper function to observe an arrow\n   *\n   * @see [[ArrowObserver]]\n   */\n  def arrow[In, Out](statsReceiver: StatsReceiver, scopes: String*): ArrowObserver[In, Out] =\n    new ArrowObserver[In, Out](statsReceiver, scopes)\n\n  /**\n   * Helper function to observe a future\n   *\n   * @see [[FutureObserver]]\n   */\n  def future[T](statsReceiver: StatsReceiver, scopes: String*): FutureObserver[T] =\n    new FutureObserver[T](statsReceiver, scopes)\n\n  /**\n   * Helper function to observe a function\n   *\n   * @see [[FunctionObserver]]\n   */\n  def function[T](statsReceiver: StatsReceiver, scopes: String*): FunctionObserver[T] =\n    new FunctionObserver[T](statsReceiver, scopes)\n\n  /**\n   * [[StitchObserver]] can record latency stats, success counters, and\n   * detailed failure stats for the results of a Stitch computation.\n   */\n  class StitchObserver[T](\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends Observer[T] {\n\n    /**\n     * Record stats for the provided Stitch.\n     * The result of the computation is passed through.\n     *\n     * @note the provided Stitch must contain the parts that need to be timed.\n     *       Using this on just the result of the computation the latency stat\n     *       will be incorrect.\n     */\n    def apply(stitch: => Stitch[T]): Stitch[T] =\n      Stitch.time(stitch).map(observe.tupled).lowerFromTry\n  }\n\n  /**\n   * [[ArrowObserver]] can record the latency stats, success counters, and\n   * detailed failure stats for the result of an Arrow computation.\n   * The result of the computation is passed through.\n   */\n  class ArrowObserver[In, Out](\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends Observer[Out] {\n\n    /**\n     * Returns a new Arrow that records stats when it's run.\n     * The result of the Arrow is passed through.\n     *\n     * @note the provided Arrow must contain the parts that need to be timed.\n     *       Using this on just the result of the computation the latency stat\n     *       will be incorrect.\n     */\n    def apply(arrow: Arrow[In, Out]): Arrow[In, Out] =\n      Arrow.time(arrow).map(observe.tupled).lowerFromTry\n  }\n\n  /**\n   * [[FutureObserver]] can record latency stats, success counters, and\n   * detailed failure stats for the results of a Future computation.\n   */\n  class FutureObserver[T](\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends Observer[T] {\n\n    /**\n     * Record stats for the provided Future.\n     * The result of the computation is passed through.\n     *\n     * @note the provided Future must contain the parts that need to be timed.\n     *       Using this on just the result of the computation the latency stat\n     *       will be incorrect.\n     */\n    def apply(future: => Future[T]): Future[T] =\n      Stat\n        .timeFuture(latencyStat)(future)\n        .onSuccess(observeSuccess)\n        .onFailure(observeFailure)\n  }\n\n  /**\n   * [[FunctionObserver]] can record latency stats, success counters, and\n   * detailed failure stats for the results of a computation computation.\n   */\n  class FunctionObserver[T](\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends Observer[T] {\n\n    /**\n     * Record stats for the provided computation.\n     * The result of the computation is passed through.\n     *\n     * @note the provided computation must contain the parts that need to be timed.\n     *       Using this on just the result of the computation the latency stat\n     *       will be incorrect.\n     */\n    def apply(f: => T): T = {\n      Try(Stat.time(latencyStat)(f))\n        .onSuccess(observeSuccess)\n        .onFailure(observeFailure)\n        .apply()\n    }\n  }\n\n  /** [[Observer]] provides methods for recording latency, success, and failure stats */\n  trait Observer[T] {\n    protected val statsReceiver: StatsReceiver\n\n    /** Scopes that prefix all stats */\n    protected val scopes: Seq[String]\n\n    private val rollupStatsReceiver = new RollupStatsReceiver(statsReceiver.scope(scopes: _*))\n    private val requestsCounter: Counter = statsReceiver.counter(scopes :+ Requests: _*)\n    private val successCounter: Counter = statsReceiver.counter(scopes :+ Success: _*)\n\n    // create the stats so their metrics paths are always present but\n    // defer to the [[RollupStatsReceiver]] to increment these stats\n    rollupStatsReceiver.counter(Failures)\n    rollupStatsReceiver.counter(Cancelled)\n\n    /** Serialize a throwable and it's causes into a seq of Strings for scoping metrics */\n    protected def serializeThrowable(throwable: Throwable): Seq[String] =\n      Throwables.mkString(throwable)\n\n    /** Used to record latency in milliseconds */\n    protected val latencyStat: Stat = statsReceiver.stat(scopes :+ Latency: _*)\n\n    /** Records the latency from a [[Duration]] */\n    protected val observeLatency: Duration => Unit = { latency =>\n      latencyStat.add(latency.inMilliseconds)\n    }\n\n    /** Records successes */\n    protected val observeSuccess: T => Unit = { _ =>\n      requestsCounter.incr()\n      successCounter.incr()\n    }\n\n    /** Records failures and failure details */\n    protected val observeFailure: Throwable => Unit = {\n      case CancelledExceptionExtractor(throwable) =>\n        requestsCounter.incr()\n        rollupStatsReceiver.counter(Cancelled +: serializeThrowable(throwable): _*).incr()\n      case throwable =>\n        requestsCounter.incr()\n        rollupStatsReceiver.counter(Failures +: serializeThrowable(throwable): _*).incr()\n    }\n\n    /** Records the latency, successes, and failures */\n    protected val observe: (Try[T], Duration) => Try[T] =\n      (response: Try[T], runDuration: Duration) => {\n        observeLatency(runDuration)\n        response\n          .onSuccess(observeSuccess)\n          .onFailure(observeFailure)\n      }\n  }\n}\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/observer/ResultsObserver.scala",
    "content": "package com.twitter.product_mixer.shared_library.observer\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.shared_library.observer.Observer.ArrowObserver\nimport com.twitter.product_mixer.shared_library.observer.Observer.FunctionObserver\nimport com.twitter.product_mixer.shared_library.observer.Observer.FutureObserver\nimport com.twitter.product_mixer.shared_library.observer.Observer.Observer\nimport com.twitter.product_mixer.shared_library.observer.Observer.StitchObserver\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport com.twitter.util.Try\n\n/**\n * Helper functions to observe requests, successes, failures, cancellations, exceptions, latency,\n * and result counts. Supports native functions and asynchronous operations.\n */\nobject ResultsObserver {\n  val Total = \"total\"\n  val Found = \"found\"\n  val NotFound = \"not_found\"\n\n  /**\n   * Helper function to observe a stitch and result counts\n   *\n   * @see [[StitchResultsObserver]]\n   */\n  def stitchResults[T](\n    size: T => Int,\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): StitchResultsObserver[T] = {\n    new StitchResultsObserver[T](size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function to observe a stitch and traversable (e.g. Seq, Set) result counts\n   *\n   * @see [[StitchResultsObserver]]\n   */\n  def stitchResults[T <: TraversableOnce[_]](\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): StitchResultsObserver[T] = {\n    new StitchResultsObserver[T](_.size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function to observe an arrow and result counts\n   *\n   * @see [[ArrowResultsObserver]]\n   */\n  def arrowResults[In, Out](\n    size: Out => Int,\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): ArrowResultsObserver[In, Out] = {\n    new ArrowResultsObserver[In, Out](size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function to observe an arrow and traversable (e.g. Seq, Set) result counts\n   *\n   * @see [[ArrowResultsObserver]]\n   */\n  def arrowResults[In, Out <: TraversableOnce[_]](\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): ArrowResultsObserver[In, Out] = {\n    new ArrowResultsObserver[In, Out](_.size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function to observe an arrow and result counts\n   *\n   * @see [[TransformingArrowResultsObserver]]\n   */\n  def transformingArrowResults[In, Out, Transformed](\n    transformer: Out => Try[Transformed],\n    size: Transformed => Int,\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): TransformingArrowResultsObserver[In, Out, Transformed] = {\n    new TransformingArrowResultsObserver[In, Out, Transformed](\n      transformer,\n      size,\n      statsReceiver,\n      scopes)\n  }\n\n  /**\n   * Helper function to observe an arrow and traversable (e.g. Seq, Set) result counts\n   *\n   * @see [[TransformingArrowResultsObserver]]\n   */\n  def transformingArrowResults[In, Out, Transformed <: TraversableOnce[_]](\n    transformer: Out => Try[Transformed],\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): TransformingArrowResultsObserver[In, Out, Transformed] = {\n    new TransformingArrowResultsObserver[In, Out, Transformed](\n      transformer,\n      _.size,\n      statsReceiver,\n      scopes)\n  }\n\n  /**\n   * Helper function to observe a future and result counts\n   *\n   * @see [[FutureResultsObserver]]\n   */\n  def futureResults[T](\n    size: T => Int,\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): FutureResultsObserver[T] = {\n    new FutureResultsObserver[T](size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function to observe a future and traversable (e.g. Seq, Set) result counts\n   *\n   * @see [[FutureResultsObserver]]\n   */\n  def futureResults[T <: TraversableOnce[_]](\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): FutureResultsObserver[T] = {\n    new FutureResultsObserver[T](_.size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function to observe a function and result counts\n   *\n   * @see [[FunctionResultsObserver]]\n   */\n  def functionResults[T](\n    size: T => Int,\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): FunctionResultsObserver[T] = {\n    new FunctionResultsObserver[T](size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function to observe a function and traversable (e.g. Seq, Set) result counts\n   *\n   * @see [[FunctionResultsObserver]]\n   */\n  def functionResults[T <: TraversableOnce[_]](\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): FunctionResultsObserver[T] = {\n    new FunctionResultsObserver[T](_.size, statsReceiver, scopes)\n  }\n\n  /** [[StitchObserver]] that also records result size */\n  class StitchResultsObserver[T](\n    override val size: T => Int,\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends StitchObserver[T](statsReceiver, scopes)\n      with ResultsObserver[T] {\n\n    override def apply(stitch: => Stitch[T]): Stitch[T] =\n      super\n        .apply(stitch)\n        .onSuccess(observeResults)\n  }\n\n  /** [[ArrowObserver]] that also records result size */\n  class ArrowResultsObserver[In, Out](\n    override val size: Out => Int,\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends ArrowObserver[In, Out](statsReceiver, scopes)\n      with ResultsObserver[Out] {\n\n    override def apply(arrow: Arrow[In, Out]): Arrow[In, Out] =\n      super\n        .apply(arrow)\n        .onSuccess(observeResults)\n  }\n\n  /**\n   * [[TransformingArrowResultsObserver]] functions like an [[ArrowObserver]] except\n   * that it transforms the result using [[transformer]] before recording stats.\n   *\n   * The original non-transformed result is then returned.\n   */\n  class TransformingArrowResultsObserver[In, Out, Transformed](\n    val transformer: Out => Try[Transformed],\n    override val size: Transformed => Int,\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends Observer[Transformed]\n      with ResultsObserver[Transformed] {\n\n    /**\n     * Returns a new Arrow that records stats on the result after applying [[transformer]] when it's run.\n     * The original, non-transformed, result of the Arrow is passed through.\n     *\n     * @note the provided Arrow must contain the parts that need to be timed.\n     *       Using this on just the result of the computation the latency stat\n     *       will be incorrect.\n     */\n    def apply(arrow: Arrow[In, Out]): Arrow[In, Out] = {\n      Arrow\n        .time(arrow)\n        .map {\n          case (response, stitchRunDuration) =>\n            observe(response.flatMap(transformer), stitchRunDuration)\n              .onSuccess(observeResults)\n            response\n        }.lowerFromTry\n    }\n  }\n\n  /** [[FutureObserver]] that also records result size */\n  class FutureResultsObserver[T](\n    override val size: T => Int,\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends FutureObserver[T](statsReceiver, scopes)\n      with ResultsObserver[T] {\n\n    override def apply(future: => Future[T]): Future[T] =\n      super\n        .apply(future)\n        .onSuccess(observeResults)\n  }\n\n  /** [[FunctionObserver]] that also records result size */\n  class FunctionResultsObserver[T](\n    override val size: T => Int,\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends FunctionObserver[T](statsReceiver, scopes)\n      with ResultsObserver[T] {\n\n    override def apply(f: => T): T = observeResults(super.apply(f))\n  }\n\n  /** [[ResultsObserver]] provides methods for recording stats for the result size */\n  trait ResultsObserver[T] {\n    protected val statsReceiver: StatsReceiver\n\n    /** Scopes that prefix all stats */\n    protected val scopes: Seq[String]\n\n    protected val totalCounter: Counter = statsReceiver.counter(scopes :+ Total: _*)\n    protected val foundCounter: Counter = statsReceiver.counter(scopes :+ Found: _*)\n    protected val notFoundCounter: Counter = statsReceiver.counter(scopes :+ NotFound: _*)\n\n    /** given a [[T]] returns it's size. */\n    protected val size: T => Int\n\n    /** Records the size of the `results` using [[size]] and return the original value. */\n    protected def observeResults(results: T): T = {\n      val resultsSize = size(results)\n      observeResultsWithSize(results, resultsSize)\n    }\n\n    /**\n     * Records the `resultsSize` and returns the `results`\n     *\n     * This is useful if the size is already available and is expensive to calculate.\n     */\n    protected def observeResultsWithSize(results: T, resultsSize: Int): T = {\n      if (resultsSize > 0) {\n        totalCounter.incr(resultsSize)\n        foundCounter.incr()\n      } else {\n        notFoundCounter.incr()\n      }\n      results\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/observer/ResultsStatsObserver.scala",
    "content": "package com.twitter.product_mixer.shared_library.observer\n\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.product_mixer.shared_library.observer.Observer.ArrowObserver\nimport com.twitter.product_mixer.shared_library.observer.Observer.FunctionObserver\nimport com.twitter.product_mixer.shared_library.observer.Observer.FutureObserver\nimport com.twitter.product_mixer.shared_library.observer.Observer.Observer\nimport com.twitter.product_mixer.shared_library.observer.Observer.StitchObserver\nimport com.twitter.product_mixer.shared_library.observer.ResultsObserver.ResultsObserver\nimport com.twitter.stitch.Arrow\nimport com.twitter.stitch.Stitch\nimport com.twitter.util.Future\nimport com.twitter.util.Try\n\n/**\n * Helper functions to observe requests, successes, failures, cancellations, exceptions, latency,\n * and result counts and time-series stats. Supports native functions and asynchronous operations.\n *\n * Note that since time-series stats are expensive to compute (relative to counters), prefer\n * [[ResultsObserver]] unless a time-series stat is needed.\n */\nobject ResultsStatsObserver {\n  val Size = \"size\"\n\n  /**\n   * Helper function to observe a stitch and result counts and time-series stats\n   */\n  def stitchResultsStats[T](\n    size: T => Int,\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): StitchResultsStatsObserver[T] = {\n    new StitchResultsStatsObserver[T](size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function to observe a stitch and traversable (e.g. Seq, Set) result counts and\n   * time-series stats\n   */\n  def stitchResultsStats[T <: TraversableOnce[_]](\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): StitchResultsStatsObserver[T] = {\n    new StitchResultsStatsObserver[T](_.size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function to observe an arrow and result counts and time-series stats\n   */\n  def arrowResultsStats[T, U](\n    size: U => Int,\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): ArrowResultsStatsObserver[T, U] = {\n    new ArrowResultsStatsObserver[T, U](size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function to observe an arrow and traversable (e.g. Seq, Set) result counts and\n   * * time-series stats\n   */\n  def arrowResultsStats[T, U <: TraversableOnce[_]](\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): ArrowResultsStatsObserver[T, U] = {\n    new ArrowResultsStatsObserver[T, U](_.size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function to observe an arrow and result counts\n   *\n   * @see [[TransformingArrowResultsStatsObserver]]\n   */\n  def transformingArrowResultsStats[In, Out, Transformed](\n    transformer: Out => Try[Transformed],\n    size: Transformed => Int,\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): TransformingArrowResultsStatsObserver[In, Out, Transformed] = {\n    new TransformingArrowResultsStatsObserver[In, Out, Transformed](\n      transformer,\n      size,\n      statsReceiver,\n      scopes)\n  }\n\n  /**\n   * Helper function to observe an arrow and traversable (e.g. Seq, Set) result counts\n   *\n   * @see [[TransformingArrowResultsStatsObserver]]\n   */\n  def transformingArrowResultsStats[In, Out, Transformed <: TraversableOnce[_]](\n    transformer: Out => Try[Transformed],\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): TransformingArrowResultsStatsObserver[In, Out, Transformed] = {\n    new TransformingArrowResultsStatsObserver[In, Out, Transformed](\n      transformer,\n      _.size,\n      statsReceiver,\n      scopes)\n  }\n\n  /**\n   * Helper function to observe a future and result counts and time-series stats\n   */\n  def futureResultsStats[T](\n    size: T => Int,\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): FutureResultsStatsObserver[T] = {\n    new FutureResultsStatsObserver[T](size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function to observe a future and traversable (e.g. Seq, Set) result counts and\n   * time-series stats\n   */\n  def futureResultsStats[T <: TraversableOnce[_]](\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): FutureResultsStatsObserver[T] = {\n    new FutureResultsStatsObserver[T](_.size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function observe a function and result counts and time-series stats\n   */\n  def functionResultsStats[T](\n    size: T => Int,\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): FunctionResultsStatsObserver[T] = {\n    new FunctionResultsStatsObserver[T](size, statsReceiver, scopes)\n  }\n\n  /**\n   * Helper function observe a function and traversable (e.g. Seq, Set) result counts and\n   * time-series stats\n   */\n  def functionResultsStats[T <: TraversableOnce[_]](\n    statsReceiver: StatsReceiver,\n    scopes: String*\n  ): FunctionResultsStatsObserver[T] = {\n    new FunctionResultsStatsObserver[T](_.size, statsReceiver, scopes)\n  }\n\n  class StitchResultsStatsObserver[T](\n    override val size: T => Int,\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends StitchObserver[T](statsReceiver, scopes)\n      with ResultsStatsObserver[T] {\n\n    override def apply(stitch: => Stitch[T]): Stitch[T] =\n      super\n        .apply(stitch)\n        .onSuccess(observeResults)\n  }\n\n  class ArrowResultsStatsObserver[T, U](\n    override val size: U => Int,\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends ArrowObserver[T, U](statsReceiver, scopes)\n      with ResultsStatsObserver[U] {\n\n    override def apply(arrow: Arrow[T, U]): Arrow[T, U] =\n      super\n        .apply(arrow)\n        .onSuccess(observeResults)\n  }\n\n  /**\n   * [[TransformingArrowResultsStatsObserver]] functions like an [[ArrowObserver]] except\n   * that it transforms the result using [[transformer]] before recording stats.\n   *\n   * The original non-transformed result is then returned.\n   */\n  class TransformingArrowResultsStatsObserver[In, Out, Transformed](\n    val transformer: Out => Try[Transformed],\n    override val size: Transformed => Int,\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends Observer[Transformed]\n      with ResultsStatsObserver[Transformed] {\n\n    /**\n     * Returns a new Arrow that records stats on the result after applying [[transformer]] when it's run.\n     * The original, non-transformed, result of the Arrow is passed through.\n     *\n     * @note the provided Arrow must contain the parts that need to be timed.\n     *       Using this on just the result of the computation the latency stat\n     *       will be incorrect.\n     */\n    def apply(arrow: Arrow[In, Out]): Arrow[In, Out] = {\n      Arrow\n        .time(arrow)\n        .map {\n          case (response, stitchRunDuration) =>\n            observe(response.flatMap(transformer), stitchRunDuration)\n              .onSuccess(observeResults)\n            response\n        }.lowerFromTry\n    }\n  }\n\n  class FutureResultsStatsObserver[T](\n    override val size: T => Int,\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends FutureObserver[T](statsReceiver, scopes)\n      with ResultsStatsObserver[T] {\n\n    override def apply(future: => Future[T]): Future[T] =\n      super\n        .apply(future)\n        .onSuccess(observeResults)\n  }\n\n  class FunctionResultsStatsObserver[T](\n    override val size: T => Int,\n    override val statsReceiver: StatsReceiver,\n    override val scopes: Seq[String])\n      extends FunctionObserver[T](statsReceiver, scopes)\n      with ResultsStatsObserver[T] {\n\n    override def apply(f: => T): T = {\n      observeResults(super.apply(f))\n    }\n  }\n\n  trait ResultsStatsObserver[T] extends ResultsObserver[T] {\n    private val sizeStat: Stat = statsReceiver.stat(scopes :+ Size: _*)\n\n    protected override def observeResults(results: T): T = {\n      val resultsSize = size(results)\n      sizeStat.add(resultsSize)\n      observeResultsWithSize(results, resultsSize)\n    }\n  }\n}\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/thrift_client/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"finatra-internal/mtls-http/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"util/util-core\",\n    ],\n    exports = [\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"finatra-internal/mtls-http/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"util/util-core\",\n    ],\n)\n"
  },
  {
    "path": "product-mixer/shared-library/src/main/scala/com/twitter/product_mixer/shared_library/thrift_client/FinagleThriftClientBuilder.scala",
    "content": "package com.twitter.product_mixer.shared_library.thrift_client\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.thrift.service.Filterable\nimport com.twitter.finagle.thrift.service.MethodPerEndpointBuilder\nimport com.twitter.finagle.thrift.service.ServicePerEndpointBuilder\nimport com.twitter.finagle.thriftmux.MethodBuilder\nimport com.twitter.util.Duration\nimport org.apache.thrift.protocol.TProtocolFactory\n\nsealed trait Idempotency\ncase object NonIdempotent extends Idempotency\ncase class Idempotent(maxExtraLoadPercent: Double) extends Idempotency\n\nobject FinagleThriftClientBuilder {\n\n  /**\n   * Library to build a Finagle Thrift method per endpoint client is a less error-prone way when\n   * compared to the builders in Finagle. This is achieved by requiring values for fields that should\n   * always be set in practice. For example, request timeouts in Finagle are unbounded when not\n   * explicitly set, and this method requires that timeout durations are passed into the method and\n   * set on the Finagle builder.\n   *\n   * Usage of\n   * [[com.twitter.inject.thrift.modules.ThriftMethodBuilderClientModule]] is almost always preferred,\n   * and the Product Mixer component library [[com.twitter.product_mixer.component_library.module]]\n   * package contains numerous examples. However, if multiple versions of a client are needed e.g.\n   * for different timeout settings, this method is useful to easily provide multiple variants.\n   *\n   * @example\n   * {{{\n   *   final val SampleServiceClientName = \"SampleServiceClient\"\n   *   @Provides\n   *   @Singleton\n   *   @Named(SampleServiceClientName)\n   *   def provideSampleServiceClient(\n   *     serviceIdentifier: ServiceIdentifier,\n   *     clientId: ClientId,\n   *     statsReceiver: StatsReceiver,\n   *   ): SampleService.MethodPerEndpoint =\n   *     buildFinagleMethodPerEndpoint[SampleService.ServicePerEndpoint, SampleService.MethodPerEndpoint](\n   *       serviceIdentifier = serviceIdentifier,\n   *       clientId = clientId,\n   *       dest = \"/s/sample/sample\",\n   *       label = \"sample\",\n   *       statsReceiver = statsReceiver,\n   *       idempotency = Idempotent(5.percent),\n   *       timeoutPerRequest = 200.milliseconds,\n   *       timeoutTotal = 400.milliseconds\n   *     )\n   * }}}\n   * @param serviceIdentifier         Service ID used to S2S Auth\n   * @param clientId                  Client ID\n   * @param dest                      Destination as a Wily path e.g. \"/s/sample/sample\"\n   * @param label                     Label of the client\n   * @param statsReceiver             Stats\n   * @param idempotency               Idempotency semantics of the client\n   * @param timeoutPerRequest         Thrift client timeout per request. The Finagle default is\n   *                                  unbounded which is almost never optimal.\n   * @param timeoutTotal              Thrift client total timeout. The Finagle default is\n   *                                  unbounded which is almost never optimal.\n   *                                  If the client is set as idempotent, which adds a\n   *                                  [[com.twitter.finagle.client.BackupRequestFilter]],\n   *                                  be sure to leave enough room for the backup request. A\n   *                                  reasonable (albeit usually too large) starting point is to\n   *                                  make the total timeout 2x relative to the per request timeout.\n   *                                  If the client is set as non-idempotent, the total timeout and\n   *                                  the per request timeout should be the same, as there will be\n   *                                  no backup requests.\n   * @param connectTimeout            Thrift client transport connect timeout. The Finagle default\n   *                                  of one second is reasonable but we lower this to match\n   *                                  acquisitionTimeout for consistency.\n   * @param acquisitionTimeout        Thrift client session acquisition timeout. The Finagle default\n   *                                  is unbounded which is almost never optimal.\n   * @param protocolFactoryOverride   Override the default protocol factory\n   *                                  e.g. [[org.apache.thrift.protocol.TCompactProtocol.Factory]]\n   * @param servicePerEndpointBuilder implicit service per endpoint builder\n   * @param methodPerEndpointBuilder  implicit method per endpoint builder\n   *\n   * @see [[https://twitter.github.io/finagle/guide/MethodBuilder.html user guide]]\n   * @see [[https://twitter.github.io/finagle/guide/MethodBuilder.html#idempotency user guide]]\n   * @return method per endpoint Finagle Thrift Client\n   */\n  def buildFinagleMethodPerEndpoint[\n    ServicePerEndpoint <: Filterable[ServicePerEndpoint],\n    MethodPerEndpoint\n  ](\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    dest: String,\n    label: String,\n    statsReceiver: StatsReceiver,\n    idempotency: Idempotency,\n    timeoutPerRequest: Duration,\n    timeoutTotal: Duration,\n    connectTimeout: Duration = 500.milliseconds,\n    acquisitionTimeout: Duration = 500.milliseconds,\n    protocolFactoryOverride: Option[TProtocolFactory] = None,\n  )(\n    implicit servicePerEndpointBuilder: ServicePerEndpointBuilder[ServicePerEndpoint],\n    methodPerEndpointBuilder: MethodPerEndpointBuilder[ServicePerEndpoint, MethodPerEndpoint]\n  ): MethodPerEndpoint = {\n    val service: ServicePerEndpoint = buildFinagleServicePerEndpoint(\n      serviceIdentifier = serviceIdentifier,\n      clientId = clientId,\n      dest = dest,\n      label = label,\n      statsReceiver = statsReceiver,\n      idempotency = idempotency,\n      timeoutPerRequest = timeoutPerRequest,\n      timeoutTotal = timeoutTotal,\n      connectTimeout = connectTimeout,\n      acquisitionTimeout = acquisitionTimeout,\n      protocolFactoryOverride = protocolFactoryOverride\n    )\n\n    ThriftMux.Client.methodPerEndpoint(service)\n  }\n\n  /**\n   * Build a Finagle Thrift service per endpoint client.\n   *\n   * @note [[buildFinagleMethodPerEndpoint]] should be preferred over the service per endpoint variant\n   *\n   * @param serviceIdentifier       Service ID used to S2S Auth\n   * @param clientId                Client ID\n   * @param dest                    Destination as a Wily path e.g. \"/s/sample/sample\"\n   * @param label                   Label of the client\n   * @param statsReceiver           Stats\n   * @param idempotency             Idempotency semantics of the client\n   * @param timeoutPerRequest       Thrift client timeout per request. The Finagle default is\n   *                                unbounded which is almost never optimal.\n   * @param timeoutTotal            Thrift client total timeout. The Finagle default is\n   *                                unbounded which is almost never optimal.\n   *                                If the client is set as idempotent, which adds a\n   *                                [[com.twitter.finagle.client.BackupRequestFilter]],\n   *                                be sure to leave enough room for the backup request. A\n   *                                reasonable (albeit usually too large) starting point is to\n   *                                make the total timeout 2x relative to the per request timeout.\n   *                                If the client is set as non-idempotent, the total timeout and\n   *                                the per request timeout should be the same, as there will be\n   *                                no backup requests.\n   * @param connectTimeout          Thrift client transport connect timeout. The Finagle default\n   *                                of one second is reasonable but we lower this to match\n   *                                acquisitionTimeout for consistency.\n   * @param acquisitionTimeout      Thrift client session acquisition timeout. The Finagle default\n   *                                is unbounded which is almost never optimal.\n   * @param protocolFactoryOverride Override the default protocol factory\n   *                                e.g. [[org.apache.thrift.protocol.TCompactProtocol.Factory]]\n   *\n   * @return service per endpoint Finagle Thrift Client\n   */\n  def buildFinagleServicePerEndpoint[ServicePerEndpoint <: Filterable[ServicePerEndpoint]](\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    dest: String,\n    label: String,\n    statsReceiver: StatsReceiver,\n    idempotency: Idempotency,\n    timeoutPerRequest: Duration,\n    timeoutTotal: Duration,\n    connectTimeout: Duration = 500.milliseconds,\n    acquisitionTimeout: Duration = 500.milliseconds,\n    protocolFactoryOverride: Option[TProtocolFactory] = None,\n  )(\n    implicit servicePerEndpointBuilder: ServicePerEndpointBuilder[ServicePerEndpoint]\n  ): ServicePerEndpoint = {\n    val thriftMux: ThriftMux.Client = ThriftMux.client\n      .withMutualTls(serviceIdentifier)\n      .withClientId(clientId)\n      .withLabel(label)\n      .withStatsReceiver(statsReceiver)\n      .withTransport.connectTimeout(connectTimeout)\n      .withSession.acquisitionTimeout(acquisitionTimeout)\n\n    val protocolThriftMux: ThriftMux.Client = protocolFactoryOverride\n      .map { protocolFactory =>\n        thriftMux.withProtocolFactory(protocolFactory)\n      }.getOrElse(thriftMux)\n\n    val methodBuilder: MethodBuilder = protocolThriftMux\n      .methodBuilder(dest)\n      .withTimeoutPerRequest(timeoutPerRequest)\n      .withTimeoutTotal(timeoutTotal)\n\n    val idempotencyMethodBuilder: MethodBuilder = idempotency match {\n      case NonIdempotent => methodBuilder.nonIdempotent\n      case Idempotent(maxExtraLoad) => methodBuilder.idempotent(maxExtraLoad = maxExtraLoad)\n    }\n\n    idempotencyMethodBuilder.servicePerEndpoint[ServicePerEndpoint]\n  }\n}\n"
  },
  {
    "path": "pushservice/BUILD.bazel",
    "content": "alias(\n    name = \"frigate-pushservice\",\n    target = \":frigate-pushservice_lib\",\n)\n\ntarget(\n    name = \"frigate-pushservice_lib\",\n    dependencies = [\n        \"frigate/frigate-pushservice-opensource/src/main/scala/com/twitter/frigate/pushservice\",\n    ],\n)\n\njvm_binary(\n    name = \"bin\",\n    basename = \"frigate-pushservice\",\n    main = \"com.twitter.frigate.pushservice.PushServiceMain\",\n    runtime_platform = \"java11\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/ch/qos/logback:logback-classic\",\n        \"finatra/inject/inject-logback/src/main/scala\",\n        \"frigate/frigate-pushservice-opensource/src/main/scala/com/twitter/frigate/pushservice\",\n        \"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback\",\n        \"twitter-server/logback-classic/src/main/scala\",\n    ],\n    excludes = [\n        exclude(\"com.twitter.translations\", \"translations-twitter\"),\n        exclude(\"org.apache.hadoop\", \"hadoop-aws\"),\n        exclude(\"org.tensorflow\"),\n        scala_exclude(\"com.twitter\", \"ckoia-scala\"),\n    ],\n)\n\njvm_app(\n    name = \"bundle\",\n    basename = \"frigate-pushservice-package-dist\",\n    archive = \"zip\",\n    binary = \":bin\",\n    tags = [\"bazel-compatible\"],\n)\n\npython3_library(\n    name = \"mr_model_constants\",\n    sources = [\n        \"config/deepbird/constants.py\",\n    ],\n    tags = [\"bazel-compatible\"],\n)\n"
  },
  {
    "path": "pushservice/README.md",
    "content": "# Pushservice\n\nPushservice is the main push recommendation service at Twitter used to generate recommendation-based notifications for users. It currently powers two functionalities:\n\n- RefreshForPushHandler: This handler determines whether to send a recommendation push to a user based on their ID. It generates the best push recommendation item and coordinates with downstream services to deliver it\n- SendHandler: This handler determines and manage whether send the push to users based on the given target user details and the provided push recommendation item\n\n## Overview\n\n### RefreshForPushHandler\n\nRefreshForPushHandler follows these steps:\n\n- Building Target and checking eligibility\n    - Builds a target user object based on the given user ID\n    - Performs target-level filterings to determine if the target is eligible for a recommendation push\n- Fetch Candidates\n    - Retrieves a list of potential candidates for the push by querying various candidate sources using the target\n- Candidate Hydration\n    - Hydrates the candidate details with batch calls to different downstream services\n- Pre-rank Filtering, also called Light Filtering\n    - Filters the hydrated candidates with lightweight RPC calls\n- Rank\n    - Perform feature hydration for candidates and target user\n    - Performs light ranking on candidates\n    - Performs heavy ranking on candidates\n- Take Step, also called Heavy Filtering\n    - Takes the top-ranked candidates one by one and applies heavy filtering until one candidate passes all filter steps\n- Send\n    - Calls the appropriate downstream service to deliver the eligible candidate as a push and in-app notification to the target user\n\n### SendHandler\n\nSendHandler follows these steps:\n\n- Building Target\n    - Builds a target user object based on the given user ID\n- Candidate Hydration\n    - Hydrates the candidate details with batch calls to different downstream services\n- Feature Hydration\n    - Perform feature hydration for candidates and target user\n- Take Step, also called Heavy Filtering\n    - Perform filterings and validation checking for the given candidate\n- Send\n    - Calls the appropriate downstream service to deliver the given candidate as a push and/or in-app notification to the target user"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/BUILD",
    "content": "python37_binary(\n    name = \"update_warm_start_checkpoint\",\n    source = \"update_warm_start_checkpoint.py\",\n    tags = [\"no-mypy\"],\n    dependencies = [\n        \":deep_norm_lib\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:update_warm_start_checkpoint\",\n    ],\n)\n\npython3_library(\n    name = \"params_lib\",\n    sources = [\"params.py\"],\n    tags = [\"no-mypy\"],\n    dependencies = [\n        \"3rdparty/python/pydantic:default\",\n        \"src/python/twitter/deepbird/projects/magic_recs/v11/lib:params_lib\",\n    ],\n)\n\npython3_library(\n    name = \"features_lib\",\n    sources = [\"features.py\"],\n    tags = [\"no-mypy\"],\n    dependencies = [\n        \":params_lib\",\n        \"src/python/twitter/deepbird/projects/magic_recs/libs\",\n        \"twml:twml-nodeps\",\n    ],\n)\n\npython3_library(\n    name = \"model_pools_lib\",\n    sources = [\"model_pools.py\"],\n    tags = [\"no-mypy\"],\n    dependencies = [\n        \":features_lib\",\n        \":params_lib\",\n        \"src/python/twitter/deepbird/projects/magic_recs/v11/lib:model_lib\",\n    ],\n)\n\npython3_library(\n    name = \"graph_lib\",\n    sources = [\"graph.py\"],\n    tags = [\"no-mypy\"],\n    dependencies = [\n        \":params_lib\",\n        \"src/python/twitter/deepbird/projects/magic_recs/libs\",\n    ],\n)\n\npython3_library(\n    name = \"run_args_lib\",\n    sources = [\"run_args.py\"],\n    tags = [\"no-mypy\"],\n    dependencies = [\n        \":features_lib\",\n        \":params_lib\",\n        \"twml:twml-nodeps\",\n    ],\n)\n\npython3_library(\n    name = \"deep_norm_lib\",\n    sources = [\"deep_norm.py\"],\n    tags = [\"no-mypy\"],\n    dependencies = [\n        \":features_lib\",\n        \":graph_lib\",\n        \":model_pools_lib\",\n        \":params_lib\",\n        \":run_args_lib\",\n        \"src/python/twitter/deepbird/projects/magic_recs/libs\",\n        \"src/python/twitter/deepbird/util/data\",\n        \"twml:twml-nodeps\",\n    ],\n)\n\npython3_library(\n    name = \"eval_lib\",\n    sources = [\"eval.py\"],\n    tags = [\"no-mypy\"],\n    dependencies = [\n        \":features_lib\",\n        \":graph_lib\",\n        \":model_pools_lib\",\n        \":params_lib\",\n        \":run_args_lib\",\n        \"src/python/twitter/deepbird/projects/magic_recs/libs\",\n        \"twml:twml-nodeps\",\n    ],\n)\n\npython37_binary(\n    name = \"deep_norm\",\n    source = \"deep_norm.py\",\n    dependencies = [\n        \":deep_norm_lib\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:deep_norm\",\n        \"twml\",\n    ],\n)\n\npython37_binary(\n    name = \"eval\",\n    source = \"eval.py\",\n    dependencies = [\n        \":eval_lib\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:eval\",\n        \"twml\",\n    ],\n)\n\npython3_library(\n    name = \"mlwf_libs\",\n    tags = [\"no-mypy\"],\n    dependencies = [\n        \":deep_norm_lib\",\n        \"twml\",\n    ],\n)\n\npython37_binary(\n    name = \"train_model\",\n    source = \"deep_norm.py\",\n    dependencies = [\n        \":deep_norm_lib\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:train_model\",\n    ],\n)\n\npython37_binary(\n    name = \"train_model_local\",\n    source = \"deep_norm.py\",\n    dependencies = [\n        \":deep_norm_lib\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:train_model_local\",\n        \"twml\",\n    ],\n)\n\npython37_binary(\n    name = \"eval_model_local\",\n    source = \"eval.py\",\n    dependencies = [\n        \":eval_lib\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:eval_model_local\",\n        \"twml\",\n    ],\n)\n\npython37_binary(\n    name = \"eval_model\",\n    source = \"eval.py\",\n    dependencies = [\n        \":eval_lib\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:eval_model\",\n    ],\n)\n\npython37_binary(\n    name = \"mlwf_model\",\n    source = \"deep_norm.py\",\n    dependencies = [\n        \":mlwf_libs\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:mlwf_model\",\n    ],\n)\n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/README.md",
    "content": "# Notification Heavy Ranker Model\n\n## Model Context\nThere are 4 major components of Twitter notifications recommendation system: 1) candidate generation 2) light ranking 3) heavy ranking & 4) quality control. This notification heavy ranker model is the core ranking model for the personalised notifications recommendation. It's a multi-task learning model to predict the probabilities that the target users will open and engage with the sent notifications. \n\n\n## Directory Structure\n- BUILD: this file defines python library dependencies\n- deep_norm.py: this file contains how to set up continuous training, model evaluation and model exporting for the notification heavy ranker model\n- eval.py: the main python entry file to set up the overall model evaluation pipeline\n- features.py: this file contains importing feature list and support functions for feature engineering\n- graph.py: this file defines how to build the tensorflow graph with specified model architecture, loss function and training configuration\n- model_pools.py: this file defines the available model types for the heavy ranker\n- params.py: this file defines hyper-parameters used in the notification heavy ranker \n- run_args.py: this file defines command line parameters to run model training & evaluation\n- update_warm_start_checkpoint.py: this file contains the support to modify checkpoints of the given saved heavy ranker model\n- lib/BUILD: this file defines python library dependencies for tensorflow model architecture\n- lib/layers.py: this file defines different type of convolution layers to be used in the heavy ranker model\n- lib/model.py: this file defines the module containing ClemNet, the heavy ranker model type\n- lib/params.py: this file defines parameters used in the heavy ranker model \n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/__init__.py",
    "content": ""
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/deep_norm.py",
    "content": "\"\"\"\nTraining job for the heavy ranker of the push notification service.\n\"\"\"\nfrom datetime import datetime\nimport json\nimport os\n\nimport twml\n\nfrom ..libs.metric_fn_utils import flip_disliked_labels, get_metric_fn\nfrom ..libs.model_utils import read_config\nfrom ..libs.warm_start_utils import get_feature_list_for_heavy_ranking, warm_start_checkpoint\nfrom .features import get_feature_config\nfrom .model_pools import ALL_MODELS\nfrom .params import load_graph_params\nfrom .run_args import get_training_arg_parser\n\nimport tensorflow.compat.v1 as tf\nfrom tensorflow.compat.v1 import logging\n\n\ndef main() -> None:\n  args, _ = get_training_arg_parser().parse_known_args()\n  logging.info(f\"Parsed args: {args}\")\n\n  params = load_graph_params(args)\n  logging.info(f\"Loaded graph params: {params}\")\n\n  param_file = os.path.join(args.save_dir, \"params.json\")\n  logging.info(f\"Saving graph params to: {param_file}\")\n  with tf.io.gfile.GFile(param_file, mode=\"w\") as file:\n    json.dump(params.json(), file, ensure_ascii=False, indent=4)\n\n  logging.info(f\"Get Feature Config: {args.feature_list}\")\n  feature_list = read_config(args.feature_list).items()\n  feature_config = get_feature_config(\n    data_spec_path=args.data_spec,\n    params=params,\n    feature_list_provided=feature_list,\n  )\n  feature_list_path = args.feature_list\n\n  warm_start_from = args.warm_start_from\n  if args.warm_start_base_dir:\n    logging.info(f\"Get warm started model from: {args.warm_start_base_dir}.\")\n\n    continuous_binary_feat_list_save_path = os.path.join(\n      args.warm_start_base_dir, \"continuous_binary_feat_list.json\"\n    )\n    warm_start_folder = os.path.join(args.warm_start_base_dir, \"best_checkpoint\")\n    job_name = os.path.basename(args.save_dir)\n    ws_output_ckpt_folder = os.path.join(args.warm_start_base_dir, f\"warm_start_for_{job_name}\")\n    if tf.io.gfile.exists(ws_output_ckpt_folder):\n      tf.io.gfile.rmtree(ws_output_ckpt_folder)\n\n    tf.io.gfile.mkdir(ws_output_ckpt_folder)\n\n    warm_start_from = warm_start_checkpoint(\n      warm_start_folder,\n      continuous_binary_feat_list_save_path,\n      feature_list_path,\n      args.data_spec,\n      ws_output_ckpt_folder,\n    )\n    logging.info(f\"Created warm_start_from_ckpt {warm_start_from}.\")\n\n  logging.info(\"Build Trainer.\")\n  metric_fn = get_metric_fn(\"OONC_Engagement\" if len(params.tasks) == 2 else \"OONC\", False)\n\n  trainer = twml.trainers.DataRecordTrainer(\n    name=\"magic_recs\",\n    params=args,\n    build_graph_fn=lambda *args: ALL_MODELS[params.model.name](params=params)(*args),\n    save_dir=args.save_dir,\n    run_config=None,\n    feature_config=feature_config,\n    metric_fn=flip_disliked_labels(metric_fn),\n    warm_start_from=warm_start_from,\n  )\n\n  logging.info(\"Build train and eval input functions.\")\n  train_input_fn = trainer.get_train_input_fn(shuffle=True)\n  eval_input_fn = trainer.get_eval_input_fn(repeat=False, shuffle=False)\n\n  learn = trainer.learn\n  if args.distributed or args.num_workers is not None:\n    learn = trainer.train_and_evaluate\n\n  if not args.directly_export_best:\n    logging.info(\"Starting training\")\n    start = datetime.now()\n    learn(\n      early_stop_minimize=False,\n      early_stop_metric=\"pr_auc_unweighted_OONC\",\n      early_stop_patience=args.early_stop_patience,\n      early_stop_tolerance=args.early_stop_tolerance,\n      eval_input_fn=eval_input_fn,\n      train_input_fn=train_input_fn,\n    )\n    logging.info(f\"Total training time: {datetime.now() - start}\")\n  else:\n    logging.info(\"Directly exporting the model\")\n\n  if not args.export_dir:\n    args.export_dir = os.path.join(args.save_dir, \"exported_models\")\n\n  logging.info(f\"Exporting the model to {args.export_dir}.\")\n  start = datetime.now()\n  twml.contrib.export.export_fn.export_all_models(\n    trainer=trainer,\n    export_dir=args.export_dir,\n    parse_fn=feature_config.get_parse_fn(),\n    serving_input_receiver_fn=feature_config.get_serving_input_receiver_fn(),\n    export_output_fn=twml.export_output_fns.batch_prediction_continuous_output_fn,\n  )\n\n  logging.info(f\"Total model export time: {datetime.now() - start}\")\n  logging.info(f\"The MLP directory is: {args.save_dir}\")\n\n  continuous_binary_feat_list_save_path = os.path.join(\n    args.save_dir, \"continuous_binary_feat_list.json\"\n  )\n  logging.info(\n    f\"Saving the list of continuous and binary features to {continuous_binary_feat_list_save_path}.\"\n  )\n  continuous_binary_feat_list = get_feature_list_for_heavy_ranking(\n    feature_list_path, args.data_spec\n  )\n  twml.util.write_file(\n    continuous_binary_feat_list_save_path, continuous_binary_feat_list, encode=\"json\"\n  )\n\n\nif __name__ == \"__main__\":\n  main()\n  logging.info(\"Done.\")\n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/eval.py",
    "content": "\"\"\"\nEvaluation job for the heavy ranker of the push notification service.\n\"\"\"\n\nfrom datetime import datetime\n\nimport twml\n\nfrom ..libs.metric_fn_utils import get_metric_fn\nfrom ..libs.model_utils import read_config\nfrom .features import get_feature_config\nfrom .model_pools import ALL_MODELS\nfrom .params import load_graph_params\nfrom .run_args import get_eval_arg_parser\n\nfrom tensorflow.compat.v1 import logging\n\n\ndef main():\n  args, _ = get_eval_arg_parser().parse_known_args()\n  logging.info(f\"Parsed args: {args}\")\n\n  params = load_graph_params(args)\n  logging.info(f\"Loaded graph params: {params}\")\n\n  logging.info(f\"Get Feature Config: {args.feature_list}\")\n  feature_list = read_config(args.feature_list).items()\n  feature_config = get_feature_config(\n    data_spec_path=args.data_spec,\n    params=params,\n    feature_list_provided=feature_list,\n  )\n\n  logging.info(\"Build DataRecordTrainer.\")\n  metric_fn = get_metric_fn(\"OONC_Engagement\" if len(params.tasks) == 2 else \"OONC\", False)\n\n  trainer = twml.trainers.DataRecordTrainer(\n    name=\"magic_recs\",\n    params=args,\n    build_graph_fn=lambda *args: ALL_MODELS[params.model.name](params=params)(*args),\n    save_dir=args.save_dir,\n    run_config=None,\n    feature_config=feature_config,\n    metric_fn=metric_fn,\n  )\n\n  logging.info(\"Run the evaluation.\")\n  start = datetime.now()\n  trainer._estimator.evaluate(\n    input_fn=trainer.get_eval_input_fn(repeat=False, shuffle=False),\n    steps=None if (args.eval_steps is not None and args.eval_steps < 0) else args.eval_steps,\n    checkpoint_path=args.eval_checkpoint,\n  )\n  logging.info(f\"Evaluating time: {datetime.now() - start}.\")\n\n\nif __name__ == \"__main__\":\n  main()\n  logging.info(\"Job done.\")\n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/features.py",
    "content": "import os\nfrom typing import Dict\n\nfrom twitter.deepbird.projects.magic_recs.libs.model_utils import filter_nans_and_infs\nimport twml\nfrom twml.layers import full_sparse, sparse_max_norm\n\nfrom .params import FeaturesParams, GraphParams, SparseFeaturesParams\n\nimport tensorflow as tf\nfrom tensorflow import Tensor\nimport tensorflow.compat.v1 as tf1\n\n\nFEAT_CONFIG_DEFAULT_VAL = 0\nDEFAULT_FEATURE_LIST_PATH = \"./feature_list_default.yaml\"\nFEATURE_LIST_DEFAULT_PATH = os.path.join(\n  os.path.dirname(os.path.realpath(__file__)), DEFAULT_FEATURE_LIST_PATH\n)\n\n\ndef get_feature_config(data_spec_path=None, feature_list_provided=[], params: GraphParams = None):\n\n  a_string_feat_list = [feat for feat, feat_type in feature_list_provided if feat_type != \"S\"]\n\n  builder = twml.contrib.feature_config.FeatureConfigBuilder(\n    data_spec_path=data_spec_path, debug=False\n  )\n\n  builder = builder.extract_feature_group(\n    feature_regexes=a_string_feat_list,\n    group_name=\"continuous_features\",\n    default_value=FEAT_CONFIG_DEFAULT_VAL,\n    type_filter=[\"CONTINUOUS\"],\n  )\n\n  builder = builder.extract_feature_group(\n    feature_regexes=a_string_feat_list,\n    group_name=\"binary_features\",\n    type_filter=[\"BINARY\"],\n  )\n\n  if params.model.features.sparse_features:\n    builder = builder.extract_features_as_hashed_sparse(\n      feature_regexes=a_string_feat_list,\n      hash_space_size_bits=params.model.features.sparse_features.bits,\n      type_filter=[\"DISCRETE\", \"STRING\", \"SPARSE_BINARY\"],\n      output_tensor_name=\"sparse_not_continuous\",\n    )\n\n    builder = builder.extract_features_as_hashed_sparse(\n      feature_regexes=[feat for feat, feat_type in feature_list_provided if feat_type == \"S\"],\n      hash_space_size_bits=params.model.features.sparse_features.bits,\n      type_filter=[\"SPARSE_CONTINUOUS\"],\n      output_tensor_name=\"sparse_continuous\",\n    )\n\n  builder = builder.add_labels([task.label for task in params.tasks] + [\"label.ntabDislike\"])\n\n  if params.weight:\n    builder = builder.define_weight(params.weight)\n\n  return builder.build()\n\n\ndef dense_features(features: Dict[str, Tensor], training: bool) -> Tensor:\n  \"\"\"\n  Performs feature transformations on the raw dense features (continuous and binary).\n  \"\"\"\n  with tf.name_scope(\"dense_features\"):\n    x = filter_nans_and_infs(features[\"continuous_features\"])\n\n    x = tf.sign(x) * tf.math.log(tf.abs(x) + 1)\n    x = tf1.layers.batch_normalization(\n      x, momentum=0.9999, training=training, renorm=training, axis=1\n    )\n    x = tf.clip_by_value(x, -5, 5)\n\n    transformed_continous_features = tf.where(tf.math.is_nan(x), tf.zeros_like(x), x)\n\n    binary_features = filter_nans_and_infs(features[\"binary_features\"])\n    binary_features = tf.dtypes.cast(binary_features, tf.float32)\n\n    output = tf.concat([transformed_continous_features, binary_features], axis=1)\n\n  return output\n\n\ndef sparse_features(\n  features: Dict[str, Tensor], training: bool, params: SparseFeaturesParams\n) -> Tensor:\n  \"\"\"\n  Performs feature transformations on the raw sparse features.\n  \"\"\"\n\n  with tf.name_scope(\"sparse_features\"):\n    with tf.name_scope(\"sparse_not_continuous\"):\n      sparse_not_continuous = full_sparse(\n        inputs=features[\"sparse_not_continuous\"],\n        output_size=params.embedding_size,\n        use_sparse_grads=training,\n        use_binary_values=False,\n      )\n\n    with tf.name_scope(\"sparse_continuous\"):\n      shape_enforced_input = twml.util.limit_sparse_tensor_size(\n        sparse_tf=features[\"sparse_continuous\"], input_size_bits=params.bits, mask_indices=False\n      )\n\n      normalized_continuous_sparse = sparse_max_norm(\n        inputs=shape_enforced_input, is_training=training\n      )\n\n      sparse_continuous = full_sparse(\n        inputs=normalized_continuous_sparse,\n        output_size=params.embedding_size,\n        use_sparse_grads=training,\n        use_binary_values=False,\n      )\n\n    output = tf.concat([sparse_not_continuous, sparse_continuous], axis=1)\n\n  return output\n\n\ndef get_features(features: Dict[str, Tensor], training: bool, params: FeaturesParams) -> Tensor:\n  \"\"\"\n  Performs feature transformations on the dense and sparse features and combine the resulting\n  tensors into a single one.\n  \"\"\"\n  with tf.name_scope(\"features\"):\n    x = dense_features(features, training)\n    tf1.logging.info(f\"Dense features: {x.shape}\")\n\n    if params.sparse_features:\n      x = tf.concat([x, sparse_features(features, training, params.sparse_features)], axis=1)\n\n  return x\n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/graph.py",
    "content": "\"\"\"\nGraph class defining methods to obtain key quantities such as:\n  * the logits\n  * the probabilities\n  * the final score\n  * the loss function\n  * the training operator\n\"\"\"\nfrom __future__ import annotations\n\nfrom abc import ABC, abstractmethod\nfrom typing import Any, Dict\n\nfrom twitter.deepbird.hparam import HParams\nimport twml\n\nfrom ..libs.model_utils import generate_disliked_mask\nfrom .params import GraphParams\n\nimport tensorflow as tf\nimport tensorflow.compat.v1 as tf1\n\n\nclass Graph(ABC):\n  def __init__(self, params: GraphParams):\n    self.params = params\n\n  @abstractmethod\n  def get_logits(self, features: Dict[str, tf.Tensor], mode: tf.estimator.ModeKeys) -> tf.Tensor:\n    pass\n\n  def get_probabilities(self, logits: tf.Tensor) -> tf.Tensor:\n    return tf.math.cumprod(tf.nn.sigmoid(logits), axis=1, name=\"probabilities\")\n\n  def get_task_weights(self, labels: tf.Tensor) -> tf.Tensor:\n    oonc_label = tf.reshape(labels[:, 0], shape=(-1, 1))\n    task_weights = tf.concat([tf.ones_like(oonc_label), oonc_label], axis=1)\n\n    n_labels = len(self.params.tasks)\n    task_weights = tf.reshape(task_weights[:, 0:n_labels], shape=(-1, n_labels))\n\n    return task_weights\n\n  def get_loss(self, labels: tf.Tensor, logits: tf.Tensor, **kwargs: Any) -> tf.Tensor:\n    with tf.name_scope(\"weights\"):\n      disliked_mask = generate_disliked_mask(labels)\n\n      labels = tf.reshape(labels[:, 0:2], shape=[-1, 2])\n\n      labels = labels * tf.cast(tf.logical_not(disliked_mask), dtype=labels.dtype)\n\n      with tf.name_scope(\"task_weight\"):\n        task_weights = self.get_task_weights(labels)\n\n      with tf.name_scope(\"batch_size\"):\n        batch_size = tf.cast(tf.shape(labels)[0], dtype=tf.float32, name=\"batch_size\")\n\n      weights = task_weights / batch_size\n\n    with tf.name_scope(\"loss\"):\n      loss = tf.reduce_sum(\n        tf.nn.sigmoid_cross_entropy_with_logits(labels=labels, logits=logits) * weights,\n      )\n\n    return loss\n\n  def get_score(self, probabilities: tf.Tensor) -> tf.Tensor:\n    with tf.name_scope(\"score_weight\"):\n      score_weights = tf.constant([task.score_weight for task in self.params.tasks])\n      score_weights = score_weights / tf.reduce_sum(score_weights, axis=0)\n\n    with tf.name_scope(\"score\"):\n      score = tf.reshape(tf.reduce_sum(probabilities * score_weights, axis=1), shape=[-1, 1])\n\n    return score\n\n  def get_train_op(self, loss: tf.Tensor, twml_params) -> Any:\n    with tf.name_scope(\"optimizer\"):\n      learning_rate = twml_params.learning_rate\n      optimizer = tf1.train.GradientDescentOptimizer(learning_rate=learning_rate)\n\n    update_ops = set(tf1.get_collection(tf1.GraphKeys.UPDATE_OPS))\n    with tf.control_dependencies(update_ops):\n      train_op = twml.optimizers.optimize_loss(\n        loss=loss,\n        variables=tf1.trainable_variables(),\n        global_step=tf1.train.get_global_step(),\n        optimizer=optimizer,\n        learning_rate=None,\n      )\n\n    return train_op\n\n  def __call__(\n    self,\n    features: Dict[str, tf.Tensor],\n    labels: tf.Tensor,\n    mode: tf.estimator.ModeKeys,\n    params: HParams,\n    config=None,\n  ) -> Dict[str, tf.Tensor]:\n    training = mode == tf.estimator.ModeKeys.TRAIN\n    logits = self.get_logits(features=features, training=training)\n    probabilities = self.get_probabilities(logits=logits)\n    score = None\n    loss = None\n    train_op = None\n\n    if mode == tf.estimator.ModeKeys.PREDICT:\n      score = self.get_score(probabilities=probabilities)\n      output = {\"loss\": loss, \"train_op\": train_op, \"prediction\": score}\n\n    elif mode in (tf.estimator.ModeKeys.TRAIN, tf.estimator.ModeKeys.EVAL):\n      loss = self.get_loss(labels=labels, logits=logits)\n\n      if mode == tf.estimator.ModeKeys.TRAIN:\n        train_op = self.get_train_op(loss=loss, twml_params=params)\n\n      output = {\"loss\": loss, \"train_op\": train_op, \"output\": probabilities}\n\n    else:\n      raise ValueError(\n        f\"\"\"\n        Invalid mode. Possible values are: {tf.estimator.ModeKeys.PREDICT}, {tf.estimator.ModeKeys.TRAIN}, and {tf.estimator.ModeKeys.EVAL}\n        . Passed: {mode}\n      \"\"\"\n      )\n\n    return output\n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/lib/BUILD",
    "content": "python3_library(\n    name = \"params_lib\",\n    sources = [\n        \"params.py\",\n    ],\n    tags = [\n        \"bazel-compatible\",\n        \"no-mypy\",\n    ],\n    dependencies = [\n        \"3rdparty/python/pydantic:default\",\n    ],\n)\n\npython3_library(\n    name = \"layers_lib\",\n    sources = [\n        \"layers.py\",\n    ],\n    tags = [\n        \"bazel-compatible\",\n        \"no-mypy\",\n    ],\n    dependencies = [\n    ],\n)\n\npython3_library(\n    name = \"model_lib\",\n    sources = [\n        \"model.py\",\n    ],\n    tags = [\n        \"bazel-compatible\",\n        \"no-mypy\",\n    ],\n    dependencies = [\n        \":layers_lib\",\n        \":params_lib\",\n        \"3rdparty/python/absl-py:default\",\n    ],\n)\n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/lib/layers.py",
    "content": "\"\"\"\nDifferent type of convolution layers to be used in the ClemNet.\n\"\"\"\nfrom typing import Any\n\nimport tensorflow as tf\n\n\nclass KerasConv1D(tf.keras.layers.Layer):\n  \"\"\"\n  Basic Conv1D layer in a wrapper to be compatible with ClemNet.\n  \"\"\"\n\n  def __init__(\n    self,\n    kernel_size: int,\n    filters: int,\n    strides: int,\n    padding: str,\n    use_bias: bool = True,\n    kernel_initializer: str = \"glorot_uniform\",\n    bias_initializer: str = \"zeros\",\n    **kwargs: Any,\n  ):\n    super(KerasConv1D, self).__init__(**kwargs)\n    self.kernel_size = kernel_size\n    self.filters = filters\n    self.use_bias = use_bias\n    self.kernel_initializer = kernel_initializer\n    self.bias_initializer = bias_initializer\n    self.strides = strides\n    self.padding = padding\n\n  def build(self, input_shape: tf.TensorShape) -> None:\n    assert (\n      len(input_shape) == 3\n    ), f\"Tensor shape must be of length 3. Passed tensor of shape {input_shape}.\"\n\n    self.features = input_shape[1]\n\n    self.w = tf.keras.layers.Conv1D(\n      kernel_size=self.kernel_size,\n      filters=self.filters,\n      strides=self.strides,\n      padding=self.padding,\n      use_bias=self.use_bias,\n      kernel_initializer=self.kernel_initializer,\n      bias_initializer=self.bias_initializer,\n      name=self.name,\n    )\n\n  def call(self, inputs: tf.Tensor, **kwargs: Any) -> tf.Tensor:\n    return self.w(inputs)\n\n\nclass ChannelWiseDense(tf.keras.layers.Layer):\n  \"\"\"\n  Dense layer is applied to each channel separately. This is more memory and computationally\n  efficient than flattening the channels and performing single dense layers over it which is the\n  default behavior in tf1.\n  \"\"\"\n\n  def __init__(\n    self,\n    output_size: int,\n    use_bias: bool,\n    kernel_initializer: str = \"uniform_glorot\",\n    bias_initializer: str = \"zeros\",\n    **kwargs: Any,\n  ):\n    super(ChannelWiseDense, self).__init__(**kwargs)\n    self.output_size = output_size\n    self.use_bias = use_bias\n    self.kernel_initializer = kernel_initializer\n    self.bias_initializer = bias_initializer\n\n  def build(self, input_shape: tf.TensorShape) -> None:\n    assert (\n      len(input_shape) == 3\n    ), f\"Tensor shape must be of length 3. Passed tensor of shape {input_shape}.\"\n\n    input_size = input_shape[1]\n    channels = input_shape[2]\n\n    self.kernel = self.add_weight(\n      name=\"kernel\",\n      shape=(channels, input_size, self.output_size),\n      initializer=self.kernel_initializer,\n      trainable=True,\n    )\n\n    self.bias = self.add_weight(\n      name=\"bias\",\n      shape=(channels, self.output_size),\n      initializer=self.bias_initializer,\n      trainable=self.use_bias,\n    )\n\n  def call(self, inputs: tf.Tensor, **kwargs: Any) -> tf.Tensor:\n    x = inputs\n\n    transposed_x = tf.transpose(x, perm=[2, 0, 1])\n    transposed_residual = (\n      tf.transpose(tf.matmul(transposed_x, self.kernel), perm=[1, 0, 2]) + self.bias\n    )\n    output = tf.transpose(transposed_residual, perm=[0, 2, 1])\n\n    return output\n\n\nclass ResidualLayer(tf.keras.layers.Layer):\n  \"\"\"\n  Layer implementing a 3D-residual connection.\n  \"\"\"\n\n  def build(self, input_shape: tf.TensorShape) -> None:\n    assert (\n      len(input_shape) == 3\n    ), f\"Tensor shape must be of length 3. Passed tensor of shape {input_shape}.\"\n\n  def call(self, inputs: tf.Tensor, residual: tf.Tensor, **kwargs: Any) -> tf.Tensor:\n    shortcut = tf.keras.layers.Conv1D(\n      filters=int(residual.shape[2]), strides=1, kernel_size=1, padding=\"SAME\", use_bias=False\n    )(inputs)\n\n    output = tf.add(shortcut, residual)\n\n    return output\n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/lib/model.py",
    "content": "\"\"\"\nModule containing ClemNet.\n\"\"\"\nfrom typing import Any\n\nfrom .layers import ChannelWiseDense, KerasConv1D, ResidualLayer\nfrom .params import BlockParams, ClemNetParams\n\nimport tensorflow as tf\nimport tensorflow.compat.v1 as tf1\n\n\nclass Block2(tf.keras.layers.Layer):\n  \"\"\"\n  Possible ClemNet block. Architecture is as follow:\n    Optional(DenseLayer + BN + Act)\n    Optional(ConvLayer + BN + Act)\n    Optional(Residual Layer)\n\n  \"\"\"\n\n  def __init__(self, params: BlockParams, **kwargs: Any):\n    super(Block2, self).__init__(**kwargs)\n    self.params = params\n\n  def build(self, input_shape: tf.TensorShape) -> None:\n    assert (\n      len(input_shape) == 3\n    ), f\"Tensor shape must be of length 3. Passed tensor of shape {input_shape}.\"\n\n  def call(self, inputs: tf.Tensor, training: bool) -> tf.Tensor:\n    x = inputs\n    if self.params.dense:\n      x = ChannelWiseDense(**self.params.dense.dict())(inputs=x, training=training)\n      x = tf1.layers.batch_normalization(x, momentum=0.9999, training=training, axis=1)\n      x = tf.keras.layers.Activation(self.params.activation)(x)\n\n    if self.params.conv:\n      x = KerasConv1D(**self.params.conv.dict())(inputs=x, training=training)\n      x = tf1.layers.batch_normalization(x, momentum=0.9999, training=training, axis=1)\n      x = tf.keras.layers.Activation(self.params.activation)(x)\n\n    if self.params.residual:\n      x = ResidualLayer()(inputs=inputs, residual=x)\n\n    return x\n\n\nclass ClemNet(tf.keras.layers.Layer):\n  \"\"\"\n  A residual network stacking residual blocks composed of dense layers and convolutions.\n  \"\"\"\n\n  def __init__(self, params: ClemNetParams, **kwargs: Any):\n    super(ClemNet, self).__init__(**kwargs)\n    self.params = params\n\n  def build(self, input_shape: tf.TensorShape) -> None:\n    assert len(input_shape) in (\n      2,\n      3,\n    ), f\"Tensor shape must be of length 3. Passed tensor of shape {input_shape}.\"\n\n  def call(self, inputs: tf.Tensor, training: bool) -> tf.Tensor:\n    if len(inputs.shape) < 3:\n      inputs = tf.expand_dims(inputs, axis=-1)\n\n    x = inputs\n    for block_params in self.params.blocks:\n      x = Block2(block_params)(inputs=x, training=training)\n\n    x = tf.keras.layers.Flatten(name=\"flattened\")(x)\n    if self.params.top:\n      x = tf.keras.layers.Dense(units=self.params.top.n_labels, name=\"logits\")(x)\n\n    return x\n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/lib/params.py",
    "content": "\"\"\"\nParameters used in ClemNet.\n\"\"\"\nfrom typing import List, Optional\n\nfrom pydantic import BaseModel, Extra, Field, PositiveInt\n\n\n# checkstyle: noqa\n\n\nclass ExtendedBaseModel(BaseModel):\n  class Config:\n    extra = Extra.forbid\n\n\nclass DenseParams(ExtendedBaseModel):\n  name: Optional[str]\n  bias_initializer: str = \"zeros\"\n  kernel_initializer: str = \"glorot_uniform\"\n  output_size: PositiveInt\n  use_bias: bool = Field(True)\n\n\nclass ConvParams(ExtendedBaseModel):\n  name: Optional[str]\n  bias_initializer: str = \"zeros\"\n  filters: PositiveInt\n  kernel_initializer: str = \"glorot_uniform\"\n  kernel_size: PositiveInt\n  padding: str = \"SAME\"\n  strides: PositiveInt = 1\n  use_bias: bool = Field(True)\n\n\nclass BlockParams(ExtendedBaseModel):\n  activation: Optional[str]\n  conv: Optional[ConvParams]\n  dense: Optional[DenseParams]\n  residual: Optional[bool]\n\n\nclass TopLayerParams(ExtendedBaseModel):\n  n_labels: PositiveInt\n\n\nclass ClemNetParams(ExtendedBaseModel):\n  blocks: List[BlockParams] = []\n  top: Optional[TopLayerParams]\n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/model_pools.py",
    "content": "\"\"\"\nCandidate architectures for each task's.\n\"\"\"\n\nfrom __future__ import annotations\n\nfrom typing import Dict\n\nfrom .features import get_features\nfrom .graph import Graph\nfrom .lib.model import ClemNet\nfrom .params import ModelTypeEnum\n\nimport tensorflow as tf\n\n\nclass MagicRecsClemNet(Graph):\n  def get_logits(self, features: Dict[str, tf.Tensor], training: bool) -> tf.Tensor:\n\n    with tf.name_scope(\"logits\"):\n      inputs = get_features(features=features, training=training, params=self.params.model.features)\n\n      with tf.name_scope(\"OONC_logits\"):\n        model = ClemNet(params=self.params.model.architecture)\n        oonc_logit = model(inputs=inputs, training=training)\n\n      with tf.name_scope(\"EngagementGivenOONC_logits\"):\n        model = ClemNet(params=self.params.model.architecture)\n        eng_logits = model(inputs=inputs, training=training)\n\n      return tf.concat([oonc_logit, eng_logits], axis=1)\n\n\nALL_MODELS = {ModelTypeEnum.clemnet: MagicRecsClemNet}\n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/params.py",
    "content": "import enum\nimport json\nfrom typing import List, Optional\n\nfrom .lib.params import BlockParams, ClemNetParams, ConvParams, DenseParams, TopLayerParams\n\nfrom pydantic import BaseModel, Extra, NonNegativeFloat\nimport tensorflow.compat.v1 as tf\n\n\n# checkstyle: noqa\n\n\nclass ExtendedBaseModel(BaseModel):\n  class Config:\n    extra = Extra.forbid\n\n\nclass SparseFeaturesParams(ExtendedBaseModel):\n  bits: int\n  embedding_size: int\n\n\nclass FeaturesParams(ExtendedBaseModel):\n  sparse_features: Optional[SparseFeaturesParams]\n\n\nclass ModelTypeEnum(str, enum.Enum):\n  clemnet: str = \"clemnet\"\n\n\nclass ModelParams(ExtendedBaseModel):\n  name: ModelTypeEnum\n  features: FeaturesParams\n  architecture: ClemNetParams\n\n\nclass TaskNameEnum(str, enum.Enum):\n  oonc: str = \"OONC\"\n  engagement: str = \"Engagement\"\n\n\nclass Task(ExtendedBaseModel):\n  name: TaskNameEnum\n  label: str\n  score_weight: NonNegativeFloat\n\n\nDEFAULT_TASKS = [\n  Task(name=TaskNameEnum.oonc, label=\"label\", score_weight=0.9),\n  Task(name=TaskNameEnum.engagement, label=\"label.engagement\", score_weight=0.1),\n]\n\n\nclass GraphParams(ExtendedBaseModel):\n  tasks: List[Task] = DEFAULT_TASKS\n  model: ModelParams\n  weight: Optional[str]\n\n\nDEFAULT_ARCHITECTURE_PARAMS = ClemNetParams(\n  blocks=[\n    BlockParams(\n      activation=\"relu\",\n      conv=ConvParams(kernel_size=3, filters=5),\n      dense=DenseParams(output_size=output_size),\n      residual=False,\n    )\n    for output_size in [1024, 512, 256, 128]\n  ],\n  top=TopLayerParams(n_labels=1),\n)\n\nDEFAULT_GRAPH_PARAMS = GraphParams(\n  model=ModelParams(\n    name=ModelTypeEnum.clemnet,\n    architecture=DEFAULT_ARCHITECTURE_PARAMS,\n    features=FeaturesParams(sparse_features=SparseFeaturesParams(bits=18, embedding_size=50)),\n  ),\n)\n\n\ndef load_graph_params(args) -> GraphParams:\n  params = DEFAULT_GRAPH_PARAMS\n  if args.param_file:\n    with tf.io.gfile.GFile(args.param_file, mode=\"r+\") as file:\n      params = GraphParams.parse_obj(json.load(file))\n\n  return params\n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/run_args.py",
    "content": "from twml.trainers import DataRecordTrainer\n\nfrom .features import FEATURE_LIST_DEFAULT_PATH\n\n\ndef get_training_arg_parser():\n  parser = DataRecordTrainer.add_parser_arguments()\n\n  parser.add_argument(\n    \"--feature_list\",\n    default=FEATURE_LIST_DEFAULT_PATH,\n    type=str,\n    help=\"Which features to use for training\",\n  )\n\n  parser.add_argument(\n    \"--param_file\",\n    default=None,\n    type=str,\n    help=\"Path to JSON file containing the graph parameters. If None, model will load default parameters.\",\n  )\n\n  parser.add_argument(\n    \"--directly_export_best\",\n    default=False,\n    action=\"store_true\",\n    help=\"whether to directly_export best_checkpoint\",\n  )\n\n  parser.add_argument(\n    \"--warm_start_from\", default=None, type=str, help=\"model dir to warm start from\"\n  )\n\n  parser.add_argument(\n    \"--warm_start_base_dir\",\n    default=None,\n    type=str,\n    help=\"latest ckpt in this folder will be used to \",\n  )\n\n  parser.add_argument(\n    \"--model_type\",\n    default=None,\n    type=str,\n    help=\"Which type of model to train.\",\n  )\n  return parser\n\n\ndef get_eval_arg_parser():\n  parser = get_training_arg_parser()\n  parser.add_argument(\n    \"--eval_checkpoint\",\n    default=None,\n    type=str,\n    help=\"Which checkpoint to use for evaluation\",\n  )\n\n  return parser\n"
  },
  {
    "path": "pushservice/src/main/python/models/heavy_ranking/update_warm_start_checkpoint.py",
    "content": "\"\"\"\nModel for modifying the checkpoints of the magic recs cnn Model with addition, deletion, and reordering\nof continuous and binary features.\n\"\"\"\n\nimport os\n\nfrom twitter.deepbird.projects.magic_recs.libs.get_feat_config import FEATURE_LIST_DEFAULT_PATH\nfrom twitter.deepbird.projects.magic_recs.libs.warm_start_utils_v11 import (\n  get_feature_list_for_heavy_ranking,\n  mkdirp,\n  rename_dir,\n  rmdir,\n  warm_start_checkpoint,\n)\nimport twml\nfrom twml.trainers import DataRecordTrainer\n\nimport tensorflow.compat.v1 as tf\nfrom tensorflow.compat.v1 import logging\n\n\ndef get_arg_parser():\n  parser = DataRecordTrainer.add_parser_arguments()\n  parser.add_argument(\n    \"--model_type\",\n    default=\"deepnorm_gbdt_inputdrop2_rescale\",\n    type=str,\n    help=\"specify the model type to use.\",\n  )\n\n  parser.add_argument(\n    \"--model_trainer_name\",\n    default=\"None\",\n    type=str,\n    help=\"deprecated, added here just for api compatibility.\",\n  )\n\n  parser.add_argument(\n    \"--warm_start_base_dir\",\n    default=\"none\",\n    type=str,\n    help=\"latest ckpt in this folder will be used.\",\n  )\n\n  parser.add_argument(\n    \"--output_checkpoint_dir\",\n    default=\"none\",\n    type=str,\n    help=\"Output folder for warm started ckpt. If none, it will move warm_start_base_dir to backup, and overwrite it\",\n  )\n\n  parser.add_argument(\n    \"--feature_list\",\n    default=\"none\",\n    type=str,\n    help=\"Which features to use for training\",\n  )\n\n  parser.add_argument(\n    \"--old_feature_list\",\n    default=\"none\",\n    type=str,\n    help=\"Which features to use for training\",\n  )\n\n  return parser\n\n\ndef get_params(args=None):\n  parser = get_arg_parser()\n  if args is None:\n    return parser.parse_args()\n  else:\n    return parser.parse_args(args)\n\n\ndef _main():\n  opt = get_params()\n  logging.info(\"parse is: \")\n  logging.info(opt)\n\n  if opt.feature_list == \"none\":\n    feature_list_path = FEATURE_LIST_DEFAULT_PATH\n  else:\n    feature_list_path = opt.feature_list\n\n  if opt.warm_start_base_dir != \"none\" and tf.io.gfile.exists(opt.warm_start_base_dir):\n    if opt.output_checkpoint_dir == \"none\" or opt.output_checkpoint_dir == opt.warm_start_base_dir:\n      _warm_start_base_dir = os.path.normpath(opt.warm_start_base_dir) + \"_backup_warm_start\"\n      _output_folder_dir = opt.warm_start_base_dir\n\n      rename_dir(opt.warm_start_base_dir, _warm_start_base_dir)\n      tf.logging.info(f\"moved {opt.warm_start_base_dir} to {_warm_start_base_dir}\")\n    else:\n      _warm_start_base_dir = opt.warm_start_base_dir\n      _output_folder_dir = opt.output_checkpoint_dir\n\n    continuous_binary_feat_list_save_path = os.path.join(\n      _warm_start_base_dir, \"continuous_binary_feat_list.json\"\n    )\n\n    if opt.old_feature_list != \"none\":\n      tf.logging.info(\"getting old continuous_binary_feat_list\")\n      continuous_binary_feat_list = get_feature_list_for_heavy_ranking(\n        opt.old_feature_list, opt.data_spec\n      )\n      rmdir(continuous_binary_feat_list_save_path)\n      twml.util.write_file(\n        continuous_binary_feat_list_save_path, continuous_binary_feat_list, encode=\"json\"\n      )\n      tf.logging.info(f\"Finish writting files to {continuous_binary_feat_list_save_path}\")\n\n    warm_start_folder = os.path.join(_warm_start_base_dir, \"best_checkpoint\")\n    if not tf.io.gfile.exists(warm_start_folder):\n      warm_start_folder = _warm_start_base_dir\n\n    rmdir(_output_folder_dir)\n    mkdirp(_output_folder_dir)\n\n    new_ckpt = warm_start_checkpoint(\n      warm_start_folder,\n      continuous_binary_feat_list_save_path,\n      feature_list_path,\n      opt.data_spec,\n      _output_folder_dir,\n      opt.model_type,\n    )\n    logging.info(f\"Created new ckpt {new_ckpt} from {warm_start_folder}\")\n\n    tf.logging.info(\"getting new continuous_binary_feat_list\")\n    new_continuous_binary_feat_list_save_path = os.path.join(\n      _output_folder_dir, \"continuous_binary_feat_list.json\"\n    )\n    continuous_binary_feat_list = get_feature_list_for_heavy_ranking(\n      feature_list_path, opt.data_spec\n    )\n    rmdir(new_continuous_binary_feat_list_save_path)\n    twml.util.write_file(\n      new_continuous_binary_feat_list_save_path, continuous_binary_feat_list, encode=\"json\"\n    )\n    tf.logging.info(f\"Finish writting files to {new_continuous_binary_feat_list_save_path}\")\n\n\nif __name__ == \"__main__\":\n  _main()\n"
  },
  {
    "path": "pushservice/src/main/python/models/libs/BUILD",
    "content": "python3_library(\n    name = \"libs\",\n    sources = [\"*.py\"],\n    tags = [\n        \"bazel-compatible\",\n        \"no-mypy\",\n    ],\n    dependencies = [\n        \"cortex/recsys/src/python/twitter/cortex/recsys/utils\",\n        \"magicpony/common/file_access/src/python/twitter/magicpony/common/file_access\",\n        \"src/python/twitter/cortex/ml/embeddings/deepbird\",\n        \"src/python/twitter/cortex/ml/embeddings/deepbird/grouped_metrics\",\n        \"src/python/twitter/deepbird/util/data\",\n        \"twml:twml-nodeps\",\n    ],\n)\n"
  },
  {
    "path": "pushservice/src/main/python/models/libs/__init__.py",
    "content": ""
  },
  {
    "path": "pushservice/src/main/python/models/libs/customized_full_sparse.py",
    "content": "# pylint: disable=no-member, arguments-differ, attribute-defined-outside-init, unused-argument\n\"\"\"\nImplementing Full Sparse Layer, allow specify use_binary_value in call() to\noveride default action.\n\"\"\"\n\nfrom twml.layers import FullSparse as defaultFullSparse\nfrom twml.layers.full_sparse import sparse_dense_matmul\n\nimport tensorflow.compat.v1 as tf\n\n\nclass FullSparse(defaultFullSparse):\n  def call(self, inputs, use_binary_values=None, **kwargs):  # pylint: disable=unused-argument\n    \"\"\"The logic of the layer lives here.\n\n    Arguments:\n      inputs:\n        A SparseTensor or a list of SparseTensors.\n        If `inputs` is a list, all tensors must have same `dense_shape`.\n\n    Returns:\n      - If `inputs` is `SparseTensor`, then returns `bias + inputs * dense_b`.\n      - If `inputs` is a `list[SparseTensor`, then returns\n       `bias + add_n([sp_a * dense_b for sp_a in inputs])`.\n    \"\"\"\n\n    if use_binary_values is not None:\n      default_use_binary_values = use_binary_values\n    else:\n      default_use_binary_values = self.use_binary_values\n\n    if isinstance(default_use_binary_values, (list, tuple)):\n      raise ValueError(\n        \"use_binary_values can not be %s when inputs is %s\"\n        % (type(default_use_binary_values), type(inputs))\n      )\n\n    outputs = sparse_dense_matmul(\n      inputs,\n      self.weight,\n      self.use_sparse_grads,\n      default_use_binary_values,\n      name=\"sparse_mm\",\n      partition_axis=self.partition_axis,\n      num_partitions=self.num_partitions,\n      compress_ids=self._use_compression,\n      cast_indices_dtype=self._cast_indices_dtype,\n    )\n\n    if self.bias is not None:\n      outputs = tf.nn.bias_add(outputs, self.bias)\n\n    if self.activation is not None:\n      return self.activation(outputs)  # pylint: disable=not-callable\n    return outputs\n"
  },
  {
    "path": "pushservice/src/main/python/models/libs/get_feat_config.py",
    "content": "import os\n\nfrom twitter.deepbird.projects.magic_recs.libs.metric_fn_utils import USER_AGE_FEATURE_NAME\nfrom twitter.deepbird.projects.magic_recs.libs.model_utils import read_config\nfrom twml.contrib import feature_config as contrib_feature_config\n\n\n# checkstyle: noqa\n\nFEAT_CONFIG_DEFAULT_VAL = -1.23456789\n\nDEFAULT_INPUT_SIZE_BITS = 18\n\nDEFAULT_FEATURE_LIST_PATH = \"./feature_list_default.yaml\"\nFEATURE_LIST_DEFAULT_PATH = os.path.join(\n  os.path.dirname(os.path.realpath(__file__)), DEFAULT_FEATURE_LIST_PATH\n)\n\nDEFAULT_FEATURE_LIST_LIGHT_RANKING_PATH = \"./feature_list_light_ranking.yaml\"\nFEATURE_LIST_DEFAULT_LIGHT_RANKING_PATH = os.path.join(\n  os.path.dirname(os.path.realpath(__file__)), DEFAULT_FEATURE_LIST_LIGHT_RANKING_PATH\n)\n\nFEATURE_LIST_DEFAULT = read_config(FEATURE_LIST_DEFAULT_PATH).items()\nFEATURE_LIST_LIGHT_RANKING_DEFAULT = read_config(FEATURE_LIST_DEFAULT_LIGHT_RANKING_PATH).items()\n\n\nLABELS = [\"label\"]\nLABELS_MTL = {\"OONC\": [\"label\"], \"OONC_Engagement\": [\"label\", \"label.engagement\"]}\nLABELS_LR = {\n  \"Sent\": [\"label.sent\"],\n  \"HeavyRankPosition\": [\"meta.ranking.is_top3\"],\n  \"HeavyRankProbability\": [\"meta.ranking.weighted_oonc_model_score\"],\n}\n\n\ndef _get_new_feature_config_base(\n  data_spec_path,\n  labels,\n  add_sparse_continous=True,\n  add_gbdt=True,\n  add_user_id=False,\n  add_timestamp=False,\n  add_user_age=False,\n  feature_list_provided=[],\n  opt=None,\n  run_light_ranking_group_metrics_in_bq=False,\n):\n  \"\"\"\n  Getter of the feature config based on specification.\n\n  Args:\n    data_spec_path: A string indicating the path of the data_spec.json file, which could be\n      either a local path or a hdfs path.\n    labels: A list of strings indicating the name of the label in the data spec.\n    add_sparse_continous: A bool indicating if sparse_continuous feature needs to be included.\n    add_gbdt: A bool indicating if gbdt feature needs to be included.\n    add_user_id: A bool indicating if user_id feature needs to be included.\n    add_timestamp: A bool indicating if timestamp feature needs to be included. This will be useful\n      for sequential models and meta learning models.\n    add_user_age: A bool indicating if the user age feature needs to be included.\n    feature_list_provided: A list of features thats need to be included. If not specified, will use\n      FEATURE_LIST_DEFAULT by default.\n    opt: A namespace of arguments indicating the hyparameters.\n    run_light_ranking_group_metrics_in_bq: A bool indicating if heavy ranker score info needs to be included to compute group metrics in BigQuery.\n\n  Returns:\n    A twml feature config object.\n  \"\"\"\n\n  input_size_bits = DEFAULT_INPUT_SIZE_BITS if opt is None else opt.input_size_bits\n\n  feature_list = feature_list_provided if feature_list_provided != [] else FEATURE_LIST_DEFAULT\n  a_string_feat_list = [f[0] for f in feature_list if f[1] != \"S\"]\n\n  builder = contrib_feature_config.FeatureConfigBuilder(data_spec_path=data_spec_path)\n\n  builder = builder.extract_feature_group(\n    feature_regexes=a_string_feat_list,\n    group_name=\"continuous\",\n    default_value=FEAT_CONFIG_DEFAULT_VAL,\n    type_filter=[\"CONTINUOUS\"],\n  )\n\n  builder = builder.extract_features_as_hashed_sparse(\n    feature_regexes=a_string_feat_list,\n    output_tensor_name=\"sparse_no_continuous\",\n    hash_space_size_bits=input_size_bits,\n    type_filter=[\"BINARY\", \"DISCRETE\", \"STRING\", \"SPARSE_BINARY\"],\n  )\n\n  if add_gbdt:\n    builder = builder.extract_features_as_hashed_sparse(\n      feature_regexes=[\"ads\\..*\"],\n      output_tensor_name=\"gbdt_sparse\",\n      hash_space_size_bits=input_size_bits,\n    )\n\n  if add_sparse_continous:\n    s_string_feat_list = [f[0] for f in feature_list if f[1] == \"S\"]\n\n    builder = builder.extract_features_as_hashed_sparse(\n      feature_regexes=s_string_feat_list,\n      output_tensor_name=\"sparse_continuous\",\n      hash_space_size_bits=input_size_bits,\n      type_filter=[\"SPARSE_CONTINUOUS\"],\n    )\n\n  if add_user_id:\n    builder = builder.extract_feature(\"meta.user_id\")\n  if add_timestamp:\n    builder = builder.extract_feature(\"meta.timestamp\")\n  if add_user_age:\n    builder = builder.extract_feature(USER_AGE_FEATURE_NAME)\n\n  if run_light_ranking_group_metrics_in_bq:\n    builder = builder.extract_feature(\"meta.trace_id\")\n    builder = builder.extract_feature(\"meta.ranking.weighted_oonc_model_score\")\n\n  builder = builder.add_labels(labels).define_weight(\"meta.weight\")\n\n  return builder.build()\n\n\ndef get_feature_config_with_sparse_continuous(\n  data_spec_path,\n  feature_list_provided=[],\n  opt=None,\n  add_user_id=False,\n  add_timestamp=False,\n  add_user_age=False,\n):\n  task_name = opt.task_name if getattr(opt, \"task_name\", None) is not None else \"OONC\"\n  if task_name not in LABELS_MTL:\n    raise ValueError(\"Invalid Task Name !\")\n\n  return _get_new_feature_config_base(\n    data_spec_path=data_spec_path,\n    labels=LABELS_MTL[task_name],\n    add_sparse_continous=True,\n    add_user_id=add_user_id,\n    add_timestamp=add_timestamp,\n    add_user_age=add_user_age,\n    feature_list_provided=feature_list_provided,\n    opt=opt,\n  )\n\n\ndef get_feature_config_light_ranking(\n  data_spec_path,\n  feature_list_provided=[],\n  opt=None,\n  add_user_id=True,\n  add_timestamp=False,\n  add_user_age=False,\n  add_gbdt=False,\n  run_light_ranking_group_metrics_in_bq=False,\n):\n  task_name = opt.task_name if getattr(opt, \"task_name\", None) is not None else \"HeavyRankPosition\"\n  if task_name not in LABELS_LR:\n    raise ValueError(\"Invalid Task Name !\")\n  if not feature_list_provided:\n    feature_list_provided = FEATURE_LIST_LIGHT_RANKING_DEFAULT\n\n  return _get_new_feature_config_base(\n    data_spec_path=data_spec_path,\n    labels=LABELS_LR[task_name],\n    add_sparse_continous=False,\n    add_gbdt=add_gbdt,\n    add_user_id=add_user_id,\n    add_timestamp=add_timestamp,\n    add_user_age=add_user_age,\n    feature_list_provided=feature_list_provided,\n    opt=opt,\n    run_light_ranking_group_metrics_in_bq=run_light_ranking_group_metrics_in_bq,\n  )\n"
  },
  {
    "path": "pushservice/src/main/python/models/libs/graph_utils.py",
    "content": "\"\"\"\nUtilties that aid in building the magic recs graph.\n\"\"\"\n\nimport re\n\nimport tensorflow.compat.v1 as tf\n\n\ndef get_trainable_variables(all_trainable_variables, trainable_regexes):\n  \"\"\"Returns a subset of trainable variables for training.\n\n  Given a collection of trainable variables, this will return all those that match the given regexes.\n  Will also log those variables.\n\n  Args:\n      all_trainable_variables (a collection of trainable tf.Variable): The variables to search through.\n      trainable_regexes (a collection of regexes): Variables that match any regex will be included.\n\n  Returns a list of tf.Variable\n  \"\"\"\n  if trainable_regexes is None or len(trainable_regexes) == 0:\n    tf.logging.info(\"No trainable regexes found. Not using get_trainable_variables behavior.\")\n    return None\n\n  assert any(\n    tf.is_tensor(var) for var in all_trainable_variables\n  ), f\"Non TF variable found: {all_trainable_variables}\"\n  trainable_variables = list(\n    filter(\n      lambda var: any(re.match(regex, var.name, re.IGNORECASE) for regex in trainable_regexes),\n      all_trainable_variables,\n    )\n  )\n  tf.logging.info(f\"Using filtered trainable variables: {trainable_variables}\")\n\n  assert (\n    trainable_variables\n  ), \"Did not find trainable variables after filtering after filtering from {} number of vars originaly. All vars: {} and train regexes: {}\".format(\n    len(all_trainable_variables), all_trainable_variables, trainable_regexes\n  )\n  return trainable_variables\n"
  },
  {
    "path": "pushservice/src/main/python/models/libs/group_metrics.py",
    "content": "import os\nimport time\n\nfrom twitter.cortex.ml.embeddings.deepbird.grouped_metrics.computation import (\n  write_grouped_metrics_to_mldash,\n)\nfrom twitter.cortex.ml.embeddings.deepbird.grouped_metrics.configuration import (\n  ClassificationGroupedMetricsConfiguration,\n  NDCGGroupedMetricsConfiguration,\n)\nimport twml\n\nfrom .light_ranking_metrics import (\n  CGRGroupedMetricsConfiguration,\n  ExpectedLossGroupedMetricsConfiguration,\n  RecallGroupedMetricsConfiguration,\n)\n\nimport numpy as np\nimport tensorflow.compat.v1 as tf\nfrom tensorflow.compat.v1 import logging\n\n\n# checkstyle: noqa\n\n\ndef run_group_metrics(trainer, data_dir, model_path, parse_fn, group_feature_name=\"meta.user_id\"):\n\n  start_time = time.time()\n  logging.info(\"Evaluating with group metrics.\")\n\n  metrics = write_grouped_metrics_to_mldash(\n    trainer=trainer,\n    data_dir=data_dir,\n    model_path=model_path,\n    group_fn=lambda datarecord: str(\n      datarecord.discreteFeatures[twml.feature_id(group_feature_name)[0]]\n    ),\n    parse_fn=parse_fn,\n    metric_configurations=[\n      ClassificationGroupedMetricsConfiguration(),\n      NDCGGroupedMetricsConfiguration(k=[5, 10, 20]),\n    ],\n    total_records_to_read=1000000000,\n    shuffle=False,\n    mldash_metrics_name=\"grouped_metrics\",\n  )\n\n  end_time = time.time()\n  logging.info(f\"Evaluated Group Metics: {metrics}.\")\n  logging.info(f\"Group metrics evaluation time {end_time - start_time}.\")\n\n\ndef run_group_metrics_light_ranking(\n  trainer, data_dir, model_path, parse_fn, group_feature_name=\"meta.trace_id\"\n):\n\n  start_time = time.time()\n  logging.info(\"Evaluating with group metrics.\")\n\n  metrics = write_grouped_metrics_to_mldash(\n    trainer=trainer,\n    data_dir=data_dir,\n    model_path=model_path,\n    group_fn=lambda datarecord: str(\n      datarecord.discreteFeatures[twml.feature_id(group_feature_name)[0]]\n    ),\n    parse_fn=parse_fn,\n    metric_configurations=[\n      CGRGroupedMetricsConfiguration(lightNs=[50, 100, 200], heavyKs=[1, 3, 10, 20, 50]),\n      RecallGroupedMetricsConfiguration(n=[50, 100, 200], k=[1, 3, 10, 20, 50]),\n      ExpectedLossGroupedMetricsConfiguration(lightNs=[50, 100, 200]),\n    ],\n    total_records_to_read=10000000,\n    num_batches_to_load=50,\n    batch_size=1024,\n    shuffle=False,\n    mldash_metrics_name=\"grouped_metrics_for_light_ranking\",\n  )\n\n  end_time = time.time()\n  logging.info(f\"Evaluated Group Metics for Light Ranking: {metrics}.\")\n  logging.info(f\"Group metrics evaluation time {end_time - start_time}.\")\n\n\ndef run_group_metrics_light_ranking_in_bq(trainer, params, checkpoint_path):\n  logging.info(\"getting Test Predictions for Light Ranking Group Metrics in BigQuery !!!\")\n  eval_input_fn = trainer.get_eval_input_fn(repeat=False, shuffle=False)\n  info_pool = []\n\n  for result in trainer.estimator.predict(\n    eval_input_fn, checkpoint_path=checkpoint_path, yield_single_examples=False\n  ):\n    traceID = result[\"trace_id\"]\n    pred = result[\"prediction\"]\n    label = result[\"target\"]\n    info = np.concatenate([traceID, pred, label], axis=1)\n    info_pool.append(info)\n\n  info_pool = np.concatenate(info_pool)\n\n  locname = \"/tmp/000/\"\n  if not os.path.exists(locname):\n    os.makedirs(locname)\n\n  locfile = locname + params.pred_file_name\n  columns = [\"trace_id\", \"model_prediction\", \"meta__ranking__weighted_oonc_model_score\"]\n  np.savetxt(locfile, info_pool, delimiter=\",\", header=\",\".join(columns))\n  tf.io.gfile.copy(locfile, params.pred_file_path + params.pred_file_name, overwrite=True)\n\n  if os.path.isfile(locfile):\n    os.remove(locfile)\n\n  logging.info(\"Done Prediction for Light Ranking Group Metrics in BigQuery.\")\n"
  },
  {
    "path": "pushservice/src/main/python/models/libs/initializer.py",
    "content": "import numpy as np\nfrom tensorflow.keras import backend as K\n\n\nclass VarianceScaling(object):\n  \"\"\"Initializer capable of adapting its scale to the shape of weights.\n  With `distribution=\"normal\"`, samples are drawn from a truncated normal\n  distribution centered on zero, with `stddev = sqrt(scale / n)` where n is:\n      - number of input units in the weight tensor, if mode = \"fan_in\"\n      - number of output units, if mode = \"fan_out\"\n      - average of the numbers of input and output units, if mode = \"fan_avg\"\n  With `distribution=\"uniform\"`,\n  samples are drawn from a uniform distribution\n  within [-limit, limit], with `limit = sqrt(3 * scale / n)`.\n  # Arguments\n      scale: Scaling factor (positive float).\n      mode: One of \"fan_in\", \"fan_out\", \"fan_avg\".\n      distribution: Random distribution to use. One of \"normal\", \"uniform\".\n      seed: A Python integer. Used to seed the random generator.\n  # Raises\n      ValueError: In case of an invalid value for the \"scale\", mode\" or\n        \"distribution\" arguments.\"\"\"\n\n  def __init__(\n    self,\n    scale=1.0,\n    mode=\"fan_in\",\n    distribution=\"normal\",\n    seed=None,\n    fan_in=None,\n    fan_out=None,\n  ):\n    self.fan_in = fan_in\n    self.fan_out = fan_out\n    if scale <= 0.0:\n      raise ValueError(\"`scale` must be a positive float. Got:\", scale)\n    mode = mode.lower()\n    if mode not in {\"fan_in\", \"fan_out\", \"fan_avg\"}:\n      raise ValueError(\n        \"Invalid `mode` argument: \" 'expected on of {\"fan_in\", \"fan_out\", \"fan_avg\"} ' \"but got\",\n        mode,\n      )\n    distribution = distribution.lower()\n    if distribution not in {\"normal\", \"uniform\"}:\n      raise ValueError(\n        \"Invalid `distribution` argument: \" 'expected one of {\"normal\", \"uniform\"} ' \"but got\",\n        distribution,\n      )\n    self.scale = scale\n    self.mode = mode\n    self.distribution = distribution\n    self.seed = seed\n\n  def __call__(self, shape, dtype=None, partition_info=None):\n    fan_in = shape[-2] if self.fan_in is None else self.fan_in\n    fan_out = shape[-1] if self.fan_out is None else self.fan_out\n\n    scale = self.scale\n    if self.mode == \"fan_in\":\n      scale /= max(1.0, fan_in)\n    elif self.mode == \"fan_out\":\n      scale /= max(1.0, fan_out)\n    else:\n      scale /= max(1.0, float(fan_in + fan_out) / 2)\n    if self.distribution == \"normal\":\n      stddev = np.sqrt(scale) / 0.87962566103423978\n      return K.truncated_normal(shape, 0.0, stddev, dtype=dtype, seed=self.seed)\n    else:\n      limit = np.sqrt(3.0 * scale)\n      return K.random_uniform(shape, -limit, limit, dtype=dtype, seed=self.seed)\n\n  def get_config(self):\n    return {\n      \"scale\": self.scale,\n      \"mode\": self.mode,\n      \"distribution\": self.distribution,\n      \"seed\": self.seed,\n    }\n\n\ndef customized_glorot_uniform(seed=None, fan_in=None, fan_out=None):\n  \"\"\"Glorot uniform initializer, also called Xavier uniform initializer.\n  It draws samples from a uniform distribution within [-limit, limit]\n  where `limit` is `sqrt(6 / (fan_in + fan_out))`\n  where `fan_in` is the number of input units in the weight tensor\n  and `fan_out` is the number of output units in the weight tensor.\n  # Arguments\n      seed: A Python integer. Used to seed the random generator.\n  # Returns\n      An initializer.\"\"\"\n  return VarianceScaling(\n    scale=1.0,\n    mode=\"fan_avg\",\n    distribution=\"uniform\",\n    seed=seed,\n    fan_in=fan_in,\n    fan_out=fan_out,\n  )\n\n\ndef customized_glorot_norm(seed=None, fan_in=None, fan_out=None):\n  \"\"\"Glorot norm initializer, also called Xavier uniform initializer.\n  It draws samples from a uniform distribution within [-limit, limit]\n  where `limit` is `sqrt(6 / (fan_in + fan_out))`\n  where `fan_in` is the number of input units in the weight tensor\n  and `fan_out` is the number of output units in the weight tensor.\n  # Arguments\n      seed: A Python integer. Used to seed the random generator.\n  # Returns\n      An initializer.\"\"\"\n  return VarianceScaling(\n    scale=1.0,\n    mode=\"fan_avg\",\n    distribution=\"normal\",\n    seed=seed,\n    fan_in=fan_in,\n    fan_out=fan_out,\n  )\n"
  },
  {
    "path": "pushservice/src/main/python/models/libs/light_ranking_metrics.py",
    "content": "from functools import partial\n\nfrom twitter.cortex.ml.embeddings.deepbird.grouped_metrics.configuration import (\n  GroupedMetricsConfiguration,\n)\nfrom twitter.cortex.ml.embeddings.deepbird.grouped_metrics.helpers import (\n  extract_prediction_from_prediction_record,\n)\n\n\n# checkstyle: noqa\n\n\ndef score_loss_at_n(labels, predictions, lightN):\n  \"\"\"\n  Compute the absolute ScoreLoss ranking metric\n  Args:\n    labels (list)     : A list of label values       (HeavyRanking Reference)\n    predictions (list): A list of prediction values  (LightRanking Predictions)\n    lightN (int): size of the list at which of Initial candidates to compute ScoreLoss. (LightRanking)\n  \"\"\"\n  assert len(labels) == len(predictions)\n\n  if lightN <= 0:\n    return None\n\n  labels_with_predictions = zip(labels, predictions)\n  labels_with_sorted_predictions = sorted(\n    labels_with_predictions, key=lambda x: x[1], reverse=True\n  )[:lightN]\n  labels_top1_light = max([label for label, _ in labels_with_sorted_predictions])\n  labels_top1_heavy = max(labels)\n\n  return labels_top1_heavy - labels_top1_light\n\n\ndef cgr_at_nk(labels, predictions, lightN, heavyK):\n  \"\"\"\n  Compute Cumulative Gain Ratio (CGR) ranking metric\n  Args:\n    labels (list)     : A list of label values       (HeavyRanking Reference)\n    predictions (list): A list of prediction values  (LightRanking Predictions)\n    lightN (int): size of the list at which of Initial candidates to compute CGR. (LightRanking)\n    heavyK (int): size of the list at which of Refined candidates to compute CGR. (HeavyRanking)\n  \"\"\"\n  assert len(labels) == len(predictions)\n\n  if (not lightN) or (not heavyK):\n    out = None\n  elif lightN <= 0 or heavyK <= 0:\n    out = None\n  else:\n\n    labels_with_predictions = zip(labels, predictions)\n    labels_with_sorted_predictions = sorted(\n      labels_with_predictions, key=lambda x: x[1], reverse=True\n    )[:lightN]\n    labels_topN_light = [label for label, _ in labels_with_sorted_predictions]\n\n    if lightN <= heavyK:\n      cg_light = sum(labels_topN_light)\n    else:\n      labels_topK_heavy_from_light = sorted(labels_topN_light, reverse=True)[:heavyK]\n      cg_light = sum(labels_topK_heavy_from_light)\n\n    ideal_ordering = sorted(labels, reverse=True)\n    cg_heavy = sum(ideal_ordering[: min(lightN, heavyK)])\n\n    out = 0.0\n    if cg_heavy != 0:\n      out = max(cg_light / cg_heavy, 0)\n\n  return out\n\n\ndef _get_weight(w, atK):\n  if not w:\n    return 1.0\n  elif len(w) <= atK:\n    return 0.0\n  else:\n    return w[atK]\n\n\ndef recall_at_nk(labels, predictions, n=None, k=None, w=None):\n  \"\"\"\n  Recall at N-K ranking metric\n  Args:\n    labels (list): A list of label values\n    predictions (list): A list of prediction values\n    n (int): size of the list at which of predictions to compute recall. (Light Ranking Predictions)\n             The default is None in which case the length of the provided predictions is used as L\n    k (int): size of the list at which of labels to compute recall. (Heavy Ranking Predictions)\n             The default is None in which case the length of the provided labels is used as L\n    w (list): weight vector sorted by labels\n  \"\"\"\n  assert len(labels) == len(predictions)\n\n  if not any(labels):\n    out = None\n  else:\n\n    safe_n = len(predictions) if not n else min(len(predictions), n)\n    safe_k = len(labels) if not k else min(len(labels), k)\n\n    labels_with_predictions = zip(labels, predictions)\n    sorted_labels_with_predictions = sorted(\n      labels_with_predictions, key=lambda x: x[0], reverse=True\n    )\n\n    order_sorted_labels_predictions = zip(range(len(labels)), *zip(*sorted_labels_with_predictions))\n\n    order_with_predictions = [\n      (order, pred) for order, label, pred in order_sorted_labels_predictions\n    ]\n    order_with_sorted_predictions = sorted(order_with_predictions, key=lambda x: x[1], reverse=True)\n\n    pred_sorted_order_at_n = [order for order, _ in order_with_sorted_predictions][:safe_n]\n\n    intersection_weight = [\n      _get_weight(w, order) if order < safe_k else 0 for order in pred_sorted_order_at_n\n    ]\n\n    intersection_score = sum(intersection_weight)\n    full_score = sum(w) if w else float(safe_k)\n\n    out = 0.0\n    if full_score != 0:\n      out = intersection_score / full_score\n\n  return out\n\n\nclass ExpectedLossGroupedMetricsConfiguration(GroupedMetricsConfiguration):\n  \"\"\"\n  This is the Expected Loss Grouped metric computation configuration.\n  \"\"\"\n\n  def __init__(self, lightNs=[]):\n    \"\"\"\n    Args:\n      lightNs (list): size of the list at which of Initial candidates to compute Expected Loss. (LightRanking)\n    \"\"\"\n    self.lightNs = lightNs\n\n  @property\n  def name(self):\n    return \"ExpectedLoss\"\n\n  @property\n  def metrics_dict(self):\n    metrics_to_compute = {}\n    for lightN in self.lightNs:\n      metric_name = \"ExpectedLoss_atLight_\" + str(lightN)\n      metrics_to_compute[metric_name] = partial(score_loss_at_n, lightN=lightN)\n    return metrics_to_compute\n\n  def extract_label(self, prec, drec, drec_label):\n    return drec_label\n\n  def extract_prediction(self, prec, drec, drec_label):\n    return extract_prediction_from_prediction_record(prec)\n\n\nclass CGRGroupedMetricsConfiguration(GroupedMetricsConfiguration):\n  \"\"\"\n  This is the Cumulative Gain Ratio (CGR) Grouped metric computation configuration.\n  CGR at the max length of each session is the default.\n  CGR at additional positions can be computed by specifying a list of 'n's and 'k's\n  \"\"\"\n\n  def __init__(self, lightNs=[], heavyKs=[]):\n    \"\"\"\n    Args:\n      lightNs (list): size of the list at which of Initial candidates to compute CGR. (LightRanking)\n      heavyK (int):   size of the list at which of Refined candidates to compute CGR. (HeavyRanking)\n    \"\"\"\n    self.lightNs = lightNs\n    self.heavyKs = heavyKs\n\n  @property\n  def name(self):\n    return \"cgr\"\n\n  @property\n  def metrics_dict(self):\n    metrics_to_compute = {}\n    for lightN in self.lightNs:\n      for heavyK in self.heavyKs:\n        metric_name = \"cgr_atLight_\" + str(lightN) + \"_atHeavy_\" + str(heavyK)\n        metrics_to_compute[metric_name] = partial(cgr_at_nk, lightN=lightN, heavyK=heavyK)\n    return metrics_to_compute\n\n  def extract_label(self, prec, drec, drec_label):\n    return drec_label\n\n  def extract_prediction(self, prec, drec, drec_label):\n    return extract_prediction_from_prediction_record(prec)\n\n\nclass RecallGroupedMetricsConfiguration(GroupedMetricsConfiguration):\n  \"\"\"\n  This is the Recall Grouped metric computation configuration.\n  Recall at the max length of each session is the default.\n  Recall at additional positions can be computed by specifying a list of 'n's and 'k's\n  \"\"\"\n\n  def __init__(self, n=[], k=[], w=[]):\n    \"\"\"\n    Args:\n      n (list): A list of ints. List of prediction rank thresholds (for light)\n      k (list): A list of ints. List of label rank thresholds (for heavy)\n    \"\"\"\n    self.predN = n\n    self.labelK = k\n    self.weight = w\n\n  @property\n  def name(self):\n    return \"group_recall\"\n\n  @property\n  def metrics_dict(self):\n    metrics_to_compute = {\"group_recall_unweighted\": recall_at_nk}\n    if not self.weight:\n      metrics_to_compute[\"group_recall_weighted\"] = partial(recall_at_nk, w=self.weight)\n\n    if self.predN and self.labelK:\n      for n in self.predN:\n        for k in self.labelK:\n          if n >= k:\n            metrics_to_compute[\n              \"group_recall_unweighted_at_L\" + str(n) + \"_at_H\" + str(k)\n            ] = partial(recall_at_nk, n=n, k=k)\n            if self.weight:\n              metrics_to_compute[\n                \"group_recall_weighted_at_L\" + str(n) + \"_at_H\" + str(k)\n              ] = partial(recall_at_nk, n=n, k=k, w=self.weight)\n\n    if self.labelK and not self.predN:\n      for k in self.labelK:\n        metrics_to_compute[\"group_recall_unweighted_at_full_at_H\" + str(k)] = partial(\n          recall_at_nk, k=k\n        )\n        if self.weight:\n          metrics_to_compute[\"group_recall_weighted_at_full_at_H\" + str(k)] = partial(\n            recall_at_nk, k=k, w=self.weight\n          )\n    return metrics_to_compute\n\n  def extract_label(self, prec, drec, drec_label):\n    return drec_label\n\n  def extract_prediction(self, prec, drec, drec_label):\n    return extract_prediction_from_prediction_record(prec)\n"
  },
  {
    "path": "pushservice/src/main/python/models/libs/metric_fn_utils.py",
    "content": "\"\"\"\nUtilties for constructing a metric_fn for magic recs.\n\"\"\"\n\nfrom twml.contrib.metrics.metrics import (\n  get_dual_binary_tasks_metric_fn,\n  get_numeric_metric_fn,\n  get_partial_multi_binary_class_metric_fn,\n  get_single_binary_task_metric_fn,\n)\n\nfrom .model_utils import generate_disliked_mask\n\nimport tensorflow.compat.v1 as tf\n\n\nMETRIC_BOOK = {\n  \"OONC\": [\"OONC\"],\n  \"OONC_Engagement\": [\"OONC\", \"Engagement\"],\n  \"Sent\": [\"Sent\"],\n  \"HeavyRankPosition\": [\"HeavyRankPosition\"],\n  \"HeavyRankProbability\": [\"HeavyRankProbability\"],\n}\n\nUSER_AGE_FEATURE_NAME = \"accountAge\"\nNEW_USER_AGE_CUTOFF = 0\n\n\ndef remove_padding_and_flatten(tensor, valid_batch_size):\n  \"\"\"Remove the padding of the input padded tensor given the valid batch size tensor,\n    then flatten the output with respect to the first dimension.\n  Args:\n    tensor: A tensor of size [META_BATCH_SIZE, BATCH_SIZE, FEATURE_DIM].\n    valid_batch_size: A tensor of size [META_BATCH_SIZE], with each element indicating\n      the effective batch size of the BATCH_SIZE dimension.\n\n  Returns:\n    A tesnor of size [tf.reduce_sum(valid_batch_size), FEATURE_DIM].\n  \"\"\"\n  unpadded_ragged_tensor = tf.RaggedTensor.from_tensor(tensor=tensor, lengths=valid_batch_size)\n\n  return unpadded_ragged_tensor.flat_values\n\n\ndef safe_mask(values, mask):\n  \"\"\"Mask values if possible.\n\n  Boolean mask inputed values if and only if values is a tensor of the same dimension as mask (or can be broadcasted to that dimension).\n\n  Args:\n      values (Any or Tensor): Input tensor to mask. Dim 0 should be size N.\n      mask (boolean tensor): A boolean tensor of size N.\n\n  Returns Values or Values masked.\n  \"\"\"\n  if values is None:\n    return values\n  if not tf.is_tensor(values):\n    return values\n  values_shape = values.get_shape()\n  if not values_shape or len(values_shape) == 0:\n    return values\n  if not mask.get_shape().is_compatible_with(values_shape[0]):\n    return values\n  return tf.boolean_mask(values, mask)\n\n\ndef add_new_user_metrics(metric_fn):\n  \"\"\"Will stratify the metric_fn by adding new user metrics.\n\n  Given an input metric_fn, double every metric: One will be the orignal and the other will only include those for new users.\n\n  Args:\n      metric_fn (python function): Base twml metric_fn.\n\n  Returns a metric_fn with new user metrics included.\n  \"\"\"\n\n  def metric_fn_with_new_users(graph_output, labels, weights):\n    if USER_AGE_FEATURE_NAME not in graph_output:\n      raise ValueError(\n        \"In order to get metrics stratified by user age, {name} feature should be added to model graph output. However, only the following output keys were found: {keys}.\".format(\n          name=USER_AGE_FEATURE_NAME, keys=graph_output.keys()\n        )\n      )\n\n    metric_ops = metric_fn(graph_output, labels, weights)\n\n    is_new = tf.reshape(\n      tf.math.less_equal(\n        tf.cast(graph_output[USER_AGE_FEATURE_NAME], tf.int64),\n        tf.cast(NEW_USER_AGE_CUTOFF, tf.int64),\n      ),\n      [-1],\n    )\n\n    labels = safe_mask(labels, is_new)\n    weights = safe_mask(weights, is_new)\n    graph_output = {key: safe_mask(values, is_new) for key, values in graph_output.items()}\n\n    new_user_metric_ops = metric_fn(graph_output, labels, weights)\n    new_user_metric_ops = {name + \"_new_users\": ops for name, ops in new_user_metric_ops.items()}\n    metric_ops.update(new_user_metric_ops)\n    return metric_ops\n\n  return metric_fn_with_new_users\n\n\ndef get_meta_learn_single_binary_task_metric_fn(\n  metrics, classnames, top_k=(5, 5, 5), use_top_k=False\n):\n  \"\"\"Wrapper function to use the metric_fn with meta learning evaluation scheme.\n\n  Args:\n    metrics: A list of string representing metric names.\n    classnames: A list of string repsenting class names, In case of multiple binary class models,\n      the names for each class or label.\n    top_k: A tuple of int to specify top K metrics.\n    use_top_k: A boolean value indicating of top K of metrics is used.\n\n  Returns:\n    A customized metric_fn function.\n  \"\"\"\n\n  def get_eval_metric_ops(graph_output, labels, weights):\n    \"\"\"The op func of the eval_metrics. Comparing with normal version,\n      the difference is we flatten the output, label, and weights.\n\n    Args:\n      graph_output: A dict of tensors.\n      labels: A tensor of int32 be the value of either 0 or 1.\n      weights: A tensor of float32 to indicate the per record weight.\n\n    Returns:\n      A dict of metric names and values.\n    \"\"\"\n    metric_op_weighted = get_partial_multi_binary_class_metric_fn(\n      metrics, predcols=0, classes=classnames\n    )\n    classnames_unweighted = [\"unweighted_\" + classname for classname in classnames]\n    metric_op_unweighted = get_partial_multi_binary_class_metric_fn(\n      metrics, predcols=0, classes=classnames_unweighted\n    )\n\n    valid_batch_size = graph_output[\"valid_batch_size\"]\n    graph_output[\"output\"] = remove_padding_and_flatten(graph_output[\"output\"], valid_batch_size)\n    labels = remove_padding_and_flatten(labels, valid_batch_size)\n    weights = remove_padding_and_flatten(weights, valid_batch_size)\n\n    tf.ensure_shape(graph_output[\"output\"], [None, 1])\n    tf.ensure_shape(labels, [None, 1])\n    tf.ensure_shape(weights, [None, 1])\n\n    metrics_weighted = metric_op_weighted(graph_output, labels, weights)\n    metrics_unweighted = metric_op_unweighted(graph_output, labels, None)\n    metrics_weighted.update(metrics_unweighted)\n\n    if use_top_k:\n      metric_op_numeric = get_numeric_metric_fn(metrics=None, topK=top_k, predcol=0, labelcol=1)\n      metrics_numeric = metric_op_numeric(graph_output, labels, weights)\n      metrics_weighted.update(metrics_numeric)\n    return metrics_weighted\n\n  return get_eval_metric_ops\n\n\ndef get_meta_learn_dual_binary_tasks_metric_fn(\n  metrics, classnames, top_k=(5, 5, 5), use_top_k=False\n):\n  \"\"\"Wrapper function to use the metric_fn with meta learning evaluation scheme.\n\n  Args:\n    metrics: A list of string representing metric names.\n    classnames: A list of string repsenting class names, In case of multiple binary class models,\n      the names for each class or label.\n    top_k: A tuple of int to specify top K metrics.\n    use_top_k: A boolean value indicating of top K of metrics is used.\n\n  Returns:\n    A customized metric_fn function.\n  \"\"\"\n\n  def get_eval_metric_ops(graph_output, labels, weights):\n    \"\"\"The op func of the eval_metrics. Comparing with normal version,\n      the difference is we flatten the output, label, and weights.\n\n    Args:\n      graph_output: A dict of tensors.\n      labels: A tensor of int32 be the value of either 0 or 1.\n      weights: A tensor of float32 to indicate the per record weight.\n\n    Returns:\n      A dict of metric names and values.\n    \"\"\"\n    metric_op_weighted = get_partial_multi_binary_class_metric_fn(\n      metrics, predcols=[0, 1], classes=classnames\n    )\n    classnames_unweighted = [\"unweighted_\" + classname for classname in classnames]\n    metric_op_unweighted = get_partial_multi_binary_class_metric_fn(\n      metrics, predcols=[0, 1], classes=classnames_unweighted\n    )\n\n    valid_batch_size = graph_output[\"valid_batch_size\"]\n    graph_output[\"output\"] = remove_padding_and_flatten(graph_output[\"output\"], valid_batch_size)\n    labels = remove_padding_and_flatten(labels, valid_batch_size)\n    weights = remove_padding_and_flatten(weights, valid_batch_size)\n\n    tf.ensure_shape(graph_output[\"output\"], [None, 2])\n    tf.ensure_shape(labels, [None, 2])\n    tf.ensure_shape(weights, [None, 1])\n\n    metrics_weighted = metric_op_weighted(graph_output, labels, weights)\n    metrics_unweighted = metric_op_unweighted(graph_output, labels, None)\n    metrics_weighted.update(metrics_unweighted)\n\n    if use_top_k:\n      metric_op_numeric = get_numeric_metric_fn(metrics=None, topK=top_k, predcol=2, labelcol=2)\n      metrics_numeric = metric_op_numeric(graph_output, labels, weights)\n      metrics_weighted.update(metrics_numeric)\n    return metrics_weighted\n\n  return get_eval_metric_ops\n\n\ndef get_metric_fn(task_name, use_stratify_metrics, use_meta_batch=False):\n  \"\"\"Will retrieve the metric_fn for magic recs.\n\n  Args:\n    task_name (string): Which task is being used for this model.\n    use_stratify_metrics (boolean): Should we add stratified metrics (new user metrics).\n    use_meta_batch (boolean): If the output/label/weights are passed in 3D shape instead of\n    2D shape.\n\n  Returns:\n    A metric_fn function to pass in twml Trainer.\n  \"\"\"\n  if task_name not in METRIC_BOOK:\n    raise ValueError(\n      \"Task name of {task_name} not recognized. Unable to retrieve metrics.\".format(\n        task_name=task_name\n      )\n    )\n  class_names = METRIC_BOOK[task_name]\n  if use_meta_batch:\n    get_n_binary_task_metric_fn = (\n      get_meta_learn_single_binary_task_metric_fn\n      if len(class_names) == 1\n      else get_meta_learn_dual_binary_tasks_metric_fn\n    )\n  else:\n    get_n_binary_task_metric_fn = (\n      get_single_binary_task_metric_fn if len(class_names) == 1 else get_dual_binary_tasks_metric_fn\n    )\n\n  metric_fn = get_n_binary_task_metric_fn(metrics=None, classnames=METRIC_BOOK[task_name])\n\n  if use_stratify_metrics:\n    metric_fn = add_new_user_metrics(metric_fn)\n\n  return metric_fn\n\n\ndef flip_disliked_labels(metric_fn):\n  \"\"\"This function returns an adapted metric_fn which flips the labels of the OONCed evaluation data to 0 if it is disliked.\n  Args:\n    metric_fn: A metric_fn function to pass in twml Trainer.\n\n  Returns:\n    _adapted_metric_fn: A customized metric_fn function with disliked OONC labels flipped.\n  \"\"\"\n\n  def _adapted_metric_fn(graph_output, labels, weights):\n    \"\"\"A customized metric_fn function with disliked OONC labels flipped.\n\n    Args:\n      graph_output: A dict of tensors.\n      labels: labels of training samples, which is a 2D tensor of shape batch_size x 3: [OONCs, engagements, dislikes]\n      weights: A tensor of float32 to indicate the per record weight.\n\n    Returns:\n      A dict of metric names and values.\n    \"\"\"\n    # We want to multiply the label of the observation by 0 only when it is disliked\n    disliked_mask = generate_disliked_mask(labels)\n\n    # Extract OONC and engagement labels only.\n    labels = tf.reshape(labels[:, 0:2], shape=[-1, 2])\n\n    # Labels will be set to 0 if it is disliked.\n    adapted_labels = labels * tf.cast(tf.logical_not(disliked_mask), dtype=labels.dtype)\n\n    return metric_fn(graph_output, adapted_labels, weights)\n\n  return _adapted_metric_fn\n"
  },
  {
    "path": "pushservice/src/main/python/models/libs/model_args.py",
    "content": "from twml.trainers import DataRecordTrainer\n\n\n# checkstyle: noqa\n\n\ndef get_arg_parser():\n  parser = DataRecordTrainer.add_parser_arguments()\n\n  parser.add_argument(\n    \"--input_size_bits\",\n    type=int,\n    default=18,\n    help=\"number of bits allocated to the input size\",\n  )\n  parser.add_argument(\n    \"--model_trainer_name\",\n    default=\"magic_recs_mlp_calibration_MTL_OONC_Engagement\",\n    type=str,\n    help=\"specify the model trainer name.\",\n  )\n\n  parser.add_argument(\n    \"--model_type\",\n    default=\"deepnorm_gbdt_inputdrop2_rescale\",\n    type=str,\n    help=\"specify the model type to use.\",\n  )\n  parser.add_argument(\n    \"--feat_config_type\",\n    default=\"get_feature_config_with_sparse_continuous\",\n    type=str,\n    help=\"specify the feature configure function to use.\",\n  )\n\n  parser.add_argument(\n    \"--directly_export_best\",\n    default=False,\n    action=\"store_true\",\n    help=\"whether to directly_export best_checkpoint\",\n  )\n\n  parser.add_argument(\n    \"--warm_start_base_dir\",\n    default=\"none\",\n    type=str,\n    help=\"latest ckpt in this folder will be used to \",\n  )\n\n  parser.add_argument(\n    \"--feature_list\",\n    default=\"none\",\n    type=str,\n    help=\"Which features to use for training\",\n  )\n  parser.add_argument(\n    \"--warm_start_from\", default=None, type=str, help=\"model dir to warm start from\"\n  )\n\n  parser.add_argument(\n    \"--momentum\", default=0.99999, type=float, help=\"Momentum term for batch normalization\"\n  )\n  parser.add_argument(\n    \"--dropout\",\n    default=0.2,\n    type=float,\n    help=\"input_dropout_rate to rescale output by (1 - input_dropout_rate)\",\n  )\n  parser.add_argument(\n    \"--out_layer_1_size\", default=256, type=int, help=\"Size of MLP_branch layer 1\"\n  )\n  parser.add_argument(\n    \"--out_layer_2_size\", default=128, type=int, help=\"Size of MLP_branch layer 2\"\n  )\n  parser.add_argument(\"--out_layer_3_size\", default=64, type=int, help=\"Size of MLP_branch layer 3\")\n  parser.add_argument(\n    \"--sparse_embedding_size\", default=50, type=int, help=\"Dimensionality of sparse embedding layer\"\n  )\n  parser.add_argument(\n    \"--dense_embedding_size\", default=128, type=int, help=\"Dimensionality of dense embedding layer\"\n  )\n\n  parser.add_argument(\n    \"--use_uam_label\",\n    default=False,\n    type=str,\n    help=\"Whether to use uam_label or not\",\n  )\n\n  parser.add_argument(\n    \"--task_name\",\n    default=\"OONC_Engagement\",\n    type=str,\n    help=\"specify the task name to use: OONC or OONC_Engagement.\",\n  )\n  parser.add_argument(\n    \"--init_weight\",\n    default=0.9,\n    type=float,\n    help=\"Initial OONC Task Weight MTL: OONC+Engagement.\",\n  )\n  parser.add_argument(\n    \"--use_engagement_weight\",\n    default=False,\n    action=\"store_true\",\n    help=\"whether to use engagement weight for base model.\",\n  )\n  parser.add_argument(\n    \"--mtl_num_extra_layers\",\n    type=int,\n    default=1,\n    help=\"Number of Hidden Layers for each TaskBranch.\",\n  )\n  parser.add_argument(\n    \"--mtl_neuron_scale\", type=int, default=4, help=\"Scaling Factor of Neurons in MTL Extra Layers.\"\n  )\n  parser.add_argument(\n    \"--use_oonc_score\",\n    default=False,\n    action=\"store_true\",\n    help=\"whether to use oonc score only or combined score.\",\n  )\n  parser.add_argument(\n    \"--use_stratified_metrics\",\n    default=False,\n    action=\"store_true\",\n    help=\"Use stratified metrics: Break out new-user metrics.\",\n  )\n  parser.add_argument(\n    \"--run_group_metrics\",\n    default=False,\n    action=\"store_true\",\n    help=\"Will run evaluation metrics grouped by user.\",\n  )\n  parser.add_argument(\n    \"--use_full_scope\",\n    default=False,\n    action=\"store_true\",\n    help=\"Will add extra scope and naming to graph.\",\n  )\n  parser.add_argument(\n    \"--trainable_regexes\",\n    default=None,\n    nargs=\"*\",\n    help=\"The union of variables specified by the list of regexes will be considered trainable.\",\n  )\n  parser.add_argument(\n    \"--fine_tuning.ckpt_to_initialize_from\",\n    dest=\"fine_tuning_ckpt_to_initialize_from\",\n    type=str,\n    default=None,\n    help=\"Checkpoint path from which to warm start. Indicates the pre-trained model.\",\n  )\n  parser.add_argument(\n    \"--fine_tuning.warm_start_scope_regex\",\n    dest=\"fine_tuning_warm_start_scope_regex\",\n    type=str,\n    default=None,\n    help=\"All variables matching this will be restored.\",\n  )\n\n  return parser\n\n\ndef get_params(args=None):\n  parser = get_arg_parser()\n  if args is None:\n    return parser.parse_args()\n  else:\n    return parser.parse_args(args)\n\n\ndef get_arg_parser_light_ranking():\n  parser = get_arg_parser()\n\n  parser.add_argument(\n    \"--use_record_weight\",\n    default=False,\n    action=\"store_true\",\n    help=\"whether to use record weight for base model.\",\n  )\n  parser.add_argument(\n    \"--min_record_weight\", default=0.0, type=float, help=\"Minimum record weight to use.\"\n  )\n  parser.add_argument(\n    \"--smooth_weight\", default=0.0, type=float, help=\"Factor to smooth Rank Position Weight.\"\n  )\n\n  parser.add_argument(\n    \"--num_mlp_layers\", type=int, default=3, help=\"Number of Hidden Layers for MLP model.\"\n  )\n  parser.add_argument(\n    \"--mlp_neuron_scale\", type=int, default=4, help=\"Scaling Factor of Neurons in MLP Layers.\"\n  )\n  parser.add_argument(\n    \"--run_light_ranking_group_metrics\",\n    default=False,\n    action=\"store_true\",\n    help=\"Will run evaluation metrics grouped by user for Light Ranking.\",\n  )\n  parser.add_argument(\n    \"--use_missing_sub_branch\",\n    default=False,\n    action=\"store_true\",\n    help=\"Whether to use missing value sub-branch for Light Ranking.\",\n  )\n  parser.add_argument(\n    \"--use_gbdt_features\",\n    default=False,\n    action=\"store_true\",\n    help=\"Whether to use GBDT features for Light Ranking.\",\n  )\n  parser.add_argument(\n    \"--run_light_ranking_group_metrics_in_bq\",\n    default=False,\n    action=\"store_true\",\n    help=\"Whether to get_predictions for Light Ranking to compute group metrics in BigQuery.\",\n  )\n  parser.add_argument(\n    \"--pred_file_path\",\n    default=None,\n    type=str,\n    help=\"path\",\n  )\n  parser.add_argument(\n    \"--pred_file_name\",\n    default=None,\n    type=str,\n    help=\"path\",\n  )\n  return parser\n"
  },
  {
    "path": "pushservice/src/main/python/models/libs/model_utils.py",
    "content": "import sys\n\nimport twml\n\nfrom .initializer import customized_glorot_uniform\n\nimport tensorflow.compat.v1 as tf\nimport yaml\n\n\n# checkstyle: noqa\n\n\ndef read_config(whitelist_yaml_file):\n  with tf.gfile.FastGFile(whitelist_yaml_file) as f:\n    try:\n      return yaml.safe_load(f)\n    except yaml.YAMLError as exc:\n      print(exc)\n      sys.exit(1)\n\n\ndef _sparse_feature_fixup(features, input_size_bits):\n  \"\"\"Rebuild a sparse tensor feature so that its dense shape attribute is present.\n\n  Arguments:\n    features (SparseTensor): Sparse feature tensor of shape ``(B, sparse_feature_dim)``.\n    input_size_bits (int): Number of columns in ``log2`` scale. Must be positive.\n\n  Returns:\n    SparseTensor: Rebuilt and non-faulty version of `features`.\"\"\"\n  sparse_feature_dim = tf.constant(2**input_size_bits, dtype=tf.int64)\n  sparse_shape = tf.stack([features.dense_shape[0], sparse_feature_dim])\n  sparse_tf = tf.SparseTensor(features.indices, features.values, sparse_shape)\n  return sparse_tf\n\n\ndef self_atten_dense(input, out_dim, activation=None, use_bias=True, name=None):\n  def safe_concat(base, suffix):\n    \"\"\"Concats variables name components if base is given.\"\"\"\n    if not base:\n      return base\n    return f\"{base}:{suffix}\"\n\n  input_dim = input.shape.as_list()[1]\n\n  sigmoid_out = twml.layers.FullDense(\n    input_dim, dtype=tf.float32, activation=tf.nn.sigmoid, name=safe_concat(name, \"sigmoid_out\")\n  )(input)\n  atten_input = sigmoid_out * input\n  mlp_out = twml.layers.FullDense(\n    out_dim,\n    dtype=tf.float32,\n    activation=activation,\n    use_bias=use_bias,\n    name=safe_concat(name, \"mlp_out\"),\n  )(atten_input)\n  return mlp_out\n\n\ndef get_dense_out(input, out_dim, activation, dense_type):\n  if dense_type == \"full_dense\":\n    out = twml.layers.FullDense(out_dim, dtype=tf.float32, activation=activation)(input)\n  elif dense_type == \"self_atten_dense\":\n    out = self_atten_dense(input, out_dim, activation=activation)\n  return out\n\n\ndef get_input_trans_func(bn_normalized_dense, is_training):\n  gw_normalized_dense = tf.expand_dims(bn_normalized_dense, -1)\n  group_num = bn_normalized_dense.shape.as_list()[1]\n\n  gw_normalized_dense = GroupWiseTrans(group_num, 1, 8, name=\"groupwise_1\", activation=tf.tanh)(\n    gw_normalized_dense\n  )\n  gw_normalized_dense = GroupWiseTrans(group_num, 8, 4, name=\"groupwise_2\", activation=tf.tanh)(\n    gw_normalized_dense\n  )\n  gw_normalized_dense = GroupWiseTrans(group_num, 4, 1, name=\"groupwise_3\", activation=tf.tanh)(\n    gw_normalized_dense\n  )\n\n  gw_normalized_dense = tf.squeeze(gw_normalized_dense, [-1])\n\n  bn_gw_normalized_dense = tf.layers.batch_normalization(\n    gw_normalized_dense,\n    training=is_training,\n    renorm_momentum=0.9999,\n    momentum=0.9999,\n    renorm=is_training,\n    trainable=True,\n  )\n\n  return bn_gw_normalized_dense\n\n\ndef tensor_dropout(\n  input_tensor,\n  rate,\n  is_training,\n  sparse_tensor=None,\n):\n  \"\"\"\n  Implements dropout layer for both dense and sparse input_tensor\n\n  Arguments:\n    input_tensor:\n      B x D dense tensor, or a sparse tensor\n    rate (float32):\n      dropout rate\n    is_training (bool):\n      training stage or not.\n    sparse_tensor (bool):\n      whether the input_tensor is sparse tensor or not. Default to be None, this value has to be passed explicitly.\n    rescale_sparse_dropout (bool):\n      Do we need to do rescaling or not.\n  Returns:\n    tensor dropped out\"\"\"\n  if sparse_tensor == True:\n    if is_training:\n      with tf.variable_scope(\"sparse_dropout\"):\n        values = input_tensor.values\n        keep_mask = tf.keras.backend.random_binomial(\n          tf.shape(values), p=1 - rate, dtype=tf.float32, seed=None\n        )\n        keep_mask.set_shape([None])\n        keep_mask = tf.cast(keep_mask, tf.bool)\n\n        keep_indices = tf.boolean_mask(input_tensor.indices, keep_mask, axis=0)\n        keep_values = tf.boolean_mask(values, keep_mask, axis=0)\n\n        dropped_tensor = tf.SparseTensor(keep_indices, keep_values, input_tensor.dense_shape)\n        return dropped_tensor\n    else:\n      return input_tensor\n  elif sparse_tensor == False:\n    return tf.layers.dropout(input_tensor, rate=rate, training=is_training)\n\n\ndef adaptive_transformation(bn_normalized_dense, is_training, func_type=\"default\"):\n  assert func_type in [\n    \"default\",\n    \"tiny\",\n  ], f\"fun_type can only be one of default and tiny, but get {func_type}\"\n\n  gw_normalized_dense = tf.expand_dims(bn_normalized_dense, -1)\n  group_num = bn_normalized_dense.shape.as_list()[1]\n\n  if func_type == \"default\":\n    gw_normalized_dense = FastGroupWiseTrans(\n      group_num, 1, 8, name=\"groupwise_1\", activation=tf.tanh, init_multiplier=8\n    )(gw_normalized_dense)\n\n    gw_normalized_dense = FastGroupWiseTrans(\n      group_num, 8, 4, name=\"groupwise_2\", activation=tf.tanh, init_multiplier=8\n    )(gw_normalized_dense)\n\n    gw_normalized_dense = FastGroupWiseTrans(\n      group_num, 4, 1, name=\"groupwise_3\", activation=tf.tanh, init_multiplier=8\n    )(gw_normalized_dense)\n  elif func_type == \"tiny\":\n    gw_normalized_dense = FastGroupWiseTrans(\n      group_num, 1, 2, name=\"groupwise_1\", activation=tf.tanh, init_multiplier=8\n    )(gw_normalized_dense)\n\n    gw_normalized_dense = FastGroupWiseTrans(\n      group_num, 2, 1, name=\"groupwise_2\", activation=tf.tanh, init_multiplier=8\n    )(gw_normalized_dense)\n\n    gw_normalized_dense = FastGroupWiseTrans(\n      group_num, 1, 1, name=\"groupwise_3\", activation=tf.tanh, init_multiplier=8\n    )(gw_normalized_dense)\n\n  gw_normalized_dense = tf.squeeze(gw_normalized_dense, [-1])\n  bn_gw_normalized_dense = tf.layers.batch_normalization(\n    gw_normalized_dense,\n    training=is_training,\n    renorm_momentum=0.9999,\n    momentum=0.9999,\n    renorm=is_training,\n    trainable=True,\n  )\n\n  return bn_gw_normalized_dense\n\n\nclass FastGroupWiseTrans(object):\n  \"\"\"\n  used to apply group-wise fully connected layers to the input.\n  it applies a tiny, unique MLP to each individual feature.\"\"\"\n\n  def __init__(self, group_num, input_dim, out_dim, name, activation=None, init_multiplier=1):\n    self.group_num = group_num\n    self.input_dim = input_dim\n    self.out_dim = out_dim\n    self.activation = activation\n    self.init_multiplier = init_multiplier\n\n    self.w = tf.get_variable(\n      name + \"_group_weight\",\n      [1, group_num, input_dim, out_dim],\n      initializer=customized_glorot_uniform(\n        fan_in=input_dim * init_multiplier, fan_out=out_dim * init_multiplier\n      ),\n      trainable=True,\n    )\n    self.b = tf.get_variable(\n      name + \"_group_bias\",\n      [1, group_num, out_dim],\n      initializer=tf.constant_initializer(0.0),\n      trainable=True,\n    )\n\n  def __call__(self, input_tensor):\n    \"\"\"\n    input_tensor: batch_size x group_num x input_dim\n    output_tensor:  batch_size x group_num x out_dim\"\"\"\n    input_tensor_expand = tf.expand_dims(input_tensor, axis=-1)\n\n    output_tensor = tf.add(\n      tf.reduce_sum(tf.multiply(input_tensor_expand, self.w), axis=-2, keepdims=False),\n      self.b,\n    )\n\n    if self.activation is not None:\n      output_tensor = self.activation(output_tensor)\n    return output_tensor\n\n\nclass GroupWiseTrans(object):\n  \"\"\"\n  Used to apply group fully connected layers to the input.\n  \"\"\"\n\n  def __init__(self, group_num, input_dim, out_dim, name, activation=None):\n    self.group_num = group_num\n    self.input_dim = input_dim\n    self.out_dim = out_dim\n    self.activation = activation\n\n    w_list, b_list = [], []\n    for idx in range(out_dim):\n      this_w = tf.get_variable(\n        name + f\"_group_weight_{idx}\",\n        [1, group_num, input_dim],\n        initializer=tf.keras.initializers.glorot_uniform(),\n        trainable=True,\n      )\n      this_b = tf.get_variable(\n        name + f\"_group_bias_{idx}\",\n        [1, group_num, 1],\n        initializer=tf.constant_initializer(0.0),\n        trainable=True,\n      )\n      w_list.append(this_w)\n      b_list.append(this_b)\n    self.w_list = w_list\n    self.b_list = b_list\n\n  def __call__(self, input_tensor):\n    \"\"\"\n    input_tensor: batch_size x group_num x input_dim\n    output_tensor: batch_size x group_num x out_dim\n    \"\"\"\n    out_tensor_list = []\n    for idx in range(self.out_dim):\n      this_res = (\n        tf.reduce_sum(input_tensor * self.w_list[idx], axis=-1, keepdims=True) + self.b_list[idx]\n      )\n      out_tensor_list.append(this_res)\n    output_tensor = tf.concat(out_tensor_list, axis=-1)\n\n    if self.activation is not None:\n      output_tensor = self.activation(output_tensor)\n    return output_tensor\n\n\ndef add_scalar_summary(var, name, name_scope=\"hist_dense_feature/\"):\n  with tf.name_scope(\"summaries/\"):\n    with tf.name_scope(name_scope):\n      tf.summary.scalar(name, var)\n\n\ndef add_histogram_summary(var, name, name_scope=\"hist_dense_feature/\"):\n  with tf.name_scope(\"summaries/\"):\n    with tf.name_scope(name_scope):\n      tf.summary.histogram(name, tf.reshape(var, [-1]))\n\n\ndef sparse_clip_by_value(sparse_tf, min_val, max_val):\n  new_vals = tf.clip_by_value(sparse_tf.values, min_val, max_val)\n  return tf.SparseTensor(sparse_tf.indices, new_vals, sparse_tf.dense_shape)\n\n\ndef check_numerics_with_msg(tensor, message=\"\", sparse_tensor=False):\n  if sparse_tensor:\n    values = tf.debugging.check_numerics(tensor.values, message=message)\n    return tf.SparseTensor(tensor.indices, values, tensor.dense_shape)\n  else:\n    return tf.debugging.check_numerics(tensor, message=message)\n\n\ndef pad_empty_sparse_tensor(tensor):\n  dummy_tensor = tf.SparseTensor(\n    indices=[[0, 0]],\n    values=[0.00001],\n    dense_shape=tensor.dense_shape,\n  )\n  result = tf.cond(\n    tf.equal(tf.size(tensor.values), 0),\n    lambda: dummy_tensor,\n    lambda: tensor,\n  )\n  return result\n\n\ndef filter_nans_and_infs(tensor, sparse_tensor=False):\n  if sparse_tensor:\n    sparse_values = tensor.values\n    filtered_val = tf.where(\n      tf.logical_or(tf.is_nan(sparse_values), tf.is_inf(sparse_values)),\n      tf.zeros_like(sparse_values),\n      sparse_values,\n    )\n    return tf.SparseTensor(tensor.indices, filtered_val, tensor.dense_shape)\n  else:\n    return tf.where(\n      tf.logical_or(tf.is_nan(tensor), tf.is_inf(tensor)), tf.zeros_like(tensor), tensor\n    )\n\n\ndef generate_disliked_mask(labels):\n  \"\"\"Generate a disliked mask where only samples with dislike labels are set to 1 otherwise set to 0.\n  Args:\n    labels: labels of training samples, which is a 2D tensor of shape batch_size x 3: [OONCs, engagements, dislikes]\n  Returns:\n    1D tensor of shape batch_size x 1: [dislikes (booleans)]\n  \"\"\"\n  return tf.equal(tf.reshape(labels[:, 2], shape=[-1, 1]), 1)\n"
  },
  {
    "path": "pushservice/src/main/python/models/libs/warm_start_utils.py",
    "content": "from collections import OrderedDict\nimport json\nimport os\nfrom os.path import join\n\nfrom twitter.magicpony.common import file_access\nimport twml\n\nfrom .model_utils import read_config\n\nimport numpy as np\nfrom scipy import stats\nimport tensorflow.compat.v1 as tf\n\n\n# checkstyle: noqa\n\n\ndef get_model_type_to_tensors_to_change_axis():\n  model_type_to_tensors_to_change_axis = {\n    \"magic_recs/model/batch_normalization/beta\": ([0], \"continuous\"),\n    \"magic_recs/model/batch_normalization/gamma\": ([0], \"continuous\"),\n    \"magic_recs/model/batch_normalization/moving_mean\": ([0], \"continuous\"),\n    \"magic_recs/model/batch_normalization/moving_stddev\": ([0], \"continuous\"),\n    \"magic_recs/model/batch_normalization/moving_variance\": ([0], \"continuous\"),\n    \"magic_recs/model/batch_normalization/renorm_mean\": ([0], \"continuous\"),\n    \"magic_recs/model/batch_normalization/renorm_stddev\": ([0], \"continuous\"),\n    \"magic_recs/model/logits/EngagementGivenOONC_logits/clem_net_1/block2_4/channel_wise_dense_4/kernel\": (\n      [1],\n      \"all\",\n    ),\n    \"magic_recs/model/logits/OONC_logits/clem_net/block2/channel_wise_dense/kernel\": ([1], \"all\"),\n  }\n\n  return model_type_to_tensors_to_change_axis\n\n\ndef mkdirp(dirname):\n  if not tf.io.gfile.exists(dirname):\n    tf.io.gfile.makedirs(dirname)\n\n\ndef rename_dir(dirname, dst):\n  file_access.hdfs.mv(dirname, dst)\n\n\ndef rmdir(dirname):\n  if tf.io.gfile.exists(dirname):\n    if tf.io.gfile.isdir(dirname):\n      tf.io.gfile.rmtree(dirname)\n    else:\n      tf.io.gfile.remove(dirname)\n\n\ndef get_var_dict(checkpoint_path):\n  checkpoint = tf.train.get_checkpoint_state(checkpoint_path)\n  var_dict = OrderedDict()\n  with tf.Session() as sess:\n    all_var_list = tf.train.list_variables(checkpoint_path)\n    for var_name, _ in all_var_list:\n      # Load the variable\n      var = tf.train.load_variable(checkpoint_path, var_name)\n      var_dict[var_name] = var\n  return var_dict\n\n\ndef get_continunous_mapping_from_feat_list(old_feature_list, new_feature_list):\n  \"\"\"\n  get var_ind for old_feature and corresponding var_ind for new_feature\n  \"\"\"\n  new_var_ind, old_var_ind = [], []\n  for this_new_id, this_new_name in enumerate(new_feature_list):\n    if this_new_name in old_feature_list:\n      this_old_id = old_feature_list.index(this_new_name)\n      new_var_ind.append(this_new_id)\n      old_var_ind.append(this_old_id)\n  return np.asarray(old_var_ind), np.asarray(new_var_ind)\n\n\ndef get_continuous_mapping_from_feat_dict(old_feature_dict, new_feature_dict):\n  \"\"\"\n  get var_ind for old_feature and corresponding var_ind for new_feature\n  \"\"\"\n  old_cont = old_feature_dict[\"continuous\"]\n  old_bin = old_feature_dict[\"binary\"]\n\n  new_cont = new_feature_dict[\"continuous\"]\n  new_bin = new_feature_dict[\"binary\"]\n\n  _dummy_sparse_feat = [f\"sparse_feature_{_idx}\" for _idx in range(100)]\n\n  cont_old_var_ind, cont_new_var_ind = get_continunous_mapping_from_feat_list(old_cont, new_cont)\n\n  all_old_var_ind, all_new_var_ind = get_continunous_mapping_from_feat_list(\n    old_cont + old_bin + _dummy_sparse_feat, new_cont + new_bin + _dummy_sparse_feat\n  )\n\n  _res = {\n    \"continuous\": (cont_old_var_ind, cont_new_var_ind),\n    \"all\": (all_old_var_ind, all_new_var_ind),\n  }\n\n  return _res\n\n\ndef warm_start_from_var_dict(\n  old_ckpt_path,\n  var_ind_dict,\n  output_dir,\n  new_len_var,\n  var_to_change_dict_fn=get_model_type_to_tensors_to_change_axis,\n):\n  \"\"\"\n  Parameters:\n      old_ckpt_path (str): path to the old checkpoint path\n      new_var_ind (array of int): index to overlapping features in new var between old and new feature list.\n      old_var_ind (array of int): index to overlapping features in old var between old and new feature list.\n\n      output_dir (str): dir that used to write modified checkpoint\n      new_len_var ({str:int}): number of feature in the new feature list.\n      var_to_change_dict_fn (dict): A function to get the dictionary of format {var_name: dim_to_change}\n  \"\"\"\n  old_var_dict = get_var_dict(old_ckpt_path)\n\n  ckpt_file_name = os.path.basename(old_ckpt_path)\n  mkdirp(output_dir)\n  output_path = join(output_dir, ckpt_file_name)\n\n  tensors_to_change = var_to_change_dict_fn()\n  tf.compat.v1.reset_default_graph()\n\n  with tf.Session() as sess:\n    var_name_shape_list = tf.train.list_variables(old_ckpt_path)\n    count = 0\n\n    for var_name, var_shape in var_name_shape_list:\n      old_var = old_var_dict[var_name]\n      if var_name in tensors_to_change.keys():\n        _info_tuple = tensors_to_change[var_name]\n        dims_to_remove_from, var_type = _info_tuple\n\n        new_var_ind, old_var_ind = var_ind_dict[var_type]\n\n        this_shape = list(old_var.shape)\n        for this_dim in dims_to_remove_from:\n          this_shape[this_dim] = new_len_var[var_type]\n\n        stddev = np.std(old_var)\n        truncated_norm_generator = stats.truncnorm(-0.5, 0.5, loc=0, scale=stddev)\n        size = np.prod(this_shape)\n        new_var = truncated_norm_generator.rvs(size).reshape(this_shape)\n        new_var = new_var.astype(old_var.dtype)\n\n        new_var = copy_feat_based_on_mapping(\n          new_var, old_var, dims_to_remove_from, new_var_ind, old_var_ind\n        )\n        count = count + 1\n      else:\n        new_var = old_var\n      var = tf.Variable(new_var, name=var_name)\n    assert count == len(tensors_to_change.keys()), \"not all variables are exchanged.\\n\"\n    saver = tf.train.Saver()\n    sess.run(tf.global_variables_initializer())\n    saver.save(sess, output_path)\n  return output_path\n\n\ndef copy_feat_based_on_mapping(new_array, old_array, dims_to_remove_from, new_var_ind, old_var_ind):\n  if dims_to_remove_from == [0, 1]:\n    for this_new_ind, this_old_ind in zip(new_var_ind, old_var_ind):\n      new_array[this_new_ind, new_var_ind] = old_array[this_old_ind, old_var_ind]\n  elif dims_to_remove_from == [0]:\n    new_array[new_var_ind] = old_array[old_var_ind]\n  elif dims_to_remove_from == [1]:\n    new_array[:, new_var_ind] = old_array[:, old_var_ind]\n  else:\n    raise RuntimeError(f\"undefined dims_to_remove_from pattern: ({dims_to_remove_from})\")\n  return new_array\n\n\ndef read_file(filename, decode=False):\n  \"\"\"\n  Reads contents from a file and optionally decodes it.\n\n  Arguments:\n    filename:\n      path to file where the contents will be loaded from.\n      Accepts HDFS and local paths.\n    decode:\n      False or 'json'. When decode='json', contents is decoded\n      with json.loads. When False, contents is returned as is.\n  \"\"\"\n  graph = tf.Graph()\n  with graph.as_default():\n    read = tf.read_file(filename)\n\n  with tf.Session(graph=graph) as sess:\n    contents = sess.run(read)\n    if not isinstance(contents, str):\n      contents = contents.decode()\n\n  if decode == \"json\":\n    contents = json.loads(contents)\n\n  return contents\n\n\ndef read_feat_list_from_disk(file_path):\n  return read_file(file_path, decode=\"json\")\n\n\ndef get_feature_list_for_light_ranking(feature_list_path, data_spec_path):\n  feature_list = read_config(feature_list_path).items()\n  string_feat_list = [f[0] for f in feature_list if f[1] != \"S\"]\n\n  feature_config_builder = twml.contrib.feature_config.FeatureConfigBuilder(\n    data_spec_path=data_spec_path\n  )\n  feature_config_builder = feature_config_builder.extract_feature_group(\n    feature_regexes=string_feat_list,\n    group_name=\"continuous\",\n    default_value=-1,\n    type_filter=[\"CONTINUOUS\"],\n  )\n  feature_config = feature_config_builder.build()\n  feature_list = feature_config_builder._feature_group_extraction_configs[0].feature_map[\n    \"CONTINUOUS\"\n  ]\n  return feature_list\n\n\ndef get_feature_list_for_heavy_ranking(feature_list_path, data_spec_path):\n  feature_list = read_config(feature_list_path).items()\n  string_feat_list = [f[0] for f in feature_list if f[1] != \"S\"]\n\n  feature_config_builder = twml.contrib.feature_config.FeatureConfigBuilder(\n    data_spec_path=data_spec_path\n  )\n  feature_config_builder = feature_config_builder.extract_feature_group(\n    feature_regexes=string_feat_list,\n    group_name=\"continuous\",\n    default_value=-1,\n    type_filter=[\"CONTINUOUS\"],\n  )\n\n  feature_config_builder = feature_config_builder.extract_feature_group(\n    feature_regexes=string_feat_list,\n    group_name=\"binary\",\n    default_value=False,\n    type_filter=[\"BINARY\"],\n  )\n\n  feature_config_builder = feature_config_builder.build()\n\n  continuous_feature_list = feature_config_builder._feature_group_extraction_configs[0].feature_map[\n    \"CONTINUOUS\"\n  ]\n\n  binary_feature_list = feature_config_builder._feature_group_extraction_configs[1].feature_map[\n    \"BINARY\"\n  ]\n  return {\"continuous\": continuous_feature_list, \"binary\": binary_feature_list}\n\n\ndef warm_start_checkpoint(\n  old_best_ckpt_folder,\n  old_feature_list_path,\n  feature_allow_list_path,\n  data_spec_path,\n  output_ckpt_folder,\n  *args,\n):\n  \"\"\"\n  Reads old checkpoint and the old feature list, and create a new ckpt warm started from old ckpt using new features .\n\n  Arguments:\n    old_best_ckpt_folder:\n      path to the best_checkpoint_folder for old model\n    old_feature_list_path:\n      path to the json file that stores the list of continuous features used in old models.\n    feature_allow_list_path:\n      yaml file that contain the feature allow list.\n    data_spec_path:\n      path to the data_spec file\n    output_ckpt_folder:\n      folder that contains the modified ckpt.\n\n  Returns:\n    path to the modified ckpt.\"\"\"\n  old_ckpt_path = tf.train.latest_checkpoint(old_best_ckpt_folder, latest_filename=None)\n\n  new_feature_dict = get_feature_list(feature_allow_list_path, data_spec_path)\n  old_feature_dict = read_feat_list_from_disk(old_feature_list_path)\n\n  var_ind_dict = get_continuous_mapping_from_feat_dict(new_feature_dict, old_feature_dict)\n\n  new_len_var = {\n    \"continuous\": len(new_feature_dict[\"continuous\"]),\n    \"all\": len(new_feature_dict[\"continuous\"] + new_feature_dict[\"binary\"]) + 100,\n  }\n\n  warm_started_ckpt_path = warm_start_from_var_dict(\n    old_ckpt_path,\n    var_ind_dict,\n    output_dir=output_ckpt_folder,\n    new_len_var=new_len_var,\n  )\n\n  return warm_started_ckpt_path\n"
  },
  {
    "path": "pushservice/src/main/python/models/light_ranking/BUILD",
    "content": "#\":mlwf_libs\",\n\npython37_binary(\n    name = \"eval_model\",\n    source = \"eval_model.py\",\n    dependencies = [\n        \":libs\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/light_ranking:eval_model\",\n    ],\n)\n\npython37_binary(\n    name = \"train_model\",\n    source = \"deep_norm.py\",\n    dependencies = [\n        \":libs\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/light_ranking:train_model\",\n    ],\n)\n\npython37_binary(\n    name = \"train_model_local\",\n    source = \"deep_norm.py\",\n    dependencies = [\n        \":libs\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/light_ranking:train_model_local\",\n        \"twml\",\n    ],\n)\n\npython37_binary(\n    name = \"eval_model_local\",\n    source = \"eval_model.py\",\n    dependencies = [\n        \":libs\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/light_ranking:eval_model_local\",\n        \"twml\",\n    ],\n)\n\npython37_binary(\n    name = \"mlwf_model\",\n    source = \"deep_norm.py\",\n    dependencies = [\n        \":mlwf_libs\",\n        \"3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/light_ranking:mlwf_model\",\n    ],\n)\n\npython3_library(\n    name = \"libs\",\n    sources = [\"**/*.py\"],\n    tags = [\"no-mypy\"],\n    dependencies = [\n        \"src/python/twitter/deepbird/projects/magic_recs/libs\",\n        \"src/python/twitter/deepbird/util/data\",\n        \"twml:twml-nodeps\",\n    ],\n)\n\npython3_library(\n    name = \"mlwf_libs\",\n    sources = [\"**/*.py\"],\n    tags = [\"no-mypy\"],\n    dependencies = [\n        \"src/python/twitter/deepbird/projects/magic_recs/libs\",\n        \"twml\",\n    ],\n)\n"
  },
  {
    "path": "pushservice/src/main/python/models/light_ranking/README.md",
    "content": "# Notification Light Ranker Model\n\n## Model Context\nThere are 4 major components of Twitter notifications recommendation system: 1) candidate generation 2) light ranking 3) heavy ranking & 4) quality control. This notification light ranker model bridges candidate generation and heavy ranking by pre-selecting highly-relevant candidates from the initial huge candidate pool. It’s a light-weight model to reduce system cost during heavy ranking without hurting user experience.\n\n## Directory Structure\n- BUILD: this file defines python library dependencies\n- model_pools_mlp.py: this file defines tensorflow model architecture for the notification light ranker model\n- deep_norm.py: this file contains 1) how to build the tensorflow graph with specified model architecture, loss function and training configuration. 2) how to set up the overall model training & evaluation pipeline\n- eval_model.py: the main python entry file to set up the overall model evaluation pipeline\n\n\n\n\n"
  },
  {
    "path": "pushservice/src/main/python/models/light_ranking/__init__.py",
    "content": ""
  },
  {
    "path": "pushservice/src/main/python/models/light_ranking/deep_norm.py",
    "content": "from datetime import datetime\nfrom functools import partial\nimport os\n\nfrom twitter.cortex.ml.embeddings.common.helpers import decode_str_or_unicode\nimport twml\nfrom twml.trainers import DataRecordTrainer\n\nfrom ..libs.get_feat_config import get_feature_config_light_ranking, LABELS_LR\nfrom ..libs.graph_utils import get_trainable_variables\nfrom ..libs.group_metrics import (\n  run_group_metrics_light_ranking,\n  run_group_metrics_light_ranking_in_bq,\n)\nfrom ..libs.metric_fn_utils import get_metric_fn\nfrom ..libs.model_args import get_arg_parser_light_ranking\nfrom ..libs.model_utils import read_config\nfrom ..libs.warm_start_utils import get_feature_list_for_light_ranking\nfrom .model_pools_mlp import light_ranking_mlp_ngbdt\n\nimport tensorflow.compat.v1 as tf\nfrom tensorflow.compat.v1 import logging\n\n\n# checkstyle: noqa\n\n\ndef build_graph(\n  features, label, mode, params, config=None, run_light_ranking_group_metrics_in_bq=False\n):\n  is_training = mode == tf.estimator.ModeKeys.TRAIN\n  this_model_func = light_ranking_mlp_ngbdt\n  model_output = this_model_func(features, is_training, params, label)\n\n  logits = model_output[\"output\"]\n  graph_output = {}\n  # --------------------------------------------------------\n  #            define graph output dict\n  # --------------------------------------------------------\n  if mode == tf.estimator.ModeKeys.PREDICT:\n    loss = None\n    output_label = \"prediction\"\n    if params.task_name in LABELS_LR:\n      output = tf.nn.sigmoid(logits)\n      output = tf.clip_by_value(output, 0, 1)\n\n      if run_light_ranking_group_metrics_in_bq:\n        graph_output[\"trace_id\"] = features[\"meta.trace_id\"]\n        graph_output[\"target\"] = features[\"meta.ranking.weighted_oonc_model_score\"]\n\n    else:\n      raise ValueError(\"Invalid Task Name !\")\n\n  else:\n    output_label = \"output\"\n    weights = tf.cast(features[\"weights\"], dtype=tf.float32, name=\"RecordWeights\")\n\n    if params.task_name in LABELS_LR:\n      if params.use_record_weight:\n        weights = tf.clip_by_value(\n          1.0 / (1.0 + weights + params.smooth_weight), params.min_record_weight, 1.0\n        )\n\n        loss = tf.reduce_sum(\n          tf.nn.sigmoid_cross_entropy_with_logits(labels=label, logits=logits) * weights\n        ) / (tf.reduce_sum(weights))\n      else:\n        loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=label, logits=logits))\n      output = tf.nn.sigmoid(logits)\n\n    else:\n      raise ValueError(\"Invalid Task Name !\")\n\n  train_op = None\n  if mode == tf.estimator.ModeKeys.TRAIN:\n    # --------------------------------------------------------\n    #                get train_op\n    # --------------------------------------------------------\n    optimizer = tf.train.GradientDescentOptimizer(learning_rate=params.learning_rate)\n    update_ops = set(tf.get_collection(tf.GraphKeys.UPDATE_OPS))\n    variables = get_trainable_variables(\n      all_trainable_variables=tf.trainable_variables(), trainable_regexes=params.trainable_regexes\n    )\n    with tf.control_dependencies(update_ops):\n      train_op = twml.optimizers.optimize_loss(\n        loss=loss,\n        variables=variables,\n        global_step=tf.train.get_global_step(),\n        optimizer=optimizer,\n        learning_rate=params.learning_rate,\n        learning_rate_decay_fn=twml.learning_rate_decay.get_learning_rate_decay_fn(params),\n      )\n\n  graph_output[output_label] = output\n  graph_output[\"loss\"] = loss\n  graph_output[\"train_op\"] = train_op\n  return graph_output\n\n\ndef get_params(args=None):\n  parser = get_arg_parser_light_ranking()\n  if args is None:\n    return parser.parse_args()\n  else:\n    return parser.parse_args(args)\n\n\ndef _main():\n  opt = get_params()\n  logging.info(\"parse is: \")\n  logging.info(opt)\n\n  feature_list = read_config(opt.feature_list).items()\n  feature_config = get_feature_config_light_ranking(\n    data_spec_path=opt.data_spec,\n    feature_list_provided=feature_list,\n    opt=opt,\n    add_gbdt=opt.use_gbdt_features,\n    run_light_ranking_group_metrics_in_bq=opt.run_light_ranking_group_metrics_in_bq,\n  )\n  feature_list_path = opt.feature_list\n\n  # --------------------------------------------------------\n  #               Create Trainer\n  # --------------------------------------------------------\n  trainer = DataRecordTrainer(\n    name=opt.model_trainer_name,\n    params=opt,\n    build_graph_fn=build_graph,\n    save_dir=opt.save_dir,\n    run_config=None,\n    feature_config=feature_config,\n    metric_fn=get_metric_fn(opt.task_name, use_stratify_metrics=False),\n  )\n  if opt.directly_export_best:\n    logging.info(\"Directly exporting the model without training\")\n  else:\n    # ----------------------------------------------------\n    #        Model Training & Evaluation\n    # ----------------------------------------------------\n    eval_input_fn = trainer.get_eval_input_fn(repeat=False, shuffle=False)\n    train_input_fn = trainer.get_train_input_fn(shuffle=True)\n\n    if opt.distributed or opt.num_workers is not None:\n      learn = trainer.train_and_evaluate\n    else:\n      learn = trainer.learn\n    logging.info(\"Training...\")\n    start = datetime.now()\n\n    early_stop_metric = \"rce_unweighted_\" + opt.task_name\n    learn(\n      early_stop_minimize=False,\n      early_stop_metric=early_stop_metric,\n      early_stop_patience=opt.early_stop_patience,\n      early_stop_tolerance=opt.early_stop_tolerance,\n      eval_input_fn=eval_input_fn,\n      train_input_fn=train_input_fn,\n    )\n\n    end = datetime.now()\n    logging.info(\"Training time: \" + str(end - start))\n\n    logging.info(\"Exporting the models...\")\n\n  # --------------------------------------------------------\n  #      Do the model exporting\n  # --------------------------------------------------------\n  start = datetime.now()\n  if not opt.export_dir:\n    opt.export_dir = os.path.join(opt.save_dir, \"exported_models\")\n\n  raw_model_path = twml.contrib.export.export_fn.export_all_models(\n    trainer=trainer,\n    export_dir=opt.export_dir,\n    parse_fn=feature_config.get_parse_fn(),\n    serving_input_receiver_fn=feature_config.get_serving_input_receiver_fn(),\n    export_output_fn=twml.export_output_fns.batch_prediction_continuous_output_fn,\n  )\n  export_model_dir = decode_str_or_unicode(raw_model_path)\n\n  logging.info(\"Model export time: \" + str(datetime.now() - start))\n  logging.info(\"The saved model directory is: \" + opt.save_dir)\n\n  tf.logging.info(\"getting default continuous_feature_list\")\n  continuous_feature_list = get_feature_list_for_light_ranking(feature_list_path, opt.data_spec)\n  continous_feature_list_save_path = os.path.join(opt.save_dir, \"continuous_feature_list.json\")\n  twml.util.write_file(continous_feature_list_save_path, continuous_feature_list, encode=\"json\")\n  tf.logging.info(f\"Finish writting files to {continous_feature_list_save_path}\")\n\n  if opt.run_light_ranking_group_metrics:\n    # --------------------------------------------\n    # Run Light Ranking Group Metrics\n    # --------------------------------------------\n    run_group_metrics_light_ranking(\n      trainer=trainer,\n      data_dir=os.path.join(opt.eval_data_dir, opt.eval_start_datetime),\n      model_path=export_model_dir,\n      parse_fn=feature_config.get_parse_fn(),\n    )\n\n  if opt.run_light_ranking_group_metrics_in_bq:\n    # ----------------------------------------------------------------------------------------\n    # Get Light/Heavy Ranker Predictions for Light Ranking Group Metrics in BigQuery\n    # ----------------------------------------------------------------------------------------\n    trainer_pred = DataRecordTrainer(\n      name=opt.model_trainer_name,\n      params=opt,\n      build_graph_fn=partial(build_graph, run_light_ranking_group_metrics_in_bq=True),\n      save_dir=opt.save_dir + \"/tmp/\",\n      run_config=None,\n      feature_config=feature_config,\n      metric_fn=get_metric_fn(opt.task_name, use_stratify_metrics=False),\n    )\n    checkpoint_folder = os.path.join(opt.save_dir, \"best_checkpoint\")\n    checkpoint = tf.train.latest_checkpoint(checkpoint_folder, latest_filename=None)\n    tf.logging.info(\"\\n\\nPrediction from Checkpoint: {:}.\\n\\n\".format(checkpoint))\n    run_group_metrics_light_ranking_in_bq(\n      trainer=trainer_pred, params=opt, checkpoint_path=checkpoint\n    )\n\n  tf.logging.info(\"Done Training & Prediction.\")\n\n\nif __name__ == \"__main__\":\n  _main()\n"
  },
  {
    "path": "pushservice/src/main/python/models/light_ranking/eval_model.py",
    "content": "from datetime import datetime\nfrom functools import partial\nimport os\n\nfrom ..libs.group_metrics import (\n  run_group_metrics_light_ranking,\n  run_group_metrics_light_ranking_in_bq,\n)\nfrom ..libs.metric_fn_utils import get_metric_fn\nfrom ..libs.model_args import get_arg_parser_light_ranking\nfrom ..libs.model_utils import read_config\nfrom .deep_norm import build_graph, DataRecordTrainer, get_config_func, logging\n\n\n# checkstyle: noqa\n\nif __name__ == \"__main__\":\n  parser = get_arg_parser_light_ranking()\n  parser.add_argument(\n    \"--eval_checkpoint\",\n    default=None,\n    type=str,\n    help=\"Which checkpoint to use for evaluation\",\n  )\n  parser.add_argument(\n    \"--saved_model_path\",\n    default=None,\n    type=str,\n    help=\"Path to saved model for evaluation\",\n  )\n  parser.add_argument(\n    \"--run_binary_metrics\",\n    default=False,\n    action=\"store_true\",\n    help=\"Whether to compute the basic binary metrics for Light Ranking.\",\n  )\n\n  opt = parser.parse_args()\n  logging.info(\"parse is: \")\n  logging.info(opt)\n\n  feature_list = read_config(opt.feature_list).items()\n  feature_config = get_config_func(opt.feat_config_type)(\n    data_spec_path=opt.data_spec,\n    feature_list_provided=feature_list,\n    opt=opt,\n    add_gbdt=opt.use_gbdt_features,\n    run_light_ranking_group_metrics_in_bq=opt.run_light_ranking_group_metrics_in_bq,\n  )\n\n  # -----------------------------------------------\n  #        Create Trainer\n  # -----------------------------------------------\n  trainer = DataRecordTrainer(\n    name=opt.model_trainer_name,\n    params=opt,\n    build_graph_fn=partial(build_graph, run_light_ranking_group_metrics_in_bq=True),\n    save_dir=opt.save_dir,\n    run_config=None,\n    feature_config=feature_config,\n    metric_fn=get_metric_fn(opt.task_name, use_stratify_metrics=False),\n  )\n\n  # -----------------------------------------------\n  #         Model Evaluation\n  # -----------------------------------------------\n  logging.info(\"Evaluating...\")\n  start = datetime.now()\n\n  if opt.run_binary_metrics:\n    eval_input_fn = trainer.get_eval_input_fn(repeat=False, shuffle=False)\n    eval_steps = None if (opt.eval_steps is not None and opt.eval_steps < 0) else opt.eval_steps\n    trainer.estimator.evaluate(eval_input_fn, steps=eval_steps, checkpoint_path=opt.eval_checkpoint)\n\n  if opt.run_light_ranking_group_metrics_in_bq:\n    run_group_metrics_light_ranking_in_bq(\n      trainer=trainer, params=opt, checkpoint_path=opt.eval_checkpoint\n    )\n\n  if opt.run_light_ranking_group_metrics:\n    run_group_metrics_light_ranking(\n      trainer=trainer,\n      data_dir=os.path.join(opt.eval_data_dir, opt.eval_start_datetime),\n      model_path=opt.saved_model_path,\n      parse_fn=feature_config.get_parse_fn(),\n    )\n\n  end = datetime.now()\n  logging.info(\"Evaluating time: \" + str(end - start))\n"
  },
  {
    "path": "pushservice/src/main/python/models/light_ranking/model_pools_mlp.py",
    "content": "import warnings\n\nfrom twml.contrib.layers import ZscoreNormalization\n\nfrom ...libs.customized_full_sparse import FullSparse\nfrom ...libs.get_feat_config import FEAT_CONFIG_DEFAULT_VAL as MISSING_VALUE_MARKER\nfrom ...libs.model_utils import (\n  _sparse_feature_fixup,\n  adaptive_transformation,\n  filter_nans_and_infs,\n  get_dense_out,\n  tensor_dropout,\n)\n\nimport tensorflow.compat.v1 as tf\n# checkstyle: noqa\n\ndef light_ranking_mlp_ngbdt(features, is_training, params, label=None):\n  return deepnorm_light_ranking(\n    features,\n    is_training,\n    params,\n    label=label,\n    decay=params.momentum,\n    dense_emb_size=params.dense_embedding_size,\n    base_activation=tf.keras.layers.LeakyReLU(),\n    input_dropout_rate=params.dropout,\n    use_gbdt=False,\n  )\n\n\ndef deepnorm_light_ranking(\n  features,\n  is_training,\n  params,\n  label=None,\n  decay=0.99999,\n  dense_emb_size=128,\n  base_activation=None,\n  input_dropout_rate=None,\n  input_dense_type=\"self_atten_dense\",\n  emb_dense_type=\"self_atten_dense\",\n  mlp_dense_type=\"self_atten_dense\",\n  use_gbdt=False,\n):\n  # --------------------------------------------------------\n  #            Initial Parameter Checking\n  # --------------------------------------------------------\n  if base_activation is None:\n    base_activation = tf.keras.layers.LeakyReLU()\n\n  if label is not None:\n    warnings.warn(\n      \"Label is unused in deepnorm_gbdt. Stop using this argument.\",\n      DeprecationWarning,\n    )\n\n  with tf.variable_scope(\"helper_layers\"):\n    full_sparse_layer = FullSparse(\n      output_size=params.sparse_embedding_size,\n      activation=base_activation,\n      use_sparse_grads=is_training,\n      use_binary_values=False,\n      dtype=tf.float32,\n    )\n    input_normalizing_layer = ZscoreNormalization(decay=decay, name=\"input_normalizing_layer\")\n\n  # --------------------------------------------------------\n  #            Feature Selection & Embedding\n  # --------------------------------------------------------\n  if use_gbdt:\n    sparse_gbdt_features = _sparse_feature_fixup(features[\"gbdt_sparse\"], params.input_size_bits)\n    if input_dropout_rate is not None:\n      sparse_gbdt_features = tensor_dropout(\n        sparse_gbdt_features, input_dropout_rate, is_training, sparse_tensor=True\n      )\n\n    total_embed = full_sparse_layer(sparse_gbdt_features, use_binary_values=True)\n\n    if (input_dropout_rate is not None) and is_training:\n      total_embed = total_embed / (1 - input_dropout_rate)\n\n  else:\n    with tf.variable_scope(\"dense_branch\"):\n      dense_continuous_features = filter_nans_and_infs(features[\"continuous\"])\n\n      if params.use_missing_sub_branch:\n        is_missing = tf.equal(dense_continuous_features, MISSING_VALUE_MARKER)\n        continuous_features_filled = tf.where(\n          is_missing,\n          tf.zeros_like(dense_continuous_features),\n          dense_continuous_features,\n        )\n        normalized_features = input_normalizing_layer(\n          continuous_features_filled, is_training, tf.math.logical_not(is_missing)\n        )\n\n        with tf.variable_scope(\"missing_sub_branch\"):\n          missing_feature_embed = get_dense_out(\n            tf.cast(is_missing, tf.float32),\n            dense_emb_size,\n            activation=base_activation,\n            dense_type=input_dense_type,\n          )\n\n      else:\n        continuous_features_filled = dense_continuous_features\n        normalized_features = input_normalizing_layer(continuous_features_filled, is_training)\n\n      with tf.variable_scope(\"continuous_sub_branch\"):\n        normalized_features = adaptive_transformation(\n          normalized_features, is_training, func_type=\"tiny\"\n        )\n\n        if input_dropout_rate is not None:\n          normalized_features = tensor_dropout(\n            normalized_features,\n            input_dropout_rate,\n            is_training,\n            sparse_tensor=False,\n          )\n        filled_feature_embed = get_dense_out(\n          normalized_features,\n          dense_emb_size,\n          activation=base_activation,\n          dense_type=input_dense_type,\n        )\n\n      if params.use_missing_sub_branch:\n        dense_embed = tf.concat(\n          [filled_feature_embed, missing_feature_embed], axis=1, name=\"merge_dense_emb\"\n        )\n      else:\n        dense_embed = filled_feature_embed\n\n    with tf.variable_scope(\"sparse_branch\"):\n      sparse_discrete_features = _sparse_feature_fixup(\n        features[\"sparse_no_continuous\"], params.input_size_bits\n      )\n      if input_dropout_rate is not None:\n        sparse_discrete_features = tensor_dropout(\n          sparse_discrete_features, input_dropout_rate, is_training, sparse_tensor=True\n        )\n\n      discrete_features_embed = full_sparse_layer(sparse_discrete_features, use_binary_values=True)\n\n      if (input_dropout_rate is not None) and is_training:\n        discrete_features_embed = discrete_features_embed / (1 - input_dropout_rate)\n\n    total_embed = tf.concat(\n      [dense_embed, discrete_features_embed],\n      axis=1,\n      name=\"total_embed\",\n    )\n\n  total_embed = tf.layers.batch_normalization(\n    total_embed,\n    training=is_training,\n    renorm_momentum=decay,\n    momentum=decay,\n    renorm=is_training,\n    trainable=True,\n  )\n\n  # --------------------------------------------------------\n  #                MLP Layers\n  # --------------------------------------------------------\n  with tf.variable_scope(\"MLP_branch\"):\n\n    assert params.num_mlp_layers >= 0\n    embed_list = [total_embed] + [None for _ in range(params.num_mlp_layers)]\n    dense_types = [emb_dense_type] + [mlp_dense_type for _ in range(params.num_mlp_layers - 1)]\n\n    for xl in range(1, params.num_mlp_layers + 1):\n      neurons = params.mlp_neuron_scale ** (params.num_mlp_layers + 1 - xl)\n      embed_list[xl] = get_dense_out(\n        embed_list[xl - 1], neurons, activation=base_activation, dense_type=dense_types[xl - 1]\n      )\n\n    if params.task_name in [\"Sent\", \"HeavyRankPosition\", \"HeavyRankProbability\"]:\n      logits = get_dense_out(embed_list[-1], 1, activation=None, dense_type=mlp_dense_type)\n\n    else:\n      raise ValueError(\"Invalid Task Name !\")\n\n  output_dict = {\"output\": logits}\n  return output_dict\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/BUILD.bazel",
    "content": "scala_library(\n    sources = [\"**/*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\n        \"bazel-compatible\",\n    ],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/bijection:scrooge\",\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"abdecider\",\n        \"abuse/detection/src/main/thrift/com/twitter/abuse/detection/scoring:thrift-scala\",\n        \"ann/src/main/scala/com/twitter/ann/common\",\n        \"ann/src/main/thrift/com/twitter/ann/common:ann-common-scala\",\n        \"audience-rewards/thrift/src/main/thrift:thrift-scala\",\n        \"communities/thrift/src/main/thrift/com/twitter/communities:thrift-scala\",\n        \"configapi/configapi-core\",\n        \"configapi/configapi-decider\",\n        \"content-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"content-recommender/thrift/src/main/thrift:thrift-scala\",\n        \"copyselectionservice/server/src/main/scala/com/twitter/copyselectionservice/algorithms\",\n        \"copyselectionservice/thrift/src/main/thrift:copyselectionservice-scala\",\n        \"cortex-deepbird/thrift/src/main/thrift:thrift-java\",\n        \"cr-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"cuad/projects/hashspace/thrift:thrift-scala\",\n        \"cuad/projects/tagspace/thrift/src/main/thrift:thrift-scala\",\n        \"detopic/thrift/src/main/thrift:thrift-scala\",\n        \"discovery-common/src/main/scala/com/twitter/discovery/common/configapi\",\n        \"discovery-common/src/main/scala/com/twitter/discovery/common/ddg\",\n        \"discovery-common/src/main/scala/com/twitter/discovery/common/environment\",\n        \"discovery-common/src/main/scala/com/twitter/discovery/common/fatigue\",\n        \"discovery-common/src/main/scala/com/twitter/discovery/common/nackwarmupfilter\",\n        \"discovery-common/src/main/scala/com/twitter/discovery/common/server\",\n        \"discovery-ds/src/main/thrift/com/twitter/dds/scio/searcher_aggregate_history_srp:searcher_aggregate_history_srp-scala\",\n        \"escherbird/src/scala/com/twitter/escherbird/util/metadatastitch\",\n        \"escherbird/src/scala/com/twitter/escherbird/util/uttclient\",\n        \"escherbird/src/thrift/com/twitter/escherbird/utt:strato-columns-scala\",\n        \"eventbus/client\",\n        \"eventdetection/event_context/src/main/scala/com/twitter/eventdetection/event_context/util\",\n        \"events-recos/events-recos-service/src/main/thrift:events-recos-thrift-scala\",\n        \"explore/explore-ranker/thrift/src/main/thrift:thrift-scala\",\n        \"featureswitches/featureswitches-core/src/main/scala\",\n        \"featureswitches/featureswitches-core/src/main/scala:dynmap\",\n        \"featureswitches/featureswitches-core/src/main/scala:recipient\",\n        \"featureswitches/featureswitches-core/src/main/scala:useragent\",\n        \"featureswitches/featureswitches-core/src/main/scala/com/twitter/featureswitches/v2/builder\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/server\",\n        \"finagle-internal/ostrich-stats\",\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"finagle/finagle-memcached/src/main/scala\",\n        \"finagle/finagle-stats\",\n        \"finagle/finagle-thriftmux\",\n        \"finagle/finagle-tunable/src/main/scala\",\n        \"finagle/finagle-zipkin-scribe\",\n        \"finatra-internal/abdecider\",\n        \"finatra-internal/decider\",\n        \"finatra-internal/mtls-http/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/http-client/src/main/scala\",\n        \"finatra/http-core/src/main/java/com/twitter/finatra/http\",\n        \"finatra/http-core/src/main/scala/com/twitter/finatra/http/response\",\n        \"finatra/http-server/src/main/scala/com/twitter/finatra/http\",\n        \"finatra/http-server/src/main/scala/com/twitter/finatra/http/filters\",\n        \"finatra/inject/inject-app/src/main/java/com/twitter/inject/annotations\",\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-server/src/main/scala\",\n        \"finatra/inject/inject-slf4j/src/main/scala/com/twitter/inject\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"finatra/inject/inject-utils/src/main/scala\",\n        \"finatra/utils/src/main/java/com/twitter/finatra/annotations\",\n        \"fleets/fleets-proxy/thrift/src/main/thrift:fleet-scala\",\n        \"fleets/fleets-proxy/thrift/src/main/thrift/service:baseservice-scala\",\n        \"flock-client/src/main/scala\",\n        \"flock-client/src/main/thrift:thrift-scala\",\n        \"follow-recommendations-service/thrift/src/main/thrift:thrift-scala\",\n        \"frigate/frigate-common:base\",\n        \"frigate/frigate-common:config\",\n        \"frigate/frigate-common:debug\",\n        \"frigate/frigate-common:entity_graph_client\",\n        \"frigate/frigate-common:history\",\n        \"frigate/frigate-common:logger\",\n        \"frigate/frigate-common:ml-base\",\n        \"frigate/frigate-common:ml-feature\",\n        \"frigate/frigate-common:ml-prediction\",\n        \"frigate/frigate-common:ntab\",\n        \"frigate/frigate-common:predicate\",\n        \"frigate/frigate-common:rec_types\",\n        \"frigate/frigate-common:score_summary\",\n        \"frigate/frigate-common:util\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/candidate\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/experiments\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/filter\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/modules/store:semantic_core_stores\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/deviceinfo\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/interests\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/strato\",\n        \"frigate/push-mixer/thrift/src/main/thrift:thrift-scala\",\n        \"geo/geo-prediction/src/main/thrift:local-viral-tweets-thrift-scala\",\n        \"geoduck/service/src/main/scala/com/twitter/geoduck/service/common/clientmodules\",\n        \"geoduck/util/country\",\n        \"gizmoduck/client/src/main/scala/com/twitter/gizmoduck/testusers/client\",\n        \"hermit/hermit-core:model-user_state\",\n        \"hermit/hermit-core:predicate\",\n        \"hermit/hermit-core:predicate-gizmoduck\",\n        \"hermit/hermit-core:predicate-scarecrow\",\n        \"hermit/hermit-core:predicate-socialgraph\",\n        \"hermit/hermit-core:predicate-tweetypie\",\n        \"hermit/hermit-core:store-labeled_push_recs\",\n        \"hermit/hermit-core:store-metastore\",\n        \"hermit/hermit-core:store-timezone\",\n        \"hermit/hermit-core:store-tweetypie\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/constants\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/model\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/gizmoduck\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/scarecrow\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/semantic_core\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/user_htl_session_store\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/user_interest\",\n        \"hmli/hss/src/main/thrift/com/twitter/hss:thrift-scala\",\n        \"ibis2/service/src/main/scala/com/twitter/ibis2/lib\",\n        \"ibis2/service/src/main/thrift/com/twitter/ibis2/service:ibis2-service-scala\",\n        \"interests-service/thrift/src/main/thrift:thrift-scala\",\n        \"interests_discovery/thrift/src/main/thrift:batch-thrift-scala\",\n        \"interests_discovery/thrift/src/main/thrift:service-thrift-scala\",\n        \"kujaku/thrift/src/main/thrift:domain-scala\",\n        \"live-video-timeline/client/src/main/scala/com/twitter/livevideo/timeline/client/v2\",\n        \"live-video-timeline/domain/src/main/scala/com/twitter/livevideo/timeline/domain\",\n        \"live-video-timeline/domain/src/main/scala/com/twitter/livevideo/timeline/domain/v2\",\n        \"live-video-timeline/thrift/src/main/thrift/com/twitter/livevideo/timeline:thrift-scala\",\n        \"live-video/common/src/main/scala/com/twitter/livevideo/common/domain/v2\",\n        \"live-video/common/src/main/scala/com/twitter/livevideo/common/ids\",\n        \"notifications-platform/inbound-notifications/src/main/thrift/com/twitter/inbound_notifications:exception-scala\",\n        \"notifications-platform/inbound-notifications/src/main/thrift/com/twitter/inbound_notifications:thrift-scala\",\n        \"notifications-platform/platform-lib/src/main/thrift/com/twitter/notifications/platform:custom-notification-actions-scala\",\n        \"notifications-platform/platform-lib/src/main/thrift/com/twitter/notifications/platform:thrift-scala\",\n        \"notifications-relevance/src/scala/com/twitter/nrel/heavyranker\",\n        \"notifications-relevance/src/scala/com/twitter/nrel/hydration/base\",\n        \"notifications-relevance/src/scala/com/twitter/nrel/hydration/frigate\",\n        \"notifications-relevance/src/scala/com/twitter/nrel/hydration/push\",\n        \"notifications-relevance/src/scala/com/twitter/nrel/lightranker\",\n        \"notificationservice/common/src/main/scala/com/twitter/notificationservice/genericfeedbackstore\",\n        \"notificationservice/common/src/main/scala/com/twitter/notificationservice/model:alias\",\n        \"notificationservice/common/src/main/scala/com/twitter/notificationservice/model/service\",\n        \"notificationservice/common/src/test/scala/com/twitter/notificationservice/mocks\",\n        \"notificationservice/scribe/src/main/scala/com/twitter/notificationservice/scribe/manhattan:mh_wrapper\",\n        \"notificationservice/thrift/src/main/thrift/com/twitter/notificationservice/api:thrift-scala\",\n        \"notificationservice/thrift/src/main/thrift/com/twitter/notificationservice/badgecount-api:thrift-scala\",\n        \"notificationservice/thrift/src/main/thrift/com/twitter/notificationservice/generic_notifications:thrift-scala\",\n        \"notifinfra/ni-lib/src/main/scala/com/twitter/ni/lib/logged_out_transform\",\n        \"observability/observability-manhattan-client/src/main/scala\",\n        \"onboarding/service/src/main/scala/com/twitter/onboarding/task/service/models/external\",\n        \"onboarding/service/thrift/src/main/thrift:thrift-scala\",\n        \"people-discovery/api/thrift/src/main/thrift:thrift-scala\",\n        \"periscope/api-proxy-thrift/thrift/src/main/thrift:thrift-scala\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module\",\n        \"product-mixer/core/src/main/scala/com/twitter/product_mixer/core/module/stringcenter\",\n        \"product-mixer/core/src/main/thrift/com/twitter/product_mixer/core:thrift-scala\",\n        \"qig-ranker/thrift/src/main/thrift:thrift-scala\",\n        \"rux-ds/src/main/thrift/com/twitter/ruxds/jobs/user_past_aggregate:user_past_aggregate-scala\",\n        \"rux/common/src/main/scala/com/twitter/rux/common/encode\",\n        \"rux/common/thrift/src/main/thrift/rux-context:rux-context-scala\",\n        \"rux/common/thrift/src/main/thrift/strato:strato-scala\",\n        \"scribelib/marshallers/src/main/scala/com/twitter/scribelib/marshallers\",\n        \"scrooge/scrooge-core\",\n        \"scrooge/scrooge-serializer/src/main/scala\",\n        \"sensitive-ds/src/main/thrift/com/twitter/scio/nsfw_user_segmentation:nsfw_user_segmentation-scala\",\n        \"servo/decider/src/main/scala\",\n        \"servo/request/src/main/scala\",\n        \"servo/util/src/main/scala\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/ml/prediction/core\",\n        \"src/scala/com/twitter/frigate/data_pipeline/common\",\n        \"src/scala/com/twitter/frigate/data_pipeline/embedding_cg:embedding_cg-test-user-ids\",\n        \"src/scala/com/twitter/frigate/data_pipeline/features_common\",\n        \"src/scala/com/twitter/frigate/news_article_recs/news_articles_metadata:thrift-scala\",\n        \"src/scala/com/twitter/frontpage/stream/util\",\n        \"src/scala/com/twitter/language/normalization\",\n        \"src/scala/com/twitter/ml/api/embedding\",\n        \"src/scala/com/twitter/ml/api/util:datarecord\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/entities/core\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/entities/magicrecs\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/core:aggregate\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/cuad:aggregate\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/embeddings\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/magicrecs:aggregate\",\n        \"src/scala/com/twitter/ml/featurestore/catalog/features/topic_signals:aggregate\",\n        \"src/scala/com/twitter/ml/featurestore/lib\",\n        \"src/scala/com/twitter/ml/featurestore/lib/data\",\n        \"src/scala/com/twitter/ml/featurestore/lib/dynamic\",\n        \"src/scala/com/twitter/ml/featurestore/lib/entity\",\n        \"src/scala/com/twitter/ml/featurestore/lib/online\",\n        \"src/scala/com/twitter/recommendation/interests/discovery/core/config\",\n        \"src/scala/com/twitter/recommendation/interests/discovery/core/deploy\",\n        \"src/scala/com/twitter/recommendation/interests/discovery/core/model\",\n        \"src/scala/com/twitter/recommendation/interests/discovery/popgeo/deploy\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/scala/com/twitter/storehaus_internal/manhattan\",\n        \"src/scala/com/twitter/storehaus_internal/manhattan/config\",\n        \"src/scala/com/twitter/storehaus_internal/memcache\",\n        \"src/scala/com/twitter/storehaus_internal/memcache/config\",\n        \"src/scala/com/twitter/storehaus_internal/util\",\n        \"src/scala/com/twitter/taxi/common\",\n        \"src/scala/com/twitter/taxi/config\",\n        \"src/scala/com/twitter/taxi/deploy\",\n        \"src/scala/com/twitter/taxi/trending/common\",\n        \"src/thrift/com/twitter/ads/adserver:adserver_rpc-scala\",\n        \"src/thrift/com/twitter/clientapp/gen:clientapp-scala\",\n        \"src/thrift/com/twitter/core_workflows/user_model:user_model-scala\",\n        \"src/thrift/com/twitter/escherbird/common:constants-scala\",\n        \"src/thrift/com/twitter/escherbird/metadata:megadata-scala\",\n        \"src/thrift/com/twitter/escherbird/metadata:metadata-service-scala\",\n        \"src/thrift/com/twitter/escherbird/search:search-service-scala\",\n        \"src/thrift/com/twitter/expandodo:only-scala\",\n        \"src/thrift/com/twitter/frigate:frigate-common-thrift-scala\",\n        \"src/thrift/com/twitter/frigate:frigate-ml-thrift-scala\",\n        \"src/thrift/com/twitter/frigate:frigate-notification-thrift-scala\",\n        \"src/thrift/com/twitter/frigate:frigate-secondary-accounts-thrift-scala\",\n        \"src/thrift/com/twitter/frigate:frigate-thrift-scala\",\n        \"src/thrift/com/twitter/frigate:frigate-user-media-representation-thrift-scala\",\n        \"src/thrift/com/twitter/frigate/data_pipeline:frigate-user-history-thrift-scala\",\n        \"src/thrift/com/twitter/frigate/dau_model:frigate-dau-thrift-scala\",\n        \"src/thrift/com/twitter/frigate/magic_events:frigate-magic-events-thrift-scala\",\n        \"src/thrift/com/twitter/frigate/magic_events/scribe:thrift-scala\",\n        \"src/thrift/com/twitter/frigate/pushcap:frigate-pushcap-thrift-scala\",\n        \"src/thrift/com/twitter/frigate/pushservice:frigate-pushservice-thrift-scala\",\n        \"src/thrift/com/twitter/frigate/scribe:frigate-scribe-thrift-scala\",\n        \"src/thrift/com/twitter/frigate/subscribed_search:frigate-subscribed-search-thrift-scala\",\n        \"src/thrift/com/twitter/frigate/user_states:frigate-userstates-thrift-scala\",\n        \"src/thrift/com/twitter/geoduck:geoduck-scala\",\n        \"src/thrift/com/twitter/gizmoduck:thrift-scala\",\n        \"src/thrift/com/twitter/gizmoduck:user-thrift-scala\",\n        \"src/thrift/com/twitter/hermit:hermit-scala\",\n        \"src/thrift/com/twitter/hermit/pop_geo:hermit-pop-geo-scala\",\n        \"src/thrift/com/twitter/hermit/stp:hermit-stp-scala\",\n        \"src/thrift/com/twitter/ibis:service-scala\",\n        \"src/thrift/com/twitter/manhattan:v1-scala\",\n        \"src/thrift/com/twitter/manhattan:v2-scala\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/ml/api:data-scala\",\n        \"src/thrift/com/twitter/ml/featurestore/timelines:ml-features-timelines-scala\",\n        \"src/thrift/com/twitter/ml/featurestore/timelines:ml-features-timelines-strato\",\n        \"src/thrift/com/twitter/ml/prediction_service:prediction_service-java\",\n        \"src/thrift/com/twitter/permissions_storage:thrift-scala\",\n        \"src/thrift/com/twitter/pink-floyd/thrift:thrift-scala\",\n        \"src/thrift/com/twitter/recos:recos-common-scala\",\n        \"src/thrift/com/twitter/recos/user_tweet_entity_graph:user_tweet_entity_graph-scala\",\n        \"src/thrift/com/twitter/recos/user_user_graph:user_user_graph-scala\",\n        \"src/thrift/com/twitter/relevance/feature_store:feature_store-scala\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"src/thrift/com/twitter/search/common:features-scala\",\n        \"src/thrift/com/twitter/search/query_interaction_graph:query_interaction_graph-scala\",\n        \"src/thrift/com/twitter/search/query_interaction_graph/service:qig-service-scala\",\n        \"src/thrift/com/twitter/service/metastore/gen:thrift-scala\",\n        \"src/thrift/com/twitter/service/scarecrow/gen:scarecrow-scala\",\n        \"src/thrift/com/twitter/service/scarecrow/gen:tiered-actions-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"src/thrift/com/twitter/spam/rtf:safety-level-scala\",\n        \"src/thrift/com/twitter/timelinemixer:thrift-scala\",\n        \"src/thrift/com/twitter/timelinemixer/server/internal:thrift-scala\",\n        \"src/thrift/com/twitter/timelines/author_features/user_health:thrift-scala\",\n        \"src/thrift/com/twitter/timelines/real_graph:real_graph-scala\",\n        \"src/thrift/com/twitter/timelinescorer:thrift-scala\",\n        \"src/thrift/com/twitter/timelinescorer/server/internal:thrift-scala\",\n        \"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala\",\n        \"src/thrift/com/twitter/timelineservice/server/suggests/logging:thrift-scala\",\n        \"src/thrift/com/twitter/trends/common:common-scala\",\n        \"src/thrift/com/twitter/trends/trip_v1:trip-tweets-thrift-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n        \"src/thrift/com/twitter/user_session_store:thrift-scala\",\n        \"src/thrift/com/twitter/wtf/candidate:wtf-candidate-scala\",\n        \"src/thrift/com/twitter/wtf/interest:interest-thrift-scala\",\n        \"src/thrift/com/twitter/wtf/scalding/common:thrift-scala\",\n        \"stitch/stitch-core\",\n        \"stitch/stitch-gizmoduck\",\n        \"stitch/stitch-socialgraph/src/main/scala\",\n        \"stitch/stitch-storehaus/src/main/scala\",\n        \"stitch/stitch-tweetypie/src/main/scala\",\n        \"storage/clients/manhattan/client/src/main/scala\",\n        \"strato/config/columns/clients:clients-strato-client\",\n        \"strato/config/columns/geo/user:user-strato-client\",\n        \"strato/config/columns/globe/curation:curation-strato-client\",\n        \"strato/config/columns/interests:interests-strato-client\",\n        \"strato/config/columns/ml/featureStore:featureStore-strato-client\",\n        \"strato/config/columns/notifications:notifications-strato-client\",\n        \"strato/config/columns/notifinfra:notifinfra-strato-client\",\n        \"strato/config/columns/periscope:periscope-strato-client\",\n        \"strato/config/columns/rux\",\n        \"strato/config/columns/rux:rux-strato-client\",\n        \"strato/config/columns/rux/open-app:open-app-strato-client\",\n        \"strato/config/columns/socialgraph/graphs:graphs-strato-client\",\n        \"strato/config/columns/socialgraph/service/soft_users:soft_users-strato-client\",\n        \"strato/config/columns/translation/service:service-strato-client\",\n        \"strato/config/columns/translation/service/platform:platform-strato-client\",\n        \"strato/config/columns/trends/trip:trip-strato-client\",\n        \"strato/config/src/thrift/com/twitter/strato/columns/frigate:logged-out-web-notifications-scala\",\n        \"strato/config/src/thrift/com/twitter/strato/columns/notifications:thrift-scala\",\n        \"strato/src/main/scala/com/twitter/strato/config\",\n        \"strato/src/main/scala/com/twitter/strato/response\",\n        \"thrift-web-forms\",\n        \"timeline-training-service/service/thrift/src/main/thrift:thrift-scala\",\n        \"timelines/src/main/scala/com/twitter/timelines/features/app\",\n        \"topic-social-proof/server/src/main/thrift:thrift-scala\",\n        \"topiclisting/topiclisting-core/src/main/scala/com/twitter/topiclisting\",\n        \"topiclisting/topiclisting-utt/src/main/scala/com/twitter/topiclisting/utt\",\n        \"trends/common/src/main/thrift/com/twitter/trends/common:thrift-scala\",\n        \"tweetypie/src/scala/com/twitter/tweetypie/tweettext\",\n        \"twitter-context/src/main/scala\",\n        \"twitter-server-internal\",\n        \"twitter-server/server/src/main/scala\",\n        \"twitter-text/lib/java/src/main/java/com/twitter/twittertext\",\n        \"twml/runtime/src/main/scala/com/twitter/deepbird/runtime/prediction_engine:prediction_engine_mkl\",\n        \"ubs/common/src/main/thrift/com/twitter/ubs:broadcast-thrift-scala\",\n        \"ubs/common/src/main/thrift/com/twitter/ubs:seller_application-thrift-scala\",\n        \"user_session_store/src/main/scala/com/twitter/user_session_store/impl/manhattan/readwrite\",\n        \"util-internal/scribe\",\n        \"util-internal/tunable/src/main/scala/com/twitter/util/tunable\",\n        \"util/util-app\",\n        \"util/util-hashing/src/main/scala\",\n        \"util/util-slf4j-api/src/main/scala\",\n        \"util/util-stats/src/main/scala\",\n        \"visibility/lib/src/main/scala/com/twitter/visibility/builder\",\n        \"visibility/lib/src/main/scala/com/twitter/visibility/interfaces/push_service\",\n        \"visibility/lib/src/main/scala/com/twitter/visibility/interfaces/spaces\",\n        \"visibility/lib/src/main/scala/com/twitter/visibility/util\",\n    ],\n    exports = [\n        \"strato/config/src/thrift/com/twitter/strato/columns/frigate:logged-out-web-notifications-scala\",\n    ],\n)\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/PushMixerThriftServerWarmupHandler.scala",
    "content": "package com.twitter.frigate.pushservice\n\nimport com.google.inject.Inject\nimport com.google.inject.Singleton\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finatra.thrift.routing.ThriftWarmup\nimport com.twitter.util.logging.Logging\nimport com.twitter.inject.utils.Handler\nimport com.twitter.frigate.pushservice.{thriftscala => t}\nimport com.twitter.frigate.thriftscala.NotificationDisplayLocation\nimport com.twitter.util.Stopwatch\nimport com.twitter.scrooge.Request\nimport com.twitter.scrooge.Response\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport com.twitter.util.Try\n\n/**\n * Warms up the refresh request path.\n * If service is running as pushservice-send then the warmup does nothing.\n *\n * When making the warmup refresh requests we\n *  - Set skipFilters to true to execute as much of the request path as possible\n *  - Set darkWrite to true to prevent sending a push\n */\n@Singleton\nclass PushMixerThriftServerWarmupHandler @Inject() (\n  warmup: ThriftWarmup,\n  serviceIdentifier: ServiceIdentifier)\n    extends Handler\n    with Logging {\n\n  private val clientId = ClientId(\"thrift-warmup-client\")\n\n  def handle(): Unit = {\n    val refreshServices = Set(\n      \"frigate-pushservice\",\n      \"frigate-pushservice-canary\",\n      \"frigate-pushservice-canary-control\",\n      \"frigate-pushservice-canary-treatment\"\n    )\n    val isRefresh = refreshServices.contains(serviceIdentifier.service)\n    if (isRefresh && !serviceIdentifier.isLocal) refreshWarmup()\n  }\n\n  def refreshWarmup(): Unit = {\n    val elapsed = Stopwatch.start()\n    val testIds = Seq(\n      1,\n      2,\n      3\n    )\n    try {\n      clientId.asCurrent {\n        testIds.foreach { id =>\n          val warmupReq = warmupQuery(id)\n          info(s\"Sending warm-up request to service with query: $warmupReq\")\n          warmup.sendRequest(\n            method = t.PushService.Refresh,\n            req = Request(t.PushService.Refresh.Args(warmupReq)))(assertWarmupResponse)\n        }\n      }\n    } catch {\n      case e: Throwable =>\n        error(e.getMessage, e)\n    }\n    info(s\"Warm up complete. Time taken: ${elapsed().toString}\")\n  }\n\n  private def warmupQuery(userId: Long): t.RefreshRequest = {\n    t.RefreshRequest(\n      userId = userId,\n      notificationDisplayLocation = NotificationDisplayLocation.PushToMobileDevice,\n      context = Some(\n        t.PushContext(\n          skipFilters = Some(true),\n          darkWrite = Some(true)\n        ))\n    )\n  }\n\n  private def assertWarmupResponse(\n    result: Try[Response[t.PushService.Refresh.SuccessType]]\n  ): Unit = {\n    result match {\n      case Return(_) => // ok\n      case Throw(exception) =>\n        warn(\"Error performing warm-up request.\")\n        error(exception.getMessage, exception)\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/PushServiceMain.scala",
    "content": "package com.twitter.frigate.pushservice\n\nimport com.twitter.discovery.common.environment.modules.EnvironmentModule\nimport com.twitter.finagle.Filter\nimport com.twitter.finatra.annotations.DarkTrafficFilterType\nimport com.twitter.finatra.decider.modules.DeciderModule\nimport com.twitter.finatra.http.HttpServer\nimport com.twitter.finatra.http.filters.CommonFilters\nimport com.twitter.finatra.http.routing.HttpRouter\nimport com.twitter.finatra.mtls.http.{Mtls => HttpMtls}\nimport com.twitter.finatra.mtls.thriftmux.{Mtls => ThriftMtls}\nimport com.twitter.finatra.mtls.thriftmux.filters.MtlsServerSessionTrackerFilter\nimport com.twitter.finatra.thrift.ThriftServer\nimport com.twitter.finatra.thrift.filters.ExceptionMappingFilter\nimport com.twitter.finatra.thrift.filters.LoggingMDCFilter\nimport com.twitter.finatra.thrift.filters.StatsFilter\nimport com.twitter.finatra.thrift.filters.ThriftMDCFilter\nimport com.twitter.finatra.thrift.filters.TraceIdMDCFilter\nimport com.twitter.finatra.thrift.routing.ThriftRouter\nimport com.twitter.frigate.common.logger.MRLoggerGlobalVariables\nimport com.twitter.frigate.pushservice.controller.PushServiceController\nimport com.twitter.frigate.pushservice.module._\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flags\nimport com.twitter.inject.thrift.modules.ThriftClientIdModule\nimport com.twitter.logging.BareFormatter\nimport com.twitter.logging.Level\nimport com.twitter.logging.LoggerFactory\nimport com.twitter.logging.{Logging => JLogging}\nimport com.twitter.logging.QueueingHandler\nimport com.twitter.logging.ScribeHandler\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule\nimport com.twitter.product_mixer.core.module.ABDeciderModule\nimport com.twitter.product_mixer.core.module.FeatureSwitchesModule\nimport com.twitter.product_mixer.core.module.StratoClientModule\n\nobject PushServiceMain extends PushServiceFinatraServer\n\nclass PushServiceFinatraServer\n    extends ThriftServer\n    with ThriftMtls\n    with HttpServer\n    with HttpMtls\n    with JLogging {\n\n  override val name = \"PushService\"\n\n  override val modules: Seq[TwitterModule] = {\n    Seq(\n      ABDeciderModule,\n      DeciderModule,\n      FeatureSwitchesModule,\n      FilterModule,\n      FlagModule,\n      EnvironmentModule,\n      ThriftClientIdModule,\n      DeployConfigModule,\n      ProductMixerFlagModule,\n      StratoClientModule,\n      PushHandlerModule,\n      PushTargetUserBuilderModule,\n      PushServiceDarkTrafficModule,\n      LoggedOutPushTargetUserBuilderModule,\n      new ThriftWebFormsModule(this),\n    )\n  }\n\n  override def configureThrift(router: ThriftRouter): Unit = {\n    router\n      .filter[ExceptionMappingFilter]\n      .filter[LoggingMDCFilter]\n      .filter[TraceIdMDCFilter]\n      .filter[ThriftMDCFilter]\n      .filter[MtlsServerSessionTrackerFilter]\n      .filter[StatsFilter]\n      .filter[Filter.TypeAgnostic, DarkTrafficFilterType]\n      .add[PushServiceController]\n  }\n\n  override def configureHttp(router: HttpRouter): Unit =\n    router\n      .filter[CommonFilters]\n\n  override protected def start(): Unit = {\n    MRLoggerGlobalVariables.setRequiredFlags(\n      traceLogFlag = injector.instance[Boolean](Flags.named(FlagModule.mrLoggerIsTraceAll.name)),\n      nthLogFlag = injector.instance[Boolean](Flags.named(FlagModule.mrLoggerNthLog.name)),\n      nthLogValFlag = injector.instance[Long](Flags.named(FlagModule.mrLoggerNthVal.name))\n    )\n  }\n\n  override protected def warmup(): Unit = {\n    handle[PushMixerThriftServerWarmupHandler]()\n  }\n\n  override protected def configureLoggerFactories(): Unit = {\n    loggerFactories.foreach { _() }\n  }\n\n  override def loggerFactories: List[LoggerFactory] = {\n    val scribeScope = statsReceiver.scope(\"scribe\")\n    List(\n      LoggerFactory(\n        level = Some(levelFlag()),\n        handlers = handlers\n      ),\n      LoggerFactory(\n        node = \"request_scribe\",\n        level = Some(Level.INFO),\n        useParents = false,\n        handlers = QueueingHandler(\n          maxQueueSize = 10000,\n          handler = ScribeHandler(\n            category = \"frigate_pushservice_log\",\n            formatter = BareFormatter,\n            statsReceiver = scribeScope.scope(\"frigate_pushservice_log\")\n          )\n        ) :: Nil\n      ),\n      LoggerFactory(\n        node = \"notification_scribe\",\n        level = Some(Level.INFO),\n        useParents = false,\n        handlers = QueueingHandler(\n          maxQueueSize = 10000,\n          handler = ScribeHandler(\n            category = \"frigate_notifier\",\n            formatter = BareFormatter,\n            statsReceiver = scribeScope.scope(\"frigate_notifier\")\n          )\n        ) :: Nil\n      ),\n      LoggerFactory(\n        node = \"push_scribe\",\n        level = Some(Level.INFO),\n        useParents = false,\n        handlers = QueueingHandler(\n          maxQueueSize = 10000,\n          handler = ScribeHandler(\n            category = \"test_frigate_push\",\n            formatter = BareFormatter,\n            statsReceiver = scribeScope.scope(\"test_frigate_push\")\n          )\n        ) :: Nil\n      ),\n      LoggerFactory(\n        node = \"push_subsample_scribe\",\n        level = Some(Level.INFO),\n        useParents = false,\n        handlers = QueueingHandler(\n          maxQueueSize = 2500,\n          handler = ScribeHandler(\n            category = \"magicrecs_candidates_subsample_scribe\",\n            maxMessagesPerTransaction = 250,\n            maxMessagesToBuffer = 2500,\n            formatter = BareFormatter,\n            statsReceiver = scribeScope.scope(\"magicrecs_candidates_subsample_scribe\")\n          )\n        ) :: Nil\n      ),\n      LoggerFactory(\n        node = \"mr_request_scribe\",\n        level = Some(Level.INFO),\n        useParents = false,\n        handlers = QueueingHandler(\n          maxQueueSize = 2500,\n          handler = ScribeHandler(\n            category = \"mr_request_scribe\",\n            maxMessagesPerTransaction = 250,\n            maxMessagesToBuffer = 2500,\n            formatter = BareFormatter,\n            statsReceiver = scribeScope.scope(\"mr_request_scribe\")\n          )\n        ) :: Nil\n      ),\n      LoggerFactory(\n        node = \"high_quality_candidates_scribe\",\n        level = Some(Level.INFO),\n        useParents = false,\n        handlers = QueueingHandler(\n          maxQueueSize = 2500,\n          handler = ScribeHandler(\n            category = \"frigate_high_quality_candidates_log\",\n            maxMessagesPerTransaction = 250,\n            maxMessagesToBuffer = 2500,\n            formatter = BareFormatter,\n            statsReceiver = scribeScope.scope(\"high_quality_candidates_scribe\")\n          )\n        ) :: Nil\n      ),\n    )\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/ContentRecommenderMixerAdaptor.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.contentrecommender.thriftscala.MetricTag\nimport com.twitter.cr_mixer.thriftscala.CrMixerTweetRequest\nimport com.twitter.cr_mixer.thriftscala.NotificationsContext\nimport com.twitter.cr_mixer.thriftscala.Product\nimport com.twitter.cr_mixer.thriftscala.ProductContext\nimport com.twitter.cr_mixer.thriftscala.{MetricTag => CrMixerMetricTag}\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.AlgorithmScore\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.CandidateSourceEligible\nimport com.twitter.frigate.common.base.CrMixerCandidate\nimport com.twitter.frigate.common.base.TopicCandidate\nimport com.twitter.frigate.common.base.TopicProofTweetCandidate\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.predicate.CommonOutNetworkTweetCandidatesSourcePredicates.filterOutInNetworkTweets\nimport com.twitter.frigate.common.predicate.CommonOutNetworkTweetCandidatesSourcePredicates.filterOutReplyTweet\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.store.CrMixerTweetStore\nimport com.twitter.frigate.pushservice.store.UttEntityHydrationStore\nimport com.twitter.frigate.pushservice.util.AdaptorUtils\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.frigate.pushservice.util.TopicsUtil\nimport com.twitter.frigate.pushservice.util.TweetWithTopicProof\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.product_mixer.core.thriftscala.ClientContext\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.topiclisting.utt.LocalizedEntity\nimport com.twitter.tsp.thriftscala.TopicSocialProofRequest\nimport com.twitter.tsp.thriftscala.TopicSocialProofResponse\nimport com.twitter.util.Future\nimport scala.collection.Map\n\ncase class ContentRecommenderMixerAdaptor(\n  crMixerTweetStore: CrMixerTweetStore,\n  tweetyPieStore: ReadableStore[Long, TweetyPieResult],\n  edgeStore: ReadableStore[RelationEdge, Boolean],\n  topicSocialProofServiceStore: ReadableStore[TopicSocialProofRequest, TopicSocialProofResponse],\n  uttEntityHydrationStore: UttEntityHydrationStore,\n  globalStats: StatsReceiver)\n    extends CandidateSource[Target, RawCandidate]\n    with CandidateSourceEligible[Target, RawCandidate] {\n\n  override val name: String = this.getClass.getSimpleName\n\n  private[this] val stats = globalStats.scope(\"ContentRecommenderMixerAdaptor\")\n  private[this] val numOfValidAuthors = stats.stat(\"num_of_valid_authors\")\n  private[this] val numOutOfMaximumDropped = stats.stat(\"dropped_due_out_of_maximum\")\n  private[this] val totalInputRecs = stats.counter(\"input_recs\")\n  private[this] val totalOutputRecs = stats.stat(\"output_recs\")\n  private[this] val totalRequests = stats.counter(\"total_requests\")\n  private[this] val nonReplyTweetsCounter = stats.counter(\"non_reply_tweets\")\n  private[this] val totalOutNetworkRecs = stats.counter(\"out_network_tweets\")\n  private[this] val totalInNetworkRecs = stats.counter(\"in_network_tweets\")\n\n  /**\n   * Builds OON raw candidates based on input OON Tweets\n   */\n  def buildOONRawCandidates(\n    inputTarget: Target,\n    oonTweets: Seq[TweetyPieResult],\n    tweetScoreMap: Map[Long, Double],\n    tweetIdToTagsMap: Map[Long, Seq[CrMixerMetricTag]],\n    maxNumOfCandidates: Int\n  ): Option[Seq[RawCandidate]] = {\n    val cands = oonTweets.flatMap { tweetResult =>\n      val tweetId = tweetResult.tweet.id\n      generateOONRawCandidate(\n        inputTarget,\n        tweetId,\n        Some(tweetResult),\n        tweetScoreMap,\n        tweetIdToTagsMap\n      )\n    }\n\n    val candidates = restrict(\n      maxNumOfCandidates,\n      cands,\n      numOutOfMaximumDropped,\n      totalOutputRecs\n    )\n\n    Some(candidates)\n  }\n\n  /**\n   * Builds a single RawCandidate With TopicProofTweetCandidate\n   */\n  def buildTopicTweetRawCandidate(\n    inputTarget: Target,\n    tweetWithTopicProof: TweetWithTopicProof,\n    localizedEntity: LocalizedEntity,\n    tags: Option[Seq[MetricTag]],\n  ): RawCandidate with TopicProofTweetCandidate = {\n    new RawCandidate with TopicProofTweetCandidate {\n      override def target: Target = inputTarget\n      override def topicListingSetting: Option[String] = Some(\n        tweetWithTopicProof.topicListingSetting)\n      override def tweetId: Long = tweetWithTopicProof.tweetId\n      override def tweetyPieResult: Option[TweetyPieResult] = Some(\n        tweetWithTopicProof.tweetyPieResult)\n      override def semanticCoreEntityId: Option[Long] = Some(tweetWithTopicProof.topicId)\n      override def localizedUttEntity: Option[LocalizedEntity] = Some(localizedEntity)\n      override def algorithmCR: Option[String] = tweetWithTopicProof.algorithmCR\n      override def tagsCR: Option[Seq[MetricTag]] = tags\n      override def isOutOfNetwork: Boolean = tweetWithTopicProof.isOON\n    }\n  }\n\n  /**\n   * Takes a group of TopicTweets and transforms them into RawCandidates\n   */\n  def buildTopicTweetRawCandidates(\n    inputTarget: Target,\n    topicProofCandidates: Seq[TweetWithTopicProof],\n    tweetIdToTagsMap: Map[Long, Seq[CrMixerMetricTag]],\n    maxNumberOfCands: Int\n  ): Future[Option[Seq[RawCandidate]]] = {\n    val semanticCoreEntityIds = topicProofCandidates\n      .map(_.topicId)\n      .toSet\n\n    TopicsUtil\n      .getLocalizedEntityMap(inputTarget, semanticCoreEntityIds, uttEntityHydrationStore)\n      .map { localizedEntityMap =>\n        val rawCandidates = topicProofCandidates.collect {\n          case topicSocialProof: TweetWithTopicProof\n              if localizedEntityMap.contains(topicSocialProof.topicId) =>\n            // Once we deprecate CR calls, we should replace this code to use the CrMixerMetricTag\n            val tags = tweetIdToTagsMap.get(topicSocialProof.tweetId).map {\n              _.flatMap { tag => MetricTag.get(tag.value) }\n            }\n            buildTopicTweetRawCandidate(\n              inputTarget,\n              topicSocialProof,\n              localizedEntityMap(topicSocialProof.topicId),\n              tags\n            )\n        }\n\n        val candResult = restrict(\n          maxNumberOfCands,\n          rawCandidates,\n          numOutOfMaximumDropped,\n          totalOutputRecs\n        )\n\n        Some(candResult)\n      }\n  }\n\n  private def generateOONRawCandidate(\n    inputTarget: Target,\n    id: Long,\n    result: Option[TweetyPieResult],\n    tweetScoreMap: Map[Long, Double],\n    tweetIdToTagsMap: Map[Long, Seq[CrMixerMetricTag]]\n  ): Option[RawCandidate with TweetCandidate] = {\n    val tagsFromCR = tweetIdToTagsMap.get(id).map { _.flatMap { tag => MetricTag.get(tag.value) } }\n    val candidate = new RawCandidate with CrMixerCandidate with TopicCandidate with AlgorithmScore {\n      override val tweetId = id\n      override val target = inputTarget\n      override val tweetyPieResult = result\n      override val localizedUttEntity = None\n      override val semanticCoreEntityId = None\n      override def commonRecType =\n        getMediaBasedCRT(\n          CommonRecommendationType.TwistlyTweet,\n          CommonRecommendationType.TwistlyPhoto,\n          CommonRecommendationType.TwistlyVideo)\n      override def tagsCR = tagsFromCR\n      override def algorithmScore = tweetScoreMap.get(id)\n      override def algorithmCR = None\n    }\n    Some(candidate)\n  }\n\n  private def restrict(\n    maxNumToReturn: Int,\n    candidates: Seq[RawCandidate],\n    numOutOfMaximumDropped: Stat,\n    totalOutputRecs: Stat\n  ): Seq[RawCandidate] = {\n    val newCandidates = candidates.take(maxNumToReturn)\n    val numDropped = candidates.length - newCandidates.length\n    numOutOfMaximumDropped.add(numDropped)\n    totalOutputRecs.add(newCandidates.size)\n    newCandidates\n  }\n\n  private def buildCrMixerRequest(\n    target: Target,\n    countryCode: Option[String],\n    language: Option[String],\n    seenTweets: Seq[Long]\n  ): CrMixerTweetRequest = {\n    CrMixerTweetRequest(\n      clientContext = ClientContext(\n        userId = Some(target.targetId),\n        countryCode = countryCode,\n        languageCode = language\n      ),\n      product = Product.Notifications,\n      productContext = Some(ProductContext.NotificationsContext(NotificationsContext())),\n      excludedTweetIds = Some(seenTweets)\n    )\n  }\n\n  private def selectCandidatesToSendBasedOnSettings(\n    isRecommendationsEligible: Boolean,\n    isTopicsEligible: Boolean,\n    oonRawCandidates: Option[Seq[RawCandidate]],\n    topicTweetCandidates: Option[Seq[RawCandidate]]\n  ): Option[Seq[RawCandidate]] = {\n    if (isRecommendationsEligible && isTopicsEligible) {\n      Some(topicTweetCandidates.getOrElse(Seq.empty) ++ oonRawCandidates.getOrElse(Seq.empty))\n    } else if (isRecommendationsEligible) {\n      oonRawCandidates\n    } else if (isTopicsEligible) {\n      topicTweetCandidates\n    } else None\n  }\n\n  override def get(target: Target): Future[Option[Seq[RawCandidate]]] = {\n    Future\n      .join(\n        target.seenTweetIds,\n        target.countryCode,\n        target.inferredUserDeviceLanguage,\n        PushDeviceUtil.isTopicsEligible(target),\n        PushDeviceUtil.isRecommendationsEligible(target)\n      ).flatMap {\n        case (seenTweets, countryCode, language, isTopicsEligible, isRecommendationsEligible) =>\n          val request = buildCrMixerRequest(target, countryCode, language, seenTweets)\n          crMixerTweetStore.getTweetRecommendations(request).flatMap {\n            case Some(response) =>\n              totalInputRecs.incr(response.tweets.size)\n              totalRequests.incr()\n              AdaptorUtils\n                .getTweetyPieResults(\n                  response.tweets.map(_.tweetId).toSet,\n                  tweetyPieStore).flatMap { tweetyPieResultMap =>\n                  filterOutInNetworkTweets(\n                    target,\n                    filterOutReplyTweet(tweetyPieResultMap.toMap, nonReplyTweetsCounter),\n                    edgeStore,\n                    numOfValidAuthors).flatMap {\n                    outNetworkTweetsWithId: Seq[(Long, TweetyPieResult)] =>\n                      totalOutNetworkRecs.incr(outNetworkTweetsWithId.size)\n                      totalInNetworkRecs.incr(response.tweets.size - outNetworkTweetsWithId.size)\n                      val outNetworkTweets: Seq[TweetyPieResult] = outNetworkTweetsWithId.map {\n                        case (_, tweetyPieResult) => tweetyPieResult\n                      }\n\n                      val tweetIdToTagsMap = response.tweets.map { tweet =>\n                        tweet.tweetId -> tweet.metricTags.getOrElse(Seq.empty)\n                      }.toMap\n\n                      val tweetScoreMap = response.tweets.map { tweet =>\n                        tweet.tweetId -> tweet.score\n                      }.toMap\n\n                      val maxNumOfCandidates =\n                        target.params(PushFeatureSwitchParams.NumberOfMaxCrMixerCandidatesParam)\n\n                      val oonRawCandidates =\n                        buildOONRawCandidates(\n                          target,\n                          outNetworkTweets,\n                          tweetScoreMap,\n                          tweetIdToTagsMap,\n                          maxNumOfCandidates)\n\n                      TopicsUtil\n                        .getTopicSocialProofs(\n                          target,\n                          outNetworkTweets,\n                          topicSocialProofServiceStore,\n                          edgeStore,\n                          PushFeatureSwitchParams.TopicProofTweetCandidatesTopicScoreThreshold).flatMap {\n                          tweetsWithTopicProof =>\n                            buildTopicTweetRawCandidates(\n                              target,\n                              tweetsWithTopicProof,\n                              tweetIdToTagsMap,\n                              maxNumOfCandidates)\n                        }.map { topicTweetCandidates =>\n                          selectCandidatesToSendBasedOnSettings(\n                            isRecommendationsEligible,\n                            isTopicsEligible,\n                            oonRawCandidates,\n                            topicTweetCandidates)\n                        }\n                  }\n                }\n            case _ => Future.None\n          }\n      }\n  }\n\n  /**\n   * For a user to be available the following news to happen\n   */\n  override def isCandidateSourceAvailable(target: Target): Future[Boolean] = {\n    Future\n      .join(\n        PushDeviceUtil.isRecommendationsEligible(target),\n        PushDeviceUtil.isTopicsEligible(target)\n      ).map {\n        case (isRecommendationsEligible, isTopicsEligible) =>\n          (isRecommendationsEligible || isTopicsEligible) &&\n            target.params(PushParams.ContentRecommenderMixerAdaptorDecider)\n      }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/EarlyBirdFirstDegreeCandidateAdaptor.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.candidate._\nimport com.twitter.frigate.common.predicate.CommonOutNetworkTweetCandidatesSourcePredicates.filterOutReplyTweet\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.hermit.store.tweetypie.UserTweet\nimport com.twitter.recos.recos_common.thriftscala.SocialProofType\nimport com.twitter.search.common.features.thriftscala.ThriftSearchResultFeatures\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport scala.collection.Map\n\ncase class EarlyBirdFirstDegreeCandidateAdaptor(\n  earlyBirdFirstDegreeCandidates: CandidateSource[\n    EarlybirdCandidateSource.Query,\n    EarlybirdCandidate\n  ],\n  tweetyPieStore: ReadableStore[Long, TweetyPieResult],\n  tweetyPieStoreNoVF: ReadableStore[Long, TweetyPieResult],\n  userTweetTweetyPieStore: ReadableStore[UserTweet, TweetyPieResult],\n  maxResultsParam: Param[Int],\n  globalStats: StatsReceiver)\n    extends CandidateSource[Target, RawCandidate]\n    with CandidateSourceEligible[Target, RawCandidate] {\n\n  type EBCandidate = EarlybirdCandidate with TweetDetails\n  private val stats = globalStats.scope(\"EarlyBirdFirstDegreeAdaptor\")\n  private val earlyBirdCandsStat: Stat = stats.stat(\"early_bird_cands_dist\")\n  private val emptyEarlyBirdCands = stats.counter(\"empty_early_bird_candidates\")\n  private val seedSetEmpty = stats.counter(\"empty_seedset\")\n  private val seenTweetsStat = stats.stat(\"filtered_by_seen_tweets\")\n  private val emptyTweetyPieResult = stats.stat(\"empty_tweetypie_result\")\n  private val nonReplyTweetsCounter = stats.counter(\"non_reply_tweets\")\n  private val enableRetweets = stats.counter(\"enable_retweets\")\n  private val f1withoutSocialContexts = stats.counter(\"f1_without_social_context\")\n  private val userTweetTweetyPieStoreCounter = stats.counter(\"user_tweet_tweetypie_store\")\n\n  override val name: String = earlyBirdFirstDegreeCandidates.name\n\n  private def getAllSocialContextActions(\n    socialProofTypes: Seq[(SocialProofType, Seq[Long])]\n  ): Seq[SocialContextAction] = {\n    socialProofTypes.flatMap {\n      case (SocialProofType.Favorite, scIds) =>\n        scIds.map { scId =>\n          SocialContextAction(\n            scId,\n            Time.now.inMilliseconds,\n            socialContextActionType = Some(SocialContextActionType.Favorite)\n          )\n        }\n      case (SocialProofType.Retweet, scIds) =>\n        scIds.map { scId =>\n          SocialContextAction(\n            scId,\n            Time.now.inMilliseconds,\n            socialContextActionType = Some(SocialContextActionType.Retweet)\n          )\n        }\n      case (SocialProofType.Reply, scIds) =>\n        scIds.map { scId =>\n          SocialContextAction(\n            scId,\n            Time.now.inMilliseconds,\n            socialContextActionType = Some(SocialContextActionType.Reply)\n          )\n        }\n      case (SocialProofType.Tweet, scIds) =>\n        scIds.map { scId =>\n          SocialContextAction(\n            scId,\n            Time.now.inMilliseconds,\n            socialContextActionType = Some(SocialContextActionType.Tweet)\n          )\n        }\n      case _ => Nil\n    }\n  }\n\n  private def generateRetweetCandidate(\n    inputTarget: Target,\n    candidate: EBCandidate,\n    scIds: Seq[Long],\n    socialProofTypes: Seq[(SocialProofType, Seq[Long])]\n  ): RawCandidate = {\n    val scActions = scIds.map { scId => SocialContextAction(scId, Time.now.inMilliseconds) }\n    new RawCandidate with TweetRetweetCandidate with EarlybirdTweetFeatures {\n      override val socialContextActions = scActions\n      override val socialContextAllTypeActions = getAllSocialContextActions(socialProofTypes)\n      override val tweetId = candidate.tweetId\n      override val target = inputTarget\n      override val tweetyPieResult = candidate.tweetyPieResult\n      override val features = candidate.features\n    }\n  }\n\n  private def generateF1CandidateWithoutSocialContext(\n    inputTarget: Target,\n    candidate: EBCandidate\n  ): RawCandidate = {\n    f1withoutSocialContexts.incr()\n    new RawCandidate with F1FirstDegree with EarlybirdTweetFeatures {\n      override val tweetId = candidate.tweetId\n      override val target = inputTarget\n      override val tweetyPieResult = candidate.tweetyPieResult\n      override val features = candidate.features\n    }\n  }\n\n  private def generateEarlyBirdCandidate(\n    id: Long,\n    result: Option[TweetyPieResult],\n    ebFeatures: Option[ThriftSearchResultFeatures]\n  ): EBCandidate = {\n    new EarlybirdCandidate with TweetDetails {\n      override val tweetyPieResult: Option[TweetyPieResult] = result\n      override val tweetId: Long = id\n      override val features: Option[ThriftSearchResultFeatures] = ebFeatures\n    }\n  }\n\n  private def filterOutSeenTweets(seenTweetIds: Seq[Long], inputTweetIds: Seq[Long]): Seq[Long] = {\n    inputTweetIds.filterNot(seenTweetIds.contains)\n  }\n\n  private def filterInvalidTweets(\n    tweetIds: Seq[Long],\n    target: Target\n  ): Future[Seq[(Long, TweetyPieResult)]] = {\n\n    val resMap = {\n      if (target.params(PushFeatureSwitchParams.EnableF1FromProtectedTweetAuthors)) {\n        userTweetTweetyPieStoreCounter.incr()\n        val keys = tweetIds.map { tweetId =>\n          UserTweet(tweetId, Some(target.targetId))\n        }\n\n        userTweetTweetyPieStore\n          .multiGet(keys.toSet).map {\n            case (userTweet, resultFut) =>\n              userTweet.tweetId -> resultFut\n          }.toMap\n      } else {\n        (target.params(PushFeatureSwitchParams.EnableVFInTweetypie) match {\n          case true => tweetyPieStore\n          case false => tweetyPieStoreNoVF\n        }).multiGet(tweetIds.toSet)\n      }\n    }\n    Future.collect(resMap).map { tweetyPieResultMap =>\n      val cands = filterOutReplyTweet(tweetyPieResultMap, nonReplyTweetsCounter).collect {\n        case (id: Long, Some(result)) =>\n          id -> result\n      }\n\n      emptyTweetyPieResult.add(tweetyPieResultMap.size - cands.size)\n      cands.toSeq\n    }\n  }\n\n  private def getEBRetweetCandidates(\n    inputTarget: Target,\n    retweets: Seq[(Long, TweetyPieResult)]\n  ): Seq[RawCandidate] = {\n    retweets.flatMap {\n      case (_, tweetypieResult) =>\n        tweetypieResult.tweet.coreData.flatMap { coreData =>\n          tweetypieResult.sourceTweet.map { sourceTweet =>\n            val tweetId = sourceTweet.id\n            val scId = coreData.userId\n            val socialProofTypes = Seq((SocialProofType.Retweet, Seq(scId)))\n            val candidate = generateEarlyBirdCandidate(\n              tweetId,\n              Some(TweetyPieResult(sourceTweet, None, None)),\n              None\n            )\n            generateRetweetCandidate(\n              inputTarget,\n              candidate,\n              Seq(scId),\n              socialProofTypes\n            )\n          }\n        }\n    }\n  }\n\n  private def getEBFirstDegreeCands(\n    tweets: Seq[(Long, TweetyPieResult)],\n    ebTweetIdMap: Map[Long, Option[ThriftSearchResultFeatures]]\n  ): Seq[EBCandidate] = {\n    tweets.map {\n      case (id, tweetypieResult) =>\n        val features = ebTweetIdMap.getOrElse(id, None)\n        generateEarlyBirdCandidate(id, Some(tweetypieResult), features)\n    }\n  }\n\n  /**\n   * Returns a combination of raw candidates made of: f1 recs, topic social proof recs, sc recs and retweet candidates\n   */\n  def buildRawCandidates(\n    inputTarget: Target,\n    firstDegreeCandidates: Seq[EBCandidate],\n    retweetCandidates: Seq[RawCandidate]\n  ): Seq[RawCandidate] = {\n    val hydratedF1Recs =\n      firstDegreeCandidates.map(generateF1CandidateWithoutSocialContext(inputTarget, _))\n    hydratedF1Recs ++ retweetCandidates\n  }\n\n  override def get(inputTarget: Target): Future[Option[Seq[RawCandidate]]] = {\n    inputTarget.seedsWithWeight.flatMap { seedsetOpt =>\n      val seedsetMap = seedsetOpt.getOrElse(Map.empty)\n\n      if (seedsetMap.isEmpty) {\n        seedSetEmpty.incr()\n        Future.None\n      } else {\n        val maxResultsToReturn = inputTarget.params(maxResultsParam)\n        val maxTweetAge = inputTarget.params(PushFeatureSwitchParams.F1CandidateMaxTweetAgeParam)\n        val earlybirdQuery = EarlybirdCandidateSource.Query(\n          maxNumResultsToReturn = maxResultsToReturn,\n          seedset = seedsetMap,\n          maxConsecutiveResultsByTheSameUser = Some(1),\n          maxTweetAge = maxTweetAge,\n          disableTimelinesMLModel = false,\n          searcherId = Some(inputTarget.targetId),\n          isProtectTweetsEnabled =\n            inputTarget.params(PushFeatureSwitchParams.EnableF1FromProtectedTweetAuthors),\n          followedUserIds = Some(seedsetMap.keySet.toSeq)\n        )\n\n        Future\n          .join(inputTarget.seenTweetIds, earlyBirdFirstDegreeCandidates.get(earlybirdQuery))\n          .flatMap {\n            case (seenTweetIds, Some(candidates)) =>\n              earlyBirdCandsStat.add(candidates.size)\n\n              val ebTweetIdMap = candidates.map { cand => cand.tweetId -> cand.features }.toMap\n\n              val ebTweetIds = ebTweetIdMap.keys.toSeq\n\n              val tweetIds = filterOutSeenTweets(seenTweetIds, ebTweetIds)\n              seenTweetsStat.add(ebTweetIds.size - tweetIds.size)\n\n              filterInvalidTweets(tweetIds, inputTarget)\n                .map { validTweets =>\n                  val (retweets, tweets) = validTweets.partition {\n                    case (_, tweetypieResult) =>\n                      tweetypieResult.sourceTweet.isDefined\n                  }\n\n                  val firstDegreeCandidates = getEBFirstDegreeCands(tweets, ebTweetIdMap)\n\n                  val retweetCandidates = {\n                    if (inputTarget.params(PushParams.EarlyBirdSCBasedCandidatesParam) &&\n                      inputTarget.params(PushParams.MRTweetRetweetRecsParam)) {\n                      enableRetweets.incr()\n                      getEBRetweetCandidates(inputTarget, retweets)\n                    } else Nil\n                  }\n\n                  Some(\n                    buildRawCandidates(\n                      inputTarget,\n                      firstDegreeCandidates,\n                      retweetCandidates\n                    ))\n                }\n\n            case _ =>\n              emptyEarlyBirdCands.incr()\n              Future.None\n          }\n      }\n    }\n  }\n\n  override def isCandidateSourceAvailable(target: Target): Future[Boolean] = {\n    PushDeviceUtil.isRecommendationsEligible(target)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/ExploreVideoTweetCandidateAdaptor.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.explore_ranker.thriftscala.ExploreRankerProductResponse\nimport com.twitter.explore_ranker.thriftscala.ExploreRankerRequest\nimport com.twitter.explore_ranker.thriftscala.ExploreRankerResponse\nimport com.twitter.explore_ranker.thriftscala.ExploreRecommendation\nimport com.twitter.explore_ranker.thriftscala.ImmersiveRecsResponse\nimport com.twitter.explore_ranker.thriftscala.ImmersiveRecsResult\nimport com.twitter.explore_ranker.thriftscala.NotificationsVideoRecs\nimport com.twitter.explore_ranker.thriftscala.Product\nimport com.twitter.explore_ranker.thriftscala.ProductContext\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.CandidateSourceEligible\nimport com.twitter.frigate.common.base.OutOfNetworkTweetCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.util.AdaptorUtils\nimport com.twitter.frigate.pushservice.util.MediaCRT\nimport com.twitter.frigate.pushservice.util.PushAdaptorUtil\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.product_mixer.core.thriftscala.ClientContext\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\ncase class ExploreVideoTweetCandidateAdaptor(\n  exploreRankerStore: ReadableStore[ExploreRankerRequest, ExploreRankerResponse],\n  tweetyPieStore: ReadableStore[Long, TweetyPieResult],\n  globalStats: StatsReceiver)\n    extends CandidateSource[Target, RawCandidate]\n    with CandidateSourceEligible[Target, RawCandidate] {\n\n  override def name: String = this.getClass.getSimpleName\n  private[this] val stats = globalStats.scope(\"ExploreVideoTweetCandidateAdaptor\")\n  private[this] val totalInputRecs = stats.stat(\"input_recs\")\n  private[this] val totalRequests = stats.counter(\"total_requests\")\n  private[this] val totalEmptyResponse = stats.counter(\"total_empty_response\")\n\n  private def buildExploreRankerRequest(\n    target: Target,\n    countryCode: Option[String],\n    language: Option[String],\n  ): ExploreRankerRequest = {\n    ExploreRankerRequest(\n      clientContext = ClientContext(\n        userId = Some(target.targetId),\n        countryCode = countryCode,\n        languageCode = language,\n      ),\n      product = Product.NotificationsVideoRecs,\n      productContext = Some(ProductContext.NotificationsVideoRecs(NotificationsVideoRecs())),\n      maxResults = Some(target.params(PushFeatureSwitchParams.MaxExploreVideoTweets))\n    )\n  }\n\n  override def get(target: Target): Future[Option[Seq[RawCandidate]]] = {\n    Future\n      .join(\n        target.countryCode,\n        target.inferredUserDeviceLanguage\n      ).flatMap {\n        case (countryCode, language) =>\n          val request = buildExploreRankerRequest(target, countryCode, language)\n          exploreRankerStore.get(request).flatMap {\n            case Some(response) =>\n              val exploreResonseTweetIds = response match {\n                case ExploreRankerResponse(ExploreRankerProductResponse\n                      .ImmersiveRecsResponse(ImmersiveRecsResponse(immersiveRecsResult))) =>\n                  immersiveRecsResult.collect {\n                    case ImmersiveRecsResult(ExploreRecommendation\n                          .ExploreTweetRecommendation(exploreTweetRecommendation)) =>\n                      exploreTweetRecommendation.tweetId\n                  }\n                case _ =>\n                  Seq.empty\n              }\n\n              totalInputRecs.add(exploreResonseTweetIds.size)\n              totalRequests.incr()\n              AdaptorUtils\n                .getTweetyPieResults(exploreResonseTweetIds.toSet, tweetyPieStore).map {\n                  tweetyPieResultMap =>\n                    val candidates = tweetyPieResultMap.values.flatten\n                      .map(buildVideoRawCandidates(target, _))\n                    Some(candidates.toSeq)\n                }\n            case _ =>\n              totalEmptyResponse.incr()\n              Future.None\n          }\n        case _ =>\n          Future.None\n      }\n  }\n\n  override def isCandidateSourceAvailable(target: Target): Future[Boolean] = {\n    PushDeviceUtil.isRecommendationsEligible(target).map { userRecommendationsEligible =>\n      userRecommendationsEligible && target.params(PushFeatureSwitchParams.EnableExploreVideoTweets)\n    }\n  }\n  private def buildVideoRawCandidates(\n    target: Target,\n    tweetyPieResult: TweetyPieResult\n  ): RawCandidate with OutOfNetworkTweetCandidate = {\n    PushAdaptorUtil.generateOutOfNetworkTweetCandidates(\n      inputTarget = target,\n      id = tweetyPieResult.tweet.id,\n      mediaCRT = MediaCRT(\n        CommonRecommendationType.ExploreVideoTweet,\n        CommonRecommendationType.ExploreVideoTweet,\n        CommonRecommendationType.ExploreVideoTweet\n      ),\n      result = Some(tweetyPieResult),\n      localizedEntity = None\n    )\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/FRSTweetCandidateAdaptor.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.cr_mixer.thriftscala.FrsTweetRequest\nimport com.twitter.cr_mixer.thriftscala.NotificationsContext\nimport com.twitter.cr_mixer.thriftscala.Product\nimport com.twitter.cr_mixer.thriftscala.ProductContext\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.CandidateSourceEligible\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.predicate.CommonOutNetworkTweetCandidatesSourcePredicates.filterOutReplyTweet\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.store.CrMixerTweetStore\nimport com.twitter.frigate.pushservice.store.UttEntityHydrationStore\nimport com.twitter.frigate.pushservice.util.MediaCRT\nimport com.twitter.frigate.pushservice.util.PushAdaptorUtil\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.frigate.pushservice.util.TopicsUtil\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.hermit.constants.AlgorithmFeedbackTokens\nimport com.twitter.hermit.model.Algorithm.Algorithm\nimport com.twitter.hermit.model.Algorithm.CrowdSearchAccounts\nimport com.twitter.hermit.model.Algorithm.ForwardEmailBook\nimport com.twitter.hermit.model.Algorithm.ForwardPhoneBook\nimport com.twitter.hermit.model.Algorithm.ReverseEmailBookIbis\nimport com.twitter.hermit.model.Algorithm.ReversePhoneBook\nimport com.twitter.hermit.store.tweetypie.UserTweet\nimport com.twitter.product_mixer.core.thriftscala.ClientContext\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.tsp.thriftscala.TopicSocialProofRequest\nimport com.twitter.tsp.thriftscala.TopicSocialProofResponse\nimport com.twitter.util.Future\n\nobject FRSAlgorithmFeedbackTokenUtil {\n  private val crtsByAlgoToken = Map(\n    getAlgorithmToken(ReverseEmailBookIbis) -> CommonRecommendationType.ReverseAddressbookTweet,\n    getAlgorithmToken(ReversePhoneBook) -> CommonRecommendationType.ReverseAddressbookTweet,\n    getAlgorithmToken(ForwardEmailBook) -> CommonRecommendationType.ForwardAddressbookTweet,\n    getAlgorithmToken(ForwardPhoneBook) -> CommonRecommendationType.ForwardAddressbookTweet,\n    getAlgorithmToken(CrowdSearchAccounts) -> CommonRecommendationType.CrowdSearchTweet\n  )\n\n  def getAlgorithmToken(algorithm: Algorithm): Int = {\n    AlgorithmFeedbackTokens.AlgorithmToFeedbackTokenMap(algorithm)\n  }\n\n  def getCRTForAlgoToken(algorithmToken: Int): Option[CommonRecommendationType] = {\n    crtsByAlgoToken.get(algorithmToken)\n  }\n}\n\ncase class FRSTweetCandidateAdaptor(\n  crMixerTweetStore: CrMixerTweetStore,\n  tweetyPieStore: ReadableStore[Long, TweetyPieResult],\n  tweetyPieStoreNoVF: ReadableStore[Long, TweetyPieResult],\n  userTweetTweetyPieStore: ReadableStore[UserTweet, TweetyPieResult],\n  uttEntityHydrationStore: UttEntityHydrationStore,\n  topicSocialProofServiceStore: ReadableStore[TopicSocialProofRequest, TopicSocialProofResponse],\n  globalStats: StatsReceiver)\n    extends CandidateSource[Target, RawCandidate]\n    with CandidateSourceEligible[Target, RawCandidate] {\n\n  private val stats = globalStats.scope(this.getClass.getSimpleName)\n  private val crtStats = stats.scope(\"CandidateDistribution\")\n  private val totalRequests = stats.counter(\"total_requests\")\n\n  // Candidate Distribution stats\n  private val reverseAddressbookCounter = crtStats.counter(\"reverse_addressbook\")\n  private val forwardAddressbookCounter = crtStats.counter(\"forward_addressbook\")\n  private val frsTweetCounter = crtStats.counter(\"frs_tweet\")\n  private val nonReplyTweetsCounter = stats.counter(\"non_reply_tweets\")\n  private val crtToCounterMapping: Map[CommonRecommendationType, Counter] = Map(\n    CommonRecommendationType.ReverseAddressbookTweet -> reverseAddressbookCounter,\n    CommonRecommendationType.ForwardAddressbookTweet -> forwardAddressbookCounter,\n    CommonRecommendationType.FrsTweet -> frsTweetCounter\n  )\n\n  private val emptyTweetyPieResult = stats.stat(\"empty_tweetypie_result\")\n\n  private[this] val numberReturnedCandidates = stats.stat(\"returned_candidates_from_earlybird\")\n  private[this] val numberCandidateWithTopic: Counter = stats.counter(\"num_can_with_topic\")\n  private[this] val numberCandidateWithoutTopic: Counter = stats.counter(\"num_can_without_topic\")\n\n  private val userTweetTweetyPieStoreCounter = stats.counter(\"user_tweet_tweetypie_store\")\n\n  override val name: String = this.getClass.getSimpleName\n\n  private def filterInvalidTweets(\n    tweetIds: Seq[Long],\n    target: Target\n  ): Future[Map[Long, TweetyPieResult]] = {\n    val resMap = {\n      if (target.params(PushFeatureSwitchParams.EnableF1FromProtectedTweetAuthors)) {\n        userTweetTweetyPieStoreCounter.incr()\n        val keys = tweetIds.map { tweetId =>\n          UserTweet(tweetId, Some(target.targetId))\n        }\n        userTweetTweetyPieStore\n          .multiGet(keys.toSet).map {\n            case (userTweet, resultFut) =>\n              userTweet.tweetId -> resultFut\n          }.toMap\n      } else {\n        (if (target.params(PushFeatureSwitchParams.EnableVFInTweetypie)) {\n           tweetyPieStore\n         } else {\n           tweetyPieStoreNoVF\n         }).multiGet(tweetIds.toSet)\n      }\n    }\n\n    Future.collect(resMap).map { tweetyPieResultMap =>\n      // Filter out replies and generate earlybird candidates only for non-empty tweetypie result\n      val cands = filterOutReplyTweet(tweetyPieResultMap, nonReplyTweetsCounter).collect {\n        case (id: Long, Some(result)) =>\n          id -> result\n      }\n\n      emptyTweetyPieResult.add(tweetyPieResultMap.size - cands.size)\n      cands\n    }\n  }\n\n  private def buildRawCandidates(\n    target: Target,\n    ebCandidates: Seq[FRSTweetCandidate]\n  ): Future[Option[Seq[RawCandidate with TweetCandidate]]] = {\n\n    val enableTopic = target.params(PushFeatureSwitchParams.EnableFrsTweetCandidatesTopicAnnotation)\n    val topicScoreThre =\n      target.params(PushFeatureSwitchParams.FrsTweetCandidatesTopicScoreThreshold)\n\n    val ebTweets = ebCandidates.map { ebCandidate =>\n      ebCandidate.tweetId -> ebCandidate.tweetyPieResult\n    }.toMap\n\n    val tweetIdLocalizedEntityMapFut = TopicsUtil.getTweetIdLocalizedEntityMap(\n      target,\n      ebTweets,\n      uttEntityHydrationStore,\n      topicSocialProofServiceStore,\n      enableTopic,\n      topicScoreThre\n    )\n\n    Future.join(target.deviceInfo, tweetIdLocalizedEntityMapFut).map {\n      case (Some(deviceInfo), tweetIdLocalizedEntityMap) =>\n        val candidates = ebCandidates\n          .map { ebCandidate =>\n            val crt = ebCandidate.commonRecType\n            crtToCounterMapping.get(crt).foreach(_.incr())\n\n            val tweetId = ebCandidate.tweetId\n            val localizedEntityOpt = {\n              if (tweetIdLocalizedEntityMap\n                  .contains(tweetId) && tweetIdLocalizedEntityMap.contains(\n                  tweetId) && deviceInfo.isTopicsEligible) {\n                tweetIdLocalizedEntityMap(tweetId)\n              } else {\n                None\n              }\n            }\n\n            PushAdaptorUtil.generateOutOfNetworkTweetCandidates(\n              inputTarget = target,\n              id = ebCandidate.tweetId,\n              mediaCRT = MediaCRT(\n                crt,\n                crt,\n                crt\n              ),\n              result = ebCandidate.tweetyPieResult,\n              localizedEntity = localizedEntityOpt)\n          }.filter { candidate =>\n            // If user only has the topic setting enabled, filter out all non-topic cands\n            deviceInfo.isRecommendationsEligible || (deviceInfo.isTopicsEligible && candidate.semanticCoreEntityId.nonEmpty)\n          }\n\n        candidates.map { candidate =>\n          if (candidate.semanticCoreEntityId.nonEmpty) {\n            numberCandidateWithTopic.incr()\n          } else {\n            numberCandidateWithoutTopic.incr()\n          }\n        }\n\n        numberReturnedCandidates.add(candidates.length)\n        Some(candidates)\n      case _ => Some(Seq.empty)\n    }\n  }\n\n  def getTweetCandidatesFromCrMixer(\n    inputTarget: Target,\n    showAllResultsFromFrs: Boolean,\n  ): Future[Option[Seq[RawCandidate with TweetCandidate]]] = {\n    Future\n      .join(\n        inputTarget.seenTweetIds,\n        inputTarget.pushRecItems,\n        inputTarget.countryCode,\n        inputTarget.targetLanguage).flatMap {\n        case (seenTweetIds, pastRecItems, countryCode, language) =>\n          val pastUserRecs = pastRecItems.userIds.toSeq\n          val request = FrsTweetRequest(\n            clientContext = ClientContext(\n              userId = Some(inputTarget.targetId),\n              countryCode = countryCode,\n              languageCode = language\n            ),\n            product = Product.Notifications,\n            productContext = Some(ProductContext.NotificationsContext(NotificationsContext())),\n            excludedUserIds = Some(pastUserRecs),\n            excludedTweetIds = Some(seenTweetIds)\n          )\n          crMixerTweetStore.getFRSTweetCandidates(request).flatMap {\n            case Some(response) =>\n              val tweetIds = response.tweets.map(_.tweetId)\n              val validTweets = filterInvalidTweets(tweetIds, inputTarget)\n              validTweets.flatMap { tweetypieMap =>\n                val ebCandidates = response.tweets\n                  .map { frsTweet =>\n                    val candidateTweetId = frsTweet.tweetId\n                    val resultFromTweetyPie = tweetypieMap.get(candidateTweetId)\n                    new FRSTweetCandidate {\n                      override val tweetId = candidateTweetId\n                      override val features = None\n                      override val tweetyPieResult = resultFromTweetyPie\n                      override val feedbackToken = frsTweet.frsPrimarySource\n                      override val commonRecType: CommonRecommendationType = feedbackToken\n                        .flatMap(token =>\n                          FRSAlgorithmFeedbackTokenUtil.getCRTForAlgoToken(token)).getOrElse(\n                          CommonRecommendationType.FrsTweet)\n                    }\n                  }.filter { ebCandidate =>\n                    showAllResultsFromFrs || ebCandidate.commonRecType == CommonRecommendationType.ReverseAddressbookTweet\n                  }\n\n                numberReturnedCandidates.add(ebCandidates.length)\n                buildRawCandidates(\n                  inputTarget,\n                  ebCandidates\n                )\n              }\n            case _ => Future.None\n          }\n      }\n  }\n\n  override def get(inputTarget: Target): Future[Option[Seq[RawCandidate with TweetCandidate]]] = {\n    totalRequests.incr()\n    val enableResultsFromFrs =\n      inputTarget.params(PushFeatureSwitchParams.EnableResultFromFrsCandidates)\n    getTweetCandidatesFromCrMixer(inputTarget, enableResultsFromFrs)\n  }\n\n  override def isCandidateSourceAvailable(target: Target): Future[Boolean] = {\n    lazy val enableFrsCandidates = target.params(PushFeatureSwitchParams.EnableFrsCandidates)\n    PushDeviceUtil.isRecommendationsEligible(target).flatMap { isEnabledForRecosSetting =>\n      PushDeviceUtil.isTopicsEligible(target).map { topicSettingEnabled =>\n        val isEnabledForTopics =\n          topicSettingEnabled && target.params(\n            PushFeatureSwitchParams.EnableFrsTweetCandidatesTopicSetting)\n        (isEnabledForRecosSetting || isEnabledForTopics) && enableFrsCandidates\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/GenericCandidateAdaptor.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.candidate._\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nobject GenericCandidates {\n  type Target =\n    TargetUser\n      with UserDetails\n      with TargetDecider\n      with TargetABDecider\n      with TweetImpressionHistory\n      with HTLVisitHistory\n      with MaxTweetAge\n      with NewUserDetails\n      with FrigateHistory\n      with TargetWithSeedUsers\n}\n\ncase class GenericCandidateAdaptor(\n  genericCandidates: CandidateSource[GenericCandidates.Target, Candidate],\n  tweetyPieStore: ReadableStore[Long, TweetyPieResult],\n  tweetyPieStoreNoVF: ReadableStore[Long, TweetyPieResult],\n  stats: StatsReceiver)\n    extends CandidateSource[Target, RawCandidate]\n    with CandidateSourceEligible[Target, RawCandidate] {\n\n  override val name: String = genericCandidates.name\n\n  private def generateTweetFavCandidate(\n    _target: Target,\n    _tweetId: Long,\n    _socialContextActions: Seq[SocialContextAction],\n    socialContextActionsAllTypes: Seq[SocialContextAction],\n    _tweetyPieResult: Option[TweetyPieResult]\n  ): RawCandidate = {\n    new RawCandidate with TweetFavoriteCandidate {\n      override val socialContextActions = _socialContextActions\n      override val socialContextAllTypeActions =\n        socialContextActionsAllTypes\n      val tweetId = _tweetId\n      val target = _target\n      val tweetyPieResult = _tweetyPieResult\n    }\n  }\n\n  private def generateTweetRetweetCandidate(\n    _target: Target,\n    _tweetId: Long,\n    _socialContextActions: Seq[SocialContextAction],\n    socialContextActionsAllTypes: Seq[SocialContextAction],\n    _tweetyPieResult: Option[TweetyPieResult]\n  ): RawCandidate = {\n    new RawCandidate with TweetRetweetCandidate {\n      override val socialContextActions = _socialContextActions\n      override val socialContextAllTypeActions = socialContextActionsAllTypes\n      val tweetId = _tweetId\n      val target = _target\n      val tweetyPieResult = _tweetyPieResult\n    }\n  }\n\n  override def get(inputTarget: Target): Future[Option[Seq[RawCandidate]]] = {\n    genericCandidates.get(inputTarget).map { candidatesOpt =>\n      candidatesOpt\n        .map { candidates =>\n          val candidatesSeq =\n            candidates.collect {\n              case tweetRetweet: TweetRetweetCandidate\n                  if inputTarget.params(PushParams.MRTweetRetweetRecsParam) =>\n                generateTweetRetweetCandidate(\n                  inputTarget,\n                  tweetRetweet.tweetId,\n                  tweetRetweet.socialContextActions,\n                  tweetRetweet.socialContextAllTypeActions,\n                  tweetRetweet.tweetyPieResult)\n              case tweetFavorite: TweetFavoriteCandidate\n                  if inputTarget.params(PushParams.MRTweetFavRecsParam) =>\n                generateTweetFavCandidate(\n                  inputTarget,\n                  tweetFavorite.tweetId,\n                  tweetFavorite.socialContextActions,\n                  tweetFavorite.socialContextAllTypeActions,\n                  tweetFavorite.tweetyPieResult)\n            }\n          candidatesSeq.foreach { candidate =>\n            stats.counter(s\"${candidate.commonRecType}_count\").incr()\n          }\n          candidatesSeq\n        }\n    }\n  }\n\n  override def isCandidateSourceAvailable(target: Target): Future[Boolean] = {\n    PushDeviceUtil.isRecommendationsEligible(target).map { isAvailable =>\n      isAvailable && target.params(PushParams.GenericCandidateAdaptorDecider)\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/HighQualityTweetsAdaptor.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.CandidateSourceEligible\nimport com.twitter.frigate.common.store.interests.InterestsLookupRequestWithContext\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.HighQualityCandidateGroupEnum\nimport com.twitter.frigate.pushservice.params.HighQualityCandidateGroupEnum._\nimport com.twitter.frigate.pushservice.params.PushConstants.targetUserAgeFeatureName\nimport com.twitter.frigate.pushservice.params.PushConstants.targetUserPreferredLanguage\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FS}\nimport com.twitter.frigate.pushservice.predicate.TargetPredicates\nimport com.twitter.frigate.pushservice.util.MediaCRT\nimport com.twitter.frigate.pushservice.util.PushAdaptorUtil\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.frigate.pushservice.util.TopicsUtil\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.interests.thriftscala.InterestId.SemanticCore\nimport com.twitter.interests.thriftscala.UserInterests\nimport com.twitter.language.normalization.UserDisplayLanguage\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripDomain\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripTweet\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripTweets\nimport com.twitter.util.Future\n\nobject HighQualityTweetsHelper {\n  def getFollowedTopics(\n    target: Target,\n    interestsWithLookupContextStore: ReadableStore[\n      InterestsLookupRequestWithContext,\n      UserInterests\n    ],\n    followedTopicsStats: Stat\n  ): Future[Seq[Long]] = {\n    TopicsUtil\n      .getTopicsFollowedByUser(target, interestsWithLookupContextStore, followedTopicsStats).map {\n        userInterestsOpt =>\n          val userInterests = userInterestsOpt.getOrElse(Seq.empty)\n          val extractedTopicIds = userInterests.flatMap {\n            _.interestId match {\n              case SemanticCore(semanticCore) => Some(semanticCore.id)\n              case _ => None\n            }\n          }\n          extractedTopicIds\n      }\n  }\n\n  def getTripQueries(\n    target: Target,\n    enabledGroups: Set[HighQualityCandidateGroupEnum.Value],\n    interestsWithLookupContextStore: ReadableStore[\n      InterestsLookupRequestWithContext,\n      UserInterests\n    ],\n    sourceIds: Seq[String],\n    stat: Stat\n  ): Future[Set[TripDomain]] = {\n\n    val followedTopicIdsSetFut: Future[Set[Long]] = if (enabledGroups.contains(Topic)) {\n      getFollowedTopics(target, interestsWithLookupContextStore, stat).map(topicIds =>\n        topicIds.toSet)\n    } else {\n      Future.value(Set.empty)\n    }\n\n    Future\n      .join(target.featureMap, target.inferredUserDeviceLanguage, followedTopicIdsSetFut).map {\n        case (\n              featureMap,\n              deviceLanguageOpt,\n              followedTopicIds\n            ) =>\n          val ageBucketOpt = if (enabledGroups.contains(AgeBucket)) {\n            featureMap.categoricalFeatures.get(targetUserAgeFeatureName)\n          } else {\n            None\n          }\n\n          val languageOptions: Set[Option[String]] = if (enabledGroups.contains(Language)) {\n            val userPreferredLanguages = featureMap.sparseBinaryFeatures\n              .getOrElse(targetUserPreferredLanguage, Set.empty[String])\n            if (userPreferredLanguages.nonEmpty) {\n              userPreferredLanguages.map(lang => Some(UserDisplayLanguage.toTweetLanguage(lang)))\n            } else {\n              Set(deviceLanguageOpt.map(UserDisplayLanguage.toTweetLanguage))\n            }\n          } else Set(None)\n\n          val followedTopicOptions: Set[Option[Long]] = if (followedTopicIds.nonEmpty) {\n            followedTopicIds.map(topic => Some(topic))\n          } else Set(None)\n\n          val tripQueries = followedTopicOptions.flatMap { topicOption =>\n            languageOptions.flatMap { languageOption =>\n              sourceIds.map { sourceId =>\n                TripDomain(\n                  sourceId = sourceId,\n                  language = languageOption,\n                  placeId = None,\n                  topicId = topicOption,\n                  gender = None,\n                  ageBucket = ageBucketOpt\n                )\n              }\n            }\n          }\n\n          tripQueries\n      }\n  }\n}\n\ncase class HighQualityTweetsAdaptor(\n  tripTweetCandidateStore: ReadableStore[TripDomain, TripTweets],\n  interestsWithLookupContextStore: ReadableStore[InterestsLookupRequestWithContext, UserInterests],\n  tweetyPieStore: ReadableStore[Long, TweetyPieResult],\n  tweetyPieStoreNoVF: ReadableStore[Long, TweetyPieResult],\n  globalStats: StatsReceiver)\n    extends CandidateSource[Target, RawCandidate]\n    with CandidateSourceEligible[Target, RawCandidate] {\n\n  override def name: String = this.getClass.getSimpleName\n\n  private val stats = globalStats.scope(\"HighQualityCandidateAdaptor\")\n  private val followedTopicsStats = stats.stat(\"followed_topics\")\n  private val missingResponseCounter = stats.counter(\"missing_respond_counter\")\n  private val crtFatigueCounter = stats.counter(\"fatigue_by_crt\")\n  private val fallbackRequestsCounter = stats.counter(\"fallback_requests\")\n\n  override def isCandidateSourceAvailable(target: Target): Future[Boolean] = {\n    PushDeviceUtil.isRecommendationsEligible(target).map {\n      _ && target.params(FS.HighQualityCandidatesEnableCandidateSource)\n    }\n  }\n\n  private val highQualityCandidateFrequencyPredicate = {\n    TargetPredicates\n      .pushRecTypeFatiguePredicate(\n        CommonRecommendationType.TripHqTweet,\n        FS.HighQualityTweetsPushInterval,\n        FS.MaxHighQualityTweetsPushGivenInterval,\n        stats\n      )\n  }\n\n  private def getTripCandidatesStrato(\n    target: Target\n  ): Future[Map[Long, Set[TripDomain]]] = {\n    val tripQueriesF: Future[Set[TripDomain]] = HighQualityTweetsHelper.getTripQueries(\n      target = target,\n      enabledGroups = target.params(FS.HighQualityCandidatesEnableGroups).toSet,\n      interestsWithLookupContextStore = interestsWithLookupContextStore,\n      sourceIds = target.params(FS.TripTweetCandidateSourceIds),\n      stat = followedTopicsStats\n    )\n\n    lazy val fallbackTripQueriesFut: Future[Set[TripDomain]] =\n      if (target.params(FS.HighQualityCandidatesEnableFallback))\n        HighQualityTweetsHelper.getTripQueries(\n          target = target,\n          enabledGroups = target.params(FS.HighQualityCandidatesFallbackEnabledGroups).toSet,\n          interestsWithLookupContextStore = interestsWithLookupContextStore,\n          sourceIds = target.params(FS.HighQualityCandidatesFallbackSourceIds),\n          stat = followedTopicsStats\n        )\n      else Future.value(Set.empty)\n\n    val initialTweetsFut: Future[Map[TripDomain, Seq[TripTweet]]] = tripQueriesF.flatMap {\n      tripQueries => getTripTweetsByDomains(tripQueries)\n    }\n\n    val tweetsByDomainFut: Future[Map[TripDomain, Seq[TripTweet]]] =\n      if (target.params(FS.HighQualityCandidatesEnableFallback)) {\n        initialTweetsFut.flatMap { candidates =>\n          val minCandidatesForFallback: Int =\n            target.params(FS.HighQualityCandidatesMinNumOfCandidatesToFallback)\n          val validCandidates = candidates.filter(_._2.size >= minCandidatesForFallback)\n\n          if (validCandidates.nonEmpty) {\n            Future.value(validCandidates)\n          } else {\n            fallbackTripQueriesFut.flatMap { fallbackTripDomains =>\n              fallbackRequestsCounter.incr(fallbackTripDomains.size)\n              getTripTweetsByDomains(fallbackTripDomains)\n            }\n          }\n        }\n      } else {\n        initialTweetsFut\n      }\n\n    val numOfCandidates: Int = target.params(FS.HighQualityCandidatesNumberOfCandidates)\n    tweetsByDomainFut.map(tweetsByDomain => reformatDomainTweetMap(tweetsByDomain, numOfCandidates))\n  }\n\n  private def getTripTweetsByDomains(\n    tripQueries: Set[TripDomain]\n  ): Future[Map[TripDomain, Seq[TripTweet]]] = {\n    Future.collect(tripTweetCandidateStore.multiGet(tripQueries)).map { response =>\n      response\n        .filter(p => p._2.exists(_.tweets.nonEmpty))\n        .mapValues(_.map(_.tweets).getOrElse(Seq.empty))\n    }\n  }\n\n  private def reformatDomainTweetMap(\n    tweetsByDomain: Map[TripDomain, Seq[TripTweet]],\n    numOfCandidates: Int\n  ): Map[Long, Set[TripDomain]] = tweetsByDomain\n    .flatMap {\n      case (tripDomain, tripTweets) =>\n        tripTweets\n          .sortBy(_.score)(Ordering[Double].reverse)\n          .take(numOfCandidates)\n          .map { tweet => (tweet.tweetId, tripDomain) }\n    }.groupBy(_._1).mapValues(_.map(_._2).toSet)\n\n  private def buildRawCandidate(\n    target: Target,\n    tweetyPieResult: TweetyPieResult,\n    tripDomain: Option[scala.collection.Set[TripDomain]]\n  ): RawCandidate = {\n    PushAdaptorUtil.generateOutOfNetworkTweetCandidates(\n      inputTarget = target,\n      id = tweetyPieResult.tweet.id,\n      mediaCRT = MediaCRT(\n        CommonRecommendationType.TripHqTweet,\n        CommonRecommendationType.TripHqTweet,\n        CommonRecommendationType.TripHqTweet\n      ),\n      result = Some(tweetyPieResult),\n      tripTweetDomain = tripDomain\n    )\n  }\n\n  private def getTweetyPieResults(\n    target: Target,\n    tweetToTripDomain: Map[Long, Set[TripDomain]]\n  ): Future[Map[Long, Option[TweetyPieResult]]] = {\n    Future.collect((if (target.params(FS.EnableVFInTweetypie)) {\n                      tweetyPieStore\n                    } else {\n                      tweetyPieStoreNoVF\n                    }).multiGet(tweetToTripDomain.keySet))\n  }\n\n  override def get(target: Target): Future[Option[Seq[RawCandidate]]] = {\n    for {\n      tweetsToTripDomainMap <- getTripCandidatesStrato(target)\n      tweetyPieResults <- getTweetyPieResults(target, tweetsToTripDomainMap)\n    } yield {\n      val candidates = tweetyPieResults.flatMap {\n        case (tweetId, tweetyPieResultOpt) =>\n          tweetyPieResultOpt.map(buildRawCandidate(target, _, tweetsToTripDomainMap.get(tweetId)))\n      }\n      if (candidates.nonEmpty) {\n        highQualityCandidateFrequencyPredicate(Seq(target))\n          .map(_.head)\n          .map { isTargetFatigueEligible =>\n            if (isTargetFatigueEligible) Some(candidates)\n            else {\n              crtFatigueCounter.incr()\n              None\n            }\n          }\n\n        Some(candidates.toSeq)\n      } else {\n        missingResponseCounter.incr()\n        None\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/ListsToRecommendCandidateAdaptor.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.CandidateSourceEligible\nimport com.twitter.frigate.common.base.ListPushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.predicate.TargetPredicates\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.geoduck.service.thriftscala.LocationResponse\nimport com.twitter.interests_discovery.thriftscala.DisplayLocation\nimport com.twitter.interests_discovery.thriftscala.NonPersonalizedRecommendedLists\nimport com.twitter.interests_discovery.thriftscala.RecommendedListsRequest\nimport com.twitter.interests_discovery.thriftscala.RecommendedListsResponse\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\ncase class ListsToRecommendCandidateAdaptor(\n  listRecommendationsStore: ReadableStore[String, NonPersonalizedRecommendedLists],\n  geoDuckV2Store: ReadableStore[Long, LocationResponse],\n  idsStore: ReadableStore[RecommendedListsRequest, RecommendedListsResponse],\n  globalStats: StatsReceiver)\n    extends CandidateSource[Target, RawCandidate]\n    with CandidateSourceEligible[Target, RawCandidate] {\n\n  override val name: String = this.getClass.getSimpleName\n\n  private[this] val stats = globalStats.scope(name)\n  private[this] val noLocationCodeCounter = stats.counter(\"no_location_code\")\n  private[this] val noCandidatesCounter = stats.counter(\"no_candidates_for_geo\")\n  private[this] val disablePopGeoListsCounter = stats.counter(\"disable_pop_geo_lists\")\n  private[this] val disableIDSListsCounter = stats.counter(\"disable_ids_lists\")\n\n  private def getListCandidate(\n    targetUser: Target,\n    _listId: Long\n  ): RawCandidate with ListPushCandidate = {\n    new RawCandidate with ListPushCandidate {\n      override val listId: Long = _listId\n\n      override val commonRecType: CommonRecommendationType = CommonRecommendationType.List\n\n      override val target: Target = targetUser\n    }\n  }\n\n  private def getListsRecommendedFromHistory(\n    target: Target\n  ): Future[Seq[Long]] = {\n    target.history.map { history =>\n      history.sortedHistory.flatMap {\n        case (_, notif) if notif.commonRecommendationType == List =>\n          notif.listNotification.map(_.listId)\n        case _ => None\n      }\n    }\n  }\n\n  private def getIDSListRecs(\n    target: Target,\n    historicalListIds: Seq[Long]\n  ): Future[Seq[Long]] = {\n    val request = RecommendedListsRequest(\n      target.targetId,\n      DisplayLocation.ListDiscoveryPage,\n      Some(historicalListIds)\n    )\n    if (target.params(PushFeatureSwitchParams.EnableIDSListRecommendations)) {\n      idsStore.get(request).map {\n        case Some(response) =>\n          response.channels.map(_.id)\n        case _ => Nil\n      }\n    } else {\n      disableIDSListsCounter.incr()\n      Future.Nil\n    }\n  }\n\n  private def getPopGeoLists(\n    target: Target,\n    historicalListIds: Seq[Long]\n  ): Future[Seq[Long]] = {\n    if (target.params(PushFeatureSwitchParams.EnablePopGeoListRecommendations)) {\n      geoDuckV2Store.get(target.targetId).flatMap {\n        case Some(locationResponse) if locationResponse.geohash.isDefined =>\n          val geoHashLength =\n            target.params(PushFeatureSwitchParams.ListRecommendationsGeoHashLength)\n          val geoHash = locationResponse.geohash.get.take(geoHashLength)\n          listRecommendationsStore\n            .get(s\"geohash_$geoHash\")\n            .map {\n              case Some(recommendedLists) =>\n                recommendedLists.recommendedListsByAlgo.flatMap { topLists =>\n                  topLists.lists.collect {\n                    case list if !historicalListIds.contains(list.listId) => list.listId\n                  }\n                }\n              case _ => Nil\n            }\n        case _ =>\n          noLocationCodeCounter.incr()\n          Future.Nil\n      }\n    } else {\n      disablePopGeoListsCounter.incr()\n      Future.Nil\n    }\n  }\n\n  override def get(target: Target): Future[Option[Seq[RawCandidate]]] = {\n    getListsRecommendedFromHistory(target).flatMap { historicalListIds =>\n      Future\n        .join(\n          getPopGeoLists(target, historicalListIds),\n          getIDSListRecs(target, historicalListIds)\n        )\n        .map {\n          case (popGeoListsIds, idsListIds) =>\n            val candidates = (idsListIds ++ popGeoListsIds).map(getListCandidate(target, _))\n            Some(candidates)\n          case _ =>\n            noCandidatesCounter.incr()\n            None\n        }\n    }\n  }\n\n  private val pushCapFatiguePredicate = TargetPredicates.pushRecTypeFatiguePredicate(\n    CommonRecommendationType.List,\n    PushFeatureSwitchParams.ListRecommendationsPushInterval,\n    PushFeatureSwitchParams.MaxListRecommendationsPushGivenInterval,\n    stats,\n  )\n  override def isCandidateSourceAvailable(target: Target): Future[Boolean] = {\n\n    val isNotFatigued = pushCapFatiguePredicate.apply(Seq(target)).map(_.head)\n\n    Future\n      .join(\n        PushDeviceUtil.isRecommendationsEligible(target),\n        isNotFatigued\n      ).map {\n        case (userRecommendationsEligible, isUnderCAP) =>\n          userRecommendationsEligible && isUnderCAP && target.params(\n            PushFeatureSwitchParams.EnableListRecommendations)\n      }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/LoggedOutPushCandidateSourceGenerator.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.CandidateSourceEligible\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.geoduck.service.thriftscala.LocationResponse\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripDomain\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripTweets\nimport com.twitter.content_mixer.thriftscala.ContentMixerRequest\nimport com.twitter.content_mixer.thriftscala.ContentMixerResponse\nimport com.twitter.geoduck.common.thriftscala.Location\nimport com.twitter.hermit.pop_geo.thriftscala.PopTweetsInPlace\nimport com.twitter.recommendation.interests.discovery.core.model.InterestDomain\n\nclass LoggedOutPushCandidateSourceGenerator(\n  tripTweetCandidateStore: ReadableStore[TripDomain, TripTweets],\n  geoDuckV2Store: ReadableStore[Long, LocationResponse],\n  safeCachedTweetyPieStoreV2: ReadableStore[Long, TweetyPieResult],\n  cachedTweetyPieStoreV2NoVF: ReadableStore[Long, TweetyPieResult],\n  cachedTweetyPieStoreV2: ReadableStore[Long, TweetyPieResult],\n  contentMixerStore: ReadableStore[ContentMixerRequest, ContentMixerResponse],\n  softUserLocationStore: ReadableStore[Long, Location],\n  topTweetsByGeoStore: ReadableStore[InterestDomain[String], Map[String, List[(Long, Double)]]],\n  topTweetsByGeoV2VersionedStore: ReadableStore[String, PopTweetsInPlace],\n)(\n  implicit val globalStats: StatsReceiver) {\n  val sources: Seq[CandidateSource[Target, RawCandidate] with CandidateSourceEligible[\n    Target,\n    RawCandidate\n  ]] = {\n    Seq(\n      TripGeoCandidatesAdaptor(\n        tripTweetCandidateStore,\n        contentMixerStore,\n        safeCachedTweetyPieStoreV2,\n        cachedTweetyPieStoreV2NoVF,\n        globalStats\n      ),\n      TopTweetsByGeoAdaptor(\n        geoDuckV2Store,\n        softUserLocationStore,\n        topTweetsByGeoStore,\n        topTweetsByGeoV2VersionedStore,\n        cachedTweetyPieStoreV2,\n        cachedTweetyPieStoreV2NoVF,\n        globalStats\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/OnboardingPushCandidateAdaptor.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.CandidateSourceEligible\nimport com.twitter.frigate.common.base.DiscoverTwitterCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FS}\nimport com.twitter.frigate.pushservice.predicate.DiscoverTwitterPredicate\nimport com.twitter.frigate.pushservice.predicate.TargetPredicates\nimport com.twitter.frigate.pushservice.util.PushAppPermissionUtil\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.frigate.thriftscala.{CommonRecommendationType => CRT}\nimport com.twitter.util.Future\n\nclass OnboardingPushCandidateAdaptor(\n  globalStats: StatsReceiver)\n    extends CandidateSource[Target, RawCandidate]\n    with CandidateSourceEligible[Target, RawCandidate] {\n\n  override val name: String = this.getClass.getSimpleName\n\n  private[this] val stats = globalStats.scope(name)\n  private[this] val requestNum = stats.counter(\"request_num\")\n  private[this] val addressBookCandNum = stats.counter(\"address_book_cand_num\")\n  private[this] val completeOnboardingCandNum = stats.counter(\"complete_onboarding_cand_num\")\n\n  private def generateOnboardingPushRawCandidate(\n    _target: Target,\n    _commonRecType: CRT\n  ): RawCandidate = {\n    new RawCandidate with DiscoverTwitterCandidate {\n      override val target = _target\n      override val commonRecType = _commonRecType\n    }\n  }\n\n  private def getEligibleCandsForTarget(\n    target: Target\n  ): Future[Option[Seq[RawCandidate]]] = {\n    val addressBookFatigue =\n      TargetPredicates\n        .pushRecTypeFatiguePredicate(\n          CRT.AddressBookUploadPush,\n          FS.FatigueForOnboardingPushes,\n          FS.MaxOnboardingPushInInterval,\n          stats)(Seq(target)).map(_.head)\n    val completeOnboardingFatigue =\n      TargetPredicates\n        .pushRecTypeFatiguePredicate(\n          CRT.CompleteOnboardingPush,\n          FS.FatigueForOnboardingPushes,\n          FS.MaxOnboardingPushInInterval,\n          stats)(Seq(target)).map(_.head)\n\n    Future\n      .join(\n        target.appPermissions,\n        addressBookFatigue,\n        completeOnboardingFatigue\n      ).map {\n        case (appPermissionOpt, addressBookPredicate, completeOnboardingPredicate) =>\n          val addressBookUploaded =\n            PushAppPermissionUtil.hasTargetUploadedAddressBook(appPermissionOpt)\n          val abUploadCandidate =\n            if (!addressBookUploaded && addressBookPredicate && target.params(\n                FS.EnableAddressBookPush)) {\n              addressBookCandNum.incr()\n              Some(generateOnboardingPushRawCandidate(target, CRT.AddressBookUploadPush))\n            } else if (!addressBookUploaded && (completeOnboardingPredicate ||\n              target.params(FS.DisableOnboardingPushFatigue)) && target.params(\n                FS.EnableCompleteOnboardingPush)) {\n              completeOnboardingCandNum.incr()\n              Some(generateOnboardingPushRawCandidate(target, CRT.CompleteOnboardingPush))\n            } else None\n\n          val allCandidates =\n            Seq(abUploadCandidate).filter(_.isDefined).flatten\n          if (allCandidates.nonEmpty) Some(allCandidates) else None\n      }\n  }\n\n  override def get(inputTarget: Target): Future[Option[Seq[RawCandidate]]] = {\n    requestNum.incr()\n    val minDurationForMRElapsed =\n      DiscoverTwitterPredicate\n        .minDurationElapsedSinceLastMrPushPredicate(\n          name,\n          FS.MrMinDurationSincePushForOnboardingPushes,\n          stats)(Seq(inputTarget)).map(_.head)\n    minDurationForMRElapsed.flatMap { minDurationElapsed =>\n      if (minDurationElapsed) getEligibleCandsForTarget(inputTarget) else Future.None\n    }\n  }\n\n  override def isCandidateSourceAvailable(target: Target): Future[Boolean] = {\n    PushDeviceUtil\n      .isRecommendationsEligible(target).map(_ && target.params(FS.EnableOnboardingPushes))\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/PushCandidateSourceGenerator.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.content_mixer.thriftscala.ContentMixerRequest\nimport com.twitter.content_mixer.thriftscala.ContentMixerResponse\nimport com.twitter.explore_ranker.thriftscala.ExploreRankerRequest\nimport com.twitter.explore_ranker.thriftscala.ExploreRankerResponse\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.candidate._\nimport com.twitter.frigate.common.store.RecentTweetsQuery\nimport com.twitter.frigate.common.store.interests.InterestsLookupRequestWithContext\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.store._\nimport com.twitter.geoduck.common.thriftscala.Location\nimport com.twitter.geoduck.service.thriftscala.LocationResponse\nimport com.twitter.hermit.pop_geo.thriftscala.PopTweetsInPlace\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.hermit.store.tweetypie.UserTweet\nimport com.twitter.interests.thriftscala.UserInterests\nimport com.twitter.interests_discovery.thriftscala.NonPersonalizedRecommendedLists\nimport com.twitter.interests_discovery.thriftscala.RecommendedListsRequest\nimport com.twitter.interests_discovery.thriftscala.RecommendedListsResponse\nimport com.twitter.recommendation.interests.discovery.core.model.InterestDomain\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripDomain\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripTweets\nimport com.twitter.tsp.thriftscala.TopicSocialProofRequest\nimport com.twitter.tsp.thriftscala.TopicSocialProofResponse\n\n/**\n * PushCandidateSourceGenerator generates candidate source list for a given Target user\n */\nclass PushCandidateSourceGenerator(\n  earlybirdCandidates: CandidateSource[EarlybirdCandidateSource.Query, EarlybirdCandidate],\n  userTweetEntityGraphCandidates: CandidateSource[UserTweetEntityGraphCandidates.Target, Candidate],\n  cachedTweetyPieStoreV2: ReadableStore[Long, TweetyPieResult],\n  safeCachedTweetyPieStoreV2: ReadableStore[Long, TweetyPieResult],\n  userTweetTweetyPieStore: ReadableStore[UserTweet, TweetyPieResult],\n  safeUserTweetTweetyPieStore: ReadableStore[UserTweet, TweetyPieResult],\n  cachedTweetyPieStoreV2NoVF: ReadableStore[Long, TweetyPieResult],\n  edgeStore: ReadableStore[RelationEdge, Boolean],\n  interestsLookupStore: ReadableStore[InterestsLookupRequestWithContext, UserInterests],\n  uttEntityHydrationStore: UttEntityHydrationStore,\n  geoDuckV2Store: ReadableStore[Long, LocationResponse],\n  topTweetsByGeoStore: ReadableStore[InterestDomain[String], Map[String, List[(Long, Double)]]],\n  topTweetsByGeoV2VersionedStore: ReadableStore[String, PopTweetsInPlace],\n  tweetImpressionsStore: TweetImpressionsStore,\n  recommendedTrendsCandidateSource: RecommendedTrendsCandidateSource,\n  recentTweetsByAuthorStore: ReadableStore[RecentTweetsQuery, Seq[Seq[Long]]],\n  topicSocialProofServiceStore: ReadableStore[TopicSocialProofRequest, TopicSocialProofResponse],\n  crMixerStore: CrMixerTweetStore,\n  contentMixerStore: ReadableStore[ContentMixerRequest, ContentMixerResponse],\n  exploreRankerStore: ReadableStore[ExploreRankerRequest, ExploreRankerResponse],\n  softUserLocationStore: ReadableStore[Long, Location],\n  tripTweetCandidateStore: ReadableStore[TripDomain, TripTweets],\n  listRecsStore: ReadableStore[String, NonPersonalizedRecommendedLists],\n  idsStore: ReadableStore[RecommendedListsRequest, RecommendedListsResponse]\n)(\n  implicit val globalStats: StatsReceiver) {\n\n  private val earlyBirdFirstDegreeCandidateAdaptor = EarlyBirdFirstDegreeCandidateAdaptor(\n    earlybirdCandidates,\n    cachedTweetyPieStoreV2,\n    cachedTweetyPieStoreV2NoVF,\n    userTweetTweetyPieStore,\n    PushFeatureSwitchParams.NumberOfMaxEarlybirdInNetworkCandidatesParam,\n    globalStats\n  )\n\n  private val frsTweetCandidateAdaptor = FRSTweetCandidateAdaptor(\n    crMixerStore,\n    cachedTweetyPieStoreV2,\n    cachedTweetyPieStoreV2NoVF,\n    userTweetTweetyPieStore,\n    uttEntityHydrationStore,\n    topicSocialProofServiceStore,\n    globalStats\n  )\n\n  private val contentRecommenderMixerAdaptor = ContentRecommenderMixerAdaptor(\n    crMixerStore,\n    safeCachedTweetyPieStoreV2,\n    edgeStore,\n    topicSocialProofServiceStore,\n    uttEntityHydrationStore,\n    globalStats\n  )\n\n  private val tripGeoCandidatesAdaptor = TripGeoCandidatesAdaptor(\n    tripTweetCandidateStore,\n    contentMixerStore,\n    safeCachedTweetyPieStoreV2,\n    cachedTweetyPieStoreV2NoVF,\n    globalStats\n  )\n\n  val sources: Seq[\n    CandidateSource[Target, RawCandidate] with CandidateSourceEligible[\n      Target,\n      RawCandidate\n    ]\n  ] = {\n    Seq(\n      earlyBirdFirstDegreeCandidateAdaptor,\n      GenericCandidateAdaptor(\n        userTweetEntityGraphCandidates,\n        cachedTweetyPieStoreV2,\n        cachedTweetyPieStoreV2NoVF,\n        globalStats.scope(\"UserTweetEntityGraphCandidates\")\n      ),\n      new OnboardingPushCandidateAdaptor(globalStats),\n      TopTweetsByGeoAdaptor(\n        geoDuckV2Store,\n        softUserLocationStore,\n        topTweetsByGeoStore,\n        topTweetsByGeoV2VersionedStore,\n        cachedTweetyPieStoreV2,\n        cachedTweetyPieStoreV2NoVF,\n        globalStats\n      ),\n      frsTweetCandidateAdaptor,\n      TopTweetImpressionsCandidateAdaptor(\n        recentTweetsByAuthorStore,\n        cachedTweetyPieStoreV2,\n        cachedTweetyPieStoreV2NoVF,\n        tweetImpressionsStore,\n        globalStats\n      ),\n      TrendsCandidatesAdaptor(\n        softUserLocationStore,\n        recommendedTrendsCandidateSource,\n        safeCachedTweetyPieStoreV2,\n        cachedTweetyPieStoreV2NoVF,\n        safeUserTweetTweetyPieStore,\n        globalStats\n      ),\n      contentRecommenderMixerAdaptor,\n      tripGeoCandidatesAdaptor,\n      HighQualityTweetsAdaptor(\n        tripTweetCandidateStore,\n        interestsLookupStore,\n        cachedTweetyPieStoreV2,\n        cachedTweetyPieStoreV2NoVF,\n        globalStats\n      ),\n      ExploreVideoTweetCandidateAdaptor(\n        exploreRankerStore,\n        cachedTweetyPieStoreV2,\n        globalStats\n      ),\n      ListsToRecommendCandidateAdaptor(\n        listRecsStore,\n        geoDuckV2Store,\n        idsStore,\n        globalStats\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/TopTweetImpressionsCandidateAdaptor.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.CandidateSourceEligible\nimport com.twitter.frigate.common.base.TopTweetImpressionsCandidate\nimport com.twitter.frigate.common.store.RecentTweetsQuery\nimport com.twitter.frigate.common.util.SnowflakeUtils\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FS}\nimport com.twitter.frigate.pushservice.store.TweetImpressionsStore\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.FutureOps\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\ncase class TweetImpressionsCandidate(\n  tweetId: Long,\n  tweetyPieResultOpt: Option[TweetyPieResult],\n  impressionsCountOpt: Option[Long])\n\ncase class TopTweetImpressionsCandidateAdaptor(\n  recentTweetsFromTflockStore: ReadableStore[RecentTweetsQuery, Seq[Seq[Long]]],\n  tweetyPieStore: ReadableStore[Long, TweetyPieResult],\n  tweetyPieStoreNoVF: ReadableStore[Long, TweetyPieResult],\n  tweetImpressionsStore: TweetImpressionsStore,\n  globalStats: StatsReceiver)\n    extends CandidateSource[Target, RawCandidate]\n    with CandidateSourceEligible[Target, RawCandidate] {\n\n  private val stats = globalStats.scope(\"TopTweetImpressionsAdaptor\")\n  private val tweetImpressionsCandsStat = stats.stat(\"top_tweet_impressions_cands_dist\")\n\n  private val eligibleUsersCounter = stats.counter(\"eligible_users\")\n  private val noneligibleUsersCounter = stats.counter(\"noneligible_users\")\n  private val meetsMinTweetsRequiredCounter = stats.counter(\"meets_min_tweets_required\")\n  private val belowMinTweetsRequiredCounter = stats.counter(\"below_min_tweets_required\")\n  private val aboveMaxInboundFavoritesCounter = stats.counter(\"above_max_inbound_favorites\")\n  private val meetsImpressionsRequiredCounter = stats.counter(\"meets_impressions_required\")\n  private val belowImpressionsRequiredCounter = stats.counter(\"below_impressions_required\")\n  private val meetsFavoritesThresholdCounter = stats.counter(\"meets_favorites_threshold\")\n  private val aboveFavoritesThresholdCounter = stats.counter(\"above_favorites_threshold\")\n  private val emptyImpressionsMapCounter = stats.counter(\"empty_impressions_map\")\n\n  private val tflockResultsStat = stats.stat(\"tflock\", \"results\")\n  private val emptyTflockResult = stats.counter(\"tflock\", \"empty_result\")\n  private val nonEmptyTflockResult = stats.counter(\"tflock\", \"non_empty_result\")\n\n  private val originalTweetsStat = stats.stat(\"tweets\", \"original_tweets\")\n  private val retweetsStat = stats.stat(\"tweets\", \"retweets\")\n  private val allRetweetsOnlyCounter = stats.counter(\"tweets\", \"all_retweets_only\")\n  private val allOriginalTweetsOnlyCounter = stats.counter(\"tweets\", \"all_original_tweets_only\")\n\n  private val emptyTweetypieMap = stats.counter(\"\", \"empty_tweetypie_map\")\n  private val emptyTweetyPieResult = stats.stat(\"\", \"empty_tweetypie_result\")\n  private val allEmptyTweetypieResults = stats.counter(\"\", \"all_empty_tweetypie_results\")\n\n  private val eligibleUsersAfterImpressionsFilter =\n    stats.counter(\"eligible_users_after_impressions_filter\")\n  private val eligibleUsersAfterFavoritesFilter =\n    stats.counter(\"eligible_users_after_favorites_filter\")\n  private val eligibleUsersWithEligibleTweets =\n    stats.counter(\"eligible_users_with_eligible_tweets\")\n\n  private val eligibleTweetCands = stats.stat(\"eligible_tweet_cands\")\n  private val getCandsRequestCounter =\n    stats.counter(\"top_tweet_impressions_get_request\")\n\n  override val name: String = this.getClass.getSimpleName\n\n  override def get(inputTarget: Target): Future[Option[Seq[RawCandidate]]] = {\n    getCandsRequestCounter.incr()\n    val eligibleCandidatesFut = getTweetImpressionsCandidates(inputTarget)\n    eligibleCandidatesFut.map { eligibleCandidates =>\n      if (eligibleCandidates.nonEmpty) {\n        eligibleUsersWithEligibleTweets.incr()\n        eligibleTweetCands.add(eligibleCandidates.size)\n        val candidate = getMostImpressionsTweet(eligibleCandidates)\n        Some(\n          Seq(\n            generateTopTweetImpressionsCandidate(\n              inputTarget,\n              candidate.tweetId,\n              candidate.tweetyPieResultOpt,\n              candidate.impressionsCountOpt.getOrElse(0L))))\n      } else None\n    }\n  }\n\n  private def getTweetImpressionsCandidates(\n    inputTarget: Target\n  ): Future[Seq[TweetImpressionsCandidate]] = {\n    val originalTweets = getRecentOriginalTweetsForUser(inputTarget)\n    originalTweets.flatMap { tweetyPieResultsMap =>\n      val numDaysSearchForOriginalTweets =\n        inputTarget.params(FS.TopTweetImpressionsOriginalTweetsNumDaysSearch)\n      val moreRecentTweetIds =\n        getMoreRecentTweetIds(tweetyPieResultsMap.keySet.toSeq, numDaysSearchForOriginalTweets)\n      val isEligible = isEligibleUser(inputTarget, tweetyPieResultsMap, moreRecentTweetIds)\n      if (isEligible) filterByEligibility(inputTarget, tweetyPieResultsMap, moreRecentTweetIds)\n      else Future.Nil\n    }\n  }\n\n  private def getRecentOriginalTweetsForUser(\n    targetUser: Target\n  ): Future[Map[Long, TweetyPieResult]] = {\n    val tweetyPieResultsMapFut = getTflockStoreResults(targetUser).flatMap { recentTweetIds =>\n      FutureOps.mapCollect((targetUser.params(FS.EnableVFInTweetypie) match {\n        case true => tweetyPieStore\n        case false => tweetyPieStoreNoVF\n      }).multiGet(recentTweetIds.toSet))\n    }\n    tweetyPieResultsMapFut.map { tweetyPieResultsMap =>\n      if (tweetyPieResultsMap.isEmpty) {\n        emptyTweetypieMap.incr()\n        Map.empty\n      } else removeRetweets(tweetyPieResultsMap)\n    }\n  }\n\n  private def getTflockStoreResults(targetUser: Target): Future[Seq[Long]] = {\n    val maxResults = targetUser.params(FS.TopTweetImpressionsRecentTweetsByAuthorStoreMaxResults)\n    val maxAge = targetUser.params(FS.TopTweetImpressionsTotalFavoritesLimitNumDaysSearch)\n    val recentTweetsQuery =\n      RecentTweetsQuery(\n        userIds = Seq(targetUser.targetId),\n        maxResults = maxResults,\n        maxAge = maxAge.days\n      )\n    recentTweetsFromTflockStore\n      .get(recentTweetsQuery).map {\n        case Some(tweetIdsAll) =>\n          val tweetIds = tweetIdsAll.headOption.getOrElse(Seq.empty)\n          val numTweets = tweetIds.size\n          if (numTweets > 0) {\n            tflockResultsStat.add(numTweets)\n            nonEmptyTflockResult.incr()\n          } else emptyTflockResult.incr()\n          tweetIds\n        case _ => Nil\n      }\n  }\n\n  private def removeRetweets(\n    tweetyPieResultsMap: Map[Long, Option[TweetyPieResult]]\n  ): Map[Long, TweetyPieResult] = {\n    val nonEmptyTweetyPieResults: Map[Long, TweetyPieResult] = tweetyPieResultsMap.collect {\n      case (key, Some(value)) => (key, value)\n    }\n    emptyTweetyPieResult.add(tweetyPieResultsMap.size - nonEmptyTweetyPieResults.size)\n\n    if (nonEmptyTweetyPieResults.nonEmpty) {\n      val originalTweets = nonEmptyTweetyPieResults.filter {\n        case (_, tweetyPieResult) =>\n          tweetyPieResult.sourceTweet.isEmpty\n      }\n      val numOriginalTweets = originalTweets.size\n      val numRetweets = nonEmptyTweetyPieResults.size - originalTweets.size\n      originalTweetsStat.add(numOriginalTweets)\n      retweetsStat.add(numRetweets)\n      if (numRetweets == 0) allOriginalTweetsOnlyCounter.incr()\n      if (numOriginalTweets == 0) allRetweetsOnlyCounter.incr()\n      originalTweets\n    } else {\n      allEmptyTweetypieResults.incr()\n      Map.empty\n    }\n  }\n\n  private def getMoreRecentTweetIds(\n    tweetIds: Seq[Long],\n    numDays: Int\n  ): Seq[Long] = {\n    tweetIds.filter { tweetId =>\n      SnowflakeUtils.isRecent(tweetId, numDays.days)\n    }\n  }\n\n  private def isEligibleUser(\n    inputTarget: Target,\n    tweetyPieResults: Map[Long, TweetyPieResult],\n    recentTweetIds: Seq[Long]\n  ): Boolean = {\n    val minNumTweets = inputTarget.params(FS.TopTweetImpressionsMinNumOriginalTweets)\n    lazy val totalFavoritesLimit =\n      inputTarget.params(FS.TopTweetImpressionsTotalInboundFavoritesLimit)\n    if (recentTweetIds.size >= minNumTweets) {\n      meetsMinTweetsRequiredCounter.incr()\n      val isUnderLimit = isUnderTotalInboundFavoritesLimit(tweetyPieResults, totalFavoritesLimit)\n      if (isUnderLimit) eligibleUsersCounter.incr()\n      else {\n        aboveMaxInboundFavoritesCounter.incr()\n        noneligibleUsersCounter.incr()\n      }\n      isUnderLimit\n    } else {\n      belowMinTweetsRequiredCounter.incr()\n      noneligibleUsersCounter.incr()\n      false\n    }\n  }\n\n  private def getFavoriteCounts(\n    tweetyPieResult: TweetyPieResult\n  ): Long = tweetyPieResult.tweet.counts.flatMap(_.favoriteCount).getOrElse(0L)\n\n  private def isUnderTotalInboundFavoritesLimit(\n    tweetyPieResults: Map[Long, TweetyPieResult],\n    totalFavoritesLimit: Long\n  ): Boolean = {\n    val favoritesIterator = tweetyPieResults.valuesIterator.map(getFavoriteCounts)\n    val totalInboundFavorites = favoritesIterator.sum\n    totalInboundFavorites <= totalFavoritesLimit\n  }\n\n  def filterByEligibility(\n    inputTarget: Target,\n    tweetyPieResults: Map[Long, TweetyPieResult],\n    tweetIds: Seq[Long]\n  ): Future[Seq[TweetImpressionsCandidate]] = {\n    lazy val minNumImpressions: Long = inputTarget.params(FS.TopTweetImpressionsMinRequired)\n    lazy val maxNumLikes: Long = inputTarget.params(FS.TopTweetImpressionsMaxFavoritesPerTweet)\n    for {\n      filteredImpressionsMap <- getFilteredImpressionsMap(tweetIds, minNumImpressions)\n      tweetIdsFilteredByFavorites <-\n        getTweetIdsFilteredByFavorites(filteredImpressionsMap.keySet, tweetyPieResults, maxNumLikes)\n    } yield {\n      if (filteredImpressionsMap.nonEmpty) eligibleUsersAfterImpressionsFilter.incr()\n      if (tweetIdsFilteredByFavorites.nonEmpty) eligibleUsersAfterFavoritesFilter.incr()\n\n      val candidates = tweetIdsFilteredByFavorites.map { tweetId =>\n        TweetImpressionsCandidate(\n          tweetId,\n          tweetyPieResults.get(tweetId),\n          filteredImpressionsMap.get(tweetId))\n      }\n      tweetImpressionsCandsStat.add(candidates.length)\n      candidates\n    }\n  }\n\n  private def getFilteredImpressionsMap(\n    tweetIds: Seq[Long],\n    minNumImpressions: Long\n  ): Future[Map[Long, Long]] = {\n    getImpressionsCounts(tweetIds).map { impressionsMap =>\n      if (impressionsMap.isEmpty) emptyImpressionsMapCounter.incr()\n      impressionsMap.filter {\n        case (_, numImpressions) =>\n          val isValid = numImpressions >= minNumImpressions\n          if (isValid) {\n            meetsImpressionsRequiredCounter.incr()\n          } else {\n            belowImpressionsRequiredCounter.incr()\n          }\n          isValid\n      }\n    }\n  }\n\n  private def getTweetIdsFilteredByFavorites(\n    filteredTweetIds: Set[Long],\n    tweetyPieResults: Map[Long, TweetyPieResult],\n    maxNumLikes: Long\n  ): Future[Seq[Long]] = {\n    val filteredByFavoritesTweetIds = filteredTweetIds.filter { tweetId =>\n      val tweetyPieResultOpt = tweetyPieResults.get(tweetId)\n      val isValid = tweetyPieResultOpt.exists { tweetyPieResult =>\n        getFavoriteCounts(tweetyPieResult) <= maxNumLikes\n      }\n      if (isValid) meetsFavoritesThresholdCounter.incr()\n      else aboveFavoritesThresholdCounter.incr()\n      isValid\n    }\n    Future(filteredByFavoritesTweetIds.toSeq)\n  }\n\n  private def getMostImpressionsTweet(\n    filteredResults: Seq[TweetImpressionsCandidate]\n  ): TweetImpressionsCandidate = {\n    val maxImpressions: Long = filteredResults.map {\n      _.impressionsCountOpt.getOrElse(0L)\n    }.max\n\n    val mostImpressionsCandidates: Seq[TweetImpressionsCandidate] =\n      filteredResults.filter(_.impressionsCountOpt.getOrElse(0L) == maxImpressions)\n\n    mostImpressionsCandidates.maxBy(_.tweetId)\n  }\n\n  private def getImpressionsCounts(\n    tweetIds: Seq[Long]\n  ): Future[Map[Long, Long]] = {\n    val impressionCountMap = tweetIds.map { tweetId =>\n      tweetId -> tweetImpressionsStore\n        .getCounts(tweetId).map(_.getOrElse(0L))\n    }.toMap\n    Future.collect(impressionCountMap)\n  }\n\n  private def generateTopTweetImpressionsCandidate(\n    inputTarget: Target,\n    _tweetId: Long,\n    result: Option[TweetyPieResult],\n    _impressionsCount: Long\n  ): RawCandidate = {\n    new RawCandidate with TopTweetImpressionsCandidate {\n      override val target: Target = inputTarget\n      override val tweetId: Long = _tweetId\n      override val tweetyPieResult: Option[TweetyPieResult] = result\n      override val impressionsCount: Long = _impressionsCount\n    }\n  }\n\n  override def isCandidateSourceAvailable(target: Target): Future[Boolean] = {\n    val enabledTopTweetImpressionsNotification =\n      target.params(FS.EnableTopTweetImpressionsNotification)\n\n    PushDeviceUtil\n      .isRecommendationsEligible(target).map(_ && enabledTopTweetImpressionsNotification)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/TopTweetsByGeoAdaptor.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.CandidateSourceEligible\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.predicate.CommonOutNetworkTweetCandidatesSourcePredicates.filterOutReplyTweet\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.model.PushTypes\nimport com.twitter.frigate.pushservice.params.PopGeoTweetVersion\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.params.TopTweetsForGeoCombination\nimport com.twitter.frigate.pushservice.params.TopTweetsForGeoRankingFunction\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FS}\nimport com.twitter.frigate.pushservice.predicate.DiscoverTwitterPredicate\nimport com.twitter.frigate.pushservice.predicate.TargetPredicates\nimport com.twitter.frigate.pushservice.util.MediaCRT\nimport com.twitter.frigate.pushservice.util.PushAdaptorUtil\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.geoduck.common.thriftscala.{Location => GeoLocation}\nimport com.twitter.geoduck.service.thriftscala.LocationResponse\nimport com.twitter.gizmoduck.thriftscala.UserType\nimport com.twitter.hermit.pop_geo.thriftscala.PopTweetsInPlace\nimport com.twitter.recommendation.interests.discovery.core.model.InterestDomain\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.FutureOps\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport scala.collection.Map\n\ncase class PlaceTweetScore(place: String, tweetId: Long, score: Double) {\n  def toTweetScore: (Long, Double) = (tweetId, score)\n}\ncase class TopTweetsByGeoAdaptor(\n  geoduckStoreV2: ReadableStore[Long, LocationResponse],\n  softUserGeoLocationStore: ReadableStore[Long, GeoLocation],\n  topTweetsByGeoStore: ReadableStore[InterestDomain[String], Map[String, List[(Long, Double)]]],\n  topTweetsByGeoStoreV2: ReadableStore[String, PopTweetsInPlace],\n  tweetyPieStore: ReadableStore[Long, TweetyPieResult],\n  tweetyPieStoreNoVF: ReadableStore[Long, TweetyPieResult],\n  globalStats: StatsReceiver)\n    extends CandidateSource[Target, RawCandidate]\n    with CandidateSourceEligible[Target, RawCandidate] {\n\n  override def name: String = this.getClass.getSimpleName\n\n  private[this] val stats = globalStats.scope(\"TopTweetsByGeoAdaptor\")\n  private[this] val noGeohashUserCounter: Counter = stats.counter(\"users_with_no_geohash_counter\")\n  private[this] val incomingRequestCounter: Counter = stats.counter(\"incoming_request_counter\")\n  private[this] val incomingLoggedOutRequestCounter: Counter =\n    stats.counter(\"incoming_logged_out_request_counter\")\n  private[this] val loggedOutRawCandidatesCounter =\n    stats.counter(\"logged_out_raw_candidates_counter\")\n  private[this] val emptyLoggedOutRawCandidatesCounter =\n    stats.counter(\"logged_out_empty_raw_candidates\")\n  private[this] val outputTopTweetsByGeoCounter: Stat =\n    stats.stat(\"output_top_tweets_by_geo_counter\")\n  private[this] val loggedOutPopByGeoV2CandidatesCounter: Counter =\n    stats.counter(\"logged_out_pop_by_geo_candidates\")\n  private[this] val dormantUsersSince14DaysCounter: Counter =\n    stats.counter(\"dormant_user_since_14_days_counter\")\n  private[this] val dormantUsersSince30DaysCounter: Counter =\n    stats.counter(\"dormant_user_since_30_days_counter\")\n  private[this] val nonDormantUsersSince14DaysCounter: Counter =\n    stats.counter(\"non_dormant_user_since_14_days_counter\")\n  private[this] val topTweetsByGeoTake100Counter: Counter =\n    stats.counter(\"top_tweets_by_geo_take_100_counter\")\n  private[this] val combinationRequestsCounter =\n    stats.scope(\"combination_method_request_counter\")\n  private[this] val popGeoTweetVersionCounter =\n    stats.scope(\"popgeo_tweet_version_counter\")\n  private[this] val nonReplyTweetsCounter = stats.counter(\"non_reply_tweets\")\n\n  val MaxGeoHashSize = 4\n\n  private def constructKeys(\n    geohash: Option[String],\n    accountCountryCode: Option[String],\n    keyLengths: Seq[Int],\n    version: PopGeoTweetVersion.Value\n  ): Set[String] = {\n    val geohashKeys = geohash match {\n      case Some(hash) => keyLengths.map { version + \"_geohash_\" + hash.take(_) }\n      case _ => Seq.empty\n    }\n\n    val accountCountryCodeKeys =\n      accountCountryCode.toSeq.map(version + \"_country_\" + _.toUpperCase)\n    (geohashKeys ++ accountCountryCodeKeys).toSet\n  }\n\n  def convertToPlaceTweetScore(\n    popTweetsInPlace: Seq[PopTweetsInPlace]\n  ): Seq[PlaceTweetScore] = {\n    popTweetsInPlace.flatMap {\n      case p =>\n        p.popTweets.map {\n          case popTweet => PlaceTweetScore(p.place, popTweet.tweetId, popTweet.score)\n        }\n    }\n  }\n\n  def sortGeoHashTweets(\n    placeTweetScores: Seq[PlaceTweetScore],\n    rankingFunction: TopTweetsForGeoRankingFunction.Value\n  ): Seq[PlaceTweetScore] = {\n    rankingFunction match {\n      case TopTweetsForGeoRankingFunction.Score =>\n        placeTweetScores.sortBy(_.score)(Ordering[Double].reverse)\n      case TopTweetsForGeoRankingFunction.GeohashLengthAndThenScore =>\n        placeTweetScores\n          .sortBy(row => (row.place.length, row.score))(Ordering[(Int, Double)].reverse)\n    }\n  }\n\n  def getResultsForLambdaStore(\n    inputTarget: Target,\n    geohash: Option[String],\n    store: ReadableStore[String, PopTweetsInPlace],\n    topk: Int,\n    version: PopGeoTweetVersion.Value\n  ): Future[Seq[(Long, Double)]] = {\n    inputTarget.accountCountryCode.flatMap { countryCode =>\n      val keys = {\n        if (inputTarget.params(FS.EnableCountryCodeBackoffTopTweetsByGeo))\n          constructKeys(geohash, countryCode, inputTarget.params(FS.GeoHashLengthList), version)\n        else\n          constructKeys(geohash, None, inputTarget.params(FS.GeoHashLengthList), version)\n      }\n      FutureOps\n        .mapCollect(store.multiGet(keys)).map {\n          case geohashTweetMap =>\n            val popTweets =\n              geohashTweetMap.values.flatten.toSeq\n            val results = sortGeoHashTweets(\n              convertToPlaceTweetScore(popTweets),\n              inputTarget.params(FS.RankingFunctionForTopTweetsByGeo))\n              .map(_.toTweetScore).take(topk)\n            results\n        }\n    }\n  }\n\n  def getPopGeoTweetsForLoggedOutUsers(\n    inputTarget: Target,\n    store: ReadableStore[String, PopTweetsInPlace]\n  ): Future[Seq[(Long, Double)]] = {\n    inputTarget.countryCode.flatMap { countryCode =>\n      val keys = constructKeys(None, countryCode, Seq(4), PopGeoTweetVersion.Prod)\n      FutureOps.mapCollect(store.multiGet(keys)).map {\n        case tweetMap =>\n          val tweets = tweetMap.values.flatten.toSeq\n          loggedOutPopByGeoV2CandidatesCounter.incr(tweets.size)\n          val popTweets = sortGeoHashTweets(\n            convertToPlaceTweetScore(tweets),\n            TopTweetsForGeoRankingFunction.Score).map(_.toTweetScore)\n          popTweets\n      }\n    }\n  }\n\n  def getRankedTweets(\n    inputTarget: Target,\n    geohash: Option[String]\n  ): Future[Seq[(Long, Double)]] = {\n    val MaxTopTweetsByGeoCandidatesToTake =\n      inputTarget.params(FS.MaxTopTweetsByGeoCandidatesToTake)\n    val scoringFn: String = inputTarget.params(FS.ScoringFuncForTopTweetsByGeo)\n    val combinationMethod = inputTarget.params(FS.TopTweetsByGeoCombinationParam)\n    val popGeoTweetVersion = inputTarget.params(FS.PopGeoTweetVersionParam)\n\n    inputTarget.isHeavyUserState.map { isHeavyUser =>\n      stats\n        .scope(combinationMethod.toString).scope(popGeoTweetVersion.toString).scope(\n          \"IsHeavyUser_\" + isHeavyUser.toString).counter().incr()\n    }\n    combinationRequestsCounter.scope(combinationMethod.toString).counter().incr()\n    popGeoTweetVersionCounter.scope(popGeoTweetVersion.toString).counter().incr()\n    lazy val geoStoreResults = if (geohash.isDefined) {\n      val hash = geohash.get.take(MaxGeoHashSize)\n      topTweetsByGeoStore\n        .get(\n          InterestDomain[String](hash)\n        )\n        .map {\n          case Some(scoringFnToTweetsMapOpt) =>\n            val tweetsWithScore = scoringFnToTweetsMapOpt\n              .getOrElse(scoringFn, List.empty)\n            val sortedResults = sortGeoHashTweets(\n              tweetsWithScore.map {\n                case (tweetId, score) => PlaceTweetScore(hash, tweetId, score)\n              },\n              TopTweetsForGeoRankingFunction.Score\n            ).map(_.toTweetScore).take(\n                MaxTopTweetsByGeoCandidatesToTake\n              )\n            sortedResults\n          case _ => Seq.empty\n        }\n    } else Future.value(Seq.empty)\n    lazy val versionPopGeoTweetResults =\n      getResultsForLambdaStore(\n        inputTarget,\n        geohash,\n        topTweetsByGeoStoreV2,\n        MaxTopTweetsByGeoCandidatesToTake,\n        popGeoTweetVersion\n      )\n    combinationMethod match {\n      case TopTweetsForGeoCombination.Default => geoStoreResults\n      case TopTweetsForGeoCombination.AccountsTweetFavAsBackfill =>\n        Future.join(geoStoreResults, versionPopGeoTweetResults).map {\n          case (geoStoreTweets, versionPopGeoTweets) =>\n            (geoStoreTweets ++ versionPopGeoTweets).take(MaxTopTweetsByGeoCandidatesToTake)\n        }\n      case TopTweetsForGeoCombination.AccountsTweetFavIntermixed =>\n        Future.join(geoStoreResults, versionPopGeoTweetResults).map {\n          case (geoStoreTweets, versionPopGeoTweets) =>\n            CandidateSource.interleaveSeqs(Seq(geoStoreTweets, versionPopGeoTweets))\n        }\n    }\n  }\n\n  override def get(inputTarget: Target): Future[Option[Seq[RawCandidate]]] = {\n    if (inputTarget.isLoggedOutUser) {\n      incomingLoggedOutRequestCounter.incr()\n      val rankedTweets = getPopGeoTweetsForLoggedOutUsers(inputTarget, topTweetsByGeoStoreV2)\n      val rawCandidates = {\n        rankedTweets.map { rt =>\n          FutureOps\n            .mapCollect(\n              tweetyPieStore\n                .multiGet(rt.map { case (tweetId, _) => tweetId }.toSet))\n            .map { tweetyPieResultMap =>\n              val results = buildTopTweetsByGeoRawCandidates(\n                inputTarget,\n                None,\n                tweetyPieResultMap\n              )\n              if (results.isEmpty) {\n                emptyLoggedOutRawCandidatesCounter.incr()\n              }\n              loggedOutRawCandidatesCounter.incr(results.size)\n              Some(results)\n            }\n        }.flatten\n      }\n      rawCandidates\n    } else {\n      incomingRequestCounter.incr()\n      getGeoHashForUsers(inputTarget).flatMap { geohash =>\n        if (geohash.isEmpty) noGeohashUserCounter.incr()\n        getRankedTweets(inputTarget, geohash).map { rt =>\n          if (rt.size == 100) {\n            topTweetsByGeoTake100Counter.incr(1)\n          }\n          FutureOps\n            .mapCollect((inputTarget.params(FS.EnableVFInTweetypie) match {\n              case true => tweetyPieStore\n              case false => tweetyPieStoreNoVF\n            }).multiGet(rt.map { case (tweetId, _) => tweetId }.toSet))\n            .map { tweetyPieResultMap =>\n              Some(\n                buildTopTweetsByGeoRawCandidates(\n                  inputTarget,\n                  None,\n                  filterOutReplyTweet(\n                    tweetyPieResultMap,\n                    nonReplyTweetsCounter\n                  )\n                )\n              )\n            }\n        }.flatten\n      }\n    }\n  }\n\n  private def getGeoHashForUsers(\n    inputTarget: Target\n  ): Future[Option[String]] = {\n\n    inputTarget.targetUser.flatMap {\n      case Some(user) =>\n        user.userType match {\n          case UserType.Soft =>\n            softUserGeoLocationStore\n              .get(inputTarget.targetId)\n              .map(_.flatMap(_.geohash.flatMap(_.stringGeohash)))\n\n          case _ =>\n            geoduckStoreV2.get(inputTarget.targetId).map(_.flatMap(_.geohash))\n        }\n\n      case None => Future.None\n    }\n  }\n\n  private def buildTopTweetsByGeoRawCandidates(\n    target: PushTypes.Target,\n    locationName: Option[String],\n    topTweets: Map[Long, Option[TweetyPieResult]]\n  ): Seq[RawCandidate with TweetCandidate] = {\n    val candidates = topTweets.map { tweetIdTweetyPieResultMap =>\n      PushAdaptorUtil.generateOutOfNetworkTweetCandidates(\n        inputTarget = target,\n        id = tweetIdTweetyPieResultMap._1,\n        mediaCRT = MediaCRT(\n          CommonRecommendationType.GeoPopTweet,\n          CommonRecommendationType.GeoPopTweet,\n          CommonRecommendationType.GeoPopTweet\n        ),\n        result = tweetIdTweetyPieResultMap._2,\n        localizedEntity = None\n      )\n    }.toSeq\n    outputTopTweetsByGeoCounter.add(candidates.length)\n    candidates\n  }\n\n  private val topTweetsByGeoFrequencyPredicate = {\n    TargetPredicates\n      .pushRecTypeFatiguePredicate(\n        CommonRecommendationType.GeoPopTweet,\n        FS.TopTweetsByGeoPushInterval,\n        FS.MaxTopTweetsByGeoPushGivenInterval,\n        stats\n      )\n  }\n\n  def getAvailabilityForDormantUser(target: Target): Future[Boolean] = {\n    lazy val isDormantUserNotFatigued = topTweetsByGeoFrequencyPredicate(Seq(target)).map(_.head)\n    lazy val enableTopTweetsByGeoForDormantUsers =\n      target.params(FS.EnableTopTweetsByGeoCandidatesForDormantUsers)\n\n    target.lastHTLVisitTimestamp.flatMap {\n      case Some(lastHTLTimestamp) =>\n        val minTimeSinceLastLogin =\n          target.params(FS.MinimumTimeSinceLastLoginForGeoPopTweetPush).ago\n        val timeSinceInactive = target.params(FS.TimeSinceLastLoginForGeoPopTweetPush).ago\n        val lastActiveTimestamp = Time.fromMilliseconds(lastHTLTimestamp)\n        if (lastActiveTimestamp > minTimeSinceLastLogin) {\n          nonDormantUsersSince14DaysCounter.incr()\n          Future.False\n        } else {\n          dormantUsersSince14DaysCounter.incr()\n          isDormantUserNotFatigued.map { isUserNotFatigued =>\n            lastActiveTimestamp < timeSinceInactive &&\n            enableTopTweetsByGeoForDormantUsers &&\n            isUserNotFatigued\n          }\n        }\n      case _ =>\n        dormantUsersSince30DaysCounter.incr()\n        isDormantUserNotFatigued.map { isUserNotFatigued =>\n          enableTopTweetsByGeoForDormantUsers && isUserNotFatigued\n        }\n    }\n  }\n\n  def getAvailabilityForPlaybookSetUp(target: Target): Future[Boolean] = {\n    lazy val enableTopTweetsByGeoForNewUsers = target.params(FS.EnableTopTweetsByGeoCandidates)\n    val isTargetEligibleForMrFatigueCheck = target.isAccountAtleastNDaysOld(\n      target.params(FS.MrMinDurationSincePushForTopTweetsByGeoPushes))\n    val isMrFatigueCheckEnabled =\n      target.params(FS.EnableMrMinDurationSinceMrPushFatigue)\n    val applyPredicateForTopTweetsByGeo =\n      if (isMrFatigueCheckEnabled) {\n        if (isTargetEligibleForMrFatigueCheck) {\n          DiscoverTwitterPredicate\n            .minDurationElapsedSinceLastMrPushPredicate(\n              name,\n              FS.MrMinDurationSincePushForTopTweetsByGeoPushes,\n              stats\n            ).andThen(\n              topTweetsByGeoFrequencyPredicate\n            )(Seq(target)).map(_.head)\n        } else {\n          Future.False\n        }\n      } else {\n        topTweetsByGeoFrequencyPredicate(Seq(target)).map(_.head)\n      }\n    applyPredicateForTopTweetsByGeo.map { predicateResult =>\n      predicateResult && enableTopTweetsByGeoForNewUsers\n    }\n  }\n\n  override def isCandidateSourceAvailable(target: Target): Future[Boolean] = {\n    if (target.isLoggedOutUser) {\n      Future.True\n    } else {\n      PushDeviceUtil\n        .isRecommendationsEligible(target).map(\n          _ && target.params(PushParams.PopGeoCandidatesDecider)).flatMap { isAvailable =>\n          if (isAvailable) {\n            Future\n              .join(getAvailabilityForDormantUser(target), getAvailabilityForPlaybookSetUp(target))\n              .map {\n                case (isAvailableForDormantUser, isAvailableForPlaybook) =>\n                  isAvailableForDormantUser || isAvailableForPlaybook\n                case _ => false\n              }\n          } else Future.False\n        }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/TrendsCandidatesAdaptor.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.events.recos.thriftscala.DisplayLocation\nimport com.twitter.events.recos.thriftscala.TrendsContext\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.CandidateSourceEligible\nimport com.twitter.frigate.common.base.TrendTweetCandidate\nimport com.twitter.frigate.common.base.TrendsCandidate\nimport com.twitter.frigate.common.candidate.RecommendedTrendsCandidateSource\nimport com.twitter.frigate.common.candidate.RecommendedTrendsCandidateSource.Query\nimport com.twitter.frigate.common.predicate.CommonOutNetworkTweetCandidatesSourcePredicates.filterOutReplyTweet\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.adaptor.TrendsCandidatesAdaptor._\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.predicate.TargetPredicates\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.geoduck.common.thriftscala.Location\nimport com.twitter.gizmoduck.thriftscala.UserType\nimport com.twitter.hermit.store.tweetypie.UserTweet\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport scala.collection.Map\n\nobject TrendsCandidatesAdaptor {\n  type TweetId = Long\n  type EventId = Long\n}\n\ncase class TrendsCandidatesAdaptor(\n  softUserGeoLocationStore: ReadableStore[Long, Location],\n  recommendedTrendsCandidateSource: RecommendedTrendsCandidateSource,\n  tweetyPieStore: ReadableStore[Long, TweetyPieResult],\n  tweetyPieStoreNoVF: ReadableStore[Long, TweetyPieResult],\n  safeUserTweetTweetyPieStore: ReadableStore[UserTweet, TweetyPieResult],\n  statsReceiver: StatsReceiver)\n    extends CandidateSource[Target, RawCandidate]\n    with CandidateSourceEligible[Target, RawCandidate] {\n  override val name = this.getClass.getSimpleName\n\n  private val trendAdaptorStats = statsReceiver.scope(\"TrendsCandidatesAdaptor\")\n  private val trendTweetCandidateNumber = trendAdaptorStats.counter(\"trend_tweet_candidate\")\n  private val nonReplyTweetsCounter = trendAdaptorStats.counter(\"non_reply_tweets\")\n\n  private def getQuery(target: Target): Future[Query] = {\n    def getUserCountryCode(target: Target): Future[Option[String]] = {\n      target.targetUser.flatMap {\n        case Some(user) if user.userType == UserType.Soft =>\n          softUserGeoLocationStore\n            .get(user.id)\n            .map(_.flatMap(_.simpleRgcResult.flatMap(_.countryCodeAlpha2)))\n\n        case _ => target.accountCountryCode\n      }\n    }\n\n    for {\n      countryCode <- getUserCountryCode(target)\n      inferredLanguage <- target.inferredUserDeviceLanguage\n    } yield {\n      Query(\n        userId = target.targetId,\n        displayLocation = DisplayLocation.MagicRecs,\n        languageCode = inferredLanguage,\n        countryCode = countryCode,\n        maxResults = target.params(PushFeatureSwitchParams.MaxRecommendedTrendsToQuery)\n      )\n    }\n  }\n\n  /**\n   * Query candidates only if sent at most [[PushFeatureSwitchParams.MaxTrendTweetNotificationsInDuration]]\n   * trend tweet notifications in [[PushFeatureSwitchParams.TrendTweetNotificationsFatigueDuration]]\n   */\n  val trendTweetFatiguePredicate = TargetPredicates.pushRecTypeFatiguePredicate(\n    CommonRecommendationType.TrendTweet,\n    PushFeatureSwitchParams.TrendTweetNotificationsFatigueDuration,\n    PushFeatureSwitchParams.MaxTrendTweetNotificationsInDuration,\n    trendAdaptorStats\n  )\n\n  private val recommendedTrendsWithTweetsCandidateSource: CandidateSource[\n    Target,\n    RawCandidate with TrendsCandidate\n  ] = recommendedTrendsCandidateSource\n    .convert[Target, TrendsCandidate](\n      getQuery,\n      recommendedTrendsCandidateSource.identityCandidateMapper\n    )\n    .batchMapValues[Target, RawCandidate with TrendsCandidate](\n      trendsCandidatesToTweetCandidates(_, _, getTweetyPieResults))\n\n  private def getTweetyPieResults(\n    tweetIds: Seq[TweetId],\n    target: Target\n  ): Future[Map[TweetId, TweetyPieResult]] = {\n    if (target.params(PushFeatureSwitchParams.EnableSafeUserTweetTweetypieStore)) {\n      Future\n        .collect(\n          safeUserTweetTweetyPieStore.multiGet(\n            tweetIds.toSet.map(UserTweet(_, Some(target.targetId))))).map {\n          _.collect {\n            case (userTweet, Some(tweetyPieResult)) => userTweet.tweetId -> tweetyPieResult\n          }\n        }\n    } else {\n      Future\n        .collect((target.params(PushFeatureSwitchParams.EnableVFInTweetypie) match {\n          case true => tweetyPieStore\n          case false => tweetyPieStoreNoVF\n        }).multiGet(tweetIds.toSet)).map { tweetyPieResultMap =>\n          filterOutReplyTweet(tweetyPieResultMap, nonReplyTweetsCounter).collect {\n            case (tweetId, Some(tweetyPieResult)) => tweetId -> tweetyPieResult\n          }\n        }\n    }\n  }\n\n  /**\n   *\n   * @param _target: [[Target]] object representing notificaion recipient user\n   * @param trendsCandidates: Sequence of [[TrendsCandidate]] returned from ERS\n   * @return: Seq of trends candidates expanded to associated tweets.\n   */\n  private def trendsCandidatesToTweetCandidates(\n    _target: Target,\n    trendsCandidates: Seq[TrendsCandidate],\n    getTweetyPieResults: (Seq[TweetId], Target) => Future[Map[TweetId, TweetyPieResult]]\n  ): Future[Seq[RawCandidate with TrendsCandidate]] = {\n\n    def generateTrendTweetCandidates(\n      trendCandidate: TrendsCandidate,\n      tweetyPieResults: Map[TweetId, TweetyPieResult]\n    ) = {\n      val tweetIds = trendCandidate.context.curatedRepresentativeTweets.getOrElse(Seq.empty) ++\n        trendCandidate.context.algoRepresentativeTweets.getOrElse(Seq.empty)\n\n      tweetIds.flatMap { tweetId =>\n        tweetyPieResults.get(tweetId).map { _tweetyPieResult =>\n          new RawCandidate with TrendTweetCandidate {\n            override val trendId: String = trendCandidate.trendId\n            override val trendName: String = trendCandidate.trendName\n            override val landingUrl: String = trendCandidate.landingUrl\n            override val timeBoundedLandingUrl: Option[String] =\n              trendCandidate.timeBoundedLandingUrl\n            override val context: TrendsContext = trendCandidate.context\n            override val tweetyPieResult: Option[TweetyPieResult] = Some(_tweetyPieResult)\n            override val tweetId: TweetId = _tweetyPieResult.tweet.id\n            override val target: Target = _target\n          }\n        }\n      }\n    }\n\n    // collect all tweet ids associated with all trends\n    val allTweetIds = trendsCandidates.flatMap { trendsCandidate =>\n      val context = trendsCandidate.context\n      context.curatedRepresentativeTweets.getOrElse(Seq.empty) ++\n        context.algoRepresentativeTweets.getOrElse(Seq.empty)\n    }\n\n    getTweetyPieResults(allTweetIds, _target)\n      .map { tweetIdToTweetyPieResult =>\n        val trendTweetCandidates = trendsCandidates.flatMap { trendCandidate =>\n          val allTrendTweetCandidates = generateTrendTweetCandidates(\n            trendCandidate,\n            tweetIdToTweetyPieResult\n          )\n\n          val (tweetCandidatesFromCuratedTrends, tweetCandidatesFromNonCuratedTrends) =\n            allTrendTweetCandidates.partition(_.isCuratedTrend)\n\n          tweetCandidatesFromCuratedTrends.filter(\n            _.target.params(PushFeatureSwitchParams.EnableCuratedTrendTweets)) ++\n            tweetCandidatesFromNonCuratedTrends.filter(\n              _.target.params(PushFeatureSwitchParams.EnableNonCuratedTrendTweets))\n        }\n\n        trendTweetCandidateNumber.incr(trendTweetCandidates.size)\n        trendTweetCandidates\n      }\n  }\n\n  /**\n   *\n   * @param target: [[Target]] user\n   * @return: true if customer is eligible to receive trend tweet notifications\n   *\n   */\n  override def isCandidateSourceAvailable(target: Target): Future[Boolean] = {\n    PushDeviceUtil\n      .isRecommendationsEligible(target)\n      .map(target.params(PushParams.TrendsCandidateDecider) && _)\n  }\n\n  override def get(target: Target): Future[Option[Seq[RawCandidate with TrendsCandidate]]] = {\n    recommendedTrendsWithTweetsCandidateSource\n      .get(target)\n      .flatMap {\n        case Some(candidates) if candidates.nonEmpty =>\n          trendTweetFatiguePredicate(Seq(target))\n            .map(_.head)\n            .map { isTargetFatigueEligible =>\n              if (isTargetFatigueEligible) Some(candidates)\n              else None\n            }\n\n        case _ => Future.None\n      }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/adaptor/TripGeoCandidatesAdaptor.scala",
    "content": "package com.twitter.frigate.pushservice.adaptor\n\nimport com.twitter.content_mixer.thriftscala.ContentMixerProductResponse\nimport com.twitter.content_mixer.thriftscala.ContentMixerRequest\nimport com.twitter.content_mixer.thriftscala.ContentMixerResponse\nimport com.twitter.content_mixer.thriftscala.NotificationsTripTweetsProductContext\nimport com.twitter.content_mixer.thriftscala.Product\nimport com.twitter.content_mixer.thriftscala.ProductContext\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.CandidateSourceEligible\nimport com.twitter.frigate.common.predicate.CommonOutNetworkTweetCandidatesSourcePredicates.filterOutReplyTweet\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.util.MediaCRT\nimport com.twitter.frigate.pushservice.util.PushAdaptorUtil\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.geoduck.util.country.CountryInfo\nimport com.twitter.product_mixer.core.thriftscala.ClientContext\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripDomain\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripTweets\nimport com.twitter.util.Future\n\ncase class TripGeoCandidatesAdaptor(\n  tripTweetCandidateStore: ReadableStore[TripDomain, TripTweets],\n  contentMixerStore: ReadableStore[ContentMixerRequest, ContentMixerResponse],\n  tweetyPieStore: ReadableStore[Long, TweetyPieResult],\n  tweetyPieStoreNoVF: ReadableStore[Long, TweetyPieResult],\n  statsReceiver: StatsReceiver)\n    extends CandidateSource[Target, RawCandidate]\n    with CandidateSourceEligible[Target, RawCandidate] {\n\n  override def name: String = this.getClass.getSimpleName\n\n  private val stats = statsReceiver.scope(name.stripSuffix(\"$\"))\n\n  private val contentMixerRequests = stats.counter(\"getTripCandidatesContentMixerRequests\")\n  private val loggedOutTripTweetIds = stats.counter(\"logged_out_trip_tweet_ids_count\")\n  private val loggedOutRawCandidates = stats.counter(\"logged_out_raw_candidates_count\")\n  private val rawCandidates = stats.counter(\"raw_candidates_count\")\n  private val loggedOutEmptyplaceId = stats.counter(\"logged_out_empty_place_id_count\")\n  private val loggedOutPlaceId = stats.counter(\"logged_out_place_id_count\")\n  private val nonReplyTweetsCounter = stats.counter(\"non_reply_tweets\")\n\n  override def isCandidateSourceAvailable(target: Target): Future[Boolean] = {\n    if (target.isLoggedOutUser) {\n      Future.True\n    } else {\n      for {\n        isRecommendationsSettingEnabled <- PushDeviceUtil.isRecommendationsEligible(target)\n        inferredLanguage <- target.inferredUserDeviceLanguage\n      } yield {\n        isRecommendationsSettingEnabled &&\n        inferredLanguage.nonEmpty &&\n        target.params(PushParams.TripGeoTweetCandidatesDecider)\n      }\n    }\n\n  }\n\n  private def buildRawCandidate(target: Target, tweetyPieResult: TweetyPieResult): RawCandidate = {\n    PushAdaptorUtil.generateOutOfNetworkTweetCandidates(\n      inputTarget = target,\n      id = tweetyPieResult.tweet.id,\n      mediaCRT = MediaCRT(\n        CommonRecommendationType.TripGeoTweet,\n        CommonRecommendationType.TripGeoTweet,\n        CommonRecommendationType.TripGeoTweet\n      ),\n      result = Some(tweetyPieResult),\n      localizedEntity = None\n    )\n  }\n\n  override def get(target: Target): Future[Option[Seq[RawCandidate]]] = {\n    if (target.isLoggedOutUser) {\n      for {\n        tripTweetIds <- getTripCandidatesForLoggedOutTarget(target)\n        tweetyPieResults <- Future.collect(tweetyPieStoreNoVF.multiGet(tripTweetIds))\n      } yield {\n        val candidates = tweetyPieResults.values.flatten.map(buildRawCandidate(target, _))\n        if (candidates.nonEmpty) {\n          loggedOutRawCandidates.incr(candidates.size)\n          Some(candidates.toSeq)\n        } else None\n      }\n    } else {\n      for {\n        tripTweetIds <- getTripCandidatesContentMixer(target)\n        tweetyPieResults <-\n          Future.collect((target.params(PushFeatureSwitchParams.EnableVFInTweetypie) match {\n            case true => tweetyPieStore\n            case false => tweetyPieStoreNoVF\n          }).multiGet(tripTweetIds))\n      } yield {\n        val nonReplyTweets = filterOutReplyTweet(tweetyPieResults, nonReplyTweetsCounter)\n        val candidates = nonReplyTweets.values.flatten.map(buildRawCandidate(target, _))\n        if (candidates.nonEmpty && target.params(\n            PushFeatureSwitchParams.TripTweetCandidateReturnEnable)) {\n          rawCandidates.incr(candidates.size)\n          Some(candidates.toSeq)\n        } else None\n      }\n    }\n  }\n\n  private def getTripCandidatesContentMixer(\n    target: Target\n  ): Future[Set[Long]] = {\n    contentMixerRequests.incr()\n    Future\n      .join(\n        target.inferredUserDeviceLanguage,\n        target.deviceInfo\n      )\n      .flatMap {\n        case (languageOpt, deviceInfoOpt) =>\n          contentMixerStore\n            .get(\n              ContentMixerRequest(\n                clientContext = ClientContext(\n                  userId = Some(target.targetId),\n                  languageCode = languageOpt,\n                  userAgent = deviceInfoOpt.flatMap(_.guessedPrimaryDeviceUserAgent.map(_.toString))\n                ),\n                product = Product.NotificationsTripTweets,\n                productContext = Some(\n                  ProductContext.NotificationsTripTweetsProductContext(\n                    NotificationsTripTweetsProductContext()\n                  )),\n                cursor = None,\n                maxResults =\n                  Some(target.params(PushFeatureSwitchParams.TripTweetMaxTotalCandidates))\n              )\n            ).map {\n              _.map { rawResponse =>\n                val tripResponse =\n                  rawResponse.contentMixerProductResponse\n                    .asInstanceOf[\n                      ContentMixerProductResponse.NotificationsTripTweetsProductResponse]\n                    .notificationsTripTweetsProductResponse\n\n                tripResponse.results.map(_.tweetResult.tweetId).toSet\n              }.getOrElse(Set.empty)\n            }\n      }\n  }\n\n  private def getTripCandidatesForLoggedOutTarget(\n    target: Target\n  ): Future[Set[Long]] = {\n    Future.join(target.targetLanguage, target.countryCode).flatMap {\n      case (Some(lang), Some(country)) =>\n        val placeId = CountryInfo.lookupByCode(country).map(_.placeIdLong)\n        if (placeId.nonEmpty) {\n          loggedOutPlaceId.incr()\n        } else {\n          loggedOutEmptyplaceId.incr()\n        }\n        val tripSource = \"TOP_GEO_V3_LR\"\n        val tripQuery = TripDomain(\n          sourceId = tripSource,\n          language = Some(lang),\n          placeId = placeId,\n          topicId = None\n        )\n        val response = tripTweetCandidateStore.get(tripQuery)\n        val tripTweetIds =\n          response.map { res =>\n            if (res.isDefined) {\n              res.get.tweets\n                .sortBy(_.score)(Ordering[Double].reverse).map(_.tweetId).toSet\n            } else {\n              Set.empty[Long]\n            }\n          }\n        tripTweetIds.map { ids => loggedOutTripTweetIds.incr(ids.size) }\n        tripTweetIds\n\n      case (_, _) => Future.value(Set.empty)\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/config/Config.scala",
    "content": "package com.twitter.frigate.pushservice.config\n\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.abuse.detection.scoring.thriftscala.TweetScoringRequest\nimport com.twitter.abuse.detection.scoring.thriftscala.TweetScoringResponse\nimport com.twitter.audience_rewards.thriftscala.HasSuperFollowingRelationshipRequest\nimport com.twitter.channels.common.thriftscala.ApiList\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala._\nimport com.twitter.decider.Decider\nimport com.twitter.discovery.common.configapi.ConfigParamsBuilder\nimport com.twitter.escherbird.common.thriftscala.QualifiedId\nimport com.twitter.escherbird.metadata.thriftscala.EntityMegadata\nimport com.twitter.eventbus.client.EventBusPublisher\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.candidate._\nimport com.twitter.frigate.common.history._\nimport com.twitter.frigate.common.ml.base._\nimport com.twitter.frigate.common.ml.feature._\nimport com.twitter.frigate.common.store._\nimport com.twitter.frigate.common.store.deviceinfo.DeviceInfo\nimport com.twitter.frigate.common.store.interests.InterestsLookupRequestWithContext\nimport com.twitter.frigate.common.store.interests.UserId\nimport com.twitter.frigate.common.util._\nimport com.twitter.frigate.data_pipeline.features_common._\nimport com.twitter.frigate.data_pipeline.thriftscala.UserHistoryKey\nimport com.twitter.frigate.data_pipeline.thriftscala.UserHistoryValue\nimport com.twitter.frigate.dau_model.thriftscala.DauProbability\nimport com.twitter.frigate.magic_events.thriftscala.FanoutEvent\nimport com.twitter.frigate.pushcap.thriftscala.PushcapUserHistory\nimport com.twitter.frigate.pushservice.ml._\nimport com.twitter.frigate.pushservice.params.DeciderKey\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitches\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.send_handler.SendHandlerPushCandidateHydrator\nimport com.twitter.frigate.pushservice.refresh_handler.PushCandidateHydrator\nimport com.twitter.frigate.pushservice.store._\nimport com.twitter.frigate.pushservice.store.{Ibis2Store => PushIbis2Store}\nimport com.twitter.frigate.pushservice.take.NotificationServiceRequest\nimport com.twitter.frigate.pushservice.thriftscala.PushRequestScribe\nimport com.twitter.frigate.scribe.thriftscala.NotificationScribe\nimport com.twitter.frigate.thriftscala._\nimport com.twitter.frigate.user_states.thriftscala.MRUserHmmState\nimport com.twitter.geoduck.common.thriftscala.{Location => GeoLocation}\nimport com.twitter.geoduck.service.thriftscala.LocationResponse\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.pop_geo.thriftscala.PopTweetsInPlace\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.hermit.predicate.tweetypie.Perspective\nimport com.twitter.hermit.predicate.tweetypie.UserTweet\nimport com.twitter.hermit.store.semantic_core.SemanticEntityForQuery\nimport com.twitter.hermit.store.tweetypie.{UserTweet => TweetyPieUserTweet}\nimport com.twitter.hermit.stp.thriftscala.STPResult\nimport com.twitter.hss.api.thriftscala.UserHealthSignalResponse\nimport com.twitter.interests.thriftscala.InterestId\nimport com.twitter.interests.thriftscala.{UserInterests => Interests}\nimport com.twitter.interests_discovery.thriftscala.NonPersonalizedRecommendedLists\nimport com.twitter.interests_discovery.thriftscala.RecommendedListsRequest\nimport com.twitter.interests_discovery.thriftscala.RecommendedListsResponse\nimport com.twitter.livevideo.timeline.domain.v2.{Event => LiveEvent}\nimport com.twitter.ml.api.thriftscala.{DataRecord => ThriftDataRecord}\nimport com.twitter.ml.featurestore.lib.dynamic.DynamicFeatureStoreClient\nimport com.twitter.notificationservice.genericfeedbackstore.FeedbackPromptValue\nimport com.twitter.notificationservice.genericfeedbackstore.GenericFeedbackStore\nimport com.twitter.notificationservice.scribe.manhattan.GenericNotificationsFeedbackRequest\nimport com.twitter.notificationservice.thriftscala.CaretFeedbackDetails\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationResponse\nimport com.twitter.nrel.heavyranker.CandidateFeatureHydrator\nimport com.twitter.nrel.heavyranker.{FeatureHydrator => MRFeatureHydrator}\nimport com.twitter.nrel.heavyranker.{TargetFeatureHydrator => RelevanceTargetFeatureHydrator}\nimport com.twitter.onboarding.task.service.thriftscala.FatigueFlowEnrollment\nimport com.twitter.permissions_storage.thriftscala.AppPermission\nimport com.twitter.recommendation.interests.discovery.core.model.InterestDomain\nimport com.twitter.recos.user_tweet_entity_graph.thriftscala.RecommendTweetEntityRequest\nimport com.twitter.recos.user_tweet_entity_graph.thriftscala.RecommendTweetEntityResponse\nimport com.twitter.recos.user_user_graph.thriftscala.RecommendUserRequest\nimport com.twitter.recos.user_user_graph.thriftscala.RecommendUserResponse\nimport com.twitter.rux.common.strato.thriftscala.UserTargetingProperty\nimport com.twitter.scio.nsfw_user_segmentation.thriftscala.NSFWProducer\nimport com.twitter.scio.nsfw_user_segmentation.thriftscala.NSFWUserSegmentation\nimport com.twitter.search.common.features.thriftscala.ThriftSearchResultFeatures\nimport com.twitter.search.earlybird.thriftscala.EarlybirdRequest\nimport com.twitter.search.earlybird.thriftscala.ThriftSearchResult\nimport com.twitter.service.gen.scarecrow.thriftscala.Event\nimport com.twitter.service.gen.scarecrow.thriftscala.TieredActionResult\nimport com.twitter.service.metastore.gen.thriftscala.Location\nimport com.twitter.service.metastore.gen.thriftscala.UserLanguages\nimport com.twitter.servo.decider.DeciderGateBuilder\nimport com.twitter.simclusters_v2.thriftscala.SimClustersInferredEntities\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.columns.frigate.logged_out_web_notifications.thriftscala.LOWebNotificationMetadata\nimport com.twitter.strato.columns.notifications.thriftscala.SourceDestUserRequest\nimport com.twitter.strato.client.{UserId => StratoUserId}\nimport com.twitter.timelines.configapi\nimport com.twitter.timelines.configapi.CompositeConfig\nimport com.twitter.timelinescorer.thriftscala.v1.ScoredTweet\nimport com.twitter.topiclisting.TopicListing\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripDomain\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripTweets\nimport com.twitter.tsp.thriftscala.TopicSocialProofRequest\nimport com.twitter.tsp.thriftscala.TopicSocialProofResponse\nimport com.twitter.ubs.thriftscala.SellerTrack\nimport com.twitter.ubs.thriftscala.AudioSpace\nimport com.twitter.ubs.thriftscala.Participants\nimport com.twitter.ubs.thriftscala.SellerApplicationState\nimport com.twitter.user_session_store.thriftscala.UserSession\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.wtf.scalding.common.thriftscala.UserFeatures\n\ntrait Config {\n  self =>\n\n  def isServiceLocal: Boolean\n\n  def localConfigRepoPath: String\n\n  def inMemCacheOff: Boolean\n\n  def historyStore: PushServiceHistoryStore\n\n  def emailHistoryStore: PushServiceHistoryStore\n\n  def strongTiesStore: ReadableStore[Long, STPResult]\n\n  def safeUserStore: ReadableStore[Long, User]\n\n  def deviceInfoStore: ReadableStore[Long, DeviceInfo]\n\n  def edgeStore: ReadableStore[RelationEdge, Boolean]\n\n  def socialGraphServiceProcessStore: ReadableStore[RelationEdge, Boolean]\n\n  def userUtcOffsetStore: ReadableStore[Long, Duration]\n\n  def cachedTweetyPieStoreV2: ReadableStore[Long, TweetyPieResult]\n\n  def safeCachedTweetyPieStoreV2: ReadableStore[Long, TweetyPieResult]\n\n  def userTweetTweetyPieStore: ReadableStore[TweetyPieUserTweet, TweetyPieResult]\n\n  def safeUserTweetTweetyPieStore: ReadableStore[TweetyPieUserTweet, TweetyPieResult]\n\n  def cachedTweetyPieStoreV2NoVF: ReadableStore[Long, TweetyPieResult]\n\n  def tweetContentFeatureCacheStore: ReadableStore[Long, ThriftDataRecord]\n\n  def scarecrowCheckEventStore: ReadableStore[Event, TieredActionResult]\n\n  def userTweetPerspectiveStore: ReadableStore[UserTweet, Perspective]\n\n  def userCountryStore: ReadableStore[Long, Location]\n\n  def pushInfoStore: ReadableStore[Long, UserForPushTargeting]\n\n  def loggedOutPushInfoStore: ReadableStore[Long, LOWebNotificationMetadata]\n\n  def tweetImpressionStore: ReadableStore[Long, Seq[Long]]\n\n  def audioSpaceStore: ReadableStore[String, AudioSpace]\n\n  def basketballGameScoreStore: ReadableStore[QualifiedId, BasketballGameLiveUpdate]\n\n  def baseballGameScoreStore: ReadableStore[QualifiedId, BaseballGameLiveUpdate]\n\n  def cricketMatchScoreStore: ReadableStore[QualifiedId, CricketMatchLiveUpdate]\n\n  def soccerMatchScoreStore: ReadableStore[QualifiedId, SoccerMatchLiveUpdate]\n\n  def nflGameScoreStore: ReadableStore[QualifiedId, NflFootballGameLiveUpdate]\n\n  def topicSocialProofServiceStore: ReadableStore[TopicSocialProofRequest, TopicSocialProofResponse]\n\n  def spaceDeviceFollowStore: ReadableStore[SourceDestUserRequest, Boolean]\n\n  def audioSpaceParticipantsStore: ReadableStore[String, Participants]\n\n  def notificationServiceSender: ReadableStore[\n    NotificationServiceRequest,\n    CreateGenericNotificationResponse\n  ]\n\n  def ocfFatigueStore: ReadableStore[OCFHistoryStoreKey, FatigueFlowEnrollment]\n\n  def dauProbabilityStore: ReadableStore[Long, DauProbability]\n\n  def hydratedLabeledPushRecsStore: ReadableStore[UserHistoryKey, UserHistoryValue]\n\n  def userHTLLastVisitStore: ReadableStore[Long, Seq[Long]]\n\n  def userLanguagesStore: ReadableStore[Long, UserLanguages]\n\n  def topTweetsByGeoStore: ReadableStore[InterestDomain[String], Map[String, List[\n    (Long, Double)\n  ]]]\n\n  def topTweetsByGeoV2VersionedStore: ReadableStore[String, PopTweetsInPlace]\n\n  lazy val pushRecItemStore: ReadableStore[PushRecItemsKey, RecItems] = PushRecItemStore(\n    hydratedLabeledPushRecsStore\n  )\n\n  lazy val labeledPushRecsVerifyingStore: ReadableStore[\n    LabeledPushRecsVerifyingStoreKey,\n    LabeledPushRecsVerifyingStoreResponse\n  ] =\n    LabeledPushRecsVerifyingStore(\n      hydratedLabeledPushRecsStore,\n      historyStore\n    )\n\n  lazy val labeledPushRecsDecideredStore: ReadableStore[LabeledPushRecsStoreKey, UserHistoryValue] =\n    LabeledPushRecsDecideredStore(\n      labeledPushRecsVerifyingStore,\n      useHydratedLabeledSendsForFeaturesDeciderKey,\n      verifyHydratedLabeledSendsForFeaturesDeciderKey\n    )\n\n  def onlineUserHistoryStore: ReadableStore[OnlineUserHistoryKey, UserHistoryValue]\n\n  def nsfwConsumerStore: ReadableStore[Long, NSFWUserSegmentation]\n\n  def nsfwProducerStore: ReadableStore[Long, NSFWProducer]\n\n  def popGeoLists: ReadableStore[String, NonPersonalizedRecommendedLists]\n\n  def listAPIStore: ReadableStore[Long, ApiList]\n\n  def openedPushByHourAggregatedStore: ReadableStore[Long, Map[Int, Int]]\n\n  def userHealthSignalStore: ReadableStore[Long, UserHealthSignalResponse]\n\n  def reactivatedUserInfoStore: ReadableStore[Long, String]\n\n  def weightedOpenOrNtabClickModelScorer: PushMLModelScorer\n\n  def optoutModelScorer: PushMLModelScorer\n\n  def filteringModelScorer: PushMLModelScorer\n\n  def recentFollowsStore: ReadableStore[Long, Seq[Long]]\n\n  def geoDuckV2Store: ReadableStore[UserId, LocationResponse]\n\n  def realGraphScoresTop500InStore: ReadableStore[Long, Map[Long, Double]]\n\n  def tweetEntityGraphStore: ReadableStore[\n    RecommendTweetEntityRequest,\n    RecommendTweetEntityResponse\n  ]\n\n  def userUserGraphStore: ReadableStore[RecommendUserRequest, RecommendUserResponse]\n\n  def userFeaturesStore: ReadableStore[Long, UserFeatures]\n\n  def userTargetingPropertyStore: ReadableStore[Long, UserTargetingProperty]\n\n  def timelinesUserSessionStore: ReadableStore[Long, UserSession]\n\n  def optOutUserInterestsStore: ReadableStore[UserId, Seq[InterestId]]\n\n  def ntabCaretFeedbackStore: ReadableStore[GenericNotificationsFeedbackRequest, Seq[\n    CaretFeedbackDetails\n  ]]\n\n  def genericFeedbackStore: ReadableStore[FeedbackRequest, Seq[\n    FeedbackPromptValue\n  ]]\n\n  def genericNotificationFeedbackStore: GenericFeedbackStore\n\n  def semanticCoreMegadataStore: ReadableStore[\n    SemanticEntityForQuery,\n    EntityMegadata\n  ]\n\n  def tweetHealthScoreStore: ReadableStore[TweetScoringRequest, TweetScoringResponse]\n\n  def earlybirdFeatureStore: ReadableStore[Long, ThriftSearchResultFeatures]\n\n  def earlybirdFeatureBuilder: FeatureBuilder[Long]\n\n  // Feature builders\n\n  def tweetAuthorLocationFeatureBuilder: FeatureBuilder[Location]\n\n  def tweetAuthorLocationFeatureBuilderById: FeatureBuilder[Long]\n\n  def socialContextActionsFeatureBuilder: FeatureBuilder[SocialContextActions]\n\n  def tweetContentFeatureBuilder: FeatureBuilder[Long]\n\n  def tweetAuthorRecentRealGraphFeatureBuilder: FeatureBuilder[RealGraphEdge]\n\n  def socialContextRecentRealGraphFeatureBuilder: FeatureBuilder[Set[RealGraphEdge]]\n\n  def tweetSocialProofFeatureBuilder: FeatureBuilder[TweetSocialProofKey]\n\n  def targetUserFullRealGraphFeatureBuilder: FeatureBuilder[TargetFullRealGraphFeatureKey]\n\n  def postProcessingFeatureBuilder: PostProcessingFeatureBuilder\n\n  def mrOfflineUserCandidateSparseAggregatesFeatureBuilder: FeatureBuilder[\n    OfflineSparseAggregateKey\n  ]\n\n  def mrOfflineUserAggregatesFeatureBuilder: FeatureBuilder[Long]\n\n  def mrOfflineUserCandidateAggregatesFeatureBuilder: FeatureBuilder[OfflineAggregateKey]\n\n  def tweetAnnotationsFeatureBuilder: FeatureBuilder[Long]\n\n  def targetUserMediaRepresentationFeatureBuilder: FeatureBuilder[Long]\n\n  def targetLevelFeatureBuilder: FeatureBuilder[MrRequestContextForFeatureStore]\n\n  def candidateLevelFeatureBuilder: FeatureBuilder[EntityRequestContextForFeatureStore]\n\n  def targetFeatureHydrator: RelevanceTargetFeatureHydrator\n\n  def useHydratedLabeledSendsForFeaturesDeciderKey: String =\n    DeciderKey.useHydratedLabeledSendsForFeaturesDeciderKey.toString\n\n  def verifyHydratedLabeledSendsForFeaturesDeciderKey: String =\n    DeciderKey.verifyHydratedLabeledSendsForFeaturesDeciderKey.toString\n\n  def lexServiceStore: ReadableStore[EventRequest, LiveEvent]\n\n  def userMediaRepresentationStore: ReadableStore[Long, UserMediaRepresentation]\n\n  def producerMediaRepresentationStore: ReadableStore[Long, UserMediaRepresentation]\n\n  def mrUserStatePredictionStore: ReadableStore[Long, MRUserHmmState]\n\n  def pushcapDynamicPredictionStore: ReadableStore[Long, PushcapUserHistory]\n\n  def earlybirdCandidateSource: EarlybirdCandidateSource\n\n  def earlybirdSearchStore: ReadableStore[EarlybirdRequest, Seq[ThriftSearchResult]]\n\n  def earlybirdSearchDest: String\n\n  def pushserviceThriftClientId: ClientId\n\n  def simClusterToEntityStore: ReadableStore[Int, SimClustersInferredEntities]\n\n  def fanoutMetadataStore: ReadableStore[(Long, Long), FanoutEvent]\n\n  /**\n   * PostRanking Feature Store Client\n   */\n  def postRankingFeatureStoreClient: DynamicFeatureStoreClient[MrRequestContextForFeatureStore]\n\n  /**\n   * ReadableStore to fetch [[UserInterests]] from INTS service\n   */\n  def interestsWithLookupContextStore: ReadableStore[InterestsLookupRequestWithContext, Interests]\n\n  /**\n   *\n   * @return: [[TopicListing]] object to fetch paused topics and scope from productId\n   */\n  def topicListing: TopicListing\n\n  /**\n   *\n   * @return: [[UttEntityHydrationStore]] object\n   */\n  def uttEntityHydrationStore: UttEntityHydrationStore\n\n  def appPermissionStore: ReadableStore[(Long, (String, String)), AppPermission]\n\n  lazy val userTweetEntityGraphCandidates: UserTweetEntityGraphCandidates =\n    UserTweetEntityGraphCandidates(\n      cachedTweetyPieStoreV2,\n      tweetEntityGraphStore,\n      PushParams.UTEGTweetCandidateSourceParam,\n      PushFeatureSwitchParams.NumberOfMaxUTEGCandidatesQueriedParam,\n      PushParams.AllowOneSocialProofForTweetInUTEGParam,\n      PushParams.OutNetworkTweetsOnlyForUTEGParam,\n      PushFeatureSwitchParams.MaxTweetAgeParam\n    )(statsReceiver)\n\n  def pushSendEventBusPublisher: EventBusPublisher[NotificationScribe]\n\n  // miscs.\n\n  def isProd: Boolean\n\n  implicit def statsReceiver: StatsReceiver\n\n  def decider: Decider\n\n  def abDecider: LoggingABDecider\n\n  def casLock: CasLock\n\n  def pushIbisV2Store: PushIbis2Store\n\n  // scribe\n  def notificationScribe(data: NotificationScribe): Unit\n\n  def requestScribe(data: PushRequestScribe): Unit\n\n  def init(): Future[Unit] = Future.Done\n\n  def configParamsBuilder: ConfigParamsBuilder\n\n  def candidateFeatureHydrator: CandidateFeatureHydrator\n\n  def featureHydrator: MRFeatureHydrator\n\n  def candidateHydrator: PushCandidateHydrator\n\n  def sendHandlerCandidateHydrator: SendHandlerPushCandidateHydrator\n\n  lazy val overridesConfig: configapi.Config = {\n    val pushFeatureSwitchConfigs: configapi.Config = PushFeatureSwitches(\n      deciderGateBuilder = new DeciderGateBuilder(decider),\n      statsReceiver = statsReceiver\n    ).config\n\n    new CompositeConfig(Seq(pushFeatureSwitchConfigs))\n  }\n\n  def realTimeClientEventStore: RealTimeClientEventStore\n\n  def inlineActionHistoryStore: ReadableStore[Long, Seq[(Long, String)]]\n\n  def softUserGeoLocationStore: ReadableStore[Long, GeoLocation]\n\n  def tweetTranslationStore: ReadableStore[TweetTranslationStore.Key, TweetTranslationStore.Value]\n\n  def tripTweetCandidateStore: ReadableStore[TripDomain, TripTweets]\n\n  def softUserFollowingStore: ReadableStore[User, Seq[Long]]\n\n  def superFollowEligibilityUserStore: ReadableStore[Long, Boolean]\n\n  def superFollowCreatorTweetCountStore: ReadableStore[StratoUserId, Int]\n\n  def hasSuperFollowingRelationshipStore: ReadableStore[\n    HasSuperFollowingRelationshipRequest,\n    Boolean\n  ]\n\n  def superFollowApplicationStatusStore: ReadableStore[(Long, SellerTrack), SellerApplicationState]\n\n  def recentHistoryCacheClient: RecentHistoryCacheClient\n\n  def openAppUserStore: ReadableStore[Long, Boolean]\n\n  def loggedOutHistoryStore: PushServiceHistoryStore\n\n  def idsStore: ReadableStore[RecommendedListsRequest, RecommendedListsResponse]\n\n  def htlScoreStore(userId: Long): ReadableStore[Long, ScoredTweet]\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/config/DeployConfig.scala",
    "content": "package com.twitter.frigate.pushservice.config\n\nimport com.twitter.abuse.detection.scoring.thriftscala.TweetScoringRequest\nimport com.twitter.abuse.detection.scoring.thriftscala.TweetScoringResponse\nimport com.twitter.audience_rewards.thriftscala.HasSuperFollowingRelationshipRequest\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.bijection.scrooge.CompactScalaCodec\nimport com.twitter.channels.common.thriftscala.ApiList\nimport com.twitter.channels.common.thriftscala.ApiListDisplayLocation\nimport com.twitter.channels.common.thriftscala.ApiListView\nimport com.twitter.content_mixer.thriftscala.ContentMixer\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.cortex.deepbird.thriftjava.DeepbirdPredictionService\nimport com.twitter.cr_mixer.thriftscala.CrMixer\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.BaseballGameLiveUpdate\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.BasketballGameLiveUpdate\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.CricketMatchLiveUpdate\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.NflFootballGameLiveUpdate\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.SoccerMatchLiveUpdate\nimport com.twitter.discovery.common.configapi.ConfigParamsBuilder\nimport com.twitter.discovery.common.configapi.FeatureContextBuilder\nimport com.twitter.discovery.common.environment.{Environment => NotifEnvironment}\nimport com.twitter.escherbird.common.thriftscala.Domains\nimport com.twitter.escherbird.common.thriftscala.QualifiedId\nimport com.twitter.escherbird.metadata.thriftscala.EntityMegadata\nimport com.twitter.escherbird.metadata.thriftscala.MetadataService\nimport com.twitter.escherbird.util.metadatastitch.MetadataStitchClient\nimport com.twitter.escherbird.util.uttclient\nimport com.twitter.escherbird.util.uttclient.CacheConfigV2\nimport com.twitter.escherbird.util.uttclient.CachedUttClientV2\nimport com.twitter.escherbird.utt.strato.thriftscala.Environment\nimport com.twitter.eventbus.client.EventBusPublisherBuilder\nimport com.twitter.events.recos.thriftscala.EventsRecosService\nimport com.twitter.explore_ranker.thriftscala.ExploreRanker\nimport com.twitter.featureswitches.v2.FeatureSwitches\nimport com.twitter.finagle.Memcached\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.client.BackupRequestFilter\nimport com.twitter.finagle.client.ClientRegistry\nimport com.twitter.finagle.loadbalancer.Balancers\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient._\nimport com.twitter.finagle.mux.transport.OpportunisticTls\nimport com.twitter.finagle.service.Retries\nimport com.twitter.finagle.service.RetryPolicy\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.thrift.RichClientParam\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.flockdb.client._\nimport com.twitter.flockdb.client.thriftscala.FlockDB\nimport com.twitter.frigate.common.base.RandomRanker\nimport com.twitter.frigate.common.candidate._\nimport com.twitter.frigate.common.config.RateLimiterGenerator\nimport com.twitter.frigate.common.entity_graph_client.RecommendedTweetEntitiesStore\nimport com.twitter.frigate.common.filter.DynamicRequestMeterFilter\nimport com.twitter.frigate.common.history._\nimport com.twitter.frigate.common.ml.feature._\nimport com.twitter.frigate.common.store._\nimport com.twitter.frigate.common.store.deviceinfo.DeviceInfoStore\nimport com.twitter.frigate.common.store.deviceinfo.MobileSdkStore\nimport com.twitter.frigate.common.store.interests._\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.frigate.common.store.strato.StratoScannableStore\nimport com.twitter.frigate.common.util.Finagle.readOnlyThriftService\nimport com.twitter.frigate.common.util._\nimport com.twitter.frigate.data_pipeline.features_common.FeatureStoreUtil\nimport com.twitter.frigate.data_pipeline.features_common._\nimport com.twitter.frigate.data_pipeline.thriftscala.UserHistoryKey\nimport com.twitter.frigate.data_pipeline.thriftscala.UserHistoryValue\nimport com.twitter.frigate.dau_model.thriftscala.DauProbability\nimport com.twitter.frigate.magic_events.thriftscala.FanoutEvent\nimport com.twitter.frigate.pushcap.thriftscala.PushcapUserHistory\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.adaptor.LoggedOutPushCandidateSourceGenerator\nimport com.twitter.frigate.pushservice.adaptor.PushCandidateSourceGenerator\nimport com.twitter.frigate.pushservice.config.mlconfig.DeepbirdV2ModelConfig\nimport com.twitter.frigate.pushservice.ml._\nimport com.twitter.frigate.pushservice.params._\nimport com.twitter.frigate.pushservice.rank.LoggedOutRanker\nimport com.twitter.frigate.pushservice.rank.RFPHLightRanker\nimport com.twitter.frigate.pushservice.rank.RFPHRanker\nimport com.twitter.frigate.pushservice.rank.SubscriptionCreatorRanker\nimport com.twitter.frigate.pushservice.refresh_handler._\nimport com.twitter.frigate.pushservice.refresh_handler.cross.CandidateCopyExpansion\nimport com.twitter.frigate.pushservice.send_handler.SendHandlerPushCandidateHydrator\nimport com.twitter.frigate.pushservice.store._\nimport com.twitter.frigate.pushservice.take.CandidateNotifier\nimport com.twitter.frigate.pushservice.take.NotificationSender\nimport com.twitter.frigate.pushservice.take.NotificationServiceRequest\nimport com.twitter.frigate.pushservice.take.NotificationServiceSender\nimport com.twitter.frigate.pushservice.take.NtabOnlyChannelSelector\nimport com.twitter.frigate.pushservice.take.history.EventBusWriter\nimport com.twitter.frigate.pushservice.take.history.HistoryWriter\nimport com.twitter.frigate.pushservice.take.sender.Ibis2Sender\nimport com.twitter.frigate.pushservice.take.sender.NtabSender\nimport com.twitter.frigate.pushservice.take.LoggedOutRefreshForPushNotifier\nimport com.twitter.frigate.pushservice.util.RFPHTakeStepUtil\nimport com.twitter.frigate.pushservice.util.SendHandlerPredicateUtil\nimport com.twitter.frigate.scribe.thriftscala.NotificationScribe\nimport com.twitter.frigate.thriftscala._\nimport com.twitter.frigate.user_states.thriftscala.MRUserHmmState\nimport com.twitter.geoduck.backend.hydration.thriftscala.Hydration\nimport com.twitter.geoduck.common.thriftscala.PlaceQueryFields\nimport com.twitter.geoduck.common.thriftscala.PlaceType\nimport com.twitter.geoduck.common.thriftscala.{Location => GeoLocation}\nimport com.twitter.geoduck.service.common.clientmodules.GeoduckUserLocate\nimport com.twitter.geoduck.service.common.clientmodules.GeoduckUserLocateModule\nimport com.twitter.geoduck.service.thriftscala.LocationResponse\nimport com.twitter.geoduck.thriftscala.LocationService\nimport com.twitter.gizmoduck.context.thriftscala.ReadConfig\nimport com.twitter.gizmoduck.context.thriftscala.TestUserConfig\nimport com.twitter.gizmoduck.testusers.client.TestUserClientBuilder\nimport com.twitter.gizmoduck.thriftscala.LookupContext\nimport com.twitter.gizmoduck.thriftscala.QueryFields\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.gizmoduck.thriftscala.UserService\nimport com.twitter.hermit.pop_geo.thriftscala.PopTweetsInPlace\nimport com.twitter.hermit.predicate.socialgraph.SocialGraphPredicate\nimport com.twitter.hermit.predicate.tweetypie.PerspectiveReadableStore\nimport com.twitter.hermit.store._\nimport com.twitter.hermit.store.common._\nimport com.twitter.hermit.store.gizmoduck.GizmoduckUserStore\nimport com.twitter.hermit.store.metastore.UserCountryStore\nimport com.twitter.hermit.store.metastore.UserLanguagesStore\nimport com.twitter.hermit.store.scarecrow.ScarecrowCheckEventStore\nimport com.twitter.hermit.store.semantic_core.MetaDataReadableStore\nimport com.twitter.hermit.store.semantic_core.SemanticEntityForQuery\nimport com.twitter.hermit.store.timezone.GizmoduckUserUtcOffsetStore\nimport com.twitter.hermit.store.timezone.UtcOffsetStore\nimport com.twitter.hermit.store.tweetypie.TweetyPieStore\nimport com.twitter.hermit.store.tweetypie.UserTweet\nimport com.twitter.hermit.store.user_htl_session_store.UserHTLLastVisitReadableStore\nimport com.twitter.hermit.stp.thriftscala.STPResult\nimport com.twitter.hss.api.thriftscala.UserHealthSignal\nimport com.twitter.hss.api.thriftscala.UserHealthSignal._\nimport com.twitter.hss.api.thriftscala.UserHealthSignalResponse\nimport com.twitter.interests.thriftscala.InterestId\nimport com.twitter.interests.thriftscala.InterestsThriftService\nimport com.twitter.interests.thriftscala.{UserInterests => Interests}\nimport com.twitter.interests_discovery.thriftscala.InterestsDiscoveryService\nimport com.twitter.interests_discovery.thriftscala.NonPersonalizedRecommendedLists\nimport com.twitter.interests_discovery.thriftscala.RecommendedListsRequest\nimport com.twitter.interests_discovery.thriftscala.RecommendedListsResponse\nimport com.twitter.kujaku.domain.thriftscala.MachineTranslationResponse\nimport com.twitter.livevideo.timeline.client.v2.LiveVideoTimelineClient\nimport com.twitter.livevideo.timeline.domain.v2.{Event => LiveEvent}\nimport com.twitter.livevideo.timeline.thrift.thriftscala.TimelineService\nimport com.twitter.logging.Logger\nimport com.twitter.ml.api.thriftscala.{DataRecord => ThriftDataRecord}\nimport com.twitter.ml.featurestore.catalog.entities.core.{Author => TweetAuthorEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{User => TargetUserEntity}\nimport com.twitter.ml.featurestore.catalog.entities.core.{UserAuthor => UserAuthorEntity}\nimport com.twitter.ml.featurestore.catalog.entities.magicrecs.{SocialContext => SocialContextEntity}\nimport com.twitter.ml.featurestore.catalog.entities.magicrecs.{UserSocialContext => TargetUserSocialContextEntity}\nimport com.twitter.ml.featurestore.timelines.thriftscala.TimelineScorerScoreView\nimport com.twitter.notificationservice.api.thriftscala.DeleteCurrentTimelineForUserRequest\nimport com.twitter.notificationservice.genericfeedbackstore.FeedbackPromptValue\nimport com.twitter.notificationservice.genericfeedbackstore.GenericFeedbackStore\nimport com.twitter.notificationservice.genericfeedbackstore.GenericFeedbackStoreBuilder\nimport com.twitter.notificationservice.scribe.manhattan.FeedbackSignalManhattanClient\nimport com.twitter.notificationservice.scribe.manhattan.GenericNotificationsFeedbackRequest\nimport com.twitter.notificationservice.thriftscala.CaretFeedbackDetails\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationRequest\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationResponse\nimport com.twitter.notificationservice.thriftscala.DeleteGenericNotificationRequest\nimport com.twitter.notificationservice.thriftscala.GenericNotificationOverrideKey\nimport com.twitter.notificationservice.thriftscala.NotificationService$FinagleClient\nimport com.twitter.nrel.heavyranker.CandidateFeatureHydrator\nimport com.twitter.nrel.heavyranker.FeatureHydrator\nimport com.twitter.nrel.heavyranker.{PushPredictionServiceStore => RelevancePushPredictionServiceStore}\nimport com.twitter.nrel.heavyranker.{TargetFeatureHydrator => RelevanceTargetFeatureHydrator}\nimport com.twitter.nrel.lightranker.MagicRecsServeDataRecordLightRanker\nimport com.twitter.nrel.lightranker.{Config => LightRankerConfig}\nimport com.twitter.onboarding.task.service.thriftscala.FatigueFlowEnrollment\nimport com.twitter.periscope.api.thriftscala.AudioSpacesLookupContext\nimport com.twitter.permissions_storage.thriftscala.AppPermission\nimport com.twitter.recommendation.interests.discovery.core.config.{DeployConfig => InterestDeployConfig}\nimport com.twitter.recommendation.interests.discovery.popgeo.deploy.PopGeoInterestProvider\nimport com.twitter.recos.user_tweet_entity_graph.thriftscala.UserTweetEntityGraph\nimport com.twitter.recos.user_user_graph.thriftscala.UserUserGraph\nimport com.twitter.rux.common.strato.thriftscala.UserTargetingProperty\nimport com.twitter.scio.nsfw_user_segmentation.thriftscala.NSFWProducer\nimport com.twitter.scio.nsfw_user_segmentation.thriftscala.NSFWUserSegmentation\nimport com.twitter.search.earlybird.thriftscala.EarlybirdService\nimport com.twitter.service.gen.scarecrow.thriftscala.ScarecrowService\nimport com.twitter.service.metastore.gen.thriftscala.Location\nimport com.twitter.simclusters_v2.thriftscala.SimClustersInferredEntities\nimport com.twitter.socialgraph.thriftscala.SocialGraphService\nimport com.twitter.spam.rtf.thriftscala.SafetyLevel\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storage.client.manhattan.kv.Guarantee\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClient\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpoint\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpointBuilder\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.Apollo\nimport com.twitter.storehaus_internal.manhattan.Athena\nimport com.twitter.storehaus_internal.manhattan.Dataset\nimport com.twitter.storehaus_internal.manhattan.ManhattanStore\nimport com.twitter.storehaus_internal.manhattan.Nash\nimport com.twitter.storehaus_internal.manhattan.Omega\nimport com.twitter.storehaus_internal.memcache.MemcacheStore\nimport com.twitter.storehaus_internal.util.ClientName\nimport com.twitter.storehaus_internal.util.ZkEndPoint\nimport com.twitter.strato.catalog.Scan.Slice\nimport com.twitter.strato.client.Strato\nimport com.twitter.strato.client.UserId\nimport com.twitter.strato.columns.frigate.logged_out_web_notifications.thriftscala.LOWebNotificationMetadata\nimport com.twitter.strato.columns.notifications.thriftscala.SourceDestUserRequest\nimport com.twitter.strato.generated.client.geo.user.FrequentSoftUserLocationClientColumn\nimport com.twitter.strato.generated.client.ml.featureStore.TimelineScorerTweetScoresV1ClientColumn\nimport com.twitter.strato.generated.client.notifications.space_device_follow_impl.SpaceDeviceFollowingClientColumn\nimport com.twitter.strato.generated.client.periscope.CoreOnAudioSpaceClientColumn\nimport com.twitter.strato.generated.client.periscope.ParticipantsOnAudioSpaceClientColumn\nimport com.twitter.strato.generated.client.rux.TargetingPropertyOnUserClientColumn\nimport com.twitter.strato.generated.client.socialgraph.graphs.creatorSubscriptionTimeline.{CountEdgesBySourceClientColumn => CreatorSubscriptionNumTweetsColumn}\nimport com.twitter.strato.generated.client.translation.service.IsTweetTranslatableClientColumn\nimport com.twitter.strato.generated.client.translation.service.platform.MachineTranslateTweetClientColumn\nimport com.twitter.strato.generated.client.trends.trip.TripTweetsAirflowProdClientColumn\nimport com.twitter.strato.thrift.ScroogeConvImplicits._\nimport com.twitter.taxi.common.AppId\nimport com.twitter.taxi.deploy.Cluster\nimport com.twitter.taxi.deploy.Env\nimport com.twitter.topiclisting.TopicListing\nimport com.twitter.topiclisting.TopicListingBuilder\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripDomain\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripTweets\nimport com.twitter.tsp.thriftscala.TopicSocialProofRequest\nimport com.twitter.tsp.thriftscala.TopicSocialProofResponse\nimport com.twitter.tweetypie.thriftscala.GetTweetOptions\nimport com.twitter.tweetypie.thriftscala.Tweet.VisibleTextRangeField\nimport com.twitter.tweetypie.thriftscala.TweetService\nimport com.twitter.ubs.thriftscala.AudioSpace\nimport com.twitter.ubs.thriftscala.Participants\nimport com.twitter.ubs.thriftscala.SellerApplicationState\nimport com.twitter.user_session_store.thriftscala.UserSession\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Timer\nimport com.twitter.util.tunable.TunableMap\nimport com.twitter.wtf.scalding.common.thriftscala.UserFeatures\nimport org.apache.thrift.protocol.TCompactProtocol\nimport com.twitter.timelinescorer.thriftscala.v1.ScoredTweet\nimport com.twitter.ubs.thriftscala.SellerTrack\nimport com.twitter.wtf.candidate.thriftscala.CandidateSeq\n\ntrait DeployConfig extends Config {\n  // Any finagle clients should not be defined as lazy. If defined lazy,\n  // ClientRegistry.expAllRegisteredClientsResolved() call in init will not ensure that the clients\n  // are active before thrift endpoint is active. We want the clients to be active, because zookeeper\n  // resolution triggered by first request(s) might result in the request(s) failing.\n\n  def serviceIdentifier: ServiceIdentifier\n\n  def tunableMap: TunableMap\n\n  def featureSwitches: FeatureSwitches\n\n  override val isProd: Boolean =\n    serviceIdentifier.environment == PushConstants.ServiceProdEnvironmentName\n\n  def shardParams: ShardParams\n\n  def log: Logger\n\n  implicit def statsReceiver: StatsReceiver\n\n  implicit val timer: Timer = DefaultTimer\n\n  def notifierThriftClientId: ClientId\n\n  def loggedOutNotifierThriftClientId: ClientId\n\n  def pushserviceThriftClientId: ClientId\n\n  def deepbirdv2PredictionServiceDest: String\n\n  def featureStoreUtil: FeatureStoreUtil\n\n  def targetLevelFeaturesConfig: PushFeaturesConfig\n\n  private val manhattanClientMtlsParams = ManhattanKVClientMtlsParams(\n    serviceIdentifier = serviceIdentifier,\n    opportunisticTls = OpportunisticTls.Required\n  )\n\n  // Commonly used clients\n  val gizmoduckClient = {\n\n    val client = ThriftMux.client\n      .withMutualTls(serviceIdentifier)\n      .withClientId(pushserviceThriftClientId)\n      .build[UserService.MethodPerEndpoint](\n        dest = \"/s/gizmoduck/gizmoduck\"\n      )\n\n    /**\n     * RequestContext test user config to allow reading test user accounts on pushservice for load\n     * testing\n     */\n    val GizmoduckTestUserConfig = TestUserConfig(\n      clientId = Some(pushserviceThriftClientId.name),\n      readConfig = Some(ReadConfig(includeTestUsers = true))\n    )\n\n    TestUserClientBuilder[UserService.MethodPerEndpoint]\n      .withClient(client)\n      .withConfig(GizmoduckTestUserConfig)\n      .build()\n  }\n\n  val sgsClient = {\n    val service = readOnlyThriftService(\n      \"\",\n      \"/s/socialgraph/socialgraph\",\n      statsReceiver,\n      pushserviceThriftClientId,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    )\n    new SocialGraphService.FinagledClient(service)\n  }\n\n  val tweetyPieClient = {\n    val service = readOnlyThriftService(\n      \"\",\n      \"/s/tweetypie/tweetypie\",\n      statsReceiver,\n      notifierThriftClientId,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    )\n    new TweetService.FinagledClient(service)\n  }\n\n  lazy val geoduckHydrationClient: Hydration.MethodPerEndpoint = {\n    val servicePerEndpoint = ThriftMux.client\n      .withLabel(\"geoduck_hydration\")\n      .withClientId(pushserviceThriftClientId)\n      .withMutualTls(serviceIdentifier)\n      .methodBuilder(\"/s/geo/hydration\")\n      .withTimeoutPerRequest(10.seconds)\n      .withTimeoutTotal(10.seconds)\n      .idempotent(maxExtraLoad = 0.0)\n      .servicePerEndpoint[Hydration.ServicePerEndpoint]\n    Hydration.MethodPerEndpoint(servicePerEndpoint)\n  }\n\n  lazy val geoduckLocationClient: LocationService.MethodPerEndpoint = {\n    val servicePerEndpoint = ThriftMux.client\n      .withLabel(\"geoduck_location\")\n      .withClientId(pushserviceThriftClientId)\n      .withMutualTls(serviceIdentifier)\n      .methodBuilder(\"/s/geo/geoduck_locationservice\")\n      .withTimeoutPerRequest(10.seconds)\n      .withTimeoutTotal(10.seconds)\n      .idempotent(maxExtraLoad = 0.0)\n      .servicePerEndpoint[LocationService.ServicePerEndpoint]\n    LocationService.MethodPerEndpoint(servicePerEndpoint)\n  }\n\n  override val geoDuckV2Store: ReadableStore[Long, LocationResponse] = {\n    val geoduckLocate: GeoduckUserLocate = GeoduckUserLocateModule.providesGeoduckUserLocate(\n      locationServiceClient = geoduckLocationClient,\n      hydrationClient = geoduckHydrationClient,\n      unscopedStatsReceiver = statsReceiver\n    )\n\n    val store: ReadableStore[Long, LocationResponse] = ReadableStore\n      .convert[GeoduckRequest, Long, LocationResponse, LocationResponse](\n        GeoduckStoreV2(geoduckLocate))({ userId: Long =>\n        GeoduckRequest(\n          userId,\n          placeTypes = Set(\n            PlaceType.City,\n            PlaceType.Metro,\n            PlaceType.Country,\n            PlaceType.ZipCode,\n            PlaceType.Admin0,\n            PlaceType.Admin1),\n          placeFields = Set(PlaceQueryFields.PlaceNames),\n          includeCountryCode = true\n        )\n      })({ locationResponse: LocationResponse => Future.value(locationResponse) })\n\n    val _cacheName = \"geoduckv2_in_memory_cache\"\n    ObservedCachedReadableStore.from(\n      store,\n      ttl = 20.seconds,\n      maxKeys = 1000,\n      cacheName = _cacheName,\n      windowSize = 10000L\n    )(statsReceiver.scope(_cacheName))\n  }\n\n  private val deepbirdServiceBase = ThriftMux.client\n    .withClientId(pushserviceThriftClientId)\n    .withMutualTls(serviceIdentifier)\n    .withLoadBalancer(Balancers.p2c())\n    .newService(deepbirdv2PredictionServiceDest, \"DeepbirdV2PredictionService\")\n  val deepbirdPredictionServiceClient = new DeepbirdPredictionService.ServiceToClient(\n    Finagle\n      .retryReadFilter(\n        tries = 3,\n        statsReceiver = statsReceiver.scope(\"DeepbirdV2PredictionService\"))\n      .andThen(Finagle.timeoutFilter(timeout = 10.seconds))\n      .andThen(deepbirdServiceBase),\n    RichClientParam(serviceName = \"DeepbirdV2PredictionService\", clientStats = statsReceiver)\n  )\n\n  val manhattanStarbuckAppId = \"frigate_pushservice_starbuck\"\n  val metastoreLocationAppId = \"frigate_notifier_metastore_location\"\n  val manhattanMetastoreAppId = \"frigate_pushservice_penguin\"\n\n  def pushServiceMHCacheDest: String\n  def pushServiceCoreSvcsCacheDest: String\n  def poptartImpressionsCacheDest: String = \"/srv#/prod/local/cache/poptart_impressions\"\n  def entityGraphCacheDest: String\n\n  val pushServiceCacheClient: Client = MemcacheStore.memcachedClient(\n    name = ClientName(\"memcache-pushservice\"),\n    dest = ZkEndPoint(pushServiceMHCacheDest),\n    statsReceiver = statsReceiver,\n    timeout = 2.seconds,\n    serviceIdentifier = serviceIdentifier\n  )\n\n  val pushServiceCoreSvcsCacheClient: Client =\n    MemcacheStore.memcachedClient(\n      name = ClientName(\"memcache-pushservice-core-svcs\"),\n      dest = ZkEndPoint(pushServiceCoreSvcsCacheDest),\n      statsReceiver = statsReceiver,\n      serviceIdentifier = serviceIdentifier,\n      timeout = 2.seconds,\n    )\n\n  val poptartImpressionsCacheClient: Client =\n    MemcacheStore.memcachedClient(\n      name = ClientName(\"memcache-pushservice-poptart-impressions\"),\n      dest = ZkEndPoint(poptartImpressionsCacheDest),\n      statsReceiver = statsReceiver,\n      serviceIdentifier = serviceIdentifier,\n      timeout = 2.seconds\n    )\n\n  val entityGraphCacheClient: Client = MemcacheStore.memcachedClient(\n    name = ClientName(\"memcache-pushservice-entity-graph\"),\n    dest = ZkEndPoint(entityGraphCacheDest),\n    statsReceiver = statsReceiver,\n    serviceIdentifier = serviceIdentifier,\n    timeout = 2.seconds\n  )\n\n  val stratoClient = {\n    val pushserviceThriftClient = ThriftMux.client.withClientId(pushserviceThriftClientId)\n    val baseBuilder = Strato\n      .Client(pushserviceThriftClient)\n      .withMutualTls(serviceIdentifier)\n    val finalBuilder = if (isServiceLocal) {\n      baseBuilder.withRequestTimeout(Duration.fromSeconds(15))\n    } else {\n      baseBuilder.withRequestTimeout(Duration.fromSeconds(3))\n    }\n    finalBuilder.build()\n  }\n\n  val interestThriftServiceClient = ThriftMux.client\n    .withClientId(pushserviceThriftClientId)\n    .withMutualTls(serviceIdentifier)\n    .withRequestTimeout(3.seconds)\n    .configured(Retries.Policy(RetryPolicy.tries(1)))\n    .configured(BackupRequestFilter.Configured(maxExtraLoad = 0.0, sendInterrupts = false))\n    .withStatsReceiver(statsReceiver)\n    .build[InterestsThriftService.MethodPerEndpoint](\n      dest = \"/s/interests-thrift-service/interests-thrift-service\",\n      label = \"interests-lookup\"\n    )\n\n  def memcacheCASDest: String\n\n  override val casLock: CasLock = {\n    val magicrecsCasMemcacheClient = Memcached.client\n      .withMutualTls(serviceIdentifier)\n      .withLabel(\"mr-cas-memcache-client\")\n      .withRequestTimeout(3.seconds)\n      .withStatsReceiver(statsReceiver)\n      .configured(Retries.Policy(RetryPolicy.tries(3)))\n      .newTwemcacheClient(memcacheCASDest)\n      .withStrings\n\n    MemcacheCasLock(magicrecsCasMemcacheClient)\n  }\n\n  override val pushInfoStore: ReadableStore[Long, UserForPushTargeting] = {\n    StratoFetchableStore.withUnitView[Long, UserForPushTargeting](\n      stratoClient,\n      \"frigate/magicrecs/pushRecsTargeting.User\")\n  }\n\n  override val loggedOutPushInfoStore: ReadableStore[Long, LOWebNotificationMetadata] = {\n    StratoFetchableStore.withUnitView[Long, LOWebNotificationMetadata](\n      stratoClient,\n      \"frigate/magicrecs/web/loggedOutWebUserStoreMh\"\n    )\n  }\n\n  // Setting up model stores\n  override val dauProbabilityStore: ReadableStore[Long, DauProbability] = {\n    StratoFetchableStore\n      .withUnitView[Long, DauProbability](stratoClient, \"frigate/magicrecs/dauProbability.User\")\n  }\n\n  override val nsfwConsumerStore = {\n    StratoFetchableStore.withUnitView[Long, NSFWUserSegmentation](\n      stratoClient,\n      \"frigate/nsfw-user-segmentation/nsfwUserSegmentation.User\")\n  }\n\n  override val nsfwProducerStore = {\n    StratoFetchableStore.withUnitView[Long, NSFWProducer](\n      stratoClient,\n      \"frigate/nsfw-user-segmentation/nsfwProducer.User\"\n    )\n  }\n\n  override val idsStore: ReadableStore[RecommendedListsRequest, RecommendedListsResponse] = {\n    val service = Finagle.readOnlyThriftService(\n      name = \"interests-discovery-service\",\n      dest = \"/s/interests_discovery/interests_discovery\",\n      statsReceiver,\n      pushserviceThriftClientId,\n      requestTimeout = 4.seconds,\n      tries = 2,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    )\n    val client = new InterestsDiscoveryService.FinagledClient(\n      service = service,\n      RichClientParam(serviceName = \"interests-discovery-service\")\n    )\n\n    InterestDiscoveryStore(client)\n  }\n\n  override val popGeoLists = {\n    StratoFetchableStore.withUnitView[String, NonPersonalizedRecommendedLists](\n      stratoClient,\n      column = \"recommendations/interests_discovery/recommendations_mh/OrganicPopgeoLists\"\n    )\n  }\n\n  override val listAPIStore = {\n    val fetcher = stratoClient\n      .fetcher[Long, ApiListView, ApiList](\"channels/hydration/apiList.List\")\n    StratoFetchableStore.withView[Long, ApiListView, ApiList](\n      fetcher,\n      ApiListView(ApiListDisplayLocation.Recommendations)\n    )\n  }\n\n  override val reactivatedUserInfoStore = {\n    val stratoFetchableStore = StratoFetchableStore\n      .withUnitView[Long, String](stratoClient, \"ml/featureStore/recentReactivationTime.User\")\n\n    ObservedReadableStore(\n      stratoFetchableStore\n    )(statsReceiver.scope(\"RecentReactivationTime\"))\n  }\n\n  override val openedPushByHourAggregatedStore: ReadableStore[Long, Map[Int, Int]] = {\n    StratoFetchableStore\n      .withUnitView[Long, Map[Int, Int]](\n        stratoClient,\n        \"frigate/magicrecs/opendPushByHourAggregated.User\")\n  }\n\n  private val lexClient: LiveVideoTimelineClient = {\n    val lexService =\n      new TimelineService.FinagledClient(\n        readOnlyThriftService(\n          name = \"lex\",\n          dest = lexServiceDest,\n          statsReceiver = statsReceiver.scope(\"lex-service\"),\n          thriftClientId = pushserviceThriftClientId,\n          requestTimeout = 5.seconds,\n          mTLSServiceIdentifier = Some(serviceIdentifier)\n        ),\n        clientParam = RichClientParam(serviceName = \"lex\")\n      )\n    new LiveVideoTimelineClient(lexService)\n  }\n\n  override val lexServiceStore = {\n    ObservedCachedReadableStore.from[EventRequest, LiveEvent](\n      buildStore(LexServiceStore(lexClient), \"lexServiceStore\"),\n      ttl = 1.hour,\n      maxKeys = 1000,\n      cacheName = \"lexServiceStore_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"lexServiceStore_cache\"))\n  }\n\n  val inferredEntitiesFromInterestedInKeyedByClusterColumn =\n    \"recommendations/simclusters_v2/inferred_entities/inferredEntitiesFromInterestedInKeyedByCluster\"\n  override val simClusterToEntityStore: ReadableStore[Int, SimClustersInferredEntities] = {\n    val store = StratoFetchableStore\n      .withUnitView[Int, SimClustersInferredEntities](\n        stratoClient,\n        inferredEntitiesFromInterestedInKeyedByClusterColumn)\n    ObservedCachedReadableStore.from[Int, SimClustersInferredEntities](\n      buildStore(store, \"simcluster_entity_store_cache\"),\n      ttl = 6.hours,\n      maxKeys = 1000,\n      cacheName = \"simcluster_entity_store_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"simcluster_entity_store_cache\"))\n  }\n\n  def fanoutMetadataColumn: String\n\n  override val fanoutMetadataStore: ReadableStore[(Long, Long), FanoutEvent] = {\n    val store = StratoFetchableStore\n      .withUnitView[(Long, Long), FanoutEvent](stratoClient, fanoutMetadataColumn)\n    ObservedCachedReadableStore.from[(Long, Long), FanoutEvent](\n      buildStore(store, \"fanoutMetadataStore\"),\n      ttl = 10.minutes,\n      maxKeys = 1000,\n      cacheName = \"fanoutMetadataStore_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"fanoutMetadataStore_cache\"))\n  }\n\n  /**\n   * PostRanking Feature Store Client\n   */\n  override def postRankingFeatureStoreClient = {\n    val clientStats = statsReceiver.scope(\"post_ranking_feature_store_client\")\n    val clientConfig =\n      FeatureStoreClientBuilder.getClientConfig(PostRankingFeaturesConfig(), featureStoreUtil)\n\n    FeatureStoreClientBuilder.getDynamicFeatureStoreClient(clientConfig, clientStats)\n  }\n\n  /**\n   * Interests lookup store\n   */\n  override val interestsWithLookupContextStore = {\n    ObservedCachedReadableStore.from[InterestsLookupRequestWithContext, Interests](\n      buildStore(\n        new InterestsWithLookupContextStore(interestThriftServiceClient, statsReceiver),\n        \"InterestsWithLookupContextStore\"\n      ),\n      ttl = 1.minute,\n      maxKeys = 1000,\n      cacheName = \"interestsWithLookupContextStore_cache\",\n      windowSize = 10000L\n    )\n  }\n\n  /**\n   * OptOutInterestsStore\n   */\n  override lazy val optOutUserInterestsStore: ReadableStore[Long, Seq[InterestId]] = {\n    buildStore(\n      InterestsOptOutwithLookUpContextStore(interestThriftServiceClient),\n      \"InterestsOptOutStore\"\n    )\n  }\n\n  override val topicListing: TopicListing =\n    if (isServiceLocal) {\n      new TopicListingBuilder(statsReceiver.scope(\"topiclisting\"), Some(localConfigRepoPath)).build\n    } else {\n      new TopicListingBuilder(statsReceiver.scope(\"topiclisting\"), None).build\n    }\n\n  val cachedUttClient = {\n    val DefaultUttCacheConfig = CacheConfigV2(capacity = 100)\n    val uttClientCacheConfigs = uttclient.UttClientCacheConfigsV2(\n      DefaultUttCacheConfig,\n      DefaultUttCacheConfig,\n      DefaultUttCacheConfig,\n      DefaultUttCacheConfig\n    )\n    new CachedUttClientV2(stratoClient, Environment.Prod, uttClientCacheConfigs, statsReceiver)\n  }\n\n  override val uttEntityHydrationStore =\n    new UttEntityHydrationStore(cachedUttClient, statsReceiver, log)\n\n  private lazy val dbv2PredictionServiceScoreStore: RelevancePushPredictionServiceStore =\n    DeepbirdV2ModelConfig.buildPredictionServiceScoreStore(\n      deepbirdPredictionServiceClient,\n      \"deepbirdv2_magicrecs\"\n    )\n\n  // Customized model to PredictionServiceStoreMap\n  // It is used to specify the predictionServiceStore for the models not in the default dbv2PredictionServiceScoreStore\n  private lazy val modelToPredictionServiceStoreMap: Map[\n    WeightedOpenOrNtabClickModel.ModelNameType,\n    RelevancePushPredictionServiceStore\n  ] = Map()\n\n  override lazy val weightedOpenOrNtabClickModelScorer = new PushMLModelScorer(\n    PushMLModel.WeightedOpenOrNtabClickProbability,\n    modelToPredictionServiceStoreMap,\n    dbv2PredictionServiceScoreStore,\n    statsReceiver.scope(\"weighted_oonc_scoring\")\n  )\n\n  override lazy val optoutModelScorer = new PushMLModelScorer(\n    PushMLModel.OptoutProbability,\n    Map.empty,\n    dbv2PredictionServiceScoreStore,\n    statsReceiver.scope(\"optout_scoring\")\n  )\n\n  override lazy val filteringModelScorer = new PushMLModelScorer(\n    PushMLModel.FilteringProbability,\n    Map.empty,\n    dbv2PredictionServiceScoreStore,\n    statsReceiver.scope(\"filtering_scoring\")\n  )\n\n  private val queryFields: Set[QueryFields] = Set(\n    QueryFields.Profile,\n    QueryFields.Account,\n    QueryFields.Roles,\n    QueryFields.Discoverability,\n    QueryFields.Safety,\n    QueryFields.Takedowns,\n    QueryFields.Labels,\n    QueryFields.Counts,\n    QueryFields.ExtendedProfile\n  )\n\n  // Setting up safeUserStore\n  override val safeUserStore =\n    // in-memory cache\n    ObservedCachedReadableStore.from[Long, User](\n      ObservedReadableStore(\n        GizmoduckUserStore.safeStore(\n          client = gizmoduckClient,\n          queryFields = queryFields,\n          safetyLevel = SafetyLevel.FilterNone,\n          statsReceiver = statsReceiver\n        )\n      )(statsReceiver.scope(\"SafeUserStore\")),\n      ttl = 1.minute,\n      maxKeys = 5e4.toInt,\n      cacheName = \"safeUserStore_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"safeUserStore_cache\"))\n\n  val mobileSdkStore = MobileSdkStore(\n    \"frigate_mobile_sdk_version_apollo\",\n    \"mobile_sdk_versions_scalding\",\n    manhattanClientMtlsParams,\n    Apollo\n  )\n\n  val deviceUserStore = ObservedReadableStore(\n    GizmoduckUserStore(\n      client = gizmoduckClient,\n      queryFields = Set(QueryFields.Devices),\n      context = LookupContext(includeSoftUsers = true),\n      statsReceiver = statsReceiver\n    )\n  )(statsReceiver.scope(\"devicesUserStore\"))\n\n  override val deviceInfoStore = DeviceInfoStore(\n    ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = ObservedReadableStore(\n        mobileSdkStore\n      )(statsReceiver.scope(\"uncachedMobileSdkVersionsStore\")),\n      cacheClient = pushServiceCacheClient,\n      ttl = 12.hours\n    )(\n      valueInjection = BinaryScalaCodec(SdkVersionValue),\n      statsReceiver = statsReceiver.scope(\"MobileSdkVersionsStore\"),\n      keyToString = {\n        case SdkVersionKey(Some(userId), Some(clientId)) =>\n          s\"DeviceInfoStore/$userId/$clientId\"\n        case SdkVersionKey(Some(userId), None) => s\"DeviceInfoStore/$userId/_\"\n        case SdkVersionKey(None, Some(clientId)) =>\n          s\"DeviceInfoStore/_/$clientId\"\n        case SdkVersionKey(None, None) => s\"DeviceInfoStore/_\"\n      }\n    ),\n    deviceUserStore\n  )\n\n  // Setting up edgeStore\n  override val edgeStore = SocialGraphPredicate.buildEdgeStore(sgsClient)\n\n  override val socialGraphServiceProcessStore = SocialGraphServiceProcessStore(edgeStore)\n\n  def userTweetEntityGraphDest: String\n  def userUserGraphDest: String\n  def lexServiceDest: String\n\n  // Setting up the history store\n  def frigateHistoryCacheDest: String\n\n  val notificationHistoryStore: NotificationHistoryStore = {\n\n    val manhattanStackBasedClient = ThriftMux.client\n      .withClientId(notifierThriftClientId)\n      .withOpportunisticTls(OpportunisticTls.Required)\n      .withMutualTls(\n        serviceIdentifier\n      )\n\n    val manhattanHistoryMethodBuilder = manhattanStackBasedClient\n      .withLabel(\"manhattan_history_v2\")\n      .withRequestTimeout(10.seconds)\n      .withStatsReceiver(statsReceiver)\n      .methodBuilder(Omega.wilyName)\n      .withMaxRetries(3)\n\n    NotificationHistoryStore.build(\n      \"frigate_notifier\",\n      \"frigate_notifications_v2\",\n      manhattanHistoryMethodBuilder,\n      maxRetryCount = 3\n    )\n  }\n\n  val emailNotificationHistoryStore: ReadOnlyHistoryStore = {\n    val client = ManhattanKVClient(\n      appId = \"frigate_email_history\",\n      dest = \"/s/manhattan/omega.native-thrift\",\n      mtlsParams = ManhattanKVClientMtlsParams(\n        serviceIdentifier = serviceIdentifier,\n        opportunisticTls = OpportunisticTls.Required\n      )\n    )\n    val endpoint = ManhattanKVEndpointBuilder(client)\n      .defaultGuarantee(Guarantee.SoftDcReadMyWrites)\n      .statsReceiver(statsReceiver)\n      .build()\n\n    ReadOnlyHistoryStore(ManhattanKVHistoryStore(endpoint, dataset = \"frigate_email_history\"))(\n      statsReceiver)\n  }\n\n  val manhattanKVLoggedOutHistoryStoreEndpoint: ManhattanKVEndpoint = {\n    val mhClient = ManhattanKVClient(\n      \"frigate_notification_logged_out_history\",\n      Nash.wilyName,\n      manhattanClientMtlsParams)\n    ManhattanKVEndpointBuilder(mhClient)\n      .defaultGuarantee(Guarantee.SoftDcReadMyWrites)\n      .defaultMaxTimeout(5.seconds)\n      .maxRetryCount(3)\n      .statsReceiver(statsReceiver)\n      .build()\n  }\n\n  val manhattanKVNtabHistoryStoreEndpoint: ManhattanKVEndpoint = {\n    val mhClient = ManhattanKVClient(\"frigate_ntab\", Omega.wilyName, manhattanClientMtlsParams)\n    ManhattanKVEndpointBuilder(mhClient)\n      .defaultGuarantee(Guarantee.SoftDcReadMyWrites)\n      .defaultMaxTimeout(5.seconds)\n      .maxRetryCount(3)\n      .statsReceiver(statsReceiver)\n      .build()\n  }\n\n  val nTabHistoryStore: ReadableWritableStore[(Long, String), GenericNotificationOverrideKey] = {\n    ObservedReadableWritableStore(\n      NTabHistoryStore(manhattanKVNtabHistoryStoreEndpoint, \"frigate_ntab_generic_notif_history\")\n    )(statsReceiver.scope(\"NTabHistoryStore\"))\n  }\n\n  override lazy val ocfFatigueStore: ReadableStore[OCFHistoryStoreKey, FatigueFlowEnrollment] =\n    new OCFPromptHistoryStore(\n      manhattanAppId = \"frigate_pushservice_ocf_fatigue_store\",\n      dataset = \"fatigue_v1\",\n      manhattanClientMtlsParams\n    )\n\n  def historyStore: PushServiceHistoryStore\n\n  def emailHistoryStore: PushServiceHistoryStore\n\n  def loggedOutHistoryStore: PushServiceHistoryStore\n\n  override val hydratedLabeledPushRecsStore: ReadableStore[UserHistoryKey, UserHistoryValue] = {\n    val labeledHistoryMemcacheClient = {\n      MemcacheStore.memcachedClient(\n        name = ClientName(\"history-memcache\"),\n        dest = ZkEndPoint(frigateHistoryCacheDest),\n        statsReceiver = statsReceiver,\n        timeout = 2.seconds,\n        serviceIdentifier = serviceIdentifier\n      )\n    }\n\n    implicit val keyCodec = CompactScalaCodec(UserHistoryKey)\n    implicit val valueCodec = CompactScalaCodec(UserHistoryValue)\n    val dataset: Dataset[UserHistoryKey, UserHistoryValue] =\n      Dataset(\n        \"\",\n        \"frigate_data_pipeline_pushservice\",\n        \"labeled_push_recs_aggregated_hydrated\",\n        Athena\n      )\n    ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = ObservedReadableStore(buildManhattanStore(dataset))(\n        statsReceiver.scope(\"UncachedHydratedLabeledPushRecsStore\")\n      ),\n      cacheClient = labeledHistoryMemcacheClient,\n      ttl = 6.hours\n    )(\n      valueInjection = valueCodec,\n      statsReceiver = statsReceiver.scope(\"HydratedLabeledPushRecsStore\"),\n      keyToString = {\n        case UserHistoryKey.UserId(userId) => s\"HLPRS/$userId\"\n        case unknownKey =>\n          throw new IllegalArgumentException(s\"Unknown userHistoryStore cache key $unknownKey\")\n      }\n    )\n  }\n\n  override val realTimeClientEventStore: RealTimeClientEventStore = {\n    val client = ManhattanKVClient(\n      \"frigate_eventstream\",\n      \"/s/manhattan/omega.native-thrift\",\n      manhattanClientMtlsParams\n    )\n    val endpoint =\n      ManhattanKVEndpointBuilder(client)\n        .defaultGuarantee(Guarantee.SoftDcReadMyWrites)\n        .defaultMaxTimeout(3.seconds)\n        .statsReceiver(statsReceiver)\n        .build()\n\n    ManhattanRealTimeClientEventStore(endpoint, \"realtime_client_events\", statsReceiver, None)\n  }\n\n  override val onlineUserHistoryStore: ReadableStore[OnlineUserHistoryKey, UserHistoryValue] = {\n    OnlineUserHistoryStore(realTimeClientEventStore)\n  }\n\n  override val userMediaRepresentationStore = UserMediaRepresentationStore(\n    \"user_media_representation\",\n    \"user_media_representation_dataset\",\n    manhattanClientMtlsParams\n  )\n\n  override val producerMediaRepresentationStore = ObservedMemcachedReadableStore.fromCacheClient(\n    backingStore = UserMediaRepresentationStore(\n      \"user_media_representation\",\n      \"producer_media_representation_dataset\",\n      manhattanClientMtlsParams\n    )(statsReceiver.scope(\"UncachedProducerMediaRepStore\")),\n    cacheClient = pushServiceCacheClient,\n    ttl = 4.hours\n  )(\n    valueInjection = BinaryScalaCodec(UserMediaRepresentation),\n    keyToString = { k: Long => s\"ProducerMediaRepStore/$k\" },\n    statsReceiver.scope(\"ProducerMediaRepStore\")\n  )\n\n  override val mrUserStatePredictionStore = {\n    StratoFetchableStore.withUnitView[Long, MRUserHmmState](\n      stratoClient,\n      \"frigate/magicrecs/mrUserStatePrediction.User\")\n  }\n\n  override val userHTLLastVisitStore =\n    UserHTLLastVisitReadableStore(\n      \"pushservice_htl_user_session\",\n      \"tls_user_session_store\",\n      statsReceiver.scope(\"userHTLLastVisitStore\"),\n      manhattanClientMtlsParams\n    )\n\n  val crMixerClient: CrMixer.MethodPerEndpoint = new CrMixer.FinagledClient(\n    readOnlyThriftService(\n      \"cr-mixer\",\n      \"/s/cr-mixer/cr-mixer-plus\",\n      statsReceiver,\n      pushserviceThriftClientId,\n      requestTimeout = 5.seconds,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    ),\n    clientParam = RichClientParam(serviceName = \"cr-mixer\")\n  )\n\n  val crMixerStore = CrMixerTweetStore(crMixerClient)(statsReceiver.scope(\"CrMixerTweetStore\"))\n\n  val contentMixerClient: ContentMixer.MethodPerEndpoint = new ContentMixer.FinagledClient(\n    readOnlyThriftService(\n      \"content-mixer\",\n      \"/s/corgi-shared/content-mixer\",\n      statsReceiver,\n      pushserviceThriftClientId,\n      requestTimeout = 5.seconds,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    ),\n    clientParam = RichClientParam(serviceName = \"content-mixer\")\n  )\n\n  val exploreRankerClient: ExploreRanker.MethodPerEndpoint =\n    new ExploreRanker.FinagledClient(\n      readOnlyThriftService(\n        \"explore-ranker\",\n        \"/s/explore-ranker/explore-ranker\",\n        statsReceiver,\n        pushserviceThriftClientId,\n        requestTimeout = 5.seconds,\n        mTLSServiceIdentifier = Some(serviceIdentifier)\n      ),\n      clientParam = RichClientParam(serviceName = \"explore-ranker\")\n    )\n\n  val contentMixerStore = {\n    ObservedReadableStore(ContentMixerStore(contentMixerClient))(\n      statsReceiver.scope(\"ContentMixerStore\"))\n  }\n\n  val exploreRankerStore = {\n    ObservedReadableStore(ExploreRankerStore(exploreRankerClient))(\n      statsReceiver.scope(\"ExploreRankerStore\")\n    )\n  }\n\n  val gizmoduckUtcOffsetStore = ObservedReadableStore(\n    GizmoduckUserUtcOffsetStore.fromUserStore(safeUserStore)\n  )(statsReceiver.scope(\"GizmoUserUtcOffsetStore\"))\n\n  override val userUtcOffsetStore =\n    UtcOffsetStore\n      .makeMemcachedUtcOffsetStore(\n        gizmoduckUtcOffsetStore,\n        pushServiceCoreSvcsCacheClient,\n        ReadableStore.empty,\n        manhattanStarbuckAppId,\n        manhattanClientMtlsParams\n      )(statsReceiver)\n      .mapValues(Duration.fromSeconds)\n\n  override val cachedTweetyPieStoreV2 = {\n    val getTweetOptions = Some(\n      GetTweetOptions(\n        safetyLevel = Some(SafetyLevel.MagicRecsV2),\n        includeRetweetCount = true,\n        includeReplyCount = true,\n        includeFavoriteCount = true,\n        includeQuotedTweet = true,\n        additionalFieldIds = Seq(VisibleTextRangeField.id)\n      )\n    )\n    buildCachedTweetyPieStore(getTweetOptions, \"tp_v2\")\n  }\n\n  override val cachedTweetyPieStoreV2NoVF = {\n    val getTweetOptions = Some(\n      GetTweetOptions(\n        safetyLevel = Some(SafetyLevel.FilterDefault),\n        includeRetweetCount = true,\n        includeReplyCount = true,\n        includeFavoriteCount = true,\n        includeQuotedTweet = true,\n        additionalFieldIds = Seq(VisibleTextRangeField.id),\n      )\n    )\n    buildCachedTweetyPieStore(getTweetOptions, \"tp_v2_noVF\")\n  }\n\n  override val safeCachedTweetyPieStoreV2 = {\n    val getTweetOptions = Some(\n      GetTweetOptions(\n        safetyLevel = Some(SafetyLevel.MagicRecsAggressiveV2),\n        includeRetweetCount = true,\n        includeReplyCount = true,\n        includeFavoriteCount = true,\n        includeQuotedTweet = true,\n        additionalFieldIds = Seq(VisibleTextRangeField.id)\n      )\n    )\n    buildCachedTweetyPieStore(getTweetOptions, \"sftp_v2\")\n  }\n\n  override val userTweetTweetyPieStore: ReadableStore[UserTweet, TweetyPieResult] = {\n    val getTweetOptions = Some(\n      GetTweetOptions(\n        safetyLevel = Some(SafetyLevel.MagicRecsV2),\n        includeRetweetCount = true,\n        includeReplyCount = true,\n        includeFavoriteCount = true,\n        includeQuotedTweet = true,\n        additionalFieldIds = Seq(VisibleTextRangeField.id)\n      )\n    )\n    TweetyPieStore.buildUserTweetStore(\n      client = tweetyPieClient,\n      options = getTweetOptions\n    )\n  }\n\n  override val safeUserTweetTweetyPieStore: ReadableStore[UserTweet, TweetyPieResult] = {\n    val getTweetOptions = Some(\n      GetTweetOptions(\n        safetyLevel = Some(SafetyLevel.MagicRecsAggressiveV2),\n        includeRetweetCount = true,\n        includeReplyCount = true,\n        includeFavoriteCount = true,\n        includeQuotedTweet = true,\n        additionalFieldIds = Seq(VisibleTextRangeField.id)\n      )\n    )\n    TweetyPieStore.buildUserTweetStore(\n      client = tweetyPieClient,\n      options = getTweetOptions\n    )\n  }\n\n  override val tweetContentFeatureCacheStore: ReadableStore[Long, ThriftDataRecord] = {\n    ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = TweetContentFeatureReadableStore(stratoClient),\n      cacheClient = poptartImpressionsCacheClient,\n      ttl = 12.hours\n    )(\n      valueInjection = BinaryScalaCodec(ThriftDataRecord),\n      statsReceiver = statsReceiver.scope(\"TweetContentFeaturesCacheStore\"),\n      keyToString = { k: Long => s\"tcf/$k\" }\n    )\n  }\n\n  lazy val tweetTranslationStore: ReadableStore[\n    TweetTranslationStore.Key,\n    TweetTranslationStore.Value\n  ] = {\n    val isTweetTranslatableStore =\n      StratoFetchableStore\n        .withUnitView[IsTweetTranslatableClientColumn.Key, Boolean](\n          fetcher = new IsTweetTranslatableClientColumn(stratoClient).fetcher\n        )\n\n    val translateTweetStore =\n      StratoFetchableStore\n        .withUnitView[MachineTranslateTweetClientColumn.Key, MachineTranslationResponse](\n          fetcher = new MachineTranslateTweetClientColumn(stratoClient).fetcher\n        )\n\n    ObservedReadableStore(\n      TweetTranslationStore(translateTweetStore, isTweetTranslatableStore, statsReceiver)\n    )(statsReceiver.scope(\"tweetTranslationStore\"))\n  }\n\n  val scarecrowClient = new ScarecrowService.FinagledClient(\n    readOnlyThriftService(\n      \"\",\n      \"/s/abuse/scarecrow\",\n      statsReceiver,\n      notifierThriftClientId,\n      requestTimeout = 5.second,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    ),\n    clientParam = RichClientParam(serviceName = \"\")\n  )\n\n  // Setting up scarecrow store\n  override val scarecrowCheckEventStore = {\n    ScarecrowCheckEventStore(scarecrowClient)\n  }\n\n  // setting up the perspective store\n  override val userTweetPerspectiveStore = {\n    val service = new DynamicRequestMeterFilter(\n      tunableMap(PushServiceTunableKeys.TweetPerspectiveStoreQpsLimit),\n      RateLimiterGenerator.asTuple(_, shardParams.numShards, 40),\n      PushQPSLimitConstants.PerspectiveStoreQPS)(timer)\n      .andThen(\n        readOnlyThriftService(\n          \"tweetypie_perspective_service\",\n          \"/s/tweetypie/tweetypie\",\n          statsReceiver,\n          notifierThriftClientId,\n          mTLSServiceIdentifier = Some(serviceIdentifier)\n        )\n      )\n\n    val client = new TweetService.FinagledClient(\n      service,\n      clientParam = RichClientParam(serviceName = \"tweetypie_perspective_client\"))\n    ObservedReadableStore(\n      PerspectiveReadableStore(client)\n    )(statsReceiver.scope(\"TweetPerspectiveStore\"))\n  }\n\n  //user country code store, used in RecsWithheldContentPredicate - wrapped by memcache based cache\n  override val userCountryStore =\n    ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = ObservedReadableStore(\n        UserCountryStore(metastoreLocationAppId, manhattanClientMtlsParams)\n      )(statsReceiver.scope(\"userCountryStore\")),\n      cacheClient = pushServiceCacheClient,\n      ttl = 12.hours\n    )(\n      valueInjection = BinaryScalaCodec(Location),\n      statsReceiver = statsReceiver.scope(\"UserCountryStore\"),\n      keyToString = { k: Long => s\"UserCountryStore/$k\" }\n    )\n\n  override val audioSpaceParticipantsStore: ReadableStore[String, Participants] = {\n    val store = StratoFetchableStore\n      .DefaultStratoFetchableStore(\n        fetcher = new ParticipantsOnAudioSpaceClientColumn(stratoClient).fetcher\n      ).composeKeyMapping[String](broadcastId =>\n        (broadcastId, AudioSpacesLookupContext(forUserId = None)))\n\n    ObservedCachedReadableStore\n      .from(\n        store = buildStore(store, \"AudioSpaceParticipantsStore\"),\n        ttl = 20.seconds,\n        maxKeys = 200,\n        cacheName = \"AudioSpaceParticipantsStore\",\n        windowSize = 200\n      )\n\n  }\n\n  override val topicSocialProofServiceStore: ReadableStore[\n    TopicSocialProofRequest,\n    TopicSocialProofResponse\n  ] = {\n    StratoFetchableStore.withUnitView[TopicSocialProofRequest, TopicSocialProofResponse](\n      stratoClient,\n      \"topic-signals/tsp/topic-social-proof\")\n  }\n\n  override val spaceDeviceFollowStore: ReadableStore[SourceDestUserRequest, Boolean] = {\n    StratoFetchableStore.withUnitView(\n      fetcher = new SpaceDeviceFollowingClientColumn(stratoClient).fetcher\n    )\n  }\n\n  override val audioSpaceStore: ReadableStore[String, AudioSpace] = {\n    val store = StratoFetchableStore\n      .DefaultStratoFetchableStore(\n        fetcher = new CoreOnAudioSpaceClientColumn(stratoClient).fetcher\n      ).composeKeyMapping[String] { broadcastId =>\n        (broadcastId, AudioSpacesLookupContext(forUserId = None))\n      }\n\n    ObservedCachedReadableStore\n      .from(\n        store = buildStore(store, \"AudioSpaceVisibilityStore\"),\n        ttl = 1.minute,\n        maxKeys = 5000,\n        cacheName = \"AudioSpaceVisibilityStore\",\n        windowSize = 10000L)\n  }\n\n  override val userLanguagesStore = UserLanguagesStore(\n    manhattanMetastoreAppId,\n    manhattanClientMtlsParams,\n    statsReceiver.scope(\"user_languages_store\")\n  )\n\n  val tflockClient: TFlockClient = new TFlockClient(\n    new FlockDB.FinagledClient(\n      readOnlyThriftService(\n        \"tflockClient\",\n        \"/s/tflock/tflock\",\n        statsReceiver,\n        pushserviceThriftClientId,\n        mTLSServiceIdentifier = Some(serviceIdentifier)\n      ),\n      serviceName = \"tflock\",\n      stats = statsReceiver\n    ),\n    defaultPageSize = 1000\n  )\n\n  val rawFlockClient = ThriftMux.client\n    .withClientId(pushserviceThriftClientId)\n    .withMutualTls(serviceIdentifier)\n    .build[FlockDB.MethodPerEndpoint](\"/s/flock/flock\")\n\n  val flockClient: FlockClient = new FlockClient(\n    rawFlockClient,\n    defaultPageSize = 100\n  )\n\n  override val recentFollowsStore: FlockFollowStore = {\n    val dStats = statsReceiver.scope(\"FlockRecentFollowsStore\")\n    FlockFollowStore(flockClient, dStats)\n  }\n\n  def notificationServiceClient: NotificationService$FinagleClient\n\n  def notificationServiceSend(\n    target: Target,\n    request: CreateGenericNotificationRequest\n  ): Future[CreateGenericNotificationResponse]\n\n  def notificationServiceDelete(\n    request: DeleteGenericNotificationRequest\n  ): Future[Unit]\n\n  def notificationServiceDeleteTimeline(\n    request: DeleteCurrentTimelineForUserRequest\n  ): Future[Unit]\n\n  override val notificationServiceSender: ReadableStore[\n    NotificationServiceRequest,\n    CreateGenericNotificationResponse\n  ] = {\n    new NotificationServiceSender(\n      notificationServiceSend,\n      PushParams.EnableWritesToNotificationServiceParam,\n      PushParams.EnableWritesToNotificationServiceForAllEmployeesParam,\n      PushParams.EnableWritesToNotificationServiceForEveryoneParam\n    )\n  }\n\n  val eventRecosServiceClient = {\n    val dest = \"/s/events-recos/events-recos-service\"\n    new EventsRecosService.FinagledClient(\n      readOnlyThriftService(\n        \"EventRecosService\",\n        dest,\n        statsReceiver,\n        pushserviceThriftClientId,\n        mTLSServiceIdentifier = Some(serviceIdentifier)\n      ),\n      clientParam = RichClientParam(serviceName = \"EventRecosService\")\n    )\n  }\n\n  lazy val recommendedTrendsCandidateSource = RecommendedTrendsCandidateSource(\n    TrendsRecommendationStore(eventRecosServiceClient, statsReceiver))\n\n  override val softUserGeoLocationStore: ReadableStore[Long, GeoLocation] =\n    StratoFetchableStore.withUnitView[Long, GeoLocation](fetcher =\n      new FrequentSoftUserLocationClientColumn(stratoClient).fetcher)\n\n  lazy val candidateSourceGenerator = new PushCandidateSourceGenerator(\n    earlybirdCandidateSource,\n    userTweetEntityGraphCandidates,\n    cachedTweetyPieStoreV2,\n    safeCachedTweetyPieStoreV2,\n    userTweetTweetyPieStore,\n    safeUserTweetTweetyPieStore,\n    cachedTweetyPieStoreV2NoVF,\n    edgeStore,\n    interestsWithLookupContextStore,\n    uttEntityHydrationStore,\n    geoDuckV2Store,\n    topTweetsByGeoStore,\n    topTweetsByGeoV2VersionedStore,\n    ruxTweetImpressionsStore,\n    recommendedTrendsCandidateSource,\n    recentTweetsByAuthorsStore,\n    topicSocialProofServiceStore,\n    crMixerStore,\n    contentMixerStore,\n    exploreRankerStore,\n    softUserGeoLocationStore,\n    tripTweetCandidateStore,\n    popGeoLists,\n    idsStore\n  )\n\n  lazy val loCandidateSourceGenerator = new LoggedOutPushCandidateSourceGenerator(\n    tripTweetCandidateStore,\n    geoDuckV2Store,\n    safeCachedTweetyPieStoreV2,\n    cachedTweetyPieStoreV2NoVF,\n    cachedTweetyPieStoreV2,\n    contentMixerStore,\n    softUserGeoLocationStore,\n    topTweetsByGeoStore,\n    topTweetsByGeoV2VersionedStore\n  )\n\n  lazy val rfphStatsRecorder = new RFPHStatsRecorder()\n\n  lazy val rfphRestrictStep = new RFPHRestrictStep()\n\n  lazy val rfphTakeStepUtil = new RFPHTakeStepUtil()(statsReceiver)\n\n  lazy val rfphPrerankFilter = new RFPHPrerankFilter()(statsReceiver)\n\n  lazy val rfphLightRanker = new RFPHLightRanker(lightRanker, statsReceiver)\n\n  lazy val sendHandlerPredicateUtil = new SendHandlerPredicateUtil()(statsReceiver)\n\n  lazy val ntabSender =\n    new NtabSender(\n      notificationServiceSender,\n      nTabHistoryStore,\n      notificationServiceDelete,\n      notificationServiceDeleteTimeline\n    )\n\n  lazy val ibis2Sender = new Ibis2Sender(pushIbisV2Store, tweetTranslationStore, statsReceiver)\n\n  lazy val historyWriter = new HistoryWriter(historyStore, statsReceiver)\n\n  lazy val loggedOutHistoryWriter = new HistoryWriter(loggedOutHistoryStore, statsReceiver)\n\n  lazy val eventBusWriter = new EventBusWriter(pushSendEventBusPublisher, statsReceiver)\n\n  lazy val ntabOnlyChannelSelector = new NtabOnlyChannelSelector\n\n  lazy val notificationSender =\n    new NotificationSender(\n      ibis2Sender,\n      ntabSender,\n      statsReceiver,\n      notificationScribe\n    )\n\n  lazy val candidateNotifier =\n    new CandidateNotifier(\n      notificationSender,\n      casLock = casLock,\n      historyWriter = historyWriter,\n      eventBusWriter = eventBusWriter,\n      ntabOnlyChannelSelector = ntabOnlyChannelSelector\n    )(statsReceiver)\n\n  lazy val loggedOutCandidateNotifier = new CandidateNotifier(\n    notificationSender,\n    casLock = casLock,\n    historyWriter = loggedOutHistoryWriter,\n    eventBusWriter = null,\n    ntabOnlyChannelSelector = ntabOnlyChannelSelector\n  )(statsReceiver)\n\n  lazy val rfphNotifier =\n    new RefreshForPushNotifier(rfphStatsRecorder, candidateNotifier)(statsReceiver)\n\n  lazy val loRfphNotifier =\n    new LoggedOutRefreshForPushNotifier(rfphStatsRecorder, loggedOutCandidateNotifier)(\n      statsReceiver)\n\n  lazy val rfphRanker = {\n    val randomRanker = RandomRanker[Target, PushCandidate]()\n    val subscriptionCreatorRanker =\n      new SubscriptionCreatorRanker(superFollowEligibilityUserStore, statsReceiver)\n    new RFPHRanker(\n      randomRanker,\n      weightedOpenOrNtabClickModelScorer,\n      subscriptionCreatorRanker,\n      userHealthSignalStore,\n      producerMediaRepresentationStore,\n      statsReceiver\n    )\n  }\n\n  lazy val rfphFeatureHydrator = new RFPHFeatureHydrator(featureHydrator)\n  lazy val loggedOutRFPHRanker = new LoggedOutRanker(cachedTweetyPieStoreV2, statsReceiver)\n\n  override val userFeaturesStore: ReadableStore[Long, UserFeatures] = {\n    implicit val valueCodec = new BinaryScalaCodec(UserFeatures)\n    val dataset: Dataset[Long, UserFeatures] =\n      Dataset(\n        \"\",\n        \"user_features_pushservice_apollo\",\n        \"recommendations_user_features_apollo\",\n        Apollo)\n\n    ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = ObservedReadableStore(buildManhattanStore(dataset))(\n        statsReceiver.scope(\"UncachedUserFeaturesStore\")\n      ),\n      cacheClient = pushServiceCacheClient,\n      ttl = 24.hours\n    )(\n      valueInjection = valueCodec,\n      statsReceiver = statsReceiver.scope(\"UserFeaturesStore\"),\n      keyToString = { k: Long => s\"ufts/$k\" }\n    )\n  }\n\n  override def htlScoreStore(userId: Long): ReadableStore[Long, ScoredTweet] = {\n    val fetcher = new TimelineScorerTweetScoresV1ClientColumn(stratoClient).fetcher\n    val htlStore = buildStore(\n      StratoFetchableStore.withView[Long, TimelineScorerScoreView, ScoredTweet](\n        fetcher,\n        TimelineScorerScoreView(Some(userId))\n      ),\n      \"htlScoreStore\"\n    )\n    htlStore\n  }\n\n  override val userTargetingPropertyStore: ReadableStore[Long, UserTargetingProperty] = {\n    val name = \"userTargetingPropertyStore\"\n    val store = StratoFetchableStore\n      .withUnitView(new TargetingPropertyOnUserClientColumn(stratoClient).fetcher)\n    buildStore(store, name)\n  }\n\n  override val timelinesUserSessionStore: ReadableStore[Long, UserSession] = {\n    implicit val valueCodec = new CompactScalaCodec(UserSession)\n    val dataset: Dataset[Long, UserSession] = Dataset[Long, UserSession](\n      \"\",\n      \"frigate_realgraph\",\n      \"real_graph_user_features\",\n      Apollo\n    )\n\n    ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = ObservedReadableStore(buildManhattanStore(dataset))(\n        statsReceiver.scope(\"UncachedTimelinesUserSessionStore\")\n      ),\n      cacheClient = pushServiceCacheClient,\n      ttl = 6.hours\n    )(\n      valueInjection = valueCodec,\n      statsReceiver = statsReceiver.scope(\"timelinesUserSessionStore\"),\n      keyToString = { k: Long => s\"tluss/$k\" }\n    )\n  }\n\n  lazy val recentTweetsFromTflockStore: ReadableStore[Long, Seq[Long]] =\n    ObservedReadableStore(\n      RecentTweetsByAuthorsStore.usingRecentTweetsConfig(\n        tflockClient,\n        RecentTweetsConfig(maxResults = 1, maxAge = 3.days)\n      )\n    )(statsReceiver.scope(\"RecentTweetsFromTflockStore\"))\n\n  lazy val recentTweetsByAuthorsStore: ReadableStore[RecentTweetsQuery, Seq[Seq[Long]]] =\n    ObservedReadableStore(\n      RecentTweetsByAuthorsStore(tflockClient)\n    )(statsReceiver.scope(\"RecentTweetsByAuthorsStore\"))\n\n  val jobConfig = PopGeoInterestProvider\n    .getPopularTweetsJobConfig(\n      InterestDeployConfig(\n        AppId(\"PopularTweetsByInterestProd\"),\n        Cluster.ATLA,\n        Env.Prod,\n        serviceIdentifier,\n        manhattanClientMtlsParams\n      ))\n    .withManhattanAppId(\"frigate_pop_by_geo_tweets\")\n\n  override val topTweetsByGeoStore = TopTweetsStore.withMemCache(\n    jobConfig,\n    pushServiceCacheClient,\n    10.seconds\n  )(statsReceiver)\n\n  override val topTweetsByGeoV2VersionedStore: ReadableStore[String, PopTweetsInPlace] = {\n    StratoFetchableStore.withUnitView[String, PopTweetsInPlace](\n      stratoClient,\n      \"recommendations/popgeo/popGeoTweetsVersioned\")\n  }\n\n  override lazy val pushcapDynamicPredictionStore: ReadableStore[Long, PushcapUserHistory] = {\n    StratoFetchableStore.withUnitView[Long, PushcapUserHistory](\n      stratoClient,\n      \"frigate/magicrecs/pushcapDynamicPrediction.User\")\n  }\n\n  override val tweetAuthorLocationFeatureBuilder =\n    UserLocationFeatureBuilder(Some(\"TweetAuthor\"))\n      .withStats()\n\n  override val tweetAuthorLocationFeatureBuilderById =\n    UserLocationFeatureBuilderById(\n      userCountryStore,\n      tweetAuthorLocationFeatureBuilder\n    ).withStats()\n\n  override val socialContextActionsFeatureBuilder =\n    SocialContextActionsFeatureBuilder().withStats()\n\n  override val tweetContentFeatureBuilder =\n    TweetContentFeatureBuilder(tweetContentFeatureCacheStore).withStats()\n\n  override val tweetAuthorRecentRealGraphFeatureBuilder =\n    RecentRealGraphFeatureBuilder(\n      stratoClient,\n      UserAuthorEntity,\n      TargetUserEntity,\n      TweetAuthorEntity,\n      TweetAuthorRecentRealGraphFeatures(statsReceiver.scope(\"TweetAuthorRecentRealGraphFeatures\"))\n    ).withStats()\n\n  override val socialContextRecentRealGraphFeatureBuilder =\n    SocialContextRecentRealGraphFeatureBuilder(\n      RecentRealGraphFeatureBuilder(\n        stratoClient,\n        TargetUserSocialContextEntity,\n        TargetUserEntity,\n        SocialContextEntity,\n        SocialContextRecentRealGraphFeatures(\n          statsReceiver.scope(\"SocialContextRecentRealGraphFeatures\"))\n      )(statsReceiver\n        .scope(\"SocialContextRecentRealGraphFeatureBuilder\").scope(\"RecentRealGraphFeatureBuilder\"))\n    ).withStats()\n\n  override val tweetSocialProofFeatureBuilder =\n    TweetSocialProofFeatureBuilder(Some(\"TargetUser\")).withStats()\n\n  override val targetUserFullRealGraphFeatureBuilder =\n    TargetFullRealGraphFeatureBuilder(Some(\"TargetUser\")).withStats()\n\n  override val postProcessingFeatureBuilder: PostProcessingFeatureBuilder =\n    PostProcessingFeatureBuilder()\n\n  override val mrOfflineUserCandidateSparseAggregatesFeatureBuilder =\n    MrOfflineUserCandidateSparseAggregatesFeatureBuilder(stratoClient, featureStoreUtil).withStats()\n\n  override val mrOfflineUserAggregatesFeatureBuilder =\n    MrOfflineUserAggregatesFeatureBuilder(stratoClient, featureStoreUtil).withStats()\n\n  override val mrOfflineUserCandidateAggregatesFeatureBuilder =\n    MrOfflineUserCandidateAggregatesFeatureBuilder(stratoClient, featureStoreUtil).withStats()\n\n  override val tweetAnnotationsFeatureBuilder =\n    TweetAnnotationsFeatureBuilder(stratoClient).withStats()\n\n  override val targetUserMediaRepresentationFeatureBuilder =\n    UserMediaRepresentationFeatureBuilder(userMediaRepresentationStore).withStats()\n\n  override val targetLevelFeatureBuilder =\n    TargetLevelFeatureBuilder(featureStoreUtil, targetLevelFeaturesConfig).withStats()\n\n  override val candidateLevelFeatureBuilder =\n    CandidateLevelFeatureBuilder(featureStoreUtil).withStats()\n\n  override lazy val targetFeatureHydrator = RelevanceTargetFeatureHydrator(\n    targetUserFullRealGraphFeatureBuilder,\n    postProcessingFeatureBuilder,\n    targetUserMediaRepresentationFeatureBuilder,\n    targetLevelFeatureBuilder\n  )\n\n  override lazy val featureHydrator =\n    FeatureHydrator(targetFeatureHydrator, candidateFeatureHydrator)\n\n  val pushServiceLightRankerConfig: LightRankerConfig = new LightRankerConfig(\n    pushserviceThriftClientId,\n    serviceIdentifier,\n    statsReceiver.scope(\"lightRanker\"),\n    deepbirdv2PredictionServiceDest,\n    \"DeepbirdV2PredictionService\"\n  )\n  val lightRanker: MagicRecsServeDataRecordLightRanker =\n    pushServiceLightRankerConfig.lightRanker\n\n  override val tweetImpressionStore: ReadableStore[Long, Seq[Long]] = {\n    val name = \"htl_impression_store\"\n    val store = buildStore(\n      HtlTweetImpressionStore.createStoreWithTweetIds(\n        requestTimeout = 6.seconds,\n        label = \"htl_tweet_impressions\",\n        serviceIdentifier = serviceIdentifier,\n        statsReceiver = statsReceiver\n      ),\n      name\n    )\n    val numTweetsReturned =\n      statsReceiver.scope(name).stat(\"num_tweets_returned_per_user\")\n    new TransformedReadableStore(store)((userId: Long, tweetIds: Seq[Long]) => {\n      numTweetsReturned.add(tweetIds.size)\n      Future.value(Some(tweetIds))\n    })\n  }\n\n  val ruxTweetImpressionsStore = new TweetImpressionsStore(stratoClient)\n\n  override val strongTiesStore: ReadableStore[Long, STPResult] = {\n    implicit val valueCodec = new BinaryScalaCodec(STPResult)\n    val strongTieScoringDataset: Dataset[Long, STPResult] =\n      Dataset(\"\", \"frigate_stp\", \"stp_result_rerank\", Athena)\n    buildManhattanStore(strongTieScoringDataset)\n  }\n\n  override lazy val earlybirdFeatureStore = ObservedReadableStore(\n    EarlybirdFeatureStore(\n      clientId = pushserviceThriftClientId.name,\n      earlybirdSearchStore = earlybirdSearchStore\n    )\n  )(statsReceiver.scope(\"EarlybirdFeatureStore\"))\n\n  override lazy val earlybirdFeatureBuilder = EarlybirdFeatureBuilder(earlybirdFeatureStore)\n\n  override lazy val earlybirdSearchStore = {\n    val earlybirdClientName: String = \"earlybird\"\n    val earlybirdSearchStoreName: String = \"EarlybirdSearchStore\"\n\n    val earlybirdClient = new EarlybirdService.FinagledClient(\n      readOnlyThriftService(\n        earlybirdClientName,\n        earlybirdSearchDest,\n        statsReceiver,\n        pushserviceThriftClientId,\n        tries = 1,\n        requestTimeout = 3.seconds,\n        mTLSServiceIdentifier = Some(serviceIdentifier)\n      ),\n      clientParam = RichClientParam(protocolFactory = new TCompactProtocol.Factory)\n    )\n\n    ObservedReadableStore(\n      EarlybirdSearchStore(earlybirdClient)(statsReceiver.scope(earlybirdSearchStoreName))\n    )(statsReceiver.scope(earlybirdSearchStoreName))\n  }\n\n  override lazy val earlybirdCandidateSource: EarlybirdCandidateSource = EarlybirdCandidateSource(\n    clientId = pushserviceThriftClientId.name,\n    earlybirdSearchStore = earlybirdSearchStore\n  )\n\n  override val realGraphScoresTop500InStore: RealGraphScoresTop500InStore = {\n    val stratoRealGraphInStore =\n      StratoFetchableStore\n        .withUnitView[Long, CandidateSeq](\n          stratoClient,\n          \"frigate/magicrecs/fanoutCoi500pRealGraphV2\")\n\n    RealGraphScoresTop500InStore(\n      ObservedMemcachedReadableStore.fromCacheClient(\n        backingStore = stratoRealGraphInStore,\n        cacheClient = entityGraphCacheClient,\n        ttl = 24.hours\n      )(\n        valueInjection = BinaryScalaCodec(CandidateSeq),\n        statsReceiver = statsReceiver.scope(\"CachedRealGraphScoresTop500InStore\"),\n        keyToString = { k: Long => s\"500p_test/$k\" }\n      )\n    )\n  }\n\n  override val tweetEntityGraphStore = {\n    val tweetEntityGraphClient = new UserTweetEntityGraph.FinagledClient(\n      Finagle.readOnlyThriftService(\n        \"user_tweet_entity_graph\",\n        userTweetEntityGraphDest,\n        statsReceiver,\n        pushserviceThriftClientId,\n        requestTimeout = 5.seconds,\n        mTLSServiceIdentifier = Some(serviceIdentifier)\n      )\n    )\n    ObservedReadableStore(\n      RecommendedTweetEntitiesStore(\n        tweetEntityGraphClient,\n        statsReceiver.scope(\"RecommendedTweetEntitiesStore\")\n      )\n    )(statsReceiver.scope(\"RecommendedTweetEntitiesStore\"))\n  }\n\n  override val userUserGraphStore = {\n    val userUserGraphClient = new UserUserGraph.FinagledClient(\n      Finagle.readOnlyThriftService(\n        \"user_user_graph\",\n        userUserGraphDest,\n        statsReceiver,\n        pushserviceThriftClientId,\n        requestTimeout = 5.seconds,\n        mTLSServiceIdentifier = Some(serviceIdentifier)\n      ),\n      clientParam = RichClientParam(serviceName = \"user_user_graph\")\n    )\n    ObservedReadableStore(\n      UserUserGraphStore(userUserGraphClient, statsReceiver.scope(\"UserUserGraphStore\"))\n    )(statsReceiver.scope(\"UserUserGraphStore\"))\n  }\n\n  override val ntabCaretFeedbackStore: ReadableStore[GenericNotificationsFeedbackRequest, Seq[\n    CaretFeedbackDetails\n  ]] = {\n    val client = ManhattanKVClient(\n      \"pushservice_ntab_caret_feedback_omega\",\n      Omega.wilyName,\n      manhattanClientMtlsParams\n    )\n    val endpoint = ManhattanKVEndpointBuilder(client)\n      .defaultGuarantee(Guarantee.SoftDcReadMyWrites)\n      .defaultMaxTimeout(3.seconds)\n      .maxRetryCount(2)\n      .statsReceiver(statsReceiver)\n      .build()\n\n    val feedbackSignalManhattanClient =\n      FeedbackSignalManhattanClient(endpoint, statsReceiver.scope(\"FeedbackSignalManhattanClient\"))\n    NtabCaretFeedbackStore(feedbackSignalManhattanClient)\n  }\n\n  override val genericFeedbackStore: ReadableStore[FeedbackRequest, Seq[\n    FeedbackPromptValue\n  ]] = {\n    FeedbackStore(\n      GenericFeedbackStoreBuilder.build(\n        manhattanKVClientAppId = \"frigate_pushservice_ntabfeedback_prompt\",\n        environment = NotifEnvironment.apply(serviceIdentifier.environment),\n        svcIdentifier = serviceIdentifier,\n        statsReceiver = statsReceiver\n      ))\n  }\n\n  override val genericNotificationFeedbackStore: GenericFeedbackStore = {\n\n    GenericFeedbackStoreBuilder.build(\n      manhattanKVClientAppId = \"frigate_pushservice_ntabfeedback_prompt\",\n      environment = NotifEnvironment.apply(serviceIdentifier.environment),\n      svcIdentifier = serviceIdentifier,\n      statsReceiver = statsReceiver\n    )\n  }\n\n  override val earlybirdSearchDest = \"/s/earlybird-root-superroot/root-superroot\"\n\n  // low latency as compared to default `semanticCoreMetadataClient`\n  private val lowLatencySemanticCoreMetadataClient: MetadataService.MethodPerEndpoint =\n    new MetadataService.FinagledClient(\n      Finagle.readOnlyThriftService(\n        name = \"semantic_core_metadata_service\",\n        dest = \"/s/escherbird/metadataservice\",\n        statsReceiver = statsReceiver,\n        thriftClientId = pushserviceThriftClientId,\n        tries = 2, // total number of tries. number of retries = tries - 1\n        requestTimeout = 2.seconds,\n        mTLSServiceIdentifier = Some(serviceIdentifier)\n      )\n    )\n\n  private val semanticCoreMetadataStitchClient = new MetadataStitchClient(\n    lowLatencySemanticCoreMetadataClient\n  )\n\n  override val semanticCoreMegadataStore: ReadableStore[SemanticEntityForQuery, EntityMegadata] = {\n    val name = \"semantic_core_megadata_store_cached\"\n    val store = MetaDataReadableStore.getMegadataReadableStore(\n      metadataStitchClient = semanticCoreMetadataStitchClient,\n      typedMetadataDomains = Some(Set(Domains.EventsEntityService))\n    )\n    ObservedCachedReadableStore\n      .from(\n        store = ObservedReadableStore(store)(\n          statsReceiver\n            .scope(\"store\")\n            .scope(\"semantic_core_megadata_store\")\n        ),\n        ttl = 1.hour,\n        maxKeys = 1000,\n        cacheName = \"semantic_core_megadata_cache\",\n        windowSize = 10000L\n      )(statsReceiver.scope(\"store\", name))\n  }\n\n  override val basketballGameScoreStore: ReadableStore[QualifiedId, BasketballGameLiveUpdate] = {\n    StratoFetchableStore.withUnitView[QualifiedId, BasketballGameLiveUpdate](\n      stratoClient,\n      \"semanticCore/basketballGameScore.Entity\")\n  }\n\n  override val baseballGameScoreStore: ReadableStore[QualifiedId, BaseballGameLiveUpdate] = {\n    StratoFetchableStore.withUnitView[QualifiedId, BaseballGameLiveUpdate](\n      stratoClient,\n      \"semanticCore/baseballGameScore.Entity\")\n  }\n\n  override val cricketMatchScoreStore: ReadableStore[QualifiedId, CricketMatchLiveUpdate] = {\n    StratoFetchableStore.withUnitView[QualifiedId, CricketMatchLiveUpdate](\n      stratoClient,\n      \"semanticCore/cricketMatchScore.Entity\")\n  }\n\n  override val soccerMatchScoreStore: ReadableStore[QualifiedId, SoccerMatchLiveUpdate] = {\n    ObservedCachedReadableStore\n      .from(\n        store = StratoFetchableStore.withUnitView[QualifiedId, SoccerMatchLiveUpdate](\n          stratoClient,\n          \"semanticCore/soccerMatchScore.Entity\"),\n        ttl = 10.seconds,\n        maxKeys = 100,\n        cacheName = \"SoccerMatchCachedStore\",\n        windowSize = 100L\n      )(statsReceiver.scope(\"SoccerMatchCachedStore\"))\n\n  }\n\n  override val nflGameScoreStore: ReadableStore[QualifiedId, NflFootballGameLiveUpdate] = {\n    ObservedCachedReadableStore\n      .from(\n        store = StratoFetchableStore.withUnitView[QualifiedId, NflFootballGameLiveUpdate](\n          stratoClient,\n          \"semanticCore/nflFootballGameScore.Entity\"),\n        ttl = 10.seconds,\n        maxKeys = 100,\n        cacheName = \"NFLMatchCachedStore\",\n        windowSize = 100L\n      )(statsReceiver.scope(\"NFLMatchCachedStore\"))\n\n  }\n\n  override val userHealthSignalStore: ReadableStore[Long, UserHealthSignalResponse] = {\n    val userHealthSignalFetcher =\n      stratoClient.fetcher[Long, Seq[UserHealthSignal], UserHealthSignalResponse](\n        \"hss/user_signals/api/healthSignals.User\"\n      )\n\n    val store = buildStore(\n      StratoFetchableStore.withView[Long, Seq[UserHealthSignal], UserHealthSignalResponse](\n        userHealthSignalFetcher,\n        Seq(\n          AgathaRecentAbuseStrikeDouble,\n          AgathaCalibratedNsfwDouble,\n          AgathaCseDouble,\n          NsfwTextUserScoreDouble,\n          NsfwConsumerScoreDouble)),\n      \"UserHealthSignalFetcher\"\n    )\n    if (!inMemCacheOff) {\n      ObservedCachedReadableStore\n        .from(\n          store = ObservedReadableStore(store)(\n            statsReceiver.scope(\"store\").scope(\"user_health_model_score_store\")),\n          ttl = 12.hours,\n          maxKeys = 16777215,\n          cacheName = \"user_health_model_score_store_cache\",\n          windowSize = 10000L\n        )(statsReceiver.scope(\"store\", \"user_health_model_score_store_cached\"))\n    } else {\n      store\n    }\n  }\n\n  override val tweetHealthScoreStore: ReadableStore[TweetScoringRequest, TweetScoringResponse] = {\n    val tweetHealthScoreFetcher =\n      stratoClient.fetcher[TweetScoringRequest, Unit, TweetScoringResponse](\n        \"abuse/detection/tweetHealthModelScore\"\n      )\n\n    val store = buildStore(\n      StratoFetchableStore.withUnitView(tweetHealthScoreFetcher),\n      \"TweetHealthScoreFetcher\"\n    )\n\n    ObservedCachedReadableStore\n      .from(\n        store = ObservedReadableStore(store)(\n          statsReceiver.scope(\"store\").scope(\"tweet_health_model_score_store\")),\n        ttl = 30.minutes,\n        maxKeys = 1000,\n        cacheName = \"tweet_health_model_score_store_cache\",\n        windowSize = 10000L\n      )(statsReceiver.scope(\"store\", \"tweet_health_model_score_store_cached\"))\n  }\n\n  override val appPermissionStore: ReadableStore[(Long, (String, String)), AppPermission] = {\n    val store = StratoFetchableStore\n      .withUnitView[(Long, (String, String)), AppPermission](\n        stratoClient,\n        \"clients/permissionsState\")\n    ObservedCachedReadableStore.from[(Long, (String, String)), AppPermission](\n      buildStore(store, \"mr_app_permission_store\"),\n      ttl = 30.minutes,\n      maxKeys = 1000,\n      cacheName = \"mr_app_permission_store_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"mr_app_permission_store_cached\"))\n  }\n\n  def pushSendEventStreamName: String\n\n  override val pushSendEventBusPublisher = EventBusPublisherBuilder()\n    .clientId(\"frigate_pushservice\")\n    .streamName(pushSendEventStreamName)\n    .thriftStruct(NotificationScribe)\n    .statsReceiver(statsReceiver.scope(\"push_send_eventbus\"))\n    .build()\n\n  override lazy val candidateFeatureHydrator: CandidateFeatureHydrator =\n    CandidateFeatureHydrator(\n      socialContextActionsFeatureBuilder = Some(socialContextActionsFeatureBuilder),\n      tweetSocialProofFeatureBuilder = Some(tweetSocialProofFeatureBuilder),\n      earlybirdFeatureBuilder = Some(earlybirdFeatureBuilder),\n      tweetContentFeatureBuilder = Some(tweetContentFeatureBuilder),\n      tweetAuthorRecentRealGraphFeatureBuilder = Some(tweetAuthorRecentRealGraphFeatureBuilder),\n      socialContextRecentRealGraphFeatureBuilder = Some(socialContextRecentRealGraphFeatureBuilder),\n      tweetAnnotationsFeatureBuilder = Some(tweetAnnotationsFeatureBuilder),\n      mrOfflineUserCandidateSparseAggregatesFeatureBuilder =\n        Some(mrOfflineUserCandidateSparseAggregatesFeatureBuilder),\n      candidateLevelFeatureBuilder = Some(candidateLevelFeatureBuilder)\n    )(statsReceiver.scope(\"push_feature_hydrator\"))\n\n  private val candidateCopyCross =\n    new CandidateCopyExpansion(statsReceiver.scope(\"refresh_handler/cross\"))\n\n  override lazy val candidateHydrator: PushCandidateHydrator =\n    PushCandidateHydrator(\n      this.socialGraphServiceProcessStore,\n      safeUserStore,\n      listAPIStore,\n      candidateCopyCross)(\n      statsReceiver.scope(\"push_candidate_hydrator\"),\n      weightedOpenOrNtabClickModelScorer)\n\n  override lazy val sendHandlerCandidateHydrator: SendHandlerPushCandidateHydrator =\n    SendHandlerPushCandidateHydrator(\n      lexServiceStore,\n      fanoutMetadataStore,\n      semanticCoreMegadataStore,\n      safeUserStore,\n      simClusterToEntityStore,\n      audioSpaceStore,\n      interestsWithLookupContextStore,\n      uttEntityHydrationStore,\n      superFollowCreatorTweetCountStore\n    )(\n      statsReceiver.scope(\"push_candidate_hydrator\"),\n      weightedOpenOrNtabClickModelScorer\n    )\n\n  def mrRequestScriberNode: String\n  def loggedOutMrRequestScriberNode: String\n\n  override lazy val configParamsBuilder: ConfigParamsBuilder = ConfigParamsBuilder(\n    config = overridesConfig,\n    featureContextBuilder = FeatureContextBuilder(featureSwitches),\n    statsReceiver = statsReceiver\n  )\n\n  def buildStore[K, V](store: ReadableStore[K, V], name: String): ReadableStore[K, V] = {\n    ObservedReadableStore(store)(statsReceiver.scope(\"store\").scope(name))\n  }\n\n  def buildManhattanStore[K, V](dataset: Dataset[K, V]): ReadableStore[K, V] = {\n    val manhattanKVClientParams = ManhattanKVClientMtlsParams(\n      serviceIdentifier = serviceIdentifier,\n      opportunisticTls = OpportunisticTls.Required\n    )\n    ManhattanStore\n      .fromDatasetWithMtls[K, V](\n        dataset,\n        mtlsParams = manhattanKVClientParams,\n        statsReceiver = statsReceiver.scope(dataset.datasetName))\n  }\n\n  def buildCachedTweetyPieStore(\n    getTweetOptions: Option[GetTweetOptions],\n    keyPrefix: String\n  ): ReadableStore[Long, TweetyPieResult] = {\n    def discardAdditionalMediaInfo(tweetypieResult: TweetyPieResult) = {\n      val updatedMedia = tweetypieResult.tweet.media.map { mediaSeq =>\n        mediaSeq.map { media => media.copy(additionalMetadata = None, sizes = Nil.toSet) }\n      }\n      val updatedTweet = tweetypieResult.tweet.copy(media = updatedMedia)\n      tweetypieResult.copy(tweet = updatedTweet)\n    }\n\n    val tweetypieStoreWithoutAdditionalMediaInfo = TweetyPieStore(\n      tweetyPieClient,\n      getTweetOptions,\n      transformTweetypieResult = discardAdditionalMediaInfo\n    )(statsReceiver.scope(\"tweetypie_without_additional_media_info\"))\n\n    ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = tweetypieStoreWithoutAdditionalMediaInfo,\n      cacheClient = pushServiceCoreSvcsCacheClient,\n      ttl = 12.hours\n    )(\n      valueInjection = TweetyPieResultInjection,\n      statsReceiver = statsReceiver.scope(\"TweetyPieStore\"),\n      keyToString = { k: Long => s\"$keyPrefix/$k\" }\n    )\n  }\n\n  override def init(): Future[Unit] =\n    ClientRegistry.expAllRegisteredClientsResolved().map { clients =>\n      log.info(\"Done resolving clients: \" + clients.mkString(\"[\", \", \", \"]\"))\n    }\n\n  val InlineActionsMhColumn =\n    \"frigate/magicrecs/inlineActionsMh\"\n\n  override val inlineActionHistoryStore: ReadableStore[Long, Seq[(Long, String)]] =\n    StratoScannableStore\n      .withUnitView[(Long, Slice[Long]), (Long, Long), String](stratoClient, InlineActionsMhColumn)\n      .composeKeyMapping[Long] { userId =>\n        (userId, Slice[Long](from = None, to = None, limit = None))\n      }.mapValues { response =>\n        response.map {\n          case (key, value) => (key._2, value)\n        }\n      }\n\n  override val tripTweetCandidateStore: ReadableStore[TripDomain, TripTweets] = {\n    StratoFetchableStore\n      .withUnitView[TripDomain, TripTweets](\n        new TripTweetsAirflowProdClientColumn(stratoClient).fetcher)\n  }\n\n  override val softUserFollowingStore: ReadableStore[User, Seq[Long]] = new SoftUserFollowingStore(\n    stratoClient)\n\n  override val superFollowEligibilityUserStore: ReadableStore[Long, Boolean] = {\n    StratoFetchableStore.withUnitView[Long, Boolean](\n      stratoClient,\n      \"audiencerewards/audienceRewardsService/getSuperFollowEligibility.User\")\n  }\n\n  override val superFollowCreatorTweetCountStore: ReadableStore[UserId, Int] = {\n    ObservedCachedReadableStore\n      .from(\n        store = StratoFetchableStore\n          .withUnitView[UserId, Int](new CreatorSubscriptionNumTweetsColumn(stratoClient).fetcher),\n        ttl = 5.minutes,\n        maxKeys = 1000,\n        cacheName = \"SuperFollowCreatorTweetCountStore\",\n        windowSize = 10000L\n      )(statsReceiver.scope(\"SuperFollowCreatorTweetCountStore\"))\n\n  }\n\n  override val hasSuperFollowingRelationshipStore: ReadableStore[\n    HasSuperFollowingRelationshipRequest,\n    Boolean\n  ] = {\n    StratoFetchableStore.withUnitView[HasSuperFollowingRelationshipRequest, Boolean](\n      stratoClient,\n      \"audiencerewards/superFollows/hasSuperFollowingRelationshipV2\")\n  }\n\n  override val superFollowApplicationStatusStore: ReadableStore[\n    (Long, SellerTrack),\n    SellerApplicationState\n  ] = {\n    StratoFetchableStore.withUnitView[(Long, SellerTrack), SellerApplicationState](\n      stratoClient,\n      \"periscope/eligibility/applicationStatus\")\n  }\n\n  def historyStoreMemcacheDest: String\n\n  override lazy val recentHistoryCacheClient = {\n    RecentHistoryCacheClient.build(historyStoreMemcacheDest, serviceIdentifier, statsReceiver)\n  }\n\n  override val openAppUserStore: ReadableStore[Long, Boolean] = {\n    buildStore(OpenAppUserStore(stratoClient), \"OpenAppUserStore\")\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/config/ExperimentsWithStats.scala",
    "content": "package com.twitter.frigate.pushservice.config\n\nimport com.twitter.frigate.common.util.Experiments\n\nobject ExperimentsWithStats {\n\n  /**\n   * Add an experiment here to collect detailed pushservice stats.\n   *\n   * ! Important !\n   * Keep this set small and remove experiments when you don't need the stats anymore.\n   */\n  final val PushExperiments: Set[String] = Set(\n    Experiments.MRAndroidInlineActionHoldback.exptName,\n  )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/config/ProdConfig.scala",
    "content": "package com.twitter.frigate.pushservice.config\n\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.bijection.Base64String\nimport com.twitter.bijection.Injection\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.decider.Decider\nimport com.twitter.featureswitches.v2.FeatureSwitches\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.thrift.RichClientParam\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.frigate.common.config.RateLimiterGenerator\nimport com.twitter.frigate.common.filter.DynamicRequestMeterFilter\nimport com.twitter.frigate.common.history.ManhattanHistoryStore\nimport com.twitter.frigate.common.history.InvalidatingAfterWritesPushServiceHistoryStore\nimport com.twitter.frigate.common.history.ManhattanKVHistoryStore\nimport com.twitter.frigate.common.history.PushServiceHistoryStore\nimport com.twitter.frigate.common.history.SimplePushServiceHistoryStore\nimport com.twitter.frigate.common.util._\nimport com.twitter.frigate.data_pipeline.features_common.FeatureStoreUtil\nimport com.twitter.frigate.data_pipeline.features_common.TargetLevelFeaturesConfig\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.DeciderKey\nimport com.twitter.frigate.pushservice.params.PushQPSLimitConstants\nimport com.twitter.frigate.pushservice.params.PushServiceTunableKeys\nimport com.twitter.frigate.pushservice.params.ShardParams\nimport com.twitter.frigate.pushservice.store.PushIbis2Store\nimport com.twitter.frigate.pushservice.thriftscala.PushRequestScribe\nimport com.twitter.frigate.scribe.thriftscala.NotificationScribe\nimport com.twitter.ibis2.service.thriftscala.Ibis2Service\nimport com.twitter.logging.Logger\nimport com.twitter.notificationservice.api.thriftscala.DeleteCurrentTimelineForUserRequest\nimport com.twitter.notificationservice.api.thriftscala.NotificationApi\nimport com.twitter.notificationservice.api.thriftscala.NotificationApi$FinagleClient\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationRequest\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationResponse\nimport com.twitter.notificationservice.thriftscala.DeleteGenericNotificationRequest\nimport com.twitter.notificationservice.thriftscala.NotificationService\nimport com.twitter.notificationservice.thriftscala.NotificationService$FinagleClient\nimport com.twitter.servo.decider.DeciderGateBuilder\nimport com.twitter.util.tunable.TunableMap\nimport com.twitter.util.Future\nimport com.twitter.util.Timer\n\ncase class ProdConfig(\n  override val isServiceLocal: Boolean,\n  override val localConfigRepoPath: String,\n  override val inMemCacheOff: Boolean,\n  override val decider: Decider,\n  override val abDecider: LoggingABDecider,\n  override val featureSwitches: FeatureSwitches,\n  override val shardParams: ShardParams,\n  override val serviceIdentifier: ServiceIdentifier,\n  override val tunableMap: TunableMap,\n)(\n  implicit val statsReceiver: StatsReceiver)\n    extends {\n  // Due to trait initialization logic in Scala, any abstract members declared in Config or\n  // DeployConfig should be declared in this block. Otherwise the abstract member might initialize to\n  // null if invoked before object creation finishing.\n\n  val log = Logger(\"ProdConfig\")\n\n  // Deciders\n  val isPushserviceCanaryDeepbirdv2CanaryClusterEnabled = decider\n    .feature(DeciderKey.enablePushserviceDeepbirdv2CanaryClusterDeciderKey.toString).isAvailable\n\n  // Client ids\n  val notifierThriftClientId = ClientId(\"frigate-notifier.prod\")\n  val loggedOutNotifierThriftClientId = ClientId(\"frigate-logged-out-notifier.prod\")\n  val pushserviceThriftClientId: ClientId = ClientId(\"frigate-pushservice.prod\")\n\n  // Dests\n  val frigateHistoryCacheDest = \"/s/cache/frigate_history\"\n  val memcacheCASDest = \"/s/cache/magic_recs_cas:twemcaches\"\n  val historyStoreMemcacheDest =\n    \"/srv#/prod/local/cache/magic_recs_history:twemcaches\"\n\n  val deepbirdv2PredictionServiceDest =\n    if (serviceIdentifier.service.equals(\"frigate-pushservice-canary\") &&\n      isPushserviceCanaryDeepbirdv2CanaryClusterEnabled)\n      \"/s/frigate/deepbirdv2-magicrecs-canary\"\n    else \"/s/frigate/deepbirdv2-magicrecs\"\n\n  override val fanoutMetadataColumn = \"frigate/magicfanout/prod/mh/fanoutMetadata\"\n\n  override val timer: Timer = DefaultTimer\n  override val featureStoreUtil = FeatureStoreUtil.withParams(Some(serviceIdentifier))\n  override val targetLevelFeaturesConfig = TargetLevelFeaturesConfig()\n  val pushServiceMHCacheDest = \"/s/cache/pushservice_mh\"\n\n  val pushServiceCoreSvcsCacheDest = \"/srv#/prod/local/cache/pushservice_core_svcs\"\n\n  val userTweetEntityGraphDest = \"/s/cassowary/user_tweet_entity_graph\"\n  val userUserGraphDest = \"/s/cassowary/user_user_graph\"\n  val lexServiceDest = \"/s/live-video/timeline-thrift\"\n  val entityGraphCacheDest = \"/s/cache/pushservice_entity_graph\"\n\n  override val pushIbisV2Store = {\n    val service = Finagle.readOnlyThriftService(\n      \"ibis-v2-service\",\n      \"/s/ibis2/ibis2\",\n      statsReceiver,\n      notifierThriftClientId,\n      requestTimeout = 3.seconds,\n      tries = 3,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    )\n\n    // according to ibis team, it is safe to retry on timeout, write & channel closed exceptions.\n    val pushIbisClient = new Ibis2Service.FinagledClient(\n      new DynamicRequestMeterFilter(\n        tunableMap(PushServiceTunableKeys.IbisQpsLimitTunableKey),\n        RateLimiterGenerator.asTuple(_, shardParams.numShards, 20),\n        PushQPSLimitConstants.IbisOrNTabQPSForRFPH\n      )(timer).andThen(service),\n      RichClientParam(serviceName = \"ibis-v2-service\")\n    )\n\n    PushIbis2Store(pushIbisClient)\n  }\n\n  val notificationServiceClient: NotificationService$FinagleClient = {\n    val service = Finagle.readWriteThriftService(\n      \"notificationservice\",\n      \"/s/notificationservice/notificationservice\",\n      statsReceiver,\n      pushserviceThriftClientId,\n      requestTimeout = 10.seconds,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    )\n\n    new NotificationService.FinagledClient(\n      new DynamicRequestMeterFilter(\n        tunableMap(PushServiceTunableKeys.NtabQpsLimitTunableKey),\n        RateLimiterGenerator.asTuple(_, shardParams.numShards, 20),\n        PushQPSLimitConstants.IbisOrNTabQPSForRFPH)(timer).andThen(service),\n      RichClientParam(serviceName = \"notificationservice\")\n    )\n  }\n\n  val notificationServiceApiClient: NotificationApi$FinagleClient = {\n    val service = Finagle.readWriteThriftService(\n      \"notificationservice-api\",\n      \"/s/notificationservice/notificationservice-api:thrift\",\n      statsReceiver,\n      pushserviceThriftClientId,\n      requestTimeout = 10.seconds,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    )\n\n    new NotificationApi.FinagledClient(\n      new DynamicRequestMeterFilter(\n        tunableMap(PushServiceTunableKeys.NtabQpsLimitTunableKey),\n        RateLimiterGenerator.asTuple(_, shardParams.numShards, 20),\n        PushQPSLimitConstants.IbisOrNTabQPSForRFPH)(timer).andThen(service),\n      RichClientParam(serviceName = \"notificationservice-api\")\n    )\n  }\n\n  val mrRequestScriberNode = \"mr_request_scribe\"\n  val loggedOutMrRequestScriberNode = \"lo_mr_request_scribe\"\n\n  override val pushSendEventStreamName = \"frigate_pushservice_send_event_prod\"\n} with DeployConfig {\n  // Scribe\n  private val notificationScribeLog = Logger(\"notification_scribe\")\n  private val notificationScribeInjection: Injection[NotificationScribe, String] = BinaryScalaCodec(\n    NotificationScribe\n  ) andThen Injection.connect[Array[Byte], Base64String, String]\n\n  override def notificationScribe(data: NotificationScribe): Unit = {\n    val logEntry: String = notificationScribeInjection(data)\n    notificationScribeLog.info(logEntry)\n  }\n\n  // History Store - Invalidates cached history after writes\n  override val historyStore = new InvalidatingAfterWritesPushServiceHistoryStore(\n    ManhattanHistoryStore(notificationHistoryStore, statsReceiver),\n    recentHistoryCacheClient,\n    new DeciderGateBuilder(decider)\n      .idGate(DeciderKey.enableInvalidatingCachedHistoryStoreAfterWrites)\n  )\n\n  override val emailHistoryStore: PushServiceHistoryStore = {\n    statsReceiver.scope(\"frigate_email_history\").counter(\"request\").incr()\n    new SimplePushServiceHistoryStore(emailNotificationHistoryStore)\n  }\n\n  override val loggedOutHistoryStore =\n    new InvalidatingAfterWritesPushServiceHistoryStore(\n      ManhattanKVHistoryStore(\n        manhattanKVLoggedOutHistoryStoreEndpoint,\n        \"frigate_notification_logged_out_history\"),\n      recentHistoryCacheClient,\n      new DeciderGateBuilder(decider)\n        .idGate(DeciderKey.enableInvalidatingCachedLoggedOutHistoryStoreAfterWrites)\n    )\n\n  private val requestScribeLog = Logger(\"request_scribe\")\n  private val requestScribeInjection: Injection[PushRequestScribe, String] = BinaryScalaCodec(\n    PushRequestScribe\n  ) andThen Injection.connect[Array[Byte], Base64String, String]\n\n  override def requestScribe(data: PushRequestScribe): Unit = {\n    val logEntry: String = requestScribeInjection(data)\n    requestScribeLog.info(logEntry)\n  }\n\n  // generic notification server\n  override def notificationServiceSend(\n    target: Target,\n    request: CreateGenericNotificationRequest\n  ): Future[CreateGenericNotificationResponse] =\n    notificationServiceClient.createGenericNotification(request)\n\n  // generic notification server\n  override def notificationServiceDelete(\n    request: DeleteGenericNotificationRequest\n  ): Future[Unit] = notificationServiceClient.deleteGenericNotification(request)\n\n  // NTab-api\n  override def notificationServiceDeleteTimeline(\n    request: DeleteCurrentTimelineForUserRequest\n  ): Future[Unit] = notificationServiceApiClient.deleteCurrentTimelineForUser(request)\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/config/StagingConfig.scala",
    "content": "package com.twitter.frigate.pushservice.config\n\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.decider.Decider\nimport com.twitter.featureswitches.v2.FeatureSwitches\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finagle.thrift.RichClientParam\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.frigate.common.config.RateLimiterGenerator\nimport com.twitter.frigate.common.filter.DynamicRequestMeterFilter\nimport com.twitter.frigate.common.history.InvalidatingAfterWritesPushServiceHistoryStore\nimport com.twitter.frigate.common.history.ManhattanHistoryStore\nimport com.twitter.frigate.common.history.ManhattanKVHistoryStore\nimport com.twitter.frigate.common.history.ReadOnlyHistoryStore\nimport com.twitter.frigate.common.history.PushServiceHistoryStore\nimport com.twitter.frigate.common.history.SimplePushServiceHistoryStore\nimport com.twitter.frigate.common.util.Finagle\nimport com.twitter.frigate.data_pipeline.features_common.FeatureStoreUtil\nimport com.twitter.frigate.data_pipeline.features_common.TargetLevelFeaturesConfig\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.DeciderKey\nimport com.twitter.frigate.pushservice.params.PushQPSLimitConstants\nimport com.twitter.frigate.pushservice.params.PushServiceTunableKeys\nimport com.twitter.frigate.pushservice.params.ShardParams\nimport com.twitter.frigate.pushservice.store._\nimport com.twitter.frigate.pushservice.thriftscala.PushRequestScribe\nimport com.twitter.frigate.scribe.thriftscala.NotificationScribe\nimport com.twitter.ibis2.service.thriftscala.Ibis2Service\nimport com.twitter.logging.Logger\nimport com.twitter.notificationservice.api.thriftscala.DeleteCurrentTimelineForUserRequest\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationRequest\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationResponse\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationResponseType\nimport com.twitter.notificationservice.thriftscala.DeleteGenericNotificationRequest\nimport com.twitter.notificationservice.thriftscala.NotificationService\nimport com.twitter.notificationservice.thriftscala.NotificationService$FinagleClient\nimport com.twitter.servo.decider.DeciderGateBuilder\nimport com.twitter.util.tunable.TunableMap\nimport com.twitter.util.Future\nimport com.twitter.util.Timer\n\ncase class StagingConfig(\n  override val isServiceLocal: Boolean,\n  override val localConfigRepoPath: String,\n  override val inMemCacheOff: Boolean,\n  override val decider: Decider,\n  override val abDecider: LoggingABDecider,\n  override val featureSwitches: FeatureSwitches,\n  override val shardParams: ShardParams,\n  override val serviceIdentifier: ServiceIdentifier,\n  override val tunableMap: TunableMap,\n)(\n  implicit val statsReceiver: StatsReceiver)\n    extends {\n  // Due to trait initialization logic in Scala, any abstract members declared in Config or\n  // DeployConfig should be declared in this block. Otherwise the abstract member might initialize to\n  // null if invoked before object creation finishing.\n\n  val log = Logger(\"StagingConfig\")\n\n  // Client ids\n  val notifierThriftClientId = ClientId(\"frigate-notifier.dev\")\n  val loggedOutNotifierThriftClientId = ClientId(\"frigate-logged-out-notifier.dev\")\n  val pushserviceThriftClientId: ClientId = ClientId(\"frigate-pushservice.staging\")\n\n  override val fanoutMetadataColumn = \"frigate/magicfanout/staging/mh/fanoutMetadata\"\n\n  // dest\n  val frigateHistoryCacheDest = \"/srv#/test/local/cache/twemcache_frigate_history\"\n  val memcacheCASDest = \"/srv#/test/local/cache/twemcache_magic_recs_cas_dev:twemcaches\"\n  val pushServiceMHCacheDest = \"/srv#/test/local/cache/twemcache_pushservice_test\"\n  val entityGraphCacheDest = \"/srv#/test/local/cache/twemcache_pushservice_test\"\n  val pushServiceCoreSvcsCacheDest = \"/srv#/test/local/cache/twemcache_pushservice_core_svcs_test\"\n  val historyStoreMemcacheDest = \"/srv#/test/local/cache/twemcache_eventstream_test:twemcaches\"\n  val userTweetEntityGraphDest = \"/cluster/local/cassowary/staging/user_tweet_entity_graph\"\n  val userUserGraphDest = \"/cluster/local/cassowary/staging/user_user_graph\"\n  val lexServiceDest = \"/srv#/staging/local/live-video/timeline-thrift\"\n  val deepbirdv2PredictionServiceDest = \"/cluster/local/frigate/staging/deepbirdv2-magicrecs\"\n\n  override val featureStoreUtil = FeatureStoreUtil.withParams(Some(serviceIdentifier))\n  override val targetLevelFeaturesConfig = TargetLevelFeaturesConfig()\n  val mrRequestScriberNode = \"validation_mr_request_scribe\"\n  val loggedOutMrRequestScriberNode = \"lo_mr_request_scribe\"\n\n  override val timer: Timer = DefaultTimer\n\n  override val pushSendEventStreamName = \"frigate_pushservice_send_event_staging\"\n\n  override val pushIbisV2Store = {\n    val service = Finagle.readWriteThriftService(\n      \"ibis-v2-service\",\n      \"/s/ibis2/ibis2\",\n      statsReceiver,\n      notifierThriftClientId,\n      requestTimeout = 6.seconds,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    )\n\n    val pushIbisClient = new Ibis2Service.FinagledClient(\n      new DynamicRequestMeterFilter(\n        tunableMap(PushServiceTunableKeys.IbisQpsLimitTunableKey),\n        RateLimiterGenerator.asTuple(_, shardParams.numShards, 20),\n        PushQPSLimitConstants.IbisOrNTabQPSForRFPH\n      )(timer).andThen(service),\n      RichClientParam(serviceName = \"ibis-v2-service\")\n    )\n\n    StagingIbis2Store(PushIbis2Store(pushIbisClient))\n  }\n\n  val notificationServiceClient: NotificationService$FinagleClient = {\n    val service = Finagle.readWriteThriftService(\n      \"notificationservice\",\n      \"/s/notificationservice/notificationservice\",\n      statsReceiver,\n      pushserviceThriftClientId,\n      requestTimeout = 10.seconds,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    )\n\n    new NotificationService.FinagledClient(\n      new DynamicRequestMeterFilter(\n        tunableMap(PushServiceTunableKeys.NtabQpsLimitTunableKey),\n        RateLimiterGenerator.asTuple(_, shardParams.numShards, 20),\n        PushQPSLimitConstants.IbisOrNTabQPSForRFPH)(timer).andThen(service),\n      RichClientParam(serviceName = \"notificationservice\")\n    )\n  }\n} with DeployConfig {\n\n  // Scribe\n  private val notificationScribeLog = Logger(\"StagingNotificationScribe\")\n\n  override def notificationScribe(data: NotificationScribe): Unit = {\n    notificationScribeLog.info(data.toString)\n  }\n  private val requestScribeLog = Logger(\"StagingRequestScribe\")\n\n  override def requestScribe(data: PushRequestScribe): Unit = {\n    requestScribeLog.info(data.toString)\n  }\n\n  // history store\n  override val historyStore = new InvalidatingAfterWritesPushServiceHistoryStore(\n    ReadOnlyHistoryStore(\n      ManhattanHistoryStore(notificationHistoryStore, statsReceiver)\n    ),\n    recentHistoryCacheClient,\n    new DeciderGateBuilder(decider)\n      .idGate(DeciderKey.enableInvalidatingCachedHistoryStoreAfterWrites)\n  )\n\n  override val emailHistoryStore: PushServiceHistoryStore = new SimplePushServiceHistoryStore(\n    emailNotificationHistoryStore)\n\n  // history store\n  override val loggedOutHistoryStore =\n    new InvalidatingAfterWritesPushServiceHistoryStore(\n      ReadOnlyHistoryStore(\n        ManhattanKVHistoryStore(\n          manhattanKVLoggedOutHistoryStoreEndpoint,\n          \"frigate_notification_logged_out_history\")),\n      recentHistoryCacheClient,\n      new DeciderGateBuilder(decider)\n        .idGate(DeciderKey.enableInvalidatingCachedLoggedOutHistoryStoreAfterWrites)\n    )\n\n  override def notificationServiceSend(\n    target: Target,\n    request: CreateGenericNotificationRequest\n  ): Future[CreateGenericNotificationResponse] =\n    target.isTeamMember.flatMap { isTeamMember =>\n      if (isTeamMember) {\n        notificationServiceClient.createGenericNotification(request)\n      } else {\n        log.info(s\"Mock creating generic notification $request for user: ${target.targetId}\")\n        Future.value(\n          CreateGenericNotificationResponse(CreateGenericNotificationResponseType.Success)\n        )\n      }\n    }\n\n  override def notificationServiceDelete(\n    request: DeleteGenericNotificationRequest\n  ): Future[Unit] = Future.Unit\n\n  override def notificationServiceDeleteTimeline(\n    request: DeleteCurrentTimelineForUserRequest\n  ): Future[Unit] = Future.Unit\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/config/mlconfig/DeepbirdV2ModelConfig.scala",
    "content": "package com.twitter.frigate.pushservice.config.mlconfig\n\nimport com.twitter.cortex.deepbird.thriftjava.DeepbirdPredictionService\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.ml.prediction.DeepbirdPredictionEngineServiceStore\nimport com.twitter.nrel.heavyranker.PushDBv2PredictionServiceStore\n\nobject DeepbirdV2ModelConfig {\n  def buildPredictionServiceScoreStore(\n    predictionServiceClient: DeepbirdPredictionService.ServiceToClient,\n    serviceName: String\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): PushDBv2PredictionServiceStore = {\n\n    val stats = statsReceiver.scope(serviceName)\n    val serviceStats = statsReceiver.scope(\"dbv2PredictionServiceStore\")\n\n    new PushDBv2PredictionServiceStore(\n      DeepbirdPredictionEngineServiceStore(predictionServiceClient, batchSize = Some(32))(stats)\n    )(serviceStats)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/controller/PushServiceController.scala",
    "content": "package com.twitter.frigate.pushservice.controller\n\nimport com.google.inject.Inject\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finatra.thrift.Controller\nimport com.twitter.frigate.pushservice.exception.DisplayLocationNotSupportedException\nimport com.twitter.frigate.pushservice.refresh_handler.RefreshForPushHandler\nimport com.twitter.frigate.pushservice.send_handler.SendHandler\nimport com.twitter.frigate.pushservice.refresh_handler.LoggedOutRefreshForPushHandler\nimport com.twitter.frigate.pushservice.thriftscala.PushService.Loggedout\nimport com.twitter.frigate.pushservice.thriftscala.PushService.Refresh\nimport com.twitter.frigate.pushservice.thriftscala.PushService.Send\nimport com.twitter.frigate.pushservice.{thriftscala => t}\nimport com.twitter.frigate.thriftscala.NotificationDisplayLocation\nimport com.twitter.util.logging.Logging\nimport com.twitter.util.Future\n\nclass PushServiceController @Inject() (\n  sendHandler: SendHandler,\n  refreshForPushHandler: RefreshForPushHandler,\n  loggedOutRefreshForPushHandler: LoggedOutRefreshForPushHandler,\n  statsReceiver: StatsReceiver)\n    extends Controller(t.PushService)\n    with Logging {\n\n  private val stats: StatsReceiver = statsReceiver.scope(s\"${this.getClass.getSimpleName}\")\n  private val failureCount = stats.counter(\"failures\")\n  private val failureStatsScope = stats.scope(\"failures\")\n  private val uncaughtErrorCount = failureStatsScope.counter(\"uncaught\")\n  private val uncaughtErrorScope = failureStatsScope.scope(\"uncaught\")\n  private val clientIdScope = stats.scope(\"client_id\")\n\n  handle(t.PushService.Send) { request: Send.Args =>\n    send(request)\n  }\n\n  handle(t.PushService.Refresh) { args: Refresh.Args =>\n    refresh(args)\n  }\n\n  handle(t.PushService.Loggedout) { request: Loggedout.Args =>\n    loggedOutRefresh(request)\n  }\n\n  private def loggedOutRefresh(\n    request: t.PushService.Loggedout.Args\n  ): Future[t.PushService.Loggedout.SuccessType] = {\n    val fut = request.request.notificationDisplayLocation match {\n      case NotificationDisplayLocation.PushToMobileDevice =>\n        loggedOutRefreshForPushHandler.refreshAndSend(request.request)\n      case _ =>\n        Future.exception(\n          new DisplayLocationNotSupportedException(\n            \"Specified notification display location is not supported\"))\n    }\n    fut.onFailure { ex =>\n      logger.error(\n        s\"Failure in push service for logged out refresh request: $request - ${ex.getMessage} - ${ex.getStackTrace\n          .mkString(\", \\n\\t\")}\",\n        ex)\n      failureCount.incr()\n      uncaughtErrorCount.incr()\n      uncaughtErrorScope.counter(ex.getClass.getCanonicalName).incr()\n    }\n  }\n\n  private def refresh(\n    request: t.PushService.Refresh.Args\n  ): Future[t.PushService.Refresh.SuccessType] = {\n\n    val fut = request.request.notificationDisplayLocation match {\n      case NotificationDisplayLocation.PushToMobileDevice =>\n        val clientId: String =\n          ClientId.current\n            .flatMap { cid => Option(cid.name) }\n            .getOrElse(\"none\")\n        clientIdScope.counter(clientId).incr()\n        refreshForPushHandler.refreshAndSend(request.request)\n      case _ =>\n        Future.exception(\n          new DisplayLocationNotSupportedException(\n            \"Specified notification display location is not supported\"))\n    }\n    fut.onFailure { ex =>\n      logger.error(\n        s\"Failure in push service for refresh request: $request - ${ex.getMessage} - ${ex.getStackTrace\n          .mkString(\", \\n\\t\")}\",\n        ex\n      )\n\n      failureCount.incr()\n      uncaughtErrorCount.incr()\n      uncaughtErrorScope.counter(ex.getClass.getCanonicalName).incr()\n    }\n\n  }\n\n  private def send(\n    request: t.PushService.Send.Args\n  ): Future[t.PushService.Send.SuccessType] = {\n    sendHandler(request.request).onFailure { ex =>\n      logger.error(\n        s\"Failure in push service for send request: $request - ${ex.getMessage} - ${ex.getStackTrace\n          .mkString(\", \\n\\t\")}\",\n        ex\n      )\n\n      failureCount.incr()\n      uncaughtErrorCount.incr()\n      uncaughtErrorScope.counter(ex.getClass.getCanonicalName).incr()\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/exception/DisplayLocationNotSupportedException.scala",
    "content": "package com.twitter.frigate.pushservice.exception\n\nimport scala.util.control.NoStackTrace\n\n/**\n * Throw exception if DisplayLocation is not supported\n *\n * @param message Exception message\n */\nclass DisplayLocationNotSupportedException(private val message: String)\n    extends Exception(message)\n    with NoStackTrace\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/exception/InvalidSportDomainException.scala",
    "content": "package com.twitter.frigate.pushservice.exception\n\nimport scala.util.control.NoStackTrace\n\n/**\n * Throw exception if the sport domain is not supported by MagicFanoutSports\n *\n * @param message Exception message\n */\nclass InvalidSportDomainException(private val message: String)\n    extends Exception(message)\n    with NoStackTrace\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/exception/TweetNTabRequestHydratorException.scala",
    "content": "package com.twitter.frigate.pushservice.exception\n\nimport scala.util.control.NoStackTrace\n\nclass TweetNTabRequestHydratorException(private val message: String)\n    extends Exception(message)\n    with NoStackTrace\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/exception/UnsupportedCrtException.scala",
    "content": "package com.twitter.frigate.pushservice.exception\n\nimport scala.util.control.NoStackTrace\n\n/**\n * Exception for CRT not expected in the scope\n * @param message Exception message to log the UnsupportedCrt\n */\nclass UnsupportedCrtException(private val message: String)\n    extends Exception(message)\n    with NoStackTrace\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/exception/UttEntityNotFoundException.scala",
    "content": "package com.twitter.frigate.pushservice.exception\n\nimport scala.util.control.NoStackTrace\n\n/**\n * Throw exception if UttEntity is not found where it might be a required data field\n *\n * @param message Exception message\n */\nclass UttEntityNotFoundException(private val message: String)\n    extends Exception(message)\n    with NoStackTrace\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/ml/HealthFeatureGetter.scala",
    "content": "package com.twitter.frigate.pushservice.ml\n\nimport com.twitter.abuse.detection.scoring.thriftscala.{Model => TweetHealthModel}\nimport com.twitter.abuse.detection.scoring.thriftscala.TweetScoringRequest\nimport com.twitter.abuse.detection.scoring.thriftscala.TweetScoringResponse\nimport com.twitter.frigate.common.base.FeatureMap\nimport com.twitter.frigate.common.base.TweetAuthor\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.predicate.HealthPredicates.userHealthSignalValueToDouble\nimport com.twitter.frigate.pushservice.util.CandidateHydrationUtil\nimport com.twitter.frigate.pushservice.util.CandidateUtil\nimport com.twitter.frigate.pushservice.util.MediaAnnotationsUtil\nimport com.twitter.frigate.thriftscala.UserMediaRepresentation\nimport com.twitter.hss.api.thriftscala.SignalValue\nimport com.twitter.hss.api.thriftscala.UserHealthSignal\nimport com.twitter.hss.api.thriftscala.UserHealthSignal.AgathaCalibratedNsfwDouble\nimport com.twitter.hss.api.thriftscala.UserHealthSignal.NsfwTextUserScoreDouble\nimport com.twitter.hss.api.thriftscala.UserHealthSignalResponse\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\nobject HealthFeatureGetter {\n\n  def getFeatures(\n    pushCandidate: PushCandidate,\n    producerMediaRepresentationStore: ReadableStore[Long, UserMediaRepresentation],\n    userHealthScoreStore: ReadableStore[Long, UserHealthSignalResponse],\n    tweetHealthScoreStoreOpt: Option[ReadableStore[TweetScoringRequest, TweetScoringResponse]] =\n      None\n  ): Future[FeatureMap] = {\n\n    pushCandidate match {\n      case cand: PushCandidate with TweetCandidate with TweetAuthor with TweetAuthorDetails =>\n        val pMediaNsfwRequest =\n          TweetScoringRequest(cand.tweetId, TweetHealthModel.ExperimentalHealthModelScore4)\n        val pTweetTextNsfwRequest =\n          TweetScoringRequest(cand.tweetId, TweetHealthModel.ExperimentalHealthModelScore1)\n\n        cand.authorId match {\n          case Some(authorId) =>\n            Future\n              .join(\n                userHealthScoreStore.get(authorId),\n                producerMediaRepresentationStore.get(authorId),\n                tweetHealthScoreStoreOpt.map(_.get(pMediaNsfwRequest)).getOrElse(Future.None),\n                tweetHealthScoreStoreOpt.map(_.get(pTweetTextNsfwRequest)).getOrElse(Future.None),\n                cand.tweetAuthor\n              ).map {\n                case (\n                      healthSignalsResponseOpt,\n                      producerMuOpt,\n                      pMediaNsfwOpt,\n                      pTweetTextNsfwOpt,\n                      tweetAuthorOpt) =>\n                  val healthSignalScoreMap = healthSignalsResponseOpt\n                    .map(_.signalValues).getOrElse(Map.empty[UserHealthSignal, SignalValue])\n                  val agathaNSFWScore = userHealthSignalValueToDouble(\n                    healthSignalScoreMap\n                      .getOrElse(AgathaCalibratedNsfwDouble, SignalValue.DoubleValue(0.5)))\n                  val userTextNSFWScore = userHealthSignalValueToDouble(\n                    healthSignalScoreMap\n                      .getOrElse(NsfwTextUserScoreDouble, SignalValue.DoubleValue(0.15)))\n                  val pMediaNsfwScore = pMediaNsfwOpt.map(_.score).getOrElse(0.0)\n                  val pTweetTextNsfwScore = pTweetTextNsfwOpt.map(_.score).getOrElse(0.0)\n\n                  val mediaRepresentationMap =\n                    producerMuOpt.map(_.mediaRepresentation).getOrElse(Map.empty[String, Double])\n                  val sumScore: Double = mediaRepresentationMap.values.sum\n                  val nudityRate =\n                    if (sumScore > 0)\n                      mediaRepresentationMap.getOrElse(\n                        MediaAnnotationsUtil.nudityCategoryId,\n                        0.0) / sumScore\n                    else 0.0\n                  val beautyRate =\n                    if (sumScore > 0)\n                      mediaRepresentationMap.getOrElse(\n                        MediaAnnotationsUtil.beautyCategoryId,\n                        0.0) / sumScore\n                    else 0.0\n                  val singlePersonRate =\n                    if (sumScore > 0)\n                      mediaRepresentationMap.getOrElse(\n                        MediaAnnotationsUtil.singlePersonCategoryId,\n                        0.0) / sumScore\n                    else 0.0\n                  val dislikeCt = cand.numericFeatures.getOrElse(\n                    \"tweet.magic_recs_tweet_real_time_aggregates_v2.pair.v2.magicrecs.realtime.is_ntab_disliked.any_feature.Duration.Top.count\",\n                    0.0)\n                  val sentCt = cand.numericFeatures.getOrElse(\n                    \"tweet.magic_recs_tweet_real_time_aggregates_v2.pair.v2.magicrecs.realtime.is_sent.any_feature.Duration.Top.count\",\n                    0.0)\n                  val dislikeRate = if (sentCt > 0) dislikeCt / sentCt else 0.0\n\n                  val authorDislikeCt = cand.numericFeatures.getOrElse(\n                    \"tweet_author_aggregate.pair.label.ntab.isDisliked.any_feature.28.days.count\",\n                    0.0)\n                  val authorReportCt = cand.numericFeatures.getOrElse(\n                    \"tweet_author_aggregate.pair.label.reportTweetDone.any_feature.28.days.count\",\n                    0.0)\n                  val authorSentCt = cand.numericFeatures\n                    .getOrElse(\n                      \"tweet_author_aggregate.pair.any_label.any_feature.28.days.count\",\n                      0.0)\n                  val authorDislikeRate =\n                    if (authorSentCt > 0) authorDislikeCt / authorSentCt else 0.0\n                  val authorReportRate =\n                    if (authorSentCt > 0) authorReportCt / authorSentCt else 0.0\n\n                  val (isNsfwAccount, authorAccountAge) = tweetAuthorOpt match {\n                    case Some(tweetAuthor) =>\n                      (\n                        CandidateHydrationUtil.isNsfwAccount(\n                          tweetAuthor,\n                          cand.target.params(PushFeatureSwitchParams.NsfwTokensParam)),\n                        (Time.now - Time.fromMilliseconds(tweetAuthor.createdAtMsec)).inHours\n                      )\n                    case _ => (false, 0)\n                  }\n\n                  val tweetSemanticCoreIds = cand.sparseBinaryFeatures\n                    .getOrElse(PushConstants.TweetSemanticCoreIdFeature, Set.empty[String])\n\n                  val continuousFeatures = Map[String, Double](\n                    \"agathaNsfwScore\" -> agathaNSFWScore,\n                    \"textNsfwScore\" -> userTextNSFWScore,\n                    \"pMediaNsfwScore\" -> pMediaNsfwScore,\n                    \"pTweetTextNsfwScore\" -> pTweetTextNsfwScore,\n                    \"nudityRate\" -> nudityRate,\n                    \"beautyRate\" -> beautyRate,\n                    \"singlePersonRate\" -> singlePersonRate,\n                    \"numSources\" -> CandidateUtil.getTagsCRCount(cand),\n                    \"favCount\" -> cand.numericFeatures\n                      .getOrElse(\"tweet.core.tweet_counts.favorite_count\", 0.0),\n                    \"activeFollowers\" -> cand.numericFeatures\n                      .getOrElse(\"RecTweetAuthor.User.ActiveFollowers\", 0.0),\n                    \"favorsRcvd28Days\" -> cand.numericFeatures\n                      .getOrElse(\"RecTweetAuthor.User.FavorsRcvd28Days\", 0.0),\n                    \"tweets28Days\" -> cand.numericFeatures\n                      .getOrElse(\"RecTweetAuthor.User.Tweets28Days\", 0.0),\n                    \"dislikeCount\" -> dislikeCt,\n                    \"dislikeRate\" -> dislikeRate,\n                    \"sentCount\" -> sentCt,\n                    \"authorDislikeCount\" -> authorDislikeCt,\n                    \"authorDislikeRate\" -> authorDislikeRate,\n                    \"authorReportCount\" -> authorReportCt,\n                    \"authorReportRate\" -> authorReportRate,\n                    \"authorSentCount\" -> authorSentCt,\n                    \"authorAgeInHour\" -> authorAccountAge.toDouble\n                  )\n\n                  val booleanFeatures = Map[String, Boolean](\n                    \"isSimclusterBased\" -> RecTypes.simclusterBasedTweets\n                      .contains(cand.commonRecType),\n                    \"isTopicTweet\" -> RecTypes.isTopicTweetType(cand.commonRecType),\n                    \"isHashSpace\" -> RecTypes.tagspaceTypes.contains(cand.commonRecType),\n                    \"isFRS\" -> RecTypes.frsTypes.contains(cand.commonRecType),\n                    \"isModelingBased\" -> RecTypes.mrModelingBasedTypes.contains(cand.commonRecType),\n                    \"isGeoPop\" -> RecTypes.GeoPopTweetTypes.contains(cand.commonRecType),\n                    \"hasPhoto\" -> cand.booleanFeatures\n                      .getOrElse(\"RecTweet.TweetyPieResult.HasPhoto\", false),\n                    \"hasVideo\" -> cand.booleanFeatures\n                      .getOrElse(\"RecTweet.TweetyPieResult.HasVideo\", false),\n                    \"hasUrl\" -> cand.booleanFeatures\n                      .getOrElse(\"RecTweet.TweetyPieResult.HasUrl\", false),\n                    \"isMrTwistly\" -> CandidateUtil.isMrTwistlyCandidate(cand),\n                    \"abuseStrikeTop2Percent\" -> tweetSemanticCoreIds.contains(\n                      PushConstants.AbuseStrike_Top2Percent_Id),\n                    \"abuseStrikeTop1Percent\" -> tweetSemanticCoreIds.contains(\n                      PushConstants.AbuseStrike_Top1Percent_Id),\n                    \"abuseStrikeTop05Percent\" -> tweetSemanticCoreIds.contains(\n                      PushConstants.AbuseStrike_Top05Percent_Id),\n                    \"abuseStrikeTop025Percent\" -> tweetSemanticCoreIds.contains(\n                      PushConstants.AbuseStrike_Top025Percent_Id),\n                    \"allSpamReportsPerFavTop1Percent\" -> tweetSemanticCoreIds.contains(\n                      PushConstants.AllSpamReportsPerFav_Top1Percent_Id),\n                    \"reportsPerFavTop1Percent\" -> tweetSemanticCoreIds.contains(\n                      PushConstants.ReportsPerFav_Top1Percent_Id),\n                    \"reportsPerFavTop2Percent\" -> tweetSemanticCoreIds.contains(\n                      PushConstants.ReportsPerFav_Top2Percent_Id),\n                    \"isNudity\" -> tweetSemanticCoreIds.contains(\n                      PushConstants.MediaUnderstanding_Nudity_Id),\n                    \"beautyStyleFashion\" -> tweetSemanticCoreIds.contains(\n                      PushConstants.MediaUnderstanding_Beauty_Id),\n                    \"singlePerson\" -> tweetSemanticCoreIds.contains(\n                      PushConstants.MediaUnderstanding_SinglePerson_Id),\n                    \"pornList\" -> tweetSemanticCoreIds.contains(PushConstants.PornList_Id),\n                    \"pornographyAndNsfwContent\" -> tweetSemanticCoreIds.contains(\n                      PushConstants.PornographyAndNsfwContent_Id),\n                    \"sexLife\" -> tweetSemanticCoreIds.contains(PushConstants.SexLife_Id),\n                    \"sexLifeOrSexualOrientation\" -> tweetSemanticCoreIds.contains(\n                      PushConstants.SexLifeOrSexualOrientation_Id),\n                    \"profanity\" -> tweetSemanticCoreIds.contains(PushConstants.ProfanityFilter_Id),\n                    \"isVerified\" -> cand.booleanFeatures\n                      .getOrElse(\"RecTweetAuthor.User.IsVerified\", false),\n                    \"hasNsfwToken\" -> isNsfwAccount\n                  )\n\n                  val stringFeatures = Map[String, String](\n                    \"tweetLanguage\" -> cand.categoricalFeatures\n                      .getOrElse(\"tweet.core.tweet_text.language\", \"\")\n                  )\n\n                  FeatureMap(\n                    booleanFeatures = booleanFeatures,\n                    numericFeatures = continuousFeatures,\n                    categoricalFeatures = stringFeatures)\n              }\n          case _ => Future.value(FeatureMap())\n        }\n      case _ => Future.value(FeatureMap())\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/ml/HydrationContextBuilder.scala",
    "content": "package com.twitter.frigate.pushservice.ml\n\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.ml.feature.TweetSocialProofKey\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.predicate.quality_model_predicate.PDauCohortUtil\nimport com.twitter.nrel.hydration.base.FeatureInput\nimport com.twitter.nrel.hydration.push.HydrationContext\nimport com.twitter.nrel.hydration.frigate.{FeatureInputs => FI}\nimport com.twitter.util.Future\n\nobject HydrationContextBuilder {\n\n  private def getRecUserInputs(\n    pushCandidate: PushCandidate\n  ): Set[FI.RecUser] = {\n    pushCandidate match {\n      case userCandidate: UserCandidate =>\n        Set(FI.RecUser(userCandidate.userId))\n      case _ => Set.empty\n    }\n  }\n\n  private def getRecTweetInputs(\n    pushCandidate: PushCandidate\n  ): Set[FI.RecTweet] =\n    pushCandidate match {\n      case tweetCandidateWithAuthor: TweetCandidate with TweetAuthor with TweetAuthorDetails =>\n        val authorIdOpt = tweetCandidateWithAuthor.authorId\n        Set(FI.RecTweet(tweetCandidateWithAuthor.tweetId, authorIdOpt))\n      case _ => Set.empty\n    }\n\n  private def getMediaInputs(\n    pushCandidate: PushCandidate\n  ): Set[FI.Media] =\n    pushCandidate match {\n      case tweetCandidateWithMedia: TweetCandidate with TweetDetails =>\n        tweetCandidateWithMedia.mediaKeys\n          .map { mk =>\n            Set(FI.Media(mk))\n          }.getOrElse(Set.empty)\n      case _ => Set.empty\n    }\n\n  private def getEventInputs(\n    pushCandidate: PushCandidate\n  ): Set[FI.Event] = pushCandidate match {\n    case mrEventCandidate: EventCandidate =>\n      Set(FI.Event(mrEventCandidate.eventId))\n    case mfEventCandidate: MagicFanoutEventCandidate =>\n      Set(FI.Event(mfEventCandidate.eventId))\n    case _ => Set.empty\n  }\n\n  private def getTopicInputs(\n    pushCandidate: PushCandidate\n  ): Set[FI.Topic] =\n    pushCandidate match {\n      case mrTopicCandidate: TopicCandidate =>\n        mrTopicCandidate.semanticCoreEntityId match {\n          case Some(topicId) => Set(FI.Topic(topicId))\n          case _ => Set.empty\n        }\n      case _ => Set.empty\n    }\n\n  private def getTweetSocialProofKey(\n    pushCandidate: PushCandidate\n  ): Future[Set[FI.SocialProofKey]] = {\n    pushCandidate match {\n      case candidate: TweetCandidate with SocialContextActions =>\n        val target = pushCandidate.target\n        target.seedsWithWeight.map { seedsWithWeightOpt =>\n          Set(\n            FI.SocialProofKey(\n              TweetSocialProofKey(\n                seedsWithWeightOpt.getOrElse(Map.empty),\n                candidate.socialContextAllTypeActions\n              ))\n          )\n        }\n      case _ => Future.value(Set.empty)\n    }\n  }\n\n  private def getSocialContextInputs(\n    pushCandidate: PushCandidate\n  ): Future[Set[FeatureInput]] =\n    pushCandidate match {\n      case candidateWithSC: Candidate with SocialContextActions =>\n        val tweetSocialProofKeyFut = getTweetSocialProofKey(pushCandidate)\n        tweetSocialProofKeyFut.map { tweetSocialProofKeyOpt =>\n          val socialContextUsers = FI.SocialContextUsers(candidateWithSC.socialContextUserIds.toSet)\n          val socialContextActions =\n            FI.SocialContextActions(candidateWithSC.socialContextAllTypeActions)\n          val socialProofKeyOpt = tweetSocialProofKeyOpt\n          Set(Set(socialContextUsers), Set(socialContextActions), socialProofKeyOpt).flatten\n        }\n      case _ => Future.value(Set.empty)\n    }\n\n  private def getPushStringGroupInputs(\n    pushCandidate: PushCandidate\n  ): Set[FI.PushStringGroup] =\n    Set(\n      FI.PushStringGroup(\n        pushCandidate.getPushCopy.flatMap(_.pushStringGroup).map(_.toString).getOrElse(\"\")\n      ))\n\n  private def getCRTInputs(\n    pushCandidate: PushCandidate\n  ): Set[FI.CommonRecommendationType] =\n    Set(FI.CommonRecommendationType(pushCandidate.commonRecType))\n\n  private def getFrigateNotification(\n    pushCandidate: PushCandidate\n  ): Set[FI.CandidateFrigateNotification] =\n    Set(FI.CandidateFrigateNotification(pushCandidate.frigateNotification))\n\n  private def getCopyId(\n    pushCandidate: PushCandidate\n  ): Set[FI.CopyId] =\n    Set(FI.CopyId(pushCandidate.pushCopyId, pushCandidate.ntabCopyId))\n\n  def build(candidate: PushCandidate): Future[HydrationContext] = {\n    val socialContextInputsFut = getSocialContextInputs(candidate)\n    socialContextInputsFut.map { socialContextInputs =>\n      val featureInputs: Set[FeatureInput] =\n        socialContextInputs ++\n          getRecUserInputs(candidate) ++\n          getRecTweetInputs(candidate) ++\n          getEventInputs(candidate) ++\n          getTopicInputs(candidate) ++\n          getCRTInputs(candidate) ++\n          getPushStringGroupInputs(candidate) ++\n          getMediaInputs(candidate) ++\n          getFrigateNotification(candidate) ++\n          getCopyId(candidate)\n\n      HydrationContext(\n        candidate.target.targetId,\n        featureInputs\n      )\n    }\n  }\n\n  def build(target: Target): Future[HydrationContext] = {\n    val realGraphFeaturesFut = target.realGraphFeatures\n    for {\n      realGraphFeaturesOpt <- realGraphFeaturesFut\n      dauProb <- PDauCohortUtil.getDauProb(target)\n      mrUserStateOpt <- target.targetMrUserState\n      historyInputOpt <-\n        if (target.params(PushFeatureSwitchParams.EnableHydratingOnlineMRHistoryFeatures)) {\n          target.onlineLabeledPushRecs.map { mrHistoryValueOpt =>\n            mrHistoryValueOpt.map(FI.MrHistory)\n          }\n        } else Future.None\n    } yield {\n      val realGraphFeaturesInputOpt = realGraphFeaturesOpt.map { realGraphFeatures =>\n        FI.TargetRealGraphFeatures(realGraphFeatures)\n      }\n      val dauProbInput = FI.DauProb(dauProb)\n      val mrUserStateInput = FI.MrUserState(mrUserStateOpt.map(_.name).getOrElse(\"unknown\"))\n      HydrationContext(\n        target.targetId,\n        Seq(\n          realGraphFeaturesInputOpt,\n          historyInputOpt,\n          Some(dauProbInput),\n          Some(mrUserStateInput)\n        ).flatten.toSet\n      )\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/ml/PushMLModelScorer.scala",
    "content": "package com.twitter.frigate.pushservice.ml\n\nimport com.twitter.cortex.deepbird.thriftjava.ModelSelector\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.FeatureMap\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushMLModel\nimport com.twitter.frigate.pushservice.params.PushModelName\nimport com.twitter.frigate.pushservice.params.WeightedOpenOrNtabClickModel\nimport com.twitter.nrel.heavyranker.PushCandidateHydrationContextWithModel\nimport com.twitter.nrel.heavyranker.PushPredictionServiceStore\nimport com.twitter.nrel.heavyranker.TargetFeatureMapWithModel\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.util.Future\n\n/**\n * PushMLModelScorer scores the Candidates and populates their ML scores\n *\n * @param pushMLModel Enum to specify which model to use for scoring the Candidates\n * @param modelToPredictionServiceStoreMap Supports all other prediction services. Specifies model ID -> dbv2 ReadableStore\n * @param defaultDBv2PredictionServiceStore: Supports models that are not specified in the previous maps (which will be directly configured in the config repo)\n * @param scoringStats StatsReceiver for scoping stats\n */\nclass PushMLModelScorer(\n  pushMLModel: PushMLModel.Value,\n  modelToPredictionServiceStoreMap: Map[\n    WeightedOpenOrNtabClickModel.ModelNameType,\n    PushPredictionServiceStore\n  ],\n  defaultDBv2PredictionServiceStore: PushPredictionServiceStore,\n  scoringStats: StatsReceiver) {\n\n  val queriesOutsideTheModelMaps: StatsReceiver =\n    scoringStats.scope(\"queries_outside_the_model_maps\")\n  val totalQueriesOutsideTheModelMaps: Counter =\n    queriesOutsideTheModelMaps.counter(\"total\")\n\n  private def scoreByBatchPredictionForModelFromMultiModelService(\n    predictionServiceStore: PushPredictionServiceStore,\n    modelVersion: WeightedOpenOrNtabClickModel.ModelNameType,\n    candidatesDetails: Seq[CandidateDetails[PushCandidate]],\n    useCommonFeatures: Boolean,\n    overridePushMLModel: PushMLModel.Value\n  ): Seq[CandidateDetails[PushCandidate]] = {\n    val modelName =\n      PushModelName(overridePushMLModel, modelVersion).toString\n    val modelSelector = new ModelSelector()\n    modelSelector.setId(modelName)\n\n    val candidateHydrationWithFeaturesMap = candidatesDetails.map { candidatesDetail =>\n      (\n        candidatesDetail.candidate.candidateHydrationContext,\n        candidatesDetail.candidate.candidateFeatureMap())\n    }\n    if (candidatesDetails.nonEmpty) {\n      val candidatesWithScore = predictionServiceStore.getBatchPredictionsForModel(\n        candidatesDetails.head.candidate.target.targetHydrationContext,\n        candidatesDetails.head.candidate.target.featureMap,\n        candidateHydrationWithFeaturesMap,\n        Some(modelSelector),\n        useCommonFeatures\n      )\n      candidatesDetails.zip(candidatesWithScore).foreach {\n        case (candidateDetail, (_, scoreOptFut)) =>\n          candidateDetail.candidate.populateQualityModelScore(\n            overridePushMLModel,\n            modelVersion,\n            scoreOptFut\n          )\n      }\n    }\n\n    candidatesDetails\n  }\n\n  private def scoreByBatchPrediction(\n    modelVersion: WeightedOpenOrNtabClickModel.ModelNameType,\n    candidatesDetails: Seq[CandidateDetails[PushCandidate]],\n    useCommonFeaturesForDBv2Service: Boolean,\n    overridePushMLModel: PushMLModel.Value\n  ): Seq[CandidateDetails[PushCandidate]] = {\n    if (modelToPredictionServiceStoreMap.contains(modelVersion)) {\n      scoreByBatchPredictionForModelFromMultiModelService(\n        modelToPredictionServiceStoreMap(modelVersion),\n        modelVersion,\n        candidatesDetails,\n        useCommonFeaturesForDBv2Service,\n        overridePushMLModel\n      )\n    } else {\n      totalQueriesOutsideTheModelMaps.incr()\n      queriesOutsideTheModelMaps.counter(modelVersion).incr()\n      scoreByBatchPredictionForModelFromMultiModelService(\n        defaultDBv2PredictionServiceStore,\n        modelVersion,\n        candidatesDetails,\n        useCommonFeaturesForDBv2Service,\n        overridePushMLModel\n      )\n    }\n  }\n\n  def scoreByBatchPredictionForModelVersion(\n    target: Target,\n    candidatesDetails: Seq[CandidateDetails[PushCandidate]],\n    modelVersionParam: FSParam[WeightedOpenOrNtabClickModel.ModelNameType],\n    useCommonFeaturesForDBv2Service: Boolean = true,\n    overridePushMLModelOpt: Option[PushMLModel.Value] = None\n  ): Seq[CandidateDetails[PushCandidate]] = {\n    scoreByBatchPrediction(\n      target.params(modelVersionParam),\n      candidatesDetails,\n      useCommonFeaturesForDBv2Service,\n      overridePushMLModelOpt.getOrElse(pushMLModel)\n    )\n  }\n\n  def singlePredicationForModelVersion(\n    modelVersion: String,\n    candidate: PushCandidate,\n    overridePushMLModelOpt: Option[PushMLModel.Value] = None\n  ): Future[Option[Double]] = {\n    val modelSelector = new ModelSelector()\n    modelSelector.setId(\n      PushModelName(overridePushMLModelOpt.getOrElse(pushMLModel), modelVersion).toString\n    )\n    if (modelToPredictionServiceStoreMap.contains(modelVersion)) {\n      modelToPredictionServiceStoreMap(modelVersion).get(\n        PushCandidateHydrationContextWithModel(\n          candidate.target.targetHydrationContext,\n          candidate.target.featureMap,\n          candidate.candidateHydrationContext,\n          candidate.candidateFeatureMap(),\n          Some(modelSelector)\n        )\n      )\n    } else {\n      totalQueriesOutsideTheModelMaps.incr()\n      queriesOutsideTheModelMaps.counter(modelVersion).incr()\n      defaultDBv2PredictionServiceStore.get(\n        PushCandidateHydrationContextWithModel(\n          candidate.target.targetHydrationContext,\n          candidate.target.featureMap,\n          candidate.candidateHydrationContext,\n          candidate.candidateFeatureMap(),\n          Some(modelSelector)\n        )\n      )\n    }\n  }\n\n  def singlePredictionForTargetLevel(\n    modelVersion: String,\n    targetId: Long,\n    featureMap: Future[FeatureMap]\n  ): Future[Option[Double]] = {\n    val modelSelector = new ModelSelector()\n    modelSelector.setId(\n      PushModelName(pushMLModel, modelVersion).toString\n    )\n    defaultDBv2PredictionServiceStore.getForTargetLevel(\n      TargetFeatureMapWithModel(targetId, featureMap, Some(modelSelector))\n    )\n  }\n\n  def getScoreHistogramCounters(\n    stats: StatsReceiver,\n    scopeName: String,\n    histogramBinSize: Double\n  ): IndexedSeq[Counter] = {\n    val histogramScopedStatsReceiver = stats.scope(scopeName)\n    val numBins = math.ceil(1.0 / histogramBinSize).toInt\n\n    (0 to numBins) map { k =>\n      if (k == 0)\n        histogramScopedStatsReceiver.counter(\"candidates_with_scores_zero\")\n      else {\n        val counterName = \"candidates_with_scores_from_%s_to_%s\".format(\n          \"%.2f\".format(histogramBinSize * (k - 1)).replace(\".\", \"\"),\n          \"%.2f\".format(math.min(1.0, histogramBinSize * k)).replace(\".\", \"\"))\n        histogramScopedStatsReceiver.counter(counterName)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/DiscoverTwitter.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.DiscoverTwitterCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.DiscoverTwitterPushIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.DiscoverTwitterNtabRequestHydrator\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.predicate.PredicatesForCandidate\nimport com.twitter.frigate.pushservice.take.predicates.BasicRFPHPredicates\nimport com.twitter.frigate.pushservice.take.predicates.OutOfNetworkTweetPredicates\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.hermit.predicate.NamedPredicate\n\nclass DiscoverTwitterPushCandidate(\n  candidate: RawCandidate with DiscoverTwitterCandidate,\n  copyIds: CopyIds,\n)(\n  implicit val statsScoped: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with DiscoverTwitterCandidate\n    with DiscoverTwitterPushIbis2Hydrator\n    with DiscoverTwitterNtabRequestHydrator {\n\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n\n  override val target: Target = candidate.target\n\n  override lazy val commonRecType: CommonRecommendationType = candidate.commonRecType\n\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n\n  override val statsReceiver: StatsReceiver =\n    statsScoped.scope(\"DiscoverTwitterPushCandidate\")\n}\n\ncase class AddressBookPushCandidatePredicates(config: Config)\n    extends BasicRFPHPredicates[DiscoverTwitterPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val predicates: List[\n    NamedPredicate[DiscoverTwitterPushCandidate]\n  ] =\n    List(\n      PredicatesForCandidate.paramPredicate(\n        PushFeatureSwitchParams.EnableAddressBookPush\n      )\n    )\n}\n\ncase class CompleteOnboardingPushCandidatePredicates(config: Config)\n    extends BasicRFPHPredicates[DiscoverTwitterPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val predicates: List[\n    NamedPredicate[DiscoverTwitterPushCandidate]\n  ] =\n    List(\n      PredicatesForCandidate.paramPredicate(\n        PushFeatureSwitchParams.EnableCompleteOnboardingPush\n      )\n    )\n}\n\ncase class PopGeoTweetCandidatePredicates(override val config: Config)\n    extends OutOfNetworkTweetPredicates[OutOfNetworkTweetPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override def postCandidateSpecificPredicates: List[\n    NamedPredicate[OutOfNetworkTweetPushCandidate]\n  ] = List(\n    PredicatesForCandidate.htlFatiguePredicate(\n      PushFeatureSwitchParams.NewUserPlaybookAllowedLastLoginHours\n    )\n  )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/F1FirstdegreeTweet.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.F1FirstDegree\nimport com.twitter.frigate.common.base.SocialContextAction\nimport com.twitter.frigate.common.base.SocialGraphServiceRelationshipMap\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes._\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.F1FirstDegreeTweetIbis2HydratorForCandidate\nimport com.twitter.frigate.pushservice.model.ntab.F1FirstDegreeTweetNTabRequestHydrator\nimport com.twitter.frigate.pushservice.take.predicates.BasicTweetPredicatesForRFPHWithoutSGSPredicates\nimport com.twitter.frigate.pushservice.util.CandidateHydrationUtil.TweetWithSocialContextTraits\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.stitch.tweetypie.TweetyPie\nimport com.twitter.util.Future\n\nclass F1TweetPushCandidate(\n  candidate: RawCandidate with TweetWithSocialContextTraits,\n  author: Future[Option[User]],\n  socialGraphServiceResultMap: Map[RelationEdge, Boolean],\n  copyIds: CopyIds\n)(\n  implicit stats: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with F1FirstDegree\n    with TweetAuthorDetails\n    with SocialGraphServiceRelationshipMap\n    with F1FirstDegreeTweetNTabRequestHydrator\n    with F1FirstDegreeTweetIbis2HydratorForCandidate {\n  override val socialContextActions: Seq[SocialContextAction] =\n    candidate.socialContextActions\n  override val socialContextAllTypeActions: Seq[SocialContextAction] =\n    candidate.socialContextActions\n  override val statsReceiver: StatsReceiver = stats\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n  override val tweetId: Long = candidate.tweetId\n  override lazy val tweetyPieResult: Option[TweetyPie.TweetyPieResult] =\n    candidate.tweetyPieResult\n  override lazy val tweetAuthor: Future[Option[User]] = author\n  override val target: PushTypes.Target = candidate.target\n  override lazy val commonRecType: CommonRecommendationType =\n    candidate.commonRecType\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n\n  override val relationshipMap: Map[RelationEdge, Boolean] = socialGraphServiceResultMap\n}\n\ncase class F1TweetCandidatePredicates(override val config: Config)\n    extends BasicTweetPredicatesForRFPHWithoutSGSPredicates[F1TweetPushCandidate] {\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ListRecommendationPushCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.channels.common.thriftscala.ApiList\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.ListPushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.ListIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.ListCandidateNTabRequestHydrator\nimport com.twitter.frigate.pushservice.predicate.ListPredicates\nimport com.twitter.frigate.pushservice.take.predicates.BasicRFPHPredicates\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nclass ListRecommendationPushCandidate(\n  val apiListStore: ReadableStore[Long, ApiList],\n  candidate: RawCandidate with ListPushCandidate,\n  copyIds: CopyIds\n)(\n  implicit stats: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with ListPushCandidate\n    with ListIbis2Hydrator\n    with ListCandidateNTabRequestHydrator {\n\n  override val commonRecType: CommonRecommendationType = candidate.commonRecType\n\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n\n  override val statsReceiver: StatsReceiver = stats\n\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n\n  override val target: PushTypes.Target = candidate.target\n\n  override val listId: Long = candidate.listId\n\n  lazy val apiList: Future[Option[ApiList]] = apiListStore.get(listId)\n\n  lazy val listName: Future[Option[String]] = apiList.map { apiListOpt =>\n    apiListOpt.map(_.name)\n  }\n\n  lazy val listOwnerId: Future[Option[Long]] = apiList.map { apiListOpt =>\n    apiListOpt.map(_.ownerId)\n  }\n\n}\n\ncase class ListRecommendationPredicates(config: Config)\n    extends BasicRFPHPredicates[ListRecommendationPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val predicates: List[NamedPredicate[ListRecommendationPushCandidate]] = List(\n    ListPredicates.listNameExistsPredicate(),\n    ListPredicates.listAuthorExistsPredicate(),\n    ListPredicates.listAuthorAcceptableToTargetUser(config.edgeStore),\n    ListPredicates.listAcceptablePredicate(),\n    ListPredicates.listSubscriberCountPredicate()\n  )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/MagicFanoutCreatorEventPushCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.HydratedMagicFanoutCreatorEventCandidate\nimport com.twitter.frigate.common.base.MagicFanoutCreatorEventCandidate\nimport com.twitter.frigate.magic_events.thriftscala.CreatorFanoutType\nimport com.twitter.frigate.magic_events.thriftscala.MagicEventsReason\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.MagicFanoutCreatorEventIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.MagicFanoutCreatorEventNtabRequestHydrator\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.predicate.PredicatesForCandidate\nimport com.twitter.frigate.pushservice.predicate.magic_fanout.MagicFanoutPredicatesForCandidate\nimport com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue.MagicFanoutNtabCaretFatiguePredicate\nimport com.twitter.frigate.pushservice.take.predicates.BasicSendHandlerPredicates\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.UserId\nimport com.twitter.util.Future\nimport scala.util.control.NoStackTrace\n\nclass MagicFanoutCreatorEventPushCandidateHydratorException(private val message: String)\n    extends Exception(message)\n    with NoStackTrace\n\nclass MagicFanoutCreatorEventPushCandidate(\n  candidate: RawCandidate with MagicFanoutCreatorEventCandidate,\n  creatorUser: Option[User],\n  copyIds: CopyIds,\n  creatorTweetCountStore: ReadableStore[UserId, Int]\n)(\n  implicit val statsScoped: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with MagicFanoutCreatorEventIbis2Hydrator\n    with MagicFanoutCreatorEventNtabRequestHydrator\n    with MagicFanoutCreatorEventCandidate\n    with HydratedMagicFanoutCreatorEventCandidate {\n  override def creatorId: Long = candidate.creatorId\n\n  override def hydratedCreator: Option[User] = creatorUser\n\n  override lazy val numberOfTweetsFut: Future[Option[Int]] =\n    creatorTweetCountStore.get(UserId(creatorId))\n\n  lazy val userProfile = hydratedCreator\n    .flatMap(_.profile).getOrElse(\n      throw new MagicFanoutCreatorEventPushCandidateHydratorException(\n        s\"Unable to get user profile to generate tapThrough for userId: $creatorId\"))\n\n  override val frigateNotification: FrigateNotification = candidate.frigateNotification\n\n  override def subscriberId: Option[Long] = candidate.subscriberId\n\n  override def creatorFanoutType: CreatorFanoutType = candidate.creatorFanoutType\n\n  override def target: PushTypes.Target = candidate.target\n\n  override def pushId: Long = candidate.pushId\n\n  override def candidateMagicEventsReasons: Seq[MagicEventsReason] =\n    candidate.candidateMagicEventsReasons\n\n  override def statsReceiver: StatsReceiver = statsScoped\n\n  override def pushCopyId: Option[Int] = copyIds.pushCopyId\n\n  override def ntabCopyId: Option[Int] = copyIds.ntabCopyId\n\n  override def copyAggregationId: Option[String] = copyIds.aggregationId\n\n  override def commonRecType: CommonRecommendationType = candidate.commonRecType\n\n  override def weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n\n}\n\ncase class MagicFanouCreatorSubscriptionEventPushPredicates(config: Config)\n    extends BasicSendHandlerPredicates[MagicFanoutCreatorEventPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val preCandidateSpecificPredicates: List[\n    NamedPredicate[MagicFanoutCreatorEventPushCandidate]\n  ] =\n    List(\n      PredicatesForCandidate.paramPredicate(\n        PushFeatureSwitchParams.EnableCreatorSubscriptionPush\n      ),\n      PredicatesForCandidate.isDeviceEligibleForCreatorPush,\n      MagicFanoutPredicatesForCandidate.creatorPushTargetIsNotCreator(),\n      MagicFanoutPredicatesForCandidate.duplicateCreatorPredicate,\n      MagicFanoutPredicatesForCandidate.magicFanoutCreatorPushFatiguePredicate(),\n    )\n\n  override val postCandidateSpecificPredicates: List[\n    NamedPredicate[MagicFanoutCreatorEventPushCandidate]\n  ] =\n    List(\n      MagicFanoutNtabCaretFatiguePredicate(),\n      MagicFanoutPredicatesForCandidate.isSuperFollowingCreator()(config, statsReceiver).flip\n    )\n}\n\ncase class MagicFanoutNewCreatorEventPushPredicates(config: Config)\n    extends BasicSendHandlerPredicates[MagicFanoutCreatorEventPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val preCandidateSpecificPredicates: List[\n    NamedPredicate[MagicFanoutCreatorEventPushCandidate]\n  ] =\n    List(\n      PredicatesForCandidate.paramPredicate(\n        PushFeatureSwitchParams.EnableNewCreatorPush\n      ),\n      PredicatesForCandidate.isDeviceEligibleForCreatorPush,\n      MagicFanoutPredicatesForCandidate.duplicateCreatorPredicate,\n      MagicFanoutPredicatesForCandidate.magicFanoutCreatorPushFatiguePredicate,\n    )\n\n  override val postCandidateSpecificPredicates: List[\n    NamedPredicate[MagicFanoutCreatorEventPushCandidate]\n  ] =\n    List(\n      MagicFanoutNtabCaretFatiguePredicate(),\n      MagicFanoutPredicatesForCandidate.isSuperFollowingCreator()(config, statsReceiver).flip\n    )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/MagicFanoutEventPushCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.escherbird.metadata.thriftscala.EntityMegadata\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.MagicFanoutEventCandidate\nimport com.twitter.frigate.common.base.RecommendationType\nimport com.twitter.frigate.common.store.interests.InterestsLookupRequestWithContext\nimport com.twitter.frigate.common.util.HighPriorityLocaleUtil\nimport com.twitter.frigate.magic_events.thriftscala.FanoutEvent\nimport com.twitter.frigate.magic_events.thriftscala.FanoutMetadata\nimport com.twitter.frigate.magic_events.thriftscala.MagicEventsReason\nimport com.twitter.frigate.magic_events.thriftscala.NewsForYouMetadata\nimport com.twitter.frigate.magic_events.thriftscala.ReasonSource\nimport com.twitter.frigate.magic_events.thriftscala.TargetID\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.Ibis2HydratorForCandidate\nimport com.twitter.frigate.pushservice.model.ntab.EventNTabRequestHydrator\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.predicate.magic_fanout.MagicFanoutPredicatesUtil\nimport com.twitter.frigate.pushservice.store.EventRequest\nimport com.twitter.frigate.pushservice.store.UttEntityHydrationStore\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.frigate.pushservice.util.TopicsUtil\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.frigate.thriftscala.MagicFanoutEventNotificationDetails\nimport com.twitter.hermit.store.semantic_core.SemanticEntityForQuery\nimport com.twitter.interests.thriftscala.InterestId.SemanticCore\nimport com.twitter.interests.thriftscala.UserInterests\nimport com.twitter.livevideo.common.ids.CountryId\nimport com.twitter.livevideo.common.ids.UserId\nimport com.twitter.livevideo.timeline.domain.v2.Event\nimport com.twitter.livevideo.timeline.domain.v2.HydrationOptions\nimport com.twitter.livevideo.timeline.domain.v2.LookupContext\nimport com.twitter.simclusters_v2.thriftscala.SimClustersInferredEntities\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.topiclisting.utt.LocalizedEntity\nimport com.twitter.util.Future\n\nabstract class MagicFanoutEventPushCandidate(\n  candidate: RawCandidate with MagicFanoutEventCandidate with RecommendationType,\n  copyIds: CopyIds,\n  override val fanoutEvent: Option[FanoutEvent],\n  override val semanticEntityResults: Map[SemanticEntityForQuery, Option[EntityMegadata]],\n  simClusterToEntities: Map[Int, Option[SimClustersInferredEntities]],\n  lexServiceStore: ReadableStore[EventRequest, Event],\n  interestsLookupStore: ReadableStore[InterestsLookupRequestWithContext, UserInterests],\n  uttEntityHydrationStore: UttEntityHydrationStore\n)(\n  implicit statsScoped: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with MagicFanoutEventHydratedCandidate\n    with MagicFanoutEventCandidate\n    with EventNTabRequestHydrator\n    with RecommendationType\n    with Ibis2HydratorForCandidate {\n\n  override lazy val eventFut: Future[Option[Event]] = {\n    eventRequestFut.flatMap {\n      case Some(eventRequest) => lexServiceStore.get(eventRequest)\n      case _ => Future.None\n    }\n  }\n\n  override val frigateNotification: FrigateNotification = candidate.frigateNotification\n\n  override val pushId: Long = candidate.pushId\n\n  override val candidateMagicEventsReasons: Seq[MagicEventsReason] =\n    candidate.candidateMagicEventsReasons\n\n  override val eventId: Long = candidate.eventId\n\n  override val momentId: Option[Long] = candidate.momentId\n\n  override val target: Target = candidate.target\n\n  override val eventLanguage: Option[String] = candidate.eventLanguage\n\n  override val details: Option[MagicFanoutEventNotificationDetails] = candidate.details\n\n  override lazy val stats: StatsReceiver = statsScoped.scope(\"MagicFanoutEventPushCandidate\")\n\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n\n  override val statsReceiver: StatsReceiver = statsScoped.scope(\"MagicFanoutEventPushCandidate\")\n\n  override val effectiveMagicEventsReasons: Option[Seq[MagicEventsReason]] = Some(\n    candidateMagicEventsReasons)\n\n  lazy val newsForYouMetadata: Option[NewsForYouMetadata] =\n    fanoutEvent.flatMap { event =>\n      {\n        event.fanoutMetadata.collect {\n          case FanoutMetadata.NewsForYouMetadata(nfyMetadata) => nfyMetadata\n        }\n      }\n    }\n\n  val reverseIndexedTopicIds = candidate.candidateMagicEventsReasons\n    .filter(_.source.contains(ReasonSource.UttTopicFollowGraph))\n    .map(_.reason).collect {\n      case TargetID.SemanticCoreID(semanticCoreID) => semanticCoreID.entityId\n    }.toSet\n\n  val ergSemanticCoreIds = candidate.candidateMagicEventsReasons\n    .filter(_.source.contains(ReasonSource.ErgShortTermInterestSemanticCore)).map(\n      _.reason).collect {\n      case TargetID.SemanticCoreID(semanticCoreID) => semanticCoreID.entityId\n    }.toSet\n\n  override lazy val ergLocalizedEntities = TopicsUtil\n    .getLocalizedEntityMap(target, ergSemanticCoreIds, uttEntityHydrationStore)\n    .map { localizedEntityMap =>\n      ergSemanticCoreIds.collect {\n        case topicId if localizedEntityMap.contains(topicId) => localizedEntityMap(topicId)\n      }\n    }\n\n  val eventSemanticCoreEntityIds: Seq[Long] = {\n    val entityIds = for {\n      event <- fanoutEvent\n      targets <- event.targets\n    } yield {\n      targets.flatMap {\n        _.whitelist.map {\n          _.collect {\n            case TargetID.SemanticCoreID(semanticCoreID) => semanticCoreID.entityId\n          }\n        }\n      }\n    }\n\n    entityIds.map(_.flatten).getOrElse(Seq.empty)\n  }\n\n  val eventSemanticCoreDomainIds: Seq[Long] = {\n    val domainIds = for {\n      event <- fanoutEvent\n      targets <- event.targets\n    } yield {\n      targets.flatMap {\n        _.whitelist.map {\n          _.collect {\n            case TargetID.SemanticCoreID(semanticCoreID) => semanticCoreID.domainId\n          }\n        }\n      }\n    }\n\n    domainIds.map(_.flatten).getOrElse(Seq.empty)\n  }\n\n  override lazy val followedTopicLocalizedEntities: Future[Set[LocalizedEntity]] = {\n\n    val isNewSignupTargetingReason = candidateMagicEventsReasons.size == 1 &&\n      candidateMagicEventsReasons.headOption.exists(_.source.contains(ReasonSource.NewSignup))\n\n    val shouldFetchTopicFollows = reverseIndexedTopicIds.nonEmpty || isNewSignupTargetingReason\n\n    val topicFollows = if (shouldFetchTopicFollows) {\n      TopicsUtil\n        .getTopicsFollowedByUser(\n          candidate.target,\n          interestsLookupStore,\n          stats.stat(\"followed_topics\")\n        ).map { _.getOrElse(Seq.empty) }.map {\n          _.flatMap {\n            _.interestId match {\n              case SemanticCore(semanticCore) => Some(semanticCore.id)\n              case _ => None\n            }\n          }\n        }\n    } else Future.Nil\n\n    topicFollows.flatMap { followedTopicIds =>\n      val topicIds = if (isNewSignupTargetingReason) {\n        // if new signup is the only targeting reason then we check the event targeting reason\n        // against realtime topic follows.\n        eventSemanticCoreEntityIds.toSet.intersect(followedTopicIds.toSet)\n      } else {\n        // check against the fanout reason of topics\n        followedTopicIds.toSet.intersect(reverseIndexedTopicIds)\n      }\n\n      TopicsUtil\n        .getLocalizedEntityMap(target, topicIds, uttEntityHydrationStore)\n        .map { localizedEntityMap =>\n          topicIds.collect {\n            case topicId if localizedEntityMap.contains(topicId) => localizedEntityMap(topicId)\n          }\n        }\n    }\n  }\n\n  lazy val simClusterToEntityMapping: Map[Int, Seq[Long]] =\n    simClusterToEntities.flatMap {\n      case (clusterId, Some(inferredEntities)) =>\n        statsReceiver.counter(\"with_cluster_to_entity_mapping\").incr()\n        Some(\n          (\n            clusterId,\n            inferredEntities.entities\n              .map(_.entityId)))\n      case _ =>\n        statsReceiver.counter(\"without_cluster_to_entity_mapping\").incr()\n        None\n    }\n\n  lazy val annotatedAndInferredSemanticCoreEntities: Seq[Long] =\n    (simClusterToEntityMapping, eventFanoutReasonEntities) match {\n      case (entityMapping, eventFanoutReasons) =>\n        entityMapping.values.flatten.toSeq ++\n          eventFanoutReasons.semanticCoreIds.map(_.entityId)\n    }\n\n  lazy val shouldHydrateSquareImage = target.deviceInfo.map { deviceInfo =>\n    (PushDeviceUtil.isPrimaryDeviceIOS(deviceInfo) &&\n    target.params(PushFeatureSwitchParams.EnableEventSquareMediaIosMagicFanoutNewsEvent)) ||\n    (PushDeviceUtil.isPrimaryDeviceAndroid(deviceInfo) &&\n    target.params(PushFeatureSwitchParams.EnableEventSquareMediaAndroid))\n  }\n\n  lazy val shouldHydratePrimaryImage: Future[Boolean] = target.deviceInfo.map { deviceInfo =>\n    (PushDeviceUtil.isPrimaryDeviceAndroid(deviceInfo) &&\n    target.params(PushFeatureSwitchParams.EnableEventPrimaryMediaAndroid))\n  }\n\n  lazy val eventRequestFut: Future[Option[EventRequest]] =\n    Future\n      .join(\n        target.inferredUserDeviceLanguage,\n        target.accountCountryCode,\n        shouldHydrateSquareImage,\n        shouldHydratePrimaryImage).map {\n        case (\n              inferredUserDeviceLanguage,\n              accountCountryCode,\n              shouldHydrateSquareImage,\n              shouldHydratePrimaryImage) =>\n          if (shouldHydrateSquareImage || shouldHydratePrimaryImage) {\n            Some(\n              EventRequest(\n                eventId,\n                lookupContext = LookupContext(\n                  hydrationOptions = HydrationOptions(\n                    includeSquareImage = shouldHydrateSquareImage,\n                    includePrimaryImage = shouldHydratePrimaryImage\n                  ),\n                  language = inferredUserDeviceLanguage,\n                  countryCode = accountCountryCode,\n                  userId = Some(UserId(target.targetId))\n                )\n              ))\n          } else {\n            Some(\n              EventRequest(\n                eventId,\n                lookupContext = LookupContext(\n                  language = inferredUserDeviceLanguage,\n                  countryCode = accountCountryCode\n                )\n              ))\n          }\n        case _ => None\n      }\n\n  lazy val isHighPriorityEvent: Future[Boolean] = target.accountCountryCode.map { countryCodeOpt =>\n    val isHighPriorityPushOpt = for {\n      countryCode <- countryCodeOpt\n      nfyMetadata <- newsForYouMetadata\n      eventContext <- nfyMetadata.eventContextScribe\n    } yield {\n      val highPriorityLocales = HighPriorityLocaleUtil.getHighPriorityLocales(\n        eventContext = eventContext,\n        defaultLocalesOpt = nfyMetadata.locales)\n      val highPriorityGeos = HighPriorityLocaleUtil.getHighPriorityGeos(\n        eventContext = eventContext,\n        defaultGeoPlaceIdsOpt = nfyMetadata.placeIds)\n      val isHighPriorityLocalePush =\n        highPriorityLocales.flatMap(_.country).map(CountryId(_)).contains(CountryId(countryCode))\n      val isHighPriorityGeoPush = MagicFanoutPredicatesUtil\n        .geoPlaceIdsFromReasons(candidateMagicEventsReasons)\n        .intersect(highPriorityGeos.toSet)\n        .nonEmpty\n      stats.scope(\"is_high_priority_locale_push\").counter(s\"$isHighPriorityLocalePush\").incr()\n      stats.scope(\"is_high_priority_geo_push\").counter(s\"$isHighPriorityGeoPush\").incr()\n      isHighPriorityLocalePush || isHighPriorityGeoPush\n    }\n    isHighPriorityPushOpt.getOrElse(false)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/MagicFanoutHydratedCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.escherbird.common.thriftscala.QualifiedId\nimport com.twitter.escherbird.metadata.thriftscala.BasicMetadata\nimport com.twitter.escherbird.metadata.thriftscala.EntityIndexFields\nimport com.twitter.escherbird.metadata.thriftscala.EntityMegadata\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.MagicFanoutCandidate\nimport com.twitter.frigate.common.base.MagicFanoutEventCandidate\nimport com.twitter.frigate.common.base.RichEventFutCandidate\nimport com.twitter.frigate.magic_events.thriftscala\nimport com.twitter.frigate.magic_events.thriftscala.AnnotationAlg\nimport com.twitter.frigate.magic_events.thriftscala.FanoutEvent\nimport com.twitter.frigate.magic_events.thriftscala.MagicEventsReason\nimport com.twitter.frigate.magic_events.thriftscala.SemanticCoreID\nimport com.twitter.frigate.magic_events.thriftscala.SimClusterID\nimport com.twitter.frigate.magic_events.thriftscala.TargetID\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.hermit.store.semantic_core.SemanticEntityForQuery\nimport com.twitter.livevideo.timeline.domain.v2.Event\nimport com.twitter.topiclisting.utt.LocalizedEntity\nimport com.twitter.util.Future\n\ncase class FanoutReasonEntities(\n  userIds: Set[Long],\n  placeIds: Set[Long],\n  semanticCoreIds: Set[SemanticCoreID],\n  simclusterIds: Set[SimClusterID]) {\n  val qualifiedIds: Set[QualifiedId] =\n    semanticCoreIds.map(e => QualifiedId(e.domainId, e.entityId))\n}\n\nobject FanoutReasonEntities {\n  val empty = FanoutReasonEntities(\n    userIds = Set.empty,\n    placeIds = Set.empty,\n    semanticCoreIds = Set.empty,\n    simclusterIds = Set.empty\n  )\n\n  def from(reasons: Seq[TargetID]): FanoutReasonEntities = {\n    val userIds: Set[Long] = reasons.collect {\n      case TargetID.UserID(userId) => userId.id\n    }.toSet\n    val placeIds: Set[Long] = reasons.collect {\n      case TargetID.PlaceID(placeId) => placeId.id\n    }.toSet\n    val semanticCoreIds: Set[SemanticCoreID] = reasons.collect {\n      case TargetID.SemanticCoreID(semanticCoreID) => semanticCoreID\n    }.toSet\n    val simclusterIds: Set[SimClusterID] = reasons.collect {\n      case TargetID.SimClusterID(simClusterID) => simClusterID\n    }.toSet\n\n    FanoutReasonEntities(\n      userIds = userIds,\n      placeIds,\n      semanticCoreIds = semanticCoreIds,\n      simclusterIds = simclusterIds\n    )\n  }\n}\n\ntrait MagicFanoutHydratedCandidate extends PushCandidate with MagicFanoutCandidate {\n  lazy val fanoutReasonEntities: FanoutReasonEntities =\n    FanoutReasonEntities.from(candidateMagicEventsReasons.map(_.reason))\n}\n\ntrait MagicFanoutEventHydratedCandidate\n    extends MagicFanoutHydratedCandidate\n    with MagicFanoutEventCandidate\n    with RichEventFutCandidate {\n\n  def target: PushTypes.Target\n\n  def stats: StatsReceiver\n\n  def fanoutEvent: Option[FanoutEvent]\n\n  def eventFut: Future[Option[Event]]\n\n  def semanticEntityResults: Map[SemanticEntityForQuery, Option[EntityMegadata]]\n\n  def effectiveMagicEventsReasons: Option[Seq[MagicEventsReason]]\n\n  def followedTopicLocalizedEntities: Future[Set[LocalizedEntity]]\n\n  def ergLocalizedEntities: Future[Set[LocalizedEntity]]\n\n  lazy val entityAnnotationAlg: Map[TargetID, Set[AnnotationAlg]] =\n    fanoutEvent\n      .flatMap { metadata =>\n        metadata.eventAnnotationInfo.map { eventAnnotationInfo =>\n          eventAnnotationInfo.map {\n            case (target, annotationInfoSet) => target -> annotationInfoSet.map(_.alg).toSet\n          }.toMap\n        }\n      }.getOrElse(Map.empty)\n\n  lazy val eventSource: Option[String] = fanoutEvent.map { metadata =>\n    val source = metadata.eventSource.getOrElse(\"undefined\")\n    stats.scope(\"eventSource\").counter(source).incr()\n    source\n  }\n\n  lazy val semanticCoreEntityTags: Map[(Long, Long), Set[String]] =\n    semanticEntityResults.flatMap {\n      case (semanticEntityForQuery, entityMegadataOpt: Option[EntityMegadata]) =>\n        for {\n          entityMegadata <- entityMegadataOpt\n          basicMetadata: BasicMetadata <- entityMegadata.basicMetadata\n          indexableFields: EntityIndexFields <- basicMetadata.indexableFields\n          tags <- indexableFields.tags\n        } yield {\n          ((semanticEntityForQuery.domainId, semanticEntityForQuery.entityId), tags.toSet)\n        }\n    }\n\n  lazy val owningTwitterUserIds: Seq[Long] = semanticEntityResults.values.flatten\n    .flatMap {\n      _.basicMetadata.flatMap(_.twitter.flatMap(_.owningTwitterUserIds))\n    }.flatten\n    .toSeq\n    .distinct\n\n  lazy val eventFanoutReasonEntities: FanoutReasonEntities =\n    fanoutEvent match {\n      case Some(fanout) =>\n        fanout.targets\n          .map { targets: Seq[thriftscala.Target] =>\n            FanoutReasonEntities.from(targets.flatMap(_.whitelist).flatten)\n          }.getOrElse(FanoutReasonEntities.empty)\n      case _ => FanoutReasonEntities.empty\n    }\n\n  override lazy val eventResultFut: Future[Event] = eventFut.map {\n    case Some(eventResult) => eventResult\n    case _ =>\n      throw new IllegalArgumentException(\"event is None for MagicFanoutEventHydratedCandidate\")\n  }\n  override val rankScore: Option[Double] = None\n  override val predictionScore: Option[Double] = None\n}\n\ncase class MagicFanoutEventHydratedInfo(\n  fanoutEvent: Option[FanoutEvent],\n  semanticEntityResults: Map[SemanticEntityForQuery, Option[EntityMegadata]])\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/MagicFanoutNewsEvent.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.escherbird.metadata.thriftscala.EntityMegadata\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.MagicFanoutNewsEventCandidate\nimport com.twitter.frigate.common.store.interests.InterestsLookupRequestWithContext\nimport com.twitter.frigate.magic_events.thriftscala.FanoutEvent\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.MagicFanoutNewsEventIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.MagicFanoutNewsEventNTabRequestHydrator\nimport com.twitter.frigate.pushservice.predicate.PredicatesForCandidate\nimport com.twitter.frigate.pushservice.predicate.event.EventPredicatesForCandidate\nimport com.twitter.frigate.pushservice.predicate.magic_fanout.MagicFanoutPredicatesForCandidate\nimport com.twitter.frigate.pushservice.predicate.magic_fanout.MagicFanoutTargetingPredicateWrappersForCandidate\nimport com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue.MagicFanoutNtabCaretFatiguePredicate\nimport com.twitter.frigate.pushservice.store.EventRequest\nimport com.twitter.frigate.pushservice.store.UttEntityHydrationStore\nimport com.twitter.frigate.pushservice.take.predicates.BasicSendHandlerPredicates\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.store.semantic_core.SemanticEntityForQuery\nimport com.twitter.interests.thriftscala.UserInterests\nimport com.twitter.livevideo.timeline.domain.v2.Event\nimport com.twitter.simclusters_v2.thriftscala.SimClustersInferredEntities\nimport com.twitter.storehaus.ReadableStore\n\nclass MagicFanoutNewsEventPushCandidate(\n  candidate: RawCandidate with MagicFanoutNewsEventCandidate,\n  copyIds: CopyIds,\n  override val fanoutEvent: Option[FanoutEvent],\n  override val semanticEntityResults: Map[SemanticEntityForQuery, Option[EntityMegadata]],\n  simClusterToEntities: Map[Int, Option[SimClustersInferredEntities]],\n  lexServiceStore: ReadableStore[EventRequest, Event],\n  interestsLookupStore: ReadableStore[InterestsLookupRequestWithContext, UserInterests],\n  uttEntityHydrationStore: UttEntityHydrationStore\n)(\n  implicit statsScoped: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends MagicFanoutEventPushCandidate(\n      candidate,\n      copyIds,\n      fanoutEvent,\n      semanticEntityResults,\n      simClusterToEntities,\n      lexServiceStore,\n      interestsLookupStore,\n      uttEntityHydrationStore\n    )(statsScoped, pushModelScorer)\n    with MagicFanoutNewsEventCandidate\n    with MagicFanoutNewsEventIbis2Hydrator\n    with MagicFanoutNewsEventNTabRequestHydrator {\n\n  override lazy val stats: StatsReceiver = statsScoped.scope(\"MagicFanoutNewsEventPushCandidate\")\n  override val statsReceiver: StatsReceiver = statsScoped.scope(\"MagicFanoutNewsEventPushCandidate\")\n}\n\ncase class MagicFanoutNewsEventCandidatePredicates(config: Config)\n    extends BasicSendHandlerPredicates[MagicFanoutNewsEventPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val preCandidateSpecificPredicates: List[\n    NamedPredicate[MagicFanoutNewsEventPushCandidate]\n  ] =\n    List(\n      EventPredicatesForCandidate.accountCountryPredicateWithAllowlist,\n      PredicatesForCandidate.isDeviceEligibleForNewsOrSports,\n      MagicFanoutPredicatesForCandidate.inferredUserDeviceLanguagePredicate,\n      PredicatesForCandidate.secondaryDormantAccountPredicate(statsReceiver),\n      MagicFanoutPredicatesForCandidate.highPriorityNewsEventExceptedPredicate(\n        MagicFanoutTargetingPredicateWrappersForCandidate\n          .magicFanoutTargetingPredicate(statsReceiver, config)\n      )(config),\n      MagicFanoutPredicatesForCandidate.geoOptOutPredicate(config.safeUserStore),\n      EventPredicatesForCandidate.isNotDuplicateWithEventId,\n      MagicFanoutPredicatesForCandidate.highPriorityNewsEventExceptedPredicate(\n        MagicFanoutPredicatesForCandidate.newsNotificationFatigue()\n      )(config),\n      MagicFanoutPredicatesForCandidate.highPriorityNewsEventExceptedPredicate(\n        MagicFanoutNtabCaretFatiguePredicate()\n      )(config),\n      MagicFanoutPredicatesForCandidate.escherbirdMagicfanoutEventParam()(statsReceiver),\n      MagicFanoutPredicatesForCandidate.hasCustomTargetingForNewsEventsParam(\n        statsReceiver\n      )\n    )\n\n  override val postCandidateSpecificPredicates: List[\n    NamedPredicate[MagicFanoutNewsEventPushCandidate]\n  ] =\n    List(\n      MagicFanoutPredicatesForCandidate.magicFanoutNoOptoutInterestPredicate,\n      MagicFanoutPredicatesForCandidate.geoTargetingHoldback(),\n      MagicFanoutPredicatesForCandidate.userGeneratedEventsPredicate,\n      EventPredicatesForCandidate.hasTitle,\n    )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/MagicFanoutProductLaunchPushCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.MagicFanoutProductLaunchCandidate\nimport com.twitter.frigate.common.util.{FeatureSwitchParams => FS}\nimport com.twitter.frigate.magic_events.thriftscala.MagicEventsReason\nimport com.twitter.frigate.magic_events.thriftscala.ProductType\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.predicate.magic_fanout.MagicFanoutPredicatesUtil\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.MagicFanoutProductLaunchIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.MagicFanoutProductLaunchNtabRequestHydrator\nimport com.twitter.frigate.pushservice.predicate.PredicatesForCandidate\nimport com.twitter.frigate.pushservice.predicate.magic_fanout.MagicFanoutPredicatesForCandidate\nimport com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue.MagicFanoutNtabCaretFatiguePredicate\nimport com.twitter.frigate.pushservice.take.predicates.BasicSendHandlerPredicates\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.hermit.predicate.NamedPredicate\n\nclass MagicFanoutProductLaunchPushCandidate(\n  candidate: RawCandidate with MagicFanoutProductLaunchCandidate,\n  copyIds: CopyIds\n)(\n  implicit val statsScoped: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with MagicFanoutProductLaunchCandidate\n    with MagicFanoutProductLaunchIbis2Hydrator\n    with MagicFanoutProductLaunchNtabRequestHydrator {\n\n  override val frigateNotification: FrigateNotification = candidate.frigateNotification\n\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n\n  override val pushId: Long = candidate.pushId\n\n  override val productLaunchType: ProductType = candidate.productLaunchType\n\n  override val candidateMagicEventsReasons: Seq[MagicEventsReason] =\n    candidate.candidateMagicEventsReasons\n\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n\n  override val target: Target = candidate.target\n\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n\n  override val statsReceiver: StatsReceiver =\n    statsScoped.scope(\"MagicFanoutProductLaunchPushCandidate\")\n}\n\ncase class MagicFanoutProductLaunchPushCandidatePredicates(config: Config)\n    extends BasicSendHandlerPredicates[MagicFanoutProductLaunchPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val preCandidateSpecificPredicates: List[\n    NamedPredicate[MagicFanoutProductLaunchPushCandidate]\n  ] =\n    List(\n      PredicatesForCandidate.isDeviceEligibleForCreatorPush,\n      PredicatesForCandidate.exceptedPredicate(\n        \"excepted_is_target_blue_verified\",\n        MagicFanoutPredicatesUtil.shouldSkipBlueVerifiedCheckForCandidate,\n        PredicatesForCandidate.isTargetBlueVerified.flip\n      ), // no need to send if target is already Blue Verified\n      PredicatesForCandidate.exceptedPredicate(\n        \"excepted_is_target_legacy_verified\",\n        MagicFanoutPredicatesUtil.shouldSkipLegacyVerifiedCheckForCandidate,\n        PredicatesForCandidate.isTargetLegacyVerified.flip\n      ), // no need to send if target is already Legacy Verified\n      PredicatesForCandidate.exceptedPredicate(\n        \"excepted_is_target_super_follow_creator\",\n        MagicFanoutPredicatesUtil.shouldSkipSuperFollowCreatorCheckForCandidate,\n        PredicatesForCandidate.isTargetSuperFollowCreator.flip\n      ), // no need to send if target is already Super Follow Creator\n      PredicatesForCandidate.paramPredicate(\n        FS.EnableMagicFanoutProductLaunch\n      ),\n      MagicFanoutPredicatesForCandidate.magicFanoutProductLaunchFatigue(),\n    )\n\n  override val postCandidateSpecificPredicates: List[\n    NamedPredicate[MagicFanoutProductLaunchPushCandidate]\n  ] =\n    List(\n      MagicFanoutNtabCaretFatiguePredicate(),\n    )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/MagicFanoutSportsPushCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.escherbird.metadata.thriftscala.EntityMegadata\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.BaseGameScore\nimport com.twitter.frigate.common.base.MagicFanoutSportsEventCandidate\nimport com.twitter.frigate.common.base.MagicFanoutSportsScoreInformation\nimport com.twitter.frigate.common.base.TeamInfo\nimport com.twitter.frigate.common.store.interests.InterestsLookupRequestWithContext\nimport com.twitter.frigate.magic_events.thriftscala.FanoutEvent\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.MagicFanoutSportsEventIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.MagicFanoutSportsEventNTabRequestHydrator\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.predicate.PredicatesForCandidate\nimport com.twitter.frigate.pushservice.predicate.magic_fanout.MagicFanoutPredicatesForCandidate\nimport com.twitter.frigate.pushservice.predicate.magic_fanout.MagicFanoutTargetingPredicateWrappersForCandidate\nimport com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue.MagicFanoutNtabCaretFatiguePredicate\nimport com.twitter.frigate.pushservice.store.EventRequest\nimport com.twitter.frigate.pushservice.store.UttEntityHydrationStore\nimport com.twitter.frigate.pushservice.take.predicates.BasicSendHandlerPredicates\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.store.semantic_core.SemanticEntityForQuery\nimport com.twitter.interests.thriftscala.UserInterests\nimport com.twitter.livevideo.timeline.domain.v2.Event\nimport com.twitter.livevideo.timeline.domain.v2.HydrationOptions\nimport com.twitter.livevideo.timeline.domain.v2.LookupContext\nimport com.twitter.simclusters_v2.thriftscala.SimClustersInferredEntities\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nclass MagicFanoutSportsPushCandidate(\n  candidate: RawCandidate\n    with MagicFanoutSportsEventCandidate\n    with MagicFanoutSportsScoreInformation,\n  copyIds: CopyIds,\n  override val fanoutEvent: Option[FanoutEvent],\n  override val semanticEntityResults: Map[SemanticEntityForQuery, Option[EntityMegadata]],\n  simClusterToEntities: Map[Int, Option[SimClustersInferredEntities]],\n  lexServiceStore: ReadableStore[EventRequest, Event],\n  interestsLookupStore: ReadableStore[InterestsLookupRequestWithContext, UserInterests],\n  uttEntityHydrationStore: UttEntityHydrationStore\n)(\n  implicit statsScoped: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends MagicFanoutEventPushCandidate(\n      candidate,\n      copyIds,\n      fanoutEvent,\n      semanticEntityResults,\n      simClusterToEntities,\n      lexServiceStore,\n      interestsLookupStore,\n      uttEntityHydrationStore)(statsScoped, pushModelScorer)\n    with MagicFanoutSportsEventCandidate\n    with MagicFanoutSportsScoreInformation\n    with MagicFanoutSportsEventNTabRequestHydrator\n    with MagicFanoutSportsEventIbis2Hydrator {\n\n  override val isScoreUpdate: Boolean = candidate.isScoreUpdate\n  override val gameScores: Future[Option[BaseGameScore]] = candidate.gameScores\n  override val homeTeamInfo: Future[Option[TeamInfo]] = candidate.homeTeamInfo\n  override val awayTeamInfo: Future[Option[TeamInfo]] = candidate.awayTeamInfo\n\n  override lazy val stats: StatsReceiver = statsScoped.scope(\"MagicFanoutSportsPushCandidate\")\n  override val statsReceiver: StatsReceiver = statsScoped.scope(\"MagicFanoutSportsPushCandidate\")\n\n  override lazy val eventRequestFut: Future[Option[EventRequest]] = {\n    Future.join(target.inferredUserDeviceLanguage, target.accountCountryCode).map {\n      case (inferredUserDeviceLanguage, accountCountryCode) =>\n        Some(\n          EventRequest(\n            eventId,\n            lookupContext = LookupContext(\n              hydrationOptions = HydrationOptions(\n                includeSquareImage = true,\n                includePrimaryImage = true\n              ),\n              language = inferredUserDeviceLanguage,\n              countryCode = accountCountryCode\n            )\n          ))\n    }\n  }\n}\n\ncase class MagicFanoutSportsEventCandidatePredicates(config: Config)\n    extends BasicSendHandlerPredicates[MagicFanoutSportsPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val preCandidateSpecificPredicates: List[\n    NamedPredicate[MagicFanoutSportsPushCandidate]\n  ] =\n    List(\n      PredicatesForCandidate.paramPredicate(PushFeatureSwitchParams.EnableScoreFanoutNotification)\n    )\n\n  override val postCandidateSpecificPredicates: List[\n    NamedPredicate[MagicFanoutSportsPushCandidate]\n  ] =\n    List(\n      PredicatesForCandidate.isDeviceEligibleForNewsOrSports,\n      MagicFanoutPredicatesForCandidate.inferredUserDeviceLanguagePredicate,\n      MagicFanoutPredicatesForCandidate.highPriorityEventExceptedPredicate(\n        MagicFanoutTargetingPredicateWrappersForCandidate\n          .magicFanoutTargetingPredicate(statsReceiver, config)\n      )(config),\n      PredicatesForCandidate.secondaryDormantAccountPredicate(\n        statsReceiver\n      ),\n      MagicFanoutPredicatesForCandidate.highPriorityEventExceptedPredicate(\n        MagicFanoutNtabCaretFatiguePredicate()\n      )(config),\n    )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/OutOfNetworkTweetPushCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.contentrecommender.thriftscala.MetricTag\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.OutOfNetworkTweetCandidate\nimport com.twitter.frigate.common.base.TopicCandidate\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.OutOfNetworkTweetIbis2HydratorForCandidate\nimport com.twitter.frigate.pushservice.model.ntab.OutOfNetworkTweetNTabRequestHydrator\nimport com.twitter.frigate.pushservice.predicate.HealthPredicates\nimport com.twitter.frigate.pushservice.take.predicates.OutOfNetworkTweetPredicates\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.stitch.tweetypie.TweetyPie\nimport com.twitter.topiclisting.utt.LocalizedEntity\nimport com.twitter.util.Future\n\nclass OutOfNetworkTweetPushCandidate(\n  candidate: RawCandidate with OutOfNetworkTweetCandidate with TopicCandidate,\n  author: Future[Option[User]],\n  copyIds: CopyIds\n)(\n  implicit stats: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with OutOfNetworkTweetCandidate\n    with TopicCandidate\n    with TweetAuthorDetails\n    with OutOfNetworkTweetNTabRequestHydrator\n    with OutOfNetworkTweetIbis2HydratorForCandidate {\n  override val statsReceiver: StatsReceiver = stats\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n  override val tweetId: Long = candidate.tweetId\n  override lazy val tweetyPieResult: Option[TweetyPie.TweetyPieResult] =\n    candidate.tweetyPieResult\n  override lazy val tweetAuthor: Future[Option[User]] = author\n  override val target: PushTypes.Target = candidate.target\n  override lazy val commonRecType: CommonRecommendationType =\n    candidate.commonRecType\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n  override lazy val semanticCoreEntityId: Option[Long] = candidate.semanticCoreEntityId\n  override lazy val localizedUttEntity: Option[LocalizedEntity] = candidate.localizedUttEntity\n  override lazy val algorithmCR: Option[String] = candidate.algorithmCR\n  override lazy val isMrBackfillCR: Option[Boolean] = candidate.isMrBackfillCR\n  override lazy val tagsCR: Option[Seq[MetricTag]] = candidate.tagsCR\n}\n\ncase class OutOfNetworkTweetCandidatePredicates(override val config: Config)\n    extends OutOfNetworkTweetPredicates[OutOfNetworkTweetPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override def postCandidateSpecificPredicates: List[\n    NamedPredicate[OutOfNetworkTweetPushCandidate]\n  ] =\n    List(\n      HealthPredicates.agathaAbusiveTweetAuthorPredicateMrTwistly(),\n    )\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/PushTypes.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.candidate.UserLanguage\nimport com.twitter.frigate.common.candidate._\nimport com.twitter.frigate.data_pipeline.features_common.RequestContextForFeatureStore\nimport com.twitter.frigate.pushservice.model.candidate.CopyInfo\nimport com.twitter.frigate.pushservice.model.candidate.MLScores\nimport com.twitter.frigate.pushservice.model.candidate.QualityScribing\nimport com.twitter.frigate.pushservice.model.candidate.Scriber\nimport com.twitter.frigate.pushservice.model.ibis.Ibis2HydratorForCandidate\nimport com.twitter.frigate.pushservice.model.ntab.NTabRequest\nimport com.twitter.frigate.pushservice.take.ChannelForCandidate\nimport com.twitter.frigate.pushservice.target._\nimport com.twitter.util.Time\n\nobject PushTypes {\n\n  trait Target\n      extends TargetUser\n      with UserDetails\n      with TargetWithPushContext\n      with TargetDecider\n      with TargetABDecider\n      with FrigateHistory\n      with PushTargeting\n      with TargetScoringDetails\n      with TweetImpressionHistory\n      with CustomConfigForExpt\n      with CaretFeedbackHistory\n      with NotificationFeedbackHistory\n      with PromptFeedbackHistory\n      with HTLVisitHistory\n      with MaxTweetAge\n      with NewUserDetails\n      with ResurrectedUserDetails\n      with TargetWithSeedUsers\n      with MagicFanoutHistory\n      with OptOutUserInterests\n      with RequestContextForFeatureStore\n      with TargetAppPermissions\n      with UserLanguage\n      with InlineActionHistory\n      with TargetPlaces\n\n  trait RawCandidate extends Candidate with TargetInfo[PushTypes.Target] with RecommendationType {\n\n    val createdAt: Time = Time.now\n  }\n\n  trait PushCandidate\n      extends RawCandidate\n      with CandidateScoringDetails\n      with MLScores\n      with QualityScribing\n      with CopyInfo\n      with Scriber\n      with Ibis2HydratorForCandidate\n      with NTabRequest\n      with ChannelForCandidate\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ScheduledSpaceSpeaker.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.ScheduledSpaceSpeakerCandidate\nimport com.twitter.frigate.common.base.SpaceCandidateFanoutDetails\nimport com.twitter.frigate.common.util.FeatureSwitchParams\nimport com.twitter.frigate.magic_events.thriftscala.SpaceMetadata\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.ScheduledSpaceSpeakerIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.ScheduledSpaceSpeakerNTabRequestHydrator\nimport com.twitter.frigate.pushservice.predicate.PredicatesForCandidate\nimport com.twitter.frigate.pushservice.predicate.SpacePredicate\nimport com.twitter.frigate.pushservice.take.predicates.BasicSendHandlerPredicates\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.ubs.thriftscala.AudioSpace\nimport com.twitter.util.Future\n\nclass ScheduledSpaceSpeakerPushCandidate(\n  candidate: RawCandidate with ScheduledSpaceSpeakerCandidate,\n  hostUser: Option[User],\n  copyIds: CopyIds,\n  audioSpaceStore: ReadableStore[String, AudioSpace]\n)(\n  implicit val statsScoped: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with ScheduledSpaceSpeakerCandidate\n    with ScheduledSpaceSpeakerIbis2Hydrator\n    with SpaceCandidateFanoutDetails\n    with ScheduledSpaceSpeakerNTabRequestHydrator {\n\n  override val startTime: Long = candidate.startTime\n\n  override val hydratedHost: Option[User] = hostUser\n\n  override val spaceId: String = candidate.spaceId\n\n  override val hostId: Option[Long] = candidate.hostId\n\n  override val speakerIds: Option[Seq[Long]] = candidate.speakerIds\n\n  override val listenerIds: Option[Seq[Long]] = candidate.listenerIds\n\n  override val frigateNotification: FrigateNotification = candidate.frigateNotification\n\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n\n  override val target: Target = candidate.target\n\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n\n  override lazy val audioSpaceFut: Future[Option[AudioSpace]] = audioSpaceStore.get(spaceId)\n\n  override val spaceFanoutMetadata: Option[SpaceMetadata] = None\n\n  override val statsReceiver: StatsReceiver =\n    statsScoped.scope(\"ScheduledSpaceSpeakerCandidate\")\n}\n\ncase class ScheduledSpaceSpeakerCandidatePredicates(config: Config)\n    extends BasicSendHandlerPredicates[ScheduledSpaceSpeakerPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val preCandidateSpecificPredicates: List[\n    NamedPredicate[ScheduledSpaceSpeakerPushCandidate]\n  ] = List(\n    SpacePredicate.scheduledSpaceStarted(\n      config.audioSpaceStore\n    ),\n    PredicatesForCandidate.paramPredicate(FeatureSwitchParams.EnableScheduledSpaceSpeakers)\n  )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ScheduledSpaceSubscriber.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.ScheduledSpaceSubscriberCandidate\nimport com.twitter.frigate.common.base.SpaceCandidateFanoutDetails\nimport com.twitter.frigate.common.util.FeatureSwitchParams\nimport com.twitter.frigate.magic_events.thriftscala.SpaceMetadata\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.ScheduledSpaceSubscriberIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.ScheduledSpaceSubscriberNTabRequestHydrator\nimport com.twitter.frigate.pushservice.predicate._\nimport com.twitter.frigate.pushservice.take.predicates.BasicSendHandlerPredicates\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.ubs.thriftscala.AudioSpace\nimport com.twitter.util.Future\n\nclass ScheduledSpaceSubscriberPushCandidate(\n  candidate: RawCandidate with ScheduledSpaceSubscriberCandidate,\n  hostUser: Option[User],\n  copyIds: CopyIds,\n  audioSpaceStore: ReadableStore[String, AudioSpace]\n)(\n  implicit val statsScoped: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with ScheduledSpaceSubscriberCandidate\n    with SpaceCandidateFanoutDetails\n    with ScheduledSpaceSubscriberIbis2Hydrator\n    with ScheduledSpaceSubscriberNTabRequestHydrator {\n\n  override val startTime: Long = candidate.startTime\n\n  override val hydratedHost: Option[User] = hostUser\n\n  override val spaceId: String = candidate.spaceId\n\n  override val hostId: Option[Long] = candidate.hostId\n\n  override val speakerIds: Option[Seq[Long]] = candidate.speakerIds\n\n  override val listenerIds: Option[Seq[Long]] = candidate.listenerIds\n\n  override val frigateNotification: FrigateNotification = candidate.frigateNotification\n\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n\n  override val target: Target = candidate.target\n\n  override lazy val audioSpaceFut: Future[Option[AudioSpace]] = audioSpaceStore.get(spaceId)\n\n  override val spaceFanoutMetadata: Option[SpaceMetadata] = None\n\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n\n  override val statsReceiver: StatsReceiver =\n    statsScoped.scope(\"ScheduledSpaceSubscriberCandidate\")\n}\n\ncase class ScheduledSpaceSubscriberCandidatePredicates(config: Config)\n    extends BasicSendHandlerPredicates[ScheduledSpaceSubscriberPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val preCandidateSpecificPredicates: List[\n    NamedPredicate[ScheduledSpaceSubscriberPushCandidate]\n  ] =\n    List(\n      PredicatesForCandidate.paramPredicate(FeatureSwitchParams.EnableScheduledSpaceSubscribers),\n      SpacePredicate.narrowCastSpace,\n      SpacePredicate.targetInSpace(config.audioSpaceParticipantsStore),\n      SpacePredicate.spaceHostTargetUserBlocking(config.edgeStore),\n      PredicatesForCandidate.duplicateSpacesPredicate\n    )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/SubscribedSearchTweetPushCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.SubscribedSearchTweetCandidate\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.SubscribedSearchTweetIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.SubscribedSearchTweetNtabRequestHydrator\nimport com.twitter.frigate.pushservice.take.predicates.BasicTweetPredicatesForRFPH\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.stitch.tweetypie.TweetyPie\nimport com.twitter.util.Future\n\nclass SubscribedSearchTweetPushCandidate(\n  candidate: RawCandidate with SubscribedSearchTweetCandidate,\n  author: Option[User],\n  copyIds: CopyIds\n)(\n  implicit stats: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with SubscribedSearchTweetCandidate\n    with TweetAuthorDetails\n    with SubscribedSearchTweetIbis2Hydrator\n    with SubscribedSearchTweetNtabRequestHydrator {\n  override def tweetAuthor: Future[Option[User]] = Future.value(author)\n\n  override def weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n\n  override def tweetId: Long = candidate.tweetId\n\n  override def pushCopyId: Option[Int] = copyIds.pushCopyId\n\n  override def ntabCopyId: Option[Int] = copyIds.ntabCopyId\n\n  override def copyAggregationId: Option[String] = copyIds.aggregationId\n\n  override def target: PushTypes.Target = candidate.target\n\n  override def searchTerm: String = candidate.searchTerm\n\n  override def timeBoundedLandingUrl: Option[String] = None\n\n  override def statsReceiver: StatsReceiver = stats\n\n  override def tweetyPieResult: Option[TweetyPie.TweetyPieResult] = candidate.tweetyPieResult\n}\n\ncase class SubscribedSearchTweetCandidatePredicates(override val config: Config)\n    extends BasicTweetPredicatesForRFPH[SubscribedSearchTweetPushCandidate] {\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/TopTweetImpressionsPushCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TopTweetImpressionsCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.TopTweetImpressionsCandidateIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.TopTweetImpressionsNTabRequestHydrator\nimport com.twitter.frigate.pushservice.predicate.TopTweetImpressionsPredicates\nimport com.twitter.frigate.pushservice.take.predicates.BasicTweetPredicatesForRFPH\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.notificationservice.thriftscala.StoryContext\nimport com.twitter.notificationservice.thriftscala.StoryContextValue\nimport com.twitter.stitch.tweetypie.TweetyPie\n\n/**\n * This class defines a hydrated [[TopTweetImpressionsCandidate]]\n *\n * @param candidate: [[TopTweetImpressionsCandidate]] for the candidate representing the user's Tweet with the most impressions\n * @param copyIds: push and ntab notification copy\n * @param stats: finagle scoped states receiver\n * @param pushModelScorer: ML model score object for fetching prediction scores\n */\nclass TopTweetImpressionsPushCandidate(\n  candidate: RawCandidate with TopTweetImpressionsCandidate,\n  copyIds: CopyIds\n)(\n  implicit stats: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with TopTweetImpressionsCandidate\n    with TopTweetImpressionsNTabRequestHydrator\n    with TopTweetImpressionsCandidateIbis2Hydrator {\n  override val target: PushTypes.Target = candidate.target\n  override val commonRecType: CommonRecommendationType = candidate.commonRecType\n  override val tweetId: Long = candidate.tweetId\n  override lazy val tweetyPieResult: Option[TweetyPie.TweetyPieResult] =\n    candidate.tweetyPieResult\n  override val impressionsCount: Long = candidate.impressionsCount\n\n  override val statsReceiver: StatsReceiver = stats.scope(getClass.getSimpleName)\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n  override val storyContext: Option[StoryContext] =\n    Some(StoryContext(altText = \"\", value = Some(StoryContextValue.Tweets(Seq(tweetId)))))\n}\n\ncase class TopTweetImpressionsPushCandidatePredicates(config: Config)\n    extends BasicTweetPredicatesForRFPH[TopTweetImpressionsPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val preCandidateSpecificPredicates: List[\n    NamedPredicate[TopTweetImpressionsPushCandidate]\n  ] = List(\n    TopTweetImpressionsPredicates.topTweetImpressionsFatiguePredicate\n  )\n\n  override val postCandidateSpecificPredicates: List[\n    NamedPredicate[TopTweetImpressionsPushCandidate]\n  ] = List(\n    TopTweetImpressionsPredicates.topTweetImpressionsThreshold()\n  )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/TopicProofTweetPushCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TopicProofTweetCandidate\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.TopicProofTweetIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.TopicProofTweetNtabRequestHydrator\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.predicate.PredicatesForCandidate\nimport com.twitter.frigate.pushservice.take.predicates.BasicTweetPredicatesForRFPH\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.stitch.tweetypie.TweetyPie\nimport com.twitter.util.Future\n\n/**\n * This class defines a hydrated [[TopicProofTweetCandidate]]\n *\n * @param candidate       : [[TopicProofTweetCandidate]] for the candidate representint a Tweet recommendation for followed Topic\n * @param author          : Tweet author representated as Gizmoduck user object\n * @param copyIds         : push and ntab notification copy\n * @param stats           : finagle scoped states receiver\n * @param pushModelScorer : ML model score object for fetching prediction scores\n */\nclass TopicProofTweetPushCandidate(\n  candidate: RawCandidate with TopicProofTweetCandidate,\n  author: Option[User],\n  copyIds: CopyIds\n)(\n  implicit stats: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with TopicProofTweetCandidate\n    with TweetAuthorDetails\n    with TopicProofTweetNtabRequestHydrator\n    with TopicProofTweetIbis2Hydrator {\n  override val statsReceiver: StatsReceiver = stats\n  override val target: PushTypes.Target = candidate.target\n  override val tweetId: Long = candidate.tweetId\n  override lazy val tweetyPieResult: Option[TweetyPie.TweetyPieResult] = candidate.tweetyPieResult\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n  override val semanticCoreEntityId = candidate.semanticCoreEntityId\n  override val localizedUttEntity = candidate.localizedUttEntity\n  override val tweetAuthor = Future.value(author)\n  override val topicListingSetting = candidate.topicListingSetting\n  override val algorithmCR = candidate.algorithmCR\n  override val commonRecType: CommonRecommendationType = candidate.commonRecType\n  override val tagsCR = candidate.tagsCR\n  override val isOutOfNetwork = candidate.isOutOfNetwork\n}\n\ncase class TopicProofTweetCandidatePredicates(override val config: Config)\n    extends BasicTweetPredicatesForRFPH[TopicProofTweetPushCandidate] {\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val preCandidateSpecificPredicates: List[NamedPredicate[TopicProofTweetPushCandidate]] =\n    List(\n      PredicatesForCandidate.paramPredicate(\n        PushFeatureSwitchParams.EnableTopicProofTweetRecs\n      ),\n    )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/TrendTweetPushCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.events.recos.thriftscala.TrendsContext\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TrendTweetCandidate\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.TrendTweetIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.TrendTweetNtabHydrator\nimport com.twitter.frigate.pushservice.take.predicates.BasicTweetPredicatesForRFPH\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.stitch.tweetypie.TweetyPie\nimport com.twitter.util.Future\n\nclass TrendTweetPushCandidate(\n  candidate: RawCandidate with TrendTweetCandidate,\n  author: Option[User],\n  copyIds: CopyIds\n)(\n  implicit stats: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with TrendTweetCandidate\n    with TweetAuthorDetails\n    with TrendTweetIbis2Hydrator\n    with TrendTweetNtabHydrator {\n  override val statsReceiver: StatsReceiver = stats\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n  override val tweetId: Long = candidate.tweetId\n  override lazy val tweetyPieResult: Option[TweetyPie.TweetyPieResult] = candidate.tweetyPieResult\n  override lazy val tweetAuthor: Future[Option[User]] = Future.value(author)\n  override val target: PushTypes.Target = candidate.target\n  override val landingUrl: String = candidate.landingUrl\n  override val timeBoundedLandingUrl: Option[String] = candidate.timeBoundedLandingUrl\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n  override val trendId: String = candidate.trendId\n  override val trendName: String = candidate.trendName\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n  override val context: TrendsContext = candidate.context\n}\n\ncase class TrendTweetPredicates(override val config: Config)\n    extends BasicTweetPredicatesForRFPH[TrendTweetPushCandidate] {\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/TripTweetPushCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.OutOfNetworkTweetCandidate\nimport com.twitter.frigate.common.base.TopicCandidate\nimport com.twitter.frigate.common.base.TripCandidate\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.OutOfNetworkTweetIbis2HydratorForCandidate\nimport com.twitter.frigate.pushservice.model.ntab.OutOfNetworkTweetNTabRequestHydrator\nimport com.twitter.frigate.pushservice.take.predicates.OutOfNetworkTweetPredicates\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.stitch.tweetypie.TweetyPie\nimport com.twitter.topiclisting.utt.LocalizedEntity\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripDomain\nimport com.twitter.util.Future\n\nclass TripTweetPushCandidate(\n  candidate: RawCandidate with OutOfNetworkTweetCandidate with TripCandidate,\n  author: Future[Option[User]],\n  copyIds: CopyIds\n)(\n  implicit stats: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with TripCandidate\n    with TopicCandidate\n    with OutOfNetworkTweetCandidate\n    with TweetAuthorDetails\n    with OutOfNetworkTweetNTabRequestHydrator\n    with OutOfNetworkTweetIbis2HydratorForCandidate {\n  override val statsReceiver: StatsReceiver = stats\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n  override val tweetId: Long = candidate.tweetId\n  override lazy val tweetyPieResult: Option[TweetyPie.TweetyPieResult] =\n    candidate.tweetyPieResult\n  override lazy val tweetAuthor: Future[Option[User]] = author\n  override val target: PushTypes.Target = candidate.target\n  override lazy val commonRecType: CommonRecommendationType =\n    candidate.commonRecType\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n  override lazy val semanticCoreEntityId: Option[Long] = None\n  override lazy val localizedUttEntity: Option[LocalizedEntity] = None\n  override lazy val algorithmCR: Option[String] = None\n  override val tripDomain: Option[collection.Set[TripDomain]] = candidate.tripDomain\n}\n\ncase class TripTweetCandidatePredicates(override val config: Config)\n    extends OutOfNetworkTweetPredicates[TripTweetPushCandidate] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/TweetAction.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.SocialContextActions\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.base.TweetDetails\nimport com.twitter.frigate.pushservice.model.PushTypes._\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.predicate._\nimport com.twitter.frigate.pushservice.take.predicates.BasicTweetPredicatesForRFPH\n\ncase class TweetActionCandidatePredicates(override val config: Config)\n    extends BasicTweetPredicatesForRFPH[\n      PushCandidate with TweetCandidate with TweetDetails with SocialContextActions\n    ] {\n\n  implicit val statsReceiver: StatsReceiver = config.statsReceiver.scope(getClass.getSimpleName)\n\n  override val preCandidateSpecificPredicates = List(PredicatesForCandidate.minSocialContext(1))\n\n  override val postCandidateSpecificPredicates = List(\n    PredicatesForCandidate.socialContextBeingFollowed(config.edgeStore),\n    PredicatesForCandidate.socialContextBlockingOrMuting(config.edgeStore),\n    PredicatesForCandidate.socialContextNotRetweetFollowing(config.edgeStore)\n  )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/TweetFavorite.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.SocialContextAction\nimport com.twitter.frigate.common.base.SocialContextUserDetails\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetFavoriteCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.TweetFavoriteCandidateIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.TweetFavoriteNTabRequestHydrator\nimport com.twitter.frigate.pushservice.util.CandidateHydrationUtil.TweetWithSocialContextTraits\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.stitch.tweetypie.TweetyPie\nimport com.twitter.util.Future\n\nclass TweetFavoritePushCandidate(\n  candidate: RawCandidate with TweetWithSocialContextTraits,\n  socialContextUserMap: Future[Map[Long, Option[User]]],\n  author: Future[Option[User]],\n  copyIds: CopyIds\n)(\n  implicit stats: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with TweetFavoriteCandidate\n    with SocialContextUserDetails\n    with TweetAuthorDetails\n    with TweetFavoriteNTabRequestHydrator\n    with TweetFavoriteCandidateIbis2Hydrator {\n  override val statsReceiver: StatsReceiver = stats\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n  override val tweetId: Long = candidate.tweetId\n  override val socialContextActions: Seq[SocialContextAction] =\n    candidate.socialContextActions\n\n  override val socialContextAllTypeActions: Seq[SocialContextAction] =\n    candidate.socialContextAllTypeActions\n\n  override lazy val scUserMap: Future[Map[Long, Option[User]]] = socialContextUserMap\n  override lazy val tweetAuthor: Future[Option[User]] = author\n  override lazy val commonRecType: CommonRecommendationType =\n    candidate.commonRecType\n  override val target: PushTypes.Target = candidate.target\n  override lazy val tweetyPieResult: Option[TweetyPie.TweetyPieResult] =\n    candidate.tweetyPieResult\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/TweetRetweet.scala",
    "content": "package com.twitter.frigate.pushservice.model\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.SocialContextAction\nimport com.twitter.frigate.common.base.SocialContextUserDetails\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetRetweetCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.model.ibis.TweetRetweetCandidateIbis2Hydrator\nimport com.twitter.frigate.pushservice.model.ntab.TweetRetweetNTabRequestHydrator\nimport com.twitter.frigate.pushservice.util.CandidateHydrationUtil.TweetWithSocialContextTraits\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.stitch.tweetypie.TweetyPie\nimport com.twitter.util.Future\n\nclass TweetRetweetPushCandidate(\n  candidate: RawCandidate with TweetWithSocialContextTraits,\n  socialContextUserMap: Future[Map[Long, Option[User]]],\n  author: Future[Option[User]],\n  copyIds: CopyIds\n)(\n  implicit stats: StatsReceiver,\n  pushModelScorer: PushMLModelScorer)\n    extends PushCandidate\n    with TweetRetweetCandidate\n    with SocialContextUserDetails\n    with TweetAuthorDetails\n    with TweetRetweetNTabRequestHydrator\n    with TweetRetweetCandidateIbis2Hydrator {\n  override val statsReceiver: StatsReceiver = stats\n  override val weightedOpenOrNtabClickModelScorer: PushMLModelScorer = pushModelScorer\n  override val tweetId: Long = candidate.tweetId\n  override val socialContextActions: Seq[SocialContextAction] =\n    candidate.socialContextActions\n\n  override val socialContextAllTypeActions: Seq[SocialContextAction] =\n    candidate.socialContextAllTypeActions\n\n  override lazy val scUserMap: Future[Map[Long, Option[User]]] = socialContextUserMap\n  override lazy val tweetAuthor: Future[Option[User]] = author\n  override lazy val commonRecType: CommonRecommendationType = candidate.commonRecType\n  override val target: PushTypes.Target = candidate.target\n  override lazy val tweetyPieResult: Option[TweetyPie.TweetyPieResult] = candidate.tweetyPieResult\n  override val pushCopyId: Option[Int] = copyIds.pushCopyId\n  override val ntabCopyId: Option[Int] = copyIds.ntabCopyId\n  override val copyAggregationId: Option[String] = copyIds.aggregationId\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/candidate/CopyInfo.scala",
    "content": "package com.twitter.frigate.pushservice.model.candidate\n\nimport com.twitter.frigate.common.util.MRPushCopy\nimport com.twitter.frigate.common.util.MrPushCopyObjects\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.util.CandidateUtil\n\ncase class CopyIds(\n  pushCopyId: Option[Int] = None,\n  ntabCopyId: Option[Int] = None,\n  aggregationId: Option[String] = None)\n\ntrait CopyInfo {\n  self: PushCandidate =>\n\n  import com.twitter.frigate.data_pipeline.common.FrigateNotificationUtil._\n\n  def getPushCopy: Option[MRPushCopy] =\n    pushCopyId match {\n      case Some(pushCopyId) => MrPushCopyObjects.getCopyFromId(pushCopyId)\n      case _ =>\n        crt2PushCopy(\n          commonRecType,\n          CandidateUtil.getSocialContextActionsFromCandidate(self).size\n        )\n    }\n\n  def pushCopyId: Option[Int]\n\n  def ntabCopyId: Option[Int]\n\n  def copyAggregationId: Option[String]\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/candidate/MLScores.scala",
    "content": "package com.twitter.frigate.pushservice.model.candidate\n\nimport com.twitter.frigate.common.base.FeatureMap\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.ml.HydrationContextBuilder\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushMLModel\nimport com.twitter.frigate.pushservice.params.WeightedOpenOrNtabClickModel\nimport com.twitter.nrel.hydration.push.HydrationContext\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.util.Future\nimport java.util.concurrent.ConcurrentHashMap\nimport scala.collection.concurrent.{Map => CMap}\nimport scala.collection.convert.decorateAsScala._\n\ntrait MLScores {\n\n  self: PushCandidate =>\n\n  lazy val candidateHydrationContext: Future[HydrationContext] = HydrationContextBuilder.build(self)\n\n  def weightedOpenOrNtabClickModelScorer: PushMLModelScorer\n\n  // Used to store the scores and avoid duplicate prediction\n  private val qualityModelScores: CMap[\n    (PushMLModel.Value, WeightedOpenOrNtabClickModel.ModelNameType),\n    Future[Option[Double]]\n  ] =\n    new ConcurrentHashMap[(PushMLModel.Value, WeightedOpenOrNtabClickModel.ModelNameType), Future[\n      Option[Double]\n    ]]().asScala\n\n  def populateQualityModelScore(\n    pushMLModel: PushMLModel.Value,\n    modelVersion: WeightedOpenOrNtabClickModel.ModelNameType,\n    prob: Future[Option[Double]]\n  ) = {\n    val modelAndVersion = (pushMLModel, modelVersion)\n    if (!qualityModelScores.contains(modelAndVersion)) {\n      qualityModelScores += modelAndVersion -> prob\n    }\n  }\n\n  // The ML scores that also depend on other candidates and are only available after all candidates are processed\n  // For example, the likelihood info for Importance Sampling\n  private lazy val crossCandidateMlScores: CMap[String, Double] =\n    new ConcurrentHashMap[String, Double]().asScala\n\n  def populateCrossCandidateMlScores(scoreName: String, score: Double): Unit = {\n    if (crossCandidateMlScores.contains(scoreName)) {\n      throw new Exception(\n        s\"$scoreName has been populated in the CrossCandidateMlScores!\\n\" +\n          s\"Existing crossCandidateMlScores are ${crossCandidateMlScores}\\n\"\n      )\n    }\n    crossCandidateMlScores += scoreName -> score\n  }\n\n  def getMLModelScore(\n    pushMLModel: PushMLModel.Value,\n    modelVersion: WeightedOpenOrNtabClickModel.ModelNameType\n  ): Future[Option[Double]] = {\n    qualityModelScores.getOrElseUpdate(\n      (pushMLModel, modelVersion),\n      weightedOpenOrNtabClickModelScorer\n        .singlePredicationForModelVersion(modelVersion, self, Some(pushMLModel))\n    )\n  }\n\n  def getMLModelScoreWithoutUpdate(\n    pushMLModel: PushMLModel.Value,\n    modelVersion: WeightedOpenOrNtabClickModel.ModelNameType\n  ): Future[Option[Double]] = {\n    qualityModelScores.getOrElse(\n      (pushMLModel, modelVersion),\n      Future.None\n    )\n  }\n\n  def getWeightedOpenOrNtabClickModelScore(\n    weightedOONCModelParam: FSParam[WeightedOpenOrNtabClickModel.ModelNameType]\n  ): Future[Option[Double]] = {\n    getMLModelScore(\n      PushMLModel.WeightedOpenOrNtabClickProbability,\n      target.params(weightedOONCModelParam)\n    )\n  }\n\n  /* After we unify the ranking and filtering models, we follow the iteration process below\n     When improving the WeightedOONC model,\n     1) Run experiment which only replace the ranking model\n     2) Make decisions according to the experiment results\n     3) Use the ranking model for filtering\n     4) Adjust percentile thresholds if necessary\n   */\n  lazy val mrWeightedOpenOrNtabClickRankingProbability: Future[Option[Double]] =\n    target.rankingModelParam.flatMap { modelParam =>\n      getWeightedOpenOrNtabClickModelScore(modelParam)\n    }\n\n  def getBigFilteringScore(\n    pushMLModel: PushMLModel.Value,\n    modelVersion: WeightedOpenOrNtabClickModel.ModelNameType\n  ): Future[Option[Double]] = {\n    mrWeightedOpenOrNtabClickRankingProbability.flatMap {\n      case Some(rankingScore) =>\n        // Adds ranking score to feature map (we must ensure the feature key is also in the feature context)\n        mergeFeatures(\n          FeatureMap(\n            numericFeatures = Map(\"scribe.WeightedOpenOrNtabClickProbability\" -> rankingScore)\n          )\n        )\n        getMLModelScore(pushMLModel, modelVersion)\n      case _ => Future.None\n    }\n  }\n\n  def getWeightedOpenOrNtabClickScoreForScribing(): Seq[Future[Map[String, Double]]] = {\n    Seq(\n      mrWeightedOpenOrNtabClickRankingProbability.map {\n        case Some(score) => Map(PushMLModel.WeightedOpenOrNtabClickProbability.toString -> score)\n        case _ => Map.empty[String, Double]\n      },\n      Future\n        .join(\n          target.rankingModelParam,\n          mrWeightedOpenOrNtabClickRankingProbability\n        ).map {\n          case (rankingModelParam, Some(score)) =>\n            Map(target.params(rankingModelParam).toString -> score)\n          case _ => Map.empty[String, Double]\n        }\n    )\n  }\n\n  def getNsfwScoreForScribing(): Seq[Future[Map[String, Double]]] = {\n    val nsfwScoreFut = getMLModelScoreWithoutUpdate(\n      PushMLModel.HealthNsfwProbability,\n      target.params(PushFeatureSwitchParams.BqmlHealthModelTypeParam))\n    Seq(nsfwScoreFut.map { nsfwScoreOpt =>\n      nsfwScoreOpt\n        .map(nsfwScore => Map(PushMLModel.HealthNsfwProbability.toString -> nsfwScore)).getOrElse(\n          Map.empty[String, Double])\n    })\n  }\n\n  def getBigFilteringSupervisedScoresForScribing(): Seq[Future[Map[String, Double]]] = {\n    if (target.params(\n        PushFeatureSwitchParams.EnableMrRequestScribingBigFilteringSupervisedScores)) {\n      Seq(\n        mrBigFilteringSupervisedSendingScore.map {\n          case Some(score) =>\n            Map(PushMLModel.BigFilteringSupervisedSendingModel.toString -> score)\n          case _ => Map.empty[String, Double]\n        },\n        mrBigFilteringSupervisedWithoutSendingScore.map {\n          case Some(score) =>\n            Map(PushMLModel.BigFilteringSupervisedWithoutSendingModel.toString -> score)\n          case _ => Map.empty[String, Double]\n        }\n      )\n    } else Seq.empty[Future[Map[String, Double]]]\n  }\n\n  def getBigFilteringRLScoresForScribing(): Seq[Future[Map[String, Double]]] = {\n    if (target.params(PushFeatureSwitchParams.EnableMrRequestScribingBigFilteringRLScores)) {\n      Seq(\n        mrBigFilteringRLSendingScore.map {\n          case Some(score) => Map(PushMLModel.BigFilteringRLSendingModel.toString -> score)\n          case _ => Map.empty[String, Double]\n        },\n        mrBigFilteringRLWithoutSendingScore.map {\n          case Some(score) => Map(PushMLModel.BigFilteringRLWithoutSendingModel.toString -> score)\n          case _ => Map.empty[String, Double]\n        }\n      )\n    } else Seq.empty[Future[Map[String, Double]]]\n  }\n\n  def buildModelScoresSeqForScribing(): Seq[Future[Map[String, Double]]] = {\n    getWeightedOpenOrNtabClickScoreForScribing() ++\n      getBigFilteringSupervisedScoresForScribing() ++\n      getBigFilteringRLScoresForScribing() ++\n      getNsfwScoreForScribing()\n  }\n\n  lazy val mrBigFilteringSupervisedSendingScore: Future[Option[Double]] =\n    getBigFilteringScore(\n      PushMLModel.BigFilteringSupervisedSendingModel,\n      target.params(PushFeatureSwitchParams.BigFilteringSupervisedSendingModelParam)\n    )\n\n  lazy val mrBigFilteringSupervisedWithoutSendingScore: Future[Option[Double]] =\n    getBigFilteringScore(\n      PushMLModel.BigFilteringSupervisedWithoutSendingModel,\n      target.params(PushFeatureSwitchParams.BigFilteringSupervisedWithoutSendingModelParam)\n    )\n\n  lazy val mrBigFilteringRLSendingScore: Future[Option[Double]] =\n    getBigFilteringScore(\n      PushMLModel.BigFilteringRLSendingModel,\n      target.params(PushFeatureSwitchParams.BigFilteringRLSendingModelParam)\n    )\n\n  lazy val mrBigFilteringRLWithoutSendingScore: Future[Option[Double]] =\n    getBigFilteringScore(\n      PushMLModel.BigFilteringRLWithoutSendingModel,\n      target.params(PushFeatureSwitchParams.BigFilteringRLWithoutSendingModelParam)\n    )\n\n  lazy val mrWeightedOpenOrNtabClickFilteringProbability: Future[Option[Double]] =\n    getWeightedOpenOrNtabClickModelScore(\n      target.filteringModelParam\n    )\n\n  lazy val mrQualityUprankingProbability: Future[Option[Double]] =\n    getMLModelScore(\n      PushMLModel.FilteringProbability,\n      target.params(PushFeatureSwitchParams.QualityUprankingModelTypeParam)\n    )\n\n  lazy val mrNsfwScore: Future[Option[Double]] =\n    getMLModelScoreWithoutUpdate(\n      PushMLModel.HealthNsfwProbability,\n      target.params(PushFeatureSwitchParams.BqmlHealthModelTypeParam)\n    )\n\n  // MR quality upranking param\n  private val qualityUprankingBoost: String = \"QualityUprankingBoost\"\n  private val producerQualityUprankingBoost: String = \"ProducerQualityUprankingBoost\"\n  private val qualityUprankingInfo: CMap[String, Double] =\n    new ConcurrentHashMap[String, Double]().asScala\n\n  lazy val mrQualityUprankingBoost: Option[Double] =\n    qualityUprankingInfo.get(qualityUprankingBoost)\n  lazy val mrProducerQualityUprankingBoost: Option[Double] =\n    qualityUprankingInfo.get(producerQualityUprankingBoost)\n\n  def setQualityUprankingBoost(boost: Double) =\n    if (qualityUprankingInfo.contains(qualityUprankingBoost)) {\n      qualityUprankingInfo(qualityUprankingBoost) = boost\n    } else {\n      qualityUprankingInfo += qualityUprankingBoost -> boost\n    }\n  def setProducerQualityUprankingBoost(boost: Double) =\n    if (qualityUprankingInfo.contains(producerQualityUprankingBoost)) {\n      qualityUprankingInfo(producerQualityUprankingBoost) = boost\n    } else {\n      qualityUprankingInfo += producerQualityUprankingBoost -> boost\n    }\n\n  private lazy val mrModelScoresFut: Future[Map[String, Double]] = {\n    if (self.target.isLoggedOutUser) {\n      Future.value(Map.empty[String, Double])\n    } else {\n      Future\n        .collectToTry {\n          buildModelScoresSeqForScribing()\n        }.map { scoreTrySeq =>\n          scoreTrySeq\n            .collect {\n              case result if result.isReturn => result.get()\n            }.reduce(_ ++ _)\n        }\n    }\n  }\n\n  // Internal model scores (scores that are independent of other candidates) for scribing\n  lazy val modelScores: Future[Map[String, Double]] =\n    target.dauProbability.flatMap { dauProbabilityOpt =>\n      val dauProbScoreMap = dauProbabilityOpt\n        .map(_.probability).map { dauProb =>\n          PushMLModel.DauProbability.toString -> dauProb\n        }.toMap\n\n      // Avoid unnecessary MR model scribing\n      if (target.isDarkWrite) {\n        mrModelScoresFut.map(dauProbScoreMap ++ _)\n      } else if (RecTypes.isSendHandlerType(commonRecType) && !RecTypes\n          .sendHandlerTypesUsingMrModel(commonRecType)) {\n        Future.value(dauProbScoreMap)\n      } else {\n        mrModelScoresFut.map(dauProbScoreMap ++ _)\n      }\n    }\n\n  // We will scribe both internal ML scores and cross-Candidate scores\n  def getModelScoresforScribing(): Future[Map[String, Double]] = {\n    if (RecTypes.notEligibleForModelScoreTracking(commonRecType) || self.target.isLoggedOutUser) {\n      Future.value(Map.empty[String, Double])\n    } else {\n      modelScores.map { internalScores =>\n        if (internalScores.keySet.intersect(crossCandidateMlScores.keySet).nonEmpty) {\n          throw new Exception(\n            \"crossCandidateMlScores overlap internalModelScores\\n\" +\n              s\"internalScores keySet: ${internalScores.keySet}\\n\" +\n              s\"crossCandidateScores keySet: ${crossCandidateMlScores.keySet}\\n\"\n          )\n        }\n\n        internalScores ++ crossCandidateMlScores\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/candidate/QualityScribing.scala",
    "content": "package com.twitter.frigate.pushservice.model.candidate\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.HighQualityScribingScores\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushMLModel\nimport com.twitter.util.Future\nimport java.util.concurrent.ConcurrentHashMap\nimport scala.collection.concurrent.{Map => CMap}\nimport scala.collection.convert.decorateAsScala._\n\ntrait QualityScribing {\n  self: PushCandidate with MLScores =>\n\n  // Use to store other scores (to avoid duplicate queries to other services, e.g. HSS)\n  private val externalCachedScores: CMap[String, Future[Option[Double]]] =\n    new ConcurrentHashMap[String, Future[Option[Double]]]().asScala\n\n  /**\n   * Retrieves the model version as specified by the corresponding FS param.\n   * This model version will be used for getting the cached score or triggering\n   * a prediction request.\n   *\n   * @param modelName The score we will like to scribe\n   */\n  private def getModelVersion(\n    modelName: HighQualityScribingScores.Name\n  ): String = {\n    modelName match {\n      case HighQualityScribingScores.HeavyRankingScore =>\n        target.params(PushFeatureSwitchParams.HighQualityCandidatesHeavyRankingModel)\n      case HighQualityScribingScores.NonPersonalizedQualityScoreUsingCnn =>\n        target.params(PushFeatureSwitchParams.HighQualityCandidatesNonPersonalizedQualityCnnModel)\n      case HighQualityScribingScores.BqmlNsfwScore =>\n        target.params(PushFeatureSwitchParams.HighQualityCandidatesBqmlNsfwModel)\n      case HighQualityScribingScores.BqmlReportScore =>\n        target.params(PushFeatureSwitchParams.HighQualityCandidatesBqmlReportModel)\n    }\n  }\n\n  /**\n   * Retrieves the score for scribing either from a cached value or\n   * by generating a prediction request. This will increase model QPS\n   *\n   * @param pushMLModel This represents the prefix of the model name (i.e. [pushMLModel]_[version])\n   * @param scoreName   The name to be use when scribing this score\n   */\n  def getScribingScore(\n    pushMLModel: PushMLModel.Value,\n    scoreName: HighQualityScribingScores.Name\n  ): Future[(String, Option[Double])] = {\n    getMLModelScore(\n      pushMLModel,\n      getModelVersion(scoreName)\n    ).map { scoreOpt =>\n      scoreName.toString -> scoreOpt\n    }\n  }\n\n  /**\n   * Retrieves the score for scribing if it has been computed/cached before otherwise\n   * it will return Future.None\n   *\n   * @param pushMLModel This represents the prefix of the model name (i.e. [pushMLModel]_[version])\n   * @param scoreName   The name to be use when scribing this score\n   */\n  def getScribingScoreWithoutUpdate(\n    pushMLModel: PushMLModel.Value,\n    scoreName: HighQualityScribingScores.Name\n  ): Future[(String, Option[Double])] = {\n    getMLModelScoreWithoutUpdate(\n      pushMLModel,\n      getModelVersion(scoreName)\n    ).map { scoreOpt =>\n      scoreName.toString -> scoreOpt\n    }\n  }\n\n  /**\n   * Caches the given score future\n   *\n   * @param scoreName The name to be use when scribing this score\n   * @param scoreFut Future mapping scoreName -> scoreOpt\n   */\n  def cacheExternalScore(scoreName: String, scoreFut: Future[Option[Double]]) = {\n    if (!externalCachedScores.contains(scoreName)) {\n      externalCachedScores += scoreName -> scoreFut\n    }\n  }\n\n  /**\n   * Returns all external scores future cached as a sequence\n   */\n  def getExternalCachedScores: Seq[Future[(String, Option[Double])]] = {\n    externalCachedScores.map {\n      case (modelName, scoreFut) =>\n        scoreFut.map { scoreOpt => modelName -> scoreOpt }\n    }.toSeq\n  }\n\n  def getExternalCachedScoreByName(name: String): Future[Option[Double]] = {\n    externalCachedScores.getOrElse(name, Future.None)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/candidate/Scriber.scala",
    "content": "package com.twitter.frigate.pushservice.model.candidate\n\nimport com.twitter.frigate.data_pipeline.features_common.PushQualityModelFeatureContext.featureContext\nimport com.twitter.frigate.data_pipeline.features_common.PushQualityModelUtil\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.common.util.NotificationScribeUtil\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.OutOfNetworkTweetPushCandidate\nimport com.twitter.frigate.pushservice.model.TopicProofTweetPushCandidate\nimport com.twitter.frigate.pushservice.ml.HydrationContextBuilder\nimport com.twitter.frigate.pushservice.predicate.quality_model_predicate.PDauCohort\nimport com.twitter.frigate.pushservice.predicate.quality_model_predicate.PDauCohortUtil\nimport com.twitter.frigate.pushservice.util.Candidate2FrigateNotification\nimport com.twitter.frigate.pushservice.util.MediaAnnotationsUtil.sensitiveMediaCategoryFeatureName\nimport com.twitter.frigate.scribe.thriftscala.FrigateNotificationScribeType\nimport com.twitter.frigate.scribe.thriftscala.NotificationScribe\nimport com.twitter.frigate.scribe.thriftscala.PredicateDetailedInfo\nimport com.twitter.frigate.scribe.thriftscala.PushCapInfo\nimport com.twitter.frigate.thriftscala.ChannelName\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.frigate.thriftscala.OverrideInfo\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.model.user_state.UserState.UserState\nimport com.twitter.ibis2.service.thriftscala.Ibis2Response\nimport com.twitter.ml.api.util.ScalaToJavaDataRecordConversions\nimport com.twitter.nrel.heavyranker.FeatureHydrator\nimport com.twitter.util.Future\nimport java.util.UUID\nimport java.util.concurrent.ConcurrentHashMap\nimport scala.collection.concurrent.{Map => CMap}\nimport scala.collection.Map\nimport scala.collection.convert.decorateAsScala._\n\ntrait Scriber {\n  self: PushCandidate =>\n\n  def statsReceiver: StatsReceiver\n\n  def frigateNotification: FrigateNotification = Candidate2FrigateNotification\n    .getFrigateNotification(self)(statsReceiver)\n    .copy(copyAggregationId = self.copyAggregationId)\n\n  lazy val impressionId: String = UUID.randomUUID.toString.replaceAll(\"-\", \"\")\n\n  // Used to store the score and threshold for predicates\n  // Map(predicate name, (score, threshold, filter?))\n  private val predicateScoreAndThreshold: CMap[String, PredicateDetailedInfo] =\n    new ConcurrentHashMap[String, PredicateDetailedInfo]().asScala\n\n  def cachePredicateInfo(\n    predName: String,\n    predScore: Double,\n    predThreshold: Double,\n    predResult: Boolean,\n    additionalInformation: Option[Map[String, Double]] = None\n  ) = {\n    if (!predicateScoreAndThreshold.contains(predName)) {\n      predicateScoreAndThreshold += predName -> PredicateDetailedInfo(\n        predName,\n        predScore,\n        predThreshold,\n        predResult,\n        additionalInformation)\n    }\n  }\n\n  def getCachedPredicateInfo(): Seq[PredicateDetailedInfo] = predicateScoreAndThreshold.values.toSeq\n\n  def frigateNotificationForPersistence(\n    channels: Seq[ChannelName],\n    isSilentPush: Boolean,\n    overrideInfoOpt: Option[OverrideInfo] = None,\n    copyFeaturesList: Set[String]\n  ): Future[FrigateNotification] = {\n\n    // record display location for frigate notification\n    statsReceiver\n      .scope(\"FrigateNotificationForPersistence\")\n      .scope(\"displayLocation\")\n      .counter(frigateNotification.notificationDisplayLocation.name)\n      .incr()\n\n    val getModelScores = self.getModelScoresforScribing()\n\n    Future.join(getModelScores, self.target.targetMrUserState).map {\n      case (mlScores, mrUserState) =>\n        frigateNotification.copy(\n          impressionId = Some(impressionId),\n          isSilentPush = Some(isSilentPush),\n          overrideInfo = overrideInfoOpt,\n          mlModelScores = Some(mlScores),\n          mrUserState = mrUserState.map(_.name),\n          copyFeatures = Some(copyFeaturesList.toSeq)\n        )\n    }\n  }\n  // scribe data\n  private def getNotificationScribe(\n    notifForPersistence: FrigateNotification,\n    userState: Option[UserState],\n    dauCohort: PDauCohort.Value,\n    ibis2Response: Option[Ibis2Response],\n    tweetAuthorId: Option[Long],\n    recUserId: Option[Long],\n    modelScoresMap: Option[Map[String, Double]],\n    primaryClient: Option[String],\n    isMrBackfillCR: Option[Boolean] = None,\n    tagsCR: Option[Seq[String]] = None,\n    gizmoduckTargetUser: Option[User],\n    predicateDetailedInfoList: Option[Seq[PredicateDetailedInfo]] = None,\n    pushCapInfoList: Option[Seq[PushCapInfo]] = None\n  ): NotificationScribe = {\n    NotificationScribe(\n      FrigateNotificationScribeType.SendMessage,\n      System.currentTimeMillis(),\n      targetUserId = Some(self.target.targetId),\n      timestampKeyForHistoryV2 = Some(createdAt.inSeconds),\n      sendType = NotificationScribeUtil.convertToScribeDisplayLocation(\n        self.frigateNotification.notificationDisplayLocation\n      ),\n      recommendationType = NotificationScribeUtil.convertToScribeRecommendationType(\n        self.frigateNotification.commonRecommendationType\n      ),\n      commonRecommendationType = Some(self.frigateNotification.commonRecommendationType),\n      fromPushService = Some(true),\n      frigateNotification = Some(notifForPersistence),\n      impressionId = Some(impressionId),\n      skipModelInfo = target.skipModelInfo,\n      ibis2Response = ibis2Response,\n      tweetAuthorId = tweetAuthorId,\n      scribeFeatures = Some(target.noSkipButScribeFeatures),\n      userState = userState.map(_.toString),\n      pDauCohort = Some(dauCohort.toString),\n      recommendedUserId = recUserId,\n      modelScores = modelScoresMap,\n      primaryClient = primaryClient,\n      isMrBackfillCR = isMrBackfillCR,\n      tagsCR = tagsCR,\n      targetUserType = gizmoduckTargetUser.map(_.userType),\n      predicateDetailedInfoList = predicateDetailedInfoList,\n      pushCapInfoList = pushCapInfoList\n    )\n  }\n\n  def scribeData(\n    ibis2Response: Option[Ibis2Response] = None,\n    isSilentPush: Boolean = false,\n    overrideInfoOpt: Option[OverrideInfo] = None,\n    copyFeaturesList: Set[String] = Set.empty,\n    channels: Seq[ChannelName] = Seq.empty\n  ): Future[NotificationScribe] = {\n\n    val recTweetAuthorId = self match {\n      case t: TweetCandidate with TweetAuthor => t.authorId\n      case _ => None\n    }\n\n    val recUserId = self match {\n      case u: UserCandidate => Some(u.userId)\n      case _ => None\n    }\n\n    val isMrBackfillCR = self match {\n      case t: OutOfNetworkTweetPushCandidate => t.isMrBackfillCR\n      case _ => None\n    }\n\n    val tagsCR = self match {\n      case t: OutOfNetworkTweetPushCandidate =>\n        t.tagsCR.map { tags =>\n          tags.map(_.toString)\n        }\n      case t: TopicProofTweetPushCandidate =>\n        t.tagsCR.map { tags =>\n          tags.map(_.toString)\n        }\n      case _ => None\n    }\n\n    Future\n      .join(\n        frigateNotificationForPersistence(\n          channels = channels,\n          isSilentPush = isSilentPush,\n          overrideInfoOpt = overrideInfoOpt,\n          copyFeaturesList = copyFeaturesList\n        ),\n        target.targetUserState,\n        PDauCohortUtil.getPDauCohort(target),\n        target.deviceInfo,\n        target.targetUser\n      )\n      .flatMap {\n        case (notifForPersistence, userState, dauCohort, deviceInfo, gizmoduckTargetUserOpt) =>\n          val primaryClient = deviceInfo.flatMap(_.guessedPrimaryClient).map(_.toString)\n          val cachedPredicateInfo =\n            if (self.target.params(PushParams.EnablePredicateDetailedInfoScribing)) {\n              Some(getCachedPredicateInfo())\n            } else None\n\n          val cachedPushCapInfo =\n            if (self.target\n                .params(PushParams.EnablePushCapInfoScribing)) {\n              Some(target.finalPushcapAndFatigue.values.toSeq)\n            } else None\n\n          val data = getNotificationScribe(\n            notifForPersistence,\n            userState,\n            dauCohort,\n            ibis2Response,\n            recTweetAuthorId,\n            recUserId,\n            notifForPersistence.mlModelScores,\n            primaryClient,\n            isMrBackfillCR,\n            tagsCR,\n            gizmoduckTargetUserOpt,\n            cachedPredicateInfo,\n            cachedPushCapInfo\n          )\n          //Don't scribe features for CRTs not eligible for ML Layer\n          if ((target.isModelTrainingData || target.scribeFeatureWithoutHydratingNewFeatures)\n            && !RecTypes.notEligibleForModelScoreTracking(self.commonRecType)) {\n            // scribe all the features for the model training data\n            self.getFeaturesForScribing.map { scribedFeatureMap =>\n              if (target.params(PushParams.EnableScribingMLFeaturesAsDataRecord) && !target.params(\n                  PushFeatureSwitchParams.EnableMrScribingMLFeaturesAsFeatureMapForStaging)) {\n                val scribedFeatureDataRecord =\n                  ScalaToJavaDataRecordConversions.javaDataRecord2ScalaDataRecord(\n                    PushQualityModelUtil.adaptToDataRecord(scribedFeatureMap, featureContext))\n                data.copy(\n                  featureDataRecord = Some(scribedFeatureDataRecord)\n                )\n              } else {\n                data.copy(features =\n                  Some(PushQualityModelUtil.convertFeatureMapToFeatures(scribedFeatureMap)))\n              }\n            }\n          } else Future.value(data)\n      }\n  }\n\n  def getFeaturesForScribing: Future[FeatureMap] = {\n    target.featureMap\n      .flatMap { targetFeatureMap =>\n        val onlineFeatureMap = targetFeatureMap ++ self\n          .candidateFeatureMap() // targetFeatureMap includes target core user history features\n\n        val filteredFeatureMap = {\n          onlineFeatureMap.copy(\n            sparseContinuousFeatures = onlineFeatureMap.sparseContinuousFeatures.filterKeys(\n              !_.equals(sensitiveMediaCategoryFeatureName))\n          )\n        }\n\n        val targetHydrationContext = HydrationContextBuilder.build(self.target)\n        val candidateHydrationContext = HydrationContextBuilder.build(self)\n\n        val featureMapFut = targetHydrationContext.join(candidateHydrationContext).flatMap {\n          case (targetContext, candidateContext) =>\n            FeatureHydrator.getFeatures(\n              candidateHydrationContext = candidateContext,\n              targetHydrationContext = targetContext,\n              onlineFeatures = filteredFeatureMap,\n              statsReceiver = statsReceiver)\n        }\n\n        featureMapFut\n      }\n  }\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/CustomConfigurationMapForIbis.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.ibis2.lib.util.JsonMarshal\nimport com.twitter.util.Future\n\ntrait CustomConfigurationMapForIbis {\n  self: PushCandidate =>\n\n  lazy val customConfigMapsJsonFut: Future[String] = {\n    customFieldsMapFut.map { customFields =>\n      JsonMarshal.toJson(customFields)\n    }\n  }\n\n  lazy val customConfigMapsFut: Future[Map[String, String]] = {\n    if (self.target.isLoggedOutUser) {\n      Future.value(Map.empty[String, String])\n    } else {\n      customConfigMapsJsonFut.map { customConfigMapsJson =>\n        Map(\"custom_config\" -> customConfigMapsJson)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/DiscoverTwitterPushIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.common.base.DiscoverTwitterCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.util.PushIbisUtil.mergeFutModelValues\nimport com.twitter.util.Future\n\ntrait DiscoverTwitterPushIbis2Hydrator extends Ibis2HydratorForCandidate {\n  self: PushCandidate with DiscoverTwitterCandidate =>\n\n  private lazy val targetModelValues: Map[String, String] = Map(\n    \"target_user\" -> target.targetId.toString\n  )\n\n  override lazy val modelValues: Future[Map[String, String]] =\n    mergeFutModelValues(super.modelValues, Future.value(targetModelValues))\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/F1FirstDegreeTweetIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.F1FirstDegree\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.util.Future\n\ntrait F1FirstDegreeTweetIbis2HydratorForCandidate\n    extends TweetCandidateIbis2Hydrator\n    with RankedSocialContextIbis2Hydrator {\n  self: PushCandidate with F1FirstDegree with TweetAuthorDetails =>\n\n  override lazy val scopedStats: StatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n\n  override lazy val tweetModelValues: Future[Map[String, String]] = {\n    for {\n      superModelValues <- super.tweetModelValues\n      tweetInlineModelValues <- tweetInlineActionModelValue\n    } yield {\n      superModelValues ++ otherModelValues ++ mediaModelValue ++ tweetInlineModelValues ++ inlineVideoMediaMap\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/Ibis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.common.util.MRPushCopy\nimport com.twitter.frigate.common.util.MrPushCopyObjects\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FS}\nimport com.twitter.ibis2.service.thriftscala.Flags\nimport com.twitter.ibis2.service.thriftscala.Ibis2Request\nimport com.twitter.ibis2.service.thriftscala.RecipientSelector\nimport com.twitter.ibis2.service.thriftscala.ResponseFlags\nimport com.twitter.util.Future\nimport scala.util.control.NoStackTrace\nimport com.twitter.ni.lib.logged_out_transform.Ibis2RequestTransform\n\nclass PushCopyIdNotFoundException(private val message: String)\n    extends Exception(message)\n    with NoStackTrace\n\nclass InvalidPushCopyIdException(private val message: String)\n    extends Exception(message)\n    with NoStackTrace\n\ntrait Ibis2HydratorForCandidate\n    extends CandidatePushCopy\n    with OverrideForIbis2Request\n    with CustomConfigurationMapForIbis {\n  self: PushCandidate =>\n\n  lazy val silentPushModelValue: Map[String, String] =\n    if (RecTypes.silentPushDefaultEnabledCrts.contains(commonRecType)) {\n      Map.empty\n    } else {\n      Map(\"is_silent_push\" -> \"true\")\n    }\n\n  private def transformRelevanceScore(\n    mlScore: Double,\n    scoreRange: Seq[Double]\n  ): Double = {\n    val (lowerBound, upperBound) = (scoreRange.head, scoreRange.last)\n    (mlScore * (upperBound - lowerBound)) + lowerBound\n  }\n\n  private def getBoundedMlScore(mlScore: Double): Double = {\n    if (RecTypes.isMagicFanoutEventType(commonRecType)) {\n      val mfScoreRange = target.params(FS.MagicFanoutRelevanceScoreRange)\n      transformRelevanceScore(mlScore, mfScoreRange)\n    } else {\n      val mrScoreRange = target.params(FS.MagicRecsRelevanceScoreRange)\n      transformRelevanceScore(mlScore, mrScoreRange)\n    }\n  }\n\n  lazy val relevanceScoreMapFut: Future[Map[String, String]] = {\n    mrWeightedOpenOrNtabClickRankingProbability.map {\n      case Some(mlScore) if target.params(FS.IncludeRelevanceScoreInIbis2Payload) =>\n        val boundedMlScore = getBoundedMlScore(mlScore)\n        Map(\"relevance_score\" -> boundedMlScore.toString)\n      case _ => Map.empty[String, String]\n    }\n  }\n\n  def customFieldsMapFut: Future[Map[String, String]] = relevanceScoreMapFut\n\n  //override is only enabled for RFPH CRT\n  def modelValues: Future[Map[String, String]] = {\n    Future.join(overrideModelValueFut, customConfigMapsFut).map {\n      case (overrideModelValue, customConfig) =>\n        overrideModelValue ++ silentPushModelValue ++ customConfig\n    }\n  }\n\n  def modelName: String = pushCopy.ibisPushModelName\n\n  def senderId: Option[Long] = None\n\n  def ibis2Request: Future[Option[Ibis2Request]] = {\n    Future.join(self.target.loggedOutMetadata, modelValues).map {\n      case (Some(metadata), modelVals) =>\n        Some(\n          Ibis2RequestTransform\n            .apply(metadata, modelName, modelVals).copy(\n              senderId = senderId,\n              flags = Some(Flags(\n                darkWrite = Some(target.isDarkWrite),\n                skipDupcheck = target.pushContext.flatMap(_.useDebugHandler),\n                responseFlags = Some(ResponseFlags(stringTelemetry = Some(true)))\n              ))\n            ))\n      case (None, modelVals) =>\n        Some(\n          Ibis2Request(\n            recipientSelector = RecipientSelector(Some(target.targetId)),\n            modelName = modelName,\n            modelValues = Some(modelVals),\n            senderId = senderId,\n            flags = Some(\n              Flags(\n                darkWrite = Some(target.isDarkWrite),\n                skipDupcheck = target.pushContext.flatMap(_.useDebugHandler),\n                responseFlags = Some(ResponseFlags(stringTelemetry = Some(true)))\n              )\n            )\n          ))\n    }\n  }\n}\n\ntrait CandidatePushCopy {\n  self: PushCandidate =>\n\n  final lazy val pushCopy: MRPushCopy =\n    pushCopyId match {\n      case Some(pushCopyId) =>\n        MrPushCopyObjects\n          .getCopyFromId(pushCopyId)\n          .getOrElse(\n            throw new InvalidPushCopyIdException(\n              s\"Invalid push copy id: $pushCopyId for ${self.commonRecType}\"))\n\n      case None =>\n        throw new PushCopyIdNotFoundException(\n          s\"PushCopy not found in frigateNotification for ${self.commonRecType}\"\n        )\n    }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/InlineActionIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.util.InlineActionUtil\nimport com.twitter.util.Future\n\ntrait InlineActionIbis2Hydrator {\n  self: PushCandidate =>\n\n  lazy val tweetInlineActionModelValue: Future[Map[String, String]] =\n    InlineActionUtil.getTweetInlineActionValue(target)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/ListIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.pushservice.model.ListRecommendationPushCandidate\nimport com.twitter.util.Future\n\ntrait ListIbis2Hydrator extends Ibis2HydratorForCandidate {\n  self: ListRecommendationPushCandidate =>\n\n  override lazy val senderId: Option[Long] = Some(0L)\n\n  override lazy val modelValues: Future[Map[String, String]] =\n    Future.join(listName, listOwnerId).map {\n      case (nameOpt, authorId) =>\n        Map(\n          \"list\" -> listId.toString,\n          \"list_name\" -> nameOpt\n            .getOrElse(\"\"),\n          \"list_author\" -> s\"${authorId.getOrElse(0L)}\"\n        )\n    }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/MagicFanoutCreatorEventIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.magic_events.thriftscala.CreatorFanoutType\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutCreatorEventPushCandidate\nimport com.twitter.frigate.pushservice.util.PushIbisUtil.mergeModelValues\nimport com.twitter.util.Future\n\ntrait MagicFanoutCreatorEventIbis2Hydrator\n    extends CustomConfigurationMapForIbis\n    with Ibis2HydratorForCandidate {\n  self: PushCandidate with MagicFanoutCreatorEventPushCandidate =>\n\n  val userMap = Map(\n    \"handle\" -> userProfile.screenName,\n    \"display_name\" -> userProfile.name\n  )\n\n  override val senderId = hydratedCreator.map(_.id)\n\n  override lazy val modelValues: Future[Map[String, String]] =\n    mergeModelValues(super.modelValues, userMap)\n\n  override val ibis2Request = creatorFanoutType match {\n    case CreatorFanoutType.UserSubscription => Future.None\n    case CreatorFanoutType.NewCreator => super.ibis2Request\n    case _ => super.ibis2Request\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/MagicFanoutNewsEventIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutEventHydratedCandidate\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.predicate.magic_fanout.MagicFanoutPredicatesUtil\nimport com.twitter.frigate.pushservice.util.PushIbisUtil._\nimport com.twitter.util.Future\n\ntrait MagicFanoutNewsEventIbis2Hydrator extends Ibis2HydratorForCandidate {\n  self: PushCandidate with MagicFanoutEventHydratedCandidate =>\n\n  override lazy val senderId: Option[Long] = {\n    val isUgmMoment = self.semanticCoreEntityTags.values.flatten.toSet\n      .contains(MagicFanoutPredicatesUtil.UgmMomentTag)\n\n    owningTwitterUserIds.headOption match {\n      case Some(owningTwitterUserId)\n          if isUgmMoment && target.params(\n            PushFeatureSwitchParams.MagicFanoutNewsUserGeneratedEventsEnable) =>\n        Some(owningTwitterUserId)\n      case _ => None\n    }\n  }\n\n  lazy val stats = self.statsReceiver.scope(\"MagicFanout\")\n  lazy val defaultImageCounter = stats.counter(\"default_image\")\n  lazy val requestImageCounter = stats.counter(\"request_num\")\n  lazy val noneImageCounter = stats.counter(\"none_num\")\n\n  private def getModelValueMediaUrl(\n    urlOpt: Option[String],\n    mapKey: String\n  ): Option[(String, String)] = {\n    requestImageCounter.incr()\n    urlOpt match {\n      case Some(PushConstants.DefaultEventMediaUrl) =>\n        defaultImageCounter.incr()\n        None\n      case Some(url) => Some(mapKey -> url)\n      case None =>\n        noneImageCounter.incr()\n        None\n    }\n  }\n\n  private lazy val eventModelValuesFut: Future[Map[String, String]] = {\n    for {\n      title <- eventTitleFut\n      squareImageUrl <- squareImageUrlFut\n      primaryImageUrl <- primaryImageUrlFut\n      eventDescriptionOpt <- eventDescriptionFut\n    } yield {\n\n      val authorId = owningTwitterUserIds.headOption match {\n        case Some(author)\n            if target.params(PushFeatureSwitchParams.MagicFanoutNewsUserGeneratedEventsEnable) =>\n          Some(\"author\" -> author.toString)\n        case _ => None\n      }\n\n      val eventDescription = eventDescriptionOpt match {\n        case Some(description)\n            if target.params(PushFeatureSwitchParams.MagicFanoutNewsEnableDescriptionCopy) =>\n          Some(\"event_description\" -> description)\n        case _ =>\n          None\n      }\n\n      Map(\n        \"event_id\" -> s\"$eventId\",\n        \"event_title\" -> title\n      ) ++\n        getModelValueMediaUrl(squareImageUrl, \"square_media_url\") ++\n        getModelValueMediaUrl(primaryImageUrl, \"media_url\") ++\n        authorId ++\n        eventDescription\n    }\n  }\n\n  private lazy val topicValuesFut: Future[Map[String, String]] = {\n    if (target.params(PushFeatureSwitchParams.EnableTopicCopyForMF)) {\n      followedTopicLocalizedEntities.map(_.headOption).flatMap {\n        case Some(localizedEntity) =>\n          Future.value(Map(\"topic_name\" -> localizedEntity.localizedNameForDisplay))\n        case _ =>\n          ergLocalizedEntities.map(_.headOption).map {\n            case Some(localizedEntity)\n                if target.params(PushFeatureSwitchParams.EnableTopicCopyForImplicitTopics) =>\n              Map(\"topic_name\" -> localizedEntity.localizedNameForDisplay)\n            case _ => Map.empty[String, String]\n          }\n      }\n    } else {\n      Future.value(Map.empty[String, String])\n    }\n  }\n\n  override lazy val modelValues: Future[Map[String, String]] =\n    mergeFutModelValues(super.modelValues, mergeFutModelValues(eventModelValuesFut, topicValuesFut))\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/MagicFanoutProductLaunchIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.common.base.MagicFanoutProductLaunchCandidate\nimport com.twitter.frigate.magic_events.thriftscala.ProductInfo\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.util.PushIbisUtil.mergeModelValues\nimport com.twitter.util.Future\n\ntrait MagicFanoutProductLaunchIbis2Hydrator\n    extends CustomConfigurationMapForIbis\n    with Ibis2HydratorForCandidate {\n  self: PushCandidate with MagicFanoutProductLaunchCandidate =>\n\n  private def getProductInfoMap(productInfo: ProductInfo): Map[String, String] = {\n    val titleMap = productInfo.title\n      .map { title =>\n        Map(\"title\" -> title)\n      }.getOrElse(Map.empty)\n    val bodyMap = productInfo.body\n      .map { body =>\n        Map(\"body\" -> body)\n      }.getOrElse(Map.empty)\n    val deeplinkMap = productInfo.deeplink\n      .map { deeplink =>\n        Map(\"deeplink\" -> deeplink)\n      }.getOrElse(Map.empty)\n\n    titleMap ++ bodyMap ++ deeplinkMap\n  }\n\n  private lazy val landingPage: Map[String, String] = {\n    val urlFromFS = target.params(PushFeatureSwitchParams.ProductLaunchLandingPageDeepLink)\n    Map(\"push_land_url\" -> urlFromFS)\n  }\n\n  private lazy val customProductLaunchPushDetails: Map[String, String] = {\n    frigateNotification.magicFanoutProductLaunchNotification match {\n      case Some(productLaunchNotif) =>\n        productLaunchNotif.productInfo match {\n          case Some(productInfo) =>\n            getProductInfoMap(productInfo)\n          case _ => Map.empty\n        }\n      case _ => Map.empty\n    }\n  }\n\n  override lazy val customFieldsMapFut: Future[Map[String, String]] =\n    mergeModelValues(super.customFieldsMapFut, customProductLaunchPushDetails)\n\n  override lazy val modelValues: Future[Map[String, String]] =\n    mergeModelValues(super.modelValues, landingPage)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/MagicFanoutSportsEventIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.common.base.BaseGameScore\nimport com.twitter.frigate.common.base.MagicFanoutSportsEventCandidate\nimport com.twitter.frigate.common.base.MagicFanoutSportsScoreInformation\nimport com.twitter.frigate.common.base.NflGameScore\nimport com.twitter.frigate.common.base.SoccerGameScore\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutEventHydratedCandidate\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.pushservice.predicate.magic_fanout.MagicFanoutSportsUtil\nimport com.twitter.frigate.pushservice.util.PushIbisUtil._\nimport com.twitter.util.Future\n\ntrait MagicFanoutSportsEventIbis2Hydrator extends Ibis2HydratorForCandidate {\n  self: PushCandidate\n    with MagicFanoutEventHydratedCandidate\n    with MagicFanoutSportsEventCandidate\n    with MagicFanoutSportsScoreInformation =>\n\n  lazy val stats = self.statsReceiver.scope(\"MagicFanoutSportsEvent\")\n  lazy val defaultImageCounter = stats.counter(\"default_image\")\n  lazy val requestImageCounter = stats.counter(\"request_num\")\n  lazy val noneImageCounter = stats.counter(\"none_num\")\n\n  override lazy val relevanceScoreMapFut = Future.value(Map.empty[String, String])\n\n  private def getModelValueMediaUrl(\n    urlOpt: Option[String],\n    mapKey: String\n  ): Option[(String, String)] = {\n    requestImageCounter.incr()\n    urlOpt match {\n      case Some(PushConstants.DefaultEventMediaUrl) =>\n        defaultImageCounter.incr()\n        None\n      case Some(url) => Some(mapKey -> url)\n      case None =>\n        noneImageCounter.incr()\n        None\n    }\n  }\n\n  private lazy val eventModelValuesFut: Future[Map[String, String]] = {\n    for {\n      title <- eventTitleFut\n      squareImageUrl <- squareImageUrlFut\n      primaryImageUrl <- primaryImageUrlFut\n    } yield {\n      Map(\n        \"event_id\" -> s\"$eventId\",\n        \"event_title\" -> title\n      ) ++\n        getModelValueMediaUrl(squareImageUrl, \"square_media_url\") ++\n        getModelValueMediaUrl(primaryImageUrl, \"media_url\")\n    }\n  }\n\n  private lazy val sportsScoreValues: Future[Map[String, String]] = {\n    for {\n      scores <- gameScores\n      homeName <- homeTeamInfo.map(_.map(_.name))\n      awayName <- awayTeamInfo.map(_.map(_.name))\n    } yield {\n      if (awayName.isDefined && homeName.isDefined && scores.isDefined) {\n        scores.get match {\n          case game: SoccerGameScore =>\n            MagicFanoutSportsUtil.getSoccerIbisMap(game) ++ Map(\n              \"away_team\" -> awayName.get,\n              \"home_team\" -> homeName.get\n            )\n          case game: NflGameScore =>\n            MagicFanoutSportsUtil.getNflIbisMap(game) ++ Map(\n              \"away_team\" -> MagicFanoutSportsUtil.getNFLReadableName(awayName.get),\n              \"home_team\" -> MagicFanoutSportsUtil.getNFLReadableName(homeName.get)\n            )\n          case baseGameScore: BaseGameScore =>\n            Map.empty[String, String]\n        }\n      } else Map.empty[String, String]\n    }\n  }\n\n  override lazy val customFieldsMapFut: Future[Map[String, String]] =\n    mergeFutModelValues(super.customFieldsMapFut, sportsScoreValues)\n\n  override lazy val modelValues: Future[Map[String, String]] =\n    mergeFutModelValues(super.modelValues, eventModelValuesFut)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/OutOfNetworkTweetIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.common.base.OutOfNetworkTweetCandidate\nimport com.twitter.frigate.common.base.TopicCandidate\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.rec_types.RecTypes._\nimport com.twitter.frigate.common.util.MrPushCopyObjects\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.util.InlineActionUtil\nimport com.twitter.frigate.pushservice.util.PushIbisUtil.mergeModelValues\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.util.Future\n\ntrait OutOfNetworkTweetIbis2HydratorForCandidate extends TweetCandidateIbis2Hydrator {\n  self: PushCandidate with OutOfNetworkTweetCandidate with TopicCandidate with TweetAuthorDetails =>\n\n  private lazy val useNewOonCopyValue =\n    if (target.params(PushFeatureSwitchParams.EnableNewMROONCopyForPush)) {\n      Map(\n        \"use_new_oon_copy\" -> \"true\"\n      )\n    } else Map.empty[String, String]\n\n  override lazy val tweetDynamicInlineActionsModelValues =\n    if (target.params(PushFeatureSwitchParams.EnableOONGeneratedInlineActions)) {\n      val actions = target.params(PushFeatureSwitchParams.OONTweetDynamicInlineActionsList)\n      InlineActionUtil.getGeneratedTweetInlineActions(target, statsReceiver, actions)\n    } else Map.empty[String, String]\n\n  private lazy val ibtModelValues: Map[String, String] =\n    Map(\n      \"is_tweet\" -> s\"${!(hasPhoto || hasVideo)}\",\n      \"is_photo\" -> s\"$hasPhoto\",\n      \"is_video\" -> s\"$hasVideo\"\n    )\n\n  private lazy val launchVideosInImmersiveExploreValue =\n    Map(\n      \"launch_videos_in_immersive_explore\" -> s\"${hasVideo && target.params(PushFeatureSwitchParams.EnableLaunchVideosInImmersiveExplore)}\"\n    )\n\n  private lazy val oonTweetModelValues =\n    useNewOonCopyValue ++ ibtModelValues ++ tweetDynamicInlineActionsModelValues ++ launchVideosInImmersiveExploreValue\n\n  lazy val useTopicCopyForMBCGIbis = mrModelingBasedTypes.contains(commonRecType) && target.params(\n    PushFeatureSwitchParams.EnableMrModelingBasedCandidatesTopicCopy)\n  lazy val useTopicCopyForFrsIbis = frsTypes.contains(commonRecType) && target.params(\n    PushFeatureSwitchParams.EnableFrsTweetCandidatesTopicCopy)\n  lazy val useTopicCopyForTagspaceIbis = tagspaceTypes.contains(commonRecType) && target.params(\n    PushFeatureSwitchParams.EnableHashspaceCandidatesTopicCopy)\n\n  override lazy val modelName: String = {\n    if (localizedUttEntity.isDefined &&\n      (useTopicCopyForMBCGIbis || useTopicCopyForFrsIbis || useTopicCopyForTagspaceIbis)) {\n      MrPushCopyObjects.TopicTweet.ibisPushModelName // uses topic copy\n    } else super.modelName\n  }\n\n  lazy val exploreVideoParams: Map[String, String] = {\n    if (self.commonRecType == CommonRecommendationType.ExploreVideoTweet) {\n      Map(\n        \"is_explore_video\" -> \"true\"\n      )\n    } else Map.empty[String, String]\n  }\n\n  override lazy val customFieldsMapFut: Future[Map[String, String]] =\n    mergeModelValues(super.customFieldsMapFut, exploreVideoParams)\n\n  override lazy val tweetModelValues: Future[Map[String, String]] =\n    if (localizedUttEntity.isDefined &&\n      (useTopicCopyForMBCGIbis || useTopicCopyForFrsIbis || useTopicCopyForTagspaceIbis)) {\n      lazy val topicTweetModelValues: Map[String, String] =\n        Map(\"topic_name\" -> s\"${localizedUttEntity.get.localizedNameForDisplay}\")\n      for {\n        superModelValues <- super.tweetModelValues\n        tweetInlineModelValue <- tweetInlineActionModelValue\n      } yield {\n        superModelValues ++ topicTweetModelValues ++ tweetInlineModelValue\n      }\n    } else {\n      for {\n        superModelValues <- super.tweetModelValues\n        tweetInlineModelValues <- tweetInlineActionModelValue\n      } yield {\n        superModelValues ++ mediaModelValue ++ oonTweetModelValues ++ tweetInlineModelValues ++ inlineVideoMediaMap\n      }\n    }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/OverrideForIbis2Request.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.common.store.deviceinfo.DeviceInfo\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FSParams}\nimport com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue.ContinuousFunction\nimport com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue.ContinuousFunctionParam\nimport com.twitter.frigate.pushservice.util.OverrideNotificationUtil\nimport com.twitter.frigate.pushservice.util.PushCapUtil\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.frigate.thriftscala.CommonRecommendationType.MagicFanoutSportsEvent\nimport com.twitter.ibis2.lib.util.JsonMarshal\nimport com.twitter.util.Future\n\ntrait OverrideForIbis2Request {\n  self: PushCandidate =>\n\n  private lazy val overrideStats = self.statsReceiver.scope(\"override_for_ibis2\")\n\n  private lazy val addedOverrideAndroidCounter =\n    overrideStats.scope(\"android\").counter(\"added_override_for_ibis2_request\")\n  private lazy val addedSmartPushConfigAndroidCounter =\n    overrideStats.scope(\"android\").counter(\"added_smart_push_config_for_ibis2_request\")\n  private lazy val addedOverrideIosCounter =\n    overrideStats.scope(\"ios\").counter(\"added_override_for_ibis2_request\")\n  private lazy val noOverrideCounter = overrideStats.counter(\"no_override_for_ibis2_request\")\n  private lazy val noOverrideDueToDeviceInfoCounter =\n    overrideStats.counter(\"no_override_due_to_device_info\")\n  private lazy val addedMlScoreToPayloadAndroid =\n    overrideStats.scope(\"android\").counter(\"added_ml_score\")\n  private lazy val noMlScoreAddedToPayload =\n    overrideStats.counter(\"no_ml_score\")\n  private lazy val addedNSlotsToPayload =\n    overrideStats.counter(\"added_n_slots\")\n  private lazy val noNSlotsAddedToPayload =\n    overrideStats.counter(\"no_n_slots\")\n  private lazy val addedCustomThreadIdToPayload =\n    overrideStats.counter(\"added_custom_thread_id\")\n  private lazy val noCustomThreadIdAddedToPayload =\n    overrideStats.counter(\"no_custom_thread_id\")\n  private lazy val enableTargetIdOverrideForMagicFanoutSportsEventCounter =\n    overrideStats.counter(\"enable_target_id_override_for_mf_sports_event\")\n\n  lazy val candidateModelScoreFut: Future[Option[Double]] = {\n    if (RecTypes.notEligibleForModelScoreTracking(commonRecType)) Future.None\n    else mrWeightedOpenOrNtabClickRankingProbability\n  }\n\n  lazy val overrideModelValueFut: Future[Map[String, String]] = {\n    if (self.target.isLoggedOutUser) {\n      Future.value(Map.empty[String, String])\n    } else {\n      Future\n        .join(\n          target.deviceInfo,\n          target.accountCountryCode,\n          OverrideNotificationUtil.getCollapseAndImpressionIdForOverride(self),\n          candidateModelScoreFut,\n          target.dynamicPushcap,\n          target.optoutAdjustedPushcap,\n          PushCapUtil.getDefaultPushCap(target)\n        ).map {\n          case (\n                deviceInfoOpt,\n                countryCodeOpt,\n                Some((collapseId, impressionIds)),\n                mlScore,\n                dynamicPushcapOpt,\n                optoutAdjustedPushcapOpt,\n                defaultPushCap) =>\n            val pushCap: Int = (dynamicPushcapOpt, optoutAdjustedPushcapOpt) match {\n              case (_, Some(optoutAdjustedPushcap)) => optoutAdjustedPushcap\n              case (Some(pushcapInfo), _) => pushcapInfo.pushcap\n              case _ => defaultPushCap\n            }\n            getClientSpecificOverrideModelValues(\n              target,\n              deviceInfoOpt,\n              countryCodeOpt,\n              collapseId,\n              impressionIds,\n              mlScore,\n              pushCap)\n          case _ =>\n            noOverrideCounter.incr()\n            Map.empty[String, String]\n        }\n    }\n  }\n\n  /**\n   * Determines the appropriate Override Notification model values based on the client\n   * @param target          Target that will be receiving the push recommendation\n   * @param deviceInfoOpt   Target's Device Info\n   * @param collapseId      Collapse ID determined by OverrideNotificationUtil\n   * @param impressionIds   Impression IDs of previously sent Override Notifications\n   * @param mlScore         Open/NTab click ranking score of the current push candidate\n   * @param pushCap         Push cap for the target\n   * @return                Map consisting of the model values that need to be added to the Ibis2 Request\n   */\n  def getClientSpecificOverrideModelValues(\n    target: Target,\n    deviceInfoOpt: Option[DeviceInfo],\n    countryCodeOpt: Option[String],\n    collapseId: String,\n    impressionIds: Seq[String],\n    mlScoreOpt: Option[Double],\n    pushCap: Int\n  ): Map[String, String] = {\n\n    val primaryDeviceIos = PushDeviceUtil.isPrimaryDeviceIOS(deviceInfoOpt)\n    val primaryDeviceAndroid = PushDeviceUtil.isPrimaryDeviceAndroid(deviceInfoOpt)\n\n    if (primaryDeviceIos ||\n      (primaryDeviceAndroid &&\n      target.params(FSParams.EnableOverrideNotificationsSmartPushConfigForAndroid))) {\n\n      if (primaryDeviceIos) addedOverrideIosCounter.incr()\n      else addedSmartPushConfigAndroidCounter.incr()\n\n      val impressionIdsSeq = {\n        if (target.params(FSParams.EnableTargetIdsInSmartPushPayload)) {\n          if (target.params(FSParams.EnableOverrideNotificationsMultipleTargetIds))\n            impressionIds\n          else Seq(impressionIds.head)\n        }\n        // Explicitly enable targetId-based override for MagicFanoutSportsEvent candidates (live sport update notifications)\n        else if (self.commonRecType == MagicFanoutSportsEvent && target.params(\n            FSParams.EnableTargetIdInSmartPushPayloadForMagicFanoutSportsEvent)) {\n          enableTargetIdOverrideForMagicFanoutSportsEventCounter.incr()\n          Seq(impressionIds.head)\n        } else Seq.empty[String]\n      }\n\n      val mlScoreMap = mlScoreOpt match {\n        case Some(mlScore)\n            if target.params(FSParams.EnableOverrideNotificationsScoreBasedOverride) =>\n          addedMlScoreToPayloadAndroid.incr()\n          Map(\"score\" -> mlScore)\n        case _ =>\n          noMlScoreAddedToPayload.incr()\n          Map.empty\n      }\n\n      val nSlotsMap = {\n        if (target.params(FSParams.EnableOverrideNotificationsNSlots)) {\n          if (target.params(FSParams.EnableOverrideMaxSlotFn)) {\n            val nslotFnParam = ContinuousFunctionParam(\n              target\n                .params(PushFeatureSwitchParams.OverrideMaxSlotFnPushCapKnobs),\n              target\n                .params(PushFeatureSwitchParams.OverrideMaxSlotFnNSlotKnobs),\n              target\n                .params(PushFeatureSwitchParams.OverrideMaxSlotFnPowerKnobs),\n              target\n                .params(PushFeatureSwitchParams.OverrideMaxSlotFnWeight),\n              target.params(FSParams.OverrideNotificationsMaxNumOfSlots)\n            )\n            val numOfSlots = ContinuousFunction.safeEvaluateFn(\n              pushCap,\n              nslotFnParam,\n              overrideStats.scope(\"max_nslot_fn\"))\n            overrideStats.counter(\"max_notification_slots_num_\" + numOfSlots.toString).incr()\n            addedNSlotsToPayload.incr()\n            Map(\"max_notification_slots\" -> numOfSlots)\n          } else {\n            addedNSlotsToPayload.incr()\n            val numOfSlots = target.params(FSParams.OverrideNotificationsMaxNumOfSlots)\n            Map(\"max_notification_slots\" -> numOfSlots)\n          }\n        } else {\n          noNSlotsAddedToPayload.incr()\n          Map.empty\n        }\n      }\n\n      val baseActionDetailsMap = Map(\"target_ids\" -> impressionIdsSeq)\n\n      val actionDetailsMap =\n        Map(\"action_details\" -> (baseActionDetailsMap ++ nSlotsMap))\n\n      val baseSmartPushConfigMap = Map(\"notification_action\" -> \"REPLACE\")\n\n      val customThreadId = {\n        if (target.params(FSParams.EnableCustomThreadIdForOverride)) {\n          addedCustomThreadIdToPayload.incr()\n          Map(\"custom_thread_id\" -> impressionId)\n        } else {\n          noCustomThreadIdAddedToPayload.incr()\n          Map.empty\n        }\n      }\n\n      val smartPushConfigMap =\n        JsonMarshal.toJson(\n          baseSmartPushConfigMap ++ actionDetailsMap ++ mlScoreMap ++ customThreadId)\n\n      Map(\"smart_notification_configuration\" -> smartPushConfigMap)\n    } else if (primaryDeviceAndroid) {\n      addedOverrideAndroidCounter.incr()\n      Map(\"notification_id\" -> collapseId, \"overriding_impression_id\" -> impressionIds.head)\n    } else {\n      noOverrideDueToDeviceInfoCounter.incr()\n      Map.empty[String, String]\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/PushOverrideInfo.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.history.History\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.frigate.thriftscala.OverrideInfo\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\n\nobject PushOverrideInfo {\n\n  private val name: String = this.getClass.getSimpleName\n\n  /**\n   * Gets all eligible time + override push notification pairs from a target's History\n   *\n   * @param history: history of push notifications\n   * @param lookbackDuration: duration to look back up in history for overriding notifications\n   * @return: list of notifications with send timestamps which are eligible for overriding\n   */\n  def getOverrideEligibleHistory(\n    history: History,\n    lookbackDuration: Duration,\n  ): Seq[(Time, FrigateNotification)] = {\n    history.sortedHistory\n      .takeWhile { case (notifTimestamp, _) => lookbackDuration.ago < notifTimestamp }\n      .filter {\n        case (_, notification) => notification.overrideInfo.isDefined\n      }\n  }\n\n  /**\n   * Gets all eligible override push notifications from a target's History\n   *\n   * @param history           Target's History\n   * @param lookbackDuration  Duration in which we would like to obtain the eligible push notifications\n   * @param stats             StatsReceiver to track stats for this function\n   * @return                  Returns a list of FrigateNotification\n   */\n  def getOverrideEligiblePushNotifications(\n    history: History,\n    lookbackDuration: Duration,\n    stats: StatsReceiver,\n  ): Seq[FrigateNotification] = {\n    val eligibleNotificationsDistribution =\n      stats.scope(name).stat(\"eligible_notifications_size_distribution\")\n    val eligibleNotificationsSeq =\n      getOverrideEligibleHistory(history, lookbackDuration)\n        .collect {\n          case (_, notification) => notification\n        }\n\n    eligibleNotificationsDistribution.add(eligibleNotificationsSeq.size)\n    eligibleNotificationsSeq\n  }\n\n  /**\n   * Gets the OverrideInfo for the last eligible Override Notification FrigateNotification, if it exists\n   * @param history           Target's History\n   * @param lookbackDuration  Duration in which we would like to obtain the last override notification\n   * @param stats             StatsReceiver to track stats for this function\n   * @return                  Returns OverrideInfo of the last MR push, else None\n   */\n  def getOverrideInfoOfLastEligiblePushNotif(\n    history: History,\n    lookbackDuration: Duration,\n    stats: StatsReceiver\n  ): Option[OverrideInfo] = {\n    val overrideInfoEmptyOfLastPush = stats.scope(name).counter(\"override_info_empty_of_last_push\")\n    val overrideInfoExistsForLastPush =\n      stats.scope(name).counter(\"override_info_exists_for_last_push\")\n    val overrideHistory =\n      getOverrideEligiblePushNotifications(history, lookbackDuration, stats)\n    if (overrideHistory.isEmpty) {\n      overrideInfoEmptyOfLastPush.incr()\n      None\n    } else {\n      overrideInfoExistsForLastPush.incr()\n      overrideHistory.head.overrideInfo\n    }\n  }\n\n  /**\n   * Gets all the MR Push Notifications in the specified override chain\n   * @param history           Target's History\n   * @param overrideChainId   Override Chain Identifier\n   * @param stats             StatsReceiver to track stats for this function\n   * @return                  Returns a sequence of FrigateNotification that exist in the override chain\n   */\n  def getMrPushNotificationsInOverrideChain(\n    history: History,\n    overrideChainId: String,\n    stats: StatsReceiver\n  ): Seq[FrigateNotification] = {\n    val notificationInOverrideChain = stats.scope(name).counter(\"notification_in_override_chain\")\n    val notificationNotInOverrideChain =\n      stats.scope(name).counter(\"notification_not_in_override_chain\")\n    history.sortedHistory.flatMap {\n      case (_, notification)\n          if isNotificationInOverrideChain(notification, overrideChainId, stats) =>\n        notificationInOverrideChain.incr()\n        Some(notification)\n      case _ =>\n        notificationNotInOverrideChain.incr()\n        None\n    }\n  }\n\n  /**\n   * Gets the timestamp (in milliseconds) for the specified FrigateNotification\n   * @param notification      The FrigateNotification that we would like the timestamp for\n   * @param history           Target's History\n   * @param stats             StatsReceiver to track stats for this function\n   * @return                  Returns the timestamp in milliseconds for the specified notification\n   *                          if it exists History, else None\n   */\n  def getTimestampInMillisForFrigateNotification(\n    notification: FrigateNotification,\n    history: History,\n    stats: StatsReceiver\n  ): Option[Long] = {\n    val foundTimestampOfNotificationInHistory =\n      stats.scope(name).counter(\"found_timestamp_of_notification_in_history\")\n    history.sortedHistory\n      .find(_._2.equals(notification)).map {\n        case (time, _) =>\n          foundTimestampOfNotificationInHistory.incr()\n          time.inMilliseconds\n      }\n  }\n\n  /**\n   * Gets the oldest frigate notification based on the user's NTab last read position\n   * @param overrideCandidatesMap     All the NTab Notifications in the override chain\n   * @return                          Returns the oldest frigate notification in the chain\n   */\n  def getOldestFrigateNotification(\n    overrideCandidatesMap: Map[Long, FrigateNotification],\n  ): FrigateNotification = {\n    overrideCandidatesMap.minBy(_._1)._2\n  }\n\n  /**\n   * Gets the impression ids of previous eligible push notification.\n   * @param history           Target's History\n   * @param lookbackDuration  Duration in which we would like to obtain previous impression ids\n   * @param stats             StatsReceiver to track stats for this function\n   * @return                  Returns the impression identifier for the last eligible push notif.\n   *                          if it exists in the target's History, else None.\n   */\n  def getImpressionIdsOfPrevEligiblePushNotif(\n    history: History,\n    lookbackDuration: Duration,\n    stats: StatsReceiver\n  ): Seq[String] = {\n    val foundImpressionIdOfLastEligiblePushNotif =\n      stats.scope(name).counter(\"found_impression_id_of_last_eligible_push_notif\")\n    val overrideHistoryEmptyWhenFetchingImpressionId =\n      stats.scope(name).counter(\"override_history_empty_when_fetching_impression_id\")\n    val overrideHistory = getOverrideEligiblePushNotifications(history, lookbackDuration, stats)\n      .filter(frigateNotification =>\n        // Exclude notifications of nonGenericOverrideTypes from being overridden\n        !RecTypes.nonGenericOverrideTypes.contains(frigateNotification.commonRecommendationType))\n\n    if (overrideHistory.isEmpty) {\n      overrideHistoryEmptyWhenFetchingImpressionId.incr()\n      Seq.empty\n    } else {\n      foundImpressionIdOfLastEligiblePushNotif.incr()\n      overrideHistory.flatMap(_.impressionId)\n    }\n  }\n\n  /**\n   * Gets the impressions ids by eventId, for MagicFanoutEvent candidates.\n   *\n   * @param history           Target's History\n   * @param lookbackDuration  Duration in which we would like to obtain previous impression ids\n   * @param stats             StatsReceiver to track stats for this function\n   * @param overridableType   Specific MagicFanoutEvent CRT\n   * @param eventId           Event identifier for MagicFanoutEventCandidate.\n   * @return                  Returns the impression identifiers for the last eligible, eventId-matching\n   *                          MagicFanoutEvent push notifications if they exist in the target's history, else None.\n   */\n  def getImpressionIdsForPrevEligibleMagicFanoutEventCandidates(\n    history: History,\n    lookbackDuration: Duration,\n    stats: StatsReceiver,\n    overridableType: CommonRecommendationType,\n    eventId: Long\n  ): Seq[String] = {\n    val foundImpressionIdOfMagicFanoutEventNotif =\n      stats.scope(name).counter(\"found_impression_id_of_magic_fanout_event_notif\")\n    val overrideHistoryEmptyWhenFetchingImpressionId =\n      stats\n        .scope(name).counter(\n          \"override_history_empty_when_fetching_impression_id_for_magic_fanout_event_notif\")\n\n    val overrideHistory =\n      getOverrideEligiblePushNotifications(history, lookbackDuration, stats)\n        .filter(frigateNotification =>\n          // Only override notifications with same CRT and eventId\n          frigateNotification.commonRecommendationType == overridableType &&\n            frigateNotification.magicFanoutEventNotification.exists(_.eventId == eventId))\n\n    if (overrideHistory.isEmpty) {\n      overrideHistoryEmptyWhenFetchingImpressionId.incr()\n      Seq.empty\n    } else {\n      foundImpressionIdOfMagicFanoutEventNotif.incr()\n      overrideHistory.flatMap(_.impressionId)\n    }\n  }\n\n  /**\n   * Determines if the provided notification is part of the specified override chain\n   * @param notification      FrigateNotification that we're trying to identify as within the override chain\n   * @param overrideChainId   Override Chain Identifier\n   * @param stats             StatsReceiver to track stats for this function\n   * @return                  Returns true if the provided FrigateNotification is within the override chain, else false\n   */\n  private def isNotificationInOverrideChain(\n    notification: FrigateNotification,\n    overrideChainId: String,\n    stats: StatsReceiver\n  ): Boolean = {\n    val notifIsInOverrideChain = stats.scope(name).counter(\"notif_is_in_override_chain\")\n    val notifNotInOverrideChain = stats.scope(name).counter(\"notif_not_in_override_chain\")\n    notification.overrideInfo match {\n      case Some(overrideInfo) =>\n        val isNotifInOverrideChain = overrideInfo.collapseInfo.overrideChainId == overrideChainId\n        if (isNotifInOverrideChain) {\n          notifIsInOverrideChain.incr()\n          true\n        } else {\n          notifNotInOverrideChain.incr()\n          false\n        }\n      case _ =>\n        notifNotInOverrideChain.incr()\n        false\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/RankedSocialContextIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.common.base.SocialContextAction\nimport com.twitter.frigate.common.base.SocialContextActions\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.util.CandidateUtil\nimport com.twitter.frigate.pushservice.util.PushIbisUtil\nimport com.twitter.util.Future\n\ntrait RankedSocialContextIbis2Hydrator {\n  self: PushCandidate with SocialContextActions =>\n\n  lazy val socialContextModelValues: Future[Map[String, String]] =\n    rankedSocialContextActionsFut.map(rankedSocialContextActions =>\n      PushIbisUtil.getSocialContextModelValues(rankedSocialContextActions.map(_.userId)))\n\n  lazy val rankedSocialContextActionsFut: Future[Seq[SocialContextAction]] =\n    CandidateUtil.getRankedSocialContext(\n      socialContextActions,\n      target.seedsWithWeight,\n      defaultToRecency = false)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/ScheduledSpaceSpeakerIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.pushservice.model.ScheduledSpaceSpeakerPushCandidate\nimport com.twitter.frigate.pushservice.util.PushIbisUtil._\nimport com.twitter.frigate.thriftscala.SpaceNotificationType\nimport com.twitter.util.Future\n\ntrait ScheduledSpaceSpeakerIbis2Hydrator extends Ibis2HydratorForCandidate {\n  self: ScheduledSpaceSpeakerPushCandidate =>\n\n  override lazy val senderId: Option[Long] = None\n\n  private lazy val targetModelValues: Future[Map[String, String]] = {\n    hostId match {\n      case Some(spaceHostId) =>\n        audioSpaceFut.map { audioSpace =>\n          val isStartNow = frigateNotification.spaceNotification.exists(\n            _.spaceNotificationType.contains(SpaceNotificationType.AtSpaceBroadcast))\n\n          Map(\n            \"host_id\" -> s\"$spaceHostId\",\n            \"space_id\" -> spaceId,\n            \"is_start_now\" -> s\"$isStartNow\"\n          ) ++ audioSpace.flatMap(_.title.map(\"space_title\" -> _))\n        }\n      case _ =>\n        Future.exception(\n          new IllegalStateException(\"Unable to get host id for ScheduledSpaceSpeakerIbis2Hydrator\"))\n    }\n  }\n\n  override lazy val modelValues: Future[Map[String, String]] =\n    mergeFutModelValues(super.modelValues, targetModelValues)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/ScheduledSpaceSubscriberIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.pushservice.model.ScheduledSpaceSubscriberPushCandidate\nimport com.twitter.frigate.pushservice.util.PushIbisUtil._\nimport com.twitter.util.Future\n\ntrait ScheduledSpaceSubscriberIbis2Hydrator extends Ibis2HydratorForCandidate {\n  self: ScheduledSpaceSubscriberPushCandidate =>\n\n  override lazy val senderId: Option[Long] = hostId\n\n  private lazy val targetModelValues: Future[Map[String, String]] = {\n    hostId match {\n      case Some(spaceHostId) =>\n        audioSpaceFut.map { audioSpace =>\n          Map(\n            \"host_id\" -> s\"$spaceHostId\",\n            \"space_id\" -> spaceId,\n          ) ++ audioSpace.flatMap(_.title.map(\"space_title\" -> _))\n        }\n      case _ =>\n        Future.exception(\n          new RuntimeException(\"Unable to get host id for ScheduledSpaceSubscriberIbis2Hydrator\"))\n    }\n  }\n\n  override lazy val modelValues: Future[Map[String, String]] =\n    mergeFutModelValues(super.modelValues, targetModelValues)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/SubscribedSearchTweetIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.pushservice.model.SubscribedSearchTweetPushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.util.InlineActionUtil\nimport com.twitter.util.Future\n\ntrait SubscribedSearchTweetIbis2Hydrator extends TweetCandidateIbis2Hydrator {\n  self: SubscribedSearchTweetPushCandidate =>\n\n  override lazy val tweetDynamicInlineActionsModelValues = {\n    if (target.params(PushFeatureSwitchParams.EnableOONGeneratedInlineActions)) {\n      val actions = target.params(PushFeatureSwitchParams.TweetDynamicInlineActionsList)\n      InlineActionUtil.getGeneratedTweetInlineActions(target, statsReceiver, actions)\n    } else Map.empty[String, String]\n  }\n\n  private lazy val searchTermValue: Map[String, String] =\n    Map(\n      \"search_term\" -> searchTerm,\n      \"search_url\" -> pushLandingUrl\n    )\n\n  private lazy val searchModelValues = searchTermValue ++ tweetDynamicInlineActionsModelValues\n\n  override lazy val tweetModelValues: Future[Map[String, String]] =\n    for {\n      superModelValues <- super.tweetModelValues\n      tweetInlineModelValues <- tweetInlineActionModelValue\n    } yield {\n      superModelValues ++ mediaModelValue ++ searchModelValues ++ tweetInlineModelValues ++ inlineVideoMediaMap\n    }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/TopTweetImpressionsCandidateIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.common.base.TopTweetImpressionsCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.util.PushIbisUtil.mergeFutModelValues\nimport com.twitter.util.Future\n\ntrait TopTweetImpressionsCandidateIbis2Hydrator extends Ibis2HydratorForCandidate {\n  self: PushCandidate with TopTweetImpressionsCandidate =>\n\n  private lazy val targetModelValues: Map[String, String] = {\n    Map(\n      \"target_user\" -> target.targetId.toString,\n      \"tweet\" -> tweetId.toString,\n      \"impressions_count\" -> impressionsCount.toString\n    )\n  }\n\n  override lazy val modelValues: Future[Map[String, String]] =\n    mergeFutModelValues(super.modelValues, Future.value(targetModelValues))\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/TopicProofTweetIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.pushservice.model.TopicProofTweetPushCandidate\nimport com.twitter.frigate.pushservice.exception.UttEntityNotFoundException\nimport com.twitter.util.Future\n\ntrait TopicProofTweetIbis2Hydrator extends TweetCandidateIbis2Hydrator {\n  self: TopicProofTweetPushCandidate =>\n\n  private lazy val implicitTopicTweetModelValues: Map[String, String] = {\n    val uttEntity = localizedUttEntity.getOrElse(\n      throw new UttEntityNotFoundException(\n        s\"${getClass.getSimpleName} UttEntity missing for $tweetId\"))\n\n    Map(\n      \"topic_name\" -> uttEntity.localizedNameForDisplay,\n      \"topic_id\" -> uttEntity.entityId.toString\n    )\n  }\n\n  override lazy val modelName: String = pushCopy.ibisPushModelName\n\n  override lazy val tweetModelValues: Future[Map[String, String]] =\n    for {\n      superModelValues <- super.tweetModelValues\n      tweetInlineModelValues <- tweetInlineActionModelValue\n    } yield {\n      superModelValues ++\n        tweetInlineModelValues ++\n        implicitTopicTweetModelValues\n    }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/TrendTweetIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.common.base.TrendTweetCandidate\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\n\ntrait TrendTweetIbis2Hydrator extends TweetCandidateIbis2Hydrator {\n  self: PushCandidate with TrendTweetCandidate with TweetAuthorDetails =>\n\n  lazy val trendNameModelValue = Map(\"trend_name\" -> trendName)\n\n  override lazy val tweetModelValues = for {\n    tweetValues <- super.tweetModelValues\n    inlineActionValues <- tweetInlineActionModelValue\n  } yield tweetValues ++ inlineActionValues ++ trendNameModelValue\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/TweetCandidateIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.base.TweetDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.SubtextForAndroidPushHeader\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FS}\nimport com.twitter.frigate.pushservice.util.CopyUtil\nimport com.twitter.frigate.pushservice.util.EmailLandingPageExperimentUtil\nimport com.twitter.frigate.pushservice.util.InlineActionUtil\nimport com.twitter.frigate.pushservice.util.PushToHomeUtil\nimport com.twitter.frigate.pushservice.util.PushIbisUtil.mergeFutModelValues\nimport com.twitter.util.Future\n\ntrait TweetCandidateIbis2Hydrator\n    extends Ibis2HydratorForCandidate\n    with InlineActionIbis2Hydrator\n    with CustomConfigurationMapForIbis {\n  self: PushCandidate with TweetCandidate with TweetDetails with TweetAuthorDetails =>\n\n  lazy val scopedStats: StatsReceiver = statsReceiver.scope(getClass.getSimpleName)\n\n  lazy val tweetIdModelValue: Map[String, String] =\n    Map(\n      \"tweet\" -> tweetId.toString\n    )\n\n  lazy val authorModelValue: Map[String, String] = {\n    assert(authorId.isDefined)\n    Map(\n      \"author\" -> authorId.getOrElse(0L).toString\n    )\n  }\n\n  lazy val otherModelValues: Map[String, String] =\n    Map(\n      \"show_explanatory_text\" -> \"true\",\n      \"show_negative_feedback\" -> \"true\"\n    )\n\n  lazy val mediaModelValue: Map[String, String] =\n    Map(\n      \"show_media\" -> \"true\"\n    )\n\n  lazy val inlineVideoMediaMap: Map[String, String] = {\n    if (hasVideo) {\n      val isInlineVideoEnabled = target.params(FS.EnableInlineVideo)\n      val isAutoplayEnabled = target.params(FS.EnableAutoplayForInlineVideo)\n      Map(\n        \"enable_inline_video_for_ios\" -> isInlineVideoEnabled.toString,\n        \"enable_autoplay_for_inline_video_ios\" -> isAutoplayEnabled.toString\n      )\n    } else Map.empty\n  }\n\n  lazy val landingPageModelValues: Future[Map[String, String]] = {\n    for {\n      deviceInfoOpt <- target.deviceInfo\n    } yield {\n      PushToHomeUtil.getIbis2ModelValue(deviceInfoOpt, target, scopedStats) match {\n        case Some(pushToHomeModelValues) => pushToHomeModelValues\n        case _ =>\n          EmailLandingPageExperimentUtil.getIbis2ModelValue(\n            deviceInfoOpt,\n            target,\n            tweetId\n          )\n      }\n    }\n  }\n\n  lazy val tweetDynamicInlineActionsModelValues = {\n    if (target.params(PushFeatureSwitchParams.EnableTweetDynamicInlineActions)) {\n      val actions = target.params(PushFeatureSwitchParams.TweetDynamicInlineActionsList)\n      InlineActionUtil.getGeneratedTweetInlineActions(target, statsReceiver, actions)\n    } else Map.empty[String, String]\n  }\n\n  lazy val tweetDynamicInlineActionsModelValuesForWeb: Map[String, String] = {\n    if (target.isLoggedOutUser) {\n      Map.empty[String, String]\n    } else {\n      InlineActionUtil.getGeneratedTweetInlineActionsForWeb(\n        actions = target.params(PushFeatureSwitchParams.TweetDynamicInlineActionsListForWeb),\n        enableForDesktopWeb =\n          target.params(PushFeatureSwitchParams.EnableDynamicInlineActionsForDesktopWeb),\n        enableForMobileWeb =\n          target.params(PushFeatureSwitchParams.EnableDynamicInlineActionsForMobileWeb)\n      )\n    }\n  }\n\n  lazy val copyFeaturesFut: Future[Map[String, String]] =\n    CopyUtil.getCopyFeatures(self, scopedStats)\n\n  private def getVerifiedSymbolModelValue: Future[Map[String, String]] = {\n    self.tweetAuthor.map {\n      case Some(author) =>\n        if (author.safety.exists(_.verified)) {\n          scopedStats.counter(\"is_verified\").incr()\n          if (target.params(FS.EnablePushPresentationVerifiedSymbol)) {\n            scopedStats.counter(\"is_verified_and_add\").incr()\n            Map(\"is_author_verified\" -> \"true\")\n          } else {\n            scopedStats.counter(\"is_verified_and_NOT_add\").incr()\n            Map.empty\n          }\n        } else {\n          scopedStats.counter(\"is_NOT_verified\").incr()\n          Map.empty\n        }\n      case _ =>\n        scopedStats.counter(\"none_author\").incr()\n        Map.empty\n    }\n  }\n\n  private def subtextAndroidPushHeader: Map[String, String] = {\n    self.target.params(PushFeatureSwitchParams.SubtextInAndroidPushHeaderParam) match {\n      case SubtextForAndroidPushHeader.None =>\n        Map.empty\n      case SubtextForAndroidPushHeader.TargetHandler =>\n        Map(\"subtext_target_handler\" -> \"true\")\n      case SubtextForAndroidPushHeader.TargetTagHandler =>\n        Map(\"subtext_target_tag_handler\" -> \"true\")\n      case SubtextForAndroidPushHeader.TargetName =>\n        Map(\"subtext_target_name\" -> \"true\")\n      case SubtextForAndroidPushHeader.AuthorTagHandler =>\n        Map(\"subtext_author_tag_handler\" -> \"true\")\n      case SubtextForAndroidPushHeader.AuthorName =>\n        Map(\"subtext_author_name\" -> \"true\")\n      case _ =>\n        Map.empty\n    }\n  }\n\n  lazy val bodyPushMap: Map[String, String] = {\n    if (self.target.params(PushFeatureSwitchParams.EnableEmptyBody)) {\n      Map(\"enable_empty_body\" -> \"true\")\n    } else Map.empty[String, String]\n  }\n\n  override def customFieldsMapFut: Future[Map[String, String]] =\n    for {\n      superModelValues <- super.customFieldsMapFut\n      copyFeaturesModelValues <- copyFeaturesFut\n      verifiedSymbolModelValue <- getVerifiedSymbolModelValue\n    } yield {\n      superModelValues ++ copyFeaturesModelValues ++\n        verifiedSymbolModelValue ++ subtextAndroidPushHeader ++ bodyPushMap\n    }\n\n  override lazy val senderId: Option[Long] = authorId\n\n  def tweetModelValues: Future[Map[String, String]] =\n    landingPageModelValues.map { landingPageModelValues =>\n      tweetIdModelValue ++ authorModelValue ++ landingPageModelValues ++ tweetDynamicInlineActionsModelValues ++ tweetDynamicInlineActionsModelValuesForWeb\n    }\n\n  override lazy val modelValues: Future[Map[String, String]] =\n    mergeFutModelValues(super.modelValues, tweetModelValues)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/TweetFavoriteIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetFavoriteCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.util.Future\n\ntrait TweetFavoriteCandidateIbis2Hydrator\n    extends TweetCandidateIbis2Hydrator\n    with RankedSocialContextIbis2Hydrator {\n  self: PushCandidate with TweetFavoriteCandidate with TweetAuthorDetails =>\n\n  override lazy val tweetModelValues: Future[Map[String, String]] =\n    for {\n      socialContextModelValues <- socialContextModelValues\n      superModelValues <- super.tweetModelValues\n      tweetInlineModelValues <- tweetInlineActionModelValue\n    } yield {\n      superModelValues ++ mediaModelValue ++ otherModelValues ++ socialContextModelValues ++ tweetInlineModelValues\n    }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ibis/TweetRetweetIbis2Hydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ibis\n\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetRetweetCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.util.PushIbisUtil.mergeModelValues\n\nimport com.twitter.util.Future\n\ntrait TweetRetweetCandidateIbis2Hydrator\n    extends TweetCandidateIbis2Hydrator\n    with RankedSocialContextIbis2Hydrator {\n  self: PushCandidate with TweetRetweetCandidate with TweetAuthorDetails =>\n\n  override lazy val tweetModelValues: Future[Map[String, String]] =\n    for {\n      socialContextModelValues <- socialContextModelValues\n      superModelValues <- super.tweetModelValues\n      tweetInlineModelValues <- tweetInlineActionModelValue\n    } yield {\n      superModelValues ++ mediaModelValue ++ otherModelValues ++ socialContextModelValues ++ tweetInlineModelValues ++ inlineVideoMediaMap\n    }\n\n  lazy val socialContextForRetweetMap: Map[String, String] =\n    if (self.target.params(PushFeatureSwitchParams.EnableSocialContextForRetweet)) {\n      Map(\"enable_social_context_retweet\" -> \"true\")\n    } else Map.empty[String, String]\n\n  override lazy val customFieldsMapFut: Future[Map[String, String]] =\n    mergeModelValues(super.customFieldsMapFut, socialContextForRetweetMap)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/CandidateNTabCopy.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.util.MRNtabCopy\nimport com.twitter.frigate.common.util.MrNtabCopyObjects\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.take.InvalidNtabCopyIdException\nimport com.twitter.frigate.pushservice.take.NtabCopyIdNotFoundException\n\ntrait CandidateNTabCopy {\n  self: PushCandidate =>\n\n  def ntabCopy: MRNtabCopy =\n    ntabCopyId\n      .map(getNtabCopyFromCopyId).getOrElse(\n        throw new NtabCopyIdNotFoundException(s\"NtabCopyId not found for $commonRecType\"))\n\n  private def getNtabCopyFromCopyId(ntabCopyId: Int): MRNtabCopy =\n    MrNtabCopyObjects\n      .getCopyFromId(ntabCopyId).getOrElse(\n        throw new InvalidNtabCopyIdException(s\"Unknown NTab Copy ID: $ntabCopyId\"))\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/DiscoverTwitterNtabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.thriftscala.{CommonRecommendationType => CRT}\nimport com.twitter.notificationservice.thriftscala._\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\ntrait DiscoverTwitterNtabRequestHydrator extends NTabRequestHydrator {\n  self: PushCandidate =>\n\n  override val senderIdFut: Future[Long] = Future.value(0L)\n\n  override val tapThroughFut: Future[String] =\n    commonRecType match {\n      case CRT.AddressBookUploadPush => Future.value(PushConstants.AddressBookUploadTapThrough)\n      case CRT.InterestPickerPush => Future.value(PushConstants.InterestPickerTapThrough)\n      case CRT.CompleteOnboardingPush =>\n        Future.value(PushConstants.CompleteOnboardingInterestAddressTapThrough)\n      case _ =>\n        Future.value(PushConstants.ConnectTabPushTapThrough)\n    }\n\n  override val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] = Future.Nil\n\n  override val facepileUsersFut: Future[Seq[Long]] = Future.Nil\n\n  override val storyContext: Option[StoryContext] = None\n\n  override val inlineCard: Option[InlineCard] = None\n\n  override val socialProofDisplayText: Option[DisplayText] = Some(DisplayText())\n\n  override lazy val ntabRequest: Future[Option[CreateGenericNotificationRequest]] =\n    if (self.commonRecType == CRT.ConnectTabPush || RecTypes.isOnboardingFlowType(\n        self.commonRecType)) {\n      Future.join(senderIdFut, displayTextEntitiesFut, facepileUsersFut, tapThroughFut).map {\n        case (senderId, displayTextEntities, facepileUsers, tapThrough) =>\n          Some(\n            CreateGenericNotificationRequest(\n              userId = target.targetId,\n              senderId = senderId,\n              genericType = GenericType.RefreshableNotification,\n              displayText = DisplayText(values = displayTextEntities),\n              facepileUsers = facepileUsers,\n              timestampMillis = Time.now.inMillis,\n              tapThroughAction = Some(TapThroughAction(Some(tapThrough))),\n              impressionId = Some(impressionId),\n              socialProofText = socialProofDisplayText,\n              context = storyContext,\n              inlineCard = inlineCard,\n              refreshableType = refreshableType\n            ))\n      }\n    } else Future.None\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/EventNTabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.notificationservice.thriftscala.DisplayText\nimport com.twitter.notificationservice.thriftscala.InlineCard\nimport com.twitter.notificationservice.thriftscala.StoryContext\nimport com.twitter.util.Future\n\ntrait EventNTabRequestHydrator extends NTabRequestHydrator {\n  self: PushCandidate =>\n\n  override def senderIdFut: Future[Long] = Future.value(0L)\n\n  override def facepileUsersFut: Future[Seq[Long]] = Future.Nil\n\n  override val storyContext: Option[StoryContext] = None\n\n  override val inlineCard: Option[InlineCard] = None\n\n  override val socialProofDisplayText: Option[DisplayText] = Some(DisplayText())\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/F1FirstDegreeTweetNTabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.take.NotificationServiceSender\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.util.Future\n\ntrait F1FirstDegreeTweetNTabRequestHydrator extends TweetNTabRequestHydrator {\n  self: PushCandidate with TweetCandidate with TweetAuthorDetails =>\n\n  override val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] =\n    NotificationServiceSender.getDisplayTextEntityFromUser(tweetAuthor, \"author\", true).map(_.toSeq)\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] = senderIdFut.map(Seq(_))\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/ListCandidateNTabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.pushservice.model.ListRecommendationPushCandidate\nimport com.twitter.notificationservice.thriftscala.DisplayText\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.notificationservice.thriftscala.InlineCard\nimport com.twitter.notificationservice.thriftscala.StoryContext\nimport com.twitter.notificationservice.thriftscala.TextValue\nimport com.twitter.util.Future\n\ntrait ListCandidateNTabRequestHydrator extends NTabRequestHydrator {\n\n  self: ListRecommendationPushCandidate =>\n\n  override lazy val senderIdFut: Future[Long] =\n    listOwnerId.map(_.getOrElse(0L))\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] = Future.Nil\n\n  override lazy val storyContext: Option[StoryContext] = None\n\n  override lazy val inlineCard: Option[InlineCard] = None\n\n  override lazy val tapThroughFut: Future[String] = Future.value(s\"i/lists/${listId}\")\n\n  override lazy val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] = listName.map {\n    listNameOpt =>\n      listNameOpt.toSeq.map { name =>\n        DisplayTextEntity(name = \"title\", value = TextValue.Text(name))\n      }\n  }\n\n  override val socialProofDisplayText: Option[DisplayText] = Some(DisplayText())\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/MagicFanoutCreatorEventNtabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.magic_events.thriftscala.CreatorFanoutType\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutCreatorEventPushCandidate\nimport com.twitter.frigate.pushservice.take.NotificationServiceSender\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationRequest\nimport com.twitter.notificationservice.thriftscala.DisplayText\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.notificationservice.thriftscala.GenericType\nimport com.twitter.notificationservice.thriftscala.InlineCard\nimport com.twitter.notificationservice.thriftscala.StoryContext\nimport com.twitter.notificationservice.thriftscala.TextValue\nimport com.twitter.notificationservice.thriftscala.TapThroughAction\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\ntrait MagicFanoutCreatorEventNtabRequestHydrator extends NTabRequestHydrator {\n  self: PushCandidate with MagicFanoutCreatorEventPushCandidate =>\n\n  override val senderIdFut: Future[Long] = Future.value(creatorId)\n\n  override lazy val tapThroughFut: Future[String] =\n    Future.value(s\"/${userProfile.screenName}/superfollows/subscribe\")\n\n  lazy val optionalTweetCountEntityFut: Future[Option[DisplayTextEntity]] = {\n    creatorFanoutType match {\n      case CreatorFanoutType.UserSubscription =>\n        numberOfTweetsFut.map {\n          _.flatMap {\n            case numberOfTweets if numberOfTweets >= 10 =>\n              Some(\n                DisplayTextEntity(\n                  name = \"tweet_count\",\n                  emphasis = true,\n                  value = TextValue.Text(numberOfTweets.toString)))\n            case _ => None\n          }\n        }\n      case _ => Future.None\n    }\n  }\n\n  override lazy val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] =\n    optionalTweetCountEntityFut\n      .map { tweetCountOpt =>\n        Seq(\n          NotificationServiceSender\n            .getDisplayTextEntityFromUser(hydratedCreator, \"display_name\", isBold = true),\n          tweetCountOpt).flatten\n      }\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] = Future.value(Seq(creatorId))\n\n  override val storyContext: Option[StoryContext] = None\n\n  override val inlineCard: Option[InlineCard] = None\n\n  lazy val refreshableTypeFut = {\n    creatorFanoutType match {\n      case CreatorFanoutType.UserSubscription =>\n        numberOfTweetsFut.map {\n          _.flatMap {\n            case numberOfTweets if numberOfTweets >= 10 =>\n              Some(\"MagicFanoutCreatorSubscriptionWithTweets\")\n            case _ => super.refreshableType\n          }\n        }\n      case _ => Future.value(super.refreshableType)\n    }\n  }\n\n  override lazy val socialProofDisplayText: Option[DisplayText] = {\n    creatorFanoutType match {\n      case CreatorFanoutType.UserSubscription =>\n        Some(\n          DisplayText(values = Seq(\n            DisplayTextEntity(name = \"handle\", value = TextValue.Text(userProfile.screenName)))))\n      case CreatorFanoutType.NewCreator => None\n      case _ => None\n    }\n  }\n\n  override lazy val ntabRequest = {\n    Future\n      .join(\n        senderIdFut,\n        displayTextEntitiesFut,\n        facepileUsersFut,\n        tapThroughFut,\n        refreshableTypeFut).map {\n        case (senderId, displayTextEntities, facepileUsers, tapThrough, refreshableTypeOpt) =>\n          Some(\n            CreateGenericNotificationRequest(\n              userId = target.targetId,\n              senderId = senderId,\n              genericType = GenericType.RefreshableNotification,\n              displayText = DisplayText(values = displayTextEntities),\n              facepileUsers = facepileUsers,\n              timestampMillis = Time.now.inMillis,\n              tapThroughAction = Some(TapThroughAction(Some(tapThrough))),\n              impressionId = Some(impressionId),\n              socialProofText = socialProofDisplayText,\n              context = storyContext,\n              inlineCard = inlineCard,\n              refreshableType = refreshableTypeOpt\n            ))\n      }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/MagicFanoutNewsEventNTabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutEventHydratedCandidate\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.notificationservice.thriftscala.TextValue\nimport com.twitter.util.Future\n\ntrait MagicFanoutNewsEventNTabRequestHydrator extends EventNTabRequestHydrator {\n  self: PushCandidate with MagicFanoutEventHydratedCandidate =>\n  override lazy val tapThroughFut: Future[String] = Future.value(s\"i/events/$eventId\")\n  override lazy val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] =\n    eventTitleFut.map { eventTitle =>\n      Seq(DisplayTextEntity(name = \"title\", value = TextValue.Text(eventTitle)))\n    }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/MagicFanoutProductLaunchNtabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.base.MagicFanoutProductLaunchCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.notificationservice.thriftscala._\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\ntrait MagicFanoutProductLaunchNtabRequestHydrator extends NTabRequestHydrator {\n  self: PushCandidate with MagicFanoutProductLaunchCandidate =>\n\n  override val senderIdFut: Future[Long] = Future.value(0L)\n\n  override lazy val tapThroughFut: Future[String] = Future.value(getProductLaunchTapThrough())\n\n  override lazy val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] = {\n    Future.value(\n      frigateNotification.magicFanoutProductLaunchNotification\n        .flatMap {\n          _.productInfo.flatMap {\n            _.body.map { body =>\n              Seq(\n                DisplayTextEntity(name = \"body\", value = TextValue.Text(body)),\n              )\n            }\n          }\n        }.getOrElse(Nil))\n  }\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] = {\n    Future.value(\n      frigateNotification.magicFanoutProductLaunchNotification\n        .flatMap {\n          _.productInfo.flatMap {\n            _.facepileUsers\n          }\n        }.getOrElse(Nil))\n  }\n\n  override val storyContext: Option[StoryContext] = None\n\n  override val inlineCard: Option[InlineCard] = None\n\n  override lazy val socialProofDisplayText: Option[DisplayText] = {\n    frigateNotification.magicFanoutProductLaunchNotification.flatMap {\n      _.productInfo.flatMap {\n        _.title.map { title =>\n          DisplayText(values =\n            Seq(DisplayTextEntity(name = \"social_context\", value = TextValue.Text(title))))\n        }\n      }\n    }\n  }\n\n  lazy val defaultTapThrough = target.params(PushFeatureSwitchParams.ProductLaunchTapThrough)\n\n  private def getProductLaunchTapThrough(): String = {\n    frigateNotification.magicFanoutProductLaunchNotification match {\n      case Some(productLaunchNotif) =>\n        productLaunchNotif.productInfo match {\n          case Some(productInfo) => productInfo.tapThrough.getOrElse(defaultTapThrough)\n          case _ => defaultTapThrough\n        }\n      case _ => defaultTapThrough\n    }\n  }\n\n  private lazy val productLaunchNtabRequest: Future[Option[CreateGenericNotificationRequest]] = {\n    Future\n      .join(senderIdFut, displayTextEntitiesFut, facepileUsersFut, tapThroughFut)\n      .map {\n        case (senderId, displayTextEntities, facepileUsers, tapThrough) =>\n          Some(\n            CreateGenericNotificationRequest(\n              userId = target.targetId,\n              senderId = senderId,\n              genericType = GenericType.RefreshableNotification,\n              displayText = DisplayText(values = displayTextEntities),\n              facepileUsers = facepileUsers,\n              timestampMillis = Time.now.inMillis,\n              tapThroughAction = Some(TapThroughAction(Some(tapThrough))),\n              impressionId = Some(impressionId),\n              socialProofText = socialProofDisplayText,\n              context = storyContext,\n              inlineCard = inlineCard,\n              refreshableType = refreshableType\n            ))\n      }\n  }\n\n  override lazy val ntabRequest: Future[Option[CreateGenericNotificationRequest]] = {\n    if (target.params(PushFeatureSwitchParams.EnableNTabEntriesForProductLaunchNotifications)) {\n      productLaunchNtabRequest\n    } else Future.None\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/MagicFanoutSportsEventNTabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.base.MagicFanoutSportsEventCandidate\nimport com.twitter.frigate.common.base.MagicFanoutSportsScoreInformation\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutEventHydratedCandidate\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FS}\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationRequest\nimport com.twitter.notificationservice.thriftscala.DisplayText\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.notificationservice.thriftscala.GenericType\nimport com.twitter.notificationservice.thriftscala.TextValue\nimport com.twitter.notificationservice.thriftscala.TapThroughAction\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\ntrait MagicFanoutSportsEventNTabRequestHydrator extends EventNTabRequestHydrator {\n  self: PushCandidate\n    with MagicFanoutEventHydratedCandidate\n    with MagicFanoutSportsEventCandidate\n    with MagicFanoutSportsScoreInformation =>\n\n  lazy val stats = self.statsReceiver.scope(\"MagicFanoutSportsEventNtabHydrator\")\n  lazy val inNetworkOnlyCounter = stats.counter(\"in_network_only\")\n  lazy val facePilesEnabledCounter = stats.counter(\"face_piles_enabled\")\n  lazy val facePilesDisabledCounter = stats.counter(\"face_piles_disabled\")\n  lazy val filterPeopleWhoDontFollowMeCounter = stats.counter(\"pepole_who_dont_follow_me_counter\")\n\n  override lazy val tapThroughFut: Future[String] = {\n    Future.value(s\"i/events/$eventId\")\n  }\n  override lazy val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] =\n    eventTitleFut.map { eventTitle =>\n      Seq(DisplayTextEntity(name = \"title\", value = TextValue.Text(eventTitle)))\n    }\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] =\n    if (target.params(FS.EnableNTabFacePileForSportsEventNotifications)) {\n      Future\n        .join(\n          target.notificationsFromOnlyPeopleIFollow,\n          target.filterNotificationsFromPeopleThatDontFollowMe,\n          awayTeamInfo,\n          homeTeamInfo).map {\n          case (inNetworkOnly, filterPeopleWhoDontFollowMe, away, home)\n              if !(inNetworkOnly || filterPeopleWhoDontFollowMe) =>\n            val awayTeamId = away.flatMap(_.twitterUserId)\n            val homeTeamId = home.flatMap(_.twitterUserId)\n            facePilesEnabledCounter.incr\n            Seq(awayTeamId, homeTeamId).flatten\n          case (inNetworkOnly, filterPeopleWhoDontFollowMe, _, _) =>\n            facePilesDisabledCounter.incr\n            if (inNetworkOnly) inNetworkOnlyCounter.incr\n            if (filterPeopleWhoDontFollowMe) filterPeopleWhoDontFollowMeCounter.incr\n            Seq.empty[Long]\n        }\n    } else Future.Nil\n\n  private lazy val sportsNtabRequest: Future[Option[CreateGenericNotificationRequest]] = {\n    Future\n      .join(senderIdFut, displayTextEntitiesFut, facepileUsersFut, tapThroughFut)\n      .map {\n        case (senderId, displayTextEntities, facepileUsers, tapThrough) =>\n          Some(\n            CreateGenericNotificationRequest(\n              userId = target.targetId,\n              senderId = senderId,\n              genericType = GenericType.RefreshableNotification,\n              displayText = DisplayText(values = displayTextEntities),\n              facepileUsers = facepileUsers,\n              timestampMillis = Time.now.inMillis,\n              tapThroughAction = Some(TapThroughAction(Some(tapThrough))),\n              impressionId = Some(impressionId),\n              socialProofText = socialProofDisplayText,\n              context = storyContext,\n              inlineCard = inlineCard,\n              refreshableType = refreshableType\n            ))\n      }\n  }\n\n  override lazy val ntabRequest: Future[Option[CreateGenericNotificationRequest]] = {\n    if (target.params(FS.EnableNTabEntriesForSportsEventNotifications)) {\n      self.target.history.flatMap { pushHistory =>\n        val prevEventHistoryExists = pushHistory.sortedHistory.exists {\n          case (_, notification) =>\n            notification.magicFanoutEventNotification.exists(_.eventId == self.eventId)\n        }\n        if (prevEventHistoryExists) {\n          Future.None\n        } else sportsNtabRequest\n      }\n    } else Future.None\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/NTabRequest.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationRequest\nimport com.twitter.util.Future\n\ntrait NTabRequest {\n\n  def ntabRequest: Future[Option[CreateGenericNotificationRequest]]\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/NTabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationRequest\nimport com.twitter.notificationservice.thriftscala.DisplayText\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.notificationservice.thriftscala.GenericType\nimport com.twitter.notificationservice.thriftscala.InlineCard\nimport com.twitter.notificationservice.thriftscala.StoryContext\nimport com.twitter.notificationservice.thriftscala.TapThroughAction\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\ntrait NTabRequestHydrator extends NTabRequest with CandidateNTabCopy {\n  self: PushCandidate =>\n\n  // Represents the sender of the recommendation\n  def senderIdFut: Future[Long]\n\n  // Consists of a sequence representing the social context user ids.\n  def facepileUsersFut: Future[Seq[Long]]\n\n  // Story Context is required for Tweet Recommendations\n  // Contains the Tweet ID of the recommended Tweet\n  def storyContext: Option[StoryContext]\n\n  // Inline card used to render a generic notification.\n  def inlineCard: Option[InlineCard]\n\n  // Represents where the recommendation should land when clicked\n  def tapThroughFut: Future[String]\n\n  // Hydration for fields that are used within the NTab copy\n  def displayTextEntitiesFut: Future[Seq[DisplayTextEntity]]\n\n  // Represents the social proof text that is needed for specific NTab copies\n  def socialProofDisplayText: Option[DisplayText]\n\n  // MagicRecs NTab entries always use RefreshableType as the Generic Type\n  final val genericType: GenericType = GenericType.RefreshableNotification\n\n  def refreshableType: Option[String] = ntabCopy.refreshableType\n\n  lazy val ntabRequest: Future[Option[CreateGenericNotificationRequest]] = {\n    Future.join(senderIdFut, displayTextEntitiesFut, facepileUsersFut, tapThroughFut).map {\n      case (senderId, displayTextEntities, facepileUsers, tapThrough) =>\n        Some(\n          CreateGenericNotificationRequest(\n            userId = target.targetId,\n            senderId = senderId,\n            genericType = GenericType.RefreshableNotification,\n            displayText = DisplayText(values = displayTextEntities),\n            facepileUsers = facepileUsers,\n            timestampMillis = Time.now.inMillis,\n            tapThroughAction = Some(TapThroughAction(Some(tapThrough))),\n            impressionId = Some(impressionId),\n            socialProofText = socialProofDisplayText,\n            context = storyContext,\n            inlineCard = inlineCard,\n            refreshableType = refreshableType\n          ))\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/NTabSocialContext.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.base.SocialContextActions\nimport com.twitter.frigate.common.base.SocialContextUserDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.util.CandidateUtil\nimport com.twitter.util.Future\n\ntrait NTabSocialContext {\n  self: PushCandidate with SocialContextActions with SocialContextUserDetails =>\n\n  private def ntabDisplayUserIds: Seq[Long] =\n    socialContextUserIds.take(ntabDisplayUserIdsLength)\n\n  def ntabDisplayUserIdsLength: Int =\n    if (socialContextUserIds.size == 2) 2 else 1\n\n  def ntabDisplayNamesAndIds: Future[Seq[(String, Long)]] =\n    scUserMap.map { userObjMap =>\n      ntabDisplayUserIds.flatMap { id =>\n        userObjMap(id).flatMap(_.profile.map(_.name)).map { name => (name, id) }\n      }\n    }\n\n  def rankedNtabDisplayNamesAndIds(defaultToRecency: Boolean): Future[Seq[(String, Long)]] =\n    scUserMap.flatMap { userObjMap =>\n      val rankedSocialContextActivityFut =\n        CandidateUtil.getRankedSocialContext(\n          socialContextActions,\n          target.seedsWithWeight,\n          defaultToRecency)\n      rankedSocialContextActivityFut.map { rankedSocialContextActivity =>\n        val ntabDisplayUserIds =\n          rankedSocialContextActivity.map(_.userId).take(ntabDisplayUserIdsLength)\n        ntabDisplayUserIds.flatMap { id =>\n          userObjMap(id).flatMap(_.profile.map(_.name)).map { name => (name, id) }\n        }\n      }\n    }\n\n  def otherCount: Future[Int] =\n    ntabDisplayNamesAndIds.map {\n      case namesWithIdSeq =>\n        Math.max(0, socialContextUserIds.length - namesWithIdSeq.size)\n    }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/OutOfNetworkTweetNTabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.base.TopicCandidate\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.base.TweetDetails\nimport com.twitter.frigate.common.rec_types.RecTypes._\nimport com.twitter.frigate.common.util.MrNtabCopyObjects\nimport com.twitter.frigate.pushservice.exception.TweetNTabRequestHydratorException\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.take.NotificationServiceSender\nimport com.twitter.notificationservice.thriftscala.DisplayText\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.notificationservice.thriftscala.TextValue\nimport com.twitter.util.Future\n\ntrait OutOfNetworkTweetNTabRequestHydrator extends TweetNTabRequestHydrator {\n  self: PushCandidate\n    with TweetCandidate\n    with TweetAuthorDetails\n    with TopicCandidate\n    with TweetDetails =>\n\n  lazy val useTopicCopyForMBCGNtab = mrModelingBasedTypes.contains(commonRecType) && target.params(\n    PushFeatureSwitchParams.EnableMrModelingBasedCandidatesTopicCopy)\n  lazy val useTopicCopyForFrsNtab = frsTypes.contains(commonRecType) && target.params(\n    PushFeatureSwitchParams.EnableFrsTweetCandidatesTopicCopy)\n  lazy val useTopicCopyForTagspaceNtab = tagspaceTypes.contains(commonRecType) && target.params(\n    PushFeatureSwitchParams.EnableHashspaceCandidatesTopicCopy)\n\n  override lazy val tapThroughFut: Future[String] = {\n    if (hasVideo && self.target.params(\n        PushFeatureSwitchParams.EnableLaunchVideosInImmersiveExplore)) {\n      Future.value(\n        s\"i/immersive_timeline?display_location=notification&include_pinned_tweet=true&pinned_tweet_id=${tweetId}&tl_type=imv\")\n    } else {\n      tweetAuthor.map {\n        case Some(author) =>\n          val authorProfile = author.profile.getOrElse(\n            throw new TweetNTabRequestHydratorException(\n              s\"Unable to obtain author profile for: ${author.id}\"))\n          s\"${authorProfile.screenName}/status/${tweetId.toString}\"\n        case _ =>\n          throw new TweetNTabRequestHydratorException(\n            s\"Unable to obtain author and target details to generate tap through for Tweet: $tweetId\")\n      }\n    }\n  }\n\n  override lazy val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] =\n    if (localizedUttEntity.isDefined &&\n      (useTopicCopyForMBCGNtab || useTopicCopyForFrsNtab || useTopicCopyForTagspaceNtab)) {\n      NotificationServiceSender\n        .getDisplayTextEntityFromUser(tweetAuthor, \"tweetAuthorName\", isBold = true).map(_.toSeq)\n    } else {\n      NotificationServiceSender\n        .getDisplayTextEntityFromUser(tweetAuthor, \"author\", isBold = true).map(_.toSeq)\n    }\n\n  override lazy val refreshableType: Option[String] = {\n    if (localizedUttEntity.isDefined &&\n      (useTopicCopyForMBCGNtab || useTopicCopyForFrsNtab || useTopicCopyForTagspaceNtab)) {\n      MrNtabCopyObjects.TopicTweet.refreshableType\n    } else ntabCopy.refreshableType\n  }\n\n  override def socialProofDisplayText: Option[DisplayText] = {\n    if (localizedUttEntity.isDefined &&\n      (useTopicCopyForMBCGNtab || useTopicCopyForFrsNtab || useTopicCopyForTagspaceNtab)) {\n      localizedUttEntity.map(uttEntity =>\n        DisplayText(values =\n          Seq(DisplayTextEntity(\"topic_name\", TextValue.Text(uttEntity.localizedNameForDisplay)))))\n    } else None\n  }\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] = senderIdFut.map(Seq(_))\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/ScheduledSpaceNTabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.base.SpaceCandidate\nimport com.twitter.frigate.common.util.MrNtabCopyObjects\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.ScheduledSpaceSpeakerPushCandidate\nimport com.twitter.frigate.pushservice.model.ScheduledSpaceSubscriberPushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.take.NotificationServiceSender\nimport com.twitter.frigate.thriftscala.SpaceNotificationType\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.notificationservice.thriftscala._\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\ntrait ScheduledSpaceSpeakerNTabRequestHydrator extends ScheduledSpaceNTabRequestHydrator {\n  self: PushCandidate with ScheduledSpaceSpeakerPushCandidate =>\n\n  override def refreshableType: Option[String] = {\n    frigateNotification.spaceNotification.flatMap { spaceNotification =>\n      spaceNotification.spaceNotificationType.flatMap {\n        case SpaceNotificationType.PreSpaceBroadcast =>\n          MrNtabCopyObjects.ScheduledSpaceSpeakerSoon.refreshableType\n        case SpaceNotificationType.AtSpaceBroadcast =>\n          MrNtabCopyObjects.ScheduledSpaceSpeakerNow.refreshableType\n        case _ =>\n          throw new IllegalStateException(s\"Unexpected SpaceNotificationType\")\n      }\n    }\n  }\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] = Future.Nil\n\n  override val socialProofDisplayText: Option[DisplayText] = Some(DisplayText())\n}\n\ntrait ScheduledSpaceSubscriberNTabRequestHydrator extends ScheduledSpaceNTabRequestHydrator {\n  self: PushCandidate with ScheduledSpaceSubscriberPushCandidate =>\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] = {\n    hostId match {\n      case Some(spaceHostId) => Future.value(Seq(spaceHostId))\n      case _ =>\n        Future.exception(\n          new IllegalStateException(\n            \"Unable to get host id for ScheduledSpaceSubscriberNTabRequestHydrator\"))\n    }\n  }\n\n  override val socialProofDisplayText: Option[DisplayText] = None\n}\n\ntrait ScheduledSpaceNTabRequestHydrator extends NTabRequestHydrator {\n  self: PushCandidate with SpaceCandidate =>\n\n  def hydratedHost: Option[User]\n\n  override lazy val senderIdFut: Future[Long] = {\n    hostId match {\n      case Some(spaceHostId) => Future.value(spaceHostId)\n      case _ => throw new IllegalStateException(s\"No Space Host Id\")\n    }\n  }\n\n  override lazy val tapThroughFut: Future[String] = Future.value(s\"i/spaces/$spaceId\")\n\n  override lazy val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] =\n    NotificationServiceSender\n      .getDisplayTextEntityFromUser(\n        Future.value(hydratedHost),\n        fieldName = \"space_host_name\",\n        isBold = true\n      ).map(_.toSeq)\n\n  override val storyContext: Option[StoryContext] = None\n\n  override val inlineCard: Option[InlineCard] = None\n\n  override lazy val ntabRequest: Future[Option[CreateGenericNotificationRequest]] = {\n    Future.join(senderIdFut, displayTextEntitiesFut, facepileUsersFut, tapThroughFut).map {\n      case (senderId, displayTextEntities, facepileUsers, tapThrough) =>\n        val expiryTimeMillis = if (target.params(PushFeatureSwitchParams.EnableSpacesTtlForNtab)) {\n          Some(\n            (Time.now + target.params(\n              PushFeatureSwitchParams.SpaceNotificationsTTLDurationForNTab)).inMillis)\n        } else None\n\n        Some(\n          CreateGenericNotificationRequest(\n            userId = target.targetId,\n            senderId = senderId,\n            genericType = GenericType.RefreshableNotification,\n            displayText = DisplayText(values = displayTextEntities),\n            facepileUsers = facepileUsers,\n            timestampMillis = Time.now.inMillis,\n            tapThroughAction = Some(TapThroughAction(Some(tapThrough))),\n            impressionId = Some(impressionId),\n            socialProofText = socialProofDisplayText,\n            context = storyContext,\n            inlineCard = inlineCard,\n            refreshableType = refreshableType,\n            expiryTimeMillis = expiryTimeMillis\n          ))\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/SubscribedSearchTweetNtabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.pushservice.model.SubscribedSearchTweetPushCandidate\nimport com.twitter.frigate.pushservice.take.NotificationServiceSender\nimport com.twitter.notificationservice.thriftscala.DisplayText\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.notificationservice.thriftscala.TextValue\nimport com.twitter.util.Future\n\ntrait SubscribedSearchTweetNtabRequestHydrator extends TweetNTabRequestHydrator {\n  self: SubscribedSearchTweetPushCandidate =>\n  override def displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] = NotificationServiceSender\n    .getDisplayTextEntityFromUser(tweetAuthor, \"tweetAuthor\", isBold = true).map(_.toSeq)\n\n  override def socialProofDisplayText: Option[DisplayText] = {\n    Some(DisplayText(values = Seq(DisplayTextEntity(\"search_query\", TextValue.Text(searchTerm)))))\n  }\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] = senderIdFut.map(Seq(_))\n\n  override lazy val tapThroughFut: Future[String] =\n    Future.value(self.ntabLandingUrl)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/TopTweetImpressionsNTabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.base.TopTweetImpressionsCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.notificationservice.thriftscala.DisplayText\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.notificationservice.thriftscala.InlineCard\nimport com.twitter.notificationservice.thriftscala.StoryContext\nimport com.twitter.notificationservice.thriftscala.StoryContextValue\nimport com.twitter.notificationservice.thriftscala.TextValue\nimport com.twitter.util.Future\n\ntrait TopTweetImpressionsNTabRequestHydrator extends NTabRequestHydrator {\n  self: PushCandidate with TopTweetImpressionsCandidate =>\n\n  override lazy val tapThroughFut: Future[String] =\n    Future.value(s\"${target.targetId}/status/$tweetId\")\n\n  override val senderIdFut: Future[Long] = Future.value(0L)\n\n  override val facepileUsersFut: Future[Seq[Long]] = Future.Nil\n\n  override val storyContext: Option[StoryContext] =\n    Some(StoryContext(altText = \"\", value = Some(StoryContextValue.Tweets(Seq(tweetId)))))\n\n  override val inlineCard: Option[InlineCard] = None\n\n  override lazy val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] = {\n    Future.value(\n      Seq(\n        DisplayTextEntity(name = \"num_impressions\", value = TextValue.Number(self.impressionsCount))\n      )\n    )\n  }\n\n  override def socialProofDisplayText: Option[DisplayText] = None\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/TopicProofTweetNtabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.pushservice.model.TopicProofTweetPushCandidate\nimport com.twitter.frigate.pushservice.exception.TweetNTabRequestHydratorException\nimport com.twitter.frigate.pushservice.exception.UttEntityNotFoundException\nimport com.twitter.frigate.pushservice.take.NotificationServiceSender\nimport com.twitter.notificationservice.thriftscala.DisplayText\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.notificationservice.thriftscala.StoryContext\nimport com.twitter.notificationservice.thriftscala.StoryContextValue\nimport com.twitter.notificationservice.thriftscala.TextValue\nimport com.twitter.util.Future\n\ntrait TopicProofTweetNtabRequestHydrator extends NTabRequestHydrator {\n  self: TopicProofTweetPushCandidate =>\n\n  override def displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] = NotificationServiceSender\n    .getDisplayTextEntityFromUser(tweetAuthor, \"tweetAuthorName\", true)\n    .map(_.toSeq)\n\n  private lazy val uttEntity = localizedUttEntity.getOrElse(\n    throw new UttEntityNotFoundException(\n      s\"${getClass.getSimpleName} UttEntity missing for $tweetId\")\n  )\n\n  override lazy val tapThroughFut: Future[String] = {\n    tweetAuthor.map {\n      case Some(author) =>\n        val authorProfile = author.profile.getOrElse(\n          throw new TweetNTabRequestHydratorException(\n            s\"Unable to obtain author profile for: ${author.id}\"))\n        s\"${authorProfile.screenName}/status/${tweetId.toString}\"\n      case _ =>\n        throw new TweetNTabRequestHydratorException(\n          s\"Unable to obtain author and target details to generate tap through for Tweet: $tweetId\")\n    }\n  }\n\n  override lazy val socialProofDisplayText: Option[DisplayText] = {\n    Some(\n      DisplayText(values =\n        Seq(DisplayTextEntity(\"topic_name\", TextValue.Text(uttEntity.localizedNameForDisplay))))\n    )\n  }\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] = senderIdFut.map(Seq(_))\n\n  override val inlineCard = None\n\n  override def storyContext: Option[StoryContext] = Some(\n    StoryContext(\"\", Some(StoryContextValue.Tweets(Seq(tweetId)))))\n\n  override def senderIdFut: Future[Long] =\n    tweetAuthor.map {\n      case Some(author) => author.id\n      case _ =>\n        throw new TweetNTabRequestHydratorException(\n          s\"Unable to obtain Author ID for: $commonRecType\")\n    }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/TrendTweetNtabHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.base.TrendTweetCandidate\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.pushservice.exception.TweetNTabRequestHydratorException\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.take.NotificationServiceSender\nimport com.twitter.frigate.pushservice.util.EmailLandingPageExperimentUtil\nimport com.twitter.notificationservice.thriftscala.DisplayText\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.notificationservice.thriftscala.TextValue\nimport com.twitter.util.Future\n\ntrait TrendTweetNtabHydrator extends TweetNTabRequestHydrator {\n  self: PushCandidate with TrendTweetCandidate with TweetCandidate with TweetAuthorDetails =>\n\n  private lazy val trendTweetNtabStats = self.statsReceiver.scope(\"trend_tweet_ntab\")\n\n  private lazy val ruxLandingOnNtabCounter =\n    trendTweetNtabStats.counter(\"use_rux_landing_on_ntab\")\n\n  override lazy val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] =\n    NotificationServiceSender\n      .getDisplayTextEntityFromUser(tweetAuthor, fieldName = \"author_name\", isBold = true)\n      .map(\n        _.toSeq :+ DisplayTextEntity(\n          name = \"trend_name\",\n          value = TextValue.Text(trendName),\n          emphasis = true)\n      )\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] = senderIdFut.map(Seq(_))\n\n  override lazy val socialProofDisplayText: Option[DisplayText] = None\n\n  override def refreshableType: Option[String] = ntabCopy.refreshableType\n\n  override lazy val tapThroughFut: Future[String] = {\n    Future.join(tweetAuthor, target.deviceInfo).map {\n      case (Some(author), Some(deviceInfo)) =>\n        val enableRuxLandingPage = deviceInfo.isRuxLandingPageEligible && target.params(\n          PushFeatureSwitchParams.EnableNTabRuxLandingPage)\n        val authorProfile = author.profile.getOrElse(\n          throw new TweetNTabRequestHydratorException(\n            s\"Unable to obtain author profile for: ${author.id}\"))\n\n        if (enableRuxLandingPage) {\n          ruxLandingOnNtabCounter.incr()\n          EmailLandingPageExperimentUtil.createNTabRuxLandingURI(authorProfile.screenName, tweetId)\n        } else {\n          s\"${authorProfile.screenName}/status/${tweetId.toString}\"\n        }\n\n      case _ =>\n        throw new TweetNTabRequestHydratorException(\n          s\"Unable to obtain author and target details to generate tap through for Tweet: $tweetId\")\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/TweetFavoriteNTabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.base.SocialContextActions\nimport com.twitter.frigate.common.base.SocialContextUserDetails\nimport com.twitter.frigate.common.base.TweetAuthor\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.take.NotificationServiceSender\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.util.Future\n\ntrait TweetFavoriteNTabRequestHydrator extends TweetNTabRequestHydrator with NTabSocialContext {\n  self: PushCandidate\n    with TweetCandidate\n    with TweetAuthor\n    with TweetAuthorDetails\n    with SocialContextActions\n    with SocialContextUserDetails =>\n\n  override lazy val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] = {\n    Future\n      .join(\n        NotificationServiceSender\n          .getDisplayTextEntityFromUser(tweetAuthor, \"tweetAuthorName\", isBold = false),\n        NotificationServiceSender\n          .generateSocialContextTextEntities(\n            rankedNtabDisplayNamesAndIds(defaultToRecency = false),\n            otherCount)\n      )\n      .map {\n        case (authorDisplay, socialContextDisplay) =>\n          socialContextDisplay ++ authorDisplay\n      }\n  }\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] = Future.value(socialContextUserIds)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/TweetNTabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.pushservice.exception.TweetNTabRequestHydratorException\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.notificationservice.thriftscala.InlineCard\nimport com.twitter.notificationservice.thriftscala.StoryContext\nimport com.twitter.notificationservice.thriftscala.StoryContextValue\nimport com.twitter.frigate.pushservice.util.EmailLandingPageExperimentUtil\nimport com.twitter.notificationservice.thriftscala._\nimport com.twitter.util.Future\n\ntrait TweetNTabRequestHydrator extends NTabRequestHydrator {\n  self: PushCandidate with TweetCandidate with TweetAuthorDetails =>\n\n  override def senderIdFut: Future[Long] =\n    tweetAuthor.map {\n      case Some(author) => author.id\n      case _ =>\n        throw new TweetNTabRequestHydratorException(\n          s\"Unable to obtain Author ID for: $commonRecType\")\n    }\n\n  override def storyContext: Option[StoryContext] = Some(\n    StoryContext(\n      altText = \"\",\n      value = Some(StoryContextValue.Tweets(Seq(tweetId))),\n      details = None\n    ))\n\n  override def inlineCard: Option[InlineCard] = Some(InlineCard.TweetCard(TweetCard(tweetId)))\n\n  override lazy val tapThroughFut: Future[String] = {\n    Future.join(tweetAuthor, target.deviceInfo).map {\n      case (Some(author), Some(deviceInfo)) =>\n        val enableRuxLandingPage = deviceInfo.isRuxLandingPageEligible && target.params(\n          PushFeatureSwitchParams.EnableNTabRuxLandingPage)\n        val authorProfile = author.profile.getOrElse(\n          throw new TweetNTabRequestHydratorException(\n            s\"Unable to obtain author profile for: ${author.id}\"))\n        if (enableRuxLandingPage) {\n          EmailLandingPageExperimentUtil.createNTabRuxLandingURI(authorProfile.screenName, tweetId)\n        } else {\n          s\"${authorProfile.screenName}/status/${tweetId.toString}\"\n        }\n      case _ =>\n        throw new TweetNTabRequestHydratorException(\n          s\"Unable to obtain author and target details to generate tap through for Tweet: $tweetId\")\n    }\n  }\n\n  override def socialProofDisplayText: Option[DisplayText] = None\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/model/ntab/TweetRetweetNTabRequestHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.model.ntab\n\nimport com.twitter.frigate.common.base.SocialContextActions\nimport com.twitter.frigate.common.base.SocialContextUserDetails\nimport com.twitter.frigate.common.base.TweetAuthor\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.take.NotificationServiceSender\nimport com.twitter.notificationservice.thriftscala.DisplayTextEntity\nimport com.twitter.util.Future\n\ntrait TweetRetweetNTabRequestHydrator extends TweetNTabRequestHydrator with NTabSocialContext {\n  self: PushCandidate\n    with TweetCandidate\n    with TweetAuthor\n    with TweetAuthorDetails\n    with SocialContextActions\n    with SocialContextUserDetails =>\n\n  override lazy val displayTextEntitiesFut: Future[Seq[DisplayTextEntity]] = {\n    Future\n      .join(\n        NotificationServiceSender\n          .getDisplayTextEntityFromUser(tweetAuthor, \"tweetAuthorName\", isBold = false),\n        NotificationServiceSender\n          .generateSocialContextTextEntities(\n            rankedNtabDisplayNamesAndIds(defaultToRecency = false),\n            otherCount)\n      )\n      .map {\n        case (authorDisplay, socialContextDisplay) =>\n          socialContextDisplay ++ authorDisplay\n      }\n  }\n\n  override lazy val facepileUsersFut: Future[Seq[Long]] = Future.value(socialContextUserIds)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/module/DeployConfigModule.scala",
    "content": "package com.twitter.frigate.pushservice.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.decider.Decider\nimport com.twitter.featureswitches.v2.FeatureSwitches\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.tunable.StandardTunableMap\nimport com.twitter.frigate.pushservice.config.DeployConfig\nimport com.twitter.frigate.pushservice.config.ProdConfig\nimport com.twitter.frigate.pushservice.config.StagingConfig\nimport com.twitter.frigate.pushservice.params.ShardParams\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ConfigRepoLocalPath\nimport com.twitter.product_mixer.core.module.product_mixer_flags.ProductMixerFlagModule.ServiceLocal\n\nobject DeployConfigModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def providesDeployConfig(\n    @Flag(FlagName.numShards) numShards: Int,\n    @Flag(FlagName.shardId) shardId: Int,\n    @Flag(FlagName.isInMemCacheOff) inMemCacheOff: Boolean,\n    @Flag(ServiceLocal) isServiceLocal: Boolean,\n    @Flag(ConfigRepoLocalPath) localConfigRepoPath: String,\n    serviceIdentifier: ServiceIdentifier,\n    decider: Decider,\n    abDecider: LoggingABDecider,\n    featureSwitches: FeatureSwitches,\n    statsReceiver: StatsReceiver\n  ): DeployConfig = {\n    val tunableMap = if (serviceIdentifier.service.contains(\"canary\")) {\n      StandardTunableMap(id = \"frigate-pushservice-canary\")\n    } else { StandardTunableMap(id = serviceIdentifier.service) }\n    val shardParams = ShardParams(numShards, shardId)\n    serviceIdentifier.environment match {\n      case \"devel\" | \"staging\" =>\n        StagingConfig(\n          isServiceLocal = isServiceLocal,\n          localConfigRepoPath = localConfigRepoPath,\n          inMemCacheOff = inMemCacheOff,\n          decider = decider,\n          abDecider = abDecider,\n          featureSwitches = featureSwitches,\n          serviceIdentifier = serviceIdentifier,\n          tunableMap = tunableMap,\n          shardParams = shardParams\n        )(statsReceiver)\n      case \"prod\" =>\n        ProdConfig(\n          isServiceLocal = isServiceLocal,\n          localConfigRepoPath = localConfigRepoPath,\n          inMemCacheOff = inMemCacheOff,\n          decider = decider,\n          abDecider = abDecider,\n          featureSwitches = featureSwitches,\n          serviceIdentifier = serviceIdentifier,\n          tunableMap = tunableMap,\n          shardParams = shardParams\n        )(statsReceiver)\n      case env => throw new Exception(s\"Unknown environment $env\")\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/module/FilterModule.scala",
    "content": "package com.twitter.frigate.pushservice.module\n\nimport com.google.inject.Provides\nimport javax.inject.Singleton\nimport com.twitter.discovery.common.nackwarmupfilter.NackWarmupFilter\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.util.Duration\n\nobject FilterModule extends TwitterModule {\n  @Singleton\n  @Provides\n  def providesNackWarmupFilter(\n    @Flag(FlagName.nackWarmupDuration) warmupDuration: Duration\n  ): NackWarmupFilter = new NackWarmupFilter(warmupDuration)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/module/FlagModule.scala",
    "content": "package com.twitter.frigate.pushservice.module\n\nimport com.twitter.app.Flag\nimport com.twitter.inject.TwitterModule\nimport com.twitter.util.Duration\nimport com.twitter.conversions.DurationOps._\n\nobject FlagName {\n  final val shardId = \"service.shard\"\n  final val numShards = \"service.num_shards\"\n  final val nackWarmupDuration = \"service.nackWarmupDuration\"\n  final val isInMemCacheOff = \"service.isInMemCacheOff\"\n}\n\nobject FlagModule extends TwitterModule {\n\n  val shardId: Flag[Int] = flag[Int](\n    name = FlagName.shardId,\n    help = \"Service shard id\"\n  )\n\n  val numShards: Flag[Int] = flag[Int](\n    name = FlagName.numShards,\n    help = \"Number of shards\"\n  )\n\n  val mrLoggerIsTraceAll: Flag[Boolean] = flag[Boolean](\n    name = \"service.isTraceAll\",\n    help = \"atraceflag\",\n    default = false\n  )\n\n  val mrLoggerNthLog: Flag[Boolean] = flag[Boolean](\n    name = \"service.nthLog\",\n    help = \"nthlog\",\n    default = false\n  )\n\n  val inMemCacheOff: Flag[Boolean] = flag[Boolean](\n    name = FlagName.isInMemCacheOff,\n    help = \"is inMemCache Off (currently only applies for user_health_model_score_store_cache)\",\n    default = false\n  )\n\n  val mrLoggerNthVal: Flag[Long] = flag[Long](\n    name = \"service.nthVal\",\n    help = \"nthlogval\",\n    default = 0,\n  )\n\n  val nackWarmupDuration: Flag[Duration] = flag[Duration](\n    name = FlagName.nackWarmupDuration,\n    help = \"duration to nack at startup\",\n    default = 0.seconds\n  )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/module/LoggedOutPushTargetUserBuilderModule.scala",
    "content": "package com.twitter.frigate.pushservice.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.decider.Decider\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.target.LoggedOutPushTargetUserBuilder\nimport com.twitter.frigate.pushservice.config.DeployConfig\nimport com.twitter.inject.TwitterModule\n\nobject LoggedOutPushTargetUserBuilderModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def providesLoggedOutPushTargetUserBuilder(\n    decider: Decider,\n    config: DeployConfig,\n    statsReceiver: StatsReceiver\n  ): LoggedOutPushTargetUserBuilder = {\n    LoggedOutPushTargetUserBuilder(\n      historyStore = config.loggedOutHistoryStore,\n      inputDecider = decider,\n      inputAbDecider = config.abDecider,\n      loggedOutPushInfoStore = config.loggedOutPushInfoStore\n    )(statsReceiver)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/module/PushHandlerModule.scala",
    "content": "package com.twitter.frigate.pushservice.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.target.LoggedOutPushTargetUserBuilder\nimport com.twitter.frigate.pushservice.refresh_handler.RefreshForPushHandler\nimport com.twitter.frigate.pushservice.config.DeployConfig\nimport com.twitter.frigate.pushservice.send_handler.SendHandler\nimport com.twitter.frigate.pushservice.take.candidate_validator.RFPHCandidateValidator\nimport com.twitter.frigate.pushservice.take.candidate_validator.SendHandlerPostCandidateValidator\nimport com.twitter.frigate.pushservice.take.candidate_validator.SendHandlerPreCandidateValidator\nimport com.twitter.frigate.pushservice.refresh_handler.LoggedOutRefreshForPushHandler\nimport com.twitter.frigate.pushservice.take.SendHandlerNotifier\nimport com.twitter.frigate.pushservice.target.PushTargetUserBuilder\nimport com.twitter.inject.TwitterModule\n\nobject PushHandlerModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def providesRefreshForPushHandler(\n    pushTargetUserBuilder: PushTargetUserBuilder,\n    config: DeployConfig,\n    statsReceiver: StatsReceiver\n  ): RefreshForPushHandler = {\n    new RefreshForPushHandler(\n      pushTargetUserBuilder = pushTargetUserBuilder,\n      candSourceGenerator = config.candidateSourceGenerator,\n      rfphRanker = config.rfphRanker,\n      candidateHydrator = config.candidateHydrator,\n      candidateValidator = new RFPHCandidateValidator(config),\n      rfphTakeStepUtil = config.rfphTakeStepUtil,\n      rfphRestrictStep = config.rfphRestrictStep,\n      rfphNotifier = config.rfphNotifier,\n      rfphStatsRecorder = config.rfphStatsRecorder,\n      mrRequestScriberNode = config.mrRequestScriberNode,\n      rfphFeatureHydrator = config.rfphFeatureHydrator,\n      rfphPrerankFilter = config.rfphPrerankFilter,\n      rfphLightRanker = config.rfphLightRanker\n    )(statsReceiver)\n  }\n\n  @Provides\n  @Singleton\n  def providesSendHandler(\n    pushTargetUserBuilder: PushTargetUserBuilder,\n    config: DeployConfig,\n    statsReceiver: StatsReceiver\n  ): SendHandler = {\n    new SendHandler(\n      pushTargetUserBuilder,\n      new SendHandlerPreCandidateValidator(config),\n      new SendHandlerPostCandidateValidator(config),\n      new SendHandlerNotifier(config.candidateNotifier, statsReceiver.scope(\"SendHandlerNotifier\")),\n      config.sendHandlerCandidateHydrator,\n      config.featureHydrator,\n      config.sendHandlerPredicateUtil,\n      config.mrRequestScriberNode)(statsReceiver, config)\n  }\n\n  @Provides\n  @Singleton\n  def providesLoggedOutRefreshForPushHandler(\n    loPushTargetUserBuilder: LoggedOutPushTargetUserBuilder,\n    config: DeployConfig,\n    statsReceiver: StatsReceiver\n  ): LoggedOutRefreshForPushHandler = {\n    new LoggedOutRefreshForPushHandler(\n      loPushTargetUserBuilder = loPushTargetUserBuilder,\n      loPushCandidateSourceGenerator = config.loCandidateSourceGenerator,\n      candidateHydrator = config.candidateHydrator,\n      loRanker = config.loggedOutRFPHRanker,\n      loRfphNotifier = config.loRfphNotifier,\n      loMrRequestScriberNode = config.loggedOutMrRequestScriberNode,\n    )(statsReceiver)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/module/PushServiceDarkTrafficModule.scala",
    "content": "package com.twitter.frigate.pushservice.module\n\nimport com.google.inject.Singleton\nimport com.twitter.decider.Decider\nimport com.twitter.decider.RandomRecipient\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsClient\nimport com.twitter.frigate.pushservice.thriftscala.PushService\nimport com.twitter.inject.Injector\nimport com.twitter.inject.thrift.modules.ReqRepDarkTrafficFilterModule\n\n/**\n * The darkTraffic filter sample all requests by default\n  and set the diffy dest to nil for non prod environments\n */\n@Singleton\nobject PushServiceDarkTrafficModule\n    extends ReqRepDarkTrafficFilterModule[PushService.ReqRepServicePerEndpoint]\n    with MtlsClient {\n\n  override def label: String = \"frigate-pushservice-diffy-proxy\"\n\n  /**\n   * Function to determine if the request should be \"sampled\", e.g.\n   * sent to the dark service.\n   *\n   * @param injector the [[com.twitter.inject.Injector]] for use in determining if a given request\n   *                 should be forwarded or not.\n   */\n  override protected def enableSampling(injector: Injector): Any => Boolean = {\n    val decider = injector.instance[Decider]\n    _ => decider.isAvailable(\"frigate_pushservice_dark_traffic_percent\", Some(RandomRecipient))\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/module/PushTargetUserBuilderModule.scala",
    "content": "package com.twitter.frigate.pushservice.module\n\nimport com.google.inject.Provides\nimport com.google.inject.Singleton\nimport com.twitter.decider.Decider\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.config.DeployConfig\nimport com.twitter.frigate.pushservice.target.PushTargetUserBuilder\nimport com.twitter.inject.TwitterModule\n\nobject PushTargetUserBuilderModule extends TwitterModule {\n\n  @Provides\n  @Singleton\n  def providesPushTargetUserBuilder(\n    decider: Decider,\n    config: DeployConfig,\n    statsReceiver: StatsReceiver\n  ): PushTargetUserBuilder = {\n    PushTargetUserBuilder(\n      historyStore = config.historyStore,\n      emailHistoryStore = config.emailHistoryStore,\n      labeledPushRecsStore = config.labeledPushRecsDecideredStore,\n      onlineUserHistoryStore = config.onlineUserHistoryStore,\n      pushRecItemsStore = config.pushRecItemStore,\n      userStore = config.safeUserStore,\n      pushInfoStore = config.pushInfoStore,\n      userCountryStore = config.userCountryStore,\n      userUtcOffsetStore = config.userUtcOffsetStore,\n      dauProbabilityStore = config.dauProbabilityStore,\n      nsfwConsumerStore = config.nsfwConsumerStore,\n      genericNotificationFeedbackStore = config.genericNotificationFeedbackStore,\n      userFeatureStore = config.userFeaturesStore,\n      mrUserStateStore = config.mrUserStatePredictionStore,\n      tweetImpressionStore = config.tweetImpressionStore,\n      timelinesUserSessionStore = config.timelinesUserSessionStore,\n      cachedTweetyPieStore = config.cachedTweetyPieStoreV2,\n      strongTiesStore = config.strongTiesStore,\n      userHTLLastVisitStore = config.userHTLLastVisitStore,\n      userLanguagesStore = config.userLanguagesStore,\n      inputDecider = decider,\n      inputAbDecider = config.abDecider,\n      realGraphScoresTop500InStore = config.realGraphScoresTop500InStore,\n      recentFollowsStore = config.recentFollowsStore,\n      resurrectedUserStore = config.reactivatedUserInfoStore,\n      configParamsBuilder = config.configParamsBuilder,\n      optOutUserInterestsStore = config.optOutUserInterestsStore,\n      deviceInfoStore = config.deviceInfoStore,\n      pushcapDynamicPredictionStore = config.pushcapDynamicPredictionStore,\n      appPermissionStore = config.appPermissionStore,\n      optoutModelScorer = config.optoutModelScorer,\n      userTargetingPropertyStore = config.userTargetingPropertyStore,\n      ntabCaretFeedbackStore = config.ntabCaretFeedbackStore,\n      genericFeedbackStore = config.genericFeedbackStore,\n      inlineActionHistoryStore = config.inlineActionHistoryStore,\n      featureHydrator = config.featureHydrator,\n      openAppUserStore = config.openAppUserStore,\n      openedPushByHourAggregatedStore = config.openedPushByHourAggregatedStore,\n      geoduckStoreV2 = config.geoDuckV2Store,\n      superFollowEligibilityUserStore = config.superFollowEligibilityUserStore,\n      superFollowApplicationStatusStore = config.superFollowApplicationStatusStore\n    )(statsReceiver)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/module/ThriftWebFormsModule.scala",
    "content": "package com.twitter.frigate.pushservice.module\n\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsThriftWebFormsModule\nimport com.twitter.finatra.thrift.ThriftServer\nimport com.twitter.frigate.pushservice.thriftscala.PushService\n\nclass ThriftWebFormsModule(server: ThriftServer)\n    extends MtlsThriftWebFormsModule[PushService.MethodPerEndpoint](server) {\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/params/DeciderKey.scala",
    "content": "package com.twitter.frigate.pushservice.params\n\nimport com.twitter.servo.decider.DeciderKeyEnum\n\nobject DeciderKey extends DeciderKeyEnum {\n  val disableAllRelevance = Value(\"frigate_pushservice_disable_all_relevance\")\n  val disableHeavyRanking = Value(\"frigate_pushservice_disable_heavy_ranking\")\n  val restrictLightRanking = Value(\"frigate_pushservice_restrict_light_ranking\")\n  val downSampleLightRankingScribeCandidates = Value(\n    \"frigate_pushservice_down_sample_light_ranking_scribe_candidates\")\n  val entityGraphTweetRecsDeciderKey = Value(\"user_tweet_entity_graph_tweet_recs\")\n  val enablePushserviceWritesToNotificationServiceDeciderKey = Value(\n    \"frigate_pushservice_enable_writes_to_notification_service\")\n  val enablePushserviceWritesToNotificationServiceForAllEmployeesDeciderKey = Value(\n    \"frigate_pushservice_enable_writes_to_notification_service_for_employees\")\n  val enablePushserviceWritesToNotificationServiceForEveryoneDeciderKey = Value(\n    \"frigate_pushservice_enable_writes_to_notification_service_for_everyone\")\n  val enablePromptFeedbackFatigueResponseNoPredicateDeciderKey = Value(\n    \"frigate_pushservice_enable_ntab_feedback_prompt_response_no_filter_predicate\")\n  val enablePushserviceDeepbirdv2CanaryClusterDeciderKey = Value(\n    \"frigate_pushservice_canary_enable_deepbirdv2_canary_cluster\")\n  val enableUTEGSCForEarlybirdTweetsDecider = Value(\n    \"frigate_pushservice_enable_uteg_sc_for_eb_tweets\")\n  val enableTweetFavRecs = Value(\"frigate_pushservice_enable_tweet_fav_recs\")\n  val enableTweetRetweetRecs = Value(\"frigate_pushservice_enable_tweet_retweet_recs\")\n  val enablePushSendEventBus = Value(\"frigate_pushservice_enable_push_send_eventbus\")\n\n  val enableModelBasedPushcapAssignments = Value(\n    \"frigate_pushservice_enable_model_based_pushcap_assignments\")\n\n  val enableTweetAnnotationFeatureHydration = Value(\n    \"frigate_pushservice_enable_tweet_annotation_features_hydration\")\n  val enableMrRequestScribing = Value(\"frigate_pushservice_enable_mr_request_scribing\")\n  val enableHighQualityCandidateScoresScribing = Value(\n    \"frigate_pushservice_enable_high_quality_candidate_scribing\")\n  val enableHtlUserAuthorRealTimeAggregateFeatureHydration = Value(\n    \"frigate_pushservice_enable_htl_new_user_user_author_rta_hydration\")\n  val enableMrUserSemanticCoreFeaturesHydration = Value(\n    \"frigate_pushservice_enable_mr_user_semantic_core_feature_hydration\")\n  val enableMrUserSemanticCoreNoZeroFeaturesHydration = Value(\n    \"frigate_pushservice_enable_mr_user_semantic_core_no_zero_feature_hydration\")\n  val enableHtlOfflineUserAggregateExtendedFeaturesHydration = Value(\n    \"frigate_pushservice_enable_htl_offline_user_aggregate_extended_features_hydration\")\n  val enableNerErgFeaturesHydration = Value(\"frigate_pushservice_enable_ner_erg_features_hydration\")\n  val enableDaysSinceRecentResurrectionFeatureHydration = Value(\n    \"frigate_pushservice_enable_days_since_recent_resurrection_features_hydration\")\n  val enableUserPastAggregatesFeatureHydration = Value(\n    \"frigate_pushservice_enable_user_past_aggregates_features_hydration\")\n  val enableUserSignalLanguageFeatureHydration = Value(\n    \"frigate_pushservice_enable_user_signal_language_features_hydration\")\n  val enableUserPreferredLanguageFeatureHydration = Value(\n    \"frigate_pushservice_enable_user_preferred_language_features_hydration\")\n  val enablePredicateDetailedInfoScribing = Value(\n    \"frigate_pushservice_enable_predicate_detailed_info_scribing\")\n  val enablePushCapInfoScribing = Value(\"frigate_pushservice_enable_push_cap_info_scribing\")\n  val disableMLInFiltering = Value(\"frigate_pushservice_disable_ml_in_filtering\")\n  val useHydratedLabeledSendsForFeaturesDeciderKey = Value(\n    \"use_hydrated_labeled_sends_for_features\")\n  val verifyHydratedLabeledSendsForFeaturesDeciderKey = Value(\n    \"verify_hydrated_labeled_sends_for_features\")\n  val trainingDataDeciderKey = Value(\"frigate_notifier_quality_model_training_data\")\n  val skipMlModelPredicateDeciderKey = Value(\"skip_ml_model_predicate\")\n  val scribeModelFeaturesDeciderKey = Value(\"scribe_model_features\")\n  val scribeModelFeaturesWithoutHydratingNewFeaturesDeciderKey = Value(\n    \"scribe_model_features_without_hydrating_new_features\")\n  val scribeModelFeaturesForRequestScribe = Value(\"scribe_model_features_for_request_scribe\")\n  val enableMrUserSimclusterV2020FeaturesHydration = Value(\n    \"frigate_pushservice_enable_mr_user_simcluster_v2020_hydration\")\n  val enableMrUserSimclusterV2020NoZeroFeaturesHydration = Value(\n    \"frigate_pushservice_enable_mr_user_simcluster_v2020_no_zero_feature_hydration\")\n  val enableMrUserEngagedTweetTokensFeaturesHydration = Value(\n    \"frigate_pushservice_enable_mr_user_engaged_tweet_tokens_feature_hydration\")\n  val enableMrCandidateTweetTokensFeaturesHydration = Value(\n    \"frigate_pushservice_enable_mr_candidate_tweet_tokens_feature_hydration\")\n  val enableTopicEngagementRealTimeAggregatesFeatureHydration = Value(\n    \"frigate_pushservice_enable_topic_engagement_real_time_aggregates_feature_hydration\"\n  )\n  val enableUserTopicAggregatesFeatureHydration = Value(\n    \"frigate_pushservice_enable_user_topic_aggregates_feature_hydration\"\n  )\n  val enableDurationSinceLastVisitFeatureHydration = Value(\n    \"frigate_pushservice_enable_duration_since_last_visit_features_hydration\"\n  )\n  val enableTwistlyAggregatesFeatureHydration = Value(\n    \"frigate_pushservice_enable_twistly_agg_feature_hydration\"\n  )\n  val enableTwHINUserEngagementFeaturesHydration = Value(\n    \"frigate_pushservice_enable_twhin_user_engagement_features_hydration\"\n  )\n  val enableTwHINUserFollowFeaturesHydration = Value(\n    \"frigate_pushservice_enable_twhin_user_follow_features_hydration\"\n  )\n  val enableTwHINAuthorFollowFeaturesHydration = Value(\n    \"frigate_pushservice_enable_twhin_author_follow_features_hydration\"\n  )\n  val enableTweetTwHINFavFeaturesHydration = Value(\n    \"frigate_pushservice_enable_tweet_twhin_fav_features_hydration\"\n  )\n  val enableSpaceVisibilityLibraryFiltering = Value(\n    \"frigate_pushservice_enable_space_visibility_library_filtering\"\n  )\n  val enableVfFeatureHydrationSpaceShim = Value(\n    \"frigate_pushservice_enable_visibility_filtering_feature_hydration_in_space_shim\")\n  val enableUserTopicFollowFeatureSet = Value(\n    \"frigate_pushservice_enable_user_topic_follow_feature_hydration\")\n  val enableOnboardingNewUserFeatureSet = Value(\n    \"frigate_pushservice_enable_onboarding_new_user_feature_hydration\")\n  val enableMrUserTopicSparseContFeatureSet = Value(\n    \"frigate_pushservice_enable_mr_user_topic_sparse_cont_feature_hydration\"\n  )\n  val enableUserPenguinLanguageFeatureSet = Value(\n    \"frigate_pushservice_enable_user_penguin_language_feature_hydration\")\n  val enableMrUserHashspaceEmbeddingFeatureSet = Value(\n    \"frigate_pushservice_enable_mr_user_hashspace_embedding_feature_hydration\")\n  val enableMrUserAuthorSparseContFeatureSet = Value(\n    \"frigate_pushservice_enable_mr_user_author_sparse_cont_feature_hydration\"\n  )\n  val enableMrTweetSentimentFeatureSet = Value(\n    \"frigate_pushservice_enable_mr_tweet_sentiment_feature_hydration\"\n  )\n  val enableMrTweetAuthorAggregatesFeatureSet = Value(\n    \"frigate_pushservice_enable_mr_tweet_author_aggregates_feature_hydration\"\n  )\n  val enableUserGeoFeatureSet = Value(\"frigate_pushservice_enable_user_geo_feature_hydration\")\n  val enableAuthorGeoFeatureSet = Value(\"frigate_pushservice_enable_author_geo_feature_hydration\")\n\n  val rampupUserGeoFeatureSet = Value(\"frigate_pushservice_ramp_up_user_geo_feature_hydration\")\n  val rampupAuthorGeoFeatureSet = Value(\"frigate_pushservice_ramp_up_author_geo_feature_hydration\")\n\n  val enablePopGeoTweets = Value(\"frigate_pushservice_enable_pop_geo_tweets\")\n  val enableTrendsTweets = Value(\"frigate_pushservice_enable_trends_tweets\")\n  val enableTripGeoTweetCandidates = Value(\"frigate_pushservice_enable_trip_geo_tweets\")\n  val enableContentRecommenderMixerAdaptor = Value(\n    \"frigate_pushservice_enable_content_recommender_mixer_adaptor\")\n  val enableGenericCandidateAdaptor = Value(\"frigate_pushservice_enable_generic_candidate_adaptor\")\n  val enableTripGeoTweetContentMixerDarkTraffic = Value(\n    \"frigate_pushservice_enable_trip_geo_tweets_content_mixer_dark_traffic\")\n\n  val enableInsTraffic = Value(\"frigate_pushservice_enable_ins_traffic\")\n  val enableIsTweetTranslatable = Value(\"frigate_pushservice_enable_is_tweet_translatable\")\n\n  val enableMrTweetSimClusterFeatureSet = Value(\n    \"frigate_pushservice_enable_mr_tweet_simcluster_feature_hydration\")\n\n  val enableMrOfflineUserTweetTopicAggregate = Value(\n    \"frigate_pushservice_enable_mr_offline_user_tweet_topic_aggregate_hydration\")\n\n  val enableMrOfflineUserTweetSimClusterAggregate = Value(\n    \"frigate_pushservice_enable_mr_offline_user_tweet_simcluster_aggregate_hydration\"\n  )\n  val enableRealGraphV2FeatureHydration = Value(\n    \"frigate_pushservice_enable_real_graph_v2_features_hydration\")\n\n  val enableTweetBeTFeatureHydration = Value(\n    \"frigate_pushservice_enable_tweet_bet_features_hydration\")\n\n  val enableInvalidatingCachedHistoryStoreAfterWrites = Value(\n    \"frigate_pushservice_enable_invalidating_cached_history_store_after_writes\")\n\n  val enableInvalidatingCachedLoggedOutHistoryStoreAfterWrites = Value(\n    \"frigate_pushservice_enable_invalidating_cached_logged_out_history_store_after_writes\")\n\n  val enableUserSendTimeFeatureHydration = Value(\n    \"frigate_pushservice_enable_user_send_time_feature_hydration\"\n  )\n\n  val enablePnegMultimodalPredictionForF1Tweets = Value(\n    \"frigate_pushservice_enable_pneg_multimodal_prediction_for_f1_tweets\"\n  )\n\n  val enableScribingOonFavScoreForF1Tweets = Value(\n    \"frigate_pushservice_enable_oon_fav_scribe_for_f1_tweets\"\n  )\n\n  val enableMrUserUtcSendTimeAggregateFeaturesHydration = Value(\n    \"frigate_pushservice_enable_mr_user_utc_send_time_aggregate_hydration\"\n  )\n\n  val enableMrUserLocalSendTimeAggregateFeaturesHydration = Value(\n    \"frigate_pushservice_enable_mr_user_local_send_time_aggregate_hydration\"\n  )\n\n  val enableBqmlReportModelPredictionForF1Tweets = Value(\n    \"frigate_pushservice_enable_bqml_report_model_prediction_for_f1_tweets\"\n  )\n\n  val enableUserTwhinEmbeddingFeatureHydration = Value(\n    \"frigate_pushservice_enable_user_twhin_embedding_feature_hydration\"\n  )\n\n  val enableAuthorFollowTwhinEmbeddingFeatureHydration = Value(\n    \"frigate_pushservice_enable_author_follow_twhin_embedding_feature_hydration\"\n  )\n\n  val enableScribingMLFeaturesAsDataRecord = Value(\n    \"frigate_pushservice_enable_scribing_ml_features_as_datarecord\"\n  )\n\n  val enableDirectHydrationForUserFeatures = Value(\n    \"frigate_pushservice_enable_direct_hydration_for_user_features\"\n  )\n\n  val enableAuthorVerifiedFeatureHydration = Value(\n    \"frigate_pushservice_enable_author_verified_feature_hydration\"\n  )\n\n  val enableAuthorCreatorSubscriptionFeatureHydration = Value(\n    \"frigate_pushservice_enable_author_creator_subscription_feature_hydration\"\n  )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/params/PushConstants.scala",
    "content": "package com.twitter.frigate.pushservice.params\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.frigate.user_states.thriftscala.UserState\nimport java.util.Locale\n\nobject PushConstants {\n\n  final val ServiceProdEnvironmentName = \"prod\"\n\n  final val RestrictLightRankingCandidatesThreshold = 1\n\n  final val DownSampleLightRankingScribeCandidatesRate = 1\n\n  final val NewUserLookbackWindow = 1.days\n\n  final val PushCapInactiveUserAndroid = 1\n  final val PushCapInactiveUserIos = 1\n  final val PushCapLightOccasionalOpenerUserAndroid = 1\n  final val PushCapLightOccasionalOpenerUserIos = 1\n\n  final val UserStateToPushCapIos = Map(\n    UserState.Inactive.name -> PushCapInactiveUserIos,\n    UserState.LightOccasionalOpener.name -> PushCapLightOccasionalOpenerUserIos\n  )\n  final val UserStateToPushCapAndroid = Map(\n    UserState.Inactive.name -> PushCapInactiveUserAndroid,\n    UserState.LightOccasionalOpener.name -> PushCapLightOccasionalOpenerUserAndroid\n  )\n\n  final val AcceptableTimeSinceLastNegativeResponse = 1.days\n\n  final val DefaultLookBackForHistory = 1.hours\n\n  final val DefaultEventMediaUrl = \"\"\n\n  final val ConnectTabPushTapThrough = \"i/connect_people\"\n\n  final val AddressBookUploadTapThrough = \"i/flow/mr-address-book-upload\"\n  final val InterestPickerTapThrough = \"i/flow/mr-interest-picker\"\n  final val CompleteOnboardingInterestAddressTapThrough = \"i/flow/mr-interest-address\"\n\n  final val IndiaCountryCode = \"IN\"\n  final val JapanCountryCode = Locale.JAPAN.getCountry.toUpperCase\n  final val UKCountryCode = Locale.UK.getCountry.toUpperCase\n\n  final val IndiaTimeZoneCode = \"Asia/Kolkata\"\n  final val JapanTimeZoneCode = \"Asia/Tokyo\"\n  final val UKTimeZoneCode = \"Europe/London\"\n\n  final val countryCodeToTimeZoneMap = Map(\n    IndiaCountryCode -> IndiaTimeZoneCode,\n    JapanCountryCode -> JapanTimeZoneCode,\n    UKCountryCode -> UKTimeZoneCode\n  )\n\n  final val AbuseStrike_Top2Percent_Id = \"AbuseStrike_Top2Percent_Id\"\n  final val AbuseStrike_Top1Percent_Id = \"AbuseStrike_Top1Percent_Id\"\n  final val AbuseStrike_Top05Percent_Id = \"AbuseStrike_Top05Percent_Id\"\n  final val AbuseStrike_Top025Percent_Id = \"AbuseStrike_Top025Percent_Id\"\n  final val AllSpamReportsPerFav_Top1Percent_Id = \"AllSpamReportsPerFav_Top1Percent_Id\"\n  final val ReportsPerFav_Top1Percent_Id = \"ReportsPerFav_Top1Percent_Id\"\n  final val ReportsPerFav_Top2Percent_Id = \"ReportsPerFav_Top2Percent_Id\"\n  final val MediaUnderstanding_Nudity_Id = \"MediaUnderstanding_Nudity_Id\"\n  final val MediaUnderstanding_Beauty_Id = \"MediaUnderstanding_Beauty_Id\"\n  final val MediaUnderstanding_SinglePerson_Id = \"MediaUnderstanding_SinglePerson_Id\"\n  final val PornList_Id = \"PornList_Id\"\n  final val PornographyAndNsfwContent_Id = \"PornographyAndNsfwContent_Id\"\n  final val SexLife_Id = \"SexLife_Id\"\n  final val SexLifeOrSexualOrientation_Id = \"SexLifeOrSexualOrientation_Id\"\n  final val ProfanityFilter_Id = \"ProfanityFilter_Id\"\n  final val TweetSemanticCoreIdFeature = \"tweet.core.tweet.semantic_core_annotations\"\n  final val targetUserGenderFeatureName = \"Target.User.Gender\"\n  final val targetUserAgeFeatureName = \"Target.User.AgeBucket\"\n  final val targetUserPreferredLanguage = \"user.language.user.preferred_contents\"\n  final val tweetAgeInHoursFeatureName = \"RecTweet.TweetyPieResult.TweetAgeInHrs\"\n  final val authorActiveFollowerFeatureName = \"RecTweetAuthor.User.ActiveFollowers\"\n  final val favFeatureName = \"tweet.core.tweet_counts.favorite_count\"\n  final val sentFeatureName =\n    \"tweet.magic_recs_tweet_real_time_aggregates_v2.pair.v2.magicrecs.realtime.is_sent.any_feature.Duration.Top.count\"\n  final val authorSendCountFeatureName =\n    \"tweet_author_aggregate.pair.any_label.any_feature.28.days.count\"\n  final val authorReportCountFeatureName =\n    \"tweet_author_aggregate.pair.label.reportTweetDone.any_feature.28.days.count\"\n  final val authorDislikeCountFeatureName =\n    \"tweet_author_aggregate.pair.label.ntab.isDisliked.any_feature.28.days.count\"\n  final val TweetLikesFeatureName = \"tweet.core.tweet_counts.favorite_count\"\n  final val TweetRepliesFeatureName = \"tweet.core.tweet_counts.reply_count\"\n\n  final val EnableCopyFeaturesForIbis2ModelValues = \"has_copy_features\"\n\n  final val EmojiFeatureNameForIbis2ModelValues = \"emoji\"\n\n  final val TargetFeatureNameForIbis2ModelValues = \"target\"\n\n  final val CopyBodyExpIbisModelValues = \"enable_body_exp\"\n\n  final val TweetMediaEmbeddingBQKeyIds = Seq(\n    230, 110, 231, 111, 232, 233, 112, 113, 234, 235, 114, 236, 115, 237, 116, 117, 238, 118, 239,\n    119, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 240, 120, 241, 121, 242, 0, 1, 122, 243, 244, 123,\n    2, 124, 245, 3, 4, 246, 125, 5, 126, 247, 127, 248, 6, 128, 249, 7, 8, 129, 9, 20, 21, 22, 23,\n    24, 25, 26, 27, 28, 29, 250, 130, 251, 252, 131, 132, 253, 133, 254, 134, 255, 135, 136, 137,\n    138, 139, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 140, 141, 142, 143, 144, 145, 146, 147, 148,\n    149, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159,\n    50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 60,\n    61, 62, 63, 64, 65, 66, 67, 68, 69, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 70, 71,\n    72, 73, 74, 75, 76, 77, 78, 79, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 80, 81, 82,\n    83, 84, 85, 86, 87, 88, 89, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 90, 91, 92, 93,\n    94, 95, 96, 97, 98, 99, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213,\n    214, 215, 216, 217, 218, 219, 220, 100, 221, 101, 222, 223, 102, 224, 103, 104, 225, 105, 226,\n    227, 106, 107, 228, 108, 229, 109\n  )\n\n  final val SportsEventDomainId = 6L\n\n  final val OoncQualityCombinedScore = \"OoncQualityCombinedScore\"\n}\n\nobject PushQPSLimitConstants {\n\n  final val PerspectiveStoreQPS = 100000\n\n  final val IbisOrNTabQPSForRFPH = 100000\n\n  final val SocialGraphServiceBatchSize = 100000\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/params/PushEnums.scala",
    "content": "package com.twitter.frigate.pushservice.params\n\n/**\n * Enum for naming scores we will scribe for non-personalized high quality candidate generation\n */\nobject HighQualityScribingScores extends Enumeration {\n  type Name = Value\n  val HeavyRankingScore = Value\n  val NonPersonalizedQualityScoreUsingCnn = Value\n  val BqmlNsfwScore = Value\n  val BqmlReportScore = Value\n}\n\n/**\n * Enum for quality upranking transform\n */\nobject MrQualityUprankingTransformTypeEnum extends Enumeration {\n  val Linear = Value\n  val Sigmoid = Value\n}\n\n/**\n * Enum for quality partial upranking transform\n */\nobject MrQualityUprankingPartialTypeEnum extends Enumeration {\n  val All = Value\n  val Oon = Value\n}\n\n/**\n * Enum for bucket membership in DDG 10220 Mr Bold Title Favorite Retweet Notification experiment\n */\nobject MRBoldTitleFavoriteAndRetweetExperimentEnum extends Enumeration {\n  val ShortTitle = Value\n}\n\n/**\n * Enum for ML filtering predicates\n */\nobject QualityPredicateEnum extends Enumeration {\n  val WeightedOpenOrNtabClick = Value\n  val ExplicitOpenOrNtabClickFilter = Value\n  val AlwaysTrue = Value // Disable ML filtering\n}\n\n/**\n * Enum to specify normalization used in BigFiltering experiments\n */\nobject BigFilteringNormalizationEnum extends Enumeration {\n  val NormalizationDisabled = Value\n  val NormalizeByNotSendingScore = Value\n}\n\n/**\n * Enum for inline actions\n */\nobject InlineActionsEnum extends Enumeration {\n  val Favorite = Value\n  val Follow = Value\n  val Reply = Value\n  val Retweet = Value\n}\n\n/**\n * Enum for template format\n */\nobject IbisTemplateFormatEnum extends Enumeration {\n  val template1 = Value\n}\n\n/**\n * Enum for Store name for Top Tweets By Geo\n */\nobject TopTweetsForGeoCombination extends Enumeration {\n  val Default = Value\n  val AccountsTweetFavAsBackfill = Value\n  val AccountsTweetFavIntermixed = Value\n}\n\n/**\n * Enum for scoring function for Top Tweets By Geo\n */\nobject TopTweetsForGeoRankingFunction extends Enumeration {\n  val Score = Value\n  val GeohashLengthAndThenScore = Value\n}\n\n/**\n * Enum for which version of popgeo tweets to be using\n */\nobject PopGeoTweetVersion extends Enumeration {\n  val Prod = Value\n}\n\n/**\n * Enum for Subtext in Android header\n */\nobject SubtextForAndroidPushHeader extends Enumeration {\n  val None = Value\n  val TargetHandler = Value\n  val TargetTagHandler = Value\n  val TargetName = Value\n  val AuthorTagHandler = Value\n  val AuthorName = Value\n}\n\nobject NsfwTextDetectionModel extends Enumeration {\n  val ProdModel = Value\n  val RetrainedModel = Value\n}\n\nobject HighQualityCandidateGroupEnum extends Enumeration {\n  val AgeBucket = Value\n  val Language = Value\n  val Topic = Value\n  val Country = Value\n  val Admin0 = Value\n  val Admin1 = Value\n}\n\nobject CrtGroupEnum extends Enumeration {\n  val Twistly = Value\n  val Frs = Value\n  val F1 = Value\n  val Topic = Value\n  val Trip = Value\n  val GeoPop = Value\n  val Other = Value\n  val None = Value\n}\n\nobject SportGameEnum extends Enumeration {\n  val Soccer = Value\n  val Nfl = Value\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/params/PushFeatureSwitchParams.scala",
    "content": "package com.twitter.frigate.pushservice.params\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.frigate.pushservice.params.InlineActionsEnum._\nimport com.twitter.frigate.pushservice.params.HighQualityCandidateGroupEnum._\nimport com.twitter.timelines.configapi.DurationConversion\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.FSEnumParam\nimport com.twitter.timelines.configapi.FSEnumSeqParam\nimport com.twitter.timelines.configapi.FSParam\nimport com.twitter.timelines.configapi.HasDurationConversion\nimport com.twitter.util.Duration\n\nobject PushFeatureSwitchParams {\n\n  /**\n   * List of CRTs to uprank. Last CRT in sequence ends up on top of list\n   */\n  object ListOfCrtsToUpRank\n      extends FSParam[Seq[String]](\"rerank_candidates_crt_to_top\", default = Seq.empty[String])\n\n  object ListOfCrtsForOpenApp\n      extends FSParam[Seq[String]](\n        \"open_app_allowed_crts\",\n        default = Seq(\n          \"f1firstdegreetweet\",\n          \"f1firstdegreephoto\",\n          \"f1firstdegreevideo\",\n          \"geopoptweet\",\n          \"frstweet\",\n          \"trendtweet\",\n          \"hermituser\",\n          \"triangularloopuser\"\n        ))\n\n  /**\n   * List of CRTs to downrank. Last CRT in sequence ends up on bottom of list\n   */\n  object ListOfCrtsToDownRank\n      extends FSParam[Seq[String]](\n        name = \"rerank_candidates_crt_to_downrank\",\n        default = Seq.empty[String])\n\n  /**\n   * Param to enable VF filtering in Tweetypie (vs using VisibilityLibrary)\n   */\n  object EnableVFInTweetypie\n      extends FSParam[Boolean](\n        name = \"visibility_filtering_enable_vf_in_tweetypie\",\n        default = true\n      )\n\n  /**\n   * Number of max earlybird candidates\n   */\n  object NumberOfMaxEarlybirdInNetworkCandidatesParam\n      extends FSBoundedParam(\n        name = \"frigate_push_max_earlybird_in_network_candidates\",\n        default = 100,\n        min = 0,\n        max = 800\n      )\n\n  /**\n   * Number of max UserTweetEntityGraph candidates to query\n   */\n  object NumberOfMaxUTEGCandidatesQueriedParam\n      extends FSBoundedParam(\n        name = \"frigate_push_max_uteg_candidates_queried\",\n        default = 30,\n        min = 0,\n        max = 300\n      )\n\n  /**\n   * Param to control the max tweet age for users\n   */\n  object MaxTweetAgeParam\n      extends FSBoundedParam[Duration](\n        name = \"tweet_age_max_hours\",\n        default = 24.hours,\n        min = 1.hours,\n        max = 72.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the max tweet age for modeling-based candidates\n   */\n  object ModelingBasedCandidateMaxTweetAgeParam\n      extends FSBoundedParam[Duration](\n        name = \"tweet_age_candidate_generation_model_max_hours\",\n        default = 24.hours,\n        min = 1.hours,\n        max = 72.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the max tweet age for simcluster-based candidates\n   */\n  object GeoPopTweetMaxAgeInHours\n      extends FSBoundedParam[Duration](\n        name = \"tweet_age_geo_pop_max_hours\",\n        default = 24.hours,\n        min = 1.hours,\n        max = 120.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the max tweet age for simcluster-based candidates\n   */\n  object SimclusterBasedCandidateMaxTweetAgeParam\n      extends FSBoundedParam[Duration](\n        name = \"tweet_age_simcluster_max_hours\",\n        default = 24.hours,\n        min = 24.hours,\n        max = 48.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the max tweet age for Detopic-based candidates\n   */\n  object DetopicBasedCandidateMaxTweetAgeParam\n      extends FSBoundedParam[Duration](\n        name = \"tweet_age_detopic_max_hours\",\n        default = 24.hours,\n        min = 24.hours,\n        max = 48.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the max tweet age for F1 candidates\n   */\n  object F1CandidateMaxTweetAgeParam\n      extends FSBoundedParam[Duration](\n        name = \"tweet_age_f1_max_hours\",\n        default = 24.hours,\n        min = 1.hours,\n        max = 96.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the max tweet age for Explore Video Tweet\n   */\n  object ExploreVideoTweetAgeParam\n      extends FSBoundedParam[Duration](\n        name = \"explore_video_tweets_age_max_hours\",\n        default = 48.hours,\n        min = 1.hours,\n        max = 336.hours // Two weeks\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to no send for new user playbook push if user login for past hours\n   */\n  object NewUserPlaybookAllowedLastLoginHours\n      extends FSBoundedParam[Duration](\n        name = \"new_user_playbook_allowed_last_login_hours\",\n        default = 0.hours,\n        min = 0.hours,\n        max = 72.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * The batch size of RefreshForPushHandler's Take step\n   */\n  object NumberOfMaxCandidatesToBatchInRFPHTakeStep\n      extends FSBoundedParam(\n        name = \"frigate_push_rfph_batch_take_max_size\",\n        default = 1,\n        min = 1,\n        max = 10\n      )\n\n  /**\n   * The maximum number of candidates to batch for Importance Sampling\n   */\n  object NumberOfMaxCandidatesToBatchForImportanceSampling\n      extends FSBoundedParam(\n        name = \"frigate_push_rfph_max_candidates_to_batch_for_importance_sampling\",\n        default = 65,\n        min = 1,\n        max = 500\n      )\n\n  /**\n   * Maximum number of regular MR push in 24.hours/daytime/nighttime\n   */\n  object MaxMrPushSends24HoursParam\n      extends FSBoundedParam(\n        name = \"pushcap_max_sends_24hours\",\n        default = 5,\n        min = 0,\n        max = 12\n      )\n\n  /**\n   * Maximum number of regular MR ntab only channel in 24.hours/daytime/nighttime\n   */\n  object MaxMrNtabOnlySends24HoursParamV3\n      extends FSBoundedParam(\n        name = \"pushcap_max_sends_24hours_ntabonly_v3\",\n        default = 5,\n        min = 0,\n        max = 12\n      )\n\n  /**\n   * Maximum number of regular MR ntab only in 24.hours/daytime/nighttime\n   */\n  object MaxMrPushSends24HoursNtabOnlyUsersParam\n      extends FSBoundedParam(\n        name = \"pushcap_max_sends_24hours_ntab_only\",\n        default = 5,\n        min = 0,\n        max = 10\n      )\n\n  /**\n   * Customized PushCap offset (e.g., to the predicted value)\n   */\n  object CustomizedPushCapOffset\n      extends FSBoundedParam[Int](\n        name = \"pushcap_customized_offset\",\n        default = 0,\n        min = -2,\n        max = 4\n      )\n\n  /**\n   * Param to enable restricting minimum pushcap assigned with ML models\n   * */\n  object EnableRestrictedMinModelPushcap\n      extends FSParam[Boolean](\n        name = \"pushcap_restricted_model_min_enable\",\n        default = false\n      )\n\n  /**\n   * Param to specify the minimum pushcap allowed to be assigned with ML models\n   * */\n  object RestrictedMinModelPushcap\n      extends FSBoundedParam[Int](\n        name = \"pushcap_restricted_model_min_value\",\n        default = 1,\n        min = 0,\n        max = 9\n      )\n\n  object EnablePushcapRefactor\n      extends FSParam[Boolean](\n        name = \"pushcap_enable_refactor\",\n        default = false\n      )\n\n  /**\n   * Enables the restrict step in pushservice for a given user\n   *\n   * Setting this to false may cause a large number of candidates to be passed on to filtering/take\n   * step in RefreshForPushHandler, increasing the service latency significantly\n   */\n  object EnableRestrictStep extends FSParam[Boolean](\"frigate_push_rfph_restrict_step_enable\", true)\n\n  /**\n   * The number of candidates that are able to pass through the restrict step.\n   */\n  object RestrictStepSize\n      extends FSBoundedParam(\n        name = \"frigate_push_rfph_restrict_step_size\",\n        default = 65,\n        min = 65,\n        max = 200\n      )\n\n  /**\n   * Number of max crMixer candidates to send.\n   */\n  object NumberOfMaxCrMixerCandidatesParam\n      extends FSBoundedParam(\n        name = \"cr_mixer_migration_max_num_of_candidates_to_return\",\n        default = 400,\n        min = 0,\n        max = 2000\n      )\n\n  /**\n   * Duration between two MR pushes\n   */\n  object MinDurationSincePushParam\n      extends FSBoundedParam[Duration](\n        name = \"pushcap_min_duration_since_push_hours\",\n        default = 4.hours,\n        min = 0.hours,\n        max = 72.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Each Phase duration to gradually ramp up MagicRecs for new users\n   */\n  object GraduallyRampUpPhaseDurationDays\n      extends FSBoundedParam[Duration](\n        name = \"pushcap_gradually_ramp_up_phase_duration_days\",\n        default = 3.days,\n        min = 2.days,\n        max = 7.days\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Param to specify interval for target pushcap fatigue\n   */\n  object TargetPushCapFatigueIntervalHours\n      extends FSBoundedParam[Duration](\n        name = \"pushcap_fatigue_interval_hours\",\n        default = 24.hours,\n        min = 1.hour,\n        max = 240.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to specify interval for target ntabOnly fatigue\n   */\n  object TargetNtabOnlyCapFatigueIntervalHours\n      extends FSBoundedParam[Duration](\n        name = \"pushcap_ntabonly_fatigue_interval_hours\",\n        default = 24.hours,\n        min = 1.hour,\n        max = 240.hours\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to use completely explicit push cap instead of LTV/modeling-based\n   */\n  object EnableExplicitPushCap\n      extends FSParam[Boolean](\n        name = \"pushcap_explicit_enable\",\n        default = false\n      )\n\n  /**\n   * Param to control explicit push cap (non-LTV)\n   */\n  object ExplicitPushCap\n      extends FSBoundedParam[Int](\n        name = \"pushcap_explicit_value\",\n        default = 1,\n        min = 0,\n        max = 20\n      )\n\n  /**\n   * Parameters for percentile thresholds of OpenOrNtabClick model in MR filtering model refreshing DDG\n   */\n  object PercentileThresholdCohort1\n      extends FSBoundedParam[Double](\n        name = \"frigate_push_modeling_percentile_threshold_cohort1\",\n        default = 0.65,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object PercentileThresholdCohort2\n      extends FSBoundedParam[Double](\n        name = \"frigate_push_modeling_percentile_threshold_cohort2\",\n        default = 0.03,\n        min = 0.0,\n        max = 1.0\n      )\n  object PercentileThresholdCohort3\n      extends FSBoundedParam[Double](\n        name = \"frigate_push_modeling_percentile_threshold_cohort3\",\n        default = 0.03,\n        min = 0.0,\n        max = 1.0\n      )\n  object PercentileThresholdCohort4\n      extends FSBoundedParam[Double](\n        name = \"frigate_push_modeling_percentile_threshold_cohort4\",\n        default = 0.06,\n        min = 0.0,\n        max = 1.0\n      )\n  object PercentileThresholdCohort5\n      extends FSBoundedParam[Double](\n        name = \"frigate_push_modeling_percentile_threshold_cohort5\",\n        default = 0.06,\n        min = 0.0,\n        max = 1.0\n      )\n  object PercentileThresholdCohort6\n      extends FSBoundedParam[Double](\n        name = \"frigate_push_modeling_percentile_threshold_cohort6\",\n        default = 0.8,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Parameters for percentile threshold list of OpenOrNtabCLick model in MR percentile grid search experiments\n   */\n  object MrPercentileGridSearchThresholdsCohort1\n      extends FSParam[Seq[Double]](\n        name = \"frigate_push_modeling_percentile_grid_search_thresholds_cohort1\",\n        default = Seq(0.8, 0.75, 0.65, 0.55, 0.45, 0.35, 0.25)\n      )\n  object MrPercentileGridSearchThresholdsCohort2\n      extends FSParam[Seq[Double]](\n        name = \"frigate_push_modeling_percentile_grid_search_thresholds_cohort2\",\n        default = Seq(0.15, 0.12, 0.1, 0.08, 0.06, 0.045, 0.03)\n      )\n  object MrPercentileGridSearchThresholdsCohort3\n      extends FSParam[Seq[Double]](\n        name = \"frigate_push_modeling_percentile_grid_search_thresholds_cohort3\",\n        default = Seq(0.15, 0.12, 0.1, 0.08, 0.06, 0.045, 0.03)\n      )\n  object MrPercentileGridSearchThresholdsCohort4\n      extends FSParam[Seq[Double]](\n        name = \"frigate_push_modeling_percentile_grid_search_thresholds_cohort4\",\n        default = Seq(0.15, 0.12, 0.1, 0.08, 0.06, 0.045, 0.03)\n      )\n  object MrPercentileGridSearchThresholdsCohort5\n      extends FSParam[Seq[Double]](\n        name = \"frigate_push_modeling_percentile_grid_search_thresholds_cohort5\",\n        default = Seq(0.3, 0.2, 0.15, 0.1, 0.08, 0.06, 0.05)\n      )\n  object MrPercentileGridSearchThresholdsCohort6\n      extends FSParam[Seq[Double]](\n        name = \"frigate_push_modeling_percentile_grid_search_thresholds_cohort6\",\n        default = Seq(0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2)\n      )\n\n  /**\n   * Parameters for threshold list of OpenOrNtabClick model in MF grid search experiments\n   */\n  object MfGridSearchThresholdsCohort1\n      extends FSParam[Seq[Double]](\n        name = \"frigate_push_modeling_mf_grid_search_thresholds_cohort1\",\n        default = Seq(0.030, 0.040, 0.050, 0.062, 0.070, 0.080, 0.090) // default: 0.062\n      )\n  object MfGridSearchThresholdsCohort2\n      extends FSParam[Seq[Double]](\n        name = \"frigate_push_modeling_mf_grid_search_thresholds_cohort2\",\n        default = Seq(0.005, 0.010, 0.015, 0.020, 0.030, 0.040, 0.050) // default: 0.020\n      )\n  object MfGridSearchThresholdsCohort3\n      extends FSParam[Seq[Double]](\n        name = \"frigate_push_modeling_mf_grid_search_thresholds_cohort3\",\n        default = Seq(0.010, 0.015, 0.020, 0.025, 0.035, 0.045, 0.055) // default: 0.025\n      )\n  object MfGridSearchThresholdsCohort4\n      extends FSParam[Seq[Double]](\n        name = \"frigate_push_modeling_mf_grid_search_thresholds_cohort4\",\n        default = Seq(0.015, 0.020, 0.025, 0.030, 0.040, 0.050, 0.060) // default: 0.030\n      )\n  object MfGridSearchThresholdsCohort5\n      extends FSParam[Seq[Double]](\n        name = \"frigate_push_modeling_mf_grid_search_thresholds_cohort5\",\n        default = Seq(0.035, 0.040, 0.045, 0.050, 0.060, 0.070, 0.080) // default: 0.050\n      )\n  object MfGridSearchThresholdsCohort6\n      extends FSParam[Seq[Double]](\n        name = \"frigate_push_modeling_mf_grid_search_thresholds_cohort6\",\n        default = Seq(0.040, 0.045, 0.050, 0.055, 0.065, 0.075, 0.085) // default: 0.055\n      )\n\n  /**\n   * Param to specify which global optout models to use to first predict the global scores for users\n   */\n  object GlobalOptoutModelParam\n      extends FSParam[Seq[OptoutModel.ModelNameType]](\n        name = \"optout_model_global_model_ids\",\n        default = Seq.empty[OptoutModel.ModelNameType]\n      )\n\n  /**\n   * Param to specify which optout model to use according to the experiment bucket\n   */\n  object BucketOptoutModelParam\n      extends FSParam[OptoutModel.ModelNameType](\n        name = \"optout_model_bucket_model_id\",\n        default = OptoutModel.D0_has_realtime_features\n      )\n\n  /*\n   * Param to enable candidate generation model\n   * */\n  object EnableCandidateGenerationModelParam\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_enable\",\n        default = false\n      )\n\n  object EnableOverrideForSportsCandidates\n      extends FSParam[Boolean](name = \"magicfanout_sports_event_enable_override\", default = true)\n\n  object EnableEventIdBasedOverrideForSportsCandidates\n      extends FSParam[Boolean](\n        name = \"magicfanout_sports_event_enable_event_id_based_override\",\n        default = true)\n\n  /**\n   * Param to specify the threshold to determine if a user’s optout score is high enough to enter the experiment.\n   */\n  object GlobalOptoutThresholdParam\n      extends FSParam[Seq[Double]](\n        name = \"optout_model_global_thresholds\",\n        default = Seq(1.0, 1.0)\n      )\n\n  /**\n   * Param to specify the threshold to determine if a user’s optout score is high enough to be assigned\n   * with a reduced pushcap based on the bucket membership.\n   */\n  object BucketOptoutThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"optout_model_bucket_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Param to specify the reduced pushcap value if the optout probability predicted by the bucket\n   * optout model is higher than the specified bucket optout threshold.\n   */\n  object OptoutExptPushCapParam\n      extends FSBoundedParam[Int](\n        name = \"optout_model_expt_push_cap\",\n        default = 10,\n        min = 0,\n        max = 10\n      )\n\n  /**\n   * Param to specify the thresholds to determine which push cap slot the user should be assigned to\n   * according to the optout score. For example,the slot thresholds are [0.1, 0.2, ..., 1.0], the user\n   * is assigned to the second slot if the optout score is in (0.1, 0.2].\n   */\n  object BucketOptoutSlotThresholdParam\n      extends FSParam[Seq[Double]](\n        name = \"optout_model_bucket_slot_thresholds\",\n        default = Seq.empty[Double]\n      )\n\n  /**\n   * Param to specify the adjusted push cap of each slot. For example, if the slot push caps are [1, 2, ..., 10]\n   * and the user is assigned to the 2nd slot according to the optout score, the push cap of the user\n   * will be adjusted to 2.\n   */\n  object BucketOptoutSlotPushcapParam\n      extends FSParam[Seq[Int]](\n        name = \"optout_model_bucket_slot_pushcaps\",\n        default = Seq.empty[Int]\n      )\n\n  /**\n   * Param to specify if the optout score based push cap adjustment is enabled\n   */\n  object EnableOptoutAdjustedPushcap\n      extends FSParam[Boolean](\n        \"optout_model_enable_optout_adjusted_pushcap\",\n        false\n      )\n\n  /**\n   * Param to specify which weighted open or ntab click model to use\n   */\n  object WeightedOpenOrNtabClickRankingModelParam\n      extends FSParam[WeightedOpenOrNtabClickModel.ModelNameType](\n        name = \"frigate_push_modeling_oonc_ranking_model_id\",\n        default = WeightedOpenOrNtabClickModel.Periodically_Refreshed_Prod_Model\n      )\n\n  /**\n   * Param to disable heavy ranker\n   */\n  object DisableHeavyRankingModelFSParam\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_disable_heavy_ranking\",\n        default = false\n      )\n\n  /**\n   * Param to specify which weighted open or ntab click model to use for Android modelling experiment\n   */\n  object WeightedOpenOrNtabClickRankingModelForAndroidParam\n      extends FSParam[WeightedOpenOrNtabClickModel.ModelNameType](\n        name = \"frigate_push_modeling_oonc_ranking_model_for_android_id\",\n        default = WeightedOpenOrNtabClickModel.Periodically_Refreshed_Prod_Model\n      )\n\n  /**\n   * Param to specify which weighted open or ntab click model to use for FILTERING\n   */\n  object WeightedOpenOrNtabClickFilteringModelParam\n      extends FSParam[WeightedOpenOrNtabClickModel.ModelNameType](\n        name = \"frigate_push_modeling_oonc_filtering_model_id\",\n        default = WeightedOpenOrNtabClickModel.Periodically_Refreshed_Prod_Model\n      )\n\n  /**\n   * Param to specify which quality predicate to use for ML filtering\n   */\n  object QualityPredicateIdParam\n      extends FSEnumParam[QualityPredicateEnum.type](\n        name = \"frigate_push_modeling_quality_predicate_id\",\n        default = QualityPredicateEnum.WeightedOpenOrNtabClick,\n        enum = QualityPredicateEnum\n      )\n\n  /**\n   * Param to control threshold for any quality predicates using explicit thresholds\n   */\n  object QualityPredicateExplicitThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"frigate_push_modeling_quality_predicate_explicit_threshold\",\n        default = 0.1,\n        min = 0,\n        max = 1)\n\n  /**\n   * MagicFanout relaxed eventID fatigue interval (when we want to enable multiple updates for the same event)\n   */\n  object MagicFanoutRelaxedEventIdFatigueIntervalInHours\n      extends FSBoundedParam[Int](\n        name = \"frigate_push_magicfanout_relaxed_event_id_fatigue_interval_in_hours\",\n        default = 24,\n        min = 0,\n        max = 720\n      )\n\n  /**\n   * MagicFanout DenyListed Countries\n   */\n  object MagicFanoutDenyListedCountries\n      extends FSParam[Seq[String]](\n        \"frigate_push_magicfanout_denylisted_countries\",\n        Seq.empty[String])\n\n  object MagicFanoutSportsEventDenyListedCountries\n      extends FSParam[Seq[String]](\n        \"magicfanout_sports_event_denylisted_countries\",\n        Seq.empty[String])\n\n  /**\n   * MagicFanout maximum erg rank for a given push event for non heavy users\n   */\n  object MagicFanoutRankErgThresholdNonHeavy\n      extends FSBoundedParam[Int](\n        name = \"frigate_push_magicfanout_erg_rank_threshold_non_heavy\",\n        default = 25,\n        min = 1,\n        max = 50\n      )\n\n  /**\n   * MagicFanout maximum erg rank for a given push event for heavy users\n   */\n  object MagicFanoutRankErgThresholdHeavy\n      extends FSBoundedParam[Int](\n        name = \"frigate_push_magicfanout_erg_rank_threshold_heavy\",\n        default = 20,\n        min = 1,\n        max = 50\n      )\n\n  object EnablePushMixerReplacingAllSources\n      extends FSParam[Boolean](\n        name = \"push_mixer_enable_replacing_all_sources\",\n        default = false\n      )\n\n  object EnablePushMixerReplacingAllSourcesWithControl\n      extends FSParam[Boolean](\n        name = \"push_mixer_enable_replacing_all_sources_with_control\",\n        default = false\n      )\n\n  object EnablePushMixerReplacingAllSourcesWithExtra\n      extends FSParam[Boolean](\n        name = \"push_mixer_enable_replacing_all_sources_with_extra\",\n        default = false\n      )\n\n  object EnablePushMixerSource\n      extends FSParam[Boolean](\n        name = \"push_mixer_enable_source\",\n        default = false\n      )\n\n  object PushMixerMaxResults\n      extends FSBoundedParam[Int](\n        name = \"push_mixer_max_results\",\n        default = 10,\n        min = 1,\n        max = 5000\n      )\n\n  /**\n   * Enable tweets from trends that have been annotated by curators\n   */\n  object EnableCuratedTrendTweets\n      extends FSParam[Boolean](name = \"trend_tweet_curated_trends_enable\", default = false)\n\n  /**\n   * Enable tweets from trends that haven't been annotated by curators\n   */\n  object EnableNonCuratedTrendTweets\n      extends FSParam[Boolean](name = \"trend_tweet_non_curated_trends_enable\", default = false)\n\n  /**\n   * Maximum trend tweet notifications in fixed duration\n   */\n  object MaxTrendTweetNotificationsInDuration\n      extends FSBoundedParam[Int](\n        name = \"trend_tweet_max_notifications_in_duration\",\n        min = 0,\n        default = 0,\n        max = 20)\n\n  /**\n   * Duration in days over which trend tweet notifications fatigue is applied\n   */\n  object TrendTweetNotificationsFatigueDuration\n      extends FSBoundedParam[Duration](\n        name = \"trend_tweet_notifications_fatigue_in_days\",\n        default = 1.day,\n        min = Duration.Bottom,\n        max = Duration.Top\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Maximum number of trends candidates to query from event-recos endpoint\n   */\n  object MaxRecommendedTrendsToQuery\n      extends FSBoundedParam[Int](\n        name = \"trend_tweet_max_trends_to_query\",\n        min = 0,\n        default = 0,\n        max = 100)\n\n  /**\n   * Fix missing event-associated interests in MagicFanoutNoOptoutInterestsPredicate\n   */\n  object MagicFanoutFixNoOptoutInterestsBugParam\n      extends FSParam[Boolean](\"frigate_push_magicfanout_fix_no_optout_interests\", default = true)\n\n  object EnableSimclusterOfflineAggFeatureForExpt\n      extends FSParam[Boolean](\"frigate_enable_simcluster_offline_agg_feature\", false)\n\n  /**\n   * Param to enable removal of UTT domain for\n   */\n  object ApplyMagicFanoutBroadEntityInterestRankThresholdPredicate\n      extends FSParam[Boolean](\n        \"frigate_push_magicfanout_broad_entity_interest_rank_threshold_predicate\",\n        false\n      )\n\n  object HydrateEventReasonsFeatures\n      extends FSParam[Boolean](\n        name = \"frigate_push_magicfanout_hydrate_event_reasons_features\",\n        false\n      )\n\n  /**\n   * Param to enable online MR history features\n   */\n  object EnableHydratingOnlineMRHistoryFeatures\n      extends FSParam[Boolean](\n        name = \"feature_hydration_online_mr_history\",\n        default = false\n      )\n\n  /**\n   * Param to enable bold title on favorite and retweet push copy for Android in DDG 10220\n   */\n  object MRBoldTitleFavoriteAndRetweetParam\n      extends FSEnumParam[MRBoldTitleFavoriteAndRetweetExperimentEnum.type](\n        name = \"frigate_push_bold_title_favorite_and_retweet_id\",\n        default = MRBoldTitleFavoriteAndRetweetExperimentEnum.ShortTitle,\n        enum = MRBoldTitleFavoriteAndRetweetExperimentEnum\n      )\n\n  /**\n   * Param to enable high priority push\n   */\n  object EnableHighPriorityPush\n      extends FSParam[Boolean](\"frigate_push_magicfanout_enable_high_priority_push\", false)\n\n  /**\n   * Param to redirect sports crt event to a custom url\n   */\n  object EnableSearchURLRedirectForSportsFanout\n      extends FSParam[Boolean](\"magicfanout_sports_event_enable_search_url_redirect\", false)\n\n  /**\n   * Param to enable score fanout notification for sports\n   */\n  object EnableScoreFanoutNotification\n      extends FSParam[Boolean](\"magicfanout_sports_event_enable_score_fanout\", false)\n\n  /**\n   * Param to add custom search url for sports crt event\n   */\n  object SearchURLRedirectForSportsFanout\n      extends FSParam[String](\n        name = \"magicfanout_sports_event_search_url_redirect\",\n        default = \"https://twitter.com/explore/tabs/ipl\",\n      )\n\n  /**\n   * Param to enable high priority sports push\n   */\n  object EnableHighPrioritySportsPush\n      extends FSParam[Boolean](\"magicfanout_sports_event_enable_high_priority_push\", false)\n\n  /**\n   * Param to control rank threshold for magicfanout user follow\n   */\n  object MagicFanoutRealgraphRankThreshold\n      extends FSBoundedParam[Int](\n        name = \"magicfanout_realgraph_threshold\",\n        default = 500,\n        max = 500,\n        min = 100\n      )\n\n  /**\n   * Topic score threshold for topic proof tweet candidates topic annotations\n   * */\n  object TopicProofTweetCandidatesTopicScoreThreshold\n      extends FSBoundedParam[Double](\n        name = \"topics_as_social_proof_topic_score_threshold\",\n        default = 0.0,\n        min = 0.0,\n        max = 100.0\n      )\n\n  /**\n   * Enable Topic Proof Tweet Recs\n   */\n  object EnableTopicProofTweetRecs\n      extends FSParam[Boolean](name = \"topics_as_social_proof_enable\", default = true)\n\n  /**\n   * Enable health filters for topic tweet notifications\n   */\n  object EnableHealthFiltersForTopicProofTweet\n      extends FSParam[Boolean](\n        name = \"topics_as_social_proof_enable_health_filters\",\n        default = false)\n\n  /**\n   * Disable health filters for CrMixer candidates\n   */\n  object DisableHealthFiltersForCrMixerCandidates\n      extends FSParam[Boolean](\n        name = \"health_and_quality_filter_disable_for_crmixer_candidates\",\n        default = false)\n\n  object EnableMagicFanoutNewsForYouNtabCopy\n      extends FSParam[Boolean](name = \"send_handler_enable_nfy_ntab_copy\", default = false)\n\n  /**\n   * Param to enable semi-personalized high quality candidates in pushservice\n   * */\n  object HighQualityCandidatesEnableCandidateSource\n      extends FSParam[Boolean](\n        name = \"high_quality_candidates_enable_candidate_source\",\n        default = false\n      )\n\n  /**\n   * Param to decide semi-personalized high quality candidates\n   * */\n  object HighQualityCandidatesEnableGroups\n      extends FSEnumSeqParam[HighQualityCandidateGroupEnum.type](\n        name = \"high_quality_candidates_enable_groups_ids\",\n        default = Seq(AgeBucket, Language),\n        enum = HighQualityCandidateGroupEnum\n      )\n\n  /**\n   * Param to decide semi-personalized high quality candidates\n   * */\n  object HighQualityCandidatesNumberOfCandidates\n      extends FSBoundedParam[Int](\n        name = \"high_quality_candidates_number_of_candidates\",\n        default = 0,\n        min = 0,\n        max = Int.MaxValue\n      )\n\n  /**\n   * Param to enable small domain falling back to bigger domains for high quality candidates in pushservice\n   * */\n  object HighQualityCandidatesEnableFallback\n      extends FSParam[Boolean](\n        name = \"high_quality_candidates_enable_fallback\",\n        default = false\n      )\n\n  /**\n   * Param to decide whether to fallback to bigger domain for high quality candidates\n   * */\n  object HighQualityCandidatesMinNumOfCandidatesToFallback\n      extends FSBoundedParam[Int](\n        name = \"high_quality_candidates_min_num_of_candidates_to_fallback\",\n        default = 50,\n        min = 0,\n        max = Int.MaxValue\n      )\n\n  /**\n   * Param to specific source ids for high quality candidates\n   * */\n  object HighQualityCandidatesFallbackSourceIds\n      extends FSParam[Seq[String]](\n        name = \"high_quality_candidates_fallback_source_ids\",\n        default = Seq(\"HQ_C_COUNT_PASS_QUALITY_SCORES\"))\n\n  /**\n   * Param to decide groups for semi-personalized high quality candidates\n   * */\n  object HighQualityCandidatesFallbackEnabledGroups\n      extends FSEnumSeqParam[HighQualityCandidateGroupEnum.type](\n        name = \"high_quality_candidates_fallback_enabled_groups_ids\",\n        default = Seq(Country),\n        enum = HighQualityCandidateGroupEnum\n      )\n\n  /**\n   * Param to control what heavy ranker model to use for scribing scores\n   */\n  object HighQualityCandidatesHeavyRankingModel\n      extends FSParam[String](\n        name = \"high_quality_candidates_heavy_ranking_model\",\n        default = \"Periodically_Refreshed_Prod_Model_V11\"\n      )\n\n  /**\n   * Param to control what non personalized quality \"Cnn\" model to use for scribing scores\n   */\n  object HighQualityCandidatesNonPersonalizedQualityCnnModel\n      extends FSParam[String](\n        name = \"high_quality_candidates_non_personalized_quality_cnn_model\",\n        default = \"Q1_2023_Mr_Tf_Quality_Model_cnn\"\n      )\n\n  /**\n   * Param to control what nsfw health model to use for scribing scores\n   */\n  object HighQualityCandidatesBqmlNsfwModel\n      extends FSParam[String](\n        name = \"high_quality_candidates_bqml_nsfw_model\",\n        default = \"Q2_2022_Mr_Bqml_Health_Model_NsfwV0\"\n      )\n\n  /**\n   * Param to control what reportodel to use for scribing scores\n   */\n  object HighQualityCandidatesBqmlReportModel\n      extends FSParam[String](\n        name = \"high_quality_candidates_bqml_report_model\",\n        default = \"Q3_2022_15266_Mr_Bqml_Non_Personalized_Report_Model_with_Media_Embeddings\"\n      )\n\n  /**\n   * Param to specify the threshold to determine if a tweet contains nudity media\n   */\n  object TweetMediaSensitiveCategoryThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"tweet_media_sensitive_category_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Param to boost candidates from subscription creators\n   */\n  object BoostCandidatesFromSubscriptionCreators\n      extends FSParam[Boolean](\n        name = \"subscription_enable_boost_candidates_from_active_creators\",\n        default = false\n      )\n\n  /**\n   * Param to soft rank candidates from subscription creators\n   */\n  object SoftRankCandidatesFromSubscriptionCreators\n      extends FSParam[Boolean](\n        name = \"subscription_enable_soft_rank_candidates_from_active_creators\",\n        default = false\n      )\n\n  /**\n   * Param as factor to control how much we want to boost creator tweets\n   */\n  object SoftRankFactorForSubscriptionCreators\n      extends FSBoundedParam[Double](\n        name = \"subscription_soft_rank_factor_for_boost\",\n        default = 1.0,\n        min = 0.0,\n        max = Double.MaxValue\n      )\n\n  /**\n   * Param to enable new OON copy for Push Notifications\n   */\n  object EnableNewMROONCopyForPush\n      extends FSParam[Boolean](\n        name = \"mr_copy_enable_new_mr_oon_copy_push\",\n        default = true\n      )\n\n  /**\n   * Param to enable generated inline actions on OON Notifications\n   */\n  object EnableOONGeneratedInlineActions\n      extends FSParam[Boolean](\n        name = \"mr_inline_enable_oon_generated_actions\",\n        default = false\n      )\n\n  /**\n   * Param to control dynamic inline actions for Out-of-Network copies\n   */\n  object OONTweetDynamicInlineActionsList\n      extends FSEnumSeqParam[InlineActionsEnum.type](\n        name = \"mr_inline_oon_tweet_dynamic_action_ids\",\n        default = Seq(Follow, Retweet, Favorite),\n        enum = InlineActionsEnum\n      )\n\n  object HighOONCTweetFormat\n      extends FSEnumParam[IbisTemplateFormatEnum.type](\n        name = \"mr_copy_high_oonc_format_id\",\n        default = IbisTemplateFormatEnum.template1,\n        enum = IbisTemplateFormatEnum\n      )\n\n  object LowOONCTweetFormat\n      extends FSEnumParam[IbisTemplateFormatEnum.type](\n        name = \"mr_copy_low_oonc_format_id\",\n        default = IbisTemplateFormatEnum.template1,\n        enum = IbisTemplateFormatEnum\n      )\n\n  /**\n   * Param to enable dynamic inline actions based on FSParams for Tweet copies (not OON)\n   */\n  object EnableTweetDynamicInlineActions\n      extends FSParam[Boolean](\n        name = \"mr_inline_enable_tweet_dynamic_actions\",\n        default = false\n      )\n\n  /**\n   * Param to control dynamic inline actions for Tweet copies (not OON)\n   */\n  object TweetDynamicInlineActionsList\n      extends FSEnumSeqParam[InlineActionsEnum.type](\n        name = \"mr_inline_tweet_dynamic_action_ids\",\n        default = Seq(Reply, Retweet, Favorite),\n        enum = InlineActionsEnum\n      )\n\n  object UseInlineActionsV1\n      extends FSParam[Boolean](\n        name = \"mr_inline_use_inline_action_v1\",\n        default = true\n      )\n\n  object UseInlineActionsV2\n      extends FSParam[Boolean](\n        name = \"mr_inline_use_inline_action_v2\",\n        default = false\n      )\n\n  object EnableInlineFeedbackOnPush\n      extends FSParam[Boolean](\n        name = \"mr_inline_enable_inline_feedback_on_push\",\n        default = false\n      )\n\n  object InlineFeedbackSubstitutePosition\n      extends FSBoundedParam[Int](\n        name = \"mr_inline_feedback_substitute_position\",\n        min = 0,\n        max = 2,\n        default = 2, // default to substitute or append last inline action\n      )\n\n  /**\n   * Param to control dynamic inline actions for web notifications\n   */\n  object EnableDynamicInlineActionsForDesktopWeb\n      extends FSParam[Boolean](\n        name = \"mr_inline_enable_dynamic_actions_for_desktop_web\",\n        default = false\n      )\n\n  object EnableDynamicInlineActionsForMobileWeb\n      extends FSParam[Boolean](\n        name = \"mr_inline_enable_dynamic_actions_for_mobile_web\",\n        default = false\n      )\n\n  /**\n   * Param to define dynamic inline action types for web notifications (both desktop web + mobile web)\n   */\n  object TweetDynamicInlineActionsListForWeb\n      extends FSEnumSeqParam[InlineActionsEnum.type](\n        name = \"mr_inline_tweet_dynamic_action_for_web_ids\",\n        default = Seq(Retweet, Favorite),\n        enum = InlineActionsEnum\n      )\n\n  /**\n   * Param to enable MR Override Notifications for Android\n   */\n  object EnableOverrideNotificationsForAndroid\n      extends FSParam[Boolean](\n        name = \"mr_override_enable_override_notifications_for_android\",\n        default = false\n      )\n\n  /**\n   * Param to enable MR Override Notifications for iOS\n   */\n  object EnableOverrideNotificationsForIos\n      extends FSParam[Boolean](\n        name = \"mr_override_enable_override_notifications_for_ios\",\n        default = false\n      )\n\n  /**\n   * Param to enable gradually ramp up notification\n   */\n  object EnableGraduallyRampUpNotification\n      extends FSParam[Boolean](\n        name = \"pushcap_gradually_ramp_up_enable\",\n        default = false\n      )\n\n  /**\n   * Param to control the minInrerval for fatigue between consecutive MFNFY pushes\n   */\n  object MFMinIntervalFatigue\n      extends FSBoundedParam[Duration](\n        name = \"frigate_push_magicfanout_fatigue_min_interval_consecutive_pushes_minutes\",\n        default = 240.minutes,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromMinutes\n  }\n\n  /**\n   * Param to control the interval for MFNFY pushes\n   */\n  object MFPushIntervalInHours\n      extends FSBoundedParam[Duration](\n        name = \"frigate_push_magicfanout_fatigue_push_interval_in_hours\",\n        default = 24.hours,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the maximum number of Sports MF pushes in a period of time\n   */\n  object SportsMaxNumberOfPushesInInterval\n      extends FSBoundedParam[Int](\n        name = \"magicfanout_sports_event_fatigue_max_pushes_in_interval\",\n        default = 2,\n        min = 0,\n        max = 6)\n\n  /**\n   * Param to control the minInterval for fatigue between consecutive sports pushes\n   */\n  object SportsMinIntervalFatigue\n      extends FSBoundedParam[Duration](\n        name = \"magicfanout_sports_event_fatigue_min_interval_consecutive_pushes_minutes\",\n        default = 240.minutes,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromMinutes\n  }\n\n  /**\n   * Param to control the interval for sports pushes\n   */\n  object SportsPushIntervalInHours\n      extends FSBoundedParam[Duration](\n        name = \"magicfanout_sports_event_fatigue_push_interval_in_hours\",\n        default = 24.hours,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the maximum number of same event sports MF pushes in a period of time\n   */\n  object SportsMaxNumberOfPushesInIntervalPerEvent\n      extends FSBoundedParam[Int](\n        name = \"magicfanout_sports_event_fatigue_max_pushes_in_per_event_interval\",\n        default = 2,\n        min = 0,\n        max = 6)\n\n  /**\n   * Param to control the minInterval for fatigue between consecutive same event sports pushes\n   */\n  object SportsMinIntervalFatiguePerEvent\n      extends FSBoundedParam[Duration](\n        name = \"magicfanout_sports_event_fatigue_min_interval_consecutive_pushes_per_event_minutes\",\n        default = 240.minutes,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromMinutes\n  }\n\n  /**\n   * Param to control the interval for same event sports pushes\n   */\n  object SportsPushIntervalInHoursPerEvent\n      extends FSBoundedParam[Duration](\n        name = \"magicfanout_sports_event_fatigue_push_interval_per_event_in_hours\",\n        default = 24.hours,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the maximum number of MF pushes in a period of time\n   */\n  object MFMaxNumberOfPushesInInterval\n      extends FSBoundedParam[Int](\n        name = \"frigate_push_magicfanout_fatigue_max_pushes_in_interval\",\n        default = 2,\n        min = 0,\n        max = 6)\n\n  /**\n   * Param to enable custom duration for fatiguing\n   */\n  object GPEnableCustomMagicFanoutCricketFatigue\n      extends FSParam[Boolean](\n        name = \"global_participation_cricket_magicfanout_enable_custom_fatigue\",\n        default = false\n      )\n\n  /**\n   * Param to enable e2e scribing for target filtering step\n   */\n  object EnableMrRequestScribingForTargetFiltering\n      extends FSParam[Boolean](\n        name = \"mr_request_scribing_enable_for_target_filtering\",\n        default = false\n      )\n\n  /**\n   * Param to enable e2e scribing for candidate filtering step\n   */\n  object EnableMrRequestScribingForCandidateFiltering\n      extends FSParam[Boolean](\n        name = \"mr_request_scribing_enable_for_candidate_filtering\",\n        default = false\n      )\n\n  /**\n   * Param to enable e2e scribing with feature hydrating\n   */\n  object EnableMrRequestScribingWithFeatureHydrating\n      extends FSParam[Boolean](\n        name = \"mr_request_scribing_enable_with_feature_hydrating\",\n        default = false\n      )\n\n  /*\n   * TargetLevel Feature list for Mr request scribing\n   */\n  object TargetLevelFeatureListForMrRequestScribing\n      extends FSParam[Seq[String]](\n        name = \"mr_request_scribing_target_level_feature_list\",\n        default = Seq.empty\n      )\n\n  /**\n   * Param to enable \\eps-greedy exploration for BigFiltering/LTV-based filtering\n   */\n  object EnableMrRequestScribingForEpsGreedyExploration\n      extends FSParam[Boolean](\n        name = \"mr_request_scribing_eps_greedy_exploration_enable\",\n        default = false\n      )\n\n  /**\n   * Param to control epsilon in \\eps-greedy exploration for BigFiltering/LTV-based filtering\n   */\n  object MrRequestScribingEpsGreedyExplorationRatio\n      extends FSBoundedParam[Double](\n        name = \"mr_request_scribing_eps_greedy_exploration_ratio\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Param to enable scribing dismiss model score\n   */\n  object EnableMrRequestScribingDismissScore\n      extends FSParam[Boolean](\n        name = \"mr_request_scribing_dismiss_score_enable\",\n        default = false\n      )\n\n  /**\n   * Param to enable scribing BigFiltering supervised model(s) score(s)\n   */\n  object EnableMrRequestScribingBigFilteringSupervisedScores\n      extends FSParam[Boolean](\n        name = \"mr_request_scribing_bigfiltering_supervised_scores_enable\",\n        default = false\n      )\n\n  /**\n   * Param to enable scribing BigFiltering RL model(s) score(s)\n   */\n  object EnableMrRequestScribingBigFilteringRLScores\n      extends FSParam[Boolean](\n        name = \"mr_request_scribing_bigfiltering_rl_scores_enable\",\n        default = false\n      )\n\n  /**\n   * Param to flatten mr request scribe\n   */\n  object EnableFlattenMrRequestScribing\n      extends FSParam[Boolean](\n        name = \"mr_request_scribing_enable_flatten\",\n        default = false\n      )\n\n  /**\n   * Param to enable NSFW token based filtering\n   */\n  object EnableNsfwTokenBasedFiltering\n      extends FSParam[Boolean](\n        name = \"health_and_quality_filter_enable_nsfw_token_based_filtering\",\n        default = false\n      )\n\n  object NsfwTokensParam\n      extends FSParam[Seq[String]](\n        name = \"health_and_quality_filter_nsfw_tokens\",\n        default = Seq(\"nsfw\", \"18+\", \"\\uD83D\\uDD1E\"))\n\n  object MinimumAllowedAuthorAccountAgeInHours\n      extends FSBoundedParam[Int](\n        name = \"health_and_quality_filter_minimum_allowed_author_account_age_in_hours\",\n        default = 0,\n        min = 0,\n        max = 168\n      )\n\n  /**\n   * Param to enable the profanity filter\n   */\n  object EnableProfanityFilterParam\n      extends FSParam[Boolean](\n        name = \"health_and_quality_filter_enable_profanity_filter\",\n        default = false\n      )\n\n  /**\n   * Param to enable query the author media representation store\n   */\n  object EnableQueryAuthorMediaRepresentationStore\n      extends FSParam[Boolean](\n        name = \"health_and_quality_filter_enable_query_author_media_representation_store\",\n        default = false\n      )\n\n  /**\n   * Threshold to filter a tweet based on the author sensitive media score\n   */\n  object AuthorSensitiveMediaFilteringThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_and_quality_filter_author_sensitive_media_filtering_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Threshold to filter a tweet based on the author sensitive media score\n   */\n  object AuthorSensitiveMediaFilteringThresholdForMrTwistly\n      extends FSBoundedParam[Double](\n        name = \"health_and_quality_filter_author_sensitive_media_filtering_threshold_for_mrtwistly\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Param to enable filtering the SimCluster tweet if it has AbuseStrike_Top2Percent entitiy\n   */\n  object EnableAbuseStrikeTop2PercentFilterSimCluster\n      extends FSParam[Boolean](\n        name = \"health_signal_store_enable_abuse_strike_top_2_percent_filter_sim_cluster\",\n        default = false\n      )\n\n  /**\n   * Param to enable filtering the SimCluster tweet if it has AbuseStrike_Top1Percent entitiy\n   */\n  object EnableAbuseStrikeTop1PercentFilterSimCluster\n      extends FSParam[Boolean](\n        name = \"health_signal_store_enable_abuse_strike_top_1_percent_filter_sim_cluster\",\n        default = false\n      )\n\n  /**\n   * Param to enable filtering the SimCluster tweet if it has AbuseStrike_Top0.5Percent entitiy\n   */\n  object EnableAbuseStrikeTop05PercentFilterSimCluster\n      extends FSParam[Boolean](\n        name = \"health_signal_store_enable_abuse_strike_top_05_percent_filter_sim_cluster\",\n        default = false\n      )\n\n  object EnableAgathaUserHealthModelPredicate\n      extends FSParam[Boolean](\n        name = \"health_signal_store_enable_agatha_user_health_model_predicate\",\n        default = false\n      )\n\n  /**\n   * Threshold to filter a tweet based on the agatha_calibrated_nsfw score of its author for MrTwistly\n   */\n  object AgathaCalibratedNSFWThresholdForMrTwistly\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_agatha_calibrated_nsfw_threshold_for_mrtwistly\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Threshold to filter a tweet based on the agatha_calibrated_nsfw score of its author\n   */\n  object AgathaCalibratedNSFWThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_agatha_calibrated_nsfw_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Threshold to filter a tweet based on the agatha_nsfw_text_user score of its author for MrTwistly\n   */\n  object AgathaTextNSFWThresholdForMrTwistly\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_agatha_text_nsfw_threshold_for_mrtwistly\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Threshold to filter a tweet based on the agatha_nsfw_text_user score of its author\n   */\n  object AgathaTextNSFWThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_agatha_text_nsfw_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Threshold to bucket a user based on the agatha_calibrated_nsfw score of the tweet author\n   */\n  object AgathaCalibratedNSFWBucketThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_agatha_calibrated_nsfw_bucket_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Threshold to bucket a user based on the agatha_nsfw_text_user score of the tweet author\n   */\n  object AgathaTextNSFWBucketThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_agatha_text_nsfw_bucket_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Param to enable filtering using pnsfw_text_tweet model.\n   */\n  object EnableHealthSignalStorePnsfwTweetTextPredicate\n      extends FSParam[Boolean](\n        name = \"health_signal_store_enable_pnsfw_tweet_text_predicate\",\n        default = false\n      )\n\n  /**\n   * Threshold score for filtering based on pnsfw_text_tweet Model.\n   */\n  object PnsfwTweetTextThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_pnsfw_tweet_text_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Threshold score for bucketing based on pnsfw_text_tweet Model.\n   */\n  object PnsfwTweetTextBucketingThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_pnsfw_tweet_text_bucketing_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Enable filtering tweets with media based on pnsfw_media_tweet Model for OON tweets only.\n   */\n  object PnsfwTweetMediaFilterOonOnly\n      extends FSParam[Boolean](\n        name = \"health_signal_store_pnsfw_tweet_media_filter_oon_only\",\n        default = true\n      )\n\n  /**\n   * Threshold score for filtering tweets with media based on pnsfw_media_tweet Model.\n   */\n  object PnsfwTweetMediaThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_pnsfw_tweet_media_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Threshold score for filtering tweets with images based on pnsfw_media_tweet Model.\n   */\n  object PnsfwTweetImageThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_pnsfw_tweet_image_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Threshold score for filtering quote/reply tweets based on source tweet's media\n   */\n  object PnsfwQuoteTweetThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_pnsfw_quote_tweet_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Threshold score for bucketing based on pnsfw_media_tweet Model.\n   */\n  object PnsfwTweetMediaBucketingThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_pnsfw_tweet_media_bucketing_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Param to enable filtering using multilingual psnfw predicate\n   */\n  object EnableHealthSignalStoreMultilingualPnsfwTweetTextPredicate\n      extends FSParam[Boolean](\n        name = \"health_signal_store_enable_multilingual_pnsfw_tweet_text_predicate\",\n        default = false\n      )\n\n  /**\n   * Language sequence we will query pnsfw scores for\n   */\n  object MultilingualPnsfwTweetTextSupportedLanguages\n      extends FSParam[Seq[String]](\n        name = \"health_signal_store_multilingual_pnsfw_tweet_supported_languages\",\n        default = Seq.empty[String],\n      )\n\n  /**\n   * Threshold score per language for bucketing based on pnsfw scores.\n   */\n  object MultilingualPnsfwTweetTextBucketingThreshold\n      extends FSParam[Seq[Double]](\n        name = \"health_signal_store_multilingual_pnsfw_tweet_text_bucketing_thresholds\",\n        default = Seq.empty[Double],\n      )\n\n  /**\n   * Threshold score per language for filtering based on pnsfw scores.\n   */\n  object MultilingualPnsfwTweetTextFilteringThreshold\n      extends FSParam[Seq[Double]](\n        name = \"health_signal_store_multilingual_pnsfw_tweet_text_filtering_thresholds\",\n        default = Seq.empty[Double],\n      )\n\n  /**\n   * List of models to threshold scores for bucketing purposes\n   */\n  object MultilingualPnsfwTweetTextBucketingModelList\n      extends FSEnumSeqParam[NsfwTextDetectionModel.type](\n        name = \"health_signal_store_multilingual_pnsfw_tweet_text_bucketing_models_ids\",\n        default = Seq(NsfwTextDetectionModel.ProdModel),\n        enum = NsfwTextDetectionModel\n      )\n\n  object MultilingualPnsfwTweetTextModel\n      extends FSEnumParam[NsfwTextDetectionModel.type](\n        name = \"health_signal_store_multilingual_pnsfw_tweet_text_model\",\n        default = NsfwTextDetectionModel.ProdModel,\n        enum = NsfwTextDetectionModel\n      )\n\n  /**\n   * Param to determine media should be enabled for android\n   */\n  object EnableEventSquareMediaAndroid\n      extends FSParam[Boolean](\n        name = \"mr_enable_event_media_square_android\",\n        default = false\n      )\n\n  /**\n   * Param to determine expanded media should be enabled for android\n   */\n  object EnableEventPrimaryMediaAndroid\n      extends FSParam[Boolean](\n        name = \"mr_enable_event_media_primary_android\",\n        default = false\n      )\n\n  /**\n   * Param to determine media should be enabled for ios for MagicFanout\n   */\n  object EnableEventSquareMediaIosMagicFanoutNewsEvent\n      extends FSParam[Boolean](\n        name = \"mr_enable_event_media_square_ios_mf\",\n        default = false\n      )\n\n  /**\n   * Param to configure HTL Visit fatigue\n   */\n  object HTLVisitFatigueTime\n      extends FSBoundedParam[Int](\n        name = \"frigate_push_htl_visit_fatigue_time\",\n        default = 20,\n        min = 0,\n        max = 72) {\n\n    // Fatigue duration for HTL visit\n    final val DefaultHoursToFatigueAfterHtlVisit = 20\n    final val OldHoursToFatigueAfterHtlVisit = 8\n  }\n\n  object MagicFanoutNewsUserGeneratedEventsEnable\n      extends FSParam[Boolean](\n        name = \"magicfanout_news_user_generated_events_enable\",\n        default = false)\n\n  object MagicFanoutSkipAccountCountryPredicate\n      extends FSParam[Boolean](\"magicfanout_news_skip_account_country_predicate\", false)\n\n  object MagicFanoutNewsEnableDescriptionCopy\n      extends FSParam[Boolean](name = \"magicfanout_news_enable_description_copy\", default = false)\n\n  /**\n   *  Enables Custom Targeting for MagicFnaout News events in Pushservice\n   */\n  object MagicFanoutEnableCustomTargetingNewsEvent\n      extends FSParam[Boolean](\"magicfanout_news_event_custom_targeting_enable\", false)\n\n  /**\n   * Enable Topic Copy in MF\n   */\n  object EnableTopicCopyForMF\n      extends FSParam[Boolean](\n        name = \"magicfanout_enable_topic_copy\",\n        default = false\n      )\n\n  /**\n   * Enable Topic Copy in MF for implicit topics\n   */\n  object EnableTopicCopyForImplicitTopics\n      extends FSParam[Boolean](\n        name = \"magicfanout_enable_topic_copy_erg_interests\",\n        default = false\n      )\n\n  /**\n   * Enable NewCreator push\n   */\n  object EnableNewCreatorPush\n      extends FSParam[Boolean](\n        name = \"new_creator_enable_push\",\n        default = false\n      )\n\n  /**\n   * Enable CreatorSubscription push\n   */\n  object EnableCreatorSubscriptionPush\n      extends FSParam[Boolean](\n        name = \"creator_subscription_enable_push\",\n        default = false\n      )\n\n  /**\n   * Featureswitch param to enable/disable push recommendations\n   */\n  object EnablePushRecommendationsParam\n      extends FSParam[Boolean](name = \"push_recommendations_enabled\", default = false)\n\n  object DisableMlInFilteringFeatureSwitchParam\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_disable_ml_in_filtering\",\n        default = false\n      )\n\n  object EnableMinDurationModifier\n      extends FSParam[Boolean](\n        name = \"min_duration_modifier_enable_hour_modifier\",\n        default = false\n      )\n\n  object EnableMinDurationModifierV2\n      extends FSParam[Boolean](\n        name = \"min_duration_modifier_enable_hour_modifier_v2\",\n        default = false\n      )\n\n  object MinDurationModifierStartHourList\n      extends FSParam[Seq[Int]](\n        name = \"min_duration_modifier_start_time_list\",\n        default = Seq(),\n      )\n\n  object MinDurationModifierEndHourList\n      extends FSParam[Seq[Int]](\n        name = \"min_duration_modifier_start_end_list\",\n        default = Seq(),\n      )\n\n  object MinDurationTimeModifierConst\n      extends FSParam[Seq[Int]](\n        name = \"min_duration_modifier_const_list\",\n        default = Seq(),\n      )\n\n  object EnableQueryUserOpenedHistory\n      extends FSParam[Boolean](\n        name = \"min_duration_modifier_enable_query_user_opened_history\",\n        default = false\n      )\n\n  object EnableMinDurationModifierByUserHistory\n      extends FSParam[Boolean](\n        name = \"min_duration_modifier_enable_hour_modifier_by_user_history\",\n        default = false\n      )\n\n  object EnableRandomHourForQuickSend\n      extends FSParam[Boolean](\n        name = \"min_duration_modifier_enable_random_hour_for_quick_send\",\n        default = false\n      )\n\n  object SendTimeByUserHistoryMaxOpenedThreshold\n      extends FSBoundedParam[Int](\n        name = \"min_duration_modifier_max_opened_threshold\",\n        default = 4,\n        min = 0,\n        max = 100)\n\n  object SendTimeByUserHistoryNoSendsHours\n      extends FSBoundedParam[Int](\n        name = \"min_duration_modifier_no_sends_hours\",\n        default = 1,\n        min = 0,\n        max = 24)\n\n  object SendTimeByUserHistoryQuickSendBeforeHours\n      extends FSBoundedParam[Int](\n        name = \"min_duration_modifier_quick_send_before_hours\",\n        default = 0,\n        min = 0,\n        max = 24)\n\n  object SendTimeByUserHistoryQuickSendAfterHours\n      extends FSBoundedParam[Int](\n        name = \"min_duration_modifier_quick_send_after_hours\",\n        default = 0,\n        min = 0,\n        max = 24)\n\n  object SendTimeByUserHistoryQuickSendMinDurationInMinute\n      extends FSBoundedParam[Int](\n        name = \"min_duration_modifier_quick_send_min_duration\",\n        default = 0,\n        min = 0,\n        max = 1440)\n\n  object SendTimeByUserHistoryNoSendMinDuration\n      extends FSBoundedParam[Int](\n        name = \"min_duration_modifier_no_send_min_duration\",\n        default = 24,\n        min = 0,\n        max = 24)\n\n  object EnableMfGeoTargeting\n      extends FSParam[Boolean](\n        name = \"frigate_push_magicfanout_geo_targeting_enable\",\n        default = false)\n\n  /**\n   * Enable RUX Tweet landing page for push open. When this param is enabled, user will go to RUX\n   * landing page instead of Tweet details page when opening MagicRecs push.\n   */\n  object EnableRuxLandingPage\n      extends FSParam[Boolean](name = \"frigate_push_enable_rux_landing_page\", default = false)\n\n  /**\n   * Enable RUX Tweet landing page for Ntab Click. When this param is enabled, user will go to RUX\n   * landing page instead of Tweet details page when click MagicRecs entry on Ntab.\n   */\n  object EnableNTabRuxLandingPage\n      extends FSParam[Boolean](name = \"frigate_push_enable_ntab_rux_landing_page\", default = false)\n\n  /**\n   * Param to enable Onboarding Pushes\n   */\n  object EnableOnboardingPushes\n      extends FSParam[Boolean](\n        name = \"onboarding_push_enable\",\n        default = false\n      )\n\n  /**\n   * Param to enable Address Book Pushes\n   */\n  object EnableAddressBookPush\n      extends FSParam[Boolean](\n        name = \"onboarding_push_enable_address_book_push\",\n        default = false\n      )\n\n  /**\n   * Param to enable Complete Onboarding Pushes\n   */\n  object EnableCompleteOnboardingPush\n      extends FSParam[Boolean](\n        name = \"onboarding_push_enable_complete_onboarding_push\",\n        default = false\n      )\n\n  /**\n   * Param to enable Smart Push Config for MR Override Notifs on Android\n   */\n  object EnableOverrideNotificationsSmartPushConfigForAndroid\n      extends FSParam[Boolean](\n        name = \"mr_override_enable_smart_push_config_for_android\",\n        default = false)\n\n  /**\n   * Param to control the min duration since last MR push for Onboarding Pushes\n   */\n  object MrMinDurationSincePushForOnboardingPushes\n      extends FSBoundedParam[Duration](\n        name = \"onboarding_push_min_duration_since_push_days\",\n        default = 4.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Param to control the push fatigue for Onboarding Pushes\n   */\n  object FatigueForOnboardingPushes\n      extends FSBoundedParam[Duration](\n        name = \"onboarding_push_fatigue_days\",\n        default = 30.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Param to specify the maximum number of Onboarding Push Notifs in a specified period of time\n   */\n  object MaxOnboardingPushInInterval\n      extends FSBoundedParam[Int](\n        name = \"onboarding_push_max_in_interval\",\n        default = 1,\n        min = 0,\n        max = 10\n      )\n\n  /**\n   * Param to disable the Onboarding Push Notif Fatigue\n   */\n  object DisableOnboardingPushFatigue\n      extends FSParam[Boolean](\n        name = \"onboarding_push_disable_push_fatigue\",\n        default = false\n      )\n\n  /**\n   * Param to control the inverter for fatigue between consecutive TopTweetsByGeoPush\n   */\n  object TopTweetsByGeoPushInterval\n      extends FSBoundedParam[Duration](\n        name = \"top_tweets_by_geo_interval_days\",\n        default = 0.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Param to control the inverter for fatigue between consecutive TripTweets\n   */\n  object HighQualityTweetsPushInterval\n      extends FSBoundedParam[Duration](\n        name = \"high_quality_candidates_push_interval_days\",\n        default = 1.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Expiry TTL duration for Tweet Notification types written to history store\n   */\n  object FrigateHistoryTweetNotificationWriteTtl\n      extends FSBoundedParam[Duration](\n        name = \"frigate_notification_history_tweet_write_ttl_days\",\n        default = 60.days,\n        min = Duration.Bottom,\n        max = Duration.Top\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Expiry TTL duration for Notification written to history store\n   */\n  object FrigateHistoryOtherNotificationWriteTtl\n      extends FSBoundedParam[Duration](\n        name = \"frigate_notification_history_other_write_ttl_days\",\n        default = 90.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Param to control maximum number of TopTweetsByGeoPush pushes to receive in an interval\n   */\n  object MaxTopTweetsByGeoPushGivenInterval\n      extends FSBoundedParam[Int](\n        name = \"top_tweets_by_geo_push_given_interval\",\n        default = 1,\n        min = 0,\n        max = 10\n      )\n\n  /**\n   * Param to control maximum number of HighQualityTweet pushes to receive in an interval\n   */\n  object MaxHighQualityTweetsPushGivenInterval\n      extends FSBoundedParam[Int](\n        name = \"high_quality_candidates_max_push_given_interval\",\n        default = 3,\n        min = 0,\n        max = 10\n      )\n\n  /**\n   * Param to downrank/backfill top tweets by geo candidates\n   */\n  object BackfillRankTopTweetsByGeoCandidates\n      extends FSParam[Boolean](\n        name = \"top_tweets_by_geo_backfill_rank\",\n        default = false\n      )\n\n  /**\n   * Determine whether to use aggressive thresholds for Health filtering on SearchTweet\n   */\n  object PopGeoTweetEnableAggressiveThresholds\n      extends FSParam[Boolean](\n        name = \"top_tweets_by_geo_enable_aggressive_health_thresholds\",\n        default = false\n      )\n\n  /**\n   * Param to apply different scoring functions to select top tweets by geo candidates\n   */\n  object ScoringFuncForTopTweetsByGeo\n      extends FSParam[String](\n        name = \"top_tweets_by_geo_scoring_function\",\n        default = \"Pop8H\",\n      )\n\n  /**\n   * Param to query different stores in pop geo service.\n   */\n  object TopTweetsByGeoCombinationParam\n      extends FSEnumParam[TopTweetsForGeoCombination.type](\n        name = \"top_tweets_by_geo_combination_id\",\n        default = TopTweetsForGeoCombination.Default,\n        enum = TopTweetsForGeoCombination\n      )\n\n  /**\n   * Param for popgeo tweet version\n   */\n  object PopGeoTweetVersionParam\n      extends FSEnumParam[PopGeoTweetVersion.type](\n        name = \"top_tweets_by_geo_version_id\",\n        default = PopGeoTweetVersion.Prod,\n        enum = PopGeoTweetVersion\n      )\n\n  /**\n   * Param to query what length of hash for geoh store\n   */\n  object GeoHashLengthList\n      extends FSParam[Seq[Int]](\n        name = \"top_tweets_by_geo_hash_length_list\",\n        default = Seq(4),\n      )\n\n  /**\n   * Param to include country code results as back off .\n   */\n  object EnableCountryCodeBackoffTopTweetsByGeo\n      extends FSParam[Boolean](\n        name = \"top_tweets_by_geo_enable_country_code_backoff\",\n        default = false,\n      )\n\n  /**\n   * Param to decide ranking function for fetched top tweets by geo\n   */\n  object RankingFunctionForTopTweetsByGeo\n      extends FSEnumParam[TopTweetsForGeoRankingFunction.type](\n        name = \"top_tweets_by_geo_ranking_function_id\",\n        default = TopTweetsForGeoRankingFunction.Score,\n        enum = TopTweetsForGeoRankingFunction\n      )\n\n  /**\n   * Param to enable top tweets by geo candidates\n   */\n  object EnableTopTweetsByGeoCandidates\n      extends FSParam[Boolean](\n        name = \"top_tweets_by_geo_enable_candidate_source\",\n        default = false\n      )\n\n  /**\n   * Param to enable top tweets by geo candidates for dormant users\n   */\n  object EnableTopTweetsByGeoCandidatesForDormantUsers\n      extends FSParam[Boolean](\n        name = \"top_tweets_by_geo_enable_candidate_source_dormant_users\",\n        default = false\n      )\n\n  /**\n   * Param to specify the maximum number of Top Tweets by Geo candidates to take\n   */\n  object MaxTopTweetsByGeoCandidatesToTake\n      extends FSBoundedParam[Int](\n        name = \"top_tweets_by_geo_candidates_to_take\",\n        default = 10,\n        min = 0,\n        max = 100\n      )\n\n  /**\n   * Param to min duration since last MR push for top tweets by geo pushes\n   */\n  object MrMinDurationSincePushForTopTweetsByGeoPushes\n      extends FSBoundedParam[Duration](\n        name = \"top_tweets_by_geo_min_duration_since_last_mr_days\",\n        default = 3.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Param to enable FRS candidate tweets\n   */\n  object EnableFrsCandidates\n      extends FSParam[Boolean](\n        name = \"frs_tweet_candidate_enable_adaptor\",\n        default = false\n      )\n\n  /**\n   * Param to enable FRSTweet candidates for topic setting users\n   * */\n  object EnableFrsTweetCandidatesTopicSetting\n      extends FSParam[Boolean](\n        name = \"frs_tweet_candidate_enable_adaptor_for_topic_setting\",\n        default = false\n      )\n\n  /**\n   * Param to enable topic annotations for FRSTweet candidates tweets\n   * */\n  object EnableFrsTweetCandidatesTopicAnnotation\n      extends FSParam[Boolean](\n        name = \"frs_tweet_candidate_enable_topic_annotation\",\n        default = false\n      )\n\n  /**\n   * Param to enable topic copy for FRSTweet candidates tweets\n   * */\n  object EnableFrsTweetCandidatesTopicCopy\n      extends FSParam[Boolean](\n        name = \"frs_tweet_candidate_enable_topic_copy\",\n        default = false\n      )\n\n  /**\n   * Topic score threshold for FRSTweet candidates topic annotations\n   * */\n  object FrsTweetCandidatesTopicScoreThreshold\n      extends FSBoundedParam[Double](\n        name = \"frs_tweet_candidate_topic_score_threshold\",\n        default = 0.0,\n        min = 0.0,\n        max = 100.0\n      )\n\n  /**\n   * Param to enable mr modeling-based candidates tweets\n   * */\n  object EnableMrModelingBasedCandidates\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_enable_adaptor\",\n        default = false\n      )\n\n  /**\n   Param to enable mr modeling-based candidates tweets for topic setting users\n   * */\n  object EnableMrModelingBasedCandidatesTopicSetting\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_enable_adaptor_for_topic_setting\",\n        default = false\n      )\n\n  /**\n   * Param to enable topic annotations for mr modeling-based candidates tweets\n   * */\n  object EnableMrModelingBasedCandidatesTopicAnnotation\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_enable_adaptor_topic_annotation\",\n        default = false\n      )\n\n  /**\n   * Topic score threshold for mr modeling based candidates topic annotations\n   * */\n  object MrModelingBasedCandidatesTopicScoreThreshold\n      extends FSBoundedParam[Double](\n        name = \"candidate_generation_model_topic_score_threshold\",\n        default = 0.0,\n        min = 0.0,\n        max = 100.0\n      )\n\n  /**\n   * Param to enable topic copy for mr modeling-based candidates tweets\n   * */\n  object EnableMrModelingBasedCandidatesTopicCopy\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_enable_topic_copy\",\n        default = false\n      )\n\n  /**\n   * Number of max mr modeling based candidates\n   * */\n  object NumberOfMaxMrModelingBasedCandidates\n      extends FSBoundedParam[Int](\n        name = \"candidate_generation_model_max_mr_modeling_based_candidates\",\n        default = 200,\n        min = 0,\n        max = 1000\n      )\n\n  /**\n   * Enable the traffic to use fav threshold\n   * */\n  object EnableThresholdOfFavMrModelingBasedCandidates\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_enable_fav_threshold\",\n        default = false\n      )\n\n  /**\n   * Threshold of fav for mr modeling based candidates\n   * */\n  object ThresholdOfFavMrModelingBasedCandidates\n      extends FSBoundedParam[Int](\n        name = \"candidate_generation_model_fav_threshold\",\n        default = 0,\n        min = 0,\n        max = 500\n      )\n\n  /**\n   * Filtered threshold for mr modeling based candidates\n   * */\n  object CandidateGenerationModelCosineThreshold\n      extends FSBoundedParam[Double](\n        name = \"candidate_generation_model_cosine_threshold\",\n        default = 0.9,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /*\n   * ANN hyparameters\n   * */\n  object ANNEfQuery\n      extends FSBoundedParam[Int](\n        name = \"candidate_generation_model_ann_ef_query\",\n        default = 300,\n        min = 50,\n        max = 1500\n      )\n\n  /**\n   * Param to do real A/B impression for FRS candidates to avoid dilution\n   */\n  object EnableResultFromFrsCandidates\n      extends FSParam[Boolean](\n        name = \"frs_tweet_candidate_enable_returned_result\",\n        default = false\n      )\n\n  /**\n   * Param to enable hashspace candidate tweets\n   */\n  object EnableHashspaceCandidates\n      extends FSParam[Boolean](\n        name = \"hashspace_candidate_enable_adaptor\",\n        default = false\n      )\n\n  /**\n   * Param to enable hashspace candidates tweets for topic setting users\n   * */\n  object EnableHashspaceCandidatesTopicSetting\n      extends FSParam[Boolean](\n        name = \"hashspace_candidate_enable_adaptor_for_topic_setting\",\n        default = false\n      )\n\n  /**\n   * Param to enable topic annotations for hashspace candidates tweets\n   * */\n  object EnableHashspaceCandidatesTopicAnnotation\n      extends FSParam[Boolean](\n        name = \"hashspace_candidate_enable_topic_annotation\",\n        default = false\n      )\n\n  /**\n   * Param to enable topic copy for hashspace candidates tweets\n   * */\n  object EnableHashspaceCandidatesTopicCopy\n      extends FSParam[Boolean](\n        name = \"hashspace_candidate_enable_topic_copy\",\n        default = false\n      )\n\n  /**\n   * Topic score threshold for hashspace candidates topic annotations\n   * */\n  object HashspaceCandidatesTopicScoreThreshold\n      extends FSBoundedParam[Double](\n        name = \"hashspace_candidate_topic_score_threshold\",\n        default = 0.0,\n        min = 0.0,\n        max = 100.0\n      )\n\n  /**\n   * Param to do real A/B impression for hashspace candidates to avoid dilution\n   */\n  object EnableResultFromHashspaceCandidates\n      extends FSParam[Boolean](\n        name = \"hashspace_candidate_enable_returned_result\",\n        default = false\n      )\n\n  /**\n   * Param to enable detopic tweet candidates in adaptor\n   */\n  object EnableDeTopicTweetCandidates\n      extends FSParam[Boolean](\n        name = \"detopic_tweet_candidate_enable_adaptor\",\n        default = false\n      )\n\n  /**\n   * Param to enable detopic tweet candidates results (to avoid dilution)\n   */\n  object EnableDeTopicTweetCandidateResults\n      extends FSParam[Boolean](\n        name = \"detopic_tweet_candidate_enable_results\",\n        default = false\n      )\n\n  /**\n   * Param to specify whether to provide a custom list of topics in request\n   */\n  object EnableDeTopicTweetCandidatesCustomTopics\n      extends FSParam[Boolean](\n        name = \"detopic_tweet_candidate_enable_custom_topics\",\n        default = false\n      )\n\n  /**\n   * Param to specify whether to provide a custom language in request\n   */\n  object EnableDeTopicTweetCandidatesCustomLanguages\n      extends FSParam[Boolean](\n        name = \"detopic_tweet_candidate_enable_custom_languages\",\n        default = false\n      )\n\n  /**\n   * Number of detopic tweet candidates in the request\n   * */\n  object NumberOfDeTopicTweetCandidates\n      extends FSBoundedParam[Int](\n        name = \"detopic_tweet_candidate_num_candidates_in_request\",\n        default = 600,\n        min = 0,\n        max = 3000\n      )\n\n  /**\n   * Max Number of detopic tweet candidates returned in adaptor\n   * */\n  object NumberOfMaxDeTopicTweetCandidatesReturned\n      extends FSBoundedParam[Int](\n        name = \"detopic_tweet_candidate_max_num_candidates_returned\",\n        default = 200,\n        min = 0,\n        max = 3000\n      )\n\n  /**\n   * Param to enable F1 from protected Authors\n   */\n  object EnableF1FromProtectedTweetAuthors\n      extends FSParam[Boolean](\n        \"f1_enable_protected_tweets\",\n        false\n      )\n\n  /**\n   * Param to enable safe user tweet tweetypie store\n   */\n  object EnableSafeUserTweetTweetypieStore\n      extends FSParam[Boolean](\n        \"mr_infra_enable_use_safe_user_tweet_tweetypie\",\n        false\n      )\n\n  /**\n   * Param to min duration since last MR push for top tweets by geo pushes\n   */\n  object EnableMrMinDurationSinceMrPushFatigue\n      extends FSParam[Boolean](\n        name = \"top_tweets_by_geo_enable_min_duration_since_mr_fatigue\",\n        default = false\n      )\n\n  /**\n   * Param to check time since last time user logged in for geo top tweets by geo push\n   */\n  object TimeSinceLastLoginForGeoPopTweetPush\n      extends FSBoundedParam[Duration](\n        name = \"top_tweets_by_geo_time_since_last_login_in_days\",\n        default = 14.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Param to check time since last time user logged in for geo top tweets by geo push\n   */\n  object MinimumTimeSinceLastLoginForGeoPopTweetPush\n      extends FSBoundedParam[Duration](\n        name = \"top_tweets_by_geo_minimum_time_since_last_login_in_days\",\n        default = 14.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /** How long we wait after a user visited the app before sending them a space fanout rec */\n  object SpaceRecsAppFatigueDuration\n      extends FSBoundedParam[Duration](\n        name = \"space_recs_app_fatigue_duration_hours\",\n        default = 4.hours,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /** The fatigue time-window for OON space fanout recs, e.g. 1 push every 3 days */\n  object OONSpaceRecsFatigueDuration\n      extends FSBoundedParam[Duration](\n        name = \"space_recs_oon_fatigue_duration_days\",\n        default = 1.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /** The global fatigue time-window for space fanout recs, e.g. 1 push every 3 days */\n  object SpaceRecsGlobalFatigueDuration\n      extends FSBoundedParam[Duration](\n        name = \"space_recs_global_fatigue_duration_days\",\n        default = 1.day,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /** The min-interval between space fanout recs.\n   * After receiving a space fanout rec, they must wait a minimum of this\n   * interval before eligibile for another */\n  object SpaceRecsFatigueMinIntervalDuration\n      extends FSBoundedParam[Duration](\n        name = \"space_recs_fatigue_mininterval_duration_minutes\",\n        default = 30.minutes,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromMinutes\n  }\n\n  /** Space fanout user-follow rank threshold.\n   * Users targeted by a follow that is above this threshold will be filtered */\n  object SpaceRecsRealgraphThreshold\n      extends FSBoundedParam[Int](\n        name = \"space_recs_realgraph_threshold\",\n        default = 50,\n        max = 500,\n        min = 0\n      )\n\n  object EnableHydratingRealGraphTargetUserFeatures\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_enable_hydrating_real_graph_target_user_feature\",\n        default = true\n      )\n\n  /** Param to reduce dillution when checking if a space is featured or not */\n  object CheckFeaturedSpaceOON\n      extends FSParam[Boolean](name = \"space_recs_check_if_its_featured_space\", default = false)\n\n  /** Enable Featured Spaces Rules for OON spaces */\n  object EnableFeaturedSpacesOON\n      extends FSParam[Boolean](name = \"space_recs_enable_featured_spaces_oon\", default = false)\n\n  /** Enable Geo Targeting */\n  object EnableGeoTargetingForSpaces\n      extends FSParam[Boolean](name = \"space_recs_enable_geo_targeting\", default = false)\n\n  /** Number of max pushes within the fatigue duration for OON Space Recs */\n  object OONSpaceRecsPushLimit\n      extends FSBoundedParam[Int](\n        name = \"space_recs_oon_push_limit\",\n        default = 1,\n        max = 3,\n        min = 0\n      )\n\n  /** Space fanout recs, number of max pushes within the fatigue duration */\n  object SpaceRecsGlobalPushLimit\n      extends FSBoundedParam[Int](\n        name = \"space_recs_global_push_limit\",\n        default = 3,\n        max = 50,\n        min = 0\n      )\n\n  /**\n   * Param to enable score based override.\n   */\n  object EnableOverrideNotificationsScoreBasedOverride\n      extends FSParam[Boolean](\n        name = \"mr_override_enable_score_ranking\",\n        default = false\n      )\n\n  /**\n   * Param to determine the lookback duration when searching for override info.\n   */\n  object OverrideNotificationsLookbackDurationForOverrideInfo\n      extends FSBoundedParam[Duration](\n        name = \"mr_override_lookback_duration_override_info_in_days\",\n        default = 30.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Param to determine the lookback duration when searching for impression ids.\n   */\n  object OverrideNotificationsLookbackDurationForImpressionId\n      extends FSBoundedParam[Duration](\n        name = \"mr_override_lookback_duration_impression_id_in_days\",\n        default = 30.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Param to enable sending multiple target ids in the payload.\n   */\n  object EnableOverrideNotificationsMultipleTargetIds\n      extends FSParam[Boolean](\n        name = \"mr_override_enable_multiple_target_ids\",\n        default = false\n      )\n\n  /**\n   * Param for MR Web Notifications holdback\n   */\n  object MRWebHoldbackParam\n      extends FSParam[Boolean](\n        name = \"mr_web_notifications_holdback\",\n        default = false\n      )\n\n  object CommonRecommendationTypeDenyListPushHoldbacks\n      extends FSParam[Seq[String]](\n        name = \"crt_to_exclude_from_holdbacks_push_holdbacks\",\n        default = Seq.empty[String]\n      )\n\n  /**\n   * Param to enable sending number of slots to maintain in the payload.\n   */\n  object EnableOverrideNotificationsNSlots\n      extends FSParam[Boolean](\n        name = \"mr_override_enable_n_slots\",\n        default = false\n      )\n\n  /**\n   * Enable down ranking of NUPS and pop geo topic follow candidates for new user playbook.\n   */\n  object EnableDownRankOfNewUserPlaybookTopicFollowPush\n      extends FSParam[Boolean](\n        name = \"topic_follow_new_user_playbook_enable_down_rank\",\n        default = false\n      )\n\n  /**\n   * Enable down ranking of NUPS and pop geo topic tweet candidates for new user playbook.\n   */\n  object EnableDownRankOfNewUserPlaybookTopicTweetPush\n      extends FSParam[Boolean](\n        name = \"topic_tweet_new_user_playbook_enable_down_rank\",\n        default = false\n      )\n\n  /**\n   * Param to enable/disable employee only spaces for fanout of notifications\n   */\n  object EnableEmployeeOnlySpaceNotifications\n      extends FSParam[Boolean](name = \"space_recs_employee_only_enable\", default = false)\n\n  /**\n   * NTab spaces ttl experiments\n   */\n  object EnableSpacesTtlForNtab\n      extends FSParam[Boolean](\n        name = \"ntab_spaces_ttl_enable\",\n        default = false\n      )\n\n  /**\n   * Param to determine the ttl duration for space notifications on NTab.\n   */\n  object SpaceNotificationsTTLDurationForNTab\n      extends FSBoundedParam[Duration](\n        name = \"ntab_spaces_ttl_hours\",\n        default = 1.hour,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /*\n   * NTab override experiments\n   * see go/ntab-override experiment brief for more details\n   */\n\n  /**\n   * Override notifications for Spaces on lockscreen.\n   */\n  object EnableOverrideForSpaces\n      extends FSParam[Boolean](\n        name = \"mr_override_spaces\",\n        default = false\n      )\n\n  /**\n   * Param to enable storing the Generic Notification Key.\n   */\n  object EnableStoringNtabGenericNotifKey\n      extends FSParam[Boolean](\n        name = \"ntab_enable_storing_generic_notif_key\",\n        default = false\n      )\n\n  /**\n   * Param to enable deleting the Target's timeline.\n   */\n  object EnableDeletingNtabTimeline\n      extends FSParam[Boolean](\n        name = \"ntab_enable_delete_timeline\",\n        default = false\n      )\n\n  /**\n   * Param to enable sending the overrideId\n   * to NTab which enables override support in NTab-api\n   */\n  object EnableOverrideIdNTabRequest\n      extends FSParam[Boolean](\n        name = \"ntab_enable_override_id_in_request\",\n        default = false\n      )\n\n  /**\n   * [Override Workstream] Param to enable NTab override n-slot feature.\n   */\n  object EnableNslotsForOverrideOnNtab\n      extends FSParam[Boolean](\n        name = \"ntab_enable_override_max_count\",\n        default = false\n      )\n\n  /**\n   * Param to determine the lookback duration for override candidates on NTab.\n   */\n  object OverrideNotificationsLookbackDurationForNTab\n      extends FSBoundedParam[Duration](\n        name = \"ntab_override_lookback_duration_days\",\n        default = 30.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Param to determine the max count for candidates on NTab.\n   */\n  object OverrideNotificationsMaxCountForNTab\n      extends FSBoundedParam[Int](\n        name = \"ntab_override_limit\",\n        min = 0,\n        max = Int.MaxValue,\n        default = 4)\n\n  //// end override experiments ////\n  /**\n   * Param to enable top tweet impressions notification\n   */\n  object EnableTopTweetImpressionsNotification\n      extends FSParam[Boolean](\n        name = \"top_tweet_impressions_notification_enable\",\n        default = false\n      )\n\n  /**\n   * Param to control the inverter for fatigue between consecutive TweetImpressions\n   */\n  object TopTweetImpressionsNotificationInterval\n      extends FSBoundedParam[Duration](\n        name = \"top_tweet_impressions_notification_interval_days\",\n        default = 7.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * The min-interval between TweetImpressions notifications.\n   * After receiving a TweetImpressions notif, they must wait a minimum of this\n   * interval before being eligible for another\n   */\n  object TopTweetImpressionsFatigueMinIntervalDuration\n      extends FSBoundedParam[Duration](\n        name = \"top_tweet_impressions_fatigue_mininterval_duration_days\",\n        default = 1.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Maximum number of top tweet impressions notifications to receive in an interval\n   */\n  object MaxTopTweetImpressionsNotifications\n      extends FSBoundedParam(\n        name = \"top_tweet_impressions_fatigue_max_in_interval\",\n        default = 0,\n        min = 0,\n        max = 10\n      )\n\n  /**\n   * Param for min number of impressions counts to be eligible for lonely_birds_tweet_impressions model\n   */\n  object TopTweetImpressionsMinRequired\n      extends FSBoundedParam[Int](\n        name = \"top_tweet_impressions_min_required\",\n        default = 25,\n        min = 0,\n        max = Int.MaxValue\n      )\n\n  /**\n   * Param for threshold of impressions counts to notify for lonely_birds_tweet_impressions model\n   */\n  object TopTweetImpressionsThreshold\n      extends FSBoundedParam[Int](\n        name = \"top_tweet_impressions_threshold\",\n        default = 25,\n        min = 0,\n        max = Int.MaxValue\n      )\n\n  /**\n   * Param for the number of days to search up to for a user's original tweets\n   */\n  object TopTweetImpressionsOriginalTweetsNumDaysSearch\n      extends FSBoundedParam[Int](\n        name = \"top_tweet_impressions_original_tweets_num_days_search\",\n        default = 3,\n        min = 0,\n        max = 21\n      )\n\n  /**\n   * Param for the minimum number of original tweets a user needs to be considered an original author\n   */\n  object TopTweetImpressionsMinNumOriginalTweets\n      extends FSBoundedParam[Int](\n        name = \"top_tweet_impressions_num_original_tweets\",\n        default = 3,\n        min = 0,\n        max = Int.MaxValue\n      )\n\n  /**\n   * Param for the max number of favorites any original Tweet can have\n   */\n  object TopTweetImpressionsMaxFavoritesPerTweet\n      extends FSBoundedParam[Int](\n        name = \"top_tweet_impressions_max_favorites_per_tweet\",\n        default = 3,\n        min = 0,\n        max = Int.MaxValue\n      )\n\n  /**\n   * Param for the max number of total inbound favorites for a user's tweets\n   */\n  object TopTweetImpressionsTotalInboundFavoritesLimit\n      extends FSBoundedParam[Int](\n        name = \"top_tweet_impressions_total_inbound_favorites_limit\",\n        default = 60,\n        min = 0,\n        max = Int.MaxValue\n      )\n\n  /**\n   * Param for the number of days to search for tweets to count the total inbound favorites\n   */\n  object TopTweetImpressionsTotalFavoritesLimitNumDaysSearch\n      extends FSBoundedParam[Int](\n        name = \"top_tweet_impressions_total_favorites_limit_num_days_search\",\n        default = 7,\n        min = 0,\n        max = 21\n      )\n\n  /**\n   * Param for the max number of recent tweets Tflock should return\n   */\n  object TopTweetImpressionsRecentTweetsByAuthorStoreMaxResults\n      extends FSBoundedParam[Int](\n        name = \"top_tweet_impressions_recent_tweets_by_author_store_max_results\",\n        default = 50,\n        min = 0,\n        max = 1000\n      )\n\n  /*\n   * Param to represent the max number of slots to maintain for Override Notifications\n   */\n  object OverrideNotificationsMaxNumOfSlots\n      extends FSBoundedParam[Int](\n        name = \"mr_override_max_num_slots\",\n        default = 1,\n        max = 10,\n        min = 1\n      )\n\n  object EnableOverrideMaxSlotFn\n      extends FSParam[Boolean](\n        name = \"mr_override_enable_max_num_slots_fn\",\n        default = false\n      )\n\n  object OverrideMaxSlotFnPushCapKnobs\n      extends FSParam[Seq[Double]](\"mr_override_fn_pushcap_knobs\", default = Seq.empty[Double])\n\n  object OverrideMaxSlotFnNSlotKnobs\n      extends FSParam[Seq[Double]](\"mr_override_fn_nslot_knobs\", default = Seq.empty[Double])\n\n  object OverrideMaxSlotFnPowerKnobs\n      extends FSParam[Seq[Double]](\"mr_override_fn_power_knobs\", default = Seq.empty[Double])\n\n  object OverrideMaxSlotFnWeight\n      extends FSBoundedParam[Double](\n        \"mr_override_fn_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = Double.MaxValue)\n\n  /**\n   * Use to enable sending target ids in the Smart Push Payload\n   */\n  object EnableTargetIdsInSmartPushPayload\n      extends FSParam[Boolean](name = \"mr_override_enable_target_ids\", default = true)\n\n  /**\n   * Param to enable override by target id for MagicFanoutSportsEvent candidates\n   */\n  object EnableTargetIdInSmartPushPayloadForMagicFanoutSportsEvent\n      extends FSParam[Boolean](\n        name = \"mr_override_enable_target_id_for_magic_fanout_sports_event\",\n        default = true)\n\n  /**\n   * Param to enable secondary account predicate on MF NFY\n   */\n  object EnableSecondaryAccountPredicateMF\n      extends FSParam[Boolean](\n        name = \"frigate_push_magicfanout_secondary_account_predicate\",\n        default = false\n      )\n\n  /**\n   * Enables showing our customers videos on their notifications\n   */\n  object EnableInlineVideo\n      extends FSParam[Boolean](name = \"mr_inline_enable_inline_video\", default = false)\n\n  /**\n   * Enables autoplay for inline videos\n   */\n  object EnableAutoplayForInlineVideo\n      extends FSParam[Boolean](name = \"mr_inline_enable_autoplay_for_inline_video\", default = false)\n\n  /**\n   * Enable OON filtering based on MentionFilter.\n   */\n  object EnableOONFilteringBasedOnUserSettings\n      extends FSParam[Boolean](name = \"oon_filtering_enable_based_on_user_settings\", false)\n\n  /**\n   * Enables Custom Thread Ids which is used to ungroup notifications for N-slots on iOS\n   */\n  object EnableCustomThreadIdForOverride\n      extends FSParam[Boolean](name = \"mr_override_enable_custom_thread_id\", default = false)\n\n  /**\n   * Enables showing verified symbol in the push presentation\n   */\n  object EnablePushPresentationVerifiedSymbol\n      extends FSParam[Boolean](name = \"push_presentation_enable_verified_symbol\", default = false)\n\n  /**\n   * Decide subtext in Android push header\n   */\n  object SubtextInAndroidPushHeaderParam\n      extends FSEnumParam[SubtextForAndroidPushHeader.type](\n        name = \"push_presentation_subtext_in_android_push_header_id\",\n        default = SubtextForAndroidPushHeader.None,\n        enum = SubtextForAndroidPushHeader)\n\n  /**\n   * Enable SimClusters Targeting For Spaces. If false we just drop all candidates with such targeting reason\n   */\n  object EnableSimClusterTargetingSpaces\n      extends FSParam[Boolean](name = \"space_recs_send_simcluster_recommendations\", default = false)\n\n  /**\n   * Param to control threshold for dot product of simcluster based targeting on Spaces\n   */\n  object SpacesTargetingSimClusterDotProductThreshold\n      extends FSBoundedParam[Double](\n        \"space_recs_simclusters_dot_product_threshold\",\n        default = 0.0,\n        min = 0.0,\n        max = 10.0)\n\n  /**\n   * Param to control top-k clusters simcluster based targeting on Spaces\n   */\n  object SpacesTopKSimClusterCount\n      extends FSBoundedParam[Int](\n        \"space_recs_simclusters_top_k_count\",\n        default = 1,\n        min = 1,\n        max = 50)\n\n  /** SimCluster users host/speaker must meet this follower count minimum threshold to be considered for sends */\n  object SpaceRecsSimClusterUserMinimumFollowerCount\n      extends FSBoundedParam[Int](\n        name = \"space_recs_simcluster_user_min_follower_count\",\n        default = 5000,\n        max = Int.MaxValue,\n        min = 0\n      )\n\n  /**\n   * Target has been bucketed into the Inline Action App Visit Fatigue Experiment\n   */\n  object TargetInInlineActionAppVisitFatigue\n      extends FSParam[Boolean](name = \"inline_action_target_in_app_visit_fatigue\", default = false)\n\n  /**\n   * Enables Inline Action App Visit Fatigue\n   */\n  object EnableInlineActionAppVisitFatigue\n      extends FSParam[Boolean](name = \"inline_action_enable_app_visit_fatigue\", default = false)\n\n  /**\n   * Determines the fatigue that we should apply when the target user has performed an inline action\n   */\n  object InlineActionAppVisitFatigue\n      extends FSBoundedParam[Duration](\n        name = \"inline_action_app_visit_fatigue_hours\",\n        default = 8.hours,\n        min = 1.hour,\n        max = 48.hours)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Weight for reranking(oonc - weight * nudityRate)\n   */\n  object AuthorSensitiveScoreWeightInReranking\n      extends FSBoundedParam[Double](\n        name = \"rerank_candidates_author_sensitive_score_weight_in_reranking\",\n        default = 0.0,\n        min = -100.0,\n        max = 100.0\n      )\n\n  /**\n   * Param to control the last active space listener threshold to filter out based on that\n   */\n  object SpaceParticipantHistoryLastActiveThreshold\n      extends FSBoundedParam[Duration](\n        name = \"space_recs_last_active_space_listener_threshold_in_hours\",\n        default = 0.hours,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /*\n   * Param to enable mr user simcluster feature set (v2020) hydration for modeling-based candidate generation\n   * */\n  object HydrateMrUserSimclusterV2020InModelingBasedCG\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_hydrate_mr_user_simcluster_v2020\",\n        default = false)\n\n  /*\n   * Param to enable mr semantic core feature set hydration for modeling-based candidate generation\n   * */\n  object HydrateMrUserSemanticCoreInModelingBasedCG\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_hydrate_mr_user_semantic_core\",\n        default = false)\n\n  /*\n   * Param to enable mr semantic core feature set hydration for modeling-based candidate generation\n   * */\n  object HydrateOnboardingInModelingBasedCG\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_hydrate_onboarding\",\n        default = false)\n\n  /*\n   * Param to enable mr topic follow feature set hydration for modeling-based candidate generation\n   * */\n  object HydrateTopicFollowInModelingBasedCG\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_hydrate_topic_follow\",\n        default = false)\n\n  /*\n   * Param to enable mr user topic feature set hydration for modeling-based candidate generation\n   * */\n  object HydrateMrUserTopicInModelingBasedCG\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_hydrate_mr_user_topic\",\n        default = false)\n\n  /*\n   * Param to enable mr user topic feature set hydration for modeling-based candidate generation\n   * */\n  object HydrateMrUserAuthorInModelingBasedCG\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_hydrate_mr_user_author\",\n        default = false)\n\n  /*\n   * Param to enable user penguin language feature set hydration for modeling-based candidate generation\n   * */\n  object HydrateUserPenguinLanguageInModelingBasedCG\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_hydrate_user_penguin_language\",\n        default = false)\n  /*\n   * Param to enable user geo feature set hydration for modeling-based candidate generation\n   * */\n  object HydrateUseGeoInModelingBasedCG\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_hydrate_user_geo\",\n        default = false)\n\n  /*\n   * Param to enable mr user hashspace embedding feature set hydration for modeling-based candidate generation\n   * */\n  object HydrateMrUserHashspaceEmbeddingInModelingBasedCG\n      extends FSParam[Boolean](\n        name = \"candidate_generation_model_hydrate_mr_user_hashspace_embedding\",\n        default = false)\n  /*\n   * Param to enable user tweet text feature hydration\n   * */\n  object EnableMrUserEngagedTweetTokensFeature\n      extends FSParam[Boolean](\n        name = \"feature_hydration_mr_user_engaged_tweet_tokens\",\n        default = false)\n\n  /**\n   * Params for CRT based see less often fatigue rules\n   */\n  object EnableF1TriggerSeeLessOftenFatigue\n      extends FSParam[Boolean](\n        name = \"seelessoften_enable_f1_trigger_fatigue\",\n        default = false\n      )\n\n  object EnableNonF1TriggerSeeLessOftenFatigue\n      extends FSParam[Boolean](\n        name = \"seelessoften_enable_nonf1_trigger_fatigue\",\n        default = false\n      )\n\n  /**\n   * Adjust the NtabCaretClickFatigue for candidates if it is triggered by\n   * TripHqTweet candidates\n   */\n  object AdjustTripHqTweetTriggeredNtabCaretClickFatigue\n      extends FSParam[Boolean](\n        name = \"seelessoften_adjust_trip_hq_tweet_triggered_fatigue\",\n        default = false\n      )\n\n  object NumberOfDaysToFilterForSeeLessOftenForF1TriggerF1\n      extends FSBoundedParam[Duration](\n        name = \"seelessoften_for_f1_trigger_f1_tofiltermr_days\",\n        default = 7.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  object NumberOfDaysToReducePushCapForSeeLessOftenForF1TriggerF1\n      extends FSBoundedParam[Duration](\n        name = \"seelessoften_for_f1_trigger_f1_toreduce_pushcap_days\",\n        default = 30.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  object NumberOfDaysToFilterForSeeLessOftenForF1TriggerNonF1\n      extends FSBoundedParam[Duration](\n        name = \"seelessoften_for_f1_trigger_nonf1_tofiltermr_days\",\n        default = 7.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  object NumberOfDaysToReducePushCapForSeeLessOftenForF1TriggerNonF1\n      extends FSBoundedParam[Duration](\n        name = \"seelessoften_for_f1_trigger_non_f1_toreduce_pushcap_days\",\n        default = 30.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  object NumberOfDaysToFilterForSeeLessOftenForNonF1TriggerF1\n      extends FSBoundedParam[Duration](\n        name = \"seelessoften_for_nonf1_trigger_f1_tofiltermr_days\",\n        default = 7.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  object NumberOfDaysToReducePushCapForSeeLessOftenForNonF1TriggerF1\n      extends FSBoundedParam[Duration](\n        name = \"seelessoften_for_nonf1_trigger_f1_toreduce_pushcap_days\",\n        default = 30.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  object NumberOfDaysToFilterForSeeLessOftenForNonF1TriggerNonF1\n      extends FSBoundedParam[Duration](\n        name = \"seelessoften_for_nonf1_trigger_nonf1_tofiltermr_days\",\n        default = 7.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  object NumberOfDaysToReducePushCapForSeeLessOftenForNonF1TriggerNonF1\n      extends FSBoundedParam[Duration](\n        name = \"seelessoften_for_nonf1_trigger_nonf1_toreduce_pushcap_days\",\n        default = 30.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  object EnableContFnF1TriggerSeeLessOftenFatigue\n      extends FSParam[Boolean](\n        name = \"seelessoften_fn_enable_f1_trigger_fatigue\",\n        default = false\n      )\n\n  object EnableContFnNonF1TriggerSeeLessOftenFatigue\n      extends FSParam[Boolean](\n        name = \"seelessoften_fn_enable_nonf1_trigger_fatigue\",\n        default = false\n      )\n\n  object SeeLessOftenListOfDayKnobs\n      extends FSParam[Seq[Double]](\"seelessoften_fn_day_knobs\", default = Seq.empty[Double])\n\n  object SeeLessOftenListOfPushCapWeightKnobs\n      extends FSParam[Seq[Double]](\"seelessoften_fn_pushcap_knobs\", default = Seq.empty[Double])\n\n  object SeeLessOftenListOfPowerKnobs\n      extends FSParam[Seq[Double]](\"seelessoften_fn_power_knobs\", default = Seq.empty[Double])\n\n  object SeeLessOftenF1TriggerF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_f1_trigger_f1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object SeeLessOftenF1TriggerNonF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_f1_trigger_nonf1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object SeeLessOftenNonF1TriggerF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_nonf1_trigger_f1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object SeeLessOftenNonF1TriggerNonF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_nonf1_trigger_nonf1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object SeeLessOftenTripHqTweetTriggerF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_trip_hq_tweet_trigger_f1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object SeeLessOftenTripHqTweetTriggerNonF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_trip_hq_tweet_trigger_nonf1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object SeeLessOftenTripHqTweetTriggerTripHqTweetPushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_trip_hq_tweet_trigger_trip_hq_tweet_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object SeeLessOftenTopicTriggerTopicPushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_topic_trigger_topic_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = Double.MaxValue)\n\n  object SeeLessOftenTopicTriggerF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_topic_trigger_f1_weight\",\n        default = 100000.0,\n        min = 0.0,\n        max = Double.MaxValue)\n\n  object SeeLessOftenTopicTriggerOONPushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_topic_trigger_oon_weight\",\n        default = 100000.0,\n        min = 0.0,\n        max = Double.MaxValue)\n\n  object SeeLessOftenF1TriggerTopicPushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_f1_trigger_topic_weight\",\n        default = 100000.0,\n        min = 0.0,\n        max = Double.MaxValue)\n\n  object SeeLessOftenOONTriggerTopicPushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_oon_trigger_topic_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = Double.MaxValue)\n\n  object SeeLessOftenDefaultPushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_default_weight\",\n        default = 100000.0,\n        min = 0.0,\n        max = Double.MaxValue)\n\n  object SeeLessOftenNtabOnlyNotifUserPushCapWeight\n      extends FSBoundedParam[Double](\n        \"seelessoften_fn_ntab_only_user_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = Double.MaxValue)\n\n  // Params for inline feedback fatigue\n  object EnableContFnF1TriggerInlineFeedbackFatigue\n      extends FSParam[Boolean](\n        name = \"feedback_inline_fn_enable_f1_trigger_fatigue\",\n        default = false\n      )\n\n  object EnableContFnNonF1TriggerInlineFeedbackFatigue\n      extends FSParam[Boolean](\n        name = \"feedback_inline_fn_enable_nonf1_trigger_fatigue\",\n        default = false\n      )\n\n  object UseInlineDislikeForFatigue\n      extends FSParam[Boolean](\n        name = \"feedback_inline_fn_use_dislike\",\n        default = true\n      )\n  object UseInlineDismissForFatigue\n      extends FSParam[Boolean](\n        name = \"feedback_inline_fn_use_dismiss\",\n        default = false\n      )\n  object UseInlineSeeLessForFatigue\n      extends FSParam[Boolean](\n        name = \"feedback_inline_fn_use_see_less\",\n        default = false\n      )\n  object UseInlineNotRelevantForFatigue\n      extends FSParam[Boolean](\n        name = \"feedback_inline_fn_use_not_relevant\",\n        default = false\n      )\n  object InlineFeedbackListOfDayKnobs\n      extends FSParam[Seq[Double]](\"feedback_inline_fn_day_knobs\", default = Seq.empty[Double])\n\n  object InlineFeedbackListOfPushCapWeightKnobs\n      extends FSParam[Seq[Double]](\"feedback_inline_fn_pushcap_knobs\", default = Seq.empty[Double])\n\n  object InlineFeedbackListOfPowerKnobs\n      extends FSParam[Seq[Double]](\"feedback_inline_fn_power_knobs\", default = Seq.empty[Double])\n\n  object InlineFeedbackF1TriggerF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"feedback_inline_fn_f1_trigger_f1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object InlineFeedbackF1TriggerNonF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"feedback_inline_fn_f1_trigger_nonf1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object InlineFeedbackNonF1TriggerF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"feedback_inline_fn_nonf1_trigger_f1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object InlineFeedbackNonF1TriggerNonF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"feedback_inline_fn_nonf1_trigger_nonf1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  // Params for prompt feedback\n  object EnableContFnF1TriggerPromptFeedbackFatigue\n      extends FSParam[Boolean](\n        name = \"feedback_prompt_fn_enable_f1_trigger_fatigue\",\n        default = false\n      )\n\n  object EnableContFnNonF1TriggerPromptFeedbackFatigue\n      extends FSParam[Boolean](\n        name = \"feedback_prompt_fn_enable_nonf1_trigger_fatigue\",\n        default = false\n      )\n  object PromptFeedbackListOfDayKnobs\n      extends FSParam[Seq[Double]](\"feedback_prompt_fn_day_knobs\", default = Seq.empty[Double])\n\n  object PromptFeedbackListOfPushCapWeightKnobs\n      extends FSParam[Seq[Double]](\"feedback_prompt_fn_pushcap_knobs\", default = Seq.empty[Double])\n\n  object PromptFeedbackListOfPowerKnobs\n      extends FSParam[Seq[Double]](\"feedback_prompt_fn_power_knobs\", default = Seq.empty[Double])\n\n  object PromptFeedbackF1TriggerF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"feedback_prompt_fn_f1_trigger_f1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object PromptFeedbackF1TriggerNonF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"feedback_prompt_fn_f1_trigger_nonf1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object PromptFeedbackNonF1TriggerF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"feedback_prompt_fn_nonf1_trigger_f1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  object PromptFeedbackNonF1TriggerNonF1PushCapWeight\n      extends FSBoundedParam[Double](\n        \"feedback_prompt_fn_nonf1_trigger_nonf1_weight\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000000.0)\n\n  /*\n   * Param to enable cohost join event notif\n   */\n  object EnableSpaceCohostJoinEvent\n      extends FSParam[Boolean](name = \"space_recs_cohost_join_enable\", default = true)\n\n  /*\n   * Param to bypass global push cap when target is device following host/speaker.\n   */\n  object BypassGlobalSpacePushCapForSoftDeviceFollow\n      extends FSParam[Boolean](name = \"space_recs_bypass_global_pushcap_for_soft_follow\", false)\n\n  /*\n   * Param to bypass active listener predicate when target is device following host/speaker.\n   */\n  object CheckActiveListenerPredicateForSoftDeviceFollow\n      extends FSParam[Boolean](name = \"space_recs_check_active_listener_for_soft_follow\", false)\n\n  object SpreadControlRatioParam\n      extends FSBoundedParam[Double](\n        name = \"oon_spread_control_ratio\",\n        default = 1000.0,\n        min = 0.0,\n        max = 100000.0\n      )\n\n  object FavOverSendThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_spread_control_fav_over_send_threshold\",\n        default = 0.14,\n        min = 0.0,\n        max = 1000.0\n      )\n\n  object AuthorReportRateThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_spread_control_author_report_rate_threshold\",\n        default = 7.4e-6,\n        min = 0.0,\n        max = 1000.0\n      )\n\n  object AuthorDislikeRateThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_spread_control_author_dislike_rate_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1000.0\n      )\n\n  object MinTweetSendsThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_spread_control_min_tweet_sends_threshold\",\n        default = 10000000000.0,\n        min = 0.0,\n        max = 10000000000.0\n      )\n\n  object MinAuthorSendsThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_spread_control_min_author_sends_threshold\",\n        default = 10000000000.0,\n        min = 0.0,\n        max = 10000000000.0\n      )\n\n  /*\n   * Tweet Ntab-dislike predicate related params\n   */\n  object TweetNtabDislikeCountThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_tweet_ntab_dislike_count_threshold\",\n        default = 10000.0,\n        min = 0.0,\n        max = 10000.0\n      )\n  object TweetNtabDislikeRateThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_tweet_ntab_dislike_rate_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Param for tweet language feature name\n   */\n  object TweetLanguageFeatureNameParam\n      extends FSParam[String](\n        name = \"language_tweet_language_feature_name\",\n        default = \"tweet.language.tweet.identified\")\n\n  /**\n   * Threshold for user inferred language filtering\n   */\n  object UserInferredLanguageThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"language_user_inferred_language_threshold\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Threshold for user device language filtering\n   */\n  object UserDeviceLanguageThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"language_user_device_language_threshold\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Param to enable/disable tweet language filter\n   */\n  object EnableTweetLanguageFilter\n      extends FSParam[Boolean](\n        name = \"language_enable_tweet_language_filter\",\n        default = false\n      )\n\n  /**\n   * Param to skip language filter for media tweets\n   */\n  object SkipLanguageFilterForMediaTweets\n      extends FSParam[Boolean](\n        name = \"language_skip_language_filter_for_media_tweets\",\n        default = false\n      )\n\n  /*\n   * Tweet Ntab-dislike predicate related params for MrTwistly\n   */\n  object TweetNtabDislikeCountThresholdForMrTwistlyParam\n      extends FSBoundedParam[Double](\n        name = \"oon_tweet_ntab_dislike_count_threshold_for_mrtwistly\",\n        default = 10000.0,\n        min = 0.0,\n        max = 10000.0\n      )\n  object TweetNtabDislikeRateThresholdForMrTwistlyParam\n      extends FSBoundedParam[Double](\n        name = \"oon_tweet_ntab_dislike_rate_threshold_for_mrtwistly\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object TweetNtabDislikeCountBucketThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_tweet_ntab_dislike_count_bucket_threshold\",\n        default = 10.0,\n        min = 0.0,\n        max = 10000.0\n      )\n\n  /*\n   * Tweet engagement ratio predicate related params\n   */\n  object TweetQTtoNtabClickRatioThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_tweet_engagement_filter_qt_to_ntabclick_ratio_threshold\",\n        default = 0.0,\n        min = 0.0,\n        max = 100000.0\n      )\n\n  /**\n   * Lower bound threshold to filter a tweet based on its reply to like ratio\n   */\n  object TweetReplytoLikeRatioThresholdLowerBound\n      extends FSBoundedParam[Double](\n        name = \"oon_tweet_engagement_filter_reply_to_like_ratio_threshold_lower_bound\",\n        default = Double.MaxValue,\n        min = 0.0,\n        max = Double.MaxValue\n      )\n\n  /**\n   * Upper bound threshold to filter a tweet based on its reply to like ratio\n   */\n  object TweetReplytoLikeRatioThresholdUpperBound\n      extends FSBoundedParam[Double](\n        name = \"oon_tweet_engagement_filter_reply_to_like_ratio_threshold_upper_bound\",\n        default = 0.0,\n        min = 0.0,\n        max = Double.MaxValue\n      )\n\n  /**\n   * Upper bound threshold to filter a tweet based on its reply to like ratio\n   */\n  object TweetReplytoLikeRatioReplyCountThreshold\n      extends FSBoundedParam[Int](\n        name = \"oon_tweet_engagement_filter_reply_count_threshold\",\n        default = Int.MaxValue,\n        min = 0,\n        max = Int.MaxValue\n      )\n\n  /*\n   * oonTweetLengthBasedPrerankingPredicate related params\n   */\n  object OonTweetLengthPredicateUpdatedMediaLogic\n      extends FSParam[Boolean](\n        name = \"oon_quality_filter_tweet_length_updated_media_logic\",\n        default = false\n      )\n\n  object OonTweetLengthPredicateUpdatedQuoteTweetLogic\n      extends FSParam[Boolean](\n        name = \"oon_quality_filter_tweet_length_updated_quote_tweet_logic\",\n        default = false\n      )\n\n  object OonTweetLengthPredicateMoreStrictForUndefinedLanguages\n      extends FSParam[Boolean](\n        name = \"oon_quality_filter_tweet_length_more_strict_for_undefined_languages\",\n        default = false\n      )\n\n  object EnablePrerankingTweetLengthPredicate\n      extends FSParam[Boolean](\n        name = \"oon_quality_filter_enable_preranking_filter\",\n        default = false\n      )\n\n  /*\n   * LengthLanguageBasedOONTweetCandidatesQualityPredicate related params\n   */\n  object SautOonWithMediaTweetLengthThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_quality_filter_tweet_length_threshold_for_saut_oon_with_media\",\n        default = 0.0,\n        min = 0.0,\n        max = 70.0\n      )\n  object NonSautOonWithMediaTweetLengthThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_quality_filter_tweet_length_threshold_for_non_saut_oon_with_media\",\n        default = 0.0,\n        min = 0.0,\n        max = 70.0\n      )\n  object SautOonWithoutMediaTweetLengthThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_quality_filter_tweet_length_threshold_for_saut_oon_without_media\",\n        default = 0.0,\n        min = 0.0,\n        max = 70.0\n      )\n  object NonSautOonWithoutMediaTweetLengthThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_quality_filter_tweet_length_threshold_for_non_saut_oon_without_media\",\n        default = 0.0,\n        min = 0.0,\n        max = 70.0\n      )\n\n  object ArgfOonWithMediaTweetWordLengthThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_quality_filter_tweet_word_length_threshold_for_argf_oon_with_media\",\n        default = 0.0,\n        min = 0.0,\n        max = 18.0\n      )\n  object EsfthOonWithMediaTweetWordLengthThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"oon_quality_filter_tweet_word_length_threshold_for_esfth_oon_with_media\",\n        default = 0.0,\n        min = 0.0,\n        max = 10.0\n      )\n\n  /**\n   * Param to enable/disable sentiment feature hydration\n   */\n  object EnableMrTweetSentimentFeatureHydrationFS\n      extends FSParam[Boolean](\n        name = \"feature_hydration_enable_mr_tweet_sentiment_feature\",\n        default = false\n      )\n\n  /**\n   * Param to enable/disable feature map scribing for staging test log\n   */\n  object EnableMrScribingMLFeaturesAsFeatureMapForStaging\n      extends FSParam[Boolean](\n        name = \"frigate_pushservice_enable_scribing_ml_features_as_featuremap_for_staging\",\n        default = false\n      )\n\n  /**\n   * Param to enable timeline health signal hydration\n   * */\n  object EnableTimelineHealthSignalHydration\n      extends FSParam[Boolean](\n        name = \"timeline_health_signal_hydration\",\n        default = false\n      )\n\n  /**\n   * Param to enable timeline health signal hydration for model training\n   * */\n  object EnableTimelineHealthSignalHydrationForModelTraining\n      extends FSParam[Boolean](\n        name = \"timeline_health_signal_hydration_for_model_training\",\n        default = false\n      )\n\n  /**\n   * Param to enable/disable mr user social context agg feature hydration\n   */\n  object EnableMrUserSocialContextAggregateFeatureHydration\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_hydrate_mr_user_social_context_agg_feature\",\n        default = true\n      )\n\n  /**\n   * Param to enable/disable mr user semantic core agg feature hydration\n   */\n  object EnableMrUserSemanticCoreAggregateFeatureHydration\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_hydrate_mr_user_semantic_core_agg_feature\",\n        default = true\n      )\n\n  /**\n   * Param to enable/disable mr user candidate sparse agg feature hydration\n   */\n  object EnableMrUserCandidateSparseOfflineAggregateFeatureHydration\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_hydrate_mr_user_candidate_sparse_agg_feature\",\n        default = true\n      )\n\n  /**\n   * Param to enable/disable mr user candidate agg feature hydration\n   */\n  object EnableMrUserCandidateOfflineAggregateFeatureHydration\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_hydrate_mr_user_candidate_agg_feature\",\n        default = true\n      )\n\n  /**\n   * Param to enable/disable mr user candidate compact agg feature hydration\n   */\n  object EnableMrUserCandidateOfflineCompactAggregateFeatureHydration\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_hydrate_mr_user_candidate_compact_agg_feature\",\n        default = false\n      )\n\n  /**\n   * Param to enable/disable mr real graph user-author/social-context feature hydration\n   */\n  object EnableRealGraphUserAuthorAndSocialContxtFeatureHydration\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_hydrate_real_graph_user_social_feature\",\n        default = true\n      )\n\n  /**\n   * Param to enable/disable mr user author agg feature hydration\n   */\n  object EnableMrUserAuthorOfflineAggregateFeatureHydration\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_hydrate_mr_user_author_agg_feature\",\n        default = true\n      )\n\n  /**\n   * Param to enable/disable mr user author compact agg feature hydration\n   */\n  object EnableMrUserAuthorOfflineCompactAggregateFeatureHydration\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_hydrate_mr_user_author_compact_agg_feature\",\n        default = false\n      )\n\n  /**\n   * Param to enable/disable mr user compact agg feature hydration\n   */\n  object EnableMrUserOfflineCompactAggregateFeatureHydration\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_hydrate_mr_user_compact_agg_feature\",\n        default = false\n      )\n\n  /**\n   * Param to enable/disable mr user simcluster agg feature hydration\n   */\n  object EnableMrUserSimcluster2020AggregateFeatureHydration\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_hydrate_mr_user_simcluster_agg_feature\",\n        default = true\n      )\n\n  /**\n   * Param to enable/disable mr user agg feature hydration\n   */\n  object EnableMrUserOfflineAggregateFeatureHydration\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_hydrate_mr_user_agg_feature\",\n        default = true\n      )\n\n  /**\n   * Param to enable/disable topic engagement RTA in the ranking model\n   */\n  object EnableTopicEngagementRealTimeAggregatesFS\n      extends FSParam[Boolean](\n        \"feature_hydration_enable_htl_topic_engagement_real_time_agg_feature\",\n        false)\n\n  /*\n   * Param to enable mr user semantic core feature hydration for heavy ranker\n   * */\n  object EnableMrUserSemanticCoreFeatureForExpt\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_hydrate_mr_user_semantic_core\",\n        default = false)\n\n  /**\n   * Param to enable hydrating user duration since last visit features\n   */\n  object EnableHydratingUserDurationSinceLastVisitFeatures\n      extends FSParam[Boolean](\n        name = \"feature_hydration_user_duration_since_last_visit\",\n        default = false)\n\n  /**\n    Param to enable/disable user-topic aggregates in the ranking model\n   */\n  object EnableUserTopicAggregatesFS\n      extends FSParam[Boolean](\"feature_hydration_enable_htl_topic_user_agg_feature\", false)\n\n  /*\n   * PNegMultimodalPredicate related params\n   */\n  object EnablePNegMultimodalPredicateParam\n      extends FSParam[Boolean](\n        name = \"pneg_multimodal_filter_enable_param\",\n        default = false\n      )\n  object PNegMultimodalPredicateModelThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"pneg_multimodal_filter_model_threshold_param\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n  object PNegMultimodalPredicateBucketThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"pneg_multimodal_filter_bucket_threshold_param\",\n        default = 0.4,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /*\n   * NegativeKeywordsPredicate related params\n   */\n  object EnableNegativeKeywordsPredicateParam\n      extends FSParam[Boolean](\n        name = \"negative_keywords_filter_enable_param\",\n        default = false\n      )\n  object NegativeKeywordsPredicateDenylist\n      extends FSParam[Seq[String]](\n        name = \"negative_keywords_filter_denylist\",\n        default = Seq.empty[String]\n      )\n  /*\n   * LightRanking related params\n   */\n  object EnableLightRankingParam\n      extends FSParam[Boolean](\n        name = \"light_ranking_enable_param\",\n        default = false\n      )\n  object LightRankingNumberOfCandidatesParam\n      extends FSBoundedParam[Int](\n        name = \"light_ranking_number_of_candidates_param\",\n        default = 100,\n        min = 0,\n        max = 1000\n      )\n  object LightRankingModelTypeParam\n      extends FSParam[String](\n        name = \"light_ranking_model_type_param\",\n        default = \"WeightedOpenOrNtabClickProbability_Q4_2021_13172_Mr_Light_Ranker_Dbv2_Top3\")\n  object EnableRandomBaselineLightRankingParam\n      extends FSParam[Boolean](\n        name = \"light_ranking_random_baseline_enable_param\",\n        default = false\n      )\n\n  object LightRankingScribeCandidatesDownSamplingParam\n      extends FSBoundedParam[Double](\n        name = \"light_ranking_scribe_candidates_down_sampling_param\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /*\n   * Quality Upranking related params\n   */\n  object EnableProducersQualityBoostingForHeavyRankingParam\n      extends FSParam[Boolean](\n        name = \"quality_upranking_enable_producers_quality_boosting_for_heavy_ranking_param\",\n        default = false\n      )\n\n  object QualityUprankingBoostForHighQualityProducersParam\n      extends FSBoundedParam[Double](\n        name = \"quality_upranking_boost_for_high_quality_producers_param\",\n        default = 1.0,\n        min = 0.0,\n        max = 10000.0\n      )\n\n  object QualityUprankingDownboostForLowQualityProducersParam\n      extends FSBoundedParam[Double](\n        name = \"quality_upranking_downboost_for_low_quality_producers_param\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object EnableQualityUprankingForHeavyRankingParam\n      extends FSParam[Boolean](\n        name = \"quality_upranking_enable_for_heavy_ranking_param\",\n        default = false\n      )\n  object QualityUprankingModelTypeParam\n      extends FSParam[WeightedOpenOrNtabClickModel.ModelNameType](\n        name = \"quality_upranking_model_id\",\n        default = \"Q4_2022_Mr_Bqml_Quality_Model_wALL\"\n      )\n  object QualityUprankingTransformTypeParam\n      extends FSEnumParam[MrQualityUprankingTransformTypeEnum.type](\n        name = \"quality_upranking_transform_id\",\n        default = MrQualityUprankingTransformTypeEnum.Sigmoid,\n        enum = MrQualityUprankingTransformTypeEnum\n      )\n\n  object QualityUprankingBoostForHeavyRankingParam\n      extends FSBoundedParam[Double](\n        name = \"quality_upranking_boost_for_heavy_ranking_param\",\n        default = 1.0,\n        min = -10.0,\n        max = 10.0\n      )\n  object QualityUprankingSigmoidBiasForHeavyRankingParam\n      extends FSBoundedParam[Double](\n        name = \"quality_upranking_sigmoid_bias_for_heavy_ranking_param\",\n        default = 0.0,\n        min = -10.0,\n        max = 10.0\n      )\n  object QualityUprankingSigmoidWeightForHeavyRankingParam\n      extends FSBoundedParam[Double](\n        name = \"quality_upranking_sigmoid_weight_for_heavy_ranking_param\",\n        default = 1.0,\n        min = -10.0,\n        max = 10.0\n      )\n  object QualityUprankingLinearBarForHeavyRankingParam\n      extends FSBoundedParam[Double](\n        name = \"quality_upranking_linear_bar_for_heavy_ranking_param\",\n        default = 1.0,\n        min = 0.0,\n        max = 10.0\n      )\n  object EnableQualityUprankingCrtScoreStatsForHeavyRankingParam\n      extends FSParam[Boolean](\n        name = \"quality_upranking_enable_crt_score_stats_for_heavy_ranking_param\",\n        default = false\n      )\n  /*\n   * BQML Health Model related params\n   */\n  object EnableBqmlHealthModelPredicateParam\n      extends FSParam[Boolean](\n        name = \"bqml_health_model_filter_enable_param\",\n        default = false\n      )\n\n  object EnableBqmlHealthModelPredictionForInNetworkCandidatesParam\n      extends FSParam[Boolean](\n        name = \"bqml_health_model_enable_prediction_for_in_network_candidates_param\",\n        default = false\n      )\n\n  object BqmlHealthModelTypeParam\n      extends FSParam[HealthNsfwModel.ModelNameType](\n        name = \"bqml_health_model_id\",\n        default = HealthNsfwModel.Q2_2022_Mr_Bqml_Health_Model_NsfwV0\n      )\n  object BqmlHealthModelPredicateFilterThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"bqml_health_model_filter_threshold_param\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n  object BqmlHealthModelPredicateBucketThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"bqml_health_model_bucket_threshold_param\",\n        default = 0.005,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object EnableBqmlHealthModelScoreHistogramParam\n      extends FSParam[Boolean](\n        name = \"bqml_health_model_score_histogram_enable_param\",\n        default = false\n      )\n\n  /*\n   * BQML Quality Model related params\n   */\n  object EnableBqmlQualityModelPredicateParam\n      extends FSParam[Boolean](\n        name = \"bqml_quality_model_filter_enable_param\",\n        default = false\n      )\n  object EnableBqmlQualityModelScoreHistogramParam\n      extends FSParam[Boolean](\n        name = \"bqml_quality_model_score_histogram_enable_param\",\n        default = false\n      )\n  object BqmlQualityModelTypeParam\n      extends FSParam[WeightedOpenOrNtabClickModel.ModelNameType](\n        name = \"bqml_quality_model_id\",\n        default = \"Q1_2022_13562_Mr_Bqml_Quality_Model_V2\"\n      )\n\n  /**\n   * Param to specify which quality models to use to get the scores for determining\n   * whether to bucket a user for the DDG\n   */\n  object BqmlQualityModelBucketModelIdListParam\n      extends FSParam[Seq[WeightedOpenOrNtabClickModel.ModelNameType]](\n        name = \"bqml_quality_model_bucket_model_id_list\",\n        default = Seq(\n          \"Q1_2022_13562_Mr_Bqml_Quality_Model_V2\",\n          \"Q2_2022_DDG14146_Mr_Personalised_BQML_Quality_Model\",\n          \"Q2_2022_DDG14146_Mr_NonPersonalised_BQML_Quality_Model\"\n        )\n      )\n\n  object BqmlQualityModelPredicateThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"bqml_quality_model_filter_threshold_param\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Param to specify the threshold to determine if a user’s quality score is high enough to enter the experiment.\n   */\n  object BqmlQualityModelBucketThresholdListParam\n      extends FSParam[Seq[Double]](\n        name = \"bqml_quality_model_bucket_threshold_list\",\n        default = Seq(0.7, 0.7, 0.7)\n      )\n\n  /*\n   * TweetAuthorAggregates related params\n   */\n  object EnableTweetAuthorAggregatesFeatureHydrationParam\n      extends FSParam[Boolean](\n        name = \"tweet_author_aggregates_feature_hydration_enable_param\",\n        default = false\n      )\n\n  /**\n   * Param to determine if we should include the relevancy score of candidates in the Ibis payload\n   */\n  object IncludeRelevanceScoreInIbis2Payload\n      extends FSParam[Boolean](\n        name = \"relevance_score_include_in_ibis2_payload\",\n        default = false\n      )\n\n  /**\n   *  Param to specify supervised model to predict score by sending the notification\n   */\n  object BigFilteringSupervisedSendingModelParam\n      extends FSParam[BigFilteringSupervisedModel.ModelNameType](\n        name = \"ltv_filtering_bigfiltering_supervised_sending_model_param\",\n        default = BigFilteringSupervisedModel.V0_0_BigFiltering_Supervised_Sending_Model\n      )\n\n  /**\n   *  Param to specify supervised model to predict score by not sending the notification\n   */\n  object BigFilteringSupervisedWithoutSendingModelParam\n      extends FSParam[BigFilteringSupervisedModel.ModelNameType](\n        name = \"ltv_filtering_bigfiltering_supervised_without_sending_model_param\",\n        default = BigFilteringSupervisedModel.V0_0_BigFiltering_Supervised_Without_Sending_Model\n      )\n\n  /**\n   *  Param to specify RL model to predict score by sending the notification\n   */\n  object BigFilteringRLSendingModelParam\n      extends FSParam[BigFilteringSupervisedModel.ModelNameType](\n        name = \"ltv_filtering_bigfiltering_rl_sending_model_param\",\n        default = BigFilteringRLModel.V0_0_BigFiltering_Rl_Sending_Model\n      )\n\n  /**\n   *  Param to specify RL model to predict score by not sending the notification\n   */\n  object BigFilteringRLWithoutSendingModelParam\n      extends FSParam[BigFilteringSupervisedModel.ModelNameType](\n        name = \"ltv_filtering_bigfiltering_rl_without_sending_model_param\",\n        default = BigFilteringRLModel.V0_0_BigFiltering_Rl_Without_Sending_Model\n      )\n\n  /**\n   *  Param to specify the threshold (send notification if score >= threshold)\n   */\n  object BigFilteringThresholdParam\n      extends FSBoundedParam[Double](\n        name = \"ltv_filtering_bigfiltering_threshold_param\",\n        default = 0.0,\n        min = Double.MinValue,\n        max = Double.MaxValue\n      )\n\n  /**\n   *  Param to specify normalization used for BigFiltering\n   */\n  object BigFilteringNormalizationTypeIdParam\n      extends FSEnumParam[BigFilteringNormalizationEnum.type](\n        name = \"ltv_filtering_bigfiltering_normalization_type_id\",\n        default = BigFilteringNormalizationEnum.NormalizationDisabled,\n        enum = BigFilteringNormalizationEnum\n      )\n\n  /**\n   *  Param to specify histograms of model scores in BigFiltering\n   */\n  object BigFilteringEnableHistogramsParam\n      extends FSParam[Boolean](\n        name = \"ltv_filtering_bigfiltering_enable_histograms_param\",\n        default = false\n      )\n\n  /*\n   * Param to enable sending requests to Ins Sender\n   */\n  object EnableInsSender extends FSParam[Boolean](name = \"ins_enable_dark_traffic\", default = false)\n\n  /**\n   * Param to specify the range of relevance scores for MagicFanout types.\n   */\n  object MagicFanoutRelevanceScoreRange\n      extends FSParam[Seq[Double]](\n        name = \"relevance_score_mf_range\",\n        default = Seq(0.75, 1.0)\n      )\n\n  /**\n   * Param to specify the range of relevance scores for MR types.\n   */\n  object MagicRecsRelevanceScoreRange\n      extends FSParam[Seq[Double]](\n        name = \"relevance_score_mr_range\",\n        default = Seq(0.25, 0.5)\n      )\n\n  /**\n   * Param to enable backfilling OON candidates if number of F1 candidates is greater than a threshold K.\n   */\n  object EnableOONBackfillBasedOnF1Candidates\n      extends FSParam[Boolean](name = \"oon_enable_backfill_based_on_f1\", default = false)\n\n  /**\n   * Threshold for the minimum number of F1 candidates required to enable backfill of OON candidates.\n   */\n  object NumberOfF1CandidatesThresholdForOONBackfill\n      extends FSBoundedParam[Int](\n        name = \"oon_enable_backfill_f1_threshold\",\n        min = 0,\n        default = 5000,\n        max = 5000)\n\n  /**\n   * Event ID allowlist to skip account country predicate\n   */\n  object MagicFanoutEventAllowlistToSkipAccountCountryPredicate\n      extends FSParam[Seq[Long]](\n        name = \"magicfanout_event_allowlist_skip_account_country_predicate\",\n        default = Seq.empty[Long]\n      )\n\n  /**\n   * MagicFanout Event Semantic Core Domain Ids\n   */\n  object ListOfEventSemanticCoreDomainIds\n      extends FSParam[Seq[Long]](\n        name = \"magicfanout_automated_events_semantic_core_domain_ids\",\n        default = Seq())\n\n  /**\n   * Adhoc id for detailed rank flow stats\n   */\n  object ListOfAdhocIdsForStatsTracking\n      extends FSParam[Set[Long]](\n        name = \"stats_enable_detailed_stats_tracking_ids\",\n        default = Set.empty[Long]\n      )\n\n  object EnableGenericCRTBasedFatiguePredicate\n      extends FSParam[Boolean](\n        name = \"seelessoften_enable_generic_crt_based_fatigue_predicate\",\n        default = false)\n\n  /**\n   * Param to enable copy features such as Emojis and Target Name\n   */\n  object EnableCopyFeaturesForF1\n      extends FSParam[Boolean](name = \"mr_copy_enable_features_f1\", default = false)\n\n  /**\n   * Param to enable copy features such as Emojis and Target Name\n   */\n  object EnableCopyFeaturesForOon\n      extends FSParam[Boolean](name = \"mr_copy_enable_features_oon\", default = false)\n\n  /**\n   * Param to enable Emoji in F1 Copy\n   */\n  object EnableEmojiInF1Copy\n      extends FSParam[Boolean](name = \"mr_copy_enable_f1_emoji\", default = false)\n\n  /**\n   * Param to enable Target in F1 Copy\n   */\n  object EnableTargetInF1Copy\n      extends FSParam[Boolean](name = \"mr_copy_enable_f1_target\", default = false)\n\n  /**\n   * Param to enable Emoji in OON Copy\n   */\n  object EnableEmojiInOonCopy\n      extends FSParam[Boolean](name = \"mr_copy_enable_oon_emoji\", default = false)\n\n  /**\n   * Param to enable Target in OON Copy\n   */\n  object EnableTargetInOonCopy\n      extends FSParam[Boolean](name = \"mr_copy_enable_oon_target\", default = false)\n\n  /**\n   * Param to enable split fatigue for Target and Emoji copy for OON and F1\n   */\n  object EnableTargetAndEmojiSplitFatigue\n      extends FSParam[Boolean](name = \"mr_copy_enable_target_emoji_split_fatigue\", default = false)\n\n  /**\n   * Param to enable experimenting string on the body\n   */\n  object EnableF1CopyBody extends FSParam[Boolean](name = \"mr_copy_f1_enable_body\", default = false)\n\n  object EnableOONCopyBody\n      extends FSParam[Boolean](name = \"mr_copy_oon_enable_body\", default = false)\n\n  object EnableIosCopyBodyTruncate\n      extends FSParam[Boolean](name = \"mr_copy_enable_body_truncate\", default = false)\n\n  object EnableNsfwCopy extends FSParam[Boolean](name = \"mr_copy_enable_nsfw\", default = false)\n\n  /**\n   * Param to determine F1 candidate nsfw score threshold\n   */\n  object NsfwScoreThresholdForF1Copy\n      extends FSBoundedParam[Double](\n        name = \"mr_copy_nsfw_threshold_f1\",\n        default = 0.3,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Param to determine OON candidate nsfw score threshold\n   */\n  object NsfwScoreThresholdForOONCopy\n      extends FSBoundedParam[Double](\n        name = \"mr_copy_nsfw_threshold_oon\",\n        default = 0.2,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Param to determine the lookback duration when searching for prev copy features.\n   */\n  object CopyFeaturesHistoryLookbackDuration\n      extends FSBoundedParam[Duration](\n        name = \"mr_copy_history_lookback_duration_in_days\",\n        default = 30.days,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Param to determine the F1 emoji copy fatigue in # of hours.\n   */\n  object F1EmojiCopyFatigueDuration\n      extends FSBoundedParam[Duration](\n        name = \"mr_copy_f1_emoji_copy_fatigue_in_hours\",\n        default = 24.hours,\n        min = 0.hours,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to determine the F1 target copy fatigue in # of hours.\n   */\n  object F1TargetCopyFatigueDuration\n      extends FSBoundedParam[Duration](\n        name = \"mr_copy_f1_target_copy_fatigue_in_hours\",\n        default = 24.hours,\n        min = 0.hours,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to determine the OON emoji copy fatigue in # of hours.\n   */\n  object OonEmojiCopyFatigueDuration\n      extends FSBoundedParam[Duration](\n        name = \"mr_copy_oon_emoji_copy_fatigue_in_hours\",\n        default = 24.hours,\n        min = 0.hours,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to determine the OON target copy fatigue in # of hours.\n   */\n  object OonTargetCopyFatigueDuration\n      extends FSBoundedParam[Duration](\n        name = \"mr_copy_oon_target_copy_fatigue_in_hours\",\n        default = 24.hours,\n        min = 0.hours,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to turn on/off home timeline based fatigue rule, where once last home timeline visit\n   * is larger than the specified will evalute to not fatigue\n   */\n  object EnableHTLBasedFatigueBasicRule\n      extends FSParam[Boolean](\n        name = \"mr_copy_enable_htl_based_fatigue_basic_rule\",\n        default = false)\n\n  /**\n   * Param to determine f1 emoji copy fatigue in # of pushes\n   */\n  object F1EmojiCopyNumOfPushesFatigue\n      extends FSBoundedParam[Int](\n        name = \"mr_copy_f1_emoji_copy_number_of_pushes_fatigue\",\n        default = 0,\n        min = 0,\n        max = 200\n      )\n\n  /**\n   * Param to determine oon emoji copy fatigue in # of pushes\n   */\n  object OonEmojiCopyNumOfPushesFatigue\n      extends FSBoundedParam[Int](\n        name = \"mr_copy_oon_emoji_copy_number_of_pushes_fatigue\",\n        default = 0,\n        min = 0,\n        max = 200\n      )\n\n  /**\n   * If user haven't visited home timeline for certain duration, we will\n   * exempt user from feature copy fatigue. This param is used to control\n   * how long it is before we enter exemption.\n   */\n  object MinFatigueDurationSinceLastHTLVisit\n      extends FSBoundedParam[Duration](\n        name = \"mr_copy_min_duration_since_last_htl_visit_hours\",\n        default = Duration.Top,\n        min = 0.hour,\n        max = Duration.Top,\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * If a user haven't visit home timeline very long, the user will return\n   * to fatigue state under the home timeline based fatigue rule. There will\n   * only be a window, where the user is out of fatigue state under the rule.\n   * This param control the length of the non fatigue period.\n   */\n  object LastHTLVisitBasedNonFatigueWindow\n      extends FSBoundedParam[Duration](\n        name = \"mr_copy_last_htl_visit_based_non_fatigue_window_hours\",\n        default = 48.hours,\n        min = 0.hour,\n        max = Duration.Top,\n      )\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  object EnableOONCBasedCopy\n      extends FSParam[Boolean](\n        name = \"mr_copy_enable_oonc_based_copy\",\n        default = false\n      )\n\n  object HighOONCThresholdForCopy\n      extends FSBoundedParam[Double](\n        name = \"mr_copy_high_oonc_threshold_for_copy\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object LowOONCThresholdForCopy\n      extends FSBoundedParam[Double](\n        name = \"mr_copy_low_oonc_threshold_for_copy\",\n        default = 0.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  object EnableTweetTranslation\n      extends FSParam[Boolean](name = \"tweet_translation_enable\", default = false)\n\n  object TripTweetCandidateReturnEnable\n      extends FSParam[Boolean](name = \"trip_tweet_candidate_enable\", default = false)\n\n  object TripTweetCandidateSourceIds\n      extends FSParam[Seq[String]](\n        name = \"trip_tweet_candidate_source_ids\",\n        default = Seq(\"TOP_GEO_V3\"))\n\n  object TripTweetMaxTotalCandidates\n      extends FSBoundedParam[Int](\n        name = \"trip_tweet_max_total_candidates\",\n        default = 500,\n        min = 10,\n        max = 1000)\n\n  object EnableEmptyBody\n      extends FSParam[Boolean](name = \"push_presentation_enable_empty_body\", default = false)\n\n  object EnableSocialContextForRetweet\n      extends FSParam[Boolean](name = \"push_presentation_social_context_retweet\", default = false)\n\n  /**\n   * Param to enable/disable simcluster feature hydration\n   */\n  object EnableMrTweetSimClusterFeatureHydrationFS\n      extends FSParam[Boolean](\n        name = \"feature_hydration_enable_mr_tweet_simcluster_feature\",\n        default = false\n      )\n\n  /**\n   * Param to disable OON candidates based on tweetAuthor\n   */\n  object DisableOutNetworkTweetCandidatesFS\n      extends FSParam[Boolean](name = \"oon_filtering_disable_oon_candidates\", default = false)\n\n  /**\n   * Param to enable Local Viral Tweets\n   */\n  object EnableLocalViralTweets\n      extends FSParam[Boolean](name = \"local_viral_tweets_enable\", default = true)\n\n  /**\n   * Param to enable Explore Video Tweets\n   */\n  object EnableExploreVideoTweets\n      extends FSParam[Boolean](name = \"explore_video_tweets_enable\", default = false)\n\n  /**\n   * Param to enable List Recommendations\n   */\n  object EnableListRecommendations\n      extends FSParam[Boolean](name = \"list_recommendations_enable\", default = false)\n\n  /**\n   * Param to enable IDS List Recommendations\n   */\n  object EnableIDSListRecommendations\n      extends FSParam[Boolean](name = \"list_recommendations_ids_enable\", default = false)\n\n  /**\n   * Param to enable PopGeo List Recommendations\n   */\n  object EnablePopGeoListRecommendations\n      extends FSParam[Boolean](name = \"list_recommendations_pop_geo_enable\", default = false)\n\n  /**\n   * Param to control the inverter for fatigue between consecutive ListRecommendations\n   */\n  object ListRecommendationsPushInterval\n      extends FSBoundedParam[Duration](\n        name = \"list_recommendations_interval_days\",\n        default = 24.hours,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromDays\n  }\n\n  /**\n   * Param to control the granularity of GeoHash for ListRecommendations\n   */\n  object ListRecommendationsGeoHashLength\n      extends FSBoundedParam[Int](\n        name = \"list_recommendations_geo_hash_length\",\n        default = 5,\n        min = 3,\n        max = 5)\n\n  /**\n   * Param to control maximum number of ListRecommendation pushes to receive in an interval\n   */\n  object MaxListRecommendationsPushGivenInterval\n      extends FSBoundedParam[Int](\n        name = \"list_recommendations_push_given_interval\",\n        default = 1,\n        min = 0,\n        max = 10\n      )\n\n  /**\n   * Param to control the subscriber count for list recommendation\n   */\n  object ListRecommendationsSubscriberCount\n      extends FSBoundedParam[Int](\n        name = \"list_recommendations_subscriber_count\",\n        default = 0,\n        min = 0,\n        max = Integer.MAX_VALUE)\n\n  /**\n   * Param to define dynamic inline action types for web notifications (both desktop web + mobile web)\n   */\n  object LocalViralTweetsBucket\n      extends FSParam[String](\n        name = \"local_viral_tweets_bucket\",\n        default = \"high\",\n      )\n\n  /**\n   * List of CrTags to disable\n   */\n  object OONCandidatesDisabledCrTagParam\n      extends FSParam[Seq[String]](\n        name = \"oon_enable_oon_candidates_disabled_crtag\",\n        default = Seq.empty[String]\n      )\n\n  /**\n   * List of Crt groups to disable\n   */\n  object OONCandidatesDisabledCrtGroupParam\n      extends FSEnumSeqParam[CrtGroupEnum.type](\n        name = \"oon_enable_oon_candidates_disabled_crt_group_ids\",\n        default = Seq.empty[CrtGroupEnum.Value],\n        enum = CrtGroupEnum\n      )\n\n  /**\n   * Param to enable launching video tweets in the Immersive Explore timeline\n   */\n  object EnableLaunchVideosInImmersiveExplore\n      extends FSParam[Boolean](name = \"launch_videos_in_immersive_explore\", default = false)\n\n  /**\n   * Param to enable Ntab Entries for Sports Event Notifications\n   */\n  object EnableNTabEntriesForSportsEventNotifications\n      extends FSParam[Boolean](\n        name = \"magicfanout_sports_event_enable_ntab_entries\",\n        default = false)\n\n  /**\n   * Param to enable Ntab Facepiles for teams in Sport Notifs\n   */\n  object EnableNTabFacePileForSportsEventNotifications\n      extends FSParam[Boolean](\n        name = \"magicfanout_sports_event_enable_ntab_facepiles\",\n        default = false)\n\n  /**\n   * Param to enable Ntab Override for Sports Event Notifications\n   */\n  object EnableNTabOverrideForSportsEventNotifications\n      extends FSParam[Boolean](\n        name = \"magicfanout_sports_event_enable_ntab_override\",\n        default = false)\n\n  /**\n   * Param to control the interval for MF Product Launch Notifs\n   */\n  object ProductLaunchPushIntervalInHours\n      extends FSBoundedParam[Duration](\n        name = \"product_launch_fatigue_push_interval_in_hours\",\n        default = 24.hours,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the maximum number of MF Product Launch Notifs in a period of time\n   */\n  object ProductLaunchMaxNumberOfPushesInInterval\n      extends FSBoundedParam[Int](\n        name = \"product_launch_fatigue_max_pushes_in_interval\",\n        default = 1,\n        min = 0,\n        max = 10)\n\n  /**\n   * Param to control the minInterval for fatigue between consecutive MF Product Launch Notifs\n   */\n  object ProductLaunchMinIntervalFatigue\n      extends FSBoundedParam[Duration](\n        name = \"product_launch_fatigue_min_interval_consecutive_pushes_in_hours\",\n        default = 24.hours,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the interval for MF New Creator Notifs\n   */\n  object NewCreatorPushIntervalInHours\n      extends FSBoundedParam[Duration](\n        name = \"new_creator_fatigue_push_interval_in_hours\",\n        default = 24.hours,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the maximum number of MF New Creator Notifs in a period of time\n   */\n  object NewCreatorPushMaxNumberOfPushesInInterval\n      extends FSBoundedParam[Int](\n        name = \"new_creator_fatigue_max_pushes_in_interval\",\n        default = 1,\n        min = 0,\n        max = 10)\n\n  /**\n   * Param to control the minInterval for fatigue between consecutive MF New Creator Notifs\n   */\n  object NewCreatorPushMinIntervalFatigue\n      extends FSBoundedParam[Duration](\n        name = \"new_creator_fatigue_min_interval_consecutive_pushes_in_hours\",\n        default = 24.hours,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the interval for MF New Creator Notifs\n   */\n  object CreatorSubscriptionPushIntervalInHours\n      extends FSBoundedParam[Duration](\n        name = \"creator_subscription_fatigue_push_interval_in_hours\",\n        default = 24.hours,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to control the maximum number of MF New Creator Notifs in a period of time\n   */\n  object CreatorSubscriptionPushMaxNumberOfPushesInInterval\n      extends FSBoundedParam[Int](\n        name = \"creator_subscription_fatigue_max_pushes_in_interval\",\n        default = 1,\n        min = 0,\n        max = 10)\n\n  /**\n   * Param to control the minInterval for fatigue between consecutive MF New Creator Notifs\n   */\n  object CreatorSubscriptionPushhMinIntervalFatigue\n      extends FSBoundedParam[Duration](\n        name = \"creator_subscription_fatigue_min_interval_consecutive_pushes_in_hours\",\n        default = 24.hours,\n        min = Duration.Bottom,\n        max = Duration.Top)\n      with HasDurationConversion {\n    override val durationConversion = DurationConversion.FromHours\n  }\n\n  /**\n   * Param to define the landing page deeplink of product launch notifications\n   */\n  object ProductLaunchLandingPageDeepLink\n      extends FSParam[String](\n        name = \"product_launch_landing_page_deeplink\",\n        default = \"\"\n      )\n\n  /**\n   * Param to define the tap through of product launch notifications\n   */\n  object ProductLaunchTapThrough\n      extends FSParam[String](\n        name = \"product_launch_tap_through\",\n        default = \"\"\n      )\n\n  /**\n   * Param to skip checking isTargetBlueVerified\n   */\n  object DisableIsTargetBlueVerifiedPredicate\n      extends FSParam[Boolean](\n        name = \"product_launch_disable_is_target_blue_verified_predicate\",\n        default = false\n      )\n\n  /**\n   * Param to enable Ntab Entries for Sports Event Notifications\n   */\n  object EnableNTabEntriesForProductLaunchNotifications\n      extends FSParam[Boolean](name = \"product_launch_enable_ntab_entry\", default = true)\n\n  /**\n   * Param to skip checking isTargetLegacyVerified\n   */\n  object DisableIsTargetLegacyVerifiedPredicate\n      extends FSParam[Boolean](\n        name = \"product_launch_disable_is_target_legacy_verified_predicate\",\n        default = false\n      )\n\n  /**\n   * Param to enable checking isTargetSuperFollowCreator\n   */\n  object EnableIsTargetSuperFollowCreatorPredicate\n      extends FSParam[Boolean](\n        name = \"product_launch_is_target_super_follow_creator_predicate_enabled\",\n        default = false\n      )\n\n  /**\n   * Param to enable Spammy Tweet filter\n   */\n  object EnableSpammyTweetFilter\n      extends FSParam[Boolean](\n        name = \"health_signal_store_enable_spammy_tweet_filter\",\n        default = false)\n\n  /**\n   * Param to enable Push to Home Android\n   */\n  object EnableTweetPushToHomeAndroid\n      extends FSParam[Boolean](name = \"push_to_home_tweet_recs_android\", default = false)\n\n  /**\n   * Param to enable Push to Home iOS\n   */\n  object EnableTweetPushToHomeiOS\n      extends FSParam[Boolean](name = \"push_to_home_tweet_recs_iOS\", default = false)\n\n  /**\n   * Param to set Spammy Tweet score threshold for OON candidates\n   */\n  object SpammyTweetOonThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_spammy_tweet_oon_threshold\",\n        default = 1.1,\n        min = 0.0,\n        max = 1.1\n      )\n\n  object NumFollowerThresholdForHealthAndQualityFilters\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_num_follower_threshold_for_health_and_quality_filters\",\n        default = 10000000000.0,\n        min = 0.0,\n        max = 10000000000.0\n      )\n\n  object NumFollowerThresholdForHealthAndQualityFiltersPreranking\n      extends FSBoundedParam[Double](\n        name =\n          \"health_signal_store_num_follower_threshold_for_health_and_quality_filters_preranking\",\n        default = 10000000.0,\n        min = 0.0,\n        max = 10000000000.0\n      )\n\n  /**\n   * Param to set Spammy Tweet score threshold for IN candidates\n   */\n  object SpammyTweetInThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_spammy_tweet_in_threshold\",\n        default = 1.1,\n        min = 0.0,\n        max = 1.1\n      )\n\n  /**\n   * Param to control bucketing for the Spammy Tweet score\n   */\n  object SpammyTweetBucketingThreshold\n      extends FSBoundedParam[Double](\n        name = \"health_signal_store_spammy_tweet_bucketing_threshold\",\n        default = 1.0,\n        min = 0.0,\n        max = 1.0\n      )\n\n  /**\n   * Param to specify the maximum number of Explore Video Tweets to request\n   */\n  object MaxExploreVideoTweets\n      extends FSBoundedParam[Int](\n        name = \"explore_video_tweets_max_candidates\",\n        default = 100,\n        min = 0,\n        max = 500\n      )\n\n  /**\n   * Param to enable social context feature set\n   */\n  object EnableBoundedFeatureSetForSocialContext\n      extends FSParam[Boolean](\n        name = \"feature_hydration_user_social_context_bounded_feature_set_enable\",\n        default = true)\n\n  /**\n   * Param to enable stp user social context feature set\n   */\n  object EnableStpBoundedFeatureSetForUserSocialContext\n      extends FSParam[Boolean](\n        name = \"feature_hydration_stp_social_context_bounded_feature_set_enable\",\n        default = true)\n\n  /**\n   * Param to enable core user history social context feature set\n   */\n  object EnableCoreUserHistoryBoundedFeatureSetForSocialContext\n      extends FSParam[Boolean](\n        name = \"feature_hydration_core_user_history_social_context_bounded_feature_set_enable\",\n        default = true)\n\n  /**\n   * Param to enable skipping post-ranking filters\n   */\n  object SkipPostRankingFilters\n      extends FSParam[Boolean](\n        name = \"frigate_push_modeling_skip_post_ranking_filters\",\n        default = false)\n\n  object MagicFanoutSimClusterDotProductNonHeavyUserThreshold\n      extends FSBoundedParam[Double](\n        name = \"frigate_push_magicfanout_simcluster_non_heavy_user_dot_product_threshold\",\n        default = 0.0,\n        min = 0.0,\n        max = 100.0\n      )\n\n  object MagicFanoutSimClusterDotProductHeavyUserThreshold\n      extends FSBoundedParam[Double](\n        name = \"frigate_push_magicfanout_simcluster_heavy_user_dot_product_threshold\",\n        default = 10.0,\n        min = 0.0,\n        max = 100.0\n      )\n\n  object EnableReducedFatigueRulesForSeeLessOften\n      extends FSParam[Boolean](\n        name = \"seelessoften_enable_reduced_fatigue\",\n        default = false\n      )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/params/PushFeatureSwitches.scala",
    "content": "package com.twitter.frigate.pushservice.params\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.{FeatureSwitchParams => Common}\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => Pushservice}\nimport com.twitter.logging.Logger\nimport com.twitter.servo.decider.DeciderGateBuilder\nimport com.twitter.timelines.configapi.BaseConfigBuilder\nimport com.twitter.timelines.configapi.FeatureSwitchOverrideUtil\nimport com.twitter.timelines.configapi.decider.DeciderUtils\n\ncase class PushFeatureSwitches(\n  deciderGateBuilder: DeciderGateBuilder,\n  statsReceiver: StatsReceiver) {\n\n  private[this] val logger = Logger(classOf[PushFeatureSwitches])\n  private[this] val stat = statsReceiver.scope(\"PushFeatureSwitches\")\n\n  private val booleanDeciderOverrides = DeciderUtils.getBooleanDeciderOverrides(\n    deciderGateBuilder,\n    PushParams.DisableAllRelevanceParam,\n    PushParams.DisableHeavyRankingParam,\n    PushParams.RestrictLightRankingParam,\n    PushParams.UTEGTweetCandidateSourceParam,\n    PushParams.EnableWritesToNotificationServiceParam,\n    PushParams.EnableWritesToNotificationServiceForAllEmployeesParam,\n    PushParams.EnableWritesToNotificationServiceForEveryoneParam,\n    PushParams.EnablePromptFeedbackFatigueResponseNoPredicate,\n    PushParams.EarlyBirdSCBasedCandidatesParam,\n    PushParams.MRTweetFavRecsParam,\n    PushParams.MRTweetRetweetRecsParam,\n    PushParams.EnablePushSendEventBus,\n    PushParams.DisableMlInFilteringParam,\n    PushParams.DownSampleLightRankingScribeCandidatesParam,\n    PushParams.EnableMrRequestScribing,\n    PushParams.EnableHighQualityCandidateScoresScribing,\n    PushParams.EnablePnegMultimodalPredictionForF1Tweets,\n    PushParams.EnableScribeOonFavScoreForF1Tweets,\n    PushParams.EnableMrUserSemanticCoreFeaturesHydration,\n    PushParams.EnableMrUserSemanticCoreNoZeroFeaturesHydration,\n    PushParams.EnableHtlOfflineUserAggregatesExtendedHydration,\n    PushParams.EnableNerErgFeatureHydration,\n    PushParams.EnableDaysSinceRecentResurrectionFeatureHydration,\n    PushParams.EnableUserPastAggregatesFeatureHydration,\n    PushParams.EnableMrUserSimclusterV2020FeaturesHydration,\n    PushParams.EnableMrUserSimclusterV2020NoZeroFeaturesHydration,\n    PushParams.EnableTopicEngagementRealTimeAggregatesFeatureHydration,\n    PushParams.EnableUserTopicAggregatesFeatureHydration,\n    PushParams.EnableHtlUserAuthorRTAFeaturesFromFeatureStoreHydration,\n    PushParams.EnableDurationSinceLastVisitFeatures,\n    PushParams.EnableTweetAnnotationFeaturesHydration,\n    PushParams.EnableSpaceVisibilityLibraryFiltering,\n    PushParams.EnableUserTopicFollowFeatureSetHydration,\n    PushParams.EnableOnboardingNewUserFeatureSetHydration,\n    PushParams.EnableMrUserAuthorSparseContFeatureSetHydration,\n    PushParams.EnableMrUserTopicSparseContFeatureSetHydration,\n    PushParams.EnableUserPenguinLanguageFeatureSetHydration,\n    PushParams.EnableMrUserHashspaceEmbeddingFeatureHydration,\n    PushParams.EnableMrUserEngagedTweetTokensFeatureHydration,\n    PushParams.EnableMrCandidateTweetTokensFeatureHydration,\n    PushParams.EnableMrTweetSentimentFeatureHydration,\n    PushParams.EnableMrTweetAuthorAggregatesFeatureHydration,\n    PushParams.EnableUserGeoFeatureSetHydration,\n    PushParams.EnableAuthorGeoFeatureSetHydration,\n    PushParams.EnableTwHINUserEngagementFeaturesHydration,\n    PushParams.EnableTwHINUserFollowFeaturesHydration,\n    PushParams.EnableTwHINAuthorFollowFeaturesHydration,\n    PushParams.EnableAuthorFollowTwhinEmbeddingFeatureHydration,\n    PushParams.RampupUserGeoFeatureSetHydration,\n    PushParams.RampupAuthorGeoFeatureSetHydration,\n    PushParams.EnablePredicateDetailedInfoScribing,\n    PushParams.EnablePushCapInfoScribing,\n    PushParams.EnableUserSignalLanguageFeatureHydration,\n    PushParams.EnableUserPreferredLanguageFeatureHydration,\n    PushParams.PopGeoCandidatesDecider,\n    PushParams.TrendsCandidateDecider,\n    PushParams.EnableInsTrafficDecider,\n    PushParams.EnableModelBasedPushcapAssignments,\n    PushParams.TripGeoTweetCandidatesDecider,\n    PushParams.ContentRecommenderMixerAdaptorDecider,\n    PushParams.GenericCandidateAdaptorDecider,\n    PushParams.TripGeoTweetContentMixerDarkTrafficDecider,\n    PushParams.EnableIsTweetTranslatableCheck,\n    PushParams.EnableMrTweetSimClusterFeatureHydration,\n    PushParams.EnableTwistlyAggregatesFeatureHydration,\n    PushParams.EnableTweetTwHINFavFeatureHydration,\n    PushParams.EnableRealGraphV2FeatureHydration,\n    PushParams.EnableTweetBeTFeatureHydration,\n    PushParams.EnableMrOfflineUserTweetTopicAggregateHydration,\n    PushParams.EnableMrOfflineUserTweetSimClusterAggregateHydration,\n    PushParams.EnableUserSendTimeFeatureHydration,\n    PushParams.EnableMrUserUtcSendTimeAggregateFeaturesHydration,\n    PushParams.EnableMrUserLocalSendTimeAggregateFeaturesHydration,\n    PushParams.EnableBqmlReportModelPredictionForF1Tweets,\n    PushParams.EnableUserTwhinEmbeddingFeatureHydration,\n    PushParams.EnableScribingMLFeaturesAsDataRecord,\n    PushParams.EnableAuthorVerifiedFeatureHydration,\n    PushParams.EnableAuthorCreatorSubscriptionFeatureHydration,\n    PushParams.EnableDirectHydrationForUserFeatures\n  )\n\n  private val intFeatureSwitchOverrides = FeatureSwitchOverrideUtil.getBoundedIntFSOverrides(\n    Pushservice.SportsMaxNumberOfPushesInIntervalPerEvent,\n    Pushservice.SportsMaxNumberOfPushesInInterval,\n    Pushservice.PushMixerMaxResults,\n    Pushservice.MaxTrendTweetNotificationsInDuration,\n    Pushservice.MaxRecommendedTrendsToQuery,\n    Pushservice.NumberOfMaxEarlybirdInNetworkCandidatesParam,\n    Pushservice.NumberOfMaxCandidatesToBatchInRFPHTakeStep,\n    Pushservice.MaxMrPushSends24HoursParam,\n    Pushservice.MaxMrPushSends24HoursNtabOnlyUsersParam,\n    Pushservice.NumberOfMaxCrMixerCandidatesParam,\n    Pushservice.RestrictStepSize,\n    Pushservice.MagicFanoutRankErgThresholdHeavy,\n    Pushservice.MagicFanoutRankErgThresholdNonHeavy,\n    Pushservice.MagicFanoutRelaxedEventIdFatigueIntervalInHours,\n    Pushservice.NumberOfMaxUTEGCandidatesQueriedParam,\n    Pushservice.HTLVisitFatigueTime,\n    Pushservice.MaxOnboardingPushInInterval,\n    Pushservice.MaxTopTweetsByGeoPushGivenInterval,\n    Pushservice.MaxHighQualityTweetsPushGivenInterval,\n    Pushservice.MaxTopTweetsByGeoCandidatesToTake,\n    Pushservice.SpaceRecsRealgraphThreshold,\n    Pushservice.SpaceRecsGlobalPushLimit,\n    Pushservice.OptoutExptPushCapParam,\n    Pushservice.MaxTopTweetImpressionsNotifications,\n    Pushservice.TopTweetImpressionsMinRequired,\n    Pushservice.TopTweetImpressionsThreshold,\n    Pushservice.TopTweetImpressionsOriginalTweetsNumDaysSearch,\n    Pushservice.TopTweetImpressionsMinNumOriginalTweets,\n    Pushservice.TopTweetImpressionsMaxFavoritesPerTweet,\n    Pushservice.TopTweetImpressionsTotalInboundFavoritesLimit,\n    Pushservice.TopTweetImpressionsTotalFavoritesLimitNumDaysSearch,\n    Pushservice.TopTweetImpressionsRecentTweetsByAuthorStoreMaxResults,\n    Pushservice.ANNEfQuery,\n    Pushservice.NumberOfMaxMrModelingBasedCandidates,\n    Pushservice.ThresholdOfFavMrModelingBasedCandidates,\n    Pushservice.LightRankingNumberOfCandidatesParam,\n    Pushservice.NumberOfDeTopicTweetCandidates,\n    Pushservice.NumberOfMaxDeTopicTweetCandidatesReturned,\n    Pushservice.OverrideNotificationsMaxNumOfSlots,\n    Pushservice.OverrideNotificationsMaxCountForNTab,\n    Pushservice.MFMaxNumberOfPushesInInterval,\n    Pushservice.SpacesTopKSimClusterCount,\n    Pushservice.SpaceRecsSimClusterUserMinimumFollowerCount,\n    Pushservice.OONSpaceRecsPushLimit,\n    Pushservice.MagicFanoutRealgraphRankThreshold,\n    Pushservice.CustomizedPushCapOffset,\n    Pushservice.NumberOfF1CandidatesThresholdForOONBackfill,\n    Pushservice.MinimumAllowedAuthorAccountAgeInHours,\n    Pushservice.RestrictedMinModelPushcap,\n    Pushservice.ListRecommendationsGeoHashLength,\n    Pushservice.ListRecommendationsSubscriberCount,\n    Pushservice.MaxListRecommendationsPushGivenInterval,\n    Pushservice.SendTimeByUserHistoryMaxOpenedThreshold,\n    Pushservice.SendTimeByUserHistoryNoSendsHours,\n    Pushservice.SendTimeByUserHistoryQuickSendBeforeHours,\n    Pushservice.SendTimeByUserHistoryQuickSendAfterHours,\n    Pushservice.SendTimeByUserHistoryQuickSendMinDurationInMinute,\n    Pushservice.SendTimeByUserHistoryNoSendMinDuration,\n    Pushservice.F1EmojiCopyNumOfPushesFatigue,\n    Pushservice.OonEmojiCopyNumOfPushesFatigue,\n    Pushservice.TripTweetMaxTotalCandidates,\n    Pushservice.InlineFeedbackSubstitutePosition,\n    Pushservice.HighQualityCandidatesNumberOfCandidates,\n    Pushservice.HighQualityCandidatesMinNumOfCandidatesToFallback,\n    Pushservice.ProductLaunchMaxNumberOfPushesInInterval,\n    Pushservice.CreatorSubscriptionPushMaxNumberOfPushesInInterval,\n    Pushservice.NewCreatorPushMaxNumberOfPushesInInterval,\n    Pushservice.TweetReplytoLikeRatioReplyCountThreshold,\n    Pushservice.MaxExploreVideoTweets,\n  )\n\n  private val doubleFeatureSwitchOverrides =\n    FeatureSwitchOverrideUtil.getBoundedDoubleFSOverrides(\n      Pushservice.PercentileThresholdCohort1,\n      Pushservice.PercentileThresholdCohort2,\n      Pushservice.PercentileThresholdCohort3,\n      Pushservice.PercentileThresholdCohort4,\n      Pushservice.PercentileThresholdCohort5,\n      Pushservice.PercentileThresholdCohort6,\n      Pushservice.PnsfwTweetTextThreshold,\n      Pushservice.PnsfwTweetTextBucketingThreshold,\n      Pushservice.PnsfwTweetMediaThreshold,\n      Pushservice.PnsfwTweetImageThreshold,\n      Pushservice.PnsfwQuoteTweetThreshold,\n      Pushservice.PnsfwTweetMediaBucketingThreshold,\n      Pushservice.AgathaCalibratedNSFWThreshold,\n      Pushservice.AgathaCalibratedNSFWThresholdForMrTwistly,\n      Pushservice.AgathaTextNSFWThreshold,\n      Pushservice.AgathaTextNSFWThresholdForMrTwistly,\n      Pushservice.AgathaCalibratedNSFWBucketThreshold,\n      Pushservice.AgathaTextNSFWBucketThreshold,\n      Pushservice.BucketOptoutThresholdParam,\n      Pushservice.TweetMediaSensitiveCategoryThresholdParam,\n      Pushservice.CandidateGenerationModelCosineThreshold,\n      Pushservice.MrModelingBasedCandidatesTopicScoreThreshold,\n      Pushservice.HashspaceCandidatesTopicScoreThreshold,\n      Pushservice.FrsTweetCandidatesTopicScoreThreshold,\n      Pushservice.TopicProofTweetCandidatesTopicScoreThreshold,\n      Pushservice.SpacesTargetingSimClusterDotProductThreshold,\n      Pushservice.SautOonWithMediaTweetLengthThresholdParam,\n      Pushservice.NonSautOonWithMediaTweetLengthThresholdParam,\n      Pushservice.SautOonWithoutMediaTweetLengthThresholdParam,\n      Pushservice.NonSautOonWithoutMediaTweetLengthThresholdParam,\n      Pushservice.ArgfOonWithMediaTweetWordLengthThresholdParam,\n      Pushservice.EsfthOonWithMediaTweetWordLengthThresholdParam,\n      Pushservice.BqmlQualityModelPredicateThresholdParam,\n      Pushservice.LightRankingScribeCandidatesDownSamplingParam,\n      Pushservice.QualityUprankingBoostForHeavyRankingParam,\n      Pushservice.QualityUprankingSigmoidBiasForHeavyRankingParam,\n      Pushservice.QualityUprankingSigmoidWeightForHeavyRankingParam,\n      Pushservice.QualityUprankingLinearBarForHeavyRankingParam,\n      Pushservice.QualityUprankingBoostForHighQualityProducersParam,\n      Pushservice.QualityUprankingDownboostForLowQualityProducersParam,\n      Pushservice.BqmlHealthModelPredicateFilterThresholdParam,\n      Pushservice.BqmlHealthModelPredicateBucketThresholdParam,\n      Pushservice.PNegMultimodalPredicateModelThresholdParam,\n      Pushservice.PNegMultimodalPredicateBucketThresholdParam,\n      Pushservice.SeeLessOftenF1TriggerF1PushCapWeight,\n      Pushservice.SeeLessOftenF1TriggerNonF1PushCapWeight,\n      Pushservice.SeeLessOftenNonF1TriggerF1PushCapWeight,\n      Pushservice.SeeLessOftenNonF1TriggerNonF1PushCapWeight,\n      Pushservice.SeeLessOftenTripHqTweetTriggerF1PushCapWeight,\n      Pushservice.SeeLessOftenTripHqTweetTriggerNonF1PushCapWeight,\n      Pushservice.SeeLessOftenTripHqTweetTriggerTripHqTweetPushCapWeight,\n      Pushservice.SeeLessOftenNtabOnlyNotifUserPushCapWeight,\n      Pushservice.PromptFeedbackF1TriggerF1PushCapWeight,\n      Pushservice.PromptFeedbackF1TriggerNonF1PushCapWeight,\n      Pushservice.PromptFeedbackNonF1TriggerF1PushCapWeight,\n      Pushservice.PromptFeedbackNonF1TriggerNonF1PushCapWeight,\n      Pushservice.InlineFeedbackF1TriggerF1PushCapWeight,\n      Pushservice.InlineFeedbackF1TriggerNonF1PushCapWeight,\n      Pushservice.InlineFeedbackNonF1TriggerF1PushCapWeight,\n      Pushservice.InlineFeedbackNonF1TriggerNonF1PushCapWeight,\n      Pushservice.TweetNtabDislikeCountThresholdParam,\n      Pushservice.TweetNtabDislikeRateThresholdParam,\n      Pushservice.TweetNtabDislikeCountThresholdForMrTwistlyParam,\n      Pushservice.TweetNtabDislikeRateThresholdForMrTwistlyParam,\n      Pushservice.TweetNtabDislikeCountBucketThresholdParam,\n      Pushservice.MinAuthorSendsThresholdParam,\n      Pushservice.MinTweetSendsThresholdParam,\n      Pushservice.AuthorDislikeRateThresholdParam,\n      Pushservice.AuthorReportRateThresholdParam,\n      Pushservice.FavOverSendThresholdParam,\n      Pushservice.SpreadControlRatioParam,\n      Pushservice.TweetQTtoNtabClickRatioThresholdParam,\n      Pushservice.TweetReplytoLikeRatioThresholdLowerBound,\n      Pushservice.TweetReplytoLikeRatioThresholdUpperBound,\n      Pushservice.AuthorSensitiveMediaFilteringThreshold,\n      Pushservice.AuthorSensitiveMediaFilteringThresholdForMrTwistly,\n      Pushservice.MrRequestScribingEpsGreedyExplorationRatio,\n      Pushservice.SeeLessOftenTopicTriggerTopicPushCapWeight,\n      Pushservice.SeeLessOftenTopicTriggerF1PushCapWeight,\n      Pushservice.SeeLessOftenTopicTriggerOONPushCapWeight,\n      Pushservice.SeeLessOftenF1TriggerTopicPushCapWeight,\n      Pushservice.SeeLessOftenOONTriggerTopicPushCapWeight,\n      Pushservice.SeeLessOftenDefaultPushCapWeight,\n      Pushservice.OverrideMaxSlotFnWeight,\n      Pushservice.QualityPredicateExplicitThresholdParam,\n      Pushservice.AuthorSensitiveScoreWeightInReranking,\n      Pushservice.BigFilteringThresholdParam,\n      Pushservice.NsfwScoreThresholdForF1Copy,\n      Pushservice.NsfwScoreThresholdForOONCopy,\n      Pushservice.HighOONCThresholdForCopy,\n      Pushservice.LowOONCThresholdForCopy,\n      Pushservice.UserDeviceLanguageThresholdParam,\n      Pushservice.UserInferredLanguageThresholdParam,\n      Pushservice.SpammyTweetOonThreshold,\n      Pushservice.SpammyTweetInThreshold,\n      Pushservice.SpammyTweetBucketingThreshold,\n      Pushservice.NumFollowerThresholdForHealthAndQualityFilters,\n      Pushservice.NumFollowerThresholdForHealthAndQualityFiltersPreranking,\n      Pushservice.SoftRankFactorForSubscriptionCreators,\n      Pushservice.MagicFanoutSimClusterDotProductHeavyUserThreshold,\n      Pushservice.MagicFanoutSimClusterDotProductNonHeavyUserThreshold\n    )\n\n  private val doubleSeqFeatureSwitchOverrides =\n    FeatureSwitchOverrideUtil.getDoubleSeqFSOverrides(\n      Pushservice.MfGridSearchThresholdsCohort1,\n      Pushservice.MfGridSearchThresholdsCohort2,\n      Pushservice.MfGridSearchThresholdsCohort3,\n      Pushservice.MfGridSearchThresholdsCohort4,\n      Pushservice.MfGridSearchThresholdsCohort5,\n      Pushservice.MfGridSearchThresholdsCohort6,\n      Pushservice.MrPercentileGridSearchThresholdsCohort1,\n      Pushservice.MrPercentileGridSearchThresholdsCohort2,\n      Pushservice.MrPercentileGridSearchThresholdsCohort3,\n      Pushservice.MrPercentileGridSearchThresholdsCohort4,\n      Pushservice.MrPercentileGridSearchThresholdsCohort5,\n      Pushservice.MrPercentileGridSearchThresholdsCohort6,\n      Pushservice.GlobalOptoutThresholdParam,\n      Pushservice.BucketOptoutSlotThresholdParam,\n      Pushservice.BqmlQualityModelBucketThresholdListParam,\n      Pushservice.SeeLessOftenListOfDayKnobs,\n      Pushservice.SeeLessOftenListOfPushCapWeightKnobs,\n      Pushservice.SeeLessOftenListOfPowerKnobs,\n      Pushservice.PromptFeedbackListOfDayKnobs,\n      Pushservice.PromptFeedbackListOfPushCapWeightKnobs,\n      Pushservice.PromptFeedbackListOfPowerKnobs,\n      Pushservice.InlineFeedbackListOfDayKnobs,\n      Pushservice.InlineFeedbackListOfPushCapWeightKnobs,\n      Pushservice.InlineFeedbackListOfPowerKnobs,\n      Pushservice.OverrideMaxSlotFnPushCapKnobs,\n      Pushservice.OverrideMaxSlotFnPowerKnobs,\n      Pushservice.OverrideMaxSlotFnPushCapKnobs,\n      Pushservice.MagicRecsRelevanceScoreRange,\n      Pushservice.MagicFanoutRelevanceScoreRange,\n      Pushservice.MultilingualPnsfwTweetTextBucketingThreshold,\n      Pushservice.MultilingualPnsfwTweetTextFilteringThreshold,\n    )\n\n  private val booleanFeatureSwitchOverrides = FeatureSwitchOverrideUtil.getBooleanFSOverrides(\n    Pushservice.EnablePushRecommendationsParam,\n    Pushservice.DisableHeavyRankingModelFSParam,\n    Pushservice.EnablePushMixerReplacingAllSources,\n    Pushservice.EnablePushMixerReplacingAllSourcesWithControl,\n    Pushservice.EnablePushMixerReplacingAllSourcesWithExtra,\n    Pushservice.EnablePushMixerSource,\n    Common.EnableScheduledSpaceSpeakers,\n    Common.EnableScheduledSpaceSubscribers,\n    Pushservice.MagicFanoutNewsUserGeneratedEventsEnable,\n    Pushservice.MagicFanoutSkipAccountCountryPredicate,\n    Pushservice.MagicFanoutNewsEnableDescriptionCopy,\n    Pushservice.EnableF1TriggerSeeLessOftenFatigue,\n    Pushservice.EnableNonF1TriggerSeeLessOftenFatigue,\n    Pushservice.AdjustTripHqTweetTriggeredNtabCaretClickFatigue,\n    Pushservice.EnableCuratedTrendTweets,\n    Pushservice.EnableNonCuratedTrendTweets,\n    Pushservice.DisableMlInFilteringFeatureSwitchParam,\n    Pushservice.EnableTopicCopyForMF,\n    Pushservice.EnableTopicCopyForImplicitTopics,\n    Pushservice.EnableRestrictStep,\n    Pushservice.EnableHighPriorityPush,\n    Pushservice.BoostCandidatesFromSubscriptionCreators,\n    Pushservice.SoftRankCandidatesFromSubscriptionCreators,\n    Pushservice.EnableNewMROONCopyForPush,\n    Pushservice.EnableQueryAuthorMediaRepresentationStore,\n    Pushservice.EnableProfanityFilterParam,\n    Pushservice.EnableAbuseStrikeTop2PercentFilterSimCluster,\n    Pushservice.EnableAbuseStrikeTop1PercentFilterSimCluster,\n    Pushservice.EnableAbuseStrikeTop05PercentFilterSimCluster,\n    Pushservice.EnableAgathaUserHealthModelPredicate,\n    Pushservice.PnsfwTweetMediaFilterOonOnly,\n    Pushservice.EnableHealthSignalStorePnsfwTweetTextPredicate,\n    Pushservice.EnableHealthSignalStoreMultilingualPnsfwTweetTextPredicate,\n    Pushservice.DisableHealthFiltersForCrMixerCandidates,\n    Pushservice.EnableOverrideNotificationsForAndroid,\n    Pushservice.EnableOverrideNotificationsForIos,\n    Pushservice.EnableMrRequestScribingForTargetFiltering,\n    Pushservice.EnableMrRequestScribingForCandidateFiltering,\n    Pushservice.EnableMrRequestScribingWithFeatureHydrating,\n    Pushservice.EnableFlattenMrRequestScribing,\n    Pushservice.EnableMrRequestScribingForEpsGreedyExploration,\n    Pushservice.EnableMrRequestScribingDismissScore,\n    Pushservice.EnableMrRequestScribingBigFilteringSupervisedScores,\n    Pushservice.EnableMrRequestScribingBigFilteringRLScores,\n    Pushservice.EnableEventPrimaryMediaAndroid,\n    Pushservice.EnableEventSquareMediaIosMagicFanoutNewsEvent,\n    Pushservice.EnableEventSquareMediaAndroid,\n    Pushservice.EnableMagicFanoutNewsForYouNtabCopy,\n    Pushservice.EnableMfGeoTargeting,\n    Pushservice.EnableRuxLandingPage,\n    Pushservice.EnableNTabRuxLandingPage,\n    Pushservice.EnableGraduallyRampUpNotification,\n    Pushservice.EnableOnboardingPushes,\n    Pushservice.EnableAddressBookPush,\n    Pushservice.EnableCompleteOnboardingPush,\n    Pushservice.EnableOverrideNotificationsSmartPushConfigForAndroid,\n    Pushservice.DisableOnboardingPushFatigue,\n    Pushservice.EnableTopTweetsByGeoCandidates,\n    Pushservice.BackfillRankTopTweetsByGeoCandidates,\n    Pushservice.PopGeoTweetEnableAggressiveThresholds,\n    Pushservice.EnableMrMinDurationSinceMrPushFatigue,\n    Pushservice.EnableF1FromProtectedTweetAuthors,\n    Pushservice.MagicFanoutEnableCustomTargetingNewsEvent,\n    Pushservice.EnableSafeUserTweetTweetypieStore,\n    Pushservice.EnableMrMinDurationSinceMrPushFatigue,\n    Pushservice.EnableHydratingOnlineMRHistoryFeatures,\n    Common.SpaceRecsEnableHostNotifs,\n    Common.SpaceRecsEnableSpeakerNotifs,\n    Common.SpaceRecsEnableListenerNotifs,\n    Common.EnableMagicFanoutProductLaunch,\n    Pushservice.EnableTopTweetsByGeoCandidatesForDormantUsers,\n    Pushservice.EnableOverrideNotificationsScoreBasedOverride,\n    Pushservice.EnableOverrideNotificationsMultipleTargetIds,\n    Pushservice.EnableMinDurationModifier,\n    Pushservice.EnableMinDurationModifierV2,\n    Pushservice.EnableMinDurationModifierByUserHistory,\n    Pushservice.EnableQueryUserOpenedHistory,\n    Pushservice.EnableRandomHourForQuickSend,\n    Pushservice.EnableFrsCandidates,\n    Pushservice.EnableFrsTweetCandidatesTopicSetting,\n    Pushservice.EnableFrsTweetCandidatesTopicAnnotation,\n    Pushservice.EnableFrsTweetCandidatesTopicCopy,\n    Pushservice.EnableCandidateGenerationModelParam,\n    Pushservice.EnableOverrideForSportsCandidates,\n    Pushservice.EnableEventIdBasedOverrideForSportsCandidates,\n    Pushservice.EnableMrModelingBasedCandidates,\n    Pushservice.EnableMrModelingBasedCandidatesTopicSetting,\n    Pushservice.EnableMrModelingBasedCandidatesTopicAnnotation,\n    Pushservice.EnableMrModelingBasedCandidatesTopicCopy,\n    Pushservice.EnableResultFromFrsCandidates,\n    Pushservice.EnableHashspaceCandidates,\n    Pushservice.EnableHashspaceCandidatesTopicSetting,\n    Pushservice.EnableHashspaceCandidatesTopicAnnotation,\n    Pushservice.EnableHashspaceCandidatesTopicCopy,\n    Pushservice.EnableResultFromHashspaceCandidates,\n    Pushservice.EnableDownRankOfNewUserPlaybookTopicFollowPush,\n    Pushservice.EnableDownRankOfNewUserPlaybookTopicTweetPush,\n    Pushservice.EnableTopTweetImpressionsNotification,\n    Pushservice.EnableLightRankingParam,\n    Pushservice.EnableRandomBaselineLightRankingParam,\n    Pushservice.EnableQualityUprankingForHeavyRankingParam,\n    Pushservice.EnableQualityUprankingCrtScoreStatsForHeavyRankingParam,\n    Pushservice.EnableProducersQualityBoostingForHeavyRankingParam,\n    Pushservice.EnableMrScribingMLFeaturesAsFeatureMapForStaging,\n    Pushservice.EnableMrTweetSentimentFeatureHydrationFS,\n    Pushservice.EnableTimelineHealthSignalHydration,\n    Pushservice.EnableTopicEngagementRealTimeAggregatesFS,\n    Pushservice.EnableMrUserSemanticCoreFeatureForExpt,\n    Pushservice.EnableHydratingRealGraphTargetUserFeatures,\n    Pushservice.EnableHydratingUserDurationSinceLastVisitFeatures,\n    Pushservice.EnableRealGraphUserAuthorAndSocialContxtFeatureHydration,\n    Pushservice.EnableUserTopicAggregatesFS,\n    Pushservice.EnableTimelineHealthSignalHydrationForModelTraining,\n    Pushservice.EnableMrUserSocialContextAggregateFeatureHydration,\n    Pushservice.EnableMrUserSemanticCoreAggregateFeatureHydration,\n    Pushservice.EnableMrUserCandidateSparseOfflineAggregateFeatureHydration,\n    Pushservice.EnableMrUserCandidateOfflineAggregateFeatureHydration,\n    Pushservice.EnableMrUserCandidateOfflineCompactAggregateFeatureHydration,\n    Pushservice.EnableMrUserAuthorOfflineAggregateFeatureHydration,\n    Pushservice.EnableMrUserAuthorOfflineCompactAggregateFeatureHydration,\n    Pushservice.EnableMrUserOfflineCompactAggregateFeatureHydration,\n    Pushservice.EnableMrUserSimcluster2020AggregateFeatureHydration,\n    Pushservice.EnableMrUserOfflineAggregateFeatureHydration,\n    Pushservice.EnableBqmlQualityModelPredicateParam,\n    Pushservice.EnableBqmlQualityModelScoreHistogramParam,\n    Pushservice.EnableBqmlHealthModelPredicateParam,\n    Pushservice.EnableBqmlHealthModelPredictionForInNetworkCandidatesParam,\n    Pushservice.EnableBqmlHealthModelScoreHistogramParam,\n    Pushservice.EnablePNegMultimodalPredicateParam,\n    Pushservice.EnableNegativeKeywordsPredicateParam,\n    Pushservice.EnableTweetAuthorAggregatesFeatureHydrationParam,\n    Pushservice.OonTweetLengthPredicateUpdatedMediaLogic,\n    Pushservice.OonTweetLengthPredicateUpdatedQuoteTweetLogic,\n    Pushservice.OonTweetLengthPredicateMoreStrictForUndefinedLanguages,\n    Pushservice.EnablePrerankingTweetLengthPredicate,\n    Pushservice.EnableDeTopicTweetCandidates,\n    Pushservice.EnableDeTopicTweetCandidateResults,\n    Pushservice.EnableDeTopicTweetCandidatesCustomTopics,\n    Pushservice.EnableDeTopicTweetCandidatesCustomLanguages,\n    Pushservice.EnableMrTweetSimClusterFeatureHydrationFS,\n    Pushservice.DisableOutNetworkTweetCandidatesFS,\n    Pushservice.EnableLaunchVideosInImmersiveExplore,\n    Pushservice.EnableStoringNtabGenericNotifKey,\n    Pushservice.EnableDeletingNtabTimeline,\n    Pushservice.EnableOverrideNotificationsNSlots,\n    Pushservice.EnableNslotsForOverrideOnNtab,\n    Pushservice.EnableOverrideMaxSlotFn,\n    Pushservice.EnableTargetIdInSmartPushPayloadForMagicFanoutSportsEvent,\n    Pushservice.EnableOverrideIdNTabRequest,\n    Pushservice.EnableOverrideForSpaces,\n    Pushservice.EnableTopicProofTweetRecs,\n    Pushservice.EnableHealthFiltersForTopicProofTweet,\n    Pushservice.EnableTargetIdsInSmartPushPayload,\n    Pushservice.EnableSecondaryAccountPredicateMF,\n    Pushservice.EnableInlineVideo,\n    Pushservice.EnableAutoplayForInlineVideo,\n    Pushservice.EnableOONGeneratedInlineActions,\n    Pushservice.EnableInlineFeedbackOnPush,\n    Pushservice.UseInlineActionsV1,\n    Pushservice.UseInlineActionsV2,\n    Pushservice.EnableFeaturedSpacesOON,\n    Pushservice.CheckFeaturedSpaceOON,\n    Pushservice.EnableGeoTargetingForSpaces,\n    Pushservice.EnableEmployeeOnlySpaceNotifications,\n    Pushservice.EnableSpacesTtlForNtab,\n    Pushservice.EnableCustomThreadIdForOverride,\n    Pushservice.EnableSimClusterTargetingSpaces,\n    Pushservice.TargetInInlineActionAppVisitFatigue,\n    Pushservice.EnableInlineActionAppVisitFatigue,\n    Pushservice.EnableThresholdOfFavMrModelingBasedCandidates,\n    Pushservice.HydrateMrUserSimclusterV2020InModelingBasedCG,\n    Pushservice.HydrateMrUserSemanticCoreInModelingBasedCG,\n    Pushservice.HydrateOnboardingInModelingBasedCG,\n    Pushservice.HydrateTopicFollowInModelingBasedCG,\n    Pushservice.HydrateMrUserTopicInModelingBasedCG,\n    Pushservice.HydrateMrUserAuthorInModelingBasedCG,\n    Pushservice.HydrateUserPenguinLanguageInModelingBasedCG,\n    Pushservice.EnableMrUserEngagedTweetTokensFeature,\n    Pushservice.HydrateMrUserHashspaceEmbeddingInModelingBasedCG,\n    Pushservice.HydrateUseGeoInModelingBasedCG,\n    Pushservice.EnableSpaceCohostJoinEvent,\n    Pushservice.EnableOONFilteringBasedOnUserSettings,\n    Pushservice.EnableContFnF1TriggerSeeLessOftenFatigue,\n    Pushservice.EnableContFnNonF1TriggerSeeLessOftenFatigue,\n    Pushservice.EnableContFnF1TriggerPromptFeedbackFatigue,\n    Pushservice.EnableContFnNonF1TriggerPromptFeedbackFatigue,\n    Pushservice.EnableContFnF1TriggerInlineFeedbackFatigue,\n    Pushservice.EnableContFnNonF1TriggerInlineFeedbackFatigue,\n    Pushservice.UseInlineDislikeForFatigue,\n    Pushservice.UseInlineDismissForFatigue,\n    Pushservice.UseInlineSeeLessForFatigue,\n    Pushservice.UseInlineNotRelevantForFatigue,\n    Pushservice.GPEnableCustomMagicFanoutCricketFatigue,\n    Pushservice.IncludeRelevanceScoreInIbis2Payload,\n    Pushservice.BypassGlobalSpacePushCapForSoftDeviceFollow,\n    Pushservice.EnableCountryCodeBackoffTopTweetsByGeo,\n    Pushservice.EnableNewCreatorPush,\n    Pushservice.EnableCreatorSubscriptionPush,\n    Pushservice.EnableInsSender,\n    Pushservice.EnableOptoutAdjustedPushcap,\n    Pushservice.EnableOONBackfillBasedOnF1Candidates,\n    Pushservice.EnableVFInTweetypie,\n    Pushservice.EnablePushPresentationVerifiedSymbol,\n    Pushservice.EnableHighPrioritySportsPush,\n    Pushservice.EnableSearchURLRedirectForSportsFanout,\n    Pushservice.EnableScoreFanoutNotification,\n    Pushservice.EnableExplicitPushCap,\n    Pushservice.EnableNsfwTokenBasedFiltering,\n    Pushservice.EnableRestrictedMinModelPushcap,\n    Pushservice.EnableGenericCRTBasedFatiguePredicate,\n    Pushservice.EnableCopyFeaturesForF1,\n    Pushservice.EnableEmojiInF1Copy,\n    Pushservice.EnableTargetInF1Copy,\n    Pushservice.EnableCopyFeaturesForOon,\n    Pushservice.EnableEmojiInOonCopy,\n    Pushservice.EnableTargetInOonCopy,\n    Pushservice.EnableF1CopyBody,\n    Pushservice.EnableOONCopyBody,\n    Pushservice.EnableIosCopyBodyTruncate,\n    Pushservice.EnableHTLBasedFatigueBasicRule,\n    Pushservice.EnableTargetAndEmojiSplitFatigue,\n    Pushservice.EnableNsfwCopy,\n    Pushservice.EnableOONCopyBody,\n    Pushservice.EnableTweetDynamicInlineActions,\n    Pushservice.EnablePushcapRefactor,\n    Pushservice.BigFilteringEnableHistogramsParam,\n    Pushservice.EnableTweetTranslation,\n    Pushservice.TripTweetCandidateReturnEnable,\n    Pushservice.EnableSocialContextForRetweet,\n    Pushservice.EnableEmptyBody,\n    Pushservice.EnableLocalViralTweets,\n    Pushservice.EnableExploreVideoTweets,\n    Pushservice.EnableDynamicInlineActionsForDesktopWeb,\n    Pushservice.EnableDynamicInlineActionsForMobileWeb,\n    Pushservice.EnableNTabEntriesForSportsEventNotifications,\n    Pushservice.EnableNTabFacePileForSportsEventNotifications,\n    Pushservice.DisableIsTargetBlueVerifiedPredicate,\n    Pushservice.EnableNTabEntriesForProductLaunchNotifications,\n    Pushservice.DisableIsTargetLegacyVerifiedPredicate,\n    Pushservice.EnableNTabOverrideForSportsEventNotifications,\n    Pushservice.EnableOONCBasedCopy,\n    Pushservice.HighQualityCandidatesEnableCandidateSource,\n    Pushservice.HighQualityCandidatesEnableFallback,\n    Pushservice.EnableTweetLanguageFilter,\n    Pushservice.EnableListRecommendations,\n    Pushservice.EnableIDSListRecommendations,\n    Pushservice.EnablePopGeoListRecommendations,\n    Pushservice.SkipLanguageFilterForMediaTweets,\n    Pushservice.EnableSpammyTweetFilter,\n    Pushservice.EnableTweetPushToHomeAndroid,\n    Pushservice.EnableTweetPushToHomeiOS,\n    Pushservice.EnableBoundedFeatureSetForSocialContext,\n    Pushservice.EnableStpBoundedFeatureSetForUserSocialContext,\n    Pushservice.EnableCoreUserHistoryBoundedFeatureSetForSocialContext,\n    Pushservice.SkipPostRankingFilters,\n    Pushservice.MRWebHoldbackParam,\n    Pushservice.EnableIsTargetSuperFollowCreatorPredicate\n  )\n\n  private val longSeqFeatureSwitchOverrides =\n    FeatureSwitchOverrideUtil.getLongSeqFSOverrides(\n      Pushservice.MagicFanoutEventAllowlistToSkipAccountCountryPredicate\n    )\n\n  private val longSetFeatureSwitchOverrides =\n    FeatureSwitchOverrideUtil.getLongSetFSOverrides(\n      Pushservice.ListOfAdhocIdsForStatsTracking\n    )\n\n  private val stringSeqFeatureSwitchOverrides =\n    FeatureSwitchOverrideUtil.getStringSeqFSOverrides(\n      Pushservice.ListOfCrtsForOpenApp,\n      Pushservice.ListOfCrtsToUpRank,\n      Pushservice.OONCandidatesDisabledCrTagParam,\n      Pushservice.ListOfCrtsToDownRank,\n      Pushservice.MagicFanoutDenyListedCountries,\n      Pushservice.GlobalOptoutModelParam,\n      Pushservice.BqmlQualityModelBucketModelIdListParam,\n      Pushservice.CommonRecommendationTypeDenyListPushHoldbacks,\n      Pushservice.TargetLevelFeatureListForMrRequestScribing,\n      Pushservice.MagicFanoutSportsEventDenyListedCountries,\n      Pushservice.MultilingualPnsfwTweetTextSupportedLanguages,\n      Pushservice.NegativeKeywordsPredicateDenylist,\n      Pushservice.TripTweetCandidateSourceIds,\n      Pushservice.NsfwTokensParam,\n      Pushservice.HighQualityCandidatesFallbackSourceIds\n    )\n\n  private val intSeqFeatureSwitchOverrides =\n    FeatureSwitchOverrideUtil.getIntSeqFSOverrides(\n      Pushservice.BucketOptoutSlotPushcapParam,\n      Pushservice.GeoHashLengthList,\n      Pushservice.MinDurationModifierStartHourList,\n      Pushservice.MinDurationModifierEndHourList,\n      Pushservice.MinDurationTimeModifierConst\n    )\n\n  private val enumFeatureSwitchOverrides = FeatureSwitchOverrideUtil.getEnumFSOverrides(\n    stat,\n    logger,\n    Pushservice.MRBoldTitleFavoriteAndRetweetParam,\n    Pushservice.QualityUprankingTransformTypeParam,\n    Pushservice.QualityPredicateIdParam,\n    Pushservice.BigFilteringNormalizationTypeIdParam,\n    Common.PushcapModelType,\n    Common.MFCricketTargetingPredicate,\n    Pushservice.RankingFunctionForTopTweetsByGeo,\n    Pushservice.TopTweetsByGeoCombinationParam,\n    Pushservice.PopGeoTweetVersionParam,\n    Pushservice.SubtextInAndroidPushHeaderParam,\n    Pushservice.HighOONCTweetFormat,\n    Pushservice.LowOONCTweetFormat,\n  )\n\n  private val enumSeqFeatureSwitchOverrides = FeatureSwitchOverrideUtil.getEnumSeqFSOverrides(\n    stat,\n    logger,\n    Pushservice.OONTweetDynamicInlineActionsList,\n    Pushservice.TweetDynamicInlineActionsList,\n    Pushservice.TweetDynamicInlineActionsListForWeb,\n    Pushservice.HighQualityCandidatesEnableGroups,\n    Pushservice.HighQualityCandidatesFallbackEnabledGroups,\n    Pushservice.OONCandidatesDisabledCrtGroupParam,\n    Pushservice.MultilingualPnsfwTweetTextBucketingModelList,\n  )\n\n  private val stringFeatureSwitchOverrides = FeatureSwitchOverrideUtil.getStringFSOverrides(\n    Common.PushcapModelPredictionVersion,\n    Pushservice.WeightedOpenOrNtabClickRankingModelParam,\n    Pushservice.WeightedOpenOrNtabClickFilteringModelParam,\n    Pushservice.BucketOptoutModelParam,\n    Pushservice.ScoringFuncForTopTweetsByGeo,\n    Pushservice.LightRankingModelTypeParam,\n    Pushservice.BigFilteringSupervisedSendingModelParam,\n    Pushservice.BigFilteringSupervisedWithoutSendingModelParam,\n    Pushservice.BigFilteringRLSendingModelParam,\n    Pushservice.BigFilteringRLWithoutSendingModelParam,\n    Pushservice.BqmlQualityModelTypeParam,\n    Pushservice.BqmlHealthModelTypeParam,\n    Pushservice.QualityUprankingModelTypeParam,\n    Pushservice.SearchURLRedirectForSportsFanout,\n    Pushservice.LocalViralTweetsBucket,\n    Pushservice.HighQualityCandidatesHeavyRankingModel,\n    Pushservice.HighQualityCandidatesNonPersonalizedQualityCnnModel,\n    Pushservice.HighQualityCandidatesBqmlNsfwModel,\n    Pushservice.HighQualityCandidatesBqmlReportModel,\n    Pushservice.ProductLaunchLandingPageDeepLink,\n    Pushservice.ProductLaunchTapThrough,\n    Pushservice.TweetLanguageFeatureNameParam\n  )\n\n  private val durationFeatureSwitchOverrides =\n    FeatureSwitchOverrideUtil.getBoundedDurationFSOverrides(\n      Common.NumberOfDaysToFilterMRForSeeLessOften,\n      Common.NumberOfDaysToReducePushCapForSeeLessOften,\n      Pushservice.NumberOfDaysToFilterForSeeLessOftenForF1TriggerF1,\n      Pushservice.NumberOfDaysToReducePushCapForSeeLessOftenForF1TriggerF1,\n      Pushservice.NumberOfDaysToFilterForSeeLessOftenForF1TriggerNonF1,\n      Pushservice.NumberOfDaysToReducePushCapForSeeLessOftenForF1TriggerNonF1,\n      Pushservice.NumberOfDaysToFilterForSeeLessOftenForNonF1TriggerF1,\n      Pushservice.NumberOfDaysToReducePushCapForSeeLessOftenForNonF1TriggerF1,\n      Pushservice.NumberOfDaysToFilterForSeeLessOftenForNonF1TriggerNonF1,\n      Pushservice.NumberOfDaysToReducePushCapForSeeLessOftenForNonF1TriggerNonF1,\n      Pushservice.TrendTweetNotificationsFatigueDuration,\n      Pushservice.MinDurationSincePushParam,\n      Pushservice.MFMinIntervalFatigue,\n      Pushservice.SimclusterBasedCandidateMaxTweetAgeParam,\n      Pushservice.DetopicBasedCandidateMaxTweetAgeParam,\n      Pushservice.F1CandidateMaxTweetAgeParam,\n      Pushservice.MaxTweetAgeParam,\n      Pushservice.ModelingBasedCandidateMaxTweetAgeParam,\n      Pushservice.GeoPopTweetMaxAgeInHours,\n      Pushservice.MinDurationSincePushParam,\n      Pushservice.GraduallyRampUpPhaseDurationDays,\n      Pushservice.MrMinDurationSincePushForOnboardingPushes,\n      Pushservice.FatigueForOnboardingPushes,\n      Pushservice.FrigateHistoryOtherNotificationWriteTtl,\n      Pushservice.FrigateHistoryTweetNotificationWriteTtl,\n      Pushservice.TopTweetsByGeoPushInterval,\n      Pushservice.HighQualityTweetsPushInterval,\n      Pushservice.MrMinDurationSincePushForTopTweetsByGeoPushes,\n      Pushservice.TimeSinceLastLoginForGeoPopTweetPush,\n      Pushservice.NewUserPlaybookAllowedLastLoginHours,\n      Pushservice.SpaceRecsAppFatigueDuration,\n      Pushservice.OONSpaceRecsFatigueDuration,\n      Pushservice.SpaceRecsFatigueMinIntervalDuration,\n      Pushservice.SpaceRecsGlobalFatigueDuration,\n      Pushservice.MinimumTimeSinceLastLoginForGeoPopTweetPush,\n      Pushservice.MinFatigueDurationSinceLastHTLVisit,\n      Pushservice.LastHTLVisitBasedNonFatigueWindow,\n      Pushservice.SpaceNotificationsTTLDurationForNTab,\n      Pushservice.OverrideNotificationsLookbackDurationForOverrideInfo,\n      Pushservice.OverrideNotificationsLookbackDurationForImpressionId,\n      Pushservice.OverrideNotificationsLookbackDurationForNTab,\n      Pushservice.TopTweetImpressionsNotificationInterval,\n      Pushservice.TopTweetImpressionsFatigueMinIntervalDuration,\n      Pushservice.MFPushIntervalInHours,\n      Pushservice.InlineActionAppVisitFatigue,\n      Pushservice.SpaceParticipantHistoryLastActiveThreshold,\n      Pushservice.SportsMinIntervalFatigue,\n      Pushservice.SportsPushIntervalInHours,\n      Pushservice.SportsMinIntervalFatiguePerEvent,\n      Pushservice.SportsPushIntervalInHoursPerEvent,\n      Pushservice.TargetNtabOnlyCapFatigueIntervalHours,\n      Pushservice.TargetPushCapFatigueIntervalHours,\n      Pushservice.CopyFeaturesHistoryLookbackDuration,\n      Pushservice.F1EmojiCopyFatigueDuration,\n      Pushservice.F1TargetCopyFatigueDuration,\n      Pushservice.OonEmojiCopyFatigueDuration,\n      Pushservice.OonTargetCopyFatigueDuration,\n      Pushservice.ProductLaunchPushIntervalInHours,\n      Pushservice.ExploreVideoTweetAgeParam,\n      Pushservice.ListRecommendationsPushInterval,\n      Pushservice.ProductLaunchMinIntervalFatigue,\n      Pushservice.NewCreatorPushIntervalInHours,\n      Pushservice.NewCreatorPushMinIntervalFatigue,\n      Pushservice.CreatorSubscriptionPushIntervalInHours,\n      Pushservice.CreatorSubscriptionPushhMinIntervalFatigue\n    )\n\n  private[params] val allFeatureSwitchOverrides =\n    booleanDeciderOverrides ++\n      booleanFeatureSwitchOverrides ++\n      intFeatureSwitchOverrides ++\n      doubleFeatureSwitchOverrides ++\n      doubleSeqFeatureSwitchOverrides ++\n      enumFeatureSwitchOverrides ++\n      stringSeqFeatureSwitchOverrides ++\n      stringFeatureSwitchOverrides ++\n      durationFeatureSwitchOverrides ++\n      intSeqFeatureSwitchOverrides ++\n      longSeqFeatureSwitchOverrides ++\n      enumSeqFeatureSwitchOverrides ++\n      longSetFeatureSwitchOverrides\n\n  val config = BaseConfigBuilder(allFeatureSwitchOverrides).build()\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/params/PushMLModelParams.scala",
    "content": "package com.twitter.frigate.pushservice.params\n\n/**\n * This enum defines ML models for push\n */\nobject PushMLModel extends Enumeration {\n  type PushMLModel = Value\n\n  val WeightedOpenOrNtabClickProbability = Value\n  val DauProbability = Value\n  val OptoutProbability = Value\n  val FilteringProbability = Value\n  val BigFilteringSupervisedSendingModel = Value\n  val BigFilteringSupervisedWithoutSendingModel = Value\n  val BigFilteringRLSendingModel = Value\n  val BigFilteringRLWithoutSendingModel = Value\n  val HealthNsfwProbability = Value\n}\n\nobject WeightedOpenOrNtabClickModel {\n  type ModelNameType = String\n\n  // MR models\n  val Periodically_Refreshed_Prod_Model =\n    \"Periodically_Refreshed_Prod_Model\" // used in DBv2 service, needed for gradually migrate via feature switch\n}\n\n\nobject OptoutModel {\n  type ModelNameType = String\n  val D0_has_realtime_features = \"D0_has_realtime_features\"\n  val D0_no_realtime_features = \"D0_no_realtime_features\"\n}\n\nobject HealthNsfwModel {\n  type ModelNameType = String\n  val Q2_2022_Mr_Bqml_Health_Model_NsfwV0 = \"Q2_2022_Mr_Bqml_Health_Model_NsfwV0\"\n}\n\nobject BigFilteringSupervisedModel {\n  type ModelNameType = String\n  val V0_0_BigFiltering_Supervised_Sending_Model = \"Q3_2022_bigfiltering_supervised_send_model_v0\"\n  val V0_0_BigFiltering_Supervised_Without_Sending_Model =\n    \"Q3_2022_bigfiltering_supervised_not_send_model_v0\"\n}\n\nobject BigFilteringRLModel {\n  type ModelNameType = String\n  val V0_0_BigFiltering_Rl_Sending_Model = \"Q3_2022_bigfiltering_rl_send_model_dqn_dau_15_open\"\n  val V0_0_BigFiltering_Rl_Without_Sending_Model =\n    \"Q3_2022_bigfiltering_rl_not_send_model_dqn_dau_15_open\"\n}\n\ncase class PushModelName(\n  modelType: PushMLModel.Value,\n  version: WeightedOpenOrNtabClickModel.ModelNameType) {\n  override def toString: String = {\n    modelType.toString + \"_\" + version\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/params/PushParams.scala",
    "content": "package com.twitter.frigate.pushservice.params\n\nimport com.twitter.rux.common.context.thriftscala.ExperimentKey\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.timelines.configapi.decider.BooleanDeciderParam\n\nobject PushParams {\n\n  /**\n   * Disable ML models in filtering\n   */\n  object DisableMlInFilteringParam extends BooleanDeciderParam(DeciderKey.disableMLInFiltering)\n\n  /**\n   * Disable ML models in ranking, use random ranking instead\n   * This param is used for ML holdback and training data collection\n   */\n  object UseRandomRankingParam extends Param(false)\n\n  /**\n   * Disable feature hydration, ML ranking, and ML filtering\n   * Use default order from candidate source\n   * This param is for service continuity\n   */\n  object DisableAllRelevanceParam extends BooleanDeciderParam(DeciderKey.disableAllRelevance)\n\n  /**\n   * Disable ML heavy ranking\n   * Use default order from candidate source\n   * This param is for service continuity\n   */\n  object DisableHeavyRankingParam extends BooleanDeciderParam(DeciderKey.disableHeavyRanking)\n\n  /**\n   * Restrict ML light ranking by selecting top3 candidates\n   * Use default order from candidate source\n   * This param is for service continuity\n   */\n  object RestrictLightRankingParam extends BooleanDeciderParam(DeciderKey.restrictLightRanking)\n\n  /**\n   * Downsample ML light ranking scribed candidates\n   */\n  object DownSampleLightRankingScribeCandidatesParam\n      extends BooleanDeciderParam(DeciderKey.downSampleLightRankingScribeCandidates)\n\n  /**\n   * Set it to true only for Android only ranking experiments\n   */\n  object AndroidOnlyRankingExperimentParam extends Param(false)\n\n  /**\n   * Enable the user_tweet_entity_graph tweet candidate source.\n   */\n  object UTEGTweetCandidateSourceParam\n      extends BooleanDeciderParam(DeciderKey.entityGraphTweetRecsDeciderKey)\n\n  /**\n   * Enable writes to Notification Service\n   */\n  object EnableWritesToNotificationServiceParam\n      extends BooleanDeciderParam(DeciderKey.enablePushserviceWritesToNotificationServiceDeciderKey)\n\n  /**\n   * Enable writes to Notification Service for all employees\n   */\n  object EnableWritesToNotificationServiceForAllEmployeesParam\n      extends BooleanDeciderParam(\n        DeciderKey.enablePushserviceWritesToNotificationServiceForAllEmployeesDeciderKey)\n\n  /**\n   * Enable writes to Notification Service for everyone\n   */\n  object EnableWritesToNotificationServiceForEveryoneParam\n      extends BooleanDeciderParam(\n        DeciderKey.enablePushserviceWritesToNotificationServiceForEveryoneDeciderKey)\n\n  /**\n   * Enable fatiguing MR for Ntab caret click\n   */\n  object EnableFatigueNtabCaretClickingParam extends Param(true)\n\n  /**\n   * Param for disabling in-network Tweet candidates\n   */\n  object DisableInNetworkTweetCandidatesParam extends Param(false)\n\n  /**\n   * Decider controlled param to enable prompt feedback response NO predicate\n   */\n  object EnablePromptFeedbackFatigueResponseNoPredicate\n      extends BooleanDeciderParam(\n        DeciderKey.enablePromptFeedbackFatigueResponseNoPredicateDeciderKey)\n\n  /**\n   * Enable hydration and generation of Social context (TF, TR) based candidates for Earlybird Tweets\n   */\n  object EarlyBirdSCBasedCandidatesParam\n      extends BooleanDeciderParam(DeciderKey.enableUTEGSCForEarlybirdTweetsDecider)\n\n  /**\n   * Param to allow reduce to one social proof for tweet param in UTEG\n   */\n  object AllowOneSocialProofForTweetInUTEGParam extends Param(true)\n\n  /**\n   * Param to query UTEG for out network tweets only\n   */\n  object OutNetworkTweetsOnlyForUTEGParam extends Param(false)\n\n  object EnablePushSendEventBus extends BooleanDeciderParam(DeciderKey.enablePushSendEventBus)\n\n  /**\n   * Enable RUX Tweet landing page for push open on iOS\n   */\n  object EnableRuxLandingPageIOSParam extends Param[Boolean](true)\n\n  /**\n   * Enable RUX Tweet landing page for push open on Android\n   */\n  object EnableRuxLandingPageAndroidParam extends Param[Boolean](true)\n\n  /**\n   * Param to decide which ExperimentKey to be encoded into Rux landing page context object.\n   * The context object is sent to rux-api and rux-api applies logic (e.g. show reply module on\n   * rux landing page or not) accordingly based on the experiment key.\n   */\n  object RuxLandingPageExperimentKeyIOSParam extends Param[Option[ExperimentKey]](None)\n  object RuxLandingPageExperimentKeyAndroidParam extends Param[Option[ExperimentKey]](None)\n\n  /**\n   * Param to enable MR Tweet Fav Recs\n   */\n  object MRTweetFavRecsParam extends BooleanDeciderParam(DeciderKey.enableTweetFavRecs)\n\n  /**\n   * Param to enable MR Tweet Retweet Recs\n   */\n  object MRTweetRetweetRecsParam extends BooleanDeciderParam(DeciderKey.enableTweetRetweetRecs)\n\n  /**\n   * Param to disable writing to NTAB\n   * */\n  object DisableWritingToNTAB extends Param[Boolean](default = false)\n\n  /**\n   * Param to show RUX landing page as a modal on iOS\n   */\n  object ShowRuxLandingPageAsModalOnIOS extends Param[Boolean](default = false)\n\n  /**\n   * Param to enable mr end to end scribing\n   */\n  object EnableMrRequestScribing extends BooleanDeciderParam(DeciderKey.enableMrRequestScribing)\n\n  /**\n   * Param to enable scribing of high quality candidate scores\n   */\n  object EnableHighQualityCandidateScoresScribing\n      extends BooleanDeciderParam(DeciderKey.enableHighQualityCandidateScoresScribing)\n\n  /**\n   * Decider controlled param to pNeg multimodal predictions for F1 tweets\n   */\n  object EnablePnegMultimodalPredictionForF1Tweets\n      extends BooleanDeciderParam(DeciderKey.enablePnegMultimodalPredictionForF1Tweets)\n\n  /**\n   * Decider controlled param to scribe oonFav score for F1 tweets\n   */\n  object EnableScribeOonFavScoreForF1Tweets\n      extends BooleanDeciderParam(DeciderKey.enableScribingOonFavScoreForF1Tweets)\n\n  /**\n   * Param to enable htl user aggregates extended hydration\n   */\n  object EnableHtlOfflineUserAggregatesExtendedHydration\n      extends BooleanDeciderParam(DeciderKey.enableHtlOfflineUserAggregateExtendedFeaturesHydration)\n\n  /**\n   * Param to enable predicate detailed info scribing\n   */\n  object EnablePredicateDetailedInfoScribing\n      extends BooleanDeciderParam(DeciderKey.enablePredicateDetailedInfoScribing)\n\n  /**\n   * Param to enable predicate detailed info scribing\n   */\n  object EnablePushCapInfoScribing\n      extends BooleanDeciderParam(DeciderKey.enablePredicateDetailedInfoScribing)\n\n  /**\n   * Param to enable user signal language feature hydration\n   */\n  object EnableUserSignalLanguageFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableUserSignalLanguageFeatureHydration)\n\n  /**\n   * Param to enable user preferred language feature hydration\n   */\n  object EnableUserPreferredLanguageFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableUserPreferredLanguageFeatureHydration)\n\n  /**\n   * Param to enable ner erg feature hydration\n   */\n  object EnableNerErgFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableNerErgFeaturesHydration)\n\n  /**\n   * Param to enable inline action on push copy for Android\n   */\n  object MRAndroidInlineActionOnPushCopyParam extends Param[Boolean](default = true)\n\n  /**\n   * Param to enable hydrating mr user semantic core embedding features\n   * */\n  object EnableMrUserSemanticCoreFeaturesHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrUserSemanticCoreFeaturesHydration)\n\n  /**\n   * Param to enable hydrating mr user semantic core embedding features filtered by 0.0000001\n   * */\n  object EnableMrUserSemanticCoreNoZeroFeaturesHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrUserSemanticCoreNoZeroFeaturesHydration)\n\n  /*\n   * Param to enable days since user's recent resurrection features hydration\n   */\n  object EnableDaysSinceRecentResurrectionFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableDaysSinceRecentResurrectionFeatureHydration)\n\n  /*\n   * Param to enable days since user past aggregates features hydration\n   */\n  object EnableUserPastAggregatesFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableUserPastAggregatesFeatureHydration)\n\n  /*\n   * Param to enable mr user simcluster features (v2020) hydration\n   * */\n  object EnableMrUserSimclusterV2020FeaturesHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrUserSimclusterV2020FeaturesHydration)\n\n  /*\n   * Param to enable mr user simcluster features (v2020) hydration\n   * */\n  object EnableMrUserSimclusterV2020NoZeroFeaturesHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrUserSimclusterV2020NoZeroFeaturesHydration)\n\n  /*\n   * Param to enable HTL topic engagement realtime aggregate features\n   * */\n  object EnableTopicEngagementRealTimeAggregatesFeatureHydration\n      extends BooleanDeciderParam(\n        DeciderKey.enableTopicEngagementRealTimeAggregatesFeatureHydration)\n\n  object EnableUserTopicAggregatesFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableUserTopicAggregatesFeatureHydration)\n\n  /**\n   * Param to enable user author RTA feature hydration\n   */\n  object EnableHtlUserAuthorRTAFeaturesFromFeatureStoreHydration\n      extends BooleanDeciderParam(DeciderKey.enableHtlUserAuthorRealTimeAggregateFeatureHydration)\n\n  /**\n   * Param to enable duration since last visit features\n   */\n  object EnableDurationSinceLastVisitFeatures\n      extends BooleanDeciderParam(DeciderKey.enableDurationSinceLastVisitFeatureHydration)\n\n  object EnableTweetAnnotationFeaturesHydration\n      extends BooleanDeciderParam(DeciderKey.enableTweetAnnotationFeatureHydration)\n\n  /**\n   * Param to Enable visibility filtering through SpaceVisibilityLibrary from SpacePredicate\n   */\n  object EnableSpaceVisibilityLibraryFiltering\n      extends BooleanDeciderParam(DeciderKey.enableSpaceVisibilityLibraryFiltering)\n\n  /*\n   * Param to enable user topic follow feature set hydration\n   * */\n  object EnableUserTopicFollowFeatureSetHydration\n      extends BooleanDeciderParam(DeciderKey.enableUserTopicFollowFeatureSet)\n\n  /*\n   * Param to enable onboarding new user feature set hydration\n   * */\n  object EnableOnboardingNewUserFeatureSetHydration\n      extends BooleanDeciderParam(DeciderKey.enableOnboardingNewUserFeatureSet)\n\n  /*\n   * Param to enable mr user author sparse continuous feature set hydration\n   * */\n  object EnableMrUserAuthorSparseContFeatureSetHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrUserAuthorSparseContFeatureSet)\n\n  /*\n   * Param to enable mr user topic sparse continuous feature set hydration\n   * */\n  object EnableMrUserTopicSparseContFeatureSetHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrUserTopicSparseContFeatureSet)\n\n  /*\n   * Param to enable penguin language feature set hydration\n   * */\n  object EnableUserPenguinLanguageFeatureSetHydration\n      extends BooleanDeciderParam(DeciderKey.enableUserPenguinLanguageFeatureSet)\n\n  /*\n   * Param to enable user engaged tweet tokens feature hydration\n   * */\n  object EnableMrUserEngagedTweetTokensFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrUserEngagedTweetTokensFeaturesHydration)\n\n  /*\n   * Param to enable candidate tweet tokens feature hydration\n   * */\n  object EnableMrCandidateTweetTokensFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrCandidateTweetTokensFeaturesHydration)\n\n  /*\n   * Param to enable mr user hashspace embedding feature set hydration\n   * */\n  object EnableMrUserHashspaceEmbeddingFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrUserHashspaceEmbeddingFeatureSet)\n\n  /*\n   * Param to enable mr tweet sentiment feature set hydration\n   * */\n  object EnableMrTweetSentimentFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrTweetSentimentFeatureSet)\n\n  /*\n   * Param to enable mr tweet_author aggregates feature set hydration\n   * */\n  object EnableMrTweetAuthorAggregatesFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrTweetAuthorAggregatesFeatureSet)\n\n  /**\n   * Param to enable twistly aggregated features\n   */\n  object EnableTwistlyAggregatesFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableTwistlyAggregatesFeatureHydration)\n\n  /**\n   * Param to enable tweet twhin favoriate features\n   */\n  object EnableTweetTwHINFavFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableTweetTwHINFavFeaturesHydration)\n\n  /*\n   * Param to enable mr user geo feature set hydration\n   * */\n  object EnableUserGeoFeatureSetHydration\n      extends BooleanDeciderParam(DeciderKey.enableUserGeoFeatureSet)\n\n  /*\n   * Param to enable mr author geo feature set hydration\n   * */\n  object EnableAuthorGeoFeatureSetHydration\n      extends BooleanDeciderParam(DeciderKey.enableAuthorGeoFeatureSet)\n\n  /*\n   * Param to ramp up mr user geo feature set hydration\n   * */\n  object RampupUserGeoFeatureSetHydration\n      extends BooleanDeciderParam(DeciderKey.rampupUserGeoFeatureSet)\n\n  /*\n   * Param to ramp up mr author geo feature set hydration\n   * */\n  object RampupAuthorGeoFeatureSetHydration\n      extends BooleanDeciderParam(DeciderKey.rampupAuthorGeoFeatureSet)\n\n  /*\n   *  Decider controlled param to enable Pop Geo Tweets\n   * */\n  object PopGeoCandidatesDecider extends BooleanDeciderParam(DeciderKey.enablePopGeoTweets)\n\n  /**\n   * Decider controlled param to enable Trip Geo Tweets\n   */\n  object TripGeoTweetCandidatesDecider\n      extends BooleanDeciderParam(DeciderKey.enableTripGeoTweetCandidates)\n\n  /**\n   * Decider controlled param to enable ContentRecommenderMixerAdaptor\n   */\n  object ContentRecommenderMixerAdaptorDecider\n      extends BooleanDeciderParam(DeciderKey.enableContentRecommenderMixerAdaptor)\n\n  /**\n   * Decider controlled param to enable GenericCandidateAdaptor\n   */\n  object GenericCandidateAdaptorDecider\n      extends BooleanDeciderParam(DeciderKey.enableGenericCandidateAdaptor)\n\n  /**\n   * Decider controlled param to enable dark traffic to ContentMixer for Trip Geo Tweets\n   */\n  object TripGeoTweetContentMixerDarkTrafficDecider\n      extends BooleanDeciderParam(DeciderKey.enableTripGeoTweetContentMixerDarkTraffic)\n\n  /*\n   *  Decider controlled param to enable Pop Geo Tweets\n   * */\n  object TrendsCandidateDecider extends BooleanDeciderParam(DeciderKey.enableTrendsTweets)\n\n  /*\n   *  Decider controlled param to enable INS Traffic\n   **/\n  object EnableInsTrafficDecider extends BooleanDeciderParam(DeciderKey.enableInsTraffic)\n\n  /**\n   * Param to enable assigning pushcap with ML predictions (read from MH table).\n   * Disabling will fallback to only use heuristics and default values.\n   */\n  object EnableModelBasedPushcapAssignments\n      extends BooleanDeciderParam(DeciderKey.enableModelBasedPushcapAssignments)\n\n  /**\n   * Param to enable twhin user engagement feature hydration\n   */\n  object EnableTwHINUserEngagementFeaturesHydration\n      extends BooleanDeciderParam(DeciderKey.enableTwHINUserEngagementFeaturesHydration)\n\n  /**\n   * Param to enable twhin user follow feature hydration\n   */\n  object EnableTwHINUserFollowFeaturesHydration\n      extends BooleanDeciderParam(DeciderKey.enableTwHINUserFollowFeaturesHydration)\n\n  /**\n   * Param to enable twhin author follow feature hydration\n   */\n  object EnableTwHINAuthorFollowFeaturesHydration\n      extends BooleanDeciderParam(DeciderKey.enableTwHINAuthorFollowFeaturesHydration)\n\n  /**\n   * Param to enable calls to the IsTweetTranslatable strato column\n   */\n  object EnableIsTweetTranslatableCheck\n      extends BooleanDeciderParam(DeciderKey.enableIsTweetTranslatable)\n\n  /**\n   * Decider controlled param to enable mr tweet simcluster feature set hydration\n   */\n  object EnableMrTweetSimClusterFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrTweetSimClusterFeatureSet)\n\n  /**\n   * Decider controlled param to enable real graph v2 feature set hydration\n   */\n  object EnableRealGraphV2FeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableRealGraphV2FeatureHydration)\n\n  /**\n   * Decider controlled param to enable Tweet BeT feature set hydration\n   */\n  object EnableTweetBeTFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableTweetBeTFeatureHydration)\n\n  /**\n   * Decider controlled param to enable mr user tweet topic feature set hydration\n   */\n  object EnableMrOfflineUserTweetTopicAggregateHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrOfflineUserTweetTopicAggregate)\n\n  /**\n   * Decider controlled param to enable mr tweet simcluster feature set hydration\n   */\n  object EnableMrOfflineUserTweetSimClusterAggregateHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrOfflineUserTweetSimClusterAggregate)\n\n  /**\n   * Decider controlled param to enable user send time features\n   */\n  object EnableUserSendTimeFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableUserSendTimeFeatureHydration)\n\n  /**\n   * Decider controlled param to enable mr user utc send time aggregate features\n   */\n  object EnableMrUserUtcSendTimeAggregateFeaturesHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrUserUtcSendTimeAggregateFeaturesHydration)\n\n  /**\n   * Decider controlled param to enable mr user local send time aggregate features\n   */\n  object EnableMrUserLocalSendTimeAggregateFeaturesHydration\n      extends BooleanDeciderParam(DeciderKey.enableMrUserLocalSendTimeAggregateFeaturesHydration)\n\n  /**\n   * Decider controlled param to enable BQML report model predictions for F1 tweets\n   */\n  object EnableBqmlReportModelPredictionForF1Tweets\n      extends BooleanDeciderParam(DeciderKey.enableBqmlReportModelPredictionForF1Tweets)\n\n  /**\n   * Decider controlled param to enable user Twhin embedding feature hydration\n   */\n  object EnableUserTwhinEmbeddingFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableUserTwhinEmbeddingFeatureHydration)\n\n  /**\n   * Decider controlled param to enable author follow Twhin embedding feature hydration\n   */\n  object EnableAuthorFollowTwhinEmbeddingFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableAuthorFollowTwhinEmbeddingFeatureHydration)\n\n  object EnableScribingMLFeaturesAsDataRecord\n      extends BooleanDeciderParam(DeciderKey.enableScribingMLFeaturesAsDataRecord)\n\n  /**\n   * Decider controlled param to enable feature hydration for Verified related feature\n   */\n  object EnableAuthorVerifiedFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableAuthorVerifiedFeatureHydration)\n\n  /**\n   * Decider controlled param to enable feature hydration for creator subscription related feature\n   */\n  object EnableAuthorCreatorSubscriptionFeatureHydration\n      extends BooleanDeciderParam(DeciderKey.enableAuthorCreatorSubscriptionFeatureHydration)\n\n  /**\n   * Decider controlled param to direct MH+Memcache hydration for the UserFeaturesDataset\n   */\n  object EnableDirectHydrationForUserFeatures\n      extends BooleanDeciderParam(DeciderKey.enableDirectHydrationForUserFeatures)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/params/PushServiceTunableKeys.scala",
    "content": "package com.twitter.frigate.pushservice.params\n\nimport com.twitter.util.tunable.TunableMap\n\nobject PushServiceTunableKeys {\n  final val IbisQpsLimitTunableKey = TunableMap.Key[Int](\"ibis2.qps.limit\")\n  final val NtabQpsLimitTunableKey = TunableMap.Key[Int](\"ntab.qps.limit\")\n  final val TweetPerspectiveStoreQpsLimit = TunableMap.Key[Int](\"tweetperspective.qps.limit\")\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/params/ShardParams.scala",
    "content": "package com.twitter.frigate.pushservice.params\n\ncase class ShardParams(numShards: Int, shardId: Int)\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/BigFilteringEpsilonGreedyExplorationPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\n/*\n * A predicate for epsilon-greedy exploration;\n * We defined it as a candidate level predicate to avoid changing the predicate and scribing pipeline,\n * but it is actually a post-ranking target level predicate:\n *  if a target user IS ENABLED for \\epsilon-greedy exploration,\n *  then with probability epsilon, the user (and thus all candidates) will be blocked\n */\nobject BigFilteringEpsilonGreedyExplorationPredicate {\n\n  val name = \"BigFilteringEpsilonGreedyExplorationPredicate\"\n\n  private def shouldFilterBasedOnEpsilonGreedyExploration(\n    target: Target\n  ): Boolean = {\n    val seed = KeyHasher.FNV1A_64.hashKey(s\"${target.targetId}\".getBytes(\"UTF8\"))\n    val hashKey = KeyHasher.FNV1A_64\n      .hashKey(\n        s\"${Trace.id.traceId.toString}:${seed.toString}\".getBytes(\"UTF8\")\n      )\n\n    math.abs(hashKey).toDouble / Long.MaxValue <\n      target.params(PushFeatureSwitchParams.MrRequestScribingEpsGreedyExplorationRatio)\n  }\n\n  def apply()(implicit statsReceiver: StatsReceiver): NamedPredicate[PushCandidate] = {\n    val stats = statsReceiver.scope(s\"predicate_$name\")\n\n    val enabledForEpsilonGreedyCounter = stats.counter(\"enabled_for_eps_greedy\")\n\n    new Predicate[PushCandidate] {\n      def apply(candidates: Seq[PushCandidate]): Future[Seq[Boolean]] = {\n        val results = candidates.map { candidate =>\n          if (!candidate.target.skipFilters && candidate.target.params(\n              PushFeatureSwitchParams.EnableMrRequestScribingForEpsGreedyExploration)) {\n            enabledForEpsilonGreedyCounter.incr()\n            !shouldFilterBasedOnEpsilonGreedyExploration(candidate.target)\n          } else {\n            true\n          }\n        }\n        Future.value(results)\n      }\n    }.withStats(stats)\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/BqmlHealthModelPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.abuse.detection.scoring.thriftscala.TweetScoringRequest\nimport com.twitter.abuse.detection.scoring.thriftscala.TweetScoringResponse\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.ml.HealthFeatureGetter\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.params.PushMLModel\nimport com.twitter.util.Future\nimport com.twitter.frigate.pushservice.util.CandidateUtil\nimport com.twitter.frigate.thriftscala.UserMediaRepresentation\nimport com.twitter.hss.api.thriftscala.UserHealthSignalResponse\nimport com.twitter.storehaus.ReadableStore\n\nobject BqmlHealthModelPredicates {\n\n  def healthModelOonPredicate(\n    bqmlHealthModelScorer: PushMLModelScorer,\n    producerMediaRepresentationStore: ReadableStore[Long, UserMediaRepresentation],\n    userHealthScoreStore: ReadableStore[Long, UserHealthSignalResponse],\n    tweetHealthScoreStore: ReadableStore[TweetScoringRequest, TweetScoringResponse]\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[\n    PushCandidate with TweetCandidate with RecommendationType with TweetAuthor\n  ] = {\n    val name = \"bqml_health_model_based_predicate\"\n    val scopedStatsReceiver = stats.scope(name)\n\n    val allCandidatesCounter = scopedStatsReceiver.counter(\"all_candidates\")\n    val oonCandidatesCounter = scopedStatsReceiver.counter(\"oon_candidates\")\n    val filteredOonCandidatesCounter =\n      scopedStatsReceiver.counter(\"filtered_oon_candidates\")\n    val emptyScoreCandidatesCounter = scopedStatsReceiver.counter(\"empty_score_candidates\")\n    val healthScoreStat = scopedStatsReceiver.stat(\"health_model_dist\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate with RecommendationType =>\n        val target = candidate.target\n        val isOonCandidate = RecTypes.isOutOfNetworkTweetRecType(candidate.commonRecType) ||\n          RecTypes.outOfNetworkTopicTweetTypes.contains(candidate.commonRecType)\n\n        lazy val enableBqmlHealthModelPredicateParam =\n          target.params(PushFeatureSwitchParams.EnableBqmlHealthModelPredicateParam)\n        lazy val enableBqmlHealthModelPredictionForInNetworkCandidates =\n          target.params(\n            PushFeatureSwitchParams.EnableBqmlHealthModelPredictionForInNetworkCandidatesParam)\n        lazy val bqmlHealthModelPredicateFilterThresholdParam =\n          target.params(PushFeatureSwitchParams.BqmlHealthModelPredicateFilterThresholdParam)\n        lazy val healthModelId = target.params(PushFeatureSwitchParams.BqmlHealthModelTypeParam)\n        lazy val enableBqmlHealthModelScoreHistogramParam =\n          target.params(PushFeatureSwitchParams.EnableBqmlHealthModelScoreHistogramParam)\n        val healthModelScoreFeature = \"bqml_health_model_score\"\n\n        val histogramBinSize = 0.05\n        lazy val healthCandidateScoreHistogramCounters =\n          bqmlHealthModelScorer.getScoreHistogramCounters(\n            scopedStatsReceiver,\n            \"health_score_histogram\",\n            histogramBinSize)\n\n        candidate match {\n          case candidate: PushCandidate with TweetAuthor with TweetAuthorDetails\n              if enableBqmlHealthModelPredicateParam && (isOonCandidate || enableBqmlHealthModelPredictionForInNetworkCandidates) =>\n            HealthFeatureGetter\n              .getFeatures(\n                candidate,\n                producerMediaRepresentationStore,\n                userHealthScoreStore,\n                Some(tweetHealthScoreStore))\n              .flatMap { healthFeatures =>\n                allCandidatesCounter.incr()\n                candidate.mergeFeatures(healthFeatures)\n\n                val healthModelScoreFutOpt =\n                  if (candidate.numericFeatures.contains(healthModelScoreFeature)) {\n                    Future.value(candidate.numericFeatures.get(healthModelScoreFeature))\n                  } else\n                    bqmlHealthModelScorer.singlePredicationForModelVersion(\n                      healthModelId,\n                      candidate\n                    )\n\n                candidate.populateQualityModelScore(\n                  PushMLModel.HealthNsfwProbability,\n                  healthModelId,\n                  healthModelScoreFutOpt\n                )\n\n                healthModelScoreFutOpt.map {\n                  case Some(healthModelScore) =>\n                    healthScoreStat.add((healthModelScore * 10000).toFloat)\n                    if (enableBqmlHealthModelScoreHistogramParam) {\n                      healthCandidateScoreHistogramCounters(\n                        math.ceil(healthModelScore / histogramBinSize).toInt).incr()\n                    }\n\n                    if (CandidateUtil.shouldApplyHealthQualityFilters(\n                        candidate) && isOonCandidate) {\n                      oonCandidatesCounter.incr()\n                      val threshold = bqmlHealthModelPredicateFilterThresholdParam\n                      candidate.cachePredicateInfo(\n                        name,\n                        healthModelScore,\n                        threshold,\n                        healthModelScore > threshold)\n                      if (healthModelScore > threshold) {\n                        filteredOonCandidatesCounter.incr()\n                        false\n                      } else true\n                    } else true\n                  case _ =>\n                    emptyScoreCandidatesCounter.incr()\n                    true\n                }\n              }\n          case _ => Future.True\n        }\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/BqmlQualityModelPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.params.PushConstants.TweetMediaEmbeddingBQKeyIds\nimport com.twitter.frigate.pushservice.params.PushMLModel\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.util.CandidateUtil\nimport com.twitter.util.Future\nimport com.twitter.frigate.pushservice.util.CandidateUtil._\n\nobject BqmlQualityModelPredicates {\n\n  def ingestExtraFeatures(cand: PushCandidate): Unit = {\n    val tagsCRCountFeature = \"tagsCR_count\"\n    val hasPushOpenOrNtabClickFeature = \"has_PushOpenOrNtabClick\"\n    val onlyPushOpenOrNtabClickFeature = \"only_PushOpenOrNtabClick\"\n    val firstTweetMediaEmbeddingFeature = \"media_embedding_0\"\n    val tweetMediaEmbeddingFeature =\n      \"media.mediaunderstanding.media_embeddings.twitter_clip_as_sparse_continuous_feature\"\n\n    if (!cand.numericFeatures.contains(tagsCRCountFeature)) {\n      cand.numericFeatures(tagsCRCountFeature) = getTagsCRCount(cand)\n    }\n    if (!cand.booleanFeatures.contains(hasPushOpenOrNtabClickFeature)) {\n      cand.booleanFeatures(hasPushOpenOrNtabClickFeature) = isRelatedToMrTwistlyCandidate(cand)\n    }\n    if (!cand.booleanFeatures.contains(onlyPushOpenOrNtabClickFeature)) {\n      cand.booleanFeatures(onlyPushOpenOrNtabClickFeature) = isMrTwistlyCandidate(cand)\n    }\n    if (!cand.numericFeatures.contains(firstTweetMediaEmbeddingFeature)) {\n      val tweetMediaEmbedding = cand.sparseContinuousFeatures\n        .getOrElse(tweetMediaEmbeddingFeature, Map.empty[String, Double])\n      Seq.range(0, TweetMediaEmbeddingBQKeyIds.size).foreach { i =>\n        cand.numericFeatures(s\"media_embedding_$i\") =\n          tweetMediaEmbedding.getOrElse(TweetMediaEmbeddingBQKeyIds(i).toString, 0.0)\n      }\n    }\n  }\n\n  def BqmlQualityModelOonPredicate(\n    bqmlQualityModelScorer: PushMLModelScorer\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[\n    PushCandidate with TweetCandidate with RecommendationType\n  ] = {\n\n    val name = \"bqml_quality_model_based_predicate\"\n    val scopedStatsReceiver = stats.scope(name)\n    val oonCandidatesCounter = scopedStatsReceiver.counter(\"oon_candidates\")\n    val inCandidatesCounter = scopedStatsReceiver.counter(\"in_candidates\")\n    val filteredOonCandidatesCounter =\n      scopedStatsReceiver.counter(\"filtered_oon_candidates\")\n    val bucketedCandidatesCounter = scopedStatsReceiver.counter(\"bucketed_oon_candidates\")\n    val emptyScoreCandidatesCounter = scopedStatsReceiver.counter(\"empty_score_candidates\")\n    val histogramBinSize = 0.05\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate with RecommendationType =>\n        val target = candidate.target\n        val crt = candidate.commonRecType\n        val isOonCandidate = RecTypes.isOutOfNetworkTweetRecType(crt) ||\n          RecTypes.outOfNetworkTopicTweetTypes.contains(crt)\n\n        lazy val enableBqmlQualityModelScoreHistogramParam =\n          target.params(PushFeatureSwitchParams.EnableBqmlQualityModelScoreHistogramParam)\n\n        lazy val qualityCandidateScoreHistogramCounters =\n          bqmlQualityModelScorer.getScoreHistogramCounters(\n            scopedStatsReceiver,\n            \"quality_score_histogram\",\n            histogramBinSize)\n\n        if (CandidateUtil.shouldApplyHealthQualityFilters(candidate) && (isOonCandidate || target\n            .params(PushParams.EnableBqmlReportModelPredictionForF1Tweets))\n          && target.params(PushFeatureSwitchParams.EnableBqmlQualityModelPredicateParam)) {\n          ingestExtraFeatures(candidate)\n\n          lazy val shouldFilterFutSeq =\n            target\n              .params(PushFeatureSwitchParams.BqmlQualityModelBucketModelIdListParam)\n              .zip(target.params(PushFeatureSwitchParams.BqmlQualityModelBucketThresholdListParam))\n              .map {\n                case (modelId, bucketThreshold) =>\n                  val scoreFutOpt =\n                    bqmlQualityModelScorer.singlePredicationForModelVersion(modelId, candidate)\n\n                  candidate.populateQualityModelScore(\n                    PushMLModel.FilteringProbability,\n                    modelId,\n                    scoreFutOpt\n                  )\n\n                  if (isOonCandidate) {\n                    oonCandidatesCounter.incr()\n                    scoreFutOpt.map {\n                      case Some(score) =>\n                        if (score >= bucketThreshold) {\n                          bucketedCandidatesCounter.incr()\n                          if (modelId == target.params(\n                              PushFeatureSwitchParams.BqmlQualityModelTypeParam)) {\n                            if (enableBqmlQualityModelScoreHistogramParam) {\n                              val scoreHistogramBinId =\n                                math.ceil(score / histogramBinSize).toInt\n                              qualityCandidateScoreHistogramCounters(scoreHistogramBinId).incr()\n                            }\n                            if (score >= target.params(\n                                PushFeatureSwitchParams.BqmlQualityModelPredicateThresholdParam)) {\n                              filteredOonCandidatesCounter.incr()\n                              true\n                            } else false\n                          } else false\n                        } else false\n                      case _ =>\n                        emptyScoreCandidatesCounter.incr()\n                        false\n                    }\n                  } else {\n                    inCandidatesCounter.incr()\n                    Future.False\n                  }\n              }\n\n          Future.collect(shouldFilterFutSeq).flatMap { shouldFilterSeq =>\n            if (shouldFilterSeq.contains(true)) {\n              Future.False\n            } else Future.True\n          }\n        } else Future.True\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/CaretFeedbackHistoryFilter.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.frigate.common.base.TargetUser\nimport com.twitter.frigate.common.candidate.CaretFeedbackHistory\nimport com.twitter.frigate.common.candidate.TargetABDecider\nimport com.twitter.frigate.common.util.MrNtabCopyObjects\nimport com.twitter.notificationservice.thriftscala.CaretFeedbackDetails\nimport com.twitter.notificationservice.thriftscala.GenericNotificationMetadata\nimport com.twitter.notificationservice.thriftscala.GenericType\n\nobject CaretFeedbackHistoryFilter {\n\n  def caretFeedbackHistoryFilter(\n    categories: Seq[String]\n  ): TargetUser with TargetABDecider with CaretFeedbackHistory => Seq[CaretFeedbackDetails] => Seq[\n    CaretFeedbackDetails\n  ] = { target => caretFeedbackDetailsSeq =>\n    caretFeedbackDetailsSeq.filter { caretFeedbackDetails =>\n      caretFeedbackDetails.genericNotificationMetadata match {\n        case Some(genericNotificationMetadata) =>\n          isFeedbackSupportedGenericType(genericNotificationMetadata)\n        case None => false\n      }\n    }\n  }\n\n  private def filterCriteria(\n    caretFeedbackDetails: CaretFeedbackDetails,\n    genericTypes: Seq[GenericType]\n  ): Boolean = {\n    caretFeedbackDetails.genericNotificationMetadata match {\n      case Some(genericNotificationMetadata) =>\n        genericTypes.contains(genericNotificationMetadata.genericType)\n      case None => false\n    }\n  }\n\n  def caretFeedbackHistoryFilterByGenericType(\n    genericTypes: Seq[GenericType]\n  ): TargetUser with TargetABDecider with CaretFeedbackHistory => Seq[CaretFeedbackDetails] => Seq[\n    CaretFeedbackDetails\n  ] = { target => caretFeedbackDetailsSeq =>\n    caretFeedbackDetailsSeq.filter { caretFeedbackDetails =>\n      filterCriteria(caretFeedbackDetails, genericTypes)\n    }\n  }\n\n  def caretFeedbackHistoryFilterByGenericTypeDenyList(\n    genericTypes: Seq[GenericType]\n  ): TargetUser with TargetABDecider with CaretFeedbackHistory => Seq[CaretFeedbackDetails] => Seq[\n    CaretFeedbackDetails\n  ] = { target => caretFeedbackDetailsSeq =>\n    caretFeedbackDetailsSeq.filterNot { caretFeedbackDetails =>\n      filterCriteria(caretFeedbackDetails, genericTypes)\n    }\n  }\n\n  def caretFeedbackHistoryFilterByRefreshableType(\n    refreshableTypes: Set[Option[String]]\n  ): TargetUser with TargetABDecider with CaretFeedbackHistory => Seq[CaretFeedbackDetails] => Seq[\n    CaretFeedbackDetails\n  ] = { target => caretFeedbackDetailsSeq =>\n    caretFeedbackDetailsSeq.filter { caretFeedbackDetails =>\n      caretFeedbackDetails.genericNotificationMetadata match {\n        case Some(genericNotificationMetadata) =>\n          refreshableTypes.contains(genericNotificationMetadata.refreshableType)\n        case None => false\n      }\n    }\n  }\n\n  def caretFeedbackHistoryFilterByRefreshableTypeDenyList(\n    refreshableTypes: Set[Option[String]]\n  ): TargetUser with TargetABDecider with CaretFeedbackHistory => Seq[CaretFeedbackDetails] => Seq[\n    CaretFeedbackDetails\n  ] = { target => caretFeedbackDetailsSeq =>\n    caretFeedbackDetailsSeq.filter { caretFeedbackDetails =>\n      caretFeedbackDetails.genericNotificationMetadata match {\n        case Some(genericNotificationMetadata) =>\n          !refreshableTypes.contains(genericNotificationMetadata.refreshableType)\n        case None => true\n      }\n    }\n  }\n\n  private def isFeedbackSupportedGenericType(\n    notificationMetadata: GenericNotificationMetadata\n  ): Boolean = {\n    val genericNotificationTypeName =\n      (notificationMetadata.genericType, notificationMetadata.refreshableType) match {\n        case (GenericType.RefreshableNotification, Some(refreshableType)) => refreshableType\n        case _ => notificationMetadata.genericType.name\n      }\n\n    MrNtabCopyObjects.AllNtabCopyTypes\n      .flatMap(_.refreshableType)\n      .contains(genericNotificationTypeName)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/CasLockPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.CasLock\nimport com.twitter.frigate.common.util.CasSuccess\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\n\nobject CasLockPredicate {\n  def apply(\n    casLock: CasLock,\n    expiryDuration: Duration\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val stats = statsReceiver.scope(\"predicate_addcaslock_for_candidate\")\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        if (candidate.target.pushContext.exists(_.darkWrite.exists(_ == true))) {\n          Future.True\n        } else if (candidate.commonRecType == CommonRecommendationType.MagicFanoutSportsEvent) {\n          Future.True\n        } else {\n          candidate.target.history flatMap { h =>\n            val now = candidate.createdAt\n            val expiry = now + expiryDuration\n            val oldTimestamp = h.lastNotificationTime map {\n              _.inSeconds\n            } getOrElse 0\n            casLock.cas(candidate.target.targetId, oldTimestamp, now.inSeconds, expiry) map {\n              casResult =>\n                stats.counter(s\"cas_$casResult\").incr()\n                casResult == CasSuccess\n            }\n          }\n        }\n      }\n      .withStats(stats)\n      .withName(\"add_cas_lock\")\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/CrtDeciderPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.decider.Decider\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\n\nobject CrtDeciderPredicate {\n  val name = \"crt_decider\"\n  def apply(\n    decider: Decider\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    Predicate\n      .from { (candidate: PushCandidate) =>\n        val prefix = \"frigate_pushservice_\"\n        val deciderKey = prefix + candidate.commonRecType\n        decider.feature(deciderKey).isAvailable\n      }\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/DiscoverTwitterPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TargetUser\nimport com.twitter.frigate.common.candidate.FrigateHistory\nimport com.twitter.frigate.common.history.History\nimport com.twitter.frigate.common.predicate.FrigateHistoryFatiguePredicate\nimport com.twitter.frigate.common.predicate.{FatiguePredicate => TargetFatiguePredicate}\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\n\nobject DiscoverTwitterPredicate {\n\n  /**\n   * Predicate used to determine if a minimum duration has elapsed since the last MR push\n   * for a CRT to be valid.\n   * @param name            Identifier of the caller (used for stats)\n   * @param intervalParam   The minimum duration interval\n   * @param stats           StatsReceiver\n   * @return                Target Predicate\n   */\n  def minDurationElapsedSinceLastMrPushPredicate(\n    name: String,\n    intervalParam: Param[Duration],\n    stats: StatsReceiver\n  ): Predicate[Target] =\n    Predicate\n      .fromAsync { target: Target =>\n        val interval =\n          target.params(intervalParam)\n        FrigateHistoryFatiguePredicate(\n          minInterval = interval,\n          getSortedHistory = { h: History =>\n            val magicRecsOnlyHistory =\n              TargetFatiguePredicate.magicRecsPushOnlyFilter(h.sortedPushDmHistory)\n            TargetFatiguePredicate.magicRecsNewUserPlaybookPushFilter(magicRecsOnlyHistory)\n          }\n        ).flatContraMap { target: TargetUser with FrigateHistory =>\n            target.history\n          }.apply(Seq(target)).map {\n            _.head\n          }\n      }.withStats(stats.scope(s\"${name}_predicate_mr_push_min_interval\"))\n      .withName(s\"${name}_predicate_mr_push_min_interval\")\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/FatiguePredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.predicate.FatiguePredicate._\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.{NotificationDisplayLocation => DisplayLocation}\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.util.Duration\n\nobject FatiguePredicate {\n\n  /**\n   * Predicate that operates on a candidate, and applies custom fatigue rules for the slice of history only\n   * corresponding to a given rec type.\n   *\n   * @param interval\n   * @param maxInInterval\n   * @param minInterval\n   * @param recommendationType\n   * @param statsReceiver\n   * @return\n   */\n  def recTypeOnly(\n    interval: Duration,\n    maxInInterval: Int,\n    minInterval: Duration,\n    recommendationType: CommonRecommendationType,\n    notificationDisplayLocation: DisplayLocation = DisplayLocation.PushToMobileDevice\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    build(\n      interval = interval,\n      maxInInterval = maxInInterval,\n      minInterval = minInterval,\n      filterHistory = recOnlyFilter(recommendationType),\n      notificationDisplayLocation = notificationDisplayLocation\n    ).flatContraMap { candidate: PushCandidate => candidate.target.history }\n      .withStats(statsReceiver.scope(s\"predicate_${recTypeOnlyFatigue}\"))\n      .withName(recTypeOnlyFatigue)\n  }\n\n  /**\n   * Predicate that operates on a candidate, and applies custom fatigue rules for the slice of history only\n   * corresponding to specified rec types\n   *\n   * @param interval\n   * @param maxInInterval\n   * @param minInterval\n   * @param statsReceiver\n   * @return\n   */\n  def recTypeSetOnly(\n    interval: Duration,\n    maxInInterval: Int,\n    minInterval: Duration,\n    recTypes: Set[CommonRecommendationType],\n    notificationDisplayLocation: DisplayLocation = DisplayLocation.PushToMobileDevice\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"rec_type_set_fatigue\"\n    build(\n      interval = interval,\n      maxInInterval = maxInInterval,\n      minInterval = minInterval,\n      filterHistory = recTypesOnlyFilter(recTypes),\n      notificationDisplayLocation = notificationDisplayLocation\n    ).flatContraMap { candidate: PushCandidate => candidate.target.history }\n      .withStats(statsReceiver.scope(s\"${name}_predicate\"))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/HealthPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.abuse.detection.scoring.thriftscala.TweetScoringRequest\nimport com.twitter.abuse.detection.scoring.thriftscala.TweetScoringResponse\nimport com.twitter.abuse.detection.scoring.thriftscala.{Model => TweetHealthModel}\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.NsfwTextDetectionModel\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.util.CandidateHydrationUtil\nimport com.twitter.frigate.pushservice.util.CandidateUtil\nimport com.twitter.frigate.pushservice.util.MediaAnnotationsUtil\nimport com.twitter.frigate.thriftscala.UserMediaRepresentation\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.hss.api.thriftscala.UserHealthSignal._\nimport com.twitter.hss.api.thriftscala.SignalValue\nimport com.twitter.hss.api.thriftscala.UserHealthSignalResponse\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\nobject HealthPredicates {\n\n  private val NsfwTextDetectionModelMap: Map[NsfwTextDetectionModel.Value, TweetHealthModel] =\n    Map(\n      NsfwTextDetectionModel.ProdModel -> TweetHealthModel.PnsfwTweetText,\n      NsfwTextDetectionModel.RetrainedModel -> TweetHealthModel.ExperimentalHealthModelScore1,\n    )\n\n  private def tweetIsSupportedLanguage(\n    candidate: PushCandidate,\n    supportedLanguages: Set[String]\n  ): Boolean = {\n    val tweetLanguage =\n      candidate.categoricalFeatures.getOrElse(\"RecTweet.TweetyPieResult.Language\", \"\")\n    supportedLanguages.contains(tweetLanguage)\n  }\n\n  def tweetHealthSignalScorePredicate(\n    tweetHealthScoreStore: ReadableStore[TweetScoringRequest, TweetScoringResponse],\n    applyToQuoteTweet: Boolean = false\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetCandidate with TweetDetails] = {\n    val name = \"tweet_health_signal_store_applyToQuoteTweet_\" + applyToQuoteTweet.toString\n    val scopedStatsReceiver = stats.scope(name)\n    val numCandidatesStats = scopedStatsReceiver.scope(\"num_candidates\")\n    val numCandidatesMediaNsfwScoreStats = numCandidatesStats.scope(\"media_nsfw_score\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate with TweetDetails =>\n        numCandidatesStats.counter(\"all\").incr()\n        val target = candidate.target\n        val tweetIdOpt = if (!applyToQuoteTweet) {\n          Some(candidate.tweetId)\n        } else candidate.tweetyPieResult.flatMap(_.quotedTweet.map(_.id))\n\n        tweetIdOpt match {\n          case Some(tweetId) =>\n            val pMediaNsfwRequest =\n              TweetScoringRequest(tweetId, TweetHealthModel.ExperimentalHealthModelScore4)\n            tweetHealthScoreStore.get(pMediaNsfwRequest).map {\n              case Some(tweetScoringResponse) =>\n                numCandidatesMediaNsfwScoreStats.counter(\"non_empty\").incr()\n                val pMediaNsfwScore = tweetScoringResponse.score\n\n                if (!applyToQuoteTweet) {\n                  candidate\n                    .cacheExternalScore(\"NsfwMediaProbability\", Future.value(Some(pMediaNsfwScore)))\n                }\n\n                val pMediaNsfwShouldBucket =\n                  pMediaNsfwScore > target.params(\n                    PushFeatureSwitchParams.PnsfwTweetMediaBucketingThreshold)\n                if (CandidateUtil.shouldApplyHealthQualityFilters(\n                    candidate) && pMediaNsfwShouldBucket) {\n                  numCandidatesMediaNsfwScoreStats.counter(\"bucketed\").incr()\n                  if (target.params(PushFeatureSwitchParams.PnsfwTweetMediaFilterOonOnly)\n                    && !RecTypes.isOutOfNetworkTweetRecType(candidate.commonRecType)) {\n                    true\n                  } else {\n                    val pMediaNsfwScoreThreshold =\n                      if (applyToQuoteTweet)\n                        target.params(PushFeatureSwitchParams.PnsfwQuoteTweetThreshold)\n                      else if (candidate.hasPhoto)\n                        target.params(PushFeatureSwitchParams.PnsfwTweetImageThreshold)\n                      else target.params(PushFeatureSwitchParams.PnsfwTweetMediaThreshold)\n                    candidate.cachePredicateInfo(\n                      name + \"_nsfwMedia\",\n                      pMediaNsfwScore,\n                      pMediaNsfwScoreThreshold,\n                      pMediaNsfwScore > pMediaNsfwScoreThreshold)\n                    if (pMediaNsfwScore > pMediaNsfwScoreThreshold) {\n                      numCandidatesMediaNsfwScoreStats.counter(\"filtered\").incr()\n                      false\n                    } else true\n                  }\n                } else true\n              case _ =>\n                numCandidatesMediaNsfwScoreStats.counter(\"empty\").incr()\n                if (candidate.hasPhoto || candidate.hasVideo) {\n                  numCandidatesMediaNsfwScoreStats.counter(\"media_tweet_with_empty_score\").incr()\n                }\n                true\n            }\n          case _ => Future.True\n        }\n      }\n      .withStats(stats.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def healthSignalScoreSpammyTweetPredicate(\n    tweetHealthScoreStore: ReadableStore[TweetScoringRequest, TweetScoringResponse]\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetCandidate with TweetDetails] = {\n    val name = \"health_signal_store_spammy_tweet\"\n    val statsScope = stats.scope(name)\n    val allCandidatesCounter = statsScope.counter(\"all_candidates\")\n    val eligibleCandidatesCounter = statsScope.counter(\"eligible_candidates\")\n    val oonCandidatesCounter = statsScope.counter(\"oon_candidates\")\n    val inCandidatesCounter = statsScope.counter(\"in_candidates\")\n    val bucketedCandidatesCounter = statsScope.counter(\"num_bucketed\")\n    val nonEmptySpamScoreCounter = statsScope.counter(\"non_empty_spam_score\")\n    val filteredOonCandidatesCounter = statsScope.counter(\"num_filtered_oon\")\n    val filteredInCandidatesCounter = statsScope.counter(\"num_filtered_in\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate with TweetDetails =>\n        allCandidatesCounter.incr()\n        val crt = candidate.commonRecType\n        val isOonCandidate = RecTypes.isOutOfNetworkTweetRecType(crt) ||\n          RecTypes.outOfNetworkTopicTweetTypes.contains(crt)\n        if (isOonCandidate) {\n          oonCandidatesCounter.incr()\n        }\n        val target = candidate.target\n        if (target.params(PushFeatureSwitchParams.EnableSpammyTweetFilter)) {\n          eligibleCandidatesCounter.incr()\n          val tweetSpamScore =\n            TweetScoringRequest(candidate.tweetId, TweetHealthModel.SpammyTweetContent)\n          tweetHealthScoreStore.get(tweetSpamScore).map {\n            case (Some(tweetScoringResponse)) =>\n              nonEmptySpamScoreCounter.incr()\n              val candidateSpamScore = tweetScoringResponse.score\n\n              candidate\n                .cacheExternalScore(\"SpammyTweetScore\", Future.value(Some(candidateSpamScore)))\n\n              val tweetSpamShouldBucket =\n                candidateSpamScore > target.params(\n                  PushFeatureSwitchParams.SpammyTweetBucketingThreshold)\n              if (CandidateUtil.shouldApplyHealthQualityFilters(\n                  candidate) && tweetSpamShouldBucket) {\n                bucketedCandidatesCounter.incr()\n                if (isOonCandidate) {\n                  val spamScoreThreshold =\n                    target.params(PushFeatureSwitchParams.SpammyTweetOonThreshold)\n                  if (candidateSpamScore > spamScoreThreshold) {\n                    filteredOonCandidatesCounter.incr()\n                    false\n                  } else true\n                } else {\n                  inCandidatesCounter.incr()\n                  val spamScoreThreshold =\n                    target.params(PushFeatureSwitchParams.SpammyTweetInThreshold)\n                  if (candidateSpamScore > spamScoreThreshold) {\n                    filteredInCandidatesCounter.incr()\n                    false\n                  } else true\n                }\n              } else true\n            case _ => true\n          }\n        } else Future.True\n      }\n      .withStats(stats.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def healthSignalScorePnsfwTweetTextPredicate(\n    tweetHealthScoreStore: ReadableStore[TweetScoringRequest, TweetScoringResponse]\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetCandidate] = {\n    val name = \"health_signal_store_pnsfw_tweet_text\"\n    val statsScope = stats.scope(name)\n    val allCandidatesCounter = statsScope.counter(\"all_candidates\")\n    val nonEmptyNsfwTextScoreNum = statsScope.counter(\"non_empty_nsfw_text_score\")\n    val filteredCounter = statsScope.counter(\"num_filtered\")\n    val lowScoreCounter = statsScope.counter(\"low_score_count\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate =>\n        val target = candidate.target\n        val predEnabled =\n          target.params(PushFeatureSwitchParams.EnableHealthSignalStorePnsfwTweetTextPredicate)\n        if (CandidateUtil.shouldApplyHealthQualityFilters(\n            candidate) && predEnabled && tweetIsSupportedLanguage(candidate, Set(\"\"))) {\n          allCandidatesCounter.incr()\n          val pnsfwTextRequest =\n            TweetScoringRequest(candidate.tweetId, TweetHealthModel.PnsfwTweetText)\n          tweetHealthScoreStore.get(pnsfwTextRequest).flatMap {\n            case Some(tweetScoringResponse) => {\n              nonEmptyNsfwTextScoreNum.incr()\n              if (tweetScoringResponse.score < 1e-8) {\n                lowScoreCounter.incr()\n              }\n\n              candidate\n                .cacheExternalScore(\n                  \"NsfwTextProbability-en\",\n                  Future.value(Some(tweetScoringResponse.score)))\n              val threshold = target.params(PushFeatureSwitchParams.PnsfwTweetTextThreshold)\n              candidate.cachePredicateInfo(\n                name,\n                tweetScoringResponse.score,\n                threshold,\n                tweetScoringResponse.score > threshold)\n              if (tweetScoringResponse.score > threshold) {\n                filteredCounter.incr()\n                Future.False\n              } else Future.True\n            }\n            case _ => Future.True\n          }\n        } else Future.True\n      }\n      .withStats(stats.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def healthSignalScoreMultilingualPnsfwTweetTextPredicate(\n    tweetHealthScoreStore: ReadableStore[TweetScoringRequest, TweetScoringResponse]\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetCandidate] = {\n    val name = \"health_signal_store_multilingual_pnsfw_tweet_text\"\n    val statsScope = stats.scope(name)\n\n    val allLanguagesIdentifier = \"all\"\n    val languagesSelectedForStats =\n      Set(\"\") + allLanguagesIdentifier\n\n    val candidatesCounterMap: Map[String, Counter] = languagesSelectedForStats.map { lang =>\n      lang -> statsScope.counter(f\"candidates_$lang\")\n    }.toMap\n    val nonEmptyHealthScoreMap: Map[String, Counter] = languagesSelectedForStats.map { lang =>\n      lang -> statsScope.counter(f\"non_empty_health_score_$lang\")\n    }.toMap\n    val emptyHealthScoreMap: Map[String, Counter] = languagesSelectedForStats.map { lang =>\n      lang -> statsScope.counter(f\"empty_health_score_$lang\")\n    }.toMap\n    val bucketedCounterMap: Map[String, Counter] = languagesSelectedForStats.map { lang =>\n      lang -> statsScope.counter(f\"num_candidates_bucketed_$lang\")\n    }.toMap\n    val filteredCounterMap: Map[String, Counter] = languagesSelectedForStats.map { lang =>\n      lang -> statsScope.counter(f\"num_filtered_$lang\")\n    }.toMap\n    val lowScoreCounterMap: Map[String, Counter] = languagesSelectedForStats.map { lang =>\n      lang -> statsScope.counter(f\"low_score_count_$lang\")\n    }.toMap\n\n    val wrongBucketingModelCounter = statsScope.counter(\"wrong_bucketing_model_count\")\n    val wrongDetectionModelCounter = statsScope.counter(\"wrong_detection_model_count\")\n\n    def increaseCounterForLanguage(counterMap: Map[String, Counter], language: String): Unit = {\n      counterMap.get(allLanguagesIdentifier) match {\n        case Some(counter) => counter.incr()\n        case _ =>\n      }\n      counterMap.get(language) match {\n        case Some(counter) => counter.incr()\n        case _ =>\n      }\n    }\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate =>\n        val target = candidate.target\n\n        val languageFeatureName = \"RecTweet.TweetyPieResult.Language\"\n\n        lazy val isPredicateEnabledForTarget = target.params(\n          PushFeatureSwitchParams.EnableHealthSignalStoreMultilingualPnsfwTweetTextPredicate)\n\n        lazy val targetNsfwTextDetectionModel: NsfwTextDetectionModel.Value =\n          target.params(PushFeatureSwitchParams.MultilingualPnsfwTweetTextModel)\n\n        lazy val targetPredicateSupportedLanguageSeq: Seq[String] =\n          target.params(PushFeatureSwitchParams.MultilingualPnsfwTweetTextSupportedLanguages)\n\n        lazy val bucketingModelSeq: Seq[NsfwTextDetectionModel.Value] =\n          target.params(PushFeatureSwitchParams.MultilingualPnsfwTweetTextBucketingModelList)\n\n        lazy val bucketingThresholdPerLanguageSeq: Seq[Double] =\n          target.params(PushFeatureSwitchParams.MultilingualPnsfwTweetTextBucketingThreshold)\n\n        lazy val filteringThresholdPerLanguageSeq: Seq[Double] =\n          target.params(PushFeatureSwitchParams.MultilingualPnsfwTweetTextFilteringThreshold)\n\n        if (CandidateUtil.shouldApplyHealthQualityFilters(\n            candidate) && isPredicateEnabledForTarget) {\n          val candidateLanguage =\n            candidate.categoricalFeatures.getOrElse(languageFeatureName, \"\")\n\n          val indexOfCandidateLanguage =\n            targetPredicateSupportedLanguageSeq.indexOf(candidateLanguage)\n\n          val isCandidateLanguageSupported = indexOfCandidateLanguage >= 0\n\n          if (isCandidateLanguageSupported) {\n            increaseCounterForLanguage(candidatesCounterMap, candidateLanguage)\n\n            val bucketingModelScoreMap: Map[NsfwTextDetectionModel.Value, Future[Option[Double]]] =\n              bucketingModelSeq.map { modelName =>\n                NsfwTextDetectionModelMap.get(modelName) match {\n                  case Some(targetNsfwTextDetectionModel) =>\n                    val pnsfwTweetTextRequest: TweetScoringRequest =\n                      TweetScoringRequest(candidate.tweetId, targetNsfwTextDetectionModel)\n\n                    val scoreOptFut: Future[Option[Double]] =\n                      tweetHealthScoreStore.get(pnsfwTweetTextRequest).map(_.map(_.score))\n\n                    candidate\n                      .cacheExternalScore(\"NsfwTextProbability\", scoreOptFut)\n\n                    modelName -> scoreOptFut\n                  case _ =>\n                    wrongBucketingModelCounter.incr()\n                    modelName -> Future.None\n                }\n              }.toMap\n\n            val candidateLanguageBucketingThreshold =\n              bucketingThresholdPerLanguageSeq(indexOfCandidateLanguage)\n\n            val userShouldBeBucketedFut: Future[Boolean] =\n              Future\n                .collect(bucketingModelScoreMap.map {\n                  case (_, modelScoreOptFut) =>\n                    modelScoreOptFut.map {\n                      case Some(score) =>\n                        increaseCounterForLanguage(nonEmptyHealthScoreMap, candidateLanguage)\n                        score > candidateLanguageBucketingThreshold\n                      case _ =>\n                        increaseCounterForLanguage(emptyHealthScoreMap, candidateLanguage)\n                        false\n                    }\n                }.toSeq).map(_.contains(true))\n\n            val candidateShouldBeFilteredFut: Future[Boolean] = userShouldBeBucketedFut.flatMap {\n              userShouldBeBucketed =>\n                if (userShouldBeBucketed) {\n                  increaseCounterForLanguage(bucketedCounterMap, candidateLanguage)\n\n                  val candidateLanguageFilteringThreshold =\n                    filteringThresholdPerLanguageSeq(indexOfCandidateLanguage)\n\n                  bucketingModelScoreMap.get(targetNsfwTextDetectionModel) match {\n                    case Some(scoreOptFut) =>\n                      scoreOptFut.map {\n                        case Some(score) =>\n                          val candidateShouldBeFiltered =\n                            score > candidateLanguageFilteringThreshold\n                          if (candidateShouldBeFiltered) {\n                            increaseCounterForLanguage(filteredCounterMap, candidateLanguage)\n                          }\n                          candidateShouldBeFiltered\n                        case _ => false\n                      }\n                    case _ =>\n                      wrongDetectionModelCounter.incr()\n                      Future.False\n                  }\n                } else {\n                  increaseCounterForLanguage(lowScoreCounterMap, candidateLanguage)\n                  Future.False\n                }\n            }\n            candidateShouldBeFilteredFut.map(result => !result)\n          } else Future.True\n        } else Future.True\n      }\n      .withStats(stats.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def authorProfileBasedPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetCandidate] = {\n    val name = \"author_profile\"\n    val statsScope = stats.scope(name)\n    val filterByNsfwToken = statsScope.counter(\"filter_by_nsfw_token\")\n    val filterByAccountAge = statsScope.counter(\"filter_by_account_age\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate =>\n        val target = candidate.target\n        candidate match {\n          case cand: PushCandidate with TweetAuthorDetails =>\n            cand.tweetAuthor.map {\n              case Some(author) =>\n                val nsfwTokens = target.params(PushFeatureSwitchParams.NsfwTokensParam)\n                val accountAgeInHours =\n                  (Time.now - Time.fromMilliseconds(author.createdAtMsec)).inHours\n                val isNsfwAccount = CandidateHydrationUtil.isNsfwAccount(author, nsfwTokens)\n                val isVerified = author.safety.map(_.verified).getOrElse(false)\n\n                if (CandidateUtil.shouldApplyHealthQualityFilters(candidate) && !isVerified) {\n                  val enableNsfwTokenCheck =\n                    target.params(PushFeatureSwitchParams.EnableNsfwTokenBasedFiltering)\n                  val minimumAllowedAge =\n                    target.params(PushFeatureSwitchParams.MinimumAllowedAuthorAccountAgeInHours)\n                  cand.cachePredicateInfo(\n                    name + \"_nsfwToken\",\n                    if (isNsfwAccount) 1.0 else 0.0,\n                    0.0,\n                    enableNsfwTokenCheck && isNsfwAccount)\n                  cand.cachePredicateInfo(\n                    name + \"_authorAge\",\n                    accountAgeInHours,\n                    minimumAllowedAge,\n                    accountAgeInHours < minimumAllowedAge)\n\n                  if (enableNsfwTokenCheck && isNsfwAccount) {\n                    filterByNsfwToken.incr()\n                    false\n                  } else if (accountAgeInHours < minimumAllowedAge) {\n                    filterByAccountAge.incr()\n                    false\n                  } else true\n                } else true\n              case _ => true\n            }\n          case _ => Future.value(true)\n        }\n      }\n      .withStats(stats.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def authorSensitiveMediaPredicate(\n    producerMediaRepresentationStore: ReadableStore[Long, UserMediaRepresentation]\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor] = {\n    val name = \"author_sensitive_media_mrtwistly\"\n    val statsScope = stats.scope(name)\n    val enableQueryNum = statsScope.counter(\"enable_query\")\n    val nonEmptyMediaRepresentationNum = statsScope.counter(\"non_empty_media_representation\")\n    val filteredOON = statsScope.counter(\"filtered_oon\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetAuthor =>\n        val target = candidate.target\n        val useAggressiveThresholds = CandidateUtil.useAggressiveHealthThresholds(candidate)\n\n        if (CandidateUtil.shouldApplyHealthQualityFilters(candidate) &&\n          RecTypes.isOutOfNetworkTweetRecType(candidate.commonRecType) &&\n          target.params(PushFeatureSwitchParams.EnableQueryAuthorMediaRepresentationStore)) {\n          enableQueryNum.incr()\n\n          candidate.authorId match {\n            case Some(authorId) =>\n              producerMediaRepresentationStore.get(authorId).map {\n                case Some(mediaRepresentation) =>\n                  nonEmptyMediaRepresentationNum.incr()\n                  val sumScore: Double = mediaRepresentation.mediaRepresentation.values.sum\n                  val nudityScore: Double = mediaRepresentation.mediaRepresentation\n                    .getOrElse(MediaAnnotationsUtil.nudityCategoryId, 0.0)\n                  val nudityRate = if (sumScore > 0) nudityScore / sumScore else 0.0\n\n                  candidate\n                    .cacheExternalScore(\"AuthorNudityScore\", Future.value(Some(nudityScore)))\n                  candidate.cacheExternalScore(\"AuthorNudityRate\", Future.value(Some(nudityRate)))\n\n                  val threshold = if (useAggressiveThresholds) {\n                    target.params(\n                      PushFeatureSwitchParams.AuthorSensitiveMediaFilteringThresholdForMrTwistly)\n                  } else {\n                    target.params(PushFeatureSwitchParams.AuthorSensitiveMediaFilteringThreshold)\n                  }\n                  candidate.cachePredicateInfo(\n                    name,\n                    nudityRate,\n                    threshold,\n                    nudityRate > threshold,\n                    Some(Map[String, Double](\"sumScore\" -> sumScore, \"nudityScore\" -> nudityScore)))\n\n                  if (nudityRate > threshold) {\n                    filteredOON.incr()\n                    false\n                  } else true\n                case _ => true\n              }\n            case _ => Future.True\n          }\n        } else {\n          Future.True\n        }\n      }\n      .withStats(stats.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def sensitiveMediaCategoryPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetCandidate] = {\n    val name = \"sensitive_media_category\"\n    val tweetMediaAnnotationFeature =\n      \"tweet.mediaunderstanding.tweet_annotations.sensitive_category_probabilities\"\n    val scopedStatsReceiver = stats.scope(name)\n    val allCandidatesCounter = scopedStatsReceiver.counter(\"all_candidates\")\n    val nonZeroNudityCandidatesCounter = scopedStatsReceiver.counter(\"non_zero_nudity_candidates\")\n    val nudityScoreStats = scopedStatsReceiver.stat(\"nudity_scores\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        allCandidatesCounter.incr()\n        val target = candidate.target\n        val nudityScore = candidate.sparseContinuousFeatures\n          .getOrElse(tweetMediaAnnotationFeature, Map.empty[String, Double]).getOrElse(\n            MediaAnnotationsUtil.nudityCategoryId,\n            0.0)\n        if (nudityScore > 0) nonZeroNudityCandidatesCounter.incr()\n        nudityScoreStats.add(nudityScore.toFloat)\n        val threshold =\n          target.params(PushFeatureSwitchParams.TweetMediaSensitiveCategoryThresholdParam)\n        candidate.cachePredicateInfo(name, nudityScore, threshold, nudityScore > threshold)\n        if (CandidateUtil.shouldApplyHealthQualityFilters(candidate) && nudityScore > threshold) {\n          Future.False\n        } else {\n          Future.True\n        }\n      }\n      .withStats(stats.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def profanityPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetCandidate] = {\n    val name = \"profanity_filter\"\n    val scopedStatsReceiver = stats.scope(name)\n    val allCandidatesCounter = scopedStatsReceiver.counter(\"all_candidates\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        allCandidatesCounter.incr()\n        val target = candidate.target\n\n        lazy val enableFilter =\n          target.params(PushFeatureSwitchParams.EnableProfanityFilterParam)\n        val tweetSemanticCoreIds = candidate.sparseBinaryFeatures\n          .getOrElse(PushConstants.TweetSemanticCoreIdFeature, Set.empty[String])\n\n        if (CandidateUtil.shouldApplyHealthQualityFilters(candidate) &&\n          tweetSemanticCoreIds.contains(PushConstants.ProfanityFilter_Id) && enableFilter) {\n          Future.False\n        } else {\n          Future.True\n        }\n      }\n      .withStats(stats.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def agathaAbusiveTweetAuthorPredicateMrTwistly(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with OutOfNetworkTweetCandidate] = {\n    val name = \"agatha_abusive_tweet_author_mr_twistly\"\n    val scopedStatsReceiver = stats.scope(name)\n    val allCandidatesCounter = scopedStatsReceiver.counter(\"all_candidates\")\n    val isMrBackfillCRCandidateCounter = scopedStatsReceiver.counter(\"isMrBackfillCR_candidates\")\n    Predicate\n      .fromAsync { cand: PushCandidate with OutOfNetworkTweetCandidate =>\n        allCandidatesCounter.incr()\n        val target = cand.target\n        val tweetSemanticCoreIds = cand.sparseBinaryFeatures\n          .getOrElse(PushConstants.TweetSemanticCoreIdFeature, Set.empty[String])\n\n        val hasAbuseStrikeTop2Percent =\n          tweetSemanticCoreIds.contains(PushConstants.AbuseStrike_Top2Percent_Id)\n        val hasAbuseStrikeTop1Percent =\n          tweetSemanticCoreIds.contains(PushConstants.AbuseStrike_Top1Percent_Id)\n        val hasAbuseStrikeTop05Percent =\n          tweetSemanticCoreIds.contains(PushConstants.AbuseStrike_Top05Percent_Id)\n\n        if (hasAbuseStrikeTop2Percent) {\n          scopedStatsReceiver.counter(\"abuse_strike_top_2_percent_candidates\").incr()\n        }\n        if (hasAbuseStrikeTop1Percent) {\n          scopedStatsReceiver.counter(\"abuse_strike_top_1_percent_candidates\").incr()\n        }\n        if (hasAbuseStrikeTop05Percent) {\n          scopedStatsReceiver.counter(\"abuse_strike_top_05_percent_candidates\").incr()\n        }\n\n        if (CandidateUtil.shouldApplyHealthQualityFilters(cand) && cand.isMrBackfillCR.getOrElse(\n            false)) {\n          isMrBackfillCRCandidateCounter.incr()\n          if (hasAbuseStrikeTop2Percent) {\n            if (target.params(\n                PushFeatureSwitchParams.EnableAbuseStrikeTop2PercentFilterSimCluster) && hasAbuseStrikeTop2Percent ||\n              target.params(\n                PushFeatureSwitchParams.EnableAbuseStrikeTop1PercentFilterSimCluster) && hasAbuseStrikeTop1Percent ||\n              target.params(\n                PushFeatureSwitchParams.EnableAbuseStrikeTop05PercentFilterSimCluster) && hasAbuseStrikeTop05Percent) {\n              Future.False\n            } else {\n              Future.True\n            }\n          } else {\n            Future.True\n          }\n        } else Future.True\n      }\n      .withStats(stats.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def userHealthSignalsPredicate(\n    userHealthSignalStore: ReadableStore[Long, UserHealthSignalResponse]\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetDetails] = {\n    val name = \"agatha_user_health_model_score\"\n    val scopedStatsReceiver = stats.scope(name)\n    val allCandidatesCounter = scopedStatsReceiver.counter(\"all_candidates\")\n    val bucketedUserCandidatesCounter =\n      scopedStatsReceiver.counter(\"bucketed_user_candidates\")\n    val filteredOON = scopedStatsReceiver.counter(\"filtered_oon\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetDetails =>\n        allCandidatesCounter.incr()\n        val target = candidate.target\n        val useAggressiveThresholds = CandidateUtil.useAggressiveHealthThresholds(candidate)\n\n        if (CandidateUtil.shouldApplyHealthQualityFilters(candidate) && target.params(\n            PushFeatureSwitchParams.EnableAgathaUserHealthModelPredicate)) {\n          val healthSignalsResponseFutOpt: Future[Option[UserHealthSignalResponse]] =\n            candidate.authorId match {\n              case Some(authorId) => userHealthSignalStore.get(authorId)\n              case _ => Future.None\n            }\n          healthSignalsResponseFutOpt.map {\n            case Some(response) =>\n              val agathaRecentAbuseStrikeScore: Double = userHealthSignalValueToDouble(\n                response.signalValues\n                  .getOrElse(AgathaRecentAbuseStrikeDouble, SignalValue.DoubleValue(0.0)))\n              val agathaCalibratedNSFWScore: Double = userHealthSignalValueToDouble(\n                response.signalValues\n                  .getOrElse(AgathaCalibratedNsfwDouble, SignalValue.DoubleValue(0.0)))\n              val agathaTextNSFWScore: Double = userHealthSignalValueToDouble(response.signalValues\n                .getOrElse(NsfwTextUserScoreDouble, SignalValue.DoubleValue(0.0)))\n\n              candidate\n                .cacheExternalScore(\n                  \"agathaRecentAbuseStrikeScore\",\n                  Future.value(Some(agathaRecentAbuseStrikeScore)))\n              candidate\n                .cacheExternalScore(\n                  \"agathaCalibratedNSFWScore\",\n                  Future.value(Some(agathaCalibratedNSFWScore)))\n              candidate\n                .cacheExternalScore(\"agathaTextNSFWScore\", Future.value(Some(agathaTextNSFWScore)))\n\n              val NSFWShouldBucket = agathaCalibratedNSFWScore > target.params(\n                PushFeatureSwitchParams.AgathaCalibratedNSFWBucketThreshold)\n              val textNSFWShouldBucket = agathaTextNSFWScore > target.params(\n                PushFeatureSwitchParams.AgathaTextNSFWBucketThreshold)\n\n              if (NSFWShouldBucket || textNSFWShouldBucket) {\n                bucketedUserCandidatesCounter.incr()\n                if (NSFWShouldBucket) {\n                  scopedStatsReceiver.counter(\"calibrated_nsfw_bucketed_user_candidates\").incr()\n                }\n                if (textNSFWShouldBucket) {\n                  scopedStatsReceiver.counter(\"text_nsfw_bucketed_user_candidates\").incr()\n                }\n\n                val (thresholdAgathaNsfw, thresholdTextNsfw) = if (useAggressiveThresholds) {\n                  (\n                    target.params(\n                      PushFeatureSwitchParams.AgathaCalibratedNSFWThresholdForMrTwistly),\n                    target\n                      .params(PushFeatureSwitchParams.AgathaTextNSFWThresholdForMrTwistly))\n                } else {\n                  (\n                    target.params(PushFeatureSwitchParams.AgathaCalibratedNSFWThreshold),\n                    target.params(PushFeatureSwitchParams.AgathaTextNSFWThreshold))\n                }\n                candidate.cachePredicateInfo(\n                  name + \"_agathaNsfw\",\n                  agathaCalibratedNSFWScore,\n                  thresholdAgathaNsfw,\n                  agathaCalibratedNSFWScore > thresholdAgathaNsfw)\n                candidate.cachePredicateInfo(\n                  name + \"_authorTextNsfw\",\n                  agathaTextNSFWScore,\n                  thresholdTextNsfw,\n                  agathaTextNSFWScore > thresholdTextNsfw)\n\n                if ((agathaCalibratedNSFWScore > thresholdAgathaNsfw) ||\n                  (agathaTextNSFWScore > thresholdTextNsfw)) {\n                  filteredOON.incr()\n                  false\n                } else true\n              } else {\n                true\n              }\n            case _ => true\n          }\n        } else {\n          Future.True\n        }\n      }\n      .withStats(stats.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def userHealthSignalValueToDouble(signalValue: SignalValue): Double = {\n    signalValue match {\n      case SignalValue.DoubleValue(value) => value\n      case _ => throw new Exception(f\"Could not convert signal value to double\")\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/JointDauAndQualityModelPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams.QualityPredicateIdParam\nimport com.twitter.frigate.pushservice.predicate.quality_model_predicate._\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\nobject JointDauAndQualityModelPredicate {\n\n  val name = \"JointDauAndQualityModelPredicate\"\n\n  def apply()(implicit statsReceiver: StatsReceiver): NamedPredicate[PushCandidate] = {\n    val stats = statsReceiver.scope(s\"predicate_$name\")\n\n    val defaultPred = WeightedOpenOrNtabClickQualityPredicate()\n    val qualityPredicateMap = QualityPredicateMap()\n\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        if (!candidate.target.skipModelPredicate) {\n\n          val modelPredicate =\n            qualityPredicateMap.getOrElse(\n              candidate.target.params(QualityPredicateIdParam),\n              defaultPred)\n\n          val modelPredicateResultFut =\n            modelPredicate.apply(Seq(candidate)).map(_.headOption.getOrElse(false))\n\n          modelPredicateResultFut\n        } else Future.True\n      }\n      .withStats(stats)\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/ListPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.model.ListRecommendationPushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.hermit.predicate.socialgraph.Edge\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.hermit.predicate.socialgraph.SocialGraphPredicate\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.socialgraph.thriftscala.RelationshipType\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nobject ListPredicates {\n\n  def listNameExistsPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[ListRecommendationPushCandidate] = {\n    Predicate\n      .fromAsync { candidate: ListRecommendationPushCandidate =>\n        candidate.listName.map(_.isDefined)\n      }\n      .withStats(stats)\n      .withName(\"list_name_exists\")\n  }\n\n  def listAuthorExistsPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[ListRecommendationPushCandidate] = {\n    Predicate\n      .fromAsync { candidate: ListRecommendationPushCandidate =>\n        candidate.listOwnerId.map(_.isDefined)\n      }\n      .withStats(stats)\n      .withName(\"list_owner_exists\")\n  }\n\n  def listAuthorAcceptableToTargetUser(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[ListRecommendationPushCandidate] = {\n    val name = \"list_author_acceptable_to_target_user\"\n    val sgsPredicate = SocialGraphPredicate\n      .anyRelationExists(\n        edgeStore,\n        Set(\n          RelationshipType.Blocking,\n          RelationshipType.BlockedBy,\n          RelationshipType.Muting\n        )\n      )\n      .withStats(statsReceiver.scope(\"list_sgs_any_relation_exists\"))\n      .withName(\"list_sgs_any_relation_exists\")\n\n    Predicate\n      .fromAsync { candidate: ListRecommendationPushCandidate =>\n        candidate.listOwnerId.flatMap {\n          case Some(ownerId) =>\n            sgsPredicate.apply(Seq(Edge(candidate.target.targetId, ownerId))).map(_.head)\n          case _ => Future.True\n        }\n      }\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  /**\n   * Checks if the list is acceptable to Target user =>\n   *    - Is Target not following the list\n   *    - Is Target not muted the list\n   */\n  def listAcceptablePredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[ListRecommendationPushCandidate] = {\n    val name = \"list_acceptable_to_target_user\"\n    Predicate\n      .fromAsync { candidate: ListRecommendationPushCandidate =>\n        candidate.apiList.map {\n          case Some(apiList) =>\n            !(apiList.following.contains(true) || apiList.muting.contains(true))\n          case _ => false\n        }\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def listSubscriberCountPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[ListRecommendationPushCandidate] = {\n    val name = \"list_subscribe_count\"\n    Predicate\n      .fromAsync { candidate: ListRecommendationPushCandidate =>\n        candidate.apiList.map { apiListOpt =>\n          apiListOpt.exists { apiList =>\n            apiList.subscriberCount >= candidate.target.params(\n              PushFeatureSwitchParams.ListRecommendationsSubscriberCount)\n          }\n        }\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/LoggedOutPreRankingPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.base.TweetDetails\nimport com.twitter.frigate.common.predicate.tweet._\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.hermit.predicate.NamedPredicate\n\nclass LoggedOutPreRankingPredicatesBuilder(implicit statsReceiver: StatsReceiver) {\n\n  private val TweetPredicates = List[NamedPredicate[PushCandidate]](\n    TweetObjectExistsPredicate[\n      TweetCandidate with TweetDetails\n    ].applyOnlyToTweetCandidatesWithTweetDetails\n      .withName(\"tweet_object_exists\"),\n    PredicatesForCandidate.oldTweetRecsPredicate.applyOnlyToTweetCandidateWithTargetAndABDeciderAndMaxTweetAge\n      .withName(\"old_tweet\"),\n    PredicatesForCandidate.tweetIsNotAreply.applyOnlyToTweetCandidateWithoutSocialContextWithTweetDetails\n      .withName(\"tweet_candidate_not_a_reply\"),\n    TweetAuthorPredicates\n      .recTweetAuthorUnsuitable[TweetCandidate with TweetAuthorDetails]\n      .applyOnlyToTweetCandidateWithTweetAuthorDetails\n      .withName(\"tweet_author_unsuitable\")\n  )\n\n  final def build(): List[NamedPredicate[PushCandidate]] = {\n    TweetPredicates\n  }\n\n}\n\nobject LoggedOutPreRankingPredicates {\n  def apply(statsReceiver: StatsReceiver): List[NamedPredicate[PushCandidate]] =\n    new LoggedOutPreRankingPredicatesBuilder()(statsReceiver).build()\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/LoggedOutTargetPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.abdecider.GuestRecipient\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.common.predicate.{FatiguePredicate => CommonFatiguePredicate}\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.frigate.common.util.Experiments.LoggedOutRecsHoldback\nimport com.twitter.hermit.predicate.Predicate\n\nobject LoggedOutTargetPredicates {\n\n  def targetFatiguePredicate[T <: Target](\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n    val name = \"logged_out_target_min_duration_since_push\"\n    CommonFatiguePredicate\n      .magicRecsPushTargetFatiguePredicate(\n        minInterval = 24.hours,\n        maxInInterval = 1\n      ).withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  def loggedOutRecsHoldbackPredicate[T <: Target](\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n    val name = \"logged_out_recs_holdback\"\n    val guestIdNotFoundCounter = statsReceiver.scope(\"logged_out\").counter(\"guest_id_not_found\")\n    val controlBucketCounter = statsReceiver.scope(\"logged_out\").counter(\"holdback_control\")\n    val allowTrafficCounter = statsReceiver.scope(\"logged_out\").counter(\"allow_traffic\")\n    Predicate.from { target: T =>\n      val guestId = target.targetGuestId match {\n        case Some(guest) => guest\n        case _ =>\n          guestIdNotFoundCounter.incr()\n          throw new IllegalStateException(\"guest_id_not_found\")\n      }\n      target.abDecider\n        .bucket(LoggedOutRecsHoldback.exptName, GuestRecipient(guestId)).map(_.name) match {\n        case Some(LoggedOutRecsHoldback.control) =>\n          controlBucketCounter.incr()\n          false\n        case _ =>\n          allowTrafficCounter.incr()\n          true\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/MlModelsHoldbackExperimentPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\nobject MlModelsHoldbackExperimentPredicate {\n\n  val name = \"MlModelsHoldbackExperimentPredicate\"\n\n  private val alwaysTruePred = PredicatesForCandidate.alwaysTruePushCandidatePredicate\n\n  def getPredicateBasedOnCandidate(\n    pc: PushCandidate,\n    treatmentPred: Predicate[PushCandidate]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): Future[Predicate[PushCandidate]] = {\n\n    Future\n      .join(Future.value(pc.target.skipFilters), pc.target.isInModelExclusionList)\n      .map {\n        case (skipFilters, isInModelExclusionList) =>\n          if (skipFilters ||\n            isInModelExclusionList ||\n            pc.target.params(PushParams.DisableMlInFilteringParam) ||\n            pc.target.params(PushFeatureSwitchParams.DisableMlInFilteringFeatureSwitchParam) ||\n            pc.target.params(PushParams.DisableAllRelevanceParam) ||\n            pc.target.params(PushParams.DisableHeavyRankingParam)) {\n            alwaysTruePred\n          } else {\n            treatmentPred\n          }\n      }\n  }\n\n  def apply()(implicit statsReceiver: StatsReceiver): NamedPredicate[PushCandidate] = {\n    val stats = statsReceiver.scope(s\"predicate_$name\")\n    val statsProd = stats.scope(\"prod\")\n    val counterAcceptedByModel = statsProd.counter(\"accepted\")\n    val counterRejectedByModel = statsProd.counter(\"rejected\")\n    val counterHoldback = stats.scope(\"holdback\").counter(\"all\")\n    val jointDauQualityPredicate = JointDauAndQualityModelPredicate()\n\n    new Predicate[PushCandidate] {\n      def apply(items: Seq[PushCandidate]): Future[Seq[Boolean]] = {\n        val boolFuts = items.map { item =>\n          getPredicateBasedOnCandidate(item, jointDauQualityPredicate)(statsReceiver)\n            .flatMap { predicate =>\n              val predictionFut = predicate.apply(Seq(item)).map(_.headOption.getOrElse(false))\n              predictionFut.foreach { prediction =>\n                if (item.target.params(PushParams.DisableMlInFilteringParam) || item.target.params(\n                    PushFeatureSwitchParams.DisableMlInFilteringFeatureSwitchParam)) {\n                  counterHoldback.incr()\n                } else {\n                  if (prediction) counterAcceptedByModel.incr() else counterRejectedByModel.incr()\n                }\n              }\n              predictionFut\n            }\n        }\n        Future.collect(boolFuts)\n      }\n    }.withStats(stats)\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/OONSpreadControlPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushConstants._\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.util.CandidateUtil\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\nobject OONSpreadControlPredicate {\n\n  def oonTweetSpreadControlPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[\n    PushCandidate with TweetCandidate with RecommendationType\n  ] = {\n    val name = \"oon_tweet_spread_control_predicate\"\n    val scopedStatsReceiver = stats.scope(name)\n    val allOonCandidatesCounter = scopedStatsReceiver.counter(\"all_oon_candidates\")\n    val filteredCandidatesCounter =\n      scopedStatsReceiver.counter(\"filtered_oon_candidates\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate with RecommendationType =>\n        val target = candidate.target\n        val crt = candidate.commonRecType\n        val isOonCandidate = RecTypes.isOutOfNetworkTweetRecType(crt) ||\n          RecTypes.outOfNetworkTopicTweetTypes.contains(crt)\n\n        lazy val minTweetSendsThreshold =\n          target.params(PushFeatureSwitchParams.MinTweetSendsThresholdParam)\n        lazy val spreadControlRatio =\n          target.params(PushFeatureSwitchParams.SpreadControlRatioParam)\n        lazy val favOverSendThreshold =\n          target.params(PushFeatureSwitchParams.FavOverSendThresholdParam)\n\n        lazy val sentCount = candidate.numericFeatures.getOrElse(sentFeatureName, 0.0)\n        lazy val followerCount =\n          candidate.numericFeatures.getOrElse(authorActiveFollowerFeatureName, 0.0)\n        lazy val favCount = candidate.numericFeatures.getOrElse(favFeatureName, 0.0)\n        lazy val favOverSends = favCount / (sentCount + 1.0)\n\n        if (CandidateUtil.shouldApplyHealthQualityFilters(candidate) && isOonCandidate) {\n          allOonCandidatesCounter.incr()\n          if (sentCount > minTweetSendsThreshold &&\n            sentCount > spreadControlRatio * followerCount &&\n            favOverSends < favOverSendThreshold) {\n            filteredCandidatesCounter.incr()\n            Future.False\n          } else Future.True\n        } else Future.True\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def oonAuthorSpreadControlPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[\n    PushCandidate with TweetCandidate with RecommendationType\n  ] = {\n    val name = \"oon_author_spread_control_predicate\"\n    val scopedStatsReceiver = stats.scope(name)\n    val allOonCandidatesCounter = scopedStatsReceiver.counter(\"all_oon_candidates\")\n    val filteredCandidatesCounter =\n      scopedStatsReceiver.counter(\"filtered_oon_candidates\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate with RecommendationType =>\n        val target = candidate.target\n        val crt = candidate.commonRecType\n        val isOonCandidate = RecTypes.isOutOfNetworkTweetRecType(crt) ||\n          RecTypes.outOfNetworkTopicTweetTypes.contains(crt)\n\n        lazy val minAuthorSendsThreshold =\n          target.params(PushFeatureSwitchParams.MinAuthorSendsThresholdParam)\n        lazy val spreadControlRatio =\n          target.params(PushFeatureSwitchParams.SpreadControlRatioParam)\n        lazy val reportRateThreshold =\n          target.params(PushFeatureSwitchParams.AuthorReportRateThresholdParam)\n        lazy val dislikeRateThreshold =\n          target.params(PushFeatureSwitchParams.AuthorDislikeRateThresholdParam)\n\n        lazy val authorSentCount =\n          candidate.numericFeatures.getOrElse(authorSendCountFeatureName, 0.0)\n        lazy val authorReportCount =\n          candidate.numericFeatures.getOrElse(authorReportCountFeatureName, 0.0)\n        lazy val authorDislikeCount =\n          candidate.numericFeatures.getOrElse(authorDislikeCountFeatureName, 0.0)\n        lazy val followerCount = candidate.numericFeatures\n          .getOrElse(authorActiveFollowerFeatureName, 0.0)\n        lazy val reportRate =\n          authorReportCount / (authorSentCount + 1.0)\n        lazy val dislikeRate =\n          authorDislikeCount / (authorSentCount + 1.0)\n\n        if (CandidateUtil.shouldApplyHealthQualityFilters(candidate) && isOonCandidate) {\n          allOonCandidatesCounter.incr()\n          if (authorSentCount > minAuthorSendsThreshold &&\n            authorSentCount > spreadControlRatio * followerCount &&\n            (reportRate > reportRateThreshold || dislikeRate > dislikeRateThreshold)) {\n            filteredCandidatesCounter.incr()\n            Future.False\n          } else Future.True\n        } else Future.True\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/OONTweetNegativeFeedbackBasedPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.util.CandidateUtil\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\nobject OONTweetNegativeFeedbackBasedPredicate {\n\n  def ntabDislikeBasedPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[\n    PushCandidate with TweetCandidate with RecommendationType\n  ] = {\n    val name = \"oon_tweet_dislike_based_predicate\"\n    val scopedStatsReceiver = stats.scope(name)\n    val allOonCandidatesCounter = scopedStatsReceiver.counter(\"all_oon_candidates\")\n    val oonCandidatesImpressedCounter =\n      scopedStatsReceiver.counter(\"oon_candidates_impressed\")\n    val filteredCandidatesCounter =\n      scopedStatsReceiver.counter(\"filtered_oon_candidates\")\n\n    val ntabDislikeCountFeature =\n      \"tweet.magic_recs_tweet_real_time_aggregates_v2.pair.v2.magicrecs.realtime.is_ntab_disliked.any_feature.Duration.Top.count\"\n    val sentFeature =\n      \"tweet.magic_recs_tweet_real_time_aggregates_v2.pair.v2.magicrecs.realtime.is_sent.any_feature.Duration.Top.count\"\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate with RecommendationType =>\n        val target = candidate.target\n        val crt = candidate.commonRecType\n        val isOonCandidate = RecTypes.isOutOfNetworkTweetRecType(crt) ||\n          RecTypes.outOfNetworkTopicTweetTypes.contains(crt)\n\n        lazy val ntabDislikeCountThreshold =\n          target.params(PushFeatureSwitchParams.TweetNtabDislikeCountThresholdParam)\n        lazy val ntabDislikeRateThreshold =\n          target.params(PushFeatureSwitchParams.TweetNtabDislikeRateThresholdParam)\n        lazy val ntabDislikeCountThresholdForMrTwistly =\n          target.params(PushFeatureSwitchParams.TweetNtabDislikeCountThresholdForMrTwistlyParam)\n        lazy val ntabDislikeRateThresholdForMrTwistly =\n          target.params(PushFeatureSwitchParams.TweetNtabDislikeRateThresholdForMrTwistlyParam)\n\n        val isMrTwistly = CandidateUtil.isMrTwistlyCandidate(candidate)\n\n        lazy val dislikeCount = candidate.numericFeatures.getOrElse(ntabDislikeCountFeature, 0.0)\n        lazy val sentCount = candidate.numericFeatures.getOrElse(sentFeature, 0.0)\n        lazy val dislikeRate = if (sentCount > 0) dislikeCount / sentCount else 0.0\n\n        if (CandidateUtil.shouldApplyHealthQualityFilters(candidate) && isOonCandidate) {\n          allOonCandidatesCounter.incr()\n          val (countThreshold, rateThreshold) = if (isMrTwistly) {\n            (ntabDislikeCountThresholdForMrTwistly, ntabDislikeRateThresholdForMrTwistly)\n          } else {\n            (ntabDislikeCountThreshold, ntabDislikeRateThreshold)\n          }\n          candidate.cachePredicateInfo(\n            name + \"_count\",\n            dislikeCount,\n            countThreshold,\n            dislikeCount > countThreshold)\n          candidate.cachePredicateInfo(\n            name + \"_rate\",\n            dislikeRate,\n            rateThreshold,\n            dislikeRate > rateThreshold)\n          if (dislikeCount > countThreshold && dislikeRate > rateThreshold) {\n            filteredCandidatesCounter.incr()\n            Future.False\n          } else Future.True\n        } else Future.True\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/OutOfNetworkCandidatesQualityPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.candidate.TargetABDecider\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.data_pipeline.features_common.MrRequestContextForFeatureStore\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.ml.featurestore.lib.dynamic.DynamicFeatureStoreClient\nimport com.twitter.util.Future\nimport com.twitter.frigate.pushservice.predicate.PostRankingPredicateHelper._\nimport com.twitter.frigate.pushservice.util.CandidateUtil\n\nobject OutOfNetworkCandidatesQualityPredicates {\n\n  def getTweetCharLengthThreshold(\n    target: TargetUser with TargetABDecider,\n    language: String,\n    useMediaThresholds: Boolean\n  ): Double = {\n    lazy val sautOonWithMediaTweetLengthThreshold =\n      target.params(PushFeatureSwitchParams.SautOonWithMediaTweetLengthThresholdParam)\n    lazy val nonSautOonWithMediaTweetLengthThreshold =\n      target.params(PushFeatureSwitchParams.NonSautOonWithMediaTweetLengthThresholdParam)\n    lazy val sautOonWithoutMediaTweetLengthThreshold =\n      target.params(PushFeatureSwitchParams.SautOonWithoutMediaTweetLengthThresholdParam)\n    lazy val nonSautOonWithoutMediaTweetLengthThreshold =\n      target.params(PushFeatureSwitchParams.NonSautOonWithoutMediaTweetLengthThresholdParam)\n    val moreStrictForUndefinedLanguages =\n      target.params(PushFeatureSwitchParams.OonTweetLengthPredicateMoreStrictForUndefinedLanguages)\n    val isSautLanguage = if (moreStrictForUndefinedLanguages) {\n      isTweetLanguageInSautOrUndefined(language)\n    } else isTweetLanguageInSaut(language)\n\n    (useMediaThresholds, isSautLanguage) match {\n      case (true, true) =>\n        sautOonWithMediaTweetLengthThreshold\n      case (true, false) =>\n        nonSautOonWithMediaTweetLengthThreshold\n      case (false, true) =>\n        sautOonWithoutMediaTweetLengthThreshold\n      case (false, false) =>\n        nonSautOonWithoutMediaTweetLengthThreshold\n      case _ => -1\n    }\n  }\n\n  def getTweetWordLengthThreshold(\n    target: TargetUser with TargetABDecider,\n    language: String,\n    useMediaThresholds: Boolean\n  ): Double = {\n    lazy val argfOonWithMediaTweetWordLengthThresholdParam =\n      target.params(PushFeatureSwitchParams.ArgfOonWithMediaTweetWordLengthThresholdParam)\n    lazy val esfthOonWithMediaTweetWordLengthThresholdParam =\n      target.params(PushFeatureSwitchParams.EsfthOonWithMediaTweetWordLengthThresholdParam)\n\n    lazy val argfOonCandidatesWithMediaCondition =\n      isTweetLanguageInArgf(language) && useMediaThresholds\n    lazy val esfthOonCandidatesWithMediaCondition =\n      isTweetLanguageInEsfth(language) && useMediaThresholds\n    lazy val afirfOonCandidatesWithoutMediaCondition =\n      isTweetLanguageInAfirf(language) && !useMediaThresholds\n\n    val afirfOonCandidatesWithoutMediaTweetWordLengthThreshold = 5\n    if (argfOonCandidatesWithMediaCondition) {\n      argfOonWithMediaTweetWordLengthThresholdParam\n    } else if (esfthOonCandidatesWithMediaCondition) {\n      esfthOonWithMediaTweetWordLengthThresholdParam\n    } else if (afirfOonCandidatesWithoutMediaCondition) {\n      afirfOonCandidatesWithoutMediaTweetWordLengthThreshold\n    } else -1\n  }\n\n  def oonTweetLengthBasedPrerankingPredicate(\n    characterBased: Boolean\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[OutOfNetworkTweetCandidate with TargetInfo[\n    TargetUser with TargetABDecider\n  ]] = {\n    val name = \"oon_tweet_length_based_preranking_predicate\"\n    val scopedStats = stats.scope(s\"${name}_charBased_$characterBased\")\n\n    Predicate\n      .fromAsync {\n        cand: OutOfNetworkTweetCandidate with TargetInfo[TargetUser with TargetABDecider] =>\n          cand match {\n            case candidate: TweetAuthorDetails =>\n              val target = candidate.target\n              val crt = candidate.commonRecType\n\n              val updatedMediaLogic =\n                target.params(PushFeatureSwitchParams.OonTweetLengthPredicateUpdatedMediaLogic)\n              val updatedQuoteTweetLogic =\n                target.params(PushFeatureSwitchParams.OonTweetLengthPredicateUpdatedQuoteTweetLogic)\n              val useMediaThresholds = if (updatedMediaLogic || updatedQuoteTweetLogic) {\n                val hasMedia = updatedMediaLogic && (candidate.hasPhoto || candidate.hasVideo)\n                val hasQuoteTweet = updatedQuoteTweetLogic && candidate.quotedTweet.nonEmpty\n                hasMedia || hasQuoteTweet\n              } else RecTypes.isMediaType(crt)\n              val enableFilter =\n                target.params(PushFeatureSwitchParams.EnablePrerankingTweetLengthPredicate)\n\n              val language = candidate.tweet.flatMap(_.language.map(_.language)).getOrElse(\"\")\n              val tweetTextOpt = candidate.tweet.flatMap(_.coreData.map(_.text))\n\n              val (length: Double, threshold: Double) = if (characterBased) {\n                (\n                  tweetTextOpt.map(_.size.toDouble).getOrElse(9999.0),\n                  getTweetCharLengthThreshold(target, language, useMediaThresholds))\n              } else {\n                (\n                  tweetTextOpt.map(getTweetWordLength).getOrElse(999.0),\n                  getTweetWordLengthThreshold(target, language, useMediaThresholds))\n              }\n              scopedStats.counter(\"threshold_\" + threshold.toString).incr()\n\n              CandidateUtil.shouldApplyHealthQualityFiltersForPrerankingPredicates(candidate).map {\n                case true if enableFilter =>\n                  length > threshold\n                case _ => true\n              }\n            case _ =>\n              scopedStats.counter(\"author_is_not_hydrated\").incr()\n              Future.True\n          }\n      }.withStats(scopedStats)\n      .withName(name)\n  }\n\n  private def isTweetLanguageInAfirf(candidateLanguage: String): Boolean = {\n    val setAFIRF: Set[String] = Set(\"\")\n    setAFIRF.contains(candidateLanguage)\n  }\n  private def isTweetLanguageInEsfth(candidateLanguage: String): Boolean = {\n    val setESFTH: Set[String] = Set(\"\")\n    setESFTH.contains(candidateLanguage)\n  }\n  private def isTweetLanguageInArgf(candidateLanguage: String): Boolean = {\n    val setARGF: Set[String] = Set(\"\")\n    setARGF.contains(candidateLanguage)\n  }\n\n  private def isTweetLanguageInSaut(candidateLanguage: String): Boolean = {\n    val setSAUT = Set(\"\")\n    setSAUT.contains(candidateLanguage)\n  }\n\n  private def isTweetLanguageInSautOrUndefined(candidateLanguage: String): Boolean = {\n    val setSautOrUndefined = Set(\"\")\n    setSautOrUndefined.contains(candidateLanguage)\n  }\n\n  def containTargetNegativeKeywords(text: String, denylist: Seq[String]): Boolean = {\n    if (denylist.isEmpty)\n      false\n    else {\n      denylist\n        .map { negativeKeyword =>\n          text.toLowerCase().contains(negativeKeyword)\n        }.reduce(_ || _)\n    }\n  }\n\n  def NegativeKeywordsPredicate(\n    postRankingFeatureStoreClient: DynamicFeatureStoreClient[MrRequestContextForFeatureStore]\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[\n    PushCandidate with TweetCandidate with RecommendationType\n  ] = {\n\n    val name = \"negative_keywords_predicate\"\n    val scopedStatsReceiver = stats.scope(name)\n    val allOonCandidatesCounter = scopedStatsReceiver.counter(\"all_oon_candidates\")\n    val filteredOonCandidatesCounter = scopedStatsReceiver.counter(\"filtered_oon_candidates\")\n    val tweetLanguageFeature = \"RecTweet.TweetyPieResult.Language\"\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate with RecommendationType =>\n        val target = candidate.target\n        val crt = candidate.commonRecType\n        val isTwistlyCandidate = RecTypes.twistlyTweets.contains(crt)\n\n        lazy val enableNegativeKeywordsPredicateParam =\n          target.params(PushFeatureSwitchParams.EnableNegativeKeywordsPredicateParam)\n        lazy val negativeKeywordsPredicateDenylist =\n          target.params(PushFeatureSwitchParams.NegativeKeywordsPredicateDenylist)\n        lazy val candidateLanguage =\n          candidate.categoricalFeatures.getOrElse(tweetLanguageFeature, \"\")\n\n        if (CandidateUtil.shouldApplyHealthQualityFilters(candidate) && candidateLanguage.equals(\n            \"en\") && isTwistlyCandidate && enableNegativeKeywordsPredicateParam) {\n          allOonCandidatesCounter.incr()\n\n          val tweetTextFuture: Future[String] =\n            getTweetText(candidate, postRankingFeatureStoreClient)\n\n          tweetTextFuture.map { tweetText =>\n            val containsNegativeWords =\n              containTargetNegativeKeywords(tweetText, negativeKeywordsPredicateDenylist)\n            candidate.cachePredicateInfo(\n              name,\n              if (containsNegativeWords) 1.0 else 0.0,\n              0.0,\n              containsNegativeWords)\n            if (containsNegativeWords) {\n              filteredOonCandidatesCounter.incr()\n              false\n            } else true\n          }\n        } else Future.True\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/PNegMultimodalPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.abuse.detection.scoring.thriftscala.Model\nimport com.twitter.abuse.detection.scoring.thriftscala.TweetScoringRequest\nimport com.twitter.abuse.detection.scoring.thriftscala.TweetScoringResponse\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.util.CandidateUtil\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nobject PNegMultimodalPredicates {\n\n  def healthSignalScorePNegMultimodalPredicate(\n    tweetHealthScoreStore: ReadableStore[TweetScoringRequest, TweetScoringResponse]\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetCandidate] = {\n    val name = \"pneg_multimodal_predicate\"\n    val statsScope = stats.scope(name)\n    val oonCandidatesCounter = statsScope.counter(\"oon_candidates\")\n    val nonEmptyModelScoreCounter = statsScope.counter(\"non_empty_model_score\")\n    val bucketedCounter = statsScope.counter(\"bucketed_oon_candidates\")\n    val filteredCounter = statsScope.counter(\"filtered_oon_candidates\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate =>\n        val target = candidate.target\n        val crt = candidate.commonRecType\n        val isOonCandidate = RecTypes.isOutOfNetworkTweetRecType(crt) ||\n          RecTypes.outOfNetworkTopicTweetTypes.contains(crt)\n\n        lazy val enablePNegMultimodalPredicateParam =\n          target.params(PushFeatureSwitchParams.EnablePNegMultimodalPredicateParam)\n        lazy val pNegMultimodalPredicateModelThresholdParam =\n          target.params(PushFeatureSwitchParams.PNegMultimodalPredicateModelThresholdParam)\n        lazy val pNegMultimodalPredicateBucketThresholdParam =\n          target.params(PushFeatureSwitchParams.PNegMultimodalPredicateBucketThresholdParam)\n        val pNegMultimodalEnabledForF1Tweets =\n          target.params(PushParams.EnablePnegMultimodalPredictionForF1Tweets)\n\n        if (CandidateUtil.shouldApplyHealthQualityFilters(\n            candidate) && (isOonCandidate || pNegMultimodalEnabledForF1Tweets) && enablePNegMultimodalPredicateParam) {\n\n          val pNegMultimodalRequest = TweetScoringRequest(candidate.tweetId, Model.PNegMultimodal)\n          tweetHealthScoreStore.get(pNegMultimodalRequest).map {\n            case Some(tweetScoringResponse) =>\n              nonEmptyModelScoreCounter.incr()\n\n              val pNegMultimodalScore = 1.0 - tweetScoringResponse.score\n\n              candidate\n                .cacheExternalScore(\"PNegMultimodalScore\", Future.value(Some(pNegMultimodalScore)))\n\n              if (isOonCandidate) {\n                oonCandidatesCounter.incr()\n\n                if (pNegMultimodalScore > pNegMultimodalPredicateBucketThresholdParam) {\n                  bucketedCounter.incr()\n                  if (pNegMultimodalScore > pNegMultimodalPredicateModelThresholdParam) {\n                    filteredCounter.incr()\n                    false\n                  } else true\n                } else true\n              } else {\n                true\n              }\n            case _ => true\n          }\n        } else {\n          Future.True\n        }\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/PostRankingPredicateHelper.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.data_pipeline.features_common.MrRequestContextForFeatureStore\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.ml.featurestore.catalog.entities.core.Tweet\nimport com.twitter.ml.featurestore.catalog.features.core.Tweet.Text\nimport com.twitter.ml.featurestore.lib.TweetId\nimport com.twitter.ml.featurestore.lib.dynamic.DynamicFeatureStoreClient\nimport com.twitter.ml.featurestore.lib.online.FeatureStoreRequest\nimport com.twitter.util.Future\n\nobject PostRankingPredicateHelper {\n\n  val tweetTextFeature = \"tweet.core.tweet.text\"\n\n  def getTweetText(\n    candidate: PushCandidate with TweetCandidate,\n    dynamicClient: DynamicFeatureStoreClient[MrRequestContextForFeatureStore]\n  ): Future[String] = {\n    if (candidate.categoricalFeatures.contains(tweetTextFeature)) {\n      Future.value(candidate.categoricalFeatures.getOrElse(tweetTextFeature, \"\"))\n    } else {\n      val candidateTweetEntity = Tweet.withId(TweetId(candidate.tweetId))\n      val featureStoreRequests = Seq(\n        FeatureStoreRequest(\n          entityIds = Seq(candidateTweetEntity)\n        ))\n      val predictionRecords = dynamicClient(\n        featureStoreRequests,\n        requestContext = candidate.target.mrRequestContextForFeatureStore)\n\n      predictionRecords.map { records =>\n        val tweetText = records.head\n          .getFeatureValue(candidateTweetEntity, Text).getOrElse(\n            \"\"\n          )\n        candidate.categoricalFeatures(tweetTextFeature) = tweetText\n        tweetText\n      }\n    }\n  }\n\n  def getTweetWordLength(tweetText: String): Double = {\n    val tweetTextWithoutUrl: String =\n      tweetText.replaceAll(\"https?://\\\\S+\\\\s?\", \"\").replaceAll(\"[\\\\s]+\", \" \")\n    tweetTextWithoutUrl.trim().split(\" \").length.toDouble\n  }\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/PreRankingPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.SocialContextActions\nimport com.twitter.frigate.common.base.SocialContextUserDetails\nimport com.twitter.frigate.common.base.TargetInfo\nimport com.twitter.frigate.common.base.TargetUser\nimport com.twitter.frigate.common.base.TweetAuthor\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.base.TweetDetails\nimport com.twitter.frigate.common.candidate.FrigateHistory\nimport com.twitter.frigate.common.candidate.TargetABDecider\nimport com.twitter.frigate.common.candidate.TweetImpressionHistory\nimport com.twitter.frigate.common.predicate.socialcontext.{Predicates => SocialContextPredicates, _}\nimport com.twitter.frigate.common.predicate.tweet._\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue.NtabCaretClickContFnFatiguePredicate\nimport com.twitter.hermit.predicate.NamedPredicate\n\nclass PreRankingPredicatesBuilder(\n)(\n  implicit statsReceiver: StatsReceiver) {\n\n  private val SocialProofPredicates = List[NamedPredicate[PushCandidate]](\n    SocialContextPredicates\n      .authorInSocialContext()\n      .applyOnlyToTweetAuthorWithSocialContextActions\n      .withName(\"author_social_context\"),\n    SocialContextPredicates\n      .selfInSocialContext[TargetUser, SocialContextActions with TargetInfo[TargetUser]]()\n      .applyOnlyToSocialContextActionsWithTargetUser\n      .withName(\"self_social_context\"),\n    SocialContextPredicates\n      .duplicateSocialContext[SocialContextActions]()\n      .applyOnlyToSocialContextActions\n      .withName(\"duplicate_social_context\"),\n    SocialContextPredicates\n      .socialContextProtected[SocialContextUserDetails]()\n      .applyOnlyToSocialContextUserDetails\n      .withName(\"social_context_protected\"),\n    SocialContextPredicates\n      .socialContextUnsuitable[SocialContextUserDetails]()\n      .applyOnlyToSocialContextUserDetails\n      .withName(\"social_context_unsuitable\"),\n    SocialContextPredicates\n      .socialContextBlink[SocialContextUserDetails]()\n      .applyOnlyToSocialContextUserDetails\n      .withName(\"social_context_blink\")\n  )\n\n  private val CommonPredicates = List[NamedPredicate[PushCandidate]](\n    PredicatesForCandidate.candidateEnabledForEmailPredicate(),\n    PredicatesForCandidate.openAppExperimentUserCandidateAllowList(statsReceiver)\n  )\n\n  private val TweetPredicates = List[NamedPredicate[PushCandidate]](\n    PredicatesForCandidate.tweetCandidateWithLessThan2SocialContextsIsAReply.applyOnlyToTweetCandidatesWithSocialContextActions\n      .withName(\"tweet_candidate_with_less_than_2_social_contexts_is_not_a_reply\"),\n    PredicatesForCandidate.filterOONCandidatePredicate(),\n    PredicatesForCandidate.oldTweetRecsPredicate.applyOnlyToTweetCandidateWithTargetAndABDeciderAndMaxTweetAge\n      .withName(\"old_tweet\"),\n    DuplicatePushTweetPredicate\n      .apply[\n        TargetUser with FrigateHistory,\n        TweetCandidate with TargetInfo[TargetUser with FrigateHistory]\n      ]\n      .applyOnlyToTweetCandidateWithTargetAndFrigateHistory\n      .withName(\"duplicate_push_tweet\"),\n    DuplicateEmailTweetPredicate\n      .apply[\n        TargetUser with FrigateHistory,\n        TweetCandidate with TargetInfo[TargetUser with FrigateHistory]\n      ]\n      .applyOnlyToTweetCandidateWithTargetAndFrigateHistory\n      .withName(\"duplicate_email_tweet\"),\n    TweetAuthorPredicates\n      .recTweetAuthorUnsuitable[TweetCandidate with TweetAuthorDetails]\n      .applyOnlyToTweetCandidateWithTweetAuthorDetails\n      .withName(\"tweet_author_unsuitable\"),\n    TweetObjectExistsPredicate[\n      TweetCandidate with TweetDetails\n    ].applyOnlyToTweetCandidatesWithTweetDetails\n      .withName(\"tweet_object_exists\"),\n    TweetImpressionPredicate[\n      TargetUser with TweetImpressionHistory,\n      TweetCandidate with TargetInfo[TargetUser with TweetImpressionHistory]\n    ].applyOnlyToTweetCandidateWithTargetAndTweetImpressionHistory\n      .withStats(statsReceiver.scope(\"tweet_impression\"))\n      .withName(\"tweet_impression\"),\n    SelfTweetPredicate[\n      TargetUser,\n      TweetAuthor with TargetInfo[TargetUser]]().applyOnlyToTweetAuthorWithTargetInfo\n      .withName(\"self_author\"),\n    PredicatesForCandidate.tweetIsNotAreply.applyOnlyToTweetCandidateWithoutSocialContextWithTweetDetails\n      .withName(\"tweet_candidate_not_a_reply\"),\n    PredicatesForCandidate.f1CandidateIsNotAReply.applyOnlyToF1CandidateWithTargetAndABDecider\n      .withName(\"f1_candidate_is_not_a_reply\"),\n    PredicatesForCandidate.outOfNetworkTweetCandidateIsNotAReply.applyOnlyToOutOfNetworkTweetCandidateWithTargetAndABDecider\n      .withName(\"out_of_network_tweet_candidate_is_not_a_reply\"),\n    PredicatesForCandidate.outOfNetworkTweetCandidateEnabledCrTag.applyOnlyToOutOfNetworkTweetCandidateWithTargetAndABDecider\n      .withName(\"out_of_network_tweet_candidate_enabled_crtag\"),\n    PredicatesForCandidate.outOfNetworkTweetCandidateEnabledCrtGroup.applyOnlyToOutOfNetworkTweetCandidateWithTargetAndABDecider\n      .withName(\"out_of_network_tweet_candidate_enabled_crt_group\"),\n    OutOfNetworkCandidatesQualityPredicates\n      .oonTweetLengthBasedPrerankingPredicate(characterBased = true)\n      .applyOnlyToOutOfNetworkTweetCandidateWithTargetAndABDecider\n      .withName(\"oon_tweet_char_length_too_short\"),\n    OutOfNetworkCandidatesQualityPredicates\n      .oonTweetLengthBasedPrerankingPredicate(characterBased = false)\n      .applyOnlyToOutOfNetworkTweetCandidateWithTargetAndABDecider\n      .withName(\"oon_tweet_word_length_too_short\"),\n    PredicatesForCandidate\n      .protectedTweetF1ExemptPredicate[\n        TargetUser with TargetABDecider,\n        TweetCandidate with TweetAuthorDetails with TargetInfo[\n          TargetUser with TargetABDecider\n        ]\n      ]\n      .applyOnlyToTweetCandidateWithAuthorDetailsWithTargetABDecider\n      .withName(\"f1_exempt_tweet_author_protected\"),\n  )\n\n  private val SgsPreRankingPredicates = List[NamedPredicate[PushCandidate]](\n    SGSPredicatesForCandidate.authorBeingFollowed.applyOnlyToAuthorBeingFollowPredicates\n      .withName(\"author_not_being_followed\"),\n    SGSPredicatesForCandidate.authorNotBeingDeviceFollowed.applyOnlyToBasicTweetPredicates\n      .withName(\"author_being_device_followed\"),\n    SGSPredicatesForCandidate.recommendedTweetAuthorAcceptableToTargetUser.applyOnlyToBasicTweetPredicates\n      .withName(\"recommended_tweet_author_not_acceptable_to_target_user\"),\n    SGSPredicatesForCandidate.disableInNetworkTweetPredicate.applyOnlyToBasicTweetPredicates\n      .withName(\"enable_in_network_tweet\"),\n    SGSPredicatesForCandidate.disableOutNetworkTweetPredicate.applyOnlyToBasicTweetPredicates\n      .withName(\"enable_out_network_tweet\")\n  )\n\n  private val SeeLessOftenPredicates = List[NamedPredicate[PushCandidate]](\n    NtabCaretClickContFnFatiguePredicate\n      .ntabCaretClickContFnFatiguePredicates(\n      )\n      .withName(\"seelessoften_cont_fn_fatigue\")\n  )\n\n  final def build(): List[NamedPredicate[PushCandidate]] = {\n    TweetPredicates ++\n      CommonPredicates ++\n      SocialProofPredicates ++\n      SgsPreRankingPredicates ++\n      SeeLessOftenPredicates\n  }\n}\n\nobject PreRankingPredicates {\n  def apply(\n    statsReceiver: StatsReceiver\n  ): List[NamedPredicate[PushCandidate]] =\n    new PreRankingPredicatesBuilder()(statsReceiver).build()\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/PredicatesForCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.candidate.MaxTweetAge\nimport com.twitter.frigate.common.candidate.TargetABDecider\nimport com.twitter.frigate.common.predicate.tweet.TweetAuthorPredicates\nimport com.twitter.frigate.common.predicate._\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.common.util.SnowflakeUtils\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.util.CandidateUtil\nimport com.twitter.frigate.thriftscala.ChannelName\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.gizmoduck.thriftscala.UserType\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.hermit.predicate.gizmoduck._\nimport com.twitter.hermit.predicate.socialgraph.Edge\nimport com.twitter.hermit.predicate.socialgraph.MultiEdge\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.hermit.predicate.socialgraph.SocialGraphPredicate\nimport com.twitter.service.metastore.gen.thriftscala.Location\nimport com.twitter.socialgraph.thriftscala.RelationshipType\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\n\nobject PredicatesForCandidate {\n\n  def oldTweetRecsPredicate(implicit stats: StatsReceiver): Predicate[\n    TweetCandidate with RecommendationType with TargetInfo[\n      TargetUser with TargetABDecider with MaxTweetAge\n    ]\n  ] = {\n    val name = \"old_tweet\"\n    Predicate\n      .from[TweetCandidate with RecommendationType with TargetInfo[\n        TargetUser with TargetABDecider with MaxTweetAge\n      ]] { candidate =>\n        {\n          val crt = candidate.commonRecType\n          val defaultAge = if (RecTypes.mrModelingBasedTypes.contains(crt)) {\n            candidate.target.params(PushFeatureSwitchParams.ModelingBasedCandidateMaxTweetAgeParam)\n          } else if (RecTypes.GeoPopTweetTypes.contains(crt)) {\n            candidate.target.params(PushFeatureSwitchParams.GeoPopTweetMaxAgeInHours)\n          } else if (RecTypes.simclusterBasedTweets.contains(crt)) {\n            candidate.target.params(\n              PushFeatureSwitchParams.SimclusterBasedCandidateMaxTweetAgeParam)\n          } else if (RecTypes.detopicTypes.contains(crt)) {\n            candidate.target.params(PushFeatureSwitchParams.DetopicBasedCandidateMaxTweetAgeParam)\n          } else if (RecTypes.f1FirstDegreeTypes.contains(crt)) {\n            candidate.target.params(PushFeatureSwitchParams.F1CandidateMaxTweetAgeParam)\n          } else if (crt == CommonRecommendationType.ExploreVideoTweet) {\n            candidate.target.params(PushFeatureSwitchParams.ExploreVideoTweetAgeParam)\n          } else\n            candidate.target.params(PushFeatureSwitchParams.MaxTweetAgeParam)\n          SnowflakeUtils.isRecent(candidate.tweetId, defaultAge)\n        }\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def tweetIsNotAreply(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[TweetCandidate with TweetDetails] = {\n    val name = \"tweet_candidate_not_a_reply\"\n    Predicate\n      .from[TweetCandidate with TweetDetails] { c =>\n        c.isReply match {\n          case Some(true) => false\n          case _ => true\n        }\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  /**\n   * Check if tweet contains any optouted free form interests.\n   * Currently, we use it for media categories and semantic core\n   * @param stats\n   * @return\n   */\n  def noOptoutFreeFormInterestPredicate(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"free_form_interest_opt_out\"\n    val tweetMediaAnnotationFeature =\n      \"tweet.mediaunderstanding.tweet_annotations.safe_category_probabilities\"\n    val tweetSemanticCoreFeature =\n      \"tweet.core.tweet.semantic_core_annotations\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    val withOptOutFreeFormInterestsCounter = stats.counter(\"with_optout_interests\")\n    val withoutOptOutInterestsCounter = stats.counter(\"without_optout_interests\")\n    val withOptOutFreeFormInterestsFromMediaAnnotationCounter =\n      stats.counter(\"with_optout_interests_from_media_annotation\")\n    val withOptOutFreeFormInterestsFromSemanticCoreCounter =\n      stats.counter(\"with_optout_interests_from_semantic_core\")\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        val tweetSemanticCoreEntityIds = candidate.sparseBinaryFeatures\n          .getOrElse(tweetSemanticCoreFeature, Set.empty[String]).map { id =>\n            id.split('.')(2)\n          }.toSet\n        val tweetMediaAnnotationIds = candidate.sparseContinuousFeatures\n          .getOrElse(tweetMediaAnnotationFeature, Map.empty[String, Double]).keys.toSet\n\n        candidate.target.optOutFreeFormUserInterests.map {\n          case optOutUserInterests: Seq[String] =>\n            withOptOutFreeFormInterestsCounter.incr()\n            val optOutUserInterestsSet = optOutUserInterests.toSet\n            val mediaAnnoIntersect = optOutUserInterestsSet.intersect(tweetMediaAnnotationIds)\n            val semanticCoreIntersect = optOutUserInterestsSet.intersect(tweetSemanticCoreEntityIds)\n            if (!mediaAnnoIntersect.isEmpty) {\n              withOptOutFreeFormInterestsFromMediaAnnotationCounter.incr()\n            }\n            if (!semanticCoreIntersect.isEmpty) {\n              withOptOutFreeFormInterestsFromSemanticCoreCounter.incr()\n            }\n            semanticCoreIntersect.isEmpty && mediaAnnoIntersect.isEmpty\n          case _ =>\n            withoutOptOutInterestsCounter.incr()\n            true\n        }\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  def tweetCandidateWithLessThan2SocialContextsIsAReply(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[TweetCandidate with TweetDetails with SocialContextActions] = {\n    val name = \"tweet_candidate_with_less_than_2_social_contexts_is_not_a_reply\"\n    Predicate\n      .from[TweetCandidate with TweetDetails with SocialContextActions] { cand =>\n        cand.isReply match {\n          case Some(true) if cand.socialContextTweetIds.size < 2 => false\n          case _ => true\n        }\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def f1CandidateIsNotAReply(implicit stats: StatsReceiver): NamedPredicate[F1Candidate] = {\n    val name = \"f1_candidate_is_not_a_reply\"\n    Predicate\n      .from[F1Candidate] { candidate =>\n        candidate.isReply match {\n          case Some(true) => false\n          case _ => true\n        }\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def outOfNetworkTweetCandidateEnabledCrTag(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[OutOfNetworkTweetCandidate with TargetInfo[TargetUser with TargetABDecider]] = {\n    val name = \"out_of_network_tweet_candidate_enabled_crtag\"\n    val scopedStats = stats.scope(name)\n    Predicate\n      .from[OutOfNetworkTweetCandidate with TargetInfo[TargetUser with TargetABDecider]] { cand =>\n        val disabledCrTag = cand.target\n          .params(PushFeatureSwitchParams.OONCandidatesDisabledCrTagParam)\n        val candGeneratedByDisabledSignal = cand.tagsCR.exists { tagsCR =>\n          val tagsCRSet = tagsCR.map(_.toString).toSet\n          tagsCRSet.nonEmpty && tagsCRSet.subsetOf(disabledCrTag.toSet)\n        }\n        if (candGeneratedByDisabledSignal) {\n          cand.tagsCR.getOrElse(Nil).foreach(tag => scopedStats.counter(tag.toString).incr())\n          false\n        } else true\n      }\n      .withStats(scopedStats)\n      .withName(name)\n  }\n\n  def outOfNetworkTweetCandidateEnabledCrtGroup(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[OutOfNetworkTweetCandidate with TargetInfo[TargetUser with TargetABDecider]] = {\n    val name = \"out_of_network_tweet_candidate_enabled_crt_group\"\n    val scopedStats = stats.scope(name)\n    Predicate\n      .from[OutOfNetworkTweetCandidate with TargetInfo[TargetUser with TargetABDecider]] { cand =>\n        val disabledCrtGroup = cand.target\n          .params(PushFeatureSwitchParams.OONCandidatesDisabledCrtGroupParam)\n        val crtGroup = CandidateUtil.getCrtGroup(cand.commonRecType)\n        val candGeneratedByDisabledCrt = disabledCrtGroup.contains(crtGroup)\n        if (candGeneratedByDisabledCrt) {\n          scopedStats.counter(\"filter_\" + crtGroup.toString).incr()\n          false\n        } else true\n      }\n      .withStats(scopedStats)\n      .withName(name)\n  }\n\n  def outOfNetworkTweetCandidateIsNotAReply(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[OutOfNetworkTweetCandidate] = {\n    val name = \"out_of_network_tweet_candidate_is_not_a_reply\"\n    Predicate\n      .from[OutOfNetworkTweetCandidate] { cand =>\n        cand.isReply match {\n          case Some(true) => false\n          case _ => true\n        }\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def recommendedTweetIsAuthoredBySelf(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate] =\n    Predicate\n      .from[PushCandidate] {\n        case tweetCandidate: PushCandidate with TweetDetails =>\n          tweetCandidate.authorId match {\n            case Some(authorId) => authorId != tweetCandidate.target.targetId\n            case None => true\n          }\n        case _ =>\n          true\n      }\n      .withStats(statsReceiver.scope(\"predicate_self_author\"))\n      .withName(\"self_author\")\n\n  def authorInSocialContext(implicit statsReceiver: StatsReceiver): NamedPredicate[PushCandidate] =\n    Predicate\n      .from[PushCandidate] {\n        case tweetCandidate: PushCandidate with TweetDetails with SocialContextActions =>\n          tweetCandidate.authorId match {\n            case Some(authorId) =>\n              !tweetCandidate.socialContextUserIds.contains(authorId)\n            case None => true\n          }\n        case _ => true\n      }\n      .withStats(statsReceiver.scope(\"predicate_author_social_context\"))\n      .withName(\"author_social_context\")\n\n  def selfInSocialContext(implicit statsReceiver: StatsReceiver): NamedPredicate[PushCandidate] = {\n    val name = \"self_social_context\"\n    Predicate\n      .from[PushCandidate] {\n        case candidate: PushCandidate with SocialContextActions =>\n          !candidate.socialContextUserIds.contains(candidate.target.targetId)\n        case _ =>\n          true\n      }\n      .withStats(statsReceiver.scope(s\"${name}_predicate\"))\n      .withName(name)\n  }\n\n  def minSocialContext(\n    threshold: Int\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with SocialContextActions] = {\n    Predicate\n      .from { candidate: PushCandidate with SocialContextActions =>\n        candidate.socialContextUserIds.size >= threshold\n      }\n      .withStats(statsReceiver.scope(\"predicate_min_social_context\"))\n      .withName(\"min_social_context\")\n  }\n\n  private def anyWithheldContent(\n    userStore: ReadableStore[Long, User],\n    userCountryStore: ReadableStore[Long, Location]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): Predicate[TargetRecUser] =\n    GizmoduckUserPredicate.withheldContentPredicate(\n      userStore = userStore,\n      userCountryStore = userCountryStore,\n      statsReceiver = statsReceiver,\n      checkAllCountries = true\n    )\n\n  def targetUserExists(implicit statsReceiver: StatsReceiver): NamedPredicate[PushCandidate] = {\n    TargetUserPredicates\n      .targetUserExists()(statsReceiver)\n      .flatContraMap { candidate: PushCandidate => Future.value(candidate.target) }\n      .withName(\"target_user_exists\")\n  }\n\n  def secondaryDormantAccountPredicate(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"secondary_dormant_account\"\n    TargetUserPredicates\n      .secondaryDormantAccountPredicate()(statsReceiver)\n      .on { candidate: PushCandidate => candidate.target }\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def socialContextBeingFollowed(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with SocialContextActions] =\n    SocialGraphPredicate\n      .allRelationEdgesExist(edgeStore, RelationshipType.Following)\n      .on { candidate: PushCandidate with SocialContextActions =>\n        candidate.socialContextUserIds.map { u => Edge(candidate.target.targetId, u) }\n      }\n      .withStats(statsReceiver.scope(\"predicate_social_context_being_followed\"))\n      .withName(\"social_context_being_followed\")\n\n  private def edgeFromCandidate(candidate: PushCandidate with TweetAuthor): Option[Edge] = {\n    candidate.authorId map { authorId => Edge(candidate.target.targetId, authorId) }\n  }\n\n  def authorNotBeingDeviceFollowed(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor] = {\n    SocialGraphPredicate\n      .relationExists(edgeStore, RelationshipType.DeviceFollowing)\n      .optionalOn(\n        edgeFromCandidate,\n        missingResult = false\n      )\n      .flip\n      .withStats(statsReceiver.scope(\"predicate_author_not_device_followed\"))\n      .withName(\"author_not_device_followed\")\n  }\n\n  def authorBeingFollowed(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor] = {\n    SocialGraphPredicate\n      .relationExists(edgeStore, RelationshipType.Following)\n      .optionalOn(\n        edgeFromCandidate,\n        missingResult = false\n      )\n      .withStats(statsReceiver.scope(\"predicate_author_being_followed\"))\n      .withName(\"author_being_followed\")\n  }\n\n  def authorNotBeingFollowed(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor] = {\n    SocialGraphPredicate\n      .relationExists(edgeStore, RelationshipType.Following)\n      .optionalOn(\n        edgeFromCandidate,\n        missingResult = false\n      )\n      .flip\n      .withStats(statsReceiver.scope(\"predicate_author_not_being_followed\"))\n      .withName(\"author_not_being_followed\")\n  }\n\n  def recommendedTweetAuthorAcceptableToTargetUser(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor] = {\n    val name = \"recommended_tweet_author_acceptable_to_target_user\"\n    SocialGraphPredicate\n      .anyRelationExists(\n        edgeStore,\n        Set(\n          RelationshipType.Blocking,\n          RelationshipType.BlockedBy,\n          RelationshipType.HideRecommendations,\n          RelationshipType.Muting\n        )\n      )\n      .flip\n      .optionalOn(\n        edgeFromCandidate,\n        missingResult = false\n      )\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def relationNotExistsPredicate(\n    edgeStore: ReadableStore[RelationEdge, Boolean],\n    relations: Set[RelationshipType]\n  ): Predicate[(Long, Iterable[Long])] =\n    SocialGraphPredicate\n      .anyRelationExistsForMultiEdge(\n        edgeStore,\n        relations\n      )\n      .flip\n      .on {\n        case (targetUserId, userIds) =>\n          MultiEdge(targetUserId, userIds.toSet)\n      }\n\n  def blocking(edgeStore: ReadableStore[RelationEdge, Boolean]): Predicate[(Long, Iterable[Long])] =\n    relationNotExistsPredicate(\n      edgeStore,\n      Set(RelationshipType.BlockedBy, RelationshipType.Blocking)\n    )\n\n  def blockingOrMuting(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  ): Predicate[(Long, Iterable[Long])] =\n    relationNotExistsPredicate(\n      edgeStore,\n      Set(RelationshipType.BlockedBy, RelationshipType.Blocking, RelationshipType.Muting)\n    )\n\n  def socialContextNotRetweetFollowing(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with SocialContextActions] = {\n    val name = \"social_context_not_retweet_following\"\n    relationNotExistsPredicate(edgeStore, Set(RelationshipType.NotRetweetFollowing))\n      .optionalOn[PushCandidate with SocialContextActions](\n        {\n          case candidate: PushCandidate with SocialContextActions\n              if RecTypes.isTweetRetweetType(candidate.commonRecType) =>\n            Some((candidate.target.targetId, candidate.socialContextUserIds))\n          case _ =>\n            None\n        },\n        missingResult = true\n      )\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def socialContextBlockingOrMuting(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with SocialContextActions] =\n    blockingOrMuting(edgeStore)\n      .on { candidate: PushCandidate with SocialContextActions =>\n        (candidate.target.targetId, candidate.socialContextUserIds)\n      }\n      .withStats(statsReceiver.scope(\"predicate_social_context_blocking_or_muting\"))\n      .withName(\"social_context_blocking_or_muting\")\n\n  /**\n   * Use hyrated Tweet object for F1 Protected experiment for checking null cast as Tweetypie hydration\n   * fails for protected Authors without passing in Target id. We do this specifically for\n   * F1 Protected Tweet Experiment in Earlybird Adaptor.\n   * For rest of the traffic refer to existing Nullcast Predicate\n   */\n  def nullCastF1ProtectedExperientPredicate(\n    tweetypieStore: ReadableStore[Long, TweetyPieResult]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetCandidate with TweetDetails] = {\n    val name = \"f1_exempted_null_cast_tweet\"\n    val f1NullCastCheckCounter = statsReceiver.scope(name).counter(\"f1_null_cast_check\")\n    Predicate\n      .fromAsync { tweetCandidate: PushCandidate with TweetCandidate with TweetDetails =>\n        if (RecTypes.f1FirstDegreeTypes(tweetCandidate.commonRecType) && tweetCandidate.target\n            .params(PushFeatureSwitchParams.EnableF1FromProtectedTweetAuthors)) {\n          f1NullCastCheckCounter.incr()\n          tweetCandidate.tweet match {\n            case Some(tweetObj) =>\n              baseNullCastTweet().apply(Seq(TweetyPieResult(tweetObj, None, None))).map(_.head)\n            case _ => Future.False\n          }\n        } else {\n          nullCastTweet(tweetypieStore).apply(Seq(tweetCandidate)).map(_.head)\n        }\n      }\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  private def baseNullCastTweet(): Predicate[TweetyPieResult] =\n    Predicate.from { t: TweetyPieResult => !t.tweet.coreData.exists { cd => cd.nullcast } }\n\n  def nullCastTweet(\n    tweetyPieStore: ReadableStore[Long, TweetyPieResult]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetCandidate] = {\n    val name = \"null_cast_tweet\"\n    baseNullCastTweet()\n      .flatOptionContraMap[PushCandidate with TweetCandidate](\n        f = (tweetCandidate: PushCandidate\n          with TweetCandidate) => tweetyPieStore.get(tweetCandidate.tweetId),\n        missingResult = false\n      )\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  /**\n   * Use the predicate except fn is true.\n   */\n  def exceptedPredicate[T <: PushCandidate](\n    name: String,\n    fn: T => Future[Boolean],\n    predicate: Predicate[T]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n    Predicate\n      .fromAsync { e: T => fn(e) }\n      .or(predicate)\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  /**\n   *\n   * @param edgeStore [[ReadableStore[RelationEdge, Boolean]]]\n   * @return - allow only out-network tweets if in-network tweets are disabled\n   */\n  def disableInNetworkTweetPredicate(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor] = {\n    val name = \"disable_in_network_tweet\"\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetAuthor =>\n        if (candidate.target.params(PushParams.DisableInNetworkTweetCandidatesParam)) {\n          authorNotBeingFollowed(edgeStore)\n            .apply(Seq(candidate))\n            .map(_.head)\n        } else Future.True\n      }.withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  /**\n   *\n   * @param edgeStore [[ReadableStore[RelationEdge, Boolean]]]\n   * @return - allow only in-network tweets if out-network tweets are disabled\n   */\n  def disableOutNetworkTweetPredicate(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor] = {\n    val name = \"disable_out_network_tweet\"\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetAuthor =>\n        if (candidate.target.params(PushFeatureSwitchParams.DisableOutNetworkTweetCandidatesFS)) {\n          authorBeingFollowed(edgeStore)\n            .apply(Seq(candidate))\n            .map(_.head)\n        } else Future.True\n      }.withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  def alwaysTruePredicate: NamedPredicate[PushCandidate] = {\n    Predicate\n      .all[PushCandidate]\n      .withName(\"predicate_AlwaysTrue\")\n  }\n\n  def alwaysTruePushCandidatePredicate: NamedPredicate[PushCandidate] = {\n    Predicate\n      .all[PushCandidate]\n      .withName(\"predicate_AlwaysTrue\")\n  }\n\n  def alwaysFalsePredicate(implicit statsReceiver: StatsReceiver): NamedPredicate[PushCandidate] = {\n    val name = \"predicate_AlwaysFalse\"\n    val scopedStatsReceiver = statsReceiver.scope(name)\n    Predicate\n      .from { candidate: PushCandidate => false }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  def accountCountryPredicate(\n    allowedCountries: Set[String]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"AccountCountryPredicate\"\n    val stats = statsReceiver.scope(name)\n    AccountCountryPredicate(allowedCountries)\n      .on { candidate: PushCandidate => candidate.target }\n      .withStats(stats)\n      .withName(name)\n  }\n\n  def paramPredicate[T <: PushCandidate](\n    param: Param[Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n    val name = param.getClass.getSimpleName.stripSuffix(\"$\")\n    TargetPredicates\n      .paramPredicate(param)\n      .on { candidate: PushCandidate => candidate.target }\n      .withStats(statsReceiver.scope(s\"param_${name}_controlled_predicate\"))\n      .withName(s\"param_${name}_controlled_predicate\")\n  }\n\n  def isDeviceEligibleForNewsOrSports(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"is_device_eligible_for_news_or_sports\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        candidate.target.deviceInfo.map(_.exists(_.isNewsEligible))\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  def isDeviceEligibleForCreatorPush(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"is_device_eligible_for_creator_push\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        candidate.target.deviceInfo.map(_.exists(settings =>\n          settings.isNewsEligible || settings.isRecommendationsEligible))\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  /**\n   * Like [[TargetUserPredicates.homeTimelineFatigue()]] but for candidate.\n   */\n  def htlFatiguePredicate(\n    fatigueDuration: Param[Duration]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"htl_fatigue\"\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        val _fatigueDuration = candidate.target.params(fatigueDuration)\n        TargetUserPredicates\n          .homeTimelineFatigue(\n            fatigueDuration = _fatigueDuration\n          ).apply(Seq(candidate.target)).map(_.head)\n      }\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  def mrWebHoldbackPredicate(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"mr_web_holdback_for_candidate\"\n    val scopedStats = stats.scope(name)\n    PredicatesForCandidate.exludeCrtFromPushHoldback\n      .or(\n        TargetPredicates\n          .webNotifsHoldback()\n          .on { candidate: PushCandidate => candidate.target }\n      )\n      .withStats(scopedStats)\n      .withName(name)\n  }\n\n  def candidateEnabledForEmailPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"candidates_enabled_for_email\"\n    Predicate\n      .from { candidate: PushCandidate =>\n        if (candidate.target.isEmailUser)\n          candidate.isInstanceOf[TweetCandidate with TweetAuthor with RecommendationType]\n        else true\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def protectedTweetF1ExemptPredicate[\n    T <: TargetUser with TargetABDecider,\n    Cand <: TweetCandidate with TweetAuthorDetails with TargetInfo[T]\n  ](\n    implicit stats: StatsReceiver\n  ): NamedPredicate[\n    TweetCandidate with TweetAuthorDetails with TargetInfo[\n      TargetUser with TargetABDecider\n    ]\n  ] = {\n    val name = \"f1_exempt_tweet_author_protected\"\n    val skipForProtectedAuthorScope = stats.scope(name).scope(\"skip_protected_author_for_f1\")\n    val authorIsProtectedCounter = skipForProtectedAuthorScope.counter(\"author_protected_true\")\n    val authorIsNotProtectedCounter = skipForProtectedAuthorScope.counter(\"author_protected_false\")\n    val authorNotFoundCounter = stats.scope(name).counter(\"author_not_found\")\n    Predicate\n      .fromAsync[TweetCandidate with TweetAuthorDetails with TargetInfo[\n        TargetUser with TargetABDecider\n      ]] {\n        case candidate: F1Candidate\n            if candidate.target.params(PushFeatureSwitchParams.EnableF1FromProtectedTweetAuthors) =>\n          candidate.tweetAuthor.foreach {\n            case Some(author) =>\n              if (GizmoduckUserPredicate.isProtected(author)) {\n                authorIsProtectedCounter.incr()\n              } else authorIsNotProtectedCounter.incr()\n            case _ => authorNotFoundCounter.incr()\n          }\n          Future.True\n        case cand =>\n          TweetAuthorPredicates.recTweetAuthorProtected.apply(Seq(cand)).map(_.head)\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  /**\n   * filter a notification if user has already received ANY prior notification about the space id\n   * @param stats\n   * @return\n   */\n  def duplicateSpacesPredicate(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[Space with PushCandidate] = {\n    val name = \"duplicate_spaces_predicate\"\n    Predicate\n      .fromAsync { c: Space with PushCandidate =>\n        c.target.pushRecItems.map { pushRecItems =>\n          !pushRecItems.spaceIds.contains(c.spaceId)\n        }\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def filterOONCandidatePredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"filter_oon_candidate\"\n\n    Predicate\n      .fromAsync[PushCandidate] { cand =>\n        val crt = cand.commonRecType\n        val isOONCandidate =\n          RecTypes.isOutOfNetworkTweetRecType(crt) || RecTypes.outOfNetworkTopicTweetTypes\n            .contains(crt) || RecTypes.isOutOfNetworkSpaceType(crt) || RecTypes.userTypes.contains(\n            crt)\n        if (isOONCandidate) {\n          cand.target.notificationsFromOnlyPeopleIFollow.map { inNetworkOnly =>\n            if (inNetworkOnly) {\n              stats.scope(name, crt.toString).counter(\"inNetworkOnlyOn\").incr()\n            } else {\n              stats.scope(name, crt.toString).counter(\"inNetworkOnlyOff\").incr()\n            }\n            !(inNetworkOnly && cand.target.params(\n              PushFeatureSwitchParams.EnableOONFilteringBasedOnUserSettings))\n          }\n        } else Future.True\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def exludeCrtFromPushHoldback(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = Predicate\n    .from { candidate: PushCandidate =>\n      val crtName = candidate.commonRecType.name\n      val target = candidate.target\n      target\n        .params(PushFeatureSwitchParams.CommonRecommendationTypeDenyListPushHoldbacks)\n        .exists(crtName.equalsIgnoreCase)\n    }\n    .withStats(stats.scope(\"exclude_crt_from_push_holdbacks\"))\n\n  def enableSendHandlerCandidates(implicit stats: StatsReceiver): NamedPredicate[PushCandidate] = {\n    val name = \"sendhandler_enable_push_recommendations\"\n    PredicatesForCandidate.exludeCrtFromPushHoldback\n      .or(PredicatesForCandidate.paramPredicate(\n        PushFeatureSwitchParams.EnablePushRecommendationsParam))\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def openAppExperimentUserCandidateAllowList(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"open_app_experiment_user_candidate_allow_list\"\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        val target = candidate.target\n        Future.join(target.isOpenAppExperimentUser, target.targetUser).map {\n          case (isOpenAppUser, targetUser) =>\n            val shouldLimitOpenAppCrts =\n              isOpenAppUser || targetUser.exists(_.userType == UserType.Soft)\n\n            if (shouldLimitOpenAppCrts) {\n              val listOfAllowedCrt = target\n                .params(PushFeatureSwitchParams.ListOfCrtsForOpenApp)\n                .flatMap(CommonRecommendationType.valueOf)\n              listOfAllowedCrt.contains(candidate.commonRecType)\n            } else true\n        }\n      }.withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def isTargetBlueVerified(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"is_target_already_blue_verified\"\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        val target = candidate.target\n        target.isBlueVerified.map(_.getOrElse(false))\n      }.withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def isTargetLegacyVerified(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"is_target_already_legacy_verified\"\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        val target = candidate.target\n        target.isVerified.map(_.getOrElse(false))\n      }.withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def isTargetSuperFollowCreator(implicit stats: StatsReceiver): NamedPredicate[PushCandidate] = {\n    val name = \"is_target_already_super_follow_creator\"\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        val target = candidate.target\n        target.isSuperFollowCreator.map(\n          _.getOrElse(false)\n        )\n      }.withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def isChannelValidPredicate(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"is_channel_valid\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        candidate\n          .getChannels().map(channels =>\n            !(channels.toSet.size == 1 && channels.head == ChannelName.None))\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/SGSPredicatesForCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.SocialGraphServiceRelationshipMap\nimport com.twitter.frigate.common.base.TweetAuthor\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.gizmoduck.thriftscala.UserType\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.hermit.predicate.socialgraph.Edge\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.socialgraph.thriftscala.RelationshipType\nimport com.twitter.util.Future\n\n/**\n * Refactor SGS predicates so that predicates can use relationshipMap we generate in hydrate step\n */\nobject SGSPredicatesForCandidate {\n\n  case class RelationshipMapEdge(edge: Edge, relationshipMap: Map[RelationEdge, Boolean])\n\n  private def relationshipMapEdgeFromCandidate(\n    candidate: PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap\n  ): Option[RelationshipMapEdge] = {\n    candidate.authorId map { authorId =>\n      RelationshipMapEdge(Edge(candidate.target.targetId, authorId), candidate.relationshipMap)\n    }\n  }\n\n  def authorBeingFollowed(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap] = {\n    val name = \"author_not_being_followed\"\n    val stats = statsReceiver.scope(name)\n    val softUserCounter = stats.counter(\"soft_user\")\n\n    val sgsAuthorBeingFollowedPredicate = Predicate\n      .from { relationshipMapEdge: RelationshipMapEdge =>\n        anyRelationExist(relationshipMapEdge, Set(RelationshipType.Following))\n      }\n\n    Predicate\n      .fromAsync {\n        candidate: PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap =>\n          val target = candidate.target\n          target.targetUser.flatMap {\n            case Some(gizmoduckUser) if gizmoduckUser.userType == UserType.Soft =>\n              softUserCounter.incr()\n              target.seedsWithWeight.map { followedUsersWithWeightOpt =>\n                candidate.authorId match {\n                  case Some(authorId) =>\n                    val followedUsers = followedUsersWithWeightOpt.getOrElse(Map.empty).keys\n                    followedUsers.toSet.contains(authorId)\n\n                  case None => false\n                }\n              }\n\n            case _ =>\n              sgsAuthorBeingFollowedPredicate\n                .optionalOn(relationshipMapEdgeFromCandidate, missingResult = false)\n                .apply(Seq(candidate))\n                .map(_.head)\n          }\n      }.withStats(stats)\n      .withName(name)\n  }\n\n  def authorNotBeingDeviceFollowed(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap] = {\n    val name = \"author_being_device_followed\"\n    Predicate\n      .from { relationshipMapEdge: RelationshipMapEdge =>\n        {\n          anyRelationExist(relationshipMapEdge, Set(RelationshipType.DeviceFollowing))\n        }\n      }\n      .optionalOn(relationshipMapEdgeFromCandidate, missingResult = false)\n      .flip\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  def recommendedTweetAuthorAcceptableToTargetUser(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap] = {\n    val name = \"recommended_tweet_author_not_acceptable_to_target_user\"\n    Predicate\n      .from { relationshipMapEdge: RelationshipMapEdge =>\n        {\n          anyRelationExist(\n            relationshipMapEdge,\n            Set(\n              RelationshipType.Blocking,\n              RelationshipType.BlockedBy,\n              RelationshipType.HideRecommendations,\n              RelationshipType.Muting\n            ))\n        }\n      }\n      .flip\n      .optionalOn(relationshipMapEdgeFromCandidate, missingResult = false)\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  def authorNotBeingFollowed(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap] = {\n    Predicate\n      .from { relationshipMapEdge: RelationshipMapEdge =>\n        {\n          anyRelationExist(relationshipMapEdge, Set(RelationshipType.Following))\n        }\n      }\n      .optionalOn(relationshipMapEdgeFromCandidate, missingResult = false)\n      .flip\n      .withStats(statsReceiver.scope(\"predicate_author_not_being_followed_pre_ranking\"))\n      .withName(\"author_not_being_followed\")\n  }\n\n  def disableInNetworkTweetPredicate(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap] = {\n    val name = \"enable_in_network_tweet\"\n    Predicate\n      .fromAsync {\n        candidate: PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap =>\n          if (candidate.target.params(PushParams.DisableInNetworkTweetCandidatesParam)) {\n            authorNotBeingFollowed\n              .apply(Seq(candidate))\n              .map(_.head)\n          } else Future.True\n      }.withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  def disableOutNetworkTweetPredicate(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap] = {\n    val name = \"enable_out_network_tweet\"\n    Predicate\n      .fromAsync {\n        candidate: PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap =>\n          if (candidate.target.params(PushFeatureSwitchParams.DisableOutNetworkTweetCandidatesFS)) {\n            authorBeingFollowed\n              .apply(Seq(candidate))\n              .map(_.head)\n          } else Future.True\n      }.withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  /**\n   * Returns true if the provided relationshipEdge exists among\n   * @param candidate candidate\n   * @param relationships relaionships\n   * @return Boolean result\n   */\n  private def anyRelationExist(\n    relationshipMapEdge: RelationshipMapEdge,\n    relationships: Set[RelationshipType]\n  ): Boolean = {\n    val resultSeq = relationships.map { relationship =>\n      relationshipMapEdge.relationshipMap.getOrElse(\n        RelationEdge(relationshipMapEdge.edge, relationship),\n        false)\n    }.toSeq\n    resultSeq.contains(true)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/ScarecrowPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.thriftscala._\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.scarecrow.{ScarecrowPredicate => HermitScarecrowPredicate}\nimport com.twitter.relevance.feature_store.thriftscala.FeatureData\nimport com.twitter.relevance.feature_store.thriftscala.FeatureValue\nimport com.twitter.service.gen.scarecrow.thriftscala.Event\nimport com.twitter.service.gen.scarecrow.thriftscala.TieredActionResult\nimport com.twitter.storehaus.ReadableStore\n\nobject ScarecrowPredicate {\n  val name = \"\"\n\n  def candidateToEvent(candidate: PushCandidate): Event = {\n    val recommendedUserIdOpt = candidate match {\n      case tweetCandidate: TweetCandidate with TweetAuthor =>\n        tweetCandidate.authorId\n      case userCandidate: UserCandidate =>\n        Some(userCandidate.userId)\n      case _ => None\n    }\n    val hashtagsInTweet = candidate match {\n      case tweetCandidate: TweetCandidate with TweetDetails =>\n        tweetCandidate.tweetyPieResult\n          .flatMap { tweetPieResult =>\n            tweetPieResult.tweet.hashtags.map(_.map(_.text))\n          }.getOrElse(Nil)\n      case _ =>\n        Nil\n    }\n    val urlsInTweet = candidate match {\n      case tweetCandidate: TweetCandidate with TweetDetails =>\n        tweetCandidate.tweetyPieResult\n          .flatMap { tweetPieResult =>\n            tweetPieResult.tweet.urls.map(_.flatMap(_.expanded))\n          }\n      case _ => None\n    }\n    val tweetIdOpt = candidate match {\n      case tweetCandidate: TweetCandidate =>\n        Some(tweetCandidate.tweetId)\n      case _ =>\n        None\n    }\n    val urlOpt = candidate match {\n      case candidate: UrlCandidate =>\n        Some(candidate.url)\n      case _ =>\n        None\n    }\n    val scUserIds = candidate match {\n      case hasSocialContext: SocialContextActions => Some(hasSocialContext.socialContextUserIds)\n      case _ => None\n    }\n\n    val eventTitleOpt = candidate match {\n      case eventCandidate: EventCandidate with EventDetails =>\n        Some(eventCandidate.eventTitle)\n      case _ =>\n        None\n    }\n\n    val urlTitleOpt = candidate match {\n      case candidate: UrlCandidate =>\n        candidate.title\n      case _ =>\n        None\n    }\n\n    val urlDescriptionOpt = candidate match {\n      case candidate: UrlCandidate with UrlCandidateWithDetails =>\n        candidate.description\n      case _ =>\n        None\n    }\n\n    Event(\n      \"magicrecs_recommendation_write\",\n      Map(\n        \"targetUserId\" -> FeatureData(Some(FeatureValue.LongValue(candidate.target.targetId))),\n        \"type\" -> FeatureData(\n          Some(\n            FeatureValue.StrValue(candidate.commonRecType.name)\n          )\n        ),\n        \"recommendedUserId\" -> FeatureData(recommendedUserIdOpt map { id =>\n          FeatureValue.LongValue(id)\n        }),\n        \"tweetId\" -> FeatureData(tweetIdOpt map { id =>\n          FeatureValue.LongValue(id)\n        }),\n        \"url\" -> FeatureData(urlOpt map { url =>\n          FeatureValue.StrValue(url)\n        }),\n        \"hashtagsInTweet\" -> FeatureData(Some(FeatureValue.StrListValue(hashtagsInTweet))),\n        \"urlsInTweet\" -> FeatureData(urlsInTweet.map(FeatureValue.StrListValue)),\n        \"socialContexts\" -> FeatureData(scUserIds.map { sc =>\n          FeatureValue.LongListValue(sc)\n        }),\n        \"eventTitle\" -> FeatureData(eventTitleOpt.map { eventTitle =>\n          FeatureValue.StrValue(eventTitle)\n        }),\n        \"urlTitle\" -> FeatureData(urlTitleOpt map { title =>\n          FeatureValue.StrValue(title)\n        }),\n        \"urlDescription\" -> FeatureData(urlDescriptionOpt map { des =>\n          FeatureValue.StrValue(des)\n        })\n      )\n    )\n  }\n\n  def candidateToPossibleEvent(c: PushCandidate): Option[Event] = {\n    if (c.frigateNotification.notificationDisplayLocation == NotificationDisplayLocation.PushToMobileDevice) {\n      Some(candidateToEvent(c))\n    } else {\n      None\n    }\n  }\n\n  def apply(\n    scarecrowCheckEventStore: ReadableStore[Event, TieredActionResult]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    HermitScarecrowPredicate(scarecrowCheckEventStore)\n      .optionalOn(\n        candidateToPossibleEvent,\n        missingResult = true\n      )\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/SpacePredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.SpaceCandidate\nimport com.twitter.frigate.common.base.SpaceCandidateDetails\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.hermit.predicate.socialgraph.Edge\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.hermit.predicate.socialgraph.SocialGraphPredicate\nimport com.twitter.socialgraph.thriftscala.RelationshipType\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.response.Err\nimport com.twitter.ubs.thriftscala.AudioSpace\nimport com.twitter.ubs.thriftscala.BroadcastState\nimport com.twitter.ubs.thriftscala.ParticipantUser\nimport com.twitter.ubs.thriftscala.Participants\nimport com.twitter.util.Future\n\nobject SpacePredicate {\n\n  /** Filters the request if the target is present in the space as a listener, speakeTestConfigr, or admin */\n  def targetInSpace(\n    audioSpaceParticipantsStore: ReadableStore[String, Participants]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[SpaceCandidateDetails with RawCandidate] = {\n    val name = \"target_in_space\"\n    Predicate\n      .fromAsync[SpaceCandidateDetails with RawCandidate] { spaceCandidate =>\n        audioSpaceParticipantsStore.get(spaceCandidate.spaceId).map {\n          case Some(participants) =>\n            val allParticipants: Seq[ParticipantUser] =\n              (participants.admins ++ participants.speakers ++ participants.listeners).flatten.toSeq\n            val isInSpace = allParticipants.exists { participant =>\n              participant.twitterUserId.contains(spaceCandidate.target.targetId)\n            }\n            !isInSpace\n          case None => false\n        }\n      }.withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  /**\n   *\n   * @param audioSpaceStore: space metadata store\n   * @param statsReceiver: record stats\n   * @return: true if the space not started ELSE false to filter out notification\n   */\n  def scheduledSpaceStarted(\n    audioSpaceStore: ReadableStore[String, AudioSpace]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[SpaceCandidate with RawCandidate] = {\n    val name = \"scheduled_space_started\"\n    Predicate\n      .fromAsync[SpaceCandidate with RawCandidate] { spaceCandidate =>\n        audioSpaceStore\n          .get(spaceCandidate.spaceId)\n          .map(_.exists(_.state.contains(BroadcastState.NotStarted)))\n          .rescue {\n            case Err(Err.Authorization, _, _) =>\n              Future.False\n          }\n      }\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  private def relationshipMapEdgeFromSpaceCandidate(\n    candidate: RawCandidate with SpaceCandidate\n  ): Option[(Long, Seq[Long])] = {\n    candidate.hostId.map { spaceHostId =>\n      (candidate.target.targetId, Seq(spaceHostId))\n    }\n  }\n\n  /**\n   * Check only host block for scheduled space reminders\n   * @return: True if no blocking relation between host and target user, else False\n   */\n  def spaceHostTargetUserBlocking(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[SpaceCandidate with RawCandidate] = {\n    val name = \"space_host_target_user_blocking\"\n    PredicatesForCandidate\n      .blocking(edgeStore)\n      .optionalOn(relationshipMapEdgeFromSpaceCandidate, false)\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  private def edgeFromCandidate(\n    candidate: PushCandidate with TweetAuthorDetails\n  ): Future[Option[Edge]] = {\n    candidate.tweetAuthor.map(_.map { author => Edge(candidate.target.targetId, author.id) })\n  }\n\n  def recommendedTweetAuthorAcceptableToTargetUser(\n    edgeStore: ReadableStore[RelationEdge, Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetAuthorDetails] = {\n    val name = \"recommended_tweet_author_acceptable_to_target_user\"\n    SocialGraphPredicate\n      .anyRelationExists(\n        edgeStore,\n        Set(\n          RelationshipType.Blocking,\n          RelationshipType.BlockedBy,\n          RelationshipType.HideRecommendations,\n          RelationshipType.Muting\n        )\n      )\n      .flip\n      .flatOptionContraMap(\n        edgeFromCandidate,\n        missingResult = false\n      )\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def narrowCastSpace(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[SpaceCandidateDetails with RawCandidate] = {\n    val name = \"narrow_cast_space\"\n    val narrowCastSpaceScope = statsReceiver.scope(name)\n    val employeeSpaceCounter = narrowCastSpaceScope.counter(\"employees\")\n    val superFollowerSpaceCounter = narrowCastSpaceScope.counter(\"super_followers\")\n\n    Predicate\n      .fromAsync[SpaceCandidateDetails with RawCandidate] { candidate =>\n        candidate.audioSpaceFut.map {\n          case Some(audioSpace) if audioSpace.narrowCastSpaceType.contains(1L) =>\n            employeeSpaceCounter.incr()\n            candidate.target.params(PushFeatureSwitchParams.EnableEmployeeOnlySpaceNotifications)\n          case Some(audioSpace) if audioSpace.narrowCastSpaceType.contains(2L) =>\n            superFollowerSpaceCounter.incr()\n            false\n          case _ => true\n        }\n      }.withStats(narrowCastSpaceScope)\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/TargetEngagementPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.tweetypie.EngagementsPredicate\nimport com.twitter.hermit.predicate.tweetypie.Perspective\nimport com.twitter.hermit.predicate.tweetypie.UserTweet\nimport com.twitter.storehaus.ReadableStore\n\nobject TargetEngagementPredicate {\n  val name = \"target_engagement\"\n  def apply(\n    perspectiveStore: ReadableStore[UserTweet, Perspective],\n    defaultForMissing: Boolean\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetCandidate] = {\n    EngagementsPredicate(perspectiveStore, defaultForMissing)\n      .on { candidate: PushCandidate with TweetCandidate =>\n        UserTweet(candidate.target.targetId, candidate.tweetId)\n      }\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/TargetNtabCaretClickFatiguePredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TargetUser\nimport com.twitter.frigate.common.candidate.CaretFeedbackHistory\nimport com.twitter.frigate.common.candidate.FrigateHistory\nimport com.twitter.frigate.common.candidate.HTLVisitHistory\nimport com.twitter.frigate.common.candidate.TargetABDecider\nimport com.twitter.frigate.common.history.History\nimport com.twitter.frigate.common.predicate.FrigateHistoryFatiguePredicate.TimeSeries\nimport com.twitter.frigate.common.predicate.ntab_caret_fatigue.NtabCaretClickFatiguePredicateHelper\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.common.util.FeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.notificationservice.thriftscala.CaretFeedbackDetails\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.frigate.common.predicate.{FatiguePredicate => CommonFatiguePredicate}\n\nobject TargetNtabCaretClickFatiguePredicate {\n  import NtabCaretClickFatiguePredicateHelper._\n\n  private val MagicRecsCategory = \"MagicRecs\"\n\n  def apply[\n    T <: TargetUser with TargetABDecider with CaretFeedbackHistory with FrigateHistory with HTLVisitHistory\n  ](\n    filterHistory: TimeSeries => TimeSeries =\n      CommonFatiguePredicate.recTypesOnlyFilter(RecTypes.sharedNTabCaretFatigueTypes),\n    filterCaretFeedbackHistory: TargetUser with TargetABDecider with CaretFeedbackHistory => Seq[\n      CaretFeedbackDetails\n    ] => Seq[CaretFeedbackDetails] =\n      CaretFeedbackHistoryFilter.caretFeedbackHistoryFilter(Seq(MagicRecsCategory)),\n    calculateFatiguePeriod: Seq[CaretFeedbackDetails] => Duration = calculateFatiguePeriodMagicRecs,\n    useMostRecentDislikeTime: Boolean = false,\n    name: String = \"NtabCaretClickFatiguePredicate\"\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n\n    val scopedStats = statsReceiver.scope(name)\n    val crtStats = scopedStats.scope(\"crt\")\n    Predicate\n      .fromAsync { target: T =>\n        Future.join(target.history, target.caretFeedbacks).map {\n          case (history, Some(feedbackDetails)) => {\n            val feedbackDetailsDeduped = dedupFeedbackDetails(\n              filterCaretFeedbackHistory(target)(feedbackDetails),\n              scopedStats\n            )\n\n            val fatiguePeriod =\n              if (hasUserDislikeInLast30Days(feedbackDetailsDeduped) && target.params(\n                  PushFeatureSwitchParams.EnableReducedFatigueRulesForSeeLessOften)) {\n                durationToFilterMRForSeeLessOftenExpt(\n                  feedbackDetailsDeduped,\n                  target.params(FeatureSwitchParams.NumberOfDaysToFilterMRForSeeLessOften),\n                  target.params(FeatureSwitchParams.NumberOfDaysToReducePushCapForSeeLessOften),\n                  scopedStats\n                )\n              } else {\n                calculateFatiguePeriod(feedbackDetailsDeduped)\n              }\n\n            val crtlist = feedbackDetailsDeduped\n              .flatMap { fd =>\n                fd.genericNotificationMetadata.map { gm =>\n                  gm.genericType.name\n                }\n              }.distinct.sorted.mkString(\"-\")\n\n            if (fatiguePeriod > 0.days) {\n              crtStats.scope(crtlist).counter(\"fatigued\").incr()\n            } else {\n              crtStats.scope(crtlist).counter(\"non_fatigued\").incr()\n            }\n\n            val hasRecentSent =\n              hasRecentSend(History(filterHistory(history.history.toSeq).toMap), fatiguePeriod)\n            !hasRecentSent\n          }\n          case _ => true\n        }\n      }\n      .withStats(scopedStats)\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/TargetPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TargetUser\nimport com.twitter.frigate.common.candidate.FrigateHistory\nimport com.twitter.frigate.common.candidate.HTLVisitHistory\nimport com.twitter.frigate.common.candidate.TargetABDecider\nimport com.twitter.frigate.common.candidate.UserDetails\nimport com.twitter.frigate.common.predicate.TargetUserPredicates\nimport com.twitter.frigate.common.predicate.{FatiguePredicate => CommonFatiguePredicate}\nimport com.twitter.frigate.common.store.deviceinfo.MobileClientType\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.target.TargetScoringDetails\nimport com.twitter.frigate.pushservice.util.PushCapUtil\nimport com.twitter.frigate.thriftscala.NotificationDisplayLocation\nimport com.twitter.frigate.thriftscala.{CommonRecommendationType => CRT}\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.timelines.configapi.FSBoundedParam\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\n\nobject TargetPredicates {\n\n  def paramPredicate[T <: Target](\n    param: Param[Boolean]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n    val name = param.getClass.getSimpleName.stripSuffix(\"$\")\n    Predicate\n      .from { target: T => target.params(param) }\n      .withStats(statsReceiver.scope(s\"param_${name}_controlled_predicate\"))\n      .withName(s\"param_${name}_controlled_predicate\")\n  }\n\n  /**\n   * Use the predicate except fn is true., Same as the candidate version but for Target\n   */\n  def exceptedPredicate[T <: TargetUser](\n    name: String,\n    fn: T => Future[Boolean],\n    predicate: Predicate[T]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n    Predicate\n      .fromAsync { e: T => fn(e) }\n      .or(predicate)\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  /**\n   * Refresh For push handler target user predicate to fatigue on visiting Home timeline\n   */\n  def targetHTLVisitPredicate[\n    T <: TargetUser with UserDetails with TargetABDecider with HTLVisitHistory\n  ](\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n    val name = \"target_htl_visit_predicate\"\n    Predicate\n      .fromAsync { target: T =>\n        val hoursToFatigue = target.params(PushFeatureSwitchParams.HTLVisitFatigueTime)\n        TargetUserPredicates\n          .homeTimelineFatigue(hoursToFatigue.hours)\n          .apply(Seq(target))\n          .map(_.head)\n      }\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  def targetPushBitEnabledPredicate[T <: Target](\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n    val name = \"push_bit_enabled\"\n    val scopedStats = statsReceiver.scope(s\"targetpredicate_$name\")\n\n    Predicate\n      .fromAsync { target: T =>\n        target.deviceInfo\n          .map { info =>\n            info.exists { deviceInfo =>\n              deviceInfo.isRecommendationsEligible ||\n              deviceInfo.isNewsEligible ||\n              deviceInfo.isTopicsEligible ||\n              deviceInfo.isSpacesEligible\n            }\n          }\n      }.withStats(scopedStats)\n      .withName(name)\n  }\n\n  def targetFatiguePredicate[T <: Target](\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n    val name = \"target_fatigue_predicate\"\n    val predicateStatScope = statsReceiver.scope(name)\n    Predicate\n      .fromAsync { target: T =>\n        PushCapUtil\n          .getPushCapFatigue(target, predicateStatScope)\n          .flatMap { pushCapInfo =>\n            CommonFatiguePredicate\n              .magicRecsPushTargetFatiguePredicate(\n                interval = pushCapInfo.fatigueInterval,\n                maxInInterval = pushCapInfo.pushcap\n              )\n              .apply(Seq(target))\n              .map(_.headOption.getOrElse(false))\n          }\n      }\n      .withStats(predicateStatScope)\n      .withName(name)\n  }\n\n  def teamExceptedPredicate[T <: TargetUser](\n    predicate: NamedPredicate[T]\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[T] = {\n    Predicate\n      .fromAsync { t: T => t.isTeamMember }\n      .or(predicate)\n      .withStats(stats.scope(predicate.name))\n      .withName(predicate.name)\n  }\n\n  def targetValidMobileSDKPredicate[T <: Target](\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n    val name = \"valid_mobile_sdk\"\n    val scopedStats = statsReceiver.scope(s\"targetpredicate_$name\")\n\n    Predicate\n      .fromAsync { target: T =>\n        TargetUserPredicates.validMobileSDKPredicate\n          .apply(Seq(target)).map(_.headOption.getOrElse(false))\n      }.withStats(scopedStats)\n      .withName(name)\n  }\n\n  def magicRecsMinDurationSinceSent[T <: Target](\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n    val name = \"target_min_duration_since_push\"\n    Predicate\n      .fromAsync { target: T =>\n        PushCapUtil.getMinDurationSincePush(target, statsReceiver).flatMap { minDurationSincePush =>\n          CommonFatiguePredicate\n            .magicRecsMinDurationSincePush(interval = minDurationSincePush)\n            .apply(Seq(target)).map(_.head)\n        }\n      }\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  def optoutProbPredicate[\n    T <: TargetUser with TargetABDecider with TargetScoringDetails with FrigateHistory\n  ](\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[T] = {\n    val name = \"target_has_high_optout_probability\"\n    Predicate\n      .fromAsync { target: T =>\n        val isNewUser = target.is30DayNewUserFromSnowflakeIdTime\n        if (isNewUser) {\n          statsReceiver.scope(name).counter(\"all_new_users\").incr()\n        }\n        target.bucketOptoutProbability\n          .flatMap {\n            case Some(optoutProb) =>\n              if (optoutProb >= target.params(PushFeatureSwitchParams.BucketOptoutThresholdParam)) {\n                CommonFatiguePredicate\n                  .magicRecsPushTargetFatiguePredicate(\n                    interval = 24.hours,\n                    maxInInterval = target.params(PushFeatureSwitchParams.OptoutExptPushCapParam)\n                  )\n                  .apply(Seq(target))\n                  .map { values =>\n                    val isValid = values.headOption.getOrElse(false)\n                    if (!isValid && isNewUser) {\n                      statsReceiver.scope(name).counter(\"filtered_new_users\").incr()\n                    }\n                    isValid\n                  }\n              } else Future.True\n            case _ => Future.True\n          }\n      }\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  /**\n   * Predicate used to specify CRT fatigue given interval and max number of candidates within interval.\n   * @param crt                   The specific CRT that this predicate is being applied to\n   * @param intervalParam         The fatigue interval\n   * @param maxInIntervalParam    The max number of the given CRT's candidates that are acceptable\n   *                              in the interval\n   * @param stats                 StatsReceiver\n   * @return                      Target Predicate\n   */\n  def pushRecTypeFatiguePredicate(\n    crt: CRT,\n    intervalParam: Param[Duration],\n    maxInIntervalParam: FSBoundedParam[Int],\n    stats: StatsReceiver\n  ): Predicate[Target] =\n    Predicate.fromAsync { target: Target =>\n      val interval = target.params(intervalParam)\n      val maxIninterval = target.params(maxInIntervalParam)\n      CommonFatiguePredicate\n        .recTypeTargetFatiguePredicate(\n          interval = interval,\n          maxInInterval = maxIninterval,\n          recommendationType = crt,\n          notificationDisplayLocation = NotificationDisplayLocation.PushToMobileDevice,\n          minInterval = 30.minutes\n        )(stats.scope(s\"${crt}_push_candidate_fatigue\")).apply(Seq(target)).map(_.head)\n    }\n\n  def inlineActionFatiguePredicate(\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[Target] = {\n    val name = \"inline_action_fatigue\"\n    val predicateRequests = statsReceiver.scope(name).counter(\"requests\")\n    val targetIsInExpt = statsReceiver.scope(name).counter(\"target_in_expt\")\n    val predicateEnabled = statsReceiver.scope(name).counter(\"enabled\")\n    val predicateDisabled = statsReceiver.scope(name).counter(\"disabled\")\n    val inlineFatigueDisabled = statsReceiver.scope(name).counter(\"inline_fatigue_disabled\")\n\n    Predicate\n      .fromAsync { target: Target =>\n        predicateRequests.incr()\n        if (target.params(PushFeatureSwitchParams.TargetInInlineActionAppVisitFatigue)) {\n          targetIsInExpt.incr()\n          target.inlineActionHistory.map { inlineHistory =>\n            if (inlineHistory.nonEmpty && target.params(\n                PushFeatureSwitchParams.EnableInlineActionAppVisitFatigue)) {\n              predicateEnabled.incr()\n              val inlineFatigue = target.params(PushFeatureSwitchParams.InlineActionAppVisitFatigue)\n              val lookbackInMs = inlineFatigue.ago.inMilliseconds\n              val filteredHistory = inlineHistory.filter {\n                case (time, _) => time > lookbackInMs\n              }\n              filteredHistory.isEmpty\n            } else {\n              inlineFatigueDisabled.incr()\n              true\n            }\n          }\n        } else {\n          predicateDisabled.incr()\n          Future.True\n        }\n      }\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  def webNotifsHoldback[T <: TargetUser with UserDetails with TargetABDecider](\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[T] = {\n    val name = \"mr_web_notifs_holdback\"\n    Predicate\n      .fromAsync { targetUserContext: T =>\n        targetUserContext.deviceInfo.map { deviceInfoOpt =>\n          val isPrimaryWeb = deviceInfoOpt.exists {\n            _.guessedPrimaryClient.exists { clientType =>\n              clientType == MobileClientType.Web\n            }\n          }\n          !(isPrimaryWeb && targetUserContext.params(PushFeatureSwitchParams.MRWebHoldbackParam))\n        }\n      }\n      .withStats(stats.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/TopTweetImpressionsPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.model.TopTweetImpressionsPushCandidate\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FS}\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\n\nobject TopTweetImpressionsPredicates {\n\n  def topTweetImpressionsFatiguePredicate(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[TopTweetImpressionsPushCandidate] = {\n    val name = \"top_tweet_impressions_fatigue\"\n    val scopedStats = stats.scope(name)\n    val bucketImpressionCounter = scopedStats.counter(\"bucket_impression_count\")\n    Predicate\n      .fromAsync { candidate: TopTweetImpressionsPushCandidate =>\n        val interval = candidate.target.params(FS.TopTweetImpressionsNotificationInterval)\n        val maxInInterval = candidate.target.params(FS.MaxTopTweetImpressionsNotifications)\n        val minInterval = candidate.target.params(FS.TopTweetImpressionsFatigueMinIntervalDuration)\n        bucketImpressionCounter.incr()\n\n        val fatiguePredicate = FatiguePredicate.recTypeOnly(\n          interval = interval,\n          maxInInterval = maxInInterval,\n          minInterval = minInterval,\n          recommendationType = CommonRecommendationType.TweetImpressions\n        )\n        fatiguePredicate.apply(Seq(candidate)).map(_.head)\n      }\n      .withStats(stats.scope(s\"predicate_${name}\"))\n      .withName(name)\n  }\n\n  def topTweetImpressionsThreshold(\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[TopTweetImpressionsPushCandidate] = {\n    val name = \"top_tweet_impressions_threshold\"\n    val scopedStats = statsReceiver.scope(name)\n    val meetsImpressionsCounter = scopedStats.counter(\"meets_impressions_count\")\n    val bucketImpressionCounter = scopedStats.counter(\"bucket_impression_count\")\n    Predicate\n      .from[TopTweetImpressionsPushCandidate] { candidate =>\n        val meetsImpressionsThreshold =\n          candidate.impressionsCount >= candidate.target.params(FS.TopTweetImpressionsThreshold)\n        if (meetsImpressionsThreshold) meetsImpressionsCounter.incr()\n        bucketImpressionCounter.incr()\n        meetsImpressionsThreshold\n      }\n      .withStats(statsReceiver.scope(s\"predicate_${name}\"))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/TweetEngagementRatioPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.util.CandidateUtil\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\nobject TweetEngagementRatioPredicate {\n\n  def QTtoNtabClickBasedPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[\n    PushCandidate with TweetCandidate with RecommendationType\n  ] = {\n    val name = \"oon_tweet_engagement_filter_qt_to_ntabclick_ratio_based_predicate\"\n    val scopedStatsReceiver = stats.scope(name)\n    val allOonCandidatesCounter = scopedStatsReceiver.counter(\"all_oon_candidates\")\n    val filteredCandidatesCounter =\n      scopedStatsReceiver.counter(\"filtered_oon_candidates\")\n\n    val quoteCountFeature =\n      \"tweet.core.tweet_counts.quote_count\"\n    val ntabClickCountFeature =\n      \"tweet.magic_recs_tweet_real_time_aggregates_v2.pair.v2.magicrecs.realtime.is_ntab_clicked.any_feature.Duration.Top.count\"\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetCandidate with RecommendationType =>\n        val target = candidate.target\n        val crt = candidate.commonRecType\n        val isOonCandidate = RecTypes.isOutOfNetworkTweetRecType(crt) ||\n          RecTypes.outOfNetworkTopicTweetTypes.contains(crt)\n\n        lazy val QTtoNtabClickRatioThreshold =\n          target.params(PushFeatureSwitchParams.TweetQTtoNtabClickRatioThresholdParam)\n        lazy val quoteCount = candidate.numericFeatures.getOrElse(quoteCountFeature, 0.0)\n        lazy val ntabClickCount = candidate.numericFeatures.getOrElse(ntabClickCountFeature, 0.0)\n        lazy val quoteRate = if (ntabClickCount > 0) quoteCount / ntabClickCount else 1.0\n\n        if (isOonCandidate) allOonCandidatesCounter.incr()\n        if (CandidateUtil.shouldApplyHealthQualityFilters(candidate) && isOonCandidate) {\n          val ntabClickThreshold = 1000\n          candidate.cachePredicateInfo(\n            name + \"_count\",\n            ntabClickCount,\n            ntabClickThreshold,\n            ntabClickCount >= ntabClickThreshold)\n          candidate.cachePredicateInfo(\n            name + \"_ratio\",\n            quoteRate,\n            QTtoNtabClickRatioThreshold,\n            quoteRate < QTtoNtabClickRatioThreshold)\n          if (ntabClickCount >= ntabClickThreshold && quoteRate < QTtoNtabClickRatioThreshold) {\n            filteredCandidatesCounter.incr()\n            Future.False\n          } else Future.True\n        } else Future.True\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  def TweetReplyLikeRatioPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetCandidate] = {\n    val name = \"tweet_reply_like_ratio\"\n    val scopedStatsReceiver = stats.scope(name)\n    val allCandidatesCounter = scopedStatsReceiver.counter(\"all_candidates\")\n    val filteredCandidatesCounter = scopedStatsReceiver.counter(\"filtered_candidates\")\n    val bucketedCandidatesCounter = scopedStatsReceiver.counter(\"bucketed_candidates\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        allCandidatesCounter.incr()\n        val target = candidate.target\n        val likeCount = candidate.numericFeatures\n          .getOrElse(PushConstants.TweetLikesFeatureName, 0.0)\n        val replyCount = candidate.numericFeatures\n          .getOrElse(PushConstants.TweetRepliesFeatureName, 0.0)\n        val ratio = replyCount / likeCount.max(1)\n        val isOonCandidate = RecTypes.isOutOfNetworkTweetRecType(candidate.commonRecType) ||\n          RecTypes.outOfNetworkTopicTweetTypes.contains(candidate.commonRecType)\n\n        if (isOonCandidate\n          && CandidateUtil.shouldApplyHealthQualityFilters(candidate)\n          && replyCount > target.params(\n            PushFeatureSwitchParams.TweetReplytoLikeRatioReplyCountThreshold)) {\n          bucketedCandidatesCounter.incr()\n          if (ratio > target.params(\n              PushFeatureSwitchParams.TweetReplytoLikeRatioThresholdLowerBound)\n            && ratio < target.params(\n              PushFeatureSwitchParams.TweetReplytoLikeRatioThresholdUpperBound)) {\n            filteredCandidatesCounter.incr()\n            Future.False\n          } else {\n            Future.True\n          }\n        } else {\n          Future.True\n        }\n      }\n      .withStats(stats.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/TweetLanguagePredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.util.CandidateUtil\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.language.normalization.UserDisplayLanguage\nimport com.twitter.util.Future\n\nobject TweetLanguagePredicate {\n\n  def oonTweeetLanguageMatch(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[\n    PushCandidate with RecommendationType with TweetDetails\n  ] = {\n    val name = \"oon_tweet_language_predicate\"\n    val scopedStatsReceiver = stats.scope(name)\n    val oonCandidatesCounter =\n      scopedStatsReceiver.counter(\"oon_candidates\")\n    val enableFilterCounter =\n      scopedStatsReceiver.counter(\"enabled_filter\")\n    val skipMediaTweetsCounter =\n      scopedStatsReceiver.counter(\"skip_media_tweets\")\n\n    Predicate\n      .fromAsync { candidate: PushCandidate with RecommendationType with TweetDetails =>\n        val target = candidate.target\n        val crt = candidate.commonRecType\n        val isOonCandidate = RecTypes.isOutOfNetworkTweetRecType(crt) ||\n          RecTypes.outOfNetworkTopicTweetTypes.contains(crt)\n\n        if (CandidateUtil.shouldApplyHealthQualityFilters(candidate) && isOonCandidate) {\n          oonCandidatesCounter.incr()\n\n          target.featureMap.map { featureMap =>\n            val userPreferredLanguages = featureMap.sparseBinaryFeatures\n              .getOrElse(\"user.language.user.preferred_contents\", Set.empty[String])\n            val userEngagementLanguages = featureMap.sparseContinuousFeatures.getOrElse(\n              \"user.language.user.engagements\",\n              Map.empty[String, Double])\n            val userFollowLanguages = featureMap.sparseContinuousFeatures.getOrElse(\n              \"user.language.user.following_accounts\",\n              Map.empty[String, Double])\n            val userProducedTweetLanguages = featureMap.sparseContinuousFeatures\n              .getOrElse(\"user.language.user.produced_tweets\", Map.empty)\n            val userDeviceLanguages = featureMap.sparseContinuousFeatures.getOrElse(\n              \"user.language.user.recent_devices\",\n              Map.empty[String, Double])\n            val tweetLanguageOpt = candidate.categoricalFeatures\n              .get(target.params(PushFeatureSwitchParams.TweetLanguageFeatureNameParam))\n\n            if (userPreferredLanguages.isEmpty)\n              scopedStatsReceiver.counter(\"userPreferredLanguages_empty\").incr()\n            if (userEngagementLanguages.isEmpty)\n              scopedStatsReceiver.counter(\"userEngagementLanguages_empty\").incr()\n            if (userFollowLanguages.isEmpty)\n              scopedStatsReceiver.counter(\"userFollowLanguages_empty\").incr()\n            if (userProducedTweetLanguages.isEmpty)\n              scopedStatsReceiver\n                .counter(\"userProducedTweetLanguages_empty\")\n                .incr()\n            if (userDeviceLanguages.isEmpty)\n              scopedStatsReceiver.counter(\"userDeviceLanguages_empty\").incr()\n            if (tweetLanguageOpt.isEmpty) scopedStatsReceiver.counter(\"tweetLanguage_empty\").incr()\n\n            val tweetLanguage = tweetLanguageOpt.getOrElse(\"und\")\n            val undefinedTweetLanguages = Set(\"\")\n\n            if (!undefinedTweetLanguages.contains(tweetLanguage)) {\n              lazy val userInferredLanguageThreshold =\n                target.params(PushFeatureSwitchParams.UserInferredLanguageThresholdParam)\n              lazy val userDeviceLanguageThreshold =\n                target.params(PushFeatureSwitchParams.UserDeviceLanguageThresholdParam)\n              lazy val enableTweetLanguageFilter =\n                target.params(PushFeatureSwitchParams.EnableTweetLanguageFilter)\n              lazy val skipLanguageFilterForMediaTweets =\n                target.params(PushFeatureSwitchParams.SkipLanguageFilterForMediaTweets)\n\n              lazy val allLanguages = userPreferredLanguages ++\n                userEngagementLanguages.filter(_._2 > userInferredLanguageThreshold).keySet ++\n                userFollowLanguages.filter(_._2 > userInferredLanguageThreshold).keySet ++\n                userProducedTweetLanguages.filter(_._2 > userInferredLanguageThreshold).keySet ++\n                userDeviceLanguages.filter(_._2 > userDeviceLanguageThreshold).keySet\n\n              if (enableTweetLanguageFilter && allLanguages.nonEmpty) {\n                enableFilterCounter.incr()\n                val hasMedia = candidate.hasPhoto || candidate.hasVideo\n\n                if (hasMedia && skipLanguageFilterForMediaTweets) {\n                  skipMediaTweetsCounter.incr()\n                  true\n                } else {\n                  allLanguages.map(UserDisplayLanguage.toTweetLanguage).contains(tweetLanguage)\n                }\n              } else true\n            } else true\n          }\n        } else Future.True\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/TweetWithheldContentPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TweetDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.hermit.predicate.tweetypie.UserLocationAndTweet\nimport com.twitter.hermit.predicate.tweetypie.WithheldTweetPredicate\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.service.metastore.gen.thriftscala.Location\nimport com.twitter.util.Future\n\nobject TweetWithheldContentPredicate {\n  val name = \"withheld_content\"\n  val defaultLocation = Location(city = \"\", region = \"\", countryCode = \"\", confidence = 0.0)\n\n  def apply(\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with TweetDetails] = {\n    Predicate\n      .fromAsync { candidate: PushCandidate with TweetDetails =>\n        candidate.tweet match {\n          case Some(tweet) =>\n            WithheldTweetPredicate(checkAllCountries = true)\n              .apply(Seq(UserLocationAndTweet(defaultLocation, tweet)))\n              .map(_.head)\n          case None =>\n            Future.value(false)\n        }\n      }\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/event/EventPredicatesForCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.event\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.EventCandidate\nimport com.twitter.frigate.common.base.TargetInfo\nimport com.twitter.frigate.common.base.TargetUser\nimport com.twitter.frigate.common.candidate.FrigateHistory\nimport com.twitter.frigate.common.history.RecItems\nimport com.twitter.frigate.magic_events.thriftscala.Locale\nimport com.twitter.frigate.pushservice.model.MagicFanoutEventHydratedCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutEventPushCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutNewsEventPushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.predicate.magic_fanout.MagicFanoutPredicatesUtil._\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\nobject EventPredicatesForCandidate {\n  def hasTitle(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[MagicFanoutEventHydratedCandidate] = {\n    val name = \"event_title_available\"\n    val scopedStatsReceiver = statsReceiver.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync { candidate: MagicFanoutEventHydratedCandidate =>\n        candidate.eventTitleFut.map(_.nonEmpty)\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  def isNotDuplicateWithEventId(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[MagicFanoutEventHydratedCandidate] = {\n    val name = \"duplicate_event_id\"\n    Predicate\n      .fromAsync { candidate: MagicFanoutEventHydratedCandidate =>\n        val useRelaxedFatigueLengthFut: Future[Boolean] =\n          candidate match {\n            case mfNewsEvent: MagicFanoutNewsEventPushCandidate =>\n              mfNewsEvent.isHighPriorityEvent\n            case _ => Future.value(false)\n          }\n        Future.join(candidate.target.history, useRelaxedFatigueLengthFut).map {\n          case (history, useRelaxedFatigueLength) =>\n            val filteredNotifications = if (useRelaxedFatigueLength) {\n              val relaxedFatigueInterval =\n                candidate.target\n                  .params(\n                    PushFeatureSwitchParams.MagicFanoutRelaxedEventIdFatigueIntervalInHours).hours\n              history.notificationMap.filterKeys { time =>\n                time.untilNow <= relaxedFatigueInterval\n              }.values\n            } else history.notificationMap.values\n            !RecItems(filteredNotifications.toSeq).events.exists(_.eventId == candidate.eventId)\n        }\n      }\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n\n  def isNotDuplicateWithEventIdForCandidate[\n    T <: TargetUser with FrigateHistory,\n    Cand <: EventCandidate with TargetInfo[T]\n  ](\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[Cand] = {\n    val name = \"is_not_duplicate_event\"\n    Predicate\n      .fromAsync { candidate: Cand =>\n        candidate.target.pushRecItems.map {\n          !_.events.map(_.eventId).contains(candidate.eventId)\n        }\n      }\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  def accountCountryPredicateWithAllowlist(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[MagicFanoutEventPushCandidate] = {\n    val name = \"account_country_predicate_with_allowlist\"\n    val scopedStats = stats.scope(name)\n\n    val skipPredicate = Predicate\n      .from { candidate: MagicFanoutEventPushCandidate =>\n        candidate.target.params(PushFeatureSwitchParams.MagicFanoutSkipAccountCountryPredicate)\n      }\n      .withStats(stats.scope(\"skip_account_country_predicate_mf\"))\n      .withName(\"skip_account_country_predicate_mf\")\n\n    val excludeEventFromAccountCountryPredicateFiltering = Predicate\n      .from { candidate: MagicFanoutEventPushCandidate =>\n        val eventId = candidate.eventId\n        val target = candidate.target\n        target\n          .params(PushFeatureSwitchParams.MagicFanoutEventAllowlistToSkipAccountCountryPredicate)\n          .exists(eventId.equals)\n      }\n      .withStats(stats.scope(\"exclude_event_from_account_country_predicate_filtering\"))\n      .withName(\"exclude_event_from_account_country_predicate_filtering\")\n\n    skipPredicate\n      .or(excludeEventFromAccountCountryPredicateFiltering)\n      .or(accountCountryPredicate)\n      .withStats(scopedStats)\n      .withName(name)\n  }\n\n  /**\n   * Check if user's country is targeted\n   * @param stats\n   */\n  def accountCountryPredicate(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[MagicFanoutEventPushCandidate] = {\n    val name = \"account_country_predicate\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    val internationalLocalePassedCounter =\n      scopedStatsReceiver.counter(\"international_locale_passed\")\n    val internationalLocaleFilteredCounter =\n      scopedStatsReceiver.counter(\"international_locale_filtered\")\n    Predicate\n      .fromAsync { candidate: MagicFanoutEventPushCandidate =>\n        candidate.target.countryCode.map {\n          case Some(countryCode) =>\n            val denyListedCountryCodes: Seq[String] =\n              if (candidate.commonRecType == CommonRecommendationType.MagicFanoutNewsEvent) {\n                candidate.target\n                  .params(PushFeatureSwitchParams.MagicFanoutDenyListedCountries)\n              } else if (candidate.commonRecType == CommonRecommendationType.MagicFanoutSportsEvent) {\n                candidate.target\n                  .params(PushFeatureSwitchParams.MagicFanoutSportsEventDenyListedCountries)\n              } else Seq()\n            val eventCountries =\n              candidate.newsForYouMetadata\n                .flatMap(_.locales).getOrElse(Seq.empty[Locale]).flatMap(_.country)\n            if (isInCountryList(countryCode, eventCountries)\n              && !isInCountryList(countryCode, denyListedCountryCodes)) {\n              internationalLocalePassedCounter.incr()\n              true\n            } else {\n              internationalLocaleFilteredCounter.incr()\n              false\n            }\n          case _ => false\n        }\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/magic_fanout/MagicFanoutPredicatesForCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.magic_fanout\n\nimport com.twitter.audience_rewards.thriftscala.HasSuperFollowingRelationshipRequest\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.MagicFanoutCandidate\nimport com.twitter.frigate.common.base.MagicFanoutCreatorEventCandidate\nimport com.twitter.frigate.common.base.MagicFanoutProductLaunchCandidate\nimport com.twitter.frigate.common.history.RecItems\nimport com.twitter.frigate.common.predicate.FatiguePredicate.build\nimport com.twitter.frigate.common.predicate.FatiguePredicate.productLaunchTypeRecTypesOnlyFilter\nimport com.twitter.frigate.common.predicate.FatiguePredicate.recOnlyFilter\nimport com.twitter.frigate.common.store.interests.InterestsLookupRequestWithContext\nimport com.twitter.frigate.common.store.interests.SemanticCoreEntityId\nimport com.twitter.frigate.common.util.IbisAppPushDeviceSettingsUtil\nimport com.twitter.frigate.magic_events.thriftscala.CreatorFanoutType\nimport com.twitter.frigate.magic_events.thriftscala.ProductType\nimport com.twitter.frigate.magic_events.thriftscala.TargetID\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutEventHydratedCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutEventPushCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutNewsEventPushCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.predicate.FatiguePredicate\nimport com.twitter.frigate.pushservice.predicate.PredicatesForCandidate\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.NotificationDisplayLocation\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.interests.thriftscala.UserInterests\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\n\nobject MagicFanoutPredicatesForCandidate {\n\n  /**\n   * Check if Semantic Core reasons satisfy rank threshold ( for heavy users a non broad entity should satisfy the threshold)\n   */\n  def magicFanoutErgInterestRankThresholdPredicate(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[MagicFanoutEventHydratedCandidate] = {\n    val name = \"magicfanout_interest_erg_rank_threshold\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync { candidate: MagicFanoutEventHydratedCandidate =>\n        candidate.target.isHeavyUserState.map { isHeavyUser =>\n          lazy val rankThreshold =\n            if (isHeavyUser) {\n              candidate.target.params(PushFeatureSwitchParams.MagicFanoutRankErgThresholdHeavy)\n            } else {\n              candidate.target.params(PushFeatureSwitchParams.MagicFanoutRankErgThresholdNonHeavy)\n            }\n          MagicFanoutPredicatesUtil\n            .checkIfValidErgScEntityReasonExists(\n              candidate.effectiveMagicEventsReasons,\n              rankThreshold\n            )\n        }\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  def newsNotificationFatigue(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val name = \"news_notification_fatigue\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        FatiguePredicate\n          .recTypeSetOnly(\n            notificationDisplayLocation = NotificationDisplayLocation.PushToMobileDevice,\n            recTypes = Set(CommonRecommendationType.MagicFanoutNewsEvent),\n            maxInInterval =\n              candidate.target.params(PushFeatureSwitchParams.MFMaxNumberOfPushesInInterval),\n            interval = candidate.target.params(PushFeatureSwitchParams.MFPushIntervalInHours),\n            minInterval = candidate.target.params(PushFeatureSwitchParams.MFMinIntervalFatigue)\n          )\n          .apply(Seq(candidate))\n          .map(_.headOption.getOrElse(false))\n\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  /**\n   * Check if reason contains any optouted semantic core entity interests.\n   *\n   * @param stats\n   *\n   * @return\n   */\n  def magicFanoutNoOptoutInterestPredicate(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[MagicFanoutEventPushCandidate] = {\n    val name = \"magicfanout_optout_interest_predicate\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    val withOptOutInterestsCounter = stats.counter(\"with_optout_interests\")\n    val withoutOptOutInterestsCounter = stats.counter(\"without_optout_interests\")\n    Predicate\n      .fromAsync { candidate: MagicFanoutEventPushCandidate =>\n        candidate.target.optOutSemanticCoreInterests.map {\n          case (\n                optOutUserInterests: Seq[SemanticCoreEntityId]\n              ) =>\n            withOptOutInterestsCounter.incr()\n            optOutUserInterests\n              .intersect(candidate.annotatedAndInferredSemanticCoreEntities).isEmpty\n          case _ =>\n            withoutOptOutInterestsCounter.incr()\n            true\n        }\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  /**\n   * Checks if the target has only one device language language,\n   * and that language is targeted for that event\n   *\n   * @param statsReceiver\n   *\n   * @return\n   */\n  def inferredUserDeviceLanguagePredicate(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[MagicFanoutEventPushCandidate] = {\n    val name = \"inferred_device_language\"\n    val scopedStats = statsReceiver.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync { candidate: MagicFanoutEventPushCandidate =>\n        val target = candidate.target\n        target.deviceInfo.map {\n          _.flatMap { deviceInfo =>\n            val languages = deviceInfo.deviceLanguages.getOrElse(Seq.empty[String])\n            val distinctDeviceLanguages =\n              IbisAppPushDeviceSettingsUtil.distinctDeviceLanguages(languages)\n\n            candidate.newsForYouMetadata.map { newsForYouMetadata =>\n              val eventLocales = newsForYouMetadata.locales.getOrElse(Seq.empty)\n              val eventLanguages = eventLocales.flatMap(_.language).map(_.toLowerCase).distinct\n\n              eventLanguages.intersect(distinctDeviceLanguages).nonEmpty\n            }\n          }.getOrElse(false)\n        }\n      }\n      .withStats(scopedStats)\n      .withName(name)\n  }\n\n  /**\n   * Bypass predicate if high priority push\n   */\n  def highPriorityNewsEventExceptedPredicate(\n    predicate: NamedPredicate[MagicFanoutNewsEventPushCandidate]\n  )(\n    implicit config: Config\n  ): NamedPredicate[MagicFanoutNewsEventPushCandidate] = {\n    PredicatesForCandidate.exceptedPredicate(\n      name = \"high_priority_excepted_\" + predicate.name,\n      fn = MagicFanoutPredicatesUtil.checkIfHighPriorityNewsEventForCandidate,\n      predicate\n    )(config.statsReceiver)\n  }\n\n  /**\n   * Bypass predicate if high priority push\n   */\n  def highPriorityEventExceptedPredicate(\n    predicate: NamedPredicate[MagicFanoutEventPushCandidate]\n  )(\n    implicit config: Config\n  ): NamedPredicate[MagicFanoutEventPushCandidate] = {\n    PredicatesForCandidate.exceptedPredicate(\n      name = \"high_priority_excepted_\" + predicate.name,\n      fn = MagicFanoutPredicatesUtil.checkIfHighPriorityEventForCandidate,\n      predicate\n    )(config.statsReceiver)\n  }\n\n  def magicFanoutSimClusterTargetingPredicate(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[MagicFanoutEventPushCandidate] = {\n    val name = \"simcluster_targeting\"\n    val scopedStats = stats.scope(s\"predicate_$name\")\n    val userStateCounters = scopedStats.scope(\"user_state\")\n    Predicate\n      .fromAsync { candidate: MagicFanoutEventPushCandidate =>\n        candidate.target.isHeavyUserState.map { isHeavyUser =>\n          val simClusterEmbeddings = candidate.newsForYouMetadata.flatMap(\n            _.eventContextScribe.flatMap(_.simClustersEmbeddings))\n          val TopKSimClustersCount = 50\n          val eventSimClusterVectorOpt: Option[MagicFanoutPredicatesUtil.SimClusterScores] =\n            MagicFanoutPredicatesUtil.getEventSimClusterVector(\n              simClusterEmbeddings.map(_.toMap),\n              (ModelVersion.Model20m145kUpdated, EmbeddingType.FollowBasedTweet),\n              TopKSimClustersCount\n            )\n          val userSimClusterVectorOpt: Option[MagicFanoutPredicatesUtil.SimClusterScores] =\n            MagicFanoutPredicatesUtil.getUserSimClusterVector(candidate.effectiveMagicEventsReasons)\n          (eventSimClusterVectorOpt, userSimClusterVectorOpt) match {\n            case (\n                  Some(eventSimClusterVector: MagicFanoutPredicatesUtil.SimClusterScores),\n                  Some(userSimClusterVector)) =>\n              val score = eventSimClusterVector\n                .normedDotProduct(userSimClusterVector, eventSimClusterVector)\n              val threshold = if (isHeavyUser) {\n                candidate.target.params(\n                  PushFeatureSwitchParams.MagicFanoutSimClusterDotProductHeavyUserThreshold)\n              } else {\n                candidate.target.params(\n                  PushFeatureSwitchParams.MagicFanoutSimClusterDotProductNonHeavyUserThreshold)\n              }\n              val isPassed = score >= threshold\n              userStateCounters.scope(isHeavyUser.toString).counter(s\"$isPassed\").incr()\n              isPassed\n\n            case (None, Some(userSimClusterVector)) =>\n              candidate.commonRecType == CommonRecommendationType.MagicFanoutSportsEvent\n\n            case _ => false\n          }\n        }\n      }\n      .withStats(scopedStats)\n      .withName(name)\n  }\n\n  def geoTargetingHoldback(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with MagicFanoutCandidate] = {\n    Predicate\n      .from[PushCandidate with MagicFanoutCandidate] { candidate =>\n        if (MagicFanoutPredicatesUtil.reasonsContainGeoTarget(\n            candidate.candidateMagicEventsReasons)) {\n          candidate.target.params(PushFeatureSwitchParams.EnableMfGeoTargeting)\n        } else true\n      }\n      .withStats(stats.scope(\"geo_targeting_holdback\"))\n      .withName(\"geo_targeting_holdback\")\n  }\n\n  def geoOptOutPredicate(\n    userStore: ReadableStore[Long, User]\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with MagicFanoutCandidate] = {\n    Predicate\n      .fromAsync[PushCandidate with MagicFanoutCandidate] { candidate =>\n        if (MagicFanoutPredicatesUtil.reasonsContainGeoTarget(\n            candidate.candidateMagicEventsReasons)) {\n          userStore.get(candidate.target.targetId).map { userOpt =>\n            val isGeoAllowed = userOpt\n              .flatMap(_.account)\n              .exists(_.allowLocationHistoryPersonalization)\n            isGeoAllowed\n          }\n        } else {\n          Future.True\n        }\n      }\n      .withStats(stats.scope(\"geo_opt_out_predicate\"))\n      .withName(\"geo_opt_out_predicate\")\n  }\n\n  /**\n   * Check if Semantic Core reasons contains valid utt reason & reason is within top k topics followed by user\n   */\n  def magicFanoutTopicFollowsTargetingPredicate(\n    implicit stats: StatsReceiver,\n    interestsLookupStore: ReadableStore[InterestsLookupRequestWithContext, UserInterests]\n  ): NamedPredicate[MagicFanoutEventHydratedCandidate] = {\n    val name = \"magicfanout_topic_follows_targeting\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync[PushCandidate with MagicFanoutEventHydratedCandidate] { candidate =>\n        candidate.followedTopicLocalizedEntities.map(_.nonEmpty)\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  /** Requires the magicfanout candidate to have a UserID reason which ranks below the follow\n   * rank threshold. If no UserID target exists the candidate is dropped. */\n  def followRankThreshold(\n    threshold: Param[Int]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with MagicFanoutCandidate] = {\n    val name = \"follow_rank_threshold\"\n    Predicate\n      .from[PushCandidate with MagicFanoutCandidate] { c =>\n        c.candidateMagicEventsReasons.exists { fanoutReason =>\n          fanoutReason.reason match {\n            case TargetID.UserID(_) =>\n              fanoutReason.rank.exists { rank =>\n                rank <= c.target.params(threshold)\n              }\n            case _ => false\n          }\n        }\n      }\n      .withStats(statsReceiver.scope(name))\n      .withName(name)\n  }\n\n  def userGeneratedEventsPredicate(\n    implicit statsReceiver: StatsReceiver\n  ): NamedPredicate[PushCandidate with MagicFanoutEventHydratedCandidate] = {\n    val name = \"user_generated_moments\"\n    val stats = statsReceiver.scope(name)\n\n    Predicate\n      .from { candidate: PushCandidate with MagicFanoutEventHydratedCandidate =>\n        val isUgmMoment = candidate.semanticCoreEntityTags.values.flatten.toSet\n          .contains(MagicFanoutPredicatesUtil.UgmMomentTag)\n        if (isUgmMoment) {\n          candidate.target.params(PushFeatureSwitchParams.MagicFanoutNewsUserGeneratedEventsEnable)\n        } else true\n      }.withStats(stats)\n      .withName(name)\n  }\n  def escherbirdMagicfanoutEventParam(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with MagicFanoutEventPushCandidate] = {\n    val name = \"magicfanout_escherbird_fs\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n\n    Predicate\n      .fromAsync[PushCandidate with MagicFanoutEventPushCandidate] { candidate =>\n        val candidateFrigateNotif = candidate.frigateNotification.magicFanoutEventNotification\n        val isEscherbirdEvent = candidateFrigateNotif.exists(_.isEscherbirdEvent.contains(true))\n        scopedStatsReceiver.counter(s\"with_escherbird_flag_$isEscherbirdEvent\").incr()\n\n        if (isEscherbirdEvent) {\n\n          val listOfEventsSemanticCoreDomainIds =\n            candidate.target.params(PushFeatureSwitchParams.ListOfEventSemanticCoreDomainIds)\n\n          val candScDomainEvent =\n            if (listOfEventsSemanticCoreDomainIds.nonEmpty) {\n              candidate.eventSemanticCoreDomainIds\n                .intersect(listOfEventsSemanticCoreDomainIds).nonEmpty\n            } else {\n              false\n            }\n          scopedStatsReceiver\n            .counter(\n              s\"with_escherbird_fs_in_list_of_event_semantic_core_domains_$candScDomainEvent\").incr()\n          Future.value(candScDomainEvent)\n        } else {\n          Future.True\n        }\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  /**\n   *  Checks if the user has custom targeting enabled.If so, bucket the user in experiment. This custom targeting refers to adding\n   *  tweet authors as targets in the eventfanout service.\n   * @param stats [StatsReceiver]\n   * @return NamedPredicate[PushCandidate with MagicFanoutEventPushCandidate]\n   */\n  def hasCustomTargetingForNewsEventsParam(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with MagicFanoutEventPushCandidate] = {\n    val name = \"magicfanout_hascustomtargeting\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n\n    Predicate\n      .from[PushCandidate with MagicFanoutEventPushCandidate] { candidate =>\n        candidate.candidateMagicEventsReasons.exists { fanoutReason =>\n          fanoutReason.reason match {\n            case userIdReason: TargetID.UserID =>\n              if (userIdReason.userID.hasCustomTargeting.contains(true)) {\n                candidate.target.params(\n                  PushFeatureSwitchParams.MagicFanoutEnableCustomTargetingNewsEvent)\n              } else true\n            case _ => true\n          }\n        }\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n\n  }\n\n  def magicFanoutProductLaunchFatigue(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with MagicFanoutProductLaunchCandidate] = {\n    val name = \"magic_fanout_product_launch_fatigue\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync { candidate: PushCandidate with MagicFanoutProductLaunchCandidate =>\n        val target = candidate.target\n        val (interval, maxInInterval, minInterval) = {\n          candidate.productLaunchType match {\n            case ProductType.BlueVerified =>\n              (\n                target.params(PushFeatureSwitchParams.ProductLaunchPushIntervalInHours),\n                target.params(PushFeatureSwitchParams.ProductLaunchMaxNumberOfPushesInInterval),\n                target.params(PushFeatureSwitchParams.ProductLaunchMinIntervalFatigue))\n            case _ =>\n              (Duration.fromDays(1), 0, Duration.Zero)\n          }\n        }\n        build(\n          interval = interval,\n          maxInInterval = maxInInterval,\n          minInterval = minInterval,\n          filterHistory = productLaunchTypeRecTypesOnlyFilter(\n            Set(CommonRecommendationType.MagicFanoutProductLaunch),\n            candidate.productLaunchType.toString),\n          notificationDisplayLocation = NotificationDisplayLocation.PushToMobileDevice\n        ).flatContraMap { candidate: PushCandidate => candidate.target.history }\n          .apply(Seq(candidate))\n          .map(_.headOption.getOrElse(false))\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  def creatorPushTargetIsNotCreator(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with MagicFanoutCreatorEventCandidate] = {\n    val name = \"magic_fanout_creator_is_self\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    Predicate\n      .from { candidate: PushCandidate with MagicFanoutCreatorEventCandidate =>\n        candidate.target.targetId != candidate.creatorId\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  def duplicateCreatorPredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with MagicFanoutCreatorEventCandidate] = {\n    val name = \"magic_fanout_creator_duplicate_creator_id\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync { cand: PushCandidate with MagicFanoutCreatorEventCandidate =>\n        cand.target.pushRecItems.map { recItems: RecItems =>\n          !recItems.creatorIds.contains(cand.creatorId)\n        }\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  def isSuperFollowingCreator(\n  )(\n    implicit config: Config,\n    stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with MagicFanoutCreatorEventCandidate] = {\n    val name = \"magic_fanout_is_already_superfollowing_creator\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync { cand: PushCandidate with MagicFanoutCreatorEventCandidate =>\n        config.hasSuperFollowingRelationshipStore\n          .get(\n            HasSuperFollowingRelationshipRequest(\n              sourceUserId = cand.target.targetId,\n              targetUserId = cand.creatorId)).map(_.getOrElse(false))\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n\n  def magicFanoutCreatorPushFatiguePredicate(\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[PushCandidate with MagicFanoutCreatorEventCandidate] = {\n    val name = \"magic_fanout_creator_fatigue\"\n    val scopedStatsReceiver = stats.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync { candidate: PushCandidate with MagicFanoutCreatorEventCandidate =>\n        val target = candidate.target\n        val (interval, maxInInterval, minInterval) = {\n          candidate.creatorFanoutType match {\n            case CreatorFanoutType.UserSubscription =>\n              (\n                target.params(PushFeatureSwitchParams.CreatorSubscriptionPushIntervalInHours),\n                target.params(\n                  PushFeatureSwitchParams.CreatorSubscriptionPushMaxNumberOfPushesInInterval),\n                target.params(PushFeatureSwitchParams.CreatorSubscriptionPushhMinIntervalFatigue))\n            case CreatorFanoutType.NewCreator =>\n              (\n                target.params(PushFeatureSwitchParams.NewCreatorPushIntervalInHours),\n                target.params(PushFeatureSwitchParams.NewCreatorPushMaxNumberOfPushesInInterval),\n                target.params(PushFeatureSwitchParams.NewCreatorPushMinIntervalFatigue))\n            case _ =>\n              (Duration.fromDays(1), 0, Duration.Zero)\n          }\n        }\n        build(\n          interval = interval,\n          maxInInterval = maxInInterval,\n          minInterval = minInterval,\n          filterHistory = recOnlyFilter(candidate.commonRecType),\n          notificationDisplayLocation = NotificationDisplayLocation.PushToMobileDevice\n        ).flatContraMap { candidate: PushCandidate => candidate.target.history }\n          .apply(Seq(candidate))\n          .map(_.headOption.getOrElse(false))\n      }\n      .withStats(scopedStatsReceiver)\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/magic_fanout/MagicFanoutPredicatesUtil.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.magic_fanout\n\nimport com.twitter.eventdetection.event_context.util.SimClustersUtil\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.magic_events.thriftscala._\nimport com.twitter.frigate.pushservice.model.MagicFanoutEventPushCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutNewsEventPushCandidate\nimport com.twitter.frigate.pushservice.model.MagicFanoutProductLaunchPushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.{SimClustersEmbedding => ThriftSimClustersEmbedding}\nimport com.twitter.util.Future\n\nobject MagicFanoutPredicatesUtil {\n\n  val UttDomain: Long = 0L\n  type DomainId = Long\n  type EntityId = Long\n  val BroadCategoryTag = \"utt:broad_category\"\n  val UgmMomentTag = \"MMTS.isUGMMoment\"\n  val TopKSimClustersCount = 50\n\n  case class SimClusterScores(simClusterScoreVector: Map[Int, Double]) {\n    def dotProduct(other: SimClusterScores): Double = {\n      simClusterScoreVector\n        .map {\n          case (clusterId, score) => other.simClusterScoreVector.getOrElse(clusterId, 0.0) * score\n        }.foldLeft(0.0) { _ + _ }\n    }\n\n    def norm(): Double = {\n      val sumOfSquares: Double = simClusterScoreVector\n        .map {\n          case (clusterId, score) => score * score\n        }.foldLeft(0.0)(_ + _)\n      scala.math.sqrt(sumOfSquares)\n    }\n\n    def normedDotProduct(other: SimClusterScores, normalizer: SimClusterScores): Double = {\n      val denominator = normalizer.norm()\n      val score = dotProduct(other)\n      if (denominator != 0.0) {\n        score / denominator\n      } else {\n        score\n      }\n    }\n  }\n\n  private def isSemanticCoreEntityBroad(\n    semanticCoreEntityTags: Map[(DomainId, EntityId), Set[String]],\n    scEntityId: SemanticCoreID\n  ): Boolean = {\n    semanticCoreEntityTags\n      .getOrElse((scEntityId.domainId, scEntityId.entityId), Set.empty).contains(BroadCategoryTag)\n  }\n\n  def isInCountryList(accountCountryCode: String, locales: Seq[String]): Boolean = {\n    locales.map(_.toLowerCase).contains(accountCountryCode.toLowerCase)\n  }\n\n  /**\n   * Boolean check of if a MagicFanout is high priority push\n   */\n  def checkIfHighPriorityNewsEventForCandidate(\n    candidate: MagicFanoutNewsEventPushCandidate\n  ): Future[Boolean] = {\n    candidate.isHighPriorityEvent.map { isHighPriority =>\n      isHighPriority && (candidate.target.params(PushFeatureSwitchParams.EnableHighPriorityPush))\n    }\n  }\n\n  /**\n   * Boolean check of if a MagicFanout event is high priority push\n   */\n  def checkIfHighPriorityEventForCandidate(\n    candidate: MagicFanoutEventPushCandidate\n  ): Future[Boolean] = {\n    candidate.isHighPriorityEvent.map { isHighPriority =>\n      candidate.commonRecType match {\n        case CommonRecommendationType.MagicFanoutSportsEvent =>\n          isHighPriority && (candidate.target.params(\n            PushFeatureSwitchParams.EnableHighPrioritySportsPush))\n        case _ => false\n      }\n    }\n  }\n\n  /**\n   * Boolean check if to skip target blue verified\n   */\n  def shouldSkipBlueVerifiedCheckForCandidate(\n    candidate: MagicFanoutProductLaunchPushCandidate\n  ): Future[Boolean] =\n    Future.value(\n      candidate.target.params(PushFeatureSwitchParams.DisableIsTargetBlueVerifiedPredicate))\n\n  /**\n   * Boolean check if to skip target is legacy verified\n   */\n  def shouldSkipLegacyVerifiedCheckForCandidate(\n    candidate: MagicFanoutProductLaunchPushCandidate\n  ): Future[Boolean] =\n    Future.value(\n      candidate.target.params(PushFeatureSwitchParams.DisableIsTargetLegacyVerifiedPredicate))\n\n  def shouldSkipSuperFollowCreatorCheckForCandidate(\n    candidate: MagicFanoutProductLaunchPushCandidate\n  ): Future[Boolean] =\n    Future.value(\n      !candidate.target.params(PushFeatureSwitchParams.EnableIsTargetSuperFollowCreatorPredicate))\n\n  /**\n   * Boolean check of if a reason of a MagicFanout is higher than the rank threshold of an event\n   */\n  def checkIfErgScEntityReasonMeetsThreshold(\n    rankThreshold: Int,\n    reason: MagicEventsReason,\n  ): Boolean = {\n    reason.reason match {\n      case TargetID.SemanticCoreID(scEntityId: SemanticCoreID) =>\n        reason.rank match {\n          case Some(rank) => rank < rankThreshold\n          case _ => false\n        }\n      case _ => false\n    }\n  }\n\n  /**\n   * Check if MagicEventsReasons contains a reason that matches the thresholdw\n   */\n  def checkIfValidErgScEntityReasonExists(\n    magicEventsReasons: Option[Seq[MagicEventsReason]],\n    rankThreshold: Int\n  )(\n    implicit stats: StatsReceiver\n  ): Boolean = {\n    magicEventsReasons match {\n      case Some(reasons) if reasons.exists(_.isNewUser.contains(true)) => true\n      case Some(reasons) =>\n        reasons.exists { reason =>\n          reason.source.contains(ReasonSource.ErgShortTermInterestSemanticCore) &&\n          checkIfErgScEntityReasonMeetsThreshold(\n            rankThreshold,\n            reason\n          )\n        }\n\n      case _ => false\n    }\n  }\n\n  /**\n   * Get event simcluster vector from event context\n   */\n  def getEventSimClusterVector(\n    simClustersEmbeddingOption: Option[Map[SimClustersEmbeddingId, ThriftSimClustersEmbedding]],\n    embeddingMapKey: (ModelVersion, EmbeddingType),\n    topKSimClustersCount: Int\n  ): Option[SimClusterScores] = {\n    simClustersEmbeddingOption.map { thriftSimClustersEmbeddings =>\n      val simClustersEmbeddings: Map[SimClustersEmbeddingId, SimClustersEmbedding] =\n        thriftSimClustersEmbeddings.map {\n          case (simClustersEmbeddingId, simClustersEmbeddingValue) =>\n            (simClustersEmbeddingId, SimClustersEmbedding(simClustersEmbeddingValue))\n        }.toMap\n      val emptySeq = Seq[(Int, Double)]()\n      val simClusterScoreTuple: Map[(ModelVersion, EmbeddingType), Seq[(Int, Double)]] =\n        SimClustersUtil\n          .getMaxTopKTweetSimClusters(simClustersEmbeddings, topKSimClustersCount)\n      SimClusterScores(simClusterScoreTuple.getOrElse(embeddingMapKey, emptySeq).toMap)\n    }\n  }\n\n  /**\n   * Get user simcluster vector magic events reasons\n   */\n  def getUserSimClusterVector(\n    magicEventsReasonsOpt: Option[Seq[MagicEventsReason]]\n  ): Option[SimClusterScores] = {\n    magicEventsReasonsOpt.map { magicEventsReasons: Seq[MagicEventsReason] =>\n      val reasons: Seq[(Int, Double)] = magicEventsReasons.flatMap { reason =>\n        reason.reason match {\n          case TargetID.SimClusterID(simClusterId: SimClusterID) =>\n            Some((simClusterId.clusterId, reason.score.getOrElse(0.0)))\n          case _ =>\n            None\n        }\n      }\n      SimClusterScores(reasons.toMap)\n    }\n  }\n\n  def reasonsContainGeoTarget(reasons: Seq[MagicEventsReason]): Boolean = {\n    reasons.exists { reason =>\n      val isGeoGraphSource = reason.source.contains(ReasonSource.GeoGraph)\n      reason.reason match {\n        case TargetID.PlaceID(_) if isGeoGraphSource => true\n        case _ => false\n      }\n    }\n  }\n\n  def geoPlaceIdsFromReasons(reasons: Seq[MagicEventsReason]): Set[Long] = {\n    reasons.flatMap { reason =>\n      val isGeoGraphSource = reason.source.contains(ReasonSource.GeoGraph)\n      reason.reason match {\n        case TargetID.PlaceID(PlaceID(id)) if isGeoGraphSource => Some(id)\n        case _ => None\n      }\n    }.toSet\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/magic_fanout/MagicFanoutSportsUtil.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.magic_fanout\n\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.NflFootballGameLiveUpdate\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.SoccerMatchLiveUpdate\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.SoccerPeriod\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.SportsEventHomeAwayTeamScore\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.SportsEventStatus\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.SportsEventTeamAlignment.Away\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.SportsEventTeamAlignment.Home\nimport com.twitter.escherbird.metadata.thriftscala.EntityMegadata\nimport com.twitter.frigate.pushservice.params.SportGameEnum\nimport com.twitter.frigate.common.base.GenericGameScore\nimport com.twitter.frigate.common.base.NflGameScore\nimport com.twitter.frigate.common.base.SoccerGameScore\nimport com.twitter.frigate.common.base.TeamInfo\nimport com.twitter.frigate.common.base.TeamScore\nimport com.twitter.hermit.store.semantic_core.SemanticEntityForQuery\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nobject MagicFanoutSportsUtil {\n\n  def transformSoccerGameScore(game: SoccerMatchLiveUpdate): Option[SoccerGameScore] = {\n    require(game.status.isDefined)\n    val gameScore = transformToGameScore(game.score, game.status.get)\n    val _penaltyKicks = transformToGameScore(game.penaltyScore, game.status.get)\n    gameScore.map { score =>\n      val _isGameEnd = game.status.get match {\n        case SportsEventStatus.Completed(_) => true\n        case _ => false\n      }\n\n      val _isHalfTime = game.period.exists { period =>\n        period match {\n          case SoccerPeriod.Halftime(_) => true\n          case _ => false\n        }\n      }\n\n      val _isOvertime = game.period.exists { period =>\n        period match {\n          case SoccerPeriod.PreOvertime(_) => true\n          case _ => false\n        }\n      }\n\n      val _isPenaltyKicks = game.period.exists { period =>\n        period match {\n          case SoccerPeriod.PrePenalty(_) => true\n          case SoccerPeriod.Penalty(_) => true\n          case _ => false\n        }\n      }\n\n      val _gameMinute = game.gameMinute.map { soccerGameMinute =>\n        game.minutesInInjuryTime match {\n          case Some(injuryTime) => s\"($soccerGameMinute+$injuryTime′)\"\n          case None => s\"($soccerGameMinute′)\"\n        }\n      }\n\n      SoccerGameScore(\n        score.home,\n        score.away,\n        isGameOngoing = score.isGameOngoing,\n        penaltyKicks = _penaltyKicks,\n        gameMinute = _gameMinute,\n        isHalfTime = _isHalfTime,\n        isOvertime = _isOvertime,\n        isPenaltyKicks = _isPenaltyKicks,\n        isGameEnd = _isGameEnd\n      )\n    }\n  }\n\n  def transformNFLGameScore(game: NflFootballGameLiveUpdate): Option[NflGameScore] = {\n    require(game.status.isDefined)\n\n    val gameScore = transformToGameScore(game.score, game.status.get)\n    gameScore.map { score =>\n      val _isGameEnd = game.status.get match {\n        case SportsEventStatus.Completed(_) => true\n        case _ => false\n      }\n\n      val _matchTime = (game.quarter, game.remainingSecondsInQuarter) match {\n        case (Some(quarter), Some(remainingSeconds)) if remainingSeconds != 0L =>\n          val m = (remainingSeconds / 60) % 60\n          val s = remainingSeconds % 60\n          val formattedSeconds = \"%02d:%02d\".format(m, s)\n          s\"(Q$quarter - $formattedSeconds)\"\n        case (Some(quarter), None) => s\"(Q$quarter)\"\n        case _ => \"\"\n      }\n\n      NflGameScore(\n        score.home,\n        score.away,\n        isGameOngoing = score.isGameOngoing,\n        isGameEnd = _isGameEnd,\n        matchTime = _matchTime\n      )\n    }\n  }\n\n  /**\n   Takes a score from Strato columns and turns it into an easier to handle structure (GameScore class)\n   We do this to easily access the home/away scenario for copy setting\n   */\n  def transformToGameScore(\n    scoreOpt: Option[SportsEventHomeAwayTeamScore],\n    status: SportsEventStatus\n  ): Option[GenericGameScore] = {\n    val isGameOngoing = status match {\n      case SportsEventStatus.InProgress(_) => true\n      case SportsEventStatus.Completed(_) => false\n      case _ => false\n    }\n\n    val scoresWithTeam = scoreOpt\n      .map { score =>\n        score.scores.map { score => (score.score, score.participantAlignment, score.participantId) }\n      }.getOrElse(Seq())\n\n    val tuple = scoresWithTeam match {\n      case Seq(teamOne, teamTwo, _*) => Some((teamOne, teamTwo))\n      case _ => None\n    }\n    tuple.flatMap {\n      case ((Some(teamOneScore), teamOneAlignment, teamOne), (Some(teamTwoScore), _, teamTwo)) =>\n        teamOneAlignment.flatMap {\n          case Home(_) =>\n            val home = TeamScore(teamOneScore, teamOne.entityId, teamOne.domainId)\n            val away = TeamScore(teamTwoScore, teamTwo.entityId, teamTwo.domainId)\n            Some(GenericGameScore(home, away, isGameOngoing))\n          case Away(_) =>\n            val away = TeamScore(teamOneScore, teamOne.entityId, teamOne.domainId)\n            val home = TeamScore(teamTwoScore, teamTwo.entityId, teamTwo.domainId)\n            Some(GenericGameScore(home, away, isGameOngoing))\n          case _ => None\n        }\n      case _ => None\n    }\n  }\n\n  def getTeamInfo(\n    team: TeamScore,\n    semanticCoreMegadataStore: ReadableStore[SemanticEntityForQuery, EntityMegadata]\n  ): Future[Option[TeamInfo]] = {\n    semanticCoreMegadataStore\n      .get(SemanticEntityForQuery(team.teamDomainId, team.teamEntityId)).map {\n        _.flatMap {\n          _.basicMetadata.map { metadata =>\n            TeamInfo(\n              name = metadata.name,\n              twitterUserId = metadata.twitter.flatMap(_.preferredTwitterUserId))\n          }\n        }\n      }\n  }\n\n  def getNFLReadableName(name: String): String = {\n    val teamNames =\n      Seq(\"\")\n    teamNames.find(teamName => name.contains(teamName)).getOrElse(name)\n  }\n\n  def getSoccerIbisMap(game: SoccerGameScore): Map[String, String] = {\n    val gameMinuteMap = game.gameMinute\n      .map { gameMinute => Map(\"match_time\" -> gameMinute) }\n      .getOrElse(Map.empty)\n\n    val updateTypeMap = {\n      if (game.isGameEnd) Map(\"is_game_end\" -> \"true\")\n      else if (game.isHalfTime) Map(\"is_half_time\" -> \"true\")\n      else if (game.isOvertime) Map(\"is_overtime\" -> \"true\")\n      else if (game.isPenaltyKicks) Map(\"is_penalty_kicks\" -> \"true\")\n      else Map(\"is_score_update\" -> \"true\")\n    }\n\n    val awayScore = game match {\n      case SoccerGameScore(_, away, _, None, _, _, _, _, _) =>\n        away.score.toString\n      case SoccerGameScore(_, away, _, Some(penaltyKick), _, _, _, _, _) =>\n        s\"${away.score} (${penaltyKick.away.score}) \"\n      case _ => \"\"\n    }\n\n    val homeScore = game match {\n      case SoccerGameScore(home, _, _, None, _, _, _, _, _) =>\n        home.score.toString\n      case SoccerGameScore(home, _, _, Some(penaltyKick), _, _, _, _, _) =>\n        s\"${home.score} (${penaltyKick.home.score}) \"\n      case _ => \"\"\n    }\n\n    val scoresMap = Map(\n      \"away_score\" -> awayScore,\n      \"home_score\" -> homeScore,\n    )\n\n    gameType(SportGameEnum.Soccer) ++ updateTypeMap ++ gameMinuteMap ++ scoresMap\n  }\n\n  def getNflIbisMap(game: NflGameScore): Map[String, String] = {\n    val gameMinuteMap = Map(\"match_time\" -> game.matchTime)\n\n    val updateTypeMap = {\n      if (game.isGameEnd) Map(\"is_game_end\" -> \"true\")\n      else Map(\"is_score_update\" -> \"true\")\n    }\n\n    val awayScore = game.away.score\n    val homeScore = game.home.score\n\n    val scoresMap = Map(\n      \"away_score\" -> awayScore.toString,\n      \"home_score\" -> homeScore.toString,\n    )\n\n    gameType(SportGameEnum.Nfl) ++ updateTypeMap ++ gameMinuteMap ++ scoresMap\n  }\n\n  private def gameType(game: SportGameEnum.Value): Map[String, String] = {\n    game match {\n      case SportGameEnum.Soccer => Map(\"is_soccer_game\" -> \"true\")\n      case SportGameEnum.Nfl => Map(\"is_nfl_game\" -> \"true\")\n      case _ => Map.empty\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/magic_fanout/MagicFanoutTargetingPredicateWrappersForCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.magic_fanout\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.interests.InterestsLookupRequestWithContext\nimport com.twitter.frigate.common.util.FeatureSwitchParams\nimport com.twitter.frigate.common.util.MagicFanoutTargetingPredicatesEnum\nimport com.twitter.frigate.common.util.MagicFanoutTargetingPredicatesEnum.MagicFanoutTargetingPredicatesEnum\nimport com.twitter.frigate.pushservice.model.MagicFanoutEventPushCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.interests.thriftscala.UserInterests\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi.FSEnumParam\n\nobject MagicFanoutTargetingPredicateWrappersForCandidate {\n\n  /**\n   * Combine Prod and Experimental Targeting predicate logic\n   * @return: NamedPredicate[MagicFanoutNewsEventPushCandidate]\n   */\n  def magicFanoutTargetingPredicate(\n    stats: StatsReceiver,\n    config: Config\n  ): NamedPredicate[MagicFanoutEventPushCandidate] = {\n    val name = \"magic_fanout_targeting_predicate\"\n    Predicate\n      .fromAsync { candidate: MagicFanoutEventPushCandidate =>\n        val mfTargetingPredicateParam = getTargetingPredicateParams(candidate)\n        val mfTargetingPredicate = MagicFanoutTargetingPredicateMapForCandidate\n          .apply(config)\n          .get(candidate.target.params(mfTargetingPredicateParam))\n        mfTargetingPredicate match {\n          case Some(predicate) =>\n            predicate.apply(Seq(candidate)).map(_.head)\n          case None =>\n            throw new Exception(\n              s\"MFTargetingPredicateMap doesnt contain value for TargetingParam: ${FeatureSwitchParams.MFTargetingPredicate}\")\n        }\n      }\n      .withStats(stats.scope(name))\n      .withName(name)\n  }\n\n  private def getTargetingPredicateParams(\n    candidate: MagicFanoutEventPushCandidate\n  ): FSEnumParam[MagicFanoutTargetingPredicatesEnum.type] = {\n    if (candidate.commonRecType == CommonRecommendationType.MagicFanoutSportsEvent) {\n      FeatureSwitchParams.MFCricketTargetingPredicate\n    } else FeatureSwitchParams.MFTargetingPredicate\n  }\n\n  /**\n   * SimCluster and ERG and Topic Follows Targeting Predicate\n   */\n  def simClusterErgTopicFollowsTargetingPredicate(\n    implicit stats: StatsReceiver,\n    interestsLookupStore: ReadableStore[InterestsLookupRequestWithContext, UserInterests]\n  ): NamedPredicate[MagicFanoutEventPushCandidate] = {\n    simClusterErgTargetingPredicate\n      .or(MagicFanoutPredicatesForCandidate.magicFanoutTopicFollowsTargetingPredicate)\n      .withName(\"sim_cluster_erg_topic_follows_targeting\")\n  }\n\n  /**\n   * SimCluster and ERG and Topic Follows Targeting Predicate\n   */\n  def simClusterErgTopicFollowsUserFollowsTargetingPredicate(\n    implicit stats: StatsReceiver,\n    interestsLookupStore: ReadableStore[InterestsLookupRequestWithContext, UserInterests]\n  ): NamedPredicate[MagicFanoutEventPushCandidate] = {\n    simClusterErgTopicFollowsTargetingPredicate\n      .or(\n        MagicFanoutPredicatesForCandidate.followRankThreshold(\n          PushFeatureSwitchParams.MagicFanoutRealgraphRankThreshold))\n      .withName(\"sim_cluster_erg_topic_follows_user_follows_targeting\")\n  }\n\n  /**\n   * SimCluster and ERG Targeting Predicate\n   */\n  def simClusterErgTargetingPredicate(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[MagicFanoutEventPushCandidate] = {\n    MagicFanoutPredicatesForCandidate.magicFanoutSimClusterTargetingPredicate\n      .or(MagicFanoutPredicatesForCandidate.magicFanoutErgInterestRankThresholdPredicate)\n      .withName(\"sim_cluster_erg_targeting\")\n  }\n}\n\n/**\n * Object to initalze and get predicate map\n */\nobject MagicFanoutTargetingPredicateMapForCandidate {\n\n  /**\n   * Called from the Config.scala at the time of server initialization\n   * @param statsReceiver: implict stats receiver\n   * @return Map[MagicFanoutTargetingPredicatesEnum, NamedPredicate[MagicFanoutNewsEventPushCandidate]]\n   */\n  def apply(\n    config: Config\n  ): Map[MagicFanoutTargetingPredicatesEnum, NamedPredicate[MagicFanoutEventPushCandidate]] = {\n    Map(\n      MagicFanoutTargetingPredicatesEnum.SimClusterAndERGAndTopicFollows -> MagicFanoutTargetingPredicateWrappersForCandidate\n        .simClusterErgTopicFollowsTargetingPredicate(\n          config.statsReceiver,\n          config.interestsWithLookupContextStore),\n      MagicFanoutTargetingPredicatesEnum.SimClusterAndERG -> MagicFanoutTargetingPredicateWrappersForCandidate\n        .simClusterErgTargetingPredicate(config.statsReceiver),\n      MagicFanoutTargetingPredicatesEnum.SimCluster -> MagicFanoutPredicatesForCandidate\n        .magicFanoutSimClusterTargetingPredicate(config.statsReceiver),\n      MagicFanoutTargetingPredicatesEnum.ERG -> MagicFanoutPredicatesForCandidate\n        .magicFanoutErgInterestRankThresholdPredicate(config.statsReceiver),\n      MagicFanoutTargetingPredicatesEnum.TopicFollows -> MagicFanoutPredicatesForCandidate\n        .magicFanoutTopicFollowsTargetingPredicate(\n          config.statsReceiver,\n          config.interestsWithLookupContextStore),\n      MagicFanoutTargetingPredicatesEnum.UserFollows -> MagicFanoutPredicatesForCandidate\n        .followRankThreshold(\n          PushFeatureSwitchParams.MagicFanoutRealgraphRankThreshold\n        )(config.statsReceiver),\n      MagicFanoutTargetingPredicatesEnum.SimClusterAndERGAndTopicFollowsAndUserFollows ->\n        MagicFanoutTargetingPredicateWrappersForCandidate\n          .simClusterErgTopicFollowsUserFollowsTargetingPredicate(\n            config.statsReceiver,\n            config.interestsWithLookupContextStore\n          )\n    )\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/ntab_caret_fatigue/CRTBasedNtabCaretClickFatiguePredicates.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.notificationservice.thriftscala.GenericType\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.notificationservice.genericfeedbackstore.FeedbackPromptValue\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.frigate.common.base.Candidate\nimport com.twitter.frigate.common.base.RecommendationType\nimport com.twitter.frigate.common.base.TargetInfo\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.SeeLessOftenType\nimport com.twitter.frigate.common.history.History\nimport com.twitter.frigate.common.predicate.FrigateHistoryFatiguePredicate.TimeSeries\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.common.predicate.ntab_caret_fatigue.NtabCaretClickFatiguePredicateHelper\nimport com.twitter.frigate.pushservice.predicate.CaretFeedbackHistoryFilter\nimport com.twitter.notificationservice.thriftscala.CaretFeedbackDetails\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.frigate.common.predicate.FatiguePredicate\nimport com.twitter.frigate.pushservice.util.PushCapUtil\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\n\nobject CRTBasedNtabCaretClickFatiguePredicates {\n\n  private val MagicRecsCategory = \"MagicRecs\"\n\n  private val HighQualityRefreshableTypes: Set[Option[String]] = Set(\n    Some(\"MagicRecHighQualityTweet\"),\n  )\n\n  private def getUserStateWeight(target: Target): Future[Double] = {\n    PushDeviceUtil.isNtabOnlyEligible.map {\n      case true =>\n        target.params(PushFeatureSwitchParams.SeeLessOftenNtabOnlyNotifUserPushCapWeight)\n      case _ => 1.0\n    }\n  }\n\n  def crtToSeeLessOftenType(\n    crt: CommonRecommendationType,\n    candidate: Candidate\n      with RecommendationType\n      with TargetInfo[\n        Target\n      ],\n  ): SeeLessOftenType = {\n    val crtToSeeLessOftenTypeMap: Map[CommonRecommendationType, SeeLessOftenType] = {\n      RecTypes.f1FirstDegreeTypes.map((_, SeeLessOftenType.F1Type)).toMap\n    }\n\n    crtToSeeLessOftenTypeMap.getOrElse(crt, SeeLessOftenType.OtherTypes)\n  }\n\n  def genericTypeToSeeLessOftenType(\n    genericType: GenericType,\n    candidate: Candidate\n      with RecommendationType\n      with TargetInfo[\n        Target\n      ]\n  ): SeeLessOftenType = {\n    val genericTypeToSeeLessOftenTypeMap: Map[GenericType, SeeLessOftenType] = {\n      Map(GenericType.MagicRecFirstDegreeTweetRecent -> SeeLessOftenType.F1Type)\n    }\n\n    genericTypeToSeeLessOftenTypeMap.getOrElse(genericType, SeeLessOftenType.OtherTypes)\n  }\n\n  def getWeightForCaretFeedback(\n    dislikedType: SeeLessOftenType,\n    candidate: Candidate\n      with RecommendationType\n      with TargetInfo[\n        Target\n      ]\n  ): Double = {\n    def getWeightFromDislikedAndCurrentType(\n      dislikedType: SeeLessOftenType,\n      currentType: SeeLessOftenType\n    ): Double = {\n      val weightMap: Map[(SeeLessOftenType, SeeLessOftenType), Double] = {\n\n        Map(\n          (SeeLessOftenType.F1Type, SeeLessOftenType.F1Type) -> candidate.target.params(\n            PushFeatureSwitchParams.SeeLessOftenF1TriggerF1PushCapWeight),\n          (SeeLessOftenType.OtherTypes, SeeLessOftenType.OtherTypes) -> candidate.target.params(\n            PushFeatureSwitchParams.SeeLessOftenNonF1TriggerNonF1PushCapWeight),\n          (SeeLessOftenType.F1Type, SeeLessOftenType.OtherTypes) -> candidate.target.params(\n            PushFeatureSwitchParams.SeeLessOftenF1TriggerNonF1PushCapWeight),\n          (SeeLessOftenType.OtherTypes, SeeLessOftenType.F1Type) -> candidate.target.params(\n            PushFeatureSwitchParams.SeeLessOftenNonF1TriggerF1PushCapWeight)\n        )\n      }\n\n      weightMap\n        .getOrElse(\n          (dislikedType, currentType),\n          candidate.target.params(PushFeatureSwitchParams.SeeLessOftenDefaultPushCapWeight))\n    }\n\n    getWeightFromDislikedAndCurrentType(\n      dislikedType,\n      crtToSeeLessOftenType(candidate.commonRecType, candidate))\n  }\n\n  private def isOutsideCrtBasedNtabCaretClickFatiguePeriodContFn(\n    candidate: Candidate\n      with RecommendationType\n      with TargetInfo[\n        Target\n      ],\n    history: History,\n    feedbackDetails: Seq[CaretFeedbackDetails],\n    filterHistory: TimeSeries => TimeSeries =\n      FatiguePredicate.recTypesOnlyFilter(RecTypes.sharedNTabCaretFatigueTypes),\n    filterCaretFeedbackHistory: Target => Seq[\n      CaretFeedbackDetails\n    ] => Seq[CaretFeedbackDetails] =\n      CaretFeedbackHistoryFilter.caretFeedbackHistoryFilter(Seq(MagicRecsCategory)),\n    knobs: Seq[Double],\n    pushCapKnobs: Seq[Double],\n    powerKnobs: Seq[Double],\n    f1Weight: Double,\n    nonF1Weight: Double,\n    defaultPushCap: Int,\n    stats: StatsReceiver,\n    tripHqTweetWeight: Double = 0.0,\n  ): Boolean = {\n    val filteredFeedbackDetails = filterCaretFeedbackHistory(candidate.target)(feedbackDetails)\n    val weight = {\n      if (RecTypes.HighQualityTweetTypes.contains(\n          candidate.commonRecType) && (tripHqTweetWeight != 0)) {\n        tripHqTweetWeight\n      } else if (RecTypes.isF1Type(candidate.commonRecType)) {\n        f1Weight\n      } else {\n        nonF1Weight\n      }\n    }\n    val filteredHistory = History(filterHistory(history.history.toSeq).toMap)\n    isOutsideFatiguePeriod(\n      filteredHistory,\n      filteredFeedbackDetails,\n      Seq(),\n      ContinuousFunctionParam(\n        knobs,\n        pushCapKnobs,\n        powerKnobs,\n        weight,\n        defaultPushCap\n      ),\n      stats.scope(\n        if (RecTypes.isF1Type(candidate.commonRecType)) \"mr_ntab_dislike_f1_candidate_fn\"\n        else if (RecTypes.HighQualityTweetTypes.contains(candidate.commonRecType))\n          \"mr_ntab_dislike_high_quality_candidate_fn\"\n        else \"mr_ntab_dislike_nonf1_candidate_fn\")\n    )\n  }\n\n  private def isOutsideFatiguePeriod(\n    history: History,\n    feedbackDetails: Seq[CaretFeedbackDetails],\n    feedbacks: Seq[FeedbackModel],\n    param: ContinuousFunctionParam,\n    stats: StatsReceiver\n  ): Boolean = {\n    val fatiguePeriod: Duration =\n      NtabCaretClickFatigueUtils.durationToFilterForFeedback(\n        feedbackDetails,\n        feedbacks,\n        param,\n        param.defaultValue,\n        stats\n      )\n\n    val hasRecentSent =\n      NtabCaretClickFatiguePredicateHelper.hasRecentSend(history, fatiguePeriod)\n    !hasRecentSent\n\n  }\n\n  def genericCRTBasedNtabCaretClickFnFatiguePredicate[\n    Cand <: Candidate with RecommendationType with TargetInfo[\n      Target\n    ]\n  ](\n    filterHistory: TimeSeries => TimeSeries =\n      FatiguePredicate.recTypesOnlyFilter(RecTypes.sharedNTabCaretFatigueTypes),\n    filterCaretFeedbackHistory: Target => Seq[\n      CaretFeedbackDetails\n    ] => Seq[CaretFeedbackDetails] = CaretFeedbackHistoryFilter\n      .caretFeedbackHistoryFilter(Seq(MagicRecsCategory)),\n    filterInlineFeedbackHistory: Seq[FeedbackModel] => Seq[FeedbackModel] =\n      NtabCaretClickFatigueUtils.feedbackModelFilterByCRT(RecTypes.sharedNTabCaretFatigueTypes)\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[Cand] = {\n    val predicateName = \"generic_crt_based_ntab_dislike_fatigue_fn\"\n    Predicate\n      .fromAsync[Cand] { cand: Cand =>\n        {\n          if (!cand.target.params(PushFeatureSwitchParams.EnableGenericCRTBasedFatiguePredicate)) {\n            Future.True\n          } else {\n            val scopedStats = stats.scope(predicateName)\n            val totalRequests = scopedStats.counter(\"mr_ntab_dislike_total\")\n            val total90Day =\n              scopedStats.counter(\"mr_ntab_dislike_90day_dislike\")\n            val totalDisabled =\n              scopedStats.counter(\"mr_ntab_dislike_not_90day_dislike\")\n            val totalSuccess = scopedStats.counter(\"mr_ntab_dislike_success\")\n            val totalFiltered = scopedStats.counter(\"mr_ntab_dislike_filtered\")\n            val totalWithHistory =\n              scopedStats.counter(\"mr_ntab_dislike_with_history\")\n            val totalWithoutHistory =\n              scopedStats.counter(\"mr_ntab_dislike_without_history\")\n            totalRequests.incr()\n\n            Future\n              .join(\n                cand.target.history,\n                cand.target.caretFeedbacks,\n                cand.target.dynamicPushcap,\n                cand.target.optoutAdjustedPushcap,\n                PushCapUtil.getDefaultPushCap(cand.target),\n                getUserStateWeight(cand.target)\n              ).map {\n                case (\n                      history,\n                      Some(feedbackDetails),\n                      dynamicPushcapOpt,\n                      optoutAdjustedPushcapOpt,\n                      defaultPushCap,\n                      userStateWeight) => {\n                  totalWithHistory.incr()\n\n                  val feedbackDetailsDeduped =\n                    NtabCaretClickFatiguePredicateHelper.dedupFeedbackDetails(\n                      filterCaretFeedbackHistory(cand.target)(feedbackDetails),\n                      stats\n                    )\n\n                  val pushCap: Int = (dynamicPushcapOpt, optoutAdjustedPushcapOpt) match {\n                    case (_, Some(optoutAdjustedPushcap)) => optoutAdjustedPushcap\n                    case (Some(pushcapInfo), _) => pushcapInfo.pushcap\n                    case _ => defaultPushCap\n                  }\n                  val filteredHistory = History(filterHistory(history.history.toSeq).toMap)\n\n                  val hasUserDislikeInLast90Days =\n                    NtabCaretClickFatigueUtils.hasUserDislikeInLast90Days(feedbackDetailsDeduped)\n                  val isF1TriggerFatigueEnabled = cand.target\n                    .params(PushFeatureSwitchParams.EnableContFnF1TriggerSeeLessOftenFatigue)\n                  val isNonF1TriggerFatigueEnabled = cand.target.params(\n                    PushFeatureSwitchParams.EnableContFnNonF1TriggerSeeLessOftenFatigue)\n\n                  val isOutisdeSeeLessOftenFatigue =\n                    if (hasUserDislikeInLast90Days && (isF1TriggerFatigueEnabled || isNonF1TriggerFatigueEnabled)) {\n                      total90Day.incr()\n\n                      val feedbackDetailsGroupedBySeeLessOftenType: Map[Option[\n                        SeeLessOftenType\n                      ], Seq[\n                        CaretFeedbackDetails\n                      ]] = feedbackDetails.groupBy(feedbackDetail =>\n                        feedbackDetail.genericNotificationMetadata.map(x =>\n                          genericTypeToSeeLessOftenType(x.genericType, cand)))\n\n                      val isOutsideFatiguePeriodSeq =\n                        for (elem <- feedbackDetailsGroupedBySeeLessOftenType if elem._1.isDefined)\n                          yield {\n                            val dislikedSeeLessOftenType: SeeLessOftenType = elem._1.get\n                            val seqCaretFeedbackDetails: Seq[CaretFeedbackDetails] = elem._2\n\n                            val weight = getWeightForCaretFeedback(\n                              dislikedSeeLessOftenType,\n                              cand) * userStateWeight\n\n                            if (isOutsideFatiguePeriod(\n                                history = filteredHistory,\n                                feedbackDetails = seqCaretFeedbackDetails,\n                                feedbacks = Seq(),\n                                param = ContinuousFunctionParam(\n                                  knobs = cand.target\n                                    .params(PushFeatureSwitchParams.SeeLessOftenListOfDayKnobs),\n                                  knobValues = cand.target\n                                    .params(\n                                      PushFeatureSwitchParams.SeeLessOftenListOfPushCapWeightKnobs).map(\n                                      _ * pushCap),\n                                  powers = cand.target\n                                    .params(PushFeatureSwitchParams.SeeLessOftenListOfPowerKnobs),\n                                  weight = weight,\n                                  defaultValue = pushCap\n                                ),\n                                scopedStats\n                              )) {\n                              true\n                            } else {\n                              false\n                            }\n                          }\n\n                      isOutsideFatiguePeriodSeq.forall(identity)\n                    } else {\n                      totalDisabled.incr()\n                      true\n                    }\n\n                  if (isOutisdeSeeLessOftenFatigue) {\n                    totalSuccess.incr()\n                  } else totalFiltered.incr()\n\n                  isOutisdeSeeLessOftenFatigue\n                }\n\n                case _ =>\n                  totalSuccess.incr()\n                  totalWithoutHistory.incr()\n                  true\n              }\n          }\n        }\n      }.withStats(stats.scope(predicateName))\n      .withName(predicateName)\n  }\n\n  def f1TriggeredCRTBasedNtabCaretClickFnFatiguePredicate[\n    Cand <: Candidate with RecommendationType with TargetInfo[\n      Target\n    ]\n  ](\n    filterHistory: TimeSeries => TimeSeries =\n      FatiguePredicate.recTypesOnlyFilter(RecTypes.sharedNTabCaretFatigueTypes),\n    filterCaretFeedbackHistory: Target => Seq[\n      CaretFeedbackDetails\n    ] => Seq[CaretFeedbackDetails] = CaretFeedbackHistoryFilter\n      .caretFeedbackHistoryFilter(Seq(MagicRecsCategory)),\n    filterInlineFeedbackHistory: Seq[FeedbackModel] => Seq[FeedbackModel] =\n      NtabCaretClickFatigueUtils.feedbackModelFilterByCRT(RecTypes.sharedNTabCaretFatigueTypes)\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[Cand] = {\n    val predicateName = \"f1_triggered_crt_based_ntab_dislike_fatigue_fn\"\n    Predicate\n      .fromAsync[Cand] { cand: Cand =>\n        {\n          val scopedStats = stats.scope(predicateName)\n          val totalRequests = scopedStats.counter(\"mr_ntab_dislike_total\")\n          val total90Day =\n            scopedStats.counter(\"mr_ntab_dislike_90day_dislike\")\n          val totalDisabled =\n            scopedStats.counter(\"mr_ntab_dislike_not_90day_dislike\")\n          val totalSuccess = scopedStats.counter(\"mr_ntab_dislike_success\")\n          val totalFiltered = scopedStats.counter(\"mr_ntab_dislike_filtered\")\n          val totalWithHistory =\n            scopedStats.counter(\"mr_ntab_dislike_with_history\")\n          val totalWithoutHistory =\n            scopedStats.counter(\"mr_ntab_dislike_without_history\")\n          totalRequests.incr()\n\n          Future\n            .join(\n              cand.target.history,\n              cand.target.caretFeedbacks,\n              cand.target.dynamicPushcap,\n              cand.target.optoutAdjustedPushcap,\n              cand.target.notificationFeedbacks,\n              PushCapUtil.getDefaultPushCap(cand.target),\n              getUserStateWeight(cand.target)\n            ).map {\n              case (\n                    history,\n                    Some(feedbackDetails),\n                    dynamicPushcapOpt,\n                    optoutAdjustedPushcapOpt,\n                    Some(feedbacks),\n                    defaultPushCap,\n                    userStateWeight) =>\n                totalWithHistory.incr()\n\n                val feedbackDetailsDeduped =\n                  NtabCaretClickFatiguePredicateHelper.dedupFeedbackDetails(\n                    filterCaretFeedbackHistory(cand.target)(feedbackDetails),\n                    stats\n                  )\n\n                val pushCap: Int = (dynamicPushcapOpt, optoutAdjustedPushcapOpt) match {\n                  case (_, Some(optoutAdjustedPushcap)) => optoutAdjustedPushcap\n                  case (Some(pushcapInfo), _) => pushcapInfo.pushcap\n                  case _ => defaultPushCap\n                }\n                val filteredHistory = History(filterHistory(history.history.toSeq).toMap)\n\n                val isOutsideInlineDislikeFatigue =\n                  if (cand.target\n                      .params(PushFeatureSwitchParams.EnableContFnF1TriggerInlineFeedbackFatigue)) {\n                    val weight =\n                      if (RecTypes.isF1Type(cand.commonRecType)) {\n                        cand.target\n                          .params(PushFeatureSwitchParams.InlineFeedbackF1TriggerF1PushCapWeight)\n                      } else {\n                        cand.target\n                          .params(PushFeatureSwitchParams.InlineFeedbackF1TriggerNonF1PushCapWeight)\n                      }\n\n                    val inlineFeedbackFatigueParam = ContinuousFunctionParam(\n                      cand.target\n                        .params(PushFeatureSwitchParams.InlineFeedbackListOfDayKnobs),\n                      cand.target\n                        .params(PushFeatureSwitchParams.InlineFeedbackListOfPushCapWeightKnobs)\n                        .map(_ * pushCap),\n                      cand.target\n                        .params(PushFeatureSwitchParams.InlineFeedbackListOfPowerKnobs),\n                      weight,\n                      pushCap\n                    )\n\n                    isInlineDislikeOutsideFatiguePeriod(\n                      cand,\n                      feedbacks\n                        .collect {\n                          case feedbackPromptValue: FeedbackPromptValue =>\n                            InlineFeedbackModel(feedbackPromptValue, None)\n                        },\n                      filteredHistory,\n                      Seq(\n                        filterInlineFeedbackHistory,\n                        NtabCaretClickFatigueUtils.feedbackModelFilterByCRT(\n                          RecTypes.f1FirstDegreeTypes)),\n                      inlineFeedbackFatigueParam,\n                      scopedStats\n                    )\n                  } else true\n\n                lazy val isOutsidePromptDislikeFatigue =\n                  if (cand.target\n                      .params(PushFeatureSwitchParams.EnableContFnF1TriggerPromptFeedbackFatigue)) {\n                    val weight =\n                      if (RecTypes.isF1Type(cand.commonRecType)) {\n                        cand.target\n                          .params(PushFeatureSwitchParams.PromptFeedbackF1TriggerF1PushCapWeight)\n                      } else {\n                        cand.target\n                          .params(PushFeatureSwitchParams.PromptFeedbackF1TriggerNonF1PushCapWeight)\n                      }\n\n                    val promptFeedbackFatigueParam = ContinuousFunctionParam(\n                      cand.target\n                        .params(PushFeatureSwitchParams.PromptFeedbackListOfDayKnobs),\n                      cand.target\n                        .params(PushFeatureSwitchParams.PromptFeedbackListOfPushCapWeightKnobs)\n                        .map(_ * pushCap),\n                      cand.target\n                        .params(PushFeatureSwitchParams.PromptFeedbackListOfPowerKnobs),\n                      weight,\n                      pushCap\n                    )\n\n                    isPromptDislikeOutsideFatiguePeriod(\n                      feedbacks\n                        .collect {\n                          case feedbackPromptValue: FeedbackPromptValue =>\n                            PromptFeedbackModel(feedbackPromptValue, None)\n                        },\n                      filteredHistory,\n                      Seq(\n                        filterInlineFeedbackHistory,\n                        NtabCaretClickFatigueUtils.feedbackModelFilterByCRT(\n                          RecTypes.f1FirstDegreeTypes)),\n                      promptFeedbackFatigueParam,\n                      scopedStats\n                    )\n                  } else true\n\n                isOutsideInlineDislikeFatigue && isOutsidePromptDislikeFatigue\n\n              case _ =>\n                totalSuccess.incr()\n                totalWithoutHistory.incr()\n                true\n            }\n        }\n      }.withStats(stats.scope(predicateName))\n      .withName(predicateName)\n  }\n\n  def nonF1TriggeredCRTBasedNtabCaretClickFnFatiguePredicate[\n    Cand <: Candidate with RecommendationType with TargetInfo[\n      Target\n    ]\n  ](\n    filterHistory: TimeSeries => TimeSeries =\n      FatiguePredicate.recTypesOnlyFilter(RecTypes.sharedNTabCaretFatigueTypes),\n    filterCaretFeedbackHistory: Target => Seq[\n      CaretFeedbackDetails\n    ] => Seq[CaretFeedbackDetails] = CaretFeedbackHistoryFilter\n      .caretFeedbackHistoryFilter(Seq(MagicRecsCategory)),\n    filterInlineFeedbackHistory: Seq[FeedbackModel] => Seq[FeedbackModel] =\n      NtabCaretClickFatigueUtils.feedbackModelFilterByCRT(RecTypes.sharedNTabCaretFatigueTypes)\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[Cand] = {\n    val predicateName = \"non_f1_triggered_crt_based_ntab_dislike_fatigue_fn\"\n    Predicate\n      .fromAsync[Cand] { cand: Cand =>\n        {\n          val scopedStats = stats.scope(predicateName)\n          val totalRequests = scopedStats.counter(\"mr_ntab_dislike_total\")\n          val total90Day =\n            scopedStats.counter(\"mr_ntab_dislike_90day_dislike\")\n          val totalDisabled =\n            scopedStats.counter(\"mr_ntab_dislike_not_90day_dislike\")\n          val totalSuccess = scopedStats.counter(\"mr_ntab_dislike_success\")\n          val totalFiltered = scopedStats.counter(\"mr_ntab_dislike_filtered\")\n          val totalWithHistory =\n            scopedStats.counter(\"mr_ntab_dislike_with_history\")\n          val totalWithoutHistory =\n            scopedStats.counter(\"mr_ntab_dislike_without_history\")\n          val totalFeedbackSuccess = scopedStats.counter(\"mr_total_feedback_success\")\n          totalRequests.incr()\n\n          Future\n            .join(\n              cand.target.history,\n              cand.target.caretFeedbacks,\n              cand.target.dynamicPushcap,\n              cand.target.optoutAdjustedPushcap,\n              cand.target.notificationFeedbacks,\n              PushCapUtil.getDefaultPushCap(cand.target),\n              getUserStateWeight(cand.target),\n            ).map {\n              case (\n                    history,\n                    Some(feedbackDetails),\n                    dynamicPushcapOpt,\n                    optoutAdjustedPushcapOpt,\n                    Some(feedbacks),\n                    defaultPushCap,\n                    userStateWeight) =>\n                totalWithHistory.incr()\n\n                val filteredfeedbackDetails =\n                  if (cand.target.params(\n                      PushFeatureSwitchParams.AdjustTripHqTweetTriggeredNtabCaretClickFatigue)) {\n                    val refreshableTypeFilter = CaretFeedbackHistoryFilter\n                      .caretFeedbackHistoryFilterByRefreshableTypeDenyList(\n                        HighQualityRefreshableTypes)\n                    refreshableTypeFilter(cand.target)(feedbackDetails)\n                  } else {\n                    feedbackDetails\n                  }\n\n                val feedbackDetailsDeduped =\n                  NtabCaretClickFatiguePredicateHelper.dedupFeedbackDetails(\n                    filterCaretFeedbackHistory(cand.target)(filteredfeedbackDetails),\n                    stats\n                  )\n\n                val pushCap: Int = (dynamicPushcapOpt, optoutAdjustedPushcapOpt) match {\n                  case (_, Some(optoutAdjustedPushcap)) => optoutAdjustedPushcap\n                  case (Some(pushcapInfo), _) => pushcapInfo.pushcap\n                  case _ => defaultPushCap\n                }\n                val filteredHistory = History(filterHistory(history.history.toSeq).toMap)\n\n                val isOutsideInlineDislikeFatigue =\n                  if (cand.target\n                      .params(\n                        PushFeatureSwitchParams.EnableContFnNonF1TriggerInlineFeedbackFatigue)) {\n                    val weight =\n                      if (RecTypes.isF1Type(cand.commonRecType))\n                        cand.target\n                          .params(PushFeatureSwitchParams.InlineFeedbackNonF1TriggerF1PushCapWeight)\n                      else\n                        cand.target\n                          .params(\n                            PushFeatureSwitchParams.InlineFeedbackNonF1TriggerNonF1PushCapWeight)\n\n                    val inlineFeedbackFatigueParam = ContinuousFunctionParam(\n                      cand.target\n                        .params(PushFeatureSwitchParams.InlineFeedbackListOfDayKnobs),\n                      cand.target\n                        .params(PushFeatureSwitchParams.InlineFeedbackListOfPushCapWeightKnobs)\n                        .map(_ * pushCap),\n                      cand.target\n                        .params(PushFeatureSwitchParams.InlineFeedbackListOfPowerKnobs),\n                      weight,\n                      pushCap\n                    )\n\n                    val excludedCRTs: Set[CommonRecommendationType] =\n                      if (cand.target.params(\n                          PushFeatureSwitchParams.AdjustTripHqTweetTriggeredNtabCaretClickFatigue)) {\n                        RecTypes.f1FirstDegreeTypes ++ RecTypes.HighQualityTweetTypes\n                      } else {\n                        RecTypes.f1FirstDegreeTypes\n                      }\n\n                    isInlineDislikeOutsideFatiguePeriod(\n                      cand,\n                      feedbacks\n                        .collect {\n                          case feedbackPromptValue: FeedbackPromptValue =>\n                            InlineFeedbackModel(feedbackPromptValue, None)\n                        },\n                      filteredHistory,\n                      Seq(\n                        filterInlineFeedbackHistory,\n                        NtabCaretClickFatigueUtils.feedbackModelExcludeCRT(excludedCRTs)),\n                      inlineFeedbackFatigueParam,\n                      scopedStats\n                    )\n                  } else true\n\n                lazy val isOutsidePromptDislikeFatigue =\n                  if (cand.target\n                      .params(\n                        PushFeatureSwitchParams.EnableContFnNonF1TriggerPromptFeedbackFatigue)) {\n                    val weight =\n                      if (RecTypes.isF1Type(cand.commonRecType))\n                        cand.target\n                          .params(PushFeatureSwitchParams.PromptFeedbackNonF1TriggerF1PushCapWeight)\n                      else\n                        cand.target\n                          .params(\n                            PushFeatureSwitchParams.PromptFeedbackNonF1TriggerNonF1PushCapWeight)\n\n                    val promptFeedbackFatigueParam = ContinuousFunctionParam(\n                      cand.target\n                        .params(PushFeatureSwitchParams.PromptFeedbackListOfDayKnobs),\n                      cand.target\n                        .params(PushFeatureSwitchParams.PromptFeedbackListOfPushCapWeightKnobs)\n                        .map(_ * pushCap),\n                      cand.target\n                        .params(PushFeatureSwitchParams.PromptFeedbackListOfPowerKnobs),\n                      weight,\n                      pushCap\n                    )\n\n                    isPromptDislikeOutsideFatiguePeriod(\n                      feedbacks\n                        .collect {\n                          case feedbackPromptValue: FeedbackPromptValue =>\n                            PromptFeedbackModel(feedbackPromptValue, None)\n                        },\n                      filteredHistory,\n                      Seq(\n                        filterInlineFeedbackHistory,\n                        NtabCaretClickFatigueUtils.feedbackModelExcludeCRT(\n                          RecTypes.f1FirstDegreeTypes)),\n                      promptFeedbackFatigueParam,\n                      scopedStats\n                    )\n                  } else true\n\n                isOutsideInlineDislikeFatigue && isOutsidePromptDislikeFatigue\n              case _ =>\n                totalFeedbackSuccess.incr()\n                totalWithoutHistory.incr()\n                true\n            }\n        }\n      }.withStats(stats.scope(predicateName))\n      .withName(predicateName)\n  }\n\n  def tripHqTweetTriggeredCRTBasedNtabCaretClickFnFatiguePredicate[\n    Cand <: Candidate with RecommendationType with TargetInfo[\n      Target\n    ]\n  ](\n    filterHistory: TimeSeries => TimeSeries =\n      FatiguePredicate.recTypesOnlyFilter(RecTypes.sharedNTabCaretFatigueTypes),\n    filterCaretFeedbackHistory: Target => Seq[\n      CaretFeedbackDetails\n    ] => Seq[CaretFeedbackDetails] = CaretFeedbackHistoryFilter\n      .caretFeedbackHistoryFilter(Seq(MagicRecsCategory)),\n    filterInlineFeedbackHistory: Seq[FeedbackModel] => Seq[FeedbackModel] =\n      NtabCaretClickFatigueUtils.feedbackModelFilterByCRT(RecTypes.sharedNTabCaretFatigueTypes)\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[Cand] = {\n    val predicateName = \"trip_hq_tweet_triggered_crt_based_ntab_dislike_fatigue_fn\"\n    Predicate\n      .fromAsync[Cand] { cand: Cand =>\n        {\n          val scopedStats = stats.scope(predicateName)\n          val totalRequests = scopedStats.counter(\"mr_ntab_dislike_total\")\n          val total90Day =\n            scopedStats.counter(\"mr_ntab_dislike_90day_dislike\")\n          val totalDisabled =\n            scopedStats.counter(\"mr_ntab_dislike_not_90day_dislike\")\n          val totalSuccess = scopedStats.counter(\"mr_ntab_dislike_success\")\n          val totalFiltered = scopedStats.counter(\"mr_ntab_dislike_filtered\")\n          val totalWithHistory =\n            scopedStats.counter(\"mr_ntab_dislike_with_history\")\n          val totalWithoutHistory =\n            scopedStats.counter(\"mr_ntab_dislike_without_history\")\n          val totalFeedbackSuccess = scopedStats.counter(\"mr_total_feedback_success\")\n          totalRequests.incr()\n\n          Future\n            .join(\n              cand.target.history,\n              cand.target.caretFeedbacks,\n              cand.target.dynamicPushcap,\n              cand.target.optoutAdjustedPushcap,\n              cand.target.notificationFeedbacks,\n              PushCapUtil.getDefaultPushCap(cand.target),\n              getUserStateWeight(cand.target),\n            ).map {\n              case (\n                    history,\n                    Some(feedbackDetails),\n                    dynamicPushcapOpt,\n                    optoutAdjustedPushcapOpt,\n                    Some(feedbacks),\n                    defaultPushCap,\n                    userStateWeight) =>\n                totalWithHistory.incr()\n                if (cand.target.params(\n                    PushFeatureSwitchParams.AdjustTripHqTweetTriggeredNtabCaretClickFatigue)) {\n\n                  val refreshableTypeFilter = CaretFeedbackHistoryFilter\n                    .caretFeedbackHistoryFilterByRefreshableType(HighQualityRefreshableTypes)\n                  val filteredfeedbackDetails = refreshableTypeFilter(cand.target)(feedbackDetails)\n\n                  val feedbackDetailsDeduped =\n                    NtabCaretClickFatiguePredicateHelper.dedupFeedbackDetails(\n                      filterCaretFeedbackHistory(cand.target)(filteredfeedbackDetails),\n                      stats\n                    )\n\n                  val pushCap: Int = (dynamicPushcapOpt, optoutAdjustedPushcapOpt) match {\n                    case (_, Some(optoutAdjustedPushcap)) => optoutAdjustedPushcap\n                    case (Some(pushcapInfo), _) => pushcapInfo.pushcap\n                    case _ => defaultPushCap\n                  }\n                  val filteredHistory = History(filterHistory(history.history.toSeq).toMap)\n\n                  val isOutsideInlineDislikeFatigue =\n                    if (cand.target\n                        .params(\n                          PushFeatureSwitchParams.EnableContFnNonF1TriggerInlineFeedbackFatigue)) {\n                      val weight = {\n                        if (RecTypes.HighQualityTweetTypes.contains(cand.commonRecType)) {\n                          cand.target\n                            .params(\n                              PushFeatureSwitchParams.InlineFeedbackNonF1TriggerNonF1PushCapWeight)\n                        } else {\n                          cand.target\n                            .params(\n                              PushFeatureSwitchParams.InlineFeedbackNonF1TriggerF1PushCapWeight)\n                        }\n                      }\n\n                      val inlineFeedbackFatigueParam = ContinuousFunctionParam(\n                        cand.target\n                          .params(PushFeatureSwitchParams.InlineFeedbackListOfDayKnobs),\n                        cand.target\n                          .params(PushFeatureSwitchParams.InlineFeedbackListOfPushCapWeightKnobs)\n                          .map(_ * pushCap),\n                        cand.target\n                          .params(PushFeatureSwitchParams.InlineFeedbackListOfPowerKnobs),\n                        weight,\n                        pushCap\n                      )\n\n                      val includedCRTs: Set[CommonRecommendationType] =\n                        RecTypes.HighQualityTweetTypes\n\n                      isInlineDislikeOutsideFatiguePeriod(\n                        cand,\n                        feedbacks\n                          .collect {\n                            case feedbackPromptValue: FeedbackPromptValue =>\n                              InlineFeedbackModel(feedbackPromptValue, None)\n                          },\n                        filteredHistory,\n                        Seq(\n                          filterInlineFeedbackHistory,\n                          NtabCaretClickFatigueUtils.feedbackModelFilterByCRT(includedCRTs)),\n                        inlineFeedbackFatigueParam,\n                        scopedStats\n                      )\n                    } else true\n\n                  lazy val isOutsidePromptDislikeFatigue =\n                    if (cand.target\n                        .params(\n                          PushFeatureSwitchParams.EnableContFnNonF1TriggerPromptFeedbackFatigue)) {\n                      val weight =\n                        if (RecTypes.isF1Type(cand.commonRecType))\n                          cand.target\n                            .params(\n                              PushFeatureSwitchParams.PromptFeedbackNonF1TriggerF1PushCapWeight)\n                        else\n                          cand.target\n                            .params(\n                              PushFeatureSwitchParams.PromptFeedbackNonF1TriggerNonF1PushCapWeight)\n\n                      val promptFeedbackFatigueParam = ContinuousFunctionParam(\n                        cand.target\n                          .params(PushFeatureSwitchParams.PromptFeedbackListOfDayKnobs),\n                        cand.target\n                          .params(PushFeatureSwitchParams.PromptFeedbackListOfPushCapWeightKnobs)\n                          .map(_ * pushCap),\n                        cand.target\n                          .params(PushFeatureSwitchParams.PromptFeedbackListOfPowerKnobs),\n                        weight,\n                        pushCap\n                      )\n\n                      isPromptDislikeOutsideFatiguePeriod(\n                        feedbacks\n                          .collect {\n                            case feedbackPromptValue: FeedbackPromptValue =>\n                              PromptFeedbackModel(feedbackPromptValue, None)\n                          },\n                        filteredHistory,\n                        Seq(\n                          filterInlineFeedbackHistory,\n                          NtabCaretClickFatigueUtils.feedbackModelExcludeCRT(\n                            RecTypes.f1FirstDegreeTypes)),\n                        promptFeedbackFatigueParam,\n                        scopedStats\n                      )\n                    } else true\n\n                  isOutsideInlineDislikeFatigue && isOutsidePromptDislikeFatigue\n                } else {\n                  true\n                }\n              case _ =>\n                totalFeedbackSuccess.incr()\n                totalWithoutHistory.incr()\n                true\n            }\n        }\n      }.withStats(stats.scope(predicateName))\n      .withName(predicateName)\n  }\n\n  private def getDedupedInlineFeedbackByType(\n    inlineFeedbacks: Seq[FeedbackModel],\n    feedbackType: FeedbackTypeEnum.Value,\n    revertedFeedbackType: FeedbackTypeEnum.Value\n  ): Seq[FeedbackModel] = {\n    inlineFeedbacks\n      .filter(feedback =>\n        feedback.feedbackTypeEnum == feedbackType ||\n          feedback.feedbackTypeEnum == revertedFeedbackType)\n      .groupBy(feedback => feedback.notificationImpressionId.getOrElse(\"\"))\n      .toSeq\n      .collect {\n        case (impressionId, feedbacks: Seq[FeedbackModel]) if (feedbacks.nonEmpty) =>\n          val latestFeedback = feedbacks.maxBy(feedback => feedback.timestampMs)\n          if (latestFeedback.feedbackTypeEnum == feedbackType)\n            Some(latestFeedback)\n          else None\n        case _ => None\n      }\n      .flatten\n  }\n\n  private def getDedupedInlineFeedback(\n    inlineFeedbacks: Seq[FeedbackModel],\n    target: Target\n  ): Seq[FeedbackModel] = {\n    val inlineDislikeFeedback =\n      if (target.params(PushFeatureSwitchParams.UseInlineDislikeForFatigue)) {\n        getDedupedInlineFeedbackByType(\n          inlineFeedbacks,\n          FeedbackTypeEnum.InlineDislike,\n          FeedbackTypeEnum.InlineRevertedDislike)\n      } else Seq()\n    val inlineDismissFeedback =\n      if (target.params(PushFeatureSwitchParams.UseInlineDismissForFatigue)) {\n        getDedupedInlineFeedbackByType(\n          inlineFeedbacks,\n          FeedbackTypeEnum.InlineDismiss,\n          FeedbackTypeEnum.InlineRevertedDismiss)\n      } else Seq()\n    val inlineSeeLessFeedback =\n      if (target.params(PushFeatureSwitchParams.UseInlineSeeLessForFatigue)) {\n        getDedupedInlineFeedbackByType(\n          inlineFeedbacks,\n          FeedbackTypeEnum.InlineSeeLess,\n          FeedbackTypeEnum.InlineRevertedSeeLess)\n      } else Seq()\n    val inlineNotRelevantFeedback =\n      if (target.params(PushFeatureSwitchParams.UseInlineNotRelevantForFatigue)) {\n        getDedupedInlineFeedbackByType(\n          inlineFeedbacks,\n          FeedbackTypeEnum.InlineNotRelevant,\n          FeedbackTypeEnum.InlineRevertedNotRelevant)\n      } else Seq()\n\n    inlineDislikeFeedback ++ inlineDismissFeedback ++ inlineSeeLessFeedback ++ inlineNotRelevantFeedback\n  }\n\n  private def isInlineDislikeOutsideFatiguePeriod(\n    candidate: Candidate\n      with RecommendationType\n      with TargetInfo[\n        Target\n      ],\n    inlineFeedbacks: Seq[FeedbackModel],\n    filteredHistory: History,\n    feedbackFilters: Seq[Seq[FeedbackModel] => Seq[FeedbackModel]],\n    inlineFeedbackFatigueParam: ContinuousFunctionParam,\n    stats: StatsReceiver\n  ): Boolean = {\n    val scopedStats = stats.scope(\"inline_dislike_fatigue\")\n\n    val inlineNegativeFeedback =\n      getDedupedInlineFeedback(inlineFeedbacks, candidate.target)\n\n    val hydratedInlineNegativeFeedback = FeedbackModelHydrator.HydrateNotification(\n      inlineNegativeFeedback,\n      filteredHistory.history.toSeq.map(_._2))\n\n    if (isOutsideFatiguePeriod(\n        filteredHistory,\n        Seq(),\n        feedbackFilters.foldLeft(hydratedInlineNegativeFeedback)((feedbacks, feedbackFilter) =>\n          feedbackFilter(feedbacks)),\n        inlineFeedbackFatigueParam,\n        scopedStats\n      )) {\n      scopedStats.counter(\"feedback_inline_dislike_success\").incr()\n      true\n    } else {\n      scopedStats.counter(\"feedback_inline_dislike_filtered\").incr()\n      false\n    }\n  }\n\n  private def isPromptDislikeOutsideFatiguePeriod(\n    feedbacks: Seq[FeedbackModel],\n    filteredHistory: History,\n    feedbackFilters: Seq[Seq[FeedbackModel] => Seq[FeedbackModel]],\n    inlineFeedbackFatigueParam: ContinuousFunctionParam,\n    stats: StatsReceiver\n  ): Boolean = {\n    val scopedStats = stats.scope(\"prompt_dislike_fatigue\")\n\n    val promptDislikeFeedback = feedbacks\n      .filter(feedback => feedback.feedbackTypeEnum == FeedbackTypeEnum.PromptIrrelevant)\n    val hydratedPromptDislikeFeedback = FeedbackModelHydrator.HydrateNotification(\n      promptDislikeFeedback,\n      filteredHistory.history.toSeq.map(_._2))\n\n    if (isOutsideFatiguePeriod(\n        filteredHistory,\n        Seq(),\n        feedbackFilters.foldLeft(hydratedPromptDislikeFeedback)((feedbacks, feedbackFilter) =>\n          feedbackFilter(feedbacks)),\n        inlineFeedbackFatigueParam,\n        scopedStats\n      )) {\n      scopedStats.counter(\"feedback_prompt_dislike_success\").incr()\n      true\n    } else {\n      scopedStats.counter(\"feedback_prompt_dislike_filtered\").incr()\n      false\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/ntab_caret_fatigue/ContinuousFunction.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue\n\nimport com.twitter.finagle.stats.StatsReceiver\n\ncase class ContinuousFunctionParam(\n  knobs: Seq[Double],\n  knobValues: Seq[Double],\n  powers: Seq[Double],\n  weight: Double,\n  defaultValue: Double) {\n\n  def validateParams(): Boolean = {\n    knobs.size > 0 && knobs.size - 1 == powers.size && knobs.size == knobValues.size\n  }\n}\n\nobject ContinuousFunction {\n\n  /**\n   * Evalutate the value for function f(x) = w(x - b)^power\n   * where w and b are decided by the start, startVal, end, endVal\n   * such that\n   *         w(start - b) ^ power = startVal\n   *         w(end - b) ^ power = endVal\n   *\n   * @param value the value at which we will evaluate the param\n   * @return weight * f(value)\n   */\n  def evaluateFn(\n    value: Double,\n    start: Double,\n    startVal: Double,\n    end: Double,\n    endVal: Double,\n    power: Double,\n    weight: Double\n  ): Double = {\n    val b =\n      (math.pow(startVal / endVal, 1 / power) * end - start) / (math.pow(\n        startVal / endVal,\n        1 / power) - 1)\n    val w = startVal / math.pow(start - b, power)\n    weight * w * math.pow(value - b, power)\n  }\n\n  /**\n   * Evaluate value for function f(x), and return weight * f(x)\n   *\n   * f(x) is a piecewise function\n   * f(x) = w_i * (x - b_i)^powers[i] for knobs[i] <= x < knobs[i+1]\n   * such that\n   *         w(knobs[i] - b) ^ power = knobVals[i]\n   *         w(knobs[i+1] - b) ^ power = knobVals[i+1]\n   *\n   * @return Evaluate value for weight * f(x), for the function described above. If the any of the input is invalid, returns defaultVal\n   */\n  def safeEvaluateFn(\n    value: Double,\n    knobs: Seq[Double],\n    knobVals: Seq[Double],\n    powers: Seq[Double],\n    weight: Double,\n    defaultVal: Double,\n    statsReceiver: StatsReceiver\n  ): Double = {\n    val totalStats = statsReceiver.counter(\"safe_evalfn_total\")\n    val validStats =\n      statsReceiver.counter(\"safe_evalfn_valid\")\n    val validEndCaseStats =\n      statsReceiver.counter(\"safe_evalfn_valid_endcase\")\n    val invalidStats = statsReceiver.counter(\"safe_evalfn_invalid\")\n\n    totalStats.incr()\n    if (knobs.size <= 0 || knobs.size - 1 != powers.size || knobs.size != knobVals.size) {\n      invalidStats.incr()\n      defaultVal\n    } else {\n      val endIndex = knobs.indexWhere(knob => knob > value)\n      validStats.incr()\n      endIndex match {\n        case -1 => {\n          validEndCaseStats.incr()\n          knobVals(knobVals.size - 1) * weight\n        }\n        case 0 => {\n          validEndCaseStats.incr()\n          knobVals(0) * weight\n        }\n        case _ => {\n          val startIndex = endIndex - 1\n          evaluateFn(\n            value,\n            knobs(startIndex),\n            knobVals(startIndex),\n            knobs(endIndex),\n            knobVals(endIndex),\n            powers(startIndex),\n            weight)\n        }\n      }\n    }\n  }\n\n  def safeEvaluateFn(\n    value: Double,\n    fnParams: ContinuousFunctionParam,\n    statsReceiver: StatsReceiver\n  ): Double = {\n    val totalStats = statsReceiver.counter(\"safe_evalfn_total\")\n    val validStats =\n      statsReceiver.counter(\"safe_evalfn_valid\")\n    val validEndCaseStats =\n      statsReceiver.counter(\"safe_evalfn_valid_endcase\")\n    val invalidStats = statsReceiver.counter(\"safe_evalfn_invalid\")\n\n    totalStats.incr()\n\n    if (fnParams.validateParams()) {\n      val endIndex = fnParams.knobs.indexWhere(knob => knob > value)\n      validStats.incr()\n      endIndex match {\n        case -1 => {\n          validEndCaseStats.incr()\n          fnParams.knobValues(fnParams.knobValues.size - 1) * fnParams.weight\n        }\n        case 0 => {\n          validEndCaseStats.incr()\n          fnParams.knobValues(0) * fnParams.weight\n        }\n        case _ => {\n          val startIndex = endIndex - 1\n          evaluateFn(\n            value,\n            fnParams.knobs(startIndex),\n            fnParams.knobValues(startIndex),\n            fnParams.knobs(endIndex),\n            fnParams.knobValues(endIndex),\n            fnParams.powers(startIndex),\n            fnParams.weight\n          )\n        }\n      }\n    } else {\n      invalidStats.incr()\n      fnParams.defaultValue\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/ntab_caret_fatigue/FeedbackModel.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue\n\nimport com.twitter.notificationservice.thriftscala.GenericType\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.notificationservice.genericfeedbackstore.FeedbackPromptValue\nimport com.twitter.notificationservice.thriftscala.CaretFeedbackDetails\nimport com.twitter.notificationservice.feedback.thriftscala.FeedbackMetadata\nimport com.twitter.notificationservice.feedback.thriftscala.InlineFeedback\nimport com.twitter.notificationservice.feedback.thriftscala.FeedbackValue\nimport com.twitter.notificationservice.feedback.thriftscala.YesOrNoAnswer\n\nobject FeedbackTypeEnum extends Enumeration {\n  val Unknown = Value\n  val CaretDislike = Value\n  val InlineDislike = Value\n  val InlineLike = Value\n  val InlineRevertedLike = Value\n  val InlineRevertedDislike = Value\n  val PromptRelevant = Value\n  val PromptIrrelevant = Value\n  val InlineDismiss = Value\n  val InlineRevertedDismiss = Value\n  val InlineSeeLess = Value\n  val InlineRevertedSeeLess = Value\n  val InlineNotRelevant = Value\n  val InlineRevertedNotRelevant = Value\n\n  def safeFindByName(name: String): Value =\n    values.find(_.toString.toLowerCase() == name.toLowerCase()).getOrElse(Unknown)\n}\n\ntrait FeedbackModel {\n\n  def timestampMs: Long\n\n  def feedbackTypeEnum: FeedbackTypeEnum.Value\n\n  def notificationImpressionId: Option[String]\n\n  def notification: Option[FrigateNotification] = None\n}\n\ncase class CaretFeedbackModel(\n  caretFeedbackDetails: CaretFeedbackDetails,\n  notificationOpt: Option[FrigateNotification] = None)\n    extends FeedbackModel {\n\n  override def timestampMs: Long = caretFeedbackDetails.eventTimestamp\n\n  override def feedbackTypeEnum: FeedbackTypeEnum.Value = FeedbackTypeEnum.CaretDislike\n\n  override def notificationImpressionId: Option[String] = caretFeedbackDetails.impressionId\n\n  override def notification: Option[FrigateNotification] = notificationOpt\n\n  def notificationGenericType: Option[GenericType] = {\n    caretFeedbackDetails.genericNotificationMetadata match {\n      case Some(genericNotificationMetadata) =>\n        Some(genericNotificationMetadata.genericType)\n      case None => None\n    }\n  }\n}\n\ncase class InlineFeedbackModel(\n  feedback: FeedbackPromptValue,\n  notificationOpt: Option[FrigateNotification] = None)\n    extends FeedbackModel {\n\n  override def timestampMs: Long = feedback.createdAt.inMilliseconds\n\n  override def feedbackTypeEnum: FeedbackTypeEnum.Value = {\n    feedback.feedbackValue match {\n      case FeedbackValue(\n            _,\n            _,\n            _,\n            Some(FeedbackMetadata.InlineFeedback(InlineFeedback(Some(answer))))) =>\n        FeedbackTypeEnum.safeFindByName(\"inline\" + answer)\n      case _ => FeedbackTypeEnum.Unknown\n    }\n  }\n\n  override def notificationImpressionId: Option[String] = Some(feedback.feedbackValue.impressionId)\n\n  override def notification: Option[FrigateNotification] = notificationOpt\n}\n\ncase class PromptFeedbackModel(\n  feedback: FeedbackPromptValue,\n  notificationOpt: Option[FrigateNotification] = None)\n    extends FeedbackModel {\n\n  override def timestampMs: Long = feedback.createdAt.inMilliseconds\n\n  override def feedbackTypeEnum: FeedbackTypeEnum.Value = {\n    feedback.feedbackValue match {\n      case FeedbackValue(_, _, _, Some(FeedbackMetadata.YesOrNoAnswer(answer))) =>\n        answer match {\n          case YesOrNoAnswer.Yes => FeedbackTypeEnum.PromptRelevant\n          case YesOrNoAnswer.No => FeedbackTypeEnum.PromptIrrelevant\n          case _ => FeedbackTypeEnum.Unknown\n        }\n      case _ => FeedbackTypeEnum.Unknown\n    }\n  }\n\n  override def notificationImpressionId: Option[String] = Some(feedback.feedbackValue.impressionId)\n\n  override def notification: Option[FrigateNotification] = notificationOpt\n}\n\nobject FeedbackModelHydrator {\n\n  def HydrateNotification(\n    feedbacks: Seq[FeedbackModel],\n    history: Seq[FrigateNotification]\n  ): Seq[FeedbackModel] = {\n    feedbacks.map {\n      case feedback @ (inlineFeedback: InlineFeedbackModel) =>\n        inlineFeedback.copy(notificationOpt = history.find(\n          _.impressionId\n            .equals(feedback.notificationImpressionId)))\n      case feedback @ (caretFeedback: CaretFeedbackModel) =>\n        caretFeedback.copy(notificationOpt = history.find(\n          _.impressionId\n            .equals(feedback.notificationImpressionId)))\n      case feedback @ (promptFeedback: PromptFeedbackModel) =>\n        promptFeedback.copy(notificationOpt = history.find(\n          _.impressionId\n            .equals(feedback.notificationImpressionId)))\n      case feedback => feedback\n    }\n\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/ntab_caret_fatigue/MagicFanoutNtabCaretFatiguePredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.predicate.ntab_caret_fatigue.NtabCaretClickFatiguePredicateHelper\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.hermit.predicate.NamedPredicate\n\nobject MagicFanoutNtabCaretFatiguePredicate {\n  val name = \"MagicFanoutNtabCaretFatiguePredicateForCandidate\"\n\n  private val MomentsCategory = \"Moments\"\n  private val MomentsViaMagicRecsCategory = \"MomentsViaMagicRecs\"\n\n  def apply()(implicit globalStats: StatsReceiver): NamedPredicate[PushCandidate] = {\n    val scopedStats = globalStats.scope(name)\n    val genericTypeCategories = Seq(MomentsCategory, MomentsViaMagicRecsCategory)\n    val crts = RecTypes.magicFanoutEventTypes\n    RecTypeNtabCaretClickFatiguePredicate\n      .apply(\n        genericTypeCategories,\n        crts,\n        NtabCaretClickFatiguePredicateHelper.calculateFatiguePeriodMagicRecs,\n        useMostRecentDislikeTime = true,\n        name = name\n      ).withStats(scopedStats).withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/ntab_caret_fatigue/NtabCaretClickCandidateFatiguePredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.predicate.FatiguePredicate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.frigate.common.base.Candidate\nimport com.twitter.frigate.common.base.TargetInfo\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.common.base.{RecommendationType => BaseRecommendationType}\nimport com.twitter.frigate.common.predicate.CandidateWithRecommendationTypeAndTargetInfoWithCaretFeedbackHistory\nimport com.twitter.frigate.common.predicate.FrigateHistoryFatiguePredicate.TimeSeries\nimport com.twitter.notificationservice.thriftscala.CaretFeedbackDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.predicate.CaretFeedbackHistoryFilter\n\nobject NtabCaretClickContFnFatiguePredicate {\n\n  private val MagicRecsCategory = \"MagicRecs\"\n\n  def ntabCaretClickContFnFatiguePredicates(\n    filterHistory: TimeSeries => TimeSeries =\n      FatiguePredicate.recTypesOnlyFilter(RecTypes.sharedNTabCaretFatigueTypes),\n    filterCaretFeedbackHistory: Target => Seq[\n      CaretFeedbackDetails\n    ] => Seq[CaretFeedbackDetails] =\n      CaretFeedbackHistoryFilter.caretFeedbackHistoryFilter(Seq(MagicRecsCategory)),\n    filterInlineFeedbackHistory: Seq[FeedbackModel] => Seq[FeedbackModel] =\n      NtabCaretClickFatigueUtils.feedbackModelFilterByCRT(RecTypes.sharedNTabCaretFatigueTypes),\n    name: String = \"NTabCaretClickFnCandidatePredicates\"\n  )(\n    implicit globalStats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val scopedStats = globalStats.scope(name)\n    CRTBasedNtabCaretClickFatiguePredicates\n      .f1TriggeredCRTBasedNtabCaretClickFnFatiguePredicate[\n        Candidate with BaseRecommendationType with TargetInfo[\n          Target\n        ]\n      ](\n        filterHistory = filterHistory,\n        filterCaretFeedbackHistory = filterCaretFeedbackHistory,\n        filterInlineFeedbackHistory = filterInlineFeedbackHistory\n      )\n      .applyOnlyToCandidateWithRecommendationTypeAndTargetWithCaretFeedbackHistory\n      .withName(\"f1_triggered_fn_seelessoften_fatigue\")\n      .andThen(\n        CRTBasedNtabCaretClickFatiguePredicates\n          .nonF1TriggeredCRTBasedNtabCaretClickFnFatiguePredicate[\n            Candidate with BaseRecommendationType with TargetInfo[\n              Target\n            ]\n          ](\n            filterHistory = filterHistory,\n            filterCaretFeedbackHistory = filterCaretFeedbackHistory,\n            filterInlineFeedbackHistory = filterInlineFeedbackHistory\n          )\n          .applyOnlyToCandidateWithRecommendationTypeAndTargetWithCaretFeedbackHistory)\n      .withName(\"nonf1_triggered_fn_seelessoften_fatigue\")\n      .andThen(\n        CRTBasedNtabCaretClickFatiguePredicates\n          .tripHqTweetTriggeredCRTBasedNtabCaretClickFnFatiguePredicate[\n            Candidate with BaseRecommendationType with TargetInfo[\n              Target\n            ]\n          ](\n            filterHistory = filterHistory,\n            filterCaretFeedbackHistory = filterCaretFeedbackHistory,\n            filterInlineFeedbackHistory = filterInlineFeedbackHistory\n          )\n          .applyOnlyToCandidateWithRecommendationTypeAndTargetWithCaretFeedbackHistory)\n      .withName(\"trip_hq_tweet_triggered_fn_seelessoften_fatigue\")\n      .andThen(\n        CRTBasedNtabCaretClickFatiguePredicates\n          .genericCRTBasedNtabCaretClickFnFatiguePredicate[\n            Candidate with BaseRecommendationType with TargetInfo[\n              Target\n            ]\n          ](\n            filterHistory = filterHistory,\n            filterCaretFeedbackHistory = filterCaretFeedbackHistory,\n            filterInlineFeedbackHistory = filterInlineFeedbackHistory)\n          .applyOnlyToCandidateWithRecommendationTypeAndTargetWithCaretFeedbackHistory\n          .withName(\"generic_fn_seelessoften_fatigue\")\n      )\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/ntab_caret_fatigue/NtabCaretClickFatiguePredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.predicate.ntab_caret_fatigue.NtabCaretClickFatiguePredicateHelper\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\nobject NtabCaretClickFatiguePredicate {\n  val name = \"NtabCaretClickFatiguePredicate\"\n\n  def isSpacesTypeAndTeamMember(candidate: PushCandidate): Future[Boolean] = {\n    candidate.target.isTeamMember.map { isTeamMember =>\n      val isSpacesType = RecTypes.isRecommendedSpacesType(candidate.commonRecType)\n      isTeamMember && isSpacesType\n    }\n  }\n\n  def apply()(implicit globalStats: StatsReceiver): NamedPredicate[PushCandidate] = {\n    val scopedStats = globalStats.scope(name)\n    val genericTypeCategories = Seq(\"MagicRecs\")\n    val crts = RecTypes.sharedNTabCaretFatigueTypes\n    val recTypeNtabCaretClickFatiguePredicate =\n      RecTypeNtabCaretClickFatiguePredicate.apply(\n        genericTypeCategories,\n        crts,\n        NtabCaretClickFatiguePredicateHelper.calculateFatiguePeriodMagicRecs,\n        useMostRecentDislikeTime = false\n      )\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        isSpacesTypeAndTeamMember(candidate).flatMap { isSpacesTypeAndTeamMember =>\n          if (RecTypes.sharedNTabCaretFatigueTypes(\n              candidate.commonRecType) && !isSpacesTypeAndTeamMember) {\n            recTypeNtabCaretClickFatiguePredicate\n              .apply(Seq(candidate)).map(_.headOption.getOrElse(false))\n          } else {\n            Future.True\n          }\n        }\n      }\n      .withStats(scopedStats)\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/ntab_caret_fatigue/NtabCaretClickFatigueUtils.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.predicate.ntab_caret_fatigue.NtabCaretClickFatiguePredicateHelper\nimport com.twitter.notificationservice.thriftscala.CaretFeedbackDetails\nimport com.twitter.util.Duration\nimport com.twitter.conversions.DurationOps._\nimport scala.math.min\nimport com.twitter.util.Time\nimport com.twitter.frigate.thriftscala.{CommonRecommendationType => CRT}\n\nobject NtabCaretClickFatigueUtils {\n\n  private def pushCapForFeedback(\n    feedbackDetails: Seq[CaretFeedbackDetails],\n    feedbacks: Seq[FeedbackModel],\n    param: ContinuousFunctionParam,\n    statsReceiver: StatsReceiver\n  ): Double = {\n    val stats = statsReceiver.scope(\"mr_seelessoften_contfn_pushcap\")\n    val pushCapTotal = stats.counter(\"pushcap_total\")\n    val pushCapInvalid =\n      stats.counter(\"pushcap_invalid\")\n\n    pushCapTotal.incr()\n    val timeSinceMostRecentDislikeMs =\n      NtabCaretClickFatiguePredicateHelper.getDurationSinceMostRecentDislike(feedbackDetails)\n    val mostRecentFeedbackTimestamp: Option[Long] =\n      feedbacks\n        .map { feedback =>\n          feedback.timestampMs\n        }.reduceOption(_ max _)\n    val timeSinceMostRecentFeedback: Option[Duration] =\n      mostRecentFeedbackTimestamp.map(Time.now - Time.fromMilliseconds(_))\n\n    val nTabDislikePushCap = timeSinceMostRecentDislikeMs match {\n      case Some(lastDislikeTimeMs) => {\n        ContinuousFunction.safeEvaluateFn(lastDislikeTimeMs.inDays.toDouble, param, stats)\n      }\n      case _ => {\n        pushCapInvalid.incr()\n        param.defaultValue\n      }\n    }\n    val feedbackPushCap = timeSinceMostRecentFeedback match {\n      case Some(lastDislikeTimeVal) => {\n        ContinuousFunction.safeEvaluateFn(lastDislikeTimeVal.inDays.toDouble, param, stats)\n      }\n      case _ => {\n        pushCapInvalid.incr()\n        param.defaultValue\n      }\n    }\n\n    min(nTabDislikePushCap, feedbackPushCap)\n  }\n\n  def durationToFilterForFeedback(\n    feedbackDetails: Seq[CaretFeedbackDetails],\n    feedbacks: Seq[FeedbackModel],\n    param: ContinuousFunctionParam,\n    defaultPushCap: Double,\n    statsReceiver: StatsReceiver\n  ): Duration = {\n    val pushCap = min(\n      pushCapForFeedback(feedbackDetails, feedbacks, param, statsReceiver),\n      defaultPushCap\n    )\n    if (pushCap <= 0) {\n      Duration.Top\n    } else {\n      24.hours / pushCap\n    }\n  }\n\n  def hasUserDislikeInLast90Days(feedbackDetails: Seq[CaretFeedbackDetails]): Boolean = {\n    val timeSinceMostRecentDislike =\n      NtabCaretClickFatiguePredicateHelper.getDurationSinceMostRecentDislike(feedbackDetails)\n\n    timeSinceMostRecentDislike.exists(_ < 90.days)\n  }\n\n  def feedbackModelFilterByCRT(\n    crts: Set[CRT]\n  ): Seq[FeedbackModel] => Seq[\n    FeedbackModel\n  ] = { feedbacks =>\n    feedbacks.filter { feedback =>\n      feedback.notification match {\n        case Some(notification) => crts.contains(notification.commonRecommendationType)\n        case None => false\n      }\n    }\n  }\n\n  def feedbackModelExcludeCRT(\n    crts: Set[CRT]\n  ): Seq[FeedbackModel] => Seq[\n    FeedbackModel\n  ] = { feedbacks =>\n    feedbacks.filter { feedback =>\n      feedback.notification match {\n        case Some(notification) => !crts.contains(notification.commonRecommendationType)\n        case None => true\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/ntab_caret_fatigue/RecTypeNtabCaretFatiguePredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.predicate.FatiguePredicate\nimport com.twitter.frigate.pushservice.predicate.CaretFeedbackHistoryFilter\nimport com.twitter.frigate.pushservice.predicate.{\n  TargetNtabCaretClickFatiguePredicate => CommonNtabCaretClickFatiguePredicate\n}\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.thriftscala.NotificationDisplayLocation\nimport com.twitter.frigate.thriftscala.{CommonRecommendationType => CRT}\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.notificationservice.thriftscala.CaretFeedbackDetails\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\n\nobject RecTypeNtabCaretClickFatiguePredicate {\n  val defaultName = \"RecTypeNtabCaretClickFatiguePredicateForCandidate\"\n\n  private def candidateFatiguePredicate(\n    genericTypeCategories: Seq[String],\n    crts: Set[CRT]\n  )(\n    implicit stats: StatsReceiver\n  ): NamedPredicate[\n    PushCandidate\n  ] = {\n    val name = \"f1TriggeredCRTBasedFatiguePredciate\"\n    val scopedStats = stats.scope(s\"predicate_$name\")\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        if (candidate.frigateNotification.notificationDisplayLocation == NotificationDisplayLocation.PushToMobileDevice) {\n          if (candidate.target.params(PushParams.EnableFatigueNtabCaretClickingParam)) {\n            NtabCaretClickContFnFatiguePredicate\n              .ntabCaretClickContFnFatiguePredicates(\n                filterHistory = FatiguePredicate.recTypesOnlyFilter(crts),\n                filterCaretFeedbackHistory =\n                  CaretFeedbackHistoryFilter.caretFeedbackHistoryFilter(genericTypeCategories),\n                filterInlineFeedbackHistory =\n                  NtabCaretClickFatigueUtils.feedbackModelFilterByCRT(crts)\n              ).apply(Seq(candidate))\n              .map(_.headOption.getOrElse(false))\n          } else Future.True\n        } else {\n          Future.True\n        }\n      }.withStats(scopedStats)\n      .withName(name)\n  }\n\n  def apply(\n    genericTypeCategories: Seq[String],\n    crts: Set[CRT],\n    calculateFatiguePeriod: Seq[CaretFeedbackDetails] => Duration,\n    useMostRecentDislikeTime: Boolean,\n    name: String = defaultName\n  )(\n    implicit globalStats: StatsReceiver\n  ): NamedPredicate[PushCandidate] = {\n    val scopedStats = globalStats.scope(name)\n    val commonNtabCaretClickFatiguePredicate = CommonNtabCaretClickFatiguePredicate(\n      filterCaretFeedbackHistory =\n        CaretFeedbackHistoryFilter.caretFeedbackHistoryFilter(genericTypeCategories),\n      filterHistory = FatiguePredicate.recTypesOnlyFilter(crts),\n      calculateFatiguePeriod = calculateFatiguePeriod,\n      useMostRecentDislikeTime = useMostRecentDislikeTime,\n      name = name\n    )(globalStats)\n\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        if (candidate.frigateNotification.notificationDisplayLocation == NotificationDisplayLocation.PushToMobileDevice) {\n          if (candidate.target.params(PushParams.EnableFatigueNtabCaretClickingParam)) {\n            commonNtabCaretClickFatiguePredicate\n              .apply(Seq(candidate.target))\n              .map(_.headOption.getOrElse(false))\n          } else Future.True\n        } else {\n          Future.True\n        }\n      }.andThen(candidateFatiguePredicate(genericTypeCategories, crts))\n      .withStats(scopedStats)\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/package.scala",
    "content": "package com.twitter.frigate.pushservice\n\nimport com.twitter.frigate.common.base.Candidate\nimport com.twitter.frigate.common.base.SocialGraphServiceRelationshipMap\nimport com.twitter.frigate.common.base.TweetAuthor\nimport com.twitter.frigate.common.rec_types.RecTypes.isInNetworkTweetType\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.hermit.predicate.Predicate\n\npackage object predicate {\n  implicit class CandidatesWithAuthorFollowPredicates(\n    predicate: Predicate[\n      PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap\n    ]) {\n    def applyOnlyToAuthorBeingFollowPredicates: Predicate[Candidate] =\n      predicate.optionalOn[Candidate](\n        {\n          case candidate: PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap\n              if isInNetworkTweetType(candidate.commonRecType) =>\n            Some(candidate)\n          case _ =>\n            None\n        },\n        missingResult = true\n      )\n  }\n\n  implicit class TweetCandidateWithTweetAuthor(\n    predicate: Predicate[\n      PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap\n    ]) {\n    def applyOnlyToBasicTweetPredicates: Predicate[Candidate] =\n      predicate.optionalOn[Candidate](\n        {\n          case candidate: PushCandidate with TweetAuthor with SocialGraphServiceRelationshipMap\n              if isInNetworkTweetType(candidate.commonRecType) =>\n            Some(candidate)\n          case _ =>\n            None\n        },\n        missingResult = true\n      )\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/quality_model_predicate/OpenOrNtabClickQualityPredicate.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.quality_model_predicate\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.util.Future\n\nobject ExplicitOONCFilterPredicate extends QualityPredicateBase {\n  override lazy val name = \"open_or_ntab_click_explicit_threshold\"\n\n  override lazy val thresholdExtractor = (t: Target) =>\n    Future.value(t.params(PushFeatureSwitchParams.QualityPredicateExplicitThresholdParam))\n\n  override def scoreExtractor = (candidate: PushCandidate) =>\n    candidate.mrWeightedOpenOrNtabClickRankingProbability\n}\n\nobject WeightedOpenOrNtabClickQualityPredicate extends QualityPredicateBase {\n  override lazy val name = \"weighted_open_or_ntab_click_model\"\n\n  override lazy val thresholdExtractor = (t: Target) => {\n    Future.value(0.0)\n  }\n\n  override def scoreExtractor =\n    (candidate: PushCandidate) => candidate.mrWeightedOpenOrNtabClickFilteringProbability\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/quality_model_predicate/QualityPredicateCommon.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.quality_model_predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.model.PushTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.target.TargetScoringDetails\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\nobject PDauCohort extends Enumeration {\n  type PDauCohort = Value\n\n  val cohort1 = Value\n  val cohort2 = Value\n  val cohort3 = Value\n  val cohort4 = Value\n  val cohort5 = Value\n  val cohort6 = Value\n}\n\nobject PDauCohortUtil {\n\n  case class DauThreshold(\n    threshold1: Double,\n    threshold2: Double,\n    threshold3: Double,\n    threshold4: Double,\n    threshold5: Double)\n\n  val defaultDAUProb = 0.0\n\n  val dauProbThresholds = DauThreshold(\n    threshold1 = 0.05,\n    threshold2 = 0.14,\n    threshold3 = 0.33,\n    threshold4 = 0.7,\n    threshold5 = 0.959\n  )\n\n  val finerThresholdMap =\n    Map(\n      PDauCohort.cohort2 -> List(0.05, 0.0539, 0.0563, 0.0600, 0.0681, 0.0733, 0.0800, 0.0849,\n        0.0912, 0.0975, 0.1032, 0.1092, 0.1134, 0.1191, 0.1252, 0.1324, 0.14),\n      PDauCohort.cohort3 -> List(0.14, 0.1489, 0.1544, 0.1625, 0.1704, 0.1797, 0.1905, 0.2001,\n        0.2120, 0.2248, 0.2363, 0.2500, 0.2650, 0.2801, 0.2958, 0.3119, 0.33),\n      PDauCohort.cohort4 -> List(0.33, 0.3484, 0.3686, 0.3893, 0.4126, 0.4350, 0.4603, 0.4856,\n        0.5092, 0.5348, 0.5602, 0.5850, 0.6087, 0.6319, 0.6548, 0.6779, 0.7),\n      PDauCohort.cohort5 -> List(0.7, 0.7295, 0.7581, 0.7831, 0.8049, 0.8251, 0.8444, 0.8612,\n        0.8786, 0.8936, 0.9043, 0.9175, 0.9290, 0.9383, 0.9498, 0.9587, 0.959)\n    )\n\n  def getBucket(targetUser: PushTypes.Target, doImpression: Boolean) = {\n    implicit val stats = targetUser.stats.scope(\"PDauCohortUtil\")\n    if (doImpression) targetUser.getBucket _ else targetUser.getBucketWithoutImpression _\n  }\n\n  def threshold1(targetUser: PushTypes.Target): Double = dauProbThresholds.threshold1\n\n  def threshold2(targetUser: PushTypes.Target): Double = dauProbThresholds.threshold2\n\n  def threshold3(targetUser: PushTypes.Target): Double = dauProbThresholds.threshold3\n\n  def threshold4(targetUser: PushTypes.Target): Double = dauProbThresholds.threshold4\n\n  def threshold5(targetUser: PushTypes.Target): Double = dauProbThresholds.threshold5\n\n  def thresholdForCohort(targetUser: PushTypes.Target, dauCohort: Int): Double = {\n    if (dauCohort == 0) 0.0\n    else if (dauCohort == 1) threshold1(targetUser)\n    else if (dauCohort == 2) threshold2(targetUser)\n    else if (dauCohort == 3) threshold3(targetUser)\n    else if (dauCohort == 4) threshold4(targetUser)\n    else if (dauCohort == 5) threshold5(targetUser)\n    else 1.0\n  }\n\n  def getPDauCohort(dauProbability: Double, thresholds: DauThreshold): PDauCohort.Value = {\n    dauProbability match {\n      case dauProb if dauProb >= 0.0 && dauProb < thresholds.threshold1 => PDauCohort.cohort1\n      case dauProb if dauProb >= thresholds.threshold1 && dauProb < thresholds.threshold2 =>\n        PDauCohort.cohort2\n      case dauProb if dauProb >= thresholds.threshold2 && dauProb < thresholds.threshold3 =>\n        PDauCohort.cohort3\n      case dauProb if dauProb >= thresholds.threshold3 && dauProb < thresholds.threshold4 =>\n        PDauCohort.cohort4\n      case dauProb if dauProb >= thresholds.threshold4 && dauProb < thresholds.threshold5 =>\n        PDauCohort.cohort5\n      case dauProb if dauProb >= thresholds.threshold5 && dauProb <= 1.0 => PDauCohort.cohort6\n    }\n  }\n\n  def getDauProb(target: TargetScoringDetails): Future[Double] = {\n    target.dauProbability.map { dauProb =>\n      dauProb.map(_.probability).getOrElse(defaultDAUProb)\n    }\n  }\n\n  def getPDauCohort(target: TargetScoringDetails): Future[PDauCohort.Value] = {\n    getDauProb(target).map { getPDauCohort(_, dauProbThresholds) }\n  }\n\n  def getPDauCohortWithPDau(target: TargetScoringDetails): Future[(PDauCohort.Value, Double)] = {\n    getDauProb(target).map { prob =>\n      (getPDauCohort(prob, dauProbThresholds), prob)\n    }\n  }\n\n  def updateStats(\n    target: PushTypes.Target,\n    modelName: String,\n    predicateResult: Boolean\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): Unit = {\n    val dauCohortOp = getPDauCohort(target)\n    dauCohortOp.map { dauCohort =>\n      val cohortStats = statsReceiver.scope(modelName).scope(dauCohort.toString)\n      cohortStats.counter(s\"filter_$predicateResult\").incr()\n    }\n    if (target.isNewSignup) {\n      val newUserModelStats = statsReceiver.scope(modelName)\n      newUserModelStats.counter(s\"new_user_filter_$predicateResult\").incr()\n    }\n  }\n}\n\ntrait QualityPredicateBase {\n  def name: String\n  def thresholdExtractor: Target => Future[Double]\n  def scoreExtractor: PushCandidate => Future[Option[Double]]\n  def isPredicateEnabled: PushCandidate => Future[Boolean] = _ => Future.True\n  def comparator: (Double, Double) => Boolean =\n    (score: Double, threshold: Double) => score >= threshold\n  def updateCustomStats(\n    candidate: PushCandidate,\n    score: Double,\n    threshold: Double,\n    result: Boolean\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): Unit = {}\n\n  def apply()(implicit statsReceiver: StatsReceiver): NamedPredicate[PushCandidate] = {\n    Predicate\n      .fromAsync { candidate: PushCandidate =>\n        isPredicateEnabled(candidate).flatMap {\n          case true =>\n            scoreExtractor(candidate).flatMap { scoreOpt =>\n              thresholdExtractor(candidate.target).map { threshold =>\n                val score = scoreOpt.getOrElse(0.0)\n                val result = comparator(score, threshold)\n                PDauCohortUtil.updateStats(candidate.target, name, result)\n                updateCustomStats(candidate, score, threshold, result)\n                result\n              }\n            }\n          case _ => Future.True\n        }\n      }\n      .withStats(statsReceiver.scope(s\"predicate_$name\"))\n      .withName(name)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/predicate/quality_model_predicate/QualityPredicateMap.scala",
    "content": "package com.twitter.frigate.pushservice.predicate.quality_model_predicate\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.QualityPredicateEnum\nimport com.twitter.frigate.pushservice.predicate.PredicatesForCandidate\nimport com.twitter.hermit.predicate.NamedPredicate\n\nobject QualityPredicateMap {\n\n  def apply(\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): Map[QualityPredicateEnum.Value, NamedPredicate[PushCandidate]] = {\n    Map(\n      QualityPredicateEnum.WeightedOpenOrNtabClick -> WeightedOpenOrNtabClickQualityPredicate(),\n      QualityPredicateEnum.ExplicitOpenOrNtabClickFilter -> ExplicitOONCFilterPredicate(),\n      QualityPredicateEnum.AlwaysTrue -> PredicatesForCandidate.alwaysTruePushCandidatePredicate,\n    )\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/rank/CRTBoostRanker.scala",
    "content": "package com.twitter.frigate.pushservice.rank\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\n\n/**\n *  This Ranker re-ranks MR candidates, boosting input CRTs.\n *  Relative ranking between input CRTs and rest of the candidates doesn't change\n *\n *  Ex: T: Tweet candidate, F: input CRT candidatess\n *\n *  T3, F2, T1, T2, F1 => F2, F1, T3, T1, T2\n */\ncase class CRTBoostRanker(statsReceiver: StatsReceiver) {\n\n  private val recsToBoostStat = statsReceiver.stat(\"recs_to_boost\")\n  private val otherRecsStat = statsReceiver.stat(\"other_recs\")\n\n  private def boostCrtToTop(\n    inputCandidates: Seq[CandidateDetails[PushCandidate]],\n    crtToBoost: CommonRecommendationType\n  ): Seq[CandidateDetails[PushCandidate]] = {\n    val (upRankedCandidates, otherCandidates) =\n      inputCandidates.partition(_.candidate.commonRecType == crtToBoost)\n    recsToBoostStat.add(upRankedCandidates.size)\n    otherRecsStat.add(otherCandidates.size)\n    upRankedCandidates ++ otherCandidates\n  }\n\n  final def boostCrtsToTop(\n    inputCandidates: Seq[CandidateDetails[PushCandidate]],\n    crtsToBoost: Seq[CommonRecommendationType]\n  ): Seq[CandidateDetails[PushCandidate]] = {\n    crtsToBoost.headOption match {\n      case Some(crt) =>\n        val upRankedCandidates = boostCrtToTop(inputCandidates, crt)\n        boostCrtsToTop(upRankedCandidates, crtsToBoost.tail)\n      case None => inputCandidates\n    }\n  }\n\n  final def boostCrtsToTopStableOrder(\n    inputCandidates: Seq[CandidateDetails[PushCandidate]],\n    crtsToBoost: Seq[CommonRecommendationType]\n  ): Seq[CandidateDetails[PushCandidate]] = {\n    val crtsToBoostSet = crtsToBoost.toSet\n    val (upRankedCandidates, otherCandidates) = inputCandidates.partition(candidateDetail =>\n      crtsToBoostSet.contains(candidateDetail.candidate.commonRecType))\n\n    upRankedCandidates ++ otherCandidates\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/rank/CRTDownRanker.scala",
    "content": "package com.twitter.frigate.pushservice.rank\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\n\n/**\n *  This Ranker re-ranks MR candidates, down ranks input CRTs.\n *  Relative ranking between input CRTs and rest of the candidates doesn't change\n *\n *  Ex: T: Tweet candidate, F: input CRT candidates\n *\n *  T3, F2, T1, T2, F1 => T3, T1, T2, F2, F1\n */\ncase class CRTDownRanker(statsReceiver: StatsReceiver) {\n\n  private val recsToDownRankStat = statsReceiver.stat(\"recs_to_down_rank\")\n  private val otherRecsStat = statsReceiver.stat(\"other_recs\")\n  private val downRankerRequests = statsReceiver.counter(\"down_ranker_requests\")\n\n  private def downRank(\n    inputCandidates: Seq[CandidateDetails[PushCandidate]],\n    crtToDownRank: CommonRecommendationType\n  ): Seq[CandidateDetails[PushCandidate]] = {\n    downRankerRequests.incr()\n    val (downRankedCandidates, otherCandidates) =\n      inputCandidates.partition(_.candidate.commonRecType == crtToDownRank)\n    recsToDownRankStat.add(downRankedCandidates.size)\n    otherRecsStat.add(otherCandidates.size)\n    otherCandidates ++ downRankedCandidates\n  }\n\n  final def downRank(\n    inputCandidates: Seq[CandidateDetails[PushCandidate]],\n    crtsToDownRank: Seq[CommonRecommendationType]\n  ): Seq[CandidateDetails[PushCandidate]] = {\n    crtsToDownRank.headOption match {\n      case Some(crt) =>\n        val downRankedCandidates = downRank(inputCandidates, crt)\n        downRank(downRankedCandidates, crtsToDownRank.tail)\n      case None => inputCandidates\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/rank/LoggedOutRanker.scala",
    "content": "package com.twitter.frigate.pushservice.rank\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nclass LoggedOutRanker(tweetyPieStore: ReadableStore[Long, TweetyPieResult], stats: StatsReceiver) {\n  private val statsReceiver = stats.scope(this.getClass.getSimpleName)\n  private val rankedCandidates = statsReceiver.counter(\"ranked_candidates_count\")\n\n  def rank(\n    candidates: Seq[CandidateDetails[PushCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    val tweetIds = candidates.map { cand => cand.candidate.asInstanceOf[TweetCandidate].tweetId }\n    val results = tweetyPieStore.multiGet(tweetIds.toSet).values.toSeq\n    val futureOfResults = Future.traverseSequentially(results)(r => r)\n    val tweetsFut = futureOfResults.map { tweetyPieResults =>\n      tweetyPieResults.map(_.map(_.tweet))\n    }\n    val sortedTweetsFuture = tweetsFut.map { tweets =>\n      tweets\n        .map { tweet =>\n          if (tweet.isDefined && tweet.get.counts.isDefined) {\n            tweet.get.id -> tweet.get.counts.get.favoriteCount.getOrElse(0L)\n          } else {\n            0 -> 0L\n          }\n        }.sortBy(_._2)(Ordering[Long].reverse)\n    }\n    val finalCandidates = sortedTweetsFuture.map { sortedTweets =>\n      sortedTweets\n        .map { tweet =>\n          candidates.find(_.candidate.asInstanceOf[TweetCandidate].tweetId == tweet._1).orNull\n        }.filter { cand => cand != null }\n    }\n    finalCandidates.map { fc =>\n      rankedCandidates.incr(fc.size)\n    }\n    finalCandidates\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/rank/ModelBasedRanker.scala",
    "content": "package com.twitter.frigate.pushservice.rank\n\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.util.Future\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.params.MrQualityUprankingPartialTypeEnum\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.params.PushConstants.OoncQualityCombinedScore\n\nobject ModelBasedRanker {\n\n  def rankBySpecifiedScore(\n    candidatesDetails: Seq[CandidateDetails[PushCandidate]],\n    scoreExtractor: PushCandidate => Future[Option[Double]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n\n    val scoredCandidatesFutures = candidatesDetails.map { cand =>\n      scoreExtractor(cand.candidate).map { scoreOp => (cand, scoreOp.getOrElse(0.0)) }\n    }\n\n    Future.collect(scoredCandidatesFutures).map { scores =>\n      val sorted = scores.sortBy { candidateDetails => -1 * candidateDetails._2 }\n      sorted.map(_._1)\n    }\n  }\n\n  def populatePredictionScoreStats(\n    candidatesDetails: Seq[CandidateDetails[PushCandidate]],\n    scoreExtractor: PushCandidate => Future[Option[Double]],\n    predictionScoreStats: StatsReceiver\n  ): Unit = {\n    val scoreScaleFactorForStat = 10000\n    val statName = \"prediction_scores\"\n    candidatesDetails.map {\n      case CandidateDetails(candidate, source) =>\n        val crt = candidate.commonRecType\n        scoreExtractor(candidate).map { scoreOp =>\n          val scaledScore = (scoreOp.getOrElse(0.0) * scoreScaleFactorForStat).toFloat\n          predictionScoreStats.scope(\"all_candidates\").stat(statName).add(scaledScore)\n          predictionScoreStats.scope(crt.toString()).stat(statName).add(scaledScore)\n        }\n    }\n  }\n\n  def populateMrWeightedOpenOrNtabClickScoreStats(\n    candidatesDetails: Seq[CandidateDetails[PushCandidate]],\n    predictionScoreStats: StatsReceiver\n  ): Unit = {\n    populatePredictionScoreStats(\n      candidatesDetails,\n      candidate => candidate.mrWeightedOpenOrNtabClickRankingProbability,\n      predictionScoreStats\n    )\n  }\n\n  def populateMrQualityUprankingScoreStats(\n    candidatesDetails: Seq[CandidateDetails[PushCandidate]],\n    predictionScoreStats: StatsReceiver\n  ): Unit = {\n    populatePredictionScoreStats(\n      candidatesDetails,\n      candidate => candidate.mrQualityUprankingProbability,\n      predictionScoreStats\n    )\n  }\n\n  def rankByMrWeightedOpenOrNtabClickScore(\n    candidatesDetails: Seq[CandidateDetails[PushCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n\n    rankBySpecifiedScore(\n      candidatesDetails,\n      candidate => candidate.mrWeightedOpenOrNtabClickRankingProbability\n    )\n  }\n\n  def transformSigmoid(\n    score: Double,\n    weight: Double = 1.0,\n    bias: Double = 0.0\n  ): Double = {\n    val base = -1.0 * (weight * score + bias)\n    val cappedBase = math.max(math.min(base, 100.0), -100.0)\n    1.0 / (1.0 + math.exp(cappedBase))\n  }\n\n  def transformLinear(\n    score: Double,\n    bar: Double = 1.0\n  ): Double = {\n    val positiveBar = math.abs(bar)\n    val cappedScore = math.max(math.min(score, positiveBar), -1.0 * positiveBar)\n    cappedScore / positiveBar\n  }\n\n  def transformIdentity(\n    score: Double\n  ): Double = score\n\n  def rankByQualityOoncCombinedScore(\n    candidatesDetails: Seq[CandidateDetails[PushCandidate]],\n    qualityScoreTransform: Double => Double,\n    qualityScoreBoost: Double = 1.0\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n\n    rankBySpecifiedScore(\n      candidatesDetails,\n      candidate => {\n        val ooncScoreFutOpt: Future[Option[Double]] =\n          candidate.mrWeightedOpenOrNtabClickRankingProbability\n        val qualityScoreFutOpt: Future[Option[Double]] =\n          candidate.mrQualityUprankingProbability\n        Future\n          .join(\n            ooncScoreFutOpt,\n            qualityScoreFutOpt\n          ).map {\n            case (Some(ooncScore), Some(qualityScore)) =>\n              val transformedQualityScore = qualityScoreTransform(qualityScore)\n              val combinedScore = ooncScore * (1.0 + qualityScoreBoost * transformedQualityScore)\n              candidate\n                .cacheExternalScore(OoncQualityCombinedScore, Future.value(Some(combinedScore)))\n              Some(combinedScore)\n            case _ => None\n          }\n      }\n    )\n  }\n\n  def rerankByProducerQualityOoncCombinedScore(\n    candidateDetails: Seq[CandidateDetails[PushCandidate]]\n  )(\n    implicit stat: StatsReceiver\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    val scopedStat = stat.scope(\"producer_quality_reranking\")\n    val oonCandidates = candidateDetails.filter {\n      case CandidateDetails(pushCandidate: PushCandidate, _) =>\n        tweetCandidateSelector(pushCandidate, MrQualityUprankingPartialTypeEnum.Oon)\n    }\n\n    val rankedOonCandidatesFut = rankBySpecifiedScore(\n      oonCandidates,\n      candidate => {\n        val baseScoreFutureOpt: Future[Option[Double]] = {\n          val qualityCombinedScoreFutureOpt =\n            candidate.getExternalCachedScoreByName(OoncQualityCombinedScore)\n          val ooncScoreFutureOpt = candidate.mrWeightedOpenOrNtabClickRankingProbability\n          Future.join(qualityCombinedScoreFutureOpt, ooncScoreFutureOpt).map {\n            case (Some(qualityCombinedScore), _) =>\n              scopedStat.counter(\"quality_combined_score\").incr()\n              Some(qualityCombinedScore)\n            case (_, ooncScoreOpt) =>\n              scopedStat.counter(\"oonc_score\").incr()\n              ooncScoreOpt\n          }\n        }\n        baseScoreFutureOpt.map {\n          case Some(baseScore) =>\n            val boostRatio = candidate.mrProducerQualityUprankingBoost.getOrElse(1.0)\n            if (boostRatio > 1.0) scopedStat.counter(\"author_uprank\").incr()\n            else if (boostRatio < 1.0) scopedStat.counter(\"author_downrank\").incr()\n            else scopedStat.counter(\"author_noboost\").incr()\n            Some(baseScore * boostRatio)\n          case _ =>\n            scopedStat.counter(\"empty_score\").incr()\n            None\n        }\n      }\n    )\n\n    rankedOonCandidatesFut.map { rankedOonCandidates =>\n      val sortedOonCandidateIterator = rankedOonCandidates.toIterator\n      candidateDetails.map { ooncRankedCandidate =>\n        val isOon = tweetCandidateSelector(\n          ooncRankedCandidate.candidate,\n          MrQualityUprankingPartialTypeEnum.Oon)\n\n        if (sortedOonCandidateIterator.hasNext && isOon)\n          sortedOonCandidateIterator.next()\n        else ooncRankedCandidate\n      }\n    }\n  }\n\n  def tweetCandidateSelector(\n    pushCandidate: PushCandidate,\n    selectedCandidateType: MrQualityUprankingPartialTypeEnum.Value\n  ): Boolean = {\n    pushCandidate match {\n      case candidate: PushCandidate with TweetCandidate =>\n        selectedCandidateType match {\n          case MrQualityUprankingPartialTypeEnum.Oon =>\n            val crt = candidate.commonRecType\n            RecTypes.isOutOfNetworkTweetRecType(crt) || RecTypes.outOfNetworkTopicTweetTypes\n              .contains(crt)\n          case _ => true\n        }\n      case _ => false\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/rank/PushserviceRanker.scala",
    "content": "package com.twitter.frigate.pushservice.rank\n\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.Ranker\nimport com.twitter.util.Future\n\ntrait PushserviceRanker[T, C] extends Ranker[T, C] {\n\n  /**\n   * Initial Ranking of input candidates\n   */\n  def initialRank(target: T, candidates: Seq[CandidateDetails[C]]): Future[Seq[CandidateDetails[C]]]\n\n  /**\n   * Re-ranks input ranked candidates. Useful when a subset of candidates are ranked\n   * by a different logic, while preserving the initial ranking for the rest\n   */\n  def reRank(\n    target: T,\n    rankedCandidates: Seq[CandidateDetails[C]]\n  ): Future[Seq[CandidateDetails[C]]]\n\n  /**\n   * Final ranking that does Initial + Rerank\n   */\n  override final def rank(target: T, candidates: Seq[CandidateDetails[C]]): (\n    Future[Seq[CandidateDetails[C]]]\n  ) = {\n    initialRank(target, candidates).flatMap { rankedCandidates => reRank(target, rankedCandidates) }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/rank/RFPHLightRanker.scala",
    "content": "package com.twitter.frigate.pushservice.rank\nimport com.twitter.contentrecommender.thriftscala.LightRankingCandidate\nimport com.twitter.contentrecommender.thriftscala.LightRankingFeatureHydrationContext\nimport com.twitter.contentrecommender.thriftscala.MagicRecsFeatureHydrationContext\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.RandomRanker\nimport com.twitter.frigate.common.base.Ranker\nimport com.twitter.frigate.common.base.TweetAuthor\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.ml.featurestore.lib.UserId\nimport com.twitter.nrel.lightranker.MagicRecsServeDataRecordLightRanker\nimport com.twitter.util.Future\n\nclass RFPHLightRanker(\n  lightRanker: MagicRecsServeDataRecordLightRanker,\n  stats: StatsReceiver)\n    extends Ranker[Target, PushCandidate] {\n\n  private val statsReceiver = stats.scope(this.getClass.getSimpleName)\n\n  private val lightRankerCandidateCounter = statsReceiver.counter(\"light_ranker_candidate_count\")\n  private val lightRankerRequestCounter = statsReceiver.counter(\"light_ranker_request_count\")\n  private val lightRankingStats: StatsReceiver = statsReceiver.scope(\"light_ranking\")\n  private val restrictLightRankingCounter: Counter =\n    lightRankingStats.counter(\"restrict_light_ranking\")\n  private val selectedLightRankerScribedTargetCandidateCountStats: Stat =\n    lightRankingStats.stat(\"selected_light_ranker_scribed_target_candidate_count\")\n  private val selectedLightRankerScribedCandidatesStats: Stat =\n    lightRankingStats.stat(\"selected_light_ranker_scribed_candidates\")\n  private val lightRankingRandomBaselineStats: StatsReceiver =\n    statsReceiver.scope(\"light_ranking_random_baseline\")\n\n  override def rank(\n    target: Target,\n    candidates: Seq[CandidateDetails[PushCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    val enableLightRanker = target.params(PushFeatureSwitchParams.EnableLightRankingParam)\n    val restrictLightRanker = target.params(PushParams.RestrictLightRankingParam)\n    val lightRankerSelectionThreshold =\n      target.params(PushFeatureSwitchParams.LightRankingNumberOfCandidatesParam)\n    val randomRanker = RandomRanker[Target, PushCandidate]()(lightRankingRandomBaselineStats)\n\n    if (enableLightRanker && candidates.length > lightRankerSelectionThreshold && !target.scribeFeatureForRequestScribe) {\n      val (tweetCandidates, nonTweetCandidates) =\n        candidates.partition {\n          case CandidateDetails(pushCandidate: PushCandidate with TweetCandidate, source) => true\n          case _ => false\n        }\n      val lightRankerSelectedTweetCandidatesFut = {\n        if (restrictLightRanker) {\n          restrictLightRankingCounter.incr()\n          lightRankThenTake(\n            target,\n            tweetCandidates\n              .asInstanceOf[Seq[CandidateDetails[PushCandidate with TweetCandidate]]],\n            PushConstants.RestrictLightRankingCandidatesThreshold\n          )\n        } else if (target.params(PushFeatureSwitchParams.EnableRandomBaselineLightRankingParam)) {\n          randomRanker.rank(target, tweetCandidates).map { randomLightRankerCands =>\n            randomLightRankerCands.take(lightRankerSelectionThreshold)\n          }\n        } else {\n          lightRankThenTake(\n            target,\n            tweetCandidates\n              .asInstanceOf[Seq[CandidateDetails[PushCandidate with TweetCandidate]]],\n            lightRankerSelectionThreshold\n          )\n        }\n      }\n      lightRankerSelectedTweetCandidatesFut.map { returnedTweetCandidates =>\n        nonTweetCandidates ++ returnedTweetCandidates\n      }\n    } else if (target.scribeFeatureForRequestScribe) {\n      val downSampleRate: Double =\n        if (target.params(PushParams.DownSampleLightRankingScribeCandidatesParam))\n          PushConstants.DownSampleLightRankingScribeCandidatesRate\n        else target.params(PushFeatureSwitchParams.LightRankingScribeCandidatesDownSamplingParam)\n      val selectedCandidateCounter: Int = math.ceil(candidates.size * downSampleRate).toInt\n      selectedLightRankerScribedTargetCandidateCountStats.add(selectedCandidateCounter.toFloat)\n\n      randomRanker.rank(target, candidates).map { randomLightRankerCands =>\n        val selectedCandidates = randomLightRankerCands.take(selectedCandidateCounter)\n        selectedLightRankerScribedCandidatesStats.add(selectedCandidates.size.toFloat)\n        selectedCandidates\n      }\n    } else Future.value(candidates)\n  }\n\n  private def lightRankThenTake(\n    target: Target,\n    candidates: Seq[CandidateDetails[PushCandidate with TweetCandidate]],\n    numOfCandidates: Int\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    lightRankerCandidateCounter.incr(candidates.length)\n    lightRankerRequestCounter.incr()\n    val lightRankerCandidates: Seq[LightRankingCandidate] = candidates.map {\n      case CandidateDetails(tweetCandidate, _) =>\n        val tweetAuthor = tweetCandidate match {\n          case t: TweetCandidate with TweetAuthor => t.authorId\n          case _ => None\n        }\n        val hydrationContext: LightRankingFeatureHydrationContext =\n          LightRankingFeatureHydrationContext.MagicRecsHydrationContext(\n            MagicRecsFeatureHydrationContext(\n              tweetAuthor = tweetAuthor,\n              pushString = tweetCandidate.getPushCopy.flatMap(_.pushStringGroup).map(_.toString))\n          )\n        LightRankingCandidate(\n          tweetId = tweetCandidate.tweetId,\n          hydrationContext = Some(hydrationContext)\n        )\n    }\n    val modelName = target.params(PushFeatureSwitchParams.LightRankingModelTypeParam)\n    val lightRankedCandidatesFut = {\n      lightRanker\n        .rank(UserId(target.targetId), lightRankerCandidates, modelName)\n    }\n\n    lightRankedCandidatesFut.map { lightRankedCandidates =>\n      val lrScoreMap = lightRankedCandidates.map { lrCand =>\n        lrCand.tweetId -> lrCand.score\n      }.toMap\n      val candScoreMap: Seq[Option[Double]] = candidates.map { candidateDetails =>\n        lrScoreMap.get(candidateDetails.candidate.tweetId)\n      }\n      sortCandidatesByScore(candidates, candScoreMap)\n        .take(numOfCandidates)\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/rank/RFPHRanker.scala",
    "content": "package com.twitter.frigate.pushservice.rank\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.Ranker\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.ml.HealthFeatureGetter\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.params.MrQualityUprankingPartialTypeEnum\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushMLModel\nimport com.twitter.frigate.pushservice.params.PushModelName\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.util.MediaAnnotationsUtil.updateMediaCategoryStats\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.util.Future\nimport com.twitter.frigate.pushservice.params.MrQualityUprankingTransformTypeEnum\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.frigate.thriftscala.UserMediaRepresentation\nimport com.twitter.hss.api.thriftscala.UserHealthSignalResponse\n\nclass RFPHRanker(\n  randomRanker: Ranker[Target, PushCandidate],\n  weightedOpenOrNtabClickModelScorer: PushMLModelScorer,\n  subscriptionCreatorRanker: SubscriptionCreatorRanker,\n  userHealthSignalStore: ReadableStore[Long, UserHealthSignalResponse],\n  producerMediaRepresentationStore: ReadableStore[Long, UserMediaRepresentation],\n  stats: StatsReceiver)\n    extends PushserviceRanker[Target, PushCandidate] {\n\n  private val statsReceiver = stats.scope(this.getClass.getSimpleName)\n\n  private val boostCRTsRanker = CRTBoostRanker(statsReceiver.scope(\"boost_desired_crts\"))\n  private val crtDownRanker = CRTDownRanker(statsReceiver.scope(\"down_rank_desired_crts\"))\n\n  private val crtsToDownRank = statsReceiver.stat(\"crts_to_downrank\")\n  private val crtsToUprank = statsReceiver.stat(\"crts_to_uprank\")\n\n  private val randomRankingCounter = stats.counter(\"randomRanking\")\n  private val mlRankingCounter = stats.counter(\"mlRanking\")\n  private val disableAllRelevanceCounter = stats.counter(\"disableAllRelevance\")\n  private val disableHeavyRankingCounter = stats.counter(\"disableHeavyRanking\")\n\n  private val heavyRankerCandidateCounter = stats.counter(\"heavy_ranker_candidate_count\")\n  private val heavyRankerScoreStats = statsReceiver.scope(\"heavy_ranker_prediction_scores\")\n\n  private val producerUprankingCounter = statsReceiver.counter(\"producer_quality_upranking\")\n  private val producerBoostedCounter = statsReceiver.counter(\"producer_boosted_candidates\")\n  private val producerDownboostedCounter = statsReceiver.counter(\"producer_downboosted_candidates\")\n\n  override def initialRank(\n    target: Target,\n    candidates: Seq[CandidateDetails[PushCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n\n    heavyRankerCandidateCounter.incr(candidates.size)\n\n    updateMediaCategoryStats(candidates)(stats)\n    target.targetUserState\n      .flatMap { targetUserState =>\n        val useRandomRanking = target.skipMlRanker || target.params(\n          PushParams.UseRandomRankingParam\n        )\n\n        if (useRandomRanking) {\n          randomRankingCounter.incr()\n          randomRanker.rank(target, candidates)\n        } else if (target.params(PushParams.DisableAllRelevanceParam)) {\n          disableAllRelevanceCounter.incr()\n          Future.value(candidates)\n        } else if (target.params(PushParams.DisableHeavyRankingParam) || target.params(\n            PushFeatureSwitchParams.DisableHeavyRankingModelFSParam)) {\n          disableHeavyRankingCounter.incr()\n          Future.value(candidates)\n        } else {\n          mlRankingCounter.incr()\n\n          val scoredCandidatesFut = scoring(target, candidates)\n\n          target.rankingModelParam.map { rankingModelParam =>\n            val modelName = PushModelName(\n              PushMLModel.WeightedOpenOrNtabClickProbability,\n              target.params(rankingModelParam)).toString\n            ModelBasedRanker.populateMrWeightedOpenOrNtabClickScoreStats(\n              candidates,\n              heavyRankerScoreStats.scope(modelName)\n            )\n          }\n\n          if (target.params(\n              PushFeatureSwitchParams.EnableQualityUprankingCrtScoreStatsForHeavyRankingParam)) {\n            val modelName = PushModelName(\n              PushMLModel.FilteringProbability,\n              target.params(PushFeatureSwitchParams.QualityUprankingModelTypeParam)\n            ).toString\n            ModelBasedRanker.populateMrQualityUprankingScoreStats(\n              candidates,\n              heavyRankerScoreStats.scope(modelName)\n            )\n          }\n\n          val ooncRankedCandidatesFut =\n            scoredCandidatesFut.flatMap(ModelBasedRanker.rankByMrWeightedOpenOrNtabClickScore)\n\n          val qualityUprankedCandidatesFut =\n            if (target.params(PushFeatureSwitchParams.EnableQualityUprankingForHeavyRankingParam)) {\n              ooncRankedCandidatesFut.flatMap { ooncRankedCandidates =>\n                val transformFunc: Double => Double =\n                  target.params(PushFeatureSwitchParams.QualityUprankingTransformTypeParam) match {\n                    case MrQualityUprankingTransformTypeEnum.Linear =>\n                      ModelBasedRanker.transformLinear(\n                        _,\n                        bar = target.params(\n                          PushFeatureSwitchParams.QualityUprankingLinearBarForHeavyRankingParam))\n                    case MrQualityUprankingTransformTypeEnum.Sigmoid =>\n                      ModelBasedRanker.transformSigmoid(\n                        _,\n                        weight = target.params(\n                          PushFeatureSwitchParams.QualityUprankingSigmoidWeightForHeavyRankingParam),\n                        bias = target.params(\n                          PushFeatureSwitchParams.QualityUprankingSigmoidBiasForHeavyRankingParam)\n                      )\n                    case _ => ModelBasedRanker.transformIdentity\n                  }\n\n                ModelBasedRanker.rankByQualityOoncCombinedScore(\n                  ooncRankedCandidates,\n                  transformFunc,\n                  target.params(PushFeatureSwitchParams.QualityUprankingBoostForHeavyRankingParam)\n                )\n              }\n            } else ooncRankedCandidatesFut\n\n          if (target.params(\n              PushFeatureSwitchParams.EnableProducersQualityBoostingForHeavyRankingParam)) {\n            producerUprankingCounter.incr()\n            qualityUprankedCandidatesFut.flatMap(cands =>\n              ModelBasedRanker.rerankByProducerQualityOoncCombinedScore(cands)(statsReceiver))\n          } else qualityUprankedCandidatesFut\n        }\n      }\n  }\n\n  private def scoring(\n    target: Target,\n    candidates: Seq[CandidateDetails[PushCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n\n    val ooncScoredCandidatesFut = target.rankingModelParam.map { rankingModelParam =>\n      weightedOpenOrNtabClickModelScorer.scoreByBatchPredictionForModelVersion(\n        target,\n        candidates,\n        rankingModelParam\n      )\n    }\n\n    val scoredCandidatesFut = {\n      if (target.params(PushFeatureSwitchParams.EnableQualityUprankingForHeavyRankingParam)) {\n        ooncScoredCandidatesFut.map { candidates =>\n          weightedOpenOrNtabClickModelScorer.scoreByBatchPredictionForModelVersion(\n            target = target,\n            candidatesDetails = candidates,\n            modelVersionParam = PushFeatureSwitchParams.QualityUprankingModelTypeParam,\n            overridePushMLModelOpt = Some(PushMLModel.FilteringProbability)\n          )\n        }\n      } else ooncScoredCandidatesFut\n    }\n\n    scoredCandidatesFut.foreach { candidates =>\n      val oonCandidates = candidates.filter {\n        case CandidateDetails(pushCandidate: PushCandidate, _) =>\n          ModelBasedRanker.tweetCandidateSelector(\n            pushCandidate,\n            MrQualityUprankingPartialTypeEnum.Oon)\n      }\n      setProducerQuality(\n        target,\n        oonCandidates,\n        userHealthSignalStore,\n        producerMediaRepresentationStore)\n    }\n  }\n\n  private def setProducerQuality(\n    target: Target,\n    candidates: Seq[CandidateDetails[PushCandidate]],\n    userHealthSignalStore: ReadableStore[Long, UserHealthSignalResponse],\n    producerMediaRepresentationStore: ReadableStore[Long, UserMediaRepresentation]\n  ): Unit = {\n    lazy val boostRatio =\n      target.params(PushFeatureSwitchParams.QualityUprankingBoostForHighQualityProducersParam)\n    lazy val downboostRatio =\n      target.params(PushFeatureSwitchParams.QualityUprankingDownboostForLowQualityProducersParam)\n    candidates.foreach {\n      case CandidateDetails(pushCandidate, _) =>\n        HealthFeatureGetter\n          .getFeatures(pushCandidate, producerMediaRepresentationStore, userHealthSignalStore).map {\n            featureMap =>\n              val agathaNsfwScore = featureMap.numericFeatures.getOrElse(\"agathaNsfwScore\", 0.5)\n              val textNsfwScore = featureMap.numericFeatures.getOrElse(\"textNsfwScore\", 0.15)\n              val nudityRate = featureMap.numericFeatures.getOrElse(\"nudityRate\", 0.0)\n              val activeFollowers = featureMap.numericFeatures.getOrElse(\"activeFollowers\", 0.0)\n              val favorsRcvd28Days = featureMap.numericFeatures.getOrElse(\"favorsRcvd28Days\", 0.0)\n              val tweets28Days = featureMap.numericFeatures.getOrElse(\"tweets28Days\", 0.0)\n              val authorDislikeCount = featureMap.numericFeatures\n                .getOrElse(\"authorDislikeCount\", 0.0)\n              val authorDislikeRate = featureMap.numericFeatures.getOrElse(\"authorDislikeRate\", 0.0)\n              val authorReportRate = featureMap.numericFeatures.getOrElse(\"authorReportRate\", 0.0)\n              val abuseStrikeTop2Percent =\n                featureMap.booleanFeatures.getOrElse(\"abuseStrikeTop2Percent\", false)\n              val abuseStrikeTop1Percent =\n                featureMap.booleanFeatures.getOrElse(\"abuseStrikeTop1Percent\", false)\n              val hasNsfwToken = featureMap.booleanFeatures.getOrElse(\"hasNsfwToken\", false)\n\n              if ((activeFollowers > 3000000) ||\n                (activeFollowers > 1000000 && agathaNsfwScore < 0.7 && nudityRate < 0.01 && !hasNsfwToken && !abuseStrikeTop2Percent) ||\n                (activeFollowers > 100000 && agathaNsfwScore < 0.7 && nudityRate < 0.01 && !hasNsfwToken && !abuseStrikeTop2Percent &&\n                tweets28Days > 0 && favorsRcvd28Days / tweets28Days > 3000 && authorReportRate < 0.000001 && authorDislikeRate < 0.0005)) {\n                producerBoostedCounter.incr()\n                pushCandidate.setProducerQualityUprankingBoost(boostRatio)\n              } else if (activeFollowers < 5 || agathaNsfwScore > 0.9 || nudityRate > 0.03 || hasNsfwToken || abuseStrikeTop1Percent ||\n                textNsfwScore > 0.4 || (authorDislikeRate > 0.005 && authorDislikeCount > 5) ||\n                (tweets28Days > 56 && favorsRcvd28Days / tweets28Days < 100)) {\n                producerDownboostedCounter.incr()\n                pushCandidate.setProducerQualityUprankingBoost(downboostRatio)\n              } else pushCandidate.setProducerQualityUprankingBoost(1.0)\n          }\n    }\n  }\n\n  private def rerankBySubscriptionCreatorRanker(\n    target: Target,\n    rankedCandidates: Future[Seq[CandidateDetails[PushCandidate]]],\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    if (target.params(PushFeatureSwitchParams.SoftRankCandidatesFromSubscriptionCreators)) {\n      val factor = target.params(PushFeatureSwitchParams.SoftRankFactorForSubscriptionCreators)\n      subscriptionCreatorRanker.boostByScoreFactor(rankedCandidates, factor)\n    } else\n      subscriptionCreatorRanker.boostSubscriptionCreator(rankedCandidates)\n  }\n\n  override def reRank(\n    target: Target,\n    rankedCandidates: Seq[CandidateDetails[PushCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    val numberOfF1Candidates =\n      rankedCandidates.count(candidateDetails =>\n        RecTypes.isF1Type(candidateDetails.candidate.commonRecType))\n    lazy val threshold =\n      target.params(PushFeatureSwitchParams.NumberOfF1CandidatesThresholdForOONBackfill)\n    lazy val enableOONBackfillBasedOnF1 =\n      target.params(PushFeatureSwitchParams.EnableOONBackfillBasedOnF1Candidates)\n\n    val f1BoostedCandidates =\n      if (enableOONBackfillBasedOnF1 && numberOfF1Candidates > threshold) {\n        boostCRTsRanker.boostCrtsToTopStableOrder(\n          rankedCandidates,\n          RecTypes.f1FirstDegreeTypes.toSeq)\n      } else rankedCandidates\n\n    val topTweetsByGeoDownRankedCandidates =\n      if (target.params(PushFeatureSwitchParams.BackfillRankTopTweetsByGeoCandidates)) {\n        crtDownRanker.downRank(\n          f1BoostedCandidates,\n          Seq(CommonRecommendationType.GeoPopTweet)\n        )\n      } else f1BoostedCandidates\n\n    val reRankedCandidatesWithBoostedCrts = {\n      val listOfCrtsToUpRank = target\n        .params(PushFeatureSwitchParams.ListOfCrtsToUpRank)\n        .flatMap(CommonRecommendationType.valueOf)\n      crtsToUprank.add(listOfCrtsToUpRank.size)\n      boostCRTsRanker.boostCrtsToTop(topTweetsByGeoDownRankedCandidates, listOfCrtsToUpRank)\n    }\n\n    val reRankedCandidatesWithDownRankedCrts = {\n      val listOfCrtsToDownRank = target\n        .params(PushFeatureSwitchParams.ListOfCrtsToDownRank)\n        .flatMap(CommonRecommendationType.valueOf)\n      crtsToDownRank.add(listOfCrtsToDownRank.size)\n      crtDownRanker.downRank(reRankedCandidatesWithBoostedCrts, listOfCrtsToDownRank)\n    }\n\n    val rerankBySubscriptionCreatorFut = {\n      if (target.params(PushFeatureSwitchParams.BoostCandidatesFromSubscriptionCreators)) {\n        rerankBySubscriptionCreatorRanker(\n          target,\n          Future.value(reRankedCandidatesWithDownRankedCrts))\n      } else Future.value(reRankedCandidatesWithDownRankedCrts)\n    }\n\n    rerankBySubscriptionCreatorFut\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/rank/SubscriptionCreatorRanker.scala",
    "content": "package com.twitter.frigate.pushservice.rank\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.TweetAuthor\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.storehaus.FutureOps\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nclass SubscriptionCreatorRanker(\n  superFollowEligibilityUserStore: ReadableStore[Long, Boolean],\n  statsReceiver: StatsReceiver) {\n\n  private val scopedStats = statsReceiver.scope(\"SubscriptionCreatorRanker\")\n  private val boostStats = scopedStats.scope(\"boostSubscriptionCreator\")\n  private val softUprankStats = scopedStats.scope(\"boostByScoreFactor\")\n  private val boostTotalCandidates = boostStats.stat(\"total_input_candidates\")\n  private val softRankTotalCandidates = softUprankStats.stat(\"total_input_candidates\")\n  private val softRankNumCandidatesCreators = softUprankStats.counter(\"candidates_from_creators\")\n  private val softRankNumCandidatesNonCreators =\n    softUprankStats.counter(\"candidates_not_from_creators\")\n  private val boostNumCandidatesCreators = boostStats.counter(\"candidates_from_creators\")\n  private val boostNumCandidatesNonCreators =\n    boostStats.counter(\"candidates_not_from_creators\")\n\n  def boostSubscriptionCreator(\n    inputCandidatesFut: Future[Seq[CandidateDetails[PushCandidate]]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n\n    inputCandidatesFut.flatMap { inputCandidates =>\n      boostTotalCandidates.add(inputCandidates.size)\n      val tweetAuthorIds = inputCandidates.flatMap {\n        case CandidateDetails(candidate: TweetCandidate with TweetAuthor, s) =>\n          candidate.authorId\n        case _ => None\n      }.toSet\n\n      FutureOps\n        .mapCollect(superFollowEligibilityUserStore.multiGet(tweetAuthorIds))\n        .map { creatorAuthorMap =>\n          val (upRankedCandidates, otherCandidates) = inputCandidates.partition {\n            case CandidateDetails(candidate: TweetCandidate with TweetAuthor, s) =>\n              candidate.authorId match {\n                case Some(authorId) =>\n                  creatorAuthorMap(authorId).getOrElse(false)\n                case _ => false\n              }\n            case _ => false\n          }\n          boostNumCandidatesCreators.incr(upRankedCandidates.size)\n          boostNumCandidatesNonCreators.incr(otherCandidates.size)\n          upRankedCandidates ++ otherCandidates\n        }\n    }\n  }\n\n  def boostByScoreFactor(\n    inputCandidatesFut: Future[Seq[CandidateDetails[PushCandidate]]],\n    factor: Double = 1.0,\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n\n    inputCandidatesFut.flatMap { inputCandidates =>\n      softRankTotalCandidates.add(inputCandidates.size)\n      val tweetAuthorIds = inputCandidates.flatMap {\n        case CandidateDetails(candidate: TweetCandidate with TweetAuthor, s) =>\n          candidate.authorId\n        case _ => None\n      }.toSet\n\n      FutureOps\n        .mapCollect(superFollowEligibilityUserStore.multiGet(tweetAuthorIds))\n        .flatMap { creatorAuthorMap =>\n          val (upRankedCandidates, otherCandidates) = inputCandidates.partition {\n            case CandidateDetails(candidate: TweetCandidate with TweetAuthor, s) =>\n              candidate.authorId match {\n                case Some(authorId) =>\n                  creatorAuthorMap(authorId).getOrElse(false)\n                case _ => false\n              }\n            case _ => false\n          }\n          softRankNumCandidatesCreators.incr(upRankedCandidates.size)\n          softRankNumCandidatesNonCreators.incr(otherCandidates.size)\n\n          ModelBasedRanker.rankBySpecifiedScore(\n            inputCandidates,\n            candidate => {\n              val isFromCreator = candidate match {\n                case candidate: TweetCandidate with TweetAuthor =>\n                  candidate.authorId match {\n                    case Some(authorId) =>\n                      creatorAuthorMap(authorId).getOrElse(false)\n                    case _ => false\n                  }\n                case _ => false\n              }\n              candidate.mrWeightedOpenOrNtabClickRankingProbability.map {\n                case Some(score) =>\n                  if (isFromCreator) Some(score * factor)\n                  else Some(score)\n                case _ => None\n              }\n            }\n          )\n        }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/LoggedOutRefreshForPushHandler.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.CandidateResult\nimport com.twitter.frigate.common.base.CandidateSource\nimport com.twitter.frigate.common.base.FetchRankFlowWithHydratedCandidates\nimport com.twitter.frigate.common.base.Invalid\nimport com.twitter.frigate.common.base.OK\nimport com.twitter.frigate.common.base.Response\nimport com.twitter.frigate.common.base.Result\nimport com.twitter.frigate.common.base.Stats.track\nimport com.twitter.frigate.common.base.Stats.trackSeq\nimport com.twitter.frigate.common.logger.MRLogger\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.adaptor.LoggedOutPushCandidateSourceGenerator\nimport com.twitter.frigate.pushservice.predicate.LoggedOutPreRankingPredicates\nimport com.twitter.frigate.pushservice.predicate.LoggedOutTargetPredicates\nimport com.twitter.frigate.pushservice.rank.LoggedOutRanker\nimport com.twitter.frigate.pushservice.take.LoggedOutRefreshForPushNotifier\nimport com.twitter.frigate.pushservice.scriber.MrRequestScribeHandler\nimport com.twitter.frigate.pushservice.target.LoggedOutPushTargetUserBuilder\nimport com.twitter.frigate.pushservice.thriftscala.LoggedOutRequest\nimport com.twitter.frigate.pushservice.thriftscala.LoggedOutResponse\nimport com.twitter.frigate.pushservice.thriftscala.PushContext\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.hermit.predicate.SequentialPredicate\nimport com.twitter.util.Future\n\nclass LoggedOutRefreshForPushHandler(\n  val loPushTargetUserBuilder: LoggedOutPushTargetUserBuilder,\n  val loPushCandidateSourceGenerator: LoggedOutPushCandidateSourceGenerator,\n  candidateHydrator: PushCandidateHydrator,\n  val loRanker: LoggedOutRanker,\n  val loRfphNotifier: LoggedOutRefreshForPushNotifier,\n  loMrRequestScriberNode: String\n)(\n  globalStats: StatsReceiver)\n    extends FetchRankFlowWithHydratedCandidates[Target, RawCandidate, PushCandidate] {\n\n  val log = MRLogger(\"LORefreshForPushHandler\")\n  implicit val statsReceiver: StatsReceiver =\n    globalStats.scope(\"LORefreshForPushHandler\")\n  private val loggedOutBuildStats = statsReceiver.scope(\"logged_out_build_target\")\n  private val loggedOutProcessStats = statsReceiver.scope(\"logged_out_process\")\n  private val loggedOutNotifyStats = statsReceiver.scope(\"logged_out_notify\")\n  private val loCandidateHydrationStats: StatsReceiver =\n    statsReceiver.scope(\"logged_out_candidate_hydration\")\n  val mrLORequestCandidateScribeStats =\n    statsReceiver.scope(\"mr_logged_out_request_scribe_candidates\")\n\n  val mrRequestScribeHandler =\n    new MrRequestScribeHandler(loMrRequestScriberNode, statsReceiver.scope(\"lo_mr_request_scribe\"))\n  val loMrRequestTargetScribeStats = statsReceiver.scope(\"lo_mr_request_scribe_target\")\n\n  lazy val loCandSourceEligibleCounter: Counter =\n    loCandidateStats.counter(\"logged_out_cand_source_eligible\")\n  lazy val loCandSourceNotEligibleCounter: Counter =\n    loCandidateStats.counter(\"logged_out_cand_source_not_eligible\")\n  lazy val allCandidatesCounter: Counter = statsReceiver.counter(\"all_logged_out_candidates\")\n  val allCandidatesFilteredPreRank = filterStats.counter(\"all_logged_out_candidates_filtered\")\n\n  override def targetPredicates(target: Target): List[Predicate[Target]] = List(\n    LoggedOutTargetPredicates.targetFatiguePredicate(),\n    LoggedOutTargetPredicates.loggedOutRecsHoldbackPredicate()\n  )\n\n  override def isTargetValid(target: Target): Future[Result] = {\n    val resultFut =\n      if (target.skipFilters) {\n        Future.value(OK)\n      } else {\n        predicateSeq(target).track(Seq(target)).map { resultArr =>\n          trackTargetPredStats(resultArr(0))\n        }\n      }\n    track(targetStats)(resultFut)\n  }\n\n  override def rank(\n    target: Target,\n    candidateDetails: Seq[CandidateDetails[PushCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    loRanker.rank(candidateDetails)\n  }\n\n  override def validCandidates(\n    target: Target,\n    candidates: Seq[PushCandidate]\n  ): Future[Seq[Result]] = {\n    Future.value(candidates.map { c => OK })\n  }\n\n  override def desiredCandidateCount(target: Target): Int = 1\n\n  private val loggedOutPreRankingPredicates =\n    LoggedOutPreRankingPredicates(filterStats.scope(\"logged_out_predicates\"))\n\n  private val loggedOutPreRankingPredicateChain =\n    new SequentialPredicate[PushCandidate](loggedOutPreRankingPredicates)\n\n  override def filter(\n    target: Target,\n    candidates: Seq[CandidateDetails[PushCandidate]]\n  ): Future[\n    (Seq[CandidateDetails[PushCandidate]], Seq[CandidateResult[PushCandidate, Result]])\n  ] = {\n    val predicateChain = loggedOutPreRankingPredicateChain\n    predicateChain\n      .track(candidates.map(_.candidate))\n      .map { results =>\n        val resultForPreRankingFiltering =\n          results\n            .zip(candidates)\n            .foldLeft(\n              (\n                Seq.empty[CandidateDetails[PushCandidate]],\n                Seq.empty[CandidateResult[PushCandidate, Result]]\n              )\n            ) {\n              case ((goodCandidates, filteredCandidates), (result, candidateDetails)) =>\n                result match {\n                  case None =>\n                    (goodCandidates :+ candidateDetails, filteredCandidates)\n\n                  case Some(pred: NamedPredicate[_]) =>\n                    val r = Invalid(Some(pred.name))\n                    (\n                      goodCandidates,\n                      filteredCandidates :+ CandidateResult[PushCandidate, Result](\n                        candidateDetails.candidate,\n                        candidateDetails.source,\n                        r\n                      )\n                    )\n                  case Some(_) =>\n                    val r = Invalid(Some(\"Filtered by un-named predicate\"))\n                    (\n                      goodCandidates,\n                      filteredCandidates :+ CandidateResult[PushCandidate, Result](\n                        candidateDetails.candidate,\n                        candidateDetails.source,\n                        r\n                      )\n                    )\n                }\n            }\n        resultForPreRankingFiltering match {\n          case (validCandidates, _) if validCandidates.isEmpty && candidates.nonEmpty =>\n            allCandidatesFilteredPreRank.incr()\n          case _ => ()\n\n        }\n        resultForPreRankingFiltering\n\n      }\n\n  }\n\n  override def candidateSources(\n    target: Target\n  ): Future[Seq[CandidateSource[Target, RawCandidate]]] = {\n    Future\n      .collect(loPushCandidateSourceGenerator.sources.map { cs =>\n        cs.isCandidateSourceAvailable(target).map { isEligible =>\n          if (isEligible) {\n            loCandSourceEligibleCounter.incr()\n            Some(cs)\n          } else {\n            loCandSourceNotEligibleCounter.incr()\n            None\n          }\n        }\n      }).map(_.flatten)\n  }\n\n  override def process(\n    target: Target,\n    externalCandidates: Seq[RawCandidate] = Nil\n  ): Future[Response[PushCandidate, Result]] = {\n    isTargetValid(target).flatMap {\n      case OK =>\n        for {\n          candidatesFromSources <- trackSeq(fetchStats)(fetchCandidates(target))\n          externalCandidateDetails = externalCandidates.map(\n            CandidateDetails(_, \"logged_out_refresh_for_push_handler_external_candidates\"))\n          allCandidates = candidatesFromSources ++ externalCandidateDetails\n          hydratedCandidatesWithCopy <-\n            trackSeq(loCandidateHydrationStats)(hydrateCandidates(allCandidates))\n          (candidates, preRankingFilteredCandidates) <-\n            track(filterStats)(filter(target, hydratedCandidatesWithCopy))\n          rankedCandidates <- trackSeq(rankingStats)(rank(target, candidates))\n          allTakeCandidateResults <- track(takeStats)(\n            take(target, rankedCandidates, desiredCandidateCount(target))\n          )\n          _ <- track(mrLORequestCandidateScribeStats)(\n            mrRequestScribeHandler.scribeForCandidateFiltering(\n              target,\n              hydratedCandidatesWithCopy,\n              preRankingFilteredCandidates,\n              rankedCandidates,\n              rankedCandidates,\n              rankedCandidates,\n              allTakeCandidateResults\n            ))\n\n        } yield {\n          val takeCandidateResults = allTakeCandidateResults.filterNot { candResult =>\n            candResult.result == MoreThanDesiredCandidates\n          }\n          val allCandidateResults = takeCandidateResults ++ preRankingFilteredCandidates\n          allCandidatesCounter.incr(allCandidateResults.size)\n          Response(OK, allCandidateResults)\n        }\n\n      case result: Result =>\n        for (_ <- track(loMrRequestTargetScribeStats)(\n            mrRequestScribeHandler.scribeForTargetFiltering(target, result))) yield {\n          Response(result, Nil)\n        }\n    }\n  }\n\n  def buildTarget(\n    guestId: Long,\n    inputPushContext: Option[PushContext]\n  ): Future[Target] =\n    loPushTargetUserBuilder.buildTarget(guestId, inputPushContext)\n\n  /**\n   * Hydrate candidate by querying downstream services\n   *\n   * @param candidates - candidates\n   *\n   * @return - hydrated candidates\n   */\n  override def hydrateCandidates(\n    candidates: Seq[CandidateDetails[RawCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = candidateHydrator(candidates)\n\n  override def batchForCandidatesCheck(target: Target): Int = 1\n\n  def refreshAndSend(request: LoggedOutRequest): Future[LoggedOutResponse] = {\n    for {\n      target <- track(loggedOutBuildStats)(\n        loPushTargetUserBuilder.buildTarget(request.guestId, request.context))\n      response <- track(loggedOutProcessStats)(process(target, externalCandidates = Seq.empty))\n      loggedOutRefreshResponse <-\n        track(loggedOutNotifyStats)(loRfphNotifier.checkResponseAndNotify(response))\n    } yield {\n      loggedOutRefreshResponse\n    }\n  }\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/PushCandidateHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler\n\nimport com.twitter.channels.common.thriftscala.ApiList\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.rec_types.RecTypes.isInNetworkTweetType\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.TrendTweetPushCandidate\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.refresh_handler.cross.CandidateCopyExpansion\nimport com.twitter.frigate.pushservice.util.CandidateHydrationUtil._\nimport com.twitter.frigate.pushservice.util.MrUserStateUtil\nimport com.twitter.frigate.pushservice.util.RelationshipUtil\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\ncase class PushCandidateHydrator(\n  socialGraphServiceProcessStore: ReadableStore[RelationEdge, Boolean],\n  safeUserStore: ReadableStore[Long, User],\n  apiListStore: ReadableStore[Long, ApiList],\n  candidateCopyCross: CandidateCopyExpansion\n)(\n  implicit statsReceiver: StatsReceiver,\n  implicit val weightedOpenOrNtabClickModelScorer: PushMLModelScorer) {\n\n  lazy val candidateWithCopyNumStat = statsReceiver.stat(\"candidate_with_copy_num\")\n  lazy val hydratedCandidateStat = statsReceiver.scope(\"hydrated_candidates\")\n  lazy val mrUserStateStat = statsReceiver.scope(\"mr_user_state\")\n\n  lazy val queryStep = statsReceiver.scope(\"query_step\")\n  lazy val relationEdgeWithoutDuplicateInQueryStep =\n    queryStep.counter(\"number_of_relationEdge_without_duplicate_in_query_step\")\n  lazy val relationEdgeWithoutDuplicateInQueryStepDistribution =\n    queryStep.stat(\"number_of_relationEdge_without_duplicate_in_query_step_distribution\")\n\n  case class Entities(\n    users: Set[Long] = Set.empty[Long],\n    relationshipEdges: Set[RelationEdge] = Set.empty[RelationEdge]) {\n    def merge(otherEntities: Entities): Entities = {\n      this.copy(\n        users = this.users ++ otherEntities.users,\n        relationshipEdges =\n          this.relationshipEdges ++ otherEntities.relationshipEdges\n      )\n    }\n  }\n\n  case class EntitiesMap(\n    userMap: Map[Long, User] = Map.empty[Long, User],\n    relationshipMap: Map[RelationEdge, Boolean] = Map.empty[RelationEdge, Boolean])\n\n  private def updateCandidateAndCrtStats(\n    candidate: RawCandidate,\n    candidateType: String,\n    numEntities: Int = 1\n  ): Unit = {\n    statsReceiver\n      .scope(candidateType).scope(candidate.commonRecType.name).stat(\n        \"totalEntitiesPerCandidateTypePerCrt\").add(numEntities)\n    statsReceiver.scope(candidateType).stat(\"totalEntitiesPerCandidateType\").add(numEntities)\n  }\n\n  private def collectEntities(\n    candidateDetailsSeq: Seq[CandidateDetails[RawCandidate]]\n  ): Entities = {\n    candidateDetailsSeq\n      .map { candidateDetails =>\n        val pushCandidate = candidateDetails.candidate\n\n        val userEntities = pushCandidate match {\n          case tweetWithSocialContext: RawCandidate with TweetWithSocialContextTraits =>\n            val authorIdOpt = getAuthorIdFromTweetCandidate(tweetWithSocialContext)\n            val scUserIds = tweetWithSocialContext.socialContextUserIds.toSet\n            updateCandidateAndCrtStats(pushCandidate, \"tweetWithSocialContext\", scUserIds.size + 1)\n            Entities(users = scUserIds ++ authorIdOpt.toSet)\n\n          case _ => Entities()\n        }\n\n        val relationEntities = {\n          if (isInNetworkTweetType(pushCandidate.commonRecType)) {\n            Entities(\n              relationshipEdges =\n                RelationshipUtil.getPreCandidateRelationshipsForInNetworkTweets(pushCandidate).toSet\n            )\n          } else Entities()\n        }\n\n        userEntities.merge(relationEntities)\n      }\n      .foldLeft(Entities()) { (e1, e2) => e1.merge(e2) }\n\n  }\n\n  /**\n   * This method calls Gizmoduck and Social Graph Service, keep the results in EntitiesMap\n   * and passed onto the update candidate phase in the hydration step\n   *\n   * @param entities contains all userIds and relationEdges for all candidates\n   * @return EntitiesMap contains userMap and relationshipMap\n   */\n  private def queryEntities(entities: Entities): Future[EntitiesMap] = {\n\n    relationEdgeWithoutDuplicateInQueryStep.incr(entities.relationshipEdges.size)\n    relationEdgeWithoutDuplicateInQueryStepDistribution.add(entities.relationshipEdges.size)\n\n    val relationshipMapFuture = Future\n      .collect(socialGraphServiceProcessStore.multiGet(entities.relationshipEdges))\n      .map { resultMap =>\n        resultMap.collect {\n          case (relationshipEdge, Some(res)) => relationshipEdge -> res\n          case (relationshipEdge, None) => relationshipEdge -> false\n        }\n      }\n\n    val userMapFuture = Future\n      .collect(safeUserStore.multiGet(entities.users))\n      .map { userMap =>\n        userMap.collect {\n          case (userId, Some(user)) =>\n            userId -> user\n        }\n      }\n\n    Future.join(userMapFuture, relationshipMapFuture).map {\n      case (uMap, rMap) => EntitiesMap(userMap = uMap, relationshipMap = rMap)\n    }\n  }\n\n  /**\n   * @param candidateDetails: recommendation candidates for a user\n   * @return sequence of candidates tagged with push and ntab copy id\n   */\n  private def expandCandidatesWithCopy(\n    candidateDetails: Seq[CandidateDetails[RawCandidate]]\n  ): Future[Seq[(CandidateDetails[RawCandidate], CopyIds)]] = {\n    candidateCopyCross.expandCandidatesWithCopyId(candidateDetails)\n  }\n\n  def updateCandidates(\n    candidateDetailsWithCopies: Seq[(CandidateDetails[RawCandidate], CopyIds)],\n    entitiesMaps: EntitiesMap\n  ): Seq[CandidateDetails[PushCandidate]] = {\n    candidateDetailsWithCopies.map {\n      case (candidateDetail, copyIds) =>\n        val pushCandidate = candidateDetail.candidate\n        val userMap = entitiesMaps.userMap\n        val relationshipMap = entitiesMaps.relationshipMap\n\n        val hydratedCandidate = pushCandidate match {\n\n          case f1TweetCandidate: F1FirstDegree =>\n            getHydratedCandidateForF1FirstDegreeTweet(\n              f1TweetCandidate,\n              userMap,\n              relationshipMap,\n              copyIds)\n\n          case tweetRetweet: TweetRetweetCandidate =>\n            getHydratedCandidateForTweetRetweet(tweetRetweet, userMap, copyIds)\n\n          case tweetFavorite: TweetFavoriteCandidate =>\n            getHydratedCandidateForTweetFavorite(tweetFavorite, userMap, copyIds)\n\n          case tripTweetCandidate: OutOfNetworkTweetCandidate with TripCandidate =>\n            getHydratedCandidateForTripTweetCandidate(tripTweetCandidate, userMap, copyIds)\n\n          case outOfNetworkTweetCandidate: OutOfNetworkTweetCandidate with TopicCandidate =>\n            getHydratedCandidateForOutOfNetworkTweetCandidate(\n              outOfNetworkTweetCandidate,\n              userMap,\n              copyIds)\n\n          case topicProofTweetCandidate: TopicProofTweetCandidate =>\n            getHydratedTopicProofTweetCandidate(topicProofTweetCandidate, userMap, copyIds)\n\n          case subscribedSearchTweetCandidate: SubscribedSearchTweetCandidate =>\n            getHydratedSubscribedSearchTweetCandidate(\n              subscribedSearchTweetCandidate,\n              userMap,\n              copyIds)\n\n          case listRecommendation: ListPushCandidate =>\n            getHydratedListCandidate(apiListStore, listRecommendation, copyIds)\n\n          case discoverTwitterCandidate: DiscoverTwitterCandidate =>\n            getHydratedCandidateForDiscoverTwitterCandidate(discoverTwitterCandidate, copyIds)\n\n          case topTweetImpressionsCandidate: TopTweetImpressionsCandidate =>\n            getHydratedCandidateForTopTweetImpressionsCandidate(\n              topTweetImpressionsCandidate,\n              copyIds)\n\n          case trendTweetCandidate: TrendTweetCandidate =>\n            new TrendTweetPushCandidate(\n              trendTweetCandidate,\n              trendTweetCandidate.authorId.flatMap(userMap.get),\n              copyIds)\n\n          case unknownCandidate =>\n            throw new IllegalArgumentException(\n              s\"Incorrect candidate for hydration: ${unknownCandidate.commonRecType}\")\n        }\n\n        CandidateDetails(\n          hydratedCandidate,\n          source = candidateDetail.source\n        )\n    }\n  }\n\n  def apply(\n    candidateDetails: Seq[CandidateDetails[RawCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    val isLoggedOutRequest =\n      candidateDetails.headOption.exists(_.candidate.target.isLoggedOutUser)\n    if (!isLoggedOutRequest) {\n      candidateDetails.headOption.map { cd =>\n        MrUserStateUtil.updateMrUserStateStats(cd.candidate.target)(mrUserStateStat)\n      }\n    }\n\n    expandCandidatesWithCopy(candidateDetails).flatMap { candidateDetailsWithCopy =>\n      candidateWithCopyNumStat.add(candidateDetailsWithCopy.size)\n      val entities = collectEntities(candidateDetailsWithCopy.map(_._1))\n      queryEntities(entities).flatMap { entitiesMap =>\n        val updatedCandidates = updateCandidates(candidateDetailsWithCopy, entitiesMap)\n        updatedCandidates.foreach { cand =>\n          hydratedCandidateStat.counter(cand.candidate.commonRecType.name).incr()\n        }\n        Future.value(updatedCandidates)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/RFPHFeatureHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.FeatureMap\nimport com.twitter.frigate.data_pipeline.features_common.MrRequestContextForFeatureStore\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.ml.HydrationContextBuilder\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.util.MrUserStateUtil\nimport com.twitter.nrel.heavyranker.FeatureHydrator\nimport com.twitter.util.Future\n\nclass RFPHFeatureHydrator(\n  featureHydrator: FeatureHydrator\n)(\n  implicit globalStats: StatsReceiver) {\n\n  implicit val statsReceiver: StatsReceiver =\n    globalStats.scope(\"RefreshForPushHandler\")\n\n  //stat for feature hydration\n  private val featureHydrationEnabledCounter = statsReceiver.counter(\"featureHydrationEnabled\")\n  private val mrUserStateStat = statsReceiver.scope(\"mr_user_state\")\n\n  private def hydrateFromRelevanceHydrator(\n    candidateDetails: Seq[CandidateDetails[PushCandidate]],\n    mrRequestContextForFeatureStore: MrRequestContextForFeatureStore\n  ): Future[Unit] = {\n    val pushCandidates = candidateDetails.map(_.candidate)\n    val candidatesAndContextsFut = Future.collect(pushCandidates.map { pc =>\n      val contextFut = HydrationContextBuilder.build(pc)\n      contextFut.map { ctx => (pc, ctx) }\n    })\n    candidatesAndContextsFut.flatMap { candidatesAndContexts =>\n      val contexts = candidatesAndContexts.map(_._2)\n      val resultsFut = featureHydrator.hydrateCandidate(contexts, mrRequestContextForFeatureStore)\n      resultsFut.map { hydrationResult =>\n        candidatesAndContexts.foreach {\n          case (pushCandidate, context) =>\n            val resultFeatures = hydrationResult.getOrElse(context, FeatureMap())\n            pushCandidate.mergeFeatures(resultFeatures)\n        }\n      }\n    }\n  }\n\n  def candidateFeatureHydration(\n    candidateDetails: Seq[CandidateDetails[PushCandidate]],\n    mrRequestContextForFeatureStore: MrRequestContextForFeatureStore\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    candidateDetails.headOption match {\n      case Some(cand) =>\n        val target = cand.candidate.target\n        MrUserStateUtil.updateMrUserStateStats(target)(mrUserStateStat)\n        if (target.params(PushParams.DisableAllRelevanceParam)) {\n          Future.value(candidateDetails)\n        } else {\n          featureHydrationEnabledCounter.incr()\n          for {\n            _ <- hydrateFromRelevanceHydrator(candidateDetails, mrRequestContextForFeatureStore)\n          } yield {\n            candidateDetails\n          }\n        }\n      case _ => Future.Nil\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/RFPHPrerankFilter.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.predicate.PreRankingPredicates\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.SequentialPredicate\nimport com.twitter.util._\n\nclass RFPHPrerankFilter(\n)(\n  globalStats: StatsReceiver) {\n  def filter(\n    target: Target,\n    hydratedCandidates: Seq[CandidateDetails[PushCandidate]]\n  ): Future[\n    (Seq[CandidateDetails[PushCandidate]], Seq[CandidateResult[PushCandidate, Result]])\n  ] = {\n    lazy val filterStats: StatsReceiver = globalStats.scope(\"RefreshForPushHandler/filter\")\n    lazy val okFilterCounter: Counter = filterStats.counter(\"ok\")\n    lazy val invalidFilterCounter: Counter = filterStats.counter(\"invalid\")\n    lazy val invalidFilterStat: StatsReceiver = filterStats.scope(\"invalid\")\n    lazy val invalidFilterReasonStat: StatsReceiver = invalidFilterStat.scope(\"reason\")\n    val allCandidatesFilteredPreRank = filterStats.counter(\"all_candidates_filtered\")\n\n    lazy val preRankingPredicates = PreRankingPredicates(\n      filterStats.scope(\"predicates\")\n    )\n\n    lazy val preRankingPredicateChain =\n      new SequentialPredicate[PushCandidate](preRankingPredicates)\n\n    val predicateChain = if (target.pushContext.exists(_.predicatesToEnable.exists(_.nonEmpty))) {\n      val predicatesToEnable = target.pushContext.flatMap(_.predicatesToEnable).getOrElse(Nil)\n      new SequentialPredicate[PushCandidate](preRankingPredicates.filter { pred =>\n        predicatesToEnable.contains(pred.name)\n      })\n    } else preRankingPredicateChain\n\n    predicateChain\n      .track(hydratedCandidates.map(_.candidate))\n      .map { results =>\n        val resultForPreRankFiltering = results\n          .zip(hydratedCandidates)\n          .foldLeft(\n            (\n              Seq.empty[CandidateDetails[PushCandidate]],\n              Seq.empty[CandidateResult[PushCandidate, Result]]\n            )\n          ) {\n            case ((goodCandidates, filteredCandidates), (result, candidateDetails)) =>\n              result match {\n                case None =>\n                  okFilterCounter.incr()\n                  (goodCandidates :+ candidateDetails, filteredCandidates)\n\n                case Some(pred: NamedPredicate[_]) =>\n                  invalidFilterCounter.incr()\n                  invalidFilterReasonStat.counter(pred.name).incr()\n                  invalidFilterReasonStat\n                    .scope(candidateDetails.candidate.commonRecType.toString).counter(\n                      pred.name).incr()\n\n                  val r = Invalid(Some(pred.name))\n                  (\n                    goodCandidates,\n                    filteredCandidates :+ CandidateResult[PushCandidate, Result](\n                      candidateDetails.candidate,\n                      candidateDetails.source,\n                      r\n                    )\n                  )\n                case Some(_) =>\n                  invalidFilterCounter.incr()\n                  invalidFilterReasonStat.counter(\"unknown\").incr()\n                  invalidFilterReasonStat\n                    .scope(candidateDetails.candidate.commonRecType.toString).counter(\n                      \"unknown\").incr()\n\n                  val r = Invalid(Some(\"Filtered by un-named predicate\"))\n                  (\n                    goodCandidates,\n                    filteredCandidates :+ CandidateResult[PushCandidate, Result](\n                      candidateDetails.candidate,\n                      candidateDetails.source,\n                      r\n                    )\n                  )\n              }\n          }\n\n        resultForPreRankFiltering match {\n          case (validCandidates, _) if validCandidates.isEmpty && hydratedCandidates.nonEmpty =>\n            allCandidatesFilteredPreRank.incr()\n          case _ => ()\n        }\n\n        resultForPreRankFiltering\n      }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/RFPHRestrictStep.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler\n\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.TargetUser\nimport com.twitter.frigate.common.candidate.TargetABDecider\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.target.TargetScoringDetails\n\nclass RFPHRestrictStep()(implicit stats: StatsReceiver) {\n\n  private val statsReceiver: StatsReceiver = stats.scope(\"RefreshForPushHandler\")\n  private val restrictStepStats: StatsReceiver = statsReceiver.scope(\"restrict\")\n  private val restrictStepNumCandidatesDroppedStat: Stat =\n    restrictStepStats.stat(\"candidates_dropped\")\n\n  /**\n   * Limit the number of candidates that enter the Take step\n   */\n  def restrict(\n    target: TargetUser with TargetABDecider with TargetScoringDetails,\n    candidates: Seq[CandidateDetails[PushCandidate]]\n  ): (Seq[CandidateDetails[PushCandidate]], Seq[CandidateDetails[PushCandidate]]) = {\n    if (target.params(PushFeatureSwitchParams.EnableRestrictStep)) {\n      val restrictSizeParam = PushFeatureSwitchParams.RestrictStepSize\n      val (newCandidates, filteredCandidates) = candidates.splitAt(target.params(restrictSizeParam))\n      val numDropped = candidates.length - newCandidates.length\n      restrictStepNumCandidatesDroppedStat.add(numDropped)\n      (newCandidates, filteredCandidates)\n    } else (candidates, Seq.empty)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/RFPHStatsRecorder.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler\n\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\n\nclass RFPHStatsRecorder(implicit statsReceiver: StatsReceiver) {\n\n  private val selectedCandidateScoreStats: StatsReceiver =\n    statsReceiver.scope(\"score_of_sent_candidate_times_10000\")\n\n  private val emptyScoreStats: StatsReceiver =\n    statsReceiver.scope(\"score_of_sent_candidate_empty\")\n\n  def trackPredictionScoreStats(candidate: PushCandidate): Unit = {\n    candidate.mrWeightedOpenOrNtabClickRankingProbability.foreach {\n      case Some(s) =>\n        selectedCandidateScoreStats\n          .stat(\"weighted_open_or_ntab_click_ranking\")\n          .add((s * 10000).toFloat)\n      case None =>\n        emptyScoreStats.counter(\"weighted_open_or_ntab_click_ranking\").incr()\n    }\n    candidate.mrWeightedOpenOrNtabClickFilteringProbability.foreach {\n      case Some(s) =>\n        selectedCandidateScoreStats\n          .stat(\"weighted_open_or_ntab_click_filtering\")\n          .add((s * 10000).toFloat)\n      case None =>\n        emptyScoreStats.counter(\"weighted_open_or_ntab_click_filtering\").incr()\n    }\n    candidate.mrWeightedOpenOrNtabClickRankingProbability.foreach {\n      case Some(s) =>\n        selectedCandidateScoreStats\n          .scope(candidate.commonRecType.toString)\n          .stat(\"weighted_open_or_ntab_click_ranking\")\n          .add((s * 10000).toFloat)\n      case None =>\n        emptyScoreStats\n          .scope(candidate.commonRecType.toString)\n          .counter(\"weighted_open_or_ntab_click_ranking\")\n          .incr()\n    }\n  }\n\n  def refreshRequestExceptionStats(\n    exception: Throwable,\n    bStats: StatsReceiver\n  ): Unit = {\n    bStats.counter(\"failures\").incr()\n    bStats.scope(\"failures\").counter(exception.getClass.getCanonicalName).incr()\n  }\n\n  def loggedOutRequestExceptionStats(\n    exception: Throwable,\n    bStats: StatsReceiver\n  ): Unit = {\n    bStats.counter(\"logged_out_failures\").incr()\n    bStats.scope(\"failures\").counter(exception.getClass.getCanonicalName).incr()\n  }\n\n  def rankDistributionStats(\n    candidatesDetails: Seq[CandidateDetails[PushCandidate]],\n    numRecsPerTypeStat: (CommonRecommendationType => Stat)\n  ): Unit = {\n    candidatesDetails\n      .groupBy { c =>\n        c.candidate.commonRecType\n      }\n      .mapValues { s =>\n        s.size\n      }\n      .foreach { case (crt, numRecs) => numRecsPerTypeStat(crt).add(numRecs) }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/RefreshForPushHandler.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.Stats.track\nimport com.twitter.frigate.common.base.Stats.trackSeq\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.logger.MRLogger\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.adaptor._\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.rank.RFPHLightRanker\nimport com.twitter.frigate.pushservice.rank.RFPHRanker\nimport com.twitter.frigate.pushservice.scriber.MrRequestScribeHandler\nimport com.twitter.frigate.pushservice.take.candidate_validator.RFPHCandidateValidator\nimport com.twitter.frigate.pushservice.target.PushTargetUserBuilder\nimport com.twitter.frigate.pushservice.target.RFPHTargetPredicates\nimport com.twitter.frigate.pushservice.util.RFPHTakeStepUtil\nimport com.twitter.frigate.pushservice.util.AdhocStatsUtil\nimport com.twitter.frigate.pushservice.thriftscala.PushContext\nimport com.twitter.frigate.pushservice.thriftscala.RefreshRequest\nimport com.twitter.frigate.pushservice.thriftscala.RefreshResponse\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.timelines.configapi.FeatureValue\nimport com.twitter.util._\n\ncase class ResultWithDebugInfo(result: Result, predicateResults: Seq[PredicateWithResult])\n\nclass RefreshForPushHandler(\n  val pushTargetUserBuilder: PushTargetUserBuilder,\n  val candSourceGenerator: PushCandidateSourceGenerator,\n  rfphRanker: RFPHRanker,\n  candidateHydrator: PushCandidateHydrator,\n  candidateValidator: RFPHCandidateValidator,\n  rfphTakeStepUtil: RFPHTakeStepUtil,\n  rfphRestrictStep: RFPHRestrictStep,\n  val rfphNotifier: RefreshForPushNotifier,\n  rfphStatsRecorder: RFPHStatsRecorder,\n  mrRequestScriberNode: String,\n  rfphFeatureHydrator: RFPHFeatureHydrator,\n  rfphPrerankFilter: RFPHPrerankFilter,\n  rfphLightRanker: RFPHLightRanker\n)(\n  globalStats: StatsReceiver)\n    extends FetchRankFlowWithHydratedCandidates[Target, RawCandidate, PushCandidate] {\n\n  val log = MRLogger(\"RefreshForPushHandler\")\n\n  implicit val statsReceiver: StatsReceiver =\n    globalStats.scope(\"RefreshForPushHandler\")\n  private val maxCandidatesToBatchInTakeStat: Stat =\n    statsReceiver.stat(\"max_cands_to_batch_in_take\")\n\n  private val rfphRequestCounter = statsReceiver.counter(\"requests\")\n\n  private val buildTargetStats = statsReceiver.scope(\"build_target\")\n  private val processStats = statsReceiver.scope(\"process\")\n  private val notifyStats = statsReceiver.scope(\"notify\")\n\n  private val lightRankingStats: StatsReceiver = statsReceiver.scope(\"light_ranking\")\n  private val reRankingStats: StatsReceiver = statsReceiver.scope(\"rerank\")\n  private val featureHydrationLatency: StatsReceiver =\n    statsReceiver.scope(\"featureHydrationLatency\")\n  private val candidateHydrationStats: StatsReceiver = statsReceiver.scope(\"candidate_hydration\")\n\n  lazy val candSourceEligibleCounter: Counter =\n    candidateStats.counter(\"cand_source_eligible\")\n  lazy val candSourceNotEligibleCounter: Counter =\n    candidateStats.counter(\"cand_source_not_eligible\")\n\n  //pre-ranking stats\n  val allCandidatesFilteredPreRank = filterStats.counter(\"all_candidates_filtered\")\n\n  // total invalid candidates\n  val totalStats: StatsReceiver = statsReceiver.scope(\"total\")\n  val totalInvalidCandidatesStat: Stat = totalStats.stat(\"candidates_invalid\")\n\n  val mrRequestScribeBuiltStats: Counter = statsReceiver.counter(\"mr_request_scribe_built\")\n\n  val mrRequestCandidateScribeStats = statsReceiver.scope(\"mr_request_scribe_candidates\")\n  val mrRequestTargetScribeStats = statsReceiver.scope(\"mr_request_scribe_target\")\n\n  val mrRequestScribeHandler =\n    new MrRequestScribeHandler(mrRequestScriberNode, statsReceiver.scope(\"mr_request_scribe\"))\n\n  val adhocStatsUtil = new AdhocStatsUtil(statsReceiver.scope(\"adhoc_stats\"))\n\n  private def numRecsPerTypeStat(crt: CommonRecommendationType) =\n    fetchStats.scope(crt.toString).stat(\"dist\")\n\n  // static list of target predicates\n  private val targetPredicates = RFPHTargetPredicates(targetStats.scope(\"predicates\"))\n\n  def buildTarget(\n    userId: Long,\n    inputPushContext: Option[PushContext],\n    forcedFeatureValues: Option[Map[String, FeatureValue]] = None\n  ): Future[Target] =\n    pushTargetUserBuilder.buildTarget(userId, inputPushContext, forcedFeatureValues)\n\n  override def targetPredicates(target: Target): List[Predicate[Target]] = targetPredicates\n\n  override def isTargetValid(target: Target): Future[Result] = {\n    val resultFut = if (target.skipFilters) {\n      Future.value(trackTargetPredStats(None))\n    } else {\n      predicateSeq(target).track(Seq(target)).map { resultArr =>\n        trackTargetPredStats(resultArr(0))\n      }\n    }\n    track(targetStats)(resultFut)\n  }\n\n  override def candidateSources(\n    target: Target\n  ): Future[Seq[CandidateSource[Target, RawCandidate]]] = {\n    Future\n      .collect(candSourceGenerator.sources.map { cs =>\n        cs.isCandidateSourceAvailable(target).map { isEligible =>\n          if (isEligible) {\n            candSourceEligibleCounter.incr()\n            Some(cs)\n          } else {\n            candSourceNotEligibleCounter.incr()\n            None\n          }\n        }\n      }).map(_.flatten)\n  }\n\n  override def updateCandidateCounter(\n    candidateResults: Seq[CandidateResult[PushCandidate, Result]]\n  ): Unit = {\n    candidateResults.foreach {\n      case candidateResult if candidateResult.result == OK =>\n        okCandidateCounter.incr()\n      case candidateResult if candidateResult.result.isInstanceOf[Invalid] =>\n        invalidCandidateCounter.incr()\n      case _ =>\n    }\n  }\n\n  override def hydrateCandidates(\n    candidates: Seq[CandidateDetails[RawCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = candidateHydrator(candidates)\n\n  override def filter(\n    target: Target,\n    hydratedCandidates: Seq[CandidateDetails[PushCandidate]]\n  ): Future[\n    (Seq[CandidateDetails[PushCandidate]], Seq[CandidateResult[PushCandidate, Result]])\n  ] = rfphPrerankFilter.filter(target, hydratedCandidates)\n\n  def lightRankAndTake(\n    target: Target,\n    candidates: Seq[CandidateDetails[PushCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    rfphLightRanker.rank(target, candidates)\n  }\n\n  override def rank(\n    target: Target,\n    candidatesDetails: Seq[CandidateDetails[PushCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    val featureHydratedCandidatesFut = trackSeq(featureHydrationLatency)(\n      rfphFeatureHydrator\n        .candidateFeatureHydration(candidatesDetails, target.mrRequestContextForFeatureStore)\n    )\n    featureHydratedCandidatesFut.flatMap { featureHydratedCandidates =>\n      rfphStatsRecorder.rankDistributionStats(featureHydratedCandidates, numRecsPerTypeStat)\n      rfphRanker.initialRank(target, candidatesDetails)\n    }\n  }\n\n  def reRank(\n    target: Target,\n    rankedCandidates: Seq[CandidateDetails[PushCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    rfphRanker.reRank(target, rankedCandidates)\n  }\n\n  override def validCandidates(\n    target: Target,\n    candidates: Seq[PushCandidate]\n  ): Future[Seq[Result]] = {\n    Future.collect(candidates.map { candidate =>\n      rfphTakeStepUtil.isCandidateValid(candidate, candidateValidator).map(res => res.result)\n    })\n  }\n\n  override def desiredCandidateCount(target: Target): Int = target.desiredCandidateCount\n\n  override def batchForCandidatesCheck(target: Target): Int = {\n    val fsParam = PushFeatureSwitchParams.NumberOfMaxCandidatesToBatchInRFPHTakeStep\n    val maxToBatch = target.params(fsParam)\n    maxCandidatesToBatchInTakeStat.add(maxToBatch)\n    maxToBatch\n  }\n\n  override def process(\n    target: Target,\n    externalCandidates: Seq[RawCandidate] = Nil\n  ): Future[Response[PushCandidate, Result]] = {\n    isTargetValid(target).flatMap {\n      case OK =>\n        for {\n          candidatesFromSources <- trackSeq(fetchStats)(fetchCandidates(target))\n          externalCandidateDetails = externalCandidates.map(\n            CandidateDetails(_, \"refresh_for_push_handler_external_candidate\"))\n          allCandidates = candidatesFromSources ++ externalCandidateDetails\n          hydratedCandidatesWithCopy <-\n            trackSeq(candidateHydrationStats)(hydrateCandidates(allCandidates))\n          _ = adhocStatsUtil.getCandidateSourceStats(hydratedCandidatesWithCopy)\n          (candidates, preRankingFilteredCandidates) <-\n            track(filterStats)(filter(target, hydratedCandidatesWithCopy))\n          _ = adhocStatsUtil.getPreRankingFilterStats(preRankingFilteredCandidates)\n          lightRankerFilteredCandidates <-\n            trackSeq(lightRankingStats)(lightRankAndTake(target, candidates))\n          _ = adhocStatsUtil.getLightRankingStats(lightRankerFilteredCandidates)\n          rankedCandidates <- trackSeq(rankingStats)(rank(target, lightRankerFilteredCandidates))\n          _ = adhocStatsUtil.getRankingStats(rankedCandidates)\n          rerankedCandidates <- trackSeq(reRankingStats)(reRank(target, rankedCandidates))\n          _ = adhocStatsUtil.getReRankingStats(rerankedCandidates)\n          (restrictedCandidates, restrictFilteredCandidates) =\n            rfphRestrictStep.restrict(target, rerankedCandidates)\n          allTakeCandidateResults <- track(takeStats)(\n            take(target, restrictedCandidates, desiredCandidateCount(target))\n          )\n          _ = adhocStatsUtil.getTakeCandidateResultStats(allTakeCandidateResults)\n          _ <- track(mrRequestCandidateScribeStats)(\n            mrRequestScribeHandler.scribeForCandidateFiltering(\n              target,\n              hydratedCandidatesWithCopy,\n              preRankingFilteredCandidates,\n              rankedCandidates,\n              rerankedCandidates,\n              restrictFilteredCandidates,\n              allTakeCandidateResults\n            ))\n        } yield {\n\n          /**\n           * Take processes post restrict step candidates and returns both:\n           *  1. valid + invalid candidates\n           *  2. Candidates that are not processed (more than desired) + restricted candidates\n           * We need #2 only for importance sampling\n           */\n          val takeCandidateResults =\n            allTakeCandidateResults.filterNot { candResult =>\n              candResult.result == MoreThanDesiredCandidates\n            }\n\n          val totalInvalidCandidates = {\n            preRankingFilteredCandidates.size + //pre-ranking filtered candidates\n              (rerankedCandidates.length - restrictedCandidates.length) + //candidates reject in restrict step\n              takeCandidateResults.count(_.result != OK) //candidates reject in take step\n          }\n          takeInvalidCandidateDist.add(\n            takeCandidateResults\n              .count(_.result != OK)\n          ) // take step invalid candidates\n          totalInvalidCandidatesStat.add(totalInvalidCandidates)\n          val allCandidateResults = takeCandidateResults ++ preRankingFilteredCandidates\n          Response(OK, allCandidateResults)\n        }\n\n      case result: Result =>\n        for (_ <- track(mrRequestTargetScribeStats)(\n            mrRequestScribeHandler.scribeForTargetFiltering(target, result))) yield {\n          mrRequestScribeBuiltStats.incr()\n          Response(result, Nil)\n        }\n    }\n  }\n\n  def refreshAndSend(request: RefreshRequest): Future[RefreshResponse] = {\n    rfphRequestCounter.incr()\n    for {\n      target <- track(buildTargetStats)(\n        pushTargetUserBuilder\n          .buildTarget(request.userId, request.context))\n      response <- track(processStats)(process(target, externalCandidates = Seq.empty))\n      refreshResponse <- track(notifyStats)(rfphNotifier.checkResponseAndNotify(response, target))\n    } yield {\n      refreshResponse\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/RefreshForPushNotifier.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler\n\nimport com.twitter.finagle.stats.BroadcastStatsReceiver\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.Stats.track\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.config.CommonConstants\nimport com.twitter.frigate.common.util.PushServiceUtil.FilteredRefreshResponseFut\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.take.CandidateNotifier\nimport com.twitter.frigate.pushservice.util.ResponseStatsTrackUtils.trackStatsForResponseToRequest\nimport com.twitter.frigate.pushservice.thriftscala.PushStatus\nimport com.twitter.frigate.pushservice.thriftscala.RefreshResponse\nimport com.twitter.util.Future\nimport com.twitter.util.JavaTimer\nimport com.twitter.util.Timer\n\nclass RefreshForPushNotifier(\n  rfphStatsRecorder: RFPHStatsRecorder,\n  candidateNotifier: CandidateNotifier\n)(\n  globalStats: StatsReceiver) {\n\n  private implicit val statsReceiver: StatsReceiver =\n    globalStats.scope(\"RefreshForPushHandler\")\n\n  private val pushStats: StatsReceiver = statsReceiver.scope(\"push\")\n  private val sendLatency: StatsReceiver = statsReceiver.scope(\"send_handler\")\n  implicit private val timer: Timer = new JavaTimer(true)\n\n  private def notify(\n    candidatesResult: CandidateResult[PushCandidate, Result],\n    target: Target,\n    receivers: Seq[StatsReceiver]\n  ): Future[RefreshResponse] = {\n\n    val candidate = candidatesResult.candidate\n\n    val predsResult = candidatesResult.result\n\n    if (predsResult != OK) {\n      val invalidResult = predsResult\n      invalidResult match {\n        case Invalid(Some(reason)) =>\n          Future.value(RefreshResponse(PushStatus.Filtered, Some(reason)))\n        case _ =>\n          Future.value(RefreshResponse(PushStatus.Filtered, None))\n      }\n    } else {\n      rfphStatsRecorder.trackPredictionScoreStats(candidate)\n\n      val isQualityUprankingCandidate = candidate.mrQualityUprankingBoost.isDefined\n      val commonRecTypeStats = Seq(\n        statsReceiver.scope(candidate.commonRecType.toString),\n        globalStats.scope(candidate.commonRecType.toString)\n      )\n      val qualityUprankingStats = Seq(\n        statsReceiver.scope(\"QualityUprankingCandidates\").scope(candidate.commonRecType.toString),\n        globalStats.scope(\"QualityUprankingCandidates\").scope(candidate.commonRecType.toString)\n      )\n\n      val receiversWithRecTypeStats = {\n        if (isQualityUprankingCandidate) {\n          receivers ++ commonRecTypeStats ++ qualityUprankingStats\n        } else {\n          receivers ++ commonRecTypeStats\n        }\n      }\n      track(sendLatency)(candidateNotifier.notify(candidate).map { res =>\n        trackStatsForResponseToRequest(\n          candidate.commonRecType,\n          candidate.target,\n          res,\n          receiversWithRecTypeStats\n        )(globalStats)\n        RefreshResponse(res.status)\n      })\n    }\n  }\n\n  def checkResponseAndNotify(\n    response: Response[PushCandidate, Result],\n    targetUserContext: Target\n  ): Future[RefreshResponse] = {\n    val receivers = Seq(statsReceiver)\n    val refreshResponse = response match {\n      case Response(OK, processedCandidates) =>\n        // valid rec candidates\n        val validCandidates = processedCandidates.filter(_.result == OK)\n\n        // top rec candidate\n        validCandidates.headOption match {\n          case Some(candidatesResult) =>\n            candidatesResult.result match {\n              case OK =>\n                notify(candidatesResult, targetUserContext, receivers)\n                  .onSuccess { nr =>\n                    pushStats.scope(\"result\").counter(nr.status.name).incr()\n                  }\n              case _ =>\n                targetUserContext.isTeamMember.flatMap { isTeamMember =>\n                  FilteredRefreshResponseFut\n                }\n            }\n          case _ =>\n            FilteredRefreshResponseFut\n        }\n      case Response(Invalid(reason), _) =>\n        // invalid target with known reason\n        FilteredRefreshResponseFut.map(_.copy(targetFilteredBy = reason))\n      case _ =>\n        // invalid target\n        FilteredRefreshResponseFut\n    }\n\n    val bStats = BroadcastStatsReceiver(receivers)\n    Stat\n      .timeFuture(bStats.stat(\"latency\"))(\n        refreshResponse\n          .raiseWithin(CommonConstants.maxPushRequestDuration)\n      )\n      .onFailure { exception =>\n        rfphStatsRecorder.refreshRequestExceptionStats(exception, bStats)\n      }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/cross/BaseCopyFramework.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler.cross\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.MRNtabCopy\nimport com.twitter.frigate.common.util.MRPushCopy\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.util.Future\n\nabstract class BaseCopyFramework(statsReceiver: StatsReceiver) {\n\n  private val NoAvailableCopyStat = statsReceiver.scope(\"no_copy_for_crt\")\n  private val NoAvailableNtabCopyStat = statsReceiver.scope(\"no_ntab_copy\")\n\n  /**\n   * Instantiate push copy filters\n   */\n  protected final val copyFilters = new CopyFilters(statsReceiver.scope(\"filters\"))\n\n  /**\n   *\n   * The following method fetches all the push copies for a [[com.twitter.frigate.thriftscala.CommonRecommendationType]]\n   * associated with a candidate and then filters the eligible copies based on\n   * [[PushTypes.PushCandidate]] features. These filters are defined in\n   * [[CopyFilters]]\n   *\n   * @param rawCandidate - [[RawCandidate]] object representing a recommendation candidate\n   *\n   * @return - set of eligible push copies for a given candidate\n   */\n  protected[cross] final def getEligiblePushCopiesFromCandidate(\n    rawCandidate: RawCandidate\n  ): Future[Seq[MRPushCopy]] = {\n    val pushCopiesFromRectype = CandidateToCopy.getPushCopiesFromRectype(rawCandidate.commonRecType)\n\n    if (pushCopiesFromRectype.isEmpty) {\n      NoAvailableCopyStat.counter(rawCandidate.commonRecType.name).incr()\n      throw new IllegalStateException(s\"No Copy defined for CRT: \" + rawCandidate.commonRecType)\n    }\n    pushCopiesFromRectype\n      .map(pushCopySet => copyFilters.execute(rawCandidate, pushCopySet.toSeq))\n      .getOrElse(Future.value(Seq.empty))\n  }\n\n  /**\n   *\n   * This method essentially forms the base for cross-step for the MagicRecs Copy Framework. Given\n   * a recommendation type this returns a set of tuples wherein each tuple is a pair of push and\n   * ntab copy eligible for the said recommendation type\n   *\n   * @param rawCandidate - [[RawCandidate]] object representing a recommendation candidate\n   * @return    - Set of eligible [[MRPushCopy]], Option[[MRNtabCopy]] for a given recommendation type\n   */\n  protected[cross] final def getEligiblePushAndNtabCopiesFromCandidate(\n    rawCandidate: RawCandidate\n  ): Future[Seq[(MRPushCopy, Option[MRNtabCopy])]] = {\n\n    val eligiblePushCopies = getEligiblePushCopiesFromCandidate(rawCandidate)\n\n    eligiblePushCopies.map { pushCopies =>\n      val setBuilder = Set.newBuilder[(MRPushCopy, Option[MRNtabCopy])]\n      pushCopies.foreach { pushCopy =>\n        val ntabCopies = CandidateToCopy.getNtabcopiesFromPushcopy(pushCopy)\n        val pushNtabCopyPairs = ntabCopies match {\n          case Some(ntabCopySet) =>\n            if (ntabCopySet.isEmpty) {\n              NoAvailableNtabCopyStat.counter(s\"copy_id: ${pushCopy.copyId}\").incr()\n              Set(pushCopy -> None)\n            } // push copy only\n            else ntabCopySet.map(pushCopy -> Some(_))\n\n          case None =>\n            Set.empty[(MRPushCopy, Option[MRNtabCopy])] // no push or ntab copy\n        }\n        setBuilder ++= pushNtabCopyPairs\n      }\n      setBuilder.result().toSeq\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/cross/CandidateCopyExpansion.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler.cross\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.util.MRNtabCopy\nimport com.twitter.frigate.common.util.MRPushCopy\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.util.Future\n\n/**\n * @param statsReceiver - stats receiver object\n */\nclass CandidateCopyExpansion(statsReceiver: StatsReceiver)\n    extends BaseCopyFramework(statsReceiver) {\n\n  /**\n   *\n   * Given a [[CandidateDetails]] object representing a push recommendation candidate this method\n   * expands it to multiple candidates, each tagged with a push copy id and ntab copy id to\n   * represent the eligible copies for the given recommendation candidate\n   *\n   * @param candidateDetails - [[CandidateDetails]] objects containing a recommendation candidate\n   *\n   * @return - list of tuples of [[PushTypes.RawCandidate]] and [[CopyIds]]\n   */\n  private final def crossCandidateDetailsWithCopyId(\n    candidateDetails: CandidateDetails[RawCandidate]\n  ): Future[Seq[(CandidateDetails[RawCandidate], CopyIds)]] = {\n    val eligibleCopyPairs = getEligiblePushAndNtabCopiesFromCandidate(candidateDetails.candidate)\n    val copyPairs = eligibleCopyPairs.map(_.map {\n      case (pushCopy: MRPushCopy, ntabCopy: Option[MRNtabCopy]) =>\n        CopyIds(\n          pushCopyId = Some(pushCopy.copyId),\n          ntabCopyId = ntabCopy.map(_.copyId)\n        )\n    })\n\n    copyPairs.map(_.map((candidateDetails, _)))\n  }\n\n  /**\n   *\n   * This method takes as input a list of [[CandidateDetails]] objects which contain the push\n   * recommendation candidates for a given target user. It expands each input candidate into\n   * multiple candidates, each tagged with a push copy id and ntab copy id to represent the eligible\n   * copies for the given recommendation candidate\n   *\n   * @param candidateDetailsSeq - list of fetched candidates for push recommendation\n   * @return - list of tuples of [[RawCandidate]] and [[CopyIds]]\n   */\n  final def expandCandidatesWithCopyId(\n    candidateDetailsSeq: Seq[CandidateDetails[RawCandidate]]\n  ): Future[Seq[(CandidateDetails[RawCandidate], CopyIds)]] =\n    Future.collect(candidateDetailsSeq.map(crossCandidateDetailsWithCopyId)).map(_.flatten)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/cross/CandidateCopyPair.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler.cross\n\nimport com.twitter.frigate.common.util.MRPushCopy\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\n\n/**\n *\n * @param candidate: [[RawCandidate]] is a recommendation candidate\n * @param pushCopy: [[MRPushCopy]] eligible for candidate\n */\ncase class CandidateCopyPair(candidate: RawCandidate, pushCopy: MRPushCopy)\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/cross/CandidateToCopy.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler.cross\n\nimport com.twitter.frigate.common.util.MrNtabCopyObjects\nimport com.twitter.frigate.common.util.MrPushCopyObjects\nimport com.twitter.frigate.common.util._\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.CommonRecommendationType._\n\nobject CandidateToCopy {\n\n  // Static map from a CommonRecommendationType to set of eligible push notification copies\n  private[cross] val rectypeToPushCopy: Map[CommonRecommendationType, Set[\n    MRPushCopy\n  ]] =\n    Map[CommonRecommendationType, Set[MRPushCopy]](\n      F1FirstdegreeTweet -> Set(\n        MrPushCopyObjects.FirstDegreeJustTweetedBoldTitle\n      ),\n      F1FirstdegreePhoto -> Set(\n        MrPushCopyObjects.FirstDegreePhotoJustTweetedBoldTitle\n      ),\n      F1FirstdegreeVideo -> Set(\n        MrPushCopyObjects.FirstDegreeVideoJustTweetedBoldTitle\n      ),\n      TweetRetweet -> Set(\n        MrPushCopyObjects.TweetRetweetWithOneDisplaySocialContextsWithText,\n        MrPushCopyObjects.TweetRetweetWithTwoDisplaySocialContextsWithText,\n        MrPushCopyObjects.TweetRetweetWithOneDisplayAndKOtherSocialContextsWithText\n      ),\n      TweetRetweetPhoto -> Set(\n        MrPushCopyObjects.TweetRetweetPhotoWithOneDisplaySocialContextWithText,\n        MrPushCopyObjects.TweetRetweetPhotoWithTwoDisplaySocialContextsWithText,\n        MrPushCopyObjects.TweetRetweetPhotoWithOneDisplayAndKOtherSocialContextsWithText\n      ),\n      TweetRetweetVideo -> Set(\n        MrPushCopyObjects.TweetRetweetVideoWithOneDisplaySocialContextWithText,\n        MrPushCopyObjects.TweetRetweetVideoWithTwoDisplaySocialContextsWithText,\n        MrPushCopyObjects.TweetRetweetVideoWithOneDisplayAndKOtherSocialContextsWithText\n      ),\n      TweetFavorite -> Set(\n        MrPushCopyObjects.TweetLikeOneSocialContextWithText,\n        MrPushCopyObjects.TweetLikeTwoSocialContextWithText,\n        MrPushCopyObjects.TweetLikeMultipleSocialContextWithText\n      ),\n      TweetFavoritePhoto -> Set(\n        MrPushCopyObjects.TweetLikePhotoOneSocialContextWithText,\n        MrPushCopyObjects.TweetLikePhotoTwoSocialContextWithText,\n        MrPushCopyObjects.TweetLikePhotoMultipleSocialContextWithText\n      ),\n      TweetFavoriteVideo -> Set(\n        MrPushCopyObjects.TweetLikeVideoOneSocialContextWithText,\n        MrPushCopyObjects.TweetLikeVideoTwoSocialContextWithText,\n        MrPushCopyObjects.TweetLikeVideoMultipleSocialContextWithText\n      ),\n      UnreadBadgeCount -> Set(MrPushCopyObjects.UnreadBadgeCount),\n      InterestBasedTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      InterestBasedPhoto -> Set(MrPushCopyObjects.RecommendedForYouPhoto),\n      InterestBasedVideo -> Set(MrPushCopyObjects.RecommendedForYouVideo),\n      UserFollow -> Set(\n        MrPushCopyObjects.UserFollowWithOneSocialContext,\n        MrPushCopyObjects.UserFollowWithTwoSocialContext,\n        MrPushCopyObjects.UserFollowOneDisplayAndKOtherSocialContext\n      ),\n      HermitUser -> Set(\n        MrPushCopyObjects.HermitUserWithOneSocialContext,\n        MrPushCopyObjects.HermitUserWithTwoSocialContext,\n        MrPushCopyObjects.HermitUserWithOneDisplayAndKOtherSocialContexts\n      ),\n      TriangularLoopUser -> Set(\n        MrPushCopyObjects.TriangularLoopUserWithOneSocialContext,\n        MrPushCopyObjects.TriangularLoopUserWithTwoSocialContexts,\n        MrPushCopyObjects.TriangularLoopUserOneDisplayAndKotherSocialContext\n      ),\n      ForwardAddressbookUserFollow -> Set(MrPushCopyObjects.ForwardAddressBookUserFollow),\n      NewsArticleNewsLanding -> Set(MrPushCopyObjects.NewsArticleNewsLandingCopy),\n      TopicProofTweet -> Set(MrPushCopyObjects.TopicProofTweet),\n      UserInterestinTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      UserInterestinPhoto -> Set(MrPushCopyObjects.RecommendedForYouPhoto),\n      UserInterestinVideo -> Set(MrPushCopyObjects.RecommendedForYouVideo),\n      TwistlyTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      TwistlyPhoto -> Set(MrPushCopyObjects.RecommendedForYouPhoto),\n      TwistlyVideo -> Set(MrPushCopyObjects.RecommendedForYouVideo),\n      ElasticTimelineTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      ElasticTimelinePhoto -> Set(MrPushCopyObjects.RecommendedForYouPhoto),\n      ElasticTimelineVideo -> Set(MrPushCopyObjects.RecommendedForYouVideo),\n      ExploreVideoTweet -> Set(MrPushCopyObjects.ExploreVideoTweet),\n      List -> Set(MrPushCopyObjects.ListRecommendation),\n      InterestBasedUserFollow -> Set(MrPushCopyObjects.UserFollowInterestBasedCopy),\n      PastEmailEngagementTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      PastEmailEngagementPhoto -> Set(MrPushCopyObjects.RecommendedForYouPhoto),\n      PastEmailEngagementVideo -> Set(MrPushCopyObjects.RecommendedForYouVideo),\n      ExplorePush -> Set(MrPushCopyObjects.ExplorePush),\n      ConnectTabPush -> Set(MrPushCopyObjects.ConnectTabPush),\n      ConnectTabWithUserPush -> Set(MrPushCopyObjects.ConnectTabWithUserPush),\n      AddressBookUploadPush -> Set(MrPushCopyObjects.AddressBookPush),\n      InterestPickerPush -> Set(MrPushCopyObjects.InterestPickerPush),\n      CompleteOnboardingPush -> Set(MrPushCopyObjects.CompleteOnboardingPush),\n      GeoPopTweet -> Set(MrPushCopyObjects.GeoPopPushCopy),\n      TagSpaceTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      FrsTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      TwhinTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      MrModelingBasedTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      DetopicTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      TweetImpressions -> Set(MrPushCopyObjects.TopTweetImpressions),\n      TrendTweet -> Set(MrPushCopyObjects.TrendTweet),\n      ReverseAddressbookTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      ForwardAddressbookTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      SpaceInNetwork -> Set(MrPushCopyObjects.SpaceHost),\n      SpaceOutOfNetwork -> Set(MrPushCopyObjects.SpaceHost),\n      SubscribedSearch -> Set(MrPushCopyObjects.SubscribedSearchTweet),\n      TripGeoTweet -> Set(MrPushCopyObjects.TripGeoTweetPushCopy),\n      CrowdSearchTweet -> Set(MrPushCopyObjects.RecommendedForYouTweet),\n      Digest -> Set(MrPushCopyObjects.Digest),\n      TripHqTweet -> Set(MrPushCopyObjects.TripHqTweetPushCopy)\n    )\n\n  // Static map from a push copy to set of eligible ntab copies\n  private[cross] val pushcopyToNtabcopy: Map[MRPushCopy, Set[MRNtabCopy]] =\n    Map[MRPushCopy, Set[MRNtabCopy]](\n      MrPushCopyObjects.FirstDegreeJustTweetedBoldTitle -> Set(\n        MrNtabCopyObjects.FirstDegreeTweetRecent),\n      MrPushCopyObjects.FirstDegreePhotoJustTweetedBoldTitle -> Set(\n        MrNtabCopyObjects.FirstDegreeTweetRecent\n      ),\n      MrPushCopyObjects.FirstDegreeVideoJustTweetedBoldTitle -> Set(\n        MrNtabCopyObjects.FirstDegreeTweetRecent\n      ),\n      MrPushCopyObjects.TweetRetweetWithOneDisplaySocialContextsWithText -> Set(\n        MrNtabCopyObjects.TweetRetweetWithOneDisplaySocialContext\n      ),\n      MrPushCopyObjects.TweetRetweetWithTwoDisplaySocialContextsWithText -> Set(\n        MrNtabCopyObjects.TweetRetweetWithTwoDisplaySocialContexts\n      ),\n      MrPushCopyObjects.TweetRetweetWithOneDisplayAndKOtherSocialContextsWithText -> Set(\n        MrNtabCopyObjects.TweetRetweetWithOneDisplayAndKOtherSocialContexts\n      ),\n      MrPushCopyObjects.TweetRetweetPhotoWithOneDisplaySocialContextWithText -> Set(\n        MrNtabCopyObjects.TweetRetweetPhotoWithOneDisplaySocialContext\n      ),\n      MrPushCopyObjects.TweetRetweetPhotoWithTwoDisplaySocialContextsWithText -> Set(\n        MrNtabCopyObjects.TweetRetweetPhotoWithTwoDisplaySocialContexts\n      ),\n      MrPushCopyObjects.TweetRetweetPhotoWithOneDisplayAndKOtherSocialContextsWithText -> Set(\n        MrNtabCopyObjects.TweetRetweetPhotoWithOneDisplayAndKOtherSocialContexts\n      ),\n      MrPushCopyObjects.TweetRetweetVideoWithOneDisplaySocialContextWithText -> Set(\n        MrNtabCopyObjects.TweetRetweetVideoWithOneDisplaySocialContext\n      ),\n      MrPushCopyObjects.TweetRetweetVideoWithTwoDisplaySocialContextsWithText -> Set(\n        MrNtabCopyObjects.TweetRetweetVideoWithTwoDisplaySocialContexts\n      ),\n      MrPushCopyObjects.TweetRetweetVideoWithOneDisplayAndKOtherSocialContextsWithText -> Set(\n        MrNtabCopyObjects.TweetRetweetVideoWithOneDisplayAndKOtherSocialContexts\n      ),\n      MrPushCopyObjects.TweetLikeOneSocialContextWithText -> Set(\n        MrNtabCopyObjects.TweetLikeWithOneDisplaySocialContext\n      ),\n      MrPushCopyObjects.TweetLikeTwoSocialContextWithText -> Set(\n        MrNtabCopyObjects.TweetLikeWithTwoDisplaySocialContexts\n      ),\n      MrPushCopyObjects.TweetLikeMultipleSocialContextWithText -> Set(\n        MrNtabCopyObjects.TweetLikeWithOneDisplayAndKOtherSocialContexts\n      ),\n      MrPushCopyObjects.TweetLikePhotoOneSocialContextWithText -> Set(\n        MrNtabCopyObjects.TweetLikePhotoWithOneDisplaySocialContext\n      ),\n      MrPushCopyObjects.TweetLikePhotoTwoSocialContextWithText -> Set(\n        MrNtabCopyObjects.TweetLikePhotoWithTwoDisplaySocialContexts\n      ),\n      MrPushCopyObjects.TweetLikePhotoMultipleSocialContextWithText -> Set(\n        MrNtabCopyObjects.TweetLikePhotoWithOneDisplayAndKOtherSocialContexts\n      ),\n      MrPushCopyObjects.TweetLikeVideoOneSocialContextWithText -> Set(\n        MrNtabCopyObjects.TweetLikeVideoWithOneDisplaySocialContext\n      ),\n      MrPushCopyObjects.TweetLikeVideoTwoSocialContextWithText -> Set(\n        MrNtabCopyObjects.TweetLikeVideoWithTwoDisplaySocialContexts\n      ),\n      MrPushCopyObjects.TweetLikeVideoMultipleSocialContextWithText -> Set(\n        MrNtabCopyObjects.TweetLikeVideoWithOneDisplayAndKOtherSocialContexts\n      ),\n      MrPushCopyObjects.UnreadBadgeCount -> Set.empty[MRNtabCopy],\n      MrPushCopyObjects.RecommendedForYouTweet -> Set(MrNtabCopyObjects.RecommendedForYouCopy),\n      MrPushCopyObjects.RecommendedForYouPhoto -> Set(MrNtabCopyObjects.RecommendedForYouCopy),\n      MrPushCopyObjects.RecommendedForYouVideo -> Set(MrNtabCopyObjects.RecommendedForYouCopy),\n      MrPushCopyObjects.GeoPopPushCopy -> Set(MrNtabCopyObjects.RecommendedForYouCopy),\n      MrPushCopyObjects.UserFollowWithOneSocialContext -> Set(\n        MrNtabCopyObjects.UserFollowWithOneDisplaySocialContext\n      ),\n      MrPushCopyObjects.UserFollowWithTwoSocialContext -> Set(\n        MrNtabCopyObjects.UserFollowWithTwoDisplaySocialContexts\n      ),\n      MrPushCopyObjects.UserFollowOneDisplayAndKOtherSocialContext -> Set(\n        MrNtabCopyObjects.UserFollowWithOneDisplayAndKOtherSocialContexts\n      ),\n      MrPushCopyObjects.HermitUserWithOneSocialContext -> Set(\n        MrNtabCopyObjects.UserFollowWithOneDisplaySocialContext\n      ),\n      MrPushCopyObjects.HermitUserWithTwoSocialContext -> Set(\n        MrNtabCopyObjects.UserFollowWithTwoDisplaySocialContexts\n      ),\n      MrPushCopyObjects.HermitUserWithOneDisplayAndKOtherSocialContexts -> Set(\n        MrNtabCopyObjects.UserFollowWithOneDisplayAndKOtherSocialContexts\n      ),\n      MrPushCopyObjects.TriangularLoopUserWithOneSocialContext -> Set(\n        MrNtabCopyObjects.TriangularLoopUserWithOneSocialContext\n      ),\n      MrPushCopyObjects.TriangularLoopUserWithTwoSocialContexts -> Set(\n        MrNtabCopyObjects.TriangularLoopUserWithTwoSocialContexts\n      ),\n      MrPushCopyObjects.TriangularLoopUserOneDisplayAndKotherSocialContext -> Set(\n        MrNtabCopyObjects.TriangularLoopUserOneDisplayAndKOtherSocialContext\n      ),\n      MrPushCopyObjects.NewsArticleNewsLandingCopy -> Set(\n        MrNtabCopyObjects.NewsArticleNewsLandingCopy\n      ),\n      MrPushCopyObjects.UserFollowInterestBasedCopy -> Set(\n        MrNtabCopyObjects.UserFollowInterestBasedCopy\n      ),\n      MrPushCopyObjects.ForwardAddressBookUserFollow -> Set(\n        MrNtabCopyObjects.ForwardAddressBookUserFollow),\n      MrPushCopyObjects.ConnectTabPush -> Set(\n        MrNtabCopyObjects.ConnectTabPush\n      ),\n      MrPushCopyObjects.ExplorePush -> Set.empty[MRNtabCopy],\n      MrPushCopyObjects.ConnectTabWithUserPush -> Set(\n        MrNtabCopyObjects.UserFollowInterestBasedCopy),\n      MrPushCopyObjects.AddressBookPush -> Set(MrNtabCopyObjects.AddressBook),\n      MrPushCopyObjects.InterestPickerPush -> Set(MrNtabCopyObjects.InterestPicker),\n      MrPushCopyObjects.CompleteOnboardingPush -> Set(MrNtabCopyObjects.CompleteOnboarding),\n      MrPushCopyObjects.TopicProofTweet -> Set(MrNtabCopyObjects.TopicProofTweet),\n      MrPushCopyObjects.TopTweetImpressions -> Set(MrNtabCopyObjects.TopTweetImpressions),\n      MrPushCopyObjects.TrendTweet -> Set(MrNtabCopyObjects.TrendTweet),\n      MrPushCopyObjects.SpaceHost -> Set(MrNtabCopyObjects.SpaceHost),\n      MrPushCopyObjects.SubscribedSearchTweet -> Set(MrNtabCopyObjects.SubscribedSearchTweet),\n      MrPushCopyObjects.TripGeoTweetPushCopy -> Set(MrNtabCopyObjects.RecommendedForYouCopy),\n      MrPushCopyObjects.Digest -> Set(MrNtabCopyObjects.Digest),\n      MrPushCopyObjects.TripHqTweetPushCopy -> Set(MrNtabCopyObjects.HighQualityTweet),\n      MrPushCopyObjects.ExploreVideoTweet -> Set(MrNtabCopyObjects.ExploreVideoTweet),\n      MrPushCopyObjects.ListRecommendation -> Set(MrNtabCopyObjects.ListRecommendation),\n      MrPushCopyObjects.MagicFanoutCreatorSubscription -> Set(\n        MrNtabCopyObjects.MagicFanoutCreatorSubscription),\n      MrPushCopyObjects.MagicFanoutNewCreator -> Set(MrNtabCopyObjects.MagicFanoutNewCreator)\n    )\n\n  /**\n   *\n   * @param crt - [[CommonRecommendationType]] used for a frigate push notification\n   *\n   * @return - Set of [[MRPushCopy]] objects representing push copies eligibile for a\n   *         [[CommonRecommendationType]]\n   */\n  def getPushCopiesFromRectype(crt: CommonRecommendationType): Option[Set[MRPushCopy]] =\n    rectypeToPushCopy.get(crt)\n\n  /**\n   *\n   * @param pushcopy - [[MRPushCopy]] object representing a push notification copy\n   * @return - Set of [[MRNtabCopy]] objects that can be paired with a given [[MRPushCopy]]\n   */\n  def getNtabcopiesFromPushcopy(pushcopy: MRPushCopy): Option[Set[MRNtabCopy]] =\n    pushcopyToNtabcopy.get(pushcopy)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/cross/CopyFilters.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler.cross\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.util.MRPushCopy\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\nprivate[cross] class CopyFilters(statsReceiver: StatsReceiver) {\n\n  private val copyPredicates = new CopyPredicates(statsReceiver.scope(\"copy_predicate\"))\n\n  def execute(rawCandidate: RawCandidate, pushCopies: Seq[MRPushCopy]): Future[Seq[MRPushCopy]] = {\n    val candidateCopyPairs: Seq[CandidateCopyPair] =\n      pushCopies.map(CandidateCopyPair(rawCandidate, _))\n\n    val compositePredicate: Predicate[CandidateCopyPair] = rawCandidate match {\n      case _: F1FirstDegree | _: OutOfNetworkTweetCandidate | _: EventCandidate |\n          _: TopicProofTweetCandidate | _: ListPushCandidate | _: HermitInterestBasedUserFollow |\n          _: UserFollowWithoutSocialContextCandidate | _: DiscoverTwitterCandidate |\n          _: TopTweetImpressionsCandidate | _: TrendTweetCandidate |\n          _: SubscribedSearchTweetCandidate | _: DigestCandidate =>\n        copyPredicates.alwaysTruePredicate\n\n      case _: SocialContextActions => copyPredicates.displaySocialContextPredicate\n\n      case _ => copyPredicates.unrecognizedCandidatePredicate // block unrecognised candidates\n    }\n\n    // apply predicate to all [[MRPushCopy]] objects\n    val filterResults: Future[Seq[Boolean]] = compositePredicate(candidateCopyPairs)\n    filterResults.map { results: Seq[Boolean] =>\n      val seqBuilder = Seq.newBuilder[MRPushCopy]\n      results.zip(pushCopies).foreach {\n        case (result, pushCopy) => if (result) seqBuilder += pushCopy\n      }\n      seqBuilder.result()\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/refresh_handler/cross/CopyPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.refresh_handler.cross\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.SocialContextActions\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.hermit.predicate.Predicate\n\nclass CopyPredicates(statsReceiver: StatsReceiver) {\n  val alwaysTruePredicate = Predicate\n    .from { _: CandidateCopyPair =>\n      true\n    }.withStats(statsReceiver.scope(\"always_true_copy_predicate\"))\n\n  val unrecognizedCandidatePredicate = alwaysTruePredicate.flip\n    .withStats(statsReceiver.scope(\"unrecognized_candidate\"))\n\n  val displaySocialContextPredicate = Predicate\n    .from { candidateCopyPair: CandidateCopyPair =>\n      candidateCopyPair.candidate match {\n        case candidateWithScActions: RawCandidate with SocialContextActions =>\n          val socialContextUserIds = candidateWithScActions.socialContextActions.map(_.userId)\n          val countSocialContext = socialContextUserIds.size\n          val pushCopy = candidateCopyPair.pushCopy\n\n          countSocialContext match {\n            case 1 => pushCopy.hasOneDisplaySocialContext && !pushCopy.hasOtherSocialContext\n            case 2 => pushCopy.hasTwoDisplayContext && !pushCopy.hasOtherSocialContext\n            case c if c > 2 =>\n              pushCopy.hasOneDisplaySocialContext && pushCopy.hasOtherSocialContext\n            case _ => false\n          }\n\n        case _ => false\n      }\n    }.withStats(statsReceiver.scope(\"display_social_context_predicate\"))\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/scriber/MrRequestScribeHandler.scala",
    "content": "package com.twitter.frigate.pushservice.scriber\n\nimport com.twitter.bijection.Base64String\nimport com.twitter.bijection.Injection\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.core_workflows.user_model.thriftscala.{UserState => ThriftUserState}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.tracing.Trace\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.CandidateResult\nimport com.twitter.frigate.common.base.Invalid\nimport com.twitter.frigate.common.base.OK\nimport com.twitter.frigate.common.base.Result\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.data_pipeline.features_common.PushQualityModelFeatureContext\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.scribe.thriftscala.CandidateFilteredOutStep\nimport com.twitter.frigate.scribe.thriftscala.CandidateRequestInfo\nimport com.twitter.frigate.scribe.thriftscala.MrRequestScribe\nimport com.twitter.frigate.scribe.thriftscala.TargetUserInfo\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.frigate.thriftscala.TweetNotification\nimport com.twitter.frigate.thriftscala.{SocialContextAction => TSocialContextAction}\nimport com.twitter.logging.Logger\nimport com.twitter.ml.api.DataRecord\nimport com.twitter.ml.api.Feature\nimport com.twitter.ml.api.FeatureType\nimport com.twitter.ml.api.util.SRichDataRecord\nimport com.twitter.ml.api.util.ScalaToJavaDataRecordConversions\nimport com.twitter.nrel.heavyranker.PushPredictionHelper\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport java.util.UUID\nimport scala.collection.mutable\n\nclass MrRequestScribeHandler(mrRequestScriberNode: String, stats: StatsReceiver) {\n\n  private val mrRequestScribeLogger = Logger(mrRequestScriberNode)\n\n  private val mrRequestScribeTargetFilteringStats =\n    stats.counter(\"MrRequestScribeHandler_target_filtering\")\n  private val mrRequestScribeCandidateFilteringStats =\n    stats.counter(\"MrRequestScribeHandler_candidate_filtering\")\n  private val mrRequestScribeInvalidStats =\n    stats.counter(\"MrRequestScribeHandler_invalid_filtering\")\n  private val mrRequestScribeUnsupportedFeatureTypeStats =\n    stats.counter(\"MrRequestScribeHandler_unsupported_feature_type\")\n  private val mrRequestScribeNotIncludedFeatureStats =\n    stats.counter(\"MrRequestScribeHandler_not_included_features\")\n\n  private final val MrRequestScribeInjection: Injection[MrRequestScribe, String] = BinaryScalaCodec(\n    MrRequestScribe\n  ) andThen Injection.connect[Array[Byte], Base64String, String]\n\n  /**\n   *\n   * @param target : Target user id\n   * @param result : Result for target filtering\n   *\n   * @return\n   */\n  def scribeForTargetFiltering(target: Target, result: Result): Future[Option[MrRequestScribe]] = {\n    if (target.isLoggedOutUser || !enableTargetFilteringScribing(target)) {\n      Future.None\n    } else {\n      val predicate = result match {\n        case Invalid(reason) => reason\n        case _ =>\n          mrRequestScribeInvalidStats.incr()\n          throw new IllegalStateException(\"Invalid reason for Target Filtering \" + result)\n      }\n      buildScribeThrift(target, predicate, None).map { targetFilteredScribe =>\n        writeAtTargetFilteringStep(target, targetFilteredScribe)\n        Some(targetFilteredScribe)\n      }\n    }\n  }\n\n  /**\n   *\n   * @param target                       : Target user id\n   * @param hydratedCandidates           : Candidates hydrated with details: impressionId, frigateNotification and source\n   * @param preRankingFilteredCandidates : Candidates result filtered out at preRanking filtering step\n   * @param rankedCandidates             : Sorted candidates details ranked by ranking step\n   * @param rerankedCandidates           : Sorted candidates details ranked by reranking step\n   * @param restrictFilteredCandidates   : Candidates details filtered out at restrict step\n   * @param allTakeCandidateResults      : Candidates results at take step, include the candidates we take and the candidates filtered out at take step [with different result]\n   *\n   * @return\n   */\n  def scribeForCandidateFiltering(\n    target: Target,\n    hydratedCandidates: Seq[CandidateDetails[PushCandidate]],\n    preRankingFilteredCandidates: Seq[CandidateResult[PushCandidate, Result]],\n    rankedCandidates: Seq[CandidateDetails[PushCandidate]],\n    rerankedCandidates: Seq[CandidateDetails[PushCandidate]],\n    restrictFilteredCandidates: Seq[CandidateDetails[PushCandidate]],\n    allTakeCandidateResults: Seq[CandidateResult[PushCandidate, Result]]\n  ): Future[Seq[MrRequestScribe]] = {\n    if (target.isLoggedOutUser || target.isEmailUser) {\n      Future.Nil\n    } else if (enableCandidateFilteringScribing(target)) {\n      val hydrateFeature =\n        target.params(PushFeatureSwitchParams.EnableMrRequestScribingWithFeatureHydrating) ||\n          target.scribeFeatureForRequestScribe\n\n      val candidateRequestInfoSeq = generateCandidatesScribeInfo(\n        hydratedCandidates,\n        preRankingFilteredCandidates,\n        rankedCandidates,\n        rerankedCandidates,\n        restrictFilteredCandidates,\n        allTakeCandidateResults,\n        isFeatureHydratingEnabled = hydrateFeature\n      )\n      val flattenStructure =\n        target.params(PushFeatureSwitchParams.EnableFlattenMrRequestScribing) || hydrateFeature\n      candidateRequestInfoSeq.flatMap { candidateRequestInfos =>\n        if (flattenStructure) {\n          Future.collect {\n            candidateRequestInfos.map { candidateRequestInfo =>\n              buildScribeThrift(target, None, Some(Seq(candidateRequestInfo)))\n                .map { mrRequestScribe =>\n                  writeAtCandidateFilteringStep(target, mrRequestScribe)\n                  mrRequestScribe\n                }\n            }\n          }\n        } else {\n          buildScribeThrift(target, None, Some(candidateRequestInfos))\n            .map { mrRequestScribe =>\n              writeAtCandidateFilteringStep(target, mrRequestScribe)\n              Seq(mrRequestScribe)\n            }\n        }\n      }\n    } else Future.Nil\n\n  }\n\n  private def buildScribeThrift(\n    target: Target,\n    targetFilteredOutPredicate: Option[String],\n    candidatesRequestInfo: Option[Seq[CandidateRequestInfo]]\n  ): Future[MrRequestScribe] = {\n    Future\n      .join(\n        target.targetUserState,\n        generateTargetFeatureScribeInfo(target),\n        target.targetUser).map {\n        case (userStateOption, targetFeatureOption, gizmoduckUserOpt) =>\n          val userState = userStateOption.map(userState => ThriftUserState(userState.id))\n          val targetFeatures =\n            targetFeatureOption.map(ScalaToJavaDataRecordConversions.javaDataRecord2ScalaDataRecord)\n          val traceId = Trace.id.traceId.toLong\n\n          MrRequestScribe(\n            requestId = UUID.randomUUID.toString.replaceAll(\"-\", \"\"),\n            scribedTimeMs = Time.now.inMilliseconds,\n            targetUserId = target.targetId,\n            targetUserInfo = Some(\n              TargetUserInfo(\n                userState,\n                features = targetFeatures,\n                userType = gizmoduckUserOpt.map(_.userType))\n            ),\n            targetFilteredOutPredicate = targetFilteredOutPredicate,\n            candidates = candidatesRequestInfo,\n            traceId = Some(traceId)\n          )\n      }\n  }\n\n  private def generateTargetFeatureScribeInfo(\n    target: Target\n  ): Future[Option[DataRecord]] = {\n    val featureList =\n      target.params(PushFeatureSwitchParams.TargetLevelFeatureListForMrRequestScribing)\n    if (featureList.nonEmpty) {\n      PushPredictionHelper\n        .getDataRecordFromTargetFeatureMap(\n          target.targetId,\n          target.featureMap,\n          stats\n        ).map { dataRecord =>\n          val richRecord =\n            new SRichDataRecord(dataRecord, PushQualityModelFeatureContext.featureContext)\n\n          val selectedRecord =\n            SRichDataRecord(new DataRecord(), PushQualityModelFeatureContext.featureContext)\n          featureList.map { featureName =>\n            val feature: Feature[_] = {\n              try {\n                PushQualityModelFeatureContext.featureContext.getFeature(featureName)\n              } catch {\n                case _: Exception =>\n                  mrRequestScribeNotIncludedFeatureStats.incr()\n                  throw new IllegalStateException(\n                    \"Scribing features not included in FeatureContext: \" + featureName)\n              }\n            }\n\n            richRecord.getFeatureValueOpt(feature).foreach { featureVal =>\n              feature.getFeatureType() match {\n                case FeatureType.BINARY =>\n                  selectedRecord.setFeatureValue(\n                    feature.asInstanceOf[Feature[Boolean]],\n                    featureVal.asInstanceOf[Boolean])\n                case FeatureType.CONTINUOUS =>\n                  selectedRecord.setFeatureValue(\n                    feature.asInstanceOf[Feature[Double]],\n                    featureVal.asInstanceOf[Double])\n                case FeatureType.STRING =>\n                  selectedRecord.setFeatureValue(\n                    feature.asInstanceOf[Feature[String]],\n                    featureVal.asInstanceOf[String])\n                case FeatureType.DISCRETE =>\n                  selectedRecord.setFeatureValue(\n                    feature.asInstanceOf[Feature[Long]],\n                    featureVal.asInstanceOf[Long])\n                case _ =>\n                  mrRequestScribeUnsupportedFeatureTypeStats.incr()\n              }\n            }\n          }\n          Some(selectedRecord.getRecord)\n        }\n    } else Future.None\n  }\n\n  private def generateCandidatesScribeInfo(\n    hydratedCandidates: Seq[CandidateDetails[PushCandidate]],\n    preRankingFilteredCandidates: Seq[CandidateResult[PushCandidate, Result]],\n    rankedCandidates: Seq[CandidateDetails[PushCandidate]],\n    rerankedCandidates: Seq[CandidateDetails[PushCandidate]],\n    restrictFilteredCandidates: Seq[CandidateDetails[PushCandidate]],\n    allTakeCandidateResults: Seq[CandidateResult[PushCandidate, Result]],\n    isFeatureHydratingEnabled: Boolean\n  ): Future[Seq[CandidateRequestInfo]] = {\n    val candidatesMap = new mutable.HashMap[String, CandidateRequestInfo]\n\n    hydratedCandidates.foreach { hydratedCandidate =>\n      val frgNotif = hydratedCandidate.candidate.frigateNotification\n      val simplifiedTweetNotificationOpt = frgNotif.tweetNotification.map { tweetNotification =>\n        TweetNotification(\n          tweetNotification.tweetId,\n          Seq.empty[TSocialContextAction],\n          tweetNotification.tweetAuthorId)\n      }\n      val simplifiedFrigateNotification = FrigateNotification(\n        frgNotif.commonRecommendationType,\n        frgNotif.notificationDisplayLocation,\n        tweetNotification = simplifiedTweetNotificationOpt\n      )\n      candidatesMap(hydratedCandidate.candidate.impressionId) = CandidateRequestInfo(\n        candidateId = \"\",\n        candidateSource = hydratedCandidate.source.substring(\n          0,\n          Math.min(6, hydratedCandidate.source.length)\n        ),\n        frigateNotification = Some(simplifiedFrigateNotification),\n        modelScore = None,\n        rankPosition = None,\n        rerankPosition = None,\n        features = None,\n        isSent = Some(false)\n      )\n    }\n\n    preRankingFilteredCandidates.foreach { preRankingFilteredCandidateResult =>\n      candidatesMap(preRankingFilteredCandidateResult.candidate.impressionId) =\n        candidatesMap(preRankingFilteredCandidateResult.candidate.impressionId)\n          .copy(\n            candidateFilteredOutPredicate = preRankingFilteredCandidateResult.result match {\n              case Invalid(reason) => reason\n              case _ => {\n                mrRequestScribeInvalidStats.incr()\n                throw new IllegalStateException(\n                  \"Invalid reason for Candidate Filtering \" + preRankingFilteredCandidateResult.result)\n              }\n            },\n            candidateFilteredOutStep = Some(CandidateFilteredOutStep.PreRankFiltering)\n          )\n    }\n\n    for {\n      _ <- Future.collectToTry {\n        rankedCandidates.zipWithIndex.map {\n          case (rankedCandidateDetail, index) =>\n            val modelScoresFut = {\n              val crt = rankedCandidateDetail.candidate.commonRecType\n              if (RecTypes.notEligibleForModelScoreTracking.contains(crt)) Future.None\n              else rankedCandidateDetail.candidate.modelScores.map(Some(_))\n            }\n\n            modelScoresFut.map { modelScores =>\n              candidatesMap(rankedCandidateDetail.candidate.impressionId) =\n                candidatesMap(rankedCandidateDetail.candidate.impressionId).copy(\n                  rankPosition = Some(index),\n                  modelScore = modelScores\n                )\n            }\n        }\n      }\n\n      _ = rerankedCandidates.zipWithIndex.foreach {\n        case (rerankedCandidateDetail, index) => {\n          candidatesMap(rerankedCandidateDetail.candidate.impressionId) =\n            candidatesMap(rerankedCandidateDetail.candidate.impressionId).copy(\n              rerankPosition = Some(index)\n            )\n        }\n      }\n\n      _ <- Future.collectToTry {\n        rerankedCandidates.map { rerankedCandidateDetail =>\n          if (isFeatureHydratingEnabled) {\n            PushPredictionHelper\n              .getDataRecord(\n                rerankedCandidateDetail.candidate.target.targetHydrationContext,\n                rerankedCandidateDetail.candidate.target.featureMap,\n                rerankedCandidateDetail.candidate.candidateHydrationContext,\n                rerankedCandidateDetail.candidate.candidateFeatureMap(),\n                stats\n              ).map { features =>\n                candidatesMap(rerankedCandidateDetail.candidate.impressionId) =\n                  candidatesMap(rerankedCandidateDetail.candidate.impressionId).copy(\n                    features = Some(\n                      ScalaToJavaDataRecordConversions.javaDataRecord2ScalaDataRecord(features))\n                  )\n              }\n          } else Future.Unit\n        }\n      }\n\n      _ = restrictFilteredCandidates.foreach { restrictFilteredCandidateDetatil =>\n        candidatesMap(restrictFilteredCandidateDetatil.candidate.impressionId) =\n          candidatesMap(restrictFilteredCandidateDetatil.candidate.impressionId)\n            .copy(candidateFilteredOutStep = Some(CandidateFilteredOutStep.Restrict))\n      }\n\n      _ = allTakeCandidateResults.foreach { allTakeCandidateResult =>\n        allTakeCandidateResult.result match {\n          case OK =>\n            candidatesMap(allTakeCandidateResult.candidate.impressionId) =\n              candidatesMap(allTakeCandidateResult.candidate.impressionId).copy(isSent = Some(true))\n          case Invalid(reason) =>\n            candidatesMap(allTakeCandidateResult.candidate.impressionId) =\n              candidatesMap(allTakeCandidateResult.candidate.impressionId).copy(\n                candidateFilteredOutPredicate = reason,\n                candidateFilteredOutStep = Some(CandidateFilteredOutStep.PostRankFiltering))\n          case _ =>\n            mrRequestScribeInvalidStats.incr()\n            throw new IllegalStateException(\n              \"Invalid reason for Candidate Filtering \" + allTakeCandidateResult.result)\n        }\n      }\n    } yield candidatesMap.values.toSeq\n  }\n\n  private def enableTargetFilteringScribing(target: Target): Boolean = {\n    target.params(PushParams.EnableMrRequestScribing) && target.params(\n      PushFeatureSwitchParams.EnableMrRequestScribingForTargetFiltering)\n  }\n\n  private def enableCandidateFilteringScribing(target: Target): Boolean = {\n    target.params(PushParams.EnableMrRequestScribing) && target.params(\n      PushFeatureSwitchParams.EnableMrRequestScribingForCandidateFiltering)\n  }\n\n  private def writeAtTargetFilteringStep(target: Target, mrRequestScribe: MrRequestScribe) = {\n    logToScribe(mrRequestScribe)\n    mrRequestScribeTargetFilteringStats.incr()\n  }\n\n  private def writeAtCandidateFilteringStep(target: Target, mrRequestScribe: MrRequestScribe) = {\n    logToScribe(mrRequestScribe)\n    mrRequestScribeCandidateFilteringStats.incr()\n  }\n\n  private def logToScribe(mrRequestScribe: MrRequestScribe): Unit = {\n    val logEntry: String = MrRequestScribeInjection(mrRequestScribe)\n    mrRequestScribeLogger.info(logEntry)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/send_handler/SendHandler.scala",
    "content": "package com.twitter.frigate.pushservice.send_handler\n\nimport com.twitter.finagle.stats.BroadcastStatsReceiver\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.CandidateFilteringOnlyFlow\nimport com.twitter.frigate.common.base.CandidateResult\nimport com.twitter.frigate.common.base.FeatureMap\nimport com.twitter.frigate.common.base.OK\nimport com.twitter.frigate.common.base.Response\nimport com.twitter.frigate.common.base.Result\nimport com.twitter.frigate.common.base.Stats.track\nimport com.twitter.frigate.common.config.CommonConstants\nimport com.twitter.frigate.common.logger.MRLogger\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.common.util.InvalidRequestException\nimport com.twitter.frigate.common.util.MrNtabCopyObjects\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.ml.HydrationContextBuilder\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams.EnableMagicFanoutNewsForYouNtabCopy\nimport com.twitter.frigate.pushservice.scriber.MrRequestScribeHandler\nimport com.twitter.frigate.pushservice.send_handler.generator.PushRequestToCandidate\nimport com.twitter.frigate.pushservice.take.SendHandlerNotifier\nimport com.twitter.frigate.pushservice.take.candidate_validator.SendHandlerPostCandidateValidator\nimport com.twitter.frigate.pushservice.take.candidate_validator.SendHandlerPreCandidateValidator\nimport com.twitter.frigate.pushservice.target.PushTargetUserBuilder\nimport com.twitter.frigate.pushservice.util.ResponseStatsTrackUtils.trackStatsForResponseToRequest\nimport com.twitter.frigate.pushservice.util.SendHandlerPredicateUtil\nimport com.twitter.frigate.pushservice.thriftscala.PushRequest\nimport com.twitter.frigate.pushservice.thriftscala.PushRequestScribe\nimport com.twitter.frigate.pushservice.thriftscala.PushResponse\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.nrel.heavyranker.FeatureHydrator\nimport com.twitter.util._\n\n/**\n * A handler for sending PushRequests\n */\nclass SendHandler(\n  pushTargetUserBuilder: PushTargetUserBuilder,\n  preCandidateValidator: SendHandlerPreCandidateValidator,\n  postCandidateValidator: SendHandlerPostCandidateValidator,\n  sendHandlerNotifier: SendHandlerNotifier,\n  candidateHydrator: SendHandlerPushCandidateHydrator,\n  featureHydrator: FeatureHydrator,\n  sendHandlerPredicateUtil: SendHandlerPredicateUtil,\n  mrRequestScriberNode: String\n)(\n  implicit val statsReceiver: StatsReceiver,\n  implicit val config: Config)\n    extends CandidateFilteringOnlyFlow[Target, RawCandidate, PushCandidate] {\n\n  implicit private val timer: Timer = new JavaTimer(true)\n  val stats = statsReceiver.scope(\"SendHandler\")\n  val log = MRLogger(\"SendHandler\")\n\n  private val buildTargetStats = stats.scope(\"build_target\")\n\n  private val candidateHydrationLatency: Stat =\n    stats.stat(\"candidateHydrationLatency\")\n\n  private val candidatePreValidatorLatency: Stat =\n    stats.stat(\"candidatePreValidatorLatency\")\n\n  private val candidatePostValidatorLatency: Stat =\n    stats.stat(\"candidatePostValidatorLatency\")\n\n  private val featureHydrationLatency: StatsReceiver =\n    stats.scope(\"featureHydrationLatency\")\n\n  private val mrRequestScribeHandler =\n    new MrRequestScribeHandler(mrRequestScriberNode, stats.scope(\"mr_request_scribe\"))\n\n  def apply(request: PushRequest): Future[PushResponse] = {\n    val receivers = Seq(\n      stats,\n      stats.scope(request.notification.commonRecommendationType.toString)\n    )\n    val bStats = BroadcastStatsReceiver(receivers)\n    bStats.counter(\"requests\").incr()\n    Stat\n      .timeFuture(bStats.stat(\"latency\"))(\n        process(request).raiseWithin(CommonConstants.maxPushRequestDuration))\n      .onSuccess {\n        case (pushResp, rawCandidate) =>\n          trackStatsForResponseToRequest(\n            rawCandidate.commonRecType,\n            rawCandidate.target,\n            pushResp,\n            receivers)(statsReceiver)\n          if (!request.context.exists(_.darkWrite.contains(true))) {\n            config.requestScribe(PushRequestScribe(request, pushResp))\n          }\n      }\n      .onFailure { ex =>\n        bStats.counter(\"failures\").incr()\n        bStats.scope(\"failures\").counter(ex.getClass.getCanonicalName).incr()\n      }\n      .map {\n        case (pushResp, _) => pushResp\n      }\n  }\n\n  private def process(request: PushRequest): Future[(PushResponse, RawCandidate)] = {\n    val recType = request.notification.commonRecommendationType\n\n    track(buildTargetStats)(\n      pushTargetUserBuilder\n        .buildTarget(\n          request.userId,\n          request.context\n        )\n    ).flatMap { targetUser =>\n      val responseWithScribedInfo = request.context.exists { context =>\n        context.responseWithScribedInfo.contains(true)\n      }\n      val newRequest =\n        if (request.notification.commonRecommendationType == CommonRecommendationType.MagicFanoutNewsEvent &&\n          targetUser.params(EnableMagicFanoutNewsForYouNtabCopy)) {\n          val newNotification = request.notification.copy(ntabCopyId =\n            Some(MrNtabCopyObjects.MagicFanoutNewsForYouCopy.copyId))\n          request.copy(notification = newNotification)\n        } else request\n\n      if (RecTypes.isSendHandlerType(recType) || newRequest.context.exists(\n          _.allowCRT.contains(true))) {\n\n        val rawCandidateFut = PushRequestToCandidate.generatePushCandidate(\n          newRequest.notification,\n          targetUser\n        )\n\n        rawCandidateFut.flatMap { rawCandidate =>\n          val pushResponse = process(targetUser, Seq(rawCandidate)).flatMap {\n            sendHandlerNotifier.checkResponseAndNotify(_, responseWithScribedInfo)\n          }\n\n          pushResponse.map { pushResponse =>\n            (pushResponse, rawCandidate)\n          }\n        }\n      } else {\n        Future.exception(InvalidRequestException(s\"${recType.name} not supported in SendHandler\"))\n      }\n    }\n  }\n\n  private def hydrateFeatures(\n    candidateDetails: Seq[CandidateDetails[PushCandidate]],\n    target: Target,\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n\n    candidateDetails.headOption match {\n      case Some(candidateDetail)\n          if RecTypes.notEligibleForModelScoreTracking(candidateDetail.candidate.commonRecType) =>\n        Future.value(candidateDetails)\n\n      case Some(candidateDetail) =>\n        val hydrationContextFut = HydrationContextBuilder.build(candidateDetail.candidate)\n        hydrationContextFut.flatMap { hc =>\n          featureHydrator\n            .hydrateCandidate(Seq(hc), target.mrRequestContextForFeatureStore)\n            .map { hydrationResult =>\n              val features = hydrationResult.getOrElse(hc, FeatureMap())\n              candidateDetail.candidate.mergeFeatures(features)\n              candidateDetails\n            }\n        }\n      case _ => Future.Nil\n    }\n  }\n\n  override def process(\n    target: Target,\n    externalCandidates: Seq[RawCandidate]\n  ): Future[Response[PushCandidate, Result]] = {\n    val candidate = externalCandidates.map(CandidateDetails(_, \"realtime\"))\n\n    for {\n      hydratedCandidatesWithCopy <- hydrateCandidates(candidate)\n\n      (candidates, preHydrationFilteredCandidates) <- track(filterStats)(\n        filter(target, hydratedCandidatesWithCopy)\n      )\n\n      featureHydratedCandidates <-\n        track(featureHydrationLatency)(hydrateFeatures(candidates, target))\n\n      allTakeCandidateResults <- track(takeStats)(\n        take(target, featureHydratedCandidates, desiredCandidateCount(target))\n      )\n\n      _ <- mrRequestScribeHandler.scribeForCandidateFiltering(\n        target = target,\n        hydratedCandidates = hydratedCandidatesWithCopy,\n        preRankingFilteredCandidates = preHydrationFilteredCandidates,\n        rankedCandidates = featureHydratedCandidates,\n        rerankedCandidates = Seq.empty,\n        restrictFilteredCandidates = Seq.empty, // no restrict step\n        allTakeCandidateResults = allTakeCandidateResults\n      )\n    } yield {\n\n      /**\n       * We combine the results for all filtering steps and pass on in sequence to next step\n       *\n       * This is done to ensure the filtering reason for the candidate from multiple levels of\n       * filtering is carried all the way until [[PushResponse]] is built and returned from\n       * frigate-pushservice-send\n       */\n      Response(OK, allTakeCandidateResults ++ preHydrationFilteredCandidates)\n    }\n  }\n\n  override def hydrateCandidates(\n    candidates: Seq[CandidateDetails[RawCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    Stat.timeFuture(candidateHydrationLatency)(candidateHydrator(candidates))\n  }\n\n  // Filter Step - pre-predicates and app specific predicates\n  override def filter(\n    target: Target,\n    hydratedCandidatesDetails: Seq[CandidateDetails[PushCandidate]]\n  ): Future[\n    (Seq[CandidateDetails[PushCandidate]], Seq[CandidateResult[PushCandidate, Result]])\n  ] = {\n    Stat.timeFuture(candidatePreValidatorLatency)(\n      sendHandlerPredicateUtil.preValidationForCandidate(\n        hydratedCandidatesDetails,\n        preCandidateValidator\n      ))\n  }\n\n  // Post Validation - Take step\n  override def validCandidates(\n    target: Target,\n    candidates: Seq[PushCandidate]\n  ): Future[Seq[Result]] = {\n    Stat.timeFuture(candidatePostValidatorLatency)(Future.collect(candidates.map { candidate =>\n      sendHandlerPredicateUtil\n        .postValidationForCandidate(candidate, postCandidateValidator)\n        .map(res => res.result)\n    }))\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/send_handler/SendHandlerPushCandidateHydrator.scala",
    "content": "package com.twitter.frigate.pushservice.send_handler\n\nimport com.twitter.escherbird.metadata.thriftscala.EntityMegadata\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.store.interests.InterestsLookupRequestWithContext\nimport com.twitter.frigate.common.util.MrNtabCopyObjects\nimport com.twitter.frigate.common.util.MrPushCopyObjects\nimport com.twitter.frigate.magic_events.thriftscala.FanoutEvent\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.store.EventRequest\nimport com.twitter.frigate.pushservice.store.UttEntityHydrationStore\nimport com.twitter.frigate.pushservice.util.CandidateHydrationUtil._\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.store.semantic_core.SemanticEntityForQuery\nimport com.twitter.interests.thriftscala.UserInterests\nimport com.twitter.livevideo.timeline.domain.v2.{Event => LiveEvent}\nimport com.twitter.simclusters_v2.thriftscala.SimClustersInferredEntities\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.UserId\nimport com.twitter.ubs.thriftscala.AudioSpace\nimport com.twitter.util.Future\n\ncase class SendHandlerPushCandidateHydrator(\n  lexServiceStore: ReadableStore[EventRequest, LiveEvent],\n  fanoutMetadataStore: ReadableStore[(Long, Long), FanoutEvent],\n  semanticCoreMegadataStore: ReadableStore[SemanticEntityForQuery, EntityMegadata],\n  safeUserStore: ReadableStore[Long, User],\n  simClusterToEntityStore: ReadableStore[Int, SimClustersInferredEntities],\n  audioSpaceStore: ReadableStore[String, AudioSpace],\n  interestsLookupStore: ReadableStore[InterestsLookupRequestWithContext, UserInterests],\n  uttEntityHydrationStore: UttEntityHydrationStore,\n  superFollowCreatorTweetCountStore: ReadableStore[UserId, Int]\n)(\n  implicit statsReceiver: StatsReceiver,\n  implicit val weightedOpenOrNtabClickModelScorer: PushMLModelScorer) {\n\n  lazy val candidateWithCopyNumStat = statsReceiver.stat(\"candidate_with_copy_num\")\n  lazy val hydratedCandidateStat = statsReceiver.scope(\"hydrated_candidates\")\n\n  def updateCandidates(\n    candidateDetails: Seq[CandidateDetails[RawCandidate]],\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n\n    Future.collect {\n      candidateDetails.map { candidateDetail =>\n        val pushCandidate = candidateDetail.candidate\n\n        val copyIds = getCopyIdsByCRT(pushCandidate.commonRecType)\n\n        val hydratedCandidateFut = pushCandidate match {\n          case magicFanoutNewsEventCandidate: MagicFanoutNewsEventCandidate =>\n            getHydratedCandidateForMagicFanoutNewsEvent(\n              magicFanoutNewsEventCandidate,\n              copyIds,\n              lexServiceStore,\n              fanoutMetadataStore,\n              semanticCoreMegadataStore,\n              simClusterToEntityStore,\n              interestsLookupStore,\n              uttEntityHydrationStore\n            )\n\n          case scheduledSpaceSubscriberCandidate: ScheduledSpaceSubscriberCandidate =>\n            getHydratedCandidateForScheduledSpaceSubscriber(\n              scheduledSpaceSubscriberCandidate,\n              safeUserStore,\n              copyIds,\n              audioSpaceStore\n            )\n          case scheduledSpaceSpeakerCandidate: ScheduledSpaceSpeakerCandidate =>\n            getHydratedCandidateForScheduledSpaceSpeaker(\n              scheduledSpaceSpeakerCandidate,\n              safeUserStore,\n              copyIds,\n              audioSpaceStore\n            )\n          case magicFanoutSportsEventCandidate: MagicFanoutSportsEventCandidate with MagicFanoutSportsScoreInformation =>\n            getHydratedCandidateForMagicFanoutSportsEvent(\n              magicFanoutSportsEventCandidate,\n              copyIds,\n              lexServiceStore,\n              fanoutMetadataStore,\n              semanticCoreMegadataStore,\n              interestsLookupStore,\n              uttEntityHydrationStore\n            )\n          case magicFanoutProductLaunchCandidate: MagicFanoutProductLaunchCandidate =>\n            getHydratedCandidateForMagicFanoutProductLaunch(\n              magicFanoutProductLaunchCandidate,\n              copyIds)\n          case creatorEventCandidate: MagicFanoutCreatorEventCandidate =>\n            getHydratedCandidateForMagicFanoutCreatorEvent(\n              creatorEventCandidate,\n              safeUserStore,\n              copyIds,\n              superFollowCreatorTweetCountStore)\n          case _ =>\n            throw new IllegalArgumentException(\"Incorrect candidate type when update candidates\")\n        }\n\n        hydratedCandidateFut.map { hydratedCandidate =>\n          hydratedCandidateStat.counter(hydratedCandidate.commonRecType.name).incr()\n          CandidateDetails(\n            hydratedCandidate,\n            source = candidateDetail.source\n          )\n        }\n      }\n    }\n  }\n\n  private def getCopyIdsByCRT(crt: CommonRecommendationType): CopyIds = {\n    crt match {\n      case CommonRecommendationType.MagicFanoutNewsEvent =>\n        CopyIds(\n          pushCopyId = Some(MrPushCopyObjects.MagicFanoutNewsPushCopy.copyId),\n          ntabCopyId = Some(MrNtabCopyObjects.MagicFanoutNewsForYouCopy.copyId),\n          aggregationId = None\n        )\n\n      case CommonRecommendationType.ScheduledSpaceSubscriber =>\n        CopyIds(\n          pushCopyId = Some(MrPushCopyObjects.ScheduledSpaceSubscriber.copyId),\n          ntabCopyId = Some(MrNtabCopyObjects.ScheduledSpaceSubscriber.copyId),\n          aggregationId = None\n        )\n      case CommonRecommendationType.ScheduledSpaceSpeaker =>\n        CopyIds(\n          pushCopyId = Some(MrPushCopyObjects.ScheduledSpaceSpeaker.copyId),\n          ntabCopyId = Some(MrNtabCopyObjects.ScheduledSpaceSpeakerNow.copyId),\n          aggregationId = None\n        )\n      case CommonRecommendationType.SpaceSpeaker =>\n        CopyIds(\n          pushCopyId = Some(MrPushCopyObjects.SpaceSpeaker.copyId),\n          ntabCopyId = Some(MrNtabCopyObjects.SpaceSpeaker.copyId),\n          aggregationId = None\n        )\n      case CommonRecommendationType.SpaceHost =>\n        CopyIds(\n          pushCopyId = Some(MrPushCopyObjects.SpaceHost.copyId),\n          ntabCopyId = Some(MrNtabCopyObjects.SpaceHost.copyId),\n          aggregationId = None\n        )\n      case CommonRecommendationType.MagicFanoutSportsEvent =>\n        CopyIds(\n          pushCopyId = Some(MrPushCopyObjects.MagicFanoutSportsPushCopy.copyId),\n          ntabCopyId = Some(MrNtabCopyObjects.MagicFanoutSportsCopy.copyId),\n          aggregationId = None\n        )\n      case CommonRecommendationType.MagicFanoutProductLaunch =>\n        CopyIds(\n          pushCopyId = Some(MrPushCopyObjects.MagicFanoutProductLaunch.copyId),\n          ntabCopyId = Some(MrNtabCopyObjects.ProductLaunch.copyId),\n          aggregationId = None\n        )\n      case CommonRecommendationType.CreatorSubscriber =>\n        CopyIds(\n          pushCopyId = Some(MrPushCopyObjects.MagicFanoutCreatorSubscription.copyId),\n          ntabCopyId = Some(MrNtabCopyObjects.MagicFanoutCreatorSubscription.copyId),\n          aggregationId = None\n        )\n      case CommonRecommendationType.NewCreator =>\n        CopyIds(\n          pushCopyId = Some(MrPushCopyObjects.MagicFanoutNewCreator.copyId),\n          ntabCopyId = Some(MrNtabCopyObjects.MagicFanoutNewCreator.copyId),\n          aggregationId = None\n        )\n      case _ =>\n        throw new IllegalArgumentException(\"Incorrect candidate type when fetch copy ids\")\n    }\n  }\n\n  def apply(\n    candidateDetails: Seq[CandidateDetails[RawCandidate]]\n  ): Future[Seq[CandidateDetails[PushCandidate]]] = {\n    updateCandidates(candidateDetails)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/send_handler/generator/CandidateGenerator.scala",
    "content": "package com.twitter.frigate.pushservice.send_handler.generator\n\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.util.Future\n\ntrait CandidateGenerator {\n\n  /**\n   * Build RawCandidate from FrigateNotification\n   * @param target\n   * @param frigateNotification\n   * @return RawCandidate\n   */\n  def getCandidate(target: Target, frigateNotification: FrigateNotification): Future[RawCandidate]\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/send_handler/generator/MagicFanoutCreatorEventCandidateGenerator.scala",
    "content": "package com.twitter.frigate.pushservice.send_handler.generator\n\nimport com.twitter.frigate.common.base.MagicFanoutCreatorEventCandidate\nimport com.twitter.frigate.magic_events.thriftscala.CreatorFanoutType\nimport com.twitter.frigate.magic_events.thriftscala.MagicEventsReason\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.model.PushTypes\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.util.Future\n\nobject MagicFanoutCreatorEventCandidateGenerator extends CandidateGenerator {\n  override def getCandidate(\n    targetUser: PushTypes.Target,\n    notification: FrigateNotification\n  ): Future[PushTypes.RawCandidate] = {\n\n    require(\n      notification.commonRecommendationType == CommonRecommendationType.CreatorSubscriber || notification.commonRecommendationType == CommonRecommendationType.NewCreator,\n      \"MagicFanoutCreatorEvent: unexpected CRT \" + notification.commonRecommendationType\n    )\n    require(\n      notification.creatorSubscriptionNotification.isDefined,\n      \"MagicFanoutCreatorEvent: creatorSubscriptionNotification is not defined\")\n    require(\n      notification.creatorSubscriptionNotification.exists(_.magicFanoutPushId.isDefined),\n      \"MagicFanoutCreatorEvent: magicFanoutPushId is not defined\")\n    require(\n      notification.creatorSubscriptionNotification.exists(_.fanoutReasons.isDefined),\n      \"MagicFanoutCreatorEvent: fanoutReasons is not defined\")\n    require(\n      notification.creatorSubscriptionNotification.exists(_.creatorId.isDefined),\n      \"MagicFanoutCreatorEvent: creatorId is not defined\")\n    if (notification.commonRecommendationType == CommonRecommendationType.CreatorSubscriber) {\n      require(\n        notification.creatorSubscriptionNotification\n          .exists(_.subscriberId.isDefined),\n        \"MagicFanoutCreatorEvent: subscriber id is not defined\"\n      )\n    }\n\n    val creatorSubscriptionNotification = notification.creatorSubscriptionNotification.get\n\n    val candidate = new RawCandidate with MagicFanoutCreatorEventCandidate {\n\n      override val target: Target = targetUser\n\n      override val pushId: Long =\n        creatorSubscriptionNotification.magicFanoutPushId.get\n\n      override val candidateMagicEventsReasons: Seq[MagicEventsReason] =\n        creatorSubscriptionNotification.fanoutReasons.get\n\n      override val creatorFanoutType: CreatorFanoutType =\n        creatorSubscriptionNotification.creatorFanoutType\n\n      override val commonRecType: CommonRecommendationType =\n        notification.commonRecommendationType\n\n      override val frigateNotification: FrigateNotification = notification\n\n      override val subscriberId: Option[Long] = creatorSubscriptionNotification.subscriberId\n\n      override val creatorId: Long = creatorSubscriptionNotification.creatorId.get\n    }\n\n    Future.value(candidate)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/send_handler/generator/MagicFanoutNewsEventCandidateGenerator.scala",
    "content": "package com.twitter.frigate.pushservice.send_handler.generator\n\nimport com.twitter.frigate.common.base.MagicFanoutNewsEventCandidate\nimport com.twitter.frigate.magic_events.thriftscala.MagicEventsReason\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.frigate.thriftscala.MagicFanoutEventNotificationDetails\nimport com.twitter.util.Future\n\nobject MagicFanoutNewsEventCandidateGenerator extends CandidateGenerator {\n\n  override def getCandidate(\n    targetUser: Target,\n    notification: FrigateNotification\n  ): Future[RawCandidate] = {\n\n    /**\n     * frigateNotification recommendation type should be [[CommonRecommendationType.MagicFanoutNewsEvent]]\n     * AND pushId field should be set\n     **/\n    require(\n      notification.commonRecommendationType == CommonRecommendationType.MagicFanoutNewsEvent,\n      \"MagicFanoutNewsEvent: unexpected CRT \" + notification.commonRecommendationType\n    )\n\n    require(\n      notification.magicFanoutEventNotification.exists(_.pushId.isDefined),\n      \"MagicFanoutNewsEvent: pushId is not defined\")\n\n    val magicFanoutEventNotification = notification.magicFanoutEventNotification.get\n\n    val candidate = new RawCandidate with MagicFanoutNewsEventCandidate {\n\n      override val target: Target = targetUser\n\n      override val eventId: Long = magicFanoutEventNotification.eventId\n\n      override val pushId: Long = magicFanoutEventNotification.pushId.get\n\n      override val candidateMagicEventsReasons: Seq[MagicEventsReason] =\n        magicFanoutEventNotification.eventReasons.getOrElse(Seq.empty)\n\n      override val momentId: Option[Long] = magicFanoutEventNotification.momentId\n\n      override val eventLanguage: Option[String] = magicFanoutEventNotification.eventLanguage\n\n      override val details: Option[MagicFanoutEventNotificationDetails] =\n        magicFanoutEventNotification.details\n\n      override val frigateNotification: FrigateNotification = notification\n    }\n\n    Future.value(candidate)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/send_handler/generator/MagicFanoutProductLaunchCandidateGenerator.scala",
    "content": "package com.twitter.frigate.pushservice.send_handler.generator\n\nimport com.twitter.frigate.common.base.MagicFanoutProductLaunchCandidate\nimport com.twitter.frigate.magic_events.thriftscala.MagicEventsReason\nimport com.twitter.frigate.magic_events.thriftscala.ProductType\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.model.PushTypes\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.util.Future\n\nobject MagicFanoutProductLaunchCandidateGenerator extends CandidateGenerator {\n\n  override def getCandidate(\n    targetUser: PushTypes.Target,\n    notification: FrigateNotification\n  ): Future[PushTypes.RawCandidate] = {\n\n    require(\n      notification.commonRecommendationType == CommonRecommendationType.MagicFanoutProductLaunch,\n      \"MagicFanoutProductLaunch: unexpected CRT \" + notification.commonRecommendationType\n    )\n    require(\n      notification.magicFanoutProductLaunchNotification.isDefined,\n      \"MagicFanoutProductLaunch: magicFanoutProductLaunchNotification is not defined\")\n    require(\n      notification.magicFanoutProductLaunchNotification.exists(_.magicFanoutPushId.isDefined),\n      \"MagicFanoutProductLaunch: magicFanoutPushId is not defined\")\n    require(\n      notification.magicFanoutProductLaunchNotification.exists(_.fanoutReasons.isDefined),\n      \"MagicFanoutProductLaunch: fanoutReasons is not defined\")\n\n    val magicFanoutProductLaunchNotification = notification.magicFanoutProductLaunchNotification.get\n\n    val candidate = new RawCandidate with MagicFanoutProductLaunchCandidate {\n\n      override val target: Target = targetUser\n\n      override val pushId: Long =\n        magicFanoutProductLaunchNotification.magicFanoutPushId.get\n\n      override val candidateMagicEventsReasons: Seq[MagicEventsReason] =\n        magicFanoutProductLaunchNotification.fanoutReasons.get\n\n      override val productLaunchType: ProductType =\n        magicFanoutProductLaunchNotification.productLaunchType\n\n      override val frigateNotification: FrigateNotification = notification\n    }\n\n    Future.value(candidate)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/send_handler/generator/MagicFanoutSportsEventCandidateGenerator.scala",
    "content": "package com.twitter.frigate.pushservice.send_handler.generator\n\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.BaseballGameLiveUpdate\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.BasketballGameLiveUpdate\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.CricketMatchLiveUpdate\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.NflFootballGameLiveUpdate\nimport com.twitter.datatools.entityservice.entities.sports.thriftscala.SoccerMatchLiveUpdate\nimport com.twitter.escherbird.common.thriftscala.Domains\nimport com.twitter.escherbird.common.thriftscala.QualifiedId\nimport com.twitter.escherbird.metadata.thriftscala.EntityMegadata\nimport com.twitter.frigate.common.base.BaseGameScore\nimport com.twitter.frigate.common.base.MagicFanoutSportsEventCandidate\nimport com.twitter.frigate.common.base.MagicFanoutSportsScoreInformation\nimport com.twitter.frigate.common.base.TeamInfo\nimport com.twitter.frigate.magic_events.thriftscala.MagicEventsReason\nimport com.twitter.frigate.pushservice.exception.InvalidSportDomainException\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.pushservice.predicate.magic_fanout.MagicFanoutSportsUtil\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.frigate.thriftscala.MagicFanoutEventNotificationDetails\nimport com.twitter.hermit.store.semantic_core.SemanticEntityForQuery\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nobject MagicFanoutSportsEventCandidateGenerator {\n\n  final def getCandidate(\n    targetUser: Target,\n    notification: FrigateNotification,\n    basketballGameScoreStore: ReadableStore[QualifiedId, BasketballGameLiveUpdate],\n    baseballGameScoreStore: ReadableStore[QualifiedId, BaseballGameLiveUpdate],\n    cricketMatchScoreStore: ReadableStore[QualifiedId, CricketMatchLiveUpdate],\n    soccerMatchScoreStore: ReadableStore[QualifiedId, SoccerMatchLiveUpdate],\n    nflGameScoreStore: ReadableStore[QualifiedId, NflFootballGameLiveUpdate],\n    semanticCoreMegadataStore: ReadableStore[SemanticEntityForQuery, EntityMegadata],\n  ): Future[RawCandidate] = {\n\n    /**\n     * frigateNotification recommendation type should be [[CommonRecommendationType.MagicFanoutSportsEvent]]\n     * AND pushId field should be set\n     *\n     * */\n    require(\n      notification.commonRecommendationType == CommonRecommendationType.MagicFanoutSportsEvent,\n      \"MagicFanoutSports: unexpected CRT \" + notification.commonRecommendationType\n    )\n\n    require(\n      notification.magicFanoutEventNotification.exists(_.pushId.isDefined),\n      \"MagicFanoutSportsEvent: pushId is not defined\")\n\n    val magicFanoutEventNotification = notification.magicFanoutEventNotification.get\n    val eventId = magicFanoutEventNotification.eventId\n    val _isScoreUpdate = magicFanoutEventNotification.isScoreUpdate.getOrElse(false)\n\n    val gameScoresFut: Future[Option[BaseGameScore]] = {\n      if (_isScoreUpdate) {\n        semanticCoreMegadataStore\n          .get(SemanticEntityForQuery(PushConstants.SportsEventDomainId, eventId))\n          .flatMap {\n            case Some(megadata) =>\n              if (megadata.domains.contains(Domains.BasketballGame)) {\n                basketballGameScoreStore\n                  .get(QualifiedId(Domains.BasketballGame.value, eventId)).map {\n                    case Some(game) if game.status.isDefined =>\n                      val status = game.status.get\n                      MagicFanoutSportsUtil.transformToGameScore(game.score, status)\n                    case _ => None\n                  }\n              } else if (megadata.domains.contains(Domains.BaseballGame)) {\n                baseballGameScoreStore\n                  .get(QualifiedId(Domains.BaseballGame.value, eventId)).map {\n                    case Some(game) if game.status.isDefined =>\n                      val status = game.status.get\n                      MagicFanoutSportsUtil.transformToGameScore(game.runs, status)\n                    case _ => None\n                  }\n              } else if (megadata.domains.contains(Domains.NflFootballGame)) {\n                nflGameScoreStore\n                  .get(QualifiedId(Domains.NflFootballGame.value, eventId)).map {\n                    case Some(game) if game.status.isDefined =>\n                      val nflScore = MagicFanoutSportsUtil.transformNFLGameScore(game)\n                      nflScore\n                    case _ => None\n                  }\n              } else if (megadata.domains.contains(Domains.SoccerMatch)) {\n                soccerMatchScoreStore\n                  .get(QualifiedId(Domains.SoccerMatch.value, eventId)).map {\n                    case Some(game) if game.status.isDefined =>\n                      val soccerScore = MagicFanoutSportsUtil.transformSoccerGameScore(game)\n                      soccerScore\n                    case _ => None\n                  }\n              } else {\n                // The domains are not in our list of supported sports\n                throw new InvalidSportDomainException(\n                  s\"Domain for entity ${eventId} is not supported\")\n              }\n            case _ => Future.None\n          }\n      } else Future.None\n    }\n\n    val homeTeamInfoFut: Future[Option[TeamInfo]] = gameScoresFut.flatMap {\n      case Some(gameScore) =>\n        MagicFanoutSportsUtil.getTeamInfo(gameScore.home, semanticCoreMegadataStore)\n      case _ => Future.None\n    }\n\n    val awayTeamInfoFut: Future[Option[TeamInfo]] = gameScoresFut.flatMap {\n      case Some(gameScore) =>\n        MagicFanoutSportsUtil.getTeamInfo(gameScore.away, semanticCoreMegadataStore)\n      case _ => Future.None\n    }\n\n    val candidate = new RawCandidate\n      with MagicFanoutSportsEventCandidate\n      with MagicFanoutSportsScoreInformation {\n\n      override val target: Target = targetUser\n\n      override val eventId: Long = magicFanoutEventNotification.eventId\n\n      override val pushId: Long = magicFanoutEventNotification.pushId.get\n\n      override val candidateMagicEventsReasons: Seq[MagicEventsReason] =\n        magicFanoutEventNotification.eventReasons.getOrElse(Seq.empty)\n\n      override val momentId: Option[Long] = magicFanoutEventNotification.momentId\n\n      override val eventLanguage: Option[String] = magicFanoutEventNotification.eventLanguage\n\n      override val details: Option[MagicFanoutEventNotificationDetails] =\n        magicFanoutEventNotification.details\n\n      override val frigateNotification: FrigateNotification = notification\n\n      override val homeTeamInfo: Future[Option[TeamInfo]] = homeTeamInfoFut\n\n      override val awayTeamInfo: Future[Option[TeamInfo]] = awayTeamInfoFut\n\n      override val gameScores: Future[Option[BaseGameScore]] = gameScoresFut\n\n      override val isScoreUpdate: Boolean = _isScoreUpdate\n    }\n\n    Future.value(candidate)\n\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/send_handler/generator/PushRequestToCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.send_handler.generator\n\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.exception.UnsupportedCrtException\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.frigate.thriftscala.{CommonRecommendationType => CRT}\nimport com.twitter.util.Future\n\nobject PushRequestToCandidate {\n  final def generatePushCandidate(\n    frigateNotification: FrigateNotification,\n    target: Target\n  )(\n    implicit config: Config\n  ): Future[RawCandidate] = {\n\n    val candidateGenerator: (Target, FrigateNotification) => Future[RawCandidate] = {\n      frigateNotification.commonRecommendationType match {\n        case CRT.MagicFanoutNewsEvent => MagicFanoutNewsEventCandidateGenerator.getCandidate\n        case CRT.ScheduledSpaceSubscriber => ScheduledSpaceSubscriberCandidateGenerator.getCandidate\n        case CRT.ScheduledSpaceSpeaker => ScheduledSpaceSpeakerCandidateGenerator.getCandidate\n        case CRT.MagicFanoutSportsEvent =>\n          MagicFanoutSportsEventCandidateGenerator.getCandidate(\n            _,\n            _,\n            config.basketballGameScoreStore,\n            config.baseballGameScoreStore,\n            config.cricketMatchScoreStore,\n            config.soccerMatchScoreStore,\n            config.nflGameScoreStore,\n            config.semanticCoreMegadataStore\n          )\n        case CRT.MagicFanoutProductLaunch =>\n          MagicFanoutProductLaunchCandidateGenerator.getCandidate\n        case CRT.NewCreator =>\n          MagicFanoutCreatorEventCandidateGenerator.getCandidate\n        case CRT.CreatorSubscriber =>\n          MagicFanoutCreatorEventCandidateGenerator.getCandidate\n        case _ =>\n          throw new UnsupportedCrtException(\n            \"UnsupportedCrtException for SendHandler: \" + frigateNotification.commonRecommendationType)\n      }\n    }\n\n    candidateGenerator(target, frigateNotification)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/send_handler/generator/ScheduledSpaceSpeakerCandidateGenerator.scala",
    "content": "package com.twitter.frigate.pushservice.send_handler.generator\n\nimport com.twitter.frigate.common.base.ScheduledSpaceSpeakerCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.util.Future\n\nobject ScheduledSpaceSpeakerCandidateGenerator extends CandidateGenerator {\n\n  override def getCandidate(\n    targetUser: Target,\n    notification: FrigateNotification\n  ): Future[RawCandidate] = {\n\n    /**\n     * frigateNotification recommendation type should be [[CommonRecommendationType.ScheduledSpaceSpeaker]]\n     *\n     **/\n    require(\n      notification.commonRecommendationType == CommonRecommendationType.ScheduledSpaceSpeaker,\n      \"ScheduledSpaceSpeaker: unexpected CRT \" + notification.commonRecommendationType\n    )\n\n    val spaceNotification = notification.spaceNotification.getOrElse(\n      throw new IllegalStateException(\"ScheduledSpaceSpeaker notification object not defined\"))\n\n    require(\n      spaceNotification.hostUserId.isDefined,\n      \"ScheduledSpaceSpeaker notification - hostUserId not defined\"\n    )\n\n    val spaceHostId = spaceNotification.hostUserId\n\n    require(\n      spaceNotification.scheduledStartTime.isDefined,\n      \"ScheduledSpaceSpeaker notification - scheduledStartTime not defined\"\n    )\n\n    val scheduledStartTime = spaceNotification.scheduledStartTime.get\n\n    val candidate = new RawCandidate with ScheduledSpaceSpeakerCandidate {\n      override val target: Target = targetUser\n      override val frigateNotification: FrigateNotification = notification\n      override val spaceId: String = spaceNotification.broadcastId\n      override val hostId: Option[Long] = spaceHostId\n      override val startTime: Long = scheduledStartTime\n      override val speakerIds: Option[Seq[Long]] = spaceNotification.speakers\n      override val listenerIds: Option[Seq[Long]] = spaceNotification.listeners\n    }\n\n    Future.value(candidate)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/send_handler/generator/ScheduledSpaceSubscriberCandidateGenerator.scala",
    "content": "package com.twitter.frigate.pushservice.send_handler.generator\n\nimport com.twitter.frigate.common.base.ScheduledSpaceSubscriberCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.util.Future\n\nobject ScheduledSpaceSubscriberCandidateGenerator extends CandidateGenerator {\n\n  override def getCandidate(\n    targetUser: Target,\n    notification: FrigateNotification\n  ): Future[RawCandidate] = {\n\n    /**\n     * frigateNotification recommendation type should be [[CommonRecommendationType.ScheduledSpaceSubscriber]]\n     *\n     **/\n    require(\n      notification.commonRecommendationType == CommonRecommendationType.ScheduledSpaceSubscriber,\n      \"ScheduledSpaceSubscriber: unexpected CRT \" + notification.commonRecommendationType\n    )\n\n    val spaceNotification = notification.spaceNotification.getOrElse(\n      throw new IllegalStateException(\"ScheduledSpaceSubscriber notification object not defined\"))\n\n    require(\n      spaceNotification.hostUserId.isDefined,\n      \"ScheduledSpaceSubscriber notification - hostUserId not defined\"\n    )\n\n    val spaceHostId = spaceNotification.hostUserId\n\n    require(\n      spaceNotification.scheduledStartTime.isDefined,\n      \"ScheduledSpaceSubscriber notification - scheduledStartTime not defined\"\n    )\n\n    val scheduledStartTime = spaceNotification.scheduledStartTime.get\n\n    val candidate = new RawCandidate with ScheduledSpaceSubscriberCandidate {\n      override val target: Target = targetUser\n      override val frigateNotification: FrigateNotification = notification\n      override val spaceId: String = spaceNotification.broadcastId\n      override val hostId: Option[Long] = spaceHostId\n      override val startTime: Long = scheduledStartTime\n      override val speakerIds: Option[Seq[Long]] = spaceNotification.speakers\n      override val listenerIds: Option[Seq[Long]] = spaceNotification.listeners\n    }\n\n    Future.value(candidate)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/ContentMixerStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.content_mixer.thriftscala.ContentMixer\nimport com.twitter.content_mixer.thriftscala.ContentMixerRequest\nimport com.twitter.content_mixer.thriftscala.ContentMixerResponse\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\ncase class ContentMixerStore(contentMixer: ContentMixer.MethodPerEndpoint)\n    extends ReadableStore[ContentMixerRequest, ContentMixerResponse] {\n\n  override def get(request: ContentMixerRequest): Future[Option[ContentMixerResponse]] = {\n    contentMixer.getCandidates(request).map { response =>\n      Some(response)\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/CopySelectionServiceStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.copyselectionservice.thriftscala._\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nclass CopySelectionServiceStore(copySelectionServiceClient: CopySelectionService.FinagledClient)\n    extends ReadableStore[CopySelectionRequestV1, Copy] {\n  override def get(k: CopySelectionRequestV1): Future[Option[Copy]] =\n    copySelectionServiceClient.getSelectedCopy(CopySelectionRequest.V1(k)).map {\n      case CopySelectionResponse.V1(response) =>\n        Some(response.selectedCopy)\n      case _ => throw CopyServiceException(CopyServiceErrorCode.VersionNotFound)\n    }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/CrMixerTweetStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.cr_mixer.thriftscala.CrMixer\nimport com.twitter.cr_mixer.thriftscala.CrMixerTweetRequest\nimport com.twitter.cr_mixer.thriftscala.CrMixerTweetResponse\nimport com.twitter.cr_mixer.thriftscala.FrsTweetRequest\nimport com.twitter.cr_mixer.thriftscala.FrsTweetResponse\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.util.Future\n\n/**\n * Store to get content recs from content recommender.\n */\ncase class CrMixerTweetStore(\n  crMixer: CrMixer.MethodPerEndpoint\n)(\n  implicit statsReceiver: StatsReceiver = NullStatsReceiver) {\n\n  private val requestsCounter = statsReceiver.counter(\"requests\")\n  private val successCounter = statsReceiver.counter(\"success\")\n  private val failuresCounter = statsReceiver.counter(\"failures\")\n  private val nonEmptyCounter = statsReceiver.counter(\"non_empty\")\n  private val emptyCounter = statsReceiver.counter(\"empty\")\n  private val failuresScope = statsReceiver.scope(\"failures\")\n  private val latencyStat = statsReceiver.stat(\"latency\")\n\n  private def updateStats[T](f: => Future[Option[T]]): Future[Option[T]] = {\n    requestsCounter.incr()\n    Stat\n      .timeFuture(latencyStat)(f)\n      .onSuccess { r =>\n        if (r.isDefined) nonEmptyCounter.incr() else emptyCounter.incr()\n        successCounter.incr()\n      }\n      .onFailure { e =>\n        {\n          failuresCounter.incr()\n          failuresScope.counter(e.getClass.getName).incr()\n        }\n      }\n  }\n\n  def getTweetRecommendations(\n    request: CrMixerTweetRequest\n  ): Future[Option[CrMixerTweetResponse]] = {\n    updateStats(crMixer.getTweetRecommendations(request).map { response =>\n      Some(response)\n    })\n  }\n\n  def getFRSTweetCandidates(request: FrsTweetRequest): Future[Option[FrsTweetResponse]] = {\n    updateStats(crMixer.getFrsBasedTweetRecommendations(request).map { response =>\n      Some(response)\n    })\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/ExploreRankerStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.explore_ranker.thriftscala.ExploreRanker\nimport com.twitter.explore_ranker.thriftscala.ExploreRankerResponse\nimport com.twitter.explore_ranker.thriftscala.ExploreRankerRequest\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\n/** A Store for Video Tweet Recommendations from Explore\n *\n * @param exploreRankerService\n */\ncase class ExploreRankerStore(exploreRankerService: ExploreRanker.MethodPerEndpoint)\n    extends ReadableStore[ExploreRankerRequest, ExploreRankerResponse] {\n\n  /** Method to get video recommendations\n   *\n   * @param request explore ranker request object\n   * @return\n   */\n  override def get(\n    request: ExploreRankerRequest\n  ): Future[Option[ExploreRankerResponse]] = {\n    exploreRankerService.getRankedResults(request).map { response =>\n      Some(response)\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/FollowRecommendationsStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.follow_recommendations.thriftscala.FollowRecommendationsThriftService\nimport com.twitter.follow_recommendations.thriftscala.Recommendation\nimport com.twitter.follow_recommendations.thriftscala.RecommendationRequest\nimport com.twitter.follow_recommendations.thriftscala.RecommendationResponse\nimport com.twitter.follow_recommendations.thriftscala.UserRecommendation\nimport com.twitter.inject.Logging\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\ncase class FollowRecommendationsStore(\n  frsClient: FollowRecommendationsThriftService.MethodPerEndpoint,\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[RecommendationRequest, RecommendationResponse]\n    with Logging {\n\n  private val scopedStats = statsReceiver.scope(getClass.getSimpleName)\n  private val requests = scopedStats.counter(\"requests\")\n  private val valid = scopedStats.counter(\"valid\")\n  private val invalid = scopedStats.counter(\"invalid\")\n  private val numTotalResults = scopedStats.stat(\"total_results\")\n  private val numValidResults = scopedStats.stat(\"valid_results\")\n\n  override def get(request: RecommendationRequest): Future[Option[RecommendationResponse]] = {\n    requests.incr()\n    frsClient.getRecommendations(request).map { response =>\n      numTotalResults.add(response.recommendations.size)\n      val validRecs = response.recommendations.filter {\n        case Recommendation.User(_: UserRecommendation) =>\n          valid.incr()\n          true\n        case _ =>\n          invalid.incr()\n          false\n      }\n\n      numValidResults.add(validRecs.size)\n      Some(\n        RecommendationResponse(\n          recommendations = validRecs\n        ))\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/IbisStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.finagle.stats.BroadcastStatsReceiver\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.logger.MRLogger\nimport com.twitter.frigate.common.store\nimport com.twitter.frigate.common.store.Fail\nimport com.twitter.frigate.common.store.IbisRequestInfo\nimport com.twitter.frigate.common.store.IbisResponse\nimport com.twitter.frigate.common.store.Sent\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.ibis2.service.thriftscala.Flags\nimport com.twitter.ibis2.service.thriftscala.FlowControl\nimport com.twitter.ibis2.service.thriftscala.Ibis2Request\nimport com.twitter.ibis2.service.thriftscala.Ibis2Response\nimport com.twitter.ibis2.service.thriftscala.Ibis2ResponseStatus\nimport com.twitter.ibis2.service.thriftscala.Ibis2Service\nimport com.twitter.ibis2.service.thriftscala.NotificationNotSentCode\nimport com.twitter.ibis2.service.thriftscala.TargetFanoutResult.NotSentReason\nimport com.twitter.util.Future\n\ntrait Ibis2Store extends store.Ibis2Store {\n  def send(ibis2Request: Ibis2Request, candidate: PushCandidate): Future[IbisResponse]\n}\n\ncase class PushIbis2Store(\n  ibisClient: Ibis2Service.MethodPerEndpoint\n)(\n  implicit val statsReceiver: StatsReceiver = NullStatsReceiver)\n    extends Ibis2Store {\n  private val log = MRLogger(this.getClass.getSimpleName)\n  private val stats = statsReceiver.scope(\"ibis_v2_store\")\n  private val statsByCrt = stats.scope(\"byCrt\")\n  private val requestsByCrt = statsByCrt.scope(\"requests\")\n  private val failuresByCrt = statsByCrt.scope(\"failures\")\n  private val successByCrt = statsByCrt.scope(\"success\")\n\n  private val statsByIbisModel = stats.scope(\"byIbisModel\")\n  private val requestsByIbisModel = statsByIbisModel.scope(\"requests\")\n  private val failuresByIbisModel = statsByIbisModel.scope(\"failures\")\n  private val successByIbisModel = statsByIbisModel.scope(\"success\")\n\n  private[this] def ibisSend(\n    ibis2Request: Ibis2Request,\n    commonRecommendationType: CommonRecommendationType\n  ): Future[IbisResponse] = {\n    val ibisModel = ibis2Request.modelName\n\n    val bStats = if (ibis2Request.flags.getOrElse(Flags()).darkWrite.contains(true)) {\n      BroadcastStatsReceiver(\n        Seq(\n          stats,\n          stats.scope(\"dark_write\")\n        )\n      )\n    } else BroadcastStatsReceiver(Seq(stats))\n\n    bStats.counter(\"requests\").incr()\n    requestsByCrt.counter(commonRecommendationType.name).incr()\n    requestsByIbisModel.counter(ibisModel).incr()\n\n    retry(ibisClient, ibis2Request, 3, bStats)\n      .map { response =>\n        bStats.counter(response.status.status.name).incr()\n        successByCrt.counter(response.status.status.name, commonRecommendationType.name).incr()\n        successByIbisModel.counter(response.status.status.name, ibisModel).incr()\n        response.status.status match {\n          case Ibis2ResponseStatus.SuccessWithDeliveries |\n              Ibis2ResponseStatus.SuccessNoDeliveries =>\n            IbisResponse(Sent, Some(response))\n          case _ =>\n            IbisResponse(Fail, Some(response))\n        }\n      }\n      .onFailure { ex =>\n        bStats.counter(\"failures\").incr()\n        val exceptionName = ex.getClass.getCanonicalName\n        bStats.scope(\"failures\").counter(exceptionName).incr()\n        failuresByCrt.counter(exceptionName, commonRecommendationType.name).incr()\n        failuresByIbisModel.counter(exceptionName, ibisModel).incr()\n      }\n  }\n\n  private def getNotifNotSentReason(\n    ibis2Response: Ibis2Response\n  ): Option[NotificationNotSentCode] = {\n    ibis2Response.status.fanoutResults match {\n      case Some(fanoutResult) =>\n        fanoutResult.pushResult.flatMap { pushResult =>\n          pushResult.results.headOption match {\n            case Some(NotSentReason(notSentInfo)) => Some(notSentInfo.notSentCode)\n            case _ => None\n          }\n        }\n      case _ => None\n    }\n  }\n\n  def send(ibis2Request: Ibis2Request, candidate: PushCandidate): Future[IbisResponse] = {\n    val requestWithIID = if (ibis2Request.flowControl.exists(_.externalIid.isDefined)) {\n      ibis2Request\n    } else {\n      ibis2Request.copy(\n        flowControl = Some(\n          ibis2Request.flowControl\n            .getOrElse(FlowControl())\n            .copy(externalIid = Some(candidate.impressionId))\n        )\n      )\n    }\n\n    val commonRecommendationType = candidate.frigateNotification.commonRecommendationType\n\n    ibisSend(requestWithIID, commonRecommendationType)\n      .onSuccess { response =>\n        response.ibis2Response.foreach { ibis2Response =>\n          getNotifNotSentReason(ibis2Response).foreach { notifNotSentCode =>\n            stats.scope(ibis2Response.status.status.name).counter(s\"$notifNotSentCode\").incr()\n          }\n          if (ibis2Response.status.status != Ibis2ResponseStatus.SuccessWithDeliveries) {\n            log.warning(\n              s\"Request dropped on ibis for ${ibis2Request.recipientSelector.recipientId}: $ibis2Response\")\n          }\n        }\n      }\n      .onFailure { ex =>\n        log.warning(\n          s\"Ibis Request failure: ${ex.getClass.getCanonicalName} \\n For IbisRequest: $ibis2Request\")\n        log.error(ex, ex.getMessage)\n      }\n  }\n\n  // retry request when Ibis2ResponseStatus is PreFanoutError\n  def retry(\n    ibisClient: Ibis2Service.MethodPerEndpoint,\n    request: Ibis2Request,\n    retryCount: Int,\n    bStats: StatsReceiver\n  ): Future[Ibis2Response] = {\n    ibisClient.sendNotification(request).flatMap { response =>\n      response.status.status match {\n        case Ibis2ResponseStatus.PreFanoutError if retryCount > 0 =>\n          bStats.scope(\"requests\").counter(\"retry\").incr()\n          bStats.counter(response.status.status.name).incr()\n          retry(ibisClient, request, retryCount - 1, bStats)\n        case _ =>\n          Future.value(response)\n      }\n    }\n  }\n\n  override def send(\n    ibis2Request: Ibis2Request,\n    requestInfo: IbisRequestInfo\n  ): Future[IbisResponse] = {\n    ibisSend(ibis2Request, requestInfo.commonRecommendationType)\n  }\n}\n\ncase class StagingIbis2Store(remoteIbis2Store: PushIbis2Store) extends Ibis2Store {\n\n  final def addDarkWriteFlagIbis2Request(\n    isTeamMember: Boolean,\n    ibis2Request: Ibis2Request\n  ): Ibis2Request = {\n    val flags =\n      ibis2Request.flags.getOrElse(Flags())\n    val darkWrite: Boolean = !isTeamMember || flags.darkWrite.getOrElse(false)\n    ibis2Request.copy(flags = Some(flags.copy(darkWrite = Some(darkWrite))))\n  }\n\n  override def send(ibis2Request: Ibis2Request, candidate: PushCandidate): Future[IbisResponse] = {\n    candidate.target.isTeamMember.flatMap { isTeamMember =>\n      val ibis2Req = addDarkWriteFlagIbis2Request(isTeamMember, ibis2Request)\n      remoteIbis2Store.send(ibis2Req, candidate)\n    }\n  }\n\n  override def send(\n    ibis2Request: Ibis2Request,\n    requestInfo: IbisRequestInfo\n  ): Future[IbisResponse] = {\n    requestInfo.isTeamMember.flatMap { isTeamMember =>\n      val ibis2Req = addDarkWriteFlagIbis2Request(isTeamMember, ibis2Request)\n      remoteIbis2Store.send(ibis2Req, requestInfo)\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/InterestDiscoveryStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.interests_discovery.thriftscala.InterestsDiscoveryService\nimport com.twitter.interests_discovery.thriftscala.RecommendedListsRequest\nimport com.twitter.interests_discovery.thriftscala.RecommendedListsResponse\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\ncase class InterestDiscoveryStore(\n  client: InterestsDiscoveryService.MethodPerEndpoint)\n    extends ReadableStore[RecommendedListsRequest, RecommendedListsResponse] {\n\n  override def get(request: RecommendedListsRequest): Future[Option[RecommendedListsResponse]] = {\n    client.getListRecos(request).map(Some(_))\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/LabeledPushRecsDecideredStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.candidate.TargetDecider\nimport com.twitter.frigate.common.history.History\nimport com.twitter.frigate.common.history.HistoryStoreKeyContext\nimport com.twitter.frigate.common.history.PushServiceHistoryStore\nimport com.twitter.frigate.data_pipeline.thriftscala._\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.hermit.store.labeled_push_recs.LabeledPushRecsJoinedWithNotificationHistoryStore\nimport com.twitter.logging.Logger\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\ncase class LabeledPushRecsVerifyingStoreKey(\n  historyStoreKey: HistoryStoreKeyContext,\n  useHydratedDataset: Boolean,\n  verifyHydratedDatasetResults: Boolean) {\n  def userId: Long = historyStoreKey.targetUserId\n}\n\ncase class LabeledPushRecsVerifyingStoreResponse(\n  userHistory: UserHistoryValue,\n  unequalNotificationsUnhydratedToHydrated: Option[\n    Map[(Time, FrigateNotification), FrigateNotification]\n  ],\n  missingFromHydrated: Option[Map[Time, FrigateNotification]])\n\ncase class LabeledPushRecsVerifyingStore(\n  labeledPushRecsStore: ReadableStore[UserHistoryKey, UserHistoryValue],\n  historyStore: PushServiceHistoryStore\n)(\n  implicit stats: StatsReceiver)\n    extends ReadableStore[LabeledPushRecsVerifyingStoreKey, LabeledPushRecsVerifyingStoreResponse] {\n\n  private def getByJoiningWithRealHistory(\n    key: HistoryStoreKeyContext\n  ): Future[Option[UserHistoryValue]] = {\n    val historyFut = historyStore.get(key, Some(365.days))\n    val toJoinWithRealHistoryFut = labeledPushRecsStore.get(UserHistoryKey.UserId(key.targetUserId))\n    Future.join(historyFut, toJoinWithRealHistoryFut).map {\n      case (_, None) => None\n      case (History(realtimeHistoryMap), Some(uhValue)) =>\n        Some(\n          LabeledPushRecsJoinedWithNotificationHistoryStore\n            .joinLabeledPushRecsSentWithNotificationHistory(uhValue, realtimeHistoryMap, stats)\n        )\n    }\n  }\n\n  private def processUserHistoryValue(uhValue: UserHistoryValue): Map[Time, FrigateNotification] = {\n    uhValue.events\n      .getOrElse(Nil)\n      .collect {\n        case Event(\n              EventType.LabeledPushRecSend,\n              Some(tsMillis),\n              Some(EventUnion.LabeledPushRecSendEvent(lprs: LabeledPushRecSendEvent))\n            ) if lprs.pushRecSendEvent.frigateNotification.isDefined =>\n          Time.fromMilliseconds(tsMillis) -> lprs.pushRecSendEvent.frigateNotification.get\n      }\n      .toMap\n  }\n\n  override def get(\n    key: LabeledPushRecsVerifyingStoreKey\n  ): Future[Option[LabeledPushRecsVerifyingStoreResponse]] = {\n    val uhKey = UserHistoryKey.UserId(key.userId)\n    if (!key.useHydratedDataset) {\n      getByJoiningWithRealHistory(key.historyStoreKey).map { uhValueOpt =>\n        uhValueOpt.map { uhValue => LabeledPushRecsVerifyingStoreResponse(uhValue, None, None) }\n      }\n    } else {\n      labeledPushRecsStore.get(uhKey).flatMap { hydratedValueOpt: Option[UserHistoryValue] =>\n        if (!key.verifyHydratedDatasetResults) {\n          Future.value(hydratedValueOpt.map { uhValue =>\n            LabeledPushRecsVerifyingStoreResponse(uhValue, None, None)\n          })\n        } else {\n          getByJoiningWithRealHistory(key.historyStoreKey).map {\n            joinedWithRealHistoryOpt: Option[UserHistoryValue] =>\n              val joinedWithRealHistoryMap =\n                joinedWithRealHistoryOpt.map(processUserHistoryValue).getOrElse(Map.empty)\n              val hydratedMap = hydratedValueOpt.map(processUserHistoryValue).getOrElse(Map.empty)\n              val unequal = joinedWithRealHistoryMap.flatMap {\n                case (time, frigateNotif) =>\n                  hydratedMap.get(time).collect {\n                    case n if n != frigateNotif => ((time, frigateNotif), n)\n                  }\n              }\n              val missing = joinedWithRealHistoryMap.filter {\n                case (time, frigateNotif) => !hydratedMap.contains(time)\n              }\n              hydratedValueOpt.map { hydratedValue =>\n                LabeledPushRecsVerifyingStoreResponse(hydratedValue, Some(unequal), Some(missing))\n              }\n          }\n        }\n      }\n    }\n  }\n}\n\ncase class LabeledPushRecsStoreKey(target: TargetDecider, historyStoreKey: HistoryStoreKeyContext) {\n  def userId: Long = historyStoreKey.targetUserId\n}\n\ncase class LabeledPushRecsDecideredStore(\n  verifyingStore: ReadableStore[\n    LabeledPushRecsVerifyingStoreKey,\n    LabeledPushRecsVerifyingStoreResponse\n  ],\n  useHydratedLabeledSendsDatasetDeciderKey: String,\n  verifyHydratedLabeledSendsForHistoryDeciderKey: String\n)(\n  implicit globalStats: StatsReceiver)\n    extends ReadableStore[LabeledPushRecsStoreKey, UserHistoryValue] {\n  private val log = Logger()\n  private val stats = globalStats.scope(\"LabeledPushRecsDecideredStore\")\n  private val numComparisons = stats.counter(\"num_comparisons\")\n  private val numMissingStat = stats.stat(\"num_missing\")\n  private val numUnequalStat = stats.stat(\"num_unequal\")\n\n  override def get(key: LabeledPushRecsStoreKey): Future[Option[UserHistoryValue]] = {\n    val useHydrated = key.target.isDeciderEnabled(\n      useHydratedLabeledSendsDatasetDeciderKey,\n      stats,\n      useRandomRecipient = true\n    )\n\n    val verifyHydrated = if (useHydrated) {\n      key.target.isDeciderEnabled(\n        verifyHydratedLabeledSendsForHistoryDeciderKey,\n        stats,\n        useRandomRecipient = true\n      )\n    } else false\n\n    val newKey = LabeledPushRecsVerifyingStoreKey(key.historyStoreKey, useHydrated, verifyHydrated)\n    verifyingStore.get(newKey).map {\n      case None => None\n      case Some(LabeledPushRecsVerifyingStoreResponse(uhValue, unequalOpt, missingOpt)) =>\n        (unequalOpt, missingOpt) match {\n          case (Some(unequal), Some(missing)) =>\n            numComparisons.incr()\n            numMissingStat.add(missing.size)\n            numUnequalStat.add(unequal.size)\n          case _ => //no-op\n        }\n        Some(uhValue)\n    }\n  }\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/LexServiceStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.livevideo.common.ids.EventId\nimport com.twitter.livevideo.timeline.client.v2.LiveVideoTimelineClient\nimport com.twitter.livevideo.timeline.domain.v2.Event\nimport com.twitter.livevideo.timeline.domain.v2.LookupContext\nimport com.twitter.stitch.storehaus.ReadableStoreOfStitch\nimport com.twitter.stitch.NotFound\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\n\ncase class EventRequest(eventId: Long, lookupContext: LookupContext = LookupContext.default)\n\nobject LexServiceStore {\n  def apply(\n    liveVideoTimelineClient: LiveVideoTimelineClient\n  ): ReadableStore[EventRequest, Event] = {\n    ReadableStoreOfStitch { eventRequest =>\n      liveVideoTimelineClient.getEvent(\n        EventId(eventRequest.eventId),\n        eventRequest.lookupContext) rescue {\n        case NotFound => Stitch.NotFound\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/NTabHistoryStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.hermit.store.common.ReadableWritableStore\nimport com.twitter.notificationservice.thriftscala.GenericNotificationOverrideKey\nimport com.twitter.stitch.Stitch\nimport com.twitter.storage.client.manhattan.bijections.Bijections.BinaryCompactScalaInjection\nimport com.twitter.storage.client.manhattan.bijections.Bijections.LongInjection\nimport com.twitter.storage.client.manhattan.bijections.Bijections.StringInjection\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpoint\nimport com.twitter.storage.client.manhattan.kv.impl.Component\nimport com.twitter.storage.client.manhattan.kv.impl.DescriptorP1L1\nimport com.twitter.storage.client.manhattan.kv.impl.KeyDescriptor\nimport com.twitter.storage.client.manhattan.kv.impl.ValueDescriptor\nimport com.twitter.util.Future\n\ncase class NTabHistoryStore(mhEndpoint: ManhattanKVEndpoint, dataset: String)\n    extends ReadableWritableStore[(Long, String), GenericNotificationOverrideKey] {\n\n  private val keyDesc: DescriptorP1L1.EmptyKey[Long, String] =\n    KeyDescriptor(Component(LongInjection), Component(StringInjection))\n\n  private val genericNotifKeyValDesc: ValueDescriptor.EmptyValue[GenericNotificationOverrideKey] =\n    ValueDescriptor[GenericNotificationOverrideKey](\n      BinaryCompactScalaInjection(GenericNotificationOverrideKey)\n    )\n\n  override def get(key: (Long, String)): Future[Option[GenericNotificationOverrideKey]] = {\n    val (userId, impressionId) = key\n    val mhKey = keyDesc.withDataset(dataset).withPkey(userId).withLkey(impressionId)\n\n    Stitch\n      .run(mhEndpoint.get(mhKey, genericNotifKeyValDesc))\n      .map { optionMhValue =>\n        optionMhValue.map(_.contents)\n      }\n  }\n\n  override def put(keyValue: ((Long, String), GenericNotificationOverrideKey)): Future[Unit] = {\n    val ((userId, impressionId), genericNotifOverrideKey) = keyValue\n    val mhKey = keyDesc.withDataset(dataset).withPkey(userId).withLkey(impressionId)\n    val mhVal = genericNotifKeyValDesc.withValue(genericNotifOverrideKey)\n    Stitch.run(mhEndpoint.insert(mhKey, mhVal))\n  }\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/OCFPromptHistoryStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.onboarding.task.service.thriftscala.FatigueFlowEnrollment\nimport com.twitter.stitch.Stitch\nimport com.twitter.storage.client.manhattan.bijections.Bijections.BinaryScalaInjection\nimport com.twitter.storage.client.manhattan.bijections.Bijections.LongInjection\nimport com.twitter.storage.client.manhattan.bijections.Bijections.StringInjection\nimport com.twitter.storage.client.manhattan.kv.impl.Component\nimport com.twitter.storage.client.manhattan.kv.impl.KeyDescriptor\nimport com.twitter.storage.client.manhattan.kv.impl.ValueDescriptor\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClient\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVEndpointBuilder\nimport com.twitter.storage.client.manhattan.kv.NoMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.Omega\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\ncase class OCFHistoryStoreKey(userId: Long, fatigueDuration: Duration, fatigueGroup: String)\n\nclass OCFPromptHistoryStore(\n  manhattanAppId: String,\n  dataset: String,\n  mtlsParams: ManhattanKVClientMtlsParams = NoMtlsParams\n)(\n  implicit stats: StatsReceiver)\n    extends ReadableStore[OCFHistoryStoreKey, FatigueFlowEnrollment] {\n\n  import ManhattanInjections._\n\n  private val client = ManhattanKVClient(\n    appId = manhattanAppId,\n    dest = Omega.wilyName,\n    mtlsParams = mtlsParams,\n    label = \"ocf_history_store\"\n  )\n  private val endpoint = ManhattanKVEndpointBuilder(client, defaultMaxTimeout = 5.seconds)\n    .statsReceiver(stats.scope(\"ocf_history_store\"))\n    .build()\n\n  private val limitResultsTo = 1\n\n  private val datasetKey = keyDesc.withDataset(dataset)\n\n  override def get(storeKey: OCFHistoryStoreKey): Future[Option[FatigueFlowEnrollment]] = {\n    val userId = storeKey.userId\n    val fatigueGroup = storeKey.fatigueGroup\n    val fatigueLength = storeKey.fatigueDuration.inMilliseconds\n    val currentTime = Time.now.inMilliseconds\n    val fullKey = datasetKey\n      .withPkey(userId)\n      .from(fatigueGroup)\n      .to(fatigueGroup, fatigueLength - currentTime)\n\n    Stitch\n      .run(endpoint.slice(fullKey, valDesc, limit = Some(limitResultsTo)))\n      .map { results =>\n        if (results.nonEmpty) {\n          val (_, mhValue) = results.head\n          Some(mhValue.contents)\n        } else None\n      }\n  }\n}\n\nobject ManhattanInjections {\n  val keyDesc = KeyDescriptor(Component(LongInjection), Component(StringInjection, LongInjection))\n  val valDesc = ValueDescriptor(BinaryScalaInjection(FatigueFlowEnrollment))\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/OnlineUserHistoryStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.frigate.common.history.History\nimport com.twitter.frigate.common.store.RealTimeClientEventStore\nimport com.twitter.frigate.data_pipeline.common.HistoryJoin\nimport com.twitter.frigate.data_pipeline.thriftscala.Event\nimport com.twitter.frigate.data_pipeline.thriftscala.EventUnion\nimport com.twitter.frigate.data_pipeline.thriftscala.PushRecSendEvent\nimport com.twitter.frigate.data_pipeline.thriftscala.UserHistoryValue\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\ncase class OnlineUserHistoryKey(\n  userId: Long,\n  offlineUserHistory: Option[UserHistoryValue],\n  history: Option[History])\n\ncase class OnlineUserHistoryStore(\n  realTimeClientEventStore: RealTimeClientEventStore,\n  duration: Duration = 3.days)\n    extends ReadableStore[OnlineUserHistoryKey, UserHistoryValue] {\n\n  override def get(key: OnlineUserHistoryKey): Future[Option[UserHistoryValue]] = {\n    val now = Time.now\n\n    val pushRecSends = key.history\n      .getOrElse(History(Nil.toMap))\n      .sortedPushDmHistory\n      .filter(_._1 > now - (duration + 1.day))\n      .map {\n        case (time, frigateNotification) =>\n          val pushRecSendEvent = PushRecSendEvent(\n            frigateNotification = Some(frigateNotification),\n            impressionId = frigateNotification.impressionId\n          )\n          pushRecSendEvent -> time\n      }\n\n    realTimeClientEventStore\n      .get(key.userId, now - duration, now)\n      .map { attributedEventHistory =>\n        val attributedClientEvents = attributedEventHistory.sortedHistory.flatMap {\n          case (time, event) =>\n            event.eventUnion match {\n              case Some(eventUnion: EventUnion.AttributedPushRecClientEvent) =>\n                Some((eventUnion.attributedPushRecClientEvent, event.eventType, time))\n              case _ => None\n            }\n        }\n\n        val realtimeLabeledSends: Seq[Event] = HistoryJoin.getLabeledPushRecSends(\n          pushRecSends,\n          attributedClientEvents,\n          Seq(),\n          Seq(),\n          Seq(),\n          now\n        )\n\n        key.offlineUserHistory.map { offlineUserHistory =>\n          val combinedEvents = offlineUserHistory.events.map { offlineEvents =>\n            (offlineEvents ++ realtimeLabeledSends)\n              .map { event =>\n                event.timestampMillis -> event\n              }\n              .toMap\n              .values\n              .toSeq\n              .sortBy { event =>\n                -1 * event.timestampMillis.getOrElse(0L)\n              }\n          }\n\n          offlineUserHistory.copy(events = combinedEvents)\n        }\n      }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/OpenAppUserStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.generated.client.rux.open_app.UsersInOpenAppDdgOnUserClientColumn\n\nobject OpenAppUserStore {\n  def apply(stratoClient: Client): ReadableStore[Long, Boolean] = {\n    val fetcher = new UsersInOpenAppDdgOnUserClientColumn(stratoClient).fetcher\n    StratoFetchableStore.withUnitView(fetcher).mapValues(_ => true)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/SocialGraphServiceProcessStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.frigate.pushservice.params.PushQPSLimitConstants.SocialGraphServiceBatchSize\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\ncase class SocialGraphServiceProcessStore(edgeStore: ReadableStore[RelationEdge, Boolean])\n    extends ReadableStore[RelationEdge, Boolean] {\n  override def multiGet[T <: RelationEdge](\n    relationEdges: Set[T]\n  ): Map[T, Future[Option[Boolean]]] = {\n    val splitSet = relationEdges.grouped(SocialGraphServiceBatchSize).toSet\n    splitSet\n      .map { relationship =>\n        edgeStore.multiGet(relationship)\n      }.foldLeft(Map.empty[T, Future[Option[Boolean]]]) { (map1, map2) =>\n        map1 ++ map2\n      }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/SoftUserFollowingStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.gizmoduck.thriftscala.UserType\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.client.UserId\nimport com.twitter.strato.config.FlockCursors.BySource.Begin\nimport com.twitter.strato.config.FlockCursors.Continue\nimport com.twitter.strato.config.FlockCursors.End\nimport com.twitter.strato.config.FlockPage\nimport com.twitter.strato.generated.client.socialgraph.service.soft_users.softUserFollows.EdgeBySourceClientColumn\nimport com.twitter.util.Future\n\nobject SoftUserFollowingStore {\n  type ViewerFollowingCursor = EdgeBySourceClientColumn.Cursor\n  val MaxPagesToFetch = 2\n  val PageLimit = 50\n}\n\nclass SoftUserFollowingStore(stratoClient: Client) extends ReadableStore[User, Seq[Long]] {\n  import SoftUserFollowingStore._\n  private val softUserFollowingEdgesPaginator = new EdgeBySourceClientColumn(stratoClient).paginator\n\n  private def accumulateIds(cursor: ViewerFollowingCursor, pagesToFetch: Int): Stitch[Seq[Long]] =\n    softUserFollowingEdgesPaginator.paginate(cursor).flatMap {\n      case FlockPage(data, next, _) =>\n        next match {\n          case cont: Continue if pagesToFetch > 1 =>\n            Stitch\n              .join(\n                Stitch.value(data.map(_.to).map(_.value)),\n                accumulateIds(cont, pagesToFetch - 1))\n              .map {\n                case (a, b) => a ++ b\n              }\n\n          case _: End | _: Continue =>\n            // end pagination if last page has been fetched or [[MaxPagesToFetch]] have been fetched\n            Stitch.value(data.map(_.to).map(_.value))\n        }\n    }\n\n  private def softFollowingFromStrato(\n    sourceId: Long,\n    pageLimit: Int,\n    pagesToFetch: Int\n  ): Stitch[Seq[Long]] = {\n    val begin = Begin[UserId, UserId](UserId(sourceId), pageLimit)\n    accumulateIds(begin, pagesToFetch)\n  }\n\n  override def get(user: User): Future[Option[Seq[Long]]] = {\n    user.userType match {\n      case UserType.Soft =>\n        Stitch.run(softFollowingFromStrato(user.id, PageLimit, MaxPagesToFetch)).map(Option(_))\n      case _ => Future.None\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/TweetImpressionsStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.util.Future\n\n/**\n * Store to get inbound Tweet impressions count for a specific Tweet id.\n */\nclass TweetImpressionsStore(stratoClient: StratoClient) extends ReadableStore[Long, String] {\n\n  private val column = \"rux/impression.Tweet\"\n  private val store = StratoFetchableStore.withUnitView[Long, String](stratoClient, column)\n\n  def getCounts(tweetId: Long): Future[Option[Long]] = {\n    store.get(tweetId).map(_.map(_.toLong))\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/TweetTranslationStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.context.TwitterContext\nimport com.twitter.context.thriftscala.Viewer\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.TwitterContextPermit\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.kujaku.domain.thriftscala.CacheUsageType\nimport com.twitter.kujaku.domain.thriftscala.MachineTranslation\nimport com.twitter.kujaku.domain.thriftscala.MachineTranslationResponse\nimport com.twitter.kujaku.domain.thriftscala.TranslationSource\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.generated.client.translation.service.IsTweetTranslatableClientColumn\nimport com.twitter.strato.generated.client.translation.service.platform.MachineTranslateTweetClientColumn\nimport com.twitter.tweetypie.thriftscala.Tweet\nimport com.twitter.util.Future\nimport com.twitter.util.logging.Logging\n\nobject TweetTranslationStore {\n  case class Key(\n    target: Target,\n    tweetId: Long,\n    tweet: Option[Tweet],\n    crt: CommonRecommendationType)\n\n  case class Value(\n    translatedTweetText: String,\n    localizedSourceLanguage: String)\n\n  val allowedCRTs = Set[CommonRecommendationType](\n    CommonRecommendationType.TwistlyTweet\n  )\n}\n\ncase class TweetTranslationStore(\n  translateTweetStore: ReadableStore[\n    MachineTranslateTweetClientColumn.Key,\n    MachineTranslationResponse\n  ],\n  isTweetTranslatableStore: ReadableStore[IsTweetTranslatableClientColumn.Key, Boolean],\n  statsReceiver: StatsReceiver)\n    extends ReadableStore[TweetTranslationStore.Key, TweetTranslationStore.Value]\n    with Logging {\n\n  private val stats = statsReceiver.scope(\"tweetTranslationStore\")\n  private val isTranslatableCounter = stats.counter(\"tweetIsTranslatable\")\n  private val notTranslatableCounter = stats.counter(\"tweetIsNotTranslatable\")\n  private val protectedUserCounter = stats.counter(\"protectedUser\")\n  private val notProtectedUserCounter = stats.counter(\"notProtectedUser\")\n  private val validLanguageCounter = stats.counter(\"validTweetLanguage\")\n  private val invalidLanguageCounter = stats.counter(\"invalidTweetLanguage\")\n  private val validCrtCounter = stats.counter(\"validCrt\")\n  private val invalidCrtCounter = stats.counter(\"invalidCrt\")\n  private val paramEnabledCounter = stats.counter(\"paramEnabled\")\n  private val paramDisabledCounter = stats.counter(\"paramDisabled\")\n\n  private val twitterContext = TwitterContext(TwitterContextPermit)\n\n  override def get(k: TweetTranslationStore.Key): Future[Option[TweetTranslationStore.Value]] = {\n    k.target.inferredUserDeviceLanguage.flatMap {\n      case Some(deviceLanguage) =>\n        setTwitterContext(k.target, deviceLanguage) {\n          translateTweet(\n            target = k.target,\n            tweetId = k.tweetId,\n            tweet = k.tweet,\n            crt = k.crt,\n            deviceLanguage = deviceLanguage).map { responseOpt =>\n            responseOpt.flatMap { response =>\n              response.translatorLocalizedSourceLanguage\n                .map { localizedSourceLanguage =>\n                  TweetTranslationStore.Value(\n                    translatedTweetText = response.translation,\n                    localizedSourceLanguage = localizedSourceLanguage\n                  )\n                }.filter { _ =>\n                  response.translationSource == TranslationSource.Google\n                }\n            }\n          }\n        }\n      case None => Future.None\n    }\n\n  }\n\n  // Don't sent protected tweets to external API for translation\n  private def checkProtectedUser(target: Target): Future[Boolean] = {\n    target.targetUser.map(_.flatMap(_.safety).forall(_.isProtected)).onSuccess {\n      case true => protectedUserCounter.incr()\n      case false => notProtectedUserCounter.incr()\n    }\n  }\n\n  private def isTweetTranslatable(\n    target: Target,\n    tweetId: Long,\n    tweet: Option[Tweet],\n    crt: CommonRecommendationType,\n    deviceLanguage: String\n  ): Future[Boolean] = {\n    val tweetLangOpt = tweet.flatMap(_.language)\n    val isValidLanguage = tweetLangOpt.exists { tweetLang =>\n      tweetLang.confidence > 0.5 &&\n      tweetLang.language != deviceLanguage\n    }\n\n    if (isValidLanguage) {\n      validLanguageCounter.incr()\n    } else {\n      invalidLanguageCounter.incr()\n    }\n\n    val isValidCrt = TweetTranslationStore.allowedCRTs.contains(crt)\n    if (isValidCrt) {\n      validCrtCounter.incr()\n    } else {\n      invalidCrtCounter.incr()\n    }\n\n    if (isValidCrt && isValidLanguage && target.params(PushParams.EnableIsTweetTranslatableCheck)) {\n      checkProtectedUser(target).flatMap {\n        case false =>\n          val isTweetTranslatableKey = IsTweetTranslatableClientColumn.Key(\n            tweetId = tweetId,\n            destinationLanguage = Some(deviceLanguage),\n            translationSource = Some(TranslationSource.Google.name),\n            excludePreferredLanguages = Some(true)\n          )\n          isTweetTranslatableStore\n            .get(isTweetTranslatableKey).map { resultOpt =>\n              resultOpt.getOrElse(false)\n            }.onSuccess {\n              case true => isTranslatableCounter.incr()\n              case false => notTranslatableCounter.incr()\n            }\n        case true =>\n          Future.False\n      }\n    } else {\n      Future.False\n    }\n  }\n\n  private def translateTweet(\n    tweetId: Long,\n    deviceLanguage: String\n  ): Future[Option[MachineTranslation]] = {\n    val translateKey = MachineTranslateTweetClientColumn.Key(\n      tweetId = tweetId,\n      destinationLanguage = deviceLanguage,\n      translationSource = TranslationSource.Google,\n      translatableEntityTypes = Seq(),\n      onlyCached = false,\n      cacheUsageType = CacheUsageType.Default\n    )\n    translateTweetStore.get(translateKey).map {\n      _.collect {\n        case MachineTranslationResponse.Result(result) => result\n      }\n    }\n  }\n\n  private def translateTweet(\n    target: Target,\n    tweetId: Long,\n    tweet: Option[Tweet],\n    crt: CommonRecommendationType,\n    deviceLanguage: String\n  ): Future[Option[MachineTranslation]] = {\n    isTweetTranslatable(target, tweetId, tweet, crt, deviceLanguage).flatMap {\n      case true =>\n        val isEnabledByParam = target.params(PushFeatureSwitchParams.EnableTweetTranslation)\n        if (isEnabledByParam) {\n          paramEnabledCounter.incr()\n          translateTweet(tweetId, deviceLanguage)\n        } else {\n          paramDisabledCounter.incr()\n          Future.None\n        }\n      case false =>\n        Future.None\n    }\n  }\n\n  private def setTwitterContext[Rep](\n    target: Target,\n    deviceLanguage: String\n  )(\n    f: => Future[Rep]\n  ): Future[Rep] = {\n    twitterContext() match {\n      case Some(viewer) if viewer.userId.nonEmpty && viewer.authenticatedUserId.nonEmpty =>\n        // If the context is already setup with a user ID just use it\n        f\n      case _ =>\n        // If not, create a new context containing the viewer user id\n        twitterContext.let(\n          Viewer(\n            userId = Some(target.targetId),\n            requestLanguageCode = Some(deviceLanguage),\n            authenticatedUserId = Some(target.targetId)\n          )) {\n          f\n        }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/store/UttEntityHydrationStore.scala",
    "content": "package com.twitter.frigate.pushservice.store\n\nimport com.twitter.escherbird.util.uttclient.CachedUttClientV2\nimport com.twitter.escherbird.util.uttclient.InvalidUttEntityException\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.stitch.Stitch\nimport com.twitter.topiclisting.TopicListingViewerContext\nimport com.twitter.topiclisting.utt.LocalizedEntity\nimport com.twitter.topiclisting.utt.LocalizedEntityFactory\nimport com.twitter.util.Future\n\n/**\n *\n * @param viewerContext: [[TopicListingViewerContext]] for filtering topic\n * @param semanticCoreEntityIds: list of semantic core entities to hydrate\n */\ncase class UttEntityHydrationQuery(\n  viewerContext: TopicListingViewerContext,\n  semanticCoreEntityIds: Seq[Long])\n\n/**\n *\n * @param cachedUttClientV2\n * @param statsReceiver\n */\nclass UttEntityHydrationStore(\n  cachedUttClientV2: CachedUttClientV2,\n  statsReceiver: StatsReceiver,\n  log: Logger) {\n\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val uttEntityNotFound = stats.counter(\"invalid_utt_entity\")\n  private val deviceLanguageMismatch = stats.counter(\"language_mismatch\")\n\n  /**\n   * SemanticCore recommends setting language and country code to None to fetch all localized topic\n   * names and apply filtering for locales on our end\n   *\n   * We use [[LocalizedEntityFactory]] from [[Topiclisting]] library to filter out topic name based\n   * on user locale\n   *\n   * Some(LocalizedEntity) - LocalizedUttEntity found\n   * None - LocalizedUttEntity not found\n   */\n  def getLocalizedTopicEntities(\n    query: UttEntityHydrationQuery\n  ): Future[Seq[Option[LocalizedEntity]]] = Stitch.run {\n    Stitch.collect {\n      query.semanticCoreEntityIds.map { semanticCoreEntityId =>\n        val uttEntity = cachedUttClientV2.cachedGetUttEntity(\n          language = None,\n          country = None,\n          version = None,\n          entityId = semanticCoreEntityId)\n\n        uttEntity\n          .map { uttEntityMetadata =>\n            val localizedEntity = LocalizedEntityFactory.getLocalizedEntity(\n              uttEntityMetadata,\n              query.viewerContext,\n              enableInternationalTopics = true,\n              enableTopicDescription = true)\n            // update counter\n            localizedEntity.foreach { entity =>\n              if (!entity.nameMatchesDeviceLanguage) deviceLanguageMismatch.incr()\n            }\n\n            localizedEntity\n          }.handle {\n            case e: InvalidUttEntityException =>\n              log.error(e.getMessage)\n              uttEntityNotFound.incr()\n              None\n          }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/CandidateNotifier.scala",
    "content": "package com.twitter.frigate.pushservice.take\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.Stats.track\nimport com.twitter.frigate.common.logger.MRLogger\nimport com.twitter.frigate.common.store.Fail\nimport com.twitter.frigate.common.store.IbisResponse\nimport com.twitter.frigate.common.store.InvalidConfiguration\nimport com.twitter.frigate.common.store.NoRequest\nimport com.twitter.frigate.common.store.Sent\nimport com.twitter.frigate.common.util.CasLock\nimport com.twitter.frigate.common.util.PushServiceUtil.InvalidConfigResponse\nimport com.twitter.frigate.common.util.PushServiceUtil.NtabWriteOnlyResponse\nimport com.twitter.frigate.common.util.PushServiceUtil.SendFailedResponse\nimport com.twitter.frigate.common.util.PushServiceUtil.SentResponse\nimport com.twitter.frigate.pushservice.predicate.CasLockPredicate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.take.history._\nimport com.twitter.frigate.pushservice.util.CopyUtil\nimport com.twitter.frigate.pushservice.thriftscala.PushResponse\nimport com.twitter.frigate.pushservice.thriftscala.PushStatus\nimport com.twitter.frigate.pushservice.util.OverrideNotificationUtil\nimport com.twitter.frigate.thriftscala.ChannelName\nimport com.twitter.util.Future\n\nclass CandidateNotifier(\n  notificationSender: NotificationSender,\n  casLock: CasLock,\n  historyWriter: HistoryWriter,\n  eventBusWriter: EventBusWriter,\n  ntabOnlyChannelSelector: NtabOnlyChannelSelector\n)(\n  implicit statsReceiver: StatsReceiver) {\n\n  private lazy val casLockPredicate =\n    CasLockPredicate(casLock, expiryDuration = 10.minutes)(statsReceiver)\n  private val candidateNotifierStats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val historyWriteCounter =\n    candidateNotifierStats.counter(\"simply_notifier_history_write_num\")\n  private val loggedOutHistoryWriteCounter =\n    candidateNotifierStats.counter(\"logged_out_simply_notifier_history_write_num\")\n  private val notificationSenderLatency =\n    candidateNotifierStats.scope(\"notification_sender_send\")\n  private val log = MRLogger(\"CandidateNotifier\")\n\n  private def mapIbisResponse(ibisResponse: IbisResponse): PushResponse = {\n    ibisResponse match {\n      case IbisResponse(Sent, _) => SentResponse\n      case IbisResponse(Fail, _) => SendFailedResponse\n      case IbisResponse(InvalidConfiguration, _) => InvalidConfigResponse\n      case IbisResponse(NoRequest, _) => NtabWriteOnlyResponse\n    }\n  }\n\n  /**\n   * - write to history store\n   * - send the notification\n   * - scribe the notification\n   *\n   * final modifier is to signal that this function cannot be overriden. There's some critical logic\n   * in this function, and it's helpful to know that no sub-class overrides it.\n   */\n  final def notify(\n    candidate: PushCandidate,\n  ): Future[PushResponse] = {\n    if (candidate.target.isDarkWrite) {\n      notificationSender.sendIbisDarkWrite(candidate).map(mapIbisResponse)\n    } else {\n      casLockPredicate(Seq(candidate)).flatMap { casLockResults =>\n        if (casLockResults.head || candidate.target.pushContext\n            .exists(_.skipFilters.contains(true))) {\n          Future\n            .join(\n              candidate.target.isSilentPush,\n              OverrideNotificationUtil\n                .getOverrideInfo(candidate, candidateNotifierStats),\n              CopyUtil.getCopyFeatures(candidate, candidateNotifierStats)\n            ).flatMap {\n              case (isSilentPush, overrideInfoOpt, copyFeaturesMap) =>\n                val channels = ntabOnlyChannelSelector.selectChannel(candidate)\n                channels.flatMap { channels =>\n                  candidate\n                    .frigateNotificationForPersistence(\n                      channels,\n                      isSilentPush,\n                      overrideInfoOpt,\n                      copyFeaturesMap.keySet).flatMap { frigateNotificationForPersistence =>\n                      val result = if (candidate.target.isDarkWrite) {\n                        candidateNotifierStats.counter(\"dark_write\").incr()\n                        Future.Unit\n                      } else {\n                        historyWriteCounter.incr()\n                        historyWriter\n                          .writeSendToHistory(candidate, frigateNotificationForPersistence)\n                      }\n                      result.flatMap { _ =>\n                        track(notificationSenderLatency)(\n                          notificationSender\n                            .notify(channels, candidate)\n                            .map { ibisResponse =>\n                              eventBusWriter\n                                .writeToEventBus(candidate, frigateNotificationForPersistence)\n                              mapIbisResponse(ibisResponse)\n                            })\n                      }\n                    }\n                }\n            }\n        } else {\n          candidateNotifierStats.counter(\"filtered_by_cas_lock\").incr()\n          Future.value(PushResponse(PushStatus.Filtered, Some(casLockPredicate.name)))\n        }\n      }\n    }\n  }\n\n  final def loggedOutNotify(\n    candidate: PushCandidate,\n  ): Future[PushResponse] = {\n    if (candidate.target.isDarkWrite) {\n      notificationSender.sendIbisDarkWrite(candidate).map(mapIbisResponse)\n    } else {\n      casLockPredicate(Seq(candidate)).flatMap { casLockResults =>\n        if (casLockResults.head || candidate.target.pushContext\n            .exists(_.skipFilters.contains(true))) {\n          val response = candidate.target.isSilentPush.flatMap { isSilentPush =>\n            candidate\n              .frigateNotificationForPersistence(\n                Seq(ChannelName.PushNtab),\n                isSilentPush,\n                None,\n                Set.empty).flatMap { frigateNotificationForPersistence =>\n                val result = if (candidate.target.isDarkWrite) {\n                  candidateNotifierStats.counter(\"logged_out_dark_write\").incr()\n                  Future.Unit\n                } else {\n                  loggedOutHistoryWriteCounter.incr()\n                  historyWriter.writeSendToHistory(candidate, frigateNotificationForPersistence)\n                }\n\n                result.flatMap { _ =>\n                  track(notificationSenderLatency)(\n                    notificationSender\n                      .loggedOutNotify(candidate)\n                      .map { ibisResponse =>\n                        mapIbisResponse(ibisResponse)\n                      })\n                }\n              }\n          }\n          response\n        } else {\n          candidateNotifierStats.counter(\"filtered_by_cas_lock\").incr()\n          Future.value(PushResponse(PushStatus.Filtered, Some(casLockPredicate.name)))\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/LoggedOutRefreshForPushNotifier.scala",
    "content": "package com.twitter.frigate.pushservice.take\n\nimport com.twitter.finagle.stats.BroadcastStatsReceiver\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateResult\nimport com.twitter.frigate.common.base.Invalid\nimport com.twitter.frigate.common.base.OK\nimport com.twitter.frigate.common.base.Response\nimport com.twitter.frigate.common.base.Result\nimport com.twitter.frigate.common.base.Stats.track\nimport com.twitter.frigate.common.config.CommonConstants\nimport com.twitter.frigate.common.logger.MRLogger\nimport com.twitter.frigate.common.util.PushServiceUtil.FilteredLoggedOutResponseFut\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.refresh_handler.RFPHStatsRecorder\nimport com.twitter.frigate.pushservice.thriftscala.LoggedOutResponse\nimport com.twitter.frigate.pushservice.thriftscala.PushStatus\nimport com.twitter.util.Future\nimport com.twitter.util.JavaTimer\nimport com.twitter.util.Timer\n\nclass LoggedOutRefreshForPushNotifier(\n  rfphStatsRecorder: RFPHStatsRecorder,\n  loCandidateNotifier: CandidateNotifier\n)(\n  globalStats: StatsReceiver) {\n  private implicit val statsReceiver: StatsReceiver =\n    globalStats.scope(\"LoggedOutRefreshForPushHandler\")\n  private val loPushStats: StatsReceiver = statsReceiver.scope(\"logged_out_push\")\n  private val loSendLatency: StatsReceiver = statsReceiver.scope(\"logged_out_send\")\n  private val processedCandidatesCounter: Counter =\n    statsReceiver.counter(\"processed_candidates_count\")\n  private val validCandidatesCounter: Counter = statsReceiver.counter(\"valid_candidates_count\")\n  private val okayCandidateCounter: Counter = statsReceiver.counter(\"ok_candidate_count\")\n  private val nonOkayCandidateCounter: Counter = statsReceiver.counter(\"non_ok_candidate_count\")\n  private val successNotifyCounter: Counter = statsReceiver.counter(\"success_notify_count\")\n  private val notifyCandidate: Counter = statsReceiver.counter(\"notify_candidate\")\n  private val noneCandidateResultCounter: Counter = statsReceiver.counter(\"none_candidate_count\")\n  private val nonOkayPredsResult: Counter = statsReceiver.counter(\"non_okay_preds_result\")\n  private val invalidResultCounter: Counter = statsReceiver.counter(\"invalid_result_count\")\n  private val filteredLoggedOutResponse: Counter = statsReceiver.counter(\"filtered_response_count\")\n\n  implicit private val timer: Timer = new JavaTimer(true)\n  val log = MRLogger(\"LoggedOutRefreshForNotifier\")\n\n  private def notify(\n    candidatesResult: CandidateResult[PushCandidate, Result]\n  ): Future[LoggedOutResponse] = {\n    val candidate = candidatesResult.candidate\n    if (candidate != null)\n      notifyCandidate.incr()\n    val predsResult = candidatesResult.result\n    if (predsResult != OK) {\n      nonOkayPredsResult.incr()\n      val invalidResult = predsResult\n      invalidResult match {\n        case Invalid(Some(reason)) =>\n          invalidResultCounter.incr()\n          Future.value(LoggedOutResponse(PushStatus.Filtered, Some(reason)))\n        case _ =>\n          filteredLoggedOutResponse.incr()\n          Future.value(LoggedOutResponse(PushStatus.Filtered, None))\n      }\n    } else {\n      track(loSendLatency)(loCandidateNotifier.loggedOutNotify(candidate).map { res =>\n        LoggedOutResponse(res.status)\n      })\n    }\n  }\n\n  def checkResponseAndNotify(\n    response: Response[PushCandidate, Result]\n  ): Future[LoggedOutResponse] = {\n    val receivers = Seq(statsReceiver)\n    val loggedOutResponse = response match {\n      case Response(OK, processedCandidates) =>\n        processedCandidatesCounter.incr(processedCandidates.size)\n        val validCandidates = processedCandidates.filter(_.result == OK)\n        validCandidatesCounter.incr(validCandidates.size)\n\n        validCandidates.headOption match {\n          case Some(candidatesResult) =>\n            candidatesResult.result match {\n              case OK =>\n                okayCandidateCounter.incr()\n                notify(candidatesResult)\n                  .onSuccess { nr =>\n                    successNotifyCounter.incr()\n                    loPushStats.scope(\"lo_result\").counter(nr.status.name).incr()\n                  }\n              case _ =>\n                nonOkayCandidateCounter.incr()\n                FilteredLoggedOutResponseFut\n            }\n          case _ =>\n            noneCandidateResultCounter.incr()\n            FilteredLoggedOutResponseFut\n        }\n\n      case Response(Invalid(reason), _) =>\n        FilteredLoggedOutResponseFut.map(_.copy(filteredBy = reason))\n\n      case _ =>\n        FilteredLoggedOutResponseFut\n    }\n    val bstats = BroadcastStatsReceiver(receivers)\n    Stat\n      .timeFuture(bstats.stat(\"logged_out_latency\"))(\n        loggedOutResponse.raiseWithin(CommonConstants.maxPushRequestDuration)\n      )\n      .onFailure { exception =>\n        rfphStatsRecorder.loggedOutRequestExceptionStats(exception, bstats)\n      }\n    loggedOutResponse\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/NotificationSender.scala",
    "content": "package com.twitter.frigate.pushservice.take\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.Stats.track\nimport com.twitter.frigate.common.store.IbisResponse\nimport com.twitter.frigate.common.store.Sent\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.take.sender.Ibis2Sender\nimport com.twitter.frigate.pushservice.take.sender.NtabSender\nimport com.twitter.frigate.scribe.thriftscala.NotificationScribe\nimport com.twitter.util.Future\nimport com.twitter.frigate.thriftscala.ChannelName\n\n/**\n * NotificationSender wraps up all the notification infra send logic, and serves as an abstract layer\n * between CandidateNotifier and the respective senders including ntab, ibis, which is being\n * gated with both a decider/feature switch\n */\nclass NotificationSender(\n  ibis2Sender: Ibis2Sender,\n  ntabSender: NtabSender,\n  statsReceiver: StatsReceiver,\n  notificationScribe: NotificationScribe => Unit) {\n\n  private val notificationNotifierStats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val ibis2SendLatency = notificationNotifierStats.scope(\"ibis2_send\")\n  private val loggedOutIbis2SendLatency = notificationNotifierStats.scope(\"logged_out_ibis2_send\")\n  private val ntabSendLatency = notificationNotifierStats.scope(\"ntab_send\")\n\n  private val ntabWriteThenSkipPushCounter =\n    notificationNotifierStats.counter(\"ntab_write_then_skip_push\")\n  private val ntabWriteThenIbisSendCounter =\n    notificationNotifierStats.counter(\"ntab_write_then_ibis_send\")\n  notificationNotifierStats.counter(\"ins_dark_traffic_send\")\n\n  private val ntabOnlyChannelSenderV3Counter =\n    notificationNotifierStats.counter(\"ntab_only_channel_send_v3\")\n\n  def sendIbisDarkWrite(candidate: PushCandidate): Future[IbisResponse] = {\n    ibis2Sender.sendAsDarkWrite(candidate)\n  }\n\n  private def isNtabOnlySend(\n    channels: Seq[ChannelName]\n  ): Future[Boolean] = {\n    val isNtabOnlyChannel = channels.contains(ChannelName.NtabOnly)\n    if (isNtabOnlyChannel) ntabOnlyChannelSenderV3Counter.incr()\n\n    Future.value(isNtabOnlyChannel)\n  }\n\n  private def isPushOnly(channels: Seq[ChannelName], candidate: PushCandidate): Future[Boolean] = {\n    Future.value(channels.contains(ChannelName.PushOnly))\n  }\n\n  def notify(\n    channels: Seq[ChannelName],\n    candidate: PushCandidate\n  ): Future[IbisResponse] = {\n    Future\n      .join(isPushOnly(channels, candidate), isNtabOnlySend(channels)).map {\n        case (isPushOnly, isNtabOnly) =>\n          if (isPushOnly) {\n            track(ibis2SendLatency)(ibis2Sender.send(channels, candidate, notificationScribe, None))\n          } else {\n            track(ntabSendLatency)(\n              ntabSender\n                .send(candidate, isNtabOnly))\n              .flatMap { ntabResponse =>\n                if (isNtabOnly) {\n                  ntabWriteThenSkipPushCounter.incr()\n                  candidate\n                    .scribeData(channels = channels).foreach(notificationScribe).map(_ =>\n                      IbisResponse(Sent))\n                } else {\n                  ntabWriteThenIbisSendCounter.incr()\n                  track(ibis2SendLatency)(\n                    ibis2Sender.send(channels, candidate, notificationScribe, ntabResponse))\n                }\n              }\n\n          }\n      }.flatten\n  }\n\n  def loggedOutNotify(\n    candidate: PushCandidate\n  ): Future[IbisResponse] = {\n    val ibisResponse = {\n      track(loggedOutIbis2SendLatency)(\n        ibis2Sender.send(Seq(ChannelName.PushNtab), candidate, notificationScribe, None))\n    }\n    ibisResponse\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/NotificationServiceSender.scala",
    "content": "package com.twitter.frigate.pushservice.take\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.logger.MRLogger\nimport com.twitter.frigate.common.ntab.InvalidNTABWriteRequestException\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.notificationservice.thriftscala._\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.util.Future\nimport scala.util.control.NoStackTrace\n\nclass NtabCopyIdNotFoundException(private val message: String)\n    extends Exception(message)\n    with NoStackTrace\n\nclass InvalidNtabCopyIdException(private val message: String)\n    extends Exception(message)\n    with NoStackTrace\n\nobject NotificationServiceSender {\n\n  def generateSocialContextTextEntities(\n    ntabDisplayNamesAndIdsFut: Future[Seq[(String, Long)]],\n    otherCountFut: Future[Int]\n  ): Future[Seq[DisplayTextEntity]] = {\n    Future.join(ntabDisplayNamesAndIdsFut, otherCountFut).map {\n      case (namesWithIdInOrder, otherCount) =>\n        val displays = namesWithIdInOrder.zipWithIndex.map {\n          case ((name, id), index) =>\n            DisplayTextEntity(\n              name = \"user\" + s\"${index + 1}\",\n              value = TextValue.Text(name),\n              emphasis = true,\n              userId = Some(id)\n            )\n        } ++ Seq(\n          DisplayTextEntity(name = \"nameCount\", value = TextValue.Number(namesWithIdInOrder.size))\n        )\n\n        val otherDisplay = if (otherCount > 0) {\n          Some(\n            DisplayTextEntity(\n              name = \"otherCount\",\n              value = TextValue.Number(otherCount)\n            )\n          )\n        } else None\n        displays ++ otherDisplay\n    }\n  }\n\n  def getDisplayTextEntityFromUser(\n    userOpt: Option[User],\n    fieldName: String,\n    isBold: Boolean\n  ): Option[DisplayTextEntity] = {\n    for {\n      user <- userOpt\n      profile <- user.profile\n    } yield {\n      DisplayTextEntity(\n        name = fieldName,\n        value = TextValue.Text(profile.name),\n        emphasis = isBold,\n        userId = Some(user.id)\n      )\n    }\n  }\n\n  def getDisplayTextEntityFromUser(\n    user: Future[Option[User]],\n    fieldName: String,\n    isBold: Boolean\n  ): Future[Option[DisplayTextEntity]] = {\n    user.map { getDisplayTextEntityFromUser(_, fieldName, isBold) }\n  }\n}\n\ncase class NotificationServiceRequest(\n  candidate: PushCandidate,\n  impressionId: String,\n  isBadgeUpdate: Boolean,\n  overrideId: Option[String] = None)\n\nclass NotificationServiceSender(\n  send: (Target, CreateGenericNotificationRequest) => Future[CreateGenericNotificationResponse],\n  enableWritesParam: Param[Boolean],\n  enableForEmployeesParam: Param[Boolean],\n  enableForEveryoneParam: Param[Boolean]\n)(\n  implicit globalStats: StatsReceiver)\n    extends ReadableStore[NotificationServiceRequest, CreateGenericNotificationResponse] {\n\n  val log = MRLogger(this.getClass.getName)\n\n  val stats = globalStats.scope(\"NotificationServiceSender\")\n  val requestEmpty = stats.scope(\"request_empty\")\n  val requestNonEmpty = stats.counter(\"request_non_empty\")\n\n  val requestBadgeCount = stats.counter(\"request_badge_count\")\n\n  val successfulWrite = stats.counter(\"successful_write\")\n  val successfulWriteScope = stats.scope(\"successful_write\")\n  val failedWriteScope = stats.scope(\"failed_write\")\n  val gotNonSuccessResponse = stats.counter(\"got_non_success_response\")\n  val gotEmptyResponse = stats.counter(\"got_empty_response\")\n  val deciderTurnedOffResponse = stats.scope(\"decider_turned_off_response\")\n\n  val disabledByDeciderForCandidate = stats.scope(\"model/candidate\").counter(\"disabled_by_decider\")\n  val sentToAlphaUserForCandidate =\n    stats.scope(\"model/candidate\").counter(\"send_to_employee_or_team\")\n  val sentToNonBucketedUserForCandidate =\n    stats.scope(\"model/candidate\").counter(\"send_to_non_bucketed_decidered_user\")\n  val noSendForCandidate = stats.scope(\"model/candidate\").counter(\"no_send\")\n\n  val ineligibleUsersForCandidate = stats.scope(\"model/candidate\").counter(\"ineligible_users\")\n\n  val darkWriteRequestsForCandidate = stats.scope(\"model/candidate\").counter(\"dark_write_traffic\")\n\n  val heavyUserForCandidateCounter = stats.scope(\"model/candidate\").counter(\"target_heavy\")\n  val nonHeavyUserForCandidateCounter = stats.scope(\"model/candidate\").counter(\"target_non_heavy\")\n\n  val skipWritingToNTAB = stats.counter(\"skip_writing_to_ntab\")\n\n  val ntabWriteDisabledForCandidate = stats.scope(\"model/candidate\").counter(\"ntab_write_disabled\")\n\n  val ntabOverrideEnabledForCandidate = stats.scope(\"model/candidate\").counter(\"override_enabled\")\n  val ntabTTLForCandidate = stats.scope(\"model/candidate\").counter(\"ttl_enabled\")\n\n  override def get(\n    notifRequest: NotificationServiceRequest\n  ): Future[Option[CreateGenericNotificationResponse]] = {\n    notifRequest.candidate.target.deviceInfo.flatMap { deviceInfoOpt =>\n      val disableWritingToNtab =\n        notifRequest.candidate.target.params(PushParams.DisableWritingToNTAB)\n\n      if (disableWritingToNtab) {\n        skipWritingToNTAB.incr()\n        Future.None\n      } else {\n        if (notifRequest.overrideId.nonEmpty) { ntabOverrideEnabledForCandidate.incr() }\n        Future\n          .join(\n            notifRequest.candidate.ntabRequest,\n            ntabWritesEnabledForCandidate(notifRequest.candidate)).flatMap {\n            case (Some(ntabRequest), ntabWritesEnabled) if ntabWritesEnabled =>\n              if (ntabRequest.expiryTimeMillis.nonEmpty) { ntabTTLForCandidate.incr() }\n              sendNTabRequest(\n                ntabRequest,\n                notifRequest.candidate.target,\n                notifRequest.isBadgeUpdate,\n                notifRequest.candidate.commonRecType,\n                isFromCandidate = true,\n                overrideId = notifRequest.overrideId\n              )\n            case (Some(_), ntabWritesEnabled) if !ntabWritesEnabled =>\n              ntabWriteDisabledForCandidate.incr()\n              Future.None\n            case (None, ntabWritesEnabled) =>\n              if (!ntabWritesEnabled) ntabWriteDisabledForCandidate.incr()\n              requestEmpty.counter(s\"candidate_${notifRequest.candidate.commonRecType}\").incr()\n              Future.None\n          }\n      }\n    }\n  }\n\n  private def sendNTabRequest(\n    genericNotificationRequest: CreateGenericNotificationRequest,\n    target: Target,\n    isBadgeUpdate: Boolean,\n    crt: CommonRecommendationType,\n    isFromCandidate: Boolean,\n    overrideId: Option[String]\n  ): Future[Option[CreateGenericNotificationResponse]] = {\n    requestNonEmpty.incr()\n    val notifSvcReq =\n      genericNotificationRequest.copy(\n        sendBadgeCountUpdate = isBadgeUpdate,\n        overrideId = overrideId\n      )\n    requestBadgeCount.incr()\n    send(target, notifSvcReq)\n      .map { response =>\n        if (response.responseType.equals(CreateGenericNotificationResponseType.DecideredOff)) {\n          deciderTurnedOffResponse.counter(s\"$crt\").incr()\n          deciderTurnedOffResponse.counter(s\"${genericNotificationRequest.genericType}\").incr()\n          throw InvalidNTABWriteRequestException(\"Decider is turned off\")\n        } else {\n          Some(response)\n        }\n      }\n      .onFailure { ex =>\n        stats.counter(s\"error_${ex.getClass.getCanonicalName}\").incr()\n        failedWriteScope.counter(s\"${crt}\").incr()\n        log\n          .error(\n            ex,\n            s\"NTAB failure $notifSvcReq\"\n          )\n      }\n      .onSuccess {\n        case Some(response) =>\n          successfulWrite.incr()\n          val successfulWriteScopeString = if (isFromCandidate) \"model/candidate\" else \"envelope\"\n          successfulWriteScope.scope(successfulWriteScopeString).counter(s\"$crt\").incr()\n          if (response.responseType != CreateGenericNotificationResponseType.Success) {\n            gotNonSuccessResponse.incr()\n            log.warning(s\"NTAB dropped $notifSvcReq with response $response\")\n          }\n\n        case _ =>\n          gotEmptyResponse.incr()\n      }\n  }\n\n  private def ntabWritesEnabledForCandidate(cand: PushCandidate): Future[Boolean] = {\n    if (!cand.target.params(enableWritesParam)) {\n      disabledByDeciderForCandidate.incr()\n      Future.False\n    } else {\n      Future\n        .join(\n          cand.target.isAnEmployee,\n          cand.target.isInNotificationsServiceWhitelist,\n          cand.target.isTeamMember\n        )\n        .flatMap {\n          case (isEmployee, isInNotificationsServiceWhitelist, isTeamMember) =>\n            cand.target.deviceInfo.flatMap { deviceInfoOpt =>\n              deviceInfoOpt\n                .map { deviceInfo =>\n                  cand.target.isHeavyUserState.map { isHeavyUser =>\n                    val isAlphaTester = (isEmployee && cand.target\n                      .params(enableForEmployeesParam)) || isInNotificationsServiceWhitelist || isTeamMember\n                    if (cand.target.isDarkWrite) {\n                      stats\n                        .scope(\"model/candidate\").counter(\n                          s\"dark_write_${cand.commonRecType}\").incr()\n                      darkWriteRequestsForCandidate.incr()\n                      false\n                    } else if (isAlphaTester || deviceInfo.isMRinNTabEligible\n                      || cand.target.insertMagicrecsIntoNTabForNonPushableUsers) {\n                      if (isHeavyUser) heavyUserForCandidateCounter.incr()\n                      else nonHeavyUserForCandidateCounter.incr()\n\n                      val enabledForDesiredUsers = cand.target.params(enableForEveryoneParam)\n                      if (isAlphaTester) {\n                        sentToAlphaUserForCandidate.incr()\n                        true\n                      } else if (enabledForDesiredUsers) {\n                        sentToNonBucketedUserForCandidate.incr()\n                        true\n                      } else {\n                        noSendForCandidate.incr()\n                        false\n                      }\n                    } else {\n                      ineligibleUsersForCandidate.incr()\n                      false\n                    }\n                  }\n                }.getOrElse(Future.False)\n            }\n        }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/SendHandlerNotifier.scala",
    "content": "package com.twitter.frigate.pushservice.take\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.Invalid\nimport com.twitter.frigate.common.base.OK\nimport com.twitter.frigate.common.base.Response\nimport com.twitter.frigate.common.base.Result\nimport com.twitter.frigate.common.util.NotificationScribeUtil\nimport com.twitter.frigate.common.util.PushServiceUtil\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.thriftscala.PushResponse\nimport com.twitter.frigate.pushservice.thriftscala.PushStatus\nimport com.twitter.util.Future\n\nclass SendHandlerNotifier(\n  candidateNotifier: CandidateNotifier,\n  private val statsReceiver: StatsReceiver) {\n\n  val missingResponseCounter = statsReceiver.counter(\"missing_response\")\n  val filteredResponseCounter = statsReceiver.counter(\"filtered\")\n\n  /**\n   *\n   * @param isScribeInfoRequired: [[Boolean]] to indicate if scribe info is required\n   * @param candidate: [[PushCandidate]] to build the scribe data from\n   * @return: scribe response string\n   */\n  private def scribeInfoForResponse(\n    isScribeInfoRequired: Boolean,\n    candidate: PushCandidate\n  ): Future[Option[String]] = {\n    if (isScribeInfoRequired) {\n      candidate.scribeData().map { scribedInfo =>\n        Some(NotificationScribeUtil.convertToJsonString(scribedInfo))\n      }\n    } else Future.None\n  }\n\n  /**\n   *\n   * @param response: Candidate validation response\n   * @param responseWithScribedInfo: boolean indicating if scribe data is expected in push response\n   * @return: [[PushResponse]] containing final result of send request for [[com.twitter.frigate.pushservice.thriftscala.PushRequest]]\n   */\n  final def checkResponseAndNotify(\n    response: Response[PushCandidate, Result],\n    responseWithScribedInfo: Boolean\n  ): Future[PushResponse] = {\n\n    response match {\n      case Response(OK, processedCandidates) =>\n        val (validCandidates, invalidCandidates) = processedCandidates.partition(_.result == OK)\n        validCandidates.headOption match {\n          case Some(candidateResult) =>\n            val scribeInfo =\n              scribeInfoForResponse(responseWithScribedInfo, candidateResult.candidate)\n            scribeInfo.flatMap { scribedData =>\n              val response: Future[PushResponse] =\n                candidateNotifier.notify(candidateResult.candidate)\n              response.map(_.copy(notifScribe = scribedData))\n            }\n\n          case None =>\n            invalidCandidates.headOption match {\n              case Some(candidateResult) =>\n                filteredResponseCounter.incr()\n                val response = candidateResult.result match {\n                  case Invalid(reason) => PushResponse(PushStatus.Filtered, filteredBy = reason)\n                  case _ => PushResponse(PushStatus.Filtered, filteredBy = Some(\"unknown\"))\n                }\n\n                val scribeInfo =\n                  scribeInfoForResponse(responseWithScribedInfo, candidateResult.candidate)\n                scribeInfo.map(scribeData => response.copy(notifScribe = scribeData))\n\n              case None =>\n                missingResponseCounter.incr()\n                PushServiceUtil.FilteredPushResponseFut\n            }\n        }\n\n      case Response(Invalid(reason), _) =>\n        throw new IllegalStateException(s\"Unexpected target filtering in SendHandler: $reason\")\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/candidate_validator/CandidateValidator.scala",
    "content": "package com.twitter.frigate.pushservice.take.candidate_validator\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.logger.MRLogger\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.take.predicates.TakeCommonPredicates\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.hermit.predicate.ConcurrentPredicate\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.hermit.predicate.SequentialPredicate\nimport com.twitter.util.Future\n\ntrait CandidateValidator extends TakeCommonPredicates {\n\n  override implicit val statsReceiver: StatsReceiver = config.statsReceiver\n\n  protected val log = MRLogger(\"CandidateValidator\")\n\n  private lazy val skipFiltersCounter = statsReceiver.counter(\"enable_skip_filters\")\n  private lazy val emailUserSkipFiltersCounter =\n    statsReceiver.counter(\"email_user_enable_skip_filters\")\n  private lazy val enablePredicatesCounter = statsReceiver.counter(\"enable_predicates\")\n\n  protected def enabledPredicates[C <: PushCandidate](\n    candidate: C,\n    predicates: List[NamedPredicate[C]]\n  ): List[NamedPredicate[C]] = {\n    val target = candidate.target\n    val skipFilters: Boolean =\n      target.pushContext.flatMap(_.skipFilters).getOrElse(false) || target.params(\n        PushFeatureSwitchParams.SkipPostRankingFilters)\n\n    if (skipFilters) {\n      skipFiltersCounter.incr()\n      if (target.isEmailUser) emailUserSkipFiltersCounter.incr()\n\n      val predicatesToEnable = target.pushContext.flatMap(_.predicatesToEnable).getOrElse(Nil)\n      if (predicatesToEnable.nonEmpty) enablePredicatesCounter.incr()\n\n      // if we skip predicates on pushContext, only enable the explicitly specified predicates\n      predicates.filter(predicatesToEnable.contains)\n    } else predicates\n  }\n\n  protected def executeSequentialPredicates[C <: PushCandidate](\n    candidate: C,\n    predicates: List[NamedPredicate[C]]\n  ): Future[Option[Predicate[C]]] = {\n    val predicatesEnabled = enabledPredicates(candidate, predicates)\n    val sequentialPredicate = new SequentialPredicate(predicatesEnabled)\n\n    sequentialPredicate.track(Seq(candidate)).map(_.head)\n  }\n\n  protected def executeConcurrentPredicates[C <: PushCandidate](\n    candidate: C,\n    predicates: List[NamedPredicate[C]]\n  ): Future[List[Predicate[C]]] = {\n    val predicatesEnabled = enabledPredicates(candidate, predicates)\n    val concurrentPredicate: ConcurrentPredicate[C] = new ConcurrentPredicate[C](predicatesEnabled)\n    concurrentPredicate.track(Seq(candidate)).map(_.head)\n  }\n\n  protected val candidatePredicatesMap: Map[CommonRecommendationType, List[\n    NamedPredicate[_ <: PushCandidate]\n  ]]\n\n  protected def getCRTPredicates[C <: PushCandidate](\n    CRT: CommonRecommendationType\n  ): List[NamedPredicate[C]] = {\n    candidatePredicatesMap.get(CRT) match {\n      case Some(predicates) =>\n        predicates.asInstanceOf[List[NamedPredicate[C]]]\n      case _ =>\n        throw new IllegalStateException(\n          s\"Unknown CommonRecommendationType for Predicates: ${CRT.name}\")\n    }\n  }\n\n  def validateCandidate[C <: PushCandidate](candidate: C): Future[Option[Predicate[C]]]\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/candidate_validator/RFPHCandidateValidator.scala",
    "content": "package com.twitter.frigate.pushservice.take.candidate_validator\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.take.predicates.candidate_map.CandidatePredicatesMap\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\nclass RFPHCandidateValidator(override val config: Config) extends CandidateValidator {\n  private val rFPHCandidateValidatorStats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val concurrentPredicateCount = rFPHCandidateValidatorStats.counter(\"concurrent\")\n  private val sequentialPredicateCount = rFPHCandidateValidatorStats.counter(\"sequential\")\n\n  override protected val candidatePredicatesMap = CandidatePredicatesMap(config)\n\n  override def validateCandidate[C <: PushCandidate](candidate: C): Future[Option[Predicate[C]]] = {\n    val candidatePredicates = getCRTPredicates(candidate.commonRecType)\n    val predicates = rfphPrePredicates ++ candidatePredicates ++ postPredicates\n    if (candidate.target.isEmailUser) {\n      concurrentPredicateCount.incr()\n      executeConcurrentPredicates(candidate, predicates).map(_.headOption)\n    } else {\n      sequentialPredicateCount.incr()\n      executeSequentialPredicates(candidate, predicates)\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/candidate_validator/SendHandlerPostCandidateValidator.scala",
    "content": "package com.twitter.frigate.pushservice.take.candidate_validator\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.take.predicates.candidate_map.SendHandlerCandidatePredicatesMap\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\nclass SendHandlerPostCandidateValidator(override val config: Config) extends CandidateValidator {\n\n  override protected val candidatePredicatesMap =\n    SendHandlerCandidatePredicatesMap.postCandidatePredicates(config)\n\n  private val sendHandlerPostCandidateValidatorStats =\n    statsReceiver.counter(\"sendHandlerPostCandidateValidator_stats\")\n\n  override def validateCandidate[C <: PushCandidate](candidate: C): Future[Option[Predicate[C]]] = {\n    val candidatePredicates = getCRTPredicates(candidate.commonRecType)\n    val predicates = candidatePredicates ++ postPredicates\n\n    sendHandlerPostCandidateValidatorStats.incr()\n\n    executeConcurrentPredicates(candidate, predicates)\n      .map(_.headOption)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/candidate_validator/SendHandlerPreCandidateValidator.scala",
    "content": "package com.twitter.frigate.pushservice.take.candidate_validator\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.take.predicates.candidate_map.SendHandlerCandidatePredicatesMap\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.util.Future\n\nclass SendHandlerPreCandidateValidator(override val config: Config) extends CandidateValidator {\n\n  override protected val candidatePredicatesMap =\n    SendHandlerCandidatePredicatesMap.preCandidatePredicates(config)\n\n  private val sendHandlerPreCandidateValidatorStats =\n    statsReceiver.counter(\"sendHandlerPreCandidateValidator_stats\")\n\n  override def validateCandidate[C <: PushCandidate](candidate: C): Future[Option[Predicate[C]]] = {\n    val candidatePredicates = getCRTPredicates(candidate.commonRecType)\n    val predicates = sendHandlerPrePredicates ++ candidatePredicates\n\n    sendHandlerPreCandidateValidatorStats.incr()\n    executeSequentialPredicates(candidate, predicates)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/channel_selection/ChannelCandidate.scala",
    "content": "package com.twitter.frigate.pushservice.take\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.thriftscala.ChannelName\nimport com.twitter.util.Future\nimport java.util.concurrent.ConcurrentHashMap\nimport scala.collection.concurrent\nimport scala.collection.convert.decorateAsScala._\n\n/**\n * A class to save all the channel related information\n */\ntrait ChannelForCandidate {\n  self: PushCandidate =>\n\n  // Cache of channel selection result\n  private[this] val selectedChannels: concurrent.Map[String, Future[Seq[ChannelName]]] =\n    new ConcurrentHashMap[String, Future[Seq[ChannelName]]]().asScala\n\n  // Returns the channel information from all ChannelSelectors.\n  def getChannels(): Future[Seq[ChannelName]] = {\n    Future.collect(selectedChannels.values.toSeq).map { c => c.flatten }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/channel_selection/ChannelSelector.scala",
    "content": "package com.twitter.frigate.pushservice.take\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.thriftscala.ChannelName\nimport com.twitter.util.Future\n\nabstract class ChannelSelector {\n\n  // Returns a map of channel name, and the candidates that can be sent on that channel.\n  def selectChannel(\n    candidate: PushCandidate\n  ): Future[Seq[ChannelName]]\n\n  def getSelectorName(): String\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/channel_selection/NtabOnlyChannelSelector.scala",
    "content": "package com.twitter.frigate.pushservice.take\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.thriftscala.ChannelName\nimport com.twitter.util.Future\n\nclass NtabOnlyChannelSelector extends ChannelSelector {\n  val SELECTOR_NAME = \"NtabOnlyChannelSelector\"\n\n  def getSelectorName(): String = SELECTOR_NAME\n\n  // Returns a map of channel name, and the candidates that can be sent on that channel\n  def selectChannel(\n    candidate: PushCandidate\n  ): Future[Seq[ChannelName]] = {\n    // Check candidate channel eligible (based on setting, push cap etc\n    // Decide which candidate can be sent on what channel\n    val channelName: Future[ChannelName] = Future.value(ChannelName.PushNtab)\n    channelName.map(channel => Seq(channel))\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/history/EventBusWriter.scala",
    "content": "package com.twitter.frigate.pushservice.take.history\n\nimport com.twitter.eventbus.client.EventBusPublisher\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.NotificationScribeUtil\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.scribe.thriftscala.NotificationScribe\nimport com.twitter.frigate.thriftscala.FrigateNotification\n\nclass EventBusWriter(\n  eventBusPublisher: EventBusPublisher[NotificationScribe],\n  stats: StatsReceiver) {\n  private def writeSendEventToEventBus(\n    target: PushTypes.Target,\n    notificationScribe: NotificationScribe\n  ): Unit = {\n    if (target.params(PushParams.EnablePushSendEventBus)) {\n      val result = eventBusPublisher.publish(notificationScribe)\n      result.onFailure { _ => stats.counter(\"push_send_eventbus_failure\").incr() }\n    }\n  }\n\n  def writeToEventBus(\n    candidate: PushCandidate,\n    frigateNotificationForPersistence: FrigateNotification\n  ): Unit = {\n    val notificationScribe = NotificationScribeUtil.getNotificationScribe(\n      targetId = candidate.target.targetId,\n      impressionId = candidate.impressionId,\n      frigateNotification = frigateNotificationForPersistence,\n      createdAt = candidate.createdAt\n    )\n    writeSendEventToEventBus(candidate.target, notificationScribe)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/history/HistoryWriter.scala",
    "content": "package com.twitter.frigate.pushservice.take.history\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.history.HistoryStoreKeyContext\nimport com.twitter.frigate.common.history.PushServiceHistoryStore\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.conversions.DurationOps._\n\nclass HistoryWriter(historyStore: PushServiceHistoryStore, stats: StatsReceiver) {\n  private lazy val historyWriterStats = stats.scope(this.getClass.getSimpleName)\n  private lazy val historyWriteCounter = historyWriterStats.counter(\"history_write_num\")\n  private lazy val loggedOutHistoryWriteCounter =\n    historyWriterStats.counter(\"logged_out_history_write_num\")\n\n  private def writeTtlForHistory(candidate: PushCandidate): Duration = {\n    if (candidate.target.isLoggedOutUser) {\n      60.days\n    } else if (RecTypes.isTweetType(candidate.commonRecType)) {\n      candidate.target.params(PushFeatureSwitchParams.FrigateHistoryTweetNotificationWriteTtl)\n    } else candidate.target.params(PushFeatureSwitchParams.FrigateHistoryOtherNotificationWriteTtl)\n  }\n\n  def writeSendToHistory(\n    candidate: PushCandidate,\n    frigateNotificationForPersistence: FrigateNotification\n  ): Future[Unit] = {\n    val historyStoreKeyContext = HistoryStoreKeyContext(\n      candidate.target.targetId,\n      candidate.target.pushContext.flatMap(_.useMemcacheForHistory).getOrElse(false)\n    )\n    if (candidate.target.isLoggedOutUser) {\n      loggedOutHistoryWriteCounter.incr()\n    } else {\n      historyWriteCounter.incr()\n    }\n    historyStore\n      .put(\n        historyStoreKeyContext,\n        candidate.createdAt,\n        frigateNotificationForPersistence,\n        Some(writeTtlForHistory(candidate))\n      )\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/predicates/BasicRFPHPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.take.predicates\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.hermit.predicate.NamedPredicate\n\ntrait BasicRFPHPredicates[C <: PushCandidate] {\n  val predicates: List[NamedPredicate[C]]\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/predicates/BasicSendHandlerPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.take.predicates\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.hermit.predicate.NamedPredicate\n\ntrait BasicSendHandlerPredicates[C <: PushCandidate] {\n\n  // specific predicates per candidate type before basic SendHandler predicates\n  val preCandidateSpecificPredicates: List[NamedPredicate[C]] = List.empty\n\n  // specific predicates per candidate type after basic SendHandler predicates, could be empty\n  val postCandidateSpecificPredicates: List[NamedPredicate[C]] = List.empty\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/predicates/BasicTweetPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.take.predicates\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.predicate.BqmlHealthModelPredicates\nimport com.twitter.frigate.pushservice.predicate.BqmlQualityModelPredicates\nimport com.twitter.frigate.pushservice.predicate.HealthPredicates\nimport com.twitter.frigate.pushservice.predicate.OONSpreadControlPredicate\nimport com.twitter.frigate.pushservice.predicate.OONTweetNegativeFeedbackBasedPredicate\nimport com.twitter.frigate.pushservice.predicate.OutOfNetworkCandidatesQualityPredicates\nimport com.twitter.frigate.pushservice.predicate.PredicatesForCandidate\nimport com.twitter.frigate.pushservice.predicate.PNegMultimodalPredicates\nimport com.twitter.frigate.pushservice.predicate.TargetEngagementPredicate\nimport com.twitter.frigate.pushservice.predicate.TweetEngagementRatioPredicate\nimport com.twitter.frigate.pushservice.predicate.TweetLanguagePredicate\nimport com.twitter.frigate.pushservice.predicate.TweetWithheldContentPredicate\n\ntrait BasicTweetPredicates {\n\n  def config: Config\n\n  implicit def statsReceiver: StatsReceiver\n\n  final lazy val basicTweetPredicates =\n    List(\n      HealthPredicates.sensitiveMediaCategoryPredicate(),\n      HealthPredicates.profanityPredicate(),\n      PredicatesForCandidate.disableOutNetworkTweetPredicate(config.edgeStore),\n      TweetEngagementRatioPredicate.QTtoNtabClickBasedPredicate(),\n      TweetLanguagePredicate.oonTweeetLanguageMatch(),\n      HealthPredicates.userHealthSignalsPredicate(config.userHealthSignalStore),\n      HealthPredicates.authorSensitiveMediaPredicate(config.producerMediaRepresentationStore),\n      HealthPredicates.authorProfileBasedPredicate(),\n      PNegMultimodalPredicates.healthSignalScorePNegMultimodalPredicate(\n        config.tweetHealthScoreStore),\n      BqmlHealthModelPredicates.healthModelOonPredicate(\n        config.filteringModelScorer,\n        config.producerMediaRepresentationStore,\n        config.userHealthSignalStore,\n        config.tweetHealthScoreStore),\n      BqmlQualityModelPredicates.BqmlQualityModelOonPredicate(config.filteringModelScorer),\n      HealthPredicates.tweetHealthSignalScorePredicate(config.tweetHealthScoreStore),\n      HealthPredicates\n        .tweetHealthSignalScorePredicate(config.tweetHealthScoreStore, applyToQuoteTweet = true),\n      PredicatesForCandidate.nullCastF1ProtectedExperientPredicate(\n        config.cachedTweetyPieStoreV2\n      ),\n      OONTweetNegativeFeedbackBasedPredicate.ntabDislikeBasedPredicate(),\n      OONSpreadControlPredicate.oonTweetSpreadControlPredicate(),\n      OONSpreadControlPredicate.oonAuthorSpreadControlPredicate(),\n      HealthPredicates.healthSignalScoreMultilingualPnsfwTweetTextPredicate(\n        config.tweetHealthScoreStore),\n      PredicatesForCandidate\n        .recommendedTweetAuthorAcceptableToTargetUser(config.edgeStore),\n      HealthPredicates.healthSignalScorePnsfwTweetTextPredicate(config.tweetHealthScoreStore),\n      HealthPredicates.healthSignalScoreSpammyTweetPredicate(config.tweetHealthScoreStore),\n      OutOfNetworkCandidatesQualityPredicates.NegativeKeywordsPredicate(\n        config.postRankingFeatureStoreClient),\n      PredicatesForCandidate.authorNotBeingDeviceFollowed(config.edgeStore),\n      TweetWithheldContentPredicate(),\n      PredicatesForCandidate.noOptoutFreeFormInterestPredicate,\n      PredicatesForCandidate.disableInNetworkTweetPredicate(config.edgeStore),\n      TweetEngagementRatioPredicate.TweetReplyLikeRatioPredicate(),\n      TargetEngagementPredicate(\n        config.userTweetPerspectiveStore,\n        defaultForMissing = true\n      ),\n    )\n}\n\n/**\n * This trait is a new version of BasicTweetPredicates\n * Difference from old version is that basicTweetPredicates are different\n * basicTweetPredicates here don't include Social Graph Service related predicates\n */\ntrait BasicTweetPredicatesWithoutSGSPredicates {\n\n  def config: Config\n\n  implicit def statsReceiver: StatsReceiver\n\n  final lazy val basicTweetPredicates = {\n    List(\n      HealthPredicates.healthSignalScoreSpammyTweetPredicate(config.tweetHealthScoreStore),\n      PredicatesForCandidate.nullCastF1ProtectedExperientPredicate(\n        config.cachedTweetyPieStoreV2\n      ),\n      TweetWithheldContentPredicate(),\n      TargetEngagementPredicate(\n        config.userTweetPerspectiveStore,\n        defaultForMissing = true\n      ),\n      PredicatesForCandidate.noOptoutFreeFormInterestPredicate,\n      HealthPredicates.userHealthSignalsPredicate(config.userHealthSignalStore),\n      HealthPredicates.tweetHealthSignalScorePredicate(config.tweetHealthScoreStore),\n      BqmlQualityModelPredicates.BqmlQualityModelOonPredicate(config.filteringModelScorer),\n      BqmlHealthModelPredicates.healthModelOonPredicate(\n        config.filteringModelScorer,\n        config.producerMediaRepresentationStore,\n        config.userHealthSignalStore,\n        config.tweetHealthScoreStore),\n    )\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/predicates/BasicTweetPredicatesForRFPH.scala",
    "content": "package com.twitter.frigate.pushservice.take.predicates\n\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.base.TweetDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.hermit.predicate.NamedPredicate\n\ntrait BasicTweetPredicatesForRFPH[C <: PushCandidate with TweetCandidate with TweetDetails]\n    extends BasicTweetPredicates\n    with BasicRFPHPredicates[C] {\n\n  // specific predicates per candidate type before basic tweet predicates\n  def preCandidateSpecificPredicates: List[NamedPredicate[C]] = List.empty\n\n  // specific predicates per candidate type after basic tweet predicates\n  def postCandidateSpecificPredicates: List[NamedPredicate[C]] = List.empty\n\n  override lazy val predicates: List[NamedPredicate[C]] =\n    preCandidateSpecificPredicates ++ basicTweetPredicates ++ postCandidateSpecificPredicates\n}\n\n/**\n * This trait is a new version of BasicTweetPredicatesForRFPH\n * Difference from old version is that basicTweetPredicates are different\n * basicTweetPredicates here don't include Social Graph Service related predicates\n */\ntrait BasicTweetPredicatesForRFPHWithoutSGSPredicates[\n  C <: PushCandidate with TweetCandidate with TweetDetails]\n    extends BasicTweetPredicatesWithoutSGSPredicates\n    with BasicRFPHPredicates[C] {\n\n  // specific predicates per candidate type before basic tweet predicates\n  def preCandidateSpecificPredicates: List[NamedPredicate[C]] = List.empty\n\n  // specific predicates per candidate type after basic tweet predicates\n  def postCandidateSpecificPredicates: List[NamedPredicate[C]] = List.empty\n\n  override lazy val predicates: List[NamedPredicate[C]] =\n    preCandidateSpecificPredicates ++ basicTweetPredicates ++ postCandidateSpecificPredicates\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/predicates/OutOfNetworkTweetPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.take.predicates\n\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.base.TweetDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.predicate.PredicatesForCandidate\nimport com.twitter.hermit.predicate.NamedPredicate\n\ntrait OutOfNetworkTweetPredicates[C <: PushCandidate with TweetCandidate with TweetDetails]\n    extends BasicTweetPredicatesForRFPH[C] {\n\n  override lazy val preCandidateSpecificPredicates: List[NamedPredicate[C]] =\n    List(\n      PredicatesForCandidate.authorNotBeingFollowed(config.edgeStore)\n    )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/predicates/TakeCommonPredicates.scala",
    "content": "package com.twitter.frigate.pushservice.take.predicates\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.pushservice.predicate.CrtDeciderPredicate\nimport com.twitter.frigate.pushservice.predicate.PredicatesForCandidate\nimport com.twitter.frigate.pushservice.predicate.ScarecrowPredicate\nimport com.twitter.frigate.pushservice.predicate.ntab_caret_fatigue.NtabCaretClickFatiguePredicate\nimport com.twitter.hermit.predicate.NamedPredicate\n\ntrait TakeCommonPredicates {\n  def config: Config\n\n  implicit def statsReceiver: StatsReceiver\n\n  lazy val rfphPrePredicates: List[NamedPredicate[PushCandidate]] = List(\n    CrtDeciderPredicate(config.decider),\n    PredicatesForCandidate.isChannelValidPredicate,\n  )\n\n  lazy val sendHandlerPrePredicates: List[NamedPredicate[PushCandidate]] = List(\n    CrtDeciderPredicate(config.decider),\n    PredicatesForCandidate.enableSendHandlerCandidates,\n    PredicatesForCandidate.mrWebHoldbackPredicate,\n    PredicatesForCandidate.targetUserExists,\n    PredicatesForCandidate.authorInSocialContext,\n    PredicatesForCandidate.recommendedTweetIsAuthoredBySelf,\n    PredicatesForCandidate.selfInSocialContext,\n    NtabCaretClickFatiguePredicate()\n  )\n\n  lazy val postPredicates: List[NamedPredicate[PushCandidate]] = List(\n    ScarecrowPredicate(config.scarecrowCheckEventStore)\n  )\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/predicates/candidate_map/CandidatePredicatesMap.scala",
    "content": "package com.twitter.frigate.pushservice.take.predicates.candidate_map\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model._\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.CommonRecommendationType._\nimport com.twitter.hermit.predicate.NamedPredicate\n\nobject CandidatePredicatesMap {\n\n  def apply(\n    implicit config: Config\n  ): Map[CommonRecommendationType, List[NamedPredicate[_ <: PushCandidate]]] = {\n\n    val trendTweetCandidatePredicates = TrendTweetPredicates(config).predicates\n    val tripTweetCandidatePredicates = TripTweetCandidatePredicates(config).predicates\n    val f1TweetCandidatePredicates = F1TweetCandidatePredicates(config).predicates\n    val oonTweetCandidatePredicates = OutOfNetworkTweetCandidatePredicates(config).predicates\n    val tweetActionCandidatePredicates = TweetActionCandidatePredicates(config).predicates\n    val topicProofTweetCandidatePredicates = TopicProofTweetCandidatePredicates(config).predicates\n    val addressBookPushPredicates = AddressBookPushCandidatePredicates(config).predicates\n    val completeOnboardingPushPredicates = CompleteOnboardingPushCandidatePredicates(\n      config).predicates\n    val popGeoTweetCandidatePredicate = PopGeoTweetCandidatePredicates(config).predicates\n    val topTweetImpressionsCandidatePredicates = TopTweetImpressionsPushCandidatePredicates(\n      config).predicates\n    val listCandidatePredicates = ListRecommendationPredicates(config).predicates\n    val subscribedSearchTweetCandidatePredicates = SubscribedSearchTweetCandidatePredicates(\n      config).predicates\n\n    Map(\n      F1FirstdegreeTweet -> f1TweetCandidatePredicates,\n      F1FirstdegreePhoto -> f1TweetCandidatePredicates,\n      F1FirstdegreeVideo -> f1TweetCandidatePredicates,\n      ElasticTimelineTweet -> oonTweetCandidatePredicates,\n      ElasticTimelinePhoto -> oonTweetCandidatePredicates,\n      ElasticTimelineVideo -> oonTweetCandidatePredicates,\n      TwistlyTweet -> oonTweetCandidatePredicates,\n      TwistlyPhoto -> oonTweetCandidatePredicates,\n      TwistlyVideo -> oonTweetCandidatePredicates,\n      ExploreVideoTweet -> oonTweetCandidatePredicates,\n      UserInterestinTweet -> oonTweetCandidatePredicates,\n      UserInterestinPhoto -> oonTweetCandidatePredicates,\n      UserInterestinVideo -> oonTweetCandidatePredicates,\n      PastEmailEngagementTweet -> oonTweetCandidatePredicates,\n      PastEmailEngagementPhoto -> oonTweetCandidatePredicates,\n      PastEmailEngagementVideo -> oonTweetCandidatePredicates,\n      TagSpaceTweet -> oonTweetCandidatePredicates,\n      TwhinTweet -> oonTweetCandidatePredicates,\n      FrsTweet -> oonTweetCandidatePredicates,\n      MrModelingBasedTweet -> oonTweetCandidatePredicates,\n      TrendTweet -> trendTweetCandidatePredicates,\n      ReverseAddressbookTweet -> oonTweetCandidatePredicates,\n      ForwardAddressbookTweet -> oonTweetCandidatePredicates,\n      TripGeoTweet -> oonTweetCandidatePredicates,\n      TripHqTweet -> tripTweetCandidatePredicates,\n      DetopicTweet -> oonTweetCandidatePredicates,\n      CrowdSearchTweet -> oonTweetCandidatePredicates,\n      TweetFavorite -> tweetActionCandidatePredicates,\n      TweetFavoritePhoto -> tweetActionCandidatePredicates,\n      TweetFavoriteVideo -> tweetActionCandidatePredicates,\n      TweetRetweet -> tweetActionCandidatePredicates,\n      TweetRetweetPhoto -> tweetActionCandidatePredicates,\n      TweetRetweetVideo -> tweetActionCandidatePredicates,\n      TopicProofTweet -> topicProofTweetCandidatePredicates,\n      SubscribedSearch -> subscribedSearchTweetCandidatePredicates,\n      AddressBookUploadPush -> addressBookPushPredicates,\n      CompleteOnboardingPush -> completeOnboardingPushPredicates,\n      List -> listCandidatePredicates,\n      GeoPopTweet -> popGeoTweetCandidatePredicate,\n      TweetImpressions -> topTweetImpressionsCandidatePredicates\n    )\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/predicates/candidate_map/SendHandlerCandidatePredicatesMap.scala",
    "content": "package com.twitter.frigate.pushservice.take.predicates.candidate_map\n\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model._\nimport com.twitter.frigate.pushservice.config.Config\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.CommonRecommendationType._\nimport com.twitter.hermit.predicate.NamedPredicate\n\nobject SendHandlerCandidatePredicatesMap {\n\n  def preCandidatePredicates(\n    implicit config: Config\n  ): Map[CommonRecommendationType, List[NamedPredicate[_ <: PushCandidate]]] = {\n    val magicFanoutNewsEventCandidatePredicates =\n      MagicFanoutNewsEventCandidatePredicates(config).preCandidateSpecificPredicates\n\n    val scheduledSpaceSubscriberPredicates = ScheduledSpaceSubscriberCandidatePredicates(\n      config).preCandidateSpecificPredicates\n\n    val scheduledSpaceSpeakerPredicates = ScheduledSpaceSpeakerCandidatePredicates(\n      config).preCandidateSpecificPredicates\n\n    val magicFanoutSportsEventCandidatePredicates =\n      MagicFanoutSportsEventCandidatePredicates(config).preCandidateSpecificPredicates\n\n    val magicFanoutProductLaunchPredicates = MagicFanoutProductLaunchPushCandidatePredicates(\n      config).preCandidateSpecificPredicates\n\n    val creatorSubscriptionFanoutPredicates = MagicFanouCreatorSubscriptionEventPushPredicates(\n      config).preCandidateSpecificPredicates\n\n    val newCreatorFanoutPredicates = MagicFanoutNewCreatorEventPushPredicates(\n      config).preCandidateSpecificPredicates\n\n    Map(\n      MagicFanoutNewsEvent -> magicFanoutNewsEventCandidatePredicates,\n      ScheduledSpaceSubscriber -> scheduledSpaceSubscriberPredicates,\n      ScheduledSpaceSpeaker -> scheduledSpaceSpeakerPredicates,\n      MagicFanoutSportsEvent -> magicFanoutSportsEventCandidatePredicates,\n      MagicFanoutProductLaunch -> magicFanoutProductLaunchPredicates,\n      NewCreator -> newCreatorFanoutPredicates,\n      CreatorSubscriber -> creatorSubscriptionFanoutPredicates\n    )\n  }\n\n  def postCandidatePredicates(\n    implicit config: Config\n  ): Map[CommonRecommendationType, List[NamedPredicate[_ <: PushCandidate]]] = {\n    val magicFanoutNewsEventCandidatePredicates =\n      MagicFanoutNewsEventCandidatePredicates(config).postCandidateSpecificPredicates\n\n    val scheduledSpaceSubscriberPredicates = ScheduledSpaceSubscriberCandidatePredicates(\n      config).postCandidateSpecificPredicates\n\n    val scheduledSpaceSpeakerPredicates = ScheduledSpaceSpeakerCandidatePredicates(\n      config).postCandidateSpecificPredicates\n\n    val magicFanoutSportsEventCandidatePredicates =\n      MagicFanoutSportsEventCandidatePredicates(config).postCandidateSpecificPredicates\n    val magicFanoutProductLaunchPredicates = MagicFanoutProductLaunchPushCandidatePredicates(\n      config).postCandidateSpecificPredicates\n    val creatorSubscriptionFanoutPredicates = MagicFanouCreatorSubscriptionEventPushPredicates(\n      config).postCandidateSpecificPredicates\n    val newCreatorFanoutPredicates = MagicFanoutNewCreatorEventPushPredicates(\n      config).postCandidateSpecificPredicates\n\n    Map(\n      MagicFanoutNewsEvent -> magicFanoutNewsEventCandidatePredicates,\n      ScheduledSpaceSubscriber -> scheduledSpaceSubscriberPredicates,\n      ScheduledSpaceSpeaker -> scheduledSpaceSpeakerPredicates,\n      MagicFanoutSportsEvent -> magicFanoutSportsEventCandidatePredicates,\n      MagicFanoutProductLaunch -> magicFanoutProductLaunchPredicates,\n      NewCreator -> newCreatorFanoutPredicates,\n      CreatorSubscriber -> creatorSubscriptionFanoutPredicates\n    )\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/sender/Ibis2Sender.scala",
    "content": "package com.twitter.frigate.pushservice.take.sender\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.common.base.TweetDetails\nimport com.twitter.frigate.common.store.IbisResponse\nimport com.twitter.frigate.common.store.InvalidConfiguration\nimport com.twitter.frigate.common.store.NoRequest\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FS}\nimport com.twitter.frigate.pushservice.store.Ibis2Store\nimport com.twitter.frigate.pushservice.store.TweetTranslationStore\nimport com.twitter.frigate.pushservice.util.CopyUtil\nimport com.twitter.frigate.pushservice.util.FunctionalUtil\nimport com.twitter.frigate.pushservice.util.InlineActionUtil\nimport com.twitter.frigate.pushservice.util.OverrideNotificationUtil\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.frigate.scribe.thriftscala.NotificationScribe\nimport com.twitter.frigate.thriftscala.ChannelName\nimport com.twitter.frigate.thriftscala.NotificationDisplayLocation\nimport com.twitter.ibis2.service.thriftscala.Ibis2Request\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationResponse\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nclass Ibis2Sender(\n  pushIbisV2Store: Ibis2Store,\n  tweetTranslationStore: ReadableStore[TweetTranslationStore.Key, TweetTranslationStore.Value],\n  statsReceiver: StatsReceiver) {\n\n  private val stats = statsReceiver.scope(getClass.getSimpleName)\n  private val silentPushCounter = stats.counter(\"silent_push\")\n  private val ibisSendFailureCounter = stats.scope(\"ibis_send_failure\").counter(\"failures\")\n  private val buggyAndroidReleaseCounter = stats.counter(\"is_buggy_android_release\")\n  private val androidPrimaryCounter = stats.counter(\"android_primary_device\")\n  private val addTranslationModelValuesCounter = stats.counter(\"with_translation_model_values\")\n  private val patchNtabResponseEnabled = stats.scope(\"with_ntab_response\")\n  private val noIbisPushStats = stats.counter(\"no_ibis_push\")\n\n  private def ibisSend(\n    candidate: PushCandidate,\n    translationModelValues: Option[Map[String, String]] = None,\n    ntabResponse: Option[CreateGenericNotificationResponse] = None\n  ): Future[IbisResponse] = {\n    if (candidate.frigateNotification.notificationDisplayLocation != NotificationDisplayLocation.PushToMobileDevice) {\n      Future.value(IbisResponse(InvalidConfiguration))\n    } else {\n      candidate.ibis2Request.flatMap {\n        case Some(request) =>\n          val requestWithTranslationMV =\n            addTranslationModelValues(request, translationModelValues)\n          val patchedIbisRequest = {\n            if (candidate.target.isLoggedOutUser) {\n              requestWithTranslationMV\n            } else {\n              patchNtabResponseToIbisRequest(requestWithTranslationMV, candidate, ntabResponse)\n            }\n          }\n          pushIbisV2Store.send(patchedIbisRequest, candidate)\n        case _ =>\n          noIbisPushStats.incr()\n          Future.value(IbisResponse(sendStatus = NoRequest, ibis2Response = None))\n      }\n    }\n  }\n\n  def sendAsDarkWrite(\n    candidate: PushCandidate\n  ): Future[IbisResponse] = {\n    ibisSend(candidate)\n  }\n\n  def send(\n    channels: Seq[ChannelName],\n    pushCandidate: PushCandidate,\n    notificationScribe: NotificationScribe => Unit,\n    ntabResponse: Option[CreateGenericNotificationResponse],\n  ): Future[IbisResponse] = pushCandidate.target.isSilentPush.flatMap { isSilentPush: Boolean =>\n    if (isSilentPush) silentPushCounter.incr()\n    pushCandidate.target.deviceInfo.flatMap { deviceInfo =>\n      if (deviceInfo.exists(_.isSim40AndroidVersion)) buggyAndroidReleaseCounter.incr()\n      if (PushDeviceUtil.isPrimaryDeviceAndroid(deviceInfo)) androidPrimaryCounter.incr()\n      Future\n        .join(\n          OverrideNotificationUtil\n            .getOverrideInfo(pushCandidate, stats),\n          CopyUtil.getCopyFeatures(pushCandidate, stats),\n          getTranslationModelValues(pushCandidate)\n        ).flatMap {\n          case (overrideInfoOpt, copyFeaturesMap, translationModelValues) =>\n            ibisSend(pushCandidate, translationModelValues, ntabResponse)\n              .onSuccess { ibisResponse =>\n                pushCandidate\n                  .scribeData(\n                    ibis2Response = ibisResponse.ibis2Response,\n                    isSilentPush = isSilentPush,\n                    overrideInfoOpt = overrideInfoOpt,\n                    copyFeaturesList = copyFeaturesMap.keySet,\n                    channels = channels\n                  ).foreach(notificationScribe)\n              }.onFailure { _ =>\n                pushCandidate\n                  .scribeData(channels = channels).foreach { data =>\n                    ibisSendFailureCounter.incr()\n                    notificationScribe(data)\n                  }\n              }\n        }\n    }\n  }\n\n  private def getTranslationModelValues(\n    candidate: PushCandidate\n  ): Future[Option[Map[String, String]]] = {\n    candidate match {\n      case tweetCandidate: TweetCandidate with TweetDetails =>\n        val key = TweetTranslationStore.Key(\n          target = candidate.target,\n          tweetId = tweetCandidate.tweetId,\n          tweet = tweetCandidate.tweet,\n          crt = candidate.commonRecType\n        )\n\n        tweetTranslationStore\n          .get(key)\n          .map {\n            case Some(value) =>\n              Some(\n                Map(\n                  \"translated_tweet_text\" -> value.translatedTweetText,\n                  \"localized_source_language\" -> value.localizedSourceLanguage\n                ))\n            case None => None\n          }\n      case _ => Future.None\n    }\n  }\n\n  private def addTranslationModelValues(\n    ibisRequest: Ibis2Request,\n    translationModelValues: Option[Map[String, String]]\n  ): Ibis2Request = {\n    (translationModelValues, ibisRequest.modelValues) match {\n      case (Some(translationModelVal), Some(existingModelValues)) =>\n        addTranslationModelValuesCounter.incr()\n        ibisRequest.copy(modelValues = Some(translationModelVal ++ existingModelValues))\n      case (Some(translationModelVal), None) =>\n        addTranslationModelValuesCounter.incr()\n        ibisRequest.copy(modelValues = Some(translationModelVal))\n      case (None, _) => ibisRequest\n    }\n  }\n\n  private def patchNtabResponseToIbisRequest(\n    ibis2Req: Ibis2Request,\n    candidate: PushCandidate,\n    ntabResponse: Option[CreateGenericNotificationResponse]\n  ): Ibis2Request = {\n    if (candidate.target.params(FS.EnableInlineFeedbackOnPush)) {\n      patchNtabResponseEnabled.counter().incr()\n      val dislikePosition = candidate.target.params(FS.InlineFeedbackSubstitutePosition)\n      val dislikeActionOption = ntabResponse\n        .map(FunctionalUtil.incr(patchNtabResponseEnabled.counter(\"ntab_response_exist\")))\n        .flatMap(response => InlineActionUtil.getDislikeInlineAction(candidate, response))\n        .map(FunctionalUtil.incr(patchNtabResponseEnabled.counter(\"dislike_action_generated\")))\n\n      // Only generate patch serialized inline action when original request has existing serialized_inline_actions_v2\n      val patchedSerializedActionOption = ibis2Req.modelValues\n        .flatMap(model => model.get(\"serialized_inline_actions_v2\"))\n        .map(FunctionalUtil.incr(patchNtabResponseEnabled.counter(\"inline_action_v2_exists\")))\n        .map(serialized =>\n          InlineActionUtil\n            .patchInlineActionAtPosition(serialized, dislikeActionOption, dislikePosition))\n        .map(FunctionalUtil.incr(patchNtabResponseEnabled.counter(\"patch_inline_action_generated\")))\n\n      (ibis2Req.modelValues, patchedSerializedActionOption) match {\n        case (Some(existingModelValue), Some(patchedActionV2)) =>\n          patchNtabResponseEnabled.scope(\"patch_applied\").counter().incr()\n          ibis2Req.copy(modelValues =\n            Some(existingModelValue ++ Map(\"serialized_inline_actions_v2\" -> patchedActionV2)))\n        case _ => ibis2Req\n      }\n    } else ibis2Req\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/take/sender/NtabSender.scala",
    "content": "package com.twitter.frigate.pushservice.take.sender\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.history.History\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.ibis.PushOverrideInfo\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FSParams}\nimport com.twitter.frigate.pushservice.take.NotificationServiceRequest\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.hermit.store.common.ReadableWritableStore\nimport com.twitter.notificationservice.api.thriftscala.DeleteCurrentTimelineForUserRequest\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationResponse\nimport com.twitter.notificationservice.thriftscala.DeleteGenericNotificationRequest\nimport com.twitter.notificationservice.thriftscala.GenericNotificationKey\nimport com.twitter.notificationservice.thriftscala.GenericNotificationOverrideKey\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nobject OverrideCandidate extends Enumeration {\n  val One: String = \"overrideEntry1\"\n}\n\nclass NtabSender(\n  notificationServiceSender: ReadableStore[\n    NotificationServiceRequest,\n    CreateGenericNotificationResponse\n  ],\n  nTabHistoryStore: ReadableWritableStore[(Long, String), GenericNotificationOverrideKey],\n  nTabDelete: DeleteGenericNotificationRequest => Future[Unit],\n  nTabDeleteTimeline: DeleteCurrentTimelineForUserRequest => Future[Unit]\n)(\n  implicit statsReceiver: StatsReceiver) {\n\n  private[this] val nTabDeleteRequests = statsReceiver.counter(\"ntab_delete_request\")\n  private[this] val nTabDeleteTimelineRequests =\n    statsReceiver.counter(\"ntab_delete_timeline_request\")\n  private[this] val ntabOverrideImpressionNotFound =\n    statsReceiver.counter(\"ntab_impression_not_found\")\n  private[this] val nTabOverrideOverriddenStat =\n    statsReceiver.counter(\"ntab_override_overridden\")\n  private[this] val storeGenericNotifOverrideKey =\n    statsReceiver.counter(\"ntab_store_generic_notif_key\")\n  private[this] val prevGenericNotifKeyNotFound =\n    statsReceiver.counter(\"ntab_prev_generic_notif_key_not_found\")\n\n  private[this] val ntabOverride =\n    statsReceiver.scope(\"ntab_override\")\n  private[this] val ntabRequestWithOverrideId =\n    ntabOverride.counter(\"request\")\n  private[this] val storeGenericNotifOverrideKeyWithOverrideId =\n    ntabOverride.counter(\"store_override_key\")\n\n  def send(\n    candidate: PushCandidate,\n    isNtabOnlyNotification: Boolean\n  ): Future[Option[CreateGenericNotificationResponse]] = {\n    if (candidate.target.params(FSParams.EnableOverrideIdNTabRequest)) {\n      ntabRequestWithOverrideId.incr()\n      overridePreviousEntry(candidate).flatMap { _ =>\n        if (shouldDisableNtabOverride(candidate)) {\n          sendNewEntry(candidate, isNtabOnlyNotification, None)\n        } else {\n          sendNewEntry(candidate, isNtabOnlyNotification, Some(OverrideCandidate.One))\n        }\n      }\n    } else {\n      for {\n        notificationOverwritten <- overrideNSlot(candidate)\n        _ <- deleteCachedApiTimeline(candidate, notificationOverwritten)\n        gnResponse <- sendNewEntry(candidate, isNtabOnlyNotification)\n      } yield gnResponse\n    }\n  }\n\n  private def sendNewEntry(\n    candidate: PushCandidate,\n    isNtabOnlyNotif: Boolean,\n    overrideId: Option[String] = None\n  ): Future[Option[CreateGenericNotificationResponse]] = {\n    notificationServiceSender\n      .get(\n        NotificationServiceRequest(\n          candidate,\n          impressionId = candidate.impressionId,\n          isBadgeUpdate = isNtabOnlyNotif,\n          overrideId = overrideId\n        )).flatMap {\n        case Some(response) =>\n          storeGenericNotifKey(candidate, response, overrideId).map { _ => Some(response) }\n        case _ => Future.None\n      }\n  }\n\n  private def storeGenericNotifKey(\n    candidate: PushCandidate,\n    createGenericNotificationResponse: CreateGenericNotificationResponse,\n    overrideId: Option[String]\n  ): Future[Unit] = {\n    if (candidate.target.params(FSParams.EnableStoringNtabGenericNotifKey)) {\n      createGenericNotificationResponse.successKey match {\n        case Some(genericNotificationKey) =>\n          val userId = genericNotificationKey.userId\n          if (overrideId.nonEmpty) {\n            storeGenericNotifOverrideKeyWithOverrideId.incr()\n          }\n          val gnOverrideKey = GenericNotificationOverrideKey(\n            userId = userId,\n            hashKey = genericNotificationKey.hashKey,\n            timestampMillis = genericNotificationKey.timestampMillis,\n            overrideId = overrideId\n          )\n          val mhKeyVal =\n            ((userId, candidate.impressionId), gnOverrideKey)\n          storeGenericNotifOverrideKey.incr()\n          nTabHistoryStore.put(mhKeyVal)\n        case _ => Future.Unit\n      }\n    } else Future.Unit\n  }\n\n  private def candidateEligibleForOverride(\n    targetHistory: History,\n    targetEntries: Seq[FrigateNotification],\n  ): FrigateNotification = {\n    val timestampToEntriesMap =\n      targetEntries.map { entry =>\n        PushOverrideInfo\n          .getTimestampInMillisForFrigateNotification(entry, targetHistory, statsReceiver)\n          .getOrElse(PushConstants.DefaultLookBackForHistory.ago.inMilliseconds) -> entry\n      }.toMap\n\n    PushOverrideInfo.getOldestFrigateNotification(timestampToEntriesMap)\n  }\n\n  private def overrideNSlot(candidate: PushCandidate): Future[Boolean] = {\n    if (candidate.target.params(FSParams.EnableNslotsForOverrideOnNtab)) {\n      val targetHistoryFut = candidate.target.history\n      targetHistoryFut.flatMap { targetHistory =>\n        val nonEligibleOverrideTypes =\n          Seq(RecTypes.RecommendedSpaceFanoutTypes ++ RecTypes.ScheduledSpaceReminderTypes)\n\n        val overrideNotifs = PushOverrideInfo\n          .getOverrideEligiblePushNotifications(\n            targetHistory,\n            candidate.target.params(FSParams.OverrideNotificationsLookbackDurationForNTab),\n            statsReceiver\n          ).filterNot {\n            case notification =>\n              nonEligibleOverrideTypes.contains(notification.commonRecommendationType)\n          }\n\n        val maxNumUnreadEntries =\n          candidate.target.params(FSParams.OverrideNotificationsMaxCountForNTab)\n        if (overrideNotifs.nonEmpty && overrideNotifs.size >= maxNumUnreadEntries) {\n          val eligibleOverrideCandidateOpt = candidateEligibleForOverride(\n            targetHistory,\n            overrideNotifs\n          )\n          eligibleOverrideCandidateOpt match {\n            case overrideCandidate if overrideCandidate.impressionId.nonEmpty =>\n              deleteNTabEntryFromGenericNotificationStore(\n                candidate.target.targetId,\n                eligibleOverrideCandidateOpt.impressionId.head)\n            case _ =>\n              ntabOverrideImpressionNotFound.incr()\n              Future.False\n          }\n        } else Future.False\n      }\n    } else {\n      Future.False\n    }\n  }\n\n  private def shouldDisableNtabOverride(candidate: PushCandidate): Boolean =\n    RecTypes.isSendHandlerType(candidate.commonRecType)\n\n  private def overridePreviousEntry(candidate: PushCandidate): Future[Boolean] = {\n\n    if (shouldDisableNtabOverride(candidate)) {\n      nTabOverrideOverriddenStat.incr()\n      Future.False\n    } else {\n      val targetHistoryFut = candidate.target.history\n      targetHistoryFut.flatMap { targetHistory =>\n        val impressionIds = PushOverrideInfo.getImpressionIdsOfPrevEligiblePushNotif(\n          targetHistory,\n          candidate.target.params(FSParams.OverrideNotificationsLookbackDurationForImpressionId),\n          statsReceiver)\n\n        if (impressionIds.nonEmpty) {\n          deleteNTabEntryFromGenericNotificationStore(candidate.target.targetId, impressionIds.head)\n        } else {\n          ntabOverrideImpressionNotFound.incr()\n          Future.False // no deletes issued\n        }\n      }\n    }\n  }\n\n  private def deleteCachedApiTimeline(\n    candidate: PushCandidate,\n    isNotificationOverridden: Boolean\n  ): Future[Unit] = {\n    if (isNotificationOverridden && candidate.target.params(FSParams.EnableDeletingNtabTimeline)) {\n      val deleteTimelineRequest = DeleteCurrentTimelineForUserRequest(candidate.target.targetId)\n      nTabDeleteTimelineRequests.incr()\n      nTabDeleteTimeline(deleteTimelineRequest)\n    } else {\n      Future.Unit\n    }\n  }\n\n  private def deleteNTabEntryFromGenericNotificationStore(\n    targetUserId: Long,\n    targetImpressionId: String\n  ): Future[Boolean] = {\n    val mhKey = (targetUserId, targetImpressionId)\n    val genericNotificationKeyFut = nTabHistoryStore.get(mhKey)\n    genericNotificationKeyFut.flatMap {\n      case Some(genericNotifOverrideKey) =>\n        val gnKey = GenericNotificationKey(\n          userId = genericNotifOverrideKey.userId,\n          hashKey = genericNotifOverrideKey.hashKey,\n          timestampMillis = genericNotifOverrideKey.timestampMillis\n        )\n        val deleteEntryRequest = DeleteGenericNotificationRequest(gnKey)\n        nTabDeleteRequests.incr()\n        nTabDelete(deleteEntryRequest).map(_ => true)\n      case _ =>\n        prevGenericNotifKeyNotFound.incr()\n        Future.False\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/target/CustomFSFields.scala",
    "content": "package com.twitter.frigate.pushservice.target\n\nimport com.twitter.featureswitches.FSCustomMapInput\nimport com.twitter.featureswitches.parsing.DynMap\nimport com.twitter.frigate.common.store.deviceinfo.DeviceInfo\nimport com.twitter.frigate.pushservice.util.NsfwInfo\nimport com.twitter.gizmoduck.thriftscala.User\n\nobject CustomFSFields {\n  private val IsReturningUser = \"is_returning_user\"\n  private val DaysSinceSignup = \"days_since_signup\"\n  private val DaysSinceLogin = \"days_since_login\"\n  private val DaysSinceReactivation = \"days_since_reactivation\"\n  private val ReactivationDate = \"reactivation_date\"\n  private val FollowGraphSize = \"follow_graph_size\"\n  private val GizmoduckUserType = \"gizmoduck_user_type\"\n  private val UserAge = \"mr_user_age\"\n  private val SensitiveOptIn = \"sensitive_opt_in\"\n  private val NsfwFollowRatio = \"nsfw_follow_ratio\"\n  private val TotalFollows = \"follow_count\"\n  private val NsfwRealGraphScore = \"nsfw_real_graph_score\"\n  private val NsfwProfileVisit = \"nsfw_profile_visit\"\n  private val TotalSearches = \"total_searches\"\n  private val NsfwSearchScore = \"nsfw_search_score\"\n  private val HasReportedNsfw = \"nsfw_reported\"\n  private val HasDislikedNsfw = \"nsfw_disliked\"\n  private val UserState = \"user_state\"\n  private val MrUserState = \"mr_user_state\"\n  private val NumDaysReceivedPushInLast30Days =\n    \"num_days_received_push_in_last_30_days\"\n  private val RecommendationsSetting = \"recommendations_setting\"\n  private val TopicsSetting = \"topics_setting\"\n  private val SpacesSetting = \"spaces_setting\"\n  private val NewsSetting = \"news_setting\"\n  private val LiveVideoSetting = \"live_video_setting\"\n  private val HasRecentPushableRebDevice = \"has_recent_pushable_rweb_device\"\n  private val RequestSource = \"request_source\"\n}\n\ncase class CustomFSFields(\n  isReactivatedUser: Boolean,\n  daysSinceSignup: Int,\n  numDaysReceivedPushInLast30Days: Int,\n  daysSinceLogin: Option[Int],\n  daysSinceReactivation: Option[Int],\n  user: Option[User],\n  userState: Option[String],\n  mrUserState: Option[String],\n  reactivationDate: Option[String],\n  requestSource: Option[String],\n  userAge: Option[Int],\n  nsfwInfo: Option[NsfwInfo],\n  deviceInfo: Option[DeviceInfo]) {\n\n  import CustomFSFields._\n\n  private val keyValMap: Map[String, Any] = Map(\n    IsReturningUser -> isReactivatedUser,\n    DaysSinceSignup -> daysSinceSignup,\n    DaysSinceLogin -> daysSinceLogin,\n    NumDaysReceivedPushInLast30Days -> numDaysReceivedPushInLast30Days\n  ) ++\n    daysSinceReactivation.map(DaysSinceReactivation -> _) ++\n    reactivationDate.map(ReactivationDate -> _) ++\n    user.flatMap(_.counts.map(counts => FollowGraphSize -> counts.following)) ++\n    user.map(u => GizmoduckUserType -> u.userType.name) ++\n    userState.map(UserState -> _) ++\n    mrUserState.map(MrUserState -> _) ++\n    requestSource.map(RequestSource -> _) ++\n    userAge.map(UserAge -> _) ++\n    nsfwInfo.flatMap(_.senstiveOptIn).map(SensitiveOptIn -> _) ++\n    nsfwInfo\n      .map { nsInfo =>\n        Map[String, Any](\n          NsfwFollowRatio -> nsInfo.nsfwFollowRatio,\n          TotalFollows -> nsInfo.totalFollowCount,\n          NsfwRealGraphScore -> nsInfo.realGraphScore,\n          NsfwProfileVisit -> nsInfo.nsfwProfileVisits,\n          TotalSearches -> nsInfo.totalSearches,\n          NsfwSearchScore -> nsInfo.searchNsfwScore,\n          HasReportedNsfw -> nsInfo.hasReported,\n          HasDislikedNsfw -> nsInfo.hasDisliked\n        )\n      }.getOrElse(Map.empty[String, Any]) ++\n    deviceInfo\n      .map { deviceInfo =>\n        Map[String, Boolean](\n          RecommendationsSetting -> deviceInfo.isRecommendationsEligible,\n          TopicsSetting -> deviceInfo.isTopicsEligible,\n          SpacesSetting -> deviceInfo.isSpacesEligible,\n          LiveVideoSetting -> deviceInfo.isBroadcastsEligible,\n          NewsSetting -> deviceInfo.isNewsEligible,\n          HasRecentPushableRebDevice -> deviceInfo.hasRecentPushableRWebDevice\n        )\n      }.getOrElse(Map.empty[String, Boolean])\n\n  val fsMap = FSCustomMapInput(DynMap(keyValMap))\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/target/LoggedOutPushTargetUserBuilder.scala",
    "content": "package com.twitter.frigate.pushservice.target\n\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.decider.Decider\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.FeatureMap\nimport com.twitter.frigate.common.history.History\nimport com.twitter.frigate.common.history.HistoryStoreKeyContext\nimport com.twitter.frigate.common.history.MagicFanoutReasonHistory\nimport com.twitter.frigate.common.history.PushServiceHistoryStore\nimport com.twitter.frigate.common.history.RecItems\nimport com.twitter.frigate.common.store.deviceinfo.DeviceInfo\nimport com.twitter.frigate.common.util.ABDeciderWithOverride\nimport com.twitter.frigate.common.util.LanguageLocaleUtil\nimport com.twitter.frigate.data_pipeline.features_common.MrRequestContextForFeatureStore\nimport com.twitter.frigate.data_pipeline.thriftscala.UserHistoryValue\nimport com.twitter.frigate.dau_model.thriftscala.DauProbability\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.thriftscala.PushContext\nimport com.twitter.frigate.thriftscala.UserForPushTargeting\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.stp.thriftscala.STPResult\nimport com.twitter.interests.thriftscala.InterestId\nimport com.twitter.notificationservice.genericfeedbackstore.FeedbackPromptValue\nimport com.twitter.notificationservice.thriftscala.CaretFeedbackDetails\nimport com.twitter.nrel.hydration.push.HydrationContext\nimport com.twitter.permissions_storage.thriftscala.AppPermission\nimport com.twitter.service.metastore.gen.thriftscala.Location\nimport com.twitter.service.metastore.gen.thriftscala.UserLanguages\nimport com.twitter.stitch.Stitch\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.columns.frigate.logged_out_web_notifications.thriftscala.LOWebNotificationMetadata\nimport com.twitter.timelines.configapi\nimport com.twitter.timelines.configapi.Params\nimport com.twitter.timelines.real_graph.v1.thriftscala.RealGraphFeatures\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.wtf.scalding.common.thriftscala.UserFeatures\n\ncase class LoggedOutPushTargetUserBuilder(\n  historyStore: PushServiceHistoryStore,\n  inputDecider: Decider,\n  inputAbDecider: LoggingABDecider,\n  loggedOutPushInfoStore: ReadableStore[Long, LOWebNotificationMetadata]\n)(\n  globalStatsReceiver: StatsReceiver) {\n  private val stats = globalStatsReceiver.scope(\"LORefreshForPushHandler\")\n  private val noHistoryCounter = stats.counter(\"no_logged_out_history\")\n  private val historyFoundCounter = stats.counter(\"logged_out_history_counter\")\n  private val noLoggedOutUserCounter = stats.counter(\"no_logged_out_user\")\n  private val countryCodeCounter = stats.counter(\"country_counter\")\n  private val noCountryCodeCounter = stats.counter(\"no_country_counter\")\n  private val noLanguageCodeCounter = stats.counter(\"no_language_counter\")\n\n  def buildTarget(\n    guestId: Long,\n    inputPushContext: Option[PushContext]\n  ): Future[Target] = {\n\n    val historyStoreKeyContext = HistoryStoreKeyContext(\n      guestId,\n      inputPushContext.flatMap(_.useMemcacheForHistory).getOrElse(false)\n    )\n    if (historyStore.get(historyStoreKeyContext, Some(30.days)) == Future.None) {\n      noHistoryCounter.incr()\n    } else {\n      historyFoundCounter.incr()\n\n    }\n    if (loggedOutPushInfoStore.get(guestId) == Future.None) {\n      noLoggedOutUserCounter.incr()\n    }\n    Future\n      .join(\n        historyStore.get(historyStoreKeyContext, Some(30.days)),\n        loggedOutPushInfoStore.get(guestId)\n      ).map {\n        case (loNotifHistory, loggedOutUserPushInfo) =>\n          new Target {\n            override lazy val stats: StatsReceiver = globalStatsReceiver\n            override val targetId: Long = guestId\n            override val targetGuestId = Some(guestId)\n            override lazy val decider: Decider = inputDecider\n            override lazy val loggedOutMetadata = Future.value(loggedOutUserPushInfo)\n            val rawLanguageFut = loggedOutMetadata.map { metadata => metadata.map(_.language) }\n            override val targetLanguage: Future[Option[String]] = rawLanguageFut.map { rawLang =>\n              if (rawLang.isDefined) {\n                val lang = LanguageLocaleUtil.getStandardLanguageCode(rawLang.get)\n                if (lang.isEmpty) {\n                  noLanguageCodeCounter.incr()\n                  None\n                } else {\n                  Option(lang)\n                }\n              } else None\n            }\n            val country = loggedOutMetadata.map(_.map(_.countryCode))\n            if (country.isDefined) {\n              countryCodeCounter.incr()\n            } else {\n              noCountryCodeCounter.incr()\n            }\n            if (loNotifHistory == null) {\n              noHistoryCounter.incr()\n            } else {\n              historyFoundCounter.incr()\n            }\n            override lazy val location: Future[Option[Location]] = country.map {\n              case Some(code) =>\n                Some(\n                  Location(\n                    city = \"\",\n                    region = \"\",\n                    countryCode = code,\n                    confidence = 0.0,\n                    lat = None,\n                    lon = None,\n                    metro = None,\n                    placeIds = None,\n                    weightedLocations = None,\n                    createdAtMsec = None,\n                    ip = None,\n                    isSignupIp = None,\n                    placeMap = None\n                  ))\n              case _ => None\n            }\n\n            override lazy val pushContext: Option[PushContext] = inputPushContext\n            override lazy val history: Future[History] = Future.value(loNotifHistory)\n            override lazy val magicFanoutReasonHistory30Days: Future[MagicFanoutReasonHistory] =\n              Future.value(null)\n            override lazy val globalStats: StatsReceiver = globalStatsReceiver\n            override lazy val pushTargeting: Future[Option[UserForPushTargeting]] = Future.None\n            override lazy val appPermissions: Future[Option[AppPermission]] = Future.None\n            override lazy val lastHTLVisitTimestamp: Future[Option[Long]] = Future.None\n            override lazy val pushRecItems: Future[RecItems] = Future.value(null)\n\n            override lazy val isNewSignup: Boolean = false\n            override lazy val metastoreLanguages: Future[Option[UserLanguages]] = Future.None\n            override lazy val optOutUserInterests: Future[Option[Seq[InterestId]]] = Future.None\n            override lazy val mrRequestContextForFeatureStore: MrRequestContextForFeatureStore =\n              null\n            override lazy val targetUser: Future[Option[User]] = Future.None\n            override lazy val notificationFeedbacks: Future[Option[Seq[FeedbackPromptValue]]] =\n              Future.None\n            override lazy val promptFeedbacks: Stitch[Seq[FeedbackPromptValue]] = null\n            override lazy val seedsWithWeight: Future[Option[Map[Long, Double]]] = Future.None\n            override lazy val tweetImpressionResults: Future[Seq[Long]] = Future.Nil\n            override lazy val params: configapi.Params = Params.Empty\n            override lazy val deviceInfo: Future[Option[DeviceInfo]] = Future.None\n            override lazy val userFeatures: Future[Option[UserFeatures]] = Future.None\n            override lazy val isOpenAppExperimentUser: Future[Boolean] = Future.False\n            override lazy val featureMap: Future[FeatureMap] = Future.value(null)\n            override lazy val dauProbability: Future[Option[DauProbability]] = Future.None\n            override lazy val labeledPushRecsHydrated: Future[Option[UserHistoryValue]] =\n              Future.None\n            override lazy val onlineLabeledPushRecs: Future[Option[UserHistoryValue]] = Future.None\n            override lazy val realGraphFeatures: Future[Option[RealGraphFeatures]] = Future.None\n            override lazy val stpResult: Future[Option[STPResult]] = Future.None\n            override lazy val globalOptoutProbabilities: Seq[Future[Option[Double]]] = Seq.empty\n            override lazy val bucketOptoutProbability: Future[Option[Double]] = Future.None\n            override lazy val utcOffset: Future[Option[Duration]] = Future.None\n            override lazy val abDecider: ABDeciderWithOverride =\n              ABDeciderWithOverride(inputAbDecider, ddgOverrideOption)(globalStatsReceiver)\n            override lazy val resurrectionDate: Future[Option[String]] = Future.None\n            override lazy val isResurrectedUser: Boolean = false\n            override lazy val timeSinceResurrection: Option[Duration] = None\n            override lazy val inlineActionHistory: Future[Seq[(Long, String)]] = Future.Nil\n            override lazy val caretFeedbacks: Future[Option[Seq[CaretFeedbackDetails]]] =\n              Future.None\n\n            override def targetHydrationContext: Future[HydrationContext] = Future.value(null)\n            override def isBlueVerified: Future[Option[Boolean]] = Future.None\n            override def isVerified: Future[Option[Boolean]] = Future.None\n            override def isSuperFollowCreator: Future[Option[Boolean]] = Future.None\n          }\n      }\n  }\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/target/PushTargetUserBuilder.scala",
    "content": "package com.twitter.frigate.pushservice.target\n\nimport com.twitter.abdecider.LoggingABDecider\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.decider.Decider\nimport com.twitter.discovery.common.configapi.ConfigParamsBuilder\nimport com.twitter.discovery.common.configapi.ExperimentOverride\nimport com.twitter.featureswitches.Recipient\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.history._\nimport com.twitter.frigate.common.logger.MRLogger\nimport com.twitter.frigate.common.store.FeedbackRequest\nimport com.twitter.frigate.common.store.PushRecItemsKey\nimport com.twitter.frigate.common.store.deviceinfo.DeviceInfo\nimport com.twitter.frigate.common.store.interests.UserId\nimport com.twitter.frigate.common.util._\nimport com.twitter.frigate.data_pipeline.features_common.MrRequestContextForFeatureStore\nimport com.twitter.frigate.data_pipeline.thriftscala.UserHistoryValue\nimport com.twitter.frigate.dau_model.thriftscala.DauProbability\nimport com.twitter.frigate.pushcap.thriftscala.PushcapInfo\nimport com.twitter.frigate.pushcap.thriftscala.PushcapUserHistory\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.ml.HydrationContextBuilder\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.store.LabeledPushRecsStoreKey\nimport com.twitter.frigate.pushservice.store.OnlineUserHistoryKey\nimport com.twitter.frigate.pushservice.util.NsfwInfo\nimport com.twitter.frigate.pushservice.util.NsfwPersonalizationUtil\nimport com.twitter.frigate.pushservice.util.PushAppPermissionUtil\nimport com.twitter.frigate.pushservice.util.PushCapUtil.getMinimumRestrictedPushcapInfo\nimport com.twitter.frigate.pushservice.thriftscala.PushContext\nimport com.twitter.frigate.pushservice.thriftscala.RequestSource\nimport com.twitter.frigate.thriftscala.SecondaryAccountsByUserState\nimport com.twitter.frigate.thriftscala.UserForPushTargeting\nimport com.twitter.frigate.user_states.thriftscala.MRUserHmmState\nimport com.twitter.frigate.user_states.thriftscala.{UserState => MrUserState}\nimport com.twitter.frontpage.stream.util.SnowflakeUtil\nimport com.twitter.geoduck.common.thriftscala.Place\nimport com.twitter.geoduck.service.thriftscala.LocationResponse\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.model.user_state.UserState\nimport com.twitter.hermit.model.user_state.UserState.UserState\nimport com.twitter.hermit.stp.thriftscala.STPResult\nimport com.twitter.ibis.thriftscala.ContentRecData\nimport com.twitter.interests.thriftscala.InterestId\nimport com.twitter.notificationservice.feedback.thriftscala.FeedbackInteraction\nimport com.twitter.notificationservice.genericfeedbackstore.FeedbackPromptValue\nimport com.twitter.notificationservice.genericfeedbackstore.GenericFeedbackStore\nimport com.twitter.notificationservice.genericfeedbackstore.GenericFeedbackStoreException\nimport com.twitter.notificationservice.model.service.DismissMenuFeedbackAction\nimport com.twitter.notificationservice.scribe.manhattan.GenericNotificationsFeedbackRequest\nimport com.twitter.notificationservice.thriftscala.CaretFeedbackDetails\nimport com.twitter.nrel.heavyranker.FeatureHydrator\nimport com.twitter.nrel.hydration.push.HydrationContext\nimport com.twitter.permissions_storage.thriftscala.AppPermission\nimport com.twitter.rux.common.strato.thriftscala.UserTargetingProperty\nimport com.twitter.scio.nsfw_user_segmentation.thriftscala.NSFWUserSegmentation\nimport com.twitter.service.metastore.gen.thriftscala.Location\nimport com.twitter.service.metastore.gen.thriftscala.UserLanguages\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi\nimport com.twitter.timelines.real_graph.thriftscala.{RealGraphFeatures => RealGraphFeaturesUnion}\nimport com.twitter.timelines.real_graph.v1.thriftscala.RealGraphFeatures\nimport com.twitter.ubs.thriftscala.SellerApplicationState\nimport com.twitter.ubs.thriftscala.SellerTrack\nimport com.twitter.user_session_store.thriftscala.UserSession\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport com.twitter.wtf.scalding.common.thriftscala.UserFeatures\n\ncase class PushTargetUserBuilder(\n  historyStore: PushServiceHistoryStore,\n  emailHistoryStore: PushServiceHistoryStore,\n  labeledPushRecsStore: ReadableStore[LabeledPushRecsStoreKey, UserHistoryValue],\n  onlineUserHistoryStore: ReadableStore[OnlineUserHistoryKey, UserHistoryValue],\n  pushRecItemsStore: ReadableStore[PushRecItemsKey, RecItems],\n  userStore: ReadableStore[Long, User],\n  pushInfoStore: ReadableStore[Long, UserForPushTargeting],\n  userCountryStore: ReadableStore[Long, Location],\n  userUtcOffsetStore: ReadableStore[Long, Duration],\n  dauProbabilityStore: ReadableStore[Long, DauProbability],\n  nsfwConsumerStore: ReadableStore[Long, NSFWUserSegmentation],\n  userFeatureStore: ReadableStore[Long, UserFeatures],\n  userTargetingPropertyStore: ReadableStore[Long, UserTargetingProperty],\n  mrUserStateStore: ReadableStore[Long, MRUserHmmState],\n  tweetImpressionStore: ReadableStore[Long, Seq[Long]],\n  ntabCaretFeedbackStore: ReadableStore[GenericNotificationsFeedbackRequest, Seq[\n    CaretFeedbackDetails\n  ]],\n  genericFeedbackStore: ReadableStore[FeedbackRequest, Seq[FeedbackPromptValue]],\n  genericNotificationFeedbackStore: GenericFeedbackStore,\n  timelinesUserSessionStore: ReadableStore[Long, UserSession],\n  cachedTweetyPieStore: ReadableStore[Long, TweetyPieResult],\n  strongTiesStore: ReadableStore[Long, STPResult],\n  userHTLLastVisitStore: ReadableStore[Long, Seq[Long]],\n  userLanguagesStore: ReadableStore[Long, UserLanguages],\n  inputDecider: Decider,\n  inputAbDecider: LoggingABDecider,\n  realGraphScoresTop500InStore: ReadableStore[Long, Map[Long, Double]],\n  recentFollowsStore: ReadableStore[Long, Seq[Long]],\n  resurrectedUserStore: ReadableStore[Long, String],\n  configParamsBuilder: ConfigParamsBuilder,\n  optOutUserInterestsStore: ReadableStore[UserId, Seq[InterestId]],\n  deviceInfoStore: ReadableStore[Long, DeviceInfo],\n  pushcapDynamicPredictionStore: ReadableStore[Long, PushcapUserHistory],\n  appPermissionStore: ReadableStore[(Long, (String, String)), AppPermission],\n  optoutModelScorer: PushMLModelScorer,\n  inlineActionHistoryStore: ReadableStore[Long, Seq[(Long, String)]],\n  featureHydrator: FeatureHydrator,\n  openAppUserStore: ReadableStore[Long, Boolean],\n  openedPushByHourAggregatedStore: ReadableStore[Long, Map[Int, Int]],\n  geoduckStoreV2: ReadableStore[Long, LocationResponse],\n  superFollowEligibilityUserStore: ReadableStore[Long, Boolean],\n  superFollowApplicationStatusStore: ReadableStore[(Long, SellerTrack), SellerApplicationState]\n)(\n  globalStatsReceiver: StatsReceiver) {\n\n  implicit val statsReceiver: StatsReceiver = globalStatsReceiver\n\n  private val log = MRLogger(\"PushTargetUserBuilder\")\n  private val recentFollowscounter = statsReceiver.counter(\"query_recent_follows\")\n  private val isModelTrainingDataCounter =\n    statsReceiver.scope(\"TargetUserBuilder\").counter(\"is_model_training\")\n  private val feedbackStoreGenerationErr = statsReceiver.counter(\"feedback_store_generation_error\")\n  private val newSignUpUserStats = statsReceiver.counter(\"new_signup_user\")\n  private val pushcapSelectionStat = statsReceiver.scope(\"pushcap_modeling\")\n  private val dormantUserCount = statsReceiver.counter(\"dormant_user_counter\")\n  private val optoutModelStat = statsReceiver.scope(\"optout_modeling\")\n  private val placeFoundStat = statsReceiver.scope(\"geoduck_v2\").stat(\"places_found\")\n  private val placesNotFound = statsReceiver.scope(\"geoduck_v2\").counter(\"places_not_found\")\n  // Email history store stats\n  private val emailHistoryStats = statsReceiver.scope(\"email_tweet_history\")\n  private val emptyEmailHistoryCounter = emailHistoryStats.counter(\"empty\")\n  private val nonEmptyEmailHistoryCounter = emailHistoryStats.counter(\"non_empty\")\n\n  private val MagicRecsCategory = \"MagicRecs\"\n  private val MomentsViaMagicRecsCategory = \"MomentsViaMagicRecs\"\n  private val MomentsCategory = \"Moments\"\n\n  def buildTarget(\n    userId: Long,\n    inputPushContext: Option[PushContext],\n    forcedFeatureValues: Option[Map[String, configapi.FeatureValue]] = None\n  ): Future[Target] = {\n    val historyStoreKeyContext = HistoryStoreKeyContext(\n      userId,\n      inputPushContext.flatMap(_.useMemcacheForHistory).getOrElse(false)\n    )\n    Future\n      .join(\n        userStore.get(userId),\n        deviceInfoStore.get(userId),\n        pushInfoStore.get(userId),\n        historyStore.get(historyStoreKeyContext, Some(30.days)),\n        emailHistoryStore.get(\n          HistoryStoreKeyContext(userId, useStoreB = false),\n          Some(7.days) // we only keep 7 days of email tweet history\n        )\n      ).flatMap {\n        case (userOpt, deviceInfoOpt, userForPushTargetingInfoOpt, notifHistory, emailHistory) =>\n          getCustomFSFields(\n            userId,\n            userOpt,\n            deviceInfoOpt,\n            userForPushTargetingInfoOpt,\n            notifHistory,\n            inputPushContext.flatMap(_.requestSource)).map { customFSField =>\n            new Target {\n\n              override lazy val stats: StatsReceiver = statsReceiver\n\n              override val targetId: Long = userId\n\n              override val targetUser: Future[Option[User]] = Future.value(userOpt)\n\n              override val isEmailUser: Boolean =\n                inputPushContext.flatMap(_.requestSource) match {\n                  case Some(source) if source == RequestSource.Email => true\n                  case _ => false\n                }\n\n              override val pushContext = inputPushContext\n\n              override def globalStats: StatsReceiver = globalStatsReceiver\n\n              override lazy val abDecider: ABDeciderWithOverride =\n                ABDeciderWithOverride(inputAbDecider, ddgOverrideOption)\n\n              override lazy val pushRecItems: Future[RecItems] =\n                pushRecItemsStore\n                  .get(PushRecItemsKey(historyStoreKeyContext, history))\n                  .map(_.getOrElse(RecItems.empty))\n\n              // List of past tweet candidates sent in the past through email with timestamp\n              override lazy val emailRecItems: Future[Seq[(Time, Long)]] = {\n                Future.value {\n                  emailHistory.sortedEmailHistory.flatMap {\n                    case (timeStamp, notification) =>\n                      notification.contentRecsNotification\n                        .map { notification =>\n                          notification.recommendations.contentRecCollections.flatMap {\n                            contentRecs =>\n                              contentRecs.contentRecModules.flatMap { contentRecModule =>\n                                contentRecModule.recData match {\n                                  case ContentRecData.TweetRec(tweetRec) =>\n                                    nonEmptyEmailHistoryCounter.incr()\n                                    Seq(tweetRec.tweetId)\n                                  case _ =>\n                                    emptyEmailHistoryCounter.incr()\n                                    Nil\n                                }\n                              }\n                          }\n                        }.getOrElse {\n                          emptyEmailHistoryCounter.incr()\n                          Nil\n                        }.map(timeStamp -> _)\n                  }\n                }\n              }\n\n              override lazy val history: Future[History] = Future.value(notifHistory)\n\n              override lazy val pushTargeting: Future[Option[UserForPushTargeting]] =\n                Future.value(userForPushTargetingInfoOpt)\n\n              override lazy val decider: Decider = inputDecider\n\n              override lazy val location: Future[Option[Location]] =\n                userCountryStore.get(userId)\n\n              override lazy val deviceInfo: Future[Option[DeviceInfo]] =\n                Future.value(deviceInfoOpt)\n\n              override lazy val targetLanguage: Future[Option[String]] = targetUser map { userOpt =>\n                userOpt.flatMap(_.account.map(_.language))\n              }\n\n              override lazy val targetAgeInYears: Future[Option[Int]] =\n                Future.value(customFSField.userAge)\n\n              override lazy val metastoreLanguages: Future[Option[UserLanguages]] =\n                userLanguagesStore.get(targetId)\n\n              override lazy val utcOffset: Future[Option[Duration]] =\n                userUtcOffsetStore.get(targetId)\n\n              override lazy val userFeatures: Future[Option[UserFeatures]] =\n                userFeatureStore.get(targetId)\n\n              override lazy val targetUserState: Future[Option[UserState]] =\n                Future.value(\n                  customFSField.userState\n                    .flatMap(userState => UserState.valueOf(userState)))\n\n              override lazy val targetMrUserState: Future[Option[MrUserState]] =\n                Future.value(\n                  customFSField.mrUserState\n                    .flatMap(mrUserState => MrUserState.valueOf(mrUserState)))\n\n              override lazy val accountStateWithDeviceInfo: Future[\n                Option[SecondaryAccountsByUserState]\n              ] = Future.None\n\n              override lazy val dauProbability: Future[Option[DauProbability]] = {\n                dauProbabilityStore.get(targetId)\n              }\n\n              override lazy val labeledPushRecsHydrated: Future[Option[UserHistoryValue]] =\n                labeledPushRecsStore.get(LabeledPushRecsStoreKey(this, historyStoreKeyContext))\n\n              override lazy val onlineLabeledPushRecs: Future[Option[UserHistoryValue]] =\n                labeledPushRecsHydrated.flatMap { labeledPushRecs =>\n                  history.flatMap { history =>\n                    onlineUserHistoryStore.get(\n                      OnlineUserHistoryKey(targetId, labeledPushRecs, Some(history))\n                    )\n                  }\n                }\n\n              override lazy val tweetImpressionResults: Future[Seq[Long]] =\n                tweetImpressionStore.get(targetId).map {\n                  case Some(impressionList) =>\n                    impressionList\n                  case _ => Nil\n                }\n\n              override lazy val realGraphFeatures: Future[Option[RealGraphFeatures]] =\n                timelinesUserSessionStore.get(targetId).map { userSessionOpt =>\n                  userSessionOpt.flatMap { userSession =>\n                    userSession.realGraphFeatures.collect {\n                      case RealGraphFeaturesUnion.V1(rGFeatures) =>\n                        rGFeatures\n                    }\n                  }\n                }\n\n              override lazy val stpResult: Future[Option[STPResult]] =\n                strongTiesStore.get(targetId)\n\n              override lazy val lastHTLVisitTimestamp: Future[Option[Long]] =\n                userHTLLastVisitStore.get(targetId).map {\n                  case Some(lastVisitTimestamps) if lastVisitTimestamps.nonEmpty =>\n                    Some(lastVisitTimestamps.max)\n                  case _ => None\n                }\n\n              override lazy val caretFeedbacks: Future[Option[Seq[CaretFeedbackDetails]]] = {\n                val scribeHistoryLookbackPeriod = 365.days\n                val now = Time.now\n                val request = GenericNotificationsFeedbackRequest(\n                  userId = targetId,\n                  eventStartTimestamp = now - scribeHistoryLookbackPeriod,\n                  eventEndTimestamp = now,\n                  filterCategory =\n                    Some(Set(MagicRecsCategory, MomentsViaMagicRecsCategory, MomentsCategory)),\n                  filterFeedbackActionText =\n                    Some(Set(DismissMenuFeedbackAction.FeedbackActionTextSeeLessOften))\n                )\n                ntabCaretFeedbackStore.get(request)\n              }\n\n              override lazy val notificationFeedbacks: Future[\n                Option[Seq[FeedbackPromptValue]]\n              ] = {\n                val scribeHistoryLookbackPeriod = 30.days\n                val now = Time.now\n                val request = FeedbackRequest(\n                  userId = targetId,\n                  oldestTimestamp = scribeHistoryLookbackPeriod.ago,\n                  newestTimestamp = Time.now,\n                  feedbackInteraction = FeedbackInteraction.Feedback\n                )\n                genericFeedbackStore.get(request)\n              }\n\n              // DEPRECATED: Use notificationFeedbacks instead.\n              // This method will increase latency dramatically.\n              override lazy val promptFeedbacks: Stitch[Seq[FeedbackPromptValue]] = {\n                val scribeHistoryLookbackPeriod = 7.days\n\n                genericNotificationFeedbackStore\n                  .getAll(\n                    userId = targetId,\n                    oldestTimestamp = scribeHistoryLookbackPeriod.ago,\n                    newestTimestamp = Time.now,\n                    feedbackInteraction = FeedbackInteraction.Feedback\n                  ).handle {\n                    case _: GenericFeedbackStoreException => {\n                      feedbackStoreGenerationErr.incr()\n                      Seq.empty[FeedbackPromptValue]\n                    }\n                  }\n              }\n\n              override lazy val optOutUserInterests: Future[Option[Seq[InterestId]]] = {\n                optOutUserInterestsStore.get(targetId)\n              }\n\n              private val experimentOverride = ddgOverrideOption.map {\n                case DDGOverride(Some(exp), Some(bucket)) =>\n                  Set(ExperimentOverride(exp, bucket))\n                case _ => Set.empty[ExperimentOverride]\n              }\n\n              override val signupCountryCode =\n                Future.value(userOpt.flatMap(_.safety.flatMap(_.signupCountryCode)))\n\n              override lazy val params: configapi.Params = {\n                val fsRecipient = Recipient(\n                  userId = Some(targetId),\n                  userRoles = userOpt.flatMap(_.roles.map(_.roles.toSet)),\n                  clientApplicationId = deviceInfoOpt.flatMap(_.guessedPrimaryClientAppId),\n                  userAgent = deviceInfoOpt.flatMap(_.guessedPrimaryDeviceUserAgent),\n                  countryCode =\n                    userOpt.flatMap(_.account.flatMap(_.countryCode.map(_.toUpperCase))),\n                  customFields = Some(customFSField.fsMap),\n                  signupCountryCode =\n                    userOpt.flatMap(_.safety.flatMap(_.signupCountryCode.map(_.toUpperCase))),\n                  languageCode = deviceInfoOpt.flatMap {\n                    _.deviceLanguages.flatMap(IbisAppPushDeviceSettingsUtil.inferredDeviceLanguage)\n                  }\n                )\n\n                configParamsBuilder.build(\n                  userId = Some(targetId),\n                  experimentOverrides = experimentOverride,\n                  featureRecipient = Some(fsRecipient),\n                  forcedFeatureValues = forcedFeatureValues.getOrElse(Map.empty),\n                )\n              }\n\n              override lazy val mrRequestContextForFeatureStore =\n                MrRequestContextForFeatureStore(targetId, params, isModelTrainingData)\n\n              override lazy val dynamicPushcap: Future[Option[PushcapInfo]] = {\n                // Get the pushcap from the pushcap model prediction store\n                if (params(PushParams.EnableModelBasedPushcapAssignments)) {\n                  val originalPushcapInfoFut =\n                    PushCapUtil.getPushcapFromUserHistory(\n                      userId,\n                      pushcapDynamicPredictionStore,\n                      params(FeatureSwitchParams.PushcapModelType),\n                      params(FeatureSwitchParams.PushcapModelPredictionVersion),\n                      pushcapSelectionStat\n                    )\n                  // Modify the push cap info if there is a restricted min value for predicted push caps.\n                  val restrictedPushcap = params(PushFeatureSwitchParams.RestrictedMinModelPushcap)\n                  originalPushcapInfoFut.map {\n                    case Some(originalPushcapInfo) =>\n                      Some(\n                        getMinimumRestrictedPushcapInfo(\n                          restrictedPushcap,\n                          originalPushcapInfo,\n                          pushcapSelectionStat))\n                    case _ => None\n                  }\n                } else Future.value(None)\n              }\n\n              override lazy val targetHydrationContext: Future[HydrationContext] =\n                HydrationContextBuilder.build(this)\n\n              override lazy val featureMap: Future[FeatureMap] =\n                targetHydrationContext.flatMap { hydrationContext =>\n                  featureHydrator.hydrateTarget(\n                    hydrationContext,\n                    this.params,\n                    this.mrRequestContextForFeatureStore)\n                }\n\n              override lazy val globalOptoutProbabilities: Seq[Future[Option[Double]]] = {\n                params(PushFeatureSwitchParams.GlobalOptoutModelParam).map { model_id =>\n                  optoutModelScorer\n                    .singlePredictionForTargetLevel(model_id, targetId, featureMap)\n                }\n              }\n\n              override lazy val bucketOptoutProbability: Future[Option[Double]] = {\n                Future\n                  .collect(globalOptoutProbabilities).map {\n                    _.zip(params(PushFeatureSwitchParams.GlobalOptoutThresholdParam))\n                      .exists {\n                        case (Some(score), threshold) => score >= threshold\n                        case _ => false\n                      }\n                  }.flatMap {\n                    case true =>\n                      optoutModelScorer.singlePredictionForTargetLevel(\n                        params(PushFeatureSwitchParams.BucketOptoutModelParam),\n                        targetId,\n                        featureMap)\n                    case _ => Future.None\n                  }\n              }\n\n              override lazy val optoutAdjustedPushcap: Future[Option[Short]] = {\n                if (params(PushFeatureSwitchParams.EnableOptoutAdjustedPushcap)) {\n                  bucketOptoutProbability.map {\n                    case Some(score) =>\n                      val idx = params(PushFeatureSwitchParams.BucketOptoutSlotThresholdParam)\n                        .indexWhere(score <= _)\n                      if (idx >= 0) {\n                        val pushcap =\n                          params(PushFeatureSwitchParams.BucketOptoutSlotPushcapParam)(idx).toShort\n                        optoutModelStat.scope(\"adjusted_pushcap\").counter(f\"$pushcap\").incr()\n                        if (pushcap >= 0) Some(pushcap)\n                        else None\n                      } else None\n                    case _ => None\n                  }\n                } else Future.None\n              }\n\n              override lazy val seedsWithWeight: Future[Option[Map[Long, Double]]] = {\n                Future\n                  .join(\n                    realGraphScoresTop500InStore.get(userId),\n                    targetUserState,\n                    targetUser\n                  )\n                  .flatMap {\n                    case (seedSetOpt, userState, gizmoduckUser) =>\n                      val seedSet = seedSetOpt.getOrElse(Map.empty[Long, Double])\n\n                      //If new sign_up or New user, combine recent_follows with real graph seedset\n                      val isNewUserEnabled = {\n                        val isNewerThan7days = customFSField.daysSinceSignup <= 7\n                        val isNewUserState = userState.contains(UserState.New)\n                        isNewUserState || isNewSignup || isNewerThan7days\n                      }\n\n                      val nonSeedSetFollowsFut = gizmoduckUser match {\n                        case Some(user) if isNewUserEnabled =>\n                          recentFollowscounter.incr()\n                          recentFollowsStore.get(user.id)\n\n                        case Some(user) if this.isModelTrainingData =>\n                          recentFollowscounter.incr()\n                          isModelTrainingDataCounter.incr()\n                          recentFollowsStore.get(user.id)\n\n                        case _ => Future.None\n                      }\n                      nonSeedSetFollowsFut.map { nonSeedSetFollows =>\n                        Some(\n                          SeedsetUtil.combineRecentFollowsWithWeightedSeedset(\n                            seedSet,\n                            nonSeedSetFollows.getOrElse(Nil)\n                          )\n                        )\n                      }\n                  }\n              }\n\n              override def magicFanoutReasonHistory30Days: Future[MagicFanoutReasonHistory] =\n                history.map(history => MagicFanoutReasonHistory(history))\n\n              override val isNewSignup: Boolean =\n                pushContext.flatMap(_.isFromNewUserLoopProcessor).getOrElse(false)\n\n              override lazy val resurrectionDate: Future[Option[String]] =\n                Future.value(customFSField.reactivationDate)\n\n              override lazy val isResurrectedUser: Boolean =\n                customFSField.daysSinceReactivation.isDefined\n\n              override lazy val timeSinceResurrection: Option[Duration] =\n                customFSField.daysSinceReactivation.map(Duration.fromDays)\n\n              override lazy val appPermissions: Future[Option[AppPermission]] =\n                PushAppPermissionUtil.getAppPermission(\n                  userId,\n                  PushAppPermissionUtil.AddressBookPermissionKey,\n                  deviceInfo,\n                  appPermissionStore)\n\n              override lazy val inlineActionHistory: Future[Seq[(Long, String)]] = {\n                inlineActionHistoryStore\n                  .get(userId).map {\n                    case Some(sortedInlineActionHistory) => sortedInlineActionHistory\n                    case _ => Seq.empty\n                  }\n              }\n\n              lazy val isOpenAppExperimentUser: Future[Boolean] =\n                openAppUserStore.get(userId).map(_.contains(true))\n\n              override lazy val openedPushByHourAggregated: Future[Option[Map[Int, Int]]] =\n                openedPushByHourAggregatedStore.get(userId)\n\n              override lazy val places: Future[Seq[Place]] = {\n                geoduckStoreV2\n                  .get(targetId)\n                  .map(_.flatMap(_.places))\n                  .map {\n                    case Some(placeSeq) if placeSeq.nonEmpty =>\n                      placeFoundStat.add(placeSeq.size)\n                      placeSeq\n                    case _ =>\n                      placesNotFound.incr()\n                      Seq.empty\n                  }\n              }\n\n              override val isBlueVerified: Future[Option[Boolean]] =\n                Future.value(userOpt.flatMap(_.safety.flatMap(_.isBlueVerified)))\n\n              override val isVerified: Future[Option[Boolean]] =\n                Future.value(userOpt.flatMap(_.safety.map(_.verified)))\n\n              override lazy val isSuperFollowCreator: Future[Option[Boolean]] =\n                superFollowEligibilityUserStore.get(targetId)\n            }\n          }\n      }\n  }\n\n  /**\n   * Provide general way to add needed FS for target user, and package them in CustomFSFields.\n   * Custom Fields is a powerful feature that allows Feature Switch library users to define and\n   * match against any arbitrary fields.\n   **/\n  private def getCustomFSFields(\n    userId: Long,\n    userOpt: Option[User],\n    deviceInfo: Option[DeviceInfo],\n    userForPushTargetingInfo: Option[UserForPushTargeting],\n    notifHistory: History,\n    requestSource: Option[RequestSource]\n  ): Future[CustomFSFields] = {\n    val reactivationDateFutOpt: Future[Option[String]] = resurrectedUserStore.get(userId)\n    val reactivationTimeFutOpt: Future[Option[Time]] =\n      reactivationDateFutOpt.map(_.map(dateStr => DateUtil.dateStrToTime(dateStr)))\n\n    val isReactivatedUserFut: Future[Boolean] = reactivationTimeFutOpt.map { timeOpt =>\n      timeOpt\n        .exists { time => Time.now - time < 30.days }\n    }\n\n    val daysSinceReactivationFut: Future[Option[Int]] =\n      reactivationTimeFutOpt.map(_.map(time => Time.now.since(time).inDays))\n\n    val daysSinceSignup: Int = (Time.now - SnowflakeUtil.timeFromId(userId)).inDays\n    if (daysSinceSignup < 14) newSignUpUserStats.incr()\n\n    val targetAgeInYears = userOpt.flatMap(_.extendedProfile.flatMap(_.ageInYears))\n\n    val lastLoginFut: Future[Option[Long]] =\n      userHTLLastVisitStore.get(userId).map {\n        case Some(lastHTLVisitTimes) =>\n          val latestHTLVisitTime = lastHTLVisitTimes.max\n          userForPushTargetingInfo.flatMap(\n            _.lastActiveOnAppTimestamp\n              .map(_.max(latestHTLVisitTime)).orElse(Some(latestHTLVisitTime)))\n        case None =>\n          userForPushTargetingInfo.flatMap(_.lastActiveOnAppTimestamp)\n      }\n\n    val daysSinceLoginFut = lastLoginFut.map {\n      _.map { lastLoginTimestamp =>\n        val timeSinceLogin = Time.now - Time.fromMilliseconds(lastLoginTimestamp)\n        if (timeSinceLogin.inDays > 21) {\n          dormantUserCount.incr()\n        }\n        timeSinceLogin.inDays\n      }\n    }\n\n    /* Could add more custom FS here */\n    val userNSFWInfoFut: Future[Option[NsfwInfo]] =\n      nsfwConsumerStore\n        .get(userId).map(_.map(nsfwUserSegmentation => NsfwInfo(nsfwUserSegmentation)))\n\n    val userStateFut: Future[Option[String]] = userFeatureStore.get(userId).map { userFeaturesOpt =>\n      userFeaturesOpt.flatMap { uFeats =>\n        uFeats.userState.map(uState => uState.name)\n      }\n    }\n\n    val mrUserStateFut: Future[Option[String]] =\n      mrUserStateStore.get(userId).map { mrUserStateOpt =>\n        mrUserStateOpt.flatMap { mrUserState =>\n          mrUserState.userState.map(_.name)\n        }\n      }\n\n    Future\n      .join(\n        reactivationDateFutOpt,\n        isReactivatedUserFut,\n        userStateFut,\n        mrUserStateFut,\n        daysSinceLoginFut,\n        daysSinceReactivationFut,\n        userNSFWInfoFut\n      ).map {\n        case (\n              reactivationDate,\n              isReactivatedUser,\n              userState,\n              mrUserState,\n              daysSinceLogin,\n              daysSinceReactivation,\n              userNSFWInfo) =>\n          val numDaysReceivedPushInLast30Days: Int =\n            notifHistory.history.keys.map(_.inDays).toSet.size\n\n          NsfwPersonalizationUtil.computeNsfwUserStats(userNSFWInfo)\n\n          CustomFSFields(\n            isReactivatedUser,\n            daysSinceSignup,\n            numDaysReceivedPushInLast30Days,\n            daysSinceLogin,\n            daysSinceReactivation,\n            userOpt,\n            userState,\n            mrUserState,\n            reactivationDate,\n            requestSource.map(_.name),\n            targetAgeInYears,\n            userNSFWInfo,\n            deviceInfo\n          )\n      }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/target/RFPHTargetPredicateGenerator.scala",
    "content": "package com.twitter.frigate.pushservice.target\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.predicate.TargetPromptFeedbackFatiguePredicate\nimport com.twitter.frigate.common.predicate.TargetUserPredicates\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.predicate.TargetNtabCaretClickFatiguePredicate\nimport com.twitter.frigate.pushservice.predicate.TargetPredicates\nimport com.twitter.hermit.predicate.NamedPredicate\n\nclass RFPHTargetPredicateGenerator(implicit statsReceiver: StatsReceiver) {\n  val predicates: List[NamedPredicate[Target]] = List(\n    TargetPredicates.magicRecsMinDurationSinceSent(),\n    TargetPredicates.targetHTLVisitPredicate(),\n    TargetPredicates.inlineActionFatiguePredicate(),\n    TargetPredicates.targetFatiguePredicate(),\n    TargetUserPredicates.secondaryDormantAccountPredicate(),\n    TargetPredicates.targetValidMobileSDKPredicate,\n    TargetPredicates.targetPushBitEnabledPredicate,\n    TargetUserPredicates.targetUserExists(),\n    TargetPredicates.paramPredicate(PushFeatureSwitchParams.EnablePushRecommendationsParam),\n    TargetPromptFeedbackFatiguePredicate.responseNoPredicate(\n      PushParams.EnablePromptFeedbackFatigueResponseNoPredicate,\n      PushConstants.AcceptableTimeSinceLastNegativeResponse),\n    TargetPredicates.teamExceptedPredicate(TargetNtabCaretClickFatiguePredicate.apply()),\n    TargetPredicates.optoutProbPredicate(),\n    TargetPredicates.webNotifsHoldback()\n  )\n}\n\nobject RFPHTargetPredicates {\n  def apply(implicit statsReceiver: StatsReceiver): List[NamedPredicate[Target]] =\n    new RFPHTargetPredicateGenerator().predicates\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/target/TargetAppPermissions.scala",
    "content": "package com.twitter.frigate.pushservice.target\n\nimport com.twitter.permissions_storage.thriftscala.AppPermission\nimport com.twitter.util.Future\n\ntrait TargetAppPermissions {\n\n  def appPermissions: Future[Option[AppPermission]]\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/target/TargetScoringDetails.scala",
    "content": "package com.twitter.frigate.pushservice.target\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.FeatureMap\nimport com.twitter.frigate.common.base.TargetUser\nimport com.twitter.frigate.common.candidate.TargetABDecider\nimport com.twitter.frigate.common.candidate.TargetDecider\nimport com.twitter.frigate.common.candidate.UserDetails\nimport com.twitter.frigate.data_pipeline.thriftscala.UserHistoryValue\nimport com.twitter.frigate.dau_model.thriftscala.DauProbability\nimport com.twitter.frigate.scribe.thriftscala.SkipModelInfo\nimport com.twitter.hermit.stp.thriftscala.STPResult\nimport com.twitter.timelines.real_graph.v1.thriftscala.RealGraphFeatures\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport com.twitter.frigate.pushservice.params.DeciderKey\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.params.WeightedOpenOrNtabClickModel\nimport com.twitter.frigate.pushservice.util.PushDeviceUtil\nimport com.twitter.nrel.hydration.push.HydrationContext\nimport com.twitter.timelines.configapi.FSParam\n\ntrait TargetScoringDetails {\n  tuc: TargetUser with TargetDecider with TargetABDecider with UserDetails =>\n\n  def stats: StatsReceiver\n\n  /*\n   * We have 3 types of model training data:\n   * 1, skip ranker and model predicates\n   *    controlled by decider frigate_notifier_quality_model_training_data\n   *    the data distribution is same to the distribution in ranking\n   * 2, skip model predicates only\n   *    controlled by decider skip_ml_model_predicate\n   *    the data distribution is same to the distribution in filtering\n   * 3, no skip, only scribe features\n   *    controlled by decider scribe_model_features\n   *    the data distribution is same to production traffic\n   * The \"miscellaneous\" is used to store all misc information for selecting the data offline (e.g., ddg-bucket information)\n   * */\n  lazy val skipModelInfo: Option[SkipModelInfo] = {\n    val trainingDataDeciderKey = DeciderKey.trainingDataDeciderKey.toString\n    val skipMlModelPredicateDeciderKey = DeciderKey.skipMlModelPredicateDeciderKey.toString\n    val scribeModelFeaturesDeciderKey = DeciderKey.scribeModelFeaturesDeciderKey.toString\n    val miscellaneous = None\n\n    if (isDeciderEnabled(trainingDataDeciderKey, stats, useRandomRecipient = true)) {\n      Some(\n        SkipModelInfo(\n          skipPushOpenPredicate = Some(true),\n          skipPushRanker = Some(true),\n          miscellaneous = miscellaneous))\n    } else if (isDeciderEnabled(skipMlModelPredicateDeciderKey, stats, useRandomRecipient = true)) {\n      Some(\n        SkipModelInfo(\n          skipPushOpenPredicate = Some(true),\n          skipPushRanker = Some(false),\n          miscellaneous = miscellaneous))\n    } else if (isDeciderEnabled(scribeModelFeaturesDeciderKey, stats, useRandomRecipient = true)) {\n      Some(SkipModelInfo(noSkipButScribeFeatures = Some(true), miscellaneous = miscellaneous))\n    } else {\n      Some(SkipModelInfo(miscellaneous = miscellaneous))\n    }\n  }\n\n  lazy val scribeFeatureForRequestScribe =\n    isDeciderEnabled(\n      DeciderKey.scribeModelFeaturesForRequestScribe.toString,\n      stats,\n      useRandomRecipient = true)\n\n  lazy val rankingModelParam: Future[FSParam[WeightedOpenOrNtabClickModel.ModelNameType]] =\n    tuc.deviceInfo.map { deviceInfoOpt =>\n      if (PushDeviceUtil.isPrimaryDeviceAndroid(deviceInfoOpt) &&\n        tuc.params(PushParams.AndroidOnlyRankingExperimentParam)) {\n        PushFeatureSwitchParams.WeightedOpenOrNtabClickRankingModelForAndroidParam\n      } else {\n        PushFeatureSwitchParams.WeightedOpenOrNtabClickRankingModelParam\n      }\n    }\n\n  lazy val filteringModelParam: FSParam[WeightedOpenOrNtabClickModel.ModelNameType] =\n    PushFeatureSwitchParams.WeightedOpenOrNtabClickFilteringModelParam\n\n  def skipMlRanker: Boolean = skipModelInfo.exists(_.skipPushRanker.contains(true))\n\n  def skipModelPredicate: Boolean = skipModelInfo.exists(_.skipPushOpenPredicate.contains(true))\n\n  def noSkipButScribeFeatures: Boolean =\n    skipModelInfo.exists(_.noSkipButScribeFeatures.contains(true))\n\n  def isModelTrainingData: Boolean = skipMlRanker || skipModelPredicate || noSkipButScribeFeatures\n\n  def scribeFeatureWithoutHydratingNewFeatures: Boolean =\n    isDeciderEnabled(\n      DeciderKey.scribeModelFeaturesWithoutHydratingNewFeaturesDeciderKey.toString,\n      stats,\n      useRandomRecipient = true\n    )\n\n  def targetHydrationContext: Future[HydrationContext]\n\n  def featureMap: Future[FeatureMap]\n\n  def dauProbability: Future[Option[DauProbability]]\n\n  def labeledPushRecsHydrated: Future[Option[UserHistoryValue]]\n\n  def onlineLabeledPushRecs: Future[Option[UserHistoryValue]]\n\n  def realGraphFeatures: Future[Option[RealGraphFeatures]]\n\n  def stpResult: Future[Option[STPResult]]\n\n  def globalOptoutProbabilities: Seq[Future[Option[Double]]]\n\n  def bucketOptoutProbability: Future[Option[Double]]\n\n  val sendTime: Long = Time.now.inMillis\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/AdaptorUtils.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.FutureOps\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nobject AdaptorUtils {\n  def getTweetyPieResults(\n    tweetIds: Set[Long],\n    tweetyPieStore: ReadableStore[Long, TweetyPieResult],\n  ): Future[Map[Long, Option[TweetyPieResult]]] =\n    FutureOps\n      .mapCollect(tweetyPieStore.multiGet(tweetIds))\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/AdhocStatsUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.CandidateResult\nimport com.twitter.frigate.common.base.Invalid\nimport com.twitter.frigate.common.base.OK\nimport com.twitter.frigate.common.base.Result\nimport com.twitter.frigate.common.base.TweetAuthor\nimport com.twitter.frigate.common.base.TweetCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams.ListOfAdhocIdsForStatsTracking\n\nclass AdhocStatsUtil(stats: StatsReceiver) {\n\n  private def getAdhocIds(candidate: PushCandidate): Set[Long] =\n    candidate.target.params(ListOfAdhocIdsForStatsTracking)\n\n  private def isAdhocTweetCandidate(candidate: PushCandidate): Boolean = {\n    candidate match {\n      case tweetCandidate: RawCandidate with TweetCandidate with TweetAuthor =>\n        tweetCandidate.authorId.exists(id => getAdhocIds(candidate).contains(id))\n      case _ => false\n    }\n  }\n\n  def getCandidateSourceStats(hydratedCandidates: Seq[CandidateDetails[PushCandidate]]): Unit = {\n    hydratedCandidates.foreach { hydratedCandidate =>\n      if (isAdhocTweetCandidate(hydratedCandidate.candidate)) {\n        stats.scope(\"candidate_source\").counter(hydratedCandidate.source).incr()\n      }\n    }\n  }\n\n  def getPreRankingFilterStats(\n    preRankingFilteredCandidates: Seq[CandidateResult[PushCandidate, Result]]\n  ): Unit = {\n    preRankingFilteredCandidates.foreach { filteredCandidate =>\n      if (isAdhocTweetCandidate(filteredCandidate.candidate)) {\n        filteredCandidate.result match {\n          case Invalid(reason) =>\n            stats.scope(\"preranking_filter\").counter(reason.getOrElse(\"unknown_reason\")).incr()\n          case _ =>\n        }\n      }\n    }\n  }\n\n  def getLightRankingStats(lightRankedCandidates: Seq[CandidateDetails[PushCandidate]]): Unit = {\n    lightRankedCandidates.foreach { lightRankedCandidate =>\n      if (isAdhocTweetCandidate(lightRankedCandidate.candidate)) {\n        stats.scope(\"light_ranker\").counter(\"passed_light_ranking\").incr()\n      }\n    }\n  }\n\n  def getRankingStats(rankedCandidates: Seq[CandidateDetails[PushCandidate]]): Unit = {\n    rankedCandidates.zipWithIndex.foreach {\n      case (rankedCandidate, index) =>\n        val rankerStats = stats.scope(\"heavy_ranker\")\n        if (isAdhocTweetCandidate(rankedCandidate.candidate)) {\n          rankerStats.counter(\"ranked_candidates\").incr()\n          rankerStats.stat(\"rank\").add(index.toFloat)\n          rankedCandidate.candidate.modelScores.map { modelScores =>\n            modelScores.foreach {\n              case (modelName, score) =>\n                // mutiply score by 1000 to not lose precision while converting to Float\n                val precisionScore = (score * 100000).toFloat\n                rankerStats.stat(modelName).add(precisionScore)\n            }\n          }\n        }\n    }\n  }\n  def getReRankingStats(rankedCandidates: Seq[CandidateDetails[PushCandidate]]): Unit = {\n    rankedCandidates.zipWithIndex.foreach {\n      case (rankedCandidate, index) =>\n        val rankerStats = stats.scope(\"re_ranking\")\n        if (isAdhocTweetCandidate(rankedCandidate.candidate)) {\n          rankerStats.counter(\"re_ranked_candidates\").incr()\n          rankerStats.stat(\"re_rank\").add(index.toFloat)\n        }\n    }\n  }\n\n  def getTakeCandidateResultStats(\n    allTakeCandidateResults: Seq[CandidateResult[PushCandidate, Result]]\n  ): Unit = {\n    val takeStats = stats.scope(\"take_step\")\n    allTakeCandidateResults.foreach { candidateResult =>\n      if (isAdhocTweetCandidate(candidateResult.candidate)) {\n        candidateResult.result match {\n          case OK =>\n            takeStats.counter(\"sent\").incr()\n          case Invalid(reason) =>\n            takeStats.counter(reason.getOrElse(\"unknown_reason\")).incr()\n          case _ =>\n            takeStats.counter(\"unknown_filter\").incr()\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/Candidate2FrigateNotification.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.thriftscala.FrigateNotification\nimport com.twitter.frigate.thriftscala.NotificationDisplayLocation\n\nobject Candidate2FrigateNotification {\n\n  def getFrigateNotification(\n    candidate: PushCandidate\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): FrigateNotification = {\n    candidate match {\n\n      case topicTweetCandidate: PushCandidate with BaseTopicTweetCandidate =>\n        PushAdaptorUtil.getFrigateNotificationForTweet(\n          crt = topicTweetCandidate.commonRecType,\n          tweetId = topicTweetCandidate.tweetId,\n          scActions = Nil,\n          authorIdOpt = topicTweetCandidate.authorId,\n          pushCopyId = topicTweetCandidate.pushCopyId,\n          ntabCopyId = topicTweetCandidate.ntabCopyId,\n          simclusterId = None,\n          semanticCoreEntityIds = topicTweetCandidate.semanticCoreEntityId.map(List(_)),\n          candidateContent = topicTweetCandidate.content,\n          trendId = None\n        )\n\n      case trendTweetCandidate: PushCandidate with TrendTweetCandidate =>\n        PushAdaptorUtil.getFrigateNotificationForTweet(\n          trendTweetCandidate.commonRecType,\n          trendTweetCandidate.tweetId,\n          Nil,\n          trendTweetCandidate.authorId,\n          trendTweetCandidate.pushCopyId,\n          trendTweetCandidate.ntabCopyId,\n          None,\n          None,\n          trendTweetCandidate.content,\n          Some(trendTweetCandidate.trendId)\n        )\n\n      case tripTweetCandidate: PushCandidate with OutOfNetworkTweetCandidate with TripCandidate =>\n        PushAdaptorUtil.getFrigateNotificationForTweet(\n          crt = tripTweetCandidate.commonRecType,\n          tweetId = tripTweetCandidate.tweetId,\n          scActions = Nil,\n          authorIdOpt = tripTweetCandidate.authorId,\n          pushCopyId = tripTweetCandidate.pushCopyId,\n          ntabCopyId = tripTweetCandidate.ntabCopyId,\n          simclusterId = None,\n          semanticCoreEntityIds = None,\n          candidateContent = tripTweetCandidate.content,\n          trendId = None,\n          tweetTripDomain = tripTweetCandidate.tripDomain\n        )\n\n      case outOfNetworkTweetCandidate: PushCandidate with OutOfNetworkTweetCandidate =>\n        PushAdaptorUtil.getFrigateNotificationForTweet(\n          crt = outOfNetworkTweetCandidate.commonRecType,\n          tweetId = outOfNetworkTweetCandidate.tweetId,\n          scActions = Nil,\n          authorIdOpt = outOfNetworkTweetCandidate.authorId,\n          pushCopyId = outOfNetworkTweetCandidate.pushCopyId,\n          ntabCopyId = outOfNetworkTweetCandidate.ntabCopyId,\n          simclusterId = None,\n          semanticCoreEntityIds = None,\n          candidateContent = outOfNetworkTweetCandidate.content,\n          trendId = None\n        )\n\n      case userCandidate: PushCandidate with UserCandidate with SocialContextActions =>\n        PushAdaptorUtil.getFrigateNotificationForUser(\n          userCandidate.commonRecType,\n          userCandidate.userId,\n          userCandidate.socialContextActions,\n          userCandidate.pushCopyId,\n          userCandidate.ntabCopyId\n        )\n\n      case userCandidate: PushCandidate with UserCandidate =>\n        PushAdaptorUtil.getFrigateNotificationForUser(\n          userCandidate.commonRecType,\n          userCandidate.userId,\n          Nil,\n          userCandidate.pushCopyId,\n          userCandidate.ntabCopyId\n        )\n\n      case tweetCandidate: PushCandidate with TweetCandidate with TweetDetails with SocialContextActions =>\n        PushAdaptorUtil.getFrigateNotificationForTweetWithSocialContextActions(\n          tweetCandidate.commonRecType,\n          tweetCandidate.tweetId,\n          tweetCandidate.socialContextActions,\n          tweetCandidate.authorId,\n          tweetCandidate.pushCopyId,\n          tweetCandidate.ntabCopyId,\n          candidateContent = tweetCandidate.content,\n          semanticCoreEntityIds = None,\n          trendId = None\n        )\n      case pushCandidate: PushCandidate =>\n        FrigateNotification(\n          commonRecommendationType = pushCandidate.commonRecType,\n          notificationDisplayLocation = NotificationDisplayLocation.PushToMobileDevice,\n          pushCopyId = pushCandidate.pushCopyId,\n          ntabCopyId = pushCandidate.ntabCopyId\n        )\n\n      case _ =>\n        statsReceiver\n          .scope(s\"${candidate.commonRecType}\").counter(\"frigate_notification_error\").incr()\n        throw new IllegalStateException(\"Incorrect candidate type when create FrigateNotification\")\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/CandidateHydrationUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.channels.common.thriftscala.ApiList\nimport com.twitter.escherbird.common.thriftscala.Domains\nimport com.twitter.escherbird.metadata.thriftscala.EntityMegadata\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base._\nimport com.twitter.frigate.common.store.interests.InterestsLookupRequestWithContext\nimport com.twitter.frigate.magic_events.thriftscala.FanoutEvent\nimport com.twitter.frigate.magic_events.thriftscala.MagicEventsReason\nimport com.twitter.frigate.magic_events.thriftscala.TargetID\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model._\nimport com.twitter.frigate.pushservice.model.FanoutReasonEntities\nimport com.twitter.frigate.pushservice.ml.PushMLModelScorer\nimport com.twitter.frigate.pushservice.model.candidate.CopyIds\nimport com.twitter.frigate.pushservice.store.EventRequest\nimport com.twitter.frigate.pushservice.store.UttEntityHydrationStore\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.hermit.store.semantic_core.SemanticEntityForQuery\nimport com.twitter.interests.thriftscala.UserInterests\nimport com.twitter.livevideo.timeline.domain.v2.{Event => LiveEvent}\nimport com.twitter.simclusters_v2.thriftscala.SimClustersInferredEntities\nimport com.twitter.storehaus.FutureOps\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.UserId\nimport com.twitter.ubs.thriftscala.AudioSpace\nimport com.twitter.util.Future\n\nobject CandidateHydrationUtil {\n\n  def getAuthorIdFromTweetCandidate(tweetCandidate: TweetCandidate): Option[Long] = {\n    tweetCandidate match {\n      case candidate: TweetCandidate with TweetAuthor =>\n        candidate.authorId\n      case _ => None\n    }\n  }\n\n  private def getCandidateAuthorFromUserMap(\n    tweetCandidate: TweetCandidate,\n    userMap: Map[Long, User]\n  ): Option[User] = {\n    getAuthorIdFromTweetCandidate(tweetCandidate) match {\n      case Some(id) =>\n        userMap.get(id)\n      case _ =>\n        None\n    }\n  }\n\n  private def getRelationshipMapForInNetworkCandidate(\n    candidate: RawCandidate with TweetAuthor,\n    relationshipMap: Map[RelationEdge, Boolean]\n  ): Map[RelationEdge, Boolean] = {\n    val relationEdges =\n      RelationshipUtil.getPreCandidateRelationshipsForInNetworkTweets(candidate).toSet\n    relationEdges.map { relationEdge =>\n      (relationEdge, relationshipMap(relationEdge))\n    }.toMap\n  }\n\n  private def getTweetCandidateSocialContextUsers(\n    candidate: RawCandidate with SocialContextActions,\n    userMap: Map[Long, User]\n  ): Map[Long, Option[User]] = {\n    candidate.socialContextUserIds.map { userId => userId -> userMap.get(userId) }.toMap\n  }\n\n  type TweetWithSocialContextTraits = TweetCandidate with TweetDetails with SocialContextActions\n\n  def getHydratedCandidateForTweetRetweet(\n    candidate: RawCandidate with TweetWithSocialContextTraits,\n    userMap: Map[Long, User],\n    copyIds: CopyIds\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): TweetRetweetPushCandidate = {\n    new TweetRetweetPushCandidate(\n      candidate = candidate,\n      socialContextUserMap = Future.value(getTweetCandidateSocialContextUsers(candidate, userMap)),\n      author = Future.value(getCandidateAuthorFromUserMap(candidate, userMap)),\n      copyIds: CopyIds\n    )\n  }\n\n  def getHydratedCandidateForTweetFavorite(\n    candidate: RawCandidate with TweetWithSocialContextTraits,\n    userMap: Map[Long, User],\n    copyIds: CopyIds\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): TweetFavoritePushCandidate = {\n    new TweetFavoritePushCandidate(\n      candidate = candidate,\n      socialContextUserMap = Future.value(getTweetCandidateSocialContextUsers(candidate, userMap)),\n      author = Future.value(getCandidateAuthorFromUserMap(candidate, userMap)),\n      copyIds = copyIds\n    )\n  }\n\n  def getHydratedCandidateForF1FirstDegreeTweet(\n    candidate: RawCandidate with F1FirstDegree,\n    userMap: Map[Long, User],\n    relationshipMap: Map[RelationEdge, Boolean],\n    copyIds: CopyIds\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): F1TweetPushCandidate = {\n    new F1TweetPushCandidate(\n      candidate = candidate,\n      author = Future.value(getCandidateAuthorFromUserMap(candidate, userMap)),\n      socialGraphServiceResultMap =\n        getRelationshipMapForInNetworkCandidate(candidate, relationshipMap),\n      copyIds = copyIds\n    )\n  }\n  def getHydratedTopicProofTweetCandidate(\n    candidate: RawCandidate with TopicProofTweetCandidate,\n    userMap: Map[Long, User],\n    copyIds: CopyIds\n  )(\n    implicit stats: StatsReceiver,\n    pushMLModelScorer: PushMLModelScorer\n  ): TopicProofTweetPushCandidate =\n    new TopicProofTweetPushCandidate(\n      candidate,\n      getCandidateAuthorFromUserMap(candidate, userMap),\n      copyIds\n    )\n\n  def getHydratedSubscribedSearchTweetCandidate(\n    candidate: RawCandidate with SubscribedSearchTweetCandidate,\n    userMap: Map[Long, User],\n    copyIds: CopyIds\n  )(\n    implicit stats: StatsReceiver,\n    pushMLModelScorer: PushMLModelScorer\n  ): SubscribedSearchTweetPushCandidate =\n    new SubscribedSearchTweetPushCandidate(\n      candidate,\n      getCandidateAuthorFromUserMap(candidate, userMap),\n      copyIds)\n\n  def getHydratedListCandidate(\n    apiListStore: ReadableStore[Long, ApiList],\n    candidate: RawCandidate with ListPushCandidate,\n    copyIds: CopyIds\n  )(\n    implicit stats: StatsReceiver,\n    pushMLModelScorer: PushMLModelScorer\n  ): ListRecommendationPushCandidate = {\n    new ListRecommendationPushCandidate(apiListStore, candidate, copyIds)\n  }\n\n  def getHydratedCandidateForOutOfNetworkTweetCandidate(\n    candidate: RawCandidate with OutOfNetworkTweetCandidate with TopicCandidate,\n    userMap: Map[Long, User],\n    copyIds: CopyIds\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): OutOfNetworkTweetPushCandidate = {\n    new OutOfNetworkTweetPushCandidate(\n      candidate: RawCandidate with OutOfNetworkTweetCandidate with TopicCandidate,\n      author = Future.value(getCandidateAuthorFromUserMap(candidate, userMap)),\n      copyIds: CopyIds\n    )\n  }\n\n  def getHydratedCandidateForTripTweetCandidate(\n    candidate: RawCandidate with OutOfNetworkTweetCandidate with TripCandidate,\n    userMap: Map[Long, User],\n    copyIds: CopyIds\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): TripTweetPushCandidate = {\n    new TripTweetPushCandidate(\n      candidate: RawCandidate with OutOfNetworkTweetCandidate with TripCandidate,\n      author = Future.value(getCandidateAuthorFromUserMap(candidate, userMap)),\n      copyIds: CopyIds\n    )\n  }\n\n  def getHydratedCandidateForDiscoverTwitterCandidate(\n    candidate: RawCandidate with DiscoverTwitterCandidate,\n    copyIds: CopyIds\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): DiscoverTwitterPushCandidate = {\n    new DiscoverTwitterPushCandidate(\n      candidate = candidate,\n      copyIds = copyIds\n    )\n  }\n\n  /**\n   * /*\n   * This method can be reusable for hydrating event candidates\n   **/\n   * @param candidate\n   * @param fanoutMetadataStore\n   * @param semanticCoreMegadataStore\n   * @return (hydratedEvent, hydratedFanoutEvent, hydratedSemanticEntityResults, hydratedSemanticCoreMegadata)\n   */\n  private def hydrateMagicFanoutEventCandidate(\n    candidate: RawCandidate with MagicFanoutEventCandidate,\n    fanoutMetadataStore: ReadableStore[(Long, Long), FanoutEvent],\n    semanticCoreMegadataStore: ReadableStore[SemanticEntityForQuery, EntityMegadata]\n  ): Future[MagicFanoutEventHydratedInfo] = {\n\n    val fanoutEventFut = fanoutMetadataStore.get((candidate.eventId, candidate.pushId))\n\n    val semanticEntityForQueries: Seq[SemanticEntityForQuery] = {\n      val semanticCoreEntityIdQueries = candidate.candidateMagicEventsReasons match {\n        case magicEventsReasons: Seq[MagicEventsReason] =>\n          magicEventsReasons.map(_.reason).collect {\n            case TargetID.SemanticCoreID(scInterest) =>\n              SemanticEntityForQuery(domainId = scInterest.domainId, entityId = scInterest.entityId)\n          }\n        case _ => Seq.empty\n      }\n      val eventEntityQuery = SemanticEntityForQuery(\n        domainId = Domains.EventsEntityService.value,\n        entityId = candidate.eventId)\n      semanticCoreEntityIdQueries :+ eventEntityQuery\n    }\n\n    val semanticEntityResultsFut = FutureOps.mapCollect(\n      semanticCoreMegadataStore.multiGet(semanticEntityForQueries.toSet)\n    )\n\n    Future\n      .join(fanoutEventFut, semanticEntityResultsFut).map {\n        case (fanoutEvent, semanticEntityResults) =>\n          MagicFanoutEventHydratedInfo(\n            fanoutEvent,\n            semanticEntityResults\n          )\n        case _ =>\n          throw new IllegalArgumentException(\n            \"event candidate hydration errors\" + candidate.frigateNotification.toString)\n      }\n  }\n\n  def getHydratedCandidateForMagicFanoutNewsEvent(\n    candidate: RawCandidate with MagicFanoutNewsEventCandidate,\n    copyIds: CopyIds,\n    lexServiceStore: ReadableStore[EventRequest, LiveEvent],\n    fanoutMetadataStore: ReadableStore[(Long, Long), FanoutEvent],\n    semanticCoreMegadataStore: ReadableStore[SemanticEntityForQuery, EntityMegadata],\n    simClusterToEntityStore: ReadableStore[Int, SimClustersInferredEntities],\n    interestsLookupStore: ReadableStore[InterestsLookupRequestWithContext, UserInterests],\n    uttEntityHydrationStore: UttEntityHydrationStore\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): Future[MagicFanoutNewsEventPushCandidate] = {\n    val magicFanoutEventHydratedInfoFut = hydrateMagicFanoutEventCandidate(\n      candidate,\n      fanoutMetadataStore,\n      semanticCoreMegadataStore\n    )\n\n    lazy val simClusterToEntityMappingFut: Future[Map[Int, Option[SimClustersInferredEntities]]] =\n      Future.collect {\n        simClusterToEntityStore.multiGet(\n          FanoutReasonEntities\n            .from(candidate.candidateMagicEventsReasons.map(_.reason)).simclusterIds.map(\n              _.clusterId)\n        )\n      }\n\n    Future\n      .join(\n        magicFanoutEventHydratedInfoFut,\n        simClusterToEntityMappingFut\n      ).map {\n        case (magicFanoutEventHydratedInfo, simClusterToEntityMapping) =>\n          new MagicFanoutNewsEventPushCandidate(\n            candidate = candidate,\n            copyIds = copyIds,\n            fanoutEvent = magicFanoutEventHydratedInfo.fanoutEvent,\n            semanticEntityResults = magicFanoutEventHydratedInfo.semanticEntityResults,\n            simClusterToEntities = simClusterToEntityMapping,\n            lexServiceStore = lexServiceStore,\n            interestsLookupStore = interestsLookupStore,\n            uttEntityHydrationStore = uttEntityHydrationStore\n          )\n      }\n  }\n\n  def getHydratedCandidateForMagicFanoutSportsEvent(\n    candidate: RawCandidate\n      with MagicFanoutSportsEventCandidate\n      with MagicFanoutSportsScoreInformation,\n    copyIds: CopyIds,\n    lexServiceStore: ReadableStore[EventRequest, LiveEvent],\n    fanoutMetadataStore: ReadableStore[(Long, Long), FanoutEvent],\n    semanticCoreMegadataStore: ReadableStore[SemanticEntityForQuery, EntityMegadata],\n    interestsLookupStore: ReadableStore[InterestsLookupRequestWithContext, UserInterests],\n    uttEntityHydrationStore: UttEntityHydrationStore\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): Future[MagicFanoutSportsPushCandidate] = {\n    val magicFanoutEventHydratedInfoFut = hydrateMagicFanoutEventCandidate(\n      candidate,\n      fanoutMetadataStore,\n      semanticCoreMegadataStore\n    )\n\n    magicFanoutEventHydratedInfoFut.map { magicFanoutEventHydratedInfo =>\n      new MagicFanoutSportsPushCandidate(\n        candidate = candidate,\n        copyIds = copyIds,\n        fanoutEvent = magicFanoutEventHydratedInfo.fanoutEvent,\n        semanticEntityResults = magicFanoutEventHydratedInfo.semanticEntityResults,\n        simClusterToEntities = Map.empty,\n        lexServiceStore = lexServiceStore,\n        interestsLookupStore = interestsLookupStore,\n        uttEntityHydrationStore = uttEntityHydrationStore\n      )\n    }\n  }\n\n  def getHydratedCandidateForMagicFanoutProductLaunch(\n    candidate: RawCandidate with MagicFanoutProductLaunchCandidate,\n    copyIds: CopyIds\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): Future[MagicFanoutProductLaunchPushCandidate] =\n    Future.value(new MagicFanoutProductLaunchPushCandidate(candidate, copyIds))\n\n  def getHydratedCandidateForMagicFanoutCreatorEvent(\n    candidate: RawCandidate with MagicFanoutCreatorEventCandidate,\n    safeUserStore: ReadableStore[Long, User],\n    copyIds: CopyIds,\n    creatorTweetCountStore: ReadableStore[UserId, Int]\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): Future[MagicFanoutCreatorEventPushCandidate] = {\n    safeUserStore.get(candidate.creatorId).map { hydratedCreatorUser =>\n      new MagicFanoutCreatorEventPushCandidate(\n        candidate,\n        hydratedCreatorUser,\n        copyIds,\n        creatorTweetCountStore)\n    }\n  }\n\n  def getHydratedCandidateForScheduledSpaceSubscriber(\n    candidate: RawCandidate with ScheduledSpaceSubscriberCandidate,\n    safeUserStore: ReadableStore[Long, User],\n    copyIds: CopyIds,\n    audioSpaceStore: ReadableStore[String, AudioSpace]\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): Future[ScheduledSpaceSubscriberPushCandidate] = {\n\n    candidate.hostId match {\n      case Some(spaceHostId) =>\n        safeUserStore.get(spaceHostId).map { hydratedHost =>\n          new ScheduledSpaceSubscriberPushCandidate(\n            candidate = candidate,\n            hostUser = hydratedHost,\n            copyIds = copyIds,\n            audioSpaceStore = audioSpaceStore\n          )\n        }\n      case _ =>\n        Future.exception(\n          new IllegalStateException(\n            \"Missing Space Host Id for hydrating ScheduledSpaceSubscriberCandidate\"))\n    }\n  }\n\n  def getHydratedCandidateForScheduledSpaceSpeaker(\n    candidate: RawCandidate with ScheduledSpaceSpeakerCandidate,\n    safeUserStore: ReadableStore[Long, User],\n    copyIds: CopyIds,\n    audioSpaceStore: ReadableStore[String, AudioSpace]\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): Future[ScheduledSpaceSpeakerPushCandidate] = {\n\n    candidate.hostId match {\n      case Some(spaceHostId) =>\n        safeUserStore.get(spaceHostId).map { hydratedHost =>\n          new ScheduledSpaceSpeakerPushCandidate(\n            candidate = candidate,\n            hostUser = hydratedHost,\n            copyIds = copyIds,\n            audioSpaceStore = audioSpaceStore\n          )\n        }\n      case _ =>\n        Future.exception(\n          new RuntimeException(\n            \"Missing Space Host Id for hydrating ScheduledSpaceSpeakerCandidate\"))\n    }\n  }\n\n  def getHydratedCandidateForTopTweetImpressionsCandidate(\n    candidate: RawCandidate with TopTweetImpressionsCandidate,\n    copyIds: CopyIds\n  )(\n    implicit stats: StatsReceiver,\n    pushModelScorer: PushMLModelScorer\n  ): TopTweetImpressionsPushCandidate = {\n    new TopTweetImpressionsPushCandidate(\n      candidate = candidate,\n      copyIds = copyIds\n    )\n  }\n\n  def isNsfwAccount(user: User, nsfwTokens: Seq[String]): Boolean = {\n    def hasNsfwToken(str: String): Boolean = nsfwTokens.exists(str.toLowerCase().contains(_))\n\n    val name = user.profile.map(_.name).getOrElse(\"\")\n    val screenName = user.profile.map(_.screenName).getOrElse(\"\")\n    val location = user.profile.map(_.location).getOrElse(\"\")\n    val description = user.profile.map(_.description).getOrElse(\"\")\n    val hasNsfwFlag =\n      user.safety.map(safety => safety.nsfwUser || safety.nsfwAdmin).getOrElse(false)\n    hasNsfwToken(name) || hasNsfwToken(screenName) || hasNsfwToken(location) || hasNsfwToken(\n      description) || hasNsfwFlag\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/CandidateUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.contentrecommender.thriftscala.MetricTag\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.OutOfNetworkTweetCandidate\nimport com.twitter.frigate.common.base.SocialContextAction\nimport com.twitter.frigate.common.base.SocialContextActions\nimport com.twitter.frigate.common.base.TargetInfo\nimport com.twitter.frigate.common.base.TargetUser\nimport com.twitter.frigate.common.base.TopicProofTweetCandidate\nimport com.twitter.frigate.common.base.TweetAuthorDetails\nimport com.twitter.frigate.common.candidate.TargetABDecider\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.params.CrtGroupEnum\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.CommonRecommendationType.TripGeoTweet\nimport com.twitter.frigate.thriftscala.CommonRecommendationType.TripHqTweet\nimport com.twitter.frigate.thriftscala.{SocialContextAction => TSocialContextAction}\nimport com.twitter.util.Future\n\nobject CandidateUtil {\n  private val mrTwistlyMetricTags =\n    Seq(MetricTag.PushOpenOrNtabClick, MetricTag.RequestHealthFilterPushOpenBasedTweetEmbedding)\n\n  def getSocialContextActionsFromCandidate(candidate: RawCandidate): Seq[TSocialContextAction] = {\n    candidate match {\n      case candidateWithSocialContex: RawCandidate with SocialContextActions =>\n        candidateWithSocialContex.socialContextActions.map { scAction =>\n          TSocialContextAction(\n            scAction.userId,\n            scAction.timestampInMillis,\n            scAction.tweetId\n          )\n        }\n      case _ => Seq.empty\n    }\n  }\n\n  /**\n   * Ranking Social Context based on the Real Graph weight\n   * @param socialContextActions  Sequence of Social Context Actions\n   * @param seedsWithWeight       Real Graph map consisting of User ID as key and RG weight as the value\n   * @param defaultToRecency      Boolean to represent if we should use the timestamp of the SC to rank\n   * @return                      Returns the ranked sequence of SC Actions\n   */\n  def getRankedSocialContext(\n    socialContextActions: Seq[SocialContextAction],\n    seedsWithWeight: Future[Option[Map[Long, Double]]],\n    defaultToRecency: Boolean\n  ): Future[Seq[SocialContextAction]] = {\n    seedsWithWeight.map {\n      case Some(followingsMap) =>\n        socialContextActions.sortBy { action => -followingsMap.getOrElse(action.userId, 0.0) }\n      case _ =>\n        if (defaultToRecency) socialContextActions.sortBy(-_.timestampInMillis)\n        else socialContextActions\n    }\n  }\n\n  def shouldApplyHealthQualityFiltersForPrerankingPredicates(\n    candidate: TweetAuthorDetails with TargetInfo[TargetUser with TargetABDecider]\n  )(\n    implicit stats: StatsReceiver\n  ): Future[Boolean] = {\n    candidate.tweetAuthor.map {\n      case Some(user) =>\n        val numFollowers: Double = user.counts.map(_.followers.toDouble).getOrElse(0.0)\n        numFollowers < candidate.target\n          .params(PushFeatureSwitchParams.NumFollowerThresholdForHealthAndQualityFiltersPreranking)\n      case _ => true\n    }\n  }\n\n  def shouldApplyHealthQualityFilters(\n    candidate: PushCandidate\n  )(\n    implicit stats: StatsReceiver\n  ): Boolean = {\n    val numFollowers =\n      candidate.numericFeatures.getOrElse(\"RecTweetAuthor.User.ActiveFollowers\", 0.0)\n    numFollowers < candidate.target\n      .params(PushFeatureSwitchParams.NumFollowerThresholdForHealthAndQualityFilters)\n  }\n\n  def useAggressiveHealthThresholds(cand: PushCandidate): Boolean =\n    isMrTwistlyCandidate(cand) ||\n      (cand.commonRecType == CommonRecommendationType.GeoPopTweet && cand.target.params(\n        PushFeatureSwitchParams.PopGeoTweetEnableAggressiveThresholds))\n\n  def isMrTwistlyCandidate(cand: PushCandidate): Boolean =\n    cand match {\n      case oonCandidate: PushCandidate with OutOfNetworkTweetCandidate =>\n        oonCandidate.tagsCR\n          .getOrElse(Seq.empty).intersect(mrTwistlyMetricTags).nonEmpty && oonCandidate.tagsCR\n          .map(_.toSet.size).getOrElse(0) == 1\n      case oonCandidate: PushCandidate with TopicProofTweetCandidate\n          if cand.target.params(PushFeatureSwitchParams.EnableHealthFiltersForTopicProofTweet) =>\n        oonCandidate.tagsCR\n          .getOrElse(Seq.empty).intersect(mrTwistlyMetricTags).nonEmpty && oonCandidate.tagsCR\n          .map(_.toSet.size).getOrElse(0) == 1\n      case _ => false\n    }\n\n  def getTagsCRCount(cand: PushCandidate): Double =\n    cand match {\n      case oonCandidate: PushCandidate with OutOfNetworkTweetCandidate =>\n        oonCandidate.tagsCR.map(_.toSet.size).getOrElse(0).toDouble\n      case oonCandidate: PushCandidate with TopicProofTweetCandidate\n          if cand.target.params(PushFeatureSwitchParams.EnableHealthFiltersForTopicProofTweet) =>\n        oonCandidate.tagsCR.map(_.toSet.size).getOrElse(0).toDouble\n      case _ => 0.0\n    }\n\n  def isRelatedToMrTwistlyCandidate(cand: PushCandidate): Boolean =\n    cand match {\n      case oonCandidate: PushCandidate with OutOfNetworkTweetCandidate =>\n        oonCandidate.tagsCR.getOrElse(Seq.empty).intersect(mrTwistlyMetricTags).nonEmpty\n      case oonCandidate: PushCandidate with TopicProofTweetCandidate\n          if cand.target.params(PushFeatureSwitchParams.EnableHealthFiltersForTopicProofTweet) =>\n        oonCandidate.tagsCR.getOrElse(Seq.empty).intersect(mrTwistlyMetricTags).nonEmpty\n      case _ => false\n    }\n\n  def getCrtGroup(commonRecType: CommonRecommendationType): CrtGroupEnum.Value = {\n    commonRecType match {\n      case crt if RecTypes.twistlyTweets(crt) => CrtGroupEnum.Twistly\n      case crt if RecTypes.frsTypes(crt) => CrtGroupEnum.Frs\n      case crt if RecTypes.f1RecTypes(crt) => CrtGroupEnum.F1\n      case crt if crt == TripGeoTweet || crt == TripHqTweet => CrtGroupEnum.Trip\n      case crt if RecTypes.TopicTweetTypes(crt) => CrtGroupEnum.Topic\n      case crt if RecTypes.isGeoPopTweetType(crt) => CrtGroupEnum.GeoPop\n      case _ => CrtGroupEnum.Other\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/CopyUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FS}\nimport com.twitter.ibis2.lib.util.JsonMarshal\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\nobject CopyUtil {\n\n  /**\n   * Get a list of history feature copy alone with metadata in the look back period, the metadata\n   * can be used to calculate number of copy pushed after the current feature copy\n   * @param candidate the candidate to be pushed to the user\n   * @return Future[Seq((..,))], which is a seq of the history FEATURE copy along with\n   *         metadata within the look back period. In the tuple, the 4 elements represents:\n   *         1. Timestamp of the past feature copy\n   *         2. Option[Seq()] of copy feature names of the past copy\n   *         3. Index of the particular feature copy in look back history if normal copy presents\n   */\n  private def getPastCopyFeaturesList(\n    candidate: PushCandidate\n  ): Future[Seq[(Time, Option[Seq[String]], Int)]] = {\n    val target = candidate.target\n\n    target.history.map { targetHistory =>\n      val historyLookbackDuration = target.params(FS.CopyFeaturesHistoryLookbackDuration)\n      val notificationHistoryInLookbackDuration = targetHistory.sortedHistory\n        .takeWhile {\n          case (notifTimestamp, _) => historyLookbackDuration.ago < notifTimestamp\n        }\n      notificationHistoryInLookbackDuration.zipWithIndex\n        .filter {\n          case ((_, notification), _) =>\n            notification.copyFeatures match {\n              case Some(copyFeatures) => copyFeatures.nonEmpty\n              case _ => false\n            }\n        }\n        .collect {\n          case ((timestamp, notification), notificationIndex) =>\n            (timestamp, notification.copyFeatures, notificationIndex)\n        }\n    }\n  }\n\n  private def getPastCopyFeaturesListForF1(\n    candidate: PushCandidate\n  ): Future[Seq[(Time, Option[Seq[String]], Int)]] = {\n    val target = candidate.target\n    target.history.map { targetHistory =>\n      val historyLookbackDuration = target.params(FS.CopyFeaturesHistoryLookbackDuration)\n      val notificationHistoryInLookbackDuration = targetHistory.sortedHistory\n        .takeWhile {\n          case (notifTimestamp, _) => historyLookbackDuration.ago < notifTimestamp\n        }\n      notificationHistoryInLookbackDuration.zipWithIndex\n        .filter {\n          case ((_, notification), _) =>\n            notification.copyFeatures match {\n              case Some(copyFeatures) =>\n                RecTypes.isF1Type(notification.commonRecommendationType) && copyFeatures.nonEmpty\n              case _ => false\n            }\n        }\n        .collect {\n          case ((timestamp, notification), notificationIndex) =>\n            (timestamp, notification.copyFeatures, notificationIndex)\n        }\n    }\n  }\n\n  private def getPastCopyFeaturesListForOON(\n    candidate: PushCandidate\n  ): Future[Seq[(Time, Option[Seq[String]], Int)]] = {\n    val target = candidate.target\n    target.history.map { targetHistory =>\n      val historyLookbackDuration = target.params(FS.CopyFeaturesHistoryLookbackDuration)\n      val notificationHistoryInLookbackDuration = targetHistory.sortedHistory\n        .takeWhile {\n          case (notifTimestamp, _) => historyLookbackDuration.ago < notifTimestamp\n        }\n      notificationHistoryInLookbackDuration.zipWithIndex\n        .filter {\n          case ((_, notification), _) =>\n            notification.copyFeatures match {\n              case Some(copyFeatures) =>\n                !RecTypes.isF1Type(notification.commonRecommendationType) && copyFeatures.nonEmpty\n\n              case _ => false\n            }\n        }\n        .collect {\n          case ((timestamp, notification), notificationIndex) =>\n            (timestamp, notification.copyFeatures, notificationIndex)\n        }\n    }\n  }\n  private def getEmojiFeaturesMap(\n    candidate: PushCandidate,\n    copyFeatureHistory: Seq[(Time, Option[Seq[String]], Int)],\n    lastHTLVisitTimestamp: Option[Long],\n    stats: StatsReceiver\n  ): Map[String, String] = {\n    val (emojiFatigueDuration, emojiFatigueNumOfPushes) = {\n      if (RecTypes.isF1Type(candidate.commonRecType)) {\n        (\n          candidate.target.params(FS.F1EmojiCopyFatigueDuration),\n          candidate.target.params(FS.F1EmojiCopyNumOfPushesFatigue))\n      } else {\n        (\n          candidate.target.params(FS.OonEmojiCopyFatigueDuration),\n          candidate.target.params(FS.OonEmojiCopyNumOfPushesFatigue))\n      }\n    }\n\n    val scopedStats = stats\n      .scope(\"getEmojiFeaturesMap\").scope(candidate.commonRecType.toString).scope(\n        emojiFatigueDuration.toString)\n    val addedEmojiCopyFeature = scopedStats.counter(\"added_emoji\")\n    val fatiguedEmojiCopyFeature = scopedStats.counter(\"no_emoji\")\n\n    val copyFeatureType = PushConstants.EmojiFeatureNameForIbis2ModelValues\n\n    val durationFatigueCarryFunc = () =>\n      isUnderDurationFatigue(copyFeatureHistory, copyFeatureType, emojiFatigueDuration)\n\n    val enableHTLBasedFatigueBasicRule = candidate.target.params(FS.EnableHTLBasedFatigueBasicRule)\n    val minDuration = candidate.target.params(FS.MinFatigueDurationSinceLastHTLVisit)\n    val lastHTLVisitBasedNonFatigueWindow =\n      candidate.target.params(FS.LastHTLVisitBasedNonFatigueWindow)\n    val htlBasedCopyFatigueCarryFunc = () =>\n      isUnderHTLBasedFatigue(lastHTLVisitTimestamp, minDuration, lastHTLVisitBasedNonFatigueWindow)\n\n    val isUnderFatigue = getIsUnderFatigue(\n      Seq(\n        (durationFatigueCarryFunc, true),\n        (htlBasedCopyFatigueCarryFunc, enableHTLBasedFatigueBasicRule),\n      ),\n      scopedStats\n    )\n\n    if (!isUnderFatigue) {\n      addedEmojiCopyFeature.incr()\n      Map(PushConstants.EmojiFeatureNameForIbis2ModelValues -> \"true\")\n    } else {\n      fatiguedEmojiCopyFeature.incr()\n      Map.empty[String, String]\n    }\n  }\n\n  private def getTargetFeaturesMap(\n    candidate: PushCandidate,\n    copyFeatureHistory: Seq[(Time, Option[Seq[String]], Int)],\n    lastHTLVisitTimestamp: Option[Long],\n    stats: StatsReceiver\n  ): Map[String, String] = {\n    val targetFatigueDuration = {\n      if (RecTypes.isF1Type(candidate.commonRecType)) {\n        candidate.target.params(FS.F1TargetCopyFatigueDuration)\n      } else {\n        candidate.target.params(FS.OonTargetCopyFatigueDuration)\n      }\n    }\n\n    val scopedStats = stats\n      .scope(\"getTargetFeaturesMap\").scope(candidate.commonRecType.toString).scope(\n        targetFatigueDuration.toString)\n    val addedTargetCopyFeature = scopedStats.counter(\"added_target\")\n    val fatiguedTargetCopyFeature = scopedStats.counter(\"no_target\")\n\n    val featureCopyType = PushConstants.TargetFeatureNameForIbis2ModelValues\n    val durationFatigueCarryFunc = () =>\n      isUnderDurationFatigue(copyFeatureHistory, featureCopyType, targetFatigueDuration)\n\n    val enableHTLBasedFatigueBasicRule = candidate.target.params(FS.EnableHTLBasedFatigueBasicRule)\n    val minDuration = candidate.target.params(FS.MinFatigueDurationSinceLastHTLVisit)\n    val lastHTLVisitBasedNonFatigueWindow =\n      candidate.target.params(FS.LastHTLVisitBasedNonFatigueWindow)\n    val htlBasedCopyFatigueCarryFunc = () =>\n      isUnderHTLBasedFatigue(lastHTLVisitTimestamp, minDuration, lastHTLVisitBasedNonFatigueWindow)\n\n    val isUnderFatigue = getIsUnderFatigue(\n      Seq(\n        (durationFatigueCarryFunc, true),\n        (htlBasedCopyFatigueCarryFunc, enableHTLBasedFatigueBasicRule),\n      ),\n      scopedStats\n    )\n\n    if (!isUnderFatigue) {\n      addedTargetCopyFeature.incr()\n      Map(PushConstants.TargetFeatureNameForIbis2ModelValues -> \"true\")\n    } else {\n\n      fatiguedTargetCopyFeature.incr()\n      Map.empty[String, String]\n    }\n  }\n\n  type FatigueRuleFlag = Boolean\n  type FatigueRuleFunc = () => Boolean\n\n  def getIsUnderFatigue(\n    fatigueRulesWithFlags: Seq[(FatigueRuleFunc, FatigueRuleFlag)],\n    statsReceiver: StatsReceiver,\n  ): Boolean = {\n    val defaultFatigue = true\n    val finalFatigueRes =\n      fatigueRulesWithFlags.zipWithIndex.foldLeft(defaultFatigue)(\n        (fatigueSoFar, fatigueRuleFuncWithFlagAndIndex) => {\n          val ((fatigueRuleFunc, flag), index) = fatigueRuleFuncWithFlagAndIndex\n          val funcScopedStats = statsReceiver.scope(s\"fatigueFunction${index}\")\n          if (flag) {\n            val shouldFatigueForTheRule = fatigueRuleFunc()\n            funcScopedStats.scope(s\"eval_${shouldFatigueForTheRule}\").counter().incr()\n            val f = fatigueSoFar && shouldFatigueForTheRule\n            f\n          } else {\n            fatigueSoFar\n          }\n        })\n    statsReceiver.scope(s\"final_fatigue_${finalFatigueRes}\").counter().incr()\n    finalFatigueRes\n  }\n\n  private def isUnderDurationFatigue(\n    copyFeatureHistory: Seq[(Time, Option[Seq[String]], Int)],\n    copyFeatureType: String,\n    fatigueDuration: com.twitter.util.Duration,\n  ): Boolean = {\n    copyFeatureHistory.exists {\n      case (notifTimestamp, Some(copyFeatures), _) if copyFeatures.contains(copyFeatureType) =>\n        notifTimestamp > fatigueDuration.ago\n      case _ => false\n    }\n  }\n\n  private def isUnderHTLBasedFatigue(\n    lastHTLVisitTimestamp: Option[Long],\n    minDurationSinceLastHTLVisit: com.twitter.util.Duration,\n    lastHTLVisitBasedNonFatigueWindow: com.twitter.util.Duration,\n  ): Boolean = {\n    val lastHTLVisit = lastHTLVisitTimestamp.map(t => Time.fromMilliseconds(t)).getOrElse(Time.now)\n    val first = Time.now < (lastHTLVisit + minDurationSinceLastHTLVisit)\n    val second =\n      Time.now > (lastHTLVisit + minDurationSinceLastHTLVisit + lastHTLVisitBasedNonFatigueWindow)\n    first || second\n  }\n\n  def getOONCBasedFeature(\n    candidate: PushCandidate,\n    stats: StatsReceiver\n  ): Future[Map[String, String]] = {\n    val target = candidate.target\n    val metric = stats.scope(\"getOONCBasedFeature\")\n    if (target.params(FS.EnableOONCBasedCopy)) {\n      candidate.mrWeightedOpenOrNtabClickRankingProbability.map {\n        case Some(score) if score >= target.params(FS.HighOONCThresholdForCopy) =>\n          metric.counter(\"high_OONC\").incr()\n          metric.counter(FS.HighOONCTweetFormat.toString).incr()\n          Map(\n            \"whole_template\" -> JsonMarshal.toJson(\n              Map(\n                target.params(FS.HighOONCTweetFormat).toString -> true\n              )))\n        case Some(score) if score <= target.params(FS.LowOONCThresholdForCopy) =>\n          metric.counter(\"low_OONC\").incr()\n          metric.counter(FS.LowOONCThresholdForCopy.toString).incr()\n          Map(\n            \"whole_template\" -> JsonMarshal.toJson(\n              Map(\n                target.params(FS.LowOONCTweetFormat).toString -> true\n              )))\n        case _ =>\n          metric.counter(\"not_in_OONC_range\").incr()\n          Map.empty[String, String]\n      }\n    } else {\n      Future.value(Map.empty[String, String])\n    }\n  }\n\n  def getCopyFeatures(\n    candidate: PushCandidate,\n    stats: StatsReceiver,\n  ): Future[Map[String, String]] = {\n    if (candidate.target.isLoggedOutUser) {\n      Future.value(Map.empty[String, String])\n    } else {\n      val featureMaps = getCopyBodyFeatures(candidate, stats)\n      for {\n        titleFeat <- getCopyTitleFeatures(candidate, stats)\n        nsfwFeat <- getNsfwCopyFeatures(candidate, stats)\n        ooncBasedFeature <- getOONCBasedFeature(candidate, stats)\n      } yield {\n        titleFeat ++ featureMaps ++ nsfwFeat ++ ooncBasedFeature\n      }\n    }\n  }\n\n  private def getCopyTitleFeatures(\n    candidate: PushCandidate,\n    stats: StatsReceiver\n  ): Future[Map[String, String]] = {\n    val scopedStats = stats.scope(\"CopyUtil\").scope(\"getCopyTitleFeatures\")\n\n    val target = candidate.target\n\n    if ((RecTypes.isSimClusterBasedType(candidate.commonRecType) && target.params(\n        FS.EnableCopyFeaturesForOon)) || (RecTypes.isF1Type(candidate.commonRecType) && target\n        .params(FS.EnableCopyFeaturesForF1))) {\n\n      val enableTargetAndEmojiSplitFatigue = target.params(FS.EnableTargetAndEmojiSplitFatigue)\n      val isTargetF1Type = RecTypes.isF1Type(candidate.commonRecType)\n\n      val copyFeatureHistoryFuture = if (enableTargetAndEmojiSplitFatigue && isTargetF1Type) {\n        getPastCopyFeaturesListForF1(candidate)\n      } else if (enableTargetAndEmojiSplitFatigue && !isTargetF1Type) {\n        getPastCopyFeaturesListForOON(candidate)\n      } else {\n        getPastCopyFeaturesList(candidate)\n      }\n\n      Future\n        .join(\n          copyFeatureHistoryFuture,\n          target.lastHTLVisitTimestamp,\n        ).map {\n          case (copyFeatureHistory, lastHTLVisitTimestamp) =>\n            val emojiFeatures = {\n              if ((RecTypes.isF1Type(candidate.commonRecType) && target.params(\n                  FS.EnableEmojiInF1Copy))\n                || RecTypes.isSimClusterBasedType(candidate.commonRecType) && target.params(\n                  FS.EnableEmojiInOonCopy)) {\n                getEmojiFeaturesMap(\n                  candidate,\n                  copyFeatureHistory,\n                  lastHTLVisitTimestamp,\n                  scopedStats)\n              } else Map.empty[String, String]\n            }\n\n            val targetFeatures = {\n              if ((RecTypes.isF1Type(candidate.commonRecType) && target.params(\n                  FS.EnableTargetInF1Copy)) || (RecTypes.isSimClusterBasedType(\n                  candidate.commonRecType) && target.params(FS.EnableTargetInOonCopy))) {\n                getTargetFeaturesMap(\n                  candidate,\n                  copyFeatureHistory,\n                  lastHTLVisitTimestamp,\n                  scopedStats)\n              } else Map.empty[String, String]\n            }\n\n            val baseCopyFeaturesMap =\n              if (emojiFeatures.nonEmpty || targetFeatures.nonEmpty)\n                Map(PushConstants.EnableCopyFeaturesForIbis2ModelValues -> \"true\")\n              else Map.empty[String, String]\n            baseCopyFeaturesMap ++ emojiFeatures ++ targetFeatures\n          case _ =>\n            Map.empty[String, String]\n        }\n    } else Future.value(Map.empty[String, String])\n  }\n\n  private def getCopyBodyTruncateFeatures(\n    candidate: PushCandidate,\n  ): Map[String, String] = {\n    if (candidate.target.params(FS.EnableIosCopyBodyTruncate)) {\n      Map(\"enable_body_truncate_ios\" -> \"true\")\n    } else {\n      Map.empty[String, String]\n    }\n  }\n\n  private def getNsfwCopyFeatures(\n    candidate: PushCandidate,\n    stats: StatsReceiver\n  ): Future[Map[String, String]] = {\n    val scopedStats = stats.scope(\"CopyUtil\").scope(\"getNsfwCopyBodyFeatures\")\n    val hasNsfwScoreF1Counter = scopedStats.counter(\"f1_has_nsfw_score\")\n    val hasNsfwScoreOonCounter = scopedStats.counter(\"oon_has_nsfw_score\")\n    val noNsfwScoreCounter = scopedStats.counter(\"no_nsfw_score\")\n    val nsfwScoreF1 = scopedStats.stat(\"f1_nsfw_score\")\n    val nsfwScoreOon = scopedStats.stat(\"oon_nsfw_score\")\n    val isNsfwF1Counter = scopedStats.counter(\"is_f1_nsfw\")\n    val isNsfwOonCounter = scopedStats.counter(\"is_oon_nsfw\")\n\n    val target = candidate.target\n    val nsfwScoreFut = if (target.params(FS.EnableNsfwCopy)) {\n      candidate.mrNsfwScore\n    } else Future.None\n\n    nsfwScoreFut.map {\n      case Some(nsfwScore) =>\n        if (RecTypes.isF1Type(candidate.commonRecType)) {\n          hasNsfwScoreF1Counter.incr()\n          nsfwScoreF1.add(nsfwScore.toFloat * 10000)\n          if (nsfwScore > target.params(FS.NsfwScoreThresholdForF1Copy)) {\n            isNsfwF1Counter.incr()\n            Map(\"is_f1_nsfw\" -> \"true\")\n          } else {\n            Map.empty[String, String]\n          }\n        } else if (RecTypes.isOutOfNetworkTweetRecType(candidate.commonRecType)) {\n          nsfwScoreOon.add(nsfwScore.toFloat * 10000)\n          hasNsfwScoreOonCounter.incr()\n          if (nsfwScore > target.params(FS.NsfwScoreThresholdForOONCopy)) {\n            isNsfwOonCounter.incr()\n            Map(\"is_oon_nsfw\" -> \"true\")\n          } else {\n            Map.empty[String, String]\n          }\n        } else {\n          Map.empty[String, String]\n        }\n      case _ =>\n        noNsfwScoreCounter.incr()\n        Map.empty[String, String]\n    }\n  }\n\n  private def getCopyBodyFeatures(\n    candidate: PushCandidate,\n    stats: StatsReceiver\n  ): Map[String, String] = {\n    val target = candidate.target\n    val scopedStats = stats.scope(\"CopyUtil\").scope(\"getCopyBodyFeatures\")\n\n    val copyBodyFeatures = {\n      if (RecTypes.isF1Type(candidate.commonRecType) && target.params(FS.EnableF1CopyBody)) {\n        scopedStats.counter(\"f1BodyExpEnabled\").incr()\n        Map(PushConstants.CopyBodyExpIbisModelValues -> \"true\")\n      } else if (RecTypes.isOutOfNetworkTweetRecType(candidate.commonRecType) && target.params(\n          FS.EnableOONCopyBody)) {\n        scopedStats.counter(\"oonBodyExpEnabled\").incr()\n        Map(PushConstants.CopyBodyExpIbisModelValues -> \"true\")\n      } else\n        Map.empty[String, String]\n    }\n    val copyBodyTruncateFeatures = getCopyBodyTruncateFeatures(candidate)\n    copyBodyFeatures ++ copyBodyTruncateFeatures\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/EmailLandingPageExperimentUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.frigate.common.store.deviceinfo.DeviceInfo\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams.EnableRuxLandingPage\nimport com.twitter.frigate.pushservice.params.PushParams.EnableRuxLandingPageAndroidParam\nimport com.twitter.frigate.pushservice.params.PushParams.EnableRuxLandingPageIOSParam\nimport com.twitter.frigate.pushservice.params.PushParams.RuxLandingPageExperimentKeyAndroidParam\nimport com.twitter.frigate.pushservice.params.PushParams.RuxLandingPageExperimentKeyIOSParam\nimport com.twitter.frigate.pushservice.params.PushParams.ShowRuxLandingPageAsModalOnIOS\nimport com.twitter.rux.common.context.thriftscala.MagicRecsNTabTweet\nimport com.twitter.rux.common.context.thriftscala.MagicRecsPushTweet\nimport com.twitter.rux.common.context.thriftscala.RuxContext\nimport com.twitter.rux.common.context.thriftscala.Source\nimport com.twitter.rux.common.encode.RuxContextEncoder\n\n/**\n * This class provides utility functions for email landing page for push\n */\nobject EmailLandingPageExperimentUtil {\n  val ruxCxtEncoder = new RuxContextEncoder()\n\n  def getIbis2ModelValue(\n    deviceInfoOpt: Option[DeviceInfo],\n    target: Target,\n    tweetId: Long\n  ): Map[String, String] = {\n    val enable = enablePushEmailLanding(deviceInfoOpt, target)\n    if (enable) {\n      val ruxCxt = if (deviceInfoOpt.exists(_.isRuxLandingPageEligible)) {\n        val encodedCxt = getRuxContext(tweetId, target, deviceInfoOpt)\n        Map(\"rux_cxt\" -> encodedCxt)\n      } else Map.empty[String, String]\n      val enableModal = if (showModalForIOS(deviceInfoOpt, target)) {\n        Map(\"enable_modal\" -> \"true\")\n      } else Map.empty[String, String]\n\n      Map(\"land_on_email_landing_page\" -> \"true\") ++ ruxCxt ++ enableModal\n    } else Map.empty[String, String]\n  }\n\n  def createNTabRuxLandingURI(screenName: String, tweetId: Long): String = {\n    val encodedCxt =\n      ruxCxtEncoder.encode(RuxContext(Some(Source.MagicRecsNTabTweet(MagicRecsNTabTweet(tweetId)))))\n    s\"$screenName/status/${tweetId.toString}?cxt=$encodedCxt\"\n  }\n\n  private def getRuxContext(\n    tweetId: Long,\n    target: Target,\n    deviceInfoOpt: Option[DeviceInfo]\n  ): String = {\n    val isDeviceIOS = PushDeviceUtil.isPrimaryDeviceIOS(deviceInfoOpt)\n    val isDeviceAndroid = PushDeviceUtil.isPrimaryDeviceAndroid(deviceInfoOpt)\n    val keyOpt = if (isDeviceIOS) {\n      target.params(RuxLandingPageExperimentKeyIOSParam)\n    } else if (isDeviceAndroid) {\n      target.params(RuxLandingPageExperimentKeyAndroidParam)\n    } else None\n    val context = RuxContext(Some(Source.MagicRecsTweet(MagicRecsPushTweet(tweetId))), None, keyOpt)\n    ruxCxtEncoder.encode(context)\n  }\n\n  private def enablePushEmailLanding(\n    deviceInfoOpt: Option[DeviceInfo],\n    target: Target\n  ): Boolean =\n    deviceInfoOpt.exists(deviceInfo =>\n      if (deviceInfo.isEmailLandingPageEligible) {\n        val isRuxLandingPageEnabled = target.params(EnableRuxLandingPage)\n        isRuxLandingPageEnabled && isRuxLandingEnabledBasedOnDeviceInfo(deviceInfoOpt, target)\n      } else false)\n\n  private def showModalForIOS(deviceInfoOpt: Option[DeviceInfo], target: Target): Boolean = {\n    deviceInfoOpt.exists { deviceInfo =>\n      deviceInfo.isRuxLandingPageAsModalEligible && target.params(ShowRuxLandingPageAsModalOnIOS)\n    }\n  }\n\n  private def isRuxLandingEnabledBasedOnDeviceInfo(\n    deviceInfoOpt: Option[DeviceInfo],\n    target: Target\n  ): Boolean = {\n    val isDeviceIOS = PushDeviceUtil.isPrimaryDeviceIOS(deviceInfoOpt)\n    val isDeviceAndroid = PushDeviceUtil.isPrimaryDeviceAndroid(deviceInfoOpt)\n    if (isDeviceIOS) {\n      target.params(EnableRuxLandingPageIOSParam)\n    } else if (isDeviceAndroid) {\n      target.params(EnableRuxLandingPageAndroidParam)\n    } else true\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/FunctionalUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.Counter\n\nobject FunctionalUtil {\n  def incr[T](counter: Counter): T => T = { x =>\n    {\n      counter.incr()\n      x\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/IbisScribeTargets.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.CommonRecommendationType._\n\nobject IbisScribeTargets {\n  val User2 = \"magic_rec_user_2\"\n  val User4 = \"magic_rec_user_4\"\n  val Tweet2 = \"magic_rec_tweet_2\"\n  val Tweet4 = \"magic_rec_tweet_4\"\n  val Tweet5 = \"magic_rec_tweet_5\"\n  val Tweet9 = \"magic_rec_tweet_9\"\n  val Tweet10 = \"magic_rec_tweet_10\"\n  val Tweet11 = \"magic_rec_tweet_11\"\n  val Tweet12 = \"magic_rec_tweet_12\"\n  val Tweet16 = \"magic_rec_tweet_16\"\n  val Hashtag = \"magic_rec_hashtag\"\n  val UnreadBadgeCount17 = \"magic_rec_unread_badge_count_17\"\n  val Highlights = \"highlights\"\n  val TweetAnalytics = \"magic_rec_tweet_analytics\"\n  val Untracked = \"untracked\"\n\n  def crtToScribeTarget(crt: CommonRecommendationType): String = crt match {\n    case UserFollow =>\n      User2\n    case HermitUser =>\n      User4\n    case TweetRetweet | TweetFavorite =>\n      Tweet2\n    case TweetRetweetPhoto | TweetFavoritePhoto =>\n      Tweet4\n    case TweetRetweetVideo | TweetFavoriteVideo =>\n      Tweet5\n    case UrlTweetLanding =>\n      Tweet9\n    case F1FirstdegreeTweet | F1FirstdegreePhoto | F1FirstdegreeVideo =>\n      Tweet10\n    case AuthorTargetingTweet =>\n      Tweet11\n    case PeriscopeShare =>\n      Tweet12\n    case CommonRecommendationType.Highlights =>\n      Highlights\n    case HashtagTweet | HashtagTweetRetweet =>\n      Hashtag\n    case PinnedTweet =>\n      Tweet16\n    case UnreadBadgeCount =>\n      UnreadBadgeCount17\n    case TweetImpressions =>\n      TweetAnalytics\n    case _ =>\n      Untracked\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/InlineActionUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.google.common.io.BaseEncoding\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.InlineActionsEnum\nimport com.twitter.frigate.pushservice.params.PushParams\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.ibis2.lib.util.JsonMarshal\nimport com.twitter.notifications.platform.thriftscala._\nimport com.twitter.notificationservice.thriftscala.CreateGenericNotificationResponse\nimport com.twitter.scrooge.BinaryThriftStructSerializer\nimport com.twitter.util.Future\n\n/**\n * This class provides utility functions for inline action for push\n */\nobject InlineActionUtil {\n\n  def scopedStats(statsReceiver: StatsReceiver): StatsReceiver =\n    statsReceiver.scope(getClass.getSimpleName)\n\n  /**\n   * Util function to build web inline actions for Ibis\n   * @param actions list of inline actions to be hydrated depending on the CRT\n   * @param enableForDesktopWeb if web inline actions should be shown on desktop RWeb, for experimentation purpose\n   * @param enableForMobileWeb if web inline actions should be shwon on mobile RWeb, for experimentation purpose\n   * @return Params for web inline actions to be consumed by `smart.inline.actions.web.mustache` in Ibis\n   */\n  def getGeneratedTweetInlineActionsForWeb(\n    actions: Seq[InlineActionsEnum.Value],\n    enableForDesktopWeb: Boolean,\n    enableForMobileWeb: Boolean\n  ): Map[String, String] = {\n    if (!enableForDesktopWeb && !enableForMobileWeb) {\n      Map.empty\n    } else {\n      val inlineActions = buildEnrichedInlineActionsMap(actions) ++ Map(\n        \"enable_for_desktop_web\" -> enableForDesktopWeb.toString,\n        \"enable_for_mobile_web\" -> enableForMobileWeb.toString\n      )\n      Map(\n        \"inline_action_details_web\" -> JsonMarshal.toJson(inlineActions),\n      )\n    }\n  }\n\n  def getGeneratedTweetInlineActionsV1(\n    actions: Seq[InlineActionsEnum.Value]\n  ): Map[String, String] = {\n    val inlineActions = buildEnrichedInlineActionsMap(actions)\n    Map(\n      \"inline_action_details\" -> JsonMarshal.toJson(inlineActions)\n    )\n  }\n\n  private def buildEnrichedInlineActionsMap(\n    actions: Seq[InlineActionsEnum.Value]\n  ): Map[String, Seq[Map[String, Any]]] = {\n    Map(\n      \"actions\" -> actions\n        .map(_.toString.toLowerCase)\n        .zipWithIndex\n        .map {\n          case (a: String, i: Int) =>\n            Map(\"action\" -> a) ++ Map(\n              s\"use_${a}_stringcenter_key\" -> true,\n              \"last\" -> (i == (actions.length - 1))\n            )\n        }.seq\n    )\n  }\n\n  def getGeneratedTweetInlineActionsV2(\n    actions: Seq[InlineActionsEnum.Value]\n  ): Map[String, String] = {\n    val v2CustomActions = actions\n      .map {\n        case InlineActionsEnum.Favorite =>\n          NotificationCustomAction(\n            Some(\"mr_inline_favorite_title\"),\n            CustomActionData.LegacyAction(LegacyAction(ActionIdentifier.Favorite))\n          )\n        case InlineActionsEnum.Follow =>\n          NotificationCustomAction(\n            Some(\"mr_inline_follow_title\"),\n            CustomActionData.LegacyAction(LegacyAction(ActionIdentifier.Follow)))\n        case InlineActionsEnum.Reply =>\n          NotificationCustomAction(\n            Some(\"mr_inline_reply_title\"),\n            CustomActionData.LegacyAction(LegacyAction(ActionIdentifier.Reply)))\n        case InlineActionsEnum.Retweet =>\n          NotificationCustomAction(\n            Some(\"mr_inline_retweet_title\"),\n            CustomActionData.LegacyAction(LegacyAction(ActionIdentifier.Retweet)))\n        case _ =>\n          NotificationCustomAction(\n            Some(\"mr_inline_favorite_title\"),\n            CustomActionData.LegacyAction(LegacyAction(ActionIdentifier.Favorite))\n          )\n      }\n    val notifications = NotificationCustomActions(v2CustomActions)\n    Map(\"serialized_inline_actions_v2\" -> serializeActionsToBase64(notifications))\n  }\n\n  def getDislikeInlineAction(\n    candidate: PushCandidate,\n    ntabResponse: CreateGenericNotificationResponse\n  ): Option[NotificationCustomAction] = {\n    ntabResponse.successKey.map(successKey => {\n      val urlParams = Map[String, String](\n        \"answer\" -> \"dislike\",\n        \"notification_hash\" -> successKey.hashKey.toString,\n        \"upstream_uid\" -> candidate.impressionId,\n        \"notification_timestamp\" -> successKey.timestampMillis.toString\n      )\n      val urlParamsString = urlParams.map(kvp => f\"${kvp._1}=${kvp._2}\").mkString(\"&\")\n\n      val httpPostRequest = HttpRequest.PostRequest(\n        PostRequest(url = f\"/2/notifications/feedback.json?$urlParamsString\", bodyParams = None))\n      val httpRequestAction = HttpRequestAction(\n        httpRequest = httpPostRequest,\n        scribeAction = Option(\"dislike_scribe_action\"),\n        isAuthorizationRequired = Option(true),\n        isDestructive = Option(false),\n        undoable = None\n      )\n      val dislikeAction = CustomActionData.HttpRequestAction(httpRequestAction)\n      NotificationCustomAction(title = Option(\"mr_inline_dislike_title\"), action = dislikeAction)\n    })\n  }\n\n  /**\n   * Given a serialized inline action v2, update the action at index to the given new action.\n   * If given index is bigger than current action length, append the given inline action at the end.\n   * @param serialized_inline_actions_v2 the original action in serialized version\n   * @param actionOption an Option of the new action to replace the old one\n   * @param index the position where the old action will be replaced\n   * @return a new serialized inline action v2\n   */\n  def patchInlineActionAtPosition(\n    serialized_inline_actions_v2: String,\n    actionOption: Option[NotificationCustomAction],\n    index: Int\n  ): String = {\n    val originalActions: Seq[NotificationCustomAction] = deserializeActionsFromString(\n      serialized_inline_actions_v2).actions\n    val newActions = actionOption match {\n      case Some(action) if index >= originalActions.size => originalActions ++ Seq(action)\n      case Some(action) => originalActions.updated(index, action)\n      case _ => originalActions\n    }\n    serializeActionsToBase64(NotificationCustomActions(newActions))\n  }\n\n  /**\n   * Return list of available inline actions for ibis2 model\n   */\n  def getGeneratedTweetInlineActions(\n    target: Target,\n    statsReceiver: StatsReceiver,\n    actions: Seq[InlineActionsEnum.Value],\n  ): Map[String, String] = {\n    val scopedStatsReceiver = scopedStats(statsReceiver)\n    val useV1 = target.params(PushFeatureSwitchParams.UseInlineActionsV1)\n    val useV2 = target.params(PushFeatureSwitchParams.UseInlineActionsV2)\n    if (useV1 && useV2) {\n      scopedStatsReceiver.counter(\"use_v1_and_use_v2\").incr()\n      getGeneratedTweetInlineActionsV1(actions) ++ getGeneratedTweetInlineActionsV2(actions)\n    } else if (useV1 && !useV2) {\n      scopedStatsReceiver.counter(\"only_use_v1\").incr()\n      getGeneratedTweetInlineActionsV1(actions)\n    } else if (!useV1 && useV2) {\n      scopedStatsReceiver.counter(\"only_use_v2\").incr()\n      getGeneratedTweetInlineActionsV2(actions)\n    } else {\n      scopedStatsReceiver.counter(\"use_neither_v1_nor_v2\").incr()\n      Map.empty[String, String]\n    }\n  }\n\n  /**\n   * Return Tweet inline action ibis2 model values after applying experiment logic\n   */\n  def getTweetInlineActionValue(target: Target): Future[Map[String, String]] = {\n    if (target.isLoggedOutUser) {\n      Future(\n        Map(\n          \"show_inline_action\" -> \"false\"\n        )\n      )\n    } else {\n      val showInlineAction: Boolean = target.params(PushParams.MRAndroidInlineActionOnPushCopyParam)\n      Future(\n        Map(\n          \"show_inline_action\" -> s\"$showInlineAction\"\n        )\n      )\n    }\n  }\n\n  private val binaryThriftStructSerializer: BinaryThriftStructSerializer[\n    NotificationCustomActions\n  ] = BinaryThriftStructSerializer.apply(NotificationCustomActions)\n  private val base64Encoding = BaseEncoding.base64()\n\n  def serializeActionsToBase64(notificationCustomActions: NotificationCustomActions): String = {\n    val actionsAsByteArray: Array[Byte] =\n      binaryThriftStructSerializer.toBytes(notificationCustomActions)\n    base64Encoding.encode(actionsAsByteArray)\n  }\n\n  def deserializeActionsFromString(serializedInlineActionV2: String): NotificationCustomActions = {\n    val actionAsByteArray = base64Encoding.decode(serializedInlineActionV2)\n    binaryThriftStructSerializer.fromBytes(actionAsByteArray)\n  }\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/MediaAnnotationsUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\n\nobject MediaAnnotationsUtil {\n\n  val mediaIdToCategoryMapping = Map(\"0\" -> \"0\")\n\n  val nudityCategoryId = \"0\"\n  val beautyCategoryId = \"0\"\n  val singlePersonCategoryId = \"0\"\n  val sensitiveMediaCategoryFeatureName =\n    \"tweet.mediaunderstanding.tweet_annotations.sensitive_category_probabilities\"\n\n  def updateMediaCategoryStats(\n    candidates: Seq[CandidateDetails[PushCandidate]]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ) = {\n\n    val statScope = statsReceiver.scope(\"mediaStats\")\n    val filteredCandidates = candidates.filter { candidate =>\n      !candidate.candidate.sparseContinuousFeatures\n        .getOrElse(sensitiveMediaCategoryFeatureName, Map.empty[String, Double]).contains(\n          nudityCategoryId)\n    }\n\n    if (filteredCandidates.isEmpty)\n      statScope.counter(\"emptyCandidateListAfterNudityFilter\").incr()\n    else\n      statScope.counter(\"nonEmptyCandidateListAfterNudityFilter\").incr()\n    candidates.foreach { candidate =>\n      statScope.counter(\"totalCandidates\").incr()\n      val mediaFeature = candidate.candidate.sparseContinuousFeatures\n        .getOrElse(sensitiveMediaCategoryFeatureName, Map.empty[String, Double])\n      if (mediaFeature.nonEmpty) {\n        val mediaCategoryByMaxScore = mediaFeature.maxBy(_._2)._1\n        statScope\n          .scope(\"mediaCategoryByMaxScore\").counter(mediaIdToCategoryMapping\n            .getOrElse(mediaCategoryByMaxScore, \"undefined\")).incr()\n\n        mediaFeature.keys.map { feature =>\n          statScope\n            .scope(\"mediaCategory\").counter(mediaIdToCategoryMapping\n              .getOrElse(feature, \"undefined\")).incr()\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/MinDurationModifierCalculator.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.TimeUtil\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FSParams}\nimport com.twitter.util.Future\nimport com.twitter.util.Time\nimport java.util.Calendar\nimport java.util.TimeZone\n\ncase class MinDurationModifierCalculator() {\n\n  private def mapCountryCodeToTimeZone(\n    countryCode: String,\n    stats: StatsReceiver\n  ): Option[Calendar] = {\n    PushConstants.countryCodeToTimeZoneMap\n      .get(countryCode.toUpperCase).map(timezone =>\n        Calendar.getInstance(TimeZone.getTimeZone(timezone)))\n  }\n\n  private def transformToHour(\n    dayOfHour: Int\n  ): Int = {\n    if (dayOfHour < 0) dayOfHour + 24\n    else dayOfHour\n  }\n\n  private def getMinDurationByHourOfDay(\n    hourOfDay: Int,\n    startTimeList: Seq[Int],\n    endTimeList: Seq[Int],\n    minDurationTimeModifierConst: Seq[Int],\n    stats: StatsReceiver\n  ): Option[Int] = {\n    val scopedStats = stats.scope(\"getMinDurationByHourOfDay\")\n    scopedStats.counter(\"request\").incr()\n    val durationOpt = (startTimeList, endTimeList, minDurationTimeModifierConst).zipped.toList\n      .filter {\n        case (startTime, endTime, _) =>\n          if (startTime <= endTime) hourOfDay >= startTime && hourOfDay < endTime\n          else (hourOfDay >= startTime) || hourOfDay < endTime\n        case _ => false\n      }.map {\n        case (_, _, modifier) => modifier\n      }.headOption\n    durationOpt match {\n      case Some(duration) => scopedStats.counter(s\"$duration.minutes\").incr()\n      case _ => scopedStats.counter(\"none\").incr()\n    }\n    durationOpt\n  }\n\n  def getMinDurationModifier(\n    target: Target,\n    calendar: Calendar,\n    stats: StatsReceiver\n  ): Option[Int] = {\n    val startTimeList = target.params(FSParams.MinDurationModifierStartHourList)\n    val endTimeList = target.params(FSParams.MinDurationModifierEndHourList)\n    val minDurationTimeModifierConst = target.params(FSParams.MinDurationTimeModifierConst)\n    if (startTimeList.length != endTimeList.length || minDurationTimeModifierConst.length != startTimeList.length) {\n      None\n    } else {\n      val hourOfDay = calendar.get(Calendar.HOUR_OF_DAY)\n      getMinDurationByHourOfDay(\n        hourOfDay,\n        startTimeList,\n        endTimeList,\n        minDurationTimeModifierConst,\n        stats)\n    }\n  }\n\n  def getMinDurationModifier(\n    target: Target,\n    countryCodeOpt: Option[String],\n    stats: StatsReceiver\n  ): Option[Int] = {\n    val scopedStats = stats\n      .scope(\"getMinDurationModifier\")\n    scopedStats.counter(\"total_requests\").incr()\n\n    countryCodeOpt match {\n      case Some(countryCode) =>\n        scopedStats\n          .counter(\"country_code_exists\").incr()\n        val calendarOpt = mapCountryCodeToTimeZone(countryCode, scopedStats)\n        calendarOpt.flatMap(calendar => getMinDurationModifier(target, calendar, scopedStats))\n      case _ => None\n    }\n  }\n\n  def getMinDurationModifier(target: Target, stats: StatsReceiver): Future[Option[Int]] = {\n    val scopedStats = stats\n      .scope(\"getMinDurationModifier\")\n    scopedStats.counter(\"total_requests\").incr()\n\n    val startTimeList = target.params(FSParams.MinDurationModifierStartHourList)\n    val endTimeList = target.params(FSParams.MinDurationModifierEndHourList)\n    val minDurationTimeModifierConst = target.params(FSParams.MinDurationTimeModifierConst)\n    if (startTimeList.length != endTimeList.length || minDurationTimeModifierConst.length != startTimeList.length) {\n      Future.value(None)\n    } else {\n      target.localTimeInHHMM.map {\n        case (hourOfDay, _) =>\n          getMinDurationByHourOfDay(\n            hourOfDay,\n            startTimeList,\n            endTimeList,\n            minDurationTimeModifierConst,\n            scopedStats)\n        case _ => None\n      }\n    }\n  }\n\n  def getMinDurationModifierByUserOpenedHistory(\n    target: Target,\n    openedPushByHourAggregatedOpt: Option[Map[Int, Int]],\n    stats: StatsReceiver\n  ): Option[Int] = {\n    val scopedStats = stats\n      .scope(\"getMinDurationModifierByUserOpenedHistory\")\n    scopedStats.counter(\"total_requests\").incr()\n    openedPushByHourAggregatedOpt match {\n      case Some(openedPushByHourAggregated) =>\n        if (openedPushByHourAggregated.isEmpty) {\n          scopedStats.counter(\"openedPushByHourAggregated_empty\").incr()\n          None\n        } else {\n          val currentUTCHour = TimeUtil.hourOfDay(Time.now)\n          val utcHourWithMaxOpened = if (target.params(FSParams.EnableRandomHourForQuickSend)) {\n            (target.targetId % 24).toInt\n          } else {\n            openedPushByHourAggregated.maxBy(_._2)._1\n          }\n          val numOfMaxOpened = openedPushByHourAggregated.maxBy(_._2)._2\n          if (numOfMaxOpened >= target.params(FSParams.SendTimeByUserHistoryMaxOpenedThreshold)) {\n            scopedStats.counter(\"pass_experiment_bucket_threshold\").incr()\n            if (numOfMaxOpened >= target\n                .params(FSParams.SendTimeByUserHistoryMaxOpenedThreshold)) { // only update if number of opened pushes meet threshold\n              scopedStats.counter(\"pass_max_threshold\").incr()\n              val quickSendBeforeHours =\n                target.params(FSParams.SendTimeByUserHistoryQuickSendBeforeHours)\n              val quickSendAfterHours =\n                target.params(FSParams.SendTimeByUserHistoryQuickSendAfterHours)\n\n              val hoursToLessSend = target.params(FSParams.SendTimeByUserHistoryNoSendsHours)\n\n              val quickSendTimeMinDurationInMinute =\n                target.params(FSParams.SendTimeByUserHistoryQuickSendMinDurationInMinute)\n              val noSendTimeMinDuration =\n                target.params(FSParams.SendTimeByUserHistoryNoSendMinDuration)\n\n              val startTimeForNoSend = transformToHour(\n                utcHourWithMaxOpened - quickSendBeforeHours - hoursToLessSend)\n              val startTimeForQuickSend = transformToHour(\n                utcHourWithMaxOpened - quickSendBeforeHours)\n              val endTimeForNoSend =\n                transformToHour(utcHourWithMaxOpened - quickSendBeforeHours)\n              val endTimeForQuickSend =\n                transformToHour(utcHourWithMaxOpened + quickSendAfterHours) + 1\n\n              val startTimeList = Seq(startTimeForNoSend, startTimeForQuickSend)\n              val endTimeList = Seq(endTimeForNoSend, endTimeForQuickSend)\n              val minDurationTimeModifierConst =\n                Seq(noSendTimeMinDuration, quickSendTimeMinDurationInMinute)\n\n              getMinDurationByHourOfDay(\n                currentUTCHour,\n                startTimeList,\n                endTimeList,\n                minDurationTimeModifierConst,\n                scopedStats)\n\n            } else None\n          } else None\n        }\n      case _ =>\n        None\n    }\n  }\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/MrUserStateUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TargetUser\n\nobject MrUserStateUtil {\n  def updateMrUserStateStats(target: TargetUser)(implicit statsReceiver: StatsReceiver) = {\n    statsReceiver.counter(\"AllUserStates\").incr()\n    target.targetMrUserState.map {\n      case Some(state) =>\n        statsReceiver.counter(state.name).incr()\n      case _ =>\n        statsReceiver.counter(\"UnknownUserState\").incr()\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/NsfwPersonalizationUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.scio.nsfw_user_segmentation.thriftscala.NSFWUserSegmentation\n\nobject NsfwPersonalizationUtil {\n  def computeNsfwUserStats(\n    targetNsfwInfo: Option[NsfwInfo]\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): Unit = {\n\n    def computeNsfwProfileVisitStats(sReceiver: StatsReceiver, nsfwProfileVisits: Long): Unit = {\n      if (nsfwProfileVisits >= 1)\n        sReceiver.counter(\"nsfwProfileVisits_gt_1\").incr()\n      if (nsfwProfileVisits >= 2)\n        sReceiver.counter(\"nsfwProfileVisits_gt_2\").incr()\n      if (nsfwProfileVisits >= 3)\n        sReceiver.counter(\"nsfwProfileVisits_gt_3\").incr()\n      if (nsfwProfileVisits >= 5)\n        sReceiver.counter(\"nsfwProfileVisits_gt_5\").incr()\n      if (nsfwProfileVisits >= 8)\n        sReceiver.counter(\"nsfwProfileVisits_gt_8\").incr()\n    }\n\n    def computeRatioStats(\n      sReceiver: StatsReceiver,\n      ratio: Int,\n      statName: String,\n      intervals: List[Double] = List(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)\n    ): Unit = {\n      intervals.foreach { i =>\n        if (ratio > i * 10000)\n          sReceiver.counter(f\"${statName}_greater_than_${i}\").incr()\n      }\n    }\n    val sReceiver = statsReceiver.scope(\"nsfw_personalization\")\n    sReceiver.counter(\"AllUsers\").incr()\n\n    (targetNsfwInfo) match {\n      case (Some(nsfwInfo)) =>\n        val sensitive = nsfwInfo.senstiveOptIn.getOrElse(false)\n        val nsfwFollowRatio =\n          nsfwInfo.nsfwFollowRatio\n        val totalFollows = nsfwInfo.totalFollowCount\n        val numNsfwProfileVisits = nsfwInfo.nsfwProfileVisits\n        val nsfwRealGraphScore = nsfwInfo.realGraphScore\n        val nsfwSearchScore = nsfwInfo.searchNsfwScore\n        val totalSearches = nsfwInfo.totalSearches\n        val realGraphScore = nsfwInfo.realGraphScore\n        val searchScore = nsfwInfo.searchNsfwScore\n\n        if (sensitive)\n          sReceiver.counter(\"sensitiveOptInEnabled\").incr()\n        else\n          sReceiver.counter(\"sensitiveOptInDisabled\").incr()\n\n        computeRatioStats(sReceiver, nsfwFollowRatio, \"nsfwRatio\")\n        computeNsfwProfileVisitStats(sReceiver, numNsfwProfileVisits)\n        computeRatioStats(sReceiver, nsfwRealGraphScore.toInt, \"nsfwRealGraphScore\")\n\n        if (totalSearches >= 10)\n          computeRatioStats(sReceiver, nsfwSearchScore.toInt, \"nsfwSearchScore\")\n        if (searchScore == 0)\n          sReceiver.counter(\"lowSearchScore\").incr()\n        if (realGraphScore < 500)\n          sReceiver.counter(\"lowRealScore\").incr()\n        if (numNsfwProfileVisits == 0)\n          sReceiver.counter(\"lowProfileVisit\").incr()\n        if (nsfwFollowRatio == 0)\n          sReceiver.counter(\"lowFollowScore\").incr()\n\n        if (totalSearches > 10 && searchScore > 5000)\n          sReceiver.counter(\"highSearchScore\").incr()\n        if (realGraphScore > 7000)\n          sReceiver.counter(\"highRealScore\").incr()\n        if (numNsfwProfileVisits > 5)\n          sReceiver.counter(\"highProfileVisit\").incr()\n        if (totalFollows > 10 && nsfwFollowRatio > 7000)\n          sReceiver.counter(\"highFollowScore\").incr()\n\n        if (searchScore == 0 && realGraphScore <= 500 && numNsfwProfileVisits == 0 && nsfwFollowRatio == 0)\n          sReceiver.counter(\"lowIntent\").incr()\n        if ((totalSearches > 10 && searchScore > 5000) || realGraphScore > 7000 || numNsfwProfileVisits > 5 || (totalFollows > 10 && nsfwFollowRatio > 7000))\n          sReceiver.counter(\"highIntent\").incr()\n      case _ =>\n    }\n  }\n}\n\ncase class NsfwInfo(nsfwUserSegmentation: NSFWUserSegmentation) {\n\n  val scalingFactor = 10000 // to convert float to int as custom fields cannot be float\n  val senstiveOptIn: Option[Boolean] = nsfwUserSegmentation.nsfwView\n  val totalFollowCount: Long = nsfwUserSegmentation.totalFollowCnt.getOrElse(0L)\n  val nsfwFollowCnt: Long =\n    nsfwUserSegmentation.nsfwAdminOrHighprecOrAgathaGtP98FollowsCnt.getOrElse(0L)\n  val nsfwFollowRatio: Int = {\n    if (totalFollowCount != 0) {\n      (nsfwFollowCnt * scalingFactor / totalFollowCount).toInt\n    } else 0\n  }\n  val nsfwProfileVisits: Long =\n    nsfwUserSegmentation.nsfwAdminOrHighPrecOrAgathaGtP98Visits\n      .map(_.numProfilesInLast14Days).getOrElse(0L)\n  val realGraphScore: Int =\n    nsfwUserSegmentation.realGraphMetrics\n      .map { rm =>\n        if (rm.totalOutboundRGScore != 0)\n          rm.totalNsfwAdmHPAgthGtP98OutboundRGScore * scalingFactor / rm.totalOutboundRGScore\n        else 0d\n      }.getOrElse(0d).toInt\n  val totalSearches: Long =\n    nsfwUserSegmentation.searchMetrics.map(_.numNonTrndSrchInLast14Days).getOrElse(0L)\n  val searchNsfwScore: Int = nsfwUserSegmentation.searchMetrics\n    .map { sm =>\n      if (sm.numNonTrndNonHshtgSrchInLast14Days != 0)\n        sm.numNonTrndNonHshtgGlobalNsfwSrchInLast14Days.toDouble * scalingFactor / sm.numNonTrndNonHshtgSrchInLast14Days\n      else 0\n    }.getOrElse(0d).toInt\n  val hasReported: Boolean =\n    nsfwUserSegmentation.notifFeedbackMetrics.exists(_.notifReportMetrics.exists(_.countTotal != 0))\n  val hasDisliked: Boolean =\n    nsfwUserSegmentation.notifFeedbackMetrics\n      .exists(_.notifDislikeMetrics.exists(_.countTotal != 0))\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/OverrideNotificationUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.MagicFanoutEventCandidate\nimport com.twitter.frigate.common.history.History\nimport com.twitter.frigate.common.rec_types.RecTypes\nimport com.twitter.frigate.common.store.deviceinfo.DeviceInfo\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes\nimport com.twitter.frigate.pushservice.model.ibis.PushOverrideInfo\nimport com.twitter.frigate.pushservice.params.PushConstants\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.pushservice.params.{PushFeatureSwitchParams => FSParams}\nimport com.twitter.frigate.thriftscala.CollapseInfo\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\nimport com.twitter.frigate.thriftscala.CommonRecommendationType.MagicFanoutSportsEvent\nimport com.twitter.frigate.thriftscala.OverrideInfo\nimport com.twitter.util.Future\nimport java.util.UUID\n\nobject OverrideNotificationUtil {\n\n  /**\n   * Gets Override Info for the current notification.\n   * @param candidate [[PushCandidate]] object representing the recommendation candidate\n   * @param stats     StatsReceiver to track stats for this function as well as the subsequent funcs. called\n   * @return          Returns OverrideInfo if CollapseInfo exists, else None\n   */\n\n  def getOverrideInfo(\n    candidate: PushCandidate,\n    stats: StatsReceiver\n  ): Future[Option[OverrideInfo]] = {\n    if (candidate.target.isLoggedOutUser) {\n      Future.None\n    } else if (isOverrideEnabledForCandidate(candidate))\n      getCollapseInfo(candidate, stats).map(_.map(OverrideInfo(_)))\n    else Future.None\n  }\n\n  private def getCollapseInfo(\n    candidate: PushCandidate,\n    stats: StatsReceiver\n  ): Future[Option[CollapseInfo]] = {\n    val target = candidate.target\n    for {\n      targetHistory <- target.history\n      deviceInfo <- target.deviceInfo\n    } yield getCollapseInfo(target, targetHistory, deviceInfo, stats)\n  }\n\n  /**\n   * Get Collapse Info for the current notification.\n   * @param target          Push Target - recipient of the notification\n   * @param targetHistory   Target's History\n   * @param deviceInfoOpt   `Option` of the Target's Device Info\n   * @param stats           StatsReceiver to track stats for this function as well as the subsequent funcs. called\n   * @return                Returns CollapseInfo if the Target is eligible for Override Notifs, else None\n   */\n  def getCollapseInfo(\n    target: PushTypes.Target,\n    targetHistory: History,\n    deviceInfoOpt: Option[DeviceInfo],\n    stats: StatsReceiver\n  ): Option[CollapseInfo] = {\n    val overrideInfoOfLastNotif =\n      PushOverrideInfo.getOverrideInfoOfLastEligiblePushNotif(\n        targetHistory,\n        target.params(FSParams.OverrideNotificationsLookbackDurationForOverrideInfo),\n        stats)\n    overrideInfoOfLastNotif match {\n      case Some(prevOverrideInfo) if isOverrideEnabled(target, deviceInfoOpt, stats) =>\n        val notifsInLastOverrideChain =\n          PushOverrideInfo.getMrPushNotificationsInOverrideChain(\n            targetHistory,\n            prevOverrideInfo.collapseInfo.overrideChainId,\n            stats)\n        val numNotifsInLastOverrideChain = notifsInLastOverrideChain.size\n        val timestampOfFirstNotifInOverrideChain =\n          PushOverrideInfo\n            .getTimestampInMillisForFrigateNotification(\n              notifsInLastOverrideChain.last,\n              targetHistory,\n              stats).getOrElse(PushConstants.DefaultLookBackForHistory.ago.inMilliseconds)\n        if (numNotifsInLastOverrideChain < target.params(FSParams.MaxMrPushSends24HoursParam) &&\n          timestampOfFirstNotifInOverrideChain > PushConstants.DefaultLookBackForHistory.ago.inMilliseconds) {\n          Some(prevOverrideInfo.collapseInfo)\n        } else {\n          val prevCollapseId = prevOverrideInfo.collapseInfo.collapseId\n          val newOverrideChainId = UUID.randomUUID.toString.replaceAll(\"-\", \"\")\n          Some(CollapseInfo(prevCollapseId, newOverrideChainId))\n        }\n      case None if isOverrideEnabled(target, deviceInfoOpt, stats) =>\n        val newOverrideChainId = UUID.randomUUID.toString.replaceAll(\"-\", \"\")\n        Some(CollapseInfo(\"\", newOverrideChainId))\n      case _ => None // Override is disabled for everything else\n    }\n  }\n\n  /**\n   * Gets the collapse and impression identifier for the current override notification\n   * @param target  Push Target - recipient of the notification\n   * @param stats   StatsReceiver to track stats for this function as well as the subsequent funcs. called\n   * @return        A Future of Collapse ID as well as the Impression ID.\n   */\n  def getCollapseAndImpressionIdForOverride(\n    candidate: PushCandidate\n  ): Future[Option[(String, Seq[String])]] = {\n    if (isOverrideEnabledForCandidate(candidate)) {\n      val target = candidate.target\n      val stats = candidate.statsReceiver\n      Future.join(target.history, target.deviceInfo).map {\n        case (targetHistory, deviceInfoOpt) =>\n          val collapseInfoOpt = getCollapseInfo(target, targetHistory, deviceInfoOpt, stats)\n\n          val impressionIds = candidate.commonRecType match {\n            case MagicFanoutSportsEvent\n                if target.params(FSParams.EnableEventIdBasedOverrideForSportsCandidates) =>\n              PushOverrideInfo.getImpressionIdsForPrevEligibleMagicFanoutEventCandidates(\n                targetHistory,\n                target.params(FSParams.OverrideNotificationsLookbackDurationForImpressionId),\n                stats,\n                MagicFanoutSportsEvent,\n                candidate\n                  .asInstanceOf[RawCandidate with MagicFanoutEventCandidate].eventId\n              )\n            case _ =>\n              PushOverrideInfo.getImpressionIdsOfPrevEligiblePushNotif(\n                targetHistory,\n                target.params(FSParams.OverrideNotificationsLookbackDurationForImpressionId),\n                stats)\n          }\n\n          collapseInfoOpt match {\n            case Some(collapseInfo) if impressionIds.nonEmpty =>\n              val notifsInLastOverrideChain =\n                PushOverrideInfo.getMrPushNotificationsInOverrideChain(\n                  targetHistory,\n                  collapseInfo.overrideChainId,\n                  stats)\n              stats\n                .scope(\"OverrideNotificationUtil\").stat(\"number_of_notifications_sent\").add(\n                  notifsInLastOverrideChain.size + 1)\n              Some((collapseInfo.collapseId, impressionIds))\n            case _ => None\n          }\n        case _ => None\n      }\n    } else Future.None\n  }\n\n  /**\n   * Checks to see if override notifications are enabled based on the Target's Device Info and Params\n   * @param target          Push Target - recipient of the notification\n   * @param deviceInfoOpt   `Option` of the Target's Device Info\n   * @param stats           StatsReceiver to track stats for this function\n   * @return                Returns True if Override Notifications are enabled for the provided\n   *                        Target, else False.\n   */\n  private def isOverrideEnabled(\n    target: PushTypes.Target,\n    deviceInfoOpt: Option[DeviceInfo],\n    stats: StatsReceiver\n  ): Boolean = {\n    val scopedStats = stats.scope(\"OverrideNotificationUtil\").scope(\"isOverrideEnabled\")\n    val enabledForAndroidCounter = scopedStats.counter(\"android_enabled\")\n    val disabledForAndroidCounter = scopedStats.counter(\"android_disabled\")\n    val enabledForIosCounter = scopedStats.counter(\"ios_enabled\")\n    val disabledForIosCounter = scopedStats.counter(\"ios_disabled\")\n    val disabledForOtherDevicesCounter = scopedStats.counter(\"other_disabled\")\n\n    val isPrimaryDeviceAndroid = PushDeviceUtil.isPrimaryDeviceAndroid(deviceInfoOpt)\n    val isPrimaryDeviceIos = PushDeviceUtil.isPrimaryDeviceIOS(deviceInfoOpt)\n\n    lazy val validAndroidDevice =\n      isPrimaryDeviceAndroid && target.params(FSParams.EnableOverrideNotificationsForAndroid)\n    lazy val validIosDevice =\n      isPrimaryDeviceIos && target.params(FSParams.EnableOverrideNotificationsForIos)\n\n    if (isPrimaryDeviceAndroid) {\n      if (validAndroidDevice) enabledForAndroidCounter.incr() else disabledForAndroidCounter.incr()\n    } else if (isPrimaryDeviceIos) {\n      if (validIosDevice) enabledForIosCounter.incr() else disabledForIosCounter.incr()\n    } else {\n      disabledForOtherDevicesCounter.incr()\n    }\n\n    validAndroidDevice || validIosDevice\n  }\n\n  /**\n   * Checks if override is enabled for the currently supported types for SendHandler or not.\n   * This method is package private for unit testing.\n   * @param candidate [[PushCandidate]]\n   * @param stats StatsReceiver to track statistics for this function\n   * @return      Returns True if override notifications are enabled for the current type, otherwise False.\n   */\n  private def isOverrideEnabledForSendHandlerCandidate(\n    candidate: PushCandidate\n  ): Boolean = {\n    val scopedStats = candidate.statsReceiver\n      .scope(\"OverrideNotificationUtil\").scope(\"isOverrideEnabledForSendHandlerType\")\n\n    val overrideSupportedTypesForSpaces: Set[CommonRecommendationType] = Set(\n      CommonRecommendationType.SpaceSpeaker,\n      CommonRecommendationType.SpaceHost\n    )\n\n    val isOverrideSupportedForSpaces = {\n      overrideSupportedTypesForSpaces.contains(candidate.commonRecType) &&\n      candidate.target.params(FSParams.EnableOverrideForSpaces)\n    }\n\n    val isOverrideSupportedForSports = {\n      candidate.commonRecType == CommonRecommendationType.MagicFanoutSportsEvent &&\n      candidate.target\n        .params(PushFeatureSwitchParams.EnableOverrideForSportsCandidates)\n    }\n\n    val isOverrideSupported = isOverrideSupportedForSpaces || isOverrideSupportedForSports\n\n    scopedStats.counter(s\"$isOverrideSupported\").incr()\n    isOverrideSupported\n  }\n\n  private[util] def isOverrideEnabledForCandidate(candidate: PushCandidate) =\n    !RecTypes.isSendHandlerType(\n      candidate.commonRecType) || isOverrideEnabledForSendHandlerCandidate(candidate)\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/PushAdaptorUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.contentrecommender.thriftscala.MetricTag\nimport com.twitter.frigate.common.base.AlgorithmScore\nimport com.twitter.frigate.common.base.OutOfNetworkTweetCandidate\nimport com.twitter.frigate.common.base.SocialContextAction\nimport com.twitter.frigate.common.base.TopicCandidate\nimport com.twitter.frigate.common.base.TripCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.thriftscala.{SocialContextAction => TSocialContextAction}\nimport com.twitter.frigate.thriftscala.{CommonRecommendationType => CRT}\nimport com.twitter.frigate.thriftscala._\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.topiclisting.utt.LocalizedEntity\nimport com.twitter.trends.trip_v1.trip_tweets.thriftscala.TripDomain\nimport scala.collection.Seq\n\ncase class MediaCRT(\n  crt: CRT,\n  photoCRT: CRT,\n  videoCRT: CRT)\n\nobject PushAdaptorUtil {\n\n  def getFrigateNotificationForUser(\n    crt: CRT,\n    userId: Long,\n    scActions: Seq[SocialContextAction],\n    pushCopyId: Option[Int],\n    ntabCopyId: Option[Int]\n  ): FrigateNotification = {\n\n    val thriftSCActions = scActions.map { scAction =>\n      TSocialContextAction(\n        scAction.userId,\n        scAction.timestampInMillis,\n        scAction.tweetId\n      )\n    }\n    FrigateNotification(\n      crt,\n      NotificationDisplayLocation.PushToMobileDevice,\n      userNotification = Some(UserNotification(userId, thriftSCActions)),\n      pushCopyId = pushCopyId,\n      ntabCopyId = ntabCopyId\n    )\n  }\n\n  def getFrigateNotificationForTweet(\n    crt: CRT,\n    tweetId: Long,\n    scActions: Seq[TSocialContextAction],\n    authorIdOpt: Option[Long],\n    pushCopyId: Option[Int],\n    ntabCopyId: Option[Int],\n    simclusterId: Option[Int],\n    semanticCoreEntityIds: Option[List[Long]],\n    candidateContent: Option[CandidateContent],\n    trendId: Option[String],\n    tweetTripDomain: Option[scala.collection.Set[TripDomain]] = None\n  ): FrigateNotification = {\n    FrigateNotification(\n      crt,\n      NotificationDisplayLocation.PushToMobileDevice,\n      tweetNotification = Some(\n        TweetNotification(\n          tweetId,\n          scActions,\n          authorIdOpt,\n          simclusterId,\n          semanticCoreEntityIds,\n          trendId,\n          tripDomain = tweetTripDomain)\n      ),\n      pushCopyId = pushCopyId,\n      ntabCopyId = ntabCopyId,\n      candidateContent = candidateContent\n    )\n  }\n\n  def getFrigateNotificationForTweetWithSocialContextActions(\n    crt: CRT,\n    tweetId: Long,\n    scActions: Seq[SocialContextAction],\n    authorIdOpt: Option[Long],\n    pushCopyId: Option[Int],\n    ntabCopyId: Option[Int],\n    candidateContent: Option[CandidateContent],\n    semanticCoreEntityIds: Option[List[Long]],\n    trendId: Option[String]\n  ): FrigateNotification = {\n\n    val thriftSCActions = scActions.map { scAction =>\n      TSocialContextAction(\n        scAction.userId,\n        scAction.timestampInMillis,\n        scAction.tweetId\n      )\n    }\n\n    getFrigateNotificationForTweet(\n      crt = crt,\n      tweetId = tweetId,\n      scActions = thriftSCActions,\n      authorIdOpt = authorIdOpt,\n      pushCopyId = pushCopyId,\n      ntabCopyId = ntabCopyId,\n      simclusterId = None,\n      candidateContent = candidateContent,\n      semanticCoreEntityIds = semanticCoreEntityIds,\n      trendId = trendId\n    )\n  }\n\n  def generateOutOfNetworkTweetCandidates(\n    inputTarget: Target,\n    id: Long,\n    mediaCRT: MediaCRT,\n    result: Option[TweetyPieResult],\n    localizedEntity: Option[LocalizedEntity] = None,\n    isMrBackfillFromCR: Option[Boolean] = None,\n    tagsFromCR: Option[Seq[MetricTag]] = None,\n    score: Option[Double] = None,\n    algorithmTypeCR: Option[String] = None,\n    tripTweetDomain: Option[scala.collection.Set[TripDomain]] = None\n  ): RawCandidate\n    with OutOfNetworkTweetCandidate\n    with TopicCandidate\n    with TripCandidate\n    with AlgorithmScore = {\n    new RawCandidate\n      with OutOfNetworkTweetCandidate\n      with TopicCandidate\n      with TripCandidate\n      with AlgorithmScore {\n      override val tweetId: Long = id\n      override val target: Target = inputTarget\n      override val tweetyPieResult: Option[TweetyPieResult] = result\n      override val localizedUttEntity: Option[LocalizedEntity] = localizedEntity\n      override val semanticCoreEntityId: Option[Long] = localizedEntity.map(_.entityId)\n      override def commonRecType: CRT =\n        getMediaBasedCRT(mediaCRT.crt, mediaCRT.photoCRT, mediaCRT.videoCRT)\n      override def isMrBackfillCR: Option[Boolean] = isMrBackfillFromCR\n      override def tagsCR: Option[Seq[MetricTag]] = tagsFromCR\n      override def algorithmScore: Option[Double] = score\n      override def algorithmCR: Option[String] = algorithmTypeCR\n      override def tripDomain: Option[collection.Set[TripDomain]] = tripTweetDomain\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/PushAppPermissionUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.frigate.common.store.deviceinfo.DeviceInfo\nimport com.twitter.onboarding.task.service.models.external.PermissionState\nimport com.twitter.permissions_storage.thriftscala.AppPermission\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nobject PushAppPermissionUtil {\n\n  final val AddressBookPermissionKey = \"addressBook\"\n  final val SyncStateKey = \"syncState\"\n  final val SyncStateOnValue = \"on\"\n\n  /**\n   * Obtains the specified target's App Permissions, based on their primary device.\n   * @param targetId            Target's Identifier\n   * @param permissionName      The permission type we are querying for (address book, geolocation, etc.)\n   * @param deviceInfoFut       Device info of the Target, presented as a Future\n   * @param appPermissionStore  Readable Store which allows us to query the App Permission Strato Column\n   * @return                    Returns the AppPermission of the Target, presented as a Future\n   */\n  def getAppPermission(\n    targetId: Long,\n    permissionName: String,\n    deviceInfoFut: Future[Option[DeviceInfo]],\n    appPermissionStore: ReadableStore[(Long, (String, String)), AppPermission]\n  ): Future[Option[AppPermission]] = {\n    deviceInfoFut.flatMap { deviceInfoOpt =>\n      val primaryDeviceIdOpt = deviceInfoOpt.flatMap(_.primaryDeviceId)\n      primaryDeviceIdOpt match {\n        case Some(primaryDeviceId) =>\n          val queryKey = (targetId, (primaryDeviceId, permissionName))\n          appPermissionStore.get(queryKey)\n        case _ => Future.None\n      }\n    }\n  }\n\n  def hasTargetUploadedAddressBook(\n    appPermissionOpt: Option[AppPermission]\n  ): Boolean = {\n    appPermissionOpt.exists { appPermission =>\n      val syncState = appPermission.metadata.get(SyncStateKey)\n      appPermission.systemPermissionState == PermissionState.On && syncState\n        .exists(_.equalsIgnoreCase(SyncStateOnValue))\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/PushCapUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.TargetUser\nimport com.twitter.frigate.common.candidate.FrigateHistory\nimport com.twitter.frigate.common.candidate.ResurrectedUserDetails\nimport com.twitter.frigate.common.candidate.TargetABDecider\nimport com.twitter.frigate.common.candidate.UserDetails\nimport com.twitter.frigate.pushcap.thriftscala.ModelType\nimport com.twitter.frigate.pushcap.thriftscala.PushcapInfo\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nimport com.twitter.frigate.scribe.thriftscala.PushCapInfo\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\n\ncase class PushCapFatigueInfo(\n  pushcap: Int,\n  fatigueInterval: Duration) {}\n\nobject PushCapUtil {\n\n  def getDefaultPushCap(target: Target): Future[Int] = {\n    Future.value(target.params(PushFeatureSwitchParams.MaxMrPushSends24HoursParam))\n  }\n\n  def getMinimumRestrictedPushcapInfo(\n    restrictedPushcap: Int,\n    originalPushcapInfo: PushcapInfo,\n    statsReceiver: StatsReceiver\n  ): PushcapInfo = {\n    if (originalPushcapInfo.pushcap < restrictedPushcap) {\n      statsReceiver\n        .scope(\"minModelPushcapRestrictions\").counter(\n          f\"num_users_adjusted_from_${originalPushcapInfo.pushcap}_to_${restrictedPushcap}\").incr()\n      PushcapInfo(\n        pushcap = restrictedPushcap.toShort,\n        modelType = ModelType.NoModel,\n        timestamp = 0L,\n        fatigueMinutes = Some((24L / restrictedPushcap) * 60L)\n      )\n    } else originalPushcapInfo\n  }\n\n  def getPushCapFatigue(\n    target: Target,\n    statsReceiver: StatsReceiver\n  ): Future[PushCapFatigueInfo] = {\n    val pushCapStats = statsReceiver.scope(\"pushcap_stats\")\n    target.dynamicPushcap\n      .map { dynamicPushcapOpt =>\n        val pushCap: Int = dynamicPushcapOpt match {\n          case Some(pushcapInfo) => pushcapInfo.pushcap\n          case _ => target.params(PushFeatureSwitchParams.MaxMrPushSends24HoursParam)\n        }\n\n        pushCapStats.stat(\"pushCapValueStats\").add(pushCap)\n        pushCapStats\n          .scope(\"pushCapValueCount\").counter(f\"num_users_with_pushcap_$pushCap\").incr()\n\n        target.finalPushcapAndFatigue += \"pushPushCap\" -> PushCapInfo(\"pushPushCap\", pushCap.toByte)\n\n        PushCapFatigueInfo(pushCap, 24.hours)\n      }\n  }\n\n  def getMinDurationsSincePushWithoutUsingPushCap(\n    target: TargetUser\n      with TargetABDecider\n      with FrigateHistory\n      with UserDetails\n      with ResurrectedUserDetails\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): Duration = {\n    val minDurationSincePush =\n      if (target.params(PushFeatureSwitchParams.EnableGraduallyRampUpNotification)) {\n        val daysInterval =\n          target.params(PushFeatureSwitchParams.GraduallyRampUpPhaseDurationDays).inDays.toDouble\n        val daysSinceActivation =\n          if (target.isResurrectedUser && target.timeSinceResurrection.isDefined) {\n            target.timeSinceResurrection.map(_.inDays.toDouble).get\n          } else {\n            target.timeElapsedAfterSignup.inDays.toDouble\n          }\n        val phaseInterval =\n          Math.max(\n            1,\n            Math.ceil(daysSinceActivation / daysInterval).toInt\n          )\n        val minDuration = 24 / phaseInterval\n        val finalMinDuration =\n          Math.max(4, minDuration).hours\n        statsReceiver\n          .scope(\"GraduallyRampUpFinalMinDuration\").counter(s\"$finalMinDuration.hours\").incr()\n        finalMinDuration\n      } else {\n        target.params(PushFeatureSwitchParams.MinDurationSincePushParam)\n      }\n    statsReceiver\n      .scope(\"minDurationsSincePushWithoutUsingPushCap\").counter(\n        s\"$minDurationSincePush.hours\").incr()\n    minDurationSincePush\n  }\n\n  def getMinDurationSincePush(\n    target: Target,\n    statsReceiver: StatsReceiver\n  ): Future[Duration] = {\n    val minDurationStats: StatsReceiver = statsReceiver.scope(\"pushcapMinDuration_stats\")\n    val minDurationModifierCalculator =\n      MinDurationModifierCalculator()\n    val openedPushByHourAggregatedFut =\n      if (target.params(PushFeatureSwitchParams.EnableQueryUserOpenedHistory))\n        target.openedPushByHourAggregated\n      else Future.None\n    Future\n      .join(\n        target.dynamicPushcap,\n        target.accountCountryCode,\n        openedPushByHourAggregatedFut\n      )\n      .map {\n        case (dynamicPushcapOpt, countryCodeOpt, openedPushByHourAggregated) =>\n          val minDurationSincePush: Duration = {\n            val isGraduallyRampingUpResurrected = target.isResurrectedUser && target.params(\n              PushFeatureSwitchParams.EnableGraduallyRampUpNotification)\n            if (isGraduallyRampingUpResurrected || target.params(\n                PushFeatureSwitchParams.EnableExplicitPushCap)) {\n              getMinDurationsSincePushWithoutUsingPushCap(target)(minDurationStats)\n            } else {\n              dynamicPushcapOpt match {\n                case Some(pushcapInfo) =>\n                  pushcapInfo.fatigueMinutes match {\n                    case Some(fatigueMinutes) => (fatigueMinutes / 60).hours\n                    case _ if pushcapInfo.pushcap > 0 => (24 / pushcapInfo.pushcap).hours\n                    case _ => getMinDurationsSincePushWithoutUsingPushCap(target)(minDurationStats)\n                  }\n                case _ =>\n                  getMinDurationsSincePushWithoutUsingPushCap(target)(minDurationStats)\n              }\n            }\n          }\n\n          val modifiedMinDurationSincePush =\n            if (target.params(PushFeatureSwitchParams.EnableMinDurationModifier)) {\n              val modifierHourOpt =\n                minDurationModifierCalculator.getMinDurationModifier(\n                  target,\n                  countryCodeOpt,\n                  statsReceiver.scope(\"MinDuration\"))\n              modifierHourOpt match {\n                case Some(modifierHour) => modifierHour.hours\n                case _ => minDurationSincePush\n              }\n            } else if (target.params(\n                PushFeatureSwitchParams.EnableMinDurationModifierByUserHistory)) {\n              val modifierMinuteOpt =\n                minDurationModifierCalculator.getMinDurationModifierByUserOpenedHistory(\n                  target,\n                  openedPushByHourAggregated,\n                  statsReceiver.scope(\"MinDuration\"))\n\n              modifierMinuteOpt match {\n                case Some(modifierMinute) => modifierMinute.minutes\n                case _ => minDurationSincePush\n              }\n            } else minDurationSincePush\n\n          target.finalPushcapAndFatigue += \"pushFatigue\" -> PushCapInfo(\n            \"pushFatigue\",\n            modifiedMinDurationSincePush.inHours.toByte)\n\n          minDurationStats\n            .stat(\"minDurationSincePushValueStats\").add(modifiedMinDurationSincePush.inHours)\n          minDurationStats\n            .scope(\"minDurationSincePushValueCount\").counter(\n              s\"$modifiedMinDurationSincePush\").incr()\n\n          modifiedMinDurationSincePush\n      }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/PushDeviceUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.frigate.common.store.deviceinfo.DeviceInfo\nimport com.twitter.frigate.common.store.deviceinfo.MobileClientType\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.util.Future\nimport com.twitter.finagle.stats.NullStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\n\nobject PushDeviceUtil {\n\n  def isPrimaryDeviceAndroid(deviceInfoOpt: Option[DeviceInfo]): Boolean = {\n    deviceInfoOpt.exists {\n      _.guessedPrimaryClient.exists { clientType =>\n        (clientType == MobileClientType.Android) || (clientType == MobileClientType.AndroidLite)\n      }\n    }\n  }\n\n  def isPrimaryDeviceIOS(deviceInfoOpt: Option[DeviceInfo]): Boolean = {\n    deviceInfoOpt.exists {\n      _.guessedPrimaryClient.exists { clientType =>\n        (clientType == MobileClientType.Iphone) || (clientType == MobileClientType.Ipad)\n      }\n    }\n  }\n\n  def isPushRecommendationsEligible(target: Target): Future[Boolean] =\n    target.deviceInfo.map(_.exists(_.isRecommendationsEligible))\n\n  def isTopicsEligible(\n    target: Target,\n    statsReceiver: StatsReceiver = NullStatsReceiver\n  ): Future[Boolean] = {\n    val isTopicsSkipFatigue = Future.True\n\n    Future.join(isTopicsSkipFatigue, target.deviceInfo.map(_.exists(_.isTopicsEligible))).map {\n      case (isTopicsNotFatigue, isTopicsEligibleSetting) =>\n        isTopicsNotFatigue && isTopicsEligibleSetting\n    }\n  }\n\n  def isSpacesEligible(target: Target): Future[Boolean] =\n    target.deviceInfo.map(_.exists(_.isSpacesEligible))\n\n  def isNtabOnlyEligible: Future[Boolean] = {\n    Future.False\n  }\n\n  def isRecommendationsEligible(target: Target): Future[Boolean] = {\n    Future.join(isPushRecommendationsEligible(target), isNtabOnlyEligible).map {\n      case (isPushRecommendation, isNtabOnly) => isPushRecommendation || isNtabOnly\n      case _ => false\n    }\n  }\n\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/PushIbisUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.util.Future\n\nobject PushIbisUtil {\n\n  def getSocialContextModelValues(socialContextUserIds: Seq[Long]): Map[String, String] = {\n\n    val socialContextSize = socialContextUserIds.size\n\n    val (displaySocialContexts, otherCount) = {\n      if (socialContextSize < 3) (socialContextUserIds, 0)\n      else (socialContextUserIds.take(1), socialContextSize - 1)\n    }\n\n    val usersValue = displaySocialContexts.map(_.toString).mkString(\",\")\n\n    if (otherCount > 0) Map(\"social_users\" -> s\"$usersValue+$otherCount\")\n    else Map(\"social_users\" -> usersValue)\n  }\n\n  def mergeFutModelValues(\n    mvFut1: Future[Map[String, String]],\n    mvFut2: Future[Map[String, String]]\n  ): Future[Map[String, String]] = {\n    Future.join(mvFut1, mvFut2).map {\n      case (mv1, mv2) => mv1 ++ mv2\n    }\n  }\n\n  def mergeModelValues(\n    mvFut1: Future[Map[String, String]],\n    mv2: Map[String, String]\n  ): Future[Map[String, String]] =\n    mvFut1.map { mv1 => mv1 ++ mv2 }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/PushToHomeUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.deviceinfo.DeviceInfo\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.params.PushFeatureSwitchParams\nobject PushToHomeUtil {\n  def getIbis2ModelValue(\n    deviceInfoOpt: Option[DeviceInfo],\n    target: Target,\n    stats: StatsReceiver\n  ): Option[Map[String, String]] = {\n    deviceInfoOpt.flatMap { deviceInfo =>\n      val isAndroidEnabled = deviceInfo.isLandOnHomeAndroid && target.params(\n        PushFeatureSwitchParams.EnableTweetPushToHomeAndroid)\n      val isIOSEnabled = deviceInfo.isLandOnHomeiOS && target.params(\n        PushFeatureSwitchParams.EnableTweetPushToHomeiOS)\n      if (isAndroidEnabled || isIOSEnabled) {\n        stats.counter(\"enable_push_to_home\").incr()\n        Some(Map(\"is_land_on_home\" -> \"true\"))\n      } else None\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/RFPHTakeStepUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.Invalid\nimport com.twitter.frigate.common.base.OK\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.refresh_handler.ResultWithDebugInfo\nimport com.twitter.frigate.pushservice.predicate.BigFilteringEpsilonGreedyExplorationPredicate\nimport com.twitter.frigate.pushservice.predicate.MlModelsHoldbackExperimentPredicate\nimport com.twitter.frigate.pushservice.take.candidate_validator.RFPHCandidateValidator\nimport com.twitter.frigate.pushservice.thriftscala.PushStatus\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.util.Future\n\nclass RFPHTakeStepUtil()(globalStats: StatsReceiver) {\n\n  implicit val statsReceiver: StatsReceiver =\n    globalStats.scope(\"RefreshForPushHandler\")\n  private val takeStats: StatsReceiver = statsReceiver.scope(\"take\")\n  private val notifierStats = takeStats.scope(\"notifier\")\n  private val validatorStats = takeStats.scope(\"validator\")\n  private val validatorLatency: Stat = validatorStats.stat(\"latency\")\n\n  private val executedPredicatesInTandem: Counter =\n    takeStats.counter(\"predicates_executed_in_tandem\")\n\n  private val bigFilteringEpsGreedyPredicate: NamedPredicate[PushCandidate] =\n    BigFilteringEpsilonGreedyExplorationPredicate()(takeStats)\n  private val bigFilteringEpsGreedyStats: StatsReceiver =\n    takeStats.scope(\"big_filtering_eps_greedy_predicate\")\n\n  private val modelPredicate: NamedPredicate[PushCandidate] =\n    MlModelsHoldbackExperimentPredicate()(takeStats)\n  private val mlPredicateStats: StatsReceiver = takeStats.scope(\"ml_predicate\")\n\n  private def updateFilteredStatusExptStats(candidate: PushCandidate, predName: String): Unit = {\n\n    val recTypeStat = globalStats.scope(\n      candidate.commonRecType.toString\n    )\n\n    recTypeStat.counter(PushStatus.Filtered.toString).incr()\n    recTypeStat\n      .scope(PushStatus.Filtered.toString)\n      .counter(predName)\n      .incr()\n  }\n\n  def isCandidateValid(\n    candidate: PushCandidate,\n    candidateValidator: RFPHCandidateValidator\n  ): Future[ResultWithDebugInfo] = {\n    val predResultFuture = Stat.timeFuture(validatorLatency) {\n      Future\n        .join(\n          bigFilteringEpsGreedyPredicate.apply(Seq(candidate)),\n          modelPredicate.apply(Seq(candidate))\n        ).flatMap {\n          case (Seq(true), Seq(true)) =>\n            executedPredicatesInTandem.incr()\n\n            bigFilteringEpsGreedyStats\n              .scope(candidate.commonRecType.toString)\n              .counter(\"passed\")\n              .incr()\n\n            mlPredicateStats\n              .scope(candidate.commonRecType.toString)\n              .counter(\"passed\")\n              .incr()\n            candidateValidator.validateCandidate(candidate).map((_, Nil))\n          case (Seq(false), _) =>\n            bigFilteringEpsGreedyStats\n              .scope(candidate.commonRecType.toString)\n              .counter(\"filtered\")\n              .incr()\n            Future.value((Some(bigFilteringEpsGreedyPredicate), Nil))\n          case (_, _) =>\n            mlPredicateStats\n              .scope(candidate.commonRecType.toString)\n              .counter(\"filtered\")\n              .incr()\n            Future.value((Some(modelPredicate), Nil))\n        }\n    }\n\n    predResultFuture.map {\n      case (Some(pred: NamedPredicate[_]), candPredicateResults) =>\n        takeStats.counter(\"filtered_by_named_general_predicate\").incr()\n        updateFilteredStatusExptStats(candidate, pred.name)\n        ResultWithDebugInfo(\n          Invalid(Some(pred.name)),\n          candPredicateResults\n        )\n\n      case (Some(_), candPredicateResults) =>\n        takeStats.counter(\"filtered_by_unnamed_general_predicate\").incr()\n        updateFilteredStatusExptStats(candidate, predName = \"unk\")\n        ResultWithDebugInfo(\n          Invalid(Some(\"unnamed_candidate_predicate\")),\n          candPredicateResults\n        )\n\n      case (None, candPredicateResults) =>\n        takeStats.counter(\"accepted_push_ok\").incr()\n        ResultWithDebugInfo(\n          OK,\n          candPredicateResults\n        )\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/RelationshipUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.frigate.common.base.TweetAuthor\nimport com.twitter.frigate.pushservice.model.PushTypes.RawCandidate\nimport com.twitter.hermit.predicate.socialgraph.Edge\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.socialgraph.thriftscala.RelationshipType\n\n/**\n * This class provides utility functions for relationshipEdge for each Candidate type.\n */\nobject RelationshipUtil {\n\n  /**\n   * Form relationEdges\n   * @param candidate PushCandidate\n   * @param relationship relationshipTypes for different candidate types\n   * @return relationEdges for different candidate types\n   */\n  private def formRelationEdgeWithTargetIdAndAuthorId(\n    candidate: RawCandidate,\n    relationship: List[RelationshipType with Product]\n  ): List[RelationEdge] = {\n    candidate match {\n      case candidate: RawCandidate with TweetAuthor =>\n        candidate.authorId match {\n          case Some(authorId) =>\n            val edge = Edge(candidate.target.targetId, authorId)\n            for {\n              r <- relationship\n            } yield RelationEdge(edge, r)\n          case _ => List.empty[RelationEdge]\n        }\n      case _ => List.empty[RelationEdge]\n    }\n  }\n\n  /**\n   * Form all relationshipEdges for basicTweetRelationShips\n   * @param candidate PushCandidate\n   * @return List of relationEdges for basicTweetRelationShips\n   */\n  def getBasicTweetRelationships(candidate: RawCandidate): List[RelationEdge] = {\n    val relationship = List(\n      RelationshipType.DeviceFollowing,\n      RelationshipType.Blocking,\n      RelationshipType.BlockedBy,\n      RelationshipType.HideRecommendations,\n      RelationshipType.Muting)\n    formRelationEdgeWithTargetIdAndAuthorId(candidate, relationship)\n  }\n\n  /**\n   * Form all relationshipEdges for F1tweetsRelationships\n   * @param candidate PushCandidate\n   * @return List of relationEdges for F1tweetsRelationships\n   */\n  def getPreCandidateRelationshipsForInNetworkTweets(\n    candidate: RawCandidate\n  ): List[RelationEdge] = {\n    val relationship = List(RelationshipType.Following)\n    getBasicTweetRelationships(candidate) ++ formRelationEdgeWithTargetIdAndAuthorId(\n      candidate,\n      relationship)\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/ResponseStatsTrackUtils.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.BroadcastStatsReceiver\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.thriftscala.PushResponse\nimport com.twitter.frigate.pushservice.thriftscala.PushStatus\nimport com.twitter.frigate.thriftscala.CommonRecommendationType\n\nobject ResponseStatsTrackUtils {\n  def trackStatsForResponseToRequest(\n    crt: CommonRecommendationType,\n    target: Target,\n    response: PushResponse,\n    receivers: Seq[StatsReceiver]\n  )(\n    originalStats: StatsReceiver\n  ): Unit = {\n    val newReceivers = Seq(\n      originalStats\n        .scope(\"is_model_training_data\")\n        .scope(target.isModelTrainingData.toString),\n      originalStats.scope(\"scribe_target\").scope(IbisScribeTargets.crtToScribeTarget(crt))\n    )\n\n    val broadcastStats = BroadcastStatsReceiver(receivers)\n    val broadcastStatsWithExpts = BroadcastStatsReceiver(newReceivers ++ receivers)\n\n    if (response.status == PushStatus.Sent) {\n      if (target.isModelTrainingData) {\n        broadcastStats.counter(\"num_training_data_recs_sent\").incr()\n      }\n    }\n    broadcastStatsWithExpts.counter(response.status.toString).incr()\n    if (response.status == PushStatus.Filtered) {\n      broadcastStats\n        .scope(response.status.toString)\n        .counter(response.filteredBy.getOrElse(\"None\"))\n        .incr()\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/SendHandlerPredicateUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.CandidateDetails\nimport com.twitter.frigate.common.base.CandidateResult\nimport com.twitter.frigate.common.base.Invalid\nimport com.twitter.frigate.common.base.OK\nimport com.twitter.frigate.common.base.Result\nimport com.twitter.frigate.pushservice.model.PushTypes.PushCandidate\nimport com.twitter.frigate.pushservice.refresh_handler.ResultWithDebugInfo\nimport com.twitter.frigate.pushservice.take.candidate_validator.SendHandlerPostCandidateValidator\nimport com.twitter.frigate.pushservice.take.candidate_validator.SendHandlerPreCandidateValidator\nimport com.twitter.frigate.pushservice.thriftscala.PushStatus\nimport com.twitter.hermit.predicate.NamedPredicate\nimport com.twitter.util.Future\n\nclass SendHandlerPredicateUtil()(globalStats: StatsReceiver) {\n  implicit val statsReceiver: StatsReceiver =\n    globalStats.scope(\"SendHandler\")\n  private val validateStats: StatsReceiver = statsReceiver.scope(\"validate\")\n\n  private def updateFilteredStatusExptStats(candidate: PushCandidate, predName: String): Unit = {\n\n    val recTypeStat = globalStats.scope(\n      candidate.commonRecType.toString\n    )\n\n    recTypeStat.counter(PushStatus.Filtered.toString).incr()\n    recTypeStat\n      .scope(PushStatus.Filtered.toString)\n      .counter(predName)\n      .incr()\n  }\n\n  /**\n   * Parsing the candidateValidtor result into desired format for preValidation before ml filtering\n   * @param hydratedCandidates\n   * @param candidateValidator\n   * @return\n   */\n  def preValidationForCandidate(\n    hydratedCandidates: Seq[CandidateDetails[PushCandidate]],\n    candidateValidator: SendHandlerPreCandidateValidator\n  ): Future[\n    (Seq[CandidateDetails[PushCandidate]], Seq[CandidateResult[PushCandidate, Result]])\n  ] = {\n    val predResultFuture =\n      Future.collect(\n        hydratedCandidates.map(hydratedCandidate =>\n          candidateValidator.validateCandidate(hydratedCandidate.candidate))\n      )\n\n    predResultFuture.map { results =>\n      results\n        .zip(hydratedCandidates)\n        .foldLeft(\n          (\n            Seq.empty[CandidateDetails[PushCandidate]],\n            Seq.empty[CandidateResult[PushCandidate, Result]]\n          )\n        ) {\n          case ((goodCandidates, filteredCandidates), (result, candidateDetails)) =>\n            result match {\n              case None =>\n                (goodCandidates :+ candidateDetails, filteredCandidates)\n              case Some(pred: NamedPredicate[_]) =>\n                val r = Invalid(Some(pred.name))\n                (\n                  goodCandidates,\n                  filteredCandidates :+ CandidateResult[PushCandidate, Result](\n                    candidateDetails.candidate,\n                    candidateDetails.source,\n                    r\n                  )\n                )\n              case Some(_) =>\n                val r = Invalid(Some(\"Filtered by un-named predicate\"))\n                (\n                  goodCandidates,\n                  filteredCandidates :+ CandidateResult[PushCandidate, Result](\n                    candidateDetails.candidate,\n                    candidateDetails.source,\n                    r\n                  )\n                )\n            }\n        }\n    }\n  }\n\n  /**\n   * Parsing the candidateValidtor result into desired format for postValidation including and after ml filtering\n   * @param candidate\n   * @param candidateValidator\n   * @return\n   */\n  def postValidationForCandidate(\n    candidate: PushCandidate,\n    candidateValidator: SendHandlerPostCandidateValidator\n  ): Future[ResultWithDebugInfo] = {\n    val predResultFuture =\n      candidateValidator.validateCandidate(candidate)\n\n    predResultFuture.map {\n      case (Some(pred: NamedPredicate[_])) =>\n        validateStats.counter(\"filtered_by_named_general_predicate\").incr()\n        updateFilteredStatusExptStats(candidate, pred.name)\n        ResultWithDebugInfo(\n          Invalid(Some(pred.name)),\n          Nil\n        )\n\n      case Some(_) =>\n        validateStats.counter(\"filtered_by_unnamed_general_predicate\").incr()\n        updateFilteredStatusExptStats(candidate, predName = \"unk\")\n        ResultWithDebugInfo(\n          Invalid(Some(\"unnamed_candidate_predicate\")),\n          Nil\n        )\n\n      case _ =>\n        validateStats.counter(\"accepted_push_ok\").incr()\n        ResultWithDebugInfo(\n          OK,\n          Nil\n        )\n    }\n  }\n}\n"
  },
  {
    "path": "pushservice/src/main/scala/com/twitter/frigate/pushservice/util/TopicsUtil.scala",
    "content": "package com.twitter.frigate.pushservice.util\n\nimport com.twitter.contentrecommender.thriftscala.DisplayLocation\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.frigate.common.base.TargetUser\nimport com.twitter.frigate.common.predicate.CommonOutNetworkTweetCandidatesSourcePredicates.authorNotBeingFollowedPredicate\nimport com.twitter.frigate.common.store.interests.InterestsLookupRequestWithContext\nimport com.twitter.frigate.pushservice.model.PushTypes.Target\nimport com.twitter.frigate.pushservice.model.PushTypes\nimport com.twitter.frigate.pushservice.store.UttEntityHydrationQuery\nimport com.twitter.frigate.pushservice.store.UttEntityHydrationStore\nimport com.twitter.hermit.predicate.Predicate\nimport com.twitter.hermit.predicate.socialgraph.RelationEdge\nimport com.twitter.interests.thriftscala.InterestRelationType\nimport com.twitter.interests.thriftscala.InterestRelationship\nimport com.twitter.interests.thriftscala.InterestedInInterestLookupContext\nimport com.twitter.interests.thriftscala.InterestedInInterestModel\nimport com.twitter.interests.thriftscala.ProductId\nimport com.twitter.interests.thriftscala.UserInterest\nimport com.twitter.interests.thriftscala.UserInterestData\nimport com.twitter.interests.thriftscala.UserInterests\nimport com.twitter.interests.thriftscala.{TopicListingViewerContext => TopicListingViewerContextCR}\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.timelines.configapi.Param\nimport com.twitter.topiclisting.TopicListingViewerContext\nimport com.twitter.topiclisting.utt.LocalizedEntity\nimport com.twitter.tsp.thriftscala.TopicListingSetting\nimport com.twitter.tsp.thriftscala.TopicSocialProofRequest\nimport com.twitter.tsp.thriftscala.TopicSocialProofResponse\nimport com.twitter.tsp.thriftscala.TopicWithScore\nimport com.twitter.util.Future\nimport scala.collection.Map\n\ncase class TweetWithTopicProof(\n  tweetId: Long,\n  topicId: Long,\n  authorId: Option[Long],\n  score: Double,\n  tweetyPieResult: TweetyPieResult,\n  topicListingSetting: String,\n  algorithmCR: Option[String],\n  isOON: Boolean)\n\nobject TopicsUtil {\n\n  /**\n   * Obtains the Localized Entities for the provided SC Entity IDs\n   * @param target                  The target user for which we're obtaining candidates\n   * @param semanticCoreEntityIds   The seq. of entity ids for which we would like to obtain the Localized Entities\n   * @param uttEntityHydrationStore Store to query the actual LocalizedEntities\n   * @return                        A Future Map consisting of the entity id as the key and LocalizedEntity as the value\n   */\n  def getLocalizedEntityMap(\n    target: Target,\n    semanticCoreEntityIds: Set[Long],\n    uttEntityHydrationStore: UttEntityHydrationStore\n  ): Future[Map[Long, LocalizedEntity]] = {\n    buildTopicListingViewerContext(target)\n      .flatMap { topicListingViewerContext =>\n        val query = UttEntityHydrationQuery(topicListingViewerContext, semanticCoreEntityIds.toSeq)\n        val localizedTopicEntitiesFut =\n          uttEntityHydrationStore.getLocalizedTopicEntities(query).map(_.flatten)\n        localizedTopicEntitiesFut.map { localizedTopicEntities =>\n          localizedTopicEntities.map { localizedTopicEntity =>\n            localizedTopicEntity.entityId -> localizedTopicEntity\n          }.toMap\n        }\n      }\n  }\n\n  /**\n   * Fetch explict followed interests i.e Topics for targetUser\n   *\n   * @param targetUser: [[Target]] object representing a user eligible for MagicRecs notification\n   * @return: list of all Topics(Interests) Followed by targetUser\n   */\n  def getTopicsFollowedByUser(\n    targetUser: Target,\n    interestsWithLookupContextStore: ReadableStore[\n      InterestsLookupRequestWithContext,\n      UserInterests\n    ],\n    followedTopicsStats: Stat\n  ): Future[Option[Seq[UserInterest]]] = {\n    buildTopicListingViewerContext(targetUser).flatMap { topicListingViewerContext =>\n      // explicit interests relation query\n      val explicitInterestsLookupRequest = InterestsLookupRequestWithContext(\n        targetUser.targetId,\n        Some(\n          InterestedInInterestLookupContext(\n            explicitContext = None,\n            inferredContext = None,\n            productId = Some(ProductId.Followable),\n            topicListingViewerContext = Some(topicListingViewerContext.toThrift),\n            disableExplicit = None,\n            disableImplicit = Some(true)\n          )\n        )\n      )\n\n      // filter explicit follow relationships from response\n      interestsWithLookupContextStore.get(explicitInterestsLookupRequest).map {\n        _.flatMap { userInterests =>\n          val followedTopics = userInterests.interests.map {\n            _.filter {\n              case UserInterest(_, Some(interestData)) =>\n                interestData match {\n                  case UserInterestData.InterestedIn(interestedIn) =>\n                    interestedIn.exists {\n                      case InterestedInInterestModel.ExplicitModel(explicitModel) =>\n                        explicitModel match {\n                          case InterestRelationship.V1(v1) =>\n                            v1.relation == InterestRelationType.Followed\n\n                          case _ => false\n                        }\n\n                      case _ => false\n                    }\n\n                  case _ => false\n                }\n\n              case _ => false // interestData unavailable\n            }\n          }\n          followedTopicsStats.add(followedTopics.getOrElse(Seq.empty[UserInterest]).size)\n          followedTopics\n        }\n      }\n    }\n  }\n\n  /**\n   *\n   * @param target : [[Target]] object respresenting MagicRecs user\n   *\n   * @return: [[TopicListingViewerContext]] for querying topics\n   */\n  def buildTopicListingViewerContext(target: Target): Future[TopicListingViewerContext] = {\n    Future.join(target.inferredUserDeviceLanguage, target.countryCode, target.targetUser).map {\n      case (inferredLanguage, countryCode, userInfo) =>\n        TopicListingViewerContext(\n          userId = Some(target.targetId),\n          guestId = None,\n          deviceId = None,\n          clientApplicationId = None,\n          userAgent = None,\n          languageCode = inferredLanguage,\n          countryCode = countryCode,\n          userRoles = userInfo.flatMap(_.roles.map(_.roles.toSet))\n        )\n    }\n  }\n\n  /**\n   *\n   * @param target : [[Target]] object respresenting MagicRecs user\n   *\n   * @return: [[TopicListingViewerContext]] for querying topics\n   */\n  def buildTopicListingViewerContextForCR(target: Target): Future[TopicListingViewerContextCR] = {\n    TopicsUtil.buildTopicListingViewerContext(target).map(_.toThrift)\n  }\n\n  /**\n   *\n   * @param target : [[Target]] object respresenting MagicRecs user\n   * @param tweets : [[Seq[TweetyPieResult]]] object representing Tweets to get TSP for\n   * @param topicSocialProofServiceStore: [[ReadableStore[TopicSocialProofRequest, TopicSocialProofResponse]]]\n   * @param edgeStore: [[ReadableStore[RelationEdge, Boolean]]]]\n   *\n   * @return: [[Future[Seq[TweetWithTopicProof]]]] Tweets with topic proof\n   */\n  def getTopicSocialProofs(\n    inputTarget: Target,\n    tweets: Seq[TweetyPieResult],\n    topicSocialProofServiceStore: ReadableStore[TopicSocialProofRequest, TopicSocialProofResponse],\n    edgeStore: ReadableStore[RelationEdge, Boolean],\n    scoreThresholdParam: Param[Double]\n  ): Future[Seq[TweetWithTopicProof]] = {\n    buildTopicListingViewerContextForCR(inputTarget).flatMap { topicListingContext =>\n      val tweetIds: Set[Long] = tweets.map(_.tweet.id).toSet\n      val tweetIdsToTweetyPie = tweets.map(tp => tp.tweet.id -> tp).toMap\n      val topicSocialProofRequest =\n        TopicSocialProofRequest(\n          inputTarget.targetId,\n          tweetIds,\n          DisplayLocation.MagicRecsRecommendTopicTweets,\n          TopicListingSetting.Followable,\n          topicListingContext)\n\n      topicSocialProofServiceStore\n        .get(topicSocialProofRequest).flatMap {\n          case Some(topicSocialProofResponse) =>\n            val topicProofCandidates = topicSocialProofResponse.socialProofs.collect {\n              case (tweetId, topicsWithScore)\n                  if topicsWithScore.nonEmpty && topicsWithScore\n                    .maxBy(_.score).score >= inputTarget\n                    .params(scoreThresholdParam) =>\n                // Get the topic with max score if there are any topics returned\n                val topicWithScore = topicsWithScore.maxBy(_.score)\n                TweetWithTopicProof(\n                  tweetId,\n                  topicWithScore.topicId,\n                  tweetIdsToTweetyPie(tweetId).tweet.coreData.map(_.userId),\n                  topicWithScore.score,\n                  tweetIdsToTweetyPie(tweetId),\n                  topicWithScore.topicFollowType.map(_.name).getOrElse(\"\"),\n                  topicWithScore.algorithmType.map(_.name),\n                  isOON = true\n                )\n            }.toSeq\n\n            hydrateTopicProofCandidatesWithEdgeStore(inputTarget, topicProofCandidates, edgeStore)\n          case _ => Future.value(Seq.empty[TweetWithTopicProof])\n        }\n    }\n  }\n\n  /**\n   * Obtain TopicWithScores for provided tweet candidates and target\n   * @param target   target user\n   * @param Tweets   tweet candidates represented in a (tweetId, TweetyPieResult) map\n   * @param topicSocialProofServiceStore store to query topic social proof\n   * @param enableTopicAnnotation whether to enable topic annotation\n   * @param topicScoreThreshold  threshold for topic score\n   * @return a (tweetId, TopicWithScore) map where the topic with highest topic score (if exists) is chosen\n   */\n  def getTopicsWithScoreMap(\n    target: PushTypes.Target,\n    Tweets: Map[Long, Option[TweetyPieResult]],\n    topicSocialProofServiceStore: ReadableStore[TopicSocialProofRequest, TopicSocialProofResponse],\n    enableTopicAnnotation: Boolean,\n    topicScoreThreshold: Double\n  ): Future[Option[Map[Long, TopicWithScore]]] = {\n\n    if (enableTopicAnnotation) {\n      TopicsUtil\n        .buildTopicListingViewerContextForCR(target).flatMap { topicListingContext =>\n          val tweetIds = Tweets.keySet\n          val topicSocialProofRequest =\n            TopicSocialProofRequest(\n              target.targetId,\n              tweetIds,\n              DisplayLocation.MagicRecsRecommendTopicTweets,\n              TopicListingSetting.Followable,\n              topicListingContext)\n\n          topicSocialProofServiceStore\n            .get(topicSocialProofRequest).map {\n              _.map { topicSocialProofResponse =>\n                topicSocialProofResponse.socialProofs\n                  .collect {\n                    case (tweetId, topicsWithScore)\n                        if topicsWithScore.nonEmpty && Tweets(tweetId).nonEmpty\n                          && topicsWithScore.maxBy(_.score).score >= topicScoreThreshold =>\n                      tweetId -> topicsWithScore.maxBy(_.score)\n                  }\n\n              }\n            }\n        }\n    } else {\n      Future.None\n    }\n\n  }\n\n  /**\n   * Obtain LocalizedEntities for provided tweet candidates and target\n   * @param target target user\n   * @param Tweets tweet candidates represented in a (tweetId, TweetyPieResult) map\n   * @param uttEntityHydrationStore store to query the actual LocalizedEntities\n   * @param topicSocialProofServiceStore store to query topic social proof\n   * @param enableTopicAnnotation whether to enable topic annotation\n   * @param topicScoreThreshold threshold for topic score\n   * @return a (tweetId, LocalizedEntity Option) Future map that stores Localized Entity (can be empty) for given tweetId\n   */\n  def getTweetIdLocalizedEntityMap(\n    target: PushTypes.Target,\n    Tweets: Map[Long, Option[TweetyPieResult]],\n    uttEntityHydrationStore: UttEntityHydrationStore,\n    topicSocialProofServiceStore: ReadableStore[TopicSocialProofRequest, TopicSocialProofResponse],\n    enableTopicAnnotation: Boolean,\n    topicScoreThreshold: Double\n  ): Future[Map[Long, Option[LocalizedEntity]]] = {\n\n    val topicWithScoreMap = getTopicsWithScoreMap(\n      target,\n      Tweets,\n      topicSocialProofServiceStore,\n      enableTopicAnnotation,\n      topicScoreThreshold)\n\n    topicWithScoreMap.flatMap { topicWithScores =>\n      topicWithScores match {\n        case Some(topics) =>\n          val topicIds = topics.collect { case (_, topic) => topic.topicId }.toSet\n          val LocalizedEntityMapFut =\n            getLocalizedEntityMap(target, topicIds, uttEntityHydrationStore)\n\n          LocalizedEntityMapFut.map { LocalizedEntityMap =>\n            topics.map {\n              case (tweetId, topic) =>\n                tweetId -> LocalizedEntityMap.get(topic.topicId)\n            }\n          }\n        case _ => Future.value(Map[Long, Option[LocalizedEntity]]())\n      }\n    }\n\n  }\n\n  /**\n   * Hydrate TweetWithTopicProof candidates with isOON field info,\n   * based on the following relationship between target user and candidate author in edgeStore\n   * @return TweetWithTopicProof candidates with isOON field populated\n   */\n  def hydrateTopicProofCandidatesWithEdgeStore(\n    inputTarget: TargetUser,\n    topicProofCandidates: Seq[TweetWithTopicProof],\n    edgeStore: ReadableStore[RelationEdge, Boolean],\n  ): Future[Seq[TweetWithTopicProof]] = {\n    // IDs of all authors of TopicProof candidates that are OON with respect to inputTarget\n    val validOONAuthorIdsFut =\n      Predicate.filter(\n        topicProofCandidates.flatMap(_.authorId).distinct,\n        authorNotBeingFollowedPredicate(inputTarget, edgeStore))\n\n    validOONAuthorIdsFut.map { validOONAuthorIds =>\n      topicProofCandidates.map(candidate => {\n        candidate.copy(isOON =\n          candidate.authorId.isDefined && validOONAuthorIds.contains(candidate.authorId.get))\n      })\n    }\n  }\n\n}\n"
  },
  {
    "path": "recos-injector/BUILD.bazel",
    "content": "# This prevents SQ query from grabbing //:all since it traverses up once to find a BUILD (DPB-14048)\n"
  },
  {
    "path": "recos-injector/CONFIG.ini",
    "content": "; See http://go/CONFIG.ini\n\n[jira]\nproject: SD\n\n[docbird]\nproject_name = recos-injector\n\n[kite]\nproject: recos-injector\n"
  },
  {
    "path": "recos-injector/README.md",
    "content": "# Recos-Injector\n\nRecos-Injector is a streaming event processor used to build input streams for GraphJet-based services. It is a general-purpose tool that consumes arbitrary incoming event streams (e.g., Fav, RT, Follow, client_events, etc.), applies filtering, and combines and publishes cleaned up events to corresponding GraphJet services. Each GraphJet-based service subscribes to a dedicated Kafka topic, and Recos-Injector enables GraphJet-based services to consume any event they want.\n\n## How to run Recos-Injector server tests\n\nYou can run tests by using the following command from your project's root directory:\n\n    $ bazel build recos-injector/...\n    $ bazel test recos-injector/...\n\n## How to run recos-injector-server in development on a local machine\n\nThe simplest way to stand up a service is to run it locally. To run\nrecos-injector-server in development mode, compile the project and then\nexecute it with `bazel run`:\n\n    $ bazel build recos-injector/server:bin\n    $ bazel run recos-injector/server:bin\n\nA tunnel can be set up in order for downstream queries to work properly.\nUpon successful server startup, try to `curl` its admin endpoint in another\nterminal:\n\n    $ curl -s localhost:9990/admin/ping\n    pong\n\nRun `curl -s localhost:9990/admin` to see a list of all available admin endpoints.\n\n## Querying Recos-Injector server from a Scala console\n\nRecos-Injector does not have a Thrift endpoint. Instead, it reads Event Bus and Kafka queues and writes to the Recos-Injector Kafka.\n\n## Generating a package for deployment\n\nTo package your service into a zip file for deployment, run:\n\n    $ bazel bundle recos-injector/server:bin --bundle-jvm-archive=zip\n\nIf the command is successful, a file named `dist/recos-injector-server.zip` will be created.\n"
  },
  {
    "path": "recos-injector/server/BUILD",
    "content": "target(\n    name = \"server\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector\",\n    ],\n)\n\ntest_suite(\n    name = \"tests\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"recos-injector/server/src/test/scala/com/twitter/recosinjector\",\n    ],\n)\n\njvm_binary(\n    name = \"bin\",\n    basename = \"recos-injector-server\",\n    main = \"com.twitter.recosinjector.Main\",\n    platform = \"java11\",\n    runtime_platform = \"java11\",\n    tags = [\n        \"bazel-compatible:migrated\",\n    ],\n    dependencies = [\n        \":server\",\n        \"3rdparty/jvm/org/slf4j:slf4j-jdk14\",\n    ],\n)\n\njvm_app(\n    name = \"bundle\",\n    basename = \"recos-injector\",\n    binary = \":bin\",\n    bundles = [bundle(\n        fileset = [\"config/*\"],\n        owning_target = \"recos-injector/server/config:files\",\n    )],\n    tags = [\n        \"bazel-compatible:migrated\",\n    ],\n)\n"
  },
  {
    "path": "recos-injector/server/config/BUILD",
    "content": "resources(\n    sources = [\n        \"!*.pyc\",\n        \"!BUILD*\",\n        \"*\",\n    ],\n    tags = [\"bazel-compatible\"],\n)\n\n# Created for Bazel compatibility.\n# In Bazel, loose files must be part of a target to be included into a bundle.\n# See also http://go/bazel-compatibility/bundle_does_not_match_any_files\nfiles(\n    name = \"files\",\n    sources = [\n        \"!BUILD\",\n        \"**/*\",\n    ],\n    tags = [\"bazel-compatible\"],\n)\n"
  },
  {
    "path": "recos-injector/server/config/change_log_config.ini",
    "content": "[Configs]\nDCS = all\nROLE = recos-injector\nJOB = recos-injector\nENV = prod\nPACKAGE = recos-injector-release\nPATH = recos-injector\n"
  },
  {
    "path": "recos-injector/server/config/decider.yml",
    "content": "tweet_event_transformer_user_tweet_entity_edges:\n  comment: \"Enables the generation of UserTweetEntity edges in tweet event transformer\"\n  default_availability: 0\n\nenable_emit_tweet_edge_from_reply:\n  comment: \"Decides when processing a Reply edge, whether to generate a Tweet edge for it as well\"\n  default_availability: 0\n\nenable_unfavorite_edge:\n  comment: \"Decides when processing a UnfavoriteEvent from Timeline events, whether to process unfav edges\"\n  default_availability: 0\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/BUILD",
    "content": "scala_library(\n    platform = \"java11\",\n    strict_deps = False,\n    tags = [\n        \"bazel-compatible\",\n    ],\n    dependencies = [\n        \"3rdparty/jvm/io/netty:netty4-tcnative-boringssl-static\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"eventbus/client\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/client\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/server\",\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"finagle/finagle-stats\",\n        \"finagle/finagle-thriftmux\",\n        \"recos-injector/server/config\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/clients\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/config\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/decider\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/edges\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/event_processors\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/publishers\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/uua_processors\",\n        \"src/thrift/com/twitter/clientapp/gen:clientapp-scala\",\n        \"src/thrift/com/twitter/gizmoduck:user-thrift-scala\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala\",\n        \"src/thrift/com/twitter/tweetypie:events-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n        \"thrift-web-forms\",\n        \"twitter-server-internal\",\n        \"twitter-server/server/src/main/scala\",\n        \"twitter-server/slf4j-jdk14/src/main/scala/com/twitter/server/logging\",\n        \"util/util-app\",\n        \"util/util-logging/src/main/scala\",\n        \"util/util-stats/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/Main.scala",
    "content": "package com.twitter.recosinjector\n\nimport com.twitter.app.Flag\nimport com.twitter.finagle.http.HttpMuxer\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.ElfOwlFilter\nimport com.twitter.recosinjector.clients.Gizmoduck\nimport com.twitter.recosinjector.clients.RecosHoseEntitiesCache\nimport com.twitter.recosinjector.clients.SocialGraph\nimport com.twitter.recosinjector.clients.Tweetypie\nimport com.twitter.recosinjector.clients.UrlResolver\nimport com.twitter.recosinjector.config._\nimport com.twitter.recosinjector.edges.SocialWriteEventToUserUserGraphBuilder\nimport com.twitter.recosinjector.edges.TimelineEventToUserTweetEntityGraphBuilder\nimport com.twitter.recosinjector.edges.TweetEventToUserTweetEntityGraphBuilder\nimport com.twitter.recosinjector.edges.TweetEventToUserUserGraphBuilder\nimport com.twitter.recosinjector.edges.UnifiedUserActionToUserVideoGraphBuilder\nimport com.twitter.recosinjector.edges.UnifiedUserActionToUserAdGraphBuilder\nimport com.twitter.recosinjector.edges.UnifiedUserActionToUserTweetGraphPlusBuilder\nimport com.twitter.recosinjector.edges.UserTweetEntityEdgeBuilder\nimport com.twitter.recosinjector.event_processors.SocialWriteEventProcessor\nimport com.twitter.recosinjector.event_processors.TimelineEventProcessor\nimport com.twitter.recosinjector.event_processors.TweetEventProcessor\nimport com.twitter.recosinjector.publishers.KafkaEventPublisher\nimport com.twitter.recosinjector.uua_processors.UnifiedUserActionProcessor\nimport com.twitter.recosinjector.uua_processors.UnifiedUserActionsConsumer\nimport com.twitter.server.logging.{Logging => JDK14Logging}\nimport com.twitter.server.Deciderable\nimport com.twitter.server.TwitterServer\nimport com.twitter.socialgraph.thriftscala.WriteEvent\nimport com.twitter.timelineservice.thriftscala.{Event => TimelineEvent}\nimport com.twitter.tweetypie.thriftscala.TweetEvent\nimport com.twitter.util.Await\nimport com.twitter.util.Duration\nimport java.util.concurrent.TimeUnit\n\nobject Main extends TwitterServer with JDK14Logging with Deciderable { self =>\n\n  implicit val stats: StatsReceiver = statsReceiver\n\n  private val dataCenter: Flag[String] = flag(\"service.cluster\", \"atla\", \"Data Center\")\n  private val serviceRole: Flag[String] = flag(\"service.role\", \"Service Role\")\n  private val serviceEnv: Flag[String] = flag(\"service.env\", \"Service Env\")\n  private val serviceName: Flag[String] = flag(\"service.name\", \"Service Name\")\n  private val shardId = flag(\"shardId\", 0, \"Shard ID\")\n  private val numShards = flag(\"numShards\", 1, \"Number of shards for this service\")\n  private val truststoreLocation =\n    flag[String](\"truststore_location\", \"\", \"Truststore file location\")\n\n  def main(): Unit = {\n    val serviceIdentifier = ServiceIdentifier(\n      role = serviceRole(),\n      service = serviceName(),\n      environment = serviceEnv(),\n      zone = dataCenter()\n    )\n    println(\"ServiceIdentifier = \" + serviceIdentifier.toString)\n    log.info(\"ServiceIdentifier = \" + serviceIdentifier.toString)\n\n    val shard = shardId()\n    val numOfShards = numShards()\n    val environment = serviceEnv()\n\n    implicit val config: DeployConfig = {\n      environment match {\n        case \"prod\" => ProdConfig(serviceIdentifier)(stats)\n        case \"staging\" | \"devel\" => StagingConfig(serviceIdentifier)\n        case env => throw new Exception(s\"Unknown environment $env\")\n      }\n    }\n\n    // Initialize the config and wait for initialization to finish\n    Await.ready(config.init())\n\n    log.info(\n      \"Starting Recos Injector: environment %s, clientId %s\",\n      environment,\n      config.recosInjectorThriftClientId\n    )\n    log.info(\"Starting shard Id: %d of %d shards...\".format(shard, numOfShards))\n\n    // Client wrappers\n    val cache = new RecosHoseEntitiesCache(config.recosInjectorCoreSvcsCacheClient)\n    val gizmoduck = new Gizmoduck(config.userStore)\n    val socialGraph = new SocialGraph(config.socialGraphIdStore)\n    val tweetypie = new Tweetypie(config.tweetyPieStore)\n    val urlResolver = new UrlResolver(config.urlInfoStore)\n\n    // Edge builders\n    val userTweetEntityEdgeBuilder = new UserTweetEntityEdgeBuilder(cache, urlResolver)\n\n    // Publishers\n    val kafkaEventPublisher = KafkaEventPublisher(\n      \"/s/kafka/recommendations:kafka-tls\",\n      config.outputKafkaTopicPrefix,\n      config.recosInjectorThriftClientId,\n      truststoreLocation())\n\n    // Message Builders\n    val socialWriteToUserUserMessageBuilder =\n      new SocialWriteEventToUserUserGraphBuilder()(\n        statsReceiver.scope(\"SocialWriteEventToUserUserGraphBuilder\")\n      )\n\n    val timelineToUserTweetEntityMessageBuilder = new TimelineEventToUserTweetEntityGraphBuilder(\n      userTweetEntityEdgeBuilder = userTweetEntityEdgeBuilder\n    )(statsReceiver.scope(\"TimelineEventToUserTweetEntityGraphBuilder\"))\n\n    val tweetEventToUserTweetEntityGraphBuilder = new TweetEventToUserTweetEntityGraphBuilder(\n      userTweetEntityEdgeBuilder = userTweetEntityEdgeBuilder,\n      tweetCreationStore = config.tweetCreationStore,\n      decider = config.recosInjectorDecider\n    )(statsReceiver.scope(\"TweetEventToUserTweetEntityGraphBuilder\"))\n\n    val socialWriteEventProcessor = new SocialWriteEventProcessor(\n      eventBusStreamName = s\"recos_injector_social_write_event_$environment\",\n      thriftStruct = WriteEvent,\n      serviceIdentifier = serviceIdentifier,\n      kafkaEventPublisher = kafkaEventPublisher,\n      userUserGraphTopic = KafkaEventPublisher.UserUserTopic,\n      userUserGraphMessageBuilder = socialWriteToUserUserMessageBuilder\n    )(statsReceiver.scope(\"SocialWriteEventProcessor\"))\n\n    val tweetToUserUserMessageBuilder = new TweetEventToUserUserGraphBuilder()(\n      statsReceiver.scope(\"TweetEventToUserUserGraphBuilder\")\n    )\n\n    val unifiedUserActionToUserVideoGraphBuilder = new UnifiedUserActionToUserVideoGraphBuilder(\n      userTweetEntityEdgeBuilder = userTweetEntityEdgeBuilder\n    )(statsReceiver.scope(\"UnifiedUserActionToUserVideoGraphBuilder\"))\n\n    val unifiedUserActionToUserAdGraphBuilder = new UnifiedUserActionToUserAdGraphBuilder(\n      userTweetEntityEdgeBuilder = userTweetEntityEdgeBuilder\n    )(statsReceiver.scope(\"UnifiedUserActionToUserAdGraphBuilder\"))\n\n    val unifiedUserActionToUserTweetGraphPlusBuilder =\n      new UnifiedUserActionToUserTweetGraphPlusBuilder(\n        userTweetEntityEdgeBuilder = userTweetEntityEdgeBuilder\n      )(statsReceiver.scope(\"UnifiedUserActionToUserTweetGraphPlusBuilder\"))\n\n    // Processors\n    val tweetEventProcessor = new TweetEventProcessor(\n      eventBusStreamName = s\"recos_injector_tweet_events_$environment\",\n      thriftStruct = TweetEvent,\n      serviceIdentifier = serviceIdentifier,\n      userUserGraphMessageBuilder = tweetToUserUserMessageBuilder,\n      userUserGraphTopic = KafkaEventPublisher.UserUserTopic,\n      userTweetEntityGraphMessageBuilder = tweetEventToUserTweetEntityGraphBuilder,\n      userTweetEntityGraphTopic = KafkaEventPublisher.UserTweetEntityTopic,\n      kafkaEventPublisher = kafkaEventPublisher,\n      socialGraph = socialGraph,\n      tweetypie = tweetypie,\n      gizmoduck = gizmoduck\n    )(statsReceiver.scope(\"TweetEventProcessor\"))\n\n    val timelineEventProcessor = new TimelineEventProcessor(\n      eventBusStreamName = s\"recos_injector_timeline_events_prototype_$environment\",\n      thriftStruct = TimelineEvent,\n      serviceIdentifier = serviceIdentifier,\n      kafkaEventPublisher = kafkaEventPublisher,\n      userTweetEntityGraphTopic = KafkaEventPublisher.UserTweetEntityTopic,\n      userTweetEntityGraphMessageBuilder = timelineToUserTweetEntityMessageBuilder,\n      decider = config.recosInjectorDecider,\n      gizmoduck = gizmoduck,\n      tweetypie = tweetypie\n    )(statsReceiver.scope(\"TimelineEventProcessor\"))\n\n    val eventBusProcessors = Seq(\n      timelineEventProcessor,\n      socialWriteEventProcessor,\n      tweetEventProcessor\n    )\n\n    val uuaProcessor = new UnifiedUserActionProcessor(\n      gizmoduck = gizmoduck,\n      tweetypie = tweetypie,\n      kafkaEventPublisher = kafkaEventPublisher,\n      userVideoGraphTopic = KafkaEventPublisher.UserVideoTopic,\n      userVideoGraphBuilder = unifiedUserActionToUserVideoGraphBuilder,\n      userAdGraphTopic = KafkaEventPublisher.UserAdTopic,\n      userAdGraphBuilder = unifiedUserActionToUserAdGraphBuilder,\n      userTweetGraphPlusTopic = KafkaEventPublisher.UserTweetPlusTopic,\n      userTweetGraphPlusBuilder = unifiedUserActionToUserTweetGraphPlusBuilder)(\n      statsReceiver.scope(\"UnifiedUserActionProcessor\"))\n\n    val uuaConsumer = new UnifiedUserActionsConsumer(uuaProcessor, truststoreLocation())\n\n    // Start-up init and graceful shutdown setup\n\n    // wait a bit for services to be ready\n    Thread.sleep(5000L)\n\n    log.info(\"Starting the event processors\")\n    eventBusProcessors.foreach(_.start())\n\n    log.info(\"Starting the uua processors\")\n    uuaConsumer.atLeastOnceProcessor.start()\n\n    this.addAdminRoute(ElfOwlFilter.getPostbackRoute())\n\n    onExit {\n      log.info(\"Shutting down the event processors\")\n      eventBusProcessors.foreach(_.stop())\n      log.info(\"Shutting down the uua processors\")\n      uuaConsumer.atLeastOnceProcessor.close()\n      log.info(\"done exit\")\n    }\n\n    // Wait on the thriftServer so that shutdownTimeout is respected.\n    Await.result(adminHttpServer)\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/clients/BUILD",
    "content": "scala_library(\n    platform = \"java11\",\n    strict_deps = False,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"finagle/finagle-memcached/src/main/scala\",\n        \"finagle/finagle-stats\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/base\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/util\",\n        \"servo/repo/src/main/scala\",\n        \"src/thrift/com/twitter/gizmoduck:thrift-scala\",\n        \"src/thrift/com/twitter/gizmoduck:user-thrift-scala\",\n        \"src/thrift/com/twitter/recos:recos-internal-scala\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"stitch/stitch-tweetypie/src/main/scala\",\n        \"util/util-logging/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/clients/Gizmoduck.scala",
    "content": "package com.twitter.recosinjector.clients\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.logging.Logger\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nclass Gizmoduck(\n  userStore: ReadableStore[Long, User]\n)(\n  implicit statsReceiver: StatsReceiver) {\n  private val log = Logger()\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n\n  def getUser(userId: Long): Future[Option[User]] = {\n    userStore\n      .get(userId)\n      .rescue {\n        case e =>\n          stats.scope(\"getUserFailure\").counter(e.getClass.getSimpleName).incr()\n          log.error(s\"Failed with message ${e.toString}\")\n          Future.None\n      }\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/clients/RecosHoseEntitiesCache.scala",
    "content": "package com.twitter.recosinjector.clients\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.io.Buf\nimport com.twitter.recos.internal.thriftscala.{RecosHoseEntities, RecosHoseEntity}\nimport com.twitter.servo.cache.ThriftSerializer\nimport com.twitter.util.{Duration, Future, Time}\nimport org.apache.thrift.protocol.TBinaryProtocol\n\ncase class CacheEntityEntry(\n  cachePrefix: String,\n  hashedEntityId: Int,\n  entity: String) {\n  val fullKey: String = cachePrefix + hashedEntityId\n}\n\nobject RecosHoseEntitiesCache {\n  val EntityTTL: Duration = 30.hours\n  val EntitiesSerializer =\n    new ThriftSerializer[RecosHoseEntities](RecosHoseEntities, new TBinaryProtocol.Factory())\n\n  val HashtagPrefix: String = \"h\"\n  val UrlPrefix: String = \"u\"\n}\n\n/**\n * A cache layer to store entities.\n * Graph services like user_tweet_entity_graph and user_url_graph store user interactions with\n * entities in a tweet, such as HashTags and URLs. These entities are string values that can be\n * potentially very big. Therefore, we instead store a hashed id in the graph edge, and keep a\n * (hashedId -> entity) mapping in this cache. The actual entity values can be recovered\n * by the graph service at serving time using this cache.\n */\nclass RecosHoseEntitiesCache(client: Client) {\n  import RecosHoseEntitiesCache._\n\n  private def isEntityWithinTTL(entity: RecosHoseEntity, ttlInMillis: Long): Boolean = {\n    entity.timestamp.exists(timestamp => Time.now.inMilliseconds - timestamp <= ttlInMillis)\n  }\n\n  /**\n   * Add a new RecosHoseEntity into RecosHoseEntities\n   */\n  private def updateRecosHoseEntities(\n    existingEntitiesOpt: Option[RecosHoseEntities],\n    newEntityString: String,\n    stats: StatsReceiver\n  ): RecosHoseEntities = {\n    val existingEntities = existingEntitiesOpt.map(_.entities).getOrElse(Nil)\n\n    // Discard expired and duplicate existing entities\n    val validExistingEntities = existingEntities\n      .filter(entity => isEntityWithinTTL(entity, EntityTTL.inMillis))\n      .filter(_.entity != newEntityString)\n\n    val newRecosHoseEntity = RecosHoseEntity(newEntityString, Some(Time.now.inMilliseconds))\n    RecosHoseEntities(validExistingEntities :+ newRecosHoseEntity)\n  }\n\n  private def getRecosHoseEntitiesCache(\n    cacheEntries: Seq[CacheEntityEntry],\n    stats: StatsReceiver\n  ): Future[Map[String, Option[RecosHoseEntities]]] = {\n    client\n      .get(cacheEntries.map(_.fullKey))\n      .map(_.map {\n        case (cacheKey, buf) =>\n          val recosHoseEntitiesTry = EntitiesSerializer.from(Buf.ByteArray.Owned.extract(buf))\n          if (recosHoseEntitiesTry.isThrow) {\n            stats.counter(\"cache_get_deserialization_failure\").incr()\n          }\n          cacheKey -> recosHoseEntitiesTry.toOption\n      })\n      .onSuccess { _ => stats.counter(\"get_cache_success\").incr() }\n      .onFailure { ex =>\n        stats.scope(\"get_cache_failure\").counter(ex.getClass.getSimpleName).incr()\n      }\n  }\n\n  private def putRecosHoseEntitiesCache(\n    cacheKey: String,\n    recosHoseEntities: RecosHoseEntities,\n    stats: StatsReceiver\n  ): Unit = {\n    val serialized = EntitiesSerializer.to(recosHoseEntities)\n    if (serialized.isThrow) {\n      stats.counter(\"cache_put_serialization_failure\").incr()\n    }\n    serialized.toOption.map { bytes =>\n      client\n        .set(cacheKey, 0, EntityTTL.fromNow, Buf.ByteArray.Owned(bytes))\n        .onSuccess { _ => stats.counter(\"put_cache_success\").incr() }\n        .onFailure { ex =>\n          stats.scope(\"put_cache_failure\").counter(ex.getClass.getSimpleName).incr()\n        }\n    }\n  }\n\n  /**\n   * Store a list of new entities into the cache by their cacheKeys, and remove expired/invalid\n   * values in the existing cache entries at the same time\n   */\n  def updateEntitiesCache(\n    newCacheEntries: Seq[CacheEntityEntry],\n    stats: StatsReceiver\n  ): Future[Unit] = {\n    stats.counter(\"update_cache_request\").incr()\n    getRecosHoseEntitiesCache(newCacheEntries, stats)\n      .map { existingCacheEntries =>\n        newCacheEntries.foreach { newCacheEntry =>\n          val fullKey = newCacheEntry.fullKey\n          val existingRecosHoseEntities = existingCacheEntries.get(fullKey).flatten\n          stats.stat(\"num_existing_entities\").add(existingRecosHoseEntities.size)\n          if (existingRecosHoseEntities.isEmpty) {\n            stats.counter(\"existing_entities_empty\").incr()\n          }\n\n          val updatedRecosHoseEntities = updateRecosHoseEntities(\n            existingRecosHoseEntities,\n            newCacheEntry.entity,\n            stats\n          )\n          stats.stat(\"num_updated_entities\").add(updatedRecosHoseEntities.entities.size)\n\n          if (updatedRecosHoseEntities.entities.nonEmpty) {\n            putRecosHoseEntitiesCache(fullKey, updatedRecosHoseEntities, stats)\n          }\n        }\n      }\n      .onSuccess { _ => stats.counter(\"update_cache_success\").incr() }\n      .onFailure { ex =>\n        stats.scope(\"update_cache_failure\").counter(ex.getClass.getSimpleName).incr()\n      }\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/clients/SocialGraph.scala",
    "content": "package com.twitter.recosinjector.clients\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.socialgraph.thriftscala._\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\nclass SocialGraph(\n  socialGraphIdStore: ReadableStore[IdsRequest, IdsResult]\n)(\n  implicit statsReceiver: StatsReceiver) {\n  import SocialGraph._\n  private val log = Logger()\n\n  private val followedByNotMutedByStats = statsReceiver.scope(\"followedByNotMutedBy\")\n\n  private def fetchIdsFromSocialGraph(\n    userId: Long,\n    ids: Seq[Long],\n    relationshipTypes: Map[RelationshipType, Boolean],\n    lookupContext: Option[LookupContext] = IncludeInactiveUnionLookupContext,\n    stats: StatsReceiver\n  ): Future[Seq[Long]] = {\n    if (ids.isEmpty) {\n      stats.counter(\"fetchIdsEmpty\").incr()\n      Future.Nil\n    } else {\n      val relationships = relationshipTypes.map {\n        case (relationshipType, hasRelationship) =>\n          SrcRelationship(\n            source = userId,\n            relationshipType = relationshipType,\n            hasRelationship = hasRelationship,\n            targets = Some(ids)\n          )\n      }.toSeq\n      val idsRequest = IdsRequest(\n        relationships = relationships,\n        pageRequest = SelectAllPageRequest,\n        context = lookupContext\n      )\n      socialGraphIdStore\n        .get(idsRequest)\n        .map { _.map(_.ids).getOrElse(Nil) }\n        .rescue {\n          case e =>\n            stats.scope(\"fetchIdsFailure\").counter(e.getClass.getSimpleName).incr()\n            log.error(s\"Failed with message ${e.toString}\")\n            Future.Nil\n        }\n    }\n  }\n\n  // which of the users in candidates follow userId and have not muted userId\n  def followedByNotMutedBy(userId: Long, candidates: Seq[Long]): Future[Seq[Long]] = {\n    fetchIdsFromSocialGraph(\n      userId,\n      candidates,\n      FollowedByNotMutedRelationships,\n      IncludeInactiveLookupContext,\n      followedByNotMutedByStats\n    )\n  }\n\n}\n\nobject SocialGraph {\n  val SelectAllPageRequest = Some(PageRequest(selectAll = Some(true)))\n\n  val IncludeInactiveLookupContext = Some(LookupContext(includeInactive = true))\n  val IncludeInactiveUnionLookupContext = Some(\n    LookupContext(includeInactive = true, performUnion = Some(true))\n  )\n\n  val FollowedByNotMutedRelationships: Map[RelationshipType, Boolean] = Map(\n    RelationshipType.FollowedBy -> true,\n    RelationshipType.MutedBy -> false\n  )\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/clients/Tweetypie.scala",
    "content": "package com.twitter.recosinjector.clients\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.stitch.tweetypie.TweetyPie.{TweetyPieException, TweetyPieResult}\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.tweetypie.thriftscala.Tweet\nimport com.twitter.util.Future\n\nclass Tweetypie(\n  tweetyPieStore: ReadableStore[Long, TweetyPieResult]\n)(\n  implicit statsReceiver: StatsReceiver) {\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val failureStats = stats.scope(\"getTweetFailure\")\n\n  def getTweet(tweetId: Long): Future[Option[Tweet]] = {\n    tweetyPieStore\n      .get(tweetId)\n      .map { _.map(_.tweet) }\n      .rescue {\n        case e: TweetyPieException =>\n          // Usually results from trying to query a protected or unsafe tweet\n          failureStats.scope(\"TweetyPieException\").counter(e.result.tweetState.toString).incr()\n          Future.None\n        case e =>\n          failureStats.counter(e.getClass.getSimpleName).incr()\n          Future.None\n      }\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/clients/UrlResolver.scala",
    "content": "package com.twitter.recosinjector.clients\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.frigate.common.util.{SnowflakeUtils, UrlInfo}\nimport com.twitter.storehaus.{FutureOps, ReadableStore}\nimport com.twitter.util.{Duration, Future, Timer}\n\nclass UrlResolver(\n  urlInfoStore: ReadableStore[String, UrlInfo]\n)(\n  implicit statsReceiver: StatsReceiver) {\n  private val EmptyFutureMap = Future.value(Map.empty[String, String])\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val twitterResolvedUrlCounter = stats.counter(\"twitterResolvedUrl\")\n  private val resolvedUrlCounter = stats.counter(\"resolvedUrl\")\n  private val noResolvedUrlCounter = stats.counter(\"noResolvedUrl\")\n\n  private val numNoDelayCounter = stats.counter(\"urlResolver_no_delay\")\n  private val numDelayCounter = stats.counter(\"urlResolver_delay\")\n\n  implicit val timer: Timer = DefaultTimer\n\n  /**\n   * Get the resolved URL map of the input raw URLs\n   *\n   * @param rawUrls list of raw URLs to query\n   * @return map of raw URL to resolved URL\n   */\n  def getResolvedUrls(rawUrls: Set[String]): Future[Map[String, String]] = {\n    FutureOps\n      .mapCollect(urlInfoStore.multiGet[String](rawUrls))\n      .map { resolvedUrlsMap =>\n        resolvedUrlsMap.flatMap {\n          case (\n                url,\n                Some(\n                  UrlInfo(\n                    Some(resolvedUrl),\n                    Some(_),\n                    Some(domain),\n                    _,\n                    _,\n                    _,\n                    _,\n                    Some(_),\n                    _,\n                    _,\n                    _,\n                    _))) =>\n            if (domain == \"Twitter\") { // Filter out Twitter based URLs\n              twitterResolvedUrlCounter.incr()\n              None\n            } else {\n              resolvedUrlCounter.incr()\n              Some(url -> resolvedUrl)\n            }\n          case _ =>\n            noResolvedUrlCounter.incr()\n            None\n        }\n      }\n  }\n\n  /**\n   *  Get resolved url maps given a list of urls, grouping urls that point to the same webpage\n   */\n  def getResolvedUrls(urls: Seq[String], tweetId: Long): Future[Map[String, String]] = {\n    if (urls.isEmpty) {\n      EmptyFutureMap\n    } else {\n      Future\n        .sleep(getUrlResolverDelayDuration(tweetId))\n        .before(getResolvedUrls(urls.toSet))\n    }\n  }\n\n  /**\n   * Given a tweet, return the amount of delay needed before attempting to resolve the Urls\n   */\n  private def getUrlResolverDelayDuration(\n    tweetId: Long\n  ): Duration = {\n    val urlResolverDelaySinceCreation = 12.seconds\n    val urlResolverDelayDuration = 4.seconds\n    val noDelay = 0.seconds\n\n    // Check whether the tweet was created more than the specified delay duration before now.\n    // If the tweet ID is not based on Snowflake, this is false, and the delay is applied.\n    val isCreatedBeforeDelayThreshold = SnowflakeUtils\n      .tweetCreationTime(tweetId)\n      .map(_.untilNow)\n      .exists(_ > urlResolverDelaySinceCreation)\n\n    if (isCreatedBeforeDelayThreshold) {\n      numNoDelayCounter.incr()\n      noDelay\n    } else {\n      numDelayCounter.incr()\n      urlResolverDelayDuration\n    }\n  }\n\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/config/BUILD",
    "content": "scala_library(\n    platform = \"java11\",\n    strict_deps = False,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/bijection:scrooge\",\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"abdecider\",\n        \"decider/src/main/scala\",\n        \"finagle/finagle-memcached/src/main/scala\",\n        \"finagle/finagle-stats\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/util\",\n        \"hermit/hermit-core:store\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/gizmoduck\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/tweetypie\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/decider\",\n        \"scribelib/validators/src/main/scala/com/twitter/scribelib/validators\",\n        \"src/scala/com/twitter/storehaus_internal/memcache\",\n        \"src/scala/com/twitter/storehaus_internal/memcache/config\",\n        \"src/scala/com/twitter/storehaus_internal/util\",\n        \"src/thrift/com/twitter/gizmoduck:thrift-java\",\n        \"src/thrift/com/twitter/gizmoduck:thrift-scala\",\n        \"src/thrift/com/twitter/gizmoduck:user-thrift-scala\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"src/thrift/com/twitter/spam/rtf:safety-level-scala\",\n        \"src/thrift/com/twitter/tweetypie:service-scala\",\n        \"stitch/stitch-core\",\n        \"stitch/stitch-socialgraph\",\n        \"stitch/stitch-storehaus/src/main/scala\",\n        \"stitch/stitch-tweetypie/src/main/scala\",\n        \"util/util-hashing/src/main/scala\",\n        \"util/util-logging/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/config/CacheConfig.scala",
    "content": "package com.twitter.recosinjector.config\n\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.storehaus_internal.memcache.MemcacheStore\nimport com.twitter.storehaus_internal.util.{ClientName, ZkEndPoint}\n\ntrait CacheConfig {\n  implicit def statsReceiver: StatsReceiver\n\n  def serviceIdentifier: ServiceIdentifier\n\n  def recosInjectorCoreSvcsCacheDest: String\n\n  val recosInjectorCoreSvcsCacheClient: Client = MemcacheStore.memcachedClient(\n    name = ClientName(\"memcache-recos-injector\"),\n    dest = ZkEndPoint(recosInjectorCoreSvcsCacheDest),\n    statsReceiver = statsReceiver,\n    serviceIdentifier = serviceIdentifier\n  )\n\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/config/Config.scala",
    "content": "package com.twitter.recosinjector.config\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.frigate.common.store.TweetCreationTimeMHStore\nimport com.twitter.frigate.common.util.UrlInfo\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.recosinjector.decider.RecosInjectorDecider\nimport com.twitter.socialgraph.thriftscala.{IdsRequest, IdsResult}\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\ntrait Config { self =>\n  implicit def statsReceiver: StatsReceiver\n\n  // ReadableStores\n  def tweetyPieStore: ReadableStore[Long, TweetyPieResult]\n\n  def userStore: ReadableStore[Long, User]\n\n  def socialGraphIdStore: ReadableStore[IdsRequest, IdsResult]\n\n  def urlInfoStore: ReadableStore[String, UrlInfo]\n\n  // Manhattan stores\n  def tweetCreationStore: TweetCreationTimeMHStore\n\n  // Decider\n  def recosInjectorDecider: RecosInjectorDecider\n\n  // Constants\n  def recosInjectorThriftClientId: ClientId\n\n  def serviceIdentifier: ServiceIdentifier\n\n  def outputKafkaTopicPrefix: String\n\n  def init(): Future[Unit] = Future.Done\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/config/DeployConfig.scala",
    "content": "package com.twitter.recosinjector.config\n\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.client.ClientRegistry\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.TweetCreationTimeMHStore\nimport com.twitter.frigate.common.util.Finagle._\nimport com.twitter.frigate.common.util.{UrlInfo, UrlInfoInjection, UrlResolver}\nimport com.twitter.gizmoduck.thriftscala.{LookupContext, QueryFields, User, UserService}\nimport com.twitter.hermit.store.common.{ObservedCachedReadableStore, ObservedMemcachedReadableStore}\nimport com.twitter.hermit.store.gizmoduck.GizmoduckUserStore\nimport com.twitter.hermit.store.tweetypie.TweetyPieStore\nimport com.twitter.logging.Logger\nimport com.twitter.pink_floyd.thriftscala.{ClientIdentifier, Storer}\nimport com.twitter.socialgraph.thriftscala.{IdsRequest, SocialGraphService}\nimport com.twitter.spam.rtf.thriftscala.SafetyLevel\nimport com.twitter.stitch.socialgraph.SocialGraph\nimport com.twitter.stitch.storehaus.ReadableStoreOfStitch\nimport com.twitter.stitch.tweetypie.TweetyPie.TweetyPieResult\nimport com.twitter.storage.client.manhattan.kv.{\n  ManhattanKVClient,\n  ManhattanKVClientMtlsParams,\n  ManhattanKVEndpointBuilder\n}\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.tweetypie.thriftscala.{GetTweetOptions, TweetService}\nimport com.twitter.util.Future\n\n/*\n * Any finagle clients should not be defined as lazy. If defined lazy,\n * ClientRegistry.expAllRegisteredClientsResolved() call in init will not ensure that the clients\n * are active before thrift endpoint is active. We want the clients to be active, because zookeeper\n * resolution triggered by first request(s) might result in the request(s) failing.\n */\ntrait DeployConfig extends Config with CacheConfig {\n  implicit def statsReceiver: StatsReceiver\n\n  def log: Logger\n\n  // Clients\n  val gizmoduckClient = new UserService.FinagledClient(\n    readOnlyThriftService(\n      \"gizmoduck\",\n      \"/s/gizmoduck/gizmoduck\",\n      statsReceiver,\n      recosInjectorThriftClientId,\n      requestTimeout = 450.milliseconds,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    )\n  )\n  val tweetyPieClient = new TweetService.FinagledClient(\n    readOnlyThriftService(\n      \"tweetypie\",\n      \"/s/tweetypie/tweetypie\",\n      statsReceiver,\n      recosInjectorThriftClientId,\n      requestTimeout = 450.milliseconds,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    )\n  )\n\n  val sgsClient = new SocialGraphService.FinagledClient(\n    readOnlyThriftService(\n      \"socialgraph\",\n      \"/s/socialgraph/socialgraph\",\n      statsReceiver,\n      recosInjectorThriftClientId,\n      requestTimeout = 450.milliseconds,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    )\n  )\n\n  val pinkStoreClient = new Storer.FinagledClient(\n    readOnlyThriftService(\n      \"pink_store\",\n      \"/s/spiderduck/pink-store\",\n      statsReceiver,\n      recosInjectorThriftClientId,\n      requestTimeout = 450.milliseconds,\n      mTLSServiceIdentifier = Some(serviceIdentifier)\n    )\n  )\n\n  // Stores\n  private val _gizmoduckStore = {\n    val queryFields: Set[QueryFields] = Set(\n      QueryFields.Discoverability,\n      QueryFields.Labels,\n      QueryFields.Safety\n    )\n    val context: LookupContext = LookupContext(\n      includeDeactivated = true,\n      safetyLevel = Some(SafetyLevel.Recommendations)\n    )\n\n    GizmoduckUserStore(\n      client = gizmoduckClient,\n      queryFields = queryFields,\n      context = context,\n      statsReceiver = statsReceiver\n    )\n  }\n\n  override val userStore: ReadableStore[Long, User] = {\n    // memcache based cache\n    ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = _gizmoduckStore,\n      cacheClient = recosInjectorCoreSvcsCacheClient,\n      ttl = 2.hours\n    )(\n      valueInjection = BinaryScalaCodec(User),\n      statsReceiver = statsReceiver.scope(\"UserStore\"),\n      keyToString = { k: Long =>\n        s\"usri/$k\"\n      }\n    )\n  }\n\n  /**\n   * TweetyPie store, used to fetch tweet objects when unavailable, and also as a source of\n   * tweet SafetyLevel filtering.\n   * Note: we do NOT cache TweetyPie calls, as it makes tweet SafetyLevel filtering less accurate.\n   * TweetyPie QPS is < 20K/cluster.\n   * More info is here:\n   * https://cgit.twitter.biz/source/tree/src/thrift/com/twitter/spam/rtf/safety_level.thrift\n   */\n  override val tweetyPieStore: ReadableStore[Long, TweetyPieResult] = {\n    val getTweetOptions = Some(\n      GetTweetOptions(\n        includeCards = true,\n        safetyLevel = Some(SafetyLevel.RecosWritePath)\n      )\n    )\n    TweetyPieStore(\n      tweetyPieClient,\n      getTweetOptions,\n      convertExceptionsToNotFound = false // Do not suppress TweetyPie errors. Leave it to caller\n    )\n  }\n\n  private val _urlInfoStore = {\n    //Initialize pink store client, for parsing url\n    UrlResolver(\n      pinkStoreClient,\n      statsReceiver.scope(\"urlFetcher\"),\n      clientId = ClientIdentifier.Recoshose\n    )\n  }\n\n  override val urlInfoStore: ReadableStore[String, UrlInfo] = {\n    // memcache based cache\n    val memcachedStore = ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = _urlInfoStore,\n      cacheClient = recosInjectorCoreSvcsCacheClient,\n      ttl = 2.hours\n    )(\n      valueInjection = UrlInfoInjection,\n      statsReceiver = statsReceiver.scope(\"UrlInfoStore\"),\n      keyToString = { k: String =>\n        s\"uisri/$k\"\n      }\n    )\n\n    ObservedCachedReadableStore.from(\n      memcachedStore,\n      ttl = 1.minutes,\n      maxKeys = 1e5.toInt,\n      windowSize = 10000L,\n      cacheName = \"url_store_in_proc_cache\"\n    )(statsReceiver.scope(\"url_store_in_proc_cache\"))\n  }\n\n  override val socialGraphIdStore = ReadableStoreOfStitch { idsRequest: IdsRequest =>\n    SocialGraph(sgsClient).ids(idsRequest)\n  }\n\n  /**\n   * MH Store for updating the last time user created a tweet\n   */\n  val tweetCreationStore: TweetCreationTimeMHStore = {\n    val client = ManhattanKVClient(\n      appId = \"recos_tweet_creation_info\",\n      dest = \"/s/manhattan/omega.native-thrift\",\n      mtlsParams = ManhattanKVClientMtlsParams(serviceIdentifier)\n    )\n\n    val endpoint = ManhattanKVEndpointBuilder(client)\n      .defaultMaxTimeout(700.milliseconds)\n      .statsReceiver(\n        statsReceiver\n          .scope(serviceIdentifier.zone)\n          .scope(serviceIdentifier.environment)\n          .scope(\"recos_injector_tweet_creation_info_store\")\n      )\n      .build()\n\n    val dataset = if (serviceIdentifier.environment == \"prod\") {\n      \"recos_injector_tweet_creation_info\"\n    } else {\n      \"recos_injector_tweet_creation_info_staging\"\n    }\n\n    new TweetCreationTimeMHStore(\n      cluster = serviceIdentifier.zone,\n      endpoint = endpoint,\n      dataset = dataset,\n      writeTtl = Some(14.days),\n      statsReceiver.scope(\"recos_injector_tweet_creation_info_store\")\n    )\n  }\n\n  // wait for all serversets to populate\n  override def init(): Future[Unit] = ClientRegistry.expAllRegisteredClientsResolved().unit\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/config/ProdConfig.scala",
    "content": "package com.twitter.recosinjector.config\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.logging.Logger\nimport com.twitter.recosinjector.decider.RecosInjectorDecider\n\ncase class ProdConfig(\n  override val serviceIdentifier: ServiceIdentifier\n)(implicit val statsReceiver: StatsReceiver) extends {\n  // Due to trait initialization logic in Scala, any abstract members declared in Config or\n  // DeployConfig should be declared in this block. Otherwise the abstract member might initialize\n  // to null if invoked before before object creation finishing.\n\n  val recosInjectorThriftClientId = ClientId(\"recos-injector.prod\")\n\n  val outputKafkaTopicPrefix = \"recos_injector\"\n\n  val log = Logger(\"ProdConfig\")\n\n  val recosInjectorCoreSvcsCacheDest = \"/srv#/prod/local/cache/recos_metadata\"\n\n  val recosInjectorDecider = RecosInjectorDecider(\n    isProd = true,\n    dataCenter = serviceIdentifier.zone\n  )\n\n} with DeployConfig\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/config/StagingConfig.scala",
    "content": "package com.twitter.recosinjector.config\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.logging.Logger\nimport com.twitter.recosinjector.decider.RecosInjectorDecider\n\ncase class StagingConfig(\n  override val serviceIdentifier: ServiceIdentifier\n)(\n  implicit val statsReceiver: StatsReceiver)\n    extends {\n  // Due to trait initialization logic in Scala, any abstract members declared in Config or\n  // DeployConfig should be declared in this block. Otherwise the abstract member might initialize\n  // to null if invoked before before object creation finishing.\n\n  val recosInjectorThriftClientId = ClientId(\"recos-injector.staging\")\n\n  val outputKafkaTopicPrefix = \"staging_recos_injector\"\n\n  val log = Logger(\"StagingConfig\")\n\n  val recosInjectorCoreSvcsCacheDest = \"/srv#/test/local/cache/twemcache_recos\"\n\n  val recosInjectorDecider = RecosInjectorDecider(\n    isProd = false,\n    dataCenter = serviceIdentifier.zone\n  )\n\n  val abDeciderLoggerNode = \"staging_abdecider_scribe\"\n\n} with DeployConfig\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/decider/BUILD",
    "content": "scala_library(\n    platform = \"java11\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"decider/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/decider/RecosInjectorDecider.scala",
    "content": "package com.twitter.recosinjector.decider\n\nimport com.twitter.decider.{Decider, DeciderFactory, RandomRecipient, Recipient}\n\ncase class RecosInjectorDecider(isProd: Boolean, dataCenter: String) {\n  lazy val decider: Decider = DeciderFactory(\n    Some(\"config/decider.yml\"),\n    Some(getOverlayPath(isProd, dataCenter))\n  )()\n\n  private def getOverlayPath(isProd: Boolean, dataCenter: String): String = {\n    if (isProd) {\n      s\"/usr/local/config/overlays/recos-injector/recos-injector/prod/$dataCenter/decider_overlay.yml\"\n    } else {\n      s\"/usr/local/config/overlays/recos-injector/recos-injector/staging/$dataCenter/decider_overlay.yml\"\n    }\n  }\n\n  def getDecider: Decider = decider\n\n  def isAvailable(feature: String, recipient: Option[Recipient]): Boolean = {\n    decider.isAvailable(feature, recipient)\n  }\n\n  def isAvailable(feature: String): Boolean = isAvailable(feature, Some(RandomRecipient))\n}\n\nobject RecosInjectorDeciderConstants {\n  val TweetEventTransformerUserTweetEntityEdgesDecider =\n    \"tweet_event_transformer_user_tweet_entity_edges\"\n  val EnableEmitTweetEdgeFromReply = \"enable_emit_tweet_edge_from_reply\"\n  val EnableUnfavoriteEdge = \"enable_unfavorite_edge\"\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/BUILD",
    "content": "scala_library(\n    platform = \"java11\",\n    strict_deps = False,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"finagle/finagle-stats\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/predicate/socialgraph\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/clients\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/decider\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/filters\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/publishers\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/util\",\n        \"src/scala/com/twitter/recos/util:recos-util\",\n        \"src/thrift/com/twitter/recos:recos-injector-scala\",\n        \"src/thrift/com/twitter/recos:recos-internal-scala\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala\",\n        \"src/thrift/com/twitter/tweetypie:events-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n    ],\n)\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/Edges.scala",
    "content": "package com.twitter.recosinjector.edges\n\nimport com.twitter.recos.internal.thriftscala.RecosHoseMessage\nimport com.twitter.recos.recos_injector.thriftscala.{Features, UserTweetAuthorGraphMessage}\nimport com.twitter.recos.util.Action.Action\nimport com.twitter.recosinjector.util.TweetDetails\nimport scala.collection.Map\n\ntrait Edge {\n  // RecosHoseMessage is the thrift struct that the graphs consume.\n  def convertToRecosHoseMessage: RecosHoseMessage\n\n  // UserTweetAuthorGraphMessage is the thrift struct that user_tweet_author_graph consumes.\n  def convertToUserTweetAuthorGraphMessage: UserTweetAuthorGraphMessage\n}\n\n/**\n * Edge corresponding to UserTweetEntityEdge.\n * It captures user-tweet interactions: Create, Like, Retweet, Reply etc.\n */\ncase class UserTweetEntityEdge(\n  sourceUser: Long,\n  targetTweet: Long,\n  action: Action,\n  cardInfo: Option[Byte],\n  metadata: Option[Long],\n  entitiesMap: Option[Map[Byte, Seq[Int]]],\n  tweetDetails: Option[TweetDetails])\n    extends Edge {\n\n  override def convertToRecosHoseMessage: RecosHoseMessage = {\n    RecosHoseMessage(\n      leftId = sourceUser,\n      rightId = targetTweet,\n      action = action.id.toByte,\n      card = cardInfo,\n      entities = entitiesMap,\n      edgeMetadata = metadata\n    )\n  }\n\n  private def getFeatures(tweetDetails: TweetDetails): Features = {\n    Features(\n      hasPhoto = Some(tweetDetails.hasPhoto),\n      hasVideo = Some(tweetDetails.hasVideo),\n      hasUrl = Some(tweetDetails.hasUrl),\n      hasHashtag = Some(tweetDetails.hasHashtag)\n    )\n  }\n\n  override def convertToUserTweetAuthorGraphMessage: UserTweetAuthorGraphMessage = {\n    UserTweetAuthorGraphMessage(\n      leftId = sourceUser,\n      rightId = targetTweet,\n      action = action.id.toByte,\n      card = cardInfo,\n      authorId = tweetDetails.flatMap(_.authorId),\n      features = tweetDetails.map(getFeatures)\n    )\n  }\n}\n\n/**\n * Edge corresponding to UserUserGraph.\n * It captures user-user interactions: Follow, Mention, Mediatag.\n */\ncase class UserUserEdge(\n  sourceUser: Long,\n  targetUser: Long,\n  action: Action,\n  metadata: Option[Long])\n    extends Edge {\n  override def convertToRecosHoseMessage: RecosHoseMessage = {\n    RecosHoseMessage(\n      leftId = sourceUser,\n      rightId = targetUser,\n      action = action.id.toByte,\n      edgeMetadata = metadata\n    )\n  }\n\n  override def convertToUserTweetAuthorGraphMessage: UserTweetAuthorGraphMessage = {\n    throw new RuntimeException(\n      \"convertToUserTweetAuthorGraphMessage not implemented in UserUserEdge.\")\n  }\n\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/EventToMessageBuilder.scala",
    "content": "package com.twitter.recosinjector.edges\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.Stats.track\nimport com.twitter.util.Future\n\n/**\n * This is the generic interface that converts incoming Events (ex. TweetEvent, FavEvent, etc)\n * into Edge for a specific output graph. It applies the following flow:\n *\n * event -> update event stats -> build edges -> filter edges\n *\n * Top-level statistics are provided for each step, such as latency and number of events\n */\ntrait EventToMessageBuilder[Event, E <: Edge] {\n  implicit val statsReceiver: StatsReceiver\n\n  private lazy val processEventStats = statsReceiver.scope(\"process_event\")\n  private lazy val numEventsStats = statsReceiver.counter(\"num_process_event\")\n  private lazy val rejectEventStats = statsReceiver.counter(\"num_reject_event\")\n  private lazy val buildEdgesStats = statsReceiver.scope(\"build\")\n  private lazy val numAllEdgesStats = buildEdgesStats.counter(\"num_all_edges\")\n  private lazy val filterEdgesStats = statsReceiver.scope(\"filter\")\n  private lazy val numValidEdgesStats = statsReceiver.counter(\"num_valid_edges\")\n  private lazy val numRecosHoseMessageStats = statsReceiver.counter(\"num_RecosHoseMessage\")\n\n  /**\n   * Given an incoming event, process and convert it into a sequence of RecosHoseMessages\n   * @param event\n   * @return\n   */\n  def processEvent(event: Event): Future[Seq[Edge]] = {\n    track(processEventStats) {\n      shouldProcessEvent(event).flatMap {\n        case true =>\n          numEventsStats.incr()\n          updateEventStatus(event)\n          for {\n            allEdges <- track(buildEdgesStats)(buildEdges(event))\n            filteredEdges <- track(filterEdgesStats)(filterEdges(event, allEdges))\n          } yield {\n            numAllEdgesStats.incr(allEdges.size)\n            numValidEdgesStats.incr(filteredEdges.size)\n            numRecosHoseMessageStats.incr(filteredEdges.size)\n            filteredEdges\n          }\n        case false =>\n          rejectEventStats.incr()\n          Future.Nil\n      }\n    }\n  }\n\n  /**\n   * Pre-process filter that determines whether the given event should be used to build edges.\n   * @param event\n   * @return\n   */\n  def shouldProcessEvent(event: Event): Future[Boolean]\n\n  /**\n   * Update cache/event logging related to the specific event.\n   * By default, no action will be taken. Override when necessary\n   * @param event\n   */\n  def updateEventStatus(event: Event): Unit = {}\n\n  /**\n   * Given an event, extract info and build a sequence of edges\n   * @param event\n   * @return\n   */\n  def buildEdges(event: Event): Future[Seq[E]]\n\n  /**\n   * Given a sequence of edges, filter and return the valid edges\n   * @param event\n   * @param edges\n   * @return\n   */\n  def filterEdges(event: Event, edges: Seq[E]): Future[Seq[E]]\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/SocialWriteEventToUserUserGraphBuilder.scala",
    "content": "package com.twitter.recosinjector.edges\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.util.Action\nimport com.twitter.socialgraph.thriftscala.{\n  Action => SocialGraphAction,\n  FollowGraphEvent,\n  FollowType,\n  WriteEvent\n}\nimport com.twitter.util.Future\n\n/**\n * Converts a WriteEvent to UserUserGraph's messages, including Mention and Mediatag messages\n */\nclass SocialWriteEventToUserUserGraphBuilder()(override implicit val statsReceiver: StatsReceiver)\n    extends EventToMessageBuilder[WriteEvent, UserUserEdge] {\n  private val followOrFrictionlessFollowCounter =\n    statsReceiver.counter(\"num_follow_or_frictionless\")\n  private val notFollowOrFrictionlessFollowCounter =\n    statsReceiver.counter(\"num_not_follow_or_frictionless\")\n  private val followEdgeCounter = statsReceiver.counter(\"num_follow_edge\")\n\n  /**\n   * For now, we are only interested in Follow events\n   */\n  override def shouldProcessEvent(event: WriteEvent): Future[Boolean] = {\n    event.action match {\n      case SocialGraphAction.Follow | SocialGraphAction.FrictionlessFollow =>\n        followOrFrictionlessFollowCounter.incr()\n        Future(true)\n      case _ =>\n        notFollowOrFrictionlessFollowCounter.incr()\n        Future(false)\n    }\n  }\n\n  /**\n   * Determine whether a Follow event is valid/error free.\n   */\n  private def isValidFollowEvent(followEvent: FollowGraphEvent): Boolean = {\n    followEvent.followType match {\n      case Some(FollowType.NormalFollow) | Some(FollowType.FrictionlessFollow) =>\n        followEvent.result.validationError.isEmpty\n      case _ =>\n        false\n    }\n  }\n\n  override def buildEdges(event: WriteEvent): Future[Seq[UserUserEdge]] = {\n    val userUserEdges = event.follow\n      .map(_.collect {\n        case followEvent if isValidFollowEvent(followEvent) =>\n          val sourceUserId = followEvent.result.request.source\n          val targetUserId = followEvent.result.request.target\n          followEdgeCounter.incr()\n          UserUserEdge(\n            sourceUserId,\n            targetUserId,\n            Action.Follow,\n            Some(System.currentTimeMillis())\n          )\n      }).getOrElse(Nil)\n    Future(userUserEdges)\n  }\n\n  override def filterEdges(\n    event: WriteEvent,\n    edges: Seq[UserUserEdge]\n  ): Future[Seq[UserUserEdge]] = {\n    Future(edges)\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/TimelineEventToUserTweetEntityGraphBuilder.scala",
    "content": "package com.twitter.recosinjector.edges\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.util.Action\nimport com.twitter.recosinjector.util.TweetFavoriteEventDetails\nimport com.twitter.util.Future\n\nclass TimelineEventToUserTweetEntityGraphBuilder(\n  userTweetEntityEdgeBuilder: UserTweetEntityEdgeBuilder\n)(\n  override implicit val statsReceiver: StatsReceiver)\n    extends EventToMessageBuilder[TweetFavoriteEventDetails, UserTweetEntityEdge] {\n\n  private val numFavEdgeCounter = statsReceiver.counter(\"num_favorite_edge\")\n  private val numUnfavEdgeCounter = statsReceiver.counter(\"num_unfavorite_edge\")\n\n  override def shouldProcessEvent(event: TweetFavoriteEventDetails): Future[Boolean] = {\n    Future(true)\n  }\n\n  override def buildEdges(details: TweetFavoriteEventDetails): Future[Seq[UserTweetEntityEdge]] = {\n    val engagement = details.userTweetEngagement\n    val tweetDetails = engagement.tweetDetails\n\n    val entitiesMapFut = userTweetEntityEdgeBuilder.getEntitiesMapAndUpdateCache(\n      tweetId = engagement.tweetId,\n      tweetDetails = tweetDetails\n    )\n\n    entitiesMapFut\n      .map { entitiesMap =>\n        UserTweetEntityEdge(\n          sourceUser = engagement.engageUserId,\n          targetTweet = engagement.tweetId,\n          action = engagement.action,\n          metadata = engagement.engagementTimeMillis,\n          cardInfo = engagement.tweetDetails.map(_.cardInfo.toByte),\n          entitiesMap = entitiesMap,\n          tweetDetails = tweetDetails\n        )\n      }\n      .map { edge =>\n        edge match {\n          case fav if fav.action == Action.Favorite =>\n            numFavEdgeCounter.incr()\n          case unfav if unfav.action == Action.Unfavorite =>\n            numUnfavEdgeCounter.incr()\n          case _ =>\n        }\n        Seq(edge)\n      }\n  }\n\n  override def filterEdges(\n    event: TweetFavoriteEventDetails,\n    edges: Seq[UserTweetEntityEdge]\n  ): Future[Seq[UserTweetEntityEdge]] = {\n    Future(edges)\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/TimelineEventToUserTweetGraphBuilder.scala",
    "content": "package com.twitter.recosinjector.edges\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.util.Action\nimport com.twitter.recosinjector.util.TweetFavoriteEventDetails\nimport com.twitter.util.Future\n\nclass TimelineEventToUserTweetGraphBuilder(\n  userTweetEntityEdgeBuilder: UserTweetEntityEdgeBuilder\n)(\n  override implicit val statsReceiver: StatsReceiver)\n    extends EventToMessageBuilder[TweetFavoriteEventDetails, UserTweetEntityEdge] {\n\n  override def shouldProcessEvent(event: TweetFavoriteEventDetails): Future[Boolean] = {\n    Future(true)\n  }\n\n  override def buildEdges(details: TweetFavoriteEventDetails): Future[Seq[UserTweetEntityEdge]] = {\n    val engagement = details.userTweetEngagement\n\n    engagement.action match {\n      case Action.Favorite =>\n        val tweetDetails = engagement.tweetDetails\n\n        val entitiesMapFut = userTweetEntityEdgeBuilder.getEntitiesMapAndUpdateCache(\n          tweetId = engagement.tweetId,\n          tweetDetails = tweetDetails\n        )\n\n        entitiesMapFut\n          .map { entitiesMap =>\n            UserTweetEntityEdge(\n              sourceUser = engagement.engageUserId,\n              targetTweet = engagement.tweetId,\n              action = engagement.action,\n              metadata = engagement.engagementTimeMillis,\n              cardInfo = engagement.tweetDetails.map(_.cardInfo.toByte),\n              entitiesMap = entitiesMap,\n              tweetDetails = tweetDetails\n            )\n          }\n          .map(Seq(_))\n\n      case _ => Future.Nil\n    }\n  }\n\n  override def filterEdges(\n    event: TweetFavoriteEventDetails,\n    edges: Seq[UserTweetEntityEdge]\n  ): Future[Seq[UserTweetEntityEdge]] = {\n    Future(edges)\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/TweetEventToUserTweetEntityGraphBuilder.scala",
    "content": "package com.twitter.recosinjector.edges\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.TweetCreationTimeMHStore\nimport com.twitter.frigate.common.util.SnowflakeUtils\nimport com.twitter.recos.internal.thriftscala.{RecosUserTweetInfo, TweetType}\nimport com.twitter.recos.util.Action\nimport com.twitter.recosinjector.decider.RecosInjectorDecider\nimport com.twitter.recosinjector.decider.RecosInjectorDeciderConstants\nimport com.twitter.recosinjector.util.TweetCreateEventDetails\nimport com.twitter.util.{Future, Time}\n\nclass TweetEventToUserTweetEntityGraphBuilder(\n  userTweetEntityEdgeBuilder: UserTweetEntityEdgeBuilder,\n  tweetCreationStore: TweetCreationTimeMHStore,\n  decider: RecosInjectorDecider\n)(\n  override implicit val statsReceiver: StatsReceiver)\n    extends EventToMessageBuilder[TweetCreateEventDetails, UserTweetEntityEdge] {\n\n  // TweetCreationStore counters\n  private val lastTweetTimeNotInMh = statsReceiver.counter(\"last_tweet_time_not_in_mh\")\n  private val tweetCreationStoreInserts = statsReceiver.counter(\"tweet_creation_store_inserts\")\n\n  private val numInvalidActionCounter = statsReceiver.counter(\"num_invalid_tweet_action\")\n\n  private val numTweetEdgesCounter = statsReceiver.counter(\"num_tweet_edge\")\n  private val numRetweetEdgesCounter = statsReceiver.counter(\"num_retweet_edge\")\n  private val numReplyEdgesCounter = statsReceiver.counter(\"num_reply_edge\")\n  private val numQuoteEdgesCounter = statsReceiver.counter(\"num_quote_edge\")\n  private val numIsMentionedEdgesCounter = statsReceiver.counter(\"num_isMentioned_edge\")\n  private val numIsMediataggedEdgesCounter = statsReceiver.counter(\"num_isMediatagged_edge\")\n\n  private val numIsDecider = statsReceiver.counter(\"num_decider_enabled\")\n  private val numIsNotDecider = statsReceiver.counter(\"num_decider_not_enabled\")\n\n  override def shouldProcessEvent(event: TweetCreateEventDetails): Future[Boolean] = {\n    val isDecider = decider.isAvailable(\n      RecosInjectorDeciderConstants.TweetEventTransformerUserTweetEntityEdgesDecider\n    )\n    if (isDecider) {\n      numIsDecider.incr()\n      Future(true)\n    } else {\n      numIsNotDecider.incr()\n      Future(false)\n    }\n  }\n\n  /**\n   * Build edges Reply event. Reply event emits 2 edges:\n   * author -> Reply -> SourceTweetId\n   * author -> Tweet -> ReplyId\n   * Do not associate entities in reply tweet to the source tweet\n   */\n  private def buildReplyEdge(event: TweetCreateEventDetails) = {\n    val userTweetEngagement = event.userTweetEngagement\n    val authorId = userTweetEngagement.engageUserId\n\n    val replyEdgeFut = event.sourceTweetDetails\n      .map { sourceTweetDetails =>\n        val sourceTweetId = sourceTweetDetails.tweet.id\n        val sourceTweetEntitiesMapFut = userTweetEntityEdgeBuilder.getEntitiesMapAndUpdateCache(\n          tweetId = sourceTweetId,\n          tweetDetails = Some(sourceTweetDetails)\n        )\n\n        sourceTweetEntitiesMapFut.map { sourceTweetEntitiesMap =>\n          val replyEdge = UserTweetEntityEdge(\n            sourceUser = authorId,\n            targetTweet = sourceTweetId,\n            action = Action.Reply,\n            metadata = Some(userTweetEngagement.tweetId),\n            cardInfo = Some(sourceTweetDetails.cardInfo.toByte),\n            entitiesMap = sourceTweetEntitiesMap,\n            tweetDetails = Some(sourceTweetDetails)\n          )\n          numReplyEdgesCounter.incr()\n          Some(replyEdge)\n        }\n      }.getOrElse(Future.None)\n\n    val tweetCreationEdgeFut =\n      if (decider.isAvailable(RecosInjectorDeciderConstants.EnableEmitTweetEdgeFromReply)) {\n        getAndUpdateLastTweetCreationTime(\n          authorId = authorId,\n          tweetId = userTweetEngagement.tweetId,\n          tweetType = TweetType.Reply\n        ).map { lastTweetTime =>\n          val edge = UserTweetEntityEdge(\n            sourceUser = authorId,\n            targetTweet = userTweetEngagement.tweetId,\n            action = Action.Tweet,\n            metadata = lastTweetTime,\n            cardInfo = userTweetEngagement.tweetDetails.map(_.cardInfo.toByte),\n            entitiesMap = None,\n            tweetDetails = userTweetEngagement.tweetDetails\n          )\n          numTweetEdgesCounter.incr()\n          Some(edge)\n        }\n      } else {\n        Future.None\n      }\n\n    Future.join(replyEdgeFut, tweetCreationEdgeFut).map {\n      case (replyEdgeOpt, tweetCreationEdgeOpt) =>\n        tweetCreationEdgeOpt.toSeq ++ replyEdgeOpt.toSeq\n    }\n  }\n\n  /**\n   * Build a Retweet UTEG edge: author -> RT -> SourceTweetId.\n   */\n  private def buildRetweetEdge(event: TweetCreateEventDetails) = {\n    val userTweetEngagement = event.userTweetEngagement\n    val tweetId = userTweetEngagement.tweetId\n\n    event.sourceTweetDetails\n      .map { sourceTweetDetails =>\n        val sourceTweetId = sourceTweetDetails.tweet.id // Id of the tweet being Retweeted\n        val sourceTweetEntitiesMapFut = userTweetEntityEdgeBuilder.getEntitiesMapAndUpdateCache(\n          tweetId = sourceTweetId,\n          tweetDetails = Some(sourceTweetDetails)\n        )\n\n        sourceTweetEntitiesMapFut.map { sourceTweetEntitiesMap =>\n          val edge = UserTweetEntityEdge(\n            sourceUser = userTweetEngagement.engageUserId,\n            targetTweet = sourceTweetId,\n            action = Action.Retweet,\n            metadata = Some(tweetId), // metadata is the tweetId\n            cardInfo = Some(sourceTweetDetails.cardInfo.toByte),\n            entitiesMap = sourceTweetEntitiesMap,\n            tweetDetails = Some(sourceTweetDetails)\n          )\n          numRetweetEdgesCounter.incr()\n          Seq(edge)\n        }\n      }.getOrElse(Future.Nil)\n  }\n\n  /**\n   * Build edges for a Quote event. Quote tweet emits 2 edges:\n   * 1. A quote social proof: author -> Quote -> SourceTweetId\n   * 2. A tweet creation edge: author -> Tweet -> QuoteTweetId\n   */\n  private def buildQuoteEdges(\n    event: TweetCreateEventDetails\n  ): Future[Seq[UserTweetEntityEdge]] = {\n    val userTweetEngagement = event.userTweetEngagement\n    val tweetId = userTweetEngagement.tweetId\n    val authorId = userTweetEngagement.engageUserId\n\n    // do not associate entities in quote tweet to the source tweet,\n    // but associate entities to quote tweet in tweet creation event\n    val quoteTweetEdgeFut = event.sourceTweetDetails\n      .map { sourceTweetDetails =>\n        val sourceTweetId = sourceTweetDetails.tweet.id // Id of the tweet being quoted\n        val sourceTweetEntitiesMapFut = userTweetEntityEdgeBuilder.getEntitiesMapAndUpdateCache(\n          tweetId = sourceTweetId,\n          tweetDetails = event.sourceTweetDetails\n        )\n\n        sourceTweetEntitiesMapFut.map { sourceTweetEntitiesMap =>\n          val edge = UserTweetEntityEdge(\n            sourceUser = authorId,\n            targetTweet = sourceTweetId,\n            action = Action.Quote,\n            metadata = Some(tweetId), // metadata is tweetId\n            cardInfo = Some(sourceTweetDetails.cardInfo.toByte), // cardInfo of the source tweet\n            entitiesMap = sourceTweetEntitiesMap,\n            tweetDetails = Some(sourceTweetDetails)\n          )\n          numQuoteEdgesCounter.incr()\n          Seq(edge)\n        }\n      }.getOrElse(Future.Nil)\n\n    val tweetCreationEdgeFut = getAndUpdateLastTweetCreationTime(\n      authorId = authorId,\n      tweetId = tweetId,\n      tweetType = TweetType.Quote\n    ).map { lastTweetTime =>\n      val metadata = lastTweetTime\n      val cardInfo = userTweetEngagement.tweetDetails.map(_.cardInfo.toByte)\n      val edge = UserTweetEntityEdge(\n        sourceUser = authorId,\n        targetTweet = tweetId,\n        action = Action.Tweet,\n        metadata = metadata,\n        cardInfo = cardInfo,\n        entitiesMap = None,\n        tweetDetails = userTweetEngagement.tweetDetails\n      )\n      numTweetEdgesCounter.incr()\n      Seq(edge)\n    }\n\n    Future.join(quoteTweetEdgeFut, tweetCreationEdgeFut).map {\n      case (quoteEdge, creationEdge) =>\n        quoteEdge ++ creationEdge\n    }\n  }\n\n  /**\n   * Build edges for a Tweet event. A Tweet emits 3 tyes edges:\n   * 1. A tweet creation edge: author -> Tweet -> TweetId\n   * 2. IsMentioned edges: mentionedUserId -> IsMentioned -> TweetId\n   * 3. IsMediatagged edges: mediataggedUserId -> IsMediatagged -> TweetId\n   */\n  private def buildTweetEdges(event: TweetCreateEventDetails): Future[Seq[UserTweetEntityEdge]] = {\n    val userTweetEngagement = event.userTweetEngagement\n    val tweetDetails = userTweetEngagement.tweetDetails\n    val tweetId = userTweetEngagement.tweetId\n    val authorId = userTweetEngagement.engageUserId\n\n    val cardInfo = tweetDetails.map(_.cardInfo.toByte)\n\n    val entitiesMapFut = userTweetEntityEdgeBuilder.getEntitiesMapAndUpdateCache(\n      tweetId = tweetId,\n      tweetDetails = tweetDetails\n    )\n\n    val lastTweetTimeFut = getAndUpdateLastTweetCreationTime(\n      authorId = authorId,\n      tweetId = tweetId,\n      tweetType = TweetType.Tweet\n    )\n\n    Future.join(entitiesMapFut, lastTweetTimeFut).map {\n      case (entitiesMap, lastTweetTime) =>\n        val tweetCreationEdge = UserTweetEntityEdge(\n          sourceUser = authorId,\n          targetTweet = tweetId,\n          action = Action.Tweet,\n          metadata = lastTweetTime,\n          cardInfo = cardInfo,\n          entitiesMap = entitiesMap,\n          tweetDetails = userTweetEngagement.tweetDetails\n        )\n        numTweetEdgesCounter.incr()\n\n        val isMentionedEdges = event.validMentionUserIds\n          .map(_.map { mentionedUserId =>\n            UserTweetEntityEdge(\n              sourceUser = mentionedUserId,\n              targetTweet = tweetId,\n              action = Action.IsMentioned,\n              metadata = Some(tweetId),\n              cardInfo = cardInfo,\n              entitiesMap = entitiesMap,\n              tweetDetails = userTweetEngagement.tweetDetails\n            )\n          }).getOrElse(Nil)\n        numIsMentionedEdgesCounter.incr(isMentionedEdges.size)\n\n        val isMediataggedEdges = event.validMediatagUserIds\n          .map(_.map { mediataggedUserId =>\n            UserTweetEntityEdge(\n              sourceUser = mediataggedUserId,\n              targetTweet = tweetId,\n              action = Action.IsMediaTagged,\n              metadata = Some(tweetId),\n              cardInfo = cardInfo,\n              entitiesMap = entitiesMap,\n              tweetDetails = userTweetEngagement.tweetDetails\n            )\n          }).getOrElse(Nil)\n        numIsMediataggedEdgesCounter.incr(isMediataggedEdges.size)\n\n        Seq(tweetCreationEdge) ++ isMentionedEdges ++ isMediataggedEdges\n    }\n  }\n\n  /**\n   * For a given user, read the user's last time tweeted from the MH store, and\n   * write the new tweet time into the MH store before returning.\n   * Note this function is async, so the MH write operations will continue to execute on its own.\n   * This might create a read/write race condition, but it's expected.\n   */\n  private def getAndUpdateLastTweetCreationTime(\n    authorId: Long,\n    tweetId: Long,\n    tweetType: TweetType\n  ): Future[Option[Long]] = {\n    val newTweetInfo = RecosUserTweetInfo(\n      authorId,\n      tweetId,\n      tweetType,\n      SnowflakeUtils.tweetCreationTime(tweetId).map(_.inMillis).getOrElse(Time.now.inMillis)\n    )\n\n    tweetCreationStore\n      .get(authorId)\n      .map(_.map { previousTweetInfoSeq =>\n        val lastTweetTime = previousTweetInfoSeq\n          .filter(info => info.tweetType == TweetType.Tweet || info.tweetType == TweetType.Quote)\n          .map(_.tweetTimestamp)\n          .sortBy(-_)\n          .headOption // Fetch the latest time user Tweeted or Quoted\n          .getOrElse(\n            Time.Bottom.inMillis\n          ) // Last tweet time never recorded in MH, default to oldest point in time\n\n        if (lastTweetTime == Time.Bottom.inMillis) lastTweetTimeNotInMh.incr()\n        lastTweetTime\n      })\n      .ensure {\n        tweetCreationStore\n          .put(authorId, newTweetInfo)\n          .onSuccess(_ => tweetCreationStoreInserts.incr())\n          .onFailure { e =>\n            statsReceiver.counter(\"write_failed_with_ex:\" + e.getClass.getName).incr()\n          }\n      }\n  }\n\n  override def buildEdges(event: TweetCreateEventDetails): Future[Seq[UserTweetEntityEdge]] = {\n    val userTweetEngagement = event.userTweetEngagement\n    userTweetEngagement.action match {\n      case Action.Reply =>\n        buildReplyEdge(event)\n      case Action.Retweet =>\n        buildRetweetEdge(event)\n      case Action.Tweet =>\n        buildTweetEdges(event)\n      case Action.Quote =>\n        buildQuoteEdges(event)\n      case _ =>\n        numInvalidActionCounter.incr()\n        Future.Nil\n    }\n\n  }\n\n  override def filterEdges(\n    event: TweetCreateEventDetails,\n    edges: Seq[UserTweetEntityEdge]\n  ): Future[Seq[UserTweetEntityEdge]] = {\n    Future(edges) // No filtering for now. Add more if needed\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/TweetEventToUserTweetGraphBuilder.scala",
    "content": "package com.twitter.recosinjector.edges\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.TweetCreationTimeMHStore\nimport com.twitter.frigate.common.util.SnowflakeUtils\nimport com.twitter.recos.internal.thriftscala.RecosUserTweetInfo\nimport com.twitter.recos.internal.thriftscala.TweetType\nimport com.twitter.recos.util.Action\nimport com.twitter.recosinjector.decider.RecosInjectorDecider\nimport com.twitter.recosinjector.decider.RecosInjectorDeciderConstants\nimport com.twitter.recosinjector.util.TweetCreateEventDetails\nimport com.twitter.util.Future\nimport com.twitter.util.Time\n\nclass TweetEventToUserTweetGraphBuilder(\n  userTweetEntityEdgeBuilder: UserTweetEntityEdgeBuilder,\n  tweetCreationStore: TweetCreationTimeMHStore,\n  decider: RecosInjectorDecider\n)(\n  override implicit val statsReceiver: StatsReceiver)\n    extends EventToMessageBuilder[TweetCreateEventDetails, UserTweetEntityEdge] {\n\n  private val numRetweetEdgesCounter = statsReceiver.counter(\"num_retweet_edge\")\n  private val numIsDecider = statsReceiver.counter(\"num_decider_enabled\")\n  private val numIsNotDecider = statsReceiver.counter(\"num_decider_not_enabled\")\n\n  override def shouldProcessEvent(event: TweetCreateEventDetails): Future[Boolean] = {\n    val isDecider = decider.isAvailable(\n      RecosInjectorDeciderConstants.TweetEventTransformerUserTweetEntityEdgesDecider\n    )\n    if (isDecider) {\n      numIsDecider.incr()\n      Future(true)\n    } else {\n      numIsNotDecider.incr()\n      Future(false)\n    }\n  }\n\n  /**\n   * Build a Retweet edge: author -> RT -> SourceTweetId.\n   */\n  private def buildRetweetEdge(event: TweetCreateEventDetails) = {\n    val userTweetEngagement = event.userTweetEngagement\n    val tweetId = userTweetEngagement.tweetId\n\n    event.sourceTweetDetails\n      .map { sourceTweetDetails =>\n        val sourceTweetId = sourceTweetDetails.tweet.id // Id of the tweet being Retweeted\n        val sourceTweetEntitiesMapFut = userTweetEntityEdgeBuilder.getEntitiesMapAndUpdateCache(\n          tweetId = sourceTweetId,\n          tweetDetails = Some(sourceTweetDetails)\n        )\n\n        sourceTweetEntitiesMapFut.map { sourceTweetEntitiesMap =>\n          val edge = UserTweetEntityEdge(\n            sourceUser = userTweetEngagement.engageUserId,\n            targetTweet = sourceTweetId,\n            action = Action.Retweet,\n            metadata = Some(tweetId), // metadata is the tweetId\n            cardInfo = Some(sourceTweetDetails.cardInfo.toByte),\n            entitiesMap = sourceTweetEntitiesMap,\n            tweetDetails = Some(sourceTweetDetails)\n          )\n          numRetweetEdgesCounter.incr()\n          Seq(edge)\n        }\n      }.getOrElse(Future.Nil)\n  }\n\n  override def buildEdges(event: TweetCreateEventDetails): Future[Seq[UserTweetEntityEdge]] = {\n    val userTweetEngagement = event.userTweetEngagement\n    userTweetEngagement.action match {\n      case Action.Retweet =>\n        buildRetweetEdge(event)\n      case _ =>\n        Future.Nil\n    }\n\n  }\n\n  override def filterEdges(\n    event: TweetCreateEventDetails,\n    edges: Seq[UserTweetEntityEdge]\n  ): Future[Seq[UserTweetEntityEdge]] = {\n    Future(edges) // No filtering for now. Add more if needed\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/TweetEventToUserUserGraphBuilder.scala",
    "content": "package com.twitter.recosinjector.edges\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.util.Action\nimport com.twitter.recosinjector.util.TweetCreateEventDetails\nimport com.twitter.util.Future\n\n/**\n * Given a tweet creation event, parse for UserUserGraph edges. Specifically, when a new tweet is\n * created, extract the valid mentioned and mediatagged users in the tweet and create edges for them\n */\nclass TweetEventToUserUserGraphBuilder(\n)(\n  override implicit val statsReceiver: StatsReceiver)\n    extends EventToMessageBuilder[TweetCreateEventDetails, UserUserEdge] {\n  private val tweetOrQuoteEventCounter = statsReceiver.counter(\"num_tweet_or_quote_event\")\n  private val nonTweetOrQuoteEventCounter = statsReceiver.counter(\"num_non_tweet_or_quote_event\")\n  private val mentionEdgeCounter = statsReceiver.counter(\"num_mention_edge\")\n  private val mediatagEdgeCounter = statsReceiver.counter(\"num_mediatag_edge\")\n\n  override def shouldProcessEvent(event: TweetCreateEventDetails): Future[Boolean] = {\n    // For user interactions, only new tweets and quotes are considered (no replies or retweets)\n    event.userTweetEngagement.action match {\n      case Action.Tweet | Action.Quote =>\n        tweetOrQuoteEventCounter.incr()\n        Future(true)\n      case _ =>\n        nonTweetOrQuoteEventCounter.incr()\n        Future(false)\n    }\n  }\n\n  override def buildEdges(event: TweetCreateEventDetails): Future[Seq[UserUserEdge]] = {\n    val mentionEdges = event.validMentionUserIds\n      .map(_.map { mentionUserId =>\n        UserUserEdge(\n          sourceUser = event.userTweetEngagement.engageUserId,\n          targetUser = mentionUserId,\n          action = Action.Mention,\n          metadata = Some(System.currentTimeMillis())\n        )\n      }).getOrElse(Nil)\n\n    val mediatagEdges = event.validMediatagUserIds\n      .map(_.map { mediatagUserId =>\n        UserUserEdge(\n          sourceUser = event.userTweetEngagement.engageUserId,\n          targetUser = mediatagUserId,\n          action = Action.MediaTag,\n          metadata = Some(System.currentTimeMillis())\n        )\n      }).getOrElse(Nil)\n\n    mentionEdgeCounter.incr(mentionEdges.size)\n    mediatagEdgeCounter.incr(mediatagEdges.size)\n    Future(mentionEdges ++ mediatagEdges)\n  }\n\n  override def filterEdges(\n    event: TweetCreateEventDetails,\n    edges: Seq[UserUserEdge]\n  ): Future[Seq[UserUserEdge]] = {\n    Future(edges)\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/UnifiedUserActionToUserAdGraphBuilder.scala",
    "content": "package com.twitter.recosinjector.edges\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.util.Action\nimport com.twitter.recosinjector.util.UuaEngagementEventDetails\nimport com.twitter.util.Future\n\nclass UnifiedUserActionToUserAdGraphBuilder(\n  userTweetEntityEdgeBuilder: UserTweetEntityEdgeBuilder\n)(\n  override implicit val statsReceiver: StatsReceiver)\n    extends EventToMessageBuilder[UuaEngagementEventDetails, UserTweetEntityEdge] {\n\n  override def shouldProcessEvent(event: UuaEngagementEventDetails): Future[Boolean] = {\n    event.userTweetEngagement.action match {\n      case Action.Click | Action.VideoPlayback75 | Action.Favorite => Future(true)\n      case _ => Future(false)\n    }\n  }\n\n  override def buildEdges(details: UuaEngagementEventDetails): Future[Seq[UserTweetEntityEdge]] = {\n    val engagement = details.userTweetEngagement\n    val tweetDetails = engagement.tweetDetails\n\n    Future.value(\n      Seq(\n        UserTweetEntityEdge(\n          sourceUser = engagement.engageUserId,\n          targetTweet = engagement.tweetId,\n          action = engagement.action,\n          metadata = engagement.engagementTimeMillis,\n          cardInfo = engagement.tweetDetails.map(_.cardInfo.toByte),\n          entitiesMap = None,\n          tweetDetails = tweetDetails\n        )))\n  }\n\n  override def filterEdges(\n    event: UuaEngagementEventDetails,\n    edges: Seq[UserTweetEntityEdge]\n  ): Future[Seq[UserTweetEntityEdge]] = {\n    Future(edges)\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/UnifiedUserActionToUserTweetGraphPlusBuilder.scala",
    "content": "package com.twitter.recosinjector.edges\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.util.Action\nimport com.twitter.recosinjector.util.UuaEngagementEventDetails\nimport com.twitter.util.Future\n\nclass UnifiedUserActionToUserTweetGraphPlusBuilder(\n  userTweetEntityEdgeBuilder: UserTweetEntityEdgeBuilder\n)(\n  override implicit val statsReceiver: StatsReceiver)\n    extends EventToMessageBuilder[UuaEngagementEventDetails, UserTweetEntityEdge] {\n\n  override def shouldProcessEvent(event: UuaEngagementEventDetails): Future[Boolean] = {\n    event.userTweetEngagement.action match {\n      case Action.Click | Action.VideoQualityView => Future(true)\n      case Action.Favorite | Action.Retweet | Action.Share => Future(true)\n      case Action.NotificationOpen | Action.EmailClick => Future(true)\n      case Action.Quote | Action.Reply => Future(true)\n      case Action.TweetNotInterestedIn | Action.TweetNotRelevant | Action.TweetSeeFewer |\n          Action.TweetReport | Action.TweetMuteAuthor | Action.TweetBlockAuthor =>\n        Future(true)\n      case _ => Future(false)\n    }\n  }\n\n  override def buildEdges(details: UuaEngagementEventDetails): Future[Seq[UserTweetEntityEdge]] = {\n    val engagement = details.userTweetEngagement\n    val tweetDetails = engagement.tweetDetails\n\n    Future\n      .value(\n        UserTweetEntityEdge(\n          sourceUser = engagement.engageUserId,\n          targetTweet = engagement.tweetId,\n          action = engagement.action,\n          metadata = engagement.engagementTimeMillis,\n          cardInfo = engagement.tweetDetails.map(_.cardInfo.toByte),\n          entitiesMap = None,\n          tweetDetails = tweetDetails\n        )\n      ).map(Seq(_))\n  }\n\n  override def filterEdges(\n    event: UuaEngagementEventDetails,\n    edges: Seq[UserTweetEntityEdge]\n  ): Future[Seq[UserTweetEntityEdge]] = {\n    Future(edges)\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/UnifiedUserActionToUserVideoGraphBuilder.scala",
    "content": "package com.twitter.recosinjector.edges\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.util.Action\nimport com.twitter.recosinjector.util.UuaEngagementEventDetails\nimport com.twitter.util.Future\n\nclass UnifiedUserActionToUserVideoGraphBuilder(\n  userTweetEntityEdgeBuilder: UserTweetEntityEdgeBuilder\n)(\n  override implicit val statsReceiver: StatsReceiver)\n    extends EventToMessageBuilder[UuaEngagementEventDetails, UserTweetEntityEdge] {\n\n  private val numVideoPlayback50EdgeCounter = statsReceiver.counter(\"num_video_playback50_edge\")\n  private val numUnVideoPlayback50Counter = statsReceiver.counter(\"num_non_video_playback50_edge\")\n\n  override def shouldProcessEvent(event: UuaEngagementEventDetails): Future[Boolean] = {\n    event.userTweetEngagement.action match {\n      case Action.VideoPlayback50 => Future(true)\n      case _ => Future(false)\n    }\n  }\n\n  override def buildEdges(details: UuaEngagementEventDetails): Future[Seq[UserTweetEntityEdge]] = {\n    val engagement = details.userTweetEngagement\n    val tweetDetails = engagement.tweetDetails\n\n    Future\n      .value(\n        UserTweetEntityEdge(\n          sourceUser = engagement.engageUserId,\n          targetTweet = engagement.tweetId,\n          action = engagement.action,\n          metadata = engagement.engagementTimeMillis,\n          cardInfo = engagement.tweetDetails.map(_.cardInfo.toByte),\n          entitiesMap = None,\n          tweetDetails = tweetDetails\n        )\n      ).map { edge =>\n        edge match {\n          case videoPlayback50 if videoPlayback50.action == Action.VideoPlayback50 =>\n            numVideoPlayback50EdgeCounter.incr()\n          case _ =>\n            numUnVideoPlayback50Counter.incr()\n        }\n        Seq(edge)\n      }\n  }\n\n  override def filterEdges(\n    event: UuaEngagementEventDetails,\n    edges: Seq[UserTweetEntityEdge]\n  ): Future[Seq[UserTweetEntityEdge]] = {\n    Future(edges)\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/edges/UserTweetEntityEdgeBuilder.scala",
    "content": "package com.twitter.recosinjector.edges\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.graphjet.algorithms.RecommendationType\nimport com.twitter.recosinjector.clients.CacheEntityEntry\nimport com.twitter.recosinjector.clients.RecosHoseEntitiesCache\nimport com.twitter.recosinjector.clients.UrlResolver\nimport com.twitter.recosinjector.util.TweetDetails\nimport com.twitter.util.Future\nimport scala.collection.Map\nimport scala.util.hashing.MurmurHash3\n\nclass UserTweetEntityEdgeBuilder(\n  cache: RecosHoseEntitiesCache,\n  urlResolver: UrlResolver\n)(\n  implicit val stats: StatsReceiver) {\n\n  def getHashedEntities(entities: Seq[String]): Seq[Int] = {\n    entities.map(MurmurHash3.stringHash)\n  }\n\n  /**\n   * Given the entities and their corresponding hashedIds, store the hashId->entity mapping into a\n   * cache.\n   * This is because UTEG edges only store the hashIds, and relies on the cache values to\n   * recover the actual entities. This allows us to store integer values instead of string in the\n   * edges to save space.\n   */\n  private def storeEntitiesInCache(\n    urlEntities: Seq[String],\n    urlHashIds: Seq[Int]\n  ): Future[Unit] = {\n    val urlCacheEntries = urlHashIds.zip(urlEntities).map {\n      case (hashId, url) =>\n        CacheEntityEntry(RecosHoseEntitiesCache.UrlPrefix, hashId, url)\n    }\n    cache.updateEntitiesCache(\n      newCacheEntries = urlCacheEntries,\n      stats = stats.scope(\"urlCache\")\n    )\n  }\n\n  /**\n   * Return an entity mapping from GraphJet recType -> hash(entity)\n   */\n  private def getEntitiesMap(\n    urlHashIds: Seq[Int]\n  ) = {\n    val entitiesMap = Seq(\n      RecommendationType.URL.getValue.toByte -> urlHashIds\n    ).collect {\n      case (keys, ids) if ids.nonEmpty => keys -> ids\n    }.toMap\n    if (entitiesMap.isEmpty) None else Some(entitiesMap)\n  }\n\n  def getEntitiesMapAndUpdateCache(\n    tweetId: Long,\n    tweetDetails: Option[TweetDetails]\n  ): Future[Option[Map[Byte, Seq[Int]]]] = {\n    val resolvedUrlFut = urlResolver\n      .getResolvedUrls(\n        urls = tweetDetails.flatMap(_.urls).getOrElse(Nil),\n        tweetId = tweetId\n      ).map(_.values.toSeq)\n\n    resolvedUrlFut.map { resolvedUrls =>\n      val urlEntities = resolvedUrls\n      val urlHashIds = getHashedEntities(urlEntities)\n\n      // Async call to cache\n      storeEntitiesInCache(\n        urlEntities = urlEntities,\n        urlHashIds = urlHashIds\n      )\n      getEntitiesMap(urlHashIds)\n    }\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/event_processors/BUILD",
    "content": "scala_library(\n    platform = \"java11\",\n    strict_deps = False,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"eventbus/client\",\n        \"recos-injector/server/config\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/clients\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/config\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/decider\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/edges\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/publishers\",\n        \"src/thrift/com/twitter/clientapp/gen:clientapp-scala\",\n        \"src/thrift/com/twitter/gizmoduck:user-thrift-scala\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala\",\n        \"src/thrift/com/twitter/tweetypie:events-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n    ],\n)\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/event_processors/EventBusProcessor.scala",
    "content": "package com.twitter.recosinjector.event_processors\n\nimport com.twitter.eventbus.client.{EventBusSubscriber, EventBusSubscriberBuilder}\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.logging.Logger\nimport com.twitter.scrooge.{ThriftStruct, ThriftStructCodec}\nimport com.twitter.util.Future\n\n/**\n * Main processor class that handles incoming EventBus events, which take forms of a ThriftStruct.\n * This class is responsible for setting up the EventBus streams, and provides a processEvent()\n * where child classes can decide what to do with incoming events\n */\ntrait EventBusProcessor[Event <: ThriftStruct] {\n  private val log = Logger()\n\n  implicit def statsReceiver: StatsReceiver\n\n  /**\n   * Full name of the EventBus stream this processor listens to\n   */\n  val eventBusStreamName: String\n\n  /**\n   * the thriftStruct definition of the objects passed in from the EventBus streams, such as\n   * TweetEvent, WriteEvent, etc.\n   */\n  val thriftStruct: ThriftStructCodec[Event]\n\n  val serviceIdentifier: ServiceIdentifier\n\n  def processEvent(event: Event): Future[Unit]\n\n  private def getEventBusSubscriberBuilder: EventBusSubscriberBuilder[Event] =\n    EventBusSubscriberBuilder()\n      .subscriberId(eventBusStreamName)\n      .serviceIdentifier(serviceIdentifier)\n      .thriftStruct(thriftStruct)\n      .numThreads(8)\n      .fromAllZones(true) // Receives traffic from all data centers\n      .skipToLatest(false) // Ensures we don't miss out on events during restart\n      .statsReceiver(statsReceiver)\n\n  // lazy val ensures the subscriber is only initialized when start() is called\n  private lazy val eventBusSubscriber = getEventBusSubscriberBuilder.build(processEvent)\n\n  def start(): EventBusSubscriber[Event] = eventBusSubscriber\n\n  def stop(): Unit = {\n    eventBusSubscriber\n      .close()\n      .onSuccess { _ =>\n        log.info(s\"EventBus processor ${this.getClass.getSimpleName} is stopped\")\n      }\n      .onFailure { ex: Throwable =>\n        log.error(ex, s\"Exception while stopping EventBus processor ${this.getClass.getSimpleName}\")\n      }\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/event_processors/SocialWriteEventProcessor.scala",
    "content": "package com.twitter.recosinjector.event_processors\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recosinjector.edges.{EventToMessageBuilder, UserUserEdge}\nimport com.twitter.recosinjector.publishers.KafkaEventPublisher\nimport com.twitter.scrooge.ThriftStructCodec\nimport com.twitter.socialgraph.thriftscala.WriteEvent\nimport com.twitter.util.Future\n\n/**\n * This processor listens to events from social graphs services. In particular, a major use case is\n * to listen to user-user follow events.\n */\nclass SocialWriteEventProcessor(\n  override val eventBusStreamName: String,\n  override val thriftStruct: ThriftStructCodec[WriteEvent],\n  override val serviceIdentifier: ServiceIdentifier,\n  kafkaEventPublisher: KafkaEventPublisher,\n  userUserGraphTopic: String,\n  userUserGraphMessageBuilder: EventToMessageBuilder[WriteEvent, UserUserEdge]\n)(\n  override implicit val statsReceiver: StatsReceiver)\n    extends EventBusProcessor[WriteEvent] {\n\n  override def processEvent(event: WriteEvent): Future[Unit] = {\n    userUserGraphMessageBuilder.processEvent(event).map { edges =>\n      edges.foreach { edge =>\n        kafkaEventPublisher.publish(edge.convertToRecosHoseMessage, userUserGraphTopic)\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/event_processors/TimelineEventProcessor.scala",
    "content": "package com.twitter.recosinjector.event_processors\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.util.Action\nimport com.twitter.recosinjector.clients.Gizmoduck\nimport com.twitter.recosinjector.clients.Tweetypie\nimport com.twitter.recosinjector.decider.RecosInjectorDecider\nimport com.twitter.recosinjector.decider.RecosInjectorDeciderConstants\nimport com.twitter.recosinjector.edges.TimelineEventToUserTweetEntityGraphBuilder\nimport com.twitter.recosinjector.filters.TweetFilter\nimport com.twitter.recosinjector.filters.UserFilter\nimport com.twitter.recosinjector.publishers.KafkaEventPublisher\nimport com.twitter.recosinjector.util.TweetDetails\nimport com.twitter.recosinjector.util.TweetFavoriteEventDetails\nimport com.twitter.recosinjector.util.UserTweetEngagement\nimport com.twitter.scrooge.ThriftStructCodec\nimport com.twitter.timelineservice.thriftscala.FavoriteEvent\nimport com.twitter.timelineservice.thriftscala.UnfavoriteEvent\nimport com.twitter.timelineservice.thriftscala.{Event => TimelineEvent}\nimport com.twitter.util.Future\n\n/**\n * Processor for Timeline events, such as Favorite (liking) tweets\n */\nclass TimelineEventProcessor(\n  override val eventBusStreamName: String,\n  override val thriftStruct: ThriftStructCodec[TimelineEvent],\n  override val serviceIdentifier: ServiceIdentifier,\n  kafkaEventPublisher: KafkaEventPublisher,\n  userTweetEntityGraphTopic: String,\n  userTweetEntityGraphMessageBuilder: TimelineEventToUserTweetEntityGraphBuilder,\n  decider: RecosInjectorDecider,\n  gizmoduck: Gizmoduck,\n  tweetypie: Tweetypie\n)(\n  override implicit val statsReceiver: StatsReceiver)\n    extends EventBusProcessor[TimelineEvent] {\n\n  private val processEventDeciderCounter = statsReceiver.counter(\"num_process_timeline_event\")\n  private val numFavoriteEventCounter = statsReceiver.counter(\"num_favorite_event\")\n  private val numUnFavoriteEventCounter = statsReceiver.counter(\"num_unfavorite_event\")\n  private val numNotFavoriteEventCounter = statsReceiver.counter(\"num_not_favorite_event\")\n\n  private val numSelfFavoriteCounter = statsReceiver.counter(\"num_self_favorite_event\")\n  private val numNullCastTweetCounter = statsReceiver.counter(\"num_null_cast_tweet\")\n  private val numTweetFailSafetyLevelCounter = statsReceiver.counter(\"num_fail_tweetypie_safety\")\n  private val numFavoriteUserUnsafeCounter = statsReceiver.counter(\"num_favorite_user_unsafe\")\n  private val engageUserFilter = new UserFilter(gizmoduck)(statsReceiver.scope(\"engage_user\"))\n  private val tweetFilter = new TweetFilter(tweetypie)\n\n  private val numProcessFavorite = statsReceiver.counter(\"num_process_favorite\")\n  private val numNoProcessFavorite = statsReceiver.counter(\"num_no_process_favorite\")\n\n  private def getFavoriteEventDetails(\n    favoriteEvent: FavoriteEvent\n  ): TweetFavoriteEventDetails = {\n\n    val engagement = UserTweetEngagement(\n      engageUserId = favoriteEvent.userId,\n      engageUser = favoriteEvent.user,\n      action = Action.Favorite,\n      engagementTimeMillis = Some(favoriteEvent.eventTimeMs),\n      tweetId = favoriteEvent.tweetId, // the tweet, or source tweet if target tweet is a retweet\n      tweetDetails = favoriteEvent.tweet.map(TweetDetails) // tweet always exists\n    )\n    TweetFavoriteEventDetails(userTweetEngagement = engagement)\n  }\n\n  private def getUnfavoriteEventDetails(\n    unfavoriteEvent: UnfavoriteEvent\n  ): TweetFavoriteEventDetails = {\n    val engagement = UserTweetEngagement(\n      engageUserId = unfavoriteEvent.userId,\n      engageUser = unfavoriteEvent.user,\n      action = Action.Unfavorite,\n      engagementTimeMillis = Some(unfavoriteEvent.eventTimeMs),\n      tweetId = unfavoriteEvent.tweetId, // the tweet, or source tweet if target tweet is a retweet\n      tweetDetails = unfavoriteEvent.tweet.map(TweetDetails) // tweet always exists\n    )\n    TweetFavoriteEventDetails(userTweetEngagement = engagement)\n  }\n\n  private def shouldProcessFavoriteEvent(event: TweetFavoriteEventDetails): Future[Boolean] = {\n    val engagement = event.userTweetEngagement\n    val engageUserId = engagement.engageUserId\n    val tweetId = engagement.tweetId\n    val authorIdOpt = engagement.tweetDetails.flatMap(_.authorId)\n\n    val isSelfFavorite = authorIdOpt.contains(engageUserId)\n    val isNullCastTweet = engagement.tweetDetails.forall(_.isNullCastTweet)\n    val isEngageUserSafeFut = engageUserFilter.filterByUserId(engageUserId)\n    val isTweetPassSafetyFut = tweetFilter.filterForTweetypieSafetyLevel(tweetId)\n\n    Future.join(isEngageUserSafeFut, isTweetPassSafetyFut).map {\n      case (isEngageUserSafe, isTweetPassSafety) =>\n        if (isSelfFavorite) numSelfFavoriteCounter.incr()\n        if (isNullCastTweet) numNullCastTweetCounter.incr()\n        if (!isEngageUserSafe) numFavoriteUserUnsafeCounter.incr()\n        if (!isTweetPassSafety) numTweetFailSafetyLevelCounter.incr()\n\n        !isSelfFavorite && !isNullCastTweet && isEngageUserSafe && isTweetPassSafety\n    }\n  }\n\n  private def processFavoriteEvent(favoriteEvent: FavoriteEvent): Future[Unit] = {\n    val eventDetails = getFavoriteEventDetails(favoriteEvent)\n    shouldProcessFavoriteEvent(eventDetails).map {\n      case true =>\n        numProcessFavorite.incr()\n        // Convert the event for UserTweetEntityGraph\n        userTweetEntityGraphMessageBuilder.processEvent(eventDetails).map { edges =>\n          edges.foreach { edge =>\n            kafkaEventPublisher.publish(edge.convertToRecosHoseMessage, userTweetEntityGraphTopic)\n          }\n        }\n      case false =>\n        numNoProcessFavorite.incr()\n    }\n  }\n\n  private def processUnFavoriteEvent(unFavoriteEvent: UnfavoriteEvent): Future[Unit] = {\n    if (decider.isAvailable(RecosInjectorDeciderConstants.EnableUnfavoriteEdge)) {\n      val eventDetails = getUnfavoriteEventDetails(unFavoriteEvent)\n      // Convert the event for UserTweetEntityGraph\n      userTweetEntityGraphMessageBuilder.processEvent(eventDetails).map { edges =>\n        edges.foreach { edge =>\n          kafkaEventPublisher.publish(edge.convertToRecosHoseMessage, userTweetEntityGraphTopic)\n        }\n      }\n    } else {\n      Future.Unit\n    }\n  }\n\n  override def processEvent(event: TimelineEvent): Future[Unit] = {\n    processEventDeciderCounter.incr()\n    event match {\n      case TimelineEvent.Favorite(favoriteEvent: FavoriteEvent) =>\n        numFavoriteEventCounter.incr()\n        processFavoriteEvent(favoriteEvent)\n      case TimelineEvent.Unfavorite(unFavoriteEvent: UnfavoriteEvent) =>\n        numUnFavoriteEventCounter.incr()\n        processUnFavoriteEvent(unFavoriteEvent)\n      case _ =>\n        numNotFavoriteEventCounter.incr()\n        Future.Unit\n    }\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/event_processors/TweetEventProcessor.scala",
    "content": "package com.twitter.recosinjector.event_processors\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.SnowflakeUtils\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.recos.util.Action\nimport com.twitter.recos.util.Action.Action\nimport com.twitter.recosinjector.clients.Gizmoduck\nimport com.twitter.recosinjector.clients.SocialGraph\nimport com.twitter.recosinjector.clients.Tweetypie\nimport com.twitter.recosinjector.edges.TweetEventToUserTweetEntityGraphBuilder\nimport com.twitter.recosinjector.edges.TweetEventToUserUserGraphBuilder\nimport com.twitter.recosinjector.filters.TweetFilter\nimport com.twitter.recosinjector.filters.UserFilter\nimport com.twitter.recosinjector.publishers.KafkaEventPublisher\nimport com.twitter.recosinjector.util.TweetCreateEventDetails\nimport com.twitter.recosinjector.util.TweetDetails\nimport com.twitter.recosinjector.util.UserTweetEngagement\nimport com.twitter.scrooge.ThriftStructCodec\nimport com.twitter.tweetypie.thriftscala.Tweet\nimport com.twitter.tweetypie.thriftscala.TweetCreateEvent\nimport com.twitter.tweetypie.thriftscala.TweetEvent\nimport com.twitter.tweetypie.thriftscala.TweetEventData\nimport com.twitter.util.Future\n\n/**\n * Event processor for tweet_events EventBus stream from Tweetypie. This stream provides all the\n * key events related to a new tweet, like Creation, Retweet, Quote Tweet, and Replying.\n * It also carries the entities/metadata information in a tweet, including\n * @ Mention, HashTag, MediaTag, URL, etc.\n */\nclass TweetEventProcessor(\n  override val eventBusStreamName: String,\n  override val thriftStruct: ThriftStructCodec[TweetEvent],\n  override val serviceIdentifier: ServiceIdentifier,\n  userUserGraphMessageBuilder: TweetEventToUserUserGraphBuilder,\n  userUserGraphTopic: String,\n  userTweetEntityGraphMessageBuilder: TweetEventToUserTweetEntityGraphBuilder,\n  userTweetEntityGraphTopic: String,\n  kafkaEventPublisher: KafkaEventPublisher,\n  socialGraph: SocialGraph,\n  gizmoduck: Gizmoduck,\n  tweetypie: Tweetypie\n)(\n  override implicit val statsReceiver: StatsReceiver)\n    extends EventBusProcessor[TweetEvent] {\n\n  private val tweetCreateEventCounter = statsReceiver.counter(\"num_tweet_create_events\")\n  private val nonTweetCreateEventCounter = statsReceiver.counter(\"num_non_tweet_create_events\")\n\n  private val tweetActionStats = statsReceiver.scope(\"tweet_action\")\n  private val numUrlCounter = statsReceiver.counter(\"num_tweet_url\")\n  private val numMediaUrlCounter = statsReceiver.counter(\"num_tweet_media_url\")\n  private val numHashTagCounter = statsReceiver.counter(\"num_tweet_hashtag\")\n\n  private val numMentionsCounter = statsReceiver.counter(\"num_tweet_mention\")\n  private val numMediatagCounter = statsReceiver.counter(\"num_tweet_mediatag\")\n  private val numValidMentionsCounter = statsReceiver.counter(\"num_tweet_valid_mention\")\n  private val numValidMediatagCounter = statsReceiver.counter(\"num_tweet_valid_mediatag\")\n\n  private val numNullCastTweetCounter = statsReceiver.counter(\"num_null_cast_tweet\")\n  private val numNullCastSourceTweetCounter = statsReceiver.counter(\"num_null_cast_source_tweet\")\n  private val numTweetFailSafetyLevelCounter = statsReceiver.counter(\"num_fail_tweetypie_safety\")\n  private val numAuthorUnsafeCounter = statsReceiver.counter(\"num_author_unsafe\")\n  private val numProcessTweetCounter = statsReceiver.counter(\"num_process_tweet\")\n  private val numNoProcessTweetCounter = statsReceiver.counter(\"num_no_process_tweet\")\n\n  private val selfRetweetCounter = statsReceiver.counter(\"num_retweets_self\")\n\n  private val engageUserFilter = new UserFilter(gizmoduck)(statsReceiver.scope(\"author_user\"))\n  private val tweetFilter = new TweetFilter(tweetypie)\n\n  private def trackTweetCreateEventStats(details: TweetCreateEventDetails): Unit = {\n    tweetActionStats.counter(details.userTweetEngagement.action.toString).incr()\n\n    details.userTweetEngagement.tweetDetails.foreach { tweetDetails =>\n      tweetDetails.mentionUserIds.foreach(mention => numMentionsCounter.incr(mention.size))\n      tweetDetails.mediatagUserIds.foreach(mediatag => numMediatagCounter.incr(mediatag.size))\n      tweetDetails.urls.foreach(urls => numUrlCounter.incr(urls.size))\n      tweetDetails.mediaUrls.foreach(mediaUrls => numMediaUrlCounter.incr(mediaUrls.size))\n      tweetDetails.hashtags.foreach(hashtags => numHashTagCounter.incr(hashtags.size))\n    }\n\n    details.validMentionUserIds.foreach(mentions => numValidMentionsCounter.incr(mentions.size))\n    details.validMediatagUserIds.foreach(mediatags => numValidMediatagCounter.incr(mediatags.size))\n  }\n\n  /**\n   * Given a created tweet, return what type of tweet it is, i.e. Tweet, Retweet, Quote, or Reply。\n   * Retweet, Quote, or Reply are responsive actions to a source tweet, so for these tweets,\n   * we also return the tweet id and author of the source tweet (ex. the tweet being retweeted).\n   */\n  private def getTweetAction(tweetDetails: TweetDetails): Action = {\n    (tweetDetails.replySourceId, tweetDetails.retweetSourceId, tweetDetails.quoteSourceId) match {\n      case (Some(_), _, _) =>\n        Action.Reply\n      case (_, Some(_), _) =>\n        Action.Retweet\n      case (_, _, Some(_)) =>\n        Action.Quote\n      case _ =>\n        Action.Tweet\n    }\n  }\n\n  /**\n   * Given a list of mentioned users and mediatagged users in the tweet, return the users who\n   * actually follow the source user.\n   */\n  private def getFollowedByIds(\n    sourceUserId: Long,\n    mentionUserIds: Option[Seq[Long]],\n    mediatagUserIds: Option[Seq[Long]]\n  ): Future[Seq[Long]] = {\n    val uniqueEntityUserIds =\n      (mentionUserIds.getOrElse(Nil) ++ mediatagUserIds.getOrElse(Nil)).distinct\n    if (uniqueEntityUserIds.isEmpty) {\n      Future.Nil\n    } else {\n      socialGraph.followedByNotMutedBy(sourceUserId, uniqueEntityUserIds)\n    }\n  }\n\n  private def getSourceTweet(tweetDetails: TweetDetails): Future[Option[Tweet]] = {\n    tweetDetails.sourceTweetId match {\n      case Some(sourceTweetId) =>\n        tweetypie.getTweet(sourceTweetId)\n      case _ =>\n        Future.None\n    }\n  }\n\n  /**\n   * Extract and return the details when the source user created a new tweet.\n   */\n  private def getTweetDetails(\n    tweet: Tweet,\n    engageUser: User\n  ): Future[TweetCreateEventDetails] = {\n    val tweetDetails = TweetDetails(tweet)\n\n    val action = getTweetAction(tweetDetails)\n    val tweetCreationTimeMillis = SnowflakeUtils.tweetCreationTime(tweet.id).map(_.inMilliseconds)\n    val engageUserId = engageUser.id\n    val userTweetEngagement = UserTweetEngagement(\n      engageUserId = engageUserId,\n      engageUser = Some(engageUser),\n      action = action,\n      engagementTimeMillis = tweetCreationTimeMillis,\n      tweetId = tweet.id,\n      tweetDetails = Some(tweetDetails)\n    )\n\n    val sourceTweetFut = getSourceTweet(tweetDetails)\n    val followedByIdsFut = getFollowedByIds(\n      engageUserId,\n      tweetDetails.mentionUserIds,\n      tweetDetails.mediatagUserIds\n    )\n\n    Future.join(followedByIdsFut, sourceTweetFut).map {\n      case (followedByIds, sourceTweet) =>\n        TweetCreateEventDetails(\n          userTweetEngagement = userTweetEngagement,\n          validEntityUserIds = followedByIds,\n          sourceTweetDetails = sourceTweet.map(TweetDetails)\n        )\n    }\n  }\n\n  /**\n   * Exclude any Retweets of one's own tweets\n   */\n  private def isEventSelfRetweet(tweetEvent: TweetCreateEventDetails): Boolean = {\n    (tweetEvent.userTweetEngagement.action == Action.Retweet) &&\n    tweetEvent.userTweetEngagement.tweetDetails.exists(\n      _.sourceTweetUserId.contains(\n        tweetEvent.userTweetEngagement.engageUserId\n      ))\n  }\n\n  private def isTweetPassSafetyFilter(tweetEvent: TweetCreateEventDetails): Future[Boolean] = {\n    tweetEvent.userTweetEngagement.action match {\n      case Action.Reply | Action.Retweet | Action.Quote =>\n        tweetEvent.userTweetEngagement.tweetDetails\n          .flatMap(_.sourceTweetId).map { sourceTweetId =>\n            tweetFilter.filterForTweetypieSafetyLevel(sourceTweetId)\n          }.getOrElse(Future(false))\n      case Action.Tweet =>\n        tweetFilter.filterForTweetypieSafetyLevel(tweetEvent.userTweetEngagement.tweetId)\n    }\n  }\n\n  private def shouldProcessTweetEvent(event: TweetCreateEventDetails): Future[Boolean] = {\n    val engagement = event.userTweetEngagement\n    val engageUserId = engagement.engageUserId\n\n    val isNullCastTweet = engagement.tweetDetails.forall(_.isNullCastTweet)\n    val isNullCastSourceTweet = event.sourceTweetDetails.exists(_.isNullCastTweet)\n    val isSelfRetweet = isEventSelfRetweet(event)\n    val isEngageUserSafeFut = engageUserFilter.filterByUserId(engageUserId)\n    val isTweetPassSafetyFut = isTweetPassSafetyFilter(event)\n\n    Future.join(isEngageUserSafeFut, isTweetPassSafetyFut).map {\n      case (isEngageUserSafe, isTweetPassSafety) =>\n        if (isNullCastTweet) numNullCastTweetCounter.incr()\n        if (isNullCastSourceTweet) numNullCastSourceTweetCounter.incr()\n        if (!isEngageUserSafe) numAuthorUnsafeCounter.incr()\n        if (isSelfRetweet) selfRetweetCounter.incr()\n        if (!isTweetPassSafety) numTweetFailSafetyLevelCounter.incr()\n\n        !isNullCastTweet &&\n        !isNullCastSourceTweet &&\n        !isSelfRetweet &&\n        isEngageUserSafe &&\n        isTweetPassSafety\n    }\n  }\n\n  override def processEvent(event: TweetEvent): Future[Unit] = {\n    event.data match {\n      case TweetEventData.TweetCreateEvent(event: TweetCreateEvent) =>\n        getTweetDetails(\n          tweet = event.tweet,\n          engageUser = event.user\n        ).flatMap { eventWithDetails =>\n          tweetCreateEventCounter.incr()\n\n          shouldProcessTweetEvent(eventWithDetails).map {\n            case true =>\n              numProcessTweetCounter.incr()\n              trackTweetCreateEventStats(eventWithDetails)\n              // Convert the event for UserUserGraph\n              userUserGraphMessageBuilder.processEvent(eventWithDetails).map { edges =>\n                edges.foreach { edge =>\n                  kafkaEventPublisher.publish(edge.convertToRecosHoseMessage, userUserGraphTopic)\n                }\n              }\n              // Convert the event for UserTweetEntityGraph\n              userTweetEntityGraphMessageBuilder.processEvent(eventWithDetails).map { edges =>\n                edges.foreach { edge =>\n                  kafkaEventPublisher\n                    .publish(edge.convertToRecosHoseMessage, userTweetEntityGraphTopic)\n                }\n              }\n            case false =>\n              numNoProcessTweetCounter.incr()\n          }\n        }\n      case _ =>\n        nonTweetCreateEventCounter.incr()\n        Future.Unit\n    }\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/filters/BUILD",
    "content": "scala_library(\n    platform = \"java11\",\n    strict_deps = False,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-stats\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/clients\",\n        \"src/thrift/com/twitter/gizmoduck:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/filters/NullCastTweetFilter.scala",
    "content": "package com.twitter.recosinjector.filters\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recosinjector.clients.Tweetypie\nimport com.twitter.util.Future\n\n/**\n * Filters tweets that are null cast, i.e. tweet is not delivered to a user's followers,\n * not shown in the user's timeline, and does not appear in search results.\n * They are mainly ads tweets.\n */\nclass NullCastTweetFilter(\n  tweetypie: Tweetypie\n)(\n  implicit statsReceiver: StatsReceiver) {\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val requests = stats.counter(\"requests\")\n  private val filtered = stats.counter(\"filtered\")\n\n  // Return Future(True) to keep the Tweet.\n  def filter(tweetId: Long): Future[Boolean] = {\n    requests.incr()\n    tweetypie\n      .getTweet(tweetId)\n      .map { tweetOpt =>\n        // If the null cast bit is Some(true), drop the tweet.\n        val isNullCastTweet = tweetOpt.flatMap(_.coreData).exists(_.nullcast)\n        if (isNullCastTweet) {\n          filtered.incr()\n        }\n        !isNullCastTweet\n      }\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/filters/TweetFilter.scala",
    "content": "package com.twitter.recosinjector.filters\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recosinjector.clients.Tweetypie\nimport com.twitter.util.Future\n\nclass TweetFilter(\n  tweetypie: Tweetypie\n)(\n  implicit statsReceiver: StatsReceiver) {\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val requests = stats.counter(\"requests\")\n  private val filtered = stats.counter(\"filtered\")\n\n  /**\n   * Query Tweetypie to see if we can fetch a tweet object successfully. TweetyPie applies a safety\n   * filter and will not return the tweet object if the filter does not pass.\n   */\n  def filterForTweetypieSafetyLevel(tweetId: Long): Future[Boolean] = {\n    requests.incr()\n    tweetypie\n      .getTweet(tweetId)\n      .map {\n        case Some(_) =>\n          true\n        case _ =>\n          filtered.incr()\n          false\n      }\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/filters/UserFilter.scala",
    "content": "package com.twitter.recosinjector.filters\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.gizmoduck.thriftscala.{LabelValue, User}\nimport com.twitter.recosinjector.clients.Gizmoduck\nimport com.twitter.util.Future\n\nclass UserFilter(\n  gizmoduck: Gizmoduck\n)(\n  implicit statsReceiver: StatsReceiver) {\n  private val stats = statsReceiver.scope(this.getClass.getSimpleName)\n  private val requests = stats.counter(\"requests\")\n  private val filtered = stats.counter(\"filtered\")\n\n  private def isUnsafe(user: User): Boolean =\n    user.safety.exists { s =>\n      s.deactivated || s.suspended || s.restricted || s.nsfwUser || s.nsfwAdmin || s.isProtected\n    }\n\n  private def hasNsfwHighPrecisionLabel(user: User): Boolean =\n    user.labels.exists {\n      _.labels.exists(_.labelValue == LabelValue.NsfwHighPrecision)\n    }\n\n  /**\n   * NOTE: This will by-pass Gizmoduck's safety level, and might allow invalid users to pass filter.\n   * Consider using filterByUserId instead.\n   * Return true if the user is valid, otherwise return false.\n   * It will first attempt to use the user object provided by the caller, and will call Gizmoduck\n   * to back fill if the caller does not provide it. This helps reduce Gizmoduck traffic.\n   */\n  def filterByUser(\n    userId: Long,\n    userOpt: Option[User] = None\n  ): Future[Boolean] = {\n    requests.incr()\n    val userFut = userOpt match {\n      case Some(user) => Future(Some(user))\n      case _ => gizmoduck.getUser(userId)\n    }\n\n    userFut.map(_.exists { user =>\n      val isValidUser = !isUnsafe(user) && !hasNsfwHighPrecisionLabel(user)\n      if (!isValidUser) filtered.incr()\n      isValidUser\n    })\n  }\n\n  /**\n   * Given a userId, return true if the user is valid. This id done in 2 steps:\n   * 1. Applying Gizmoduck's safety level while querying for the user from Gizmoduck\n   * 2. If a user passes Gizmoduck's safety level, check its specific user status\n   */\n  def filterByUserId(userId: Long): Future[Boolean] = {\n    requests.incr()\n    gizmoduck\n      .getUser(userId)\n      .map { userOpt =>\n        val isValidUser = userOpt.exists { user =>\n          !(isUnsafe(user) || hasNsfwHighPrecisionLabel(user))\n        }\n        if (!isValidUser) {\n          filtered.incr()\n        }\n        isValidUser\n      }\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/publishers/BUILD",
    "content": "scala_library(\n    platform = \"java11\",\n    strict_deps = False,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"finatra-internal/messaging/kafka/src/main/scala\",\n        \"servo/repo/src/main/scala\",\n        \"src/thrift/com/twitter/recos:recos-injector-scala\",\n        \"src/thrift/com/twitter/recos:recos-internal-scala\",\n    ],\n)\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/publishers/KafkaEventPublisher.scala",
    "content": "package com.twitter.recosinjector.publishers\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.finatra.kafka.producers.FinagleKafkaProducerBuilder\nimport com.twitter.finatra.kafka.serde.ScalaSerdes\nimport com.twitter.recos.internal.thriftscala.RecosHoseMessage\nimport org.apache.kafka.clients.CommonClientConfigs\nimport org.apache.kafka.clients.producer.ProducerRecord\nimport org.apache.kafka.common.config.SaslConfigs\nimport org.apache.kafka.common.config.SslConfigs\nimport org.apache.kafka.common.security.auth.SecurityProtocol\nimport org.apache.kafka.common.serialization.StringSerializer\n\ncase class KafkaEventPublisher(\n  kafkaDest: String,\n  outputKafkaTopicPrefix: String,\n  clientId: ClientId,\n  truststoreLocation: String) {\n\n  private val producer = FinagleKafkaProducerBuilder[String, RecosHoseMessage]()\n    .dest(kafkaDest)\n    .clientId(clientId.name)\n    .keySerializer(new StringSerializer)\n    .valueSerializer(ScalaSerdes.Thrift[RecosHoseMessage].serializer)\n    .withConfig(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, SecurityProtocol.SASL_SSL.toString)\n    .withConfig(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, truststoreLocation)\n    .withConfig(SaslConfigs.SASL_MECHANISM, SaslConfigs.GSSAPI_MECHANISM)\n    .withConfig(SaslConfigs.SASL_KERBEROS_SERVICE_NAME, \"kafka\")\n    .withConfig(SaslConfigs.SASL_KERBEROS_SERVER_NAME, \"kafka\")\n    // Use Native Kafka Client\n    .buildClient()\n\n  def publish(\n    message: RecosHoseMessage,\n    topic: String\n  )(\n    implicit statsReceiver: StatsReceiver\n  ): Unit = {\n    val topicName = s\"${outputKafkaTopicPrefix}_$topic\"\n    // Kafka Producer is thread-safe. No extra Future-pool protect.\n    producer.send(new ProducerRecord(topicName, message))\n    statsReceiver.counter(topicName + \"_written_msg_success\").incr()\n  }\n}\n\nobject KafkaEventPublisher {\n  // Kafka topics available for publishing\n  val UserVideoTopic = \"user_video\"\n  val UserTweetEntityTopic = \"user_tweet_entity\"\n  val UserUserTopic = \"user_user\"\n  val UserAdTopic = \"user_tweet\"\n  val UserTweetPlusTopic = \"user_tweet_plus\"\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/util/BUILD",
    "content": "scala_library(\n    platform = \"java11\",\n    strict_deps = False,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/twitter/graphjet\",\n        \"finagle/finagle-stats\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/clients\",\n        \"src/scala/com/twitter/recos/util:recos-util\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n    ],\n)\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/util/EventDetails.scala",
    "content": "package com.twitter.recosinjector.util\n\nimport com.twitter.frigate.common.base.TweetUtil\nimport com.twitter.gizmoduck.thriftscala.User\nimport com.twitter.recos.util.Action.Action\nimport com.twitter.tweetypie.thriftscala.Tweet\n\n/**\n * This is used to store information about a newly created tweet\n * @param validEntityUserIds For users mentioned or mediatagged in the tweet, these follow the\n *                           engage user and only they are are considered valid\n * @param sourceTweetDetails For Reply, Quote, or RT, source tweet is the tweet being actioned on\n */\ncase class TweetCreateEventDetails(\n  userTweetEngagement: UserTweetEngagement,\n  validEntityUserIds: Seq[Long],\n  sourceTweetDetails: Option[TweetDetails]) {\n  // A mention is only valid if the mentioned user follows the source user\n  val validMentionUserIds: Option[Seq[Long]] = {\n    userTweetEngagement.tweetDetails.flatMap(_.mentionUserIds.map(_.intersect(validEntityUserIds)))\n  }\n\n  // A mediatag is only valid if the mediatagged user follows the source user\n  val validMediatagUserIds: Option[Seq[Long]] = {\n    userTweetEngagement.tweetDetails.flatMap(_.mediatagUserIds.map(_.intersect(validEntityUserIds)))\n  }\n}\n\n/**\n * Stores information about a favorite/unfav engagement.\n * NOTE: This could either be Likes, or UNLIKEs (i.e. when user cancels the Like)\n * @param userTweetEngagement the engagement details\n */\ncase class TweetFavoriteEventDetails(\n  userTweetEngagement: UserTweetEngagement)\n\n/**\n * Stores information about a unified user action engagement.\n * @param userTweetEngagement the engagement details\n */\ncase class UuaEngagementEventDetails(\n  userTweetEngagement: UserTweetEngagement)\n\n/**\n * Details about a user-tweet engagement, like when a user tweeted/liked a tweet\n * @param engageUserId User that engaged with the tweet\n * @param action The action the user took on the tweet\n * @param tweetId The type of engagement the user took on the tweet\n */\ncase class UserTweetEngagement(\n  engageUserId: Long,\n  engageUser: Option[User],\n  action: Action,\n  engagementTimeMillis: Option[Long],\n  tweetId: Long,\n  tweetDetails: Option[TweetDetails])\n\n/**\n * Helper class that decomposes a tweet object and provides related details about this tweet\n */\ncase class TweetDetails(tweet: Tweet) {\n  val authorId: Option[Long] = tweet.coreData.map(_.userId)\n\n  val urls: Option[Seq[String]] = tweet.urls.map(_.map(_.url))\n\n  val mediaUrls: Option[Seq[String]] = tweet.media.map(_.map(_.expandedUrl))\n\n  val hashtags: Option[Seq[String]] = tweet.hashtags.map(_.map(_.text))\n\n  // mentionUserIds include reply user ids at the beginning of a tweet\n  val mentionUserIds: Option[Seq[Long]] = tweet.mentions.map(_.flatMap(_.userId))\n\n  val mediatagUserIds: Option[Seq[Long]] = tweet.mediaTags.map {\n    _.tagMap.flatMap {\n      case (_, mediaTag) => mediaTag.flatMap(_.userId)\n    }.toSeq\n  }\n\n  val replySourceId: Option[Long] = tweet.coreData.flatMap(_.reply.flatMap(_.inReplyToStatusId))\n  val replyUserId: Option[Long] = tweet.coreData.flatMap(_.reply.map(_.inReplyToUserId))\n\n  val retweetSourceId: Option[Long] = tweet.coreData.flatMap(_.share.map(_.sourceStatusId))\n  val retweetUserId: Option[Long] = tweet.coreData.flatMap(_.share.map(_.sourceUserId))\n\n  val quoteSourceId: Option[Long] = tweet.quotedTweet.map(_.tweetId)\n  val quoteUserId: Option[Long] = tweet.quotedTweet.map(_.userId)\n  val quoteTweetUrl: Option[String] = tweet.quotedTweet.flatMap(_.permalink.map(_.shortUrl))\n\n  //If the tweet is retweet/reply/quote, this is the tweet that the new tweet responds to\n  val (sourceTweetId, sourceTweetUserId) = {\n    (replySourceId, retweetSourceId, quoteSourceId) match {\n      case (Some(replyId), _, _) =>\n        (Some(replyId), replyUserId)\n      case (_, Some(retweetId), _) =>\n        (Some(retweetId), retweetUserId)\n      case (_, _, Some(quoteId)) =>\n        (Some(quoteId), quoteUserId)\n      case _ =>\n        (None, None)\n    }\n  }\n\n  // Boolean information\n  val hasPhoto: Boolean = TweetUtil.containsPhotoTweet(tweet)\n\n  val hasVideo: Boolean = TweetUtil.containsVideoTweet(tweet)\n\n  // TweetyPie does not populate url fields in a quote tweet create event, even though we\n  // consider quote tweets as url tweets. This boolean helps make up for it.\n  // Details: https://groups.google.com/a/twitter.com/d/msg/eng/BhK1XAcSSWE/F8Gc4_5uDwAJ\n  val hasQuoteTweetUrl: Boolean = tweet.quotedTweet.exists(_.permalink.isDefined)\n\n  val hasUrl: Boolean = this.urls.exists(_.nonEmpty) || hasQuoteTweetUrl\n\n  val hasHashtag: Boolean = this.hashtags.exists(_.nonEmpty)\n\n  val isCard: Boolean = hasUrl | hasPhoto | hasVideo\n\n  implicit def bool2Long(b: Boolean): Long = if (b) 1L else 0L\n\n  // Return a hashed long that contains card type information of the tweet\n  val cardInfo: Long = isCard | (hasUrl << 1) | (hasPhoto << 2) | (hasVideo << 3)\n\n  // nullcast tweet is one that is purposefully not broadcast to followers, ex. an ad tweet.\n  val isNullCastTweet: Boolean = tweet.coreData.exists(_.nullcast)\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/uua_processors/BUILD",
    "content": "scala_library(\n    platform = \"java11\",\n    strict_deps = False,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"eventbus/client\",\n        \"kafka/finagle-kafka/finatra-kafka/src/main/scala\",\n        \"kafka/libs/src/main/scala/com/twitter/kafka/client/headers\",\n        \"kafka/libs/src/main/scala/com/twitter/kafka/client/processor\",\n        \"recos-injector/server/config\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/clients\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/config\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/decider\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/edges\",\n        \"recos-injector/server/src/main/scala/com/twitter/recosinjector/publishers\",\n        \"src/thrift/com/twitter/clientapp/gen:clientapp-scala\",\n        \"src/thrift/com/twitter/gizmoduck:user-thrift-scala\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"src/thrift/com/twitter/timelineservice/server/internal:thrift-scala\",\n        \"src/thrift/com/twitter/tweetypie:events-scala\",\n        \"src/thrift/com/twitter/tweetypie:tweet-scala\",\n        \"unified_user_actions/thrift/src/main/thrift/com/twitter/unified_user_actions:unified_user_actions-scala\",\n    ],\n)\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/uua_processors/UnifiedUserActionProcessor.scala",
    "content": "package com.twitter.recosinjector.uua_processors\n\nimport org.apache.kafka.clients.consumer.ConsumerRecord\nimport com.twitter.finatra.kafka.serde.UnKeyed\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.recos.util.Action\nimport com.twitter.recos.util.Action.Action\nimport com.twitter.recosinjector.clients.Gizmoduck\nimport com.twitter.recosinjector.clients.Tweetypie\nimport com.twitter.recosinjector.edges.UnifiedUserActionToUserVideoGraphBuilder\nimport com.twitter.recosinjector.edges.UnifiedUserActionToUserAdGraphBuilder\nimport com.twitter.recosinjector.edges.UnifiedUserActionToUserTweetGraphPlusBuilder\nimport com.twitter.unified_user_actions.thriftscala.UnifiedUserAction\nimport com.twitter.unified_user_actions.thriftscala.ActionType\nimport com.twitter.unified_user_actions.thriftscala.Item\nimport com.twitter.recosinjector.filters.UserFilter\nimport com.twitter.recosinjector.publishers.KafkaEventPublisher\nimport com.twitter.recosinjector.util.TweetDetails\nimport com.twitter.recosinjector.util.UserTweetEngagement\nimport com.twitter.recosinjector.util.UuaEngagementEventDetails\nimport com.twitter.unified_user_actions.thriftscala.NotificationContent\nimport com.twitter.unified_user_actions.thriftscala.NotificationInfo\nimport com.twitter.util.Future\n\nclass UnifiedUserActionProcessor(\n  gizmoduck: Gizmoduck,\n  tweetypie: Tweetypie,\n  kafkaEventPublisher: KafkaEventPublisher,\n  userVideoGraphTopic: String,\n  userVideoGraphBuilder: UnifiedUserActionToUserVideoGraphBuilder,\n  userAdGraphTopic: String,\n  userAdGraphBuilder: UnifiedUserActionToUserAdGraphBuilder,\n  userTweetGraphPlusTopic: String,\n  userTweetGraphPlusBuilder: UnifiedUserActionToUserTweetGraphPlusBuilder\n)(\n  implicit statsReceiver: StatsReceiver) {\n\n  val messagesProcessedCount = statsReceiver.counter(\"messages_processed\")\n\n  val eventsByTypeCounts = statsReceiver.scope(\"events_by_type\")\n  private val numSelfEngageCounter = statsReceiver.counter(\"num_self_engage_event\")\n  private val numTweetFailSafetyLevelCounter = statsReceiver.counter(\"num_fail_tweetypie_safety\")\n  private val numNullCastTweetCounter = statsReceiver.counter(\"num_null_cast_tweet\")\n  private val numEngageUserUnsafeCounter = statsReceiver.counter(\"num_engage_user_unsafe\")\n  private val engageUserFilter = new UserFilter(gizmoduck)(statsReceiver.scope(\"engage_user\"))\n  private val numNoProcessTweetCounter = statsReceiver.counter(\"num_no_process_tweet\")\n  private val numProcessTweetCounter = statsReceiver.counter(\"num_process_tweet\")\n\n  private def getUuaEngagementEventDetails(\n    unifiedUserAction: UnifiedUserAction\n  ): Option[Future[UuaEngagementEventDetails]] = {\n    val userIdOpt = unifiedUserAction.userIdentifier.userId\n    val tweetIdOpt = unifiedUserAction.item match {\n      case Item.TweetInfo(tweetInfo) => Some(tweetInfo.actionTweetId)\n      case Item.NotificationInfo(\n            NotificationInfo(_, NotificationContent.TweetNotification(notification))) =>\n        Some(notification.tweetId)\n      case _ => None\n    }\n    val timestamp = unifiedUserAction.eventMetadata.sourceTimestampMs\n    val action = getTweetAction(unifiedUserAction.actionType)\n\n    tweetIdOpt\n      .flatMap { tweetId =>\n        userIdOpt.map { engageUserId =>\n          val tweetFut = tweetypie.getTweet(tweetId)\n          tweetFut.map { tweetOpt =>\n            val tweetDetailsOpt = tweetOpt.map(TweetDetails)\n            val engagement = UserTweetEngagement(\n              engageUserId = engageUserId,\n              action = action,\n              engagementTimeMillis = Some(timestamp),\n              tweetId = tweetId,\n              engageUser = None,\n              tweetDetails = tweetDetailsOpt\n            )\n            UuaEngagementEventDetails(engagement)\n          }\n        }\n      }\n  }\n\n  private def getTweetAction(action: ActionType): Action = {\n    action match {\n      case ActionType.ClientTweetVideoPlayback50 => Action.VideoPlayback50\n      case ActionType.ClientTweetClick => Action.Click\n      case ActionType.ClientTweetVideoPlayback75 => Action.VideoPlayback75\n      case ActionType.ClientTweetVideoQualityView => Action.VideoQualityView\n      case ActionType.ServerTweetFav => Action.Favorite\n      case ActionType.ServerTweetReply => Action.Reply\n      case ActionType.ServerTweetRetweet => Action.Retweet\n      case ActionType.ClientTweetQuote => Action.Quote\n      case ActionType.ClientNotificationOpen => Action.NotificationOpen\n      case ActionType.ClientTweetEmailClick => Action.EmailClick\n      case ActionType.ClientTweetShareViaBookmark => Action.Share\n      case ActionType.ClientTweetShareViaCopyLink => Action.Share\n      case ActionType.ClientTweetSeeFewer => Action.TweetSeeFewer\n      case ActionType.ClientTweetNotRelevant => Action.TweetNotRelevant\n      case ActionType.ClientTweetNotInterestedIn => Action.TweetNotInterestedIn\n      case ActionType.ServerTweetReport => Action.TweetReport\n      case ActionType.ClientTweetMuteAuthor => Action.TweetMuteAuthor\n      case ActionType.ClientTweetBlockAuthor => Action.TweetBlockAuthor\n      case _ => Action.UnDefined\n    }\n  }\n  private def shouldProcessTweetEngagement(\n    event: UuaEngagementEventDetails,\n    isAdsUseCase: Boolean = false\n  ): Future[Boolean] = {\n    val engagement = event.userTweetEngagement\n    val engageUserId = engagement.engageUserId\n    val authorIdOpt = engagement.tweetDetails.flatMap(_.authorId)\n\n    val isSelfEngage = authorIdOpt.contains(engageUserId)\n    val isNullCastTweet = engagement.tweetDetails.forall(_.isNullCastTweet)\n    val isEngageUserSafeFut = engageUserFilter.filterByUserId(engageUserId)\n    val isTweetPassSafety =\n      engagement.tweetDetails.isDefined // Tweetypie can fetch a tweet object successfully\n\n    isEngageUserSafeFut.map { isEngageUserSafe =>\n      if (isSelfEngage) numSelfEngageCounter.incr()\n      if (isNullCastTweet) numNullCastTweetCounter.incr()\n      if (!isEngageUserSafe) numEngageUserUnsafeCounter.incr()\n      if (!isTweetPassSafety) numTweetFailSafetyLevelCounter.incr()\n\n      !isSelfEngage && (!isNullCastTweet && !isAdsUseCase || isNullCastTweet && isAdsUseCase) && isEngageUserSafe && isTweetPassSafety\n    }\n  }\n\n  def apply(record: ConsumerRecord[UnKeyed, UnifiedUserAction]): Future[Unit] = {\n\n    messagesProcessedCount.incr()\n    val unifiedUserAction = record.value\n    eventsByTypeCounts.counter(unifiedUserAction.actionType.toString).incr()\n\n    getTweetAction(unifiedUserAction.actionType) match {\n      case Action.UnDefined =>\n        numNoProcessTweetCounter.incr()\n        Future.Unit\n      case action =>\n        getUuaEngagementEventDetails(unifiedUserAction)\n          .map {\n            _.flatMap { detail =>\n              // The following cases are set up specifically for an ads relevance demo.\n              val actionForAds = Set(Action.Click, Action.Favorite, Action.VideoPlayback75)\n              if (actionForAds.contains(action))\n                shouldProcessTweetEngagement(detail, isAdsUseCase = true).map {\n                  case true =>\n                    userAdGraphBuilder.processEvent(detail).map { edges =>\n                      edges.foreach { edge =>\n                        kafkaEventPublisher\n                          .publish(edge.convertToRecosHoseMessage, userAdGraphTopic)\n                      }\n                    }\n                    numProcessTweetCounter.incr()\n                  case _ =>\n                }\n\n              shouldProcessTweetEngagement(detail).map {\n                case true =>\n                  userVideoGraphBuilder.processEvent(detail).map { edges =>\n                    edges.foreach { edge =>\n                      kafkaEventPublisher\n                        .publish(edge.convertToRecosHoseMessage, userVideoGraphTopic)\n                    }\n                  }\n\n                  userTweetGraphPlusBuilder.processEvent(detail).map { edges =>\n                    edges.foreach { edge =>\n                      kafkaEventPublisher\n                        .publish(edge.convertToRecosHoseMessage, userTweetGraphPlusTopic)\n                    }\n                  }\n                  numProcessTweetCounter.incr()\n                case _ =>\n              }\n            }\n          }.getOrElse(Future.Unit)\n    }\n  }\n}\n"
  },
  {
    "path": "recos-injector/server/src/main/scala/com/twitter/recosinjector/uua_processors/UnifiedUserActionsConsumer.scala",
    "content": "package com.twitter.recosinjector.uua_processors\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finatra.kafka.consumers.FinagleKafkaConsumerBuilder\nimport com.twitter.finatra.kafka.domain.KafkaGroupId\nimport com.twitter.finatra.kafka.domain.SeekStrategy\nimport com.twitter.finatra.kafka.serde.ScalaSerdes\nimport com.twitter.finatra.kafka.serde.UnKeyed\nimport com.twitter.finatra.kafka.serde.UnKeyedSerde\nimport org.apache.kafka.clients.CommonClientConfigs\nimport org.apache.kafka.common.config.SaslConfigs\nimport org.apache.kafka.common.config.SslConfigs\nimport org.apache.kafka.common.security.auth.SecurityProtocol\nimport com.twitter.unified_user_actions.thriftscala.UnifiedUserAction\nimport com.twitter.kafka.client.processor.AtLeastOnceProcessor\nimport com.twitter.kafka.client.processor.ThreadSafeKafkaConsumerClient\nimport com.twitter.conversions.StorageUnitOps._\n\nclass UnifiedUserActionsConsumer(\n  processor: UnifiedUserActionProcessor,\n  truststoreLocation: String\n)(\n  implicit statsReceiver: StatsReceiver) {\n  import UnifiedUserActionsConsumer._\n\n  private val kafkaClient = new ThreadSafeKafkaConsumerClient[UnKeyed, UnifiedUserAction](\n    FinagleKafkaConsumerBuilder[UnKeyed, UnifiedUserAction]()\n      .groupId(KafkaGroupId(uuaRecosInjectorGroupId))\n      .keyDeserializer(UnKeyedSerde.deserializer)\n      .valueDeserializer(ScalaSerdes.Thrift[UnifiedUserAction].deserializer)\n      .dest(uuaDest)\n      .maxPollRecords(maxPollRecords)\n      .maxPollInterval(maxPollInterval)\n      .fetchMax(fetchMax)\n      .seekStrategy(SeekStrategy.END)\n      .enableAutoCommit(false) // AtLeastOnceProcessor performs commits manually\n      .withConfig(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, SecurityProtocol.SASL_SSL.toString)\n      .withConfig(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, truststoreLocation)\n      .withConfig(SaslConfigs.SASL_MECHANISM, SaslConfigs.GSSAPI_MECHANISM)\n      .withConfig(SaslConfigs.SASL_KERBEROS_SERVICE_NAME, \"kafka\")\n      .withConfig(SaslConfigs.SASL_KERBEROS_SERVER_NAME, \"kafka\")\n      .config)\n\n  val atLeastOnceProcessor: AtLeastOnceProcessor[UnKeyed, UnifiedUserAction] = {\n    AtLeastOnceProcessor[UnKeyed, UnifiedUserAction](\n      name = processorName,\n      topic = uuaTopic,\n      consumer = kafkaClient,\n      processor = processor.apply,\n      maxPendingRequests = maxPendingRequests,\n      workerThreads = workerThreads,\n      commitIntervalMs = commitIntervalMs,\n      statsReceiver = statsReceiver.scope(processorName)\n    )\n  }\n\n}\n\nobject UnifiedUserActionsConsumer {\n  val maxPollRecords = 1000\n  val maxPollInterval = 5.minutes\n  val fetchMax = 1.megabytes\n  val maxPendingRequests = 1000\n  val workerThreads = 16\n  val commitIntervalMs = 10.seconds.inMilliseconds\n  val processorName = \"unified_user_actions_processor\"\n  val uuaTopic = \"unified_user_actions_engagements\"\n  val uuaDest = \"/s/kafka/bluebird-1:kafka-tls\"\n  val uuaRecosInjectorGroupId = \"recos-injector\"\n}\n"
  },
  {
    "path": "representation-manager/BUILD.bazel",
    "content": "# This prevents SQ query from grabbing //:all since it traverses up once to find a BUILD\n"
  },
  {
    "path": "representation-manager/README.md",
    "content": "# Representation Manager #\n\n**Representation Manager** (RMS) serves as a centralized embedding management system, providing SimClusters or other embeddings as facade of the underlying storage or services. \n\n"
  },
  {
    "path": "representation-manager/bin/deploy.sh",
    "content": "#!/usr/bin/env bash\n\nJOB=representation-manager bazel run --ui_event_filters=-info,-stdout,-stderr --noshow_progress \\\n\t//relevance-platform/src/main/python/deploy -- \"$@\"\n"
  },
  {
    "path": "representation-manager/client/src/main/scala/com/twitter/representation_manager/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-thrift-client\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/strato\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/readablestore\",\n        \"representation-manager/client/src/main/scala/com/twitter/representation_manager/config\",\n        \"representation-manager/server/src/main/thrift:thrift-scala\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"stitch/stitch-storehaus\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n)\n"
  },
  {
    "path": "representation-manager/client/src/main/scala/com/twitter/representation_manager/StoreBuilder.scala",
    "content": "package com.twitter.representation_manager\n\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.hermit.store.common.ObservedCachedReadableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.representation_manager.config.ClientConfig\nimport com.twitter.representation_manager.config.DisabledInMemoryCacheParams\nimport com.twitter.representation_manager.config.EnabledInMemoryCacheParams\nimport com.twitter.representation_manager.thriftscala.SimClustersEmbeddingView\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.LocaleEntityId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.TopicId\nimport com.twitter.simclusters_v2.thriftscala.{SimClustersEmbedding => ThriftSimClustersEmbedding}\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.strato.thrift.ScroogeConvImplicits._\n\n/**\n * This is the class that offers features to build readable stores for a given\n * SimClustersEmbeddingView (i.e. embeddingType and modelVersion). It applies ClientConfig\n * for a particular service and build ReadableStores which implement that config.\n */\nclass StoreBuilder(\n  clientConfig: ClientConfig,\n  stratoClient: StratoClient,\n  memCachedClient: MemcachedClient,\n  globalStats: StatsReceiver,\n) {\n  private val stats =\n    globalStats.scope(\"representation_manager_client\").scope(this.getClass.getSimpleName)\n\n  // Column consts\n  private val ColPathPrefix = \"recommendations/representation_manager/\"\n  private val SimclustersTweetColPath = ColPathPrefix + \"simClustersEmbedding.Tweet\"\n  private val SimclustersUserColPath = ColPathPrefix + \"simClustersEmbedding.User\"\n  private val SimclustersTopicIdColPath = ColPathPrefix + \"simClustersEmbedding.TopicId\"\n  private val SimclustersLocaleEntityIdColPath =\n    ColPathPrefix + \"simClustersEmbedding.LocaleEntityId\"\n\n  def buildSimclustersTweetEmbeddingStore(\n    embeddingColumnView: SimClustersEmbeddingView\n  ): ReadableStore[Long, SimClustersEmbedding] = {\n    val rawStore = StratoFetchableStore\n      .withView[Long, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n        stratoClient,\n        SimclustersTweetColPath,\n        embeddingColumnView)\n      .mapValues(SimClustersEmbedding(_))\n\n    addCacheLayer(rawStore, embeddingColumnView)\n  }\n\n  def buildSimclustersUserEmbeddingStore(\n    embeddingColumnView: SimClustersEmbeddingView\n  ): ReadableStore[Long, SimClustersEmbedding] = {\n    val rawStore = StratoFetchableStore\n      .withView[Long, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n        stratoClient,\n        SimclustersUserColPath,\n        embeddingColumnView)\n      .mapValues(SimClustersEmbedding(_))\n\n    addCacheLayer(rawStore, embeddingColumnView)\n  }\n\n  def buildSimclustersTopicIdEmbeddingStore(\n    embeddingColumnView: SimClustersEmbeddingView\n  ): ReadableStore[TopicId, SimClustersEmbedding] = {\n    val rawStore = StratoFetchableStore\n      .withView[TopicId, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n        stratoClient,\n        SimclustersTopicIdColPath,\n        embeddingColumnView)\n      .mapValues(SimClustersEmbedding(_))\n\n    addCacheLayer(rawStore, embeddingColumnView)\n  }\n\n  def buildSimclustersLocaleEntityIdEmbeddingStore(\n    embeddingColumnView: SimClustersEmbeddingView\n  ): ReadableStore[LocaleEntityId, SimClustersEmbedding] = {\n    val rawStore = StratoFetchableStore\n      .withView[LocaleEntityId, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n        stratoClient,\n        SimclustersLocaleEntityIdColPath,\n        embeddingColumnView)\n      .mapValues(SimClustersEmbedding(_))\n\n    addCacheLayer(rawStore, embeddingColumnView)\n  }\n\n  def buildSimclustersTweetEmbeddingStoreWithEmbeddingIdAsKey(\n    embeddingColumnView: SimClustersEmbeddingView\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n    val rawStore = StratoFetchableStore\n      .withView[Long, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n        stratoClient,\n        SimclustersTweetColPath,\n        embeddingColumnView)\n      .mapValues(SimClustersEmbedding(_))\n    val embeddingIdAsKeyStore = rawStore.composeKeyMapping[SimClustersEmbeddingId] {\n      case SimClustersEmbeddingId(_, _, InternalId.TweetId(tweetId)) =>\n        tweetId\n    }\n\n    addCacheLayer(embeddingIdAsKeyStore, embeddingColumnView)\n  }\n\n  def buildSimclustersUserEmbeddingStoreWithEmbeddingIdAsKey(\n    embeddingColumnView: SimClustersEmbeddingView\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n    val rawStore = StratoFetchableStore\n      .withView[Long, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n        stratoClient,\n        SimclustersUserColPath,\n        embeddingColumnView)\n      .mapValues(SimClustersEmbedding(_))\n    val embeddingIdAsKeyStore = rawStore.composeKeyMapping[SimClustersEmbeddingId] {\n      case SimClustersEmbeddingId(_, _, InternalId.UserId(userId)) =>\n        userId\n    }\n\n    addCacheLayer(embeddingIdAsKeyStore, embeddingColumnView)\n  }\n\n  def buildSimclustersTopicEmbeddingStoreWithEmbeddingIdAsKey(\n    embeddingColumnView: SimClustersEmbeddingView\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n    val rawStore = StratoFetchableStore\n      .withView[TopicId, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n        stratoClient,\n        SimclustersTopicIdColPath,\n        embeddingColumnView)\n      .mapValues(SimClustersEmbedding(_))\n    val embeddingIdAsKeyStore = rawStore.composeKeyMapping[SimClustersEmbeddingId] {\n      case SimClustersEmbeddingId(_, _, InternalId.TopicId(topicId)) =>\n        topicId\n    }\n\n    addCacheLayer(embeddingIdAsKeyStore, embeddingColumnView)\n  }\n\n  def buildSimclustersTopicIdEmbeddingStoreWithEmbeddingIdAsKey(\n    embeddingColumnView: SimClustersEmbeddingView\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n    val rawStore = StratoFetchableStore\n      .withView[TopicId, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n        stratoClient,\n        SimclustersTopicIdColPath,\n        embeddingColumnView)\n      .mapValues(SimClustersEmbedding(_))\n    val embeddingIdAsKeyStore = rawStore.composeKeyMapping[SimClustersEmbeddingId] {\n      case SimClustersEmbeddingId(_, _, InternalId.TopicId(topicId)) =>\n        topicId\n    }\n\n    addCacheLayer(embeddingIdAsKeyStore, embeddingColumnView)\n  }\n\n  def buildSimclustersLocaleEntityIdEmbeddingStoreWithEmbeddingIdAsKey(\n    embeddingColumnView: SimClustersEmbeddingView\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n    val rawStore = StratoFetchableStore\n      .withView[LocaleEntityId, SimClustersEmbeddingView, ThriftSimClustersEmbedding](\n        stratoClient,\n        SimclustersLocaleEntityIdColPath,\n        embeddingColumnView)\n      .mapValues(SimClustersEmbedding(_))\n    val embeddingIdAsKeyStore = rawStore.composeKeyMapping[SimClustersEmbeddingId] {\n      case SimClustersEmbeddingId(_, _, InternalId.LocaleEntityId(localeEntityId)) =>\n        localeEntityId\n    }\n\n    addCacheLayer(embeddingIdAsKeyStore, embeddingColumnView)\n  }\n\n  private def addCacheLayer[K](\n    rawStore: ReadableStore[K, SimClustersEmbedding],\n    embeddingColumnView: SimClustersEmbeddingView,\n  ): ReadableStore[K, SimClustersEmbedding] = {\n    // Add in-memory caching based on ClientConfig\n    val inMemCacheParams = clientConfig.inMemoryCacheConfig\n      .getCacheSetup(embeddingColumnView.embeddingType, embeddingColumnView.modelVersion)\n\n    val statsPerStore = stats\n      .scope(embeddingColumnView.embeddingType.name).scope(embeddingColumnView.modelVersion.name)\n\n    inMemCacheParams match {\n      case DisabledInMemoryCacheParams =>\n        ObservedReadableStore(\n          store = rawStore\n        )(statsPerStore)\n      case EnabledInMemoryCacheParams(ttl, maxKeys, cacheName) =>\n        ObservedCachedReadableStore.from[K, SimClustersEmbedding](\n          rawStore,\n          ttl = ttl,\n          maxKeys = maxKeys,\n          cacheName = cacheName,\n          windowSize = 10000L\n        )(statsPerStore)\n    }\n  }\n\n}\n"
  },
  {
    "path": "representation-manager/client/src/main/scala/com/twitter/representation_manager/config/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-thrift-client\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/common\",\n        \"representation-manager/server/src/main/thrift:thrift-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n)\n"
  },
  {
    "path": "representation-manager/client/src/main/scala/com/twitter/representation_manager/config/ClientConfig.scala",
    "content": "package com.twitter.representation_manager.config\n\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\n\n/*\n * This is RMS client config class.\n * We only support setting up in memory cache params for now, but we expect to enable other\n * customisations in the near future e.g. request timeout\n *\n * --------------------------------------------\n * PLEASE NOTE:\n * Having in-memory cache is not necessarily a free performance win, anyone considering it should\n * investigate rather than blindly enabling it\n * */\nclass ClientConfig(inMemCacheParamsOverrides: Map[\n  (EmbeddingType, ModelVersion),\n  InMemoryCacheParams\n] = Map.empty) {\n  // In memory cache config per embedding\n  val inMemCacheParams = DefaultInMemoryCacheConfig.cacheParamsMap ++ inMemCacheParamsOverrides\n  val inMemoryCacheConfig = new InMemoryCacheConfig(inMemCacheParams)\n}\n\nobject DefaultClientConfig extends ClientConfig\n"
  },
  {
    "path": "representation-manager/client/src/main/scala/com/twitter/representation_manager/config/InMemoryCacheConfig.scala",
    "content": "package com.twitter.representation_manager.config\n\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.util.Duration\n\n/*\n * --------------------------------------------\n * PLEASE NOTE:\n * Having in-memory cache is not necessarily a free performance win, anyone considering it should\n * investigate rather than blindly enabling it\n * --------------------------------------------\n * */\n\nsealed trait InMemoryCacheParams\n\n/*\n * This holds params that is required to set up a in-mem cache for a single embedding store\n */\ncase class EnabledInMemoryCacheParams(\n  ttl: Duration,\n  maxKeys: Int,\n  cacheName: String)\n  extends InMemoryCacheParams\nobject DisabledInMemoryCacheParams extends InMemoryCacheParams\n\n/*\n * This is the class for the in-memory cache config. Client could pass in their own cacheParamsMap to\n * create a new InMemoryCacheConfig instead of using the DefaultInMemoryCacheConfig object below\n * */\nclass InMemoryCacheConfig(\n  cacheParamsMap: Map[\n    (EmbeddingType, ModelVersion),\n    InMemoryCacheParams\n  ] = Map.empty) {\n\n  def getCacheSetup(\n    embeddingType: EmbeddingType,\n    modelVersion: ModelVersion\n  ): InMemoryCacheParams = {\n    // When requested embedding type doesn't exist, we return DisabledInMemoryCacheParams\n    cacheParamsMap.getOrElse((embeddingType, modelVersion), DisabledInMemoryCacheParams)\n  }\n}\n\n/*\n * Default config for the in-memory cache\n * Clients can directly import and use this one if they don't want to set up a customised config\n * */\nobject DefaultInMemoryCacheConfig extends InMemoryCacheConfig {\n  // set default to no in-memory caching\n  val cacheParamsMap = Map.empty\n}\n"
  },
  {
    "path": "representation-manager/server/BUILD",
    "content": "jvm_binary(\n    name = \"bin\",\n    basename = \"representation-manager\",\n    main = \"com.twitter.representation_manager.RepresentationManagerFedServerMain\",\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-logback/src/main/scala\",\n        \"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback\",\n        \"representation-manager/server/src/main/resources\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager\",\n        \"twitter-server/logback-classic/src/main/scala\",\n    ],\n)\n\n#  Aurora Workflows build phase convention requires a jvm_app named with ${project-name}-app\njvm_app(\n    name = \"representation-manager-app\",\n    archive = \"zip\",\n    binary = \":bin\",\n)\n"
  },
  {
    "path": "representation-manager/server/src/main/resources/BUILD",
    "content": "resources(\n    sources = [\n        \"*.xml\",\n        \"config/*.yml\",\n    ],\n    tags = [\"bazel-compatible\"],\n)\n"
  },
  {
    "path": "representation-manager/server/src/main/resources/config/decider.yml",
    "content": "# ---------- traffic percentage by embedding type and model version ----------\n# Decider strings are build dynamically following the rule in there\n# i.e. s\"enable_${embeddingType.name}_${modelVersion.name}\"\n# Hence this should be updated accordingly if usage is changed in the embedding stores\n\n# Tweet embeddings\n\"enable_LogFavBasedTweet_Model20m145k2020\":\n  comment: \"Enable x% read traffic (0<=x<=10000, e.g. 1000=10%) for LogFavBasedTweet - Model20m145k2020. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_LogFavBasedTweet_Model20m145kUpdated\":\n  comment: \"Enable x% read traffic (0<=x<=10000, e.g. 1000=10%) for LogFavBasedTweet - Model20m145kUpdated. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_LogFavLongestL2EmbeddingTweet_Model20m145k2020\":\n  comment: \"Enable x% read traffic (0<=x<=10000, e.g. 1000=10%) for LogFavLongestL2EmbeddingTweet - Model20m145k2020. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_LogFavLongestL2EmbeddingTweet_Model20m145kUpdated\":\n  comment: \"Enable x% read traffic (0<=x<=10000, e.g. 1000=10%) for LogFavLongestL2EmbeddingTweet - Model20m145kUpdated. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n# Topic embeddings\n\"enable_FavTfgTopic_Model20m145k2020\":\n  comment: \"Enable the read traffic to FavTfgTopic - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_LogFavBasedKgoApeTopic_Model20m145k2020\":\n  comment: \"Enable the read traffic to LogFavBasedKgoApeTopic - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n# User embeddings - KnownFor\n\"enable_FavBasedProducer_Model20m145kUpdated\":\n  comment: \"Enable the read traffic to FavBasedProducer - Model20m145kUpdated from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_FavBasedProducer_Model20m145k2020\":\n  comment: \"Enable the read traffic to FavBasedProducer - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_FollowBasedProducer_Model20m145k2020\":\n  comment: \"Enable the read traffic to FollowBasedProducer - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_AggregatableFavBasedProducer_Model20m145kUpdated\":\n  comment: \"Enable the read traffic to AggregatableFavBasedProducer - Model20m145kUpdated from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_AggregatableFavBasedProducer_Model20m145k2020\":\n  comment: \"Enable the read traffic to AggregatableFavBasedProducer - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_AggregatableLogFavBasedProducer_Model20m145kUpdated\":\n  comment: \"Enable the read traffic to AggregatableLogFavBasedProducer - Model20m145kUpdated from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_AggregatableLogFavBasedProducer_Model20m145k2020\":\n  comment: \"Enable the read traffic to AggregatableLogFavBasedProducer - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_RelaxedAggregatableLogFavBasedProducer_Model20m145kUpdated:\n  comment: \"Enable the read traffic to RelaxedAggregatableLogFavBasedProducer - Model20m145kUpdated from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_RelaxedAggregatableLogFavBasedProducer_Model20m145k2020:\n  comment: \"Enable the read traffic to RelaxedAggregatableLogFavBasedProducer - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n# User embeddings - InterestedIn\n\"enable_LogFavBasedUserInterestedInFromAPE_Model20m145k2020\":\n  comment: \"Enable the read traffic to LogFavBasedUserInterestedInFromAPE - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_FollowBasedUserInterestedInFromAPE_Model20m145k2020\":\n  comment: \"Enable the read traffic to FollowBasedUserInterestedInFromAPE - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_FavBasedUserInterestedIn_Model20m145kUpdated\":\n  comment: \"Enable the read traffic to FavBasedUserInterestedIn - Model20m145kUpdated from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_FavBasedUserInterestedIn_Model20m145k2020\":\n  comment: \"Enable the read traffic to FavBasedUserInterestedIn - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_FollowBasedUserInterestedIn_Model20m145k2020\":\n  comment: \"Enable the read traffic to FollowBasedUserInterestedIn - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_LogFavBasedUserInterestedIn_Model20m145k2020\":\n  comment: \"Enable the read traffic to LogFavBasedUserInterestedIn - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_FavBasedUserInterestedInFromPE_Model20m145kUpdated\":\n  comment: \"Enable the read traffic to FavBasedUserInterestedInFromPE - Model20m145kUpdated from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_FilteredUserInterestedIn_Model20m145kUpdated\":\n  comment: \"Enable the read traffic to FilteredUserInterestedIn - Model20m145kUpdated from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_FilteredUserInterestedIn_Model20m145k2020\":\n  comment: \"Enable the read traffic to FilteredUserInterestedIn - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_FilteredUserInterestedInFromPE_Model20m145kUpdated\":\n  comment: \"Enable the read traffic to FilteredUserInterestedInFromPE - Model20m145kUpdated from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_UnfilteredUserInterestedIn_Model20m145kUpdated\":\n  comment: \"Enable the read traffic to UnfilteredUserInterestedIn - Model20m145kUpdated from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_UnfilteredUserInterestedIn_Model20m145k2020\":\n  comment: \"Enable the read traffic to UnfilteredUserInterestedIn - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_UserNextInterestedIn_Model20m145k2020\":\n  comment: \"Enable the read traffic to UserNextInterestedIn - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_LogFavBasedUserInterestedMaxpoolingAddressBookFromIIAPE_Model20m145k2020\":\n  comment: \"Enable the read traffic to LogFavBasedUserInterestedMaxpoolingAddressBookFromIIAPE - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_LogFavBasedUserInterestedAverageAddressBookFromIIAPE_Model20m145k2020\":\n  comment: \"Enable the read traffic to LogFavBasedUserInterestedAverageAddressBookFromIIAPE - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_LogFavBasedUserInterestedBooktypeMaxpoolingAddressBookFromIIAPE_Model20m145k2020\":\n  comment: \"Enable the read traffic to LogFavBasedUserInterestedMaxpoolingAddressBookFromIIAPE - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_LogFavBasedUserInterestedLargestDimMaxpoolingAddressBookFromIIAPE_Model20m145k2020\":\n  comment: \"Enable the read traffic to LogFavBasedUserInterestedAverageAddressBookFromIIAPE - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020\":\n  comment: \"Enable the read traffic to LogFavBasedUserInterestedMaxpoolingAddressBookFromIIAPE - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\"enable_LogFavBasedUserInterestedConnectedMaxpoolingAddressBookFromIIAPE_Model20m145k2020\":\n  comment: \"Enable the read traffic to LogFavBasedUserInterestedAverageAddressBookFromIIAPE - Model20m145k2020 from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n# ---------- load shedding by caller id ----------\n# To create a new decider, add here with the same format and caller's details :\n# \"representation-manager_load_shed_by_caller_id_twtr:{{role}}:{{name}}:{{environment}}:{{cluster}}\"\n# All the deciders below are generated by this script:\n# ./strato/bin/fed deciders representation-manager --service-role=representation-manager --service-name=representation-manager\n# If you need to run the script and paste the output, add ONLY the prod deciders here.\n\"representation-manager_load_shed_by_caller_id_all\":\n  comment: \"Reject all traffic from caller id: all\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:cr-mixer:cr-mixer:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:cr-mixer:cr-mixer:prod:atla\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:cr-mixer:cr-mixer:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:cr-mixer:cr-mixer:prod:pdxa\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:simclusters-ann:simclusters-ann-1:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:simclusters-ann:simclusters-ann-1:prod:atla\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:simclusters-ann:simclusters-ann-1:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:simclusters-ann:simclusters-ann-1:prod:pdxa\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:simclusters-ann:simclusters-ann-3:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:simclusters-ann:simclusters-ann-3:prod:atla\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:simclusters-ann:simclusters-ann-3:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:simclusters-ann:simclusters-ann-3:prod:pdxa\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:simclusters-ann:simclusters-ann-4:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:simclusters-ann:simclusters-ann-4:prod:atla\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:simclusters-ann:simclusters-ann-4:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:simclusters-ann:simclusters-ann-4:prod:pdxa\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:simclusters-ann:simclusters-ann-experimental:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:simclusters-ann:simclusters-ann-experimental:prod:atla\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:simclusters-ann:simclusters-ann-experimental:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:simclusters-ann:simclusters-ann-experimental:prod:pdxa\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:simclusters-ann:simclusters-ann:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:simclusters-ann:simclusters-ann:prod:atla\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:simclusters-ann:simclusters-ann:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:simclusters-ann:simclusters-ann:prod:pdxa\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:stratostore:stratoapi:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:stratostore:stratoapi:prod:atla\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:stratostore:stratoserver:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:stratostore:stratoserver:prod:atla\"\n  default_availability: 0\n\n\"representation-manager_load_shed_by_caller_id_twtr:svc:stratostore:stratoserver:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:stratostore:stratoserver:prod:pdxa\"\n  default_availability: 0\n\n# ---------- Dark Traffic Proxy ----------\nrepresentation-manager_forward_dark_traffic:\n  comment: \"Defines the percentage of traffic to forward to diffy-proxy. Set to 0 to disable dark traffic forwarding\"\n  default_availability: 0\n"
  },
  {
    "path": "representation-manager/server/src/main/resources/logback.xml",
    "content": "<configuration>\n    <shutdownHook class=\"ch.qos.logback.core.hook.DelayingShutdownHook\"/>\n\n    <!-- ===================================================== -->\n    <!-- Service Config -->\n    <!-- ===================================================== -->\n    <property name=\"DEFAULT_SERVICE_PATTERN\"\n              value=\"%-16X{traceId} %-12X{clientId:--} %-16X{method} %-25logger{0} %msg\"/>\n\n    <property name=\"DEFAULT_ACCESS_PATTERN\"\n              value=\"%msg\"/>\n\n    <!-- ===================================================== -->\n    <!-- Common Config -->\n    <!-- ===================================================== -->\n\n    <!-- JUL/JDK14 to Logback bridge -->\n    <contextListener class=\"ch.qos.logback.classic.jul.LevelChangePropagator\">\n        <resetJUL>true</resetJUL>\n    </contextListener>\n\n    <!-- ====================================================================================== -->\n    <!-- NOTE: The following appenders use a simple TimeBasedRollingPolicy configuration.       -->\n    <!--       You may want to consider using a more advanced SizeAndTimeBasedRollingPolicy.    -->\n    <!--       See: https://logback.qos.ch/manual/appenders.html#SizeAndTimeBasedRollingPolicy  -->\n    <!-- ====================================================================================== -->\n\n    <!-- Service Log (rollover daily, keep maximum of 21 days of gzip compressed logs) -->\n    <appender name=\"SERVICE\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n        <file>${log.service.output}</file>\n        <rollingPolicy class=\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\">\n            <!-- daily rollover -->\n            <fileNamePattern>${log.service.output}.%d.gz</fileNamePattern>\n            <!-- the maximum total size of all the log files -->\n            <totalSizeCap>3GB</totalSizeCap>\n            <!-- keep maximum 21 days' worth of history -->\n            <maxHistory>21</maxHistory>\n            <cleanHistoryOnStart>true</cleanHistoryOnStart>\n        </rollingPolicy>\n        <encoder>\n            <pattern>%date %.-3level ${DEFAULT_SERVICE_PATTERN}%n</pattern>\n        </encoder>\n    </appender>\n\n    <!-- Access Log (rollover daily, keep maximum of 21 days of gzip compressed logs) -->\n    <appender name=\"ACCESS\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n        <file>${log.access.output}</file>\n        <rollingPolicy class=\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\">\n            <!-- daily rollover -->\n            <fileNamePattern>${log.access.output}.%d.gz</fileNamePattern>\n            <!-- the maximum total size of all the log files -->\n            <totalSizeCap>100MB</totalSizeCap>\n            <!-- keep maximum 7 days' worth of history -->\n            <maxHistory>7</maxHistory>\n            <cleanHistoryOnStart>true</cleanHistoryOnStart>\n        </rollingPolicy>\n        <encoder>\n            <pattern>${DEFAULT_ACCESS_PATTERN}%n</pattern>\n        </encoder>\n    </appender>\n\n    <!--LogLens -->\n    <appender name=\"LOGLENS\" class=\"com.twitter.loglens.logback.LoglensAppender\">\n        <mdcAdditionalContext>true</mdcAdditionalContext>\n        <category>${log.lens.category}</category>\n        <index>${log.lens.index}</index>\n        <tag>${log.lens.tag}/service</tag>\n        <encoder>\n            <pattern>%msg</pattern>\n        </encoder>\n    </appender>\n\n    <!-- LogLens Access -->\n    <appender name=\"LOGLENS-ACCESS\" class=\"com.twitter.loglens.logback.LoglensAppender\">\n        <mdcAdditionalContext>true</mdcAdditionalContext>\n        <category>${log.lens.category}</category>\n        <index>${log.lens.index}</index>\n        <tag>${log.lens.tag}/access</tag>\n        <encoder>\n            <pattern>%msg</pattern>\n        </encoder>\n    </appender>\n\n    <!-- Pipeline Execution Logs -->\n    <appender name=\"ALLOW-LISTED-PIPELINE-EXECUTIONS\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n        <file>allow_listed_pipeline_executions.log</file>\n        <rollingPolicy class=\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\">\n            <!-- daily rollover -->\n            <fileNamePattern>allow_listed_pipeline_executions.log.%d.gz</fileNamePattern>\n             <!-- the maximum total size of all the log files -->\n            <totalSizeCap>100MB</totalSizeCap>\n            <!-- keep maximum 7 days' worth of history -->\n            <maxHistory>7</maxHistory>\n            <cleanHistoryOnStart>true</cleanHistoryOnStart>\n        </rollingPolicy>\n        <encoder>\n            <pattern>%date %.-3level ${DEFAULT_SERVICE_PATTERN}%n</pattern>\n        </encoder>\n    </appender>\n\n    <!-- ===================================================== -->\n    <!-- Primary Async Appenders -->\n    <!-- ===================================================== -->\n\n    <property name=\"async_queue_size\" value=\"${queue.size:-50000}\"/>\n    <property name=\"async_max_flush_time\" value=\"${max.flush.time:-0}\"/>\n\n    <appender name=\"ASYNC-SERVICE\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"SERVICE\"/>\n    </appender>\n\n    <appender name=\"ASYNC-ACCESS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"ACCESS\"/>\n    </appender>\n\n    <appender name=\"ASYNC-ALLOW-LISTED-PIPELINE-EXECUTIONS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"ALLOW-LISTED-PIPELINE-EXECUTIONS\"/>\n    </appender>\n\n    <appender name=\"ASYNC-LOGLENS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"LOGLENS\"/>\n    </appender>\n\n    <appender name=\"ASYNC-LOGLENS-ACCESS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"LOGLENS-ACCESS\"/>\n    </appender>\n\n    <!-- ===================================================== -->\n    <!-- Package Config -->\n    <!-- ===================================================== -->\n\n    <!-- Per-Package Config -->\n    <logger name=\"com.twitter\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.wilyns\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.configbus.client.file\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.finagle.mux\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.finagle.serverset2\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.logging.ScribeHandler\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.zookeeper.client.internal\" level=\"INHERITED\"/>\n\n    <!-- Root Config -->\n    <!-- For all logs except access logs, disable logging below log_level level by default. This can be overriden in the per-package loggers, and dynamically in the admin panel of individual instances. -->\n    <root level=\"${log_level:-INFO}\">\n        <appender-ref ref=\"ASYNC-SERVICE\"/>\n        <appender-ref ref=\"ASYNC-LOGLENS\"/>\n    </root>\n\n    <!-- Access Logging -->\n    <!-- Access logs are turned off by default -->\n    <logger name=\"com.twitter.finatra.thrift.filters.AccessLoggingFilter\" level=\"OFF\" additivity=\"false\">\n        <appender-ref ref=\"ASYNC-ACCESS\"/>\n        <appender-ref ref=\"ASYNC-LOGLENS-ACCESS\"/>\n    </logger>\n\n</configuration>\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-thrift-client\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/columns/topic\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/columns/tweet\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/columns/user\",\n        \"strato/src/main/scala/com/twitter/strato/fed\",\n        \"strato/src/main/scala/com/twitter/strato/fed/server\",\n    ],\n)\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/RepresentationManagerFedServer.scala",
    "content": "package com.twitter.representation_manager\n\nimport com.google.inject.Module\nimport com.twitter.inject.thrift.modules.ThriftClientIdModule\nimport com.twitter.representation_manager.columns.topic.LocaleEntityIdSimClustersEmbeddingCol\nimport com.twitter.representation_manager.columns.topic.TopicIdSimClustersEmbeddingCol\nimport com.twitter.representation_manager.columns.tweet.TweetSimClustersEmbeddingCol\nimport com.twitter.representation_manager.columns.user.UserSimClustersEmbeddingCol\nimport com.twitter.representation_manager.modules.CacheModule\nimport com.twitter.representation_manager.modules.InterestsThriftClientModule\nimport com.twitter.representation_manager.modules.LegacyRMSConfigModule\nimport com.twitter.representation_manager.modules.StoreModule\nimport com.twitter.representation_manager.modules.TimerModule\nimport com.twitter.representation_manager.modules.UttClientModule\nimport com.twitter.strato.fed._\nimport com.twitter.strato.fed.server._\n\nobject RepresentationManagerFedServerMain extends RepresentationManagerFedServer\n\ntrait RepresentationManagerFedServer extends StratoFedServer {\n  override def dest: String = \"/s/representation-manager/representation-manager\"\n  override val modules: Seq[Module] =\n    Seq(\n      CacheModule,\n      InterestsThriftClientModule,\n      LegacyRMSConfigModule,\n      StoreModule,\n      ThriftClientIdModule,\n      TimerModule,\n      UttClientModule\n    )\n\n  override def columns: Seq[Class[_ <: StratoFed.Column]] =\n    Seq(\n      classOf[TweetSimClustersEmbeddingCol],\n      classOf[UserSimClustersEmbeddingCol],\n      classOf[TopicIdSimClustersEmbeddingCol],\n      classOf[LocaleEntityIdSimClustersEmbeddingCol]\n    )\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/columns/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"strato/src/main/scala/com/twitter/strato/fed\",\n        \"strato/src/main/scala/com/twitter/strato/fed/server\",\n    ],\n)\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/columns/ColumnConfigBase.scala",
    "content": "package com.twitter.representation_manager.columns\n\nimport com.twitter.strato.access.Access.LdapGroup\nimport com.twitter.strato.config.ContactInfo\nimport com.twitter.strato.config.FromColumns\nimport com.twitter.strato.config.Has\nimport com.twitter.strato.config.Prefix\nimport com.twitter.strato.config.ServiceIdentifierPattern\n\nobject ColumnConfigBase {\n\n  /****************** Internal permissions *******************/\n  val recosPermissions: Seq[com.twitter.strato.config.Policy] = Seq()\n\n  /****************** External permissions *******************/\n  // This is used to grant limited access to members outside of RP team.\n  val externalPermissions: Seq[com.twitter.strato.config.Policy] = Seq()\n\n  val contactInfo: ContactInfo = ContactInfo(\n    description = \"Please contact Relevance Platform for more details\",\n    contactEmail = \"no-reply@twitter.com\",\n    ldapGroup = \"ldap\",\n    jiraProject = \"JIRA\",\n    links = Seq(\"http://go/rms-runbook\")\n  )\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/columns/topic/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/columns\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/modules\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/store\",\n        \"representation-manager/server/src/main/thrift:thrift-scala\",\n        \"strato/src/main/scala/com/twitter/strato/fed\",\n        \"strato/src/main/scala/com/twitter/strato/fed/server\",\n    ],\n)\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/columns/topic/LocaleEntityIdSimClustersEmbeddingCol.scala",
    "content": "package com.twitter.representation_manager.columns.topic\n\nimport com.twitter.representation_manager.columns.ColumnConfigBase\nimport com.twitter.representation_manager.store.TopicSimClustersEmbeddingStore\nimport com.twitter.representation_manager.thriftscala.SimClustersEmbeddingView\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbedding\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.LocaleEntityId\nimport com.twitter.stitch\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.storehaus.StitchOfReadableStore\nimport com.twitter.strato.catalog.OpMetadata\nimport com.twitter.strato.config.AnyOf\nimport com.twitter.strato.config.ContactInfo\nimport com.twitter.strato.config.FromColumns\nimport com.twitter.strato.config.Policy\nimport com.twitter.strato.config.Prefix\nimport com.twitter.strato.data.Conv\nimport com.twitter.strato.data.Description.PlainText\nimport com.twitter.strato.data.Lifecycle\nimport com.twitter.strato.fed._\nimport com.twitter.strato.thrift.ScroogeConv\nimport javax.inject.Inject\n\nclass LocaleEntityIdSimClustersEmbeddingCol @Inject() (\n  embeddingStore: TopicSimClustersEmbeddingStore)\n    extends StratoFed.Column(\n      \"recommendations/representation_manager/simClustersEmbedding.LocaleEntityId\")\n    with StratoFed.Fetch.Stitch {\n\n  private val storeStitch: SimClustersEmbeddingId => Stitch[SimClustersEmbedding] =\n    StitchOfReadableStore(embeddingStore.topicSimClustersEmbeddingStore.mapValues(_.toThrift))\n\n  val colPermissions: Seq[com.twitter.strato.config.Policy] =\n    ColumnConfigBase.recosPermissions ++ ColumnConfigBase.externalPermissions :+ FromColumns(\n      Set(\n        Prefix(\"ml/featureStore/simClusters\"),\n      ))\n\n  override val policy: Policy = AnyOf({\n    colPermissions\n  })\n\n  override type Key = LocaleEntityId\n  override type View = SimClustersEmbeddingView\n  override type Value = SimClustersEmbedding\n\n  override val keyConv: Conv[Key] = ScroogeConv.fromStruct[LocaleEntityId]\n  override val viewConv: Conv[View] = ScroogeConv.fromStruct[SimClustersEmbeddingView]\n  override val valueConv: Conv[Value] = ScroogeConv.fromStruct[SimClustersEmbedding]\n\n  override val contactInfo: ContactInfo = ColumnConfigBase.contactInfo\n\n  override val metadata: OpMetadata = OpMetadata(\n    lifecycle = Some(Lifecycle.Production),\n    description = Some(\n      PlainText(\n        \"The Topic SimClusters Embedding Endpoint in Representation Management Service with LocaleEntityId.\" +\n          \" TDD: http://go/rms-tdd\"))\n  )\n\n  override def fetch(key: Key, view: View): Stitch[Result[Value]] = {\n    val embeddingId = SimClustersEmbeddingId(\n      view.embeddingType,\n      view.modelVersion,\n      InternalId.LocaleEntityId(key)\n    )\n\n    storeStitch(embeddingId)\n      .map(embedding => found(embedding))\n      .handle {\n        case stitch.NotFound => missing\n      }\n  }\n\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/columns/topic/TopicIdSimClustersEmbeddingCol.scala",
    "content": "package com.twitter.representation_manager.columns.topic\n\nimport com.twitter.representation_manager.columns.ColumnConfigBase\nimport com.twitter.representation_manager.store.TopicSimClustersEmbeddingStore\nimport com.twitter.representation_manager.thriftscala.SimClustersEmbeddingView\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbedding\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.TopicId\nimport com.twitter.stitch\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.storehaus.StitchOfReadableStore\nimport com.twitter.strato.catalog.OpMetadata\nimport com.twitter.strato.config.AnyOf\nimport com.twitter.strato.config.ContactInfo\nimport com.twitter.strato.config.FromColumns\nimport com.twitter.strato.config.Policy\nimport com.twitter.strato.config.Prefix\nimport com.twitter.strato.data.Conv\nimport com.twitter.strato.data.Description.PlainText\nimport com.twitter.strato.data.Lifecycle\nimport com.twitter.strato.fed._\nimport com.twitter.strato.thrift.ScroogeConv\nimport javax.inject.Inject\n\nclass TopicIdSimClustersEmbeddingCol @Inject() (embeddingStore: TopicSimClustersEmbeddingStore)\n    extends StratoFed.Column(\"recommendations/representation_manager/simClustersEmbedding.TopicId\")\n    with StratoFed.Fetch.Stitch {\n\n  private val storeStitch: SimClustersEmbeddingId => Stitch[SimClustersEmbedding] =\n    StitchOfReadableStore(embeddingStore.topicSimClustersEmbeddingStore.mapValues(_.toThrift))\n\n  val colPermissions: Seq[com.twitter.strato.config.Policy] =\n    ColumnConfigBase.recosPermissions ++ ColumnConfigBase.externalPermissions :+ FromColumns(\n      Set(\n        Prefix(\"ml/featureStore/simClusters\"),\n      ))\n\n  override val policy: Policy = AnyOf({\n    colPermissions\n  })\n\n  override type Key = TopicId\n  override type View = SimClustersEmbeddingView\n  override type Value = SimClustersEmbedding\n\n  override val keyConv: Conv[Key] = ScroogeConv.fromStruct[TopicId]\n  override val viewConv: Conv[View] = ScroogeConv.fromStruct[SimClustersEmbeddingView]\n  override val valueConv: Conv[Value] = ScroogeConv.fromStruct[SimClustersEmbedding]\n\n  override val contactInfo: ContactInfo = ColumnConfigBase.contactInfo\n\n  override val metadata: OpMetadata = OpMetadata(\n    lifecycle = Some(Lifecycle.Production),\n    description = Some(PlainText(\n      \"The Topic SimClusters Embedding Endpoint in Representation Management Service with TopicId.\" +\n        \" TDD: http://go/rms-tdd\"))\n  )\n\n  override def fetch(key: Key, view: View): Stitch[Result[Value]] = {\n    val embeddingId = SimClustersEmbeddingId(\n      view.embeddingType,\n      view.modelVersion,\n      InternalId.TopicId(key)\n    )\n\n    storeStitch(embeddingId)\n      .map(embedding => found(embedding))\n      .handle {\n        case stitch.NotFound => missing\n      }\n  }\n\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/columns/tweet/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/columns\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/modules\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/store\",\n        \"representation-manager/server/src/main/thrift:thrift-scala\",\n        \"strato/src/main/scala/com/twitter/strato/fed\",\n        \"strato/src/main/scala/com/twitter/strato/fed/server\",\n    ],\n)\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/columns/tweet/TweetSimClustersEmbeddingCol.scala",
    "content": "package com.twitter.representation_manager.columns.tweet\n\nimport com.twitter.representation_manager.columns.ColumnConfigBase\nimport com.twitter.representation_manager.store.TweetSimClustersEmbeddingStore\nimport com.twitter.representation_manager.thriftscala.SimClustersEmbeddingView\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbedding\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.stitch\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.storehaus.StitchOfReadableStore\nimport com.twitter.strato.catalog.OpMetadata\nimport com.twitter.strato.config.AnyOf\nimport com.twitter.strato.config.ContactInfo\nimport com.twitter.strato.config.FromColumns\nimport com.twitter.strato.config.Policy\nimport com.twitter.strato.config.Prefix\nimport com.twitter.strato.data.Conv\nimport com.twitter.strato.data.Description.PlainText\nimport com.twitter.strato.data.Lifecycle\nimport com.twitter.strato.fed._\nimport com.twitter.strato.thrift.ScroogeConv\nimport javax.inject.Inject\n\nclass TweetSimClustersEmbeddingCol @Inject() (embeddingStore: TweetSimClustersEmbeddingStore)\n    extends StratoFed.Column(\"recommendations/representation_manager/simClustersEmbedding.Tweet\")\n    with StratoFed.Fetch.Stitch {\n\n  private val storeStitch: SimClustersEmbeddingId => Stitch[SimClustersEmbedding] =\n    StitchOfReadableStore(embeddingStore.tweetSimClustersEmbeddingStore.mapValues(_.toThrift))\n\n  val colPermissions: Seq[com.twitter.strato.config.Policy] =\n    ColumnConfigBase.recosPermissions ++ ColumnConfigBase.externalPermissions :+ FromColumns(\n      Set(\n        Prefix(\"ml/featureStore/simClusters\"),\n      ))\n\n  override val policy: Policy = AnyOf({\n    colPermissions\n  })\n\n  override type Key = Long // TweetId\n  override type View = SimClustersEmbeddingView\n  override type Value = SimClustersEmbedding\n\n  override val keyConv: Conv[Key] = Conv.long\n  override val viewConv: Conv[View] = ScroogeConv.fromStruct[SimClustersEmbeddingView]\n  override val valueConv: Conv[Value] = ScroogeConv.fromStruct[SimClustersEmbedding]\n\n  override val contactInfo: ContactInfo = ColumnConfigBase.contactInfo\n\n  override val metadata: OpMetadata = OpMetadata(\n    lifecycle = Some(Lifecycle.Production),\n    description = Some(\n      PlainText(\"The Tweet SimClusters Embedding Endpoint in Representation Management Service.\" +\n        \" TDD: http://go/rms-tdd\"))\n  )\n\n  override def fetch(key: Key, view: View): Stitch[Result[Value]] = {\n    val embeddingId = SimClustersEmbeddingId(\n      view.embeddingType,\n      view.modelVersion,\n      InternalId.TweetId(key)\n    )\n\n    storeStitch(embeddingId)\n      .map(embedding => found(embedding))\n      .handle {\n        case stitch.NotFound => missing\n      }\n  }\n\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/columns/user/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/columns\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/modules\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/store\",\n        \"representation-manager/server/src/main/thrift:thrift-scala\",\n        \"strato/src/main/scala/com/twitter/strato/fed\",\n        \"strato/src/main/scala/com/twitter/strato/fed/server\",\n    ],\n)\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/columns/user/UserSimClustersEmbeddingCol.scala",
    "content": "package com.twitter.representation_manager.columns.user\n\nimport com.twitter.representation_manager.columns.ColumnConfigBase\nimport com.twitter.representation_manager.store.UserSimClustersEmbeddingStore\nimport com.twitter.representation_manager.thriftscala.SimClustersEmbeddingView\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbedding\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.stitch\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.storehaus.StitchOfReadableStore\nimport com.twitter.strato.catalog.OpMetadata\nimport com.twitter.strato.config.AnyOf\nimport com.twitter.strato.config.ContactInfo\nimport com.twitter.strato.config.FromColumns\nimport com.twitter.strato.config.Policy\nimport com.twitter.strato.config.Prefix\nimport com.twitter.strato.data.Conv\nimport com.twitter.strato.data.Description.PlainText\nimport com.twitter.strato.data.Lifecycle\nimport com.twitter.strato.fed._\nimport com.twitter.strato.thrift.ScroogeConv\nimport javax.inject.Inject\n\nclass UserSimClustersEmbeddingCol @Inject() (embeddingStore: UserSimClustersEmbeddingStore)\n    extends StratoFed.Column(\"recommendations/representation_manager/simClustersEmbedding.User\")\n    with StratoFed.Fetch.Stitch {\n\n  private val storeStitch: SimClustersEmbeddingId => Stitch[SimClustersEmbedding] =\n    StitchOfReadableStore(embeddingStore.userSimClustersEmbeddingStore.mapValues(_.toThrift))\n\n  val colPermissions: Seq[com.twitter.strato.config.Policy] =\n    ColumnConfigBase.recosPermissions ++ ColumnConfigBase.externalPermissions :+ FromColumns(\n      Set(\n        Prefix(\"ml/featureStore/simClusters\"),\n      ))\n\n  override val policy: Policy = AnyOf({\n    colPermissions\n  })\n\n  override type Key = Long // UserId\n  override type View = SimClustersEmbeddingView\n  override type Value = SimClustersEmbedding\n\n  override val keyConv: Conv[Key] = Conv.long\n  override val viewConv: Conv[View] = ScroogeConv.fromStruct[SimClustersEmbeddingView]\n  override val valueConv: Conv[Value] = ScroogeConv.fromStruct[SimClustersEmbedding]\n\n  override val contactInfo: ContactInfo = ColumnConfigBase.contactInfo\n\n  override val metadata: OpMetadata = OpMetadata(\n    lifecycle = Some(Lifecycle.Production),\n    description = Some(\n      PlainText(\"The User SimClusters Embedding Endpoint in Representation Management Service.\" +\n        \" TDD: http://go/rms-tdd\"))\n  )\n\n  override def fetch(key: Key, view: View): Stitch[Result[Value]] = {\n    val embeddingId = SimClustersEmbeddingId(\n      view.embeddingType,\n      view.modelVersion,\n      InternalId.UserId(key)\n    )\n\n    storeStitch(embeddingId)\n      .map(embedding => found(embedding))\n      .handle {\n        case stitch.NotFound => missing\n      }\n  }\n\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/common/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"decider/src/main/scala\",\n        \"finagle/finagle-memcached\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/injection\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/common/MemCacheConfig.scala",
    "content": "package com.twitter.representation_manager.common\n\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.relevance_platform.common.injection.LZ4Injection\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.common.SimClustersEmbeddingIdCacheKeyBuilder\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType._\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion._\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.{SimClustersEmbedding => ThriftSimClustersEmbedding}\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Duration\n\n/*\n * NOTE - ALL the cache configs here are just placeholders, NONE of them is used anyweher in RMS yet\n * */\nsealed trait MemCacheParams\nsealed trait MemCacheConfig\n\n/*\n * This holds params that is required to set up a memcache cache for a single embedding store\n * */\ncase class EnabledMemCacheParams(ttl: Duration) extends MemCacheParams\nobject DisabledMemCacheParams extends MemCacheParams\n\n/*\n * We use this MemcacheConfig as the single source to set up the memcache for all RMS use cases\n * NO OVERRIDE FROM CLIENT\n * */\nobject MemCacheConfig {\n  val keyHasher: KeyHasher = KeyHasher.FNV1A_64\n  val hashKeyPrefix: String = \"RMS\"\n  val simclustersEmbeddingCacheKeyBuilder =\n    SimClustersEmbeddingIdCacheKeyBuilder(keyHasher.hashKey, hashKeyPrefix)\n\n  val cacheParamsMap: Map[\n    (EmbeddingType, ModelVersion),\n    MemCacheParams\n  ] = Map(\n    // Tweet Embeddings\n    (LogFavBasedTweet, Model20m145kUpdated) -> EnabledMemCacheParams(ttl = 10.minutes),\n    (LogFavBasedTweet, Model20m145k2020) -> EnabledMemCacheParams(ttl = 10.minutes),\n    (LogFavLongestL2EmbeddingTweet, Model20m145kUpdated) -> EnabledMemCacheParams(ttl = 10.minutes),\n    (LogFavLongestL2EmbeddingTweet, Model20m145k2020) -> EnabledMemCacheParams(ttl = 10.minutes),\n    // User - KnownFor Embeddings\n    (FavBasedProducer, Model20m145kUpdated) -> EnabledMemCacheParams(ttl = 12.hours),\n    (FavBasedProducer, Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (FollowBasedProducer, Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (AggregatableLogFavBasedProducer, Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (RelaxedAggregatableLogFavBasedProducer, Model20m145kUpdated) -> EnabledMemCacheParams(ttl =\n      12.hours),\n    (RelaxedAggregatableLogFavBasedProducer, Model20m145k2020) -> EnabledMemCacheParams(ttl =\n      12.hours),\n    // User - InterestedIn Embeddings\n    (LogFavBasedUserInterestedInFromAPE, Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (FollowBasedUserInterestedInFromAPE, Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (FavBasedUserInterestedIn, Model20m145kUpdated) -> EnabledMemCacheParams(ttl = 12.hours),\n    (FavBasedUserInterestedIn, Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (FollowBasedUserInterestedIn, Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (LogFavBasedUserInterestedIn, Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (FavBasedUserInterestedInFromPE, Model20m145kUpdated) -> EnabledMemCacheParams(ttl = 12.hours),\n    (FilteredUserInterestedIn, Model20m145kUpdated) -> EnabledMemCacheParams(ttl = 12.hours),\n    (FilteredUserInterestedIn, Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (FilteredUserInterestedInFromPE, Model20m145kUpdated) -> EnabledMemCacheParams(ttl = 12.hours),\n    (UnfilteredUserInterestedIn, Model20m145kUpdated) -> EnabledMemCacheParams(ttl = 12.hours),\n    (UnfilteredUserInterestedIn, Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (UserNextInterestedIn, Model20m145k2020) -> EnabledMemCacheParams(ttl =\n      30.minutes), //embedding is updated every 2 hours, keeping it lower to avoid staleness\n    (\n      LogFavBasedUserInterestedMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (\n      LogFavBasedUserInterestedAverageAddressBookFromIIAPE,\n      Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (\n      LogFavBasedUserInterestedBooktypeMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (\n      LogFavBasedUserInterestedLargestDimMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (\n      LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (\n      LogFavBasedUserInterestedConnectedMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    // Topic Embeddings\n    (FavTfgTopic, Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n    (LogFavBasedKgoApeTopic, Model20m145k2020) -> EnabledMemCacheParams(ttl = 12.hours),\n  )\n\n  def getCacheSetup(\n    embeddingType: EmbeddingType,\n    modelVersion: ModelVersion\n  ): MemCacheParams = {\n    // When requested (embeddingType, modelVersion) doesn't exist, we return DisabledMemCacheParams\n    cacheParamsMap.getOrElse((embeddingType, modelVersion), DisabledMemCacheParams)\n  }\n\n  def getCacheKeyPrefix(embeddingType: EmbeddingType, modelVersion: ModelVersion) =\n    s\"${embeddingType.value}_${modelVersion.value}_\"\n\n  def getStatsName(embeddingType: EmbeddingType, modelVersion: ModelVersion) =\n    s\"${embeddingType.name}_${modelVersion.name}_mem_cache\"\n\n  /**\n   * Build a ReadableStore based on MemCacheConfig.\n   *\n   * If memcache is disabled, it will return a normal readable store wrapper of the rawStore,\n   * with SimClustersEmbedding as value;\n   * If memcache is enabled, it will return a ObservedMemcachedReadableStore wrapper of the rawStore,\n   * with memcache set up according to the EnabledMemCacheParams\n   * */\n  def buildMemCacheStoreForSimClustersEmbedding(\n    rawStore: ReadableStore[SimClustersEmbeddingId, ThriftSimClustersEmbedding],\n    cacheClient: Client,\n    embeddingType: EmbeddingType,\n    modelVersion: ModelVersion,\n    stats: StatsReceiver\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n    val cacheParams = getCacheSetup(embeddingType, modelVersion)\n    val store = cacheParams match {\n      case DisabledMemCacheParams => rawStore\n      case EnabledMemCacheParams(ttl) =>\n        val memCacheKeyPrefix = MemCacheConfig.getCacheKeyPrefix(\n          embeddingType,\n          modelVersion\n        )\n        val statsName = MemCacheConfig.getStatsName(\n          embeddingType,\n          modelVersion\n        )\n        ObservedMemcachedReadableStore.fromCacheClient(\n          backingStore = rawStore,\n          cacheClient = cacheClient,\n          ttl = ttl\n        )(\n          valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n          statsReceiver = stats.scope(statsName),\n          keyToString = { k => memCacheKeyPrefix + k.toString }\n        )\n    }\n    store.mapValues(SimClustersEmbedding(_))\n  }\n\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/common/RepresentationManagerDecider.scala",
    "content": "package com.twitter.representation_manager.common\n\nimport com.twitter.decider.Decider\nimport com.twitter.decider.RandomRecipient\nimport com.twitter.decider.Recipient\nimport com.twitter.simclusters_v2.common.DeciderGateBuilderWithIdHashing\nimport javax.inject.Inject\n\ncase class RepresentationManagerDecider @Inject() (decider: Decider) {\n\n  val deciderGateBuilder = new DeciderGateBuilderWithIdHashing(decider)\n\n  def isAvailable(feature: String, recipient: Option[Recipient]): Boolean = {\n    decider.isAvailable(feature, recipient)\n  }\n\n  /**\n   * When useRandomRecipient is set to false, the decider is either completely on or off.\n   * When useRandomRecipient is set to true, the decider is on for the specified % of traffic.\n   */\n  def isAvailable(feature: String, useRandomRecipient: Boolean = true): Boolean = {\n    if (useRandomRecipient) isAvailable(feature, Some(RandomRecipient))\n    else isAvailable(feature, None)\n  }\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/migration/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"content-recommender/server/src/main/scala/com/twitter/contentrecommender:representation-manager-deps\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/strato\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/util\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/injection\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/readablestore\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/common\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/store\",\n        \"src/scala/com/twitter/ml/api/embedding\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/scala/com/twitter/simclusters_v2/score\",\n        \"src/scala/com/twitter/simclusters_v2/summingbird/stores\",\n        \"src/scala/com/twitter/storehaus_internal/manhattan\",\n        \"src/scala/com/twitter/storehaus_internal/util\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"src/thrift/com/twitter/socialgraph:thrift-scala\",\n        \"storage/clients/manhattan/client/src/main/scala\",\n        \"tweetypie/src/scala/com/twitter/tweetypie/util\",\n    ],\n)\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/migration/LegacyRMS.scala",
    "content": "package com.twitter.representation_manager.migration\n\nimport com.twitter.bijection.Injection\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.contentrecommender.store.ApeEntityEmbeddingStore\nimport com.twitter.contentrecommender.store.InterestsOptOutStore\nimport com.twitter.contentrecommender.store.SemanticCoreTopicSeedStore\nimport com.twitter.contentrecommender.twistly\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.decider.Decider\nimport com.twitter.escherbird.util.uttclient.CacheConfigV2\nimport com.twitter.escherbird.util.uttclient.CachedUttClientV2\nimport com.twitter.escherbird.util.uttclient.UttClientCacheConfigsV2\nimport com.twitter.escherbird.utt.strato.thriftscala.Environment\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient.MtlsThriftMuxClientSyntax\nimport com.twitter.finagle.mux.ClientDiscardedRequestException\nimport com.twitter.finagle.service.ReqRep\nimport com.twitter.finagle.service.ResponseClass\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.frigate.common.util.SeqLongInjection\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.hermit.store.common.DeciderableReadableStore\nimport com.twitter.hermit.store.common.ObservedCachedReadableStore\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.interests.thriftscala.InterestsThriftService\nimport com.twitter.relevance_platform.common.injection.LZ4Injection\nimport com.twitter.relevance_platform.common.readablestore.ReadableStoreWithTimeout\nimport com.twitter.representation_manager.common.RepresentationManagerDecider\nimport com.twitter.representation_manager.store.DeciderConstants\nimport com.twitter.representation_manager.store.DeciderKey\nimport com.twitter.simclusters_v2.common.ModelVersions\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.common.SimClustersEmbeddingIdCacheKeyBuilder\nimport com.twitter.simclusters_v2.stores.SimClustersEmbeddingStore\nimport com.twitter.simclusters_v2.summingbird.stores.PersistentTweetEmbeddingStore\nimport com.twitter.simclusters_v2.summingbird.stores.ProducerClusterEmbeddingReadableStores\nimport com.twitter.simclusters_v2.summingbird.stores.UserInterestedInReadableStore\nimport com.twitter.simclusters_v2.thriftscala.ClustersUserIsInterestedIn\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType._\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion.Model20m145k2020\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion.Model20m145kUpdated\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersMultiEmbedding\nimport com.twitter.simclusters_v2.thriftscala.SimClustersMultiEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.{SimClustersEmbedding => ThriftSimClustersEmbedding}\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.Athena\nimport com.twitter.storehaus_internal.manhattan.ManhattanRO\nimport com.twitter.storehaus_internal.manhattan.ManhattanROConfig\nimport com.twitter.storehaus_internal.util.ApplicationID\nimport com.twitter.storehaus_internal.util.DatasetName\nimport com.twitter.storehaus_internal.util.HDFSPath\nimport com.twitter.strato.client.Strato\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.strato.thrift.ScroogeConvImplicits._\nimport com.twitter.tweetypie.util.UserId\nimport com.twitter.util.Duration\nimport com.twitter.util.Future\nimport com.twitter.util.Throw\nimport com.twitter.util.Timer\nimport javax.inject.Inject\nimport javax.inject.Named\nimport scala.reflect.ClassTag\n\nclass LegacyRMS @Inject() (\n  serviceIdentifier: ServiceIdentifier,\n  cacheClient: Client,\n  stats: StatsReceiver,\n  decider: Decider,\n  clientId: ClientId,\n  timer: Timer,\n  @Named(\"cacheHashKeyPrefix\") val cacheHashKeyPrefix: String = \"RMS\",\n  @Named(\"useContentRecommenderConfiguration\") val useContentRecommenderConfiguration: Boolean =\n    false) {\n\n  private val mhMtlsParams: ManhattanKVClientMtlsParams = ManhattanKVClientMtlsParams(\n    serviceIdentifier)\n  private val rmsDecider = RepresentationManagerDecider(decider)\n  val keyHasher: KeyHasher = KeyHasher.FNV1A_64\n\n  private val embeddingCacheKeyBuilder =\n    SimClustersEmbeddingIdCacheKeyBuilder(keyHasher.hashKey, cacheHashKeyPrefix)\n  private val statsReceiver = stats.scope(\"representation_management\")\n\n  // Strato client, default timeout = 280ms\n  val stratoClient: StratoClient =\n    Strato.client\n      .withMutualTls(serviceIdentifier)\n      .build()\n\n  // Builds ThriftMux client builder for Content-Recommender service\n  private def makeThriftClientBuilder(\n    requestTimeout: Duration\n  ): ThriftMux.Client = {\n    ThriftMux.client\n      .withClientId(clientId)\n      .withMutualTls(serviceIdentifier)\n      .withRequestTimeout(requestTimeout)\n      .withStatsReceiver(statsReceiver.scope(\"clnt\"))\n      .withResponseClassifier {\n        case ReqRep(_, Throw(_: ClientDiscardedRequestException)) => ResponseClass.Ignorable\n      }\n  }\n\n  private def makeThriftClient[ThriftServiceType: ClassTag](\n    dest: String,\n    label: String,\n    requestTimeout: Duration = 450.milliseconds\n  ): ThriftServiceType = {\n    makeThriftClientBuilder(requestTimeout)\n      .build[ThriftServiceType](dest, label)\n  }\n\n  /** *** SimCluster Embedding Stores ******/\n  implicit val simClustersEmbeddingIdInjection: Injection[SimClustersEmbeddingId, Array[Byte]] =\n    BinaryScalaCodec(SimClustersEmbeddingId)\n  implicit val simClustersEmbeddingInjection: Injection[ThriftSimClustersEmbedding, Array[Byte]] =\n    BinaryScalaCodec(ThriftSimClustersEmbedding)\n  implicit val simClustersMultiEmbeddingInjection: Injection[SimClustersMultiEmbedding, Array[\n    Byte\n  ]] =\n    BinaryScalaCodec(SimClustersMultiEmbedding)\n  implicit val simClustersMultiEmbeddingIdInjection: Injection[SimClustersMultiEmbeddingId, Array[\n    Byte\n  ]] =\n    BinaryScalaCodec(SimClustersMultiEmbeddingId)\n\n  def getEmbeddingsDataset(\n    mhMtlsParams: ManhattanKVClientMtlsParams,\n    datasetName: String\n  ): ReadableStore[SimClustersEmbeddingId, ThriftSimClustersEmbedding] = {\n    ManhattanRO.getReadableStoreWithMtls[SimClustersEmbeddingId, ThriftSimClustersEmbedding](\n      ManhattanROConfig(\n        HDFSPath(\"\"), // not needed\n        ApplicationID(\"content_recommender_athena\"),\n        DatasetName(datasetName), // this should be correct\n        Athena\n      ),\n      mhMtlsParams\n    )\n  }\n\n  lazy val logFavBasedLongestL2Tweet20M145K2020EmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore =\n      PersistentTweetEmbeddingStore\n        .longestL2NormTweetEmbeddingStoreManhattan(\n          mhMtlsParams,\n          PersistentTweetEmbeddingStore.LogFavBased20m145k2020Dataset,\n          statsReceiver,\n          maxLength = 10,\n        ).mapValues(_.toThrift)\n\n    val memcachedStore = ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = rawStore,\n      cacheClient = cacheClient,\n      ttl = 15.minutes\n    )(\n      valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n      statsReceiver =\n        statsReceiver.scope(\"log_fav_based_longest_l2_tweet_embedding_20m145k2020_mem_cache\"),\n      keyToString = { k =>\n        s\"scez_l2:${LogFavBasedTweet}_${ModelVersions.Model20M145K2020}_$k\"\n      }\n    )\n\n    val inMemoryCacheStore: ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] =\n      memcachedStore\n        .composeKeyMapping[SimClustersEmbeddingId] {\n          case SimClustersEmbeddingId(\n                LogFavLongestL2EmbeddingTweet,\n                Model20m145k2020,\n                InternalId.TweetId(tweetId)) =>\n            tweetId\n        }\n        .mapValues(SimClustersEmbedding(_))\n\n    ObservedCachedReadableStore.from[SimClustersEmbeddingId, SimClustersEmbedding](\n      inMemoryCacheStore,\n      ttl = 12.minute,\n      maxKeys = 1048575,\n      cacheName = \"log_fav_based_longest_l2_tweet_embedding_20m145k2020_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"log_fav_based_longest_l2_tweet_embedding_20m145k2020_store\"))\n  }\n\n  lazy val logFavBased20M145KUpdatedTweetEmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore =\n      PersistentTweetEmbeddingStore\n        .mostRecentTweetEmbeddingStoreManhattan(\n          mhMtlsParams,\n          PersistentTweetEmbeddingStore.LogFavBased20m145kUpdatedDataset,\n          statsReceiver\n        ).mapValues(_.toThrift)\n\n    val memcachedStore = ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = rawStore,\n      cacheClient = cacheClient,\n      ttl = 10.minutes\n    )(\n      valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n      statsReceiver = statsReceiver.scope(\"log_fav_based_tweet_embedding_mem_cache\"),\n      keyToString = { k =>\n        // SimClusters_embedding_LZ4/embeddingType_modelVersion_tweetId\n        s\"scez:${LogFavBasedTweet}_${ModelVersions.Model20M145KUpdated}_$k\"\n      }\n    )\n\n    val inMemoryCacheStore: ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n      memcachedStore\n        .composeKeyMapping[SimClustersEmbeddingId] {\n          case SimClustersEmbeddingId(\n                LogFavBasedTweet,\n                Model20m145kUpdated,\n                InternalId.TweetId(tweetId)) =>\n            tweetId\n        }\n        .mapValues(SimClustersEmbedding(_))\n    }\n\n    ObservedCachedReadableStore.from[SimClustersEmbeddingId, SimClustersEmbedding](\n      inMemoryCacheStore,\n      ttl = 5.minute,\n      maxKeys = 1048575, // 200MB\n      cacheName = \"log_fav_based_tweet_embedding_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"log_fav_based_tweet_embedding_store\"))\n  }\n\n  lazy val logFavBased20M145K2020TweetEmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore =\n      PersistentTweetEmbeddingStore\n        .mostRecentTweetEmbeddingStoreManhattan(\n          mhMtlsParams,\n          PersistentTweetEmbeddingStore.LogFavBased20m145k2020Dataset,\n          statsReceiver,\n          maxLength = 10,\n        ).mapValues(_.toThrift)\n\n    val memcachedStore = ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = rawStore,\n      cacheClient = cacheClient,\n      ttl = 15.minutes\n    )(\n      valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n      statsReceiver = statsReceiver.scope(\"log_fav_based_tweet_embedding_20m145k2020_mem_cache\"),\n      keyToString = { k =>\n        // SimClusters_embedding_LZ4/embeddingType_modelVersion_tweetId\n        s\"scez:${LogFavBasedTweet}_${ModelVersions.Model20M145K2020}_$k\"\n      }\n    )\n\n    val inMemoryCacheStore: ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] =\n      memcachedStore\n        .composeKeyMapping[SimClustersEmbeddingId] {\n          case SimClustersEmbeddingId(\n                LogFavBasedTweet,\n                Model20m145k2020,\n                InternalId.TweetId(tweetId)) =>\n            tweetId\n        }\n        .mapValues(SimClustersEmbedding(_))\n\n    ObservedCachedReadableStore.from[SimClustersEmbeddingId, SimClustersEmbedding](\n      inMemoryCacheStore,\n      ttl = 12.minute,\n      maxKeys = 16777215,\n      cacheName = \"log_fav_based_tweet_embedding_20m145k2020_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"log_fav_based_tweet_embedding_20m145k2020_store\"))\n  }\n\n  lazy val favBasedTfgTopicEmbedding2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val stratoStore =\n      StratoFetchableStore\n        .withUnitView[SimClustersEmbeddingId, ThriftSimClustersEmbedding](\n          stratoClient,\n          \"recommendations/simclusters_v2/embeddings/favBasedTFGTopic20M145K2020\")\n\n    val truncatedStore = stratoStore.mapValues { embedding =>\n      SimClustersEmbedding(embedding, truncate = 50)\n    }\n\n    ObservedCachedReadableStore.from(\n      ObservedReadableStore(truncatedStore)(\n        statsReceiver.scope(\"fav_tfg_topic_embedding_2020_cache_backing_store\")),\n      ttl = 12.hours,\n      maxKeys = 262143, // 200MB\n      cacheName = \"fav_tfg_topic_embedding_2020_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"fav_tfg_topic_embedding_2020_cache\"))\n  }\n\n  lazy val logFavBasedApe20M145K2020EmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    ObservedReadableStore(\n      StratoFetchableStore\n        .withUnitView[SimClustersEmbeddingId, ThriftSimClustersEmbedding](\n          stratoClient,\n          \"recommendations/simclusters_v2/embeddings/logFavBasedAPE20M145K2020\")\n        .composeKeyMapping[SimClustersEmbeddingId] {\n          case SimClustersEmbeddingId(\n                AggregatableLogFavBasedProducer,\n                Model20m145k2020,\n                internalId) =>\n            SimClustersEmbeddingId(AggregatableLogFavBasedProducer, Model20m145k2020, internalId)\n        }\n        .mapValues(embedding => SimClustersEmbedding(embedding, 50))\n    )(statsReceiver.scope(\"aggregatable_producer_embeddings_by_logfav_score_2020\"))\n  }\n\n  val interestService: InterestsThriftService.MethodPerEndpoint =\n    makeThriftClient[InterestsThriftService.MethodPerEndpoint](\n      \"/s/interests-thrift-service/interests-thrift-service\",\n      \"interests_thrift_service\"\n    )\n\n  val interestsOptOutStore: InterestsOptOutStore = InterestsOptOutStore(interestService)\n\n  // Save 2 ^ 18 UTTs. Promising 100% cache rate\n  lazy val defaultCacheConfigV2: CacheConfigV2 = CacheConfigV2(262143)\n  lazy val uttClientCacheConfigsV2: UttClientCacheConfigsV2 = UttClientCacheConfigsV2(\n    getTaxonomyConfig = defaultCacheConfigV2,\n    getUttTaxonomyConfig = defaultCacheConfigV2,\n    getLeafIds = defaultCacheConfigV2,\n    getLeafUttEntities = defaultCacheConfigV2\n  )\n\n  // CachedUttClient to use StratoClient\n  lazy val cachedUttClientV2: CachedUttClientV2 = new CachedUttClientV2(\n    stratoClient = stratoClient,\n    env = Environment.Prod,\n    cacheConfigs = uttClientCacheConfigsV2,\n    statsReceiver = statsReceiver.scope(\"cached_utt_client\")\n  )\n\n  lazy val semanticCoreTopicSeedStore: ReadableStore[\n    SemanticCoreTopicSeedStore.Key,\n    Seq[UserId]\n  ] = {\n    /*\n      Up to 1000 Long seeds per topic/language = 62.5kb per topic/language (worst case)\n      Assume ~10k active topic/languages ~= 650MB (worst case)\n     */\n    val underlying = new SemanticCoreTopicSeedStore(cachedUttClientV2, interestsOptOutStore)(\n      statsReceiver.scope(\"semantic_core_topic_seed_store\"))\n\n    val memcacheStore = ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = underlying,\n      cacheClient = cacheClient,\n      ttl = 12.hours\n    )(\n      valueInjection = SeqLongInjection,\n      statsReceiver = statsReceiver.scope(\"topic_producer_seed_store_mem_cache\"),\n      keyToString = { k => s\"tpss:${k.entityId}_${k.languageCode}\" }\n    )\n\n    ObservedCachedReadableStore.from[SemanticCoreTopicSeedStore.Key, Seq[UserId]](\n      store = memcacheStore,\n      ttl = 6.hours,\n      maxKeys = 20e3.toInt,\n      cacheName = \"topic_producer_seed_store_cache\",\n      windowSize = 5000\n    )(statsReceiver.scope(\"topic_producer_seed_store_cache\"))\n  }\n\n  lazy val logFavBasedApeEntity20M145K2020EmbeddingStore: ApeEntityEmbeddingStore = {\n    val apeStore = logFavBasedApe20M145K2020EmbeddingStore.composeKeyMapping[UserId]({ id =>\n      SimClustersEmbeddingId(\n        AggregatableLogFavBasedProducer,\n        Model20m145k2020,\n        InternalId.UserId(id))\n    })\n\n    new ApeEntityEmbeddingStore(\n      semanticCoreSeedStore = semanticCoreTopicSeedStore,\n      aggregatableProducerEmbeddingStore = apeStore,\n      statsReceiver = statsReceiver.scope(\"log_fav_based_ape_entity_2020_embedding_store\"))\n  }\n\n  lazy val logFavBasedApeEntity20M145K2020EmbeddingCachedStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val truncatedStore =\n      logFavBasedApeEntity20M145K2020EmbeddingStore.mapValues(_.truncate(50).toThrift)\n\n    val memcachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = truncatedStore,\n        cacheClient = cacheClient,\n        ttl = 12.hours\n      )(\n        valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n        statsReceiver = statsReceiver.scope(\"log_fav_based_ape_entity_2020_embedding_mem_cache\"),\n        keyToString = { k => embeddingCacheKeyBuilder.apply(k) }\n      ).mapValues(SimClustersEmbedding(_))\n\n    val inMemoryCachedStore =\n      ObservedCachedReadableStore.from[SimClustersEmbeddingId, SimClustersEmbedding](\n        memcachedStore,\n        ttl = 6.hours,\n        maxKeys = 262143,\n        cacheName = \"log_fav_based_ape_entity_2020_embedding_cache\",\n        windowSize = 10000L\n      )(statsReceiver.scope(\"log_fav_based_ape_entity_2020_embedding_cached_store\"))\n\n    DeciderableReadableStore(\n      inMemoryCachedStore,\n      rmsDecider.deciderGateBuilder.idGateWithHashing[SimClustersEmbeddingId](\n        DeciderKey.enableLogFavBasedApeEntity20M145K2020EmbeddingCachedStore),\n      statsReceiver.scope(\"log_fav_based_ape_entity_2020_embedding_deciderable_store\")\n    )\n  }\n\n  lazy val relaxedLogFavBasedApe20M145K2020EmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    ObservedReadableStore(\n      StratoFetchableStore\n        .withUnitView[SimClustersEmbeddingId, ThriftSimClustersEmbedding](\n          stratoClient,\n          \"recommendations/simclusters_v2/embeddings/logFavBasedAPERelaxedFavEngagementThreshold20M145K2020\")\n        .composeKeyMapping[SimClustersEmbeddingId] {\n          case SimClustersEmbeddingId(\n                RelaxedAggregatableLogFavBasedProducer,\n                Model20m145k2020,\n                internalId) =>\n            SimClustersEmbeddingId(\n              RelaxedAggregatableLogFavBasedProducer,\n              Model20m145k2020,\n              internalId)\n        }\n        .mapValues(embedding => SimClustersEmbedding(embedding).truncate(50))\n    )(statsReceiver.scope(\n      \"aggregatable_producer_embeddings_by_logfav_score_relaxed_fav_engagement_threshold_2020\"))\n  }\n\n  lazy val relaxedLogFavBasedApe20M145K2020EmbeddingCachedStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val truncatedStore =\n      relaxedLogFavBasedApe20M145K2020EmbeddingStore.mapValues(_.truncate(50).toThrift)\n\n    val memcachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = truncatedStore,\n        cacheClient = cacheClient,\n        ttl = 12.hours\n      )(\n        valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n        statsReceiver =\n          statsReceiver.scope(\"relaxed_log_fav_based_ape_entity_2020_embedding_mem_cache\"),\n        keyToString = { k: SimClustersEmbeddingId => embeddingCacheKeyBuilder.apply(k) }\n      ).mapValues(SimClustersEmbedding(_))\n\n    ObservedCachedReadableStore.from[SimClustersEmbeddingId, SimClustersEmbedding](\n      memcachedStore,\n      ttl = 6.hours,\n      maxKeys = 262143,\n      cacheName = \"relaxed_log_fav_based_ape_entity_2020_embedding_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"relaxed_log_fav_based_ape_entity_2020_embedding_cache_store\"))\n  }\n\n  lazy val favBasedProducer20M145K2020EmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val underlyingStore = ProducerClusterEmbeddingReadableStores\n      .getProducerTopKSimClusters2020EmbeddingsStore(\n        mhMtlsParams\n      ).composeKeyMapping[SimClustersEmbeddingId] {\n        case SimClustersEmbeddingId(\n              FavBasedProducer,\n              Model20m145k2020,\n              InternalId.UserId(userId)) =>\n          userId\n      }.mapValues { topSimClustersWithScore =>\n        ThriftSimClustersEmbedding(topSimClustersWithScore.topClusters.take(10))\n      }\n\n    // same memcache config as for favBasedUserInterestedIn20M145K2020Store\n    val memcachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = underlyingStore,\n        cacheClient = cacheClient,\n        ttl = 24.hours\n      )(\n        valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n        statsReceiver = statsReceiver.scope(\"fav_based_producer_embedding_20M_145K_2020_mem_cache\"),\n        keyToString = { k => embeddingCacheKeyBuilder.apply(k) }\n      ).mapValues(SimClustersEmbedding(_))\n\n    ObservedCachedReadableStore.from[SimClustersEmbeddingId, SimClustersEmbedding](\n      memcachedStore,\n      ttl = 12.hours,\n      maxKeys = 16777215,\n      cacheName = \"fav_based_producer_embedding_20M_145K_2020_embedding_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"fav_based_producer_embedding_20M_145K_2020_embedding_store\"))\n  }\n\n  // Production\n  lazy val interestedIn20M145KUpdatedStore: ReadableStore[UserId, ClustersUserIsInterestedIn] = {\n    UserInterestedInReadableStore.defaultStoreWithMtls(\n      mhMtlsParams,\n      modelVersion = ModelVersions.Model20M145KUpdated\n    )\n  }\n\n  // Production\n  lazy val interestedIn20M145K2020Store: ReadableStore[UserId, ClustersUserIsInterestedIn] = {\n    UserInterestedInReadableStore.defaultStoreWithMtls(\n      mhMtlsParams,\n      modelVersion = ModelVersions.Model20M145K2020\n    )\n  }\n\n  // Production\n  lazy val InterestedInFromPE20M145KUpdatedStore: ReadableStore[\n    UserId,\n    ClustersUserIsInterestedIn\n  ] = {\n    UserInterestedInReadableStore.defaultIIPEStoreWithMtls(\n      mhMtlsParams,\n      modelVersion = ModelVersions.Model20M145KUpdated)\n  }\n\n  lazy val simClustersInterestedInStore: ReadableStore[\n    (UserId, ModelVersion),\n    ClustersUserIsInterestedIn\n  ] = {\n    new ReadableStore[(UserId, ModelVersion), ClustersUserIsInterestedIn] {\n      override def get(k: (UserId, ModelVersion)): Future[Option[ClustersUserIsInterestedIn]] = {\n        k match {\n          case (userId, Model20m145kUpdated) =>\n            interestedIn20M145KUpdatedStore.get(userId)\n          case (userId, Model20m145k2020) =>\n            interestedIn20M145K2020Store.get(userId)\n          case _ =>\n            Future.None\n        }\n      }\n    }\n  }\n\n  lazy val simClustersInterestedInFromProducerEmbeddingsStore: ReadableStore[\n    (UserId, ModelVersion),\n    ClustersUserIsInterestedIn\n  ] = {\n    new ReadableStore[(UserId, ModelVersion), ClustersUserIsInterestedIn] {\n      override def get(k: (UserId, ModelVersion)): Future[Option[ClustersUserIsInterestedIn]] = {\n        k match {\n          case (userId, ModelVersion.Model20m145kUpdated) =>\n            InterestedInFromPE20M145KUpdatedStore.get(userId)\n          case _ =>\n            Future.None\n        }\n      }\n    }\n  }\n\n  lazy val userInterestedInStore =\n    new twistly.interestedin.EmbeddingStore(\n      interestedInStore = simClustersInterestedInStore,\n      interestedInFromProducerEmbeddingStore = simClustersInterestedInFromProducerEmbeddingsStore,\n      statsReceiver = statsReceiver\n    )\n\n  // Production\n  lazy val favBasedUserInterestedIn20M145KUpdatedStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val underlyingStore =\n      UserInterestedInReadableStore\n        .defaultSimClustersEmbeddingStoreWithMtls(\n          mhMtlsParams,\n          EmbeddingType.FavBasedUserInterestedIn,\n          ModelVersion.Model20m145kUpdated)\n        .mapValues(_.toThrift)\n\n    val memcachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = underlyingStore,\n        cacheClient = cacheClient,\n        ttl = 12.hours\n      )(\n        valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n        statsReceiver = statsReceiver.scope(\"fav_based_user_interested_in_mem_cache\"),\n        keyToString = { k => embeddingCacheKeyBuilder.apply(k) }\n      ).mapValues(SimClustersEmbedding(_))\n\n    ObservedCachedReadableStore.from[SimClustersEmbeddingId, SimClustersEmbedding](\n      memcachedStore,\n      ttl = 6.hours,\n      maxKeys = 262143,\n      cacheName = \"fav_based_user_interested_in_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"fav_based_user_interested_in_store\"))\n  }\n\n  // Production\n  lazy val LogFavBasedInterestedInFromAPE20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val underlyingStore =\n      UserInterestedInReadableStore\n        .defaultIIAPESimClustersEmbeddingStoreWithMtls(\n          mhMtlsParams,\n          EmbeddingType.LogFavBasedUserInterestedInFromAPE,\n          ModelVersion.Model20m145k2020)\n        .mapValues(_.toThrift)\n\n    val memcachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = underlyingStore,\n        cacheClient = cacheClient,\n        ttl = 12.hours\n      )(\n        valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n        statsReceiver = statsReceiver.scope(\"log_fav_based_user_interested_in_from_ape_mem_cache\"),\n        keyToString = { k => embeddingCacheKeyBuilder.apply(k) }\n      ).mapValues(SimClustersEmbedding(_))\n\n    ObservedCachedReadableStore.from[SimClustersEmbeddingId, SimClustersEmbedding](\n      memcachedStore,\n      ttl = 6.hours,\n      maxKeys = 262143,\n      cacheName = \"log_fav_based_user_interested_in_from_ape_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"log_fav_based_user_interested_in_from_ape_store\"))\n  }\n\n  // Production\n  lazy val FollowBasedInterestedInFromAPE20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val underlyingStore =\n      UserInterestedInReadableStore\n        .defaultIIAPESimClustersEmbeddingStoreWithMtls(\n          mhMtlsParams,\n          EmbeddingType.FollowBasedUserInterestedInFromAPE,\n          ModelVersion.Model20m145k2020)\n        .mapValues(_.toThrift)\n\n    val memcachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = underlyingStore,\n        cacheClient = cacheClient,\n        ttl = 12.hours\n      )(\n        valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n        statsReceiver = statsReceiver.scope(\"follow_based_user_interested_in_from_ape_mem_cache\"),\n        keyToString = { k => embeddingCacheKeyBuilder.apply(k) }\n      ).mapValues(SimClustersEmbedding(_))\n\n    ObservedCachedReadableStore.from[SimClustersEmbeddingId, SimClustersEmbedding](\n      memcachedStore,\n      ttl = 6.hours,\n      maxKeys = 262143,\n      cacheName = \"follow_based_user_interested_in_from_ape_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"follow_based_user_interested_in_from_ape_store\"))\n  }\n\n  // production\n  lazy val favBasedUserInterestedIn20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val underlyingStore: ReadableStore[SimClustersEmbeddingId, ThriftSimClustersEmbedding] =\n      UserInterestedInReadableStore\n        .defaultSimClustersEmbeddingStoreWithMtls(\n          mhMtlsParams,\n          EmbeddingType.FavBasedUserInterestedIn,\n          ModelVersion.Model20m145k2020).mapValues(_.toThrift)\n\n    ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = underlyingStore,\n        cacheClient = cacheClient,\n        ttl = 12.hours\n      )(\n        valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n        statsReceiver = statsReceiver.scope(\"fav_based_user_interested_in_2020_mem_cache\"),\n        keyToString = { k => embeddingCacheKeyBuilder.apply(k) }\n      ).mapValues(SimClustersEmbedding(_))\n  }\n\n  // Production\n  lazy val logFavBasedUserInterestedIn20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val underlyingStore =\n      UserInterestedInReadableStore\n        .defaultSimClustersEmbeddingStoreWithMtls(\n          mhMtlsParams,\n          EmbeddingType.LogFavBasedUserInterestedIn,\n          ModelVersion.Model20m145k2020)\n\n    val memcachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = underlyingStore.mapValues(_.toThrift),\n        cacheClient = cacheClient,\n        ttl = 12.hours\n      )(\n        valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n        statsReceiver = statsReceiver.scope(\"log_fav_based_user_interested_in_2020_store\"),\n        keyToString = { k => embeddingCacheKeyBuilder.apply(k) }\n      ).mapValues(SimClustersEmbedding(_))\n\n    ObservedCachedReadableStore.from[SimClustersEmbeddingId, SimClustersEmbedding](\n      memcachedStore,\n      ttl = 6.hours,\n      maxKeys = 262143,\n      cacheName = \"log_fav_based_user_interested_in_2020_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"log_fav_based_user_interested_in_2020_store\"))\n  }\n\n  // Production\n  lazy val favBasedUserInterestedInFromPE20M145KUpdatedStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val underlyingStore =\n      UserInterestedInReadableStore\n        .defaultIIPESimClustersEmbeddingStoreWithMtls(\n          mhMtlsParams,\n          EmbeddingType.FavBasedUserInterestedInFromPE,\n          ModelVersion.Model20m145kUpdated)\n        .mapValues(_.toThrift)\n\n    val memcachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = underlyingStore,\n        cacheClient = cacheClient,\n        ttl = 12.hours\n      )(\n        valueInjection = LZ4Injection.compose(BinaryScalaCodec(ThriftSimClustersEmbedding)),\n        statsReceiver = statsReceiver.scope(\"fav_based_user_interested_in_from_pe_mem_cache\"),\n        keyToString = { k => embeddingCacheKeyBuilder.apply(k) }\n      ).mapValues(SimClustersEmbedding(_))\n\n    ObservedCachedReadableStore.from[SimClustersEmbeddingId, SimClustersEmbedding](\n      memcachedStore,\n      ttl = 6.hours,\n      maxKeys = 262143,\n      cacheName = \"fav_based_user_interested_in_from_pe_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"fav_based_user_interested_in_from_pe_cache\"))\n  }\n\n  private val underlyingStores: Map[\n    (EmbeddingType, ModelVersion),\n    ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding]\n  ] = Map(\n    // Tweet Embeddings\n    (LogFavBasedTweet, Model20m145kUpdated) -> logFavBased20M145KUpdatedTweetEmbeddingStore,\n    (LogFavBasedTweet, Model20m145k2020) -> logFavBased20M145K2020TweetEmbeddingStore,\n    (\n      LogFavLongestL2EmbeddingTweet,\n      Model20m145k2020) -> logFavBasedLongestL2Tweet20M145K2020EmbeddingStore,\n    // Entity Embeddings\n    (FavTfgTopic, Model20m145k2020) -> favBasedTfgTopicEmbedding2020Store,\n    (\n      LogFavBasedKgoApeTopic,\n      Model20m145k2020) -> logFavBasedApeEntity20M145K2020EmbeddingCachedStore,\n    // KnownFor Embeddings\n    (FavBasedProducer, Model20m145k2020) -> favBasedProducer20M145K2020EmbeddingStore,\n    (\n      RelaxedAggregatableLogFavBasedProducer,\n      Model20m145k2020) -> relaxedLogFavBasedApe20M145K2020EmbeddingCachedStore,\n    // InterestedIn Embeddings\n    (\n      LogFavBasedUserInterestedInFromAPE,\n      Model20m145k2020) -> LogFavBasedInterestedInFromAPE20M145K2020Store,\n    (\n      FollowBasedUserInterestedInFromAPE,\n      Model20m145k2020) -> FollowBasedInterestedInFromAPE20M145K2020Store,\n    (FavBasedUserInterestedIn, Model20m145kUpdated) -> favBasedUserInterestedIn20M145KUpdatedStore,\n    (FavBasedUserInterestedIn, Model20m145k2020) -> favBasedUserInterestedIn20M145K2020Store,\n    (LogFavBasedUserInterestedIn, Model20m145k2020) -> logFavBasedUserInterestedIn20M145K2020Store,\n    (\n      FavBasedUserInterestedInFromPE,\n      Model20m145kUpdated) -> favBasedUserInterestedInFromPE20M145KUpdatedStore,\n    (FilteredUserInterestedIn, Model20m145kUpdated) -> userInterestedInStore,\n    (FilteredUserInterestedIn, Model20m145k2020) -> userInterestedInStore,\n    (FilteredUserInterestedInFromPE, Model20m145kUpdated) -> userInterestedInStore,\n    (UnfilteredUserInterestedIn, Model20m145kUpdated) -> userInterestedInStore,\n    (UnfilteredUserInterestedIn, Model20m145k2020) -> userInterestedInStore,\n  )\n\n  val simClustersEmbeddingStore: ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n    val underlying: ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] =\n      SimClustersEmbeddingStore.buildWithDecider(\n        underlyingStores = underlyingStores,\n        decider = rmsDecider.decider,\n        statsReceiver = statsReceiver.scope(\"simClusters_embeddings_store_deciderable\")\n      )\n\n    val underlyingWithTimeout: ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] =\n      new ReadableStoreWithTimeout(\n        rs = underlying,\n        decider = rmsDecider.decider,\n        enableTimeoutDeciderKey = DeciderConstants.enableSimClustersEmbeddingStoreTimeouts,\n        timeoutValueKey = DeciderConstants.simClustersEmbeddingStoreTimeoutValueMillis,\n        timer = timer,\n        statsReceiver = statsReceiver.scope(\"simClusters_embedding_store_timeouts\")\n      )\n\n    ObservedReadableStore(\n      store = underlyingWithTimeout\n    )(statsReceiver.scope(\"simClusters_embeddings_store\"))\n  }\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/modules/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finagle/finagle-stats\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/util\",\n        \"interests-service/thrift/src/main/thrift:thrift-scala\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/common\",\n        \"servo/util\",\n        \"src/scala/com/twitter/storehaus_internal/manhattan\",\n        \"src/scala/com/twitter/storehaus_internal/memcache\",\n        \"src/scala/com/twitter/storehaus_internal/util\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n    ],\n)\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/modules/CacheModule.scala",
    "content": "package com.twitter.representation_manager.modules\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.memcached.Client\nimport javax.inject.Singleton\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.inject.TwitterModule\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.storehaus_internal.memcache.MemcacheStore\nimport com.twitter.storehaus_internal.util.ClientName\nimport com.twitter.storehaus_internal.util.ZkEndPoint\n\nobject CacheModule extends TwitterModule {\n\n  private val cacheDest = flag[String](\"cache_module.dest\", \"Path to memcache service\")\n  private val timeout = flag[Int](\"memcache.timeout\", \"Memcache client timeout\")\n  private val retries = flag[Int](\"memcache.retries\", \"Memcache timeout retries\")\n\n  @Singleton\n  @Provides\n  def providesCache(\n    serviceIdentifier: ServiceIdentifier,\n    stats: StatsReceiver\n  ): Client =\n    MemcacheStore.memcachedClient(\n      name = ClientName(\"memcache_representation_manager\"),\n      dest = ZkEndPoint(cacheDest()),\n      timeout = timeout().milliseconds,\n      retries = retries(),\n      statsReceiver = stats.scope(\"cache_client\"),\n      serviceIdentifier = serviceIdentifier\n    )\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/modules/InterestsThriftClientModule.scala",
    "content": "package com.twitter.representation_manager.modules\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.ThriftMux\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.mtls.client.MtlsStackClient.MtlsThriftMuxClientSyntax\nimport com.twitter.finagle.mux.ClientDiscardedRequestException\nimport com.twitter.finagle.service.ReqRep\nimport com.twitter.finagle.service.ResponseClass\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.inject.TwitterModule\nimport com.twitter.interests.thriftscala.InterestsThriftService\nimport com.twitter.util.Throw\nimport javax.inject.Singleton\n\nobject InterestsThriftClientModule extends TwitterModule {\n\n  @Singleton\n  @Provides\n  def providesInterestsThriftClient(\n    clientId: ClientId,\n    serviceIdentifier: ServiceIdentifier,\n    statsReceiver: StatsReceiver\n  ): InterestsThriftService.MethodPerEndpoint = {\n    ThriftMux.client\n      .withClientId(clientId)\n      .withMutualTls(serviceIdentifier)\n      .withRequestTimeout(450.milliseconds)\n      .withStatsReceiver(statsReceiver.scope(\"InterestsThriftClient\"))\n      .withResponseClassifier {\n        case ReqRep(_, Throw(_: ClientDiscardedRequestException)) => ResponseClass.Ignorable\n      }\n      .build[InterestsThriftService.MethodPerEndpoint](\n        dest = \"/s/interests-thrift-service/interests-thrift-service\",\n        label = \"interests_thrift_service\"\n      )\n  }\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/modules/LegacyRMSConfigModule.scala",
    "content": "package com.twitter.representation_manager.modules\n\nimport com.google.inject.Provides\nimport com.twitter.inject.TwitterModule\nimport javax.inject.Named\nimport javax.inject.Singleton\n\nobject LegacyRMSConfigModule extends TwitterModule {\n  @Singleton\n  @Provides\n  @Named(\"cacheHashKeyPrefix\")\n  def providesCacheHashKeyPrefix: String = \"RMS\"\n\n  @Singleton\n  @Provides\n  @Named(\"useContentRecommenderConfiguration\")\n  def providesUseContentRecommenderConfiguration: Boolean = false\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/modules/StoreModule.scala",
    "content": "package com.twitter.representation_manager.modules\n\nimport com.google.inject.Provides\nimport javax.inject.Singleton\nimport com.twitter.inject.TwitterModule\nimport com.twitter.decider.Decider\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.representation_manager.common.RepresentationManagerDecider\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\n\nobject StoreModule extends TwitterModule {\n  @Singleton\n  @Provides\n  def providesMhMtlsParams(\n    serviceIdentifier: ServiceIdentifier\n  ): ManhattanKVClientMtlsParams = ManhattanKVClientMtlsParams(serviceIdentifier)\n\n  @Singleton\n  @Provides\n  def providesRmsDecider(\n    decider: Decider\n  ): RepresentationManagerDecider = RepresentationManagerDecider(decider)\n\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/modules/TimerModule.scala",
    "content": "package com.twitter.representation_manager.modules\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.inject.TwitterModule\nimport com.twitter.util.Timer\nimport javax.inject.Singleton\n\nobject TimerModule extends TwitterModule {\n  @Singleton\n  @Provides\n  def providesTimer: Timer = DefaultTimer\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/modules/UttClientModule.scala",
    "content": "package com.twitter.representation_manager.modules\n\nimport com.google.inject.Provides\nimport com.twitter.escherbird.util.uttclient.CacheConfigV2\nimport com.twitter.escherbird.util.uttclient.CachedUttClientV2\nimport com.twitter.escherbird.util.uttclient.UttClientCacheConfigsV2\nimport com.twitter.escherbird.utt.strato.thriftscala.Environment\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.strato.client.{Client => StratoClient}\nimport javax.inject.Singleton\n\nobject UttClientModule extends TwitterModule {\n\n  @Singleton\n  @Provides\n  def providesUttClient(\n    stratoClient: StratoClient,\n    statsReceiver: StatsReceiver\n  ): CachedUttClientV2 = {\n    // Save 2 ^ 18 UTTs. Promising 100% cache rate\n    val defaultCacheConfigV2: CacheConfigV2 = CacheConfigV2(262143)\n\n    val uttClientCacheConfigsV2: UttClientCacheConfigsV2 = UttClientCacheConfigsV2(\n      getTaxonomyConfig = defaultCacheConfigV2,\n      getUttTaxonomyConfig = defaultCacheConfigV2,\n      getLeafIds = defaultCacheConfigV2,\n      getLeafUttEntities = defaultCacheConfigV2\n    )\n\n    // CachedUttClient to use StratoClient\n    new CachedUttClientV2(\n      stratoClient = stratoClient,\n      env = Environment.Prod,\n      cacheConfigs = uttClientCacheConfigsV2,\n      statsReceiver = statsReceiver.scope(\"cached_utt_client\")\n    )\n  }\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/store/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"content-recommender/server/src/main/scala/com/twitter/contentrecommender:representation-manager-deps\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/util\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/common\",\n        \"src/scala/com/twitter/simclusters_v2/stores\",\n        \"src/scala/com/twitter/simclusters_v2/summingbird/stores\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"storage/clients/manhattan/client/src/main/scala\",\n        \"tweetypie/src/scala/com/twitter/tweetypie/util\",\n    ],\n)\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/store/DeciderConstants.scala",
    "content": "package com.twitter.representation_manager.store\n\nimport com.twitter.servo.decider.DeciderKeyEnum\n\nobject DeciderConstants {\n  // Deciders inherited from CR and RSX and only used in LegacyRMS\n  // Their value are manipulated by CR and RSX's yml file and their decider dashboard\n  // We will remove them after migration completed\n  val enableLogFavBasedApeEntity20M145KUpdatedEmbeddingCachedStore =\n    \"enableLogFavBasedApeEntity20M145KUpdatedEmbeddingCachedStore\"\n\n  val enableLogFavBasedApeEntity20M145K2020EmbeddingCachedStore =\n    \"enableLogFavBasedApeEntity20M145K2020EmbeddingCachedStore\"\n\n  val enablelogFavBased20M145K2020TweetEmbeddingStoreTimeouts =\n    \"enable_log_fav_based_tweet_embedding_20m145k2020_timeouts\"\n  val logFavBased20M145K2020TweetEmbeddingStoreTimeoutValueMillis =\n    \"log_fav_based_tweet_embedding_20m145k2020_timeout_value_millis\"\n\n  val enablelogFavBased20M145KUpdatedTweetEmbeddingStoreTimeouts =\n    \"enable_log_fav_based_tweet_embedding_20m145kUpdated_timeouts\"\n  val logFavBased20M145KUpdatedTweetEmbeddingStoreTimeoutValueMillis =\n    \"log_fav_based_tweet_embedding_20m145kUpdated_timeout_value_millis\"\n\n  val enableSimClustersEmbeddingStoreTimeouts = \"enable_sim_clusters_embedding_store_timeouts\"\n  val simClustersEmbeddingStoreTimeoutValueMillis =\n    \"sim_clusters_embedding_store_timeout_value_millis\"\n}\n\n// Necessary for using servo Gates\nobject DeciderKey extends DeciderKeyEnum {\n  val enableLogFavBasedApeEntity20M145KUpdatedEmbeddingCachedStore: Value = Value(\n    DeciderConstants.enableLogFavBasedApeEntity20M145KUpdatedEmbeddingCachedStore\n  )\n\n  val enableLogFavBasedApeEntity20M145K2020EmbeddingCachedStore: Value = Value(\n    DeciderConstants.enableLogFavBasedApeEntity20M145K2020EmbeddingCachedStore\n  )\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/store/TopicSimClustersEmbeddingStore.scala",
    "content": "package com.twitter.representation_manager.store\n\nimport com.twitter.contentrecommender.store.ApeEntityEmbeddingStore\nimport com.twitter.contentrecommender.store.InterestsOptOutStore\nimport com.twitter.contentrecommender.store.SemanticCoreTopicSeedStore\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.escherbird.util.uttclient.CachedUttClientV2\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.frigate.common.util.SeqLongInjection\nimport com.twitter.hermit.store.common.ObservedCachedReadableStore\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.interests.thriftscala.InterestsThriftService\nimport com.twitter.representation_manager.common.MemCacheConfig\nimport com.twitter.representation_manager.common.RepresentationManagerDecider\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.stores.SimClustersEmbeddingStore\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType._\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion._\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.TopicId\nimport com.twitter.simclusters_v2.thriftscala.LocaleEntityId\nimport com.twitter.simclusters_v2.thriftscala.{SimClustersEmbedding => ThriftSimClustersEmbedding}\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.tweetypie.util.UserId\nimport javax.inject.Inject\n\nclass TopicSimClustersEmbeddingStore @Inject() (\n  stratoClient: StratoClient,\n  cacheClient: Client,\n  globalStats: StatsReceiver,\n  mhMtlsParams: ManhattanKVClientMtlsParams,\n  rmsDecider: RepresentationManagerDecider,\n  interestService: InterestsThriftService.MethodPerEndpoint,\n  uttClient: CachedUttClientV2) {\n\n  private val stats = globalStats.scope(this.getClass.getSimpleName)\n  private val interestsOptOutStore = InterestsOptOutStore(interestService)\n\n  /**\n   * Note this is NOT an embedding store. It is a list of author account ids we use to represent\n   * topics\n   */\n  private val semanticCoreTopicSeedStore: ReadableStore[\n    SemanticCoreTopicSeedStore.Key,\n    Seq[UserId]\n  ] = {\n    /*\n      Up to 1000 Long seeds per topic/language = 62.5kb per topic/language (worst case)\n      Assume ~10k active topic/languages ~= 650MB (worst case)\n     */\n    val underlying = new SemanticCoreTopicSeedStore(uttClient, interestsOptOutStore)(\n      stats.scope(\"semantic_core_topic_seed_store\"))\n\n    val memcacheStore = ObservedMemcachedReadableStore.fromCacheClient(\n      backingStore = underlying,\n      cacheClient = cacheClient,\n      ttl = 12.hours)(\n      valueInjection = SeqLongInjection,\n      statsReceiver = stats.scope(\"topic_producer_seed_store_mem_cache\"),\n      keyToString = { k => s\"tpss:${k.entityId}_${k.languageCode}\" }\n    )\n\n    ObservedCachedReadableStore.from[SemanticCoreTopicSeedStore.Key, Seq[UserId]](\n      store = memcacheStore,\n      ttl = 6.hours,\n      maxKeys = 20e3.toInt,\n      cacheName = \"topic_producer_seed_store_cache\",\n      windowSize = 5000\n    )(stats.scope(\"topic_producer_seed_store_cache\"))\n  }\n\n  private val favBasedTfgTopicEmbedding20m145k2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore =\n      StratoFetchableStore\n        .withUnitView[SimClustersEmbeddingId, ThriftSimClustersEmbedding](\n          stratoClient,\n          \"recommendations/simclusters_v2/embeddings/favBasedTFGTopic20M145K2020\").mapValues(\n          embedding => SimClustersEmbedding(embedding, truncate = 50).toThrift)\n        .composeKeyMapping[LocaleEntityId] { localeEntityId =>\n          SimClustersEmbeddingId(\n            FavTfgTopic,\n            Model20m145k2020,\n            InternalId.LocaleEntityId(localeEntityId))\n        }\n\n    buildLocaleEntityIdMemCacheStore(rawStore, FavTfgTopic, Model20m145k2020)\n  }\n\n  private val logFavBasedApeEntity20M145K2020EmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val apeStore = StratoFetchableStore\n      .withUnitView[SimClustersEmbeddingId, ThriftSimClustersEmbedding](\n        stratoClient,\n        \"recommendations/simclusters_v2/embeddings/logFavBasedAPE20M145K2020\")\n      .mapValues(embedding => SimClustersEmbedding(embedding, truncate = 50))\n      .composeKeyMapping[UserId]({ id =>\n        SimClustersEmbeddingId(\n          AggregatableLogFavBasedProducer,\n          Model20m145k2020,\n          InternalId.UserId(id))\n      })\n    val rawStore = new ApeEntityEmbeddingStore(\n      semanticCoreSeedStore = semanticCoreTopicSeedStore,\n      aggregatableProducerEmbeddingStore = apeStore,\n      statsReceiver = stats.scope(\"log_fav_based_ape_entity_2020_embedding_store\"))\n      .mapValues(embedding => SimClustersEmbedding(embedding.toThrift, truncate = 50).toThrift)\n      .composeKeyMapping[TopicId] { topicId =>\n        SimClustersEmbeddingId(\n          LogFavBasedKgoApeTopic,\n          Model20m145k2020,\n          InternalId.TopicId(topicId))\n      }\n\n    buildTopicIdMemCacheStore(rawStore, LogFavBasedKgoApeTopic, Model20m145k2020)\n  }\n\n  private def buildTopicIdMemCacheStore(\n    rawStore: ReadableStore[TopicId, ThriftSimClustersEmbedding],\n    embeddingType: EmbeddingType,\n    modelVersion: ModelVersion\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n    val observedStore: ObservedReadableStore[TopicId, ThriftSimClustersEmbedding] =\n      ObservedReadableStore(\n        store = rawStore\n      )(stats.scope(embeddingType.name).scope(modelVersion.name))\n\n    val storeWithKeyMapping = observedStore.composeKeyMapping[SimClustersEmbeddingId] {\n      case SimClustersEmbeddingId(_, _, InternalId.TopicId(topicId)) =>\n        topicId\n    }\n\n    MemCacheConfig.buildMemCacheStoreForSimClustersEmbedding(\n      storeWithKeyMapping,\n      cacheClient,\n      embeddingType,\n      modelVersion,\n      stats\n    )\n  }\n\n  private def buildLocaleEntityIdMemCacheStore(\n    rawStore: ReadableStore[LocaleEntityId, ThriftSimClustersEmbedding],\n    embeddingType: EmbeddingType,\n    modelVersion: ModelVersion\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n    val observedStore: ObservedReadableStore[LocaleEntityId, ThriftSimClustersEmbedding] =\n      ObservedReadableStore(\n        store = rawStore\n      )(stats.scope(embeddingType.name).scope(modelVersion.name))\n\n    val storeWithKeyMapping = observedStore.composeKeyMapping[SimClustersEmbeddingId] {\n      case SimClustersEmbeddingId(_, _, InternalId.LocaleEntityId(localeEntityId)) =>\n        localeEntityId\n    }\n\n    MemCacheConfig.buildMemCacheStoreForSimClustersEmbedding(\n      storeWithKeyMapping,\n      cacheClient,\n      embeddingType,\n      modelVersion,\n      stats\n    )\n  }\n\n  private val underlyingStores: Map[\n    (EmbeddingType, ModelVersion),\n    ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding]\n  ] = Map(\n    // Topic Embeddings\n    (FavTfgTopic, Model20m145k2020) -> favBasedTfgTopicEmbedding20m145k2020Store,\n    (LogFavBasedKgoApeTopic, Model20m145k2020) -> logFavBasedApeEntity20M145K2020EmbeddingStore,\n  )\n\n  val topicSimClustersEmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    SimClustersEmbeddingStore.buildWithDecider(\n      underlyingStores = underlyingStores,\n      decider = rmsDecider.decider,\n      statsReceiver = stats\n    )\n  }\n\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/store/TweetSimClustersEmbeddingStore.scala",
    "content": "package com.twitter.representation_manager.store\n\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.representation_manager.common.MemCacheConfig\nimport com.twitter.representation_manager.common.RepresentationManagerDecider\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.stores.SimClustersEmbeddingStore\nimport com.twitter.simclusters_v2.summingbird.stores.PersistentTweetEmbeddingStore\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType._\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion._\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.{SimClustersEmbedding => ThriftSimClustersEmbedding}\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Inject\n\nclass TweetSimClustersEmbeddingStore @Inject() (\n  cacheClient: Client,\n  globalStats: StatsReceiver,\n  mhMtlsParams: ManhattanKVClientMtlsParams,\n  rmsDecider: RepresentationManagerDecider) {\n\n  private val stats = globalStats.scope(this.getClass.getSimpleName)\n\n  val logFavBasedLongestL2Tweet20M145KUpdatedEmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore =\n      PersistentTweetEmbeddingStore\n        .longestL2NormTweetEmbeddingStoreManhattan(\n          mhMtlsParams,\n          PersistentTweetEmbeddingStore.LogFavBased20m145kUpdatedDataset,\n          stats\n        ).mapValues(_.toThrift)\n\n    buildMemCacheStore(rawStore, LogFavLongestL2EmbeddingTweet, Model20m145kUpdated)\n  }\n\n  val logFavBasedLongestL2Tweet20M145K2020EmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore =\n      PersistentTweetEmbeddingStore\n        .longestL2NormTweetEmbeddingStoreManhattan(\n          mhMtlsParams,\n          PersistentTweetEmbeddingStore.LogFavBased20m145k2020Dataset,\n          stats\n        ).mapValues(_.toThrift)\n\n    buildMemCacheStore(rawStore, LogFavLongestL2EmbeddingTweet, Model20m145k2020)\n  }\n\n  val logFavBased20M145KUpdatedTweetEmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore =\n      PersistentTweetEmbeddingStore\n        .mostRecentTweetEmbeddingStoreManhattan(\n          mhMtlsParams,\n          PersistentTweetEmbeddingStore.LogFavBased20m145kUpdatedDataset,\n          stats\n        ).mapValues(_.toThrift)\n\n    buildMemCacheStore(rawStore, LogFavBasedTweet, Model20m145kUpdated)\n  }\n\n  val logFavBased20M145K2020TweetEmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore =\n      PersistentTweetEmbeddingStore\n        .mostRecentTweetEmbeddingStoreManhattan(\n          mhMtlsParams,\n          PersistentTweetEmbeddingStore.LogFavBased20m145k2020Dataset,\n          stats\n        ).mapValues(_.toThrift)\n\n    buildMemCacheStore(rawStore, LogFavBasedTweet, Model20m145k2020)\n  }\n\n  private def buildMemCacheStore(\n    rawStore: ReadableStore[TweetId, ThriftSimClustersEmbedding],\n    embeddingType: EmbeddingType,\n    modelVersion: ModelVersion\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n    val observedStore: ObservedReadableStore[TweetId, ThriftSimClustersEmbedding] =\n      ObservedReadableStore(\n        store = rawStore\n      )(stats.scope(embeddingType.name).scope(modelVersion.name))\n\n    val storeWithKeyMapping = observedStore.composeKeyMapping[SimClustersEmbeddingId] {\n      case SimClustersEmbeddingId(_, _, InternalId.TweetId(tweetId)) =>\n        tweetId\n    }\n\n    MemCacheConfig.buildMemCacheStoreForSimClustersEmbedding(\n      storeWithKeyMapping,\n      cacheClient,\n      embeddingType,\n      modelVersion,\n      stats\n    )\n  }\n\n  private val underlyingStores: Map[\n    (EmbeddingType, ModelVersion),\n    ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding]\n  ] = Map(\n    // Tweet Embeddings\n    (LogFavBasedTweet, Model20m145kUpdated) -> logFavBased20M145KUpdatedTweetEmbeddingStore,\n    (LogFavBasedTweet, Model20m145k2020) -> logFavBased20M145K2020TweetEmbeddingStore,\n    (\n      LogFavLongestL2EmbeddingTweet,\n      Model20m145kUpdated) -> logFavBasedLongestL2Tweet20M145KUpdatedEmbeddingStore,\n    (\n      LogFavLongestL2EmbeddingTweet,\n      Model20m145k2020) -> logFavBasedLongestL2Tweet20M145K2020EmbeddingStore,\n  )\n\n  val tweetSimClustersEmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    SimClustersEmbeddingStore.buildWithDecider(\n      underlyingStores = underlyingStores,\n      decider = rmsDecider.decider,\n      statsReceiver = stats\n    )\n  }\n\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/scala/com/twitter/representation_manager/store/UserSimClustersEmbeddingStore.scala",
    "content": "package com.twitter.representation_manager.store\n\nimport com.twitter.contentrecommender.twistly\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.store.strato.StratoFetchableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.representation_manager.common.MemCacheConfig\nimport com.twitter.representation_manager.common.RepresentationManagerDecider\nimport com.twitter.simclusters_v2.common.ModelVersions\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.stores.SimClustersEmbeddingStore\nimport com.twitter.simclusters_v2.summingbird.stores.ProducerClusterEmbeddingReadableStores\nimport com.twitter.simclusters_v2.summingbird.stores.UserInterestedInReadableStore\nimport com.twitter.simclusters_v2.summingbird.stores.UserInterestedInReadableStore.getStore\nimport com.twitter.simclusters_v2.summingbird.stores.UserInterestedInReadableStore.modelVersionToDatasetMap\nimport com.twitter.simclusters_v2.summingbird.stores.UserInterestedInReadableStore.knownModelVersions\nimport com.twitter.simclusters_v2.summingbird.stores.UserInterestedInReadableStore.toSimClustersEmbedding\nimport com.twitter.simclusters_v2.thriftscala.ClustersUserIsInterestedIn\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType._\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion._\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.{SimClustersEmbedding => ThriftSimClustersEmbedding}\nimport com.twitter.storage.client.manhattan.kv.ManhattanKVClientMtlsParams\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.storehaus_internal.manhattan.Apollo\nimport com.twitter.storehaus_internal.manhattan.ManhattanCluster\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.strato.thrift.ScroogeConvImplicits._\nimport com.twitter.tweetypie.util.UserId\nimport com.twitter.util.Future\nimport javax.inject.Inject\n\nclass UserSimClustersEmbeddingStore @Inject() (\n  stratoClient: StratoClient,\n  cacheClient: Client,\n  globalStats: StatsReceiver,\n  mhMtlsParams: ManhattanKVClientMtlsParams,\n  rmsDecider: RepresentationManagerDecider) {\n\n  private val stats = globalStats.scope(this.getClass.getSimpleName)\n\n  private val favBasedProducer20M145KUpdatedEmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore = ProducerClusterEmbeddingReadableStores\n      .getProducerTopKSimClustersEmbeddingsStore(\n        mhMtlsParams\n      ).mapValues { topSimClustersWithScore =>\n        ThriftSimClustersEmbedding(topSimClustersWithScore.topClusters)\n      }.composeKeyMapping[SimClustersEmbeddingId] {\n        case SimClustersEmbeddingId(_, _, InternalId.UserId(userId)) =>\n          userId\n      }\n\n    buildMemCacheStore(rawStore, FavBasedProducer, Model20m145kUpdated)\n  }\n\n  private val favBasedProducer20M145K2020EmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore = ProducerClusterEmbeddingReadableStores\n      .getProducerTopKSimClusters2020EmbeddingsStore(\n        mhMtlsParams\n      ).mapValues { topSimClustersWithScore =>\n        ThriftSimClustersEmbedding(topSimClustersWithScore.topClusters)\n      }.composeKeyMapping[SimClustersEmbeddingId] {\n        case SimClustersEmbeddingId(_, _, InternalId.UserId(userId)) =>\n          userId\n      }\n\n    buildMemCacheStore(rawStore, FavBasedProducer, Model20m145k2020)\n  }\n\n  private val followBasedProducer20M145K2020EmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore = ProducerClusterEmbeddingReadableStores\n      .getProducerTopKSimClustersEmbeddingsByFollowStore(\n        mhMtlsParams\n      ).mapValues { topSimClustersWithScore =>\n        ThriftSimClustersEmbedding(topSimClustersWithScore.topClusters)\n      }.composeKeyMapping[SimClustersEmbeddingId] {\n        case SimClustersEmbeddingId(_, _, InternalId.UserId(userId)) =>\n          userId\n      }\n\n    buildMemCacheStore(rawStore, FollowBasedProducer, Model20m145k2020)\n  }\n\n  private val logFavBasedApe20M145K2020EmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore = StratoFetchableStore\n      .withUnitView[SimClustersEmbeddingId, ThriftSimClustersEmbedding](\n        stratoClient,\n        \"recommendations/simclusters_v2/embeddings/logFavBasedAPE20M145K2020\")\n      .mapValues(embedding => SimClustersEmbedding(embedding, truncate = 50).toThrift)\n\n    buildMemCacheStore(rawStore, AggregatableLogFavBasedProducer, Model20m145k2020)\n  }\n\n  private val rawRelaxedLogFavBasedApe20M145K2020EmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    ThriftSimClustersEmbedding\n  ] = {\n    StratoFetchableStore\n      .withUnitView[SimClustersEmbeddingId, ThriftSimClustersEmbedding](\n        stratoClient,\n        \"recommendations/simclusters_v2/embeddings/logFavBasedAPERelaxedFavEngagementThreshold20M145K2020\")\n      .mapValues(embedding => SimClustersEmbedding(embedding, truncate = 50).toThrift)\n  }\n\n  private val relaxedLogFavBasedApe20M145K2020EmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildMemCacheStore(\n      rawRelaxedLogFavBasedApe20M145K2020EmbeddingStore,\n      RelaxedAggregatableLogFavBasedProducer,\n      Model20m145k2020)\n  }\n\n  private val relaxedLogFavBasedApe20m145kUpdatedEmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore = rawRelaxedLogFavBasedApe20M145K2020EmbeddingStore\n      .composeKeyMapping[SimClustersEmbeddingId] {\n        case SimClustersEmbeddingId(\n              RelaxedAggregatableLogFavBasedProducer,\n              Model20m145kUpdated,\n              internalId) =>\n          SimClustersEmbeddingId(\n            RelaxedAggregatableLogFavBasedProducer,\n            Model20m145k2020,\n            internalId)\n      }\n\n    buildMemCacheStore(rawStore, RelaxedAggregatableLogFavBasedProducer, Model20m145kUpdated)\n  }\n\n  private val logFavBasedInterestedInFromAPE20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildUserInterestedInStore(\n      UserInterestedInReadableStore.defaultIIAPESimClustersEmbeddingStoreWithMtls,\n      LogFavBasedUserInterestedInFromAPE,\n      Model20m145k2020)\n  }\n\n  private val followBasedInterestedInFromAPE20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildUserInterestedInStore(\n      UserInterestedInReadableStore.defaultIIAPESimClustersEmbeddingStoreWithMtls,\n      FollowBasedUserInterestedInFromAPE,\n      Model20m145k2020)\n  }\n\n  private val favBasedUserInterestedIn20M145KUpdatedStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildUserInterestedInStore(\n      UserInterestedInReadableStore.defaultSimClustersEmbeddingStoreWithMtls,\n      FavBasedUserInterestedIn,\n      Model20m145kUpdated)\n  }\n\n  private val favBasedUserInterestedIn20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildUserInterestedInStore(\n      UserInterestedInReadableStore.defaultSimClustersEmbeddingStoreWithMtls,\n      FavBasedUserInterestedIn,\n      Model20m145k2020)\n  }\n\n  private val followBasedUserInterestedIn20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildUserInterestedInStore(\n      UserInterestedInReadableStore.defaultSimClustersEmbeddingStoreWithMtls,\n      FollowBasedUserInterestedIn,\n      Model20m145k2020)\n  }\n\n  private val logFavBasedUserInterestedIn20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildUserInterestedInStore(\n      UserInterestedInReadableStore.defaultSimClustersEmbeddingStoreWithMtls,\n      LogFavBasedUserInterestedIn,\n      Model20m145k2020)\n  }\n\n  private val favBasedUserInterestedInFromPE20M145KUpdatedStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildUserInterestedInStore(\n      UserInterestedInReadableStore.defaultIIPESimClustersEmbeddingStoreWithMtls,\n      FavBasedUserInterestedInFromPE,\n      Model20m145kUpdated)\n  }\n\n  private val twistlyUserInterestedInStore: ReadableStore[\n    SimClustersEmbeddingId,\n    ThriftSimClustersEmbedding\n  ] = {\n    val interestedIn20M145KUpdatedStore = {\n      UserInterestedInReadableStore.defaultStoreWithMtls(\n        mhMtlsParams,\n        modelVersion = ModelVersions.Model20M145KUpdated\n      )\n    }\n    val interestedIn20M145K2020Store = {\n      UserInterestedInReadableStore.defaultStoreWithMtls(\n        mhMtlsParams,\n        modelVersion = ModelVersions.Model20M145K2020\n      )\n    }\n    val interestedInFromPE20M145KUpdatedStore = {\n      UserInterestedInReadableStore.defaultIIPEStoreWithMtls(\n        mhMtlsParams,\n        modelVersion = ModelVersions.Model20M145KUpdated)\n    }\n    val simClustersInterestedInStore: ReadableStore[\n      (UserId, ModelVersion),\n      ClustersUserIsInterestedIn\n    ] = {\n      new ReadableStore[(UserId, ModelVersion), ClustersUserIsInterestedIn] {\n        override def get(k: (UserId, ModelVersion)): Future[Option[ClustersUserIsInterestedIn]] = {\n          k match {\n            case (userId, Model20m145kUpdated) =>\n              interestedIn20M145KUpdatedStore.get(userId)\n            case (userId, Model20m145k2020) =>\n              interestedIn20M145K2020Store.get(userId)\n            case _ =>\n              Future.None\n          }\n        }\n      }\n    }\n    val simClustersInterestedInFromProducerEmbeddingsStore: ReadableStore[\n      (UserId, ModelVersion),\n      ClustersUserIsInterestedIn\n    ] = {\n      new ReadableStore[(UserId, ModelVersion), ClustersUserIsInterestedIn] {\n        override def get(k: (UserId, ModelVersion)): Future[Option[ClustersUserIsInterestedIn]] = {\n          k match {\n            case (userId, ModelVersion.Model20m145kUpdated) =>\n              interestedInFromPE20M145KUpdatedStore.get(userId)\n            case _ =>\n              Future.None\n          }\n        }\n      }\n    }\n    new twistly.interestedin.EmbeddingStore(\n      interestedInStore = simClustersInterestedInStore,\n      interestedInFromProducerEmbeddingStore = simClustersInterestedInFromProducerEmbeddingsStore,\n      statsReceiver = stats\n    ).mapValues(_.toThrift)\n  }\n\n  private val userNextInterestedIn20m145k2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildUserInterestedInStore(\n      UserInterestedInReadableStore.defaultNextInterestedInStoreWithMtls,\n      UserNextInterestedIn,\n      Model20m145k2020)\n  }\n\n  private val filteredUserInterestedIn20m145kUpdatedStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildMemCacheStore(twistlyUserInterestedInStore, FilteredUserInterestedIn, Model20m145kUpdated)\n  }\n\n  private val filteredUserInterestedIn20m145k2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildMemCacheStore(twistlyUserInterestedInStore, FilteredUserInterestedIn, Model20m145k2020)\n  }\n\n  private val filteredUserInterestedInFromPE20m145kUpdatedStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildMemCacheStore(\n      twistlyUserInterestedInStore,\n      FilteredUserInterestedInFromPE,\n      Model20m145kUpdated)\n  }\n\n  private val unfilteredUserInterestedIn20m145kUpdatedStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildMemCacheStore(\n      twistlyUserInterestedInStore,\n      UnfilteredUserInterestedIn,\n      Model20m145kUpdated)\n  }\n\n  private val unfilteredUserInterestedIn20m145k2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    buildMemCacheStore(twistlyUserInterestedInStore, UnfilteredUserInterestedIn, Model20m145k2020)\n  }\n\n  // [Experimental] User InterestedIn, generated by aggregating IIAPE embedding from AddressBook\n\n  private val logFavBasedInterestedMaxpoolingAddressBookFromIIAPE20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val datasetName = \"addressbook_sims_embedding_iiape_maxpooling\"\n    val appId = \"wtf_embedding_apollo\"\n    buildUserInterestedInStoreGeneric(\n      simClustersEmbeddingStoreWithMtls,\n      LogFavBasedUserInterestedMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020,\n      datasetName = datasetName,\n      appId = appId,\n      manhattanCluster = Apollo\n    )\n  }\n\n  private val logFavBasedInterestedAverageAddressBookFromIIAPE20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val datasetName = \"addressbook_sims_embedding_iiape_average\"\n    val appId = \"wtf_embedding_apollo\"\n    buildUserInterestedInStoreGeneric(\n      simClustersEmbeddingStoreWithMtls,\n      LogFavBasedUserInterestedAverageAddressBookFromIIAPE,\n      Model20m145k2020,\n      datasetName = datasetName,\n      appId = appId,\n      manhattanCluster = Apollo\n    )\n  }\n\n  private val logFavBasedUserInterestedBooktypeMaxpoolingAddressBookFromIIAPE20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val datasetName = \"addressbook_sims_embedding_iiape_booktype_maxpooling\"\n    val appId = \"wtf_embedding_apollo\"\n    buildUserInterestedInStoreGeneric(\n      simClustersEmbeddingStoreWithMtls,\n      LogFavBasedUserInterestedBooktypeMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020,\n      datasetName = datasetName,\n      appId = appId,\n      manhattanCluster = Apollo\n    )\n  }\n\n  private val logFavBasedUserInterestedLargestDimMaxpoolingAddressBookFromIIAPE20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val datasetName = \"addressbook_sims_embedding_iiape_largestdim_maxpooling\"\n    val appId = \"wtf_embedding_apollo\"\n    buildUserInterestedInStoreGeneric(\n      simClustersEmbeddingStoreWithMtls,\n      LogFavBasedUserInterestedLargestDimMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020,\n      datasetName = datasetName,\n      appId = appId,\n      manhattanCluster = Apollo\n    )\n  }\n\n  private val logFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val datasetName = \"addressbook_sims_embedding_iiape_louvain_maxpooling\"\n    val appId = \"wtf_embedding_apollo\"\n    buildUserInterestedInStoreGeneric(\n      simClustersEmbeddingStoreWithMtls,\n      LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020,\n      datasetName = datasetName,\n      appId = appId,\n      manhattanCluster = Apollo\n    )\n  }\n\n  private val logFavBasedUserInterestedConnectedMaxpoolingAddressBookFromIIAPE20M145K2020Store: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val datasetName = \"addressbook_sims_embedding_iiape_connected_maxpooling\"\n    val appId = \"wtf_embedding_apollo\"\n    buildUserInterestedInStoreGeneric(\n      simClustersEmbeddingStoreWithMtls,\n      LogFavBasedUserInterestedConnectedMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020,\n      datasetName = datasetName,\n      appId = appId,\n      manhattanCluster = Apollo\n    )\n  }\n\n  /**\n   * Helper func to build a readable store for some UserInterestedIn embeddings with\n   *    1. A storeFunc from UserInterestedInReadableStore\n   *    2. EmbeddingType\n   *    3. ModelVersion\n   *    4. MemCacheConfig\n   * */\n  private def buildUserInterestedInStore(\n    storeFunc: (ManhattanKVClientMtlsParams, EmbeddingType, ModelVersion) => ReadableStore[\n      SimClustersEmbeddingId,\n      SimClustersEmbedding\n    ],\n    embeddingType: EmbeddingType,\n    modelVersion: ModelVersion\n  ): ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore = storeFunc(mhMtlsParams, embeddingType, modelVersion)\n      .mapValues(_.toThrift)\n    val observedStore = ObservedReadableStore(\n      store = rawStore\n    )(stats.scope(embeddingType.name).scope(modelVersion.name))\n\n    MemCacheConfig.buildMemCacheStoreForSimClustersEmbedding(\n      observedStore,\n      cacheClient,\n      embeddingType,\n      modelVersion,\n      stats\n    )\n  }\n\n  private def buildUserInterestedInStoreGeneric(\n    storeFunc: (ManhattanKVClientMtlsParams, EmbeddingType, ModelVersion, String, String,\n      ManhattanCluster) => ReadableStore[\n      SimClustersEmbeddingId,\n      SimClustersEmbedding\n    ],\n    embeddingType: EmbeddingType,\n    modelVersion: ModelVersion,\n    datasetName: String,\n    appId: String,\n    manhattanCluster: ManhattanCluster\n  ): ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    val rawStore =\n      storeFunc(mhMtlsParams, embeddingType, modelVersion, datasetName, appId, manhattanCluster)\n        .mapValues(_.toThrift)\n    val observedStore = ObservedReadableStore(\n      store = rawStore\n    )(stats.scope(embeddingType.name).scope(modelVersion.name))\n\n    MemCacheConfig.buildMemCacheStoreForSimClustersEmbedding(\n      observedStore,\n      cacheClient,\n      embeddingType,\n      modelVersion,\n      stats\n    )\n  }\n\n  private def simClustersEmbeddingStoreWithMtls(\n    mhMtlsParams: ManhattanKVClientMtlsParams,\n    embeddingType: EmbeddingType,\n    modelVersion: ModelVersion,\n    datasetName: String,\n    appId: String,\n    manhattanCluster: ManhattanCluster\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n\n    if (!modelVersionToDatasetMap.contains(ModelVersions.toKnownForModelVersion(modelVersion))) {\n      throw new IllegalArgumentException(\n        \"Unknown model version: \" + modelVersion + \". Known model versions: \" + knownModelVersions)\n    }\n    getStore(appId, mhMtlsParams, datasetName, manhattanCluster)\n      .composeKeyMapping[SimClustersEmbeddingId] {\n        case SimClustersEmbeddingId(theEmbeddingType, theModelVersion, InternalId.UserId(userId))\n            if theEmbeddingType == embeddingType && theModelVersion == modelVersion =>\n          userId\n      }.mapValues(toSimClustersEmbedding(_, embeddingType))\n  }\n\n  private def buildMemCacheStore(\n    rawStore: ReadableStore[SimClustersEmbeddingId, ThriftSimClustersEmbedding],\n    embeddingType: EmbeddingType,\n    modelVersion: ModelVersion\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n    val observedStore = ObservedReadableStore(\n      store = rawStore\n    )(stats.scope(embeddingType.name).scope(modelVersion.name))\n\n    MemCacheConfig.buildMemCacheStoreForSimClustersEmbedding(\n      observedStore,\n      cacheClient,\n      embeddingType,\n      modelVersion,\n      stats\n    )\n  }\n\n  private val underlyingStores: Map[\n    (EmbeddingType, ModelVersion),\n    ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding]\n  ] = Map(\n    // KnownFor Embeddings\n    (FavBasedProducer, Model20m145kUpdated) -> favBasedProducer20M145KUpdatedEmbeddingStore,\n    (FavBasedProducer, Model20m145k2020) -> favBasedProducer20M145K2020EmbeddingStore,\n    (FollowBasedProducer, Model20m145k2020) -> followBasedProducer20M145K2020EmbeddingStore,\n    (AggregatableLogFavBasedProducer, Model20m145k2020) -> logFavBasedApe20M145K2020EmbeddingStore,\n    (\n      RelaxedAggregatableLogFavBasedProducer,\n      Model20m145kUpdated) -> relaxedLogFavBasedApe20m145kUpdatedEmbeddingStore,\n    (\n      RelaxedAggregatableLogFavBasedProducer,\n      Model20m145k2020) -> relaxedLogFavBasedApe20M145K2020EmbeddingStore,\n    // InterestedIn Embeddings\n    (\n      LogFavBasedUserInterestedInFromAPE,\n      Model20m145k2020) -> logFavBasedInterestedInFromAPE20M145K2020Store,\n    (\n      FollowBasedUserInterestedInFromAPE,\n      Model20m145k2020) -> followBasedInterestedInFromAPE20M145K2020Store,\n    (FavBasedUserInterestedIn, Model20m145kUpdated) -> favBasedUserInterestedIn20M145KUpdatedStore,\n    (FavBasedUserInterestedIn, Model20m145k2020) -> favBasedUserInterestedIn20M145K2020Store,\n    (FollowBasedUserInterestedIn, Model20m145k2020) -> followBasedUserInterestedIn20M145K2020Store,\n    (LogFavBasedUserInterestedIn, Model20m145k2020) -> logFavBasedUserInterestedIn20M145K2020Store,\n    (\n      FavBasedUserInterestedInFromPE,\n      Model20m145kUpdated) -> favBasedUserInterestedInFromPE20M145KUpdatedStore,\n    (FilteredUserInterestedIn, Model20m145kUpdated) -> filteredUserInterestedIn20m145kUpdatedStore,\n    (FilteredUserInterestedIn, Model20m145k2020) -> filteredUserInterestedIn20m145k2020Store,\n    (\n      FilteredUserInterestedInFromPE,\n      Model20m145kUpdated) -> filteredUserInterestedInFromPE20m145kUpdatedStore,\n    (\n      UnfilteredUserInterestedIn,\n      Model20m145kUpdated) -> unfilteredUserInterestedIn20m145kUpdatedStore,\n    (UnfilteredUserInterestedIn, Model20m145k2020) -> unfilteredUserInterestedIn20m145k2020Store,\n    (UserNextInterestedIn, Model20m145k2020) -> userNextInterestedIn20m145k2020Store,\n    (\n      LogFavBasedUserInterestedMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020) -> logFavBasedInterestedMaxpoolingAddressBookFromIIAPE20M145K2020Store,\n    (\n      LogFavBasedUserInterestedAverageAddressBookFromIIAPE,\n      Model20m145k2020) -> logFavBasedInterestedAverageAddressBookFromIIAPE20M145K2020Store,\n    (\n      LogFavBasedUserInterestedBooktypeMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020) -> logFavBasedUserInterestedBooktypeMaxpoolingAddressBookFromIIAPE20M145K2020Store,\n    (\n      LogFavBasedUserInterestedLargestDimMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020) -> logFavBasedUserInterestedLargestDimMaxpoolingAddressBookFromIIAPE20M145K2020Store,\n    (\n      LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020) -> logFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE20M145K2020Store,\n    (\n      LogFavBasedUserInterestedConnectedMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020) -> logFavBasedUserInterestedConnectedMaxpoolingAddressBookFromIIAPE20M145K2020Store,\n  )\n\n  val userSimClustersEmbeddingStore: ReadableStore[\n    SimClustersEmbeddingId,\n    SimClustersEmbedding\n  ] = {\n    SimClustersEmbeddingStore.buildWithDecider(\n      underlyingStores = underlyingStores,\n      decider = rmsDecider.decider,\n      statsReceiver = stats\n    )\n  }\n\n}\n"
  },
  {
    "path": "representation-manager/server/src/main/thrift/BUILD",
    "content": "create_thrift_libraries(\n    base_name = \"thrift\",\n    sources = [\n        \"com/twitter/representation_manager/service.thrift\",\n    ],\n    platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n    ],\n    dependency_roots = [\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift\",\n    ],\n    generate_languages = [\n        \"java\",\n        \"scala\",\n        \"strato\",\n    ],\n)\n"
  },
  {
    "path": "representation-manager/server/src/main/thrift/com/twitter/representation_manager/service.thrift",
    "content": "namespace java com.twitter.representation_manager.thriftjava\n#@namespace scala com.twitter.representation_manager.thriftscala\n#@namespace strato com.twitter.representation_manager\n\ninclude \"com/twitter/simclusters_v2/online_store.thrift\"\ninclude \"com/twitter/simclusters_v2/identifier.thrift\"\n\n/**\n  * A uniform column view for all kinds of SimClusters based embeddings.\n  **/\nstruct SimClustersEmbeddingView {\n  1: required identifier.EmbeddingType embeddingType\n  2: required online_store.ModelVersion modelVersion\n}(persisted = 'false', hasPersonalData = 'false')\n"
  },
  {
    "path": "representation-scorer/BUILD.bazel",
    "content": "# This prevents SQ query from grabbing //:all since it traverses up once to find a BUILD\n"
  },
  {
    "path": "representation-scorer/README.md",
    "content": "# Representation Scorer #\n\n**Representation Scorer** (RSX) serves as a centralized scoring system, offering SimClusters or other embedding-based scoring solutions as machine learning features.\n\nThe Representation Scorer acquires user behavior data from the User Signal Service (USS) and extracts embeddings from the Representation Manager (RMS). It then calculates both pairwise and listwise features. These features are used at various stages, including candidate retrieval and ranking."
  },
  {
    "path": "representation-scorer/bin/canary-check.sh",
    "content": "#!/bin/bash\n\nexport CANARY_CHECK_ROLE=\"representation-scorer\"\nexport CANARY_CHECK_NAME=\"representation-scorer\"\nexport CANARY_CHECK_INSTANCES=\"0-19\"\n\npython3 relevance-platform/tools/canary_check.py \"$@\"\n\n"
  },
  {
    "path": "representation-scorer/bin/deploy.sh",
    "content": "#!/usr/bin/env bash\n\nJOB=representation-scorer bazel run --ui_event_filters=-info,-stdout,-stderr --noshow_progress \\\n\t//relevance-platform/src/main/python/deploy -- \"$@\"\n"
  },
  {
    "path": "representation-scorer/bin/remote-debug-tunnel.sh",
    "content": "#!/bin/bash\n\nset -o nounset\nset -eu\n\nDC=\"atla\"\nROLE=\"$USER\"\nSERVICE=\"representation-scorer\"\nINSTANCE=\"0\"\nKEY=\"$DC/$ROLE/devel/$SERVICE/$INSTANCE\"\n\nwhile test $# -gt 0; do\n  case \"$1\" in\n    -h|--help)\n      echo \"$0 Set up an ssh tunnel for $SERVICE remote debugging and disable aurora health checks\"\n      echo \" \"\n      echo \"See representation-scorer/README.md for details of how to use this script, and go/remote-debug for\"\n      echo \"general information about remote debugging in Aurora\"\n      echo \" \"\n      echo \"Default instance if called with no args:\"\n      echo \"  $KEY\"\n      echo \" \"\n      echo \"Positional args:\"\n      echo \"  $0 [datacentre] [role] [service_name] [instance]\"\n      echo \" \"\n      echo \"Options:\"\n      echo \"  -h, --help                show brief help\"\n      exit 0\n      ;;\n    *)\n      break\n      ;;\n  esac\ndone\n\nif [ -n \"${1-}\" ]; then\n  DC=\"$1\"\nfi\n\nif [ -n \"${2-}\" ]; then\n  ROLE=\"$2\"\nfi\n\nif [ -n \"${3-}\" ]; then\n  SERVICE=\"$3\"\nfi\n\nif [ -n \"${4-}\" ]; then\n  INSTANCE=\"$4\"\nfi\n\nKEY=\"$DC/$ROLE/devel/$SERVICE/$INSTANCE\"\nread -p \"Set up remote debugger tunnel for $KEY? (y/n) \" -r CONFIRM\nif [[ ! $CONFIRM =~ ^[Yy]$ ]]; then\n  echo \"Exiting, tunnel not created\"\n  exit 1\nfi\n\necho \"Disabling health check and opening tunnel. Exit with control-c when you're finished\"\nCMD=\"aurora task ssh $KEY -c 'touch .healthchecksnooze' && aurora task ssh $KEY -L '5005:debug' --ssh-options '-N -S none -v '\"\n\necho \"Running $CMD\"\neval \"$CMD\"\n\n\n\n"
  },
  {
    "path": "representation-scorer/docs/index.rst",
    "content": "Representation Scorer (RSX)\n###########################\n\nOverview\n========\n\nRepresentation Scorer (RSX) is a StratoFed service which serves scores for pairs of entities (User, Tweet, Topic...) based on some representation of those entities. For example, it serves User-Tweet scores based on the cosine similarity of SimClusters embeddings for each of these.  It aims to provide these with low latency and at high scale, to support applications such as scoring for ANN candidate generation and feature hydration via feature store.\n\n\nCurrent use cases\n-----------------\n\nRSX currently serves traffic for the following use cases:\n\n- User-Tweet similarity scores for Home ranking, using SimClusters embedding dot product\n- Topic-Tweet similarity scores for topical tweet candidate generation and topic social proof, using SimClusters embedding cosine similarity and CERTO scores\n- Tweet-Tweet and User-Tweet similarity scores for ANN candidate generation, using SimClusters embedding cosine similarity \n- (in development) User-Tweet similarity scores for Home ranking, based on various aggregations of similarities with recent faves, retweets and follows performed by the user\n\nGetting Started\n===============\n\nFetching scores\n---------------\n\nScores are served from the recommendations/representation_scorer/score column.\n\nUsing RSX for your application\n------------------------------\n\nRSX may be a good fit for your application if you need scores based on combinations of SimCluster embeddings for core nouns. We also plan to support other embeddings and scoring approaches in the future.\n\n.. toctree::\n   :maxdepth: 2\n   :hidden:\n\n   index\n   \n\n"
  },
  {
    "path": "representation-scorer/server/BUILD",
    "content": "jvm_binary(\n    name = \"bin\",\n    basename = \"representation-scorer\",\n    main = \"com.twitter.representationscorer.RepresentationScorerFedServerMain\",\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finatra/inject/inject-logback/src/main/scala\",\n        \"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback\",\n        \"representation-scorer/server/src/main/resources\",\n        \"representation-scorer/server/src/main/scala/com/twitter/representationscorer\",\n        \"twitter-server/logback-classic/src/main/scala\",\n    ],\n)\n\n#  Aurora Workflows build phase convention requires a jvm_app named with ${project-name}-app\njvm_app(\n    name = \"representation-scorer-app\",\n    archive = \"zip\",\n    binary = \":bin\",\n    tags = [\"bazel-compatible\"],\n)\n"
  },
  {
    "path": "representation-scorer/server/src/main/resources/BUILD",
    "content": "resources(\n    sources = [\n        \"*.xml\",\n        \"*.yml\",\n        \"com/twitter/slo/slo.json\",\n        \"config/*.yml\",\n    ],\n    tags = [\"bazel-compatible\"],\n)\n"
  },
  {
    "path": "representation-scorer/server/src/main/resources/com/twitter/slo/slo.json",
    "content": "{\n  \"servers\": [\n    {\n      \"name\": \"strato\",\n      \"indicators\": [\n        {\n          \"id\": \"success_rate_3m\",\n          \"indicator_type\": \"SuccessRateIndicator\",\n          \"duration\": 3,\n          \"duration_unit\": \"MINUTES\"\n        }, {\n          \"id\": \"latency_3m_p99\",\n          \"indicator_type\": \"LatencyIndicator\",\n          \"duration\": 3,\n          \"duration_unit\": \"MINUTES\",\n          \"percentile\": 0.99\n        }\n      ],\n      \"objectives\": [\n        {\n          \"indicator\": \"success_rate_3m\",\n          \"objective_type\": \"SuccessRateObjective\",\n          \"operator\": \">=\",\n          \"threshold\": 0.995\n        },\n        {\n          \"indicator\": \"latency_3m_p99\",\n          \"objective_type\": \"LatencyObjective\",\n          \"operator\": \"<=\",\n          \"threshold\": 50\n        }\n      ],\n      \"long_term_objectives\": [\n        {\n          \"id\": \"success_rate_28_days\",\n          \"objective_type\": \"SuccessRateObjective\",\n          \"operator\": \">=\",\n          \"threshold\": 0.993,\n          \"duration\": 28,\n          \"duration_unit\": \"DAYS\"\n        },\n        {\n          \"id\": \"latency_p99_28_days\",\n          \"objective_type\": \"LatencyObjective\",\n          \"operator\": \"<=\",\n          \"threshold\": 60,\n          \"duration\": 28,\n          \"duration_unit\": \"DAYS\",\n          \"percentile\": 0.99\n        }\n      ]\n    }\n  ],\n  \"@version\": 1\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/resources/config/decider.yml",
    "content": "enableLogFavBasedApeEntity20M145KUpdatedEmbeddingCachedStore:\n  comment: \"Enable to use the non-empty store for logFavBasedApeEntity20M145KUpdatedEmbeddingCachedStore (from 0% to 100%). 0 means use EMPTY readable store for all requests.\"\n  default_availability: 0\n\nenableLogFavBasedApeEntity20M145K2020EmbeddingCachedStore:\n  comment: \"Enable to use the non-empty store for logFavBasedApeEntity20M145K2020EmbeddingCachedStore (from 0% to 100%). 0 means use EMPTY readable store for all requests.\"\n  default_availability: 0\n\nrepresentation-scorer_forward_dark_traffic:\n  comment: \"Defines the percentage of traffic to forward to diffy-proxy. Set to 0 to disable dark traffic forwarding\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_non_prod_callers\":\n  comment: \"Discard traffic from all non-prod callers\"\n  default_availability: 0\n\nenable_log_fav_based_tweet_embedding_20m145k2020_timeouts:\n  comment: \"If enabled, set a timeout on calls to the logFavBased20M145K2020TweetEmbeddingStore\"\n  default_availability: 0\n\nlog_fav_based_tweet_embedding_20m145k2020_timeout_value_millis:\n  comment: \"The value of this decider defines the timeout (in milliseconds) to use on calls to the logFavBased20M145K2020TweetEmbeddingStore, i.e. 1.50% is 150ms. Only applied if enable_log_fav_based_tweet_embedding_20m145k2020_timeouts is true\"\n  default_availability: 2000\n\nenable_log_fav_based_tweet_embedding_20m145kUpdated_timeouts:\n  comment: \"If enabled, set a timeout on calls to the logFavBased20M145KUpdatedTweetEmbeddingStore\"\n  default_availability: 0\n\nlog_fav_based_tweet_embedding_20m145kUpdated_timeout_value_millis:\n  comment: \"The value of this decider defines the timeout (in milliseconds) to use on calls to the logFavBased20M145KUpdatedTweetEmbeddingStore, i.e. 1.50% is 150ms. Only applied if enable_log_fav_based_tweet_embedding_20m145kUpdated_timeouts is true\"\n  default_availability: 2000\n\nenable_cluster_tweet_index_store_timeouts:\n  comment: \"If enabled, set a timeout on calls to the ClusterTweetIndexStore\"\n  default_availability: 0\n\ncluster_tweet_index_store_timeout_value_millis:\n  comment: \"The value of this decider defines the timeout (in milliseconds) to use on calls to the ClusterTweetIndexStore, i.e. 1.50% is 150ms. Only applied if enable_cluster_tweet_index_store_timeouts is true\"\n  default_availability: 2000\n\nrepresentation_scorer_fetch_signal_share:\n  comment: \"If enabled, fetches share signals from USS\"\n  default_availability: 0\n\nrepresentation_scorer_fetch_signal_reply:\n  comment: \"If enabled, fetches reply signals from USS\"\n  default_availability: 0\n\nrepresentation_scorer_fetch_signal_original_tweet:\n  comment: \"If enabled, fetches original tweet signals from USS\"\n  default_availability: 0\n\nrepresentation_scorer_fetch_signal_video_playback:\n  comment: \"If enabled, fetches video playback signals from USS\"\n  default_availability: 0\n\nrepresentation_scorer_fetch_signal_block:\n  comment: \"If enabled, fetches account block signals from USS\"\n  default_availability: 0\n\nrepresentation_scorer_fetch_signal_mute:\n  comment: \"If enabled, fetches account mute signals from USS\"\n  default_availability: 0\n\nrepresentation_scorer_fetch_signal_report:\n  comment: \"If enabled, fetches tweet report signals from USS\"\n  default_availability: 0\n\nrepresentation_scorer_fetch_signal_dont_like:\n  comment: \"If enabled, fetches tweet don't like signals from USS\"\n  default_availability: 0\n\nrepresentation_scorer_fetch_signal_see_fewer:\n  comment: \"If enabled, fetches tweet see fewer signals from USS\"\n  default_availability: 0\n\n# To create a new decider, add here with the same format and caller's details : \"representation-scorer_load_shed_by_caller_id_twtr:{{role}}:{{name}}:{{environment}}:{{cluster}}\"\n# All the deciders below are generated by this script - ./strato/bin/fed deciders ./ --service-role=representation-scorer --service-name=representation-scorer\n# If you need to run the script and paste the output, add only the prod deciders here. Non-prod ones are being taken care of by representation-scorer_load_shed_non_prod_callers\n\n\"representation-scorer_load_shed_by_caller_id_all\":\n  comment: \"Reject all traffic from caller id: all\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:frigate:frigate-pushservice-canary:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:frigate:frigate-pushservice-canary:prod:atla\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:frigate:frigate-pushservice-canary:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:frigate:frigate-pushservice-canary:prod:pdxa\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:frigate:frigate-pushservice-send:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:frigate:frigate-pushservice-send:prod:atla\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:frigate:frigate-pushservice:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:frigate:frigate-pushservice:prod:atla\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:frigate:frigate-pushservice:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:frigate:frigate-pushservice:prod:pdxa\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:frigate:frigate-pushservice:staging:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:frigate:frigate-pushservice:staging:atla\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:frigate:frigate-pushservice:staging:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:frigate:frigate-pushservice:staging:pdxa\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:home-scorer:home-scorer:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:home-scorer:home-scorer:prod:atla\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:home-scorer:home-scorer:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:home-scorer:home-scorer:prod:pdxa\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:stratostore:stratoapi:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:stratostore:stratoapi:prod:atla\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:stratostore:stratoserver:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:stratostore:stratoserver:prod:atla\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:stratostore:stratoserver:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:stratostore:stratoserver:prod:pdxa\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:timelinescorer:timelinescorer:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:timelinescorer:timelinescorer:prod:atla\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:timelinescorer:timelinescorer:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:timelinescorer:timelinescorer:prod:pdxa\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:topic-social-proof:topic-social-proof:prod:atla\":\n  comment: \"Reject all traffic from caller id: twtr:svc:topic-social-proof:topic-social-proof:prod:atla\"\n  default_availability: 0\n\n\"representation-scorer_load_shed_by_caller_id_twtr:svc:topic-social-proof:topic-social-proof:prod:pdxa\":\n  comment: \"Reject all traffic from caller id: twtr:svc:topic-social-proof:topic-social-proof:prod:pdxa\"\n  default_availability: 0\n\n\"enable_sim_clusters_embedding_store_timeouts\":\n  comment: \"If enabled, set a timeout on calls to the SimClustersEmbeddingStore\"\n  default_availability: 10000\n\nsim_clusters_embedding_store_timeout_value_millis:\n  comment: \"The value of this decider defines the timeout (in milliseconds) to use on calls to the SimClustersEmbeddingStore, i.e. 1.50% is 150ms. Only applied if enable_sim_clusters_embedding_store_timeouts is true\"\n  default_availability: 2000\n"
  },
  {
    "path": "representation-scorer/server/src/main/resources/logback.xml",
    "content": "<configuration>\n    <shutdownHook class=\"ch.qos.logback.core.hook.DelayingShutdownHook\"/>\n\n    <!-- ===================================================== -->\n    <!-- Service Config -->\n    <!-- ===================================================== -->\n    <property name=\"DEFAULT_SERVICE_PATTERN\"\n              value=\"%-16X{traceId} %-12X{clientId:--} %-16X{method} %-25logger{0} %msg\"/>\n\n    <property name=\"DEFAULT_ACCESS_PATTERN\"\n              value=\"%msg\"/>\n\n    <!-- ===================================================== -->\n    <!-- Common Config -->\n    <!-- ===================================================== -->\n\n    <!-- JUL/JDK14 to Logback bridge -->\n    <contextListener class=\"ch.qos.logback.classic.jul.LevelChangePropagator\">\n        <resetJUL>true</resetJUL>\n    </contextListener>\n\n    <!-- ====================================================================================== -->\n    <!-- NOTE: The following appenders use a simple TimeBasedRollingPolicy configuration.       -->\n    <!--       You may want to consider using a more advanced SizeAndTimeBasedRollingPolicy.    -->\n    <!--       See: https://logback.qos.ch/manual/appenders.html#SizeAndTimeBasedRollingPolicy  -->\n    <!-- ====================================================================================== -->\n\n    <!-- Service Log (rollover daily, keep maximum of 21 days of gzip compressed logs) -->\n    <appender name=\"SERVICE\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n        <file>${log.service.output}</file>\n        <rollingPolicy class=\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\">\n            <!-- daily rollover -->\n            <fileNamePattern>${log.service.output}.%d.gz</fileNamePattern>\n            <!-- the maximum total size of all the log files -->\n            <totalSizeCap>3GB</totalSizeCap>\n            <!-- keep maximum 21 days' worth of history -->\n            <maxHistory>21</maxHistory>\n            <cleanHistoryOnStart>true</cleanHistoryOnStart>\n        </rollingPolicy>\n        <encoder>\n            <pattern>%date %.-3level ${DEFAULT_SERVICE_PATTERN}%n</pattern>\n        </encoder>\n    </appender>\n\n    <!-- Access Log (rollover daily, keep maximum of 21 days of gzip compressed logs) -->\n    <appender name=\"ACCESS\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n        <file>${log.access.output}</file>\n        <rollingPolicy class=\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\">\n            <!-- daily rollover -->\n            <fileNamePattern>${log.access.output}.%d.gz</fileNamePattern>\n            <!-- the maximum total size of all the log files -->\n            <totalSizeCap>100MB</totalSizeCap>\n            <!-- keep maximum 7 days' worth of history -->\n            <maxHistory>7</maxHistory>\n            <cleanHistoryOnStart>true</cleanHistoryOnStart>\n        </rollingPolicy>\n        <encoder>\n            <pattern>${DEFAULT_ACCESS_PATTERN}%n</pattern>\n        </encoder>\n    </appender>\n\n    <!--LogLens -->\n    <appender name=\"LOGLENS\" class=\"com.twitter.loglens.logback.LoglensAppender\">\n        <mdcAdditionalContext>true</mdcAdditionalContext>\n        <category>${log.lens.category}</category>\n        <index>${log.lens.index}</index>\n        <tag>${log.lens.tag}/service</tag>\n        <encoder>\n            <pattern>%msg</pattern>\n        </encoder>\n    </appender>\n\n    <!-- LogLens Access -->\n    <appender name=\"LOGLENS-ACCESS\" class=\"com.twitter.loglens.logback.LoglensAppender\">\n        <mdcAdditionalContext>true</mdcAdditionalContext>\n        <category>${log.lens.category}</category>\n        <index>${log.lens.index}</index>\n        <tag>${log.lens.tag}/access</tag>\n        <encoder>\n            <pattern>%msg</pattern>\n        </encoder>\n    </appender>\n\n    <!-- Pipeline Execution Logs -->\n    <appender name=\"ALLOW-LISTED-PIPELINE-EXECUTIONS\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n        <file>allow_listed_pipeline_executions.log</file>\n        <rollingPolicy class=\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\">\n            <!-- daily rollover -->\n            <fileNamePattern>allow_listed_pipeline_executions.log.%d.gz</fileNamePattern>\n            <!-- the maximum total size of all the log files -->\n            <totalSizeCap>100MB</totalSizeCap>\n            <!-- keep maximum 7 days' worth of history -->\n            <maxHistory>7</maxHistory>\n            <cleanHistoryOnStart>true</cleanHistoryOnStart>\n        </rollingPolicy>\n        <encoder>\n            <pattern>%date %.-3level ${DEFAULT_SERVICE_PATTERN}%n</pattern>\n        </encoder>\n    </appender>\n\n    <!-- ===================================================== -->\n    <!-- Primary Async Appenders -->\n    <!-- ===================================================== -->\n\n    <property name=\"async_queue_size\" value=\"${queue.size:-50000}\"/>\n    <property name=\"async_max_flush_time\" value=\"${max.flush.time:-0}\"/>\n\n    <appender name=\"ASYNC-SERVICE\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"SERVICE\"/>\n    </appender>\n\n    <appender name=\"ASYNC-ACCESS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"ACCESS\"/>\n    </appender>\n\n    <appender name=\"ASYNC-ALLOW-LISTED-PIPELINE-EXECUTIONS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"ALLOW-LISTED-PIPELINE-EXECUTIONS\"/>\n    </appender>\n\n    <appender name=\"ASYNC-LOGLENS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"LOGLENS\"/>\n    </appender>\n\n    <appender name=\"ASYNC-LOGLENS-ACCESS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"LOGLENS-ACCESS\"/>\n    </appender>\n\n    <!-- ===================================================== -->\n    <!-- Package Config -->\n    <!-- ===================================================== -->\n\n    <!-- Per-Package Config -->\n    <logger name=\"com.twitter\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.wilyns\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.configbus.client.file\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.finagle.mux\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.finagle.serverset2\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.logging.ScribeHandler\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.zookeeper.client.internal\" level=\"INHERITED\"/>\n\n    <!-- Root Config -->\n    <!-- For all logs except access logs, disable logging below log_level level by default. This can be overriden in the per-package loggers, and dynamically in the admin panel of individual instances. -->\n    <root level=\"${log_level:-INFO}\">\n        <appender-ref ref=\"ASYNC-SERVICE\"/>\n        <appender-ref ref=\"ASYNC-LOGLENS\"/>\n    </root>\n\n    <!-- Access Logging -->\n    <!-- Access logs are turned off by default -->\n    <logger name=\"com.twitter.finatra.thrift.filters.AccessLoggingFilter\" level=\"OFF\" additivity=\"false\">\n        <appender-ref ref=\"ASYNC-ACCESS\"/>\n        <appender-ref ref=\"ASYNC-LOGLENS-ACCESS\"/>\n    </logger>\n\n</configuration>\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle-internal/slo/src/main/scala/com/twitter/finagle/slo\",\n        \"finatra/inject/inject-thrift-client\",\n        \"representation-scorer/server/src/main/scala/com/twitter/representationscorer/columns\",\n        \"strato/src/main/scala/com/twitter/strato/fed\",\n        \"strato/src/main/scala/com/twitter/strato/fed/server\",\n        \"twitter-server-internal/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/RepresentationScorerFedServer.scala",
    "content": "package com.twitter.representationscorer\n\nimport com.google.inject.Module\nimport com.twitter.inject.thrift.modules.ThriftClientIdModule\nimport com.twitter.representationscorer.columns.ListScoreColumn\nimport com.twitter.representationscorer.columns.ScoreColumn\nimport com.twitter.representationscorer.columns.SimClustersRecentEngagementSimilarityColumn\nimport com.twitter.representationscorer.columns.SimClustersRecentEngagementSimilarityUserTweetEdgeColumn\nimport com.twitter.representationscorer.modules.CacheModule\nimport com.twitter.representationscorer.modules.EmbeddingStoreModule\nimport com.twitter.representationscorer.modules.RMSConfigModule\nimport com.twitter.representationscorer.modules.TimerModule\nimport com.twitter.representationscorer.twistlyfeatures.UserSignalServiceRecentEngagementsClientModule\nimport com.twitter.strato.fed._\nimport com.twitter.strato.fed.server._\n\nobject RepresentationScorerFedServerMain extends RepresentationScorerFedServer\n\ntrait RepresentationScorerFedServer extends StratoFedServer {\n  override def dest: String = \"/s/representation-scorer/representation-scorer\"\n  override val modules: Seq[Module] =\n    Seq(\n      CacheModule,\n      ThriftClientIdModule,\n      UserSignalServiceRecentEngagementsClientModule,\n      TimerModule,\n      RMSConfigModule,\n      EmbeddingStoreModule\n    )\n\n  override def columns: Seq[Class[_ <: StratoFed.Column]] =\n    Seq(\n      classOf[ListScoreColumn],\n      classOf[ScoreColumn],\n      classOf[SimClustersRecentEngagementSimilarityUserTweetEdgeColumn],\n      classOf[SimClustersRecentEngagementSimilarityColumn]\n    )\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/columns/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"content-recommender/thrift/src/main/thrift:thrift-scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"representation-scorer/server/src/main/scala/com/twitter/representationscorer/common\",\n        \"representation-scorer/server/src/main/scala/com/twitter/representationscorer/modules\",\n        \"representation-scorer/server/src/main/scala/com/twitter/representationscorer/scorestore\",\n        \"representation-scorer/server/src/main/scala/com/twitter/representationscorer/twistlyfeatures\",\n        \"representation-scorer/server/src/main/thrift:thrift-scala\",\n        \"strato/src/main/scala/com/twitter/strato/fed\",\n        \"strato/src/main/scala/com/twitter/strato/fed/server\",\n    ],\n)\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/columns/Info.scala",
    "content": "package com.twitter.representationscorer.columns\n\nimport com.twitter.strato.config.{ContactInfo => StratoContactInfo}\n\nobject Info {\n  val contactInfo: StratoContactInfo = StratoContactInfo(\n    description = \"Please contact Relevance Platform team for more details\",\n    contactEmail = \"no-reply@twitter.com\",\n    ldapGroup = \"representation-scorer-admins\",\n    jiraProject = \"JIRA\",\n    links = Seq(\"http://go.twitter.biz/rsx-runbook\")\n  )\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/columns/ListScoreColumn.scala",
    "content": "package com.twitter.representationscorer.columns\n\nimport com.twitter.representationscorer.thriftscala.ListScoreId\nimport com.twitter.representationscorer.thriftscala.ListScoreResponse\nimport com.twitter.representationscorer.scorestore.ScoreStore\nimport com.twitter.representationscorer.thriftscala.ScoreResult\nimport com.twitter.simclusters_v2.common.SimClustersEmbeddingId.LongInternalId\nimport com.twitter.simclusters_v2.common.SimClustersEmbeddingId.LongSimClustersEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.Score\nimport com.twitter.simclusters_v2.thriftscala.ScoreId\nimport com.twitter.simclusters_v2.thriftscala.ScoreInternalId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingPairScoreId\nimport com.twitter.stitch\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.OpMetadata\nimport com.twitter.strato.config.ContactInfo\nimport com.twitter.strato.config.Policy\nimport com.twitter.strato.data.Conv\nimport com.twitter.strato.data.Description.PlainText\nimport com.twitter.strato.data.Lifecycle\nimport com.twitter.strato.fed._\nimport com.twitter.strato.thrift.ScroogeConv\nimport com.twitter.util.Future\nimport com.twitter.util.Return\nimport com.twitter.util.Throw\nimport javax.inject.Inject\n\nclass ListScoreColumn @Inject() (scoreStore: ScoreStore)\n    extends StratoFed.Column(\"recommendations/representation_scorer/listScore\")\n    with StratoFed.Fetch.Stitch {\n\n  override val policy: Policy = Common.rsxReadPolicy\n\n  override type Key = ListScoreId\n  override type View = Unit\n  override type Value = ListScoreResponse\n\n  override val keyConv: Conv[Key] = ScroogeConv.fromStruct[ListScoreId]\n  override val viewConv: Conv[View] = Conv.ofType\n  override val valueConv: Conv[Value] = ScroogeConv.fromStruct[ListScoreResponse]\n\n  override val contactInfo: ContactInfo = Info.contactInfo\n\n  override val metadata: OpMetadata = OpMetadata(\n    lifecycle = Some(Lifecycle.Production),\n    description = Some(\n      PlainText(\n        \"Scoring for multiple candidate entities against a single target entity\"\n      ))\n  )\n\n  override def fetch(key: Key, view: View): Stitch[Result[Value]] = {\n\n    val target = SimClustersEmbeddingId(\n      embeddingType = key.targetEmbeddingType,\n      modelVersion = key.modelVersion,\n      internalId = key.targetId\n    )\n    val scoreIds = key.candidateIds.map { candidateId =>\n      val candidate = SimClustersEmbeddingId(\n        embeddingType = key.candidateEmbeddingType,\n        modelVersion = key.modelVersion,\n        internalId = candidateId\n      )\n      ScoreId(\n        algorithm = key.algorithm,\n        internalId = ScoreInternalId.SimClustersEmbeddingPairScoreId(\n          SimClustersEmbeddingPairScoreId(target, candidate)\n        )\n      )\n    }\n\n    Stitch\n      .callFuture {\n        val (keys: Iterable[ScoreId], vals: Iterable[Future[Option[Score]]]) =\n          scoreStore.uniformScoringStore.multiGet(scoreIds.toSet).unzip\n        val results: Future[Iterable[Option[Score]]] = Future.collectToTry(vals.toSeq) map {\n          tryOptVals =>\n            tryOptVals map {\n              case Return(Some(v)) => Some(v)\n              case Return(None) => None\n              case Throw(_) => None\n            }\n        }\n        val scoreMap: Future[Map[Long, Double]] = results.map { scores =>\n          keys\n            .zip(scores).collect {\n              case (\n                    ScoreId(\n                      _,\n                      ScoreInternalId.SimClustersEmbeddingPairScoreId(\n                        SimClustersEmbeddingPairScoreId(\n                          _,\n                          LongSimClustersEmbeddingId(candidateId)))),\n                    Some(score)) =>\n                (candidateId, score.score)\n            }.toMap\n        }\n        scoreMap\n      }\n      .map { (scores: Map[Long, Double]) =>\n        val orderedScores = key.candidateIds.collect {\n          case LongInternalId(id) => ScoreResult(scores.get(id))\n          case _ =>\n            // This will return None scores for candidates which don't have Long ids, but that's fine:\n            // at the moment we're only scoring for Tweets\n            ScoreResult(None)\n        }\n        found(ListScoreResponse(orderedScores))\n      }\n      .handle {\n        case stitch.NotFound => missing\n      }\n  }\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/columns/ScoreColumn.scala",
    "content": "package com.twitter.representationscorer.columns\n\nimport com.twitter.contentrecommender.thriftscala.ScoringResponse\nimport com.twitter.representationscorer.scorestore.ScoreStore\nimport com.twitter.simclusters_v2.thriftscala.ScoreId\nimport com.twitter.stitch\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.config.ContactInfo\nimport com.twitter.strato.config.Policy\nimport com.twitter.strato.catalog.OpMetadata\nimport com.twitter.strato.data.Conv\nimport com.twitter.strato.data.Lifecycle\nimport com.twitter.strato.data.Description.PlainText\nimport com.twitter.strato.fed._\nimport com.twitter.strato.thrift.ScroogeConv\nimport javax.inject.Inject\n\nclass ScoreColumn @Inject() (scoreStore: ScoreStore)\n    extends StratoFed.Column(\"recommendations/representation_scorer/score\")\n    with StratoFed.Fetch.Stitch {\n\n  override val policy: Policy = Common.rsxReadPolicy\n\n  override type Key = ScoreId\n  override type View = Unit\n  override type Value = ScoringResponse\n\n  override val keyConv: Conv[Key] = ScroogeConv.fromStruct[ScoreId]\n  override val viewConv: Conv[View] = Conv.ofType\n  override val valueConv: Conv[Value] = ScroogeConv.fromStruct[ScoringResponse]\n\n  override val contactInfo: ContactInfo = Info.contactInfo\n\n  override val metadata: OpMetadata = OpMetadata(\n    lifecycle = Some(Lifecycle.Production),\n    description = Some(PlainText(\n      \"The Uniform Scoring Endpoint in Representation Scorer for the Content-Recommender.\" +\n        \" TDD: http://go/representation-scorer-tdd Guideline: http://go/uniform-scoring-guideline\"))\n  )\n\n  override def fetch(key: Key, view: View): Stitch[Result[Value]] =\n    scoreStore\n      .uniformScoringStoreStitch(key)\n      .map(score => found(ScoringResponse(Some(score))))\n      .handle {\n        case stitch.NotFound => missing\n      }\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/columns/SimClustersRecentEngagementSimilarityColumn.scala",
    "content": "package com.twitter.representationscorer.columns\n\nimport com.twitter.representationscorer.common.TweetId\nimport com.twitter.representationscorer.common.UserId\nimport com.twitter.representationscorer.thriftscala.RecentEngagementSimilaritiesResponse\nimport com.twitter.representationscorer.twistlyfeatures.Scorer\nimport com.twitter.stitch\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.OpMetadata\nimport com.twitter.strato.config.ContactInfo\nimport com.twitter.strato.config.Policy\nimport com.twitter.strato.data.Conv\nimport com.twitter.strato.data.Description.PlainText\nimport com.twitter.strato.data.Lifecycle\nimport com.twitter.strato.fed._\nimport com.twitter.strato.thrift.ScroogeConv\nimport javax.inject.Inject\n\nclass SimClustersRecentEngagementSimilarityColumn @Inject() (scorer: Scorer)\n    extends StratoFed.Column(\n      \"recommendations/representation_scorer/simClustersRecentEngagementSimilarity\")\n    with StratoFed.Fetch.Stitch {\n\n  override val policy: Policy = Common.rsxReadPolicy\n\n  override type Key = (UserId, Seq[TweetId])\n  override type View = Unit\n  override type Value = RecentEngagementSimilaritiesResponse\n\n  override val keyConv: Conv[Key] = Conv.ofType[(Long, Seq[Long])]\n  override val viewConv: Conv[View] = Conv.ofType\n  override val valueConv: Conv[Value] =\n    ScroogeConv.fromStruct[RecentEngagementSimilaritiesResponse]\n\n  override val contactInfo: ContactInfo = Info.contactInfo\n\n  override val metadata: OpMetadata = OpMetadata(\n    lifecycle = Some(Lifecycle.Production),\n    description = Some(\n      PlainText(\n        \"User-Tweet scores based on the user's recent engagements for multiple tweets.\"\n      ))\n  )\n\n  override def fetch(key: Key, view: View): Stitch[Result[Value]] =\n    scorer\n      .get(key._1, key._2)\n      .map(results => found(RecentEngagementSimilaritiesResponse(results)))\n      .handle {\n        case stitch.NotFound => missing\n      }\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/columns/SimClustersRecentEngagementSimilarityUserTweetEdgeColumn.scala",
    "content": "package com.twitter.representationscorer.columns\n\nimport com.twitter.representationscorer.common.TweetId\nimport com.twitter.representationscorer.common.UserId\nimport com.twitter.representationscorer.thriftscala.SimClustersRecentEngagementSimilarities\nimport com.twitter.representationscorer.twistlyfeatures.Scorer\nimport com.twitter.stitch\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.catalog.OpMetadata\nimport com.twitter.strato.config.ContactInfo\nimport com.twitter.strato.config.Policy\nimport com.twitter.strato.data.Conv\nimport com.twitter.strato.data.Description.PlainText\nimport com.twitter.strato.data.Lifecycle\nimport com.twitter.strato.fed._\nimport com.twitter.strato.thrift.ScroogeConv\nimport javax.inject.Inject\n\nclass SimClustersRecentEngagementSimilarityUserTweetEdgeColumn @Inject() (scorer: Scorer)\n    extends StratoFed.Column(\n      \"recommendations/representation_scorer/simClustersRecentEngagementSimilarity.UserTweetEdge\")\n    with StratoFed.Fetch.Stitch {\n\n  override val policy: Policy = Common.rsxReadPolicy\n\n  override type Key = (UserId, TweetId)\n  override type View = Unit\n  override type Value = SimClustersRecentEngagementSimilarities\n\n  override val keyConv: Conv[Key] = Conv.ofType[(Long, Long)]\n  override val viewConv: Conv[View] = Conv.ofType\n  override val valueConv: Conv[Value] =\n    ScroogeConv.fromStruct[SimClustersRecentEngagementSimilarities]\n\n  override val contactInfo: ContactInfo = Info.contactInfo\n\n  override val metadata: OpMetadata = OpMetadata(\n    lifecycle = Some(Lifecycle.Production),\n    description = Some(\n      PlainText(\n        \"User-Tweet scores based on the user's recent engagements\"\n      ))\n  )\n\n  override def fetch(key: Key, view: View): Stitch[Result[Value]] =\n    scorer\n      .get(key._1, key._2)\n      .map(found(_))\n      .handle {\n        case stitch.NotFound => missing\n      }\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/common/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"decider/src/main/scala\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n    ],\n)\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/common/DeciderConstants.scala",
    "content": "package com.twitter.representationscorer\n\nobject DeciderConstants {\n  val enableSimClustersEmbeddingStoreTimeouts = \"enable_sim_clusters_embedding_store_timeouts\"\n  val simClustersEmbeddingStoreTimeoutValueMillis =\n    \"sim_clusters_embedding_store_timeout_value_millis\"\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/common/RepresentationScorerDecider.scala",
    "content": "package com.twitter.representationscorer.common\n\nimport com.twitter.decider.Decider\nimport com.twitter.decider.RandomRecipient\nimport com.twitter.decider.Recipient\nimport com.twitter.simclusters_v2.common.DeciderGateBuilderWithIdHashing\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\ncase class RepresentationScorerDecider @Inject() (decider: Decider) {\n\n  val deciderGateBuilder = new DeciderGateBuilderWithIdHashing(decider)\n\n  def isAvailable(feature: String, recipient: Option[Recipient]): Boolean = {\n    decider.isAvailable(feature, recipient)\n  }\n\n  /**\n   * When useRandomRecipient is set to false, the decider is either completely on or off.\n   * When useRandomRecipient is set to true, the decider is on for the specified % of traffic.\n   */\n  def isAvailable(feature: String, useRandomRecipient: Boolean = true): Boolean = {\n    if (useRandomRecipient) isAvailable(feature, Some(RandomRecipient))\n    else isAvailable(feature, None)\n  }\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/common/package.scala",
    "content": "package com.twitter.representationscorer\n\npackage object common {\n  type UserId = Long\n  type TweetId = Long\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/modules/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finagle/finagle-stats\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"representation-manager/client/src/main/scala/com/twitter/representation_manager\",\n        \"representation-manager/client/src/main/scala/com/twitter/representation_manager/config\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/migration\",\n        \"representation-scorer/server/src/main/scala/com/twitter/representationscorer/common\",\n        \"servo/util\",\n        \"src/scala/com/twitter/simclusters_v2/stores\",\n        \"src/scala/com/twitter/storehaus_internal/memcache\",\n        \"src/scala/com/twitter/storehaus_internal/util\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/modules/CacheModule.scala",
    "content": "package com.twitter.representationscorer.modules\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.memcached.Client\nimport javax.inject.Singleton\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.inject.TwitterModule\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.storehaus_internal.memcache.MemcacheStore\nimport com.twitter.storehaus_internal.util.ClientName\nimport com.twitter.storehaus_internal.util.ZkEndPoint\n\nobject CacheModule extends TwitterModule {\n\n  private val cacheDest = flag[String](\"cache_module.dest\", \"Path to memcache service\")\n  private val timeout = flag[Int](\"memcache.timeout\", \"Memcache client timeout\")\n  private val retries = flag[Int](\"memcache.retries\", \"Memcache timeout retries\")\n\n  @Singleton\n  @Provides\n  def providesCache(\n    serviceIdentifier: ServiceIdentifier,\n    stats: StatsReceiver\n  ): Client =\n    MemcacheStore.memcachedClient(\n      name = ClientName(\"memcache_representation_manager\"),\n      dest = ZkEndPoint(cacheDest()),\n      timeout = timeout().milliseconds,\n      retries = retries(),\n      statsReceiver = stats.scope(\"cache_client\"),\n      serviceIdentifier = serviceIdentifier\n    )\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/modules/EmbeddingStoreModule.scala",
    "content": "package com.twitter.representationscorer.modules\n\nimport com.google.inject.Provides\nimport com.twitter.decider.Decider\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.finagle.thrift.ClientId\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.relevance_platform.common.readablestore.ReadableStoreWithTimeout\nimport com.twitter.representation_manager.migration.LegacyRMS\nimport com.twitter.representationscorer.DeciderConstants\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.stores.SimClustersEmbeddingStore\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType._\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion._\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Timer\nimport javax.inject.Singleton\n\nobject EmbeddingStoreModule extends TwitterModule {\n  @Singleton\n  @Provides\n  def providesEmbeddingStore(\n    memCachedClient: MemcachedClient,\n    serviceIdentifier: ServiceIdentifier,\n    clientId: ClientId,\n    timer: Timer,\n    decider: Decider,\n    stats: StatsReceiver\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n    val cacheHashKeyPrefix: String = \"RMS\"\n    val embeddingStoreClient = new LegacyRMS(\n      serviceIdentifier,\n      memCachedClient,\n      stats,\n      decider,\n      clientId,\n      timer,\n      cacheHashKeyPrefix\n    )\n\n    val underlyingStores: Map[\n      (EmbeddingType, ModelVersion),\n      ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding]\n    ] = Map(\n      // Tweet Embeddings\n      (\n        LogFavBasedTweet,\n        Model20m145k2020) -> embeddingStoreClient.logFavBased20M145K2020TweetEmbeddingStore,\n      (\n        LogFavLongestL2EmbeddingTweet,\n        Model20m145k2020) -> embeddingStoreClient.logFavBasedLongestL2Tweet20M145K2020EmbeddingStore,\n      // InterestedIn Embeddings\n      (\n        LogFavBasedUserInterestedInFromAPE,\n        Model20m145k2020) -> embeddingStoreClient.LogFavBasedInterestedInFromAPE20M145K2020Store,\n      (\n        FavBasedUserInterestedIn,\n        Model20m145k2020) -> embeddingStoreClient.favBasedUserInterestedIn20M145K2020Store,\n      // Author Embeddings\n      (\n        FavBasedProducer,\n        Model20m145k2020) -> embeddingStoreClient.favBasedProducer20M145K2020EmbeddingStore,\n      // Entity Embeddings\n      (\n        LogFavBasedKgoApeTopic,\n        Model20m145k2020) -> embeddingStoreClient.logFavBasedApeEntity20M145K2020EmbeddingCachedStore,\n      (FavTfgTopic, Model20m145k2020) -> embeddingStoreClient.favBasedTfgTopicEmbedding2020Store,\n    )\n\n    val simClustersEmbeddingStore: ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n      val underlying: ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] =\n        SimClustersEmbeddingStore.buildWithDecider(\n          underlyingStores = underlyingStores,\n          decider = decider,\n          statsReceiver = stats.scope(\"simClusters_embeddings_store_deciderable\")\n        )\n\n      val underlyingWithTimeout: ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] =\n        new ReadableStoreWithTimeout(\n          rs = underlying,\n          decider = decider,\n          enableTimeoutDeciderKey = DeciderConstants.enableSimClustersEmbeddingStoreTimeouts,\n          timeoutValueKey = DeciderConstants.simClustersEmbeddingStoreTimeoutValueMillis,\n          timer = timer,\n          statsReceiver = stats.scope(\"simClusters_embedding_store_timeouts\")\n        )\n\n      ObservedReadableStore(\n        store = underlyingWithTimeout\n      )(stats.scope(\"simClusters_embeddings_store\"))\n    }\n    simClustersEmbeddingStore\n  }\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/modules/RMSConfigModule.scala",
    "content": "package com.twitter.representationscorer.modules\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.inject.TwitterModule\nimport com.twitter.representation_manager.config.ClientConfig\nimport com.twitter.representation_manager.config.EnabledInMemoryCacheParams\nimport com.twitter.representation_manager.config.InMemoryCacheParams\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType._\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion._\nimport javax.inject.Singleton\n\nobject RMSConfigModule extends TwitterModule {\n  def getCacheName(embedingType: EmbeddingType, modelVersion: ModelVersion): String =\n    s\"${embedingType.name}_${modelVersion.name}_in_mem_cache\"\n\n  @Singleton\n  @Provides\n  def providesRMSClientConfig: ClientConfig = {\n    val cacheParamsMap: Map[\n      (EmbeddingType, ModelVersion),\n      InMemoryCacheParams\n    ] = Map(\n      // Tweet Embeddings\n      (LogFavBasedTweet, Model20m145k2020) -> EnabledInMemoryCacheParams(\n        ttl = 10.minutes,\n        maxKeys = 1048575, // 800MB\n        cacheName = getCacheName(LogFavBasedTweet, Model20m145k2020)),\n      (LogFavLongestL2EmbeddingTweet, Model20m145k2020) -> EnabledInMemoryCacheParams(\n        ttl = 5.minute,\n        maxKeys = 1048575, // 800MB\n        cacheName = getCacheName(LogFavLongestL2EmbeddingTweet, Model20m145k2020)),\n      // User - KnownFor Embeddings\n      (FavBasedProducer, Model20m145k2020) -> EnabledInMemoryCacheParams(\n        ttl = 1.day,\n        maxKeys = 500000, // 400MB\n        cacheName = getCacheName(FavBasedProducer, Model20m145k2020)),\n      // User - InterestedIn Embeddings\n      (LogFavBasedUserInterestedInFromAPE, Model20m145k2020) -> EnabledInMemoryCacheParams(\n        ttl = 6.hours,\n        maxKeys = 262143,\n        cacheName = getCacheName(LogFavBasedUserInterestedInFromAPE, Model20m145k2020)),\n      (FavBasedUserInterestedIn, Model20m145k2020) -> EnabledInMemoryCacheParams(\n        ttl = 6.hours,\n        maxKeys = 262143,\n        cacheName = getCacheName(FavBasedUserInterestedIn, Model20m145k2020)),\n      // Topic Embeddings\n      (FavTfgTopic, Model20m145k2020) -> EnabledInMemoryCacheParams(\n        ttl = 12.hours,\n        maxKeys = 262143, // 200MB\n        cacheName = getCacheName(FavTfgTopic, Model20m145k2020)),\n      (LogFavBasedKgoApeTopic, Model20m145k2020) -> EnabledInMemoryCacheParams(\n        ttl = 6.hours,\n        maxKeys = 262143,\n        cacheName = getCacheName(LogFavBasedKgoApeTopic, Model20m145k2020)),\n    )\n\n    new ClientConfig(inMemCacheParamsOverrides = cacheParamsMap)\n  }\n\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/modules/TimerModule.scala",
    "content": "package com.twitter.representationscorer.modules\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.util.DefaultTimer\nimport com.twitter.inject.TwitterModule\nimport com.twitter.util.Timer\nimport javax.inject.Singleton\n\nobject TimerModule extends TwitterModule {\n  @Singleton\n  @Provides\n  def providesTimer: Timer = DefaultTimer\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/scorestore/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/util\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/injection\",\n        \"representation-manager/client/src/main/scala/com/twitter/representation_manager\",\n        \"representation-manager/client/src/main/scala/com/twitter/representation_manager/config\",\n        \"representation-scorer/server/src/main/scala/com/twitter/representationscorer/common\",\n        \"src/scala/com/twitter/simclusters_v2/score\",\n        \"src/scala/com/twitter/topic_recos/common\",\n        \"src/scala/com/twitter/topic_recos/stores\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"src/thrift/com/twitter/topic_recos:topic_recos-thrift-scala\",\n        \"stitch/stitch-storehaus\",\n    ],\n)\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/scorestore/ScoreStore.scala",
    "content": "package com.twitter.representationscorer.scorestore\n\nimport com.twitter.bijection.scrooge.BinaryScalaCodec\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hashing.KeyHasher\nimport com.twitter.hermit.store.common.ObservedCachedReadableStore\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.hermit.store.common.ObservedReadableStore\nimport com.twitter.relevance_platform.common.injection.LZ4Injection\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.score.ScoreFacadeStore\nimport com.twitter.simclusters_v2.score.SimClustersEmbeddingPairScoreStore\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType.FavTfgTopic\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType.LogFavBasedKgoApeTopic\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType.LogFavBasedTweet\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion.Model20m145kUpdated\nimport com.twitter.simclusters_v2.thriftscala.Score\nimport com.twitter.simclusters_v2.thriftscala.ScoreId\nimport com.twitter.simclusters_v2.thriftscala.ScoringAlgorithm\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.stitch.storehaus.StitchOfReadableStore\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport com.twitter.topic_recos.stores.CertoTweetTopicScoresStore\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton()\nclass ScoreStore @Inject() (\n  simClustersEmbeddingStore: ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding],\n  stratoClient: StratoClient,\n  representationScorerCacheClient: Client,\n  stats: StatsReceiver) {\n\n  private val keyHasher = KeyHasher.FNV1A_64\n  private val statsReceiver = stats.scope(\"score_store\")\n\n  /** ** Score Store *****/\n  private val simClustersEmbeddingCosineSimilarityScoreStore =\n    ObservedReadableStore(\n      SimClustersEmbeddingPairScoreStore\n        .buildCosineSimilarityStore(simClustersEmbeddingStore)\n        .toThriftStore\n    )(statsReceiver.scope(\"simClusters_embedding_cosine_similarity_score_store\"))\n\n  private val simClustersEmbeddingDotProductScoreStore =\n    ObservedReadableStore(\n      SimClustersEmbeddingPairScoreStore\n        .buildDotProductStore(simClustersEmbeddingStore)\n        .toThriftStore\n    )(statsReceiver.scope(\"simClusters_embedding_dot_product_score_store\"))\n\n  private val simClustersEmbeddingJaccardSimilarityScoreStore =\n    ObservedReadableStore(\n      SimClustersEmbeddingPairScoreStore\n        .buildJaccardSimilarityStore(simClustersEmbeddingStore)\n        .toThriftStore\n    )(statsReceiver.scope(\"simClusters_embedding_jaccard_similarity_score_store\"))\n\n  private val simClustersEmbeddingEuclideanDistanceScoreStore =\n    ObservedReadableStore(\n      SimClustersEmbeddingPairScoreStore\n        .buildEuclideanDistanceStore(simClustersEmbeddingStore)\n        .toThriftStore\n    )(statsReceiver.scope(\"simClusters_embedding_euclidean_distance_score_store\"))\n\n  private val simClustersEmbeddingManhattanDistanceScoreStore =\n    ObservedReadableStore(\n      SimClustersEmbeddingPairScoreStore\n        .buildManhattanDistanceStore(simClustersEmbeddingStore)\n        .toThriftStore\n    )(statsReceiver.scope(\"simClusters_embedding_manhattan_distance_score_store\"))\n\n  private val simClustersEmbeddingLogCosineSimilarityScoreStore =\n    ObservedReadableStore(\n      SimClustersEmbeddingPairScoreStore\n        .buildLogCosineSimilarityStore(simClustersEmbeddingStore)\n        .toThriftStore\n    )(statsReceiver.scope(\"simClusters_embedding_log_cosine_similarity_score_store\"))\n\n  private val simClustersEmbeddingExpScaledCosineSimilarityScoreStore =\n    ObservedReadableStore(\n      SimClustersEmbeddingPairScoreStore\n        .buildExpScaledCosineSimilarityStore(simClustersEmbeddingStore)\n        .toThriftStore\n    )(statsReceiver.scope(\"simClusters_embedding_exp_scaled_cosine_similarity_score_store\"))\n\n  // Use the default setting\n  private val topicTweetRankingScoreStore =\n    TopicTweetRankingScoreStore.buildTopicTweetRankingStore(\n      FavTfgTopic,\n      LogFavBasedKgoApeTopic,\n      LogFavBasedTweet,\n      Model20m145kUpdated,\n      consumerEmbeddingMultiplier = 1.0,\n      producerEmbeddingMultiplier = 1.0\n    )\n\n  private val topicTweetsCortexThresholdStore = TopicTweetsCosineSimilarityAggregateStore(\n    TopicTweetsCosineSimilarityAggregateStore.DefaultScoreKeys,\n    statsReceiver.scope(\"topic_tweets_cortex_threshold_store\")\n  )\n\n  val topicTweetCertoScoreStore: ObservedCachedReadableStore[ScoreId, Score] = {\n    val underlyingStore = ObservedReadableStore(\n      TopicTweetCertoScoreStore(CertoTweetTopicScoresStore.prodStore(stratoClient))\n    )(statsReceiver.scope(\"topic_tweet_certo_score_store\"))\n\n    val memcachedStore = ObservedMemcachedReadableStore\n      .fromCacheClient(\n        backingStore = underlyingStore,\n        cacheClient = representationScorerCacheClient,\n        ttl = 10.minutes\n      )(\n        valueInjection = LZ4Injection.compose(BinaryScalaCodec(Score)),\n        statsReceiver = statsReceiver.scope(\"topic_tweet_certo_store_memcache\"),\n        keyToString = { k: ScoreId =>\n          s\"certocs:${keyHasher.hashKey(k.toString.getBytes)}\"\n        }\n      )\n\n    ObservedCachedReadableStore.from[ScoreId, Score](\n      memcachedStore,\n      ttl = 5.minutes,\n      maxKeys = 1000000,\n      cacheName = \"topic_tweet_certo_store_cache\",\n      windowSize = 10000L\n    )(statsReceiver.scope(\"topic_tweet_certo_store_cache\"))\n  }\n\n  val uniformScoringStore: ReadableStore[ScoreId, Score] =\n    ScoreFacadeStore.buildWithMetrics(\n      readableStores = Map(\n        ScoringAlgorithm.PairEmbeddingCosineSimilarity ->\n          simClustersEmbeddingCosineSimilarityScoreStore,\n        ScoringAlgorithm.PairEmbeddingDotProduct ->\n          simClustersEmbeddingDotProductScoreStore,\n        ScoringAlgorithm.PairEmbeddingJaccardSimilarity ->\n          simClustersEmbeddingJaccardSimilarityScoreStore,\n        ScoringAlgorithm.PairEmbeddingEuclideanDistance ->\n          simClustersEmbeddingEuclideanDistanceScoreStore,\n        ScoringAlgorithm.PairEmbeddingManhattanDistance ->\n          simClustersEmbeddingManhattanDistanceScoreStore,\n        ScoringAlgorithm.PairEmbeddingLogCosineSimilarity ->\n          simClustersEmbeddingLogCosineSimilarityScoreStore,\n        ScoringAlgorithm.PairEmbeddingExpScaledCosineSimilarity ->\n          simClustersEmbeddingExpScaledCosineSimilarityScoreStore,\n        // Certo normalized cosine score between topic-tweet pairs\n        ScoringAlgorithm.CertoNormalizedCosineScore\n          -> topicTweetCertoScoreStore,\n        // Certo normalized dot-product score between topic-tweet pairs\n        ScoringAlgorithm.CertoNormalizedDotProductScore\n          -> topicTweetCertoScoreStore\n      ),\n      aggregatedStores = Map(\n        ScoringAlgorithm.WeightedSumTopicTweetRanking ->\n          topicTweetRankingScoreStore,\n        ScoringAlgorithm.CortexTopicTweetLabel ->\n          topicTweetsCortexThresholdStore,\n      ),\n      statsReceiver = stats\n    )\n\n  val uniformScoringStoreStitch: ScoreId => com.twitter.stitch.Stitch[Score] =\n    StitchOfReadableStore(uniformScoringStore)\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/scorestore/TopicTweetCertoScoreStore.scala",
    "content": "package com.twitter.representationscorer.scorestore\n\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.ScoreInternalId.GenericPairScoreId\nimport com.twitter.simclusters_v2.thriftscala.ScoringAlgorithm.CertoNormalizedDotProductScore\nimport com.twitter.simclusters_v2.thriftscala.ScoringAlgorithm.CertoNormalizedCosineScore\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.TopicId\nimport com.twitter.simclusters_v2.thriftscala.{Score => ThriftScore}\nimport com.twitter.simclusters_v2.thriftscala.{ScoreId => ThriftScoreId}\nimport com.twitter.storehaus.FutureOps\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.topic_recos.thriftscala.Scores\nimport com.twitter.topic_recos.thriftscala.TopicToScores\nimport com.twitter.util.Future\n\n/**\n * Score store to get Certo <topic, tweet> scores.\n * Currently, the store supports two Scoring Algorithms (i.e., two types of Certo scores):\n * 1. NormalizedDotProduct\n * 2. NormalizedCosine\n * Querying with corresponding scoring algorithms results in different Certo scores.\n */\ncase class TopicTweetCertoScoreStore(certoStratoStore: ReadableStore[TweetId, TopicToScores])\n    extends ReadableStore[ThriftScoreId, ThriftScore] {\n\n  override def multiGet[K1 <: ThriftScoreId](ks: Set[K1]): Map[K1, Future[Option[ThriftScore]]] = {\n    val tweetIds =\n      ks.map(_.internalId).collect {\n        case GenericPairScoreId(scoreId) =>\n          ((scoreId.id1, scoreId.id2): @annotation.nowarn(\n            \"msg=may not be exhaustive|max recursion depth\")) match {\n            case (InternalId.TweetId(tweetId), _) => tweetId\n            case (_, InternalId.TweetId(tweetId)) => tweetId\n          }\n      }\n\n    val result = for {\n      certoScores <- Future.collect(certoStratoStore.multiGet(tweetIds))\n    } yield {\n      ks.map { k =>\n        (k.algorithm, k.internalId) match {\n          case (CertoNormalizedDotProductScore, GenericPairScoreId(scoreId)) =>\n            (scoreId.id1, scoreId.id2) match {\n              case (InternalId.TweetId(tweetId), InternalId.TopicId(topicId)) =>\n                (\n                  k,\n                  extractScore(\n                    tweetId,\n                    topicId,\n                    certoScores,\n                    _.followerL2NormalizedDotProduct8HrHalfLife))\n              case (InternalId.TopicId(topicId), InternalId.TweetId(tweetId)) =>\n                (\n                  k,\n                  extractScore(\n                    tweetId,\n                    topicId,\n                    certoScores,\n                    _.followerL2NormalizedDotProduct8HrHalfLife))\n              case _ => (k, None)\n            }\n          case (CertoNormalizedCosineScore, GenericPairScoreId(scoreId)) =>\n            (scoreId.id1, scoreId.id2) match {\n              case (InternalId.TweetId(tweetId), InternalId.TopicId(topicId)) =>\n                (\n                  k,\n                  extractScore(\n                    tweetId,\n                    topicId,\n                    certoScores,\n                    _.followerL2NormalizedCosineSimilarity8HrHalfLife))\n              case (InternalId.TopicId(topicId), InternalId.TweetId(tweetId)) =>\n                (\n                  k,\n                  extractScore(\n                    tweetId,\n                    topicId,\n                    certoScores,\n                    _.followerL2NormalizedCosineSimilarity8HrHalfLife))\n              case _ => (k, None)\n            }\n          case _ => (k, None)\n        }\n      }.toMap\n    }\n    FutureOps.liftValues(ks, result)\n  }\n\n  /**\n   * Given tweetToCertoScores, extract certain Certo score between the given tweetId and topicId.\n   * The Certo score of interest is specified using scoreExtractor.\n   */\n  def extractScore(\n    tweetId: TweetId,\n    topicId: TopicId,\n    tweetToCertoScores: Map[TweetId, Option[TopicToScores]],\n    scoreExtractor: Scores => Double\n  ): Option[ThriftScore] = {\n    tweetToCertoScores.get(tweetId).flatMap {\n      case Some(topicToScores) =>\n        topicToScores.topicToScores.flatMap(_.get(topicId).map(scoreExtractor).map(ThriftScore(_)))\n      case _ => Some(ThriftScore(0.0))\n    }\n  }\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/scorestore/TopicTweetRankingScoreStore.scala",
    "content": "package com.twitter.representationscorer.scorestore\n\nimport com.twitter.simclusters_v2.score.WeightedSumAggregatedScoreStore\nimport com.twitter.simclusters_v2.score.WeightedSumAggregatedScoreStore.WeightedSumAggregatedScoreParameter\nimport com.twitter.simclusters_v2.thriftscala.{EmbeddingType, ModelVersion, ScoringAlgorithm}\n\nobject TopicTweetRankingScoreStore {\n  val producerEmbeddingScoreMultiplier = 1.0\n  val consumerEmbeddingScoreMultiplier = 1.0\n\n  /**\n   * Build the scoring store for TopicTweet Ranking based on Default Multipliers.\n   * If you want to compare the ranking between different multipliers, register a new\n   * ScoringAlgorithm and let the upstream uses different scoringAlgorithm by params.\n   */\n  def buildTopicTweetRankingStore(\n    consumerEmbeddingType: EmbeddingType,\n    producerEmbeddingType: EmbeddingType,\n    tweetEmbeddingType: EmbeddingType,\n    modelVersion: ModelVersion,\n    consumerEmbeddingMultiplier: Double = consumerEmbeddingScoreMultiplier,\n    producerEmbeddingMultiplier: Double = producerEmbeddingScoreMultiplier\n  ): WeightedSumAggregatedScoreStore = {\n    WeightedSumAggregatedScoreStore(\n      List(\n        WeightedSumAggregatedScoreParameter(\n          ScoringAlgorithm.PairEmbeddingCosineSimilarity,\n          consumerEmbeddingMultiplier,\n          WeightedSumAggregatedScoreStore.genericPairScoreIdToSimClustersEmbeddingPairScoreId(\n            consumerEmbeddingType,\n            tweetEmbeddingType,\n            modelVersion\n          )\n        ),\n        WeightedSumAggregatedScoreParameter(\n          ScoringAlgorithm.PairEmbeddingCosineSimilarity,\n          producerEmbeddingMultiplier,\n          WeightedSumAggregatedScoreStore.genericPairScoreIdToSimClustersEmbeddingPairScoreId(\n            producerEmbeddingType,\n            tweetEmbeddingType,\n            modelVersion\n          )\n        )\n      )\n    )\n  }\n\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/scorestore/TopicTweetsCosineSimilarityAggregateStore.scala",
    "content": "package com.twitter.representationscorer.scorestore\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.util.StatsUtil\nimport com.twitter.representationscorer.scorestore.TopicTweetsCosineSimilarityAggregateStore.ScoreKey\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.score.AggregatedScoreStore\nimport com.twitter.simclusters_v2.thriftscala.ScoreInternalId.GenericPairScoreId\nimport com.twitter.simclusters_v2.thriftscala.ScoringAlgorithm.CortexTopicTweetLabel\nimport com.twitter.simclusters_v2.thriftscala.{\n  EmbeddingType,\n  InternalId,\n  ModelVersion,\n  ScoreInternalId,\n  ScoringAlgorithm,\n  SimClustersEmbeddingId,\n  TopicId,\n  Score => ThriftScore,\n  ScoreId => ThriftScoreId,\n  SimClustersEmbeddingPairScoreId => ThriftSimClustersEmbeddingPairScoreId\n}\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.topic_recos.common.Configs.{DefaultModelVersion, MinCosineSimilarityScore}\nimport com.twitter.topic_recos.common._\nimport com.twitter.util.Future\n\n/**\n * Calculates the cosine similarity scores of arbitrary combinations of TopicEmbeddings and\n * TweetEmbeddings.\n * The class has 2 uses:\n * 1. For internal uses. TSP will call this store to fetch the raw scores for (topic, tweet) with\n * all available embedding types. We calculate all the scores here, so the caller can do filtering\n * & score caching on their side. This will make it possible to DDG different embedding scores.\n *\n * 2. For external calls from Cortex. We return true (or 1.0) for any given (topic, tweet) if their\n * cosine similarity passes the threshold for any of the embedding types.\n * The expected input type is\n * ScoreId(\n *  PairEmbeddingCosineSimilarity,\n *  GenericPairScoreId(TopicId, TweetId)\n * )\n */\ncase class TopicTweetsCosineSimilarityAggregateStore(\n  scoreKeys: Seq[ScoreKey],\n  statsReceiver: StatsReceiver)\n    extends AggregatedScoreStore {\n\n  def toCortexScore(scoresMap: Map[ScoreKey, Double]): Double = {\n    val passThreshold = scoresMap.exists {\n      case (_, score) => score >= MinCosineSimilarityScore\n    }\n    if (passThreshold) 1.0 else 0.0\n  }\n\n  /**\n   * To be called by Cortex through Unified Score API ONLY. Calculates all possible (topic, tweet),\n   * return 1.0 if any of the embedding scores passes the minimum threshold.\n   *\n   * Expect a GenericPairScoreId(PairEmbeddingCosineSimilarity, (TopicId, TweetId)) as input\n   */\n  override def get(k: ThriftScoreId): Future[Option[ThriftScore]] = {\n    StatsUtil.trackOptionStats(statsReceiver) {\n      (k.algorithm, k.internalId) match {\n        case (CortexTopicTweetLabel, GenericPairScoreId(genericPairScoreId)) =>\n          (genericPairScoreId.id1, genericPairScoreId.id2) match {\n            case (InternalId.TopicId(topicId), InternalId.TweetId(tweetId)) =>\n              TopicTweetsCosineSimilarityAggregateStore\n                .getRawScoresMap(topicId, tweetId, scoreKeys, scoreFacadeStore)\n                .map { scoresMap => Some(ThriftScore(toCortexScore(scoresMap))) }\n            case (InternalId.TweetId(tweetId), InternalId.TopicId(topicId)) =>\n              TopicTweetsCosineSimilarityAggregateStore\n                .getRawScoresMap(topicId, tweetId, scoreKeys, scoreFacadeStore)\n                .map { scoresMap => Some(ThriftScore(toCortexScore(scoresMap))) }\n            case _ =>\n              Future.None\n            // Do not accept other InternalId combinations\n          }\n        case _ =>\n          // Do not accept other Id types for now\n          Future.None\n      }\n    }\n  }\n}\n\nobject TopicTweetsCosineSimilarityAggregateStore {\n\n  val TopicEmbeddingTypes: Seq[EmbeddingType] =\n    Seq(\n      EmbeddingType.FavTfgTopic,\n      EmbeddingType.LogFavBasedKgoApeTopic\n    )\n\n  // Add the new embedding types if want to test the new Tweet embedding performance.\n  val TweetEmbeddingTypes: Seq[EmbeddingType] = Seq(EmbeddingType.LogFavBasedTweet)\n\n  val ModelVersions: Seq[ModelVersion] =\n    Seq(DefaultModelVersion)\n\n  val DefaultScoreKeys: Seq[ScoreKey] = {\n    for {\n      modelVersion <- ModelVersions\n      topicEmbeddingType <- TopicEmbeddingTypes\n      tweetEmbeddingType <- TweetEmbeddingTypes\n    } yield {\n      ScoreKey(\n        topicEmbeddingType = topicEmbeddingType,\n        tweetEmbeddingType = tweetEmbeddingType,\n        modelVersion = modelVersion\n      )\n    }\n  }\n  case class ScoreKey(\n    topicEmbeddingType: EmbeddingType,\n    tweetEmbeddingType: EmbeddingType,\n    modelVersion: ModelVersion)\n\n  def getRawScoresMap(\n    topicId: TopicId,\n    tweetId: TweetId,\n    scoreKeys: Seq[ScoreKey],\n    uniformScoringStore: ReadableStore[ThriftScoreId, ThriftScore]\n  ): Future[Map[ScoreKey, Double]] = {\n    val scoresMapFut = scoreKeys.map { key =>\n      val scoreInternalId = ScoreInternalId.SimClustersEmbeddingPairScoreId(\n        ThriftSimClustersEmbeddingPairScoreId(\n          buildTopicEmbedding(topicId, key.topicEmbeddingType, key.modelVersion),\n          SimClustersEmbeddingId(\n            key.tweetEmbeddingType,\n            key.modelVersion,\n            InternalId.TweetId(tweetId))\n        ))\n      val scoreFut = uniformScoringStore\n        .get(\n          ThriftScoreId(\n            algorithm = ScoringAlgorithm.PairEmbeddingCosineSimilarity, // Hard code as cosine sim\n            internalId = scoreInternalId\n          ))\n      key -> scoreFut\n    }.toMap\n\n    Future\n      .collect(scoresMapFut).map(_.collect {\n        case (key, Some(ThriftScore(score))) =>\n          (key, score)\n      })\n  }\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/twistlyfeatures/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/github/ben-manes/caffeine\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"representation-scorer/server/src/main/scala/com/twitter/representationscorer/common\",\n        \"representation-scorer/server/src/main/scala/com/twitter/representationscorer/scorestore\",\n        \"representation-scorer/server/src/main/thrift:thrift-scala\",\n        \"src/thrift/com/twitter/twistly:twistly-scala\",\n        \"stitch/stitch-core\",\n        \"stitch/stitch-core:cache\",\n        \"strato/config/columns/recommendations/twistly:twistly-strato-client\",\n        \"strato/config/columns/recommendations/user-signal-service:user-signal-service-strato-client\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"user-signal-service/thrift/src/main/thrift:thrift-scala\",\n        \"util/util-core\",\n    ],\n)\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/twistlyfeatures/Engagements.scala",
    "content": "package com.twitter.representationscorer.twistlyfeatures\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\n\ncase class Engagements(\n  favs7d: Seq[UserSignal] = Nil,\n  retweets7d: Seq[UserSignal] = Nil,\n  follows30d: Seq[UserSignal] = Nil,\n  shares7d: Seq[UserSignal] = Nil,\n  replies7d: Seq[UserSignal] = Nil,\n  originalTweets7d: Seq[UserSignal] = Nil,\n  videoPlaybacks7d: Seq[UserSignal] = Nil,\n  block30d: Seq[UserSignal] = Nil,\n  mute30d: Seq[UserSignal] = Nil,\n  report30d: Seq[UserSignal] = Nil,\n  dontlike30d: Seq[UserSignal] = Nil,\n  seeFewer30d: Seq[UserSignal] = Nil) {\n\n  import Engagements._\n\n  private val now = Time.now\n  private val oneDayAgo = (now - OneDaySpan).inMillis\n  private val sevenDaysAgo = (now - SevenDaysSpan).inMillis\n\n  // All ids from the signals grouped by type (tweetIds, userIds, etc)\n  val tweetIds: Seq[Long] =\n    (favs7d ++ retweets7d ++ shares7d\n      ++ replies7d ++ originalTweets7d ++ videoPlaybacks7d\n      ++ report30d ++ dontlike30d ++ seeFewer30d)\n      .map(_.targetId)\n  val authorIds: Seq[Long] = (follows30d ++ block30d ++ mute30d).map(_.targetId)\n\n  // Tweet signals\n  val dontlike7d: Seq[UserSignal] = dontlike30d.filter(_.timestamp > sevenDaysAgo)\n  val seeFewer7d: Seq[UserSignal] = seeFewer30d.filter(_.timestamp > sevenDaysAgo)\n\n  val favs1d: Seq[UserSignal] = favs7d.filter(_.timestamp > oneDayAgo)\n  val retweets1d: Seq[UserSignal] = retweets7d.filter(_.timestamp > oneDayAgo)\n  val shares1d: Seq[UserSignal] = shares7d.filter(_.timestamp > oneDayAgo)\n  val replies1d: Seq[UserSignal] = replies7d.filter(_.timestamp > oneDayAgo)\n  val originalTweets1d: Seq[UserSignal] = originalTweets7d.filter(_.timestamp > oneDayAgo)\n  val videoPlaybacks1d: Seq[UserSignal] = videoPlaybacks7d.filter(_.timestamp > oneDayAgo)\n  val dontlike1d: Seq[UserSignal] = dontlike7d.filter(_.timestamp > oneDayAgo)\n  val seeFewer1d: Seq[UserSignal] = seeFewer7d.filter(_.timestamp > oneDayAgo)\n\n  // User signals\n  val follows7d: Seq[UserSignal] = follows30d.filter(_.timestamp > sevenDaysAgo)\n  val block7d: Seq[UserSignal] = block30d.filter(_.timestamp > sevenDaysAgo)\n  val mute7d: Seq[UserSignal] = mute30d.filter(_.timestamp > sevenDaysAgo)\n  val report7d: Seq[UserSignal] = report30d.filter(_.timestamp > sevenDaysAgo)\n\n  val block1d: Seq[UserSignal] = block7d.filter(_.timestamp > oneDayAgo)\n  val mute1d: Seq[UserSignal] = mute7d.filter(_.timestamp > oneDayAgo)\n  val report1d: Seq[UserSignal] = report7d.filter(_.timestamp > oneDayAgo)\n}\n\nobject Engagements {\n  val OneDaySpan: Duration = 1.days\n  val SevenDaysSpan: Duration = 7.days\n  val ThirtyDaysSpan: Duration = 30.days\n}\n\ncase class UserSignal(targetId: Long, timestamp: Long)\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/twistlyfeatures/ScoreResult.scala",
    "content": "package com.twitter.representationscorer.twistlyfeatures\n\ncase class ScoreResult(id: Long, score: Option[Double])\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/twistlyfeatures/Scorer.scala",
    "content": "package com.twitter.representationscorer.twistlyfeatures\n\nimport com.twitter.finagle.stats.Counter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.representationscorer.common.TweetId\nimport com.twitter.representationscorer.common.UserId\nimport com.twitter.representationscorer.scorestore.ScoreStore\nimport com.twitter.representationscorer.thriftscala.SimClustersRecentEngagementSimilarities\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.ScoreId\nimport com.twitter.simclusters_v2.thriftscala.ScoreInternalId\nimport com.twitter.simclusters_v2.thriftscala.ScoringAlgorithm\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingPairScoreId\nimport com.twitter.stitch.Stitch\nimport javax.inject.Inject\n\nclass Scorer @Inject() (\n  fetchEngagementsFromUSS: Long => Stitch[Engagements],\n  scoreStore: ScoreStore,\n  stats: StatsReceiver) {\n\n  import Scorer._\n\n  private val scoreStats = stats.scope(\"score\")\n  private val scoreCalculationStats = scoreStats.scope(\"calculation\")\n  private val scoreResultStats = scoreStats.scope(\"result\")\n\n  private val scoresNonEmptyCounter = scoreResultStats.scope(\"all\").counter(\"nonEmpty\")\n  private val scoresNonZeroCounter = scoreResultStats.scope(\"all\").counter(\"nonZero\")\n\n  private val tweetScoreStats = scoreCalculationStats.scope(\"tweetScore\").stat(\"latency\")\n  private val userScoreStats = scoreCalculationStats.scope(\"userScore\").stat(\"latency\")\n\n  private val favNonZero = scoreResultStats.scope(\"favs\").counter(\"nonZero\")\n  private val favNonEmpty = scoreResultStats.scope(\"favs\").counter(\"nonEmpty\")\n\n  private val retweetsNonZero = scoreResultStats.scope(\"retweets\").counter(\"nonZero\")\n  private val retweetsNonEmpty = scoreResultStats.scope(\"retweets\").counter(\"nonEmpty\")\n\n  private val followsNonZero = scoreResultStats.scope(\"follows\").counter(\"nonZero\")\n  private val followsNonEmpty = scoreResultStats.scope(\"follows\").counter(\"nonEmpty\")\n\n  private val sharesNonZero = scoreResultStats.scope(\"shares\").counter(\"nonZero\")\n  private val sharesNonEmpty = scoreResultStats.scope(\"shares\").counter(\"nonEmpty\")\n\n  private val repliesNonZero = scoreResultStats.scope(\"replies\").counter(\"nonZero\")\n  private val repliesNonEmpty = scoreResultStats.scope(\"replies\").counter(\"nonEmpty\")\n\n  private val originalTweetsNonZero = scoreResultStats.scope(\"originalTweets\").counter(\"nonZero\")\n  private val originalTweetsNonEmpty = scoreResultStats.scope(\"originalTweets\").counter(\"nonEmpty\")\n\n  private val videoViewsNonZero = scoreResultStats.scope(\"videoViews\").counter(\"nonZero\")\n  private val videoViewsNonEmpty = scoreResultStats.scope(\"videoViews\").counter(\"nonEmpty\")\n\n  private val blockNonZero = scoreResultStats.scope(\"block\").counter(\"nonZero\")\n  private val blockNonEmpty = scoreResultStats.scope(\"block\").counter(\"nonEmpty\")\n\n  private val muteNonZero = scoreResultStats.scope(\"mute\").counter(\"nonZero\")\n  private val muteNonEmpty = scoreResultStats.scope(\"mute\").counter(\"nonEmpty\")\n\n  private val reportNonZero = scoreResultStats.scope(\"report\").counter(\"nonZero\")\n  private val reportNonEmpty = scoreResultStats.scope(\"report\").counter(\"nonEmpty\")\n\n  private val dontlikeNonZero = scoreResultStats.scope(\"dontlike\").counter(\"nonZero\")\n  private val dontlikeNonEmpty = scoreResultStats.scope(\"dontlike\").counter(\"nonEmpty\")\n\n  private val seeFewerNonZero = scoreResultStats.scope(\"seeFewer\").counter(\"nonZero\")\n  private val seeFewerNonEmpty = scoreResultStats.scope(\"seeFewer\").counter(\"nonEmpty\")\n\n  private def getTweetScores(\n    candidateTweetId: TweetId,\n    sourceTweetIds: Seq[TweetId]\n  ): Stitch[Seq[ScoreResult]] = {\n    val getScoresStitch = Stitch.traverse(sourceTweetIds) { sourceTweetId =>\n      scoreStore\n        .uniformScoringStoreStitch(getTweetScoreId(sourceTweetId, candidateTweetId))\n        .liftNotFoundToOption\n        .map(score => ScoreResult(sourceTweetId, score.map(_.score)))\n    }\n\n    Stitch.time(getScoresStitch).flatMap {\n      case (tryResult, duration) =>\n        tweetScoreStats.add(duration.inMillis)\n        Stitch.const(tryResult)\n    }\n  }\n\n  private def getUserScores(\n    tweetId: TweetId,\n    authorIds: Seq[UserId]\n  ): Stitch[Seq[ScoreResult]] = {\n    val getScoresStitch = Stitch.traverse(authorIds) { authorId =>\n      scoreStore\n        .uniformScoringStoreStitch(getAuthorScoreId(authorId, tweetId))\n        .liftNotFoundToOption\n        .map(score => ScoreResult(authorId, score.map(_.score)))\n    }\n\n    Stitch.time(getScoresStitch).flatMap {\n      case (tryResult, duration) =>\n        userScoreStats.add(duration.inMillis)\n        Stitch.const(tryResult)\n    }\n  }\n\n  /**\n   * Get the [[SimClustersRecentEngagementSimilarities]] result containing the similarity\n   * features for the given userId-TweetId.\n   */\n  def get(\n    userId: UserId,\n    tweetId: TweetId\n  ): Stitch[SimClustersRecentEngagementSimilarities] = {\n    get(userId, Seq(tweetId)).map(x => x.head)\n  }\n\n  /**\n   * Get a list of [[SimClustersRecentEngagementSimilarities]] results containing the similarity\n   * features for the given tweets of the user Id.\n   * Guaranteed to be the same number/order as requested.\n   */\n  def get(\n    userId: UserId,\n    tweetIds: Seq[TweetId]\n  ): Stitch[Seq[SimClustersRecentEngagementSimilarities]] = {\n    fetchEngagementsFromUSS(userId)\n      .flatMap(engagements => {\n        // For each tweet received in the request, compute the similarity scores between them\n        // and the user signals fetched from USS.\n        Stitch\n          .join(\n            Stitch.traverse(tweetIds)(id => getTweetScores(id, engagements.tweetIds)),\n            Stitch.traverse(tweetIds)(id => getUserScores(id, engagements.authorIds)),\n          )\n          .map {\n            case (tweetScoresSeq, userScoreSeq) =>\n              // All seq have = size because when scores don't exist, they are returned as Option\n              (tweetScoresSeq, userScoreSeq).zipped.map { (tweetScores, userScores) =>\n                computeSimilarityScoresPerTweet(\n                  engagements,\n                  tweetScores.groupBy(_.id),\n                  userScores.groupBy(_.id))\n              }\n          }\n      })\n  }\n\n  /**\n   *\n   * Computes the [[SimClustersRecentEngagementSimilarities]]\n   * using the given tweet-tweet and user-tweet scores in TweetScoresMap\n   * and the user signals in [[Engagements]].\n   */\n  private def computeSimilarityScoresPerTweet(\n    engagements: Engagements,\n    tweetScores: Map[TweetId, Seq[ScoreResult]],\n    authorScores: Map[UserId, Seq[ScoreResult]]\n  ): SimClustersRecentEngagementSimilarities = {\n    val favs7d = engagements.favs7d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val favs1d = engagements.favs1d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val retweets7d = engagements.retweets7d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val retweets1d = engagements.retweets1d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val follows30d = engagements.follows30d.view\n      .flatMap(s => authorScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val follows7d = engagements.follows7d.view\n      .flatMap(s => authorScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val shares7d = engagements.shares7d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val shares1d = engagements.shares1d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val replies7d = engagements.replies7d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val replies1d = engagements.replies1d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val originalTweets7d = engagements.originalTweets7d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val originalTweets1d = engagements.originalTweets1d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val videoViews7d = engagements.videoPlaybacks7d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val videoViews1d = engagements.videoPlaybacks1d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val block30d = engagements.block30d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val block7d = engagements.block7d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val block1d = engagements.block1d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val mute30d = engagements.mute30d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val mute7d = engagements.mute7d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val mute1d = engagements.mute1d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val report30d = engagements.report30d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val report7d = engagements.report7d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val report1d = engagements.report1d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val dontlike30d = engagements.dontlike30d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val dontlike7d = engagements.dontlike7d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val dontlike1d = engagements.dontlike1d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val seeFewer30d = engagements.seeFewer30d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val seeFewer7d = engagements.seeFewer7d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val seeFewer1d = engagements.seeFewer1d.view\n      .flatMap(s => tweetScores.get(s.targetId))\n      .flatten.flatMap(_.score)\n      .force\n\n    val result = SimClustersRecentEngagementSimilarities(\n      fav1dLast10Max = max(favs1d),\n      fav1dLast10Avg = avg(favs1d),\n      fav7dLast10Max = max(favs7d),\n      fav7dLast10Avg = avg(favs7d),\n      retweet1dLast10Max = max(retweets1d),\n      retweet1dLast10Avg = avg(retweets1d),\n      retweet7dLast10Max = max(retweets7d),\n      retweet7dLast10Avg = avg(retweets7d),\n      follow7dLast10Max = max(follows7d),\n      follow7dLast10Avg = avg(follows7d),\n      follow30dLast10Max = max(follows30d),\n      follow30dLast10Avg = avg(follows30d),\n      share1dLast10Max = max(shares1d),\n      share1dLast10Avg = avg(shares1d),\n      share7dLast10Max = max(shares7d),\n      share7dLast10Avg = avg(shares7d),\n      reply1dLast10Max = max(replies1d),\n      reply1dLast10Avg = avg(replies1d),\n      reply7dLast10Max = max(replies7d),\n      reply7dLast10Avg = avg(replies7d),\n      originalTweet1dLast10Max = max(originalTweets1d),\n      originalTweet1dLast10Avg = avg(originalTweets1d),\n      originalTweet7dLast10Max = max(originalTweets7d),\n      originalTweet7dLast10Avg = avg(originalTweets7d),\n      videoPlayback1dLast10Max = max(videoViews1d),\n      videoPlayback1dLast10Avg = avg(videoViews1d),\n      videoPlayback7dLast10Max = max(videoViews7d),\n      videoPlayback7dLast10Avg = avg(videoViews7d),\n      block1dLast10Max = max(block1d),\n      block1dLast10Avg = avg(block1d),\n      block7dLast10Max = max(block7d),\n      block7dLast10Avg = avg(block7d),\n      block30dLast10Max = max(block30d),\n      block30dLast10Avg = avg(block30d),\n      mute1dLast10Max = max(mute1d),\n      mute1dLast10Avg = avg(mute1d),\n      mute7dLast10Max = max(mute7d),\n      mute7dLast10Avg = avg(mute7d),\n      mute30dLast10Max = max(mute30d),\n      mute30dLast10Avg = avg(mute30d),\n      report1dLast10Max = max(report1d),\n      report1dLast10Avg = avg(report1d),\n      report7dLast10Max = max(report7d),\n      report7dLast10Avg = avg(report7d),\n      report30dLast10Max = max(report30d),\n      report30dLast10Avg = avg(report30d),\n      dontlike1dLast10Max = max(dontlike1d),\n      dontlike1dLast10Avg = avg(dontlike1d),\n      dontlike7dLast10Max = max(dontlike7d),\n      dontlike7dLast10Avg = avg(dontlike7d),\n      dontlike30dLast10Max = max(dontlike30d),\n      dontlike30dLast10Avg = avg(dontlike30d),\n      seeFewer1dLast10Max = max(seeFewer1d),\n      seeFewer1dLast10Avg = avg(seeFewer1d),\n      seeFewer7dLast10Max = max(seeFewer7d),\n      seeFewer7dLast10Avg = avg(seeFewer7d),\n      seeFewer30dLast10Max = max(seeFewer30d),\n      seeFewer30dLast10Avg = avg(seeFewer30d),\n    )\n    trackStats(result)\n    result\n  }\n\n  private def trackStats(result: SimClustersRecentEngagementSimilarities): Unit = {\n    val scores = Seq(\n      result.fav7dLast10Max,\n      result.retweet7dLast10Max,\n      result.follow30dLast10Max,\n      result.share1dLast10Max,\n      result.share7dLast10Max,\n      result.reply7dLast10Max,\n      result.originalTweet7dLast10Max,\n      result.videoPlayback7dLast10Max,\n      result.block30dLast10Max,\n      result.mute30dLast10Max,\n      result.report30dLast10Max,\n      result.dontlike30dLast10Max,\n      result.seeFewer30dLast10Max\n    )\n\n    val nonEmpty = scores.exists(_.isDefined)\n    val nonZero = scores.exists { case Some(score) if score > 0 => true; case _ => false }\n\n    if (nonEmpty) {\n      scoresNonEmptyCounter.incr()\n    }\n\n    if (nonZero) {\n      scoresNonZeroCounter.incr()\n    }\n\n    // We use the largest window of a given type of score,\n    // because the largest window is inclusive of smaller windows.\n    trackSignalStats(favNonEmpty, favNonZero, result.fav7dLast10Avg)\n    trackSignalStats(retweetsNonEmpty, retweetsNonZero, result.retweet7dLast10Avg)\n    trackSignalStats(followsNonEmpty, followsNonZero, result.follow30dLast10Avg)\n    trackSignalStats(sharesNonEmpty, sharesNonZero, result.share7dLast10Avg)\n    trackSignalStats(repliesNonEmpty, repliesNonZero, result.reply7dLast10Avg)\n    trackSignalStats(originalTweetsNonEmpty, originalTweetsNonZero, result.originalTweet7dLast10Avg)\n    trackSignalStats(videoViewsNonEmpty, videoViewsNonZero, result.videoPlayback7dLast10Avg)\n    trackSignalStats(blockNonEmpty, blockNonZero, result.block30dLast10Avg)\n    trackSignalStats(muteNonEmpty, muteNonZero, result.mute30dLast10Avg)\n    trackSignalStats(reportNonEmpty, reportNonZero, result.report30dLast10Avg)\n    trackSignalStats(dontlikeNonEmpty, dontlikeNonZero, result.dontlike30dLast10Avg)\n    trackSignalStats(seeFewerNonEmpty, seeFewerNonZero, result.seeFewer30dLast10Avg)\n  }\n\n  private def trackSignalStats(nonEmpty: Counter, nonZero: Counter, score: Option[Double]): Unit = {\n    if (score.nonEmpty) {\n      nonEmpty.incr()\n\n      if (score.get > 0)\n        nonZero.incr()\n    }\n  }\n}\n\nobject Scorer {\n  def avg(s: Traversable[Double]): Option[Double] =\n    if (s.isEmpty) None else Some(s.sum / s.size)\n  def max(s: Traversable[Double]): Option[Double] =\n    if (s.isEmpty) None else Some(s.foldLeft(0.0D) { (curr, _max) => math.max(curr, _max) })\n\n  private def getAuthorScoreId(\n    userId: UserId,\n    tweetId: TweetId\n  ) = {\n    ScoreId(\n      algorithm = ScoringAlgorithm.PairEmbeddingCosineSimilarity,\n      internalId = ScoreInternalId.SimClustersEmbeddingPairScoreId(\n        SimClustersEmbeddingPairScoreId(\n          SimClustersEmbeddingId(\n            internalId = InternalId.UserId(userId),\n            modelVersion = ModelVersion.Model20m145k2020,\n            embeddingType = EmbeddingType.FavBasedProducer\n          ),\n          SimClustersEmbeddingId(\n            internalId = InternalId.TweetId(tweetId),\n            modelVersion = ModelVersion.Model20m145k2020,\n            embeddingType = EmbeddingType.LogFavBasedTweet\n          )\n        ))\n    )\n  }\n\n  private def getTweetScoreId(\n    sourceTweetId: TweetId,\n    candidateTweetId: TweetId\n  ) = {\n    ScoreId(\n      algorithm = ScoringAlgorithm.PairEmbeddingCosineSimilarity,\n      internalId = ScoreInternalId.SimClustersEmbeddingPairScoreId(\n        SimClustersEmbeddingPairScoreId(\n          SimClustersEmbeddingId(\n            internalId = InternalId.TweetId(sourceTweetId),\n            modelVersion = ModelVersion.Model20m145k2020,\n            embeddingType = EmbeddingType.LogFavLongestL2EmbeddingTweet\n          ),\n          SimClustersEmbeddingId(\n            internalId = InternalId.TweetId(candidateTweetId),\n            modelVersion = ModelVersion.Model20m145k2020,\n            embeddingType = EmbeddingType.LogFavBasedTweet\n          )\n        ))\n    )\n  }\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/twistlyfeatures/UserSignalServiceRecentEngagementsClient.scala",
    "content": "package com.twitter.representationscorer.twistlyfeatures\n\nimport com.twitter.decider.SimpleRecipient\nimport com.twitter.finagle.stats.Stat\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.representationscorer.common._\nimport com.twitter.representationscorer.twistlyfeatures.Engagements._\nimport com.twitter.simclusters_v2.common.SimClustersEmbeddingId.LongInternalId\nimport com.twitter.stitch.Stitch\nimport com.twitter.strato.generated.client.recommendations.user_signal_service.SignalsClientColumn\nimport com.twitter.strato.generated.client.recommendations.user_signal_service.SignalsClientColumn.Value\nimport com.twitter.usersignalservice.thriftscala.BatchSignalRequest\nimport com.twitter.usersignalservice.thriftscala.SignalRequest\nimport com.twitter.usersignalservice.thriftscala.SignalType\nimport com.twitter.util.Time\nimport scala.collection.mutable.ArrayBuffer\nimport com.twitter.usersignalservice.thriftscala.ClientIdentifier\n\nclass UserSignalServiceRecentEngagementsClient(\n  stratoClient: SignalsClientColumn,\n  decider: RepresentationScorerDecider,\n  stats: StatsReceiver) {\n\n  import UserSignalServiceRecentEngagementsClient._\n\n  private val signalStats = stats.scope(\"user-signal-service\", \"signal\")\n  private val signalTypeStats: Map[SignalType, Stat] =\n    SignalType.list.map(s => (s, signalStats.scope(s.name).stat(\"size\"))).toMap\n\n  def get(userId: UserId): Stitch[Engagements] = {\n    val request = buildRequest(userId)\n    stratoClient.fetcher.fetch(request).map(_.v).lowerFromOption().map { response =>\n      val now = Time.now\n      val sevenDaysAgo = now - SevenDaysSpan\n      val thirtyDaysAgo = now - ThirtyDaysSpan\n\n      Engagements(\n        favs7d = getUserSignals(response, SignalType.TweetFavorite, sevenDaysAgo),\n        retweets7d = getUserSignals(response, SignalType.Retweet, sevenDaysAgo),\n        follows30d = getUserSignals(response, SignalType.AccountFollowWithDelay, thirtyDaysAgo),\n        shares7d = getUserSignals(response, SignalType.TweetShareV1, sevenDaysAgo),\n        replies7d = getUserSignals(response, SignalType.Reply, sevenDaysAgo),\n        originalTweets7d = getUserSignals(response, SignalType.OriginalTweet, sevenDaysAgo),\n        videoPlaybacks7d =\n          getUserSignals(response, SignalType.VideoView90dPlayback50V1, sevenDaysAgo),\n        block30d = getUserSignals(response, SignalType.AccountBlock, thirtyDaysAgo),\n        mute30d = getUserSignals(response, SignalType.AccountMute, thirtyDaysAgo),\n        report30d = getUserSignals(response, SignalType.TweetReport, thirtyDaysAgo),\n        dontlike30d = getUserSignals(response, SignalType.TweetDontLike, thirtyDaysAgo),\n        seeFewer30d = getUserSignals(response, SignalType.TweetSeeFewer, thirtyDaysAgo),\n      )\n    }\n  }\n\n  private def getUserSignals(\n    response: Value,\n    signalType: SignalType,\n    earliestValidTimestamp: Time\n  ): Seq[UserSignal] = {\n    val signals = response.signalResponse\n      .getOrElse(signalType, Seq.empty)\n      .view\n      .filter(_.timestamp > earliestValidTimestamp.inMillis)\n      .map(s => s.targetInternalId.collect { case LongInternalId(id) => (id, s.timestamp) })\n      .collect { case Some((id, engagedAt)) => UserSignal(id, engagedAt) }\n      .take(EngagementsToScore)\n      .force\n\n    signalTypeStats(signalType).add(signals.size)\n    signals\n  }\n\n  private def buildRequest(userId: Long) = {\n    val recipient = Some(SimpleRecipient(userId))\n\n    // Signals RSX always fetches\n    val requestSignals = ArrayBuffer(\n      SignalRequestFav,\n      SignalRequestRetweet,\n      SignalRequestFollow\n    )\n\n    // Signals under experimentation. We use individual deciders to disable them if necessary.\n    // If experiments are successful, they will become permanent.\n    if (decider.isAvailable(FetchSignalShareDeciderKey, recipient))\n      requestSignals.append(SignalRequestShare)\n\n    if (decider.isAvailable(FetchSignalReplyDeciderKey, recipient))\n      requestSignals.append(SignalRequestReply)\n\n    if (decider.isAvailable(FetchSignalOriginalTweetDeciderKey, recipient))\n      requestSignals.append(SignalRequestOriginalTweet)\n\n    if (decider.isAvailable(FetchSignalVideoPlaybackDeciderKey, recipient))\n      requestSignals.append(SignalRequestVideoPlayback)\n\n    if (decider.isAvailable(FetchSignalBlockDeciderKey, recipient))\n      requestSignals.append(SignalRequestBlock)\n\n    if (decider.isAvailable(FetchSignalMuteDeciderKey, recipient))\n      requestSignals.append(SignalRequestMute)\n\n    if (decider.isAvailable(FetchSignalReportDeciderKey, recipient))\n      requestSignals.append(SignalRequestReport)\n\n    if (decider.isAvailable(FetchSignalDontlikeDeciderKey, recipient))\n      requestSignals.append(SignalRequestDontlike)\n\n    if (decider.isAvailable(FetchSignalSeeFewerDeciderKey, recipient))\n      requestSignals.append(SignalRequestSeeFewer)\n\n    BatchSignalRequest(userId, requestSignals, Some(ClientIdentifier.RepresentationScorerHome))\n  }\n}\n\nobject UserSignalServiceRecentEngagementsClient {\n  val FetchSignalShareDeciderKey = \"representation_scorer_fetch_signal_share\"\n  val FetchSignalReplyDeciderKey = \"representation_scorer_fetch_signal_reply\"\n  val FetchSignalOriginalTweetDeciderKey = \"representation_scorer_fetch_signal_original_tweet\"\n  val FetchSignalVideoPlaybackDeciderKey = \"representation_scorer_fetch_signal_video_playback\"\n  val FetchSignalBlockDeciderKey = \"representation_scorer_fetch_signal_block\"\n  val FetchSignalMuteDeciderKey = \"representation_scorer_fetch_signal_mute\"\n  val FetchSignalReportDeciderKey = \"representation_scorer_fetch_signal_report\"\n  val FetchSignalDontlikeDeciderKey = \"representation_scorer_fetch_signal_dont_like\"\n  val FetchSignalSeeFewerDeciderKey = \"representation_scorer_fetch_signal_see_fewer\"\n\n  val EngagementsToScore = 10\n  private val engagementsToScoreOpt: Option[Long] = Some(EngagementsToScore)\n\n  val SignalRequestFav: SignalRequest =\n    SignalRequest(engagementsToScoreOpt, SignalType.TweetFavorite)\n  val SignalRequestRetweet: SignalRequest = SignalRequest(engagementsToScoreOpt, SignalType.Retweet)\n  val SignalRequestFollow: SignalRequest =\n    SignalRequest(engagementsToScoreOpt, SignalType.AccountFollowWithDelay)\n  // New experimental signals\n  val SignalRequestShare: SignalRequest =\n    SignalRequest(engagementsToScoreOpt, SignalType.TweetShareV1)\n  val SignalRequestReply: SignalRequest = SignalRequest(engagementsToScoreOpt, SignalType.Reply)\n  val SignalRequestOriginalTweet: SignalRequest =\n    SignalRequest(engagementsToScoreOpt, SignalType.OriginalTweet)\n  val SignalRequestVideoPlayback: SignalRequest =\n    SignalRequest(engagementsToScoreOpt, SignalType.VideoView90dPlayback50V1)\n\n  // Negative signals\n  val SignalRequestBlock: SignalRequest =\n    SignalRequest(engagementsToScoreOpt, SignalType.AccountBlock)\n  val SignalRequestMute: SignalRequest =\n    SignalRequest(engagementsToScoreOpt, SignalType.AccountMute)\n  val SignalRequestReport: SignalRequest =\n    SignalRequest(engagementsToScoreOpt, SignalType.TweetReport)\n  val SignalRequestDontlike: SignalRequest =\n    SignalRequest(engagementsToScoreOpt, SignalType.TweetDontLike)\n  val SignalRequestSeeFewer: SignalRequest =\n    SignalRequest(engagementsToScoreOpt, SignalType.TweetSeeFewer)\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/scala/com/twitter/representationscorer/twistlyfeatures/UserSignalServiceRecentEngagementsClientModule.scala",
    "content": "package com.twitter.representationscorer.twistlyfeatures\n\nimport com.github.benmanes.caffeine.cache.Caffeine\nimport com.twitter.stitch.cache.EvictingCache\nimport com.google.inject.Provides\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.representationscorer.common.RepresentationScorerDecider\nimport com.twitter.stitch.Stitch\nimport com.twitter.stitch.cache.ConcurrentMapCache\nimport com.twitter.stitch.cache.MemoizeQuery\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.generated.client.recommendations.user_signal_service.SignalsClientColumn\nimport java.util.concurrent.ConcurrentMap\nimport java.util.concurrent.TimeUnit\nimport javax.inject.Singleton\n\nobject UserSignalServiceRecentEngagementsClientModule extends TwitterModule {\n\n  @Singleton\n  @Provides\n  def provide(\n    client: Client,\n    decider: RepresentationScorerDecider,\n    statsReceiver: StatsReceiver\n  ): Long => Stitch[Engagements] = {\n    val stratoClient = new SignalsClientColumn(client)\n\n    /*\n    This cache holds a users recent engagements for a short period of time, such that batched requests\n    for multiple (userid, tweetid) pairs don't all need to fetch them.\n\n    [1] Caffeine cache keys/values must be objects, so we cannot use the `Long` primitive directly.\n    The boxed java.lang.Long works as a key, since it is an object. In most situations the compiler\n    can see where auto(un)boxing can occur. However, here we seem to need some wrapper functions\n    with explicit types to allow the boxing to happen.\n     */\n    val mapCache: ConcurrentMap[java.lang.Long, Stitch[Engagements]] =\n      Caffeine\n        .newBuilder()\n        .expireAfterWrite(5, TimeUnit.SECONDS)\n        .maximumSize(\n          1000 // We estimate 5M unique users in a 5m period - with 2k RSX instances, assume that one will see < 1k in a 5s period\n        )\n        .build[java.lang.Long, Stitch[Engagements]]\n        .asMap\n\n    statsReceiver.provideGauge(\"ussRecentEngagementsClient\", \"cache_size\") { mapCache.size.toFloat }\n\n    val engagementsClient =\n      new UserSignalServiceRecentEngagementsClient(stratoClient, decider, statsReceiver)\n\n    val f = (l: java.lang.Long) => engagementsClient.get(l) // See note [1] above\n    val cachedCall = MemoizeQuery(f, EvictingCache.lazily(new ConcurrentMapCache(mapCache)))\n    (l: Long) => cachedCall(l) // see note [1] above\n  }\n}\n"
  },
  {
    "path": "representation-scorer/server/src/main/thrift/BUILD",
    "content": "create_thrift_libraries(\n    base_name = \"thrift\",\n    sources = [\n        \"com/twitter/representationscorer/service.thrift\",\n    ],\n    platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n    ],\n    dependency_roots = [\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift\",\n    ],\n    generate_languages = [\n        \"java\",\n        \"scala\",\n        \"strato\",\n    ],\n    provides_java_name = \"representationscorer-service-thrift-java\",\n    provides_scala_name = \"representationscorer-service-thrift-scala\",\n)\n"
  },
  {
    "path": "representation-scorer/server/src/main/thrift/com/twitter/representationscorer/service.thrift",
    "content": "namespace java com.twitter.representationscorer.thriftjava\n#@namespace scala com.twitter.representationscorer.thriftscala\n#@namespace strato com.twitter.representationscorer\n\ninclude \"com/twitter/simclusters_v2/identifier.thrift\"\ninclude \"com/twitter/simclusters_v2/online_store.thrift\"\ninclude \"com/twitter/simclusters_v2/score.thrift\"\n\nstruct SimClustersRecentEngagementSimilarities {\n  // All scores computed using cosine similarity\n  // 1 - 1000 Positive Signals\n  1: optional double fav1dLast10Max // max score from last 10 faves in the last 1 day\n  2: optional double fav1dLast10Avg // avg score from last 10 faves in the last 1 day\n  3: optional double fav7dLast10Max // max score from last 10 faves in the last 7 days\n  4: optional double fav7dLast10Avg // avg score from last 10 faves in the last 7 days\n  5: optional double retweet1dLast10Max // max score from last 10 retweets in the last 1 days\n  6: optional double retweet1dLast10Avg // avg score from last 10 retweets in the last 1 days\n  7: optional double retweet7dLast10Max // max score from last 10 retweets in the last 7 days\n  8: optional double retweet7dLast10Avg // avg score from last 10 retweets in the last 7 days\n  9: optional double follow7dLast10Max // max score from the last 10 follows in the last 7 days\n  10: optional double follow7dLast10Avg // avg score from the last 10 follows in the last 7 days\n  11: optional double follow30dLast10Max // max score from the last 10 follows in the last 30 days\n  12: optional double follow30dLast10Avg // avg score from the last 10 follows in the last 30 days\n  13: optional double share1dLast10Max // max score from last 10 shares in the last 1 day\n  14: optional double share1dLast10Avg // avg score from last 10 shares in the last 1 day\n  15: optional double share7dLast10Max // max score from last 10 shares in the last 7 days\n  16: optional double share7dLast10Avg // avg score from last 10 shares in the last 7 days\n  17: optional double reply1dLast10Max // max score from last 10 replies in the last 1 day\n  18: optional double reply1dLast10Avg // avg score from last 10 replies in the last 1 day\n  19: optional double reply7dLast10Max // max score from last 10 replies in the last 7 days\n  20: optional double reply7dLast10Avg // avg score from last 10 replies in the last 7 days\n  21: optional double originalTweet1dLast10Max // max score from last 10 original tweets in the last 1 day\n  22: optional double originalTweet1dLast10Avg // avg score from last 10 original tweets in the last 1 day\n  23: optional double originalTweet7dLast10Max // max score from last 10 original tweets in the last 7 days\n  24: optional double originalTweet7dLast10Avg // avg score from last 10 original tweets in the last 7 days\n  25: optional double videoPlayback1dLast10Max // max score from last 10 video playback50 in the last 1 day\n  26: optional double videoPlayback1dLast10Avg // avg score from last 10 video playback50 in the last 1 day\n  27: optional double videoPlayback7dLast10Max // max score from last 10 video playback50 in the last 7 days\n  28: optional double videoPlayback7dLast10Avg // avg score from last 10 video playback50 in the last 7 days\n\n  // 1001 - 2000 Implicit Signals\n\n  // 2001 - 3000 Negative Signals\n  // Block Series\n  2001: optional double block1dLast10Avg\n  2002: optional double block1dLast10Max\n  2003: optional double block7dLast10Avg\n  2004: optional double block7dLast10Max\n  2005: optional double block30dLast10Avg\n  2006: optional double block30dLast10Max\n  // Mute Series\n  2101: optional double mute1dLast10Avg\n  2102: optional double mute1dLast10Max\n  2103: optional double mute7dLast10Avg\n  2104: optional double mute7dLast10Max\n  2105: optional double mute30dLast10Avg\n  2106: optional double mute30dLast10Max\n  // Report Series\n  2201: optional double report1dLast10Avg\n  2202: optional double report1dLast10Max\n  2203: optional double report7dLast10Avg\n  2204: optional double report7dLast10Max\n  2205: optional double report30dLast10Avg\n  2206: optional double report30dLast10Max\n  // Dontlike\n  2301: optional double dontlike1dLast10Avg\n  2302: optional double dontlike1dLast10Max\n  2303: optional double dontlike7dLast10Avg\n  2304: optional double dontlike7dLast10Max\n  2305: optional double dontlike30dLast10Avg\n  2306: optional double dontlike30dLast10Max\n  // SeeFewer\n  2401: optional double seeFewer1dLast10Avg\n  2402: optional double seeFewer1dLast10Max\n  2403: optional double seeFewer7dLast10Avg\n  2404: optional double seeFewer7dLast10Max\n  2405: optional double seeFewer30dLast10Avg\n  2406: optional double seeFewer30dLast10Max\n}(persisted='true', hasPersonalData = 'true')\n\n/*\n * List score API\n */\nstruct ListScoreId {\n  1: required score.ScoringAlgorithm algorithm\n  2: required online_store.ModelVersion modelVersion\n  3: required identifier.EmbeddingType targetEmbeddingType\n  4: required identifier.InternalId targetId\n  5: required identifier.EmbeddingType candidateEmbeddingType\n  6: required list<identifier.InternalId> candidateIds\n}(hasPersonalData = 'true')\n\nstruct ScoreResult {\n  // This api does not communicate why a score is missing. For example, it may be unavailable\n  // because the referenced entities do not exist (e.g. the embedding was not found) or because\n  // timeouts prevented us from calculating it.\n  1: optional double score\n}\n\nstruct ListScoreResponse {\n  1: required list<ScoreResult> scores // Guaranteed to be the same number/order as requested\n}\n\nstruct RecentEngagementSimilaritiesResponse {\n  1: required list<SimClustersRecentEngagementSimilarities> results // Guaranteed to be the same number/order as requested\n}\n"
  },
  {
    "path": "science/search/ingester/config/README.md",
    "content": "## Ingester Configs\nThis directory contains pipeline configurations for the tweet ingesters (realtime, protected and realtime_cg) and the user-updates ingester. The pipeline configurations define an ordered sequence of stages that the tweet or user update goes through before reaching Earlybird. Source code for the various stages referenced in the configs can be found at src/java/com/twitter/search/ingester/pipeline/twitter."
  },
  {
    "path": "science/search/ingester/config/pipeline-indexer.userupdates.xml",
    "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n  \n\n<!--\n     This indexer reads UserModification from user_modification Kafka topic, converts the\n     data into AntisocialUserUpdate by querying Gizmoduck and then writes the data to the\n     the search_user_updates Kafka topic.\n-->\n<pipeline>\n  <property\n      propName=\"validator\"\n      className=\"org.apache.commons.pipeline.validation.SimplePipelineValidator\"/>\n  <listener className=\"org.apache.commons.pipeline.listener.ObjectProcessedEventCounter\"/>\n  <driverFactory\n      className=\"org.apache.commons.pipeline.driver.DedicatedThreadStageDriverFactory\"\n      id=\"pipeline\">\n\n    <!-- This queue is a factor of batchSize larger than inner queues because it is unbatched -->\n    <property\n        propName=\"queueFactory\"\n        className=\"org.apache.commons.pipeline.util.BlockingQueueFactory$ArrayBlockingQueueFactory\"\n        capacity=\"500\"\n        fair=\"false\"/>\n  </driverFactory>\n\n  <stage\n    className=\"com.twitter.search.ingester.pipeline.twitter.userupdates.UserUpdatesPipelineStage\"\n    environment=\"prod\"\n    driverFactoryId=\"pipeline\"/>\n</pipeline>\n"
  },
  {
    "path": "science/search/ingester/config/pipeline-ingester.protected.xml",
    "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n\n<!-- Ingesters process tweet create events from TweetyPie and write them to a queue for Earlybird\nto index. -->\n<pipeline>\n  <property\n      propName=\"validator\"\n      className=\"org.apache.commons.pipeline.validation.SimplePipelineValidator\"/>\n  <listener\n      className=\"org.apache.commons.pipeline.listener.ObjectProcessedEventCounter\"/>\n  <driverFactory\n      className=\"org.apache.commons.pipeline.driver.DedicatedThreadStageDriverFactory\"\n      id=\"kafka\">\n\n    <property\n        propName=\"queueFactory\"\n        className=\"org.apache.commons.pipeline.util.BlockingQueueFactory$ArrayBlockingQueueFactory\"\n        capacity=\"1000\"\n        fair=\"false\"/>\n  </driverFactory>\n\n  <!-- Read tweets from the thrift kafka queue. The reader loops forever. -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.kafka.KafkaRawRecordConsumerStage\"\n      kafkaClusterPath=\"\"\n      kafkaClientId=\"\"\n      kafkaTopicName=\"\"\n      kafkaConsumerGroupId=\"\"\n      maxPollRecords=\"1\"\n      pollTimeoutMs=\"1000\"\n      partitioned=\"false\"\n      deciderKey=\"\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Deserialize the bytes into TweetData -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.TweetEventDeserializerStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Filter to only have the safetytype for this cluster -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.FilterEventsBySafetyTypeStage\"\n      tweetCreateLatencyLogThresholdMillis=\"5000\"\n      safetyType=\"PROTECTED\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Parse to TwitterMessage -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ThriftTweetParserStage\"\n      tweetDeleteEventBranchNames=\"kafka_update_events_delete\"\n      driverFactoryId=\"kafka\"/>\n\n  <branch>\n    <pipeline key=\"kafka_update_events_delete\">\n      <property\n          propName=\"validator\"\n          className=\"org.apache.commons.pipeline.validation.SimplePipelineValidator\"/>\n      <listener\n          className=\"org.apache.commons.pipeline.listener.ObjectProcessedEventCounter\"/>\n      <driverFactory\n          className=\"org.apache.commons.pipeline.driver.DedicatedThreadStageDriverFactory\"\n          id=\"kafka_update_events_delete\">\n\n        <!-- we are willing to queue more deletes than other stages,\n             to make sure we don't slow down the incoming tweets -->\n        <property\n            propName=\"queueFactory\"\n            className=\"org.apache.commons.pipeline.util.BlockingQueueFactory$ArrayBlockingQueueFactory\"\n            capacity=\"1000\"\n            fair=\"false\"/>\n      </driverFactory>\n\n      <stage\n          className=\"com.twitter.search.ingester.pipeline.twitter.kafka.DeleteUpdateEventsKafkaProducerStage\"\n          kafkaClusterPath=\"\"\n          kafkaClientId=\"\"\n          kafkaTopicName=\"\"\n          driverFactoryId=\"kafka_update_events_delete\"/>\n    </pipeline>\n  </branch>\n\n\n  <!-- filters out messages that are not formatted correctly -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.FilterTwitterMessageStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- retrieves space ids from space urls if the tweet has space urls -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.RetrieveSpaceIdsStage\"\n      driverFactoryId=\"kafka\"/>\n\n\n  <!-- looks up user reputation scores for each message -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.LookupUserPropertiesBatchedStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- extract text features of the message -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.TextFeatureExtractionWorkersStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- compute text quality score of the message -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.TextQualityEvaluationWorkerStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Extract lat/lon pairs from the text, and geocode them -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.SingleTweetExtractAndGeocodeLatLonStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- adds coded locations -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.PopulateCodedLocationsBatchedStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Parse the TwitterMessages into ThriftStatuses -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ConvertMessageToThriftStage\"\n      thriftVersionedEventsBranchName=\"kafka_base_tweets\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Branch for tweets -->\n  <branch>\n    <pipeline key=\"kafka_base_tweets\">\n      <property\n          propName=\"validator\"\n          className=\"org.apache.commons.pipeline.validation.SimplePipelineValidator\"/>\n      <listener\n          className=\"org.apache.commons.pipeline.listener.ObjectProcessedEventCounter\"/>\n      <driverFactory\n          className=\"org.apache.commons.pipeline.driver.DedicatedThreadStageDriverFactory\"\n          id=\"kafka_base_tweets\">\n\n        <property\n            propName=\"queueFactory\"\n            className=\"org.apache.commons.pipeline.util.BlockingQueueFactory$ArrayBlockingQueueFactory\"\n            capacity=\"1000\"\n            fair=\"false\"/>\n      </driverFactory>\n\n      <stage\n          className=\"com.twitter.search.ingester.pipeline.twitter.kafka.TweetThriftVersionedEventsKafkaProducerStage\"\n          kafkaClusterPath=\"\"\n          kafkaClientId=\"search_ingester_indexing_events\"\n          kafkaTopicName=\"search_ingester_indexing_events_protected_prod\"\n          driverFactoryId=\"kafka_base_tweets\"/>\n    </pipeline>\n  </branch>\n\n  <!-- Resolve compressed URL via Pink -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ResolveCompressedUrlsBatchedStage\"\n      pinkClientId=\"INGESTER\"\n      batchedStageBatchSize=\"10\"\n      tweetMaxAgeToResolve=\"10000\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Retrieve card information -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.RetrieveCardBatchedStage\"\n      tweetypieClientId=\"ingester.prod\"\n      filterProtected=\"false\"\n      internalBatchSize=\"50\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Retrieve named entities -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.RetrieveNamedEntitiesSingleTweetStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- retrieves space admins and title for a tweet if the tweet has space urls -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.RetrieveSpaceAdminsAndTitleStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- extract text features of the message -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.TextUrlsFeatureExtractionStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Compute the tweet signature -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ComputeTweetSignatureStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Parse the TwitterMessages into ThriftStatuses -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ConvertDelayedMessageToThriftStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.kafka.TweetThriftVersionedEventsKafkaProducerStage\"\n      kafkaClusterPath=\"\"\n      stageName=\"UpdateEvents\"\n      kafkaClientId=\"\"\n      kafkaTopicName=\"\"\n      driverFactoryId=\"kafka\"/>\n</pipeline>\n"
  },
  {
    "path": "science/search/ingester/config/pipeline-ingester.realtime.xml",
    "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n\n<!-- Ingesters process tweet create events from TweetyPie and write them to a queue for Earlybird\nto index. -->\n<pipeline>\n  <property\n      propName=\"validator\"\n      className=\"org.apache.commons.pipeline.validation.SimplePipelineValidator\"/>\n  <listener\n      className=\"org.apache.commons.pipeline.listener.ObjectProcessedEventCounter\"/>\n  <driverFactory\n      className=\"org.apache.commons.pipeline.driver.DedicatedThreadStageDriverFactory\"\n      id=\"kafka\">\n\n    <property\n        propName=\"queueFactory\"\n        className=\"org.apache.commons.pipeline.util.BlockingQueueFactory$ArrayBlockingQueueFactory\"\n        capacity=\"1000\"\n        fair=\"false\"/>\n  </driverFactory>\n\n  <!-- Read tweets from the thrift kafka queue. The reader loops forever. -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.kafka.KafkaRawRecordConsumerStage\"\n      kafkaClusterPath=\"\"\n      kafkaClientId=\"\"\n      kafkaTopicName=\"\"\n      kafkaConsumerGroupId=\"\"\n      maxPollRecords=\"1\"\n      pollTimeoutMs=\"1000\"\n      partitioned=\"false\"\n      deciderKey=\"\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Deserialize the bytes into TweetData -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.TweetEventDeserializerStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Filter to only have the safetytype for this cluster -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.FilterEventsBySafetyTypeStage\"\n      tweetCreateLatencyLogThresholdMillis=\"5000\"\n      safetyType=\"PUBLIC\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Parse to TwitterMessage -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ThriftTweetParserStage\"\n      tweetCreateEventBranchNames=\"kafka_retweet_and_reply\"\n      tweetDeleteEventBranchNames=\"kafka_update_events_delete\"\n      driverFactoryId=\"kafka\"/>\n\n  <branch>\n    <pipeline key=\"kafka_update_events_delete\">\n      <property\n          propName=\"validator\"\n          className=\"org.apache.commons.pipeline.validation.SimplePipelineValidator\"/>\n      <listener\n          className=\"org.apache.commons.pipeline.listener.ObjectProcessedEventCounter\"/>\n      <driverFactory\n          className=\"org.apache.commons.pipeline.driver.DedicatedThreadStageDriverFactory\"\n          id=\"kafka_update_events_delete\">\n\n        <!-- we are willing to queue more deletes than other stages,\n             to make sure we don't slow down the incoming tweets -->\n        <property\n            propName=\"queueFactory\"\n            className=\"org.apache.commons.pipeline.util.BlockingQueueFactory$ArrayBlockingQueueFactory\"\n            capacity=\"1000\"\n            fair=\"false\"/>\n      </driverFactory>\n\n      <stage\n          className=\"com.twitter.search.ingester.pipeline.twitter.kafka.DeleteUpdateEventsKafkaProducerStage\"\n          kafkaClusterPath=\"\"\n          kafkaClientId=\"\"\n          kafkaTopicName=\"\"\n          driverFactoryId=\"kafka_update_events_delete\"/>\n    </pipeline>\n  </branch>\n\n  <!-- Processes retweets and replies to tweets -->\n  <branch>\n    <pipeline key=\"kafka_retweet_and_reply\">\n      <property\n          propName=\"validator\"\n          className=\"org.apache.commons.pipeline.validation.SimplePipelineValidator\"/>\n      <listener\n          className=\"org.apache.commons.pipeline.listener.ObjectProcessedEventCounter\"/>\n      <driverFactory\n          className=\"org.apache.commons.pipeline.driver.DedicatedThreadStageDriverFactory\"\n          id=\"kafka_retweet_and_reply\">\n\n        <property\n            propName=\"queueFactory\"\n            className=\"org.apache.commons.pipeline.util.BlockingQueueFactory$ArrayBlockingQueueFactory\"\n            capacity=\"1000\"\n            fair=\"false\"/>\n      </driverFactory>\n\n      <!-- An incoming reply to this stage can either be a tweet directed at someone using @mention, or\n           a tweet that is a direct reply to another tweet. This stage filters retweets and tweets that are\n           direct replies to other tweets into the retweet_and_reply pipeline -->\n      <stage\n          className=\"com.twitter.search.ingester.pipeline.twitter.FilterRetweetsAndRepliesStage\"\n          driverFactoryId=\"kafka_retweet_and_reply\"/>\n\n      <stage\n          className=\"com.twitter.search.ingester.pipeline.twitter.ConvertToThriftVersionedEventsStage\"\n          driverFactoryId=\"kafka_retweet_and_reply\"/>\n\n      <stage\n          className=\"com.twitter.search.ingester.pipeline.twitter.kafka.RetweetAndReplyUpdateEventsKafkaProducerStage\"\n          kafkaClusterPath=\"\"\n          kafkaClientId=\"\"\n          kafkaTopicName=\"\"\n          driverFactoryId=\"kafka_retweet_and_reply\"/>\n    </pipeline>\n  </branch>\n\n  <!-- filters out messages that are not formatted correctly -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.FilterTwitterMessageStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- retrieves space ids from space urls if the tweet has space urls -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.RetrieveSpaceIdsStage\"\n      driverFactoryId=\"kafka\"/>\n\n\n  <!-- looks up user reputation scores for each message -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.LookupUserPropertiesBatchedStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- extract text features of the message -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.TextFeatureExtractionWorkersStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- compute text quality score of the message -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.TextQualityEvaluationWorkerStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Extract lat/lon pairs from the text, and geocode them -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.SingleTweetExtractAndGeocodeLatLonStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- adds coded locations -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.PopulateCodedLocationsBatchedStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Parse the TwitterMessages into ThriftStatuses -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ConvertMessageToThriftStage\"\n      thriftVersionedEventsBranchName=\"kafka_base_tweets\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Branch for tweets -->\n  <branch>\n    <pipeline key=\"kafka_base_tweets\">\n      <property\n          propName=\"validator\"\n          className=\"org.apache.commons.pipeline.validation.SimplePipelineValidator\"/>\n      <listener\n          className=\"org.apache.commons.pipeline.listener.ObjectProcessedEventCounter\"/>\n      <driverFactory\n          className=\"org.apache.commons.pipeline.driver.DedicatedThreadStageDriverFactory\"\n          id=\"kafka_base_tweets\">\n\n        <property\n            propName=\"queueFactory\"\n            className=\"org.apache.commons.pipeline.util.BlockingQueueFactory$ArrayBlockingQueueFactory\"\n            capacity=\"1000\"\n            fair=\"false\"/>\n      </driverFactory>\n\n      <stage\n          className=\"com.twitter.search.ingester.pipeline.twitter.kafka.TweetThriftVersionedEventsKafkaProducerStage\"\n          kafkaClusterPath=\"\"\n          kafkaClientId=\"\"\n          kafkaTopicName=\"search_ingester_indexing_events_realtime_prod\"\n          driverFactoryId=\"kafka_base_tweets\"/>\n    </pipeline>\n  </branch>\n\n  <!-- Resolve compressed URL via Pink -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ResolveCompressedUrlsBatchedStage\"\n      pinkClientId=\"INGESTER\"\n      batchedStageBatchSize=\"10\"\n      tweetMaxAgeToResolve=\"10000\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Retrieve card information -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.RetrieveCardBatchedStage\"\n      tweetypieClientId=\"ingester.prod\"\n      internalBatchSize=\"50\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Retrieve named entities -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.RetrieveNamedEntitiesSingleTweetStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- retrieves space admins and title for a tweet if the tweet has space urls -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.RetrieveSpaceAdminsAndTitleStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- extract text features of the message -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.TextUrlsFeatureExtractionStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Compute the tweet signature -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ComputeTweetSignatureStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Parse the TwitterMessages into ThriftStatuses -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ConvertDelayedMessageToThriftStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.kafka.TweetThriftVersionedEventsKafkaProducerStage\"\n      kafkaClusterPath=\"\"\n      stageName=\"UpdateEvents\"\n      kafkaClientId=\"\"\n      kafkaTopicName=\"\"\n      driverFactoryId=\"kafka\"/>\n</pipeline>\n"
  },
  {
    "path": "science/search/ingester/config/pipeline-ingester.realtime_cg.xml",
    "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n<!-- Ingesters process tweet create events from TweetyPie and write them to a queue for Earlybird\nto index. -->\n<pipeline>\n  <property\n      propName=\"validator\"\n      className=\"org.apache.commons.pipeline.validation.SimplePipelineValidator\"/>\n  <listener\n      className=\"org.apache.commons.pipeline.listener.ObjectProcessedEventCounter\"/>\n  <driverFactory\n      className=\"org.apache.commons.pipeline.driver.DedicatedThreadStageDriverFactory\"\n      id=\"kafka\">\n\n    <property\n        propName=\"queueFactory\"\n        className=\"org.apache.commons.pipeline.util.BlockingQueueFactory$ArrayBlockingQueueFactory\"\n        capacity=\"1000\"\n        fair=\"false\"/>\n  </driverFactory>\n\n  <!-- Read tweets from the thrift kafka queue. The reader loops forever. -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.kafka.KafkaRawRecordConsumerStage\"\n      kafkaClusterPath=\"\"\n      kafkaClientId=\"\"\n      kafkaTopicName=\"\"\n      kafkaConsumerGroupId=\"\"\n      maxPollRecords=\"1\"\n      pollTimeoutMs=\"1000\"\n      partitioned=\"false\"\n      deciderKey=\"\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Deserialize the bytes into TweetData -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.TweetEventDeserializerStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Filter to only have the safetytype = PUBLIC or PROTECTED -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.FilterEventsBySafetyTypeStage\"\n      tweetCreateLatencyLogThresholdMillis=\"5000\"\n      safetyType=\"PUBLIC_OR_PROTECTED\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Parse to TwitterMessage -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ThriftTweetParserStage\"\n      tweetDeleteEventBranchNames=\"kafka_update_events_delete\"\n      driverFactoryId=\"kafka\"/>\n\n  <branch>\n    <pipeline key=\"kafka_update_events_delete\">\n      <property\n          propName=\"validator\"\n          className=\"org.apache.commons.pipeline.validation.SimplePipelineValidator\"/>\n      <listener\n          className=\"org.apache.commons.pipeline.listener.ObjectProcessedEventCounter\"/>\n      <driverFactory\n          className=\"org.apache.commons.pipeline.driver.DedicatedThreadStageDriverFactory\"\n          id=\"kafka_update_events_delete\">\n\n        <!-- we are willing to queue more deletes than other stages,\n             to make sure we don't slow down the incoming tweets -->\n        <property\n            propName=\"queueFactory\"\n            className=\"org.apache.commons.pipeline.util.BlockingQueueFactory$ArrayBlockingQueueFactory\"\n            capacity=\"1000\"\n            fair=\"false\"/>\n      </driverFactory>\n\n      <stage\n          className=\"com.twitter.search.ingester.pipeline.twitter.kafka.DeleteUpdateEventsKafkaProducerStage\"\n          kafkaClusterPath=\"\"\n          kafkaClientId=\"\"\n          kafkaTopicName=\"\"\n          driverFactoryId=\"kafka_update_events_delete\"/>\n    </pipeline>\n  </branch>\n\n  <!-- filters out messages that are not formatted correctly -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.FilterTwitterMessageStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- retrieves space ids from space urls if the tweet has space urls -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.RetrieveSpaceIdsStage\"\n      driverFactoryId=\"kafka\"/>\n\n\n  <!-- looks up user reputation scores for each message -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.LookupUserPropertiesBatchedStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- extract text features of the message -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.TextFeatureExtractionWorkersStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- compute text quality score of the message -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.TextQualityEvaluationWorkerStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Extract lat/lon pairs from the text, and geocode them -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.SingleTweetExtractAndGeocodeLatLonStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- adds coded locations -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.PopulateCodedLocationsBatchedStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Parse the TwitterMessages into ThriftStatuses -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ConvertMessageToThriftStage\"\n      thriftVersionedEventsBranchName=\"kafka_base_tweets\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Branch for tweets -->\n  <branch>\n    <pipeline key=\"kafka_base_tweets\">\n      <property\n          propName=\"validator\"\n          className=\"org.apache.commons.pipeline.validation.SimplePipelineValidator\"/>\n      <listener\n          className=\"org.apache.commons.pipeline.listener.ObjectProcessedEventCounter\"/>\n      <driverFactory\n          className=\"org.apache.commons.pipeline.driver.DedicatedThreadStageDriverFactory\"\n          id=\"kafka_base_tweets\">\n\n        <property\n            propName=\"queueFactory\"\n            className=\"org.apache.commons.pipeline.util.BlockingQueueFactory$ArrayBlockingQueueFactory\"\n            capacity=\"1000\"\n            fair=\"false\"/>\n      </driverFactory>\n\n      <stage\n          className=\"com.twitter.search.ingester.pipeline.twitter.kafka.TweetThriftVersionedEventsKafkaProducerStage\"\n          kafkaClusterPath=\"\"\n          kafkaClientId=\"\"\n          kafkaTopicName=\"\"\n          driverFactoryId=\"kafka_base_tweets\"/>\n    </pipeline>\n  </branch>\n\n  <!-- Resolve compressed URL via Pink -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ResolveCompressedUrlsBatchedStage\"\n      pinkClientId=\"INGESTER\"\n      batchedStageBatchSize=\"10\"\n      tweetMaxAgeToResolve=\"10000\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Retrieve card information -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.RetrieveCardBatchedStage\"\n      tweetypieClientId=\"\"\n      internalBatchSize=\"50\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Retrieve named entities -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.RetrieveNamedEntitiesSingleTweetStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- retrieves space admins and title for a tweet if the tweet has space urls -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.RetrieveSpaceAdminsAndTitleStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- extract text features of the message -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.TextUrlsFeatureExtractionStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Compute the tweet signature -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ComputeTweetSignatureStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <!-- Parse the TwitterMessages into ThriftStatuses -->\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.ConvertDelayedMessageToThriftStage\"\n      driverFactoryId=\"kafka\"/>\n\n  <stage\n      className=\"com.twitter.search.ingester.pipeline.twitter.kafka.TweetThriftVersionedEventsKafkaProducerStage\"\n      kafkaClusterPath=\"\"\n      stageName=\"UpdateEvents\"\n      kafkaClientId=\"\"\n      kafkaTopicName=\"\"\n      driverFactoryId=\"kafka\"/>\n</pipeline>\n"
  },
  {
    "path": "simclusters-ann/BUILD.bazel",
    "content": "# This prevents SQ query from grabbing //:all since it traverses up once to find a BUILD\n"
  },
  {
    "path": "simclusters-ann/README.md",
    "content": "# SimClusters ANN\n\nSimClusters ANN is a service that returns tweet candidate recommendations given a SimClusters embedding. The service implements tweet recommendations based on the Approximate Cosine Similarity algorithm.\n\nThe cosine similarity between two Tweet SimClusters Embedding represents the relevance level of two tweets in SimCluster space. The traditional algorithm for calculating cosine similarity is expensive and hard to support by the existing infrastructure. Therefore, the Approximate Cosine Similarity algorithm is introduced to save response time by reducing I/O operations.\n\n## Background\nSimClusters V2 runtime infra introduces the SimClusters and its online and offline approaches. A heron job builds the mapping between SimClusters and Tweets. The job saves top 400 Tweets for a SimClusters and top 100 SimClusters for a Tweet. Favorite score and follow score are two types of tweet score.  In the document, the top 100 SimClusters based on the favorite score for a Tweet stands for the Tweet SimClusters Embedding. \n\nThe cosine similarity between two Tweet SimClusters Embedding presents the relevant level of two tweets in SimCluster space. The score varies from 0 to 1. The high cosine similarity score(>= 0.7 in Prod) means that the users who like two tweets share the same SimClusters. \n\n\nSimClusters from the Linear Algebra Perspective discussed the difference between the dot-product and cosine similarity in SimCluster space. We believe the cosine similarity approach is better because it avoids the bias of tweet popularity.\n\n However, calculating the cosine similarity between two Tweets is pretty expensive in Tweet candidate generation. In TWISTLY, we scan at most 15,000 (6 source tweets * 25 clusters * 100 tweets per clusters) tweet candidates for every Home Timeline request. The traditional algorithm needs to make API calls to fetch 15,000 tweet SimCluster embeddings. Consider that we need to process over 6,000 RPS, it’s hard to support by the existing infrastructure.  \n\n\n## SimClusters Approximate Cosine Similarity Core Algorithm\n\n1. Provide a source SimCluster Embedding *SV*, *SV = [(SC1, Score), (SC2, Score), (SC3, Score) …]*\n\n2. Fetch top *M* tweets for each Top *N* SimClusters based on SV. In Prod, *M = 400*, *N = 50*.  Tweets may appear in multiple SimClusters. \n \n|   |   |   |   |\n|---|---|---|---|\n| SC1  | T1:Score  | T2: Score  | ...   |\n| SC2 |  T3: Score | T4: Score  |  ... |\n\n\n3. Based on the previous table, generate an *(M x N) x N* Matrix *R*. The *R* represents the approximate SimCluster embeddings for *MxN* tweets. The embedding only contains top *N* SimClusters from *SV*. Only top *M* tweets from each SimCluster have the score. Others are 0. \n\n|   |  SC1 |  SC2 | ...   |\n|---|---|---|---|\n| T1  | Score  | 0  | ...   |\n| T2 |  Score | 0 |  ... |\n| T3 |  0 | Score  |  ... |\n\n4. Compute the dot product between source vector and the approximate vectors for each tweet. (Calculate *R • SV^T*). Take top *X* tweets. In Prod, *X = 200*\n\n5. Fetch *X* tweet SimClusters Embedding, Calculate Cosine Similarity between *X* tweets and *SV*, Return top *Y* above a certain threshold *Z*.\n\nApproximate Cosine Similarity is an approximate algorithm. Instead of fetching *M * N* tweets embedding, it only fetches *X* tweets embedding. In prod, *X / M * N * 100% = 6%*. Based on the metrics during TWISTLY development, most of the response time is consumed by I/O operation. The Approximate Cosine Similarity is a good approach to save a large amount of response time. \n\nThe idea of the approximate algorithm is based on the assumption that the higher dot-product between source tweets’ SimCluster embedding and candidate tweet’s limited SimCluster Embedding, the possibility that these two tweets are relevant is higher. Additional Cosine Similarity filter is to guarantee that the results are not affected by popularity bias.  \n\nAdjusting the M, N, X, Y, Z is able to balance the precision and recall for different products. The implementation of approximate cosine similarity is used by TWISTLY, Interest-based tweet recommendation, Similar Tweet in RUX, and Author based recommendation. This algorithm is also suitable for future user or entity recommendation based on SimClusters Embedding. \n\n\n# -------------------------------\n# Build and Test\n# -------------------------------\nCompile the service\n\n    $ ./bazel build simclusters-ann/server:bin\n\nUnit tests\n\n    $ ./bazel test simclusters-ann/server:bin\n\n# -------------------------------\n# Deploy\n# -------------------------------\n\n## Prerequisite for devel deployments\nFirst of all, you need to generate Service to Service certificates for use while developing locally. This only needs to be done ONCE:\n\nTo add cert files to Aurora (if you want to deploy to DEVEL):\n```\n$ developer-cert-util --env devel --job simclusters-ann\n```\n\n## Deploying to devel/staging from a local build\nReference -\n    \n    $ ./simclusters-ann/bin/deploy.sh --help\n\nUse the script to build the service in your local branch, upload it to packer and deploy in devel aurora:\n\n    $ ./simclusters-ann/bin/deploy.sh atla $USER devel simclusters-ann\n\nYou can also deploy to staging with this script. E.g. to deploy to instance 1:\n\n    $ ./simclusters-ann/bin/deploy.sh atla simclusters-ann staging simclusters-ann <instance-number>\n\n## Deploying to production\n\nProduction deploys should be managed by Workflows. \n_Do not_ deploy to production unless it is an emergency and you have approval from oncall.\n\n##### It is not recommended to deploy from Command Lines into production environments, unless 1) you're testing a small change in Canary shard [0,9]. 2) Tt is an absolute emergency. Be sure to make oncalls aware of the changes you're deploying.\n\n    $ ./simclusters-ann/bin/deploy.sh atla simclusters-ann prod simclusters-ann <instance-number>\nIn the case of multiple instances,\n\n    $ ./simclusters-ann/bin/deploy.sh atla simclusters-ann prod simclusters-ann <instance-number-start>-<instance-number-end>\n\n## Checking Deployed Version and Rolling Back\n\nWherever possible, roll back using Workflows by finding an earlier good version and clicking the \"rollback\" button in the UI. This is the safest and least error-prone method.\n"
  },
  {
    "path": "simclusters-ann/server/BUILD",
    "content": "jvm_binary(\n    name = \"bin\",\n    basename = \"simclusters-ann\",\n    main = \"com.twitter.simclustersann.SimClustersAnnServerMain\",\n    runtime_platform = \"java11\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-zipkin-scribe/src/main/scala\",\n        \"finatra/inject/inject-logback/src/main/scala\",\n        \"loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback\",\n        \"simclusters-ann/server/src/main/scala/com/twitter/simclustersann\",\n        \"twitter-server-internal/src/main/scala\",\n        \"twitter-server/logback-classic/src/main/scala\",\n    ],\n)\n\n#  Aurora Workflows build phase convention requires a jvm_app named with ${project-name}-app\njvm_app(\n    name = \"simclusters-ann-app\",\n    archive = \"zip\",\n    binary = \":bin\",\n    tags = [\"bazel-compatible\"],\n)\n"
  },
  {
    "path": "simclusters-ann/server/src/main/resources/BUILD",
    "content": "resources(\n    sources = [\n        \"*.xml\",\n        \"config/*.yml\",\n    ],\n    tags = [\"bazel-compatible\"],\n)\n"
  },
  {
    "path": "simclusters-ann/server/src/main/resources/config/decider.yml",
    "content": "# SimClusters embedding store enable / disable decider values\n\n# ---------- Dark Traffic Proxy ----------\ndark_traffic_filter:\n  comment: Proportion of the requests that are forwarded as dark traffic to the proxy\n  default_availability: 0\n\n# Tweet embeddings\nenable_LogFavBasedTweet_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_LogFavLongestL2EmbeddingTweet_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n# Entity embeddings\nenable_FavTfgTopic_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n\nenable_LogFavBasedKgoApeTopic_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n# KnownFor embeddings\nenable_FavBasedProducer_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_FollowBasedProducer_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_RelaxedAggregatableLogFavBasedProducer_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\n# InterestedIn embeddings\nenable_LogFavBasedUserInterestedInFromAPE_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_FollowBasedUserInterestedInFromAPE_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_FavBasedUserInterestedIn_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_FollowBasedUserInterestedIn_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_LogFavBasedUserInterestedIn_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_FilteredUserInterestedIn_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_UnfilteredUserInterestedIn_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_LogFavBasedUserInterestedMaxpoolingAddressBookFromIIAPE_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_LogFavBasedUserInterestedAverageAddressBookFromIIAPE_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_LogFavBasedUserInterestedBooktypeMaxpoolingAddressBookFromIIAPE_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_LogFavBasedUserInterestedLargestDimMaxpoolingAddressBookFromIIAPE_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_LogFavBasedUserInterestedConnectedMaxpoolingAddressBookFromIIAPE_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n\nenable_UserNextInterestedIn_Model20m145k2020:\n  comment: \"Enable the read traffic to (embeddingType, modelVersion) from 0% to 100%. 0 means return EMPTY for all requests.\"\n  default_availability: 10000\n"
  },
  {
    "path": "simclusters-ann/server/src/main/resources/logback.xml",
    "content": "<configuration>\n    <shutdownHook class=\"ch.qos.logback.core.hook.DelayingShutdownHook\"/>\n\n    <!-- ===================================================== -->\n    <!-- Service Config -->\n    <!-- ===================================================== -->\n    <property name=\"DEFAULT_SERVICE_PATTERN\"\n              value=\"%-16X{traceId} %-12X{clientId:--} %-16X{method} %-25logger{0} %msg\"/>\n\n    <property name=\"DEFAULT_ACCESS_PATTERN\"\n              value=\"%msg\"/>\n\n    <!-- ===================================================== -->\n    <!-- Common Config -->\n    <!-- ===================================================== -->\n\n    <!-- JUL/JDK14 to Logback bridge -->\n    <contextListener class=\"ch.qos.logback.classic.jul.LevelChangePropagator\">\n        <resetJUL>true</resetJUL>\n    </contextListener>\n\n    <!-- ====================================================================================== -->\n    <!-- NOTE: The following appenders use a simple TimeBasedRollingPolicy configuration.       -->\n    <!--       You may want to consider using a more advanced SizeAndTimeBasedRollingPolicy.    -->\n    <!--       See: https://logback.qos.ch/manual/appenders.html#SizeAndTimeBasedRollingPolicy  -->\n    <!-- ====================================================================================== -->\n\n    <!-- Service Log (rollover daily, keep maximum of 21 days of gzip compressed logs) -->\n    <appender name=\"SERVICE\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n        <file>${log.service.output}</file>\n        <rollingPolicy class=\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\">\n            <!-- daily rollover -->\n            <fileNamePattern>${log.service.output}.%d.gz</fileNamePattern>\n            <!-- the maximum total size of all the log files -->\n            <totalSizeCap>3GB</totalSizeCap>\n            <!-- keep maximum 21 days' worth of history -->\n            <maxHistory>21</maxHistory>\n            <cleanHistoryOnStart>true</cleanHistoryOnStart>\n        </rollingPolicy>\n        <encoder>\n            <pattern>%date %.-3level ${DEFAULT_SERVICE_PATTERN}%n</pattern>\n        </encoder>\n    </appender>\n\n    <!-- Access Log (rollover daily, keep maximum of 21 days of gzip compressed logs) -->\n    <appender name=\"ACCESS\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n        <file>${log.access.output}</file>\n        <rollingPolicy class=\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\">\n            <!-- daily rollover -->\n            <fileNamePattern>${log.access.output}.%d.gz</fileNamePattern>\n            <!-- the maximum total size of all the log files -->\n            <totalSizeCap>100MB</totalSizeCap>\n            <!-- keep maximum 7 days' worth of history -->\n            <maxHistory>7</maxHistory>\n            <cleanHistoryOnStart>true</cleanHistoryOnStart>\n        </rollingPolicy>\n        <encoder>\n            <pattern>${DEFAULT_ACCESS_PATTERN}%n</pattern>\n        </encoder>\n    </appender>\n\n    <!--LogLens -->\n    <appender name=\"LOGLENS\" class=\"com.twitter.loglens.logback.LoglensAppender\">\n        <mdcAdditionalContext>true</mdcAdditionalContext>\n        <category>${log.lens.category}</category>\n        <index>${log.lens.index}</index>\n        <tag>${log.lens.tag}/service</tag>\n        <encoder>\n            <pattern>%msg</pattern>\n        </encoder>\n    </appender>\n\n    <!-- LogLens Access -->\n    <appender name=\"LOGLENS-ACCESS\" class=\"com.twitter.loglens.logback.LoglensAppender\">\n        <mdcAdditionalContext>true</mdcAdditionalContext>\n        <category>${log.lens.category}</category>\n        <index>${log.lens.index}</index>\n        <tag>${log.lens.tag}/access</tag>\n        <encoder>\n            <pattern>%msg</pattern>\n        </encoder>\n    </appender>\n\n    <!-- Pipeline Execution Logs -->\n    <appender name=\"ALLOW-LISTED-PIPELINE-EXECUTIONS\" class=\"ch.qos.logback.core.rolling.RollingFileAppender\">\n        <file>allow_listed_pipeline_executions.log</file>\n        <rollingPolicy class=\"ch.qos.logback.core.rolling.TimeBasedRollingPolicy\">\n            <!-- daily rollover -->\n            <fileNamePattern>allow_listed_pipeline_executions.log.%d.gz</fileNamePattern>\n             <!-- the maximum total size of all the log files -->\n            <totalSizeCap>100MB</totalSizeCap>\n            <!-- keep maximum 7 days' worth of history -->\n            <maxHistory>7</maxHistory>\n            <cleanHistoryOnStart>true</cleanHistoryOnStart>\n        </rollingPolicy>\n        <encoder>\n            <pattern>%date %.-3level ${DEFAULT_SERVICE_PATTERN}%n</pattern>\n        </encoder>\n    </appender>\n\n    <!-- ===================================================== -->\n    <!-- Primary Async Appenders -->\n    <!-- ===================================================== -->\n\n    <property name=\"async_queue_size\" value=\"${queue.size:-50000}\"/>\n    <property name=\"async_max_flush_time\" value=\"${max.flush.time:-0}\"/>\n\n    <appender name=\"ASYNC-SERVICE\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"SERVICE\"/>\n    </appender>\n\n    <appender name=\"ASYNC-ACCESS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"ACCESS\"/>\n    </appender>\n\n    <appender name=\"ASYNC-ALLOW-LISTED-PIPELINE-EXECUTIONS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"ALLOW-LISTED-PIPELINE-EXECUTIONS\"/>\n    </appender>\n\n    <appender name=\"ASYNC-LOGLENS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"LOGLENS\"/>\n    </appender>\n\n    <appender name=\"ASYNC-LOGLENS-ACCESS\" class=\"com.twitter.inject.logback.AsyncAppender\">\n        <queueSize>${async_queue_size}</queueSize>\n        <maxFlushTime>${async_max_flush_time}</maxFlushTime>\n        <appender-ref ref=\"LOGLENS-ACCESS\"/>\n    </appender>\n\n    <!-- ===================================================== -->\n    <!-- Package Config -->\n    <!-- ===================================================== -->\n\n    <!-- Per-Package Config -->\n    <logger name=\"com.twitter\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.wilyns\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.configbus.client.file\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.finagle.mux\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.finagle.serverset2\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.logging.ScribeHandler\" level=\"INHERITED\"/>\n    <logger name=\"com.twitter.zookeeper.client.internal\" level=\"INHERITED\"/>\n    <!-- Disable deadline exceeded logs by default. This can be overriden dynamically in the admin panel of individual instances. -->\n    <logger name=\"com.twitter.relevance_platform.common.exceptions.DeadlineExceededExceptionMapper\" level=\"OFF\"/>\n\n    <!-- Root Config -->\n    <!-- For all logs except access logs, disable logging below log_level level by default. This can be overriden in the per-package loggers, and dynamically in the admin panel of individual instances. -->\n    <root level=\"${log_level:-INFO}\">\n        <appender-ref ref=\"ASYNC-SERVICE\"/>\n        <appender-ref ref=\"ASYNC-LOGLENS\"/>\n    </root>\n\n    <!-- Access Logging -->\n   <!-- Access logs are turned off by default -->\n    <logger name=\"com.twitter.finatra.thrift.filters.AccessLoggingFilter\" level=\"OFF\" additivity=\"false\">\n        <appender-ref ref=\"ASYNC-ACCESS\"/>\n        <appender-ref ref=\"ASYNC-LOGLENS-ACCESS\"/>\n    </logger>\n\n</configuration>\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-http/src/main/scala\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"finatra-internal/decider/src/main/scala\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra/inject/inject-app/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-server/src/main/scala\",\n        \"finatra/inject/inject-thrift-client/src/main/scala\",\n        \"finatra/inject/inject-utils/src/main/scala\",\n        \"finatra/utils/src/main/java/com/twitter/finatra/annotations\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/exceptions\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/filters\",\n        \"simclusters-ann/server/src/main/resources\",\n        \"simclusters-ann/server/src/main/scala/com/twitter/simclustersann/controllers\",\n        \"simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules\",\n        \"simclusters-ann/thrift/src/main/thrift:thrift-scala\",\n        \"src/thrift/com/twitter/search:earlybird-scala\",\n        \"thrift-web-forms/src/main/scala/com/twitter/thriftwebforms/view\",\n        \"twitter-server/server/src/main/scala\",\n        \"util/util-app/src/main/scala\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/SimclustersAnnServer.scala",
    "content": "package com.twitter.simclustersann\n\nimport com.google.inject.Module\nimport com.twitter.finatra.decider.modules.DeciderModule\nimport com.twitter.finatra.mtls.thriftmux.Mtls\nimport com.twitter.finatra.thrift.ThriftServer\nimport com.twitter.finatra.thrift.filters._\nimport com.twitter.finatra.thrift.routing.ThriftRouter\nimport com.twitter.inject.thrift.modules.ThriftClientIdModule\nimport com.twitter.relevance_platform.common.exceptions._\nimport com.twitter.simclustersann.controllers.SimClustersANNController\nimport com.twitter.simclustersann.exceptions.InvalidRequestForSimClustersAnnVariantExceptionMapper\nimport com.twitter.simclustersann.modules._\nimport com.twitter.simclustersann.thriftscala.SimClustersANNService\nimport com.twitter.finagle.Filter\nimport com.twitter.finatra.annotations.DarkTrafficFilterType\nimport com.twitter.inject.annotations.Flags\nimport com.twitter.relevance_platform.common.filters.DarkTrafficFilterModule\nimport com.twitter.relevance_platform.common.filters.ClientStatsFilter\nimport com.twitter.simclustersann.common.FlagNames.DisableWarmup\n\nobject SimClustersAnnServerMain extends SimClustersAnnServer\n\nclass SimClustersAnnServer extends ThriftServer with Mtls {\n  flag(\n    name = DisableWarmup,\n    default = false,\n    help = \"If true, no warmup will be run.\"\n  )\n\n  override val name = \"simclusters-ann-server\"\n\n  override val modules: Seq[Module] = Seq(\n    CacheModule,\n    ServiceNameMapperModule,\n    ClusterConfigMapperModule,\n    ClusterConfigModule,\n    ClusterTweetIndexProviderModule,\n    DeciderModule,\n    EmbeddingStoreModule,\n    FlagsModule,\n    FuturePoolProvider,\n    RateLimiterModule,\n    SimClustersANNCandidateSourceModule,\n    StratoClientProviderModule,\n    ThriftClientIdModule,\n    new CustomMtlsThriftWebFormsModule[SimClustersANNService.MethodPerEndpoint](this),\n    new DarkTrafficFilterModule[SimClustersANNService.ReqRepServicePerEndpoint]()\n  )\n\n  def configureThrift(router: ThriftRouter): Unit = {\n    router\n      .filter[LoggingMDCFilter]\n      .filter[TraceIdMDCFilter]\n      .filter[ThriftMDCFilter]\n      .filter[ClientStatsFilter]\n      .filter[ExceptionMappingFilter]\n      .filter[Filter.TypeAgnostic, DarkTrafficFilterType]\n      .exceptionMapper[InvalidRequestForSimClustersAnnVariantExceptionMapper]\n      .exceptionMapper[DeadlineExceededExceptionMapper]\n      .exceptionMapper[UnhandledExceptionMapper]\n      .add[SimClustersANNController]\n  }\n\n  override protected def warmup(): Unit = {\n    if (!injector.instance[Boolean](Flags.named(DisableWarmup))) {\n      handle[SimclustersAnnWarmupHandler]()\n    }\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/SimclustersAnnWarmupHandler.scala",
    "content": "package com.twitter.simclustersann\n\nimport com.twitter.inject.Logging\nimport com.twitter.inject.utils.Handler\nimport javax.inject.Inject\nimport scala.util.control.NonFatal\nimport com.google.common.util.concurrent.RateLimiter\nimport com.twitter.conversions.DurationOps.richDurationFromInt\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.simclusters_v2.common.ClusterId\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Await\nimport com.twitter.util.ExecutorServiceFuturePool\nimport com.twitter.util.Future\n\nclass SimclustersAnnWarmupHandler @Inject() (\n  clusterTweetCandidatesStore: ReadableStore[ClusterId, Seq[(TweetId, Double)]],\n  futurePool: ExecutorServiceFuturePool,\n  rateLimiter: RateLimiter,\n  statsReceiver: StatsReceiver)\n    extends Handler\n    with Logging {\n\n  private val stats = statsReceiver.scope(this.getClass.getName)\n\n  private val scopedStats = stats.scope(\"fetchFromCache\")\n  private val clusters = scopedStats.counter(\"clusters\")\n  private val fetchedKeys = scopedStats.counter(\"keys\")\n  private val failures = scopedStats.counter(\"failures\")\n  private val success = scopedStats.counter(\"success\")\n\n  private val SimclustersNumber = 144428\n\n  override def handle(): Unit = {\n    try {\n      val clusterIds = List.range(1, SimclustersNumber)\n      val futures: Seq[Future[Unit]] = clusterIds\n        .map { clusterId =>\n          clusters.incr()\n          futurePool {\n            rateLimiter.acquire()\n\n            Await.result(\n              clusterTweetCandidatesStore\n                .get(clusterId)\n                .onSuccess { _ =>\n                  success.incr()\n                }\n                .handle {\n                  case NonFatal(e) =>\n                    failures.incr()\n                },\n              timeout = 10.seconds\n            )\n            fetchedKeys.incr()\n          }\n        }\n\n      Await.result(Future.collect(futures), timeout = 10.minutes)\n\n    } catch {\n      case NonFatal(e) => error(e.getMessage, e)\n    } finally {\n      try {\n        futurePool.executor.shutdown()\n      } catch {\n        case NonFatal(_) =>\n      }\n      info(\"Warmup done.\")\n    }\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/candidate_source/ApproximateCosineSimilarity.scala",
    "content": "package com.twitter.simclustersann.candidate_source\n\nimport com.twitter.simclusters_v2.common.ClusterId\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclustersann.thriftscala.ScoringAlgorithm\nimport com.twitter.simclustersann.thriftscala.SimClustersANNConfig\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\nimport scala.collection.mutable\n\n/**\n * This store looks for tweets whose similarity is close to a Source SimClustersEmbeddingId.\n *\n * Approximate cosine similarity is the core algorithm to drive this store.\n *\n * Step 1 - 4 are in \"fetchCandidates\" method.\n * 1. Retrieve the SimClusters Embedding by the SimClustersEmbeddingId\n * 2. Fetch top N clusters' top tweets from the clusterTweetCandidatesStore (TopTweetsPerCluster index).\n * 3. Calculate all the tweet candidates' dot-product or approximate cosine similarity to source tweets.\n * 4. Take top M tweet candidates by the step 3's score\n */\ntrait ApproximateCosineSimilarity {\n  type ScoredTweet = (Long, Double)\n  def apply(\n    sourceEmbedding: SimClustersEmbedding,\n    sourceEmbeddingId: SimClustersEmbeddingId,\n    config: SimClustersANNConfig,\n    candidateScoresStat: Int => Unit,\n    clusterTweetsMap: Map[ClusterId, Option[Seq[(TweetId, Double)]]],\n    clusterTweetsMapArray: Map[ClusterId, Option[Array[(TweetId, Double)]]] = Map.empty\n  ): Seq[ScoredTweet]\n}\n\nobject ApproximateCosineSimilarity extends ApproximateCosineSimilarity {\n\n  final val InitialCandidateMapSize = 16384\n  val MaxNumResultsUpperBound = 1000\n  final val MaxTweetCandidateAgeUpperBound = 175200\n\n  private class HashMap[A, B](initSize: Int) extends mutable.HashMap[A, B] {\n    override def initialSize: Int = initSize // 16 - by default\n  }\n\n  private def parseTweetId(embeddingId: SimClustersEmbeddingId): Option[TweetId] = {\n    embeddingId.internalId match {\n      case InternalId.TweetId(tweetId) =>\n        Some(tweetId)\n      case _ =>\n        None\n    }\n  }\n\n  override def apply(\n    sourceEmbedding: SimClustersEmbedding,\n    sourceEmbeddingId: SimClustersEmbeddingId,\n    config: SimClustersANNConfig,\n    candidateScoresStat: Int => Unit,\n    clusterTweetsMap: Map[ClusterId, Option[Seq[(TweetId, Double)]]] = Map.empty,\n    clusterTweetsMapArray: Map[ClusterId, Option[Array[(TweetId, Double)]]] = Map.empty\n  ): Seq[ScoredTweet] = {\n    val now = Time.now\n    val earliestTweetId =\n      if (config.maxTweetCandidateAgeHours >= MaxTweetCandidateAgeUpperBound)\n        0L // Disable max tweet age filter\n      else\n        SnowflakeId.firstIdFor(now - Duration.fromHours(config.maxTweetCandidateAgeHours))\n    val latestTweetId =\n      SnowflakeId.firstIdFor(now - Duration.fromHours(config.minTweetCandidateAgeHours))\n\n    // Use Mutable map to optimize performance. The method is thread-safe.\n\n    // Set initial map size to around p75 of map size distribution to avoid too many copying\n    // from extending the size of the mutable hashmap\n    val candidateScoresMap =\n      new HashMap[TweetId, Double](InitialCandidateMapSize)\n    val candidateNormalizationMap =\n      new HashMap[TweetId, Double](InitialCandidateMapSize)\n\n    clusterTweetsMap.foreach {\n      case (clusterId, Some(tweetScores)) if sourceEmbedding.contains(clusterId) =>\n        val sourceClusterScore = sourceEmbedding.getOrElse(clusterId)\n\n        for (i <- 0 until Math.min(tweetScores.size, config.maxTopTweetsPerCluster)) {\n          val (tweetId, score) = tweetScores(i)\n\n          if (!parseTweetId(sourceEmbeddingId).contains(tweetId) &&\n            tweetId >= earliestTweetId && tweetId <= latestTweetId) {\n            candidateScoresMap.put(\n              tweetId,\n              candidateScoresMap.getOrElse(tweetId, 0.0) + score * sourceClusterScore)\n            candidateNormalizationMap\n              .put(tweetId, candidateNormalizationMap.getOrElse(tweetId, 0.0) + score * score)\n          }\n        }\n      case _ => ()\n    }\n\n    candidateScoresStat(candidateScoresMap.size)\n\n    // Re-Rank the candidate by configuration\n    val processedCandidateScores: Seq[(TweetId, Double)] = candidateScoresMap.map {\n      case (candidateId, score) =>\n        // Enable Partial Normalization\n        val processedScore = {\n          // We applied the \"log\" version of partial normalization when we rank candidates\n          // by log cosine similarity\n          config.annAlgorithm match {\n            case ScoringAlgorithm.LogCosineSimilarity =>\n              score / sourceEmbedding.logNorm / math.log(1 + candidateNormalizationMap(candidateId))\n            case ScoringAlgorithm.CosineSimilarity =>\n              score / sourceEmbedding.l2norm / math.sqrt(candidateNormalizationMap(candidateId))\n            case ScoringAlgorithm.CosineSimilarityNoSourceEmbeddingNormalization =>\n              score / math.sqrt(candidateNormalizationMap(candidateId))\n            case ScoringAlgorithm.DotProduct => score\n          }\n        }\n        candidateId -> processedScore\n    }.toSeq\n\n    processedCandidateScores\n      .filter(_._2 >= config.minScore)\n      .sortBy(-_._2)\n      .take(Math.min(config.maxNumResults, MaxNumResultsUpperBound))\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/candidate_source/BUILD",
    "content": "scala_library(\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/twitter/storehaus:core\",\n        \"frigate/frigate-common:base\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/base\",\n        \"simclusters-ann/thrift/src/main/thrift:thrift-scala\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/scala/com/twitter/simclusters_v2/summingbird/stores\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n        \"util/util-stats/src/main/scala/com/twitter/finagle/stats\",\n    ],\n)\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/candidate_source/ExperimentalApproximateCosineSimilarity.scala",
    "content": "package com.twitter.simclustersann.candidate_source\n\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclustersann.thriftscala.ScoringAlgorithm\nimport com.twitter.simclustersann.thriftscala.SimClustersANNConfig\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\nimport com.google.common.collect.Comparators\nimport com.twitter.simclusters_v2.common.ClusterId\n\n/**\n * A modified version of OptimizedApproximateCosineSimilarity which uses more java streams to avoid\n * materializing intermediate collections. Its performance is still under investigation.\n */\nobject ExperimentalApproximateCosineSimilarity extends ApproximateCosineSimilarity {\n\n  final val InitialCandidateMapSize = 16384\n  val MaxNumResultsUpperBound = 1000\n  final val MaxTweetCandidateAgeUpperBound = 175200\n\n  private def parseTweetId(embeddingId: SimClustersEmbeddingId): Option[TweetId] = {\n    embeddingId.internalId match {\n      case InternalId.TweetId(tweetId) =>\n        Some(tweetId)\n      case _ =>\n        None\n    }\n  }\n  private val CompareByScore: java.util.Comparator[(Long, Double)] =\n    new java.util.Comparator[(Long, Double)] {\n      override def compare(o1: (Long, Double), o2: (Long, Double)): Int = {\n        java.lang.Double.compare(o1._2, o2._2)\n      }\n    }\n  class Scores(var score: Double, var norm: Double)\n\n  override def apply(\n    sourceEmbedding: SimClustersEmbedding,\n    sourceEmbeddingId: SimClustersEmbeddingId,\n    config: SimClustersANNConfig,\n    candidateScoresStat: Int => Unit,\n    clusterTweetsMap: Map[ClusterId, Option[Seq[(TweetId, Double)]]] = Map.empty,\n    clusterTweetsMapArray: Map[ClusterId, Option[Array[(TweetId, Double)]]] = Map.empty\n  ): Seq[ScoredTweet] = {\n    val now = Time.now\n    val earliestTweetId =\n      if (config.maxTweetCandidateAgeHours >= MaxTweetCandidateAgeUpperBound)\n        0L // Disable max tweet age filter\n      else\n        SnowflakeId.firstIdFor(now - Duration.fromHours(config.maxTweetCandidateAgeHours))\n    val latestTweetId =\n      SnowflakeId.firstIdFor(now - Duration.fromHours(config.minTweetCandidateAgeHours))\n\n    val candidateScoresMap = new java.util.HashMap[Long, Scores](InitialCandidateMapSize)\n    val sourceTweetId = parseTweetId(sourceEmbeddingId).getOrElse(0L)\n\n    clusterTweetsMap.foreach {\n      case (clusterId, Some(tweetScores)) =>\n        val sourceClusterScore = sourceEmbedding.getOrElse(clusterId)\n\n        for (i <- 0 until Math.min(tweetScores.size, config.maxTopTweetsPerCluster)) {\n          val (tweetId, score) = tweetScores(i)\n\n          if (tweetId >= earliestTweetId &&\n            tweetId <= latestTweetId &&\n            tweetId != sourceTweetId) {\n\n            val scores = candidateScoresMap.get(tweetId)\n            if (scores == null) {\n              val scorePair = new Scores(\n                score = score * sourceClusterScore,\n                norm = score * score\n              )\n              candidateScoresMap.put(tweetId, scorePair)\n            } else {\n              scores.score = scores.score + (score * sourceClusterScore)\n              scores.norm = scores.norm + (score * score)\n            }\n          }\n        }\n      case _ => ()\n    }\n\n    candidateScoresStat(candidateScoresMap.size)\n\n    val normFn: (Long, Scores) => (Long, Double) = config.annAlgorithm match {\n      case ScoringAlgorithm.LogCosineSimilarity =>\n        (candidateId: Long, score: Scores) =>\n          (\n            candidateId,\n            score.score / sourceEmbedding.logNorm / math.log(1 + score.norm)\n          )\n      case ScoringAlgorithm.CosineSimilarity =>\n        (candidateId: Long, score: Scores) =>\n          (\n            candidateId,\n            score.score / sourceEmbedding.l2norm / math.sqrt(score.norm)\n          )\n      case ScoringAlgorithm.CosineSimilarityNoSourceEmbeddingNormalization =>\n        (candidateId: Long, score: Scores) =>\n          (\n            candidateId,\n            score.score / math.sqrt(score.norm)\n          )\n      case ScoringAlgorithm.DotProduct =>\n        (candidateId: Long, score: Scores) =>\n          (\n            candidateId,\n            score.score\n          )\n    }\n\n    import scala.collection.JavaConverters._\n\n    val topKCollector = Comparators.greatest(\n      Math.min(config.maxNumResults, MaxNumResultsUpperBound),\n      CompareByScore\n    )\n\n    candidateScoresMap\n      .entrySet().stream()\n      .map[(Long, Double)]((e: java.util.Map.Entry[Long, Scores]) => normFn(e.getKey, e.getValue))\n      .filter((s: (Long, Double)) => s._2 >= config.minScore)\n      .collect(topKCollector)\n      .asScala\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/candidate_source/OptimizedApproximateCosineSimilarity.scala",
    "content": "package com.twitter.simclustersann.candidate_source\n\nimport com.twitter.simclusters_v2.common.ClusterId\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclustersann.thriftscala.ScoringAlgorithm\nimport com.twitter.simclustersann.thriftscala.SimClustersANNConfig\nimport com.twitter.snowflake.id.SnowflakeId\nimport com.twitter.util.Duration\nimport com.twitter.util.Time\n\n/**\n * Compared with ApproximateCosineSimilarity, this implementation:\n * - moves some computation aroudn to reduce allocations\n * - uses a single hashmap to store both scores and normalization coefficients\n * - uses some java collections in place of scala ones\n * Testing is still in progress, but this implementation shows significant (> 2x) improvements in\n * CPU utilization and allocations with 800 tweets per cluster.\n */\nobject OptimizedApproximateCosineSimilarity extends ApproximateCosineSimilarity {\n\n  final val InitialCandidateMapSize = 16384\n  val MaxNumResultsUpperBound = 1000\n  final val MaxTweetCandidateAgeUpperBound = 175200\n\n  private def parseTweetId(embeddingId: SimClustersEmbeddingId): Option[TweetId] = {\n    embeddingId.internalId match {\n      case InternalId.TweetId(tweetId) =>\n        Some(tweetId)\n      case _ =>\n        None\n    }\n  }\n\n  override def apply(\n    sourceEmbedding: SimClustersEmbedding,\n    sourceEmbeddingId: SimClustersEmbeddingId,\n    config: SimClustersANNConfig,\n    candidateScoresStat: Int => Unit,\n    clusterTweetsMap: Map[ClusterId, Option[Seq[(TweetId, Double)]]] = Map.empty,\n    clusterTweetsMapArray: Map[ClusterId, Option[Array[(TweetId, Double)]]] = Map.empty\n  ): Seq[ScoredTweet] = {\n    val now = Time.now\n    val earliestTweetId =\n      if (config.maxTweetCandidateAgeHours >= MaxTweetCandidateAgeUpperBound)\n        0L // Disable max tweet age filter\n      else\n        SnowflakeId.firstIdFor(now - Duration.fromHours(config.maxTweetCandidateAgeHours))\n    val latestTweetId =\n      SnowflakeId.firstIdFor(now - Duration.fromHours(config.minTweetCandidateAgeHours))\n\n    val candidateScoresMap = new java.util.HashMap[Long, (Double, Double)](InitialCandidateMapSize)\n\n    val sourceTweetId = parseTweetId(sourceEmbeddingId).getOrElse(0L)\n\n    clusterTweetsMap.foreach {\n      case (clusterId, Some(tweetScores)) if sourceEmbedding.contains(clusterId) =>\n        val sourceClusterScore = sourceEmbedding.getOrElse(clusterId)\n\n        for (i <- 0 until Math.min(tweetScores.size, config.maxTopTweetsPerCluster)) {\n          val (tweetId, score) = tweetScores(i)\n\n          if (tweetId >= earliestTweetId &&\n            tweetId <= latestTweetId &&\n            tweetId != sourceTweetId) {\n\n            val scores = candidateScoresMap.getOrDefault(tweetId, (0.0, 0.0))\n            val newScores = (\n              scores._1 + score * sourceClusterScore,\n              scores._2 + score * score,\n            )\n            candidateScoresMap.put(tweetId, newScores)\n          }\n        }\n      case _ => ()\n    }\n\n    candidateScoresStat(candidateScoresMap.size)\n\n    val normFn: (Long, (Double, Double)) => (Long, Double) = config.annAlgorithm match {\n      case ScoringAlgorithm.LogCosineSimilarity =>\n        (candidateId: Long, score: (Double, Double)) =>\n          candidateId -> score._1 / sourceEmbedding.logNorm / math.log(1 + score._2)\n      case ScoringAlgorithm.CosineSimilarity =>\n        (candidateId: Long, score: (Double, Double)) =>\n          candidateId -> score._1 / sourceEmbedding.l2norm / math.sqrt(score._2)\n      case ScoringAlgorithm.CosineSimilarityNoSourceEmbeddingNormalization =>\n        (candidateId: Long, score: (Double, Double)) =>\n          candidateId -> score._1 / math.sqrt(score._2)\n      case ScoringAlgorithm.DotProduct =>\n        (candidateId: Long, score: (Double, Double)) => (candidateId, score._1)\n    }\n\n    val scoredTweets: java.util.ArrayList[(Long, Double)] =\n      new java.util.ArrayList(candidateScoresMap.size)\n\n    val it = candidateScoresMap.entrySet().iterator()\n    while (it.hasNext) {\n      val mapEntry = it.next()\n      val normedScore = normFn(mapEntry.getKey, mapEntry.getValue)\n      if (normedScore._2 >= config.minScore)\n        scoredTweets.add(normedScore)\n    }\n    import scala.collection.JavaConverters._\n\n    scoredTweets.asScala\n      .sortBy(-_._2)\n      .take(Math.min(config.maxNumResults, MaxNumResultsUpperBound))\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/candidate_source/SimClustersANNCandidateSource.scala",
    "content": "package com.twitter.simclustersann.candidate_source\n\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.frigate.common.base.Stats\nimport com.twitter.simclusters_v2.common.ClusterId\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.simclustersann.thriftscala.SimClustersANNConfig\nimport com.twitter.simclustersann.thriftscala.SimClustersANNTweetCandidate\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.util.Future\n\n/**\n * This store looks for tweets whose similarity is close to a Source SimClustersEmbeddingId.\n *\n * Approximate cosine similarity is the core algorithm to drive this store.\n *\n * Step 1 - 4 are in \"fetchCandidates\" method.\n * 1. Retrieve the SimClusters Embedding by the SimClustersEmbeddingId\n * 2. Fetch top N clusters' top tweets from the clusterTweetCandidatesStore (TopTweetsPerCluster index).\n * 3. Calculate all the tweet candidates' dot-product or approximate cosine similarity to source tweets.\n * 4. Take top M tweet candidates by the step 3's score\n */\ncase class SimClustersANNCandidateSource(\n  approximateCosineSimilarity: ApproximateCosineSimilarity,\n  clusterTweetCandidatesStore: ReadableStore[ClusterId, Seq[(TweetId, Double)]],\n  simClustersEmbeddingStore: ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding],\n  statsReceiver: StatsReceiver) {\n  private val stats = statsReceiver.scope(this.getClass.getName)\n  private val fetchSourceEmbeddingStat = stats.scope(\"fetchSourceEmbedding\")\n  private val fetchCandidatesStat = stats.scope(\"fetchCandidates\")\n  private val candidateScoresStat = stats.stat(\"candidateScoresMap\")\n\n  def get(\n    query: SimClustersANNCandidateSource.Query\n  ): Future[Option[Seq[SimClustersANNTweetCandidate]]] = {\n    val sourceEmbeddingId = query.sourceEmbeddingId\n    val config = query.config\n    for {\n      maybeSimClustersEmbedding <- Stats.track(fetchSourceEmbeddingStat) {\n        simClustersEmbeddingStore.get(query.sourceEmbeddingId)\n      }\n      maybeFilteredCandidates <- maybeSimClustersEmbedding match {\n        case Some(sourceEmbedding) =>\n          for {\n            candidates <- Stats.trackSeq(fetchCandidatesStat) {\n              fetchCandidates(sourceEmbeddingId, sourceEmbedding, config)\n            }\n          } yield {\n            fetchCandidatesStat\n              .stat(sourceEmbeddingId.embeddingType.name, sourceEmbeddingId.modelVersion.name).add(\n                candidates.size)\n            Some(candidates)\n          }\n        case None =>\n          fetchCandidatesStat\n            .stat(sourceEmbeddingId.embeddingType.name, sourceEmbeddingId.modelVersion.name).add(0)\n          Future.None\n      }\n    } yield {\n      maybeFilteredCandidates\n    }\n  }\n\n  private def fetchCandidates(\n    sourceEmbeddingId: SimClustersEmbeddingId,\n    sourceEmbedding: SimClustersEmbedding,\n    config: SimClustersANNConfig\n  ): Future[Seq[SimClustersANNTweetCandidate]] = {\n\n    val clusterIds =\n      sourceEmbedding\n        .truncate(config.maxScanClusters).getClusterIds()\n        .toSet\n\n    Future\n      .collect {\n        clusterTweetCandidatesStore.multiGet(clusterIds)\n      }.map { clusterTweetsMap =>\n        approximateCosineSimilarity(\n          sourceEmbedding = sourceEmbedding,\n          sourceEmbeddingId = sourceEmbeddingId,\n          config = config,\n          candidateScoresStat = (i: Int) => candidateScoresStat.add(i),\n          clusterTweetsMap = clusterTweetsMap\n        ).map {\n          case (tweetId, score) =>\n            SimClustersANNTweetCandidate(\n              tweetId = tweetId,\n              score = score\n            )\n        }\n      }\n  }\n}\n\nobject SimClustersANNCandidateSource {\n  case class Query(\n    sourceEmbeddingId: SimClustersEmbeddingId,\n    config: SimClustersANNConfig)\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/common/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [],\n)\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/common/FlagNames.scala",
    "content": "package com.twitter.simclustersann.common\n\nobject FlagNames {\n\n  /**\n   * Global Settings\n   */\n  final val ServiceTimeout = \"service.timeout\"\n  final val DarkTrafficFilterDeciderKey = \"thrift.dark.traffic.filter.decider_key\"\n\n  /**\n   * Cache Setting\n   */\n  final val CacheDest = \"cache_module.dest\"\n  final val CacheTimeout = \"cache_module.timeout\"\n  // Only turn on the async update when the SANN Cluster has the production taffic.\n  final val CacheAsyncUpdate = \"cache_module.async_update\"\n\n  /**\n   * Warmup Settings\n   */\n  final val DisableWarmup = \"warmup.disable\"\n  final val NumberOfThreads = \"warmup.thread_number\"\n  final val RateLimiterQPS = \"warmup.rate_limiter_qps\"\n\n  /**\n   * Algorithm Parameters\n   */\n  final val MaxTopTweetPerCluster = \"sim_clusters.ann.max_top_tweets_per_cluster\"\n\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/controllers/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/javax/inject:javax.inject\",\n        \"3rdparty/jvm/net/codingwell:scala-guice\",\n        \"decider/src/main/scala\",\n        \"finagle/finagle-core/src/main\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift:controller\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/exceptions\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/filters\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/modules\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/response\",\n        \"finatra/thrift/src/main/scala/com/twitter/finatra/thrift/routing\",\n        \"representation-manager/server/src/main/scala/com/twitter/representation_manager/migration\",\n        \"scrooge/scrooge-core/src/main/scala\",\n        \"simclusters-ann/server/src/main/scala/com/twitter/simclustersann/candidate_source\",\n        \"simclusters-ann/server/src/main/scala/com/twitter/simclustersann/common\",\n        \"simclusters-ann/server/src/main/scala/com/twitter/simclustersann/filters\",\n        \"simclusters-ann/thrift/src/main/thrift:thrift-scala\",\n        \"src/scala/com/twitter/simclusters_v2/candidate_source\",\n        \"twitter-server/server/src/main/scala\",\n        \"util/util-core:scala\",\n        \"util/util-slf4j-api/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/controllers/SimClustersANNController.scala",
    "content": "package com.twitter.simclustersann.controllers\n\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.finatra.thrift.Controller\nimport com.twitter.simclustersann.thriftscala.SimClustersANNService.GetTweetCandidates\nimport com.twitter.simclustersann.thriftscala.SimClustersANNService\nimport com.twitter.simclustersann.thriftscala.Query\nimport com.twitter.simclustersann.thriftscala.SimClustersANNTweetCandidate\nimport com.twitter.scrooge.Request\nimport com.twitter.scrooge.Response\nimport javax.inject.Inject\nimport com.twitter.finagle.Service\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.simclustersann.candidate_source.{\n  SimClustersANNCandidateSource => SANNSimClustersANNCandidateSource\n}\nimport com.twitter.simclustersann.common.FlagNames\nimport com.twitter.simclustersann.filters.GetTweetCandidatesResponseStatsFilter\nimport com.twitter.simclustersann.filters.SimClustersAnnVariantFilter\nimport com.twitter.util.Future\nimport com.twitter.util.JavaTimer\nimport com.twitter.util.Timer\n\nclass SimClustersANNController @Inject() (\n  @Flag(FlagNames.ServiceTimeout) serviceTimeout: Int,\n  variantFilter: SimClustersAnnVariantFilter,\n  getTweetCandidatesResponseStatsFilter: GetTweetCandidatesResponseStatsFilter,\n  sannCandidateSource: SANNSimClustersANNCandidateSource,\n  globalStats: StatsReceiver)\n    extends Controller(SimClustersANNService) {\n\n  import SimClustersANNController._\n\n  private val stats: StatsReceiver = globalStats.scope(this.getClass.getCanonicalName)\n  private val timer: Timer = new JavaTimer(true)\n\n  val filteredService: Service[Request[GetTweetCandidates.Args], Response[\n    Seq[SimClustersANNTweetCandidate]\n  ]] = {\n    variantFilter\n      .andThen(getTweetCandidatesResponseStatsFilter)\n      .andThen(Service.mk(handler))\n  }\n\n  handle(GetTweetCandidates).withService(filteredService)\n\n  private def handler(\n    request: Request[GetTweetCandidates.Args]\n  ): Future[Response[Seq[SimClustersANNTweetCandidate]]] = {\n    val query: Query = request.args.query\n    val simClustersANNCandidateSourceQuery = SANNSimClustersANNCandidateSource.Query(\n      sourceEmbeddingId = query.sourceEmbeddingId,\n      config = query.config\n    )\n\n    val result = sannCandidateSource\n      .get(simClustersANNCandidateSourceQuery).map {\n        case Some(tweetCandidatesSeq) =>\n          Response(tweetCandidatesSeq.map { tweetCandidate =>\n            SimClustersANNTweetCandidate(\n              tweetId = tweetCandidate.tweetId,\n              score = tweetCandidate.score\n            )\n          })\n        case None =>\n          DefaultResponse\n      }\n\n    result.raiseWithin(serviceTimeout.milliseconds)(timer).rescue {\n      case e: Throwable =>\n        stats.scope(\"failures\").counter(e.getClass.getCanonicalName).incr()\n        Future.value(DefaultResponse)\n    }\n  }\n}\n\nobject SimClustersANNController {\n  val DefaultResponse: Response[Seq[SimClustersANNTweetCandidate]] = Response(Seq.empty)\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/exceptions/BUILD",
    "content": "scala_library(\n    sources = [\"*.scala\"],\n    compiler_option_sets = [\"fatal_warnings\"],\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-core/src/main\",\n        \"finatra-internal/mtls-thriftmux/src/main/scala\",\n        \"finatra-internal/thrift/src/main/thrift:thrift-scala\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/exceptions/InvalidRequestForSimClustersAnnVariantException.scala",
    "content": "package com.twitter.simclustersann.exceptions\n\nimport com.twitter.finagle.RequestException\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\n\ncase class InvalidRequestForSimClustersAnnVariantException(\n  modelVersion: ModelVersion,\n  embeddingType: EmbeddingType,\n  actualServiceName: String,\n  expectedServiceName: Option[String])\n    extends RequestException(\n      s\"Request with model version ($modelVersion) and embedding type ($embeddingType) cannot be \" +\n        s\"processed by service variant ($actualServiceName).\" +\n        s\" Expected service variant: $expectedServiceName.\",\n      null)\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/exceptions/InvalidRequestForSimClustersAnnVariantExceptionMapper.scala",
    "content": "package com.twitter.simclustersann.exceptions\n\nimport com.twitter.finatra.thrift.exceptions.ExceptionMapper\nimport com.twitter.finatra.thrift.thriftscala.ClientError\nimport com.twitter.finatra.thrift.thriftscala.ClientErrorCause\nimport com.twitter.util.Future\nimport com.twitter.util.logging.Logging\nimport javax.inject.Singleton\n\n/**\n * An exception mapper designed to handle\n * [[com.twitter.simclustersann.exceptions.InvalidRequestForSimClustersAnnVariantException]]\n * by returning a Thrift IDL defined Client Error.\n */\n@Singleton\nclass InvalidRequestForSimClustersAnnVariantExceptionMapper\n    extends ExceptionMapper[InvalidRequestForSimClustersAnnVariantException, Nothing]\n    with Logging {\n\n  override def handleException(\n    throwable: InvalidRequestForSimClustersAnnVariantException\n  ): Future[Nothing] = {\n    error(\"Invalid Request For SimClusters Ann Variant Exception\", throwable)\n\n    Future.exception(ClientError(ClientErrorCause.BadRequest, throwable.getMessage()))\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/exceptions/MissingClusterConfigForSimClustersAnnVariantException.scala",
    "content": "package com.twitter.simclustersann.exceptions\n\ncase class MissingClusterConfigForSimClustersAnnVariantException(sannServiceName: String)\n    extends IllegalStateException(\n      s\"No cluster configuration found for service ($sannServiceName)\",\n      null)\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/filters/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle/finagle-core/src/main\",\n        \"finatra/inject/inject-app/src/main/java/com/twitter/inject/annotations\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/simclustersann/multicluster\",\n        \"scrooge/scrooge-core/src/main/scala\",\n        \"simclusters-ann/server/src/main/scala/com/twitter/simclustersann/exceptions\",\n        \"simclusters-ann/thrift/src/main/thrift:thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/filters/GetTweetCandidatesResponseStatsFilter.scala",
    "content": "package com.twitter.simclustersann.filters\n\nimport com.twitter.finagle.Service\nimport com.twitter.finagle.SimpleFilter\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.scrooge.Request\nimport com.twitter.scrooge.Response\nimport com.twitter.simclustersann.thriftscala.SimClustersANNService\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass GetTweetCandidatesResponseStatsFilter @Inject() (\n  statsReceiver: StatsReceiver)\n    extends SimpleFilter[Request[SimClustersANNService.GetTweetCandidates.Args], Response[\n      SimClustersANNService.GetTweetCandidates.SuccessType\n    ]] {\n\n  private[this] val stats = statsReceiver.scope(\"method_response_stats\").scope(\"getTweetCandidates\")\n  private[this] val candidateScoreStats = stats.stat(\"candidate_score_x1000\")\n  private[this] val emptyResponseCounter = stats.counter(\"empty\")\n  private[this] val nonEmptyResponseCounter = stats.counter(\"non_empty\")\n  override def apply(\n    request: Request[SimClustersANNService.GetTweetCandidates.Args],\n    service: Service[Request[SimClustersANNService.GetTweetCandidates.Args], Response[\n      SimClustersANNService.GetTweetCandidates.SuccessType\n    ]]\n  ): Future[Response[SimClustersANNService.GetTweetCandidates.SuccessType]] = {\n    val response = service(request)\n\n    response.onSuccess { successResponse =>\n      if (successResponse.value.size == 0)\n        emptyResponseCounter.incr()\n      else\n        nonEmptyResponseCounter.incr()\n      successResponse.value.foreach { candidate =>\n        candidateScoreStats.add(candidate.score.toFloat * 1000)\n      }\n    }\n    response\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/filters/SimClustersAnnVariantFilter.scala",
    "content": "package com.twitter.simclustersann.filters\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.Service\nimport com.twitter.finagle.SimpleFilter\nimport com.twitter.relevance_platform.simclustersann.multicluster.ServiceNameMapper\nimport com.twitter.scrooge.Request\nimport com.twitter.scrooge.Response\nimport com.twitter.simclustersann.exceptions.InvalidRequestForSimClustersAnnVariantException\nimport com.twitter.simclustersann.thriftscala.SimClustersANNService\nimport com.twitter.util.Future\nimport javax.inject.Inject\nimport javax.inject.Singleton\n\n@Singleton\nclass SimClustersAnnVariantFilter @Inject() (\n  serviceNameMapper: ServiceNameMapper,\n  serviceIdentifier: ServiceIdentifier,\n) extends SimpleFilter[Request[SimClustersANNService.GetTweetCandidates.Args], Response[\n      SimClustersANNService.GetTweetCandidates.SuccessType\n    ]] {\n  override def apply(\n    request: Request[SimClustersANNService.GetTweetCandidates.Args],\n    service: Service[Request[SimClustersANNService.GetTweetCandidates.Args], Response[\n      SimClustersANNService.GetTweetCandidates.SuccessType\n    ]]\n  ): Future[Response[SimClustersANNService.GetTweetCandidates.SuccessType]] = {\n\n    validateRequest(request)\n    service(request)\n  }\n\n  private def validateRequest(\n    request: Request[SimClustersANNService.GetTweetCandidates.Args]\n  ): Unit = {\n    val modelVersion = request.args.query.sourceEmbeddingId.modelVersion\n    val embeddingType = request.args.query.config.candidateEmbeddingType\n\n    val actualServiceName = serviceIdentifier.service\n\n    val expectedServiceName = serviceNameMapper.getServiceName(modelVersion, embeddingType)\n\n    expectedServiceName match {\n      case Some(name) if name == actualServiceName => ()\n      case _ =>\n        throw InvalidRequestForSimClustersAnnVariantException(\n          modelVersion,\n          embeddingType,\n          actualServiceName,\n          expectedServiceName)\n    }\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/BUILD",
    "content": "scala_library(\n    compiler_option_sets = [\"fatal_warnings\"],\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finagle/finagle-stats\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"frigate/frigate-common/src/main/scala/com/twitter/frigate/common/store/strato\",\n        \"hermit/hermit-core/src/main/scala/com/twitter/hermit/store/common\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/injection\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/common/readablestore\",\n        \"relevance-platform/src/main/scala/com/twitter/relevance_platform/simclustersann/multicluster\",\n        \"representation-manager/client/src/main/scala/com/twitter/representation_manager\",\n        \"simclusters-ann/server/src/main/scala/com/twitter/simclustersann/candidate_source\",\n        \"simclusters-ann/server/src/main/scala/com/twitter/simclustersann/common\",\n        \"simclusters-ann/server/src/main/scala/com/twitter/simclustersann/exceptions\",\n        \"simclusters-ann/thrift/src/main/thrift:thrift-scala\",\n        \"src/scala/com/twitter/simclusters_v2/common\",\n        \"src/scala/com/twitter/simclusters_v2/summingbird\",\n        \"src/scala/com/twitter/storehaus_internal/memcache\",\n        \"src/scala/com/twitter/storehaus_internal/util\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift-scala\",\n    ],\n)\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/CacheModule.scala",
    "content": "package com.twitter.simclustersann.modules\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.memcached.Client\nimport javax.inject.Singleton\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.inject.TwitterModule\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.simclustersann.common.FlagNames\nimport com.twitter.storehaus_internal.memcache.MemcacheStore\nimport com.twitter.storehaus_internal.util.ClientName\nimport com.twitter.storehaus_internal.util.ZkEndPoint\n\nobject CacheModule extends TwitterModule {\n\n  @Singleton\n  @Provides\n  def providesCache(\n    @Flag(FlagNames.CacheDest) cacheDest: String,\n    @Flag(FlagNames.CacheTimeout) cacheTimeout: Int,\n    serviceIdentifier: ServiceIdentifier,\n    stats: StatsReceiver\n  ): Client =\n    MemcacheStore.memcachedClient(\n      name = ClientName(\"memcache_simclusters_ann\"),\n      dest = ZkEndPoint(cacheDest),\n      timeout = cacheTimeout.milliseconds,\n      retries = 0,\n      statsReceiver = stats.scope(\"cache_client\"),\n      serviceIdentifier = serviceIdentifier\n    )\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/ClusterConfigMapperModule.scala",
    "content": "package com.twitter.simclustersann.modules\n\nimport com.google.inject.Provides\nimport com.twitter.inject.TwitterModule\nimport com.twitter.relevance_platform.simclustersann.multicluster.ClusterConfigMapper\nimport javax.inject.Singleton\n\nobject ClusterConfigMapperModule extends TwitterModule {\n  @Singleton\n  @Provides\n  def providesClusterConfigMapper(\n  ): ClusterConfigMapper = {\n    ClusterConfigMapper\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/ClusterConfigModule.scala",
    "content": "package com.twitter.simclustersann.modules\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.inject.TwitterModule\nimport com.twitter.relevance_platform.simclustersann.multicluster.ClusterConfig\nimport com.twitter.relevance_platform.simclustersann.multicluster.ClusterConfigMapper\nimport com.twitter.simclustersann.exceptions.MissingClusterConfigForSimClustersAnnVariantException\nimport javax.inject.Singleton\n\nobject ClusterConfigModule extends TwitterModule {\n  @Singleton\n  @Provides\n  def providesClusterConfig(\n    serviceIdentifier: ServiceIdentifier,\n    clusterConfigMapper: ClusterConfigMapper\n  ): ClusterConfig = {\n    val serviceName = serviceIdentifier.service\n\n    clusterConfigMapper.getClusterConfig(serviceName) match {\n      case Some(config) => config\n      case None => throw MissingClusterConfigForSimClustersAnnVariantException(serviceName)\n    }\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/ClusterTweetIndexProviderModule.scala",
    "content": "package com.twitter.simclustersann.modules\n\nimport com.google.inject.Provides\nimport com.twitter.conversions.DurationOps._\nimport com.twitter.decider.Decider\nimport com.twitter.finagle.memcached.Client\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.hermit.store.common.ObservedCachedReadableStore\nimport com.twitter.hermit.store.common.ObservedMemcachedReadableStore\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.relevance_platform.common.injection.LZ4Injection\nimport com.twitter.relevance_platform.common.injection.SeqObjectInjection\nimport com.twitter.relevance_platform.simclustersann.multicluster.ClusterConfig\nimport com.twitter.relevance_platform.simclustersann.multicluster.ClusterTweetIndexStoreConfig\nimport com.twitter.simclusters_v2.common.ClusterId\nimport com.twitter.simclusters_v2.common.ModelVersions\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.summingbird.stores.ClusterKey\nimport com.twitter.simclusters_v2.summingbird.stores.TopKTweetsForClusterKeyReadableStore\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclustersann.common.FlagNames\nimport com.twitter.storehaus.ReadableStore\n\nimport javax.inject.Singleton\n\nobject ClusterTweetIndexProviderModule extends TwitterModule {\n\n  @Singleton\n  @Provides\n  // Provides ClusterTweetIndex Store based on different maxResults settings on the same store\n  // Create a different provider if index is in a different store\n  def providesClusterTweetIndex(\n    @Flag(FlagNames.MaxTopTweetPerCluster) maxTopTweetPerCluster: Int,\n    @Flag(FlagNames.CacheAsyncUpdate) asyncUpdate: Boolean,\n    clusterConfig: ClusterConfig,\n    serviceIdentifier: ServiceIdentifier,\n    stats: StatsReceiver,\n    decider: Decider,\n    simClustersANNCacheClient: Client\n  ): ReadableStore[ClusterId, Seq[(TweetId, Double)]] = {\n    // Build the underling cluster-to-tweet store\n    val topTweetsForClusterStore = clusterConfig.clusterTweetIndexStoreConfig match {\n      // If the config returns Manhattan tweet index config, we read from a RO MH store\n      case manhattanConfig: ClusterTweetIndexStoreConfig.Manhattan =>\n        TopKTweetsForClusterKeyReadableStore.getClusterToTopKTweetsStoreFromManhattanRO(\n          maxTopTweetPerCluster,\n          manhattanConfig,\n          serviceIdentifier)\n      case memCacheConfig: ClusterTweetIndexStoreConfig.Memcached =>\n        TopKTweetsForClusterKeyReadableStore.getClusterToTopKTweetsStoreFromMemCache(\n          maxTopTweetPerCluster,\n          memCacheConfig,\n          serviceIdentifier)\n      case _ =>\n        // Bad instance\n        ReadableStore.empty\n    }\n\n    val embeddingType: EmbeddingType = clusterConfig.candidateTweetEmbeddingType\n    val modelVersion: String = ModelVersions.toKnownForModelVersion(clusterConfig.modelVersion)\n\n    val store: ReadableStore[ClusterId, Seq[(TweetId, Double)]] =\n      topTweetsForClusterStore.composeKeyMapping { id: ClusterId =>\n        ClusterKey(id, modelVersion, embeddingType)\n      }\n\n    val memcachedTopTweetsForClusterStore =\n      ObservedMemcachedReadableStore.fromCacheClient(\n        backingStore = store,\n        cacheClient = simClustersANNCacheClient,\n        ttl = 15.minutes,\n        asyncUpdate = asyncUpdate\n      )(\n        valueInjection = LZ4Injection.compose(SeqObjectInjection[(Long, Double)]()),\n        statsReceiver = stats.scope(\"cluster_tweet_index_mem_cache\"),\n        keyToString = { k =>\n          // prod cache key : SimClusters_LZ4/cluster_to_tweet/clusterId_embeddingType_modelVersion\n          s\"scz:c2t:${k}_${embeddingType}_${modelVersion}_$maxTopTweetPerCluster\"\n        }\n      )\n\n    val cachedStore: ReadableStore[ClusterId, Seq[(TweetId, Double)]] = {\n      ObservedCachedReadableStore.from[ClusterId, Seq[(TweetId, Double)]](\n        memcachedTopTweetsForClusterStore,\n        ttl = 10.minute,\n        maxKeys = 150000,\n        cacheName = \"cluster_tweet_index_cache\",\n        windowSize = 10000L\n      )(stats.scope(\"cluster_tweet_index_store\"))\n    }\n    cachedStore\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/CustomMtlsThriftWebFormsModule.scala",
    "content": "package com.twitter.simclustersann.modules\n\nimport com.twitter.finatra.mtls.thriftmux.modules.MtlsThriftWebFormsModule\nimport com.twitter.finatra.thrift.ThriftServer\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.InternalId\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.thriftwebforms.MethodOptions\nimport com.twitter.thriftwebforms.view.ServiceResponseView\nimport com.twitter.util.Future\nimport com.twitter.simclustersann.thriftscala.SimClustersANNTweetCandidate\nimport com.twitter.simclustersann.thriftscala.Query\nimport com.twitter.simclustersann.thriftscala.SimClustersANNConfig\nimport com.twitter.simclustersann.thriftscala.ScoringAlgorithm\nimport com.twitter.thriftwebforms.MethodOptions.Access\nimport scala.reflect.ClassTag\nimport com.twitter.simclustersann.thriftscala.SimClustersANNService\nimport scala.collection.mutable\n\nclass CustomMtlsThriftWebFormsModule[T: ClassTag](server: ThriftServer)\n    extends MtlsThriftWebFormsModule[T](server: ThriftServer) {\n\n  private val Nbsp = \"&nbsp;\"\n  private val LdapGroups = Seq(\"recosplat-sensitive-data-medium\", \"simclusters-ann-admins\")\n\n  override protected def methodOptions: Map[String, MethodOptions] = {\n    val tweetId = 1568796529690902529L\n    val sannDefaultQuery = SimClustersANNService.GetTweetCandidates.Args(\n      query = Query(\n        sourceEmbeddingId = SimClustersEmbeddingId(\n          embeddingType = EmbeddingType.LogFavLongestL2EmbeddingTweet,\n          modelVersion = ModelVersion.Model20m145k2020,\n          internalId = InternalId.TweetId(tweetId)\n        ),\n        config = SimClustersANNConfig(\n          maxNumResults = 10,\n          minScore = 0.0,\n          candidateEmbeddingType = EmbeddingType.LogFavBasedTweet,\n          maxTopTweetsPerCluster = 400,\n          maxScanClusters = 50,\n          maxTweetCandidateAgeHours = 24,\n          minTweetCandidateAgeHours = 0,\n          annAlgorithm = ScoringAlgorithm.CosineSimilarity\n        )\n      ))\n\n    Seq(\"getTweetCandidates\")\n      .map(\n        _ -> MethodOptions(\n          defaultRequestValue = Some(sannDefaultQuery),\n          responseRenderers = Seq(renderTimeline),\n          allowedAccessOverride = Some(Access.ByLdapGroup(LdapGroups))\n        )).toMap\n  }\n\n  val FullAccessLdapGroups: Seq[String] =\n    Seq(\n      \"recosplat-sensitive-data-medium\",\n      \"simclusters-ann-admins\",\n      \"recos-platform-admins\"\n    )\n\n  override protected def defaultMethodAccess: MethodOptions.Access = {\n    MethodOptions.Access.ByLdapGroup(FullAccessLdapGroups)\n  }\n\n  def renderTimeline(r: AnyRef): Future[ServiceResponseView] = {\n    val simClustersANNTweetCandidates = r match {\n      case response: Iterable[_] =>\n        response.map(x => x.asInstanceOf[SimClustersANNTweetCandidate]).toSeq\n      case _ => Seq()\n    }\n    renderTweets(simClustersANNTweetCandidates)\n  }\n\n  private def renderTweets(\n    simClustersANNTweetCandidates: Seq[SimClustersANNTweetCandidate]\n  ): Future[ServiceResponseView] = {\n    val htmlSb = new mutable.StringBuilder()\n    val headerHtml = s\"\"\"<h3>Tweet Candidates</h3>\"\"\"\n    val tweetsHtml = simClustersANNTweetCandidates.map { simClustersANNTweetCandidate =>\n      val tweetId = simClustersANNTweetCandidate.tweetId\n      val score = simClustersANNTweetCandidate.score\n      s\"\"\"<blockquote class=\"twitter-tweet\"><a href=\"https://twitter.com/tweet/statuses/$tweetId\"></a></blockquote> <b>score:</b> $score <br><br>\"\"\"\n    }.mkString\n\n    htmlSb ++= headerHtml\n    htmlSb ++= Nbsp\n    htmlSb ++= tweetsHtml\n    Future.value(\n      ServiceResponseView(\n        \"SimClusters ANN Tweet Candidates\",\n        htmlSb.toString(),\n        Seq(\"//platform.twitter.com/widgets.js\")\n      )\n    )\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/EmbeddingStoreModule.scala",
    "content": "package com.twitter.simclustersann.modules\n\nimport com.google.inject.Provides\nimport com.twitter.decider.Decider\nimport com.twitter.finagle.memcached.{Client => MemcachedClient}\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.representation_manager.StoreBuilder\nimport com.twitter.representation_manager.config.{\n  DefaultClientConfig => RepresentationManagerDefaultClientConfig\n}\nimport com.twitter.representation_manager.thriftscala.SimClustersEmbeddingView\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.stores.SimClustersEmbeddingStore\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType\nimport com.twitter.simclusters_v2.thriftscala.EmbeddingType._\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion\nimport com.twitter.simclusters_v2.thriftscala.ModelVersion._\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.storehaus.ReadableStore\nimport com.twitter.strato.client.{Client => StratoClient}\nimport javax.inject.Singleton\n\nobject EmbeddingStoreModule extends TwitterModule {\n\n  val TweetEmbeddings: Set[SimClustersEmbeddingView] = Set(\n    SimClustersEmbeddingView(LogFavLongestL2EmbeddingTweet, Model20m145kUpdated),\n    SimClustersEmbeddingView(LogFavLongestL2EmbeddingTweet, Model20m145k2020)\n  )\n\n  val UserEmbeddings: Set[SimClustersEmbeddingView] = Set(\n    // KnownFor\n    SimClustersEmbeddingView(FavBasedProducer, Model20m145kUpdated),\n    SimClustersEmbeddingView(FavBasedProducer, Model20m145k2020),\n    SimClustersEmbeddingView(FollowBasedProducer, Model20m145k2020),\n    SimClustersEmbeddingView(AggregatableLogFavBasedProducer, Model20m145k2020),\n    // InterestedIn\n    SimClustersEmbeddingView(UnfilteredUserInterestedIn, Model20m145k2020),\n    SimClustersEmbeddingView(\n      LogFavBasedUserInterestedMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020),\n    SimClustersEmbeddingView(\n      LogFavBasedUserInterestedAverageAddressBookFromIIAPE,\n      Model20m145k2020),\n    SimClustersEmbeddingView(\n      LogFavBasedUserInterestedBooktypeMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020),\n    SimClustersEmbeddingView(\n      LogFavBasedUserInterestedLargestDimMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020),\n    SimClustersEmbeddingView(\n      LogFavBasedUserInterestedLouvainMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020),\n    SimClustersEmbeddingView(\n      LogFavBasedUserInterestedConnectedMaxpoolingAddressBookFromIIAPE,\n      Model20m145k2020),\n    SimClustersEmbeddingView(UserNextInterestedIn, Model20m145k2020),\n    SimClustersEmbeddingView(LogFavBasedUserInterestedInFromAPE, Model20m145k2020)\n  )\n\n  @Singleton\n  @Provides\n  def providesEmbeddingStore(\n    stratoClient: StratoClient,\n    memCachedClient: MemcachedClient,\n    decider: Decider,\n    stats: StatsReceiver\n  ): ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding] = {\n\n    val rmsStoreBuilder = new StoreBuilder(\n      clientConfig = RepresentationManagerDefaultClientConfig,\n      stratoClient = stratoClient,\n      memCachedClient = memCachedClient,\n      globalStats = stats,\n    )\n\n    val underlyingStores: Map[\n      (EmbeddingType, ModelVersion),\n      ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding]\n    ] = {\n      val tweetEmbeddingStores: Map[\n        (EmbeddingType, ModelVersion),\n        ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding]\n      ] = TweetEmbeddings\n        .map(embeddingView =>\n          (\n            (embeddingView.embeddingType, embeddingView.modelVersion),\n            rmsStoreBuilder\n              .buildSimclustersTweetEmbeddingStoreWithEmbeddingIdAsKey(embeddingView))).toMap\n\n      val userEmbeddingStores: Map[\n        (EmbeddingType, ModelVersion),\n        ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding]\n      ] = UserEmbeddings\n        .map(embeddingView =>\n          (\n            (embeddingView.embeddingType, embeddingView.modelVersion),\n            rmsStoreBuilder\n              .buildSimclustersUserEmbeddingStoreWithEmbeddingIdAsKey(embeddingView))).toMap\n\n      tweetEmbeddingStores ++ userEmbeddingStores\n    }\n\n    SimClustersEmbeddingStore.buildWithDecider(\n      underlyingStores = underlyingStores,\n      decider = decider,\n      statsReceiver = stats\n    )\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/FlagsModule.scala",
    "content": "package com.twitter.simclustersann.modules\n\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclustersann.common.FlagNames\n\nobject FlagsModule extends TwitterModule {\n\n  flag[Int](\n    name = FlagNames.ServiceTimeout,\n    default = 40,\n    help = \"The threshold of Request Timeout\"\n  )\n\n  flag[String](\n    name = FlagNames.DarkTrafficFilterDeciderKey,\n    default = \"dark_traffic_filter\",\n    help = \"Dark traffic filter decider key\"\n  )\n\n  flag[String](\n    name = FlagNames.CacheDest,\n    default = \"/s/cache/content_recommender_unified_v2\",\n    help = \"Path to memcache service. Currently using CR uniform scoring cache\"\n  )\n\n  flag[Int](\n    name = FlagNames.CacheTimeout,\n    default = 15,\n    help = \"The threshold of MemCache Timeout\"\n  )\n\n  flag[Boolean](\n    name = FlagNames.CacheAsyncUpdate,\n    default = false,\n    help = \"Whether to enable the async update for the MemCache\"\n  )\n\n  flag[Int](\n    name = FlagNames.MaxTopTweetPerCluster,\n    default = 200,\n    help = \"Maximum number of tweets to take per each simclusters\"\n  )\n\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/FuturePoolProvider.scala",
    "content": "package com.twitter.simclustersann.modules\n\nimport com.google.inject.Provides\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.simclustersann.common.FlagNames.NumberOfThreads\nimport com.twitter.util.ExecutorServiceFuturePool\nimport java.util.concurrent.Executors\nimport javax.inject.Singleton\nobject FuturePoolProvider extends TwitterModule {\n  flag[Int](\n    name = NumberOfThreads,\n    default = 20,\n    help = \"The number of threads in the future pool.\"\n  )\n\n  @Singleton\n  @Provides\n  def providesFuturePool(\n    @Flag(NumberOfThreads) numberOfThreads: Int\n  ): ExecutorServiceFuturePool = {\n    val threadPool = Executors.newFixedThreadPool(numberOfThreads)\n    new ExecutorServiceFuturePool(threadPool) {\n      override def toString: String = s\"warmup-future-pool-$executor)\"\n    }\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/RateLimiterModule.scala",
    "content": "package com.twitter.simclustersann.modules\n\nimport com.google.common.util.concurrent.RateLimiter\nimport com.google.inject.Provides\nimport com.twitter.inject.TwitterModule\nimport com.twitter.inject.annotations.Flag\nimport com.twitter.simclustersann.common.FlagNames.RateLimiterQPS\nimport javax.inject.Singleton\n\nobject RateLimiterModule extends TwitterModule {\n  flag[Int](\n    name = RateLimiterQPS,\n    default = 1000,\n    help = \"The QPS allowed by the rate limiter.\"\n  )\n\n  @Singleton\n  @Provides\n  def providesRateLimiter(\n    @Flag(RateLimiterQPS) rateLimiterQps: Int\n  ): RateLimiter =\n    RateLimiter.create(rateLimiterQps)\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/ServiceNameMapperModule.scala",
    "content": "package com.twitter.simclustersann.modules\n\nimport com.google.inject.Provides\nimport com.twitter.inject.TwitterModule\nimport com.twitter.relevance_platform.simclustersann.multicluster.ServiceNameMapper\nimport javax.inject.Singleton\n\nobject ServiceNameMapperModule extends TwitterModule {\n  @Singleton\n  @Provides\n  def providesServiceNameMapper(\n  ): ServiceNameMapper = {\n    ServiceNameMapper\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/SimClustersANNCandidateSourceModule.scala",
    "content": "package com.twitter.simclustersann.modules\n\nimport com.google.inject.Provides\nimport com.twitter.finagle.stats.StatsReceiver\nimport com.twitter.inject.TwitterModule\nimport com.twitter.simclusters_v2.common.ClusterId\nimport com.twitter.simclusters_v2.common.SimClustersEmbedding\nimport com.twitter.simclusters_v2.common.TweetId\nimport com.twitter.simclusters_v2.thriftscala.SimClustersEmbeddingId\nimport com.twitter.storehaus.ReadableStore\nimport javax.inject.Singleton\nimport com.twitter.simclustersann.candidate_source.ApproximateCosineSimilarity\nimport com.twitter.simclustersann.candidate_source.ExperimentalApproximateCosineSimilarity\nimport com.twitter.simclustersann.candidate_source.OptimizedApproximateCosineSimilarity\nimport com.twitter.simclustersann.candidate_source.SimClustersANNCandidateSource\n\nobject SimClustersANNCandidateSourceModule extends TwitterModule {\n\n  val acsFlag = flag[String](\n    name = \"approximate_cosine_similarity\",\n    default = \"original\",\n    help =\n      \"Select different implementations of the approximate cosine similarity algorithm, for testing optimizations\",\n  )\n  @Singleton\n  @Provides\n  def provides(\n    embeddingStore: ReadableStore[SimClustersEmbeddingId, SimClustersEmbedding],\n    cachedClusterTweetIndexStore: ReadableStore[ClusterId, Seq[(TweetId, Double)]],\n    statsReceiver: StatsReceiver\n  ): SimClustersANNCandidateSource = {\n\n    val approximateCosineSimilarity = acsFlag() match {\n      case \"original\" => ApproximateCosineSimilarity\n      case \"optimized\" => OptimizedApproximateCosineSimilarity\n      case \"experimental\" => ExperimentalApproximateCosineSimilarity\n      case _ => ApproximateCosineSimilarity\n    }\n\n    new SimClustersANNCandidateSource(\n      approximateCosineSimilarity = approximateCosineSimilarity,\n      clusterTweetCandidatesStore = cachedClusterTweetIndexStore,\n      simClustersEmbeddingStore = embeddingStore,\n      statsReceiver = statsReceiver.scope(\"simClustersANNCandidateSource\")\n    )\n  }\n}\n"
  },
  {
    "path": "simclusters-ann/server/src/main/scala/com/twitter/simclustersann/modules/StratoClientProviderModule.scala",
    "content": "package com.twitter.simclustersann.modules\n\nimport com.google.inject.Provides\nimport javax.inject.Singleton\nimport com.twitter.inject.TwitterModule\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier\nimport com.twitter.strato.client.Client\nimport com.twitter.strato.client.Strato\n\nobject StratoClientProviderModule extends TwitterModule {\n\n  @Singleton\n  @Provides\n  def providesCache(\n    serviceIdentifier: ServiceIdentifier,\n  ): Client = Strato.client\n    .withMutualTls(serviceIdentifier)\n    .build()\n\n}\n"
  },
  {
    "path": "simclusters-ann/thrift/src/main/thrift/BUILD",
    "content": "create_thrift_libraries(\n    base_name = \"thrift\",\n    sources = [\"**/*.thrift\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependency_roots = [\n        \"finatra-internal/thrift/src/main/thrift\",\n        \"src/thrift/com/twitter/simclusters_v2:simclusters_v2-thrift\",\n    ],\n    generate_languages = [\n        \"java\",\n        \"scala\",\n    ],\n    provides_java_name = \"simclusters-ann-thrift-java\",\n    provides_scala_name = \"simclusters-ann-thrift-scala\",\n)\n"
  },
  {
    "path": "simclusters-ann/thrift/src/main/thrift/simClustersAnn.thrift",
    "content": "namespace java com.twitter.simclustersann.thriftjava\n#@namespace scala com.twitter.simclustersann.thriftscala\n\ninclude \"finatra-thrift/finatra_thrift_exceptions.thrift\"\ninclude \"com/twitter/simclusters_v2/identifier.thrift\"\ninclude \"com/twitter/simclusters_v2/score.thrift\"\n\nstruct Query {\n    1: required identifier.SimClustersEmbeddingId sourceEmbeddingId;\n    2: required SimClustersANNConfig config;\n}\n\nstruct SimClustersANNTweetCandidate {\n    1: required i64 tweetId (personalDataType = 'TweetId');\n    2: required double score;\n}\n\nstruct SimClustersANNConfig {\n    1: required i32 maxNumResults;\n    2: required double minScore;\n    3: required identifier.EmbeddingType candidateEmbeddingType;\n    4: required i32 maxTopTweetsPerCluster;\n    5: required i32 maxScanClusters;\n    6: required i32 maxTweetCandidateAgeHours;\n    7: required i32 minTweetCandidateAgeHours;\n    8: required ScoringAlgorithm annAlgorithm;\n}\n\n/**\n  * The algorithm type to identify the score algorithm.\n  **/\nenum ScoringAlgorithm {\n\tDotProduct = 1,\n\tCosineSimilarity = 2,\n  LogCosineSimilarity = 3,\n  CosineSimilarityNoSourceEmbeddingNormalization = 4,  // Score = (Source dot Candidate) / candidate_l2_norm\n}(hasPersonalData = 'false')\n\nenum InvalidResponseParameter {\n\tINVALID_EMBEDDING_TYPE = 1,\n\tINVALID_MODEL_VERSION = 2,\n}\n\nexception InvalidResponseParameterException {\n\t1: required InvalidResponseParameter errorCode,\n\t2: optional string message // failure reason\n}\n\nservice SimClustersANNService {\n\n    list<SimClustersANNTweetCandidate> getTweetCandidates(\n        1: required Query query;\n    ) throws (\n      1: InvalidResponseParameterException e;\n      2: finatra_thrift_exceptions.ServerError serverError;\n      3: finatra_thrift_exceptions.ClientError clientError;\n    );\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/README.md",
    "content": "# Tweet Search System (Earlybird)\n> **TL;DR** Tweet Search System (Earlybird) find tweets from people you follow, rank them, and serve the tweets to Home.\n\n## What is Tweet Search System (Earlybird)? \n[Earlybird](http://notes.stephenholiday.com/Earlybird.pdf) is a **real-time search system** based on [Apache Lucene](https://lucene.apache.org/) to support the high volume of queries and content updates. The major use cases are Relevance Search (specifically, Text search) and Timeline In-network Tweet retrieval (or UserID based search). It is designed to enable the efficient indexing and querying of billions of tweets, and to provide low-latency search results, even with heavy query loads.\n\n## How it is related to the Home Timeline Recommendation Algorithm\n\n![in-network](img/in-network.png)\n\nAt Twitter, we use Tweet Search System (Earlybird) to do Home Timeline In-network Tweet retrieval: given a list of following users, find their recently posted tweets. Earlybird (Search Index) is the major candidate source for in-network tweets across Following tab and For You tab.\n\n\n## High-level architecture\nWe split our entire tweet search index into three clusters: a **realtime** cluster indexing all public tweets posted in about the last 7 days, a **protected** cluster indexing all protected tweets for the same timeframe; and an **archive** cluster indexing all tweets ever posted, up to about two days ago. \n\nEarlybird addresses the challenges of scaling real-time search by splitting each cluster across multiple **partitions**, each responsible for a portion of the index. The architecture uses a distributed *inverted index* that is sharded and replicated. This design allows for efficient index updates and query processing. \n\nThe system also employs an incremental indexing approach, enabling it to process and index new tweets in real-time as they arrive. With single writer, multiple reader structure, Earlybird can handle a large number of real-time updates and queries concurrently while maintaining low query latency. The system can achieve high query throughput and low query latency while maintaining a high degree of index freshness. \n\n\n### Indexing \n* Ingesters read tweets and user modifications from kafka topics, extract fields and features from them and write the extracted data to intermediate kafka topics for Earlybirds to consume, index and serve.\n* Feature Update Service feeds feature updates such as up-to-date engagement (like, retweets, replies) counts to Earlybird.\n![indexing](img/indexing.png)\n\n### Serving\nEarlybird roots fanout requests to different Earlybird clusters or partitions. Upon receiving responses from the clusters or partitions, roots merge the responses before finally returning the merged response to the client. \n![serving](img/serving.png)\n\n## Use cases\n\n1. Tweet Search\n  * Top search\n  * Latest search\n\n![top](img/top-search.png)\n\n2. Candidate generation\n  * Timeline (For You Tab, Following Tab)\n  * Notifications\n\n![home](img/foryou.png)\n\n## References\n* \"Earlybird: Real-Time Search at Twitter\" (http://notes.stephenholiday.com/Earlybird.pdf)\n* \"Reducing search indexing latency to one second\" (https://blog.twitter.com/engineering/en_us/topics/infrastructure/2020/reducing-search-indexing-latency-to-one-second)\n* \"Omnisearch index formats\" (https://blog.twitter.com/engineering/en_us/topics/infrastructure/2016/omnisearch-index-formats)\n\n\n"
  },
  {
    "path": "src/java/com/twitter/search/common/README.md",
    "content": "Contains code that is common to multiple earlybird services (ingesters, roots and earlybird)."
  },
  {
    "path": "src/java/com/twitter/search/common/converter/earlybird/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/twitter/elephantbird:core\",\n        \"3rdparty/jvm/geo/google:geoGoogle\",\n        \"3rdparty/jvm/joda-time\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/httpcomponents:httpcore\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"cuad/projects/ner/thrift/src/main/thrift:thrift-java\",\n        \"decider/src/main/scala\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/collections\",\n        \"src/java/com/twitter/common/text/language:locale-util\",\n        \"src/java/com/twitter/common/text/token\",\n        \"src/java/com/twitter/common/text/util:token-util\",\n        \"src/java/com/twitter/common_internal/text:text-penguin7\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/search/common/config\",\n        \"src/java/com/twitter/search/common/constants\",\n        \"src/java/com/twitter/search/common/debug\",\n        \"src/java/com/twitter/search/common/decider\",\n        \"src/java/com/twitter/search/common/encoding/docvalues\",\n        \"src/java/com/twitter/search/common/encoding/features\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/partitioning/base\",\n        \"src/java/com/twitter/search/common/partitioning/snowflakeparser\",\n        \"src/java/com/twitter/search/common/relevance:entities_and_filters\",\n        \"src/java/com/twitter/search/common/relevance:text\",\n        \"src/java/com/twitter/search/common/relevance/features\",\n        \"src/java/com/twitter/search/common/schema\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/common/util:longintconverter\",\n        \"src/java/com/twitter/search/common/util/analysis\",\n        \"src/java/com/twitter/search/common/util/lang\",\n        \"src/java/com/twitter/search/common/util/spatial\",\n        \"src/java/com/twitter/search/common/util/text\",\n        \"src/java/com/twitter/search/common/util/text/regex\",\n        \"src/java/com/twitter/search/common/util/thrift:thrift-utils\",\n        \"src/java/com/twitter/search/common/util/url\",\n        \"src/java/com/twitter/search/ingester/model\",\n        \"src/thrift/com/twitter/search/common:constants-java\",\n        \"src/thrift/com/twitter/search/common:indexing-java\",\n        \"src/thrift/com/twitter/search/common:schema-java\",\n        \"src/thrift/com/twitter/search/common/debug:debug-java\",\n        \"src/thrift/com/twitter/service/spiderduck/gen:metadata-store-java\",\n        \"src/thrift/com/twitter/tweetypie:tweet-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/converter/earlybird/BasicIndexingConverter.java",
    "content": "package com.twitter.search.common.converter.earlybird;\n\nimport java.io.IOException;\nimport java.util.Date;\nimport java.util.List;\nimport java.util.Optional;\nimport javax.annotation.concurrent.NotThreadSafe;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.commons.collections.CollectionUtils;\nimport org.joda.time.DateTime;\nimport org.joda.time.DateTimeZone;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.search.common.converter.earlybird.EncodedFeatureBuilder.TweetFeatureWithEncodeFeatures;\nimport com.twitter.search.common.indexing.thriftjava.Place;\nimport com.twitter.search.common.indexing.thriftjava.PotentialLocation;\nimport com.twitter.search.common.indexing.thriftjava.ProfileGeoEnrichment;\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.indexing.thriftjava.VersionedTweetFeatures;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.search.common.relevance.entities.GeoObject;\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\nimport com.twitter.search.common.relevance.entities.TwitterQuotedMessage;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdEncodedFeatures;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.schema.earlybird.EarlybirdThriftDocumentBuilder;\nimport com.twitter.search.common.schema.thriftjava.ThriftDocument;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEventType;\nimport com.twitter.search.common.util.spatial.GeoUtil;\nimport com.twitter.search.common.util.text.NormalizerHelper;\nimport com.twitter.tweetypie.thriftjava.ComposerSource;\n\n/**\n * Converts a TwitterMessage into a ThriftVersionedEvents. This is only responsible for data that\n * is available immediately when a Tweet is created. Some data, like URL data, isn't available\n * immediately, and so it is processed later, in the DelayedIndexingConverter and sent as an\n * update. In order to achieve this we create the document in 2 passes:\n *\n * 1. BasicIndexingConverter builds thriftVersionedEvents with the fields that do not require\n * external services.\n *\n * 2. DelayedIndexingConverter builds all the document fields depending on external services, once\n * those services have processed the relevant Tweet and we have retrieved that data.\n */\n@NotThreadSafe\npublic class BasicIndexingConverter {\n  private static final Logger LOG = LoggerFactory.getLogger(BasicIndexingConverter.class);\n\n  private static final SearchCounter NUM_NULLCAST_FEATURE_FLAG_SET_TWEETS =\n      SearchCounter.export(\"num_nullcast_feature_flag_set_tweets\");\n  private static final SearchCounter NUM_NULLCAST_TWEETS =\n      SearchCounter.export(\"num_nullcast_tweets\");\n  private static final SearchCounter NUM_NON_NULLCAST_TWEETS =\n      SearchCounter.export(\"num_non_nullcast_tweets\");\n  private static final SearchCounter ADJUSTED_BAD_CREATED_AT_COUNTER =\n      SearchCounter.export(\"adjusted_incorrect_created_at_timestamp\");\n  private static final SearchCounter INCONSISTENT_TWEET_ID_AND_CREATED_AT_MS =\n      SearchCounter.export(\"inconsistent_tweet_id_and_created_at_ms\");\n  private static final SearchCounter NUM_SELF_THREAD_TWEETS =\n      SearchCounter.export(\"num_self_thread_tweets\");\n  private static final SearchCounter NUM_EXCLUSIVE_TWEETS =\n      SearchCounter.export(\"num_exclusive_tweets\");\n\n  // If a tweet carries a timestamp smaller than this timestamp, we consider the timestamp invalid,\n  // because twitter does not even exist back then before: Sun, 01 Jan 2006 00:00:00 GMT\n  private static final long VALID_CREATION_TIME_THRESHOLD_MILLIS =\n      new DateTime(2006, 1, 1, 0, 0, 0, DateTimeZone.UTC).getMillis();\n\n  private final EncodedFeatureBuilder featureBuilder;\n  private final Schema schema;\n  private final EarlybirdCluster cluster;\n\n  public BasicIndexingConverter(Schema schema, EarlybirdCluster cluster) {\n    this.featureBuilder = new EncodedFeatureBuilder();\n    this.schema = schema;\n    this.cluster = cluster;\n  }\n\n  /**\n   * This function converts TwitterMessage to ThriftVersionedEvents, which is a generic data\n   * structure that can be consumed by Earlybird directly.\n   */\n  public ThriftVersionedEvents convertMessageToThrift(\n      TwitterMessage message,\n      boolean strict,\n      List<PenguinVersion> penguinVersions) throws IOException {\n    Preconditions.checkNotNull(message);\n    Preconditions.checkNotNull(penguinVersions);\n\n    ThriftVersionedEvents versionedEvents = new ThriftVersionedEvents()\n        .setId(message.getId());\n\n    ImmutableSchemaInterface schemaSnapshot = schema.getSchemaSnapshot();\n\n    for (PenguinVersion penguinVersion : penguinVersions) {\n      ThriftDocument document =\n          buildDocumentForPenguinVersion(schemaSnapshot, message, strict, penguinVersion);\n\n      ThriftIndexingEvent thriftIndexingEvent = new ThriftIndexingEvent()\n          .setDocument(document)\n          .setEventType(ThriftIndexingEventType.INSERT)\n          .setSortId(message.getId());\n      message.getFromUserTwitterId().map(thriftIndexingEvent::setUid);\n      versionedEvents.putToVersionedEvents(penguinVersion.getByteValue(), thriftIndexingEvent);\n    }\n\n    return versionedEvents;\n  }\n\n  private ThriftDocument buildDocumentForPenguinVersion(\n      ImmutableSchemaInterface schemaSnapshot,\n      TwitterMessage message,\n      boolean strict,\n      PenguinVersion penguinVersion) throws IOException {\n    TweetFeatureWithEncodeFeatures tweetFeature =\n        featureBuilder.createTweetFeaturesFromTwitterMessage(\n            message, penguinVersion, schemaSnapshot);\n\n    EarlybirdThriftDocumentBuilder builder =\n        buildBasicFields(message, schemaSnapshot, cluster, tweetFeature);\n\n    buildUserFields(builder, message, tweetFeature.versionedFeatures, penguinVersion);\n    buildGeoFields(builder, message, tweetFeature.versionedFeatures);\n    buildRetweetAndReplyFields(builder, message, strict);\n    buildQuotesFields(builder, message);\n    buildVersionedFeatureFields(builder, tweetFeature.versionedFeatures);\n    buildAnnotationFields(builder, message);\n    buildNormalizedMinEngagementFields(builder, tweetFeature.encodedFeatures, cluster);\n    buildDirectedAtFields(builder, message);\n\n    builder.withSpaceIdFields(message.getSpaceIds());\n\n    return builder.build();\n  }\n\n  /**\n   * Build the basic fields for a tweet.\n   */\n  public static EarlybirdThriftDocumentBuilder buildBasicFields(\n      TwitterMessage message,\n      ImmutableSchemaInterface schemaSnapshot,\n      EarlybirdCluster cluster,\n      TweetFeatureWithEncodeFeatures tweetFeature) {\n    EarlybirdEncodedFeatures extendedEncodedFeatures = tweetFeature.extendedEncodedFeatures;\n    if (extendedEncodedFeatures == null && EarlybirdCluster.isTwitterMemoryFormatCluster(cluster)) {\n      extendedEncodedFeatures = EarlybirdEncodedFeatures.newEncodedTweetFeatures(\n          schemaSnapshot, EarlybirdFieldConstant.EXTENDED_ENCODED_TWEET_FEATURES_FIELD);\n    }\n    EarlybirdThriftDocumentBuilder builder = new EarlybirdThriftDocumentBuilder(\n        tweetFeature.encodedFeatures,\n        extendedEncodedFeatures,\n        new EarlybirdFieldConstants(),\n        schemaSnapshot);\n\n    builder.withID(message.getId());\n\n    final Date createdAt = message.getDate();\n    long createdAtMs = createdAt == null ? 0L : createdAt.getTime();\n\n    createdAtMs = fixCreatedAtTimeStampIfNecessary(message.getId(), createdAtMs);\n\n    if (createdAtMs > 0L) {\n      builder.withCreatedAt((int) (createdAtMs / 1000));\n    }\n\n    builder.withTweetSignature(tweetFeature.versionedFeatures.getTweetSignature());\n\n    if (message.getConversationId() > 0) {\n      long conversationId = message.getConversationId();\n      builder.withLongField(\n          EarlybirdFieldConstant.CONVERSATION_ID_CSF.getFieldName(), conversationId);\n      // We only index conversation ID when it is different from the tweet ID.\n      if (message.getId() != conversationId) {\n        builder.withLongField(\n            EarlybirdFieldConstant.CONVERSATION_ID_FIELD.getFieldName(), conversationId);\n      }\n    }\n\n    if (message.getComposerSource().isPresent()) {\n      ComposerSource composerSource = message.getComposerSource().get();\n      builder.withIntField(\n          EarlybirdFieldConstant.COMPOSER_SOURCE.getFieldName(), composerSource.getValue());\n      if (composerSource == ComposerSource.CAMERA) {\n        builder.withCameraComposerSourceFlag();\n      }\n    }\n\n    EarlybirdEncodedFeatures encodedFeatures = tweetFeature.encodedFeatures;\n    if (encodedFeatures.isFlagSet(EarlybirdFieldConstant.FROM_VERIFIED_ACCOUNT_FLAG)) {\n      builder.addFilterInternalFieldTerm(EarlybirdFieldConstant.VERIFIED_FILTER_TERM);\n    }\n    if (encodedFeatures.isFlagSet(EarlybirdFieldConstant.FROM_BLUE_VERIFIED_ACCOUNT_FLAG)) {\n      builder.addFilterInternalFieldTerm(EarlybirdFieldConstant.BLUE_VERIFIED_FILTER_TERM);\n    }\n\n    if (encodedFeatures.isFlagSet(EarlybirdFieldConstant.IS_OFFENSIVE_FLAG)) {\n      builder.withOffensiveFlag();\n    }\n\n    if (message.getNullcast()) {\n      NUM_NULLCAST_TWEETS.increment();\n      builder.addFilterInternalFieldTerm(EarlybirdFieldConstant.NULLCAST_FILTER_TERM);\n    } else {\n      NUM_NON_NULLCAST_TWEETS.increment();\n    }\n    if (encodedFeatures.isFlagSet(EarlybirdFieldConstant.IS_NULLCAST_FLAG)) {\n      NUM_NULLCAST_FEATURE_FLAG_SET_TWEETS.increment();\n    }\n    if (message.isSelfThread()) {\n      builder.addFilterInternalFieldTerm(\n          EarlybirdFieldConstant.SELF_THREAD_FILTER_TERM);\n      NUM_SELF_THREAD_TWEETS.increment();\n    }\n\n    if (message.isExclusive()) {\n      builder.addFilterInternalFieldTerm(EarlybirdFieldConstant.EXCLUSIVE_FILTER_TERM);\n      builder.withLongField(\n          EarlybirdFieldConstant.EXCLUSIVE_CONVERSATION_AUTHOR_ID_CSF.getFieldName(),\n          message.getExclusiveConversationAuthorId());\n      NUM_EXCLUSIVE_TWEETS.increment();\n    }\n\n    builder.withLanguageCodes(message.getLanguage(), message.getBCP47LanguageTag());\n\n    return builder;\n  }\n\n  /**\n   * Build the user fields.\n   */\n  public static void buildUserFields(\n      EarlybirdThriftDocumentBuilder builder,\n      TwitterMessage message,\n      VersionedTweetFeatures versionedTweetFeatures,\n      PenguinVersion penguinVersion) {\n    // 1. Set all the from user fields.\n    if (message.getFromUserTwitterId().isPresent()) {\n      builder.withLongField(EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName(),\n          message.getFromUserTwitterId().get())\n      // CSF\n      .withLongField(EarlybirdFieldConstant.FROM_USER_ID_CSF.getFieldName(),\n          message.getFromUserTwitterId().get());\n    } else {\n      LOG.warn(\"fromUserTwitterId is not set in TwitterMessage! Status id: \" + message.getId());\n    }\n\n    if (message.getFromUserScreenName().isPresent()) {\n      String fromUser = message.getFromUserScreenName().get();\n      String normalizedFromUser =\n          NormalizerHelper.normalizeWithUnknownLocale(fromUser, penguinVersion);\n\n      builder\n          .withWhiteSpaceTokenizedScreenNameField(\n              EarlybirdFieldConstant.TOKENIZED_FROM_USER_FIELD.getFieldName(),\n              normalizedFromUser)\n          .withStringField(EarlybirdFieldConstant.FROM_USER_FIELD.getFieldName(),\n              normalizedFromUser);\n\n      if (message.getTokenizedFromUserScreenName().isPresent()) {\n        builder.withCamelCaseTokenizedScreenNameField(\n            EarlybirdFieldConstant.CAMELCASE_USER_HANDLE_FIELD.getFieldName(),\n            fromUser,\n            normalizedFromUser,\n            message.getTokenizedFromUserScreenName().get());\n      }\n    }\n\n    Optional<String> toUserScreenName = message.getToUserLowercasedScreenName();\n    if (toUserScreenName.isPresent() && !toUserScreenName.get().isEmpty()) {\n      builder.withStringField(\n          EarlybirdFieldConstant.TO_USER_FIELD.getFieldName(),\n          NormalizerHelper.normalizeWithUnknownLocale(toUserScreenName.get(), penguinVersion));\n    }\n\n    if (versionedTweetFeatures.isSetUserDisplayNameTokenStreamText()) {\n      builder.withTokenStreamField(EarlybirdFieldConstant.TOKENIZED_USER_NAME_FIELD.getFieldName(),\n          versionedTweetFeatures.getUserDisplayNameTokenStreamText(),\n          versionedTweetFeatures.getUserDisplayNameTokenStream());\n    }\n  }\n\n  /**\n   * Build the geo fields.\n   */\n  public static void buildGeoFields(\n      EarlybirdThriftDocumentBuilder builder,\n      TwitterMessage message,\n      VersionedTweetFeatures versionedTweetFeatures) {\n    double lat = GeoUtil.ILLEGAL_LATLON;\n    double lon = GeoUtil.ILLEGAL_LATLON;\n    if (message.getGeoLocation() != null) {\n      GeoObject location = message.getGeoLocation();\n      builder.withGeoField(EarlybirdFieldConstant.GEO_HASH_FIELD.getFieldName(),\n          location.getLatitude(), location.getLongitude(), location.getAccuracy());\n\n      if (location.getSource() != null) {\n        builder.withStringField(EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n            EarlybirdFieldConstants.formatGeoType(location.getSource()));\n      }\n\n      if (GeoUtil.validateGeoCoordinates(location.getLatitude(), location.getLongitude())) {\n        lat = location.getLatitude();\n        lon = location.getLongitude();\n      }\n    }\n\n    // See SEARCH-14317 for investigation on how much space geo filed is used in archive cluster.\n    // In lucene archives, this CSF is needed regardless of whether geoLocation is set.\n    builder.withLatLonCSF(lat, lon);\n\n    if (versionedTweetFeatures.isSetTokenizedPlace()) {\n      Place place = versionedTweetFeatures.getTokenizedPlace();\n      Preconditions.checkArgument(place.isSetId(), \"Place ID not set for tweet \"\n          + message.getId());\n      Preconditions.checkArgument(place.isSetFullName(),\n          \"Place full name not set for tweet \" + message.getId());\n      builder.addFilterInternalFieldTerm(EarlybirdFieldConstant.PLACE_ID_FIELD.getFieldName());\n      builder\n          .withStringField(EarlybirdFieldConstant.PLACE_ID_FIELD.getFieldName(), place.getId())\n          .withStringField(EarlybirdFieldConstant.PLACE_FULL_NAME_FIELD.getFieldName(),\n              place.getFullName());\n      if (place.isSetCountryCode()) {\n        builder.withStringField(EarlybirdFieldConstant.PLACE_COUNTRY_CODE_FIELD.getFieldName(),\n            place.getCountryCode());\n      }\n    }\n\n    if (versionedTweetFeatures.isSetTokenizedProfileGeoEnrichment()) {\n      ProfileGeoEnrichment profileGeoEnrichment =\n          versionedTweetFeatures.getTokenizedProfileGeoEnrichment();\n      Preconditions.checkArgument(\n          profileGeoEnrichment.isSetPotentialLocations(),\n          \"ProfileGeoEnrichment.potentialLocations not set for tweet \"\n              + message.getId());\n      List<PotentialLocation> potentialLocations = profileGeoEnrichment.getPotentialLocations();\n      Preconditions.checkArgument(\n          !potentialLocations.isEmpty(),\n          \"Found tweet with an empty ProfileGeoEnrichment.potentialLocations: \"\n              + message.getId());\n      builder.addFilterInternalFieldTerm(EarlybirdFieldConstant.PROFILE_GEO_FILTER_TERM);\n      for (PotentialLocation potentialLocation : potentialLocations) {\n        if (potentialLocation.isSetCountryCode()) {\n          builder.withStringField(\n              EarlybirdFieldConstant.PROFILE_GEO_COUNTRY_CODE_FIELD.getFieldName(),\n              potentialLocation.getCountryCode());\n        }\n        if (potentialLocation.isSetRegion()) {\n          builder.withStringField(EarlybirdFieldConstant.PROFILE_GEO_REGION_FIELD.getFieldName(),\n              potentialLocation.getRegion());\n        }\n        if (potentialLocation.isSetLocality()) {\n          builder.withStringField(EarlybirdFieldConstant.PROFILE_GEO_LOCALITY_FIELD.getFieldName(),\n              potentialLocation.getLocality());\n        }\n      }\n    }\n\n    builder.withPlacesField(message.getPlaces());\n  }\n\n  /**\n   * Build the retweet and reply fields.\n   */\n  public static void buildRetweetAndReplyFields(\n      EarlybirdThriftDocumentBuilder builder,\n      TwitterMessage message,\n      boolean strict) {\n    long retweetUserIdVal = -1;\n    long sharedStatusIdVal = -1;\n    if (message.getRetweetMessage() != null) {\n      if (message.getRetweetMessage().getSharedId() != null) {\n        sharedStatusIdVal = message.getRetweetMessage().getSharedId();\n      }\n      if (message.getRetweetMessage().hasSharedUserTwitterId()) {\n        retweetUserIdVal = message.getRetweetMessage().getSharedUserTwitterId();\n      }\n    }\n\n    long inReplyToStatusIdVal = -1;\n    long inReplyToUserIdVal = -1;\n    if (message.isReply()) {\n      if (message.getInReplyToStatusId().isPresent()) {\n        inReplyToStatusIdVal = message.getInReplyToStatusId().get();\n      }\n      if (message.getToUserTwitterId().isPresent()) {\n        inReplyToUserIdVal = message.getToUserTwitterId().get();\n      }\n    }\n\n    buildRetweetAndReplyFields(\n        retweetUserIdVal,\n        sharedStatusIdVal,\n        inReplyToStatusIdVal,\n        inReplyToUserIdVal,\n        strict,\n        builder);\n  }\n\n  /**\n   * Build the quotes fields.\n   */\n  public static void buildQuotesFields(\n      EarlybirdThriftDocumentBuilder builder,\n      TwitterMessage message) {\n    if (message.getQuotedMessage() != null) {\n      TwitterQuotedMessage quoted = message.getQuotedMessage();\n      if (quoted != null && quoted.getQuotedStatusId() > 0 && quoted.getQuotedUserId() > 0) {\n        builder.withQuote(quoted.getQuotedStatusId(), quoted.getQuotedUserId());\n      }\n    }\n  }\n\n  /**\n   * Build directed at field.\n   */\n  public static void buildDirectedAtFields(\n      EarlybirdThriftDocumentBuilder builder,\n      TwitterMessage message) {\n    if (message.getDirectedAtUserId().isPresent() && message.getDirectedAtUserId().get() > 0) {\n      builder.withDirectedAtUser(message.getDirectedAtUserId().get());\n      builder.addFilterInternalFieldTerm(EarlybirdFieldConstant.DIRECTED_AT_FILTER_TERM);\n    }\n  }\n\n  /**\n   * Build the versioned features for a tweet.\n   */\n  public static void buildVersionedFeatureFields(\n      EarlybirdThriftDocumentBuilder builder,\n      VersionedTweetFeatures versionedTweetFeatures) {\n    builder\n        .withHashtagsField(versionedTweetFeatures.getHashtags())\n        .withMentionsField(versionedTweetFeatures.getMentions())\n        .withStocksFields(versionedTweetFeatures.getStocks())\n        .withResolvedLinksText(versionedTweetFeatures.getNormalizedResolvedUrlText())\n        .withTokenStreamField(EarlybirdFieldConstant.TEXT_FIELD.getFieldName(),\n            versionedTweetFeatures.getTweetTokenStreamText(),\n            versionedTweetFeatures.isSetTweetTokenStream()\n                ? versionedTweetFeatures.getTweetTokenStream() : null)\n        .withStringField(EarlybirdFieldConstant.SOURCE_FIELD.getFieldName(),\n            versionedTweetFeatures.getSource())\n        .withStringField(EarlybirdFieldConstant.NORMALIZED_SOURCE_FIELD.getFieldName(),\n            versionedTweetFeatures.getNormalizedSource());\n\n    // Internal fields for smileys and question marks\n    if (versionedTweetFeatures.hasPositiveSmiley) {\n      builder.withStringField(\n          EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n          EarlybirdFieldConstant.HAS_POSITIVE_SMILEY);\n    }\n    if (versionedTweetFeatures.hasNegativeSmiley) {\n      builder.withStringField(\n          EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n          EarlybirdFieldConstant.HAS_NEGATIVE_SMILEY);\n    }\n    if (versionedTweetFeatures.hasQuestionMark) {\n      builder.withStringField(EarlybirdFieldConstant.TEXT_FIELD.getFieldName(),\n          EarlybirdThriftDocumentBuilder.QUESTION_MARK);\n    }\n  }\n\n  /**\n   * Build the escherbird annotations for a tweet.\n   */\n  public static void buildAnnotationFields(\n      EarlybirdThriftDocumentBuilder builder,\n      TwitterMessage message) {\n    List<TwitterMessage.EscherbirdAnnotation> escherbirdAnnotations =\n        message.getEscherbirdAnnotations();\n    if (CollectionUtils.isEmpty(escherbirdAnnotations)) {\n      return;\n    }\n\n    builder.addFacetSkipList(EarlybirdFieldConstant.ENTITY_ID_FIELD.getFieldName());\n\n    for (TwitterMessage.EscherbirdAnnotation annotation : escherbirdAnnotations) {\n      String groupDomainEntity = String.format(\"%d.%d.%d\",\n          annotation.groupId, annotation.domainId, annotation.entityId);\n      String domainEntity = String.format(\"%d.%d\", annotation.domainId, annotation.entityId);\n      String entity = String.format(\"%d\", annotation.entityId);\n\n      builder.withStringField(EarlybirdFieldConstant.ENTITY_ID_FIELD.getFieldName(),\n          groupDomainEntity);\n      builder.withStringField(EarlybirdFieldConstant.ENTITY_ID_FIELD.getFieldName(),\n          domainEntity);\n      builder.withStringField(EarlybirdFieldConstant.ENTITY_ID_FIELD.getFieldName(),\n          entity);\n    }\n  }\n\n  /**\n   * Build the correct ThriftIndexingEvent's fields based on retweet and reply status.\n   */\n  public static void buildRetweetAndReplyFields(\n      long retweetUserIdVal,\n      long sharedStatusIdVal,\n      long inReplyToStatusIdVal,\n      long inReplyToUserIdVal,\n      boolean strict,\n      EarlybirdThriftDocumentBuilder builder) {\n    Optional<Long> retweetUserId = Optional.of(retweetUserIdVal).filter(x -> x > 0);\n    Optional<Long> sharedStatusId = Optional.of(sharedStatusIdVal).filter(x -> x > 0);\n    Optional<Long> inReplyToUserId = Optional.of(inReplyToUserIdVal).filter(x -> x > 0);\n    Optional<Long> inReplyToStatusId = Optional.of(inReplyToStatusIdVal).filter(x -> x > 0);\n\n    // We have six combinations here. A Tweet can be\n    //   1) a reply to another tweet (then it has both in-reply-to-user-id and\n    //      in-reply-to-status-id set),\n    //   2) directed-at a user (then it only has in-reply-to-user-id set),\n    //   3) not a reply at all.\n    // Additionally, it may or may not be a Retweet (if it is, then it has retweet-user-id and\n    // retweet-status-id set).\n    //\n    // We want to set some fields unconditionally, and some fields (reference-author-id and\n    // shared-status-id) depending on the reply/retweet combination.\n    //\n    // 1. Normal tweet (not a reply, not a retweet). None of the fields should be set.\n    //\n    // 2. Reply to a tweet (both in-reply-to-user-id and in-reply-to-status-id set).\n    //   IN_REPLY_TO_USER_ID_FIELD    should be set to in-reply-to-user-id\n    //   SHARED_STATUS_ID_CSF         should be set to in-reply-to-status-id\n    //   IS_REPLY_FLAG                should be set\n    //\n    // 3. Directed-at a user (only in-reply-to-user-id is set).\n    //   IN_REPLY_TO_USER_ID_FIELD    should be set to in-reply-to-user-id\n    //   IS_REPLY_FLAG                should be set\n    //\n    // 4. Retweet of a normal tweet (retweet-user-id and retweet-status-id are set).\n    //   RETWEET_SOURCE_USER_ID_FIELD should be set to retweet-user-id\n    //   SHARED_STATUS_ID_CSF         should be set to retweet-status-id\n    //   IS_RETWEET_FLAG              should be set\n    //\n    // 5. Retweet of a reply (both in-reply-to-user-id and in-reply-to-status-id set,\n    // retweet-user-id and retweet-status-id are set).\n    //   RETWEET_SOURCE_USER_ID_FIELD should be set to retweet-user-id\n    //   SHARED_STATUS_ID_CSF         should be set to retweet-status-id (retweet beats reply!)\n    //   IS_RETWEET_FLAG              should be set\n    //   IN_REPLY_TO_USER_ID_FIELD    should be set to in-reply-to-user-id\n    //   IS_REPLY_FLAG                should NOT be set\n    //\n    // 6. Retweet of a directed-at tweet (only in-reply-to-user-id is set,\n    // retweet-user-id and retweet-status-id are set).\n    //   RETWEET_SOURCE_USER_ID_FIELD should be set to retweet-user-id\n    //   SHARED_STATUS_ID_CSF         should be set to retweet-status-id\n    //   IS_RETWEET_FLAG              should be set\n    //   IN_REPLY_TO_USER_ID_FIELD    should be set to in-reply-to-user-id\n    //   IS_REPLY_FLAG                should NOT be set\n    //\n    // In other words:\n    // SHARED_STATUS_ID_CSF logic: if this is a retweet SHARED_STATUS_ID_CSF should be set to\n    // retweet-status-id, otherwise if it's a reply to a tweet, it should be set to\n    // in-reply-to-status-id.\n\n    Preconditions.checkState(retweetUserId.isPresent() == sharedStatusId.isPresent());\n\n    if (retweetUserId.isPresent()) {\n      builder.withNativeRetweet(retweetUserId.get(), sharedStatusId.get());\n\n      if (inReplyToUserId.isPresent()) {\n        // Set IN_REPLY_TO_USER_ID_FIELD even if this is a retweet of a reply.\n        builder.withInReplyToUserID(inReplyToUserId.get());\n      }\n    } else {\n      // If this is a retweet of a reply, we don't want to mark it as a reply, or override fields\n      // set by the retweet logic.\n      // If we are in this branch, this is not a retweet. Potentially, we set the reply flag,\n      // and override shared-status-id and reference-author-id.\n\n      if (inReplyToStatusId.isPresent()) {\n        if (strict) {\n          // Enforcing that if this is a reply to a tweet, then it also has a replied-to user.\n          Preconditions.checkState(inReplyToUserId.isPresent());\n        }\n        builder.withReplyFlag();\n        builder.withLongField(\n            EarlybirdFieldConstant.SHARED_STATUS_ID_CSF.getFieldName(),\n            inReplyToStatusId.get());\n        builder.withLongField(\n            EarlybirdFieldConstant.IN_REPLY_TO_TWEET_ID_FIELD.getFieldName(),\n            inReplyToStatusId.get());\n      }\n      if (inReplyToUserId.isPresent()) {\n        builder.withReplyFlag();\n        builder.withInReplyToUserID(inReplyToUserId.get());\n      }\n    }\n  }\n\n  /**\n   * Build the engagement fields.\n   */\n  public static void buildNormalizedMinEngagementFields(\n      EarlybirdThriftDocumentBuilder builder,\n      EarlybirdEncodedFeatures encodedFeatures,\n      EarlybirdCluster cluster) throws IOException {\n    if (EarlybirdCluster.isArchive(cluster)) {\n      int favoriteCount = encodedFeatures.getFeatureValue(EarlybirdFieldConstant.FAVORITE_COUNT);\n      int retweetCount = encodedFeatures.getFeatureValue(EarlybirdFieldConstant.RETWEET_COUNT);\n      int replyCount = encodedFeatures.getFeatureValue(EarlybirdFieldConstant.REPLY_COUNT);\n      builder\n          .withNormalizedMinEngagementField(\n              EarlybirdFieldConstant.NORMALIZED_FAVORITE_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD\n                  .getFieldName(),\n              favoriteCount);\n      builder\n          .withNormalizedMinEngagementField(\n              EarlybirdFieldConstant.NORMALIZED_RETWEET_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD\n                  .getFieldName(),\n              retweetCount);\n      builder\n          .withNormalizedMinEngagementField(\n              EarlybirdFieldConstant.NORMALIZED_REPLY_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD\n                  .getFieldName(),\n              replyCount);\n    }\n  }\n\n  /**\n   * As seen in SEARCH-5617, we sometimes have incorrect createdAt. This method tries to fix them\n   * by extracting creation time from snowflake when possible.\n   */\n  public static long fixCreatedAtTimeStampIfNecessary(long id, long createdAtMs) {\n    if (createdAtMs < VALID_CREATION_TIME_THRESHOLD_MILLIS\n        && id > SnowflakeIdParser.SNOWFLAKE_ID_LOWER_BOUND) {\n      // This tweet has a snowflake ID, and we can extract timestamp from the ID.\n      ADJUSTED_BAD_CREATED_AT_COUNTER.increment();\n      return SnowflakeIdParser.getTimestampFromTweetId(id);\n    } else if (!SnowflakeIdParser.isTweetIDAndCreatedAtConsistent(id, createdAtMs)) {\n      LOG.error(\n          \"Found inconsistent tweet ID and created at timestamp: [statusID={}], [createdAtMs={}]\",\n          id, createdAtMs);\n      INCONSISTENT_TWEET_ID_AND_CREATED_AT_MS.increment();\n    }\n\n    return createdAtMs;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/converter/earlybird/CombinedIndexingConverter.java",
    "content": "package com.twitter.search.common.converter.earlybird;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport javax.annotation.concurrent.NotThreadSafe;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdThriftDocumentBuilder;\nimport com.twitter.search.common.schema.thriftjava.ThriftDocument;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEventType;\n\n/**\n * CombinedIndexingConverter builds objects from TwitterMessage to ThriftVersionedEvent.\n *\n * It is used in tests and in offline jobs, so all data is available on the TwitterMessage. This\n * means that we don't need to split up the ThriftVersionedEvents into basic events and update\n * events, like we do in the realtime pipeline using the BasicIndexingConverter and the\n * DelayedIndexingConverter.\n */\n@NotThreadSafe\npublic class CombinedIndexingConverter {\n  private final EncodedFeatureBuilder featureBuilder;\n  private final Schema schema;\n  private final EarlybirdCluster cluster;\n\n  public CombinedIndexingConverter(Schema schema, EarlybirdCluster cluster) {\n    this.featureBuilder = new EncodedFeatureBuilder();\n    this.schema = schema;\n    this.cluster = cluster;\n  }\n\n  /**\n   * Converts a TwitterMessage to a Thrift representation.\n   */\n  public ThriftVersionedEvents convertMessageToThrift(\n      TwitterMessage message,\n      boolean strict,\n      List<PenguinVersion> penguinVersions) throws IOException {\n    Preconditions.checkNotNull(message);\n    Preconditions.checkNotNull(penguinVersions);\n\n    ThriftVersionedEvents versionedEvents = new ThriftVersionedEvents()\n        .setId(message.getId());\n\n    ImmutableSchemaInterface schemaSnapshot = schema.getSchemaSnapshot();\n\n    for (PenguinVersion penguinVersion : penguinVersions) {\n      ThriftDocument document =\n          buildDocumentForPenguinVersion(schemaSnapshot, message, strict, penguinVersion);\n\n      ThriftIndexingEvent thriftIndexingEvent = new ThriftIndexingEvent()\n          .setDocument(document)\n          .setEventType(ThriftIndexingEventType.INSERT)\n          .setSortId(message.getId());\n      message.getFromUserTwitterId().map(thriftIndexingEvent::setUid);\n      versionedEvents.putToVersionedEvents(penguinVersion.getByteValue(), thriftIndexingEvent);\n    }\n\n    return versionedEvents;\n  }\n\n  private ThriftDocument buildDocumentForPenguinVersion(\n      ImmutableSchemaInterface schemaSnapshot,\n      TwitterMessage message,\n      boolean strict,\n      PenguinVersion penguinVersion) throws IOException {\n    EncodedFeatureBuilder.TweetFeatureWithEncodeFeatures tweetFeature =\n        featureBuilder.createTweetFeaturesFromTwitterMessage(\n            message, penguinVersion, schemaSnapshot);\n\n    EarlybirdThriftDocumentBuilder builder =\n        BasicIndexingConverter.buildBasicFields(message, schemaSnapshot, cluster, tweetFeature);\n\n    BasicIndexingConverter\n        .buildUserFields(builder, message, tweetFeature.versionedFeatures, penguinVersion);\n    BasicIndexingConverter.buildGeoFields(builder, message, tweetFeature.versionedFeatures);\n    DelayedIndexingConverter.buildURLFields(builder, message, tweetFeature.encodedFeatures);\n    BasicIndexingConverter.buildRetweetAndReplyFields(builder, message, strict);\n    BasicIndexingConverter.buildQuotesFields(builder, message);\n    BasicIndexingConverter.buildVersionedFeatureFields(builder, tweetFeature.versionedFeatures);\n    DelayedIndexingConverter.buildCardFields(builder, message, penguinVersion);\n    BasicIndexingConverter.buildAnnotationFields(builder, message);\n    BasicIndexingConverter.buildNormalizedMinEngagementFields(\n        builder, tweetFeature.encodedFeatures, cluster);\n    DelayedIndexingConverter.buildNamedEntityFields(builder, message);\n    BasicIndexingConverter.buildDirectedAtFields(builder, message);\n\n    return builder.build();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/converter/earlybird/DelayedIndexingConverter.java",
    "content": "package com.twitter.search.common.converter.earlybird;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Locale;\nimport java.util.Map;\nimport java.util.Set;\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Joiner;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\n\nimport org.apache.commons.lang.StringUtils;\nimport org.apache.http.annotation.NotThreadSafe;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.text.token.TokenizedCharSequenceStream;\nimport com.twitter.common.text.util.TokenStreamSerializer;\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.cuad.ner.plain.thriftjava.NamedEntity;\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.constants.SearchCardType;\nimport com.twitter.search.common.decider.DeciderUtil;\nimport com.twitter.search.common.indexing.thriftjava.SearchCard2;\nimport com.twitter.search.common.indexing.thriftjava.ThriftExpandedUrl;\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.indexing.thriftjava.TwitterPhotoUrl;\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\nimport com.twitter.search.common.relevance.entities.TwitterMessageUser;\nimport com.twitter.search.common.relevance.features.TweetTextFeatures;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdEncodedFeatures;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.common.schema.earlybird.EarlybirdThriftDocumentBuilder;\nimport com.twitter.search.common.schema.thriftjava.ThriftDocument;\nimport com.twitter.search.common.schema.thriftjava.ThriftField;\nimport com.twitter.search.common.schema.thriftjava.ThriftFieldData;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEventType;\nimport com.twitter.search.common.util.lang.ThriftLanguageUtil;\nimport com.twitter.search.common.util.text.LanguageIdentifierHelper;\nimport com.twitter.search.common.util.text.NormalizerHelper;\nimport com.twitter.search.common.util.text.TokenizerHelper;\nimport com.twitter.search.common.util.text.TokenizerResult;\nimport com.twitter.search.common.util.text.TweetTokenStreamSerializer;\nimport com.twitter.service.spiderduck.gen.MediaTypes;\nimport com.twitter.search.common.metrics.SearchCounter;\n\n/**\n * Create and populate ThriftVersionedEvents from the URL data, card data, and named entities\n * contained in a TwitterMessage. This data is delayed because these services take a few seconds\n * to process tweets, and we want to send the basic data available in the BasicIndexingConverter as\n * soon as possible, so we send the additional data a few seconds later, as an update.\n *\n * Prefer to add data and processing to the BasicIndexingConverter when possible. Only add data here\n * if your data source _requires_ data from an external service AND the external service takes at\n * least a few seconds to process new tweets.\n */\n@NotThreadSafe\npublic class DelayedIndexingConverter {\n  private static final SearchCounter NUM_TWEETS_WITH_CARD_URL =\n      SearchCounter.export(\"tweets_with_card_url\");\n  private static final SearchCounter NUM_TWEETS_WITH_NUMERIC_CARD_URI =\n      SearchCounter.export(\"tweets_with_numeric_card_uri\");\n  private static final SearchCounter NUM_TWEETS_WITH_INVALID_CARD_URI =\n      SearchCounter.export(\"tweets_with_invalid_card_uri\");\n  private static final SearchCounter TOTAL_URLS =\n      SearchCounter.export(\"total_urls_on_tweets\");\n  private static final SearchCounter MEDIA_URLS_ON_TWEETS =\n      SearchCounter.export(\"media_urls_on_tweets\");\n  private static final SearchCounter NON_MEDIA_URLS_ON_TWEETS =\n      SearchCounter.export(\"non_media_urls_on_tweets\");\n  public static final String INDEX_URL_DESCRIPTION_AND_TITLE_DECIDER =\n      \"index_url_description_and_title\";\n\n  private static class ThriftDocumentWithEncodedTweetFeatures {\n    private final ThriftDocument document;\n    private final EarlybirdEncodedFeatures encodedFeatures;\n\n    public ThriftDocumentWithEncodedTweetFeatures(ThriftDocument document,\n                                                  EarlybirdEncodedFeatures encodedFeatures) {\n      this.document = document;\n      this.encodedFeatures = encodedFeatures;\n    }\n\n    public ThriftDocument getDocument() {\n      return document;\n    }\n\n    public EarlybirdEncodedFeatures getEncodedFeatures() {\n      return encodedFeatures;\n    }\n  }\n\n  // The list of all the encoded_tweet_features flags that might be updated by this converter.\n  // No extended_encoded_tweet_features are updated (otherwise they should be in this list too).\n  private static final List<EarlybirdFieldConstants.EarlybirdFieldConstant> UPDATED_FLAGS =\n      Lists.newArrayList(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.IS_OFFENSIVE_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_LINK_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.IS_SENSITIVE_CONTENT,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.TEXT_SCORE,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.TWEET_SIGNATURE,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.LINK_LANGUAGE,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_IMAGE_URL_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_VIDEO_URL_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_NEWS_URL_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_EXPANDO_CARD_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_MULTIPLE_MEDIA_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_CARD_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_VISIBLE_LINK_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_CONSUMER_VIDEO_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_PRO_VIDEO_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_VINE_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_PERISCOPE_FLAG,\n          EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_NATIVE_IMAGE_FLAG\n      );\n\n  private static final Logger LOG = LoggerFactory.getLogger(DelayedIndexingConverter.class);\n  private static final String AMPLIFY_CARD_NAME = \"amplify\";\n  private static final String PLAYER_CARD_NAME = \"player\";\n\n  private final EncodedFeatureBuilder featureBuilder = new EncodedFeatureBuilder();\n\n  private final Schema schema;\n  private final Decider decider;\n\n  public DelayedIndexingConverter(Schema schema, Decider decider) {\n    this.schema = schema;\n    this.decider = decider;\n  }\n\n  /**\n   * Converts the given message to two ThriftVersionedEvents instances: the first one is a feature\n   * update event for all link and card related flags, and the second one is the append event that\n   * might contain updates to all link and card related fields.\n   *\n   * We need to split the updates to fields and flags into two separate events because:\n   *  - When a tweet is created, earlybirds get the \"main\" event, which does not have resolved URLs.\n   *  - Then the earlybirds might get a feature update from the signal ingesters, marking the tweet\n   *    as spam.\n   *  - Then the ingesters resolve the URLs and send an update event. At this point, the ingesters\n   *    need to send updates for link-related flags too (HAS_LINK_FLAG, etc.). And there are a few\n   *    ways to do this:\n   *    1. Encode these flags into encoded_tweet_features and extended_encoded_tweet_features and\n   *       add these fields to the update event. The problem is that earlybirds will then override\n   *       the encoded_tweet_features ane extended_encoded_tweet_features fields in the index for\n   *       this tweet, which will override the feature update the earlybirds got earlier, which\n   *       means that a spammy tweet might no longer be marked as spam in the index.\n   *    2. Send updates only for the flags that might've been updated by this converter. Since\n   *       ThriftIndexingEvent already has a map of field -> value, it seems like the natural place\n   *       to add these updates to. However, earlybirds can correctly process flag updates only if\n   *       they come in a feature update event (PARTIAL_UPDATE). So we need to send the field\n   *       updates in an OUT_OF_ORDER_UPDATE event, and the flag updates in a PARTIAL_UPDATE event.\n   *\n   * We need to send the feature update event before the append event to avoid issues like the one\n   * in SEARCH-30919 where tweets were returned from the card name field index before the HAS_CARD\n   * feature was updated to true.\n   *\n   * @param message The TwitterMessage to convert.\n   * @param penguinVersions The Penguin versions for which ThriftIndexingEvents should be created.\n   * @return An out of order update event for all link- and card-related fields and a feature update\n   *         event for all link- and card-related flags.\n   */\n  public List<ThriftVersionedEvents> convertMessageToOutOfOrderAppendAndFeatureUpdate(\n      TwitterMessage message, List<PenguinVersion> penguinVersions) {\n    Preconditions.checkNotNull(message);\n    Preconditions.checkNotNull(penguinVersions);\n\n    ThriftVersionedEvents featureUpdateVersionedEvents = new ThriftVersionedEvents();\n    ThriftVersionedEvents outOfOrderAppendVersionedEvents = new ThriftVersionedEvents();\n    ImmutableSchemaInterface schemaSnapshot = schema.getSchemaSnapshot();\n\n    for (PenguinVersion penguinVersion : penguinVersions) {\n      ThriftDocumentWithEncodedTweetFeatures documentWithEncodedFeatures =\n          buildDocumentForPenguinVersion(schemaSnapshot, message, penguinVersion);\n\n      ThriftIndexingEvent featureUpdateThriftIndexingEvent = new ThriftIndexingEvent();\n      featureUpdateThriftIndexingEvent.setEventType(ThriftIndexingEventType.PARTIAL_UPDATE);\n      featureUpdateThriftIndexingEvent.setUid(message.getId());\n      featureUpdateThriftIndexingEvent.setDocument(\n          buildFeatureUpdateDocument(documentWithEncodedFeatures.getEncodedFeatures()));\n      featureUpdateVersionedEvents.putToVersionedEvents(\n          penguinVersion.getByteValue(), featureUpdateThriftIndexingEvent);\n\n      ThriftIndexingEvent outOfOrderAppendThriftIndexingEvent = new ThriftIndexingEvent();\n      outOfOrderAppendThriftIndexingEvent.setDocument(documentWithEncodedFeatures.getDocument());\n      outOfOrderAppendThriftIndexingEvent.setEventType(ThriftIndexingEventType.OUT_OF_ORDER_APPEND);\n      message.getFromUserTwitterId().ifPresent(outOfOrderAppendThriftIndexingEvent::setUid);\n      outOfOrderAppendThriftIndexingEvent.setSortId(message.getId());\n      outOfOrderAppendVersionedEvents.putToVersionedEvents(\n          penguinVersion.getByteValue(), outOfOrderAppendThriftIndexingEvent);\n    }\n\n    featureUpdateVersionedEvents.setId(message.getId());\n    outOfOrderAppendVersionedEvents.setId(message.getId());\n\n    return Lists.newArrayList(featureUpdateVersionedEvents, outOfOrderAppendVersionedEvents);\n  }\n\n  private ThriftDocument buildFeatureUpdateDocument(EarlybirdEncodedFeatures encodedFeatures) {\n    ThriftDocument document = new ThriftDocument();\n    for (EarlybirdFieldConstants.EarlybirdFieldConstant flag : UPDATED_FLAGS) {\n      ThriftField field = new ThriftField();\n      field.setFieldConfigId(flag.getFieldId());\n      field.setFieldData(new ThriftFieldData().setIntValue(encodedFeatures.getFeatureValue(flag)));\n      document.addToFields(field);\n    }\n    return document;\n  }\n\n  private ThriftDocumentWithEncodedTweetFeatures buildDocumentForPenguinVersion(\n      ImmutableSchemaInterface schemaSnapshot,\n      TwitterMessage message,\n      PenguinVersion penguinVersion) {\n\n    EarlybirdEncodedFeatures encodedFeatures = featureBuilder.createTweetFeaturesFromTwitterMessage(\n        message, penguinVersion, schemaSnapshot).encodedFeatures;\n\n    EarlybirdThriftDocumentBuilder builder = new EarlybirdThriftDocumentBuilder(\n        encodedFeatures,\n        null,\n        new EarlybirdFieldConstants(),\n        schemaSnapshot);\n\n    builder.setAddLatLonCSF(false);\n    builder.withID(message.getId());\n    buildFieldsFromUrlInfo(builder, message, penguinVersion, encodedFeatures);\n    buildCardFields(builder, message, penguinVersion);\n    buildNamedEntityFields(builder, message);\n    builder.withTweetSignature(message.getTweetSignature(penguinVersion));\n\n    buildSpaceAdminAndTitleFields(builder, message, penguinVersion);\n\n    builder.setAddEncodedTweetFeatures(false);\n\n    return new ThriftDocumentWithEncodedTweetFeatures(builder.build(), encodedFeatures);\n  }\n\n  public static void buildNamedEntityFields(\n      EarlybirdThriftDocumentBuilder builder, TwitterMessage message) {\n    for (NamedEntity namedEntity : message.getNamedEntities()) {\n      builder.withNamedEntity(namedEntity);\n    }\n  }\n\n  private void buildFieldsFromUrlInfo(\n      EarlybirdThriftDocumentBuilder builder,\n      TwitterMessage message,\n      PenguinVersion penguinVersion,\n      EarlybirdEncodedFeatures encodedFeatures) {\n    // We need to update the RESOLVED_LINKS_TEXT_FIELD, since we might have new resolved URLs.\n    // Use the same logic as in EncodedFeatureBuilder.java.\n    TweetTextFeatures textFeatures = message.getTweetTextFeatures(penguinVersion);\n    String resolvedUrlsText = Joiner.on(\" \").skipNulls().join(textFeatures.getResolvedUrlTokens());\n    builder.withResolvedLinksText(resolvedUrlsText);\n\n    buildURLFields(builder, message, encodedFeatures);\n    buildAnalyzedURLFields(builder, message, penguinVersion);\n  }\n\n  private void buildAnalyzedURLFields(\n      EarlybirdThriftDocumentBuilder builder, TwitterMessage message, PenguinVersion penguinVersion\n  ) {\n    TOTAL_URLS.add(message.getExpandedUrls().size());\n    if (DeciderUtil.isAvailableForRandomRecipient(\n        decider,\n        INDEX_URL_DESCRIPTION_AND_TITLE_DECIDER)) {\n      for (ThriftExpandedUrl expandedUrl : message.getExpandedUrls()) {\n      /*\n        Consumer Media URLs are added to the expanded URLs in\n        TweetEventParserHelper.addMediaEntitiesToMessage. These Twitter.com media URLs contain\n        the tweet text as the description and the title is \"<User Name> on Twitter\". This is\n        redundant information at best and misleading at worst. We will ignore these URLs to avoid\n        polluting the url_description and url_title field as well as saving space.\n       */\n        if (!expandedUrl.isSetConsumerMedia() || !expandedUrl.isConsumerMedia()) {\n          NON_MEDIA_URLS_ON_TWEETS.increment();\n          if (expandedUrl.isSetDescription()) {\n            buildTweetTokenizerTokenizedField(builder,\n                EarlybirdFieldConstants.EarlybirdFieldConstant.URL_DESCRIPTION_FIELD.getFieldName(),\n                expandedUrl.getDescription(),\n                penguinVersion);\n          }\n          if (expandedUrl.isSetTitle()) {\n            buildTweetTokenizerTokenizedField(builder,\n                EarlybirdFieldConstants.EarlybirdFieldConstant.URL_TITLE_FIELD.getFieldName(),\n                expandedUrl.getTitle(),\n                penguinVersion);\n          }\n        } else {\n          MEDIA_URLS_ON_TWEETS.increment();\n        }\n      }\n    }\n  }\n\n  /**\n   * Build the URL based fields from a tweet.\n   */\n  public static void buildURLFields(\n      EarlybirdThriftDocumentBuilder builder,\n      TwitterMessage message,\n      EarlybirdEncodedFeatures encodedFeatures\n  ) {\n    Map<String, ThriftExpandedUrl> expandedUrlMap = message.getExpandedUrlMap();\n\n    for (ThriftExpandedUrl expandedUrl : expandedUrlMap.values()) {\n      if (expandedUrl.getMediaType() == MediaTypes.NATIVE_IMAGE) {\n        EncodedFeatureBuilder.addPhotoUrl(message, expandedUrl.getCanonicalLastHopUrl());\n      }\n    }\n\n    // now add all twitter photos links that came with the tweet's payload\n    Map<Long, String> photos = message.getPhotoUrls();\n    List<TwitterPhotoUrl> photoURLs = new ArrayList<>();\n    if (photos != null) {\n      for (Map.Entry<Long, String> entry : photos.entrySet()) {\n        TwitterPhotoUrl photo = new TwitterPhotoUrl(entry.getKey());\n        String mediaUrl = entry.getValue();\n        if (mediaUrl != null) {\n          photo.setMediaUrl(mediaUrl);\n        }\n        photoURLs.add(photo);\n      }\n    }\n\n    try {\n      builder\n          .withURLs(Lists.newArrayList(expandedUrlMap.values()))\n          .withTwimgURLs(photoURLs);\n    } catch (IOException ioe) {\n      LOG.error(\"URL field creation threw an IOException\", ioe);\n    }\n\n\n    if (encodedFeatures.isFlagSet(\n        EarlybirdFieldConstants.EarlybirdFieldConstant.IS_OFFENSIVE_FLAG)) {\n      builder.withOffensiveFlag();\n    }\n    if (encodedFeatures.isFlagSet(\n        EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_CONSUMER_VIDEO_FLAG)) {\n      builder.addFilterInternalFieldTerm(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.CONSUMER_VIDEO_FILTER_TERM);\n    }\n    if (encodedFeatures.isFlagSet(\n        EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_PRO_VIDEO_FLAG)) {\n      builder.addFilterInternalFieldTerm(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.PRO_VIDEO_FILTER_TERM);\n    }\n    if (encodedFeatures.isFlagSet(EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_VINE_FLAG)) {\n      builder.addFilterInternalFieldTerm(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.VINE_FILTER_TERM);\n    }\n    if (encodedFeatures.isFlagSet(\n        EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_PERISCOPE_FLAG)) {\n      builder.addFilterInternalFieldTerm(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.PERISCOPE_FILTER_TERM);\n    }\n  }\n\n  /**\n   * Build the card information inside ThriftIndexingEvent's fields.\n   */\n  static void buildCardFields(EarlybirdThriftDocumentBuilder builder,\n                              TwitterMessage message,\n                              PenguinVersion penguinVersion) {\n    if (message.hasCard()) {\n      SearchCard2 card = buildSearchCardFromTwitterMessage(\n          message,\n          TweetTokenStreamSerializer.getTweetTokenStreamSerializer(),\n          penguinVersion);\n      buildCardFeatures(message.getId(), builder, card);\n    }\n  }\n\n  private static SearchCard2 buildSearchCardFromTwitterMessage(\n      TwitterMessage message,\n      TokenStreamSerializer streamSerializer,\n      PenguinVersion penguinVersion) {\n    SearchCard2 card = new SearchCard2();\n    card.setCardName(message.getCardName());\n    if (message.getCardDomain() != null) {\n      card.setCardDomain(message.getCardDomain());\n    }\n    if (message.getCardLang() != null) {\n      card.setCardLang(message.getCardLang());\n    }\n    if (message.getCardUrl() != null) {\n      card.setCardUrl(message.getCardUrl());\n    }\n\n    if (message.getCardTitle() != null && !message.getCardTitle().isEmpty()) {\n      String normalizedTitle = NormalizerHelper.normalize(\n          message.getCardTitle(), message.getLocale(), penguinVersion);\n      TokenizerResult result = TokenizerHelper.tokenizeTweet(\n          normalizedTitle, message.getLocale(), penguinVersion);\n      TokenizedCharSequenceStream tokenSeqStream = new TokenizedCharSequenceStream();\n      tokenSeqStream.reset(result.tokenSequence);\n      try {\n        card.setCardTitleTokenStream(streamSerializer.serialize(tokenSeqStream));\n        card.setCardTitleTokenStreamText(result.tokenSequence.toString());\n      } catch (IOException e) {\n        LOG.error(\"TwitterTokenStream serialization error! Could not serialize card title: \"\n            + result.tokenSequence);\n        card.unsetCardTitleTokenStream();\n        card.unsetCardTitleTokenStreamText();\n      }\n    }\n    if (message.getCardDescription() != null && !message.getCardDescription().isEmpty()) {\n      String normalizedDesc = NormalizerHelper.normalize(\n          message.getCardDescription(), message.getLocale(), penguinVersion);\n      TokenizerResult result = TokenizerHelper.tokenizeTweet(\n          normalizedDesc, message.getLocale(), penguinVersion);\n      TokenizedCharSequenceStream tokenSeqStream = new TokenizedCharSequenceStream();\n      tokenSeqStream.reset(result.tokenSequence);\n      try {\n        card.setCardDescriptionTokenStream(streamSerializer.serialize(tokenSeqStream));\n        card.setCardDescriptionTokenStreamText(result.tokenSequence.toString());\n      } catch (IOException e) {\n        LOG.error(\"TwitterTokenStream serialization error! Could not serialize card description: \"\n            + result.tokenSequence);\n        card.unsetCardDescriptionTokenStream();\n        card.unsetCardDescriptionTokenStreamText();\n      }\n    }\n\n    return card;\n  }\n\n  /**\n   * Builds card features.\n   */\n  private static void buildCardFeatures(\n      long tweetId, EarlybirdThriftDocumentBuilder builder, SearchCard2 card) {\n    if (card == null) {\n      return;\n    }\n    builder\n        .withTokenStreamField(\n            EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_TITLE_FIELD.getFieldName(),\n            card.getCardTitleTokenStreamText(),\n            card.isSetCardTitleTokenStream() ? card.getCardTitleTokenStream() : null)\n        .withTokenStreamField(\n            EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_DESCRIPTION_FIELD.getFieldName(),\n            card.getCardDescriptionTokenStreamText(),\n            card.isSetCardDescriptionTokenStream() ? card.getCardDescriptionTokenStream() : null)\n        .withStringField(\n            EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_NAME_FIELD.getFieldName(),\n            card.getCardName())\n        .withIntField(\n            EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_TYPE_CSF_FIELD.getFieldName(),\n            SearchCardType.cardTypeFromStringName(card.getCardName()).getByteValue());\n\n    if (card.getCardLang() != null) {\n      builder.withStringField(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_LANG.getFieldName(),\n          card.getCardLang()).withIntField(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_LANG_CSF.getFieldName(),\n          ThriftLanguageUtil.getThriftLanguageOf(card.getCardLang()).getValue());\n    }\n    if (card.getCardDomain() != null) {\n      builder.withStringField(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_DOMAIN_FIELD.getFieldName(),\n          card.getCardDomain());\n    }\n    if (card.getCardUrl() != null) {\n      NUM_TWEETS_WITH_CARD_URL.increment();\n      if (card.getCardUrl().startsWith(\"card://\")) {\n        String suffix = card.getCardUrl().replace(\"card://\", \"\");\n        if (StringUtils.isNumeric(suffix)) {\n          NUM_TWEETS_WITH_NUMERIC_CARD_URI.increment();\n          builder.withLongField(\n              EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_URI_CSF.getFieldName(),\n              Long.parseLong(suffix));\n          LOG.debug(String.format(\n              \"Good card URL for tweet %s: %s\",\n              tweetId,\n              card.getCardUrl()));\n        } else {\n          NUM_TWEETS_WITH_INVALID_CARD_URI.increment();\n          LOG.debug(String.format(\n              \"Card URL starts with \\\"card://\\\" but followed by non-numeric for tweet %s: %s\",\n              tweetId,\n              card.getCardUrl()));\n        }\n      }\n    }\n    if (isCardVideo(card)) {\n      // Add into \"internal\" field so that this tweet is returned by filter:videos.\n      builder.addFacetSkipList(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.VIDEO_LINKS_FIELD.getFieldName());\n    }\n  }\n\n  /**\n   * Determines if a card is a video.\n   */\n  private static boolean isCardVideo(@Nullable SearchCard2 card) {\n    if (card == null) {\n      return false;\n    }\n    return AMPLIFY_CARD_NAME.equalsIgnoreCase(card.getCardName())\n        || PLAYER_CARD_NAME.equalsIgnoreCase(card.getCardName());\n  }\n\n  private void buildSpaceAdminAndTitleFields(\n      EarlybirdThriftDocumentBuilder builder,\n      TwitterMessage message,\n      PenguinVersion penguinVersion) {\n\n    buildSpaceAdminFields(builder, message.getSpaceAdmins(), penguinVersion);\n\n    // build the space title field.\n    buildTweetTokenizerTokenizedField(\n        builder,\n        EarlybirdFieldConstants.EarlybirdFieldConstant.SPACE_TITLE_FIELD.getFieldName(),\n        message.getSpaceTitle(),\n        penguinVersion);\n  }\n\n  private void buildSpaceAdminFields(\n      EarlybirdThriftDocumentBuilder builder,\n      Set<TwitterMessageUser> spaceAdmins,\n      PenguinVersion penguinVersion) {\n\n    for (TwitterMessageUser spaceAdmin : spaceAdmins) {\n      if (spaceAdmin.getScreenName().isPresent()) {\n        // build screen name (aka handle) fields.\n        String screenName = spaceAdmin.getScreenName().get();\n        String normalizedScreenName =\n            NormalizerHelper.normalizeWithUnknownLocale(screenName, penguinVersion);\n\n        builder.withStringField(\n            EarlybirdFieldConstants.EarlybirdFieldConstant.SPACE_ADMIN_FIELD.getFieldName(),\n            normalizedScreenName);\n        builder.withWhiteSpaceTokenizedScreenNameField(\n            EarlybirdFieldConstants\n                .EarlybirdFieldConstant.TOKENIZED_SPACE_ADMIN_FIELD.getFieldName(),\n            normalizedScreenName);\n\n        if (spaceAdmin.getTokenizedScreenName().isPresent()) {\n          builder.withCamelCaseTokenizedScreenNameField(\n              EarlybirdFieldConstants\n                  .EarlybirdFieldConstant.CAMELCASE_TOKENIZED_SPACE_ADMIN_FIELD.getFieldName(),\n              screenName,\n              normalizedScreenName,\n              spaceAdmin.getTokenizedScreenName().get());\n        }\n      }\n\n      if (spaceAdmin.getDisplayName().isPresent()) {\n        buildTweetTokenizerTokenizedField(\n            builder,\n            EarlybirdFieldConstants\n                .EarlybirdFieldConstant.TOKENIZED_SPACE_ADMIN_DISPLAY_NAME_FIELD.getFieldName(),\n            spaceAdmin.getDisplayName().get(),\n            penguinVersion);\n      }\n    }\n  }\n\n  private void buildTweetTokenizerTokenizedField(\n      EarlybirdThriftDocumentBuilder builder,\n      String fieldName,\n      String text,\n      PenguinVersion penguinVersion) {\n\n    if (StringUtils.isNotEmpty(text)) {\n      Locale locale = LanguageIdentifierHelper\n          .identifyLanguage(text);\n      String normalizedText = NormalizerHelper.normalize(\n          text, locale, penguinVersion);\n      TokenizerResult result = TokenizerHelper\n          .tokenizeTweet(normalizedText, locale, penguinVersion);\n      TokenizedCharSequenceStream tokenSeqStream = new TokenizedCharSequenceStream();\n      tokenSeqStream.reset(result.tokenSequence);\n      TokenStreamSerializer streamSerializer =\n          TweetTokenStreamSerializer.getTweetTokenStreamSerializer();\n      try {\n        builder.withTokenStreamField(\n            fieldName,\n            result.tokenSequence.toString(),\n            streamSerializer.serialize(tokenSeqStream));\n      } catch (IOException e) {\n        LOG.error(\"TwitterTokenStream serialization error! Could not serialize: \" + text);\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/converter/earlybird/EncodedFeatureBuilder.java",
    "content": "package com.twitter.search.common.converter.earlybird;\n\nimport java.io.IOException;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Locale;\nimport java.util.Map;\nimport java.util.Optional;\nimport java.util.Set;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\nimport java.util.stream.Collectors;\n\nimport com.google.common.base.Joiner;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.apache.commons.lang.StringUtils;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.text.token.TokenizedCharSequence;\nimport com.twitter.common.text.token.TokenizedCharSequenceStream;\nimport com.twitter.common.text.util.TokenStreamSerializer;\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.search.common.indexing.thriftjava.Place;\nimport com.twitter.search.common.indexing.thriftjava.PotentialLocation;\nimport com.twitter.search.common.indexing.thriftjava.ProfileGeoEnrichment;\nimport com.twitter.search.common.indexing.thriftjava.ThriftExpandedUrl;\nimport com.twitter.search.common.indexing.thriftjava.VersionedTweetFeatures;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.relevance.entities.PotentialLocationObject;\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\nimport com.twitter.search.common.relevance.features.FeatureSink;\nimport com.twitter.search.common.relevance.features.MutableFeatureNormalizers;\nimport com.twitter.search.common.relevance.features.RelevanceSignalConstants;\nimport com.twitter.search.common.relevance.features.TweetTextFeatures;\nimport com.twitter.search.common.relevance.features.TweetTextQuality;\nimport com.twitter.search.common.relevance.features.TweetUserFeatures;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdEncodedFeatures;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.util.lang.ThriftLanguageUtil;\nimport com.twitter.search.common.util.text.LanguageIdentifierHelper;\nimport com.twitter.search.common.util.text.NormalizerHelper;\nimport com.twitter.search.common.util.text.SourceNormalizer;\nimport com.twitter.search.common.util.text.TokenizerHelper;\nimport com.twitter.search.common.util.text.TokenizerResult;\nimport com.twitter.search.common.util.text.TweetTokenStreamSerializer;\nimport com.twitter.search.common.util.url.LinkVisibilityUtils;\nimport com.twitter.search.common.util.url.NativeVideoClassificationUtils;\nimport com.twitter.search.ingester.model.VisibleTokenRatioUtil;\n\n/**\n * EncodedFeatureBuilder helps to build encoded features for TwitterMessage.\n *\n * This is stateful so should only be used one tweet at a time\n */\npublic class EncodedFeatureBuilder {\n  private static final Logger LOG = LoggerFactory.getLogger(EncodedFeatureBuilder.class);\n\n  private static final SearchCounter NUM_TWEETS_WITH_INVALID_TWEET_ID_IN_PHOTO_URL =\n      SearchCounter.export(\"tweets_with_invalid_tweet_id_in_photo_url\");\n\n  // TwitterTokenStream for converting TokenizedCharSequence into a stream for serialization\n  // This is stateful so should only be used one tweet at a time\n  private final TokenizedCharSequenceStream tokenSeqStream = new TokenizedCharSequenceStream();\n\n  // SUPPRESS CHECKSTYLE:OFF LineLength\n  private static final Pattern TWITTER_PHOTO_PERMA_LINK_PATTERN =\n          Pattern.compile(\"(?i:^(?:(?:https?\\\\:\\\\/\\\\/)?(?:www\\\\.)?)?twitter\\\\.com\\\\/(?:\\\\?[^#]+)?(?:#!?\\\\/?)?\\\\w{1,20}\\\\/status\\\\/(\\\\d+)\\\\/photo\\\\/\\\\d*$)\");\n\n  private static final Pattern TWITTER_PHOTO_COPY_PASTE_LINK_PATTERN =\n          Pattern.compile(\"(?i:^(?:(?:https?\\\\:\\\\/\\\\/)?(?:www\\\\.)?)?twitter\\\\.com\\\\/(?:#!?\\\\/)?\\\\w{1,20}\\\\/status\\\\/(\\\\d+)\\\\/photo\\\\/\\\\d*$)\");\n  // SUPPRESS CHECKSTYLE:ON LineLength\n\n  private static final VisibleTokenRatioUtil VISIBLE_TOKEN_RATIO = new VisibleTokenRatioUtil();\n\n  private static final Map<PenguinVersion, SearchCounter> SERIALIZE_FAILURE_COUNTERS_MAP =\n      Maps.newEnumMap(PenguinVersion.class);\n  static {\n    for (PenguinVersion penguinVersion : PenguinVersion.values()) {\n      SERIALIZE_FAILURE_COUNTERS_MAP.put(\n          penguinVersion,\n          SearchCounter.export(\n              \"tokenstream_serialization_failure_\" + penguinVersion.name().toLowerCase()));\n    }\n  }\n\n  public static class TweetFeatureWithEncodeFeatures {\n    public final VersionedTweetFeatures versionedFeatures;\n    public final EarlybirdEncodedFeatures encodedFeatures;\n    public final EarlybirdEncodedFeatures extendedEncodedFeatures;\n\n    public TweetFeatureWithEncodeFeatures(\n        VersionedTweetFeatures versionedFeatures,\n        EarlybirdEncodedFeatures encodedFeatures,\n        EarlybirdEncodedFeatures extendedEncodedFeatures) {\n      this.versionedFeatures = versionedFeatures;\n      this.encodedFeatures = encodedFeatures;\n      this.extendedEncodedFeatures = extendedEncodedFeatures;\n    }\n  }\n\n  /**\n   * Create tweet text features and the encoded features.\n   *\n   * @param message the tweet message\n   * @param penguinVersion the based penguin version to create the features\n   * @param schemaSnapshot the schema associated with the features\n   * @return the text features and the encoded features\n   */\n  public TweetFeatureWithEncodeFeatures createTweetFeaturesFromTwitterMessage(\n      TwitterMessage message,\n      PenguinVersion penguinVersion,\n      ImmutableSchemaInterface schemaSnapshot) {\n    VersionedTweetFeatures versionedTweetFeatures = new VersionedTweetFeatures();\n\n    // Write extendedPackedFeatures.\n    EarlybirdEncodedFeatures extendedEncodedFeatures =\n        createExtendedEncodedFeaturesFromTwitterMessage(message, penguinVersion, schemaSnapshot);\n    if (extendedEncodedFeatures != null) {\n      extendedEncodedFeatures\n          .writeExtendedFeaturesToVersionedTweetFeatures(versionedTweetFeatures);\n    }\n\n    setSourceAndNormalizedSource(\n        message.getStrippedSource(), versionedTweetFeatures, penguinVersion);\n\n    TweetTextFeatures textFeatures = message.getTweetTextFeatures(penguinVersion);\n\n    ///////////////////////////////\n    // Add hashtags and mentions\n    textFeatures.getHashtags().forEach(versionedTweetFeatures::addToHashtags);\n    textFeatures.getMentions().forEach(versionedTweetFeatures::addToMentions);\n\n    ///////////////////////////////\n    // Extract some extra information from the message text.\n    // Index stock symbols with $ prepended\n    textFeatures.getStocks().stream()\n        .filter(stock -> stock != null)\n        .forEach(stock -> versionedTweetFeatures.addToStocks(stock.toLowerCase()));\n\n    // Question marks\n    versionedTweetFeatures.setHasQuestionMark(textFeatures.hasQuestionMark());\n    // Smileys\n    versionedTweetFeatures.setHasPositiveSmiley(textFeatures.hasPositiveSmiley());\n    versionedTweetFeatures.setHasNegativeSmiley(textFeatures.hasNegativeSmiley());\n\n    TokenStreamSerializer streamSerializer =\n        TweetTokenStreamSerializer.getTweetTokenStreamSerializer();\n    TokenizedCharSequence tokenSeq = textFeatures.getTokenSequence();\n    tokenSeqStream.reset(tokenSeq);\n    int tokenPercent = VISIBLE_TOKEN_RATIO.extractAndNormalizeTokenPercentage(tokenSeqStream);\n    tokenSeqStream.reset(tokenSeq);\n\n    // Write packedFeatures.\n    EarlybirdEncodedFeatures encodedFeatures = createEncodedFeaturesFromTwitterMessage(\n        message, penguinVersion, schemaSnapshot, tokenPercent);\n    encodedFeatures.writeFeaturesToVersionedTweetFeatures(versionedTweetFeatures);\n\n    try {\n      versionedTweetFeatures.setTweetTokenStream(streamSerializer.serialize(tokenSeqStream));\n      versionedTweetFeatures.setTweetTokenStreamText(tokenSeq.toString());\n    } catch (IOException e) {\n      LOG.error(\"TwitterTokenStream serialization error! Could not serialize: \"\n          + tokenSeq.toString());\n      SERIALIZE_FAILURE_COUNTERS_MAP.get(penguinVersion).increment();\n      versionedTweetFeatures.unsetTweetTokenStream();\n      versionedTweetFeatures.unsetTweetTokenStreamText();\n    }\n\n    // User name features\n    if (message.getFromUserDisplayName().isPresent()) {\n      Locale locale = LanguageIdentifierHelper\n          .identifyLanguage(message.getFromUserDisplayName().get());\n      String normalizedDisplayName = NormalizerHelper.normalize(\n          message.getFromUserDisplayName().get(), locale, penguinVersion);\n      TokenizerResult result = TokenizerHelper\n          .tokenizeTweet(normalizedDisplayName, locale, penguinVersion);\n      tokenSeqStream.reset(result.tokenSequence);\n      try {\n        versionedTweetFeatures.setUserDisplayNameTokenStream(\n            streamSerializer.serialize(tokenSeqStream));\n        versionedTweetFeatures.setUserDisplayNameTokenStreamText(result.tokenSequence.toString());\n      } catch (IOException e) {\n        LOG.error(\"TwitterTokenStream serialization error! Could not serialize: \"\n            + message.getFromUserDisplayName().get());\n        SERIALIZE_FAILURE_COUNTERS_MAP.get(penguinVersion).increment();\n        versionedTweetFeatures.unsetUserDisplayNameTokenStream();\n        versionedTweetFeatures.unsetUserDisplayNameTokenStreamText();\n      }\n    }\n\n    String resolvedUrlsText = Joiner.on(\" \").skipNulls().join(textFeatures.getResolvedUrlTokens());\n    versionedTweetFeatures.setNormalizedResolvedUrlText(resolvedUrlsText);\n\n    addPlace(message, versionedTweetFeatures, penguinVersion);\n    addProfileGeoEnrichment(message, versionedTweetFeatures, penguinVersion);\n\n    versionedTweetFeatures.setTweetSignature(message.getTweetSignature(penguinVersion));\n\n    return new TweetFeatureWithEncodeFeatures(\n        versionedTweetFeatures, encodedFeatures, extendedEncodedFeatures);\n  }\n\n\n  protected static void setSourceAndNormalizedSource(\n      String strippedSource,\n      VersionedTweetFeatures versionedTweetFeatures,\n      PenguinVersion penguinVersion) {\n\n    if (strippedSource != null && !strippedSource.isEmpty()) {\n      // normalize source for searchable field - replaces whitespace with underscores (???).\n      versionedTweetFeatures.setNormalizedSource(\n          SourceNormalizer.normalize(strippedSource, penguinVersion));\n\n      // source facet has simpler normalization.\n      Locale locale = LanguageIdentifierHelper.identifyLanguage(strippedSource);\n      versionedTweetFeatures.setSource(NormalizerHelper.normalizeKeepCase(\n          strippedSource, locale, penguinVersion));\n    }\n  }\n\n  /**\n   * Adds the given photo url to the thrift status if it is a twitter photo permalink.\n   * Returns true, if this was indeed a twitter photo, false otherwise.\n   */\n  public static boolean addPhotoUrl(TwitterMessage message, String photoPermalink) {\n    Matcher matcher = TWITTER_PHOTO_COPY_PASTE_LINK_PATTERN.matcher(photoPermalink);\n    if (!matcher.matches() || matcher.groupCount() < 1) {\n      matcher = TWITTER_PHOTO_PERMA_LINK_PATTERN.matcher(photoPermalink);\n    }\n\n    if (matcher.matches() && matcher.groupCount() == 1) {\n      // this is a native photo url which we need to store in a separate field\n      String idStr = matcher.group(1);\n      if (idStr != null) {\n        // idStr should be a valid tweet ID (and therefore, should fit into a Long), but we have\n        // tweets for which idStr is a long sequence of digits that does not fit into a Long.\n        try {\n          long photoStatusId = Long.parseLong(idStr);\n          message.addPhotoUrl(photoStatusId, null);\n        } catch (NumberFormatException e) {\n          LOG.warn(\"Found a tweet with a photo URL with an invalid tweet ID: \" + message);\n          NUM_TWEETS_WITH_INVALID_TWEET_ID_IN_PHOTO_URL.increment();\n        }\n      }\n      return true;\n    }\n    return false;\n  }\n\n  private void addPlace(TwitterMessage message,\n                        VersionedTweetFeatures versionedTweetFeatures,\n                        PenguinVersion penguinVersion) {\n    String placeId = message.getPlaceId();\n    if (placeId == null) {\n      return;\n    }\n\n    // Tweet.Place.id and Tweet.Place.full_name are both required fields.\n    String placeFullName = message.getPlaceFullName();\n    Preconditions.checkNotNull(placeFullName, \"Tweet.Place without full_name.\");\n\n    Locale placeFullNameLocale = LanguageIdentifierHelper.identifyLanguage(placeFullName);\n    String normalizedPlaceFullName =\n        NormalizerHelper.normalize(placeFullName, placeFullNameLocale, penguinVersion);\n    String tokenizedPlaceFullName = StringUtils.join(\n        TokenizerHelper.tokenizeQuery(normalizedPlaceFullName, placeFullNameLocale, penguinVersion),\n        \" \");\n\n    Place place = new Place(placeId, tokenizedPlaceFullName);\n    String placeCountryCode = message.getPlaceCountryCode();\n    if (placeCountryCode != null) {\n      Locale placeCountryCodeLocale = LanguageIdentifierHelper.identifyLanguage(placeCountryCode);\n      place.setCountryCode(\n          NormalizerHelper.normalize(placeCountryCode, placeCountryCodeLocale, penguinVersion));\n    }\n\n    versionedTweetFeatures.setTokenizedPlace(place);\n  }\n\n  private void addProfileGeoEnrichment(TwitterMessage message,\n                                       VersionedTweetFeatures versionedTweetFeatures,\n                                       PenguinVersion penguinVersion) {\n    List<PotentialLocationObject> potentialLocations = message.getPotentialLocations();\n    if (potentialLocations.isEmpty()) {\n      return;\n    }\n\n    List<PotentialLocation> thriftPotentialLocations = Lists.newArrayList();\n    for (PotentialLocationObject potentialLocation : potentialLocations) {\n      thriftPotentialLocations.add(potentialLocation.toThriftPotentialLocation(penguinVersion));\n    }\n    versionedTweetFeatures.setTokenizedProfileGeoEnrichment(\n        new ProfileGeoEnrichment(thriftPotentialLocations));\n  }\n\n  /** Returns the encoded features. */\n  public static EarlybirdEncodedFeatures createEncodedFeaturesFromTwitterMessage(\n      TwitterMessage message,\n      PenguinVersion penguinVersion,\n      ImmutableSchemaInterface schema,\n      int normalizedTokenPercentBucket) {\n    FeatureSink sink = new FeatureSink(schema);\n\n    // Static features\n    sink.setBooleanValue(EarlybirdFieldConstant.IS_RETWEET_FLAG, message.isRetweet())\n        .setBooleanValue(EarlybirdFieldConstant.IS_REPLY_FLAG, message.isReply())\n        .setBooleanValue(\n            EarlybirdFieldConstant.FROM_VERIFIED_ACCOUNT_FLAG, message.isUserVerified())\n        .setBooleanValue(\n            EarlybirdFieldConstant.FROM_BLUE_VERIFIED_ACCOUNT_FLAG, message.isUserBlueVerified())\n        .setBooleanValue(EarlybirdFieldConstant.IS_SENSITIVE_CONTENT, message.isSensitiveContent());\n\n    TweetTextFeatures textFeatures = message.getTweetTextFeatures(penguinVersion);\n    if (textFeatures != null) {\n      final FeatureConfiguration featureConfigNumHashtags = schema.getFeatureConfigurationByName(\n          EarlybirdFieldConstant.NUM_HASHTAGS.getFieldName());\n      final FeatureConfiguration featureConfigNumMentions = schema.getFeatureConfigurationByName(\n          EarlybirdFieldConstant.NUM_MENTIONS.getFieldName());\n\n      sink.setNumericValue(\n              EarlybirdFieldConstant.NUM_HASHTAGS,\n              Math.min(textFeatures.getHashtagsSize(), featureConfigNumHashtags.getMaxValue()))\n          .setNumericValue(\n              EarlybirdFieldConstant.NUM_MENTIONS,\n              Math.min(textFeatures.getMentionsSize(), featureConfigNumMentions.getMaxValue()))\n          .setBooleanValue(\n              EarlybirdFieldConstant.HAS_MULTIPLE_HASHTAGS_OR_TRENDS_FLAG,\n              TwitterMessage.hasMultipleHashtagsOrTrends(textFeatures))\n          .setBooleanValue(\n              EarlybirdFieldConstant.HAS_TREND_FLAG,\n              textFeatures.getTrendingTermsSize() > 0);\n    }\n\n    TweetTextQuality textQuality = message.getTweetTextQuality(penguinVersion);\n    if (textQuality != null) {\n      sink.setNumericValue(EarlybirdFieldConstant.TEXT_SCORE, textQuality.getTextScore());\n      sink.setBooleanValue(\n          EarlybirdFieldConstant.IS_OFFENSIVE_FLAG,\n          textQuality.hasBoolQuality(TweetTextQuality.BooleanQualityType.OFFENSIVE)\n              || textQuality.hasBoolQuality(TweetTextQuality.BooleanQualityType.OFFENSIVE_USER)\n              // Note: if json message \"possibly_sensitive\" flag is set, we consider the tweet\n              // sensitive and is currently filtered out in safe search mode via a hacky setup:\n              // earlybird does not create _filter_sensitive_content field, only\n              // _is_offensive field is created, and used in filter:safe operator\n              || textQuality.hasBoolQuality(TweetTextQuality.BooleanQualityType.SENSITIVE));\n      if (textQuality.hasBoolQuality(TweetTextQuality.BooleanQualityType.SENSITIVE)) {\n        sink.setBooleanValue(EarlybirdFieldConstant.IS_SENSITIVE_CONTENT, true);\n      }\n    } else {\n      // we don't have text score, for whatever reason, set to sentinel value so we won't be\n      // skipped by scoring function\n      sink.setNumericValue(EarlybirdFieldConstant.TEXT_SCORE,\n          RelevanceSignalConstants.UNSET_TEXT_SCORE_SENTINEL);\n    }\n\n    if (message.isSetLocale()) {\n      sink.setNumericValue(EarlybirdFieldConstant.LANGUAGE,\n          ThriftLanguageUtil.getThriftLanguageOf(message.getLocale()).getValue());\n    }\n\n    // User features\n    TweetUserFeatures userFeatures = message.getTweetUserFeatures(penguinVersion);\n    if (userFeatures != null) {\n      sink.setBooleanValue(EarlybirdFieldConstant.IS_USER_SPAM_FLAG, userFeatures.isSpam())\n          .setBooleanValue(EarlybirdFieldConstant.IS_USER_NSFW_FLAG, userFeatures.isNsfw())\n          .setBooleanValue(EarlybirdFieldConstant.IS_USER_BOT_FLAG, userFeatures.isBot());\n    }\n    if (message.getUserReputation() != TwitterMessage.DOUBLE_FIELD_NOT_PRESENT) {\n      sink.setNumericValue(EarlybirdFieldConstant.USER_REPUTATION,\n          (byte) message.getUserReputation());\n    } else {\n      sink.setNumericValue(EarlybirdFieldConstant.USER_REPUTATION,\n          RelevanceSignalConstants.UNSET_REPUTATION_SENTINEL);\n    }\n\n    sink.setBooleanValue(EarlybirdFieldConstant.IS_NULLCAST_FLAG, message.getNullcast());\n\n    // Realtime Ingestion does not write engagement features. Updater does that.\n    if (message.getNumFavorites() > 0) {\n      sink.setNumericValue(EarlybirdFieldConstant.FAVORITE_COUNT,\n          MutableFeatureNormalizers.BYTE_NORMALIZER.normalize(message.getNumFavorites()));\n    }\n    if (message.getNumRetweets() > 0) {\n      sink.setNumericValue(EarlybirdFieldConstant.RETWEET_COUNT,\n          MutableFeatureNormalizers.BYTE_NORMALIZER.normalize(message.getNumRetweets()));\n    }\n    if (message.getNumReplies() > 0) {\n      sink.setNumericValue(EarlybirdFieldConstant.REPLY_COUNT,\n          MutableFeatureNormalizers.BYTE_NORMALIZER.normalize(message.getNumReplies()));\n    }\n\n    sink.setNumericValue(EarlybirdFieldConstant.VISIBLE_TOKEN_RATIO, normalizedTokenPercentBucket);\n\n    EarlybirdEncodedFeatures encodedFeatures =\n        (EarlybirdEncodedFeatures) sink.getFeaturesForBaseField(\n            EarlybirdFieldConstant.ENCODED_TWEET_FEATURES_FIELD.getFieldName());\n    updateLinkEncodedFeatures(encodedFeatures, message);\n    return encodedFeatures;\n  }\n\n  /**\n   * Returns the extended encoded features.\n   */\n  public static EarlybirdEncodedFeatures createExtendedEncodedFeaturesFromTwitterMessage(\n    TwitterMessage message,\n    PenguinVersion penguinVersion,\n    ImmutableSchemaInterface schema) {\n    FeatureSink sink = new FeatureSink(schema);\n\n    TweetTextFeatures textFeatures = message.getTweetTextFeatures(penguinVersion);\n\n    if (textFeatures != null) {\n      setExtendedEncodedFeatureIntValue(sink, schema,\n          EarlybirdFieldConstant.NUM_HASHTAGS_V2, textFeatures.getHashtagsSize());\n      setExtendedEncodedFeatureIntValue(sink, schema,\n          EarlybirdFieldConstant.NUM_MENTIONS_V2, textFeatures.getMentionsSize());\n      setExtendedEncodedFeatureIntValue(sink, schema,\n          EarlybirdFieldConstant.NUM_STOCKS, textFeatures.getStocksSize());\n    }\n\n    Optional<Long> referenceAuthorId = message.getReferenceAuthorId();\n    if (referenceAuthorId.isPresent()) {\n      setEncodedReferenceAuthorId(sink, referenceAuthorId.get());\n    }\n\n    return (EarlybirdEncodedFeatures) sink.getFeaturesForBaseField(\n        EarlybirdFieldConstant.EXTENDED_ENCODED_TWEET_FEATURES_FIELD.getFieldName());\n  }\n\n  /**\n   * Updates all URL-related features, based on the values stored in the given message.\n   *\n   * @param encodedFeatures The features to be updated.\n   * @param message The message.\n   */\n  public static void updateLinkEncodedFeatures(\n      EarlybirdEncodedFeatures encodedFeatures, TwitterMessage message) {\n    if (message.getLinkLocale() != null) {\n      encodedFeatures.setFeatureValue(\n          EarlybirdFieldConstant.LINK_LANGUAGE,\n          ThriftLanguageUtil.getThriftLanguageOf(message.getLinkLocale()).getValue());\n    }\n\n    if (message.hasCard()) {\n      encodedFeatures.setFlag(EarlybirdFieldConstant.HAS_CARD_FLAG);\n    }\n\n    // Set HAS_IMAGE HAS_NEWS HAS_VIDEO etc. flags for expanded urls.\n    if (message.getExpandedUrlMapSize() > 0) {\n      encodedFeatures.setFlag(EarlybirdFieldConstant.HAS_LINK_FLAG);\n\n      for (ThriftExpandedUrl url : message.getExpandedUrlMap().values()) {\n        if (url.isSetMediaType()) {\n          switch (url.getMediaType()) {\n            case NATIVE_IMAGE:\n              encodedFeatures.setFlag(EarlybirdFieldConstant.HAS_IMAGE_URL_FLAG);\n              encodedFeatures.setFlag(EarlybirdFieldConstant.HAS_NATIVE_IMAGE_FLAG);\n              break;\n            case IMAGE:\n              encodedFeatures.setFlag(EarlybirdFieldConstant.HAS_IMAGE_URL_FLAG);\n              break;\n            case VIDEO:\n              encodedFeatures.setFlag(EarlybirdFieldConstant.HAS_VIDEO_URL_FLAG);\n              break;\n            case NEWS:\n              encodedFeatures.setFlag(EarlybirdFieldConstant.HAS_NEWS_URL_FLAG);\n              break;\n            case UNKNOWN:\n              break;\n            default:\n              throw new IllegalStateException(\"Unexpected enum value: \" + url.getMediaType());\n          }\n        }\n      }\n    }\n\n    Set<String> canonicalLastHopUrlsStrings = message.getCanonicalLastHopUrls();\n    Set<String> expandedUrlsStrings = message.getExpandedUrls()\n        .stream()\n        .map(ThriftExpandedUrl::getExpandedUrl)\n        .collect(Collectors.toSet());\n    Set<String> expandedAndLastHopUrlsStrings = new HashSet<>();\n    expandedAndLastHopUrlsStrings.addAll(expandedUrlsStrings);\n    expandedAndLastHopUrlsStrings.addAll(canonicalLastHopUrlsStrings);\n    // Check both expanded and last hop url for consumer videos as consumer video urls are\n    // sometimes redirected to the url of the tweets containing the videos (SEARCH-42612).\n    if (NativeVideoClassificationUtils.hasConsumerVideo(expandedAndLastHopUrlsStrings)) {\n      encodedFeatures.setFlag(EarlybirdFieldConstant.HAS_CONSUMER_VIDEO_FLAG);\n    }\n    if (NativeVideoClassificationUtils.hasProVideo(canonicalLastHopUrlsStrings)) {\n      encodedFeatures.setFlag(EarlybirdFieldConstant.HAS_PRO_VIDEO_FLAG);\n    }\n    if (NativeVideoClassificationUtils.hasVine(canonicalLastHopUrlsStrings)) {\n      encodedFeatures.setFlag(EarlybirdFieldConstant.HAS_VINE_FLAG);\n    }\n    if (NativeVideoClassificationUtils.hasPeriscope(canonicalLastHopUrlsStrings)) {\n      encodedFeatures.setFlag(EarlybirdFieldConstant.HAS_PERISCOPE_FLAG);\n    }\n    if (LinkVisibilityUtils.hasVisibleLink(message.getExpandedUrls())) {\n      encodedFeatures.setFlag(EarlybirdFieldConstant.HAS_VISIBLE_LINK_FLAG);\n    }\n  }\n\n  private static void setExtendedEncodedFeatureIntValue(\n      FeatureSink sink,\n      ImmutableSchemaInterface schema,\n      EarlybirdFieldConstant field,\n      int value) {\n    boolean fieldInSchema = schema.hasField(field.getFieldName());\n    if (fieldInSchema) {\n      FeatureConfiguration featureConfig =\n          schema.getFeatureConfigurationByName(field.getFieldName());\n      sink.setNumericValue(field, Math.min(value, featureConfig.getMaxValue()));\n    }\n  }\n\n  private static void setEncodedReferenceAuthorId(FeatureSink sink, long referenceAuthorId) {\n    LongIntConverter.IntegerRepresentation ints =\n        LongIntConverter.convertOneLongToTwoInt(referenceAuthorId);\n    sink.setNumericValue(\n        EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_LEAST_SIGNIFICANT_INT, ints.leastSignificantInt);\n    sink.setNumericValue(\n        EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_MOST_SIGNIFICANT_INT, ints.mostSignificantInt);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/docvalues/BUILD",
    "content": "# Java library for docvalues and common stride field encoding utilities.\njava_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    provides = artifact(\n        org = \"com.twitter.search.common\",\n        name = \"encoding-docvalues\",\n        repo = artifactory,\n    ),\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-common\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-facet\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/thrift/com/twitter/search/common:schema-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/docvalues/CSFTypeUtil.java",
    "content": "package com.twitter.search.common.encoding.docvalues;\n\npublic final class CSFTypeUtil {\n  private CSFTypeUtil() {\n  }\n\n  /**\n   * Convert a long into a byte array, stored into dest.\n   */\n  public static void convertToBytes(byte[] dest, int valueIndex, int value) {\n    int offset = valueIndex * Integer.BYTES;\n    dest[offset] = (byte) (value >>> 24);\n    dest[offset + 1] = (byte) (value >>> 16);\n    dest[offset + 2] = (byte) (value >>> 8);\n    dest[offset + 3] = (byte) value;\n  }\n\n  /**\n   * Convert bytes into a long value. Inverse function of convertToBytes.\n   */\n  public static int convertFromBytes(byte[] data, int startOffset, int valueIndex) {\n    // This should rarely happen, eg. when we get a corrupt ThriftIndexingEvent, we insert a new\n    // Document which is blank. Such a document results in a length 0 BytesRef.\n    if (data.length == 0) {\n      return 0;\n    }\n\n    int offset = startOffset + valueIndex * Integer.BYTES;\n    return ((data[offset] & 0xFF) << 24)\n        | ((data[offset + 1] & 0xFF) << 16)\n        | ((data[offset + 2] & 0xFF) << 8)\n        | (data[offset + 3] & 0xFF);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/features/BUILD",
    "content": "# Java library for feature encoding and decoding utilities.\njava_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    provides = artifact(\n        org = \"com.twitter.search.common\",\n        name = \"encoding-features\",\n        repo = artifactory,\n    ),\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/thrift/com/twitter/search/common:indexing-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/features/BinByteNormalizer.java",
    "content": "package com.twitter.search.common.encoding.features;\n\nimport java.util.Map;\nimport java.util.SortedSet;\nimport java.util.TreeMap;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Maps;\nimport com.google.common.collect.Sets;\n\n/**\n * Normalizes values to predefined bins.\n * If the value to normalize is lower than the lowest bin defined, normalizes to Byte.MIN_VALUE.\n */\npublic class BinByteNormalizer extends ByteNormalizer {\n\n  private final TreeMap<Double, Byte> bins = Maps.newTreeMap();\n  private final TreeMap<Byte, Double> reverseBins = Maps.newTreeMap();\n\n  /**\n   * Constructs a normalizer using predefined bins.\n   * @param bins A mapping between the upper bound of a value and the bin it should normalize to.\n   * For example providing a map with 2 entries, {5=>1, 10=>2} will normalize as follows:\n   *   values under 5: Byte.MIN_VALUE\n   *   values between 5 and 10: 1\n   *   values over 10: 2\n   */\n  public BinByteNormalizer(final Map<Double, Byte> bins) {\n    Preconditions.checkNotNull(bins);\n    Preconditions.checkArgument(!bins.isEmpty(), \"No bins provided\");\n    Preconditions.checkArgument(hasIncreasingValues(bins));\n    this.bins.putAll(bins);\n    for (Map.Entry<Double, Byte> entry : bins.entrySet()) {\n      reverseBins.put(entry.getValue(), entry.getKey());\n    }\n  }\n\n  /**\n   * check that if key1 > key2 then val1 > val2 in the {@code map}.\n   */\n  private static boolean hasIncreasingValues(final Map<Double, Byte> map) {\n    SortedSet<Double> orderedKeys = Sets.newTreeSet(map.keySet());\n    byte prev = Byte.MIN_VALUE;\n    for (Double key : orderedKeys) { // save the unboxing\n      byte cur = map.get(key);\n      if (cur <= prev) {\n        return false;\n      }\n      prev = cur;\n    }\n    return true;\n  }\n\n  @Override\n  public byte normalize(double val) {\n    Map.Entry<Double, Byte> lowerBound = bins.floorEntry(val);\n    return lowerBound == null\n        ? Byte.MIN_VALUE\n        : lowerBound.getValue();\n    }\n\n  @Override\n  public double unnormLowerBound(byte norm) {\n    return reverseBins.get(reverseBins.floorKey(norm));\n  }\n\n  @Override\n  public double unnormUpperBound(byte norm) {\n    return norm == reverseBins.lastKey()\n        ? Double.POSITIVE_INFINITY\n        : reverseBins.get(reverseBins.floorKey((byte) (1 + norm)));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/features/ByteNormalizer.java",
    "content": "package com.twitter.search.common.encoding.features;\n\n/**\n * Interface for compressing unbounded float values to a signed byte. It includes both\n * normalization of values and encoding of values in a byte.\n */\npublic abstract class ByteNormalizer {\n  public static byte intToUnsignedByte(int i) {\n    return (byte) i;\n  }\n\n  public static int unsignedByteToInt(byte b) {\n    return (int) b & 0xFF;\n  }\n\n  /**\n   * Returns the byte-compressed value of {@code val}.\n   */\n  public abstract byte normalize(double val);\n\n  /**\n   * Returns a lower bound to the unnormalized range of {@code norm}.\n   */\n  public abstract double unnormLowerBound(byte norm);\n\n  /**\n   * Returns an upper bound to the unnormalized range of {@code norm}.\n   */\n  public abstract double unnormUpperBound(byte norm);\n\n  /**\n   * Returns true if the normalized value of {@code val} is different than the normalized value of\n   * {@code val - 1}\n   */\n  public boolean changedNorm(double val) {\n    return normalize(val) != normalize(val - 1);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/features/ClampByteNormalizer.java",
    "content": "package com.twitter.search.common.encoding.features;\n\nimport com.google.common.base.Preconditions;\n\n/**\n * A byte normalizer that restricts the values to the given range before normalizing them.\n */\npublic class ClampByteNormalizer extends ByteNormalizer {\n  private final int minUnnormalizedValue;\n  private final int maxUnnormalizedValue;\n\n  /**\n   * Creates a new ClampByteNormalizer instance.\n   *\n   * @param minValue The smallest allowed unnormalized value.\n   * @param maxValue The largest allowed unnormalized value.\n   */\n  public ClampByteNormalizer(int minUnnormalizedValue, int maxUnnormalizedValue) {\n    Preconditions.checkState(minUnnormalizedValue <= maxUnnormalizedValue);\n    Preconditions.checkState(minUnnormalizedValue >= 0);\n    Preconditions.checkState(maxUnnormalizedValue <= 255);\n    this.minUnnormalizedValue = minUnnormalizedValue;\n    this.maxUnnormalizedValue = maxUnnormalizedValue;\n  }\n\n  @Override\n  public byte normalize(double val) {\n    int adjustedValue = (int) val;\n    if (adjustedValue < minUnnormalizedValue) {\n      adjustedValue = minUnnormalizedValue;\n    }\n    if (adjustedValue > maxUnnormalizedValue) {\n      adjustedValue = maxUnnormalizedValue;\n    }\n    return ByteNormalizer.intToUnsignedByte(adjustedValue);\n  }\n\n  @Override\n  public double unnormLowerBound(byte norm) {\n    return ByteNormalizer.unsignedByteToInt(norm);\n  }\n\n  @Override\n  public double unnormUpperBound(byte norm) {\n    return ByteNormalizer.unsignedByteToInt(norm) + 1;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/features/EncodedFeatures.java",
    "content": "package com.twitter.search.common.encoding.features;\n\n/**\n * Encodes multiple values (bytes or bits) into an integer.\n */\npublic class EncodedFeatures {\n  private int value;\n\n  public final void setSerializedValue(int val) {\n    this.value = val;\n  }\n\n  public final int getSerializedValue() {\n    return value;\n  }\n\n  // setByte is agnostic to signed / unsigned bytes.\n  protected final EncodedFeatures setByte(byte count, int bitshift, long inverseMask) {\n    value = (int) ((value & inverseMask) | ((count & 0xffL) << bitshift));\n    return this;\n  }\n\n  /**\n   * Sets the value but only if greater. setByteIfGreater assumes unsigned bytes.\n   */\n  public final EncodedFeatures setByteIfGreater(byte newCount, int bitshift, long inversemask) {\n    if ((getByte(bitshift) & 0xff) < (newCount & 0xff)) {\n      setByte(newCount, bitshift, inversemask);\n    }\n    return this;\n  }\n\n  protected final int getByte(int bitshift) {\n    return (int) (((value & 0xffffffffL) >>> bitshift) & 0xffL);\n  }\n\n  protected final int getByteMasked(int bitshift, long mask) {\n    return (int) (((value & mask) >>> bitshift) & 0xffL);\n  }\n\n  protected final EncodedFeatures setBit(int bit, boolean flag) {\n    if (flag) {\n      value |= bit;\n    } else {\n      value &= ~bit;\n    }\n    return this;\n  }\n\n  protected final boolean getBit(int bit) {\n    return (value & bit) != 0;\n  }\n\n  @Override\n  public String toString() {\n    return String.format(\"%x\", value);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/features/IntNormalizer.java",
    "content": "package com.twitter.search.common.encoding.features;\n\n/**\n * Interface for processing different feature values into an int. It provides a one-way translation\n * of encoding using com.twitter.search.common.encoding.features.ByteNormalizer and supports all the\n * old normalizers. The difference is that we directly return the normalized int value\n * (instead of converting from byte).\n */\npublic interface IntNormalizer {\n  /**\n   * Returns the normalized value of {@code val}.\n   * The value may be byte-compressed or as-is depending on the normalizer type\n   */\n  int normalize(double val);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/features/IntegerEncodedFeatures.java",
    "content": "package com.twitter.search.common.encoding.features;\n\nimport java.util.List;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\n\nimport com.twitter.search.common.indexing.thriftjava.PackedFeatures;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\n\n/**\n * Class used to read/write integers encoded according to\n * {@link com.twitter.search.common.schema.base.FeatureConfiguration}\n *\n * Implementations must override {@link #getInt(int pos)} and {@link #setInt(int pos, int value)}.\n */\npublic abstract class IntegerEncodedFeatures {\n  /**\n   * Returns the value at the given position.\n   */\n  public abstract int getInt(int pos);\n\n  /**\n   * Sets the given value at the given position.\n   */\n  public abstract void setInt(int pos, int value);\n\n  /**\n   * Get the maximum number of integers to hold features.\n   * @return the number of integers to represent all features.\n   */\n  public abstract int getNumInts();\n\n  /**\n   * Test to see if the given feature is true or non-zero. Useful for one bit features.\n   * @param feature feature to examine\n   * @return true if feature is non-zero\n   */\n  public boolean isFlagSet(FeatureConfiguration feature) {\n    return (getInt(feature.getValueIndex()) & feature.getBitMask()) != 0;\n  }\n\n  public IntegerEncodedFeatures setFlag(FeatureConfiguration feature) {\n    setInt(feature.getValueIndex(), getInt(feature.getValueIndex()) | feature.getBitMask());\n    return this;\n  }\n\n  public IntegerEncodedFeatures clearFlag(FeatureConfiguration feature) {\n    setInt(feature.getValueIndex(), getInt(feature.getValueIndex()) & feature.getInverseBitMask());\n    return this;\n  }\n\n  /**\n   * Sets a boolean flag.\n   */\n  public IntegerEncodedFeatures setFlagValue(FeatureConfiguration feature, boolean value) {\n    if (value) {\n      setFlag(feature);\n    } else {\n      clearFlag(feature);\n    }\n    return this;\n  }\n\n  /**\n   * Get feature value\n   * @param feature feature to get\n   * @return the value of the feature\n   */\n  public int getFeatureValue(FeatureConfiguration feature) {\n    return (getInt(feature.getValueIndex()) & feature.getBitMask())\n            >>> feature.getBitStartPosition();\n  }\n\n  /**\n   * Set feature value\n   * @param feature feature to modify\n   * @param value value to set.\n   */\n  public IntegerEncodedFeatures setFeatureValue(FeatureConfiguration feature, int value) {\n    Preconditions.checkState(\n        value <= feature.getMaxValue(),\n        \"Feature value, %s, is greater than the max value allowed for this feature. \"\n            + \"Feature: %s, Max value: %s\",\n        value, feature.getName(), feature.getMaxValue());\n\n    // Clear the value of the given feature in its int.\n    int temp = getInt(feature.getValueIndex()) & feature.getInverseBitMask();\n\n    // Set the new feature value. Applying the bit mask here ensures that other features in the\n    // same int are not modified by mistake.\n    temp |= (value << feature.getBitStartPosition()) & feature.getBitMask();\n\n    setInt(feature.getValueIndex(), temp);\n    return this;\n  }\n\n  /**\n   * Sets feature value if greater than current value\n   * @param feature feature to modify\n   * @param value new value\n   */\n  public IntegerEncodedFeatures setFeatureValueIfGreater(FeatureConfiguration feature, int value) {\n    if (value > getFeatureValue(feature)) {\n      setFeatureValue(feature, value);\n    }\n    return this;\n  }\n\n  /**\n   * Increment a feature if its not at its maximum value.\n   * @return whether the feature is incremented.\n   */\n  public boolean incrementIfNotMaximum(FeatureConfiguration feature) {\n    int newValue = getFeatureValue(feature) + 1;\n    if (newValue <= feature.getMaxValue()) {\n      setFeatureValue(feature, newValue);\n      return true;\n    } else {\n      return false;\n    }\n  }\n\n  /**\n   * Copy these encoded features to a new PackedFeatures thrift struct.\n   */\n  public PackedFeatures copyToPackedFeatures() {\n    return copyToPackedFeatures(new PackedFeatures());\n  }\n\n  /**\n    * Copy these encoded features to a PackedFeatures thrift struct.\n    */\n  public PackedFeatures copyToPackedFeatures(PackedFeatures packedFeatures) {\n    Preconditions.checkNotNull(packedFeatures);\n    final List<Integer> integers = Lists.newArrayListWithCapacity(getNumInts());\n    for (int i = 0; i < getNumInts(); i++) {\n      integers.add(getInt(i));\n    }\n    packedFeatures.setDeprecated_featureConfigurationVersion(0);\n    packedFeatures.setFeatures(integers);\n    return packedFeatures;\n  }\n\n  /**\n   * Copy features from a packed features struct.\n   */\n  public void readFromPackedFeatures(PackedFeatures packedFeatures) {\n    Preconditions.checkNotNull(packedFeatures);\n    List<Integer> ints = packedFeatures.getFeatures();\n    for (int i = 0; i < getNumInts(); i++) {\n      if (i < ints.size()) {\n        setInt(i, ints.get(i));\n      } else {\n        setInt(i, 0);\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/features/LogByteNormalizer.java",
    "content": "package com.twitter.search.common.encoding.features;\n\nimport com.google.common.base.Preconditions;\n\n/**\n * Normalizes values as follows:\n *   Positive numbers normalize to (1 + round(log_baseN(value))).\n *   Negative numbers throw.\n *   0 will normalize to 0.\n * The log base is 2 by default.\n */\npublic class LogByteNormalizer extends ByteNormalizer {\n\n  private static final double DEFAULT_BASE = 2;\n  private final double base;\n  private final double logBase;\n\n  public LogByteNormalizer(double base) {\n    Preconditions.checkArgument(base > 0);\n    this.base = base;\n    logBase = Math.log(base);\n  }\n\n  public LogByteNormalizer() {\n    this(DEFAULT_BASE);\n  }\n\n  @Override\n  public byte normalize(double val) {\n    if (val < 0) {\n      throw new IllegalArgumentException(\"Can't log-normalize negative value \" + val);\n    } else if (val == 0) {\n      return 0;\n    } else {\n      long logVal = 1 + (long) Math.floor(Math.log(val) / logBase);\n      return logVal > Byte.MAX_VALUE ? Byte.MAX_VALUE : (byte) logVal;\n    }\n  }\n\n  @Override\n  public double unnormLowerBound(byte norm) {\n    return norm < 0\n        ? Double.NEGATIVE_INFINITY\n        : Math.floor(Math.pow(base, norm - 1));\n  }\n\n  @Override\n  public double unnormUpperBound(byte norm) {\n    return norm == Byte.MAX_VALUE\n        ? Double.POSITIVE_INFINITY\n        : Math.floor(Math.pow(base, norm));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/features/PredictionScoreNormalizer.java",
    "content": "package com.twitter.search.common.encoding.features;\n\nimport com.google.common.base.Preconditions;\n\n/**\n * A normalizer that normalizes the prediction score from a machine learning classifier, which\n * ranges within [0.0, 1.0], to an integer value by multiplying by (10 ^ precision), and returns\n * the rounded value. The lower the precision, the less amount of bits it takes to encode the score.\n * @see #precision\n *\n * This normalizer also could denormalize the normalized value from integer back to double using the\n * same precision.\n */\npublic class PredictionScoreNormalizer {\n\n  private final int precision;\n  private final double normalizingBase;\n\n  public PredictionScoreNormalizer(int precision) {\n    this.precision = precision;\n    this.normalizingBase = Math.pow(10, this.precision);\n  }\n\n  /**\n   * Returns the normalized int value for prediction score {@code score} by multiplying\n   * by {@code normalizingBase}, and round the result.\n   * @throws IllegalArgumentException when parameter {@code score} is not within [0.0, 1.0]\n   */\n  public int normalize(double score) {\n    Preconditions.checkArgument(isScoreWithinRange(score));\n    return (int) Math.round(score * this.normalizingBase);\n  }\n\n  /**\n   * Converts the normalized int value back to a double score by dividing by {@code normalizingBase}\n   * @throws IllegalStateException when the denormalized value is not within [0.0, 1.0]\n   */\n  public double denormalize(int normalizedScore) {\n    double denormalizedValue = normalizedScore / this.normalizingBase;\n    if (!isScoreWithinRange(denormalizedValue)) {\n      throw new IllegalStateException(\n          String.format(\"The denormalized value %s is not within [0.0, 1.0]\", denormalizedValue)\n      );\n    }\n    return denormalizedValue;\n  }\n\n  private static boolean isScoreWithinRange(double score) {\n    return 0.0 <= score && score <= 1.0;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/features/SingleBytePositiveFloatNormalizer.java",
    "content": "package com.twitter.search.common.encoding.features;\n\n/**\n * Normalizes using the logic described in {@link SingleBytePositiveFloatUtil}.\n */\npublic class SingleBytePositiveFloatNormalizer extends ByteNormalizer {\n\n  @Override\n  public byte normalize(double val) {\n    return SingleBytePositiveFloatUtil.toSingleBytePositiveFloat((float) val);\n  }\n\n  @Override\n  public double unnormLowerBound(byte norm) {\n    return SingleBytePositiveFloatUtil.toJavaFloat(norm);\n  }\n\n  /**\n   * Get the upper bound of the raw value for a normalized byte.\n   * @deprecated This is wrongly implemented, always use unnormLowerBound(),\n   * or use SmartIntegerNormalizer.\n   */\n  @Override @Deprecated\n  public double unnormUpperBound(byte norm) {\n    return 1 + SingleBytePositiveFloatUtil.toJavaFloat(norm);\n  }\n\n  /**\n   * Return the the post-log2 unnormalized value. This is only used for some legacy Earlybird\n   * features and scoring functions.\n   */\n  public double unnormAndLog2(byte norm) {\n    return SingleBytePositiveFloatUtil.toLog2Double(norm);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/features/SingleBytePositiveFloatUtil.java",
    "content": "package com.twitter.search.common.encoding.features;\n\n/**\n * Util used to:\n *   - Encode a positive Java float into a single byte float\n *   - Decode a single byte into a positive Java float\n *\n * Configuration:\n *   - Exponent: higher 4 bits, base 10.\n *   - Mantissa: lower 4 bit, representing 1.0 to 9.0\n *   - Exponent bias is 1.\n *\n * Formula:\n *   Max(Mantissa, 9) * 10 ^ (Exponent - 1)\n *\n * Smallest float: 0.0                        (0000 0000)\n * Smallest positive float: 1.0 * 10^-1       (0000 0001)\n * Largest float: 9.0 * 10^13                 (1110 1111)\n * Infinity:                                  (1111 0000)\n * NaN:                                       (1111 1000)\n */\npublic final class SingleBytePositiveFloatUtil {\n  private SingleBytePositiveFloatUtil() { }\n\n  // 4 bits mantissa. Range [1.0, 10.0) is divided into 16 steps\n  public static final byte MAX_BYTE_VALUE = (byte) 0xEF;\n  public static final byte INFINITY = (byte) 0xF0;\n  public static final byte NOT_A_NUMBER = (byte) 0xF8;\n  private static final float STEP_SIZE = 1.0f;\n  private static final int EXPONENT_BIAS = 1;\n  private static final byte MIN_EXPONENT = -EXPONENT_BIAS;\n  private static final int MAX_EXPONENT = 14 - EXPONENT_BIAS;\n  private static final byte MANTISSA_MASK = 0x0F;\n\n  /**\n   * Converts the given float into a single byte floating point number.\n   * This is used in the updater and OK to be a bit slow.\n   */\n  public static byte toSingleBytePositiveFloat(float f) {\n    if (f < 0) {\n      throw new UnsupportedOperationException(\n          \"Cannot encode negative floats into SingleBytePostiveFloat.\");\n    }\n\n    if (Float.compare(f, Float.POSITIVE_INFINITY) == 0) {\n      return INFINITY;\n    }\n\n    if (Float.compare(f, Float.NaN) == 0) {\n      return NOT_A_NUMBER;\n    }\n\n    int mantissa = 0;\n    int exponent = (int) Math.floor(Math.log10(f));\n    // Overflow (Number too large), just return the largest possible value\n    if (exponent > MAX_EXPONENT) {\n      return MAX_BYTE_VALUE;\n    }\n\n    // Underflow (Number too small), just return 0\n    if (exponent < MIN_EXPONENT) {\n      return 0;\n    }\n\n    int frac = Math.round(f / (float) Math.pow(10.0f, exponent) / STEP_SIZE);\n    mantissa = fractionToMantissaTable[frac];\n\n    return (byte) (((exponent + EXPONENT_BIAS) << 4) | mantissa);\n  }\n\n  /**\n   * Called in Earlybird per hit and needs to be fast.\n   */\n  public static float toJavaFloat(byte b) {\n    return BYTE_TO_FLOAT_CONVERSION_TABLE[b & 0xff];\n  }\n\n  // Table used for converting mantissa into a significant\n  private static float[] mantissaToFractionTable = {\n    //   Decimal        Matisa value\n      STEP_SIZE * 0,   // 0000\n      STEP_SIZE * 1,   // 0001\n      STEP_SIZE * 1,   // 0010\n      STEP_SIZE * 2,   // 0011\n      STEP_SIZE * 2,   // 0100\n      STEP_SIZE * 3,   // 0101\n      STEP_SIZE * 3,   // 0110\n      STEP_SIZE * 4,   // 0111\n      STEP_SIZE * 4,   // 1000\n      STEP_SIZE * 5,   // 1001\n      STEP_SIZE * 5,   // 1010\n      STEP_SIZE * 6,   // 1011\n      STEP_SIZE * 6,   // 1100\n      STEP_SIZE * 7,   // 1101\n      STEP_SIZE * 8,   // 1110\n      STEP_SIZE * 9    // 1111\n  };\n\n  // Table used for converting fraction into mantissa.\n  // Reverse operation of the above\n  private static int[] fractionToMantissaTable = {\n      0,  // 0\n      1,  // 1\n      3,  // 2\n      5,  // 3\n      7,  // 4\n      9,  // 5\n      11,  // 6\n      13,  // 7\n      14,  // 8\n      15,  // 9\n      15,  // 10 (Edge case: because we round the fraction, we can get 10 here.)\n  };\n\n  public static final byte LARGEST_FRACTION_UNDER_ONE = (byte) (toSingleBytePositiveFloat(1f) - 1);\n\n  /**\n   * Converts the given byte to java float.\n   */\n  private static float toJavaFloatSlow(byte b) {\n    if (b == INFINITY) {\n      return Float.POSITIVE_INFINITY;\n    }\n\n    if ((b & 0xff) > (INFINITY & 0xff)) {\n      return Float.NaN;\n    }\n\n    int exponent = ((b & 0xff) >>> 4) - EXPONENT_BIAS;\n    int mantissa = b & MANTISSA_MASK;\n    return mantissaToFractionTable[mantissa] * (float) Math.pow(10.0f, exponent);\n  }\n\n  // Cached results from byte to float conversion\n  private static final float[] BYTE_TO_FLOAT_CONVERSION_TABLE = new float[256];\n  private static final double[] BYTE_TO_LOG2_CONVERSION_TABLE = new double[256];\n  private static final byte[] OLD_TO_NEW_BYTE_CONVERSION_TABLE = new byte[256];\n\n  static {\n    LogByteNormalizer normalizer = new LogByteNormalizer();\n    for (int i = 0; i < 256; i++) {\n      byte b = (byte) i;\n      BYTE_TO_FLOAT_CONVERSION_TABLE[i] = toJavaFloatSlow(b);\n      BYTE_TO_LOG2_CONVERSION_TABLE[i] =\n          0xff & normalizer.normalize(BYTE_TO_FLOAT_CONVERSION_TABLE[i]);\n      if (b == 0) {\n        OLD_TO_NEW_BYTE_CONVERSION_TABLE[i] = 0;\n      } else if (b > 0) {\n        OLD_TO_NEW_BYTE_CONVERSION_TABLE[i] =\n            toSingleBytePositiveFloat((float) normalizer.unnormLowerBound(b));\n      } else {\n        // should not get here.\n        OLD_TO_NEW_BYTE_CONVERSION_TABLE[i] = MAX_BYTE_VALUE;\n      }\n    }\n  }\n\n  /**\n   * Convert a normalized byte to the log2() version of its original value\n   */\n  static double toLog2Double(byte b) {\n    return BYTE_TO_LOG2_CONVERSION_TABLE[b & 0xff];\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/encoding/features/SmartIntegerNormalizer.java",
    "content": "package com.twitter.search.common.encoding.features;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\n/**\n * A smart integer normalizer that converts an integer of a known range to a small integer up to\n * 8 bits long. This normalizer generates a boundary value array in the constructor as the buckets\n * for different values.\n * <p/>\n * The normalized value has a nice properties:\n * 1) it maintains the order of original value: if a > b, then normalize(a) > normalize(b).\n * 2) the value 0 is always normalized to byte 0.\n * 3) the normalized values are (almost) evenly distributed on the log scale\n * 4) no waste in code space, all possible values representable by normalized bits are used,\n * each corresponding to a different value.\n */\npublic class SmartIntegerNormalizer extends ByteNormalizer {\n  // The max value we want to support in this normalizer. If the input is larger than this value,\n  // it's normalized as if it's the maxValue.\n  private final int maxValue;\n  // Number of bits used for normalized value, the largest normalized value\n  // would be (1 << numBits) - 1.\n  private final int numBits;\n  // The inclusive lower bounds of all buckets. A normalized value k corresponds to original values\n  // in the inclusive-exclusive range\n  //   [ boundaryValues[k], boundaryValues[k+1] )\n  private final int[] boundaryValues;\n  // The length of the boundaryValues array, or the number of buckets.\n  private final int length;\n\n  /**\n   * Construct a normalizer.\n   *\n   * @param maxValue max value it supports, must be larger than minValue. Anything larger than this\n   * would be treated as maxValue.\n   * @param numBits number of bits you want to use for this normalization, between 1 and 8.\n   * higher resolution for the lower numbers.\n   */\n  public SmartIntegerNormalizer(int maxValue, int numBits) {\n    Preconditions.checkArgument(maxValue > 0);\n    Preconditions.checkArgument(numBits > 0 && numBits <= 8);\n\n    this.maxValue = maxValue;\n    this.numBits = numBits;\n\n    this.length = 1 << numBits;\n    this.boundaryValues = new int[length];\n\n\n    int index;\n    for (index = length - 1; index >= 0; --index) {\n      // values are evenly distributed on the log scale\n      int boundary = (int) Math.pow(maxValue, (double) index / length);\n      // we have more byte slots left than we have possible boundary values (buckets),\n      // just give consecutive boundary values to all remaining slots, starting from 0.\n      if (boundary <= index) {\n        break;\n      }\n      boundaryValues[index] = boundary;\n    }\n    if (index >= 0) {\n      for (int i = 1; i <= index; ++i) {\n        boundaryValues[i] = i;\n      }\n    }\n    boundaryValues[0] = 0;  // the first one is always 0.\n  }\n\n  @Override\n  public byte normalize(double val) {\n    int intVal = (int) (val > maxValue ? maxValue : val);\n    return intToUnsignedByte(binarySearch(intVal, boundaryValues));\n  }\n\n  /**\n   * Return the lower bound of the bucket represent by norm. This simply returns the boundary\n   * value indexed by current norm.\n   */\n  @Override\n  public double unnormLowerBound(byte norm) {\n    return boundaryValues[unsignedByteToInt(norm)];\n  }\n\n  /**\n   * Return the upper bound of the bucket represent by norm. This returns the next boundary value\n   * minus 1. If norm represents the last bucket, it returns the maxValue.\n   */\n  @Override\n  public double unnormUpperBound(byte norm) {\n    // if it's already the last possible normalized value, just return the corresponding last\n    // boundary value.\n    int intNorm = unsignedByteToInt(norm);\n    if (intNorm == length - 1) {\n      return maxValue;\n    }\n    return boundaryValues[intNorm + 1] - 1;\n  }\n\n  /**\n   * Do a binary search on array and find the index of the item that's no bigger than value.\n   */\n  private static int binarySearch(int value, int[] array) {\n    // corner cases\n    if (value <= array[0]) {\n      return 0;\n    } else if (value >= array[array.length - 1]) {\n      return array.length - 1;\n    }\n    int left = 0;\n    int right = array.length - 1;\n    int pivot = (left + right) >> 1;\n    do {\n      int midVal = array[pivot];\n      if (value == midVal) {\n        break;\n      } else if (value > midVal) {\n        left = pivot;\n      } else {\n        right = pivot;\n      }\n      pivot = (left + right) >> 1;\n    } while (pivot != left);\n    return pivot;\n  }\n\n  @Override\n  public String toString() {\n    StringBuilder sb = new StringBuilder(String.format(\n        \"Smart Integer Normalizer (numBits = %d, max = %d)\\n\",\n        this.numBits, this.maxValue));\n    for (int i = 0; i < this.length; i++) {\n      sb.append(String.format(\n          \"[%2d] boundary = %6d, range [ %6d, %6d ), norm: %4d | %4d | %4d %s\\n\",\n          i, boundaryValues[i],\n          (int) unnormLowerBound(intToUnsignedByte(i)),\n          (int) unnormUpperBound(intToUnsignedByte(i)),\n          unsignedByteToInt(normalize(boundaryValues[i] - 1)),\n          unsignedByteToInt(normalize(boundaryValues[i])),\n          unsignedByteToInt(normalize(boundaryValues[i] + 1)),\n          i == boundaryValues[i] ? \"*\" : \"\"));\n    }\n    return sb.toString();\n  }\n\n  @VisibleForTesting\n  int[] getBoundaryValues() {\n    return boundaryValues;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/code/findbugs:jsr305\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-common\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-smartcn\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-facet\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-queries\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/search/common/features\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/common/util/analysis\",\n        \"src/java/com/twitter/search/queryparser\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/java/com/twitter/search/queryparser/query/search:search-query-nodes\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/BoostUtils.java",
    "content": "package com.twitter.search.common.query;\n\nimport org.apache.lucene.search.BoostQuery;\nimport org.apache.lucene.search.Query;\n\n/**\n * A class of utilities related to query boosts.\n */\npublic final class BoostUtils {\n  private BoostUtils() {\n  }\n\n  /**\n   * Wraps the given query into a BoostQuery, if {@code boost} is not equal to 1.0f.\n   *\n   * @param query The query.\n   * @param boost The boost.\n   * @return If {@code boost} is equal to 1.0f, then {@code query} is returned; otherwise,\n   *         {@code query} is wrapped into a {@code BoostQuery} instance with the given boost.\n   */\n  public static Query maybeWrapInBoostQuery(Query query, float boost) {\n    if (boost == 1.0f) {\n      return query;\n    }\n    return new BoostQuery(query, boost);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/CollectAnnotationsVisitor.java",
    "content": "package com.twitter.search.common.query;\n\n\nimport java.util.Map;\nimport java.util.Set;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Maps;\n\nimport com.twitter.search.queryparser.query.BooleanQuery;\nimport com.twitter.search.queryparser.query.Conjunction;\nimport com.twitter.search.queryparser.query.Disjunction;\nimport com.twitter.search.queryparser.query.Operator;\nimport com.twitter.search.queryparser.query.Phrase;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.QueryVisitor;\nimport com.twitter.search.queryparser.query.SpecialTerm;\nimport com.twitter.search.queryparser.query.Term;\nimport com.twitter.search.queryparser.query.annotation.Annotation;\n\n/**\n * Collect the nodes with a specified annotation type in the given query.\n */\npublic class CollectAnnotationsVisitor extends QueryVisitor<Boolean> {\n\n  protected final Annotation.Type type;\n\n  protected final Map<Query, Boolean> nodeToTypeMap = Maps.newIdentityHashMap();\n\n  public CollectAnnotationsVisitor(Annotation.Type type) {\n    this.type = Preconditions.checkNotNull(type);\n  }\n\n  @Override\n  public Boolean visit(Disjunction disjunction) throws QueryParserException {\n    return visitBooleanQuery(disjunction);\n  }\n\n  @Override\n  public Boolean visit(Conjunction conjunction) throws QueryParserException {\n    return visitBooleanQuery(conjunction);\n  }\n\n  @Override\n  public Boolean visit(Phrase phrase) throws QueryParserException {\n    return visitQuery(phrase);\n  }\n\n  @Override\n  public Boolean visit(Term term) throws QueryParserException {\n    return visitQuery(term);\n  }\n\n  @Override\n  public Boolean visit(Operator operator) throws QueryParserException {\n    return visitQuery(operator);\n  }\n\n  @Override\n  public Boolean visit(SpecialTerm special) throws QueryParserException {\n    return visitQuery(special);\n  }\n\n  protected boolean visitQuery(Query query) throws QueryParserException {\n    if (query.hasAnnotationType(type)) {\n      collectNode(query);\n      return true;\n    }\n    return false;\n  }\n\n  protected void collectNode(Query query) {\n    nodeToTypeMap.put(query, true);\n  }\n\n  protected boolean visitBooleanQuery(BooleanQuery query) throws QueryParserException {\n    boolean found = false;\n    if (query.hasAnnotationType(type)) {\n      collectNode(query);\n      found = true;\n    }\n    for (Query child : query.getChildren()) {\n      found |= child.accept(this);\n    }\n    return found;\n  }\n\n  public Set<Query> getNodes() {\n    return nodeToTypeMap.keySet();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/CollectQueryTypeVisitor.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.util.Map;\nimport java.util.Set;\n\nimport com.google.common.collect.Maps;\n\nimport com.twitter.search.queryparser.query.BooleanQuery;\nimport com.twitter.search.queryparser.query.Conjunction;\nimport com.twitter.search.queryparser.query.Disjunction;\nimport com.twitter.search.queryparser.query.Operator;\nimport com.twitter.search.queryparser.query.Phrase;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.QueryVisitor;\nimport com.twitter.search.queryparser.query.SpecialTerm;\nimport com.twitter.search.queryparser.query.Term;\n\n/**\n * Collects the nodes with a specified query type in the given query.\n */\npublic class CollectQueryTypeVisitor extends QueryVisitor<Boolean> {\n\n  protected final Query.QueryType queryType;\n\n  protected final Map<Query, Boolean> nodeToTypeMap = Maps.newIdentityHashMap();\n\n  public CollectQueryTypeVisitor(Query.QueryType queryType) {\n    this.queryType = queryType;\n  }\n\n  @Override\n  public Boolean visit(Disjunction disjunction) throws QueryParserException {\n    return visitBooleanQuery(disjunction);\n  }\n\n  @Override\n  public Boolean visit(Conjunction conjunction) throws QueryParserException {\n    return visitBooleanQuery(conjunction);\n  }\n\n  @Override\n  public Boolean visit(Phrase phrase) throws QueryParserException {\n    return visitQuery(phrase);\n  }\n\n  @Override\n  public Boolean visit(Term term) throws QueryParserException {\n    return visitQuery(term);\n  }\n\n  @Override\n  public Boolean visit(Operator operator) throws QueryParserException {\n    return visitQuery(operator);\n  }\n\n  @Override\n  public Boolean visit(SpecialTerm special) throws QueryParserException {\n    return visitQuery(special);\n  }\n\n  public Set<Query> getCollectedNodes() {\n    return nodeToTypeMap.keySet();\n  }\n\n  protected boolean visitQuery(Query query) throws QueryParserException {\n    if (query.isTypeOf(queryType)) {\n      collectNode(query);\n      return true;\n    }\n    return false;\n  }\n\n  protected void collectNode(Query query) {\n    nodeToTypeMap.put(query, true);\n  }\n\n  protected boolean visitBooleanQuery(BooleanQuery query) throws QueryParserException {\n    boolean found = false;\n    if (query.isTypeOf(queryType)) {\n      collectNode(query);\n      found = true;\n    }\n    for (Query child : query.getChildren()) {\n      found |= child.accept(this);\n    }\n    return found;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/CollectVariantVisitor.java",
    "content": "package com.twitter.search.common.query;\n\nimport com.twitter.search.queryparser.query.annotation.Annotation;\n\n\n/**\n * A visitor that collects the nodes that have :v annotation\n */\npublic class CollectVariantVisitor extends CollectAnnotationsVisitor {\n  public CollectVariantVisitor() {\n    super(Annotation.Type.VARIANT);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/DefaultFilterWeight.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.io.IOException;\nimport java.util.Set;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.ConstantScoreScorer;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.Explanation;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\n/**\n * An abstract Weight implementation that can be used by all \"filter\" classes (Query instances that\n * should not contribute to the overall query score).\n */\npublic abstract class DefaultFilterWeight extends Weight {\n  public DefaultFilterWeight(Query query) {\n    super(query);\n  }\n\n  @Override\n  public void extractTerms(Set<Term> terms) {\n  }\n\n  @Override\n  public Explanation explain(LeafReaderContext context, int doc) throws IOException {\n    Scorer scorer = scorer(context);\n    if ((scorer != null) && (scorer.iterator().advance(doc) == doc)) {\n      return Explanation.match(0f, \"Match on id \" + doc);\n    }\n    return Explanation.match(0f, \"No match on id \" + doc);\n  }\n\n  @Override\n  public Scorer scorer(LeafReaderContext context) throws IOException {\n    DocIdSetIterator disi = getDocIdSetIterator(context);\n    if (disi == null) {\n      return null;\n    }\n\n    return new ConstantScoreScorer(this, 0.0f, ScoreMode.COMPLETE_NO_SCORES, disi);\n  }\n\n  @Override\n  public boolean isCacheable(LeafReaderContext ctx) {\n    return false;\n  }\n\n  /**\n   * Returns the DocIdSetIterator over which the scorers created by this weight need to iterate.\n   *\n   * @param context The LeafReaderContext instance used to create the scorer.\n   */\n  protected abstract DocIdSetIterator getDocIdSetIterator(LeafReaderContext context)\n      throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/DocIdFilter.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.io.IOException;\nimport java.util.Set;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.ConstantScoreScorer;\nimport org.apache.lucene.search.Explanation;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\n/**\n * Lucene filter on top of a known docid\n *\n */\npublic class DocIdFilter extends Query {\n  private final int docid;\n\n  public DocIdFilter(int docid) {\n    this.docid = docid;\n  }\n\n  @Override\n  public Weight createWeight(\n      IndexSearcher searcher, ScoreMode scoreMode, float boost) throws IOException {\n    return new Weight(this) {\n      @Override\n      public void extractTerms(Set<Term> terms) {\n      }\n\n      @Override\n      public Explanation explain(LeafReaderContext context, int doc) throws IOException {\n        Scorer scorer = scorer(context);\n        if ((scorer != null) && (scorer.iterator().advance(doc) == doc)) {\n          return Explanation.match(0f, \"Match on id \" + doc);\n        }\n        return Explanation.match(0f, \"No match on id \" + doc);\n      }\n\n      @Override\n      public Scorer scorer(LeafReaderContext context) throws IOException {\n        return new ConstantScoreScorer(this, 0.0f, scoreMode, new SingleDocDocIdSetIterator(docid));\n      }\n\n      @Override\n      public boolean isCacheable(LeafReaderContext ctx) {\n        return true;\n      }\n    };\n  }\n\n  @Override\n  public int hashCode() {\n    return docid;\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof DocIdFilter)) {\n      return false;\n    }\n\n    return docid == DocIdFilter.class.cast(obj).docid;\n  }\n\n  @Override\n  public String toString(String field) {\n    return \"DOC_ID_FILTER[docId=\" + docid + \" + ]\";\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/FieldRankHitInfo.java",
    "content": "package com.twitter.search.common.query;\n\n/**\n * When a hit (on a part of the query tree) occurs, this class is passed to HitAttributeCollector\n * for collection.\n *\n * This implementation carries the following info:\n * <ul>\n *   <li>The field that matched (the field ID is recorded)</li>\n *   <li>The query node that matched (the query node rank is recorded)</li>\n *   <li>The ID of the last doc that matched this query</li>\n * </ul>\n *\n * Each IdentifiableQuery should be associated with one FieldRankHitInfo, which is passed to a\n * HitAttributeCollector when a hit occurs.\n */\npublic class FieldRankHitInfo {\n  protected static final int UNSET_DOC_ID = -1;\n\n  private final int fieldId;\n  private final int rank;\n  private int docId = UNSET_DOC_ID;\n\n  public FieldRankHitInfo(int fieldId, int rank) {\n    this.fieldId = fieldId;\n    this.rank = rank;\n  }\n\n  public int getFieldId() {\n    return fieldId;\n  }\n\n  public int getRank() {\n    return rank;\n  }\n\n  public int getDocId() {\n    return docId;\n  }\n\n  public void setDocId(int docId) {\n    this.docId = docId;\n  }\n\n  public void resetDocId() {\n    this.docId = UNSET_DOC_ID;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/FieldWeightUtil.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.util.Collections;\nimport java.util.EnumSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Enums;\nimport com.google.common.base.Function;\nimport com.google.common.base.Functions;\nimport com.google.common.base.Predicates;\nimport com.google.common.collect.FluentIterable;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Iterables;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.schema.base.FieldWeightDefault;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.annotation.Annotation;\nimport com.twitter.search.queryparser.query.annotation.FieldAnnotationUtils;\nimport com.twitter.search.queryparser.query.annotation.FieldNameWithBoost;\n\npublic final class FieldWeightUtil {\n  private static final Logger LOG = LoggerFactory.getLogger(FieldWeightUtil.class);\n  private FieldWeightUtil() {\n  }\n\n  /**\n   * Combines default field weight configuration with field annotations and returns a\n   * field-to-weight map.\n   *\n   * @param query The query whose annotations we will look into\n   * @param defaultFieldWeightMap field-to-FieldWeightDefault map\n   * @param enabledFieldWeightMap for optimization, this is the field-to-weight map inferred from\n   * the field-to-FieldWeightDefault map\n   * @param fieldNameToTyped A function that can turn string field name to typed field\n   * @param <T> The typed field\n   */\n  public static <T> ImmutableMap<T, Float> combineDefaultWithAnnotation(\n      Query query,\n      Map<T, FieldWeightDefault> defaultFieldWeightMap,\n      Map<T, Float> enabledFieldWeightMap,\n      Function<String, T> fieldNameToTyped) throws QueryParserException {\n    return combineDefaultWithAnnotation(\n        query,\n        defaultFieldWeightMap,\n        enabledFieldWeightMap,\n        fieldNameToTyped,\n        Collections.<MappableField, T>emptyMap(),\n        Functions.forMap(Collections.<T, String>emptyMap(), \"\"));\n  }\n\n  /**\n   * Combines default field weight configuration with field annotations and returns a\n   * field-to-weight map. Also maps generic mappable fields to field weight boosts and resolves them\n   *\n   * @param query The query whose annotations we will look into\n   * @param defaultFieldWeightMap field-to-FieldWeightDefault map\n   * @param enabledFieldWeightMap for optimization, this is the field-to-weight map inferred from\n   * the field-to-FieldWeightDefault map\n   * @param fieldNameToTyped A function that can turn a string field name to typed field\n   * @param mappableFieldMap mapping of mappable fields to the corresponding typed fields\n   * @param typedToFieldName A function that can turn a typed field into a string field name\n   * @param <T> The typed field\n   *\n   * Note: As a result of discussion on SEARCH-24029, we now allow replace and remove annotations\n   * on a single term. See http://go/fieldweight for info on field weight annotations.\n   */\n  public static <T> ImmutableMap<T, Float> combineDefaultWithAnnotation(\n        Query query,\n        Map<T, FieldWeightDefault> defaultFieldWeightMap,\n        Map<T, Float> enabledFieldWeightMap,\n        Function<String, T> fieldNameToTyped,\n        Map<MappableField, T> mappableFieldMap,\n        Function<T, String> typedToFieldName) throws QueryParserException {\n    List<Annotation> fieldAnnotations = query.getAllAnnotationsOf(Annotation.Type.FIELD);\n    List<Annotation> mappableFieldAnnotations =\n      query.getAllAnnotationsOf(Annotation.Type.MAPPABLE_FIELD);\n\n    if (fieldAnnotations.isEmpty() && mappableFieldAnnotations.isEmpty()) {\n      return ImmutableMap.copyOf(enabledFieldWeightMap);\n    }\n\n    // Convert mapped fields to field annotations\n    Iterable<Annotation> fieldAnnotationsForMappedFields =\n        FluentIterable.from(mappableFieldAnnotations)\n            .transform(FieldWeightUtil.fieldAnnotationForMappableField(mappableFieldMap,\n                                                                       typedToFieldName))\n            .filter(Predicates.notNull());\n\n    Iterable<Annotation> annotations =\n        Iterables.concat(fieldAnnotationsForMappedFields, fieldAnnotations);\n\n    // Sanitize the field annotations first, remove the ones we don't know\n    // for REPLACE and REMOVE.\n    List<FieldNameWithBoost> sanitizedFields = Lists.newArrayList();\n    Set<FieldNameWithBoost.FieldModifier> seenModifierTypes =\n        EnumSet.noneOf(FieldNameWithBoost.FieldModifier.class);\n\n    for (Annotation annotation : annotations) {\n      FieldNameWithBoost fieldNameWithBoost = (FieldNameWithBoost) annotation.getValue();\n      T typedField = fieldNameToTyped.apply(fieldNameWithBoost.getFieldName());\n      FieldNameWithBoost.FieldModifier modifier = fieldNameWithBoost.getFieldModifier();\n      if (defaultFieldWeightMap.containsKey(typedField)) {\n        seenModifierTypes.add(modifier);\n        sanitizedFields.add(fieldNameWithBoost);\n      }\n    }\n\n    // Even if there is no mapping for a mapped annotation, if a query is replaced by an unknown\n    // mapping, it should not map to other fields, so we need to detect a REPLACE annotation\n    if (seenModifierTypes.isEmpty()\n        && FieldAnnotationUtils.hasReplaceAnnotation(mappableFieldAnnotations)) {\n      seenModifierTypes.add(FieldNameWithBoost.FieldModifier.REPLACE);\n    }\n\n    boolean onlyHasReplace = seenModifierTypes.size() == 1\n      && seenModifierTypes.contains(FieldNameWithBoost.FieldModifier.REPLACE);\n\n    // If we only have replace, start with an empty map, otherwise, start with all enabled fields.\n    Map<T, Float> actualMap = onlyHasReplace\n        ? Maps.<T, Float>newLinkedHashMap()\n        : Maps.newLinkedHashMap(enabledFieldWeightMap);\n\n    // Go over all field annotations and apply them.\n    for (FieldNameWithBoost fieldAnnotation : sanitizedFields) {\n      T typedField = fieldNameToTyped.apply(fieldAnnotation.getFieldName());\n      FieldNameWithBoost.FieldModifier modifier = fieldAnnotation.getFieldModifier();\n      switch (modifier) {\n        case REMOVE:\n          actualMap.remove(typedField);\n          break;\n\n        case ADD:\n        case REPLACE:\n          if (fieldAnnotation.getBoost().isPresent()) {\n            actualMap.put(typedField, fieldAnnotation.getBoost().get());\n          } else {\n            // When annotation does not specify weight, use default weight\n            actualMap.put(\n                typedField,\n                defaultFieldWeightMap.get(typedField).getWeight());\n          }\n          break;\n        default:\n          throw new QueryParserException(\"Unknown field annotation type: \" + fieldAnnotation);\n      }\n    }\n\n    return ImmutableMap.copyOf(actualMap);\n  }\n\n  public static ImmutableMap<String, Float> combineDefaultWithAnnotation(\n      Query query,\n      Map<String, FieldWeightDefault> defaultFieldWeightMap,\n      Map<String, Float> enabledFieldWeightMap) throws QueryParserException {\n\n    return combineDefaultWithAnnotation(\n        query, defaultFieldWeightMap, enabledFieldWeightMap, Functions.<String>identity());\n  }\n\n  /**\n   * Create an annotation of the FIELD type from annotations of the MAPPED_FIELD type\n   * @param mappableFieldMap mapping of mappable fields to the corresponding typed fields\n   * @param typedToFieldName A function that can turn a typed field into a string field name\n   * @param <T> The typed field\n   * @return an Annotation with the same modifier and boost for a FIELD as the incoming MAPPED_FIELD\n   * annotation\n   */\n  private static <T> Function<Annotation, Annotation> fieldAnnotationForMappableField(\n      final Map<MappableField, T> mappableFieldMap,\n      final Function<T, String> typedToFieldName) {\n    return new Function<Annotation, Annotation>() {\n      @Nullable\n      @Override\n      public Annotation apply(Annotation mappableAnnotation) {\n        FieldNameWithBoost fieldNameWithBoost = (FieldNameWithBoost) mappableAnnotation.getValue();\n        MappableField mappedField =\n            Enums.getIfPresent(\n                MappableField.class,\n                fieldNameWithBoost.getFieldName().toUpperCase()).orNull();\n        T typedFieldName = mappableFieldMap.get(mappedField);\n        Annotation fieldAnnotation = null;\n        if (typedFieldName != null) {\n          String fieldName = typedToFieldName.apply(typedFieldName);\n          FieldNameWithBoost mappedFieldBoost =\n              new FieldNameWithBoost(\n                  fieldName,\n                  fieldNameWithBoost.getBoost(),\n                  fieldNameWithBoost.getFieldModifier());\n          fieldAnnotation = Annotation.Type.FIELD.newInstance(mappedFieldBoost);\n        }\n        return fieldAnnotation;\n      }\n    };\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/FilteredQuery.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.io.IOException;\nimport java.util.Set;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.IndexReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.Explanation;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\n/**\n * A pairing of a query and a filter. The hits traversal is driven by the query's DocIdSetIterator,\n * and the filter is used only to do post-filtering. In other words, the filter is never used to\n * find the next doc ID: it's only used to filter out the doc IDs returned by the query's\n * DocIdSetIterator. This is useful when we need to have a conjunction between a query that can\n * quickly iterate through doc IDs (eg. a posting list), and an expensive filter (eg. a filter based\n * on the values stored in a CSF).\n *\n * For example, let say we want to build a query that returns all docs that have at least 100 faves.\n *   1. One option is to go with the [min_faves 100] query. This would be very expensive though,\n *      because this query would have to walk through every doc in the segment and for each one of\n *      them it would have to extract the number of faves from the forward index.\n *   2. Another option is to go with a conjunction between this query and the HAS_ENGAGEMENT filter:\n *      (+[min_faves 100] +[cached_filter has_engagements]). The HAS_ENGAGEMENT filter could\n *      traverse the doc ID space faster (if it's backed by a posting list). But this approach would\n *      still be slow, because as soon as the HAS_ENGAGEMENT filter finds a doc ID, the conjunction\n *      scorer would trigger an advance(docID) call on the min_faves part of the query, which has\n *      the same problem as the first option.\n *   3. Finally, a better option for this particular case would be to drive by the HAS_ENGAGEMENT\n *      filter (because it can quickly jump over all docs that do not have any engagement), and use\n *      the min_faves filter as a post-processing step, on a much smaller set of docs.\n */\npublic class FilteredQuery extends Query {\n  /**\n   * A doc ID predicate that determines if the given doc ID should be accepted.\n   */\n  @FunctionalInterface\n  public static interface DocIdFilter {\n    /**\n     * Determines if the given doc ID should be accepted.\n     */\n    boolean accept(int docId) throws IOException;\n  }\n\n  /**\n   * A factory for creating DocIdFilter instances based on a given LeafReaderContext instance.\n   */\n  @FunctionalInterface\n  public static interface DocIdFilterFactory {\n    /**\n     * Returns a DocIdFilter instance for the given LeafReaderContext instance.\n     */\n    DocIdFilter getDocIdFilter(LeafReaderContext context) throws IOException;\n  }\n\n  private static class FilteredQueryDocIdSetIterator extends DocIdSetIterator {\n    private final DocIdSetIterator queryScorerIterator;\n    private final DocIdFilter docIdFilter;\n\n    public FilteredQueryDocIdSetIterator(\n        DocIdSetIterator queryScorerIterator, DocIdFilter docIdFilter) {\n      this.queryScorerIterator = Preconditions.checkNotNull(queryScorerIterator);\n      this.docIdFilter = Preconditions.checkNotNull(docIdFilter);\n    }\n\n    @Override\n    public int docID() {\n      return queryScorerIterator.docID();\n    }\n\n    @Override\n    public int nextDoc() throws IOException {\n      int docId;\n      do {\n        docId = queryScorerIterator.nextDoc();\n      } while (docId != NO_MORE_DOCS && !docIdFilter.accept(docId));\n      return docId;\n    }\n\n    @Override\n    public int advance(int target) throws IOException {\n      int docId = queryScorerIterator.advance(target);\n      if (docId == NO_MORE_DOCS || docIdFilter.accept(docId)) {\n        return docId;\n      }\n      return nextDoc();\n    }\n\n    @Override\n    public long cost() {\n      return queryScorerIterator.cost();\n    }\n  }\n\n  private static class FilteredQueryScorer extends Scorer {\n    private final Scorer queryScorer;\n    private final DocIdFilter docIdFilter;\n\n    public FilteredQueryScorer(Weight weight, Scorer queryScorer, DocIdFilter docIdFilter) {\n      super(weight);\n      this.queryScorer = Preconditions.checkNotNull(queryScorer);\n      this.docIdFilter = Preconditions.checkNotNull(docIdFilter);\n    }\n\n    @Override\n    public int docID() {\n      return queryScorer.docID();\n    }\n\n    @Override\n    public float score() throws IOException {\n      return queryScorer.score();\n    }\n\n    @Override\n    public DocIdSetIterator iterator() {\n      return new FilteredQueryDocIdSetIterator(queryScorer.iterator(), docIdFilter);\n    }\n\n    @Override\n    public float getMaxScore(int upTo) throws IOException {\n      return queryScorer.getMaxScore(upTo);\n    }\n  }\n\n  private static class FilteredQueryWeight extends Weight {\n    private final Weight queryWeight;\n    private final DocIdFilterFactory docIdFilterFactory;\n\n    public FilteredQueryWeight(\n        FilteredQuery query, Weight queryWeight, DocIdFilterFactory docIdFilterFactory) {\n      super(query);\n      this.queryWeight = Preconditions.checkNotNull(queryWeight);\n      this.docIdFilterFactory = Preconditions.checkNotNull(docIdFilterFactory);\n    }\n\n    @Override\n    public void extractTerms(Set<Term> terms) {\n      queryWeight.extractTerms(terms);\n    }\n\n    @Override\n    public Explanation explain(LeafReaderContext context, int doc) throws IOException {\n      return queryWeight.explain(context, doc);\n    }\n\n    @Override\n    public Scorer scorer(LeafReaderContext context) throws IOException {\n      Scorer queryScorer = queryWeight.scorer(context);\n      if (queryScorer == null) {\n        return null;\n      }\n\n      return new FilteredQueryScorer(this, queryScorer, docIdFilterFactory.getDocIdFilter(context));\n    }\n\n    @Override\n    public boolean isCacheable(LeafReaderContext ctx) {\n      return queryWeight.isCacheable(ctx);\n    }\n  }\n\n  private final Query query;\n  private final DocIdFilterFactory docIdFilterFactory;\n\n  public FilteredQuery(Query query, DocIdFilterFactory docIdFilterFactory) {\n    this.query = Preconditions.checkNotNull(query);\n    this.docIdFilterFactory = Preconditions.checkNotNull(docIdFilterFactory);\n  }\n\n  public Query getQuery() {\n    return query;\n  }\n\n  @Override\n  public Query rewrite(IndexReader reader) throws IOException {\n    Query rewrittenQuery = query.rewrite(reader);\n    if (rewrittenQuery != query) {\n      return new FilteredQuery(rewrittenQuery, docIdFilterFactory);\n    }\n    return this;\n  }\n\n  @Override\n  public int hashCode() {\n    return query.hashCode() * 13 + docIdFilterFactory.hashCode();\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof FilteredQuery)) {\n      return false;\n    }\n\n    FilteredQuery filteredQuery = FilteredQuery.class.cast(obj);\n    return query.equals(filteredQuery.query)\n        && docIdFilterFactory.equals(filteredQuery.docIdFilterFactory);\n  }\n\n  @Override\n  public String toString(String field) {\n    StringBuilder sb = new StringBuilder();\n    sb.append(\"FilteredQuery(\")\n        .append(query)\n        .append(\" -> \")\n        .append(docIdFilterFactory)\n        .append(\")\");\n    return sb.toString();\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost)\n      throws IOException {\n    Weight queryWeight = Preconditions.checkNotNull(query.createWeight(searcher, scoreMode, boost));\n    return new FilteredQueryWeight(this, queryWeight, docIdFilterFactory);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/FilteredScorer.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.Weight;\n\npublic class FilteredScorer extends Scorer {\n  protected final Scorer inner;\n\n  public FilteredScorer(Weight weight, Scorer inner) {\n    super(weight);\n    this.inner = inner;\n  }\n\n  @Override\n  public float score() throws IOException {\n    return inner.score();\n  }\n\n  @Override\n  public int docID() {\n    return inner.docID();\n  }\n\n  @Override\n  public DocIdSetIterator iterator() {\n    return inner.iterator();\n  }\n\n  @Override\n  public float getMaxScore(int upTo) throws IOException {\n    return inner.getMaxScore(upTo);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/HitAttributeCollector.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.util.List;\nimport java.util.Map;\nimport java.util.function.BiFunction;\nimport java.util.function.Function;\n\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.search.Query;\n\n/**\n * Not threadsafe, but should be reused across different queries unless the size of the existing\n * one is too small for a new huge serialized query.\n */\npublic class HitAttributeCollector {\n  private final List<FieldRankHitInfo> hitInfos = Lists.newArrayList();\n  private final BiFunction<Integer, Integer, FieldRankHitInfo> hitInfoSupplier;\n\n  private int docBase = 0;\n\n  public HitAttributeCollector() {\n    this.hitInfoSupplier = FieldRankHitInfo::new;\n  }\n\n  /**\n   * Constructs a new {@code HitAttributionCollector} with the specified {@code FieldRankHitInfo}\n   * supplier.\n   *\n   * @param hitInfoSupplier function to supply a {@code FieldRankHitInfo} instance\n   */\n  public HitAttributeCollector(BiFunction<Integer, Integer, FieldRankHitInfo> hitInfoSupplier) {\n    this.hitInfoSupplier = hitInfoSupplier;\n  }\n\n  /**\n   * Creates a new IdentifiableQuery for the given query, fieldId and rank, and \"registers\"\n   * the fieldId and the rank with this collector.\n   *\n   * @param query the query to be wrapped.\n   * @param fieldId the ID of the field to be searched.\n   * @param rank The rank of this query.\n   * @return A new IdentifiableQuery instance for the given query, fieldId and rank.\n   */\n  public IdentifiableQuery newIdentifiableQuery(Query query, int fieldId, int rank) {\n    FieldRankHitInfo fieldRankHitInfo = hitInfoSupplier.apply(fieldId, rank);\n    hitInfos.add(fieldRankHitInfo);\n    return new IdentifiableQuery(query, fieldRankHitInfo, this);\n  }\n\n  public void clearHitAttributions(LeafReaderContext ctx, FieldRankHitInfo hitInfo) {\n    docBase = ctx.docBase;\n    hitInfo.resetDocId();\n  }\n\n  public void collectScorerAttribution(int docId, FieldRankHitInfo hitInfo) {\n    hitInfo.setDocId(docId + docBase);\n  }\n\n  /**\n   * This method should be called when a global hit occurs.\n   * This method returns hit attribution summary for the whole query tree.\n   * This supports getting hit attribution for only the curDoc.\n   *\n   * @param docId docId passed in for checking against curDoc.\n   * @return Returns a map from node rank to a set of matching field IDs. This map does not contain\n   *         entries for ranks that did not hit at all.\n   */\n  public Map<Integer, List<Integer>> getHitAttribution(int docId) {\n    return getHitAttribution(docId, (fieldId) -> fieldId);\n  }\n\n  /**\n   * This method should be called when a global hit occurs.\n   * This method returns hit attribution summary for the whole query tree.\n   * This supports getting hit attribution for only the curDoc.\n   *\n   * @param docId docId passed in for checking against curDoc.\n   * @param fieldIdFunc The mapping of field IDs to objects of type T.\n   * @return Returns a map from node rank to a set of matching objects (usually field IDs or names).\n   *         This map does not contain entries for ranks that did not hit at all.\n   */\n  public <T> Map<Integer, List<T>> getHitAttribution(int docId, Function<Integer, T> fieldIdFunc) {\n    int key = docId + docBase;\n    Map<Integer, List<T>> hitMap = Maps.newHashMap();\n\n    // Manually iterate through all hitInfos elements. It's slightly faster than using an Iterator.\n    for (FieldRankHitInfo hitInfo : hitInfos) {\n      if (hitInfo.getDocId() == key) {\n        int rank = hitInfo.getRank();\n        List<T> rankHits = hitMap.computeIfAbsent(rank, k -> Lists.newArrayList());\n        T fieldDescription = fieldIdFunc.apply(hitInfo.getFieldId());\n        rankHits.add(fieldDescription);\n      }\n    }\n\n    return hitMap;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/HitAttributeHelper.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.util.List;\nimport java.util.Map;\nimport java.util.function.Function;\n\nimport com.google.common.collect.Maps;\n\nimport com.twitter.search.queryparser.query.Query;\n\nimport static com.twitter.search.common.query.FieldRankHitInfo.UNSET_DOC_ID;\n\n/**\n * Generic helper class containing the data needed to set up and collect field hit attributions.\n */\npublic class HitAttributeHelper implements HitAttributeProvider {\n  private final HitAttributeCollector collector;\n  private final Function<Integer, String> fieldIdsToFieldNames;\n\n  // This is a mapping of type T query nodes to rank id\n  private final Map<Query, Integer> nodeToRankMap;\n\n  // This is meant to expand individual Query nodes into multiple ranks,\n  // for example, expanding a multi_term_disjunction to include a rank for each disjunction value.\n  private final Map<Query, List<Integer>> expandedNodeToRankMap;\n\n  // A single-entry cache for hit attribution, so we can reuse the immediate result. Will be used\n  // only when lastDocId matches\n  private ThreadLocal<Map<Integer, List<String>>> lastHitAttrHolder = new ThreadLocal<>();\n  private ThreadLocal<Integer> lastDocIdHolder = ThreadLocal.withInitial(() -> UNSET_DOC_ID);\n\n  protected HitAttributeHelper(\n      HitAttributeCollector collector,\n      Function<Integer, String> fieldIdsToFieldNames,\n      Map<Query, Integer> nodeToRankMap,\n      Map<Query, List<Integer>> expandedNodeToRankMap) {\n    this.collector = collector;\n    this.fieldIdsToFieldNames = fieldIdsToFieldNames;\n    this.nodeToRankMap = nodeToRankMap;\n    this.expandedNodeToRankMap = expandedNodeToRankMap;\n  }\n\n  /**\n   * Constructs a new {@code HitAttributeHelper} with the specified {@code HitAttributeCollector}\n   * instance and fields.\n   *\n   * @param collector a collector instance\n   * @param fieldIdsToFieldNames a list of field names indexed by id\n   */\n  public HitAttributeHelper(HitAttributeCollector collector, String[] fieldIdsToFieldNames) {\n    this(collector,\n        (fieldId) -> fieldIdsToFieldNames[fieldId],\n        Maps.newHashMap(),\n        Maps.newHashMap());\n  }\n\n  public HitAttributeCollector getFieldRankHitAttributeCollector() {\n    return collector;\n  }\n\n  /**\n   * Returns hit attribution information indexed by node rank\n   *\n   * @param docId the document id\n   * @return a mapping from the query's node rank to a list of field names that were hit.\n   */\n  public Map<Integer, List<String>> getHitAttribution(int docId) {\n    // check cache first so we don't have to recompute the same thing.\n    if (lastDocIdHolder.get() == docId) {\n      return lastHitAttrHolder.get();\n    }\n\n    lastDocIdHolder.set(docId);\n    Map<Integer, List<String>> hitAttribution =\n        collector.getHitAttribution(docId, fieldIdsToFieldNames);\n    lastHitAttrHolder.set(hitAttribution);\n    return hitAttribution;\n  }\n\n  /**\n   * Adds a new node and its respective rank to the helper's node-to-rank map\n   * Will throw an exception if attempting to add/update an existing node\n   *\n   * @param node the query node\n   * @param rank the rank associated with the node\n   */\n  public void addNodeRank(Query node, int rank) {\n    // if there are two of the same terms, just map them to the first rank, they should get the same\n    // hits back\n    if (!nodeToRankMap.containsKey(node)) {\n      nodeToRankMap.put(node, rank);\n    }\n  }\n\n  public Map<Query, Integer> getNodeToRankMap() {\n    return nodeToRankMap;\n  }\n\n  public Map<Query, List<Integer>> getExpandedNodeToRankMap() {\n    return expandedNodeToRankMap;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/HitAttributeProvider.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.util.List;\nimport java.util.Map;\n\n/**\n * The interface for objects that can provide hit attributes for a document.\n */\npublic interface HitAttributeProvider {\n  /** Returns the hit attributes for the given document. */\n  Map<Integer, List<String>> getHitAttribution(int docId);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/IDDisjunctionQuery.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.Objects;\nimport java.util.Set;\nimport java.util.stream.Collectors;\n\nimport org.apache.lucene.index.FilteredTermsEnum;\nimport org.apache.lucene.index.IndexReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.index.TermState;\nimport org.apache.lucene.index.TermStates;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.search.BooleanClause.Occur;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.BulkScorer;\nimport org.apache.lucene.search.ConstantScoreQuery;\nimport org.apache.lucene.search.ConstantScoreScorer;\nimport org.apache.lucene.search.ConstantScoreWeight;\nimport org.apache.lucene.search.DocIdSet;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.MultiTermQuery;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.TermQuery;\nimport org.apache.lucene.search.Weight;\nimport org.apache.lucene.util.AttributeSource;\nimport org.apache.lucene.util.BytesRef;\nimport org.apache.lucene.util.DocIdSetBuilder;\n\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.IndexedNumericFieldSettings;\nimport com.twitter.search.common.util.analysis.LongTermAttributeImpl;\nimport com.twitter.search.common.util.analysis.SortableLongTermAttributeImpl;\nimport com.twitter.search.queryparser.query.QueryParserException;\n\n/**\n * An extension of Lucene's MultiTermQuery which creates a disjunction of\n * long ID terms. Lucene tries to rewrite the Query depending on the number\n * of clauses to perform as efficiently as possible.\n */\npublic class IDDisjunctionQuery extends MultiTermQuery {\n  private final List<Long> ids;\n  private final boolean useOrderPreservingEncoding;\n\n  /** Creates a new IDDisjunctionQuery instance. */\n  public IDDisjunctionQuery(List<Long> ids, String field, ImmutableSchemaInterface schemaSnapshot)\n      throws QueryParserException {\n    super(field);\n    this.ids = ids;\n\n    setRewriteMethod(new Rewrite());\n\n    if (!schemaSnapshot.hasField(field)) {\n      throw new QueryParserException(\n          \"Tried to search a field which does not exist in schema: \" + field);\n    }\n\n    IndexedNumericFieldSettings numericFieldSettings =\n        schemaSnapshot.getFieldInfo(field).getFieldType().getNumericFieldSettings();\n\n    if (numericFieldSettings == null) {\n      throw new QueryParserException(\"Requested id field is not numerical: \" + field);\n    }\n\n    this.useOrderPreservingEncoding = numericFieldSettings.isUseSortableEncoding();\n  }\n\n  /**\n   * Work around for an issue where LongTerms are not valid utf8, so calling\n   * toString on any TermQuery containing a LongTerm may cause exceptions.\n   */\n  private class Rewrite extends RewriteMethod {\n    @Override\n    public Query rewrite(IndexReader reader, MultiTermQuery query) throws IOException {\n      Query result = new MultiTermQueryConstantScoreWrapper(\n          (IDDisjunctionQuery) query, useOrderPreservingEncoding);\n      return result;\n    }\n  }\n\n  @Override\n  protected TermsEnum getTermsEnum(final Terms terms, AttributeSource atts) throws IOException {\n    final Iterator<Long> it = this.ids.iterator();\n    final TermsEnum termsEnum = terms.iterator();\n\n    return new FilteredTermsEnum(termsEnum) {\n      private final BytesRef term = useOrderPreservingEncoding\n          ? SortableLongTermAttributeImpl.newBytesRef()\n          : LongTermAttributeImpl.newBytesRef();\n\n      @Override protected AcceptStatus accept(BytesRef term) throws IOException {\n        return AcceptStatus.YES;\n      }\n\n      @Override public BytesRef next() throws IOException {\n        while (it.hasNext()) {\n          Long longTerm = it.next();\n          if (useOrderPreservingEncoding) {\n            SortableLongTermAttributeImpl.copyLongToBytesRef(term, longTerm);\n          } else {\n            LongTermAttributeImpl.copyLongToBytesRef(term, longTerm);\n          }\n          if (termsEnum.seekExact(term)) {\n            return term;\n          }\n        }\n\n        return null;\n      }\n    };\n  }\n\n  @Override\n  public String toString(String field) {\n    StringBuilder builder = new StringBuilder();\n    builder.append(\"IDDisjunction[\").append(this.field).append(\":\");\n    for (Long id : this.ids) {\n      builder.append(id);\n      builder.append(\",\");\n    }\n    builder.setLength(builder.length() - 1);\n    builder.append(\"]\");\n    return builder.toString();\n  }\n\n  private static class TermQueryWithToString extends TermQuery {\n    private final boolean useOrderPreservingEncoding;\n\n    public TermQueryWithToString(Term t, TermStates states, boolean useOrderPreservingEncoding) {\n      super(t, states);\n      this.useOrderPreservingEncoding = useOrderPreservingEncoding;\n    }\n\n    @Override\n    public String toString(String field) {\n      StringBuilder buffer = new StringBuilder();\n      if (!getTerm().field().equals(field)) {\n        buffer.append(getTerm().field());\n        buffer.append(\":\");\n      }\n      long longTerm;\n      BytesRef termBytes = getTerm().bytes();\n      if (useOrderPreservingEncoding) {\n        longTerm = SortableLongTermAttributeImpl.copyBytesRefToLong(termBytes);\n      } else {\n        longTerm = LongTermAttributeImpl.copyBytesRefToLong(termBytes);\n      }\n      buffer.append(longTerm);\n      return buffer.toString();\n    }\n  }\n\n  /**\n   * This class provides the functionality behind {@link MultiTermQuery#CONSTANT_SCORE_REWRITE}.\n   * It tries to rewrite per-segment as a boolean query that returns a constant score and otherwise\n   * fills a DocIdSet with matches and builds a Scorer on top of this DocIdSet.\n   */\n  static final class MultiTermQueryConstantScoreWrapper extends Query {\n    // disable the rewrite option which will scan all posting lists sequentially and perform\n    // the intersection using a temporary DocIdSet. In earlybird this mode is slower than a \"normal\"\n    // disjunctive BooleanQuery, due to early termination and the fact that everything is in memory.\n    private static final int BOOLEAN_REWRITE_TERM_COUNT_THRESHOLD = 3000;\n\n    private static class TermAndState {\n      private final BytesRef term;\n      private final TermState state;\n      private final int docFreq;\n      private final long totalTermFreq;\n\n      TermAndState(BytesRef term, TermState state, int docFreq, long totalTermFreq) {\n        this.term = term;\n        this.state = state;\n        this.docFreq = docFreq;\n        this.totalTermFreq = totalTermFreq;\n      }\n    }\n\n    private static class WeightOrDocIdSet {\n      private final Weight weight;\n      private final DocIdSet docIdSet;\n\n      WeightOrDocIdSet(Weight weight) {\n        this.weight = Objects.requireNonNull(weight);\n        this.docIdSet = null;\n      }\n\n      WeightOrDocIdSet(DocIdSet docIdSet) {\n        this.docIdSet = docIdSet;\n        this.weight = null;\n      }\n    }\n\n    protected final IDDisjunctionQuery query;\n    private final boolean useOrderPreservingEncoding;\n\n    /**\n     * Wrap a {@link MultiTermQuery} as a Filter.\n     */\n    protected MultiTermQueryConstantScoreWrapper(\n        IDDisjunctionQuery query,\n        boolean useOrderPreservingEncoding) {\n      this.query = query;\n      this.useOrderPreservingEncoding = useOrderPreservingEncoding;\n    }\n\n    @Override\n    public String toString(String field) {\n      // query.toString should be ok for the filter, too, if the query boost is 1.0f\n      return query.toString(field);\n    }\n\n    @Override\n    public boolean equals(Object obj) {\n      if (!(obj instanceof MultiTermQueryConstantScoreWrapper)) {\n        return false;\n      }\n\n      return query.equals(MultiTermQueryConstantScoreWrapper.class.cast(obj).query);\n    }\n\n    @Override\n    public int hashCode() {\n      return query == null ? 0 : query.hashCode();\n    }\n\n    /** Returns the field name for this query */\n    public String getField() {\n      return query.getField();\n    }\n\n    private List<Long> getIDs() {\n      return query.ids;\n    }\n\n    @Override\n    public Weight createWeight(\n        final IndexSearcher searcher,\n        final ScoreMode scoreMode,\n        final float boost) throws IOException {\n      return new ConstantScoreWeight(this, boost) {\n        /** Try to collect terms from the given terms enum and return true iff all\n         *  terms could be collected. If {@code false} is returned, the enum is\n         *  left positioned on the next term. */\n        private boolean collectTerms(LeafReaderContext context,\n                                     TermsEnum termsEnum,\n                                     List<TermAndState> terms) throws IOException {\n          final int threshold = Math.min(BOOLEAN_REWRITE_TERM_COUNT_THRESHOLD,\n                                         BooleanQuery.getMaxClauseCount());\n          for (int i = 0; i < threshold; ++i) {\n            final BytesRef term = termsEnum.next();\n            if (term == null) {\n              return true;\n            }\n            TermState state = termsEnum.termState();\n            terms.add(new TermAndState(BytesRef.deepCopyOf(term),\n                                       state,\n                                       termsEnum.docFreq(),\n                                       termsEnum.totalTermFreq()));\n          }\n          return termsEnum.next() == null;\n        }\n\n        /**\n         * On the given leaf context, try to either rewrite to a disjunction if\n         * there are few terms, or build a DocIdSet containing matching docs.\n         */\n        private WeightOrDocIdSet rewrite(LeafReaderContext context)\n            throws IOException {\n          final Terms terms = context.reader().terms(query.getField());\n          if (terms == null) {\n            // field does not exist\n            return new WeightOrDocIdSet((DocIdSet) null);\n          }\n\n          final TermsEnum termsEnum = query.getTermsEnum(terms);\n          assert termsEnum != null;\n\n          PostingsEnum docs = null;\n\n          final List<TermAndState> collectedTerms = new ArrayList<>();\n          if (collectTerms(context, termsEnum, collectedTerms)) {\n            // build a boolean query\n            BooleanQuery.Builder bqBuilder = new BooleanQuery.Builder();\n            for (TermAndState t : collectedTerms) {\n              final TermStates termStates = new TermStates(searcher.getTopReaderContext());\n              termStates.register(t.state, context.ord, t.docFreq, t.totalTermFreq);\n              final Term term = new Term(query.getField(), t.term);\n              bqBuilder.add(\n                  new TermQueryWithToString(term, termStates, useOrderPreservingEncoding),\n                  Occur.SHOULD);\n            }\n            Query q = BoostUtils.maybeWrapInBoostQuery(\n                new ConstantScoreQuery(bqBuilder.build()), score());\n            return new WeightOrDocIdSet(\n                searcher.rewrite(q).createWeight(searcher, scoreMode, boost));\n          }\n\n          // Too many terms: go back to the terms we already collected and start building\n          // the DocIdSet\n          DocIdSetBuilder builder = new DocIdSetBuilder(context.reader().maxDoc());\n          if (!collectedTerms.isEmpty()) {\n            TermsEnum termsEnum2 = terms.iterator();\n            for (TermAndState t : collectedTerms) {\n              termsEnum2.seekExact(t.term, t.state);\n              docs = termsEnum2.postings(docs, PostingsEnum.NONE);\n              builder.add(docs);\n            }\n          }\n\n          // Then keep filling the DocIdSet with remaining terms\n          do {\n            docs = termsEnum.postings(docs, PostingsEnum.NONE);\n            builder.add(docs);\n          } while (termsEnum.next() != null);\n\n          return new WeightOrDocIdSet(builder.build());\n        }\n\n        private Scorer scorer(DocIdSet set) throws IOException {\n          if (set == null) {\n            return null;\n          }\n          final DocIdSetIterator disi = set.iterator();\n          if (disi == null) {\n            return null;\n          }\n          return new ConstantScoreScorer(this, score(), ScoreMode.COMPLETE_NO_SCORES, disi);\n        }\n\n        @Override\n        public BulkScorer bulkScorer(LeafReaderContext context) throws IOException {\n          final WeightOrDocIdSet weightOrDocIdSet = rewrite(context);\n          if (weightOrDocIdSet.weight != null) {\n            return weightOrDocIdSet.weight.bulkScorer(context);\n          } else {\n            final Scorer scorer = scorer(weightOrDocIdSet.docIdSet);\n            if (scorer == null) {\n              return null;\n            }\n            return new DefaultBulkScorer(scorer);\n          }\n        }\n\n        @Override\n        public Scorer scorer(LeafReaderContext context) throws IOException {\n          final WeightOrDocIdSet weightOrDocIdSet = rewrite(context);\n          if (weightOrDocIdSet.weight != null) {\n            return weightOrDocIdSet.weight.scorer(context);\n          } else {\n            return scorer(weightOrDocIdSet.docIdSet);\n          }\n        }\n\n        @Override\n        public void extractTerms(Set<Term> terms) {\n          terms.addAll(getIDs()\n              .stream()\n              .map(id -> new Term(getField(), LongTermAttributeImpl.copyIntoNewBytesRef(id)))\n              .collect(Collectors.toSet()));\n        }\n\n        @Override\n        public boolean isCacheable(LeafReaderContext ctx) {\n          return false;\n        }\n      };\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/IdentifiableQuery.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.io.IOException;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.IndexReader;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\n/**\n * Query implementation adds attribute collection support for an underlying query.\n */\npublic class IdentifiableQuery extends Query {\n  protected final Query inner;\n  private final FieldRankHitInfo queryId;\n  private final HitAttributeCollector attrCollector;\n\n  public IdentifiableQuery(Query inner, FieldRankHitInfo queryId,\n                           HitAttributeCollector attrCollector) {\n    this.inner = Preconditions.checkNotNull(inner);\n    this.queryId = queryId;\n    this.attrCollector = Preconditions.checkNotNull(attrCollector);\n  }\n\n  @Override\n  public Weight createWeight(\n      IndexSearcher searcher, ScoreMode scoreMode, float boost) throws IOException {\n    Weight innerWeight = inner.createWeight(searcher, scoreMode, boost);\n    return new IdentifiableQueryWeight(this, innerWeight, queryId, attrCollector);\n  }\n\n  @Override\n  public Query rewrite(IndexReader reader) throws IOException {\n    Query rewritten = inner.rewrite(reader);\n    if (rewritten != inner) {\n      return new IdentifiableQuery(rewritten, queryId, attrCollector);\n    }\n    return this;\n  }\n\n  @Override\n  public int hashCode() {\n    return inner.hashCode() * 13 + (queryId == null ? 0 : queryId.hashCode());\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof IdentifiableQuery)) {\n      return false;\n    }\n\n    IdentifiableQuery identifiableQuery = IdentifiableQuery.class.cast(obj);\n    return inner.equals(identifiableQuery.inner)\n        && (queryId == null\n            ? identifiableQuery.queryId == null\n            : queryId.equals(identifiableQuery.queryId));\n  }\n\n  @Override\n  public String toString(String field) {\n    return inner.toString(field);\n  }\n\n  @VisibleForTesting\n  public Query getQueryForTest() {\n    return inner;\n  }\n\n  @VisibleForTesting\n  public FieldRankHitInfo getQueryIdForTest() {\n    return queryId;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/IdentifiableQueryScorer.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.Weight;\n\n/**\n * Scorer implementation that adds attribute collection support for an underlying query.\n * Meant to be used in conjunction with {@link IdentifiableQuery}.\n */\npublic class IdentifiableQueryScorer extends FilteredScorer {\n  private final FieldRankHitInfo queryId;\n  private final HitAttributeCollector attrCollector;\n\n  public IdentifiableQueryScorer(Weight weight, Scorer inner, FieldRankHitInfo queryId,\n                                 HitAttributeCollector attrCollector) {\n    super(weight, inner);\n    this.queryId = queryId;\n    this.attrCollector = Preconditions.checkNotNull(attrCollector);\n  }\n\n  @Override\n  public DocIdSetIterator iterator() {\n    final DocIdSetIterator superDISI = super.iterator();\n\n    return new DocIdSetIterator() {\n      @Override\n      public int docID() {\n        return superDISI.docID();\n      }\n\n      @Override\n      public int nextDoc() throws IOException {\n        int docid = superDISI.nextDoc();\n        if (docid != NO_MORE_DOCS) {\n          attrCollector.collectScorerAttribution(docid, queryId);\n        }\n        return docid;\n      }\n\n      @Override\n      public int advance(int target) throws IOException {\n        int docid = superDISI.advance(target);\n        if (docid != NO_MORE_DOCS) {\n          attrCollector.collectScorerAttribution(docid, queryId);\n        }\n        return docid;\n      }\n\n      @Override\n      public long cost() {\n        return superDISI.cost();\n      }\n    };\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/IdentifiableQueryWeight.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.io.IOException;\nimport java.util.Set;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.Explanation;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.Weight;\n\n/**\n * Weight implementation that adds attribute collection support for an underlying query.\n * Meant to be used in conjunction with {@link IdentifiableQuery}.\n */\npublic class IdentifiableQueryWeight extends Weight {\n  private final Weight inner;\n  private final FieldRankHitInfo queryId;\n  private final HitAttributeCollector attrCollector;\n\n  /** Creates a new IdentifiableQueryWeight instance. */\n  public IdentifiableQueryWeight(IdentifiableQuery query, Weight inner, FieldRankHitInfo queryId,\n                                 HitAttributeCollector attrCollector) {\n    super(query);\n    this.inner = inner;\n    this.queryId = queryId;\n    this.attrCollector = Preconditions.checkNotNull(attrCollector);\n  }\n\n  @Override\n  public Explanation explain(LeafReaderContext context, int doc)\n      throws IOException {\n    return inner.explain(context, doc);\n  }\n\n  @Override\n  public Scorer scorer(LeafReaderContext context) throws IOException {\n    attrCollector.clearHitAttributions(context, queryId);\n    Scorer innerScorer = inner.scorer(context);\n    if (innerScorer != null) {\n      return new IdentifiableQueryScorer(this, innerScorer, queryId, attrCollector);\n    } else {\n      return null;\n    }\n  }\n\n  @Override\n  public void extractTerms(Set<Term> terms) {\n    inner.extractTerms(terms);\n  }\n\n  @Override\n  public boolean isCacheable(LeafReaderContext ctx) {\n    return inner.isCacheable(ctx);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/MappableField.java",
    "content": "package com.twitter.search.common.query;\n\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Maps;\n\n/**\n * The indices may map the fields declared here to fields internally without exposing their schemas\n * to other services. This can be used, for example, to set boosts for URL-like fields in Earlybird\n * without direct knowledge of the internal Earlybird field name\n */\npublic enum MappableField {\n  REFERRAL,\n  URL;\n\n  static {\n    ImmutableMap.Builder<MappableField, String> builder = ImmutableMap.builder();\n    for (MappableField mappableField : MappableField.values()) {\n      builder.put(mappableField, mappableField.toString().toLowerCase());\n    }\n    MAPPABLE_FIELD_TO_NAME_MAP = Maps.immutableEnumMap(builder.build());\n  }\n\n  private static final ImmutableMap<MappableField, String> MAPPABLE_FIELD_TO_NAME_MAP;\n\n  /** Returns the name of the given MappableField. */\n  public static String mappableFieldName(MappableField mappableField) {\n    return MAPPABLE_FIELD_TO_NAME_MAP.get(mappableField);\n  }\n\n  /** Returns the name of this MappableField. */\n  public String getName() {\n    return MAPPABLE_FIELD_TO_NAME_MAP.get(this);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/MultiTermDisjunctionQuery.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.io.IOException;\nimport java.util.Iterator;\nimport java.util.Set;\n\nimport org.apache.lucene.index.FilteredTermsEnum;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.search.MultiTermQuery;\nimport org.apache.lucene.util.AttributeSource;\nimport org.apache.lucene.util.BytesRef;\n\n\npublic class MultiTermDisjunctionQuery extends MultiTermQuery {\n\n  private final Set<BytesRef> values;\n\n  /** Creates a new MultiTermDisjunctionQuery instance. */\n  public MultiTermDisjunctionQuery(String field, Set<BytesRef> values) {\n    super(field);\n    this.values = values;\n  }\n\n  @Override\n  protected TermsEnum getTermsEnum(Terms terms, AttributeSource atts)\n      throws IOException {\n    final TermsEnum termsEnum = terms.iterator();\n    final Iterator<BytesRef> it = values.iterator();\n\n    return new FilteredTermsEnum(termsEnum) {\n      @Override protected AcceptStatus accept(BytesRef term) throws IOException {\n        return AcceptStatus.YES;\n      }\n\n      @Override public BytesRef next() throws IOException {\n        while (it.hasNext()) {\n          BytesRef termRef = it.next();\n          if (termsEnum.seekExact(termRef)) {\n            return termRef;\n          }\n        }\n\n        return null;\n      }\n    };\n  }\n\n  @Override\n  public String toString(String field) {\n    StringBuilder builder = new StringBuilder();\n    builder.append(\"MultiTermDisjunctionQuery[\");\n    for (BytesRef termVal : this.values) {\n      builder.append(termVal);\n      builder.append(\",\");\n    }\n    builder.setLength(builder.length() - 1);\n    builder.append(\"]\");\n    return builder.toString();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/QueryCommonFieldHitsVisitor.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\n\nimport com.google.common.collect.Sets;\n\nimport com.twitter.search.queryparser.query.Conjunction;\nimport com.twitter.search.queryparser.query.Disjunction;\nimport com.twitter.search.queryparser.query.Phrase;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.SpecialTerm;\nimport com.twitter.search.queryparser.query.Term;\nimport com.twitter.search.queryparser.query.search.Link;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\nimport com.twitter.search.queryparser.query.search.SearchQueryVisitor;\n\n/**\n * Visitor to track the fields hits of each node\n * Returns the common fields among conjunctions and the union of the fields amongst disjunctions\n */\npublic final class QueryCommonFieldHitsVisitor extends SearchQueryVisitor<Set<String>> {\n\n  private static final Logger LOG = Logger.getLogger(QueryCommonFieldHitsVisitor.class.getName());\n\n  private Map<Query, Integer> nodeToRankMap;\n  private Map<Integer, List<String>> hitFieldsByRank;\n\n  /**\n   * Find query term hit intersections based on hitmap given by HitAttributeHelper\n   *\n   * @param hitAttributeHelper the HitAttributeHelper\n   * @param docID documentID\n   * @param query the query searched\n   * @return a set of hit fields in String representation\n   */\n  public static Set<String> findIntersection(\n      HitAttributeHelper hitAttributeHelper,\n      int docID,\n      Query query) {\n    return findIntersection(hitAttributeHelper.getNodeToRankMap(),\n                            hitAttributeHelper.getHitAttribution(docID),\n                            query);\n  }\n\n  /**\n   * Find query term hit intersections based on hitmap given by HitAttributeHelper\n   *\n   * @param nodeToRankMap the map of query node to its integer rank value\n   * @param hitFieldsByRank map of rank to list of hit fields in String representation\n   * @param query the query searched\n   * @return a set of hit fields in String representation\n   */\n  public static Set<String> findIntersection(\n      Map<Query, Integer> nodeToRankMap,\n      Map<Integer, List<String>> hitFieldsByRank,\n      Query query) {\n    QueryCommonFieldHitsVisitor visitor =\n        new QueryCommonFieldHitsVisitor(nodeToRankMap, hitFieldsByRank);\n    try {\n      Set<String> returnSet = query.accept(visitor);\n      return returnSet;\n    } catch (QueryParserException e) {\n      LOG.log(Level.SEVERE, \"Could not find intersection for query [\" + query + \"]: \", e);\n      return Collections.emptySet();\n    }\n  }\n\n  private QueryCommonFieldHitsVisitor(Map<Query, Integer> nodeToRankMap,\n                                      Map<Integer, List<String>> hitFieldsByRank) {\n    this.nodeToRankMap = nodeToRankMap;\n    this.hitFieldsByRank = hitFieldsByRank;\n  }\n\n  @Override\n  public Set<String> visit(Disjunction disjunction) throws QueryParserException {\n    Set<String> fieldHitIntersections = Sets.newHashSet();\n    for (Query child : disjunction.getChildren()) {\n      fieldHitIntersections.addAll(child.accept(this));\n    }\n    return fieldHitIntersections;\n  }\n\n  @Override\n  public Set<String> visit(Conjunction conjunction) throws QueryParserException {\n    List<Query> children = conjunction.getChildren();\n    if (!children.isEmpty()) {\n      boolean initializedIntersections = false;\n      Set<String> fieldHitIntersections = Sets.newHashSet();\n      for (Query child : children) {\n        Set<String> hits = child.accept(this);\n        if (hits.isEmpty()) {\n          // if it is empty, it means this query node is not of term type\n          // and we do not include these in the field intersection\n          // eg. cache filters, proximity groups\n          continue;\n        }\n        if (!initializedIntersections) {\n          fieldHitIntersections.addAll(hits);\n          initializedIntersections = true;\n        } else {\n          fieldHitIntersections.retainAll(hits);\n        }\n      }\n      return fieldHitIntersections;\n    }\n    return Collections.emptySet();\n  }\n\n  @Override\n  public Set<String> visit(Term term) throws QueryParserException {\n    Set<String> fieldHitIntersections = Sets.newHashSet();\n    Integer rank = nodeToRankMap.get(term);\n    if (rank != null) {\n      List<String> fields = hitFieldsByRank.get(rank);\n      // for disjunction cases where a term may not have any hits\n      if (fields != null) {\n        fieldHitIntersections.addAll(fields);\n      }\n    }\n    return fieldHitIntersections;\n  }\n\n  @Override\n  public Set<String> visit(SpecialTerm specialTerm) throws QueryParserException {\n    // This is way of splitting @mentions ensures consistency with way the lucene query is built in\n    // expertsearch\n    if (specialTerm.getType() == SpecialTerm.Type.MENTION && specialTerm.getValue().contains(\"_\")) {\n      Phrase phrase = new Phrase(specialTerm.getValue().split(\"_\"));\n      return phrase.accept(this);\n    }\n    return specialTerm.toTermOrPhrase().accept(this);\n  }\n\n  @Override\n  public Set<String> visit(SearchOperator operator) throws QueryParserException {\n    return Collections.emptySet();\n  }\n\n  @Override\n  public Set<String> visit(Link link) throws QueryParserException {\n    return link.toPhrase().accept(this);\n  }\n\n  @Override\n  public Set<String> visit(Phrase phrase) throws QueryParserException {\n    // All terms in the phrase should return the same hits fields, just check the first one\n    List<String> terms = phrase.getTerms();\n    if (!terms.isEmpty()) {\n      Term term = new Term(phrase.getTerms().get(0));\n      return term.accept(this);\n    }\n    return Collections.emptySet();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/QueryHitAttributeHelper.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.util.Collections;\nimport java.util.IdentityHashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.function.Function;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.visitors.MultiTermDisjunctionRankVisitor;\nimport com.twitter.search.queryparser.visitors.NodeRankAnnotator;\nimport com.twitter.search.queryparser.visitors.QueryTreeIndex;\n\n/**\n * A helper class to collect field and query node hit attributions.\n */\npublic class QueryHitAttributeHelper extends HitAttributeHelper {\n  private final Query annotatedQuery;\n\n  protected QueryHitAttributeHelper(HitAttributeCollector collector,\n                                    Function<Integer, String> fieldIdsToFieldNames,\n                                    IdentityHashMap<Query, Integer> nodeToRankMap,\n                                    Query annotatedQuery,\n                                    Map<Query, List<Integer>> expandedRanksMap) {\n    super(collector, fieldIdsToFieldNames, nodeToRankMap, expandedRanksMap);\n    this.annotatedQuery = annotatedQuery;\n  }\n\n  /**\n   * Constructor specific for com.twitter.search.queryParser.query.Query\n   *\n   * This helper visits a parsed query to construct a node-to-rank mapping,\n   * and uses a schema to determine all of the possible fields to be tracked.\n   * A collector is then created.\n   *\n   * @param query the query for which we will collect hit attribution.\n   * @param schema the indexing schema.\n   */\n  public static QueryHitAttributeHelper from(Query query, final Schema schema)\n      throws QueryParserException {\n    IdentityHashMap<Query, Integer> nodeToRankMap;\n    Query annotatedQuery;\n\n    // First see if the query already has node rank annotations on it. If so, we'll just use those\n    // to identify query nodes.\n    // We enforce that all provided ranks are in the range of [0, N-1] so not to blow up the size\n    // of the collection array.\n    QueryRankVisitor rankVisitor = new QueryRankVisitor();\n    if (query.accept(rankVisitor)) {\n      nodeToRankMap = rankVisitor.getNodeToRankMap();\n      annotatedQuery = query;\n    } else {\n      // Otherwise, we will assign all nodes in-order ranks, and use those to track per-node hit\n      // attribution\n      QueryTreeIndex queryTreeIndex = QueryTreeIndex.buildFor(query);\n      NodeRankAnnotator annotator = new NodeRankAnnotator(queryTreeIndex.getNodeToIndexMap());\n      annotatedQuery = query.accept(annotator);\n      nodeToRankMap = annotator.getUpdatedNodeToRankMap();\n    }\n\n    // Extract ranks for multi_term_disjunction operators\n    MultiTermDisjunctionRankVisitor multiTermDisjunctionRankVisitor =\n        new MultiTermDisjunctionRankVisitor(Collections.max(nodeToRankMap.values()));\n    annotatedQuery.accept(multiTermDisjunctionRankVisitor);\n    Map<Query, List<Integer>> expandedRanksMap =\n        multiTermDisjunctionRankVisitor.getMultiTermDisjunctionRankExpansionsMap();\n\n    return new QueryHitAttributeHelper(\n        new HitAttributeCollector(),\n        (fieldId) -> schema.getFieldName(fieldId),\n        nodeToRankMap,\n        annotatedQuery,\n        expandedRanksMap);\n  }\n\n  public Query getAnnotatedQuery() {\n    return annotatedQuery;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/QueryRankVisitor.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.util.IdentityHashMap;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Maps;\n\nimport com.twitter.search.queryparser.query.BooleanQuery;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.annotation.Annotation;\nimport com.twitter.search.queryparser.visitors.DetectAnnotationVisitor;\n\n/**\n * A visitor that collects node ranks from :r annotation in the query\n */\npublic class QueryRankVisitor extends DetectAnnotationVisitor {\n  private final IdentityHashMap<Query, Integer> nodeToRankMap = Maps.newIdentityHashMap();\n\n  public QueryRankVisitor() {\n    super(Annotation.Type.NODE_RANK);\n  }\n\n  @Override\n  protected boolean visitBooleanQuery(BooleanQuery query) throws QueryParserException {\n    if (query.hasAnnotationType(Annotation.Type.NODE_RANK)) {\n      collectNodeRank(query.getAnnotationOf(Annotation.Type.NODE_RANK).get(), query);\n    }\n\n    boolean found = false;\n    for (Query child : query.getChildren()) {\n      found |= child.accept(this);\n    }\n    return found;\n  }\n\n  @Override\n  protected boolean visitQuery(Query query) throws QueryParserException {\n    if (query.hasAnnotationType(Annotation.Type.NODE_RANK)) {\n      collectNodeRank(query.getAnnotationOf(Annotation.Type.NODE_RANK).get(), query);\n      return true;\n    }\n\n    return false;\n  }\n\n  private void collectNodeRank(Annotation anno, Query query) {\n    Preconditions.checkArgument(anno.getType() == Annotation.Type.NODE_RANK);\n    int rank = (Integer) anno.getValue();\n    nodeToRankMap.put(query, rank);\n  }\n\n  public IdentityHashMap<Query, Integer> getNodeToRankMap() {\n    return nodeToRankMap;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/SingleDocDocIdSetIterator.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.search.DocIdSetIterator;\n\npublic class SingleDocDocIdSetIterator extends DocIdSetIterator {\n\n  // the only docid in the list\n  private final int doc;\n\n  private int docid = -1;\n\n  public SingleDocDocIdSetIterator(int doc) {\n    this.doc = doc;\n  }\n\n  @Override\n  public int docID() {\n    return docid;\n  }\n\n  @Override\n  public int nextDoc() throws IOException {\n    if (docid == -1) {\n      docid = doc;\n    } else {\n      docid = NO_MORE_DOCS;\n    }\n    return docid;\n  }\n\n  @Override\n  public int advance(int target) throws IOException {\n    if (docid == NO_MORE_DOCS) {\n      return docid;\n    } else if (doc < target) {\n      docid = NO_MORE_DOCS;\n      return docid;\n    } else {\n      docid = doc;\n    }\n    return docid;\n  }\n\n  @Override\n  public long cost() {\n    return 1;\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/query/StaticHitAttributeProvider.java",
    "content": "package com.twitter.search.common.query;\n\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Map;\n\n/**\n * A hit attribute provider based on the static data\n */\npublic class StaticHitAttributeProvider implements HitAttributeProvider {\n  private int currentDocId;\n  private Map<Integer, List<String>> currentHitAttr;\n\n  public StaticHitAttributeProvider() {\n  }\n\n  /**\n   * Set a fake last doc id and hit attribution, this is only used to generate explanation.\n   */\n  public void setCurrentHitAttr(int docId, Map<Integer, List<String>> hitAttr) {\n    this.currentDocId = docId;\n    this.currentHitAttr = hitAttr;\n  }\n\n  @Override\n  public Map<Integer, List<String>> getHitAttribution(int docId) {\n    if (docId == currentDocId) {\n      return currentHitAttr;\n    }\n    return Collections.EMPTY_MAP;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/BUILD",
    "content": "java_library(\n    name = \"utils\",\n    sources = [\"utils/*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/twitter/elephantbird:core\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/text/language:locale-util\",\n        \"src/java/com/twitter/search/common/partitioning/snowflakeparser\",\n        \"src/java/com/twitter/search/common/relevance/features\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/common/tweetypie\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n        \"src/thrift/com/twitter/search/common:schema-java\",\n    ],\n)\n\njava_library(\n    name = \"ranking\",\n    sources = [\"ranking/**/*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":utils\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/twitter/elephantbird:core\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/search/common/logging\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/relevance/features\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n    ],\n)\n\nTRENDS_DATA_SERVICE_SOURCES = [\n    \"TrendsThriftDataServiceManager.java\",\n    \"NGramCache.java\",\n]\n\njava_library(\n    name = \"trends-data-service\",\n    sources = TRENDS_DATA_SERVICE_SOURCES,\n    platform = \"java8\",\n    provides = artifact(\n        org = \"com.twitter.search.common.relevance\",\n        name = \"trends-data-service\",\n        repo = artifactory,\n    ),\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/io/netty:netty4-tcnative-boringssl-static\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/client\",\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-thrift/src/main/java\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/penguin/search/filter\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/thrift/com/twitter/trends/plus:trends-plus-java\",\n        \"src/thrift/com/twitter/trends/service/gen:trends_service-java\",\n        \"src/thrift/com/twitter/trends/trending_content:trending-content-service-java\",\n        \"trends/trends_metadata/thrift/src/main/thrift/com/twitter/trends/trends_metadata:thrift-java\",\n        \"twitter-server-internal/src/main/scala\",\n        \"util/util-core:scala\",\n        \"util/util-stats/src/main/scala\",\n    ],\n)\n\njava_library(\n    name = \"feature-update-reader\",\n    sources = [\"readers/*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/org/apache/bookkeeper:bookkeeper-server\",\n        \"3rdparty/jvm/org/apache/bookkeeper:bookkeeper-twitter-science-provider\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/util:system-mocks\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/util/io:record-reader-api\",\n        \"src/java/com/twitter/search/common/util/thrift:text-protocol\",\n        \"src/thrift/com/twitter/search/common:schema-java\",\n    ],\n)\n\ntarget(\n    dependencies = [\n        \":feature-update-reader\",\n        \":trends-data-service\",\n        \"src/java/com/twitter/search/common/relevance/features\",\n    ],\n)\n\njava_library(\n    name = \"config\",\n    sources = [\"config/**/*.java\"],\n    platform = \"java8\",\n    provides = artifact(\n        org = \"com.twitter.search.common.relevance\",\n        name = \"config\",\n        repo = artifactory,\n    ),\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"src/java/com/twitter/search/common/config\",\n        \"src/resources/com/twitter/search/common/relevance/config\",\n    ],\n)\n\njava_library(\n    name = \"classifiers\",\n    sources = [\"classifiers/**/*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":config\",\n        \":entities_and_filters\",\n        \":trends-data-service\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/twitter/elephantbird:core\",\n        \"3rdparty/jvm/commons-lang\",\n        \"3rdparty/jvm/geo/google:geoGoogle\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/collections\",\n        \"src/java/com/twitter/common/text/language:locale-util\",\n        \"src/java/com/twitter/common/text/token\",\n        \"src/java/com/twitter/common/text/transformer\",\n        \"src/java/com/twitter/common_internal/text:text-penguin7\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/search/common/config\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/relevance/features\",\n        \"src/java/com/twitter/search/common/util/io/periodic\",\n        \"src/java/com/twitter/search/common/util/text\",\n        \"twitter-text/lib/java/src/main/java/com/twitter/twittertext\",\n    ],\n)\n\njava_library(\n    name = \"text\",\n    sources = [\"text/**/*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":entities_and_filters\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"src/java/com/twitter/common/text/token\",\n        \"src/java/com/twitter/common/text/util:char-seq-util\",\n        \"src/java/com/twitter/common_internal/text:text-penguin7\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/search/common/relevance/features\",\n        \"src/java/com/twitter/search/common/util/text\",\n        \"src/java/com/twitter/search/common/util/text/regex\",\n        \"src/thrift/com/twitter/search/common:indexing-java\",\n    ],\n)\n\njava_library(\n    name = \"scorers\",\n    sources = [\"scorers/**/*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":classifiers\",\n        \":config\",\n        \":entities_and_filters\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/twitter/elephantbird:core\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/search/common/encoding/features\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/relevance/features\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n    ],\n)\n\njava_library(\n    name = \"entities_and_filters\",\n    sources = [\n        \"entities/**/*.java\",\n        \"filters/**/*.java\",\n    ],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/code/findbugs:jsr305\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/commons-lang\",\n        \"3rdparty/jvm/geo/google:geoGoogle\",\n        \"3rdparty/jvm/org/apache/commons:commons-lang3\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"cuad/projects/ner/thrift/src/main/thrift:thrift-java\",\n        \"decider/src/main/scala\",\n        \"src/java/com/twitter/common/text/extractor\",\n        \"src/java/com/twitter/common/text/language:locale-util\",\n        \"src/java/com/twitter/common/text/pipeline\",\n        \"src/java/com/twitter/common/text/token\",\n        \"src/java/com/twitter/common/text/transformer\",\n        \"src/java/com/twitter/common_internal/text:text-penguin7\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/search/common/decider\",\n        \"src/java/com/twitter/search/common/encoding/features\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/partitioning/snowflakeparser\",\n        \"src/java/com/twitter/search/common/relevance/features\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/common/util/text\",\n        \"src/thrift/com/twitter/search/common:indexing-java\",\n        \"src/thrift/com/twitter/service/spiderduck/gen:metadata-store-java\",\n        \"src/thrift/com/twitter/tweetypie:tweet-java\",\n        \"util/util-core:scala\",\n    ],\n)\n\njava_library(\n    name = \"scores\",\n    sources = [\"scores/**/*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/NGramCache.java",
    "content": "package com.twitter.search.common.relevance;\n\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Locale;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.cache.CacheBuilder;\nimport com.google.common.collect.ImmutableList;\n\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.penguin.search.filter.StringMatchFilter;\nimport com.twitter.util.Duration;\n\n/**\n * the Cache for Trends\n */\npublic class NGramCache {\n  private static final int DEFAULT_MAX_CACHE_SIZE = 5000;\n  private static final long DEFAULT_CACHE_ITEM_TTL_SEC = 24 * 3600; // 1 day\n\n  private final PenguinVersion penguinVersion;\n\n  // Keys are trends. Values are empty strings.\n  private final Map<String, String> trendsCache;\n\n  private volatile StringMatchFilter trendsMatcher = null;\n\n  /**\n   * Extract Trends from a list of normalized tokens\n   */\n  public List<String> extractTrendsFromNormalized(List<String> tokens) {\n    if (trendsMatcher == null) {\n      return Collections.emptyList();\n    }\n\n    ImmutableList.Builder<String> trends = ImmutableList.builder();\n    for (String trend : trendsMatcher.extractNormalized(tokens)) {\n      if (trendsCache.containsKey(trend)) {\n        trends.add(trend);\n      }\n    }\n\n    return trends.build();\n  }\n\n  /**\n   * Extract Trends from a list of tokens\n   */\n  public List<String> extractTrendsFrom(List<String> tokens, Locale language) {\n    if (trendsMatcher == null) {\n      return Collections.emptyList();\n    }\n    return trendsMatcher.extract(language, tokens);\n  }\n\n  /**\n   * Extract Trends from a given CharSequence\n   */\n  public List<String> extractTrendsFrom(CharSequence text, Locale language) {\n    if (trendsMatcher == null) {\n      return Collections.emptyList();\n    }\n\n    ImmutableList.Builder<String> trends = ImmutableList.builder();\n    for (String trend : trendsMatcher.extract(language, text)) {\n      if (trendsCache.containsKey(trend)) {\n        trends.add(trend);\n      }\n    }\n\n    return trends.build();\n  }\n\n  public long numTrendingTerms() {\n    return trendsCache.size();\n  }\n\n  public Set<String> getTrends() {\n    return trendsCache.keySet();\n  }\n\n  public void clear() {\n    trendsCache.clear();\n    trendsMatcher = null;\n  }\n\n  /** Adds all trends to this NGramCache. */\n  public void addAll(Iterable<String> trends) {\n    for (String trend : trends) {\n      trendsCache.put(trend, \"\");\n    }\n\n    trendsMatcher = new StringMatchFilter(trendsCache.keySet(), penguinVersion);\n  }\n\n  public static Builder builder() {\n    return new Builder();\n  }\n\n  public static class Builder {\n    private int maxCacheSize = DEFAULT_MAX_CACHE_SIZE;\n    private long cacheItemTTLSecs = DEFAULT_CACHE_ITEM_TTL_SEC; // 1 day\n    private PenguinVersion penguinVersion = PenguinVersion.PENGUIN_4;\n\n    public Builder maxCacheSize(int cacheSize) {\n      this.maxCacheSize = cacheSize;\n      return this;\n    }\n\n    public Builder cacheItemTTL(long cacheItemTTL) {\n      this.cacheItemTTLSecs = cacheItemTTL;\n      return this;\n    }\n\n    public Builder penguinVersion(PenguinVersion newPenguinVersion) {\n      this.penguinVersion = Preconditions.checkNotNull(newPenguinVersion);\n      return this;\n    }\n\n    /** Builds an NGramCache instance. */\n    public NGramCache build() {\n      return new NGramCache(\n          maxCacheSize,\n          Duration.apply(cacheItemTTLSecs, TimeUnit.SECONDS),\n          penguinVersion);\n    }\n  }\n\n  // Should be used only in tests that want to mock out this class.\n  @VisibleForTesting\n  public NGramCache() {\n    this(DEFAULT_MAX_CACHE_SIZE,\n         Duration.apply(DEFAULT_CACHE_ITEM_TTL_SEC, TimeUnit.SECONDS),\n         PenguinVersion.PENGUIN_4);\n  }\n\n  private NGramCache(int maxCacheSize, Duration cacheItemTTL, PenguinVersion penguinVersion) {\n    // we only have 1 refresher thread that writes to the cache\n    this.trendsCache = CacheBuilder.newBuilder()\n        .concurrencyLevel(1)\n        .expireAfterWrite(cacheItemTTL.inSeconds(), TimeUnit.SECONDS)\n        .maximumSize(maxCacheSize)\n        .<String, String>build()\n        .asMap();\n    this.penguinVersion = penguinVersion;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/TrendsThriftDataServiceManager.java",
    "content": "package com.twitter.search.common.relevance;\n\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.concurrent.Executors;\nimport java.util.concurrent.ScheduledExecutorService;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.atomic.AtomicLong;\nimport java.util.stream.Collectors;\n\nimport scala.runtime.BoxedUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Sets;\nimport com.google.common.util.concurrent.ThreadFactoryBuilder;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.ThriftMux;\nimport com.twitter.finagle.builder.ClientBuilder;\nimport com.twitter.finagle.builder.ClientConfig;\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier;\nimport com.twitter.finagle.mtls.client.MtlsClientBuilder;\nimport com.twitter.finagle.stats.DefaultStatsReceiver;\nimport com.twitter.finagle.thrift.ThriftClientRequest;\nimport com.twitter.search.common.metrics.RelevanceStats;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.trends.plus.Module;\nimport com.twitter.trends.plus.TrendsPlusRequest;\nimport com.twitter.trends.plus.TrendsPlusResponse;\nimport com.twitter.trends.service.gen.Location;\nimport com.twitter.trends.trending_content.thriftjava.TrendingContentService;\nimport com.twitter.trends.trends_metadata.thriftjava.TrendsMetadataService;\nimport com.twitter.util.Duration;\nimport com.twitter.util.Future;\nimport com.twitter.util.Try;\n\n/**\n * Manages trends data retrieved from trends thrift API and perform automatic refresh.\n */\npublic final class TrendsThriftDataServiceManager {\n  private static final Logger LOG =\n    LoggerFactory.getLogger(TrendsThriftDataServiceManager.class.getName());\n\n  private static final int DEFAULT_TIME_TO_KILL_SEC = 60;\n\n  @VisibleForTesting\n  protected static final Map<String, String> DEFAULT_TRENDS_PARAMS_MAP = ImmutableMap.of(\n      \"MAX_ITEMS_TO_RETURN\", \"10\");   // we only take top 10 for each woeid.\n\n  @VisibleForTesting\n  protected static final int MAX_TRENDS_PER_WOEID = 10;\n\n  private final Duration requestTimeout;\n  private final Duration refreshDelayDuration;\n  private final Duration reloadIntervalDuration;\n  private final int numRetries;\n\n  // a list of trends cache we want to update\n  private final List<NGramCache> trendsCacheList;\n\n  private final SearchCounter getAvailableSuccessCounter =\n      RelevanceStats.exportLong(\"trends_extractor_get_available_success\");\n  private final SearchCounter getAvailableFailureCounter =\n      RelevanceStats.exportLong(\"trends_extractor_get_available_failure\");\n  private final SearchCounter getTrendsSuccessCounter =\n      RelevanceStats.exportLong(\"trends_extractor_success_fetch\");\n  private final SearchCounter getTrendsFailureCounter =\n      RelevanceStats.exportLong(\"trends_extractor_failed_fetch\");\n  private final SearchCounter updateFailureCounter =\n      RelevanceStats.exportLong(\"trends_extractor_failed_update\");\n\n  private final ServiceIdentifier serviceIdentifier;\n  private ScheduledExecutorService scheduler;\n\n\n  @VisibleForTesting\n  protected Service<ThriftClientRequest, byte[]> contentService;\n  protected TrendingContentService.ServiceToClient contentClient;\n  protected Service<ThriftClientRequest, byte[]> metadataService;\n  protected TrendsMetadataService.ServiceToClient metadataClient;\n\n  @VisibleForTesting\n  protected TrendsUpdater trendsUpdater;\n\n  /**\n   * Returns an instance of TrendsThriftDataServiceManager.\n   * @param serviceIdentifier The service that wants to call\n   * into Trend's services.\n   * @param numRetries The number of retries in the event of\n   * request failures.\n   * @param requestTimeout The amount of time we wait before we consider a\n   * a request as failed.\n   * @param initTrendsCacheDelay How long to wait before the initial\n   * filling of the Trends cache in milliseconds.\n   * @param reloadInterval How often to refresh the cache with updated trends.\n   * @param trendsCacheList The cache of trends.\n   * @return An instance of TrendsThriftDataServiceManager configured\n   * with respect to the params provided.\n   */\n  public static TrendsThriftDataServiceManager newInstance(\n      ServiceIdentifier serviceIdentifier,\n      int numRetries,\n      Duration requestTimeout,\n      Duration initTrendsCacheDelay,\n      Duration reloadInterval,\n      List<NGramCache> trendsCacheList) {\n    return new TrendsThriftDataServiceManager(\n        serviceIdentifier,\n        numRetries,\n        requestTimeout,\n        initTrendsCacheDelay,\n        reloadInterval,\n        trendsCacheList);\n  }\n\n  /**\n   * Resume auto refresh. Always called in constructor. Can be invoked after a\n   * stopAuthRefresh call to resume auto refreshing. Invoking it after shutDown is undefined.\n   */\n  public synchronized void startAutoRefresh() {\n    if (scheduler == null) {\n      scheduler = Executors.newSingleThreadScheduledExecutor(\n          new ThreadFactoryBuilder().setDaemon(true).setNameFormat(\n              \"trends-data-refresher[%d]\").build());\n      scheduler.scheduleAtFixedRate(\n          trendsUpdater,\n          refreshDelayDuration.inSeconds(),\n          reloadIntervalDuration.inSeconds(),\n          TimeUnit.SECONDS);\n    }\n  }\n\n  /**\n   * Stop auto refresh. Wait for the current execution thread to finish.\n   * This is a blocking call.\n   */\n  public synchronized void stopAutoRefresh() {\n    if (scheduler != null) {\n      scheduler.shutdown(); // Disable new tasks from being submitted\n      try {\n        // Wait a while for existing tasks to terminate\n        if (!scheduler.awaitTermination(DEFAULT_TIME_TO_KILL_SEC, TimeUnit.SECONDS)) {\n          scheduler.shutdownNow(); // Cancel currently executing tasks\n          // Wait a while for tasks to respond to being cancelled\n          if (!scheduler.awaitTermination(DEFAULT_TIME_TO_KILL_SEC, TimeUnit.SECONDS)) {\n            LOG.info(\"Executor thread pool did not terminate.\");\n          }\n        }\n      } catch (InterruptedException ie) {\n        // (Re-)Cancel if current thread also interrupted\n        scheduler.shutdownNow();\n        // Preserve interrupt status\n        Thread.currentThread().interrupt();\n      }\n      scheduler = null;\n    }\n  }\n\n  /** Shuts down the manager. */\n  public void shutDown() {\n    stopAutoRefresh();\n    // clear the cache\n    for (NGramCache cache : trendsCacheList) {\n      cache.clear();\n    }\n\n    if (contentService != null) {\n      contentService.close();\n    }\n\n    if (metadataService != null) {\n      metadataService.close();\n    }\n  }\n\n  private TrendsThriftDataServiceManager(\n      ServiceIdentifier serviceIdentifier,\n      int numRetries,\n      Duration requestTimeoutMS,\n      Duration refreshDelayDuration,\n      Duration reloadIntervalDuration,\n      List<NGramCache> trendsCacheList) {\n    this.numRetries = numRetries;\n    this.requestTimeout = requestTimeoutMS;\n    this.refreshDelayDuration = refreshDelayDuration;\n    this.reloadIntervalDuration = reloadIntervalDuration;\n    this.serviceIdentifier = serviceIdentifier;\n    this.trendsCacheList = Preconditions.checkNotNull(trendsCacheList);\n    trendsUpdater = new TrendsUpdater();\n    metadataService = buildMetadataService();\n    metadataClient = buildMetadataClient(metadataService);\n    contentService = buildContentService();\n    contentClient = buildContentClient(contentService);\n  }\n\n  @VisibleForTesting\n  protected Service<ThriftClientRequest, byte[]> buildContentService() {\n    ClientBuilder<\n        ThriftClientRequest,\n        byte[], ClientConfig.Yes,\n        ClientConfig.Yes,\n        ClientConfig.Yes\n        >\n        builder = ClientBuilder.get()\n          .stack(ThriftMux.client())\n          .name(\"trends_thrift_data_service_manager_content\")\n          .dest(\"\")\n          .retries(numRetries)\n          .reportTo(DefaultStatsReceiver.get())\n          .tcpConnectTimeout(requestTimeout)\n          .requestTimeout(requestTimeout);\n    ClientBuilder mtlsBuilder =\n        new MtlsClientBuilder.MtlsClientBuilderSyntax<>(builder).mutualTls(serviceIdentifier);\n\n    return ClientBuilder.safeBuild(mtlsBuilder);\n  }\n\n  @VisibleForTesting\n  protected TrendingContentService.ServiceToClient buildContentClient(\n      Service<ThriftClientRequest, byte[]> service) {\n    return new TrendingContentService.ServiceToClient(service);\n  }\n\n  @VisibleForTesting\n  protected Service<ThriftClientRequest, byte[]> buildMetadataService() {\n    ClientBuilder<\n        ThriftClientRequest,\n        byte[],\n        ClientConfig.Yes,\n        ClientConfig.Yes,\n        ClientConfig.Yes\n        >\n        builder = ClientBuilder.get()\n          .stack(ThriftMux.client())\n          .name(\"trends_thrift_data_service_manager_metadata\")\n          .dest(\"\")\n          .retries(numRetries)\n          .reportTo(DefaultStatsReceiver.get())\n          .tcpConnectTimeout(requestTimeout)\n          .requestTimeout(requestTimeout);\n    ClientBuilder mtlsBuilder =\n        new MtlsClientBuilder.MtlsClientBuilderSyntax<>(builder).mutualTls(serviceIdentifier);\n\n    return ClientBuilder.safeBuild(mtlsBuilder);\n  }\n\n  @VisibleForTesting\n  protected TrendsMetadataService.ServiceToClient buildMetadataClient(\n      Service<ThriftClientRequest, byte[]> service) {\n    return new TrendsMetadataService.ServiceToClient(service);\n  }\n\n  /**\n   * Updater that fetches available woeids and corresponding trending terms.\n   */\n  @VisibleForTesting\n  protected class TrendsUpdater implements Runnable {\n    @Override\n    public void run() {\n      populateCacheFromTrendsService();\n    }\n\n    private Future<BoxedUnit> populateCacheFromTrendsService() {\n      long startTime = System.currentTimeMillis();\n      AtomicLong numTrendsReceived = new AtomicLong(0);\n      return metadataClient.getAvailable().flatMap(locations -> {\n        if (locations == null) {\n          getAvailableFailureCounter.increment();\n          LOG.warn(\"Failed to get woeids from trends.\");\n          return Future.value(BoxedUnit.UNIT);\n        }\n        getAvailableSuccessCounter.increment();\n        return populateCacheFromTrendLocations(locations, numTrendsReceived);\n      }).onFailure(throwable -> {\n        LOG.info(\"Update failed\", throwable);\n        updateFailureCounter.increment();\n        return BoxedUnit.UNIT;\n      }).ensure(() -> {\n        logRefreshStatus(startTime, numTrendsReceived);\n        return BoxedUnit.UNIT;\n      });\n    }\n\n    private Future<BoxedUnit> populateCacheFromTrendLocations(\n        List<Location> locations,\n        AtomicLong numTrendsReceived) {\n      List<Future<TrendsPlusResponse>> trendsPlusFutures = locations.stream()\n          .map(location -> makeTrendsPlusRequest(location))\n          .collect(Collectors.toList());\n\n      Future<List<Try<TrendsPlusResponse>>> trendsPlusFuture =\n          Future.collectToTry(trendsPlusFutures);\n      return trendsPlusFuture.map(tryResponses -> {\n        populateCacheFromResponses(tryResponses, numTrendsReceived);\n        return BoxedUnit.UNIT;\n      });\n    }\n\n    private Future<TrendsPlusResponse> makeTrendsPlusRequest(Location location) {\n      TrendsPlusRequest request = new TrendsPlusRequest()\n          .setWoeid(location.getWoeid())\n          .setMaxTrends(MAX_TRENDS_PER_WOEID);\n      long startTime = System.currentTimeMillis();\n      return contentClient.getTrendsPlus(request)\n          .onSuccess(response -> {\n            getTrendsSuccessCounter.increment();\n            return BoxedUnit.UNIT;\n          }).onFailure(throwable -> {\n            getTrendsFailureCounter.increment();\n            return BoxedUnit.UNIT;\n          });\n    }\n\n    private void populateCacheFromResponses(\n        List<Try<TrendsPlusResponse>> tryResponses,\n        AtomicLong numTrendsReceived) {\n      Set<String> trendStrings = Sets.newHashSet();\n\n      for (Try<TrendsPlusResponse> tryResponse : tryResponses) {\n        if (tryResponse.isThrow()) {\n          LOG.warn(\"Failed to fetch trends:\" + tryResponse.toString());\n          continue;\n        }\n\n        TrendsPlusResponse trendsPlusResponse = tryResponse.get();\n        numTrendsReceived.addAndGet(trendsPlusResponse.modules.size());\n        for (Module module : trendsPlusResponse.modules) {\n          trendStrings.add(module.getTrend().name);\n        }\n      }\n\n      for (NGramCache cache : trendsCacheList) {\n        cache.addAll(trendStrings);\n      }\n    }\n  }\n\n  private void logRefreshStatus(long startTime, AtomicLong numTrendsReceived) {\n    LOG.info(String.format(\"Refresh done in [%dms] :\\nfetchSuccess[%d] fetchFailure[%d] \"\n            + \"updateFailure[%d] num trends received [%d]\",\n        System.currentTimeMillis() - startTime,\n        getTrendsSuccessCounter.get(),\n        getTrendsFailureCounter.get(),\n        updateFailureCounter.get(),\n        numTrendsReceived.get()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/classifiers/TweetClassifier.java",
    "content": "package com.twitter.search.common.relevance.classifiers;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\n\n/**\n * Interface to perform feature classification for a single\n * @TwitterMessage object, or a group of them.\n *\n * Classification includes two steps: feature extraction, and\n * quality evaluation. During feature extraction, any interesting\n * feature that is deemed useful for subsequent quality analysis\n * is extracted from the @TwitterMessage object. Quality evaluation\n * is then done by a group of @TweetEvaluator objects associated\n * with the classifier, by using the various features extracted in the\n * previous step.\n *\n * Feature extraction and quality evaluation results are stored in\n * @TweetFeatures field of the @TwitterMessage object, which is defined\n * in src/main/thrift/classifier.thrift.\n */\npublic abstract class TweetClassifier {\n  /**\n   * A list of TweetQualityEvaluators which are invoked after\n   * feature extraction is done. If null, no quality evaluation\n   * is done.\n   */\n  protected Iterable<TweetEvaluator> qualityEvaluators = null;\n\n  /**\n   * Passed in TwitterMessage is examined and any extractable\n   * features are saved in TweetFeatures field of TwitterMessage.\n   * Then TweetQualityEvaluators are applied to compute various\n   * quality values.\n   *\n   * @param tweet TwitterMessage to perform classification on.\n   */\n  public void classifyTweet(final TwitterMessage tweet) {\n    Preconditions.checkNotNull(tweet);\n\n    // extract features\n    extractFeatures(tweet);\n\n    // compute quality\n    evaluate(tweet);\n  }\n\n  /**\n   * Classify a group of TwitterMessages and store features in their corresponding\n   * TweetFeatures fields.\n   *\n   * This default implementation just iterates through the map and classifies each\n   * individual tweet. Batching for better performance, if applicable, can be implemented by\n   * concrete subclasses.\n   *\n   * @param tweets TwitterMessages to perform classification on.\n   */\n  public void classifyTweets(final Iterable<TwitterMessage> tweets) {\n    extractFeatures(tweets);\n    evaluate(tweets);\n  }\n\n  /**\n   * Use the specified list of TweetQualityEvaluators for this classifier.\n   *\n   * @param evaluators list of TweetQualityEvaluators to be used with this classifier.\n   */\n  protected void setQualityEvaluators(final Iterable<TweetEvaluator> qualityEvaluators) {\n    Preconditions.checkNotNull(qualityEvaluators);\n    this.qualityEvaluators = qualityEvaluators;\n  }\n\n\n  /**\n   * Extract interesting features from a single TwitterMessage for classification.\n   *\n   * @param tweet TwitterMessage to extract interesting features for\n   */\n  protected abstract void extractFeatures(final TwitterMessage tweet);\n\n  /**\n   * Extract interesting features from a list of TwitterMessages for classification.\n   * @param tweets list of TwitterMessages to extract interesting features for\n   */\n  protected void extractFeatures(final Iterable<TwitterMessage> tweets) {\n    for (TwitterMessage tweet: tweets) {\n      extractFeatures(tweet);\n    }\n  }\n\n  /**\n   * Given a TwitterMessage which already has its features extracted,\n   * perform quality evaluation.\n   *\n   * @param tweet TwitterMessage to perform quality evaluation for\n   */\n  protected void evaluate(final TwitterMessage tweet) {\n    if (qualityEvaluators == null) {\n      return;\n    }\n    for (TweetEvaluator evaluator : qualityEvaluators) {\n      evaluator.evaluate(tweet);\n    }\n  }\n\n  /**\n   * Given a list of TwitterMessages which already have their features extracted,\n   * perform quality evaluation.\n   *\n   * @param tweets list of TwitterMessages to perform quality evaluation for\n   */\n  protected void evaluate(final Iterable<TwitterMessage> tweets) {\n    for (TwitterMessage tweet: tweets) {\n      evaluate(tweet);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/classifiers/TweetEvaluator.java",
    "content": "package com.twitter.search.common.relevance.classifiers;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\n\n/**\n * Interface to perform quality evaluation for a single @TwitterMessage\n * object or a group of them.\n *\n */\npublic abstract class TweetEvaluator {\n  /**\n   * Passed in TwitterMessage is examined and any extractable\n   * features are stored in TweetFeatures field of TwitterMessage.\n   *\n   * @param tweet TwitterMessage to perform classification on.\n   */\n  public abstract void evaluate(final TwitterMessage tweet);\n\n  /**\n   * Classify a group of TwitterMessages and store the features in their corresponding\n   * TweetFeatures fields.\n   *\n   * This default implementation just iterates through the map and classifies each\n   * individual tweet. Batching for better performance, if applicable, can be implemented by\n   * concrete subclasses.\n   *\n   * @param tweets TwitterMessages to perform classification on.\n   */\n   public void evaluate(final Iterable<TwitterMessage> tweets) {\n    Preconditions.checkNotNull(tweets);\n    for (TwitterMessage tweet: tweets) {\n      evaluate(tweet);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/classifiers/TweetOffensiveEvaluator.java",
    "content": "package com.twitter.search.common.relevance.classifiers;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.concurrent.Executors;\nimport java.util.concurrent.ScheduledExecutorService;\nimport java.util.concurrent.atomic.AtomicReference;\n\nimport com.google.common.base.Joiner;\nimport com.google.common.base.Preconditions;\nimport com.google.common.io.ByteSource;\nimport com.google.common.util.concurrent.ThreadFactoryBuilder;\n\nimport org.apache.commons.io.IOUtils;\nimport org.apache.commons.lang.StringUtils;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.text.language.LocaleUtil;\nimport com.twitter.common.text.token.TokenizedCharSequence;\nimport com.twitter.common.text.token.attribute.TokenType;\nimport com.twitter.common.util.Clock;\nimport com.twitter.common_internal.text.pipeline.TwitterNgramGenerator;\nimport com.twitter.common_internal.text.topic.BlacklistedTopics;\nimport com.twitter.common_internal.text.topic.BlacklistedTopics.FilterMode;\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.search.common.metrics.RelevanceStats;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\nimport com.twitter.search.common.relevance.features.TweetTextFeatures;\nimport com.twitter.search.common.relevance.features.TweetTextQuality;\nimport com.twitter.search.common.util.io.periodic.PeriodicFileLoader;\nimport com.twitter.search.common.util.text.NormalizerHelper;\nimport com.twitter.search.common.util.text.TokenizerHelper;\n\n/**\n * Determines if tweet text or username contains potentially offensive language.\n */\npublic class TweetOffensiveEvaluator extends TweetEvaluator {\n  private static final Logger LOG = LoggerFactory.getLogger(TweetOffensiveEvaluator.class);\n\n  private static final int MAX_OFFENSIVE_TERMS = 2;\n\n  private final File filterDirectory;\n  private static final File DEFAULT_FILTER_DIR = new File(\"\");\n  private static final String ADULT_TOKEN_FILE_NAME = \"adult_tokens.txt\";\n  private static final String OFFENSIVE_TOPIC_FILE_NAME = \"offensive_topics.txt\";\n  private static final String OFFENSIVE_SUBSTRING_FILE_NAME = \"offensive_substrings.txt\";\n\n  private static final ThreadLocal<TwitterNgramGenerator> NGRAM_GENERATOR_HOLDER =\n      new ThreadLocal<TwitterNgramGenerator>() {\n        @Override\n        protected TwitterNgramGenerator initialValue() {\n          // It'll generate ngrams from TokenizedCharSequence, which contains tokenization results,\n          // so it doesn't matter which Penguin version to use here.\n          return new TwitterNgramGenerator.Builder(PenguinVersion.PENGUIN_6)\n              .setSize(1, MAX_OFFENSIVE_TERMS)\n              .build();\n        }\n      };\n\n  private final AtomicReference<BlacklistedTopics> offensiveTopics =\n    new AtomicReference<>();\n  private final AtomicReference<BlacklistedTopics> offensiveUsersTopics =\n    new AtomicReference<>();\n\n  private final AtomicReference<ByteSource> adultTokenFileContents = new AtomicReference<>();\n  private final AtomicReference<ByteSource> offensiveTokenFileContents = new AtomicReference<>();\n  private final AtomicReference<ByteSource> offensiveSubstringFileContents = new\n    AtomicReference<>();\n\n  private final SearchCounter sensitiveTextCounter =\n      RelevanceStats.exportLong(\"num_sensitive_text\");\n\n  public TweetOffensiveEvaluator() {\n    this(DEFAULT_FILTER_DIR);\n  }\n\n  public TweetOffensiveEvaluator(\n    File filterDirectory\n  ) {\n    this.filterDirectory = filterDirectory;\n    adultTokenFileContents.set(BlacklistedTopics.getResource(\n      BlacklistedTopics.DATA_PREFIX + ADULT_TOKEN_FILE_NAME));\n    offensiveTokenFileContents.set(BlacklistedTopics.getResource(\n      BlacklistedTopics.DATA_PREFIX + OFFENSIVE_TOPIC_FILE_NAME));\n    offensiveSubstringFileContents.set(BlacklistedTopics.getResource(\n      BlacklistedTopics.DATA_PREFIX + OFFENSIVE_SUBSTRING_FILE_NAME));\n\n    try {\n      rebuildBlacklistedTopics();\n    } catch (IOException e) {\n      throw new RuntimeException(e);\n    }\n\n    ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(\n      new ThreadFactoryBuilder()\n        .setNameFormat(\"offensive-evaluator-blacklist-reloader\")\n        .setDaemon(true)\n        .build());\n    initPeriodicFileLoader(adultTokenFileContents, ADULT_TOKEN_FILE_NAME, executor);\n    initPeriodicFileLoader(offensiveTokenFileContents, OFFENSIVE_TOPIC_FILE_NAME, executor);\n    initPeriodicFileLoader(offensiveSubstringFileContents, OFFENSIVE_SUBSTRING_FILE_NAME, executor);\n  }\n\n  private void initPeriodicFileLoader(\n    AtomicReference<ByteSource> byteSource,\n    String fileName,\n    ScheduledExecutorService executor) {\n    File file = new File(filterDirectory, fileName);\n    try {\n      PeriodicFileLoader loader = new PeriodicFileLoader(\n        \"offensive-evaluator-\" + fileName,\n        file.getPath(),\n        executor,\n        Clock.SYSTEM_CLOCK) {\n        @Override\n        protected void accept(InputStream stream) throws IOException {\n          byteSource.set(ByteSource.wrap(IOUtils.toByteArray(stream)));\n          rebuildBlacklistedTopics();\n        }\n      };\n      loader.init();\n    } catch (Exception e) {\n      // Not the end of the world if we couldn't load the file, we already loaded the resource.\n      LOG.error(\"Could not load offensive topic filter \" + fileName + \" from ConfigBus\", e);\n    }\n  }\n\n  private void rebuildBlacklistedTopics() throws IOException {\n    offensiveTopics.set(new BlacklistedTopics.Builder(false)\n      .loadFilterFromSource(adultTokenFileContents.get(), FilterMode.EXACT)\n      .loadFilterFromSource(offensiveSubstringFileContents.get(), FilterMode.SUBSTRING)\n      .build());\n\n    offensiveUsersTopics.set(new BlacklistedTopics.Builder(false)\n      .loadFilterFromSource(offensiveTokenFileContents.get(), FilterMode.EXACT)\n      .loadFilterFromSource(offensiveSubstringFileContents.get(), FilterMode.SUBSTRING)\n      .build());\n  }\n\n  @Override\n  public void evaluate(final TwitterMessage tweet) {\n    BlacklistedTopics offensiveFilter = this.offensiveTopics.get();\n    BlacklistedTopics offensiveUsersFilter = this.offensiveUsersTopics.get();\n\n    if (offensiveFilter == null || offensiveUsersFilter == null) {\n      return;\n    }\n\n    if (tweet.isSensitiveContent()) {\n      sensitiveTextCounter.increment();\n    }\n\n    // Check for user name.\n    Preconditions.checkState(tweet.getFromUserScreenName().isPresent(),\n        \"Missing from-user screen name\");\n\n    for (PenguinVersion penguinVersion : tweet.getSupportedPenguinVersions()) {\n      TweetTextQuality textQuality = tweet.getTweetTextQuality(penguinVersion);\n\n      if (tweet.isSensitiveContent()) {\n        textQuality.addBoolQuality(TweetTextQuality.BooleanQualityType.SENSITIVE);\n      }\n\n      // Check if username has an offensive term\n      if (isUserNameOffensive(\n          tweet.getFromUserScreenName().get(), offensiveUsersFilter, penguinVersion)) {\n        SearchRateCounter offensiveUserCounter = RelevanceStats.exportRate(\n            \"num_offensive_user_\" + penguinVersion.name().toLowerCase());\n        offensiveUserCounter.increment();\n        textQuality.addBoolQuality(TweetTextQuality.BooleanQualityType.OFFENSIVE_USER);\n      }\n\n      // Check if tweet has an offensive term\n      if (isTweetOffensive(tweet, offensiveFilter, penguinVersion)) {\n        SearchRateCounter offensiveTextCounter = RelevanceStats.exportRate(\n            \"num_offensive_text_\" + penguinVersion.name().toLowerCase());\n        offensiveTextCounter.increment();\n        textQuality.addBoolQuality(TweetTextQuality.BooleanQualityType.OFFENSIVE);\n      }\n    }\n  }\n\n  private boolean isUserNameOffensive(String userName,\n                                      BlacklistedTopics offensiveUsersFilter,\n                                      PenguinVersion penguinVersion) {\n    String normalizedUserName = NormalizerHelper.normalizeKeepCase(\n        userName, LocaleUtil.UNKNOWN, penguinVersion);\n    List<String> termsToCheck = new ArrayList(TokenizerHelper.getSubtokens(normalizedUserName));\n    termsToCheck.add(normalizedUserName.toLowerCase());\n\n    for (String userNameToken : termsToCheck) {\n      if (!StringUtils.isBlank(userNameToken) && offensiveUsersFilter.filter(userNameToken)) {\n        return true;\n      }\n    }\n    return false;\n  }\n\n  private boolean isTweetOffensive(final TwitterMessage tweet,\n                                   BlacklistedTopics offensiveFilter,\n                                   PenguinVersion penguinVersion) {\n    TweetTextFeatures textFeatures = tweet.getTweetTextFeatures(penguinVersion);\n\n    boolean tweetHasOffensiveTerm = false;\n\n    // Check for tweet text.\n    List<TokenizedCharSequence> ngrams =\n        NGRAM_GENERATOR_HOLDER.get().generateNgramsAsTokenizedCharSequence(\n            textFeatures.getTokenSequence(), tweet.getLocale());\n    for (TokenizedCharSequence ngram : ngrams) {\n      // skip URL ngram\n      if (!ngram.getTokensOf(TokenType.URL).isEmpty()) {\n        continue;\n      }\n      String ngramStr = ngram.toString();\n      if (!StringUtils.isBlank(ngramStr) && offensiveFilter.filter(ngramStr)) {\n        tweetHasOffensiveTerm = true;\n        break;\n      }\n    }\n\n    // Due to some strangeness in Penguin, we don't get ngrams for tokens around \"\\n-\" or \"-\\n\"\n    // in the original string, this made us miss some offensive words this way. Here we do another\n    // pass of check using just the tokens generated by the tokenizer. (See SEARCHQUAL-8907)\n    if (!tweetHasOffensiveTerm) {\n      for (String ngramStr : textFeatures.getTokens()) {\n        // skip URLs\n        if (ngramStr.startsWith(\"http://\") || ngramStr.startsWith(\"https://\")) {\n          continue;\n        }\n        if (!StringUtils.isBlank(ngramStr) && offensiveFilter.filter(ngramStr)) {\n          tweetHasOffensiveTerm = true;\n          break;\n        }\n      }\n    }\n\n    if (!tweetHasOffensiveTerm) {\n      // check for resolved URLs\n      String resolvedUrlsText =\n          Joiner.on(\" \").skipNulls().join(textFeatures.getResolvedUrlTokens());\n      List<String> ngramStrs = NGRAM_GENERATOR_HOLDER.get().generateNgramsAsString(\n          resolvedUrlsText, LocaleUtil.UNKNOWN);\n      for (String ngram : ngramStrs) {\n        if (!StringUtils.isBlank(ngram) && offensiveFilter.filter(ngram)) {\n          tweetHasOffensiveTerm = true;\n          break;\n        }\n      }\n    }\n\n    return tweetHasOffensiveTerm;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/classifiers/TweetQualityFeatureExtractor.java",
    "content": "package com.twitter.search.common.relevance.classifiers;\n\nimport java.io.IOException;\nimport java.util.Set;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.text.transformer.RegexTransformer;\nimport com.twitter.common.text.transformer.RtRemovalTransformer;\nimport com.twitter.common.text.transformer.Transformer;\nimport com.twitter.common.text.transformer.TransformerChain;\nimport com.twitter.common_internal.text.duplicate.RandomSubstringExtractor;\nimport com.twitter.common_internal.text.duplicate.SignatureGenerator;\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\nimport com.twitter.search.common.relevance.features.TweetIntegerShingleSignature;\nimport com.twitter.search.common.relevance.features.TweetTextFeatures;\nimport com.twitter.search.common.util.text.NormalizerHelper;\nimport com.twitter.twittertext.Regex;\n\n/**\n * Given a tweet text, extract useful text features.\n */\npublic class TweetQualityFeatureExtractor {\n  private static final Transformer STATUS_TEXT_CLEANER =\n      TransformerChain.of(\n          // remove @reply as defined in twitter-text\n          new RegexTransformer.Builder()\n              .setRegexPattern(Regex.VALID_REPLY)\n              .setReplaceString(\"\")\n              .setTriggeringChar('@')\n              .build(),\n          // remove the old style retweet, eg RT: @mention or via @mention\n          new RtRemovalTransformer()\n      );\n\n  // for signature generation\n  private static final int MIN_NUM_FEATURES = 2;\n  private final SignatureGenerator signatureGenerator = new SignatureGenerator(\n      new RandomSubstringExtractor(\n          TweetIntegerShingleSignature.NUM_SHINGLES, // number of signatures\n          MIN_NUM_FEATURES, // each signature is generated by taking this number of features/tokens\n                            // from text\n          false, // do not consider full tweet text as a feature\n          false)); // do not do early termination\n\n  /**\n   * Given TwitterMessage, extract all interesting tweet text features and store in\n   * the returned TweetTextFeatures object.\n   *\n   * @param tweet TwitterMessage to extract features from\n   * @throws IOException\n   */\n  public void extractTweetTextFeatures(final TwitterMessage tweet) {\n    Preconditions.checkNotNull(tweet);\n\n    for (PenguinVersion penguinVersion : tweet.getSupportedPenguinVersions()) {\n      // Get basic features.\n      TweetTextFeatures textFeatures = tweet.getTweetTextFeatures(penguinVersion);\n\n      extractCharLength(textFeatures);\n\n      // Signature that hashes on text with resolved urls, aggressively remove RT tags, which\n      // accounts for more than 50% of neardups, also remove @mentions.\n      // we use resolved urls for signature since they are what matters.\n      CharSequence strippedText = tweet.getTextReplacedWithResolvedURLs();\n      strippedText = strippedText == null ? \"\" : strippedText;\n      strippedText = STATUS_TEXT_CLEANER.transform(strippedText);\n\n      // Generate the signature.\n      // will lower case, use penguin\n      String normalizedSignatureText =\n        NormalizerHelper.normalize(strippedText, tweet.getLocale(), penguinVersion);\n      if (normalizedSignatureText != null && !normalizedSignatureText.isEmpty()) {\n        Set<byte[]> rawSignature =\n          signatureGenerator.generateSignatureByteArray(normalizedSignatureText);\n        textFeatures.setSignature((new TweetIntegerShingleSignature(rawSignature)).serialize());\n      }\n    }\n  }\n\n  /**\n   * Compute number of letters in stripped tweet text, also records unsupported char counts.\n   *\n   * @param textFeatures TweetTextFeatures object to store letter length, unsupported chars, etc.\n   */\n  private static void extractCharLength(final TweetTextFeatures textFeatures) {\n    Preconditions.checkNotNull(textFeatures);\n    int length = 0;\n    int caps = 0;\n    String strippedText = textFeatures.getNormalizedStrippedText();\n    if (strippedText != null && !strippedText.isEmpty()) {\n      for (char c : strippedText.toCharArray()) {\n        if (Character.isLetter(c)) {\n          length++;\n          if (Character.isUpperCase(c)) {\n            caps++;\n          }\n        }\n      }\n    }\n    textFeatures.setLength(length);\n    textFeatures.setCaps(caps);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/classifiers/TweetTextClassifier.java",
    "content": "package com.twitter.search.common.relevance.classifiers;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\n\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier;\nimport java.util.List;\n\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.search.common.relevance.config.TweetProcessingConfig;\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\n\n/**\n * Classifier that focuses on tweet text features and their corresponding\n * quality.\n */\npublic class TweetTextClassifier extends TweetClassifier {\n  private TweetQualityFeatureExtractor featureExtractor = new TweetQualityFeatureExtractor();\n  private TweetTrendsExtractor trendsExtractor = null;\n\n  /**\n   * Constructor. Requires a list of TweetQualityEvaluator objects.\n   * @param evaluators list of TweetQualityEvaluator objects responsible for quality evaluation.\n   * @param serviceIdentifier The identifier of the calling service.\n   * @param supportedPenguinVersions A list of supported penguin versions.\n   */\n  public TweetTextClassifier(\n      final Iterable<TweetEvaluator> evaluators,\n      ServiceIdentifier serviceIdentifier,\n      List<PenguinVersion> supportedPenguinVersions) {\n    Preconditions.checkNotNull(evaluators);\n    setQualityEvaluators(evaluators);\n    TweetProcessingConfig.init();\n\n    if (TweetProcessingConfig.getBool(\"extract_trends\", false)) {\n      trendsExtractor = new TweetTrendsExtractor(serviceIdentifier, supportedPenguinVersions);\n    }\n  }\n\n  /**\n   * Extract text features for the specified TwitterMessage.\n   *\n   * @param tweet TwitterMessage to extract features from.\n   */\n  @Override\n  protected void extractFeatures(TwitterMessage tweet) {\n    extractFeatures(Lists.newArrayList(tweet));\n  }\n\n  /**\n   * Extract text features for the specified list of TwitterMessages.\n   *\n   * @param tweets list of TwitterMessages to extract interesting features for\n   */\n  @Override\n  protected void extractFeatures(Iterable<TwitterMessage> tweets) {\n    Preconditions.checkNotNull(tweets);\n    for (TwitterMessage tweet : tweets) {\n      featureExtractor.extractTweetTextFeatures(tweet);\n    }\n\n    // Optionally try to annotate trends for all the tweets.\n    if (TweetProcessingConfig.getBool(\"extract_trends\", false) && trendsExtractor != null) {\n      trendsExtractor.extractTrends(tweets);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/classifiers/TweetTextEvaluator.java",
    "content": "package com.twitter.search.common.relevance.classifiers;\n\nimport java.util.List;\nimport java.util.Map;\nimport java.util.function.Function;\nimport java.util.stream.Collectors;\n\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\nimport com.twitter.search.common.relevance.features.TweetTextFeatures;\nimport com.twitter.search.common.relevance.features.TweetTextQuality;\n\n/**\n * Calculates entropy of tweet text based on tokens.\n */\npublic class TweetTextEvaluator extends TweetEvaluator {\n\n  @Override\n  public void evaluate(final TwitterMessage tweet) {\n    for (PenguinVersion penguinVersion : tweet.getSupportedPenguinVersions()) {\n      TweetTextFeatures textFeatures = tweet.getTweetTextFeatures(penguinVersion);\n      TweetTextQuality textQuality = tweet.getTweetTextQuality(penguinVersion);\n\n      double readability = 0;\n      int numKeptWords = textFeatures.getStrippedTokensSize();\n      for (String token : textFeatures.getStrippedTokens()) {\n        readability += token.length();\n      }\n      if (numKeptWords > 0) {\n        readability = readability * Math.log(numKeptWords) / numKeptWords;\n      }\n      textQuality.setReadability(readability);\n      textQuality.setEntropy(entropy(textFeatures.getStrippedTokens()));\n      textQuality.setShout(textFeatures.getCaps() / Math.max(textFeatures.getLength(), 1.0d));\n    }\n  }\n\n  private static double entropy(List<String> tokens) {\n    Map<String, Long> tokenCounts =\n        tokens.stream().collect(Collectors.groupingBy(Function.identity(), Collectors.counting()));\n    int numItems = tokens.size();\n\n    double entropy = 0;\n    for (long count : tokenCounts.values()) {\n      double prob = (double) count / numItems;\n      entropy -= prob * log2(prob);\n    }\n    return entropy;\n  }\n\n  private static double log2(double n) {\n    return Math.log(n) / Math.log(2);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/classifiers/TweetTrendsExtractor.java",
    "content": "package com.twitter.search.common.relevance.classifiers;\n\nimport java.util.List;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.ImmutableMap;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier;\nimport com.twitter.search.common.metrics.RelevanceStats;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.relevance.NGramCache;\nimport com.twitter.search.common.relevance.TrendsThriftDataServiceManager;\nimport com.twitter.search.common.relevance.config.TweetProcessingConfig;\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\nimport com.twitter.search.common.relevance.features.TweetTextFeatures;\nimport com.twitter.util.Duration;\n\n/**\n * Determines if tweets contains trending terms.\n * Sets corresponding bits and fields to TweetTextFeatures.\n */\npublic class TweetTrendsExtractor {\n\n  // The amount of time before filling the trends cache for the first time.\n  private static final long INIT_TRENDS_CACHE_DELAY = 0;\n\n  private static final Logger LOG = LoggerFactory.getLogger(TweetTrendsExtractor.class.getName());\n\n  private static final int LOGGING_INTERVAL = 100000;\n\n  // Singleton trends data service. This is the default service used unless a different\n  // instance is injected in the constructor.\n  private static volatile TrendsThriftDataServiceManager trendsDataServiceSingleton;\n\n  // trends cache used for extracting trends from tweets\n  private static volatile ImmutableMap<PenguinVersion, NGramCache> trendsCaches;\n\n  private static synchronized void initTrendsDataServiceInstance(\n      ServiceIdentifier serviceIdentifier,\n      List<PenguinVersion> supportedPenguinVersions) {\n    if (trendsDataServiceSingleton == null) {\n      TweetProcessingConfig.init();\n      if (trendsCaches == null) {\n        ImmutableMap.Builder<PenguinVersion, NGramCache> trendsCachesBuilder =\n            ImmutableMap.builder();\n        for (PenguinVersion penguinVersion : supportedPenguinVersions) {\n          NGramCache cache = NGramCache.builder()\n              .maxCacheSize(\n                  TweetProcessingConfig.getInt(\"trends_extractor_num_trends_to_cache\", 5000))\n              .penguinVersion(penguinVersion)\n              .build();\n          trendsCachesBuilder.put(penguinVersion, cache);\n        }\n        trendsCaches = trendsCachesBuilder.build();\n      }\n      long rawTimeout = TweetProcessingConfig.getLong(\"trends_extractor_timeout_msec\", 200);\n      long rawInterval =\n          TweetProcessingConfig.getLong(\"trends_extractor_reload_interval_sec\", 600L);\n      trendsDataServiceSingleton =\n          TrendsThriftDataServiceManager.newInstance(\n              serviceIdentifier,\n              TweetProcessingConfig.getInt(\"trends_extractor_retry\", 2),\n              Duration.apply(rawTimeout, TimeUnit.MILLISECONDS),\n              Duration.apply(INIT_TRENDS_CACHE_DELAY, TimeUnit.SECONDS),\n              Duration.apply(rawInterval, TimeUnit.SECONDS),\n              trendsCaches.values().asList()\n          );\n      trendsDataServiceSingleton.startAutoRefresh();\n      LOG.info(\"Started trend extractor.\");\n    }\n  }\n\n  public TweetTrendsExtractor(\n      ServiceIdentifier serviceIdentifier,\n      List<PenguinVersion> supportedPenguinVersions) {\n    initTrendsDataServiceInstance(serviceIdentifier, supportedPenguinVersions);\n  }\n\n  /**\n   * Extract trending terms from the specified tweet.\n   * @param tweet the specified tweet\n   */\n  public void extractTrends(TwitterMessage tweet) {\n    extractTrends(ImmutableList.of(tweet));\n  }\n\n  /**\n   * Extract trending terms from the specified list of tweets.\n   * @param tweets a list of tweets\n   */\n  public void extractTrends(Iterable<TwitterMessage> tweets) {\n    Preconditions.checkNotNull(tweets);\n\n    for (TwitterMessage tweet : tweets) {\n      for (PenguinVersion penguinVersion : tweet.getSupportedPenguinVersions()) {\n        NGramCache trendsCache = trendsCaches.get(penguinVersion);\n        if (trendsCache == null) {\n          LOG.info(\"Trends cache for Penguin version \" + penguinVersion + \" is null.\");\n          continue;\n        } else if (trendsCache.numTrendingTerms() == 0) {\n          LOG.info(\"Trends cache for Penguin version \" + penguinVersion + \" is empty.\");\n          continue;\n        }\n\n        List<String> trendsInTweet = trendsCache.extractTrendsFrom(\n            tweet.getTokenizedCharSequence(penguinVersion), tweet.getLocale());\n\n        TweetTextFeatures textFeatures = tweet.getTweetTextFeatures(penguinVersion);\n        if (textFeatures == null || textFeatures.getTokens() == null) {\n          continue;\n        }\n\n        textFeatures.getTrendingTerms().addAll(trendsInTweet);\n\n        updateTrendsStats(\n            tweet,\n            textFeatures,\n            penguinVersion,\n            RelevanceStats.exportLong(\n                \"trends_extractor_has_trends_\" + penguinVersion.name().toLowerCase()),\n            RelevanceStats.exportLong(\n                \"trends_extractor_no_trends_\" + penguinVersion.name().toLowerCase()),\n            RelevanceStats.exportLong(\n                \"trends_extractor_too_many_trends_\" + penguinVersion.name().toLowerCase()));\n      }\n    }\n  }\n\n  private void updateTrendsStats(TwitterMessage tweet,\n                                 TweetTextFeatures textFeatures,\n                                 PenguinVersion penguinVersion,\n                                 SearchCounter hasTrendsCounterToUpdate,\n                                 SearchCounter noTrendsCounterToUpdate,\n                                 SearchCounter tooManyTrendsCounterToUpdate) {\n    int numTrendingTerms = textFeatures.getTrendingTerms().size();\n    if (numTrendingTerms == 0) {\n      noTrendsCounterToUpdate.increment();\n    } else {\n      if (numTrendingTerms > 1) {\n        tooManyTrendsCounterToUpdate.increment();\n      }\n      hasTrendsCounterToUpdate.increment();\n    }\n\n    long counter = noTrendsCounterToUpdate.get();\n    if (counter % LOGGING_INTERVAL == 0) {\n      long hasTrends = hasTrendsCounterToUpdate.get();\n      long noTrends = noTrendsCounterToUpdate.get();\n      long tooManyTrends = tooManyTrendsCounterToUpdate.get();\n      double ratio = 100.0d * hasTrends / (hasTrends + noTrends + 1);\n      double tooManyTrendsRatio = 100.0d * tooManyTrends / (hasTrends + 1);\n      LOG.info(String.format(\n          \"Has trends %d, no trends %d, ratio %.2f, too many trends %.2f,\"\n              + \" sample tweet id [%d] matching terms [%s] penguin version [%s]\",\n          hasTrends, noTrends, ratio, tooManyTrendsRatio, tweet.getId(),\n          textFeatures.getTrendingTerms(), penguinVersion));\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/config/TweetProcessingConfig.java",
    "content": "package com.twitter.search.common.relevance.config;\n\nimport java.io.InputStream;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.config.ConfigFile;\n\n/**\n * Config file for relevance computation.\n */\npublic final class TweetProcessingConfig {\n  private static final Logger LOG = LoggerFactory.getLogger(TweetProcessingConfig.class);\n  private static final String SCORER_CONFIG_DIR = \"common/relevance/config\";\n  public static final String DEFAULT_CONFIG_FILE = \"relevance.yml\";\n  private static ConfigFile relevanceConfig = null;\n\n  private TweetProcessingConfig() {\n  }\n\n  /** Initializes this instance from the given config file. */\n  public static void init(String configFile) {\n    if (relevanceConfig == null) {\n      synchronized (TweetProcessingConfig.class) {\n        if (relevanceConfig == null) {\n          String file = configFile == null ? DEFAULT_CONFIG_FILE : configFile;\n          relevanceConfig = new ConfigFile(SCORER_CONFIG_DIR, file);\n        }\n      }\n    }\n  }\n\n  /** Initializes this instance from the given input stream. */\n  public static void init(InputStream inputStream, String configType) {\n    if (relevanceConfig == null) {\n      synchronized (TweetProcessingConfig.class) {\n        if (relevanceConfig == null) {\n          relevanceConfig = new ConfigFile(inputStream, configType);\n        }\n      }\n    }\n  }\n\n  /** Initializes this instance. */\n  public static void init() {\n    init(null);\n  }\n\n  /**\n   * Returns the value of the given property as a double value.\n   *\n   * @param property The property.\n   * @param defaultValue The default value to return if the property is not present in the config.\n   */\n  public static double getDouble(String property, double defaultValue) {\n    return relevanceConfig.getDouble(property, defaultValue);\n  }\n\n  /**\n   * Returns the value of the given property as a string value.\n   *\n   * @param property The property.\n   * @param defaultValue The default value to return if the property is not present in the config.\n   */\n  public static String getString(String property, String defaultValue) {\n    return relevanceConfig.getString(property, defaultValue);\n  }\n\n  /**\n   * Returns the value of the given property as an integer value.\n   *\n   * @param property The property.\n   * @param defaultValue The default value to return if the property is not present in the config.\n   */\n  public static int getInt(String property, int defaultValue) {\n    return relevanceConfig.getInt(property, defaultValue);\n  }\n\n  /**\n   * Returns the value of the given property as a long value.\n   *\n   * @param property The property.\n   * @param defaultValue The default value to return if the property is not present in the config.\n   */\n  public static long getLong(String property, long defaultValue) {\n    return relevanceConfig.getLong(property, defaultValue);\n  }\n\n  /**\n   * Returns the value of the given property as a boolean value.\n   *\n   * @param property The property.\n   * @param defaultValue The default value to return if the property is not present in the config.\n   */\n  public static boolean getBool(String property, boolean defaultValue) {\n    return relevanceConfig.getBool(property, defaultValue);\n  }\n\n  /**\n   * Returns the value of the given property as a string.\n   *\n   * @param property The property.\n   * @throws ConfigurationException If the given property is not found in the config.\n   */\n  public static String getString(String property) {\n    try {\n      return relevanceConfig.getString(property);\n    } catch (ConfigurationException e) {\n      LOG.error(\"Fatal error: could not get config string \" + property, e);\n      throw new RuntimeException(e);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/entities/GeoObject.java",
    "content": "package com.twitter.search.common.relevance.entities;\n\nimport java.util.List;\nimport java.util.Optional;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftGeoLocationSource;\nimport com.twitter.search.common.indexing.thriftjava.ThriftGeoTags;\nimport com.twitter.tweetypie.thriftjava.GeoCoordinates;\nimport com.twitter.tweetypie.thriftjava.Place;\n\nimport geo.google.datamodel.GeoAddressAccuracy;\n\n/**\n * A GeoObject, extending a GeoCoordinate to include radius and accuracy\n */\npublic class GeoObject {\n\n  public static final int INT_FIELD_NOT_PRESENT = -1;\n  public static final double DOUBLE_FIELD_NOT_PRESENT = -1.0;\n\n  private double latitude = DOUBLE_FIELD_NOT_PRESENT;\n  private double longitude = DOUBLE_FIELD_NOT_PRESENT;\n  private double radius = DOUBLE_FIELD_NOT_PRESENT;\n\n  private final ThriftGeoLocationSource source;\n\n  // Valid range is 0-9. With 0 being unknown and 9 being most accurate.\n  // If this GeoObject is valid, this should be set to INT_FIELD_NOT_PRESENT\n  private int accuracy = 0;\n\n  /** Creates a new GeoObject instance. */\n  public GeoObject(double lat, double lon, ThriftGeoLocationSource source) {\n    this(lat, lon, 0, source);\n  }\n\n  /** Creates a new GeoObject instance. */\n  public GeoObject(double lat, double lon, int acc, ThriftGeoLocationSource source) {\n    latitude = lat;\n    longitude = lon;\n    accuracy = acc;\n    this.source = source;\n  }\n\n  /** Creates a new GeoObject instance. */\n  public GeoObject(ThriftGeoLocationSource source) {\n    this.source = source;\n  }\n\n  /**\n   * Tries to create a {@code GeoObject} instance from a given TweetyPie {@code Place} struct based\n   * on its bounding box coordinates.\n   *\n   * @param place\n   * @return {@code Optional} instance with {@code GeoObject} if bounding box coordinates are\n   *         available, or an empty {@code Optional}.\n   */\n  public static Optional<GeoObject> fromPlace(Place place) {\n    // Can't use place.centroid: from the sample of data, centroid seems to always be null\n    // (as of May 17 2016).\n    if (place.isSetBounding_box() && place.getBounding_boxSize() > 0) {\n      int pointsCount = place.getBounding_boxSize();\n\n      if (pointsCount == 1) {\n        GeoCoordinates point = place.getBounding_box().get(0);\n        return Optional.of(createForIngester(point.getLatitude(), point.getLongitude()));\n      } else {\n        double sumLatitude = 0.0;\n        double sumLongitude = 0.0;\n\n        List<GeoCoordinates> box = place.getBounding_box();\n\n        // Drop the last point if it's the same as the first point.\n        // The same logic is present in several other classes dealing with places.\n        // See e.g. birdherd/src/main/scala/com/twitter/birdherd/tweetypie/TweetyPiePlace.scala\n        if (box.get(pointsCount - 1).equals(box.get(0))) {\n          pointsCount--;\n        }\n\n        for (int i = 0; i < pointsCount; i++) {\n          GeoCoordinates coords = box.get(i);\n          sumLatitude += coords.getLatitude();\n          sumLongitude += coords.getLongitude();\n        }\n\n        double averageLatitude = sumLatitude / pointsCount;\n        double averageLongitude = sumLongitude / pointsCount;\n        return Optional.of(GeoObject.createForIngester(averageLatitude, averageLongitude));\n      }\n    }\n    return Optional.empty();\n  }\n\n  public void setRadius(double radius) {\n    this.radius = radius;\n  }\n\n  public Double getRadius() {\n    return radius;\n  }\n\n  public void setLatitude(double latitude) {\n    this.latitude = latitude;\n  }\n\n  public Double getLatitude() {\n    return latitude;\n  }\n\n  public void setLongitude(double longitude) {\n    this.longitude = longitude;\n  }\n\n  public Double getLongitude() {\n    return longitude;\n  }\n\n  public int getAccuracy() {\n    return accuracy;\n  }\n\n  public void setAccuracy(int accuracy) {\n    this.accuracy = accuracy;\n  }\n\n  public ThriftGeoLocationSource getSource() {\n    return source;\n  }\n\n  /** Convers this GeoObject instance to a ThriftGeoTags instance. */\n  public ThriftGeoTags toThriftGeoTags(long twitterMessageId) {\n    ThriftGeoTags geoTags = new ThriftGeoTags();\n    geoTags.setStatusId(twitterMessageId);\n    geoTags.setLatitude(getLatitude());\n    geoTags.setLongitude(getLongitude());\n    geoTags.setAccuracy(accuracy);\n    geoTags.setGeoLocationSource(source);\n    return geoTags;\n  }\n\n  private static final double COORDS_EQUALITY_THRESHOLD = 1e-7;\n\n  /**\n   * Performs an approximate comparison between the two GeoObject instances.\n   *\n   * @deprecated This code is not performant and should not be used in\n   * production code. Use only for tests. See SEARCH-5148.\n   */\n  @Deprecated\n  @VisibleForTesting\n  public static boolean approxEquals(GeoObject a, GeoObject b) {\n    if (a == null && b == null) {\n      return true;\n    }\n    if ((a == null && b != null) || (a != null && b == null)) {\n      return false;\n    }\n\n    if (a.accuracy != b.accuracy) {\n      return false;\n    }\n    if (Math.abs(a.latitude - b.latitude) > COORDS_EQUALITY_THRESHOLD) {\n      return false;\n    }\n    if (Math.abs(a.longitude - b.longitude) > COORDS_EQUALITY_THRESHOLD) {\n      return false;\n    }\n    if (Double.compare(a.radius, b.radius) != 0) {\n      return false;\n    }\n    if (a.source != b.source) {\n      return false;\n    }\n\n    return true;\n  }\n\n  @Override\n  public String toString() {\n    return \"GeoObject{\"\n        + \"latitude=\" + latitude\n        + \", longitude=\" + longitude\n        + \", radius=\" + radius\n        + \", source=\" + source\n        + \", accuracy=\" + accuracy\n        + '}';\n  }\n\n  /**\n   * Convenience factory method for ingester purposes.\n   */\n  public static GeoObject createForIngester(double latitude, double longitude) {\n    return new GeoObject(\n        latitude,\n        longitude,\n        // store with highest level of accuracy: POINT_LEVEL\n        GeoAddressAccuracy.POINT_LEVEL.getCode(),\n        ThriftGeoLocationSource.GEOTAG);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/entities/PotentialLocationObject.java",
    "content": "package com.twitter.search.common.relevance.entities;\n\nimport java.util.Locale;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.commons.lang.StringUtils;\n\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.search.common.indexing.thriftjava.PotentialLocation;\nimport com.twitter.search.common.util.text.LanguageIdentifierHelper;\nimport com.twitter.search.common.util.text.NormalizerHelper;\nimport com.twitter.search.common.util.text.TokenizerHelper;\n\n/**\n * An immutable tuple to wrap a country code, region and locality. Based on the PotentialLocation\n * struct in status.thrift.\n */\npublic class PotentialLocationObject {\n  private final String countryCode;\n  private final String region;\n  private final String locality;\n\n  /**\n   * Creates a new PotentialLocationObject instance.\n   *\n   * @param countryCode The country code.\n   * @param region The region.\n   * @param locality The locality.\n   */\n  public PotentialLocationObject(String countryCode, String region, String locality) {\n    this.countryCode = countryCode;\n    this.region = region;\n    this.locality = locality;\n  }\n\n  public String getCountryCode() {\n    return countryCode;\n  }\n\n  public String getRegion() {\n    return region;\n  }\n\n  public String getLocality() {\n    return locality;\n  }\n\n  /**\n   * Converts this PotentialLocationObject instance to a PotentialLocation thrift struct.\n   *\n   * @param penguinVersion The penguin version to use for normalization and tokenization.\n   */\n  public PotentialLocation toThriftPotentialLocation(PenguinVersion penguinVersion) {\n    Preconditions.checkNotNull(penguinVersion);\n\n    String normalizedCountryCode = null;\n    if (countryCode != null) {\n      Locale countryCodeLocale = LanguageIdentifierHelper.identifyLanguage(countryCode);\n      normalizedCountryCode =\n          NormalizerHelper.normalize(countryCode, countryCodeLocale, penguinVersion);\n    }\n\n    String tokenizedRegion = null;\n    if (region != null) {\n      Locale regionLocale = LanguageIdentifierHelper.identifyLanguage(region);\n      String normalizedRegion = NormalizerHelper.normalize(region, regionLocale, penguinVersion);\n      tokenizedRegion = StringUtils.join(\n          TokenizerHelper.tokenizeQuery(normalizedRegion, regionLocale, penguinVersion), \" \");\n    }\n\n    String tokenizedLocality = null;\n    if (locality != null) {\n      Locale localityLocale = LanguageIdentifierHelper.identifyLanguage(locality);\n      String normalizedLocality =\n          NormalizerHelper.normalize(locality, localityLocale, penguinVersion);\n      tokenizedLocality =\n          StringUtils.join(TokenizerHelper.tokenizeQuery(\n                               normalizedLocality, localityLocale, penguinVersion), \" \");\n    }\n\n    return new PotentialLocation()\n        .setCountryCode(normalizedCountryCode)\n        .setRegion(tokenizedRegion)\n        .setLocality(tokenizedLocality);\n  }\n\n  @Override\n  public int hashCode() {\n    return ((countryCode == null) ? 0 : countryCode.hashCode())\n        + 13 * ((region == null) ? 0 : region.hashCode())\n        + 19 * ((locality == null) ? 0 : locality.hashCode());\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof PotentialLocationObject)) {\n      return false;\n    }\n\n    PotentialLocationObject entry = (PotentialLocationObject) obj;\n    return (countryCode == null\n            ? entry.countryCode == null\n            : countryCode.equals(entry.countryCode))\n        && (region == null\n            ? entry.region == null\n            : region.equals(entry.region))\n        && (locality == null\n            ? entry.locality == null\n            : locality.equals(entry.locality));\n  }\n\n  @Override\n  public String toString() {\n    return new StringBuilder(\"PotentialLocationObject {\")\n        .append(\"countryCode=\").append(countryCode)\n        .append(\", region=\").append(region)\n        .append(\", locality=\").append(locality)\n        .append(\"}\")\n        .toString();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/entities/TwitterMessage.java",
    "content": "package com.twitter.search.common.relevance.entities;\n\nimport java.text.DateFormat;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.Date;\nimport java.util.HashSet;\nimport java.util.LinkedHashMap;\nimport java.util.List;\nimport java.util.Locale;\nimport java.util.Map;\nimport java.util.Optional;\nimport java.util.Set;\nimport javax.annotation.Nonnull;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ComparisonChain;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\nimport com.google.common.collect.Sets;\n\nimport org.apache.commons.lang.StringUtils;\nimport org.apache.commons.lang3.builder.EqualsBuilder;\nimport org.apache.commons.lang3.builder.HashCodeBuilder;\nimport org.apache.commons.lang3.builder.ToStringBuilder;\nimport org.apache.lucene.analysis.TokenStream;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.text.language.LocaleUtil;\nimport com.twitter.common.text.pipeline.TwitterLanguageIdentifier;\nimport com.twitter.common.text.token.TokenizedCharSequence;\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.cuad.ner.plain.thriftjava.NamedEntity;\nimport com.twitter.search.common.indexing.thriftjava.ThriftExpandedUrl;\nimport com.twitter.search.common.relevance.features.TweetFeatures;\nimport com.twitter.search.common.relevance.features.TweetTextFeatures;\nimport com.twitter.search.common.relevance.features.TweetTextQuality;\nimport com.twitter.search.common.relevance.features.TweetUserFeatures;\nimport com.twitter.search.common.util.text.NormalizerHelper;\nimport com.twitter.service.spiderduck.gen.MediaTypes;\nimport com.twitter.tweetypie.thriftjava.ComposerSource;\nimport com.twitter.util.TwitterDateFormat;\n\n/**\n * A representation of tweets used as an intermediate object during ingestion. As we proceed\n * in ingestion, we fill this object with data. We then convert it to ThriftVersionedEvents (which\n * itself represents a single tweet too, in different penguin versions potentially).\n */\npublic class TwitterMessage {\n  private static final Logger LOG = LoggerFactory.getLogger(TwitterMessage.class);\n\n  public static class EscherbirdAnnotation implements Comparable<EscherbirdAnnotation> {\n    public final long groupId;\n    public final long domainId;\n    public final long entityId;\n\n    public EscherbirdAnnotation(long groupId, long domainId, long entityId) {\n      this.groupId = groupId;\n      this.domainId = domainId;\n      this.entityId = entityId;\n    }\n\n    @Override\n    public boolean equals(Object o2) {\n      if (o2 instanceof EscherbirdAnnotation) {\n        EscherbirdAnnotation a2 = (EscherbirdAnnotation) o2;\n        return groupId == a2.groupId && domainId == a2.domainId && entityId == a2.entityId;\n      }\n      return false;\n    }\n\n    @Override\n    public int hashCode() {\n      return new HashCodeBuilder()\n          .append(groupId)\n          .append(domainId)\n          .append(entityId)\n          .toHashCode();\n    }\n\n    @Override\n    public int compareTo(EscherbirdAnnotation o) {\n      return ComparisonChain.start()\n          .compare(this.groupId, o.groupId)\n          .compare(this.domainId, o.domainId)\n          .compare(this.entityId, o.entityId)\n          .result();\n    }\n  }\n\n  private final List<EscherbirdAnnotation> escherbirdAnnotations = Lists.newArrayList();\n\n  // tweet features for multiple penguin versions\n  private static class VersionedTweetFeatures {\n    // TweetFeatures populated by relevance classifiers, structure defined\n    // in src/main/thrift/classifier.thrift.\n    private TweetFeatures tweetFeatures = new TweetFeatures();\n    private TokenizedCharSequence tokenizedCharSequence = null;\n    private Set<String> normalizedHashtags = Sets.newHashSet();\n\n    public TweetFeatures getTweetFeatures() {\n      return this.tweetFeatures;\n    }\n\n    public void setTweetFeatures(final TweetFeatures tweetFeatures) {\n      this.tweetFeatures = tweetFeatures;\n    }\n\n    public TweetTextQuality getTweetTextQuality() {\n      return this.tweetFeatures.getTweetTextQuality();\n    }\n\n    public TweetTextFeatures getTweetTextFeatures() {\n      return this.tweetFeatures.getTweetTextFeatures();\n    }\n\n    public TweetUserFeatures getTweetUserFeatures() {\n      return this.tweetFeatures.getTweetUserFeatures();\n    }\n\n    public TokenizedCharSequence getTokenizedCharSequence() {\n      return this.tokenizedCharSequence;\n    }\n\n    public void setTokenizedCharSequence(TokenizedCharSequence sequence) {\n      this.tokenizedCharSequence = sequence;\n    }\n\n    public Set<String> getNormalizedHashtags() {\n      return this.normalizedHashtags;\n    }\n\n    public void addNormalizedHashtags(String normalizedHashtag) {\n      this.normalizedHashtags.add(normalizedHashtag);\n    }\n  }\n\n  public static final int INT_FIELD_NOT_PRESENT = -1;\n  public static final long LONG_FIELD_NOT_PRESENT = -1;\n  public static final double DOUBLE_FIELD_NOT_PRESENT = -1;\n  public static final int MAX_USER_REPUTATION = 100;\n\n  private final long tweetId;\n\n  private String text;\n\n  private Date date;\n  @Nonnull\n  private Optional<TwitterMessageUser> optionalFromUser = Optional.empty();\n  @Nonnull\n  private Optional<TwitterMessageUser> optionalToUser = Optional.empty();\n  private Locale locale = null;\n  private Locale linkLocale = null;\n\n  // Original source text.\n  private String origSource;\n  // Source with HTML tags removed and truncated.\n  private String strippedSource;\n\n  // Original location text.\n  private String origLocation;\n\n  // Location truncated for mysql field-width reasons (see TwitterMessageUtil.java).\n  private String truncatedNormalizedLocation;\n\n  // User's country\n  private String fromUserLocCountry;\n\n  private Integer followersCount = INT_FIELD_NOT_PRESENT;\n  private boolean deleted = false;\n\n  // Fields extracted from entities (in the JSON object)\n  private List<TwitterMessageUser> mentions = new ArrayList<>();\n  private Set<String> hashtags = Sets.newHashSet();\n  // Lat/lon and region accuracy tuples extracted from tweet text, or null.\n  private GeoObject geoLocation = null;\n  private boolean uncodeableLocation = false;\n  // This is set if the tweet is geotagged. (i.e. \"geo\" or \"coordinate\" section is present\n  // in the json)\n  // This field has only a getter but no setter --- it is filled in when the json is parsed.\n  private GeoObject geoTaggedLocation = null;\n\n  private double userReputation = DOUBLE_FIELD_NOT_PRESENT;\n  private boolean geocodeRequired = false;\n  private boolean sensitiveContent = false;\n  private boolean userProtected;\n  private boolean userVerified;\n  private boolean userBlueVerified;\n  private TwitterRetweetMessage retweetMessage;\n  private TwitterQuotedMessage quotedMessage;\n  private List<String> places;\n  // maps from original url (the t.co url) to ThriftExpandedUrl, which contains the\n  // expanded url and the spiderduck response (canoicalLastHopUrl and mediatype)\n  private final Map<String, ThriftExpandedUrl> expandedUrls;\n  // maps the photo status id to the media url\n  private Map<Long, String> photoUrls;\n  private Optional<Long> inReplyToStatusId = Optional.empty();\n  private Optional<Long> directedAtUserId = Optional.empty();\n\n  private long conversationId = -1;\n\n  // True if tweet is nullcasted.\n  private boolean nullcast = false;\n\n  // True if tweet is a self-threaded tweet\n  private boolean selfThread = false;\n\n  // If the tweet is a part of an exclusive conversation, the author who started\n  // that conversation.\n  private Optional<Long> exclusiveConversationAuthorId = Optional.empty();\n\n  // tweet features map for multiple versions of penguin\n  private Map<PenguinVersion, VersionedTweetFeatures> versionedTweetFeaturesMap;\n\n  // Engagments count: favorites, retweets and replies\n  private int numFavorites = 0;\n  private int numRetweets = 0;\n  private int numReplies = 0;\n\n  // Card information\n  private String cardName;\n  private String cardDomain;\n  private String cardTitle;\n  private String cardDescription;\n  private String cardLang;\n  private String cardUrl;\n\n  private String placeId;\n  private String placeFullName;\n  private String placeCountryCode;\n\n  private Set<NamedEntity> namedEntities = Sets.newHashSet();\n\n  // Spaces data\n  private Set<String> spaceIds = Sets.newHashSet();\n  private Set<TwitterMessageUser> spaceAdmins = Sets.newHashSet();\n  private String spaceTitle;\n\n  private Optional<ComposerSource> composerSource = Optional.empty();\n\n  private final List<PotentialLocationObject> potentialLocations = Lists.newArrayList();\n\n  // one or two penguin versions supported by this system\n  private final List<PenguinVersion> supportedPenguinVersions;\n\n  public TwitterMessage(Long tweetId, List<PenguinVersion> supportedPenguinVersions) {\n    this.tweetId = tweetId;\n    this.places = new ArrayList<>();\n    this.expandedUrls = new LinkedHashMap<>();\n    // make sure we support at least one, but no more than two versions of penguin\n    this.supportedPenguinVersions = supportedPenguinVersions;\n    this.versionedTweetFeaturesMap = getVersionedTweetFeaturesMap();\n    Preconditions.checkArgument(this.supportedPenguinVersions.size() <= 2\n        && this.supportedPenguinVersions.size() > 0);\n  }\n\n  /**\n   * Replace to-user with in-reply-to user if needed.\n   */\n  public void replaceToUserWithInReplyToUserIfNeeded(\n      String inReplyToScreenName, long inReplyToUserId) {\n    if (shouldUseReplyUserAsToUser(optionalToUser, inReplyToUserId)) {\n      TwitterMessageUser replyUser =\n          TwitterMessageUser.createWithNamesAndId(inReplyToScreenName, \"\", inReplyToUserId);\n      optionalToUser = Optional.of(replyUser);\n    }\n  }\n\n  // To-user could have been inferred from the mention at the position 0.\n  // But if there is an explicit in-reply-to user, we might need to use it as to-user instead.\n  private static boolean shouldUseReplyUserAsToUser(\n      Optional<TwitterMessageUser> currentToUser,\n      long inReplyToUserId) {\n    if (!currentToUser.isPresent()) {\n      // There is no mention in the tweet that qualifies as to-user.\n      return true;\n    }\n\n    // We already have a mention in the tweet that qualifies as to-user.\n    TwitterMessageUser toUser = currentToUser.get();\n    if (!toUser.getId().isPresent()) {\n      // The to-user from the mention is a stub.\n      return true;\n    }\n\n    long toUserId = toUser.getId().get();\n    if (toUserId != inReplyToUserId) {\n      // The to-user from the mention is different that the in-reply-to user,\n      // use in-reply-to user instead.\n      return true;\n    }\n\n    return false;\n  }\n\n  public double getUserReputation() {\n    return userReputation;\n  }\n\n  /**\n   * Sets the user reputation.\n   */\n  public TwitterMessage setUserReputation(double newUserReputation) {\n    if (newUserReputation > MAX_USER_REPUTATION) {\n      LOG.warn(\"Out of bounds user reputation {} for status id {}\", newUserReputation, tweetId);\n      this.userReputation = (float) MAX_USER_REPUTATION;\n    } else {\n      this.userReputation = newUserReputation;\n    }\n    return this;\n  }\n\n  public String getText() {\n    return text;\n  }\n\n  public Optional<TwitterMessageUser> getOptionalToUser() {\n    return optionalToUser;\n  }\n\n  public void setOptionalToUser(Optional<TwitterMessageUser> optionalToUser) {\n    this.optionalToUser = optionalToUser;\n  }\n\n  public void setText(String text) {\n    this.text = text;\n  }\n\n  public Date getDate() {\n    return date;\n  }\n\n  public void setDate(Date date) {\n    this.date = date;\n  }\n\n  public void setFromUser(@Nonnull TwitterMessageUser fromUser) {\n    Preconditions.checkNotNull(fromUser, \"Don't set a null fromUser\");\n    optionalFromUser = Optional.of(fromUser);\n  }\n\n  public Optional<String> getFromUserScreenName() {\n    return optionalFromUser.isPresent()\n        ? optionalFromUser.get().getScreenName()\n        : Optional.empty();\n  }\n\n  /**\n   * Sets the fromUserScreenName.\n   */\n  public void setFromUserScreenName(@Nonnull String fromUserScreenName) {\n    TwitterMessageUser newFromUser = optionalFromUser.isPresent()\n        ? optionalFromUser.get().copyWithScreenName(fromUserScreenName)\n        : TwitterMessageUser.createWithScreenName(fromUserScreenName);\n\n    optionalFromUser = Optional.of(newFromUser);\n  }\n\n  public Optional<TokenStream> getTokenizedFromUserScreenName() {\n    return optionalFromUser.flatMap(TwitterMessageUser::getTokenizedScreenName);\n  }\n\n  public Optional<String> getFromUserDisplayName() {\n    return optionalFromUser.flatMap(TwitterMessageUser::getDisplayName);\n  }\n\n  /**\n   * Sets the fromUserDisplayName.\n   */\n  public void setFromUserDisplayName(@Nonnull String fromUserDisplayName) {\n    TwitterMessageUser newFromUser = optionalFromUser.isPresent()\n        ? optionalFromUser.get().copyWithDisplayName(fromUserDisplayName)\n        : TwitterMessageUser.createWithDisplayName(fromUserDisplayName);\n\n    optionalFromUser = Optional.of(newFromUser);\n  }\n\n  public Optional<Long> getFromUserTwitterId() {\n    return optionalFromUser.flatMap(TwitterMessageUser::getId);\n  }\n\n  /**\n   * Sets the fromUserId.\n   */\n  public void setFromUserId(long fromUserId) {\n    TwitterMessageUser newFromUser = optionalFromUser.isPresent()\n        ? optionalFromUser.get().copyWithId(fromUserId)\n        : TwitterMessageUser.createWithId(fromUserId);\n\n    optionalFromUser = Optional.of(newFromUser);\n  }\n\n  public long getConversationId() {\n    return conversationId;\n  }\n\n  public void setConversationId(long conversationId) {\n    this.conversationId = conversationId;\n  }\n\n  public boolean isUserProtected() {\n    return this.userProtected;\n  }\n\n  public void setUserProtected(boolean userProtected) {\n    this.userProtected = userProtected;\n  }\n\n  public boolean isUserVerified() {\n    return this.userVerified;\n  }\n\n  public void setUserVerified(boolean userVerified) {\n    this.userVerified = userVerified;\n  }\n\n  public boolean isUserBlueVerified() {\n    return this.userBlueVerified;\n  }\n\n  public void setUserBlueVerified(boolean userBlueVerified) {\n    this.userBlueVerified = userBlueVerified;\n  }\n\n  public void setIsSensitiveContent(boolean isSensitiveContent) {\n    this.sensitiveContent = isSensitiveContent;\n  }\n\n  public boolean isSensitiveContent() {\n    return this.sensitiveContent;\n  }\n\n  public Optional<TwitterMessageUser> getToUserObject() {\n    return optionalToUser;\n  }\n\n  public void setToUserObject(@Nonnull TwitterMessageUser user) {\n    Preconditions.checkNotNull(user, \"Don't set a null to-user\");\n    optionalToUser = Optional.of(user);\n  }\n\n  public Optional<Long> getToUserTwitterId() {\n    return optionalToUser.flatMap(TwitterMessageUser::getId);\n  }\n\n  /**\n   * Sets toUserId.\n   */\n  public void setToUserTwitterId(long toUserId) {\n    TwitterMessageUser newToUser = optionalToUser.isPresent()\n        ? optionalToUser.get().copyWithId(toUserId)\n        : TwitterMessageUser.createWithId(toUserId);\n\n    optionalToUser = Optional.of(newToUser);\n  }\n\n  public Optional<String> getToUserLowercasedScreenName() {\n    return optionalToUser.flatMap(TwitterMessageUser::getScreenName).map(String::toLowerCase);\n  }\n\n  public Optional<String> getToUserScreenName() {\n    return optionalToUser.flatMap(TwitterMessageUser::getScreenName);\n  }\n\n  /**\n   * Sets toUserScreenName.\n   */\n  public void setToUserScreenName(@Nonnull String screenName) {\n    Preconditions.checkNotNull(screenName, \"Don't set a null to-user screenname\");\n\n    TwitterMessageUser newToUser = optionalToUser.isPresent()\n        ? optionalToUser.get().copyWithScreenName(screenName)\n        : TwitterMessageUser.createWithScreenName(screenName);\n\n    optionalToUser = Optional.of(newToUser);\n  }\n\n  // to use from TweetEventParseHelper\n  public void setDirectedAtUserId(Optional<Long> directedAtUserId) {\n    this.directedAtUserId = directedAtUserId;\n  }\n\n  @VisibleForTesting\n  public Optional<Long> getDirectedAtUserId() {\n    return directedAtUserId;\n  }\n\n  /**\n   * Returns the referenceAuthorId.\n   */\n  public Optional<Long> getReferenceAuthorId() {\n    // The semantics of reference-author-id:\n    // - if the tweet is a retweet, it should be the user id of the author of the original tweet\n    // - else, if the tweet is directed at a user, it should be the id of the user it's directed at.\n    // - else, if the tweet is a reply in a root self-thread, directed-at is not set, so it's\n    //   the id of the user who started the self-thread.\n    //\n    // For definitive info on replies and directed-at, take a look at go/replies. To view these\n    // for a certain tweet, use http://go/t.\n    //\n    // Note that if directed-at is set, reply is always set.\n    // If reply is set, directed-at is not necessarily set.\n    if (isRetweet() && retweetMessage.hasSharedUserTwitterId()) {\n      long retweetedUserId = retweetMessage.getSharedUserTwitterId();\n      return Optional.of(retweetedUserId);\n    } else if (directedAtUserId.isPresent()) {\n      // Why not replace directedAtUserId with reply and make this function depend\n      // on the \"reply\" field of TweetCoreData?\n      // Well, verified by counters, it seems for ~1% of tweets, which contain both directed-at\n      // and reply, directed-at-user is different than the reply-to-user id. This happens in the\n      // following case:\n      //\n      //       author / reply-to / directed-at\n      //  T1   A        -          -\n      //  T2   B        A          A\n      //  T3   B        B          A\n      //\n      //  T2 is a reply to T1, T3 is a reply to T2.\n      //\n      // It's up to us to decide who this tweet is \"referencing\", but with the current code,\n      // we choose that T3 is referencing user A.\n      return directedAtUserId;\n    } else {\n      // This is the case of a root self-thread reply. directed-at is not set.\n      Optional<Long> fromUserId = this.getFromUserTwitterId();\n      Optional<Long> toUserId = this.getToUserTwitterId();\n\n      if (fromUserId.isPresent() && fromUserId.equals(toUserId)) {\n        return fromUserId;\n      }\n    }\n    return Optional.empty();\n  }\n\n  public void setNumFavorites(int numFavorites) {\n    this.numFavorites = numFavorites;\n  }\n\n  public void setNumRetweets(int numRetweets) {\n    this.numRetweets = numRetweets;\n  }\n\n  public void setNumReplies(int numRepliess) {\n    this.numReplies = numRepliess;\n  }\n\n  public void addEscherbirdAnnotation(EscherbirdAnnotation annotation) {\n    escherbirdAnnotations.add(annotation);\n  }\n\n  public List<EscherbirdAnnotation> getEscherbirdAnnotations() {\n    return escherbirdAnnotations;\n  }\n\n  public List<PotentialLocationObject> getPotentialLocations() {\n    return potentialLocations;\n  }\n\n  public void setPotentialLocations(Collection<PotentialLocationObject> potentialLocations) {\n    this.potentialLocations.clear();\n    this.potentialLocations.addAll(potentialLocations);\n  }\n\n  @Override\n  public String toString() {\n    return ToStringBuilder.reflectionToString(this);\n  }\n\n  // Tweet language related getters and setters.\n\n  /**\n   * Returns the locale.\n   * <p>\n   * Note the getLocale() will never return null, this is for the convenience of text related\n   * processing in the ingester. If you want the real locale, you need to check isSetLocale()\n   * first to see if we really have any information about the locale of this tweet.\n   */\n  public Locale getLocale() {\n    if (locale == null) {\n      return TwitterLanguageIdentifier.UNKNOWN;\n    } else {\n      return locale;\n    }\n  }\n\n  public void setLocale(Locale locale) {\n    this.locale = locale;\n  }\n\n  /**\n   * Determines if the locate is set.\n   */\n  public boolean isSetLocale() {\n    return locale != null;\n  }\n\n  /**\n   * Returns the language of the locale. E.g. zh\n   */\n  public String getLanguage() {\n    if (isSetLocale()) {\n      return getLocale().getLanguage();\n    } else {\n      return null;\n    }\n  }\n\n  /**\n   * Returns the IETF BCP 47 Language Tag of the locale. E.g. zh-CN\n   */\n  public String getBCP47LanguageTag() {\n    if (isSetLocale()) {\n      return getLocale().toLanguageTag();\n    } else {\n      return null;\n    }\n  }\n\n  public void setLanguage(String language) {\n    if (language != null) {\n      locale = LocaleUtil.getLocaleOf(language);\n    }\n  }\n\n  // Tweet link language related getters and setters.\n  public Locale getLinkLocale() {\n    return linkLocale;\n  }\n\n  public void setLinkLocale(Locale linkLocale) {\n    this.linkLocale = linkLocale;\n  }\n\n  /**\n   * Returns the language of the link locale.\n   */\n  public String getLinkLanguage() {\n    if (this.linkLocale == null) {\n      return null;\n    } else {\n      return this.linkLocale.getLanguage();\n    }\n  }\n\n  public String getOrigSource() {\n    return origSource;\n  }\n\n  public void setOrigSource(String origSource) {\n    this.origSource = origSource;\n  }\n\n  public String getStrippedSource() {\n    return strippedSource;\n  }\n\n  public void setStrippedSource(String strippedSource) {\n    this.strippedSource = strippedSource;\n  }\n\n  public String getOrigLocation() {\n    return origLocation;\n  }\n\n  public String getLocation() {\n    return truncatedNormalizedLocation;\n  }\n\n  public void setOrigLocation(String origLocation) {\n    this.origLocation = origLocation;\n  }\n\n  public void setTruncatedNormalizedLocation(String truncatedNormalizedLocation) {\n    this.truncatedNormalizedLocation = truncatedNormalizedLocation;\n  }\n\n  public boolean hasFromUserLocCountry() {\n    return fromUserLocCountry != null;\n  }\n\n  public String getFromUserLocCountry() {\n    return fromUserLocCountry;\n  }\n\n  public void setFromUserLocCountry(String fromUserLocCountry) {\n    this.fromUserLocCountry = fromUserLocCountry;\n  }\n\n  public String getTruncatedNormalizedLocation() {\n    return truncatedNormalizedLocation;\n  }\n\n  public Integer getFollowersCount() {\n    return followersCount;\n  }\n\n  public void setFollowersCount(Integer followersCount) {\n    this.followersCount = followersCount;\n  }\n\n  public boolean hasFollowersCount() {\n    return followersCount != INT_FIELD_NOT_PRESENT;\n  }\n\n  public boolean isDeleted() {\n    return deleted;\n  }\n\n  public void setDeleted(boolean deleted) {\n    this.deleted = deleted;\n  }\n\n  public boolean hasCard() {\n    return !StringUtils.isBlank(getCardName());\n  }\n\n  @Override\n  public int hashCode() {\n    return ((Long) getId()).hashCode();\n  }\n\n  /**\n   * Parses the given date using the TwitterDateFormat.\n   */\n  public static Date parseDate(String date) {\n    DateFormat parser = TwitterDateFormat.apply(\"EEE MMM d HH:mm:ss Z yyyy\");\n    try {\n      return parser.parse(date);\n    } catch (Exception e) {\n      return null;\n    }\n  }\n\n  public boolean hasGeoLocation() {\n    return geoLocation != null;\n  }\n\n  public void setGeoLocation(GeoObject location) {\n    this.geoLocation = location;\n  }\n\n  public GeoObject getGeoLocation() {\n    return geoLocation;\n  }\n\n  public String getPlaceId() {\n    return placeId;\n  }\n\n  public void setPlaceId(String placeId) {\n    this.placeId = placeId;\n  }\n\n  public String getPlaceFullName() {\n    return placeFullName;\n  }\n\n  public void setPlaceFullName(String placeFullName) {\n    this.placeFullName = placeFullName;\n  }\n\n  public String getPlaceCountryCode() {\n    return placeCountryCode;\n  }\n\n  public void setPlaceCountryCode(String placeCountryCode) {\n    this.placeCountryCode = placeCountryCode;\n  }\n\n  public void setGeoTaggedLocation(GeoObject geoTaggedLocation) {\n    this.geoTaggedLocation = geoTaggedLocation;\n  }\n\n  public GeoObject getGeoTaggedLocation() {\n    return geoTaggedLocation;\n  }\n\n  public void setLatLon(double latitude, double longitude) {\n    geoLocation = new GeoObject(latitude, longitude, null);\n  }\n\n  public Double getLatitude() {\n    return hasGeoLocation() ? geoLocation.getLatitude() : null;\n  }\n\n  public Double getLongitude() {\n    return hasGeoLocation() ? geoLocation.getLongitude() : null;\n  }\n\n  public boolean isUncodeableLocation() {\n    return uncodeableLocation;\n  }\n\n  public void setUncodeableLocation() {\n    uncodeableLocation = true;\n  }\n\n  public void setGeocodeRequired() {\n    this.geocodeRequired = true;\n  }\n\n  public boolean isGeocodeRequired() {\n    return geocodeRequired;\n  }\n\n  public Map<Long, String> getPhotoUrls() {\n    return photoUrls;\n  }\n\n  /**\n   * Associates the given mediaUrl with the given photoStatusId.\n   */\n  public void addPhotoUrl(long photoStatusId, String mediaUrl) {\n    if (photoUrls == null) {\n      photoUrls = new LinkedHashMap<>();\n    }\n    photoUrls.putIfAbsent(photoStatusId, mediaUrl);\n  }\n\n  public Map<String, ThriftExpandedUrl> getExpandedUrlMap() {\n    return expandedUrls;\n  }\n\n  public int getExpandedUrlMapSize() {\n    return expandedUrls.size();\n  }\n\n  /**\n   * Associates the given originalUrl with the given expanderUrl.\n   */\n  public void addExpandedUrl(String originalUrl, ThriftExpandedUrl expandedUrl) {\n    this.expandedUrls.put(originalUrl, expandedUrl);\n  }\n\n  /**\n   * Replaces urls with resolved ones.\n   */\n  public String getTextReplacedWithResolvedURLs() {\n    String retText = text;\n    for (Map.Entry<String, ThriftExpandedUrl> entry : expandedUrls.entrySet()) {\n      ThriftExpandedUrl urlInfo = entry.getValue();\n      String resolvedUrl;\n      String canonicalLastHopUrl = urlInfo.getCanonicalLastHopUrl();\n      String expandedUrl = urlInfo.getExpandedUrl();\n      if (canonicalLastHopUrl != null) {\n        resolvedUrl = canonicalLastHopUrl;\n        LOG.debug(\"{} has canonical last hop url set\", urlInfo);\n      } else if (expandedUrl != null) {\n        LOG.debug(\"{} has no canonical last hop url set, using expanded url instead\", urlInfo);\n        resolvedUrl = expandedUrl;\n      } else {\n        LOG.debug(\"{} has no canonical last hop url or expanded url set, skipping\", urlInfo);\n        continue;\n      }\n      retText = retText.replace(entry.getKey(), resolvedUrl);\n    }\n    return retText;\n  }\n\n  public long getId() {\n    return tweetId;\n  }\n\n  public boolean isRetweet() {\n    return retweetMessage != null;\n  }\n\n  public boolean hasQuote() {\n    return quotedMessage != null;\n  }\n\n  public boolean isReply() {\n    return getToUserScreenName().isPresent()\n        || getToUserTwitterId().isPresent()\n        || getInReplyToStatusId().isPresent();\n  }\n\n  public boolean isReplyToTweet() {\n    return getInReplyToStatusId().isPresent();\n  }\n\n  public TwitterRetweetMessage getRetweetMessage() {\n    return retweetMessage;\n  }\n\n  public void setRetweetMessage(TwitterRetweetMessage retweetMessage) {\n    this.retweetMessage = retweetMessage;\n  }\n\n  public TwitterQuotedMessage getQuotedMessage() {\n    return quotedMessage;\n  }\n\n  public void setQuotedMessage(TwitterQuotedMessage quotedMessage) {\n    this.quotedMessage = quotedMessage;\n  }\n\n  public List<String> getPlaces() {\n    return places;\n  }\n\n  public void addPlace(String place) {\n    // Places are used for earlybird serialization\n    places.add(place);\n  }\n\n  public Optional<Long> getInReplyToStatusId() {\n    return inReplyToStatusId;\n  }\n\n  public void setInReplyToStatusId(long inReplyToStatusId) {\n    Preconditions.checkArgument(inReplyToStatusId > 0, \"In-reply-to status ID should be positive\");\n    this.inReplyToStatusId = Optional.of(inReplyToStatusId);\n  }\n\n  public boolean getNullcast() {\n    return nullcast;\n  }\n\n  public void setNullcast(boolean nullcast) {\n    this.nullcast = nullcast;\n  }\n\n  public List<PenguinVersion> getSupportedPenguinVersions() {\n    return supportedPenguinVersions;\n  }\n\n  private VersionedTweetFeatures getVersionedTweetFeatures(PenguinVersion penguinVersion) {\n    VersionedTweetFeatures versionedTweetFeatures = versionedTweetFeaturesMap.get(penguinVersion);\n    return Preconditions.checkNotNull(versionedTweetFeatures);\n  }\n\n  public TweetFeatures getTweetFeatures(PenguinVersion penguinVersion) {\n    return getVersionedTweetFeatures(penguinVersion).getTweetFeatures();\n  }\n\n  @VisibleForTesting\n  // only used in Tests\n  public void setTweetFeatures(PenguinVersion penguinVersion, TweetFeatures tweetFeatures) {\n    versionedTweetFeaturesMap.get(penguinVersion).setTweetFeatures(tweetFeatures);\n  }\n\n  public int getTweetSignature(PenguinVersion penguinVersion) {\n    return getVersionedTweetFeatures(penguinVersion).getTweetTextFeatures().getSignature();\n  }\n\n  public TweetTextQuality getTweetTextQuality(PenguinVersion penguinVersion) {\n    return getVersionedTweetFeatures(penguinVersion).getTweetTextQuality();\n  }\n\n  public TweetTextFeatures getTweetTextFeatures(PenguinVersion penguinVersion) {\n    return getVersionedTweetFeatures(penguinVersion).getTweetTextFeatures();\n  }\n\n  public TweetUserFeatures getTweetUserFeatures(PenguinVersion penguinVersion) {\n    return getVersionedTweetFeatures(penguinVersion).getTweetUserFeatures();\n  }\n\n  public TokenizedCharSequence getTokenizedCharSequence(PenguinVersion penguinVersion) {\n    return getVersionedTweetFeatures(penguinVersion).getTokenizedCharSequence();\n  }\n\n  public void setTokenizedCharSequence(PenguinVersion penguinVersion,\n                                       TokenizedCharSequence sequence) {\n    getVersionedTweetFeatures(penguinVersion).setTokenizedCharSequence(sequence);\n  }\n\n  // True if the features contain multiple hash tags or multiple trends.\n  // This is intended as an anti-trend-spam measure.\n  public static boolean hasMultipleHashtagsOrTrends(TweetTextFeatures textFeatures) {\n    // Allow at most 1 trend and 2 hashtags.\n    return textFeatures.getTrendingTermsSize() > 1 || textFeatures.getHashtagsSize() > 2;\n  }\n\n  /**\n   * Returns the expanded URLs.\n   */\n  public Collection<ThriftExpandedUrl> getExpandedUrls() {\n    return expandedUrls.values();\n  }\n\n  /**\n   * Returns the canonical last hop URLs.\n   */\n  public Set<String> getCanonicalLastHopUrls() {\n    Set<String> result = new HashSet<>(expandedUrls.size());\n    for (ThriftExpandedUrl url : expandedUrls.values()) {\n      result.add(url.getCanonicalLastHopUrl());\n    }\n    return result;\n  }\n\n  public String getCardName() {\n    return cardName;\n  }\n\n  public void setCardName(String cardName) {\n    this.cardName = cardName;\n  }\n\n  public String getCardDomain() {\n    return cardDomain;\n  }\n\n  public void setCardDomain(String cardDomain) {\n    this.cardDomain = cardDomain;\n  }\n\n  public String getCardTitle() {\n    return cardTitle;\n  }\n\n  public void setCardTitle(String cardTitle) {\n    this.cardTitle = cardTitle;\n  }\n\n  public String getCardDescription() {\n    return cardDescription;\n  }\n\n  public void setCardDescription(String cardDescription) {\n    this.cardDescription = cardDescription;\n  }\n\n  public String getCardLang() {\n    return cardLang;\n  }\n\n  public void setCardLang(String cardLang) {\n    this.cardLang = cardLang;\n  }\n\n  public String getCardUrl() {\n    return cardUrl;\n  }\n\n  public void setCardUrl(String cardUrl) {\n    this.cardUrl = cardUrl;\n  }\n\n  public List<TwitterMessageUser> getMentions() {\n    return this.mentions;\n  }\n\n  public void setMentions(List<TwitterMessageUser> mentions) {\n    this.mentions = mentions;\n  }\n\n  public List<String> getLowercasedMentions() {\n    return Lists.transform(getMentions(), user -> {\n      // This condition is also checked in addUserToMentions().\n      Preconditions.checkState(user.getScreenName().isPresent(), \"Invalid mention\");\n      return user.getScreenName().get().toLowerCase();\n    });\n  }\n\n  public Set<String> getHashtags() {\n    return this.hashtags;\n  }\n\n  public Set<String> getNormalizedHashtags(PenguinVersion penguinVersion) {\n    return getVersionedTweetFeatures(penguinVersion).getNormalizedHashtags();\n  }\n\n  public void addNormalizedHashtag(String normalizedHashtag, PenguinVersion penguinVersion) {\n    getVersionedTweetFeatures(penguinVersion).addNormalizedHashtags(normalizedHashtag);\n  }\n\n  public Optional<ComposerSource> getComposerSource() {\n    return composerSource;\n  }\n\n  public void setComposerSource(ComposerSource composerSource) {\n    Preconditions.checkNotNull(composerSource, \"composerSource should not be null\");\n    this.composerSource = Optional.of(composerSource);\n  }\n\n  public boolean isSelfThread() {\n    return selfThread;\n  }\n\n  public void setSelfThread(boolean selfThread) {\n    this.selfThread = selfThread;\n  }\n\n  public boolean isExclusive() {\n    return exclusiveConversationAuthorId.isPresent();\n  }\n\n  public long getExclusiveConversationAuthorId() {\n    return exclusiveConversationAuthorId.get();\n  }\n\n  public void setExclusiveConversationAuthorId(long exclusiveConversationAuthorId) {\n    this.exclusiveConversationAuthorId = Optional.of(exclusiveConversationAuthorId);\n  }\n\n  /**\n   * Adds an expanded media url based on the given parameters.\n   */\n  public void addExpandedMediaUrl(String originalUrl,\n                                  String expandedUrl,\n                                  @Nullable MediaTypes mediaType) {\n    if (!StringUtils.isBlank(originalUrl) && !StringUtils.isBlank(expandedUrl)) {\n      ThriftExpandedUrl thriftExpandedUrl = new ThriftExpandedUrl();\n      if (mediaType != null) {\n        thriftExpandedUrl.setMediaType(mediaType);\n      }\n      thriftExpandedUrl.setOriginalUrl(originalUrl);\n      thriftExpandedUrl.setExpandedUrl(expandedUrl);  // This will be tokenized and indexed\n      // Note that the mediaURL is not indexed. We could also index it, but it is not indexed\n      // to reduce RAM usage.\n      thriftExpandedUrl.setCanonicalLastHopUrl(expandedUrl); // This will be tokenized and indexed\n      addExpandedUrl(originalUrl, thriftExpandedUrl);\n      thriftExpandedUrl.setConsumerMedia(true);\n    }\n  }\n\n  /**\n   * Adds an expanded non-media url based on the given parameters.\n   */\n  public void addExpandedNonMediaUrl(String originalUrl, String expandedUrl) {\n    if (!StringUtils.isBlank(originalUrl)) {\n      ThriftExpandedUrl thriftExpandedUrl = new ThriftExpandedUrl(originalUrl);\n      if (!StringUtils.isBlank(expandedUrl)) {\n        thriftExpandedUrl.setExpandedUrl(expandedUrl);\n      }\n      addExpandedUrl(originalUrl, thriftExpandedUrl);\n      thriftExpandedUrl.setConsumerMedia(false);\n    }\n  }\n\n  /**\n   * Only used in tests.\n   *\n   * Simulates resolving compressed URLs, which is usually done by ResolveCompressedUrlsStage.\n   */\n  @VisibleForTesting\n  public void replaceUrlsWithResolvedUrls(Map<String, String> resolvedUrls) {\n    for (Map.Entry<String, ThriftExpandedUrl> urlEntry : expandedUrls.entrySet()) {\n      String tcoUrl = urlEntry.getKey();\n      if (resolvedUrls.containsKey(tcoUrl)) {\n        ThriftExpandedUrl expandedUrl = urlEntry.getValue();\n        expandedUrl.setCanonicalLastHopUrl(resolvedUrls.get(tcoUrl));\n      }\n    }\n  }\n\n  /**\n   * Adds a mention for a user with the given screen name.\n   */\n  public void addMention(String screenName) {\n    TwitterMessageUser user = TwitterMessageUser.createWithScreenName(screenName);\n    addUserToMentions(user);\n  }\n\n  /**\n   * Adds the given user to mentions.\n   */\n  public void addUserToMentions(TwitterMessageUser user) {\n    Preconditions.checkArgument(user.getScreenName().isPresent(), \"Don't add invalid mentions\");\n    this.mentions.add(user);\n  }\n\n  /**\n   * Adds the given hashtag.\n   */\n  public void addHashtag(String hashtag) {\n    this.hashtags.add(hashtag);\n    for (PenguinVersion penguinVersion : supportedPenguinVersions) {\n      addNormalizedHashtag(NormalizerHelper.normalize(hashtag, getLocale(), penguinVersion),\n          penguinVersion);\n    }\n  }\n\n  private Map<PenguinVersion, VersionedTweetFeatures> getVersionedTweetFeaturesMap() {\n    Map<PenguinVersion, VersionedTweetFeatures> versionedMap =\n        Maps.newEnumMap(PenguinVersion.class);\n    for (PenguinVersion penguinVersion : getSupportedPenguinVersions()) {\n      versionedMap.put(penguinVersion, new VersionedTweetFeatures());\n    }\n\n    return versionedMap;\n  }\n\n  public int getNumFavorites() {\n    return numFavorites;\n  }\n\n  public int getNumRetweets() {\n    return numRetweets;\n  }\n\n  public int getNumReplies() {\n    return numReplies;\n  }\n\n  public Set<NamedEntity> getNamedEntities() {\n    return namedEntities;\n  }\n\n  public void addNamedEntity(NamedEntity namedEntity) {\n    namedEntities.add(namedEntity);\n  }\n\n  public Set<String> getSpaceIds() {\n    return spaceIds;\n  }\n\n  public void setSpaceIds(Set<String> spaceIds) {\n    this.spaceIds = Sets.newHashSet(spaceIds);\n  }\n\n  public Set<TwitterMessageUser> getSpaceAdmins() {\n    return spaceAdmins;\n  }\n\n  public void addSpaceAdmin(TwitterMessageUser admin) {\n    spaceAdmins.add(admin);\n  }\n\n  public String getSpaceTitle() {\n    return spaceTitle;\n  }\n\n  public void setSpaceTitle(String spaceTitle) {\n    this.spaceTitle = spaceTitle;\n  }\n\n  private static boolean equals(List<EscherbirdAnnotation> l1, List<EscherbirdAnnotation> l2) {\n    EscherbirdAnnotation[] arr1 = l1.toArray(new EscherbirdAnnotation[l1.size()]);\n    Arrays.sort(arr1);\n    EscherbirdAnnotation[] arr2 = l1.toArray(new EscherbirdAnnotation[l2.size()]);\n    Arrays.sort(arr2);\n    return Arrays.equals(arr1, arr2);\n  }\n\n  /**\n   * Compares the given messages using reflection and determines if they're approximately equal.\n   */\n  public static boolean reflectionApproxEquals(\n      TwitterMessage a,\n      TwitterMessage b,\n      List<String> additionalExcludeFields) {\n    List<String> excludeFields = Lists.newArrayList(\n        \"versionedTweetFeaturesMap\",\n        \"geoLocation\",\n        \"geoTaggedLocation\",\n        \"escherbirdAnnotations\"\n    );\n    excludeFields.addAll(additionalExcludeFields);\n\n    return EqualsBuilder.reflectionEquals(a, b, excludeFields)\n        && GeoObject.approxEquals(a.getGeoLocation(), b.getGeoLocation())\n        && GeoObject.approxEquals(a.getGeoTaggedLocation(), b.getGeoTaggedLocation())\n        && equals(a.getEscherbirdAnnotations(), b.getEscherbirdAnnotations());\n  }\n\n  public static boolean reflectionApproxEquals(TwitterMessage a, TwitterMessage b) {\n    return reflectionApproxEquals(a, b, Collections.emptyList());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/entities/TwitterMessageUser.java",
    "content": "package com.twitter.search.common.relevance.entities;\n\nimport java.util.Optional;\nimport javax.annotation.Nonnull;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.commons.lang3.builder.EqualsBuilder;\nimport org.apache.commons.lang3.builder.HashCodeBuilder;\nimport org.apache.lucene.analysis.TokenStream;\n\nimport com.twitter.search.common.util.text.TokenizerHelper;\n\n// Represents from-user, to-user, mentions and audioSpace admins in TwitterMessage.\npublic final class TwitterMessageUser {\n\n  @Nonnull private final Optional<String> screenName;  // a.k.a. user handle or username\n  @Nonnull private final Optional<String> displayName;\n\n  @Nonnull private Optional<TokenStream> tokenizedScreenName;\n\n  @Nonnull private final Optional<Long> id; // twitter ID\n\n  public static final class Builder {\n    @Nonnull private Optional<String> screenName = Optional.empty();\n    @Nonnull private Optional<String> displayName = Optional.empty();\n    @Nonnull private Optional<TokenStream> tokenizedScreenName = Optional.empty();\n    @Nonnull private Optional<Long> id = Optional.empty();\n\n    public Builder() {\n    }\n\n    /**\n     * Initialized Builder based on an existing TwitterMessageUser\n     */\n    public Builder(TwitterMessageUser user) {\n      this.screenName = user.screenName;\n      this.displayName = user.displayName;\n      this.tokenizedScreenName = user.tokenizedScreenName;\n      this.id = user.id;\n    }\n\n    /**\n     * Initialized Builder screen name (handle/the name following the \"@\") and do tokenization\n     * for it.\n     */\n    public Builder withScreenName(Optional<String> newScreenName) {\n      this.screenName = newScreenName;\n      if (newScreenName.isPresent()) {\n        this.tokenizedScreenName = Optional.of(\n            TokenizerHelper.getNormalizedCamelcaseTokenStream(newScreenName.get()));\n      }\n      return this;\n    }\n\n    /**\n     * Initialized Builder display name\n     */\n    public Builder withDisplayName(Optional<String> newDisplayName) {\n      this.displayName = newDisplayName;\n      return this;\n    }\n\n    public Builder withId(Optional<Long> newId) {\n      this.id = newId;\n      return this;\n    }\n\n    public TwitterMessageUser build() {\n      return new TwitterMessageUser(\n          screenName, displayName, tokenizedScreenName, id);\n    }\n  }\n\n  /** Creates a TwitterMessageUser instance with the given screen name. */\n  public static TwitterMessageUser createWithScreenName(@Nonnull String screenName) {\n    Preconditions.checkNotNull(screenName, \"Don't set a null screen name\");\n    return new Builder()\n        .withScreenName(Optional.of(screenName))\n        .build();\n  }\n\n  /** Creates a TwitterMessageUser instance with the given display name. */\n  public static TwitterMessageUser createWithDisplayName(@Nonnull String displayName) {\n    Preconditions.checkNotNull(displayName, \"Don't set a null display name\");\n    return new Builder()\n        .withDisplayName(Optional.of(displayName))\n        .build();\n  }\n\n  /** Creates a TwitterMessageUser instance with the given ID. */\n  public static TwitterMessageUser createWithId(long id) {\n    Preconditions.checkArgument(id >= 0, \"Don't sent a negative user ID\");\n    return new Builder()\n        .withId(Optional.of(id))\n        .build();\n  }\n\n  /** Creates a TwitterMessageUser instance with the given parameters. */\n  public static TwitterMessageUser createWithNamesAndId(\n      @Nonnull String screenName,\n      @Nonnull String displayName,\n      long id) {\n    Preconditions.checkNotNull(screenName, \"Use another method instead of passing null name\");\n    Preconditions.checkNotNull(displayName, \"Use another method instead of passing null name\");\n    Preconditions.checkArgument(id >= 0, \"Use another method instead of passing negative ID\");\n    return new Builder()\n        .withScreenName(Optional.of(screenName))\n        .withDisplayName(Optional.of(displayName))\n        .withId(Optional.of(id))\n        .build();\n  }\n\n  /** Creates a TwitterMessageUser instance with the given parameters. */\n  public static TwitterMessageUser createWithNames(\n      @Nonnull String screenName,\n      @Nonnull String displayName) {\n    Preconditions.checkNotNull(screenName, \"Use another method instead of passing null name\");\n    Preconditions.checkNotNull(displayName, \"Use another method instead of passing null name\");\n    return new Builder()\n        .withScreenName(Optional.of(screenName))\n        .withDisplayName(Optional.of(displayName))\n        .build();\n  }\n\n  /** Creates a TwitterMessageUser instance with the given parameters. */\n  public static TwitterMessageUser createWithOptionalNamesAndId(\n      @Nonnull Optional<String> optScreenName,\n      @Nonnull Optional<String> optDisplayName,\n      @Nonnull Optional<Long> optId) {\n    Preconditions.checkNotNull(optScreenName, \"Pass Optional.absent() instead of null\");\n    Preconditions.checkNotNull(optDisplayName, \"Pass Optional.absent() instead of null\");\n    Preconditions.checkNotNull(optId, \"Pass Optional.absent() instead of null\");\n    return new Builder()\n        .withScreenName(optScreenName)\n        .withDisplayName(optDisplayName)\n        .withId(optId)\n        .build();\n  }\n\n  private TwitterMessageUser(\n      @Nonnull Optional<String> screenName,\n      @Nonnull Optional<String> displayName,\n      @Nonnull Optional<TokenStream> tokenizedScreenName,\n      @Nonnull Optional<Long> id) {\n    this.screenName = screenName;\n    this.displayName = displayName;\n    this.tokenizedScreenName = tokenizedScreenName;\n    this.id = id;\n  }\n\n  /** Creates a copy of this TwitterMessageUser instance, with the given screen name. */\n  public TwitterMessageUser copyWithScreenName(@Nonnull String newScreenName) {\n    Preconditions.checkNotNull(newScreenName, \"Don't set a null screen name\");\n    return new Builder(this)\n        .withScreenName(Optional.of(newScreenName))\n        .build();\n  }\n\n  /** Creates a copy of this TwitterMessageUser instance, with the given display name. */\n  public TwitterMessageUser copyWithDisplayName(@Nonnull String newDisplayName) {\n    Preconditions.checkNotNull(newDisplayName, \"Don't set a null display name\");\n    return new Builder(this)\n        .withDisplayName(Optional.of(newDisplayName))\n        .build();\n  }\n\n  /** Creates a copy of this TwitterMessageUser instance, with the given ID. */\n  public TwitterMessageUser copyWithId(long newId) {\n    Preconditions.checkArgument(newId >= 0, \"Don't set a negative user ID\");\n    return new Builder(this)\n        .withId(Optional.of(newId))\n        .build();\n  }\n\n  public Optional<String> getScreenName() {\n    return screenName;\n  }\n\n  public Optional<String> getDisplayName() {\n    return displayName;\n  }\n\n  public Optional<TokenStream> getTokenizedScreenName() {\n    return tokenizedScreenName;\n  }\n\n  public Optional<Long> getId() {\n    return id;\n  }\n\n  @Override\n  public String toString() {\n    return \"[\" + screenName + \", \" + displayName + \", \" + id + \"]\";\n  }\n\n  /**\n   * Compares this TwitterMessageUser instance to the given object.\n   *\n   * @deprecated deprecated.\n   */\n  @Deprecated\n  @Override\n  public boolean equals(Object o) {\n    if (o == null) {\n      return false;\n    }\n    if (o == this) {\n      return true;\n    }\n    if (o.getClass() != getClass()) {\n      return false;\n    }\n    TwitterMessageUser other = (TwitterMessageUser) o;\n    return new EqualsBuilder()\n        .append(screenName, other.screenName)\n        .append(displayName, other.displayName)\n        .isEquals();\n  }\n\n  /**\n   * Returns a hash code for this TwitterMessageUser instance.\n   *\n   * @deprecated deprecated.\n   */\n  @Deprecated\n  @Override\n  public int hashCode() {\n    return HashCodeBuilder.reflectionHashCode(this);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/entities/TwitterMessageUtil.java",
    "content": "package com.twitter.search.common.relevance.entities;\n\nimport java.text.Normalizer;\nimport java.util.Map;\nimport java.util.NavigableMap;\nimport java.util.Set;\nimport java.util.TreeMap;\nimport java.util.concurrent.ConcurrentMap;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Maps;\n\nimport org.apache.commons.lang.StringUtils;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.text.transformer.HTMLTagRemovalTransformer;\nimport com.twitter.common_internal.text.extractor.EmojiExtractor;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\n\npublic final class TwitterMessageUtil {\n  private static final Logger LOG = LoggerFactory.getLogger(TwitterMessageUtil.class);\n\n  private TwitterMessageUtil() {\n  }\n\n  @VisibleForTesting\n  static final ConcurrentMap<Field, Counters> COUNTERS_MAP = Maps.newConcurrentMap();\n  // We truncate the location string because we used to use a MySQL table to store the geocoding\n  // information.  In the MySQL table, the location string was fix width of 30 characters.\n  // We have migrated to Manhattan and the location string is no longer limited to 30 character.\n  // However, in order to correctly lookup location geocode from Manhattan, we still need to\n  // truncate the location just like we did before.\n  private static final int MAX_LOCATION_LEN = 30;\n\n  // Note: we strip tags to index source, as typically source contains <a href=...> tags.\n  // Sometimes we get a source where stripping fails, as the URL in the tag was\n  // excessively long.  We drop these sources, as there is little reason to index them.\n  private static final int MAX_SOURCE_LEN = 64;\n\n  private static HTMLTagRemovalTransformer tagRemovalTransformer = new HTMLTagRemovalTransformer();\n\n  private static final String STAT_PREFIX = \"twitter_message_\";\n\n  public enum Field {\n    FROM_USER_DISPLAY_NAME,\n    NORMALIZED_LOCATION,\n    ORIG_LOCATION,\n    ORIG_SOURCE,\n    SHARED_USER_DISPLAY_NAME,\n    SOURCE,\n    TEXT,\n    TO_USER_SCREEN_NAME;\n\n    public String getNameForStats() {\n      return name().toLowerCase();\n    }\n  }\n\n  @VisibleForTesting\n  static class Counters {\n    private final SearchRateCounter truncatedCounter;\n    private final SearchRateCounter tweetsWithStrippedSupplementaryCharsCounter;\n    private final SearchRateCounter strippedSupplementaryCharsCounter;\n    private final SearchRateCounter nonStrippedEmojiCharsCounter;\n    private final SearchRateCounter emojisAtTruncateBoundaryCounter;\n\n    Counters(Field field) {\n      String fieldNameForStats = field.getNameForStats();\n      truncatedCounter = SearchRateCounter.export(\n          STAT_PREFIX + \"truncated_\" + fieldNameForStats);\n      tweetsWithStrippedSupplementaryCharsCounter = SearchRateCounter.export(\n          STAT_PREFIX + \"tweets_with_stripped_supplementary_chars_\" + fieldNameForStats);\n      strippedSupplementaryCharsCounter = SearchRateCounter.export(\n          STAT_PREFIX + \"stripped_supplementary_chars_\" + fieldNameForStats);\n      nonStrippedEmojiCharsCounter = SearchRateCounter.export(\n          STAT_PREFIX + \"non_stripped_emoji_chars_\" + fieldNameForStats);\n      emojisAtTruncateBoundaryCounter = SearchRateCounter.export(\n          STAT_PREFIX + \"emojis_at_truncate_boundary_\" + fieldNameForStats);\n    }\n\n    SearchRateCounter getTruncatedCounter() {\n      return truncatedCounter;\n    }\n\n    SearchRateCounter getTweetsWithStrippedSupplementaryCharsCounter() {\n      return tweetsWithStrippedSupplementaryCharsCounter;\n    }\n\n    SearchRateCounter getStrippedSupplementaryCharsCounter() {\n      return strippedSupplementaryCharsCounter;\n    }\n\n    SearchRateCounter getNonStrippedEmojiCharsCounter() {\n      return nonStrippedEmojiCharsCounter;\n    }\n\n    SearchRateCounter getEmojisAtTruncateBoundaryCounter() {\n      return emojisAtTruncateBoundaryCounter;\n    }\n  }\n\n  static {\n    for (Field field : Field.values()) {\n      COUNTERS_MAP.put(field, new Counters(field));\n    }\n  }\n\n  // Note: the monorail enforces a limit of 15 characters for screen names,\n  // but some users with up to 20 character names were grandfathered-in.  To allow\n  // those users to be searchable, support up to 20 chars.\n  private static final int MAX_SCREEN_NAME_LEN = 20;\n\n  // Note: we expect the current limit to be 10K. Also, all supplementary unicode characters (with\n  // the exception of emojis, maybe) will be removed and not counted as total length. Added alert\n  // for text truncation rate as well. SEARCH-9512\n  private static final int MAX_TWEET_TEXT_LEN = 10000;\n\n  @VisibleForTesting\n  static final SearchRateCounter FILTERED_NO_STATUS_ID =\n      SearchRateCounter.export(STAT_PREFIX + \"filtered_no_status_id\");\n  @VisibleForTesting\n  static final SearchRateCounter FILTERED_NO_FROM_USER =\n      SearchRateCounter.export(STAT_PREFIX + \"filtered_no_from_user\");\n  @VisibleForTesting\n  static final SearchRateCounter FILTERED_LONG_SCREEN_NAME =\n      SearchRateCounter.export(STAT_PREFIX + \"filtered_long_screen_name\");\n  @VisibleForTesting\n  static final SearchRateCounter FILTERED_NO_TEXT =\n      SearchRateCounter.export(STAT_PREFIX + \"filtered_no_text\");\n  @VisibleForTesting\n  static final SearchRateCounter FILTERED_NO_DATE =\n      SearchRateCounter.export(STAT_PREFIX + \"filtered_no_date\");\n  @VisibleForTesting\n  static final SearchRateCounter NULLCAST_TWEET =\n      SearchRateCounter.export(STAT_PREFIX + \"filter_nullcast_tweet\");\n  @VisibleForTesting\n  static final SearchRateCounter NULLCAST_TWEET_ACCEPTED =\n      SearchRateCounter.export(STAT_PREFIX + \"nullcast_tweet_accepted\");\n  @VisibleForTesting\n  static final SearchRateCounter INCONSISTENT_TWEET_ID_AND_CREATED_AT =\n      SearchRateCounter.export(STAT_PREFIX + \"inconsistent_tweet_id_and_created_at_ms\");\n\n  /** Strips the given source from the message with the given ID. */\n  private static String stripSource(String source, Long messageId) {\n    if (source == null) {\n      return null;\n    }\n    // Always strip emojis from sources: they don't really make sense in this field.\n    String strippedSource = stripSupplementaryChars(\n        tagRemovalTransformer.transform(source).toString(), Field.SOURCE, true);\n    if (strippedSource.length() > MAX_SOURCE_LEN) {\n      LOG.warn(\"Message \"\n          + messageId\n          + \" contains stripped source that exceeds MAX_SOURCE_LEN. Removing: \"\n          + strippedSource);\n      COUNTERS_MAP.get(Field.SOURCE).getTruncatedCounter().increment();\n      return null;\n    }\n    return strippedSource;\n  }\n\n  /**\n   * Strips and truncates the location of the message with the given ID.\n   *\n   */\n  private static String stripAndTruncateLocation(String location) {\n    // Always strip emojis from locations: they don't really make sense in this field.\n    String strippedLocation = stripSupplementaryChars(location, Field.NORMALIZED_LOCATION, true);\n    return truncateString(strippedLocation, MAX_LOCATION_LEN, Field.NORMALIZED_LOCATION, true);\n  }\n\n  /**\n   * Sets the origSource and strippedSource fields on a TwitterMessage\n   *\n   */\n  public static void setSourceOnMessage(TwitterMessage message, String modifiedDeviceSource) {\n    // Always strip emojis from sources: they don't really make sense in this field.\n    message.setOrigSource(stripSupplementaryChars(modifiedDeviceSource, Field.ORIG_SOURCE, true));\n    message.setStrippedSource(stripSource(modifiedDeviceSource, message.getId()));\n  }\n\n  /**\n   * Sets the origLocation to the stripped location, and sets\n   * the truncatedNormalizedLocation to the truncated and normalized location.\n   */\n  public static void setAndTruncateLocationOnMessage(\n      TwitterMessage message,\n      String newOrigLocation) {\n    // Always strip emojis from locations: they don't really make sense in this field.\n    message.setOrigLocation(stripSupplementaryChars(newOrigLocation, Field.ORIG_LOCATION, true));\n\n    // Locations in the new locations table require additional normalization. It can also change\n    // the length of the string, so we must do this before truncation.\n    if (newOrigLocation != null) {\n      String normalized =\n          Normalizer.normalize(newOrigLocation, Normalizer.Form.NFKC).toLowerCase().trim();\n      message.setTruncatedNormalizedLocation(stripAndTruncateLocation(normalized));\n    } else {\n      message.setTruncatedNormalizedLocation(null);\n    }\n  }\n\n  /**\n   * Validates the given TwitterMessage.\n   *\n   * @param message The message to validate.\n   * @param stripEmojisForFields The set of fields for which emojis should be stripped.\n   * @param acceptNullcastMessage Determines if this message should be accepted, if it's a nullcast\n   *                              message.\n   * @return {@code true} if the given message is valid; {@code false} otherwise.\n   */\n  public static boolean validateTwitterMessage(\n      TwitterMessage message,\n      Set<Field> stripEmojisForFields,\n      boolean acceptNullcastMessage) {\n    if (message.getNullcast()) {\n      NULLCAST_TWEET.increment();\n      if (!acceptNullcastMessage) {\n        LOG.info(\"Dropping nullcasted message \" + message.getId());\n        return false;\n      }\n      NULLCAST_TWEET_ACCEPTED.increment();\n    }\n\n    if (!message.getFromUserScreenName().isPresent()\n        || StringUtils.isBlank(message.getFromUserScreenName().get())) {\n      LOG.error(\"Message \" + message.getId() + \" contains no from user. Skipping.\");\n      FILTERED_NO_FROM_USER.increment();\n      return false;\n    }\n    String fromUserScreenName = message.getFromUserScreenName().get();\n\n    if (fromUserScreenName.length() > MAX_SCREEN_NAME_LEN) {\n      LOG.warn(\"Message \" + message.getId() + \" has a user screen name longer than \"\n               + MAX_SCREEN_NAME_LEN + \" characters: \" + message.getFromUserScreenName()\n               + \". Skipping.\");\n      FILTERED_LONG_SCREEN_NAME.increment();\n      return false;\n    }\n\n    // Remove supplementary characters and truncate these text fields.\n    if (message.getFromUserDisplayName().isPresent()) {\n      message.setFromUserDisplayName(stripSupplementaryChars(\n          message.getFromUserDisplayName().get(),\n          Field.FROM_USER_DISPLAY_NAME,\n          stripEmojisForFields.contains(Field.FROM_USER_DISPLAY_NAME)));\n    }\n    if (message.getToUserScreenName().isPresent()) {\n      String strippedToUserScreenName = stripSupplementaryChars(\n          message.getToUserLowercasedScreenName().get(),\n          Field.TO_USER_SCREEN_NAME,\n          stripEmojisForFields.contains(Field.TO_USER_SCREEN_NAME));\n      message.setToUserScreenName(\n          truncateString(\n              strippedToUserScreenName,\n              MAX_SCREEN_NAME_LEN,\n              Field.TO_USER_SCREEN_NAME,\n              stripEmojisForFields.contains(Field.TO_USER_SCREEN_NAME)));\n    }\n\n    String strippedText = stripSupplementaryChars(\n        message.getText(),\n        Field.TEXT,\n        stripEmojisForFields.contains(Field.TEXT));\n    message.setText(truncateString(\n        strippedText,\n        MAX_TWEET_TEXT_LEN,\n        Field.TEXT,\n        stripEmojisForFields.contains(Field.TEXT)));\n\n    if (StringUtils.isBlank(message.getText())) {\n      FILTERED_NO_TEXT.increment();\n      return false;\n    }\n\n    if (message.getDate() == null) {\n      LOG.error(\"Message \" + message.getId() + \" contains no date. Skipping.\");\n      FILTERED_NO_DATE.increment();\n      return false;\n    }\n\n    if (message.isRetweet()) {\n      return validateRetweetMessage(message.getRetweetMessage(), stripEmojisForFields);\n    }\n\n    // Track if both the snowflake ID and created at timestamp are consistent.\n    if (!SnowflakeIdParser.isTweetIDAndCreatedAtConsistent(message.getId(), message.getDate())) {\n      LOG.error(\"Found inconsistent tweet ID and created at timestamp: [messageID=\"\n                + message.getId() + \"], [messageDate=\" + message.getDate() + \"].\");\n      INCONSISTENT_TWEET_ID_AND_CREATED_AT.increment();\n    }\n\n    return true;\n  }\n\n  private static boolean validateRetweetMessage(\n      TwitterRetweetMessage message, Set<Field> stripEmojisForFields) {\n    if (message.getSharedId() == null || message.getRetweetId() == null) {\n      LOG.error(\"Retweet Message contains a null twitter id. Skipping.\");\n      FILTERED_NO_STATUS_ID.increment();\n      return false;\n    }\n\n    if (message.getSharedDate() == null) {\n      LOG.error(\"Retweet Message \" + message.getRetweetId() + \" contains no date. Skipping.\");\n      return false;\n    }\n\n    // Remove supplementary characters from these text fields.\n    message.setSharedUserDisplayName(stripSupplementaryChars(\n        message.getSharedUserDisplayName(),\n        Field.SHARED_USER_DISPLAY_NAME,\n        stripEmojisForFields.contains(Field.SHARED_USER_DISPLAY_NAME)));\n\n    return true;\n  }\n\n  /**\n   * Strips non indexable chars from the text.\n   *\n   * Returns the resulting string, which may be the same object as the text argument when\n   * no stripping or truncation is necessary.\n   *\n   * Non-indexed characters are \"supplementary unicode\" that are not emojis. Note that\n   * supplementary unicode are still characters that seem worth indexing, as many characters\n   * in CJK languages are supplementary. However this would make the size of our index\n   * explode (~186k supplementary characters exist), so it's not feasible.\n   *\n   * @param text The text to strip\n   * @param field The field this text is from\n   * @param stripSupplementaryEmojis Whether or not to strip supplementary emojis. Note that this\n   * parameter name isn't 100% accurate. This parameter is meant to replicate behavior prior to\n   * adding support for *not* stripping supplementary emojis. The prior behavior would turn an\n   * emoji such as a keycap \"1\\uFE0F\\u20E3\" (http://www.iemoji.com/view/emoji/295/symbols/keycap-1)\n   * into just '1'. So the keycap emoji is not completely stripped, only the portion after the '1'.\n   *\n   */\n  @VisibleForTesting\n  public static String stripSupplementaryChars(\n      String text,\n      Field field,\n      boolean stripSupplementaryEmojis) {\n    if (text == null || text.isEmpty()) {\n      return text;\n    }\n\n    // Initialize an empty map so that if we choose not to strip emojis,\n    // then no emojipositions will be found and we don't need a null\n    // check before checking if an emoji is at a certain spot.\n    NavigableMap<Integer, Integer> emojiPositions = new TreeMap<>();\n\n    if (!stripSupplementaryEmojis) {\n      emojiPositions = EmojiExtractor.getEmojiPositions(text);\n    }\n\n    StringBuilder strippedTextBuilder = new StringBuilder();\n    int sequenceStart = 0;\n    int i = 0;\n    while (i < text.length()) {\n      if (Character.isSupplementaryCodePoint(text.codePointAt(i))) {\n        // Check if this supplementary character is an emoji\n        if (!emojiPositions.containsKey(i)) {\n          // It's not an emoji, or we want to strip emojis, so strip it\n\n          // text[i] and text[i + 1] are part of a supplementary code point.\n          strippedTextBuilder.append(text.substring(sequenceStart, i));\n          sequenceStart = i + 2;  // skip 2 chars\n          i = sequenceStart;\n          COUNTERS_MAP.get(field).getStrippedSupplementaryCharsCounter().increment();\n        } else {\n          // It's an emoji, keep it\n          i += emojiPositions.get(i);\n          COUNTERS_MAP.get(field).getNonStrippedEmojiCharsCounter().increment();\n        }\n      } else {\n        ++i;\n      }\n    }\n    if (sequenceStart < text.length()) {\n      strippedTextBuilder.append(text.substring(sequenceStart));\n    }\n\n    String strippedText = strippedTextBuilder.toString();\n    if (strippedText.length() < text.length()) {\n      COUNTERS_MAP.get(field).getTweetsWithStrippedSupplementaryCharsCounter().increment();\n    }\n    return strippedText;\n  }\n\n  /**\n   * Truncates the given string to the given length.\n   *\n   * Note that we are truncating based on the # of UTF-16 characters a given emoji takes up.\n   * So if a single emoji takes up 4 UTF-16 characters, that counts as 4 for the truncation,\n   * not just 1.\n   *\n   * @param text The text to truncate\n   * @param maxLength The maximum length of the string after truncation\n   * @param field The field from which this string cames\n   * @param splitEmojisAtMaxLength If true, don't worry about emojis and just truncate at maxLength,\n   * potentially splitting them. If false, truncate before the emoji if truncating at maxLength\n   * would cause the emoji to be split.\n   */\n  @VisibleForTesting\n  static String truncateString(\n      String text,\n      int maxLength,\n      Field field,\n      boolean splitEmojisAtMaxLength) {\n    Preconditions.checkArgument(maxLength > 0);\n\n    if ((text == null) || (text.length() <= maxLength)) {\n      return text;\n    }\n\n    int truncatePoint = maxLength;\n    NavigableMap<Integer, Integer> emojiPositions;\n    // If we want to consider emojis we should not strip on an emoji boundary.\n    if (!splitEmojisAtMaxLength) {\n      emojiPositions = EmojiExtractor.getEmojiPositions(text);\n\n      // Get the last emoji before maxlength.\n      Map.Entry<Integer, Integer> lastEmojiBeforeMaxLengthEntry =\n          emojiPositions.lowerEntry(maxLength);\n\n      if (lastEmojiBeforeMaxLengthEntry != null) {\n        int lowerEmojiEnd = lastEmojiBeforeMaxLengthEntry.getKey()\n            + lastEmojiBeforeMaxLengthEntry.getValue();\n\n        // If the last emoji would be truncated, truncate before the last emoji.\n        if (lowerEmojiEnd > truncatePoint) {\n          truncatePoint = lastEmojiBeforeMaxLengthEntry.getKey();\n          COUNTERS_MAP.get(field).getEmojisAtTruncateBoundaryCounter().increment();\n        }\n      }\n    }\n\n    COUNTERS_MAP.get(field).getTruncatedCounter().increment();\n    return text.substring(0, truncatePoint);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/entities/TwitterQuotedMessage.java",
    "content": "package com.twitter.search.common.relevance.entities;\n\nimport org.apache.commons.lang3.builder.EqualsBuilder;\nimport org.apache.commons.lang3.builder.HashCodeBuilder;\nimport org.apache.commons.lang3.builder.ToStringBuilder;\n\n/**\n * The object for quoted message\n  */\npublic class TwitterQuotedMessage {\n  private final long quotedStatusId;\n  private final long quotedUserId;\n\n  public TwitterQuotedMessage(long quotedStatusId, long quotedUserId) {\n    this.quotedStatusId = quotedStatusId;\n    this.quotedUserId = quotedUserId;\n  }\n\n  public long getQuotedStatusId() {\n    return quotedStatusId;\n  }\n\n  public long getQuotedUserId() {\n    return quotedUserId;\n  }\n\n  @Override\n  public boolean equals(Object o) {\n    return EqualsBuilder.reflectionEquals(this, o);\n  }\n\n  @Override\n  public int hashCode() {\n    return HashCodeBuilder.reflectionHashCode(this);\n  }\n\n  @Override\n  public String toString() {\n    return ToStringBuilder.reflectionToString(this);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/entities/TwitterRetweetMessage.java",
    "content": "package com.twitter.search.common.relevance.entities;\n\nimport java.util.Date;\n\nimport org.apache.commons.lang3.builder.EqualsBuilder;\nimport org.apache.commons.lang3.builder.HashCodeBuilder;\nimport org.apache.commons.lang3.builder.ToStringBuilder;\n\npublic class TwitterRetweetMessage {\n  // based on original tweet\n  private Long sharedId;\n\n  // TwitterMessageUtil checks them\n  private String sharedUserDisplayName;\n  private Long sharedUserTwitterId = TwitterMessage.LONG_FIELD_NOT_PRESENT;\n\n  private Date sharedDate = null;\n\n  // based on retweet\n  private Long retweetId;\n\n  public Long getRetweetId() {\n    return retweetId;\n  }\n\n  public void setRetweetId(Long retweetId) {\n    this.retweetId = retweetId;\n  }\n\n  public Long getSharedId() {\n    return sharedId;\n  }\n\n  public void setSharedId(Long sharedId) {\n    this.sharedId = sharedId;\n  }\n\n  public String getSharedUserDisplayName() {\n    return sharedUserDisplayName;\n  }\n\n  public void setSharedUserDisplayName(String sharedUserDisplayName) {\n    this.sharedUserDisplayName = sharedUserDisplayName;\n  }\n\n  public Long getSharedUserTwitterId() {\n    return sharedUserTwitterId;\n  }\n\n  public boolean hasSharedUserTwitterId() {\n    return sharedUserTwitterId != TwitterMessage.LONG_FIELD_NOT_PRESENT;\n  }\n\n  public void setSharedUserTwitterId(Long sharedUserTwitterId) {\n    this.sharedUserTwitterId = sharedUserTwitterId;\n  }\n\n  public Date getSharedDate() {\n    return sharedDate;\n  }\n\n  public void setSharedDate(Date sharedDate) {\n    this.sharedDate = sharedDate;\n  }\n\n  @Override\n  public boolean equals(Object o) {\n    return EqualsBuilder.reflectionEquals(this, o);\n  }\n\n  @Override\n  public int hashCode() {\n    return HashCodeBuilder.reflectionHashCode(this);\n  }\n\n  @Override\n  public String toString() {\n    return ToStringBuilder.reflectionToString(this);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/AgeDecay.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.base.Preconditions;\n\n/**\n * Utility to compute an age decay multiplier based on a sigmoid function.\n */\npublic class AgeDecay {\n  public static final double SLOPE_COEFF = 4.0;\n  public static final double LN_HALF = Math.log(0.5);\n  public final double halflife;\n  public final double maxBoost;\n  public final double base;\n  public final double slope;\n\n  /** Creates a new AgeDecay instance. */\n  public AgeDecay(double base, double maxBoost, double halflife, double slope) {\n    this.maxBoost = maxBoost;\n    this.base = base;\n    this.halflife = halflife;\n    this.slope = slope;\n  }\n\n  /** Creates a new AgeDecay instance. */\n  public AgeDecay(double base, double halflife, double slope) {\n    this(base, 1.0, halflife, slope);\n  }\n\n  /**\n   * Compute the age decay, using the provided halflife.\n   *\n   * @param tweetAge The tweet age.\n   * @param unit The unit of the tweetAge parameter.\n   */\n  public double getAgeDecayMultiplier(long tweetAge, TimeUnit unit) {\n    return getAgeDecayMultiplier(TimeUnit.SECONDS.convert(tweetAge, unit));\n  }\n\n  /**\n   * Compute the age decay, assuming the halflife in the constructor is in minutes.\n   * @param ageInSeconds the age in seconds\n   */\n  public double getAgeDecayMultiplier(long ageInSeconds) {\n    long minutesSinceTweet = TimeUnit.MINUTES.convert(ageInSeconds, TimeUnit.SECONDS);\n    return compute(minutesSinceTweet);\n  }\n\n  /**\n   * Compute age decay given an age, the age has to be in the same unit as halflife, which you\n   * construct the object with.\n   */\n  public double compute(double age) {\n    return compute(base, maxBoost, halflife, slope, age);\n  }\n\n  /**\n   * Compute the age decay given all parameters. Use this if you don't need to reuse an AgeDecay\n   * object.\n   */\n  public static double compute(\n      double base, double maxBoost, double halflife, double slope, double age) {\n    return base + ((maxBoost - base) / (1 + Math.exp(slope * (age - halflife))));\n  }\n\n  public static double compute(\n      double base, double maxBoost, double halflife, double age) {\n    Preconditions.checkArgument(halflife != 0);\n    return compute(base, maxBoost, halflife, SLOPE_COEFF / halflife, age);\n  }\n\n  /**\n   * Another nicer exponential decay function. Returns a value in (0, 1]\n   */\n  public static double computeExponential(double halflife, double exp, double age) {\n    return Math.exp(LN_HALF * Math.pow(age, exp) / Math.pow(halflife, exp));\n  }\n\n  /**\n   * Exponential decay with remapping of the value from (0,1] to (min,max]\n   */\n  public static double computeExponential(double halflife, double exp, double age,\n                                          double minBoost, double maxBoost) {\n    double decay = computeExponential(halflife, exp, age);  // in (0, 1]\n    return (maxBoost - minBoost) * decay + minBoost;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/BUILD",
    "content": "# Java library for tweet features and utilities.\njava_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/twitter/elephantbird:core\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/text/token\",\n        \"src/java/com/twitter/search/common/encoding/features\",\n        \"src/java/com/twitter/search/common/features\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/common/util/lang\",\n        \"src/thrift/com/twitter/search/common:constants-java\",\n        \"src/thrift/com/twitter/search/common:features-java\",\n        \"src/thrift/com/twitter/search/common:schema-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/EarlybirdDocumentFeatures.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport java.io.IOException;\nimport java.util.Map;\nimport java.util.function.Function;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Maps;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.NumericDocValues;\n\nimport com.twitter.search.common.features.thrift.ThriftSearchResultFeatures;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFType;\nimport com.twitter.search.common.schema.thriftjava.ThriftFeatureNormalizationType;\n\npublic class EarlybirdDocumentFeatures {\n  private static final Map<Integer, SearchCounter> FEATURE_CONFIG_IS_NULL_MAP = Maps.newHashMap();\n  private static final Map<Integer, SearchCounter> FEATURE_OUTPUT_TYPE_IS_NULL_MAP =\n      Maps.newHashMap();\n  private static final Map<Integer, SearchCounter> NO_SCHEMA_FIELD_FOR_FEATURE_MAP =\n      Maps.newHashMap();\n  private static final String FEATURE_CONFIG_IS_NULL_COUNTER_PATTERN =\n      \"null_feature_config_for_feature_id_%d\";\n  private static final String FEATURE_OUTPUT_TYPE_IS_NULL_COUNTER_PATTERN =\n      \"null_output_type_for_feature_id_%d\";\n  private static final String NO_SCHEMA_FIELD_FOR_FEATURE_COUNTER_PATTERN =\n      \"no_schema_field_for_feature_id_%d\";\n  private static final SearchCounter UNKNOWN_FEATURE_OUTPUT_TYPE_COUNTER =\n      SearchCounter.export(\"unknown_feature_output_type\");\n\n  private final Map<String, NumericDocValues> numericDocValues = Maps.newHashMap();\n  private final LeafReader leafReader;\n  private int docId = -1;\n\n  /**\n   * Creates a new EarlybirdDocumentFeatures instance that will return feature values based on the\n   * NumericDocValues stored in the given LeafReader for the given document.\n   */\n  public EarlybirdDocumentFeatures(LeafReader leafReader) {\n    this.leafReader = Preconditions.checkNotNull(leafReader);\n  }\n\n  /**\n   * Advances this instance to the given doc ID. The new doc ID must be greater than or equal to the\n   * current doc ID stored in this instance.\n   */\n  public void advance(int target) {\n    Preconditions.checkArgument(\n        target >= 0,\n        \"Target (%s) cannot be negative.\",\n        target);\n    Preconditions.checkArgument(\n        target >= docId,\n        \"Target (%s) smaller than current doc ID (%s).\",\n        target,\n        docId);\n    Preconditions.checkArgument(\n        target < leafReader.maxDoc(),\n        \"Target (%s) cannot be greater than or equal to the max doc ID (%s).\",\n        target,\n        leafReader.maxDoc());\n    docId = target;\n  }\n\n  /**\n   * Returns the feature value for the given field.\n   */\n  public long getFeatureValue(EarlybirdFieldConstant field) throws IOException {\n    // The index might not have a NumericDocValues instance for this feature.\n    // This might happen if we dynamically update the feature schema, for example.\n    //\n    // Cache the NumericDocValues instances for all accessed features, even if they're null.\n    String fieldName = field.getFieldName();\n    NumericDocValues docValues;\n    if (numericDocValues.containsKey(fieldName)) {\n      docValues = numericDocValues.get(fieldName);\n    } else {\n      docValues = leafReader.getNumericDocValues(fieldName);\n      numericDocValues.put(fieldName, docValues);\n    }\n    return docValues != null && docValues.advanceExact(docId) ? docValues.longValue() : 0L;\n  }\n\n  /**\n   * Determines if the given flag is set.\n   */\n  public boolean isFlagSet(EarlybirdFieldConstant field) throws IOException {\n    return getFeatureValue(field) != 0;\n  }\n\n  /**\n   * Returns the unnormalized value for the given field.\n   */\n  public double getUnnormalizedFeatureValue(EarlybirdFieldConstant field) throws IOException {\n    long featureValue = getFeatureValue(field);\n    ThriftFeatureNormalizationType normalizationType = field.getFeatureNormalizationType();\n    if (normalizationType == null) {\n      normalizationType = ThriftFeatureNormalizationType.NONE;\n    }\n    switch (normalizationType) {\n      case NONE:\n        return featureValue;\n      case LEGACY_BYTE_NORMALIZER:\n        return MutableFeatureNormalizers.BYTE_NORMALIZER.unnormLowerBound((byte) featureValue);\n      case LEGACY_BYTE_NORMALIZER_WITH_LOG2:\n        return MutableFeatureNormalizers.BYTE_NORMALIZER.unnormAndLog2((byte) featureValue);\n      case SMART_INTEGER_NORMALIZER:\n        return MutableFeatureNormalizers.SMART_INTEGER_NORMALIZER.unnormUpperBound(\n            (byte) featureValue);\n      case PREDICTION_SCORE_NORMALIZER:\n        return IntNormalizers.PREDICTION_SCORE_NORMALIZER.denormalize((int) featureValue);\n      default:\n        throw new IllegalArgumentException(\n            \"Unsupported normalization type \" + normalizationType + \" for feature \"\n                + field.getFieldName());\n    }\n  }\n\n  /**\n   * Creates a ThriftSearchResultFeatures instance populated with values for all available features\n   * that have a non-zero value set.\n   */\n  public ThriftSearchResultFeatures getSearchResultFeatures(ImmutableSchemaInterface schema)\n      throws IOException {\n    return getSearchResultFeatures(schema, (featureId) -> true);\n  }\n\n  /**\n   * Creates a ThriftSearchResultFeatures instance populated with values for all available features\n   * that have a non-zero value set.\n   *\n   * @param schema The schema.\n   * @param shouldCollectFeatureId A predicate that determines which features should be collected.\n   */\n  public ThriftSearchResultFeatures getSearchResultFeatures(\n      ImmutableSchemaInterface schema,\n      Function<Integer, Boolean> shouldCollectFeatureId) throws IOException {\n    Map<Integer, Boolean> boolValues = Maps.newHashMap();\n    Map<Integer, Double> doubleValues = Maps.newHashMap();\n    Map<Integer, Integer> intValues = Maps.newHashMap();\n    Map<Integer, Long> longValues = Maps.newHashMap();\n\n    Map<Integer, FeatureConfiguration> idToFeatureConfigMap = schema.getFeatureIdToFeatureConfig();\n    for (int featureId : schema.getSearchFeatureSchema().getEntries().keySet()) {\n      if (!shouldCollectFeatureId.apply(featureId)) {\n        continue;\n      }\n\n      FeatureConfiguration featureConfig = idToFeatureConfigMap.get(featureId);\n      if (featureConfig == null) {\n        FEATURE_CONFIG_IS_NULL_MAP.computeIfAbsent(\n            featureId,\n            (fId) -> SearchCounter.export(\n                String.format(FEATURE_CONFIG_IS_NULL_COUNTER_PATTERN, fId))).increment();\n        continue;\n      }\n\n      ThriftCSFType outputType = featureConfig.getOutputType();\n      if (outputType == null) {\n        FEATURE_OUTPUT_TYPE_IS_NULL_MAP.computeIfAbsent(\n            featureId,\n            (fId) -> SearchCounter.export(\n                String.format(FEATURE_OUTPUT_TYPE_IS_NULL_COUNTER_PATTERN, fId))).increment();\n        continue;\n      }\n\n      if (!EarlybirdFieldConstants.hasFieldConstant(featureId)) {\n        // Should only happen for features that were dynamically added to the schema.\n        NO_SCHEMA_FIELD_FOR_FEATURE_MAP.computeIfAbsent(\n            featureId,\n            (fId) -> SearchCounter.export(\n                String.format(NO_SCHEMA_FIELD_FOR_FEATURE_COUNTER_PATTERN, fId))).increment();\n        continue;\n      }\n\n      EarlybirdFieldConstant field = EarlybirdFieldConstants.getFieldConstant(featureId);\n      switch (outputType) {\n        case BOOLEAN:\n          if (isFlagSet(field)) {\n            boolValues.put(featureId, true);\n          }\n          break;\n        case BYTE:\n          // It's unclear why we don't add this feature to a separate byteValues map...\n          byte byteFeatureValue = (byte) getFeatureValue(field);\n          if (byteFeatureValue != 0) {\n            intValues.put(featureId, (int) byteFeatureValue);\n          }\n          break;\n        case INT:\n          int intFeatureValue = (int) getFeatureValue(field);\n          if (intFeatureValue != 0) {\n            intValues.put(featureId, intFeatureValue);\n          }\n          break;\n        case LONG:\n          long longFeatureValue = getFeatureValue(field);\n          if (longFeatureValue != 0) {\n            longValues.put(featureId, longFeatureValue);\n          }\n          break;\n        case FLOAT:\n          // It's unclear why we don't add this feature to a separate floatValues map...\n          float floatFeatureValue = (float) getFeatureValue(field);\n          if (floatFeatureValue != 0) {\n            doubleValues.put(featureId, (double) floatFeatureValue);\n          }\n          break;\n        case DOUBLE:\n          double doubleFeatureValue = getUnnormalizedFeatureValue(field);\n          if (doubleFeatureValue != 0) {\n            doubleValues.put(featureId, doubleFeatureValue);\n          }\n          break;\n        default:\n          UNKNOWN_FEATURE_OUTPUT_TYPE_COUNTER.increment();\n      }\n    }\n\n    return new ThriftSearchResultFeatures()\n        .setBoolValues(boolValues)\n        .setIntValues(intValues)\n        .setLongValues(longValues)\n        .setDoubleValues(doubleValues);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/FeatureSink.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport java.util.Map;\n\nimport com.google.common.collect.Maps;\n\nimport com.twitter.search.common.encoding.features.IntegerEncodedFeatures;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdEncodedFeatures;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\n\n/**\n * FeatureSink is used to write features based on feature configuration or feature name.  After\n * all feature is written, the class can return the base field integer array values.\n *\n * This class is not thread-safe.\n */\npublic class FeatureSink {\n  private ImmutableSchemaInterface schema;\n  private final Map<String, IntegerEncodedFeatures> encodedFeatureMap;\n\n  /** Creates a new FeatureSink instance. */\n  public FeatureSink(ImmutableSchemaInterface schema) {\n    this.schema = schema;\n    this.encodedFeatureMap = Maps.newHashMap();\n  }\n\n  private IntegerEncodedFeatures getFeatures(String baseFieldName) {\n    IntegerEncodedFeatures features = encodedFeatureMap.get(baseFieldName);\n    if (features == null) {\n      features = EarlybirdEncodedFeatures.newEncodedTweetFeatures(schema, baseFieldName);\n      encodedFeatureMap.put(baseFieldName, features);\n    }\n    return features;\n  }\n\n  /** Sets the given numeric value for the field. */\n  public FeatureSink setNumericValue(EarlybirdFieldConstant field, int value) {\n    return setNumericValue(field.getFieldName(), value);\n  }\n\n  /** Sets the given numeric value for the feature with the given name. */\n  public FeatureSink setNumericValue(String featureName, int value) {\n    final FeatureConfiguration featureConfig = schema.getFeatureConfigurationByName(featureName);\n    if (featureConfig != null) {\n      getFeatures(featureConfig.getBaseField()).setFeatureValue(featureConfig, value);\n    }\n    return this;\n  }\n\n  /** Sets the given boolean value for the given field. */\n  public FeatureSink setBooleanValue(EarlybirdFieldConstant field, boolean value) {\n    return setBooleanValue(field.getFieldName(), value);\n  }\n\n  /** Sets the given boolean value for the feature with the given name. */\n  public FeatureSink setBooleanValue(String featureName, boolean value) {\n    final FeatureConfiguration featureConfig = schema.getFeatureConfigurationByName(featureName);\n    if (featureConfig != null) {\n      getFeatures(featureConfig.getBaseField()).setFlagValue(featureConfig, value);\n    }\n    return this;\n  }\n\n  /** Returns the features for the given base field. */\n  public IntegerEncodedFeatures getFeaturesForBaseField(EarlybirdFieldConstant baseField) {\n    return getFeaturesForBaseField(baseField.getFieldName());\n  }\n\n  /** Returns the features for the given base field. */\n  public IntegerEncodedFeatures getFeaturesForBaseField(String baseFieldName) {\n    return encodedFeatureMap.get(baseFieldName);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/IntNormalizers.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport java.util.concurrent.TimeUnit;\n\nimport com.twitter.search.common.encoding.features.ByteNormalizer;\nimport com.twitter.search.common.encoding.features.IntNormalizer;\nimport com.twitter.search.common.encoding.features.PredictionScoreNormalizer;\n\n/**\n * Int value normalizers used to push feature values into earlybird db. For the\n * 8-bit feature types, this class wraps the\n * com.twitter.search.common.relevance.features.MutableFeatureNormalizers\n */\npublic final class IntNormalizers {\n  private IntNormalizers() {\n  }\n\n  public static final IntNormalizer LEGACY_NORMALIZER =\n      val -> ByteNormalizer.unsignedByteToInt(\n          MutableFeatureNormalizers.BYTE_NORMALIZER.normalize(val));\n\n  public static final IntNormalizer SMART_INTEGER_NORMALIZER =\n      val -> ByteNormalizer.unsignedByteToInt(\n          MutableFeatureNormalizers.SMART_INTEGER_NORMALIZER.normalize(val));\n\n  // The PARUS_SCORE feature is deprecated and is never set in our indexes. However, we still need\n  // this normalizer for now, because some models do not work properly with \"missing\" features, so\n  // for now we still need to set the PARUS_SCORE feature to 0.\n  public static final IntNormalizer PARUS_SCORE_NORMALIZER = val -> 0;\n\n  public static final IntNormalizer BOOLEAN_NORMALIZER =\n      val -> val == 0 ? 0 : 1;\n\n  public static final IntNormalizer TIMESTAMP_SEC_TO_HR_NORMALIZER =\n      val -> (int) TimeUnit.SECONDS.toHours((long) val);\n\n  public static final PredictionScoreNormalizer PREDICTION_SCORE_NORMALIZER =\n      new PredictionScoreNormalizer(3);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/MutableFeatureNormalizers.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport com.twitter.search.common.encoding.features.ByteNormalizer;\nimport com.twitter.search.common.encoding.features.SingleBytePositiveFloatNormalizer;\nimport com.twitter.search.common.encoding.features.SmartIntegerNormalizer;\n\n/**\n * Byte value normalizers used to push feature values into earlybird db.\n */\npublic abstract class MutableFeatureNormalizers {\n  // The max value we support in SMART_INTEGER_NORMALIZER below, this should be enough for all kinds\n  // of engagements we see on Twitter, anything larger than this would be represented as the same\n  // value (255, if using a byte).\n  private static final int MAX_COUNTER_VALUE_SUPPORTED = 50000000;\n\n  // Avoid using this normalizer for procesing any new data, always use SmartIntegerNormalizer\n  // below.\n  public static final SingleBytePositiveFloatNormalizer BYTE_NORMALIZER =\n      new SingleBytePositiveFloatNormalizer();\n\n  public static final ByteNormalizer SMART_INTEGER_NORMALIZER =\n      new SmartIntegerNormalizer(MAX_COUNTER_VALUE_SUPPORTED, 8);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/QueryFeatureType.java",
    "content": "package com.twitter.search.common.relevance.features;\n\n/**\n * An enum to hold different types of query-specific features (these are not indexed in Earlybird)\n */\npublic enum QueryFeatureType {\n  SOCIAL_ENGAGEMENTS,\n  CLICKS\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/RelevanceSignalConstants.java",
    "content": "package com.twitter.search.common.relevance.features;\n\n/**\n * Defines relevance related constants that are used at both ingestion time and\n * earlybird scoring time.\n */\npublic final class RelevanceSignalConstants {\n  // user reputation\n  public static final byte UNSET_REPUTATION_SENTINEL = Byte.MIN_VALUE;\n  public static final byte MAX_REPUTATION = 100;\n  public static final byte MIN_REPUTATION = 0;\n  // below overall CDF of ~10%, default value for new users,\n  // given as a goodwill value in case it is unset\n  public static final byte GOODWILL_REPUTATION = 17;\n\n  // text score\n  public static final byte UNSET_TEXT_SCORE_SENTINEL = Byte.MIN_VALUE;\n  // roughly at overall CDF of ~10%, given as a goodwill value in case it is unset\n  public static final byte GOODWILL_TEXT_SCORE = 19;\n\n  private RelevanceSignalConstants() {\n  }\n\n  // check whether the specified user rep value is valid\n  public static boolean isValidUserReputation(int userRep) {\n    return userRep != UNSET_REPUTATION_SENTINEL\n           && userRep >= MIN_REPUTATION\n           && userRep < MAX_REPUTATION;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/ScoringUtils.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport com.google.common.base.Preconditions;\n\n/**\n * Scoring utilities\n */\npublic final class ScoringUtils {\n  private ScoringUtils() { }\n\n  /**\n   * normalize a positive value of arbitrary range to [0.0, 1.0], with a slop\n   * @param value the value to normalize.\n   * @param halfval a reference value that will be normalized to 0.5\n   * @param exp an exponential parameter (must be positive) to control the converging speed,\n   * the smaller the value the faster it reaches the halfval but slower it reaches the maximum.\n   * @return a normalized value\n   */\n  public static float normalize(float value, double halfval, double exp) {\n    Preconditions.checkArgument(exp > 0.0 && exp <= 1.0);\n    return (float) (Math.pow(value, exp) / (Math.pow(value, exp) + Math.pow(halfval, exp)));\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/TermVector.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport java.util.Map;\n\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Maps;\n\nimport com.twitter.common.base.Function;\n\n/**\n * Class to keep String-Double of term vectors\n * It can calculate magnitude, dot product, and cosine similarity\n */\npublic class TermVector {\n  private static final double MIN_MAGNITUDE = 0.00001;\n  private final double magnitude;\n  private final ImmutableMap<String, Double> termWeights;\n\n  /** Creates a new TermVector instance. */\n  public TermVector(Map<String, Double> termWeights) {\n    this.termWeights = ImmutableMap.copyOf(termWeights);\n    double sum = 0.0;\n    for (Map.Entry<String, Double> entry : termWeights.entrySet()) {\n      double value = entry.getValue();\n      sum += value * value;\n    }\n    magnitude = Math.sqrt(sum);\n  }\n\n  public ImmutableMap<String, Double> getTermWeights() {\n    return termWeights;\n  }\n\n  public double getMagnitude() {\n    return magnitude;\n  }\n\n  /**\n   * Normalize term vector into unit magnitude\n   * @return           the unit normalized TermVector with magnitude equals 1\n   *                   return null if magnitude is very low\n   */\n  public TermVector getUnitNormalized() {\n    if (magnitude < MIN_MAGNITUDE) {\n      return null;\n    }\n    return new TermVector(\n        Maps.transformValues(termWeights, (Function<Double, Double>) weight -> weight / magnitude));\n  }\n\n  /**\n   * Calculate the dot product with another term vector\n   * @param other      the other term vector\n   * @return           the dot product of the two vectors\n   */\n  public double getDotProduct(TermVector other) {\n    double sum = 0.0;\n    for (Map.Entry<String, Double> entry : termWeights.entrySet()) {\n      Double value2 = other.termWeights.get(entry.getKey());\n      if (value2 != null) {\n        sum += entry.getValue() * value2;\n      }\n    }\n    return sum;\n  }\n\n  /**\n   * Calculate the cosine similarity of with another term vector\n   * @param other     the other term vector\n   * @return          the cosine similarity.\n   *                  if either has very small magnitude, it returns 0 (dotProduct close to 0)\n   */\n  public double getCosineSimilarity(TermVector other) {\n    if (magnitude < MIN_MAGNITUDE || other.magnitude < MIN_MAGNITUDE) {\n      return 0;\n    }\n    return getDotProduct(other) / (magnitude * other.magnitude);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/TweetEngagementFeatures.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport com.twitter.search.common.encoding.features.EncodedFeatures;\n\n/**\n * Holds engagement features for a particular tweet and encodes them as a single int.\n * The features are: retweet count, favorite count, itweet score, reply count.\n */\npublic class TweetEngagementFeatures extends EncodedFeatures {\n  private static final int RETWEET_COUNT_BIT_SHIFT = 0;\n  private static final long RETWEET_COUNT_INVERSE_BIT_MASK =  0xffffff00L;\n\n  private static final int ITWEET_SCORE_BIT_SHIFT = 8;\n  private static final long ITWEET_SCORE_INVERSE_BIT_MASK = 0xffff00ffL;\n\n  private static final int FAV_COUNT_BIT_SHIFT = 16;\n  private static final long FAV_COUNT_INVERSE_BIT_MASK =    0xff00ffffL;\n\n  private static final int REPLY_COUNT_BIT_SHIFT = 24;\n  private static final long REPLY_COUNT_INVERSE_BIT_MASK =    0x00ffffffL;\n\n  public TweetEngagementFeatures setRetweetCount(byte count) {\n    setByteIfGreater(count, RETWEET_COUNT_BIT_SHIFT, RETWEET_COUNT_INVERSE_BIT_MASK);\n    return this;\n  }\n\n  public int getRetweetCount() {\n    return getByte(RETWEET_COUNT_BIT_SHIFT);\n  }\n\n  public TweetEngagementFeatures setITweetScore(byte score) {\n    setByteIfGreater(score, ITWEET_SCORE_BIT_SHIFT, ITWEET_SCORE_INVERSE_BIT_MASK);\n    return this;\n  }\n\n  public int getITweetScore() {\n    return getByte(ITWEET_SCORE_BIT_SHIFT);\n  }\n\n  public TweetEngagementFeatures setFavCount(byte count) {\n    setByteIfGreater(count, FAV_COUNT_BIT_SHIFT, FAV_COUNT_INVERSE_BIT_MASK);\n    return this;\n  }\n\n  public int getFavCount() {\n    return getByte(FAV_COUNT_BIT_SHIFT);\n  }\n\n  public TweetEngagementFeatures setReplyCount(byte count) {\n    setByteIfGreater(count, REPLY_COUNT_BIT_SHIFT, REPLY_COUNT_INVERSE_BIT_MASK);\n    return this;\n  }\n\n  public int getReplyCount() {\n    return getByte(REPLY_COUNT_BIT_SHIFT);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/TweetFeatureType.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport java.util.Map;\nimport java.util.Set;\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.ImmutableSet;\n\nimport com.twitter.search.common.encoding.features.IntNormalizer;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\n\nimport static com.twitter.search.common.relevance.features.IntNormalizers.BOOLEAN_NORMALIZER;\nimport static com.twitter.search.common.relevance.features.IntNormalizers.LEGACY_NORMALIZER;\nimport static com.twitter.search.common.relevance.features.IntNormalizers.PARUS_SCORE_NORMALIZER;\nimport static com.twitter.search.common.relevance.features.IntNormalizers.SMART_INTEGER_NORMALIZER;\nimport static com.twitter.search.common.relevance.features.IntNormalizers.TIMESTAMP_SEC_TO_HR_NORMALIZER;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\n\n/**\n * An enum to represent all dynamic/realtime feature types we can update in the Signal Ingester.\n * It provides information for their normalization and their corresponding earlybird feature fields\n * and provides utils both producer (Signal Ingester) and consumer (Earlybird) side.\n *\n */\npublic enum TweetFeatureType {\n  RETWEET                         (true,  0,  LEGACY_NORMALIZER,\n      EarlybirdFieldConstant.RETWEET_COUNT),\n  REPLY                           (true,  1,  LEGACY_NORMALIZER,\n      EarlybirdFieldConstant.REPLY_COUNT),\n  FAVORITE                        (true,  4,  LEGACY_NORMALIZER,\n      EarlybirdFieldConstant.FAVORITE_COUNT),\n  PARUS_SCORE                     (false, 3,  PARUS_SCORE_NORMALIZER,\n      EarlybirdFieldConstant.PARUS_SCORE),\n  EMBEDS_IMP_COUNT                (true,  10, LEGACY_NORMALIZER,\n      EarlybirdFieldConstant.EMBEDS_IMPRESSION_COUNT),\n  EMBEDS_URL_COUNT                (true,  11, LEGACY_NORMALIZER,\n      EarlybirdFieldConstant.EMBEDS_URL_COUNT),\n  VIDEO_VIEW                      (false, 12, LEGACY_NORMALIZER,\n      EarlybirdFieldConstant.VIDEO_VIEW_COUNT),\n  // v2 engagement counters, they will eventually replace v1 counters above\n  RETWEET_V2                      (true,  13, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.RETWEET_COUNT_V2),\n  REPLY_V2                        (true,  14, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.REPLY_COUNT_V2),\n  FAVORITE_V2                     (true,  15, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.FAVORITE_COUNT_V2),\n  EMBEDS_IMP_COUNT_V2             (true,  16, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.EMBEDS_IMPRESSION_COUNT_V2),\n  EMBEDS_URL_COUNT_V2             (true,  17, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.EMBEDS_URL_COUNT_V2),\n  VIDEO_VIEW_V2                   (false, 18, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.VIDEO_VIEW_COUNT_V2),\n  // other new items\n  QUOTE                           (true,  19, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.QUOTE_COUNT),\n  // weighted engagement counters\n  WEIGHTED_RETWEET                (true,  20, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.WEIGHTED_RETWEET_COUNT),\n  WEIGHTED_REPLY                  (true,  21, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.WEIGHTED_REPLY_COUNT),\n  WEIGHTED_FAVORITE               (true,  22, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.WEIGHTED_FAVORITE_COUNT),\n  WEIGHTED_QUOTE                  (true,  23, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.WEIGHTED_QUOTE_COUNT),\n\n  // tweet-level safety labels\n  LABEL_ABUSIVE                   (false, 24, BOOLEAN_NORMALIZER,\n      EarlybirdFieldConstant.LABEL_ABUSIVE_FLAG),\n  LABEL_ABUSIVE_HI_RCL            (false, 25, BOOLEAN_NORMALIZER,\n      EarlybirdFieldConstant.LABEL_ABUSIVE_HI_RCL_FLAG),\n  LABEL_DUP_CONTENT               (false, 26, BOOLEAN_NORMALIZER,\n      EarlybirdFieldConstant.LABEL_DUP_CONTENT_FLAG),\n  LABEL_NSFW_HI_PRC               (false, 27, BOOLEAN_NORMALIZER,\n      EarlybirdFieldConstant.LABEL_NSFW_HI_PRC_FLAG),\n  LABEL_NSFW_HI_RCL               (false, 28, BOOLEAN_NORMALIZER,\n      EarlybirdFieldConstant.LABEL_NSFW_HI_RCL_FLAG),\n  LABEL_SPAM                      (false, 29, BOOLEAN_NORMALIZER,\n      EarlybirdFieldConstant.LABEL_SPAM_FLAG),\n  LABEL_SPAM_HI_RCL               (false, 30, BOOLEAN_NORMALIZER,\n      EarlybirdFieldConstant.LABEL_SPAM_HI_RCL_FLAG),\n\n  PERISCOPE_EXISTS                (false, 32, BOOLEAN_NORMALIZER,\n      EarlybirdFieldConstant.PERISCOPE_EXISTS),\n  PERISCOPE_HAS_BEEN_FEATURED     (false, 33, BOOLEAN_NORMALIZER,\n      EarlybirdFieldConstant.PERISCOPE_HAS_BEEN_FEATURED),\n  PERISCOPE_IS_CURRENTLY_FEATURED (false, 34, BOOLEAN_NORMALIZER,\n      EarlybirdFieldConstant.PERISCOPE_IS_CURRENTLY_FEATURED),\n  PERISCOPE_IS_FROM_QUALITY_SOURCE(false, 35, BOOLEAN_NORMALIZER,\n      EarlybirdFieldConstant.PERISCOPE_IS_FROM_QUALITY_SOURCE),\n  PERISCOPE_IS_LIVE               (false, 36, BOOLEAN_NORMALIZER,\n      EarlybirdFieldConstant.PERISCOPE_IS_LIVE),\n\n  // decayed engagement counters\n  DECAYED_RETWEET                 (true,  37, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.DECAYED_RETWEET_COUNT),\n  DECAYED_REPLY                   (true,  38, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.DECAYED_REPLY_COUNT),\n  DECAYED_FAVORITE                (true,  39, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.DECAYED_FAVORITE_COUNT),\n  DECAYED_QUOTE                   (true,  40, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.DECAYED_QUOTE_COUNT),\n\n  // timestamp of last engagement types\n  LAST_RETWEET_SINCE_CREATION_HR  (false, 41, TIMESTAMP_SEC_TO_HR_NORMALIZER,\n      EarlybirdFieldConstant.LAST_RETWEET_SINCE_CREATION_HRS),\n  LAST_REPLY_SINCE_CREATION_HR    (false, 42, TIMESTAMP_SEC_TO_HR_NORMALIZER,\n      EarlybirdFieldConstant.LAST_REPLY_SINCE_CREATION_HRS),\n  LAST_FAVORITE_SINCE_CREATION_HR (false, 43, TIMESTAMP_SEC_TO_HR_NORMALIZER,\n      EarlybirdFieldConstant.LAST_FAVORITE_SINCE_CREATION_HRS),\n  LAST_QUOTE_SINCE_CREATION_HR    (false, 44, TIMESTAMP_SEC_TO_HR_NORMALIZER,\n      EarlybirdFieldConstant.LAST_QUOTE_SINCE_CREATION_HRS),\n\n  // fake engagement counters\n  FAKE_RETWEET                    (true,  45, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.FAKE_RETWEET_COUNT),\n  FAKE_REPLY                      (true,  46, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.FAKE_REPLY_COUNT),\n  FAKE_FAVORITE                   (true,  47, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.FAKE_FAVORITE_COUNT),\n  FAKE_QUOTE                      (true,  48, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.FAKE_QUOTE_COUNT),\n\n  // blink engagement counters\n  BLINK_RETWEET                   (true,  49, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.BLINK_RETWEET_COUNT),\n  BLINK_REPLY                     (true,  50, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.BLINK_REPLY_COUNT),\n  BLINK_FAVORITE                  (true,  51, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.BLINK_FAVORITE_COUNT),\n  BLINK_QUOTE                     (true,  52, SMART_INTEGER_NORMALIZER,\n      EarlybirdFieldConstant.BLINK_QUOTE_COUNT),\n\n  /* semicolon in a single line to avoid polluting git blame */;\n\n  private static final Map<TweetFeatureType, TweetFeatureType> V2_COUNTER_MAP =\n      ImmutableMap.<TweetFeatureType, TweetFeatureType>builder()\n          .put(RETWEET,          RETWEET_V2)\n          .put(REPLY,            REPLY_V2)\n          .put(FAVORITE,         FAVORITE_V2)\n          .put(EMBEDS_IMP_COUNT, EMBEDS_IMP_COUNT_V2)\n          .put(EMBEDS_URL_COUNT, EMBEDS_URL_COUNT_V2)\n          .put(VIDEO_VIEW,       VIDEO_VIEW_V2)\n      .build();\n\n  private static final Map<TweetFeatureType, TweetFeatureType> WEIGHTED_COUNTER_MAP =\n      ImmutableMap.<TweetFeatureType, TweetFeatureType>builder()\n          .put(RETWEET,          WEIGHTED_RETWEET)\n          .put(REPLY,            WEIGHTED_REPLY)\n          .put(FAVORITE,         WEIGHTED_FAVORITE)\n          .put(QUOTE,            WEIGHTED_QUOTE)\n          .build();\n\n  private static final Map<TweetFeatureType, TweetFeatureType> DECAYED_COUNTER_MAP =\n      ImmutableMap.<TweetFeatureType, TweetFeatureType>builder()\n          .put(RETWEET,          DECAYED_RETWEET)\n          .put(REPLY,            DECAYED_REPLY)\n          .put(FAVORITE,         DECAYED_FAVORITE)\n          .put(QUOTE,            DECAYED_QUOTE)\n          .build();\n\n  private static final Map<TweetFeatureType, TweetFeatureType> DECAYED_COUNTER_TO_ELAPSED_TIME =\n      ImmutableMap.<TweetFeatureType, TweetFeatureType>builder()\n          .put(DECAYED_RETWEET,  LAST_RETWEET_SINCE_CREATION_HR)\n          .put(DECAYED_REPLY,    LAST_REPLY_SINCE_CREATION_HR)\n          .put(DECAYED_FAVORITE, LAST_FAVORITE_SINCE_CREATION_HR)\n          .put(DECAYED_QUOTE,    LAST_QUOTE_SINCE_CREATION_HR)\n          .build();\n\n  private static final Set<TweetFeatureType> DECAYED_FEATURES =\n      ImmutableSet.of(DECAYED_RETWEET, DECAYED_REPLY, DECAYED_FAVORITE, DECAYED_QUOTE);\n\n  private static final Set<TweetFeatureType> FAKE_ENGAGEMENT_FEATURES =\n      ImmutableSet.of(FAKE_RETWEET, FAKE_REPLY, FAKE_FAVORITE, FAKE_QUOTE);\n\n  private static final Set<TweetFeatureType> BLINK_ENGAGEMENT_FEATURES =\n      ImmutableSet.of(BLINK_RETWEET, BLINK_REPLY, BLINK_FAVORITE, BLINK_QUOTE);\n\n  @Nullable\n  public TweetFeatureType getV2Type() {\n    return V2_COUNTER_MAP.get(this);\n  }\n\n  @Nullable\n  public static TweetFeatureType getWeightedType(TweetFeatureType type) {\n    return WEIGHTED_COUNTER_MAP.get(type);\n  }\n\n  @Nullable\n  public static TweetFeatureType getDecayedType(TweetFeatureType type) {\n    return DECAYED_COUNTER_MAP.get(type);\n  }\n\n  // Whether this feature is incremental or direct value.\n  private final boolean incremental;\n\n  // This normalizer is used to (1) normalize the output value in DLIndexEventOutputBolt,\n  // (2) check value change.\n  private final IntNormalizer normalizer;\n\n  // value for composing cache key. It has to be unique and in increasing order.\n  private final int typeInt;\n\n  private final EarlybirdFieldConstants.EarlybirdFieldConstant earlybirdField;\n\n  private final IncrementChecker incrementChecker;\n\n  /**\n   * Constructing an enum for a type. The earlybirdField can be null if it's not prepared, they\n   * can be here as placeholders but they can't be outputted.\n   * The normalizer is null for the timestamp features that do not require normalization\n   */\n  TweetFeatureType(boolean incremental,\n                   int typeInt,\n                   IntNormalizer normalizer,\n                   @Nullable EarlybirdFieldConstant earlybirdField) {\n    this.incremental = incremental;\n    this.typeInt = typeInt;\n    this.normalizer = normalizer;\n    this.earlybirdField = earlybirdField;\n    this.incrementChecker = new IncrementChecker(this);\n  }\n\n  public boolean isIncremental() {\n    return incremental;\n  }\n\n  public IntNormalizer getNormalizer() {\n    return normalizer;\n  }\n\n  public int getTypeInt() {\n    return typeInt;\n  }\n\n  public int normalize(double value) {\n    return normalizer.normalize(value);\n  }\n\n  public IncrementChecker getIncrementChecker() {\n    return incrementChecker;\n  }\n\n  public EarlybirdFieldConstant getEarlybirdField() {\n    return Preconditions.checkNotNull(earlybirdField);\n  }\n\n  public boolean hasEarlybirdField() {\n    return earlybirdField != null;\n  }\n\n  public boolean isDecayed() {\n    return DECAYED_FEATURES.contains(this);\n  }\n\n  @Nullable\n  public TweetFeatureType getElapsedTimeFeatureType() {\n    return DECAYED_COUNTER_TO_ELAPSED_TIME.get(this);\n  }\n\n  public boolean isFakeEngagement() {\n    return FAKE_ENGAGEMENT_FEATURES.contains(this);\n  }\n\n  public boolean isBlinkEngagement() {\n    return BLINK_ENGAGEMENT_FEATURES.contains(this);\n  }\n\n  /**\n   * Check if an increment is eligible for emitting\n   */\n  public static class IncrementChecker {\n    private final IntNormalizer normalizer;\n\n    public IncrementChecker(IntNormalizer normalizer) {\n      this.normalizer = normalizer;\n    }\n\n    IncrementChecker(TweetFeatureType type) {\n      this(type.getNormalizer());\n    }\n\n    /**\n     * Check if a value change is eligible for output\n     */\n    public boolean eligibleForEmit(int oldValue, int newValue) {\n      return normalizer.normalize(oldValue) != normalizer.normalize(newValue);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/TweetFeatures.java",
    "content": "package com.twitter.search.common.relevance.features;\n\npublic class TweetFeatures {\n  private final TweetTextQuality tweetTextQuality = new TweetTextQuality();\n  private final TweetTextFeatures tweetTextFeatures = new TweetTextFeatures();\n  private final TweetUserFeatures tweetUserFeatures = new TweetUserFeatures();\n\n  public TweetTextFeatures getTweetTextFeatures() {\n    return tweetTextFeatures;\n  }\n\n  public TweetTextQuality getTweetTextQuality() {\n    return tweetTextQuality;\n  }\n\n  public TweetUserFeatures getTweetUserFeatures() {\n    return tweetUserFeatures;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/TweetIntegerShingleSignature.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport java.nio.ByteBuffer;\nimport java.util.Arrays;\n\nimport com.google.common.base.Preconditions;\n\n/**\n * A TweetIntegerShingleSignature object consists of 4 bytes, each representing the signature of\n * a status text sample. The signature bytes are sorted in ascending order and compacted to an\n * integer in big endian for serialization.\n *\n * Fuzzy matching of two TweetIntegerShingleSignature objects is met when the number of matching\n * bytes between the two is equal to or above 3.\n */\npublic class TweetIntegerShingleSignature {\n  public static final int NUM_SHINGLES = Integer.SIZE / Byte.SIZE;\n  public static final int DEFAULT_NO_SIGNATURE = 0;\n  public static final TweetIntegerShingleSignature NO_SIGNATURE_HANDLE =\n    deserialize(DEFAULT_NO_SIGNATURE);\n  public static final int DEFAULT_MIN_SHINGLES_MATCH = 3;\n  private final int minShinglesMatch;\n\n  private final byte[] shingles;\n  private final int signature;  // redundant information, for easier comparison.\n\n  /**\n   * Construct from a byte array.\n   */\n  public TweetIntegerShingleSignature(byte[] shingles, int minShinglesMatch) {\n    Preconditions.checkArgument(shingles.length == NUM_SHINGLES);\n    this.shingles = shingles;\n    // sort to byte's natural ascending order\n    Arrays.sort(this.shingles);\n    this.minShinglesMatch = minShinglesMatch;\n    this.signature = serializeInternal(shingles);\n  }\n\n  /**\n   * Construct from a byte array.\n   */\n  public TweetIntegerShingleSignature(byte[] shingles) {\n    this(shingles, DEFAULT_MIN_SHINGLES_MATCH);\n  }\n\n  /**\n   * Construct from a serialized integer signature.\n   */\n  public TweetIntegerShingleSignature(int signature, int minShinglesMatch) {\n    this.shingles = deserializeInternal(signature);\n    // sort to byte's natural ascending order\n    Arrays.sort(this.shingles);\n    this.minShinglesMatch = minShinglesMatch;\n    // now store the sorted shingles into signature field, may be different from what passed in.\n    this.signature = serializeInternal(shingles);\n  }\n\n  /**\n   * Construct from a serialized integer signature.\n   */\n  public TweetIntegerShingleSignature(int signature) {\n    this(signature, DEFAULT_MIN_SHINGLES_MATCH);\n  }\n\n  /**\n   * Used by ingester to generate signature.\n   * Raw signatures are in byte arrays per sample, and can be more or less\n   * than what is asked for.\n   *\n   * @param rawSignature\n   */\n  public TweetIntegerShingleSignature(Iterable<byte[]> rawSignature) {\n    byte[] condensedSignature = new byte[NUM_SHINGLES];\n    int i = 0;\n    for (byte[] signatureItem : rawSignature) {\n      condensedSignature[i++] = signatureItem[0];\n      if (i == NUM_SHINGLES) {\n        break;\n      }\n    }\n    this.shingles = condensedSignature;\n    Arrays.sort(this.shingles);\n    this.minShinglesMatch = DEFAULT_MIN_SHINGLES_MATCH;\n    this.signature = serializeInternal(shingles);\n  }\n\n  /**\n   * When used in a hashtable for dup detection, take the first byte of each signature for fast\n   * pass for majority case of no fuzzy matching. For top queries, this optimization losses about\n   * only 4% of all fuzzy matches.\n   *\n   * @return most significant byte of this signature as its hashcode.\n   */\n  @Override\n  public int hashCode() {\n    return shingles[0] & 0xFF;\n  }\n\n  /**\n   * Perform fuzzy matching between two TweetIntegerShingleSignature objects.\n   *\n   * @param other TweetIntegerShingleSignature object to perform fuzzy match against\n   * @return true if at least minMatch number of bytes match\n   */\n  @Override\n  public boolean equals(Object other) {\n    if (this == other) {\n      return true;\n    }\n    if (other == null) {\n      return false;\n    }\n    if (getClass() != other.getClass()) {\n      return false;\n    }\n\n    final TweetIntegerShingleSignature otherSignatureInteger = (TweetIntegerShingleSignature) other;\n\n    int otherSignature = otherSignatureInteger.serialize();\n    if (signature == otherSignature) {\n      // Both serialized signature is the same\n      return true;\n    } else if (signature != DEFAULT_NO_SIGNATURE && otherSignature != DEFAULT_NO_SIGNATURE) {\n      // Neither is NO_SIGNATURE, need to compare shingles.\n      byte[] otherShingles = otherSignatureInteger.getShingles();\n      int numberMatchesNeeded = minShinglesMatch;\n      // expect bytes are in ascending sorted order\n      int i = 0;\n      int j = 0;\n      while (((numberMatchesNeeded <= (NUM_SHINGLES - i)) // early termination for i\n              || (numberMatchesNeeded <= (NUM_SHINGLES - j))) // early termination j\n             && (i < NUM_SHINGLES) && (j < NUM_SHINGLES)) {\n        if (shingles[i] == otherShingles[j]) {\n          if (shingles[i] != 0) {  // we only consider two shingles equal if they are non zero\n            numberMatchesNeeded--;\n            if (numberMatchesNeeded == 0) {\n              return true;\n            }\n          }\n          i++;\n          j++;\n        } else if (shingles[i] < otherShingles[j]) {\n          i++;\n        } else if (shingles[i] > otherShingles[j]) {\n          j++;\n        }\n      }\n    }\n    // One is NO_SIGNATURE and one is not.\n    return false;\n  }\n\n  /**\n   * Returns the sorted array of signature bytes.\n   */\n  public byte[] getShingles() {\n    return shingles;\n  }\n\n  /**\n   * Serialize 4 sorted signature bytes into an integer in big endian order.\n   *\n   * @return compacted int signature\n   */\n  private static int serializeInternal(byte[] shingles) {\n    ByteBuffer byteBuffer = ByteBuffer.allocate(NUM_SHINGLES);\n    byteBuffer.put(shingles, 0, NUM_SHINGLES);\n    return byteBuffer.getInt(0);\n  }\n\n  /**\n   * Deserialize an integer into a 4-byte array.\n   * @param signature The signature integer.\n   * @return A byte array with 4 elements.\n   */\n  private static byte[] deserializeInternal(int signature) {\n    return ByteBuffer.allocate(NUM_SHINGLES).putInt(signature).array();\n  }\n\n  public int serialize() {\n    return signature;\n  }\n\n  public static boolean isFuzzyMatch(int signature1, int signature2) {\n    return TweetIntegerShingleSignature.deserialize(signature1).equals(\n        TweetIntegerShingleSignature.deserialize(signature2));\n  }\n\n  public static TweetIntegerShingleSignature deserialize(int signature) {\n    return new TweetIntegerShingleSignature(signature);\n  }\n\n  public static TweetIntegerShingleSignature deserialize(int signature, int minMatchSingles) {\n    return new TweetIntegerShingleSignature(signature, minMatchSingles);\n  }\n\n  @Override\n  public String toString() {\n    return String.format(\"%d %d %d %d\", shingles[0], shingles[1], shingles[2], shingles[3]);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/TweetSignatureUtil.java",
    "content": "package com.twitter.search.common.relevance.features;\n\npublic final class TweetSignatureUtil {\n  private TweetSignatureUtil() {\n  }\n\n  /** Converts the signature in args[0] to a TweetIntegerShingleSignature. */\n  public static void main(String[] args) throws Exception {\n    if (args.length < 1) {\n      throw new RuntimeException(\"Please provide signature value.\");\n    }\n    int signature = Integer.parseInt(args[0]);\n    System.out.println(TweetIntegerShingleSignature.deserialize(signature).toString());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/TweetTextFeatures.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport java.util.Collection;\nimport java.util.List;\nimport java.util.Set;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.Sets;\n\nimport com.twitter.common.text.token.TokenizedCharSequence;\n\npublic class TweetTextFeatures {\n  // Basic Features, always extracted.\n  // normalized, lower cased tweet text, w/o resolved urls\n  private String normalizedText;\n\n  // tokens from normalizedText, w/o resolved urls, lower cased.\n  private List<String> tokens;\n\n  // tokens from resolved urls, lower cased.\n  private List<String> resolvedUrlsTokens;\n\n  // tokens in the form of a TokenizedCharSeq, NOT LOWER CASED\n  private TokenizedCharSequence tokenSequence;\n\n  // strippedTokens above joined with space\n  private String normalizedStrippedText;\n\n  // normalized, original case tokens, without @mention, #hashtag or urls.\n  private List<String> strippedTokens;\n\n  // all hash tags, without \"#\", lower cased\n  private Set<String> hashtags = Sets.newHashSet();\n\n  // all mentions, without \"@\", lower cased\n  private Set<String> mentions = Sets.newHashSet();\n\n  // whether this tweet has a question mark that's not in url.\n  private boolean hasQuestionMark = false;\n\n  private boolean hasPositiveSmiley = false;\n  private boolean hasNegativeSmiley = false;\n\n  // normalized, original case smileys\n  private List<String> smileys;\n\n  // lower cased, normalized stock names, without \"$\"\n  private List<String> stocks;\n\n  // Extra features for text quality evaluation only.\n  private int signature = TweetIntegerShingleSignature.DEFAULT_NO_SIGNATURE;\n  private Set<String> trendingTerms = Sets.newHashSet();\n  private int length;\n  private int caps;\n\n  public String getNormalizedText() {\n    return normalizedText;\n  }\n\n  public void setNormalizedText(String normalizedText) {\n    this.normalizedText = normalizedText;\n  }\n\n  public List<String> getTokens() {\n    return tokens;\n  }\n\n  public int getTokensSize() {\n    return tokens == null ? 0 : tokens.size();\n  }\n\n  public void setTokens(List<String> tokens) {\n    this.tokens = tokens;\n  }\n\n  public List<String> getResolvedUrlTokens() {\n    return resolvedUrlsTokens;\n  }\n\n  public int getResolvedUrlTokensSize() {\n    return resolvedUrlsTokens == null ? 0 : resolvedUrlsTokens.size();\n  }\n\n  public void setResolvedUrlTokens(List<String> tokensResolvedUrls) {\n    this.resolvedUrlsTokens = tokensResolvedUrls;\n  }\n\n  public TokenizedCharSequence getTokenSequence() {\n    return tokenSequence;\n  }\n\n  public void setTokenSequence(TokenizedCharSequence tokenSequence) {\n    this.tokenSequence = tokenSequence;\n  }\n\n  public String getNormalizedStrippedText() {\n    return normalizedStrippedText;\n  }\n\n  public void setNormalizedStrippedText(String normalizedStrippedText) {\n    this.normalizedStrippedText = normalizedStrippedText;\n  }\n\n  public List<String> getStrippedTokens() {\n    return strippedTokens;\n  }\n\n  public int getStrippedTokensSize() {\n    return strippedTokens == null ? 0 : strippedTokens.size();\n  }\n\n  public void setStrippedTokens(List<String> strippedTokens) {\n    this.strippedTokens = strippedTokens;\n  }\n\n  public Set<String> getHashtags() {\n    return hashtags;\n  }\n\n  public int getHashtagsSize() {\n    return hashtags.size();\n  }\n\n  public void setHashtags(Collection<String> hashtags) {\n    this.hashtags = Sets.newHashSet(hashtags);\n  }\n\n  public Set<String> getMentions() {\n    return mentions;\n  }\n\n  public int getMentionsSize() {\n    return mentions.size();\n  }\n\n  public void setMentions(Collection<String> mentions) {\n    this.mentions = Sets.newHashSet(mentions);\n  }\n\n  public boolean hasQuestionMark() {\n    return hasQuestionMark;\n  }\n\n  public void setHasQuestionMark(boolean hasQuestionMark) {\n    this.hasQuestionMark = hasQuestionMark;\n  }\n\n  public boolean hasPositiveSmiley() {\n    return hasPositiveSmiley;\n  }\n\n  public void setHasPositiveSmiley(boolean hasPositiveSmiley) {\n    this.hasPositiveSmiley = hasPositiveSmiley;\n  }\n\n  public boolean hasNegativeSmiley() {\n    return hasNegativeSmiley;\n  }\n\n  public void setHasNegativeSmiley(boolean hasNegativeSmiley) {\n    this.hasNegativeSmiley = hasNegativeSmiley;\n  }\n\n  public List<String> getSmileys() {\n    return smileys;\n  }\n\n  public int getSmileysSize() {\n    return smileys == null ? 0 : smileys.size();\n  }\n\n  public void setSmileys(List<String> smileys) {\n    this.smileys = smileys;\n  }\n\n  public List<String> getStocks() {\n    return stocks;\n  }\n\n  public int getStocksSize() {\n    return stocks == null ? 0 : stocks.size();\n  }\n\n  public void setStocks(List<String> stocks) {\n    this.stocks = stocks;\n  }\n\n  public int getSignature() {\n    return signature;\n  }\n\n  public void setSignature(int signature) {\n    this.signature = signature;\n  }\n\n  /** Returns the trending terms. */\n  public Set<String> getTrendingTerms() {\n    return trendingTerms;\n  }\n\n  public int getTrendingTermsSize() {\n    return trendingTerms.size();\n  }\n\n  @VisibleForTesting\n  public void setTrendingTerms(Set<String> trendingTerms) {\n    this.trendingTerms = trendingTerms;\n  }\n\n  public int getLength() {\n    return length;\n  }\n\n  public void setLength(int length) {\n    this.length = length;\n  }\n\n  public int getCaps() {\n    return caps;\n  }\n\n  public void setCaps(int caps) {\n    this.caps = caps;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/TweetTextQuality.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport java.util.Set;\n\nimport com.google.common.collect.Sets;\n\npublic class TweetTextQuality {\n\n  public static enum BooleanQualityType {\n    OFFENSIVE,          // tweet text is offensive\n    OFFENSIVE_USER,     // user name is offensive\n    HASHTAG_NAME_MATCH,  // hashtag matches username\n    SENSITIVE,           // tweet is marked as sensitive when it comes in\n  }\n\n  public static final double ENTROPY_NOT_SET = Double.MIN_VALUE;\n\n  public static final byte UNSET_TEXT_SCORE = -128;\n\n  private double readability;\n  private double shout;\n  private double entropy = ENTROPY_NOT_SET;\n  private final Set<BooleanQualityType> boolQualities = Sets.newHashSet();\n  private byte textScore = UNSET_TEXT_SCORE;\n\n  public double getReadability() {\n    return readability;\n  }\n\n  public void setReadability(double readability) {\n    this.readability = readability;\n  }\n\n  public double getShout() {\n    return shout;\n  }\n\n  public void setShout(double shout) {\n    this.shout = shout;\n  }\n\n  public double getEntropy() {\n    return entropy;\n  }\n\n  public void setEntropy(double entropy) {\n    this.entropy = entropy;\n  }\n\n  public void addBoolQuality(BooleanQualityType type) {\n    boolQualities.add(type);\n  }\n\n  public boolean hasBoolQuality(BooleanQualityType type) {\n    return boolQualities.contains(type);\n  }\n\n  public Set<BooleanQualityType> getBoolQualities() {\n    return boolQualities;\n  }\n\n  public byte getTextScore() {\n    return textScore;\n  }\n\n  public void setTextScore(byte textScore) {\n    this.textScore = textScore;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/features/TweetUserFeatures.java",
    "content": "package com.twitter.search.common.relevance.features;\n\nimport java.util.Map;\n\npublic class TweetUserFeatures {\n  private String lang;\n  private double langConfidence;\n  private int followers;\n  private int following;\n  private int reputation;\n  private int tweets;\n  private int retweets;\n  private int retweeted;\n  private Map<String, Double> knownForTopics;\n  private boolean isSpam;\n  private boolean isNsfw;\n  private boolean isBot;\n\n  public String getLang() {\n    return lang;\n  }\n\n  public void setLang(String lang) {\n    this.lang = lang;\n  }\n\n  public double getLangConfidence() {\n    return langConfidence;\n  }\n\n  public void setLangConfidence(double langConfidence) {\n    this.langConfidence = langConfidence;\n  }\n\n  public int getFollowers() {\n    return followers;\n  }\n\n  public void setFollowers(int followers) {\n    this.followers = followers;\n  }\n\n  public int getFollowing() {\n    return following;\n  }\n\n  public void setFollowing(int following) {\n    this.following = following;\n  }\n\n  public int getReputation() {\n    return reputation;\n  }\n\n  public void setReputation(int reputation) {\n    this.reputation = reputation;\n  }\n\n  public int getTweets() {\n    return tweets;\n  }\n\n  public void setTweets(int tweets) {\n    this.tweets = tweets;\n  }\n\n  public int getRetweets() {\n    return retweets;\n  }\n\n  public void setRetweets(int retweets) {\n    this.retweets = retweets;\n  }\n\n  public int getRetweeted() {\n    return retweeted;\n  }\n\n  public void setRetweeted(int retweeted) {\n    this.retweeted = retweeted;\n  }\n\n  public Map<String, Double> getKnownForTopics() {\n    return knownForTopics;\n  }\n\n  public void setKnownForTopics(Map<String, Double> knownForTopics) {\n    this.knownForTopics = knownForTopics;\n  }\n\n  public boolean isSpam() {\n    return isSpam;\n  }\n\n  public void setSpam(boolean spam) {\n    isSpam = spam;\n  }\n\n  public boolean isNsfw() {\n    return isNsfw;\n  }\n\n  public void setNsfw(boolean nsfw) {\n    isNsfw = nsfw;\n  }\n\n  public boolean isBot() {\n    return isBot;\n  }\n\n  public void setBot(boolean bot) {\n    isBot = bot;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/scorers/TweetScorer.java",
    "content": "package com.twitter.search.common.relevance.scorers;\n\nimport com.twitter.search.common.relevance.classifiers.TweetClassifier;\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\n\n/**\n * Interface to compute feature scores for a single @TwitterMessage\n * object, or a group of them, after they have been processed by\n * feature classifiers.\n *\n * Intentionally kept Scorers separate from Classifiers, since they\n * may be run at different stages and in different batching manners.\n * Convenience methods are provided to run classification and scoring\n * in one call.\n */\npublic abstract class TweetScorer {\n  /**\n   * Compute and store feature score in TwitterMessage based on its\n   * TweetFeatures.\n   *\n   * @param tweet tweet message to compute and store score to.\n   */\n  public abstract void scoreTweet(final TwitterMessage tweet);\n\n  /**\n   * Score a group of TwitterMessages based on their corresponding TweetFeatures\n   * and store feature scores in TwitterMessages.\n   *\n   * This default implementation just iterates through the map and scores each\n   * individual tweet. Batching for better performance, if applicable, can be implemented by\n   * concrete subclasses.\n   *\n   * @param tweets TwitterMessages to score.\n   */\n  public void scoreTweets(Iterable<TwitterMessage> tweets) {\n    for (TwitterMessage tweet: tweets) {\n      scoreTweet(tweet);\n    }\n  }\n\n  /**\n   * Convenience method.\n   * Classify tweet using the specified list of classifiers, then compute score.\n   *\n   * @param classifier list of classifiers to use for classification.\n   * @param tweet tweet to classify and score\n   */\n  public void classifyAndScoreTweet(TweetClassifier classifier, TwitterMessage tweet) {\n    classifier.classifyTweet(tweet);\n    scoreTweet(tweet);\n  }\n\n  /**\n   * Convenience method.\n   * Classify tweets using the specified list of classifiers, then compute score.\n   *\n   * @param classifier classifier to use for classification.\n   * @param tweets tweets to classify and score\n   */\n  public void classifyAndScoreTweets(TweetClassifier classifier, Iterable<TwitterMessage> tweets) {\n    for (TwitterMessage tweet: tweets) {\n      classifyAndScoreTweet(classifier, tweet);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/scorers/TweetTextScorer.java",
    "content": "package com.twitter.search.common.relevance.scorers;\n\nimport java.util.Map;\nimport java.util.concurrent.ConcurrentMap;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.search.common.metrics.RelevanceStats;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.relevance.config.TweetProcessingConfig;\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\nimport com.twitter.search.common.relevance.features.TweetFeatures;\nimport com.twitter.search.common.relevance.features.TweetTextFeatures;\nimport com.twitter.search.common.relevance.features.TweetTextQuality;\n\n/**\n * Compute a text score for TwitterMessage based on its offensiveness,\n * shoutness, length, readability and hashtag properties extracted from\n * tweet text.\n * <p/>\n * Formula:\n * text_score = offensive_text_damping * offensive_username_damping *\n * Sigma(feature_score_weight * feature_score)\n * <p/>\n * scored features are: length, readability, shout, entropy, links\n */\npublic class TweetTextScorer extends TweetScorer {\n  private static final Logger LOG = LoggerFactory.getLogger(TweetTextScorer.class);\n\n  private static final double DEFAULT_OFFENSIVE_TERM_DAMPING = 0.2d;\n  private static final double DEFAULT_OFFENSIVE_NAME_DAMPING = 0.2d;\n\n  // Sigma of all weights = 1.0d\n  private static final double DEFAULT_LENGTH_WEIGHT = 0.5d;\n  private static final double DEFAULT_READABILITY_WEIGHT = 0.1d;\n  private static final double DEFAULT_SHOUT_WEIGHT = 0.1d;\n  private static final double DEFAULT_ENTROPY_WEIGHT = 0.25d;\n  private static final double DEFAULT_LINK_WEIGHT = 0.05d;\n\n  private static final double DEFAULT_NO_DAMPING = 1.0d;\n\n  // Sigmoid alpha values for normalization\n  private static final double DEFAULT_READABILITY_ALPHA = 0.05d;\n  private static final double DEFAULT_ENTROPY_ALPHA = 0.5d;\n  private static final double DEFAULT_LENGTH_ALPHA = 0.03d;\n\n  private static final ConcurrentMap<String, SearchRateCounter> RATE_COUNTERS =\n      Maps.newConcurrentMap();\n  private static final ConcurrentMap<PenguinVersion, Map<Integer, SearchRateCounter>>\n      SCORE_HISTOGRAMS = Maps.newConcurrentMap();\n\n  private double offensiveTermDamping = DEFAULT_OFFENSIVE_TERM_DAMPING;\n  private double offensiveNameDamping = DEFAULT_OFFENSIVE_NAME_DAMPING;\n\n  private double lengthWeight = DEFAULT_LENGTH_WEIGHT;\n  private double readabilityWeight = DEFAULT_READABILITY_WEIGHT;\n  private double shoutWeight = DEFAULT_SHOUT_WEIGHT;\n  private double entropyWeight = DEFAULT_ENTROPY_WEIGHT;\n  private double linkWeight = DEFAULT_LINK_WEIGHT;\n\n  private double readabilityAlpha = DEFAULT_READABILITY_ALPHA;\n  private double entropyAlpha = DEFAULT_ENTROPY_ALPHA;\n  private double lengthAlpha = DEFAULT_LENGTH_ALPHA;\n\n  /** Configure from a config file, validate the configuration. */\n  public TweetTextScorer(String configFile) {\n    TweetProcessingConfig.init(configFile);\n\n    // get dampings\n    checkWeightRange(offensiveTermDamping = TweetProcessingConfig\n        .getDouble(\"offensive_term_damping\", DEFAULT_OFFENSIVE_TERM_DAMPING));\n    checkWeightRange(offensiveNameDamping = TweetProcessingConfig\n        .getDouble(\"offensive_name_damping\", DEFAULT_OFFENSIVE_NAME_DAMPING));\n\n    // get weights\n    checkWeightRange(lengthWeight = TweetProcessingConfig\n        .getDouble(\"length_weight\", DEFAULT_LENGTH_WEIGHT));\n    checkWeightRange(readabilityWeight = TweetProcessingConfig\n        .getDouble(\"readability_weight\", DEFAULT_READABILITY_WEIGHT));\n    checkWeightRange(shoutWeight = TweetProcessingConfig\n        .getDouble(\"shout_weight\", DEFAULT_SHOUT_WEIGHT));\n    checkWeightRange(entropyWeight = TweetProcessingConfig\n        .getDouble(\"entropy_weight\", DEFAULT_ENTROPY_WEIGHT));\n    checkWeightRange(linkWeight = TweetProcessingConfig\n        .getDouble(\"link_weight\", DEFAULT_LINK_WEIGHT));\n\n    // check sigma of weights\n    Preconditions.checkArgument(\n        lengthWeight + readabilityWeight + shoutWeight + entropyWeight + linkWeight == 1.0d);\n\n    readabilityAlpha = TweetProcessingConfig\n        .getDouble(\"readability_alpha\", DEFAULT_READABILITY_ALPHA);\n    entropyAlpha = TweetProcessingConfig.getDouble(\"entropy_alpha\", DEFAULT_ENTROPY_ALPHA);\n    lengthAlpha = TweetProcessingConfig.getDouble(\"length_alpha\", DEFAULT_LENGTH_ALPHA);\n  }\n\n  /** Creates a new TweetTextScorer instance. */\n  public TweetTextScorer() {\n  }\n\n  /** Scores the given tweet. */\n  public void scoreTweet(final TwitterMessage tweet) {\n    Preconditions.checkNotNull(tweet);\n\n    for (PenguinVersion penguinVersion : tweet.getSupportedPenguinVersions()) {\n      TweetFeatures features = Preconditions.checkNotNull(tweet.getTweetFeatures(penguinVersion));\n      TweetTextFeatures textFeatures = Preconditions.checkNotNull(features.getTweetTextFeatures());\n      TweetTextQuality textQuality = Preconditions.checkNotNull(features.getTweetTextQuality());\n      boolean isOffensiveText = textQuality.hasBoolQuality(\n          TweetTextQuality.BooleanQualityType.OFFENSIVE);\n      boolean isOffensiveScreenName = textQuality.hasBoolQuality(\n          TweetTextQuality.BooleanQualityType.OFFENSIVE_USER);\n      double shoutScore = DEFAULT_NO_DAMPING - textQuality.getShout();\n      double lengthScore = normalize(textFeatures.getLength(), lengthAlpha);\n      double readabilityScore = normalize(textQuality.getReadability(), readabilityAlpha);\n      double entropyScore = normalize(textQuality.getEntropy(), entropyAlpha);\n\n      double score = (isOffensiveText ? offensiveTermDamping : DEFAULT_NO_DAMPING)\n        * (isOffensiveScreenName ? offensiveNameDamping : DEFAULT_NO_DAMPING)\n        * (lengthWeight * lengthScore\n           + readabilityWeight * readabilityScore\n           + shoutWeight * shoutScore\n           + entropyWeight * entropyScore\n           + linkWeight * (tweet.getExpandedUrlMapSize() > 0 ? 1 : 0));\n\n      // scale to [0, 100] byte\n      textQuality.setTextScore((byte) (score * 100));\n\n      updateStats(\n          isOffensiveText,\n          isOffensiveScreenName,\n          textFeatures,\n          score,\n          getRateCounterStat(\"num_offensive_text_\", penguinVersion),\n          getRateCounterStat(\"num_offensive_user_\", penguinVersion),\n          getRateCounterStat(\"num_no_trends_\", penguinVersion),\n          getRateCounterStat(\"num_has_trends_\", penguinVersion),\n          getRateCounterStat(\"num_too_many_trends_\", penguinVersion),\n          getRateCounterStat(\"num_scored_tweets_\", penguinVersion),\n          getScoreHistogram(penguinVersion));\n\n      if (LOG.isDebugEnabled()) {\n        LOG.debug(String.format(\n            \"Tweet length [%.2f] weighted length [%.2f], readability [%.2f] \"\n            + \"weighted readability [%.2f], shout [%.2f] weighted shout [%.2f], \"\n            + \"entropy [%.2f], weighted entropy [%.2f], \"\n            + \"score [%.2f], text [%s], penguin version [%s]\",\n            lengthScore,\n            lengthWeight * lengthScore,\n            readabilityScore,\n            readabilityWeight * readabilityScore,\n            shoutScore,\n            shoutWeight * shoutScore,\n            entropyScore,\n            entropyWeight * entropyScore,\n            score,\n            tweet.getText(),\n            penguinVersion));\n      }\n    }\n  }\n\n  private void updateStats(boolean isOffensiveText,\n                           boolean isOffensiveScreenName,\n                           TweetTextFeatures textFeatures,\n                           double score,\n                           SearchRateCounter offensiveTextCounter,\n                           SearchRateCounter offensiveUserNameCounter,\n                           SearchRateCounter noTrendsCounter,\n                           SearchRateCounter hasTrendsCounter,\n                           SearchRateCounter tooManyTrendsHashtagsCounter,\n                           SearchRateCounter scoredTweets,\n                           Map<Integer, SearchRateCounter> scoreHistogram) {\n    // set stats\n    if (isOffensiveText) {\n      offensiveTextCounter.increment();\n    }\n    if (isOffensiveScreenName) {\n      offensiveUserNameCounter.increment();\n    }\n    if (textFeatures.getTrendingTermsSize() == 0) {\n      noTrendsCounter.increment();\n    } else {\n      hasTrendsCounter.increment();\n    }\n    if (TwitterMessage.hasMultipleHashtagsOrTrends(textFeatures)) {\n      tooManyTrendsHashtagsCounter.increment();\n    }\n    scoredTweets.increment();\n\n    int bucket = (int) Math.floor(score * 10) * 10;\n    scoreHistogram.get(bucket).increment();\n  }\n\n  // normalize the passed in value to smoothed [0, 1.0d] range\n  private static double normalize(double value, double alpha) {\n    return 2 * (1.0d / (1.0d + Math.exp(-(alpha * value))) - 0.5);\n  }\n\n  // Make sure weight values are within the range of [0.0, 1.0]\n  private void checkWeightRange(double value) {\n    Preconditions.checkArgument(value >= 0.0d && value <= 1.0d);\n  }\n\n  private Map<Integer, SearchRateCounter> getScoreHistogram(PenguinVersion penguinVersion) {\n    Map<Integer, SearchRateCounter> scoreHistogram = SCORE_HISTOGRAMS.get(penguinVersion);\n    if (scoreHistogram == null) {\n      scoreHistogram = Maps.newHashMap();\n      String statsName = \"num_text_score_%d_%s\";\n\n      for (int i = 0; i <= 100; i += 10) {\n        scoreHistogram.put(i, RelevanceStats.exportRate(\n                               String.format(statsName, i, penguinVersion.name().toLowerCase())));\n      }\n\n      scoreHistogram = SCORE_HISTOGRAMS.putIfAbsent(penguinVersion, scoreHistogram);\n      if (scoreHistogram == null) {\n        scoreHistogram = SCORE_HISTOGRAMS.get(penguinVersion);\n      }\n    }\n\n    return scoreHistogram;\n  }\n\n  private SearchRateCounter getRateCounterStat(String statPrefix, PenguinVersion penguinVersion) {\n    String statName = statPrefix + penguinVersion.name().toLowerCase();\n    SearchRateCounter rateCounter = RATE_COUNTERS.get(statName);\n    if (rateCounter == null) {\n      // Only one RateCounter instance is created for each stat name. So we don't need to worry\n      // that another thread might've created this instance in the meantime: we can just create/get\n      // it, and store it in the map.\n      rateCounter = RelevanceStats.exportRate(statName);\n      RATE_COUNTERS.put(statName, rateCounter);\n    }\n    return rateCounter;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/text/LocationUtils.java",
    "content": "package com.twitter.search.common.relevance.text;\n\nimport java.util.regex.Matcher;\n\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\nimport com.twitter.search.common.util.text.regex.Regex;\n\npublic final class LocationUtils {\n  private LocationUtils() {\n  }\n\n  /**\n   * Extract lat/lon information from a twitter message.\n   * @param message The twitter message.\n   * @return A two-element double array for the lat/lon information.\n   */\n  public static double[] extractLatLon(TwitterMessage message) {\n    // first look in text for L:, then fall back to profile\n    Matcher loc = Regex.LAT_LON_LOC_PATTERN.matcher(message.getText());\n    if (loc.find() || message.getOrigLocation() != null\n        && (loc = Regex.LAT_LON_PATTERN.matcher(message.getOrigLocation())).find()) {\n      final double lat = Double.parseDouble(loc.group(2));\n      final double lon = Double.parseDouble(loc.group(3));\n\n      if (Math.abs(lat) > 90.0) {\n        throw new NumberFormatException(\"Latitude cannot exceed +-90 degrees: \" + lat);\n      }\n      if (Math.abs(lon) > 180.0) {\n        throw new NumberFormatException(\"Longitude cannot exceed +-180 degrees: \" + lon);\n      }\n\n      // Reject these common \"bogus\" regions.\n      if ((lat == 0 && lon == 0) || lat == -1 || lon == -1) {\n        return null;\n      }\n\n      return new double[]{lat, lon};\n    }\n    return null;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/text/TweetParser.java",
    "content": "package com.twitter.search.common.relevance.text;\n\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Locale;\nimport java.util.Set;\n\nimport com.google.common.base.Joiner;\nimport com.google.common.collect.Sets;\n\nimport com.twitter.common.text.util.CharSequenceUtils;\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.search.common.indexing.thriftjava.ThriftExpandedUrl;\nimport com.twitter.search.common.relevance.entities.TwitterMessage;\nimport com.twitter.search.common.relevance.features.TweetTextFeatures;\nimport com.twitter.search.common.util.text.NormalizerHelper;\nimport com.twitter.search.common.util.text.Smileys;\nimport com.twitter.search.common.util.text.TokenizerHelper;\nimport com.twitter.search.common.util.text.TokenizerResult;\n\n/**\n * A parser to extract very basic information from a tweet.\n */\npublic class TweetParser {\n  private static final boolean DO_NOT_REMOVE_WWW = false;\n\n  /** Parses the given TwitterMessage. */\n  public void parseTweet(TwitterMessage message) {\n    parseTweet(message, false, true);\n  }\n\n  /** Parses the given TwitterMessage. */\n  public void parseTweet(TwitterMessage message,\n                         boolean useEntitiesFromTweetText,\n                         boolean parseUrls) {\n    for (PenguinVersion penguinVersion : message.getSupportedPenguinVersions()) {\n      parseTweet(message, useEntitiesFromTweetText, parseUrls, penguinVersion);\n    }\n  }\n\n  /** Parses the given TwitterMessage. */\n  public void parseTweet(TwitterMessage message,\n                         boolean useEntitiesFromTweetText,\n                         boolean parseUrls,\n                         PenguinVersion penguinVersion) {\n    TweetTextFeatures textFeatures = message.getTweetTextFeatures(penguinVersion);\n    String rawText = message.getText();\n    Locale locale = message.getLocale();\n\n    // don't lower case first.\n    String normalizedText = NormalizerHelper.normalizeKeepCase(rawText, locale, penguinVersion);\n    String lowercasedNormalizedText =\n      CharSequenceUtils.toLowerCase(normalizedText, locale).toString();\n\n    textFeatures.setNormalizedText(lowercasedNormalizedText);\n\n    TokenizerResult result = TokenizerHelper.tokenizeTweet(normalizedText, locale, penguinVersion);\n    List<String> tokens = new ArrayList<>(result.tokens);\n    textFeatures.setTokens(tokens);\n    textFeatures.setTokenSequence(result.tokenSequence);\n\n    if (parseUrls) {\n      parseUrls(message, textFeatures);\n    }\n\n    textFeatures.setStrippedTokens(result.strippedDownTokens);\n    textFeatures.setNormalizedStrippedText(Joiner.on(\" \").skipNulls()\n                                                 .join(result.strippedDownTokens));\n\n    // Sanity checks, make sure there is no null token list.\n    if (textFeatures.getTokens() == null) {\n      textFeatures.setTokens(Collections.<String>emptyList());\n    }\n    if (textFeatures.getResolvedUrlTokens() == null) {\n      textFeatures.setResolvedUrlTokens(Collections.<String>emptyList());\n    }\n    if (textFeatures.getStrippedTokens() == null) {\n      textFeatures.setStrippedTokens(Collections.<String>emptyList());\n    }\n\n    setHashtagsAndMentions(message, textFeatures, penguinVersion);\n    textFeatures.setStocks(sanitizeTokenizerResults(result.stocks, '$'));\n    textFeatures.setHasQuestionMark(findQuestionMark(textFeatures));\n\n    // Set smiley polarities.\n    textFeatures.setSmileys(result.smileys);\n    for (String smiley : textFeatures.getSmileys()) {\n      if (Smileys.isValidSmiley(smiley)) {\n        boolean polarity = Smileys.getPolarity(smiley);\n        if (polarity) {\n          textFeatures.setHasPositiveSmiley(true);\n        } else {\n          textFeatures.setHasNegativeSmiley(true);\n        }\n      }\n    }\n    message.setTokenizedCharSequence(penguinVersion, result.rawSequence);\n\n    if (useEntitiesFromTweetText) {\n      takeEntities(message, textFeatures, result, penguinVersion);\n    }\n  }\n\n  /** Parse the URLs in the given TwitterMessage. */\n  public void parseUrls(TwitterMessage message) {\n    for (PenguinVersion penguinVersion : message.getSupportedPenguinVersions()) {\n      parseUrls(message, message.getTweetTextFeatures(penguinVersion));\n    }\n  }\n\n  /** Parse the URLs in the given TwitterMessage. */\n  public void parseUrls(TwitterMessage message, TweetTextFeatures textFeatures) {\n    if (message.getExpandedUrlMap() != null) {\n      Set<String> urlsToTokenize = Sets.newLinkedHashSet();\n      for (ThriftExpandedUrl url : message.getExpandedUrlMap().values()) {\n        if (url.isSetExpandedUrl()) {\n          urlsToTokenize.add(url.getExpandedUrl());\n        }\n        if (url.isSetCanonicalLastHopUrl()) {\n          urlsToTokenize.add(url.getCanonicalLastHopUrl());\n        }\n      }\n      TokenizerResult resolvedUrlResult =\n          TokenizerHelper.tokenizeUrls(urlsToTokenize, message.getLocale(), DO_NOT_REMOVE_WWW);\n      List<String> urlTokens = new ArrayList<>(resolvedUrlResult.tokens);\n      textFeatures.setResolvedUrlTokens(urlTokens);\n    }\n  }\n\n  private void takeEntities(TwitterMessage message,\n                            TweetTextFeatures textFeatures,\n                            TokenizerResult result,\n                            PenguinVersion penguinVersion) {\n    if (message.getHashtags().isEmpty()) {\n      // add hashtags to TwitterMessage if it doens't already have them, from\n      // JSON entities, this happens when we do offline indexing\n      for (String hashtag : sanitizeTokenizerResults(result.hashtags, '#')) {\n        message.addHashtag(hashtag);\n      }\n    }\n\n    if (message.getMentions().isEmpty()) {\n      // add mentions to TwitterMessage if it doens't already have them, from\n      // JSON entities, this happens when we do offline indexing\n      for (String mention : sanitizeTokenizerResults(result.mentions, '@')) {\n        message.addMention(mention);\n      }\n    }\n\n    setHashtagsAndMentions(message, textFeatures, penguinVersion);\n  }\n\n  private void setHashtagsAndMentions(TwitterMessage message,\n                                      TweetTextFeatures textFeatures,\n                                      PenguinVersion penguinVersion) {\n    textFeatures.setHashtags(message.getNormalizedHashtags(penguinVersion));\n    textFeatures.setMentions(message.getLowercasedMentions());\n  }\n\n  // The strings in the mentions, hashtags and stocks lists in TokenizerResult should already have\n  // the leading characters ('@', '#' and '$') stripped. So in most cases, this sanitization is not\n  // needed. However, sometimes Penguin tokenizes hashtags, cashtags and mentions incorrectly\n  // (for example, when using the Korean tokenizer for tokens like ~@mention or ?#hashtag -- see\n  // SEARCHQUAL-11924 for more details). So we're doing this extra sanitization here to try to work\n  // around these tokenization issues.\n  private List<String> sanitizeTokenizerResults(List<String> tokens, char tokenSymbol) {\n    List<String> sanitizedTokens = new ArrayList<String>();\n    for (String token : tokens) {\n      int indexOfTokenSymbol = token.indexOf(tokenSymbol);\n      if (indexOfTokenSymbol < 0) {\n        sanitizedTokens.add(token);\n      } else {\n        String sanitizedToken = token.substring(indexOfTokenSymbol + 1);\n        if (!sanitizedToken.isEmpty()) {\n          sanitizedTokens.add(sanitizedToken);\n        }\n      }\n    }\n    return sanitizedTokens;\n  }\n\n  /** Determines if the normalized text of the given features contain a question mark. */\n  public static boolean findQuestionMark(TweetTextFeatures textFeatures) {\n    // t.co links don't contain ?'s, so it's not necessary to subtract ?'s occurring in Urls\n    // the tweet text always contains t.co, even if the display url is different\n    // all links on twitter are now wrapped into t.co\n    return textFeatures.getNormalizedText().contains(\"?\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/relevance/text/VisibleTokenRatioNormalizer.java",
    "content": "package com.twitter.search.common.relevance.text;\n\npublic class VisibleTokenRatioNormalizer {\n\n  private static final int NORMALIZE_TO_BITS = 4;\n  private final int normalizeToSize;\n\n  /**\n   * constructor\n   */\n  public VisibleTokenRatioNormalizer(int normalizeToBits) {\n    int size = 2 << (normalizeToBits - 1);\n    // Let's say normalizeSize is set to 16....\n    // If you multiply 1.0 * 16, it is 16\n    // If you multiply 0.0 * 16, it is 0\n    // That would be occupying 17 ints, not 16, so we subtract 1 here...\n    this.normalizeToSize = size - 1;\n  }\n\n  /**\n   * method\n   */\n  public int normalize(double percent) {\n    if (percent > 1 || percent < 0) {\n      throw new IllegalArgumentException(\"percent should be less than 1 and greater than 0\");\n    }\n    int bucket = (int) (percent * normalizeToSize);\n    return normalizeToSize - bucket;\n  }\n\n  public double denormalize(int reverseBucket) {\n    int bucket = normalizeToSize - reverseBucket;\n    return bucket / (double) normalizeToSize;\n  }\n\n  public static VisibleTokenRatioNormalizer createInstance() {\n    return new VisibleTokenRatioNormalizer(NORMALIZE_TO_BITS);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/AnalyzerFactory.java",
    "content": "package com.twitter.search.common.schema;\n\nimport java.io.Reader;\nimport java.text.ParseException;\nimport java.util.Map;\n\nimport com.google.common.base.Splitter;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Sets;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport org.apache.lucene.analysis.Analyzer;\nimport org.apache.lucene.analysis.CharArraySet;\nimport org.apache.lucene.analysis.CharFilter;\nimport org.apache.lucene.analysis.TokenStream;\nimport org.apache.lucene.analysis.Tokenizer;\nimport org.apache.lucene.analysis.charfilter.HTMLStripCharFilter;\nimport org.apache.lucene.analysis.core.WhitespaceAnalyzer;\nimport org.apache.lucene.analysis.fa.PersianCharFilter;\nimport org.apache.lucene.analysis.standard.StandardAnalyzer;\nimport org.apache.lucene.util.Version;\n\nimport com.twitter.search.common.schema.thriftjava.ThriftAnalyzer;\nimport com.twitter.search.common.schema.thriftjava.ThriftClassInstantiater;\nimport com.twitter.search.common.schema.thriftjava.ThriftCustomAnalyzer;\n\npublic class AnalyzerFactory {\n  private static final Logger LOG = LoggerFactory.getLogger(AnalyzerFactory.class);\n\n  private static final String MATCH_VERSION_ARG_NAME = \"matchVersion\";\n  private static final String STANDARD_ANALYZER = \"StandardAnalyzer\";\n  private static final String WHITESPACE_ANALYZER = \"WhitespaceAnalyzer\";\n  private static final String SEARCH_WHITESPACE_ANALYZER = \"SearchWhitespaceAnalyzer\";\n  private static final String HTML_STRIP_CHAR_FILTER = \"HTMLStripCharFilter\";\n  private static final String PERSIAN_CHAR_FILTER = \"PersianCharFilter\";\n\n  /**\n   * Return a Lucene Analyzer based on the given ThriftAnalyzer.\n   */\n  public Analyzer getAnalyzer(ThriftAnalyzer analyzer) {\n    if (analyzer.isSetAnalyzer()) {\n      return resolveAnalyzerClass(analyzer.getAnalyzer());\n    } else if (analyzer.isSetCustomAnalyzer()) {\n      return buildCustomAnalyzer(analyzer.getCustomAnalyzer());\n    }\n    return new SearchWhitespaceAnalyzer();\n  }\n\n  private Analyzer resolveAnalyzerClass(ThriftClassInstantiater classDef) {\n    Map<String, String> params = classDef.getParams();\n    Version matchVersion = Version.LUCENE_8_5_2;\n\n    String matchVersionName = getArg(params, MATCH_VERSION_ARG_NAME);\n    if (matchVersionName != null) {\n      try {\n        matchVersion = Version.parse(matchVersionName);\n      } catch (ParseException e) {\n        // ignore and use default version\n        LOG.warn(\"Unable to parse match version: \" + matchVersionName\n                + \". Will use default version of 8.5.2.\");\n      }\n    }\n\n    if (classDef.getClassName().equals(STANDARD_ANALYZER)) {\n      String stopwords = getArg(params, \"stopwords\");\n      if (stopwords != null) {\n\n        CharArraySet stopwordSet = new CharArraySet(\n                Lists.newLinkedList(Splitter.on(\",\").split(stopwords)),\n                false);\n        return new StandardAnalyzer(stopwordSet);\n      } else {\n        return new StandardAnalyzer();\n      }\n    } else if (classDef.getClassName().equals(WHITESPACE_ANALYZER)) {\n      return new WhitespaceAnalyzer();\n    } else if (classDef.getClassName().equals(SEARCH_WHITESPACE_ANALYZER)) {\n      return new SearchWhitespaceAnalyzer();\n    }\n\n    return null;\n  }\n\n  private Analyzer buildCustomAnalyzer(final ThriftCustomAnalyzer customAnalyzer) {\n    return new Analyzer() {\n      @Override\n      protected TokenStreamComponents createComponents(String fieldName) {\n        final Tokenizer tokenizer = resolveTokenizerClass(customAnalyzer.getTokenizer());\n\n        TokenStream filter = tokenizer;\n\n        if (customAnalyzer.isSetFilters()) {\n          for (ThriftClassInstantiater filterClass : customAnalyzer.getFilters()) {\n            filter = resolveTokenFilterClass(filterClass, filter);\n          }\n        }\n\n        return new TokenStreamComponents(tokenizer, filter);\n      }\n    };\n  }\n\n  private Tokenizer resolveTokenizerClass(ThriftClassInstantiater classDef) {\n    return null;\n  }\n\n  private TokenStream resolveTokenFilterClass(ThriftClassInstantiater classDef, TokenStream input) {\n    return null;\n  }\n\n  private CharFilter resolveCharFilterClass(ThriftClassInstantiater classDef, Reader input) {\n    if (classDef.getClassName().equals(HTML_STRIP_CHAR_FILTER)) {\n      String escapedTags = getArg(classDef.getParams(), \"excapedTags\");\n      if (escapedTags != null) {\n        return new HTMLStripCharFilter(input, Sets.newHashSet(Splitter.on(\",\").split(escapedTags)));\n      } else {\n        return new HTMLStripCharFilter(input);\n      }\n    } else if (classDef.getClassName().equals(PERSIAN_CHAR_FILTER)) {\n      return new PersianCharFilter(input);\n    }\n\n\n    throw new ClassNotSupportedException(\"CharFilter\", classDef);\n  }\n\n  private String getArg(Map<String, String> args, String arg) {\n    if (args == null) {\n      return null;\n    }\n\n    return args.get(arg);\n  }\n\n  public final class ClassNotSupportedException extends RuntimeException {\n    private ClassNotSupportedException(String type, ThriftClassInstantiater classDef) {\n      super(type + \" class with name \" + classDef.getClassName() + \" currently not supported.\");\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/BUILD",
    "content": "# Library for schema builder and related analysis utilities.\njava_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-common\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-smartcn\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-facet\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/collections\",\n        \"src/java/com/twitter/common/text/token\",\n        \"src/java/com/twitter/common/text/util:token-util\",\n        \"src/java/com/twitter/search/common/encoding/docvalues\",\n        \"src/java/com/twitter/search/common/features\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/util/analysis\",\n        \"src/java/com/twitter/search/common/util/io\",\n        \"src/java/com/twitter/search/common/util/io:record-reader-api\",\n        \"src/java/com/twitter/search/common/util/spatial\",\n        \"src/java/com/twitter/search/common/util/text\",\n        \"src/java/com/twitter/search/common/util/thrift:thrift-utils\",\n        \"src/thrift/com/twitter/search/common:features-java\",\n        \"src/thrift/com/twitter/search/common:schema-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/DynamicSchema.java",
    "content": "package com.twitter.search.common.schema;\n\nimport java.util.Collection;\nimport java.util.Map;\nimport java.util.concurrent.atomic.AtomicReference;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Predicate;\nimport com.google.common.collect.ImmutableCollection;\nimport com.google.common.collect.ImmutableMap;\n\nimport org.apache.lucene.analysis.Analyzer;\nimport org.apache.lucene.facet.FacetsConfig;\nimport org.apache.lucene.index.FieldInfos;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchema;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\nimport com.twitter.search.common.schema.base.FieldWeightDefault;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.thriftjava.ThriftAnalyzer;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFType;\nimport com.twitter.search.common.schema.thriftjava.ThriftFieldConfiguration;\n\n/**\n * A schema implementation that allow minor version increments at run time.\n */\npublic class DynamicSchema implements Schema {\n  private static final Logger LOG = LoggerFactory.getLogger(DynamicSchema.class);\n\n  private final AtomicReference<ImmutableSchema> schema;\n\n  public DynamicSchema(ImmutableSchema schema) {\n    this.schema = new AtomicReference<>(schema);\n  }\n\n  public ImmutableSchemaInterface getSchemaSnapshot() {\n    return schema.get();\n  }\n\n  /**\n   * Update the schema reference inside this DynamicSchema.\n   */\n  public synchronized void updateSchema(ImmutableSchema newSchema) throws SchemaUpdateException {\n    ImmutableSchema oldSchema = schema.get();\n    if (newSchema.getMajorVersionNumber() != oldSchema.getMajorVersionNumber()) {\n      throw new SchemaUpdateException(\"Dynamic major version update is not supported.\");\n    } else {\n      if (newSchema.getMinorVersionNumber() <= oldSchema.getMinorVersionNumber()) {\n        throw new SchemaUpdateException(\"Dynamic backward minor version update is not supported.\");\n      } else {\n        LOG.info(\"DynamicSchema accepted update. Old version is {}.{}; new version is {}.{}\",\n            oldSchema.getMajorVersionNumber(),\n            oldSchema.getMinorVersionNumber(),\n            newSchema.getMajorVersionNumber(),\n            newSchema.getMinorVersionNumber());\n        schema.set(newSchema);\n      }\n    }\n  }\n\n  public static class SchemaUpdateException extends Exception {\n    public SchemaUpdateException(String message) {\n      super(message);\n    }\n  }\n\n  // The below are all methods in the Schema interface delegated to the underlying ImmutableSchema.\n  // The below is generated by IntelliJ, and reviewers can stop reviewing this file here.\n  // If you are adding logic into this class, please do so above this line.\n  @Override\n  public FieldInfos getLuceneFieldInfos(\n      Predicate<String> acceptedFields) {\n    return schema.get().getLuceneFieldInfos(acceptedFields);\n  }\n\n  @Override\n  public FacetsConfig getFacetsConfig() {\n    return schema.get().getFacetsConfig();\n  }\n\n  @Override\n  public Analyzer getDefaultAnalyzer(\n      ThriftAnalyzer override) {\n    return schema.get().getDefaultAnalyzer(override);\n  }\n\n  @Override\n  public ImmutableCollection<FieldInfo> getFieldInfos() {\n    return schema.get().getFieldInfos();\n  }\n\n  @Override\n  public boolean hasField(int fieldConfigId) {\n    return schema.get().hasField(fieldConfigId);\n  }\n\n  @Override\n  public boolean hasField(String fieldName) {\n    return schema.get().hasField(fieldName);\n  }\n\n  @Override\n  @Nullable\n  public FieldInfo getFieldInfo(int fieldConfigId) {\n    return schema.get().getFieldInfo(fieldConfigId);\n  }\n\n  @Override\n  @Nullable\n  public FieldInfo getFieldInfo(String fieldName) {\n    return schema.get().getFieldInfo(fieldName);\n  }\n\n  @Override\n  public String getFieldName(int fieldConfigId) {\n    return schema.get().getFieldName(fieldConfigId);\n  }\n\n  @Override\n  public FieldInfo getFieldInfo(int fieldConfigId,\n                                ThriftFieldConfiguration override) {\n    return schema.get().getFieldInfo(fieldConfigId, override);\n  }\n\n  @Override\n  public int getNumFacetFields() {\n    return schema.get().getNumFacetFields();\n  }\n\n  @Override\n  public FieldInfo getFacetFieldByFacetName(\n      String facetName) {\n    return schema.get().getFacetFieldByFacetName(facetName);\n  }\n\n  @Override\n  public FieldInfo getFacetFieldByFieldName(\n      String fieldName) {\n    return schema.get().getFacetFieldByFieldName(fieldName);\n  }\n\n  @Override\n  public Collection<FieldInfo> getFacetFields() {\n    return schema.get().getFacetFields();\n  }\n\n  @Override\n  public Collection<FieldInfo> getCsfFacetFields() {\n    return schema.get().getCsfFacetFields();\n  }\n\n  @Override\n  public String getVersionDescription() {\n    return schema.get().getVersionDescription();\n  }\n\n  @Override\n  public int getMajorVersionNumber() {\n    return schema.get().getMajorVersionNumber();\n  }\n\n  @Override\n  public int getMinorVersionNumber() {\n    return schema.get().getMinorVersionNumber();\n  }\n\n  @Override\n  public boolean isVersionOfficial() {\n    return schema.get().isVersionOfficial();\n  }\n\n  @Override\n  public Map<String, FieldWeightDefault> getFieldWeightMap() {\n    return schema.get().getFieldWeightMap();\n  }\n\n  @Override\n  public FeatureConfiguration getFeatureConfigurationByName(\n      String featureName) {\n    return schema.get().getFeatureConfigurationByName(featureName);\n  }\n\n  @Override\n  public FeatureConfiguration getFeatureConfigurationById(int featureFieldId) {\n    return Preconditions.checkNotNull(schema.get().getFeatureConfigurationById(featureFieldId));\n  }\n\n  @Override\n  @Nullable\n  public ThriftCSFType getCSFFieldType(\n      String fieldName) {\n    return schema.get().getCSFFieldType(fieldName);\n  }\n\n  @Override\n  public ThriftSearchFeatureSchema getSearchFeatureSchema() {\n    return schema.get().getSearchFeatureSchema();\n  }\n\n  @Override\n  public ImmutableMap<Integer, FeatureConfiguration> getFeatureIdToFeatureConfig() {\n    return schema.get().getFeatureIdToFeatureConfig();\n  }\n\n  @Override\n  public ImmutableMap<String, FeatureConfiguration> getFeatureNameToFeatureConfig() {\n    return schema.get().getFeatureNameToFeatureConfig();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/ImmutableSchema.java",
    "content": "package com.twitter.search.common.schema;\n\nimport java.io.IOException;\nimport java.io.ObjectOutputStream;\nimport java.util.Collection;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.SortedMap;\nimport java.util.TreeMap;\nimport java.util.concurrent.atomic.AtomicLong;\nimport javax.annotation.Nullable;\nimport javax.annotation.concurrent.Immutable;\nimport javax.annotation.concurrent.ThreadSafe;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Predicate;\nimport com.google.common.collect.ImmutableCollection;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.ImmutableSet;\nimport com.google.common.collect.ImmutableSortedMap;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\nimport com.google.common.collect.Sets;\n\nimport org.apache.lucene.analysis.Analyzer;\nimport org.apache.lucene.facet.FacetsConfig;\nimport org.apache.lucene.index.DocValuesType;\nimport org.apache.lucene.index.FieldInfos;\nimport org.apache.lucene.index.IndexOptions;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.common.text.util.TokenStreamSerializer;\nimport com.twitter.search.common.features.ExternalTweetFeature;\nimport com.twitter.search.common.features.SearchResultFeature;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchema;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchemaEntry;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchemaSpecifier;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureType;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\nimport com.twitter.search.common.schema.base.FieldWeightDefault;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.IndexedNumericFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftAnalyzer;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFType;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFViewSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftFacetFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftFieldConfiguration;\nimport com.twitter.search.common.schema.thriftjava.ThriftFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexedFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftSchema;\nimport com.twitter.search.common.schema.thriftjava.ThriftSearchFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftTokenStreamSerializer;\n\n/**\n * A schema instance that does not change at run time.\n */\n@Immutable @ThreadSafe\npublic class ImmutableSchema implements ImmutableSchemaInterface {\n  private static final Logger LOG = LoggerFactory.getLogger(ImmutableSchema.class);\n  private static final ImmutableSet<ThriftCSFType> CAN_FACET_ON_CSF_TYPES =\n      ImmutableSet.<ThriftCSFType>builder()\n          .add(ThriftCSFType.BYTE)\n          .add(ThriftCSFType.INT)\n          .add(ThriftCSFType.LONG)\n          .build();\n\n  private static final SearchCounter FEATURES_EXISTED_IN_OLD_SCHEMA =\n      SearchCounter.export(\"features_existed_in_old_schema\");\n\n  // Currently our index uses 4 bits to store the facet field id.\n  public static final int MAX_FACET_FIELD_ID = 15;\n\n  public static final String HF_TERM_PAIRS_FIELD = \"hf_term_pairs\";\n  public static final String HF_PHRASE_PAIRS_FIELD = \"hf_phrase_pairs\";\n\n  private final ImmutableMap<Integer, FieldInfo> fieldSettingsMapById;\n  private final ImmutableMap<String, FieldInfo> fieldSettingsMapByName;\n  private final ImmutableMap<String, FeatureConfiguration> featureConfigMapByName;\n  private final ImmutableMap<Integer, FeatureConfiguration> featureConfigMapById;\n\n  @Nullable\n  private final ThriftAnalyzer defaultAnalyzer;\n  private final AnalyzerFactory analyzerFactory;\n\n  private final ImmutableMap<String, FieldWeightDefault> fieldWeightMap;\n  private final Map<String, FieldInfo> facetNameToFieldMap = Maps.newHashMap();\n  private final int numFacetFields;\n  private final ImmutableSet<FieldInfo> csfFacetFields;\n\n  // This is the search result feature schema - it has the definition for all the column stride\n  // view fields.\n  private final ThriftSearchFeatureSchema searchFeatureSchema;\n\n  private final int majorVersionNumber;\n  private final int minorVersionNumber;\n  private final String versionDesc;\n  private final boolean isVersionOfficial;\n\n  /**\n   * Construct a Schema instance with the given ThriftSchema and AnalyzerFactory.\n   */\n  public ImmutableSchema(ThriftSchema thriftSchema,\n                         AnalyzerFactory analyzerFactory,\n                         String featureSchemaVersionPrefix) throws SchemaValidationException {\n    Pair<Integer, String> versionPair = parseVersionString(thriftSchema.getVersion());\n    this.majorVersionNumber = thriftSchema.getMajorVersionNumber();\n    this.minorVersionNumber = thriftSchema.getMinorVersionNumber();\n    this.versionDesc = versionPair.getSecond();\n    this.isVersionOfficial = thriftSchema.isVersionIsOfficial();\n\n    this.analyzerFactory = analyzerFactory;\n\n    Map<Integer, FieldInfo> tmpMap = Maps.newLinkedHashMap();\n    Set<FieldInfo> tmpSet = Sets.newHashSet();\n\n    if (thriftSchema.isSetDefaultAnalyzer()) {\n      this.defaultAnalyzer = thriftSchema.getDefaultAnalyzer().deepCopy();\n    } else {\n      this.defaultAnalyzer = null;\n    }\n\n    Map<Integer, ThriftFieldConfiguration> configs = thriftSchema.getFieldConfigs();\n\n    // Collect all the CSF Views, so that we can later verify that they are appropriately\n    // configured once we've processed all the other field settings.\n    Map<Integer, ThriftFieldConfiguration> csfViewFields = Maps.newHashMap();\n    boolean requiresHfPairFields = false;\n    boolean hasHfTermPairField = false;\n    boolean hasHfPhrasePairField = false;\n    int numFacets = 0;\n    for (Map.Entry<Integer, ThriftFieldConfiguration> entry : configs.entrySet()) {\n      int fieldId = entry.getKey();\n\n      if (tmpMap.containsKey(fieldId)) {\n        throw new SchemaValidationException(\"Duplicate field id \" + fieldId);\n      }\n\n      ThriftFieldConfiguration config = entry.getValue();\n      FieldInfo fieldInfo = parseThriftFieldSettings(fieldId, config, csfViewFields);\n      validate(fieldInfo);\n      if (fieldInfo.getFieldType().isFacetField()) {\n        if (numFacets > MAX_FACET_FIELD_ID) {\n          throw new SchemaValidationException(\n              \"Maximum supported facet field ID is:  \" + MAX_FACET_FIELD_ID);\n        }\n        numFacets++;\n        facetNameToFieldMap.put(fieldInfo.getFieldType().getFacetName(), fieldInfo);\n\n        if (fieldInfo.getFieldType().isUseCSFForFacetCounting()) {\n          tmpSet.add(fieldInfo);\n        }\n      }\n\n      tmpMap.put(fieldId, fieldInfo);\n\n      if (fieldInfo.getFieldType().isIndexHFTermPairs()) {\n        requiresHfPairFields = true;\n      }\n      if (fieldInfo.getName().equals(HF_TERM_PAIRS_FIELD)) {\n        hasHfTermPairField = true;\n      }\n      if (fieldInfo.getName().equals(HF_PHRASE_PAIRS_FIELD)) {\n        hasHfPhrasePairField = true;\n      }\n    }\n\n    this.numFacetFields = numFacets;\n    this.csfFacetFields = ImmutableSet.copyOf(tmpSet);\n\n    // If any field requires high frequency term/phrase pair fields, make sure they exist\n    if (requiresHfPairFields) {\n      if (!hasHfTermPairField || !hasHfPhrasePairField) {\n        throw new SchemaValidationException(\n            \"High frequency term/phrase pair fields do not exist in the schema.\");\n      }\n    }\n\n    this.fieldSettingsMapById = ImmutableMap.copyOf(tmpMap);\n\n    Pair<ImmutableMap<String, FeatureConfiguration>, ImmutableMap<Integer, FeatureConfiguration>>\n        featureConfigMapPair = buildFeatureMaps(csfViewFields);\n    this.featureConfigMapByName = featureConfigMapPair.getFirst();\n    this.featureConfigMapById = featureConfigMapPair.getSecond();\n\n    for (ThriftFieldConfiguration csfViewField : csfViewFields.values()) {\n      SchemaBuilder.verifyCSFViewSettings(configs, csfViewField);\n    }\n\n    ImmutableMap.Builder<String, FieldInfo> builder = ImmutableMap.builder();\n\n    for (FieldInfo info : fieldSettingsMapById.values()) {\n      info.getFieldType().freeze();\n      builder.put(info.getName(), info);\n    }\n    this.fieldSettingsMapByName = builder.build();\n\n    ImmutableMap.Builder<String, FieldWeightDefault> fieldWeightMapBuilder = ImmutableMap.builder();\n\n    for (FieldInfo fi : getFieldInfos()) {\n      // CSF fields are not searchable. All other fields are.\n      if (fi.getFieldType().isIndexedField()) {\n        fieldWeightMapBuilder.put(\n            fi.getName(),\n            new FieldWeightDefault(\n                fi.getFieldType().isTextSearchableByDefault(),\n                fi.getFieldType().getTextSearchableFieldWeight()));\n      }\n    }\n\n    this.fieldWeightMap = fieldWeightMapBuilder.build();\n    // Create features with extra Earlybird derived fields, extra fields won't change the version\n    // but they do change the checksum.\n    this.searchFeatureSchema = createSearchResultFeatureSchema(\n        featureSchemaVersionPrefix, fieldSettingsMapByName, featureConfigMapByName);\n  }\n\n  /**\n   * Add a set of features to a schema if they don't exist yet, and update the schema checksum.\n   * if there's conflict, RuntimeException will be thrown.\n   * Old map won't be touched, a new map will be returned will old and new data combined.\n   */\n  public static Map<Integer, ThriftSearchFeatureSchemaEntry> appendToFeatureSchema(\n      Map<Integer, ThriftSearchFeatureSchemaEntry> oldEntryMap,\n      Set<? extends SearchResultFeature> features) throws SchemaValidationException {\n    if (oldEntryMap == null) {\n      throw new SchemaValidationException(\n          \"Cannot append features to schema, the entryMap is null\");\n    }\n    // make a copy of the existing map\n    ImmutableMap.Builder<Integer, ThriftSearchFeatureSchemaEntry> builder =\n        ImmutableSortedMap.<Integer, ThriftSearchFeatureSchemaEntry>naturalOrder()\n            .putAll(oldEntryMap);\n\n    for (SearchResultFeature feature : features) {\n      if (oldEntryMap.containsKey(feature.getId())) {\n        FEATURES_EXISTED_IN_OLD_SCHEMA.increment();\n      } else {\n        builder.put(feature.getId(), new ThriftSearchFeatureSchemaEntry()\n            .setFeatureName(feature.getName())\n            .setFeatureType(feature.getType()));\n      }\n    }\n    return builder.build();\n  }\n\n  /**\n   * Append external features to create a new schema.\n   * @param oldSchema The old schema to build on top of\n   * @param features a list of features to be appended to the schema\n   * @param versionSuffix the version suffix, if not-null, it will be attached to the end of\n   * original schema's version.\n   * @return A new schema object with the appended fields\n   * @throws SchemaValidationException thrown when the checksum cannot be computed\n   */\n  public static ThriftSearchFeatureSchema appendToCreateNewFeatureSchema(\n      ThriftSearchFeatureSchema oldSchema,\n      Set<ExternalTweetFeature> features,\n      @Nullable String versionSuffix) throws SchemaValidationException {\n\n    ThriftSearchFeatureSchema newSchema = new ThriftSearchFeatureSchema();\n    // copy over all the entries plus the new ones\n    newSchema.setEntries(appendToFeatureSchema(oldSchema.getEntries(), features));\n\n    ThriftSearchFeatureSchemaSpecifier spec = new ThriftSearchFeatureSchemaSpecifier();\n    // the version is directly inherited or with a suffix\n    Preconditions.checkArgument(versionSuffix == null || !versionSuffix.isEmpty());\n    spec.setVersion(versionSuffix == null\n        ? oldSchema.getSchemaSpecifier().getVersion()\n        : oldSchema.getSchemaSpecifier().getVersion() + versionSuffix);\n    spec.setChecksum(getChecksum(newSchema.getEntries()));\n    newSchema.setSchemaSpecifier(spec);\n    return newSchema;\n  }\n\n  @Override\n  public FieldInfos getLuceneFieldInfos(Predicate<String> acceptedFields) {\n    List<org.apache.lucene.index.FieldInfo> acceptedFieldInfos = Lists.newArrayList();\n    for (FieldInfo fi : getFieldInfos()) {\n      if (acceptedFields == null || acceptedFields.apply(fi.getName())) {\n        acceptedFieldInfos.add(convert(fi.getName(), fi.getFieldId(), fi.getFieldType()));\n      }\n    }\n    return new FieldInfos(acceptedFieldInfos.toArray(\n        new org.apache.lucene.index.FieldInfo[acceptedFieldInfos.size()]));\n  }\n\n  private FieldInfo parseThriftFieldSettings(int fieldId, ThriftFieldConfiguration fieldConfig,\n                                             Map<Integer, ThriftFieldConfiguration> csfViewFields)\n      throws SchemaValidationException {\n    FieldInfo fieldInfo\n        = new FieldInfo(fieldId, fieldConfig.getFieldName(), new EarlybirdFieldType());\n    ThriftFieldSettings fieldSettings = fieldConfig.getSettings();\n\n\n    boolean settingFound = false;\n\n    if (fieldSettings.isSetIndexedFieldSettings()) {\n      if (fieldSettings.isSetCsfFieldSettings() || fieldSettings.isSetCsfViewSettings()) {\n        throw new SchemaValidationException(\"ThriftFieldSettings: Only one of \"\n            + \"'indexedFieldSettings', 'csfFieldSettings', 'csfViewSettings' can be set.\");\n      }\n\n      applyIndexedFieldSettings(fieldInfo, fieldSettings.getIndexedFieldSettings());\n      settingFound = true;\n    }\n\n    if (fieldSettings.isSetCsfFieldSettings()) {\n      if (fieldSettings.isSetIndexedFieldSettings() || fieldSettings.isSetCsfViewSettings()) {\n        throw new SchemaValidationException(\"ThriftFieldSettings: Only one of \"\n            + \"'indexedFieldSettings', 'csfFieldSettings', 'csfViewSettings' can be set.\");\n      }\n\n      applyCsfFieldSettings(fieldInfo, fieldSettings.getCsfFieldSettings());\n      settingFound = true;\n    }\n\n    if (fieldSettings.isSetFacetFieldSettings()) {\n      if (!fieldSettings.isSetIndexedFieldSettings() && !(fieldSettings.isSetCsfFieldSettings()\n          && fieldSettings.getFacetFieldSettings().isUseCSFForFacetCounting()\n          && CAN_FACET_ON_CSF_TYPES.contains(fieldSettings.getCsfFieldSettings().getCsfType()))) {\n        throw new SchemaValidationException(\"ThriftFieldSettings: 'facetFieldSettings' can only be \"\n            + \"used in combination with 'indexedFieldSettings' or with 'csfFieldSettings' \"\n            + \"where 'isUseCSFForFacetCounting' was set to true and ThriftCSFType is a type that \"\n            + \"can be faceted on.\");\n      }\n\n      applyFacetFieldSettings(fieldInfo, fieldSettings.getFacetFieldSettings());\n      settingFound = true;\n    }\n\n    if (fieldSettings.isSetCsfViewSettings()) {\n      if (fieldSettings.isSetIndexedFieldSettings() || fieldSettings.isSetCsfFieldSettings()) {\n        throw new SchemaValidationException(\"ThriftFieldSettings: Only one of \"\n            + \"'indexedFieldSettings', 'csfFieldSettings', 'csfViewSettings' can be set.\");\n      }\n\n      // add this field now, but apply settings later to make sure the base field was added properly\n      // before\n      csfViewFields.put(fieldId, fieldConfig);\n      settingFound = true;\n    }\n\n    if (!settingFound) {\n      throw new SchemaValidationException(\"ThriftFieldSettings: One of 'indexedFieldSettings', \"\n          + \"'csfFieldSettings' or 'facetFieldSettings' must be set.\");\n    }\n\n    // search field settings are optional\n    if (fieldSettings.isSetSearchFieldSettings()) {\n      if (!fieldSettings.isSetIndexedFieldSettings()) {\n        throw new SchemaValidationException(\n            \"ThriftFieldSettings: 'searchFieldSettings' can only be \"\n                + \"used in combination with 'indexedFieldSettings'\");\n      }\n\n      applySearchFieldSettings(fieldInfo, fieldSettings.getSearchFieldSettings());\n    }\n\n    return fieldInfo;\n  }\n\n  private void applyCsfFieldSettings(FieldInfo fieldInfo, ThriftCSFFieldSettings settings)\n      throws SchemaValidationException {\n    // csfType is required - no need to check if it's set\n    fieldInfo.getFieldType().setDocValuesType(DocValuesType.NUMERIC);\n    fieldInfo.getFieldType().setCsfType(settings.getCsfType());\n\n    if (settings.isVariableLength()) {\n      fieldInfo.getFieldType().setDocValuesType(DocValuesType.BINARY);\n      fieldInfo.getFieldType().setCsfVariableLength();\n    } else {\n      if (settings.isSetFixedLengthSettings()) {\n        fieldInfo.getFieldType().setCsfFixedLengthSettings(\n            settings.getFixedLengthSettings().getNumValuesPerDoc(),\n            settings.getFixedLengthSettings().isUpdateable());\n        if (settings.getFixedLengthSettings().getNumValuesPerDoc() > 1) {\n          fieldInfo.getFieldType().setDocValuesType(DocValuesType.BINARY);\n        }\n      } else {\n        throw new SchemaValidationException(\n            \"ThriftCSFFieldSettings: Either variableLength should be set to 'true', \"\n                + \"or fixedLengthSettings should be set.\");\n      }\n    }\n\n    fieldInfo.getFieldType().setCsfLoadIntoRam(settings.isLoadIntoRAM());\n    if (settings.isSetDefaultValue()) {\n      fieldInfo.getFieldType().setCsfDefaultValue(settings.getDefaultValue());\n    }\n  }\n\n  private void applyCsfViewFieldSettings(FieldInfo fieldInfo, FieldInfo baseField,\n                                         ThriftCSFViewSettings settings)\n      throws SchemaValidationException {\n    // csfType is required - no need to check if it's set\n    fieldInfo.getFieldType().setDocValuesType(DocValuesType.NUMERIC);\n    fieldInfo.getFieldType().setCsfType(settings.getCsfType());\n\n    fieldInfo.getFieldType().setCsfFixedLengthSettings(1 /* numValuesPerDoc*/,\n        false /* updateable*/);\n\n    fieldInfo.getFieldType().setCsfViewSettings(fieldInfo.getName(), settings, baseField);\n  }\n\n  private void applyFacetFieldSettings(FieldInfo fieldInfo, ThriftFacetFieldSettings settings) {\n    if (settings.isSetFacetName()) {\n      fieldInfo.getFieldType().setFacetName(settings.getFacetName());\n    } else {\n      // fall back to field name if no facet name is explicitly provided\n      fieldInfo.getFieldType().setFacetName(fieldInfo.getName());\n    }\n    fieldInfo.getFieldType().setStoreFacetSkiplist(settings.isStoreSkiplist());\n    fieldInfo.getFieldType().setStoreFacetOffensiveCounters(settings.isStoreOffensiveCounters());\n    fieldInfo.getFieldType().setUseCSFForFacetCounting(settings.isUseCSFForFacetCounting());\n  }\n\n  private void applyIndexedFieldSettings(FieldInfo fieldInfo, ThriftIndexedFieldSettings settings)\n      throws SchemaValidationException {\n    fieldInfo.getFieldType().setIndexedField(true);\n    fieldInfo.getFieldType().setStored(settings.isStored());\n    fieldInfo.getFieldType().setTokenized(settings.isTokenized());\n    fieldInfo.getFieldType().setStoreTermVectors(settings.isStoreTermVectors());\n    fieldInfo.getFieldType().setStoreTermVectorOffsets(settings.isStoreTermVectorOffsets());\n    fieldInfo.getFieldType().setStoreTermVectorPositions(settings.isStoreTermVectorPositions());\n    fieldInfo.getFieldType().setStoreTermVectorPayloads(settings.isStoreTermVectorPayloads());\n    fieldInfo.getFieldType().setOmitNorms(settings.isOmitNorms());\n    fieldInfo.getFieldType().setIndexHFTermPairs(settings.isIndexHighFreqTermPairs());\n    fieldInfo.getFieldType().setUseTweetSpecificNormalization(\n        settings.deprecated_performTweetSpecificNormalizations);\n\n    if (settings.isSetIndexOptions()) {\n      switch (settings.getIndexOptions()) {\n        case DOCS_ONLY :\n          fieldInfo.getFieldType().setIndexOptions(IndexOptions.DOCS);\n          break;\n        case DOCS_AND_FREQS :\n          fieldInfo.getFieldType().setIndexOptions(IndexOptions.DOCS_AND_FREQS);\n          break;\n        case DOCS_AND_FREQS_AND_POSITIONS :\n          fieldInfo.getFieldType().setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS);\n          break;\n        case DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS :\n          fieldInfo.getFieldType().setIndexOptions(\n              IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);\n          break;\n        default:\n          throw new SchemaValidationException(\"Unknown value for IndexOptions: \"\n              + settings.getIndexOptions());\n      }\n    } else if (settings.isIndexed()) {\n      // default for backward-compatibility\n      fieldInfo.getFieldType().setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS);\n    }\n\n    fieldInfo.getFieldType().setStorePerPositionPayloads(settings.isStorePerPositionPayloads());\n    fieldInfo.getFieldType().setDefaultPayloadLength(\n        settings.getDefaultPerPositionPayloadLength());\n    fieldInfo.getFieldType().setBecomesImmutable(!settings.isSupportOutOfOrderAppends());\n    fieldInfo.getFieldType().setSupportOrderedTerms(settings.isSupportOrderedTerms());\n    fieldInfo.getFieldType().setSupportTermTextLookup(settings.isSupportTermTextLookup());\n\n    if (settings.isSetNumericFieldSettings()) {\n      fieldInfo.getFieldType().setNumericFieldSettings(\n          new IndexedNumericFieldSettings(settings.getNumericFieldSettings()));\n    }\n\n    if (settings.isSetTokenStreamSerializer()) {\n      fieldInfo.getFieldType().setTokenStreamSerializerBuilder(\n          buildTokenStreamSerializerProvider(settings.getTokenStreamSerializer()));\n    }\n  }\n\n  private void applySearchFieldSettings(FieldInfo fieldInfo, ThriftSearchFieldSettings settings)\n      throws SchemaValidationException {\n    fieldInfo.getFieldType().setTextSearchableFieldWeight(\n        (float) settings.getTextSearchableFieldWeight());\n    fieldInfo.getFieldType().setTextSearchableByDefault(settings.isTextDefaultSearchable());\n  }\n\n  private void validate(FieldInfo fieldInfo) throws SchemaValidationException {\n  }\n\n  private TokenStreamSerializer.Builder buildTokenStreamSerializerProvider(\n      final ThriftTokenStreamSerializer settings) {\n    TokenStreamSerializer.Builder builder = TokenStreamSerializer.builder();\n    for (String serializerName : settings.getAttributeSerializerClassNames()) {\n      try {\n        builder.add((TokenStreamSerializer.AttributeSerializer) Class.forName(serializerName)\n            .newInstance());\n      } catch (InstantiationException e) {\n        throw new RuntimeException(\n            \"Unable to instantiate AttributeSerializer for name \" + serializerName);\n      } catch (IllegalAccessException e) {\n        throw new RuntimeException(\n            \"Unable to instantiate AttributeSerializer for name \" + serializerName);\n      } catch (ClassNotFoundException e) {\n        throw new RuntimeException(\n            \"Unable to instantiate AttributeSerializer for name \" + serializerName);\n      }\n    }\n    return builder;\n  }\n\n  @Override\n  public FacetsConfig getFacetsConfig() {\n    FacetsConfig facetsConfig = new FacetsConfig();\n\n    for (String facetName : facetNameToFieldMap.keySet()) {\n      // set multiValued = true as default, since we're using SortedSetDocValues facet, in which,\n      // there is no difference between multiValued true or false for the real facet, but only the\n      // checking of the values.\n      facetsConfig.setMultiValued(facetName, true);\n    }\n\n    return facetsConfig;\n  }\n\n  @Override\n  public Analyzer getDefaultAnalyzer(ThriftAnalyzer override) {\n    if (override != null) {\n      return analyzerFactory.getAnalyzer(override);\n    }\n\n    if (defaultAnalyzer != null) {\n      return analyzerFactory.getAnalyzer(defaultAnalyzer);\n    }\n\n    return new SearchWhitespaceAnalyzer();\n  }\n\n  @Override\n  public ImmutableCollection<FieldInfo> getFieldInfos() {\n    return fieldSettingsMapById.values();\n  }\n\n  /**\n   * This is the preferred method to check whether a field configuration is in schema.\n   * One can also use getFieldInfo and do null checks, but should be careful about excessive\n   * warning logging resulting from looking up fields not in schema.\n   */\n  @Override\n  public boolean hasField(int fieldConfigId) {\n    return fieldSettingsMapById.containsKey(fieldConfigId);\n  }\n\n  /**\n   * This is the preferred method to check whether a field configuration is in schema.\n   * One can also use getFieldInfo and do null checks, but should be careful about excessive\n   * warning logging resulting from looking up fields not in schema.\n   */\n  @Override\n  public boolean hasField(String fieldName) {\n    return fieldSettingsMapByName.containsKey(fieldName);\n  }\n\n  /**\n   * Get FieldInfo for the given field id.\n   * If the goal is to check whether a field is in the schema, use {@link #hasField(int)} instead.\n   * This method logs a warning whenever it returns null.\n   */\n  @Override\n  @Nullable\n  public FieldInfo getFieldInfo(int fieldConfigId) {\n    return getFieldInfo(fieldConfigId, null);\n  }\n\n  private org.apache.lucene.index.FieldInfo convert(String fieldName,\n                                                    int index,\n                                                    EarlybirdFieldType type) {\n    return new org.apache.lucene.index.FieldInfo(\n        fieldName,                          // String name\n        index,                              // int number\n        type.storeTermVectors(),            // boolean storeTermVector\n        type.omitNorms(),                   // boolean omitNorms\n        type.isStorePerPositionPayloads(),  // boolean storePayloads\n        type.indexOptions(),                // IndexOptions indexOptions\n        type.docValuesType(),               // DocValuesType docValues\n        -1,                                 // long dvGen\n        Maps.<String, String>newHashMap(),  // Map<String, String> attributes\n        0,                                  // int pointDataDimensionCount\n        0,                                  // int pointIndexDimensionCount\n        0,                                  // int pointNumBytes\n        false);                             // boolean softDeletesField\n  }\n\n  /**\n   * Get FieldInfo for the given field name, or null if the field does not exist.\n   */\n  @Override\n  @Nullable\n  public FieldInfo getFieldInfo(String fieldName) {\n    return fieldSettingsMapByName.get(fieldName);\n  }\n\n  @Override\n  public String getFieldName(int fieldConfigId) {\n    FieldInfo fieldInfo = fieldSettingsMapById.get(fieldConfigId);\n    return fieldInfo != null ? fieldInfo.getName() : null;\n  }\n\n  @Override\n  public FieldInfo getFieldInfo(int fieldConfigId, ThriftFieldConfiguration override) {\n    FieldInfo fieldInfo = fieldSettingsMapById.get(fieldConfigId);\n    if (fieldInfo == null) {\n      // This method is used to check the availability of fields by IDs,\n      // so no warning is logged here (would be too verbose otherwise).\n      return null;\n    }\n\n    if (override != null) {\n      try {\n        return merge(fieldConfigId, fieldInfo, override);\n      } catch (SchemaValidationException e) {\n        throw new RuntimeException(e);\n      }\n    }\n\n    return fieldInfo;\n  }\n\n  @Override\n  public int getNumFacetFields() {\n    return numFacetFields;\n  }\n\n  @Override\n  public FieldInfo getFacetFieldByFacetName(String facetName) {\n    return facetNameToFieldMap.get(facetName);\n  }\n\n  @Override\n  public FieldInfo getFacetFieldByFieldName(String fieldName) {\n    FieldInfo fieldInfo = getFieldInfo(fieldName);\n    return fieldInfo != null && fieldInfo.getFieldType().isFacetField() ? fieldInfo : null;\n  }\n\n  @Override\n  public Collection<FieldInfo> getFacetFields() {\n    return facetNameToFieldMap.values();\n  }\n\n  @Override\n  public Collection<FieldInfo> getCsfFacetFields() {\n    return csfFacetFields;\n  }\n\n  @Override\n  public String getVersionDescription() {\n    return versionDesc;\n  }\n\n  @Override\n  public int getMajorVersionNumber() {\n    return majorVersionNumber;\n  }\n\n  @Override\n  public int getMinorVersionNumber() {\n    return minorVersionNumber;\n  }\n\n  @Override\n  public boolean isVersionOfficial() {\n    return isVersionOfficial;\n  }\n\n  /**\n   * Parses a version string like \"16: renamed field x into y\" into a version number and\n   * a string description.\n   * @return a Pair of the version number and the description\n   */\n  private static Pair<Integer, String> parseVersionString(String version)\n      throws SchemaValidationException {\n    Preconditions.checkNotNull(version, \"Schema must have a version number and description.\");\n    int colonIndex = version.indexOf(':');\n    if (colonIndex == -1) {\n      throw new SchemaValidationException(\"Malformed version string: \" + version);\n    }\n    try {\n      int versionNumber = Integer.parseInt(version.substring(0, colonIndex));\n      String versionDesc = version.substring(colonIndex + 1);\n      return Pair.of(versionNumber, versionDesc);\n    } catch (Exception e) {\n      throw new SchemaValidationException(\"Malformed version string: \" + version, e);\n    }\n  }\n\n  @Override\n  public Map<String, FieldWeightDefault> getFieldWeightMap() {\n    return fieldWeightMap;\n  }\n\n  /**\n   * Build the feature maps so that we can use feature name to get the feature configuration.\n   * @return: an immutable map keyed on fieldName.\n   */\n  private Pair<ImmutableMap<String, FeatureConfiguration>,\n      ImmutableMap<Integer, FeatureConfiguration>> buildFeatureMaps(\n      final Map<Integer, ThriftFieldConfiguration> csvViewFields)\n      throws SchemaValidationException {\n\n    final ImmutableMap.Builder<String, FeatureConfiguration> featureConfigMapByNameBuilder =\n        ImmutableMap.builder();\n    final ImmutableMap.Builder<Integer, FeatureConfiguration> featureConfigMapByIdBuilder =\n        ImmutableMap.builder();\n\n    for (final Map.Entry<Integer, ThriftFieldConfiguration> entry : csvViewFields.entrySet()) {\n      ThriftFieldSettings fieldSettings = entry.getValue().getSettings();\n      FieldInfo fieldInfo = getFieldInfo(entry.getKey());\n      FieldInfo baseFieldInfo =\n          getFieldInfo(fieldSettings.getCsfViewSettings().getBaseFieldConfigId());\n      if (baseFieldInfo == null) {\n        throw new SchemaValidationException(\"Base field (id=\"\n            + fieldSettings.getCsfViewSettings().getBaseFieldConfigId() + \") not found.\");\n      }\n      applyCsfViewFieldSettings(fieldInfo, baseFieldInfo, fieldSettings.getCsfViewSettings());\n\n      FeatureConfiguration featureConfig = fieldInfo.getFieldType()\n          .getCsfViewFeatureConfiguration();\n      if (featureConfig != null) {\n        featureConfigMapByNameBuilder.put(fieldInfo.getName(), featureConfig);\n        featureConfigMapByIdBuilder.put(fieldInfo.getFieldId(), featureConfig);\n      }\n    }\n\n    return Pair.of(featureConfigMapByNameBuilder.build(), featureConfigMapByIdBuilder.build());\n  }\n\n  @Override\n  public FeatureConfiguration getFeatureConfigurationByName(String featureName) {\n    return featureConfigMapByName.get(featureName);\n  }\n\n  @Override\n  public FeatureConfiguration getFeatureConfigurationById(int featureFieldId) {\n    return Preconditions.checkNotNull(featureConfigMapById.get(featureFieldId),\n        \"Field ID: \" + featureFieldId);\n  }\n\n  @Override\n  @Nullable\n  public ThriftCSFType getCSFFieldType(String fieldName) {\n    FieldInfo fieldInfo = getFieldInfo(fieldName);\n    if (fieldInfo == null) {\n      return null;\n    }\n\n    EarlybirdFieldType fieldType = fieldInfo.getFieldType();\n    if (fieldType.docValuesType() != org.apache.lucene.index.DocValuesType.NUMERIC) {\n      return null;\n    }\n\n    return fieldType.getCsfType();\n  }\n\n  @Override\n  public ImmutableSchemaInterface getSchemaSnapshot() {\n    return this;\n  }\n\n  private FieldInfo merge(int fieldConfigId,\n                          FieldInfo fieldInfo,\n                          ThriftFieldConfiguration overrideConfig)\n      throws SchemaValidationException {\n\n    throw new UnsupportedOperationException(\"Field override config not supported\");\n  }\n\n  @Override\n  public ThriftSearchFeatureSchema getSearchFeatureSchema() {\n    return searchFeatureSchema;\n  }\n\n  @Override\n  public ImmutableMap<Integer, FeatureConfiguration> getFeatureIdToFeatureConfig() {\n    return featureConfigMapById;\n  }\n\n  @Override\n  public ImmutableMap<String, FeatureConfiguration> getFeatureNameToFeatureConfig() {\n    return featureConfigMapByName;\n  }\n\n  private ThriftSearchFeatureSchema createSearchResultFeatureSchema(\n      String featureSchemaVersionPrefix,\n      Map<String, FieldInfo> allFieldSettings,\n      Map<String, FeatureConfiguration> featureConfigurations) throws SchemaValidationException {\n    final ImmutableMap.Builder<Integer, ThriftSearchFeatureSchemaEntry> builder =\n        new ImmutableMap.Builder<>();\n\n    for (Map.Entry<String, FieldInfo> field : allFieldSettings.entrySet()) {\n      FeatureConfiguration featureConfig = featureConfigurations.get(field.getKey());\n      if (featureConfig == null) {\n        // This is either a not csf related field or a csf field.\n        continue;\n      }\n\n      // This is a csfView field.\n      if (featureConfig.getOutputType() == null) {\n        LOG.info(\"Skip unused fieldschemas: {} for search feature schema.\", field.getKey());\n        continue;\n      }\n\n      ThriftSearchFeatureType featureType = getResultFeatureType(featureConfig.getOutputType());\n      if (featureType != null) {\n        builder.put(\n            field.getValue().getFieldId(),\n            new ThriftSearchFeatureSchemaEntry(field.getKey(), featureType));\n      } else {\n        LOG.error(\"Invalid CSFType encountered for csf field: {}\", field.getKey());\n      }\n    }\n    Map<Integer, ThriftSearchFeatureSchemaEntry> indexOnlySchemaEntries = builder.build();\n\n    // Add earlybird derived features, they are defined in ExternalTweetFeatures and used in the\n    // scoring function. They are no different from those auto-generated index-based features\n    // viewed from outside Earlybird.\n    Map<Integer, ThriftSearchFeatureSchemaEntry> entriesWithEBFeatures =\n        appendToFeatureSchema(\n            indexOnlySchemaEntries, ExternalTweetFeature.EARLYBIRD_DERIVED_FEATURES);\n\n    // Add other features needed for tweet ranking from EarlybirdRankingDerivedFeature.\n    Map<Integer, ThriftSearchFeatureSchemaEntry> allSchemaEntries = appendToFeatureSchema(\n        entriesWithEBFeatures, ExternalTweetFeature.EARLYBIRD_RANKING_DERIVED_FEATURES);\n\n    long schemaEntriesChecksum = getChecksum(allSchemaEntries);\n    SearchLongGauge.export(\"feature_schema_checksum\", new AtomicLong(schemaEntriesChecksum));\n\n    String schemaVersion = String.format(\n        \"%s.%d.%d\", featureSchemaVersionPrefix, majorVersionNumber, minorVersionNumber);\n    ThriftSearchFeatureSchemaSpecifier schemaSpecifier =\n        new ThriftSearchFeatureSchemaSpecifier(schemaVersion, schemaEntriesChecksum);\n\n    ThriftSearchFeatureSchema schema = new ThriftSearchFeatureSchema();\n    schema.setSchemaSpecifier(schemaSpecifier);\n    schema.setEntries(allSchemaEntries);\n\n    return schema;\n  }\n\n  // Serializes schemaEntries to a byte array, and computes a CRC32 checksum of the array.\n  // The serialization needs to be stable: if schemaEntries1.equals(schemaEntries2), we want\n  // this method to produce the same checksum for schemaEntrie1 and schemaEntrie2, even if\n  // the checksums are computed in different JVMs, etc.\n  private static long getChecksum(Map<Integer, ThriftSearchFeatureSchemaEntry> schemaEntries)\n      throws SchemaValidationException {\n    SortedMap<Integer, ThriftSearchFeatureSchemaEntry> sortedSchemaEntries =\n        new TreeMap<Integer, ThriftSearchFeatureSchemaEntry>(schemaEntries);\n\n    CRC32OutputStream crc32OutputStream = new CRC32OutputStream();\n    ObjectOutputStream objectOutputStream = null;\n    try {\n      objectOutputStream = new ObjectOutputStream(crc32OutputStream);\n      for (Integer fieldId : sortedSchemaEntries.keySet()) {\n        objectOutputStream.writeObject(fieldId);\n        ThriftSearchFeatureSchemaEntry schemaEntry = sortedSchemaEntries.get(fieldId);\n        objectOutputStream.writeObject(schemaEntry.getFeatureName());\n        objectOutputStream.writeObject(schemaEntry.getFeatureType());\n      }\n      objectOutputStream.flush();\n      return crc32OutputStream.getValue();\n    } catch (IOException e) {\n      throw new SchemaValidationException(\"Could not serialize feature schema entries.\", e);\n    } finally {\n      Preconditions.checkNotNull(objectOutputStream);\n      try {\n        objectOutputStream.close();\n      } catch (IOException e) {\n        throw new SchemaValidationException(\"Could not close ObjectOutputStream.\", e);\n      }\n    }\n  }\n\n  /**\n   * Get the search feature type based on the csf type.\n   * @param csfType the column stride field type for the data\n   * @return the corresponding search feature type\n   */\n  @VisibleForTesting\n  public static ThriftSearchFeatureType getResultFeatureType(ThriftCSFType csfType) {\n    switch (csfType) {\n      case INT:\n      case BYTE:\n        return ThriftSearchFeatureType.INT32_VALUE;\n      case BOOLEAN:\n        return ThriftSearchFeatureType.BOOLEAN_VALUE;\n      case FLOAT:\n      case DOUBLE:\n        return ThriftSearchFeatureType.DOUBLE_VALUE;\n      case LONG:\n        return ThriftSearchFeatureType.LONG_VALUE;\n      default:\n        return null;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/NumericField.java",
    "content": "package com.twitter.search.common.schema;\n\nimport org.apache.lucene.document.Field;\nimport org.apache.lucene.document.FieldType;\nimport org.apache.lucene.index.IndexOptions;\n\n/**\n * A Lucene numeric field, similar to the LegacyIntField, LegacyLongField, etc. Lucene classes that\n * were removed in Lucene 7.0.0.\n */\npublic final class NumericField extends Field {\n  private static final FieldType NUMERIC_FIELD_TYPE = new FieldType();\n  static {\n    NUMERIC_FIELD_TYPE.setTokenized(true);\n    NUMERIC_FIELD_TYPE.setOmitNorms(true);\n    NUMERIC_FIELD_TYPE.setIndexOptions(IndexOptions.DOCS);\n    NUMERIC_FIELD_TYPE.freeze();\n  }\n\n  /**\n   * Creates a new integer field with the given name and value.\n   */\n  public static NumericField newIntField(String fieldName, int value) {\n    NumericField field = new NumericField(fieldName);\n    field.fieldsData = Integer.valueOf(value);\n    return field;\n  }\n\n  /**\n   * Creates a new long field with the given name and value.\n   */\n  public static NumericField newLongField(String fieldName, long value) {\n    NumericField field = new NumericField(fieldName);\n    field.fieldsData = Long.valueOf(value);\n    return field;\n  }\n\n  // We could replace the static methods with constructors, but I think that would make it much\n  // easier to accidentally use NumericField(String, int) instead of NumericField(String, long),\n  // for example, leading to hard to debug errors.\n  private NumericField(String fieldName) {\n    super(fieldName, NUMERIC_FIELD_TYPE);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/SchemaBuilder.java",
    "content": "package com.twitter.search.common.schema;\n\nimport java.util.Map;\nimport java.util.Set;\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.Sets;\n\nimport com.twitter.common.text.util.CharSequenceTermAttributeSerializer;\nimport com.twitter.common.text.util.PositionIncrementAttributeSerializer;\nimport com.twitter.common.text.util.TokenStreamSerializer;\nimport com.twitter.common.text.util.TokenTypeAttributeSerializer;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\nimport com.twitter.search.common.schema.base.FieldNameToIdMapping;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFType;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFViewSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftFacetFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftFeatureNormalizationType;\nimport com.twitter.search.common.schema.thriftjava.ThriftFeatureUpdateConstraint;\nimport com.twitter.search.common.schema.thriftjava.ThriftFieldConfiguration;\nimport com.twitter.search.common.schema.thriftjava.ThriftFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftFixedLengthCSFSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexOptions;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexedFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexedNumericFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftNumericType;\nimport com.twitter.search.common.schema.thriftjava.ThriftSchema;\nimport com.twitter.search.common.schema.thriftjava.ThriftSearchFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftTokenStreamSerializer;\nimport com.twitter.search.common.util.analysis.CharTermAttributeSerializer;\nimport com.twitter.search.common.util.analysis.IntTermAttributeSerializer;\nimport com.twitter.search.common.util.analysis.LongTermAttributeSerializer;\nimport com.twitter.search.common.util.analysis.PayloadAttributeSerializer;\n\npublic class SchemaBuilder {\n\n  public static final String CSF_VIEW_NAME_SEPARATOR = \".\";\n  protected final ThriftSchema schema = new ThriftSchema();\n  protected final FieldNameToIdMapping idMapping;\n  protected final int tokenStreamSerializerVersion;\n\n  // As of now, we do not allow two fields to share the same field name.\n  // This set is used to perform this check.\n  private final Set<String> fieldNameSet = Sets.newHashSet();\n\n  /**\n   * Construct a schema builder with the given FieldNameToIdMapper.\n   * A SchemaBuilder is used to build a ThriftSchema incrementally.\n   */\n  public SchemaBuilder(FieldNameToIdMapping idMapping,\n                       TokenStreamSerializer.Version tokenStreamSerializerVersion) {\n    this.idMapping = idMapping;\n    Preconditions.checkArgument(\n        tokenStreamSerializerVersion == TokenStreamSerializer.Version.VERSION_2);\n    this.tokenStreamSerializerVersion = tokenStreamSerializerVersion.ordinal();\n  }\n\n  /**\n   * Build ThriftSchema using settings accumulated so far.\n   */\n  public final ThriftSchema build() {\n    return schema;\n  }\n\n  /**\n   * Uses fieldName also as facetName.\n   */\n  public final SchemaBuilder withFacetConfigs(String fieldName,\n      boolean storeSkipList,\n      boolean storeOffensiveCounters,\n      boolean useCSFForFacetCounting) {\n    return withFacetConfigs(\n        fieldName,\n        fieldName,\n        storeSkipList,\n        storeOffensiveCounters,\n        useCSFForFacetCounting);\n  }\n\n  /**\n   * Add facet field configuration.\n   */\n  public final SchemaBuilder withFacetConfigs(String fieldName,\n      String facetName,\n      boolean storeSkipList,\n      boolean storeOffensiveCounters,\n      boolean useCSFForFacetCounting) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFacetFieldSettings facetSettings = new ThriftFacetFieldSettings();\n    // As of now, all our facet names are the same as field names\n    facetSettings.setFacetName(facetName);\n    facetSettings.setStoreSkiplist(storeSkipList);\n    facetSettings.setStoreOffensiveCounters(storeOffensiveCounters);\n    facetSettings.setUseCSFForFacetCounting(useCSFForFacetCounting);\n\n    int fieldId = idMapping.getFieldID(fieldName);\n    ThriftFieldConfiguration fieldConfiguration = schema.getFieldConfigs().get(fieldId);\n    Preconditions.checkNotNull(fieldConfiguration,\n        \"In Earlybird, a facet field must be indexed. \"\n            + \"No ThriftIndexedFieldSettings found for field \" + fieldName);\n    fieldConfiguration.getSettings().setFacetFieldSettings(facetSettings);\n    return this;\n  }\n\n  /**\n   * Configure the given field ID to be used for partitioning.\n   */\n  public final SchemaBuilder withPartitionFieldId(int partitionFieldId) {\n    schema.setPartitionFieldId(partitionFieldId);\n    return this;\n  }\n\n  /**\n   * Add a column stride field into schema.\n   */\n  public final SchemaBuilder withColumnStrideField(String fieldName,\n      ThriftCSFType type,\n      int numValuesPerDoc,\n      boolean updatable,\n      boolean loadIntoRam) {\n    return withColumnStrideField(fieldName, type, numValuesPerDoc, updatable, loadIntoRam, null);\n  }\n\n  /**\n   * Add a column stride field into schema that is variable length.\n   */\n  public final SchemaBuilder withBinaryColumnStrideField(String fieldName,\n                                                         boolean loadIntoRam) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftCSFFieldSettings csfFieldSettings = new ThriftCSFFieldSettings();\n    csfFieldSettings.setCsfType(ThriftCSFType.BYTE)\n        .setVariableLength(true)\n        .setLoadIntoRAM(loadIntoRam);\n\n    ThriftFieldSettings fieldSettings =\n        new ThriftFieldSettings().setCsfFieldSettings(csfFieldSettings);\n    ThriftFieldConfiguration fieldConf =\n        new ThriftFieldConfiguration(fieldName).setSettings(fieldSettings);\n    putIntoFieldConfigs(idMapping.getFieldID(fieldName), fieldConf);\n    return this;\n  }\n\n  /**\n   * Add a column stride field into schema which has a default value.\n   */\n  public final SchemaBuilder withColumnStrideField(String fieldName,\n      ThriftCSFType type,\n      int numValuesPerDoc,\n      boolean updatable,\n      boolean loadIntoRam,\n      Long defaultValue) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftCSFFieldSettings csfFieldSettings = new ThriftCSFFieldSettings();\n    csfFieldSettings.setCsfType(type)\n        .setVariableLength(false)\n        .setFixedLengthSettings(\n            new ThriftFixedLengthCSFSettings()\n                .setNumValuesPerDoc(numValuesPerDoc)\n                .setUpdateable(updatable))\n        .setLoadIntoRAM(loadIntoRam);\n\n    if (defaultValue != null) {\n      csfFieldSettings.setDefaultValue(defaultValue);\n    }\n\n    ThriftFieldSettings fieldSettings =\n        new ThriftFieldSettings().setCsfFieldSettings(csfFieldSettings);\n    ThriftFieldConfiguration fieldConf =\n        new ThriftFieldConfiguration(fieldName).setSettings(fieldSettings);\n    putIntoFieldConfigs(idMapping.getFieldID(fieldName), fieldConf);\n    return this;\n  }\n\n  /**\n   * Add a CSF view into schema. A view is a portion of another CSF.\n   */\n  public final SchemaBuilder withColumnStrideFieldView(\n      String fieldName,\n      ThriftCSFType csfType,\n      ThriftCSFType outputCSFType,\n      String baseFieldName,\n      int valueIndex,\n      int bitStartPosition,\n      int bitLength,\n      ThriftFeatureNormalizationType featureNormalizationType,\n      @Nullable Set<ThriftFeatureUpdateConstraint> constraints) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n\n    int baseFieldConfigID = idMapping.getFieldID(baseFieldName);\n\n    ThriftCSFViewSettings csfViewSettings = new ThriftCSFViewSettings()\n            .setBaseFieldConfigId(baseFieldConfigID)\n            .setCsfType(csfType)\n            .setValueIndex(valueIndex)\n            .setBitStartPosition(bitStartPosition)\n            .setBitLength(bitLength);\n    if (outputCSFType != null) {\n      csfViewSettings.setOutputCSFType(outputCSFType);\n    }\n    if (featureNormalizationType != ThriftFeatureNormalizationType.NONE) {\n      csfViewSettings.setNormalizationType(featureNormalizationType);\n    }\n    if (constraints != null) {\n      csfViewSettings.setFeatureUpdateConstraints(constraints);\n    }\n    ThriftFieldSettings fieldSettings = new ThriftFieldSettings()\n            .setCsfViewSettings(csfViewSettings);\n    ThriftFieldConfiguration fieldConf = new ThriftFieldConfiguration(fieldName)\n            .setSettings(fieldSettings);\n\n    Map<Integer, ThriftFieldConfiguration> fieldConfigs = schema.getFieldConfigs();\n    verifyCSFViewSettings(fieldConfigs, fieldConf);\n\n    putIntoFieldConfigs(idMapping.getFieldID(fieldName), fieldConf);\n    return this;\n  }\n\n  /**\n   * Sanity checks for CSF view settings.\n   */\n  public static void verifyCSFViewSettings(Map<Integer, ThriftFieldConfiguration> fieldConfigs,\n      ThriftFieldConfiguration fieldConf) {\n    Preconditions.checkNotNull(fieldConf.getSettings());\n    Preconditions.checkNotNull(fieldConf.getSettings().getCsfViewSettings());\n    ThriftCSFViewSettings csfViewSettings = fieldConf.getSettings().getCsfViewSettings();\n\n    if (fieldConfigs != null) {\n      ThriftFieldConfiguration baseFieldConfig = fieldConfigs.get(\n              csfViewSettings.getBaseFieldConfigId());\n      if (baseFieldConfig != null) {\n        String baseFieldName = baseFieldConfig.getFieldName();\n        String expectedViewNamePrefix = baseFieldName + CSF_VIEW_NAME_SEPARATOR;\n        if (fieldConf.getFieldName().startsWith(expectedViewNamePrefix)) {\n          ThriftFieldSettings baseFieldSettings = baseFieldConfig.getSettings();\n          ThriftCSFFieldSettings baseFieldCSFSettings = baseFieldSettings.getCsfFieldSettings();\n\n          if (baseFieldCSFSettings != null) {\n             if (!baseFieldCSFSettings.isVariableLength()\n                 && baseFieldCSFSettings.getFixedLengthSettings() != null) {\n\n               ThriftCSFType baseCSFType = baseFieldCSFSettings.getCsfType();\n               switch (baseCSFType) {\n                 case BYTE:\n                   checkCSFViewPositions(baseFieldCSFSettings, 8, csfViewSettings);\n                   break;\n                 case INT:\n                   checkCSFViewPositions(baseFieldCSFSettings, 32, csfViewSettings);\n                   break;\n                 default:\n                   throw new IllegalStateException(\"Base field: \" + baseFieldName\n                           + \" is of a non-supported CSFType: \" + baseCSFType);\n               }\n             } else {\n               throw new IllegalStateException(\"Base field: \" + baseFieldName\n                       + \" must be a fixed-length CSF field\");\n             }\n          } else {\n            throw new IllegalStateException(\"Base field: \" + baseFieldName + \" is not a CSF field\");\n          }\n        } else {\n          throw new IllegalStateException(\"View field name for baseFieldConfigID: \"\n                  + csfViewSettings.getBaseFieldConfigId() + \" must start with: '\"\n                  + expectedViewNamePrefix + \"'\");\n        }\n      } else {\n        throw new IllegalStateException(\"Can't add a view, no field defined for base fieldID: \"\n                + csfViewSettings.getBaseFieldConfigId());\n      }\n    } else {\n      throw new IllegalStateException(\"Can't add a view, no field configs defined.\");\n    }\n  }\n\n  private static void checkCSFViewPositions(ThriftCSFFieldSettings baseFieldCSFSettings,\n      int bitsPerValue,\n      ThriftCSFViewSettings csfViewSettings) {\n    ThriftFixedLengthCSFSettings fixedLengthCSFSettings =\n            baseFieldCSFSettings.getFixedLengthSettings();\n    Preconditions.checkNotNull(fixedLengthCSFSettings);\n\n    int numValues = fixedLengthCSFSettings.getNumValuesPerDoc();\n    Preconditions.checkState(csfViewSettings.getValueIndex() >= 0,\n        \"value index must be positive: \" + csfViewSettings.getValueIndex());\n    Preconditions.checkState(csfViewSettings.getValueIndex() < numValues, \"value index \"\n        + csfViewSettings.getValueIndex() + \" must be less than numValues: \" + numValues);\n\n    Preconditions.checkState(csfViewSettings.getBitStartPosition() >= 0,\n        \"bitStartPosition must be positive: \" + csfViewSettings.getBitStartPosition());\n    Preconditions.checkState(csfViewSettings.getBitStartPosition() < bitsPerValue,\n        \"bitStartPosition \" + csfViewSettings.getBitStartPosition()\n            + \" must be less than bitsPerValue \" + bitsPerValue);\n\n    Preconditions.checkState(csfViewSettings.getBitLength() >= 1,\n        \"bitLength must be positive: \" + csfViewSettings.getBitLength());\n\n    Preconditions.checkState(\n        csfViewSettings.getBitStartPosition() + csfViewSettings.getBitLength() <= bitsPerValue,\n        String.format(\"bitStartPosition (%d) + bitLength (%d) must be less than bitsPerValue (%d)\",\n        csfViewSettings.getBitStartPosition(), csfViewSettings.getBitLength(), bitsPerValue));\n  }\n\n  // No position; no freq; not pretokenized; not tokenized.\n  /**\n   * Norm is disabled as default. Like Lucene string field, or int/long fields.\n   */\n  public final SchemaBuilder withIndexedNotTokenizedField(String fieldName) {\n    return withIndexedNotTokenizedField(fieldName, false);\n  }\n\n  /**\n   * Add an indexed but not tokenized field. This is similar to Lucene's StringField.\n   */\n  public final SchemaBuilder withIndexedNotTokenizedField(String fieldName,\n                                                          boolean supportOutOfOrderAppends) {\n    return withIndexedNotTokenizedField(fieldName, supportOutOfOrderAppends, true);\n  }\n\n  private final SchemaBuilder withIndexedNotTokenizedField(String fieldName,\n                                                          boolean supportOutOfOrderAppends,\n                                                          boolean omitNorms) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFieldSettings settings = getNoPositionNoFreqSettings(supportOutOfOrderAppends);\n    settings.getIndexedFieldSettings().setOmitNorms(omitNorms);\n    ThriftFieldConfiguration config = new ThriftFieldConfiguration(fieldName)\n        .setSettings(settings);\n    putIntoFieldConfigs(idMapping.getFieldID(fieldName), config);\n    return this;\n  }\n\n\n  /** Makes the given field searchable by default, with the given weight. */\n  public final SchemaBuilder withSearchFieldByDefault(\n      String fieldName, float textSearchableFieldWeight) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n\n    ThriftFieldSettings settings =\n        schema.getFieldConfigs().get(idMapping.getFieldID(fieldName)).getSettings();\n    settings.setSearchFieldSettings(\n        new ThriftSearchFieldSettings()\n            .setTextSearchableFieldWeight(textSearchableFieldWeight)\n            .setTextDefaultSearchable(true));\n\n    return this;\n  }\n\n  /**\n   * Similar to Lucene's TextField. The string is analyzed using the default/override analyzer.\n   * @param fieldName\n   * @param addHfPairIfHfFieldsArePresent Add hfPair fields if they exists in the schema.\n   *            For certain text fields, adding hfPair fields are usually preferred, but they may\n   *            not exist in the schema, in which case the hfPair fields will not be added.\n   */\n  public final SchemaBuilder withTextField(String fieldName,\n                                           boolean addHfPairIfHfFieldsArePresent) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFieldConfiguration config = new ThriftFieldConfiguration(fieldName).setSettings(\n        getDefaultSettings(ThriftIndexOptions.DOCS_AND_FREQS_AND_POSITIONS));\n\n    if (addHfPairIfHfFieldsArePresent) {\n      // Add hfPair fields only if they exist in the schema for the cluster\n      boolean hfPair = shouldIncludeField(ImmutableSchema.HF_TERM_PAIRS_FIELD)\n                       && shouldIncludeField(ImmutableSchema.HF_PHRASE_PAIRS_FIELD);\n      config.getSettings().getIndexedFieldSettings().setIndexHighFreqTermPairs(hfPair);\n    }\n\n    config.getSettings().getIndexedFieldSettings().setTokenized(true);\n    putIntoFieldConfigs(idMapping.getFieldID(fieldName), config);\n    return this;\n  }\n\n  /**\n   * Marked the given field as having per position payload.\n   */\n  public final SchemaBuilder withPerPositionPayload(String fieldName, int defaultPayloadLength) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFieldSettings settings =\n            schema.getFieldConfigs().get(idMapping.getFieldID(fieldName)).getSettings();\n\n    settings.getIndexedFieldSettings().setStorePerPositionPayloads(true);\n    settings.getIndexedFieldSettings().setDefaultPerPositionPayloadLength(defaultPayloadLength);\n    return this;\n  }\n\n  /**\n   * Add field into schema that is pre-tokenized and does not have position.\n   * E.g. hashtags / stocks / card_domain\n   */\n  public final SchemaBuilder withPretokenizedNoPositionField(String fieldName) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFieldConfiguration config = new ThriftFieldConfiguration(fieldName)\n        .setSettings(getPretokenizedNoPositionFieldSetting());\n    // Add hfPair fields only if they exist in the schema for the cluster\n    boolean hfPair = shouldIncludeField(ImmutableSchema.HF_TERM_PAIRS_FIELD)\n                         && shouldIncludeField(ImmutableSchema.HF_PHRASE_PAIRS_FIELD);\n    config.getSettings().getIndexedFieldSettings().setIndexHighFreqTermPairs(hfPair);\n    putIntoFieldConfigs(idMapping.getFieldID(fieldName), config);\n    return this;\n  }\n\n  /**\n   * Mark the field to have ordered term dictionary.\n   * In Lucene, term dictionary is sorted. In Earlybird, term dictionary order is not\n   * guaranteed unless this is turned on.\n   */\n  public final SchemaBuilder withOrderedTerms(String fieldName) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFieldSettings settings =\n        schema.getFieldConfigs().get(idMapping.getFieldID(fieldName)).getSettings();\n\n    settings.getIndexedFieldSettings().setSupportOrderedTerms(true);\n    return this;\n  }\n\n  /**\n   * Support lookup of term text by term id in the term dictionary.\n   */\n  public final SchemaBuilder withTermTextLookup(String fieldName) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFieldSettings settings =\n        schema.getFieldConfigs().get(idMapping.getFieldID(fieldName)).getSettings();\n\n    settings.getIndexedFieldSettings().setSupportTermTextLookup(true);\n    return this;\n  }\n\n  /**\n   * Add a text field that is pre-tokenized, so not analyzed again in the index (e.g. Earlybird).\n   *\n   * Note that the token streams MUST be created using the attributes defined in\n   * {@link com.twitter.search.common.util.text.TweetTokenStreamSerializer}.\n   */\n  public final SchemaBuilder withPretokenizedTextField(\n      String fieldName,\n      boolean addHfPairIfHfFieldsArePresent) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFieldConfiguration config = new ThriftFieldConfiguration(fieldName)\n        .setSettings(getDefaultPretokenizedSettings(\n            ThriftIndexOptions.DOCS_AND_FREQS_AND_POSITIONS));\n    putIntoFieldConfigs(idMapping.getFieldID(fieldName), config);\n    // Add hfPair fields only if they exist in the schema for the cluster\n    if (addHfPairIfHfFieldsArePresent) {\n      // Add hfPair fields only if they exist in the schema for the cluster\n      boolean hfPair = shouldIncludeField(ImmutableSchema.HF_TERM_PAIRS_FIELD)\n                       && shouldIncludeField(ImmutableSchema.HF_PHRASE_PAIRS_FIELD);\n      config.getSettings().getIndexedFieldSettings().setIndexHighFreqTermPairs(hfPair);\n    }\n    return this;\n  }\n\n  /**\n   * Add a feature configuration\n   */\n  public final SchemaBuilder withFeatureConfiguration(String baseFieldName, String viewName,\n                                                      FeatureConfiguration featureConfiguration) {\n    return withColumnStrideFieldView(\n        viewName,\n        // Defaulting all encoded tweet features to int since the underlying encoded tweet features\n        // are ints.\n        ThriftCSFType.INT,\n        featureConfiguration.getOutputType(),\n        baseFieldName,\n        featureConfiguration.getValueIndex(),\n        featureConfiguration.getBitStartPosition(),\n        featureConfiguration.getBitLength(),\n        featureConfiguration.getFeatureNormalizationType(),\n        featureConfiguration.getUpdateConstraints()\n    );\n  }\n\n  /**\n   * Add a long field in schema. This field uses LongTermAttribute.\n   */\n  private SchemaBuilder addLongTermField(String fieldName, boolean useSortableEncoding) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFieldSettings longTermSettings = getEarlybirdNumericFieldSettings();\n    ThriftTokenStreamSerializer tokenStreamSerializer =\n        new ThriftTokenStreamSerializer(tokenStreamSerializerVersion);\n    tokenStreamSerializer.setAttributeSerializerClassNames(\n        ImmutableList.<String>of(LongTermAttributeSerializer.class.getName()));\n    longTermSettings.getIndexedFieldSettings().setTokenStreamSerializer(tokenStreamSerializer);\n\n    ThriftIndexedNumericFieldSettings numericFieldSettings =\n        new ThriftIndexedNumericFieldSettings(true);\n    numericFieldSettings.setNumericType(ThriftNumericType.LONG);\n    numericFieldSettings.setUseSortableEncoding(useSortableEncoding);\n    longTermSettings.getIndexedFieldSettings().setNumericFieldSettings(numericFieldSettings);\n\n    putIntoFieldConfigs(idMapping.getFieldID(fieldName),\n        new ThriftFieldConfiguration(fieldName).setSettings(longTermSettings));\n    return this;\n  }\n\n  public final SchemaBuilder withSortableLongTermField(String fieldName) {\n    return addLongTermField(fieldName, true);\n  }\n\n  public final SchemaBuilder withLongTermField(String fieldName) {\n    return addLongTermField(fieldName, false);\n  }\n\n  /**\n   * Add an int field in schema. This field uses IntTermAttribute.\n   */\n  public final SchemaBuilder withIntTermField(String fieldName) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFieldSettings intTermSettings = getEarlybirdNumericFieldSettings();\n    ThriftTokenStreamSerializer attributeSerializer =\n        new ThriftTokenStreamSerializer(tokenStreamSerializerVersion);\n    attributeSerializer.setAttributeSerializerClassNames(\n        ImmutableList.<String>of(IntTermAttributeSerializer.class.getName()));\n    intTermSettings.getIndexedFieldSettings().setTokenStreamSerializer(attributeSerializer);\n\n    ThriftIndexedNumericFieldSettings numericFieldSettings =\n        new ThriftIndexedNumericFieldSettings(true);\n    numericFieldSettings.setNumericType(ThriftNumericType.INT);\n    intTermSettings.getIndexedFieldSettings().setNumericFieldSettings(numericFieldSettings);\n\n    putIntoFieldConfigs(idMapping.getFieldID(fieldName),\n        new ThriftFieldConfiguration(fieldName).setSettings(intTermSettings));\n    return this;\n  }\n\n  /**\n   * Timeline and ExpertSearch uses\n   * {@link com.twitter.search.common.util.analysis.PayloadWeightedTokenizer} to store weighted\n   * values.\n   *\n   * E.g. for the PRODUCED_LANGUAGES and CONSUMED_LANGUAGES fields, they contain not a single,\n   * value, but instead a list of values with a weight associated with each value.\n   *\n   * This method adds an indexed field that uses\n   * {@link com.twitter.search.common.util.analysis.PayloadWeightedTokenizer}.\n   */\n  public final SchemaBuilder withCharTermPayloadWeightedField(String fieldName) {\n    ThriftFieldConfiguration config = new ThriftFieldConfiguration(fieldName)\n        .setSettings(getPayloadWeightedSettings(ThriftIndexOptions.DOCS_AND_FREQS_AND_POSITIONS));\n    putIntoFieldConfigs(idMapping.getFieldID(fieldName), config);\n    return this;\n  }\n\n  /**\n   * Set the version and description of this schema.\n   */\n  public final SchemaBuilder withSchemaVersion(\n      int majorVersionNumber,\n      int minorVersionNumber,\n      String versionDesc,\n      boolean isOfficial) {\n    schema.setMajorVersionNumber(majorVersionNumber);\n    schema.setMinorVersionNumber(minorVersionNumber);\n\n    schema.setVersion(majorVersionNumber + \":\" + versionDesc);\n    schema.setVersionIsOfficial(isOfficial);\n\n    return this;\n  }\n\n  public final SchemaBuilder withSchemaVersion(\n      int majorVersionNumber,\n      String versionDesc,\n      boolean isOfficial) {\n    return withSchemaVersion(majorVersionNumber, 0, versionDesc, isOfficial);\n  }\n\n  protected void putIntoFieldConfigs(int id, ThriftFieldConfiguration config) {\n    if (schema.getFieldConfigs() != null && schema.getFieldConfigs().containsKey(id)) {\n      throw new IllegalStateException(\"Already have a ThriftFieldConfiguration for field id \" + id);\n    }\n\n    if (fieldNameSet.contains(config.getFieldName())) {\n      throw new IllegalStateException(\"Already have a ThriftFieldConfiguration for field \"\n          + config.getFieldName());\n    }\n    fieldNameSet.add(config.getFieldName());\n    schema.putToFieldConfigs(id, config);\n  }\n\n  // Default field settings. Most field settings are similar to this.\n  protected ThriftFieldSettings getDefaultSettings(ThriftIndexOptions indexOption) {\n    return getDefaultSettings(indexOption, false);\n  }\n\n  protected ThriftFieldSettings getDefaultSettings(ThriftIndexOptions indexOption,\n                                                   boolean supportOutOfOrderAppends) {\n    ThriftFieldSettings fieldSettings = new ThriftFieldSettings();\n    ThriftIndexedFieldSettings indexedFieldSettings = new ThriftIndexedFieldSettings();\n    indexedFieldSettings\n        .setIndexed(true)\n        .setStored(false)\n        .setTokenized(false)\n        .setStoreTermVectors(false)\n        .setStoreTermVectorOffsets(false)\n        .setStoreTermVectorPayloads(false)\n        .setStoreTermVectorPositions(false)\n        .setSupportOutOfOrderAppends(supportOutOfOrderAppends)\n        .setIndexOptions(indexOption)\n        .setOmitNorms(true); // All Earlybird fields omit norms.\n    fieldSettings.setIndexedFieldSettings(indexedFieldSettings);\n    return fieldSettings;\n  }\n\n  /**\n   * Default field settings for fields that are pretokenized\n   *\n   * The fields that use these settings will need to be tokenized using a serializer with the\n   * attributes defined in {@link com.twitter.search.common.util.text.TweetTokenStreamSerializer}.\n   */\n  protected final ThriftFieldSettings getDefaultPretokenizedSettings(\n      ThriftIndexOptions indexOption) {\n    ThriftFieldSettings fieldSettings = getDefaultSettings(indexOption);\n    fieldSettings.getIndexedFieldSettings().setTokenized(true);\n    ThriftTokenStreamSerializer attributeSerializer =\n        new ThriftTokenStreamSerializer(tokenStreamSerializerVersion);\n    attributeSerializer.setAttributeSerializerClassNames(\n        ImmutableList.<String>of(\n            CharSequenceTermAttributeSerializer.class.getName(),\n            PositionIncrementAttributeSerializer.class.getName(),\n            TokenTypeAttributeSerializer.class.getName()));\n\n    fieldSettings.getIndexedFieldSettings().setTokenStreamSerializer(attributeSerializer);\n    return fieldSettings;\n  }\n\n  protected final ThriftFieldSettings getPretokenizedNoPositionFieldSetting() {\n    return getDefaultPretokenizedSettings(ThriftIndexOptions.DOCS_AND_FREQS);\n  }\n\n  protected final ThriftFieldSettings getNoPositionNoFreqSettings() {\n    return getNoPositionNoFreqSettings(false);\n  }\n\n  protected final ThriftFieldSettings getNoPositionNoFreqSettings(\n      boolean supportOutOfOrderAppends) {\n    return getDefaultSettings(ThriftIndexOptions.DOCS_ONLY, supportOutOfOrderAppends);\n  }\n\n  protected final ThriftFieldSettings getEarlybirdNumericFieldSettings() {\n    // Supposedly numeric fields are not tokenized.\n    // However, Earlybird uses SingleTokenTokenStream to handle int/long fields.\n    // So we need to set indexed to true for these fields.\n    ThriftFieldSettings settings = getNoPositionNoFreqSettings();\n    settings.getIndexedFieldSettings().setTokenized(true);\n    return settings;\n  }\n\n  private ThriftFieldSettings getPayloadWeightedSettings(ThriftIndexOptions indexOption) {\n    ThriftFieldSettings fieldSettings = getDefaultSettings(indexOption);\n    fieldSettings.getIndexedFieldSettings().setTokenized(true);\n    ThriftTokenStreamSerializer attributeSerializer =\n        new ThriftTokenStreamSerializer(tokenStreamSerializerVersion);\n    attributeSerializer.setAttributeSerializerClassNames(\n        ImmutableList.<String>of(CharTermAttributeSerializer.class.getName(),\n            PositionIncrementAttributeSerializer.class.getName(),\n            PayloadAttributeSerializer.class.getName()));\n    fieldSettings.getIndexedFieldSettings().setTokenStreamSerializer(attributeSerializer);\n    return fieldSettings;\n  }\n\n  protected boolean shouldIncludeField(String fieldName) {\n    return true;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/SchemaDocumentFactory.java",
    "content": "package com.twitter.search.common.schema;\n\nimport java.io.IOException;\nimport java.io.StringReader;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Set;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.Sets;\n\nimport org.apache.lucene.analysis.Analyzer;\nimport org.apache.lucene.analysis.TokenStream;\nimport org.apache.lucene.analysis.tokenattributes.CharTermAttribute;\nimport org.apache.lucene.analysis.tokenattributes.TermToBytesRefAttribute;\nimport org.apache.lucene.document.Document;\nimport org.apache.lucene.document.Field;\nimport org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetField;\nimport org.apache.lucene.util.BytesRef;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.text.token.TwitterTokenStream;\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.schema.base.IndexedNumericFieldSettings;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.thriftjava.ThriftDocument;\nimport com.twitter.search.common.schema.thriftjava.ThriftField;\nimport com.twitter.search.common.schema.thriftjava.ThriftFieldData;\nimport com.twitter.search.common.schema.thriftjava.ThriftGeoCoordinate;\nimport com.twitter.search.common.util.analysis.IntTermAttribute;\nimport com.twitter.search.common.util.analysis.LongTermAttribute;\nimport com.twitter.search.common.util.analysis.SortableLongTermAttribute;\nimport com.twitter.search.common.util.spatial.GeoUtil;\nimport com.twitter.search.common.util.text.HighFrequencyTermPairs;\nimport com.twitter.search.common.util.text.OmitNormTextField;\nimport com.twitter.search.common.util.text.SingleTokenStream;\n\n/**\n * A document factory that converts {@link ThriftDocument} into Lucene {@link Document}s\n * using the provided {@link com.twitter.search.common.schema.base.Schema}.\n */\npublic class SchemaDocumentFactory {\n  private static final Logger LOG = LoggerFactory.getLogger(SchemaDocumentFactory.class);\n\n  private final Schema schema;\n  private final ImmutableList<TokenStreamRewriter> tokenStreamRewriters;\n\n  /**\n   * Creates a SchemaDocumentFactory with a schema and the tokenStreamRewriters.\n   *\n   * @param tokenStreamRewriters a list of token stream rewriters, which will be applied in order.\n   */\n  public SchemaDocumentFactory(\n      Schema schema,\n      List<TokenStreamRewriter> tokenStreamRewriters) {\n    this.schema = schema;\n    this.tokenStreamRewriters = ImmutableList.copyOf(tokenStreamRewriters);\n  }\n\n  /**\n   * Creates a SchemaDocumentFactory with no tokenStreamRewriters.\n   */\n  public SchemaDocumentFactory(Schema schema) {\n    this(schema, Collections.EMPTY_LIST);\n  }\n\n  public final Document newDocument(ThriftDocument document) throws IOException {\n    return innerNewDocument(document);\n  }\n\n  /**\n   * Create a Lucene document from the ThriftDocument.\n   */\n  @VisibleForTesting\n  public Document innerNewDocument(ThriftDocument document) throws IOException {\n    Document luceneDocument = new Document();\n    Set<String> hfTerms = Sets.newHashSet();\n    Set<String> hfPhrases = Sets.newHashSet();\n\n    Analyzer defaultAnalyzer = schema.getDefaultAnalyzer(document.getDefaultAnalyzerOverride());\n\n    for (ThriftField field : document.getFields()) {\n      boolean successful = false;\n      try {\n        addLuceneFields(field, defaultAnalyzer, luceneDocument, hfTerms, hfPhrases);\n        successful = true;\n      } finally {\n        if (!successful) {\n          LOG.warn(\"Unexpected exception while trying to add field. Field ID: \"\n              + field.getFieldConfigId() + \" Field Name: \"\n              + schema.getFieldName(field.getFieldConfigId()));\n        }\n      }\n    }\n\n    for (String token : hfTerms) {\n      for (String token2 : hfTerms) {\n        if (token.compareTo(token2) < 0) {\n          luceneDocument.add(new Field(ImmutableSchema.HF_TERM_PAIRS_FIELD,\n                                          HighFrequencyTermPairs.createPair(token, token2),\n                                          OmitNormTextField.TYPE_NOT_STORED));\n        }\n      }\n    }\n\n    for (String phrase : hfPhrases) {\n      // Tokens in the phrase set are not terms and have already been processed with\n      // HighFrequencyTermPairs.createPhrasePair.\n      luceneDocument.add(new Field(ImmutableSchema.HF_PHRASE_PAIRS_FIELD, phrase,\n                                      OmitNormTextField.TYPE_NOT_STORED));\n    }\n\n    return schema.getFacetsConfig().build(luceneDocument);\n  }\n\n  private void addLuceneFields(ThriftField field, Analyzer analyzer, Document doc,\n                               Set<String> hfTerms, Set<String> hfPhrases) throws IOException {\n    Schema.FieldInfo fieldInfo =\n        schema.getFieldInfo(field.getFieldConfigId(), field.getFieldConfigOverride());\n\n    if (fieldInfo == null) {\n      // field not defined in schema - skip it\n      return;\n    }\n\n    ThriftFieldData fieldData = field.getFieldData();\n    if (fieldInfo.getFieldType().getCsfType() !=  null) {\n      addCSFField(doc, fieldInfo, fieldData);\n      return;\n    }\n\n    // Checking which data type is set is not sufficient here. We also need to check schema to\n    // see what the type the field is configured to be. See SEARCH-5173 for more details.\n    // The problem is that Pig, while converting Tuples to Thrift, sets all primitive type\n    // fields to 0. (i.e. the isSet calls will return true).\n    IndexedNumericFieldSettings numericSettings =\n        fieldInfo.getFieldType().getNumericFieldSettings();\n    if (fieldData.isSetTokenStreamValue()) {\n      addTokenField(doc, hfTerms, hfPhrases, fieldInfo, fieldData);\n    } else if (fieldData.isSetStringValue()) {\n      addStringField(analyzer, doc, hfTerms, hfPhrases, fieldInfo, fieldData);\n    } else if (fieldData.isSetBytesValue()) {\n      addBytesField(doc, fieldInfo, fieldData);\n    } else if (fieldData.isSetGeoCoordinate()) {\n      addGeoField(doc, fieldInfo, fieldData);\n    } else if (numericSettings != null) {\n      // handle numeric fields.\n      switch (numericSettings.getNumericType()) {\n        case INT:\n          Preconditions.checkState(fieldData.isSetIntValue(),\n              \"Int field does not have int value set. Field name: %s\", fieldInfo.getName());\n          addIntField(doc, fieldInfo, fieldData);\n          break;\n        case LONG:\n          Preconditions.checkState(fieldData.isSetLongValue(),\n              \"Long field does not have long value set. Field name: %s\", fieldInfo.getName());\n          addLongField(doc, fieldInfo, fieldData);\n          break;\n        case FLOAT:\n          Preconditions.checkState(fieldData.isSetFloatValue(),\n              \"Float field does not have float value set. Field name: %s \", fieldInfo.getName());\n          addFloatField();\n          break;\n        case DOUBLE:\n          Preconditions.checkState(fieldData.isSetDoubleValue(),\n              \"Double field does not have double value set. Field name: %s\", fieldInfo.getName());\n          addDoubleFIeld();\n          break;\n        default:\n          throw new UnsupportedOperationException(\"Earlybird does not know how to handle field \"\n              + field.getFieldConfigId() + \" \" + field);\n      }\n    } else {\n      throw new UnsupportedOperationException(\"Earlybird does not know how to handle field \"\n          + field.getFieldConfigId() + \" \" + field);\n    }\n  }\n\n  private void addCSFField(Document doc, Schema.FieldInfo fieldInfo, ThriftFieldData fieldData) {\n    if (fieldInfo.getFieldType().getCsfFixedLengthNumValuesPerDoc() > 1) {\n\n      // As an optimization, TBinaryProtocol stores a byte array field as a part of a larger byte\n      // array field.  Must call fieldData.getBytesValue().  fieldData.bytesValue.array() will\n      // return extraneous data. See: SEARCH-3996\n      doc.add(new Field(fieldInfo.getName(), fieldData.getBytesValue(), fieldInfo.getFieldType()));\n    } else {\n      doc.add(new CSFField(fieldInfo.getName(), fieldInfo.getFieldType(), fieldData));\n    }\n  }\n\n  private void addTokenField(\n      Document doc,\n      Set<String> hfTerms,\n      Set<String> hfPhrases,\n      Schema.FieldInfo fieldInfo,\n      ThriftFieldData fieldData) throws IOException {\n    TwitterTokenStream twitterTokenStream\n        = fieldInfo.getFieldType().getTokenStreamSerializer().deserialize(\n        fieldData.getTokenStreamValue(), fieldData.getStringValue());\n\n    try {\n      for (TokenStreamRewriter rewriter : tokenStreamRewriters) {\n        twitterTokenStream = rewriter.rewrite(fieldInfo, twitterTokenStream);\n      }\n\n      expandStream(doc, fieldInfo, twitterTokenStream, hfTerms, hfPhrases);\n      doc.add(new Field(fieldInfo.getName(), twitterTokenStream, fieldInfo.getFieldType()));\n    } finally {\n      twitterTokenStream.close();\n    }\n  }\n\n  private void addStringField(Analyzer analyzer, Document doc, Set<String> hfTerms,\n                              Set<String> hfPhrases, Schema.FieldInfo fieldInfo,\n                              ThriftFieldData fieldData) {\n    doc.add(new Field(fieldInfo.getName(), fieldData.getStringValue(), fieldInfo.getFieldType()));\n    if (fieldInfo.getFieldType().tokenized()) {\n      try {\n        TokenStream tokenStream = analyzer.tokenStream(fieldInfo.getName(),\n                new StringReader(fieldData.getStringValue()));\n        try {\n          expandStream(\n              doc,\n              fieldInfo,\n              tokenStream,\n              hfTerms,\n              hfPhrases);\n        } finally {\n          tokenStream.close();\n        }\n      } catch (IOException e) {\n        LOG.error(\"IOException expanding token stream\", e);\n      }\n    } else {\n      addFacetField(doc, fieldInfo, fieldData.getStringValue());\n    }\n  }\n\n  private void addBytesField(Document doc, Schema.FieldInfo fieldInfo, ThriftFieldData fieldData) {\n    doc.add(new Field(fieldInfo.getName(), fieldData.getBytesValue(), fieldInfo.getFieldType()));\n  }\n\n  private void addIntField(Document doc, Schema.FieldInfo fieldInfo,\n                           ThriftFieldData fieldData) {\n    int value = fieldData.getIntValue();\n    addFacetField(doc, fieldInfo, String.valueOf(value));\n\n    if (fieldInfo.getFieldType().getNumericFieldSettings() == null) {\n      // No NumericFieldSettings. Even though the data is numeric, this field is not\n      // really a numerical field. Just add as a string.\n      doc.add(new Field(fieldInfo.getName(), String.valueOf(value), fieldInfo.getFieldType()));\n    } else if (fieldInfo.getFieldType().getNumericFieldSettings().isUseTwitterFormat()) {\n      addIntTermAttributeField(value, fieldInfo, doc);\n    } else {\n      // Use lucene style numerical fields\n      doc.add(NumericField.newIntField(fieldInfo.getName(), value));\n    }\n  }\n\n  private void addIntTermAttributeField(int value,\n                                        Schema.FieldInfo fieldInfo,\n                                        Document doc) {\n    SingleTokenStream singleToken = new SingleTokenStream();\n    IntTermAttribute termAtt = singleToken.addAttribute(IntTermAttribute.class);\n    termAtt.setTerm(value);\n    doc.add(new Field(fieldInfo.getName(), singleToken, fieldInfo.getFieldType()));\n  }\n\n  private void addLongField(Document doc, Schema.FieldInfo fieldInfo,\n                            ThriftFieldData fieldData) {\n    long value = fieldData.getLongValue();\n    addFacetField(doc, fieldInfo, String.valueOf(value));\n\n    if (fieldInfo.getFieldType().getNumericFieldSettings() == null) {\n      // No NumericFieldSettings. Even though the data is numeric, this field is not\n      // really a numerical field. Just add as a string.\n      doc.add(new Field(fieldInfo.getName(), String.valueOf(value), fieldInfo.getFieldType()));\n    } else if (fieldInfo.getFieldType().getNumericFieldSettings().isUseTwitterFormat()) {\n      // Twitter style numerical field: use LongTermAttribute\n      addLongTermAttributeField(value, fieldInfo, doc);\n    } else {\n      // Use lucene style numerical fields\n      doc.add(NumericField.newLongField(fieldInfo.getName(), value));\n    }\n  }\n\n  private void addLongTermAttributeField(long value,\n                                         Schema.FieldInfo fieldInfo,\n                                         Document doc) {\n    SingleTokenStream singleToken = new SingleTokenStream();\n    boolean useSortableEncoding =\n        fieldInfo.getFieldType().getNumericFieldSettings().isUseSortableEncoding();\n\n    if (useSortableEncoding) {\n      SortableLongTermAttribute termAtt = singleToken.addAttribute(SortableLongTermAttribute.class);\n      termAtt.setTerm(value);\n    } else {\n      LongTermAttribute termAtt = singleToken.addAttribute(LongTermAttribute.class);\n      termAtt.setTerm(value);\n    }\n    doc.add(new Field(fieldInfo.getName(), singleToken, fieldInfo.getFieldType()));\n  }\n\n  private void addFloatField() {\n    throw new UnsupportedOperationException(\"Earlybird does not support float values yet.\");\n  }\n\n  private void addDoubleFIeld() {\n    throw new UnsupportedOperationException(\"Earlybird does not support double values yet.\");\n  }\n\n  private void addGeoField(Document doc, Schema.FieldInfo fieldInfo, ThriftFieldData fieldData) {\n    ThriftGeoCoordinate coord = fieldData.getGeoCoordinate();\n    if (GeoUtil.validateGeoCoordinates(coord.getLat(), coord.getLon())) {\n      GeoUtil.fillGeoFields(doc, fieldInfo.getName(),\n          coord.getLat(), coord.getLon(), coord.getAccuracy());\n    }\n  }\n\n  private void addFacetField(Document doc, Schema.FieldInfo fieldInfo, String value) {\n    Preconditions.checkArgument(doc != null);\n    Preconditions.checkArgument(fieldInfo != null);\n    Preconditions.checkArgument(value != null);\n\n    if (fieldInfo.getFieldType().getFacetName() != null) {\n      doc.add(new SortedSetDocValuesFacetField(fieldInfo.getFieldType().getFacetName(), value));\n    }\n  }\n\n  private String getTerm(TermToBytesRefAttribute attr) {\n    if (attr instanceof CharTermAttribute) {\n      return ((CharTermAttribute) attr).toString();\n    } else if (attr instanceof IntTermAttribute) {\n      return String.valueOf(((IntTermAttribute) attr).getTerm());\n    } else if (attr instanceof LongTermAttribute) {\n      return String.valueOf(((LongTermAttribute) attr).getTerm());\n    } else {\n      return attr.getBytesRef().utf8ToString();\n    }\n  }\n\n  /**\n   * Expand the TwitterTokenStream and populate high-frequency terms, phrases and/or facet category paths.\n   */\n  private void expandStream(\n      Document doc,\n      Schema.FieldInfo fieldInfo,\n      TokenStream stream,\n      Set<String> hfTerms,\n      Set<String> hfPhrases) throws IOException {\n    // Checkstyle does not allow assignment to parameters.\n    Set<String> facetHfTerms = hfTerms;\n    Set<String> facetHfPhrases = hfPhrases;\n\n    if (!(HighFrequencyTermPairs.INDEX_HF_TERM_PAIRS\n        && fieldInfo.getFieldType().isIndexHFTermPairs())) {\n      // high-frequency terms and phrases are not needed\n      if (fieldInfo.getFieldType().getFacetName() == null) {\n        // Facets are not needed either, simply return, would do nothing otherwise\n        return;\n      }\n      facetHfTerms = null;\n      facetHfPhrases = null;\n    }\n\n    final TermToBytesRefAttribute attr = stream.getAttribute(TermToBytesRefAttribute.class);\n    stream.reset();\n\n    String lastHFTerm = null;\n    while (stream.incrementToken()) {\n      String term = getTerm(attr);\n      if (fieldInfo.getFieldType().getFacetName() != null) {\n        addFacetField(doc, fieldInfo, term);\n      }\n      if (HighFrequencyTermPairs.HF_TERM_SET.contains(term)) {\n        if (facetHfTerms != null) {\n          facetHfTerms.add(term);\n        }\n        if (lastHFTerm != null) {\n          if (facetHfPhrases != null) {\n            facetHfPhrases.add(HighFrequencyTermPairs.createPhrasePair(lastHFTerm, term));\n          }\n        }\n        lastHFTerm = term;\n      } else {\n        lastHFTerm = null;\n      }\n    }\n  }\n\n  public static final class CSFField extends Field {\n    /**\n     * Create a CSFField with the given fieldType, containing the given field data.\n     */\n    public CSFField(String name, EarlybirdFieldType fieldType, ThriftFieldData data) {\n      super(name, fieldType);\n\n      if (fieldType.isCsfVariableLength()) {\n        fieldsData = new BytesRef(data.getBytesValue());\n      } else {\n        switch (fieldType.getCsfType()) {\n          case BYTE:\n            fieldsData = Long.valueOf(data.getByteValue());\n            break;\n          case INT:\n            fieldsData = Long.valueOf(data.getIntValue());\n            break;\n          case LONG:\n            fieldsData = Long.valueOf(data.getLongValue());\n            break;\n          case FLOAT:\n            fieldsData = Long.valueOf(Float.floatToRawIntBits((float) data.getFloatValue()));\n            break;\n          case DOUBLE:\n            fieldsData = Long.valueOf(Double.doubleToRawLongBits(data.getDoubleValue()));\n            break;\n          default:\n            throw new IllegalArgumentException(\"Unknown csf type: \" + fieldType.getCsfType());\n        }\n      }\n    }\n  }\n\n  public interface TokenStreamRewriter {\n    /**\n     * Rewrite the token stream.\n     */\n    TwitterTokenStream rewrite(Schema.FieldInfo fieldInfo, TwitterTokenStream stream);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/SchemaUtil.java",
    "content": "package com.twitter.search.common.schema;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.DocValuesType;\nimport org.apache.lucene.index.IndexOptions;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.IndexedNumericFieldSettings;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFType;\nimport com.twitter.search.common.schema.thriftjava.ThriftNumericType;\nimport com.twitter.search.common.util.analysis.IntTermAttributeImpl;\nimport com.twitter.search.common.util.analysis.LongTermAttributeImpl;\nimport com.twitter.search.common.util.analysis.SortableLongTermAttributeImpl;\n\npublic final class SchemaUtil {\n  private SchemaUtil() {\n  }\n\n  /**\n   * Get the a fixed CSF field's number of values per doc.\n   * @param schema the Schema for the index\n   * @param fieldId the field id the CSF field - the field must be of binary integer type and\n   *                in fixed size\n   * @return the number of values per doc\n   */\n  public static int getCSFFieldFixedLength(ImmutableSchemaInterface schema, int fieldId) {\n    final Schema.FieldInfo fieldInfo = Preconditions.checkNotNull(schema.getFieldInfo(fieldId));\n    return getCSFFieldFixedLength(fieldInfo);\n  }\n\n  /**\n   * Get the a fixed CSF field's number of values per doc.\n   * @param schema the Schema for the index\n   * @param fieldName the field name of the CSF field - the field must be of binary integer type\n   *                  and in fixed size\n   * @return the number of values per doc\n   */\n  public static int getCSFFieldFixedLength(ImmutableSchemaInterface schema, String fieldName) {\n    final Schema.FieldInfo fieldInfo = Preconditions.checkNotNull(schema.getFieldInfo(fieldName));\n    return getCSFFieldFixedLength(fieldInfo);\n  }\n\n  /**\n   * Get the a fixed CSF field's number of values per doc.\n   * @param fieldInfo the field of the CSF field - the field must be of binary integer type\n   *                  and in fixed size\n   * @return the number of values per doc\n   */\n  public static int getCSFFieldFixedLength(Schema.FieldInfo fieldInfo) {\n    final EarlybirdFieldType fieldType = fieldInfo.getFieldType();\n    Preconditions.checkState(fieldType.docValuesType() == DocValuesType.BINARY\n        && fieldType.getCsfType() == ThriftCSFType.INT);\n    return fieldType.getCsfFixedLengthNumValuesPerDoc();\n  }\n\n  /** Converts the given value to a BytesRef instance, according to the type of the given field. */\n  public static BytesRef toBytesRef(Schema.FieldInfo fieldInfo, String value) {\n    EarlybirdFieldType fieldType = fieldInfo.getFieldType();\n    Preconditions.checkArgument(fieldType.indexOptions() != IndexOptions.NONE);\n    IndexedNumericFieldSettings numericSetting = fieldType.getNumericFieldSettings();\n    if (numericSetting != null) {\n      if (!numericSetting.isUseTwitterFormat()) {\n        throw new UnsupportedOperationException(\n            \"Numeric field not using Twitter format: cannot drill down.\");\n      }\n\n      ThriftNumericType numericType = numericSetting.getNumericType();\n      switch (numericType) {\n        case INT:\n          try {\n            return IntTermAttributeImpl.copyIntoNewBytesRef(Integer.parseInt(value));\n          } catch (NumberFormatException e) {\n            throw new UnsupportedOperationException(\n                String.format(\"Cannot parse value for int field %s: %s\",\n                              fieldInfo.getName(), value),\n                e);\n          }\n        case LONG:\n          try {\n            return numericSetting.isUseSortableEncoding()\n                ? SortableLongTermAttributeImpl.copyIntoNewBytesRef(Long.parseLong(value))\n                : LongTermAttributeImpl.copyIntoNewBytesRef(Long.parseLong(value));\n          } catch (NumberFormatException e) {\n            throw new UnsupportedOperationException(\n                String.format(\"Cannot parse value for long field %s: %s\",\n                              fieldInfo.getName(), value),\n                e);\n          }\n        default:\n          throw new UnsupportedOperationException(\n              String.format(\"Unsupported numeric type for field %s: %s\",\n                            fieldInfo.getName(), numericType));\n      }\n    }\n\n    return new BytesRef(value);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/SearchWhitespaceAnalyzer.java",
    "content": "package com.twitter.search.common.schema;\n\nimport org.apache.lucene.analysis.Analyzer;\nimport org.apache.lucene.analysis.core.WhitespaceTokenizer;\n\n/**\n * The majority of the code is copied from Lucene 3.1 analysis.core.WhitespaceAnalyzer. The only\n * new code is the getPositionIncrementGap()\n */\npublic final class SearchWhitespaceAnalyzer extends Analyzer {\n  @Override\n  protected TokenStreamComponents createComponents(final String fieldName) {\n    return new TokenStreamComponents(new WhitespaceTokenizer());\n  }\n\n  /**\n   * Make sure that phrase queries do not match across 2 instances of the text field.\n   *\n   * See the Javadoc for Analyzer.getPositionIncrementGap() for a good explanation of how this\n   * method works.\n   */\n  @Override\n  public int getPositionIncrementGap(String fieldName) {\n    // Hard-code \"text\" here, because we can't depend on EarlybirdFieldConstants.\n    return \"text\".equals(fieldName) ? 1 : super.getPositionIncrementGap(fieldName);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/ThriftDocumentBuilder.java",
    "content": "package com.twitter.search.common.schema;\n\nimport java.io.IOException;\nimport java.util.List;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\n\nimport javax.annotation.Nullable;\n\nimport com.twitter.common.text.util.PositionIncrementAttributeSerializer;\nimport com.twitter.common.text.util.TokenStreamSerializer;\nimport com.twitter.search.common.schema.base.FieldNameToIdMapping;\nimport com.twitter.search.common.schema.thriftjava.ThriftDocument;\nimport com.twitter.search.common.schema.thriftjava.ThriftField;\nimport com.twitter.search.common.schema.thriftjava.ThriftFieldData;\nimport com.twitter.search.common.schema.thriftjava.ThriftGeoCoordinate;\nimport com.twitter.search.common.util.analysis.CharTermAttributeSerializer;\nimport com.twitter.search.common.util.analysis.LongTermAttributeSerializer;\nimport com.twitter.search.common.util.analysis.LongTermsTokenStream;\nimport com.twitter.search.common.util.analysis.PayloadAttributeSerializer;\nimport com.twitter.search.common.util.analysis.PayloadWeightedTokenizer;\nimport com.twitter.search.common.util.spatial.GeoUtil;\n\n/**\n * Builder class for building ThriftDocuments.\n */\npublic class ThriftDocumentBuilder {\n  private static final Logger LOG = Logger.getLogger(ThriftDocumentBuilder.class.getName());\n\n  protected final ThriftDocument doc = new ThriftDocument();\n  protected final FieldNameToIdMapping idMapping;\n\n  private static final ThreadLocal<TokenStreamSerializer> PAYLOAD_WEIGHTED_SERIALIZER_PER_THREAD =\n      new ThreadLocal<TokenStreamSerializer>() {\n        @Override\n        protected TokenStreamSerializer initialValue() {\n          return TokenStreamSerializer.builder()\n              .add(new CharTermAttributeSerializer())\n              .add(new PositionIncrementAttributeSerializer())\n              .add(new PayloadAttributeSerializer())\n              .build();\n        }\n      };\n\n  private static final ThreadLocal<TokenStreamSerializer> LONG_TERM_SERIALIZER_PER_THREAD =\n          new ThreadLocal<TokenStreamSerializer>() {\n            @Override\n            protected TokenStreamSerializer initialValue() {\n              return TokenStreamSerializer.builder()\n                  .add(new LongTermAttributeSerializer())\n                  .build();\n            }\n          };\n\n  public ThriftDocumentBuilder(FieldNameToIdMapping idMapping) {\n    this.idMapping = idMapping;\n  }\n\n  protected void prepareToBuild() {\n    // left empty, subclass can override this.\n  }\n\n  public ThriftDocument build() {\n    prepareToBuild();\n    return doc;\n  }\n\n  /**\n   * Add a long field. This is indexed as a\n   * {@link com.twitter.search.common.util.analysis.LongTermAttribute}\n   */\n  public final ThriftDocumentBuilder withLongField(String fieldName, long value) {\n    ThriftFieldData fieldData = new ThriftFieldData().setLongValue(value);\n    ThriftField field = new ThriftField()\n        .setFieldConfigId(idMapping.getFieldID(fieldName)).setFieldData(fieldData);\n    doc.addToFields(field);\n    return this;\n  }\n\n  /**\n   * Add an int field. This is indexed as a\n   * {@link com.twitter.search.common.util.analysis.IntTermAttribute}\n   */\n  public final ThriftDocumentBuilder withIntField(String fieldName, int value) {\n    ThriftFieldData fieldData = new ThriftFieldData().setIntValue(value);\n    ThriftField field = new ThriftField()\n        .setFieldConfigId(idMapping.getFieldID(fieldName)).setFieldData(fieldData);\n    doc.addToFields(field);\n    return this;\n  }\n\n  /**\n   * Add a field whose value is a single byte.\n   */\n  public final ThriftDocumentBuilder withByteField(String fieldName, byte value) {\n    ThriftFieldData fieldData = new ThriftFieldData().setByteValue(value);\n    ThriftField field = new ThriftField()\n        .setFieldConfigId(idMapping.getFieldID(fieldName)).setFieldData(fieldData);\n    doc.addToFields(field);\n    return this;\n  }\n\n  /**\n   * Add a field whose value is a byte array.\n   */\n  public final ThriftDocumentBuilder withBytesField(String fieldName, byte[] value) {\n    ThriftFieldData fieldData = new ThriftFieldData().setBytesValue(value);\n    ThriftField field = new ThriftField()\n        .setFieldConfigId(idMapping.getFieldID(fieldName)).setFieldData(fieldData);\n    doc.addToFields(field);\n    return this;\n  }\n\n  /**\n   * Add a field whose value is a float.\n   */\n  public final ThriftDocumentBuilder withFloatField(String fieldName, float value) {\n    ThriftFieldData fieldData = new ThriftFieldData().setFloatValue(value);\n    ThriftField field = new ThriftField()\n        .setFieldConfigId(idMapping.getFieldID(fieldName)).setFieldData(fieldData);\n    doc.addToFields(field);\n    return this;\n  }\n\n  /**\n   * Added a field whose value is a Lucene TokenStream.\n   * The Lucene TokenStream is serialized using Twitter's\n   * {@link com.twitter.common.text.util.TokenStreamSerializer}\n   */\n  public final ThriftDocumentBuilder withTokenStreamField(String fieldName,\n                                                          @Nullable String tokenStreamText,\n                                                          byte[] tokenStream) {\n    if (tokenStream == null) {\n      return this;\n    }\n    ThriftFieldData fieldData = new ThriftFieldData()\n        .setStringValue(tokenStreamText).setTokenStreamValue(tokenStream);\n    ThriftField field = new ThriftField()\n        .setFieldConfigId(idMapping.getFieldID(fieldName)).setFieldData(fieldData);\n    doc.addToFields(field);\n    return this;\n  }\n\n  /**\n   * Add a field whose value is a String.\n   * @param fieldName Name of the field where the string will be added.\n   * @param text This string is indexed as is (not analyzed).\n   */\n  public final ThriftDocumentBuilder withStringField(String fieldName, String text) {\n    if (text == null || text.isEmpty()) {\n      return this;\n    }\n\n    ThriftFieldData fieldData = new ThriftFieldData().setStringValue(text);\n    ThriftField field = new ThriftField()\n        .setFieldConfigId(idMapping.getFieldID(fieldName)).setFieldData(fieldData);\n    doc.addToFields(field);\n    return this;\n  }\n\n  /**\n   * Add a field whose value is a geo coordinate.\n   * Earlybird will process the coordinates into geo hashes before indexing.\n   */\n  public final ThriftDocumentBuilder withGeoField(String fieldName,\n                                                  double lat, double lon, int acc) {\n    if (!GeoUtil.validateGeoCoordinates(lat, lon)) {\n      // If the geo coordinates are invalid, don't add any field.\n      return this;\n    }\n    ThriftGeoCoordinate coord = new ThriftGeoCoordinate();\n    coord.setLat(lat);\n    coord.setLon(lon);\n    coord.setAccuracy(acc);\n\n    ThriftFieldData fieldData = new ThriftFieldData().setGeoCoordinate(coord);\n    ThriftField field = new ThriftField()\n        .setFieldConfigId(idMapping.getFieldID(fieldName)).setFieldData(fieldData);\n    doc.addToFields(field);\n    return this;\n  }\n\n  /**\n   * Added a list of tokens that are weighted. The weights are stored inside payload.\n   * See {@link com.twitter.search.common.util.analysis.PayloadWeightedTokenizer} for more details.\n   */\n  public final ThriftDocumentBuilder withPayloadWeightTokenStreamField(String fieldName,\n                                                                       String tokens) {\n    byte[] serialized;\n    try {\n      PayloadWeightedTokenizer tokenizer = new PayloadWeightedTokenizer(tokens);\n      serialized = PAYLOAD_WEIGHTED_SERIALIZER_PER_THREAD.get().serialize(tokenizer);\n      tokenizer.close();\n    } catch (IOException e) {\n      LOG.log(Level.WARNING,\n          \"Failed to add PayloadWeightedTokenizer field. Bad token weight list: \" + tokens, e);\n      return this;\n    } catch (NumberFormatException e) {\n      LOG.log(Level.WARNING,\n          \"Failed to add PayloadWeightedTokenizer field. Cannot parse token weight: \" + tokens, e);\n      return this;\n    }\n    withTokenStreamField(fieldName, tokens, serialized);\n    return this;\n  }\n\n  /**\n   * Add a field whose value is a list of longs.\n   * Each long is encoded into a LongTermAttribute.\n   * The field will contain a LongTermTokenStream.\n   */\n  public final ThriftDocumentBuilder withLongIDsField(String fieldName,\n      List<Long> longList)  throws IOException {\n\n    if (longList == null || longList.isEmpty()) {\n        return this;\n    }\n    LongTermsTokenStream stream = new LongTermsTokenStream(longList);\n    stream.reset();\n    byte[] serializedStream = LONG_TERM_SERIALIZER_PER_THREAD.get().serialize(stream);\n\n    ThriftFieldData fieldData = new ThriftFieldData().setTokenStreamValue(serializedStream);\n    ThriftField field = new ThriftField()\n        .setFieldConfigId(idMapping.getFieldID(fieldName)).setFieldData(fieldData);\n    doc.addToFields(field);\n    return this;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/base/BUILD",
    "content": "# Library for Schema.java and other utilities with minimal dependencies.\njava_library(\n    name = \"base\",\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    provides = artifact(\n        org = \"com.twitter.search.common\",\n        name = \"schema-base\",\n        repo = artifactory,\n    ),\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/code/findbugs:jsr305\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/commons-lang\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-common\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-facet\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/text/util:token-util\",\n        \"src/thrift/com/twitter/search/common:features-java\",\n        \"src/thrift/com/twitter/search/common:schema-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/base/EarlybirdFieldType.java",
    "content": "package com.twitter.search.common.schema.base;\n\nimport javax.annotation.Nullable;\n\nimport org.apache.commons.lang.StringUtils;\nimport org.apache.lucene.document.FieldType;\nimport org.apache.lucene.index.DocValuesType;\nimport org.apache.lucene.index.IndexOptions;\n\nimport com.twitter.common.text.util.TokenStreamSerializer;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFType;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFViewSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftFeatureUpdateConstraint;\n\n/**\n * An extension of Lucene's {@link FieldType} that contains additional Earlybird-specific settings.\n * Lucene IndexingChains can downcast the FieldType object to access these additional settings.\n */\npublic class EarlybirdFieldType extends FieldType {\n\n  public static final EarlybirdFieldType LONG_CSF_FIELD_TYPE = new EarlybirdFieldType();\n  public static final EarlybirdFieldType INT_CSF_FIELD_TYPE = new EarlybirdFieldType();\n  public static final EarlybirdFieldType BYTE_CSF_FIELD_TYPE = new EarlybirdFieldType();\n\n  static {\n    LONG_CSF_FIELD_TYPE.setCsfType(ThriftCSFType.LONG);\n    LONG_CSF_FIELD_TYPE.setDocValuesType(DocValuesType.NUMERIC);\n    LONG_CSF_FIELD_TYPE.setCsfLoadIntoRam(true);\n    LONG_CSF_FIELD_TYPE.freeze();\n\n    INT_CSF_FIELD_TYPE.setCsfType(ThriftCSFType.INT);\n    INT_CSF_FIELD_TYPE.setDocValuesType(DocValuesType.NUMERIC);\n    INT_CSF_FIELD_TYPE.setCsfLoadIntoRam(true);\n    INT_CSF_FIELD_TYPE.freeze();\n\n    BYTE_CSF_FIELD_TYPE.setCsfType(ThriftCSFType.BYTE);\n    BYTE_CSF_FIELD_TYPE.setDocValuesType(DocValuesType.NUMERIC);\n    BYTE_CSF_FIELD_TYPE.setCsfLoadIntoRam(true);\n    BYTE_CSF_FIELD_TYPE.freeze();\n  }\n\n\n  private boolean storePerPositionPayloads;\n  private int defaultPayloadLength;\n  // This is true for fields that become immutable after optimization\n  private boolean becomesImmutable = true;\n  private boolean supportOrderedTerms;\n  private boolean supportTermTextLookup;\n  private boolean indexHFTermPairs;\n\n  /**\n   * This flag turns on tweet specific normalizations.\n   * This turns on the following two token processors:\n   * {@link com.twitter.search.common.util.text.splitter.HashtagMentionPunctuationSplitter}\n   * {@link com.twitter.search.common.util.text.filter.NormalizedTokenFilter}\n   *\n   * HashtagMentionPunctuationSplitter would break a mention or hashtag like @ab_cd or #ab_cd into\n   * tokens {ab, cd}.\n   * NormalizedTokenFilter strips out the # @ $ from the tokens.\n   *\n   *\n   * @deprecated we should remove this flag. It is confusing to have Earlybird apply additional\n   * tokenization on top of what ingester produced.\n   */\n  @Deprecated\n  private boolean useTweetSpecificNormalization;\n\n  @Nullable\n  private TokenStreamSerializer.Builder tokenStreamSerializerProvider = null;\n\n  // csf type settings\n  private ThriftCSFType csfType;\n  private boolean csfVariableLength;\n  private int csfFixedLengthNumValuesPerDoc;\n  private boolean csfFixedLengthUpdateable;\n  private boolean csfLoadIntoRam;\n  private boolean csfDefaultValueSet;\n  private long csfDefaultValue;\n  // True if this is a CSF field which is a view on top of a different CSF field\n  private boolean csfViewField;\n  // If this field is a csf view, this is the ID of the CSF field backing the view\n  private int csfViewBaseFieldId;\n  private FeatureConfiguration csfViewFeatureConfiguration;\n\n  // facet field settings\n  private String facetName;\n  private boolean storeFacetSkiplist;\n  private boolean storeFacetOffensiveCounters;\n  private boolean useCSFForFacetCounting;\n\n  // Determines if this field is indexed\n  private boolean indexedField = false;\n\n  // search field settings\n  // whether a field should be searched by default\n  private boolean textSearchableByDefault = false;\n  private float textSearchableFieldWeight = 1.0f;\n\n  // For indexed numerical fields\n  private IndexedNumericFieldSettings numericFieldSettings = null;\n\n  public boolean isStorePerPositionPayloads() {\n    return storePerPositionPayloads;\n  }\n\n  public void setStorePerPositionPayloads(boolean storePerPositionPayloads) {\n    checkIfFrozen();\n    this.storePerPositionPayloads = storePerPositionPayloads;\n  }\n\n  public int getDefaultPayloadLength() {\n    return defaultPayloadLength;\n  }\n\n  public void setDefaultPayloadLength(int defaultPayloadLength) {\n    checkIfFrozen();\n    this.defaultPayloadLength = defaultPayloadLength;\n  }\n\n  public boolean becomesImmutable() {\n    return becomesImmutable;\n  }\n\n  public void setBecomesImmutable(boolean becomesImmutable) {\n    checkIfFrozen();\n    this.becomesImmutable = becomesImmutable;\n  }\n\n  public boolean isSupportOrderedTerms() {\n    return supportOrderedTerms;\n  }\n\n  public void setSupportOrderedTerms(boolean supportOrderedTerms) {\n    checkIfFrozen();\n    this.supportOrderedTerms = supportOrderedTerms;\n  }\n\n  public boolean isSupportTermTextLookup() {\n    return supportTermTextLookup;\n  }\n\n  public void setSupportTermTextLookup(boolean supportTermTextLookup) {\n    this.supportTermTextLookup = supportTermTextLookup;\n  }\n\n  @Nullable\n  public TokenStreamSerializer getTokenStreamSerializer() {\n    return tokenStreamSerializerProvider == null ? null : tokenStreamSerializerProvider.safeBuild();\n  }\n\n  public void setTokenStreamSerializerBuilder(TokenStreamSerializer.Builder provider) {\n    checkIfFrozen();\n    this.tokenStreamSerializerProvider = provider;\n  }\n\n  public ThriftCSFType getCsfType() {\n    return csfType;\n  }\n\n  public void setCsfType(ThriftCSFType csfType) {\n    checkIfFrozen();\n    this.csfType = csfType;\n  }\n\n  public boolean isCsfVariableLength() {\n    return csfVariableLength;\n  }\n\n  public int getCsfFixedLengthNumValuesPerDoc() {\n    return csfFixedLengthNumValuesPerDoc;\n  }\n\n  public void setCsfVariableLength() {\n    checkIfFrozen();\n    this.csfVariableLength = true;\n  }\n\n  /**\n   * Make the field a fixed length CSF, with the given length.\n   */\n  public void setCsfFixedLengthSettings(int csfFixedLengthNumValuesPerDocument,\n                                        boolean isCsfFixedLengthUpdateable) {\n    checkIfFrozen();\n    this.csfVariableLength = false;\n    this.csfFixedLengthNumValuesPerDoc = csfFixedLengthNumValuesPerDocument;\n    this.csfFixedLengthUpdateable = isCsfFixedLengthUpdateable;\n  }\n\n  public boolean isCsfFixedLengthUpdateable() {\n    return csfFixedLengthUpdateable;\n  }\n\n  public boolean isCsfLoadIntoRam() {\n    return csfLoadIntoRam;\n  }\n\n  public void setCsfLoadIntoRam(boolean csfLoadIntoRam) {\n    checkIfFrozen();\n    this.csfLoadIntoRam = csfLoadIntoRam;\n  }\n\n  public void setCsfDefaultValue(long defaultValue) {\n    checkIfFrozen();\n    this.csfDefaultValue = defaultValue;\n    this.csfDefaultValueSet = true;\n  }\n\n  public long getCsfDefaultValue() {\n    return csfDefaultValue;\n  }\n\n  public boolean isCsfDefaultValueSet() {\n    return csfDefaultValueSet;\n  }\n\n  public String getFacetName() {\n    return facetName;\n  }\n\n  public void setFacetName(String facetName) {\n    checkIfFrozen();\n    this.facetName = facetName;\n  }\n\n  public boolean isStoreFacetSkiplist() {\n    return storeFacetSkiplist;\n  }\n\n  public void setStoreFacetSkiplist(boolean storeFacetSkiplist) {\n    checkIfFrozen();\n    this.storeFacetSkiplist = storeFacetSkiplist;\n  }\n\n  public boolean isStoreFacetOffensiveCounters() {\n    return storeFacetOffensiveCounters;\n  }\n\n  public void setStoreFacetOffensiveCounters(boolean storeFacetOffensiveCounters) {\n    checkIfFrozen();\n    this.storeFacetOffensiveCounters = storeFacetOffensiveCounters;\n  }\n\n  public boolean isUseCSFForFacetCounting() {\n    return useCSFForFacetCounting;\n  }\n\n  public void setUseCSFForFacetCounting(boolean useCSFForFacetCounting) {\n    checkIfFrozen();\n    this.useCSFForFacetCounting = useCSFForFacetCounting;\n  }\n\n  public boolean isFacetField() {\n    return facetName != null && !StringUtils.isEmpty(facetName);\n  }\n\n  public boolean isIndexHFTermPairs() {\n    return indexHFTermPairs;\n  }\n\n  public void setIndexHFTermPairs(boolean indexHFTermPairs) {\n    checkIfFrozen();\n    this.indexHFTermPairs = indexHFTermPairs;\n  }\n\n  public boolean acceptPretokenizedField() {\n    return tokenStreamSerializerProvider != null;\n  }\n\n  /**\n   * set this field to use additional twitter specific tokenization.\n   * @deprecated should avoid doing additional tokenizations on top of what ingester produced.\n   */\n  @Deprecated\n  public boolean useTweetSpecificNormalization() {\n    return useTweetSpecificNormalization;\n  }\n\n  /**\n   * test whether this field uses additional twitter specific tokenization.\n   * @deprecated should avoid doing additional tokenizations on top of what ingester produced.\n   */\n  @Deprecated\n  public void setUseTweetSpecificNormalization(boolean useTweetSpecificNormalization) {\n    checkIfFrozen();\n    this.useTweetSpecificNormalization = useTweetSpecificNormalization;\n  }\n\n  public boolean isIndexedField() {\n    return indexedField;\n  }\n\n  public void setIndexedField(boolean indexedField) {\n    this.indexedField = indexedField;\n  }\n\n  public boolean isTextSearchableByDefault() {\n    return textSearchableByDefault;\n  }\n\n  public void setTextSearchableByDefault(boolean textSearchableByDefault) {\n    checkIfFrozen();\n    this.textSearchableByDefault = textSearchableByDefault;\n  }\n\n  public float getTextSearchableFieldWeight() {\n    return textSearchableFieldWeight;\n  }\n\n  public void setTextSearchableFieldWeight(float textSearchableFieldWeight) {\n    checkIfFrozen();\n    this.textSearchableFieldWeight = textSearchableFieldWeight;\n  }\n\n  /**\n   * Convenience method to find out if this field stores positions. {@link #indexOptions()} can also\n   * be used to determine the index options for this field.\n   */\n  public final boolean hasPositions() {\n    return indexOptions() == IndexOptions.DOCS_AND_FREQS_AND_POSITIONS\n            || indexOptions() == IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS;\n  }\n\n  public boolean isCsfViewField() {\n    return csfViewField;\n  }\n\n  public int getCsfViewBaseFieldId() {\n    return csfViewBaseFieldId;\n  }\n\n  public FeatureConfiguration getCsfViewFeatureConfiguration() {\n    return csfViewFeatureConfiguration;\n  }\n\n  /**\n   * Set the CSF view settings. A CSF view is a portion of an another CSF.\n   */\n  public void setCsfViewSettings(String fieldName,\n                                 ThriftCSFViewSettings csfViewSettings,\n                                 Schema.FieldInfo baseField) {\n    checkIfFrozen();\n    this.csfViewField = true;\n    this.csfViewBaseFieldId = csfViewSettings.getBaseFieldConfigId();\n    FeatureConfiguration.Builder builder = FeatureConfiguration.builder()\n            .withName(fieldName)\n            .withType(csfViewSettings.csfType)\n            .withBitRange(csfViewSettings.getValueIndex(),\n                csfViewSettings.getBitStartPosition(),\n                csfViewSettings.getBitLength())\n            .withBaseField(baseField.getName());\n    if (csfViewSettings.isSetOutputCSFType()) {\n      builder.withOutputType(csfViewSettings.getOutputCSFType());\n    }\n    if (csfViewSettings.isSetNormalizationType()) {\n      builder.withFeatureNormalizationType(csfViewSettings.getNormalizationType());\n    }\n    if (csfViewSettings.isSetFeatureUpdateConstraints()) {\n      for (ThriftFeatureUpdateConstraint c : csfViewSettings.getFeatureUpdateConstraints()) {\n        builder.withFeatureUpdateConstraint(c);\n      }\n    }\n\n    this.csfViewFeatureConfiguration = builder.build();\n  }\n\n  public IndexedNumericFieldSettings getNumericFieldSettings() {\n    return numericFieldSettings;\n  }\n\n  public void setNumericFieldSettings(IndexedNumericFieldSettings numericFieldSettings) {\n    checkIfFrozen();\n    this.numericFieldSettings = numericFieldSettings;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/base/FeatureConfiguration.java",
    "content": "package com.twitter.search.common.schema.base;\n\nimport java.util.Set;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Sets;\n\nimport com.twitter.common.base.MorePreconditions;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFType;\nimport com.twitter.search.common.schema.thriftjava.ThriftFeatureNormalizationType;\nimport com.twitter.search.common.schema.thriftjava.ThriftFeatureUpdateConstraint;\n\n// FeatureConfiguration is defined for all the column stride view fields.\npublic final class FeatureConfiguration {\n  private final String name;\n  private final int intIndex;\n  // Start position in the given int (0-31)\n  private final int bitStartPos;\n  // Length in bits of the feature\n  private final int bitLength;\n  // precomputed for reuse\n  private final int bitMask;\n  private final int inverseBitMask;\n  private final int maxValue;\n\n  private final ThriftCSFType type;\n\n  // This is the client seen feature type: if this is null, this field is unused.\n  @Nullable\n  private final ThriftCSFType outputType;\n\n  private final String baseField;\n\n  private final Set<FeatureConstraint> featureUpdateConstraints;\n\n  private final ThriftFeatureNormalizationType featureNormalizationType;\n\n  /**\n   * Creates a new FeatureConfiguration with a base field.\n   *\n   * @param intIndex which integer is the feature in (0 based).\n   * @param bitStartPos at which bit does the feature start (0-31).\n   * @param bitLength length in bits of the feature\n   * @param baseField the CSF this feature is stored within.\n   */\n  private FeatureConfiguration(\n          String name,\n          ThriftCSFType type,\n          ThriftCSFType outputType,\n          int intIndex,\n          int bitStartPos,\n          int bitLength,\n          String baseField,\n          Set<FeatureConstraint> featureUpdateConstraints,\n          ThriftFeatureNormalizationType featureNormalizationType) {\n    Preconditions.checkState(bitStartPos + bitLength <= Integer.SIZE,\n            \"Feature must not cross int boundary.\");\n    this.name = MorePreconditions.checkNotBlank(name);\n    this.type = Preconditions.checkNotNull(type);\n    this.outputType = outputType;\n    this.intIndex = intIndex;\n    this.bitStartPos = bitStartPos;\n    this.bitLength = bitLength;\n    // Technically, int-sized features can use all 32 bits to store a positive value greater than\n    // Integer.MAX_VALUE. But in practice, we will convert the values of those features to Java ints\n    // on the read side, so the max value for those features will still be Integer.MAX_VALUE.\n    this.maxValue = (1 << Math.min(bitLength, Integer.SIZE - 1)) - 1;\n    this.bitMask = (int) (((1L << bitLength) - 1) << bitStartPos);\n    this.inverseBitMask = ~bitMask;\n    this.baseField = baseField;\n    this.featureUpdateConstraints = featureUpdateConstraints;\n    this.featureNormalizationType = Preconditions.checkNotNull(featureNormalizationType);\n  }\n\n  public String getName() {\n    return name;\n  }\n\n  public int getMaxValue() {\n    return maxValue;\n  }\n\n  @Override\n  public String toString() {\n    return new StringBuilder().append(name)\n            .append(\" (\").append(intIndex).append(\", \")\n            .append(bitStartPos).append(\", \")\n            .append(bitLength).append(\") \").toString();\n  }\n\n  public int getValueIndex() {\n    return intIndex;\n  }\n\n  public int getBitStartPosition() {\n    return bitStartPos;\n  }\n\n  public int getBitLength() {\n    return bitLength;\n  }\n\n  public int getBitMask() {\n    return bitMask;\n  }\n\n  public int getInverseBitMask() {\n    return inverseBitMask;\n  }\n\n  public String getBaseField() {\n    return baseField;\n  }\n\n  public ThriftCSFType getType() {\n    return type;\n  }\n\n  @Nullable\n  public ThriftCSFType getOutputType() {\n    return outputType;\n  }\n\n  public ThriftFeatureNormalizationType getFeatureNormalizationType() {\n    return featureNormalizationType;\n  }\n\n  /**\n   * Returns the update constraint for the feature.\n   */\n  public Set<ThriftFeatureUpdateConstraint> getUpdateConstraints() {\n    if (featureUpdateConstraints == null) {\n      return null;\n    }\n    Set<ThriftFeatureUpdateConstraint> constraintSet = Sets.newHashSet();\n    for (FeatureConstraint constraint : featureUpdateConstraints) {\n      constraintSet.add(constraint.getType());\n    }\n    return constraintSet;\n  }\n\n  /**\n   * Returns true if the given update satisfies all feature update constraints.\n   */\n  public boolean validateFeatureUpdate(final Number oldValue, final Number newValue) {\n    if (featureUpdateConstraints != null) {\n      for (FeatureConstraint contraint : featureUpdateConstraints) {\n        if (!contraint.apply(oldValue, newValue)) {\n          return false;\n        }\n      }\n    }\n\n    return true;\n  }\n\n  @Override\n  public int hashCode() {\n    return (name == null ? 0 : name.hashCode())\n        + intIndex * 7\n        + bitStartPos * 13\n        + bitLength * 23\n        + bitMask * 31\n        + inverseBitMask * 43\n        + (int) maxValue * 53\n        + (type == null ? 0 : type.hashCode()) * 61\n        + (outputType == null ? 0 : outputType.hashCode()) * 71\n        + (baseField == null ? 0 : baseField.hashCode()) * 83\n        + (featureUpdateConstraints == null ? 0 : featureUpdateConstraints.hashCode()) * 87\n        + (featureNormalizationType == null ? 0 : featureNormalizationType.hashCode()) * 97;\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof FeatureConfiguration)) {\n      return false;\n    }\n\n    FeatureConfiguration featureConfiguration = FeatureConfiguration.class.cast(obj);\n    return (name == featureConfiguration.name)\n        && (bitStartPos == featureConfiguration.bitStartPos)\n        && (bitLength == featureConfiguration.bitLength)\n        && (bitMask == featureConfiguration.bitMask)\n        && (inverseBitMask == featureConfiguration.inverseBitMask)\n        && (maxValue == featureConfiguration.maxValue)\n        && (type == featureConfiguration.type)\n        && (outputType == featureConfiguration.outputType)\n        && (baseField == featureConfiguration.baseField)\n        && (featureUpdateConstraints == null\n            ? featureConfiguration.featureUpdateConstraints == null\n            : featureUpdateConstraints.equals(featureConfiguration.featureUpdateConstraints))\n        && (featureNormalizationType == null\n            ? featureConfiguration.featureNormalizationType == null\n            : featureNormalizationType.equals(featureConfiguration.featureNormalizationType));\n  }\n\n  private interface FeatureConstraint {\n    boolean apply(Number oldValue, Number newValue);\n    ThriftFeatureUpdateConstraint getType();\n  }\n\n  public static Builder builder() {\n    return new Builder();\n  }\n\n  public static final class Builder {\n    private String name;\n    private ThriftCSFType type;\n    private ThriftCSFType outputType;\n    private int intIndex;\n    // Start position in the given int (0-31)\n    private int bitStartPos;\n    // Length in bits of the feature\n    private int bitLength;\n\n    private String baseField;\n\n    private Set<FeatureConstraint> featureUpdateConstraints;\n\n    private ThriftFeatureNormalizationType featureNormalizationType =\n        ThriftFeatureNormalizationType.NONE;\n\n    public FeatureConfiguration build() {\n      return new FeatureConfiguration(name, type, outputType, intIndex, bitStartPos, bitLength,\n              baseField, featureUpdateConstraints, featureNormalizationType);\n    }\n\n    public Builder withName(String n) {\n      this.name = n;\n      return this;\n    }\n\n    public Builder withType(ThriftCSFType featureType) {\n      this.type = featureType;\n      return this;\n    }\n\n    public Builder withOutputType(ThriftCSFType featureFeatureType) {\n      this.outputType = featureFeatureType;\n      return this;\n    }\n\n    public Builder withFeatureNormalizationType(\n        ThriftFeatureNormalizationType normalizationType) {\n      this.featureNormalizationType = Preconditions.checkNotNull(normalizationType);\n      return this;\n    }\n\n    /**\n     * Sets the bit range at the given intIndex, startPos and length.\n     */\n    public Builder withBitRange(int index, int startPos, int length) {\n      this.intIndex = index;\n      this.bitStartPos = startPos;\n      this.bitLength = length;\n      return this;\n    }\n\n    public Builder withBaseField(String baseFieldName) {\n      this.baseField = baseFieldName;\n      return this;\n    }\n\n    /**\n     * Adds a feature update constraint.\n     */\n    public Builder withFeatureUpdateConstraint(final ThriftFeatureUpdateConstraint constraint) {\n      if (featureUpdateConstraints == null) {\n        featureUpdateConstraints = Sets.newHashSet();\n      }\n\n      switch (constraint) {\n        case IMMUTABLE:\n          featureUpdateConstraints.add(new FeatureConstraint() {\n            @Override public boolean apply(Number oldValue, Number newValue) {\n              return false;\n            }\n            @Override public ThriftFeatureUpdateConstraint getType() {\n              return ThriftFeatureUpdateConstraint.IMMUTABLE;\n            }\n          });\n          break;\n        case INC_ONLY:\n          featureUpdateConstraints.add(new FeatureConstraint() {\n            @Override  public boolean apply(Number oldValue, Number newValue) {\n              return newValue.intValue() > oldValue.intValue();\n            }\n            @Override public ThriftFeatureUpdateConstraint getType() {\n              return ThriftFeatureUpdateConstraint.INC_ONLY;\n            }\n          });\n          break;\n        case POSITIVE:\n          featureUpdateConstraints.add(new FeatureConstraint() {\n            @Override  public boolean apply(Number oldValue, Number newValue) {\n              return newValue.intValue() >= 0;\n            }\n            @Override public ThriftFeatureUpdateConstraint getType() {\n              return ThriftFeatureUpdateConstraint.POSITIVE;\n            }\n          });\n          break;\n        default:\n      }\n\n      return this;\n    }\n\n    private Builder() {\n\n    }\n  }\n}\n\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/base/FieldNameToIdMapping.java",
    "content": "package com.twitter.search.common.schema.base;\n\nimport java.util.Map;\n\nimport com.google.common.collect.ImmutableMap;\n\n/**\n * Maps from fieldName to fieldIDs.\n */\npublic abstract class FieldNameToIdMapping {\n  /**\n   * Returns field ID for the given fieldName.\n   * Can throw unchecked exceptions is the fieldName is not known to Earlybird.\n   */\n  public abstract int getFieldID(String fieldName);\n\n  /**\n   * Wrap the given map into a fieldNameToIdMapping instance.\n   */\n  public static FieldNameToIdMapping newFieldNameToIdMapping(Map<String, Integer> map) {\n    final ImmutableMap<String, Integer> immutableMap = ImmutableMap.copyOf(map);\n    return new FieldNameToIdMapping() {\n      @Override public int getFieldID(String fieldName) {\n        return immutableMap.get(fieldName);\n      }\n    };\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/base/FieldWeightDefault.java",
    "content": "package com.twitter.search.common.schema.base;\n\nimport java.util.LinkedHashMap;\nimport java.util.Map;\n\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Maps;\n\nimport static com.google.common.base.Preconditions.checkNotNull;\n\n/**\n * Records whether a field's enabled for search by default and its default weight. Note that these\n * two are decoupled -- a field can have a default weight but not enabled for search by default.\n * In a query it can be enabled by an annotation that does not specify a weight (e.g., \":f:foo\"),\n * which would then use the default weight.\n *\n * Instances are mutable.\n */\npublic class FieldWeightDefault {\n  private final boolean enabled;\n  private final float weight;\n\n  public FieldWeightDefault(boolean enabled, float weight) {\n    this.enabled = enabled;\n    this.weight = weight;\n  }\n\n  public static FieldWeightDefault fromSignedWeight(float signedValue) {\n    return new FieldWeightDefault(signedValue >= 0, Math.abs(signedValue));\n  }\n\n  /**\n   * Returns an immutable map from field name to default field weights for only enabled fields.\n   * Fields that are not enabled for search by default will not be included.\n   */\n  public static <T> ImmutableMap<T, Float> getOnlyEnabled(\n      Map<T, FieldWeightDefault> map) {\n\n    ImmutableMap.Builder<T, Float> builder = ImmutableMap.builder();\n    for (Map.Entry<T, FieldWeightDefault> entry : map.entrySet()) {\n      if (entry.getValue().isEnabled()) {\n        builder.put(entry.getKey(), entry.getValue().getWeight());\n      }\n    }\n    return builder.build();\n  }\n\n  public boolean isEnabled() {\n    return enabled;\n  }\n\n  public float getWeight() {\n    return weight;\n  }\n\n  /**\n   * Overlays the base field-weight map with the given one. Since it is an overlay, a\n   * field that does not exist in the base map will never be added. Also, negative value means\n   * the field is not enabled for search by default, but if it is, the absolute value would serve as\n   * the default.\n   */\n  public static ImmutableMap<String, FieldWeightDefault> overrideFieldWeightMap(\n      Map<String, FieldWeightDefault> base,\n      Map<String, Double> fieldWeightMapOverride) {\n\n    checkNotNull(base);\n    if (fieldWeightMapOverride == null) {\n      return ImmutableMap.copyOf(base);\n    }\n\n    LinkedHashMap<String, FieldWeightDefault> map = Maps.newLinkedHashMap(base);\n    for (Map.Entry<String, Double> entry : fieldWeightMapOverride.entrySet()) {\n      if (base.containsKey(entry.getKey())\n          && entry.getValue() >= -Float.MAX_VALUE\n          && entry.getValue() <= Float.MAX_VALUE) {\n\n        map.put(\n            entry.getKey(),\n            FieldWeightDefault.fromSignedWeight(entry.getValue().floatValue()));\n      }\n    }\n\n    return ImmutableMap.copyOf(map);\n  }\n\n  /**\n   * Creates a field-to-FieldWeightDefault map from the given field-to-weight map, where negative\n   * weight means the the field is not enabled for search by default, but if it is (e.g.,\n   * by annotation), the absolute value of the weight shall be used.\n   */\n  public static <T> ImmutableMap<T, FieldWeightDefault> fromSignedWeightMap(\n      Map<T, ? extends Number> signedWeightMap) {\n\n    ImmutableMap.Builder<T, FieldWeightDefault> builder = ImmutableMap.builder();\n    for (Map.Entry<T, ? extends Number> entry : signedWeightMap.entrySet()) {\n      // If double to float conversion failed, we will get a float infinity.\n      // See http://stackoverflow.com/a/10075093/716468\n      float floatValue = entry.getValue().floatValue();\n      if (floatValue != Float.NEGATIVE_INFINITY\n          && floatValue != Float.POSITIVE_INFINITY) {\n\n        builder.put(\n            entry.getKey(),\n            FieldWeightDefault.fromSignedWeight(floatValue));\n      }\n    }\n\n    return builder.build();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/base/ImmutableSchemaInterface.java",
    "content": "package com.twitter.search.common.schema.base;\n\nimport javax.annotation.concurrent.Immutable;\nimport javax.annotation.concurrent.ThreadSafe;\n\n/**\n * This interface carries the same signature as Schema with the only difference that this schema\n * is immutable.  This should be used by short sessions and the class would guarantee the schema\n * would not change for the session.  A typical usage is like a search query session.\n */\n@Immutable\n@ThreadSafe\npublic interface ImmutableSchemaInterface extends Schema {\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/base/IndexedNumericFieldSettings.java",
    "content": "package com.twitter.search.common.schema.base;\n\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexedNumericFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftNumericType;\n\npublic class IndexedNumericFieldSettings {\n  private final ThriftNumericType numericType;\n  private final int numericPrecisionStep;\n  private final boolean useTwitterFormat;\n  private final boolean useSortableEncoding;\n\n  /**\n   * Create a IndexedNumericFieldSettings from a ThriftIndexedNumericFieldSettings\n   */\n  public IndexedNumericFieldSettings(ThriftIndexedNumericFieldSettings numericFieldSettings) {\n    this.numericType            = numericFieldSettings.getNumericType();\n    this.numericPrecisionStep   = numericFieldSettings.getNumericPrecisionStep();\n    this.useTwitterFormat       = numericFieldSettings.isUseTwitterFormat();\n    this.useSortableEncoding    = numericFieldSettings.isUseSortableEncoding();\n  }\n\n  public ThriftNumericType getNumericType() {\n    return numericType;\n  }\n\n  public int getNumericPrecisionStep() {\n    return numericPrecisionStep;\n  }\n\n  public boolean isUseTwitterFormat() {\n    return useTwitterFormat;\n  }\n\n  public boolean isUseSortableEncoding() {\n    return useSortableEncoding;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/base/Schema.java",
    "content": "package com.twitter.search.common.schema.base;\n\nimport java.util.Collection;\nimport java.util.Map;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Predicate;\nimport com.google.common.collect.ImmutableCollection;\nimport com.google.common.collect.ImmutableMap;\n\nimport org.apache.lucene.analysis.Analyzer;\nimport org.apache.lucene.facet.FacetsConfig;\nimport org.apache.lucene.index.FieldInfos;\n\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchema;\nimport com.twitter.search.common.schema.thriftjava.ThriftAnalyzer;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFType;\nimport com.twitter.search.common.schema.thriftjava.ThriftFieldConfiguration;\n\n/**\n * Search Schema.\n */\npublic interface Schema {\n  /**\n   * Certain Schema implementations can evolve at run time.  This call returns a snapshot of\n   * of the schema which is guaranteed to not change.\n   */\n  ImmutableSchemaInterface getSchemaSnapshot();\n\n  /**\n   * Returns a string describing the current schema version.\n   */\n  String getVersionDescription();\n\n  /**\n   * Returns whether the schema version is official. Only official segments are uploaded to HDFS.\n   */\n  boolean isVersionOfficial();\n\n  /**\n   * Returns the schema's major version.\n   */\n  int getMajorVersionNumber();\n\n  /**\n   * Returns the schema's minor version.\n   */\n  int getMinorVersionNumber();\n\n  /**\n   * Returns the default analyzer. This analyzer is used when none is specified on the field info.\n   */\n  Analyzer getDefaultAnalyzer(ThriftAnalyzer override);\n\n  /**\n   * Returns whether the given field is configured in the schema.\n   */\n  boolean hasField(int fieldConfigId);\n\n  /**\n   * Returns whether the given field is configured in the schema.\n   */\n  boolean hasField(String fieldName);\n\n  /**\n   * Get the field name corresponding to the given field id.\n   */\n  String getFieldName(int fieldConfigId);\n\n  /**\n   * Return the FieldInfo of all fields.\n   */\n  ImmutableCollection<FieldInfo> getFieldInfos();\n\n  /**\n   * Get the field info for the given field id. If an override is given, attempt to merge the\n   * base field info with the override config.\n   */\n  FieldInfo getFieldInfo(int fieldConfigId, ThriftFieldConfiguration override);\n\n\n  /**\n   * Get the field info for the given field id. No override.\n   */\n  @Nullable\n  FieldInfo getFieldInfo(int fieldConfigId);\n\n  /**\n   * Get the field info for the given field name. No override.\n   */\n  @Nullable\n  FieldInfo getFieldInfo(String fieldName);\n\n  /**\n   * Builds a lucene FieldInfos instance, usually used for indexing.\n   */\n  FieldInfos getLuceneFieldInfos(Predicate<String> acceptedFields);\n\n  /**\n   * Returns the number of facet fields in this schema.\n   */\n  int getNumFacetFields();\n\n  /**\n   * Return facet configurations.\n   */\n  FacetsConfig getFacetsConfig();\n\n  /**\n   * Get the facet field's field info by facet name.\n   */\n  FieldInfo getFacetFieldByFacetName(String facetName);\n\n  /**\n   * Get the facet field's field info by field name.\n   */\n  FieldInfo getFacetFieldByFieldName(String fieldName);\n\n  /**\n   * Get the field infos for all facet fields.\n   */\n  Collection<FieldInfo> getFacetFields();\n\n  /**\n   * Get the field infos for all facet fields backed by column stride fields.\n   */\n  Collection<FieldInfo> getCsfFacetFields();\n\n  /**\n   * Get the field weight map for text searchable fields.\n   */\n  Map<String, FieldWeightDefault> getFieldWeightMap();\n\n  /**\n   * Get scoring feature configuration by feature name.\n   */\n  FeatureConfiguration getFeatureConfigurationByName(String featureName);\n\n  /**\n   * Get scoring feature configuration by feature field id.  The feature configuration is\n   * guaranteed to be not null, or a NullPointerException will be thrown out.\n   */\n  FeatureConfiguration getFeatureConfigurationById(int featureFieldId);\n\n  /**\n   * Returns the ThriftCSFType for a CSF field.\n   * Note: for non-CSF field, null will be returned.\n   */\n  @Nullable\n  ThriftCSFType getCSFFieldType(String fieldName);\n\n  /**\n   * Get the search result feature schema for all possible features in all search results.\n   *\n   * The returned value is not really immutable (because it's a pre-generated thrift struct).\n   * We want to return it directly because we want to pre-build it once and return with the thrift\n   * search results as is.\n   */\n  ThriftSearchFeatureSchema getSearchFeatureSchema();\n\n  /**\n   * Get the mapping from feature id to feature configuration.\n   */\n  ImmutableMap<Integer, FeatureConfiguration> getFeatureIdToFeatureConfig();\n\n  /**\n   * Get the mapping from feature name to feature configuration.\n   */\n  ImmutableMap<String, FeatureConfiguration> getFeatureNameToFeatureConfig();\n\n  /**\n   * Field configuration for a single field.\n   */\n  final class FieldInfo {\n    private final int fieldId;\n    private final String name;\n    private final EarlybirdFieldType luceneFieldType;\n\n    public FieldInfo(int fieldId, String name, EarlybirdFieldType luceneFieldType) {\n      this.fieldId = fieldId;\n      this.name = name;\n      this.luceneFieldType = luceneFieldType;\n    }\n\n    public int getFieldId() {\n      return fieldId;\n    }\n\n    public String getName() {\n      return name;\n    }\n\n    public EarlybirdFieldType getFieldType() {\n      return luceneFieldType;\n    }\n\n    public String getDescription() {\n      return String.format(\n          \"(FieldInfo [fieldId: %d, name: %s, luceneFieldType: %s])\",\n          fieldId, name, luceneFieldType.getFacetName()\n      );\n    }\n\n    @Override\n    public boolean equals(Object obj) {\n      if (!(obj instanceof FieldInfo)) {\n        return false;\n      }\n      return fieldId == ((FieldInfo) obj).fieldId;\n    }\n\n    @Override\n    public int hashCode() {\n      return fieldId;\n    }\n  }\n\n  /**\n   * Exception thrown when errors or inconsistences are detected in a search schema.\n   */\n  final class SchemaValidationException extends Exception {\n    public SchemaValidationException(String msg) {\n      super(msg);\n    }\n\n    public SchemaValidationException(String msg, Exception e) {\n      super(msg, e);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/base/ThriftDocumentUtil.java",
    "content": "package com.twitter.search.common.schema.base;\n\nimport java.util.ArrayList;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Set;\n\nimport com.twitter.search.common.schema.thriftjava.ThriftDocument;\nimport com.twitter.search.common.schema.thriftjava.ThriftField;\n\n/**\n * Utility APIs for ThriftDocument.\n */\npublic final class ThriftDocumentUtil {\n  private ThriftDocumentUtil() {\n  }\n\n  /**\n   * Get ThriftField out of a ThriftDocument.\n   */\n  public static ThriftField getField(ThriftDocument thriftDoc,\n                                     String fieldName,\n                                     FieldNameToIdMapping idMap) {\n    int id = idMap.getFieldID(fieldName);\n    for (ThriftField field : thriftDoc.getFields()) {\n      int fieldId = field.getFieldConfigId();\n      if (fieldId == id) {\n        return field;\n      }\n    }\n\n    return null;\n  }\n\n  /**\n   * Get all fields out of a ThriftDocument that match the given field name.\n   */\n  public static List<ThriftField> getFields(\n      ThriftDocument thriftDoc, String fieldName, FieldNameToIdMapping idMap) {\n\n    int id = idMap.getFieldID(fieldName);\n    List<ThriftField> result = new ArrayList<>();\n\n    for (ThriftField field : thriftDoc.getFields()) {\n      int fieldId = field.getFieldConfigId();\n      if (fieldId == id) {\n        result.add(field);\n      }\n    }\n\n    return result;\n  }\n\n\n  /**\n   * Retrieve the long value from a thrift field\n   */\n  public static long getLongValue(ThriftDocument thriftDoc,\n                                  String fieldName,\n                                  FieldNameToIdMapping idMap) {\n    ThriftField f = getField(thriftDoc, fieldName, idMap);\n    return f == null ? 0L : f.getFieldData().getLongValue();\n  }\n\n  /**\n   * Retrieve the byte value from a thrift field\n   */\n  public static byte getByteValue(ThriftDocument thriftDoc,\n                                  String fieldName,\n                                  FieldNameToIdMapping idMap) {\n    ThriftField f = getField(thriftDoc, fieldName, idMap);\n    return f == null ? (byte) 0 : f.getFieldData().getByteValue();\n  }\n\n  /**\n   * Retrieve the bytes value from a thrift field\n   */\n  public static byte[] getBytesValue(ThriftDocument thriftDoc,\n                                     String fieldName,\n                                     FieldNameToIdMapping idMap) {\n    ThriftField f = getField(thriftDoc, fieldName, idMap);\n    return f == null ? null : f.getFieldData().getBytesValue();\n  }\n\n  /**\n   * Retrieve the int value from a thrift field\n   */\n  public static int getIntValue(ThriftDocument thriftDoc,\n                                String fieldName,\n                                FieldNameToIdMapping idMap) {\n    ThriftField f = getField(thriftDoc, fieldName, idMap);\n    return f == null ? 0 : f.getFieldData().getIntValue();\n  }\n\n  /**\n   * Retrieve the string value from a thrift field\n   */\n  public static String getStringValue(ThriftDocument thriftDoc,\n                                      String fieldName,\n                                      FieldNameToIdMapping idMap) {\n    ThriftField f = getField(thriftDoc, fieldName, idMap);\n    return f == null ? null : f.getFieldData().getStringValue();\n  }\n\n  /**\n   * Retrieve the string values from all thrift fields with the given fieldName.\n   */\n  public static List<String> getStringValues(\n      ThriftDocument thriftDoc,\n      String fieldName,\n      FieldNameToIdMapping idMap) {\n    List<ThriftField> fields = getFields(thriftDoc, fieldName, idMap);\n    List<String> fieldStrings = new ArrayList<>();\n\n    for (ThriftField field : fields) {\n      fieldStrings.add(field.getFieldData().getStringValue());\n    }\n    return fieldStrings;\n  }\n\n  /**\n   * Returns whether the specified document has duplicate fields.\n   */\n  public static boolean hasDuplicateFields(ThriftDocument thriftDoc) {\n    Set<Integer> seen = new HashSet<>();\n    for (ThriftField field : thriftDoc.getFields()) {\n      if (!seen.add(field.getFieldConfigId())) {\n        return true;\n      }\n    }\n    return false;\n  }\n\n  /**\n   * Get ThriftField out of a ThriftDocument.\n   */\n  public static ThriftField getField(ThriftDocument thriftDoc, int fieldId) {\n    for (ThriftField field : thriftDoc.getFields()) {\n      if (field.getFieldConfigId() == fieldId) {\n        return field;\n      }\n    }\n\n    return null;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/earlybird/BUILD",
    "content": "# Library for earlybird-specific schema.\njava_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/twitter/elephantbird:core\",\n        \"3rdparty/jvm/geo/google:geoGoogle\",\n        \"3rdparty/jvm/joda-time\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"cuad/projects/ner/thrift/src/main/thrift:thrift-java\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/collections\",\n        \"src/java/com/twitter/common/text/token\",\n        \"src/java/com/twitter/common/text/util:token-util\",\n        \"src/java/com/twitter/common_internal/text\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/search/common/config\",\n        \"src/java/com/twitter/search/common/constants\",\n        \"src/java/com/twitter/search/common/encoding/docvalues\",\n        \"src/java/com/twitter/search/common/encoding/features\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/partitioning/snowflakeparser\",\n        \"src/java/com/twitter/search/common/schema\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/util:longintconverter\",\n        \"src/java/com/twitter/search/common/util/analysis\",\n        \"src/java/com/twitter/search/common/util/spatial\",\n        \"src/java/com/twitter/search/common/util/text\",\n        \"src/java/com/twitter/search/common/util/text/regex\",\n        \"src/java/com/twitter/search/common/util/url\",\n        \"src/thrift/com/twitter/search/common:indexing-java\",\n        \"src/thrift/com/twitter/search/common:schema-java\",\n        \"src/thrift/com/twitter/service/spiderduck/gen:metadata-store-java\",\n    ],\n    exports = [\n        \"src/thrift/com/twitter/search/common:indexing-java\",\n    ],\n)\n\njava_library(\n    name = \"for-timelines\",\n    sources = [\n        \"EarlybirdCluster.java\",\n        \"EarlybirdFieldConstants.java\",\n    ],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/twitter/elephantbird:core\",\n        \"3rdparty/jvm/geo/google:geoGoogle\",\n        \"3rdparty/jvm/joda-time\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"cuad/projects/ner/thrift/src/main/thrift:thrift-java\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/collections\",\n        \"src/java/com/twitter/common/text/token\",\n        \"src/java/com/twitter/common/text/util:token-util\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/search/common/config\",\n        \"src/java/com/twitter/search/common/constants\",\n        \"src/java/com/twitter/search/common/encoding/docvalues\",\n        \"src/java/com/twitter/search/common/encoding/features\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/partitioning/snowflakeparser\",\n        \"src/java/com/twitter/search/common/schema\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/util:longintconverter\",\n        \"src/java/com/twitter/search/common/util/analysis\",\n        \"src/java/com/twitter/search/common/util/spatial\",\n        \"src/java/com/twitter/search/common/util/text\",\n        \"src/java/com/twitter/search/common/util/text/regex\",\n        \"src/java/com/twitter/search/common/util/url\",\n        \"src/thrift/com/twitter/search/common:indexing-java\",\n        \"src/thrift/com/twitter/search/common:schema-java\",\n        \"src/thrift/com/twitter/service/spiderduck/gen:metadata-store-java\",\n    ],\n    exports = [\n        \"src/thrift/com/twitter/search/common:indexing-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/earlybird/EarlybirdCluster.java",
    "content": "package com.twitter.search.common.schema.earlybird;\n\nimport java.util.Set;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.ImmutableSet;\n\n/**\n * A list of existing Earlybird clusters.\n */\npublic enum EarlybirdCluster {\n  /**\n   * Realtime earlybird cluster. Has 100% of tweet for about 7 days.\n   */\n  REALTIME,\n  /**\n   * Protected earlybird cluster. Has only tweets from protected accounts.\n   */\n  PROTECTED,\n  /**\n   * Full archive cluster. Has all tweets until about 2 days ago.\n   */\n  FULL_ARCHIVE,\n  /**\n   * SuperRoot cluster. Talks to the other clusters instead of talking directly to earlybirds.\n   */\n  SUPERROOT,\n\n  /**\n   * A dedicated cluster for Candidate Generation use cases based on Earlybird in Home/PushService\n   */\n  REALTIME_CG;\n\n  public String getNameForStats() {\n    return name().toLowerCase();\n  }\n\n  public static boolean isArchive(EarlybirdCluster cluster) {\n    return isClusterInSet(cluster, ARCHIVE_CLUSTERS);\n  }\n\n  public static boolean isTwitterMemoryFormatCluster(EarlybirdCluster cluster) {\n    return isClusterInSet(cluster, TWITTER_IN_MEMORY_INDEX_FORMAT_GENERAL_PURPOSE_CLUSTERS);\n  }\n\n  public static boolean hasEarlybirds(EarlybirdCluster cluster) {\n    return cluster != SUPERROOT;\n  }\n\n  private static boolean isClusterInSet(EarlybirdCluster cluster, Set<EarlybirdCluster> set) {\n    return set.contains(cluster);\n  }\n\n  protected static final ImmutableSet<EarlybirdCluster> ARCHIVE_CLUSTERS =\n      ImmutableSet.of(FULL_ARCHIVE);\n\n  @VisibleForTesting\n  public static final ImmutableSet<EarlybirdCluster>\n          TWITTER_IN_MEMORY_INDEX_FORMAT_GENERAL_PURPOSE_CLUSTERS =\n      ImmutableSet.of(\n          REALTIME,\n          PROTECTED);\n\n  @VisibleForTesting\n  public static final ImmutableSet<EarlybirdCluster> TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS =\n      ImmutableSet.of(\n          REALTIME,\n          PROTECTED,\n          REALTIME_CG);\n\n  /**\n   * Constant for field used in general purpose clusters,\n   * Note that GENERAL_PURPOSE_CLUSTERS does not include REALTIME_CG. If you wish to include REALTIME_CG,\n   * please use ALL_CLUSTERS\n   */\n  protected static final ImmutableSet<EarlybirdCluster> GENERAL_PURPOSE_CLUSTERS =\n      ImmutableSet.of(\n          REALTIME,\n          PROTECTED,\n          FULL_ARCHIVE,\n          SUPERROOT);\n\n  protected static final ImmutableSet<EarlybirdCluster> ALL_CLUSTERS =\n      ImmutableSet.of(\n          REALTIME,\n          PROTECTED,\n          FULL_ARCHIVE,\n          SUPERROOT,\n          REALTIME_CG);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/earlybird/EarlybirdEncodedFeatures.java",
    "content": "package com.twitter.search.common.schema.earlybird;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.encoding.features.IntegerEncodedFeatures;\nimport com.twitter.search.common.indexing.thriftjava.PackedFeatures;\nimport com.twitter.search.common.indexing.thriftjava.VersionedTweetFeatures;\nimport com.twitter.search.common.schema.SchemaUtil;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\n\n/**\n * A class for encoding earlybird features in integers\n */\npublic abstract class EarlybirdEncodedFeatures extends IntegerEncodedFeatures {\n  private final ImmutableSchemaInterface schema;\n  private final EarlybirdFieldConstant baseField;\n\n  public EarlybirdEncodedFeatures(ImmutableSchemaInterface schema,\n                                  EarlybirdFieldConstant baseField) {\n    this.schema = schema;\n    this.baseField = baseField;\n  }\n\n  /**\n   * Write this object into packedFeatures of the given VersionedTweetFeatures.\n   */\n  public void writeFeaturesToVersionedTweetFeatures(\n      VersionedTweetFeatures versionedTweetFeatures) {\n    if (!versionedTweetFeatures.isSetPackedFeatures()) {\n      versionedTweetFeatures.setPackedFeatures(new PackedFeatures());\n    }\n    copyToPackedFeatures(versionedTweetFeatures.getPackedFeatures());\n  }\n\n  /**\n   * Write this object into extendedPackedFeatures of the given VersionedTweetFeatures.\n   */\n  public void writeExtendedFeaturesToVersionedTweetFeatures(\n      VersionedTweetFeatures versionedTweetFeatures) {\n    if (!versionedTweetFeatures.isSetExtendedPackedFeatures()) {\n      versionedTweetFeatures.setExtendedPackedFeatures(new PackedFeatures());\n    }\n    copyToPackedFeatures(versionedTweetFeatures.getExtendedPackedFeatures());\n  }\n\n  @Override\n  public String toString() {\n    StringBuilder ret = new StringBuilder();\n    ret.append(\"Tweet features: \\n\");\n    for (FeatureConfiguration feature\n        : EarlybirdSchemaCreateTool.FEATURE_CONFIGURATION_MAP.values()) {\n      ret.append(feature.getName()).append(\": \").append(getFeatureValue(feature)).append(\"\\n\");\n    }\n    return ret.toString();\n  }\n\n  public boolean isFlagSet(EarlybirdFieldConstant field) {\n    return isFlagSet(schema.getFeatureConfigurationById(field.getFieldId()));\n  }\n\n  public int getFeatureValue(EarlybirdFieldConstant field) {\n    return getFeatureValue(schema.getFeatureConfigurationById(field.getFieldId()));\n  }\n\n  public EarlybirdEncodedFeatures setFlag(EarlybirdFieldConstant field) {\n    setFlag(schema.getFeatureConfigurationById(field.getFieldId()));\n    return this;\n  }\n\n  public EarlybirdEncodedFeatures clearFlag(EarlybirdFieldConstant field) {\n    clearFlag(schema.getFeatureConfigurationById(field.getFieldId()));\n    return this;\n  }\n\n  public EarlybirdEncodedFeatures setFlagValue(EarlybirdFieldConstant field,\n                                               boolean value) {\n    setFlagValue(schema.getFeatureConfigurationById(field.getFieldId()), value);\n    return this;\n  }\n\n  public EarlybirdEncodedFeatures setFeatureValue(EarlybirdFieldConstant field,\n                                                  int value) {\n    setFeatureValue(schema.getFeatureConfigurationById(field.getFieldId()), value);\n    return this;\n  }\n\n  public EarlybirdEncodedFeatures setFeatureValueIfGreater(EarlybirdFieldConstant field,\n                                                           int value) {\n    setFeatureValueIfGreater(schema.getFeatureConfigurationById(field.getFieldId()), value);\n    return this;\n  }\n\n  public boolean incrementIfNotMaximum(EarlybirdFieldConstant field) {\n    return incrementIfNotMaximum(schema.getFeatureConfigurationById(field.getFieldId()));\n  }\n\n  private static final class ArrayEncodedTweetFeatures extends EarlybirdEncodedFeatures {\n    private final int[] encodedInts;\n\n    private ArrayEncodedTweetFeatures(ImmutableSchemaInterface schema,\n                                      EarlybirdFieldConstant baseField) {\n      super(schema, baseField);\n\n      final int numIntegers = SchemaUtil.getCSFFieldFixedLength(schema, baseField.getFieldId());\n      Preconditions.checkState(numIntegers > 0);\n      this.encodedInts = new int[numIntegers];\n    }\n\n    @Override\n    public int getNumInts() {\n      return encodedInts.length;\n    }\n\n    @Override\n    public int getInt(int pos) {\n      return encodedInts[pos];\n    }\n\n    @Override\n    public void setInt(int pos, int value) {\n      encodedInts[pos] = value;\n    }\n  }\n\n  /**\n   * Create a new {@link EarlybirdEncodedFeatures} object based on schema and base field.\n   * @param schema the schema for all fields\n   * @param baseField base field's constant value\n   */\n  public static EarlybirdEncodedFeatures newEncodedTweetFeatures(\n      ImmutableSchemaInterface schema, EarlybirdFieldConstant baseField) {\n    return new ArrayEncodedTweetFeatures(schema, baseField);\n  }\n\n  /**\n   * Create a new {@link EarlybirdEncodedFeatures} object based on schema and base field name.\n   * @param schema the schema for all fields\n   * @param baseFieldName base field's name\n   */\n  public static EarlybirdEncodedFeatures newEncodedTweetFeatures(\n      ImmutableSchemaInterface schema, String baseFieldName) {\n    EarlybirdFieldConstant baseField = EarlybirdFieldConstants.getFieldConstant(baseFieldName);\n    Preconditions.checkNotNull(baseField);\n    return newEncodedTweetFeatures(schema, baseField);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/earlybird/EarlybirdEncodedFeaturesUtil.java",
    "content": "package com.twitter.search.common.schema.earlybird;\n\nimport com.twitter.search.common.encoding.docvalues.CSFTypeUtil;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\n\npublic final class EarlybirdEncodedFeaturesUtil {\n  private EarlybirdEncodedFeaturesUtil() {\n  }\n\n  /**\n   * Returns a byte array that can be stored in a ThriftDocument as bytesField.\n   */\n  public static byte[] toBytesForThriftDocument(EarlybirdEncodedFeatures features) {\n    int numInts = features.getNumInts();\n    byte[] serializedFeatures = new byte[numInts * Integer.BYTES];\n    for (int i = 0; i < numInts; i++) {\n      CSFTypeUtil.convertToBytes(serializedFeatures, i, features.getInt(i));\n    }\n    return serializedFeatures;\n  }\n\n  /**\n   * Converts data in a given byte array (starting at the provided offset) into\n   * EarlybirdEncodedFeatures.\n   */\n  public static EarlybirdEncodedFeatures fromBytes(\n      ImmutableSchemaInterface schema, EarlybirdFieldConstants.EarlybirdFieldConstant baseField,\n      byte[] data, int offset) {\n    EarlybirdEncodedFeatures features = EarlybirdEncodedFeatures.newEncodedTweetFeatures(\n        schema, baseField);\n    for (int idx = 0; idx < features.getNumInts(); ++idx) {\n      features.setInt(idx, CSFTypeUtil.convertFromBytes(data, offset, idx));\n    }\n    return features;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/earlybird/EarlybirdFieldConstants.java",
    "content": "\npackage com.twitter.search.common.schema.earlybird;\n\nimport java.util.Collection;\nimport java.util.EnumSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.ImmutableSet;\nimport com.google.common.collect.Sets;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftGeoLocationSource;\nimport com.twitter.search.common.schema.ImmutableSchema;\nimport com.twitter.search.common.schema.SchemaBuilder;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\nimport com.twitter.search.common.schema.base.FieldNameToIdMapping;\nimport com.twitter.search.common.schema.thriftjava.ThriftFeatureNormalizationType;\n\n/**\n * Field names, field IDs etc.\n */\npublic class EarlybirdFieldConstants extends FieldNameToIdMapping {\n  @VisibleForTesting\n  public static final String ENCODED_TWEET_FEATURES_FIELD_NAME = \"encoded_tweet_features\";\n\n  @VisibleForTesting\n  public static final String EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME =\n      \"extended_encoded_tweet_features\";\n\n  private enum FlagFeatureFieldType {\n    NON_FLAG_FEATURE_FIELD,\n    FLAG_FEATURE_FIELD\n  }\n\n  private enum UnusedFeatureFieldType {\n    USED_FEATURE_FIELD,\n    UNUSED_FEATURE_FIELD\n  }\n\n  /**\n   * CSF_NAME_TO_MIN_ENGAGEMENT_FIELD_MAP and MIN_ENGAGEMENT_FIELD_TO_CSF_NAME_MAP are used in\n   * EarlybirdLuceneQueryVisitor to map the CSFs REPLY_COUNT, RETWEET_COUNT, and FAVORITE_COUNT to\n   * their respective min engagement fields, and vice versa.\n   */\n  public static final ImmutableMap<String, EarlybirdFieldConstant>\n      CSF_NAME_TO_MIN_ENGAGEMENT_FIELD_MAP = ImmutableMap.<String, EarlybirdFieldConstant>builder()\n          .put(EarlybirdFieldConstant.REPLY_COUNT.getFieldName(),\n              EarlybirdFieldConstant.NORMALIZED_REPLY_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD)\n          .put(EarlybirdFieldConstant.RETWEET_COUNT.getFieldName(),\n              EarlybirdFieldConstant.NORMALIZED_RETWEET_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD)\n          .put(EarlybirdFieldConstant.FAVORITE_COUNT.getFieldName(),\n              EarlybirdFieldConstant.NORMALIZED_FAVORITE_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD)\n          .build();\n\n  public static final ImmutableMap<String, EarlybirdFieldConstant>\n      MIN_ENGAGEMENT_FIELD_TO_CSF_NAME_MAP = ImmutableMap.<String, EarlybirdFieldConstant>builder()\n      .put(EarlybirdFieldConstant.NORMALIZED_REPLY_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD\n              .getFieldName(),\n          EarlybirdFieldConstant.REPLY_COUNT)\n      .put(EarlybirdFieldConstant.NORMALIZED_RETWEET_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD\n              .getFieldName(),\n          EarlybirdFieldConstant.RETWEET_COUNT)\n      .put(EarlybirdFieldConstant.NORMALIZED_FAVORITE_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD\n              .getFieldName(),\n          EarlybirdFieldConstant.FAVORITE_COUNT)\n      .build();\n\n  /**\n   * A list of Earlybird field names and field IDs, and the clusters that need them.\n   */\n  public enum EarlybirdFieldConstant {\n    // These enums are grouped by category and sorted alphabetically.\n    // Next indexed field ID is 76\n    // Next CSF field ID is 115\n    // Next encoded_features CSF field ID is 185\n    // Next extended_encoded_features CSF field ID is 284\n\n    // Text searchable fields\n    // Provides slow ID Mapping from tweet ID to doc ID through TermsEnum.seekExact().\n    ID_FIELD(\"id\", 0, EarlybirdCluster.ALL_CLUSTERS),\n    RESOLVED_LINKS_TEXT_FIELD(\"resolved_links_text\", 1),\n    TEXT_FIELD(\"text\", 2),\n    TOKENIZED_FROM_USER_FIELD(\"tokenized_from_user\", 3),\n\n    // Other indexed fields\n    CARD_TITLE_FIELD(\"card_title\", 4),\n    CARD_DESCRIPTION_FIELD(\"card_description\", 5),\n    // We require the createdAt field to be set so we can properly filter tweets based on time.\n    CREATED_AT_FIELD(\"created_at\", 6, EarlybirdCluster.ALL_CLUSTERS),\n    // 7 was formerly EVENT_IDS_FIELD(\"event_ids\", 7, EarlybirdCluster.REALTIME)\n    ENTITY_ID_FIELD(\"entity_id\", 40),\n    // The screen name of the user that created the tweet. Should be set to the normalized value in\n    // the com.twitter.gizmoduck.thriftjava.Profile.screen_name field.\n    FROM_USER_FIELD(\"from_user\", 8),\n    // The numeric ID of the user that created the tweet.\n    FROM_USER_ID_FIELD(\"from_user_id\", 9, EarlybirdCluster.ALL_CLUSTERS),\n    CARD_DOMAIN_FIELD(\"card_domain\", 11),\n    CARD_NAME_FIELD(\"card_name\", 12),\n    GEO_HASH_FIELD(\"geo_hash\", 13),\n    HASHTAGS_FIELD(\"hashtags\", 14),\n    HF_PHRASE_PAIRS_FIELD(ImmutableSchema.HF_PHRASE_PAIRS_FIELD, 15),\n    HF_TERM_PAIRS_FIELD(ImmutableSchema.HF_TERM_PAIRS_FIELD, 16),\n    IMAGE_LINKS_FIELD(\"image_links\", 17),\n    IN_REPLY_TO_TWEET_ID_FIELD(\"in_reply_to_tweet_id\", 59),\n    IN_REPLY_TO_USER_ID_FIELD(\"in_reply_to_user_id\", 38),\n    // The internal field is used for many purposes:\n    // 1. to store facet skiplists\n    // 2. to power the filter operator, by storing posting list for terms like __filter_twimg\n    // 3. to store posting lists for positive and negative smileys\n    // 4. to store geo location types.\n    // etc.\n    INTERNAL_FIELD(\"internal\", 18, EarlybirdCluster.ALL_CLUSTERS),\n    ISO_LANGUAGE_FIELD(\"iso_lang\", 19),\n    LINK_CATEGORY_FIELD(\"link_category\", 36),\n    LINKS_FIELD(\"links\", 21),\n    MENTIONS_FIELD(\"mentions\", 22),\n    // Field 23 used to be NAMED_ENTITIES_FIELD\n    NEWS_LINKS_FIELD(\"news_links\", 24),\n    NORMALIZED_SOURCE_FIELD(\"norm_source\", 25),\n    PLACE_FIELD(\"place\", 26),\n    // Field 37 used to be PUBLICLY_INFERRED_USER_LOCATION_PLACE_ID_FIELD\n    // The ID of the source tweet. Set for retweets only.\n    RETWEET_SOURCE_TWEET_ID_FIELD(\"retweet_source_tweet_id\", 60,\n        EarlybirdCluster.ALL_CLUSTERS),\n    // The ID of the source tweet's author. Set for retweets only.\n    RETWEET_SOURCE_USER_ID_FIELD(\"retweet_source_user_id\", 39),\n    SOURCE_FIELD(\"source\", 29),\n    STOCKS_FIELD(\"stocks\", 30),\n    // The screen name of the user that a tweet was directed at.\n    TO_USER_FIELD(\"to_user\", 32),\n    // Field 33 used to be TOPIC_IDS_FIELD and is now unused. It can be reused later.\n    TWIMG_LINKS_FIELD(\"twimg_links\", 34),\n    VIDEO_LINKS_FIELD(\"video_links\", 35),\n    CAMELCASE_USER_HANDLE_FIELD(\"camelcase_tokenized_from_user\", 41),\n    // This field should be set to the the tokenized and normalized value in the\n    // com.twitter.gizmoduck.thriftjava.Profile.name field.\n    TOKENIZED_USER_NAME_FIELD(\"tokenized_from_user_display_name\", 42),\n    CONVERSATION_ID_FIELD(\"conversation_id\", 43),\n    PLACE_ID_FIELD(\"place_id\", 44),\n    PLACE_FULL_NAME_FIELD(\"place_full_name\", 45),\n    PLACE_COUNTRY_CODE_FIELD(\"place_country_code\", 46),\n    PROFILE_GEO_COUNTRY_CODE_FIELD(\"profile_geo_country_code\", 47),\n    PROFILE_GEO_REGION_FIELD(\"profile_geo_region\", 48),\n    PROFILE_GEO_LOCALITY_FIELD(\"profile_geo_locality\", 49),\n    LIKED_BY_USER_ID_FIELD(\"liked_by_user_id\", 50, EarlybirdCluster.REALTIME),\n    NORMALIZED_REPLY_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD(\n        \"normalized_reply_count_greater_than_or_equal_to\", 51, EarlybirdCluster.FULL_ARCHIVE),\n    NORMALIZED_RETWEET_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD(\n        \"normalized_retweet_count_greater_than_or_equal_to\", 52, EarlybirdCluster.FULL_ARCHIVE),\n    NORMALIZED_FAVORITE_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD(\n        \"normalized_favorite_count_greater_than_or_equal_to\", 53, EarlybirdCluster.FULL_ARCHIVE),\n    COMPOSER_SOURCE(\"composer_source\", 54),\n    QUOTED_TWEET_ID_FIELD(\"quoted_tweet_id\", 55),\n    QUOTED_USER_ID_FIELD(\"quoted_user_id\", 56),\n    RETWEETED_BY_USER_ID(\"retweeted_by_user_id\", 57, EarlybirdCluster.REALTIME),\n    REPLIED_TO_BY_USER_ID(\"replied_to_by_user_id\", 58, EarlybirdCluster.REALTIME),\n    CARD_LANG(\"card_lang\", 61),\n    // SEARCH-27823: Field ID 62 used to be named_entity, which was the combination of all\n    // named_entity* fields below. We need to leave 62 unused for backwards compatibility.\n    NAMED_ENTITY_FROM_URL_FIELD(\"named_entity_from_url\", 63),\n    NAMED_ENTITY_FROM_TEXT_FIELD(\"named_entity_from_text\", 64),\n    NAMED_ENTITY_WITH_TYPE_FROM_URL_FIELD(\"named_entity_with_type_from_url\", 65),\n    NAMED_ENTITY_WITH_TYPE_FROM_TEXT_FIELD(\"named_entity_with_type_from_text\", 66),\n    DIRECTED_AT_USER_ID_FIELD(\"directed_at_user_id\", 67),\n    SPACE_ID_FIELD(\"space_id\", 68,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_GENERAL_PURPOSE_CLUSTERS),\n    SPACE_TITLE_FIELD(\"space_title\", 69,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_GENERAL_PURPOSE_CLUSTERS),\n\n    // Detailed description of the space admin fields can be found at go/earlybirdfields.\n    SPACE_ADMIN_FIELD(\"space_admin\", 70,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_GENERAL_PURPOSE_CLUSTERS),\n    TOKENIZED_SPACE_ADMIN_FIELD(\"tokenized_space_admin\", 71,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_GENERAL_PURPOSE_CLUSTERS),\n    CAMELCASE_TOKENIZED_SPACE_ADMIN_FIELD(\"camelcase_tokenized_space_admin\", 72,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_GENERAL_PURPOSE_CLUSTERS),\n    TOKENIZED_SPACE_ADMIN_DISPLAY_NAME_FIELD(\"tokenized_space_admin_display_name\", 73,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_GENERAL_PURPOSE_CLUSTERS),\n    URL_DESCRIPTION_FIELD(\"url_description\", 74),\n    URL_TITLE_FIELD(\"url_title\", 75),\n\n    // CSF\n    CARD_TYPE_CSF_FIELD(\"card_type_csf\", 100),\n    ENCODED_TWEET_FEATURES_FIELD(ENCODED_TWEET_FEATURES_FIELD_NAME, 102,\n        EarlybirdCluster.ALL_CLUSTERS),\n    // Provides the doc ID -> original tweet ID mapping for retweets.\n    SHARED_STATUS_ID_CSF(\"shared_status_id_csf\", 106, EarlybirdCluster.ALL_CLUSTERS),\n    // Provides the doc ID -> tweet author's user ID mapping.\n    FROM_USER_ID_CSF(\"from_user_id_csf\", 103, EarlybirdCluster.ALL_CLUSTERS),\n    CREATED_AT_CSF_FIELD(\"created_at_csf\", 101, EarlybirdCluster.ARCHIVE_CLUSTERS),\n    // Provides the doc ID -> tweet ID mapping.\n    ID_CSF_FIELD(\"id_csf\", 104, EarlybirdCluster.ARCHIVE_CLUSTERS),\n    LAT_LON_CSF_FIELD(\"latlon_csf\", 105),\n    CONVERSATION_ID_CSF(\"conversation_id_csf\", 107, EarlybirdCluster.ALL_CLUSTERS),\n    QUOTED_TWEET_ID_CSF(\"quoted_tweet_id_csf\", 108),\n    QUOTED_USER_ID_CSF(\"quoted_user_id_csf\", 109),\n    CARD_LANG_CSF(\"card_lang_csf\", 110),\n    DIRECTED_AT_USER_ID_CSF(\"directed_at_user_id_csf\", 111),\n    REFERENCE_AUTHOR_ID_CSF(\"reference_author_id_csf\", 112),\n    EXCLUSIVE_CONVERSATION_AUTHOR_ID_CSF(\"exclusive_conversation_author_id_csf\", 113),\n    CARD_URI_CSF(\"card_uri_csf\", 114),\n\n    // CSF Views on top of ENCODED_TWEET_FEATURES_FIELD\n    IS_RETWEET_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"IS_RETWEET_FLAG\", 150,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    IS_OFFENSIVE_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"IS_OFFENSIVE_FLAG\", 151,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_LINK_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_LINK_FLAG\", 152,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_TREND_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_TREND_FLAG\", 153,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    IS_REPLY_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"IS_REPLY_FLAG\", 154,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    IS_SENSITIVE_CONTENT(ENCODED_TWEET_FEATURES_FIELD_NAME, \"IS_SENSITIVE_CONTENT\", 155,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_MULTIPLE_HASHTAGS_OR_TRENDS_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"HAS_MULTIPLE_HASHTAGS_OR_TRENDS_FLAG\", 156, FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.ALL_CLUSTERS),\n    FROM_VERIFIED_ACCOUNT_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"FROM_VERIFIED_ACCOUNT_FLAG\",\n        157,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    TEXT_SCORE(ENCODED_TWEET_FEATURES_FIELD_NAME, \"TEXT_SCORE\", 158,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    LANGUAGE(ENCODED_TWEET_FEATURES_FIELD_NAME, \"LANGUAGE\", 159,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    LINK_LANGUAGE(ENCODED_TWEET_FEATURES_FIELD_NAME, \"LINK_LANGUAGE\", 160,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_IMAGE_URL_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_IMAGE_URL_FLAG\", 161,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_VIDEO_URL_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_VIDEO_URL_FLAG\", 162,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_NEWS_URL_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_NEWS_URL_FLAG\", 163,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_EXPANDO_CARD_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_EXPANDO_CARD_FLAG\", 164,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_MULTIPLE_MEDIA_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_MULTIPLE_MEDIA_FLAG\", 165,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    PROFILE_IS_EGG_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"PROFILE_IS_EGG_FLAG\", 166,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    NUM_MENTIONS(ENCODED_TWEET_FEATURES_FIELD_NAME, \"NUM_MENTIONS\", 167,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    NUM_HASHTAGS(ENCODED_TWEET_FEATURES_FIELD_NAME, \"NUM_HASHTAGS\", 168,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_CARD_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_CARD_FLAG\", 169,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_VISIBLE_LINK_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_VISIBLE_LINK_FLAG\", 170,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    USER_REPUTATION(ENCODED_TWEET_FEATURES_FIELD_NAME, \"USER_REPUTATION\", 171,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    IS_USER_SPAM_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"IS_USER_SPAM_FLAG\", 172,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    IS_USER_NSFW_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"IS_USER_NSFW_FLAG\", 173,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    IS_USER_BOT_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"IS_USER_BOT_FLAG\", 174,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    IS_USER_NEW_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"IS_USER_NEW_FLAG\", 175,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    PREV_USER_TWEET_ENGAGEMENT(ENCODED_TWEET_FEATURES_FIELD_NAME, \"PREV_USER_TWEET_ENGAGEMENT\",\n        176,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    COMPOSER_SOURCE_IS_CAMERA_FLAG(\n        ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"COMPOSER_SOURCE_IS_CAMERA_FLAG\",\n        177,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.ALL_CLUSTERS),\n    RETWEET_COUNT(\n        ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"RETWEET_COUNT\",\n        178,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.LEGACY_BYTE_NORMALIZER_WITH_LOG2),\n    FAVORITE_COUNT(\n        ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"FAVORITE_COUNT\",\n        179,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.LEGACY_BYTE_NORMALIZER_WITH_LOG2),\n    REPLY_COUNT(\n        ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"REPLY_COUNT\",\n        180,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.LEGACY_BYTE_NORMALIZER_WITH_LOG2),\n    PARUS_SCORE(ENCODED_TWEET_FEATURES_FIELD_NAME, \"PARUS_SCORE\", 181,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n\n    /**\n     * This is the rough percentage of the nth token at 140 divided by num tokens\n     * and is basically n / num tokens where n is the token starting before 140 characters\n     */\n    VISIBLE_TOKEN_RATIO(ENCODED_TWEET_FEATURES_FIELD_NAME, \"VISIBLE_TOKEN_RATIO\", 182,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_QUOTE_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_QUOTE_FLAG\", 183,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n\n    FROM_BLUE_VERIFIED_ACCOUNT_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"FROM_BLUE_VERIFIED_ACCOUNT_FLAG\",\n        184,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n\n    TWEET_SIGNATURE(ENCODED_TWEET_FEATURES_FIELD_NAME, \"TWEET_SIGNATURE\", 188,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n\n    // MEDIA TYPES\n    HAS_CONSUMER_VIDEO_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_CONSUMER_VIDEO_FLAG\", 189,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_PRO_VIDEO_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_PRO_VIDEO_FLAG\", 190,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_VINE_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_VINE_FLAG\", 191,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_PERISCOPE_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_PERISCOPE_FLAG\", 192,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n    HAS_NATIVE_IMAGE_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"HAS_NATIVE_IMAGE_FLAG\", 193,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n\n    // NOTE: if possible, please reserve field ID 194 to 196 for future media types (SEARCH-9131)\n\n    IS_NULLCAST_FLAG(ENCODED_TWEET_FEATURES_FIELD_NAME, \"IS_NULLCAST_FLAG\", 197,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD, EarlybirdCluster.ALL_CLUSTERS),\n\n    // EXTENDED ENCODED TWEET FEATURES that's not available on archive clusters\n    EXTENDED_ENCODED_TWEET_FEATURES_FIELD(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME, 200,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    EMBEDS_IMPRESSION_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"EMBEDS_IMPRESSION_COUNT\",\n        221,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.LEGACY_BYTE_NORMALIZER),\n    EMBEDS_URL_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"EMBEDS_URL_COUNT\",\n        222,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.LEGACY_BYTE_NORMALIZER),\n    VIDEO_VIEW_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"VIDEO_VIEW_COUNT\",\n        223,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.LEGACY_BYTE_NORMALIZER),\n\n    // empty bits in integer 0 (starting bit 24, 8 bits)\n    EXTENDED_FEATURE_UNUSED_BITS_0_24_8(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"UNUSED_BITS_0_24_8\", 244,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        UnusedFeatureFieldType.UNUSED_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    // SEARCH-8564 - Reference Tweet Author ID\n    REFERENCE_AUTHOR_ID_LEAST_SIGNIFICANT_INT(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"REFERENCE_AUTHOR_ID_LEAST_SIGNIFICANT_INT\", 202,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n    REFERENCE_AUTHOR_ID_MOST_SIGNIFICANT_INT(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"REFERENCE_AUTHOR_ID_MOST_SIGNIFICANT_INT\", 203,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    // SEARCHQUAL-8130: engagement counters v2\n    // Integer 3\n    RETWEET_COUNT_V2(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"RETWEET_COUNT_V2\", 225,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    FAVORITE_COUNT_V2(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"FAVORITE_COUNT_V2\", 226,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    REPLY_COUNT_V2(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"REPLY_COUNT_V2\", 227,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    EMBEDS_IMPRESSION_COUNT_V2(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"EMBEDS_IMPRESSION_COUNT_V2\",\n        228,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n\n    // Integer 4\n    EMBEDS_URL_COUNT_V2(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"EMBEDS_URL_COUNT_V2\",\n        229,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    VIDEO_VIEW_COUNT_V2(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"VIDEO_VIEW_COUNT_V2\",\n        230,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    QUOTE_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"QUOTE_COUNT\",\n        231,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n\n    // Tweet Safety Labels\n    LABEL_ABUSIVE_FLAG(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"LABEL_ABUSIVE_FLAG\", 232,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    LABEL_ABUSIVE_HI_RCL_FLAG(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"LABEL_ABUSIVE_HI_RCL_FLAG\", 233,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    LABEL_DUP_CONTENT_FLAG(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"LABEL_DUP_CONTENT_FLAG\", 234,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    LABEL_NSFW_HI_PRC_FLAG(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"LABEL_NSFW_HI_PRC_FLAG\", 235,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    LABEL_NSFW_HI_RCL_FLAG(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"LABEL_NSFW_HI_RCL_FLAG\", 236,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    LABEL_SPAM_FLAG(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"LABEL_SPAM_FLAG\", 237,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    LABEL_SPAM_HI_RCL_FLAG(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"LABEL_SPAM_HI_RCL_FLAG\", 238,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    // please save this bit for other safety labels\n    EXTENDED_TEST_FEATURE_UNUSED_BITS_4_31_1(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"UNUSED_BITS_4_31_1\", 239,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        UnusedFeatureFieldType.UNUSED_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    // Integer 5\n    WEIGHTED_RETWEET_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"WEIGHTED_RETWEET_COUNT\",\n        240,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    WEIGHTED_REPLY_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"WEIGHTED_REPLY_COUNT\",\n        241,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    WEIGHTED_FAVORITE_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"WEIGHTED_FAVORITE_COUNT\",\n        242,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    WEIGHTED_QUOTE_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"WEIGHTED_QUOTE_COUNT\",\n        243,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n\n    // Integer 6\n    // Periscope features\n    PERISCOPE_EXISTS(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"PERISCOPE_EXISTS\", 245,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n    PERISCOPE_HAS_BEEN_FEATURED(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"PERISCOPE_HAS_BEEN_FEATURED\", 246,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n    PERISCOPE_IS_CURRENTLY_FEATURED(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"PERISCOPE_IS_CURRENTLY_FEATURED\", 247,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n    PERISCOPE_IS_FROM_QUALITY_SOURCE(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"PERISCOPE_IS_FROM_QUALITY_SOURCE\", 248,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n    PERISCOPE_IS_LIVE(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"PERISCOPE_IS_LIVE\", 249,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n    IS_TRENDING_NOW_FLAG(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"IS_TRENDING_NOW_FLAG\", 292,\n        FlagFeatureFieldType.FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    // remaining bits for integer 6 (starting bit 6, 26 remaining bits)\n    EXTENDED_TEST_FEATURE_UNUSED_BITS_7_6_26(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"UNUSED_BITS_7_6_26\", 250,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        UnusedFeatureFieldType.UNUSED_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    // Decaying engagement counters\n    // Integer 7\n    DECAYED_RETWEET_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"DECAYED_RETWEET_COUNT\",\n        251,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    DECAYED_REPLY_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"DECAYED_REPLY_COUNT\",\n        252,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    DECAYED_FAVORITE_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"DECAYED_FAVORITE_COUNT\",\n        253,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    DECAYED_QUOTE_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"DECAYED_QUOTE_COUNT\",\n        254,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n\n    // Fake engagement counters. The fake here is in the sense of spam, not in the sense of testing.\n    // Refer to [JIRA SEARCHQUAL-10736 Remove Fake Engagements in Search] for more details.\n    // Integer 8\n    FAKE_RETWEET_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"FAKE_RETWEET_COUNT\", 269,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    FAKE_REPLY_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"FAKE_REPLY_COUNT\", 270,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    FAKE_FAVORITE_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"FAKE_FAVORITE_COUNT\", 271,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    FAKE_QUOTE_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"FAKE_QUOTE_COUNT\", 272,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n\n    // Last engagement timestamps. These features use the Tweet's creation time as base and\n    // are incremented every 1 hour\n    // Integer 9\n    LAST_RETWEET_SINCE_CREATION_HRS(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"LAST_RETWEET_SINCE_CREATION_HRS\",\n        273,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.NONE),\n    LAST_REPLY_SINCE_CREATION_HRS(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"LAST_REPLY_SINCE_CREATION_HRS\",\n        274,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.NONE),\n    LAST_FAVORITE_SINCE_CREATION_HRS(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"LAST_FAVORITE_SINCE_CREATION_HRS\",\n        275,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.NONE),\n    LAST_QUOTE_SINCE_CREATION_HRS(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"LAST_QUOTE_SINCE_CREATION_HRS\",\n        276,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.NONE),\n\n    // 4 bits hashtag count, mention count and stock count (SEARCH-24336)\n    // Integer 10\n    NUM_HASHTAGS_V2(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"NUM_HASHTAGS_V2\",\n        277,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.NONE\n    ),\n    NUM_MENTIONS_V2(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"NUM_MENTIONS_V2\",\n        278,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.NONE\n    ),\n    NUM_STOCKS(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"NUM_STOCKS\",\n        279,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.NONE\n    ),\n\n    // Integer 11\n    // Blink engagement counters\n    BLINK_RETWEET_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"BLINK_RETWEET_COUNT\",\n        280,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    BLINK_REPLY_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"BLINK_REPLY_COUNT\",\n        281,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    BLINK_FAVORITE_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"BLINK_FAVORITE_COUNT\",\n        282,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n    BLINK_QUOTE_COUNT(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"BLINK_QUOTE_COUNT\",\n        283,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.SMART_INTEGER_NORMALIZER),\n\n    // Integer 10 (remaining)\n    // Production Toxicity and PBlock score from HML (go/toxicity, go/pblock)\n    TOXICITY_SCORE(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"TOXICITY_SCORE\", 284,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.PREDICTION_SCORE_NORMALIZER\n    ),\n    PBLOCK_SCORE(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"PBLOCK_SCORE\", 285,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.PREDICTION_SCORE_NORMALIZER\n    ),\n\n    // Integer 12\n    // Experimental health model scores from HML\n    EXPERIMENTAL_HEALTH_MODEL_SCORE_1(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"EXPERIMENTAL_HEALTH_MODEL_SCORE_1\", 286,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.PREDICTION_SCORE_NORMALIZER\n    ),\n    EXPERIMENTAL_HEALTH_MODEL_SCORE_2(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"EXPERIMENTAL_HEALTH_MODEL_SCORE_2\", 287,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.PREDICTION_SCORE_NORMALIZER\n    ),\n    EXPERIMENTAL_HEALTH_MODEL_SCORE_3(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"EXPERIMENTAL_HEALTH_MODEL_SCORE_3\", 288,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.PREDICTION_SCORE_NORMALIZER\n    ),\n    // remaining bits for index 12 (unused_bits_12)\n    EXTENDED_TEST_FEATURE_UNUSED_BITS_12_30_2(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"UNUSED_BITS_12_30_2\", 289,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        UnusedFeatureFieldType.UNUSED_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    // Integer 13\n    // Experimental health model scores from HML (cont.)\n    EXPERIMENTAL_HEALTH_MODEL_SCORE_4(\n        EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"EXPERIMENTAL_HEALTH_MODEL_SCORE_4\", 290,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.PREDICTION_SCORE_NORMALIZER\n    ),\n    // Production pSpammyTweet score from HML (go/pspammytweet)\n    P_SPAMMY_TWEET_SCORE(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"P_SPAMMY_TWEET_SCORE\", 291,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.PREDICTION_SCORE_NORMALIZER\n    ),\n    // Production pReportedTweet score from HML (go/preportedtweet)\n    P_REPORTED_TWEET_SCORE(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"P_REPORTED_TWEET_SCORE\", 293,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.PREDICTION_SCORE_NORMALIZER\n    ),\n    // remaining bits for index 13 (unused_bits_13)\n    EXTENDED_TEST_FEATURE_UNUSED_BITS_13_30_2(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"UNUSED_BITS_13_30_2\", 294,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        UnusedFeatureFieldType.UNUSED_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS\n    ),\n\n    // Integer 14\n    // Health model scores from HML (cont.)\n    // Prod Spammy Tweet Content model score from Platform Manipulation (go/spammy-tweet-content)\n    SPAMMY_TWEET_CONTENT_SCORE(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"SPAMMY_TWEET_CONTENT_SCORE\", 295,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS,\n        ThriftFeatureNormalizationType.PREDICTION_SCORE_NORMALIZER\n    ),\n    // remaining bits for index 14 (unused_bits_14)\n    EXTENDED_TEST_FEATURE_UNUSED_BITS_14_10_22(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"UNUSED_BITS_14_10_22\", 296,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        UnusedFeatureFieldType.UNUSED_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS\n    ),\n\n    // Note that the integer block index i in the names UNUSED_BITS{i}\" below is 1-based, but the\n    // index j in UNUSED_BITS_{j}_x_y above is 0-based.\n    EXTENDED_TEST_FEATURE_UNUSED_BITS_16(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"UNUSED_BITS16\", 216,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        UnusedFeatureFieldType.UNUSED_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    EXTENDED_TEST_FEATURE_UNUSED_BITS_17(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"UNUSED_BITS17\", 217,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        UnusedFeatureFieldType.UNUSED_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    EXTENDED_TEST_FEATURE_UNUSED_BITS_18(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"UNUSED_BITS18\", 218,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        UnusedFeatureFieldType.UNUSED_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    EXTENDED_TEST_FEATURE_UNUSED_BITS_19(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"UNUSED_BITS19\", 219,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        UnusedFeatureFieldType.UNUSED_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS),\n\n    EXTENDED_TEST_FEATURE_UNUSED_BITS_20(EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        \"UNUSED_BITS20\", 220,\n        FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n        UnusedFeatureFieldType.UNUSED_FEATURE_FIELD,\n        EarlybirdCluster.TWITTER_IN_MEMORY_INDEX_FORMAT_ALL_CLUSTERS);\n\n    // Filter field terms. These end up as terms in the \"internal\" field (id=18). So for example\n    // you can have a doc with field(internal) = \"__filter_nullcast\", \"__filter_vine\" and that will\n    // be a nullcast tweet with a vine link in it.\n    public static final String NULLCAST_FILTER_TERM = \"nullcast\";\n    public static final String VERIFIED_FILTER_TERM = \"verified\";\n    public static final String BLUE_VERIFIED_FILTER_TERM = \"blue_verified\";\n    public static final String NATIVE_RETWEETS_FILTER_TERM = \"nativeretweets\";\n    public static final String QUOTE_FILTER_TERM = \"quote\";\n    public static final String REPLIES_FILTER_TERM = \"replies\";\n    public static final String CONSUMER_VIDEO_FILTER_TERM = \"consumer_video\";\n    public static final String PRO_VIDEO_FILTER_TERM = \"pro_video\";\n    public static final String VINE_FILTER_TERM = \"vine\";\n    public static final String PERISCOPE_FILTER_TERM = \"periscope\";\n    public static final String PROFILE_GEO_FILTER_TERM = \"profile_geo\";\n    public static final String SELF_THREAD_FILTER_TERM = \"self_threads\";\n    public static final String DIRECTED_AT_FILTER_TERM = \"directed_at\";\n    public static final String EXCLUSIVE_FILTER_TERM = \"exclusive\";\n\n    // Reserved terms for the internal field.\n    public static final String HAS_POSITIVE_SMILEY = \"__has_positive_smiley\";\n    public static final String HAS_NEGATIVE_SMILEY = \"__has_negative_smiley\";\n    public static final String IS_OFFENSIVE = \"__is_offensive\";\n\n    // Facet fields\n    public static final String MENTIONS_FACET = \"mentions\";\n    public static final String HASHTAGS_FACET = \"hashtags\";\n    public static final String STOCKS_FACET = \"stocks\";\n    public static final String VIDEOS_FACET = \"videos\";\n    public static final String IMAGES_FACET = \"images\";\n    public static final String NEWS_FACET = \"news\";\n    public static final String LANGUAGES_FACET = \"languages\";\n    public static final String SOURCES_FACET = \"sources\";\n    public static final String TWIMG_FACET = \"twimg\";\n    public static final String FROM_USER_ID_FACET = \"user_id\";\n    public static final String RETWEETS_FACET = \"retweets\";\n    public static final String LINKS_FACET = \"links\";\n    public static final String SPACES_FACET = \"spaces\";\n\n    /**\n     * Used by the query parser to check that the operator of a [filter X] query is valid.\n     * Also used by blender, though it probably shouldn't be.\n     */\n    public static final ImmutableSet<String> FACETS = ImmutableSet.<String>builder()\n        .add(MENTIONS_FACET)\n        .add(HASHTAGS_FACET)\n        .add(STOCKS_FACET)\n        .add(VIDEOS_FACET)\n        .add(IMAGES_FACET)\n        .add(NEWS_FACET)\n        .add(LINKS_FACET)\n        .add(LANGUAGES_FACET)\n        .add(SOURCES_FACET)\n        .add(TWIMG_FACET)\n        .add(SPACES_FACET)\n        .build();\n\n    /**\n     * Used by blender to convert facet names to field names. We should find a way to get the\n     * information we need in blender without needing this map.\n     */\n    public static final ImmutableMap<String, String> FACET_TO_FIELD_MAP =\n        ImmutableMap.<String, String>builder()\n            .put(MENTIONS_FACET, MENTIONS_FIELD.getFieldName())\n            .put(HASHTAGS_FACET, HASHTAGS_FIELD.getFieldName())\n            .put(STOCKS_FACET, STOCKS_FIELD.getFieldName())\n            .put(VIDEOS_FACET, VIDEO_LINKS_FIELD.getFieldName())\n            .put(IMAGES_FACET, IMAGE_LINKS_FIELD.getFieldName())\n            .put(NEWS_FACET, NEWS_LINKS_FIELD.getFieldName())\n            .put(LANGUAGES_FACET, ISO_LANGUAGE_FIELD.getFieldName())\n            .put(SOURCES_FACET, SOURCE_FIELD.getFieldName())\n            .put(TWIMG_FACET, TWIMG_LINKS_FIELD.getFieldName())\n            .put(LINKS_FACET, LINKS_FIELD.getFieldName())\n            .put(SPACES_FACET, SPACE_ID_FIELD.getFieldName())\n            .build();\n\n    public static String getFacetSkipFieldName(String fieldName) {\n      return \"__has_\" + fieldName;\n    }\n\n    private final String fieldName;\n    private final int fieldId;\n    private final EnumSet<EarlybirdCluster> clusters;\n    private final FlagFeatureFieldType flagFeatureField;\n\n    private final UnusedFeatureFieldType unusedField;\n\n    // Only set for feature fields.\n    @Nullable\n    private final FeatureConfiguration featureConfiguration;\n\n    // Only set for feature fields.\n    private final ThriftFeatureNormalizationType featureNormalizationType;\n\n    // To simplify field configurations and reduce duplicate code, we give clusters a default value\n    EarlybirdFieldConstant(String fieldName, int fieldId) {\n      this(fieldName, fieldId, EarlybirdCluster.GENERAL_PURPOSE_CLUSTERS, null);\n    }\n\n    EarlybirdFieldConstant(String fieldName, int fieldId, Set<EarlybirdCluster> clusters) {\n      this(fieldName, fieldId, clusters, null);\n    }\n\n    EarlybirdFieldConstant(String fieldName, int fieldId, EarlybirdCluster cluster) {\n      this(fieldName, fieldId, ImmutableSet.<EarlybirdCluster>of(cluster), null);\n    }\n\n    /**\n     * Base field name is needed here in order to construct the full\n     * name of the feature. Our convention is that a feature should be named\n     * as: baseFieldName.featureName.  For example: encoded_tweet_features.retweet_count.\n     */\n    EarlybirdFieldConstant(\n        String baseName,\n        String fieldName,\n        int fieldId,\n        FlagFeatureFieldType flagFeatureField,\n        Set<EarlybirdCluster> clusters) {\n      this((baseName + SchemaBuilder.CSF_VIEW_NAME_SEPARATOR + fieldName).toLowerCase(),\n          fieldId, clusters, flagFeatureField, null);\n    }\n\n    EarlybirdFieldConstant(\n        String baseName,\n        String fieldName,\n        int fieldId,\n        FlagFeatureFieldType flagFeatureField,\n        UnusedFeatureFieldType unusedField,\n        Set<EarlybirdCluster> clusters) {\n      this((baseName + SchemaBuilder.CSF_VIEW_NAME_SEPARATOR + fieldName).toLowerCase(),\n          fieldId, clusters, flagFeatureField, unusedField, null);\n    }\n\n    EarlybirdFieldConstant(\n        String baseName,\n        String fieldName,\n        int fieldId,\n        FlagFeatureFieldType flagFeatureField,\n        Set<EarlybirdCluster> clusters,\n        ThriftFeatureNormalizationType featureNormalizationType) {\n      this((baseName + SchemaBuilder.CSF_VIEW_NAME_SEPARATOR + fieldName).toLowerCase(),\n          fieldId, clusters, flagFeatureField, UnusedFeatureFieldType.USED_FEATURE_FIELD,\n          featureNormalizationType, null);\n    }\n\n    /**\n     * Constructor.\n     */\n    EarlybirdFieldConstant(String fieldName, int fieldId, Set<EarlybirdCluster> clusters,\n                                   @Nullable FeatureConfiguration featureConfiguration) {\n      this(fieldName, fieldId, clusters, FlagFeatureFieldType.NON_FLAG_FEATURE_FIELD,\n          featureConfiguration);\n    }\n\n    /**\n     * Constructor.\n     */\n    EarlybirdFieldConstant(String fieldName,\n                           int fieldId,\n                           Set<EarlybirdCluster> clusters,\n                           FlagFeatureFieldType flagFeatureField,\n                           @Nullable FeatureConfiguration featureConfiguration) {\n      this(fieldName, fieldId, clusters, flagFeatureField,\n          UnusedFeatureFieldType.USED_FEATURE_FIELD, featureConfiguration);\n    }\n\n    /**\n     * Constructor.\n     */\n    EarlybirdFieldConstant(String fieldName,\n                           int fieldId,\n                           Set<EarlybirdCluster> clusters,\n                           FlagFeatureFieldType flagFeatureField,\n                           UnusedFeatureFieldType unusedField,\n                           @Nullable FeatureConfiguration featureConfiguration) {\n      this(fieldName, fieldId, clusters, flagFeatureField, unusedField, null, featureConfiguration);\n    }\n\n    /**\n     * Constructor.\n     */\n    EarlybirdFieldConstant(String fieldName,\n                           int fieldId,\n                           Set<EarlybirdCluster> clusters,\n                           FlagFeatureFieldType flagFeatureField,\n                           UnusedFeatureFieldType unusedField,\n                           @Nullable ThriftFeatureNormalizationType featureNormalizationType,\n                           @Nullable FeatureConfiguration featureConfiguration) {\n      this.fieldId = fieldId;\n      this.fieldName = fieldName;\n      this.clusters = EnumSet.copyOf(clusters);\n      this.flagFeatureField = flagFeatureField;\n      this.unusedField = unusedField;\n      this.featureNormalizationType = featureNormalizationType;\n      this.featureConfiguration = featureConfiguration;\n    }\n\n    // Override toString to make replacing StatusConstant Easier.\n    @Override\n    public String toString() {\n      return fieldName;\n    }\n\n    public boolean isValidFieldInCluster(EarlybirdCluster cluster) {\n      return clusters.contains(cluster);\n    }\n\n    public String getFieldName() {\n      return fieldName;\n    }\n\n    public int getFieldId() {\n      return fieldId;\n    }\n\n    public FlagFeatureFieldType getFlagFeatureField() {\n      return flagFeatureField;\n    }\n\n    public boolean isFlagFeatureField() {\n      return flagFeatureField == FlagFeatureFieldType.FLAG_FEATURE_FIELD;\n    }\n\n    public boolean isUnusedField() {\n      return unusedField == UnusedFeatureFieldType.UNUSED_FEATURE_FIELD;\n    }\n\n    @Nullable\n    public FeatureConfiguration getFeatureConfiguration() {\n      return featureConfiguration;\n    }\n\n    @Nullable\n    public ThriftFeatureNormalizationType getFeatureNormalizationType() {\n      return featureNormalizationType;\n    }\n  }\n\n  private static final Map<String, EarlybirdFieldConstant> NAME_TO_ID_MAP;\n  private static final Map<Integer, EarlybirdFieldConstant> ID_TO_FIELD_MAP;\n  static {\n    ImmutableMap.Builder<String, EarlybirdFieldConstant> nameToIdMapBuilder =\n        ImmutableMap.builder();\n    ImmutableMap.Builder<Integer, EarlybirdFieldConstant> idToFieldMapBuilder =\n        ImmutableMap.builder();\n    Set<String> fieldNameDupDetector = Sets.newHashSet();\n    Set<Integer> fieldIdDupDetector = Sets.newHashSet();\n    for (EarlybirdFieldConstant fc : EarlybirdFieldConstant.values()) {\n      if (fieldNameDupDetector.contains(fc.getFieldName())) {\n        throw new IllegalStateException(\"detected fields sharing field name: \" + fc.getFieldName());\n      }\n      if (fieldIdDupDetector.contains(fc.getFieldId())) {\n        throw new IllegalStateException(\"detected fields sharing field id: \" + fc.getFieldId());\n      }\n\n      fieldNameDupDetector.add(fc.getFieldName());\n      fieldIdDupDetector.add(fc.getFieldId());\n      nameToIdMapBuilder.put(fc.getFieldName(), fc);\n      idToFieldMapBuilder.put(fc.getFieldId(), fc);\n    }\n    NAME_TO_ID_MAP = nameToIdMapBuilder.build();\n    ID_TO_FIELD_MAP = idToFieldMapBuilder.build();\n  }\n\n  // This define the list of boolean features, but the name does not have \"flag\" inside.  This\n  // definition is only for double checking purpose to prevent code change mistakes.  The setting\n  // of the flag feature is based on FlagFeatureFieldType.FLAG_FEATURE_FIELD.\n  public static final Set<EarlybirdFieldConstants.EarlybirdFieldConstant> EXTRA_FLAG_FIELDS =\n      Sets.newHashSet(EarlybirdFieldConstants.EarlybirdFieldConstant.IS_SENSITIVE_CONTENT);\n  public static final String FLAG_STRING = \"flag\";\n\n  private static final List<EarlybirdFieldConstant> FLAG_FEATURE_FIELDS;\n  static {\n    ImmutableList.Builder<EarlybirdFieldConstant> flagFieldBuilder = ImmutableList.builder();\n    for (EarlybirdFieldConstant fc : EarlybirdFieldConstant.values()) {\n      if (fc.getFlagFeatureField() == FlagFeatureFieldType.FLAG_FEATURE_FIELD\n          && !fc.isUnusedField()) {\n        flagFieldBuilder.add(fc);\n      }\n    }\n    FLAG_FEATURE_FIELDS = flagFieldBuilder.build();\n  }\n\n  /**\n   * Get all the flag features meaning that they are boolean features with only 1 bit in the packed\n   * feature encoding.\n   */\n  public static Collection<EarlybirdFieldConstant> getFlagFeatureFields() {\n    return FLAG_FEATURE_FIELDS;\n  }\n\n  /**\n   * Get the EarlybirdFieldConstant for the specified field.\n   */\n  public static EarlybirdFieldConstant getFieldConstant(String fieldName) {\n    EarlybirdFieldConstant field = NAME_TO_ID_MAP.get(fieldName);\n    if (field == null) {\n      throw new IllegalArgumentException(\"Unknown field: \" + fieldName);\n    }\n    return field;\n  }\n\n  /**\n   * Get the EarlybirdFieldConstant for the specified field.\n   */\n  public static EarlybirdFieldConstant getFieldConstant(int fieldId) {\n    EarlybirdFieldConstant field = ID_TO_FIELD_MAP.get(fieldId);\n    if (field == null) {\n      throw new IllegalArgumentException(\"Unknown field: \" + fieldId);\n    }\n    return field;\n  }\n\n  /**\n   * Determines if there's a field with the given ID.\n   */\n  public static boolean hasFieldConstant(int fieldId) {\n    return ID_TO_FIELD_MAP.keySet().contains(fieldId);\n  }\n\n  @Override\n  public final int getFieldID(String fieldName) {\n    return getFieldConstant(fieldName).getFieldId();\n  }\n\n  public static final String formatGeoType(ThriftGeoLocationSource source) {\n    return \"__geo_location_type_\" + source.name().toLowerCase();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/earlybird/EarlybirdSchemaBuilder.java",
    "content": "package com.twitter.search.common.schema.earlybird;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableList;\n\nimport com.twitter.common.text.util.TokenStreamSerializer;\nimport com.twitter.search.common.schema.SchemaBuilder;\nimport com.twitter.search.common.schema.base.FieldNameToIdMapping;\nimport com.twitter.search.common.schema.thriftjava.ThriftFieldConfiguration;\nimport com.twitter.search.common.schema.thriftjava.ThriftFieldSettings;\nimport com.twitter.search.common.schema.thriftjava.ThriftTokenStreamSerializer;\nimport com.twitter.search.common.util.analysis.CharTermAttributeSerializer;\nimport com.twitter.search.common.util.analysis.TermPayloadAttributeSerializer;\n\n/**\n * Build class used to build a ThriftSchema\n */\npublic class EarlybirdSchemaBuilder extends SchemaBuilder {\n  private final EarlybirdCluster cluster;\n\n  public EarlybirdSchemaBuilder(FieldNameToIdMapping idMapping,\n                                EarlybirdCluster cluster,\n                                TokenStreamSerializer.Version tokenStreamSerializerVersion) {\n    super(idMapping, tokenStreamSerializerVersion);\n    this.cluster = cluster;\n  }\n\n  /**\n   * Configure the specified field to be Out-of-order.\n   * In the realtime cluster, this causes Earlybird to used the skip list posting format.\n   */\n  public final EarlybirdSchemaBuilder withOutOfOrderEnabledForField(String fieldName) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFieldSettings settings =\n        schema.getFieldConfigs().get(idMapping.getFieldID(fieldName)).getSettings();\n    Preconditions.checkState(settings.isSetIndexedFieldSettings(),\n                             \"Out of order field must be indexed\");\n    settings.getIndexedFieldSettings().setSupportOutOfOrderAppends(true);\n    return this;\n  }\n\n  /**\n   * This turns on tweet specific normalizations. This turns on the following two token processors:\n   * {@link com.twitter.search.common.util.text.splitter.HashtagMentionPunctuationSplitter}\n   * {@link com.twitter.search.common.util.text.filter.NormalizedTokenFilter}\n   * <p/>\n   * HashtagMentionPunctuationSplitter would break a mention or hashtag like @ab_cd or #ab_cd into\n   * tokens {ab, cd}.\n   * NormalizedTokenFilter strips out the # @ $ from the tokens.\n   */\n  public final EarlybirdSchemaBuilder withTweetSpecificNormalization(String fieldName) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFieldSettings settings =\n        schema.getFieldConfigs().get(idMapping.getFieldID(fieldName)).getSettings();\n    Preconditions.checkState(settings.isSetIndexedFieldSettings(),\n                             \"Tweet text field must be indexed.\");\n    settings.getIndexedFieldSettings().setDeprecated_performTweetSpecificNormalizations(true);\n    return this;\n  }\n\n  /**\n   * Add a twitter photo facet field.\n   */\n  public final EarlybirdSchemaBuilder withPhotoUrlFacetField(String fieldName) {\n    if (!shouldIncludeField(fieldName)) {\n      return this;\n    }\n    ThriftFieldSettings photoFieldSettings = getNoPositionNoFreqSettings();\n    ThriftTokenStreamSerializer tokenStreamSerializer =\n        new ThriftTokenStreamSerializer(tokenStreamSerializerVersion);\n    tokenStreamSerializer.setAttributeSerializerClassNames(\n        ImmutableList.<String>of(\n            CharTermAttributeSerializer.class.getName(),\n            TermPayloadAttributeSerializer.class.getName()));\n    photoFieldSettings\n        .getIndexedFieldSettings()\n        .setTokenStreamSerializer(tokenStreamSerializer)\n        .setTokenized(true);\n    putIntoFieldConfigs(idMapping.getFieldID(fieldName),\n                        new ThriftFieldConfiguration(fieldName).setSettings(photoFieldSettings));\n    return this;\n  }\n\n  /**\n   * Returns whether the given field should be included or dropped.\n   */\n  @Override\n  protected boolean shouldIncludeField(String fieldName) {\n    return EarlybirdFieldConstants.getFieldConstant(fieldName).isValidFieldInCluster(cluster);\n  }\n}\n\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/earlybird/EarlybirdSchemaCreateTool.java",
    "content": "package com.twitter.search.common.schema.earlybird;\n\nimport java.util.Map;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Maps;\n\nimport com.twitter.common.text.util.TokenStreamSerializer;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.schema.AnalyzerFactory;\nimport com.twitter.search.common.schema.DynamicSchema;\nimport com.twitter.search.common.schema.ImmutableSchema;\nimport com.twitter.search.common.schema.SchemaBuilder;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFType;\nimport com.twitter.search.common.schema.thriftjava.ThriftFeatureUpdateConstraint;\nimport com.twitter.search.common.schema.thriftjava.ThriftSchema;\n\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.BLINK_FAVORITE_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.BLINK_QUOTE_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.BLINK_REPLY_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.BLINK_RETWEET_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.COMPOSER_SOURCE_IS_CAMERA_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.DECAYED_FAVORITE_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.DECAYED_QUOTE_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.DECAYED_REPLY_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.DECAYED_RETWEET_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EMBEDS_IMPRESSION_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EMBEDS_IMPRESSION_COUNT_V2;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EMBEDS_URL_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EMBEDS_URL_COUNT_V2;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXPERIMENTAL_HEALTH_MODEL_SCORE_1;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXPERIMENTAL_HEALTH_MODEL_SCORE_2;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXPERIMENTAL_HEALTH_MODEL_SCORE_3;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXPERIMENTAL_HEALTH_MODEL_SCORE_4;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXTENDED_FEATURE_UNUSED_BITS_0_24_8;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXTENDED_TEST_FEATURE_UNUSED_BITS_12_30_2;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXTENDED_TEST_FEATURE_UNUSED_BITS_13_30_2;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXTENDED_TEST_FEATURE_UNUSED_BITS_14_10_22;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXTENDED_TEST_FEATURE_UNUSED_BITS_16;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXTENDED_TEST_FEATURE_UNUSED_BITS_17;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXTENDED_TEST_FEATURE_UNUSED_BITS_18;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXTENDED_TEST_FEATURE_UNUSED_BITS_19;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXTENDED_TEST_FEATURE_UNUSED_BITS_20;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXTENDED_TEST_FEATURE_UNUSED_BITS_4_31_1;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.EXTENDED_TEST_FEATURE_UNUSED_BITS_7_6_26;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.FAKE_FAVORITE_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.FAKE_QUOTE_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.FAKE_REPLY_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.FAKE_RETWEET_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.FAVORITE_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.FAVORITE_COUNT_V2;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.FROM_BLUE_VERIFIED_ACCOUNT_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.FROM_VERIFIED_ACCOUNT_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_CARD_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_CONSUMER_VIDEO_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_EXPANDO_CARD_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_IMAGE_URL_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_LINK_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_MULTIPLE_HASHTAGS_OR_TRENDS_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_MULTIPLE_MEDIA_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_NATIVE_IMAGE_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_NEWS_URL_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_PERISCOPE_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_PRO_VIDEO_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_QUOTE_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_TREND_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_VIDEO_URL_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_VINE_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.HAS_VISIBLE_LINK_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.IS_NULLCAST_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.IS_OFFENSIVE_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.IS_REPLY_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.IS_RETWEET_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.IS_SENSITIVE_CONTENT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.IS_TRENDING_NOW_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.IS_USER_BOT_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.IS_USER_NEW_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.IS_USER_NSFW_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.IS_USER_SPAM_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LABEL_ABUSIVE_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LABEL_ABUSIVE_HI_RCL_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LABEL_DUP_CONTENT_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LABEL_NSFW_HI_PRC_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LABEL_NSFW_HI_RCL_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LABEL_SPAM_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LABEL_SPAM_HI_RCL_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LANGUAGE;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LAST_FAVORITE_SINCE_CREATION_HRS;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LAST_QUOTE_SINCE_CREATION_HRS;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LAST_REPLY_SINCE_CREATION_HRS;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LAST_RETWEET_SINCE_CREATION_HRS;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.LINK_LANGUAGE;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.NORMALIZED_FAVORITE_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.NORMALIZED_REPLY_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.NORMALIZED_RETWEET_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.NUM_HASHTAGS;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.NUM_HASHTAGS_V2;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.NUM_MENTIONS;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.NUM_MENTIONS_V2;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.NUM_STOCKS;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.PARUS_SCORE;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.PBLOCK_SCORE;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.PERISCOPE_EXISTS;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.PERISCOPE_HAS_BEEN_FEATURED;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.PERISCOPE_IS_CURRENTLY_FEATURED;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.PERISCOPE_IS_FROM_QUALITY_SOURCE;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.PERISCOPE_IS_LIVE;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.PREV_USER_TWEET_ENGAGEMENT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.PROFILE_IS_EGG_FLAG;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.P_REPORTED_TWEET_SCORE;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.P_SPAMMY_TWEET_SCORE;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.QUOTE_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_LEAST_SIGNIFICANT_INT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_MOST_SIGNIFICANT_INT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.REPLY_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.REPLY_COUNT_V2;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.RETWEET_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.RETWEET_COUNT_V2;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.SPAMMY_TWEET_CONTENT_SCORE;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.TEXT_SCORE;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.TOXICITY_SCORE;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.TWEET_SIGNATURE;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.USER_REPUTATION;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.VIDEO_VIEW_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.VIDEO_VIEW_COUNT_V2;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.VISIBLE_TOKEN_RATIO;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.WEIGHTED_FAVORITE_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.WEIGHTED_QUOTE_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.WEIGHTED_REPLY_COUNT;\nimport static com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant.WEIGHTED_RETWEET_COUNT;\n\n/**\n * Field configurations for Earlybird.\n */\npublic final class EarlybirdSchemaCreateTool {\n  // How many times a schema is built\n  private static final SearchCounter SCHEMA_BUILD_COUNT =\n      SearchCounter.export(\"schema_build_count\");\n\n  // Number of integers for the column of ENCODED_TWEET_FEATURES_FIELD.\n  @VisibleForTesting\n  public static final int NUMBER_OF_INTEGERS_FOR_FEATURES = 5;\n\n  // Number of integers for the column of EXTENDED_ENCODED_TWEET_FEATURES_FIELD.\n  // extra 80 bytes\n  // In realtime cluster, assuming 19 segments total, and 8388608 docs per segment\n  // this would amount to about 12.75GB of memory needed\n  //\n  @VisibleForTesting\n  public static final int NUMBER_OF_INTEGERS_FOR_EXTENDED_FEATURES = 20;\n\n  @VisibleForTesting\n  public static final Map<String, FeatureConfiguration> FEATURE_CONFIGURATION_MAP\n      = Maps.newLinkedHashMap();\n\n  public static final String BASE_FIELD_NAME =\n      EarlybirdFieldConstant.ENCODED_TWEET_FEATURES_FIELD.getFieldName();\n\n  private static String getBaseFieldName(String fullName) {\n    int index = fullName.indexOf(SchemaBuilder.CSF_VIEW_NAME_SEPARATOR);\n    Preconditions.checkArgument(index > 0);\n    return fullName.substring(0, index);\n  }\n\n  private static String getBaseFieldName(EarlybirdFieldConstant fieldConstant) {\n    return getBaseFieldName(fieldConstant.getFieldName());\n  }\n\n  private static String getFeatureNameInField(EarlybirdFieldConstant fieldConstant) {\n    int index = fieldConstant.getFieldName().indexOf(SchemaBuilder.CSF_VIEW_NAME_SEPARATOR);\n    Preconditions.checkArgument(index > 0);\n    return fieldConstant.getFieldName().substring(index + 1);\n  }\n\n  // defining all features\n  static {\n    // Add individual tweet encoded features as views on top of\n    // EarlybirdFieldConstant.ENCODED_TWEET_FEATURES_FIELD\n\n    // int intIndex, int bitStartPos, int bitLength\n    newEarlybirdFeatureConfiguration(IS_RETWEET_FLAG, ThriftCSFType.BOOLEAN, 0, 0, 1);\n    newEarlybirdFeatureConfiguration(IS_OFFENSIVE_FLAG, ThriftCSFType.BOOLEAN, 0, 1, 1);\n    newEarlybirdFeatureConfiguration(HAS_LINK_FLAG, ThriftCSFType.BOOLEAN, 0, 2, 1);\n    newEarlybirdFeatureConfiguration(HAS_TREND_FLAG, ThriftCSFType.BOOLEAN, 0, 3, 1);\n    newEarlybirdFeatureConfiguration(IS_REPLY_FLAG, ThriftCSFType.BOOLEAN, 0, 4, 1);\n    newEarlybirdFeatureConfiguration(IS_SENSITIVE_CONTENT, ThriftCSFType.BOOLEAN, 0, 5, 1);\n    newEarlybirdFeatureConfiguration(HAS_MULTIPLE_HASHTAGS_OR_TRENDS_FLAG,\n        ThriftCSFType.BOOLEAN, 0, 6, 1);\n    newEarlybirdFeatureConfiguration(FROM_VERIFIED_ACCOUNT_FLAG, ThriftCSFType.BOOLEAN, 0, 7, 1);\n    newEarlybirdFeatureConfiguration(TEXT_SCORE, ThriftCSFType.INT, 0, 8, 8);\n    newEarlybirdFeatureConfiguration(LANGUAGE, ThriftCSFType.INT, 0, 16, 8);\n    newEarlybirdFeatureConfiguration(LINK_LANGUAGE, ThriftCSFType.INT, 0, 24, 8);\n\n    newEarlybirdFeatureConfiguration(HAS_IMAGE_URL_FLAG, ThriftCSFType.BOOLEAN, 1, 0, 1);\n    newEarlybirdFeatureConfiguration(HAS_VIDEO_URL_FLAG, ThriftCSFType.BOOLEAN, 1, 1, 1);\n    newEarlybirdFeatureConfiguration(HAS_NEWS_URL_FLAG, ThriftCSFType.BOOLEAN, 1, 2, 1);\n    newEarlybirdFeatureConfiguration(HAS_EXPANDO_CARD_FLAG, ThriftCSFType.BOOLEAN, 1, 3, 1);\n    newEarlybirdFeatureConfiguration(HAS_MULTIPLE_MEDIA_FLAG, ThriftCSFType.BOOLEAN, 1, 4, 1);\n    newEarlybirdFeatureConfiguration(PROFILE_IS_EGG_FLAG, ThriftCSFType.BOOLEAN, 1, 5, 1);\n    newEarlybirdFeatureConfiguration(NUM_MENTIONS, ThriftCSFType.INT, 1, 6, 2);     // 0, 1, 2, 3+\n    newEarlybirdFeatureConfiguration(NUM_HASHTAGS, ThriftCSFType.INT, 1, 8, 2);     // 0, 1, 2, 3+\n    newEarlybirdFeatureConfiguration(HAS_CARD_FLAG, ThriftCSFType.BOOLEAN, 1, 10, 1);\n    newEarlybirdFeatureConfiguration(HAS_VISIBLE_LINK_FLAG, ThriftCSFType.BOOLEAN, 1, 11, 1);\n    newEarlybirdFeatureConfiguration(USER_REPUTATION, ThriftCSFType.INT, 1, 12, 8);\n    newEarlybirdFeatureConfiguration(IS_USER_SPAM_FLAG, ThriftCSFType.BOOLEAN, 1, 20, 1);\n    newEarlybirdFeatureConfiguration(IS_USER_NSFW_FLAG, ThriftCSFType.BOOLEAN, 1, 21, 1);\n    newEarlybirdFeatureConfiguration(IS_USER_BOT_FLAG, ThriftCSFType.BOOLEAN, 1, 22, 1);\n    newEarlybirdFeatureConfiguration(IS_USER_NEW_FLAG, ThriftCSFType.BOOLEAN, 1, 23, 1);\n    newEarlybirdFeatureConfiguration(PREV_USER_TWEET_ENGAGEMENT, ThriftCSFType.INT, 1, 24, 6);\n    newEarlybirdFeatureConfiguration(COMPOSER_SOURCE_IS_CAMERA_FLAG,\n        ThriftCSFType.BOOLEAN, 1, 30, 1);\n    newEarlybirdFeatureConfiguration(IS_NULLCAST_FLAG, ThriftCSFType.BOOLEAN, 1, 31, 1);\n\n    newEarlybirdFeatureConfiguration(RETWEET_COUNT, ThriftCSFType.DOUBLE, 2, 0, 8,\n        ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(FAVORITE_COUNT, ThriftCSFType.DOUBLE, 2, 8, 8,\n        ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(REPLY_COUNT, ThriftCSFType.DOUBLE, 2, 16, 8,\n        ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(PARUS_SCORE, ThriftCSFType.DOUBLE, 2, 24, 8);\n\n    newEarlybirdFeatureConfiguration(HAS_CONSUMER_VIDEO_FLAG, ThriftCSFType.BOOLEAN, 3, 0, 1);\n    newEarlybirdFeatureConfiguration(HAS_PRO_VIDEO_FLAG, ThriftCSFType.BOOLEAN, 3, 1, 1);\n    newEarlybirdFeatureConfiguration(HAS_VINE_FLAG, ThriftCSFType.BOOLEAN, 3, 2, 1);\n    newEarlybirdFeatureConfiguration(HAS_PERISCOPE_FLAG, ThriftCSFType.BOOLEAN, 3, 3, 1);\n    newEarlybirdFeatureConfiguration(HAS_NATIVE_IMAGE_FLAG, ThriftCSFType.BOOLEAN, 3, 4, 1);\n    // NOTE: There are 3 bits left in the first byte of INT 3, if possible, please reserve them\n    // for future media types (SEARCH-9131)\n    // newEarlybirdFeatureConfiguration(FUTURE_MEDIA_BITS, ThriftCSFType.INT, 3, 5, 3);\n\n    newEarlybirdFeatureConfiguration(VISIBLE_TOKEN_RATIO, ThriftCSFType.INT, 3, 8, 4);\n    newEarlybirdFeatureConfiguration(HAS_QUOTE_FLAG, ThriftCSFType.BOOLEAN, 3, 12, 1);\n    newEarlybirdFeatureConfiguration(FROM_BLUE_VERIFIED_ACCOUNT_FLAG,\n        ThriftCSFType.BOOLEAN, 3, 13, 1);\n    // Unused bits from bit 14 to bit 31 (18 bits)\n    // newEarlybirdFeatureConfiguration(UNUSED_BITS, ThriftCSFType.INT, 3, 14, 18);\n\n    newEarlybirdFeatureConfiguration(TWEET_SIGNATURE, ThriftCSFType.INT, 4, 0, 32);\n\n    newEarlybirdFeatureConfiguration(EMBEDS_IMPRESSION_COUNT,\n        ThriftCSFType.DOUBLE, 0, 0, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(EMBEDS_URL_COUNT,\n        ThriftCSFType.DOUBLE, 0, 8, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(VIDEO_VIEW_COUNT,\n        ThriftCSFType.DOUBLE, 0, 16, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n\n    // Unused bits from bit 24 to bit 31 (8 bits).\n    // This used to be a feature that was decommissioned (SEARCHQUAL-10321)\n    newEarlybirdFeatureConfiguration(EXTENDED_FEATURE_UNUSED_BITS_0_24_8,\n        ThriftCSFType.INT, 0, 24, 8);\n\n    newEarlybirdFeatureConfiguration(REFERENCE_AUTHOR_ID_LEAST_SIGNIFICANT_INT,\n        ThriftCSFType.INT, 1, 0, 32, ThriftFeatureUpdateConstraint.IMMUTABLE);\n    newEarlybirdFeatureConfiguration(REFERENCE_AUTHOR_ID_MOST_SIGNIFICANT_INT,\n        ThriftCSFType.INT, 2, 0, 32, ThriftFeatureUpdateConstraint.IMMUTABLE);\n\n    newEarlybirdFeatureConfiguration(RETWEET_COUNT_V2,\n        ThriftCSFType.DOUBLE, 3, 0, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(FAVORITE_COUNT_V2,\n        ThriftCSFType.DOUBLE, 3, 8, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(REPLY_COUNT_V2,\n        ThriftCSFType.DOUBLE, 3, 16, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(EMBEDS_IMPRESSION_COUNT_V2,\n        ThriftCSFType.DOUBLE, 3, 24, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n\n    newEarlybirdFeatureConfiguration(EMBEDS_URL_COUNT_V2,\n        ThriftCSFType.DOUBLE, 4, 0, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(VIDEO_VIEW_COUNT_V2,\n        ThriftCSFType.DOUBLE, 4, 8, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(QUOTE_COUNT,\n        ThriftCSFType.DOUBLE, 4, 16, 8);\n\n    newEarlybirdFeatureConfiguration(LABEL_ABUSIVE_FLAG,        ThriftCSFType.BOOLEAN, 4, 24, 1);\n    newEarlybirdFeatureConfiguration(LABEL_ABUSIVE_HI_RCL_FLAG, ThriftCSFType.BOOLEAN, 4, 25, 1);\n    newEarlybirdFeatureConfiguration(LABEL_DUP_CONTENT_FLAG,    ThriftCSFType.BOOLEAN, 4, 26, 1);\n    newEarlybirdFeatureConfiguration(LABEL_NSFW_HI_PRC_FLAG,    ThriftCSFType.BOOLEAN, 4, 27, 1);\n    newEarlybirdFeatureConfiguration(LABEL_NSFW_HI_RCL_FLAG,    ThriftCSFType.BOOLEAN, 4, 28, 1);\n    newEarlybirdFeatureConfiguration(LABEL_SPAM_FLAG,           ThriftCSFType.BOOLEAN, 4, 29, 1);\n    newEarlybirdFeatureConfiguration(LABEL_SPAM_HI_RCL_FLAG,    ThriftCSFType.BOOLEAN, 4, 30, 1);\n\n    newEarlybirdFeatureConfiguration(EXTENDED_TEST_FEATURE_UNUSED_BITS_4_31_1,\n        ThriftCSFType.INT, 4, 31, 1);\n\n    newEarlybirdFeatureConfiguration(WEIGHTED_RETWEET_COUNT,\n        ThriftCSFType.DOUBLE, 5, 0, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(WEIGHTED_REPLY_COUNT,\n        ThriftCSFType.DOUBLE, 5, 8, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(WEIGHTED_FAVORITE_COUNT,\n        ThriftCSFType.DOUBLE, 5, 16, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(WEIGHTED_QUOTE_COUNT,\n        ThriftCSFType.DOUBLE, 5, 24, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n\n    newEarlybirdFeatureConfiguration(PERISCOPE_EXISTS,\n        ThriftCSFType.BOOLEAN, 6, 0, 1);\n    newEarlybirdFeatureConfiguration(PERISCOPE_HAS_BEEN_FEATURED,\n        ThriftCSFType.BOOLEAN, 6, 1, 1);\n    newEarlybirdFeatureConfiguration(PERISCOPE_IS_CURRENTLY_FEATURED,\n        ThriftCSFType.BOOLEAN, 6, 2, 1);\n    newEarlybirdFeatureConfiguration(PERISCOPE_IS_FROM_QUALITY_SOURCE,\n        ThriftCSFType.BOOLEAN, 6, 3, 1);\n    newEarlybirdFeatureConfiguration(PERISCOPE_IS_LIVE,\n        ThriftCSFType.BOOLEAN, 6, 4, 1);\n\n    newEarlybirdFeatureConfiguration(IS_TRENDING_NOW_FLAG,\n        ThriftCSFType.BOOLEAN, 6, 5, 1);\n\n    // remaining bits for integer 6\n    newEarlybirdFeatureConfiguration(EXTENDED_TEST_FEATURE_UNUSED_BITS_7_6_26,\n        ThriftCSFType.INT, 6, 6, 26);\n\n    // The decaying counters can become smaller\n    newEarlybirdFeatureConfiguration(DECAYED_RETWEET_COUNT,\n        ThriftCSFType.DOUBLE, 7, 0, 8, ThriftFeatureUpdateConstraint.POSITIVE);\n    newEarlybirdFeatureConfiguration(DECAYED_REPLY_COUNT,\n        ThriftCSFType.DOUBLE, 7, 8, 8, ThriftFeatureUpdateConstraint.POSITIVE);\n    newEarlybirdFeatureConfiguration(DECAYED_FAVORITE_COUNT,\n        ThriftCSFType.DOUBLE, 7, 16, 8, ThriftFeatureUpdateConstraint.POSITIVE);\n    newEarlybirdFeatureConfiguration(DECAYED_QUOTE_COUNT,\n        ThriftCSFType.DOUBLE, 7, 24, 8, ThriftFeatureUpdateConstraint.POSITIVE);\n\n    // The fake engagement counters.\n    newEarlybirdFeatureConfiguration(FAKE_RETWEET_COUNT,\n        ThriftCSFType.DOUBLE, 8, 0, 8, ThriftFeatureUpdateConstraint.POSITIVE);\n    newEarlybirdFeatureConfiguration(FAKE_REPLY_COUNT,\n        ThriftCSFType.DOUBLE, 8, 8, 8, ThriftFeatureUpdateConstraint.POSITIVE);\n    newEarlybirdFeatureConfiguration(FAKE_FAVORITE_COUNT,\n        ThriftCSFType.DOUBLE, 8, 16, 8, ThriftFeatureUpdateConstraint.POSITIVE);\n    newEarlybirdFeatureConfiguration(FAKE_QUOTE_COUNT,\n        ThriftCSFType.DOUBLE, 8, 24, 8, ThriftFeatureUpdateConstraint.POSITIVE);\n\n    newEarlybirdFeatureConfiguration(LAST_RETWEET_SINCE_CREATION_HRS,\n        ThriftCSFType.INT, 9, 0, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(LAST_REPLY_SINCE_CREATION_HRS,\n        ThriftCSFType.INT, 9, 8, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(LAST_FAVORITE_SINCE_CREATION_HRS,\n        ThriftCSFType.INT, 9, 16, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n    newEarlybirdFeatureConfiguration(LAST_QUOTE_SINCE_CREATION_HRS,\n        ThriftCSFType.INT, 9, 24, 8, ThriftFeatureUpdateConstraint.INC_ONLY);\n\n    newEarlybirdFeatureConfiguration(NUM_HASHTAGS_V2,\n        ThriftCSFType.INT, 10, 0, 4);\n    newEarlybirdFeatureConfiguration(NUM_MENTIONS_V2,\n        ThriftCSFType.INT, 10, 4, 4);\n    newEarlybirdFeatureConfiguration(NUM_STOCKS,\n        ThriftCSFType.INT, 10, 8, 4);\n\n    // Remaining bits for integer 10\n    // Production Toxicity and PBlock score from HML (go/toxicity, go/pblock)\n    newEarlybirdFeatureConfiguration(TOXICITY_SCORE,\n        ThriftCSFType.DOUBLE, 10, 12, 10);\n    newEarlybirdFeatureConfiguration(PBLOCK_SCORE,\n        ThriftCSFType.DOUBLE, 10, 22, 10);\n\n    // The blink engagement counters\n    newEarlybirdFeatureConfiguration(BLINK_RETWEET_COUNT,\n        ThriftCSFType.DOUBLE, 11, 0, 8, ThriftFeatureUpdateConstraint.POSITIVE);\n    newEarlybirdFeatureConfiguration(BLINK_REPLY_COUNT,\n        ThriftCSFType.DOUBLE, 11, 8, 8, ThriftFeatureUpdateConstraint.POSITIVE);\n    newEarlybirdFeatureConfiguration(BLINK_FAVORITE_COUNT,\n        ThriftCSFType.DOUBLE, 11, 16, 8, ThriftFeatureUpdateConstraint.POSITIVE);\n    newEarlybirdFeatureConfiguration(BLINK_QUOTE_COUNT,\n        ThriftCSFType.DOUBLE, 11, 24, 8, ThriftFeatureUpdateConstraint.POSITIVE);\n\n    // Experimental health model scores from HML\n    newEarlybirdFeatureConfiguration(EXPERIMENTAL_HEALTH_MODEL_SCORE_1,\n        ThriftCSFType.DOUBLE, 12, 0, 10);\n    newEarlybirdFeatureConfiguration(EXPERIMENTAL_HEALTH_MODEL_SCORE_2,\n        ThriftCSFType.DOUBLE, 12, 10, 10);\n    newEarlybirdFeatureConfiguration(EXPERIMENTAL_HEALTH_MODEL_SCORE_3,\n        ThriftCSFType.DOUBLE, 12, 20, 10);\n    // remaining bits for integer 12\n    newEarlybirdFeatureConfiguration(EXTENDED_TEST_FEATURE_UNUSED_BITS_12_30_2,\n        ThriftCSFType.INT, 12, 30, 2);\n\n    // Experimental health model scores from HML (cont.)\n    newEarlybirdFeatureConfiguration(EXPERIMENTAL_HEALTH_MODEL_SCORE_4,\n        ThriftCSFType.DOUBLE, 13, 0, 10);\n    // Production pSpammyTweet score from HML (go/pspammytweet)\n    newEarlybirdFeatureConfiguration(P_SPAMMY_TWEET_SCORE,\n        ThriftCSFType.DOUBLE, 13, 10, 10);\n    // Production pReportedTweet score from HML (go/preportedtweet)\n    newEarlybirdFeatureConfiguration(P_REPORTED_TWEET_SCORE,\n        ThriftCSFType.DOUBLE, 13, 20, 10);\n    // remaining bits for integer 13\n    newEarlybirdFeatureConfiguration(EXTENDED_TEST_FEATURE_UNUSED_BITS_13_30_2,\n        ThriftCSFType.INT, 13, 30, 2);\n\n    // Experimental health model scores from HML (cont.)\n    // Prod Spammy Tweet Content model score from Platform Manipulation (go/spammy-tweet-content)\n    newEarlybirdFeatureConfiguration(SPAMMY_TWEET_CONTENT_SCORE,\n        ThriftCSFType.DOUBLE, 14, 0, 10);\n    // remaining bits for integer 14\n    newEarlybirdFeatureConfiguration(EXTENDED_TEST_FEATURE_UNUSED_BITS_14_10_22,\n        ThriftCSFType.INT, 14, 10, 22);\n\n    // Note that the integer index below is 0-based, but the index j in UNUSED_BITS_{j} below\n    // is 1-based.\n    newEarlybirdFeatureConfiguration(EXTENDED_TEST_FEATURE_UNUSED_BITS_16,\n        ThriftCSFType.INT, 15, 0, 32);\n    newEarlybirdFeatureConfiguration(EXTENDED_TEST_FEATURE_UNUSED_BITS_17,\n        ThriftCSFType.INT, 16, 0, 32);\n    newEarlybirdFeatureConfiguration(EXTENDED_TEST_FEATURE_UNUSED_BITS_18,\n        ThriftCSFType.INT, 17, 0, 32);\n    newEarlybirdFeatureConfiguration(EXTENDED_TEST_FEATURE_UNUSED_BITS_19,\n        ThriftCSFType.INT, 18, 0, 32);\n    newEarlybirdFeatureConfiguration(EXTENDED_TEST_FEATURE_UNUSED_BITS_20,\n        ThriftCSFType.INT, 19, 0, 32);\n  }\n\n  private EarlybirdSchemaCreateTool() { }\n\n  /**\n   * Get schema for the Earlybird.\n   */\n  public static DynamicSchema buildSchema(EarlybirdCluster cluster)\n      throws Schema.SchemaValidationException {\n    SCHEMA_BUILD_COUNT.increment();\n    return new DynamicSchema(new ImmutableSchema(buildThriftSchema(cluster),\n                                                 new AnalyzerFactory(),\n                                                 cluster.getNameForStats()));\n  }\n\n  /**\n   * Get schema for the Earlybird, can throw runtime exception.  This is mostly for static schema\n   * usage, which does not care about schema updates.\n   */\n  @VisibleForTesting\n  public static DynamicSchema buildSchemaWithRuntimeException(EarlybirdCluster cluster) {\n    try {\n      return buildSchema(cluster);\n    } catch (Schema.SchemaValidationException e) {\n      throw new RuntimeException(e);\n    }\n  }\n\n  private static FeatureConfiguration newEarlybirdFeatureConfiguration(\n      EarlybirdFieldConstant fieldConstant,\n      ThriftCSFType type,\n      int intIndex, int bitStartPos, int bitLength,\n      ThriftFeatureUpdateConstraint... constraints) {\n\n    if (!fieldConstant.isFlagFeatureField() && type == ThriftCSFType.BOOLEAN) {\n      throw new IllegalArgumentException(\n          \"Non-flag feature field configured with boolean Thrift type: \" + fieldConstant);\n    }\n    if (fieldConstant.isFlagFeatureField() && type != ThriftCSFType.BOOLEAN) {\n      throw new IllegalArgumentException(\n          \"Flag feature field configured with non-boolean Thrift type: \" + fieldConstant);\n    }\n\n    String baseFieldName = getBaseFieldName(fieldConstant);\n    String name = getFeatureNameInField(fieldConstant);\n    FeatureConfiguration.Builder builder = FeatureConfiguration.builder()\n        .withName(name)\n        .withType(type)\n        .withBitRange(intIndex, bitStartPos, bitLength);\n    // remove the following line once we configure features purely by the schema\n    builder.withBaseField(baseFieldName);\n\n    if (!fieldConstant.isUnusedField()) {\n      builder.withOutputType(type);\n    }\n    if (fieldConstant.getFeatureNormalizationType() != null) {\n      builder.withFeatureNormalizationType(fieldConstant.getFeatureNormalizationType());\n    }\n\n    for (ThriftFeatureUpdateConstraint constraint : constraints) {\n      builder.withFeatureUpdateConstraint(constraint);\n    }\n    FeatureConfiguration featureConfiguration = builder.build();\n    FEATURE_CONFIGURATION_MAP.put(fieldConstant.getFieldName(), featureConfiguration);\n    return featureConfiguration;\n  }\n\n  /**\n   * Build ThriftSchema for the Earlybird. Note that the schema returned can be used\n   * all Earlybird clusters. However, some clusters may not use all the field configurations.\n   */\n  @VisibleForTesting\n  public static ThriftSchema buildThriftSchema(EarlybirdCluster cluster) {\n    EarlybirdSchemaBuilder builder = new EarlybirdSchemaBuilder(\n        new EarlybirdFieldConstants(), cluster, TokenStreamSerializer.Version.VERSION_2);\n\n    builder.withSchemaVersion(\n        FlushVersion.CURRENT_FLUSH_VERSION.getVersionNumber(),\n        FlushVersion.CURRENT_FLUSH_VERSION.getMinorVersion(),\n        FlushVersion.CURRENT_FLUSH_VERSION.getDescription(),\n        FlushVersion.CURRENT_FLUSH_VERSION.isOfficial());\n\n    // ID field, used for partitioning\n    builder.withPartitionFieldId(0)\n        .withSortableLongTermField(EarlybirdFieldConstant.ID_FIELD.getFieldName())\n        // Text Fields that are searched by default\n        .withTextField(EarlybirdFieldConstant.RESOLVED_LINKS_TEXT_FIELD.getFieldName(), true)\n        .withSearchFieldByDefault(\n            EarlybirdFieldConstant.RESOLVED_LINKS_TEXT_FIELD.getFieldName(), 0.1f)\n        .withPretokenizedTextField(EarlybirdFieldConstant.TEXT_FIELD.getFieldName(), true)\n        .withSearchFieldByDefault(EarlybirdFieldConstant.TEXT_FIELD.getFieldName(), 1.0f);\n    builder.withTweetSpecificNormalization(EarlybirdFieldConstant.TEXT_FIELD.getFieldName())\n        .withTextField(EarlybirdFieldConstant.TOKENIZED_FROM_USER_FIELD.getFieldName(), true)\n        .withSearchFieldByDefault(\n            EarlybirdFieldConstant.TOKENIZED_FROM_USER_FIELD.getFieldName(), 0.2f)\n\n        // Text fields not searched by default\n        .withTextField(EarlybirdFieldConstant.FROM_USER_FIELD.getFieldName(), false)\n        .withTextField(EarlybirdFieldConstant.TO_USER_FIELD.getFieldName(), false)\n\n        // cards are not searched by default, and have weight 0.\n        .withPretokenizedTextField(EarlybirdFieldConstant.CARD_TITLE_FIELD.getFieldName(), false)\n        .withPretokenizedTextField(\n            EarlybirdFieldConstant.CARD_DESCRIPTION_FIELD.getFieldName(), false)\n        .withTextField(EarlybirdFieldConstant.CARD_LANG.getFieldName(), false)\n\n        // Out-of-order append fields\n        .withLongTermField(EarlybirdFieldConstant.LIKED_BY_USER_ID_FIELD.getFieldName())\n        .withLongTermField(EarlybirdFieldConstant.RETWEETED_BY_USER_ID.getFieldName())\n        .withLongTermField(EarlybirdFieldConstant.REPLIED_TO_BY_USER_ID.getFieldName())\n\n        // No Position fields, sorted alphabetically\n        .withPretokenizedNoPositionField(EarlybirdFieldConstant.CARD_DOMAIN_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.CARD_NAME_FIELD.getFieldName())\n        .withIntTermField(EarlybirdFieldConstant.CREATED_AT_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.ENTITY_ID_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.GEO_HASH_FIELD.getFieldName())\n        .withLongTermField(EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName())\n        .withLongTermField(EarlybirdFieldConstant.IN_REPLY_TO_TWEET_ID_FIELD.getFieldName())\n        .withLongTermField(EarlybirdFieldConstant.IN_REPLY_TO_USER_ID_FIELD.getFieldName())\n        .withLongTermField(EarlybirdFieldConstant.RETWEET_SOURCE_TWEET_ID_FIELD.getFieldName())\n        .withLongTermField(EarlybirdFieldConstant.RETWEET_SOURCE_USER_ID_FIELD.getFieldName())\n        .withLongTermField(EarlybirdFieldConstant.CONVERSATION_ID_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.PLACE_ID_FIELD.getFieldName())\n        .withTextField(EarlybirdFieldConstant.PLACE_FULL_NAME_FIELD.getFieldName(), false)\n        .withIndexedNotTokenizedField(\n            EarlybirdFieldConstant.PLACE_COUNTRY_CODE_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(\n            EarlybirdFieldConstant.PROFILE_GEO_COUNTRY_CODE_FIELD.getFieldName())\n        .withTextField(EarlybirdFieldConstant.PROFILE_GEO_REGION_FIELD.getFieldName(), false)\n        .withTextField(EarlybirdFieldConstant.PROFILE_GEO_LOCALITY_FIELD.getFieldName(), false)\n        .withTermTextLookup(EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName())\n        .withTermTextLookup(EarlybirdFieldConstant.IN_REPLY_TO_USER_ID_FIELD.getFieldName())\n        .withPretokenizedNoPositionField(EarlybirdFieldConstant.HASHTAGS_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(ImmutableSchema.HF_PHRASE_PAIRS_FIELD)\n        .withIndexedNotTokenizedField(ImmutableSchema.HF_TERM_PAIRS_FIELD)\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.IMAGE_LINKS_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.ISO_LANGUAGE_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.LINKS_FIELD.getFieldName())\n        .withIntTermField(EarlybirdFieldConstant.LINK_CATEGORY_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.MENTIONS_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.NEWS_LINKS_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.NORMALIZED_SOURCE_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.PLACE_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.SOURCE_FIELD.getFieldName())\n        .withPretokenizedNoPositionField(EarlybirdFieldConstant.STOCKS_FIELD.getFieldName())\n        .withIndexedNotTokenizedField(EarlybirdFieldConstant.VIDEO_LINKS_FIELD.getFieldName())\n        .withIntTermField(NORMALIZED_FAVORITE_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD.getFieldName())\n        .withIntTermField(NORMALIZED_REPLY_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD.getFieldName())\n        .withIntTermField(NORMALIZED_RETWEET_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD.getFieldName())\n\n        .withIntTermField(EarlybirdFieldConstant.COMPOSER_SOURCE.getFieldName())\n\n        .withLongTermField(EarlybirdFieldConstant.QUOTED_TWEET_ID_FIELD.getFieldName())\n        .withLongTermField(EarlybirdFieldConstant.QUOTED_USER_ID_FIELD.getFieldName())\n        .withLongTermField(EarlybirdFieldConstant.DIRECTED_AT_USER_ID_FIELD.getFieldName())\n\n        // Named entity fields\n        .withIndexedNotTokenizedField(\n            EarlybirdFieldConstant.NAMED_ENTITY_FROM_URL_FIELD.getFieldName(), true)\n        .withIndexedNotTokenizedField(\n            EarlybirdFieldConstant.NAMED_ENTITY_FROM_TEXT_FIELD.getFieldName(), true)\n        .withIndexedNotTokenizedField(\n            EarlybirdFieldConstant.NAMED_ENTITY_WITH_TYPE_FROM_URL_FIELD.getFieldName(), true)\n        .withIndexedNotTokenizedField(\n            EarlybirdFieldConstant.NAMED_ENTITY_WITH_TYPE_FROM_TEXT_FIELD.getFieldName(), true)\n\n        // camelCase-tokenized user handles and tokenized user names, not searchable by default\n        .withPretokenizedTextField(\n            EarlybirdFieldConstant.CAMELCASE_USER_HANDLE_FIELD.getFieldName(), false)\n        .withPretokenizedTextField(\n            EarlybirdFieldConstant.TOKENIZED_USER_NAME_FIELD.getFieldName(), false)\n\n        .withIndexedNotTokenizedField(\n            EarlybirdFieldConstant.SPACE_ID_FIELD.getFieldName())\n        .withTextField(EarlybirdFieldConstant.SPACE_ADMIN_FIELD.getFieldName(), false)\n        .withPretokenizedTextField(EarlybirdFieldConstant.SPACE_TITLE_FIELD.getFieldName(), false)\n        .withTextField(EarlybirdFieldConstant.TOKENIZED_SPACE_ADMIN_FIELD.getFieldName(), true)\n        .withPretokenizedTextField(\n            EarlybirdFieldConstant.CAMELCASE_TOKENIZED_SPACE_ADMIN_FIELD.getFieldName(), false)\n        .withPretokenizedTextField(\n            EarlybirdFieldConstant.TOKENIZED_SPACE_ADMIN_DISPLAY_NAME_FIELD.getFieldName(), false)\n        .withPretokenizedTextField(\n            EarlybirdFieldConstant.URL_DESCRIPTION_FIELD.getFieldName(), false)\n        .withPretokenizedTextField(\n            EarlybirdFieldConstant.URL_TITLE_FIELD.getFieldName(), false);\n\n    builder\n        .withPhotoUrlFacetField(EarlybirdFieldConstant.TWIMG_LINKS_FIELD.getFieldName())\n        .withOutOfOrderEnabledForField(\n            EarlybirdFieldConstant.LIKED_BY_USER_ID_FIELD.getFieldName())\n        .withOutOfOrderEnabledForField(\n            EarlybirdFieldConstant.RETWEETED_BY_USER_ID.getFieldName())\n        .withOutOfOrderEnabledForField(\n            EarlybirdFieldConstant.REPLIED_TO_BY_USER_ID.getFieldName());\n\n    // ColumnStrideFields.\n    boolean loadCSFIntoRAMDefault = cluster != EarlybirdCluster.FULL_ARCHIVE;\n\n    builder\n        .withColumnStrideField(EarlybirdFieldConstants.ENCODED_TWEET_FEATURES_FIELD_NAME,\n                ThriftCSFType.INT, NUMBER_OF_INTEGERS_FOR_FEATURES,\n                true, loadCSFIntoRAMDefault)\n        .withColumnStrideField(EarlybirdFieldConstant.FROM_USER_ID_CSF.getFieldName(),\n            ThriftCSFType.LONG, 1, false, /* the full archive loads this field into RAM */ true)\n        .withColumnStrideField(EarlybirdFieldConstant.SHARED_STATUS_ID_CSF.getFieldName(),\n                ThriftCSFType.LONG, 1, false, loadCSFIntoRAMDefault)\n        .withColumnStrideField(EarlybirdFieldConstant.CARD_TYPE_CSF_FIELD.getFieldName(),\n                ThriftCSFType.BYTE, 1, false, loadCSFIntoRAMDefault)\n         // CSF Used by archive mappers\n        .withColumnStrideField(EarlybirdFieldConstant.CREATED_AT_CSF_FIELD.getFieldName(),\n            ThriftCSFType.INT, 1, false, /* the full archive loads this field into RAM */ true)\n        .withColumnStrideField(EarlybirdFieldConstant.ID_CSF_FIELD.getFieldName(),\n            ThriftCSFType.LONG, 1, false, /* the full archive loads this field into RAM */ true)\n        .withColumnStrideField(EarlybirdFieldConstant.LAT_LON_CSF_FIELD.getFieldName(),\n            ThriftCSFType.LONG, 1, false, loadCSFIntoRAMDefault)\n        .withColumnStrideField(EarlybirdFieldConstant.CONVERSATION_ID_CSF.getFieldName(),\n            ThriftCSFType.LONG, 1, false, loadCSFIntoRAMDefault)\n        .withColumnStrideField(EarlybirdFieldConstant.QUOTED_TWEET_ID_CSF.getFieldName(),\n            ThriftCSFType.LONG, 1, false, loadCSFIntoRAMDefault)\n        .withColumnStrideField(EarlybirdFieldConstant.QUOTED_USER_ID_CSF.getFieldName(),\n            ThriftCSFType.LONG, 1, false, loadCSFIntoRAMDefault)\n        .withColumnStrideField(EarlybirdFieldConstant.CARD_LANG_CSF.getFieldName(),\n            ThriftCSFType.INT, 1, false, loadCSFIntoRAMDefault)\n        .withColumnStrideField(EarlybirdFieldConstant.CARD_URI_CSF.getFieldName(),\n            ThriftCSFType.LONG, 1, false, loadCSFIntoRAMDefault)\n        .withColumnStrideField(EarlybirdFieldConstant.DIRECTED_AT_USER_ID_CSF.getFieldName(),\n            ThriftCSFType.LONG, 1, false, loadCSFIntoRAMDefault)\n        .withColumnStrideField(EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_CSF.getFieldName(),\n            ThriftCSFType.LONG, 1, false, loadCSFIntoRAMDefault)\n        .withColumnStrideField(\n            EarlybirdFieldConstant.EXCLUSIVE_CONVERSATION_AUTHOR_ID_CSF.getFieldName(),\n            ThriftCSFType.LONG, 1, false, loadCSFIntoRAMDefault)\n\n    /* Semicolon on separate line to preserve git blame. */;\n\n    builder.withColumnStrideField(\n        EarlybirdFieldConstants.EXTENDED_ENCODED_TWEET_FEATURES_FIELD_NAME,\n        ThriftCSFType.INT, NUMBER_OF_INTEGERS_FOR_EXTENDED_FEATURES,\n        true, loadCSFIntoRAMDefault);\n\n    for (Map.Entry<String, FeatureConfiguration> entry : FEATURE_CONFIGURATION_MAP.entrySet()) {\n      String fullName = entry.getKey();\n      String baseName = getBaseFieldName(fullName);\n      EarlybirdFieldConstant fieldConstant = EarlybirdFieldConstants.getFieldConstant(fullName);\n      if (fieldConstant.isValidFieldInCluster(cluster)) {\n        builder.withFeatureConfiguration(baseName, fullName, entry.getValue());\n      }\n    }\n    // Add facet settings for facet fields\n    // boolean args are respectively whether to use skiplist, whether offensive, whether to use CSF\n    builder\n        .withFacetConfigs(EarlybirdFieldConstant.MENTIONS_FIELD.getFieldName(),\n            EarlybirdFieldConstant.MENTIONS_FACET, true, false, false)\n        .withFacetConfigs(EarlybirdFieldConstant.HASHTAGS_FIELD.getFieldName(),\n            EarlybirdFieldConstant.HASHTAGS_FACET, true, false, false)\n        .withFacetConfigs(EarlybirdFieldConstant.STOCKS_FIELD.getFieldName(),\n            EarlybirdFieldConstant.STOCKS_FACET, true, false, false)\n        .withFacetConfigs(EarlybirdFieldConstant.IMAGE_LINKS_FIELD.getFieldName(),\n            EarlybirdFieldConstant.IMAGES_FACET, true, true, false)\n        .withFacetConfigs(EarlybirdFieldConstant.VIDEO_LINKS_FIELD.getFieldName(),\n            EarlybirdFieldConstant.VIDEOS_FACET, true, true, false)\n        .withFacetConfigs(EarlybirdFieldConstant.NEWS_LINKS_FIELD.getFieldName(),\n            EarlybirdFieldConstant.NEWS_FACET, true, false, false)\n        .withFacetConfigs(EarlybirdFieldConstant.ISO_LANGUAGE_FIELD.getFieldName(),\n            EarlybirdFieldConstant.LANGUAGES_FACET, false, false, false)\n        .withFacetConfigs(EarlybirdFieldConstant.SOURCE_FIELD.getFieldName(),\n            EarlybirdFieldConstant.SOURCES_FACET, false, false, false)\n        .withFacetConfigs(EarlybirdFieldConstant.TWIMG_LINKS_FIELD.getFieldName(),\n            EarlybirdFieldConstant.TWIMG_FACET, true, true, false)\n        .withFacetConfigs(EarlybirdFieldConstant.FROM_USER_ID_CSF.getFieldName(),\n            EarlybirdFieldConstant.FROM_USER_ID_FACET, false, false, true /* facet on CSF */)\n        .withFacetConfigs(EarlybirdFieldConstant.SHARED_STATUS_ID_CSF.getFieldName(),\n            EarlybirdFieldConstant.RETWEETS_FACET, false, false, true /* facet on CSF */)\n        .withFacetConfigs(EarlybirdFieldConstant.LINKS_FIELD.getFieldName(),\n            EarlybirdFieldConstant.LINKS_FACET, true, false, false)\n        .withFacetConfigs(\n            EarlybirdFieldConstant.NAMED_ENTITY_WITH_TYPE_FROM_URL_FIELD.getFieldName(),\n            true, false, false)\n        .withFacetConfigs(\n            EarlybirdFieldConstant.NAMED_ENTITY_WITH_TYPE_FROM_TEXT_FIELD.getFieldName(),\n            true, false, false)\n        .withFacetConfigs(\n            EarlybirdFieldConstant.ENTITY_ID_FIELD.getFieldName(),\n            true, false, false)\n        .withFacetConfigs(EarlybirdFieldConstant.SPACE_ID_FIELD.getFieldName(),\n            EarlybirdFieldConstant.SPACES_FACET, true, false, false);\n    return builder.build();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/earlybird/EarlybirdThriftDocumentBuilder.java",
    "content": "package com.twitter.search.common.schema.earlybird;\n\nimport java.io.IOException;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Set;\nimport javax.annotation.Nonnull;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.ImmutableSet;\nimport com.google.common.collect.Sets;\n\nimport org.apache.commons.lang.StringUtils;\nimport org.apache.lucene.analysis.TokenStream;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.common.text.util.TokenStreamSerializer;\nimport com.twitter.cuad.ner.plain.thriftjava.NamedEntity;\nimport com.twitter.cuad.ner.plain.thriftjava.NamedEntityContext;\nimport com.twitter.cuad.ner.plain.thriftjava.NamedEntityInputSourceType;\nimport com.twitter.cuad.ner.thriftjava.WholeEntityType;\nimport com.twitter.search.common.constants.SearchCardType;\nimport com.twitter.search.common.indexing.thriftjava.ThriftExpandedUrl;\nimport com.twitter.search.common.indexing.thriftjava.ThriftGeoLocationSource;\nimport com.twitter.search.common.indexing.thriftjava.TwitterPhotoUrl;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.schema.ThriftDocumentBuilder;\nimport com.twitter.search.common.schema.base.FieldNameToIdMapping;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.util.analysis.CharTermAttributeSerializer;\nimport com.twitter.search.common.util.analysis.IntTermAttributeSerializer;\nimport com.twitter.search.common.util.analysis.TermPayloadAttributeSerializer;\nimport com.twitter.search.common.util.analysis.TwitterPhotoTokenStream;\nimport com.twitter.search.common.util.spatial.GeoUtil;\nimport com.twitter.search.common.util.text.TokenizerHelper;\nimport com.twitter.search.common.util.text.TweetTokenStreamSerializer;\nimport com.twitter.search.common.util.text.regex.Regex;\nimport com.twitter.search.common.util.url.LinkVisibilityUtils;\nimport com.twitter.search.common.util.url.URLUtils;\n\nimport geo.google.datamodel.GeoAddressAccuracy;\nimport com.twitter.search.common.schema.thriftjava.ThriftDocument;\n\n/**\n * Builder class for building a {@link ThriftDocument}.\n */\npublic final class EarlybirdThriftDocumentBuilder extends ThriftDocumentBuilder {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdThriftDocumentBuilder.class);\n\n  private static final SearchCounter SERIALIZE_FAILURE_COUNT_NONPENGUIN_DEPENDENT =\n      SearchCounter.export(\"tokenstream_serialization_failure_non_penguin_dependent\");\n\n  private static final String HASHTAG_SYMBOL = \"#\";\n  private static final String CASHTAG_SYMBOL = \"$\";\n  private static final String MENTION_SYMBOL = \"@\";\n\n  private static final SearchCounter BCP47_LANGUAGE_TAG_COUNTER =\n      SearchCounter.export(\"bcp47_language_tag\");\n\n  /**\n   * Used to check if a card is video card.\n   *\n   * @see #withSearchCard\n   */\n  private static final String AMPLIFY_CARD_NAME = \"amplify\";\n  private static final String PLAYER_CARD_NAME = \"player\";\n\n  // Extra term indexed for native retweets, to ensure that the \"-rt\" query excludes them.\n  public static final String RETWEET_TERM = \"rt\";\n  public static final String QUESTION_MARK = \"?\";\n\n  private static final Set<NamedEntityInputSourceType> NAMED_ENTITY_URL_SOURCE_TYPES =\n      ImmutableSet.of(\n          NamedEntityInputSourceType.URL_TITLE, NamedEntityInputSourceType.URL_DESCRIPTION);\n\n  private final TokenStreamSerializer intTermAttributeSerializer =\n      new TokenStreamSerializer(ImmutableList.of(\n          new IntTermAttributeSerializer()));\n  private final TokenStreamSerializer photoUrlSerializer =\n      new TokenStreamSerializer(ImmutableList\n          .<TokenStreamSerializer.AttributeSerializer>of(\n              new CharTermAttributeSerializer(), new TermPayloadAttributeSerializer()));\n  private final Schema schema;\n\n  private boolean isSetLatLonCSF = false;\n  private boolean addLatLonCSF = true;\n  private boolean addEncodedTweetFeatures = true;\n\n  @Nonnull\n  private final EarlybirdEncodedFeatures encodedTweetFeatures;\n  @Nullable\n  private final EarlybirdEncodedFeatures extendedEncodedTweetFeatures;\n\n  /**\n   * Default constructor\n   */\n  public EarlybirdThriftDocumentBuilder(\n      @Nonnull EarlybirdEncodedFeatures encodedTweetFeatures,\n      @Nullable EarlybirdEncodedFeatures extendedEncodedTweetFeatures,\n      FieldNameToIdMapping idMapping,\n      Schema schema) {\n    super(idMapping);\n    this.schema = schema;\n    this.encodedTweetFeatures = Preconditions.checkNotNull(encodedTweetFeatures);\n\n    this.extendedEncodedTweetFeatures = extendedEncodedTweetFeatures;\n  }\n\n  /**\n   * Get internal {@link EarlybirdEncodedFeatures}\n   */\n  public EarlybirdEncodedFeatures getEncodedTweetFeatures() {\n    return encodedTweetFeatures;\n  }\n\n  /**\n   * Add skip list entry for the given field.\n   * This adds a term __has_fieldName in the INTERNAL field.\n   */\n  public EarlybirdThriftDocumentBuilder addFacetSkipList(String fieldName) {\n    withStringField(EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n        EarlybirdFieldConstant.getFacetSkipFieldName(fieldName));\n    return this;\n  }\n\n  /**\n   * Add a filter term in the INTERNAL field.\n   */\n  public EarlybirdThriftDocumentBuilder addFilterInternalFieldTerm(String filterName) {\n    withStringField(EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n        EarlybirdThriftDocumentUtil.formatFilter(filterName));\n    return this;\n  }\n\n  /**\n   * Add id field and id csf field.\n   */\n  public EarlybirdThriftDocumentBuilder withID(long id) {\n    withLongField(EarlybirdFieldConstant.ID_FIELD.getFieldName(), id);\n    withLongField(EarlybirdFieldConstant.ID_CSF_FIELD.getFieldName(), id);\n    return this;\n  }\n\n  /**\n   * Add created at field and created at csf field.\n   */\n  public EarlybirdThriftDocumentBuilder withCreatedAt(int createdAt) {\n    withIntField(EarlybirdFieldConstant.CREATED_AT_FIELD.getFieldName(), createdAt);\n    withIntField(EarlybirdFieldConstant.CREATED_AT_CSF_FIELD.getFieldName(), createdAt);\n    return this;\n  }\n\n  /**\n   * Add tweet text field.\n   */\n  public EarlybirdThriftDocumentBuilder withTweetText(\n      String text, byte[] textTokenStream) throws IOException {\n    withTokenStreamField(EarlybirdFieldConstants.EarlybirdFieldConstant.TEXT_FIELD.getFieldName(),\n        text, textTokenStream);\n    return this;\n  }\n\n  public EarlybirdThriftDocumentBuilder withTweetText(String text) throws IOException {\n    withTweetText(text, null);\n    return this;\n  }\n\n  /**\n   * Add a list of cashTags. Like $TWTR.\n   */\n  public EarlybirdThriftDocumentBuilder withStocksFields(List<String> cashTags) {\n    if (isNotEmpty(cashTags)) {\n      addFacetSkipList(EarlybirdFieldConstant.STOCKS_FIELD.getFieldName());\n      for (String cashTag : cashTags) {\n        withStringField(\n            EarlybirdFieldConstant.STOCKS_FIELD.getFieldName(), CASHTAG_SYMBOL + cashTag);\n      }\n    }\n    return this;\n  }\n\n  /**\n   * Add a list of hashtags.\n   */\n  public EarlybirdThriftDocumentBuilder withHashtagsField(List<String> hashtags) {\n    if (isNotEmpty(hashtags)) {\n      int numHashtags = Math.min(\n          hashtags.size(),\n          schema.getFeatureConfigurationById(\n              EarlybirdFieldConstant.NUM_HASHTAGS.getFieldId()).getMaxValue());\n      encodedTweetFeatures.setFeatureValue(EarlybirdFieldConstant.NUM_HASHTAGS, numHashtags);\n      addFacetSkipList(EarlybirdFieldConstant.HASHTAGS_FIELD.getFieldName());\n      for (String hashtag : hashtags) {\n        withStringField(\n            EarlybirdFieldConstant.HASHTAGS_FIELD.getFieldName(), HASHTAG_SYMBOL + hashtag);\n      }\n    }\n    return this;\n  }\n\n  /**\n   * Added a list of mentions.\n   */\n  public EarlybirdThriftDocumentBuilder withMentionsField(List<String> mentions) {\n    if (isNotEmpty(mentions)) {\n      int numMentions = Math.min(\n          mentions.size(),\n          schema.getFeatureConfigurationById(\n              EarlybirdFieldConstant.NUM_HASHTAGS.getFieldId()).getMaxValue());\n      encodedTweetFeatures.setFeatureValue(EarlybirdFieldConstant.NUM_MENTIONS, numMentions);\n      addFacetSkipList(EarlybirdFieldConstant.MENTIONS_FIELD.getFieldName());\n      for (String mention : mentions) {\n        withStringField(\n            EarlybirdFieldConstant.MENTIONS_FIELD.getFieldName(), MENTION_SYMBOL + mention);\n      }\n    }\n    return this;\n  }\n\n  /**\n   * Add a list of Twitter Photo URLs (twimg URLs). These are different from regular URLs, because\n   * we use the TwitterPhotoTokenStream to index them, and we also include the status ID as payload.\n   */\n  public EarlybirdThriftDocumentBuilder withTwimgURLs(\n      List<TwitterPhotoUrl> urls) throws IOException {\n    if (isNotEmpty(urls)) {\n      for (TwitterPhotoUrl photoUrl : urls) {\n        TokenStream ts = new TwitterPhotoTokenStream(photoUrl.getPhotoStatusId(),\n            photoUrl.getMediaUrl());\n        byte[] serializedTs = photoUrlSerializer.serialize(ts);\n        withTokenStreamField(EarlybirdFieldConstant.TWIMG_LINKS_FIELD.getFieldName(),\n            Long.toString(photoUrl.getPhotoStatusId()), serializedTs);\n        addFacetSkipList(EarlybirdFieldConstant.TWIMG_LINKS_FIELD.getFieldName());\n      }\n      encodedTweetFeatures.setFlag(EarlybirdFieldConstant.HAS_IMAGE_URL_FLAG);\n      encodedTweetFeatures.setFlag(EarlybirdFieldConstant.HAS_NATIVE_IMAGE_FLAG);\n    }\n    return this;\n  }\n\n  /**\n   * Add a list of URLs. This also add facet skip list terms for news / images / videos if needed.\n   */\n  public EarlybirdThriftDocumentBuilder withURLs(List<ThriftExpandedUrl> urls) {\n    if (isNotEmpty(urls)) {\n      Set<String> dedupedLinks = Sets.newHashSet();\n\n      for (ThriftExpandedUrl expandedUrl : urls) {\n        if (expandedUrl.isSetOriginalUrl()) {\n          String normalizedOriginalUrl = URLUtils.normalizePath(expandedUrl.getOriginalUrl());\n          dedupedLinks.add(normalizedOriginalUrl);\n        }\n        if (expandedUrl.isSetExpandedUrl()) {\n          dedupedLinks.add(URLUtils.normalizePath(expandedUrl.getExpandedUrl()));\n        }\n\n        if (expandedUrl.isSetCanonicalLastHopUrl()) {\n          String url = URLUtils.normalizePath(expandedUrl.getCanonicalLastHopUrl());\n          dedupedLinks.add(url);\n\n          String facetUrl = URLUtils.normalizeFacetURL(url);\n\n          if (expandedUrl.isSetMediaType()) {\n            switch (expandedUrl.getMediaType()) {\n              case NEWS:\n                withStringField(EarlybirdFieldConstant.NEWS_LINKS_FIELD.getFieldName(), url);\n                addFacetSkipList(EarlybirdFieldConstant.NEWS_LINKS_FIELD.getFieldName());\n                encodedTweetFeatures.setFlag(EarlybirdFieldConstant.HAS_NEWS_URL_FLAG);\n                break;\n              case VIDEO:\n                withStringField(EarlybirdFieldConstant.VIDEO_LINKS_FIELD.getFieldName(), facetUrl);\n                addFacetSkipList(EarlybirdFieldConstant.VIDEO_LINKS_FIELD.getFieldName());\n                encodedTweetFeatures.setFlag(EarlybirdFieldConstant.HAS_VIDEO_URL_FLAG);\n                break;\n              case IMAGE:\n                withStringField(EarlybirdFieldConstant.IMAGE_LINKS_FIELD.getFieldName(), facetUrl);\n                addFacetSkipList(EarlybirdFieldConstant.IMAGE_LINKS_FIELD.getFieldName());\n                encodedTweetFeatures.setFlag(EarlybirdFieldConstant.HAS_IMAGE_URL_FLAG);\n                break;\n              case NATIVE_IMAGE:\n                // Nothing done here. Native images are handled separately.\n                // They are in PhotoUrls instead of expandedUrls.\n                break;\n              case UNKNOWN:\n                break;\n              default:\n                throw new RuntimeException(\"Unknown Media Type: \" + expandedUrl.getMediaType());\n            }\n          }\n\n          if (expandedUrl.isSetLinkCategory()) {\n            withIntField(EarlybirdFieldConstant.LINK_CATEGORY_FIELD.getFieldName(),\n                expandedUrl.getLinkCategory().getValue());\n          }\n        }\n      }\n\n      if (!dedupedLinks.isEmpty()) {\n        encodedTweetFeatures.setFlag(EarlybirdFieldConstant.HAS_LINK_FLAG);\n\n        addFacetSkipList(EarlybirdFieldConstant.LINKS_FIELD.getFieldName());\n\n        for (String linkUrl : dedupedLinks) {\n          withStringField(EarlybirdFieldConstant.LINKS_FIELD.getFieldName(), linkUrl);\n        }\n      }\n\n      encodedTweetFeatures.setFlagValue(\n          EarlybirdFieldConstant.HAS_VISIBLE_LINK_FLAG,\n          LinkVisibilityUtils.hasVisibleLink(urls));\n    }\n\n    return this;\n  }\n\n  /**\n   * Add a list of places. The place are U64 encoded place IDs.\n   */\n  public EarlybirdThriftDocumentBuilder withPlacesField(List<String> places) {\n    if (isNotEmpty(places)) {\n      for (String place : places) {\n        withStringField(EarlybirdFieldConstant.PLACE_FIELD.getFieldName(), place);\n      }\n    }\n    return this;\n  }\n\n  /**\n   * Add tweet text signature field.\n   */\n  public EarlybirdThriftDocumentBuilder withTweetSignature(int signature) {\n    encodedTweetFeatures.setFeatureValue(EarlybirdFieldConstant.TWEET_SIGNATURE, signature);\n    return this;\n  }\n\n  /**\n   * Add geo hash field and internal filter field.\n   */\n  public EarlybirdThriftDocumentBuilder withGeoHash(double lat, double lon, int accuracy) {\n    if (GeoUtil.validateGeoCoordinates(lat, lon)) {\n      withGeoField(\n          EarlybirdFieldConstant.GEO_HASH_FIELD.getFieldName(),\n          lat, lon, accuracy);\n      withLatLonCSF(lat, lon);\n    }\n    return this;\n  }\n\n  public EarlybirdThriftDocumentBuilder withGeoHash(double lat, double lon) {\n    withGeoHash(lat, lon, GeoAddressAccuracy.UNKNOWN_LOCATION.getCode());\n    return this;\n  }\n\n  /**\n   * Add geo location source to the internal field with ThriftGeoLocationSource object.\n   */\n  public EarlybirdThriftDocumentBuilder withGeoLocationSource(\n      ThriftGeoLocationSource geoLocationSource) {\n    if (geoLocationSource != null) {\n      withGeoLocationSource(EarlybirdFieldConstants.formatGeoType(geoLocationSource));\n    }\n    return this;\n  }\n\n  /**\n   * Add geo location source to the internal field.\n   */\n  public EarlybirdThriftDocumentBuilder withGeoLocationSource(String geoLocationSource) {\n    withStringField(EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(), geoLocationSource);\n    return this;\n  }\n\n  /**\n   * Add encoded lat and lon to LatLonCSF field.\n   */\n  public EarlybirdThriftDocumentBuilder withLatLonCSF(double lat, double lon) {\n    isSetLatLonCSF = true;\n    long encodedLatLon = GeoUtil.encodeLatLonIntoInt64((float) lat, (float) lon);\n    withLongField(EarlybirdFieldConstant.LAT_LON_CSF_FIELD.getFieldName(), encodedLatLon);\n    return this;\n  }\n\n  /**\n   * Add from verified account flag to internal field.\n   */\n  public EarlybirdThriftDocumentBuilder withFromVerifiedAccountFlag() {\n    encodedTweetFeatures.setFlag(EarlybirdFieldConstant.FROM_VERIFIED_ACCOUNT_FLAG);\n    addFilterInternalFieldTerm(EarlybirdFieldConstant.VERIFIED_FILTER_TERM);\n    return this;\n  }\n\n  /**\n   * Add from blue-verified account flag to internal field.\n   */\n  public EarlybirdThriftDocumentBuilder withFromBlueVerifiedAccountFlag() {\n    encodedTweetFeatures.setFlag(EarlybirdFieldConstant.FROM_BLUE_VERIFIED_ACCOUNT_FLAG);\n    addFilterInternalFieldTerm(EarlybirdFieldConstant.BLUE_VERIFIED_FILTER_TERM);\n    return this;\n  }\n\n  /**\n   * Add offensive flag to internal field.\n   */\n  public EarlybirdThriftDocumentBuilder withOffensiveFlag() {\n    encodedTweetFeatures.setFlag(EarlybirdFieldConstant.IS_OFFENSIVE_FLAG);\n    withStringField(\n        EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n        EarlybirdFieldConstant.IS_OFFENSIVE);\n    return this;\n  }\n\n  /**\n   * Add user reputation value to encoded feature.\n   */\n  public EarlybirdThriftDocumentBuilder withUserReputation(byte score) {\n    encodedTweetFeatures.setFeatureValue(EarlybirdFieldConstant.USER_REPUTATION, score);\n    return this;\n  }\n\n  /**\n   * This method creates the fields related to document language.\n   * For most languages, their isoLanguageCode and bcp47LanguageTag are the same.\n   * For some languages with variants, these two fields are different.\n   * E.g. for simplified Chinese, their isoLanguageCode is zh, but their bcp47LanguageTag is zh-cn.\n   * <p>\n   * This method adds fields for both the isoLanguageCode and bcp47LanguageTag.\n   */\n  public EarlybirdThriftDocumentBuilder withLanguageCodes(\n      String isoLanguageCode, String bcp47LanguageTag) {\n    if (isoLanguageCode != null) {\n      withISOLanguage(isoLanguageCode);\n    }\n    if (bcp47LanguageTag != null && !bcp47LanguageTag.equals(isoLanguageCode)) {\n      BCP47_LANGUAGE_TAG_COUNTER.increment();\n      withISOLanguage(bcp47LanguageTag);\n    }\n    return this;\n  }\n\n  /**\n   * Adds a String field into the ISO_LANGUAGE_FIELD.\n   */\n  public EarlybirdThriftDocumentBuilder withISOLanguage(String languageString) {\n    withStringField(\n        EarlybirdFieldConstant.ISO_LANGUAGE_FIELD.getFieldName(), languageString.toLowerCase());\n    return this;\n  }\n\n  /**\n   * Add from user ID fields.\n   */\n  public EarlybirdThriftDocumentBuilder withFromUserID(long fromUserId) {\n    withLongField(EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName(), fromUserId);\n    withLongField(EarlybirdFieldConstant.FROM_USER_ID_CSF.getFieldName(), fromUserId);\n    return this;\n  }\n\n  /**\n   * Add from user information fields.\n   */\n  public EarlybirdThriftDocumentBuilder withFromUser(\n      long fromUserId, String fromUser) {\n    withFromUser(fromUserId, fromUser, null);\n    return this;\n  }\n\n  /**\n   * Add from user information fields.\n   */\n  public EarlybirdThriftDocumentBuilder withFromUser(String fromUser) {\n    withFromUser(fromUser, null);\n    return this;\n  }\n\n  /**\n   * Add from user information fields.\n   */\n  public EarlybirdThriftDocumentBuilder withFromUser(\n      String fromUser, String tokenizedFromUser) {\n    withStringField(EarlybirdFieldConstant.FROM_USER_FIELD.getFieldName(), fromUser);\n    withStringField(EarlybirdFieldConstant.TOKENIZED_FROM_USER_FIELD.getFieldName(),\n        isNotBlank(tokenizedFromUser) ? tokenizedFromUser : fromUser);\n    return this;\n  }\n\n  /**\n   * Add from user information fields.\n   */\n  public EarlybirdThriftDocumentBuilder withFromUser(\n      long fromUserId, String fromUser, String tokenizedFromUser) {\n    withFromUserID(fromUserId);\n    withFromUser(fromUser, tokenizedFromUser);\n    return this;\n  }\n\n  /**\n   * Add to user field.\n   */\n  public EarlybirdThriftDocumentBuilder withToUser(\n      String toUser) {\n    withStringField(EarlybirdFieldConstant.TO_USER_FIELD.getFieldName(), toUser);\n    return this;\n  }\n\n  /**\n   * Add escherbird annotation fields.\n   */\n  public EarlybirdThriftDocumentBuilder withAnnotationEntities(List<String> entities) {\n    if (isNotEmpty(entities)) {\n      for (String entity : entities) {\n        withStringField(EarlybirdFieldConstant.ENTITY_ID_FIELD.getFieldName(), entity);\n      }\n    }\n    return this;\n  }\n\n  /**\n   * Add replies to internal field and set is reply flag.\n   */\n  public EarlybirdThriftDocumentBuilder withReplyFlag() {\n    encodedTweetFeatures.setFlag(EarlybirdFieldConstant.IS_REPLY_FLAG);\n    addFilterInternalFieldTerm(EarlybirdFieldConstant.REPLIES_FILTER_TERM);\n    return this;\n  }\n\n  public EarlybirdThriftDocumentBuilder withCameraComposerSourceFlag() {\n    encodedTweetFeatures.setFlag(EarlybirdFieldConstant.COMPOSER_SOURCE_IS_CAMERA_FLAG);\n    return this;\n  }\n\n    /**\n     * Add in reply to user id.\n     * <p>\n     * Notice {@link #withReplyFlag} is not automatically called since retweet a tweet that is\n     * a reply to some other tweet is not considered a reply.\n     * The caller should call {@link #withReplyFlag} separately if this tweet is really a reply tweet.\n     */\n  public EarlybirdThriftDocumentBuilder withInReplyToUserID(long inReplyToUserID) {\n    withLongField(EarlybirdFieldConstant.IN_REPLY_TO_USER_ID_FIELD.getFieldName(), inReplyToUserID);\n    return this;\n  }\n\n  /**\n   * Add reference tweet author id.\n   */\n  public EarlybirdThriftDocumentBuilder withReferenceAuthorID(long referenceAuthorID) {\n    withLongField(EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_CSF.getFieldName(), referenceAuthorID);\n    return this;\n  }\n\n  /**\n   * Add all native retweet related fields/label\n   */\n  @VisibleForTesting\n  public EarlybirdThriftDocumentBuilder withNativeRetweet(final long retweetUserID,\n                                                          final long sharedStatusID) {\n    withLongField(EarlybirdFieldConstant.SHARED_STATUS_ID_CSF.getFieldName(), sharedStatusID);\n\n    withLongField(EarlybirdFieldConstant.RETWEET_SOURCE_TWEET_ID_FIELD.getFieldName(),\n                  sharedStatusID);\n    withLongField(EarlybirdFieldConstant.RETWEET_SOURCE_USER_ID_FIELD.getFieldName(),\n                  retweetUserID);\n    withLongField(EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_CSF.getFieldName(), retweetUserID);\n\n    encodedTweetFeatures.setFlag(EarlybirdFieldConstant.IS_RETWEET_FLAG);\n\n    // Add native retweet label to the internal field.\n    addFilterInternalFieldTerm(EarlybirdFieldConstant.NATIVE_RETWEETS_FILTER_TERM);\n    withStringField(EarlybirdFieldConstant.TEXT_FIELD.getFieldName(), RETWEET_TERM);\n    return this;\n  }\n\n  /**\n   * Add quoted tweet id and user id.\n   */\n  @VisibleForTesting\n  public EarlybirdThriftDocumentBuilder withQuote(\n      final long quotedStatusId, final long quotedUserId) {\n    withLongField(EarlybirdFieldConstant.QUOTED_TWEET_ID_FIELD.getFieldName(), quotedStatusId);\n    withLongField(EarlybirdFieldConstant.QUOTED_USER_ID_FIELD.getFieldName(), quotedUserId);\n\n    withLongField(EarlybirdFieldConstant.QUOTED_TWEET_ID_CSF.getFieldName(), quotedStatusId);\n    withLongField(EarlybirdFieldConstant.QUOTED_USER_ID_CSF.getFieldName(), quotedUserId);\n\n    encodedTweetFeatures.setFlag(EarlybirdFieldConstant.HAS_QUOTE_FLAG);\n\n    // Add quote label to the internal field.\n    addFilterInternalFieldTerm(EarlybirdFieldConstant.QUOTE_FILTER_TERM);\n    return this;\n  }\n\n  /**\n   * Add resolved links text field.\n   */\n  public EarlybirdThriftDocumentBuilder withResolvedLinksText(String linksText) {\n    withStringField(EarlybirdFieldConstant.RESOLVED_LINKS_TEXT_FIELD.getFieldName(), linksText);\n    return this;\n  }\n\n  /**\n   * Add source field.\n   */\n  public EarlybirdThriftDocumentBuilder withSource(String source) {\n    withStringField(EarlybirdFieldConstant.SOURCE_FIELD.getFieldName(), source);\n    return this;\n  }\n\n  /**\n   * Add normalized source field.\n   */\n  public EarlybirdThriftDocumentBuilder withNormalizedSource(String normalizedSource) {\n    withStringField(\n        EarlybirdFieldConstant.NORMALIZED_SOURCE_FIELD.getFieldName(), normalizedSource);\n    return this;\n  }\n\n  /**\n   * Add positive smiley to internal field.\n   */\n  public EarlybirdThriftDocumentBuilder withPositiveSmiley() {\n    withStringField(\n        EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n        EarlybirdFieldConstant.HAS_POSITIVE_SMILEY);\n    return this;\n  }\n\n  /**\n   * Add negative smiley to internal field.\n   */\n  public EarlybirdThriftDocumentBuilder withNegativeSmiley() {\n    withStringField(\n        EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n        EarlybirdFieldConstant.HAS_NEGATIVE_SMILEY);\n    return this;\n  }\n\n  /**\n   * Add question mark label to a text field.\n   */\n  public EarlybirdThriftDocumentBuilder withQuestionMark() {\n    withStringField(EarlybirdFieldConstant.TEXT_FIELD.getFieldName(), QUESTION_MARK);\n    return this;\n  }\n\n  /**\n   * Add card related fields.\n   */\n  public EarlybirdThriftDocumentBuilder withSearchCard(\n      String name,\n      String domain,\n      String title, byte[] serializedTitleStream,\n      String description, byte[] serializedDescriptionStream,\n      String lang) {\n    if (isNotBlank(title)) {\n      withTokenStreamField(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_TITLE_FIELD.getFieldName(),\n          title, serializedTitleStream);\n    }\n\n    if (isNotBlank(description)) {\n      withTokenStreamField(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_DESCRIPTION_FIELD.getFieldName(),\n          description, serializedDescriptionStream);\n    }\n\n    if (isNotBlank(lang)) {\n      withStringField(EarlybirdFieldConstant.CARD_LANG.getFieldName(), lang);\n    }\n\n    if (isNotBlank(domain)) {\n      withStringField(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_DOMAIN_FIELD.getFieldName(), domain);\n    }\n\n    if (isNotBlank(name)) {\n      withStringField(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_NAME_FIELD.getFieldName(), name);\n      withIntField(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.CARD_TYPE_CSF_FIELD.getFieldName(),\n          SearchCardType.cardTypeFromStringName(name).getByteValue());\n    }\n\n    if (AMPLIFY_CARD_NAME.equalsIgnoreCase(name)\n        || PLAYER_CARD_NAME.equalsIgnoreCase(name)) {\n      // Add into \"internal\" field so that this tweet is returned by filter:videos.\n      addFacetSkipList(\n          EarlybirdFieldConstants.EarlybirdFieldConstant.VIDEO_LINKS_FIELD.getFieldName());\n    }\n\n    return this;\n  }\n\n  public EarlybirdThriftDocumentBuilder withNormalizedMinEngagementField(\n      String fieldName, int normalizedNumEngagements) throws IOException {\n    EarlybirdThriftDocumentUtil.addNormalizedMinEngagementField(doc, fieldName,\n        normalizedNumEngagements);\n    return this;\n  }\n\n  /**\n   * Add named entity with given canonical name and type to document.\n   */\n  public EarlybirdThriftDocumentBuilder withNamedEntity(NamedEntity namedEntity) {\n    if (namedEntity.getContexts() == null) {\n      // In this unlikely case, we don't have any context for named entity type or source,\n      // so we can't properly index it in any of our fields. We'll just skip it in this case.\n      return this;\n    }\n\n    // Keep track of the fields we've applied in the builder already, to ensure we only index\n    // each term (field/value pair) once\n    Set<Pair<EarlybirdFieldConstant, String>> fieldsApplied = new HashSet<>();\n    for (NamedEntityContext context : namedEntity.getContexts()) {\n      if (context.isSetInput_source()\n          && NAMED_ENTITY_URL_SOURCE_TYPES.contains(context.getInput_source().getSource_type())) {\n        // If the source is one of the URL* types, add the named entity to the \"from_url\" fields,\n        // ensuring we add it only once\n        addNamedEntityFields(\n            fieldsApplied,\n            EarlybirdFieldConstant.NAMED_ENTITY_FROM_URL_FIELD,\n            EarlybirdFieldConstant.NAMED_ENTITY_WITH_TYPE_FROM_URL_FIELD,\n            namedEntity.getCanonical_name(),\n            context);\n      } else {\n        addNamedEntityFields(\n            fieldsApplied,\n            EarlybirdFieldConstant.NAMED_ENTITY_FROM_TEXT_FIELD,\n            EarlybirdFieldConstant.NAMED_ENTITY_WITH_TYPE_FROM_TEXT_FIELD,\n            namedEntity.getCanonical_name(),\n            context);\n      }\n    }\n\n    return this;\n  }\n\n  /**\n   * Add space id fields.\n   */\n  public EarlybirdThriftDocumentBuilder withSpaceIdFields(Set<String> spaceIds) {\n    if (!spaceIds.isEmpty()) {\n      addFacetSkipList(EarlybirdFieldConstant.SPACE_ID_FIELD.getFieldName());\n      for (String spaceId : spaceIds) {\n        withStringField(EarlybirdFieldConstant.SPACE_ID_FIELD.getFieldName(), spaceId);\n      }\n    }\n    return this;\n  }\n\n  /**\n   * Add directed at user.\n   */\n  @VisibleForTesting\n  public EarlybirdThriftDocumentBuilder withDirectedAtUser(final long directedAtUserId) {\n    withLongField(EarlybirdFieldConstant.DIRECTED_AT_USER_ID_FIELD.getFieldName(),\n        directedAtUserId);\n\n    withLongField(EarlybirdFieldConstant.DIRECTED_AT_USER_ID_CSF.getFieldName(), directedAtUserId);\n\n    return this;\n  }\n\n  /**\n   * Add a white space tokenized screen name field.\n   *\n   * Example:\n   *  screenName - \"super_hero\"\n   *  tokenized version - \"super hero\"\n   */\n  public EarlybirdThriftDocumentBuilder withWhiteSpaceTokenizedScreenNameField(\n      String fieldName,\n      String normalizedScreenName) {\n    String whiteSpaceTokenizableScreenName = StringUtils.join(\n        normalizedScreenName.split(Regex.HASHTAG_USERNAME_PUNCTUATION_REGEX), \" \");\n    withStringField(fieldName, whiteSpaceTokenizableScreenName);\n    return this;\n  }\n\n  /**\n   * Add a camel case tokenized screen name field.\n   */\n  public EarlybirdThriftDocumentBuilder withCamelCaseTokenizedScreenNameField(\n      String fieldName,\n      String screenName,\n      String normalizedScreenName,\n      TokenStream screenNameTokenStream) {\n\n    // this normalized text is consistent to how the tokenized stream is created from\n    // TokenizerHelper.getNormalizedCamelcaseTokenStream - ie. just lowercasing.\n    String camelCaseTokenizedScreenNameText =\n        TokenizerHelper.getNormalizedCamelcaseTokenStreamText(screenName);\n    try {\n      // Reset the token stream in case it has been read before.\n      screenNameTokenStream.reset();\n      byte[] camelCaseTokenizedScreenName =\n          TweetTokenStreamSerializer.getTweetTokenStreamSerializer()\n              .serialize(screenNameTokenStream);\n\n      withTokenStreamField(\n          fieldName,\n          camelCaseTokenizedScreenNameText.isEmpty()\n              ? normalizedScreenName : camelCaseTokenizedScreenNameText,\n          camelCaseTokenizedScreenName);\n    } catch (IOException e) {\n      LOG.error(\"TwitterTokenStream serialization error! Could not serialize: \" + screenName);\n      SERIALIZE_FAILURE_COUNT_NONPENGUIN_DEPENDENT.increment();\n    }\n    return this;\n  }\n\n  private void addNamedEntityFields(\n      Set<Pair<EarlybirdFieldConstant, String>> fieldsApplied,\n      EarlybirdFieldConstant nameOnlyField,\n      EarlybirdFieldConstant nameWithTypeField,\n      String name,\n      NamedEntityContext context) {\n    withOneTimeStringField(fieldsApplied, nameOnlyField, name, false);\n    if (context.isSetEntity_type()) {\n      withOneTimeStringField(fieldsApplied, nameWithTypeField,\n          formatNamedEntityString(name, context.getEntity_type()), true);\n    }\n  }\n\n  private void withOneTimeStringField(\n      Set<Pair<EarlybirdFieldConstant, String>> fieldsApplied, EarlybirdFieldConstant field,\n      String value, boolean addToFacets) {\n    Pair<EarlybirdFieldConstant, String> fieldValuePair = new Pair<>(field, value);\n    if (!fieldsApplied.contains(fieldValuePair)) {\n      if (addToFacets) {\n        addFacetSkipList(field.getFieldName());\n      }\n      withStringField(field.getFieldName(), value);\n      fieldsApplied.add(fieldValuePair);\n    }\n  }\n\n  private String formatNamedEntityString(String name, WholeEntityType type) {\n    return String.format(\"%s:%s\", name, type).toLowerCase();\n  }\n\n  /**\n   * Set whether set LAT_LON_CSF_FIELD or not before build\n   * if LAT_LON_CSF_FIELD is not set deliberately.\n   *\n   * @see #prepareToBuild()\n   */\n  public EarlybirdThriftDocumentBuilder setAddLatLonCSF(boolean isSet) {\n    addLatLonCSF = isSet;\n    return this;\n  }\n\n  /**\n   * Set if add encoded tweet feature field in the end.\n   *\n   * @see #prepareToBuild()\n   */\n  public EarlybirdThriftDocumentBuilder setAddEncodedTweetFeatures(boolean isSet) {\n    addEncodedTweetFeatures = isSet;\n    return this;\n  }\n\n  @Override\n  protected void prepareToBuild() {\n    if (!isSetLatLonCSF && addLatLonCSF) {\n      // In lucene archives, this CSF is needed regardless of whether geoLocation is set.\n      withLatLonCSF(GeoUtil.ILLEGAL_LATLON, GeoUtil.ILLEGAL_LATLON);\n    }\n\n    if (addEncodedTweetFeatures) {\n      // Add encoded_tweet_features before building the document.\n      withBytesField(\n          EarlybirdFieldConstant.ENCODED_TWEET_FEATURES_FIELD.getFieldName(),\n          EarlybirdEncodedFeaturesUtil.toBytesForThriftDocument(encodedTweetFeatures));\n    }\n\n    if (extendedEncodedTweetFeatures != null) {\n      // Add extended_encoded_tweet_features before building the document.\n      withBytesField(\n          EarlybirdFieldConstant.EXTENDED_ENCODED_TWEET_FEATURES_FIELD.getFieldName(),\n          EarlybirdEncodedFeaturesUtil.toBytesForThriftDocument(extendedEncodedTweetFeatures));\n    }\n  }\n\n  private static boolean isNotBlank(String value) {\n    return value != null && !value.isEmpty();\n  }\n\n  private static boolean isNotEmpty(List<?> value) {\n    return value != null && !value.isEmpty();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/earlybird/EarlybirdThriftDocumentUtil.java",
    "content": "package com.twitter.search.common.schema.earlybird;\n\nimport java.io.IOException;\nimport java.util.Iterator;\nimport java.util.List;\n\nimport com.google.common.collect.ImmutableList;\n\nimport com.twitter.common.text.util.TokenStreamSerializer;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.ThriftDocumentUtil;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.schema.thriftjava.ThriftDocument;\nimport com.twitter.search.common.schema.thriftjava.ThriftField;\nimport com.twitter.search.common.schema.thriftjava.ThriftFieldData;\nimport com.twitter.search.common.util.analysis.IntTermAttributeSerializer;\nimport com.twitter.search.common.util.analysis.TwitterNormalizedMinEngagementTokenStream;\n\n/**\n * Utility APIs for ThriftDocument used in Earlybird.\n */\npublic final class EarlybirdThriftDocumentUtil {\n  private static final EarlybirdFieldConstants ID_MAPPING = new EarlybirdFieldConstants();\n\n  private static final String FILTER_FORMAT_STRING = \"__filter_%s\";\n\n  /**\n   * Used to check whether a thrift document has filter nullcast internal field set.\n   * @see #isNullcastFilterSet(ThriftDocument)\n   */\n  private static final String NULLCAST_FILTER_TERM =\n      formatFilter(EarlybirdFieldConstant.NULLCAST_FILTER_TERM);\n\n  private static final String SELF_THREAD_FILTER_TERM =\n      formatFilter(EarlybirdFieldConstant.SELF_THREAD_FILTER_TERM);\n\n  private static final String DIRECTED_AT_FILTER_TERM =\n      formatFilter(EarlybirdFieldConstant.DIRECTED_AT_FILTER_TERM);\n\n  private EarlybirdThriftDocumentUtil() {\n    // Cannot instantiate.\n  }\n\n  /**\n   * Formats a regular, simple filter term. The 'filter' argument should correspond to a constant\n   * from the Operator class, matching the operand (filter:links -> \"links\").\n   */\n  public static final String formatFilter(String filter) {\n    return String.format(FILTER_FORMAT_STRING, filter);\n  }\n\n  /**\n   * Get status id.\n   */\n  public static long getID(ThriftDocument document) {\n    return ThriftDocumentUtil.getLongValue(\n        document, EarlybirdFieldConstant.ID_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get Card name.\n   */\n  public static String getCardName(ThriftDocument document) {\n    return ThriftDocumentUtil.getStringValue(\n        document, EarlybirdFieldConstant.CARD_NAME_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get Card language.\n   */\n  public static String getCardLang(ThriftDocument document) {\n    return ThriftDocumentUtil.getStringValue(\n        document, EarlybirdFieldConstant.CARD_LANG.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get Card language CSF.\n   *\n   * card language CSF is represented internally as an integer ID for a ThriftLanguage.\n   */\n  public static int getCardLangCSF(ThriftDocument document) {\n    return ThriftDocumentUtil.getIntValue(\n        document, EarlybirdFieldConstant.CARD_LANG_CSF.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get quoted tweet id.\n   */\n  public static long getQuotedTweetID(ThriftDocument document) {\n    return ThriftDocumentUtil.getLongValue(\n        document, EarlybirdFieldConstant.QUOTED_TWEET_ID_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get quoted tweet user id.\n   */\n  public static long getQuotedUserID(ThriftDocument document) {\n    return ThriftDocumentUtil.getLongValue(\n        document, EarlybirdFieldConstant.QUOTED_USER_ID_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get directed at user id.\n   */\n  public static long getDirectedAtUserId(ThriftDocument document) {\n    return ThriftDocumentUtil.getLongValue(\n        document, EarlybirdFieldConstant.DIRECTED_AT_USER_ID_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get directed at user id CSF.\n   */\n  public static long getDirectedAtUserIdCSF(ThriftDocument document) {\n    return ThriftDocumentUtil.getLongValue(\n        document, EarlybirdFieldConstant.DIRECTED_AT_USER_ID_CSF.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get reference author id CSF.\n   */\n  public static long getReferenceAuthorIdCSF(ThriftDocument document) {\n    return ThriftDocumentUtil.getLongValue(\n        document, EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_CSF.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get links.\n   */\n  public static List<String> getLinks(ThriftDocument document) {\n    return getStringValues(document, EarlybirdFieldConstant.LINKS_FIELD);\n  }\n\n  /**\n   * Get created at timestamp in sec.\n   */\n  public static int getCreatedAtSec(ThriftDocument document) {\n    return ThriftDocumentUtil.getIntValue(\n        document, EarlybirdFieldConstant.CREATED_AT_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get created at timestamp in ms.\n   */\n  public static long getCreatedAtMs(ThriftDocument document) {\n    long createdAtSec = (long) getCreatedAtSec(document);\n    return createdAtSec * 1000L;\n  }\n\n  /**\n   * Get from user id.\n   */\n  public static long getFromUserID(ThriftDocument document) {\n    return ThriftDocumentUtil.getLongValue(\n        document, EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get from user.\n   */\n  public static String getFromUser(ThriftDocument document) {\n    return ThriftDocumentUtil.getStringValue(\n        document, EarlybirdFieldConstant.FROM_USER_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get tokenized from user display name.\n   */\n  public static String getFromUserDisplayName(ThriftDocument document) {\n    return ThriftDocumentUtil.getStringValue(\n        document, EarlybirdFieldConstant.TOKENIZED_USER_NAME_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get tokenized from user.\n   */\n  public static String getTokenizedFromUser(ThriftDocument document) {\n    return ThriftDocumentUtil.getStringValue(\n        document, EarlybirdFieldConstant.TOKENIZED_FROM_USER_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get resolved links text.\n   */\n  public static String getResolvedLinksText(ThriftDocument document) {\n    return ThriftDocumentUtil.getStringValue(\n        document, EarlybirdFieldConstant.RESOLVED_LINKS_TEXT_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * Get iso language code.\n   */\n  public static List<String> getISOLanguage(ThriftDocument document) {\n    return ThriftDocumentUtil.getStringValues(\n        document, EarlybirdFieldConstant.ISO_LANGUAGE_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  /**\n   * First remove the old timestamp if they exist.\n   * Then add the created at and created at csf fields to the given thrift document.\n   */\n  public static void replaceCreatedAtAndCreatedAtCSF(ThriftDocument document, int value) {\n    removeField(document, EarlybirdFieldConstant.CREATED_AT_FIELD);\n    removeField(document, EarlybirdFieldConstant.CREATED_AT_CSF_FIELD);\n\n    addIntField(document, EarlybirdFieldConstant.CREATED_AT_FIELD, value);\n    addIntField(document, EarlybirdFieldConstant.CREATED_AT_CSF_FIELD, value);\n  }\n\n  /**\n   * Add the given int value as the given field into the given document.\n   */\n  public static ThriftDocument addIntField(\n      ThriftDocument document, EarlybirdFieldConstant fieldConstant, int value) {\n    ThriftFieldData fieldData = new ThriftFieldData().setIntValue(value);\n    ThriftField field =\n        new ThriftField().setFieldConfigId(fieldConstant.getFieldId()).setFieldData(fieldData);\n    document.addToFields(field);\n    return document;\n  }\n\n  private static EarlybirdFieldConstant getFeatureField(EarlybirdFieldConstant field) {\n    if (field.getFieldName().startsWith(\n        EarlybirdFieldConstant.ENCODED_TWEET_FEATURES_FIELD.getFieldName())) {\n      return EarlybirdFieldConstant.ENCODED_TWEET_FEATURES_FIELD;\n    } else if (field.getFieldName().startsWith(\n        EarlybirdFieldConstant.EXTENDED_ENCODED_TWEET_FEATURES_FIELD.getFieldName())) {\n      return EarlybirdFieldConstant.EXTENDED_ENCODED_TWEET_FEATURES_FIELD;\n    } else {\n      throw new IllegalArgumentException(\"Not a feature field: \" + field);\n    }\n  }\n\n  /**\n   * Get the feature value of a field.\n   */\n  public static int getFeatureValue(\n      ImmutableSchemaInterface schema,\n      ThriftDocument document,\n      EarlybirdFieldConstant field) {\n\n    EarlybirdFieldConstant featureField = getFeatureField(field);\n\n    byte[] encodedFeaturesBytes =\n        ThriftDocumentUtil.getBytesValue(document, featureField.getFieldName(), ID_MAPPING);\n\n    if (encodedFeaturesBytes == null) {\n      // Treat the feature value as 0 if there is no encoded feature field.\n      return 0;\n    } else {\n      EarlybirdEncodedFeatures encodedFeatures = EarlybirdEncodedFeaturesUtil.fromBytes(\n          schema, featureField, encodedFeaturesBytes, 0);\n      return encodedFeatures.getFeatureValue(field);\n    }\n  }\n\n  /**\n   * Check whether the feature flag is set.\n   */\n  public static boolean isFeatureBitSet(\n      ImmutableSchemaInterface schema,\n      ThriftDocument document,\n      EarlybirdFieldConstant field) {\n\n    EarlybirdFieldConstant featureField = getFeatureField(field);\n\n    byte[] encodedFeaturesBytes =\n        ThriftDocumentUtil.getBytesValue(document, featureField.getFieldName(), ID_MAPPING);\n\n    if (encodedFeaturesBytes == null) {\n      // Treat the bit as not set if there is no encoded feature field.\n      return false;\n    } else {\n      EarlybirdEncodedFeatures encodedFeatures = EarlybirdEncodedFeaturesUtil.fromBytes(\n          schema, featureField, encodedFeaturesBytes, 0);\n      return encodedFeatures.isFlagSet(field);\n    }\n  }\n\n  /**\n   * Check whether nullcast flag is set in the encoded features field.\n   */\n  public static boolean isNullcastBitSet(ImmutableSchemaInterface schema, ThriftDocument document) {\n    return isFeatureBitSet(schema, document, EarlybirdFieldConstant.IS_NULLCAST_FLAG);\n  }\n\n  /**\n   * Remove all fields with the given field constant in a document.\n   */\n  public static void removeField(ThriftDocument document, EarlybirdFieldConstant fieldConstant) {\n    List<ThriftField> fields = document.getFields();\n    if (fields != null) {\n      Iterator<ThriftField> fieldsIterator = fields.iterator();\n      while (fieldsIterator.hasNext()) {\n        if (fieldsIterator.next().getFieldConfigId() == fieldConstant.getFieldId()) {\n          fieldsIterator.remove();\n        }\n      }\n    }\n  }\n\n  /**\n   * Remove a string field with given fieldConstant and value.\n   */\n  public static void removeStringField(\n      ThriftDocument document, EarlybirdFieldConstant fieldConstant, String value) {\n    List<ThriftField> fields = document.getFields();\n    if (fields != null) {\n      for (ThriftField field : fields) {\n        if (field.getFieldConfigId() == fieldConstant.getFieldId()\n            && field.getFieldData().getStringValue().equals(value)) {\n          fields.remove(field);\n          return;\n        }\n      }\n    }\n  }\n\n  /**\n   * Adds a new TokenStream field for each engagement counter if normalizedNumEngagements >= 1.\n   */\n  public static void addNormalizedMinEngagementField(\n      ThriftDocument doc,\n      String fieldName,\n      int normalizedNumEngagements) throws IOException {\n    if (normalizedNumEngagements < 1) {\n      return;\n    }\n    TokenStreamSerializer serializer =\n        new TokenStreamSerializer(ImmutableList.of(new IntTermAttributeSerializer()));\n    TwitterNormalizedMinEngagementTokenStream stream = new\n        TwitterNormalizedMinEngagementTokenStream(normalizedNumEngagements);\n    byte[] serializedStream = serializer.serialize(stream);\n    ThriftFieldData fieldData = new ThriftFieldData().setTokenStreamValue(serializedStream);\n    ThriftField field = new ThriftField().setFieldConfigId(ID_MAPPING.getFieldID(fieldName))\n        .setFieldData(fieldData);\n    doc.addToFields(field);\n  }\n\n  public static List<String> getStringValues(\n      ThriftDocument document, EarlybirdFieldConstant field) {\n    return ThriftDocumentUtil.getStringValues(document, field.getFieldName(), ID_MAPPING);\n  }\n\n  public static boolean isNullcastFilterSet(ThriftDocument document) {\n    return isFilterSet(document, NULLCAST_FILTER_TERM);\n  }\n\n  public static boolean isSelfThreadFilterSet(ThriftDocument document) {\n    return isFilterSet(document, SELF_THREAD_FILTER_TERM);\n  }\n\n  public static String getSelfThreadFilterTerm() {\n    return SELF_THREAD_FILTER_TERM;\n  }\n\n  public static String getDirectedAtFilterTerm() {\n    return DIRECTED_AT_FILTER_TERM;\n  }\n\n  public static boolean isDirectedAtFilterSet(ThriftDocument document) {\n    return isFilterSet(document, DIRECTED_AT_FILTER_TERM);\n  }\n\n  /**\n   * Check whether given filter is set in the internal field.\n   */\n  private static boolean isFilterSet(ThriftDocument document, String filter) {\n    List<String> terms = ThriftDocumentUtil.getStringValues(\n        document, EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(), ID_MAPPING);\n    for (String term : terms) {\n      if (filter.equals(term)) {\n        return true;\n      }\n    }\n    return false;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/schema/earlybird/FlushVersion.java",
    "content": "package com.twitter.search.common.schema.earlybird;\n\nimport javax.annotation.Nullable;\n\nimport com.twitter.search.common.config.Config;\n\npublic enum FlushVersion {\n  /* =======================================================\n   * Versions\n   * ======================================================= */\n  VERSION_0(\"Initial version of partition flushing.\"),\n  VERSION_1(\"Added timestamps and corresponding mapper to SegmentData.\"),\n  VERSION_2(\"Add column stride fields.\"),\n  VERSION_3(\"Change facet field configuration.\"),\n  VERSION_4(\"Add per term offensive counters to parallel posting arrays.\"),\n  VERSION_5(\"Add native photo facet.\"),\n  VERSION_6(\"Add UserFeature column stride field\"),\n  VERSION_7(\"Index segment optimizations; new facet data structures.\"),\n  VERSION_8(\"Store statuses in memory in Earlybird.\"),\n  VERSION_9(\"Index from_user_ids into a searchable field.\"),\n  VERSION_10(\"Change from_user_id dictionary from fst to mphf\"),\n  VERSION_11(\"Write image and video facet in separate lucene field.\"),\n  VERSION_12(\"Add retweeted status ID to the sparse CSF.\"),\n  VERSION_13(\"Add isOffensive field for profanity filter.\"),\n  VERSION_14(\"Fix features column stride field corruption.\"),\n  VERSION_15(\"Upgrade Lucene version, which has a different FST serialization format.\"),\n  VERSION_16(\"Remove maxDoc in favor of lastDocID\"),\n  VERSION_17(\"Added partition and timeslice identifiers to SegmentData.\"),\n  VERSION_18(\"Per-term payloads\"),\n  VERSION_19(\"Multiple per-doc payload fields\"),\n  VERSION_20(\"Unify and fix hash codes\"),\n  VERSION_21(\"Super awesome new flexible realtime posting list format.\"),\n  VERSION_22(\"Added new geo implementation.\"),\n  VERSION_23(\"Upgrade to Lucene 4.0.0 Final\"),\n  VERSION_24(\"Added tweet topic ids.\"),\n  VERSION_25(\"Turn on skip list for mention facet.\"),\n  VERSION_26(\"Added new EncodedTweetFeaturesColumnStrideField.\"),\n  VERSION_27(\"Topic ids facet field.\"),\n  VERSION_28(\"From-user discover stories skiplist field.\"),\n  VERSION_29(\"Move tokenized screen name to the new username field\"),\n  VERSION_30(\"Enable HF term pairs index.\"),\n  VERSION_31(\"Remove reverse doc ids.\"),\n  VERSION_32(\"Switch shared status id CSF to non-sparse long CSF index.\"),\n  VERSION_33(\"New skip lists for optimized high df posting lists.\"),\n  VERSION_34(\"Store tweet signature in EarlybirdEncodedFeatures.\"),\n  VERSION_35(\"Don't store shared status id csf in archive indexes.\"),\n  VERSION_36(\"Don't store norms.\"),\n  VERSION_37(\"64 bit user ids.\"),\n  VERSION_38(\"Index links in archive.\"),\n  VERSION_39(\"Fix pic.twitter.com image link handling not setting the internal field correctly.\"),\n  VERSION_40(\"Fix all archive tweets being marked as replies.\"),\n  VERSION_41(\"Avoid flushing event_ids field; event clusters are applied as updates.\"),\n  VERSION_42(\"No position fields refactoring; made a few fields to not use position.\"),\n  VERSION_43(\"Index private geo coordinates\"),\n  VERSION_44(\"Materialize last doc id in HighDFCompressedPostinglists\", true),\n  VERSION_45(\"Removing from_user_id facets support\", true),\n  VERSION_46(\"Guard against badly out of order tweets in the search archive.\", true),\n  VERSION_47(\"Added card title and description fields.\", true),\n  VERSION_48(\"Added card type CSF.\", true),\n  VERSION_49(\"Lucene 4.4 upgrade\", true),\n  VERSION_50(\"Put mem-archive back on non-lucene optimized indexes\", true),\n  VERSION_51(\"Force index rebuild to fix blank text field. See SEARCH-2505.\", true),\n  VERSION_52(\"Refactoring of docValues/CSF.\", true),\n  VERSION_53(\"Remove SegmentData.Configuration\", true),\n  VERSION_54(\"Fix bad indices caused by SEARCH-2723.\", true),\n  VERSION_55(\"Fixed non-deterministic facetIds across restarts. SEARCH-2815.\", true),\n  VERSION_56(\"Flush FacetIDMap.\", true),\n  VERSION_57(\"Remove LatLonMapper and use standard DocValues instead.\", true),\n  VERSION_58(\"Longterm Attribute Optimization.\", true),\n  VERSION_59(\"Renamed archive segment names. Current segment is no longer mutable.\", true),\n  // Flush version 60 and 59 have the same format.\n  // Flush version is increased to trigger a rebuild, because we noticed incomplete segments.\n  // More details can be found on SEARCH-3664\n  VERSION_60(\"Flush version change to trigger segment rebuild.\", true),\n  VERSION_61(\"Adding back from_user_id\", true),\n  VERSION_62(\"Add retweet facet.\", true),\n  VERSION_63(\"Switch to new index API in com.twitter.search.core.earlybird.\", true),\n  VERSION_64(\"Sort merge archive day and part-* data. SEARCH-4692.\", true),\n  VERSION_65(\"Fix ID_FIELD and CREATED_AT_FIELD sort order. SEARCH-4004 SEARCH-912 \", true),\n  VERSION_66(\"Rebuild data for 1/5/2015. Data on HDFS fixed as part of SEARCH-5347.\", true),\n  VERSION_67(\"Upgrade to Lucene 4.10.3.\", true),\n  VERSION_68(\"Switching to Penguin v4\", true),\n  VERSION_69(\"Fix 16% archive segments: SEARCH-6073\", true),\n  VERSION_70(\"Switching to Penguin v4 for full archive cluster. SEARCH-5302\", true),\n  VERSION_71(\"Switching to Penguin v4 for ssd archive cluster.\", true),\n  VERSION_72(\"Added Escherbird annotations for full archive.\", true),\n  VERSION_73(\"Lucene 5.2.1 upgrade.\", true, 0),\n  VERSION_74(\"Hanndle geo scurbbed data and archive geo index accuracy\", true, 0),\n  VERSION_75(\"Delete from_user_id_stories from indices\", true, 0),\n  VERSION_76(\"Allow multiple index extensions.\", true, 0),\n  VERSION_77(\"Removed EarlybirdCodec\", true, 0),\n  // minor version 2: added embedded tweet features\n  // minor version 3: change embedded tweet features to INC_ONLY\n  VERSION_78(\"Added 80 bytes of extended features\", true, 3),\n  // minor version 1: SEARCH-8564 - Reference Tweet Author ID, using\n  //                  EXTENDED_TEST_FEATURE_UNUSED_BITS_2 and EXTENDED_TEST_FEATURE_UNUSED_BITS_3\n  VERSION_79(\"Renamed UNUSED_BIT to HAS_VISIBLE_LINK\", true, 1),\n  // minor version 2: SEARCH-8564 / http://go/rb/770373\n  //                  Made REFERENCE_AUTHOR_ID_LEAST_SIGNIFICANT_INT and\n  //                  REFERENCE_AUTHOR_ID_MOST_SIGNIFICANT_INT immutable field\n  VERSION_80(\"Facet for links: SEARCH-8331\", true, 2),\n  // minor version 1: added video view count\n  VERSION_81(\"Adding LowDF posting list with packed ints\", true, 1),\n  VERSION_82(\"Enabling HighDF posting list with packed ints\", true, 0),\n  // minor version 1: SEARCH-9379 - Added bitset for nullcast tweets\n  // minor version 2: SEARCH-8765 - Added visible token ratio\n  VERSION_83(\"Add bits in encoded features for media type flags. SEARCH-9131\", true, 2),\n  VERSION_84(\"Enable archive rebuild for __has_links field. SEARCH-9635\", true, 0),\n  // minor version 1: SEARCHQUAL-8130, add engagement v2\n  VERSION_85(\"New archive build gen for missing geo data. SEARCH-9894\", true, 1),\n  VERSION_86(\"Added new fields to the index\", true, 0),\n  // During this rebuild both the statuses and the engagement counts were regenerated.\n  // minor version 1: added quote_count\n  VERSION_87(\"Periodic archive full rebuild. SEARCH-9423\", true, 1),\n  // minor version 1: make new tokenized user name/handle fields textSearchable\n  //                  (see go/rb/847134/)\n  // minor version 2: added has_quote\n  VERSION_88(\"Fixing missing day in the full archive index. SEARCH-11233\", true, 2),\n  VERSION_89(\"Index and store conversation ids.\", true, 0),\n  VERSION_90(\"Fixing inconsistent days in the full archive index. SEARCH-11744\", true, 0),\n  VERSION_91(\"Making in_reply_to_user_id field use MPH. SEARCH-10836\", true, 0),\n  VERSION_92(\"Allow searches by any field. SEARCH-11251\", true, 0),\n  // During this rebuild we regenerated engagement counts and merged the annotations in the\n  // aggregate job.\n  VERSION_93(\"Periodic archive full rebuild. SEARCH-11076\", true, 0),\n  // minor version 1: add ThriftCSFViewSettings.outputCSFType\n  VERSION_94(\"Indexing a bunch of geo fields. SEARCH-10283\", true, 1),\n  VERSION_95(\"Removing topic ID fields. SEARCH-8616\", true, 0),\n    // minor version 1: add ThriftCSFViewSettings.normalizationType\n  VERSION_96(\"Enabling conversation ID for all clusters. SEARCH-11989\", true, 1),\n  // minor version 1: set several feature configuration to be correct double type\n  // minor version 2: set some more feature configuration to be correct double type\n  // minor version 3: add safety labels SEARCHQUAL-9561\n  // minor version 4: add weighted engagement counts SEARCHQUAL-9574\n  // minor version 5: add Dopamine non personalized score SEARCHQUAL-9743\n  VERSION_97(\"Changing CSF type to BOOLEAN for some has_* flags.\", true, 5),\n  VERSION_98(\"Periodic archive full rebuild. PCM-56871.\", true, 1),\n  VERSION_99(\"Removing named_entities field. SEARCH-13708\", true, 0),\n  // minor version 1: add periscope features (SEARCHQUAL-10008)\n  // minor version 2: add raw_earlybird_score to TweetExternalFeatures (SEARCHQUAL-10347)\n  VERSION_100(\"Upgrade Penguin Version from V4 to V6. SEARCH-12991\", true, 2),\n  // minor version 1: adjust for normalizer type for some engagement counters (SEARCHQUAL-9537)\n  // minor version 2: add decaying engagement counts and last engaged timestamps (SEARCHQUAL-10532)\n  VERSION_101(\"Add emoji to the index. SEARCH-12991\", true, 2),\n  VERSION_102(\"Periodic full archive rebuild. PCM-67851\", true, 0),\n  VERSION_103(\"Add liked_by_user_id field. SEARCH-15341\", true, 0),\n  // minor version 1: remove last engaged timestamp with 3-hour increment (SEARCHQUAL-10903)\n  // minor version 2: add fake engagement counts (SEARCHQUAL-10795)\n  // minor version 3: add last engaged timestamp with 1-hour increment (SEARCHQUAL-10942)\n  VERSION_104(\"Reverting to the 20170109_pc100_par30 build gen. SEARCH-15731\", true, 3),\n  VERSION_105(\"Add 3 new fields to archive index for engagement features. SEARCH-16102\", true, 0),\n  // This is the last rebuild based on /tables/statuses. Starting 9/14 this build-gen is powered\n  // by TweetSource. During this rebuild both statuses and engagement counts were rebuilt.\n  VERSION_106(\"Periodic archive full rebuild. PCM-74652\", true, 0),\n  VERSION_107(\"Removing card fields from full archive index.\", true, 0),\n  VERSION_108(\"Removing the tms_id field from all schemas.\", true, 0),\n  VERSION_109(\"Removing LAT_LON_FIELD from all schemas.\", true, 0),\n  VERSION_110(\"Adding the card fields back to the full archive index.\", true, 1),\n  // minor version 1: Add composer source csf field (SEARCH-22494)\n  VERSION_111(\"Adding composer_source to index. SEARCH-20377.\", true, 1),\n  VERSION_112(\"Partial rebuild to fix SEARCH-22529.\", true, 0),\n  VERSION_113(\"Full archive build gen 20180312_pc100_par30.\", true, 0),\n  VERSION_114(\"Fix for SEARCH-23761.\", true, 0),\n  VERSION_115(\"Add fields for quoted tweets. SEARCH-23919\", true, 0),\n  // minor version 1: Add 4 bit hashtag count, mention count and stock count (SEARCH-24336)\n  VERSION_116(\"Bump flush version for scrubbing pipeline. SEARCH-24225\", true, 1),\n  VERSION_117(\"Add retweeted_by_user_id and replied_to_by_user_id fields. SEARCH-24463\", true, 0),\n  // minor version 1: Removed dopamine_non_personalized_score (SEARCHQUAL-10321)\n  VERSION_118(\"Adding the reply and retweet source tweet IDs: SEARCH-23702, SEARCH-24502\", true, 1),\n  // minor version 1: add blink engagement counts (SEARCHQUAL-15176)\n  VERSION_119(\"Remove public inferred location: SEARCH-24235\", true, 1),\n  VERSION_120(\"Flush extensions before fields when flushing segments.\", true, 0),\n  VERSION_121(\"Flush the startingDocIdForSearch field. SEARCH-25464.\", true, 0),\n  VERSION_122(\"Do not flush the startingDocIdForSearch field.\", true, 0),\n  VERSION_123(\"Renaming the largestDocID flushed property to firstAddedDocID.\", true, 0),\n  VERSION_124(\"Use the skip list posting list for all fields.\", true, 0),\n  VERSION_125(\"Use hashmap for tweet ID lookup.\", true, 0),\n  VERSION_126(\"Use the skip list posting list for all fields.\", true, 0),\n  VERSION_127(\"Flushing the min and max doc IDs in each segment.\", true, 0),\n  VERSION_128(\"Add card_lang to index. SEARCH-26539\", true, 0),\n  VERSION_129(\"Move the tweet ID mapper to the segment data.\", true, 0),\n  VERSION_130(\"Move the time mapper to the segment data.\", true, 0),\n  VERSION_131(\"Change the facets classes to work with any doc IDs.\", true, 0),\n  VERSION_132(\"Make the CSF classes work with any doc IDs.\", true, 0),\n  VERSION_133(\"Removing smallestDocID property.\", true, 0),\n  VERSION_134(\"Optimize DeletedDocs before flushing.\", true, 0),\n  VERSION_135(\"Add payloads to skiplists.\", true, 0),\n  VERSION_136(\"Add name to int pools.\", true, 0),\n  VERSION_137(\"Add unsorted stream offset.\", true, 0),\n  VERSION_138(\"Switch to the OutOfOrderRealtimeTweetIDMapper.\", true, 0),\n  VERSION_139(\"Remove realtime posting lists.\", true, 0),\n  VERSION_140(\"Add named_entity field. SEARCH-27547\", true, 0),\n  VERSION_141(\"Flush the out of order updates count.\", true, 0),\n  VERSION_142(\"Add named_entity facet support. SEARCH-28054\", true, 0),\n  VERSION_143(\"Index updates before optimizing segment.\", true, 0),\n  VERSION_144(\"Refactor TermsArray.\", true, 0),\n  VERSION_145(\"Remove SmallestDocID.\", true, 0),\n  VERSION_146(\"Add entity_id facet support. SEARCH-28071\", true, 0),\n  VERSION_147(\"Enable updating facets\", true, 0),\n  VERSION_148(\"Rename the counter for feature updates to partial updates\", true, 0),\n  VERSION_149(\"Stop flushing offsets for sorted updates DL streams.\", true, 0),\n  VERSION_150(\"Update the name of the property for the updates DL stream offset.\", true, 0),\n  VERSION_151(\"Upgrade Lucene version to 5.5.5.\", true, 0),\n  VERSION_152(\"Upgrade Lucene version to 6.0.0.\", true, 0),\n  VERSION_153(\"Upgrade Lucene version to 6.6.6.\", true, 0),\n  VERSION_154(\"Store the timeslice ID on EarlybirdIndexSegmentData.\", true, 0),\n  VERSION_155(\"Do not flush index extensions.\", true, 0),\n  VERSION_156(\"Deprecate ThriftIndexedFieldSettings.defaultFieldBoost.\", true, 0),\n  VERSION_157(\"Load CREATED_AT_CSF_FIELD into RAM in archive.\", true, 0),\n  VERSION_158(\"Added directed at user ID field and CSF.\", true, 0),\n  VERSION_159(\"Changing deleted docs serialization format.\", true, 0),\n  VERSION_160(\"Add fields for health model scores. SEARCH-31907, HML-2099\", true, 0),\n  VERSION_161(\"Switch to the 'search' Kafka cluster.\", true, 0),\n  VERSION_162(\"Update Lucene version to 7.0.0.\", true, 0),\n  VERSION_163(\"Update Lucene version to 7.7.2.\", true, 0),\n  // minor version 1: add IS_TRENDING_NOW_FLAG\n  VERSION_164(\"Collect per-term stats in the realtime segments.\", true, 1),\n  VERSION_165(\"Update Lucene version to 8.5.2.\", true, 0),\n  VERSION_166(\"Serialize maxPosition field for InvertedRealtimeIndex\", true, 0),\n  VERSION_167(\"Add field for pSpammyTweetScore. HML-2557\", true, 0),\n  VERSION_168(\"Add field for pReportedTweetScore. HML-2644\", true, 0),\n  VERSION_169(\"Add field for spammyTweetContentScore. PFM-70\", true, 0),\n  VERSION_170(\"Add reference author id CSF. SEARCH-34715\", true, 0),\n  VERSION_171(\"Add space_id field. SEARCH-36156\", true, 0),\n  VERSION_172(\"Add facet support for space_id. SEARCH-36388\", true, 0),\n  VERSION_173(\"Add space admin and title fields. SEARCH-36986\", true, 0),\n  VERSION_174(\"Switching to Penguin v7 for realtime-exp0 cluster. SEARCH-36068\", true, 0),\n  VERSION_175(\"Adding exclusive conversation author id CSF\", true, 0),\n  VERSION_176(\"Adding card URI CSF\", true, 0),\n  // minor version 1: add FROM_BLUE_VERIFIED_ACCOUNT_FLAG\n  // minor version 2: Adding new cluster REALTIME_CG. SEARCH-45692\n  VERSION_177(\"Adding URL Description and Title fields. SEARCH-41641\", true, 2),\n\n  /**\n   * This semi colon is on a separate line to avoid polluting git blame history.\n   * Put a comma after the new enum field you're adding.\n   */;\n\n  // The current version.\n  public static final FlushVersion CURRENT_FLUSH_VERSION =\n      FlushVersion.values()[FlushVersion.values().length - 1];\n\n  public static final String DELIMITER = \"_v_\";\n\n  /* =======================================================\n   * Helper methods\n   * ======================================================= */\n  private final String description;\n  private final boolean isOfficial;\n  private final int minorVersion;\n\n  /**\n   * A flush version is not official unless explicitly stated to be official.\n   * An unofficial flush version is never uploaded to HDFS.\n   */\n  private FlushVersion(String description) {\n    this(description, false, 0);\n  }\n\n  private FlushVersion(String description, boolean isOfficial) {\n    this(description, isOfficial, 0);\n  }\n\n  private FlushVersion(String description, boolean isOfficial, int minorVersion) {\n    this.description = description;\n    this.isOfficial = isOfficial;\n    this.minorVersion = minorVersion;\n  }\n\n  /**\n   * Returns file extension with version number.\n   */\n  public String getVersionFileExtension() {\n    if (this == VERSION_0) {\n      return \"\";\n    } else {\n      return DELIMITER + ordinal();\n    }\n  }\n\n  /**\n   * Returns file extension given flush version number.\n   * If the flush version is unknown (e.g. higher than current flush version or lower than 0), null\n   * is returned.\n   */\n  @Nullable\n  public static String getVersionFileExtension(int flushVersion) {\n    if (flushVersion > CURRENT_FLUSH_VERSION.ordinal() || flushVersion < 0) {\n      return null;\n    } else {\n      return FlushVersion.values()[flushVersion].getVersionFileExtension();\n    }\n  }\n\n  /**\n   * Returns a string describing the current schema version.\n   * @deprecated Please use {@link com.twitter.search.common.schema.base.Schema#getVersionDescription()}\n   */\n  @Deprecated\n  public String getDescription() {\n    return description;\n  }\n\n  /**\n   * Returns the schema's major version.\n   * @deprecated Please use {@link com.twitter.search.common.schema.base.Schema#getMajorVersionNumber()}.\n   */\n  @Deprecated\n  public int getVersionNumber() {\n    return this.ordinal();\n  }\n\n  public boolean onOrAfter(FlushVersion other) {\n    return compareTo(other) >= 0;\n  }\n\n  /**\n   * Returns whether the schema version is official. Only official segments are uploaded to HDFS.\n   * @deprecated Please use {@link com.twitter.search.common.schema.base.Schema#isVersionOfficial()}.\n   */\n  @Deprecated\n  public boolean isOfficial() {\n    // We want the loading/flushing tests to pass locally even if the version is not meant\n    // to be an official version.\n    return isOfficial || Config.environmentIsTest();\n  }\n\n  /**\n   * As of now, this is hardcoded to 0. We will start using this soon.\n   * @deprecated Please consult schema for minor version. This should only be used to build schema.\n   */\n  @Deprecated\n  public int getMinorVersion() {\n    return minorVersion;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/AndNotDocIdSetIterator.java",
    "content": "package com.twitter.search.common.search;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.search.DocIdSetIterator;\n\npublic class AndNotDocIdSetIterator extends DocIdSetIterator {\n  private int nextDelDoc;\n  private final DocIdSetIterator baseIter;\n  private final DocIdSetIterator notIter;\n  private int currID;\n\n  /** Creates a new AndNotDocIdSetIterator instance. */\n  public AndNotDocIdSetIterator(DocIdSetIterator baseIter, DocIdSetIterator notIter)\n          throws IOException {\n    nextDelDoc = notIter.nextDoc();\n    this.baseIter = baseIter;\n    this.notIter = notIter;\n    currID = -1;\n  }\n\n  @Override\n  public int advance(int target) throws IOException {\n    currID = baseIter.advance(target);\n    if (currID == DocIdSetIterator.NO_MORE_DOCS) {\n      return currID;\n    }\n\n    if (nextDelDoc != DocIdSetIterator.NO_MORE_DOCS) {\n      if (currID < nextDelDoc) {\n        return currID;\n      } else if (currID == nextDelDoc) {\n        return nextDoc();\n      } else {\n        nextDelDoc = notIter.advance(currID);\n        if (currID == nextDelDoc) {\n          return nextDoc();\n        }\n      }\n    }\n    return currID;\n  }\n\n  @Override\n  public int docID() {\n    return currID;\n  }\n\n  @Override\n  public int nextDoc() throws IOException {\n    currID = baseIter.nextDoc();\n    if (nextDelDoc != DocIdSetIterator.NO_MORE_DOCS) {\n      while (currID != DocIdSetIterator.NO_MORE_DOCS) {\n        if (currID < nextDelDoc) {\n          return currID;\n        } else {\n          if (currID == nextDelDoc) {\n            currID = baseIter.nextDoc();\n          }\n          nextDelDoc = notIter.advance(currID);\n        }\n      }\n    }\n    return currID;\n  }\n\n  @Override\n  public long cost() {\n    return baseIter.cost();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/twitter/elephantbird:core\",\n        \"3rdparty/jvm/geo/google:geoGoogle\",\n        \"3rdparty/jvm/log4j\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-common\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-smartcn\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-facet\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-queries\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-spatial-extras\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/collections\",\n        \"src/java/com/twitter/common/util:system-mocks\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/query\",\n        \"src/java/com/twitter/search/common/schema\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/util/spatial\",\n        \"src/java/com/twitter/search/queryparser\",\n        \"src/thrift/com/twitter/search/common:facets-java\",\n        \"src/thrift/com/twitter/search/common:query-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/DelegatingEarlyTerminationCollector.java",
    "content": "package com.twitter.search.common.search;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport javax.annotation.Nullable;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.search.Collector;\nimport org.apache.lucene.search.LeafCollector;\nimport org.apache.lucene.search.Scorable;\nimport org.apache.lucene.search.ScoreMode;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.query.thriftjava.CollectorParams;\n\n/**\n * A {@link com.twitter.search.common.search.TwitterEarlyTerminationCollector}\n * that delegates actual hit collection to a sub collector.\n */\npublic final class DelegatingEarlyTerminationCollector\n    extends TwitterEarlyTerminationCollector {\n  private final Collector subCollector;\n  private LeafCollector subLeafCollector;\n\n  /** Creates a new DelegatingEarlyTerminationCollector instance. */\n  public DelegatingEarlyTerminationCollector(Collector subCollector,\n                                             CollectorParams collectorParams,\n                                             TerminationTracker terminationTracker,\n                                             @Nullable QueryCostProvider queryCostProvider,\n                                             int numDocsBetweenTimeoutChecks,\n                                             Clock clock) {\n    super(\n        collectorParams,\n        terminationTracker,\n        queryCostProvider,\n        numDocsBetweenTimeoutChecks,\n        clock);\n    this.subCollector = subCollector;\n  }\n\n  @Override\n  public void setScorer(Scorable scorer) throws IOException {\n    super.setScorer(scorer);\n    subLeafCollector.setScorer(scorer);\n  }\n\n  @Override\n  protected void doCollect() throws IOException {\n    subLeafCollector.collect(curDocId);\n  }\n\n  @Override\n  protected void doFinishSegment(int lastSearchedDocID) throws IOException {\n    if (subCollector instanceof TwitterCollector) {\n      ((TwitterCollector) subCollector).finishSegment(lastSearchedDocID);\n    }\n  }\n\n  @Override\n  public void setNextReader(LeafReaderContext context) throws IOException {\n    super.setNextReader(context);\n    subLeafCollector = subCollector.getLeafCollector(context);\n  }\n\n  @Override\n  public ScoreMode scoreMode() {\n    return subCollector.scoreMode();\n  }\n\n  @Override\n  public List<String> getDebugInfo() {\n    return null;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/DocIdTracker.java",
    "content": "package com.twitter.search.common.search;\n\n/**\n * Provide an accessor for a doc ID. This is useful for classes that iterate through doc IDs\n * and maintain a \"last seen\" doc ID.\n */\npublic interface DocIdTracker {\n  /**\n   * Retrieve current doc ID\n   */\n  int getCurrentDocId();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/EarlyTerminationState.java",
    "content": "package com.twitter.search.common.search;\n\nimport javax.annotation.Nonnull;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.metrics.SearchCounter;\n\n/**\n * This is not an enum to allow different clusters to define their own EarlyTerminationStates.\n */\npublic final class EarlyTerminationState {\n  private static final String STATS_PREFIX = \"early_termination_\";\n\n  public static final EarlyTerminationState COLLECTING =\n      new EarlyTerminationState(\"no_early_termination\", false);\n  public static final EarlyTerminationState TERMINATED_TIME_OUT_EXCEEDED =\n      new EarlyTerminationState(\"terminated_timeout_exceeded\", true);\n  public static final EarlyTerminationState TERMINATED_MAX_QUERY_COST_EXCEEDED =\n      new EarlyTerminationState(\"terminated_max_query_cost_exceeded\", true);\n  public static final EarlyTerminationState TERMINATED_MAX_HITS_EXCEEDED =\n      new EarlyTerminationState(\"terminated_max_hits_exceeded\", true);\n  public static final EarlyTerminationState TERMINATED_NUM_RESULTS_EXCEEDED =\n      new EarlyTerminationState(\"terminated_num_results_exceeded\", true);\n\n\n  // This string can be returned as a part of a search response, to tell the searcher\n  // why the search got early terminated.\n  private final String terminationReason;\n  private final boolean terminated;\n  private final SearchCounter count;\n\n  public EarlyTerminationState(@Nonnull String terminationReason, boolean terminated) {\n    this.terminationReason = Preconditions.checkNotNull(terminationReason);\n    this.terminated = terminated;\n    count = SearchCounter.export(STATS_PREFIX + terminationReason + \"_count\");\n\n  }\n\n  public boolean isTerminated() {\n    return terminated;\n  }\n\n  public String getTerminationReason() {\n    return terminationReason;\n  }\n\n  public void incrementCount() {\n    count.increment();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/GeoQuadTreeQueryBuilderUtil.java",
    "content": "package com.twitter.search.common.search;\n\nimport java.util.LinkedHashSet;\nimport java.util.Set;\n\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.spatial.prefix.tree.Cell;\nimport org.apache.lucene.spatial.prefix.tree.CellIterator;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.util.spatial.GeohashChunkImpl;\nimport com.twitter.search.queryparser.util.GeoCode;\n\nimport geo.google.datamodel.GeoAddressAccuracy;\n\npublic final class GeoQuadTreeQueryBuilderUtil {\n  private GeoQuadTreeQueryBuilderUtil() {\n  }\n\n  /**\n   * Build a geo quad tree query based around the geo code based on the geo field.\n   * @param geocode the geo location for the quad tree query\n   * @param field the field where the geohash tokens are indexed\n   * @return the corresponding for the geo quad tree query\n   */\n  public static Query buildGeoQuadTreeQuery(GeoCode geocode, String field) {\n    Set<BytesRef> geoHashSet = new LinkedHashSet<>();\n\n    // if accuracy is specified. Add a term query based on accuracy.\n    if (geocode.accuracy != GeoAddressAccuracy.UNKNOWN_LOCATION.getCode()) {\n      BytesRef termRef = new BytesRef(GeohashChunkImpl.buildGeoStringWithAccuracy(geocode.latitude,\n          geocode.longitude,\n          geocode.accuracy));\n      geoHashSet.add(termRef);\n    }\n\n    // If distance is specified. Add term queries based on distance\n    if (geocode.distanceKm != GeoCode.DOUBLE_DISTANCE_NOT_SET) {\n      // Build query based on distance\n      int treeLevel = -1;\n      // First find block containing query point with diagonal greater than 2 * radius.\n      Cell centerNode = GeohashChunkImpl.getGeoNodeByRadius(geocode.latitude, geocode.longitude,\n          geocode.distanceKm);\n      // Add center node querying term\n      if (centerNode != null) {\n        geoHashSet.add(centerNode.getTokenBytesNoLeaf(new BytesRef()));\n        treeLevel = centerNode.getLevel();\n      }\n\n      // This improves edge case recall, by adding cells also intersecting the query area.\n      CellIterator nodes = GeohashChunkImpl.getNodesIntersectingCircle(geocode.latitude,\n          geocode.longitude,\n          geocode.distanceKm,\n          treeLevel);\n      // If there are other nodes intersecting query circle, also add them in.\n      if (nodes != null) {\n        while (nodes.hasNext()) {\n          geoHashSet.add(nodes.next().getTokenBytesNoLeaf(new BytesRef()));\n        }\n      }\n    }\n\n    return new com.twitter.search.common.query.MultiTermDisjunctionQuery(field, geoHashSet);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/IntArrayDocIdSetIterator.java",
    "content": "package com.twitter.search.common.search;\n\nimport java.util.Arrays;\n\nimport org.apache.lucene.search.DocIdSetIterator;\n\n/**\n * DocIdSetIterator implementation from a sorted list of non-negative integers. If the given list of\n * doc IDs is not sorted or contains negative doc IDs, the results are undefined.\n */\npublic class IntArrayDocIdSetIterator extends DocIdSetIterator {\n  private final int[] docIds;\n  private int docId;\n  private int cursor;\n\n  public IntArrayDocIdSetIterator(int[] ids) {\n    docIds = ids;\n    reset();\n  }\n\n  /** Used for testing. */\n  public void reset() {\n    docId = -1;\n    cursor = -1;\n  }\n\n  @Override\n  public int docID() {\n    return docId;\n  }\n\n  @Override\n  public int nextDoc() {\n    return advance(docId);\n  }\n\n  @Override\n  public int advance(int target) {\n    if (docId == NO_MORE_DOCS) {\n      return docId;\n    }\n\n    if (target < docId) {\n      return docId;\n    }\n\n    if (cursor == docIds.length - 1) {\n      docId = NO_MORE_DOCS;\n      return docId;\n    }\n\n    if (target == docId) {\n      docId = docIds[++cursor];\n      return docId;\n    }\n\n    int toIndex = Math.min(cursor + (target - docId) + 1, docIds.length);\n    int targetIndex = Arrays.binarySearch(docIds, cursor + 1, toIndex, target);\n    if (targetIndex < 0) {\n      targetIndex = -targetIndex - 1;\n    }\n\n    if (targetIndex == docIds.length) {\n      docId = NO_MORE_DOCS;\n    } else {\n      cursor = targetIndex;\n      docId = docIds[cursor];\n    }\n    return docId;\n  }\n\n  @Override\n  public long cost() {\n    return docIds == null ? 0 : docIds.length;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/PairDocIdSetIterator.java",
    "content": "package com.twitter.search.common.search;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.search.DocIdSetIterator;\n/**\n * Disjunction over 2 DocIdSetIterators. This should be faster than a disjunction over N since there\n * would be no need to adjust the heap.\n */\npublic class PairDocIdSetIterator extends DocIdSetIterator {\n\n  private final DocIdSetIterator d1;\n  private final DocIdSetIterator d2;\n\n  private int doc = -1;\n\n  /** Creates a new PairDocIdSetIterator instance. */\n  public PairDocIdSetIterator(DocIdSetIterator d1, DocIdSetIterator d2) throws IOException {\n    Preconditions.checkNotNull(d1);\n    Preconditions.checkNotNull(d2);\n    this.d1 = d1;\n    this.d2 = d2;\n    // position the iterators\n    this.d1.nextDoc();\n    this.d2.nextDoc();\n  }\n\n  @Override\n  public int docID() {\n    return doc;\n  }\n\n  @Override\n  public int nextDoc() throws IOException {\n    int doc1 = d1.docID();\n    int doc2 = d2.docID();\n    DocIdSetIterator iter = null;\n    if (doc1 < doc2) {\n      doc = doc1;\n      //d1.nextDoc();\n      iter = d1;\n    } else if (doc1 > doc2) {\n      doc = doc2;\n      //d2.nextDoc();\n      iter = d2;\n    } else {\n      doc = doc1;\n      //d1.nextDoc();\n      //d2.nextDoc();\n    }\n\n    if (doc != NO_MORE_DOCS) {\n      if (iter != null) {\n        iter.nextDoc();\n      } else {\n        d1.nextDoc();\n        d2.nextDoc();\n      }\n    }\n    return doc;\n  }\n\n  @Override\n  public int advance(int target) throws IOException {\n    if (d1.docID() < target) {\n      d1.advance(target);\n    }\n    if (d2.docID() < target) {\n      d2.advance(target);\n    }\n    return (doc != NO_MORE_DOCS) ? nextDoc() : doc;\n  }\n\n  @Override\n  public long cost() {\n    // very coarse estimate\n    return d1.cost() + d2.cost();\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/QueryCostProvider.java",
    "content": "package com.twitter.search.common.search;\n\n/**\n * Any class that can track and return query cost.\n */\npublic interface QueryCostProvider {\n  /** Returns the total cost. */\n  double getTotalCost();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/TerminationTracker.java",
    "content": "package com.twitter.search.common.search;\n\nimport java.util.HashSet;\nimport java.util.Set;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.query.thriftjava.CollectorTerminationParams;\n\n/**\n * Used for tracking termination criteria for earlybird queries.\n *\n * Currently this tracks the query time out and query cost, if they are set on the\n * {@link com.twitter.search.common.query.thriftjava.CollectorTerminationParams}.\n */\npublic class TerminationTracker {\n  /** Query start time provided by client. */\n  private final long clientStartTimeMillis;\n\n  /** Timeout end times, calculated from {@link #clientStartTimeMillis}. */\n  private final long timeoutEndTimeMillis;\n\n  /** Query start time recorded at earlybird server. */\n  private final long localStartTimeMillis;\n\n  /** Tracking query cost */\n  private final double maxQueryCost;\n\n  // Sometimes, we want to early terminate before timeoutEndTimeMillis, to reserve time for\n  // work that needs to be done after early termination (E.g. merging results).\n  private final int postTerminationOverheadMillis;\n\n  // We don't check for early termination often enough. Some times requests timeout in between\n  // early termination checks. This buffer time is also substracted from deadline.\n  // To illustrate how this is used, let's use a simple example:\n  // If we spent 750ms searching 5 segments, a rough estimate is that we need 150ms to search\n  // one segment. If the timeout is set to 800ms, we should not starting searching the next segment.\n  // In this case, on can set preTerminationSafeBufferTimeMillis to 150ms, so that when early\n  // termination check computes the deadline, this buffer is also subtracted. See SEARCH-29723.\n  private int preTerminationSafeBufferTimeMillis = 0;\n\n  private EarlyTerminationState earlyTerminationState = EarlyTerminationState.COLLECTING;\n\n  // This flag determines whether the last searched doc ID trackers should be consulted when a\n  // timeout occurs.\n  private final boolean useLastSearchedDocIdOnTimeout;\n\n  private final Set<DocIdTracker> lastSearchedDocIdTrackers = new HashSet<>();\n\n  /**\n   * Creates a new termination tracker that will not specify a timeout or max query cost.\n   * Can be used for queries that explicitly do not want to use a timeout. Meant to be used for\n   * tests, and background queries running for the query cache.\n   */\n  public TerminationTracker(Clock clock) {\n    this.clientStartTimeMillis = clock.nowMillis();\n    this.localStartTimeMillis = clientStartTimeMillis;\n    this.timeoutEndTimeMillis = Long.MAX_VALUE;\n    this.maxQueryCost = Double.MAX_VALUE;\n    this.postTerminationOverheadMillis = 0;\n    this.useLastSearchedDocIdOnTimeout = false;\n  }\n\n  /**\n   * Convenient method overloading for\n   * {@link #TerminationTracker(CollectorTerminationParams, long, Clock, int)}.\n   */\n  public TerminationTracker(\n      CollectorTerminationParams terminationParams, Clock clock,\n      int postTerminationOverheadMillis) {\n    this(terminationParams, clock.nowMillis(), clock, postTerminationOverheadMillis);\n  }\n\n  /**\n   * Convenient method overloading for\n   * {@link #TerminationTracker(CollectorTerminationParams, long, Clock, int)}.\n   */\n  public TerminationTracker(\n      CollectorTerminationParams terminationParams, int postTerminationOverheadMillis) {\n    this(\n        terminationParams,\n        System.currentTimeMillis(),\n        Clock.SYSTEM_CLOCK,\n        postTerminationOverheadMillis);\n  }\n\n  /**\n   * Creates a new TerminationTracker instance.\n   *\n   * @param terminationParams  CollectorParams.CollectorTerminationParams carrying parameters\n   *                           about early termination.\n   * @param clientStartTimeMillis The query start time (in millis) specified by client. This is used\n   *                              to calculate timeout end time, like {@link #timeoutEndTimeMillis}.\n   * @param clock used to sample {@link #localStartTimeMillis}.\n   * @param postTerminationOverheadMillis How much time should be reserved.  E.g. if request time\n   *                                      out is 800ms, and this is set to 200ms, early termination\n   *                                      will kick in at 600ms mark.\n   */\n  public TerminationTracker(\n      CollectorTerminationParams terminationParams,\n      long clientStartTimeMillis,\n      Clock clock,\n      int postTerminationOverheadMillis) {\n    Preconditions.checkNotNull(terminationParams);\n    Preconditions.checkArgument(postTerminationOverheadMillis >= 0);\n\n    this.clientStartTimeMillis = clientStartTimeMillis;\n    this.localStartTimeMillis = clock.nowMillis();\n\n    if (terminationParams.isSetTimeoutMs()\n        && terminationParams.getTimeoutMs() > 0) {\n      Preconditions.checkState(terminationParams.getTimeoutMs() >= postTerminationOverheadMillis);\n      this.timeoutEndTimeMillis = this.clientStartTimeMillis + terminationParams.getTimeoutMs();\n    } else {\n      // Effectively no timeout.\n      this.timeoutEndTimeMillis = Long.MAX_VALUE;\n    }\n\n    // Tracking query cost\n    if (terminationParams.isSetMaxQueryCost()\n        && terminationParams.getMaxQueryCost() > 0) {\n      maxQueryCost = terminationParams.getMaxQueryCost();\n    } else {\n      maxQueryCost = Double.MAX_VALUE;\n    }\n\n    this.useLastSearchedDocIdOnTimeout = terminationParams.isEnforceQueryTimeout();\n    this.postTerminationOverheadMillis = postTerminationOverheadMillis;\n  }\n\n  /**\n   * Returns the reserve time to perform post termination work. Return the deadline timestamp\n   * with postTerminationWorkEstimate subtracted.\n   */\n  public long getTimeoutEndTimeWithReservation() {\n    // Return huge value if time out is disabled.\n    if (timeoutEndTimeMillis == Long.MAX_VALUE) {\n      return timeoutEndTimeMillis;\n    } else {\n      return timeoutEndTimeMillis\n          - postTerminationOverheadMillis\n          - preTerminationSafeBufferTimeMillis;\n    }\n  }\n\n  public void setPreTerminationSafeBufferTimeMillis(int preTerminationSafeBufferTimeMillis) {\n    Preconditions.checkArgument(preTerminationSafeBufferTimeMillis >= 0);\n\n    this.preTerminationSafeBufferTimeMillis = preTerminationSafeBufferTimeMillis;\n  }\n\n  public long getLocalStartTimeMillis() {\n    return localStartTimeMillis;\n  }\n\n  public long getClientStartTimeMillis() {\n    return clientStartTimeMillis;\n  }\n\n  public double getMaxQueryCost() {\n    return maxQueryCost;\n  }\n\n  public boolean isEarlyTerminated() {\n    return earlyTerminationState.isTerminated();\n  }\n\n  public EarlyTerminationState getEarlyTerminationState() {\n    return earlyTerminationState;\n  }\n\n  public void setEarlyTerminationState(EarlyTerminationState earlyTerminationState) {\n    this.earlyTerminationState = earlyTerminationState;\n  }\n\n  /**\n   * Return the minimum searched doc ID amongst all registered trackers, or -1 if there aren't any\n   * trackers. Doc IDs are stored in ascending order, and trackers update their doc IDs as they\n   * search, so the minimum doc ID reflects the most recent fully searched doc ID.\n   */\n  int getLastSearchedDocId() {\n    return lastSearchedDocIdTrackers.stream()\n        .mapToInt(DocIdTracker::getCurrentDocId).min().orElse(-1);\n  }\n\n  void resetDocIdTrackers() {\n    lastSearchedDocIdTrackers.clear();\n  }\n\n  /**\n   * Add a DocIdTracker, to keep track of the last fully-searched doc ID when early termination\n   * occurs.\n   */\n  public void addDocIdTracker(DocIdTracker docIdTracker) {\n    lastSearchedDocIdTrackers.add(docIdTracker);\n  }\n\n  public boolean useLastSearchedDocIdOnTimeout() {\n    return useLastSearchedDocIdOnTimeout;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/TwitterCollector.java",
    "content": "package com.twitter.search.common.search;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.search.Collector;\n\n/**\n * Lucene Collectors throw CollectionTerminatedException to perform early termination.\n * We don't believe that throwing Exceptions to control execution flow is ideal, so we are adding\n * this class to be a base of all Twitter Collectors.\n *\n * {@link com.twitter.search.common.search.TwitterIndexSearcher} uses the {@link #isTerminated()}\n * method to perform early termination, instead of relying on CollectionTerminatedException.\n */\npublic abstract class TwitterCollector implements Collector {\n\n  /**\n   * Subclasses should return true if they want to perform early termination.\n   * This method is called every hit and should not be expensive.\n   */\n  public abstract boolean isTerminated() throws IOException;\n\n  /**\n   * Lucene API only has a method that's called before searching a segment setNextReader().\n   * This hook is called after finishing searching a segment.\n   * @param lastSearchedDocID is the last docid searched before termination,\n   * or NO_MORE_DOCS if there was no early termination.  This doc need not be a hit,\n   * and should not be collected here.\n   */\n  public abstract void finishSegment(int lastSearchedDocID) throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/TwitterEarlyTerminationCollector.java",
    "content": "package com.twitter.search.common.search;\n\nimport java.io.IOException;\nimport java.util.List;\nimport javax.annotation.Nonnull;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.search.LeafCollector;\nimport org.apache.lucene.search.Scorable;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.query.thriftjava.CollectorParams;\nimport com.twitter.search.common.query.thriftjava.CollectorTerminationParams;\n\n/**\n * A TwitterCollector containing the most common early termination logic based on\n * timeout, cost, and max hits. This class does not do any actual hit collection---this class\n * is abstract and cannot be instantiated.\n *\n * If a Collector and all its subclasses need early termination, it should extend this class.\n *\n * However, if one just wants to add EarlyTermination to any single collector, he can just\n * use {@link DelegatingEarlyTerminationCollector}\n * as a wrapper.\n */\npublic abstract class TwitterEarlyTerminationCollector\n    extends TwitterCollector implements LeafCollector {\n  private static final Logger LOG = LoggerFactory.getLogger(TwitterEarlyTerminationCollector.class);\n  private static final SearchCounter NEGATIVE_TIME_PER_SEGMENT =\n      SearchCounter.export(\"TwitterEarlyTerminationCollector_negative_time_per_segment\");\n  private static final SearchRateCounter QUERY_TIMEOUT_ENFORCED =\n      SearchRateCounter.export(\"TwitterEarlyTerminationCollector_query_timeout_enforced\");\n\n  protected int curDocId = -1;\n\n  protected Scorable scorer = null;\n  private LeafReader curReader = null;\n  private final long maxHitsToProcess;\n  private long numHitsProcessed = 0;\n  private int lastEarlyTerminationCheckDocId = -1;\n  private final Clock clock;\n\n  @Nullable\n  private final QueryCostProvider queryCostProvider;\n\n  private final TerminationTracker terminationTracker;\n\n  // This determines how often the expensive early termination check is performed.\n  // If set to be negative, expensive early termination check only performed at segment boundaries.\n  // If set to a positive number X, this check is performed every X docs processed.\n  private int numDocsBetweenTimeoutChecks;\n\n  // Number of segments searched so far.\n  // This is used to predicatively early terminate.\n  // Expensive early termination checks may not happen often enough. Sometimes the request\n  // times out in between the termination checks.\n  // After finishing searching a segment, we estimate how much time is needed to search one\n  // segment on average.  If searching the next segment would cause a timeout, we early terminate.\n  private int numSearchedSegments = 0;\n\n  /**\n   * Creates a new TwitterEarlyTerminationCollector instance.\n   *\n   * @param collectorParams the parameters needed to guide early termination.\n   * @param terminationTracker If null is passed in, a new TerminationTrack is created. Otherwise,\n   *        the one passed in is used.\n   * @param numDocsBetweenTimeoutChecks TerminationTracker based check are performed upon a hit\n   *        every numDocsBetweenTimeoutChecks docs. If a non-positive number is passed\n   *        in, TerminationTracker based checks are disabled.\n   *        If collectorParams specifies a value as well, that value is used.\n   */\n  public TwitterEarlyTerminationCollector(\n      CollectorParams collectorParams,\n      TerminationTracker terminationTracker,\n      @Nullable QueryCostProvider queryCostProvider,\n      int numDocsBetweenTimeoutChecks,\n      Clock clock) {\n    CollectorTerminationParams terminationParams = collectorParams.getTerminationParams();\n\n    if (terminationParams == null) {\n      terminationParams = new CollectorTerminationParams()\n          .setMaxHitsToProcess(Integer.MAX_VALUE)\n          .setMaxQueryCost(Double.MAX_VALUE)\n          .setTimeoutMs(Integer.MAX_VALUE);\n    }\n\n    if (!terminationParams.isSetMaxHitsToProcess() || terminationParams.getMaxHitsToProcess() < 0) {\n      maxHitsToProcess = Integer.MAX_VALUE;\n    } else {\n      maxHitsToProcess = terminationParams.getMaxHitsToProcess();\n    }\n\n    if (terminationParams.isSetNumDocsBetweenTimeoutChecks()) {\n      this.numDocsBetweenTimeoutChecks = terminationParams.getNumDocsBetweenTimeoutChecks();\n    } else {\n      this.numDocsBetweenTimeoutChecks = numDocsBetweenTimeoutChecks;\n    }\n\n    this.terminationTracker = Preconditions.checkNotNull(terminationTracker);\n    this.queryCostProvider = queryCostProvider;\n    this.clock = clock;\n  }\n\n  public final LeafCollector getLeafCollector(LeafReaderContext context) throws IOException {\n    this.setNextReader(context);\n    return this;\n  }\n\n  /**\n   * Sub-classes may override this to add more collection logic.\n   */\n  protected abstract void doCollect() throws IOException;\n\n  /**\n   * Sub-classes may override this to add more segment completion logic.\n   * @param lastSearchedDocID is the last docid searched before termination,\n   * or NO_MORE_DOCS if there was no early termination.  This doc may not be a hit!\n   */\n  protected abstract void doFinishSegment(int lastSearchedDocID) throws IOException;\n\n  /**\n   *  sub classes can override this to perform more early termination checks.\n   */\n  public EarlyTerminationState innerShouldCollectMore() throws IOException {\n    return EarlyTerminationState.COLLECTING;\n  }\n\n  /**\n   * After early termination, this method can be used to retrieve early termination reason.\n   */\n  @Nonnull\n  public final EarlyTerminationState getEarlyTerminationState() {\n    return terminationTracker.getEarlyTerminationState();\n  }\n\n  protected final EarlyTerminationState setEarlyTerminationState(\n      EarlyTerminationState newEarlyTerminationState) {\n    terminationTracker.setEarlyTerminationState(newEarlyTerminationState);\n    return newEarlyTerminationState;\n  }\n\n  @Override\n  public final boolean isTerminated() throws IOException {\n    EarlyTerminationState earlyTerminationState = getEarlyTerminationState();\n\n    if (earlyTerminationState.isTerminated()) {\n      return true;\n    }\n\n    if (getNumHitsProcessed() >= getMaxHitsToProcess()) {\n      collectedEnoughResults();\n      if (shouldTerminate()) {\n        return setEarlyTerminationState(EarlyTerminationState.TERMINATED_MAX_HITS_EXCEEDED)\n            .isTerminated();\n      } else {\n        return false;\n      }\n    }\n\n    return innerShouldCollectMore().isTerminated();\n  }\n\n  /**\n   * Note: subclasses overriding this method are expected to call \"super.setNextReader\"\n   * in their setNextReader().\n   * @deprecated Remove this methods in favor of {@link #getLeafCollector(LeafReaderContext)}\n   */\n  @Deprecated\n  public void setNextReader(LeafReaderContext context) throws IOException {\n    if (!terminationTracker.useLastSearchedDocIdOnTimeout()) {\n      expensiveEarlyTerminationCheck();\n    }\n\n    // Reset curDocId for next segment\n    curDocId = -1;\n    lastEarlyTerminationCheckDocId = -1;\n    curReader = context.reader();\n  }\n\n  /**\n   * Sub-classes overriding this method are expected to call super.setScorer()\n   */\n  @Override\n  public void setScorer(Scorable scorer) throws IOException {\n    this.scorer = scorer;\n  }\n\n  @Override\n  public final void collect(int doc) throws IOException {\n    curDocId = doc;\n    doCollect();\n    numHitsProcessed++;\n    if (numDocsBetweenTimeoutChecks > 0\n        && (curDocId - lastEarlyTerminationCheckDocId) >= numDocsBetweenTimeoutChecks) {\n      lastEarlyTerminationCheckDocId = curDocId;\n\n      if (!terminationTracker.useLastSearchedDocIdOnTimeout()) {\n        expensiveEarlyTerminationCheck();\n      }\n    }\n  }\n\n  /**\n   * Accounting for a segment searched.\n   * @param lastSearchedDocID is the last docid searched before termination,\n   * or NO_MORE_DOCS if there was no early termination.  This doc may not be a hit!\n   */\n  protected final void trackCompleteSegment(int lastSearchedDocID) throws IOException {\n    doFinishSegment(lastSearchedDocID);\n  }\n\n  @Override\n  public final void finishSegment(int lastSearchedDocID) throws IOException {\n    // finished searching a segment. Computer average time needed to search a segment.\n    Preconditions.checkState(curReader != null, \"Did subclass call super.setNextReader()?\");\n    numSearchedSegments++;\n\n    long totalTime = clock.nowMillis() - terminationTracker.getLocalStartTimeMillis();\n\n    if (totalTime >= Integer.MAX_VALUE) {\n      String msg = String.format(\n          \"%s: A query runs for %d that is longer than Integer.MAX_VALUE ms. lastSearchedDocID: %d\",\n          getClass().getSimpleName(), totalTime, lastSearchedDocID\n      );\n      LOG.error(msg);\n      throw new IllegalStateException(msg);\n    }\n\n    int timePerSegment = ((int) totalTime) / numSearchedSegments;\n\n    if (timePerSegment < 0) {\n      NEGATIVE_TIME_PER_SEGMENT.increment();\n      timePerSegment = 0;\n    }\n\n    // If we're enforcing timeout via the last searched doc ID, we don't need to add this buffer,\n    // since we'll detect the timeout right away.\n    if (!terminationTracker.useLastSearchedDocIdOnTimeout()) {\n      terminationTracker.setPreTerminationSafeBufferTimeMillis(timePerSegment);\n    }\n\n    // Check whether we timed out and are checking for timeout at the leaves. If so, we should use\n    // the captured lastSearchedDocId from the tracker instead, which is the most up-to-date amongst\n    // the query nodes.\n    if (terminationTracker.useLastSearchedDocIdOnTimeout()\n        && EarlyTerminationState.TERMINATED_TIME_OUT_EXCEEDED.equals(\n            terminationTracker.getEarlyTerminationState())) {\n      QUERY_TIMEOUT_ENFORCED.increment();\n      trackCompleteSegment(terminationTracker.getLastSearchedDocId());\n    } else {\n      trackCompleteSegment(lastSearchedDocID);\n    }\n\n    // We finished a segment, so clear out the DocIdTrackers. The next segment will register its\n    // own trackers, and we don't need to keep the trackers from the current segment.\n    terminationTracker.resetDocIdTrackers();\n\n    curDocId = -1;\n    curReader = null;\n    scorer = null;\n  }\n\n  /**\n   * More expensive Early Termination checks, which are not called every hit.\n   * This sets EarlyTerminationState if it decides that early termination should kick in.\n   * See: SEARCH-29723.\n   */\n  private void expensiveEarlyTerminationCheck() {\n    if (queryCostProvider != null) {\n      double totalQueryCost = queryCostProvider.getTotalCost();\n      double maxQueryCost = terminationTracker.getMaxQueryCost();\n      if (totalQueryCost >= maxQueryCost) {\n        setEarlyTerminationState(EarlyTerminationState.TERMINATED_MAX_QUERY_COST_EXCEEDED);\n      }\n    }\n\n    final long nowMillis = clock.nowMillis();\n    if (nowMillis >= terminationTracker.getTimeoutEndTimeWithReservation()) {\n      setEarlyTerminationState(EarlyTerminationState.TERMINATED_TIME_OUT_EXCEEDED);\n    }\n  }\n\n  public long getMaxHitsToProcess() {\n    return maxHitsToProcess;\n  }\n\n  public final void setNumHitsProcessed(long numHitsProcessed) {\n    this.numHitsProcessed = numHitsProcessed;\n  }\n\n  protected final long getNumHitsProcessed() {\n    return numHitsProcessed;\n  }\n\n  protected final int getNumSearchedSegments() {\n    return numSearchedSegments;\n  }\n\n  protected final Clock getClock() {\n    return clock;\n  }\n\n  @VisibleForTesting\n  protected final TerminationTracker getTerminationTracker() {\n    return this.terminationTracker;\n  }\n\n  protected void collectedEnoughResults() throws IOException {\n  }\n\n  protected boolean shouldTerminate() {\n    return true;\n  }\n\n  /**\n   * Debug info collected during execution.\n   */\n  public abstract List<String> getDebugInfo();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/TwitterIndexSearcher.java",
    "content": "package com.twitter.search.common.search;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.IndexReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.MultiDocValues;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.search.CollectionStatistics;\nimport org.apache.lucene.search.Collector;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.LeafCollector;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.TermStatistics;\nimport org.apache.lucene.search.Weight;\n\n/**\n * An IndexSearch that works with TwitterEarlyTerminationCollector.\n * If a stock Lucene collector is passed into search(), this IndexSearch.search() behaves the\n * same as Lucene's stock IndexSearcher.  However, if a TwitterEarlyTerminationCollector is passed\n * in, this IndexSearcher performs early termination without relying on\n * CollectionTerminatedException.\n */\npublic class TwitterIndexSearcher extends IndexSearcher {\n  public TwitterIndexSearcher(IndexReader r) {\n    super(r);\n  }\n\n  /**\n   * search() main loop.\n   * This behaves exactly like IndexSearcher.search() if a stock Lucene collector passed in.\n   * However, if a TwitterCollector is passed in, this class performs Twitter style early\n   * termination without relying on\n   * {@link org.apache.lucene.search.CollectionTerminatedException}.\n   */\n  @Override\n  protected void search(List<LeafReaderContext> leaves, Weight weight, Collector coll)\n      throws IOException {\n\n    // If an TwitterCollector is passed in, we can do a few extra things in here, such\n    // as early termination.  Otherwise we can just fall back to IndexSearcher.search().\n    if (coll instanceof TwitterCollector) {\n      TwitterCollector collector = (TwitterCollector) coll;\n\n      for (LeafReaderContext ctx : leaves) { // search each subreader\n        if (collector.isTerminated()) {\n          return;\n        }\n\n        // Notify the collector that we're starting this segment, and check for early\n        // termination criteria again.  setNextReader() performs 'expensive' early\n        // termination checks in some implementations such as TwitterEarlyTerminationCollector.\n        LeafCollector leafCollector = collector.getLeafCollector(ctx);\n        if (collector.isTerminated()) {\n          return;\n        }\n\n        // Initialize the scorer - it should not be null.  Note that constructing the scorer\n        // may actually do real work, such as advancing to the first hit.\n        Scorer scorer = weight.scorer(ctx);\n\n        if (scorer == null) {\n          collector.finishSegment(DocIdSetIterator.NO_MORE_DOCS);\n          continue;\n        }\n\n        leafCollector.setScorer(scorer);\n\n        // Start searching.\n        DocIdSetIterator docIdSetIterator = scorer.iterator();\n        int docID = docIdSetIterator.nextDoc();\n        if (docID != DocIdSetIterator.NO_MORE_DOCS) {\n          // Collect results.  Note: check isTerminated() before calling nextDoc().\n          do {\n            leafCollector.collect(docID);\n          } while (!collector.isTerminated()\n                   && (docID = docIdSetIterator.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS);\n        }\n\n        // Always finish the segment, providing the last docID advanced to.\n        collector.finishSegment(docID);\n      }\n    } else {\n      // The collector given is not a TwitterCollector, just use stock lucene search().\n      super.search(leaves, weight, coll);\n    }\n  }\n\n  /** Returns {@link NumericDocValues} for this field, or\n   *  null if no {@link NumericDocValues} were indexed for\n   *  this field.  The returned instance should only be\n   *  used by a single thread. */\n  public NumericDocValues getNumericDocValues(String field) throws IOException {\n    return MultiDocValues.getNumericValues(getIndexReader(), field);\n  }\n\n  @Override\n  public CollectionStatistics collectionStatistics(String field) throws IOException {\n    return collectionStatistics(field, getIndexReader());\n  }\n\n  @Override\n  public TermStatistics termStatistics(Term term, int docFreq, long totalTermFreq) {\n    return termStats(term, docFreq, totalTermFreq);\n  }\n\n  /**\n   * Lucene relies on the fact that maxDocID is typically equal to the number of documents in the\n   * index, which is false when we have sparse doc IDs or when we start from 8 million docs and\n   * decrement, so in this class we pass in numDocs instead of the maximum assigned document ID.\n   * Note that the comment on {@link CollectionStatistics#maxDoc()} says that it returns the number\n   * of documents in the segment, not the maximum ID, and that it is only used this way. This is\n   * necessary for all lucene scoring methods, e.g.\n   * {@link org.apache.lucene.search.similarities.TFIDFSimilarity#idfExplain}. This method body is\n   * largely copied from {@link IndexSearcher#collectionStatistics(String)}.\n   */\n  public static CollectionStatistics collectionStatistics(String field, IndexReader indexReader)\n      throws IOException {\n    Preconditions.checkNotNull(field);\n\n    int docsWithField = 0;\n    long sumTotalTermFreq = 0;\n    long sumDocFreq = 0;\n    for (LeafReaderContext leaf : indexReader.leaves()) {\n      Terms terms = leaf.reader().terms(field);\n      if (terms == null) {\n        continue;\n      }\n\n      docsWithField += terms.getDocCount();\n      sumTotalTermFreq += terms.getSumTotalTermFreq();\n      sumDocFreq += terms.getSumDocFreq();\n    }\n\n    if (docsWithField == 0) {\n      // The CollectionStatistics API in Lucene is designed poorly. On one hand, starting with\n      // Lucene 8.0.0, searchers are expected to always produce valid CollectionStatistics instances\n      // and all int fields in these instances are expected to be strictly greater than 0. On the\n      // other hand, Lucene itself produces null CollectionStatistics instances in a few places.\n      // Also, there's no good placeholder value to indicate that a field is empty, which is a very\n      // reasonable thing to happen (for example, the first few tweets in a new segment might not\n      // have any links, so then the resolved_links_text would be empty). So to get around this\n      // issue, we do here what Lucene does: we return a CollectionStatistics instance with all\n      // fields set to 1.\n      return new CollectionStatistics(field, 1, 1, 1, 1);\n    }\n\n    // The writer could have added more docs to the index since this searcher started processing\n    // this request, or could be in the middle of adding a doc, which could mean that only some of\n    // the docsWithField, sumTotalTermFreq and sumDocFreq stats have been updated. I don't think\n    // this is a big deal, as these stats are only used for computing a hit's score, and minor\n    // inaccuracies should have very little effect on a hit's final score. But CollectionStatistic's\n    // constructor has some strict asserts for the relationship between these stats. So we need to\n    // make sure we cap the values of these stats appropriately.\n    //\n    // Adjust numDocs based on docsWithField (instead of doing the opposite), because:\n    //   1. If new documents were added to this segment after the reader was created, it seems\n    //      reasonable to take the more recent information into account.\n    //   2. The termStats() method below will return the most recent docFreq (not the value that\n    //      docFreq was set to when this reader was created). If this value is higher than numDocs,\n    //      then Lucene might end up producing negative scores, which must never happen.\n    int numDocs = Math.max(indexReader.numDocs(), docsWithField);\n    sumDocFreq = Math.max(sumDocFreq, docsWithField);\n    sumTotalTermFreq = Math.max(sumTotalTermFreq, sumDocFreq);\n    return new CollectionStatistics(field, numDocs, docsWithField, sumTotalTermFreq, sumDocFreq);\n  }\n\n  /**\n   * This method body is largely copied from {@link IndexSearcher#termStatistics(Term, int, long)}.\n   * The only difference is that we make sure all parameters we pass to the TermStatistics instance\n   * we create are set to at least 1 (because Lucene 8.0.0 expects them to be).\n   */\n  public static TermStatistics termStats(Term term, int docFreq, long totalTermFreq) {\n    // Lucene expects the doc frequency and total term frequency to be at least 1. This assumption\n    // doesn't always make sense (the segment can be empty -- see comment above), but to make Lucene\n    // happy, make sure to always set these parameters to at least 1.\n    int adjustedDocFreq = Math.max(docFreq, 1);\n    return new TermStatistics(\n        term.bytes(),\n        adjustedDocFreq,\n        Math.max(totalTermFreq, adjustedDocFreq));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/termination/BUILD",
    "content": "java_library(\n    name = \"termination\",\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-common\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-facet\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-queries\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/util:system-mocks\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/query\",\n        \"src/java/com/twitter/search/common/search\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/termination/QueryTimeout.java",
    "content": "package com.twitter.search.common.search.termination;\n\nimport com.twitter.search.common.search.DocIdTracker;\n\n/**\n * QueryTimeout provides a method for early termination of queries.\n */\npublic interface QueryTimeout {\n  /**\n   * Returns true if query processing should terminate, otherwise false.\n   */\n  boolean shouldExit();\n\n  /**\n   * Register a DocIdTracker for the scope of the query, to determine the last fully-searched\n   * doc ID after early termination.\n   */\n  void registerDocIdTracker(DocIdTracker docIdTracker);\n\n  /**\n   * Return client ID of query.\n   */\n  String getClientId();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/termination/QueryTimeoutFactory.java",
    "content": "package com.twitter.search.common.search.termination;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\n\npublic class QueryTimeoutFactory {\n  /**\n   * Creates a QueryTimeout instance for a given EarlybirdRequest and TerminationTracker, if the\n   * required conditions for leaf-level timeout checking are met. Returns null otherwise.\n   *\n   * The conditions are:\n   *   1) CollectorTerminationParams.isEnforceQueryTimeout()\n   *   2) CollectorTerminationParams.isSetTimeoutMs()\n   */\n  public QueryTimeout createQueryTimeout(\n      EarlybirdRequest request,\n      TerminationTracker tracker,\n      Clock clock) {\n    if (tracker != null\n        && request != null\n        && request.isSetSearchQuery()\n        && request.getSearchQuery().isSetCollectorParams()\n        && request.getSearchQuery().getCollectorParams().isSetTerminationParams()\n        && request.getSearchQuery().getCollectorParams().getTerminationParams()\n            .isEnforceQueryTimeout()\n        && request.getSearchQuery().getCollectorParams().getTerminationParams()\n            .isSetTimeoutMs()) {\n      return new QueryTimeoutImpl(request.getClientId(), tracker, clock);\n    } else {\n      return null;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/termination/QueryTimeoutImpl.java",
    "content": "package com.twitter.search.common.search.termination;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.search.DocIdTracker;\nimport com.twitter.search.common.search.EarlyTerminationState;\nimport com.twitter.search.common.search.TerminationTracker;\n\n/**\n * QueryTimeoutImpl provides a method for early termination of queries based on time.\n */\npublic class QueryTimeoutImpl implements QueryTimeout {\n  private final String clientId;\n  private final TerminationTracker tracker;\n  private final Clock clock;\n\n  private final SearchRateCounter shouldTerminateCounter;\n\n  public QueryTimeoutImpl(String clientId, TerminationTracker tracker, Clock clock) {\n    this.clientId = Preconditions.checkNotNull(clientId);\n    this.tracker = Preconditions.checkNotNull(tracker);\n    this.clock = Preconditions.checkNotNull(clock);\n    shouldTerminateCounter =\n        SearchRateCounter.export(\"query_timeout_should_terminate_\" + clientId);\n  }\n\n  /**\n   * Returns true when the clock's time has met or exceeded the tracker's timeout end.\n   */\n  public boolean shouldExit() {\n    if (clock.nowMillis() >= tracker.getTimeoutEndTimeWithReservation()) {\n      tracker.setEarlyTerminationState(EarlyTerminationState.TERMINATED_TIME_OUT_EXCEEDED);\n      shouldTerminateCounter.increment();\n      return true;\n    }\n    return false;\n  }\n\n  @Override\n  public void registerDocIdTracker(DocIdTracker docIdTracker) {\n    tracker.addDocIdTracker(docIdTracker);\n  }\n\n  @Override\n  public String getClientId() {\n    return clientId;\n  }\n\n  @Override\n  public int hashCode() {\n    return clientId.hashCode() * 13 + tracker.hashCode();\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof QueryTimeoutImpl)) {\n      return false;\n    }\n\n    QueryTimeoutImpl queryTimeout = QueryTimeoutImpl.class.cast(obj);\n    return clientId.equals(queryTimeout.clientId) && tracker.equals(queryTimeout.tracker);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/termination/TerminationQuery.java",
    "content": "package com.twitter.search.common.search.termination;\n\nimport java.io.IOException;\nimport java.util.Arrays;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.IndexReader;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\n/**\n * Query implementation that can timeout and return non-exhaustive results.\n */\npublic class TerminationQuery extends Query {\n  private final Query inner;\n  private final QueryTimeout timeout;\n\n  public TerminationQuery(Query inner, QueryTimeout timeout) {\n    this.inner = Preconditions.checkNotNull(inner);\n    this.timeout = Preconditions.checkNotNull(timeout);\n  }\n\n  @Override\n  public Weight createWeight(\n      IndexSearcher searcher, ScoreMode scoreMode, float boost) throws IOException {\n    Weight innerWeight = inner.createWeight(searcher, scoreMode, boost);\n    return new TerminationQueryWeight(this, innerWeight, timeout);\n  }\n\n  @Override\n  public Query rewrite(IndexReader reader) throws IOException {\n    Query rewritten = inner.rewrite(reader);\n    if (rewritten != inner) {\n      return new TerminationQuery(rewritten, timeout);\n    }\n    return this;\n  }\n\n  public QueryTimeout getTimeout() {\n    return timeout;\n  }\n\n  @Override\n  public int hashCode() {\n    return Arrays.hashCode(new Object[] {inner, timeout});\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof TerminationQuery)) {\n      return false;\n    }\n\n    TerminationQuery terminationQuery = TerminationQuery.class.cast(obj);\n    return Arrays.equals(new Object[] {inner, timeout},\n                         new Object[] {terminationQuery.inner, terminationQuery.timeout});\n  }\n\n  @Override\n  public String toString(String field) {\n    return inner.toString(field);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/termination/TerminationQueryScorer.java",
    "content": "package com.twitter.search.common.search.termination;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.query.FilteredScorer;\nimport com.twitter.search.common.search.DocIdTracker;\n\n/**\n * Scorer implementation that adds termination support for an underlying query.\n * Meant to be used in conjunction with {@link TerminationQuery}.\n */\npublic class TerminationQueryScorer extends FilteredScorer implements DocIdTracker {\n  private final QueryTimeout timeout;\n  private int lastSearchedDocId = -1;\n\n  TerminationQueryScorer(Weight weight, Scorer inner, QueryTimeout timeout) {\n    super(weight, inner);\n    this.timeout = Preconditions.checkNotNull(timeout);\n    this.timeout.registerDocIdTracker(this);\n    SearchRateCounter.export(\n        timeout.getClientId() + \"_num_termination_query_scorers_created\").increment();\n  }\n\n  @Override\n  public DocIdSetIterator iterator() {\n    final DocIdSetIterator superDISI = super.iterator();\n    return new DocIdSetIterator() {\n      // lastSearchedDocId is the ID of the last document that was traversed in the posting list.\n      // docId is the current doc ID in this iterator. In most cases, lastSearchedDocId and docId\n      // will be equal. They will be different only if the query needed to be terminated based on\n      // the timeout. In that case, docId will be set to NO_MORE_DOCS, but lastSearchedDocId will\n      // still be set to the last document that was actually traversed.\n      private int docId = -1;\n\n      @Override\n      public int docID() {\n        return docId;\n      }\n\n      @Override\n      public int nextDoc() throws IOException {\n        if (docId == NO_MORE_DOCS) {\n          return NO_MORE_DOCS;\n        }\n\n        if (timeout.shouldExit()) {\n          docId = NO_MORE_DOCS;\n        } else {\n          docId = superDISI.nextDoc();\n          lastSearchedDocId = docId;\n        }\n        return docId;\n      }\n\n      @Override\n      public int advance(int target) throws IOException {\n        if (docId == NO_MORE_DOCS) {\n          return NO_MORE_DOCS;\n        }\n\n        if (target == NO_MORE_DOCS) {\n          docId = NO_MORE_DOCS;\n          lastSearchedDocId = docId;\n        } else if (timeout.shouldExit()) {\n          docId = NO_MORE_DOCS;\n        } else {\n          docId = superDISI.advance(target);\n          lastSearchedDocId = docId;\n        }\n        return docId;\n      }\n\n      @Override\n      public long cost() {\n        return superDISI.cost();\n      }\n    };\n  }\n\n  @Override\n  public int getCurrentDocId() {\n    return lastSearchedDocId;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/search/termination/TerminationQueryWeight.java",
    "content": "package com.twitter.search.common.search.termination;\n\nimport java.io.IOException;\nimport java.util.Set;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.Explanation;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.Weight;\n\n/**\n * Weight implementation that adds termination support for an underlying query.\n * Meant to be used in conjunction with {@link TerminationQuery}.\n */\npublic class TerminationQueryWeight extends Weight {\n  private final Weight inner;\n  private final QueryTimeout timeout;\n\n  TerminationQueryWeight(TerminationQuery query, Weight inner, QueryTimeout timeout) {\n    super(query);\n    this.inner = inner;\n    this.timeout = Preconditions.checkNotNull(timeout);\n  }\n\n  @Override\n  public Explanation explain(LeafReaderContext context, int doc)\n      throws IOException {\n    return inner.explain(context, doc);\n  }\n\n  @Override\n  public Scorer scorer(LeafReaderContext context) throws IOException {\n    Scorer innerScorer = inner.scorer(context);\n    if (innerScorer != null) {\n      return new TerminationQueryScorer(this, innerScorer, timeout);\n    }\n\n    return null;\n  }\n\n  @Override\n  public void extractTerms(Set<Term> terms) {\n    inner.extractTerms(terms);\n  }\n\n  @Override\n  public boolean isCacheable(LeafReaderContext ctx) {\n    return inner.isCacheable(ctx);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/earlybird/BUILD",
    "content": "java_library(\n    sources = [\"**/*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/com/twitter/elephantbird:core\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/collections\",\n        \"src/java/com/twitter/search/common/encoding/features\",\n        \"src/java/com/twitter/search/common/logging\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/relevance:ranking\",\n        \"src/java/com/twitter/search/common/relevance:text\",\n        \"src/java/com/twitter/search/common/relevance/features\",\n        \"src/java/com/twitter/search/common/runtime\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n        \"src/thrift/com/twitter/search/adaptive:adaptive-results-java\",\n        \"src/thrift/com/twitter/search/common:constants-java\",\n        \"src/thrift/com/twitter/search/common:indexing-java\",\n        \"src/thrift/com/twitter/search/common:query-java\",\n        \"src/thrift/com/twitter/search/common:ranking-java\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/earlybird/EarlybirdResponseMergeUtil.java",
    "content": "package com.twitter.search.common.util.earlybird;\n\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.concurrent.ExecutionException;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.cache.LoadingCache;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Lists;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchRankingMode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftTweetSource;\n\n/**\n * Utility methods to merge EarlybirdResponses.\n */\npublic final class EarlybirdResponseMergeUtil {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdResponseMergeUtil.class);\n\n  private static final String INVALID_RESPONSE_STATS_PREFIX = \"invalid_response_stats_\";\n\n  // Stats for invalid earlybird response\n  private static final ImmutableMap<EarlybirdResponseCode, SearchCounter> ERROR_EXCEPTIONS;\n\n  public static final SearchCounter NULL_RESPONSE_COUNTER =\n      SearchCounter.export(INVALID_RESPONSE_STATS_PREFIX + \"null_response\");\n  public static final SearchCounter SEARCH_RESULTS_NOT_SET_COUNTER =\n      SearchCounter.export(INVALID_RESPONSE_STATS_PREFIX + \"search_results_not_set\");\n  public static final SearchCounter SEARCH_RESULTS_WITH_RESULTS_NOT_SET_COUNTER =\n      SearchCounter.export(INVALID_RESPONSE_STATS_PREFIX + \"search_results_with_results_not_set\");\n  public static final SearchCounter MAX_SEARCHED_STATUS_ID_NOT_SET_COUNTER =\n      SearchCounter.export(INVALID_RESPONSE_STATS_PREFIX + \"max_searched_status_id_not_set\");\n  public static final SearchCounter MIN_SEARCHED_STATUS_ID_NOT_SET_COUNTER =\n      SearchCounter.export(INVALID_RESPONSE_STATS_PREFIX + \"min_searched_status_id_not_set\");\n\n  static {\n    ImmutableMap.Builder<EarlybirdResponseCode, SearchCounter> builder = ImmutableMap.builder();\n\n    for (EarlybirdResponseCode responseCode : EarlybirdResponseCode.values()) {\n      if (responseCode != EarlybirdResponseCode.SUCCESS) {\n        builder.put(responseCode, SearchCounter.export(\n            INVALID_RESPONSE_STATS_PREFIX + responseCode.name().toLowerCase()));\n      }\n    }\n\n    ERROR_EXCEPTIONS = builder.build();\n  }\n\n  private EarlybirdResponseMergeUtil() {\n  }\n\n  /**\n   * Tags the results in the given EarlybirdResponse with the given ThriftTweetSource and adds them\n   * to the given list of results.\n   *\n   * @param results The list of results to which the new results will be added.\n   * @param earlybirdResponse The EarlybirdResponse whose results will be added to {@code results}.\n   * @param tweetSource The ThriftTweetSource that will be used to mark all results in\n   *                    {@code earlybirdResponse}.\n   * @return {@code false} if {@code earlybirdResponse} is {@code null} or doesn't have any results;\n   *         {@code true}, otherwise.\n   */\n  public static boolean addResultsToList(List<ThriftSearchResult> results,\n                                         EarlybirdResponse earlybirdResponse,\n                                         ThriftTweetSource tweetSource) {\n    return EarlybirdResponseUtil.hasResults(earlybirdResponse)\n      && addResultsToList(results,\n                          earlybirdResponse.getSearchResults().getResults(),\n                          tweetSource);\n  }\n\n  /**\n   * Tags the results in the given list with the given ThriftTweetSource and adds them to the given\n   * list of results.\n   *\n   * @param results The list of results to which the new results will be added.\n   * @param resultsToAdd The list of results to add.\n   * @param tweetSource The ThriftTweetSource that will be used to mark all results in\n   *                    {@code resultsToAdd}.\n   * @return {@code false} if {@code results} is {@code null} or if {@code resultsToAdd} is\n   *         {@code null} or doesn't have any results; {@code true}, otherwise.\n   */\n  public static boolean addResultsToList(List<ThriftSearchResult> results,\n                                         List<ThriftSearchResult> resultsToAdd,\n                                         ThriftTweetSource tweetSource) {\n    Preconditions.checkNotNull(results);\n    if ((resultsToAdd == null) || resultsToAdd.isEmpty()) {\n      return false;\n    }\n\n    markWithTweetSource(resultsToAdd, tweetSource);\n\n    results.addAll(resultsToAdd);\n    return true;\n  }\n\n  /**\n   * Distinct the input ThriftSearchResult by its status id. If there are duplicates, the first\n   * instance of the duplicates is returned in the distinct result. If the distinct result is the\n   * same as the input result, the initial input result is returned; otherwise, the distinct result\n   * is returned.\n   *\n   * @param results the input result\n   * @param dupsStats stats counter track duplicates source\n   * @return the input result if there is no duplicate; otherwise, return the distinct result\n   */\n  public static List<ThriftSearchResult> distinctByStatusId(\n      List<ThriftSearchResult> results,\n      LoadingCache<Pair<ThriftTweetSource, ThriftTweetSource>, SearchCounter> dupsStats) {\n    Map<Long, ThriftTweetSource> seenStatusIdToSourceMap = new HashMap<>();\n    List<ThriftSearchResult> distinctResults = Lists.newArrayListWithCapacity(results.size());\n    for (ThriftSearchResult result : results)  {\n      if (seenStatusIdToSourceMap.containsKey(result.getId())) {\n        ThriftTweetSource source1 = seenStatusIdToSourceMap.get(result.getId());\n        ThriftTweetSource source2 = result.getTweetSource();\n        if (source1 != null && source2 != null) {\n          try {\n            dupsStats.get(Pair.of(source1, source2)).increment();\n          } catch (ExecutionException e) {\n            LOG.warn(\"Could not increment stat for duplicate results from clusters \" + source1\n                + \" and \" + source2, e);\n          }\n        }\n      } else {\n        distinctResults.add(result);\n        seenStatusIdToSourceMap.put(result.getId(), result.getTweetSource());\n      }\n    }\n    return results.size() == distinctResults.size() ? results : distinctResults;\n  }\n\n  /**\n   * Tags the given results with the given ThriftTweetSource.\n   *\n   * @param results The results to be tagged.\n   * @param tweetSource The ThriftTweetSource to be used to tag the given results.\n   */\n  public static void markWithTweetSource(List<ThriftSearchResult> results,\n                                         ThriftTweetSource tweetSource) {\n    if (results != null) {\n      for (ThriftSearchResult result : results) {\n        result.setTweetSource(tweetSource);\n      }\n    }\n  }\n\n  /**\n   * Check if an Earlybird response is valid\n   */\n  public static boolean isValidResponse(final EarlybirdResponse response) {\n    if (response == null) {\n      NULL_RESPONSE_COUNTER.increment();\n      return false;\n    }\n\n    if (!EarlybirdResponseUtil.isSuccessfulResponse(response)) {\n      return false;\n    }\n\n    if (!response.isSetSearchResults()) {\n      SEARCH_RESULTS_NOT_SET_COUNTER.increment();\n      return true;\n    }\n\n    if (!response.getSearchResults().isSetResults()) {\n      SEARCH_RESULTS_WITH_RESULTS_NOT_SET_COUNTER.increment();\n    }\n\n    // In earlybird, when earlybird terminated, e.g., time out, complex queries - we don't set the\n    // min/max  searched status id.\n    boolean isEarlyTerminated = response.isSetEarlyTerminationInfo()\n        && response.getEarlyTerminationInfo().isEarlyTerminated();\n\n    if (!isEarlyTerminated && !response.getSearchResults().isSetMinSearchedStatusID()) {\n      MIN_SEARCHED_STATUS_ID_NOT_SET_COUNTER.increment();\n    }\n\n    if (!isEarlyTerminated && !response.getSearchResults().isSetMaxSearchedStatusID()) {\n      MAX_SEARCHED_STATUS_ID_NOT_SET_COUNTER.increment();\n    }\n\n    return true;\n  }\n\n  /**\n   * For invalid successful Earlybird Response, return a failed response with debug msg.\n   */\n  public static EarlybirdResponse transformInvalidResponse(final EarlybirdResponse response,\n                                                           final String debugMsg) {\n    if (response == null) {\n      return failedEarlybirdResponse(EarlybirdResponseCode.PERSISTENT_ERROR,\n          debugMsg + \", msg: null response from downstream\");\n    }\n    Preconditions.checkState(response.getResponseCode() != EarlybirdResponseCode.SUCCESS);\n\n    EarlybirdResponseCode newResponseCode;\n    EarlybirdResponseCode responseCode = response.getResponseCode();\n    switch (responseCode) {\n      case TIER_SKIPPED:\n        ERROR_EXCEPTIONS.get(responseCode).increment();\n        return response;\n      case REQUEST_BLOCKED_ERROR:\n      case CLIENT_ERROR:\n      case SERVER_TIMEOUT_ERROR:\n      case QUOTA_EXCEEDED_ERROR:\n      case CLIENT_CANCEL_ERROR:\n      case TOO_MANY_PARTITIONS_FAILED_ERROR:\n        ERROR_EXCEPTIONS.get(responseCode).increment();\n        newResponseCode = responseCode;\n        break;\n      default:\n        ERROR_EXCEPTIONS.get(responseCode).increment();\n        newResponseCode = EarlybirdResponseCode.PERSISTENT_ERROR;\n    }\n\n    String newDebugMsg = debugMsg + \", downstream response code: \" + responseCode\n      + (response.isSetDebugString() ? \", downstream msg: \" + response.getDebugString() : \"\");\n\n\n    return failedEarlybirdResponse(newResponseCode, newDebugMsg);\n  }\n\n  /**\n   * Create a new EarlybirdResponse with debug msg\n   */\n  public static EarlybirdResponse failedEarlybirdResponse(final EarlybirdResponseCode responseCode,\n                                                          final String debugMsg) {\n    EarlybirdResponse failedResponse = new EarlybirdResponse();\n    failedResponse.setResponseCode(responseCode);\n    failedResponse.setDebugString(debugMsg);\n    return failedResponse;\n  }\n\n  /**\n   * Returns the number of results to keep as part of merge-collection. Recency mode should ignore\n   * relevance options. In particular, the flag returnAllResults inside relevance options.\n   */\n  public static int computeNumResultsToKeep(EarlybirdRequest request) {\n    ThriftSearchQuery searchQuery = request.getSearchQuery();\n\n    if (searchQuery.getRankingMode() != ThriftSearchRankingMode.RECENCY\n        && searchQuery.isSetRelevanceOptions()\n        && searchQuery.getRelevanceOptions().isReturnAllResults()) {\n      return Integer.MAX_VALUE;\n    }\n\n    if (request.isSetNumResultsToReturnAtRoot()) {\n      return request.getNumResultsToReturnAtRoot();\n    }\n\n    if (searchQuery.isSetCollectorParams()) {\n      return searchQuery.getCollectorParams().getNumResultsToReturn();\n    }\n\n    return searchQuery.getNumResults();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/earlybird/EarlybirdResponseUtil.java",
    "content": "package com.twitter.search.common.util.earlybird;\n\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Set;\nimport java.util.stream.Collectors;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.adaptive.adaptive_results.thriftjava.TweetSource;\nimport com.twitter.search.common.logging.ObjectKey;\nimport com.twitter.search.common.runtime.DebugManager;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird.thrift.ThriftTweetSource;\n\n/** Utility methods that work on EarlybirdResponses. */\npublic final class EarlybirdResponseUtil {\n  private EarlybirdResponseUtil() {\n  }\n\n  /**\n   * Returns the results in the given EarlybirdResponse.\n   *\n   * @param response The EarlybirdResponse.\n   * @return The results in the given EarlybirdResponse, or {@code null} if the response is\n   *         {@code null} or the results are not set.\n   */\n  public static ThriftSearchResults getResults(EarlybirdResponse response) {\n    if ((response == null) || !response.isSetSearchResults()) {\n      return null;\n    }\n\n    return response.getSearchResults();\n  }\n\n  /**\n   * Determines if the given EarlybirdResponse has results.\n   *\n   * @param response The EarlybirdResponse.\n   * @return {@code true} if the given EarlybirdResponse has results; {@code false} otherwise.\n   */\n  public static boolean hasResults(EarlybirdResponse response) {\n    ThriftSearchResults results = getResults(response);\n    return (results != null) && results.isSetResults() && !results.getResults().isEmpty();\n  }\n\n  /**\n   * Returns the number of results in the given EarlybirdResponse.\n   *\n   * @param response The EarlybirdResponse.\n   * @return The number of results in the given EarlybirdResponse.\n   */\n  public static int getNumResults(EarlybirdResponse response) {\n    return hasResults(response) ? response.getSearchResults().getResultsSize() : 0;\n  }\n\n  /**\n   * Determines the response is early-terminated.\n   *\n   * @param response The EarlybirdResponse.\n   * @return {@code true} if the response is early-terminated; {@code false} otherwise.\n   */\n  public static boolean isEarlyTerminated(EarlybirdResponse response) {\n    Preconditions.checkNotNull(response);\n    return response.isSetEarlyTerminationInfo()\n        && response.getEarlyTerminationInfo().isEarlyTerminated();\n  }\n\n  /**\n   * Returns if the response should be considered failed for purposes of stats and logging.\n   */\n  public static boolean responseConsideredFailed(EarlybirdResponseCode code) {\n    return code != EarlybirdResponseCode.SUCCESS\n        && code != EarlybirdResponseCode.REQUEST_BLOCKED_ERROR\n        && code != EarlybirdResponseCode.TIER_SKIPPED;\n  }\n\n  /**\n   * Extract results from Earlybird response.\n   */\n  public static List<ThriftSearchResult> extractResultsFromEarlybirdResponse(\n      EarlybirdResponse response) {\n    return hasResults(response)\n        ? response.getSearchResults().getResults() : Collections.emptyList();\n  }\n\n  /**\n   * Log the Earlybird response as a candidate source.\n   */\n  public static EarlybirdResponse debugLogAsCandidateSource(\n      EarlybirdResponse response, TweetSource tweetSource) {\n    List<ThriftSearchResult> results = extractResultsFromEarlybirdResponse(response);\n    debugLogAsCandidateSourceHelper(results, tweetSource);\n    return response;\n  }\n\n  /**\n   * Log a list of ThriftSearchResult as a candidate source.\n   */\n  public static List<ThriftSearchResult> debugLogAsCandidateSource(\n      List<ThriftSearchResult> results, TweetSource tweetSource) {\n    debugLogAsCandidateSourceHelper(results, tweetSource);\n    return results;\n  }\n\n  private static void debugLogAsCandidateSourceHelper(\n      List<ThriftSearchResult> results, TweetSource tweetSource) {\n    // debug message for Earlybird relevance candidate source\n    List<String> strIds = results\n        .stream()\n        .map(ThriftSearchResult::getId)\n        .map(Object::toString)\n        .collect(Collectors.toList());\n    ObjectKey debugMsgKey = ObjectKey.createTweetCandidateSourceKey(\n        tweetSource.name());\n    DebugManager.perObjectBasic(\n        debugMsgKey,\n        String.format(\"[%s][%s] results: %s\", debugMsgKey.getType(), debugMsgKey.getId(), strIds));\n  }\n\n  /**\n   * Extract the real time response from an existing response\n   */\n  public static EarlybirdResponse extractRealtimeResponse(EarlybirdResponse response) {\n    EarlybirdResponse realtimeResponse = response.deepCopy();\n    if (EarlybirdResponseUtil.hasResults(response)) {\n      List<ThriftSearchResult> realtimeResults = realtimeResponse.getSearchResults().getResults();\n      realtimeResults.clear();\n      for (ThriftSearchResult result : response.getSearchResults().getResults()) {\n        if (result.getTweetSource() == ThriftTweetSource.REALTIME_CLUSTER) {\n          realtimeResults.add(result);\n        }\n      }\n    }\n\n    return realtimeResponse;\n  }\n\n  /**\n   * Returns an EarlybirdResponse that should be returned by roots when a tier was skipped.\n   *\n   * @param minId The minSearchedStatusID to be set on the response.\n   * @param maxId The maxSearchedStatusID to be set on the response.\n   * @param debugMsg The debug message to be set on the response.\n   * @return A response that should be returned by roots when a tier was skipped.\n   */\n  public static EarlybirdResponse tierSkippedRootResponse(long minId, long maxId, String debugMsg) {\n    return new EarlybirdResponse(EarlybirdResponseCode.SUCCESS, 0)\n      .setSearchResults(new ThriftSearchResults()\n                        .setResults(new ArrayList<>())\n                        .setMinSearchedStatusID(minId)\n                        .setMaxSearchedStatusID(maxId))\n      .setDebugString(debugMsg);\n  }\n\n  /**\n   * Determines if the given response is a success response.\n   *\n   * A response is considered successful if it's not null and has either a SUCCESS, TIER_SKIPPED or\n   * REQUEST_BLOCKED_ERROR response code.\n   *\n   * @param response The response to check.\n   * @return Whether the given response is successful or not.\n   */\n  public static boolean isSuccessfulResponse(EarlybirdResponse response) {\n    return response != null\n      && (response.getResponseCode() == EarlybirdResponseCode.SUCCESS\n          || response.getResponseCode() == EarlybirdResponseCode.TIER_SKIPPED\n          || response.getResponseCode() == EarlybirdResponseCode.REQUEST_BLOCKED_ERROR);\n  }\n\n  /**\n   * Finds all unexpected nullcast statuses within the given result. A nullcast status is\n   * unexpected iff:\n   *   1. the tweet is a nullcast tweet.\n   *   2. the tweet is NOT explicitly requested with {@link ThriftSearchQuery#searchStatusIds}\n   */\n  public static Set<Long> findUnexpectedNullcastStatusIds(\n      ThriftSearchResults thriftSearchResults, EarlybirdRequest request) {\n    Set<Long> statusIds = new HashSet<>();\n    for (ThriftSearchResult result : thriftSearchResults.getResults()) {\n      if (resultIsNullcast(result) && !isSearchStatusId(request, result.getId())) {\n        statusIds.add(result.getId());\n      }\n    }\n    return statusIds;\n  }\n\n  private static boolean isSearchStatusId(EarlybirdRequest request, long id) {\n    return request.getSearchQuery().isSetSearchStatusIds()\n        && request.getSearchQuery().getSearchStatusIds().contains(id);\n  }\n\n  private static boolean resultIsNullcast(ThriftSearchResult result) {\n    return result.isSetMetadata() && result.getMetadata().isIsNullcast();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/earlybird/FacetsResultsUtils.java",
    "content": "package com.twitter.search.common.util.earlybird;\n\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.Comparator;\nimport java.util.HashMap;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\n\nimport com.google.common.collect.Lists;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.logging.DebugMessageBuilder;\nimport com.twitter.search.common.ranking.thriftjava.ThriftFacetFinalSortOrder;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCount;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCountMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftFacetFieldRequest;\nimport com.twitter.search.earlybird.thrift.ThriftFacetFieldResults;\nimport com.twitter.search.earlybird.thrift.ThriftFacetRankingMode;\nimport com.twitter.search.earlybird.thrift.ThriftFacetRequest;\nimport com.twitter.search.earlybird.thrift.ThriftFacetResults;\nimport com.twitter.search.earlybird.thrift.ThriftTermResults;\n\n/**\n * A utility class to provide some functions for facets results processing.\n */\npublic final class FacetsResultsUtils {\n\n  private static final Logger LOG = LoggerFactory.getLogger(FacetsResultsUtils.class);\n\n  private FacetsResultsUtils() {\n  }\n\n  public static class FacetFieldInfo {\n    public ThriftFacetFieldRequest fieldRequest;\n    public int totalCounts;\n    public Map<String, ThriftFacetCount> topFacets;\n    public List<Map.Entry<ThriftLanguage, Double>> languageHistogramEntries = Lists.newLinkedList();\n  }\n\n  // Only return top languages in the language histogram which sum up to at least this much\n  // ratio, here we get first 80 percentiles.\n  public static final double MIN_PERCENTAGE_SUM_REQUIRED = 0.8;\n  // if a language ratio is over this number, we already return.\n  public static final double MIN_PERCENTAGE = 0.01;\n\n  /**\n   * Prepare facet fields with empty entries and check if we need termStats for filtering.\n   * Returns true if termStats filtering is needed (thus the termStats servie call).\n   * @param facetRequest The related facet request.\n   * @param facetFieldInfoMap The facet field info map to fill, a map from facet type to the facet\n   * fiels results info.\n   * @return {@code true} if termstats request is needed afterwards.\n   */\n  public static boolean prepareFieldInfoMap(\n      ThriftFacetRequest facetRequest,\n      final Map<String, FacetsResultsUtils.FacetFieldInfo> facetFieldInfoMap) {\n    boolean termStatsFilteringMode = false;\n\n    for (ThriftFacetFieldRequest fieldRequest : facetRequest.getFacetFields()) {\n      FacetsResultsUtils.FacetFieldInfo info = new FacetsResultsUtils.FacetFieldInfo();\n      info.fieldRequest = fieldRequest;\n      facetFieldInfoMap.put(fieldRequest.getFieldName(), info);\n      if (fieldRequest.getRankingMode() == ThriftFacetRankingMode.FILTER_WITH_TERM_STATISTICS) {\n        termStatsFilteringMode = true;\n      }\n    }\n\n    return termStatsFilteringMode;\n  }\n\n  /**\n   * Extract information from one ThriftFacetResults into facetFieldInfoMap and userIDWhitelist.\n   * @param facetResults Related facets results.\n   * @param facetFieldInfoMap The facets field info map to fill, a map from facet type to the facet\n   * fiels results info.\n   * @param userIDWhitelist The user whitelist to fill.\n   */\n  public static void fillFacetFieldInfo(\n      final ThriftFacetResults facetResults,\n      final Map<String, FacetsResultsUtils.FacetFieldInfo> facetFieldInfoMap,\n      final Set<Long> userIDWhitelist) {\n\n    for (String facetField : facetResults.getFacetFields().keySet()) {\n      FacetsResultsUtils.FacetFieldInfo info = facetFieldInfoMap.get(facetField);\n      if (info.topFacets == null) {\n        info.topFacets = new HashMap<>();\n      }\n\n      ThriftFacetFieldResults results = facetResults.getFacetFields().get(facetField);\n      if (results.isSetLanguageHistogram()) {\n        info.languageHistogramEntries.addAll(results.getLanguageHistogram().entrySet());\n      }\n      for (ThriftFacetCount newCount : results.getTopFacets()) {\n        ThriftFacetCount resultCount = info.topFacets.get(newCount.facetLabel);\n        if (resultCount == null) {\n          info.topFacets.put(newCount.facetLabel, new ThriftFacetCount(newCount));\n        } else {\n          resultCount.setFacetCount(resultCount.facetCount + newCount.facetCount);\n          resultCount.setSimpleCount(resultCount.simpleCount + newCount.simpleCount);\n          resultCount.setWeightedCount(resultCount.weightedCount + newCount.weightedCount);\n          resultCount.setPenaltyCount(resultCount.penaltyCount + newCount.penaltyCount);\n          //  this could pass the old metadata object back or a new merged one.\n          resultCount.setMetadata(\n                  mergeFacetMetadata(resultCount.getMetadata(), newCount.getMetadata(),\n                                     userIDWhitelist));\n        }\n      }\n      info.totalCounts += results.totalCount;\n    }\n  }\n\n  /**\n   * Merge a metadata into an existing one.\n   * @param baseMetadata the metadata to merge into.\n   * @param metadataUpdate the new metadata to merge.\n   * @param userIDWhitelist user id whitelist to filter user id with.\n   * @return The updated metadata.\n   */\n  public static ThriftFacetCountMetadata mergeFacetMetadata(\n          final ThriftFacetCountMetadata baseMetadata,\n          final ThriftFacetCountMetadata metadataUpdate,\n          final Set<Long> userIDWhitelist) {\n    ThriftFacetCountMetadata mergedMetadata = baseMetadata;\n    if (metadataUpdate != null) {\n      String mergedExplanation = null;\n      if (mergedMetadata != null) {\n        if (mergedMetadata.maxTweepCred < metadataUpdate.maxTweepCred) {\n          mergedMetadata.setMaxTweepCred(metadataUpdate.maxTweepCred);\n        }\n\n        if (mergedMetadata.isSetExplanation()) {\n          mergedExplanation = mergedMetadata.getExplanation();\n          if (metadataUpdate.isSetExplanation()) {\n            mergedExplanation += \"\\n\" + metadataUpdate.getExplanation();\n          }\n        } else if (metadataUpdate.isSetExplanation()) {\n          mergedExplanation = metadataUpdate.getExplanation();\n        }\n\n        if (mergedMetadata.getStatusId() == -1) {\n          if (LOG.isDebugEnabled()) {\n            LOG.debug(\"status id in facet count metadata is -1: \" + mergedMetadata);\n          }\n          mergedMetadata = metadataUpdate;\n        } else if (metadataUpdate.getStatusId() != -1\n            && metadataUpdate.getStatusId() < mergedMetadata.getStatusId()) {\n          // keep the oldest tweet, ie. the lowest status ID\n          mergedMetadata = metadataUpdate;\n        } else if (metadataUpdate.getStatusId() == mergedMetadata.getStatusId()) {\n          if (mergedMetadata.getTwitterUserId() == -1) {\n            // in this case we didn't find the user in a previous partition yet\n            // only update the user if the status id matches\n            mergedMetadata.setTwitterUserId(metadataUpdate.getTwitterUserId());\n            mergedMetadata.setDontFilterUser(metadataUpdate.isDontFilterUser());\n          }\n          if (!mergedMetadata.isSetStatusLanguage()) {\n            mergedMetadata.setStatusLanguage(metadataUpdate.getStatusLanguage());\n          }\n        }\n        if (!mergedMetadata.isSetNativePhotoUrl() && metadataUpdate.isSetNativePhotoUrl()) {\n          mergedMetadata.setNativePhotoUrl(metadataUpdate.getNativePhotoUrl());\n        }\n      } else {\n        mergedMetadata = metadataUpdate;\n      }\n\n      // this will not set an explanation if neither oldMetadata nor metadataUpdate\n      // had an explanation\n      if (mergedExplanation != null) {\n        mergedMetadata.setExplanation(mergedExplanation);\n      }\n\n      if (userIDWhitelist != null) {\n        // result must not be null now because of the if above\n        if (mergedMetadata.getTwitterUserId() != -1 && !mergedMetadata.isDontFilterUser()) {\n          mergedMetadata.setDontFilterUser(\n              userIDWhitelist.contains(mergedMetadata.getTwitterUserId()));\n        }\n      }\n    }\n\n    return mergedMetadata;\n  }\n\n  /**\n   * Appends all twimg results to the image results. Optionally resorts the image results if\n   * a comparator is passed in.\n   * Also computes the sums of totalCount, totalScore, totalPenalty.\n   */\n  public static void mergeTwimgResults(ThriftFacetResults facetResults,\n                                       Comparator<ThriftFacetCount> optionalSortComparator) {\n    if (facetResults == null || !facetResults.isSetFacetFields()) {\n      return;\n    }\n\n    ThriftFacetFieldResults imageResults =\n        facetResults.getFacetFields().get(EarlybirdFieldConstant.IMAGES_FACET);\n    ThriftFacetFieldResults twimgResults =\n        facetResults.getFacetFields().remove(EarlybirdFieldConstant.TWIMG_FACET);\n    if (imageResults == null) {\n      if (twimgResults != null) {\n        facetResults.getFacetFields().put(EarlybirdFieldConstant.IMAGES_FACET, twimgResults);\n      }\n      return;\n    }\n\n    if (twimgResults != null) {\n      imageResults.setTotalCount(imageResults.getTotalCount() + twimgResults.getTotalCount());\n      imageResults.setTotalPenalty(imageResults.getTotalPenalty() + twimgResults.getTotalPenalty());\n      imageResults.setTotalScore(imageResults.getTotalScore() + twimgResults.getTotalScore());\n      for (ThriftFacetCount count : twimgResults.getTopFacets()) {\n        imageResults.addToTopFacets(count);\n      }\n      if (optionalSortComparator != null) {\n        Collections.sort(imageResults.topFacets, optionalSortComparator);\n      }\n    }\n  }\n\n  /**\n   * Dedup twimg facets.\n   *\n   * Twimg facet uses the status ID as the facet label, instead of the twimg URL, a.k.a.\n   * native photo URL. It is possible to have the same twimg URL appearing in two different\n   * facet label (RT style retweet? copy & paste the twimg URL?). Therefore, to dedup twimg\n   * facet correctly, we need to look at ThriftFacetCount.metadata.nativePhotoUrl\n   *\n   * @param dedupSet A set holding the native URLs from the twimg facetFieldResults. By having\n   *                 the caller passing in the set, it allows the caller to dedup the facet\n   *                 across different ThriftFacetFieldResults.\n   * @param facetFieldResults The twimg facet field results to be debupped\n   * @param debugMessageBuilder\n   */\n  public static void dedupTwimgFacet(Set<String> dedupSet,\n                                     ThriftFacetFieldResults facetFieldResults,\n                                     DebugMessageBuilder debugMessageBuilder) {\n    if (facetFieldResults == null || facetFieldResults.getTopFacets() == null) {\n      return;\n    }\n\n    Iterator<ThriftFacetCount> iterator = facetFieldResults.getTopFacetsIterator();\n\n    while (iterator.hasNext()) {\n      ThriftFacetCount count = iterator.next();\n      if (count.isSetMetadata() && count.getMetadata().isSetNativePhotoUrl()) {\n        String nativeUrl = count.getMetadata().getNativePhotoUrl();\n\n        if (dedupSet.contains(nativeUrl)) {\n          iterator.remove();\n          debugMessageBuilder.detailed(\"dedupTwimgFacet removed %s\", nativeUrl);\n        } else {\n          dedupSet.add(nativeUrl);\n        }\n      }\n    }\n\n\n  }\n\n  private static final class LanguageCount {\n    private final ThriftLanguage lang;\n    private final double count;\n    private LanguageCount(ThriftLanguage lang, double count) {\n      this.lang = lang;\n      this.count = count;\n    }\n  }\n\n  /**\n   * Calculate the top languages and store them in the results.\n   */\n  public static void fillTopLanguages(FacetsResultsUtils.FacetFieldInfo info,\n                                      final ThriftFacetFieldResults results) {\n    double sumForLanguage = 0.0;\n    double[] sums = new double[ThriftLanguage.values().length];\n    for (Map.Entry<ThriftLanguage, Double> entry : info.languageHistogramEntries) {\n      sumForLanguage += entry.getValue();\n      if (entry.getKey() == null) {\n        // EB might be setting null key for unknown language. SEARCH-1294\n        continue;\n      }\n      sums[entry.getKey().getValue()] += entry.getValue();\n    }\n    if (sumForLanguage == 0.0) {\n      return;\n    }\n    List<LanguageCount> langCounts = new ArrayList<>(ThriftLanguage.values().length);\n    for (int i = 0; i < sums.length; i++) {\n      if (sums[i] > 0.0) {\n        // ThriftLanguage.findByValue() might return null, which should fall back to UNKNOWN.\n        ThriftLanguage lang = ThriftLanguage.findByValue(i);\n        lang = lang == null ? ThriftLanguage.UNKNOWN : lang;\n        langCounts.add(new LanguageCount(lang, sums[i]));\n      }\n    }\n    Collections.sort(langCounts, (left, right) -> Double.compare(right.count, left.count));\n    double percentageSum = 0.0;\n    Map<ThriftLanguage, Double> languageHistogramMap =\n        new HashMap<>(langCounts.size());\n    int numAdded = 0;\n    for (LanguageCount langCount : langCounts) {\n      if (langCount.count == 0.0) {\n        break;\n      }\n      double percentage = langCount.count / sumForLanguage;\n      if (percentageSum > MIN_PERCENTAGE_SUM_REQUIRED\n          && percentage < MIN_PERCENTAGE && numAdded >= 3) {\n        break;\n      }\n      languageHistogramMap.put(langCount.lang, percentage);\n      percentageSum += percentage;\n      numAdded++;\n    }\n    results.setLanguageHistogram(languageHistogramMap);\n  }\n\n  /**\n   * Replace \"p.twimg.com/\" part of the native photo (twimg) URL with \"pbs.twimg.com/media/\".\n   * We need to do this because of blobstore and it's suppose to be a temporary measure. This\n   * code should be removed once we verified that all native photo URL being sent to Search\n   * are prefixed with \"pbs.twimg.com/media/\" and no native photo URL in our index contains\n   * \"p.twimg.com/\"\n   *\n   * Please see SEARCH-783 and EVENTS-539 for more details.\n   *\n   * @param response response containing the facet results\n   */\n  public static void fixNativePhotoUrl(EarlybirdResponse response) {\n    if (response == null\n        || !response.isSetFacetResults()\n        || !response.getFacetResults().isSetFacetFields()) {\n      return;\n    }\n\n    for (Map.Entry<String, ThriftFacetFieldResults> facetMapEntry\n        : response.getFacetResults().getFacetFields().entrySet()) {\n      final String facetResultField = facetMapEntry.getKey();\n\n      if (EarlybirdFieldConstant.TWIMG_FACET.equals(facetResultField)\n          || EarlybirdFieldConstant.IMAGES_FACET.equals(facetResultField)) {\n        ThriftFacetFieldResults facetFieldResults = facetMapEntry.getValue();\n        for (ThriftFacetCount facetCount : facetFieldResults.getTopFacets()) {\n          replacePhotoUrl(facetCount.getMetadata());\n        }\n      }\n    }\n  }\n\n  /**\n   * Replace \"p.twimg.com/\" part of the native photo (twimg) URL with \"pbs.twimg.com/media/\".\n   * We need to do this because of blobstore and it's suppose to be a temporary measure. This\n   * code should be removed once we verified that all native photo URL being sent to Search\n   * are prefixed with \"pbs.twimg.com/media/\" and no native photo URL in our index contains\n   * \"p.twimg.com/\"\n   *\n   * Please see SEARCH-783 and EVENTS-539 for more details.\n   *\n   * @param termResultsCollection collection of ThriftTermResults containing the native photo URL\n   */\n  public static void fixNativePhotoUrl(Collection<ThriftTermResults> termResultsCollection) {\n    if (termResultsCollection == null) {\n      return;\n    }\n\n    for (ThriftTermResults termResults : termResultsCollection) {\n      if (!termResults.isSetMetadata()) {\n        continue;\n      }\n      replacePhotoUrl(termResults.getMetadata());\n    }\n  }\n\n  /**\n   * Helper function for fixNativePhotoUrl()\n   */\n  private static void replacePhotoUrl(ThriftFacetCountMetadata metadata) {\n    if (metadata != null\n        && metadata.isSetNativePhotoUrl()) {\n      String nativePhotoUrl = metadata.getNativePhotoUrl();\n      nativePhotoUrl = nativePhotoUrl.replace(\"://p.twimg.com/\", \"://pbs.twimg.com/media/\");\n      metadata.setNativePhotoUrl(nativePhotoUrl);\n    }\n  }\n\n  /**\n   * Deepcopy of an EarlybirdResponse without explanation\n   */\n  public static EarlybirdResponse deepCopyWithoutExplanation(EarlybirdResponse facetsResponse) {\n    if (facetsResponse == null) {\n      return null;\n    } else if (!facetsResponse.isSetFacetResults()\n        || facetsResponse.getFacetResults().getFacetFieldsSize() == 0) {\n      return facetsResponse.deepCopy();\n    }\n    EarlybirdResponse copy = facetsResponse.deepCopy();\n    for (Map.Entry<String, ThriftFacetFieldResults> entry\n        : copy.getFacetResults().getFacetFields().entrySet()) {\n      if (entry.getValue().getTopFacetsSize() > 0) {\n        for (ThriftFacetCount fc : entry.getValue().getTopFacets()) {\n          fc.getMetadata().unsetExplanation();\n        }\n      }\n    }\n    return copy;\n  }\n\n  /**\n   * Returns a comparator used to compare facet counts by calling\n   * getFacetCountComparator(ThriftFacetFinalSortOrder).  The sort order is determined by\n   * the facetRankingOptions on the facet request.\n   */\n  public static Comparator<ThriftFacetCount> getFacetCountComparator(\n      ThriftFacetRequest facetRequest) {\n\n    ThriftFacetFinalSortOrder sortOrder = ThriftFacetFinalSortOrder.SCORE;\n\n    if (facetRequest.isSetFacetRankingOptions()\n        && facetRequest.getFacetRankingOptions().isSetFinalSortOrder()) {\n      sortOrder = facetRequest.getFacetRankingOptions().getFinalSortOrder();\n    }\n\n    return getFacetCountComparator(sortOrder);\n  }\n\n  /**\n   * Returns a comparator using the specified order.\n   */\n  public static Comparator<ThriftFacetCount> getFacetCountComparator(\n      ThriftFacetFinalSortOrder sortOrder) {\n\n    switch (sortOrder) {\n      case SIMPLE_COUNT:   return SIMPLE_COUNT_COMPARATOR;\n      case SCORE:          return SCORE_COMPARATOR;\n      case CREATED_AT:     return CREATED_AT_COMPARATOR;\n      case WEIGHTED_COUNT: return WEIGHTED_COUNT_COMPARATOR;\n      default:             return SCORE_COMPARATOR;\n    }\n  }\n\n  private static final Comparator<ThriftFacetCount> SIMPLE_COUNT_COMPARATOR =\n      (count1, count2) -> {\n        if (count1.simpleCount > count2.simpleCount) {\n          return 1;\n        } else if (count1.simpleCount < count2.simpleCount) {\n          return -1;\n        }\n\n        return count1.facetLabel.compareTo(count2.facetLabel);\n      };\n\n  private static final Comparator<ThriftFacetCount> WEIGHTED_COUNT_COMPARATOR =\n      (count1, count2) -> {\n        if (count1.weightedCount > count2.weightedCount) {\n          return 1;\n        } else if (count1.weightedCount < count2.weightedCount) {\n          return -1;\n        }\n\n        return SIMPLE_COUNT_COMPARATOR.compare(count1, count2);\n      };\n\n  private static final Comparator<ThriftFacetCount> SCORE_COMPARATOR =\n      (count1, count2) -> {\n        if (count1.score > count2.score) {\n          return 1;\n        } else if (count1.score < count2.score) {\n          return -1;\n        }\n        return SIMPLE_COUNT_COMPARATOR.compare(count1, count2);\n      };\n\n  private static final Comparator<ThriftFacetCount> CREATED_AT_COMPARATOR =\n      (count1, count2) -> {\n        if (count1.isSetMetadata() && count1.getMetadata().isSetCreated_at()\n            && count2.isSetMetadata() && count2.getMetadata().isSetCreated_at()) {\n          // more recent items have higher created_at values\n          if (count1.getMetadata().getCreated_at() > count2.getMetadata().getCreated_at()) {\n            return 1;\n          } else if (count1.getMetadata().getCreated_at() < count2.getMetadata().getCreated_at()) {\n            return -1;\n          }\n        }\n\n        return SCORE_COMPARATOR.compare(count1, count2);\n      };\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/earlybird/ResponseMergerUtils.java",
    "content": "package com.twitter.search.common.util.earlybird;\n\nimport java.util.List;\nimport java.util.Set;\n\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Sets;\n\nimport com.twitter.search.common.query.thriftjava.EarlyTerminationInfo;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\n\npublic final class ResponseMergerUtils {\n\n  // Utility class, disallow instantiation.\n  private ResponseMergerUtils() {\n  }\n\n  /**\n   * Merges early termination infos from several earlybird responses.\n   *\n   * @param responses earlybird responses to merge the early termination infos from\n   * @return merged early termination info\n   */\n  public static EarlyTerminationInfo mergeEarlyTerminationInfo(List<EarlybirdResponse> responses) {\n    EarlyTerminationInfo etInfo = new EarlyTerminationInfo(false);\n    Set<String> etReasonSet = Sets.newHashSet();\n    // Fill in EarlyTerminationStatus\n    for (EarlybirdResponse ebResp : responses) {\n      if (ebResp.isSetEarlyTerminationInfo()\n          && ebResp.getEarlyTerminationInfo().isEarlyTerminated()) {\n        etInfo.setEarlyTerminated(true);\n        if (ebResp.getEarlyTerminationInfo().isSetEarlyTerminationReason()) {\n          etReasonSet.add(ebResp.getEarlyTerminationInfo().getEarlyTerminationReason());\n        }\n        if (ebResp.getEarlyTerminationInfo().isSetMergedEarlyTerminationReasons()) {\n          etReasonSet.addAll(ebResp.getEarlyTerminationInfo().getMergedEarlyTerminationReasons());\n        }\n      }\n    }\n    if (etInfo.isEarlyTerminated()) {\n      etInfo.setMergedEarlyTerminationReasons(Lists.newArrayList(etReasonSet));\n    }\n    return etInfo;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/earlybird/ResultsUtil.java",
    "content": "package com.twitter.search.common.util.earlybird;\n\nimport java.util.Map;\n\nimport com.google.common.base.Function;\nimport com.google.common.collect.Iterables;\nimport com.google.common.collect.Maps;\n\n/**\n * Utility class used to help merging results.\n */\npublic final class ResultsUtil {\n  private ResultsUtil() { }\n\n  /**\n   * Aggregate a list of responses in the following way.\n   * 1. For each response, mapGetter can turn the response into a map.\n   * 2. Dump all entries from the above map into a \"total\" map, which accumulates entries from\n   *    all the responses.\n   */\n  public static <T, V> Map<T, Integer> aggregateCountMap(\n          Iterable<V> responses,\n          Function<V, Map<T, Integer>> mapGetter) {\n    Map<T, Integer> total = Maps.newHashMap();\n    for (Map<T, Integer> map : Iterables.transform(responses, mapGetter)) {\n      if (map != null) {\n        for (Map.Entry<T, Integer> entry : map.entrySet()) {\n          T key = entry.getKey();\n          total.put(key, total.containsKey(key)\n              ? total.get(key) + entry.getValue() : entry.getValue());\n        }\n      }\n    }\n    return total;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/earlybird/TermStatisticsUtil.java",
    "content": "package com.twitter.search.common.util.earlybird;\n\nimport java.util.concurrent.TimeUnit;\n\nimport com.twitter.search.earlybird.thrift.ThriftHistogramSettings;\n\n/**\n * A utility class to provide some functions for TermStatistics request processing\n */\npublic final class TermStatisticsUtil {\n\n  private static final org.slf4j.Logger LOG =\n      org.slf4j.LoggerFactory.getLogger(TermStatisticsUtil.class);\n\n  private TermStatisticsUtil() {\n  }\n\n  /**\n   * Determine the binsize base on settings in ThriftHistogramSettings.granularity\n   */\n  public static int determineBinSize(ThriftHistogramSettings histogramSettings) {\n    final int DEFAULT_BINSIZE = (int) TimeUnit.HOURS.toSeconds(1);\n    int binSize;\n    switch (histogramSettings.getGranularity()) {\n      case DAYS:\n        binSize = (int) TimeUnit.DAYS.toSeconds(1);\n        break;\n      case HOURS:\n        binSize = (int) TimeUnit.HOURS.toSeconds(1);\n        break;\n      case MINUTES:\n        binSize = (int) TimeUnit.MINUTES.toSeconds(1);\n        break;\n      case CUSTOM:\n        binSize = histogramSettings.isSetBinSizeInSeconds()\n                      ? histogramSettings.getBinSizeInSeconds()\n                      : DEFAULT_BINSIZE;\n        break;\n      default:\n        binSize = DEFAULT_BINSIZE;\n        LOG.warn(\"Unknown ThriftHistogramGranularityType {} using default binsize: {}\",\n                 histogramSettings.getGranularity(), DEFAULT_BINSIZE);\n    }\n\n    return binSize;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/earlybird/ThriftSearchQueryUtil.java",
    "content": "package com.twitter.search.common.util.earlybird;\n\nimport com.twitter.search.common.query.thriftjava.CollectorParams;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\n\n/**\n * Utility class from constructing ThriftSearchQuery.\n */\npublic final class ThriftSearchQueryUtil {\n  private ThriftSearchQueryUtil() { }\n\n  /**\n   * Convenience methods for constructing a ThriftSearchQuery.\n   */\n  public static ThriftSearchQuery newSearchQuery(String serializedQuery, int numResults) {\n    ThriftSearchQuery searchQuery = new ThriftSearchQuery();\n    searchQuery.setSerializedQuery(serializedQuery);\n    searchQuery.setCollectorParams(new CollectorParams().setNumResultsToReturn(numResults));\n    return searchQuery;\n  }\n\n  /** Determines if the given request was initiated by a logged in user. */\n  public static boolean requestInitiatedByLoggedInUser(EarlybirdRequest request) {\n    ThriftSearchQuery searchQuery = request.getSearchQuery();\n    return (searchQuery != null) && searchQuery.isSetSearcherId()\n      && (searchQuery.getSearcherId() > 0);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/earlybird/ThriftSearchResultUtil.java",
    "content": "package com.twitter.search.common.util.earlybird;\n\nimport java.util.List;\nimport java.util.Map;\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Function;\nimport com.google.common.base.Predicate;\nimport com.google.common.base.Predicates;\nimport com.google.common.collect.Iterables;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.relevance.ranking.ActionChain;\nimport com.twitter.search.common.relevance.ranking.filters.ExactDuplicateFilter;\nimport com.twitter.search.common.relevance.text.VisibleTokenRatioNormalizer;\nimport com.twitter.search.common.runtime.ActionChainDebugManager;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.earlybird.thrift.ThriftFacetFieldResults;\nimport com.twitter.search.earlybird.thrift.ThriftFacetResults;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultType;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird.thrift.ThriftTweetSource;\n\n/**\n * ThriftSearchResultUtil contains some simple static methods for constructing\n * ThriftSearchResult objects.\n */\npublic final class ThriftSearchResultUtil {\n  private ThriftSearchResultUtil() { }\n\n  private static final VisibleTokenRatioNormalizer NORMALIZER =\n      VisibleTokenRatioNormalizer.createInstance();\n\n  public static final Function<ThriftSearchResults, Map<ThriftLanguage, Integer>> LANG_MAP_GETTER =\n      searchResults -> searchResults.getLanguageHistogram();\n  public static final Function<ThriftSearchResults, Map<Long, Integer>> HIT_COUNTS_MAP_GETTER =\n      searchResults -> searchResults.getHitCounts();\n\n  // Some useful Predicates\n  public static final Predicate<ThriftSearchResult> IS_OFFENSIVE_TWEET =\n      result -> {\n        if (result != null && result.isSetMetadata()) {\n          ThriftSearchResultMetadata metadata = result.getMetadata();\n          return metadata.isIsOffensive();\n        } else {\n          return false;\n        }\n      };\n\n  public static final Predicate<ThriftSearchResult> IS_TOP_TWEET =\n      result -> result != null\n             && result.isSetMetadata()\n             && result.getMetadata().isSetResultType()\n             && result.getMetadata().getResultType() == ThriftSearchResultType.POPULAR;\n\n  public static final Predicate<ThriftSearchResult> FROM_FULL_ARCHIVE =\n      result -> result != null\n             && result.isSetTweetSource()\n             && result.getTweetSource() == ThriftTweetSource.FULL_ARCHIVE_CLUSTER;\n\n  public static final Predicate<ThriftSearchResult> IS_FULL_ARCHIVE_TOP_TWEET =\n      Predicates.and(FROM_FULL_ARCHIVE, IS_TOP_TWEET);\n\n  public static final Predicate<ThriftSearchResult> IS_NSFW_BY_ANY_MEANS_TWEET =\n          result -> {\n            if (result != null && result.isSetMetadata()) {\n              ThriftSearchResultMetadata metadata = result.getMetadata();\n              return metadata.isIsUserNSFW()\n                      || metadata.isIsOffensive()\n                      || metadata.getExtraMetadata().isIsSensitiveContent();\n            } else {\n              return false;\n            }\n          };\n\n  /**\n   * Returns the number of underlying ThriftSearchResult results.\n   */\n  public static int numResults(ThriftSearchResults results) {\n    if (results == null || !results.isSetResults()) {\n      return 0;\n    } else {\n      return results.getResultsSize();\n    }\n  }\n\n  /**\n   * Returns the list of tweet IDs in ThriftSearchResults.\n   * Returns null if there's no results.\n   */\n  @Nullable\n  public static List<Long> getTweetIds(ThriftSearchResults results) {\n    if (numResults(results) > 0) {\n      return getTweetIds(results.getResults());\n    } else {\n      return null;\n    }\n  }\n\n  /**\n   * Returns the list of tweet IDs in a list of ThriftSearchResult.\n   * Returns null if there's no results.\n   */\n  public static List<Long> getTweetIds(@Nullable List<ThriftSearchResult> results) {\n    if (results != null && results.size() > 0) {\n      return Lists.newArrayList(Iterables.transform(\n          results,\n          searchResult -> searchResult.getId()\n      ));\n    }\n    return null;\n  }\n\n  /**\n   * Given ThriftSearchResults, build a map from tweet ID to the tweets metadata.\n   */\n  public static Map<Long, ThriftSearchResultMetadata> getTweetMetadataMap(\n      Schema schema, ThriftSearchResults results) {\n    Map<Long, ThriftSearchResultMetadata> resultMap = Maps.newHashMap();\n    if (results == null || results.getResultsSize() == 0) {\n      return resultMap;\n    }\n    for (ThriftSearchResult searchResult : results.getResults()) {\n      resultMap.put(searchResult.getId(), searchResult.getMetadata());\n    }\n    return resultMap;\n  }\n\n  /**\n   * Return the total number of facet results in ThriftFacetResults, by summing up the number\n   * of facet results in each field.\n   */\n  public static int numFacetResults(ThriftFacetResults results) {\n    if (results == null || !results.isSetFacetFields()) {\n      return 0;\n    } else {\n      int numResults = 0;\n      for (ThriftFacetFieldResults field : results.getFacetFields().values()) {\n        if (field.isSetTopFacets()) {\n          numResults += field.topFacets.size();\n        }\n      }\n      return numResults;\n    }\n  }\n\n  /**\n   * Updates the search statistics on base, by adding the corresponding stats from delta.\n   */\n  public static void incrementCounts(ThriftSearchResults base,\n                                     ThriftSearchResults delta) {\n    if (delta.isSetNumHitsProcessed()) {\n      base.setNumHitsProcessed(base.getNumHitsProcessed() + delta.getNumHitsProcessed());\n    }\n    if (delta.isSetNumPartitionsEarlyTerminated() && delta.getNumPartitionsEarlyTerminated() > 0) {\n      // This currently used for merging results on a single earlybird, so we don't sum up all the\n      // counts, just set it to 1 if we see one that was early terminated.\n      base.setNumPartitionsEarlyTerminated(1);\n    }\n    if (delta.isSetMaxSearchedStatusID()) {\n      long deltaMax = delta.getMaxSearchedStatusID();\n      if (!base.isSetMaxSearchedStatusID() || deltaMax > base.getMaxSearchedStatusID()) {\n        base.setMaxSearchedStatusID(deltaMax);\n      }\n    }\n    if (delta.isSetMinSearchedStatusID()) {\n      long deltaMin = delta.getMinSearchedStatusID();\n      if (!base.isSetMinSearchedStatusID() || deltaMin < base.getMinSearchedStatusID()) {\n        base.setMinSearchedStatusID(deltaMin);\n      }\n    }\n    if (delta.isSetScore()) {\n      if (base.isSetScore()) {\n        base.setScore(base.getScore() + delta.getScore());\n      } else {\n        base.setScore(delta.getScore());\n      }\n    }\n  }\n\n  /**\n   * Removes the duplicates from the given list of results.\n   *\n   * @param results The list of ThriftSearchResults.\n   * @return The given list with duplicates removed.\n   */\n  public static List<ThriftSearchResult> removeDuplicates(List<ThriftSearchResult> results) {\n    ActionChain<ThriftSearchResult> filterChain =\n      ActionChainDebugManager\n        .<ThriftSearchResult>createActionChainBuilder(\"RemoveDuplicatesFilters\")\n        .appendActions(new ExactDuplicateFilter())\n        .build();\n    return filterChain.apply(results);\n  }\n\n  /**\n   * Returns ranking score from Earlybird shard-based ranking models if any, and 0 otherwise.\n   */\n  public static double getTweetScore(@Nullable ThriftSearchResult result) {\n    if (result == null || !result.isSetMetadata() || !result.getMetadata().isSetScore()) {\n      return 0.0;\n    }\n    return result.getMetadata().getScore();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/earlybird/ThriftSearchResultsRelevanceStatsUtil.java",
    "content": "package com.twitter.search.common.util.earlybird;\n\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultsRelevanceStats;\n\npublic final class ThriftSearchResultsRelevanceStatsUtil {\n  private ThriftSearchResultsRelevanceStatsUtil() { }\n\n  /**\n   * Adding ThriftSearchResultsRelevanceStats from one set of results onto a base set.\n   * Assumes all values are set on both of the inputs.\n   *\n   * @param base the stats to add to.\n   * @param delta the stats to be added.\n   */\n  public static void addRelevanceStats(ThriftSearchResultsRelevanceStats base,\n                                       ThriftSearchResultsRelevanceStats delta) {\n    base.setNumScored(base.getNumScored() + delta.getNumScored());\n    base.setNumSkipped(base.getNumSkipped() + delta.getNumSkipped());\n    base.setNumSkippedForAntiGaming(\n            base.getNumSkippedForAntiGaming() + delta.getNumSkippedForAntiGaming());\n    base.setNumSkippedForLowReputation(\n            base.getNumSkippedForLowReputation() + delta.getNumSkippedForLowReputation());\n    base.setNumSkippedForLowTextScore(\n            base.getNumSkippedForLowTextScore() + delta.getNumSkippedForLowTextScore());\n    base.setNumSkippedForSocialFilter(\n            base.getNumSkippedForSocialFilter() + delta.getNumSkippedForSocialFilter());\n    base.setNumSkippedForLowFinalScore(\n            base.getNumSkippedForLowFinalScore() + delta.getNumSkippedForLowFinalScore());\n    if (delta.getOldestScoredTweetAgeInSeconds() > base.getOldestScoredTweetAgeInSeconds()) {\n      base.setOldestScoredTweetAgeInSeconds(delta.getOldestScoredTweetAgeInSeconds());\n    }\n\n    base.setNumFromDirectFollows(base.getNumFromDirectFollows() + delta.getNumFromDirectFollows());\n    base.setNumFromTrustedCircle(base.getNumFromTrustedCircle() + delta.getNumFromTrustedCircle());\n    base.setNumReplies(base.getNumReplies() + delta.getNumReplies());\n    base.setNumRepliesTrusted(base.getNumRepliesTrusted() + delta.getNumRepliesTrusted());\n    base.setNumRepliesOutOfNetwork(\n            base.getNumRepliesOutOfNetwork() + delta.getNumRepliesOutOfNetwork());\n    base.setNumSelfTweets(base.getNumSelfTweets() + delta.getNumSelfTweets());\n    base.setNumWithMedia(base.getNumWithMedia() + delta.getNumWithMedia());\n    base.setNumWithNews(base.getNumWithNews() + delta.getNumWithNews());\n    base.setNumSpamUser(base.getNumSpamUser() + delta.getNumSpamUser());\n    base.setNumOffensive(base.getNumOffensive() + delta.getNumOffensive());\n    base.setNumBot(base.getNumBot() + delta.getNumBot());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/lang/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    provides = artifact(\n        org = \"com.twitter.search.common.util\",\n        name = \"lang\",\n        repo = artifactory,\n    ),\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/code/findbugs:jsr305\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/text/language:locale-util\",\n        \"src/thrift/com/twitter/search/common:constants-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/lang/ThriftLanguageUtil.java",
    "content": "package com.twitter.search.common.util.lang;\n\nimport java.lang.reflect.Field;\nimport java.util.Locale;\nimport java.util.Map;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.text.language.LocaleUtil;\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\n\n/**\n * This class can be used to convert ThriftLanguage to Locale object and vise versa.\n */\npublic final class ThriftLanguageUtil {\n  private static final Logger LOG = LoggerFactory.getLogger(ThriftLanguageUtil.class.getName());\n\n  // stores ThriftLanguage.id -> Locale mapping\n  private static final Locale[] LOCALES;\n\n  // stores Locale -> ThriftLanguage mapping\n  private static final Map<Locale, ThriftLanguage> THRIFT_LANGUAGES;\n\n  static {\n    LOCALES = new Locale[ThriftLanguage.values().length];\n    Map<Locale, ThriftLanguage> thriftLanguageMap = Maps.newHashMap();\n\n    // get all languages defined in ThriftLanguage\n    Field[] fields = ThriftLanguage.class.getDeclaredFields();\n    for (Field field : fields) {\n      if (!field.isEnumConstant()) {\n        continue;\n      }\n\n      try {\n        ThriftLanguage thriftLang = (ThriftLanguage) field.get(null);\n        String thriftLanguageName = field.getName();\n\n        // get corresponding Locale declared in LocaleUtil\n        try {\n          Field localeUtilField = LocaleUtil.class.getDeclaredField(thriftLanguageName);\n          Locale localeLang = (Locale) localeUtilField.get(null);\n\n          LOCALES[thriftLang.getValue()] = localeLang;\n          thriftLanguageMap.put(localeLang, thriftLang);\n        } catch (NoSuchFieldException e) {\n          LOG.warn(\"{} is defined in ThriftLanguage, but not in LocaleUtil.\", thriftLanguageName);\n        }\n      } catch (IllegalAccessException e) {\n        // shouldn't happen.\n        LOG.warn(\"Could not get a declared field.\", e);\n      }\n    }\n\n    // Let's make sure that all Locales defined in LocaleUtil are also defined in ThriftLanguage\n    for (Locale lang : LocaleUtil.getDefinedLanguages()) {\n      if (!thriftLanguageMap.containsKey(lang)) {\n        LOG.warn(\"{} is defined in LocaleUtil but not in ThriftLanguage.\", lang.getLanguage());\n      }\n    }\n\n    THRIFT_LANGUAGES = ImmutableMap.copyOf(thriftLanguageMap);\n  }\n\n  private ThriftLanguageUtil() {\n  }\n\n  /**\n   * Returns a Locale object which corresponds to a given ThriftLanguage object.\n   * @param language ThriftLanguage object\n   * @return a corresponding Locale object\n   */\n  public static Locale getLocaleOf(ThriftLanguage language) {\n    // Note that ThriftLanguage.findByValue() can return null (thrift generated code).\n    // So ThriftLanguageUtil.getLocaleOf needs to handle null correctly.\n    if (language == null) {\n      return LocaleUtil.UNKNOWN;\n    }\n\n    Preconditions.checkArgument(language.getValue() < LOCALES.length);\n    return LOCALES[language.getValue()];\n  }\n\n  /**\n   * Returns a ThriftLanguage object which corresponds to a given Locale object.\n   *\n   * @param language Locale object\n   * @return a corresponding ThriftLanguage object, or UNKNOWN if there's no corresponding one.\n   */\n  public static ThriftLanguage getThriftLanguageOf(Locale language) {\n    Preconditions.checkNotNull(language);\n    ThriftLanguage thriftLang = THRIFT_LANGUAGES.get(language);\n    return thriftLang == null ? ThriftLanguage.UNKNOWN : thriftLang;\n  }\n\n  /**\n   * Returns a ThriftLanguage object which corresponds to a given language code.\n   *\n   * @param languageCode BCP-47 language code\n   * @return a corresponding ThriftLanguage object, or UNKNOWN if there's no corresponding one.\n   */\n  public static ThriftLanguage getThriftLanguageOf(String languageCode) {\n    Preconditions.checkNotNull(languageCode);\n    ThriftLanguage thriftLang = THRIFT_LANGUAGES.get(LocaleUtil.getLocaleOf(languageCode));\n    return thriftLang == null ? ThriftLanguage.UNKNOWN : thriftLang;\n  }\n\n  /**\n   * Returns a ThriftLanguage object which corresponds to a given int value.\n   * If value is not valid, returns ThriftLanguage.UNKNOWN\n   * @param value value of language\n   * @return a corresponding ThriftLanguage object\n   */\n  public static ThriftLanguage safeFindByValue(int value) {\n    ThriftLanguage thriftLang = ThriftLanguage.findByValue(value);\n    return thriftLang == null ? ThriftLanguage.UNKNOWN : thriftLang;\n  }\n\n  /**\n   * Returns the language code which corresponds to a given ThriftLanguage.\n   *\n   * Note that multiple ThriftLanguage entries can return the same language code.\n   *\n   * @param thriftLang ThriftLanguage object\n   * @return Corresponding language or null if thriftLang is null.\n   */\n  @Nullable\n  public static String getLanguageCodeOf(@Nullable ThriftLanguage thriftLang) {\n    if (thriftLang == null) {\n      return null;\n    }\n    return ThriftLanguageUtil.getLocaleOf(thriftLang).getLanguage();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/it/unimi/dsi:fastutil\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/java/com/twitter/search/common/util/io\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/EnumBasedLinearModel.java",
    "content": "package com.twitter.search.common.util.ml;\n\nimport java.io.IOException;\nimport java.util.EnumMap;\nimport java.util.EnumSet;\nimport java.util.Map;\nimport java.util.Set;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Predicates;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Maps;\n\nimport com.twitter.search.common.file.AbstractFile;\nimport com.twitter.search.common.util.io.TextFileLoadingUtils;\n\n/**\n * Represents a linear model for scoring and classification.\n *\n * The list of features is defined by an Enum class. The model weights and instances are\n * represented as maps that must contain an entry for all the values of the enum.\n *\n */\npublic class EnumBasedLinearModel<K extends Enum<K>> implements MapBasedLinearModel<K> {\n\n  private final EnumSet<K> features;\n  private final EnumMap<K, Float> weights;\n\n  /**\n   * Creates a model from a map of weights.\n   *\n   * @param enumType Enum used for the keys\n   * @param weights Feature weights.\n   */\n  public EnumBasedLinearModel(Class<K> enumType, Map<K, Float> weights) {\n    features = EnumSet.allOf(enumType);\n    EnumMap<K, Float> enumWeights =\n        new EnumMap<>(Maps.filterValues(weights, Predicates.notNull()));\n    Preconditions.checkArgument(features.equals(enumWeights.keySet()),\n        \"The model does not include weights for all the available features\");\n\n    this.weights = enumWeights;\n  }\n\n  public ImmutableMap<K, Float> getWeights() {\n    return Maps.immutableEnumMap(weights);\n  }\n\n  @Override\n  public float score(Map<K, Float> instance) {\n    float total = 0;\n    for (Map.Entry<K, Float> weightEntry : weights.entrySet()) {\n      Float feature = instance.get(weightEntry.getKey());\n      if (feature != null) {\n        total += weightEntry.getValue() * feature;\n      }\n    }\n    return total;\n  }\n\n  /**\n   * Determines whether an instance is positive.\n   */\n  @Override\n  public boolean classify(float threshold, Map<K, Float> instance) {\n    return score(instance) > threshold;\n  }\n\n  @Override\n  public boolean classify(Map<K, Float> instance) {\n    return classify(0, instance);\n  }\n\n  @Override\n  public String toString() {\n    return String.format(\"EnumBasedLinearModel[%s]\", weights);\n  }\n\n  /**\n   * Creates a model where all the features have the same weight.\n   * This method is useful for generating the feature vectors for training a new model.\n   */\n  public static <T extends Enum<T>> EnumBasedLinearModel<T> createWithEqualWeight(Class<T> enumType,\n                                                                                  Float weight) {\n    EnumSet<T> features = EnumSet.allOf(enumType);\n    EnumMap<T, Float> weights = Maps.newEnumMap(enumType);\n    for (T feature : features) {\n      weights.put(feature, weight);\n    }\n    return new EnumBasedLinearModel<>(enumType, weights);\n  }\n\n  /**\n   * Loads the model from a TSV file with the following format:\n   *\n   *    feature_name  \\t  weight\n   */\n  public static <T extends Enum<T>> EnumBasedLinearModel<T> createFromFile(\n      Class<T> enumType, AbstractFile path) throws IOException {\n    return new EnumBasedLinearModel<>(enumType, loadWeights(enumType, path, true));\n  }\n\n  /**\n   * Loads the model from a TSV file, using a default weight of 0 for missing features.\n   *\n   * File format:\n   *\n   *     feature_name  \\t  weight\n   */\n  public static <T extends Enum<T>> EnumBasedLinearModel<T> createFromFileSafe(\n      Class<T> enumType, AbstractFile path) throws IOException {\n    return new EnumBasedLinearModel<>(enumType, loadWeights(enumType, path, false));\n  }\n\n  /**\n   * Creates a map of (feature_name, weight) from a TSV file.\n   *\n   * If strictMode is true, it will throw an exception if the file doesn't contain all the\n   * features declared in the enum. Otherwise, it will use zero as default value.\n   *\n   */\n  private static <T extends Enum<T>> EnumMap<T, Float> loadWeights(\n      Class<T> enumType, AbstractFile fileHandle, boolean strictMode) throws IOException {\n    Map<String, Float> weightsFromFile =\n      TextFileLoadingUtils.loadMapFromFile(fileHandle, input -> Float.parseFloat(input));\n    EnumMap<T, Float> weights = Maps.newEnumMap(enumType);\n    Set<T> expectedFeatures = EnumSet.allOf(enumType);\n    if (!strictMode) {\n      for (T feature : expectedFeatures) {\n        weights.put(feature, 0f);\n      }\n    }\n    for (String featureName : weightsFromFile.keySet()) {\n      Float weight = weightsFromFile.get(featureName);\n      weights.put(Enum.valueOf(enumType, featureName.toUpperCase()), weight);\n    }\n    Preconditions.checkArgument(expectedFeatures.equals(weights.keySet()),\n        \"Model does not contain weights for all the features\");\n    return weights;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/FeatureUtils.java",
    "content": "package com.twitter.search.common.util.ml;\n\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Optional;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Sets;\n\n/**\n * Utilities for feature transformation and extraction.\n */\npublic final class FeatureUtils {\n\n  private FeatureUtils() {\n  }\n\n  /**\n   * Computes the difference between 2 values and returns the ratio of the difference over the\n   * minimum of both, according to these cases:\n   *\n   * 1. if (a > b) return  a / b\n   * 2. if (a < b) return  - b / a\n   * 3. if (a == b == 0) return 0\n   *\n   * The upper/lower limit is (-) maxRatio. For cases 1 and 2, if the denominator is 0,\n   * it returns maxRatio.\n   *\n   * This method is used to define a feature that tells how much larger or smaller is the\n   * first value with respect to the second one..\n   */\n  public static float diffRatio(float a, float b, float maxRatio) {\n    float diff = a - b;\n    if (diff == 0) {\n      return 0;\n    }\n    float denominator = Math.min(a, b);\n    float ratio = denominator != 0 ? Math.abs(diff / denominator) : maxRatio;\n    return Math.copySign(Math.min(ratio, maxRatio), diff);\n  }\n\n  /**\n   * Computes the cosine similarity between two maps that represent sparse vectors.\n   */\n  public static <K, V extends Number> double cosineSimilarity(\n      Map<K, V> vector1, Map<K, V> vector2) {\n    if (vector1 == null || vector1.isEmpty() || vector2 == null || vector2.isEmpty()) {\n      return 0;\n    }\n    double squaredSum1 = 0;\n    double squaredSum2 = 0;\n    double squaredCrossSum = 0;\n\n    for (K key : Sets.union(vector1.keySet(), vector2.keySet())) {\n      double value1 = 0;\n      double value2 = 0;\n\n      V optValue1 = vector1.get(key);\n      if (optValue1 != null) {\n        value1 = optValue1.doubleValue();\n      }\n      V optValue2 = vector2.get(key);\n      if (optValue2 != null) {\n        value2 = optValue2.doubleValue();\n      }\n\n      squaredSum1 += value1 * value1;\n      squaredSum2 += value2 * value2;\n      squaredCrossSum += value1 * value2;\n    }\n\n    if (squaredSum1 == 0 || squaredSum2 == 0) {\n      return 0;\n    } else {\n      return squaredCrossSum / Math.sqrt(squaredSum1 * squaredSum2);\n    }\n  }\n\n  /**\n   * Computes the cosine similarity between two (dense) vectors.\n   */\n  public static <V extends Number> double cosineSimilarity(\n      List<V> vector1, List<V> vector2) {\n    if (vector1 == null || vector1.isEmpty() || vector2 == null || vector2.isEmpty()) {\n      return 0;\n    }\n\n    Preconditions.checkArgument(vector1.size() == vector2.size());\n    double squaredSum1 = 0;\n    double squaredSum2 = 0;\n    double squaredCrossSum = 0;\n    for (int i = 0; i < vector1.size(); i++) {\n      double value1 = vector1.get(i).doubleValue();\n      double value2 = vector2.get(i).doubleValue();\n      squaredSum1 += value1 * value1;\n      squaredSum2 += value2 * value2;\n      squaredCrossSum += value1 * value2;\n    }\n\n    if (squaredSum1 == 0 || squaredSum2 == 0) {\n      return 0;\n    } else {\n      return squaredCrossSum / Math.sqrt(squaredSum1 * squaredSum2);\n    }\n  }\n\n  /**\n   * Finds the key of the map with the highest value (compared in natural order)\n   */\n  @SuppressWarnings(\"unchecked\")\n  public static <K, V extends Comparable> Optional<K> findMaxKey(Map<K, V> map) {\n    if (map == null || map.isEmpty()) {\n      return Optional.empty();\n    }\n\n    Optional<Map.Entry<K, V>> maxEntry = map.entrySet().stream().max(Map.Entry.comparingByValue());\n    return maxEntry.map(Map.Entry::getKey);\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/MapBasedLinearModel.java",
    "content": "package com.twitter.search.common.util.ml;\n\nimport java.util.Map;\n\n/**\n * An interface for linear models that are backed by some sort of map\n */\npublic interface MapBasedLinearModel<K> {\n  /**\n   * Evaluate using this model given a feature vector.\n   * @param instance The feature vector in format of a hashmap.\n   * @return\n   */\n  boolean classify(Map<K, Float> instance);\n\n  /**\n   * Evaluate using this model given a classification threshold and a feature vector.\n   * @param threshold Score threshold used for classification.\n   * @param instance The feature vector in format of a hashmap.\n   * @return\n   */\n  boolean classify(float threshold, Map<K, Float> instance);\n\n  /**\n   * Computes the score of an instance as a linear combination of the features and the model\n   * weights. 0 is used as default value for features or weights that are not present.\n   *\n   * @param instance The feature vector in format of a hashmap.\n   * @return The instance score according to the model.\n   */\n  float score(Map<K, Float> instance);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/StringMapBasedLinearModel.java",
    "content": "package com.twitter.search.common.util.ml;\n\nimport java.util.Map;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.base.Function;\nimport com.twitter.search.common.file.AbstractFile;\nimport com.twitter.search.common.util.io.TextFileLoadingUtils;\n\nimport it.unimi.dsi.fastutil.objects.Object2FloatMap;\nimport it.unimi.dsi.fastutil.objects.Object2FloatOpenHashMap;\n\n/**\n * Represents a linear model for scoring and classification.\n *\n * Features are represented as arbitrary strings, making this a fairly flexible implementation\n * (at the cost of some performance, since all operations require hash lookups). Instances\n * and weights are both encoded sparsely (as maps) so this implementation is well suited to\n * models with large feature sets where most features are inactive at a given time. Weights\n * for unknown features are assumed to be 0.\n *\n */\npublic class StringMapBasedLinearModel implements MapBasedLinearModel<String> {\n  private static final Logger LOG = LoggerFactory.getLogger(StringMapBasedLinearModel.class);\n\n  protected final Object2FloatMap<String> model = new Object2FloatOpenHashMap<>();\n\n  /**\n   * Creates a model from a map of weights.\n   *\n   * @param weights Feature weights.\n   */\n  public StringMapBasedLinearModel(Map<String, Float> weights) {\n    model.putAll(weights);\n    model.defaultReturnValue(0.0f);\n  }\n\n  /**\n   * Get the weight of a feature\n   * @param featureName\n   * @return\n   */\n  public float getWeight(String featureName) {\n    return model.getFloat(featureName);\n  }\n\n  /**\n   * Get the full weight map\n   */\n  @VisibleForTesting\n  protected Map<String, Float> getWeights() {\n    return model;\n  }\n\n  /**\n   * Evaluate using this model given a feature vector.\n   * @param values The feature vector in format of a hashmap.\n   * @return\n   */\n  @Override\n  public float score(Map<String, Float> values) {\n    float score = 0.0f;\n    for (Map.Entry<String, Float> value : values.entrySet()) {\n      String featureName = value.getKey();\n      float weight = getWeight(featureName);\n      if (weight != 0.0f) {\n        score += weight * value.getValue();\n        if (LOG.isDebugEnabled()) {\n          LOG.debug(String.format(\"%s = %.3f * %.3f = %.3f, \",\n              featureName, weight, value.getValue(),\n              weight * value.getValue()));\n        }\n      }\n    }\n    if (LOG.isDebugEnabled()) {\n      LOG.debug(String.format(\"Score = %.3f\", score));\n    }\n    return score;\n  }\n\n  /**\n   * Determines whether an instance is positive.\n   */\n  @Override\n  public boolean classify(Map<String, Float> values) {\n    return classify(0.0f, values);\n  }\n\n  @Override\n  public boolean classify(float threshold, Map<String, Float> values) {\n    return score(values) > threshold;\n  }\n\n  public int size() {\n    return model.size();\n  }\n\n  @Override\n  public String toString() {\n    StringBuilder sb = new StringBuilder();\n    sb.append(\"StringMapBasedLinearModel[\");\n    for (Map.Entry<String, Float> entry : model.entrySet()) {\n      sb.append(String.format(\"(%s = %.3f), \", entry.getKey(), entry.getValue()));\n    }\n    sb.append(\"]\");\n    return sb.toString();\n  }\n\n  /**\n   * Loads the model from a TSV file with the following format:\n   *\n   *    feature_name  \\t  weight\n   */\n  public static StringMapBasedLinearModel loadFromFile(AbstractFile fileHandle) {\n    Map<String, Float> weights =\n        TextFileLoadingUtils.loadMapFromFile(\n            fileHandle,\n            (Function<String, Float>) item -> Float.parseFloat(item));\n    return new StringMapBasedLinearModel(weights);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/models_manager/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    strict_deps = True,\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"3rdparty/jvm/org/yaml:snakeyaml\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/java/com/twitter/search/common/metrics\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/models_manager/BaseModelsManager.java",
    "content": "package com.twitter.search.common.util.ml.models_manager;\n\nimport java.io.BufferedReader;\nimport java.io.IOException;\nimport java.io.UncheckedIOException;\nimport java.util.Collections;\nimport java.util.Date;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Optional;\nimport java.util.Set;\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.Executors;\nimport java.util.concurrent.TimeUnit;\nimport java.util.function.Function;\nimport java.util.function.Supplier;\nimport java.util.stream.Collectors;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Strings;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.Sets;\nimport com.google.common.util.concurrent.ThreadFactoryBuilder;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport org.yaml.snakeyaml.Yaml;\n\nimport com.twitter.search.common.file.AbstractFile;\nimport com.twitter.search.common.file.FileUtils;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\n\n/**\n * Loads models from HDFS and provides an interface for reloading them periodically.\n *\n * There are 2 possible ways of detecting the active models:\n *\n * - DirectorySupplier: Uses all the subdirectories of a base path\n * - ConfigSupplier: Gets the list from from a configuration file\n *\n * Models can be updated or added. Depending on the selected method, existing models can be removed\n * if they are no longer active.\n */\npublic abstract class BaseModelsManager<T> implements Runnable {\n  private static final Logger LOG = LoggerFactory.getLogger(BaseModelsManager.class);\n\n  protected final Map<String, Long> lastModifiedMsByModel = new ConcurrentHashMap<>();\n  protected final Map<String, T> loadedModels = new ConcurrentHashMap<>();\n  protected final Supplier<Map<String, AbstractFile>> activeModelsSupplier;\n\n  protected Map<String, T> prevLoadedModels = new ConcurrentHashMap<>();\n\n  // This flag determines whether models are unloaded immediately when they're removed from\n  // activeModelsSupplier. If false, old models stay in memory until the process is restarted.\n  // This may be useful to safely change model configuration without restarting.\n  protected final boolean shouldUnloadInactiveModels;\n\n  protected final SearchLongGauge numModels;\n  protected final SearchCounter numErrors;\n  protected final SearchLongGauge lastLoadedMs;\n\n  protected Supplier<Boolean> shouldServeModels;\n  protected Supplier<Boolean> shouldLoadModels;\n\n  public BaseModelsManager(\n      Supplier<Map<String, AbstractFile>> activeModelsSupplier,\n      boolean shouldUnloadInactiveModels,\n      String statsPrefix\n  ) {\n    this(\n      activeModelsSupplier,\n      shouldUnloadInactiveModels,\n      statsPrefix,\n      () -> true,\n      () -> true\n    );\n  }\n\n  public BaseModelsManager(\n      Supplier<Map<String, AbstractFile>> activeModelsSupplier,\n      boolean shouldUnloadInactiveModels,\n      String statsPrefix,\n      Supplier<Boolean> shouldServeModels,\n      Supplier<Boolean> shouldLoadModels\n  ) {\n    this.activeModelsSupplier = activeModelsSupplier;\n    this.shouldUnloadInactiveModels = shouldUnloadInactiveModels;\n\n    this.shouldServeModels = shouldServeModels;\n    this.shouldLoadModels = shouldLoadModels;\n\n    numModels = SearchLongGauge.export(\n        String.format(\"model_loader_%s_num_models\", statsPrefix));\n    numErrors = SearchCounter.export(\n        String.format(\"model_loader_%s_num_errors\", statsPrefix));\n    lastLoadedMs = SearchLongGauge.export(\n        String.format(\"model_loader_%s_last_loaded_timestamp_ms\", statsPrefix));\n  }\n\n  /**\n   *  Retrieves a particular model.\n   */\n  public Optional<T> getModel(String name) {\n    if (shouldServeModels.get()) {\n      return Optional.ofNullable(loadedModels.get(name));\n    } else {\n      return Optional.empty();\n    }\n  }\n\n  /**\n   * Reads a model instance from the directory file instance.\n   *\n   * @param modelBaseDir AbstractFile instance representing the directory.\n   * @return Model instance parsed from the directory.\n   */\n  public abstract T readModelFromDirectory(AbstractFile modelBaseDir) throws Exception;\n\n  /**\n   * Cleans up any resources used by the model instance.\n   * This method is called after removing the model from the in-memory map.\n   * Sub-classes can provide custom overridden implementation as required.\n   *\n   * @param unloadedModel Model instance that would be unloaded from the manager.\n   */\n  protected void cleanUpUnloadedModel(T unloadedModel) { }\n\n  @Override\n  public void run() {\n    // Get available models, either from the config file or by listing the base directory\n    final Map<String, AbstractFile> modelPathsFromConfig;\n    if (!shouldLoadModels.get()) {\n      LOG.info(\"Loading models is currently disabled.\");\n      return;\n    }\n\n    modelPathsFromConfig = activeModelsSupplier.get();\n    for (Map.Entry<String, AbstractFile> nameAndPath : modelPathsFromConfig.entrySet()) {\n      String modelName = nameAndPath.getKey();\n      try {\n        AbstractFile modelDirectory = nameAndPath.getValue();\n        if (!modelDirectory.exists() && loadedModels.containsKey(modelName)) {\n          LOG.warn(\"Loaded model '{}' no longer exists at HDFS path {}, keeping loaded version; \"\n              + \"replace directory in HDFS to update model.\", modelName, modelDirectory);\n          continue;\n        }\n\n        long previousModifiedTimestamp = lastModifiedMsByModel.getOrDefault(modelName, 0L);\n        long lastModifiedMs = modelDirectory.getLastModified();\n        if (previousModifiedTimestamp == lastModifiedMs) {\n          continue;\n        }\n\n        LOG.info(\"Starting to load model. name={} path={}\", modelName, modelDirectory.getPath());\n        T model = Preconditions.checkNotNull(readModelFromDirectory(modelDirectory));\n        LOG.info(\"Model initialized: {}. Last modified: {} ({})\",\n                 modelName, lastModifiedMs, new Date(lastModifiedMs));\n        T previousModel = loadedModels.put(modelName, model);\n        lastModifiedMsByModel.put(modelName, lastModifiedMs);\n\n        if (previousModel != null) {\n          cleanUpUnloadedModel(previousModel);\n        }\n      } catch (Exception e) {\n        numErrors.increment();\n        LOG.error(\"Error initializing model: {}\", modelName, e);\n      }\n    }\n\n    // Remove any currently loaded models not present in the latest list\n    if (shouldUnloadInactiveModels) {\n      Set<String> inactiveModels =\n          Sets.difference(loadedModels.keySet(), modelPathsFromConfig.keySet()).immutableCopy();\n\n      for (String modelName : inactiveModels) {\n        T modelToUnload = loadedModels.get(modelName);\n        loadedModels.remove(modelName);\n\n        if (modelToUnload != null) {\n          // We could have an inactive model key without a model (value) if the\n          // initial readModelFromDirectory failed for the model entry.\n          // Checking for null to avoid exception.\n          cleanUpUnloadedModel(modelToUnload);\n        }\n        LOG.info(\"Unloaded model that is no longer active: {}\", modelName);\n      }\n    }\n\n    if (!prevLoadedModels.keySet().equals(loadedModels.keySet())) {\n      LOG.info(\"Finished loading models: {}\", loadedModels.keySet());\n    }\n    prevLoadedModels = loadedModels;\n    numModels.set(loadedModels.size());\n    lastLoadedMs.set(System.currentTimeMillis());\n  }\n\n  /**\n   * Schedules the loader to run periodically.\n   * @param period Period between executions\n   * @param timeUnit The time unit the period parameter.\n   */\n  public final void scheduleAtFixedRate(\n      long period, TimeUnit timeUnit, String builderThreadName) {\n    Executors.newSingleThreadScheduledExecutor(\n        new ThreadFactoryBuilder()\n            .setDaemon(true)\n            .setNameFormat(builderThreadName)\n            .build())\n        .scheduleAtFixedRate(this, 0, period, timeUnit);\n  }\n\n  /**\n   * Gets the active list of models from the subdirectories in a base directory.\n   *\n   * Each model is identified by the name of the subdirectory.\n   */\n  @VisibleForTesting\n  public static class DirectorySupplier implements Supplier<Map<String, AbstractFile>> {\n    private static final Logger LOG = LoggerFactory.getLogger(DirectorySupplier.class);\n    private final AbstractFile baseDir;\n\n    public DirectorySupplier(AbstractFile baseDir) {\n      this.baseDir = baseDir;\n    }\n\n    @Override\n    public Map<String, AbstractFile> get() {\n      try {\n        LOG.info(\"Loading models from the directories in: {}\", baseDir.getPath());\n        List<AbstractFile> modelDirs =\n            ImmutableList.copyOf(baseDir.listFiles(AbstractFile.IS_DIRECTORY));\n        LOG.info(\"Found {} model directories: {}\", modelDirs.size(), modelDirs);\n        return modelDirs.stream()\n            .collect(Collectors.toMap(\n                AbstractFile::getName,\n                Function.identity()\n            ));\n      } catch (IOException e) {\n        throw new UncheckedIOException(e);\n      }\n    }\n  }\n\n  /**\n   * Gets the active list of models by reading a YAML config file.\n   *\n   * The keys are the model names, the values are dictionaries with a single entry for the path\n   * of the model in HDFS (without the HDFS name node prefix). For example:\n   *\n   *    model_a:\n   *        path: /path/to/model_a\n   *    model_b:\n   *        path: /path/to/model_b\n   *\n   */\n  @VisibleForTesting\n  public static class ConfigSupplier implements Supplier<Map<String, AbstractFile>> {\n\n    private final AbstractFile configFile;\n\n    public ConfigSupplier(AbstractFile configFile) {\n      this.configFile = configFile;\n    }\n\n    @SuppressWarnings(\"unchecked\")\n    @Override\n    public Map<String, AbstractFile> get() {\n      try (BufferedReader configReader = configFile.getCharSource().openBufferedStream()) {\n        Yaml yamlParser = new Yaml();\n        //noinspection unchecked\n        Map<String, Map<String, String>> config =\n            (Map<String, Map<String, String>>) yamlParser.load(configReader);\n\n        if (config == null || config.isEmpty()) {\n          return Collections.emptyMap();\n        }\n\n        Map<String, AbstractFile> modelPaths = new HashMap<>();\n        for (Map.Entry<String, Map<String, String>> nameAndConfig : config.entrySet()) {\n          String path = Strings.emptyToNull(nameAndConfig.getValue().get(\"path\"));\n          Preconditions.checkNotNull(path, \"Missing path for model: %s\", nameAndConfig.getKey());\n          modelPaths.put(nameAndConfig.getKey(), FileUtils.getHdfsFileHandle(path));\n        }\n        return modelPaths;\n      } catch (IOException e) {\n        throw new UncheckedIOException(e);\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common_internal/hadoop\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/ml/api/transform\",\n        \"src/java/com/twitter/ml/common/base\",\n        \"src/java/com/twitter/ml/prediction/core\",\n        \"src/java/com/twitter/ml/tool/prediction:ModelInterpreter\",\n        \"src/java/com/twitter/ml/vw/constant\",\n        \"src/java/com/twitter/mlv2/trees/predictor\",\n        \"src/java/com/twitter/mlv2/trees/scorer\",\n        \"src/java/com/twitter/search/common/features\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/util/ml/models_manager\",\n        \"src/java/com/twitter/search/modeling/common\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/search/common:features-java\",\n    ],\n)\n\njava_library(\n    name = \"for-timelines\",\n    sources = [\n        \"BaseLegacyScoreAccumulator.java\",\n        \"BaseModelBuilder.java\",\n        \"BaseScoreAccumulator.java\",\n        \"CompositeFeatureContext.java\",\n        \"DiscretizedFeature.java\",\n        \"DiscretizedFeatureRange.java\",\n        \"LegacyModelBuilder.java\",\n        \"LightweightLinearModel.java\",\n        \"ModelBuilder.java\",\n        \"SchemaBasedModelBuilder.java\",\n    ],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common_internal/hadoop\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/ml/api/transform:DiscretizerTransform\",\n        \"src/java/com/twitter/ml/common/base\",\n        \"src/java/com/twitter/ml/tool/prediction:ModelInterpreter\",\n        \"src/java/com/twitter/ml/vw/constant\",\n        \"src/java/com/twitter/search/common/features\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/util/ml/models_manager\",\n        \"src/java/com/twitter/search/modeling/common\",\n        \"src/thrift/com/twitter/ml/api:data-java\",\n        \"src/thrift/com/twitter/search/common:features-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/BaseLegacyScoreAccumulator.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.ml.api.Feature;\n\n/**\n * Score accumulator for legacy (non-schema-based) features. It provides methods to add features\n * using Feature objects.\n *\n * @deprecated This class is retired and we suggest to switch to schema-based features.\n */\n@Deprecated\npublic abstract class BaseLegacyScoreAccumulator<D> extends BaseScoreAccumulator<D> {\n\n  public BaseLegacyScoreAccumulator(LightweightLinearModel model) {\n    super(model);\n    Preconditions.checkState(!model.isSchemaBased(),\n        \"Cannot create LegacyScoreAccumulator with a schema-based model: %s\", model.getName());\n  }\n\n  /**\n   * Add to the score the weight of a binary feature (if it's present).\n   *\n   * @deprecated This function is retired and we suggest to switch to addSchemaBooleanFeatures in\n   * SchemaBasedScoreAccumulator.\n   */\n  @Deprecated\n  protected BaseLegacyScoreAccumulator addBinaryFeature(Feature<Boolean> feature,\n                                                        boolean value) {\n    if (value) {\n      Double weight = model.binaryFeatures.get(feature);\n      if (weight != null) {\n        score += weight;\n      }\n    }\n    return this;\n  }\n\n  /**\n   * Add to the score the weight of a continuous feature.\n   * <p>\n   * If the model uses real valued features, it multiplies its weight by the provided value.\n   * Otherwise, it tries to find the discretized feature and adds its weight to the score.\n   *\n   * @deprecated This function is retired and we suggest to switch to addSchemaContinuousFeatures in\n   * SchemaBasedScoreAccumulator.\n   */\n  @Deprecated\n  protected BaseLegacyScoreAccumulator addContinuousFeature(Feature<Double> feature,\n                                                            double value) {\n    Double weightFromContinuous = model.continuousFeatures.get(feature);\n    if (weightFromContinuous != null) {\n      score += weightFromContinuous * value;\n    } else {\n      DiscretizedFeature discretizedFeature = model.discretizedFeatures.get(feature);\n      if (discretizedFeature != null) {\n        // Use only the weight of the discretized feature (there's no need to multiply it)\n        score += discretizedFeature.getWeight(value);\n      }\n    }\n    return this;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/BaseModelBuilder.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\nimport java.util.Collection;\nimport java.util.Comparator;\nimport java.util.List;\n\nimport com.google.common.collect.Lists;\n\nimport com.twitter.ml.api.FeatureParser;\nimport com.twitter.ml.api.transform.DiscretizerTransform;\nimport com.twitter.ml.tool.prediction.ModelInterpreter;\n\n/**\n * The base model builder for LightweightLinearModels.\n */\npublic abstract class BaseModelBuilder implements ModelBuilder {\n  // Ignore features that have an absolute weight lower than this value\n  protected static final double MIN_WEIGHT = 1e-9;\n  private static final String BIAS_FIELD_NAME = ModelInterpreter.BIAS_FIELD_NAME;\n  static final String DISCRETIZER_NAME_SUFFIX =\n      \".\" + DiscretizerTransform.DEFAULT_FEATURE_NAME_SUFFIX;\n\n  protected final String modelName;\n  protected double bias;\n\n  public BaseModelBuilder(String modelName) {\n    this.modelName = modelName;\n    this.bias = 0.0;\n  }\n\n  /**\n   * Collects all the ranges of a discretized feature and sorts them.\n   */\n  static DiscretizedFeature buildFeature(Collection<DiscretizedFeatureRange> ranges) {\n    List<DiscretizedFeatureRange> sortedRanges = Lists.newArrayList(ranges);\n    sortedRanges.sort(Comparator.comparingDouble(a -> a.minValue));\n\n    double[] splits = new double[ranges.size()];\n    double[] weights = new double[ranges.size()];\n\n    for (int i = 0; i < sortedRanges.size(); i++) {\n      splits[i] = sortedRanges.get(i).minValue;\n      weights[i] = sortedRanges.get(i).weight;\n    }\n    return new DiscretizedFeature(splits, weights);\n  }\n\n  /**\n   * Parses a line from the interpreted model text file. See the javadoc of the constructor for\n   * more details about how to create the text file.\n   * <p>\n   * The file uses TSV format with 3 columns:\n   * <p>\n   * Model name (Generated by ML API, but ignored by this class)\n   * Feature definition:\n   * Name of the feature or definition from the MDL discretizer.\n   * Weight:\n   * Weight of the feature using LOGIT scale.\n   * <p>\n   * When it parses each line, it stores the weights for all the features defined in the context,\n   * as well as the bias, but it ignores any other feature (e.g. label, prediction or\n   * meta.record_weight) and features with a small absolute weight (see MIN_WEIGHT).\n   * <p>\n   * Example lines:\n   * <p>\n   * model_name      bias    0.019735312089324074\n   * model_name      demo.binary_feature          0.06524706073105327\n   * model_name      demo.continuous_feature      0.0\n   * model_name      demo.continuous_feature.dz/dz_model=mdl/dz_range=-inf_3.58e-01   0.07155931927263737\n   * model_name      demo.continuous_feature.dz/dz_model=mdl/dz_range=3.58e-01_inf    -0.08979256264865387\n   *\n   * @see ModelInterpreter\n   * @see DiscretizerTransform\n   */\n  @Override\n  public ModelBuilder parseLine(String line) {\n    String[] columns = line.split(\"\\t\");\n    if (columns.length != 3) {\n      return this;\n    }\n\n    // columns[0] has the model name, which we don't need\n    String featureName = columns[1];\n    double weight = Double.parseDouble(columns[2]);\n\n    if (BIAS_FIELD_NAME.equals(featureName)) {\n      bias = weight;\n      return this;\n    }\n\n    FeatureParser parser = FeatureParser.parse(featureName);\n    String baseName = parser.getBaseName();\n\n    if (Math.abs(weight) < MIN_WEIGHT && !baseName.endsWith(DISCRETIZER_NAME_SUFFIX)) {\n      // skip, unless it represents a range of a discretized feature.\n      // discretized features with all zeros should also be removed, but will handle that later\n      return this;\n    }\n\n    addFeature(baseName, weight, parser);\n    return this;\n  }\n\n  /**\n   * Adds feature to the model\n   */\n  protected abstract void addFeature(String baseName, double weight, FeatureParser parser);\n\n  @Override\n  public abstract LightweightLinearModel build();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/BaseScoreAccumulator.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\n/**\n * The base class for a lightweight scorer based on a model and some feature data.\n *\n * @param <D> The type of feature data to be scored with\n */\npublic abstract class BaseScoreAccumulator<D> {\n  protected final LightweightLinearModel model;\n  protected double score;\n\n  public BaseScoreAccumulator(LightweightLinearModel model) {\n    this.model = model;\n    this.score = model.bias;\n  }\n\n  /**\n   * Compute score with a model and feature data\n   */\n  public final double scoreWith(D featureData, boolean useLogitScore) {\n    updateScoreWithFeatures(featureData);\n    return useLogitScore ? getLogitScore() : getSigmoidScore();\n  }\n\n  public final void reset() {\n    this.score = model.bias;\n  }\n\n  /**\n   * Update the accumulator score with features, after this function the score should already\n   * be computed.\n   */\n  protected abstract void updateScoreWithFeatures(D data);\n\n  /**\n   * Get the already accumulated score\n   */\n  protected final double getLogitScore() {\n    return score;\n  }\n\n  /**\n   * Returns the score as a value mapped between 0 and 1.\n   */\n  protected final double getSigmoidScore() {\n    return 1 / (1 + Math.exp(-score));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/CompositeFeatureContext.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\nimport java.util.function.Supplier;\nimport javax.annotation.Nullable;\n\nimport com.twitter.ml.api.FeatureContext;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchema;\n\n/**\n * An object to store feature context information to build models with.\n */\npublic class CompositeFeatureContext {\n  // legacy static feature context\n  private final FeatureContext legacyContext;\n  // a supplier for the context (well the schema itself) of the schema-based features\n  private final Supplier<ThriftSearchFeatureSchema> schemaSupplier;\n\n  public CompositeFeatureContext(\n      FeatureContext legacyContext,\n      @Nullable Supplier<ThriftSearchFeatureSchema> schemaSupplier) {\n    this.legacyContext = legacyContext;\n    this.schemaSupplier = schemaSupplier;\n  }\n\n  FeatureContext getLegacyContext() {\n    return legacyContext;\n  }\n\n  ThriftSearchFeatureSchema getFeatureSchema() {\n    if (schemaSupplier == null) {\n      throw new UnsupportedOperationException(\"Feature schema was not initialized\");\n    }\n    return schemaSupplier.get();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/DecisionForestModelsManager.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\nimport java.io.IOException;\nimport java.util.Collections;\nimport java.util.Map;\nimport java.util.function.Supplier;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.ml.api.FeatureContext;\nimport com.twitter.mlv2.trees.predictor.CartTree;\nimport com.twitter.mlv2.trees.scorer.DecisionForestScorer;\nimport com.twitter.search.common.file.AbstractFile;\nimport com.twitter.search.common.util.ml.models_manager.BaseModelsManager;\n\n/**\n * Loads Decision Forest based models and keep them in memory. Can also be scheduled to reload\n * models periodically.\n *\n * Note: Each instance is tied to a single {@link FeatureContext} instance. So, to load models\n * for different tasks, you should use different instances of the this class.\n */\npublic class DecisionForestModelsManager extends BaseModelsManager<DecisionForestScorer<CartTree>> {\n  private static final String MODEL_FILE_NAME = \"model.json\";\n\n  private final FeatureContext featureContext;\n\n  DecisionForestModelsManager(\n      Supplier<Map<String, AbstractFile>> activeModelsSupplier,\n      FeatureContext featureContext,\n      boolean shouldUnloadInactiveModels,\n      String statsPrefix\n  ) {\n    super(activeModelsSupplier, shouldUnloadInactiveModels, statsPrefix);\n    this.featureContext = featureContext;\n  }\n\n  @Override\n  public DecisionForestScorer<CartTree> readModelFromDirectory(AbstractFile modelBaseDir)\n      throws IOException {\n    String modelFilePath = modelBaseDir.getChild(MODEL_FILE_NAME).getPath();\n    return DecisionForestScorer.createCartTreeScorer(modelFilePath, featureContext);\n  }\n\n  /**\n   * Creates an instance that loads the models specified in a configuration file.\n   *\n   * Note that if the configuration file changes and it doesn't include a model that was present\n   * before, the model will be removed (i.e. it unloads models that are not active anymore).\n   */\n  public static DecisionForestModelsManager createUsingConfigFile(\n      AbstractFile configFile, FeatureContext featureContext, String statsPrefix) {\n    Preconditions.checkArgument(\n        configFile.canRead(), \"Config file is not readable: %s\", configFile.getPath());\n    return new DecisionForestModelsManager(\n        new ConfigSupplier(configFile), featureContext, true, statsPrefix);\n  }\n\n  /**\n   * Creates a no-op instance. It can be used for tests or when the models are disabled.\n   */\n  public static DecisionForestModelsManager createNoOp(String statsPrefix) {\n    return new DecisionForestModelsManager(\n        Collections::emptyMap, new FeatureContext(), false, statsPrefix) {\n      @Override\n      public void run() { }\n    };\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/DiscretizedFeature.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\nimport java.util.Arrays;\n\nimport com.google.common.base.Preconditions;\n\n/**\n * Represents a continuous feature that has been discretized into a set of disjoint ranges.\n *\n * Each range [a, b) is represented by the lower split point (a) and its associated weight.\n */\nclass DiscretizedFeature {\n\n  protected final double[] splitPoints;\n  protected final double[] weights;\n\n  /**\n   * Creates an instance from a list of split points and their corresponding weights.\n   *\n   * @param splitPoints Lower values of the ranges. The first entry must be Double.NEGATIVE_INFINITY\n   *  They must be sorted (in ascending order).\n   * @param  weights Weights for the splits.\n   */\n  protected DiscretizedFeature(double[] splitPoints, double[] weights) {\n    Preconditions.checkArgument(splitPoints.length == weights.length);\n    Preconditions.checkArgument(splitPoints.length > 1);\n    Preconditions.checkArgument(splitPoints[0] == Double.NEGATIVE_INFINITY,\n        \"First split point must be Double.NEGATIVE_INFINITY\");\n    this.splitPoints = splitPoints;\n    this.weights = weights;\n  }\n\n  public double getWeight(double value) {\n    // binarySearch returns (- insertionPoint - 1)\n    int index = Math.abs(Arrays.binarySearch(splitPoints, value) + 1) - 1;\n    return weights[index];\n  }\n\n  public boolean allValuesBelowThreshold(double minWeight) {\n    for (double weight : weights) {\n      if (Math.abs(weight) > minWeight) {\n        return false;\n      }\n    }\n    return true;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/DiscretizedFeatureRange.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\nimport com.google.common.base.Preconditions;\n\n/**\n * The discretized value range for a continous feature. After discretization a continuous feature\n * may become multiple discretized binary features, each occupying a range. This class stores this\n * range and a weight for it.\n */\npublic class DiscretizedFeatureRange {\n  protected final double minValue;\n  protected final double maxValue;\n  protected final double weight;\n\n  DiscretizedFeatureRange(double weight, String range) {\n    String[] limits = range.split(\"_\");\n    Preconditions.checkArgument(limits.length == 2);\n\n    this.minValue = parseRangeValue(limits[0]);\n    this.maxValue = parseRangeValue(limits[1]);\n    this.weight = weight;\n  }\n\n  private static double parseRangeValue(String value) {\n    if (\"inf\".equals(value)) {\n      return Double.POSITIVE_INFINITY;\n    } else if (\"-inf\".equals(value)) {\n      return Double.NEGATIVE_INFINITY;\n    } else {\n      return Double.parseDouble(value);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/LegacyModelBuilder.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\nimport java.util.Map;\n\nimport com.google.common.collect.HashMultimap;\nimport com.google.common.collect.Maps;\nimport com.google.common.collect.Multimap;\n\nimport com.twitter.ml.api.Feature;\nimport com.twitter.ml.api.FeatureContext;\nimport com.twitter.ml.api.FeatureParser;\nimport com.twitter.ml.api.transform.DiscretizerTransform;\n\n/**\n * The builder for a model based on the legacy (non-schema-based) features.\n * See also SchemaBasedModelBuilder.\n */\npublic final class LegacyModelBuilder extends BaseModelBuilder {\n\n  private final Map<String, Feature> featuresByName;\n  // for legacy features\n  private final Map<Feature<Boolean>, Double> binaryFeatures;\n  private final Map<Feature<Double>, Double> continuousFeatures;\n  private final Multimap<Feature<Double>, DiscretizedFeatureRange> discretizedFeatureRanges;\n\n  LegacyModelBuilder(String modelName, FeatureContext context) {\n    super(modelName);\n    featuresByName = getFeaturesByName(context);\n    binaryFeatures = Maps.newHashMap();\n    continuousFeatures = Maps.newHashMap();\n    discretizedFeatureRanges = HashMultimap.create();\n  }\n\n  private static Map<String, Feature> getFeaturesByName(FeatureContext featureContext) {\n    Map<String, Feature> featuresByName = Maps.newHashMap();\n    for (Feature<?> feature : featureContext.getAllFeatures()) {\n      featuresByName.put(feature.getFeatureName(), feature);\n    }\n    return featuresByName;\n  }\n\n  @Override\n  protected void addFeature(String baseName, double weight, FeatureParser parser) {\n    Feature feature = featuresByName.get(baseName);\n    if (feature != null) {\n      switch (feature.getFeatureType()) {\n        case BINARY:\n          binaryFeatures.put(feature, weight);\n          break;\n        case CONTINUOUS:\n          continuousFeatures.put(feature, weight);\n          break;\n        default:\n          throw new IllegalArgumentException(\n              String.format(\"Unsupported feature type: %s\", feature));\n      }\n    } else if (baseName.endsWith(DISCRETIZER_NAME_SUFFIX)\n        && parser.getExtension().containsKey(DiscretizerTransform.DEFAULT_RANGE_EXT)) {\n\n      String featureName =\n          baseName.substring(0, baseName.length() - DISCRETIZER_NAME_SUFFIX.length());\n\n      feature = featuresByName.get(featureName);\n      if (feature == null) {\n        return;\n      }\n\n      String rangeSpec = parser.getExtension().get(DiscretizerTransform.DEFAULT_RANGE_EXT);\n      discretizedFeatureRanges.put(feature, new DiscretizedFeatureRange(weight, rangeSpec));\n    }\n  }\n\n  @Override\n  public LightweightLinearModel build() {\n    Map<Feature<Double>, DiscretizedFeature> discretizedFeatures = Maps.newHashMap();\n    for (Feature<Double> feature : discretizedFeatureRanges.keySet()) {\n      DiscretizedFeature discretizedFeature =\n          BaseModelBuilder.buildFeature(discretizedFeatureRanges.get(feature));\n      if (!discretizedFeature.allValuesBelowThreshold(MIN_WEIGHT)) {\n        discretizedFeatures.put(feature, discretizedFeature);\n      }\n    }\n    return LightweightLinearModel.createForLegacy(\n        modelName, bias, binaryFeatures, continuousFeatures, discretizedFeatures);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/LightweightLinearModel.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\nimport java.io.BufferedReader;\nimport java.io.FileReader;\nimport java.io.IOException;\nimport java.util.Map;\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.ml.api.Feature;\nimport com.twitter.search.common.file.AbstractFile;\n\n/**\n * Provides an interface to the weights associated to the features of a linear model trained\n * with Prediction Engine.\n *\n * This class is used along with ScoreAccumulator to efficiently score instances. It supports only\n * a limited set of features:\n *\n * - Only linear models are supported.\n * - Only binary and continuous features (i.e. it doesn't support discrete/categorical features).\n * - It supports the MDL discretizer (but not the one based on trees).\n * - It doesn't support feature crossings.\n *\n * Instances of this class should be created using only the load methods (loadFromHdfs and\n * loadFromLocalFile).\n *\n * IMPORTANT:\n *\n * Use this class, and ScoreAccumulator, ONLY when runtime is a major concern. Otherwise, consider\n * using Prediction Engine as a library. Ideally, we should access directly the structures that\n * Prediction Engine creates when it loads a model, instead of parsing a text file with the\n * feature weights.\n *\n * The discretized feature bins created by MDL may be too fine to be displayed properly in the\n * parsed text file and there may be bins with the same min value. A binary search finding the\n * bin for a same feature value therefore may end up with different bins/scores in different runs,\n * producing unstable scores. See SEARCHQUAL-15957 for more detail.\n *\n * @see com.twitter.ml.tool.prediction.ModelInterpreter\n */\npublic class LightweightLinearModel {\n  protected final double bias;\n  protected final boolean schemaBased;\n  protected final String name;\n\n  // for legacy metadata based model\n  protected final Map<Feature<Boolean>, Double> binaryFeatures;\n  protected final Map<Feature<Double>, Double> continuousFeatures;\n  protected final Map<Feature<Double>, DiscretizedFeature> discretizedFeatures;\n\n  // for schema-based model\n  protected final Map<Integer, Double> binaryFeaturesById;\n  protected final Map<Integer, Double> continuousFeaturesById;\n  protected final Map<Integer, DiscretizedFeature> discretizedFeaturesById;\n\n  private static final String SCHEMA_BASED_SUFFIX = \".schema_based\";\n\n  LightweightLinearModel(\n      String modelName,\n      double bias,\n      boolean schemaBased,\n      @Nullable Map<Feature<Boolean>, Double> binaryFeatures,\n      @Nullable Map<Feature<Double>, Double> continuousFeatures,\n      @Nullable Map<Feature<Double>, DiscretizedFeature> discretizedFeatures,\n      @Nullable Map<Integer, Double> binaryFeaturesById,\n      @Nullable Map<Integer, Double> continuousFeaturesById,\n      @Nullable Map<Integer, DiscretizedFeature> discretizedFeaturesById) {\n\n    this.name = modelName;\n    this.bias = bias;\n    this.schemaBased = schemaBased;\n\n    // legacy feature maps\n    this.binaryFeatures =\n        schemaBased ? null : Preconditions.checkNotNull(binaryFeatures);\n    this.continuousFeatures =\n        schemaBased ? null : Preconditions.checkNotNull(continuousFeatures);\n    this.discretizedFeatures =\n        schemaBased ? null : Preconditions.checkNotNull(discretizedFeatures);\n\n    // schema based feature maps\n    this.binaryFeaturesById =\n        schemaBased ? Preconditions.checkNotNull(binaryFeaturesById) : null;\n    this.continuousFeaturesById =\n        schemaBased ? Preconditions.checkNotNull(continuousFeaturesById) : null;\n    this.discretizedFeaturesById =\n        schemaBased ? Preconditions.checkNotNull(discretizedFeaturesById) : null;\n  }\n\n  public String getName() {\n    return name;\n  }\n\n  /**\n   * Create model for legacy features\n   */\n  protected static LightweightLinearModel createForLegacy(\n      String modelName,\n      double bias,\n      Map<Feature<Boolean>, Double> binaryFeatures,\n      Map<Feature<Double>, Double> continuousFeatures,\n      Map<Feature<Double>, DiscretizedFeature> discretizedFeatures) {\n    return new LightweightLinearModel(modelName, bias, false,\n        binaryFeatures, continuousFeatures, discretizedFeatures,\n        null, null, null);\n  }\n\n  /**\n   * Create model for schema-based features\n   */\n  protected static LightweightLinearModel createForSchemaBased(\n      String modelName,\n      double bias,\n      Map<Integer, Double> binaryFeaturesById,\n      Map<Integer, Double> continuousFeaturesById,\n      Map<Integer, DiscretizedFeature> discretizedFeaturesById) {\n    return new LightweightLinearModel(modelName, bias, true,\n        null, null, null,\n        binaryFeaturesById, continuousFeaturesById, discretizedFeaturesById);\n  }\n\n  public boolean isSchemaBased() {\n    return schemaBased;\n  }\n\n  /**\n   * Loads a model from a text file.\n   *\n   * See the javadoc of the constructor for more details on how to create the file from a trained\n   * Prediction Engine model.\n   *\n   * If schemaBased is true, the featureContext is ignored.\n   */\n  public static LightweightLinearModel load(\n      String modelName,\n      BufferedReader reader,\n      boolean schemaBased,\n      CompositeFeatureContext featureContext) throws IOException {\n\n    ModelBuilder builder = schemaBased\n        ? new SchemaBasedModelBuilder(modelName, featureContext.getFeatureSchema())\n        : new LegacyModelBuilder(modelName, featureContext.getLegacyContext());\n    String line;\n    while ((line = reader.readLine()) != null) {\n      builder.parseLine(line);\n    }\n    return builder.build();\n  }\n\n  /**\n   * Loads a model from a local text file.\n   *\n   * See the javadoc of the constructor for more details on how to create the file from a trained\n   * Prediction Engine model.\n   */\n  public static LightweightLinearModel loadFromLocalFile(\n      String modelName,\n      CompositeFeatureContext featureContext,\n      String fileName) throws IOException {\n    try (BufferedReader reader = new BufferedReader(new FileReader(fileName))) {\n      boolean schemaBased = modelName.endsWith(SCHEMA_BASED_SUFFIX);\n      return load(modelName, reader, schemaBased, featureContext);\n    }\n  }\n\n  /**\n   * Loads a model from a file in the local filesystem or in HDFS.\n   *\n   * See the javadoc of the constructor for more details on how to create the file from a trained\n   * Prediction Engine model.\n   */\n  public static LightweightLinearModel load(\n      String modelName, CompositeFeatureContext featureContext, AbstractFile modelFile)\n      throws IOException {\n    try (BufferedReader reader = modelFile.getCharSource().openBufferedStream()) {\n      boolean schemaBased = modelName.endsWith(SCHEMA_BASED_SUFFIX);\n      return load(modelName, reader, schemaBased, featureContext);\n    }\n  }\n\n  public String toString() {\n    return String.format(\"LightweightLinearModel. {bias=%s binary=%s continuous=%s discrete=%s}\",\n        this.bias, this.binaryFeatures, this.continuousFeatures, this.discretizedFeatures);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/ModelBuilder.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\n/**\n * A builder interface to build a LightweightLinearModel.\n */\npublic interface ModelBuilder {\n  /**\n   * parses a line of the model file and updates the build state\n   */\n  ModelBuilder parseLine(String line);\n\n  /**\n   * builds the model\n   */\n  LightweightLinearModel build();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/ModelLoader.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\nimport java.io.IOException;\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.base.Optional;\nimport com.google.common.base.Supplier;\nimport com.google.common.base.Suppliers;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.file.AbstractFile;\nimport com.twitter.search.common.file.FileUtils;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\n\n/**\n * Loads LightweightLinearModel objects from a directory and provides an interface for reloading\n * them periodically.\n *\n * All the models must support the same features (defined by a FeatureContext) and they are\n * identified by the name of the subdirectory. This is the required directory structure:\n *\n *  /path/to/base-directory\n *      one-model/model.tsv\n *      another-model/model.tsv\n *      experimental-model/model.tsv\n *\n * Each subdirectory must contain a file named 'model.tsv' in the format required by\n * LightweightLinearModel.\n */\npublic class ModelLoader implements Runnable {\n\n  private static final Logger LOG = LoggerFactory.getLogger(ModelLoader.class);\n  private static final String MODEL_FILE_NAME = \"model.tsv\";\n\n  private final CompositeFeatureContext featureContext;\n  private final Supplier<AbstractFile> directorySupplier;\n\n  private final Map<String, LightweightLinearModel> models;\n  private final Map<String, Long> lastModifiedMsByModel;\n\n  private final SearchLongGauge lastModelLoadedAtMs;\n  private final SearchLongGauge numModels;\n  private final SearchCounter numLoads;\n  private final SearchCounter numErrors;\n\n  /**\n   * Creates a new instance for a feature context and a base directory.\n   *\n   * It exports 4 counters:\n   *\n   *   ${counterPrefix}_last_loaded:\n   *      Timestamp (in ms) when the last model was loaded.\n   *   ${counterPrefix}_num_models:\n   *      Number of models currently loaded.\n   *   ${counterPrefix}_num_loads:\n   *      Number of succesful model loads.\n   *   ${counterPrefix}_num_errors:\n   *      Number of errors occurred while loading the models.\n   */\n  protected ModelLoader(\n      CompositeFeatureContext featureContext,\n      Supplier<AbstractFile> directorySupplier,\n      String counterPrefix,\n      SearchStatsReceiver statsReceiver) {\n    this.featureContext = featureContext;\n\n    // This function returns the base directory every time we call 'run'. We use a function instead\n    // of using directly an AbstractFile instance, in case that we can't obtain an instance at\n    // initialization time (e.g. if there's an issue with HDFS).\n    this.directorySupplier = directorySupplier;\n    this.models = Maps.newConcurrentMap();\n    this.lastModifiedMsByModel = Maps.newConcurrentMap();\n\n    this.lastModelLoadedAtMs = statsReceiver.getLongGauge(counterPrefix + \"last_loaded\");\n    this.numModels = statsReceiver.getLongGauge(counterPrefix + \"num_models\");\n    this.numLoads = statsReceiver.getCounter(counterPrefix + \"num_loads\");\n    this.numErrors = statsReceiver.getCounter(counterPrefix + \"num_errors\");\n  }\n\n  public Optional<LightweightLinearModel> getModel(String name) {\n    return Optional.fromNullable(models.get(name));\n  }\n\n  /**\n   * Loads the models from the base directory.\n   *\n   * It doesn't load a model if its file has not been modified since the last time it was loaded.\n   *\n   * This method doesn't delete previously loaded models if their directories are not available.\n   */\n  @Override\n  public void run() {\n    try {\n      AbstractFile baseDirectory = directorySupplier.get();\n      List<AbstractFile> modelDirectories =\n          Lists.newArrayList(baseDirectory.listFiles(IS_MODEL_DIR));\n      for (AbstractFile directory : modelDirectories) {\n        try {\n          // Note that the modelName is the directory name, if it ends with \".schema_based\", the\n          // model will be loaded as a schema-based model.\n          String modelName = directory.getName();\n          AbstractFile modelFile = directory.getChild(MODEL_FILE_NAME);\n          long currentLastModified = modelFile.getLastModified();\n          Long lastModified = lastModifiedMsByModel.get(modelName);\n          if (lastModified == null || lastModified < currentLastModified) {\n            LightweightLinearModel model =\n                LightweightLinearModel.load(modelName, featureContext, modelFile);\n            if (!models.containsKey(modelName)) {\n              LOG.info(\"Loading model {}.\", modelName);\n            }\n            models.put(modelName, model);\n            lastModifiedMsByModel.put(modelName, currentLastModified);\n            lastModelLoadedAtMs.set(System.currentTimeMillis());\n            numLoads.increment();\n            LOG.debug(\"Model: {}\", model);\n          } else {\n            LOG.debug(\"Directory for model {} has not changed.\", modelName);\n          }\n        } catch (Exception e) {\n          LOG.error(\"Error loading model from directory: \" + directory.getPath(), e);\n          this.numErrors.increment();\n        }\n      }\n      if (numModels.get() != models.size()) {\n        LOG.info(\"Finished loading models. Model names: {}\", models.keySet());\n      }\n      this.numModels.set(models.size());\n    } catch (IOException e) {\n      LOG.error(\"Error loading models\", e);\n      this.numErrors.increment();\n    }\n  }\n\n  /**\n   * Creates an instance that loads models from a directory (local or from HDFS).\n   */\n  public static ModelLoader forDirectory(\n      final AbstractFile directory,\n      CompositeFeatureContext featureContext,\n      String counterPrefix,\n      SearchStatsReceiver statsReceiver) {\n    Supplier<AbstractFile> directorySupplier = Suppliers.ofInstance(directory);\n    return new ModelLoader(featureContext, directorySupplier, counterPrefix, statsReceiver);\n  }\n\n  /**\n   * Creates an instance that loads models from HDFS.\n   */\n  public static ModelLoader forHdfsDirectory(\n      final String nameNode,\n      final String directory,\n      CompositeFeatureContext featureContext,\n      String counterPrefix,\n      SearchStatsReceiver statsReceiver) {\n    Supplier<AbstractFile> directorySupplier =\n        () -> FileUtils.getHdfsFileHandle(directory, nameNode);\n    return new ModelLoader(featureContext, directorySupplier, counterPrefix, statsReceiver);\n  }\n\n  private static final AbstractFile.Filter IS_MODEL_DIR = file -> {\n    try {\n      if (file.isDirectory()) {\n        AbstractFile modelFile = file.getChild(MODEL_FILE_NAME);\n        return (modelFile != null) && modelFile.canRead();\n      }\n    } catch (IOException e) {\n      LOG.error(\"Error reading file: \" + file, e);\n    }\n    return false;\n  };\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/PredictionEngineModelsManager.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\nimport java.util.Collections;\nimport java.util.Map;\nimport java.util.function.Supplier;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.ml.prediction.core.PredictionEngine;\nimport com.twitter.ml.prediction.core.PredictionEngineFactory;\nimport com.twitter.ml.prediction.core.PredictionEngineLoadingException;\nimport com.twitter.ml.vw.constant.SnapshotConstants;\nimport com.twitter.search.common.file.AbstractFile;\nimport com.twitter.search.common.util.ml.models_manager.BaseModelsManager;\n\n/**\n * Loads PredictionEngine models from a model provider (config or fixed directory)\n * and keeps them in memory. Can also reload models periodically by querying the\n * same model provider source.\n */\npublic class PredictionEngineModelsManager extends BaseModelsManager<PredictionEngine> {\n\n  PredictionEngineModelsManager(\n      Supplier<Map<String, AbstractFile>> activeModelsSupplier,\n      boolean shouldUnloadInactiveModels,\n      String statsPrefix) {\n    super(activeModelsSupplier, shouldUnloadInactiveModels, statsPrefix);\n  }\n\n  @Override\n  public PredictionEngine readModelFromDirectory(AbstractFile modelBaseDir)\n      throws PredictionEngineLoadingException {\n    // We need to add the 'hdfs://' prefix, otherwise PredictionEngine will treat it as a\n    // path in the local filesystem.\n    PredictionEngine predictionEngine = new PredictionEngineFactory()\n        .createFromSnapshot(\n            \"hdfs://\" + modelBaseDir.getPath(), SnapshotConstants.FIXED_PATH);\n\n    predictionEngine.initialize();\n\n    return predictionEngine;\n  }\n\n  /**\n   * Creates an instance that loads the models specified in a configuration file.\n   *\n   * Note that if the configuration file changes and it doesn't include a model that was present\n   * before, the model will be removed (i.e. it unloads models that are not active anymore).\n   */\n  public static PredictionEngineModelsManager createUsingConfigFile(\n      AbstractFile configFile, String statsPrefix) {\n    Preconditions.checkArgument(\n        configFile.canRead(), \"Config file is not readable: %s\", configFile.getPath());\n    return new PredictionEngineModelsManager(new ConfigSupplier(configFile), true, statsPrefix);\n  }\n\n  /**\n   * Creates a no-op instance. It can be used for tests or when the models are disabled.\n   */\n  public static PredictionEngineModelsManager createNoOp(String statsPrefix) {\n    return new PredictionEngineModelsManager(Collections::emptyMap, false, statsPrefix) {\n      @Override\n      public void run() { }\n    };\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/SchemaBasedModelBuilder.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\nimport java.util.Map;\nimport java.util.stream.Collectors;\n\nimport com.google.common.collect.HashMultimap;\nimport com.google.common.collect.Maps;\nimport com.google.common.collect.Multimap;\n\nimport com.twitter.ml.api.FeatureParser;\nimport com.twitter.ml.api.transform.DiscretizerTransform;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchema;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchemaEntry;\n\n/**\n * Builds a model with schema-based features, here all features are tracked by Id.\n * This class is very similar to LegacyModelBuilder, which will eventually be deprecated.\n */\npublic class SchemaBasedModelBuilder extends BaseModelBuilder {\n  private final Map<String, FeatureData> featuresByName;\n  private final Map<Integer, Double> binaryFeatures;\n  private final Map<Integer, Double> continuousFeatures;\n  private final Multimap<Integer, DiscretizedFeatureRange> discretizedFeatureRanges;\n\n  /**\n   * a class to hold feature information\n   */\n  static class FeatureData {\n    private final ThriftSearchFeatureSchemaEntry entry;\n    private final int id;\n\n    public FeatureData(ThriftSearchFeatureSchemaEntry entry, int id) {\n      this.entry = entry;\n      this.id = id;\n    }\n  }\n\n  SchemaBasedModelBuilder(String modelName, ThriftSearchFeatureSchema featureSchema) {\n    super(modelName);\n    featuresByName = getFeatureDataMap(featureSchema);\n    binaryFeatures = Maps.newHashMap();\n    continuousFeatures = Maps.newHashMap();\n    discretizedFeatureRanges = HashMultimap.create();\n  }\n\n  /**\n   * Creates a map from feature name to thrift entries\n   */\n  private static Map<String, FeatureData> getFeatureDataMap(\n      ThriftSearchFeatureSchema schema) {\n    return schema.getEntries().entrySet().stream()\n        .collect(Collectors.toMap(\n            e -> e.getValue().getFeatureName(),\n            e -> new FeatureData(e.getValue(), e.getKey())\n        ));\n  }\n\n  @Override\n  protected void addFeature(String baseName, double weight, FeatureParser parser) {\n    FeatureData feature = featuresByName.get(baseName);\n    if (feature != null) {\n      switch (feature.entry.getFeatureType()) {\n        case BOOLEAN_VALUE:\n          binaryFeatures.put(feature.id, weight);\n          break;\n        case INT32_VALUE:\n        case LONG_VALUE:\n        case DOUBLE_VALUE:\n          continuousFeatures.put(feature.id, weight);\n          break;\n        default:\n          // other values are not supported yet\n          throw new IllegalArgumentException(\n              String.format(\"Unsupported feature type: %s\", feature));\n      }\n    } else if (baseName.endsWith(DISCRETIZER_NAME_SUFFIX)\n        && parser.getExtension().containsKey(DiscretizerTransform.DEFAULT_RANGE_EXT)) {\n\n      String featureName =\n          baseName.substring(0, baseName.length() - DISCRETIZER_NAME_SUFFIX.length());\n\n      feature = featuresByName.get(featureName);\n      if (feature == null) {\n        return;\n      }\n\n      String rangeSpec = parser.getExtension().get(DiscretizerTransform.DEFAULT_RANGE_EXT);\n      discretizedFeatureRanges.put(feature.id, new DiscretizedFeatureRange(weight, rangeSpec));\n    }\n  }\n\n  @Override\n  public LightweightLinearModel build() {\n    Map<Integer, DiscretizedFeature> discretizedFeatures = Maps.newHashMap();\n    for (Integer feature : discretizedFeatureRanges.keySet()) {\n      DiscretizedFeature discretizedFeature =\n          BaseModelBuilder.buildFeature(discretizedFeatureRanges.get(feature));\n      if (!discretizedFeature.allValuesBelowThreshold(MIN_WEIGHT)) {\n        discretizedFeatures.put(feature, discretizedFeature);\n      }\n    }\n    return LightweightLinearModel.createForSchemaBased(\n        modelName, bias, binaryFeatures, continuousFeatures, discretizedFeatures);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/prediction_engine/SchemaBasedScoreAccumulator.java",
    "content": "package com.twitter.search.common.util.ml.prediction_engine;\n\nimport java.util.Map;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.features.thrift.ThriftSearchResultFeatures;\nimport com.twitter.search.modeling.common.TweetFeaturesUtils;\n\n/**\n * Score accumulator for schema-based features.\n */\npublic class SchemaBasedScoreAccumulator extends BaseScoreAccumulator<ThriftSearchResultFeatures> {\n\n  public SchemaBasedScoreAccumulator(LightweightLinearModel model) {\n    super(model);\n    Preconditions.checkState(model.isSchemaBased(),\n        \"Cannot create SchemaBasedScoreAccumulator with a non-schema-based model: %s\",\n        model.getName());\n  }\n\n  @Override\n  protected final void updateScoreWithFeatures(ThriftSearchResultFeatures featureData) {\n    // go through all features available and apply all those available in the model\n    addSchemaBooleanFeatures(featureData.getBoolValues());\n    addSchemaContinuousFeatures(featureData.getIntValues());\n    addSchemaContinuousFeatures(featureData.getLongValues());\n    addSchemaContinuousFeatures(featureData.getDoubleValues());\n  }\n\n  private void addSchemaBooleanFeatures(Map<Integer, Boolean> booleanMap) {\n    if (booleanMap == null || booleanMap.isEmpty()) {\n      return;\n    }\n    for (Map.Entry<Integer, Boolean> entry : booleanMap.entrySet()) {\n      if (entry.getValue()) {\n        score += model.binaryFeaturesById.getOrDefault(entry.getKey(), 0.0);\n      }\n    }\n  }\n\n  private void addSchemaContinuousFeatures(Map<Integer, ? extends Number> valueMap) {\n    if (valueMap == null || valueMap.isEmpty()) {\n      return;\n    }\n    for (Map.Entry<Integer, ? extends Number> entry : valueMap.entrySet()) {\n      Integer id = entry.getKey();\n      if (TweetFeaturesUtils.isFeatureDiscrete(id)) {\n        continue;  // we don't process any discrete features now\n      }\n      Double weight = model.continuousFeaturesById.get(id);\n      if (weight != null) {\n        // found non-discretized entry\n        score += weight * entry.getValue().doubleValue();\n      } else {\n        DiscretizedFeature discretizedFeature = model.discretizedFeaturesById.get(id);\n        if (discretizedFeature != null) {\n          // Use only the weight of the discretized feature (there's no need to multiply it)\n          score += discretizedFeature.getWeight(entry.getValue().doubleValue());\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/tensorflow_engine/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"3rdparty/jvm/org/tensorflow\",\n        \"finatra/inject/inject-slf4j/src/main/scala/com/twitter/inject\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/java/com/twitter/search/common/schema\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/util/ml/models_manager\",\n        \"src/thrift/com/twitter/search/common:features-java\",\n        \"tensorflow/tfcompute-java/src/main/java/com/twitter/tfcompute_java\",\n        \"twml/runtime/src/main/scala/com/twitter/twml/runtime/lib\",\n        \"twml/runtime/src/main/scala/com/twitter/twml/runtime/models\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/common/util/ml/tensorflow_engine/TensorflowModelsManager.java",
    "content": "package com.twitter.search.common.util.ml.tensorflow_engine;\n\nimport java.io.IOException;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.function.Supplier;\n\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport org.tensorflow.SavedModelBundle;\nimport org.tensorflow.Session;\n\nimport com.twitter.ml.api.FeatureUtil;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchema;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchemaEntry;\nimport com.twitter.search.common.file.AbstractFile;\nimport com.twitter.search.common.schema.DynamicSchema;\nimport com.twitter.search.common.util.ml.models_manager.BaseModelsManager;\nimport com.twitter.tfcompute_java.TFModelRunner;\nimport com.twitter.tfcompute_java.TFSessionInit;\nimport com.twitter.twml.runtime.lib.TwmlLoader;\nimport com.twitter.twml.runtime.models.ModelLocator;\nimport com.twitter.twml.runtime.models.ModelLocator$;\nimport com.twitter.util.Await;\n\n/**\n * TensorflowModelsManager manages the lifecyle of TF models.\n */\npublic class TensorflowModelsManager extends BaseModelsManager<TFModelRunner>  {\n\n  private static final Logger LOG = LoggerFactory.getLogger(TensorflowModelsManager.class);\n\n  private static final String[] TF_TAGS = new String[] {\"serve\"};\n\n  private volatile Map<Integer, Long> featureSchemaIdToMlApiId = new HashMap<Integer, Long>();\n\n  static {\n    TwmlLoader.load();\n  }\n\n  public static final TensorflowModelsManager NO_OP_MANAGER =\n    createNoOp(\"no_op_manager\");\n\n  public TensorflowModelsManager(\n      Supplier<Map<String, AbstractFile>> activeModelsSupplier,\n      boolean shouldUnloadInactiveModels,\n      String statsPrefix\n  ) {\n    this(\n      activeModelsSupplier,\n      shouldUnloadInactiveModels,\n      statsPrefix,\n      () -> true,\n      () -> true,\n      null\n    );\n  }\n\n  public TensorflowModelsManager(\n      Supplier<Map<String, AbstractFile>> activeModelsSupplier,\n      boolean shouldUnloadInactiveModels,\n      String statsPrefix,\n      Supplier<Boolean> serveModels,\n      Supplier<Boolean> loadModels,\n      DynamicSchema dynamicSchema\n  ) {\n    super(\n      activeModelsSupplier,\n      shouldUnloadInactiveModels,\n      statsPrefix,\n      serveModels,\n      loadModels\n    );\n    if (dynamicSchema != null) {\n      updateFeatureSchemaIdToMlIdMap(dynamicSchema.getSearchFeatureSchema());\n    }\n  }\n\n  /**\n   * The ML API feature ids for tensorflow scoring are hashes of their feature names. This hashing\n   * could be expensive to do for every search request. Instead, allow the map from schema feature\n   * id to ML API id to be updated whenever the schema is reloaded.\n   */\n  public void updateFeatureSchemaIdToMlIdMap(ThriftSearchFeatureSchema schema) {\n    HashMap<Integer, Long> newFeatureSchemaIdToMlApiId = new HashMap<Integer, Long>();\n    Map<Integer, ThriftSearchFeatureSchemaEntry> featureEntries = schema.getEntries();\n    for (Map.Entry<Integer, ThriftSearchFeatureSchemaEntry> entry : featureEntries.entrySet()) {\n      long mlApiFeatureId = FeatureUtil.featureIdForName(entry.getValue().getFeatureName());\n      newFeatureSchemaIdToMlApiId.put(entry.getKey(), mlApiFeatureId);\n    }\n\n    featureSchemaIdToMlApiId = newFeatureSchemaIdToMlApiId;\n  }\n\n  public Map<Integer, Long> getFeatureSchemaIdToMlApiId() {\n    return featureSchemaIdToMlApiId;\n  }\n\n  /**\n   * If the manager is not enabled, it won't fetch TF models.\n   */\n  public boolean isEnabled() {\n    return true;\n  }\n\n  /**\n   * Load an individual model and make it available for inference.\n   */\n  public TFModelRunner readModelFromDirectory(\n    AbstractFile modelDir) throws IOException {\n\n    ModelLocator modelLocator =\n      ModelLocator$.MODULE$.apply(\n        modelDir.toString(),\n        modelDir.toURI()\n      );\n\n    try {\n      Await.result(modelLocator.ensureLocalPresent(true));\n    } catch (Exception e) {\n      LOG.error(\"Couldn't find model \" + modelDir.toString(), e);\n      throw new IOException(\"Couldn't find model \" + modelDir.toString());\n    }\n\n    Session session = SavedModelBundle.load(modelLocator.localPath(), TF_TAGS).session();\n\n    return new TFModelRunner(session);\n  }\n\n\n  /**\n   * Initialize Tensorflow intra and inter op thread pools.\n   * See `ConfigProto.[intra|inter]_op_parallelism_threads` documentation for more information:\n   * https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/protobuf/config.proto\n   * Initialization should happen only once.\n   * Default values for Tensorflow are:\n   * intraOpParallelismThreads = 0 which means that TF will pick an appropriate default.\n   * interOpParallelismThreads = 0 which means that TF will pick an appropriate default.\n   * operation_timeout_in_ms = 0 which means that no timeout will be applied.\n   */\n  public static void initTensorflowThreadPools(\n    int intraOpParallelismThreads,\n    int interOpParallelismThreads) {\n    new TFSessionInit(intraOpParallelismThreads, interOpParallelismThreads, 0);\n  }\n\n  /**\n   * Creates a no-op instance. It can be used for tests or when the models are disabled.\n   */\n  public static TensorflowModelsManager createNoOp(String statsPrefix) {\n    return new TensorflowModelsManager(Collections::emptyMap, false, statsPrefix) {\n      @Override\n      public void run() { }\n\n      @Override\n      public boolean isEnabled() {\n        return false;\n      }\n\n      @Override\n      public void updateFeatureSchemaIdToMlIdMap(ThriftSearchFeatureSchema schema) { }\n    };\n  }\n\n /**\n   * Creates an instance that loads the models based on a ConfigSupplier.\n   */\n  public static TensorflowModelsManager createUsingConfigFile(\n      AbstractFile configFile,\n      boolean shouldUnloadInactiveModels,\n      String statsPrefix,\n      Supplier<Boolean> serveModels,\n      Supplier<Boolean> loadModels,\n      DynamicSchema dynamicSchema) {\n    Preconditions.checkArgument(\n        configFile.canRead(), \"Config file is not readable: %s\", configFile.getPath());\n    return new TensorflowModelsManager(\n      new ConfigSupplier(configFile),\n      shouldUnloadInactiveModels,\n      statsPrefix,\n      serveModels,\n      loadModels,\n      dynamicSchema\n    );\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/BUILD",
    "content": "java_library(\n    sources = [\"**/*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/it/unimi/dsi:fastutil\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-common\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-smartcn\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-facet\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-queries\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/collections\",\n        \"src/java/com/twitter/search/common/encoding/docvalues\",\n        \"src/java/com/twitter/search/common/encoding/features\",\n        \"src/java/com/twitter/search/common/facets\",\n        \"src/java/com/twitter/search/common/hashtable\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/relevance/features\",\n        \"src/java/com/twitter/search/common/schema\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/common/search\",\n        \"src/java/com/twitter/search/common/util:log_format_util\",\n        \"src/java/com/twitter/search/common/util/analysis\",\n        \"src/java/com/twitter/search/common/util/hash\",\n        \"src/java/com/twitter/search/common/util/io:flushable\",\n        \"src/thrift/com/twitter/search/common:constants-java\",\n        \"src/thrift/com/twitter/search/common:facets-java\",\n        \"src/thrift/com/twitter/search/common:schema-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/README.md",
    "content": "# Search Index (Earlybird) core classes \n\n> **TL;DR** Earlybird (Search Index) find tweets from people you follow, rank them, and serve tweets to Home.\n\n## What is Earlybird (Search Index)\n\n[Earlybird](http://notes.stephenholiday.com/Earlybird.pdf) is a **real-time search system** based on [Apache Lucene](https://lucene.apache.org/) to support the high volume of queries and content updates. The major use cases are Relevance Search (specifically, Text search) and Timeline In-network Tweet retrieval (or UserID based search). It is designed to enable the efficient indexing and querying of billions of tweets, and to provide low-latency search results, even with heavy query loads. \n\n## Directory Structure\nThe project consists of several packages and files, which can be summarized as follows:\n\n\n* `facets/`: This subdirectory contains classes responsible for facet counting and processing. Some key classes include EarlybirdFacets, EarlybirdFacetsFactory, FacetAccumulator, and FacetCountAggregator. The classes handle facet counting, facet iterators, facet label providers, and facet response rewriting.\n* `index/`: This directory contains the indexing and search infra files, with several subdirectories for specific components.\n  * `column/`: This subdirectory contains classes related to column-stride field indexes, including ColumnStrideByteIndex, ColumnStrideIntIndex, ColumnStrideLongIndex, and various optimized versions of these indexes. These classes deal with managing and updating doc values.\n  * `extensions/`: This subdirectory contains classes for index extensions, including EarlybirdIndexExtensionsData, EarlybirdIndexExtensionsFactory, and EarlybirdRealtimeIndexExtensionsData.\n  * `inverted/`: This subdirectory focuses on the inverted index and its components, such as InMemoryFields, IndexOptimizer, InvertedIndex, and InvertedRealtimeIndex. It also contains classes for managing and processing posting lists and term dictionaries, like EarlybirdPostingsEnum, FSTTermDictionary, and MPHTermDictionary.\n  * `util/`: This subdirectory contains utility classes for managing search iterators and filters, such as AllDocsIterator, RangeDISI, RangeFilterDISI, and SearchSortUtils. The system appears to be designed to handle search indexing and facet counting efficiently. Key components include an inverted index, various types of posting lists, and term dictionaries. Facet counting and processing is handled by specialized classes within the facets subdirectory. The overall structure indicates a well-organized and modular search indexing system that can be maintained and extended as needed.\n\n## Related Services\n* The Earlybirds main classes. See `src/java/com/twitter/search/earlybird/`\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/AbstractFacetCountingArray.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.inverted.IntBlockPool;\n\n/**\n * AbstractFacetCountingArray implements a lookup from a doc ID to an unordered list of facets.\n * A facet is a pair of (term ID, field ID), which could represent,\n * for example (\"http://twitter.com\", \"links\").\n *\n * Internally, we have two data structures: A map from doc ID to an int and a pool of ints. We refer\n * to the values contained in these structures as packed values. A packed value can either be a\n * pointer to a location in the pool, an encoded facet or a sentinel value. Pointers always have\n * their high bit set to 1.\n *\n * If a document has just one facet, we will store the encoded facet in the map, and nothing in the\n * pool. Otherwise, the map will contain a pointer into the int pool.\n *\n * The int pool is encoded in a block-allocated linked list.\n * See {@link AbstractFacetCountingArray#collectForDocId} for details on how to traverse the list.\n */\npublic abstract class AbstractFacetCountingArray implements Flushable {\n  private static final Logger LOG = LoggerFactory.getLogger(AbstractFacetCountingArray.class);\n\n  private static final FacetCountIterator EMPTY_ITERATOR = new FacetCountIterator() {\n    @Override\n    public void collect(int docID) {\n      // noop\n    }\n  };\n\n  public static final AbstractFacetCountingArray EMPTY_ARRAY = new AbstractFacetCountingArray() {\n    @Override\n    public final FacetCountIterator getIterator(EarlybirdIndexSegmentAtomicReader reader,\n                                                FacetCountState<?> countState,\n                                                FacetCountIteratorFactory iteratorFactory) {\n      return EMPTY_ITERATOR;\n    }\n\n    @Override\n    public final int getFacet(int docID) {\n      return UNASSIGNED;\n    }\n\n    @Override\n    public final void setFacet(int docID, int facetID) {\n    }\n\n    @Override\n    public final AbstractFacetCountingArray rewriteAndMapIDs(\n        Map<Integer, int[]> termIDMapper,\n        DocIDToTweetIDMapper originalTweetIdMapper,\n        DocIDToTweetIDMapper optimizedTweetIdMapper) {\n      return this;\n    }\n\n    @Override\n    public <T extends Flushable> Handler<T> getFlushHandler() {\n      return null;\n    }\n  };\n\n  protected class ArrayFacetCountIterator extends FacetCountIterator {\n    @Override\n    public void collect(int docID) {\n      collectForDocId(docID, this);\n    }\n  }\n\n  private static final int NUM_BITS_TERM_ID = 27;\n  private static final int TERM_ID_MASK = (1 << NUM_BITS_TERM_ID) - 1;\n\n  private static final int NUM_BITS_FIELD_ID = 4;\n  private static final int FIELD_ID_MASK = (1 << NUM_BITS_FIELD_ID) - 1;\n\n  private static final int HIGHEST_ORDER_BIT = Integer.MIN_VALUE;  // 1L << 31\n  private static final int HIGHEST_ORDER_BIT_INVERSE_MASK = HIGHEST_ORDER_BIT - 1;\n\n  protected static final int UNASSIGNED = Integer.MAX_VALUE;\n\n  protected static final int decodeTermID(int facetID) {\n    if (facetID != UNASSIGNED) {\n      int termID = facetID & TERM_ID_MASK;\n      return termID;\n    }\n\n    return EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND;\n  }\n\n  protected static final int decodeFieldID(int facetID) {\n    return (facetID >>> NUM_BITS_TERM_ID) & FIELD_ID_MASK;\n  }\n\n  protected static final int encodeFacetID(int fieldID, int termID) {\n    return ((fieldID & FIELD_ID_MASK) << NUM_BITS_TERM_ID) | (termID & TERM_ID_MASK);\n  }\n\n  protected static final int decodePointer(int value) {\n    return value & HIGHEST_ORDER_BIT_INVERSE_MASK;\n  }\n\n  protected static final int encodePointer(int value) {\n    return value | HIGHEST_ORDER_BIT;\n  }\n\n  protected static final boolean isPointer(int value) {\n    return (value & HIGHEST_ORDER_BIT) != 0;\n  }\n\n  private final IntBlockPool facetsPool;\n\n  protected AbstractFacetCountingArray() {\n    facetsPool = new IntBlockPool(\"facets\");\n  }\n\n  protected AbstractFacetCountingArray(IntBlockPool facetsPool) {\n    this.facetsPool = facetsPool;\n  }\n\n  /**\n   * Returns an iterator to iterate all docs/facets stored in this FacetCountingArray.\n   */\n  public FacetCountIterator getIterator(\n      EarlybirdIndexSegmentAtomicReader reader,\n      FacetCountState<?> countState,\n      FacetCountIteratorFactory iteratorFactory) {\n    Preconditions.checkNotNull(countState);\n    Preconditions.checkNotNull(reader);\n\n    List<FacetCountIterator> iterators = new ArrayList<>();\n    for (Schema.FieldInfo fieldInfo : countState.getSchema().getCsfFacetFields()) {\n      if (countState.isCountField(fieldInfo)) {\n        // Rather than rely on the normal facet counting array, we read from a column stride\n        // field using a custom implementation of FacetCountIterator.\n        // This optimization is due to two factors:\n        //  1) for the from_user_id_csf facet, every document has a from user id,\n        //     but many documents contain no other facets.\n        //  2) we require from_user_id and shared_status_id to be in a column stride field\n        //     for other uses.\n        try {\n          iterators.add(iteratorFactory.getFacetCountIterator(reader, fieldInfo));\n        } catch (IOException e) {\n          String facetName = fieldInfo.getFieldType().getFacetName();\n          LOG.error(\"Failed to construct iterator for \" + facetName + \" facet\", e);\n        }\n      }\n    }\n    if (iterators.size() == 0) {\n      return new ArrayFacetCountIterator();\n    }\n    if (iterators.size() < countState.getNumFieldsToCount()) {\n      iterators.add(new ArrayFacetCountIterator());\n    }\n    return new CompositeFacetCountIterator(iterators);\n  }\n\n  /**\n   * Collects facets of the document with the provided docID.\n   * See {@link FacetCountingArrayWriter#addFacet} for details on the format of the int pool.\n   */\n  public void collectForDocId(int docID, FacetTermCollector collector) {\n    int firstValue = getFacet(docID);\n    if (firstValue == UNASSIGNED) {\n      return;  // no facet\n    }\n    if (!isPointer(firstValue)) {\n      // highest order bit not set, only one facet for this document.\n      collector.collect(docID, decodeTermID(firstValue), decodeFieldID(firstValue));\n      return;\n    }\n\n    // multiple facets, traverse the linked list to find all of the facets for this document.\n    int pointer = decodePointer(firstValue);\n    while (true) {\n      int packedValue = facetsPool.get(pointer);\n      // UNASSIGNED is a sentinel value indicating that we have reached the end of the linked list.\n      if (packedValue == UNASSIGNED) {\n        return;\n      }\n\n      if (isPointer(packedValue)) {\n        // If the packedValue is a pointer, we need to skip over some ints to reach the facets for\n        // this document.\n        pointer = decodePointer(packedValue);\n      } else {\n        // If the packedValue is not a pointer, it is an encoded facet, and we can simply decrement\n        // the pointer to collect the next value.\n        collector.collect(docID, decodeTermID(packedValue), decodeFieldID(packedValue));\n        pointer--;\n      }\n    }\n  }\n\n  /**\n   * This method can return one of three values for each given doc ID:\n   *  - UNASSIGNED, if the document has no facets\n   *  - If the highest-order bit is not set, then the (negated) returned value is the single facet\n   *    for this document.\n   *  - If the highest-order bit is set, then the document has multiple facets, and the returned\n   *    values is a pointer into facetsPool.\n   */\n  protected abstract int getFacet(int docID);\n\n  protected abstract void setFacet(int docID, int facetID);\n\n  /**\n   * Called during segment optimization to map term ids that have changed as a\n   * result of the optimization.\n   */\n  public abstract AbstractFacetCountingArray rewriteAndMapIDs(\n      Map<Integer, int[]> termIDMapper,\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException;\n\n  IntBlockPool getFacetsPool() {\n    return facetsPool;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/CSFFacetCountIterator.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.NumericDocValues;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\n/**\n * An iterator that looks up the termID from the appropriate CSF\n */\npublic class CSFFacetCountIterator extends FacetCountIterator {\n  private final int fieldID;\n  private final NumericDocValues numericDocValues;\n\n  /**\n   * Creates a new iterator for the given facet csf field.\n   */\n  public CSFFacetCountIterator(\n      EarlybirdIndexSegmentAtomicReader reader,\n      Schema.FieldInfo facetFieldInfo) throws IOException {\n    FacetIDMap.FacetField facetField = reader.getFacetIDMap().getFacetField(facetFieldInfo);\n    Preconditions.checkNotNull(facetField);\n    this.fieldID = facetField.getFacetId();\n    numericDocValues = reader.getNumericDocValues(facetFieldInfo.getName());\n    Preconditions.checkNotNull(numericDocValues);\n  }\n\n  @Override\n  public void collect(int internalDocID) throws IOException {\n    if (numericDocValues.advanceExact(internalDocID)) {\n      long termID = numericDocValues.longValue();\n      if (shouldCollect(internalDocID, termID)) {\n        collect(internalDocID, termID, fieldID);\n      }\n    }\n  }\n\n  /**\n   * Subclasses should override if they need to restrict the docs or termIDs\n   * that they collect on. For example, these may need to override if\n   *  1) Not all docs set this field, so we should not collect on\n   *     the default value of 0\n   *  2) The same CSF field means different things (in particular, shared_status_id means\n   *     retweet OR reply parent id) so we need to do some other check to determine if we should\n   *     collect\n   *\n   * @return whether we should collect on this doc/termID\n   */\n  protected boolean shouldCollect(int internalDocID, long termID) throws IOException {\n    return true;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/CompositeFacetCountIterator.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.io.IOException;\nimport java.util.Collection;\nimport java.util.List;\n\nimport com.twitter.common.collections.Pair;\n\n/**\n * Calls multiple FacetCountIterators. Currently this is used for calling the\n * default FacetCountingArray iterator and the CSF and retweet iterators\n */\npublic class CompositeFacetCountIterator extends FacetCountIterator {\n  private final Collection<FacetCountIterator> iterators;\n\n  /**\n   * Creates a new composite iterator on the provided collection of iterators.\n   */\n  public CompositeFacetCountIterator(Collection<FacetCountIterator> iterators) {\n    this.iterators = iterators;\n    for (FacetCountIterator iterator : iterators) {\n      iterator.setIncrementData(this.incrementData);\n    }\n  }\n\n  @Override\n  public void collect(int docID) throws IOException {\n    for (FacetCountIterator iterator : iterators) {\n      iterator.collect(docID);\n    }\n  }\n\n  @Override\n  protected void addProof(int docID, long termID, int fieldID) {\n    for (FacetCountIterator iterator : iterators) {\n      iterator.addProof(docID, termID, fieldID);\n    }\n  }\n\n  @Override\n  public void setProofs(List<Pair<Integer, Long>> proof) {\n    for (FacetCountIterator iterator : iterators) {\n      iterator.setProofs(proof);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/DummyFacetAccumulator.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\n/**\n * This accumulator does not accumulate the facet counts when {@link #add(long, int, int, int)}\n * is called.\n */\npublic class DummyFacetAccumulator<R> extends FacetAccumulator<R> {\n\n  @Override\n  public int add(long termID, int scoreIncrement, int penaltyCount, int tweepCred) {\n    return 0;\n  }\n\n  @Override\n  public R getAllFacets() {\n    return null;\n  }\n\n  @Override\n  public R getTopFacets(int n) {\n    return null;\n  }\n\n  @Override\n  public void reset(FacetLabelProvider facetLabelProvider) {\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/EarlybirdFacetDocValueSet.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.util.Map;\nimport java.util.Map.Entry;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.facet.FacetsConfig;\nimport org.apache.lucene.index.ReaderUtil;\nimport org.apache.lucene.index.SortedSetDocValues;\nimport org.apache.lucene.util.BytesRef;\nimport org.apache.lucene.util.BytesRefBuilder;\n\nimport com.twitter.search.core.earlybird.index.inverted.InvertedIndex;\n\npublic class EarlybirdFacetDocValueSet extends SortedSetDocValues {\n  private final AbstractFacetCountingArray countingArray;\n  private final InvertedIndex[] labelProviders;\n  private final String[] fieldNames;\n  private final int[] starts;\n  private final BytesRefBuilder ordCache;\n  private int totalTerms;\n  private int docID = -1;\n  private int currentFacet = FacetCountingArray.UNASSIGNED;\n  private int pointer = -1;\n  private boolean hasMoreOrds = false;\n\n  public static final String FIELD_NAME = FacetsConfig.DEFAULT_INDEX_FIELD_NAME;\n\n  /**\n   * Creates a new EarlybirdFacetDocValueSet from the provided FacetCountingArray.\n   */\n  public EarlybirdFacetDocValueSet(AbstractFacetCountingArray countingArray,\n                                   Map<String, FacetLabelProvider> labelProviderMap,\n                                   FacetIDMap facetIdMap) {\n    this.countingArray = countingArray;\n    labelProviders = new InvertedIndex[facetIdMap.getNumberOfFacetFields()];\n    fieldNames = new String[facetIdMap.getNumberOfFacetFields()];\n    for (Entry<String, FacetLabelProvider> entry : labelProviderMap.entrySet()) {\n      FacetLabelProvider labelProvider = entry.getValue();\n      if (labelProvider instanceof InvertedIndex) {\n        FacetIDMap.FacetField facetField = facetIdMap.getFacetFieldByFacetName(entry.getKey());\n        if (facetField != null) {\n          labelProviders[facetField.getFacetId()] = (InvertedIndex) labelProvider;\n          fieldNames[facetField.getFacetId()] = entry.getKey();\n        }\n      }\n    }\n\n    starts = new int[labelProviders.length + 1];    // build starts array\n    ordCache = new BytesRefBuilder();\n    totalTerms = 0;\n\n    for (int i = 0; i < labelProviders.length; ++i) {\n      if (labelProviders[i] != null) {\n        starts[i] = totalTerms;\n        int termCount = labelProviders[i].getNumTerms();\n        totalTerms += termCount;\n      }\n    }\n\n    // added to so that mapping from ord to index works via ReaderUtil.subIndex\n    starts[labelProviders.length] = totalTerms;\n  }\n\n  private long encodeOrd(int fieldId, int termId) {\n    assert starts[fieldId] + termId < starts[fieldId + 1];\n    return starts[fieldId] + termId;\n  }\n\n  @Override\n  public long nextOrd() {\n    if (!hasMoreOrds || currentFacet == FacetCountingArray.UNASSIGNED) {\n      return SortedSetDocValues.NO_MORE_ORDS;\n    }\n\n    // only 1 facet val\n    if (!FacetCountingArray.isPointer(currentFacet)) {\n      int termId = FacetCountingArray.decodeTermID(currentFacet);\n      int fieldId = FacetCountingArray.decodeFieldID(currentFacet);\n      hasMoreOrds = false;\n      return encodeOrd(fieldId, termId);\n    }\n\n    // multiple facets, follow the pointer to find all facets in the facetsPool.\n    if (pointer == -1) {\n      pointer = FacetCountingArray.decodePointer(currentFacet);\n    }\n    int facetID = countingArray.getFacetsPool().get(pointer);\n    int termId = FacetCountingArray.decodeTermID(facetID);\n    int fieldId = FacetCountingArray.decodeFieldID(facetID);\n\n    hasMoreOrds = FacetCountingArray.isPointer(facetID);\n    pointer++;\n    return encodeOrd(fieldId, termId);\n  }\n\n  @Override\n  public BytesRef lookupOrd(long ord) {\n    int idx = ReaderUtil.subIndex((int) ord, this.starts);\n    if (labelProviders[idx] != null) {\n      int termID = (int) ord - starts[idx];\n      BytesRef term = new BytesRef();\n      labelProviders[idx].getTerm(termID, term);\n      String name = fieldNames[idx];\n      String val = FacetsConfig.pathToString(new String[] {name, term.utf8ToString()});\n      ordCache.copyChars(val);\n    } else {\n      ordCache.copyChars(\"\");\n    }\n    return ordCache.get();\n  }\n\n  @Override\n  public long lookupTerm(BytesRef key) {\n    throw new UnsupportedOperationException();\n  }\n\n  @Override\n  public long getValueCount() {\n    return totalTerms;\n  }\n\n  @Override\n  public int docID() {\n    return docID;\n  }\n\n  @Override\n  public int nextDoc() {\n    return ++docID;\n  }\n\n  @Override\n  public int advance(int target) {\n    Preconditions.checkState(target >= docID);\n    docID = target;\n    currentFacet = countingArray.getFacet(docID);\n    pointer = -1;\n    hasMoreOrds = true;\n    return docID;\n  }\n\n  @Override\n  public boolean advanceExact(int target) {\n    return advance(target) != FacetCountingArray.UNASSIGNED;\n  }\n\n  @Override\n  public long cost() {\n    return totalTerms;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/EarlybirdFacets.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.io.IOException;\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\n\nimport org.apache.lucene.facet.FacetResult;\nimport org.apache.lucene.facet.Facets;\nimport org.apache.lucene.facet.FacetsCollector;\nimport org.apache.lucene.facet.FacetsCollector.MatchingDocs;\nimport org.apache.lucene.util.BitDocIdSet;\nimport org.apache.lucene.util.BitSet;\n\nimport com.twitter.search.common.facets.FacetSearchParam;\nimport com.twitter.search.common.facets.thriftjava.FacetFieldRequest;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\n/**\n * Lucene accumulator implementation that counts on our facet counting array data structure.\n *\n */\npublic class EarlybirdFacets extends Facets {\n\n  private final AbstractFacetCountingArray countingArray;\n  private final FacetCountAggregator aggregator;\n  private final EarlybirdIndexSegmentAtomicReader reader;\n  private final MatchingDocs matchingDocs;\n  private final Map<FacetFieldRequest, FacetResult> resultMapping;\n\n  /**\n   * Constructs an EarlybirdFacets accumulator.\n   */\n  public EarlybirdFacets(\n      List<FacetSearchParam> facetSearchParams,\n      FacetsCollector facetsCollector,\n      EarlybirdIndexSegmentAtomicReader reader) throws IOException {\n\n    Preconditions.checkArgument(facetSearchParams != null && !facetSearchParams.isEmpty());\n    Preconditions.checkArgument(\n        facetsCollector != null\n        && facetsCollector.getMatchingDocs() != null\n        && facetsCollector.getMatchingDocs().size() == 1);\n    Preconditions.checkNotNull(reader);\n\n    this.countingArray = reader.getSegmentData().getFacetCountingArray();\n    this.reader = reader;\n    this.aggregator = new FacetCountAggregator(facetSearchParams,\n        reader.getSegmentData().getSchema(),\n        reader.getFacetIDMap(),\n        reader.getSegmentData().getPerFieldMap());\n    this.matchingDocs = facetsCollector.getMatchingDocs().get(0);\n\n    this.resultMapping = count();\n  }\n\n  private Map<FacetFieldRequest, FacetResult> count() throws IOException {\n    Preconditions.checkState(matchingDocs.bits instanceof BitDocIdSet,\n            \"Assuming BitDocIdSet\");\n    final BitSet bits = ((BitDocIdSet) matchingDocs.bits).bits();\n    final int length = bits.length();\n    int doc = reader.getSmallestDocID();\n    if (doc != -1) {\n      while (doc < length && (doc = bits.nextSetBit(doc)) != -1) {\n        countingArray.collectForDocId(doc, aggregator);\n        doc++;\n      }\n    }\n    return aggregator.getTop();\n  }\n\n  @Override\n  public FacetResult getTopChildren(int topN, String dim, String... path) throws IOException {\n    FacetFieldRequest facetFieldRequest = new FacetFieldRequest(dim, topN);\n    if (path.length > 0) {\n      facetFieldRequest.setPath(Lists.newArrayList(path));\n    }\n\n    FacetResult result = resultMapping.get(facetFieldRequest);\n\n    Preconditions.checkNotNull(\n        result,\n        \"Illegal facet field request: %s, supported requests are: %s\",\n        facetFieldRequest,\n        resultMapping.keySet());\n\n    return result;\n  }\n\n  @Override\n  public Number getSpecificValue(String dim, String... path) {\n    throw new UnsupportedOperationException(\"Not supported\");\n  }\n\n  @Override\n  public List<FacetResult> getAllDims(int topN) throws IOException {\n    throw new UnsupportedOperationException(\"Not supported\");\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/EarlybirdFacetsFactory.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport org.apache.lucene.facet.Facets;\nimport org.apache.lucene.facet.FacetsCollector;\n\nimport com.twitter.search.common.facets.CountFacetSearchParam;\nimport com.twitter.search.common.facets.FacetSearchParam;\nimport com.twitter.search.common.facets.FacetsFactory;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\n/**\n * Factory for EarlybirdFacets\n */\npublic class EarlybirdFacetsFactory implements FacetsFactory {\n  private final EarlybirdIndexSegmentAtomicReader reader;\n\n  public EarlybirdFacetsFactory(EarlybirdIndexSegmentAtomicReader reader) {\n    this.reader = reader;\n  }\n\n  @Override\n  public Facets create(\n      List<FacetSearchParam> facetSearchParams,\n      FacetsCollector facetsCollector) throws IOException {\n\n    return new EarlybirdFacets(facetSearchParams, facetsCollector, reader);\n  }\n\n  @Override\n  public boolean accept(FacetSearchParam facetSearchParam) {\n    if (!(facetSearchParam instanceof CountFacetSearchParam)\n        || (facetSearchParam.getFacetFieldRequest().getPath() != null\n            && !facetSearchParam.getFacetFieldRequest().getPath().isEmpty())) {\n      return false;\n    }\n\n    String field = facetSearchParam.getFacetFieldRequest().getField();\n    Schema.FieldInfo facetInfo = reader.getSegmentData().getSchema()\n            .getFacetFieldByFacetName(field);\n\n    return facetInfo != null\n        && reader.getSegmentData().getPerFieldMap().containsKey(facetInfo.getName());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/FacetAccumulator.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\n\n/**\n * Counts facet occurrences and provides the top items\n * at the end. Actual subclass can implement this functionality differently: e.g. by using\n * a heap (priority queue) or a hashmap with pruning step.\n * The type R represents the facet results, which can e.g. be a thrift class.\n */\npublic abstract class FacetAccumulator<R> {\n  /** Called to notify the accumulator that the given termID has occurred in a document\n   *  Returns the current count of the given termID.\n   */\n  public abstract int add(long termID, int scoreIncrement, int penaltyIncrement, int tweepCred);\n\n  /** After hit collection is done this can be called to\n   * retrieve the items that occurred most often */\n  public abstract R getTopFacets(int n);\n\n  /** After hit collection is done this can be called to retrieve all the items accumulated\n   * (which may not be all that occurred) */\n  public abstract R getAllFacets();\n\n  /** Called to reset a facet accumulator for re-use.  This is an optimization\n   * which takes advantage of the fact that these accumulators may allocate\n   * large hash-tables, and we use one per-segment, which may be as many as 10-20 **/\n  public abstract void reset(FacetLabelProvider facetLabelProvider);\n\n  /** Language histogram accumulation and retrieval. They both have no-op default implementations.\n   */\n  public void recordLanguage(int languageId) { }\n\n  public LanguageHistogram getLanguageHistogram() {\n    return LanguageHistogram.EMPTY_HISTOGRAM;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/FacetCountAggregator.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Map.Entry;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Maps;\n\nimport org.apache.lucene.facet.FacetResult;\n\nimport com.twitter.search.common.facets.CountFacetSearchParam;\nimport com.twitter.search.common.facets.FacetSearchParam;\nimport com.twitter.search.common.facets.thriftjava.FacetFieldRequest;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.index.inverted.InvertedIndex;\n\n/**\n * Global facet aggregator across all fields.\n *\n */\npublic class FacetCountAggregator implements FacetTermCollector {\n\n  // keys for the following aggregators are fieldIds\n  private final Map<Integer, PerfieldFacetCountAggregator> aggregators;\n  private final Map<Integer, FacetSearchParam> facetSearchParamMap;\n\n  /**\n   * Creates a new facet aggregator.\n   */\n  public FacetCountAggregator(\n      List<FacetSearchParam> facetSearchParams,\n      Schema schema,\n      FacetIDMap facetIDMap,\n      Map<String, InvertedIndex> labelProviderMap) {\n\n    aggregators = Maps.newHashMap();\n    facetSearchParamMap = Maps.newHashMap();\n\n    // Check params:\n    for (FacetSearchParam facetSearchParam : facetSearchParams) {\n      if (!(facetSearchParam instanceof CountFacetSearchParam)) {\n        throw new IllegalArgumentException(\n            \"this collector only supports CountFacetSearchParam; got \" + facetSearchParam);\n      }\n      if (facetSearchParam.getFacetFieldRequest().getPath() != null\n          && !facetSearchParam.getFacetFieldRequest().getPath().isEmpty()) {\n        throw new IllegalArgumentException(\n            \"this collector dosen't support hierarchical facets: \"\n            + facetSearchParam.getFacetFieldRequest().getPath());\n      }\n\n      String field = facetSearchParam.getFacetFieldRequest().getField();\n      Schema.FieldInfo facetField =\n          schema == null ? null : schema.getFacetFieldByFacetName(field);\n\n      if (facetField == null || !labelProviderMap.containsKey(facetField.getName())) {\n        throw new IllegalStateException(\"facet field: \" + field + \" is not defined\");\n      }\n\n      int fieldId = facetIDMap.getFacetField(facetField).getFacetId();\n      Preconditions.checkState(!aggregators.containsKey(fieldId));\n      Preconditions.checkState(!facetSearchParamMap.containsKey(fieldId));\n      aggregators.put(fieldId, new PerfieldFacetCountAggregator(field,\n          labelProviderMap.get(facetField.getName())));\n      facetSearchParamMap.put(fieldId, facetSearchParam);\n    }\n  }\n\n  /**\n   * Returns the top facets.\n   */\n  public Map<FacetFieldRequest, FacetResult> getTop() {\n    Map<FacetFieldRequest, FacetResult> map = Maps.newHashMap();\n    for (Entry<Integer, PerfieldFacetCountAggregator> entry : aggregators.entrySet()) {\n      FacetSearchParam facetSearchParam = facetSearchParamMap.get(entry.getKey());\n      map.put(facetSearchParam.getFacetFieldRequest(), entry.getValue().getTop(facetSearchParam));\n    }\n    return map;\n  }\n\n  @Override\n  public boolean collect(int docID, long termID, int fieldID) {\n    PerfieldFacetCountAggregator perfieldAggregator = aggregators.get(fieldID);\n    if (perfieldAggregator != null) {\n      perfieldAggregator.collect((int) termID);\n      return true;\n    } else {\n      return false;\n    }\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/FacetCountIterator.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport com.twitter.common.collections.Pair;\n\n/**\n * The collect() method is called for every document for which facets shall be counted.\n * This iterator then calls the FacetAccumulators for all facets that belong to the\n * current document.\n */\npublic abstract class FacetCountIterator implements FacetTermCollector {\n\n  public static class IncrementData {\n    public FacetAccumulator[] accumulators;\n    public int weightedCountIncrement;\n    public int penaltyIncrement;\n    public int tweepCred;\n    public int languageId;\n  }\n\n  public IncrementData incrementData = new IncrementData();\n\n  private List<Pair<Integer, Long>> proofs = null;\n\n  void setIncrementData(IncrementData incrementData) {\n    this.incrementData = incrementData;\n  }\n\n  public void setProofs(List<Pair<Integer, Long>> proofs) {\n    this.proofs = proofs;\n  }\n\n  // interface method that collects a specific term in a specific field for this document.\n  @Override\n  public boolean collect(int docID, long termID, int fieldID) {\n    FacetAccumulator accumulator = incrementData.accumulators[fieldID];\n    accumulator.add(termID, incrementData.weightedCountIncrement, incrementData.penaltyIncrement,\n                    incrementData.tweepCred);\n    accumulator.recordLanguage(incrementData.languageId);\n\n    if (proofs != null) {\n      addProof(docID, termID, fieldID);\n    }\n    return true;\n  }\n\n  protected void addProof(int docID, long termID, int fieldID) {\n    proofs.add(new Pair<>(fieldID, termID));\n  }\n\n  /**\n   * Collected facets for the given document.\n   */\n  public abstract void collect(int docID) throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/FacetCountIteratorFactory.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\n/**\n * A factory for {@link FacetCountIterator}s.\n */\npublic abstract class FacetCountIteratorFactory {\n  /**\n   * For a field that is being faceted on and for which we should use a CSF for facet counting,\n   * return the iterator we should use for counting.\n   *\n   * @param reader The reader to use when getting CSF values\n   * @param fieldInfo The Schema.FieldInfo corresponding to the facet we're counting\n   * @return An iterator for this field\n   */\n  public abstract FacetCountIterator getFacetCountIterator(\n      EarlybirdIndexSegmentAtomicReader reader,\n      Schema.FieldInfo fieldInfo) throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/FacetCountState.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.Iterator;\nimport java.util.Map;\nimport java.util.Set;\n\nimport com.google.common.collect.Sets;\n\nimport com.twitter.search.common.schema.base.Schema;\n\n/**\n * Maintains internal state during one facet count request.\n */\npublic final class FacetCountState<R> {\n  private final Set<Schema.FieldInfo> fieldsToCount = new HashSet<>();\n  private final Map<String, FacetFieldResults<R>> facetfieldResults =\n      new HashMap<>();\n  private final int minNumFacetResults;\n  private final Schema schema;\n\n  public FacetCountState(Schema schema, int minNumFacetResults) {\n    this.schema = schema;\n    this.minNumFacetResults = minNumFacetResults;\n  }\n\n  /**\n   * Adds a facet to be counted in this request.\n   */\n  public void addFacet(String facetName, int numResultsRequested) {\n    facetfieldResults.put(facetName, new FacetFieldResults(facetName,\n            Math.max(numResultsRequested, minNumFacetResults)));\n    Schema.FieldInfo field = schema.getFacetFieldByFacetName(facetName);\n    fieldsToCount.add(field);\n  }\n\n  public Schema getSchema() {\n    return schema;\n  }\n\n  public int getNumFieldsToCount() {\n    return fieldsToCount.size();\n  }\n\n  /**\n   * Returns whether or not there is a field to be counted for which no skip list is stored\n   */\n  public boolean hasFieldToCountWithoutSkipList() {\n    for (Schema.FieldInfo facetField: fieldsToCount) {\n      if (!facetField.getFieldType().isStoreFacetSkiplist()) {\n        return true;\n      }\n    }\n    return false;\n  }\n\n  public Set<Schema.FieldInfo> getFacetFieldsToCountWithSkipLists() {\n    return Sets.filter(\n        fieldsToCount,\n        facetField -> facetField.getFieldType().isStoreFacetSkiplist());\n  }\n\n  public boolean isCountField(Schema.FieldInfo field) {\n    return fieldsToCount.contains(field);\n  }\n\n  public Iterator<FacetFieldResults<R>> getFacetFieldResultsIterator() {\n    return facetfieldResults.values().iterator();\n  }\n\n  public static final class FacetFieldResults<R> {\n    public final String facetName;\n    public final int numResultsRequested;\n    public R results;\n    public int numResultsFound;\n    public boolean finished = false;\n\n    private FacetFieldResults(String facetName, int numResultsRequested) {\n      this.facetName = facetName;\n      this.numResultsRequested = numResultsRequested;\n    }\n\n    public boolean isFinished() {\n      return finished || results != null && numResultsFound >= numResultsRequested;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/FacetCountingArray.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.io.IOException;\nimport java.util.Map;\n\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.inverted.IntBlockPool;\n\nimport it.unimi.dsi.fastutil.ints.Int2IntOpenHashMap;\n\npublic class FacetCountingArray extends AbstractFacetCountingArray {\n  private static final Logger LOG = LoggerFactory.getLogger(FacetCountingArray.class);\n\n  private final Int2IntOpenHashMap facetsMap;\n\n  /**\n   * Creates a new, empty FacetCountingArray with the given size.\n   */\n  public FacetCountingArray(int maxSegmentSize) {\n    super();\n    facetsMap = new Int2IntOpenHashMap(maxSegmentSize);\n    facetsMap.defaultReturnValue(UNASSIGNED);\n  }\n\n  private FacetCountingArray(Int2IntOpenHashMap facetsMap, IntBlockPool facetsPool) {\n    super(facetsPool);\n    this.facetsMap = facetsMap;\n  }\n\n  @Override\n  protected int getFacet(int docID) {\n    return facetsMap.get(docID);\n  }\n\n  @Override\n  protected void setFacet(int docID, int facetID) {\n    facetsMap.put(docID, facetID);\n  }\n\n  @Override\n  public AbstractFacetCountingArray rewriteAndMapIDs(\n      Map<Integer, int[]> termIDMapper,\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    Preconditions.checkNotNull(originalTweetIdMapper);\n    Preconditions.checkNotNull(optimizedTweetIdMapper);\n\n    // We need to rewrite the facet array, because the term ids have to be mapped to the\n    // key space of the minimum perfect hash function that replaces the hash table.\n    // We also need to remap tweet IDs to the optimized doc IDs.\n    int maxDocID = optimizedTweetIdMapper.getPreviousDocID(Integer.MAX_VALUE);\n    AbstractFacetCountingArray newArray = new OptimizedFacetCountingArray(maxDocID + 1);\n    final FacetCountingArrayWriter writer = new FacetCountingArrayWriter(newArray);\n    FacetCountIterator iterator = new ArrayFacetCountIterator() {\n      @Override\n      public boolean collect(int docID, long termID, int fieldID) {\n        int[] termIDMap = termIDMapper.get(fieldID);\n        int mappedTermID;\n        // If there isn't a map for this term, we are using the original term IDs and can continue\n        // with that term ID. If there is a term ID map, then we need to use the new term ID,\n        // because the new index will use an MPH term dictionary with new term IDs.\n        if (termIDMap == null) {\n          mappedTermID = (int) termID;\n        } else if (termID < termIDMap.length) {\n          mappedTermID = termIDMap[(int) termID];\n        } else {\n          // During segment optimization we might index a new term after the termIDMap is created\n          // in IndexOptimizer.optimizeInvertedIndexes(). We can safely ignore these terms, as\n          // they will be re-indexed later.\n          return false;\n        }\n\n        try {\n          long tweetId = originalTweetIdMapper.getTweetID(docID);\n          int newDocId = optimizedTweetIdMapper.getDocID(tweetId);\n          Preconditions.checkState(newDocId != DocIDToTweetIDMapper.ID_NOT_FOUND,\n                                   \"Did not find a mapping in the new tweet ID mapper for doc ID \"\n                                   + newDocId + \", tweet ID \" + tweetId);\n\n          writer.addFacet(newDocId, fieldID, mappedTermID);\n        } catch (IOException e) {\n          LOG.error(\"Caught an unexpected IOException while optimizing facet.\", e);\n        }\n\n        return true;\n      }\n    };\n\n    // We want to iterate the facets in increasing tweet ID order. This might not correspond to\n    // decreasing doc ID order in the original mapper (see OutOfOrderRealtimeTweetIDMapper).\n    // However, the optimized mapper should be sorted both by tweet IDs and by doc IDs (in reverse\n    // order). So we need to iterate here over the doc IDs in the optimized mapper, convert them\n    // to doc IDs in the original mapper, and pass those doc IDs to collect().\n    int docId = optimizedTweetIdMapper.getPreviousDocID(Integer.MAX_VALUE);\n    while (docId != DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      long tweetId = optimizedTweetIdMapper.getTweetID(docId);\n      int originalDocId = originalTweetIdMapper.getDocID(tweetId);\n      iterator.collect(originalDocId);\n      docId = optimizedTweetIdMapper.getPreviousDocID(docId);\n    }\n    return newArray;\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<FacetCountingArray> {\n    private static final String FACETS_POOL_PROP_NAME = \"facetsPool\";\n    private final int maxSegmentSize;\n\n    public FlushHandler(int maxSegmentSize) {\n      this.maxSegmentSize = maxSegmentSize;\n    }\n\n    public FlushHandler(FacetCountingArray objectToFlush) {\n      super(objectToFlush);\n      maxSegmentSize = -1;\n    }\n\n    @Override\n    public void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      FacetCountingArray array = getObjectToFlush();\n      out.writeInt(array.facetsMap.size());\n      for (Int2IntOpenHashMap.Entry entry : array.facetsMap.int2IntEntrySet()) {\n        out.writeInt(entry.getIntKey());\n        out.writeInt(entry.getIntValue());\n      }\n      array.getFacetsPool().getFlushHandler().flush(\n          flushInfo.newSubProperties(FACETS_POOL_PROP_NAME), out);\n    }\n\n    @Override\n    public FacetCountingArray doLoad(FlushInfo flushInfo, DataDeserializer in) throws IOException {\n      int size = in.readInt();\n      Int2IntOpenHashMap facetsMap = new Int2IntOpenHashMap(maxSegmentSize);\n      facetsMap.defaultReturnValue(UNASSIGNED);\n      for (int i = 0; i < size; i++) {\n        facetsMap.put(in.readInt(), in.readInt());\n      }\n      IntBlockPool facetsPool = new IntBlockPool.FlushHandler().load(\n          flushInfo.getSubProperties(FACETS_POOL_PROP_NAME), in);\n      return new FacetCountingArray(facetsMap, facetsPool);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/FacetCountingArrayWriter.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport com.twitter.search.core.earlybird.index.inverted.IntBlockPool;\n\npublic class FacetCountingArrayWriter {\n  private final AbstractFacetCountingArray facetCountingArray;\n  private int previousDocID = -1;\n\n  public FacetCountingArrayWriter(AbstractFacetCountingArray array) {\n    facetCountingArray = array;\n  }\n\n  /**\n   * Adds a facet for the given doc, field and term tuple.\n   *\n   * The layout of the packedValues in the term pool is:\n   *\n   * index |0 |1 |2 |3 |4 |5 |6 |7 |8 |9 |\n   * value |U |1a|1b|1c|U |2b|2c|P3|1d|1f|\n   *\n   * Where U is UNASSIGNED, P+X is a pointer to index X (e.g. P3 means pointer to index 3),\n   * or a doc ID and facet (e.g. doc ID 1 and facet a would be 1a).\n   */\n  public void addFacet(int docID, int fieldID, int termID) {\n    IntBlockPool facetsPool = facetCountingArray.getFacetsPool();\n    int packedValue = facetCountingArray.getFacet(docID);\n\n    if (packedValue == AbstractFacetCountingArray.UNASSIGNED) {\n      // first facet for this doc.\n      // keep it in the array and don't add it to the map.\n      facetCountingArray.setFacet(docID, AbstractFacetCountingArray.encodeFacetID(fieldID, termID));\n      return;\n    }\n\n    if (!FacetCountingArray.isPointer(packedValue)) {\n      // If the packedValue is not a pointer, we know that we have exactly one facet in the index\n      // for this document, so copy the existing facet into the pool.\n      facetsPool.add(AbstractFacetCountingArray.UNASSIGNED);\n      facetsPool.add(packedValue);\n    } else if (previousDocID != docID) {\n      // We have seen this document ID in a different document. Store the pointer to the first facet\n      // for this doc ID in the pool so that we can traverse the linked list.\n      facetsPool.add(packedValue);\n    }\n\n    previousDocID = docID;\n\n    // Add the new facet to the end of the FacetCountingArray.\n    facetsPool.add(AbstractFacetCountingArray.encodeFacetID(fieldID, termID));\n\n    // Set the facetValue for this document to the pointer to the facet we just added to the array.\n    int poolPointer = AbstractFacetCountingArray.encodePointer(facetsPool.length() - 1);\n    facetCountingArray.setFacet(docID, poolPointer);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/FacetIDMap.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.io.IOException;\nimport java.util.Arrays;\nimport java.util.Collection;\nimport java.util.Map;\n\nimport com.google.common.collect.Maps;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\n\n/**\n * Currently a facet is configured by:\n *   - Index field name: The Lucene field name which stores the indexed terms of this facet\n *   - Facet name:       The name of the facet that the search API specifies to request facet counts.\n *   - Facet id:         An internal id which is used to store the facet forward mapping in the facet counting\n *                       data structures.\n *\n * This is a multi-map with two different mappings:\n *   Facet name       -> Facet id\n *   Facet id         -> FieldInfo\n */\npublic final class FacetIDMap implements Flushable {\n  private final FacetField[] facetIDToFieldMap;\n  private final Map<String, Integer> facetNameToIDMap;\n\n  private FacetIDMap(FacetField[] facetIDToFieldMap) {\n    this.facetIDToFieldMap = facetIDToFieldMap;\n\n    facetNameToIDMap = Maps.newHashMapWithExpectedSize(facetIDToFieldMap.length);\n    for (int i = 0; i < facetIDToFieldMap.length; i++) {\n      facetNameToIDMap.put(facetIDToFieldMap[i].getFacetName(), i);\n    }\n  }\n\n  public FacetField getFacetField(Schema.FieldInfo fieldInfo) {\n    return fieldInfo != null && fieldInfo.getFieldType().isFacetField()\n            ? getFacetFieldByFacetName(fieldInfo.getFieldType().getFacetName()) : null;\n  }\n\n  public FacetField getFacetFieldByFacetName(String facetName) {\n    Integer facetID = facetNameToIDMap.get(facetName);\n    return facetID != null ? facetIDToFieldMap[facetID] : null;\n  }\n\n  public FacetField getFacetFieldByFacetID(int facetID) {\n    return facetIDToFieldMap[facetID];\n  }\n\n  public Collection<FacetField> getFacetFields() {\n    return Arrays.asList(facetIDToFieldMap);\n  }\n\n  public int getNumberOfFacetFields() {\n    return facetIDToFieldMap.length;\n  }\n\n  /**\n   * Builds a new FacetIDMap from the given schema.\n   */\n  public static FacetIDMap build(Schema schema) {\n    FacetField[] facetIDToFieldMap = new FacetField[schema.getNumFacetFields()];\n\n    int facetId = 0;\n\n    for (Schema.FieldInfo fieldInfo : schema.getFieldInfos()) {\n      if (fieldInfo.getFieldType().isFacetField()) {\n        facetIDToFieldMap[facetId] = new FacetField(facetId, fieldInfo);\n        facetId++;\n      }\n    }\n\n    return new FacetIDMap(facetIDToFieldMap);\n  }\n\n  public static final class FacetField {\n    private final int facetId;\n    private final Schema.FieldInfo fieldInfo;\n\n    private FacetField(int facetId, Schema.FieldInfo fieldInfo) {\n      this.facetId = facetId;\n      this.fieldInfo = fieldInfo;\n    }\n\n    public int getFacetId() {\n      return facetId;\n    }\n\n    public Schema.FieldInfo getFieldInfo() {\n      return fieldInfo;\n    }\n\n    public String getFacetName() {\n      return fieldInfo.getFieldType().getFacetName();\n    }\n\n    public String getDescription() {\n      return String.format(\n          \"(FacetField [facetId: %d, fieldInfo: %s])\",\n          getFacetId(), fieldInfo.getDescription());\n    }\n  }\n\n  @SuppressWarnings(\"unchecked\")\n  @Override\n  public FacetIDMap.FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<FacetIDMap> {\n    private static final String NUM_FACET_FIELDS_PROP_NAME = \"numFacetFields\";\n\n    private final Schema schema;\n\n    public FlushHandler(Schema schema) {\n      this.schema = schema;\n    }\n\n    public FlushHandler(FacetIDMap objectToFlush) {\n      super(objectToFlush);\n      // schema only needed here for loading, not for flushing\n      this.schema = null;\n    }\n\n    @Override\n    public void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      FacetIDMap toFlush = getObjectToFlush();\n      int[] idMap = new int[toFlush.facetIDToFieldMap.length];\n      for (int i = 0; i < toFlush.facetIDToFieldMap.length; i++) {\n        idMap[i] = toFlush.facetIDToFieldMap[i].getFieldInfo().getFieldId();\n      }\n      out.writeIntArray(idMap);\n\n      flushInfo.addIntProperty(NUM_FACET_FIELDS_PROP_NAME, idMap.length);\n    }\n\n\n    @Override\n    public FacetIDMap doLoad(FlushInfo flushInfo, DataDeserializer in) throws IOException {\n      int[] idMap = in.readIntArray();\n      if (idMap.length != schema.getNumFacetFields()) {\n        throw new IOException(\"Wrong number of facet fields. Expected by schema: \"\n                + schema.getNumFacetFields()\n                + \", but found in serialized segment: \" + idMap.length);\n      }\n\n      FacetField[] facetIDToFieldMap = new FacetField[schema.getNumFacetFields()];\n\n      for (int i = 0; i < idMap.length; i++) {\n        int fieldConfigId = idMap[i];\n        facetIDToFieldMap[i] = new FacetField(i, schema.getFieldInfo(fieldConfigId));\n      }\n\n      return new FacetIDMap(facetIDToFieldMap);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/FacetLabelProvider.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.hashtable.HashTable;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.util.analysis.IntTermAttributeImpl;\nimport com.twitter.search.common.util.analysis.LongTermAttributeImpl;\nimport com.twitter.search.common.util.analysis.SortableLongTermAttributeImpl;\nimport com.twitter.search.core.earlybird.index.inverted.InvertedIndex;\n\n/**\n * Given a termID this accessor can be used to retrieve the term bytesref and text\n * that corresponds to the termID.\n */\npublic interface FacetLabelProvider {\n  /**\n   * Returns a {@link FacetLabelAccessor} for this provider.\n   */\n  FacetLabelAccessor getLabelAccessor();\n\n  abstract class FacetLabelAccessor {\n    private int currentTermID = -1;\n\n    protected final BytesRef termRef = new BytesRef();\n    protected boolean hasTermPayload = false;\n    protected final BytesRef termPayload = new BytesRef();\n    protected int offensiveCount = 0;\n\n    protected final boolean maybeSeek(long termID) {\n      if (termID == currentTermID) {\n        return true;\n      }\n\n      if (seek(termID)) {\n        currentTermID = (int) termID;\n        return true;\n      } else {\n        currentTermID = -1;\n        return false;\n      }\n    }\n\n    // Seek to term id provided.  Returns true if term found.  Should update termRef,\n    // hasTermPayload, and termPayload as appropriate.\n    protected abstract boolean seek(long termID);\n\n    public final BytesRef getTermRef(long termID) {\n      return maybeSeek(termID) ? termRef : null;\n    }\n\n    public String getTermText(long termID) {\n      return maybeSeek(termID) ? termRef.utf8ToString() : null;\n    }\n\n    public final BytesRef getTermPayload(long termID) {\n      return maybeSeek(termID) && hasTermPayload ? termPayload : null;\n    }\n\n    public final int getOffensiveCount(long termID) {\n      return maybeSeek(termID) ? offensiveCount : 0;\n    }\n  }\n\n  /**\n   * Assumes the term is stored as an IntTermAttribute, and uses this to convert\n   * the term bytesref to an integer string facet label.\n   */\n  class IntTermFacetLabelProvider implements FacetLabelProvider {\n      private final InvertedIndex invertedIndex;\n\n    public IntTermFacetLabelProvider(InvertedIndex invertedIndex) {\n      this.invertedIndex = invertedIndex;\n    }\n\n    @Override\n    public FacetLabelAccessor getLabelAccessor() {\n      return new FacetLabelAccessor() {\n        @Override\n        protected boolean seek(long termID) {\n          if (termID != HashTable.EMPTY_SLOT) {\n            invertedIndex.getTerm((int) termID, termRef);\n            return true;\n          }\n          return false;\n        }\n\n        @Override\n        public String getTermText(long termID) {\n          return maybeSeek(termID)\n                 ? Integer.toString(IntTermAttributeImpl.copyBytesRefToInt(termRef))\n                 : null;\n        }\n      };\n    }\n  }\n\n  /**\n   * Assumes the term is stored as an LongTermAttribute, and uses this to convert\n   * the term bytesref to an long string facet label.\n   */\n  class LongTermFacetLabelProvider implements FacetLabelProvider {\n    private final InvertedIndex invertedIndex;\n\n    public LongTermFacetLabelProvider(InvertedIndex invertedIndex) {\n      this.invertedIndex = invertedIndex;\n    }\n\n    @Override\n    public FacetLabelAccessor getLabelAccessor() {\n      return new FacetLabelAccessor() {\n        @Override\n        protected boolean seek(long termID) {\n          if (termID != HashTable.EMPTY_SLOT) {\n            invertedIndex.getTerm((int) termID, termRef);\n            return true;\n          }\n          return false;\n        }\n\n        @Override\n        public String getTermText(long termID) {\n          return maybeSeek(termID)\n                 ? Long.toString(LongTermAttributeImpl.copyBytesRefToLong(termRef))\n                 : null;\n        }\n      };\n    }\n  }\n\n  class SortedLongTermFacetLabelProvider implements FacetLabelProvider {\n    private final InvertedIndex invertedIndex;\n\n    public SortedLongTermFacetLabelProvider(InvertedIndex invertedIndex) {\n      this.invertedIndex = invertedIndex;\n    }\n\n    @Override\n    public FacetLabelAccessor getLabelAccessor() {\n      return new FacetLabelAccessor() {\n        @Override\n        protected boolean seek(long termID) {\n          if (termID != HashTable.EMPTY_SLOT) {\n            invertedIndex.getTerm((int) termID, termRef);\n            return true;\n          }\n          return false;\n        }\n\n        @Override\n        public String getTermText(long termID) {\n          return maybeSeek(termID)\n              ? Long.toString(SortableLongTermAttributeImpl.copyBytesRefToLong(termRef))\n              : null;\n        }\n      };\n    }\n  }\n\n  class IdentityFacetLabelProvider implements FacetLabelProvider {\n    @Override\n    public FacetLabelAccessor getLabelAccessor() {\n      return new FacetLabelAccessor() {\n        @Override\n        protected boolean seek(long termID) {\n          return true;\n        }\n\n        @Override\n        public String getTermText(long termID) {\n          return Long.toString(termID);\n        }\n      };\n    }\n  }\n\n  /**\n   * The methods on this provider should NOT be called under normal circumstances!\n   *\n   * When a facet misses inverted index and does not use CSF, this InaccessibleFacetLabelProvider\n   * will be used as a dummy provider. Then, unexptectedFacetLabelAccess counter will be\n   * incremented when this provider is used later.\n   *\n   * Also see:\n   * {@link FacetUtil}\n   */\n  class InaccessibleFacetLabelProvider implements FacetLabelProvider {\n    private final SearchCounter unexptectedFacetLabelAccess;\n\n    public InaccessibleFacetLabelProvider(String fieldName) {\n      this.unexptectedFacetLabelAccess =\n          SearchCounter.export(\"unexpected_facet_label_access_for_field_\" + fieldName);\n    }\n\n    @Override\n    public FacetLabelAccessor getLabelAccessor() {\n      return new FacetLabelAccessor() {\n        @Override\n        protected boolean seek(long termID) {\n          unexptectedFacetLabelAccess.increment();\n          return false;\n        }\n      };\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/FacetResponseRewriter.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport com.twitter.search.common.facets.thriftjava.FacetResponse;\n\n/**\n * Rewrite facet responses\n */\npublic interface FacetResponseRewriter {\n  /**\n   * Do the response rewrite\n   *\n   * @param facetResponse the response before the rewriting\n   * @return the rewrited response\n   */\n  FacetResponse rewrite(FacetResponse facetResponse);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/FacetTermCollector.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\n/**\n * An interface for collecting all facets in an document.\n */\npublic interface FacetTermCollector {\n  /**\n   * Collect one facet term.\n   * @param docID The docID for which the facets are being collected.\n   * @param termID The termID for this facet item.\n   * @param fieldID The fieldID for this facet item.\n   * @return True if anything has actually been collected, false if this has been skipped.\n   *         Currently, this return value is not used.\n   */\n  boolean collect(int docID, long termID, int fieldID);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/FacetUtil.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.util.HashMap;\nimport java.util.Map;\n\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.schema.base.IndexedNumericFieldSettings;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.thriftjava.ThriftNumericType;\nimport com.twitter.search.core.earlybird.index.inverted.InvertedIndex;\n\n/**\n * A utility class for selecting iterators and label providers\n * for facets.\n *\n */\npublic abstract class FacetUtil {\n  private static final Logger LOG = LoggerFactory.getLogger(FacetUtil.class);\n\n  private FacetUtil() {\n    // unused\n  }\n\n  /**\n   * A utility method for choosing the right facet label provider based on the EarlybirdFieldType.\n   * Takes in a InvertedIndex since some facet label providers are or depend on the inverted\n   * index.\n   * Should never return null.\n   *\n   * @param fieldType A FieldType for the facet\n   * @param invertedField The inverted index associated with the facet. May be null.\n   * @return A non-null FacetLabelProvider\n   */\n  public static FacetLabelProvider chooseFacetLabelProvider(\n      EarlybirdFieldType fieldType,\n      InvertedIndex invertedField) {\n    Preconditions.checkNotNull(fieldType);\n\n    // In the case neither inverted index existing nor using CSF,\n    // return FacetLabelProvider.InaccessibleFacetLabelProvider to throw exception\n    // more meaningfully and explicitly.\n    if (invertedField == null && !fieldType.isUseCSFForFacetCounting()) {\n      return new FacetLabelProvider.InaccessibleFacetLabelProvider(fieldType.getFacetName());\n    }\n\n    if (fieldType.isUseCSFForFacetCounting()) {\n      return new FacetLabelProvider.IdentityFacetLabelProvider();\n    }\n    IndexedNumericFieldSettings numericSettings = fieldType.getNumericFieldSettings();\n    if (numericSettings != null && numericSettings.isUseTwitterFormat()) {\n      if (numericSettings.getNumericType() == ThriftNumericType.INT) {\n        return new FacetLabelProvider.IntTermFacetLabelProvider(invertedField);\n      } else if (numericSettings.getNumericType() == ThriftNumericType.LONG) {\n        return numericSettings.isUseSortableEncoding()\n            ? new FacetLabelProvider.SortedLongTermFacetLabelProvider(invertedField)\n            : new FacetLabelProvider.LongTermFacetLabelProvider(invertedField);\n      } else {\n        Preconditions.checkState(false,\n            \"Should never be reached, indicates incomplete handling of different kinds of facets\");\n        return null;\n      }\n    } else {\n      return invertedField;\n    }\n  }\n\n  /**\n   * Get segment-specific facet label providers based on the schema\n   * and on the fieldToInvertedIndexMapping for the segment.\n   * These will be used by facet accumulators to get the text of the termIDs\n   *\n   * @param schema the schema, for info on fields and facets\n   * @param fieldToInvertedIndexMapping map of fields to their inverted indices\n   * @return facet label provider map\n   */\n  public static Map<String, FacetLabelProvider> getFacetLabelProviders(\n      Schema schema,\n      Map<String, InvertedIndex> fieldToInvertedIndexMapping) {\n\n    HashMap<String, FacetLabelProvider> facetLabelProviderBuilder\n        = new HashMap<>();\n\n    for (Schema.FieldInfo fieldInfo : schema.getFacetFields()) {\n      EarlybirdFieldType fieldType = fieldInfo.getFieldType();\n      Preconditions.checkNotNull(fieldType);\n      String fieldName = fieldInfo.getName();\n      String facetName = fieldType.getFacetName();\n      InvertedIndex invertedIndex = fieldToInvertedIndexMapping.get(fieldName);\n      if (invertedIndex == null && !fieldType.isUseCSFForFacetCounting()) {\n        LOG.warn(\"No docs in segment had field \" + fieldName\n                + \" indexed for facet \" + facetName\n                + \" so InaccessibleFacetLabelProvider will be provided.\"\n        );\n      }\n      facetLabelProviderBuilder.put(facetName, Preconditions.checkNotNull(\n          chooseFacetLabelProvider(fieldType, invertedIndex)));\n    }\n\n    return facetLabelProviderBuilder;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/LanguageHistogram.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.util.Arrays;\nimport java.util.Map;\n\nimport com.google.common.collect.ImmutableMap;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\n\n/**\n * A util class to build a language histogram\n */\npublic class LanguageHistogram {\n  private static final Logger LOG = LoggerFactory.getLogger(LanguageHistogram.class);\n\n  public static final LanguageHistogram EMPTY_HISTOGRAM = new LanguageHistogram() {\n    // Let's make this immutable for safety.\n    @Override public void clear() {\n      throw new UnsupportedOperationException();\n    }\n\n    @Override public void increment(int languageID) {\n      throw new UnsupportedOperationException();\n    }\n\n    @Override public void add(int languageID, int value) {\n      throw new UnsupportedOperationException();\n    }\n\n    @Override public void addAll(LanguageHistogram histogram) {\n      throw new UnsupportedOperationException();\n    }\n  };\n\n  private final int[] languageHistogram = new int[ThriftLanguage.values().length];\n\n  public int[] getLanguageHistogram() {\n    return languageHistogram;\n  }\n\n  /**\n   * Returns this histogram represented as a language->count map.\n   */\n  public Map<ThriftLanguage, Integer> getLanguageHistogramAsMap() {\n    ImmutableMap.Builder<ThriftLanguage, Integer> builder = ImmutableMap.builder();\n    for (int i = 0; i < languageHistogram.length; i++) {\n      // ThriftLanguage.findByValue() might return null, which should fall back to UNKNOWN.\n      ThriftLanguage lang = ThriftLanguage.findByValue(i);\n      lang = lang == null ? ThriftLanguage.UNKNOWN : lang;\n      builder.put(lang, languageHistogram[i]);\n    }\n    return builder.build();\n  }\n\n  public void clear() {\n    Arrays.fill(languageHistogram, 0);\n  }\n\n  public void increment(int languageId) {\n    if (isValidLanguageId(languageId)) {\n      languageHistogram[languageId]++;\n    }\n  }\n\n  public void increment(ThriftLanguage language) {\n    increment(language.getValue());\n  }\n\n  public void add(int languageId, int value) {\n    if (isValidLanguageId(languageId)) {\n      languageHistogram[languageId] += value;\n    }\n  }\n\n  public void add(ThriftLanguage language, int value) {\n    add(language.getValue(), value);\n  }\n\n  /**\n   * Adds all entries from the provided histogram to this histogram.\n   */\n  public void addAll(LanguageHistogram histogram) {\n    if (histogram == EMPTY_HISTOGRAM) {\n      return;\n    }\n    for (int i = 0; i < languageHistogram.length; i++) {\n      languageHistogram[i] += histogram.languageHistogram[i];\n    }\n  }\n\n  // Check for out of bound languages.  If a language is out of bounds, we don't want it\n  // to cause the entire search to fail.\n  private boolean isValidLanguageId(int languageId) {\n    if (languageId < languageHistogram.length) {\n      return true;\n    } else {\n      LOG.error(\"Language id \" + languageId + \" out of range\");\n      return false;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/OptimizedFacetCountingArray.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.io.IOException;\nimport java.util.Arrays;\nimport java.util.Map;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.inverted.IntBlockPool;\n\npublic class OptimizedFacetCountingArray extends AbstractFacetCountingArray {\n  private final int[] facetsMap;\n\n  /**\n   * Creates a new, empty FacetCountingArray with the given size.\n   */\n  public OptimizedFacetCountingArray(int maxDocIdInclusive) {\n    super();\n    facetsMap = new int[maxDocIdInclusive];\n    Arrays.fill(facetsMap, UNASSIGNED);\n  }\n\n  private OptimizedFacetCountingArray(int[] facetsMap, IntBlockPool facetsPool) {\n    super(facetsPool);\n    this.facetsMap = facetsMap;\n  }\n\n  @Override\n  protected int getFacet(int docID) {\n    return facetsMap[docID];\n  }\n\n  @Override\n  protected void setFacet(int docID, int facetID) {\n    facetsMap[docID] = facetID;\n  }\n\n  @Override\n  public AbstractFacetCountingArray rewriteAndMapIDs(\n      Map<Integer, int[]> termIDMapper,\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) {\n    throw new UnsupportedOperationException(\n        \"OptimizedFacetCountingArray instances should never be rewritten.\");\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<OptimizedFacetCountingArray> {\n    private static final String FACETS_POOL_PROP_NAME = \"facetsPool\";\n\n    public FlushHandler() {\n    }\n\n    public FlushHandler(OptimizedFacetCountingArray objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    public void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      OptimizedFacetCountingArray objectToFlush = getObjectToFlush();\n      out.writeIntArray(objectToFlush.facetsMap);\n      objectToFlush.getFacetsPool().getFlushHandler().flush(\n          flushInfo.newSubProperties(FACETS_POOL_PROP_NAME), out);\n    }\n\n    @Override\n    public OptimizedFacetCountingArray doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      int[] facetsMap = in.readIntArray();\n      IntBlockPool facetsPool = new IntBlockPool.FlushHandler().load(\n          flushInfo.getSubProperties(FACETS_POOL_PROP_NAME), in);\n      return new OptimizedFacetCountingArray(facetsMap, facetsPool);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/PerfieldFacetCountAggregator.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.facet.FacetResult;\nimport org.apache.lucene.facet.LabelAndValue;\nimport org.apache.lucene.util.BytesRef;\nimport org.apache.lucene.util.PriorityQueue;\n\nimport com.twitter.search.common.facets.FacetSearchParam;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider.FacetLabelAccessor;\n\nimport it.unimi.dsi.fastutil.ints.Int2IntMap.Entry;\nimport it.unimi.dsi.fastutil.ints.Int2IntMap.FastEntrySet;\nimport it.unimi.dsi.fastutil.ints.Int2IntOpenHashMap;\n\npublic class PerfieldFacetCountAggregator {\n\n  private final Int2IntOpenHashMap countMap;\n  private final FacetLabelAccessor facetLabelAccessor;\n  private final String name;\n\n  /**\n   * Creates a new per-field facet aggregator.\n   */\n  public PerfieldFacetCountAggregator(String name, FacetLabelProvider facetLabelProvider) {\n    this.name = name;\n    this.countMap = new Int2IntOpenHashMap();\n    this.countMap.defaultReturnValue(0);\n    this.facetLabelAccessor = facetLabelProvider.getLabelAccessor();\n  }\n\n  public void collect(int termId) {\n    countMap.put(termId, countMap.get(termId) + 1);\n  }\n\n  /**\n   * Returns the top facets.\n   */\n  public FacetResult getTop(FacetSearchParam facetSearchParam) {\n    Preconditions.checkArgument(\n        facetSearchParam != null\n        && facetSearchParam.getFacetFieldRequest().getField().equals(name)\n        && (facetSearchParam.getFacetFieldRequest().getPath() == null\n            || facetSearchParam.getFacetFieldRequest().getPath().isEmpty()));\n\n    PriorityQueue<Entry> pq = new PriorityQueue<Entry>(\n        facetSearchParam.getFacetFieldRequest().getNumResults()) {\n\n      private BytesRef buffer = new BytesRef();\n\n      @Override\n      protected boolean lessThan(Entry a, Entry b) {\n        // first by count desc\n        int r = Integer.compare(a.getIntValue(), b.getIntValue());\n        if (r != 0) {\n          return r < 0;\n        }\n\n        // and then by label asc\n        BytesRef label1 = facetLabelAccessor.getTermRef(a.getIntKey());\n        buffer.bytes = label1.bytes;\n        buffer.offset = label1.offset;\n        buffer.length = label1.length;\n\n        return buffer.compareTo(facetLabelAccessor.getTermRef(b.getIntKey())) > 0;\n      }\n\n    };\n\n    final FastEntrySet entrySet = countMap.int2IntEntrySet();\n\n    int numValid = 0;\n    for (Entry entry : entrySet) {\n      long val = entry.getIntValue();\n      if (val > 0) {\n        numValid++;\n        pq.insertWithOverflow(entry);\n      }\n    }\n\n    int numVals = pq.size();\n    LabelAndValue[] labelValues = new LabelAndValue[numVals];\n\n    // Priority queue pops out \"least\" element first (that is the root).\n    // Least in our definition regardless of how we define what that is should be the last element.\n    for (int i = labelValues.length - 1; i >= 0; i--) {\n      Entry entry = pq.pop();\n      labelValues[i] = new LabelAndValue(\n          facetLabelAccessor.getTermText(entry.getIntKey()),\n          entry.getValue());\n    }\n\n    return new FacetResult(name, null, 0, labelValues, numValid);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/SortedSetDocValuesFacetsFactory.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.facet.Facets;\nimport org.apache.lucene.facet.FacetsCollector;\nimport org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts;\nimport org.apache.lucene.facet.sortedset.SortedSetDocValuesReaderState;\n\nimport com.twitter.search.common.facets.CountFacetSearchParam;\nimport com.twitter.search.common.facets.FacetSearchParam;\nimport com.twitter.search.common.facets.FacetsFactory;\n\n/**\n * Factory for SortedSetDocValuesFacetCounts\n */\npublic class SortedSetDocValuesFacetsFactory implements FacetsFactory {\n  private final SortedSetDocValuesReaderState state;\n\n  public SortedSetDocValuesFacetsFactory(SortedSetDocValuesReaderState state) {\n    this.state = state;\n  }\n\n  @Override\n  public Facets create(\n      List<FacetSearchParam> facetSearchParams,\n      FacetsCollector facetsCollector) throws IOException {\n\n    Preconditions.checkNotNull(facetsCollector);\n\n    return new SortedSetDocValuesFacetCounts(state, facetsCollector);\n  }\n\n  @Override\n  public boolean accept(FacetSearchParam facetSearchParam) {\n    return facetSearchParam instanceof CountFacetSearchParam\n        && (facetSearchParam.getFacetFieldRequest().getPath() == null\n            || facetSearchParam.getFacetFieldRequest().getPath().isEmpty())\n        && SortedSetDocValuesReaderStateHelper.isDimSupported(\n            state, facetSearchParam.getFacetFieldRequest().getField());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/facets/SortedSetDocValuesReaderStateHelper.java",
    "content": "package com.twitter.search.core.earlybird.facets;\n\nimport org.apache.lucene.facet.sortedset.SortedSetDocValuesReaderState;\n\n/**\n * We have to check if the facet field (dim called by lucene) is supported or\n * not by the SortedSetDocValuesReaderState. The method we have to call is\n * private to the lucene package, so we have this helper to do the call for us.\n */\npublic abstract class SortedSetDocValuesReaderStateHelper {\n  public static boolean isDimSupported(SortedSetDocValuesReaderState state, String dim) {\n    return state.getOrdRange(dim) != null;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/DocIDToTweetIDMapper.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport java.io.IOException;\n\n/**\n * An interface for mapping the doc IDs in our indexes to the corresponding tweet IDs.\n */\npublic interface DocIDToTweetIDMapper {\n  /** A constant indicating that a doc ID was not found in the mapper. */\n  int ID_NOT_FOUND = -1;\n\n  /**\n   * Returns the tweet ID corresponding to the given doc ID.\n   *\n   * @param docID The doc ID stored in our indexes.\n   * @return The tweet ID corresponding to the given doc ID.\n   */\n  long getTweetID(int docID);\n\n  /**\n   * Returns the internal doc ID corresponding to the given tweet ID. Returns ID_NOT_FOUND if the\n   * given tweet ID cannot be found in the index.\n   *\n   * @param tweetID The tweet ID.\n   * @return The doc ID corresponding to the given tweet ID.\n   */\n  int getDocID(long tweetID) throws IOException;\n\n  /**\n   * Returns the smallest valid doc ID in this mapper that's strictly higher than the given doc ID.\n   * If no such doc ID exists, ID_NOT_FOUND is returned.\n   *\n   * @param docID The current doc ID.\n   * @return The smallest valid doc ID in this mapper that's strictly higher than the given doc ID,\n   *         or a negative number, if no such doc ID exists.\n   */\n  int getNextDocID(int docID);\n\n  /**\n   * Returns the largest valid doc ID in this mapper that's strictly smaller than the given doc ID.\n   * If no such doc ID exists, ID_NOT_FOUND is returned.\n   *\n   * @param docID The current doc ID.\n   * @return The largest valid doc ID in this mapper that's strictly smaller than the given doc ID,\n   *         or a negative number, if no such doc ID exists.\n   */\n  int getPreviousDocID(int docID);\n\n  /**\n   * Returns the total number of documents stored in this mapper.\n   *\n   * @return The total number of documents stored in this mapper.\n   */\n  int getNumDocs();\n\n  /**\n   * Adds a mapping for the given tweet ID. Returns the doc ID assigned to this tweet ID.\n   * This method does not check if the tweet ID is already present in the mapper. It always assigns\n   * a new doc ID to the given tweet.\n   *\n   * @param tweetID The tweet ID to be added to the mapper.\n   * @return The doc ID assigned to the given tweet ID, or ID_NOT_FOUND if a doc ID could not be\n   *         assigned to this tweet.\n   */\n  int addMapping(long tweetID);\n\n  /**\n   * Converts the current DocIDToTweetIDMapper to a DocIDToTweetIDMapper instance with the same\n   * tweet IDs. The tweet IDs in the original and optimized instances can be mapped to different\n   * doc IDs. However, we expect doc IDs to be assigned such that tweets created later have smaller\n   * have smaller doc IDs.\n   *\n   * This method should be called when an earlybird segment is being optimized, right before\n   * flushing it to disk.\n   *\n   * @return An optimized DocIDToTweetIDMapper with the same tweet IDs.\n   */\n  DocIDToTweetIDMapper optimize() throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/EarlybirdIndexSegmentAtomicReader.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport java.io.IOException;\nimport java.util.Map;\nimport java.util.Set;\n\nimport com.google.common.collect.Sets;\n\nimport org.apache.lucene.index.FieldInfos;\nimport org.apache.lucene.index.Fields;\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.DocIdSetIterator;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.facets.AbstractFacetCountingArray;\nimport com.twitter.search.core.earlybird.facets.FacetIDMap;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.index.inverted.DeletedDocs;\n\n/**\n * Base class for atomic Earlybird segment readers.\n */\npublic abstract class EarlybirdIndexSegmentAtomicReader extends LeafReader {\n  public static final int TERM_NOT_FOUND = -1;\n\n  private final DeletedDocs.View deletesView;\n  private final EarlybirdIndexSegmentData segmentData;\n  protected final EarlybirdIndexSegmentData.SyncData syncData;\n\n  private FieldInfos fieldInfos;\n\n  /**\n   * Creates a new atomic reader for this Earlybird segment.\n   */\n  public EarlybirdIndexSegmentAtomicReader(EarlybirdIndexSegmentData segmentData) {\n    super();\n    this.segmentData = segmentData;\n    this.syncData = segmentData.getSyncData();\n    this.deletesView = segmentData.getDeletedDocs().getView();\n    // fieldInfos will be initialized lazily if required\n    this.fieldInfos = null;\n  }\n\n  public int getSmallestDocID() {\n    return syncData.getSmallestDocID();\n  }\n\n  public final FacetIDMap getFacetIDMap() {\n    return segmentData.getFacetIDMap();\n  }\n\n  public final Map<String, FacetLabelProvider> getFacetLabelProviders() {\n    return segmentData.getFacetLabelProviders();\n  }\n\n  public AbstractFacetCountingArray getFacetCountingArray() {\n    return segmentData.getFacetCountingArray();\n  }\n\n  public final FacetLabelProvider getFacetLabelProviders(Schema.FieldInfo field) {\n    String facetName = field.getFieldType().getFacetName();\n    return facetName != null && segmentData.getFacetLabelProviders() != null\n            ? segmentData.getFacetLabelProviders().get(facetName) : null;\n  }\n\n  @Override\n  public FieldInfos getFieldInfos() {\n    if (fieldInfos == null) {\n      // TwitterInMemoryIndexReader is constructed per query, and this call is only needed for\n      // optimize. We wouldn't want to create a new FieldInfos per search, so we deffer it.\n      Schema schema = segmentData.getSchema();\n      final Set<String> fieldSet = Sets.newHashSet(segmentData.getPerFieldMap().keySet());\n      fieldSet.addAll(segmentData.getDocValuesManager().getDocValueNames());\n      fieldInfos = schema.getLuceneFieldInfos(input -> input != null && fieldSet.contains(input));\n    }\n    return fieldInfos;\n  }\n\n  /**\n   * Returns the ID that was assigned to the given term in\n   * {@link com.twitter.search.core.earlybird.index.inverted.InvertedRealtimeIndex}\n   */\n  public abstract int getTermID(Term t) throws IOException;\n\n  /**\n   * Returns the oldest posting for the given term\n   * NOTE: This method may return a deleted doc id.\n   */\n  public abstract int getOldestDocID(Term t) throws IOException;\n\n  @Override\n  public abstract NumericDocValues getNumericDocValues(String field) throws IOException;\n\n  /**\n   * Determines if this reader has any documents to traverse. Note that it is possible for the tweet\n   * ID mapper to have documents, but for this reader to not see them yet. In this case, this method\n   * will return false.\n   */\n  public boolean hasDocs() {\n    return segmentData.numDocs() > 0;\n  }\n\n  /**\n   * Returns the newest posting for the given term\n   */\n  public final int getNewestDocID(Term term) throws IOException {\n    PostingsEnum td = postings(term);\n    if (td == null) {\n      return EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND;\n    }\n\n    if (td.nextDoc() != DocIdSetIterator.NO_MORE_DOCS) {\n      return td.docID();\n    } else {\n      return EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND;\n    }\n  }\n\n  public final DeletedDocs.View getDeletesView() {\n    return deletesView;\n  }\n\n  @Override\n  public final Fields getTermVectors(int docID) {\n    // Earlybird does not use term vectors.\n    return null;\n  }\n\n  public EarlybirdIndexSegmentData getSegmentData() {\n    return segmentData;\n  }\n\n  public Schema getSchema() {\n    return segmentData.getSchema();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/EarlybirdIndexSegmentData.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.DirectoryReader;\nimport org.apache.lucene.index.IndexWriterConfig;\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.store.Directory;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.facets.AbstractFacetCountingArray;\nimport com.twitter.search.core.earlybird.facets.FacetCountingArrayWriter;\nimport com.twitter.search.core.earlybird.facets.FacetIDMap;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideByteIndex;\nimport com.twitter.search.core.earlybird.index.column.DocValuesManager;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdIndexExtensionsData;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdIndexExtensionsFactory;\nimport com.twitter.search.core.earlybird.index.inverted.DeletedDocs;\nimport com.twitter.search.core.earlybird.index.inverted.InvertedIndex;\nimport com.twitter.search.core.earlybird.index.inverted.InvertedRealtimeIndex;\nimport com.twitter.search.core.earlybird.index.inverted.OptimizedMemoryIndex;\nimport com.twitter.search.core.earlybird.index.inverted.TermPointerEncoding;\n\n/**\n * Base class that references data structures belonging to an Earlybird segment.\n */\npublic abstract class EarlybirdIndexSegmentData implements Flushable {\n  /**\n   * This class has a map which contains a snapshot of max published pointers, to distinguish the\n   * documents in the skip lists that are fully indexed, and safe to return to searchers and those\n   * that are in progress and should not be returned to searchers. See\n   * \"Earlybird Indexing Latency Design Document\"\n   * for rationale and design.\n   *\n   * It also has the smallestDocID, which determines the smallest assigned doc ID in the tweet ID\n   * mapper that is safe to traverse.\n   *\n   * The pointer map and smallestDocID need to be updated atomically. See SEARCH-27650.\n   */\n  public static class SyncData {\n    private final Map<InvertedIndex, Integer> indexPointers;\n    private final int smallestDocID;\n\n    public SyncData(Map<InvertedIndex, Integer> indexPointers, int smallestDocID) {\n      this.indexPointers = indexPointers;\n      this.smallestDocID = smallestDocID;\n    }\n\n    public Map<InvertedIndex, Integer> getIndexPointers() {\n      return indexPointers;\n    }\n\n    public int getSmallestDocID() {\n      return smallestDocID;\n    }\n  }\n\n  private volatile SyncData syncData;\n\n  private final int maxSegmentSize;\n  private final long timeSliceID;\n\n  private final ConcurrentHashMap<String, QueryCacheResultForSegment> queryCacheMap =\n      new ConcurrentHashMap<>();\n  private final AbstractFacetCountingArray facetCountingArray;\n  private final boolean isOptimized;\n  private final ConcurrentHashMap<String, InvertedIndex> perFieldMap;\n  private final ConcurrentHashMap<String, ColumnStrideByteIndex> normsMap;\n\n  private final Map<String, FacetLabelProvider> facetLabelProviders;\n  private final FacetIDMap facetIDMap;\n\n  private final Schema schema;\n  private final DocValuesManager docValuesManager;\n\n  private final DeletedDocs deletedDocs;\n\n  private final DocIDToTweetIDMapper docIdToTweetIdMapper;\n  private final TimeMapper timeMapper;\n\n  static LeafReader getLeafReaderFromOptimizedDirectory(Directory directory) throws IOException {\n    List<LeafReaderContext> leaves = DirectoryReader.open(directory).getContext().leaves();\n    int leavesSize = leaves.size();\n    Preconditions.checkState(1 == leavesSize,\n        \"Expected one leaf reader in directory %s, but found %s\", directory, leavesSize);\n    return leaves.get(0).reader();\n  }\n\n  /**\n   * Creates a new SegmentData instance using the provided data.\n   */\n  public EarlybirdIndexSegmentData(\n      int maxSegmentSize,\n      long timeSliceID,\n      Schema schema,\n      boolean isOptimized,\n      int smallestDocID,\n      ConcurrentHashMap<String, InvertedIndex> perFieldMap,\n      ConcurrentHashMap<String, ColumnStrideByteIndex> normsMap,\n      AbstractFacetCountingArray facetCountingArray,\n      DocValuesManager docValuesManager,\n      Map<String, FacetLabelProvider> facetLabelProviders,\n      FacetIDMap facetIDMap,\n      DeletedDocs deletedDocs,\n      DocIDToTweetIDMapper docIdToTweetIdMapper,\n      TimeMapper timeMapper) {\n    this.maxSegmentSize = maxSegmentSize;\n    this.timeSliceID = timeSliceID;\n    this.schema = schema;\n    this.isOptimized = isOptimized;\n    this.facetCountingArray = facetCountingArray;\n    this.perFieldMap = perFieldMap;\n    this.syncData = new SyncData(buildIndexPointers(), smallestDocID);\n    this.normsMap = normsMap;\n    this.docValuesManager = docValuesManager;\n    this.facetLabelProviders = facetLabelProviders;\n    this.facetIDMap = facetIDMap;\n    this.deletedDocs = deletedDocs;\n    this.docIdToTweetIdMapper = docIdToTweetIdMapper;\n    this.timeMapper = timeMapper;\n\n    Preconditions.checkNotNull(schema);\n  }\n\n  public final Schema getSchema() {\n    return schema;\n  }\n\n  /**\n   * Returns all {@link EarlybirdIndexExtensionsData} instances contained in this segment.\n   * Since index extensions are optional, the returned map might be null or empty.\n   */\n  public abstract <S extends EarlybirdIndexExtensionsData> S getIndexExtensionsData();\n\n  public DocIDToTweetIDMapper getDocIDToTweetIDMapper() {\n    return docIdToTweetIdMapper;\n  }\n\n  public TimeMapper getTimeMapper() {\n    return timeMapper;\n  }\n\n  public final DocValuesManager getDocValuesManager() {\n    return docValuesManager;\n  }\n\n  public Map<String, FacetLabelProvider> getFacetLabelProviders() {\n    return facetLabelProviders;\n  }\n\n  public FacetIDMap getFacetIDMap() {\n    return facetIDMap;\n  }\n\n  /**\n   * Returns the QueryCacheResult for the given filter for this segment.\n   */\n  public QueryCacheResultForSegment getQueryCacheResult(String queryCacheFilterName) {\n    return queryCacheMap.get(queryCacheFilterName);\n  }\n\n  public long getQueryCachesCardinality() {\n    return queryCacheMap.values().stream().mapToLong(q -> q.getCardinality()).sum();\n  }\n\n  /**\n   * Get cache cardinality for each query cache.\n   * @return\n   */\n  public List<Pair<String, Long>> getPerQueryCacheCardinality() {\n    ArrayList<Pair<String, Long>> result = new ArrayList<>();\n\n    queryCacheMap.forEach((cacheName, queryCacheResult) -> {\n      result.add(Pair.of(cacheName, queryCacheResult.getCardinality()));\n    });\n    return result;\n  }\n\n  /**\n   * Updates the QueryCacheResult stored for the given filter for this segment\n   */\n  public QueryCacheResultForSegment updateQueryCacheResult(\n      String queryCacheFilterName, QueryCacheResultForSegment queryCacheResultForSegment) {\n    return queryCacheMap.put(queryCacheFilterName, queryCacheResultForSegment);\n  }\n\n  /**\n   * Subclasses are allowed to return null here to disable writing to a FacetCountingArray.\n   */\n  public FacetCountingArrayWriter createFacetCountingArrayWriter() {\n    return getFacetCountingArray() != null\n        ? new FacetCountingArrayWriter(getFacetCountingArray()) : null;\n  }\n\n  public int getMaxSegmentSize() {\n    return maxSegmentSize;\n  }\n\n  public long getTimeSliceID() {\n    return timeSliceID;\n  }\n\n  public void updateSmallestDocID(int smallestDocID) {\n    // Atomic swap\n    syncData = new SyncData(Collections.unmodifiableMap(buildIndexPointers()), smallestDocID);\n  }\n\n  private Map<InvertedIndex, Integer> buildIndexPointers() {\n    Map<InvertedIndex, Integer> newIndexPointers = new HashMap<>();\n    for (InvertedIndex index : perFieldMap.values()) {\n      if (index.hasMaxPublishedPointer()) {\n        newIndexPointers.put(index, index.getMaxPublishedPointer());\n      }\n    }\n\n    return newIndexPointers;\n  }\n\n  public SyncData getSyncData() {\n    return syncData;\n  }\n\n  public AbstractFacetCountingArray getFacetCountingArray() {\n    return facetCountingArray;\n  }\n\n  public void addField(String fieldName, InvertedIndex field) {\n    perFieldMap.put(fieldName, field);\n  }\n\n  public Map<String, InvertedIndex> getPerFieldMap() {\n    return Collections.unmodifiableMap(perFieldMap);\n  }\n\n  public InvertedIndex getFieldIndex(String fieldName) {\n    return perFieldMap.get(fieldName);\n  }\n\n  public Map<String, ColumnStrideByteIndex> getNormsMap() {\n    return Collections.unmodifiableMap(normsMap);\n  }\n\n  public DeletedDocs getDeletedDocs() {\n    return deletedDocs;\n  }\n\n  /**\n   * Returns the norms index for the given field name.\n   */\n  public ColumnStrideByteIndex getNormIndex(String fieldName) {\n    return normsMap == null ? null : normsMap.get(fieldName);\n  }\n\n  /**\n   * Returns the norms index for the given field name, add if not exist.\n   */\n  public ColumnStrideByteIndex createNormIndex(String fieldName) {\n    if (normsMap == null) {\n      return null;\n    }\n    ColumnStrideByteIndex csf = normsMap.get(fieldName);\n    if (csf == null) {\n      csf = new ColumnStrideByteIndex(fieldName, maxSegmentSize);\n      normsMap.put(fieldName, csf);\n    }\n    return csf;\n  }\n\n  /**\n   * Flushes this segment to disk.\n   */\n  public void flushSegment(FlushInfo flushInfo, DataSerializer out) throws IOException {\n    getFlushHandler().flush(flushInfo, out);\n  }\n\n  public final boolean isOptimized() {\n    return this.isOptimized;\n  }\n\n  /**\n   * Returns a new atomic reader for this segment.\n   */\n  public EarlybirdIndexSegmentAtomicReader createAtomicReader() throws IOException {\n    EarlybirdIndexSegmentAtomicReader reader = doCreateAtomicReader();\n    EarlybirdIndexExtensionsData indexExtension = getIndexExtensionsData();\n    if (indexExtension != null) {\n      indexExtension.setupExtensions(reader);\n    }\n    return reader;\n  }\n\n  /**\n   * Creates a new atomic reader for this segment.\n   */\n  protected abstract EarlybirdIndexSegmentAtomicReader doCreateAtomicReader() throws IOException;\n\n  /**\n   * Creates a new segment writer for this segment.\n   */\n  public abstract EarlybirdIndexSegmentWriter createEarlybirdIndexSegmentWriter(\n      IndexWriterConfig indexWriterConfig) throws IOException;\n\n  public abstract static class AbstractSegmentDataFlushHandler\n      <S extends EarlybirdIndexExtensionsData>\n      extends Flushable.Handler<EarlybirdIndexSegmentData> {\n    protected static final String MAX_SEGMENT_SIZE_PROP_NAME = \"maxSegmentSize\";\n    protected static final String TIME_SLICE_ID_PROP_NAME = \"time_slice_id\";\n    protected static final String SMALLEST_DOCID_PROP_NAME = \"smallestDocID\";\n    protected static final String DOC_ID_MAPPER_SUBPROPS_NAME = \"doc_id_mapper\";\n    protected static final String TIME_MAPPER_SUBPROPS_NAME = \"time_mapper\";\n    public static final String IS_OPTIMIZED_PROP_NAME = \"isOptimized\";\n\n    // Abstract methods child classes should implement:\n    // 1. How to additional data structures\n    protected abstract void flushAdditionalDataStructures(\n        FlushInfo flushInfo, DataSerializer out, EarlybirdIndexSegmentData toFlush)\n            throws IOException;\n\n    // 2. Load additional data structures and construct SegmentData.\n    // Common data structures should be passed into this method to avoid code duplication.\n    // Subclasses should load additional data structures and construct a SegmentData.\n    protected abstract EarlybirdIndexSegmentData constructSegmentData(\n        FlushInfo flushInfo,\n        ConcurrentHashMap<String, InvertedIndex> perFieldMap,\n        int maxSegmentSize,\n        S indexExtension,\n        DocIDToTweetIDMapper docIdToTweetIdMapper,\n        TimeMapper timeMapper,\n        DataDeserializer in) throws IOException;\n\n    protected abstract S newIndexExtension();\n\n    protected final Schema schema;\n    protected final EarlybirdIndexExtensionsFactory indexExtensionsFactory;\n    private final Flushable.Handler<? extends DocIDToTweetIDMapper> docIdMapperFlushHandler;\n    private final Flushable.Handler<? extends TimeMapper> timeMapperFlushHandler;\n\n    public AbstractSegmentDataFlushHandler(\n        Schema schema,\n        EarlybirdIndexExtensionsFactory indexExtensionsFactory,\n        Flushable.Handler<? extends DocIDToTweetIDMapper> docIdMapperFlushHandler,\n        Flushable.Handler<? extends TimeMapper> timeMapperFlushHandler) {\n      super();\n      this.schema = schema;\n      this.indexExtensionsFactory = indexExtensionsFactory;\n      this.docIdMapperFlushHandler = docIdMapperFlushHandler;\n      this.timeMapperFlushHandler = timeMapperFlushHandler;\n    }\n\n    public AbstractSegmentDataFlushHandler(EarlybirdIndexSegmentData objectToFlush) {\n      super(objectToFlush);\n      this.schema = objectToFlush.schema;\n      this.indexExtensionsFactory = null; // factory only needed for loading SegmentData from disk\n      this.docIdMapperFlushHandler = null; // docIdMapperFlushHandler needed only for loading data\n      this.timeMapperFlushHandler = null; // timeMapperFlushHandler needed only for loading data\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out)\n        throws IOException {\n      EarlybirdIndexSegmentData segmentData = getObjectToFlush();\n\n      Preconditions.checkState(segmentData.docIdToTweetIdMapper instanceof Flushable);\n      ((Flushable) segmentData.docIdToTweetIdMapper).getFlushHandler().flush(\n          flushInfo.newSubProperties(DOC_ID_MAPPER_SUBPROPS_NAME), out);\n\n      if (segmentData.timeMapper != null) {\n        segmentData.timeMapper.getFlushHandler()\n            .flush(flushInfo.newSubProperties(TIME_MAPPER_SUBPROPS_NAME), out);\n      }\n\n      flushInfo.addBooleanProperty(IS_OPTIMIZED_PROP_NAME, segmentData.isOptimized());\n      flushInfo.addIntProperty(MAX_SEGMENT_SIZE_PROP_NAME, segmentData.getMaxSegmentSize());\n      flushInfo.addLongProperty(TIME_SLICE_ID_PROP_NAME, segmentData.getTimeSliceID());\n      flushInfo.addIntProperty(SMALLEST_DOCID_PROP_NAME,\n                               segmentData.getSyncData().getSmallestDocID());\n\n      flushIndexes(flushInfo, out, segmentData);\n\n      // Flush cluster specific data structures:\n      // FacetCountingArray, TweetIDMapper, LatLonMapper, and TimeMapper\n      flushAdditionalDataStructures(flushInfo, out, segmentData);\n    }\n\n    private void flushIndexes(\n        FlushInfo flushInfo,\n        DataSerializer out,\n        EarlybirdIndexSegmentData segmentData) throws IOException {\n      Map<String, InvertedIndex> perFieldMap = segmentData.getPerFieldMap();\n      FlushInfo fieldProps = flushInfo.newSubProperties(\"fields\");\n      long sizeBeforeFlush = out.length();\n      for (Map.Entry<String, InvertedIndex> entry : perFieldMap.entrySet()) {\n        String fieldName = entry.getKey();\n        entry.getValue().getFlushHandler().flush(fieldProps.newSubProperties(fieldName), out);\n      }\n      fieldProps.setSizeInBytes(out.length() - sizeBeforeFlush);\n    }\n\n    @Override\n    protected EarlybirdIndexSegmentData doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      DocIDToTweetIDMapper docIdToTweetIdMapper = docIdMapperFlushHandler.load(\n          flushInfo.getSubProperties(DOC_ID_MAPPER_SUBPROPS_NAME), in);\n\n      FlushInfo timeMapperFlushInfo = flushInfo.getSubProperties(TIME_MAPPER_SUBPROPS_NAME);\n      TimeMapper timeMapper =\n          timeMapperFlushInfo != null ? timeMapperFlushHandler.load(timeMapperFlushInfo, in) : null;\n\n      final int maxSegmentSize = flushInfo.getIntProperty(MAX_SEGMENT_SIZE_PROP_NAME);\n      ConcurrentHashMap<String, InvertedIndex> perFieldMap = loadIndexes(flushInfo, in);\n      return constructSegmentData(\n          flushInfo,\n          perFieldMap,\n          maxSegmentSize,\n          newIndexExtension(),\n          docIdToTweetIdMapper,\n          timeMapper,\n          in);\n    }\n\n    // Move this method into EarlybirdRealtimeIndexSegmentData (careful,\n    // we may need to increment FlushVersion because EarlybirdLuceneIndexSegmentData\n    // currently has the 'fields' subproperty in its FlushInfo as well)\n    private ConcurrentHashMap<String, InvertedIndex> loadIndexes(\n        FlushInfo flushInfo, DataDeserializer in) throws IOException {\n      ConcurrentHashMap<String, InvertedIndex> perFieldMap = new ConcurrentHashMap<>();\n\n      FlushInfo fieldProps = flushInfo.getSubProperties(\"fields\");\n      Iterator<String> fieldIterator = fieldProps.getKeyIterator();\n      while (fieldIterator.hasNext()) {\n        String fieldName = fieldIterator.next();\n        EarlybirdFieldType fieldType = schema.getFieldInfo(fieldName).getFieldType();\n        FlushInfo subProp = fieldProps.getSubProperties(fieldName);\n        boolean isOptimized = subProp.getBooleanProperty(\n            OptimizedMemoryIndex.FlushHandler.IS_OPTIMIZED_PROP_NAME);\n        final InvertedIndex invertedIndex;\n        if (isOptimized) {\n          if (!fieldType.becomesImmutable()) {\n            throw new IOException(\"Tried to load an optimized field that is not immutable: \"\n                + fieldName);\n          }\n          invertedIndex = (new OptimizedMemoryIndex.FlushHandler(fieldType)).load(subProp, in);\n        } else {\n          invertedIndex = (new InvertedRealtimeIndex.FlushHandler(\n                               fieldType, TermPointerEncoding.DEFAULT_ENCODING))\n              .load(subProp, in);\n        }\n        perFieldMap.put(fieldName, invertedIndex);\n      }\n      return perFieldMap;\n    }\n  }\n\n  public int numDocs() {\n    return docIdToTweetIdMapper.getNumDocs();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/EarlybirdIndexSegmentWriter.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport java.io.Closeable;\nimport java.io.IOException;\n\nimport org.apache.lucene.document.Document;\nimport org.apache.lucene.index.IndexReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.search.Collector;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.LeafCollector;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.Scorable;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.store.Directory;\n\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideFieldIndex;\nimport com.twitter.search.core.earlybird.index.column.DocValuesUpdate;\n\n/**\n * IndexSegmentWriter combines some common functionality between the Lucene and Realtime index\n * segment writers.\n */\npublic abstract class EarlybirdIndexSegmentWriter implements Closeable {\n\n  public EarlybirdIndexSegmentWriter() {\n  }\n\n  /**\n   * Gets the segment data this segment write is associated with.\n   * @return\n   */\n  public abstract EarlybirdIndexSegmentData getSegmentData();\n\n  /**\n   * Appends terms from the document to the document matching the query. Does not replace a field or\n   * document, actually adds to the the field in the segment.\n   */\n  public final void appendOutOfOrder(Query query, Document doc) throws IOException {\n    runQuery(query, docID -> appendOutOfOrder(doc, docID));\n  }\n\n  protected abstract void appendOutOfOrder(Document doc, int docId) throws IOException;\n\n  /**\n   * Deletes a document in this segment that matches this query.\n   */\n  public void deleteDocuments(Query query) throws IOException {\n    runQuery(query, docID -> getSegmentData().getDeletedDocs().deleteDoc(docID));\n  }\n\n  /**\n   * Updates the docvalues of a document in this segment that matches this query.\n   */\n  public void updateDocValues(Query query, String field, DocValuesUpdate update)\n      throws IOException {\n    runQuery(query, docID -> {\n        ColumnStrideFieldIndex docValues =\n            getSegmentData().getDocValuesManager().getColumnStrideFieldIndex(field);\n        if (docValues == null) {\n          return;\n        }\n\n        update.update(docValues, docID);\n      });\n  }\n\n  private void runQuery(final Query query, final OnHit onHit) throws IOException {\n    try (IndexReader reader = getSegmentData().createAtomicReader()) {\n      new IndexSearcher(reader).search(query, new Collector() {\n        @Override\n        public LeafCollector getLeafCollector(LeafReaderContext context) throws IOException {\n          return new LeafCollector() {\n            @Override\n            public void setScorer(Scorable scorer) {\n            }\n\n            @Override\n            public void collect(int docID) throws IOException {\n              onHit.hit(docID);\n            }\n          };\n        }\n\n        @Override\n        public ScoreMode scoreMode() {\n          return ScoreMode.COMPLETE_NO_SCORES;\n        }\n      });\n    }\n  }\n\n  private interface OnHit {\n    void hit(int docID) throws IOException;\n  }\n\n  /**\n   * Adds a new document to this segment. In production, this method should be called only by\n   * Expertsearch.\n   */\n  public abstract void addDocument(Document doc) throws IOException;\n\n  /**\n   * Adds a new tweet to this segment. This method should be called only by Earlybird.\n   */\n  public abstract void addTweet(Document doc, long tweetId, boolean docIsOffensive)\n      throws IOException;\n\n  /**\n   * Returns the total number of documents in the segment.\n   */\n  public abstract int numDocs() throws IOException;\n\n  /**\n   * Returns the number of documents in this segment without taking deleted docs into account.\n   * E.g. if 10 documents were added to this segments, and 5 were deleted,\n   * this method still returns 10.\n   */\n  public abstract int numDocsNoDelete() throws IOException;\n\n  /**\n   * Forces the underlying index to be merged down to a single segment.\n   */\n  public abstract void forceMerge() throws IOException;\n\n  /**\n   * Appends the provides Lucene indexes to this segment.\n   */\n  public abstract void addIndexes(Directory... dirs) throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/EarlybirdIndexableField.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport org.apache.lucene.document.Field;\nimport org.apache.lucene.index.DocValuesType;\n\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\n\npublic class EarlybirdIndexableField extends Field {\n\n  /**\n   * Creates a new indexable field with the given name, value and {@link EarlybirdFieldType}.\n   */\n  public EarlybirdIndexableField(String name, Object value, EarlybirdFieldType fieldType) {\n    super(name, fieldType);\n    if (fieldType.docValuesType() == DocValuesType.NUMERIC) {\n      if (value instanceof Number) {\n        super.fieldsData = ((Number) value).longValue();\n      } else {\n        throw new IllegalArgumentException(\"value not a number: \" + value.getClass());\n      }\n    }\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/EarlybirdLuceneIndexSegmentAtomicReader.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.BinaryDocValues;\nimport org.apache.lucene.index.FieldInfos;\nimport org.apache.lucene.index.FilterLeafReader;\nimport org.apache.lucene.index.LeafMetaData;\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.index.PointValues;\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.index.SortedDocValues;\nimport org.apache.lucene.index.SortedNumericDocValues;\nimport org.apache.lucene.index.SortedSetDocValues;\nimport org.apache.lucene.index.StoredFieldVisitor;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.store.Directory;\nimport org.apache.lucene.util.Bits;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.encoding.docvalues.CSFTypeUtil;\nimport com.twitter.search.common.encoding.features.IntegerEncodedFeatures;\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\nimport com.twitter.search.common.schema.base.Schema.FieldInfo;\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideFieldDocValues;\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideFieldIndex;\n\npublic final class EarlybirdLuceneIndexSegmentAtomicReader\n    extends EarlybirdIndexSegmentAtomicReader {\n  private abstract static class DocIdSetIteratorWrapper extends NumericDocValues {\n    private final DocIdSetIterator delegate;\n\n    public DocIdSetIteratorWrapper(DocIdSetIterator delegate) {\n      this.delegate = Preconditions.checkNotNull(delegate);\n    }\n\n    @Override\n    public int docID() {\n      return delegate.docID();\n    }\n\n    @Override\n    public int nextDoc() throws IOException {\n      return delegate.nextDoc();\n    }\n\n    @Override\n    public int advance(int target) throws IOException {\n      return delegate.advance(target);\n    }\n\n    @Override\n    public long cost() {\n      return delegate.cost();\n    }\n  }\n\n  private static class BytesRefBasedIntegerEncodedFeatures extends IntegerEncodedFeatures {\n    private final BytesRef bytesRef;\n    private final int numInts;\n\n    public BytesRefBasedIntegerEncodedFeatures(BytesRef bytesRef, int numInts) {\n      this.bytesRef = bytesRef;\n      this.numInts = numInts;\n    }\n\n    @Override\n    public int getInt(int pos) {\n      return CSFTypeUtil.convertFromBytes(bytesRef.bytes, bytesRef.offset, pos);\n    }\n\n    @Override\n    public void setInt(int pos, int value) {\n      throw new UnsupportedOperationException();\n    }\n\n    @Override\n    public int getNumInts() {\n      return numInts;\n    }\n  }\n\n  private static final int OLDEST_DOC_SKIP_INTERVAL = 256;\n\n  private final LeafReader delegate;\n\n  /**\n   * Do not add public constructors to this class. EarlybirdLuceneIndexSegmentAtomicReader instances\n   * should be created only by calling EarlybirdLuceneIndexSegmentData.createAtomicReader(), to make\n   * sure everything is set up properly (such as CSF readers).\n   */\n  EarlybirdLuceneIndexSegmentAtomicReader(\n      EarlybirdIndexSegmentData segmentData, Directory directory) throws IOException {\n    super(segmentData);\n    this.delegate = getDelegateReader(directory);\n  }\n\n  private LeafReader getDelegateReader(Directory directory) throws IOException {\n    LeafReader directoryReader =\n        EarlybirdIndexSegmentData.getLeafReaderFromOptimizedDirectory(directory);\n    return new FilterLeafReader(directoryReader) {\n      @Override\n      public NumericDocValues getNumericDocValues(String field) throws IOException {\n        EarlybirdFieldType type = getSchema().getFieldInfo(field).getFieldType();\n        if ((type == null) || !type.isCsfViewField()) {\n          return in.getNumericDocValues(field);\n        }\n\n        // Compute as many things as possible once, outside the NumericDocValues.get() call.\n        String baseFieldName = getSchema().getFieldInfo(type.getCsfViewBaseFieldId()).getName();\n        FieldInfo baseFieldInfo =\n            Preconditions.checkNotNull(getSchema().getFieldInfo(baseFieldName));\n        EarlybirdFieldType baseFieldType = baseFieldInfo.getFieldType();\n        Preconditions.checkState(!baseFieldType.isCsfVariableLength());\n        int numInts = baseFieldType.getCsfFixedLengthNumValuesPerDoc();\n        FeatureConfiguration featureConfiguration =\n            Preconditions.checkNotNull(type.getCsfViewFeatureConfiguration());\n        Preconditions.checkArgument(featureConfiguration.getValueIndex() < numInts);\n\n        if (numInts == 1) {\n          // All encoded tweet features are encoded in a single integer.\n          NumericDocValues numericDocValues = in.getNumericDocValues(baseFieldName);\n          return new DocIdSetIteratorWrapper(numericDocValues) {\n            @Override\n            public long longValue() throws IOException {\n              return (numericDocValues.longValue() & featureConfiguration.getBitMask())\n                  >> featureConfiguration.getBitStartPosition();\n            }\n\n            @Override\n            public boolean advanceExact(int target) throws IOException {\n              return numericDocValues.advanceExact(target);\n            }\n          };\n        }\n\n        BinaryDocValues binaryDocValues =\n            Preconditions.checkNotNull(in.getBinaryDocValues(baseFieldName));\n        return new DocIdSetIteratorWrapper(binaryDocValues) {\n          @Override\n          public long longValue() throws IOException {\n            BytesRef data = binaryDocValues.binaryValue();\n            IntegerEncodedFeatures encodedFeatures =\n                new BytesRefBasedIntegerEncodedFeatures(data, numInts);\n            return encodedFeatures.getFeatureValue(featureConfiguration);\n          }\n\n          @Override\n          public boolean advanceExact(int target) throws IOException {\n            return binaryDocValues.advanceExact(target);\n          }\n        };\n      }\n\n      @Override\n      public CacheHelper getCoreCacheHelper() {\n        return in.getCoreCacheHelper();\n      }\n\n      @Override\n      public CacheHelper getReaderCacheHelper() {\n        return in.getReaderCacheHelper();\n      }\n    };\n  }\n\n  private TermsEnum getTermsEnumAtTerm(Term term) throws IOException {\n    Terms terms = terms(term.field());\n    if (terms == null) {\n      return null;\n    }\n\n    TermsEnum termsEnum = terms.iterator();\n    return termsEnum.seekExact(term.bytes()) ? termsEnum : null;\n  }\n\n  @Override\n  public int getOldestDocID(Term term) throws IOException {\n    TermsEnum termsEnum = getTermsEnumAtTerm(term);\n    if (termsEnum == null) {\n      return EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND;\n    }\n\n    PostingsEnum td = termsEnum.postings(null);\n    int oldestDocID = td.nextDoc();\n    if (oldestDocID == DocIdSetIterator.NO_MORE_DOCS) {\n      return EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND;\n    }\n\n    final int docFreq = termsEnum.docFreq();\n    if (docFreq > OLDEST_DOC_SKIP_INTERVAL * 16) {\n      final int skipSize = docFreq / OLDEST_DOC_SKIP_INTERVAL;\n      do {\n        oldestDocID = td.docID();\n      } while (td.advance(oldestDocID + skipSize) != DocIdSetIterator.NO_MORE_DOCS);\n\n      td = delegate.postings(term);\n      td.advance(oldestDocID);\n    }\n\n    do {\n      oldestDocID = td.docID();\n    } while (td.nextDoc() != DocIdSetIterator.NO_MORE_DOCS);\n\n    return oldestDocID;\n  }\n\n  @Override\n  public int getTermID(Term term) throws IOException {\n    TermsEnum termsEnum = getTermsEnumAtTerm(term);\n    return termsEnum != null\n        ? (int) termsEnum.ord()\n        : EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND;\n  }\n\n  @Override\n  public Terms terms(String field) throws IOException {\n    return delegate.terms(field);\n  }\n\n  @Override\n  public FieldInfos getFieldInfos() {\n    return delegate.getFieldInfos();\n  }\n\n  @Override\n  public Bits getLiveDocs() {\n    return getDeletesView().getLiveDocs();\n  }\n\n  @Override\n  public int numDocs() {\n    return delegate.numDocs();\n  }\n\n  @Override\n  public int maxDoc() {\n    return delegate.maxDoc();\n  }\n\n  @Override\n  public void document(int docID, StoredFieldVisitor visitor) throws IOException {\n    delegate.document(docID, visitor);\n  }\n\n  @Override\n  public boolean hasDeletions() {\n    return getDeletesView().hasDeletions();\n  }\n\n  @Override\n  protected void doClose() throws IOException {\n    delegate.close();\n  }\n\n  @Override\n  public NumericDocValues getNumericDocValues(String field) throws IOException {\n    FieldInfo fieldInfo = getSegmentData().getSchema().getFieldInfo(field);\n    if (fieldInfo == null) {\n      return null;\n    }\n\n    // If this field is a CSF view field or if it's not loaded in memory, get the NumericDocValues\n    // from the delegate.\n    EarlybirdFieldType fieldType = fieldInfo.getFieldType();\n    if (fieldType.isCsfViewField() || !fieldInfo.getFieldType().isCsfLoadIntoRam()) {\n      NumericDocValues delegateVals = delegate.getNumericDocValues(field);\n      if (delegateVals != null) {\n        return delegateVals;\n      }\n    }\n\n    // The field is either loaded in memory, or the delegate doesn't have NumericDocValues for it.\n    // Return the NumericDocValues for this field stored in the DocValuesManager.\n    ColumnStrideFieldIndex csf =\n        getSegmentData().getDocValuesManager().getColumnStrideFieldIndex(field);\n    return csf != null ? new ColumnStrideFieldDocValues(csf, this) : null;\n  }\n\n  @Override\n  public BinaryDocValues getBinaryDocValues(String field) throws IOException {\n    return delegate.getBinaryDocValues(field);\n  }\n\n  @Override\n  public SortedDocValues getSortedDocValues(String field) throws IOException {\n    return delegate.getSortedDocValues(field);\n  }\n\n  @Override\n  public SortedSetDocValues getSortedSetDocValues(String field) throws IOException {\n    return delegate.getSortedSetDocValues(field);\n  }\n\n  @Override\n  public NumericDocValues getNormValues(String field) throws IOException {\n    return delegate.getNormValues(field);\n  }\n\n  @Override\n  public SortedNumericDocValues getSortedNumericDocValues(String field) throws IOException {\n    return delegate.getSortedNumericDocValues(field);\n  }\n\n  @Override\n  public void checkIntegrity() throws IOException {\n    delegate.checkIntegrity();\n  }\n\n  @Override\n  public PointValues getPointValues(String field) throws IOException {\n    return delegate.getPointValues(field);\n  }\n\n  @Override\n  public LeafMetaData getMetaData() {\n    return delegate.getMetaData();\n  }\n\n  @Override\n  public CacheHelper getCoreCacheHelper() {\n    return delegate.getCoreCacheHelper();\n  }\n\n  @Override\n  public CacheHelper getReaderCacheHelper() {\n    return delegate.getReaderCacheHelper();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/EarlybirdLuceneIndexSegmentData.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport java.io.IOException;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport org.apache.lucene.index.IndexWriterConfig;\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.store.Directory;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.facets.AbstractFacetCountingArray;\nimport com.twitter.search.core.earlybird.facets.FacetCountingArrayWriter;\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideFieldIndex;\nimport com.twitter.search.core.earlybird.index.column.DocValuesManager;\nimport com.twitter.search.core.earlybird.index.column.OptimizedDocValuesManager;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdIndexExtensionsData;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdIndexExtensionsFactory;\nimport com.twitter.search.core.earlybird.index.inverted.DeletedDocs;\nimport com.twitter.search.core.earlybird.index.inverted.InvertedIndex;\n\n/**\n * Implements {@link EarlybirdIndexSegmentData} for Lucene-based on-disk Earlybird segments.\n */\npublic final class EarlybirdLuceneIndexSegmentData extends EarlybirdIndexSegmentData {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdLuceneIndexSegmentData.class);\n\n  private final Directory directory;\n  private final EarlybirdIndexExtensionsData indexExtension;\n\n  /**\n   * Creates a new Lucene-based SegmentData instance from a lucene directory.\n   */\n  public EarlybirdLuceneIndexSegmentData(\n      Directory directory,\n      int maxSegmentSize,\n      long timeSliceID,\n      Schema schema,\n      DocIDToTweetIDMapper docIdToTweetIdMapper,\n      TimeMapper timeMapper,\n      EarlybirdIndexExtensionsFactory indexExtensionsFactory) {\n    this(\n        directory,\n        maxSegmentSize,\n        timeSliceID,\n        schema,\n        false, // isOptimized\n        0, // smallestDocId\n        new ConcurrentHashMap<>(),\n        AbstractFacetCountingArray.EMPTY_ARRAY,\n        new OptimizedDocValuesManager(schema, maxSegmentSize),\n        docIdToTweetIdMapper,\n        timeMapper,\n        indexExtensionsFactory == null\n            ? null : indexExtensionsFactory.newLuceneIndexExtensionsData());\n  }\n\n  public EarlybirdLuceneIndexSegmentData(\n      Directory directory,\n      int maxSegmentSize,\n      long timeSliceID,\n      Schema schema,\n      boolean isOptimized,\n      int smallestDocID,\n      ConcurrentHashMap<String, InvertedIndex> perFieldMap,\n      AbstractFacetCountingArray facetCountingArray,\n      DocValuesManager docValuesManager,\n      DocIDToTweetIDMapper docIdToTweetIdMapper,\n      TimeMapper timeMapper,\n      EarlybirdIndexExtensionsData indexExtension) {\n    super(maxSegmentSize,\n          timeSliceID,\n          schema,\n          isOptimized,\n          smallestDocID,\n          perFieldMap,\n          new ConcurrentHashMap<>(),\n          facetCountingArray,\n          docValuesManager,\n          null, // facetLabelProviders\n          null, // facetIDMap\n          DeletedDocs.NO_DELETES,\n          docIdToTweetIdMapper,\n          timeMapper);\n    this.directory = directory;\n    this.indexExtension = indexExtension;\n  }\n\n  public Directory getLuceneDirectory() {\n    return directory;\n  }\n\n  @Override\n  public EarlybirdIndexExtensionsData getIndexExtensionsData() {\n    return indexExtension;\n  }\n\n  @Override\n  public FacetCountingArrayWriter createFacetCountingArrayWriter() {\n    return null;\n  }\n\n  @Override\n  protected EarlybirdIndexSegmentAtomicReader doCreateAtomicReader() throws IOException {\n    // EarlybirdSegment creates one single EarlybirdIndexSegmentAtomicReader instance per segment\n    // and caches it, and the cached instance is recreated only when the segment's data changes.\n    // This is why this is a good place to reload all CSFs that should be loaded in RAM. Also, it's\n    // easier and less error-prone to do it here, than trying to track down all places that mutate\n    // the segment data and do it there.\n    LeafReader reader = getLeafReaderFromOptimizedDirectory(directory);\n    for (Schema.FieldInfo fieldInfo : getSchema().getFieldInfos()) {\n      // Load CSF into RAM based on configurations in the schema.\n      if (fieldInfo.getFieldType().getCsfType() != null\n          && fieldInfo.getFieldType().isCsfLoadIntoRam()) {\n        if (reader.getNumericDocValues(fieldInfo.getName()) != null) {\n          ColumnStrideFieldIndex index = getDocValuesManager().addColumnStrideField(\n              fieldInfo.getName(), fieldInfo.getFieldType());\n          index.load(reader, fieldInfo.getName());\n        } else {\n          LOG.warn(\"Field {} does not have NumericDocValues.\", fieldInfo.getName());\n        }\n      }\n    }\n\n    return new EarlybirdLuceneIndexSegmentAtomicReader(this, directory);\n  }\n\n  @Override\n  public EarlybirdIndexSegmentWriter createEarlybirdIndexSegmentWriter(\n      IndexWriterConfig indexWriterConfig) throws IOException {\n    return new EarlybirdLuceneIndexSegmentWriter(this, indexWriterConfig);\n  }\n\n  @Override\n  public EarlybirdIndexSegmentData.AbstractSegmentDataFlushHandler getFlushHandler() {\n    return new OnDiskSegmentDataFlushHandler(this);\n  }\n\n  public static class OnDiskSegmentDataFlushHandler\n      extends AbstractSegmentDataFlushHandler<EarlybirdIndexExtensionsData> {\n    private final Directory directory;\n\n    public OnDiskSegmentDataFlushHandler(EarlybirdLuceneIndexSegmentData objectToFlush) {\n      super(objectToFlush);\n      this.directory = objectToFlush.directory;\n    }\n\n    public OnDiskSegmentDataFlushHandler(\n        Schema schema,\n        Directory directory,\n        EarlybirdIndexExtensionsFactory indexExtensionsFactory,\n        Flushable.Handler<? extends DocIDToTweetIDMapper> docIdMapperFlushHandler,\n        Flushable.Handler<? extends TimeMapper> timeMapperFlushHandler) {\n      super(schema, indexExtensionsFactory, docIdMapperFlushHandler, timeMapperFlushHandler);\n      this.directory = directory;\n    }\n\n    @Override\n    protected EarlybirdIndexExtensionsData newIndexExtension() {\n      return indexExtensionsFactory.newLuceneIndexExtensionsData();\n    }\n\n    @Override\n    protected void flushAdditionalDataStructures(\n        FlushInfo flushInfo, DataSerializer out, EarlybirdIndexSegmentData toFlush) {\n    }\n\n    @Override\n    protected EarlybirdIndexSegmentData constructSegmentData(\n        FlushInfo flushInfo,\n        ConcurrentHashMap<String, InvertedIndex> perFieldMap,\n        int maxSegmentSize,\n        EarlybirdIndexExtensionsData indexExtension,\n        DocIDToTweetIDMapper docIdToTweetIdMapper,\n        TimeMapper timeMapper,\n        DataDeserializer in) {\n      return new EarlybirdLuceneIndexSegmentData(\n          directory,\n          maxSegmentSize,\n          flushInfo.getLongProperty(TIME_SLICE_ID_PROP_NAME),\n          schema,\n          flushInfo.getBooleanProperty(IS_OPTIMIZED_PROP_NAME),\n          flushInfo.getIntProperty(SMALLEST_DOCID_PROP_NAME),\n          perFieldMap,\n          AbstractFacetCountingArray.EMPTY_ARRAY,\n          new OptimizedDocValuesManager(schema, maxSegmentSize),\n          docIdToTweetIdMapper,\n          timeMapper,\n          indexExtension);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/EarlybirdLuceneIndexSegmentWriter.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport java.io.File;\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport org.slf4j.Marker;\nimport org.slf4j.MarkerFactory;\n\nimport org.apache.lucene.document.Document;\nimport org.apache.lucene.index.IndexWriter;\nimport org.apache.lucene.index.IndexWriterConfig;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.store.Directory;\nimport org.apache.lucene.store.FSDirectory;\nimport org.apache.lucene.store.LockObtainFailedException;\n\n/**\n * EarlybirdIndexWriter implementation that's a wrapper around Lucene's {@link IndexWriter}\n * and writes Lucene segments into a {@link Directory}.\n */\npublic class EarlybirdLuceneIndexSegmentWriter extends EarlybirdIndexSegmentWriter {\n  private static final Logger LOG =\n    LoggerFactory.getLogger(EarlybirdLuceneIndexSegmentWriter.class);\n  private static final Marker FATAL = MarkerFactory.getMarker(\"FATAL\");\n\n  private final EarlybirdLuceneIndexSegmentData segmentData;\n  private final IndexWriter indexWriter;\n\n  @Override\n  public EarlybirdIndexSegmentData getSegmentData() {\n    return segmentData;\n  }\n\n  /**\n   * Construct a lucene IndexWriter-based Earlybird segment writer.\n   * This will open a Lucene IndexWriter on segmentData.getLuceneDirectory().\n   * This constructor will throw LockObtainFailedException if it cannot obtain the \"write.lock\"\n   * inside the directory segmentData.getLuceneDirectory().\n   *\n   * Don't add public constructors to this class. EarlybirdLuceneIndexSegmentWriter instances should\n   * be created only by calling EarlybirdLuceneIndexSegmentData.createEarlybirdIndexSegmentWriter(),\n   * to make sure everything is set up properly (such as CSF readers).\n   */\n  EarlybirdLuceneIndexSegmentWriter(\n      EarlybirdLuceneIndexSegmentData segmentData,\n      IndexWriterConfig indexWriterConfig) throws IOException {\n    Preconditions.checkNotNull(segmentData);\n    this.segmentData = segmentData;\n    try {\n      this.indexWriter = new IndexWriter(segmentData.getLuceneDirectory(), indexWriterConfig);\n    } catch (LockObtainFailedException e) {\n      logDebuggingInfoUponFailureToObtainLuceneWriteLock(segmentData, e);\n      // Rethrow the exception, and this Earlybird will trigger critical alerts\n      throw e;\n    }\n  }\n\n  private void logDebuggingInfoUponFailureToObtainLuceneWriteLock(\n      EarlybirdLuceneIndexSegmentData luceneIndexSegmentData,\n      LockObtainFailedException e) throws IOException {\n    // Every day, we create a new Lucene dir---we do not append into existing Lucene dirs.\n    // Supposedly, we should never fail to obtain the write lock from a fresh and empty\n    // Lucene directory.\n    // Adding debugging information for SEARCH-4454, where a timeslice roll failed because\n    // Earlybird failed to get the write lock for a new timeslice.\n    Directory dir = luceneIndexSegmentData.getLuceneDirectory();\n    LOG.error(\n      FATAL,\n      \"Unable to obtain write.lock for Lucene directory. The Lucene directory is: \" + dir,\n      e);\n\n    if (dir instanceof FSDirectory) { // this check should always be true in our current setup.\n      FSDirectory fsDir = (FSDirectory) dir;\n      // Log if the underlying directory on disk does not exist.\n      File underlyingDir = fsDir.getDirectory().toFile();\n      if (underlyingDir.exists()) {\n        LOG.info(\"Lucene directory contains the following files: \"\n            + Lists.newArrayList(fsDir.listAll()));\n      } else {\n        LOG.error(\n          FATAL,\n          \"Directory \" + underlyingDir + \" does not exist on disk.\",\n          e);\n      }\n\n      if (!underlyingDir.canWrite()) {\n        LOG.error(\n          FATAL,\n          \"Cannot write into directory \" + underlyingDir,\n          e);\n      }\n\n      File writeLockFile = new File(underlyingDir, \"write.lock\");\n      if (writeLockFile.exists()) {\n        LOG.error(\n          FATAL,\n          \"Write lock file \" + writeLockFile + \" already exists.\",\n          e);\n      }\n\n      if (!writeLockFile.canWrite()) {\n        LOG.error(\n          FATAL,\n          \"No write access to lock file: \" + writeLockFile\n            + \" Usable space: \" + underlyingDir.getUsableSpace(),\n          e);\n      }\n\n      // List all files in the segment directory\n      File segmentDir = underlyingDir.getParentFile();\n      LOG.warn(\"Segment directory contains the following files: \"\n          + Lists.newArrayList(segmentDir.list()));\n    } else {\n      LOG.warn(\"Unable to log debugging info upon failing to acquire Lucene write lock.\"\n          + \"The class of the directory is: \" + dir.getClass().getName());\n    }\n  }\n\n  @Override\n  public void addDocument(Document doc) throws IOException {\n    indexWriter.addDocument(doc);\n  }\n\n  @Override\n  public void addTweet(Document doc, long tweetId, boolean docIdOffensive) throws IOException {\n    indexWriter.addDocument(doc);\n  }\n\n  @Override\n  protected void appendOutOfOrder(Document doc, int docId) throws IOException {\n    throw new UnsupportedOperationException(\"This Lucene-based IndexWriter does not support \"\n            + \"updates and out-of-order appends.\");\n  }\n\n  @Override\n  public int numDocs() {\n    return indexWriter.getDocStats().maxDoc;\n  }\n\n  @Override\n  public int numDocsNoDelete() throws IOException {\n    return numDocs();\n  }\n\n  @Override\n  public void deleteDocuments(Query query) throws IOException {\n    super.deleteDocuments(query);\n    indexWriter.deleteDocuments(query);\n  }\n\n  @Override\n  public void addIndexes(Directory... dirs) throws IOException {\n    indexWriter.addIndexes(dirs);\n  }\n\n  @Override\n  public void forceMerge() throws IOException {\n    indexWriter.forceMerge(1);\n  }\n\n  @Override\n  public void close() throws IOException {\n    indexWriter.close();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/EarlybirdRealtimeIndexSegmentAtomicReader.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.BinaryDocValues;\nimport org.apache.lucene.index.Fields;\nimport org.apache.lucene.index.LeafMetaData;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.index.PointValues;\nimport org.apache.lucene.index.SortedDocValues;\nimport org.apache.lucene.index.SortedNumericDocValues;\nimport org.apache.lucene.index.SortedSetDocValues;\nimport org.apache.lucene.index.StoredFieldVisitor;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.search.Sort;\nimport org.apache.lucene.util.Bits;\nimport org.apache.lucene.util.Version;\n\nimport com.twitter.search.core.earlybird.facets.EarlybirdFacetDocValueSet;\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideFieldDocValues;\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideFieldIndex;\nimport com.twitter.search.core.earlybird.index.inverted.InMemoryFields;\nimport com.twitter.search.core.earlybird.index.inverted.InvertedIndex;\n\npublic final class EarlybirdRealtimeIndexSegmentAtomicReader\n    extends EarlybirdIndexSegmentAtomicReader {\n  private final Fields fields;\n  private final int maxDocId;\n  private final int numDocs;\n\n  /**\n   * Creates a new real-time reader for the given segment. Do not add public constructors to this\n   * class. EarlybirdRealtimeIndexSegmentAtomicReader instances should be created only by calling\n   * EarlybirdRealtimeIndexSegmentData.createAtomicReader(), to make sure everything is set up\n   * properly (such as CSF readers).\n   */\n  EarlybirdRealtimeIndexSegmentAtomicReader(EarlybirdRealtimeIndexSegmentData segmentData) {\n    super(segmentData);\n\n    this.fields = new InMemoryFields(segmentData.getPerFieldMap(), syncData.getIndexPointers());\n\n    // We cache the highest doc ID and the number of docs, because the reader must return the same\n    // values for its entire lifetime, and the segment will get more tweets over time.\n    // These values could be slightly out of sync with 'fields', because we don't update these\n    // values atomically with the fields.\n    this.maxDocId = segmentData.getDocIDToTweetIDMapper().getPreviousDocID(Integer.MAX_VALUE);\n    this.numDocs = segmentData.getDocIDToTweetIDMapper().getNumDocs();\n  }\n\n  @Override\n  public int maxDoc() {\n    return maxDocId + 1;\n  }\n\n  @Override\n  public int numDocs() {\n    return numDocs;\n  }\n\n  @Override\n  protected void doClose() {\n    // nothing to do\n  }\n\n  @Override\n  public void document(int docID, StoredFieldVisitor visitor) {\n    // not supported\n  }\n\n  @Override\n  public int getOldestDocID(Term t) throws IOException {\n    InvertedIndex perField = getSegmentData().getPerFieldMap().get(t.field());\n    if (perField == null) {\n      return TERM_NOT_FOUND;\n    }\n    return perField.getLargestDocIDForTerm(t.bytes());\n  }\n\n  @Override\n  public int getTermID(Term t) throws IOException {\n    InvertedIndex perField = getSegmentData().getPerFieldMap().get(t.field());\n    if (perField == null) {\n      return TERM_NOT_FOUND;\n    }\n    return perField.lookupTerm(t.bytes());\n  }\n\n  @Override\n  public Bits getLiveDocs() {\n    // liveDocs contains inverted (decreasing) docIDs.\n    return getDeletesView().getLiveDocs();\n  }\n\n  @Override\n  public boolean hasDeletions() {\n    return getDeletesView().hasDeletions();\n  }\n\n  @Override\n  public Terms terms(String field) throws IOException {\n    return fields.terms(field);\n  }\n\n  @Override\n  public NumericDocValues getNumericDocValues(String field) throws IOException {\n    ColumnStrideFieldIndex csf =\n        getSegmentData().getDocValuesManager().getColumnStrideFieldIndex(field);\n    return csf != null ? new ColumnStrideFieldDocValues(csf, this) : null;\n  }\n\n  @Override\n  public boolean hasDocs() {\n    // smallestDocID is the smallest document ID that was available when this reader was created.\n    // So we need to check its value in order to decide if this reader can see any documents,\n    // because in the meantime other documents might've been added to the tweet ID mapper.\n    return getSmallestDocID() != Integer.MAX_VALUE;\n  }\n\n  @Override\n  public BinaryDocValues getBinaryDocValues(String field) {\n    return null;\n  }\n\n  @Override\n  public SortedDocValues getSortedDocValues(String field) {\n    return null;\n  }\n\n  @Override\n  public SortedSetDocValues getSortedSetDocValues(String field) {\n    // special handling for facet field\n    if (EarlybirdFacetDocValueSet.FIELD_NAME.equals(field)) {\n      return ((EarlybirdRealtimeIndexSegmentData) getSegmentData()).getFacetDocValueSet();\n    }\n\n    return null;\n  }\n\n  @Override\n  public NumericDocValues getNormValues(String field) throws IOException {\n    ColumnStrideFieldIndex csf = getSegmentData().getNormIndex(field);\n    return csf != null ? new ColumnStrideFieldDocValues(csf, this) : null;\n  }\n\n  @Override\n  public SortedNumericDocValues getSortedNumericDocValues(String field) {\n    return null;\n  }\n\n  @Override\n  public void checkIntegrity() {\n    // nothing to do\n  }\n\n  @Override\n  public PointValues getPointValues(String field) {\n    return null;\n  }\n\n  @Override\n  public LeafMetaData getMetaData() {\n    return new LeafMetaData(Version.LATEST.major, Version.LATEST, Sort.RELEVANCE);\n  }\n\n  @Override\n  public CacheHelper getCoreCacheHelper() {\n    return null;\n  }\n\n  @Override\n  public CacheHelper getReaderCacheHelper() {\n    return null;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/EarlybirdRealtimeIndexSegmentData.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport java.io.IOException;\nimport java.util.Map;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport com.google.common.collect.Maps;\n\nimport org.apache.lucene.index.IndexWriterConfig;\nimport org.apache.lucene.search.IndexSearcher;\n\nimport com.twitter.search.common.schema.SearchWhitespaceAnalyzer;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.facets.AbstractFacetCountingArray;\nimport com.twitter.search.core.earlybird.facets.EarlybirdFacetDocValueSet;\nimport com.twitter.search.core.earlybird.facets.FacetCountingArray;\nimport com.twitter.search.core.earlybird.facets.FacetIDMap;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.facets.FacetUtil;\nimport com.twitter.search.core.earlybird.facets.OptimizedFacetCountingArray;\nimport com.twitter.search.core.earlybird.index.column.DocValuesManager;\nimport com.twitter.search.core.earlybird.index.column.OptimizedDocValuesManager;\nimport com.twitter.search.core.earlybird.index.column.UnoptimizedDocValuesManager;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdIndexExtensionsFactory;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdRealtimeIndexExtensionsData;\nimport com.twitter.search.core.earlybird.index.inverted.DeletedDocs;\nimport com.twitter.search.core.earlybird.index.inverted.IndexOptimizer;\nimport com.twitter.search.core.earlybird.index.inverted.InvertedIndex;\n\n/**\n * Implements {@link EarlybirdIndexSegmentData} for real-time in-memory Earlybird segments.\n */\npublic class EarlybirdRealtimeIndexSegmentData extends EarlybirdIndexSegmentData {\n  private final EarlybirdRealtimeIndexExtensionsData indexExtension;\n\n  private EarlybirdFacetDocValueSet facetDocValueSet;\n\n  /**\n   * Creates a new empty real-time SegmentData instance.\n   */\n  public EarlybirdRealtimeIndexSegmentData(\n      int maxSegmentSize,\n      long timeSliceID,\n      Schema schema,\n      DocIDToTweetIDMapper docIdToTweetIdMapper,\n      TimeMapper timeMapper,\n      EarlybirdIndexExtensionsFactory indexExtensionsFactory) {\n    this(\n        maxSegmentSize,\n        timeSliceID,\n        schema,\n        false, // isOptimized\n        Integer.MAX_VALUE,\n        new ConcurrentHashMap<>(),\n        new FacetCountingArray(maxSegmentSize),\n        new UnoptimizedDocValuesManager(schema, maxSegmentSize),\n        Maps.newHashMapWithExpectedSize(schema.getNumFacetFields()),\n        FacetIDMap.build(schema),\n        new DeletedDocs.Default(maxSegmentSize),\n        docIdToTweetIdMapper,\n        timeMapper,\n        indexExtensionsFactory == null\n            ? null\n            : indexExtensionsFactory.newRealtimeIndexExtensionsData());\n  }\n\n  /**\n   * Creates a new real-time SegmentData instance using the passed in data structures. Usually this\n   * constructor is used by the FlushHandler after a segment was loaded from disk, but also the\n   * {@link IndexOptimizer} uses it to create an\n   * optimized segment.\n   */\n  public EarlybirdRealtimeIndexSegmentData(\n      int maxSegmentSize,\n      long timeSliceID,\n      Schema schema,\n      boolean isOptimized,\n      int smallestDocID,\n      ConcurrentHashMap<String, InvertedIndex> perFieldMap,\n      AbstractFacetCountingArray facetCountingArray,\n      DocValuesManager docValuesManager,\n      Map<String, FacetLabelProvider> facetLabelProviders,\n      FacetIDMap facetIDMap,\n      DeletedDocs deletedDocs,\n      DocIDToTweetIDMapper docIdToTweetIdMapper,\n      TimeMapper timeMapper,\n      EarlybirdRealtimeIndexExtensionsData indexExtension) {\n    super(maxSegmentSize,\n          timeSliceID,\n          schema,\n          isOptimized,\n          smallestDocID,\n          perFieldMap,\n          new ConcurrentHashMap<>(),\n          facetCountingArray,\n          docValuesManager,\n          facetLabelProviders,\n          facetIDMap,\n          deletedDocs,\n          docIdToTweetIdMapper,\n          timeMapper);\n    this.indexExtension = indexExtension;\n    this.facetDocValueSet = null;\n  }\n\n  @Override\n  public EarlybirdRealtimeIndexExtensionsData getIndexExtensionsData() {\n    return indexExtension;\n  }\n\n  /**\n   * For realtime segments, this wraps a facet datastructure into a SortedSetDocValues to\n   * comply to Lucene facet api.\n   */\n  public EarlybirdFacetDocValueSet getFacetDocValueSet() {\n    if (facetDocValueSet == null) {\n      AbstractFacetCountingArray facetCountingArray = getFacetCountingArray();\n      if (facetCountingArray != null) {\n        facetDocValueSet = new EarlybirdFacetDocValueSet(\n            facetCountingArray, getFacetLabelProviders(), getFacetIDMap());\n      }\n    }\n    return facetDocValueSet;\n  }\n\n  @Override\n  protected EarlybirdIndexSegmentAtomicReader doCreateAtomicReader() {\n    return new EarlybirdRealtimeIndexSegmentAtomicReader(this);\n  }\n\n  /**\n   * Convenience method for creating an EarlybirdIndexSegmentWriter for this segment with a default\n   * IndexSegmentWriter config.\n   */\n  public EarlybirdIndexSegmentWriter createEarlybirdIndexSegmentWriter() {\n    return createEarlybirdIndexSegmentWriter(\n        new IndexWriterConfig(new SearchWhitespaceAnalyzer()).setSimilarity(\n            IndexSearcher.getDefaultSimilarity()));\n  }\n\n  @Override\n  public EarlybirdIndexSegmentWriter createEarlybirdIndexSegmentWriter(\n      IndexWriterConfig indexWriterConfig) {\n    // Prepare the in-memory segment with all enabled CSF fields.\n    DocValuesManager docValuesManager = getDocValuesManager();\n    for (Schema.FieldInfo fieldInfo : getSchema().getFieldInfos()) {\n      if (fieldInfo.getFieldType().getCsfType() != null) {\n        docValuesManager.addColumnStrideField(fieldInfo.getName(), fieldInfo.getFieldType());\n      }\n    }\n\n    return new EarlybirdRealtimeIndexSegmentWriter(\n        this,\n        indexWriterConfig.getAnalyzer(),\n        indexWriterConfig.getSimilarity());\n  }\n\n  @Override\n  public EarlybirdIndexSegmentData.AbstractSegmentDataFlushHandler getFlushHandler() {\n    return new InMemorySegmentDataFlushHandler(this);\n  }\n\n  public static class InMemorySegmentDataFlushHandler\n      extends AbstractSegmentDataFlushHandler<EarlybirdRealtimeIndexExtensionsData> {\n    public InMemorySegmentDataFlushHandler(EarlybirdIndexSegmentData objectToFlush) {\n      super(objectToFlush);\n    }\n\n    public InMemorySegmentDataFlushHandler(\n        Schema schema,\n        EarlybirdIndexExtensionsFactory factory,\n        Flushable.Handler<? extends DocIDToTweetIDMapper> docIdMapperFlushHandler,\n        Flushable.Handler<? extends TimeMapper> timeMapperFlushHandler) {\n      super(schema, factory, docIdMapperFlushHandler, timeMapperFlushHandler);\n    }\n\n    @Override\n    protected EarlybirdRealtimeIndexExtensionsData newIndexExtension() {\n      return indexExtensionsFactory.newRealtimeIndexExtensionsData();\n    }\n\n    @Override\n    protected void flushAdditionalDataStructures(\n        FlushInfo flushInfo,\n        DataSerializer out,\n        EarlybirdIndexSegmentData segmentData) throws IOException {\n      segmentData.getFacetCountingArray().getFlushHandler()\n          .flush(flushInfo.newSubProperties(\"facet_counting_array\"), out);\n\n      // flush all column stride fields\n      segmentData.getDocValuesManager().getFlushHandler()\n          .flush(flushInfo.newSubProperties(\"doc_values\"), out);\n\n      segmentData.getFacetIDMap().getFlushHandler()\n          .flush(flushInfo.newSubProperties(\"facet_id_map\"), out);\n\n      segmentData.getDeletedDocs().getFlushHandler()\n          .flush(flushInfo.newSubProperties(\"deleted_docs\"), out);\n    }\n\n    @Override\n    protected EarlybirdIndexSegmentData constructSegmentData(\n        FlushInfo flushInfo,\n        ConcurrentHashMap<String, InvertedIndex> perFieldMap,\n        int maxSegmentSize,\n        EarlybirdRealtimeIndexExtensionsData indexExtension,\n        DocIDToTweetIDMapper docIdToTweetIdMapper,\n        TimeMapper timeMapper,\n        DataDeserializer in) throws IOException {\n      boolean isOptimized = flushInfo.getBooleanProperty(IS_OPTIMIZED_PROP_NAME);\n\n      Flushable.Handler<? extends AbstractFacetCountingArray> facetLoader = isOptimized\n          ? new OptimizedFacetCountingArray.FlushHandler()\n          : new FacetCountingArray.FlushHandler(maxSegmentSize);\n      AbstractFacetCountingArray facetCountingArray =\n          facetLoader.load(flushInfo.getSubProperties(\"facet_counting_array\"), in);\n\n      Flushable.Handler<? extends DocValuesManager> docValuesLoader = isOptimized\n          ? new OptimizedDocValuesManager.OptimizedFlushHandler(schema)\n          : new UnoptimizedDocValuesManager.UnoptimizedFlushHandler(schema);\n      DocValuesManager docValuesManager =\n          docValuesLoader.load(flushInfo.getSubProperties(\"doc_values\"), in);\n\n      FacetIDMap facetIDMap = new FacetIDMap.FlushHandler(schema)\n          .load(flushInfo.getSubProperties(\"facet_id_map\"), in);\n\n      DeletedDocs.Default deletedDocs = new DeletedDocs.Default.FlushHandler(maxSegmentSize)\n          .load(flushInfo.getSubProperties(\"deleted_docs\"), in);\n\n      return new EarlybirdRealtimeIndexSegmentData(\n          maxSegmentSize,\n          flushInfo.getLongProperty(TIME_SLICE_ID_PROP_NAME),\n          schema,\n          isOptimized,\n          flushInfo.getIntProperty(SMALLEST_DOCID_PROP_NAME),\n          perFieldMap,\n          facetCountingArray,\n          docValuesManager,\n          FacetUtil.getFacetLabelProviders(schema, perFieldMap),\n          facetIDMap,\n          deletedDocs,\n          docIdToTweetIdMapper,\n          timeMapper,\n          indexExtension);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/EarlybirdRealtimeIndexSegmentWriter.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport org.apache.lucene.analysis.Analyzer;\nimport org.apache.lucene.analysis.TokenStream;\nimport org.apache.lucene.analysis.tokenattributes.OffsetAttribute;\nimport org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute;\nimport org.apache.lucene.analysis.tokenattributes.TermToBytesRefAttribute;\nimport org.apache.lucene.document.Document;\nimport org.apache.lucene.document.Field;\nimport org.apache.lucene.facet.FacetsConfig;\nimport org.apache.lucene.index.DocValuesType;\nimport org.apache.lucene.index.FieldInvertState;\nimport org.apache.lucene.index.IndexOptions;\nimport org.apache.lucene.index.IndexableField;\nimport org.apache.lucene.index.IndexableFieldType;\nimport org.apache.lucene.search.similarities.Similarity;\nimport org.apache.lucene.store.Directory;\nimport org.apache.lucene.util.AttributeSource;\nimport org.apache.lucene.util.BytesRef;\nimport org.apache.lucene.util.BytesRefHash;\nimport org.apache.lucene.util.Version;\n\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.core.earlybird.facets.FacetCountingArrayWriter;\nimport com.twitter.search.core.earlybird.facets.FacetIDMap.FacetField;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.facets.FacetUtil;\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideByteIndex;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdRealtimeIndexExtensionsData;\nimport com.twitter.search.core.earlybird.index.inverted.EarlybirdCSFDocValuesProcessor;\nimport com.twitter.search.core.earlybird.index.inverted.InvertedRealtimeIndex;\nimport com.twitter.search.core.earlybird.index.inverted.InvertedRealtimeIndexWriter;\nimport com.twitter.search.core.earlybird.index.inverted.TermPointerEncoding;\nimport com.twitter.search.core.earlybird.index.util.AllDocsIterator;\n\n/**\n * EarlybirdIndexWriter implementation that writes realtime in-memory segments.\n * Note that it is used by both Earlybirds and ExpertSearch.\n */\npublic final class EarlybirdRealtimeIndexSegmentWriter extends EarlybirdIndexSegmentWriter {\n  private static final Logger LOG =\n    LoggerFactory.getLogger(EarlybirdRealtimeIndexSegmentWriter.class);\n  /**\n   * Maximum tweet length is 10k, setting maximum token position to 25k in case of weird unicode.\n   */\n  private static final int MAX_POSITION = 25000;\n\n  private static final String OUT_OF_ORDER_APPEND_UNSUPPORTED_STATS_PATTERN =\n      \"out_of_order_append_unsupported_for_field_%s\";\n  private static final ConcurrentHashMap<String, SearchRateCounter>\n      UNSUPPORTED_OUT_OF_ORDER_APPEND_MAP = new ConcurrentHashMap<>();\n  private static final SearchRateCounter NUM_TWEETS_DROPPED =\n      SearchRateCounter.export(\"EarlybirdRealtimeIndexSegmentWriter_num_tweets_dropped\");\n\n  private long nextFieldGen;\n\n  private HashMap<String, PerField> fields = new HashMap<>();\n  private List<PerField> fieldsInDocument = new ArrayList<>();\n\n  private final EarlybirdCSFDocValuesProcessor docValuesProcessor;\n\n  private Map<String, InvertedRealtimeIndexWriter> termHashSync = new HashMap<>();\n  private Set<String> appendedFields = new HashSet<>();\n\n  private final Analyzer analyzer;\n  private final Similarity similarity;\n\n  private final EarlybirdRealtimeIndexSegmentData segmentData;\n\n  private final Field allDocsField;\n\n  @Nullable\n  private final FacetCountingArrayWriter facetCountingArrayWriter;\n\n  /**\n   * Creates a new writer for a real-time in-memory Earlybird segment.\n   *\n   * Do not add public constructors to this class. EarlybirdRealtimeIndexSegmentWriter instances\n   * should be created only by calling\n   * EarlybirdRealtimeIndexSegmentData.createEarlybirdIndexSegmentWriter(), to make sure everything\n   * is set up properly (such as CSF readers).\n   */\n  EarlybirdRealtimeIndexSegmentWriter(\n      EarlybirdRealtimeIndexSegmentData segmentData,\n      Analyzer analyzer,\n      Similarity similarity) {\n    Preconditions.checkNotNull(segmentData);\n    this.segmentData = segmentData;\n    this.facetCountingArrayWriter = segmentData.createFacetCountingArrayWriter();\n    this.docValuesProcessor = new EarlybirdCSFDocValuesProcessor(segmentData.getDocValuesManager());\n    this.analyzer = analyzer;\n    this.similarity = similarity;\n    this.allDocsField = buildAllDocsField(segmentData);\n  }\n\n  @Override\n  public EarlybirdRealtimeIndexSegmentData getSegmentData() {\n    return segmentData;\n  }\n\n  @Override\n  public int numDocsNoDelete() {\n    return segmentData.getDocIDToTweetIDMapper().getNumDocs();\n  }\n\n  @Override\n  public void addDocument(Document doc) throws IOException {\n    // This method should be called only from Expertsearch, not tweets Earlybirds.\n    DocIDToTweetIDMapper docIdToTweetIdMapper = segmentData.getDocIDToTweetIDMapper();\n    Preconditions.checkState(docIdToTweetIdMapper instanceof SequentialDocIDMapper);\n\n    // Make sure we have space for a new doc in this segment.\n    Preconditions.checkState(docIdToTweetIdMapper.getNumDocs() < segmentData.getMaxSegmentSize(),\n                             \"Cannot add a new document to the segment, because it's full.\");\n\n    addDocument(doc, docIdToTweetIdMapper.addMapping(-1L), false);\n  }\n\n  @Override\n  public void addTweet(Document doc, long tweetId, boolean docIsOffensive) throws IOException {\n    DocIDToTweetIDMapper docIdToTweetIdMapper = segmentData.getDocIDToTweetIDMapper();\n    Preconditions.checkState(!(docIdToTweetIdMapper instanceof SequentialDocIDMapper));\n\n    // Make sure we have space for a new doc in this segment.\n    Preconditions.checkState(docIdToTweetIdMapper.getNumDocs() < segmentData.getMaxSegmentSize(),\n                             \"Cannot add a new document to the segment, because it's full.\");\n\n    Preconditions.checkNotNull(doc.getField(\n        EarlybirdFieldConstants.EarlybirdFieldConstant.CREATED_AT_FIELD.getFieldName()));\n\n    addAllDocsField(doc);\n\n    int docId = docIdToTweetIdMapper.addMapping(tweetId);\n    // Make sure we successfully assigned a doc ID to the new document/tweet before proceeding.\n    // If the docId is DocIDToTweetIDMapper.ID_NOT_FOUND then either:\n    //  1. the tweet is older than the  OutOfOrderRealtimeTweetIDMapper.segmentBoundaryTimestamp and\n    //    is too old for this segment\n    //  2. the OutOfOrderRealtimeTweetIDMapper does not have any available doc ids left\n    if (docId == DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      LOG.info(\"Could not assign doc id for tweet. Dropping tweet id \" + tweetId\n          + \" for segment with timeslice: \" + segmentData.getTimeSliceID());\n      NUM_TWEETS_DROPPED.increment();\n      return;\n    }\n\n    addDocument(doc, docId, docIsOffensive);\n  }\n\n  private void addDocument(Document doc,\n                           int docId,\n                           boolean docIsOffensive) throws IOException {\n    fieldsInDocument.clear();\n\n    long fieldGen = nextFieldGen++;\n\n    // NOTE: we need two passes here, in case there are\n    // multi-valued fields, because we must process all\n    // instances of a given field at once, since the\n    // analyzer is free to reuse TokenStream across fields\n    // (i.e., we cannot have more than one TokenStream\n    // running \"at once\"):\n\n    try {\n      for (IndexableField field : doc) {\n        if (!skipField(field.name())) {\n          processField(docId, field, fieldGen, docIsOffensive);\n        }\n      }\n    } finally {\n      // Finish each indexed field name seen in the document:\n      for (PerField field : fieldsInDocument) {\n        field.finish(docId);\n      }\n\n      // When indexing a dummy document for out-of-order updates into a loaded segment, that\n      // document gets docID set as maxSegment size. So we have to make sure that we never\n      // sync backwards in document order.\n      int smallestDocID = Math.min(docId, segmentData.getSyncData().getSmallestDocID());\n      segmentData.updateSmallestDocID(smallestDocID);\n    }\n  }\n\n  @Override\n  protected void appendOutOfOrder(Document doc, int internalDocID) throws IOException {\n    Preconditions.checkNotNull(doc);\n    fieldsInDocument.clear();\n\n    long fieldGen = nextFieldGen++;\n\n    try {\n      for (IndexableField indexableField : doc) {\n        if (!skipField(indexableField.name())) {\n          Schema.FieldInfo fi = segmentData.getSchema().getFieldInfo(indexableField.name());\n          if (fi == null) {\n            LOG.error(\"FieldInfo for \" + indexableField.name() + \" is null!\");\n            continue;\n          }\n          if (segmentData.isOptimized() && fi.getFieldType().becomesImmutable()) {\n            UNSUPPORTED_OUT_OF_ORDER_APPEND_MAP.computeIfAbsent(\n                indexableField.name(),\n                f -> SearchRateCounter.export(\n                    String.format(OUT_OF_ORDER_APPEND_UNSUPPORTED_STATS_PATTERN, f))\n            ).increment();\n            continue;\n          }\n          processField(internalDocID, indexableField, fieldGen, false);\n          appendedFields.add(indexableField.name());\n        }\n      }\n    } finally {\n      // Finish each indexed field name seen in the document:\n      for (PerField field : fieldsInDocument) {\n        field.finish(internalDocID);\n      }\n      // force sync\n      segmentData.updateSmallestDocID(segmentData.getSyncData().getSmallestDocID());\n    }\n  }\n\n  @Override\n  public void addIndexes(Directory... dirs) {\n    throw new UnsupportedOperationException(\"In realtime mode addIndexes() is currently \"\n            + \"not supported.\");\n  }\n\n  @Override\n  public void forceMerge() {\n    // we always have a single segment in realtime-mode\n  }\n\n  @Override\n  public void close() {\n    // nothing to close\n  }\n\n  private void processField(\n      int docId,\n      IndexableField field,\n      long fieldGen,\n      boolean currentDocIsOffensive) throws IOException {\n    String fieldName = field.name();\n    IndexableFieldType fieldType = field.fieldType();\n\n    // Invert indexed fields:\n    if (fieldType.indexOptions() != IndexOptions.NONE) {\n      PerField perField = getOrAddField(fieldName, fieldType);\n\n      // Whether this is the first time we have seen this field in this document.\n      boolean first = perField.fieldGen != fieldGen;\n      perField.invert(field, docId, first, currentDocIsOffensive);\n\n      if (first) {\n        fieldsInDocument.add(perField);\n        perField.fieldGen = fieldGen;\n      }\n    } else {\n      Schema.FieldInfo facetFieldInfo =\n              segmentData.getSchema().getFacetFieldByFieldName(fieldName);\n      FacetField facetField = facetFieldInfo != null\n              ? segmentData.getFacetIDMap().getFacetField(facetFieldInfo) : null;\n      EarlybirdFieldType facetFieldType = facetFieldInfo != null\n              ? facetFieldInfo.getFieldType() : null;\n      Preconditions.checkState(\n          facetFieldInfo == null || (facetField != null && facetFieldType != null));\n      if (facetField != null && facetFieldType.isUseCSFForFacetCounting()) {\n          segmentData.getFacetLabelProviders().put(\n              facetField.getFacetName(),\n              Preconditions.checkNotNull(\n                      FacetUtil.chooseFacetLabelProvider(facetFieldType, null)));\n       }\n    }\n\n    if (fieldType.docValuesType() != DocValuesType.NONE) {\n      StoredFieldsConsumerBuilder consumerBuilder = new StoredFieldsConsumerBuilder(\n              fieldName, (EarlybirdFieldType) fieldType);\n      EarlybirdRealtimeIndexExtensionsData indexExtension = segmentData.getIndexExtensionsData();\n      if (indexExtension != null) {\n        indexExtension.createStoredFieldsConsumer(consumerBuilder);\n      }\n      if (consumerBuilder.isUseDefaultConsumer()) {\n        consumerBuilder.addConsumer(docValuesProcessor);\n      }\n\n      StoredFieldsConsumer storedFieldsConsumer = consumerBuilder.build();\n      if (storedFieldsConsumer != null) {\n        storedFieldsConsumer.addField(docId, field);\n      }\n    }\n  }\n\n  /** Returns a previously created {@link PerField}, absorbing the type information from\n   * {@link org.apache.lucene.document.FieldType}, and creates a new {@link PerField} if this field\n   * name wasn't seen yet. */\n  private PerField getOrAddField(String name, IndexableFieldType fieldType) {\n    // Note that this could be a computeIfAbsent, but that allocates a closure in the hot path and\n    // slows down indexing.\n    PerField perField = fields.get(name);\n    if (perField == null) {\n      boolean omitNorms = fieldType.omitNorms() || fieldType.indexOptions() == IndexOptions.NONE;\n      perField = new PerField(this, name, fieldType.indexOptions(), omitNorms);\n      fields.put(name, perField);\n    }\n    return perField;\n  }\n\n  /** NOTE: not static: accesses at least docState, termsHash. */\n  private static final class PerField implements Comparable<PerField> {\n\n    private final EarlybirdRealtimeIndexSegmentWriter indexSegmentWriter;\n\n    private final String fieldName;\n    private final IndexOptions indexOptions;\n    private final boolean omitNorms;\n\n    private InvertedRealtimeIndex invertedField;\n    private InvertedDocConsumer indexWriter;\n\n    /** We use this to know when a PerField is seen for the\n     *  first time in the current document. */\n    private long fieldGen = -1;\n\n    // reused\n    private TokenStream tokenStream;\n\n    private int currentPosition;\n    private int currentOffset;\n    private int currentLength;\n    private int currentOverlap;\n    private int lastStartOffset;\n    private int lastPosition;\n\n    public PerField(\n        EarlybirdRealtimeIndexSegmentWriter indexSegmentWriter,\n        String fieldName,\n        IndexOptions indexOptions,\n        boolean omitNorms) {\n      this.indexSegmentWriter = indexSegmentWriter;\n      this.fieldName = fieldName;\n      this.indexOptions = indexOptions;\n      this.omitNorms = omitNorms;\n\n      initInvertState();\n    }\n\n    void initInvertState() {\n      // it's okay if this is null - in that case TwitterTermHashPerField\n      // will not add it to the facet array\n      final Schema.FieldInfo facetFieldInfo\n          = indexSegmentWriter.segmentData.getSchema().getFacetFieldByFieldName(fieldName);\n      final FacetField facetField = facetFieldInfo != null\n              ? indexSegmentWriter.segmentData.getFacetIDMap().getFacetField(facetFieldInfo) : null;\n      final EarlybirdFieldType facetFieldType\n          = facetFieldInfo != null ? facetFieldInfo.getFieldType() : null;\n      Preconditions.checkState(\n          facetFieldInfo == null || (facetField != null && facetFieldType != null));\n\n      if (facetField != null && facetFieldType.isUseCSFForFacetCounting()) {\n        indexSegmentWriter.segmentData.getFacetLabelProviders().put(\n            facetField.getFacetName(),\n            Preconditions.checkNotNull(\n                FacetUtil.chooseFacetLabelProvider(facetFieldType, null)));\n        return;\n      }\n\n      Schema.FieldInfo fi = indexSegmentWriter.segmentData.getSchema().getFieldInfo(fieldName);\n      final EarlybirdFieldType fieldType = fi.getFieldType();\n\n      InvertedDocConsumerBuilder consumerBuilder = new InvertedDocConsumerBuilder(\n          indexSegmentWriter.segmentData, fieldName, fieldType);\n      EarlybirdRealtimeIndexExtensionsData indexExtension =\n          indexSegmentWriter.segmentData.getIndexExtensionsData();\n      if (indexExtension != null) {\n        indexExtension.createInvertedDocConsumer(consumerBuilder);\n      }\n\n      if (consumerBuilder.isUseDefaultConsumer()) {\n        if (indexSegmentWriter.segmentData.getPerFieldMap().containsKey(fieldName)) {\n          invertedField = (InvertedRealtimeIndex) indexSegmentWriter\n              .segmentData.getPerFieldMap().get(fieldName);\n        } else {\n          invertedField = new InvertedRealtimeIndex(\n              fieldType,\n              TermPointerEncoding.DEFAULT_ENCODING,\n              fieldName);\n        }\n\n        InvertedRealtimeIndexWriter fieldWriter = new InvertedRealtimeIndexWriter(\n            invertedField, facetField, indexSegmentWriter.facetCountingArrayWriter);\n\n        if (facetField != null) {\n          Map<String, FacetLabelProvider> providerMap =\n              indexSegmentWriter.segmentData.getFacetLabelProviders();\n          if (!providerMap.containsKey(facetField.getFacetName())) {\n            providerMap.put(\n                facetField.getFacetName(),\n                Preconditions.checkNotNull(\n                    FacetUtil.chooseFacetLabelProvider(facetFieldType, invertedField)));\n          }\n        }\n\n        indexSegmentWriter.segmentData.addField(fieldName, invertedField);\n\n        if (indexSegmentWriter.appendedFields.contains(fieldName)) {\n          indexSegmentWriter.termHashSync.put(fieldName, fieldWriter);\n        }\n\n        consumerBuilder.addConsumer(fieldWriter);\n      }\n\n      indexWriter = consumerBuilder.build();\n    }\n\n    @Override\n    public int compareTo(PerField other) {\n      return this.fieldName.compareTo(other.fieldName);\n    }\n\n    @Override\n    public boolean equals(Object other) {\n      if (!(other instanceof PerField)) {\n        return false;\n      }\n\n      return this.fieldName.equals(((PerField) other).fieldName);\n    }\n\n    @Override\n    public int hashCode() {\n      return fieldName.hashCode();\n    }\n\n    public void finish(int docId) {\n      if (indexWriter != null) {\n        indexWriter.finish();\n      }\n\n      if (!omitNorms) {\n        FieldInvertState state = new FieldInvertState(\n            Version.LATEST.major,\n            fieldName,\n            indexOptions,\n            currentPosition,\n            currentLength,\n            currentOverlap,\n            currentOffset,\n            0,   // maxTermFrequency\n            0);  // uniqueTermCount\n        ColumnStrideByteIndex normsIndex =\n            indexSegmentWriter.segmentData.createNormIndex(fieldName);\n        if (normsIndex != null) {\n          normsIndex.setValue(docId, (byte) indexSegmentWriter.similarity.computeNorm(state));\n        }\n      }\n    }\n\n    /** Inverts one field for one document; first is true\n     *  if this is the first time we are seeing this field\n     *  name in this document. */\n    public void invert(IndexableField field,\n                       int docId,\n                       boolean first,\n                       boolean currentDocIsOffensive) throws IOException {\n      if (indexWriter == null) {\n        return;\n      }\n      if (first) {\n        currentPosition = -1;\n        currentOffset = 0;\n        lastPosition = 0;\n        lastStartOffset = 0;\n\n        if (invertedField != null) {\n          invertedField.incrementNumDocs();\n        }\n      }\n\n      IndexableFieldType fieldType = field.fieldType();\n      final boolean analyzed = fieldType.tokenized() && indexSegmentWriter.analyzer != null;\n      boolean succeededInProcessingField = false;\n      try {\n        tokenStream = field.tokenStream(indexSegmentWriter.analyzer, tokenStream);\n        tokenStream.reset();\n\n        PositionIncrementAttribute posIncrAttribute =\n            tokenStream.addAttribute(PositionIncrementAttribute.class);\n        OffsetAttribute offsetAttribute = tokenStream.addAttribute(OffsetAttribute.class);\n        TermToBytesRefAttribute termAtt = tokenStream.addAttribute(TermToBytesRefAttribute.class);\n\n        Set<BytesRef> seenTerms = new HashSet<>();\n        indexWriter.start(tokenStream, currentDocIsOffensive);\n        while (tokenStream.incrementToken()) {\n          // If we hit an exception in stream.next below\n          // (which is fairly common, e.g. if analyzer\n          // chokes on a given document), then it's\n          // non-aborting and (above) this one document\n          // will be marked as deleted, but still\n          // consume a docID\n\n          int posIncr = posIncrAttribute.getPositionIncrement();\n          currentPosition += posIncr;\n          if (currentPosition < lastPosition) {\n            if (posIncr == 0) {\n              throw new IllegalArgumentException(\n                  \"first position increment must be > 0 (got 0) for field '\" + field.name() + \"'\");\n            } else if (posIncr < 0) {\n              throw new IllegalArgumentException(\n                  \"position increments (and gaps) must be >= 0 (got \" + posIncr + \") for field '\"\n                  + field.name() + \"'\");\n            } else {\n              throw new IllegalArgumentException(\n                  \"position overflowed Integer.MAX_VALUE (got posIncr=\" + posIncr + \" lastPosition=\"\n                  + lastPosition + \" position=\" + currentPosition + \") for field '\" + field.name()\n                  + \"'\");\n            }\n          } else if (currentPosition > MAX_POSITION) {\n            throw new IllegalArgumentException(\n                \"position \" + currentPosition + \" is too large for field '\" + field.name()\n                + \"': max allowed position is \" + MAX_POSITION);\n          }\n          lastPosition = currentPosition;\n          if (posIncr == 0) {\n            currentOverlap++;\n          }\n\n          int startOffset = currentOffset + offsetAttribute.startOffset();\n          int endOffset = currentOffset + offsetAttribute.endOffset();\n          if (startOffset < lastStartOffset || endOffset < startOffset) {\n            throw new IllegalArgumentException(\n                \"startOffset must be non-negative, and endOffset must be >= startOffset, and \"\n                + \"offsets must not go backwards startOffset=\" + startOffset + \",endOffset=\"\n                + endOffset + \",lastStartOffset=\" + lastStartOffset + \" for field '\" + field.name()\n                + \"'\");\n          }\n          lastStartOffset = startOffset;\n          indexWriter.add(docId, currentPosition);\n          currentLength++;\n\n          BytesRef term = termAtt.getBytesRef();\n          if (seenTerms.add(term) && (invertedField != null)) {\n            invertedField.incrementSumTermDocFreq();\n          }\n        }\n\n        tokenStream.end();\n\n        currentPosition += posIncrAttribute.getPositionIncrement();\n        currentOffset += offsetAttribute.endOffset();\n        succeededInProcessingField = true;\n      } catch (BytesRefHash.MaxBytesLengthExceededException e) {\n        byte[] prefix = new byte[30];\n        BytesRef bigTerm = tokenStream.getAttribute(TermToBytesRefAttribute.class).getBytesRef();\n        System.arraycopy(bigTerm.bytes, bigTerm.offset, prefix, 0, 30);\n        String msg = \"Document contains at least one immense term in field=\\\"\" + fieldName\n                + \"\\\" (whose UTF8 encoding is longer than the max length), all of \"\n                + \"which were skipped.\" + \"Please correct the analyzer to not produce such terms. \"\n                + \"The prefix of the first immense term is: '\" + Arrays.toString(prefix)\n                + \"...', original message: \" + e.getMessage();\n        LOG.warn(msg);\n        // Document will be deleted above:\n        throw new IllegalArgumentException(msg, e);\n      } finally {\n        if (!succeededInProcessingField) {\n          LOG.warn(\"An exception was thrown while processing field \" + fieldName);\n        }\n        if (tokenStream != null) {\n          try {\n            tokenStream.close();\n          } catch (IOException e) {\n            if (succeededInProcessingField) {\n              // only throw this exception if no other exception already occurred above\n              throw e;\n            } else {\n              LOG.warn(\"Exception while trying to close TokenStream.\", e);\n            }\n          }\n        }\n      }\n\n      if (analyzed) {\n        currentPosition += indexSegmentWriter.analyzer.getPositionIncrementGap(fieldName);\n        currentOffset += indexSegmentWriter.analyzer.getOffsetGap(fieldName);\n      }\n    }\n  }\n\n  @Override\n  public int numDocs() {\n    return segmentData.getDocIDToTweetIDMapper().getNumDocs();\n  }\n\n  public interface InvertedDocConsumer {\n    /**\n     * Called for each document before inversion starts.\n     */\n    void start(AttributeSource attributeSource, boolean currentDocIsOffensive);\n\n    /**\n     * Called for each token in the current document.\n     * @param docID Document id.\n     * @param position Position in the token stream for this document.\n     */\n    void add(int docID, int position) throws IOException;\n\n    /**\n     * Called after the last token was added and before the next document is processed.\n     */\n    void finish();\n  }\n\n  public interface StoredFieldsConsumer {\n    /**\n     * Adds a new stored fields.\n     */\n    void addField(int docID, IndexableField field) throws IOException;\n  }\n\n  /**\n   * This Builder allows registering listeners for a particular field of an indexable document.\n   * For each field name any number of listeners can be added.\n   *\n   * Using {@link #useDefaultConsumer} it can be specified whether this index writer will use\n   * the default consumer in addition to any additionally registered consumers.\n   */\n  public abstract static class ConsumerBuilder<T> {\n    private boolean useDefaultConsumer;\n    private final List<T> consumers;\n    private final EarlybirdFieldType fieldType;\n    private final String fieldName;\n\n    private ConsumerBuilder(String fieldName, EarlybirdFieldType fieldType) {\n      useDefaultConsumer = true;\n      consumers = Lists.newArrayList();\n      this.fieldName = fieldName;\n      this.fieldType = fieldType;\n    }\n\n    public String getFieldName() {\n      return fieldName;\n    }\n\n    public EarlybirdFieldType getFieldType() {\n      return fieldType;\n    }\n\n    /**\n     * If set to true, {@link EarlybirdRealtimeIndexSegmentWriter} will use the default consumer\n     * (e.g. build a default inverted index for an inverted field) in addition to any consumers\n     * added via {@link #addConsumer(Object)}.\n     */\n    public void setUseDefaultConsumer(boolean useDefaultConsumer) {\n      this.useDefaultConsumer = useDefaultConsumer;\n    }\n\n    public boolean isUseDefaultConsumer() {\n      return useDefaultConsumer;\n    }\n\n    /**\n     * Allows registering any number of additional consumers for the field associated with this\n     * builder.\n     */\n    public void addConsumer(T consumer) {\n      consumers.add(consumer);\n    }\n\n    T build() {\n      if (consumers.isEmpty()) {\n        return null;\n      } else if (consumers.size() == 1) {\n        return consumers.get(0);\n      } else {\n        return build(consumers);\n      }\n    }\n\n    abstract T build(List<T> consumerList);\n  }\n\n  public static final class StoredFieldsConsumerBuilder\n          extends ConsumerBuilder<StoredFieldsConsumer> {\n    private StoredFieldsConsumerBuilder(String fieldName, EarlybirdFieldType fieldType) {\n      super(fieldName, fieldType);\n    }\n\n    @Override\n    StoredFieldsConsumer build(final List<StoredFieldsConsumer> consumers) {\n      return (docID, field) -> {\n        for (StoredFieldsConsumer consumer : consumers) {\n          consumer.addField(docID, field);\n        }\n      };\n    }\n  }\n\n  public static final class InvertedDocConsumerBuilder\n      extends ConsumerBuilder<InvertedDocConsumer> {\n    private final EarlybirdIndexSegmentData segmentData;\n\n    private InvertedDocConsumerBuilder(\n        EarlybirdIndexSegmentData segmentData, String fieldName, EarlybirdFieldType fieldType) {\n      super(fieldName, fieldType);\n      this.segmentData = segmentData;\n    }\n\n    @Override\n    InvertedDocConsumer build(final List<InvertedDocConsumer> consumers) {\n      return new InvertedDocConsumer() {\n        @Override\n        public void start(AttributeSource attributeSource, boolean currentDocIsOffensive) {\n          for (InvertedDocConsumer consumer : consumers) {\n            consumer.start(attributeSource, currentDocIsOffensive);\n          }\n        }\n\n        @Override\n        public void finish() {\n          for (InvertedDocConsumer consumer : consumers) {\n            consumer.finish();\n          }\n        }\n\n        @Override\n        public void add(int docID, int position) throws IOException {\n          for (InvertedDocConsumer consumer : consumers) {\n            consumer.add(docID, position);\n          }\n        }\n      };\n    }\n\n    public EarlybirdIndexSegmentData getSegmentData() {\n      return segmentData;\n    }\n  }\n\n  /**\n   * Returns true, if a field should not be indexed.\n   * @deprecated This writer should be able to process all fields in the future.\n   */\n  @Deprecated\n  private static boolean skipField(String fieldName) {\n    // ignore lucene facet fields for realtime index, we are handling it differently for now.\n    return fieldName.startsWith(FacetsConfig.DEFAULT_INDEX_FIELD_NAME);\n  }\n\n  private static Field buildAllDocsField(EarlybirdRealtimeIndexSegmentData segmentData) {\n    String fieldName = EarlybirdFieldConstants.EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName();\n    if (segmentData.getSchema().hasField(fieldName)) {\n      Schema.FieldInfo fi = Preconditions.checkNotNull(\n          segmentData.getSchema().getFieldInfo(fieldName));\n      return new Field(fi.getName(), AllDocsIterator.ALL_DOCS_TERM, fi.getFieldType());\n    }\n\n    return null;\n  }\n\n  /**\n   * Every document must have this field and term, so that we can safely iterate through documents\n   * using {@link AllDocsIterator}. This is to prevent the problem of adding a tweet to the doc ID\n   * mapper, and returning it for a match-all query when the rest of the document hasn't been\n   * published. This could lead to queries returning incorrect results for queries that are only\n   * negations.\n   * */\n  private void addAllDocsField(Document doc) {\n    if (allDocsField != null) {\n      doc.add(allDocsField);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/QueryCacheResultForSegment.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport org.apache.lucene.search.DocIdSet;\n\n/**\n * Class to hold the actual cache which provides a doc id iterator to walk through the cache/result.\n *\n * An instance holds the results for a single query of the different ones defined in querycache.yml.\n */\npublic class QueryCacheResultForSegment {\n  private final DocIdSet docIdSet;\n  private final int smallestDocID;\n  private final long cardinality;\n\n  /**\n   * Stores query cache results.\n   *\n   * @param docIdSet Documents in the cache.\n   * @param cardinality Size of the cache.\n   * @param smallestDocID The most recently posted document contained in the cache.\n   */\n  public QueryCacheResultForSegment(DocIdSet docIdSet, long cardinality, int smallestDocID) {\n    this.docIdSet = docIdSet;\n    this.smallestDocID = smallestDocID;\n    this.cardinality = cardinality;\n  }\n\n  public DocIdSet getDocIdSet() {\n    return docIdSet;\n  }\n\n  public int getSmallestDocID() {\n    return smallestDocID;\n  }\n\n  public long getCardinality() {\n    return cardinality;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/SequentialDocIDMapper.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\n/**\n * A doc ID mapper that assigns doc IDs sequentially in decreasing order, starting with the given\n * max ID. Used by Expertsearch, which doesn't index tweets.\n */\npublic class SequentialDocIDMapper implements DocIDToTweetIDMapper {\n  private final int maxSegmentSize;\n  private int lastAssignedDocId;\n\n  public SequentialDocIDMapper(int maxSegmentSize) {\n    this.maxSegmentSize = maxSegmentSize;\n    lastAssignedDocId = maxSegmentSize;\n  }\n\n  @Override\n  public long getTweetID(int docID) {\n    // Should be used only at segment optimization time and in tests.\n    if ((docID < lastAssignedDocId) || (docID >= maxSegmentSize)) {\n      return ID_NOT_FOUND;\n    }\n\n    return docID;\n  }\n\n  @Override\n  public int getDocID(long tweetID) {\n    // Should be used only at segment optimization time and in tests.\n    if ((tweetID < lastAssignedDocId) || (tweetID >= maxSegmentSize)) {\n      return ID_NOT_FOUND;\n    }\n\n    return (int) tweetID;\n  }\n\n  @Override\n  public int getNumDocs() {\n    return maxSegmentSize - lastAssignedDocId;\n  }\n\n  @Override\n  public int getNextDocID(int docID) {\n    int nextDocID = docID + 1;\n\n    // nextDocID is larger than any doc ID that can be assigned by this mapper.\n    if (nextDocID >= maxSegmentSize) {\n      return ID_NOT_FOUND;\n    }\n\n    // nextDocID is smaller than any doc ID assigned by this mapper so far.\n    if (nextDocID < lastAssignedDocId) {\n      return lastAssignedDocId;\n    }\n\n    // nextDocID is in the range of doc IDs assigned by this mapper.\n    return nextDocID;\n  }\n\n  @Override\n  public int getPreviousDocID(int docID) {\n    int previousDocID = docID - 1;\n\n    // previousDocID is larger than any doc ID that can be assigned by this mapper.\n    if (previousDocID >= maxSegmentSize) {\n      return maxSegmentSize - 1;\n    }\n\n    // previousDocID is smaller than any doc ID assigned by this mapper so far.\n    if (previousDocID < lastAssignedDocId) {\n      return ID_NOT_FOUND;\n    }\n\n    // previousDocID is in the range of doc IDs assigned by this mapper.\n    return previousDocID;\n  }\n\n  @Override\n  public int addMapping(final long tweetID) {\n    return --lastAssignedDocId;\n  }\n\n  @Override\n  public DocIDToTweetIDMapper optimize() {\n    // Segments that use this DocIDToTweetIDMapper should never be optimized.\n    throw new UnsupportedOperationException(\"SequentialDocIDMapper cannot be optimized.\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/TimeMapper.java",
    "content": "package com.twitter.search.core.earlybird.index;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.util.io.flushable.Flushable;\n\n/**\n * Maps timestamps to the doc IDs assigned to the documents that are indexed (tweets, users, etc.).\n */\npublic interface TimeMapper extends Flushable {\n  // Unless specified, all time fields are seconds-since-epoch.\n  int ILLEGAL_TIME = Integer.MIN_VALUE;\n\n  /**\n   * Returns the time of the newest tweet in the index.\n   *\n   * @return The time of the newest tweet in the index.\n   */\n  int getLastTime();\n\n  /**\n   * Returns the time of the oldest tweet in the index.\n   *\n   * @return The time of the oldest tweet in the index.\n   */\n  int getFirstTime();\n\n  /**\n   * Returns the timestamp of the document mapped to the given doc ID, or ILLEGAL_TIME if this\n   * mapper doesn't know about this doc ID.\n   *\n   * @param docID The document's internal ID.\n   * @return The timestamp of the document mapped to the given doc ID.\n   */\n  int getTime(int docID);\n\n  /**\n   * Returns the doc ID of the first indexed document with a timestamp equal to or greater than the\n   * given timestamp.\n   *\n   * If timeSeconds is larger than the max timestamp in this mapper, smallestDocID is returned.\n   * If timeSeconds is smaller than the min timestamp in the mapper, the largest docID is returned.\n   *\n   * Note that when tweets are indexed out of order, this method might return the doc ID of a tweet\n   * with a timestamp greater than timeSeconds, even if there's a tweet with a timestamp of\n   * timeSeconds. So the callers of this method can use the returned doc ID as a starting point for\n   * iteration purposes, but should have a check that the traversed doc IDs have a timestamp in the\n   * desired range. See SinceUntilFilter.getDocIdSet() for an example.\n   *\n   * Example:\n   *   DocIds:  6, 5, 4, 3, 2, 1, 0\n   *   Times:   1, 5, 3, 4, 4, 3, 6\n   * With that data:\n   *   findFirstDocId(1, 0) should return 6.\n   *   findFirstDocId(3, 0) should return 5.\n   *   findFirstDocId(4, 0) should return 5.\n   *   findFirstDocId(5, 0) should return 5.\n   *   findFirstDocId(6, 0) should return 0.\n   *\n   * @param timeSeconds The boundary timestamp, in seconds.\n   * @param smallestDocID The doc ID to return if the given time boundary is larger than the max\n   *                      timestamp in this mapper.\n   */\n  int findFirstDocId(int timeSeconds, int smallestDocID) throws IOException;\n\n  /**\n   * Optimizes this time mapper.\n   *\n   * At segment optimization time, the doc IDs assigned to the documents in that segment might\n   * change (they might be mapped to a more compact space for performance reasons, for example).\n   * When that happens, we need to remap accordingly the doc IDs stored in the time mapper for that\n   * segment too. It would also be a good time to optimize the data stored in the time mapper.\n   *\n   * @param originalDocIdMapper The doc ID mapper used by this segment before it was optimized.\n   * @param optimizedDocIdMapper The doc ID mapper used by this segment after it was optimized.\n   * @return An optimized TimeMapper with the same tweet IDs.\n   */\n  TimeMapper optimize(DocIDToTweetIDMapper originalDocIdMapper,\n                      DocIDToTweetIDMapper optimizedDocIdMapper) throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/AbstractColumnStrideMultiIntIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.BinaryDocValues;\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.encoding.docvalues.CSFTypeUtil;\nimport com.twitter.search.common.util.io.flushable.Flushable;\n\npublic abstract class AbstractColumnStrideMultiIntIndex\n    extends ColumnStrideFieldIndex implements Flushable {\n  private static final int NUM_BYTES_PER_INT = java.lang.Integer.SIZE / java.lang.Byte.SIZE;\n\n  private final int numIntsPerField;\n\n  protected AbstractColumnStrideMultiIntIndex(String name, int numIntsPerField) {\n    super(name);\n    this.numIntsPerField = numIntsPerField;\n  }\n\n  public int getNumIntsPerField() {\n    return numIntsPerField;\n  }\n\n  @Override\n  public long get(int docID) {\n    throw new UnsupportedOperationException();\n  }\n\n  /**\n   * Returns the value stored at the given index for the given doc ID.\n   */\n  public abstract int get(int docID, int valueIndex);\n\n  /**\n   * Sets the value stored at the given index for the given doc ID.\n   */\n  public abstract void setValue(int docID, int valueIndex, int val);\n\n  @Override\n  public void load(LeafReader atomicReader, String field) throws IOException {\n    BinaryDocValues docValues = atomicReader.getBinaryDocValues(field);\n    int numBytesPerDoc = numIntsPerField * NUM_BYTES_PER_INT;\n\n    for (int docID = 0; docID < atomicReader.maxDoc(); docID++) {\n      Preconditions.checkState(docValues.advanceExact(docID));\n      BytesRef scratch = docValues.binaryValue();\n      Preconditions.checkState(\n          scratch.length == numBytesPerDoc,\n          \"Unexpected doc value length for field \" + field\n          + \": Should be \" + numBytesPerDoc + \", but was \" + scratch.length);\n\n      scratch.length = NUM_BYTES_PER_INT;\n      for (int i = 0; i < numIntsPerField; i++) {\n        setValue(docID, i, asInt(scratch));\n        scratch.offset += NUM_BYTES_PER_INT;\n      }\n    }\n  }\n\n  public void updateDocValues(BytesRef ref, int docID) {\n    for (int i = 0; i < numIntsPerField; i++) {\n      setValue(docID, i, CSFTypeUtil.convertFromBytes(ref.bytes, ref.offset, i));\n    }\n  }\n\n  private static int asInt(BytesRef b) {\n    return asInt(b, b.offset);\n  }\n\n  private static int asInt(BytesRef b, int pos) {\n    int p = pos;\n    return (b.bytes[p++] << 24) | (b.bytes[p++] << 16) | (b.bytes[p++] << 8) | (b.bytes[p] & 0xFF);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/ColumnStrideByteIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\nimport it.unimi.dsi.fastutil.ints.Int2ByteOpenHashMap;\n\npublic class ColumnStrideByteIndex extends ColumnStrideFieldIndex implements Flushable {\n  private final Int2ByteOpenHashMap values;\n  private final int maxSize;\n\n  public ColumnStrideByteIndex(String name, int maxSize) {\n    super(name);\n    values = new Int2ByteOpenHashMap(maxSize);  // default unset value is 0\n    this.maxSize = maxSize;\n  }\n\n  private ColumnStrideByteIndex(String name, Int2ByteOpenHashMap values, int maxSize) {\n    super(name);\n    this.values = values;\n    this.maxSize = maxSize;\n  }\n\n  @Override\n  public void setValue(int docID, long value) {\n    values.put(docID, (byte) value);\n  }\n\n  @Override\n  public long get(int docID) {\n    return values.get(docID);\n  }\n\n  @Override\n  public ColumnStrideFieldIndex optimize(\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    return new OptimizedColumnStrideByteIndex(this, originalTweetIdMapper, optimizedTweetIdMapper);\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<ColumnStrideByteIndex> {\n    private static final String NAME_PROP_NAME = \"fieldName\";\n    private static final String MAX_SIZE_PROP = \"maxSize\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(ColumnStrideByteIndex objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      ColumnStrideByteIndex index = getObjectToFlush();\n      flushInfo.addStringProperty(NAME_PROP_NAME, index.getName());\n      flushInfo.addIntProperty(MAX_SIZE_PROP, index.maxSize);\n\n      out.writeInt(index.values.size());\n      for (Int2ByteOpenHashMap.Entry entry : index.values.int2ByteEntrySet()) {\n        out.writeInt(entry.getIntKey());\n        out.writeByte(entry.getByteValue());\n      }\n    }\n\n    @Override\n    protected ColumnStrideByteIndex doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      int size = in.readInt();\n      int maxSize = flushInfo.getIntProperty(MAX_SIZE_PROP);\n      Int2ByteOpenHashMap map = new Int2ByteOpenHashMap(maxSize);\n      for (int i = 0; i < size; i++) {\n        map.put(in.readInt(), in.readByte());\n      }\n      return new ColumnStrideByteIndex(flushInfo.getStringProperty(NAME_PROP_NAME), map, maxSize);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/ColumnStrideFieldDocValues.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.NumericDocValues;\n\nimport com.twitter.search.core.earlybird.index.util.AllDocsIterator;\n\n/**\n * A NumericDocValues implementation that uses an AllDocsIterator to iterate through all docs, and\n * gets its values from a ColumnStrideFieldIndex instance.\n */\npublic class ColumnStrideFieldDocValues extends NumericDocValues {\n  private final ColumnStrideFieldIndex csf;\n  private final AllDocsIterator iterator;\n\n  public ColumnStrideFieldDocValues(ColumnStrideFieldIndex csf, LeafReader reader)\n      throws IOException {\n    this.csf = Preconditions.checkNotNull(csf);\n    this.iterator = new AllDocsIterator(Preconditions.checkNotNull(reader));\n  }\n\n  @Override\n  public long longValue() {\n    return csf.get(docID());\n  }\n\n  @Override\n  public int docID() {\n    return iterator.docID();\n  }\n\n  @Override\n  public int nextDoc() throws IOException {\n    return iterator.nextDoc();\n  }\n\n  @Override\n  public int advance(int target) throws IOException {\n    return iterator.advance(target);\n  }\n\n  @Override\n  public boolean advanceExact(int target) throws IOException {\n    // The javadocs for advance() and advanceExact() are inconsistent. advance() allows the target\n    // to be smaller than the current doc ID, and requires the iterator to advance the current doc\n    // ID past the target, and past the current doc ID. So essentially, advance(target) returns\n    // max(target, currentDocId + 1). At the same time, advanceExact() is undefined if the target is\n    // smaller than the current do ID (or if it's an invalid doc ID), and always returns the target.\n    // So essentially, advanceExact(target) should always set the current doc ID to the given target\n    // and if target == currentDocId, then currentDocId should not be advanced. This is why we have\n    // these extra checks here instead of moving them to advance().\n    Preconditions.checkState(\n        target >= docID(),\n        \"ColumnStrideFieldDocValues.advance() for field %s called with target %s, \"\n        + \"but the current doc ID is %s.\",\n        csf.getName(),\n        target,\n        docID());\n    if (target == docID()) {\n      return true;\n    }\n\n    // We don't need to check if we have a value for 'target', because a ColumnStrideFieldIndex\n    // instance has a value for every doc ID (though that value might be 0).\n    return advance(target) == target;\n  }\n\n  @Override\n  public long cost() {\n    return iterator.cost();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/ColumnStrideFieldIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.NumericDocValues;\n\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\n/**\n * Get an underlying data for a field by calling\n * EarlybirdIndexSegmentAtomicReader#getNumericDocValues(String).\n */\npublic abstract class ColumnStrideFieldIndex {\n  private final String name;\n\n  public ColumnStrideFieldIndex(String name) {\n    this.name = name;\n  }\n\n  public String getName() {\n    return name;\n  }\n\n  /**\n   * Returns the CSF value for the given doc ID.\n   */\n  public abstract long get(int docID);\n\n  /**\n   * Updates the CSF value for the given doc ID to the given value.\n   */\n  public void setValue(int docID, long value) {\n    throw new UnsupportedOperationException();\n  }\n\n  /**\n   * Loads the CSF from an AtomicReader.\n   */\n  public void load(LeafReader atomicReader, String field) throws IOException {\n    NumericDocValues docValues = atomicReader.getNumericDocValues(field);\n    if (docValues != null) {\n      for (int i = 0; i < atomicReader.maxDoc(); i++) {\n        if (docValues.advanceExact(i)) {\n          setValue(i, docValues.longValue());\n        }\n      }\n    }\n  }\n\n  /**\n   * Optimizes the representation of this column stride field, and remaps its doc IDs, if necessary.\n   *\n   * @param originalTweetIdMapper The original tweet ID mapper.\n   * @param optimizedTweetIdMapper The optimized tweet ID mapper.\n   * @return An optimized column stride field equivalent to this CSF,\n   *         with possibly remapped doc IDs.\n   */\n  public ColumnStrideFieldIndex optimize(\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    return this;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/ColumnStrideIntIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\nimport it.unimi.dsi.fastutil.ints.Int2IntOpenHashMap;\n\npublic class ColumnStrideIntIndex extends ColumnStrideFieldIndex implements Flushable {\n  private final Int2IntOpenHashMap values;\n  private final int maxSize;\n\n  public ColumnStrideIntIndex(String name, int maxSize) {\n    super(name);\n    values = new Int2IntOpenHashMap(maxSize);  // default unset value is 0\n    this.maxSize = maxSize;\n  }\n\n  public ColumnStrideIntIndex(String name, Int2IntOpenHashMap values, int maxSize) {\n    super(name);\n    this.values = values;\n    this.maxSize = maxSize;\n  }\n\n  @Override\n  public void setValue(int docID, long value) {\n    values.put(docID, (int) value);\n  }\n\n  @Override\n  public long get(int docID) {\n    return values.get(docID);\n  }\n\n  @Override\n  public ColumnStrideFieldIndex optimize(\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    return new OptimizedColumnStrideIntIndex(this, originalTweetIdMapper, optimizedTweetIdMapper);\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<ColumnStrideIntIndex> {\n    private static final String NAME_PROP_NAME = \"fieldName\";\n    private static final String MAX_SIZE_PROP = \"maxSize\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(ColumnStrideIntIndex objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      ColumnStrideIntIndex index = getObjectToFlush();\n      flushInfo.addStringProperty(NAME_PROP_NAME, index.getName());\n      flushInfo.addIntProperty(MAX_SIZE_PROP, index.maxSize);\n\n      out.writeInt(index.values.size());\n      for (Int2IntOpenHashMap.Entry entry : index.values.int2IntEntrySet()) {\n        out.writeInt(entry.getIntKey());\n        out.writeInt(entry.getIntValue());\n      }\n    }\n\n    @Override\n    protected ColumnStrideIntIndex doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      int size = in.readInt();\n      int maxSize = flushInfo.getIntProperty(MAX_SIZE_PROP);\n      Int2IntOpenHashMap map = new Int2IntOpenHashMap(maxSize);\n      for (int i = 0; i < size; i++) {\n        map.put(in.readInt(), in.readInt());\n      }\n      return new ColumnStrideIntIndex(flushInfo.getStringProperty(NAME_PROP_NAME), map, maxSize);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/ColumnStrideIntViewIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport com.twitter.search.common.encoding.features.IntegerEncodedFeatures;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\n/**\n * An Int CSF view on top of an {@link AbstractColumnStrideMultiIntIndex}.\n *\n * Used for decoding encoded packed features and exposing them as\n * {@link org.apache.lucene.index.NumericDocValues}.\n */\npublic class ColumnStrideIntViewIndex extends ColumnStrideFieldIndex {\n  private static class IntViewIntegerEncodedFeatures extends IntegerEncodedFeatures {\n    private final AbstractColumnStrideMultiIntIndex baseIndex;\n    private final int docID;\n\n    public IntViewIntegerEncodedFeatures(AbstractColumnStrideMultiIntIndex baseIndex, int docID) {\n      this.baseIndex = baseIndex;\n      this.docID = docID;\n    }\n\n    @Override\n    public int getInt(int pos) {\n      return baseIndex.get(docID, pos);\n    }\n\n    @Override\n    public void setInt(int pos, int value) {\n      baseIndex.setValue(docID, pos, value);\n    }\n\n    @Override\n    public int getNumInts() {\n      return baseIndex.getNumIntsPerField();\n    }\n  }\n\n  private final AbstractColumnStrideMultiIntIndex baseIndex;\n  private final FeatureConfiguration featureConfiguration;\n\n  /**\n   * Creates a new ColumnStrideIntViewIndex on top of an existing AbstractColumnStrideMultiIntIndex.\n   */\n  public ColumnStrideIntViewIndex(Schema.FieldInfo info,\n                                  AbstractColumnStrideMultiIntIndex baseIndex) {\n    super(info.getName());\n    this.baseIndex = baseIndex;\n    this.featureConfiguration = info.getFieldType().getCsfViewFeatureConfiguration();\n  }\n\n  @Override\n  public long get(int docID) {\n    IntegerEncodedFeatures encodedFeatures = new IntViewIntegerEncodedFeatures(baseIndex, docID);\n    return encodedFeatures.getFeatureValue(featureConfiguration);\n  }\n\n  @Override\n  public void setValue(int docID, long value) {\n    IntegerEncodedFeatures encodedFeatures = new IntViewIntegerEncodedFeatures(baseIndex, docID);\n    encodedFeatures.setFeatureValue(featureConfiguration, (int) value);\n  }\n\n  @Override\n  public ColumnStrideFieldIndex optimize(\n      DocIDToTweetIDMapper originalTweetIdMapper, DocIDToTweetIDMapper optimizedTweetIdMapper) {\n    throw new UnsupportedOperationException(\n        \"ColumnStrideIntViewIndex instances do not support optimization\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/ColumnStrideLongIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\nimport it.unimi.dsi.fastutil.ints.Int2LongOpenHashMap;\n\npublic class ColumnStrideLongIndex extends ColumnStrideFieldIndex implements Flushable {\n  private final Int2LongOpenHashMap values;\n  private final int maxSize;\n\n  public ColumnStrideLongIndex(String name, int maxSize) {\n    super(name);\n    values = new Int2LongOpenHashMap(maxSize);  // default unset value is 0\n    this.maxSize = maxSize;\n  }\n\n  private ColumnStrideLongIndex(String name, Int2LongOpenHashMap values, int maxSize) {\n    super(name);\n    this.values = values;\n    this.maxSize = maxSize;\n  }\n\n  @Override\n  public void setValue(int docID, long value) {\n    values.put(docID, value);\n  }\n\n  @Override\n  public long get(int docID) {\n    return values.get(docID);\n  }\n\n  @Override\n  public ColumnStrideFieldIndex optimize(\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    return new OptimizedColumnStrideLongIndex(this, originalTweetIdMapper, optimizedTweetIdMapper);\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<ColumnStrideLongIndex> {\n    private static final String NAME_PROP_NAME = \"fieldName\";\n    private static final String MAX_SIZE_PROP = \"maxSize\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(ColumnStrideLongIndex objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      ColumnStrideLongIndex index = getObjectToFlush();\n      flushInfo.addStringProperty(NAME_PROP_NAME, index.getName());\n      flushInfo.addIntProperty(MAX_SIZE_PROP, index.maxSize);\n\n      out.writeInt(index.values.size());\n      for (Int2LongOpenHashMap.Entry entry : index.values.int2LongEntrySet()) {\n        out.writeInt(entry.getIntKey());\n        out.writeLong(entry.getLongValue());\n      }\n    }\n\n    @Override\n    protected ColumnStrideLongIndex doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      int size = in.readInt();\n      int maxSize = flushInfo.getIntProperty(MAX_SIZE_PROP);\n      Int2LongOpenHashMap map = new Int2LongOpenHashMap(maxSize);\n      for (int i = 0; i < size; i++) {\n        map.put(in.readInt(), in.readLong());\n      }\n      return new ColumnStrideLongIndex(flushInfo.getStringProperty(NAME_PROP_NAME), map, maxSize);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/ColumnStrideMultiIntIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\nimport it.unimi.dsi.fastutil.ints.Int2IntOpenHashMap;\n\npublic class ColumnStrideMultiIntIndex extends AbstractColumnStrideMultiIntIndex {\n  private final Int2IntOpenHashMap[] values;\n  private final int maxSize;\n\n  public ColumnStrideMultiIntIndex(String name, int maxSize, int numIntsPerField) {\n    super(name, numIntsPerField);\n    values = new Int2IntOpenHashMap[numIntsPerField];\n    for (int i = 0; i < numIntsPerField; i++) {\n      values[i] = new Int2IntOpenHashMap(maxSize);  // default unset value is 0\n    }\n    this.maxSize = maxSize;\n  }\n\n  public ColumnStrideMultiIntIndex(String name, Int2IntOpenHashMap[] values, int maxSize) {\n    super(name, values.length);\n    this.values = values;\n    this.maxSize = maxSize;\n  }\n\n  @Override\n  public void setValue(int docID, int valueIndex, int value) {\n    values[valueIndex].put(docID, value);\n  }\n\n  @Override\n  public int get(int docID, int valueIndex) {\n    return values[valueIndex].get(docID);\n  }\n\n  @Override\n  public ColumnStrideFieldIndex optimize(\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    return new OptimizedColumnStrideMultiIntIndex(\n        this, originalTweetIdMapper, optimizedTweetIdMapper);\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<ColumnStrideMultiIntIndex> {\n    private static final String NAME_PROP_NAME = \"fieldName\";\n    private static final String MAX_SIZE_PROP = \"maxSize\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(ColumnStrideMultiIntIndex objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      ColumnStrideMultiIntIndex index = getObjectToFlush();\n      flushInfo.addStringProperty(NAME_PROP_NAME, index.getName());\n      flushInfo.addIntProperty(MAX_SIZE_PROP, index.maxSize);\n\n      out.writeInt(index.values.length);\n      for (int i = 0; i < index.values.length; i++) {\n        Int2IntOpenHashMap map = index.values[i];\n        out.writeInt(map.size());\n        for (Int2IntOpenHashMap.Entry entry : map.int2IntEntrySet()) {\n          out.writeInt(entry.getIntKey());\n          out.writeInt(entry.getIntValue());\n        }\n      }\n    }\n\n    @Override\n    protected ColumnStrideMultiIntIndex doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      int numIntsPerField = in.readInt();\n      int maxSize = flushInfo.getIntProperty(MAX_SIZE_PROP);\n      Int2IntOpenHashMap[] values = new Int2IntOpenHashMap[numIntsPerField];\n      for (int i = 0; i < numIntsPerField; i++) {\n        int size = in.readInt();\n        Int2IntOpenHashMap map = new Int2IntOpenHashMap(maxSize);\n        for (int j = 0; j < size; j++) {\n          map.put(in.readInt(), in.readInt());\n        }\n        values[i] = map;\n      }\n      return new ColumnStrideMultiIntIndex(\n          flushInfo.getStringProperty(NAME_PROP_NAME), values, maxSize);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/ConstantColumnStrideFieldIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\n/**\n * A ColumnStrideFieldIndex implementation that always returns the same value.\n */\npublic class ConstantColumnStrideFieldIndex extends ColumnStrideFieldIndex {\n  private final long defaultValue;\n\n  public ConstantColumnStrideFieldIndex(String name, long defaultValue) {\n    super(name);\n    this.defaultValue = defaultValue;\n  }\n\n  @Override\n  public long get(int docID) {\n    return defaultValue;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/DocValuesManager.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\nimport java.util.Iterator;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Maps;\nimport com.google.common.collect.Sets;\n\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\npublic abstract class DocValuesManager implements Flushable {\n  protected final Schema schema;\n  protected final int segmentSize;\n  protected final ConcurrentHashMap<String, ColumnStrideFieldIndex> columnStrideFields;\n\n  public DocValuesManager(Schema schema, int segmentSize) {\n    this(schema, segmentSize, new ConcurrentHashMap<>());\n  }\n\n  protected DocValuesManager(Schema schema,\n                             int segmentSize,\n                             ConcurrentHashMap<String, ColumnStrideFieldIndex> columnStrideFields) {\n    this.schema = Preconditions.checkNotNull(schema);\n    this.segmentSize = segmentSize;\n    this.columnStrideFields = columnStrideFields;\n  }\n\n  protected abstract ColumnStrideFieldIndex newByteCSF(String field);\n  protected abstract ColumnStrideFieldIndex newIntCSF(String field);\n  protected abstract ColumnStrideFieldIndex newLongCSF(String field);\n  protected abstract ColumnStrideFieldIndex newMultiIntCSF(String field, int numIntsPerField);\n\n  /**\n   * Optimize this doc values manager, and return a doc values manager a more compact and fast\n   * encoding for doc values (but that we can't add new doc IDs to).\n   */\n  public abstract DocValuesManager optimize(\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException;\n\n  public Set<String> getDocValueNames() {\n    return columnStrideFields.keySet();\n  }\n\n  /**\n   * Creates a new {@link ColumnStrideFieldIndex} for the given field and returns it.\n   */\n  public ColumnStrideFieldIndex addColumnStrideField(String field, EarlybirdFieldType fieldType) {\n    // For CSF view fields, we will perform the same check on the base field when we try to create\n    // a ColumnStrideFieldIndex for them in newIntViewCSF().\n    if (!fieldType.isCsfViewField()) {\n      Preconditions.checkState(\n          fieldType.isCsfLoadIntoRam(), \"Field %s is not loaded in RAM\", field);\n    }\n\n    if (columnStrideFields.containsKey(field)) {\n      return columnStrideFields.get(field);\n    }\n\n    final ColumnStrideFieldIndex index;\n    switch (fieldType.getCsfType()) {\n      case BYTE:\n        index = newByteCSF(field);\n        break;\n      case INT:\n        if (fieldType.getCsfFixedLengthNumValuesPerDoc() > 1) {\n          index = newMultiIntCSF(field, fieldType.getCsfFixedLengthNumValuesPerDoc());\n        } else if (fieldType.isCsfViewField()) {\n          index = newIntViewCSF(field);\n        } else {\n          index = newIntCSF(field);\n        }\n        break;\n      case LONG:\n        index = newLongCSF(field);\n        break;\n      default:\n        throw new RuntimeException(\"Invalid CsfType.\");\n    }\n\n    columnStrideFields.put(field, index);\n    return index;\n  }\n\n  protected ColumnStrideFieldIndex newIntViewCSF(String field) {\n    Schema.FieldInfo info = Preconditions.checkNotNull(schema.getFieldInfo(field));\n    Schema.FieldInfo baseFieldInfo = Preconditions.checkNotNull(\n        schema.getFieldInfo(info.getFieldType().getCsfViewBaseFieldId()));\n\n    Preconditions.checkState(\n        baseFieldInfo.getFieldType().isCsfLoadIntoRam(),\n        \"Field %s has a base field (%s) that is not loaded in RAM\",\n        field, baseFieldInfo.getName());\n\n    // We might not have a CSF for the base field yet.\n    ColumnStrideFieldIndex baseFieldIndex =\n        addColumnStrideField(baseFieldInfo.getName(), baseFieldInfo.getFieldType());\n    Preconditions.checkNotNull(baseFieldIndex);\n    Preconditions.checkState(baseFieldIndex instanceof AbstractColumnStrideMultiIntIndex);\n    return new ColumnStrideIntViewIndex(info, (AbstractColumnStrideMultiIntIndex) baseFieldIndex);\n  }\n\n  /**\n   * Returns the ColumnStrideFieldIndex instance for the given field.\n   */\n  public ColumnStrideFieldIndex getColumnStrideFieldIndex(String field) {\n    ColumnStrideFieldIndex docValues = columnStrideFields.get(field);\n    if (docValues == null) {\n      Schema.FieldInfo info = schema.getFieldInfo(field);\n      if (info != null && info.getFieldType().isCsfDefaultValueSet()) {\n        return new ConstantColumnStrideFieldIndex(field, info.getFieldType().getCsfDefaultValue());\n      }\n    }\n\n    return docValues;\n  }\n\n  private static final String CSF_INDEX_CLASS_NAME_PROP_NAME = \"csfIndexClassName\";\n  private static final String CSF_PROP_NAME = \"column_stride_fields\";\n  protected static final String MAX_SEGMENT_SIZE_PROP_NAME = \"maxSegmentSize\";\n\n  private static Map<String, Set<Schema.FieldInfo>> getIntViewFields(Schema schema) {\n    Map<String, Set<Schema.FieldInfo>> intViewFields = Maps.newHashMap();\n    for (Schema.FieldInfo fieldInfo : schema.getFieldInfos()) {\n      if (fieldInfo.getFieldType().isCsfViewField()) {\n        Schema.FieldInfo baseFieldInfo = Preconditions.checkNotNull(\n            schema.getFieldInfo(fieldInfo.getFieldType().getCsfViewBaseFieldId()));\n        String baseFieldName = baseFieldInfo.getName();\n        Set<Schema.FieldInfo> intViewFieldsForBaseField =\n            intViewFields.computeIfAbsent(baseFieldName, k -> Sets.newHashSet());\n        intViewFieldsForBaseField.add(fieldInfo);\n      }\n    }\n    return intViewFields;\n  }\n\n  public abstract static class FlushHandler extends Handler<DocValuesManager> {\n    private final Schema schema;\n\n    public FlushHandler(Schema schema) {\n      this.schema = schema;\n    }\n\n    public FlushHandler(DocValuesManager docValuesManager) {\n      super(docValuesManager);\n      this.schema = docValuesManager.schema;\n    }\n\n    @Override\n    public void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      long startTime = getClock().nowMillis();\n\n      DocValuesManager docValuesManager = getObjectToFlush();\n      flushInfo.addIntProperty(MAX_SEGMENT_SIZE_PROP_NAME, docValuesManager.segmentSize);\n      long sizeBeforeFlush = out.length();\n      FlushInfo csfProps = flushInfo.newSubProperties(CSF_PROP_NAME);\n      for (ColumnStrideFieldIndex csf : docValuesManager.columnStrideFields.values()) {\n      if (!(csf instanceof ColumnStrideIntViewIndex)) {\n        Preconditions.checkState(\n            csf instanceof Flushable,\n            \"Cannot flush column stride field {} of type {}\",\n            csf.getName(), csf.getClass().getCanonicalName());\n        FlushInfo info = csfProps.newSubProperties(csf.getName());\n        info.addStringProperty(CSF_INDEX_CLASS_NAME_PROP_NAME, csf.getClass().getCanonicalName());\n        ((Flushable) csf).getFlushHandler().flush(info, out);\n      }\n    }\n      csfProps.setSizeInBytes(out.length() - sizeBeforeFlush);\n      getFlushTimerStats().timerIncrement(getClock().nowMillis() - startTime);\n    }\n\n    @Override\n    public DocValuesManager doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      long startTime = getClock().nowMillis();\n      Map<String, Set<Schema.FieldInfo>> intViewFields = getIntViewFields(schema);\n\n      FlushInfo csfProps = flushInfo.getSubProperties(CSF_PROP_NAME);\n      ConcurrentHashMap<String, ColumnStrideFieldIndex> columnStrideFields =\n          new ConcurrentHashMap<>();\n\n      Iterator<String> csfPropIter = csfProps.getKeyIterator();\n      while (csfPropIter.hasNext()) {\n        String fieldName = csfPropIter.next();\n        try {\n          FlushInfo info = csfProps.getSubProperties(fieldName);\n          String className = info.getStringProperty(CSF_INDEX_CLASS_NAME_PROP_NAME);\n          Class<? extends ColumnStrideFieldIndex> fieldIndexType =\n              (Class<? extends ColumnStrideFieldIndex>) Class.forName(className);\n          Preconditions.checkNotNull(\n              fieldIndexType,\n              \"Invalid field configuration: field \" + fieldName + \" not found in config.\");\n\n          for (Class<?> c : fieldIndexType.getDeclaredClasses()) {\n            if (Handler.class.isAssignableFrom(c)) {\n              @SuppressWarnings(\"rawtypes\")\n              Handler handler = (Handler) c.newInstance();\n              ColumnStrideFieldIndex index = (ColumnStrideFieldIndex) handler.load(\n                  csfProps.getSubProperties(fieldName), in);\n              columnStrideFields.put(fieldName, index);\n\n              // If this is a base field, create ColumnStrideIntViewIndex instances for all the\n              // view fields based on it.\n              if (index instanceof AbstractColumnStrideMultiIntIndex) {\n                AbstractColumnStrideMultiIntIndex multiIntIndex =\n                    (AbstractColumnStrideMultiIntIndex) index;\n\n                // We should have AbstractColumnStrideMultiIntIndex instances only for base fields\n                // and all our base fields have views defined on top of them.\n                for (Schema.FieldInfo intViewFieldInfo : intViewFields.get(fieldName)) {\n                  columnStrideFields.put(\n                      intViewFieldInfo.getName(),\n                      new ColumnStrideIntViewIndex(intViewFieldInfo, multiIntIndex));\n                }\n              }\n\n              break;\n            }\n          }\n        } catch (ClassNotFoundException | IllegalAccessException | InstantiationException e) {\n          throw new IOException(\n              \"Invalid field configuration for column stride field: \" + fieldName, e);\n        }\n      }\n      getLoadTimerStats().timerIncrement(getClock().nowMillis() - startTime);\n\n      return createDocValuesManager(\n          schema,\n          flushInfo.getIntProperty(MAX_SEGMENT_SIZE_PROP_NAME),\n          columnStrideFields);\n    }\n\n    protected abstract DocValuesManager createDocValuesManager(\n        Schema docValuesSchema,\n        int maxSegmentSize,\n        ConcurrentHashMap<String, ColumnStrideFieldIndex> columnStrideFields);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/DocValuesUpdate.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\npublic interface DocValuesUpdate {\n  /**\n   * Performs an doc values update on the given document.\n   */\n  void update(ColumnStrideFieldIndex docValues, int docID);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/OptimizedColumnStrideByteIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\npublic class OptimizedColumnStrideByteIndex extends ColumnStrideFieldIndex implements Flushable {\n  private final byte[] values;\n\n  public OptimizedColumnStrideByteIndex(String name, int maxSize) {\n    super(name);\n    values = new byte[maxSize];\n  }\n\n  public OptimizedColumnStrideByteIndex(\n      ColumnStrideByteIndex columnStrideByteIndex,\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    super(columnStrideByteIndex.getName());\n    int maxDocId = optimizedTweetIdMapper.getPreviousDocID(Integer.MAX_VALUE);\n    values = new byte[maxDocId + 1];\n\n    int docId = optimizedTweetIdMapper.getNextDocID(Integer.MIN_VALUE);\n    while (docId != DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      int originalDocId = originalTweetIdMapper.getDocID(optimizedTweetIdMapper.getTweetID(docId));\n      setValue(docId, columnStrideByteIndex.get(originalDocId));\n      docId = optimizedTweetIdMapper.getNextDocID(docId);\n    }\n  }\n\n  private OptimizedColumnStrideByteIndex(String name, byte[] values) {\n    super(name);\n    this.values = values;\n  }\n\n  @Override\n  public void setValue(int docID, long value) {\n    this.values[docID] = (byte) value;\n  }\n\n  @Override\n  public long get(int docID) {\n    return values[docID];\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<OptimizedColumnStrideByteIndex> {\n    private static final String NAME_PROP_NAME = \"fieldName\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(OptimizedColumnStrideByteIndex objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      OptimizedColumnStrideByteIndex columnStrideByteIndex = getObjectToFlush();\n      flushInfo.addStringProperty(NAME_PROP_NAME, columnStrideByteIndex.getName());\n      out.writeByteArray(columnStrideByteIndex.values);\n    }\n\n    @Override\n    protected OptimizedColumnStrideByteIndex doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      byte[] values = in.readByteArray();\n      return new OptimizedColumnStrideByteIndex(\n          flushInfo.getStringProperty(NAME_PROP_NAME), values);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/OptimizedColumnStrideIntIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\npublic class OptimizedColumnStrideIntIndex extends ColumnStrideFieldIndex implements Flushable {\n  private final int[] values;\n\n  public OptimizedColumnStrideIntIndex(String name, int maxSize) {\n    super(name);\n    values = new int[maxSize];\n  }\n\n  public OptimizedColumnStrideIntIndex(\n      ColumnStrideIntIndex columnStrideIntIndex,\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    super(columnStrideIntIndex.getName());\n    int maxDocId = optimizedTweetIdMapper.getPreviousDocID(Integer.MAX_VALUE);\n    values = new int[maxDocId + 1];\n\n    int docId = optimizedTweetIdMapper.getNextDocID(Integer.MIN_VALUE);\n    while (docId != DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      int originalDocId = originalTweetIdMapper.getDocID(optimizedTweetIdMapper.getTweetID(docId));\n      setValue(docId, columnStrideIntIndex.get(originalDocId));\n      docId = optimizedTweetIdMapper.getNextDocID(docId);\n    }\n  }\n\n  private OptimizedColumnStrideIntIndex(String name, int[] values) {\n    super(name);\n    this.values = values;\n  }\n\n  @Override\n  public void setValue(int docID, long value) {\n    this.values[docID] = (int) value;\n  }\n\n  @Override\n  public long get(int docID) {\n    return values[docID];\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<OptimizedColumnStrideIntIndex> {\n    private static final String NAME_PROP_NAME = \"fieldName\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(OptimizedColumnStrideIntIndex objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      OptimizedColumnStrideIntIndex columnStrideIntIndex = getObjectToFlush();\n      flushInfo.addStringProperty(NAME_PROP_NAME, columnStrideIntIndex.getName());\n      out.writeIntArray(columnStrideIntIndex.values);\n    }\n\n    @Override\n    protected OptimizedColumnStrideIntIndex doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      int[] values = in.readIntArray();\n      return new OptimizedColumnStrideIntIndex(\n          flushInfo.getStringProperty(NAME_PROP_NAME), values);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/OptimizedColumnStrideLongIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\npublic class OptimizedColumnStrideLongIndex extends ColumnStrideFieldIndex implements Flushable {\n  private final long[] values;\n\n  public OptimizedColumnStrideLongIndex(String name, int maxSize) {\n    super(name);\n    values = new long[maxSize];\n  }\n\n  public OptimizedColumnStrideLongIndex(\n      ColumnStrideLongIndex columnStrideLongIndex,\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    super(columnStrideLongIndex.getName());\n    int maxDocId = optimizedTweetIdMapper.getPreviousDocID(Integer.MAX_VALUE);\n    values = new long[maxDocId + 1];\n\n    int docId = optimizedTweetIdMapper.getNextDocID(Integer.MIN_VALUE);\n    while (docId != DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      int originalDocId = originalTweetIdMapper.getDocID(optimizedTweetIdMapper.getTweetID(docId));\n      setValue(docId, columnStrideLongIndex.get(originalDocId));\n      docId = optimizedTweetIdMapper.getNextDocID(docId);\n    }\n  }\n\n  private OptimizedColumnStrideLongIndex(String name, long[] values) {\n    super(name);\n    this.values = values;\n  }\n\n  @Override\n  public void setValue(int docID, long value) {\n    this.values[docID] = value;\n  }\n\n  @Override\n  public long get(int docID) {\n    return values[docID];\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<OptimizedColumnStrideLongIndex> {\n    private static final String NAME_PROP_NAME = \"fieldName\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(OptimizedColumnStrideLongIndex objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      OptimizedColumnStrideLongIndex columnStrideLongIndex = getObjectToFlush();\n      flushInfo.addStringProperty(NAME_PROP_NAME, columnStrideLongIndex.getName());\n      out.writeLongArray(columnStrideLongIndex.values);\n    }\n\n    @Override\n    protected OptimizedColumnStrideLongIndex doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      long[] values = in.readLongArray();\n      return new OptimizedColumnStrideLongIndex(\n          flushInfo.getStringProperty(NAME_PROP_NAME), values);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/OptimizedColumnStrideMultiIntIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\npublic class OptimizedColumnStrideMultiIntIndex\n    extends AbstractColumnStrideMultiIntIndex implements Flushable {\n  private final int[] values;\n\n  public OptimizedColumnStrideMultiIntIndex(String name, int maxSize, int numIntsPerField) {\n    super(name, numIntsPerField);\n    values = new int[Math.multiplyExact(maxSize, numIntsPerField)];\n  }\n\n  public OptimizedColumnStrideMultiIntIndex(\n      ColumnStrideMultiIntIndex columnStrideMultiIntIndex,\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    super(columnStrideMultiIntIndex.getName(), columnStrideMultiIntIndex.getNumIntsPerField());\n    int maxDocId = optimizedTweetIdMapper.getPreviousDocID(Integer.MAX_VALUE);\n    values = new int[columnStrideMultiIntIndex.getNumIntsPerField() * (maxDocId + 1)];\n\n    int docId = optimizedTweetIdMapper.getNextDocID(Integer.MIN_VALUE);\n    while (docId != DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      int originalDocId = originalTweetIdMapper.getDocID(optimizedTweetIdMapper.getTweetID(docId));\n      for (int i = 0; i < columnStrideMultiIntIndex.getNumIntsPerField(); ++i) {\n        setValue(docId, i, columnStrideMultiIntIndex.get(originalDocId, i));\n      }\n      docId = optimizedTweetIdMapper.getNextDocID(docId);\n    }\n  }\n\n  private OptimizedColumnStrideMultiIntIndex(String name, int numIntsPerField, int[] values) {\n    super(name, numIntsPerField);\n    this.values = values;\n  }\n\n  @Override\n  public void setValue(int docID, int valueIndex, int value) {\n    values[docID * getNumIntsPerField() + valueIndex] = value;\n  }\n\n  @Override\n  public int get(int docID, int valueIndex) {\n    return values[docID * getNumIntsPerField() + valueIndex];\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler\n      extends Flushable.Handler<OptimizedColumnStrideMultiIntIndex> {\n    private static final String INTS_PER_FIELD_PROP_NAME = \"intsPerField\";\n    private static final String NAME_PROP_NAME = \"fieldName\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(OptimizedColumnStrideMultiIntIndex objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      OptimizedColumnStrideMultiIntIndex columnStrideMultiIntIndex = getObjectToFlush();\n      flushInfo.addStringProperty(NAME_PROP_NAME, columnStrideMultiIntIndex.getName());\n      flushInfo.addIntProperty(INTS_PER_FIELD_PROP_NAME,\n                               columnStrideMultiIntIndex.getNumIntsPerField());\n      out.writeIntArray(columnStrideMultiIntIndex.values);\n    }\n\n    @Override\n    protected OptimizedColumnStrideMultiIntIndex doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      int[] values = in.readIntArray();\n      return new OptimizedColumnStrideMultiIntIndex(\n          flushInfo.getStringProperty(NAME_PROP_NAME),\n          flushInfo.getIntProperty(INTS_PER_FIELD_PROP_NAME),\n          values);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/OptimizedDocValuesManager.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\nimport java.util.Set;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport com.google.common.collect.Sets;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\npublic class OptimizedDocValuesManager extends DocValuesManager {\n  public OptimizedDocValuesManager(Schema schema, int segmentSize) {\n    super(schema, segmentSize);\n  }\n\n  public OptimizedDocValuesManager(DocValuesManager docValuesManager,\n                                   DocIDToTweetIDMapper originalTweetIdMapper,\n                                   DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    super(docValuesManager.schema, docValuesManager.segmentSize);\n    Set<ColumnStrideIntViewIndex> intViewIndexes = Sets.newHashSet();\n    for (String fieldName : docValuesManager.columnStrideFields.keySet()) {\n      ColumnStrideFieldIndex originalColumnStrideField =\n          docValuesManager.columnStrideFields.get(fieldName);\n      if (originalColumnStrideField instanceof ColumnStrideIntViewIndex) {\n        intViewIndexes.add((ColumnStrideIntViewIndex) originalColumnStrideField);\n      } else {\n        ColumnStrideFieldIndex optimizedColumnStrideField =\n            originalColumnStrideField.optimize(originalTweetIdMapper, optimizedTweetIdMapper);\n        columnStrideFields.put(fieldName, optimizedColumnStrideField);\n      }\n    }\n\n    // We have to process the ColumnStrideIntViewIndex instances after we process all other CSFs,\n    // because we need to make sure we've optimized the CSFs for the base fields.\n    for (ColumnStrideIntViewIndex intViewIndex : intViewIndexes) {\n      String fieldName = intViewIndex.getName();\n      columnStrideFields.put(fieldName, newIntViewCSF(fieldName));\n    }\n  }\n\n  private OptimizedDocValuesManager(\n      Schema schema,\n      int segmentSize,\n      ConcurrentHashMap<String, ColumnStrideFieldIndex> columnStrideFields) {\n    super(schema, segmentSize, columnStrideFields);\n  }\n\n  @Override\n  protected ColumnStrideFieldIndex newByteCSF(String field) {\n    return new OptimizedColumnStrideByteIndex(field, segmentSize);\n  }\n\n  @Override\n  protected ColumnStrideFieldIndex newIntCSF(String field) {\n    return new OptimizedColumnStrideIntIndex(field, segmentSize);\n  }\n\n  @Override\n  protected ColumnStrideFieldIndex newLongCSF(String field) {\n    return new OptimizedColumnStrideLongIndex(field, segmentSize);\n  }\n\n  @Override\n  protected ColumnStrideFieldIndex newMultiIntCSF(String field, int numIntsPerField) {\n    return new OptimizedColumnStrideMultiIntIndex(field, segmentSize, numIntsPerField);\n  }\n\n  @Override\n  public DocValuesManager optimize(DocIDToTweetIDMapper originalTweetIdMapper,\n                                   DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    return this;\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new OptimizedFlushHandler(this);\n  }\n\n  public static class OptimizedFlushHandler extends FlushHandler {\n    public OptimizedFlushHandler(Schema schema) {\n      super(schema);\n    }\n\n    private OptimizedFlushHandler(DocValuesManager docValuesManager) {\n      super(docValuesManager);\n    }\n\n    @Override\n    protected DocValuesManager createDocValuesManager(\n        Schema schema,\n        int maxSegmentSize,\n        ConcurrentHashMap<String, ColumnStrideFieldIndex> columnStrideFields) {\n      return new OptimizedDocValuesManager(schema, maxSegmentSize, columnStrideFields);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/column/UnoptimizedDocValuesManager.java",
    "content": "package com.twitter.search.core.earlybird.index.column;\n\nimport java.io.IOException;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\npublic class UnoptimizedDocValuesManager extends DocValuesManager {\n  public UnoptimizedDocValuesManager(Schema schema, int segmentSize) {\n    super(schema, segmentSize);\n  }\n\n  private UnoptimizedDocValuesManager(\n      Schema schema,\n      int segmentSize,\n      ConcurrentHashMap<String, ColumnStrideFieldIndex> columnStrideFields) {\n    super(schema, segmentSize, columnStrideFields);\n  }\n\n  @Override\n  protected ColumnStrideFieldIndex newByteCSF(String field) {\n    return new ColumnStrideByteIndex(field, segmentSize);\n  }\n\n  @Override\n  protected ColumnStrideFieldIndex newIntCSF(String field) {\n    return new ColumnStrideIntIndex(field, segmentSize);\n  }\n\n  @Override\n  protected ColumnStrideFieldIndex newLongCSF(String field) {\n    return new ColumnStrideLongIndex(field, segmentSize);\n  }\n\n  @Override\n  protected ColumnStrideFieldIndex newMultiIntCSF(String field, int numIntsPerField) {\n    return new ColumnStrideMultiIntIndex(field, segmentSize, numIntsPerField);\n  }\n\n  @Override\n  public DocValuesManager optimize(DocIDToTweetIDMapper originalTweetIdMapper,\n                                   DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    return new OptimizedDocValuesManager(this, originalTweetIdMapper, optimizedTweetIdMapper);\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new UnoptimizedFlushHandler(this);\n  }\n\n  public static class UnoptimizedFlushHandler extends FlushHandler {\n    public UnoptimizedFlushHandler(Schema schema) {\n      super(schema);\n    }\n\n    private UnoptimizedFlushHandler(DocValuesManager docValuesManager) {\n      super(docValuesManager);\n    }\n\n    @Override\n    protected DocValuesManager createDocValuesManager(\n        Schema schema,\n        int maxSegmentSize,\n        ConcurrentHashMap<String, ColumnStrideFieldIndex> columnStrideFields) {\n      return new UnoptimizedDocValuesManager(schema, maxSegmentSize, columnStrideFields);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/extensions/EarlybirdIndexExtensionsData.java",
    "content": "package com.twitter.search.core.earlybird.index.extensions;\n\nimport java.io.IOException;\n\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\n/**\n * Base index extensions class.\n */\npublic interface EarlybirdIndexExtensionsData {\n  /**\n   * Sets up the extensions for the given reader.\n   */\n  void setupExtensions(EarlybirdIndexSegmentAtomicReader atomicReader) throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/extensions/EarlybirdIndexExtensionsFactory.java",
    "content": "package com.twitter.search.core.earlybird.index.extensions;\n\n/**\n * Base class to implement factories that create realtime and Lucene index extensions.\n *\n * The factory needs to be able to create instances for new segments, as well as load\n * index extensions of existing segments from disk.\n */\npublic abstract class EarlybirdIndexExtensionsFactory {\n  /**\n   * Returns the {@link EarlybirdRealtimeIndexExtensionsData} instance to be used for a new segment.\n   */\n  public abstract EarlybirdRealtimeIndexExtensionsData newRealtimeIndexExtensionsData();\n\n  /**\n   * Returns the {@link EarlybirdIndexExtensionsData} instance to be used for a new Lucene segment.\n   */\n  public abstract EarlybirdIndexExtensionsData newLuceneIndexExtensionsData();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/extensions/EarlybirdRealtimeIndexExtensionsData.java",
    "content": "package com.twitter.search.core.earlybird.index.extensions;\n\nimport com.twitter.search.core.earlybird.index.EarlybirdRealtimeIndexSegmentWriter;\n\n/**\n * An index extensions implementation for real-time Earlybird indexes.\n */\npublic interface EarlybirdRealtimeIndexExtensionsData extends EarlybirdIndexExtensionsData {\n  /**\n   * Optionally, an implementing class can provide a custom consumer for inverted fields (i.e. streams of tokens).\n   */\n  void createInvertedDocConsumer(\n      EarlybirdRealtimeIndexSegmentWriter.InvertedDocConsumerBuilder builder);\n\n  /**\n   * Optionally, an implementing class can provide a custom consumer for stored fields (e.g. doc values fields).\n   */\n  void createStoredFieldsConsumer(\n      EarlybirdRealtimeIndexSegmentWriter.StoredFieldsConsumerBuilder builder);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/BaseByteBlockPool.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\nimport java.util.Arrays;\n\nimport org.apache.lucene.store.DataInput;\nimport org.apache.lucene.store.DataOutput;\nimport org.apache.lucene.util.ArrayUtil;\nimport org.apache.lucene.util.ByteBlockPool;\nimport org.apache.lucene.util.BytesRef;\n\nimport static org.apache.lucene.util.RamUsageEstimator.NUM_BYTES_OBJECT_REF;\n\n/**\n * Base class for BlockPools backed by byte[] arrays.\n */\npublic abstract class BaseByteBlockPool {\n  /**\n   * The extra object with final array is necessary to guarantee visibility to\n   * other threads without synchronization/using volatile.\n   *\n   * From 'Java Concurrency in practice' by Brian Goetz, p. 349:\n   *\n   * \"Initialization safety guarantees that for properly constructed objects, all\n   *  threads will see the correct values of final fields that were set by the con-\n   *  structor, regardless of how the object is published. Further, any variables\n   *  that can be reached through a final field of a properly constructed object\n   *  (such as the elements of a final array or the contents of a HashMap refer-\n   *  enced by a final field) are also guaranteed to be visible to other threads.\"\n   */\n  public static final class Pool {\n    public final byte[][] buffers;\n\n    public Pool(byte[][] buffers) {\n      this.buffers = buffers;\n    }\n\n    public byte[][] getBlocks() {\n      return buffers;\n    }\n  }\n\n  public Pool pool = new Pool(new byte[10][]);\n  // The index of the current buffer in pool.buffers.\n  public int bufferUpto = -1;\n  // The number of bytes that have been written in the current buffer.\n  public int byteUpto = ByteBlockPool.BYTE_BLOCK_SIZE;\n  // The current buffer, i.e. a reference to pool.buffers[bufferUpto]\n  public byte[] buffer;\n  // The total number of bytes that have been used up to now, excluding the current buffer.\n  public int byteOffset = -ByteBlockPool.BYTE_BLOCK_SIZE;\n  // The one and only WriteStream for this pool.\n  private WriteStream writeStream = new WriteStream();\n\n  protected BaseByteBlockPool() { }\n\n  /**\n   * Used for loading flushed pool.\n   */\n  protected BaseByteBlockPool(Pool pool, int bufferUpto, int byteUpTo, int byteOffset) {\n    this.pool = pool;\n    this.bufferUpto = bufferUpto;\n    this.byteUpto = byteUpTo;\n    this.byteOffset = byteOffset;\n    if (bufferUpto >= 0) {\n      this.buffer = pool.buffers[bufferUpto];\n    }\n  }\n\n  /**\n   * Resets the index of the pool to 0 in the first buffer and resets the byte arrays of\n   * all previously allocated buffers to 0s.\n   */\n  public void reset() {\n    if (bufferUpto != -1) {\n      // We allocated at least one buffer\n\n      for (int i = 0; i < bufferUpto; i++) {\n        // Fully zero fill buffers that we fully used\n        Arrays.fill(pool.buffers[i], (byte) 0);\n      }\n\n      // Partial zero fill the final buffer\n      Arrays.fill(pool.buffers[bufferUpto], 0, byteUpto, (byte) 0);\n\n      bufferUpto = 0;\n      byteUpto = 0;\n      byteOffset = 0;\n      buffer = pool.buffers[0];\n    }\n  }\n\n  /**\n   * Switches to the next buffer and positions the index at its beginning.\n   */\n  public void nextBuffer() {\n    if (1 + bufferUpto == pool.buffers.length) {\n      byte[][] newBuffers = new byte[ArrayUtil.oversize(pool.buffers.length + 1,\n                                                           NUM_BYTES_OBJECT_REF)][];\n      System.arraycopy(pool.buffers, 0, newBuffers, 0, pool.buffers.length);\n      pool = new Pool(newBuffers);\n    }\n    buffer = pool.buffers[1 + bufferUpto] = new byte[ByteBlockPool.BYTE_BLOCK_SIZE];\n    bufferUpto++;\n\n    byteUpto = 0;\n    byteOffset += ByteBlockPool.BYTE_BLOCK_SIZE;\n  }\n\n  /**\n   * Returns the start offset of the next data that will be added to the pool, UNLESS the data is\n   * added using addBytes and avoidSplitting = true\n   */\n  public int getOffset() {\n    return byteOffset + byteUpto;\n  }\n\n  /**\n   * Returns the start offset of b in the pool\n   * @param b byte to put\n   */\n  public int addByte(byte b) {\n    int initOffset = byteOffset + byteUpto;\n    int remainingBytesInBuffer = ByteBlockPool.BYTE_BLOCK_SIZE - byteUpto;\n    // If the buffer is full, move on to the next one.\n    if (remainingBytesInBuffer <= 0) {\n      nextBuffer();\n    }\n    buffer[byteUpto] = b;\n    byteUpto++;\n    return initOffset;\n  }\n\n  /**\n   * Returns the start offset of the bytes in the pool.\n   *        If avoidSplitting is false, this is guaranteed to return the same value that would be\n   *        returned by getOffset()\n   * @param bytes source array\n   * @param length number of bytes to put\n   * @param avoidSplitting if possible (the length is less than ByteBlockPool.BYTE_BLOCK_SIZE),\n   *        the bytes will not be split across buffer boundaries. This is useful for small data\n   *        that will be read a lot (small amount of space wasted in return for avoiding copying\n   *        memory when calling getBytes).\n   */\n  public int addBytes(byte[] bytes, int offset, int length, boolean avoidSplitting) {\n    // The first time this is called, there may not be an existing buffer yet.\n    if (buffer == null) {\n      nextBuffer();\n    }\n\n    int remainingBytesInBuffer = ByteBlockPool.BYTE_BLOCK_SIZE - byteUpto;\n\n    if (avoidSplitting && length < ByteBlockPool.BYTE_BLOCK_SIZE) {\n      if (remainingBytesInBuffer < length) {\n        nextBuffer();\n      }\n      int initOffset = byteOffset + byteUpto;\n      System.arraycopy(bytes, offset, buffer, byteUpto, length);\n      byteUpto += length;\n      return initOffset;\n    } else {\n      int initOffset = byteOffset + byteUpto;\n      if (remainingBytesInBuffer < length) {\n        // Must split the bytes across buffers.\n        int remainingLength = length;\n        while (remainingLength > ByteBlockPool.BYTE_BLOCK_SIZE - byteUpto) {\n          int lengthToCopy = ByteBlockPool.BYTE_BLOCK_SIZE - byteUpto;\n          System.arraycopy(bytes, length - remainingLength + offset,\n                  buffer, byteUpto, lengthToCopy);\n          remainingLength -= lengthToCopy;\n          nextBuffer();\n        }\n        System.arraycopy(bytes, length - remainingLength + offset,\n                buffer, byteUpto, remainingLength);\n        byteUpto += remainingLength;\n      } else {\n        // Just add all bytes to the current buffer.\n        System.arraycopy(bytes, offset, buffer, byteUpto, length);\n        byteUpto += length;\n      }\n      return initOffset;\n    }\n  }\n\n  /**\n   * Default addBytes. Does not avoid splitting.\n   * @see #addBytes(byte[], int, boolean)\n   */\n  public int addBytes(byte[] bytes, int length) {\n    return addBytes(bytes, 0, length, false);\n  }\n\n  /**\n   * Default addBytes. Does not avoid splitting.\n   * @see #addBytes(byte[], int, boolean)\n   */\n  public int addBytes(byte[] bytes, int offset, int length) {\n    return addBytes(bytes, offset, length, false);\n  }\n\n  /**\n   * Reads one byte from the pool.\n   * @param offset location to read byte from\n   */\n  public byte getByte(int offset) {\n    int bufferIndex = offset >>> ByteBlockPool.BYTE_BLOCK_SHIFT;\n    int bufferOffset = offset & ByteBlockPool.BYTE_BLOCK_MASK;\n    return pool.buffers[bufferIndex][bufferOffset];\n  }\n\n  /**\n   * Returns false if offset is invalid or there aren't these many bytes\n   * available in the pool.\n   * @param offset location to start reading bytes from\n   * @param length number of bytes to read\n   * @param output the object to write the output to. MUST be non null.\n   */\n  public boolean getBytesToBytesRef(int offset, int length, BytesRef output) {\n    if (offset < 0 || offset + length > byteUpto + byteOffset) {\n      return false;\n    }\n    int currentBuffer = offset >>> ByteBlockPool.BYTE_BLOCK_SHIFT;\n    int currentOffset = offset & ByteBlockPool.BYTE_BLOCK_MASK;\n    // If the requested bytes are split across pools, we have to make a new array of bytes\n    // to copy them into and return a ref to that.\n    if (currentOffset + length <= ByteBlockPool.BYTE_BLOCK_SIZE) {\n      output.bytes = pool.buffers[currentBuffer];\n      output.offset = currentOffset;\n      output.length = length;\n    } else {\n      byte[] bytes = new byte[length];\n      int remainingLength = length;\n      while (remainingLength > ByteBlockPool.BYTE_BLOCK_SIZE - currentOffset) {\n        int lengthToCopy = ByteBlockPool.BYTE_BLOCK_SIZE - currentOffset;\n        System.arraycopy(pool.buffers[currentBuffer], currentOffset, bytes,\n                         length - remainingLength, lengthToCopy);\n        remainingLength -= lengthToCopy;\n        currentBuffer++;\n        currentOffset = 0;\n      }\n      System.arraycopy(pool.buffers[currentBuffer], currentOffset, bytes, length - remainingLength,\n                       remainingLength);\n      output.bytes = bytes;\n      output.length = bytes.length;\n      output.offset = 0;\n    }\n    return true;\n\n  }\n\n  /**\n   * Returns the read bytes, or null if offset is invalid or there aren't these many bytes\n   * available in the pool.\n   * @param offset location to start reading bytes from\n   * @param length number of bytes to read\n   */\n  public BytesRef getBytes(int offset, int length) {\n    BytesRef result = new BytesRef();\n    if (getBytesToBytesRef(offset, length, result)) {\n      return result;\n    } else {\n      return null;\n    }\n  }\n\n  /**\n   * get a new readStream at a given offset for this pool.\n   *\n   * Notice that individual ReadStreams are not threadsafe, but you can get as many ReadStreams as\n   * you want.\n   */\n  public ReadStream getReadStream(int offset) {\n    return new ReadStream(offset);\n  }\n\n  /**\n   * get the (one and only) WriteStream for this pool.\n   *\n   * Notice that there is exactly one WriteStream per pool, and it is not threadsafe.\n   */\n  public WriteStream getWriteStream() {\n    return writeStream;\n  }\n\n  /**\n   * A DataOutput-like interface for writing \"contiguous\" data to a ByteBlockPool.\n   *\n   * This is not threadsafe.\n   */\n  public final class WriteStream extends DataOutput {\n    private WriteStream() { }\n\n    /**\n     * Returns the start offset of the next data that will be added to the pool, UNLESS the data is\n     * added using addBytes and avoidSplitting = true\n     */\n    public int getOffset() {\n      return BaseByteBlockPool.this.getOffset();\n    }\n\n    /**\n     * Write bytes to the pool.\n     * @param bytes  source array\n     * @param offset  offset in bytes of the data to write\n     * @param length  number of bytes to put\n     * @param avoidSplitting  same as {link ByteBlockPool.addBytes}\n     * @return  the start offset of the bytes in the pool\n     */\n    public int writeBytes(byte[] bytes, int offset, int length, boolean avoidSplitting) {\n      return addBytes(bytes, offset, length, avoidSplitting);\n    }\n\n    @Override\n    public void writeBytes(byte[] b, int offset, int length) throws IOException {\n      addBytes(b, offset, length);\n    }\n\n    @Override\n    public void writeByte(byte b) {\n      addByte(b);\n    }\n  }\n\n  /**\n   * A DataInput-like interface for reading \"contiguous\" data from a ByteBlockPool.\n   *\n   * This is not threadsafe.\n   *\n   * This does not fully implement the DataInput interface - its DataInput.readBytes method throws\n   * UnsupportedOperationException because this class provides a facility for no-copy reading.\n   */\n  public final class ReadStream extends DataInput {\n    private int offset;\n\n    private ReadStream(int offset) {\n      this.offset = offset;\n    }\n\n    public BytesRef readBytes(int n) {\n      return readBytes(n, false);\n    }\n\n    /**\n     * read n bytes that were written with a given value of avoidSplitting\n     * @param n  number of bytes to read.\n     * @param avoidSplitting  this should be the same that was used at writeBytes time.\n     * @return  a reference to the bytes read or null.\n     */\n    public BytesRef readBytes(int n, boolean avoidSplitting) {\n      int currentBuffer = offset >>> ByteBlockPool.BYTE_BLOCK_SHIFT;\n      int currentOffset = offset & ByteBlockPool.BYTE_BLOCK_MASK;\n      if (avoidSplitting && n < ByteBlockPool.BYTE_BLOCK_SIZE\n          && currentOffset + n > ByteBlockPool.BYTE_BLOCK_SIZE) {\n        ++currentBuffer;\n        currentOffset = 0;\n        offset = currentBuffer << ByteBlockPool.BYTE_BLOCK_SHIFT;\n      }\n      BytesRef result = getBytes(offset, n);\n      this.offset += n;\n      return result;\n    }\n\n    @Override\n    public byte readByte() {\n      return getByte(offset++);\n    }\n\n    @Override\n    public void readBytes(byte[] b, int off, int len) throws IOException {\n      throw new UnsupportedOperationException(\"Use the no-copies version of ReadBytes instead.\");\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/ByteBlockPool.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\n\npublic class ByteBlockPool extends BaseByteBlockPool implements Flushable {\n\n  public ByteBlockPool() {\n  }\n\n  /**\n   * Used for loading flushed pool.\n   */\n  private ByteBlockPool(Pool pool, int bufferUpto, int byteUpTo, int byteOffset) {\n    super(pool, bufferUpto, byteUpTo, byteOffset);\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static class FlushHandler extends Flushable.Handler<ByteBlockPool> {\n    private static final String BUFFER_UP_TO_PROP_NAME = \"bufferUpto\";\n    private static final String BYTE_UP_TO_PROP_NAME = \"byteUpto\";\n    private static final String BYTE_OFFSET_PROP_NAME = \"byteOffset\";\n\n    public FlushHandler(ByteBlockPool objectToFlush) {\n      super(objectToFlush);\n    }\n\n    public FlushHandler() {\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      ByteBlockPool objectToFlush = getObjectToFlush();\n      out.writeByteArray2D(objectToFlush.pool.buffers, objectToFlush.bufferUpto + 1);\n      flushInfo.addIntProperty(BUFFER_UP_TO_PROP_NAME, objectToFlush.bufferUpto);\n      flushInfo.addIntProperty(BYTE_UP_TO_PROP_NAME, objectToFlush.byteUpto);\n      flushInfo.addIntProperty(BYTE_OFFSET_PROP_NAME, objectToFlush.byteOffset);\n    }\n\n    @Override\n    protected ByteBlockPool doLoad(FlushInfo flushInfo,\n                                   DataDeserializer in) throws IOException {\n      return new ByteBlockPool(\n              new BaseByteBlockPool.Pool(in.readByteArray2D()),\n              flushInfo.getIntProperty(BUFFER_UP_TO_PROP_NAME),\n              flushInfo.getIntProperty(BYTE_UP_TO_PROP_NAME),\n              flushInfo.getIntProperty(BYTE_OFFSET_PROP_NAME));\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/ByteTermUtils.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport org.apache.lucene.util.ByteBlockPool;\nimport org.apache.lucene.util.BytesRef;\nimport org.apache.lucene.util.StringHelper;\n\n/**\n * Utility class for BytePools which have each term's length encoded before the contents in the\n * ByteBlockPool\n * Another solution is to have a class that encapsulates both textStarts and the byteBlockPool and\n * knows how the byteBlockPool is used to store the strings\n **/\npublic abstract class ByteTermUtils {\n  /**\n   * Fill in a BytesRef from term's length & bytes encoded in byte block\n   */\n  public static int setBytesRef(final BaseByteBlockPool byteBlockPool,\n                                BytesRef term,\n                                final int textStart) {\n    final byte[] block = term.bytes =\n            byteBlockPool.pool.buffers[textStart >>> ByteBlockPool.BYTE_BLOCK_SHIFT];\n    final int start = textStart & ByteBlockPool.BYTE_BLOCK_MASK;\n    int pos = start;\n\n    byte b = block[pos++];\n    term.length = b & 0x7F;\n    for (int shift = 7; (b & 0x80) != 0; shift += 7) {\n      b = block[pos++];\n      term.length |= (b & 0x7F) << shift;\n    }\n    term.offset = pos;\n\n    assert term.length >= 0;\n    return textStart + (pos - start) + term.length;\n  }\n\n   /**\n    * Test whether the text for current RawPostingList p equals\n    * current tokenText in utf8.\n    */\n   public static boolean postingEquals(final BaseByteBlockPool termPool,\n       final int textStart, final BytesRef other) {\n     final byte[] block = termPool.pool.getBlocks()[textStart >>> ByteBlockPool.BYTE_BLOCK_SHIFT];\n     assert block != null;\n\n     int pos = textStart & ByteBlockPool.BYTE_BLOCK_MASK;\n\n     byte b = block[pos++];\n     int len = b & 0x7F;\n     for (int shift = 7; (b & 0x80) != 0; shift += 7) {\n       b = block[pos++];\n       len |= (b & 0x7F) << shift;\n     }\n\n     if (len == other.length) {\n       final byte[] utf8Bytes = other.bytes;\n       for (int tokenPos = other.offset;\n               tokenPos < other.length + other.offset; pos++, tokenPos++) {\n         if (utf8Bytes[tokenPos] != block[pos]) {\n           return false;\n         }\n       }\n       return true;\n     } else {\n       return false;\n     }\n   }\n\n   /**\n    * Returns the hashCode of the term stored at the given position in the block pool.\n    */\n   public static int hashCode(\n       final BaseByteBlockPool termPool, final int textStart) {\n    final byte[] block = termPool.pool.getBlocks()[textStart >>> ByteBlockPool.BYTE_BLOCK_SHIFT];\n    final int start = textStart & ByteBlockPool.BYTE_BLOCK_MASK;\n\n    int pos = start;\n\n    byte b = block[pos++];\n    int len = b & 0x7F;\n    for (int shift = 7; (b & 0x80) != 0; shift += 7) {\n      b = block[pos++];\n      len |= (b & 0x7F) << shift;\n    }\n\n    // Hash code returned here must be consistent with the one used in TermHashTable.lookupItem, so\n    // use the fixed hash seed. See TermHashTable.lookupItem for explanation of fixed hash seed.\n    return StringHelper.murmurhash3_x86_32(block, pos, len, InvertedRealtimeIndex.FIXED_HASH_SEED);\n  }\n\n  /**\n   * Copies the utf8 encoded byte ref to the termPool.\n   * @param termPool\n   * @param utf8\n   * @return The text's start position in the termPool\n   */\n  public static int copyToTermPool(BaseByteBlockPool termPool, BytesRef bytes) {\n    // Maybe grow the termPool before we write.  Assume we need 5 bytes in\n    // the worst case to store the VInt.\n    if (bytes.length + 5 + termPool.byteUpto > ByteBlockPool.BYTE_BLOCK_SIZE) {\n      // Not enough room in current block\n      termPool.nextBuffer();\n    }\n\n    final int textStart = termPool.byteUpto + termPool.byteOffset;\n\n    writeVInt(termPool, bytes.length);\n    System.arraycopy(bytes.bytes, bytes.offset, termPool.buffer, termPool.byteUpto, bytes.length);\n    termPool.byteUpto += bytes.length;\n\n    return textStart;\n  }\n\n  private static void writeVInt(final BaseByteBlockPool termPool, final int v) {\n    int value = v;\n    final byte[] block = termPool.buffer;\n    int blockUpto = termPool.byteUpto;\n\n    while ((value & ~0x7F) != 0) {\n      block[blockUpto++] = (byte) ((value & 0x7f) | 0x80);\n      value >>>= 7;\n    }\n    block[blockUpto++] =  (byte) value;\n    termPool.byteUpto = blockUpto;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/DeletedDocs.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.util.Bits;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\nimport it.unimi.dsi.fastutil.ints.Int2IntOpenHashMap;\n\npublic abstract class DeletedDocs implements Flushable {\n  private static final Logger LOG = LoggerFactory.getLogger(DeletedDocs.class);\n\n  /**\n   * Deletes the given document.\n   */\n  public abstract boolean deleteDoc(int docID);\n\n  /**\n   * Returns a point-in-time view of the deleted docs. Calling {@link #deleteDoc(int)} afterwards\n   * will not alter this View.\n   */\n  public abstract View getView();\n\n  /**\n   * Number of deletions.\n   */\n  public abstract int numDeletions();\n\n  /**\n   * Returns a DeletedDocs instance that has the same deleted tweet IDs, but mapped to the doc IDs\n   * in the optimizedTweetIdMapper.\n   *\n   * @param originalTweetIdMapper The original DocIDToTweetIDMapper instance that was used to add\n   *                              doc IDs to this DeletedDocs instance.\n   * @param optimizedTweetIdMapper The new DocIDToTweetIDMapper instance.\n   * @return An DeletedDocs instance that has the same tweets deleted, but mapped to the doc IDs in\n   *         optimizedTweetIdMapper.\n   */\n  public abstract DeletedDocs optimize(\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException;\n\n  public abstract class View {\n    /**\n     * Returns true, if the given document was deleted.\n     */\n    public abstract boolean isDeleted(int docID);\n\n    /**\n     * Returns true, if there are any deleted documents in this View.\n     */\n    public abstract boolean hasDeletions();\n\n    /**\n     * Returns {@link Bits} where all deleted documents have their bit set to 0, and\n     * all non-deleted documents have their bits set to 1.\n     */\n    public abstract Bits getLiveDocs();\n  }\n\n  public static class Default extends DeletedDocs {\n    private static final int KEY_NOT_FOUND = -1;\n\n    private final int size;\n    private final Int2IntOpenHashMap deletes;\n\n    // Each delete is marked with a unique, consecutively-increasing sequence ID.\n    private int sequenceID = 0;\n\n    public Default(int size) {\n      this.size = size;\n      deletes = new Int2IntOpenHashMap(size);\n      deletes.defaultReturnValue(KEY_NOT_FOUND);\n    }\n\n    /**\n     * Returns false, if this call was a noop, i.e. if the document was already deleted.\n     */\n    @Override\n    public boolean deleteDoc(int docID) {\n      if (deletes.putIfAbsent(docID, sequenceID) == KEY_NOT_FOUND) {\n        sequenceID++;\n        return true;\n      }\n      return false;\n    }\n\n    private boolean isDeleted(int internalID, int readerSequenceID) {\n      int deletedSequenceId = deletes.get(internalID);\n      return (deletedSequenceId >= 0) && (deletedSequenceId < readerSequenceID);\n    }\n\n    private boolean hasDeletions(int readerSequenceID) {\n      return readerSequenceID > 0;\n    }\n\n    @Override\n    public int numDeletions() {\n      return sequenceID;\n    }\n\n    @Override\n    public View getView() {\n      return new View() {\n        private final int readerSequenceID = sequenceID;\n\n        // liveDocs bitset contains inverted (decreasing) docids.\n        public final Bits liveDocs = !hasDeletions() ? null : new Bits() {\n          @Override\n          public final boolean get(int docID) {\n            return !isDeleted(docID);\n          }\n\n          @Override\n          public final int length() {\n            return size;\n          }\n        };\n\n        @Override\n        public Bits getLiveDocs() {\n          return liveDocs;\n        }\n\n\n        // Operates on internal (increasing) docids.\n        @Override\n        public final boolean isDeleted(int internalID) {\n          return DeletedDocs.Default.this.isDeleted(internalID, readerSequenceID);\n        }\n\n        @Override\n        public final boolean hasDeletions() {\n          return DeletedDocs.Default.this.hasDeletions(readerSequenceID);\n        }\n      };\n    }\n\n    @Override\n    public DeletedDocs optimize(DocIDToTweetIDMapper originalTweetIdMapper,\n                                DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n      DeletedDocs optimizedDeletedDocs = new Default(size);\n      for (int deletedDocID : deletes.keySet()) {\n        long tweetID = originalTweetIdMapper.getTweetID(deletedDocID);\n        int optimizedDeletedDocID = optimizedTweetIdMapper.getDocID(tweetID);\n        optimizedDeletedDocs.deleteDoc(optimizedDeletedDocID);\n      }\n      return optimizedDeletedDocs;\n    }\n\n    @SuppressWarnings(\"unchecked\")\n    @Override\n    public Default.FlushHandler getFlushHandler() {\n      return new Default.FlushHandler(this, size);\n    }\n\n    public static final class FlushHandler extends Flushable.Handler<Default> {\n      private final int size;\n\n      public FlushHandler(Default objectToFlush, int size) {\n        super(objectToFlush);\n        this.size = size;\n      }\n\n      public FlushHandler(int size) {\n        this.size = size;\n      }\n\n      @Override\n      protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n        long startTime = getClock().nowMillis();\n\n        Int2IntOpenHashMap deletes = getObjectToFlush().deletes;\n        out.writeIntArray(deletes.keySet().toIntArray());\n\n        getFlushTimerStats().timerIncrement(getClock().nowMillis() - startTime);\n      }\n\n      @Override\n      protected Default doLoad(FlushInfo flushInfo, DataDeserializer in) throws IOException {\n        Default deletedDocs = new Default(size);\n        long startTime = getClock().nowMillis();\n\n        int[] deletedDocIDs = in.readIntArray();\n        for (int docID : deletedDocIDs) {\n          deletedDocs.deleteDoc(docID);\n        }\n\n        getLoadTimerStats().timerIncrement(getClock().nowMillis() - startTime);\n        return deletedDocs;\n      }\n    }\n  }\n\n  public static final DeletedDocs NO_DELETES = new DeletedDocs() {\n    @Override\n    public <T extends Flushable> Handler<T> getFlushHandler() {\n      return null;\n    }\n\n    @Override\n    public boolean deleteDoc(int docID) {\n      return false;\n    }\n\n    @Override\n    public DeletedDocs optimize(DocIDToTweetIDMapper originalTweetIdMapper,\n                                DocIDToTweetIDMapper optimizedTweetIdMapper) {\n      return this;\n    }\n\n    @Override\n    public int numDeletions() {\n      return 0;\n    }\n\n    @Override\n    public View getView() {\n      return new View() {\n        @Override\n        public boolean isDeleted(int docID) {\n          return false;\n        }\n\n        @Override\n        public boolean hasDeletions() {\n          return false;\n        }\n\n        @Override\n        public Bits getLiveDocs() {\n          return null;\n        }\n\n      };\n    }\n  };\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/EarlybirdCSFDocValuesProcessor.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.facet.FacetsConfig;\nimport org.apache.lucene.index.DocValuesType;\nimport org.apache.lucene.index.IndexableField;\n\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.core.earlybird.index.EarlybirdRealtimeIndexSegmentWriter;\nimport com.twitter.search.core.earlybird.index.column.AbstractColumnStrideMultiIntIndex;\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideFieldIndex;\nimport com.twitter.search.core.earlybird.index.column.DocValuesManager;\n\n/**\n * Handler for docvalues in the indexing chain.\n */\npublic class EarlybirdCSFDocValuesProcessor\n    implements EarlybirdRealtimeIndexSegmentWriter.StoredFieldsConsumer {\n\n  private final DocValuesManager docValuesManager;\n\n  public EarlybirdCSFDocValuesProcessor(DocValuesManager docValuesManager) {\n    this.docValuesManager = docValuesManager;\n  }\n\n  @Override\n  public void addField(int docID, IndexableField field) throws IOException {\n    final DocValuesType dvType = field.fieldType().docValuesType();\n    if (dvType != null) {\n\n      // ignore lucene facet fields for realtime index, we are handling it differently\n      if (field.name().startsWith(FacetsConfig.DEFAULT_INDEX_FIELD_NAME)) {\n        return;\n      }\n      if (!(field.fieldType() instanceof EarlybirdFieldType)) {\n        throw new RuntimeException(\n            \"fieldType must be an EarlybirdFieldType instance for field \" + field.name());\n      }\n      EarlybirdFieldType fieldType = (EarlybirdFieldType) field.fieldType();\n\n      if (dvType == DocValuesType.NUMERIC) {\n        if (!(field.numericValue() instanceof Long)) {\n          throw new IllegalArgumentException(\n              \"illegal type \" + field.numericValue().getClass()\n              + \": DocValues types must be Long\");\n        }\n\n        ColumnStrideFieldIndex csfIndex =\n            docValuesManager.addColumnStrideField(field.name(), fieldType);\n        if (fieldType.getCsfFixedLengthNumValuesPerDoc() > 1) {\n          throw new UnsupportedOperationException(\"unsupported multi numeric values\");\n        } else {\n          csfIndex.setValue(docID, field.numericValue().longValue());\n        }\n\n      } else if (dvType == DocValuesType.BINARY) {\n        ColumnStrideFieldIndex csfIndex =\n            docValuesManager.addColumnStrideField(field.name(), fieldType);\n        if (fieldType.getCsfFixedLengthNumValuesPerDoc() > 1) {\n          Preconditions.checkArgument(\n              csfIndex instanceof AbstractColumnStrideMultiIntIndex,\n              \"Unsupported multi-value binary CSF class: \" + csfIndex);\n          ((AbstractColumnStrideMultiIntIndex) csfIndex).updateDocValues(\n              field.binaryValue(), docID);\n        }\n      } else {\n        throw new UnsupportedOperationException(\"unsupported DocValues.Type: \" + dvType);\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/EarlybirdOptimizedPostingsEnum.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.util.BytesRef;\n\n/**\n * Extend {@link EarlybirdPostingsEnum} to add more functionalities for docs (and positions)\n * enumerator of {@link OptimizedPostingLists}.\n */\npublic abstract class EarlybirdOptimizedPostingsEnum extends EarlybirdPostingsEnum {\n  /** Current doc and its frequency. */\n  private int currentDocID = -1;\n  private int currentFreq = 0;\n\n  /**\n   * Next doc and its frequency.\n   * These values should be set at {@link #loadNextPosting()}.\n   */\n  protected int nextDocID;\n  protected int nextFreq;\n\n  /** Pointer to the enumerated posting list. */\n  protected final int postingListPointer;\n\n  /** Total number of postings in the enumerated posting list. */\n  protected final int numPostingsTotal;\n\n  /** Query cost tracker. */\n  protected final QueryCostTracker queryCostTracker;\n\n  /**\n   * Sole constructor.\n   *\n   * @param postingListPointer pointer to the posting list for which this enumerator is created\n   * @param numPostings number of postings in the posting list for which this enumerator is created\n   */\n  public EarlybirdOptimizedPostingsEnum(int postingListPointer, int numPostings) {\n    this.postingListPointer = postingListPointer;\n    this.numPostingsTotal = numPostings;\n\n    // Get the thread local query cost tracker.\n    this.queryCostTracker = QueryCostTracker.getTracker();\n  }\n\n  /**\n   * Set {@link #currentDocID} and {@link #currentFreq} and load next posting.\n   * This method will de-dup if duplicate doc IDs are stored.\n   *\n   * @return {@link #currentDocID}\n   * @see {@link #nextDoc()}\n   */\n  @Override\n  protected final int nextDocNoDel() throws IOException {\n    currentDocID = nextDocID;\n\n    // Return immediately if exhausted.\n    if (currentDocID == NO_MORE_DOCS) {\n      return NO_MORE_DOCS;\n    }\n\n    currentFreq = nextFreq;\n    loadNextPosting();\n\n    // In case duplicate doc ID is stored.\n    while (currentDocID == nextDocID) {\n      currentFreq += nextFreq;\n      loadNextPosting();\n    }\n\n    startCurrentDoc();\n    return currentDocID;\n  }\n\n  /**\n   * Called when {@link #nextDocNoDel()} advances to a new docID.\n   * Subclasses can do extra accounting as needed.\n   */\n  protected void startCurrentDoc() {\n    // No-op in this class.\n  }\n\n  /**\n   * Loads the next posting, setting the nextDocID and nextFreq.\n   *\n   * @see #nextDocNoDel()\n   */\n  protected abstract void loadNextPosting();\n\n  /**\n   * Subclass should implement {@link #skipTo(int)}.\n   *\n   * @see org.apache.lucene.search.DocIdSetIterator#advance(int)\n   */\n  @Override\n  public final int advance(int target) throws IOException {\n    // Skipping to NO_MORE_DOCS or beyond largest doc ID.\n    if (target == NO_MORE_DOCS || target > getLargestDocID()) {\n      currentDocID = nextDocID = NO_MORE_DOCS;\n      currentFreq = nextFreq = 0;\n      return NO_MORE_DOCS;\n    }\n\n    // Skip as close as possible.\n    skipTo(target);\n\n    // Calling nextDoc to reach the target, or go beyond it if target does not exist.\n    int doc;\n    do {\n      doc = nextDoc();\n    } while (doc < target);\n\n    return doc;\n  }\n\n  /**\n   * Used in {@link #advance(int)}.\n   * This method should skip to the given target as close as possible, but NOT reach the target.\n   *\n   * @see #advance(int)\n   */\n  protected abstract void skipTo(int target);\n\n  /**\n   * Return loaded {@link #currentFreq}.\n   *\n   * @see org.apache.lucene.index.PostingsEnum#freq()\n   * @see #nextDocNoDel()\n   */\n  @Override\n  public final int freq() throws IOException {\n    return currentFreq;\n  }\n\n  /**\n   * Return loaded {@link #currentDocID}.\n   *\n   * @see org.apache.lucene.index.PostingsEnum#docID() ()\n   * @see #nextDocNoDel()\n   */\n  @Override\n  public final int docID() {\n    return currentDocID;\n  }\n\n  /*********************************************\n   * Not Supported Information                 *\n   * @see org.apache.lucene.index.PostingsEnum *\n   *********************************************/\n\n  @Override\n  public int nextPosition() throws IOException {\n    return -1;\n  }\n\n  @Override\n  public int startOffset() throws IOException {\n    return -1;\n  }\n\n  @Override\n  public int endOffset() throws IOException {\n    return -1;\n  }\n\n  @Override\n  public BytesRef getPayload() throws IOException {\n    return null;\n  }\n\n  /*********************************\n   * Helper methods for subclasses *\n   *********************************/\n\n  protected int getCurrentFreq() {\n    return currentFreq;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/EarlybirdPostingsEnum.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.PostingsEnum;\n\n/**\n * Extension of Lucene's PostingsEnum interface that adds additional funcionality.\n */\npublic abstract class EarlybirdPostingsEnum extends PostingsEnum {\n  @Override\n  public final int nextDoc() throws IOException {\n    // SEARCH-7008\n    return nextDocNoDel();\n  }\n\n  /**\n   * Advances to the next doc without paying attention to liveDocs.\n   */\n  protected abstract int nextDocNoDel() throws IOException;\n\n  /**\n   * Returns the largest docID contained in this posting list.\n   */\n  public abstract int getLargestDocID() throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/FSTTermDictionary.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\nimport java.util.Comparator;\n\nimport org.apache.lucene.index.BaseTermsEnum;\nimport org.apache.lucene.index.ImpactsEnum;\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.index.SlowImpactsEnum;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.util.BytesRef;\nimport org.apache.lucene.util.InPlaceMergeSorter;\nimport org.apache.lucene.util.IntsRefBuilder;\nimport org.apache.lucene.util.fst.BytesRefFSTEnum;\nimport org.apache.lucene.util.fst.FST;\nimport org.apache.lucene.util.fst.PositiveIntOutputs;\nimport org.apache.lucene.util.fst.Util;\nimport org.apache.lucene.util.packed.PackedInts;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\npublic class FSTTermDictionary implements TermDictionary, Flushable {\n  private final FST<Long> fst;\n\n  private final PackedInts.Reader termPointers;\n  private final ByteBlockPool termPool;\n  private final TermPointerEncoding termPointerEncoding;\n  private int numTerms;\n\n  FSTTermDictionary(int numTerms, FST<Long> fst,\n                    ByteBlockPool termPool, PackedInts.Reader termPointers,\n                    TermPointerEncoding termPointerEncoding) {\n    this.numTerms = numTerms;\n    this.fst = fst;\n    this.termPool = termPool;\n    this.termPointers = termPointers;\n    this.termPointerEncoding = termPointerEncoding;\n  }\n\n  @Override\n  public int getNumTerms() {\n    return numTerms;\n  }\n\n  @Override\n  public int lookupTerm(BytesRef term) throws IOException {\n    if (fst == null) {\n      return EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND;\n    }\n    final BytesRefFSTEnum<Long> fstEnum = new BytesRefFSTEnum<>(fst);\n\n    final BytesRefFSTEnum.InputOutput<Long> result = fstEnum.seekExact(term);\n    if (result != null && result.input.equals(term)) {\n      // -1 because 0 is not supported by the fst\n      return result.output.intValue() - 1;\n    } else {\n      return EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND;\n    }\n  }\n\n  static FSTTermDictionary buildFST(\n      final ByteBlockPool termPool,\n      int[] termPointers,\n      int numTerms,\n      final Comparator<BytesRef> comp,\n      boolean supportTermTextLookup,\n      final TermPointerEncoding termPointerEncoding) throws IOException {\n    final IntsRefBuilder scratchIntsRef = new IntsRefBuilder();\n\n    final int[] compact = new int[numTerms];\n    for (int i = 0; i < numTerms; i++) {\n      compact[i] = i;\n    }\n\n    // first sort the terms\n    new InPlaceMergeSorter() {\n      private BytesRef scratch1 = new BytesRef();\n      private BytesRef scratch2 = new BytesRef();\n\n      @Override\n      protected void swap(int i, int j) {\n        final int o = compact[i];\n        compact[i] = compact[j];\n        compact[j] = o;\n      }\n\n      @Override\n      protected int compare(int i, int j) {\n        final int ord1 = compact[i];\n        final int ord2 = compact[j];\n        ByteTermUtils.setBytesRef(termPool, scratch1,\n                                  termPointerEncoding.getTextStart(termPointers[ord1]));\n        ByteTermUtils.setBytesRef(termPool, scratch2,\n                                  termPointerEncoding.getTextStart(termPointers[ord2]));\n        return comp.compare(scratch1, scratch2);\n      }\n\n    }.sort(0, compact.length);\n\n    final PositiveIntOutputs outputs = PositiveIntOutputs.getSingleton();\n\n    final org.apache.lucene.util.fst.Builder<Long> builder =\n        new org.apache.lucene.util.fst.Builder<>(FST.INPUT_TYPE.BYTE1, outputs);\n\n    final BytesRef term = new BytesRef();\n    for (int termID : compact) {\n      ByteTermUtils.setBytesRef(termPool, term,\n              termPointerEncoding.getTextStart(termPointers[termID]));\n      // +1 because 0 is not supported by the fst\n      builder.add(Util.toIntsRef(term, scratchIntsRef), (long) termID + 1);\n    }\n\n    if (supportTermTextLookup) {\n      PackedInts.Reader packedTermPointers = OptimizedMemoryIndex.getPackedInts(termPointers);\n      return new FSTTermDictionary(\n          numTerms,\n          builder.finish(),\n          termPool,\n          packedTermPointers,\n          termPointerEncoding);\n    } else {\n      return new FSTTermDictionary(\n          numTerms,\n          builder.finish(),\n          null, // termPool\n          null, // termPointers\n          termPointerEncoding);\n    }\n  }\n\n  @Override\n  public boolean getTerm(int termID, BytesRef text, BytesRef termPayload) {\n    if (termPool == null) {\n      throw new UnsupportedOperationException(\n              \"This dictionary does not support term lookup by termID\");\n    } else {\n      int termPointer = (int) termPointers.get(termID);\n      boolean hasTermPayload = termPointerEncoding.hasPayload(termPointer);\n      int textStart = termPointerEncoding.getTextStart(termPointer);\n      // setBytesRef sets the passed in BytesRef \"text\" to the term in the termPool.\n      // As a side effect it returns the offset of the next entry in the pool after the term,\n      // which may optionally be used if this term has a payload.\n      int termPayloadStart = ByteTermUtils.setBytesRef(termPool, text, textStart);\n      if (termPayload != null && hasTermPayload) {\n        ByteTermUtils.setBytesRef(termPool, termPayload, termPayloadStart);\n      }\n\n      return hasTermPayload;\n    }\n  }\n\n  @Override\n  public TermsEnum createTermsEnum(OptimizedMemoryIndex index) {\n    return new BaseTermsEnum() {\n      private final BytesRefFSTEnum<Long> fstEnum = fst != null ? new BytesRefFSTEnum<>(fst) : null;\n      private BytesRefFSTEnum.InputOutput<Long> current;\n\n      @Override\n      public SeekStatus seekCeil(BytesRef term)\n          throws IOException {\n        if (fstEnum == null) {\n          return SeekStatus.END;\n        }\n\n        current = fstEnum.seekCeil(term);\n        if (current != null && current.input.equals(term)) {\n          return SeekStatus.FOUND;\n        } else {\n          return SeekStatus.END;\n        }\n      }\n\n      @Override\n      public boolean seekExact(BytesRef text) throws IOException {\n        current = fstEnum.seekExact(text);\n        return current != null;\n      }\n\n      // In our case the ord is the termId.\n      @Override\n      public void seekExact(long ord) {\n        current = new BytesRefFSTEnum.InputOutput<>();\n        current.input = null;\n        // +1 because 0 is not supported by the fst\n        current.output = ord + 1;\n\n        if (termPool != null) {\n          BytesRef bytesRef = new BytesRef();\n          int termId = (int) ord;\n          assert termId == ord;\n          FSTTermDictionary.this.getTerm(termId, bytesRef, null);\n          current.input = bytesRef;\n        }\n      }\n\n      @Override\n      public BytesRef next() throws IOException {\n        current = fstEnum.next();\n        if (current == null) {\n          return null;\n        }\n        return current.input;\n      }\n\n      @Override\n      public BytesRef term() {\n        return current.input;\n      }\n\n      // In our case the ord is the termId.\n      @Override\n      public long ord() {\n        // -1 because 0 is not supported by the fst\n        return current.output - 1;\n      }\n\n      @Override\n      public int docFreq() {\n        return index.getDF((int) ord());\n      }\n\n      @Override\n      public long totalTermFreq() {\n        return docFreq();\n      }\n\n      @Override\n      public PostingsEnum postings(PostingsEnum reuse, int flags) throws IOException {\n        int termID = (int) ord();\n        int postingsPointer = index.getPostingListPointer(termID);\n        int numPostings = index.getNumPostings(termID);\n        return index.getPostingLists().postings(postingsPointer, numPostings, flags);\n      }\n\n      @Override\n      public ImpactsEnum impacts(int flags) throws IOException {\n        return new SlowImpactsEnum(postings(null, flags));\n      }\n    };\n  }\n\n  @SuppressWarnings(\"unchecked\")\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static class FlushHandler extends Flushable.Handler<FSTTermDictionary> {\n    private static final String NUM_TERMS_PROP_NAME = \"numTerms\";\n    private static final String SUPPORT_TERM_TEXT_LOOKUP_PROP_NAME = \"supportTermTextLookup\";\n    private final TermPointerEncoding termPointerEncoding;\n\n    public FlushHandler(TermPointerEncoding termPointerEncoding) {\n      super();\n      this.termPointerEncoding = termPointerEncoding;\n    }\n\n    public FlushHandler(FSTTermDictionary objectToFlush) {\n      super(objectToFlush);\n      this.termPointerEncoding = objectToFlush.termPointerEncoding;\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out)\n        throws IOException {\n      FSTTermDictionary objectToFlush = getObjectToFlush();\n      flushInfo.addIntProperty(NUM_TERMS_PROP_NAME, objectToFlush.getNumTerms());\n      flushInfo.addBooleanProperty(SUPPORT_TERM_TEXT_LOOKUP_PROP_NAME,\n              objectToFlush.termPool != null);\n      if (objectToFlush.termPool != null) {\n        out.writePackedInts(objectToFlush.termPointers);\n        objectToFlush.termPool.getFlushHandler().flush(flushInfo.newSubProperties(\"termPool\"), out);\n      }\n      objectToFlush.fst.save(out.getIndexOutput());\n    }\n\n    @Override\n    protected FSTTermDictionary doLoad(FlushInfo flushInfo,\n        DataDeserializer in) throws IOException {\n      int numTerms = flushInfo.getIntProperty(NUM_TERMS_PROP_NAME);\n      boolean supportTermTextLookup =\n              flushInfo.getBooleanProperty(SUPPORT_TERM_TEXT_LOOKUP_PROP_NAME);\n      PackedInts.Reader termPointers = null;\n      ByteBlockPool termPool = null;\n      if (supportTermTextLookup) {\n        termPointers = in.readPackedInts();\n        termPool = (new ByteBlockPool.FlushHandler())\n                .load(flushInfo.getSubProperties(\"termPool\"), in);\n      }\n      final PositiveIntOutputs outputs = PositiveIntOutputs.getSingleton();\n      return new FSTTermDictionary(numTerms, new FST<>(in.getIndexInput(), outputs),\n              termPool, termPointers, termPointerEncoding);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/HighDFPackedIntsDocsAndPositionsEnum.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\n/**\n * Docs, frequencies, and positions enumerator for {@link HighDFPackedIntsPostingLists}.\n */\npublic class HighDFPackedIntsDocsAndPositionsEnum extends HighDFPackedIntsDocsEnum {\n  /**\n   * Pre-computed shifts, masks, and start int indices for {@link #positionListsReader}.\n   * These pre-computed values should be read-only and shared across all reader threads.\n   *\n   * Notice:\n   * - start int indices are NEEDED since there IS jumping within a slice in\n   *   {@link #doAdditionalSkip()} and {@link #startCurrentDoc()}.\n   */\n  private static final PackedLongsReaderPreComputedValues PRE_COMPUTED_VALUES =\n      new PackedLongsReaderPreComputedValues(\n          HighDFPackedIntsPostingLists.MAX_POSITION_BIT,\n          HighDFPackedIntsPostingLists.POSITION_SLICE_NUM_BITS_WITHOUT_HEADER,\n          HighDFPackedIntsPostingLists.POSITION_SLICE_SIZE_WITHOUT_HEADER,\n          true);\n\n  /**\n   * Int block pool holding the positions for the read posting list. This is mainly used while\n   * reading slice headers in {@link #loadNextPositionSlice()}.\n   */\n  private final IntBlockPool positionLists;\n\n  /** Packed ints reader for positions. */\n  private final IntBlockPoolPackedLongsReader positionListsReader;\n\n  /** Total number of positions in the current position slice. */\n  private int numPositionsInSliceTotal;\n\n  /**\n   * Number of remaining positions for {@link #currentDocID}; this value is decremented every time\n   * {@link #nextPosition()} is called.\n   */\n  private int numPositionsRemainingForCurrentDocID;\n\n  /**\n   * Pointer to the first int, which contains the position slice header, of the next position slice.\n   * This value is used to track which slice will be loaded when {@link #loadNextPositionSlice()} is\n   * called.\n   */\n  private int nextPositionSlicePointer;\n\n  /**\n   * Create a docs and positions enumerator.\n   */\n  public HighDFPackedIntsDocsAndPositionsEnum(\n      IntBlockPool skipLists,\n      IntBlockPool deltaFreqLists,\n      IntBlockPool positionLists,\n      int postingListPointer,\n      int numPostings,\n      boolean omitPositions) {\n    super(skipLists, deltaFreqLists, postingListPointer, numPostings, omitPositions);\n\n    this.positionLists = positionLists;\n    this.positionListsReader = new IntBlockPoolPackedLongsReader(\n        positionLists,\n        PRE_COMPUTED_VALUES,\n        queryCostTracker,\n        QueryCostTracker.CostType.LOAD_OPTIMIZED_POSTING_BLOCK);\n\n    // Load the first position slice.\n    this.nextPositionSlicePointer = skipListReader.getPositionCurrentSlicePointer();\n    loadNextPositionSlice();\n  }\n\n  /**\n   * Prepare for current doc:\n   * - skipping over unread positions for the current doc.\n   * - reset remaining positions for current doc to {@link #currentFreq}.\n   *\n   * @see #nextDocNoDel()\n   */\n  @Override\n  protected void startCurrentDoc() {\n    // Locate next position for current doc by skipping over unread positions from the previous doc.\n    if (numPositionsRemainingForCurrentDocID != 0) {\n      int numPositionsRemainingInSlice =\n          numPositionsInSliceTotal - positionListsReader.getPackedValueIndex();\n      while (numPositionsRemainingInSlice <= numPositionsRemainingForCurrentDocID) {\n        numPositionsRemainingForCurrentDocID -= numPositionsRemainingInSlice;\n        nextPositionSlicePointer += HighDFPackedIntsPostingLists.SLICE_SIZE;\n        loadNextPositionSlice();\n        numPositionsRemainingInSlice = numPositionsInSliceTotal;\n      }\n\n      positionListsReader.setPackedValueIndex(\n          positionListsReader.getPackedValueIndex() + numPositionsRemainingForCurrentDocID);\n    }\n\n    // Number of remaining positions for current doc is current freq.\n    numPositionsRemainingForCurrentDocID = getCurrentFreq();\n  }\n\n  /**\n   * Put positions reader to the start of next position slice and reset number of bits per packed\n   * value for next position slice.\n   */\n  private void loadNextPositionSlice() {\n    final int header = positionLists.get(nextPositionSlicePointer);\n    final int bitsForPosition = HighDFPackedIntsPostingLists.getNumBitsForPosition(header);\n    numPositionsInSliceTotal = HighDFPackedIntsPostingLists.getNumPositionsInSlice(header);\n\n    positionListsReader.jumpToInt(\n        nextPositionSlicePointer + HighDFPackedIntsPostingLists.POSITION_SLICE_HEADER_SIZE,\n        bitsForPosition);\n  }\n\n  /**\n   * Return next position for current doc.\n   * @see org.apache.lucene.index.PostingsEnum#nextPosition()\n   */\n  @Override\n  public int nextPosition() throws IOException {\n    // Return -1 immediately if all positions are used up for current doc.\n    if (numPositionsRemainingForCurrentDocID == 0) {\n      return -1;\n    }\n\n    if (positionListsReader.getPackedValueIndex() < numPositionsInSliceTotal)  {\n      // Read next position in current slice.\n      final int nextPosition = (int) positionListsReader.readPackedLong();\n      numPositionsRemainingForCurrentDocID--;\n      return nextPosition;\n    } else {\n      // All positions in current slice is used up, load next slice.\n      nextPositionSlicePointer += HighDFPackedIntsPostingLists.SLICE_SIZE;\n      loadNextPositionSlice();\n      return nextPosition();\n    }\n  }\n\n  /**\n   * Set {@link #positionListsReader} to the correct location and correct number of bits per packed\n   * value for the delta-freq slice on which this enum is landed after skipping.\n   *\n   * @see #skipTo(int)\n   */\n  @Override\n  protected void doAdditionalSkip() {\n    nextPositionSlicePointer = skipListReader.getPositionCurrentSlicePointer();\n    loadNextPositionSlice();\n\n    // Locate the exact position in slice.\n    final int skipListEntryEncodedMetadata = skipListReader.getEncodedMetadataCurrentSlice();\n    positionListsReader.setPackedValueIndex(\n        HighDFPackedIntsPostingLists.getPositionOffsetInSlice(skipListEntryEncodedMetadata));\n    numPositionsRemainingForCurrentDocID = 0;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/HighDFPackedIntsDocsEnum.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\n/**\n * Docs and frequencies enumerator for {@link HighDFPackedIntsPostingLists}.\n */\npublic class HighDFPackedIntsDocsEnum extends EarlybirdOptimizedPostingsEnum {\n  /**\n   * Pre-computed shifts, masks for {@link #deltaFreqListsReader}.\n   * These pre-computed values should be read-only and shared across all reader threads.\n   *\n   * Notice:\n   * - start int indices are NOT needed since there is not jumping within a slice.\n   */\n  private static final PackedLongsReaderPreComputedValues PRE_COMPUTED_VALUES =\n      new PackedLongsReaderPreComputedValues(\n          HighDFPackedIntsPostingLists.MAX_DOC_ID_BIT\n              + HighDFPackedIntsPostingLists.MAX_FREQ_BIT,\n          HighDFPackedIntsPostingLists.NUM_BITS_PER_SLICE,\n          HighDFPackedIntsPostingLists.SLICE_SIZE,\n          false);\n\n  /** Packed ints reader for delta-freq pairs. */\n  private final IntBlockPoolPackedLongsReader deltaFreqListsReader;\n\n  /** Skip list reader. */\n  protected final HighDFPackedIntsSkipListReader skipListReader;\n\n  /** Number of remaining docs (delta-freq pairs) in a slice. */\n  private int numDocsRemaining;\n\n  /**\n   * Total number of docs (delta-freq pairs) in a slice.\n   * This value is set every time a slice is loaded in {@link #loadNextDeltaFreqSlice()}.\n   */\n  private int numDocsInSliceTotal;\n\n  /**\n   * Number of bits used for frequency in a delta-freq slice.\n   * This value is set every time a slice is loaded in {@link #loadNextDeltaFreqSlice()}.\n   */\n  private int bitsForFreq;\n\n  /**\n   * Frequency mask used to extract frequency from a delta-freq pair, in a delta-freq slice.\n   * This value is set every time a slice is loaded in {@link #loadNextDeltaFreqSlice()}.\n   */\n  private int freqMask;\n  private boolean freqBitsIsZero;\n\n  /**\n   * Sole constructor.\n   *\n   * @param skipLists skip lists int block pool\n   * @param deltaFreqLists delta-freq lists int block pool\n   * @param postingListPointer pointer to the posting list for which this enumerator is created\n   * @param numPostings number of postings in the posting list for which this enumerator is created\n   * @param omitPositions whether positions are omitted in the posting list of which this enumerator\n   *                      is created\n   */\n  public HighDFPackedIntsDocsEnum(\n      IntBlockPool skipLists,\n      IntBlockPool deltaFreqLists,\n      int postingListPointer,\n      int numPostings,\n      boolean omitPositions) {\n    super(postingListPointer, numPostings);\n\n    // Create skip list reader and get first skip entry.\n    this.skipListReader = new HighDFPackedIntsSkipListReader(\n        skipLists, postingListPointer, omitPositions);\n    this.skipListReader.getNextSkipEntry();\n\n    // Set number of remaining docs in this posting list.\n    this.numDocsRemaining = skipListReader.getNumDocsTotal();\n\n    // Create a delta-freq pair packed values reader.\n    this.deltaFreqListsReader = new IntBlockPoolPackedLongsReader(\n        deltaFreqLists,\n        PRE_COMPUTED_VALUES,\n        queryCostTracker,\n        QueryCostTracker.CostType.LOAD_OPTIMIZED_POSTING_BLOCK);\n\n    loadNextDeltaFreqSlice();\n    loadNextPosting();\n  }\n\n  /**\n   * Load next delta-freq slice, return false if all docs exhausted.\n   * Notice!! The caller of this method should make sure the current slice is all used up and\n   * {@link #numDocsRemaining} is updated accordingly.\n   *\n   * @return whether a slice is loaded.\n   * @see #loadNextPosting()\n   * @see #skipTo(int)\n   */\n  private boolean loadNextDeltaFreqSlice() {\n    // Load nothing if no docs are remaining.\n    if (numDocsRemaining == 0) {\n      return false;\n    }\n\n    final int encodedMetadata = skipListReader.getEncodedMetadataCurrentSlice();\n    final int bitsForDelta = HighDFPackedIntsPostingLists.getNumBitsForDelta(encodedMetadata);\n    bitsForFreq = HighDFPackedIntsPostingLists.getNumBitsForFreq(encodedMetadata);\n    numDocsInSliceTotal = HighDFPackedIntsPostingLists.getNumDocsInSlice(encodedMetadata);\n\n    freqMask = (1 << bitsForFreq) - 1;\n    freqBitsIsZero = bitsForFreq == 0;\n\n    // Locate and reset the reader for this slice.\n    final int bitsPerPackedValue = bitsForDelta + bitsForFreq;\n    deltaFreqListsReader.jumpToInt(\n        skipListReader.getDeltaFreqCurrentSlicePointer(), bitsPerPackedValue);\n    return true;\n  }\n\n  /**\n   * Load next delta-freq pair from the current slice and set the computed\n   * {@link #nextDocID} and {@link #nextFreq}.\n   */\n  @Override\n  protected final void loadNextPosting() {\n    assert numDocsRemaining >= (numDocsInSliceTotal - deltaFreqListsReader.getPackedValueIndex())\n        : \"numDocsRemaining should be equal to or greater than number of docs remaining in slice\";\n\n    if (deltaFreqListsReader.getPackedValueIndex() < numDocsInSliceTotal) {\n      // Current slice is not exhausted.\n      final long nextDeltaFreqPair = deltaFreqListsReader.readPackedLong();\n\n      /**\n       * Optimization: No need to do shifts and masks if number of bits for frequency is 0.\n       * Also, the stored frequency is the actual frequency - 1.\n       * @see\n       * HighDFPackedIntsPostingLists#copyPostingList(org.apache.lucene.index.PostingsEnum, int)\n       */\n      if (freqBitsIsZero) {\n        nextFreq = 1;\n        nextDocID += (int) nextDeltaFreqPair;\n      } else {\n        nextFreq = (int) ((nextDeltaFreqPair & freqMask) + 1);\n        nextDocID += (int) (nextDeltaFreqPair >>> bitsForFreq);\n      }\n\n      numDocsRemaining--;\n    } else {\n      // Current slice is exhausted, get next skip entry and load next slice.\n      skipListReader.getNextSkipEntry();\n      if (loadNextDeltaFreqSlice()) {\n        // Next slice is loaded, load next posting again.\n        loadNextPosting();\n      } else {\n        // All docs are exhausted, mark this enumerator as exhausted.\n        assert numDocsRemaining == 0;\n        nextDocID = NO_MORE_DOCS;\n        nextFreq = 0;\n      }\n    }\n  }\n\n  /**\n   * Skip over slices to approach the given target as close as possible.\n   */\n  @Override\n  protected final void skipTo(int target) {\n    assert target != NO_MORE_DOCS : \"Should be handled in parent class advance method\";\n\n    int numSlicesToSkip = 0;\n    int numDocsToSkip = 0;\n    int numDocsRemainingInSlice = numDocsInSliceTotal - deltaFreqListsReader.getPackedValueIndex();\n\n    // Skipping over slices.\n    while (skipListReader.peekPreviousDocIDNextSlice() < target) {\n      skipListReader.getNextSkipEntry();\n      nextDocID = skipListReader.getPreviousDocIDCurrentSlice();\n      numDocsToSkip += numDocsRemainingInSlice;\n      int header = skipListReader.getEncodedMetadataCurrentSlice();\n      numDocsRemainingInSlice = HighDFPackedIntsPostingLists.getNumDocsInSlice(header);\n\n      numSlicesToSkip++;\n    }\n\n    // If skipped any slices, load the new slice.\n    if (numSlicesToSkip > 0) {\n      numDocsRemaining -= numDocsToSkip;\n      final boolean hasNextSlice = loadNextDeltaFreqSlice();\n      assert hasNextSlice;\n      assert numDocsRemaining >= numDocsInSliceTotal && numDocsInSliceTotal > 0;\n\n      // Do additional skip for the delta freq slice that was just loaded.\n      doAdditionalSkip();\n\n      loadNextPosting();\n    }\n  }\n\n  /**\n   * Subclass should override this method if want to do additional skip on its data structure.\n   */\n  protected void doAdditionalSkip() {\n    // No-op in this class.\n  }\n\n  /**\n   * Get the largest doc ID from {@link #skipListReader}.\n   */\n  @Override\n  public int getLargestDocID() throws IOException {\n    return skipListReader.getLargestDocID();\n  }\n\n  /**\n   * Return {@link #numDocsRemaining} as a proxy of cost.\n   *\n   * @see org.apache.lucene.index.PostingsEnum#cost()\n   */\n  @Override\n  public long cost() {\n    return numDocsRemaining;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/HighDFPackedIntsPostingLists.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport javax.annotation.Nullable;\n\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.search.DocIdSetIterator;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\n\n/**\n * An optimized posting lists implementation storing doc deltas, doc freqs, and positions as packed\n * ints in a 64 ints slice backed by {@link IntBlockPool}.\n *\n * There are three inner data structures used to store values used by a posting lists instance:\n *\n * - Skip lists, used for fast {@link PostingsEnum#advance(int)}, are stored in {@link #skipLists}\n *   int block pool.\n * - Doc deltas and freqs are stored in {@link #deltaFreqLists} int block pool.\n * - Positions are stored in {@link #positionLists} int block pool.\n *\n * For detail layout and configuration, please refer to the Javadoc of {@link #skipLists},\n * {@link #deltaFreqLists} and {@link #positionLists}.\n *\n * <b>This implementation designed for posting lists with a LARGE number of postings.</b>\n *\n * <i>Acknowledgement</i>: the concepts of slice based packed ints encoding/decoding is borrowed\n *                         from {@code HighDFCompressedPostinglists}, which will be deprecated due\n *                         to not supporting positions that are greater than 255.\n */\npublic class HighDFPackedIntsPostingLists extends OptimizedPostingLists {\n  /**\n   * A counter used to track when positions enum is required and a posting lists instance is set\n   * to omit positions.\n   *\n   * @see #postings(int, int, int)\n   */\n  private static final SearchCounter GETTING_POSITIONS_WITH_OMIT_POSITIONS =\n      SearchCounter.export(\n          \"high_df_packed_ints_posting_list_getting_positions_with_omit_positions\");\n\n  /**\n   * Information related to size of a slice.\n   */\n  static final int SLICE_SIZE_BIT = 6;\n  static final int SLICE_SIZE = 1 << SLICE_SIZE_BIT;                 //   64 ints per block\n  static final int NUM_BITS_PER_SLICE = SLICE_SIZE * Integer.SIZE;   // 2048 bits per block\n\n  /**\n   * A skip list has ONE skip list header that contains 5 ints (4 ints if positions are omitted):\n   * - 1st int: number of skip entries in this skip list.\n   * - 2nd int: largest doc ID in this posting list.\n   * - 3rd int: number of docs in this posting list.\n   * - 4th int: pointer to the start of the delta-freq list of this posting list.\n   * - 5th int: (OPTIONAL) pointer to the start of the position list of this posting list.\n   */\n  static final int SKIPLIST_HEADER_SIZE = 5;\n  static final int SKIPLIST_HEADER_SIZE_WITHOUT_POSITIONS = SKIPLIST_HEADER_SIZE - 1;\n\n  /**\n   * A skip list has MANY skip entries. Each skip entry is for one slice in delta-freq list.\n   * There are 3 ints in every skip entry (2 ints if positions are omitted):\n   * - 1st int: last doc ID in previous slice (0 for the first slice), this is mainly used during\n   *            skipping because deltas, not absolute doc IDs, are stored in a slice.\n   * - 2nd int: encoded metadata of the corresponding delta-freq slice. There are 4 piece of\n   *            information from the LOWEST bits to HIGHEST bits of this int:\n   *            11 bits: number of docs (delta-freq pairs) in this slice.\n   *             5 bits: number of bits used to encode each freq.\n   *             5 bits: number of bits used to encode each delta.\n   *            11 bits: POSITION SLICE OFFSET: an index of number of positions; this is where the\n   *                     first position of the first doc (in this delta-freq slice) is in the\n   *                     position slice. The position slice is identified by the 3rd int below.\n   *                     These two piece information uniquely identified the location of the start\n   *                     position of this delta-freq slice. This value is always 0 if position is\n   *                     omitted.\n   * - 3rd int: (OPTIONAL) POSITION SLICE INDEX: an index of of number of slices; this value\n   *            identifies the slice in which the first position of the first doc (in this\n   *            delta-freq slice) exists. The exact location inside the position slice is identified\n   *            by POSITION SLICE OFFSET that is stored in the 2nd int above.\n   *            Notice: this is not the absolute address in the block pool, but instead a relative\n   *            offset (in number of slices) on top of this term's first position slice.\n   *            This value DOES NOT EXIST if position is omitted.\n   */\n  static final int SKIPLIST_ENTRY_SIZE = 3;\n  static final int SKIPLIST_ENTRY_SIZE_WITHOUT_POSITIONS = SKIPLIST_ENTRY_SIZE - 1;\n\n  /**\n   * Shifts and masks used to encode/decode metadata from the 2nd int of a skip list entry.\n   * @see #SKIPLIST_ENTRY_SIZE\n   * @see #encodeSkipListEntryMetadata(int, int, int, int)\n   * @see #getNumBitsForDelta(int)\n   * @see #getNumBitsForFreq(int)\n   * @see #getNumDocsInSlice(int)\n   * @see #getPositionOffsetInSlice(int)\n   */\n  static final int SKIPLIST_ENTRY_POSITION_OFFSET_SHIFT = 21;\n  static final int SKIPLIST_ENTRY_NUM_BITS_DELTA_SHIFT = 16;\n  static final int SKIPLIST_ENTRY_NUM_BITS_FREQ_SHIFT = 11;\n  static final int SKIPLIST_ENTRY_POSITION_OFFSET_MASK = (1 << 11) - 1;\n  static final int SKIPLIST_ENTRY_NUM_BITS_DELTA_MASK = (1 << 5) - 1;\n  static final int SKIPLIST_ENTRY_NUM_BITS_FREQ_MASK = (1 << 5) - 1;\n  static final int SKIPLIST_ENTRY_NUM_DOCS_MASK = (1 << 11) - 1;\n\n  /**\n   * Each position slice has a header that is the 1st int in this position slice. From LOWEST bits\n   * to HIGHEST bits, there are 2 pieces of information encoded in this single int:\n   * 11 bits: number of positions in this slice.\n   *  5 bits: number of bits used to encode each position.\n   */\n  static final int POSITION_SLICE_HEADER_SIZE = 1;\n\n  /**\n   * Information related to size of a position slice. The actual size is the same as\n   * {@link #SLICE_SIZE}, but there is 1 int used for position slice header.\n   */\n  static final int POSITION_SLICE_SIZE_WITHOUT_HEADER = SLICE_SIZE - POSITION_SLICE_HEADER_SIZE;\n  static final int POSITION_SLICE_NUM_BITS_WITHOUT_HEADER =\n      POSITION_SLICE_SIZE_WITHOUT_HEADER * Integer.SIZE;\n\n  /**\n   * Shifts and masks used to encode/decode metadata from the position slice header.\n   * @see #POSITION_SLICE_HEADER_SIZE\n   * @see #encodePositionEntryHeader(int, int)\n   * @see #getNumPositionsInSlice(int)\n   * @see #getNumBitsForPosition(int)\n   */\n  static final int POSITION_SLICE_HEADER_BITS_POSITION_SHIFT = 11;\n  static final int POSITION_SLICE_HEADER_BITS_POSITION_MASK = (1 << 5) - 1;\n  static final int POSITION_SLICE_HEADER_NUM_POSITIONS_MASK = (1 << 11) - 1;\n\n  /**\n   * Stores skip list for each posting list.\n   *\n   * A skip list consists of ONE skip list header and MANY skip list entries, and each skip entry\n   * corresponds to one delta-freq slice. Also, unlike {@link #deltaFreqLists} and\n   * {@link #positionLists}, values in skip lists int pool are NOT stored in unit of slices.\n   *\n   * Example:\n   * H: skip list header int\n   * E: skip list entry int\n   * ': int boundary\n   * |: header/entry boundary (also a boundary of int)\n   *\n   *  <----- skip list A -----> <- skip list B ->\n   * |H'H'H'H'H|E'E|E'E|E'E|E'E|H'H'H'H'H|E'E|E'E|\n   */\n  private final IntBlockPool skipLists;\n\n  /**\n   * Stores delta-freq list for each posting list.\n   *\n   * A delta-freq list consists of MANY 64-int slices, and delta-freq pairs are stored compactly\n   * with a fixed number of bits within a single slice. Each slice has a corresponding skip list\n   * entry in {@link #skipLists} storing metadata about this slice.\n   *\n   * Example:\n   * |: slice boundary\n   *\n   *  <----------------- delta-freq list A -----------------> <--- delta-freq list B --->\n   * |64 ints slice|64 ints slice|64 ints slice|64 ints slice|64 ints slice|64 ints slice|\n   */\n  private final IntBlockPool deltaFreqLists;\n\n  /**\n   * Stores position list for each posting list.\n   *\n   * A position list consists of MANY 64 ints slices, and positions are stored compactly with a\n   * fixed number of bits within a single slice. The first int in each slice is used as a header to\n   * store the metadata about this position slice.\n   *\n   * Example:\n   * H: position header int\n   * ': int boundary\n   * |: slice boundary\n   *\n   *  <--------------- position list A ---------------> <---------- position list B ---------->\n   * |H'63 ints|H'63 ints|H'63 ints|H'63 ints|H'63 ints|H'63 ints|H'63 ints|H'63 ints|H'63 ints|\n   */\n  private final IntBlockPool positionLists;\n\n  /**\n   * Whether positions are omitted in this optimized posting lists.\n   */\n  private final boolean omitPositions;\n\n  /**\n   * Skip list header and entry size for this posting lists, could be different depends on whether\n   * position is omitted or not.\n   *\n   * @see #SKIPLIST_HEADER_SIZE\n   * @see #SKIPLIST_HEADER_SIZE_WITHOUT_POSITIONS\n   * @see #SKIPLIST_ENTRY_SIZE\n   * @see #SKIPLIST_ENTRY_SIZE_WITHOUT_POSITIONS\n   */\n  private final int skipListHeaderSize;\n  private final int skiplistEntrySize;\n\n  /**\n   * Buffer used in {@link #copyPostingList(PostingsEnum, int)}\n   * to queue up values needed for a slice.\n   * Loaded posting lists have them set as null.\n   */\n  private final PostingsBufferQueue docFreqQueue;\n  private final PostingsBufferQueue positionQueue;\n\n  /**\n   * Packed ints writer used to write into delta-freq int pool and position int pool.\n   * Loaded posting lists have them set as null.\n   */\n  private final IntBlockPoolPackedLongsWriter deltaFreqListsWriter;\n  private final IntBlockPoolPackedLongsWriter positionListsWriter;\n\n  /**\n   * Default constructor.\n   *\n   * @param omitPositions whether positions will be omitted in these posting lists.\n   */\n  public HighDFPackedIntsPostingLists(boolean omitPositions) {\n    this(\n        new IntBlockPool(\"high_df_packed_ints_skip_lists\"),\n        new IntBlockPool(\"high_df_packed_ints_delta_freq_lists\"),\n        new IntBlockPool(\"high_df_packed_ints_position_lists\"),\n        omitPositions,\n        new PostingsBufferQueue(NUM_BITS_PER_SLICE),\n        new PostingsBufferQueue(POSITION_SLICE_NUM_BITS_WITHOUT_HEADER));\n  }\n\n  /**\n   * Constructors used by loader.\n   *\n   * @param skipLists loaded int block pool represents skip lists\n   * @param deltaFreqLists loaded int block pool represents delta-freq lists\n   * @param positionLists loaded int block pool represents position lists\n   * @param omitPositions whether positions will be omitted in these posting lists\n   * @param docFreqQueue buffer used to queue up values used for a doc freq slice, null if loaded\n   * @param positionQueue buffer used to queue up values used for a position slice, null if loaded\n   * @see FlushHandler#doLoad(FlushInfo, DataDeserializer)\n   */\n  private HighDFPackedIntsPostingLists(\n      IntBlockPool skipLists,\n      IntBlockPool deltaFreqLists,\n      IntBlockPool positionLists,\n      boolean omitPositions,\n      @Nullable PostingsBufferQueue docFreqQueue,\n      @Nullable PostingsBufferQueue positionQueue) {\n    this.skipLists = skipLists;\n    this.deltaFreqLists = deltaFreqLists;\n    this.positionLists = positionLists;\n    this.omitPositions = omitPositions;\n\n    this.docFreqQueue = docFreqQueue;\n    this.positionQueue = positionQueue;\n\n    // docFreqQueue is null if this postingLists is loaded,\n    // we don't need to create writer at that case.\n    if (docFreqQueue == null) {\n      assert positionQueue == null;\n      this.deltaFreqListsWriter = null;\n      this.positionListsWriter = null;\n    } else {\n      this.deltaFreqListsWriter = new IntBlockPoolPackedLongsWriter(deltaFreqLists);\n      this.positionListsWriter = new IntBlockPoolPackedLongsWriter(positionLists);\n    }\n\n    if (omitPositions) {\n      skipListHeaderSize = SKIPLIST_HEADER_SIZE_WITHOUT_POSITIONS;\n      skiplistEntrySize = SKIPLIST_ENTRY_SIZE_WITHOUT_POSITIONS;\n    } else {\n      skipListHeaderSize = SKIPLIST_HEADER_SIZE;\n      skiplistEntrySize = SKIPLIST_ENTRY_SIZE;\n    }\n  }\n\n  /**\n   * A simple wrapper around assorted states used when coping positions in a posting enum.\n   * @see #copyPostingList(PostingsEnum, int)\n   */\n  private static class PositionsState {\n    /** Max position has been seen for the current position slice. */\n    private int maxPosition = 0;\n\n    /** Bits needed to encode/decode positions in the current position slice. */\n    private int bitsNeededForPosition = 0;\n\n    /** Total number of position slices created for current posting list. */\n    private int numPositionsSlices = 0;\n\n    /**\n     * Whenever a slice of doc/freq pairs is written, this will point to the first position\n     * associated with the first doc in the doc/freq slice.\n     */\n    private int currentPositionsSliceIndex = 0;\n    private int currentPositionsSliceOffset = 0;\n\n    /**\n     * Whenever a new document is processed, this points to the first position for this doc.\n     * This is used if this doc ends up being chosen as the first doc in a doc/freq slice.\n     */\n    private int nextPositionsSliceIndex = 0;\n    private int nextPositionsSliceOffset = 0;\n  }\n\n  /**\n   * Copies postings in the given postings enum into this posting lists instance.\n   *\n   * @param postingsEnum enumerator of the posting list that needs to be copied\n   * @param numPostings number of postings in the posting list that needs to be copied\n   * @return pointer to the copied posting list in this posting lists instance\n   */\n  @Override\n  public int copyPostingList(PostingsEnum postingsEnum, int numPostings) throws IOException {\n    assert docFreqQueue.isEmpty() : \"each new posting list should start with an empty queue\";\n    assert positionQueue.isEmpty() : \"each new posting list should start with an empty queue\";\n\n    final int skipListPointer = skipLists.length();\n    final int deltaFreqListPointer = deltaFreqLists.length();\n    final int positionListPointer = positionLists.length();\n    assert isSliceStart(deltaFreqListPointer) : \"each new posting list should start at a new slice\";\n    assert isSliceStart(positionListPointer) : \"each new posting list should start at a new slice\";\n\n    // Make room for skip list HEADER.\n    for (int i = 0; i < skipListHeaderSize; i++) {\n      skipLists.add(-1);\n    }\n\n    int doc;\n    int prevDoc = 0;\n    int prevWrittenDoc = 0;\n\n    int maxDelta = 0;\n    int maxFreq = 0;\n\n    int bitsNeededForDelta = 0;\n    int bitsNeededForFreq = 0;\n\n    // Keep tracking positions related info for this posting list.\n    PositionsState positionsState = new PositionsState();\n\n    int numDocs = 0;\n    int numDeltaFreqSlices = 0;\n    while ((doc = postingsEnum.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {\n      numDocs++;\n\n      int delta = doc - prevDoc;\n      assert delta <= MAX_DOC_ID;\n\n      int newBitsForDelta = bitsNeededForDelta;\n      if (delta > maxDelta) {\n        maxDelta = delta;\n        newBitsForDelta = log(maxDelta, 2);\n        assert newBitsForDelta <= MAX_DOC_ID_BIT;\n      }\n\n      /**\n       * Optimization: store freq - 1 since a freq must be positive. Save bits and improve decoding\n       * speed. At read side, the read frequency will plus 1.\n       * @see HighDFPackedIntsDocsEnum#loadNextPosting()\n       */\n      int freq = postingsEnum.freq() - 1;\n      assert freq >= 0;\n\n      int newBitsForFreq = bitsNeededForFreq;\n      if (freq > maxFreq) {\n        maxFreq = freq;\n        newBitsForFreq = log(maxFreq, 2);\n        assert newBitsForFreq <= MAX_FREQ_BIT;\n      }\n\n      // Write positions for this doc if not omit positions.\n      if (!omitPositions) {\n        writePositionsForDoc(postingsEnum, positionsState);\n      }\n\n      if ((newBitsForDelta + newBitsForFreq) * (docFreqQueue.size() + 1) > NUM_BITS_PER_SLICE) {\n        //The latest doc does not fit into this slice.\n        assert (bitsNeededForDelta + bitsNeededForFreq) * docFreqQueue.size()\n            <= NUM_BITS_PER_SLICE;\n\n        prevWrittenDoc = writeDeltaFreqSlice(\n            bitsNeededForDelta,\n            bitsNeededForFreq,\n            positionsState,\n            prevWrittenDoc);\n        numDeltaFreqSlices++;\n\n        maxDelta = delta;\n        maxFreq = freq;\n        bitsNeededForDelta = log(maxDelta, 2);\n        bitsNeededForFreq = log(maxFreq, 2);\n      } else {\n        bitsNeededForDelta = newBitsForDelta;\n        bitsNeededForFreq = newBitsForFreq;\n      }\n\n      docFreqQueue.offer(doc, freq);\n\n      prevDoc = doc;\n    }\n\n    // Some positions may be left in the buffer queue.\n    if (!positionQueue.isEmpty()) {\n      writePositionSlice(positionsState.bitsNeededForPosition);\n    }\n\n    // Some docs may be left in the buffer queue.\n    if (!docFreqQueue.isEmpty()) {\n      writeDeltaFreqSlice(\n          bitsNeededForDelta,\n          bitsNeededForFreq,\n          positionsState,\n          prevWrittenDoc);\n      numDeltaFreqSlices++;\n    }\n\n    // Write skip list header.\n    int skipListHeaderPointer = skipListPointer;\n    final int numSkipListEntries =\n        (skipLists.length() - (skipListPointer + skipListHeaderSize)) / skiplistEntrySize;\n    assert numSkipListEntries == numDeltaFreqSlices\n        : \"number of delta freq slices should be the same as number of skip list entries\";\n    skipLists.set(skipListHeaderPointer++, numSkipListEntries);\n    skipLists.set(skipListHeaderPointer++, prevDoc);\n    skipLists.set(skipListHeaderPointer++, numDocs);\n    skipLists.set(skipListHeaderPointer++, deltaFreqListPointer);\n    if (!omitPositions) {\n      skipLists.set(skipListHeaderPointer, positionListPointer);\n    }\n\n    return skipListPointer;\n  }\n\n  /**\n   * Write positions for current doc into {@link #positionLists}.\n   *\n   * @param postingsEnum postings enumerator containing the positions need to be written\n   * @param positionsState some states about {@link #positionLists} and {@link #positionQueue}\n   * @see #copyPostingList(PostingsEnum, int)\n   */\n  private void writePositionsForDoc(\n      PostingsEnum postingsEnum,\n      PositionsState positionsState) throws IOException {\n    assert !omitPositions : \"this method should not be called if positions are omitted\";\n\n    for (int i = 0; i < postingsEnum.freq(); i++) {\n      int pos = postingsEnum.nextPosition();\n\n      int newBitsForPosition = positionsState.bitsNeededForPosition;\n      if (pos > positionsState.maxPosition) {\n        positionsState.maxPosition = pos;\n        newBitsForPosition = log(positionsState.maxPosition, 2);\n        assert newBitsForPosition <= MAX_POSITION_BIT;\n      }\n\n      if (newBitsForPosition * (positionQueue.size() + 1)\n          > POSITION_SLICE_NUM_BITS_WITHOUT_HEADER\n          || positionQueue.isFull()) {\n        assert positionsState.bitsNeededForPosition * positionQueue.size()\n            <= POSITION_SLICE_NUM_BITS_WITHOUT_HEADER;\n\n        writePositionSlice(positionsState.bitsNeededForPosition);\n        positionsState.numPositionsSlices++;\n\n        positionsState.maxPosition = pos;\n        positionsState.bitsNeededForPosition = log(positionsState.maxPosition, 2);\n      } else {\n        positionsState.bitsNeededForPosition = newBitsForPosition;\n      }\n\n      // Update first position pointer if this position is the first position of a doc\n      if (i == 0) {\n        positionsState.nextPositionsSliceIndex = positionsState.numPositionsSlices;\n        positionsState.nextPositionsSliceOffset = positionQueue.size();\n      }\n\n      // Stores a dummy doc -1 since doc is unused in position list.\n      positionQueue.offer(-1, pos);\n    }\n  }\n\n  /**\n   * Write out all the buffered positions in {@link #positionQueue} into a position slice.\n   *\n   * @param bitsNeededForPosition number of bits used for each position in this position slice\n   */\n  private void writePositionSlice(final int bitsNeededForPosition) {\n    assert !omitPositions;\n    assert 0 <= bitsNeededForPosition && bitsNeededForPosition <= MAX_POSITION_BIT;\n\n    final int lengthBefore = positionLists.length();\n    assert isSliceStart(lengthBefore);\n\n    // First int in this slice stores number of bits needed for position\n    // and number of positions in this slice..\n    positionLists.add(encodePositionEntryHeader(bitsNeededForPosition, positionQueue.size()));\n\n    positionListsWriter.jumpToInt(positionLists.length(), bitsNeededForPosition);\n    while (!positionQueue.isEmpty()) {\n      int pos = PostingsBufferQueue.getSecondValue(positionQueue.poll());\n      assert log(pos, 2) <= bitsNeededForPosition;\n\n      positionListsWriter.writePackedInt(pos);\n    }\n\n    // Fill up this slice in case it is only partially filled.\n    while (positionLists.length() < lengthBefore + SLICE_SIZE) {\n      positionLists.add(0);\n    }\n\n    assert positionLists.length() - lengthBefore == SLICE_SIZE;\n  }\n\n  /**\n   * Write out all the buffered docs and frequencies in {@link #docFreqQueue} into a delta-freq\n   * slice and update the skip list entry of this slice.\n   *\n   * @param bitsNeededForDelta number of bits used for each delta in this delta-freq slice\n   * @param bitsNeededForFreq number of bits used for each freq in this delta-freq slice\n   * @param positionsState some states about {@link #positionLists} and {@link #positionQueue}\n   * @param prevWrittenDoc last doc written in previous slice\n   * @return last doc written in this slice\n   */\n  private int writeDeltaFreqSlice(\n      final int bitsNeededForDelta,\n      final int bitsNeededForFreq,\n      final PositionsState positionsState,\n      final int prevWrittenDoc) {\n    assert 0 <= bitsNeededForDelta && bitsNeededForDelta <= MAX_DOC_ID_BIT;\n    assert 0 <= bitsNeededForFreq && bitsNeededForFreq <= MAX_FREQ_BIT;\n\n    final int lengthBefore = deltaFreqLists.length();\n    assert isSliceStart(lengthBefore);\n\n    writeSkipListEntry(prevWrittenDoc, bitsNeededForDelta, bitsNeededForFreq, positionsState);\n\n    // Keep track of previous docID so that we compute the docID deltas.\n    int prevDoc = prevWrittenDoc;\n\n    // A <delta|freq> pair is stored as a packed value.\n    final int bitsPerPackedValue = bitsNeededForDelta + bitsNeededForFreq;\n    deltaFreqListsWriter.jumpToInt(deltaFreqLists.length(), bitsPerPackedValue);\n    while (!docFreqQueue.isEmpty()) {\n      long value = docFreqQueue.poll();\n      int doc = PostingsBufferQueue.getDocID(value);\n      int delta = doc - prevDoc;\n      assert log(delta, 2) <= bitsNeededForDelta;\n\n      int freq = PostingsBufferQueue.getSecondValue(value);\n      assert log(freq, 2) <= bitsNeededForFreq;\n\n      // Cast the delta to long before left shift to avoid overflow.\n      final long deltaFreqPair = (((long) delta) << bitsNeededForFreq) + freq;\n      deltaFreqListsWriter.writePackedLong(deltaFreqPair);\n      prevDoc = doc;\n    }\n\n    // Fill up this slice in case it is only partially filled.\n    while (deltaFreqLists.length() <  lengthBefore + SLICE_SIZE) {\n      deltaFreqLists.add(0);\n    }\n\n    positionsState.currentPositionsSliceIndex = positionsState.nextPositionsSliceIndex;\n    positionsState.currentPositionsSliceOffset = positionsState.nextPositionsSliceOffset;\n\n    assert deltaFreqLists.length() - lengthBefore == SLICE_SIZE;\n    return prevDoc;\n  }\n\n  /**\n   * Write the skip list entry for a delta-freq slice.\n   *\n   * @param prevWrittenDoc last doc written in previous slice\n   * @param bitsNeededForDelta number of bits used for each delta in this delta-freq slice\n   * @param bitsNeededForFreq number of bits used for each freq in this delta-freq slice\n   * @param positionsState some states about {@link #positionLists} and {@link #positionQueue}\n   * @see #writeDeltaFreqSlice(int, int, PositionsState, int)\n   * @see #SKIPLIST_ENTRY_SIZE\n   */\n  private void writeSkipListEntry(\n      int prevWrittenDoc,\n      int bitsNeededForDelta,\n      int bitsNeededForFreq,\n      PositionsState positionsState) {\n    // 1st int: last written doc ID in previous slice\n    skipLists.add(prevWrittenDoc);\n\n    // 2nd int: encoded metadata\n    skipLists.add(\n        encodeSkipListEntryMetadata(\n            positionsState.currentPositionsSliceOffset,\n            bitsNeededForDelta,\n            bitsNeededForFreq,\n            docFreqQueue.size()));\n\n    // 3rd int: optional, position slice index\n    if (!omitPositions) {\n      skipLists.add(positionsState.currentPositionsSliceIndex);\n    }\n  }\n\n  /**\n   * Create and return a docs enumerator or docs-positions enumerator based on input flag.\n   *\n   * @see org.apache.lucene.index.PostingsEnum\n   */\n  @Override\n  public EarlybirdPostingsEnum postings(\n      int postingListPointer, int numPostings, int flags) throws IOException {\n    // Positions are omitted but position enumerator are requried.\n    if (omitPositions && PostingsEnum.featureRequested(flags, PostingsEnum.POSITIONS)) {\n      GETTING_POSITIONS_WITH_OMIT_POSITIONS.increment();\n    }\n\n    if (!omitPositions && PostingsEnum.featureRequested(flags, PostingsEnum.POSITIONS)) {\n      return new HighDFPackedIntsDocsAndPositionsEnum(\n          skipLists,\n          deltaFreqLists,\n          positionLists,\n          postingListPointer,\n          numPostings,\n          false);\n    } else {\n      return new HighDFPackedIntsDocsEnum(\n          skipLists,\n          deltaFreqLists,\n          postingListPointer,\n          numPostings,\n          omitPositions);\n    }\n  }\n\n  /******************************************************\n   * Skip list entry encoded data encoding and decoding *\n   ******************************************************/\n\n  /**\n   * Encode a skip list entry metadata, which is stored in the 2nd int of the skip list entry.\n   *\n   * @see #SKIPLIST_ENTRY_SIZE\n   */\n  private static int encodeSkipListEntryMetadata(\n      int positionOffsetInSlice, int numBitsForDelta, int numBitsForFreq, int numDocsInSlice) {\n    assert 0 <= positionOffsetInSlice\n        && positionOffsetInSlice < POSITION_SLICE_NUM_BITS_WITHOUT_HEADER;\n    assert 0 <= numBitsForDelta && numBitsForDelta <= MAX_DOC_ID_BIT;\n    assert 0 <= numBitsForFreq && numBitsForFreq <= MAX_FREQ_BIT;\n    assert 0 < numDocsInSlice && numDocsInSlice <= NUM_BITS_PER_SLICE;\n    return (positionOffsetInSlice << SKIPLIST_ENTRY_POSITION_OFFSET_SHIFT)\n        + (numBitsForDelta << SKIPLIST_ENTRY_NUM_BITS_DELTA_SHIFT)\n        + (numBitsForFreq << SKIPLIST_ENTRY_NUM_BITS_FREQ_SHIFT)\n        // stores numDocsInSlice - 1 to avoid over flow since numDocsInSlice ranges in [1, 2048]\n        // and 11 bits are used to store number docs in slice\n        + (numDocsInSlice - 1);\n  }\n\n  /**\n   * Decode POSITION_SLICE_OFFSET of the delta-freq slice having the given skip entry encoded data.\n   *\n   * @see #SKIPLIST_ENTRY_SIZE\n   */\n  static int getPositionOffsetInSlice(int skipListEntryEncodedMetadata) {\n    return (skipListEntryEncodedMetadata >>> SKIPLIST_ENTRY_POSITION_OFFSET_SHIFT)\n        & SKIPLIST_ENTRY_POSITION_OFFSET_MASK;\n  }\n\n  /**\n   * Decode number of bits used for delta in the slice having the given skip entry encoded data.\n   *\n   * @see #SKIPLIST_ENTRY_SIZE\n   */\n  static int getNumBitsForDelta(int skipListEntryEncodedMetadata) {\n    return (skipListEntryEncodedMetadata >>> SKIPLIST_ENTRY_NUM_BITS_DELTA_SHIFT)\n        & SKIPLIST_ENTRY_NUM_BITS_DELTA_MASK;\n  }\n\n  /**\n   * Decode number of bits used for freqs in the slice having the given skip entry encoded data.\n   *\n   * @see #SKIPLIST_ENTRY_SIZE\n   */\n  static int getNumBitsForFreq(int skipListEntryEncodedMetadata) {\n    return (skipListEntryEncodedMetadata >>> SKIPLIST_ENTRY_NUM_BITS_FREQ_SHIFT)\n        & SKIPLIST_ENTRY_NUM_BITS_FREQ_MASK;\n  }\n\n  /**\n   * Decode number of delta-freq pairs stored in the slice having the given skip entry encoded data.\n   *\n   * @see #SKIPLIST_ENTRY_SIZE\n   */\n  static int getNumDocsInSlice(int skipListEntryEncodedMetadata) {\n    /**\n     * Add 1 to the decode value since the stored value is subtracted by 1.\n     * @see #encodeSkipListEntryMetadata(int, int, int, int)\n     */\n    return (skipListEntryEncodedMetadata & SKIPLIST_ENTRY_NUM_DOCS_MASK) + 1;\n  }\n\n  /*****************************************************\n   * Position slice entry header encoding and decoding *\n   *****************************************************/\n\n  /**\n   * Encode a position slice entry header.\n   *\n   * @param numBitsForPosition number of bits used to encode positions in this slice.\n   * @param numPositionsInSlice number of positions in this slice.\n   * @return an int as the encoded header.\n   * @see #POSITION_SLICE_HEADER_SIZE\n   */\n  private static int encodePositionEntryHeader(int numBitsForPosition, int numPositionsInSlice) {\n    assert 0 <= numBitsForPosition && numBitsForPosition <= MAX_POSITION_BIT;\n    assert 0 < numPositionsInSlice && numPositionsInSlice <= POSITION_SLICE_NUM_BITS_WITHOUT_HEADER;\n    return (numBitsForPosition << POSITION_SLICE_HEADER_BITS_POSITION_SHIFT) + numPositionsInSlice;\n  }\n\n  /**\n   * Decode number of bits used for position in the slice having the given header.\n   *\n   * @param positionEntryHeader entry header will be decoded.\n   * @see #POSITION_SLICE_HEADER_SIZE\n   */\n  static int getNumBitsForPosition(int positionEntryHeader) {\n    return (positionEntryHeader >>> POSITION_SLICE_HEADER_BITS_POSITION_SHIFT)\n        & POSITION_SLICE_HEADER_BITS_POSITION_MASK;\n  }\n\n  /**\n   * Decode number of positions stored in the slice having the given header.\n   *\n   * @param positionEntryHeader entry header will be decoded.\n   * @see #POSITION_SLICE_HEADER_SIZE\n   */\n  static int getNumPositionsInSlice(int positionEntryHeader) {\n    return positionEntryHeader & POSITION_SLICE_HEADER_NUM_POSITIONS_MASK;\n  }\n\n  /******************\n   * Helper methods *\n   ******************/\n\n  /**\n   * Check if given pointer is pointing to the slice start.\n   *\n   * @param pointer the index will be checked.\n   */\n  static boolean isSliceStart(int pointer) {\n    return pointer % HighDFPackedIntsPostingLists.SLICE_SIZE == 0;\n  }\n\n  /**\n   * Ceil of log of x in the given base.\n   *\n   * @return x == 0 ? 0 : Math.ceil(Math.log(x) / Math.log(base))\n   */\n  private static int log(int x, int base) {\n    assert base >= 2;\n    if (x == 0) {\n      return 0;\n    }\n    int ret = 1;\n    long n = base; // needs to be a long to avoid overflow\n    while (x >= n) {\n      n *= base;\n      ret++;\n    }\n    return ret;\n  }\n\n  /**********************\n   * For flush and load *\n   **********************/\n\n  @SuppressWarnings(\"unchecked\")\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static class FlushHandler extends Flushable.Handler<HighDFPackedIntsPostingLists> {\n    private static final String OMIT_POSITIONS_PROP_NAME = \"omitPositions\";\n    private static final String SKIP_LISTS_PROP_NAME = \"skipLists\";\n    private static final String DELTA_FREQ_LISTS_PROP_NAME = \"deltaFreqLists\";\n    private static final String POSITION_LISTS_PROP_NAME = \"positionLists\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(HighDFPackedIntsPostingLists objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out)\n        throws IOException {\n      HighDFPackedIntsPostingLists objectToFlush = getObjectToFlush();\n      flushInfo.addBooleanProperty(OMIT_POSITIONS_PROP_NAME, objectToFlush.omitPositions);\n      objectToFlush.skipLists.getFlushHandler()\n          .flush(flushInfo.newSubProperties(SKIP_LISTS_PROP_NAME), out);\n      objectToFlush.deltaFreqLists.getFlushHandler()\n          .flush(flushInfo.newSubProperties(DELTA_FREQ_LISTS_PROP_NAME), out);\n      objectToFlush.positionLists.getFlushHandler()\n          .flush(flushInfo.newSubProperties(POSITION_LISTS_PROP_NAME), out);\n    }\n\n    @Override\n    protected HighDFPackedIntsPostingLists doLoad(\n        FlushInfo flushInfo, DataDeserializer in) throws IOException {\n      IntBlockPool skipLists = (new IntBlockPool.FlushHandler())\n          .load(flushInfo.getSubProperties(SKIP_LISTS_PROP_NAME), in);\n      IntBlockPool deltaFreqLists = (new IntBlockPool.FlushHandler())\n          .load(flushInfo.getSubProperties(DELTA_FREQ_LISTS_PROP_NAME), in);\n      IntBlockPool positionLists = (new IntBlockPool.FlushHandler())\n          .load(flushInfo.getSubProperties(POSITION_LISTS_PROP_NAME), in);\n      return new HighDFPackedIntsPostingLists(\n          skipLists,\n          deltaFreqLists,\n          positionLists,\n          flushInfo.getBooleanProperty(OMIT_POSITIONS_PROP_NAME),\n          null,\n          null);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/HighDFPackedIntsSkipListReader.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport org.apache.lucene.search.DocIdSetIterator;\n\n/**\n * A skip list reader of a single term used {@link HighDFPackedIntsDocsEnum}.\n * @see HighDFPackedIntsPostingLists\n */\nclass HighDFPackedIntsSkipListReader {\n  /** Skip lists int pool. */\n  private final IntBlockPool skipLists;\n\n  /** Whether positions are omitted in the posting list having the read skip list. */\n  private final boolean omitPositions;\n\n  /**\n   * Last doc in the previous slice relative to the current delta-freq slice. This value is 0 if\n   * the current slice is the first delta-freq slice.\n   */\n  private int previousDocIDCurrentSlice;\n\n  /** Encoded metadata of the current delta-freq slice.*/\n  private int encodedMetadataCurrentSlice;\n\n  /**\n   * Pointer to the first int (contains the position slice header) of the position slice that has\n   * the first position of the first doc in the current delta-freq slice.\n   */\n  private int positionCurrentSliceIndex;\n\n  /** Pointer to the first int in the current delta-freq slice. */\n  private int deltaFreqCurrentSlicePointer;\n\n  /** Data of next slice. */\n  private int previousDocIDNextSlice;\n  private int encodedMetadataNextSlice;\n  private int positionNextSliceIndex;\n  private int deltaFreqNextSlicePointer;\n\n  /** Used to load blocks and read ints from skip lists int pool. */\n  private int[] currentSkipListBlock;\n  private int skipListBlockStart;\n  private int skipListBlockIndex;\n\n  /** Number of remaining skip entries for the read skip list. */\n  private int numSkipListEntriesRemaining;\n\n  /** Largest doc ID in the posting list having the read skip list. */\n  private final int largestDocID;\n\n  /** Pointer to the first int in the first slice that stores positions for this term. */\n  private final int positionListPointer;\n\n  /** Total number of docs in the posting list having the read skip list. */\n  private final int numDocsTotal;\n\n  /**\n   * Create a skip list reader specified by the given skip list pointer in the given skip lists int\n   * pool.\n   *\n   * @param skipLists int pool where the read skip list exists\n   * @param skipListPointer pointer to the read skip list\n   * @param omitPositions whether positions are omitted in the positing list to which the read skip\n   *                      list belongs\n   */\n  public HighDFPackedIntsSkipListReader(\n      final IntBlockPool skipLists,\n      final int skipListPointer,\n      final boolean omitPositions) {\n    this.skipLists = skipLists;\n    this.omitPositions = omitPositions;\n\n    this.skipListBlockStart = IntBlockPool.getBlockStart(skipListPointer);\n    this.skipListBlockIndex = IntBlockPool.getOffsetInBlock(skipListPointer);\n    this.currentSkipListBlock = skipLists.getBlock(skipListBlockStart);\n\n    // Read skip list header.\n    this.numSkipListEntriesRemaining = readNextValueFromSkipListBlock();\n    this.largestDocID = readNextValueFromSkipListBlock();\n    this.numDocsTotal = readNextValueFromSkipListBlock();\n    int deltaFreqListPointer = readNextValueFromSkipListBlock();\n    this.positionListPointer = omitPositions ? -1 : readNextValueFromSkipListBlock();\n\n    // Set it back by one slice for fetchNextSkipEntry() to advance correctly.\n    this.deltaFreqNextSlicePointer = deltaFreqListPointer - HighDFPackedIntsPostingLists.SLICE_SIZE;\n    fetchNextSkipEntry();\n  }\n\n  /**\n   * Load already fetched data in next skip entry into current data variables, and pre-fetch again.\n   */\n  public void getNextSkipEntry() {\n    previousDocIDCurrentSlice = previousDocIDNextSlice;\n    encodedMetadataCurrentSlice = encodedMetadataNextSlice;\n    positionCurrentSliceIndex = positionNextSliceIndex;\n    deltaFreqCurrentSlicePointer = deltaFreqNextSlicePointer;\n    fetchNextSkipEntry();\n  }\n\n  /**\n   * Fetch data for next skip entry if skip list is not exhausted; otherwise, set docIDNextSlice\n   * to NO_MORE_DOCS.\n   */\n  private void fetchNextSkipEntry() {\n    if (numSkipListEntriesRemaining == 0) {\n      previousDocIDNextSlice = DocIdSetIterator.NO_MORE_DOCS;\n      return;\n    }\n\n    previousDocIDNextSlice = readNextValueFromSkipListBlock();\n    encodedMetadataNextSlice = readNextValueFromSkipListBlock();\n    if (!omitPositions) {\n      positionNextSliceIndex = readNextValueFromSkipListBlock();\n    }\n    deltaFreqNextSlicePointer += HighDFPackedIntsPostingLists.SLICE_SIZE;\n    numSkipListEntriesRemaining--;\n  }\n\n  /**************************************\n   * Getters of data in skip list entry *\n   **************************************/\n\n  /**\n   * In the context of a current slice, this is the docID of the last document in the previous\n   * slice (or 0 if the current slice is the first slice).\n   *\n   * @see HighDFPackedIntsPostingLists#SKIPLIST_ENTRY_SIZE\n   */\n  public int getPreviousDocIDCurrentSlice() {\n    return previousDocIDCurrentSlice;\n  }\n\n  /**\n   * Get the encoded metadata of the current delta-freq slice.\n   *\n   * @see HighDFPackedIntsPostingLists#SKIPLIST_ENTRY_SIZE\n   */\n  public int getEncodedMetadataCurrentSlice() {\n    return encodedMetadataCurrentSlice;\n  }\n\n  /**\n   * Get the pointer to the first int, WHICH CONTAINS THE POSITION SLICE HEADER, of the position\n   * slice that contains the first position of the first doc in the delta-freq slice that\n   * is corresponding to the current skip list entry.\n   *\n   * @see HighDFPackedIntsPostingLists#SKIPLIST_ENTRY_SIZE\n   */\n  public int getPositionCurrentSlicePointer() {\n    assert !omitPositions;\n    return positionListPointer\n        + positionCurrentSliceIndex * HighDFPackedIntsPostingLists.SLICE_SIZE;\n  }\n\n  /**\n   * Get the pointer to the first int in the current delta-freq slice.\n   */\n  public int getDeltaFreqCurrentSlicePointer() {\n    return deltaFreqCurrentSlicePointer;\n  }\n\n  /**\n   * In the context of next slice, get the last doc ID in the previous slice. This is used to skip\n   * over slices.\n   *\n   * @see HighDFPackedIntsDocsEnum#skipTo(int)\n   */\n  public int peekPreviousDocIDNextSlice() {\n    return previousDocIDNextSlice;\n  }\n\n  /***************************************\n   * Getters of data in skip list header *\n   ***************************************/\n\n  public int getLargestDocID() {\n    return largestDocID;\n  }\n\n  public int getNumDocsTotal() {\n    return numDocsTotal;\n  }\n\n  /***************************************************\n   * Methods helping loading int block and read ints *\n   ***************************************************/\n\n  private int readNextValueFromSkipListBlock() {\n    if (skipListBlockIndex == IntBlockPool.BLOCK_SIZE) {\n      loadSkipListBlock();\n    }\n    return currentSkipListBlock[skipListBlockIndex++];\n  }\n\n  private void loadSkipListBlock() {\n    skipListBlockStart += IntBlockPool.BLOCK_SIZE;\n    currentSkipListBlock = skipLists.getBlock(skipListBlockStart);\n    skipListBlockIndex = 0;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/InMemoryFields.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.util.HashMap;\nimport java.util.Iterator;\nimport java.util.Map;\n\nimport org.apache.lucene.index.Fields;\nimport org.apache.lucene.index.Terms;\n\npublic class InMemoryFields extends Fields {\n  private final Map<InvertedIndex, Terms> termsCache = new HashMap<>();\n  private final Map<String, InvertedIndex> perFields;\n  private final Map<InvertedIndex, Integer> pointerIndex;\n\n  /**\n   * Returns a new {@link Fields} instance for the provided {@link InvertedIndex}es.\n   */\n  public InMemoryFields(Map<String, InvertedIndex> perFields,\n                        Map<InvertedIndex, Integer> pointerIndex) {\n    this.perFields = perFields;\n    this.pointerIndex = pointerIndex;\n  }\n\n  @Override\n  public Iterator<String> iterator() {\n    return perFields.keySet().iterator();\n  }\n\n  @Override\n  public Terms terms(String field) {\n    InvertedIndex invertedIndex = perFields.get(field);\n    if (invertedIndex == null) {\n      return null;\n    }\n\n    return termsCache.computeIfAbsent(invertedIndex,\n        index -> index.createTerms(pointerIndex.getOrDefault(invertedIndex, -1)));\n  }\n\n  @Override\n  public int size() {\n    return perFields.size();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/IndexOptimizer.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.facets.AbstractFacetCountingArray;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.facets.FacetUtil;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.EarlybirdRealtimeIndexSegmentData;\nimport com.twitter.search.core.earlybird.index.TimeMapper;\nimport com.twitter.search.core.earlybird.index.column.DocValuesManager;\n\npublic final class IndexOptimizer {\n  private static final Logger LOG = LoggerFactory.getLogger(IndexOptimizer.class);\n\n  private IndexOptimizer() {\n  }\n\n  /**\n   * Optimizes this in-memory index segment.\n   */\n  public static EarlybirdRealtimeIndexSegmentData optimize(\n      EarlybirdRealtimeIndexSegmentData source) throws IOException {\n    LOG.info(\"Starting index optimizing.\");\n\n    ConcurrentHashMap<String, InvertedIndex> targetMap = new ConcurrentHashMap<>();\n    LOG.info(String.format(\n        \"Source PerFieldMap size is %d\", source.getPerFieldMap().size()));\n\n    LOG.info(\"Optimize doc id mapper.\");\n    // Optimize the doc ID mapper first.\n    DocIDToTweetIDMapper originalTweetIdMapper = source.getDocIDToTweetIDMapper();\n    DocIDToTweetIDMapper optimizedTweetIdMapper = originalTweetIdMapper.optimize();\n\n    TimeMapper optimizedTimeMapper =\n        source.getTimeMapper() != null\n        ? source.getTimeMapper().optimize(originalTweetIdMapper, optimizedTweetIdMapper)\n        : null;\n\n    // Some fields have their terms rewritten to support the minimal perfect hash function we use\n    // (note that it's a minimal perfect hash function, not a minimal perfect hash _table_).\n    // The FacetCountingArray stores term IDs. This is a map from the facet field ID to a map from\n    // original term ID to the new, MPH term IDs.\n    Map<Integer, int[]> termIDMapper = new HashMap<>();\n\n    LOG.info(\"Optimize inverted indexes.\");\n    optimizeInvertedIndexes(\n        source, targetMap, originalTweetIdMapper, optimizedTweetIdMapper, termIDMapper);\n\n    LOG.info(\"Rewrite and map ids in facet counting array.\");\n    AbstractFacetCountingArray facetCountingArray = source.getFacetCountingArray().rewriteAndMapIDs(\n        termIDMapper, originalTweetIdMapper, optimizedTweetIdMapper);\n\n    Map<String, FacetLabelProvider> facetLabelProviders =\n        FacetUtil.getFacetLabelProviders(source.getSchema(), targetMap);\n\n    LOG.info(\"Optimize doc values manager.\");\n    DocValuesManager optimizedDocValuesManager =\n        source.getDocValuesManager().optimize(originalTweetIdMapper, optimizedTweetIdMapper);\n\n    LOG.info(\"Optimize deleted docs.\");\n    DeletedDocs optimizedDeletedDocs =\n        source.getDeletedDocs().optimize(originalTweetIdMapper, optimizedTweetIdMapper);\n\n    final boolean isOptimized = true;\n    return new EarlybirdRealtimeIndexSegmentData(\n        source.getMaxSegmentSize(),\n        source.getTimeSliceID(),\n        source.getSchema(),\n        isOptimized,\n        optimizedTweetIdMapper.getNextDocID(Integer.MIN_VALUE),\n        targetMap,\n        facetCountingArray,\n        optimizedDocValuesManager,\n        facetLabelProviders,\n        source.getFacetIDMap(),\n        optimizedDeletedDocs,\n        optimizedTweetIdMapper,\n        optimizedTimeMapper,\n        source.getIndexExtensionsData());\n  }\n\n  private static void optimizeInvertedIndexes(\n      EarlybirdRealtimeIndexSegmentData source,\n      ConcurrentHashMap<String, InvertedIndex> targetMap,\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper,\n      Map<Integer, int[]> termIDMapper\n  ) throws IOException {\n    for (Map.Entry<String, InvertedIndex> entry : source.getPerFieldMap().entrySet()) {\n      String fieldName = entry.getKey();\n      Preconditions.checkState(entry.getValue() instanceof InvertedRealtimeIndex);\n      InvertedRealtimeIndex sourceIndex = (InvertedRealtimeIndex) entry.getValue();\n      EarlybirdFieldType fieldType = source.getSchema().getFieldInfo(fieldName).getFieldType();\n\n      InvertedIndex newIndex;\n      if (fieldType.becomesImmutable() && sourceIndex.getNumTerms() > 0) {\n        Schema.FieldInfo facetField = source.getSchema().getFacetFieldByFieldName(fieldName);\n\n        newIndex = new OptimizedMemoryIndex(\n            fieldType,\n            fieldName,\n            sourceIndex,\n            termIDMapper,\n            source.getFacetIDMap().getFacetField(facetField),\n            originalTweetIdMapper,\n            optimizedTweetIdMapper);\n      } else {\n        newIndex = optimizeMutableIndex(\n            fieldType,\n            fieldName,\n            sourceIndex,\n            originalTweetIdMapper,\n            optimizedTweetIdMapper);\n      }\n\n      targetMap.put(fieldName, newIndex);\n    }\n  }\n\n  /**\n   * Optimize a mutable index.\n   */\n  private static InvertedIndex optimizeMutableIndex(\n      EarlybirdFieldType fieldType,\n      String fieldName,\n      InvertedRealtimeIndex originalIndex,\n      DocIDToTweetIDMapper originalMapper,\n      DocIDToTweetIDMapper optimizedMapper\n  ) throws IOException {\n    Preconditions.checkState(!fieldType.isStorePerPositionPayloads());\n    TermsEnum allTerms = originalIndex.createTermsEnum(originalIndex.getMaxPublishedPointer());\n\n    int numTerms = originalIndex.getNumTerms();\n\n    InvertedRealtimeIndex index = new InvertedRealtimeIndex(\n        fieldType,\n        TermPointerEncoding.DEFAULT_ENCODING,\n        fieldName);\n    index.setNumDocs(originalIndex.getNumDocs());\n\n    for (int termID = 0; termID < numTerms; termID++) {\n      allTerms.seekExact(termID);\n      PostingsEnum postingsEnum = new OptimizingPostingsEnumWrapper(\n          allTerms.postings(null), originalMapper, optimizedMapper);\n\n      BytesRef termPayload = originalIndex.getLabelAccessor().getTermPayload(termID);\n      copyPostingList(index, postingsEnum, termID, allTerms.term(), termPayload);\n    }\n    return index;\n  }\n\n\n  /**\n   * Copies the given posting list into these posting lists.\n   *\n   * @param postingsEnum enumerator of the posting list that needs to be copied\n   */\n  private static void copyPostingList(\n      InvertedRealtimeIndex index,\n      PostingsEnum postingsEnum,\n      int termID,\n      BytesRef term,\n      BytesRef termPayload\n  ) throws IOException {\n    int docId;\n    while ((docId = postingsEnum.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {\n      index.incrementSumTermDocFreq();\n      for (int i = 0; i < postingsEnum.freq(); i++) {\n        index.incrementSumTotalTermFreq();\n        int position = postingsEnum.nextPosition();\n        int newTermID = InvertedRealtimeIndexWriter.indexTerm(\n            index,\n            term,\n            docId,\n            position,\n            termPayload,\n            null, // We know that fields that remain mutable never have a posting payload.\n            TermPointerEncoding.DEFAULT_ENCODING);\n\n        // Our term lookups are very slow, so we cache term dictionaries for some fields across many\n        // segments, so we must keep the term IDs the same while remapping.\n        Preconditions.checkState(newTermID == termID);\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/IntBlockPool.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\nimport java.util.Arrays;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\n\n// Modeled after TwitterCharBlockPool, with a lot of simplification.\npublic class IntBlockPool implements Flushable {\n  private static final SearchLongGauge INT_BLOCK_POOL_MAX_LENGTH =\n      SearchLongGauge.export(\"twitter_int_block_pool_max_size\");\n  private static final String STAT_PREFIX = \"twitter_int_block_pool_size_\";\n\n  private static final int BLOCK_SHIFT = 14;\n  public static final int BLOCK_SIZE = 1 << BLOCK_SHIFT;\n  private static final int BLOCK_MASK = BLOCK_SIZE - 1;\n\n  // We can address up to 2^31 elements with an int. We use 1 << 14 bits for the block offset,\n  // so we can use the remaining 17 bits for the blocks index. Therefore the maximum number of\n  // addressable blocks is 1 << 17 or maxInt >> 14.\n  private static final int MAX_NUM_BLOCKS = Integer.MAX_VALUE >> BLOCK_SHIFT;\n\n  // Initial value written into the blocks.\n  private final int initialValue;\n\n  // Extra object with final array is necessary to guarantee visibility\n  // to other threads without synchronization / volatiles.  See comment\n  // in TwitterCharBlockPool.\n  public static final class Pool {\n    public final int[][] blocks;\n    Pool(int[][] blocks) {\n      this.blocks = blocks;\n\n      // Adjust max size if exceeded maximum value.\n      synchronized (INT_BLOCK_POOL_MAX_LENGTH) {\n        if (this.blocks != null) {\n          final long currentSize = (long) (this.blocks.length * BLOCK_SIZE);\n          if (currentSize > INT_BLOCK_POOL_MAX_LENGTH.get()) {\n            INT_BLOCK_POOL_MAX_LENGTH.set(currentSize);\n          }\n        }\n      }\n    }\n  }\n  public Pool pool;\n\n  private int currBlockIndex;   // Index into blocks array.\n  private int[] currBlock = null;\n  private int currBlockOffset;  // Index into current block.\n  private final String poolName;\n  private final SearchLongGauge sizeGauge;\n\n  public IntBlockPool(String poolName) {\n    this(0, poolName);\n  }\n\n  public IntBlockPool(int initialValue, String poolName) {\n    // Start with room for 16 initial blocks (does not allocate these blocks).\n    this.pool = new Pool(new int[16][]);\n    this.initialValue = initialValue;\n\n    // Start at the end of a previous, non-existent blocks.\n    this.currBlockIndex = -1;\n    this.currBlock = null;\n    this.currBlockOffset = BLOCK_SIZE;\n    this.poolName = poolName;\n    this.sizeGauge = createGauge(poolName, pool);\n  }\n\n  // Constructor for FlushHandler.\n  protected IntBlockPool(\n      int currBlockIndex,\n      int currBlockOffset,\n      int[][]blocks,\n      String poolName) {\n    this.initialValue = 0;\n    this.pool = new Pool(blocks);\n    this.currBlockIndex = currBlockIndex;\n    this.currBlockOffset = currBlockOffset;\n    if (currBlockIndex >= 0) {\n      this.currBlock = this.pool.blocks[currBlockIndex];\n    }\n    this.poolName = poolName;\n    this.sizeGauge = createGauge(poolName, pool);\n  }\n\n  private static SearchLongGauge createGauge(String suffix, Pool pool) {\n    SearchLongGauge gauge = SearchLongGauge.export(STAT_PREFIX + suffix);\n    if (pool.blocks != null) {\n      gauge.set(pool.blocks.length * BLOCK_SIZE);\n    }\n    return gauge;\n  }\n\n  /**\n   * Adds an int to the current block and returns it's overall index.\n   */\n  public int add(int value) {\n    if (currBlockOffset == BLOCK_SIZE) {\n      newBlock();\n    }\n    currBlock[currBlockOffset++] = value;\n    return (currBlockIndex << BLOCK_SHIFT) + currBlockOffset - 1;\n  }\n\n  // Returns number of ints in this blocks\n  public int length() {\n    return currBlockOffset + currBlockIndex * BLOCK_SIZE;\n  }\n\n  // Gets an int from the specified index.\n  public final int get(int index) {\n    return getBlock(index)[getOffsetInBlock(index)];\n  }\n\n  public static int getBlockStart(int index) {\n    return (index >>> BLOCK_SHIFT) * BLOCK_SIZE;\n  }\n\n  public static int getOffsetInBlock(int index) {\n    return index & BLOCK_MASK;\n  }\n\n  public final int[] getBlock(int index) {\n    final int blockIndex = index >>> BLOCK_SHIFT;\n    return pool.blocks[blockIndex];\n  }\n\n  // Sets an int value at the specified index.\n  public void set(int index, int value) {\n    final int blockIndex = index >>> BLOCK_SHIFT;\n    final int offset = index & BLOCK_MASK;\n    pool.blocks[blockIndex][offset] = value;\n  }\n\n  /**\n   * Evaluates whether two instances of IntBlockPool are equal by value. It is\n   * slow because it has to check every element in the pool.\n   */\n  @VisibleForTesting\n  public boolean verySlowEqualsForTests(IntBlockPool that) {\n    if (length() != that.length()) {\n      return false;\n    }\n\n    for (int i = 0; i < length(); i++) {\n      if (get(i) != that.get(i)) {\n        return false;\n      }\n    }\n\n    return true;\n  }\n\n  private void newBlock() {\n    final int newBlockIndex = 1 + currBlockIndex;\n    if (newBlockIndex >= MAX_NUM_BLOCKS) {\n      throw new RuntimeException(\n          \"Too many blocks, would overflow int index for blocks \" + poolName);\n    }\n    if (newBlockIndex == pool.blocks.length) {\n      // Blocks array is too small to add a new block.  Resize.\n      int[][] newBlocks = new int[pool.blocks.length * 2][];\n      System.arraycopy(pool.blocks, 0, newBlocks, 0, pool.blocks.length);\n      pool = new Pool(newBlocks);\n\n      sizeGauge.set(pool.blocks.length * BLOCK_SIZE);\n    }\n\n    currBlock = pool.blocks[newBlockIndex] = allocateBlock();\n    currBlockOffset = 0;\n    currBlockIndex = newBlockIndex;\n  }\n\n  private int[] allocateBlock() {\n    int[] block = new int[BLOCK_SIZE];\n    Arrays.fill(block, initialValue);\n    return block;\n  }\n\n  @SuppressWarnings(\"unchecked\")\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<IntBlockPool> {\n    private static final String CURRENT_BLOCK_INDEX_PROP_NAME = \"currentBlockIndex\";\n    private static final String CURRENT_BLOCK_OFFSET_PROP_NAME = \"currentBlockOffset\";\n    private static final String POOL_NAME = \"poolName\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(IntBlockPool objToFlush) {\n      super(objToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      IntBlockPool pool = getObjectToFlush();\n      flushInfo.addIntProperty(CURRENT_BLOCK_INDEX_PROP_NAME, pool.currBlockIndex);\n      flushInfo.addIntProperty(CURRENT_BLOCK_OFFSET_PROP_NAME, pool.currBlockOffset);\n      flushInfo.addStringProperty(POOL_NAME, pool.poolName);\n      out.writeIntArray2D(pool.pool.blocks, pool.currBlockIndex + 1);\n    }\n\n    @Override\n    protected IntBlockPool doLoad(FlushInfo flushInfo, DataDeserializer in) throws IOException {\n      String poolName = flushInfo.getStringProperty(POOL_NAME);\n      return new IntBlockPool(\n          flushInfo.getIntProperty(CURRENT_BLOCK_INDEX_PROP_NAME),\n          flushInfo.getIntProperty(CURRENT_BLOCK_OFFSET_PROP_NAME),\n          in.readIntArray2D(),\n          poolName);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/IntBlockPoolPackedLongsReader.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport javax.annotation.Nullable;\n\n/**\n * A packed ints reader reading packed values (int/long) written in {@link IntBlockPool}.\n * @see IntBlockPoolPackedLongsWriter\n *\n * A standard usage would be :\n * - set reader at an int block pool pointer and number of bits per packed value:\n *   {@link #jumpToInt(int, int)}}\n * - read: {@link #readPackedLong()}\n *\n * Example usage:\n * @see HighDFPackedIntsDocsEnum\n * @see HighDFPackedIntsDocsAndPositionsEnum\n */\npublic final class IntBlockPoolPackedLongsReader {\n  /**\n   * Mask used to convert an int to a long. We cannot just cast because it will fill in the higher\n   * 32 bits with the sign bit, but we need the higher 32 bits to be 0 instead.\n   */\n  private static final long LONG_MASK = 0xFFFFFFFFL;\n\n  /** The int block pool from which packed ints will be read. */\n  private final IntBlockPool intBlockPool;\n\n  /** Pre-computed shifts, masks, and start int indices used to decode packed ints. */\n  private final PackedLongsReaderPreComputedValues preComputedValues;\n\n  /**\n   * The underlying {@link #intBlockPool} will be read block by blocks. The current read\n   * block will be identified by {@link #startPointerForCurrentBlock} and assigned to\n   * {@link #currentBlock}. {@link #indexInCurrentBlock} will be used access values from the\n   * {@link #currentBlock}.\n   */\n  private int[] currentBlock;\n  private int indexInCurrentBlock;\n  private int startPointerForCurrentBlock = -1;\n\n  /**\n   * Whether the decoded packed values are spanning more than 1 int.\n   * @see #readPackedLong()\n   */\n  private boolean packedValueNeedsLong;\n\n  /**\n   * Masks used to extract packed values.\n   * @see #readPackedLong()\n   */\n  private long packedValueMask;\n\n  /** PRE-COMPUTED: The index of the first int that has a specific packed values. */\n  private int[] packedValueStartIndices;\n\n  /** PRE-COMPUTED: The shifts and masks used to decode packed values. */\n  private int[] packedValueLowBitsRightShift;\n  private int[] packedValueMiddleBitsLeftShift;\n  private int[] packedValueMiddleBitsMask;\n  private int[] packedValueHighBitsLeftShift;\n  private int[] packedValueHighBitsMask;\n\n  /** Index of packed values. */\n  private int packedValueIndex;\n\n  /**\n   * The {@link #indexInCurrentBlock} and {@link #startPointerForCurrentBlock} of the first int\n   * that holds packed values. This two values together uniquely form a int block pool pointer\n   * --- {@link #packedValueStartBlockStart} + {@link #packedValueStartBlockIndex} --- that points\n   * to the first int that has pointer.\n   *\n   * @see #jumpToInt(int, int)\n   */\n  private int packedValueStartBlockIndex;\n  private int packedValueStartBlockStart;\n\n  /** Current int read from {@link #currentBlock}. */\n  private int currentInt;\n\n  /**\n   * If given, query cost will be tracked every time a int block is loaded.\n   * @see #loadNextBlock()\n   */\n  private final QueryCostTracker queryCostTracker;\n  private final QueryCostTracker.CostType queryCostType;\n\n  /**\n   * Default constructor.\n   *\n   * @param intBlockPool from which packed ints will be read\n   * @param preComputedValues pre-computed shifts, masks, and start int\n   * @param queryCostTracker optional, query cost tracker used while loading a new block\n   * @param queryCostType optional, query cost type will be tracked while loading a new block\n   */\n  public IntBlockPoolPackedLongsReader(\n      IntBlockPool intBlockPool,\n      PackedLongsReaderPreComputedValues preComputedValues,\n      @Nullable QueryCostTracker queryCostTracker,\n      @Nullable QueryCostTracker.CostType queryCostType) {\n    this.intBlockPool = intBlockPool;\n    this.preComputedValues = preComputedValues;\n\n    // For query cost tracking.\n    this.queryCostTracker = queryCostTracker;\n    this.queryCostType = queryCostType;\n  }\n\n  /**\n   * Constructor with {@link #queryCostTracker} and {@link #queryCostType} set to null.\n   *\n   * @param intBlockPool from which packed ints will be read\n   * @param preComputedValues pre-computed shifts, masks, and start int\n   */\n  public IntBlockPoolPackedLongsReader(\n      IntBlockPool intBlockPool,\n      PackedLongsReaderPreComputedValues preComputedValues) {\n    this(intBlockPool, preComputedValues, null, null);\n  }\n\n  /**\n   * 1. Set the reader to starting reading at the given int block pool pointer. Correct block will\n   *    be loaded if the given pointer points to the different block than {@link #currentBlock}.\n   * 2. Update shifts, masks, and start int indices based on given number of bits per packed value.\n   * 3. Reset packed value sequence start data.\n   *\n   * @param intBlockPoolPointer points to the int from which this reader will start reading\n   * @param bitsPerPackedValue number of bits per packed value.\n   */\n  public void jumpToInt(int intBlockPoolPointer, int bitsPerPackedValue) {\n    assert  bitsPerPackedValue <= Long.SIZE;\n\n    // Update indexInCurrentBlock and load a different index if needed.\n    int newBlockStart = IntBlockPool.getBlockStart(intBlockPoolPointer);\n    indexInCurrentBlock = IntBlockPool.getOffsetInBlock(intBlockPoolPointer);\n\n    if (startPointerForCurrentBlock != newBlockStart) {\n      startPointerForCurrentBlock = newBlockStart;\n      loadNextBlock();\n    }\n\n    // Re-set shifts, masks, and start int indices for the given number bits per packed value.\n    packedValueNeedsLong = bitsPerPackedValue > Integer.SIZE;\n    packedValueMask =\n        bitsPerPackedValue == Long.SIZE ? 0xFFFFFFFFFFFFFFFFL : (1L << bitsPerPackedValue) - 1;\n    packedValueStartIndices = preComputedValues.getStartIntIndices(bitsPerPackedValue);\n    packedValueLowBitsRightShift = preComputedValues.getLowBitsRightShift(bitsPerPackedValue);\n    packedValueMiddleBitsLeftShift = preComputedValues.getMiddleBitsLeftShift(bitsPerPackedValue);\n    packedValueMiddleBitsMask = preComputedValues.getMiddleBitsMask(bitsPerPackedValue);\n    packedValueHighBitsLeftShift = preComputedValues.getHighBitsLeftShift(bitsPerPackedValue);\n    packedValueHighBitsMask = preComputedValues.getHighBitsMask(bitsPerPackedValue);\n\n    // Update packed values sequence start data.\n    packedValueIndex = 0;\n    packedValueStartBlockIndex = indexInCurrentBlock;\n    packedValueStartBlockStart = startPointerForCurrentBlock;\n\n    // Load an int to prepare for readPackedLong.\n    loadInt();\n  }\n\n  /**\n   * Read next packed value as a long.\n   *\n   * Caller could cast the returned long to an int if needed.\n   * NOTICE! Be careful of overflow while casting a long to an int.\n   *\n   * @return next packed value in a long.\n   */\n  public long readPackedLong() {\n    long packedValue;\n\n    if (packedValueNeedsLong) {\n      packedValue =\n          (LONG_MASK & currentInt)\n              >>> packedValueLowBitsRightShift[packedValueIndex] & packedValueMask;\n      packedValue |=\n          (LONG_MASK & loadInt()\n              & packedValueMiddleBitsMask[packedValueIndex])\n              << packedValueMiddleBitsLeftShift[packedValueIndex];\n      if (packedValueHighBitsLeftShift[packedValueIndex] != 0) {\n        packedValue |=\n            (LONG_MASK & loadInt()\n                & packedValueHighBitsMask[packedValueIndex])\n                << packedValueHighBitsLeftShift[packedValueIndex];\n      }\n    } else {\n      packedValue =\n          currentInt >>> packedValueLowBitsRightShift[packedValueIndex] & packedValueMask;\n      if (packedValueMiddleBitsLeftShift[packedValueIndex] != 0) {\n        packedValue |=\n            (loadInt()\n                & packedValueMiddleBitsMask[packedValueIndex])\n                << packedValueMiddleBitsLeftShift[packedValueIndex];\n      }\n    }\n\n    packedValueIndex++;\n    return packedValue;\n  }\n\n  /**\n   * A simple getter of {@link #packedValueIndex}.\n   */\n  public int getPackedValueIndex() {\n    return packedValueIndex;\n  }\n\n  /**\n   * A setter of {@link #packedValueIndex}. This setter will also set the correct\n   * {@link #indexInCurrentBlock} based on {@link #packedValueStartIndices}.\n   */\n  public void setPackedValueIndex(int packedValueIndex) {\n    this.packedValueIndex = packedValueIndex;\n    this.indexInCurrentBlock =\n        packedValueStartBlockIndex + packedValueStartIndices[packedValueIndex];\n    this.startPointerForCurrentBlock = packedValueStartBlockStart;\n    loadInt();\n  }\n\n  /**************************\n   * Private Helper Methods *\n   **************************/\n\n  /**\n   * Load a new int block, specified by {@link #startPointerForCurrentBlock}, from\n   * {@link #intBlockPool}. If {@link #queryCostTracker} is given, query cost with type\n   * {@link #queryCostType} will be tracked as well.\n   */\n  private void loadNextBlock() {\n    if (queryCostTracker != null) {\n      assert queryCostType != null;\n      queryCostTracker.track(queryCostType);\n    }\n\n    currentBlock = intBlockPool.getBlock(startPointerForCurrentBlock);\n  }\n\n  /**\n   * Load an int from {@link #currentBlock}. The loaded int will be returned as well.\n   * If the {@link #currentBlock} is used up, next block will be automatically loaded.\n   */\n  private int loadInt() {\n    while (indexInCurrentBlock >= IntBlockPool.BLOCK_SIZE) {\n      startPointerForCurrentBlock += IntBlockPool.BLOCK_SIZE;\n      loadNextBlock();\n\n      indexInCurrentBlock = Math.max(indexInCurrentBlock - IntBlockPool.BLOCK_SIZE, 0);\n    }\n\n    currentInt = currentBlock[indexInCurrentBlock++];\n    return currentInt;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/IntBlockPoolPackedLongsWriter.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\n/**\n * A packed ints writer writing packed values (int/long) into {@link IntBlockPool}.\n * @see IntBlockPoolPackedLongsReader\n *\n * A standard useage would be:\n * - set writer at an int block pool pointer and number of bits per packed value:\n *   {@link #jumpToInt(int, int)}\n * - write: {@link #writePackedInt(int)} or {@link #writePackedLong(long)}\n *\n * Example usage:\n * @see HighDFPackedIntsPostingLists\n */\npublic final class IntBlockPoolPackedLongsWriter {\n  /**\n   * Mask used to convert an int to a long. We cannot just cast because it will fill in the higher\n   * 32 bits with the sign bit, but we need the higher 32 bits to be 0 instead.\n   */\n  private static final long LONG_MASK = 0xFFFFFFFFL;\n\n  /** The int block pool into which packed ints will be written. */\n  private final IntBlockPool intBlockPool;\n\n  /** The value in the current position in the int block pool. */\n  private int currentIntValue = 0;\n\n  /** Starting bit index of unused bits in {@link #currentIntValue}. */\n  private int currentIntBitIndex = 0;\n\n  /** Pointer of {@link #currentIntValue} in {@link #intBlockPool}. */\n  private int currentIntPointer = -1;\n\n  /**\n   * Number of bits per packed value that will be written with\n   * {@link #writePackedInt(int)} or {@link #writePackedLong(long)}.\n   */\n  private int numBitsPerPackedValue = -1;\n\n  /**\n   * Mask used to extract the lower {@link #numBitsPerPackedValue} in a given value.\n   */\n  private long packedValueBitsMask = 0;\n\n  /**\n   * Sole constructor.\n   *\n   * @param intBlockPool into which packed ints will be written\n   */\n  public IntBlockPoolPackedLongsWriter(IntBlockPool intBlockPool) {\n    this.intBlockPool = intBlockPool;\n  }\n\n  /**\n   * 1. Set this writer to start writing at the given int block pool pointer.\n   * 2. Set number of bits per packed value that will be write.\n   * 3. Re-set {@link #currentIntValue} and {@link #currentIntBitIndex} to 0.\n   *\n   * @param intBlockPoolPointer the position this writer should start writing packed values. This\n   *                            pointer must be less then or equal to he length of the block pool.\n   *                            Subsequent writes will {@link IntBlockPool#add(int)} to the\n   *                            end of the int block pool if the given pointer equals to the length.\n   * @param bitsPerPackedValue must be non-negative.\n   */\n  public void jumpToInt(int intBlockPoolPointer, int bitsPerPackedValue) {\n    assert intBlockPoolPointer <= intBlockPool.length();\n    assert bitsPerPackedValue >= 0;\n\n    // Set the writer to start writing at the given int block pool pointer.\n    this.currentIntPointer = intBlockPoolPointer;\n\n    // Set number of bits that will be write per packed value.\n    this.numBitsPerPackedValue = bitsPerPackedValue;\n\n    // Compute the mask used to extract lower number of bitsPerPackedValue.\n    this.packedValueBitsMask =\n        bitsPerPackedValue == Long.SIZE ? -1L : (1L << bitsPerPackedValue) - 1;\n\n    // Reset current int data to 0.\n    this.currentIntValue = 0;\n    this.currentIntBitIndex = 0;\n  }\n\n  /**\n   * The given int value will be ZERO extended to a long and written using\n   * {@link #writePackedValueInternal(long)} (long)}.\n   *\n   * @see #LONG_MASK\n   */\n  public void writePackedInt(final int value) {\n    assert numBitsPerPackedValue <= Integer.SIZE;\n    writePackedValueInternal(LONG_MASK & value);\n  }\n\n  /**\n   * Write a long value.\n   * The given long value must bu UNABLE to fit in an int.\n   */\n  public void writePackedLong(final long value) {\n    assert numBitsPerPackedValue <= Long.SIZE;\n    writePackedValueInternal(value);\n  }\n\n  /*************************\n   * Private Helper Method *\n   *************************/\n\n  /**\n   * Write the given number of bits of the given value into this int pool as a packed int.\n   *\n   * @param value value will be written\n   */\n  private void writePackedValueInternal(final long value) {\n    // Extract lower 'numBitsPerPackedValue' from the given value.\n    long val = value & packedValueBitsMask;\n\n    assert val == value : String.format(\n        \"given value %d needs more bits than specified %d\", value, numBitsPerPackedValue);\n\n    int numBitsWrittenCurIter;\n    int numBitsRemaining = numBitsPerPackedValue;\n\n    // Each iteration of this while loop is writing part of the given value.\n    while (numBitsRemaining > 0) {\n      // Write into 'currentIntValue' int.\n      currentIntValue |= val << currentIntBitIndex;\n\n      // Calculate number of bits have been written in this iteration,\n      // we either used up all the remaining bits in 'currentIntValue' or\n      // finished up writing the value, whichever is smaller.\n      numBitsWrittenCurIter = Math.min(Integer.SIZE - currentIntBitIndex, numBitsRemaining);\n\n      // Number of bits remaining should be decremented.\n      numBitsRemaining -= numBitsWrittenCurIter;\n\n      // Right shift the value to remove the bits have been written.\n      val >>>= numBitsWrittenCurIter;\n\n      // Update bit index in current int.\n      currentIntBitIndex += numBitsWrittenCurIter;\n      assert currentIntBitIndex <= Integer.SIZE;\n\n      flush();\n\n      // if 'currentIntValue' int is used up.\n      if (currentIntBitIndex == Integer.SIZE) {\n        currentIntPointer++;\n\n        currentIntValue = 0;\n        currentIntBitIndex = 0;\n      }\n    }\n  }\n\n  /**\n   * Flush the {@link #currentIntValue} int into the int pool if the any bits of the int are used.\n   */\n  private void flush() {\n    if (currentIntPointer == intBlockPool.length()) {\n      intBlockPool.add(currentIntValue);\n      assert currentIntPointer + 1 == intBlockPool.length();\n    } else {\n      intBlockPool.set(currentIntPointer, currentIntValue);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/InvertedIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\n/**\n * Inverted index for a single field.\n *\n * Example: The field is \"hashtags\", this index contains a mapping from all the hashtags\n * that we've seen to a list of postings.\n */\npublic abstract class InvertedIndex implements FacetLabelProvider, Flushable {\n  protected final EarlybirdFieldType fieldType;\n\n  public InvertedIndex(EarlybirdFieldType fieldType) {\n    this.fieldType = fieldType;\n  }\n\n  public EarlybirdFieldType getFieldType() {\n    return fieldType;\n  }\n\n  /**\n   * Get the internal doc id of the oldest doc that includes term.\n   * @param term  the term to look for.\n   * @return  The internal docid, or TERM_NOT_FOUND.\n   */\n  public final int getLargestDocIDForTerm(BytesRef term) throws IOException {\n    final int termID = lookupTerm(term);\n    return getLargestDocIDForTerm(termID);\n  }\n\n  /**\n   * Get the document frequency for this term.\n   * @param term  the term to look for.\n   * @return  The document frequency of this term in the index.\n   */\n  public final int getDF(BytesRef term) throws IOException {\n    final int termID = lookupTerm(term);\n    if (termID == EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND) {\n      return 0;\n    }\n    return getDF(termID);\n  }\n\n  public boolean hasMaxPublishedPointer() {\n    return false;\n  }\n\n  public int getMaxPublishedPointer() {\n    return -1;\n  }\n\n  /**\n   * Create the Lucene magic Terms accessor.\n   * @param maxPublishedPointer used by the skip list to enable atomic document updates.\n   * @return  a new Terms object.\n   */\n  public abstract Terms createTerms(int maxPublishedPointer);\n\n  /**\n   * Create the Lucene magic TermsEnum accessor.\n   * @param maxPublishedPointer used by the skip list to enable atomic document updates.\n   * @return  a new TermsEnum object.\n   */\n  public abstract TermsEnum createTermsEnum(int maxPublishedPointer);\n\n  /**\n   * Returns the number of distinct terms in this inverted index.\n   * For example, if the indexed documents are:\n   *   \"i love chocolate and i love cakes\"\n   *   \"i love cookies\"\n   *\n   * then this method will return 6, because there are 6 distinct terms:\n   *   i, love, chocolate, and, cakes, cookies\n   */\n  public abstract int getNumTerms();\n\n  /**\n   * Returns the number of distinct documents in this index.\n   */\n  public abstract int getNumDocs();\n\n  /**\n   * Returns the total number of postings in this inverted index.\n   *\n   * For example, if the indexed documents are:\n   *   \"i love chocolate and i love cakes\"\n   *   \"i love cookies\"\n   *\n   * then this method will return 10, because there's a total of 10 words in these 2 documents.\n   */\n  public abstract int getSumTotalTermFreq();\n\n  /**\n   * Returns the sum of the number of documents for each term in this index.\n   *\n   * For example, if the indexed documents are:\n   *   \"i love chocolate and i love cakes\"\n   *   \"i love cookies\"\n   *\n   * then this method will return 8, because there are:\n   *   2 documents for term \"i\" (it doesn't matter that the first document has the term \"i\" twice)\n   *   2 documents for term \"love\" (same reason)\n   *   1 document for terms \"chocolate\", \"and\", \"cakes\", \"cookies\"\n   */\n  public abstract int getSumTermDocFreq();\n\n  /**\n   * Lookup a term.\n   * @param term  the term to lookup.\n   * @return  the term ID for this term.\n   */\n  public abstract int lookupTerm(BytesRef term) throws IOException;\n\n  /**\n   * Get the text for a given termID.\n   * @param termID  the term id\n   * @param text  a BytesRef that will be modified to contain the text of this termid.\n   */\n  public abstract void getTerm(int termID, BytesRef text);\n\n  /**\n   * Get the internal doc id of the oldest doc that includes this term.\n   * @param termID  The termID of the term.\n   * @return  The internal docid, or TERM_NOT_FOUND.\n   */\n  public abstract int getLargestDocIDForTerm(int termID) throws IOException;\n\n  /**\n   * Get the document frequency for a given termID\n   * @param termID  the term id\n   * @return  the document frequency of this term in this index.\n   */\n  public abstract int getDF(int termID);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/InvertedRealtimeIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\nimport java.util.Comparator;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.util.BytesRef;\nimport org.apache.lucene.util.StringHelper;\n\nimport com.twitter.search.common.hashtable.HashTable;\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.util.hash.KeysSource;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\npublic class InvertedRealtimeIndex extends InvertedIndex {\n  public static final int FIXED_HASH_SEED = 0;\n\n  public final class TermHashTable extends HashTable<BytesRef> {\n\n    private final TermPointerEncoding termPointerEncoding;\n\n    public TermHashTable(int size, TermPointerEncoding termPointerEncoding) {\n      super(size);\n      this.termPointerEncoding = termPointerEncoding;\n    }\n\n    public TermHashTable(int[] termsHash, TermPointerEncoding termPointerEncoding) {\n      super(termsHash);\n      this.termPointerEncoding = termPointerEncoding;\n    }\n\n    @Override\n    public boolean matchItem(BytesRef term, int candidateTermID) {\n      return ByteTermUtils.postingEquals(\n          getTermPool(),\n          termPointerEncoding.getTextStart(termsArray.termPointers[candidateTermID]), term);\n    }\n\n    @Override\n    public int hashCodeForItem(int itemID) {\n      return ByteTermUtils.hashCode(\n          getTermPool(), termPointerEncoding.getTextStart(termsArray.termPointers[itemID]));\n    }\n\n    /*\n     * Use a fixed hash seed to compute the hash code for the given item. This is necessary because\n     * we want the TermHashTable to be consistent for lookups in indexes that have been flushed and\n     * loaded across restarts and redeploys.\n     *\n     * Note: previously we used item.hashcode(), however that hash function relies on the seed value\n     * StringHelper.GOOD_FAST_HASH_SEED, which is initialized to System.currentTimeMillis() when the\n     * JVM process starts up.\n     */\n    public long lookupItem(BytesRef item) {\n      int itemHashCode = StringHelper.murmurhash3_x86_32(item, FIXED_HASH_SEED);\n\n      return super.lookupItem(item, itemHashCode);\n    }\n  }\n\n\n  /**\n   * Skip list comparator used by {@link #termsSkipList}. The key would be the bytesRef of the term,\n   *   and the value would be the termID of a term.\n   *\n   *   Notice this comparator is keeping states,\n   *   so different threads CANNOT share the same comparator.\n   */\n  public static final class TermsSkipListComparator implements SkipListComparator<BytesRef> {\n    private static final Comparator<BytesRef> BYTES_REF_COMPARATOR = Comparator.naturalOrder();\n\n    private static final int SENTINEL_VALUE = HashTable.EMPTY_SLOT;\n\n    // Initializing two BytesRef to use for later comparisons.\n    //   Notice different threads cannot share the same comparator.\n    private final BytesRef bytesRef1 = new BytesRef();\n    private final BytesRef bytesRef2 = new BytesRef();\n\n    /**\n     * We have to pass each part of the index in since during load process, the comparator\n     *   needs to be build before the index.\n     */\n    private final InvertedRealtimeIndex invertedIndex;\n\n    public TermsSkipListComparator(InvertedRealtimeIndex invertedIndex) {\n      this.invertedIndex = invertedIndex;\n    }\n\n    @Override\n    public int compareKeyWithValue(BytesRef key, int targetValue, int targetPosition) {\n      // No key could represent SENTINEL_VALUE and SENTINEL_VALUE is greatest.\n      if (targetValue == SENTINEL_VALUE) {\n        return -1;\n      } else {\n        getTerm(targetValue, bytesRef1);\n        return BYTES_REF_COMPARATOR.compare(key, bytesRef1);\n      }\n    }\n\n    @Override\n    public int compareValues(int v1, int v2) {\n      // SENTINEL_VALUE is greatest.\n      if (v1 != SENTINEL_VALUE && v2 != SENTINEL_VALUE) {\n        getTerm(v1, bytesRef1);\n        getTerm(v2, bytesRef2);\n        return BYTES_REF_COMPARATOR.compare(bytesRef1, bytesRef2);\n      } else if (v1 == SENTINEL_VALUE && v2 == SENTINEL_VALUE) {\n        return 0;\n      } else if (v1 == SENTINEL_VALUE) {\n        return 1;\n      } else {\n        return -1;\n      }\n    }\n\n    @Override\n    public int getSentinelValue() {\n      return SENTINEL_VALUE;\n    }\n\n    /**\n     * Get the term specified by the termID.\n     *   This method should be the same as {@link InvertedRealtimeIndex#getTerm}\n     */\n    private void getTerm(int termID, BytesRef text) {\n      invertedIndex.getTerm(termID, text);\n    }\n  }\n\n  private static final int HASHMAP_SIZE = 64 * 1024;\n\n  private SkipListContainer<BytesRef> termsSkipList;\n\n  private final TermPointerEncoding termPointerEncoding;\n  private final ByteBlockPool termPool;\n  private final SkipListPostingList postingList;\n\n  private int numTerms;\n  private int numDocs;\n  private int sumTotalTermFreq;\n  private int sumTermDocFreq;\n  private int maxPosition;\n\n  private volatile TermHashTable hashTable;\n  private TermsArray termsArray;\n\n  /**\n   * Creates a new in-memory real-time inverted index for the given field.\n   */\n  public InvertedRealtimeIndex(EarlybirdFieldType fieldType,\n                               TermPointerEncoding termPointerEncoding,\n                               String fieldName) {\n    super(fieldType);\n    this.termPool = new ByteBlockPool();\n\n    this.termPointerEncoding = termPointerEncoding;\n    this.hashTable = new TermHashTable(HASHMAP_SIZE, termPointerEncoding);\n\n    this.postingList = new SkipListPostingList(\n        fieldType.hasPositions()\n            ? SkipListContainer.HasPositions.YES\n            : SkipListContainer.HasPositions.NO,\n        fieldType.isStorePerPositionPayloads()\n            ? SkipListContainer.HasPayloads.YES\n            : SkipListContainer.HasPayloads.NO,\n        fieldName);\n\n    this.termsArray = new TermsArray(\n        HASHMAP_SIZE, fieldType.isStoreFacetOffensiveCounters());\n\n    // Create termsSkipList to maintain order if field is support ordered terms.\n    if (fieldType.isSupportOrderedTerms()) {\n      // Terms skip list does not support position.\n      this.termsSkipList = new SkipListContainer<>(\n          new TermsSkipListComparator(this),\n          SkipListContainer.HasPositions.NO,\n          SkipListContainer.HasPayloads.NO,\n          \"terms\");\n      this.termsSkipList.newSkipList();\n    } else {\n      this.termsSkipList = null;\n    }\n  }\n\n  void setTermsSkipList(SkipListContainer<BytesRef> termsSkipList) {\n    this.termsSkipList = termsSkipList;\n  }\n\n  SkipListContainer<BytesRef> getTermsSkipList() {\n    return termsSkipList;\n  }\n\n  private InvertedRealtimeIndex(\n      EarlybirdFieldType fieldType,\n      int numTerms,\n      int numDocs,\n      int sumTermDocFreq,\n      int sumTotalTermFreq,\n      int maxPosition,\n      int[] termsHash,\n      TermsArray termsArray,\n      ByteBlockPool termPool,\n      TermPointerEncoding termPointerEncoding,\n      SkipListPostingList postingList) {\n    super(fieldType);\n    this.numTerms = numTerms;\n    this.numDocs = numDocs;\n    this.sumTermDocFreq = sumTermDocFreq;\n    this.sumTotalTermFreq = sumTotalTermFreq;\n    this.maxPosition = maxPosition;\n    this.termsArray = termsArray;\n    this.termPool = termPool;\n    this.termPointerEncoding = termPointerEncoding;\n    this.hashTable = new TermHashTable(termsHash, termPointerEncoding);\n    this.postingList = postingList;\n  }\n\n  void insertToTermsSkipList(BytesRef termBytesRef, int termID) {\n    if (termsSkipList != null) {\n      // Use the comparator passed in while building the skip list since we only have one writer.\n      termsSkipList.insert(termBytesRef, termID, SkipListContainer.FIRST_LIST_HEAD);\n    }\n  }\n\n  @Override\n  public int getNumTerms() {\n    return numTerms;\n  }\n\n  @Override\n  public int getNumDocs() {\n    return numDocs;\n  }\n\n  @Override\n  public int getSumTotalTermFreq() {\n    return sumTotalTermFreq;\n  }\n\n  @Override\n  public int getSumTermDocFreq() {\n    return sumTermDocFreq;\n  }\n\n  @Override\n  public Terms createTerms(int maxPublishedPointer) {\n    return new RealtimeIndexTerms(this, maxPublishedPointer);\n  }\n\n  @Override\n  public TermsEnum createTermsEnum(int maxPublishedPointer) {\n    // Use SkipListInMemoryTermsEnum if termsSkipList is not null, which indicates field required\n    // ordered term.\n    if (termsSkipList == null) {\n      return new RealtimeIndexTerms.InMemoryTermsEnum(this, maxPublishedPointer);\n    } else {\n      return new RealtimeIndexTerms.SkipListInMemoryTermsEnum(this, maxPublishedPointer);\n    }\n  }\n\n  int getPostingListPointer(int termID) {\n    return termsArray.getPostingsPointer(termID);\n  }\n\n  @Override\n  public int getLargestDocIDForTerm(int termID) {\n    if (termID == EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND) {\n      return TermsArray.INVALID;\n    } else {\n      return postingList.getDocIDFromPosting(termsArray.largestPostings[termID]);\n    }\n  }\n\n  @Override\n  public int getDF(int termID) {\n    if (termID == HashTable.EMPTY_SLOT) {\n      return 0;\n    } else {\n      return this.postingList.getDF(termID, termsArray);\n    }\n  }\n\n  @Override\n  public int getMaxPublishedPointer() {\n    return this.postingList.getMaxPublishedPointer();\n  }\n\n  @Override\n  public int lookupTerm(BytesRef term) {\n    return HashTable.decodeItemId(hashTable.lookupItem(term));\n  }\n\n  @Override\n  public FacetLabelAccessor getLabelAccessor() {\n    final TermsArray termsArrayCopy = this.termsArray;\n\n    return new FacetLabelAccessor() {\n      @Override protected boolean seek(long termID) {\n        if (termID == HashTable.EMPTY_SLOT) {\n          return false;\n        }\n        int termPointer = termsArrayCopy.termPointers[(int) termID];\n        hasTermPayload = termPointerEncoding.hasPayload(termPointer);\n        int textStart = termPointerEncoding.getTextStart(termPointer);\n        int termPayloadStart = ByteTermUtils.setBytesRef(termPool, termRef, textStart);\n        if (hasTermPayload) {\n          ByteTermUtils.setBytesRef(termPool, termPayload, termPayloadStart);\n        }\n        offensiveCount = termsArrayCopy.offensiveCounters != null\n            ? termsArrayCopy.offensiveCounters[(int) termID] : 0;\n\n        return true;\n      }\n    };\n  }\n\n  @Override\n  public boolean hasMaxPublishedPointer() {\n    return true;\n  }\n\n  @Override\n  public void getTerm(int termID, BytesRef text) {\n    getTerm(termID, text, termsArray, termPointerEncoding, termPool);\n  }\n\n  /**\n   * Extract to helper method so the logic can be shared with\n   *   {@link TermsSkipListComparator#getTerm}\n   */\n  private static void getTerm(int termID, BytesRef text,\n                              TermsArray termsArray,\n                              TermPointerEncoding termPointerEncoding,\n                              ByteBlockPool termPool) {\n    int textStart = termPointerEncoding.getTextStart(termsArray.termPointers[termID]);\n    ByteTermUtils.setBytesRef(termPool, text, textStart);\n  }\n\n  /**\n   * Called when postings hash is too small (> 50% occupied).\n   */\n  void rehashPostings(int newSize) {\n    TermHashTable newTable = new TermHashTable(newSize, termPointerEncoding);\n    hashTable.rehash(newTable);\n    hashTable = newTable;\n  }\n\n  /**\n   * Returns per-term array containing the number of documents indexed with that term that were\n   * considered to be offensive.\n   */\n  @Nullable\n  int[] getOffensiveCounters() {\n    return this.termsArray.offensiveCounters;\n  }\n\n  /**\n   * Returns access to all the terms in this index as a {@link KeysSource}.\n   */\n  public KeysSource getKeysSource() {\n    final int localNumTerms = this.numTerms;\n    final TermsArray termsArrayCopy = this.termsArray;\n\n    return new KeysSource() {\n      private int termID = 0;\n      private BytesRef text = new BytesRef();\n\n      @Override\n      public int getNumberOfKeys() {\n        return localNumTerms;\n      }\n\n      /** Must not be called more often than getNumberOfKeys() before rewind() is called */\n      @Override\n      public BytesRef nextKey() {\n        Preconditions.checkState(termID < localNumTerms);\n        int textStart = termPointerEncoding.getTextStart(termsArrayCopy.termPointers[termID]);\n        ByteTermUtils.setBytesRef(termPool, text, textStart);\n        termID++;\n        return text;\n      }\n\n      @Override\n      public void rewind() {\n        termID = 0;\n      }\n    };\n  }\n\n  /**\n   * Returns byte pool containing term text for all terms in this index.\n   */\n  public ByteBlockPool getTermPool() {\n    return this.termPool;\n  }\n\n  /**\n   * Returns per-term array containing pointers to where the text of each term is stored in the\n   * byte pool returned by {@link #getTermPool()}.\n   */\n  public int[] getTermPointers() {\n    return this.termsArray.termPointers;\n  }\n\n  /**\n   * Returns the hash table used to look up terms in this index.\n   */\n  InvertedRealtimeIndex.TermHashTable getHashTable() {\n    return hashTable;\n  }\n\n\n  TermsArray getTermsArray() {\n    return termsArray;\n  }\n\n  TermsArray growTermsArray() {\n    termsArray = termsArray.grow();\n    return termsArray;\n  }\n\n  @SuppressWarnings(\"unchecked\")\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  TermPointerEncoding getTermPointerEncoding() {\n    return termPointerEncoding;\n  }\n\n  SkipListPostingList getPostingList() {\n    return postingList;\n  }\n\n  void incrementNumTerms() {\n    numTerms++;\n  }\n\n  void incrementSumTotalTermFreq() {\n    sumTotalTermFreq++;\n  }\n\n  public void incrementSumTermDocFreq() {\n    sumTermDocFreq++;\n  }\n\n  public void incrementNumDocs() {\n    numDocs++;\n  }\n\n  void setNumDocs(int numDocs) {\n    this.numDocs = numDocs;\n  }\n\n  void adjustMaxPosition(int position) {\n    if (position > maxPosition) {\n      maxPosition = position;\n    }\n  }\n\n  int getMaxPosition() {\n    return maxPosition;\n  }\n\n  public static class FlushHandler extends Flushable.Handler<InvertedRealtimeIndex> {\n    private static final String NUM_DOCS_PROP_NAME = \"numDocs\";\n    private static final String SUM_TOTAL_TERM_FREQ_PROP_NAME = \"sumTotalTermFreq\";\n    private static final String SUM_TERM_DOC_FREQ_PROP_NAME = \"sumTermDocFreq\";\n    private static final String NUM_TERMS_PROP_NAME = \"numTerms\";\n    private static final String POSTING_LIST_PROP_NAME = \"postingList\";\n    private static final String TERMS_SKIP_LIST_PROP_NAME = \"termsSkipList\";\n    private static final String MAX_POSITION = \"maxPosition\";\n\n    protected final EarlybirdFieldType fieldType;\n    protected final TermPointerEncoding termPointerEncoding;\n\n    public FlushHandler(EarlybirdFieldType fieldType,\n                        TermPointerEncoding termPointerEncoding) {\n      this.fieldType = fieldType;\n      this.termPointerEncoding = termPointerEncoding;\n    }\n\n    public FlushHandler(InvertedRealtimeIndex objectToFlush) {\n      super(objectToFlush);\n      this.fieldType = objectToFlush.fieldType;\n      this.termPointerEncoding = objectToFlush.getTermPointerEncoding();\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out)\n        throws IOException {\n      InvertedRealtimeIndex objectToFlush = getObjectToFlush();\n      flushInfo.addIntProperty(NUM_TERMS_PROP_NAME, objectToFlush.getNumTerms());\n      flushInfo.addIntProperty(NUM_DOCS_PROP_NAME, objectToFlush.numDocs);\n      flushInfo.addIntProperty(SUM_TERM_DOC_FREQ_PROP_NAME, objectToFlush.sumTermDocFreq);\n      flushInfo.addIntProperty(SUM_TOTAL_TERM_FREQ_PROP_NAME, objectToFlush.sumTotalTermFreq);\n      flushInfo.addIntProperty(MAX_POSITION, objectToFlush.maxPosition);\n\n      out.writeIntArray(objectToFlush.hashTable.slots());\n      objectToFlush.termsArray.getFlushHandler()\n          .flush(flushInfo.newSubProperties(\"termsArray\"), out);\n      objectToFlush.getTermPool().getFlushHandler()\n          .flush(flushInfo.newSubProperties(\"termPool\"), out);\n      objectToFlush.getPostingList().getFlushHandler()\n          .flush(flushInfo.newSubProperties(POSTING_LIST_PROP_NAME), out);\n\n      if (fieldType.isSupportOrderedTerms()) {\n        Preconditions.checkNotNull(objectToFlush.termsSkipList);\n\n        objectToFlush.termsSkipList.getFlushHandler()\n            .flush(flushInfo.newSubProperties(TERMS_SKIP_LIST_PROP_NAME), out);\n      }\n    }\n\n    @Override\n    protected InvertedRealtimeIndex doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      int[] termsHash = in.readIntArray();\n      TermsArray termsArray = (new TermsArray.FlushHandler())\n          .load(flushInfo.getSubProperties(\"termsArray\"), in);\n      ByteBlockPool termPool = (new ByteBlockPool.FlushHandler())\n          .load(flushInfo.getSubProperties(\"termPool\"), in);\n      SkipListPostingList postingList = (new SkipListPostingList.FlushHandler())\n          .load(flushInfo.getSubProperties(POSTING_LIST_PROP_NAME), in);\n\n      InvertedRealtimeIndex index = new InvertedRealtimeIndex(\n          fieldType,\n          flushInfo.getIntProperty(NUM_TERMS_PROP_NAME),\n          flushInfo.getIntProperty(NUM_DOCS_PROP_NAME),\n          flushInfo.getIntProperty(SUM_TERM_DOC_FREQ_PROP_NAME),\n          flushInfo.getIntProperty(SUM_TOTAL_TERM_FREQ_PROP_NAME),\n          flushInfo.getIntProperty(MAX_POSITION),\n          termsHash,\n          termsArray,\n          termPool,\n          termPointerEncoding,\n          postingList);\n\n      if (fieldType.isSupportOrderedTerms()) {\n        SkipListComparator<BytesRef> comparator = new TermsSkipListComparator(index);\n        index.setTermsSkipList((new SkipListContainer.FlushHandler<>(comparator))\n            .load(flushInfo.getSubProperties(TERMS_SKIP_LIST_PROP_NAME), in));\n      }\n\n      return index;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/InvertedRealtimeIndexWriter.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.analysis.tokenattributes.PayloadAttribute;\nimport org.apache.lucene.analysis.tokenattributes.TermToBytesRefAttribute;\nimport org.apache.lucene.util.AttributeSource;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.hashtable.HashTable;\nimport com.twitter.search.common.util.analysis.TermPayloadAttribute;\nimport com.twitter.search.core.earlybird.facets.FacetCountingArrayWriter;\nimport com.twitter.search.core.earlybird.facets.FacetIDMap.FacetField;\nimport com.twitter.search.core.earlybird.index.EarlybirdRealtimeIndexSegmentWriter;\n\npublic class InvertedRealtimeIndexWriter\n    implements EarlybirdRealtimeIndexSegmentWriter.InvertedDocConsumer {\n  private final InvertedRealtimeIndex invertedIndex;\n  private final FacetCountingArrayWriter facetArray;\n  private final FacetField facetField;\n\n  private TermToBytesRefAttribute termAtt;\n  private TermPayloadAttribute termPayloadAtt;\n  private PayloadAttribute payloadAtt;\n  private boolean currentDocIsOffensive;\n\n  /**\n   * Creates a new writer for writing to an inverted in-memory real-time index.\n   */\n  public InvertedRealtimeIndexWriter(\n          InvertedRealtimeIndex index,\n          FacetField facetField,\n          FacetCountingArrayWriter facetArray) {\n    super();\n    this.invertedIndex = index;\n    this.facetArray = facetArray;\n    this.facetField = facetField;\n  }\n\n  @Override\n  public void start(AttributeSource attributeSource, boolean docIsOffensive) {\n    termAtt = attributeSource.addAttribute(TermToBytesRefAttribute.class);\n    termPayloadAtt = attributeSource.addAttribute(TermPayloadAttribute.class);\n    payloadAtt = attributeSource.addAttribute(PayloadAttribute.class);\n    currentDocIsOffensive = docIsOffensive;\n  }\n\n  /**\n   * Adds a posting to the provided inverted index.\n   *\n   * @param termBytesRef is a payload that is stored with the term. It is only stored once for each\n   *                     term.\n   * @param postingPayload is a byte payload that will be stored separately for every posting.\n   * @return term id of the added posting.\n   */\n  public static int indexTerm(InvertedRealtimeIndex invertedIndex, BytesRef termBytesRef,\n      int docID, int position, BytesRef termPayload,\n      BytesRef postingPayload, TermPointerEncoding termPointerEncoding) {\n\n    InvertedRealtimeIndex.TermHashTable hashTable = invertedIndex.getHashTable();\n    BaseByteBlockPool termPool = invertedIndex.getTermPool();\n\n    TermsArray termsArray = invertedIndex.getTermsArray();\n\n    long hashTableInfoForBytesRef = hashTable.lookupItem(termBytesRef);\n    int termID = HashTable.decodeItemId(hashTableInfoForBytesRef);\n    int hashTableSlot = HashTable.decodeHashPosition(hashTableInfoForBytesRef);\n\n    invertedIndex.adjustMaxPosition(position);\n\n    if (termID == HashTable.EMPTY_SLOT) {\n      // First time we are seeing this token since we last flushed the hash.\n      // the LSB in textStart denotes whether this term has a term payload\n      int textStart = ByteTermUtils.copyToTermPool(termPool, termBytesRef);\n      boolean hasTermPayload = termPayload != null;\n      int termPointer = termPointerEncoding.encodeTermPointer(textStart, hasTermPayload);\n\n      if (hasTermPayload) {\n        ByteTermUtils.copyToTermPool(termPool, termPayload);\n      }\n\n      termID = invertedIndex.getNumTerms();\n      invertedIndex.incrementNumTerms();\n      if (termID >= termsArray.getSize()) {\n        termsArray = invertedIndex.growTermsArray();\n      }\n\n      termsArray.termPointers[termID] = termPointer;\n\n      Preconditions.checkState(hashTable.slots()[hashTableSlot] == HashTable.EMPTY_SLOT);\n      hashTable.setSlot(hashTableSlot, termID);\n\n      if (invertedIndex.getNumTerms() * 2 >= hashTable.numSlots()) {\n        invertedIndex.rehashPostings(2 * hashTable.numSlots());\n      }\n\n      // Insert termID into termsSkipList.\n      invertedIndex.insertToTermsSkipList(termBytesRef, termID);\n    }\n\n    invertedIndex.incrementSumTotalTermFreq();\n    invertedIndex.getPostingList()\n        .appendPosting(termID, termsArray, docID, position, postingPayload);\n\n    return termID;\n  }\n\n  /**\n   * Delete a posting that was inserted out of order.\n   *\n   * This function needs work before it is used in production:\n   * - It should take an isDocOffensive parameter so we can decrement the offensive\n   *   document count for the term.\n   * - It doesn't allow the same concurrency guarantees that the other posting methods do.\n   */\n  public static void deletePosting(\n      InvertedRealtimeIndex invertedIndex, BytesRef termBytesRef, int docID) {\n\n    long hashTableInfoForBytesRef = invertedIndex.getHashTable().lookupItem(termBytesRef);\n    int termID = HashTable.decodeItemId(hashTableInfoForBytesRef);\n\n    if (termID != HashTable.EMPTY_SLOT) {\n      // Have seen this term before, and the field that supports deletes.\n      invertedIndex.getPostingList().deletePosting(termID, invertedIndex.getTermsArray(), docID);\n    }\n  }\n\n  @Override\n  public void add(int docID, int position) {\n    final BytesRef payload;\n    if (payloadAtt == null) {\n      payload = null;\n    } else {\n      payload = payloadAtt.getPayload();\n    }\n\n    BytesRef termPayload = termPayloadAtt.getTermPayload();\n\n    int termID = indexTerm(invertedIndex, termAtt.getBytesRef(),\n        docID, position, termPayload, payload,\n        invertedIndex.getTermPointerEncoding());\n\n    if (termID == -1) {\n      return;\n    }\n\n    TermsArray termsArray = invertedIndex.getTermsArray();\n\n    if (currentDocIsOffensive && termsArray.offensiveCounters != null) {\n      termsArray.offensiveCounters[termID]++;\n    }\n\n    if (facetField != null) {\n      facetArray.addFacet(docID, facetField.getFacetId(), termID);\n    }\n  }\n\n  @Override\n  public void finish() {\n    payloadAtt = null;\n    termPayloadAtt = null;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/LowDFPackedIntsPostingLists.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.util.packed.PackedInts;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\n\n/**\n * A posting list intended for low-df terms, terms that have a small number of postings.\n *\n * The postings (docs and positions) are stored in PackedInts, packed based on the largest docId\n * and position across all low-df terms in a field.\n *\n * All docIds are packed together in their own PackedInts, and all positions are stored together\n * in their own PackedInts.\n *  - A docId is stored for every single posting, that is if a doc has a frequency of N, it will be\n * stored N times.\n * - For fields that omitPositions, positions are not stored at all.\n *\n * Example:\n * Postings in the form (docId, position):\n *   (1, 0), (1, 1), (2, 1), (2, 3), (2, 5), (4, 0), (5, 0)\n * Will be stored as:\n *   packedDocIds:    [1, 1, 2, 2, 2, 4, 5]\n *   packedPositions: [0, 1, 1, 3, 5, 0, 0]\n */\npublic class LowDFPackedIntsPostingLists extends OptimizedPostingLists {\n  private static final SearchCounter GETTING_POSITIONS_WITH_OMIT_POSITIONS =\n      SearchCounter.export(\"low_df_packed_ints_posting_list_getting_positions_with_omit_positions\");\n\n  /**\n   * Internal class for hiding PackedInts Readers and Writers. A Mutable instance of PackedInts is\n   * only required when we're optimizing a new index.\n   * For the read side, we only need a PackedInts.Reader.\n   * For loaded indexes, we also only need a PackedInts.Reader.\n   */\n  private static final class PackedIntsWrapper {\n    // Will be null if we are operating on a loaded in read-only index.\n    @Nullable\n    private final PackedInts.Mutable mutablePackedInts;\n    private final PackedInts.Reader readerPackedInts;\n\n    private PackedIntsWrapper(PackedInts.Mutable mutablePackedInts) {\n      this.mutablePackedInts = Preconditions.checkNotNull(mutablePackedInts);\n      this.readerPackedInts = mutablePackedInts;\n    }\n\n    private PackedIntsWrapper(PackedInts.Reader readerPackedInts) {\n      this.mutablePackedInts = null;\n      this.readerPackedInts = readerPackedInts;\n    }\n\n    public int size() {\n      return readerPackedInts.size();\n    }\n\n    public PackedInts.Reader getReader() {\n      return readerPackedInts;\n    }\n\n    public void set(int index, long value) {\n      this.mutablePackedInts.set(index, value);\n    }\n  }\n\n  private final PackedIntsWrapper packedDocIds;\n  /**\n   * Will be null for fields that omitPositions.\n   */\n  @Nullable\n  private final PackedIntsWrapper packedPositions;\n  private final boolean omitPositions;\n  private final int totalPostingsAcrossTerms;\n  private final int maxPosition;\n  private int currentPackedIntsPosition;\n\n  /**\n   * Creates a new LowDFPackedIntsPostingLists.\n   * @param omitPositions whether positions should be omitted or not.\n   * @param totalPostingsAcrossTerms how many postings across all terms this field has.\n   * @param maxPosition the largest position used in all the postings for this field.\n   */\n  public LowDFPackedIntsPostingLists(\n      boolean omitPositions,\n      int totalPostingsAcrossTerms,\n      int maxPosition) {\n    this(\n        new PackedIntsWrapper(PackedInts.getMutable(\n            totalPostingsAcrossTerms,\n            PackedInts.bitsRequired(MAX_DOC_ID),\n            PackedInts.DEFAULT)),\n        omitPositions\n            ? null\n            : new PackedIntsWrapper(PackedInts.getMutable(\n            totalPostingsAcrossTerms,\n            PackedInts.bitsRequired(maxPosition),\n            PackedInts.DEFAULT)),\n        omitPositions,\n        totalPostingsAcrossTerms,\n        maxPosition);\n  }\n\n  private LowDFPackedIntsPostingLists(\n      PackedIntsWrapper packedDocIds,\n      @Nullable\n      PackedIntsWrapper packedPositions,\n      boolean omitPositions,\n      int totalPostingsAcrossTerms,\n      int maxPosition) {\n    this.packedDocIds = packedDocIds;\n    this.packedPositions = packedPositions;\n    this.omitPositions = omitPositions;\n    this.totalPostingsAcrossTerms = totalPostingsAcrossTerms;\n    this.maxPosition = maxPosition;\n    this.currentPackedIntsPosition = 0;\n  }\n\n  @Override\n  public int copyPostingList(PostingsEnum postingsEnum, int numPostings) throws IOException {\n    int pointer = currentPackedIntsPosition;\n\n    int docId;\n\n    while ((docId = postingsEnum.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {\n      assert docId <= MAX_DOC_ID;\n      int freq = postingsEnum.freq();\n      assert freq <= numPostings;\n\n      for (int i = 0; i < freq; i++) {\n        packedDocIds.set(currentPackedIntsPosition, docId);\n        if (packedPositions != null) {\n          int position = postingsEnum.nextPosition();\n          assert position <= maxPosition;\n          packedPositions.set(currentPackedIntsPosition, position);\n        }\n        currentPackedIntsPosition++;\n      }\n    }\n\n    return pointer;\n  }\n\n  @Override\n  public EarlybirdPostingsEnum postings(\n      int postingListPointer,\n      int numPostings,\n      int flags) throws IOException {\n\n    if (PostingsEnum.featureRequested(flags, PostingsEnum.POSITIONS) && !omitPositions) {\n      assert packedPositions != null;\n      return new LowDFPackedIntsPostingsEnum(\n          packedDocIds.getReader(),\n          packedPositions.getReader(),\n          postingListPointer,\n          numPostings);\n    } else {\n      if (PostingsEnum.featureRequested(flags, PostingsEnum.POSITIONS) && omitPositions) {\n        GETTING_POSITIONS_WITH_OMIT_POSITIONS.increment();\n      }\n\n      return new LowDFPackedIntsPostingsEnum(\n          packedDocIds.getReader(),\n          null, // no positions\n          postingListPointer,\n          numPostings);\n    }\n  }\n\n  @VisibleForTesting\n  int getPackedIntsSize() {\n    return packedDocIds.size();\n  }\n\n  @VisibleForTesting\n  int getMaxPosition() {\n    return maxPosition;\n  }\n\n  @VisibleForTesting\n  boolean isOmitPositions() {\n    return omitPositions;\n  }\n\n  @SuppressWarnings(\"unchecked\")\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  static class FlushHandler extends Flushable.Handler<LowDFPackedIntsPostingLists> {\n    private static final String OMIT_POSITIONS_PROP_NAME = \"omitPositions\";\n    private static final String TOTAL_POSTINGS_PROP_NAME = \"totalPostingsAcrossTerms\";\n    private static final String MAX_POSITION_PROP_NAME = \"maxPosition\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(LowDFPackedIntsPostingLists objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      LowDFPackedIntsPostingLists objectToFlush = getObjectToFlush();\n\n      flushInfo.addBooleanProperty(OMIT_POSITIONS_PROP_NAME, objectToFlush.omitPositions);\n      flushInfo.addIntProperty(TOTAL_POSTINGS_PROP_NAME, objectToFlush.totalPostingsAcrossTerms);\n      flushInfo.addIntProperty(MAX_POSITION_PROP_NAME, objectToFlush.maxPosition);\n\n      out.writePackedInts(objectToFlush.packedDocIds.getReader());\n\n      if (!objectToFlush.omitPositions) {\n        assert objectToFlush.packedPositions != null;\n        out.writePackedInts(objectToFlush.packedPositions.getReader());\n      }\n    }\n\n    @Override\n    protected LowDFPackedIntsPostingLists doLoad(\n        FlushInfo flushInfo,\n        DataDeserializer in) throws IOException {\n\n      boolean omitPositions = flushInfo.getBooleanProperty(OMIT_POSITIONS_PROP_NAME);\n      int totalPostingsAcrossTerms = flushInfo.getIntProperty(TOTAL_POSTINGS_PROP_NAME);\n      int maxPosition = flushInfo.getIntProperty(MAX_POSITION_PROP_NAME);\n\n      PackedIntsWrapper packedDocIds = new PackedIntsWrapper(in.readPackedInts());\n\n      PackedIntsWrapper packedPositions = null;\n      if (!omitPositions) {\n        packedPositions = new PackedIntsWrapper(in.readPackedInts());\n      }\n\n      return new LowDFPackedIntsPostingLists(\n          packedDocIds,\n          packedPositions,\n          omitPositions,\n          totalPostingsAcrossTerms,\n          maxPosition);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/LowDFPackedIntsPostingsEnum.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport javax.annotation.Nullable;\n\nimport org.apache.lucene.util.packed.PackedInts;\n\n/**\n * A PostingsEnum for iterating over LowDFPackedIntsPostingLists.\n *\n * Can be used with positions and without positions.\n */\npublic class LowDFPackedIntsPostingsEnum extends EarlybirdOptimizedPostingsEnum {\n  private static final int SKIP_INTERVAL = 128;\n\n  private final PackedInts.Reader packedDocIds;\n  @Nullable\n  private final PackedInts.Reader packedPositions;\n  private final int lastPostingPointer;\n  private final int largestDocID;\n  private int currentPositionPointer;\n\n  /** Pointer to the next posting that will be loaded. */\n  private int nextPostingPointer;\n\n  /**\n   * Creates a new PostingsEnum for all postings in a given term.\n   */\n  public LowDFPackedIntsPostingsEnum(\n      PackedInts.Reader packedDocIds,\n      @Nullable\n      PackedInts.Reader packedPositions,\n      int postingListPointer,\n      int numPostings) {\n    super(postingListPointer, numPostings);\n\n    this.packedDocIds = packedDocIds;\n    this.packedPositions = packedPositions;\n    this.nextPostingPointer = postingListPointer;\n\n    this.lastPostingPointer = postingListPointer + numPostings - 1;\n    this.largestDocID = (int) packedDocIds.get(lastPostingPointer);\n\n    loadNextPosting();\n\n    // Treat each term as a single block load.\n    queryCostTracker.track(QueryCostTracker.CostType.LOAD_OPTIMIZED_POSTING_BLOCK);\n  }\n\n  @Override\n  protected void loadNextPosting() {\n    if (nextPostingPointer <= lastPostingPointer) {\n      nextDocID = (int) packedDocIds.get(nextPostingPointer);\n      nextFreq = 1;\n    } else {\n      // all postings fully processed\n      nextDocID = NO_MORE_DOCS;\n      nextFreq = 0;\n    }\n    nextPostingPointer++;\n  }\n\n  @Override\n  protected void startCurrentDoc() {\n    if (packedPositions != null) {\n      /**\n       * Remember where we were at the beginning of this doc, so that we can iterate over the\n       * positions for this doc if needed.\n       * Adjust by `- 1 - getCurrentFreq()` because we already advanced beyond the last posting in\n       * the previous loadNextPosting() calls.\n       * @see #nextDocNoDel()\n       */\n      currentPositionPointer = nextPostingPointer - 1 - getCurrentFreq();\n    }\n  }\n\n  @Override\n  protected void skipTo(int target) {\n    assert target != NO_MORE_DOCS : \"Should be handled in parent class advance method\";\n\n    // now we know there must be a doc in this block that we can return\n    int skipIndex = nextPostingPointer + SKIP_INTERVAL;\n    while (skipIndex <= lastPostingPointer && target > packedDocIds.get(skipIndex)) {\n      nextPostingPointer = skipIndex;\n      skipIndex += SKIP_INTERVAL;\n    }\n  }\n\n  @Override\n  public int nextPosition() throws IOException {\n    if (packedPositions == null) {\n      return -1;\n    } else if (currentPositionPointer < packedPositions.size()) {\n      return (int) packedPositions.get(currentPositionPointer++);\n    } else {\n      return -1;\n    }\n  }\n\n  @Override\n  public int getLargestDocID() throws IOException {\n    return largestDocID;\n  }\n\n  @Override\n  public long cost() {\n    // cost would be -1 if this enum is exhausted.\n    final int cost = lastPostingPointer - nextPostingPointer + 1;\n    return cost < 0 ? 0 : cost;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/MPHTermDictionary.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.BaseTermsEnum;\nimport org.apache.lucene.index.ImpactsEnum;\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.index.SlowImpactsEnum;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.util.BytesRef;\nimport org.apache.lucene.util.packed.PackedInts;\n\nimport com.twitter.search.common.util.hash.BDZAlgorithm;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\npublic class MPHTermDictionary implements TermDictionary, Flushable {\n  private final BDZAlgorithm termsHashFunction;\n  private final PackedInts.Reader termPointers;\n  private final ByteBlockPool termPool;\n  private final TermPointerEncoding termPointerEncoding;\n  private final int numTerms;\n\n  MPHTermDictionary(int numTerms, BDZAlgorithm termsHashFunction,\n      PackedInts.Reader termPointers, ByteBlockPool termPool,\n      TermPointerEncoding termPointerEncoding) {\n    this.numTerms = numTerms;\n    this.termsHashFunction = termsHashFunction;\n    this.termPointers = termPointers;\n    this.termPool = termPool;\n    this.termPointerEncoding = termPointerEncoding;\n  }\n\n  @Override\n  public int getNumTerms() {\n    return numTerms;\n  }\n\n  @Override\n  public int lookupTerm(BytesRef term) {\n    int termID = termsHashFunction.lookup(term);\n    if (termID >= getNumTerms() || termID < 0) {\n      return EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND;\n    }\n\n    if (ByteTermUtils.postingEquals(termPool, termPointerEncoding\n            .getTextStart((int) termPointers.get(termID)), term)) {\n      return termID;\n    } else {\n      return EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND;\n    }\n  }\n\n  @Override\n  public boolean getTerm(int termID, BytesRef text, BytesRef termPayload) {\n    int termPointer = (int) termPointers.get(termID);\n    boolean hasTermPayload = termPointerEncoding.hasPayload(termPointer);\n    int textStart = termPointerEncoding.getTextStart(termPointer);\n    // setBytesRef sets the passed in BytesRef \"text\" to the term in the termPool.\n    // As a side effect it returns the offset of the next entry in the pool after the term,\n    // which may optionally be used if this term has a payload.\n    int termPayloadStart = ByteTermUtils.setBytesRef(termPool, text, textStart);\n    if (termPayload != null && hasTermPayload) {\n      ByteTermUtils.setBytesRef(termPool, termPayload, termPayloadStart);\n    }\n\n    return hasTermPayload;\n  }\n\n  @Override\n  public TermsEnum createTermsEnum(OptimizedMemoryIndex index) {\n    return new MPHTermsEnum(index);\n  }\n\n  public static class MPHTermsEnum extends BaseTermsEnum {\n    private int termID;\n    private final BytesRef bytesRef = new BytesRef();\n    private final OptimizedMemoryIndex index;\n\n    MPHTermsEnum(OptimizedMemoryIndex index) {\n      this.index = index;\n    }\n\n    @Override\n    public int docFreq() {\n      return index.getDF(termID);\n    }\n\n    @Override\n    public PostingsEnum postings(PostingsEnum reuse, int flags) throws IOException {\n      int postingsPointer = index.getPostingListPointer(termID);\n      int numPostings = index.getNumPostings(termID);\n      return index.getPostingLists().postings(postingsPointer, numPostings, flags);\n    }\n\n    @Override\n    public ImpactsEnum impacts(int flags) throws IOException {\n      return new SlowImpactsEnum(postings(null, flags));\n    }\n\n    @Override\n    public SeekStatus seekCeil(BytesRef text) throws IOException {\n      termID = index.lookupTerm(text);\n\n      if (termID == -1) {\n        return SeekStatus.END;\n      } else {\n        return SeekStatus.FOUND;\n      }\n    }\n\n    @Override\n    public BytesRef next() {\n      return null;\n    }\n\n    @Override\n    public long ord() {\n      return termID;\n    }\n\n    @Override\n    public void seekExact(long ord) {\n      if (ord < index.getNumTerms()) {\n        termID = (int) ord;\n        index.getTerm(termID, bytesRef, null);\n      }\n    }\n\n    @Override\n    public BytesRef term() {\n      return bytesRef;\n    }\n\n    @Override\n    public long totalTermFreq() {\n      return docFreq();\n    }\n  }\n\n  @SuppressWarnings(\"unchecked\")\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static class FlushHandler extends Flushable.Handler<MPHTermDictionary> {\n    static final String NUM_TERMS_PROP_NAME = \"numTerms\";\n    private final TermPointerEncoding termPointerEncoding;\n\n    public FlushHandler(TermPointerEncoding termPointerEncoding) {\n      super();\n      this.termPointerEncoding = termPointerEncoding;\n    }\n\n    public FlushHandler(MPHTermDictionary objectToFlush) {\n      super(objectToFlush);\n      this.termPointerEncoding = objectToFlush.termPointerEncoding;\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out)\n        throws IOException {\n      MPHTermDictionary objectToFlush = getObjectToFlush();\n      flushInfo.addIntProperty(NUM_TERMS_PROP_NAME, objectToFlush.getNumTerms());\n\n      out.writePackedInts(objectToFlush.termPointers);\n      objectToFlush.termPool.getFlushHandler().flush(flushInfo.newSubProperties(\"termPool\"), out);\n      objectToFlush.termsHashFunction.getFlushHandler()\n              .flush(flushInfo.newSubProperties(\"termsHashFunction\"), out);\n    }\n\n    @Override\n    protected MPHTermDictionary doLoad(FlushInfo flushInfo,\n        DataDeserializer in) throws IOException {\n      int numTerms = flushInfo.getIntProperty(NUM_TERMS_PROP_NAME);\n      PackedInts.Reader termPointers = in.readPackedInts();\n      ByteBlockPool termPool = (new ByteBlockPool.FlushHandler()).load(\n              flushInfo.getSubProperties(\"termPool\"), in);\n      BDZAlgorithm termsHashFunction = (new BDZAlgorithm.FlushHandler()).load(\n              flushInfo.getSubProperties(\"termsHashFunction\"), in);\n\n      return new MPHTermDictionary(numTerms, termsHashFunction, termPointers,\n              termPool, termPointerEncoding);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/MultiPostingLists.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.lucene.index.PostingsEnum;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\n\npublic class MultiPostingLists extends OptimizedPostingLists {\n\n  @VisibleForTesting\n  public static final int DEFAULT_DF_THRESHOLD = 1000;\n\n  private final OptimizedPostingLists lowDF;\n  private final OptimizedPostingLists highDF;\n\n  private final int dfThreshold;\n\n  /**\n   * Given the number of postings in each term (in this field), sum up the number of postings in\n   * the low df fields.\n   * @param numPostingsPerTerm number of postings in each term in this field.\n   * @param dfThreshold the low/high df threshold.\n   */\n  private static int numPostingsInLowDfTerms(int[] numPostingsPerTerm, int dfThreshold) {\n    int sumOfAllPostings = 0;\n    for (int numPostingsInATerm : numPostingsPerTerm) {\n      if (numPostingsInATerm < dfThreshold) {\n        sumOfAllPostings += numPostingsInATerm;\n      }\n    }\n    return sumOfAllPostings;\n  }\n\n  /**\n   * Creates a new posting list delegating to either lowDF or highDF posting list.\n   * @param omitPositions whether positions should be omitted or not.\n   * @param numPostingsPerTerm number of postings in each term in this field.\n   * @param maxPosition the largest position used in all the postings for this field.\n   */\n  public MultiPostingLists(\n      boolean omitPositions,\n      int[] numPostingsPerTerm,\n      int maxPosition) {\n    this(\n        new LowDFPackedIntsPostingLists(\n            omitPositions,\n            numPostingsInLowDfTerms(numPostingsPerTerm, DEFAULT_DF_THRESHOLD),\n            maxPosition),\n        new HighDFPackedIntsPostingLists(omitPositions),\n        DEFAULT_DF_THRESHOLD);\n  }\n\n  private MultiPostingLists(\n      OptimizedPostingLists lowDF,\n      OptimizedPostingLists highDF,\n      int dfThreshold) {\n    this.lowDF = lowDF;\n    this.highDF = highDF;\n    this.dfThreshold = dfThreshold;\n  }\n\n  @Override\n  public int copyPostingList(PostingsEnum postingsEnum, int numPostings)\n      throws IOException {\n    return numPostings < dfThreshold\n          ? lowDF.copyPostingList(postingsEnum, numPostings)\n          : highDF.copyPostingList(postingsEnum, numPostings);\n  }\n\n  @Override\n  public EarlybirdPostingsEnum postings(int postingsPointer, int numPostings, int flags)\n      throws IOException {\n    return numPostings < dfThreshold\n        ? lowDF.postings(postingsPointer, numPostings, flags)\n        : highDF.postings(postingsPointer, numPostings, flags);\n  }\n\n  @SuppressWarnings(\"unchecked\")\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  @VisibleForTesting\n  OptimizedPostingLists getLowDfPostingsList() {\n    return lowDF;\n  }\n\n  @VisibleForTesting\n  OptimizedPostingLists getHighDfPostingsList() {\n    return highDF;\n  }\n\n  public static class FlushHandler extends Flushable.Handler<MultiPostingLists> {\n    private static final String DF_THRESHOLD_PROP_NAME = \"dfThresHold\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(MultiPostingLists objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out)\n        throws IOException {\n      MultiPostingLists objectToFlush = getObjectToFlush();\n      flushInfo.addIntProperty(DF_THRESHOLD_PROP_NAME, objectToFlush.dfThreshold);\n      objectToFlush.lowDF.getFlushHandler().flush(\n              flushInfo.newSubProperties(\"lowDFPostinglists\"), out);\n      objectToFlush.highDF.getFlushHandler().flush(\n              flushInfo.newSubProperties(\"highDFPostinglists\"), out);\n    }\n\n    @Override\n    protected MultiPostingLists doLoad(FlushInfo flushInfo,\n        DataDeserializer in) throws IOException {\n      OptimizedPostingLists lowDF = new LowDFPackedIntsPostingLists.FlushHandler()\n            .load(flushInfo.getSubProperties(\"lowDFPostinglists\"), in);\n      OptimizedPostingLists highDF = new HighDFPackedIntsPostingLists.FlushHandler()\n          .load(flushInfo.getSubProperties(\"highDFPostinglists\"), in);\n      return new MultiPostingLists(\n          lowDF,\n          highDF,\n          flushInfo.getIntProperty(DF_THRESHOLD_PROP_NAME));\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/MultiSegmentTermDictionary.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport com.google.common.collect.ImmutableList;\n\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\n/**\n * A term dictionary that's backed by multiple underlying segments/indexes. For a given term, will\n * be able to return the termId for each of the underlying indexes.\n */\npublic interface MultiSegmentTermDictionary {\n\n  /**\n   * Lookup a term in this multi segment term dictionary, and return the term ids for that term on\n   * all of the managed segments.\n   *\n   * @return An array containing a termId for each segment that this term dictionary is backed by.\n   * The order of segments will match the order returned by {@link #getSegmentIndexes()}.\n   *\n   * For each segment, the term id will be returned, or\n   * {@link EarlybirdIndexSegmentAtomicReader#TERM_NOT_FOUND} if that segment does not have the\n   * given term.\n   */\n  int[] lookupTermIds(BytesRef term);\n\n  /**\n   * A convenience method for checking whether a specific index/segment is backed by this term\n   * dictionary. Returning true here is equivalent to returning:\n   * <pre>\n   * getSegmentIndexes().contains(invertedIndex);\n   * </pre>\n   */\n  default boolean supportSegmentIndex(InvertedIndex invertedIndex) {\n    return getSegmentIndexes().contains(invertedIndex);\n  }\n\n  /**\n   * The list of indexes that this term dictionary is backed by. The order of indexes here will\n   * be consistent with the order of termIds returned by {@link #lookupTermIds(BytesRef)}.\n   */\n  ImmutableList<? extends InvertedIndex> getSegmentIndexes();\n\n  /**\n   * Returns the number of terms in this term dictionary.\n   *\n   * If the term \"foo\" appears in segment A and in segment B, it will be counted once. To get the\n   * total number of terms across all managed segments, see {@link #getNumTermEntries()}.\n   */\n  int getNumTerms();\n\n  /**\n   * Returns the total number of terms in this term dictionary across all managed segments.\n   *\n   * If the term \"foo\" appears in segment A and in segment B, it will have 2 entries in this term\n   * dictionary.\n   */\n  int getNumTermEntries();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/MultiSegmentTermDictionaryWithFastutil.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.util.Arrays;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.OptionalInt;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Stopwatch;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.Maps;\n\nimport org.apache.lucene.util.BytesRef;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.util.LogFormatUtil;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\nimport it.unimi.dsi.fastutil.ints.IntArrayList;\n\n/**\n * This implementation took MultiSegmentTermDictionaryWithMap and replaced some of the\n * data structures with fastutil equivalents and it also uses a more memory efficient way to\n * store the precomputed data.\n *\n * This implementation has a requirement that each term per field needs to be present at\n * most once per document, since we only have space to index 2^24 terms and we have 2^23\n * documents as of now in realtime earlybirds.\n *\n * See UserIdMultiSegmentQuery class comment for more information on how this is used.\n */\npublic class MultiSegmentTermDictionaryWithFastutil implements MultiSegmentTermDictionary {\n  private static final Logger LOG = LoggerFactory.getLogger(\n      MultiSegmentTermDictionaryWithFastutil.class);\n\n  @VisibleForTesting\n  public static final SearchTimerStats TERM_DICTIONARY_CREATION_STATS =\n      SearchTimerStats.export(\"multi_segment_term_dictionary_with_fastutil_creation\",\n          TimeUnit.MILLISECONDS, false);\n\n  private static final int MAX_TERM_ID_BITS = 24;\n  private static final int TERM_ID_MASK = (1 << MAX_TERM_ID_BITS) - 1; // First 24 bits.\n  private static final int MAX_SEGMENT_SIZE = 1 << (MAX_TERM_ID_BITS - 1);\n\n  private final ImmutableList<OptimizedMemoryIndex> indexes;\n\n  // For each term, a list of (index id, term id) packed into an integer.\n  // The integer contains:\n  // byte 0: index (segment id). Since we have ~20 segments, this fits into a byte.\n  // bytes [1-3]: term id. The terms we're building this dictionary for are user ids\n  //   associated with a tweet - from_user_id and in_reply_to_user_id. Since we have\n  //   at most 2**23 tweets in realtime, we'll have at most 2**23 unique terms per\n  //   segments. The term ids post optimization are consecutive numbers, so they will\n  //   fit in 24 bits. We don't use the term dictionary in archive, which has more\n  //   tweets per segment.\n  //\n  //   To verify the maximum amount of tweets in a segment, see max_segment_size in\n  //   earlybird-config.yml.\n  private final HashMap<BytesRef, IntArrayList> termsMap;\n  private final int numTerms;\n  private final int numTermEntries;\n\n  int encodeIndexAndTermId(int indexId, int termId) {\n    // Push the index id to the left and use the other 24 bits for the term id.\n    return (indexId << MAX_TERM_ID_BITS) | termId;\n  }\n\n  void decodeIndexAndTermId(int[] arr, int packed) {\n    arr[packed >> MAX_TERM_ID_BITS] = packed & TERM_ID_MASK;\n  }\n\n\n  /**\n   * Creates a new multi-segment term dictionary backed by a regular java map.\n   */\n  public MultiSegmentTermDictionaryWithFastutil(\n      String field,\n      List<OptimizedMemoryIndex> indexes) {\n\n    this.indexes = ImmutableList.copyOf(indexes);\n\n    // Pre-size the map with estimate of max number of terms. It should be at least that big.\n    OptionalInt optionalMax = indexes.stream().mapToInt(OptimizedMemoryIndex::getNumTerms).max();\n    int maxNumTerms = optionalMax.orElse(0);\n    this.termsMap = Maps.newHashMapWithExpectedSize(maxNumTerms);\n\n    LOG.info(\"About to merge {} indexes for field {}, estimated {} terms\",\n        indexes.size(), field, LogFormatUtil.formatInt(maxNumTerms));\n    Stopwatch stopwatch = Stopwatch.createStarted();\n\n    BytesRef termBytesRef = new BytesRef();\n\n    for (int indexId = 0; indexId < indexes.size(); indexId++) {\n      // The inverted index for this field.\n      OptimizedMemoryIndex index = indexes.get(indexId);\n\n      int indexNumTerms = index.getNumTerms();\n\n      if (indexNumTerms > MAX_SEGMENT_SIZE) {\n        throw new IllegalStateException(\"too many terms: \" + indexNumTerms);\n      }\n\n      for (int termId = 0; termId < indexNumTerms; termId++) {\n        index.getTerm(termId, termBytesRef);\n\n        IntArrayList indexTerms = termsMap.get(termBytesRef);\n        if (indexTerms == null) {\n          BytesRef term = BytesRef.deepCopyOf(termBytesRef);\n\n          indexTerms = new IntArrayList();\n          termsMap.put(term, indexTerms);\n        }\n\n        indexTerms.add(encodeIndexAndTermId(indexId, termId));\n      }\n    }\n\n    this.numTerms = termsMap.size();\n    this.numTermEntries = indexes.stream().mapToInt(OptimizedMemoryIndex::getNumTerms).sum();\n\n    TERM_DICTIONARY_CREATION_STATS.timerIncrement(stopwatch.elapsed(TimeUnit.MILLISECONDS));\n    LOG.info(\"Done merging {} segments for field {} in {} - \"\n            + \"num terms: {}, num term entries: {}.\",\n        indexes.size(), field, stopwatch,\n        LogFormatUtil.formatInt(this.numTerms),\n        LogFormatUtil.formatInt(this.numTermEntries));\n  }\n\n  @Override\n  public int[] lookupTermIds(BytesRef term) {\n    int[] termIds = new int[indexes.size()];\n    Arrays.fill(termIds, EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND);\n\n    IntArrayList indexTerms = termsMap.get(term);\n    if (indexTerms != null) {\n      for (int i = 0; i < indexTerms.size(); i++) {\n        decodeIndexAndTermId(termIds, indexTerms.getInt(i));\n      }\n    }\n\n    return termIds;\n  }\n\n  @Override\n  public ImmutableList<? extends InvertedIndex> getSegmentIndexes() {\n    return indexes;\n  }\n\n  @Override\n  public int getNumTerms() {\n    return this.numTerms;\n  }\n\n  @Override\n  public int getNumTermEntries() {\n    return this.numTermEntries;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/MultiSegmentTermDictionaryWithMap.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.util.Arrays;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.OptionalInt;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.apache.lucene.util.BytesRef;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.util.LogFormatUtil;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\n/**\n * A rather simple implementation of a MultiSegmentTermDictionary that just keeps all terms in a\n * java hash map, and all the termIds for a term in a java list.\n *\n * An alternate implementation could have an MPH for the map, and a IntBlockPool for storing\n * the term ids.\n *\n * See UserIdMultiSegmentQuery class comment for more information on how this is used.\n */\npublic class MultiSegmentTermDictionaryWithMap implements MultiSegmentTermDictionary {\n  private static final Logger LOG = LoggerFactory.getLogger(\n      MultiSegmentTermDictionaryWithMap.class);\n\n  @VisibleForTesting\n  public static final SearchTimerStats TERM_DICTIONARY_CREATION_STATS =\n      SearchTimerStats.export(\"multi_segment_term_dictionary_with_map_creation\",\n          TimeUnit.MILLISECONDS, false);\n\n  private final ImmutableList<OptimizedMemoryIndex> indexes;\n  private final HashMap<BytesRef, List<IndexTerm>> termsMap;\n  private final int numTerms;\n  private final int numTermEntries;\n\n  private static class IndexTerm {\n    private int indexId;\n    private final int termId;\n\n    public IndexTerm(int indexId, int termId) {\n      this.indexId = indexId;\n      this.termId = termId;\n    }\n  }\n\n  /**\n   * Creates a new multi-segment term dictionary backed by a regular java map.\n   */\n  public MultiSegmentTermDictionaryWithMap(\n      String field,\n      List<OptimizedMemoryIndex> indexes) {\n\n    this.indexes = ImmutableList.copyOf(indexes);\n\n    // Pre-size the map with estimate of max number of terms. It should be at least that big.\n    OptionalInt optionalMax = indexes.stream().mapToInt(OptimizedMemoryIndex::getNumTerms).max();\n    int maxNumTerms = optionalMax.orElse(0);\n    this.termsMap = Maps.newHashMapWithExpectedSize(maxNumTerms);\n\n    LOG.info(\"About to merge {} indexes for field {}, estimated {} terms\",\n        indexes.size(), field, LogFormatUtil.formatInt(maxNumTerms));\n    long start = System.currentTimeMillis();\n\n    BytesRef termText = new BytesRef();\n    long copiedBytes = 0;\n    for (int indexId = 0; indexId < indexes.size(); indexId++) {\n      // The inverted index for this field.\n      OptimizedMemoryIndex index = indexes.get(indexId);\n\n      int indexNumTerms = index.getNumTerms();\n      for (int termId = 0; termId < indexNumTerms; termId++) {\n        index.getTerm(termId, termText);\n\n        // This copies the underlying array to a new array.\n        BytesRef term = BytesRef.deepCopyOf(termText);\n        copiedBytes += term.length;\n\n        List<IndexTerm> indexTerms = termsMap.computeIfAbsent(term, k -> Lists.newArrayList());\n\n        indexTerms.add(new IndexTerm(indexId, termId));\n      }\n    }\n\n    this.numTerms = termsMap.size();\n    this.numTermEntries = indexes.stream().mapToInt(OptimizedMemoryIndex::getNumTerms).sum();\n\n    long elapsed = System.currentTimeMillis() - start;\n    TERM_DICTIONARY_CREATION_STATS.timerIncrement(elapsed);\n    LOG.info(\"Done merging {} indexes for field {} in {}ms - \"\n      + \"num terms: {}, num term entries: {}, copied bytes: {}\",\n        indexes.size(), field, elapsed,\n        LogFormatUtil.formatInt(this.numTerms), LogFormatUtil.formatInt(this.numTermEntries),\n            LogFormatUtil.formatInt(copiedBytes));\n  }\n\n  @Override\n  public int[] lookupTermIds(BytesRef term) {\n    int[] termIds = new int[indexes.size()];\n    Arrays.fill(termIds, EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND);\n\n    List<IndexTerm> indexTerms = termsMap.get(term);\n    if (indexTerms != null) {\n      for (IndexTerm indexTerm : indexTerms) {\n        termIds[indexTerm.indexId] = indexTerm.termId;\n      }\n    }\n\n    return termIds;\n  }\n\n  @Override\n  public ImmutableList<? extends InvertedIndex> getSegmentIndexes() {\n    return indexes;\n  }\n\n  @Override\n  public int getNumTerms() {\n    return this.numTerms;\n  }\n\n  @Override\n  public int getNumTermEntries() {\n    return this.numTermEntries;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/OptimizedIndexTerms.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\n\npublic class OptimizedIndexTerms extends Terms {\n  private final OptimizedMemoryIndex index;\n\n  public OptimizedIndexTerms(OptimizedMemoryIndex index) {\n    this.index = index;\n  }\n\n  @Override\n  public long size() {\n    return index.getNumTerms();\n  }\n\n  @Override\n  public TermsEnum iterator() {\n    return index.createTermsEnum(index.getMaxPublishedPointer());\n  }\n\n  @Override\n  public long getSumTotalTermFreq() {\n    return index.getSumTotalTermFreq();\n  }\n\n  @Override\n  public long getSumDocFreq() {\n    return index.getSumTermDocFreq();\n  }\n\n  @Override\n  public int getDocCount() {\n    return index.getNumDocs();\n  }\n\n  @Override\n  public boolean hasFreqs() {\n    return false;\n  }\n\n  @Override\n  public boolean hasOffsets() {\n    return false;\n  }\n\n  @Override\n  public boolean hasPositions() {\n    return true;\n  }\n\n  @Override\n  public boolean hasPayloads() {\n    return false;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/OptimizedMemoryIndex.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\nimport java.util.Comparator;\nimport java.util.Map;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.util.BytesRef;\nimport org.apache.lucene.util.packed.PackedInts;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.util.hash.BDZAlgorithm;\nimport com.twitter.search.common.util.hash.BDZAlgorithm.MPHFNotFoundException;\nimport com.twitter.search.common.util.hash.KeysSource;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.facets.FacetIDMap.FacetField;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\npublic class OptimizedMemoryIndex extends InvertedIndex implements Flushable {\n  private static final Logger LOG = LoggerFactory.getLogger(OptimizedMemoryIndex.class);\n  private static final Comparator<BytesRef> BYTES_REF_COMPARATOR = Comparator.naturalOrder();\n\n  private static final SearchCounter MPH_NOT_FOUND_COUNT =\n      SearchCounter.export(\"twitter_optimized_index_mph_not_found_count\");\n\n  private final PackedInts.Reader numPostings;\n  private final PackedInts.Reader postingListPointers;\n  private final PackedInts.Reader offensiveCounters;\n  private final MultiPostingLists postingLists;\n\n  private final TermDictionary dictionary;\n\n  private final int numDocs;\n  private final int sumTotalTermFreq;\n  private final int sumTermDocFreq;\n\n  private OptimizedMemoryIndex(EarlybirdFieldType fieldType,\n                               int numDocs,\n                               int sumTermDocFreq,\n                               int sumTotalTermFreq,\n                               PackedInts.Reader numPostings,\n                               PackedInts.Reader postingListPointers,\n                               PackedInts.Reader offensiveCounters,\n                               MultiPostingLists postingLists,\n                               TermDictionary dictionary) {\n    super(fieldType);\n    this.numDocs = numDocs;\n    this.sumTermDocFreq = sumTermDocFreq;\n    this.sumTotalTermFreq = sumTotalTermFreq;\n    this.numPostings = numPostings;\n    this.postingListPointers = postingListPointers;\n    this.offensiveCounters = offensiveCounters;\n    this.postingLists = postingLists;\n    this.dictionary = dictionary;\n  }\n\n  public OptimizedMemoryIndex(\n      EarlybirdFieldType fieldType,\n      String field,\n      InvertedRealtimeIndex source,\n      Map<Integer, int[]> termIDMapper,\n      FacetField facetField,\n      DocIDToTweetIDMapper originalTweetIdMapper,\n      DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    super(fieldType);\n\n    numDocs = source.getNumDocs();\n    sumTermDocFreq = source.getSumTermDocFreq();\n    sumTotalTermFreq = source.getSumTotalTermFreq();\n\n    Preconditions.checkNotNull(originalTweetIdMapper, \"The segment must have a tweet ID mapper.\");\n    Preconditions.checkNotNull(optimizedTweetIdMapper,\n                               \"The optimized tweet ID mapper cannot be null.\");\n\n    // We rely on the fact that new terms always have a greater term ID. We ignore all terms that\n    // are equal to or greater than numTerms, as they may be incompletely applied. If new terms are\n    // added while optimizing, they will be re-added when we re-apply updates.\n    final KeysSource termsIterator = source.getKeysSource();\n    int numTerms = termsIterator.getNumberOfKeys();\n    int maxPublishedPointer = source.getMaxPublishedPointer();\n\n    int[] tempPostingListPointers = new int[numTerms];\n\n    BDZAlgorithm termsHashFunction = null;\n\n    final boolean supportTermTextLookup = facetField != null || fieldType.isSupportTermTextLookup();\n    if (supportTermTextLookup) {\n      try {\n        termsHashFunction = new BDZAlgorithm(termsIterator);\n      } catch (MPHFNotFoundException e) {\n        // we couldn't find a mphf for this field\n        // no problem, this can happen for very small fields\n        // - just use the fst in that case\n        LOG.warn(\"Unable to build MPH for field: {}\", field);\n        MPH_NOT_FOUND_COUNT.increment();\n      }\n    }\n\n    // Make sure to only call the expensive computeNumPostings() once.\n    int[] numPostingsSource = computeNumPostings(source, numTerms, maxPublishedPointer);\n\n    // The BDZ Algorithm returns a function from bytesref to term ID. However, these term IDs are\n    // different than the original term IDs (it's a hash function, not a hash _table_), so we have\n    // to remap the term IDs to match the ones generated by BDZ. We track that using the termIDMap.\n    int[] termIDMap = null;\n\n    if (termsHashFunction != null) {\n      termsIterator.rewind();\n      termIDMap = BDZAlgorithm.createIdMap(termsHashFunction, termsIterator);\n      if (facetField != null) {\n        termIDMapper.put(facetField.getFacetId(), termIDMap);\n      }\n\n      PackedInts.Reader termPointers = getPackedInts(source.getTermPointers(), termIDMap);\n      this.numPostings = getPackedInts(numPostingsSource, termIDMap);\n      this.offensiveCounters = source.getOffensiveCounters() == null ? null\n              : getPackedInts(source.getOffensiveCounters(), termIDMap);\n\n      this.dictionary = new MPHTermDictionary(\n          numTerms,\n          termsHashFunction,\n          termPointers,\n          source.getTermPool(),\n          TermPointerEncoding.DEFAULT_ENCODING);\n    } else {\n      this.dictionary = FSTTermDictionary.buildFST(\n          source.getTermPool(),\n          source.getTermPointers(),\n          numTerms,\n          BYTES_REF_COMPARATOR,\n          supportTermTextLookup,\n          TermPointerEncoding.DEFAULT_ENCODING);\n\n      this.numPostings = getPackedInts(numPostingsSource);\n      this.offensiveCounters = source.getOffensiveCounters() == null ? null\n              : getPackedInts(source.getOffensiveCounters());\n    }\n\n    TermsEnum allTerms = source.createTermsEnum(maxPublishedPointer);\n\n    this.postingLists = new MultiPostingLists(\n        !fieldType.hasPositions(),\n        numPostingsSource,\n        source.getMaxPosition());\n\n    for (int termID = 0; termID < numTerms; termID++) {\n      allTerms.seekExact(termID);\n      PostingsEnum postingsEnum = new OptimizingPostingsEnumWrapper(\n          allTerms.postings(null), originalTweetIdMapper, optimizedTweetIdMapper);\n      int mappedTermID = termIDMap != null ? termIDMap[termID] : termID;\n      tempPostingListPointers[mappedTermID] =\n          postingLists.copyPostingList(postingsEnum, numPostingsSource[termID]);\n    }\n\n    this.postingListPointers = getPackedInts(tempPostingListPointers);\n  }\n\n  private static int[] map(int[] source, int[] map) {\n    int[] target = new int[map.length];\n    for (int i = 0; i < map.length; i++) {\n      target[map[i]] = source[i];\n    }\n    return target;\n  }\n\n  static PackedInts.Reader getPackedInts(int[] values) {\n    return getPackedInts(values, null);\n  }\n\n  private static PackedInts.Reader getPackedInts(int[] values, int[] map) {\n    int[] mappedValues = values;\n    if (map != null) {\n      mappedValues = map(mappedValues, map);\n    }\n\n    // first determine max value\n    long maxValue = Long.MIN_VALUE;\n    for (int value : mappedValues) {\n      if (value > maxValue) {\n        maxValue = value;\n      }\n    }\n\n    PackedInts.Mutable packed =\n            PackedInts.getMutable(mappedValues.length, PackedInts.bitsRequired(maxValue),\n                    PackedInts.DEFAULT);\n    for (int i = 0; i < mappedValues.length; i++) {\n      packed.set(i, mappedValues[i]);\n    }\n\n    return packed;\n  }\n\n  /**\n   * Returns per-term array containing the number of posting in this index for each term.\n   * This call is extremely slow.\n   */\n  private static int[] computeNumPostings(\n      InvertedRealtimeIndex source,\n      int numTerms,\n      int maxPublishedPointer\n  ) throws IOException {\n    int[] numPostings = new int[numTerms];\n    TermsEnum allTerms = source.createTermsEnum(maxPublishedPointer);\n\n    for (int termID = 0; termID < numTerms; termID++) {\n      allTerms.seekExact(termID);\n      PostingsEnum docsEnum = allTerms.postings(null);\n      while (docsEnum.nextDoc() != DocIdSetIterator.NO_MORE_DOCS) {\n        numPostings[termID] += docsEnum.freq();\n      }\n    }\n\n    return numPostings;\n  }\n\n  @Override\n  public int getNumDocs() {\n    return numDocs;\n  }\n\n  @Override\n  public int getSumTotalTermFreq() {\n    return sumTotalTermFreq;\n  }\n\n  @Override\n  public int getSumTermDocFreq() {\n    return sumTermDocFreq;\n  }\n\n  public OptimizedPostingLists getPostingLists() {\n    Preconditions.checkState(hasPostingLists());\n    return postingLists;\n  }\n\n  int getPostingListPointer(int termID) {\n    Preconditions.checkState(hasPostingLists());\n    return (int) postingListPointers.get(termID);\n  }\n\n  int getNumPostings(int termID) {\n    Preconditions.checkState(hasPostingLists());\n    return (int) numPostings.get(termID);\n  }\n\n  public boolean getTerm(int termID, BytesRef text, BytesRef termPayload) {\n    return dictionary.getTerm(termID, text, termPayload);\n  }\n\n  @Override\n  public FacetLabelAccessor getLabelAccessor() {\n    return new FacetLabelAccessor() {\n      @Override\n      protected boolean seek(long termID) {\n        if (termID != EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND) {\n          hasTermPayload = getTerm((int) termID, termRef, termPayload);\n          offensiveCount = offensiveCounters != null\n                  ? (int) offensiveCounters.get((int) termID) : 0;\n          return true;\n        } else {\n          return false;\n        }\n      }\n    };\n  }\n\n  @Override\n  public Terms createTerms(int maxPublishedPointer) {\n    return new OptimizedIndexTerms(this);\n  }\n\n  @Override\n  public TermsEnum createTermsEnum(int maxPublishedPointer) {\n    return dictionary.createTermsEnum(this);\n  }\n\n  @Override\n  public int lookupTerm(BytesRef term) throws IOException {\n    return dictionary.lookupTerm(term);\n  }\n\n  @Override\n  public int getLargestDocIDForTerm(int termID) throws IOException {\n    Preconditions.checkState(hasPostingLists());\n    if (termID == EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND) {\n      return EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND;\n    } else {\n      return postingLists.getLargestDocID((int) postingListPointers.get(termID),\n              (int) numPostings.get(termID));\n    }\n  }\n\n  @Override\n  public int getDF(int termID) {\n    return (int) numPostings.get(termID);\n  }\n\n  @Override\n  public int getNumTerms() {\n    return dictionary.getNumTerms();\n  }\n\n  @Override\n  public void getTerm(int termID, BytesRef text) {\n    dictionary.getTerm(termID, text, null);\n  }\n\n  @VisibleForTesting TermDictionary getTermDictionary() {\n    return dictionary;\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public boolean hasPostingLists() {\n    return postingListPointers != null\n        && postingLists != null\n        && numPostings != null;\n  }\n\n  @VisibleForTesting\n  OptimizedPostingLists getOptimizedPostingLists() {\n    return postingLists;\n  }\n\n  public static class FlushHandler extends Flushable.Handler<OptimizedMemoryIndex> {\n    private static final String NUM_DOCS_PROP_NAME = \"numDocs\";\n    private static final String SUM_TOTAL_TERM_FREQ_PROP_NAME = \"sumTotalTermFreq\";\n    private static final String SUM_TERM_DOC_FREQ_PROP_NAME = \"sumTermDocFreq\";\n    private static final String USE_MIN_PERFECT_HASH_PROP_NAME = \"useMinimumPerfectHashFunction\";\n    private static final String SKIP_POSTING_LIST_PROP_NAME = \"skipPostingLists\";\n    private static final String HAS_OFFENSIVE_COUNTERS_PROP_NAME = \"hasOffensiveCounters\";\n    public static final String IS_OPTIMIZED_PROP_NAME = \"isOptimized\";\n\n    private final EarlybirdFieldType fieldType;\n\n    public FlushHandler(EarlybirdFieldType fieldType) {\n      super();\n      this.fieldType = fieldType;\n    }\n\n    public FlushHandler(OptimizedMemoryIndex objectToFlush) {\n      super(objectToFlush);\n      fieldType = objectToFlush.fieldType;\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      long startTime = getClock().nowMillis();\n      OptimizedMemoryIndex objectToFlush = getObjectToFlush();\n      boolean useHashFunction = objectToFlush.dictionary instanceof MPHTermDictionary;\n      boolean skipPostingLists = !objectToFlush.hasPostingLists();\n\n      flushInfo.addIntProperty(NUM_DOCS_PROP_NAME, objectToFlush.numDocs);\n      flushInfo.addIntProperty(SUM_TERM_DOC_FREQ_PROP_NAME, objectToFlush.sumTermDocFreq);\n      flushInfo.addIntProperty(SUM_TOTAL_TERM_FREQ_PROP_NAME, objectToFlush.sumTotalTermFreq);\n      flushInfo.addBooleanProperty(USE_MIN_PERFECT_HASH_PROP_NAME, useHashFunction);\n      flushInfo.addBooleanProperty(SKIP_POSTING_LIST_PROP_NAME, skipPostingLists);\n      flushInfo.addBooleanProperty(HAS_OFFENSIVE_COUNTERS_PROP_NAME,\n          objectToFlush.offensiveCounters != null);\n      flushInfo.addBooleanProperty(IS_OPTIMIZED_PROP_NAME, true);\n\n      if (!skipPostingLists) {\n        out.writePackedInts(objectToFlush.postingListPointers);\n        out.writePackedInts(objectToFlush.numPostings);\n      }\n      if (objectToFlush.offensiveCounters != null) {\n        out.writePackedInts(objectToFlush.offensiveCounters);\n      }\n\n      if (!skipPostingLists) {\n        objectToFlush.postingLists.getFlushHandler().flush(\n            flushInfo.newSubProperties(\"postingLists\"), out);\n      }\n      objectToFlush.dictionary.getFlushHandler().flush(flushInfo.newSubProperties(\"dictionary\"),\n              out);\n      getFlushTimerStats().timerIncrement(getClock().nowMillis() - startTime);\n    }\n\n    @Override\n    protected OptimizedMemoryIndex doLoad(\n        FlushInfo flushInfo, DataDeserializer in) throws IOException {\n      long startTime = getClock().nowMillis();\n      boolean useHashFunction = flushInfo.getBooleanProperty(USE_MIN_PERFECT_HASH_PROP_NAME);\n      boolean skipPostingLists = flushInfo.getBooleanProperty(SKIP_POSTING_LIST_PROP_NAME);\n\n      PackedInts.Reader postingListPointers = skipPostingLists ? null : in.readPackedInts();\n      PackedInts.Reader numPostings = skipPostingLists ? null : in.readPackedInts();\n      PackedInts.Reader offensiveCounters =\n              flushInfo.getBooleanProperty(HAS_OFFENSIVE_COUNTERS_PROP_NAME)\n                  ? in.readPackedInts() : null;\n\n      MultiPostingLists postingLists =  skipPostingLists ? null\n              : (new MultiPostingLists.FlushHandler())\n                      .load(flushInfo.getSubProperties(\"postingLists\"), in);\n\n      TermDictionary dictionary;\n      if (useHashFunction) {\n        dictionary = (new MPHTermDictionary.FlushHandler(TermPointerEncoding.DEFAULT_ENCODING))\n            .load(flushInfo.getSubProperties(\"dictionary\"), in);\n      } else {\n        dictionary = (new FSTTermDictionary.FlushHandler(TermPointerEncoding.DEFAULT_ENCODING))\n            .load(flushInfo.getSubProperties(\"dictionary\"), in);\n      }\n      getLoadTimerStats().timerIncrement(getClock().nowMillis() - startTime);\n\n      return new OptimizedMemoryIndex(fieldType,\n                                      flushInfo.getIntProperty(NUM_DOCS_PROP_NAME),\n                                      flushInfo.getIntProperty(SUM_TERM_DOC_FREQ_PROP_NAME),\n                                      flushInfo.getIntProperty(SUM_TOTAL_TERM_FREQ_PROP_NAME),\n                                      numPostings,\n                                      postingListPointers,\n                                      offensiveCounters,\n                                      postingLists,\n                                      dictionary);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/OptimizedPostingLists.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.PostingsEnum;\n\nimport com.twitter.search.common.util.io.flushable.Flushable;\n\npublic abstract class OptimizedPostingLists implements Flushable {\n  static final int MAX_DOC_ID_BIT = 24;\n  static final int MAX_DOC_ID = (1 << MAX_DOC_ID_BIT) - 1;\n\n  static final int MAX_POSITION_BIT = 31;\n\n  static final int MAX_FREQ_BIT = 31;\n\n  /**\n   * Copies the given posting list into these posting lists.\n   *\n   * @param postingsEnum enumerator of the posting list that needs to be copied\n   * @param numPostings number of postings in the posting list that needs to be copied\n   * @return position index of the head of the copied posting list in these posting lists instance\n   */\n  public abstract int copyPostingList(PostingsEnum postingsEnum, int numPostings)\n      throws IOException;\n\n  /**\n   * Create and return a postings doc enumerator or doc-position enumerator based on input flag.\n   *\n   * @see org.apache.lucene.index.PostingsEnum\n   */\n  public abstract EarlybirdPostingsEnum postings(int postingListPointer, int numPostings, int flags)\n      throws IOException;\n\n  /**\n   * Returns the largest docID contained in the posting list pointed by {@code postingListPointer}.\n   */\n  public final int getLargestDocID(int postingListPointer, int numPostings) throws IOException {\n    return postings(postingListPointer, numPostings, PostingsEnum.NONE).getLargestDocID();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/OptimizingPostingsEnumWrapper.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\n/**\n * A PostingsEnum that maps doc IDs in one DocIDToTweetIDMapper instance to doc IDs in another\n * DocIDToTweetIDMapper.\n *\n * Unoptimized segments can use any DocIDToTweetIDMapper they want, which means that there are no\n * guarantees on the distribution of the doc IDs in this mapper. However, optimized segments must\n * use an OptimizedTweetIDMapper: we want to assign sequential doc IDs and use delta encondings in\n * order to save space. So when an Earlybird segment needs to be optimized, we might need to convert\n * the doc ID space of the unoptimized tweet ID mapper to the doc ID space of the optimized mapper.\n * However, once we do this, the doc IDs stored in the posting lists in that segment will no longer\n * be valid, unless we remap them too. So the goal of this class is to provide a way to do that.\n *\n * When we want to optimize a posting list, we need to traverse it and pack it. This class provides\n * a wrapper around the original posting list that does the doc ID remapping at traversal time.\n */\npublic class OptimizingPostingsEnumWrapper extends PostingsEnum {\n  private final List<Integer> docIds = Lists.newArrayList();\n  private final Map<Integer, List<Integer>> positions = Maps.newHashMap();\n\n  private int docIdIndex = -1;\n  private int positionIndex = -1;\n\n  public OptimizingPostingsEnumWrapper(PostingsEnum source,\n                                       DocIDToTweetIDMapper originalTweetIdMapper,\n                                       DocIDToTweetIDMapper newTweetIdMapper) throws IOException {\n    int docId;\n    while ((docId = source.nextDoc()) != NO_MORE_DOCS) {\n      long tweetId = originalTweetIdMapper.getTweetID(docId);\n      int newDocId = newTweetIdMapper.getDocID(tweetId);\n      Preconditions.checkState(newDocId != DocIDToTweetIDMapper.ID_NOT_FOUND,\n          \"Did not find a mapping in the new tweet ID mapper for tweet ID %s, doc ID %s\",\n          tweetId, docId);\n\n      docIds.add(newDocId);\n      List<Integer> docPositions = Lists.newArrayListWithCapacity(source.freq());\n      positions.put(newDocId, docPositions);\n      for (int i = 0; i < source.freq(); ++i) {\n        docPositions.add(source.nextPosition());\n      }\n    }\n    Collections.sort(docIds);\n  }\n\n  @Override\n  public int nextDoc() {\n    ++docIdIndex;\n    if (docIdIndex >= docIds.size()) {\n      return NO_MORE_DOCS;\n    }\n\n    positionIndex = -1;\n    return docIds.get(docIdIndex);\n  }\n\n  @Override\n  public int freq() {\n    Preconditions.checkState(docIdIndex >= 0, \"freq() called before nextDoc().\");\n    Preconditions.checkState(docIdIndex < docIds.size(),\n                             \"freq() called after nextDoc() returned NO_MORE_DOCS.\");\n    return positions.get(docIds.get(docIdIndex)).size();\n  }\n\n  @Override\n  public int nextPosition() {\n    Preconditions.checkState(docIdIndex >= 0, \"nextPosition() called before nextDoc().\");\n    Preconditions.checkState(docIdIndex < docIds.size(),\n                             \"nextPosition() called after nextDoc() returned NO_MORE_DOCS.\");\n\n    ++positionIndex;\n    Preconditions.checkState(positionIndex < positions.get(docIds.get(docIdIndex)).size(),\n                             \"nextPosition() called more than freq() times.\");\n    return positions.get(docIds.get(docIdIndex)).get(positionIndex);\n  }\n\n  // All other methods are not supported.\n\n  @Override\n  public int advance(int target) {\n    throw new UnsupportedOperationException(\n        \"OptimizingPostingsEnumWrapper.advance() is not supported.\");\n  }\n\n  @Override\n  public long cost() {\n    throw new UnsupportedOperationException(\n        \"OptimizingPostingsEnumWrapper.cost() is not supported.\");\n  }\n\n  @Override\n  public int docID() {\n    throw new UnsupportedOperationException(\n        \"OptimizingPostingsEnumWrapper.docID() is not supported.\");\n  }\n\n  @Override\n  public int endOffset() {\n    throw new UnsupportedOperationException(\n        \"OptimizingPostingsEnumWrapper.endOffset() is not supported.\");\n  }\n\n  @Override\n  public BytesRef getPayload() {\n    throw new UnsupportedOperationException(\n        \"OptimizingPostingsEnumWrapper.getPayload() is not supported.\");\n  }\n\n  @Override\n  public int startOffset() {\n    throw new UnsupportedOperationException(\n        \"OptimizingPostingsEnumWrapper.startOffset() is not supported.\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/PackedLongsReaderPreComputedValues.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\n/**\n * Pre-computed shifts, mask, and start int indices used by\n * {@link IntBlockPoolPackedLongsReader} to decode packed values from\n * {@link IntBlockPool}.\n *\n * The purpose of this class is for decoding efficiency and speed. This class is thread-safe since\n * all its usages are read-only.\n *\n * Packed ints are stored from LOWEST bits for HIGHEST bits in an int.\n *\n * Here are 3 different situations when a packed value spans 1, 2, and 3 ints:\n *\n * - A packed value spans 1 int:\n *            [High Bits ................................. Low Bits]\n *   int[n] = [possible_other_data|packed_value|possible_other_data]\n *\n *   To decode, 1 shift right and 1 mask are needed:\n *     * shift - {@link #allLowBitsRightShift}\n *     * mask - dynamically computed based on bitsPerValue (in decoded slice).\n *\n * - A packed value spans 2 ints:\n *   The data is stored as:\n *              [High Bits .................. Low Bits]\n *   int[n]   = [low_bits_of_packed_value | other_data]\n *   int[n+1] = [other_data| high_bits_of_packed_value]\n *\n *   To decode, 1 shift right, 1 shift left, and 2 masks are needed:\n *     * 1 shift right {@link #allLowBitsRightShift} and 1 mask (computed on the fly) to compute\n *       low_bits_of_packed_value\n *     * 1 mask {@link #allMiddleBitsMask} and 1 shift left {@link #allMiddleBitsLeftShift} to\n *       compute high_bits_of_packed_value\n *     * 1 OR to combine `high_bits_of_packed_value | low_bits_of_packed_value`\n *\n * - A packed value spans 3 ints:\n *   The data is stored as:\n *              [High Bits .................. Low Bits]\n *   int[n]   = [low_bits_of_packed_value | other_data]\n *   int[n+1] = [ ... middle_bits_of_packed_value ... ]\n *   int[n+2] = [other_data| high_bits_of_packed_value]\n *\n *   To decode, 1 shift right, 2 shift left, and 3 masks are needed:\n *     * 1 shift right {@link #allLowBitsRightShift} and 1 mask (computed on the fly) to compute\n *       low_bits_of_packed_value\n *     * 1 shift left {@link #allMiddleBitsLeftShift} and 1 mask {@link #allMiddleBitsMask} to\n *       compute middle_bits_of_data\n *     * 1 shift left {@link #allHighBitsLeftShift} and 1 mask {@link #allHighBitsMask} to compute\n *       high_bits_of_data\n *     * 1 OR to combine `low_bits_of_data | middle_bits_of_data | high_bits_of_data`\n *\n * Example usage:\n * @see HighDFPackedIntsDocsEnum\n * @see HighDFPackedIntsDocsAndPositionsEnum\n */\npublic final class PackedLongsReaderPreComputedValues {\n  private final int[][] allLowBitsRightShift;\n  private final int[][] allMiddleBitsLeftShift;\n  private final int[][] allMiddleBitsMask;\n  private final int[][] allHighBitsLeftShift;\n  private final int[][] allHighBitsMask;\n\n  /**\n   * 2D int arrays containing pre-computed start int indices; the 2 dimensions are\n   * int[numBitsPerPackedValue][packedValueIndex].\n   *\n   * For a given number bits per packed value and a given packed value index, this is the first\n   * int in the subsequent of ints that contains the packed value with the given packed value index.\n   */\n  private final int[][] allStartIntIndices;\n\n  /**\n   * Sole constructor.\n   *\n   * @param maxBitsPerValue max possible number of bits of packed values that will be decoded\n   * @param maxNumValues max number of values are encoded back to back\n   * @param maxNumInts max number of ints are used to store packed values\n   * @param needStartIntIndex for optimization: whether start int indices are needed\n   */\n  PackedLongsReaderPreComputedValues(\n      int maxBitsPerValue,\n      int maxNumValues,\n      int maxNumInts,\n      boolean needStartIntIndex) {\n    assert maxBitsPerValue <= Long.SIZE;\n\n    if (needStartIntIndex) {\n      this.allStartIntIndices = new int[maxBitsPerValue + 1][maxNumValues];\n    } else {\n      this.allStartIntIndices = null;\n    }\n\n    this.allLowBitsRightShift = new int[maxBitsPerValue + 1][maxNumValues];\n    this.allMiddleBitsLeftShift = new int[maxBitsPerValue + 1][maxNumValues];\n    this.allMiddleBitsMask = new int[maxBitsPerValue + 1][maxNumValues];\n\n    // Packed value could use up 2 ints.\n    if (maxBitsPerValue > Integer.SIZE) {\n      this.allHighBitsLeftShift = new int[maxBitsPerValue + 1][maxNumValues];\n      this.allHighBitsMask = new int[maxBitsPerValue + 1][maxNumValues];\n    } else {\n      this.allHighBitsLeftShift = null;\n      this.allHighBitsMask = null;\n    }\n\n    compute(maxBitsPerValue, maxNumValues, maxNumInts);\n  }\n\n  /**\n   * Compute masks, shifts and start indices.\n   */\n  private void compute(int maxBitsPerValue, int maxNumValues, int maxNumInts) {\n    // For each possible bits per packed value.\n    for (int bitsPerPackedValue = 0; bitsPerPackedValue <= maxBitsPerValue; bitsPerPackedValue++) {\n      int[] startIntIndices =\n          allStartIntIndices != null ? allStartIntIndices[bitsPerPackedValue] : null;\n      int[] lowBitsRightShift =\n          allLowBitsRightShift[bitsPerPackedValue];\n      int[] middleBitsLeftShift =\n          allMiddleBitsLeftShift[bitsPerPackedValue];\n      int[] middleBitsMask =\n          allMiddleBitsMask[bitsPerPackedValue];\n      int[] highBitsLeftShift =\n          allHighBitsLeftShift != null ? allHighBitsLeftShift[bitsPerPackedValue] : null;\n      int[] highBitsMask =\n          allHighBitsMask != null ? allHighBitsMask[bitsPerPackedValue] : null;\n\n      int shift = 0;\n      int currentIntIndex = 0;\n      int bitsRead;\n      int bitsRemaining;\n\n      // For each packed value.\n      for (int packedValueIndex = 0; packedValueIndex < maxNumValues; packedValueIndex++) {\n        if (startIntIndices != null) {\n          startIntIndices[packedValueIndex] = currentIntIndex;\n        }\n        // Packed value spans to the 1st int.\n        lowBitsRightShift[packedValueIndex] = shift;\n        bitsRead = Integer.SIZE - shift;\n        bitsRemaining = bitsPerPackedValue - bitsRead;\n\n        if (bitsRemaining >= 0) {\n          // Packed value spans to the 2nd int.\n          currentIntIndex++;\n          if (currentIntIndex == maxNumInts) {\n            break;\n          }\n          middleBitsLeftShift[packedValueIndex] = bitsRead;\n          middleBitsMask[packedValueIndex] =\n              bitsRemaining >= Integer.SIZE ? 0xFFFFFFFF : (1 << bitsRemaining) - 1;\n\n          // Packed value spans to the 3rd int.\n          bitsRead += Integer.SIZE;\n          bitsRemaining -= Integer.SIZE;\n          if (bitsRemaining >= 0) {\n            currentIntIndex++;\n            if (currentIntIndex == maxNumInts) {\n              break;\n            }\n            assert highBitsLeftShift != null;\n            assert highBitsMask != null;\n            highBitsLeftShift[packedValueIndex] = bitsRead;\n            highBitsMask[packedValueIndex] =\n                bitsRemaining >= Integer.SIZE ? 0xFFFFFFFF : (1 << bitsRemaining) - 1;\n          }\n        }\n\n        shift += bitsPerPackedValue;\n        shift = shift % Integer.SIZE;\n      }\n    }\n  }\n\n  /********************************************************************\n   * Getters of Pre-computed Values: returns should NEVER be modified *\n   ********************************************************************/\n\n  int[] getStartIntIndices(int numBitsPerValue) {\n    return allStartIntIndices == null ? null : allStartIntIndices[numBitsPerValue];\n  }\n\n  int[] getLowBitsRightShift(int numBitsPerValue) {\n    return allLowBitsRightShift[numBitsPerValue];\n  }\n\n  int[] getMiddleBitsLeftShift(int numBitsPerValue) {\n    return allMiddleBitsLeftShift[numBitsPerValue];\n  }\n\n  int[] getMiddleBitsMask(int numBitsPerValue) {\n    return allMiddleBitsMask[numBitsPerValue];\n  }\n\n  int[] getHighBitsLeftShift(int numBitsPerValue) {\n    return allHighBitsLeftShift == null ? null : allHighBitsLeftShift[numBitsPerValue];\n  }\n\n  int[] getHighBitsMask(int numBitsPerValue) {\n    return allHighBitsMask == null ? null : allHighBitsMask[numBitsPerValue];\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/PayloadUtil.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport org.apache.lucene.util.BytesRef;\n\n/**\n * Utilities for encoding and decoding BytesRefs into ints. The encoding is:\n * [0..n] n bytes big-endian decoded into integers.\n * n: number of bytes.\n *\n * Example:\n * encode([DE, AD, BE, EF, AB]) => [0xDEADBEEF, 0xAB000000, 5]\n *\n * It's necessary to store the length at the end instead of the start so that we can know how far to\n * jump backward from a skiplist entry. We can't store it after the skip list entry because there\n * can be a variable number of pointers after the skip list entry.\n *\n * An example skip list entry, with labels on the following line:\n * [0xDEADBEEF,       12,   654,         0x877,       0x78879]\n * [   payload, position, docID, level0Pointer, level1Pointer]\n */\npublic final class PayloadUtil {\n  private PayloadUtil() {\n  }\n\n  public static final int[] EMPTY_PAYLOAD = new int[]{0};\n\n  /**\n   * Encodes a {@link BytesRef} into an int array (to be inserted into a\n   * {@link IntBlockPool}. The encoder considers the input to be big-endian encoded ints.\n   */\n  public static int[] encodePayload(BytesRef payload) {\n    if (payload == null) {\n      return EMPTY_PAYLOAD;\n    }\n\n    int intsInPayload = intsForBytes(payload.length);\n\n    int[] arr = new int[1 + intsInPayload];\n\n    for (int i = 0; i < intsInPayload; i++) {\n      int n = 0;\n      for (int j = 0; j < 4; j++) {\n        int index = i * 4 + j;\n        int b;\n        if (index < payload.length) {\n          // mask off the top bits in case b is negative.\n          b = payload.bytes[index] & 0xFF;\n        } else {\n          b = 0;\n        }\n        n = n << 8 | b;\n      }\n\n      arr[i] = n;\n    }\n\n    arr[intsInPayload] = payload.length;\n\n    return arr;\n  }\n\n  /**\n   * Decodes a {@link IntBlockPool} and position into a {@link BytesRef}. The ints are\n   * converted into big-endian encoded bytes.\n   */\n  public static BytesRef decodePayload(\n      IntBlockPool b,\n      int pointer) {\n    int length = b.get(pointer);\n    BytesRef bytesRef = new BytesRef(length);\n    bytesRef.length = length;\n\n    int numInts = intsForBytes(length);\n\n    for (int i = 0; i < numInts; i++) {\n      int n = b.get(pointer - numInts + i);\n      for (int j = 0; j < 4; j++) {\n        int byteIndex = 4 * i + j;\n        if (byteIndex < length) {\n          bytesRef.bytes[byteIndex] = (byte) (n >> 8 * (3 - byteIndex % 4));\n        }\n      }\n    }\n\n    return bytesRef;\n  }\n\n  private static int intsForBytes(int byteCount) {\n    return (byteCount + 3) / 4;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/PostingsBufferQueue.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.util.NoSuchElementException;\n\nimport com.google.common.annotations.VisibleForTesting;\n\n/**\n * A posting buffer used by {@link HighDFPackedIntsPostingLists} while copying over posting list.\n */\nfinal class PostingsBufferQueue {\n  /**\n   * Mask used to convert an int to a long. We cannot just cast because doing so  will fill in the\n   * higher 32 bits with the sign bit, but we need the higher 32 bits to be 0 instead.\n   */\n  static final long LONG_MASK = (1L << 32) - 1;\n\n  /**\n   * A circular FIFO long queue used internally to store posting.\n   * @see #postingsQueue\n   */\n  @VisibleForTesting\n  static final class Queue {\n    private final long[] queue;\n    private int head = 0;\n    private int tail = 0;\n    private int size;\n\n    Queue(int maxSize) {\n      this.queue = new long[maxSize < 2 ? 2 : maxSize];\n    }\n\n    boolean isEmpty() {\n      return size() == 0;\n    }\n\n    boolean isFull() {\n      return size() == queue.length;\n    }\n\n    void offer(long value) {\n      if (size() == queue.length) {\n        throw new IllegalStateException(\"Queue is full\");\n      }\n      queue[tail] = value;\n      tail = (tail + 1) % queue.length;\n      size++;\n    }\n\n    long poll() {\n      if (isEmpty()) {\n        throw new NoSuchElementException(\"Queue is empty.\");\n      }\n      long value = queue[head];\n      head = (head + 1) % queue.length;\n      size--;\n      return value;\n    }\n\n    int size() {\n      return size;\n    }\n  }\n\n  /**\n   * Internal posting queue.\n   */\n  private final Queue postingsQueue;\n\n  /**\n   * Constructor with max size.\n   *\n   * @param maxSize max size of this buffer.\n   */\n  PostingsBufferQueue(int maxSize) {\n    this.postingsQueue = new Queue(maxSize);\n  }\n\n  /**\n   * Check if the buffer is empty.\n   *\n   * @return If this buffer is empty\n   */\n  boolean isEmpty() {\n    return postingsQueue.isEmpty();\n  }\n\n  /**\n   * Check if the buffer is full.\n   *\n   * @return If this buffer is full\n   */\n  boolean isFull() {\n    return postingsQueue.isFull();\n  }\n\n  /**\n   * Get the current size of this buffer.\n   *\n   * @return Current size of this buffer\n   */\n  int size() {\n    return postingsQueue.size();\n  }\n\n  /**\n   * Store a posting with docID and a second value that could be freq, position, or any additional\n   * info. This method will encode the offered doc ID and second value with\n   * {@link #encodePosting(int, int)}.\n   *\n   * @param docID doc ID of the posting\n   * @param secondValue an additional value of the posting\n   */\n  void offer(int docID, int secondValue) {\n    postingsQueue.offer(encodePosting(docID, secondValue));\n  }\n\n  /**\n   * Remove and return the earliest inserted posting, this is a FIFO queue.\n   *\n   * @return the earliest inserted posting.\n   */\n  long poll() {\n    return postingsQueue.poll();\n  }\n\n  /**\n   * Encode a doc ID and a second value, both are ints, into a long. The higher 32 bits store the\n   * doc ID and lower 32 bits store the second value.\n   *\n   * @param docID an int specifying doc ID of the posting\n   * @param secondValue an int specifying the second value of the posting\n   * @return an encoded long represent the posting\n   */\n  private static long encodePosting(int docID, int secondValue) {\n    return ((LONG_MASK & docID) << 32) | (LONG_MASK & secondValue);\n  }\n\n  /**\n   * Decode doc ID from the given posting.\n   * @param posting a given posting encoded with {@link #encodePosting(int, int)}\n   * @return the doc ID of the given posting.\n   */\n  static int getDocID(long posting) {\n    return (int) (posting >> 32);\n  }\n\n  /**\n   * Decode the second value from the given posting.\n   * @param posting a given posting encoded with {@link #encodePosting(int, int)}\n   * @return the second value of the given posting.\n   */\n  static int getSecondValue(long posting) {\n    return (int) posting;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/QueryCostTracker.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport org.apache.lucene.util.CloseableThreadLocal;\n\nimport com.twitter.search.common.search.QueryCostProvider;\n\npublic class QueryCostTracker implements QueryCostProvider {\n  public static enum CostType {\n    // For the realtime segment we track how many posting list blocks\n    // are accessed during the lifetime of one query.\n    LOAD_REALTIME_POSTING_BLOCK(1),\n\n    // Number of optimized posting list blocks\n    LOAD_OPTIMIZED_POSTING_BLOCK(1);\n\n    private final double cost;\n\n    private CostType(double cost) {\n      this.cost = cost;\n    }\n  }\n\n  private static final CloseableThreadLocal<QueryCostTracker> TRACKERS\n      = new CloseableThreadLocal<QueryCostTracker>() {\n    @Override protected QueryCostTracker initialValue() {\n      return new QueryCostTracker();\n    }\n  };\n\n  public static QueryCostTracker getTracker() {\n    return TRACKERS.get();\n  }\n\n  private double totalCost;\n\n  public void track(CostType costType) {\n    totalCost += costType.cost;\n  }\n\n  public void reset() {\n    totalCost = 0;\n  }\n\n  @Override\n  public double getTotalCost() {\n    return totalCost;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/RealtimeIndexTerms.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.util.Iterator;\nimport java.util.TreeSet;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.BaseTermsEnum;\nimport org.apache.lucene.index.ImpactsEnum;\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.index.SlowImpactsEnum;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.hashtable.HashTable;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.util.hash.KeysSource;\n\npublic class RealtimeIndexTerms extends Terms {\n  // Calling InMemoryTermsEnum.next() creates a full copy of the entire term dictionary, and can\n  // be quite expensive. We don't expect these calls to happen, and they shpould not happen on the\n  // regular read path. We stat them here just in case to see if there is any unexpected usage.\n  private static final SearchCounter TERMS_ENUM_NEXT_CALLS =\n      SearchCounter.export(\"in_memory_terms_enum_next_calls\");\n  private static final SearchCounter TERMS_ENUM_CREATE_TERM_SET =\n      SearchCounter.export(\"in_memory_terms_enum_next_create_term_set\");\n  private static final SearchCounter TERMS_ENUM_CREATE_TERM_SET_SIZE =\n      SearchCounter.export(\"in_memory_terms_enum_next_create_term_set_size\");\n\n  private final InvertedRealtimeIndex index;\n  private final int maxPublishedPointer;\n\n  public RealtimeIndexTerms(InvertedRealtimeIndex index, int maxPublishedPointer) {\n    this.index = index;\n    this.maxPublishedPointer = maxPublishedPointer;\n  }\n\n  @Override\n  public long size() {\n    return index.getNumTerms();\n  }\n\n  @Override\n  public TermsEnum iterator() {\n    return index.createTermsEnum(maxPublishedPointer);\n  }\n\n  /**\n   * This TermsEnum use a tree set to support {@link TermsEnum#next()} method. However, this is not\n   * efficient enough to support realtime operation. {@link TermsEnum#seekCeil} is not fully\n   * supported in this termEnum.\n   */\n  public static class InMemoryTermsEnum extends BaseTermsEnum {\n    private final InvertedRealtimeIndex index;\n    private final int maxPublishedPointer;\n    private int termID = -1;\n    private BytesRef bytesRef = new BytesRef();\n    private Iterator<BytesRef> termIter;\n    private TreeSet<BytesRef> termSet;\n\n    public InMemoryTermsEnum(InvertedRealtimeIndex index, int maxPublishedPointer) {\n      this.index = index;\n      this.maxPublishedPointer = maxPublishedPointer;\n      termIter = null;\n    }\n\n    @Override\n    public int docFreq() {\n      return index.getDF(termID);\n    }\n\n    @Override\n    public PostingsEnum postings(PostingsEnum reuse, int flags) {\n      int postingsPointer = index.getPostingListPointer(termID);\n      return index.getPostingList().postings(postingsPointer, docFreq(), maxPublishedPointer);\n    }\n\n    @Override\n    public ImpactsEnum impacts(int flags) {\n      return new SlowImpactsEnum(postings(null, flags));\n    }\n\n    @Override\n    public SeekStatus seekCeil(BytesRef text) {\n      // Nullify termIter.\n      termIter = null;\n\n      termID = index.lookupTerm(text);\n\n      if (termID == -1) {\n        return SeekStatus.END;\n      } else {\n        index.getTerm(termID, bytesRef);\n        return SeekStatus.FOUND;\n      }\n    }\n\n    @Override\n    public BytesRef next() {\n      TERMS_ENUM_NEXT_CALLS.increment();\n      if (termSet == null) {\n        termSet = new TreeSet<>();\n        KeysSource keysource = index.getKeysSource();\n        keysource.rewind();\n        int numTerms = keysource.getNumberOfKeys();\n        for (int i = 0; i < numTerms; ++i) {\n          BytesRef ref = keysource.nextKey();\n          // we need to clone the ref since the keysource is reusing the returned BytesRef\n          // instance and we are storing it\n          termSet.add(ref.clone());\n        }\n        TERMS_ENUM_CREATE_TERM_SET.increment();\n        TERMS_ENUM_CREATE_TERM_SET_SIZE.add(numTerms);\n      }\n\n      // Construct termIter from the subset.\n      if (termIter == null) {\n        termIter = termSet.tailSet(bytesRef, true).iterator();\n      }\n\n      if (termIter.hasNext()) {\n        bytesRef = termIter.next();\n        termID = index.lookupTerm(bytesRef);\n      } else {\n        termID = -1;\n        bytesRef = null;\n      }\n      return bytesRef;\n    }\n\n    @Override\n    public long ord() {\n      return termID;\n    }\n\n    @Override\n    public void seekExact(long ord) {\n      // Nullify termIter.\n      termIter = null;\n\n      if (ord < index.getNumTerms()) {\n        termID = (int) ord;\n        index.getTerm(termID, bytesRef);\n      }\n    }\n\n    @Override\n    public BytesRef term() {\n      return bytesRef;\n    }\n\n    @Override\n    public long totalTermFreq() {\n      return docFreq();\n    }\n  }\n\n  /**\n   * This TermsEnum use a {@link SkipListContainer} backed termsSkipList provided by\n   * {@link InvertedRealtimeIndex} to supported ordered terms operations like\n   * {@link TermsEnum#next()} and {@link TermsEnum#seekCeil}.\n   */\n  public static class SkipListInMemoryTermsEnum extends BaseTermsEnum {\n    private final InvertedRealtimeIndex index;\n\n    private int termID = -1;\n    private BytesRef bytesRef = new BytesRef();\n    private int nextTermIDPointer;\n\n    /**\n     * {@link #nextTermIDPointer} is used to record pointer to next termsID to accelerate\n     * {@link #next}. However, {@link #seekCeil} and {@link #seekExact} may jump to an arbitrary\n     * term so the {@link #nextTermIDPointer} may not be correct, and this flag is used to check if\n     * this happens. If this flag is false, {@link #correctNextTermIDPointer} should be called to\n     * correct the value.\n     */\n    private boolean isNextTermIDPointerCorrect;\n\n    private final SkipListContainer<BytesRef> termsSkipList;\n    private final InvertedRealtimeIndex.TermsSkipListComparator termsSkipListComparator;\n    private final int maxPublishedPointer;\n\n    /**\n     * Creates a new {@link TermsEnum} for a skip list-based sorted real-time term dictionary.\n     */\n    public SkipListInMemoryTermsEnum(InvertedRealtimeIndex index, int maxPublishedPointer) {\n      Preconditions.checkNotNull(index.getTermsSkipList());\n\n      this.index = index;\n      this.termsSkipList = index.getTermsSkipList();\n\n      // Each Terms Enum shall have their own comparators to be thread safe.\n      this.termsSkipListComparator =\n          new InvertedRealtimeIndex.TermsSkipListComparator(index);\n      this.nextTermIDPointer =\n          termsSkipList.getNextPointer(SkipListContainer.FIRST_LIST_HEAD);\n      this.isNextTermIDPointerCorrect = true;\n      this.maxPublishedPointer = maxPublishedPointer;\n    }\n\n    @Override\n    public int docFreq() {\n      return index.getDF(termID);\n    }\n\n    @Override\n    public PostingsEnum postings(PostingsEnum reuse, int flags) {\n      int postingsPointer = index.getPostingListPointer(termID);\n      return index.getPostingList().postings(postingsPointer, docFreq(), maxPublishedPointer);\n    }\n\n    @Override\n    public ImpactsEnum impacts(int flags) {\n      return new SlowImpactsEnum(postings(null, flags));\n    }\n\n    @Override\n    public SeekStatus seekCeil(BytesRef text) {\n      // Next term pointer is not correct anymore since seek ceil\n      //   will jump to an arbitrary term.\n      isNextTermIDPointerCorrect = false;\n\n      // Doing precise lookup first.\n      termID = index.lookupTerm(text);\n\n      // Doing ceil lookup if not found, otherwise we are good.\n      if (termID == -1) {\n        return seekCeilWithSkipList(text);\n      } else {\n        index.getTerm(termID, bytesRef);\n        return SeekStatus.FOUND;\n      }\n    }\n\n    /**\n     * Doing ceil terms search with terms skip list.\n     */\n    private SeekStatus seekCeilWithSkipList(BytesRef text) {\n      int termIDPointer = termsSkipList.searchCeil(text,\n          SkipListContainer.FIRST_LIST_HEAD,\n          termsSkipListComparator,\n          null);\n\n      // End reached but still cannot found a ceil term.\n      if (termIDPointer == SkipListContainer.FIRST_LIST_HEAD) {\n        termID = HashTable.EMPTY_SLOT;\n        return SeekStatus.END;\n      }\n\n      termID = termsSkipList.getValue(termIDPointer);\n\n      // Set next termID pointer and is correct flag.\n      nextTermIDPointer = termsSkipList.getNextPointer(termIDPointer);\n      isNextTermIDPointerCorrect = true;\n\n      // Found a ceil term but not the precise match.\n      index.getTerm(termID, bytesRef);\n      return SeekStatus.NOT_FOUND;\n    }\n\n    /**\n     * {@link #nextTermIDPointer} is used to record the pointer to next termID. This method is used\n     * to correct {@link #nextTermIDPointer} to correct value after {@link #seekCeil} or\n     * {@link #seekExact} dropped current term to arbitrary point.\n     */\n    private void correctNextTermIDPointer() {\n      final int curTermIDPointer = termsSkipList.search(\n          bytesRef,\n          SkipListContainer.FIRST_LIST_HEAD,\n          termsSkipListComparator,\n          null);\n      // Must be able to find the exact term.\n      assert termID == HashTable.EMPTY_SLOT\n          || termID == termsSkipList.getValue(curTermIDPointer);\n\n      nextTermIDPointer = termsSkipList.getNextPointer(curTermIDPointer);\n      isNextTermIDPointerCorrect = true;\n    }\n\n    @Override\n    public BytesRef next() {\n      // Correct nextTermIDPointer first if not correct due to seekExact or seekCeil.\n      if (!isNextTermIDPointerCorrect) {\n        correctNextTermIDPointer();\n      }\n\n      // Skip list is exhausted.\n      if (nextTermIDPointer == SkipListContainer.FIRST_LIST_HEAD) {\n        termID = HashTable.EMPTY_SLOT;\n        return null;\n      }\n\n      termID = termsSkipList.getValue(nextTermIDPointer);\n\n      index.getTerm(termID, bytesRef);\n\n      // Set next termID Pointer.\n      nextTermIDPointer = termsSkipList.getNextPointer(nextTermIDPointer);\n      return bytesRef;\n    }\n\n    @Override\n    public long ord() {\n      return termID;\n    }\n\n    @Override\n    public void seekExact(long ord) {\n      if (ord < index.getNumTerms()) {\n        termID = (int) ord;\n        index.getTerm(termID, bytesRef);\n\n        // Next term pointer is not correct anymore since seek exact\n        //   just jump to an arbitrary term.\n        isNextTermIDPointerCorrect = false;\n      }\n    }\n\n    @Override\n    public BytesRef term() {\n      return bytesRef;\n    }\n\n    @Override\n    public long totalTermFreq() {\n      return docFreq();\n    }\n  }\n\n  @Override\n  public long getSumTotalTermFreq() {\n    return index.getSumTotalTermFreq();\n  }\n\n  @Override\n  public long getSumDocFreq() {\n    return index.getSumTermDocFreq();\n  }\n\n  @Override\n  public int getDocCount() {\n    return index.getNumDocs();\n  }\n\n  @Override\n  public boolean hasFreqs() {\n    return true;\n  }\n\n  @Override\n  public boolean hasOffsets() {\n    return false;\n  }\n\n  @Override\n  public boolean hasPositions() {\n    return true;\n  }\n\n  @Override\n  public boolean hasPayloads() {\n    return true;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/SkipListComparator.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\n/**\n * Comparator interface for {@link SkipListContainer},\n * see sample implementation {@link SkipListIntegerComparator}.\n *\n * Notice: less/equal/greater here refer to the order precedence, instead of numerical value.\n */\npublic interface SkipListComparator<K> {\n\n  /**\n   * Determine the order between the given key and the key of the given targetValue.\n   * Notice, usually key of a value could be derived from the value along.\n   *\n   * Implementation of this method should consider sentinel value, see {@link #getSentinelValue()}.\n   *\n   * Can include position data (primarily for text posting lists). Position should be ignored if\n   * the skip list was constructed without positions enabled.\n   *\n   * @return negative, zero, or positive to indicate if first value is\n   *         less than, equal to, or greater than the second value, respectively.\n   */\n  int compareKeyWithValue(K key, int targetValue, int targetPosition);\n\n  /**\n   * Determine the order of two given values based on their keys.\n   * Notice, usually key of a value could be derived from the value along.\n   *\n   * Implementation of this method should consider sentinel value, see {@link #getSentinelValue()}.\n   *\n   * @return negative, zero, or positive to indicate if first value is\n   *         less than, equal to, or greater than the second value, respectively.\n   */\n  int compareValues(int v1, int v2);\n\n  /**\n   * Return a sentinel value, sentinel value should be considered by this comparator\n   * as an ADVISORY GREATEST value, which should NOT be actually inserted into the skip list.\n   *\n   * @return the sentinel value.\n   */\n  int getSentinelValue();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/SkipListContainer.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\nimport java.util.Random;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\n\nimport static com.twitter.search.core.earlybird.index.inverted.PayloadUtil.EMPTY_PAYLOAD;\n\n/**\n * This is a skip list container implementation backed by {@link IntBlockPool}.\n *\n * Skip list is a data structure similar to linked list, but with a hierarchy of lists\n * each skipping over fewer elements, and the bottom hierarchy does NOT skip any elements.\n * @see <a href=\"http://en.wikipedia.org/wiki/Skip_list\">Skip List Wikipedia</a>\n *\n * This implementation is lock free and thread safe with ONE writer thread and MULTIPLE reader\n * threads.\n *\n * This implementation could contain one or more skip lists, and they are all backed by\n * the same {@link IntBlockPool}.\n *\n * Values are actually stored as integers; however search key is implemented as a generic type.\n * Inserts of values that already exist are stored as subsequent elements. This is used to support\n * positions and term frequency.\n *\n * Also reserve the integer after value to store next ordinal pointer information. We avoid storing\n * pointers to the next element in the tower by allocating them contiguously. To descend the tower,\n * we just increment the pointer.\n *\n * This skip list can also store positions as integers. It allocates them before it allocates the\n * value (the value is a doc ID if we are using positions). This means that we can access the\n * position by simply decrementing the value pointer.\n *\n * To understand how the skip list works, first understand how insert works, then the rest will be\n * more comprehendable.\n *\n * A skip list will be implemented in a circle linked way:\n *   - the list head node will have the sentinel value, which is the advisory greatest value\n *     provided by comparator.\n *   - Real first value will be pointed by the list head node.\n *   - Real last value will point to the list head.\n *\n * Constraints:\n *   - Does NOT support negative value.\n *\n * Simple Viz:\n *\n * Empty list with max tower height 5. S = Sentinel value, I = Initial value.\n *    | s| 0| 0| 0| 0| 0| i| i| i| i| i| i| i| i| i| i|\n *\n * One possible situation after inserting 4, 6, 5.\n *    | s| 6| 6| 9| 0| 0| 4|13|13| 6| 0| 0| 0| 5| 9| 9|\n */\npublic class SkipListContainer<K> implements Flushable {\n  /**\n   * The list head of first skip list in the container, this is for convenient usage,\n   * so application use only one skip list does not need to keep track of the list head.\n   */\n  static final int FIRST_LIST_HEAD = 0;\n\n  /**\n   * Initial value used when initialize int block pool. Notice -1 is not used here in order to give\n   * application more freedom because -1 is a special value when doing bit manipulations.\n   */\n  static final int INITIAL_VALUE = -2;\n\n  /**\n   *  Maximum tower height of this skip list and chance to grow tower by level.\n   *\n   *  Notice these two values could affect the memory usage and the performance.\n   *  Ideally they should be calculated based on the potential size of the skip list.\n   *\n   *  Given n is the number of elements in the skip list, the memory usage is in O(n).\n   *\n   *  More precisely,\n   *\n   *  the memory is mainly used for the following data:\n   *\n   *  header_tower  = O(maxTowerHeight + 1)\n   *  value         = O(n)\n   *  next_pointers = O(n * (1 - growTowerChance^(maxTowerHeight + 1)) / (1 - growTowerChance))\n   *\n   * thus, the total memory usage is in O(header_tower + value + next_pointers).\n   *\n   * Default value for maximum tower height and grow tower chance, these two numbers are chosen\n   * arbitrarily now.\n   */\n  @VisibleForTesting\n  public static final int MAX_TOWER_HEIGHT = 10;\n  private static final float GROW_TOWER_CHANCE = 0.2f;\n\n  public enum HasPositions {\n    YES,\n    NO\n  }\n\n  public enum HasPayloads {\n    YES,\n    NO\n  }\n\n  static final int INVALID_POSITION = -3;\n\n  /** Memory barrier. */\n  private volatile int maxPoolPointer;\n\n  /** Actual storage data structure. */\n  private final IntBlockPool blockPool;\n\n  /**\n   * Default comparator used to determine the order between two given values or between one key and\n   * another value.\n   *\n   * Notice this comparator is shared by all threads using this skip list, so it is not thread safe\n   * if it is maintaining some states. However, {@link #search}, {@link #insert}, and\n   * {@link #searchCeil} support passed in comparator as a parameter, which should be thread safe if\n   * managed by the caller properly.\n   */\n  private final SkipListComparator<K> defaultComparator;\n\n  /** Random generator used to decide if to grow tower by one level or not. */\n  private final Random random = new Random();\n\n  /**\n   * Used by writer thread to record last pointers at each level. Notice it is ok to have it as an\n   * instance field because we would only have one writer thread.\n   */\n  private final int[] lastPointers;\n\n  /**\n   * Whether the skip list contains positions. Used for text fields.\n   */\n  private final HasPositions hasPositions;\n\n  private final HasPayloads hasPayloads;\n\n  /**\n   * Creates a new probabilistic skip list, using the provided comparator to compare keys\n   * of type K.\n   *\n   * @param comparator a comparator used to compare integer values.\n   */\n  public SkipListContainer(\n      SkipListComparator<K> comparator,\n      HasPositions hasPositions,\n      HasPayloads hasPayloads,\n      String name\n  ) {\n    this(comparator, new IntBlockPool(INITIAL_VALUE, name), hasPositions, hasPayloads);\n  }\n\n  /**\n   * Base constructor, also used by flush handler.\n   */\n  private SkipListContainer(\n      SkipListComparator<K> comparator,\n      IntBlockPool blockPool,\n      HasPositions hasPositions,\n      HasPayloads hasPayloads) {\n    // Sentinel value specified by the comparator cannot equal to INITIAL_VALUE.\n    Preconditions.checkArgument(comparator.getSentinelValue() != INITIAL_VALUE);\n\n    this.defaultComparator = comparator;\n    this.lastPointers = new int[MAX_TOWER_HEIGHT];\n    this.blockPool = blockPool;\n    this.hasPositions = hasPositions;\n    this.hasPayloads = hasPayloads;\n  }\n\n  /**\n   * Search for the index of the greatest value which has key less than or equal to the given key.\n   *\n   * This is more like a floor search function. See {@link #searchCeil} for ceil search.\n   *\n   * @param key target key will be searched.\n   * @param skipListHead index of the header tower of the skip list will be searched.\n   * @param comparator comparator used for comparison when traversing through the skip list.\n   * @param searchFinger {@link SkipListSearchFinger} to accelerate search speed,\n   *                     notice the search finger must be before the key.\n   * @return the index of the greatest value which is less than or equal to given value,\n   *         will return skipListHead if given value has no greater or equal values.\n   */\n  public int search(\n      K key,\n      int skipListHead,\n      SkipListComparator<K> comparator,\n      @Nullable SkipListSearchFinger searchFinger) {\n    assert comparator != null;\n    // Start at the header tower.\n    int currentPointer = skipListHead;\n\n    // Instantiate nextPointer and nextValue outside of the for loop so we can use the value\n    // directly after for loop.\n    int nextPointer = getForwardPointer(currentPointer, MAX_TOWER_HEIGHT - 1);\n    int nextValue = getValue(nextPointer);\n\n    // Top down traversal.\n    for (int currentLevel = MAX_TOWER_HEIGHT - 1; currentLevel >= 0; currentLevel--) {\n      nextPointer = getForwardPointer(currentPointer, currentLevel);\n      nextValue = getValue(nextPointer);\n\n      // Jump to search finger at current level.\n      if (searchFinger != null) {\n        final int fingerPointer = searchFinger.getPointer(currentLevel);\n         assert searchFinger.isInitialPointer(fingerPointer)\n            || comparator.compareKeyWithValue(key, getValue(fingerPointer), INVALID_POSITION) >= 0;\n\n        if (!searchFinger.isInitialPointer(fingerPointer)\n            && comparator.compareValues(getValue(fingerPointer), nextValue) >= 0) {\n          currentPointer = fingerPointer;\n          nextPointer = getForwardPointer(currentPointer, currentLevel);\n          nextValue = getValue(nextPointer);\n        }\n      }\n\n      // Move forward.\n      while (comparator.compareKeyWithValue(key, nextValue, INVALID_POSITION) > 0) {\n        currentPointer = nextPointer;\n\n        nextPointer = getForwardPointer(currentPointer, currentLevel);\n        nextValue = getValue(nextPointer);\n      }\n\n      // Advance search finger.\n      if (searchFinger != null && currentPointer != skipListHead) {\n        final int currentValue = getValue(currentPointer);\n        final int fingerPointer = searchFinger.getPointer(currentLevel);\n\n        if (searchFinger.isInitialPointer(fingerPointer)\n            || comparator.compareValues(currentValue, getValue(fingerPointer)) > 0) {\n          searchFinger.setPointer(currentLevel, currentPointer);\n        }\n      }\n    }\n\n    // Return next pointer if next value matches searched value; otherwise return currentPointer.\n    return comparator.compareKeyWithValue(key, nextValue, INVALID_POSITION) == 0\n        ? nextPointer : currentPointer;\n  }\n\n  /**\n   * Perform search with {@link #defaultComparator}.\n   * Notice {@link #defaultComparator} is not thread safe if it is keeping some states.\n   */\n  public int search(K key, int skipListHead, @Nullable SkipListSearchFinger searchFinger) {\n    return search(key, skipListHead, this.defaultComparator, searchFinger);\n  }\n\n  /**\n   * Ceil search on given {@param key}.\n   *\n   * @param key target key will be searched.\n   * @param skipListHead index of the header tower of the skip list will be searched.\n   * @param comparator comparator used for comparison when traversing through the skip list.\n   * @param searchFinger {@link SkipListSearchFinger} to accelerate search speed.\n   * @return index of the smallest value with key greater or equal to the given key.\n   */\n  public int searchCeil(\n      K key,\n      int skipListHead,\n      SkipListComparator<K> comparator,\n      @Nullable SkipListSearchFinger searchFinger) {\n    assert comparator != null;\n\n    // Perform regular search.\n    final int foundPointer = search(key, skipListHead, comparator, searchFinger);\n\n    // Return foundPointer if it is not the list head and the pointed value has key equal to the\n    // given key; otherwise, return next pointer.\n    if (foundPointer != skipListHead\n        && comparator.compareKeyWithValue(key, getValue(foundPointer), INVALID_POSITION) == 0) {\n      return foundPointer;\n    } else {\n      return getNextPointer(foundPointer);\n    }\n  }\n\n  /**\n   * Perform searchCeil with {@link #defaultComparator}.\n   * Notice {@link #defaultComparator} is not thread safe if it is keeping some states.\n   */\n  public int searchCeil(\n      K key, int skipListHead, @Nullable SkipListSearchFinger searchFinger) {\n    return searchCeil(key, skipListHead, this.defaultComparator, searchFinger);\n  }\n\n  /**\n   * Insert a new value into the skip list.\n   *\n   * Notice inserting supports duplicate keys and duplicate values.\n   *\n   * Duplicate keys with different values or positions will be inserted consecutively.\n   * Duplciate keys with identical values will be ignored, and the duplicate will not be stored in\n   * the posting list.\n   *\n   * @param key is the key of the given value.\n   * @param value is the value will be inserted, cannot be {@link #getSentinelValue()}.\n   * @param skipListHead index of the header tower of the skip list will accept the new value.\n   * @param comparator comparator used for comparison when traversing through the skip list.\n   * @return whether this value exists in the posting list. Note that this will return true even\n   * if it is a new position.\n   */\n  public boolean insert(K key, int value, int position, int[] payload, int skipListHead,\n                    SkipListComparator<K> comparator) {\n    Preconditions.checkArgument(comparator != null);\n    Preconditions.checkArgument(value != getSentinelValue());\n\n    // Start at the header tower.\n    int currentPointer = skipListHead;\n\n    // Initialize lastPointers.\n    for (int i = 0; i < MAX_TOWER_HEIGHT; i++) {\n      this.lastPointers[i] = INITIAL_VALUE;\n    }\n    int nextPointer = INITIAL_VALUE;\n\n    // Top down traversal.\n    for (int currentLevel = MAX_TOWER_HEIGHT - 1; currentLevel >= 0; currentLevel--) {\n      nextPointer = getForwardPointer(currentPointer, currentLevel);\n      int nextValue = getValue(nextPointer);\n\n      int nextPosition = getPosition(nextPointer);\n      while (comparator.compareKeyWithValue(key, nextValue, nextPosition) > 0) {\n        currentPointer = nextPointer;\n\n        nextPointer = getForwardPointer(currentPointer, currentLevel);\n        nextValue = getValue(nextPointer);\n        nextPosition = getPosition(nextPointer);\n      }\n\n      // Store last pointers.\n      lastPointers[currentLevel] = currentPointer;\n    }\n\n    // we use isDuplicateValue to determine if a value already exists in a posting list (even if it\n    // is a new position). We need to check both current pointer and next pointer in case this is\n    // the largest position we have seen for this value in this skip list. In that case, nextPointer\n    // will point to a larger value, but we want to check the smaller one to see if it is the same\n    // value. For example, if we have [(1, 2), (2, 4)] and we want to insert (1, 3), then\n    // nextPointer will point to (2, 4), but we want to check the doc ID of (1, 2) to see if it has\n    // the same document ID.\n    boolean isDuplicateValue = getValue(currentPointer) == value || getValue(nextPointer) == value;\n\n    if (comparator.compareKeyWithValue(key, getValue(nextPointer), getPosition(nextPointer)) != 0) {\n      if (hasPayloads == HasPayloads.YES) {\n        Preconditions.checkNotNull(payload);\n        // If this skip list has payloads, we store the payload immediately before the document ID\n        // and position (iff the position exists) in the block pool. We store payloads before\n        // positions because they are variable length, and reading past them would require knowing\n        // the size of the payload. We don't store payloads after the doc ID because we have a\n        // variable number of pointers after the doc ID, and we would have no idea where the\n        // pointers stop and the payload starts.\n        for (int n : payload) {\n          this.blockPool.add(n);\n        }\n      }\n\n      if (hasPositions == HasPositions.YES) {\n        // If this skip list has positions, we store the position before the document ID in the\n        // block pool.\n        this.blockPool.add(position);\n      }\n\n      // Insert value.\n      final int insertedPointer = this.blockPool.add(value);\n\n      // Insert outgoing pointers.\n      final int height = getRandomTowerHeight();\n      for (int currentLevel = 0; currentLevel < height; currentLevel++) {\n        this.blockPool.add(getForwardPointer(lastPointers[currentLevel], currentLevel));\n      }\n\n      this.sync();\n\n      // Update incoming pointers.\n      for (int currentLevel = 0; currentLevel < height; currentLevel++) {\n        setForwardPointer(lastPointers[currentLevel], currentLevel, insertedPointer);\n      }\n\n      this.sync();\n    }\n\n    return isDuplicateValue;\n  }\n\n  /**\n   * Delete a given key from skip list\n   *\n   * @param key the key of the given value\n   * @param skipListHead index of the header tower of the skip list will accept the new value\n   * @param comparator comparator used for comparison when traversing through the skip list\n   * @return smallest value in the container. Returns {@link #INITIAL_VALUE} if the\n   * key does not exist.\n   */\n  public int delete(K key, int skipListHead, SkipListComparator<K> comparator) {\n    boolean foundKey = false;\n\n    for (int currentLevel = MAX_TOWER_HEIGHT - 1; currentLevel >= 0; currentLevel--) {\n      int currentPointer = skipListHead;\n      int nextValue = getValue(getForwardPointer(currentPointer, currentLevel));\n\n      // First we skip over all the nodes that are smaller than our key.\n      while (comparator.compareKeyWithValue(key, nextValue, INVALID_POSITION) > 0) {\n        currentPointer = getForwardPointer(currentPointer, currentLevel);\n        nextValue = getValue(getForwardPointer(currentPointer, currentLevel));\n      }\n\n      Preconditions.checkState(currentPointer != INITIAL_VALUE);\n\n      // If we don't find the node at this level that's OK, keep searching on a lower one.\n      if (comparator.compareKeyWithValue(key, nextValue, INVALID_POSITION) != 0) {\n        continue;\n      }\n\n      // We found an element to delete.\n      foundKey = true;\n\n      // Otherwise, save the current pointer. Right now, current pointer points to the first element\n      // that has the same value as key.\n      int savedPointer = currentPointer;\n\n      currentPointer = getForwardPointer(currentPointer, currentLevel);\n      // Then, walk over every element that is equal to the key.\n      while (comparator.compareKeyWithValue(key, getValue(currentPointer), INVALID_POSITION) == 0) {\n        currentPointer = getForwardPointer(currentPointer, currentLevel);\n      }\n\n      // update the saved pointer to point to the first non-equal element of the skip list.\n      setForwardPointer(savedPointer, currentLevel, currentPointer);\n    }\n\n    // Something has changed, need to sync up here.\n    if (foundKey) {\n      this.sync();\n      // return smallest value, might be used as first postings later\n      return getSmallestValue(skipListHead);\n    }\n\n    return INITIAL_VALUE;\n  }\n\n  /**\n   * Perform insert with {@link #defaultComparator}.\n   * Notice {@link #defaultComparator} is not thread safe if it is keeping some states.\n   */\n  public boolean insert(K key, int value, int skipListHead) {\n    return insert(key, value, INVALID_POSITION, EMPTY_PAYLOAD, skipListHead,\n        this.defaultComparator);\n  }\n\n  public boolean insert(K key, int value, int position, int[] payload, int skipListHead) {\n    return insert(key, value, position, payload, skipListHead, this.defaultComparator);\n  }\n\n  /**\n   * Perform delete with {@link #defaultComparator}.\n   * Notice {@link #defaultComparator} is not thread safe if it is keeping some states.\n   */\n  public int delete(K key, int skipListHead) {\n    return delete(key, skipListHead, this.defaultComparator);\n  }\n\n  /**\n   * Get the pointer of next value pointed by the given pointer.\n   *\n   * @param pointer reference to the current value.\n   * @return pointer of next value.\n   */\n  public int getNextPointer(int pointer) {\n    return getForwardPointer(pointer, 0);\n  }\n\n  /**\n   * Get the value pointed by a pointer, this is a dereference process.\n   *\n   * @param pointer is an array index on this.blockPool.\n   * @return value pointed pointed by the pointer.\n   */\n  public int getValue(int pointer) {\n    int value = blockPool.get(pointer);\n\n    // Visibility race\n    if (value == INITIAL_VALUE) {\n      // Volatile read to cross the memory barrier again.\n      final boolean isSafe = isPointerSafe(pointer);\n      assert isSafe;\n\n      // Re-read the pointer again\n      value = blockPool.get(pointer);\n    }\n\n    return value;\n  }\n\n  public int getSmallestValue(int skipListHeader) {\n    return getValue(getForwardPointer(skipListHeader, 0));\n  }\n\n  /**\n   * Builder of a forward search finger with header tower index.\n   *\n   * @return a new {@link SkipListSearchFinger} object.\n   */\n  public SkipListSearchFinger buildSearchFinger() {\n    return new SkipListSearchFinger(MAX_TOWER_HEIGHT);\n  }\n\n  /**\n   * Added another skip list into the int pool.\n   *\n   * @return index of the header tower of the newly created skip list.\n   */\n  public int newSkipList() {\n    // Virtual value of header.\n    final int sentinelValue = getSentinelValue();\n    if (hasPositions == HasPositions.YES) {\n      this.blockPool.add(INVALID_POSITION);\n    }\n    final int skipListHead = this.blockPool.add(sentinelValue);\n\n    // Build header tower, initially point all the pointers to\n    //   itself since no value has been inserted.\n    for (int i = 0; i < MAX_TOWER_HEIGHT; i++) {\n      this.blockPool.add(skipListHead);\n    }\n\n    this.sync();\n\n    return skipListHead;\n  }\n\n  /**\n   * Check if the block pool has been initiated by {@link #newSkipList}.\n   */\n  public boolean isEmpty() {\n    return this.blockPool.length() == 0;\n  }\n\n  /**\n   * Write to the volatile variable to cross memory barrier. maxPoolPointer is the memory barrier\n   * for new appends.\n   */\n  private void sync() {\n    this.maxPoolPointer = this.blockPool.length();\n  }\n\n  /**\n   * Read from volatile variable to cross memory barrier.\n   *\n   * @param pointer is an block pool index.\n   * @return boolean indicate if given pointer is within the range of max pool pointer.\n   */\n  private boolean isPointerSafe(int pointer) {\n    return pointer <= this.maxPoolPointer;\n  }\n\n  /**\n   * Get the position associated with the doc ID pointed to by pointer.\n   * @param pointer aka doc ID pointer.\n   * @return The value of the position for that doc ID. Returns INVALID_POSITION if the skip list\n   * does not have positions, or if there is no position for that pointer.\n   */\n  public int getPosition(int pointer) {\n    if (hasPositions == HasPositions.NO) {\n      return INVALID_POSITION;\n    }\n    // if this skip list has positions, the position will always be inserted into the block pool\n    // immediately before the doc ID.\n    return getValue(pointer - 1);\n  }\n\n  /**\n   * Get the payload pointer from a normal pointer (e.g. one returned from the {@link this#search}\n   * method).\n   */\n  public int getPayloadPointer(int pointer) {\n    Preconditions.checkState(hasPayloads == HasPayloads.YES,\n        \"getPayloadPointer() should only be called on a skip list that supports payloads.\");\n\n    // if this skip list has payloads, the payload will always be inserted into the block pool\n    // before the doc ID, and before the position if there is a position.\n    int positionOffset = hasPositions == HasPositions.YES ? 1 : 0;\n\n    return pointer - 1 - positionOffset;\n  }\n\n\n  int getPoolSize() {\n    return this.blockPool.length();\n  }\n\n\n  IntBlockPool getBlockPool() {\n    return blockPool;\n  }\n\n  public HasPayloads getHasPayloads() {\n    return hasPayloads;\n  }\n\n  /******************\n   * Helper Methods *\n   ******************/\n\n  /**\n   * Get the next forward pointer on a given level.\n   *\n   * @param pointer is an array index on this.blockPool, might be SENTINEL_VALUE.\n   * @param level indicates the level of the forward pointer will be acquired. It is zero indexed.\n   * @return next forward pointer on the given level, might be SENTINEL_VALUE.\n   */\n  private int getForwardPointer(int pointer, int level) {\n    final int pointerIndex = pointer + level + 1;\n\n    int forwardPointer = blockPool.get(pointerIndex);\n\n    // Visibility race\n    if (forwardPointer == INITIAL_VALUE) {\n      // Volatile read to cross the memory barrier again.\n      final boolean isSafe = isPointerSafe(pointerIndex);\n      assert isSafe;\n\n      // Re-read the pointer again\n      forwardPointer = blockPool.get(pointerIndex);\n    }\n\n    return forwardPointer;\n  }\n\n /**\n   * Set the next forward pointer on a given level.\n   *\n   * @param pointer points to the value, of which the pointer value will be updated.\n   * @param level indicates the level of the forward pointer will be set. It is zero indexed.\n   * @param target the value fo the target pointer which will be set.\n   */\n  private void setForwardPointer(int pointer, int level, int target) {\n    // Update header tower if given pointer points to headerTower.\n    setPointer(pointer + level + 1, target);\n  }\n\n  /**\n   * Set the value pointed by pointer\n   * @param pointer point to the actual position in the pool\n   * @param target the value we are going to set\n   */\n  private void setPointer(int pointer, int target) {\n    blockPool.set(pointer, target);\n  }\n\n  /**\n   * Getter of the sentinel value used by this skip list. The sentinel value should be provided\n   * by the comparator.\n   *\n   * @return sentinel value used by this skip list.\n   */\n  int getSentinelValue() {\n    return defaultComparator.getSentinelValue();\n  }\n\n  /**\n   * Return a height h in range [1, maxTowerHeight], each number with chance\n   * growTowerChance ^ (h - 1).\n   *\n   * @return a integer indicating height.\n   */\n  private int getRandomTowerHeight() {\n    int height = 1;\n    while (height < MAX_TOWER_HEIGHT && random.nextFloat() < GROW_TOWER_CHANCE) {\n      height++;\n    }\n    return height;\n  }\n\n  @SuppressWarnings(\"unchecked\")\n  @Override\n  public FlushHandler<K> getFlushHandler() {\n    return new FlushHandler<>(this);\n  }\n\n  public static class FlushHandler<K> extends Flushable.Handler<SkipListContainer<K>> {\n    private final SkipListComparator<K> comparator;\n    private static final String BLOCK_POOL_PROP_NAME = \"blockPool\";\n    private static final String HAS_POSITIONS_PROP_NAME = \"hasPositions\";\n    private static final String HAS_PAYLOADS_PROP_NAME = \"hasPayloads\";\n\n    public FlushHandler(SkipListContainer<K> objectToFlush) {\n      super(objectToFlush);\n      this.comparator = objectToFlush.defaultComparator;\n    }\n\n    public FlushHandler(SkipListComparator<K> comparator) {\n      this.comparator = comparator;\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      long startTime = getClock().nowMillis();\n      SkipListContainer<K> objectToFlush = getObjectToFlush();\n      flushInfo.addBooleanProperty(HAS_POSITIONS_PROP_NAME,\n          objectToFlush.hasPositions == HasPositions.YES);\n      flushInfo.addBooleanProperty(HAS_PAYLOADS_PROP_NAME,\n          objectToFlush.hasPayloads == HasPayloads.YES);\n\n      objectToFlush.blockPool.getFlushHandler()\n          .flush(flushInfo.newSubProperties(BLOCK_POOL_PROP_NAME), out);\n      getFlushTimerStats().timerIncrement(getClock().nowMillis() - startTime);\n    }\n\n    @Override\n    protected SkipListContainer<K> doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      long startTime = getClock().nowMillis();\n      IntBlockPool blockPool = (new IntBlockPool.FlushHandler()).load(\n          flushInfo.getSubProperties(BLOCK_POOL_PROP_NAME), in);\n      getLoadTimerStats().timerIncrement(getClock().nowMillis() - startTime);\n\n      HasPositions hasPositions = flushInfo.getBooleanProperty(HAS_POSITIONS_PROP_NAME)\n          ? HasPositions.YES : HasPositions.NO;\n      HasPayloads hasPayloads = flushInfo.getBooleanProperty(HAS_PAYLOADS_PROP_NAME)\n          ? HasPayloads.YES : HasPayloads.NO;\n\n      return new SkipListContainer<>(\n          this.comparator,\n          blockPool,\n          hasPositions,\n          hasPayloads);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/SkipListIntegerComparator.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\n/**\n *  Example implementation of {@link SkipListComparator} with Order-Theoretic Properties.\n *\n *  Notice:\n *    Re-using key object is highly suggested!\n *    Normally the generic type should be a mutable object so it can be reused by the reader/writer.\n */\npublic class SkipListIntegerComparator implements SkipListComparator<Integer> {\n\n  @Override\n  public int compareKeyWithValue(Integer key, int targetValue, int targetPosition) {\n    return key - targetValue;\n  }\n\n  @Override\n  public int compareValues(int v1, int v2) {\n    return v1 - v2;\n  }\n\n  @Override\n  public int getSentinelValue() {\n    return Integer.MAX_VALUE;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/SkipListPostingList.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\n\nimport static com.twitter.search.core.earlybird.index.inverted.SkipListContainer.HasPayloads;\nimport static com.twitter.search.core.earlybird.index.inverted.SkipListContainer.HasPositions;\nimport static com.twitter.search.core.earlybird.index.inverted.SkipListContainer.INVALID_POSITION;\nimport static com.twitter.search.core.earlybird.index.inverted.TermsArray.INVALID;\n\n/**\n * A skip list implementation of real time posting list. Supports out of order updates.\n */\npublic class SkipListPostingList implements Flushable {\n  /** Underlying skip list. */\n  private final SkipListContainer<Key> skipListContainer;\n\n  /** Key used when inserting into the skip list. */\n  private final Key key = new Key();\n\n  public SkipListPostingList(\n      HasPositions hasPositions,\n      HasPayloads hasPayloads,\n      String field) {\n    this.skipListContainer = new SkipListContainer<>(\n        new DocIDComparator(),\n        hasPositions,\n        hasPayloads,\n        field);\n  }\n\n  /** Used by {@link SkipListPostingList.FlushHandler} */\n  private SkipListPostingList(SkipListContainer<Key> skipListContainer) {\n    this.skipListContainer = skipListContainer;\n  }\n\n  /**\n   * Appends a posting to the posting list for a term.\n   */\n  public void appendPosting(\n      int termID,\n      TermsArray termsArray,\n      int docID,\n      int position,\n      @Nullable BytesRef payload) {\n    termsArray.getLargestPostings()[termID] = Math.max(\n        termsArray.getLargestPostings()[termID],\n        docID);\n\n    // Append to an existing skip list.\n    // Notice, header tower index is stored at the last postings pointer spot.\n    int postingsPointer = termsArray.getPostingsPointer(termID);\n    if (postingsPointer == INVALID) {\n      // Create a new skip list and add the first posting.\n      postingsPointer = skipListContainer.newSkipList();\n    }\n\n    boolean havePostingForThisDoc = insertPosting(docID, position, payload, postingsPointer);\n\n    // If this is a new document ID, we need to update the document frequency for this term\n    if (!havePostingForThisDoc) {\n      termsArray.getDocumentFrequency()[termID]++;\n    }\n\n    termsArray.updatePostingsPointer(termID, postingsPointer);\n  }\n\n  /**\n   * Deletes the given doc ID from the posting list for the term.\n   */\n  public void deletePosting(int termID, TermsArray postingsArray, int docID) {\n    int docFreq = postingsArray.getDocumentFrequency()[termID];\n    if (docFreq == 0) {\n      return;\n    }\n\n    int postingsPointer = postingsArray.getPostingsPointer(termID);\n    // skipListContainer is not empty, try to delete docId from it.\n    int smallestDoc = deletePosting(docID, postingsPointer);\n    if (smallestDoc == SkipListContainer.INITIAL_VALUE) {\n      // Key does not exist.\n      return;\n    }\n\n    postingsArray.getDocumentFrequency()[termID]--;\n  }\n\n  /**\n   * Insert posting into an existing skip list.\n   *\n   * @param docID docID of the this posting.\n   * @param skipListHead header tower index of the skip list\n   *                         in which the posting will be inserted.\n   * @return whether we have already inserted this document ID into this term list.\n   */\n  private boolean insertPosting(int docID, int position, BytesRef termPayload, int skipListHead) {\n    int[] payload = PayloadUtil.encodePayload(termPayload);\n    return skipListContainer.insert(key.withDocAndPosition(docID, position), docID, position,\n        payload, skipListHead);\n  }\n\n  private int deletePosting(int docID, int skipListHead) {\n    return skipListContainer.delete(key.withDocAndPosition(docID, INVALID_POSITION), skipListHead);\n  }\n\n  /** Return a term docs enumerator with position flag on. */\n  public PostingsEnum postings(\n      int postingPointer,\n      int docFreq,\n      int maxPublishedPointer) {\n    return new SkipListPostingsEnum(\n        postingPointer, docFreq, maxPublishedPointer, skipListContainer);\n  }\n\n  /**\n   * Get the number of documents (AKA document frequency or DF) for the given term.\n   */\n  public int getDF(int termID, TermsArray postingsArray) {\n    int[] documentFrequency = postingsArray.getDocumentFrequency();\n    Preconditions.checkArgument(termID < documentFrequency.length);\n\n    return documentFrequency[termID];\n  }\n\n  public int getDocIDFromPosting(int posting) {\n    // Posting is simply the whole doc ID.\n    return posting;\n  }\n\n  public int getMaxPublishedPointer() {\n    return skipListContainer.getPoolSize();\n  }\n\n\n  @SuppressWarnings(\"unchecked\")\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static class FlushHandler extends Flushable.Handler<SkipListPostingList> {\n    private static final String SKIP_LIST_PROP_NAME = \"skipList\";\n\n    public FlushHandler(SkipListPostingList objectToFlush) {\n      super(objectToFlush);\n    }\n\n    public FlushHandler() {\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      SkipListPostingList objectToFlush = getObjectToFlush();\n\n      objectToFlush.skipListContainer.getFlushHandler()\n          .flush(flushInfo.newSubProperties(SKIP_LIST_PROP_NAME), out);\n    }\n\n    @Override\n    protected SkipListPostingList doLoad(\n        FlushInfo flushInfo, DataDeserializer in) throws IOException {\n      SkipListComparator<Key> comparator = new DocIDComparator();\n      SkipListContainer.FlushHandler<Key> flushHandler =\n          new SkipListContainer.FlushHandler<>(comparator);\n      SkipListContainer<Key> skipList =\n          flushHandler.load(flushInfo.getSubProperties(SKIP_LIST_PROP_NAME), in);\n      return new SkipListPostingList(skipList);\n    }\n  }\n\n  /**\n   * Key used to in {@link SkipListContainer} by {@link SkipListPostingList}.\n   */\n  public static class Key {\n    private int docID;\n    private int position;\n\n    public int getDocID() {\n      return docID;\n    }\n\n    public int getPosition() {\n      return position;\n    }\n\n    public Key withDocAndPosition(int withDocID, int withPosition) {\n      this.docID = withDocID;\n      this.position = withPosition;\n      return this;\n    }\n  }\n\n  /**\n   * Comparator for docID and position.\n   */\n  public static class DocIDComparator implements SkipListComparator<Key> {\n    private static final int SENTINEL_VALUE = DocIdSetIterator.NO_MORE_DOCS;\n\n    @Override\n    public int compareKeyWithValue(Key key, int targetDocID, int targetPosition) {\n      // No key could represent sentinel value and sentinel value is the largest.\n      int docCompare = key.getDocID() - targetDocID;\n      if (docCompare == 0 && targetPosition != INVALID_POSITION) {\n        return key.getPosition() - targetPosition;\n      } else {\n        return docCompare;\n      }\n    }\n\n    @Override\n    public int compareValues(int docID1, int docID2) {\n      // Sentinel value is the largest.\n      return docID1 - docID2;\n    }\n\n    @Override\n    public int getSentinelValue() {\n      return SENTINEL_VALUE;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/SkipListPostingsEnum.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.core.earlybird.index.EarlybirdRealtimeIndexSegmentData;\n\nimport static com.twitter.search.core.earlybird.index.inverted.SkipListContainer.INVALID_POSITION;\n\n/**\n * TermDocs enumerator used by {@link SkipListPostingList}.\n */\npublic class SkipListPostingsEnum extends PostingsEnum {\n  /** Initialize cur doc ID and frequency. */\n  private int curDoc = TermsArray.INVALID;\n  private int curFreq = 0;\n\n  private final int postingPointer;\n\n  private final int cost;\n\n  /**\n   * maxPublishedPointer exists to prevent us from returning documents that are partially indexed.\n   * These pointers are safe to follow, but the documents should not be returned. See\n   * {@link EarlybirdRealtimeIndexSegmentData#getSyncData()} ()}.\n   */\n  private final int maxPublishedPointer;\n\n  /** Skip list info and search key */\n  private final SkipListContainer<SkipListPostingList.Key> skiplist;\n  private final SkipListPostingList.Key key = new SkipListPostingList.Key();\n\n  /**\n   * Pointer/posting/docID of next posting in the skip list.\n   *  Notice the next here is relative to last posting with curDoc ID.\n   */\n  private int nextPostingPointer;\n  private int nextPostingDocID;\n\n  /**\n   * We save the positionPointer because we must walk the posting list to obtain term frequency\n   * before we can start iterating through document positions. To do that walk, we increment\n   * postingsPointer until it points to the first posting for the next doc, so postingsPointer is no\n   * longer what we want to use as the start of the position list. The position pointer starts out\n   * pointing to the first posting with that doc ID value. There can be duplicate doc ID values with\n   * different positions. To find subsequent positions, we simply walk the posting list using this\n   * pointer.\n   */\n  private int positionPointer = -1;\n\n  /**\n   * The payloadPointer should only be called after calling nextPosition, as it points to a payload\n   * for each position. It is not updated unless nextPosition is called.\n   */\n  private int payloadPointer = -1;\n\n  /** Search finger used in advance method. */\n  private final SkipListSearchFinger advanceSearchFinger;\n\n  /**\n   * A new {@link PostingsEnum} for a real-time skip list-based posting list.\n   */\n  public SkipListPostingsEnum(\n      int postingPointer,\n      int docFreq,\n      int maxPublishedPointer,\n      SkipListContainer<SkipListPostingList.Key> skiplist) {\n    this.postingPointer = postingPointer;\n    this.skiplist = skiplist;\n    this.advanceSearchFinger = this.skiplist.buildSearchFinger();\n    this.maxPublishedPointer = maxPublishedPointer;\n    this.nextPostingPointer = postingPointer;\n\n    // WARNING:\n    // docFreq is approximate and may not be the true document frequency of the posting list.\n    this.cost = docFreq;\n\n    if (postingPointer != -1) {\n      // Because the posting pointer is not negative 1, we know it's valid.\n      readNextPosting();\n    }\n\n    advanceSearchFinger.reset();\n  }\n\n  @Override\n  public final int nextDoc() {\n    // Notice if skip list is exhausted nextPostingPointer will point back to postingPointer since\n    // skip list is circle linked.\n    if (nextPostingPointer == postingPointer) {\n      // Skip list is exhausted.\n      curDoc = NO_MORE_DOCS;\n      curFreq = 0;\n    } else {\n      // Skip list is not exhausted.\n      curDoc = nextPostingDocID;\n      curFreq = 1;\n      positionPointer = nextPostingPointer;\n\n      // Keep reading all the posting with the same doc ID.\n      // Notice:\n      //   - posting with the same doc ID will be stored consecutively\n      //     since the skip list is sorted.\n      //   - if skip list is exhausted, nextPostingPointer will become postingPointer\n      //     since skip list is circle linked.\n      readNextPosting();\n      while (nextPostingPointer != postingPointer && nextPostingDocID == curDoc) {\n        curFreq++;\n        readNextPosting();\n      }\n    }\n\n    // Returned updated curDoc.\n    return curDoc;\n  }\n\n  /**\n   * Moves the enumerator forward by one element, then reads the information at that position.\n   * */\n  private void readNextPosting() {\n    // Move search finger forward at lowest level.\n    advanceSearchFinger.setPointer(0, nextPostingPointer);\n\n    // Read next posting pointer.\n    nextPostingPointer = skiplist.getNextPointer(nextPostingPointer);\n\n    // Read the new posting positioned under nextPostingPointer into the nextPostingDocID.\n    readNextPostingInfo();\n  }\n\n  private boolean isPointerPublished(int pointer) {\n    return pointer <= maxPublishedPointer;\n  }\n\n  /** Read next posting and doc id encoded in next posting. */\n  private void readNextPostingInfo() {\n    // We need to skip over every pointer that has not been published to this Enum, otherwise the\n    // searcher will see unpublished documents. We also end termination if we reach\n    // nextPostingPointer == postingPointer, because that means we have reached the end of the\n    // skiplist.\n    while (!isPointerPublished(nextPostingPointer) && nextPostingPointer != postingPointer) {\n      // Move search finger forward at lowest level.\n      advanceSearchFinger.setPointer(0, nextPostingPointer);\n\n      // Read next posting pointer.\n      nextPostingPointer = skiplist.getNextPointer(nextPostingPointer);\n    }\n\n    // Notice if skip list is exhausted, nextPostingPointer will be postingPointer\n    // since skip list is circle linked.\n    if (nextPostingPointer != postingPointer) {\n      nextPostingDocID = skiplist.getValue(nextPostingPointer);\n    } else {\n      nextPostingDocID = NO_MORE_DOCS;\n    }\n  }\n\n  /**\n   * Jump to the target, then use {@link #nextDoc()} to collect nextDoc info.\n   * Notice target might be smaller than curDoc or smallestDocID.\n   */\n  @Override\n  public final int advance(int target) {\n    if (target == NO_MORE_DOCS) {\n      // Exhaust the posting list, so that future calls to docID() always return NO_MORE_DOCS.\n      nextPostingPointer = postingPointer;\n    }\n\n    if (nextPostingPointer == postingPointer) {\n      // Call nextDoc to ensure that all values are updated and we don't have to duplicate that\n      // here.\n      return nextDoc();\n    }\n\n    // Jump to target if target is bigger.\n    if (target >= curDoc && target >= nextPostingDocID) {\n      jumpToTarget(target);\n    }\n\n    // Retrieve next doc.\n    return nextDoc();\n  }\n\n  /**\n   * Set the next posting pointer (and info) to the first posting\n   * with doc ID equal to or larger than the target.\n   *\n   * Notice this method does not set curDoc or curFreq.\n   */\n  private void jumpToTarget(int target) {\n    // Do a ceil search.\n    nextPostingPointer = skiplist.searchCeil(\n        key.withDocAndPosition(target, INVALID_POSITION), postingPointer, advanceSearchFinger);\n\n    // Read next posting information.\n    readNextPostingInfo();\n  }\n\n  @Override\n  public int nextPosition() {\n    // If doc ID is equal to no more docs than we are past the end of the posting list. If doc ID\n    // is invalid, then we have not called nextDoc yet, and we should not return a real position.\n    // If the position pointer is past the current doc ID, then we should not return a position\n    // until nextDoc is called again (we don't want to return positions for a different doc).\n    if (docID() == NO_MORE_DOCS\n        || docID() == TermsArray.INVALID\n        || skiplist.getValue(positionPointer) != docID()) {\n      return INVALID_POSITION;\n    }\n    payloadPointer = positionPointer;\n    int position = skiplist.getPosition(positionPointer);\n    do {\n      positionPointer = skiplist.getNextPointer(positionPointer);\n    } while (!isPointerPublished(positionPointer) && positionPointer != postingPointer);\n    return position;\n  }\n\n  @Override\n  public BytesRef getPayload() {\n    if (skiplist.getHasPayloads() == SkipListContainer.HasPayloads.NO) {\n      return null;\n    }\n\n    int pointer = skiplist.getPayloadPointer(this.payloadPointer);\n    Preconditions.checkState(pointer > 0);\n    return PayloadUtil.decodePayload(skiplist.getBlockPool(), pointer);\n  }\n\n  @Override\n  public int startOffset() {\n    return -1;\n  }\n\n  @Override\n  public int endOffset() {\n    return -1;\n  }\n\n  @Override\n  public final int docID() {\n    return curDoc;\n  }\n\n  @Override\n  public final int freq() {\n    return curFreq;\n  }\n\n  @Override\n  public long cost() {\n    return cost;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/SkipListSearchFinger.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\n/**\n * A forward search finger used, optionally, by {@link SkipListContainer#search}.\n *\n * A search finger is pointer to the result returned by last time a search method is performed.\n * @see <a href=\"http://en.wikipedia.org/wiki/Finger_search\">Finger search wikipedia</a>.\n *\n * Using a search finger on a skip list could reduce the search search time from\n * log(n) to log(k), where n is length of the skip list and k is the distance between last searched\n * key and current searched key.\n */\npublic class SkipListSearchFinger {\n  // Pointer used when initialize the search finger.\n  public static final int INITIAL_POINTER = Integer.MIN_VALUE;\n\n  private final int[] lastPointers;\n\n  /**\n   * Creates a new search finger.\n   */\n  public SkipListSearchFinger(int maxTowerHeight) {\n    lastPointers = new int[maxTowerHeight];\n\n    reset();\n  }\n\n  public void reset() {\n    for (int i = 0; i < lastPointers.length; i++) {\n      setPointer(i, INITIAL_POINTER);\n    }\n  }\n\n  public int getPointer(int level) {\n    return lastPointers[level];\n  }\n\n  public void setPointer(int level, int pointer) {\n    lastPointers[level] = pointer;\n  }\n\n  public boolean isInitialPointer(int pointer) {\n    return pointer == INITIAL_POINTER;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/TermDictionary.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\n/**\n * A two-way mapping between terms and their interned value (termID).\n *\n * Implementation of this interface must guarantee that termIDs are dense, starting at 0;\n * so they are good to be used as indices in arrays.\n */\npublic interface TermDictionary extends Flushable {\n  int TERM_NOT_FOUND = EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND;\n\n  /**\n   * Returns the number of terms in this dictionary.\n   */\n  int getNumTerms();\n\n  /**\n   * Create a TermsEnum object over this TermDictionary for a given index.\n   * @param index\n   */\n  TermsEnum createTermsEnum(OptimizedMemoryIndex index);\n\n  /**\n   * Lookup a term in this dictionary.\n   * @param term  the term to lookup.\n   * @return  the term id for this term, or TERM_NOT_FOUND\n   * @throws IOException\n   */\n  int lookupTerm(BytesRef term) throws IOException;\n\n  /**\n   * Get the term for given id and possibly its payload.\n   * @param termID  the term that we want to get.\n   * @param text  MUST be non-null. It will be filled with the term.\n   * @param termPayload  if non-null, it will be filled with the payload if the term has any.\n   * @return  Returns true, iff this term has a term payload.\n   */\n  boolean getTerm(int termID, BytesRef text, BytesRef termPayload);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/TermPointerEncoding.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\n/**\n * Encodes and decodes term pointers.\n */\npublic abstract class TermPointerEncoding {\n  /**\n   * Returns the start of the text stored in a {@link BaseByteBlockPool} of the given term.\n   */\n  public abstract int getTextStart(int termPointer);\n\n  /**\n   * Returns true, if the given term stores a per-term payload.\n   */\n  public abstract boolean hasPayload(int termPointer);\n\n  /**\n   * Encodes and returns a pointer for a term stored at the given textStart in a\n   * {@link BaseByteBlockPool}.\n   */\n  public abstract int encodeTermPointer(int textStart, boolean hasPayload);\n\n  public static final TermPointerEncoding DEFAULT_ENCODING = new TermPointerEncoding() {\n    @Override public int getTextStart(int termPointer) {\n      return termPointer >>> 1;\n    }\n\n    @Override public boolean hasPayload(int termPointer) {\n      return (termPointer & 1) != 0;\n    }\n\n    @Override\n    public int encodeTermPointer(int textStart, boolean hasPayload) {\n      int code = textStart << 1;\n      return hasPayload ? (code | 1) : code;\n    }\n  };\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/inverted/TermsArray.java",
    "content": "package com.twitter.search.core.earlybird.index.inverted;\n\nimport java.io.IOException;\nimport java.util.Arrays;\n\nimport org.apache.lucene.util.ArrayUtil;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\n\n/**\n * TermsArray provides information on each term in the posting list.\n *\n * It does not provide any concurrency guarantees. The writer must ensure that all updates are\n * visible to readers with an external memory barrier.\n */\npublic class TermsArray implements Flushable {\n  private static final int BYTES_PER_POSTING = 5 * Integer.BYTES;\n  public static final int INVALID = -1;\n\n  private final int size;\n\n  public final int[] termPointers;\n  private final int[] postingsPointers;\n\n  // Derived data. Not atomic and not reliable.\n  public final int[] largestPostings;\n  public final int[] documentFrequency;\n  public final int[] offensiveCounters;\n\n  TermsArray(int size, boolean useOffensiveCounters) {\n    this.size = size;\n\n    termPointers = new int[size];\n    postingsPointers = new int[size];\n\n    largestPostings = new int[size];\n    documentFrequency = new int[size];\n\n    if (useOffensiveCounters) {\n      offensiveCounters = new int[size];\n    } else {\n      offensiveCounters = null;\n    }\n\n    Arrays.fill(postingsPointers, INVALID);\n    Arrays.fill(largestPostings, INVALID);\n  }\n\n  private TermsArray(TermsArray oldArray, int newSize) {\n    this(newSize, oldArray.offensiveCounters != null);\n    copyFrom(oldArray);\n  }\n\n  private TermsArray(\n      int size,\n      int[] termPointers,\n      int[] postingsPointers,\n      int[] largestPostings,\n      int[] documentFrequency,\n      int[] offensiveCounters) {\n    this.size = size;\n\n    this.termPointers = termPointers;\n    this.postingsPointers = postingsPointers;\n\n    this.largestPostings = largestPostings;\n    this.documentFrequency = documentFrequency;\n    this.offensiveCounters = offensiveCounters;\n  }\n\n  TermsArray grow() {\n    int newSize = ArrayUtil.oversize(size + 1, BYTES_PER_POSTING);\n    return new TermsArray(this, newSize);\n  }\n\n\n  private void copyFrom(TermsArray from) {\n    copy(from.termPointers, termPointers);\n    copy(from.postingsPointers, postingsPointers);\n\n    copy(from.largestPostings, largestPostings);\n    copy(from.documentFrequency, documentFrequency);\n\n    if (from.offensiveCounters != null) {\n      copy(from.offensiveCounters, offensiveCounters);\n    }\n  }\n\n  private void copy(int[] from, int[] to) {\n    System.arraycopy(from, 0, to, 0, from.length);\n  }\n\n  /**\n   * Returns the size of this array.\n   */\n  public int getSize() {\n    return size;\n  }\n\n  /**\n   * Write side operation for updating the pointer to the last posting for a given term.\n   */\n  public void updatePostingsPointer(int termID, int newPointer) {\n    postingsPointers[termID] = newPointer;\n  }\n\n  /**\n   * The returned pointer is guaranteed to be memory safe to follow to its target. The data\n   * structure it points to will be consistent and safe to traverse. The posting list may contain\n   * doc IDs that the current reader should not see, and the reader should skip over these doc IDs\n   * to ensure that the readers provide an immutable view of the doc IDs in a posting list.\n   */\n  public int getPostingsPointer(int termID) {\n    return postingsPointers[termID];\n  }\n\n  public int[] getDocumentFrequency() {\n    return documentFrequency;\n  }\n\n  /**\n   * Gets the array containing the first posting for each indexed term.\n   */\n  public int[] getLargestPostings() {\n    return largestPostings;\n  }\n\n  @SuppressWarnings(\"unchecked\")\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static class FlushHandler extends Flushable.Handler<TermsArray> {\n    private static final String SIZE_PROP_NAME = \"size\";\n    private static final String HAS_OFFENSIVE_COUNTERS_PROP_NAME = \"hasOffensiveCounters\";\n\n    public FlushHandler(TermsArray objectToFlush) {\n      super(objectToFlush);\n    }\n\n    public FlushHandler() {\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      TermsArray objectToFlush = getObjectToFlush();\n      flushInfo.addIntProperty(SIZE_PROP_NAME, objectToFlush.size);\n      boolean hasOffensiveCounters = objectToFlush.offensiveCounters != null;\n      flushInfo.addBooleanProperty(HAS_OFFENSIVE_COUNTERS_PROP_NAME, hasOffensiveCounters);\n\n      out.writeIntArray(objectToFlush.termPointers);\n      out.writeIntArray(objectToFlush.postingsPointers);\n\n      out.writeIntArray(objectToFlush.largestPostings);\n      out.writeIntArray(objectToFlush.documentFrequency);\n\n      if (hasOffensiveCounters) {\n        out.writeIntArray(objectToFlush.offensiveCounters);\n      }\n    }\n\n    @Override\n    protected TermsArray doLoad(\n        FlushInfo flushInfo, DataDeserializer in) throws IOException {\n      int size = flushInfo.getIntProperty(SIZE_PROP_NAME);\n      boolean hasOffensiveCounters = flushInfo.getBooleanProperty(HAS_OFFENSIVE_COUNTERS_PROP_NAME);\n\n      int[] termPointers = in.readIntArray();\n      int[] postingsPointers = in.readIntArray();\n\n      int[] largestPostings = in.readIntArray();\n      int[] documentFrequency = in.readIntArray();\n\n      int[] offensiveCounters = hasOffensiveCounters ? in.readIntArray() : null;\n\n      return new TermsArray(\n          size,\n          termPointers,\n          postingsPointers,\n          largestPostings,\n          documentFrequency,\n          offensiveCounters);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/util/AllDocsIterator.java",
    "content": "package com.twitter.search.core.earlybird.index.util;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.core.earlybird.index.EarlybirdRealtimeIndexSegmentAtomicReader;\n\n/**\n * Used to iterate through all of the documents in an Earlybird segment. This is necessary so that\n * we can ensure all of the documents we are reading have been published to the readers. If we used\n * the doc ID mapper to iterate through documents, it would return documents that have been only\n * partially added to the index, and could return bogus search results (SEARCH-27711).\n */\npublic class AllDocsIterator extends DocIdSetIterator {\n  public static final String ALL_DOCS_TERM = \"__all_docs\";\n\n  private final DocIdSetIterator delegate;\n\n  public AllDocsIterator(LeafReader reader) throws IOException {\n    delegate = buildDISI(reader);\n  }\n\n  private static DocIdSetIterator buildDISI(LeafReader reader) throws IOException {\n    if (!isRealtimeUnoptimizedSegment(reader)) {\n      return all(reader.maxDoc());\n    }\n\n    Terms terms =\n        reader.terms(EarlybirdFieldConstants.EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName());\n    if (terms == null) {\n      return all(reader.maxDoc());\n    }\n\n    TermsEnum termsEnum = terms.iterator();\n    boolean hasTerm = termsEnum.seekExact(new BytesRef(ALL_DOCS_TERM));\n    if (hasTerm) {\n      return termsEnum.postings(null);\n    }\n\n    return empty();\n  }\n\n  @Override\n  public int docID() {\n    return delegate.docID();\n  }\n\n  @Override\n  public int nextDoc() throws IOException {\n    return delegate.nextDoc();\n  }\n\n  @Override\n  public int advance(int target) throws IOException {\n    return delegate.advance(target);\n  }\n\n  @Override\n  public long cost() {\n    return delegate.cost();\n  }\n\n  /**\n   * Returns whether this is a realtime segment in the realtime index that is still unoptimized and\n   * mutable.\n   */\n  private static boolean isRealtimeUnoptimizedSegment(LeafReader reader) {\n    if (reader instanceof EarlybirdRealtimeIndexSegmentAtomicReader) {\n      EarlybirdRealtimeIndexSegmentAtomicReader realtimeReader =\n          (EarlybirdRealtimeIndexSegmentAtomicReader) reader;\n      return !realtimeReader.getSegmentData().isOptimized();\n    }\n\n    return false;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/util/RangeDISI.java",
    "content": "package com.twitter.search.core.earlybird.index.util;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.search.DocIdSetIterator;\n\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\npublic class RangeDISI extends DocIdSetIterator {\n  private final int start;\n  private final int end;\n  private final AllDocsIterator delegate;\n\n  private int currentDocId = -1;\n\n  public RangeDISI(LeafReader reader, int start, int end) throws IOException {\n    this.delegate = new AllDocsIterator(reader);\n    this.start = start;\n    if (end == DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      this.end = Integer.MAX_VALUE;\n    } else {\n      this.end = end;\n    }\n  }\n\n  @Override\n  public int docID() {\n    return currentDocId;\n  }\n\n  @Override\n  public int nextDoc() throws IOException {\n    return advance(currentDocId + 1);\n  }\n\n  @Override\n  public int advance(int target) throws IOException {\n    currentDocId = delegate.advance(Math.max(target, start));\n    if (currentDocId > end) {\n      currentDocId = NO_MORE_DOCS;\n    }\n    return currentDocId;\n  }\n\n  @Override\n  public long cost() {\n    return delegate.cost();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/util/RangeFilterDISI.java",
    "content": "package com.twitter.search.core.earlybird.index.util;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.search.DocIdSetIterator;\n\n/**\n * A doc id set iterator that iterates over a filtered set of ids from firstId inclusive to lastId\n * inclusive.\n */\npublic class RangeFilterDISI extends DocIdSetIterator {\n  private final RangeDISI delegate;\n\n  public RangeFilterDISI(LeafReader reader) throws IOException {\n    this(reader, 0, reader.maxDoc() - 1);\n  }\n\n  public RangeFilterDISI(LeafReader reader, int smallestDocID, int largestDocID)\n      throws IOException {\n    this.delegate = new RangeDISI(reader, smallestDocID, largestDocID);\n  }\n\n  @Override\n  public int docID() {\n    return delegate.docID();\n  }\n\n  @Override\n  public int nextDoc() throws IOException {\n    delegate.nextDoc();\n    return nextValidDoc();\n  }\n\n  @Override\n  public int advance(int target) throws IOException {\n    delegate.advance(target);\n    return nextValidDoc();\n  }\n\n  private int nextValidDoc() throws IOException {\n    int doc = delegate.docID();\n    while (doc != NO_MORE_DOCS && !shouldReturnDoc()) {\n      doc = delegate.nextDoc();\n    }\n    return doc;\n  }\n\n  @Override\n  public long cost() {\n    return delegate.cost();\n  }\n\n  // Override this method to add additional filters. Should return true if the current doc is OK.\n  protected boolean shouldReturnDoc() throws IOException {\n    return true;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/core/earlybird/index/util/SearchSortUtils.java",
    "content": "package com.twitter.search.core.earlybird.index.util;\n\nimport com.google.common.base.Preconditions;\n\npublic abstract class SearchSortUtils {\n  public interface Comparator<T> {\n    /**\n     *  Compares the item represented by the given index with the provided value.\n     */\n    int compare(int index, T value);\n  }\n\n  /**\n   * Performs a binary search using the given comparator, and returns the index of the item that\n   * was found. If foundLow is true, the greatest item that's lower than the provided key\n   * is returned. Otherwise, the lowest item that's greater than the provided key is returned.\n   */\n  public static <T> int binarySearch(Comparator<T> comparator, final int begin, final int end,\n      final T key, boolean findLow) {\n    int low = begin;\n    int high = end;\n    Preconditions.checkState(comparator.compare(low, key) <= comparator.compare(high, key));\n    while (low <= high) {\n      int mid = (low + high) >>> 1;\n      int result = comparator.compare(mid, key);\n      if (result < 0) {\n        low = mid + 1;\n      } else if (result > 0) {\n        high = mid - 1;\n      } else {\n        return mid;\n      } // key found\n    }\n\n    assert low > high;\n    if (findLow) {\n      return high < begin ? begin : high;\n    } else {\n      return low > end ? end : low;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/BUILD",
    "content": "COMMON_SOURCES = [\"common/**/*.java\"]\n\nCONFIG_SOURCES = [\"config/**/*.java\"]\n\nTOOLS_SOURCES = [\"tools/**/*.java\"]\n\nINDEX_SOURCES = [\"index/facets/**/*.java\"]\n\nSEGMENT_BUILDER_SOURCES = [\"archive/segmentbuilder/**/*.java\"]\n\njava_library(\n    name = \"earlybird-lib\",\n    sources = [\"**/*.java\"] + exclude_globs(COMMON_SOURCES + CONFIG_SOURCES + TOOLS_SOURCES + SEGMENT_BUILDER_SOURCES + INDEX_SOURCES),\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/code/gson\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/twitter/distributedlog:distributedlog-core\",\n        \"3rdparty/jvm/com/twitter/elephantbird:core\",\n        \"3rdparty/jvm/commons-codec\",\n        \"3rdparty/jvm/commons-httpclient\",\n        \"3rdparty/jvm/commons-io\",\n        \"3rdparty/jvm/commons-lang\",\n        \"3rdparty/jvm/geo/google:geoGoogle\",\n        \"3rdparty/jvm/io/netty:netty4-tcnative-boringssl-static\",\n        \"3rdparty/jvm/it/unimi/dsi:fastutil\",\n        \"3rdparty/jvm/javax/servlet:servlet-api\",\n        \"3rdparty/jvm/net/java/dev/jets3t\",\n        \"3rdparty/jvm/org/apache/bookkeeper:bookkeeper-server\",\n        \"3rdparty/jvm/org/apache/bookkeeper:bookkeeper-twitter-science-provider\",\n        \"3rdparty/jvm/org/apache/commons:commons-lang3\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/httpcomponents:httpclient\",\n        \"3rdparty/jvm/org/apache/kafka:kafka-clients\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-common\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-smartcn\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-facet\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-queries\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-queryparser\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-spatial-extras\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-test-framework\",\n        \"3rdparty/jvm/org/apache/thrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/json\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"3rdparty/jvm/org/tensorflow\",\n        \"3rdparty/jvm/org/tensorflow:tensorflow-hadoop\",\n        \"3rdparty/jvm/org/yaml:snakeyaml\",\n        \"cuad/projects/ner/thrift/src/main/thrift:thrift-java\",\n        \"decider/src/main/scala\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/client\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/server\",\n        \"finagle-internal/slo/src/main/scala/com/twitter/finagle/slo\",\n        \"finagle/finagle-base-http\",\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-http\",\n        \"finagle/finagle-serversets/src/main/scala\",\n        \"finagle/finagle-stats/src/main/scala\",\n        \"finagle/finagle-thrift/src/main/java\",\n        \"finagle/finagle-thrift/src/main/scala\",\n        \"finagle/finagle-thriftmux/src/main/scala\",\n        \"finagle/finagle-zipkin-core/src/main/scala\",\n        \"finagle/finagle-zipkin-scribe/src/main/scala\",\n        \"kafka/finagle-kafka/finatra-kafka/src/main/scala\",\n        \"periscope/api-proxy-thrift/thrift/src/main/thrift:thrift-java\",\n        \"servo/decider\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"src/antlr/com/twitter/search/queryparser/antlr:queryparser-antlr\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/collections\",\n        \"src/java/com/twitter/common/net:dynamic-host-set\",\n        \"src/java/com/twitter/common/quantity\",\n        \"src/java/com/twitter/common/text/language:locale-util\",\n        \"src/java/com/twitter/common/text/token\",\n        \"src/java/com/twitter/common/text/util:token-util\",\n        \"src/java/com/twitter/common/util\",\n        \"src/java/com/twitter/common/util:system-mocks\",\n        \"src/java/com/twitter/common/zookeeper:client\",\n        \"src/java/com/twitter/common/zookeeper:group\",\n        \"src/java/com/twitter/common/zookeeper:server-set\",\n        \"src/java/com/twitter/common_internal/bloomfilter\",\n        \"src/java/com/twitter/common_internal/collections\",\n        \"src/java/com/twitter/common_internal/text:text-penguin7\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/common_internal/zookeeper\",\n        \"src/java/com/twitter/ml/api:api-base\",\n        \"src/java/com/twitter/search/common/aurora\",\n        \"src/java/com/twitter/search/common/concurrent\",\n        \"src/java/com/twitter/search/common/config\",\n        \"src/java/com/twitter/search/common/constants\",\n        \"src/java/com/twitter/search/common/dark\",\n        \"src/java/com/twitter/search/common/database\",\n        \"src/java/com/twitter/search/common/decider\",\n        \"src/java/com/twitter/search/common/encoding/docvalues\",\n        \"src/java/com/twitter/search/common/encoding/features\",\n        \"src/java/com/twitter/search/common/features\",\n        \"src/java/com/twitter/search/common/file\",\n        \"src/java/com/twitter/search/common/logging\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/partitioning/base\",\n        \"src/java/com/twitter/search/common/partitioning/snowflakeparser\",\n        \"src/java/com/twitter/search/common/partitioning/zookeeper\",\n        \"src/java/com/twitter/search/common/query\",\n        \"src/java/com/twitter/search/common/relevance:feature-update-reader\",\n        \"src/java/com/twitter/search/common/relevance:scorers\",\n        \"src/java/com/twitter/search/common/relevance:text\",\n        \"src/java/com/twitter/search/common/relevance/features\",\n        \"src/java/com/twitter/search/common/schema\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/common/search\",\n        \"src/java/com/twitter/search/common/search/termination\",\n        \"src/java/com/twitter/search/common/util:closeresourceutil\",\n        \"src/java/com/twitter/search/common/util:finagleutil\",\n        \"src/java/com/twitter/search/common/util:gcutil\",\n        \"src/java/com/twitter/search/common/util:kerberos\",\n        \"src/java/com/twitter/search/common/util:log_format_util\",\n        \"src/java/com/twitter/search/common/util:longintconverter\",\n        \"src/java/com/twitter/search/common/util:platform_stats_exporter\",\n        \"src/java/com/twitter/search/common/util:rule_based_converter\",\n        \"src/java/com/twitter/search/common/util/analysis\",\n        \"src/java/com/twitter/search/common/util/date\",\n        \"src/java/com/twitter/search/common/util/earlybird\",\n        \"src/java/com/twitter/search/common/util/hash\",\n        \"src/java/com/twitter/search/common/util/io\",\n        \"src/java/com/twitter/search/common/util/io:dl-reader-writer\",\n        \"src/java/com/twitter/search/common/util/io:flushable\",\n        \"src/java/com/twitter/search/common/util/io:record-reader-api\",\n        \"src/java/com/twitter/search/common/util/io/kafka\",\n        \"src/java/com/twitter/search/common/util/lang\",\n        \"src/java/com/twitter/search/common/util/ml/models_manager\",\n        \"src/java/com/twitter/search/common/util/ml/prediction_engine\",\n        \"src/java/com/twitter/search/common/util/ml/tensorflow_engine\",\n        \"src/java/com/twitter/search/common/util/spatial\",\n        \"src/java/com/twitter/search/common/util/text\",\n        \"src/java/com/twitter/search/common/util/text/regex\",\n        \"src/java/com/twitter/search/common/util/thrift:text-protocol\",\n        \"src/java/com/twitter/search/common/util/thrift:thrift-utils\",\n        \"src/java/com/twitter/search/common/util/url\",\n        \"src/java/com/twitter/search/common/util/zktrylock\",\n        \"src/java/com/twitter/search/common/util/zookeeper\",\n        \"src/java/com/twitter/search/core/earlybird\",\n        \"src/java/com/twitter/search/earlybird/common\",\n        \"src/java/com/twitter/search/earlybird/common/config\",\n        \"src/java/com/twitter/search/earlybird/common/userupdates\",\n        \"src/java/com/twitter/search/earlybird/config\",\n        \"src/java/com/twitter/search/earlybird/index/facets\",\n        \"src/java/com/twitter/search/ingester/pipeline/strato_fetchers\",\n        \"src/java/com/twitter/search/modeling/common\",\n        \"src/java/com/twitter/search/modeling/tweet_ranking\",\n        \"src/java/com/twitter/search/queryparser\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/java/com/twitter/search/queryparser/query/search:search-query-nodes\",\n        \"src/resources/com/twitter/search/earlybird/com/twitter\",\n        \"src/resources/com/twitter/search/earlybird/ml\",\n        \"src/thrift/com/twitter/search:common\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n        \"src/thrift/com/twitter/search/common:features-java\",\n        \"src/thrift/com/twitter/search/common:indexing-java\",\n        \"src/thrift/com/twitter/search/common:query-java\",\n        \"src/thrift/com/twitter/service/spiderduck/gen:metadata-store-java\",\n        \"src/thrift/com/twitter/tweetypie:events-java\",\n        \"src/thrift/org/apache/aurora/gen:api\",\n        \"stitch/stitch-core/src/main/scala/com/twitter/stitch\",\n        \"strato/src/main/scala/com/twitter/strato/catalog\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"strato/src/main/scala/com/twitter/strato/data\",\n        \"strato/src/main/scala/com/twitter/strato/thrift\",\n        \"tensorflow/tfcompute-java/src/main/java/com/twitter/tfcompute_java\",\n        \"thrift-web-forms/src/main/java/com/twitter/thriftwebforms\",\n        \"thrift-web-forms/src/main/scala/com/twitter/thriftwebforms\",\n        \"twitter-server-internal\",\n        \"twitter-server/server/src/main/scala\",\n        \"ubs/common/src/main/thrift/com/twitter/ubs:broadcast-thrift-java\",\n        \"ubs/common/src/main/thrift/com/twitter/ubs:events-java\",\n        \"util-internal/util-eval/src/main/scala\",\n        \"util/util-app\",\n        \"util/util-core:scala\",\n        \"util/util-function\",\n        \"util/util-lint\",\n        \"util/util-slf4j-api/src/main/scala\",\n        \"util/util-stats/src/main/scala\",\n    ],\n)\n\njvm_binary(\n    name = \"earlybird-binary\",\n    basename = \"earlybird\",\n    main = \"com.twitter.search.earlybird.EarlybirdMain\",\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \":earlybird-lib\",\n        \"loglens/loglens-log4j\",\n    ],\n)\n\njava_library(\n    name = \"tools\",\n    sources = TOOLS_SOURCES,\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \":earlybird-lib\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/guava:guava-testlib\",\n        \"3rdparty/jvm/commons-codec\",\n        \"3rdparty/jvm/commons-httpclient\",\n        \"3rdparty/jvm/geo/google:geoGoogle\",\n        \"3rdparty/jvm/junit\",\n        \"3rdparty/jvm/net/java/dev/jets3t\",\n        \"3rdparty/jvm/org/apache/bookkeeper:bookkeeper-server\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/CONFIG.ini",
    "content": "; See http://go/CONFIG.ini\n\n[jira]\nproject: SEARCH\n\n[kite]\nproject: earlybird\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/Earlybird.java",
    "content": "package com.twitter.search.earlybird;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.net.InetAddress;\nimport java.net.UnknownHostException;\nimport java.util.Arrays;\nimport java.util.Map;\nimport java.util.function.Predicate;\nimport java.util.stream.Collectors;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.app.Flag;\nimport com.twitter.app.Flaggable;\nimport com.twitter.finagle.Http;\nimport com.twitter.finagle.http.HttpMuxer;\nimport com.twitter.search.common.aurora.AuroraInstanceKey;\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.common.config.LoggerConfiguration;\nimport com.twitter.search.common.constants.SearchThriftWebFormsAccess;\nimport com.twitter.search.common.metrics.BuildInfoStats;\nimport com.twitter.search.common.util.Kerberos;\nimport com.twitter.search.common.util.PlatformStatsExporter;\nimport com.twitter.search.earlybird.admin.EarlybirdAdminManager;\nimport com.twitter.search.earlybird.admin.EarlybirdHealthHandler;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.exception.EarlybirdStartupException;\nimport com.twitter.search.earlybird.exception.UncaughtExceptionHandler;\nimport com.twitter.search.earlybird.factory.EarlybirdServerFactory;\nimport com.twitter.search.earlybird.factory.EarlybirdWireModule;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird.util.EarlybirdDecider;\nimport com.twitter.server.handler.DeciderHandler$;\nimport com.twitter.server.AbstractTwitterServer;\nimport com.twitter.thriftwebforms.DisplaySettingsConfig;\nimport com.twitter.thriftwebforms.MethodOptionsAccessConfig;\nimport com.twitter.thriftwebforms.ThriftClientSettingsConfig;\nimport com.twitter.thriftwebforms.ThriftMethodSettingsConfig;\nimport com.twitter.thriftwebforms.ThriftServiceSettings;\nimport com.twitter.thriftwebforms.ThriftWebFormsSettings;\nimport com.twitter.thriftwebforms.TwitterServerThriftWebForms;\nimport com.twitter.util.Await;\nimport com.twitter.util.TimeoutException;\n\npublic class Earlybird extends AbstractTwitterServer {\n  private static final Logger LOG = LoggerFactory.getLogger(Earlybird.class);\n\n  // Flags defined here need to be processed before setting override values to EarlybirdConfig.\n\n  private final Flag<File> configFile = flag().create(\n      \"config_file\",\n      new File(\"earlybird-search.yml\"),\n      \"specify config file\",\n      Flaggable.ofFile()\n  );\n\n  private final Flag<String> logDir = flag().create(\n      \"earlybird_log_dir\",\n      \"\",\n      \"override log dir from config file\",\n      Flaggable.ofString()\n  );\n\n  private final Map<String, Flag<?>> flagMap = Arrays.stream(EarlybirdProperty.values())\n      .collect(Collectors.toMap(\n          property -> property.name(),\n          property -> property.createFlag(flag())));\n\n  private final UncaughtExceptionHandler uncaughtExceptionHandler =\n      new UncaughtExceptionHandler();\n\n  private EarlybirdServer earlybirdServer;\n  private EarlybirdAdminManager earlybirdAdminManager;\n\n  public Earlybird() {\n    // Default health handler is added inside Lifecycle trait.  To override that we need to set it\n    // in the constructor since HttpAdminServer is started before Earlybird.preMain() is called.\n    HttpMuxer.addHandler(\"/health\", new EarlybirdHealthHandler());\n  }\n\n  /**\n   * Needs to be called from preMain and not from onInit() as flags / args parsing happens after\n   * onInit() is called.\n   */\n  @VisibleForTesting\n  void configureFromFlagsAndSetupLogging() {\n    // Makes sure the EarlybirdStats is injected with a variable repository.\n    EarlybirdConfig.init(configFile.getWithDefault().get().getName());\n\n    if (logDir.isDefined()) {\n      EarlybirdConfig.overrideLogDir(logDir.get().get());\n    }\n    new LoggerConfiguration(EarlybirdConfig.getLogPropertiesFile(),\n        EarlybirdConfig.getLogDir()).configure();\n\n    String instanceKey = System.getProperty(\"aurora.instanceKey\");\n    if (instanceKey != null) {\n      EarlybirdConfig.setAuroraInstanceKey(AuroraInstanceKey.fromInstanceKey(instanceKey));\n      LOG.info(\"Earlybird is running on Aurora\");\n      checkRequiredProperties(EarlybirdProperty::isRequiredOnAurora, \"Aurora\");\n    } else {\n      LOG.info(\"Earlybird is running on dedicated hardware\");\n      checkRequiredProperties(EarlybirdProperty::isRequiredOnDedicated, \"dedicated hardware\");\n    }\n    LOG.info(\"Config environment: {}\", Config.getEnvironment());\n\n    if (adminPort().isDefined() && adminPort().get().isDefined()) {\n      int adminPort = adminPort().get().get().getPort();\n      LOG.info(\"Admin port is {}\", adminPort);\n      EarlybirdConfig.setAdminPort(adminPort);\n    }\n\n    EarlybirdConfig.setOverrideValues(\n        flagMap.values().stream()\n            .filter(Flag::isDefined)\n            .collect(Collectors.toMap(Flag::name, flag -> flag.get().get())));\n  }\n\n  private void checkRequiredProperties(\n      Predicate<EarlybirdProperty> propertyPredicate, String location) {\n    Arrays.stream(EarlybirdProperty.values())\n        .filter(propertyPredicate)\n        .map(property -> flagMap.get(property.name()))\n        .forEach(flag ->\n            Preconditions.checkState(flag.isDefined(),\n                \"-%s is required on %s\", flag.name(), location));\n  }\n\n  private void logEarlybirdInfo() {\n    try {\n      LOG.info(\"Hostname: {}\", InetAddress.getLocalHost().getHostName());\n    } catch (UnknownHostException e) {\n      LOG.info(\"Unable to be get local host: {}\", e.getMessage());\n    }\n    LOG.info(\"Earlybird info [Name: {}, Zone: {}, Env: {}]\",\n            EarlybirdProperty.EARLYBIRD_NAME.get(),\n            EarlybirdProperty.ZONE.get(),\n            EarlybirdProperty.ENV.get());\n    LOG.info(\"Earlybird scrubgen from Aurora: {}]\",\n        EarlybirdProperty.EARLYBIRD_SCRUB_GEN.get());\n    LOG.info(\"Find final partition config by searching the log for \\\"Partition config info\\\"\");\n  }\n\n  private EarlybirdServer makeEarlybirdServer() {\n    EarlybirdWireModule earlybirdWireModule = new EarlybirdWireModule();\n    EarlybirdServerFactory earlybirdFactory = new EarlybirdServerFactory();\n    try {\n      return earlybirdFactory.makeEarlybirdServer(earlybirdWireModule);\n    } catch (IOException e) {\n      LOG.error(\"Exception while constructing EarlybirdServer.\", e);\n      throw new RuntimeException(e);\n    }\n  }\n\n  private void setupThriftWebForms() {\n    TwitterServerThriftWebForms.addAdminRoutes(this, TwitterServerThriftWebForms.apply(\n        ThriftWebFormsSettings.apply(\n            DisplaySettingsConfig.DEFAULT,\n            ThriftServiceSettings.apply(\n                EarlybirdService.ServiceIface.class.getSimpleName(),\n                EarlybirdConfig.getThriftPort()),\n            ThriftClientSettingsConfig.makeCompactRequired(\n                EarlybirdProperty.getServiceIdentifier()),\n            ThriftMethodSettingsConfig.access(\n              MethodOptionsAccessConfig.byLdapGroup(\n                SearchThriftWebFormsAccess.READ_LDAP_GROUP))),\n        scala.reflect.ClassTag$.MODULE$.apply(EarlybirdService.ServiceIface.class)));\n  }\n\n  private void setupDeciderWebForms() {\n    addAdminRoute(\n        DeciderHandler$.MODULE$.route(\n            \"earlybird\",\n            EarlybirdDecider.getMutableDecisionMaker(),\n            EarlybirdDecider.getDecider()));\n  }\n\n  @Override\n  public Http.Server configureAdminHttpServer(Http.Server server) {\n    return server.withMonitor(uncaughtExceptionHandler);\n  }\n\n  @Override\n  public void preMain() {\n    configureFromFlagsAndSetupLogging();\n    logEarlybirdInfo();\n    LOG.info(\"Starting preMain()\");\n\n    BuildInfoStats.export();\n    PlatformStatsExporter.exportPlatformStats();\n\n    // Use our own exception handler to monitor all unhandled exceptions.\n    Thread.setDefaultUncaughtExceptionHandler((thread, e) -> {\n      LOG.error(\"Invoked default uncaught exception handler.\");\n      uncaughtExceptionHandler.handle(e);\n    });\n    LOG.info(\"Registered unhandled exception monitor.\");\n\n    Kerberos.kinit(\n        EarlybirdConfig.getString(\"kerberos_user\", \"\"),\n        EarlybirdConfig.getString(\"kerberos_keytab_path\", \"\")\n    );\n\n    LOG.info(\"Creating earlybird server.\");\n    earlybirdServer = makeEarlybirdServer();\n\n    uncaughtExceptionHandler.setShutdownHook(() -> {\n      earlybirdServer.shutdown();\n      this.close();\n    });\n\n    earlybirdAdminManager = EarlybirdAdminManager.create(earlybirdServer);\n    earlybirdAdminManager.start();\n    LOG.info(\"Started admin interface.\");\n\n    setupThriftWebForms();\n    setupDeciderWebForms();\n\n    LOG.info(\"Opened thrift serving form.\");\n\n    LOG.info(\"preMain() complete.\");\n  }\n\n  @Override\n  public void main() throws InterruptedException, TimeoutException, EarlybirdStartupException {\n    innerMain();\n  }\n\n  /**\n   * Setting up an innerMain() so that tests can mock out the contents of main without interfering\n   * with reflection being done in App.scala looking for a method named \"main\".\n   */\n  @VisibleForTesting\n  void innerMain() throws TimeoutException, InterruptedException, EarlybirdStartupException {\n    LOG.info(\"Starting main().\");\n\n    // If this method throws, TwitterServer will catch the exception and call close, so we don't\n    // catch it here.\n    try {\n      earlybirdServer.start();\n    } catch (Throwable throwable) {\n      LOG.error(\"Exception while starting:\", throwable);\n      throw throwable;\n    }\n\n    Await.ready(adminHttpServer());\n    LOG.info(\"main() complete.\");\n  }\n\n  @Override\n  public void onExit() {\n    LOG.info(\"Starting onExit()\");\n    earlybirdServer.shutdown();\n    try {\n      earlybirdAdminManager.doShutdown();\n    } catch (InterruptedException e) {\n      LOG.warn(\"earlybirdAdminManager shutdown was interrupted with \" + e);\n    }\n    LOG.info(\"onExit() complete.\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/EarlybirdCPUQualityFactor.java",
    "content": "package com.twitter.search.earlybird;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.sun.management.OperatingSystemMXBean;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\n\n/**\n * Manages the quality factor for an Earlybird based on CPU usage.\n */\npublic class EarlybirdCPUQualityFactor implements QualityFactor {\n  public static final String ENABLE_QUALITY_FACTOR_DECIDER = \"enable_quality_factor\";\n  public static final String OVERRIDE_QUALITY_FACTOR_DECIDER = \"override_quality_factor\";\n\n  @VisibleForTesting\n  protected static final double CPU_USAGE_THRESHOLD = 0.8;\n  @VisibleForTesting\n  protected static final double MAX_QF_INCREMENT = 0.5;\n  @VisibleForTesting\n  protected static final double MAX_QF_DECREMENT = 0.1;\n  @VisibleForTesting\n  protected static final double MAX_CPU_USAGE = 1.0;\n\n  private static final Logger QUALITY_FACTOR_LOG =\n      LoggerFactory.getLogger(EarlybirdCPUQualityFactor.class);\n  private static final Logger EARLYBIRD_LOG = LoggerFactory.getLogger(Earlybird.class);\n\n  /**\n   * Tracks the real, underlying CPU QF value, regardless of the decider enabling\n   * it.\n   */\n  @VisibleForTesting\n  protected static final String UNDERLYING_CPU_QF_GUAGE = \"underlying_cpu_quality_factor\";\n\n  /**\n   * Reports the QF actually used to degrade Earlybirds.\n   */\n  @VisibleForTesting\n  protected static final String CPU_QF_GUAGE = \"cpu_quality_factor\";\n\n  private static final int SAMPLING_WINDOW_MILLIS = 60 * 1000;   // one minute\n\n\n  private double qualityFactor = 1;\n  private double previousQualityFactor = 1;\n\n  private final SearchDecider decider;\n  private final OperatingSystemMXBean operatingSystemMXBean;\n\n  public EarlybirdCPUQualityFactor(\n      Decider decider,\n      OperatingSystemMXBean operatingSystemMXBean,\n      SearchStatsReceiver searchStatsReceiver) {\n    this.decider = new SearchDecider(decider);\n    this.operatingSystemMXBean = operatingSystemMXBean;\n\n    searchStatsReceiver.getCustomGauge(UNDERLYING_CPU_QF_GUAGE, () -> qualityFactor);\n    searchStatsReceiver.getCustomGauge(CPU_QF_GUAGE, this::get);\n  }\n\n  /**\n   * Updates the current quality factor based on CPU usage.\n   */\n  @VisibleForTesting\n  protected void update() {\n    previousQualityFactor = qualityFactor;\n\n    double cpuUsage = operatingSystemMXBean.getSystemCpuLoad();\n\n    if (cpuUsage < CPU_USAGE_THRESHOLD) {\n      double increment =\n          ((CPU_USAGE_THRESHOLD - cpuUsage) / CPU_USAGE_THRESHOLD) * MAX_QF_INCREMENT;\n      qualityFactor = Math.min(1, qualityFactor + increment);\n    } else {\n      double decrement =\n          ((cpuUsage - CPU_USAGE_THRESHOLD) / (MAX_CPU_USAGE - CPU_USAGE_THRESHOLD))\n              * MAX_QF_DECREMENT;\n      qualityFactor = Math.max(0, qualityFactor - decrement);\n    }\n\n    if (!qualityFactorChanged()) {\n      return;\n    }\n\n    QUALITY_FACTOR_LOG.info(\n        String.format(\"CPU: %.2f Quality Factor: %.2f\", cpuUsage, qualityFactor));\n\n    if (!enabled()) {\n      return;\n    }\n\n    if (degradationBegan()) {\n      EARLYBIRD_LOG.info(\"Service degradation began.\");\n    }\n\n    if (degradationEnded()) {\n      EARLYBIRD_LOG.info(\"Service degradation ended.\");\n    }\n  }\n\n  @Override\n  public double get() {\n    if (!enabled()) {\n      return 1;\n    }\n\n    if (isOverridden()) {\n      return override();\n    }\n\n    return qualityFactor;\n  }\n\n  @Override\n  public void startUpdates() {\n    new Thread(() -> {\n      while (true) {\n        update();\n        try {\n          Thread.sleep(SAMPLING_WINDOW_MILLIS);\n        } catch (InterruptedException e) {\n          QUALITY_FACTOR_LOG.warn(\n              \"Quality factoring thread interrupted during sleep between updates\", e);\n        }\n      }\n    }).start();\n  }\n\n  /**\n   * Returns true if quality factoring is enabled by the decider.\n   * @return\n   */\n  private boolean enabled() {\n    return decider != null && decider.isAvailable(ENABLE_QUALITY_FACTOR_DECIDER);\n  }\n\n  /**\n   * Returns true if a decider has overridden the quality factor.\n   * @return\n   */\n  private boolean isOverridden() {\n    return decider != null && decider.getAvailability(OVERRIDE_QUALITY_FACTOR_DECIDER) < 10000.0;\n  }\n\n  /**\n   * Returns the override decider value.\n   * @return\n   */\n  private double override() {\n    return decider == null ? 1 : decider.getAvailability(OVERRIDE_QUALITY_FACTOR_DECIDER) / 10000.0;\n  }\n\n  /**\n   * Returns true if the quality factor has changed since the last update.\n   * @return\n   */\n  private boolean qualityFactorChanged() {\n    return Math.abs(qualityFactor - previousQualityFactor) > 0.01;\n  }\n\n  /**\n   * Returns true if we've entered a degraded state.\n   * @return\n   */\n  private boolean degradationBegan() {\n    return Math.abs(previousQualityFactor - 1.0) < 0.01 && qualityFactor < previousQualityFactor;\n  }\n\n  /**\n   * Returns true if we've left the degraded state.\n   * @return\n   */\n  private boolean degradationEnded() {\n    return Math.abs(qualityFactor - 1.0) < 0.01 && previousQualityFactor < qualityFactor;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/EarlybirdDarkProxy.java",
    "content": "package com.twitter.search.earlybird;\n\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.Lists;\n\nimport org.apache.thrift.protocol.TCompactProtocol;\n\nimport com.twitter.finagle.ThriftMux;\nimport com.twitter.finagle.builder.ClientBuilder;\nimport com.twitter.finagle.builder.ClientConfig.Yes;\nimport com.twitter.finagle.mtls.client.MtlsThriftMuxClient;\nimport com.twitter.finagle.stats.StatsReceiver;\nimport com.twitter.finagle.thrift.ClientId;\nimport com.twitter.finagle.thrift.ThriftClientRequest;\nimport com.twitter.finagle.zipkin.thrift.ZipkinTracer;\nimport com.twitter.search.common.dark.DarkProxy;\nimport com.twitter.search.common.dark.ResolverProxy;\nimport com.twitter.search.common.dark.ServerSetResolver;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.util.thrift.BytesToThriftFilter;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.util.Duration;\n\npublic class EarlybirdDarkProxy {\n  private static final String WARM_UP_DECIDER_KEY_PREFIX = \"warmup_\";\n\n  private static final int DARK_REQUESTS_TOTAL_REQUEST_TIMEOUT_MS =\n      EarlybirdConfig.getInt(\"dark_requests_total_request_timeout_ms\", 800);\n  private static final int DARK_REQUESTS_INDIVIDUAL_REQUEST_TIMEOUT_MS =\n      EarlybirdConfig.getInt(\"dark_requests_individual_request_timeout_ms\", 800);\n  private static final int DARK_REQUESTS_CONNECT_TIMEOUT_MS =\n      EarlybirdConfig.getInt(\"dark_requests_connect_timeout_ms\", 500);\n  private static final int DARK_REQUESTS_NUM_RETRIES =\n      EarlybirdConfig.getInt(\"dark_requests_num_retries\", 1);\n  private static final String DARK_REQUESTS_FINAGLE_CLIENT_ID =\n      EarlybirdConfig.getString(\"dark_requests_finagle_client_id\", \"earlybird_warmup\");\n\n  private final DarkProxy<ThriftClientRequest, byte[]> darkProxy;\n\n  public EarlybirdDarkProxy(SearchDecider searchDecider,\n                            StatsReceiver statsReceiver,\n                            EarlybirdServerSetManager earlybirdServerSetManager,\n                            EarlybirdWarmUpManager earlybirdWarmUpManager,\n                            String clusterName) {\n    darkProxy = newDarkProxy(searchDecider,\n                             statsReceiver,\n                             earlybirdServerSetManager,\n                             earlybirdWarmUpManager,\n                             clusterName);\n  }\n\n  public DarkProxy<ThriftClientRequest, byte[]> getDarkProxy() {\n    return darkProxy;\n  }\n\n  @VisibleForTesting\n  protected DarkProxy<ThriftClientRequest, byte[]> newDarkProxy(\n      SearchDecider searchDecider,\n      StatsReceiver statsReceiver,\n      EarlybirdServerSetManager earlybirdServerSetManager,\n      final EarlybirdWarmUpManager earlybirdWarmUpManager,\n      String clusterName) {\n    ResolverProxy resolverProxy = new ResolverProxy();\n    ServerSetResolver.SelfServerSetResolver selfServerSetResolver =\n        new ServerSetResolver.SelfServerSetResolver(\n            earlybirdServerSetManager.getServerSetIdentifier(), resolverProxy);\n    selfServerSetResolver.init();\n\n    final String clusterNameForDeciderKey = clusterName.toLowerCase().replaceAll(\"-\", \"_\");\n    final String warmUpServerSetIdentifier = earlybirdWarmUpManager.getServerSetIdentifier();\n    DarkProxy newDarkProxy = new DarkProxy<ThriftClientRequest, byte[]>(\n        selfServerSetResolver,\n        newClientBuilder(statsReceiver),\n        resolverProxy,\n        searchDecider,\n        Lists.newArrayList(warmUpServerSetIdentifier),\n        new BytesToThriftFilter(),\n        statsReceiver) {\n      @Override\n      protected String getServicePathDeciderKey(String servicePath) {\n        if (warmUpServerSetIdentifier.equals(servicePath)) {\n          return WARM_UP_DECIDER_KEY_PREFIX + clusterNameForDeciderKey;\n        }\n\n        return clusterNameForDeciderKey;\n      }\n    };\n\n    newDarkProxy.init();\n    return newDarkProxy;\n  }\n\n  private ClientBuilder<ThriftClientRequest, byte[], ?, Yes, Yes> newClientBuilder(\n      StatsReceiver statsReceiver) {\n    return ClientBuilder.get()\n        .daemon(true)\n        .timeout(Duration.apply(DARK_REQUESTS_TOTAL_REQUEST_TIMEOUT_MS, TimeUnit.MILLISECONDS))\n        .requestTimeout(\n            Duration.apply(DARK_REQUESTS_INDIVIDUAL_REQUEST_TIMEOUT_MS, TimeUnit.MILLISECONDS))\n        .tcpConnectTimeout(Duration.apply(DARK_REQUESTS_CONNECT_TIMEOUT_MS, TimeUnit.MILLISECONDS))\n        .retries(DARK_REQUESTS_NUM_RETRIES)\n        .reportTo(statsReceiver)\n        .tracer(ZipkinTracer.mk(statsReceiver))\n        .stack(new MtlsThriftMuxClient(\n            ThriftMux.client())\n            .withMutualTls(EarlybirdProperty.getServiceIdentifier())\n            .withProtocolFactory(new TCompactProtocol.Factory())\n            .withClientId(new ClientId(DARK_REQUESTS_FINAGLE_CLIENT_ID)));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/EarlybirdFinagleServerManager.java",
    "content": "package com.twitter.search.earlybird;\n\nimport com.twitter.finagle.thrift.ThriftClientRequest;\nimport com.twitter.search.common.dark.DarkProxy;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.util.Duration;\n\n/**\n * Manages a finagle server underneath, which can be recreated.\n *\n * This class is not thread-safe. It is up to the concrete implementations and their callers to\n * correctly synchronize calls to these methods (for example, to make sure that there is no race\n * condition if startProductionFinagleServer() and stopProductionFinagleServer() are called\n * concurrently from two different threads).\n */\npublic interface EarlybirdFinagleServerManager {\n  /**\n   * Determines if the warm up finagle server is currently running\n   */\n  boolean isWarmUpServerRunning();\n\n  /**\n   * Starts up the warm up finagle server on the given port.\n   */\n  void startWarmUpFinagleServer(\n      EarlybirdService.ServiceIface serviceIface,\n      String serviceName,\n      int port);\n\n  /**\n   * Stops the warm up finagle server, after waiting for at most the given amount of time.\n   */\n  void stopWarmUpFinagleServer(Duration serverCloseWaitTime) throws InterruptedException;\n\n  /**\n   * Determines if the production finagle server is currently running.\n   */\n  boolean isProductionServerRunning();\n\n  /**\n   * Starts up the production finagle server on the given port.\n   */\n  void startProductionFinagleServer(\n      DarkProxy<ThriftClientRequest, byte[]> darkProxy,\n      EarlybirdService.ServiceIface serviceIface,\n      String serviceName,\n      int port);\n\n  /**\n   * Stops the production finagle server after waiting for at most the given amount of time.\n   */\n  void stopProductionFinagleServer(Duration serverCloseWaitTime) throws InterruptedException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/EarlybirdFuturePoolManager.java",
    "content": "package com.twitter.search.earlybird;\n\nimport java.util.concurrent.ArrayBlockingQueue;\nimport java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\nimport java.util.concurrent.RejectedExecutionException;\nimport java.util.concurrent.ThreadFactory;\nimport java.util.concurrent.ThreadPoolExecutor;\nimport java.util.concurrent.TimeUnit;\n\nimport scala.Function0;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.util.concurrent.ThreadFactoryBuilder;\n\nimport com.twitter.search.common.concurrent.ThreadPoolExecutorStats;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.util.ExecutorServiceFuturePool;\nimport com.twitter.util.Future;\nimport com.twitter.util.FuturePool;\n\n/**\n * A future pool that delegates all calls to an underlying futurePool, which can be recreated.\n */\npublic class EarlybirdFuturePoolManager implements FuturePool {\n  private volatile ExecutorServiceFuturePool pool = null;\n\n  private final String threadName;\n  private final ThreadPoolExecutorStats threadPoolExecutorStats;\n\n  public EarlybirdFuturePoolManager(String threadName) {\n    this.threadName = threadName;\n    this.threadPoolExecutorStats = new ThreadPoolExecutorStats(threadName);\n  }\n\n  final synchronized void createUnderlyingFuturePool(int threadCount) {\n    Preconditions.checkState(pool == null, \"Cannot create a new pool before stopping the old one\");\n\n    ExecutorService executorService =\n        createExecutorService(threadCount, getMaxQueueSize());\n    if (executorService instanceof ThreadPoolExecutor) {\n      threadPoolExecutorStats.setUnderlyingExecutorForStats((ThreadPoolExecutor) executorService);\n    }\n\n    pool = new ExecutorServiceFuturePool(executorService);\n  }\n\n  final synchronized void stopUnderlyingFuturePool(long timeout, TimeUnit timeunit)\n      throws InterruptedException {\n    Preconditions.checkNotNull(pool);\n    pool.executor().shutdown();\n    pool.executor().awaitTermination(timeout, timeunit);\n    pool = null;\n  }\n\n  boolean isPoolReady() {\n    return pool != null;\n  }\n\n  @Override\n  public final <T> Future<T> apply(Function0<T> f) {\n    return Preconditions.checkNotNull(pool).apply(f);\n  }\n\n  @VisibleForTesting\n  protected ExecutorService createExecutorService(int threadCount, int maxQueueSize) {\n    if (maxQueueSize <= 0) {\n      return Executors.newFixedThreadPool(threadCount, createThreadFactory(threadName));\n    }\n\n    SearchRateCounter rejectedTaskCounter =\n        SearchRateCounter.export(threadName + \"_rejected_task_count\");\n    return new ThreadPoolExecutor(\n        threadCount, threadCount, 0, TimeUnit.MILLISECONDS,\n        new ArrayBlockingQueue<>(maxQueueSize),\n        createThreadFactory(threadName),\n        (runnable, executor) -> {\n          rejectedTaskCounter.increment();\n          throw new RejectedExecutionException(threadName + \" queue is full\");\n        });\n  }\n\n  @VisibleForTesting\n  protected int getMaxQueueSize() {\n    return EarlybirdProperty.MAX_QUEUE_SIZE.get(0);\n  }\n\n  @VisibleForTesting\n  static ThreadFactory createThreadFactory(String threadName) {\n    return new ThreadFactoryBuilder()\n        .setNameFormat(threadName + \"-%d\")\n        .setDaemon(true)\n        .build();\n  }\n\n  @Override\n  public int poolSize() {\n    return Preconditions.checkNotNull(pool).poolSize();\n  }\n\n  @Override\n  public int numActiveTasks() {\n    return Preconditions.checkNotNull(pool).numActiveTasks();\n  }\n\n  @Override\n  public long numCompletedTasks() {\n    return Preconditions.checkNotNull(pool).numCompletedTasks();\n  }\n\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/EarlybirdIndexConfig.java",
    "content": "package com.twitter.search.earlybird;\n\nimport java.io.IOException;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Predicate;\nimport com.google.common.base.Predicates;\n\nimport org.apache.lucene.index.IndexWriterConfig;\nimport org.apache.lucene.store.Directory;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.schema.DynamicSchema;\nimport com.twitter.search.common.schema.base.Schema.SchemaValidationException;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdSchemaCreateTool;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.util.CloseResourceUtil;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentData;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdIndexExtensionsFactory;\nimport com.twitter.search.earlybird.document.DocumentFactory;\nimport com.twitter.search.earlybird.document.ThriftIndexingEventDocumentFactory;\nimport com.twitter.search.earlybird.document.ThriftIndexingEventUpdateFactory;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.partition.PartitionConfig;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\nimport com.twitter.search.earlybird.partition.SegmentSyncInfo;\nimport com.twitter.search.earlybird.partition.UserPartitionUtil;\n\n/**\n * Collection of required indexing entities that differ in the various Earlybird clusters.\n */\npublic abstract class EarlybirdIndexConfig {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdIndexConfig.class);\n\n  private final EarlybirdCluster cluster;\n  private final DynamicSchema schema;\n  private final Decider decider;\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n  protected final CriticalExceptionHandler criticalExceptionHandler;\n\n  /**\n   * Creates a new index config using an applicable schema built for the provided cluster.\n   */\n  protected EarlybirdIndexConfig(\n      EarlybirdCluster cluster, Decider decider, SearchIndexingMetricSet searchIndexingMetricSet,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    this(cluster, buildSchema(cluster), decider, searchIndexingMetricSet,\n        criticalExceptionHandler);\n  }\n\n  @VisibleForTesting\n  protected EarlybirdIndexConfig(\n      EarlybirdCluster cluster,\n      DynamicSchema schema,\n      Decider decider,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    this.cluster = cluster;\n    this.schema = schema;\n    this.decider = decider;\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n    this.criticalExceptionHandler = criticalExceptionHandler;\n    LOG.info(\"This Earlybird uses index config: \" + this.getClass().getSimpleName());\n  }\n\n  private static DynamicSchema buildSchema(EarlybirdCluster cluster) {\n    try {\n      return EarlybirdSchemaCreateTool.buildSchema(cluster);\n    } catch (SchemaValidationException e) {\n      throw new RuntimeException(e);\n    }\n  }\n\n  /**\n   * Creates the appropriate document factory for this earlybird.\n   */\n  public final DocumentFactory<ThriftIndexingEvent> createDocumentFactory() {\n    return new ThriftIndexingEventDocumentFactory(\n        getSchema(), getCluster(), decider, searchIndexingMetricSet,\n        criticalExceptionHandler);\n  }\n\n  /**\n   * Creates a document factory for ThriftIndexingEvents that are updates to the index.\n   */\n  public final DocumentFactory<ThriftIndexingEvent> createUpdateFactory() {\n    return new ThriftIndexingEventUpdateFactory(\n        getSchema(), getCluster(), decider, criticalExceptionHandler);\n  }\n\n  /**\n   * Return the EarlybirdCluster enum identifying the cluster this config is for.\n   */\n  public final EarlybirdCluster getCluster() {\n    return cluster;\n  }\n\n  /**\n   * Return the default filter for UserUpdatesTable - for the archive cluster keep\n   * users that belong to the current partition.\n   */\n  public final Predicate<Long> getUserTableFilter(PartitionConfig partitionConfig) {\n    if (EarlybirdCluster.isArchive(getCluster())) {\n      return UserPartitionUtil.filterUsersByPartitionPredicate(partitionConfig);\n    }\n\n    return Predicates.alwaysTrue();\n  }\n\n  /**\n   * Creates a new Lucene {@link Directory} to be used for indexing documents.\n   */\n  public abstract Directory newLuceneDirectory(SegmentSyncInfo segmentSyncInfo) throws IOException;\n\n  /**\n   * Creates a new Lucene IndexWriterConfig that can be used for creating a segment writer for a\n   * new segment.\n   */\n  public abstract IndexWriterConfig newIndexWriterConfig();\n\n  /**\n   * Creates a new SegmentData object to add documents to.\n   */\n  public abstract EarlybirdIndexSegmentData newSegmentData(\n      int maxSegmentSize,\n      long timeSliceID,\n      Directory dir,\n      EarlybirdIndexExtensionsFactory extensionsFactory);\n\n  /**\n   * Loads a flushed index for the given segment.\n   */\n  public abstract EarlybirdIndexSegmentData loadSegmentData(\n      FlushInfo flushInfo,\n      DataDeserializer dataInputStream,\n      Directory dir,\n      EarlybirdIndexExtensionsFactory extensionsFactory) throws IOException;\n\n  /**\n   * Creates a new segment optimizer for the given segment data.\n   */\n  public abstract EarlybirdIndexSegmentData optimize(\n      EarlybirdIndexSegmentData earlybirdIndexSegmentData) throws IOException;\n\n  /**\n   * Whether the index is stored on disk or not. If an index is not on disk, it is presumed to be\n   * in memory.\n   */\n  public abstract boolean isIndexStoredOnDisk();\n\n  /**\n   * Whether documents are search in LIFO ordering (RT mode), or default (Lucene) FIFO ordering\n   */\n  public final boolean isUsingLIFODocumentOrdering() {\n    return !isIndexStoredOnDisk();\n  }\n\n  /**\n   * Whether this index supports out-of-order indexing\n   */\n  public abstract boolean supportOutOfOrderIndexing();\n\n  /**\n   * Returns a CloseResourceUtil used for closing resources.\n   */\n  public abstract CloseResourceUtil getResourceCloser();\n\n  /**\n   * Returns the schema for this index configuration.\n   */\n  public final DynamicSchema getSchema() {\n    return schema;\n  }\n\n  /**\n   * Returns the decider used by this EarlybirdIndexConfig instance.\n   */\n  public Decider getDecider() {\n    return decider;\n  }\n\n  public SearchIndexingMetricSet getSearchIndexingMetricSet() {\n    return searchIndexingMetricSet;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/EarlybirdMain.java",
    "content": "package com.twitter.search.earlybird;\n\npublic final class EarlybirdMain {\n  private EarlybirdMain() {\n  }\n\n  public static void main(String[] args) {\n    new Earlybird().main(args);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/EarlybirdProductionFinagleServerManager.java",
    "content": "package com.twitter.search.earlybird;\n\nimport java.net.InetSocketAddress;\nimport java.util.concurrent.atomic.AtomicReference;\n\nimport org.apache.thrift.protocol.TCompactProtocol;\nimport org.apache.thrift.protocol.TProtocolFactory;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.ListeningServer;\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SslException;\nimport com.twitter.finagle.ThriftMux;\nimport com.twitter.finagle.mtls.server.MtlsThriftMuxServer;\nimport com.twitter.finagle.mux.transport.OpportunisticTls;\nimport com.twitter.finagle.stats.MetricsStatsReceiver;\nimport com.twitter.finagle.thrift.ThriftClientRequest;\nimport com.twitter.finagle.util.ExitGuard;\nimport com.twitter.finagle.zipkin.thrift.ZipkinTracer;\nimport com.twitter.search.common.dark.DarkProxy;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.exception.EarlybirdFinagleServerMonitor;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.server.filter.AdmissionControl;\nimport com.twitter.server.filter.cpuAdmissionControl;\nimport com.twitter.util.Await;\nimport com.twitter.util.Duration;\nimport com.twitter.util.TimeoutException;\n\npublic class EarlybirdProductionFinagleServerManager implements EarlybirdFinagleServerManager {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(EarlybirdProductionFinagleServerManager.class);\n\n  private final AtomicReference<ListeningServer> warmUpFinagleServer = new AtomicReference<>();\n  private final AtomicReference<ListeningServer> productionFinagleServer = new AtomicReference<>();\n  private final EarlybirdFinagleServerMonitor unhandledExceptionMonitor;\n\n  public EarlybirdProductionFinagleServerManager(\n      CriticalExceptionHandler criticalExceptionHandler) {\n    this.unhandledExceptionMonitor =\n        new EarlybirdFinagleServerMonitor(criticalExceptionHandler);\n  }\n\n  @Override\n  public boolean isWarmUpServerRunning() {\n    return warmUpFinagleServer.get() != null;\n  }\n\n  @Override\n  public void startWarmUpFinagleServer(EarlybirdService.ServiceIface serviceIface,\n                                       String serviceName,\n                                       int port) {\n    TProtocolFactory protocolFactory = new TCompactProtocol.Factory();\n    startFinagleServer(warmUpFinagleServer, \"warmup\",\n      new EarlybirdService.Service(serviceIface, protocolFactory),\n      protocolFactory, serviceName, port);\n  }\n\n  @Override\n  public void stopWarmUpFinagleServer(Duration serverCloseWaitTime) throws InterruptedException {\n    stopFinagleServer(warmUpFinagleServer, serverCloseWaitTime, \"Warm up\");\n  }\n\n  @Override\n  public boolean isProductionServerRunning() {\n    return productionFinagleServer.get() != null;\n  }\n\n  @Override\n  public void startProductionFinagleServer(DarkProxy<ThriftClientRequest, byte[]> darkProxy,\n                                           EarlybirdService.ServiceIface serviceIface,\n                                           String serviceName,\n                                           int port) {\n    TProtocolFactory protocolFactory = new TCompactProtocol.Factory();\n    startFinagleServer(productionFinagleServer, \"production\",\n      darkProxy.toFilter().andThen(new EarlybirdService.Service(serviceIface, protocolFactory)),\n      protocolFactory, serviceName, port);\n  }\n\n  @Override\n  public void stopProductionFinagleServer(Duration serverCloseWaitTime)\n      throws InterruptedException {\n    stopFinagleServer(productionFinagleServer, serverCloseWaitTime, \"Production\");\n  }\n\n  private void startFinagleServer(AtomicReference target, String serverDescription,\n      Service<byte[], byte[]> service, TProtocolFactory protocolFactory, String serviceName,\n      int port) {\n    target.set(getServer(service, serviceName, port, protocolFactory));\n    LOG.info(\"Started EarlybirdServer \" + serverDescription + \" finagle server on port \" + port);\n  }\n\n  private ListeningServer getServer(\n      Service<byte[], byte[]> service, String serviceName, int port,\n      TProtocolFactory protocolFactory) {\n    MetricsStatsReceiver statsReceiver = new MetricsStatsReceiver();\n    ThriftMux.Server server = new MtlsThriftMuxServer(ThriftMux.server())\n        .withMutualTls(EarlybirdProperty.getServiceIdentifier())\n        .withServiceClass(EarlybirdService.class)\n        .withOpportunisticTls(OpportunisticTls.Required())\n        .withLabel(serviceName)\n        .withStatsReceiver(statsReceiver)\n        .withTracer(ZipkinTracer.mk(statsReceiver))\n        .withMonitor(unhandledExceptionMonitor)\n        .withProtocolFactory(protocolFactory);\n\n    if (cpuAdmissionControl.isDefined()) {\n      LOG.info(\"cpuAdmissionControl flag is set, replacing AuroraThrottlingAdmissionFilter\"\n          + \" with LinuxCpuAdmissionFilter\");\n      server = server\n          .configured(AdmissionControl.auroraThrottling().off().mk())\n          .configured(AdmissionControl.linuxCpu().useGlobalFlag().mk());\n    }\n\n    return server.serve(new InetSocketAddress(port), service);\n  }\n\n  private void stopFinagleServer(AtomicReference<ListeningServer> finagleServer,\n                                 Duration serverCloseWaitTime,\n                                 String serverDescription) throws InterruptedException {\n    try {\n      LOG.info(\"Waiting for \" + serverDescription + \" finagle server to close. \"\n               + \"Current time is \" + System.currentTimeMillis());\n      Await.result(finagleServer.get().close(), serverCloseWaitTime);\n      LOG.info(\"Stopped \" + serverDescription + \" finagle server. Current time is \"\n               + System.currentTimeMillis());\n      finagleServer.set(null);\n    } catch (TimeoutException e) {\n      LOG.warn(serverDescription + \" finagle server did not shutdown cleanly.\", e);\n    } catch (SslException e) {\n      // Closing the Thrift port seems to throw an SSLException (SSLEngine closed already).\n      // See SEARCH-29449. Log the exception and reset finagleServer, so that future calls to\n      // startProductionFinagleServer() succeed.\n      LOG.warn(\"Got a SSLException while trying to close the Thrift port.\", e);\n      finagleServer.set(null);\n    } catch (InterruptedException e) {\n      // If we catch an InterruptedException here, it means that we're probably shutting down.\n      // We should propagate this exception, and rely on EarlybirdServer.stopThriftService()\n      // to do the right thing.\n      throw e;\n    } catch (Exception e) {\n      LOG.error(e.getMessage(), e);\n    } finally {\n      // If the finagle server does not close cleanly, this line prints details about\n      // the ExitGuards.\n      LOG.info(serverDescription + \" server ExitGuard explanation: \" + ExitGuard.explainGuards());\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/EarlybirdSearcher.java",
    "content": "package com.twitter.search.earlybird;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.stream.Collectors;\nimport javax.annotation.Nonnull;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Joiner;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.ImmutableSet;\nimport com.google.common.collect.Lists;\n\nimport org.apache.commons.lang.StringUtils;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.queryparser.classic.ParseException;\nimport org.apache.lucene.queryparser.classic.QueryParser;\nimport org.apache.lucene.search.BooleanClause.Occur;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.Query;\nimport org.apache.thrift.TException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.database.DatabaseConfig;\nimport com.twitter.search.common.decider.DeciderUtil;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchema;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.partitioning.base.Segment;\nimport com.twitter.search.common.query.MappableField;\nimport com.twitter.search.common.query.QueryHitAttributeHelper;\nimport com.twitter.search.common.query.thriftjava.CollectorParams;\nimport com.twitter.search.common.query.thriftjava.CollectorTerminationParams;\nimport com.twitter.search.common.query.thriftjava.EarlyTerminationInfo;\nimport com.twitter.search.common.ranking.thriftjava.ThriftRankingParams;\nimport com.twitter.search.common.ranking.thriftjava.ThriftScoringFunctionType;\nimport com.twitter.search.common.results.thriftjava.FieldHitList;\nimport com.twitter.search.common.schema.SchemaUtil;\nimport com.twitter.search.common.schema.SearchWhitespaceAnalyzer;\nimport com.twitter.search.common.schema.base.FieldWeightDefault;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.common.search.TwitterEarlyTerminationCollector;\nimport com.twitter.search.common.search.termination.QueryTimeoutFactory;\nimport com.twitter.search.common.util.earlybird.EarlybirdResponseUtil;\nimport com.twitter.search.common.util.ml.tensorflow_engine.TensorflowModelsManager;\nimport com.twitter.search.common.util.thrift.ThriftUtils;\nimport com.twitter.search.core.earlybird.facets.FacetCountState;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.exception.ClientException;\nimport com.twitter.search.earlybird.exception.TransientException;\nimport com.twitter.search.earlybird.index.facets.FacetSkipList;\nimport com.twitter.search.earlybird.ml.ScoringModelsManager;\nimport com.twitter.search.earlybird.partition.AudioSpaceTable;\nimport com.twitter.search.earlybird.partition.MultiSegmentTermDictionaryManager;\nimport com.twitter.search.earlybird.partition.PartitionConfig;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentManager;\nimport com.twitter.search.earlybird.querycache.QueryCacheConversionRules;\nimport com.twitter.search.earlybird.querycache.QueryCacheManager;\nimport com.twitter.search.earlybird.queryparser.DetectFieldAnnotationVisitor;\nimport com.twitter.search.earlybird.queryparser.EarlybirdLuceneQueryVisitor;\nimport com.twitter.search.earlybird.queryparser.HighFrequencyTermPairRewriteVisitor;\nimport com.twitter.search.earlybird.queryparser.LuceneRelevanceQueryVisitor;\nimport com.twitter.search.earlybird.queryparser.ProtectedOperatorQueryRewriter;\nimport com.twitter.search.earlybird.search.AbstractResultsCollector;\nimport com.twitter.search.earlybird.search.AntiGamingFilter;\nimport com.twitter.search.earlybird.search.queries.BadUserRepFilter;\nimport com.twitter.search.earlybird.search.EarlybirdLuceneSearcher;\nimport com.twitter.search.earlybird.search.EarlybirdMultiSegmentSearcher;\nimport com.twitter.search.earlybird.search.queries.MatchAllDocsQuery;\nimport com.twitter.search.earlybird.search.queries.RequiredStatusIDsFilter;\nimport com.twitter.search.earlybird.search.SearchRequestInfo;\nimport com.twitter.search.earlybird.search.SearchResultsCollector;\nimport com.twitter.search.earlybird.search.SearchResultsInfo;\nimport com.twitter.search.earlybird.search.SimpleSearchResults;\nimport com.twitter.search.earlybird.search.SocialFilter;\nimport com.twitter.search.earlybird.search.SocialSearchResultsCollector;\nimport com.twitter.search.earlybird.search.queries.UserFlagsExcludeFilter;\nimport com.twitter.search.earlybird.search.queries.UserIdMultiSegmentQuery;\nimport com.twitter.search.earlybird.search.facets.EntityAnnotationCollector;\nimport com.twitter.search.earlybird.search.facets.ExpandedUrlCollector;\nimport com.twitter.search.earlybird.search.facets.ExplainFacetResultsCollector;\nimport com.twitter.search.earlybird.search.facets.FacetRankingModule;\nimport com.twitter.search.earlybird.search.facets.FacetResultsCollector;\nimport com.twitter.search.earlybird.search.facets.FacetSearchRequestInfo;\nimport com.twitter.search.earlybird.search.facets.NamedEntityCollector;\nimport com.twitter.search.earlybird.search.facets.SpaceFacetCollector;\nimport com.twitter.search.earlybird.search.facets.TermStatisticsCollector;\nimport com.twitter.search.earlybird.search.facets.TermStatisticsRequestInfo;\nimport com.twitter.search.earlybird.search.relevance.RelevanceSearchRequestInfo;\nimport com.twitter.search.earlybird.search.relevance.RelevanceSearchResults;\nimport com.twitter.search.earlybird.search.relevance.collectors.AbstractRelevanceCollector;\nimport com.twitter.search.earlybird.search.relevance.collectors.BatchRelevanceTopCollector;\nimport com.twitter.search.earlybird.search.relevance.collectors.RelevanceAllCollector;\nimport com.twitter.search.earlybird.search.relevance.collectors.RelevanceTopCollector;\nimport com.twitter.search.earlybird.search.relevance.scoring.RelevanceQuery;\nimport com.twitter.search.earlybird.search.relevance.scoring.ScoringFunction;\nimport com.twitter.search.earlybird.search.relevance.scoring.ScoringFunctionProvider;\nimport com.twitter.search.earlybird.search.relevance.scoring.TensorflowBasedScoringFunction;\nimport com.twitter.search.earlybird.stats.EarlybirdRPCStats;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.EarlybirdDebugInfo;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCount;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCountMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftFacetFieldRequest;\nimport com.twitter.search.earlybird.thrift.ThriftFacetFieldResults;\nimport com.twitter.search.earlybird.thrift.ThriftFacetRequest;\nimport com.twitter.search.earlybird.thrift.ThriftFacetResults;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchRankingMode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchRelevanceOptions;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultExtraMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadataOptions;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird.thrift.ThriftTermRequest;\nimport com.twitter.search.earlybird.thrift.ThriftTermStatisticsRequest;\nimport com.twitter.search.earlybird.thrift.ThriftTermStatisticsResults;\nimport com.twitter.search.earlybird.util.EarlybirdSearchResultUtil;\nimport com.twitter.search.queryparser.parser.SerializedQueryParser;\nimport com.twitter.search.queryparser.query.Conjunction;\nimport com.twitter.search.queryparser.query.Disjunction;\nimport com.twitter.search.queryparser.query.QueryNodeUtils;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.annotation.Annotation;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\nimport com.twitter.search.queryparser.query.search.SearchOperatorConstants;\nimport com.twitter.search.queryparser.util.IdTimeRanges;\nimport com.twitter.search.queryparser.visitors.ConversionVisitor;\nimport com.twitter.search.queryparser.visitors.DetectPositiveOperatorVisitor;\nimport com.twitter.search.queryparser.visitors.NamedDisjunctionVisitor;\nimport com.twitter.search.queryparser.visitors.ProximityGroupRewriteVisitor;\nimport com.twitter.search.queryparser.visitors.StripAnnotationsVisitor;\n\nimport static com.twitter.search.queryparser.query.search.SearchOperator.Type.UNTIL_TIME;\n\n/**\n * This class provides the basic search() method:\n * - converts the thrift request object into what lucene expects.\n * - gets the segment.\n * - handles all errors, and prepares the response in case of error.\n *\n * We have one instance of this class per search received.\n */\npublic class EarlybirdSearcher {\n  public enum QueryMode {\n    // Please think before adding more query modes: can this be implemented in a general way?\n    RECENCY(new EarlybirdRPCStats(\"search_recency\")),\n    FACETS(new EarlybirdRPCStats(\"search_facets\")),\n    TERM_STATS(new EarlybirdRPCStats(\"search_termstats\")),\n    RELEVANCE(new EarlybirdRPCStats(\"search_relevance\")),\n    TOP_TWEETS(new EarlybirdRPCStats(\"search_toptweets\"));\n\n    private final EarlybirdRPCStats requestStats;\n\n    QueryMode(EarlybirdRPCStats requestStats) {\n      this.requestStats = requestStats;\n    }\n\n    public EarlybirdRPCStats getRequestStats() {\n      return requestStats;\n    }\n  }\n\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdSearcher.class);\n  private static final String MATCH_ALL_SERIALIZED_QUERY = \"(* )\";\n  /**\n   * generic field annotations can be mapped to a concrete field in the index using this mapping\n   * via {@link com.twitter.search.queryparser.query.annotation.Annotation.Type#MAPPABLE_FIELD}\n   */\n  private static final Map<MappableField, String> MAPPABLE_FIELD_MAP =\n      ImmutableMap.of(\n          MappableField.URL,\n          EarlybirdFieldConstant.RESOLVED_LINKS_TEXT_FIELD.getFieldName());\n\n  private static final String ALLOW_QUERY_SPECIFIC_SIGNAL_DECIDER_KEY\n      = \"allow_query_specific_score_adjustments\";\n\n  @VisibleForTesting\n  public static final String ALLOW_AUTHOR_SPECIFIC_SIGNAL_DECIDER_KEY\n      = \"allow_author_specific_score_adjustments\";\n\n  private static final String USE_MULTI_TERM_DISJUNCTION_FOR_LIKED_BY_USER_IDS_DECIDER_KEY\n      = \"use_multi_term_disjunction_for_liked_by_user_ids\";\n\n  private static final String ALLOW_CAMELCASE_USERNAME_FIELD_WEIGHT_OVERRIDE_DECIDER_KEY_PREFIX\n      = \"allow_camelcase_username_field_weight_override_in_\";\n\n  private static final String ALLOW_TOKENIZED_DISPLAY_NAME_FIELD_WEIGHT_OVERRIDE_DECIDER_KEY_PREFIX\n      = \"allow_tokenized_display_name_field_weight_override_in_\";\n\n  private static final boolean ALLOW_QUERY_SPECIFIC_SIGNAL_CONFIG\n      = EarlybirdConfig.getBool(\"allow_query_specific_score_adjustments\", false);\n\n  private static final boolean ALLOW_AUTHOR_SPECIFIC_SIGNAL_CONFIG\n      = EarlybirdConfig.getBool(\"allow_author_specific_score_adjustments\", false);\n\n  public static final int DEFAULT_NUM_FACET_RESULTS = 100;\n\n  private final ImmutableSchemaInterface schemaSnapshot;\n  private final EarlybirdCluster cluster;\n\n  private final Clock clock;\n  private final Decider decider;\n\n  // The actual request thrift.\n  private final EarlybirdRequest request;\n\n  // searchQuery from inside the request.\n  private final ThriftSearchQuery searchQuery;\n\n  // CollectorParams from inside the searchQuery;\n  private final CollectorParams collectorParams;\n\n  // Parsed query (parsed from serialized query string in request).\n  private com.twitter.search.queryparser.query.Query parsedQuery;\n  private boolean parsedQueryAllowNullcast;\n  private IdTimeRanges idTimeRanges;\n\n  // Lucene version of the above.  This is what we will actually be executing.\n  private org.apache.lucene.search.Query luceneQuery;\n\n  // Used for queries where we want to collect per-field hit attribution\n  @Nullable\n  private QueryHitAttributeHelper hitAttributeHelper;\n\n  // Debugging info can be appended to this buffer.\n  private final StringBuilder messageBuffer = new StringBuilder(1024);\n  private final EarlybirdDebugInfo debugInfo = new EarlybirdDebugInfo();\n\n  // The segment we are searching, or null for the multi-searcher.\n  private Segment segment = null;\n\n  // True iff we are searching all segments (multi-searcher).\n  private final boolean searchAllSegments;\n\n  // Tracking termination criteria for this query\n  private final TerminationTracker terminationTracker;\n\n  private EarlybirdLuceneSearcher searcher = null;\n\n  private final SegmentManager segmentManager;\n  private final QueryCacheManager queryCacheManager;\n  private final ScoringModelsManager scoringModelsManager;\n  private final TensorflowModelsManager tensorflowModelsManager;\n\n  private AntiGamingFilter antiGamingFilter = null;\n\n  private final boolean searchHighFrequencyTermPairs =\n      EarlybirdConfig.getBool(\"search_high_frequency_term_pairs\", false);\n\n  // How long to allow post-termination when enforcing query timeout\n  private final int enforceQueryTimeoutBufferMillis =\n      EarlybirdConfig.getInt(\"enforce_query_timeout_buffer_millis\", 50);\n\n  private EarlybirdRPCStats requestStats;\n\n  private QueryTimeoutFactory queryTimeoutFactory;\n\n  // Exported stats\n  private final EarlybirdSearcherStats searcherStats;\n\n  @VisibleForTesting\n  public static final SearchCounter FIELD_WEIGHT_OVERRIDE_MAP_NON_NULL_COUNT =\n      SearchCounter.export(\"field_weight_override_map_non_null_count\");\n  @VisibleForTesting\n  public static final SearchCounter DROPPED_CAMELCASE_USERNAME_FIELD_WEIGHT_OVERRIDE =\n      SearchCounter.export(\"dropped_camelcase_username_field_weight_override\");\n  @VisibleForTesting\n  public static final SearchCounter DROPPED_TOKENIZED_DISPLAY_NAME_FIELD_WEIGHT_OVERRIDE =\n      SearchCounter.export(\"dropped_tokenized_display_name_field_weight_override\");\n\n  private static final SearchCounter RESPONSE_HAS_NO_THRIFT_SEARCH_RESULTS =\n      SearchCounter.export(\"tweets_earlybird_searcher_response_has_no_thrift_search_results\");\n  private static final SearchCounter CLIENT_HAS_FEATURE_SCHEMA_COUNTER =\n      SearchCounter.export(\"tweets_earlybird_searcher_client_has_feature_schema\");\n  private static final SearchCounter CLIENT_DOESNT_HAVE_FEATURE_SCHEMA_COUNTER =\n      SearchCounter.export(\"tweet_earlybird_searcher_client_doesnt_have_feature_schema\");\n  private static final SearchCounter COLLECTOR_PARAMS_MAX_HITS_TO_PROCESS_NOT_SET_COUNTER =\n      SearchCounter.export(\"collector_params_max_hits_to_process_not_set\");\n  private static final SearchCounter POSITIVE_PROTECTED_OPERATOR_DETECTED_COUNTER =\n      SearchCounter.export(\"positive_protected_operator_detected_counter\");\n\n  // Query mode we are executing.\n  private final QueryMode queryMode;\n\n  // facetRequest from inside the request (or null).\n  private final ThriftFacetRequest facetRequest;\n\n  // termStatisticsRequest from inside the request (or null).\n  private final ThriftTermStatisticsRequest termStatisticsRequest;\n\n  // Results fields filled in during searchInternal().\n  private ThriftSearchResults searchResults = null;\n  private ThriftFacetResults facetResults = null;\n  private ThriftTermStatisticsResults termStatisticsResults = null;\n  private EarlyTerminationInfo earlyTerminationInfo = null;\n\n  // Partition config used to fill in debugging info.\n  // If null, no debug info is written into results.\n  @Nullable\n  private final PartitionConfig partitionConfig;\n\n  private final MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager;\n\n  private final QualityFactor qualityFactor;\n\n  private Set<String> queriedFields;\n  private final AudioSpaceTable audioSpaceTable;\n\n  public EarlybirdSearcher(\n      EarlybirdRequest request,\n      SegmentManager segmentManager,\n      AudioSpaceTable audioSpaceTable,\n      QueryCacheManager queryCacheManager,\n      ImmutableSchemaInterface schema,\n      EarlybirdCluster cluster,\n      @Nullable PartitionConfig partitionConfig,\n      Decider decider,\n      EarlybirdSearcherStats searcherStats,\n      ScoringModelsManager scoringModelsManager,\n      TensorflowModelsManager tensorflowModelsManager,\n      Clock clock,\n      MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager,\n      QueryTimeoutFactory queryTimeoutFactory,\n      QualityFactor qualityFactor) {\n    this.queryMode = getQueryMode(request);\n    this.schemaSnapshot = schema.getSchemaSnapshot();\n    // set the request stats as early as possible, so that we can track errors that happen\n    // early on in query processing.\n    this.requestStats = queryMode.getRequestStats();\n    this.facetRequest = request.isSetFacetRequest() ? request.getFacetRequest() : null;\n    this.termStatisticsRequest = request.isSetTermStatisticsRequest()\n        ? request.getTermStatisticsRequest() : null;\n    this.partitionConfig = partitionConfig;\n    this.searcherStats = searcherStats;\n    this.multiSegmentTermDictionaryManager = multiSegmentTermDictionaryManager;\n    this.clock = clock;\n    this.decider = decider;\n    this.request = request;\n    this.segmentManager = segmentManager;\n    this.queryCacheManager = queryCacheManager;\n    this.cluster = cluster;\n    this.scoringModelsManager = scoringModelsManager;\n    this.tensorflowModelsManager = tensorflowModelsManager;\n    this.audioSpaceTable = audioSpaceTable;\n    // Note: we're deferring the validation/nullchecks until validateRequest()\n    // for more contained exception handling\n    this.searchQuery = request.getSearchQuery();\n    this.collectorParams = this.searchQuery == null ? null : this.searchQuery.getCollectorParams();\n    // Search all segments if searchSegmentId is unset.\n    this.searchAllSegments = !request.isSetSearchSegmentId();\n    if (this.collectorParams == null\n        || !this.collectorParams.isSetTerminationParams()) {\n      this.terminationTracker = new TerminationTracker(clock);\n    } else if (request.isSetClientRequestTimeMs()) {\n      this.terminationTracker = new TerminationTracker(collectorParams.getTerminationParams(),\n          request.getClientRequestTimeMs(), clock,\n          getPostTerminationOverheadMillis(collectorParams.getTerminationParams()));\n    } else {\n      this.terminationTracker = new TerminationTracker(\n          collectorParams.getTerminationParams(), clock,\n          getPostTerminationOverheadMillis(collectorParams.getTerminationParams()));\n    }\n    this.queryTimeoutFactory = queryTimeoutFactory;\n    this.qualityFactor = qualityFactor;\n  }\n\n  private int getPostTerminationOverheadMillis(CollectorTerminationParams terminationParams) {\n    // If enforcing timeouts, set the post-termination buffer to the smaller of the timeout or the\n    // configured buffer. This ensures that timeout >= buffer, and a request with a smaller timeout\n    // should just time out immediately (because timeout == buffer).\n    return (terminationParams.isEnforceQueryTimeout() && terminationParams.getTimeoutMs() > 0)\n        ? Math.min(enforceQueryTimeoutBufferMillis, terminationParams.getTimeoutMs()) : 0;\n  }\n\n  // Appends a debug string to the buffer.\n  private void appendMessage(String message) {\n    messageBuffer.append(message).append(\"\\n\");\n  }\n\n  /**\n   * Processes an Earlybird search request.\n   * @return the earlybird response for this search request.\n   */\n  public EarlybirdResponse search() {\n    try {\n      debugInfo.setHost(DatabaseConfig.getLocalHostname());\n\n      // Throws transient exception for invalid requests.\n      validateRequest();\n\n      // Throws client exception for bad queries,\n      parseEarlybirdRequest();\n\n      // Modify the Lucene query if necessary.\n      luceneQuery = postLuceneQueryProcess(luceneQuery);\n\n      // Might return PARTITION_NOT_FOUND or PARTITION_DISABLED.\n      EarlybirdResponseCode code = initSearcher();\n      if (code != EarlybirdResponseCode.SUCCESS) {\n        return respondError(code);\n      }\n\n      return searchInternal();\n\n    } catch (TransientException e) {\n      LOG.error(String.format(\"Transient exception in search() for EarlybirdRequest:\\n%s\", request),\n                e);\n      appendMessage(e.getMessage());\n      return respondError(EarlybirdResponseCode.TRANSIENT_ERROR);\n    } catch (ClientException e) {\n      LOG.warn(String.format(\"Client exception in search() %s for EarlybirdRequest:\\n %s\",\n          e, request));\n      appendMessage(e.getMessage());\n      return respondError(EarlybirdResponseCode.CLIENT_ERROR);\n    } catch (Exception e) {\n      LOG.warn(String.format(\"Uncaught exception in search() for EarlybirdRequest:\\n%s\", request),\n               e);\n      appendMessage(e.getMessage());\n      return respondError(EarlybirdResponseCode.TRANSIENT_ERROR);\n    } catch (AssertionError e) {\n      LOG.warn(String.format(\"Assertion error in search() for EarlybirdRequest:\\n%s\", request), e);\n      appendMessage(e.getMessage());\n      return respondError(EarlybirdResponseCode.TRANSIENT_ERROR);\n    } catch (Error e) {\n      // SEARCH-33166: If we got here, it means what was thrown was not an Exception, or anything\n      // we know how to handle. Log the Error for diagnostic purposes and propagate it.\n      LOG.error(\"Re-throwing uncaught error\", e);\n      throw e;\n    }\n  }\n\n  public EarlybirdRPCStats getRequestStats() {\n    return requestStats;\n  }\n\n  /**\n   * Wraps the given query with the provided filter queries.\n   *\n   * @param query the query to wrap with filters.\n   * @param filters the filters to wrap the query with.\n   * @return a BooleanQuery wrapped with filters\n   */\n  public static Query wrapFilters(Query query, Query... filters) {\n    boolean filtersEmpty = filters == null || filters.length == 0;\n\n    if (!filtersEmpty) {\n      filtersEmpty = true;\n      for (Query f : filters) {\n        if (f != null) {\n          filtersEmpty = false;\n          break;\n        }\n      }\n    }\n\n    if (filtersEmpty) {\n      if (query == null) {\n        return new MatchAllDocsQuery();\n      } else {\n        return query;\n      }\n    }\n\n    BooleanQuery.Builder bqBuilder = new BooleanQuery.Builder();\n    if (query != null) {\n      bqBuilder.add(query, Occur.MUST);\n    }\n    for (Query f : filters) {\n      if (f != null) {\n        bqBuilder.add(f, Occur.FILTER);\n      }\n    }\n    return bqBuilder.build();\n  }\n\n  // Examine all fields in the request for sanity.\n  private void validateRequest() throws TransientException, ClientException {\n    // First try thrift's internal validate.  Should always succeed.\n    try {\n      request.validate();\n    } catch (TException e) {\n      throw new TransientException(e.getMessage(), e);\n    }\n\n    if (searchQuery == null) {\n      throw new TransientException(\"No ThriftSearchQuery specified\");\n    }\n\n    if (collectorParams == null) {\n      throw new TransientException(\"No CollectorParams specified\");\n    }\n\n    validateTermStatsRequest();\n\n    if (!searchAllSegments) {\n      if (request.getSearchSegmentId() <= 0) {\n        String msg = \"Bad time slice ID: \" + request.getSearchSegmentId();\n        throw new TransientException(msg);\n      }\n\n      // Initialize the segment.\n      SegmentInfo segmentInfo = this.segmentManager.getSegmentInfo(request.getSearchSegmentId());\n      segment = segmentInfo != null ? segmentInfo.getSegment() : null;\n    }\n\n    if (collectorParams.getNumResultsToReturn() < 0) {\n      String msg = \"Invalid numResults: \" + collectorParams.getNumResultsToReturn();\n      throw new TransientException(msg);\n    }\n\n    if (searchQuery.getNamedDisjunctionMapSize() > 0 && searchQuery.isSetLuceneQuery()) {\n      throw new ClientException(\"namedMultiTermDisjunctionMap does not support with luceneQuery\");\n    }\n  }\n\n  private void validateTermStatsRequest() throws ClientException {\n    // Validate the field names and values for all ThriftTermRequests.\n    if (request.isSetTermStatisticsRequest()\n        && request.getTermStatisticsRequest().isSetTermRequests()) {\n      for (ThriftTermRequest termRequest : request.getTermStatisticsRequest().getTermRequests()) {\n        // If termRequest.fieldName is not set, it defaults to 'text', which is a string field,\n        // so we don't need to check the term.\n        if (termRequest.isSetFieldName()) {\n          String fieldName = termRequest.getFieldName();\n          Schema.FieldInfo facetFieldInfo = schemaSnapshot.getFacetFieldByFacetName(fieldName);\n          if (facetFieldInfo != null) {\n            // Facet fields are string fields, so we don't need to check the term.\n            continue;\n          }\n\n          Schema.FieldInfo fieldInfo = schemaSnapshot.getFieldInfo(fieldName);\n          if (fieldInfo == null) {\n            throw new ClientException(\"Field \" + fieldName + \" is not present in the schema.\");\n          }\n\n          try {\n            SchemaUtil.toBytesRef(fieldInfo, termRequest.getTerm());\n          } catch (UnsupportedOperationException e) {\n            throw new ClientException(\"Term \" + termRequest.getTerm() + \" is not compatible with \"\n                                      + \"the type of field \" + fieldName);\n          }\n        }\n      }\n    }\n  }\n\n  private void setQueriesInDebugInfo(\n      com.twitter.search.queryparser.query.Query parsedQ,\n      org.apache.lucene.search.Query luceneQ) {\n    debugInfo.setParsedQuery(parsedQ == null ? null : parsedQ.serialize());\n    debugInfo.setLuceneQuery(luceneQ == null ? null : luceneQ.toString());\n  }\n\n  /**\n   * Takes the EarlybirdRequest that came into the service and after various parsing and processing\n   * steps ultimately produces a Lucene query.\n   */\n  private void parseEarlybirdRequest() throws ClientException {\n    SerializedQueryParser parser = new SerializedQueryParser(EarlybirdConfig.getPenguinVersion());\n\n    try {\n      // if the deprecated iterativeQueries field is set, return an error to the client\n      // indicating that support for it has been removed.\n      if (searchQuery.isSetDeprecated_iterativeQueries()) {\n        throw new ClientException(\"Invalid request: iterativeQueries feature has been removed\");\n      }\n\n      // we parse the actual query from the user, if any\n      luceneQuery = null;\n      parsedQuery = null;  // this will be set by parseQueryHelper()\n\n      if (searchQuery.getLikedByUserIDFilter64Size() > 0\n          && searchQuery.isSetLuceneQuery()) {\n        throw new ClientException(\"likedByUserIDFilter64 does not support with luceneQuery\");\n      }\n\n      if (!StringUtils.isBlank(request.getSearchQuery().getSerializedQuery())) {\n        searcherStats.thriftQueryWithSerializedQuery.increment();\n        luceneQuery = parseSerializedQuery(searchQuery.getSerializedQuery(), parser, true);\n      } else if (!StringUtils.isBlank(request.getSearchQuery().getLuceneQuery())) {\n        searcherStats.thriftQueryWithLuceneQuery.increment();\n        luceneQuery = parseLuceneQuery(searchQuery.getLuceneQuery());\n        LOG.info(\"lucene query: {}\", searchQuery.getLuceneQuery());\n        if (luceneQuery != null) {\n          LOG.info(\"Using lucene query directly from the request: \" + luceneQuery.toString());\n        }\n      } else {\n        searcherStats.thriftQueryWithoutTextQuery.increment();\n        luceneQuery = parseSerializedQuery(\n            MATCH_ALL_SERIALIZED_QUERY,\n            parser,\n            queryMode != QueryMode.TERM_STATS);\n      }\n    } catch (QueryParserException | BooleanQuery.TooManyClauses e) {\n      LOG.info(\"Exception parsing query during search\", e);\n      appendMessage(e.getMessage());\n      throw new ClientException(e);\n    }\n  }\n\n  /**\n   * Parses a serialized query and creates a Lucene query out of it.\n   *\n   * To see how serialized queries look like, go to go/searchsyntax.\n   */\n  private Query parseSerializedQuery(\n      String serializedQuery,\n      SerializedQueryParser parser,\n      boolean shouldAdjustQueryBasedOnRequestParameters) throws QueryParserException {\n    // Parse the serialized query.\n    parsedQuery = parser.parse(serializedQuery);\n    if (parsedQuery == null) {\n      return null;\n    }\n\n    // rewrite query if positive 'protected' operator is detected\n    if (parsedQuery.accept(new DetectPositiveOperatorVisitor(SearchOperatorConstants.PROTECTED))) {\n      POSITIVE_PROTECTED_OPERATOR_DETECTED_COUNTER.increment();\n      ProtectedOperatorQueryRewriter rewriter = new ProtectedOperatorQueryRewriter();\n      parsedQuery = rewriter.rewrite(\n          parsedQuery,\n          request.followedUserIds,\n          segmentManager.getUserTable());\n    }\n\n    ThriftSearchRelevanceOptions options = searchQuery.getRelevanceOptions();\n    if (shouldAdjustQueryBasedOnRequestParameters) {\n      // If likedByUserIDFilter64 is set, combine it with query\n      // Note: we deal with likedByUserIDFilter64 here instead of in postLuceneQueryProcess as we\n      // want annotate query with ranks.\n      if (searchQuery.isSetLikedByUserIDFilter64()\n          && searchQuery.getLikedByUserIDFilter64Size() > 0) {\n        parsedQuery = combineWithLikedByUserIdFilter64(\n            parsedQuery, searchQuery.getLikedByUserIDFilter64());\n      }\n\n      // If namedListMap field is set, replace the named lists in the serialized query.\n      if (searchQuery.getNamedDisjunctionMapSize() > 0) {\n        parsedQuery = parsedQuery.accept(\n            new NamedDisjunctionVisitor(searchQuery.getNamedDisjunctionMap()));\n      }\n\n      if (searchQuery.isSetRelevanceOptions()\n          && searchQuery.getRelevanceOptions().isCollectFieldHitAttributions()) {\n        // NOTE: Before we do any modifications to the serialized query tree, annotate the query\n        // nodes with their node rank in the original query.\n        this.hitAttributeHelper =\n            QueryHitAttributeHelper.from(parsedQuery, schemaSnapshot);\n        parsedQuery = hitAttributeHelper.getAnnotatedQuery();\n      }\n\n      // Currently antisocial/nullcast tweets are dropped when we build index, but some tweets may\n      // become antisocial with realtime updates. For consistency, we should always filter out\n      // antisocial/nullcast tweets if the user is not explicitly including it.\n      final boolean allowAntisocial =\n          parsedQuery.accept(new DetectPositiveOperatorVisitor(SearchOperatorConstants.ANTISOCIAL));\n      if (!allowAntisocial) {\n        parsedQuery = QueryNodeUtils.appendAsConjunction(\n            parsedQuery,\n            QueryCacheConversionRules.CACHED_EXCLUDE_ANTISOCIAL);\n      }\n      parsedQueryAllowNullcast =\n          parsedQuery.accept(new DetectPositiveOperatorVisitor(SearchOperatorConstants.NULLCAST));\n      if (!parsedQueryAllowNullcast) {\n        parsedQuery = QueryNodeUtils.appendAsConjunction(\n            parsedQuery, new SearchOperator(\"filter\", SearchOperatorConstants.NULLCAST).negate());\n      }\n\n      // Strip all annotations from the filters that will be converted to query cache filters.\n      // See SEARCH-15552.\n      parsedQuery = parsedQuery.accept(\n          new StripAnnotationsVisitor(QueryCacheConversionRules.STRIP_ANNOTATIONS_QUERIES));\n\n      // Convert certain filters into cached filters, also consolidate them.\n      parsedQuery = parsedQuery.accept(\n          new ConversionVisitor(QueryCacheConversionRules.DEFAULT_RULES));\n\n      // add proximity if needed\n      if (options != null\n          && options.isProximityScoring()\n          && searchQuery.getRankingMode() != ThriftSearchRankingMode.RECENCY) {\n        parsedQuery = parsedQuery.accept(new ProximityGroupRewriteVisitor()).simplify();\n      }\n    }\n\n    if (request.isSkipVeryRecentTweets()) {\n      parsedQuery = restrictQueryToFullyIndexedTweets(parsedQuery);\n    }\n\n    parsedQuery = parsedQuery.simplify();\n    debugInfo.setParsedQuery(parsedQuery.serialize());\n\n    // Extract top-level since-id for pagination optimizations.\n    idTimeRanges = IdTimeRanges.fromQuery(parsedQuery);\n\n    // Does any final processing specific to EarlybirdSearch class.\n    parsedQuery = preLuceneQueryProcess(parsedQuery);\n\n    // Convert to a lucene query.\n    EarlybirdLuceneQueryVisitor luceneVisitor = getLuceneVisitor(\n        options == null ? null : options.getFieldWeightMapOverride());\n\n    if (options != null) {\n      luceneVisitor\n          .setProximityPhraseWeight((float) options.getProximityPhraseWeight())\n          .setProximityPhraseSlop(options.getProximityPhraseSlop());\n    }\n\n    // Propagate hit attribute helper to the lucene visitor if it has been setup.\n    luceneVisitor.setFieldHitAttributeHelper(this.hitAttributeHelper);\n\n    org.apache.lucene.search.Query query = parsedQuery.accept(luceneVisitor);\n    if (query != null) {\n      debugInfo.setLuceneQuery(query.toString());\n    }\n\n    queriedFields = luceneVisitor.getQueriedFields();\n\n    return query;\n  }\n\n  private Query parseLuceneQuery(String query) {\n    QueryParser parser = new QueryParser(\n        EarlybirdFieldConstant.TEXT_FIELD.getFieldName(),\n        new SearchWhitespaceAnalyzer());\n    parser.setSplitOnWhitespace(true);\n    try {\n      return parser.parse(query);\n    } catch (ParseException e) {\n      LOG.error(\"Cannot parse raw lucene query: \" + query, e);\n    } catch (NullPointerException e) {\n      LOG.error(\"NullPointerException while parsing raw lucene query: \" + query\n          + \", probably your grammar is wrong.\\n\", e);\n    }\n    return null;\n  }\n\n  private com.twitter.search.queryparser.query.Query combineWithLikedByUserIdFilter64(\n      com.twitter.search.queryparser.query.Query query,\n      List<Long> ids) throws QueryParserException {\n    return QueryNodeUtils.appendAsConjunction(query, getLikedByUserIdQuery(ids));\n  }\n\n  /**\n   * initSearcher initializes the segmentSearcher, and returns SUCCESS if OK\n   * or some other response code it not OK.\n   */\n  private EarlybirdResponseCode initSearcher() throws IOException {\n    searcher = null;\n    if (searchAllSegments) {\n      return initMultiSegmentSearcher();\n    } else {\n      return initSingleSegmentSearcher();\n    }\n  }\n\n  private EarlybirdResponseCode initSingleSegmentSearcher() throws IOException {\n    if (segment == null) {\n      String message = \"Segment not found for time slice: \" + request.getSearchSegmentId();\n      LOG.warn(message);\n      appendMessage(message);\n      return EarlybirdResponseCode.PARTITION_NOT_FOUND;\n    }\n\n    EarlybirdResponseCode code = this.segmentManager.checkSegment(segment);\n    if (code != EarlybirdResponseCode.SUCCESS) {\n      String message = \"Segment \" + segment + \" either disabled or dropped\";\n      LOG.warn(message);\n      appendMessage(message);\n      return code;\n    }\n\n    searcher = segmentManager.getSearcher(segment, schemaSnapshot);\n    if (searcher == null) {\n      String message = \"Could not construct searcher for segment \" + segment;\n      LOG.error(message);\n      appendMessage(message);\n      return EarlybirdResponseCode.PERSISTENT_ERROR;\n    } else {\n      appendMessage(\"Searching segment: \" + segment);\n      return EarlybirdResponseCode.SUCCESS;\n    }\n  }\n\n  private EarlybirdResponseCode initMultiSegmentSearcher() throws IOException {\n    EarlybirdMultiSegmentSearcher multiSearcher =\n        segmentManager.getMultiSearcher(schemaSnapshot);\n    searcher = multiSearcher;\n    Preconditions.checkNotNull(searcher);\n\n    // Set a top level since id to skip entire segments when possible.\n    multiSearcher.setIdTimeRanges(idTimeRanges);\n    return EarlybirdResponseCode.SUCCESS;\n  }\n\n  private com.twitter.search.queryparser.query.Query\n  restrictQueryToFullyIndexedTweets(com.twitter.search.queryparser.query.Query query) {\n    long untilTimeSeconds =\n        RecentTweetRestriction.recentTweetsUntilTime(decider, (int) (clock.nowMillis() / 1000));\n    if (untilTimeSeconds == 0) {\n      return query;\n    }\n\n    SearchOperator timeLimit = new SearchOperator(UNTIL_TIME, untilTimeSeconds);\n    return new Conjunction(query, timeLimit);\n  }\n\n  private EarlybirdResponse newResponse(EarlybirdResponseCode code, boolean setDebugInfo) {\n    EarlybirdResponse response = new EarlybirdResponse();\n    response.setResponseCode(code);\n    if (setDebugInfo) {\n      response.setDebugInfo(debugInfo);\n      if (messageBuffer.length() > 0) {\n        response.setDebugString(DatabaseConfig.getLocalHostname()\n                                + \":\\n\" + messageBuffer.toString());\n      }\n    }\n    return response;\n  }\n\n  private EarlybirdResponse respondError(EarlybirdResponseCode code) {\n    appendMessage(\"Responding with error code \" + code);\n    // Always respond with an error message, even when request.debug is false\n    return newResponse(code, true);\n  }\n\n  @VisibleForTesting\n  public TerminationTracker getTerminationTracker() {\n    return terminationTracker;\n  }\n\n  public void maybeSetCollectorDebugInfo(TwitterEarlyTerminationCollector collector) {\n    if (request.isSetDebugOptions() && request.getDebugOptions().isIncludeCollectorDebugInfo()) {\n      debugInfo.setCollectorDebugInfo(collector.getDebugInfo());\n    }\n  }\n\n  public void setTermStatisticsDebugInfo(List<String> termStatisticsDebugInfo) {\n    debugInfo.setTermStatisticsDebugInfo(termStatisticsDebugInfo);\n  }\n\n  private EarlybirdResponse searchInternal() throws TransientException, ClientException {\n    searchResults = new ThriftSearchResults();\n\n    SearchResultsInfo searchResultsInfo;\n    try {\n      switch (queryMode) {\n        case RECENCY:\n          searchResultsInfo = processRealtimeQuery();\n          break;\n        case RELEVANCE:\n          // Relevance search and Model-based search differ only on the scoring function used.\n          SearchTimer timer = searcherStats.createTimer();\n          timer.start();\n          searchResultsInfo = processRelevanceQuery();\n          timer.stop();\n          searcherStats.recordRelevanceStats(timer, request);\n          break;\n        case FACETS:\n          searchResultsInfo = processFacetsQuery();\n          break;\n        case TERM_STATS:\n          searchResultsInfo = processTermStatsQuery();\n          break;\n        case TOP_TWEETS:\n          searchResultsInfo = processTopTweetsQuery();\n          break;\n        default:\n          throw new TransientException(\"Unknown query mode \" + queryMode);\n      }\n\n      return respondSuccess(searchResults, facetResults, termStatisticsResults,\n          earlyTerminationInfo, searchResultsInfo);\n    } catch (IOException e) {\n      throw new TransientException(e.getMessage(), e);\n    }\n  }\n\n  /**\n   * Helper method to process facets query.\n   */\n  private SearchResultsInfo processFacetsQuery() throws ClientException, IOException {\n    // figure out which fields we need to count\n    FacetCountState facetCountState = newFacetCountState();\n\n    // Additionally wrap our query into a skip list boolean query for faster counting.\n    if (!facetRequest.isUsingQueryCache()) {\n      // Only if all fields to be counted use skip lists, then we can add a required clause\n      // that filters out all results that do not contain those fields\n      boolean cannotAddRequiredClause = facetCountState.hasFieldToCountWithoutSkipList();\n      final Query facetSkipListFilter =\n          cannotAddRequiredClause ? null : FacetSkipList.getSkipListQuery(facetCountState);\n      final Query antisocialFilter = UserFlagsExcludeFilter.getUserFlagsExcludeFilter(\n          segmentManager.getUserTable(), true, true, false);\n      luceneQuery = wrapFilters(luceneQuery,\n          facetSkipListFilter,\n          antisocialFilter);\n    }\n\n    facetResults = new ThriftFacetResults(new HashMap<>());\n\n    FacetSearchRequestInfo searchRequestInfo =\n        new FacetSearchRequestInfo(searchQuery, facetRequest.getFacetRankingOptions(),\n            luceneQuery, facetCountState, terminationTracker);\n    searchRequestInfo.setIdTimeRanges(idTimeRanges);\n    if (searchQuery.getMaxHitsPerUser() > 0) {\n      antiGamingFilter = new AntiGamingFilter(\n          searchQuery.getMaxHitsPerUser(),\n          searchQuery.getMaxTweepcredForAntiGaming(),\n          luceneQuery);\n    }\n\n    AbstractResultsCollector<\n        FacetSearchRequestInfo, EarlybirdLuceneSearcher.FacetSearchResults> collector;\n    if (request.getDebugMode() > 2) {\n      collector = new ExplainFacetResultsCollector(schemaSnapshot,\n          searchRequestInfo, antiGamingFilter, searcherStats, clock, request.debugMode);\n    } else {\n      collector = new FacetResultsCollector(schemaSnapshot,\n          searchRequestInfo, antiGamingFilter, searcherStats, clock, request.debugMode);\n    }\n\n    setQueriesInDebugInfo(parsedQuery, searchRequestInfo.getLuceneQuery());\n    searcher.search(searchRequestInfo.getLuceneQuery(), collector);\n    EarlybirdLuceneSearcher.FacetSearchResults hits = collector.getResults();\n\n    EarlybirdSearchResultUtil.setResultStatistics(searchResults, hits);\n    earlyTerminationInfo = EarlybirdSearchResultUtil.prepareEarlyTerminationInfo(hits);\n    Set<Long> userIDWhitelist =\n        antiGamingFilter != null ? antiGamingFilter.getUserIDWhitelist() : null;\n    prepareFacetResults(facetResults, hits, facetCountState, userIDWhitelist,\n        request.getDebugMode());\n    facetResults.setUserIDWhitelist(userIDWhitelist);\n\n    maybeSetCollectorDebugInfo(collector);\n\n    if (collector instanceof ExplainFacetResultsCollector) {\n      ((ExplainFacetResultsCollector) collector).setExplanations(facetResults);\n    }\n\n    return hits;\n  }\n\n  /**\n   * Helper method to process term-stats query.\n   */\n  private SearchResultsInfo processTermStatsQuery() throws IOException {\n    // first extract the terms that we need to count\n    TermStatisticsRequestInfo searchRequestInfo =\n        new TermStatisticsRequestInfo(searchQuery, luceneQuery, termStatisticsRequest,\n            terminationTracker);\n    searchRequestInfo.setIdTimeRanges(idTimeRanges);\n    setQueriesInDebugInfo(parsedQuery, searchRequestInfo.getLuceneQuery());\n    TermStatisticsCollector.TermStatisticsSearchResults hits =\n        searcher.collectTermStatistics(searchRequestInfo, this, request.getDebugMode());\n    EarlybirdSearchResultUtil.setResultStatistics(searchResults, hits);\n    earlyTerminationInfo = EarlybirdSearchResultUtil.prepareEarlyTerminationInfo(hits);\n    if (hits.results != null) {\n      termStatisticsResults = new ThriftTermStatisticsResults();\n      prepareTermStatisticsResults(termStatisticsResults, hits, request.getDebugMode());\n    }\n\n    return hits;\n  }\n\n  /**\n   * Helper method to process realtime query.\n   */\n  private SearchResultsInfo processRealtimeQuery() throws IOException, ClientException {\n    // Disable maxHitsToProcess.\n    if (!collectorParams.isSetTerminationParams()) {\n      collectorParams.setTerminationParams(new CollectorTerminationParams());\n      collectorParams.getTerminationParams().setMaxHitsToProcess(-1);\n      COLLECTOR_PARAMS_MAX_HITS_TO_PROCESS_NOT_SET_COUNTER.increment();\n    }\n\n    SearchRequestInfo searchRequestInfo = new SearchRequestInfo(\n      searchQuery, luceneQuery, terminationTracker);\n    searchRequestInfo.setIdTimeRanges(idTimeRanges);\n    searchRequestInfo.setHitAttributeHelper(hitAttributeHelper);\n    searchRequestInfo.setTimestamp(getQueryTimestamp(searchQuery));\n\n    AbstractResultsCollector<SearchRequestInfo, SimpleSearchResults> collector;\n    if (searchQuery.isSetSocialFilterType()) {\n      if (!searchRequestInfo.getSearchQuery().isSetDirectFollowFilter()\n          || !searchRequestInfo.getSearchQuery().isSetTrustedFilter()) {\n        searcherStats.unsetFiltersForSocialFilterTypeQuery.increment();\n        throw new ClientException(\n            \"SocialFilterType specified without a TrustedFilter or DirectFollowFilter\");\n      }\n      SocialFilter socialFilter = new SocialFilter(\n          searchQuery.getSocialFilterType(),\n          searchRequestInfo.getSearchQuery().getSearcherId(),\n          searchRequestInfo.getSearchQuery().getTrustedFilter(),\n          searchRequestInfo.getSearchQuery().getDirectFollowFilter());\n      collector = new SocialSearchResultsCollector(\n          schemaSnapshot,\n          searchRequestInfo,\n          socialFilter,\n          searcherStats,\n          cluster,\n          segmentManager.getUserTable(),\n          request.getDebugMode());\n    } else {\n      collector = new SearchResultsCollector(\n          schemaSnapshot,\n          searchRequestInfo,\n          clock,\n          searcherStats,\n          cluster,\n          segmentManager.getUserTable(),\n          request.getDebugMode());\n    }\n\n    setQueriesInDebugInfo(parsedQuery, luceneQuery);\n    searcher.search(luceneQuery, collector);\n\n    SimpleSearchResults hits = collector.getResults();\n\n    EarlybirdSearchResultUtil.setResultStatistics(searchResults, hits);\n    earlyTerminationInfo = EarlybirdSearchResultUtil.prepareEarlyTerminationInfo(hits);\n    EarlybirdSearchResultUtil.prepareResultsArray(\n        searchResults.getResults(), hits, request.debugMode > 0 ? partitionConfig : null);\n    searchResults.setHitCounts(collector.getHitCountMap());\n\n    maybeSetCollectorDebugInfo(collector);\n\n    addResultPayloads();\n\n    return hits;\n  }\n\n  /**\n   * Helper method to process relevance query.\n   */\n  private SearchResultsInfo processRelevanceQuery() throws IOException, ClientException {\n    if (!searchQuery.isSetRelevanceOptions()) {\n      LOG.warn(\"Relevance query with no relevance options!\");\n      searchQuery.setRelevanceOptions(new ThriftSearchRelevanceOptions());\n    }\n\n    // Note: today the assumption is that if you specify hasSpecifiedTweets,\n    // you really do want all tweets scored and returned.\n    final boolean hasSpecifiedTweets = searchQuery.getSearchStatusIdsSize() > 0;\n    if (hasSpecifiedTweets) {\n      collectorParams.setNumResultsToReturn(searchQuery.getSearchStatusIdsSize());\n    }\n    // If we have explicit user ids, we will want to look at all results from those users, and will\n    // not need to use the AntiGamingFilter.\n    final boolean hasSpecifiedFromUserIds = searchQuery.getFromUserIDFilter64Size() > 0;\n\n    createRelevanceAntiGamingFilter(hasSpecifiedTweets, hasSpecifiedFromUserIds);\n\n    if (searchQuery.getRelevanceOptions().isSetRankingParams()) {\n      ThriftRankingParams rankingParams = searchQuery.getRelevanceOptions().getRankingParams();\n\n      // The score adjustment signals that are passed in the request are disabled for the archive\n      // cluster or when the features are decidered off. If the request provides those fields,\n      // we unset them since checking the hashmap when scoring can cause a slight bump in\n      // latency.\n      //\n      // Verify that the signal query specific scores for tweets signal is enabled\n      if (rankingParams.isSetQuerySpecificScoreAdjustments()) {\n        if (ALLOW_QUERY_SPECIFIC_SIGNAL_CONFIG\n            && DeciderUtil.isAvailableForRandomRecipient(\n            decider, ALLOW_QUERY_SPECIFIC_SIGNAL_DECIDER_KEY)) {\n          searcherStats.querySpecificSignalQueriesUsed.increment();\n          searcherStats.querySpecificSignalMapTotalSize.add(\n              rankingParams.getQuerySpecificScoreAdjustmentsSize());\n        } else {\n          searchQuery.getRelevanceOptions().getRankingParams().unsetQuerySpecificScoreAdjustments();\n          searcherStats.querySpecificSignalQueriesErased.increment();\n        }\n      }\n\n      // Verify that the signal author specific scores signal is enabled\n      if (rankingParams.isSetAuthorSpecificScoreAdjustments()) {\n        if (ALLOW_AUTHOR_SPECIFIC_SIGNAL_CONFIG\n            && DeciderUtil.isAvailableForRandomRecipient(\n            decider, ALLOW_AUTHOR_SPECIFIC_SIGNAL_DECIDER_KEY)) {\n          searcherStats.authorSpecificSignalQueriesUsed.increment();\n          searcherStats.authorSpecificSignalMapTotalSize.add(\n              rankingParams.getAuthorSpecificScoreAdjustmentsSize());\n        } else {\n          searchQuery.getRelevanceOptions().getRankingParams()\n              .unsetAuthorSpecificScoreAdjustments();\n          searcherStats.authorSpecificSignalQueriesErased.increment();\n        }\n      }\n    }\n\n    ScoringFunction scoringFunction =\n        new ScoringFunctionProvider.DefaultScoringFunctionProvider(\n            request, schemaSnapshot, searchQuery, antiGamingFilter,\n            segmentManager.getUserTable(), hitAttributeHelper,\n            parsedQuery, scoringModelsManager, tensorflowModelsManager)\n            .getScoringFunction();\n    scoringFunction.setDebugMode(request.getDebugMode());\n\n    RelevanceQuery relevanceQuery = new RelevanceQuery(luceneQuery, scoringFunction);\n    RelevanceSearchRequestInfo searchRequestInfo =\n        new RelevanceSearchRequestInfo(\n            searchQuery, relevanceQuery, terminationTracker, qualityFactor);\n    searchRequestInfo.setIdTimeRanges(idTimeRanges);\n    searchRequestInfo.setHitAttributeHelper(hitAttributeHelper);\n    searchRequestInfo.setTimestamp(getQueryTimestamp(searchQuery));\n\n    if (shouldUseTensorFlowCollector()\n        && searchQuery.getRelevanceOptions().isUseRelevanceAllCollector()) {\n      throw new ClientException(\"Tensorflow scoring does not work with the RelevanceAllCollector\");\n    }\n\n    final AbstractRelevanceCollector collector;\n    // First check if the Tensorflow results collector should be used, because the\n    // TensorflowBasedScoringFunction only works with the BatchRelevanceTopCollector\n    if (shouldUseTensorFlowCollector()) {\n      // Collect top numResults.\n      collector = new BatchRelevanceTopCollector(\n          schemaSnapshot,\n          searchRequestInfo,\n          scoringFunction,\n          searcherStats,\n          cluster,\n          segmentManager.getUserTable(),\n          clock,\n          request.getDebugMode());\n    } else if (hasSpecifiedTweets\n        || searchQuery.getRelevanceOptions().isUseRelevanceAllCollector()) {\n      // Collect all.\n      collector = new RelevanceAllCollector(\n          schemaSnapshot,\n          searchRequestInfo,\n          scoringFunction,\n          searcherStats,\n          cluster,\n          segmentManager.getUserTable(),\n          clock,\n          request.getDebugMode());\n    } else {\n      // Collect top numResults.\n      collector = new RelevanceTopCollector(\n          schemaSnapshot,\n          searchRequestInfo,\n          scoringFunction,\n          searcherStats,\n          cluster,\n          segmentManager.getUserTable(),\n          clock,\n          request.getDebugMode());\n    }\n\n    // Make sure that the Tensorflow scoring function and the Tensorflow results collector are\n    // always used together. If this fails it will result in a TRANSIENT_ERROR response.\n    Preconditions.checkState((collector instanceof BatchRelevanceTopCollector)\n        == (scoringFunction instanceof TensorflowBasedScoringFunction));\n\n    setQueriesInDebugInfo(parsedQuery, searchRequestInfo.getLuceneQuery());\n    searcher.search(searchRequestInfo.getLuceneQuery(), collector);\n\n    RelevanceSearchResults hits = collector.getResults();\n    EarlybirdSearchResultUtil.setResultStatistics(searchResults, hits);\n    searchResults.setScoringTimeNanos(hits.getScoringTimeNanos());\n\n    earlyTerminationInfo = EarlybirdSearchResultUtil.prepareEarlyTerminationInfo(hits);\n    EarlybirdSearchResultUtil.setLanguageHistogram(searchResults, collector.getLanguageHistogram());\n    EarlybirdSearchResultUtil.prepareRelevanceResultsArray(\n        searchResults.getResults(),\n        hits,\n        antiGamingFilter != null ? antiGamingFilter.getUserIDWhitelist() : null,\n        request.getDebugMode() > 0 ? partitionConfig : null);\n\n    searchResults.setHitCounts(collector.getHitCountMap());\n    searchResults.setRelevanceStats(hits.getRelevanceStats());\n\n    maybeSetCollectorDebugInfo(collector);\n\n    if (explanationsEnabled(request.getDebugMode())) {\n      searcher.explainSearchResults(searchRequestInfo, hits, searchResults);\n    }\n\n    addResultPayloads();\n\n    return hits;\n  }\n\n  public static boolean explanationsEnabled(int debugLevel) {\n    return debugLevel > 1;\n  }\n\n  private boolean shouldUseTensorFlowCollector() {\n    return tensorflowModelsManager.isEnabled()\n        && searchQuery.getRelevanceOptions().isSetRankingParams()\n        && searchQuery.getRelevanceOptions().getRankingParams().isSetType()\n        && searchQuery.getRelevanceOptions().getRankingParams().getType()\n        == ThriftScoringFunctionType.TENSORFLOW_BASED;\n  }\n  /**\n   * Optionally, if requested and needed, will create a new AntiGamingFilter. Otherwize, no\n   * AntiGamingFilter will be used for this query.\n   * @param hasSpecifiedTweets whether the request has searchStatusIds specified.\n   * @param hasSpecifiedFromUserIds whether the request has fromUserIDFilter64 specified.\n   */\n  private void createRelevanceAntiGamingFilter(\n      boolean hasSpecifiedTweets, boolean hasSpecifiedFromUserIds) {\n\n    // Anti-gaming filter (turned off for specified tweets mode, or when you're explicitly asking\n    // for specific users' tweets).\n    if (searchQuery.getMaxHitsPerUser() > 0 && !hasSpecifiedTweets && !hasSpecifiedFromUserIds) {\n      searcherStats.relevanceAntiGamingFilterUsed.increment();\n      antiGamingFilter = new AntiGamingFilter(\n          searchQuery.getMaxHitsPerUser(),\n          searchQuery.getMaxTweepcredForAntiGaming(),\n          luceneQuery);\n    } else if (searchQuery.getMaxHitsPerUser() <= 0) {\n      searcherStats.relevanceAntiGamingFilterNotRequested.increment();\n    } else if (hasSpecifiedTweets && hasSpecifiedFromUserIds) {\n      searcherStats.relevanceAntiGamingFilterSpecifiedTweetsAndFromUserIds.increment();\n    } else if (hasSpecifiedTweets) {\n      searcherStats.relevanceAntiGamingFilterSpecifiedTweets.increment();\n    } else if (hasSpecifiedFromUserIds) {\n      searcherStats.relevanceAntiGamingFilterSpecifiedFromUserIds.increment();\n    }\n  }\n\n  /**\n   * Check to make sure that there are no nullcast documents in results.  If there exists nullcasts\n   * in results, we should log error and increment counters correspondingly.\n   */\n  @VisibleForTesting\n  public void logAndIncrementStatsIfNullcastInResults(ThriftSearchResults thriftSearchResults) {\n    if (!thriftSearchResults.isSetResults()) {\n      return;\n    }\n\n    Set<Long> unexpectedNullcastStatusIds =\n        EarlybirdResponseUtil.findUnexpectedNullcastStatusIds(thriftSearchResults, request);\n\n    if (!unexpectedNullcastStatusIds.isEmpty()) {\n      searcherStats.nullcastUnexpectedQueries.increment();\n      searcherStats.nullcastUnexpectedResults.add(unexpectedNullcastStatusIds.size());\n\n      String base64Request;\n      try {\n        base64Request = ThriftUtils.toBase64EncodedString(request);\n      } catch (TException e) {\n        base64Request = \"Failed to parse base 64 request\";\n      }\n      LOG.error(\n          \"Found unexpected nullcast tweets: {} | parsedQuery: {} | request: {} | response: {} | \"\n              + \"request base 64: {}\",\n          Joiner.on(\",\").join(unexpectedNullcastStatusIds),\n          parsedQuery.serialize(),\n          request,\n          thriftSearchResults,\n          base64Request);\n    }\n  }\n\n  private void addResultPayloads() throws IOException {\n    if (searchQuery.getResultMetadataOptions() != null) {\n      if (searchQuery.getResultMetadataOptions().isGetTweetUrls()) {\n        searcher.fillFacetResults(new ExpandedUrlCollector(), searchResults);\n      }\n\n      if (searchQuery.getResultMetadataOptions().isGetNamedEntities()) {\n        searcher.fillFacetResults(new NamedEntityCollector(), searchResults);\n      }\n\n      if (searchQuery.getResultMetadataOptions().isGetEntityAnnotations()) {\n        searcher.fillFacetResults(new EntityAnnotationCollector(), searchResults);\n      }\n\n      if (searchQuery.getResultMetadataOptions().isGetSpaces()) {\n        searcher.fillFacetResults(new SpaceFacetCollector(audioSpaceTable), searchResults);\n      }\n    }\n  }\n\n  /**\n   * Helper method to process top tweets query.\n   */\n  private SearchResultsInfo processTopTweetsQuery() throws IOException, ClientException {\n    // set dummy relevance options if it's not available, but this shouldn't happen in prod\n    if (!searchQuery.isSetRelevanceOptions()) {\n      searchQuery.setRelevanceOptions(new ThriftSearchRelevanceOptions());\n    }\n    if (!searchQuery.getRelevanceOptions().isSetRankingParams()) {\n      searchQuery.getRelevanceOptions().setRankingParams(\n          // this is important, or it's gonna pick DefaultScoringFunction which pretty much\n          // does nothing.\n          new ThriftRankingParams().setType(ThriftScoringFunctionType.TOPTWEETS));\n    }\n    ScoringFunction scoringFunction = new ScoringFunctionProvider.DefaultScoringFunctionProvider(\n        request, schemaSnapshot, searchQuery, null,\n        segmentManager.getUserTable(), hitAttributeHelper, parsedQuery,\n        scoringModelsManager, tensorflowModelsManager)\n        .getScoringFunction();\n    scoringFunction.setDebugMode(request.getDebugMode());\n\n    RelevanceQuery relevanceQuery = new RelevanceQuery(luceneQuery, scoringFunction);\n    RelevanceSearchRequestInfo searchRequestInfo =\n        new RelevanceSearchRequestInfo(\n            searchQuery, relevanceQuery, terminationTracker, qualityFactor);\n    searchRequestInfo.setIdTimeRanges(idTimeRanges);\n    searchRequestInfo.setTimestamp(getQueryTimestamp(searchQuery));\n\n    final AbstractRelevanceCollector collector =\n        new RelevanceTopCollector(\n            schemaSnapshot,\n            searchRequestInfo,\n            scoringFunction,\n            searcherStats,\n            cluster,\n            segmentManager.getUserTable(),\n            clock,\n            request.getDebugMode());\n\n    setQueriesInDebugInfo(parsedQuery, searchRequestInfo.getLuceneQuery());\n    searcher.search(searchRequestInfo.getLuceneQuery(), collector);\n\n    RelevanceSearchResults hits = collector.getResults();\n    EarlybirdSearchResultUtil.setResultStatistics(searchResults, hits);\n    searchResults.setScoringTimeNanos(hits.getScoringTimeNanos());\n    earlyTerminationInfo = EarlybirdSearchResultUtil.prepareEarlyTerminationInfo(hits);\n    EarlybirdSearchResultUtil.setLanguageHistogram(\n        searchResults,\n        collector.getLanguageHistogram());\n    EarlybirdSearchResultUtil.prepareRelevanceResultsArray(\n        searchResults.getResults(),\n        hits,\n        null,\n        request.getDebugMode() > 0 ? partitionConfig : null);\n\n    searchResults.setHitCounts(collector.getHitCountMap());\n    searchResults.setRelevanceStats(hits.getRelevanceStats());\n\n    maybeSetCollectorDebugInfo(collector);\n\n    if (explanationsEnabled(request.getDebugMode())\n        && searchQuery.isSetRelevanceOptions()\n        && searchQuery.getRelevanceOptions().isSetRankingParams()) {\n      searcher.explainSearchResults(searchRequestInfo, hits, searchResults);\n    }\n\n    addResultPayloads();\n\n    return hits;\n  }\n\n  private FacetCountState newFacetCountState() throws ClientException {\n    int minNumFacetResults = DEFAULT_NUM_FACET_RESULTS;\n    if (facetRequest.isSetFacetRankingOptions()\n        && facetRequest.getFacetRankingOptions().isSetNumCandidatesFromEarlybird()) {\n      minNumFacetResults = facetRequest.getFacetRankingOptions().getNumCandidatesFromEarlybird();\n    }\n\n    // figure out which fields we need to count\n    FacetCountState facetCountState = new FacetCountState(schemaSnapshot, minNumFacetResults);\n\n    // all categories if none!\n    if (facetRequest.getFacetFields() == null || facetRequest.getFacetFields().isEmpty()) {\n      for (Schema.FieldInfo facetField : schemaSnapshot.getFacetFields()) {\n        facetCountState.addFacet(\n            facetField.getFieldType().getFacetName(), DEFAULT_NUM_FACET_RESULTS);\n      }\n    } else {\n      Iterator<ThriftFacetFieldRequest> it = facetRequest.getFacetFieldsIterator();\n      while (it.hasNext()) {\n        ThriftFacetFieldRequest facetFieldRequest = it.next();\n        Schema.FieldInfo facet = schemaSnapshot.getFacetFieldByFacetName(\n            facetFieldRequest.getFieldName());\n        if (facet != null) {\n          facetCountState.addFacet(\n              facet.getFieldType().getFacetName(), facetFieldRequest.getNumResults());\n        } else {\n          throw new ClientException(\"Unknown facet field: \" + facetFieldRequest.getFieldName());\n        }\n      }\n    }\n    return facetCountState;\n  }\n\n  private com.twitter.search.queryparser.query.Query preLuceneQueryProcess(\n      com.twitter.search.queryparser.query.Query twitterQuery) throws QueryParserException {\n\n    com.twitter.search.queryparser.query.Query query = twitterQuery;\n    if (searchHighFrequencyTermPairs && !includesCardField(searchQuery, query)) {\n      // Process high frequency term pairs. Works best when query is as flat as possible.\n      query = HighFrequencyTermPairRewriteVisitor.safeRewrite(\n          query,\n          DeciderUtil.isAvailableForRandomRecipient(\n              decider, \"enable_hf_term_pair_negative_disjunction_rewrite\"));\n    }\n    return query.simplify();\n  }\n\n  private Query postLuceneQueryProcess(final Query query) throws ClientException {\n    if (StringUtils.isBlank(request.getSearchQuery().getSerializedQuery())\n        && StringUtils.isBlank(request.getSearchQuery().getLuceneQuery())) {\n      searcherStats.numRequestsWithBlankQuery.get(queryMode).increment();\n      if (searchQuery.getSearchStatusIdsSize() == 0\n          && searchQuery.getFromUserIDFilter64Size() == 0\n          && searchQuery.getLikedByUserIDFilter64Size() == 0) {\n        // No query or ids to search.  This is only allowed in some modes.\n        if (queryMode == QueryMode.RECENCY\n            || queryMode == QueryMode.RELEVANCE\n            || queryMode == QueryMode.TOP_TWEETS) {\n          throw new ClientException(\n              \"No query or status ids for \" + queryMode.toString().toLowerCase() + \" query\");\n        }\n      }\n    }\n\n    // Wrap the query as needed with additional query filters.\n    List<Query> filters = Lists.newArrayList();\n\n    // Min tweep cred filter.\n    if (searchQuery.isSetMinTweepCredFilter()) {\n      searcherStats.addedFilterBadUserRep.increment();\n      filters.add(BadUserRepFilter.getBadUserRepFilter(searchQuery.getMinTweepCredFilter()));\n    }\n\n    if (searchQuery.getFromUserIDFilter64Size() > 0) {\n      this.queriedFields.add(EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName());\n      this.searcherStats.addedFilterFromUserIds.increment();\n      try {\n        filters.add(UserIdMultiSegmentQuery.createIdDisjunctionQuery(\n            \"from_user_id_filter\",\n            searchQuery.getFromUserIDFilter64(),\n            EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName(),\n            schemaSnapshot,\n            multiSegmentTermDictionaryManager,\n            decider,\n            cluster,\n            Lists.newArrayList(),\n            null,\n            queryTimeoutFactory.createQueryTimeout(request, terminationTracker, clock)));\n      } catch (QueryParserException e) {\n        throw new ClientException(e);\n      }\n    }\n\n    // Wrap the lucene query with these filters.\n    Query wrappedQuery = wrapFilters(query, filters.toArray(new Query[filters.size()]));\n\n    // If searchStatusIds is set, additionally modify the query to search exactly these\n    // ids, using the luceneQuery only for scoring.\n    if (searchQuery.getSearchStatusIdsSize() > 0) {\n      this.searcherStats.addedFilterTweetIds.increment();\n\n      final Query queryForScoring = wrappedQuery;\n      final Query queryForRetrieval =\n          RequiredStatusIDsFilter.getRequiredStatusIDsQuery(searchQuery.getSearchStatusIds());\n\n      return new BooleanQuery.Builder()\n          .add(queryForRetrieval, Occur.MUST)\n          .add(queryForScoring, Occur.SHOULD)\n          .build();\n    }\n\n    return wrappedQuery;\n  }\n\n  private com.twitter.search.queryparser.query.Query getLikedByUserIdQuery(\n      List<Long> ids) throws QueryParserException {\n    if (DeciderUtil.isAvailableForRandomRecipient(\n        decider, USE_MULTI_TERM_DISJUNCTION_FOR_LIKED_BY_USER_IDS_DECIDER_KEY)) {\n      // rewrite LikedByUserIdFilter64 to a multi_term_disjuntion query\n      return createMultiTermDisjunctionQueryForLikedByUserIds(ids);\n    } else {\n      // rewrite LikedByUserIdFilter64 to a disjunction of multiple liked_by_user_ids query\n      return createDisjunctionQueryForLikedByUserIds(ids);\n    }\n  }\n\n  /**\n   * Returns the Lucene query visitor that should be applied to the original request.\n   *\n   * @param fieldWeightMapOverride The per-field weight overrides.\n   */\n  @VisibleForTesting\n  public EarlybirdLuceneQueryVisitor getLuceneVisitor(\n      Map<String, Double> fieldWeightMapOverride) {\n    String clusterName = cluster.getNameForStats();\n    // Iff in relevance mode _and_ intepreteSinceId is false, we turn off since_id\n    // operator by using LuceneRelevanceQueryVisitor.\n\n    if (searchQuery.getRankingMode() == ThriftSearchRankingMode.RELEVANCE\n        && searchQuery.getRelevanceOptions() != null\n        && !searchQuery.getRelevanceOptions().isInterpretSinceId()) {\n      // hack!  reset top level since id, which is the same thing LuceneRelevanceVisitor\n      // is doing.\n      idTimeRanges = null;\n      return new LuceneRelevanceQueryVisitor(\n          schemaSnapshot,\n          queryCacheManager,\n          segmentManager.getUserTable(),\n          segmentManager.getUserScrubGeoMap(),\n          terminationTracker,\n          FieldWeightDefault.overrideFieldWeightMap(\n              schemaSnapshot.getFieldWeightMap(),\n              dropBadFieldWeightOverrides(fieldWeightMapOverride, decider, clusterName)),\n          MAPPABLE_FIELD_MAP,\n          multiSegmentTermDictionaryManager,\n          decider,\n          cluster,\n          queryTimeoutFactory.createQueryTimeout(\n              request, terminationTracker, clock));\n    } else {\n      return new EarlybirdLuceneQueryVisitor(\n          schemaSnapshot,\n          queryCacheManager,\n          segmentManager.getUserTable(),\n          segmentManager.getUserScrubGeoMap(),\n          terminationTracker,\n          FieldWeightDefault.overrideFieldWeightMap(\n              schemaSnapshot.getFieldWeightMap(),\n              dropBadFieldWeightOverrides(fieldWeightMapOverride, decider, clusterName)),\n          MAPPABLE_FIELD_MAP,\n          multiSegmentTermDictionaryManager,\n          decider,\n          cluster,\n          queryTimeoutFactory.createQueryTimeout(\n              request, terminationTracker, clock));\n    }\n  }\n\n  private void prepareFacetResults(ThriftFacetResults thriftFacetResults,\n                                     EarlybirdLuceneSearcher.FacetSearchResults hits,\n                                     FacetCountState<ThriftFacetFieldResults> facetCountState,\n                                     Set<Long> userIDWhitelist,\n                                     byte debugMode) throws IOException {\n    for (FacetRankingModule rankingModule : FacetRankingModule.REGISTERED_RANKING_MODULES) {\n      rankingModule.prepareResults(hits, facetCountState);\n    }\n\n    Map<Term, ThriftFacetCount> allFacetResults = new HashMap<>();\n\n    Iterator<FacetCountState.FacetFieldResults<ThriftFacetFieldResults>> fieldResultsIterator =\n        facetCountState.getFacetFieldResultsIterator();\n    while (fieldResultsIterator.hasNext()) {\n\n      FacetCountState.FacetFieldResults<ThriftFacetFieldResults> facetFieldResults =\n          fieldResultsIterator.next();\n\n      if (facetFieldResults.results == null) {\n        // return empty resultset for this facet\n        List<ThriftFacetCount> emptyList = new ArrayList<>();\n        facetFieldResults.results = new ThriftFacetFieldResults(emptyList, 0);\n      }\n      thriftFacetResults.putToFacetFields(facetFieldResults.facetName,\n          facetFieldResults.results);\n\n      Schema.FieldInfo field = schemaSnapshot.getFacetFieldByFacetName(\n          facetFieldResults.facetName);\n\n      for (ThriftFacetCount result : facetFieldResults.results.topFacets) {\n        if (result.facetLabel != null) {\n          allFacetResults.put(new Term(field.getName(), result.facetLabel), result);\n        } else {\n          LOG.warn(\"Null facetLabel, field: {}, result: {}\", field.getName(), result);\n        }\n      }\n    }\n\n    searcher.fillFacetResultMetadata(allFacetResults, schemaSnapshot, debugMode);\n\n    if (userIDWhitelist != null) {\n      for (ThriftFacetCount facetCount : allFacetResults.values()) {\n        ThriftFacetCountMetadata metadata = facetCount.getMetadata();\n        if (metadata != null) {\n          metadata.setDontFilterUser(userIDWhitelist.contains(metadata.getTwitterUserId()));\n        }\n      }\n    }\n  }\n\n  private void prepareTermStatisticsResults(\n      ThriftTermStatisticsResults termStatistics,\n      TermStatisticsCollector.TermStatisticsSearchResults hits,\n      byte debugMode) throws IOException {\n\n    termStatistics.setBinIds(hits.binIds);\n    termStatistics.setHistogramSettings(termStatisticsRequest.getHistogramSettings());\n    termStatistics.setTermResults(hits.results);\n    setTermStatisticsDebugInfo(hits.getTermStatisticsDebugInfo());\n\n    if (hits.lastCompleteBinId != -1) {\n      termStatistics.setMinCompleteBinId(hits.lastCompleteBinId);\n    } else {\n      SearchRateCounter.export(String.format(\n          \"term_stats_%s_unset_min_complete_bin_id\", request.getClientId())).increment();\n    }\n\n    if (idTimeRanges != null\n        && idTimeRanges.getUntilTimeExclusive().isPresent()\n        && hits.getMinSearchedTime() > idTimeRanges.getUntilTimeExclusive().get()) {\n      SearchRateCounter.export(String.format(\n          \"term_stats_%s_min_searched_time_after_until_time\", request.getClientId())).increment();\n    }\n\n    searcher.fillTermStatsMetadata(termStatistics, schemaSnapshot, debugMode);\n  }\n\n  private EarlybirdResponse respondSuccess(\n      ThriftSearchResults thriftSearchResults,\n      ThriftFacetResults thriftFacetResults,\n      ThriftTermStatisticsResults termStatisticResults,\n      @Nonnull EarlyTerminationInfo earlyTerminationState,\n      @Nonnull SearchResultsInfo searchResultsInfo) {\n\n    Preconditions.checkNotNull(earlyTerminationState);\n    Preconditions.checkNotNull(searchResultsInfo);\n\n    exportEarlyTerminationStats(earlyTerminationState);\n\n    EarlybirdResponse response =\n        newResponse(EarlybirdResponseCode.SUCCESS, request.getDebugMode() > 0);\n    response.setEarlyTerminationInfo(earlyTerminationState);\n    response.setNumSearchedSegments(searchResultsInfo.getNumSearchedSegments());\n\n    if (thriftSearchResults != null) {\n      // Nullcast check is only used when parsed query is available: if there is no parsed query,\n      // we would not add possible exclude nullcast filter.\n      if (parsedQuery != null && !parsedQueryAllowNullcast) {\n        logAndIncrementStatsIfNullcastInResults(thriftSearchResults);\n      }\n      response.setSearchResults(thriftSearchResults);\n    } else {\n      RESPONSE_HAS_NO_THRIFT_SEARCH_RESULTS.increment();\n    }\n    if (thriftFacetResults != null) {\n      response.setFacetResults(thriftFacetResults);\n    }\n    if (termStatisticResults != null) {\n      response.setTermStatisticsResults(termStatisticResults);\n    }\n\n    appendFeatureSchemaIfNeeded(response);\n\n    appendLikedByUserIdsIfNeeded(response);\n\n    return response;\n  }\n\n  private void exportEarlyTerminationStats(@Nonnull EarlyTerminationInfo earlyTerminationState) {\n    if (earlyTerminationState.isSetEarlyTerminationReason()) {\n      SearchRateCounter.export(String.format(\"early_termination_%s_%s\",\n          ClientIdUtil.formatClientId(request.getClientId()),\n          earlyTerminationState.getEarlyTerminationReason())).increment();\n      SearchRateCounter.export(String.format(\"early_termination_%s_%s\",\n          ClientIdUtil.formatClientIdAndRequestType(\n              request.getClientId(), queryMode.name().toLowerCase()),\n          earlyTerminationState.getEarlyTerminationReason())).increment();\n    }\n  }\n\n  /**\n   * Builds a rank -> userId map for liked_by_user_id queries that request hit attribution, and\n   * appends the resulting map to the response.\n   */\n  private void appendLikedByUserIdsIfNeeded(EarlybirdResponse response) {\n    // Check if user asked for likedByUserIds list in response\n    ThriftSearchRelevanceOptions resultRelevanceOptions =\n        request.getSearchQuery().getRelevanceOptions();\n    if ((resultRelevanceOptions == null)\n        || !resultRelevanceOptions.isCollectFieldHitAttributions()) {\n      return;\n    }\n\n    // Make sure we have results in response and hit attribution helper is set up correctly\n    if (!response.isSetSearchResults() || hitAttributeHelper == null) {\n      return;\n    }\n\n    // Get rank to node map\n    Map<com.twitter.search.queryparser.query.Query, Integer> nodeToRankMap =\n        Preconditions.checkNotNull(hitAttributeHelper.getNodeToRankMap());\n\n    Map<com.twitter.search.queryparser.query.Query, List<Integer>> expandedNodeToRankMap =\n        Preconditions.checkNotNull(hitAttributeHelper.getExpandedNodeToRankMap());\n\n    // Build a rank to id map\n    ImmutableMap.Builder<Integer, Long> builder = ImmutableMap.builder();\n    for (com.twitter.search.queryparser.query.Query query : nodeToRankMap.keySet()) {\n      if (query instanceof SearchOperator) {\n        SearchOperator op = (SearchOperator) query;\n        if (expandedNodeToRankMap.containsKey(query)) {\n          // for multi_term_disjunction case\n          List<Integer> ranks = expandedNodeToRankMap.get(op);\n          Preconditions.checkArgument(op.getNumOperands() == ranks.size() + 1);\n          for (int i = 0; i < ranks.size(); ++i) {\n            builder.put(ranks.get(i), Long.valueOf(op.getOperands().get(i + 1)));\n          }\n        } else if (op.getOperatorType() == SearchOperator.Type.LIKED_BY_USER_ID) {\n          // for liked_by_user_id case\n          Preconditions.checkArgument(op.getAnnotationOf(Annotation.Type.NODE_RANK).isPresent());\n          builder.put(\n              (Integer) op.getAnnotationOf(Annotation.Type.NODE_RANK).get().getValue(),\n              Long.valueOf(op.getOperands().get(0)));\n        }\n      }\n    }\n    Map<Integer, Long> rankToIdMap = builder.build();\n\n    // Append liked_by_user_id filed into result\n    for (ThriftSearchResult result : response.getSearchResults().getResults()) {\n      if (result.isSetMetadata()\n          && result.getMetadata().isSetFieldHitAttribution()\n          && result.getMetadata().getFieldHitAttribution().isSetHitMap()) {\n\n        List<Long> likedByUserIdList = Lists.newArrayList();\n\n        Map<Integer, FieldHitList> hitMap =\n            result.getMetadata().getFieldHitAttribution().getHitMap();\n        // iterate hit attributions\n        for (int rank : hitMap.keySet()) {\n          if (rankToIdMap.containsKey(rank)) {\n            likedByUserIdList.add(rankToIdMap.get(rank));\n          }\n        }\n        if (!result.getMetadata().isSetExtraMetadata()) {\n          result.getMetadata().setExtraMetadata(new ThriftSearchResultExtraMetadata());\n        }\n        result.getMetadata().getExtraMetadata().setLikedByUserIds(likedByUserIdList);\n      }\n    }\n  }\n\n  private void appendFeatureSchemaIfNeeded(EarlybirdResponse response) {\n    // Do not append the schema if the client didn't request it.\n    ThriftSearchResultMetadataOptions resultMetadataOptions =\n        request.getSearchQuery().getResultMetadataOptions();\n    if ((resultMetadataOptions == null) || !resultMetadataOptions.isReturnSearchResultFeatures()) {\n      return;\n    }\n\n    if (!response.isSetSearchResults()) {\n      return;\n    }\n\n    ThriftSearchFeatureSchema featureSchema = schemaSnapshot.getSearchFeatureSchema();\n    Preconditions.checkState(\n        featureSchema.isSetSchemaSpecifier(),\n        \"The feature schema doesn't have a schema specifier set: {}\", featureSchema);\n\n    // If the client has this schema, we only need to return the schema version.\n    // If the client doesn't have this schema, we need to return the schema entries too.\n    if (resultMetadataOptions.isSetFeatureSchemasAvailableInClient()\n        && resultMetadataOptions.getFeatureSchemasAvailableInClient().contains(\n        featureSchema.getSchemaSpecifier())) {\n      CLIENT_HAS_FEATURE_SCHEMA_COUNTER.increment();\n      ThriftSearchFeatureSchema responseFeatureSchema = new ThriftSearchFeatureSchema();\n      responseFeatureSchema.setSchemaSpecifier(featureSchema.getSchemaSpecifier());\n      response.getSearchResults().setFeatureSchema(responseFeatureSchema);\n    } else {\n      CLIENT_DOESNT_HAVE_FEATURE_SCHEMA_COUNTER.increment();\n      Preconditions.checkState(featureSchema.isSetEntries(),\n          \"Entries are not set in the feature schema: \" + featureSchema);\n      response.getSearchResults().setFeatureSchema(featureSchema);\n    }\n  }\n\n  private static long getQueryTimestamp(ThriftSearchQuery query) {\n    return query != null && query.isSetTimestampMsecs() ? query.getTimestampMsecs() : 0;\n  }\n\n  private static boolean includesCardField(ThriftSearchQuery searchQuery,\n                                           com.twitter.search.queryparser.query.Query query)\n      throws QueryParserException {\n\n    if (searchQuery.isSetRelevanceOptions()) {\n      ThriftSearchRelevanceOptions options = searchQuery.getRelevanceOptions();\n      if (options.isSetFieldWeightMapOverride()\n          && (options.getFieldWeightMapOverride().containsKey(\n              EarlybirdFieldConstant.CARD_TITLE_FIELD.getFieldName())\n          || options.getFieldWeightMapOverride()\n          .containsKey(EarlybirdFieldConstant.CARD_DESCRIPTION_FIELD.getFieldName()))) {\n\n        return true;\n      }\n    }\n\n    return query.accept(new DetectFieldAnnotationVisitor(ImmutableSet.of(\n        EarlybirdFieldConstant.CARD_TITLE_FIELD.getFieldName(),\n        EarlybirdFieldConstant.CARD_DESCRIPTION_FIELD.getFieldName())));\n  }\n\n  private static QueryMode getQueryMode(EarlybirdRequest request) {\n    if (request.isSetFacetRequest()) {\n      return QueryMode.FACETS;\n    } else if (request.isSetTermStatisticsRequest()) {\n      return QueryMode.TERM_STATS;\n    }\n\n    // Recency mode until we determine otherwise.\n    QueryMode queryMode = QueryMode.RECENCY;\n    ThriftSearchQuery searchQuery = request.getSearchQuery();\n    if (searchQuery != null) {\n      switch (searchQuery.getRankingMode()) {\n        case RECENCY:\n          queryMode = QueryMode.RECENCY;\n          break;\n        case RELEVANCE:\n          queryMode = QueryMode.RELEVANCE;\n          break;\n        case TOPTWEETS:\n          queryMode = QueryMode.TOP_TWEETS;\n          break;\n        default:\n          break;\n      }\n    }\n\n    if (searchQuery == null\n        || !searchQuery.isSetSerializedQuery()\n        || searchQuery.getSerializedQuery().isEmpty()) {\n      LOG.debug(\"Search query was empty, query mode was \" + queryMode);\n    }\n\n    return queryMode;\n  }\n\n  private static <V> ImmutableMap<String, V> dropBadFieldWeightOverrides(\n      Map<String, V> map, Decider decider, String clusterName) {\n\n    if (map == null) {\n      return null;\n    }\n\n    FIELD_WEIGHT_OVERRIDE_MAP_NON_NULL_COUNT.increment();\n    ImmutableMap.Builder<String, V> builder = ImmutableMap.builder();\n\n    for (Map.Entry<String, V> entry : map.entrySet()) {\n      if (EarlybirdFieldConstant.CAMELCASE_USER_HANDLE_FIELD.getFieldName().equals(entry.getKey())\n          && !isAllowedCamelcaseUsernameFieldWeightOverride(decider, clusterName)) {\n        DROPPED_CAMELCASE_USERNAME_FIELD_WEIGHT_OVERRIDE.increment();\n      } else if (EarlybirdFieldConstant.TOKENIZED_USER_NAME_FIELD.getFieldName().equals(\n                     entry.getKey())\n          && !isAllowedTokenizedScreenNameFieldWeightOverride(decider, clusterName)) {\n        DROPPED_TOKENIZED_DISPLAY_NAME_FIELD_WEIGHT_OVERRIDE.increment();\n      } else {\n        builder.put(entry.getKey(), entry.getValue());\n      }\n    }\n\n    return builder.build();\n  }\n\n  private static boolean isAllowedCamelcaseUsernameFieldWeightOverride(\n      Decider decider, String clusterName) {\n    return DeciderUtil.isAvailableForRandomRecipient(decider,\n        ALLOW_CAMELCASE_USERNAME_FIELD_WEIGHT_OVERRIDE_DECIDER_KEY_PREFIX + clusterName);\n  }\n\n  private static boolean isAllowedTokenizedScreenNameFieldWeightOverride(\n      Decider decider, String clusterName) {\n    return DeciderUtil.isAvailableForRandomRecipient(decider,\n        ALLOW_TOKENIZED_DISPLAY_NAME_FIELD_WEIGHT_OVERRIDE_DECIDER_KEY_PREFIX + clusterName);\n  }\n\n  private static com.twitter.search.queryparser.query.Query\n  createMultiTermDisjunctionQueryForLikedByUserIds(List<Long> ids) throws QueryParserException {\n    List<String> operands = new ArrayList<>(ids.size() + 1);\n    operands.add(EarlybirdFieldConstant.LIKED_BY_USER_ID_FIELD.getFieldName());\n    for (long id : ids) {\n      operands.add(String.valueOf(id));\n    }\n    return new SearchOperator(SearchOperator.Type.MULTI_TERM_DISJUNCTION, operands)\n        .simplify();\n  }\n\n  private static com.twitter.search.queryparser.query.Query createDisjunctionQueryForLikedByUserIds(\n      List<Long> ids) throws QueryParserException {\n    return new Disjunction(\n        ids.stream()\n            .map(id -> new SearchOperator(SearchOperator.Type.LIKED_BY_USER_ID, id))\n            .collect(Collectors.toList()))\n        .simplify();\n  }\n\n  public com.twitter.search.queryparser.query.Query getParsedQuery() {\n    return parsedQuery;\n  }\n\n  /**\n   * Get the index fields that were queried after this searcher completed its job.\n   * @return\n   */\n  public Set<String> getQueriedFields() {\n    return queriedFields;\n  }\n\n  public Query getLuceneQuery() {\n    return luceneQuery;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/EarlybirdServer.java",
    "content": "package com.twitter.search.earlybird;\n\nimport java.io.BufferedWriter;\nimport java.io.Closeable;\nimport java.io.File;\nimport java.io.IOException;\nimport java.nio.file.Files;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Set;\nimport java.util.concurrent.ArrayBlockingQueue;\nimport java.util.concurrent.ExecutionException;\nimport java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\nimport java.util.concurrent.RejectedExecutionException;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.atomic.AtomicReference;\nimport javax.annotation.Nullable;\nimport javax.annotation.concurrent.GuardedBy;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Charsets;\nimport com.google.common.base.Stopwatch;\nimport com.google.common.cache.CacheBuilder;\nimport com.google.common.cache.CacheLoader;\nimport com.google.common.cache.LoadingCache;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Lists;\nimport com.google.common.util.concurrent.AtomicLongMap;\n\nimport org.apache.commons.codec.binary.Base64;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.thrift.TBase;\nimport org.apache.thrift.TException;\nimport org.apache.thrift.TSerializer;\nimport org.apache.zookeeper.KeeperException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.common.util.Clock;\nimport com.twitter.common.zookeeper.ServerSet.UpdateException;\nimport com.twitter.common.zookeeper.ZooKeeperClient;\nimport com.twitter.decider.Decider;\nimport com.twitter.finagle.Failure;\nimport com.twitter.search.common.database.DatabaseConfig;\nimport com.twitter.search.common.metrics.Percentile;\nimport com.twitter.search.common.metrics.PercentileUtil;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.metrics.Timer;\nimport com.twitter.search.common.schema.DynamicSchema;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.FlushVersion;\nimport com.twitter.search.common.search.termination.QueryTimeoutFactory;\nimport com.twitter.search.common.util.FinagleUtil;\nimport com.twitter.search.common.util.GCUtil;\nimport com.twitter.search.common.util.ml.tensorflow_engine.TensorflowModelsManager;\nimport com.twitter.search.common.util.zookeeper.ZooKeeperProxy;\nimport com.twitter.search.core.earlybird.index.inverted.QueryCostTracker;\nimport com.twitter.search.earlybird.admin.LastSearchesSummary;\nimport com.twitter.search.earlybird.admin.QueriedFieldsAndSchemaStats;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.common.EarlybirdRequestLogger;\nimport com.twitter.search.earlybird.common.EarlybirdRequestPostLogger;\nimport com.twitter.search.earlybird.common.EarlybirdRequestPreLogger;\nimport com.twitter.search.earlybird.common.EarlybirdRequestUtil;\nimport com.twitter.search.earlybird.common.RequestResponsePair;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.exception.EarlybirdStartupException;\nimport com.twitter.search.earlybird.exception.TransientException;\nimport com.twitter.search.earlybird.ml.ScoringModelsManager;\nimport com.twitter.search.earlybird.partition.AudioSpaceTable;\nimport com.twitter.search.earlybird.partition.DynamicPartitionConfig;\nimport com.twitter.search.earlybird.partition.EarlybirdStartup;\nimport com.twitter.search.earlybird.partition.MultiSegmentTermDictionaryManager;\nimport com.twitter.search.earlybird.partition.PartitionConfig;\nimport com.twitter.search.earlybird.partition.PartitionManager;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\nimport com.twitter.search.earlybird.partition.SegmentManager;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\nimport com.twitter.search.earlybird.partition.SegmentVulture;\nimport com.twitter.search.earlybird.querycache.QueryCacheManager;\nimport com.twitter.search.earlybird.stats.EarlybirdRPCStats;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.EarlybirdServerStats;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusCode;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusResponse;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird.util.OneTaskScheduledExecutorManager;\nimport com.twitter.search.earlybird.util.TermCountMonitor;\nimport com.twitter.search.earlybird.util.TweetCountMonitor;\nimport com.twitter.snowflake.id.SnowflakeId;\nimport com.twitter.util.Duration;\nimport com.twitter.util.Function;\nimport com.twitter.util.Function0;\nimport com.twitter.util.Future;\n\npublic class EarlybirdServer implements EarlybirdService.ServiceIface, ServerSetMember {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdServer.class);\n\n  private static final String EARLYBIRD_STARTUP = \"earlybird startup\";\n  public static final String SERVICE_NAME = \"Earlybird\";\n\n  private static final boolean REGISTER_WITH_ZK_ON_STARTUP =\n      EarlybirdConfig.getBool(\"register_with_zk_on_startup\", true);\n  private static final Duration SERVER_CLOSE_WAIT_TIME = Duration.apply(5L, TimeUnit.SECONDS);\n\n  private static final Failure QUEUE_FULL_FAILURE =\n      Failure.rejected(\"Rejected due to full executor queue\");\n\n  private final int port = EarlybirdConfig.getThriftPort();\n  private final int warmUpPort = EarlybirdConfig.getWarmUpThriftPort();\n  private final int numSearcherThreads = EarlybirdConfig.getSearcherThreads();\n\n  private final SearchStatsReceiver earlybirdServerStatsReceiver;\n  private final EarlybirdRPCStats searchStats = new EarlybirdRPCStats(\"search\");\n  private final EarlybirdSearcherStats tweetsSearcherStats;\n\n  private static final String REQUESTS_RECEIVED_BY_FINAGLE_ID_COUNTER_NAME_PATTERN =\n      \"requests_for_finagle_id_%s_all\";\n  private static final String REQUESTS_RECEIVED_BY_FINAGLE_ID_AND_CLIENT_ID_COUNTER_NAME_PATTERN =\n      \"requests_for_finagle_id_%s_and_client_id_%s\";\n  private static final String RESPONSES_PER_CLIENT_ID_STAT_TEMPLATE =\n      \"responses_for_client_id_%s_with_response_code_%s\";\n\n  // Loading cache for per finagle-client-id stats. Storing them in a loading cache key-ed by\n  // finagle client id so we don't export the stat multiple times.\n  private final LoadingCache<String, SearchTimerStats> requestCountersByFinagleClientId =\n      CacheBuilder.newBuilder().build(\n          new CacheLoader<String, SearchTimerStats>() {\n            @Override\n            public SearchTimerStats load(String finagleClientId) {\n              return earlybirdServerStatsReceiver.getTimerStats(\n                  String.format(\n                      REQUESTS_RECEIVED_BY_FINAGLE_ID_COUNTER_NAME_PATTERN,\n                      finagleClientId), TimeUnit.MICROSECONDS, false, true, false);\n            }\n          });\n\n  // Counters per client and response code.\n  private final LoadingCache<String, SearchCounter> responseByClientIdAndResponseCode =\n      CacheBuilder.newBuilder().build(\n          new CacheLoader<String, SearchCounter>() {\n              @Override\n              public SearchCounter load(String key) {\n                  return earlybirdServerStatsReceiver.getCounter(key);\n              }\n          });\n\n  private final LoadingCache<String, SearchCounter> resultsAgeCounter =\n      CacheBuilder.newBuilder().build(\n          new CacheLoader<String, SearchCounter>() {\n            @Override\n            public SearchCounter load(String key) {\n              return earlybirdServerStatsReceiver.getCounter(key);\n            }\n          }\n      );\n\n  // Loading cache for per finagle client id and client id stats. These are stored separate\n  // from the other stats because they are key-ed by the pair of finagle client id and client id\n  // in order to make sure the stats are only exported once.\n  // In the key-pair the first element is the finagle client id while the second element is the\n  // client id.\n  private final LoadingCache<Pair<String, String>, SearchRateCounter>\n      requestCountersByFinagleIdAndClientId = CacheBuilder.newBuilder().build(\n          new CacheLoader<Pair<String, String>, SearchRateCounter>() {\n            @Override\n            public SearchRateCounter load(Pair<String, String> clientKey) {\n              return earlybirdServerStatsReceiver.getRateCounter(\n                  String.format(\n                      REQUESTS_RECEIVED_BY_FINAGLE_ID_AND_CLIENT_ID_COUNTER_NAME_PATTERN,\n                      clientKey.getFirst(),\n                      clientKey.getSecond()));\n            }\n          });\n\n  // Loading cache for per-client-id latency stats. Stored in a loading cache here mainly because\n  // the tests assert the mock stats receiver that each stat is only exported once.\n  private final LoadingCache<String, SearchTimerStats> clientIdSearchStats =\n      CacheBuilder.newBuilder().build(\n          new CacheLoader<String, SearchTimerStats>() {\n            @Override\n            public SearchTimerStats load(String clientId) {\n              String formattedClientId = ClientIdUtil.formatClientId(clientId);\n              return earlybirdServerStatsReceiver.getTimerStats(formattedClientId,\n                  TimeUnit.MICROSECONDS, false, true, true);\n            }\n          });\n\n  private final LoadingCache<String, SearchTimerStats> clientIdScoringPerQueryStats =\n      CacheBuilder.newBuilder().build(\n          new CacheLoader<String, SearchTimerStats>() {\n            @Override\n            public SearchTimerStats load(String clientId) {\n              String statName =\n                  String.format(\"scoring_time_per_query_for_client_id_%s\", clientId);\n              return earlybirdServerStatsReceiver.getTimerStats(statName,\n                  TimeUnit.NANOSECONDS, false, true, false);\n            }\n          });\n\n  private final LoadingCache<String, SearchTimerStats> clientIdScoringPerHitStats =\n      CacheBuilder.newBuilder().build(\n          new CacheLoader<String, SearchTimerStats>() {\n            @Override\n            public SearchTimerStats load(String clientId) {\n              String statName =\n                  String.format(\"scoring_time_per_hit_for_client_id_%s\", clientId);\n              return earlybirdServerStatsReceiver.getTimerStats(statName,\n                  TimeUnit.NANOSECONDS, false, true, false);\n            }\n          });\n\n  private final LoadingCache<String, Percentile<Integer>> clientIdScoringNumHitsProcessedStats =\n      CacheBuilder.newBuilder().build(\n          new CacheLoader<String, Percentile<Integer>>() {\n            @Override\n            public Percentile<Integer> load(String clientId) {\n              String statName =\n                  String.format(\"scoring_num_hits_processed_for_client_id_%s\", clientId);\n              return PercentileUtil.createPercentile(statName);\n            }\n          });\n\n  private final LoadingCache<String, AtomicReference<RequestResponsePair>> lastRequestPerClientId =\n      CacheBuilder.newBuilder().build(\n          new CacheLoader<String, AtomicReference<RequestResponsePair>>() {\n            @Override\n            public AtomicReference<RequestResponsePair> load(String key) throws Exception {\n              return new AtomicReference<>(null);\n            }\n          });\n\n\n  private final SearchTimerStats overallScoringTimePerQueryStats;\n  private final SearchTimerStats overallScoringTimePerHitStats;\n  private final Percentile<Integer> overallScoringNumHitsProcessedStats;\n\n  private final EarlybirdIndexConfig earlybirdIndexConfig;\n  private final DynamicPartitionConfig dynamicPartitionConfig;\n  private final SegmentManager segmentManager;\n  private final UpdateableEarlybirdStateManager stateManager;\n  private final AudioSpaceTable audioSpaceTable;\n\n  private final SearchLongGauge startupTimeGauge;\n\n  // Time spent in an internal thread pool queue, between the time we get the search request\n  // from finagle until it actually starts being executed.\n  private final SearchTimerStats internalQueueWaitTimeStats;\n\n  // Tracking request that have exceeded their allocated timeout prior to us actually being able\n  // to start executing the search.\n  private final SearchCounter requestTimeoutExceededBeforeSearchCounter;\n  // Current number of running searcher threads.\n  private final SearchLongGauge numSearcherThreadsGauge;\n  private final QueryTimeoutFactory queryTimeoutFactory;\n\n  private PartitionManager partitionManager;\n  private QueryCacheManager queryCacheManager;\n\n  private final ScoringModelsManager scoringModelsManager;\n\n  private final TensorflowModelsManager tensorflowModelsManager;\n\n  private final EarlybirdRequestPreLogger requestPreLogger;\n  private final EarlybirdRequestPostLogger requestLogger;\n\n  private final TweetCountMonitor tweetCountMonitor;\n  private final TermCountMonitor termCountMonitor;\n\n  private final EarlybirdServerSetManager serverSetManager;\n  private final EarlybirdWarmUpManager warmUpManager;\n  private final MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager;\n\n  private final Object shutdownLock = new Object();\n  @GuardedBy(\"shutdownLock\")\n  private final EarlybirdFuturePoolManager futurePoolManager;\n  @GuardedBy(\"shutdownLock\")\n  private final EarlybirdFinagleServerManager finagleServerManager;\n\n  // If a search request comes in with a client-side start time, and we see that based on that\n  // the timeout has expired, whether we should drop that query immediately.\n  private final boolean skipTimedOutRequests =\n      EarlybirdConfig.getBool(\"skip_timedout_requests\", false);\n\n  // client of szookeeper.local.twitter.com.\n  // This is used to perform distributed locking and layout reading etc.\n  private final ZooKeeperProxy sZooKeeperClient;\n\n  private final Decider decider;\n\n  private final Clock clock;\n\n  private final List<Closeable> toClose = new ArrayList<>();\n\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n\n  private final EarlybirdDarkProxy earlybirdDarkProxy;\n\n  private final ImmutableMap<EarlybirdResponseCode, SearchCounter> responseCodeCounters;\n  private final SegmentSyncConfig segmentSyncConfig;\n  private final EarlybirdStartup earlybirdStartup;\n  private final QualityFactor qualityFactor;\n\n  private boolean isShutdown = false;\n  private boolean isShuttingDown = false;\n\n  private final AtomicLongMap<String> queriedFieldsCounts = AtomicLongMap.create();\n\n  public EarlybirdServer(QueryCacheManager queryCacheManager,\n                         ZooKeeperProxy sZkClient,\n                         Decider decider,\n                         EarlybirdIndexConfig earlybirdIndexConfig,\n                         DynamicPartitionConfig dynamicPartitionConfig,\n                         PartitionManager partitionManager,\n                         SegmentManager segmentManager,\n                         AudioSpaceTable audioSpaceTable,\n                         TermCountMonitor termCountMonitor,\n                         TweetCountMonitor tweetCountMonitor,\n                         UpdateableEarlybirdStateManager earlybirdStateManager,\n                         EarlybirdFuturePoolManager futurePoolManager,\n                         EarlybirdFinagleServerManager finagleServerManager,\n                         EarlybirdServerSetManager serverSetManager,\n                         EarlybirdWarmUpManager warmUpManager,\n                         SearchStatsReceiver earlybirdServerStatsReceiver,\n                         EarlybirdSearcherStats tweetsSearcherStats,\n                         ScoringModelsManager scoringModelsManager,\n                         TensorflowModelsManager tensorflowModelsManager,\n                         Clock clock,\n                         MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager,\n                         EarlybirdDarkProxy earlybirdDarkProxy,\n                         SegmentSyncConfig segmentSyncConfig,\n                         QueryTimeoutFactory queryTimeoutFactory,\n                         EarlybirdStartup earlybirdStartup,\n                         QualityFactor qualityFactor,\n                         SearchIndexingMetricSet searchIndexingMetricSet) {\n    LOG.info(\"Creating EarlybirdServer\");\n    this.decider = decider;\n    this.clock = clock;\n    this.sZooKeeperClient = sZkClient;\n    this.earlybirdIndexConfig = earlybirdIndexConfig;\n    this.dynamicPartitionConfig = dynamicPartitionConfig;\n    this.segmentManager = segmentManager;\n    this.queryCacheManager = queryCacheManager;\n    this.termCountMonitor = termCountMonitor;\n    this.tweetCountMonitor = tweetCountMonitor;\n    this.stateManager = earlybirdStateManager;\n    this.partitionManager = partitionManager;\n    this.futurePoolManager = futurePoolManager;\n    this.finagleServerManager = finagleServerManager;\n    this.serverSetManager = serverSetManager;\n    this.warmUpManager = warmUpManager;\n    this.earlybirdServerStatsReceiver = earlybirdServerStatsReceiver;\n    this.tweetsSearcherStats = tweetsSearcherStats;\n    this.scoringModelsManager = scoringModelsManager;\n    this.tensorflowModelsManager = tensorflowModelsManager;\n    this.multiSegmentTermDictionaryManager = multiSegmentTermDictionaryManager;\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n    this.earlybirdDarkProxy = earlybirdDarkProxy;\n    this.segmentSyncConfig = segmentSyncConfig;\n    this.queryTimeoutFactory = queryTimeoutFactory;\n    this.earlybirdStartup = earlybirdStartup;\n    this.qualityFactor = qualityFactor;\n    this.audioSpaceTable = audioSpaceTable;\n\n    EarlybirdStatus.setStartTime(System.currentTimeMillis());\n\n    // Our initial status code is STARTING.\n    EarlybirdStatus.setStatus(EarlybirdStatusCode.STARTING);\n    EarlybirdStatus.THRIFT_SERVICE_STARTED.set(false);\n\n    PartitionConfig partitionConfig = dynamicPartitionConfig.getCurrentPartitionConfig();\n    earlybirdServerStatsReceiver.getLongGauge(\n        \"search_cluster_\" + partitionConfig.getClusterName()).set(1);\n    earlybirdServerStatsReceiver.getLongGauge(\n        \"tier_name_\" + partitionConfig.getTierName()).set(1);\n\n    earlybirdServerStatsReceiver.getLongGauge(\"partition\").set(\n        partitionConfig.getIndexingHashPartitionID());\n    earlybirdServerStatsReceiver.getLongGauge(\"replica\").set(\n        partitionConfig.getHostPositionWithinHashPartition());\n    earlybirdServerStatsReceiver.getLongGauge(\"penguin_version\").set(\n        EarlybirdConfig.getPenguinVersionByte());\n\n    earlybirdServerStatsReceiver.getLongGauge(\"flush_version\").set(\n        FlushVersion.CURRENT_FLUSH_VERSION.ordinal());\n    String buildGen = EarlybirdConfig.getString(\"offline_segment_build_gen\", \"unknown\");\n    earlybirdServerStatsReceiver.getLongGauge(\"build_gen_\" + buildGen).set(1);\n\n    this.startupTimeGauge = earlybirdServerStatsReceiver.getLongGauge(\"startup_time_millis\");\n    this.internalQueueWaitTimeStats = earlybirdServerStatsReceiver.getTimerStats(\n        \"internal_queue_wait_time\", TimeUnit.MILLISECONDS, false, true, false);\n    this.requestTimeoutExceededBeforeSearchCounter = earlybirdServerStatsReceiver.getCounter(\n        \"request_timeout_exceeded_before_search\");\n    this.numSearcherThreadsGauge =\n        earlybirdServerStatsReceiver.getLongGauge(\"num_searcher_threads\");\n    this.overallScoringTimePerQueryStats = earlybirdServerStatsReceiver.getTimerStats(\n        \"overall_scoring_time_per_query\", TimeUnit.NANOSECONDS, false, true, false);\n\n    // For most of our scoring functions the scoring_time_per_hit records the actual time to score a\n    // single hit. However, the tensorflow based scoring function uses batch scoring, so we do not\n    // know the actual time it takes to score a single hit. We are now including batch scoring time\n    // in all scoring time stats (SEARCH-26014), which means that the scoring_time_per_hit stat may\n    // be a bit misleading for tensorflow based queries. For these queries the scoring_time_per_hit\n    // represents the ratio between total_scoring_time and the number_of_hits, instead of the actual\n    // time to score a single hit.\n    this.overallScoringTimePerHitStats = earlybirdServerStatsReceiver.getTimerStats(\n        \"overall_scoring_time_per_hit\", TimeUnit.NANOSECONDS, false, true, false);\n    this.overallScoringNumHitsProcessedStats = PercentileUtil.createPercentile(\n        \"overall_scoring_num_hits_processed\");\n\n    ImmutableMap.Builder<EarlybirdResponseCode, SearchCounter> responseCodeCountersBuilder =\n        new ImmutableMap.Builder<>();\n    for (EarlybirdResponseCode responseCode : EarlybirdResponseCode.values()) {\n      responseCodeCountersBuilder.put(\n          responseCode,\n          earlybirdServerStatsReceiver.getCounter(\n              \"responses_with_response_code_\" + responseCode.name().toLowerCase()));\n    }\n    responseCodeCounters = responseCodeCountersBuilder.build();\n\n    disableLuceneQueryCache();\n    initManagers();\n\n    requestPreLogger = EarlybirdRequestPreLogger.buildForShard(\n      EarlybirdConfig.getInt(\"latency_warn_threshold\", 100), decider);\n    requestLogger = EarlybirdRequestPostLogger.buildForShard(\n        EarlybirdConfig.getInt(\"latency_warn_threshold\", 100), decider);\n\n    this.qualityFactor.startUpdates();\n\n    LOG.info(\"Created EarlybirdServer\");\n  }\n\n  public boolean isShutdown() {\n    return this.isShutdown;\n  }\n\n  private void initManagers() {\n    LOG.info(\"Created EarlybirdIndexConfig: \" + earlybirdIndexConfig.getClass().getSimpleName());\n\n    segmentManager.addUpdateListener(queryCacheManager);\n  }\n\n  public PartitionManager getPartitionManager() {\n    return partitionManager;\n  }\n\n  public QueryCacheManager getQueryCacheManager() {\n    return queryCacheManager;\n  }\n\n  public SegmentManager getSegmentManager() {\n    return segmentManager;\n  }\n\n  public MultiSegmentTermDictionaryManager getMultiSegmentTermDictionaryManager() {\n    return this.multiSegmentTermDictionaryManager;\n  }\n\n  @VisibleForTesting\n  public int getPort() {\n    return port;\n  }\n\n  private void disableLuceneQueryCache() {\n    // SEARCH-30046: Look into possibly re-enabling the query -> weight cache.\n    // We can't use this cache until we upgrade to Lucene 6.0.0, because we have queries with a\n    // boost of 0.0, and they don't play nicely with Lucene's LRUQueryCache.get() method.\n    //\n    // Lucene 6.0.0 changes how boosts are handled: \"real\" boosts should be wrapped into BoostQuery\n    // instances, and queries with a boost of 0.0 should be rewritten as \"filters\"\n    // (BooleanQuery.add(query, BooleanClause.Occur.FILTER)). So when we upgrade to Lucene 6.0.0 we\n    // will be forced to refactor how we handle our current queries with a boost of 0.0, which might\n    // allow us to re-enable this cache.\n    //\n    // Note that disabling this cache is not a regression: it should give us the behavior that we\n    // had with Lucene 5.2.1 (and it's unclear if this cache is useful at all).\n    //\n    // WARNING: The default 'DefaultQueryCache' maintains a static reference to the weight forever,\n    // causing a memory leak. Our weights hold references to an entire segment so the memory leak is\n    // significant.\n    IndexSearcher.setDefaultQueryCache(null);\n  }\n\n  /**\n   * Starts the earlybird server.\n   */\n  public void start() throws EarlybirdStartupException {\n    // Make sure this is at the top of the function before other parts of the system start running\n    new EarlybirdBlacklistHandler(Clock.SYSTEM_CLOCK, sZooKeeperClient)\n        .blockThenExitIfBlacklisted();\n\n    Stopwatch startupWatch = Stopwatch.createStarted();\n    EarlybirdStatus.beginEvent(EARLYBIRD_STARTUP, searchIndexingMetricSet.startupInProgress);\n\n    LOG.info(\"java.library.path is: \" + System.getProperty(\"java.library.path\"));\n\n    PartitionConfig partitionConfig = dynamicPartitionConfig.getCurrentPartitionConfig();\n\n    SegmentVulture.removeUnusedSegments(partitionManager, partitionConfig,\n        earlybirdIndexConfig.getSchema().getMajorVersionNumber(), segmentSyncConfig);\n\n    // Start the schema manager\n    schedule(stateManager);\n\n    Closeable closeable = earlybirdStartup.start();\n    toClose.add(closeable);\n    if (EarlybirdStatus.getStatusCode() == EarlybirdStatusCode.STOPPING) {\n      LOG.info(\"Server is shutdown. Exiting...\");\n      return;\n    }\n\n    startupTimeGauge.set(startupWatch.elapsed(TimeUnit.MILLISECONDS));\n\n    EarlybirdStatus.endEvent(EARLYBIRD_STARTUP, searchIndexingMetricSet.startupInProgress);\n\n    GCUtil.runGC();  // Attempt to force a full GC before joining the serverset\n\n    try {\n      startThriftService(null, true);\n    } catch (InterruptedException e) {\n      LOG.info(\"Interrupted while starting thrift server, quitting earlybird\");\n      throw new EarlybirdStartupException(\"Interrupted while starting thrift server\");\n    }\n\n    EarlybirdStatus.THRIFT_SERVICE_STARTED.set(true);\n\n    // only once we're current, kick off daily tweet count monitors only for archive cluster\n    if (EarlybirdConfig.getInt(TweetCountMonitor.RUN_INTERVAL_MINUTES_CONFIG_NAME, -1) > 0) {\n      schedule(tweetCountMonitor);\n    }\n\n    // only once we're current, kick off per-field term count monitors\n    if (EarlybirdConfig.getInt(TermCountMonitor.RUN_INTERVAL_MINUTES_CONFIG_NAME, -1) > 0) {\n      schedule(termCountMonitor);\n    }\n\n    startupTimeGauge.set(startupWatch.elapsed(TimeUnit.MILLISECONDS));\n    LOG.info(\"EarlybirdServer start up time: {}\", startupWatch);\n  }\n\n  /**\n   * Starts the thrift server if the server is not running.\n   * If searcherThreads is null, it uses the value specified by EarlybirdConfig.\n   */\n  public void startThriftService(@Nullable Integer searcherThreads, boolean isStartingUp)\n      throws InterruptedException {\n    synchronized (shutdownLock) {\n      if (!finagleServerManager.isWarmUpServerRunning()\n          && !finagleServerManager.isProductionServerRunning()) {\n        int threadCount = searcherThreads != null\n            ? searcherThreads : this.numSearcherThreads;\n        LOG.info(\"Starting searcher pool with \" + threadCount + \" threads\");\n        futurePoolManager.createUnderlyingFuturePool(threadCount);\n        numSearcherThreadsGauge.set(threadCount);\n\n        // If the server is not shutting down, go through the warm up stage. If the server is\n        // instructed to shut down during warm up, warmUpManager.warmUp() should return within a\n        // second, and should leave the warm up server set. We should still shut down the warm up\n        // Finagle server.\n        if (isStartingUp && (EarlybirdStatus.getStatusCode() != EarlybirdStatusCode.STOPPING)) {\n          LOG.info(\"Opening warmup thrift port...\");\n          finagleServerManager.startWarmUpFinagleServer(this, SERVICE_NAME, warmUpPort);\n          EarlybirdStatus.WARMUP_THRIFT_PORT_OPEN.set(true);\n\n          try {\n            warmUpManager.warmUp();\n          } catch (UpdateException e) {\n            LOG.warn(\"Could not join or leave the warm up server set.\", e);\n          } finally {\n            finagleServerManager.stopWarmUpFinagleServer(SERVER_CLOSE_WAIT_TIME);\n            EarlybirdStatus.WARMUP_THRIFT_PORT_OPEN.set(false);\n          }\n        }\n\n        // If the server is not shutting down, we can start the production Finagle server and join\n        // the production server set.\n        if (EarlybirdStatus.getStatusCode() != EarlybirdStatusCode.STOPPING) {\n          LOG.info(\"Opening production thrift port...\");\n          finagleServerManager.startProductionFinagleServer(\n              earlybirdDarkProxy.getDarkProxy(), this, SERVICE_NAME, port);\n          EarlybirdStatus.THRIFT_PORT_OPEN.set(true);\n\n          if (REGISTER_WITH_ZK_ON_STARTUP) {\n            // After the earlybird starts up, register with ZooKeeper.\n            try {\n              joinServerSet(\"internal start-up\");\n\n              // Join separate server set for ServiceProxy on Archive Earlybirds\n              if (!EarlybirdConfig.isAurora()) {\n                joinServerSetForServiceProxy();\n              }\n            } catch (UpdateException e) {\n              throw new RuntimeException(\"Unable to join ServerSet during startup.\", e);\n            }\n          }\n        }\n      }\n    }\n  }\n\n  /**\n   * Stops the thrift server if the server is already running.\n   */\n  public void stopThriftService(boolean shouldShutDown) {\n    synchronized (shutdownLock) {\n      try {\n        leaveServerSet(shouldShutDown ? \"internal shutdown\" : \"admin stopThriftService\");\n      } catch (UpdateException e) {\n        LOG.warn(\"Leaving production ServerSet failed.\", e);\n      }\n\n      if (finagleServerManager.isProductionServerRunning()) {\n        try {\n          finagleServerManager.stopProductionFinagleServer(SERVER_CLOSE_WAIT_TIME);\n          futurePoolManager.stopUnderlyingFuturePool(\n              SERVER_CLOSE_WAIT_TIME.inSeconds(), TimeUnit.SECONDS);\n          numSearcherThreadsGauge.set(0);\n        } catch (InterruptedException e) {\n          LOG.error(\"Interrupted while stopping thrift service\", e);\n          Thread.currentThread().interrupt();\n        }\n        EarlybirdStatus.THRIFT_PORT_OPEN.set(false);\n      }\n    }\n  }\n\n  /**\n   * Gets a string with information about the last request we've seen from each client.\n   */\n  public Future<String> getLastSearchesByClient(boolean includeResults) {\n    LastSearchesSummary summary = new LastSearchesSummary(\n        lastRequestPerClientId, clientIdSearchStats, includeResults);\n    return Future.value(summary.getSummary());\n  }\n\n  /**\n   * The following are all the Thrift RPC methods inherited from EarlybirdService.Iface\n   */\n\n  // Thrift getName RPC.\n  @Override\n  public Future<String> getName() {\n    return Future.value(SERVICE_NAME);\n  }\n\n  // Thrift getStatus RPC.\n  @Override\n  public Future<EarlybirdStatusResponse> getStatus() {\n    EarlybirdStatusResponse response = new EarlybirdStatusResponse();\n    response.setCode(EarlybirdStatus.getStatusCode());\n    response.setAliveSince(EarlybirdStatus.getStartTime());\n    response.setMessage(EarlybirdStatus.getStatusMessage());\n    return Future.value(response);\n  }\n\n  public Future<List<String>> getSegmentMetadata() {\n    return Future.value(segmentManager.getSegmentMetadata());\n  }\n\n  public Future<String> getQueryCachesData() {\n    return Future.value(segmentManager.getQueryCachesData());\n  }\n\n  /**\n   * Get a text summary for which fields did we use in a schema.\n   */\n  public Future<String> getQueriedFieldsAndSchemaStats() {\n    ImmutableSchemaInterface schema = this.earlybirdIndexConfig.getSchema().getSchemaSnapshot();\n\n    QueriedFieldsAndSchemaStats summary = new QueriedFieldsAndSchemaStats(schema,\n        queriedFieldsCounts);\n    return Future.value(summary.getSummary());\n  }\n\n  /**\n   * Shuts down the earlybird server.\n   */\n  public void shutdown() {\n    LOG.info(\"shutdown(): status set to STOPPING\");\n    EarlybirdStatus.setStatus(EarlybirdStatusCode.STOPPING);\n    try {\n      LOG.info(\"Stopping Finagle server.\");\n      stopThriftService(true);\n      EarlybirdStatus.THRIFT_SERVICE_STARTED.set(false);\n\n      if (queryCacheManager != null) {\n        queryCacheManager.shutdown();\n      } else {\n        LOG.info(\"No queryCacheManager to shut down\");\n      }\n\n      earlybirdIndexConfig.getResourceCloser().shutdownExecutor();\n\n      isShuttingDown = true;\n      LOG.info(\"Closing {} closeables.\", toClose.size());\n      for (Closeable closeable : toClose) {\n        closeable.close();\n      }\n    } catch (InterruptedException | IOException e) {\n      EarlybirdStatus.setStatus(EarlybirdStatusCode.UNHEALTHY, e.getMessage());\n      LOG.error(\"Interrupted during shutdown, status set to UNHEALTHY\");\n    }\n    LOG.info(\"Earlybird server stopped!\");\n    isShutdown = true;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> search(final EarlybirdRequest request) {\n    final long requestReceivedTimeMillis = System.currentTimeMillis();\n    // Record clock diff as early as possible.\n    EarlybirdRequestUtil.recordClientClockDiff(request);\n\n    if (!futurePoolManager.isPoolReady()) {\n      return Future.exception(new TransientException(\"Earlybird not yet able to handle requests.\"));\n    }\n\n    return futurePoolManager.apply(new Function0<EarlybirdResponse>() {\n      @Override\n      public EarlybirdResponse apply() {\n        return doSearch(request, requestReceivedTimeMillis);\n      }\n    }).rescue(Function.func(\n        // respond with Nack when the queue is full\n        t -> Future.exception((t instanceof RejectedExecutionException) ? QUEUE_FULL_FAILURE : t)));\n  }\n\n  private EarlybirdResponse doSearch(EarlybirdRequest request, long requestReceivedTimeMillis) {\n    final long queueWaitTime = System.currentTimeMillis() - requestReceivedTimeMillis;\n    internalQueueWaitTimeStats.timerIncrement(queueWaitTime);\n\n    // request restart time, not to be confused with startTime which is server restart time\n    Timer timer = new Timer(TimeUnit.MICROSECONDS);\n\n    requestPreLogger.logRequest(request);\n\n    String clientId = ClientIdUtil.getClientIdFromRequest(request);\n    String finagleClientId = FinagleUtil.getFinagleClientName();\n    requestCountersByFinagleIdAndClientId.getUnchecked(new Pair<>(finagleClientId, clientId))\n        .increment();\n\n    EarlybirdRequestUtil.checkAndSetCollectorParams(request);\n\n    // If the thrift logger is busy logging, queue the thrift request for logging.\n    if (EarlybirdThriftRequestLoggingUtil.thriftLoggerBusy) {\n      EarlybirdThriftRequestLoggingUtil.REQUEST_BUFFER.offer(request);\n    }\n\n    EarlybirdRequestUtil.logAndFixExcessiveValues(request);\n\n    final EarlybirdSearcher searcher = new EarlybirdSearcher(\n        request,\n        segmentManager,\n        audioSpaceTable,\n        queryCacheManager,\n        earlybirdIndexConfig.getSchema().getSchemaSnapshot(),\n        earlybirdIndexConfig.getCluster(),\n        dynamicPartitionConfig.getCurrentPartitionConfig(),\n        decider,\n        tweetsSearcherStats,\n        scoringModelsManager,\n        tensorflowModelsManager,\n        clock,\n        multiSegmentTermDictionaryManager,\n        queryTimeoutFactory,\n        qualityFactor);\n\n    QueryCostTracker queryCostTracker = QueryCostTracker.getTracker();\n    EarlybirdResponse response = null;\n    try {\n      if (skipTimedOutRequests\n          && searcher.getTerminationTracker().getTimeoutEndTimeWithReservation()\n          <= clock.nowMillis()) {\n        requestTimeoutExceededBeforeSearchCounter.increment();\n        response = new EarlybirdResponse();\n        response.setResponseCode(EarlybirdResponseCode.SERVER_TIMEOUT_ERROR);\n      } else {\n        queryCostTracker.reset();\n        response = searcher.search();\n      }\n    } finally {\n      if (response == null) {\n        // This can only happen if we failed to catch an exception in the searcher.\n        LOG.error(\"Response was null: \" + request.toString());\n        response = new EarlybirdResponse();\n        response.setResponseCode(EarlybirdResponseCode.TRANSIENT_ERROR);\n      }\n\n      if (response.getSearchResults() == null) {\n        List<ThriftSearchResult> emptyResultSet = Lists.newArrayList();\n        response.setSearchResults(new ThriftSearchResults(emptyResultSet));\n      }\n\n      long reqLatency = timer.stop();\n      response.setResponseTime(reqLatency / 1000);\n      response.setResponseTimeMicros(reqLatency);\n      response.getSearchResults().setQueryCost(queryCostTracker.getTotalCost());\n\n      requestLogger.logRequest(request, response, timer);\n\n      int numResults = EarlybirdRequestLogger.numResultsForLog(response);\n      boolean success = response.getResponseCode() == EarlybirdResponseCode.SUCCESS;\n      boolean clientError = response.getResponseCode() == EarlybirdResponseCode.CLIENT_ERROR;\n      boolean earlyTerminated = (response.getSearchResults().isSetNumPartitionsEarlyTerminated()\n          && response.getSearchResults().getNumPartitionsEarlyTerminated() > 0)\n          || searcher.getTerminationTracker().isEarlyTerminated();\n      // Update termination stats.\n      searcher.getTerminationTracker().getEarlyTerminationState().incrementCount();\n\n      searchStats.requestComplete(reqLatency, numResults, success, earlyTerminated, clientError);\n      if (searcher.getRequestStats() != null) {\n        searcher.getRequestStats().requestComplete(reqLatency, numResults, success,\n            earlyTerminated, clientError);\n      }\n\n      getResponseCodeCounter(response.getResponseCode()).increment();\n      // Adding this counter to make it easier to debug cases where we see a spike in\n      // bad client request errors but don't know where they're coming from. (The\n      // alternative is to ssh to a machine in the cluster and sample\n      // /var/log/earlybird/earlybird.failed_requests).\n      getClientIdResponseCodeCounter(clientId, response.getResponseCode()).increment();\n\n      // Export request latency as a stat.\n      clientIdSearchStats.getUnchecked(clientId).timerIncrement(reqLatency);\n      requestCountersByFinagleClientId.getUnchecked(finagleClientId).timerIncrement(reqLatency);\n      addEarlybirdServerStats(response, queueWaitTime);\n      // Export scoring stats for the request.\n      exportScoringTimeStats(response, clientId);\n    }\n\n    Set<String> queriedFields = searcher.getQueriedFields();\n    if (queriedFields != null) {\n      for (String queriedField : queriedFields) {\n        queriedFieldsCounts.incrementAndGet(queriedField);\n      }\n    }\n\n    // Increment counters for age of the returned results.\n    if (response.getSearchResults() != null && response.getSearchResults().getResults() != null) {\n      long currentTime = System.currentTimeMillis();\n      for (ThriftSearchResult result : response.getSearchResults().getResults()) {\n        long tweetId = result.getId();\n        if (SnowflakeId.isSnowflakeId(tweetId)) {\n          long ageMillis = Math.max(0L,\n              currentTime - SnowflakeId.unixTimeMillisFromId(tweetId));\n          int ageDays = Duration.fromMilliseconds(ageMillis).inDays();\n\n          if (EarlybirdConfig.isRealtimeOrProtected()) {\n            String key = \"result_age_in_days_\" + ageDays;\n            resultsAgeCounter.getUnchecked(key).increment();\n          } else {\n            int ageYears = ageDays / 365;\n            String key = \"result_age_in_years_\" + ageYears;\n            resultsAgeCounter.getUnchecked(key).increment();\n          }\n        }\n      }\n    }\n\n    try {\n      lastRequestPerClientId.get(clientId).set(\n          new RequestResponsePair(request, searcher.getParsedQuery(),\n              searcher.getLuceneQuery(), response));\n    } catch (ExecutionException ex) {\n      // Not a big problem, we'll just notice that the admin page doesn't work, and it\n      // probably won't happen.\n    }\n\n\n    return response;\n  }\n\n  private void exportScoringTimeStats(EarlybirdResponse response, String clientId) {\n    if (response.isSetSearchResults()\n        && response.getSearchResults().isSetScoringTimeNanos()\n        && response.getSearchResults().isSetNumHitsProcessed()) {\n      int numHitsProcessed = response.getSearchResults().getNumHitsProcessed();\n      long scoringTimeNanos = response.getSearchResults().getScoringTimeNanos();\n\n      if (numHitsProcessed > 0) {\n        // Only compute and report scoring time per hit when we have hits. (i.e. we don't just want\n        // to report 0's for cases where there were no hits, and only want to report legit per-hit\n        // times.\n        long scoringTimePerHit = scoringTimeNanos / numHitsProcessed;\n\n        this.clientIdScoringPerHitStats.getUnchecked(clientId).timerIncrement(scoringTimePerHit);\n        this.overallScoringTimePerHitStats.timerIncrement(scoringTimePerHit);\n      }\n\n      this.clientIdScoringPerQueryStats.getUnchecked(clientId).timerIncrement(scoringTimeNanos);\n      this.overallScoringTimePerQueryStats.timerIncrement(scoringTimeNanos);\n\n      // The num hits processed stats here are scoped only to queries that were actually scored.\n      // This would exclude queries like term stats (that would otherwise have huge num hits\n      // processed).\n      this.clientIdScoringNumHitsProcessedStats.getUnchecked(clientId).record(numHitsProcessed);\n      this.overallScoringNumHitsProcessedStats.record(numHitsProcessed);\n    }\n  }\n\n  private void addEarlybirdServerStats(EarlybirdResponse response, long queueWaitTime) {\n    PartitionConfig curPartitionConfig = dynamicPartitionConfig.getCurrentPartitionConfig();\n    EarlybirdServerStats earlybirdServerStats = new EarlybirdServerStats();\n    response.setEarlybirdServerStats(earlybirdServerStats);\n    earlybirdServerStats.setHostname(DatabaseConfig.getLocalHostname());\n    earlybirdServerStats.setPartition(curPartitionConfig.getIndexingHashPartitionID());\n    earlybirdServerStats.setTierName(curPartitionConfig.getTierName());\n    earlybirdServerStats.setCurrentQps(searchStats.getRequestRate());\n    earlybirdServerStats.setQueueTimeMillis(queueWaitTime);\n    earlybirdServerStats.setAverageQueueTimeMillis(\n        (long) (double) internalQueueWaitTimeStats.read());\n    earlybirdServerStats.setAverageLatencyMicros(searchStats.getAverageLatency());\n  }\n\n  @Override\n  public void joinServerSet(String username) throws UpdateException {\n    serverSetManager.joinServerSet(username);\n  }\n\n\n  @Override\n  public int getNumberOfServerSetMembers() throws InterruptedException,\n      ZooKeeperClient.ZooKeeperConnectionException, KeeperException {\n    return serverSetManager.getNumberOfServerSetMembers();\n  }\n\n  @Override\n  public void leaveServerSet(String username) throws UpdateException {\n    serverSetManager.leaveServerSet(username);\n  }\n\n  @Override\n  public void joinServerSetForServiceProxy() {\n    serverSetManager.joinServerSetForServiceProxy();\n  }\n\n  @VisibleForTesting\n  protected static class EarlybirdThriftRequestLoggingUtil {\n    private static final int DEFAULT_MAX_ENTRIES_TO_LOG = 50000;\n    private static final int DEFAULT_BUFFER_SIZE = 10000;\n    private static final int DEFAULT_LOGGING_SLEEP_MS = 100;\n\n    @VisibleForTesting\n    protected static volatile boolean thriftLoggerBusy = false;\n    private static final ExecutorService LOGGING_EXECUTOR = Executors.newCachedThreadPool();\n\n    // Synchronized circular buffer used for buffering requests.\n    // If buffer is full, the oldest requests are replaced. This should not be a problem for\n    // logging purpose.\n    @VisibleForTesting\n    protected static final ArrayBlockingQueue<EarlybirdRequest> REQUEST_BUFFER =\n        new ArrayBlockingQueue<>(DEFAULT_BUFFER_SIZE);\n\n\n    /**\n     * Create a separate thread to log thrift request to the given file. If a thread is already\n     * logging thrift requests, this does nothing and throws an IOException indicating that the\n     * logging thread is busy.\n     *\n     * @param logFile File to log to.\n     * @param maxEntriesToLog Number of entries to log.\n     * @param postLoggingHook Code to run after logging finishes. Only used for testing as of now.\n     */\n    @VisibleForTesting\n    protected static synchronized void startThriftLogging(final File logFile,\n                                                          final int maxEntriesToLog,\n                                                          final Runnable postLoggingHook)\n        throws IOException {\n      if (thriftLoggerBusy) {\n        throw new IOException(\"Already busy logging thrift request. No action taken.\");\n      }\n\n      if (!logFile.canWrite()) {\n        throw new IOException(\"Unable to open log file for writing:  \" + logFile);\n      }\n\n      final BufferedWriter thriftLogWriter =\n          Files.newBufferedWriter(logFile.toPath(), Charsets.UTF_8);\n\n      // TSerializer used by the writer thread.\n      final TSerializer serializer = new TSerializer();\n\n      REQUEST_BUFFER.clear();\n      thriftLoggerBusy = true;\n      LOG.info(\"Started to log thrift requests into file \" + logFile.getAbsolutePath());\n      LOGGING_EXECUTOR.submit(() -> {\n        try {\n          int count = 0;\n          while (count < maxEntriesToLog) {\n            if (REQUEST_BUFFER.isEmpty()) {\n              Thread.sleep(DEFAULT_LOGGING_SLEEP_MS);\n              continue;\n            }\n\n            try {\n              EarlybirdRequest ebRequest = REQUEST_BUFFER.poll();\n              String logLine = serializeThriftObject(ebRequest, serializer);\n              thriftLogWriter.write(logLine);\n              count++;\n            } catch (TException e) {\n              LOG.warn(\"Unable to serialize EarlybirdRequest for logging.\", e);\n            }\n          }\n          return count;\n        } finally {\n          thriftLogWriter.close();\n          thriftLoggerBusy = false;\n          LOG.info(\"Finished logging thrift requests into file \" + logFile.getAbsolutePath());\n          REQUEST_BUFFER.clear();\n          if (postLoggingHook != null) {\n            postLoggingHook.run();\n          }\n        }\n      });\n    }\n\n    /**\n     * Serialize a thrift object to a base 64 encoded string.\n     */\n    private static String serializeThriftObject(TBase<?, ?> tObject, TSerializer serializer)\n        throws TException {\n      return new Base64().encodeToString(serializer.serialize(tObject)) + \"\\n\";\n    }\n  }\n\n  /**\n   * Start to log thrift EarlybirdRequests.\n   *\n   * @param logFile Log file to write to.\n   * @param numRequestsToLog Number of requests to collect.  Default value of 50000 used if\n   * 0 or negative numbers are pass in.\n   */\n  public void startThriftLogging(File logFile, int numRequestsToLog) throws IOException {\n    int requestToLog = numRequestsToLog <= 0\n        ? EarlybirdThriftRequestLoggingUtil.DEFAULT_MAX_ENTRIES_TO_LOG : numRequestsToLog;\n    EarlybirdThriftRequestLoggingUtil.startThriftLogging(logFile, requestToLog, null);\n  }\n\n  @VisibleForTesting\n  @Override\n  public boolean isInServerSet() {\n    return serverSetManager.isInServerSet();\n  }\n\n  @VisibleForTesting\n  SearchCounter getResponseCodeCounter(EarlybirdResponseCode responseCode) {\n    return responseCodeCounters.get(responseCode);\n  }\n\n  @VisibleForTesting\n  SearchCounter getClientIdResponseCodeCounter(\n      String clientId, EarlybirdResponseCode responseCode) {\n    String key = String.format(RESPONSES_PER_CLIENT_ID_STAT_TEMPLATE,\n            clientId, responseCode.name().toLowerCase());\n    return responseByClientIdAndResponseCode.getUnchecked(key);\n  }\n\n  public void setNoShutdownWhenNotInLayout(boolean noShutdown) {\n    stateManager.setNoShutdownWhenNotInLayout(noShutdown);\n  }\n\n  private void schedule(OneTaskScheduledExecutorManager manager) {\n    if (!isShuttingDown) {\n      manager.schedule();\n      toClose.add(manager);\n    }\n  }\n\n  public DynamicSchema getSchema() {\n    return earlybirdIndexConfig.getSchema();\n  }\n\n  public AudioSpaceTable getAudioSpaceTable() {\n    return audioSpaceTable;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/EarlybirdServerSetManager.java",
    "content": "package com.twitter.search.earlybird;\n\nimport java.net.InetAddress;\nimport java.net.InetSocketAddress;\nimport java.util.concurrent.atomic.AtomicLong;\n\nimport javax.annotation.concurrent.GuardedBy;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Maps;\n\nimport org.apache.zookeeper.KeeperException;\nimport org.apache.zookeeper.Watcher;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.zookeeper.ServerSet;\nimport com.twitter.common.zookeeper.ZooKeeperClient;\nimport com.twitter.common_internal.zookeeper.TwitterServerSet;\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.common.database.DatabaseConfig;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.util.zookeeper.ZooKeeperProxy;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.config.TierConfig;\nimport com.twitter.search.earlybird.exception.AlreadyInServerSetUpdateException;\nimport com.twitter.search.earlybird.exception.NotInServerSetUpdateException;\nimport com.twitter.search.earlybird.partition.PartitionConfig;\n\npublic class EarlybirdServerSetManager implements ServerSetMember {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdServerSetManager.class);\n\n  // How many times this earlybird joined/left its partition's server set\n  @VisibleForTesting\n  protected final SearchCounter leaveServerSetCounter;\n  @VisibleForTesting\n  protected final SearchCounter joinServerSetCounter;\n  private final ZooKeeperProxy discoveryZKClient;\n  private final SearchLongGauge inServerSetGauge;\n  private final PartitionConfig partitionConfig;\n  private final int port;\n  private final String serverSetNamePrefix;\n\n  @VisibleForTesting\n  protected final SearchLongGauge connectedToZooKeeper;\n\n  private final Object endpointStatusLock = new Object();\n  @GuardedBy(\"endpointStatusLock\")\n  private ServerSet.EndpointStatus endpointStatus = null;\n\n  private boolean inServerSetForServiceProxy = false;\n\n  public EarlybirdServerSetManager(\n      SearchStatsReceiver searchStatsReceiver,\n      ZooKeeperProxy discoveryZKClient,\n      final PartitionConfig partitionConfig,\n      int port,\n      String serverSetNamePrefix) {\n    this.discoveryZKClient = discoveryZKClient;\n    this.partitionConfig = partitionConfig;\n    this.port = port;\n    this.serverSetNamePrefix = serverSetNamePrefix;\n\n    // Export serverset related stats\n    Preconditions.checkNotNull(searchStatsReceiver);\n    this.joinServerSetCounter = searchStatsReceiver.getCounter(\n        serverSetNamePrefix + \"join_server_set_count\");\n    this.leaveServerSetCounter = searchStatsReceiver.getCounter(\n        serverSetNamePrefix + \"leave_server_set_count\");\n\n    // Create a new stat based on the partition number for hosts-in-partition aggregation.\n    // The value of the stat is dependent on whether the server is in the serverset so that the\n    // aggregate stat reflects the number serving traffic instead of the live process count.\n    AtomicLong sharedInServerSetStatus = new AtomicLong();\n    this.inServerSetGauge = searchStatsReceiver.getLongGauge(\n        serverSetNamePrefix + \"is_in_server_set\", sharedInServerSetStatus);\n    this.connectedToZooKeeper = searchStatsReceiver.getLongGauge(\n        serverSetNamePrefix + \"connected_to_zookeeper\");\n\n    searchStatsReceiver.getLongGauge(\n        serverSetNamePrefix + \"member_of_partition_\" + partitionConfig.getIndexingHashPartitionID(),\n        sharedInServerSetStatus);\n\n    this.discoveryZKClient.registerExpirationHandler(() -> connectedToZooKeeper.set(0));\n\n    this.discoveryZKClient.register(event -> {\n      if (event.getType() == Watcher.Event.EventType.None\n          && event.getState() == Watcher.Event.KeeperState.SyncConnected) {\n        connectedToZooKeeper.set(1);\n      }\n    });\n  }\n\n  /**\n   * Join ServerSet and update endpointStatus.\n   * This will allow Earlybird consumers, e.g. Blender, to detect when an\n   * Earlybird goes online and offline.\n   * @param username\n   */\n  @Override\n  public void joinServerSet(String username) throws ServerSet.UpdateException {\n    joinServerSetCounter.increment();\n\n    synchronized (endpointStatusLock) {\n      LOG.info(\"Joining {} ServerSet (instructed by: {}) ...\", serverSetNamePrefix, username);\n      if (endpointStatus != null) {\n        LOG.warn(\"Already in ServerSet. Nothing done.\");\n        throw new AlreadyInServerSetUpdateException(\"Already in ServerSet. Nothing done.\");\n      }\n\n      try {\n        TwitterServerSet.Service service = getServerSetService();\n\n        ServerSet serverSet = discoveryZKClient.createServerSet(service);\n        endpointStatus = serverSet.join(\n            new InetSocketAddress(InetAddress.getLocalHost().getHostName(), port),\n            Maps.newHashMap(),\n            partitionConfig.getHostPositionWithinHashPartition());\n\n        inServerSetGauge.set(1);\n\n        String path = service.getPath();\n        EarlybirdStatus.recordEarlybirdEvent(\"Joined \" + serverSetNamePrefix + \" ServerSet \" + path\n                                             + \" (instructed by: \" + username + \")\");\n        LOG.info(\"Successfully joined {} ServerSet {} (instructed by: {})\",\n                 serverSetNamePrefix, path, username);\n      } catch (Exception e) {\n        endpointStatus = null;\n        String message = \"Failed to join \" + serverSetNamePrefix + \" ServerSet of partition \"\n            + partitionConfig.getIndexingHashPartitionID();\n        LOG.error(message, e);\n        throw new ServerSet.UpdateException(message, e);\n      }\n    }\n  }\n\n  /**\n   * Takes this Earlybird out of its registered ServerSet.\n   *\n   * @throws ServerSet.UpdateException if there was a problem leaving the ServerSet,\n   * or if this Earlybird is already not in a ServerSet.\n   * @param username\n   */\n  @Override\n  public void leaveServerSet(String username) throws ServerSet.UpdateException {\n    leaveServerSetCounter.increment();\n    synchronized (endpointStatusLock) {\n      LOG.info(\"Leaving {} ServerSet (instructed by: {}) ...\", serverSetNamePrefix, username);\n      if (endpointStatus == null) {\n        String message = \"Not in a ServerSet. Nothing done.\";\n        LOG.warn(message);\n        throw new NotInServerSetUpdateException(message);\n      }\n\n      endpointStatus.leave();\n      endpointStatus = null;\n      inServerSetGauge.set(0);\n      EarlybirdStatus.recordEarlybirdEvent(\"Left \" + serverSetNamePrefix\n                                           + \" ServerSet (instructed by: \" + username + \")\");\n      LOG.info(\"Successfully left {} ServerSet. (instructed by: {})\",\n               serverSetNamePrefix, username);\n    }\n  }\n\n  @Override\n  public int getNumberOfServerSetMembers()\n      throws InterruptedException, ZooKeeperClient.ZooKeeperConnectionException, KeeperException {\n    String path = getServerSetService().getPath();\n    return discoveryZKClient.getNumberOfServerSetMembers(path);\n  }\n\n  /**\n   * Determines if this earlybird is in the server set.\n   */\n  @Override\n  public boolean isInServerSet() {\n    synchronized (endpointStatusLock) {\n      return endpointStatus != null;\n    }\n  }\n\n  /**\n   * Returns the server set that this earlybird should join.\n   */\n  public String getServerSetIdentifier() {\n    TwitterServerSet.Service service = getServerSetService();\n    return String.format(\"/cluster/local/%s/%s/%s\",\n                         service.getRole(),\n                         service.getEnv(),\n                         service.getName());\n  }\n\n  private TwitterServerSet.Service getServerSetService() {\n    // If the tier name is 'all' then it treat it as an untiered EB cluster\n    // and do not add the tier component into the ZK path it registers under.\n    String tierZKPathComponent = \"\";\n    if (!TierConfig.DEFAULT_TIER_NAME.equalsIgnoreCase(partitionConfig.getTierName())) {\n      tierZKPathComponent = \"/\" + partitionConfig.getTierName();\n    }\n    if (EarlybirdConfig.isAurora()) {\n      // ROLE, EARYLBIRD_NAME, and ENV properties are required on Aurora, thus will be set here\n      return new TwitterServerSet.Service(\n          EarlybirdProperty.ROLE.get(),\n          EarlybirdProperty.ENV.get(),\n          getServerSetPath(EarlybirdProperty.EARLYBIRD_NAME.get() + tierZKPathComponent));\n    } else {\n      return new TwitterServerSet.Service(\n          DatabaseConfig.getZooKeeperRole(),\n          Config.getEnvironment(),\n          getServerSetPath(\"earlybird\" + tierZKPathComponent));\n    }\n  }\n\n  private String getServerSetPath(String earlybirdName) {\n    return String.format(\"%s%s/hash_partition_%d\", serverSetNamePrefix, earlybirdName,\n        partitionConfig.getIndexingHashPartitionID());\n  }\n\n  /**\n   * Join ServerSet for ServiceProxy with a named admin port and with a zookeeper path that Service\n   * Proxy can translate to a domain name label that is less than 64 characters (due to the size\n   * limit for domain name labels described here: https://tools.ietf.org/html/rfc1035)\n   * This will allow us to access Earlybirds that are not on mesos via ServiceProxy.\n   */\n  @Override\n  public void joinServerSetForServiceProxy() {\n    // This additional Zookeeper server set is only necessary for Archive Earlybirds which are\n    // running on bare metal hardware, so ensure that this method is never called for services\n    // on Aurora.\n    Preconditions.checkArgument(!EarlybirdConfig.isAurora(),\n        \"Attempting to join server set for ServiceProxy on Earlybird running on Aurora\");\n\n    LOG.info(\"Attempting to join ServerSet for ServiceProxy\");\n    try {\n      TwitterServerSet.Service service = getServerSetForServiceProxyOnArchive();\n\n      ServerSet serverSet = discoveryZKClient.createServerSet(service);\n      String hostName = InetAddress.getLocalHost().getHostName();\n      int adminPort = EarlybirdConfig.getAdminPort();\n      serverSet.join(\n          new InetSocketAddress(hostName, port),\n          ImmutableMap.of(\"admin\", new InetSocketAddress(hostName, adminPort)),\n          partitionConfig.getHostPositionWithinHashPartition());\n\n      String path = service.getPath();\n      LOG.info(\"Successfully joined ServerSet for ServiceProxy {}\", path);\n      inServerSetForServiceProxy = true;\n    } catch (Exception e) {\n      String message = \"Failed to join ServerSet for ServiceProxy of partition \"\n          + partitionConfig.getIndexingHashPartitionID();\n      LOG.warn(message, e);\n    }\n  }\n\n  @VisibleForTesting\n  protected TwitterServerSet.Service getServerSetForServiceProxyOnArchive() {\n    String serverSetPath = String.format(\"proxy/%s/p_%d\",\n        partitionConfig.getTierName(),\n        partitionConfig.getIndexingHashPartitionID());\n    return new TwitterServerSet.Service(\n        DatabaseConfig.getZooKeeperRole(),\n        Config.getEnvironment(),\n        serverSetPath);\n  }\n\n  @VisibleForTesting\n  protected boolean isInServerSetForServiceProxy() {\n    return inServerSetForServiceProxy;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/EarlybirdStatus.java",
    "content": "package com.twitter.search.earlybird;\n\nimport java.text.SimpleDateFormat;\nimport java.util.Date;\nimport java.util.List;\nimport java.util.Optional;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.atomic.AtomicBoolean;\n\nimport com.google.common.collect.Lists;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.BuildInfo;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusCode;\nimport com.twitter.util.Duration;\n\n/**\n * High level status of an Earlybird server. SEARCH-28016\n */\npublic final class EarlybirdStatus {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdStatus.class);\n\n  private static final String BUILD_SHA = getBuildShaFromVars();\n\n  protected static long startTime;\n  protected static EarlybirdStatusCode statusCode;\n  protected static String statusMessage;\n  protected static final AtomicBoolean THRIFT_PORT_OPEN = new AtomicBoolean(false);\n  protected static final AtomicBoolean WARMUP_THRIFT_PORT_OPEN = new AtomicBoolean(false);\n  protected static final AtomicBoolean THRIFT_SERVICE_STARTED = new AtomicBoolean(false);\n\n  private static final List<EarlybirdEvent> EARLYBIRD_SERVER_EVENTS = Lists.newArrayList();\n  private static class EarlybirdEvent {\n    private final String eventName;\n    private final long timestampMillis;\n    private final long timeSinceServerStartMillis;\n    private final long durationMillis;\n\n    public EarlybirdEvent(String eventName, long timestampMillis) {\n      this(eventName, timestampMillis, -1);\n    }\n\n    public EarlybirdEvent(\n        String eventName,\n        long timestampMillis,\n        long eventDurationMillis) {\n      this.eventName = eventName;\n      this.timestampMillis = timestampMillis;\n      this.timeSinceServerStartMillis = timestampMillis - startTime;\n      this.durationMillis = eventDurationMillis;\n    }\n\n    public String getEventLogString() {\n      String result = String.format(\n          \"%s %s\",\n          new SimpleDateFormat(\"yyyy-MM-dd HH:mm:ss.SSS\").format(new Date(timestampMillis)),\n          eventName);\n\n      if (durationMillis > 0) {\n        result += String.format(\n            \", took: %s\", Duration.apply(durationMillis, TimeUnit.MILLISECONDS).toString());\n      }\n\n      result += String.format(\n          \", time since server start: %s\",\n          Duration.apply(timeSinceServerStartMillis, TimeUnit.MILLISECONDS).toString()\n      );\n\n      return result;\n    }\n  }\n\n  private EarlybirdStatus() {\n  }\n\n  public static synchronized void setStartTime(long time) {\n    startTime = time;\n    LOG.info(\"startTime set to \" + time);\n  }\n\n  public static synchronized void setStatus(EarlybirdStatusCode code) {\n    setStatus(code, null);\n  }\n\n  public static synchronized void setStatus(EarlybirdStatusCode code, String message) {\n    statusCode = code;\n    statusMessage = message;\n    LOG.info(\"status set to \" + code + (message != null ? \" with message \" + message : \"\"));\n  }\n\n  public static synchronized long getStartTime() {\n    return startTime;\n  }\n\n  public static synchronized boolean isStarting() {\n    return statusCode == EarlybirdStatusCode.STARTING;\n  }\n\n  public static synchronized boolean hasStarted() {\n    return statusCode == EarlybirdStatusCode.CURRENT;\n  }\n\n  public static boolean isThriftServiceStarted() {\n    return THRIFT_SERVICE_STARTED.get();\n  }\n\n  public static synchronized EarlybirdStatusCode getStatusCode() {\n    return statusCode;\n  }\n\n  public static synchronized String getStatusMessage() {\n    return (statusMessage == null ? \"\" : statusMessage + \", \")\n        + \"warmup thrift port is \" + (WARMUP_THRIFT_PORT_OPEN.get() ? \"OPEN\" : \"CLOSED\")\n        + \", production thrift port is \" + (THRIFT_PORT_OPEN.get() ? \"OPEN\" : \"CLOSED\");\n  }\n\n  public static synchronized void recordEarlybirdEvent(String eventName) {\n    long timeMillis = System.currentTimeMillis();\n    EARLYBIRD_SERVER_EVENTS.add(new EarlybirdEvent(eventName, timeMillis));\n  }\n\n  private static String getBeginEventMessage(String eventName) {\n    return \"[Begin Event] \" + eventName;\n  }\n\n  private static String getEndEventMessage(String eventName) {\n    return \"[ End Event ] \" + eventName;\n  }\n\n  /**\n   * Records the beginning of the given event.\n   *\n   * @param eventName The event name.\n   * @param startupMetric The metric that will be used to keep track of the time for this event.\n   */\n  public static synchronized void beginEvent(String eventName,\n                                             SearchIndexingMetricSet.StartupMetric startupMetric) {\n    long timeMillis = System.currentTimeMillis();\n    String eventMessage = getBeginEventMessage(eventName);\n    LOG.info(eventMessage);\n    EARLYBIRD_SERVER_EVENTS.add(new EarlybirdEvent(eventMessage, timeMillis));\n\n    startupMetric.begin();\n  }\n\n  /**\n   * Records the end of the given event.\n   *\n   * @param eventName The event name.\n   * @param startupMetric The metric used to keep track of the time for this event.\n   */\n  public static synchronized void endEvent(String eventName,\n                                           SearchIndexingMetricSet.StartupMetric startupMetric) {\n    long timeMillis = System.currentTimeMillis();\n\n    String beginEventMessage = getBeginEventMessage(eventName);\n    Optional<EarlybirdEvent> beginEventOpt = EARLYBIRD_SERVER_EVENTS.stream()\n        .filter(event -> event.eventName.equals(beginEventMessage))\n        .findFirst();\n\n    String eventMessage = getEndEventMessage(eventName);\n    LOG.info(eventMessage);\n    EarlybirdEvent endEvent = new EarlybirdEvent(\n        eventMessage,\n        timeMillis,\n        beginEventOpt.map(e -> timeMillis - e.timestampMillis).orElse(-1L));\n\n    EARLYBIRD_SERVER_EVENTS.add(endEvent);\n\n    startupMetric.end(endEvent.durationMillis);\n  }\n\n  public static synchronized void clearAllEvents() {\n    EARLYBIRD_SERVER_EVENTS.clear();\n  }\n\n  public static String getBuildSha() {\n    return BUILD_SHA;\n  }\n\n  /**\n   * Returns the list of all earlybird events that happened since the server started.\n   */\n  public static synchronized Iterable<String> getEarlybirdEvents() {\n    List<String> eventLog = Lists.newArrayListWithCapacity(EARLYBIRD_SERVER_EVENTS.size());\n    for (EarlybirdEvent event : EARLYBIRD_SERVER_EVENTS) {\n      eventLog.add(event.getEventLogString());\n    }\n    return eventLog;\n  }\n\n  private static String getBuildShaFromVars() {\n    BuildInfo buildInfo = new BuildInfo();\n    String buildSha = buildInfo.getProperties().getProperty(BuildInfo.Key.GIT_REVISION.value);\n    if (buildSha != null) {\n      return buildSha;\n    } else {\n      return \"UNKNOWN\";\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/EarlybirdWarmUpManager.java",
    "content": "package com.twitter.search.earlybird;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.common.zookeeper.ServerSet;\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.decider.DeciderUtil;\nimport com.twitter.search.earlybird.partition.PartitionConfig;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusCode;\n\npublic class EarlybirdWarmUpManager {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdWarmUpManager.class);\n  private static final String WARM_UP_ON_DURATION_DECIDER_KEY_PATTERN =\n      \"%s_warm_up_duration_seconds\";\n\n  private final EarlybirdServerSetManager earlybirdServerSetManager;\n  private final String clusterName;\n  private final SearchIndexingMetricSet.StartupMetric startUpInWarmUpMetric;\n  private final Decider decider;\n  private final Clock clock;\n\n  public EarlybirdWarmUpManager(EarlybirdServerSetManager earlybirdServerSetManager,\n                                PartitionConfig partitionConfig,\n                                SearchIndexingMetricSet searchIndexingMetricSet,\n                                Decider decider,\n                                Clock clock) {\n    this.earlybirdServerSetManager = earlybirdServerSetManager;\n    this.clusterName = partitionConfig.getClusterName();\n    this.startUpInWarmUpMetric = searchIndexingMetricSet.startupInWarmUp;\n    this.decider = decider;\n    this.clock = clock;\n  }\n\n  public String getServerSetIdentifier() {\n    return earlybirdServerSetManager.getServerSetIdentifier();\n  }\n\n  /**\n   * Warms up the earlybird. The earlybird joins a special server set that gets production dark\n   * reads, and leaves this server set after a specified period of time.\n   */\n  public void warmUp() throws InterruptedException, ServerSet.UpdateException {\n    int warmUpDurationSeconds = DeciderUtil.getAvailability(\n        decider,\n        String.format(WARM_UP_ON_DURATION_DECIDER_KEY_PATTERN, clusterName.replaceAll(\"-\", \"_\")));\n    if (warmUpDurationSeconds == 0) {\n      LOG.info(String.format(\"Warm up stage duration for cluster %s set to 0. Skipping.\",\n                             clusterName));\n      return;\n    }\n\n    earlybirdServerSetManager.joinServerSet(\"internal warm up\");\n\n    // If doWarmUp() is interrupted, try to leave the server set, and propagate the\n    // InterruptedException. Otherwise, try to leave the server set, and propagate any exception\n    // that it might throw.\n    InterruptedException warmUpInterruptedException = null;\n    try {\n      doWarmUp(warmUpDurationSeconds);\n    } catch (InterruptedException e) {\n      warmUpInterruptedException = e;\n      throw e;\n    } finally {\n      if (warmUpInterruptedException != null) {\n        try {\n          earlybirdServerSetManager.leaveServerSet(\"internal warm up\");\n        } catch (Exception e) {\n          warmUpInterruptedException.addSuppressed(e);\n        }\n      } else {\n        earlybirdServerSetManager.leaveServerSet(\"internal warm up\");\n      }\n    }\n  }\n\n  @VisibleForTesting\n  protected void doWarmUp(int warmUpDurationSeconds) throws InterruptedException {\n    long warmUpStartTimeMillis = clock.nowMillis();\n    LOG.info(String.format(\"Warming up for %d seconds.\", warmUpDurationSeconds));\n    EarlybirdStatus.beginEvent(\"warm_up\", startUpInWarmUpMetric);\n\n    // Sleep for warmUpDurationSeconds seconds, but check if the server is going down every second.\n    int count = 0;\n    try {\n      while ((count++ < warmUpDurationSeconds)\n             && (EarlybirdStatus.getStatusCode() != EarlybirdStatusCode.STOPPING)) {\n        clock.waitFor(1000);\n      }\n    } finally {\n      LOG.info(String.format(\"Done warming up after %d milliseconds.\",\n                             clock.nowMillis() - warmUpStartTimeMillis));\n      EarlybirdStatus.endEvent(\"warm_up\", startUpInWarmUpMetric);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/QualityFactor.java",
    "content": "package com.twitter.search.earlybird;\n\n/**\n * Interface defining a quality factor.\n */\npublic interface QualityFactor {\n  /**\n   * Returns the current quality factor.\n   * @return The quality factor; a number between 0.0 and 1.0.\n   */\n  double get();\n\n  /**\n   * Starts a thread to update the quality factor periodically.\n   */\n  void startUpdates();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/README.md",
    "content": "# Search Index (Earlybird) main classes\n\n> **TL;DR** Earlybird (Search Index) find tweets from people you follow, rank them, and serve them to Home.\n\n## What is Earlybird (Search Index)\n\n[Earlybird](http://notes.stephenholiday.com/Earlybird.pdf) is a **real-time search system** based on [Apache Lucene](https://lucene.apache.org/) to support the high volume of queries and content updates. The major use cases are Relevance Search (specifically, Text search) and Timeline In-network Tweet retrieval (or UserID based search). It is designed to enable the efficient indexing and querying of billions of tweets, and to provide low-latency search results, even with heavy query loads.\n\n## High-level architecture\nWe split our entire tweet search index into three clusters: a **realtime** cluster indexing all public tweets posted in about the last 7 days, a **protected** cluster indexing all protected tweets for the same timeframe; and an **archive** cluster indexing all tweets ever posted, up to about two days ago.\n\nEarlybird addresses the challenges of scaling real-time search by splitting each cluster across multiple **partitions**, each responsible for a portion of the index. The architecture uses a distributed *inverted index* that is sharded and replicated. This design allows for efficient index updates and query processing.\n\nThe system also employs an incremental indexing approach, enabling it to process and index new tweets in real-time as they arrive. With single writer, multiple reader structure, Earlybird can handle a large number of real-time updates and queries concurrently while maintaining low query latency. The system can achieve high query throughput and low query latency while maintaining a high degree of index freshness.\n\n## Main Components \n\n**Partition Manager**: Responsible for managing the configuration of partitions, as well as the mapping between users and partitions. It also handles index loading and flushing.\n\n**Real-time Indexer**: Continuously reads from a kafka stream of incoming tweets and updates the index (tweet creation, tweet updates, user updates). It also supports tweet deletion events.\n\n**Query Engine**: Handles the execution of search queries against the distributed index. It employs various optimization techniques, such as term-based pruning and caching.\n\n**Document Preprocessor**: Converts raw tweets into a document representation suitable for indexing. It handles tokenization, normalization, and analysis of tweet text and metadata. See our ingestion pipeline `src/java/com/twitter/search/ingester` for more write-path processing.\n\n**Index Writer**: Writes tweet documents to the index and maintains the index structure, including **posting lists** and **term dictionaries**.\n\n**Segment Manager**: Manages index segments within a partition. It is responsible for merging, optimizing, and flushing index segments to disk, or flush to HDFS to snapshot live segments.\n\n**Searcher**: Executes queries against the index, using techniques like caching and parallel query execution to minimize query latency. It also incorporates scoring models and ranking algorithms to provide relevant search results.\n\nThe most important two data structures for Earlybird (or Information Retrieval in general) including:\n\n* **Inverted Index** which stores a mapping between a Term to a list of Doc IDs. Essentially, we build a hash map: each key in the map is a distinct Term (e.g., `cat`, `dog`) in a tweet, and each value is the list of tweets (aka., Document) in which the word appears. We keep one inverted index per field (text, UserID, user name, links, etc.)\n* **Postings List** which optimize the storage a the list of Doc IDs mentioned above.\n\nSee more at: https://blog.twitter.com/engineering/en_us/topics/infrastructure/2016/omnisearch-index-formats\n\n## Advanced features\n\nEarlybird incorporates several advanced features such as facet search, which allows users to refine search results based on specific attributes such as user mentions, hashtags, and URLs. Furthermore, the system supports various ranking models, including machine learning-based scoring models, to provide relevant search results.\n\n## Directory Structure\nThe project consists of several packages and files, which can be summarized as follows:\n\n* At the root level, the primary focus is on the Earlybird server implementation and its associated classes. These include classes for search, CPU quality factors, server management, index config, main classes, server startup, etc.\n* `archive/`: Directory deals with the management and configuration of archived data, specifically for Earlybird Index Configurations. It also contains a `segmentbuilder/` subdirectory, which includes classes for building and updating archive index segments.\n* `common/`: Directory holds utility classes for logging, handling requests, and Thrift backend functionality. It also has two subdirectories: `config/` for Earlybird configuration and `userupdates/` for user-related data handling.\n* `config/`: Directory is dedicated to managing tier configurations specifically for archive cluster, which relate to server and search query distribution.\n* `document/`: Handles document creation and processing, including various factories and token stream writers.\n* `exception/`: Contains custom exceptions and exception handling classes related to the system.\n* `factory/`: Provides utilities and factories for configurations, Kafka consumers, and server instances.\n* `index/`: Contains index-related classes, including in-memory time mappers, tweet ID mappers, and facets.\n* `ml/`: Houses the `ScoringModelsManager` for managing machine learning models.\n* `partition/`: Manages partitions and index segments, including index loaders, segment writers, and startup indexers.\n* `querycache/`: Implements caching for queries and query results, including cache configuration and update tasks.\n* `queryparser/`: Provides query parsing functionality, including files that cover query rewriters and lhigh-frequency term extraction.\n* `search/`: Contains read path related classes, such as search request processing, result collectors, and facet collectors.\n* `segment/`: Provides classes for managing segment data providers and data reader sets.\n* `stats/`: Contains classes for tracking and reporting statistics related to the system.\n* `tools/`: Houses utility classes for deserializing thrift requests.\n* `util/`: Includes utility classes for various tasks, such as action logging, scheduled tasks, and JSON viewers.\n\n## Related Services\n\n* The Earlybirds sit behind Earlybird Root servers that fan out queries to them. See `src/java/com/twitter/search/earlybird_root/`\n* The Earlybirds are powered by multiple ingestion pipelines. See `src/java/com/twitter/search/ingester/`\n* Earlybird segments for the Archives are built offline by segment builders\n* Also, Earlybird light ranking is defined in `timelines/data_processing/ad_hoc/earlybird_ranking`\n and `src/python/twitter/deepbird/projects/timelines/scripts/models/earlybird`.\n* Search common library/packages\n\n## References\n\nSee more: \n\n* \"Earlybird: Real-Time Search at Twitter\" (http://notes.stephenholiday.com/Earlybird.pdf)\n* \"Reducing search indexing latency to one second\" (https://blog.twitter.com/engineering/en_us/topics/infrastructure/2020/reducing-search-indexing-latency-to-one-second)\n* \"Omnisearch index formats\" (https://blog.twitter.com/engineering/en_us/topics/infrastructure/2016/omnisearch-index-formats)\n\n\n\n\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/RealtimeEarlybirdIndexConfig.java",
    "content": "package com.twitter.search.earlybird;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.IndexWriterConfig;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.store.Directory;\nimport org.apache.lucene.store.RAMDirectory;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.schema.DynamicSchema;\nimport com.twitter.search.common.schema.SearchWhitespaceAnalyzer;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.util.CloseResourceUtil;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentData;\nimport com.twitter.search.core.earlybird.index.EarlybirdRealtimeIndexSegmentData;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdIndexExtensionsFactory;\nimport com.twitter.search.core.earlybird.index.inverted.IndexOptimizer;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.index.OptimizedTimeMapper;\nimport com.twitter.search.earlybird.index.OptimizedTweetIDMapper;\nimport com.twitter.search.earlybird.index.OutOfOrderRealtimeTweetIDMapper;\nimport com.twitter.search.earlybird.index.RealtimeTimeMapper;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\nimport com.twitter.search.earlybird.partition.SegmentSyncInfo;\n\n/**\n * Index config for the Real-Time in-memory Tweet cluster.\n */\npublic class RealtimeEarlybirdIndexConfig extends EarlybirdIndexConfig {\n  private final CloseResourceUtil resourceCloser = new CloseResourceUtil();\n\n  public RealtimeEarlybirdIndexConfig(\n      EarlybirdCluster cluster, Decider decider, SearchIndexingMetricSet searchIndexingMetricSet,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    super(cluster, decider, searchIndexingMetricSet, criticalExceptionHandler);\n  }\n\n  public RealtimeEarlybirdIndexConfig(\n      EarlybirdCluster cluster, DynamicSchema schema, Decider decider,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    super(cluster, schema, decider, searchIndexingMetricSet, criticalExceptionHandler);\n  }\n\n  @Override\n  public Directory newLuceneDirectory(SegmentSyncInfo segmentSyncInfo) {\n    return new RAMDirectory();\n  }\n\n  @Override\n  public IndexWriterConfig newIndexWriterConfig() {\n    return new IndexWriterConfig(new SearchWhitespaceAnalyzer())\n        .setSimilarity(IndexSearcher.getDefaultSimilarity());\n  }\n\n  @Override\n  public EarlybirdIndexSegmentData newSegmentData(\n      int maxSegmentSize,\n      long timeSliceID,\n      Directory dir,\n      EarlybirdIndexExtensionsFactory extensionsFactory) {\n    return new EarlybirdRealtimeIndexSegmentData(\n        maxSegmentSize,\n        timeSliceID,\n        getSchema(),\n        new OutOfOrderRealtimeTweetIDMapper(maxSegmentSize, timeSliceID),\n        new RealtimeTimeMapper(maxSegmentSize),\n        extensionsFactory);\n  }\n\n  @Override\n  public EarlybirdIndexSegmentData loadSegmentData(\n          FlushInfo flushInfo,\n          DataDeserializer dataInputStream,\n          Directory dir,\n          EarlybirdIndexExtensionsFactory extensionsFactory) throws IOException {\n    EarlybirdRealtimeIndexSegmentData.InMemorySegmentDataFlushHandler flushHandler;\n    boolean isOptimized = flushInfo.getBooleanProperty(\n        EarlybirdIndexSegmentData.AbstractSegmentDataFlushHandler.IS_OPTIMIZED_PROP_NAME);\n    if (isOptimized) {\n      flushHandler = new EarlybirdRealtimeIndexSegmentData.InMemorySegmentDataFlushHandler(\n          getSchema(),\n          extensionsFactory,\n          new OptimizedTweetIDMapper.FlushHandler(),\n          new OptimizedTimeMapper.FlushHandler());\n    } else {\n      flushHandler = new EarlybirdRealtimeIndexSegmentData.InMemorySegmentDataFlushHandler(\n          getSchema(),\n          extensionsFactory,\n          new OutOfOrderRealtimeTweetIDMapper.FlushHandler(),\n          new RealtimeTimeMapper.FlushHandler());\n    }\n\n\n    return flushHandler.load(flushInfo, dataInputStream);\n  }\n\n  @Override\n  public EarlybirdIndexSegmentData optimize(\n      EarlybirdIndexSegmentData earlybirdIndexSegmentData) throws IOException {\n    Preconditions.checkArgument(\n        earlybirdIndexSegmentData instanceof EarlybirdRealtimeIndexSegmentData,\n        \"Expected EarlybirdRealtimeIndexSegmentData but got %s\",\n        earlybirdIndexSegmentData.getClass());\n\n    return IndexOptimizer.optimize((EarlybirdRealtimeIndexSegmentData) earlybirdIndexSegmentData);\n  }\n\n  @Override\n  public boolean isIndexStoredOnDisk() {\n    return false;\n  }\n\n  @Override\n  public final CloseResourceUtil getResourceCloser() {\n    return resourceCloser;\n  }\n\n  @Override\n  public boolean supportOutOfOrderIndexing() {\n    return true;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/RecentTweetRestriction.java",
    "content": "package com.twitter.search.earlybird;\n\nimport scala.Option;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport com.twitter.decider.Decider;\n\npublic final class RecentTweetRestriction {\n  private static final String RECENT_TWEETS_THRESHOLD = \"recent_tweets_threshold\";\n  private static final String QUERY_CACHE_UNTIL_TIME = \"query_cache_until_time\";\n\n  @VisibleForTesting\n  public static final int DEFAULT_RECENT_TWEET_SECONDS = 15;\n\n  private RecentTweetRestriction() {\n  }\n\n  /**\n   * Returns the point in time (in seconds past the unix epoch) before which all tweets will be\n   * completely indexed. This is required by some clients, because they rely on Earlybird monotonically\n   * indexing tweets by ID and that tweets are completely indexed when they see them.\n   *\n   * @param lastTime The time at which the most recent tweet was indexed, in seconds since the unix\n   * epoch.\n   */\n  public static int recentTweetsUntilTime(Decider decider, int lastTime) {\n    return untilTimeSeconds(decider, lastTime, RECENT_TWEETS_THRESHOLD);\n  }\n\n  /**\n   * Returns the point in time (in seconds past the unix epoch) before which all tweets will be\n   * completely indexed. This is required by some clients, because they rely on Earlybird monotonically\n   * indexing tweets by ID and that tweets are completely indexed when they see them.\n   *\n   * @param lastTime The time at which the most recent tweet was indexed, in seconds since the unix\n   * epoch.\n   */\n  public static int queryCacheUntilTime(Decider decider, int lastTime) {\n    return untilTimeSeconds(decider, lastTime, QUERY_CACHE_UNTIL_TIME);\n  }\n\n  private static int untilTimeSeconds(Decider decider, int lastTime, String deciderKey) {\n    int recentTweetSeconds = getRecentTweetSeconds(decider, deciderKey);\n\n    if (recentTweetSeconds == 0) {\n      return 0;\n    }\n\n    return lastTime - recentTweetSeconds;\n  }\n\n  private static int getRecentTweetSeconds(Decider decider, String deciderKey) {\n    Option<Object> deciderValue = decider.getAvailability(deciderKey);\n    if (deciderValue.isDefined()) {\n      return (int) deciderValue.get();\n    }\n    return DEFAULT_RECENT_TWEET_SECONDS;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/ServerSetMember.java",
    "content": "package com.twitter.search.earlybird;\n\nimport org.apache.zookeeper.KeeperException;\n\nimport com.twitter.common.zookeeper.ServerSet;\nimport com.twitter.common.zookeeper.ZooKeeperClient;\n\n/**\n * Represents a server that can add and remove itself from a server set.\n */\npublic interface ServerSetMember {\n  /**\n   * Makes this server join its server set.\n   *\n   * @throws ServerSet.UpdateException\n   * @param requestSource\n   */\n  void joinServerSet(String requestSource) throws ServerSet.UpdateException;\n\n  /**\n   * Makes this server leave its server set.\n   *\n   * @throws ServerSet.UpdateException\n   * @param requestSource\n   */\n  void leaveServerSet(String requestSource) throws ServerSet.UpdateException;\n\n  /**\n   * Gets and returns the current number of members in this server's server set.\n   *\n   * @return number of members currently in this host's server set.\n   * @throws InterruptedException\n   * @throws ZooKeeperClient.ZooKeeperConnectionException\n   * @throws KeeperException\n   */\n  int getNumberOfServerSetMembers() throws InterruptedException,\n      ZooKeeperClient.ZooKeeperConnectionException, KeeperException;\n\n  /**\n   * Checks if this earlybird is in the server set.\n   *\n   * @return true if it is, false otherwise.\n   */\n  boolean isInServerSet();\n\n  /**\n   * Should only be called for Archive Earlybirds.\n   *\n   * Join ServerSet for ServiceProxy with a named admin port and with a zookeeper path that Service\n   * Proxy can translate to a domain name label that is less than 64 characters (due to the size\n   * limit for domain name labels described here: https://tools.ietf.org/html/rfc1035)\n   * This will allow us to access Earlybirds that are not on mesos via ServiceProxy.\n   */\n  void joinServerSetForServiceProxy();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/UpdateableEarlybirdStateManager.java",
    "content": "package com.twitter.search.earlybird;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.util.Random;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.atomic.AtomicLong;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Charsets;\n\nimport org.apache.thrift.TException;\nimport org.apache.zookeeper.KeeperException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.common.zookeeper.ZooKeeperClient;\nimport com.twitter.search.common.aurora.AuroraSchedulerClient;\nimport com.twitter.search.common.concurrent.ScheduledExecutorServiceFactory;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.file.LocalFile;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.schema.AnalyzerFactory;\nimport com.twitter.search.common.schema.DynamicSchema;\nimport com.twitter.search.common.schema.ImmutableSchema;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.thriftjava.ThriftSchema;\nimport com.twitter.search.common.util.ml.tensorflow_engine.TensorflowModelsManager;\nimport com.twitter.search.common.util.thrift.ThriftUtils;\nimport com.twitter.search.common.util.zookeeper.ZooKeeperProxy;\nimport com.twitter.search.earlybird.common.NonPagingAssert;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.ml.ScoringModelsManager;\nimport com.twitter.search.earlybird.partition.DynamicPartitionConfig;\nimport com.twitter.search.earlybird.partition.PartitionConfig;\nimport com.twitter.search.earlybird.partition.PartitionConfigLoader;\nimport com.twitter.search.earlybird.partition.PartitionConfigLoadingException;\nimport com.twitter.search.earlybird.util.OneTaskScheduledExecutorManager;\nimport com.twitter.search.earlybird.util.PeriodicActionParams;\nimport com.twitter.search.earlybird.util.ShutdownWaitTimeParams;\n\n/**\n * A class that keeps track of Earlybird state that may change while an Earlybird runs, and keeps\n * that state up to date. Currently keeps track of the current Earlybird schema and partition\n * configuration, and periodically updates them from Zookeeper. It also reloads periodically the\n * scoring models from HDFS.\n */\npublic class UpdateableEarlybirdStateManager extends OneTaskScheduledExecutorManager {\n  private static final Logger LOG = LoggerFactory.getLogger(UpdateableEarlybirdStateManager.class);\n  public static final String SCHEMA_SUFFIX = \".schema.v\";\n\n  private static final String THREAD_NAME_PATTERN = \"state_update-%d\";\n  private static final boolean THREAD_IS_DAEMON = true;\n  private static final long EXECUTOR_SHUTDOWN_WAIT_SEC = 5;\n\n  private static final String DEFAULT_ZK_SCHEMA_LOCATION =\n      \"/twitter/search/production/earlybird/schema\";\n  private static final String DEFAULT_LOCAL_SCHEMA_LOCATION =\n      \"/home/search/earlybird_schema_canary\";\n  private static final long DEFAULT_UPDATE_PERIOD_MILLIS =\n      TimeUnit.MINUTES.toMillis(30);\n\n  private static final String SCHEMA_MAJOR_VERSION_NAME =\n      \"schema_major_version\";\n  private static final String SCHEMA_MINOR_VERSION_NAME =\n      \"schema_minor_version\";\n  private static final String LAST_SUCCESSFUL_SCHEMA_RELOAD_TIME_MILLIS_NAME =\n      \"last_successful_schema_reload_timestamp_millis\";\n  @VisibleForTesting\n  static final String FAIL_TO_LOAD_SCHEMA_COUNT_NAME =\n      \"fail_to_load_schema_count\";\n  @VisibleForTesting\n  static final String HOST_IS_CANARY_SCHEME = \"host_is_canary_schema\";\n  @VisibleForTesting\n  static final String DID_NOT_FIND_SCHEMA_COUNT_NAME =\n      \"did_not_find_schema_count\";\n  private static final String LAST_SUCCESSFUL_PARTITION_CONFIG_RELOAD_TIME_MILLIS_NAME =\n      \"last_successful_partition_config_reload_timestamp_millis\";\n  @VisibleForTesting\n  static final String FAIL_TO_LOAD_PARTITION_CONFIG_COUNT_NAME =\n      \"fail_to_load_partition_config_count\";\n  @VisibleForTesting\n  static final String HOST_IS_IN_LAYOUT_STAT_NAME = \"host_is_in_layout\";\n  private static final String NOT_IN_LAYOUT_SHUT_DOWN_ATTEMPTED_NAME =\n      \"not_in_layout_shut_down_attempted\";\n\n  private static final String SHUT_DOWN_EARLYBIRD_WHEN_NOT_IN_LAYOUT_DECIDER_KEY =\n      \"shut_down_earlybird_when_not_in_layout\";\n\n  private static final String NO_SHUTDOWN_WHEN_NOT_IN_LAYOUT_NAME =\n      \"no_shutdown_when_not_in_layout\";\n\n  private final SearchLongGauge schemaMajorVersion;\n  private final SearchLongGauge schemaMinorVersion;\n  private final SearchLongGauge lastSuccessfulSchemaReloadTimeMillis;\n  private final SearchCounter failToLoadSchemaCount;\n  private final SearchLongGauge hostIsCanarySchema;\n  private final SearchCounter didNotFindSchemaCount;\n  private final SearchLongGauge lastSuccessfulPartitionConfigReloadTimeMillis;\n  private final SearchCounter failToLoadPartitionConfigCount;\n  private final SearchLongGauge hostIsInLayout;\n  private final SearchCounter notInLayoutShutDownAttemptedCount;\n  private final SearchLongGauge noShutdownWhenNotInLayoutGauge;\n\n  private final EarlybirdIndexConfig indexConfig;\n  private final DynamicPartitionConfig partitionConfig;\n  private final String schemaLocationOnLocal;\n  private final String schemaLocationOnZK;\n  private final ZooKeeperProxy zkClient;\n  private final AuroraSchedulerClient schedulerClient;\n  private final ScoringModelsManager scoringModelsManager;\n  private final TensorflowModelsManager tensorflowModelsManager;\n  private final SearchDecider searchDecider;\n  private final AtomicLong noShutdownWhenNotInLayout;\n  private EarlybirdServer earlybirdServer;\n  private Clock clock;\n\n  public UpdateableEarlybirdStateManager(\n      EarlybirdIndexConfig indexConfig,\n      DynamicPartitionConfig partitionConfig,\n      ZooKeeperProxy zooKeeperClient,\n      @Nullable  AuroraSchedulerClient schedulerClient,\n      ScheduledExecutorServiceFactory executorServiceFactory,\n      ScoringModelsManager scoringModelsManager,\n      TensorflowModelsManager tensorflowModelsManager,\n      SearchStatsReceiver searchStatsReceiver,\n      SearchDecider searchDecider,\n      CriticalExceptionHandler criticalExceptionHandler,\n      Clock clock) {\n    this(\n        indexConfig,\n        partitionConfig,\n        DEFAULT_LOCAL_SCHEMA_LOCATION,\n        DEFAULT_ZK_SCHEMA_LOCATION,\n        DEFAULT_UPDATE_PERIOD_MILLIS,\n        zooKeeperClient,\n        schedulerClient,\n        executorServiceFactory,\n        scoringModelsManager,\n        tensorflowModelsManager,\n        searchStatsReceiver,\n        searchDecider,\n        criticalExceptionHandler,\n        clock);\n  }\n\n  protected UpdateableEarlybirdStateManager(\n      EarlybirdIndexConfig indexConfig,\n      DynamicPartitionConfig partitionConfig,\n      String schemaLocationOnLocal,\n      String schemaLocationOnZK,\n      long updatePeriodMillis,\n      ZooKeeperProxy zkClient,\n      @Nullable  AuroraSchedulerClient schedulerClient,\n      ScheduledExecutorServiceFactory executorServiceFactory,\n      ScoringModelsManager scoringModelsManager,\n      TensorflowModelsManager tensorflowModelsManager,\n      SearchStatsReceiver searchStatsReceiver,\n      SearchDecider searchDecider,\n      CriticalExceptionHandler criticalExceptionHandler,\n      Clock clock) {\n    super(\n        executorServiceFactory,\n        THREAD_NAME_PATTERN,\n        THREAD_IS_DAEMON,\n        PeriodicActionParams.withFixedDelay(\n          updatePeriodMillis,\n          TimeUnit.MILLISECONDS\n        ),\n        new ShutdownWaitTimeParams(\n          EXECUTOR_SHUTDOWN_WAIT_SEC,\n          TimeUnit.SECONDS\n        ),\n        searchStatsReceiver,\n        criticalExceptionHandler);\n    this.indexConfig = indexConfig;\n    this.partitionConfig = partitionConfig;\n    this.schemaLocationOnLocal = schemaLocationOnLocal;\n    this.schemaLocationOnZK = schemaLocationOnZK;\n    this.zkClient = zkClient;\n    this.schedulerClient = schedulerClient;\n    this.scoringModelsManager = scoringModelsManager;\n    this.searchDecider = searchDecider;\n    this.noShutdownWhenNotInLayout = new AtomicLong(0);\n    this.tensorflowModelsManager = tensorflowModelsManager;\n    this.clock = clock;\n    this.schemaMajorVersion = getSearchStatsReceiver().getLongGauge(\n        SCHEMA_MAJOR_VERSION_NAME);\n    this.schemaMinorVersion = getSearchStatsReceiver().getLongGauge(\n        SCHEMA_MINOR_VERSION_NAME);\n    this.lastSuccessfulSchemaReloadTimeMillis = getSearchStatsReceiver().getLongGauge(\n        LAST_SUCCESSFUL_SCHEMA_RELOAD_TIME_MILLIS_NAME);\n    this.failToLoadSchemaCount = getSearchStatsReceiver().getCounter(\n        FAIL_TO_LOAD_SCHEMA_COUNT_NAME);\n    this.hostIsCanarySchema = getSearchStatsReceiver().getLongGauge(HOST_IS_CANARY_SCHEME);\n    this.didNotFindSchemaCount = getSearchStatsReceiver().getCounter(\n        DID_NOT_FIND_SCHEMA_COUNT_NAME);\n    this.lastSuccessfulPartitionConfigReloadTimeMillis = getSearchStatsReceiver().getLongGauge(\n        LAST_SUCCESSFUL_PARTITION_CONFIG_RELOAD_TIME_MILLIS_NAME);\n    this.failToLoadPartitionConfigCount = getSearchStatsReceiver().getCounter(\n        FAIL_TO_LOAD_PARTITION_CONFIG_COUNT_NAME);\n    this.hostIsInLayout = getSearchStatsReceiver().getLongGauge(\n        HOST_IS_IN_LAYOUT_STAT_NAME);\n    this.notInLayoutShutDownAttemptedCount = getSearchStatsReceiver().getCounter(\n        NOT_IN_LAYOUT_SHUT_DOWN_ATTEMPTED_NAME);\n    this.noShutdownWhenNotInLayoutGauge = getSearchStatsReceiver().getLongGauge(\n        NO_SHUTDOWN_WHEN_NOT_IN_LAYOUT_NAME, noShutdownWhenNotInLayout);\n\n    updateSchemaVersionStats(indexConfig.getSchema());\n  }\n\n  private void updateSchemaVersionStats(Schema schema) {\n    schemaMajorVersion.set(schema.getMajorVersionNumber());\n    schemaMinorVersion.set(schema.getMinorVersionNumber());\n    lastSuccessfulSchemaReloadTimeMillis.set(System.currentTimeMillis());\n    lastSuccessfulPartitionConfigReloadTimeMillis.set(System.currentTimeMillis());\n    hostIsInLayout.set(1);\n  }\n\n  private void updateSchemaVersionWithThriftSchema(ThriftSchema thriftSchema)\n      throws Schema.SchemaValidationException, DynamicSchema.SchemaUpdateException {\n\n      ImmutableSchema newSchema = new ImmutableSchema(\n          thriftSchema, new AnalyzerFactory(), indexConfig.getCluster().getNameForStats());\n      indexConfig.getSchema().updateSchema(newSchema);\n      tensorflowModelsManager.updateFeatureSchemaIdToMlIdMap(newSchema.getSearchFeatureSchema());\n      updateSchemaVersionStats(indexConfig.getSchema());\n      LOG.info(\"Schema updated. New Schema is: \\n\" + ThriftUtils.toTextFormatSafe(thriftSchema));\n  }\n\n  protected void updateSchema(ZooKeeperProxy zkClientToUse) {\n    // There are 3 cases:\n    // 1. Try to locate local schema file to canary, it might fail either because file not exist or\n    // ineligible versions.\n    // 2. Canary local schema failed, lookup schema file from zookeeper.\n    // 3. Both local and zookeeper updates failed, we do not update schema. Either schema not exists\n    // in zookeeper, or this would happened after canary schema: we updated current schema but did\n    // not rollback after finished.\n    if (updateSchemaFromLocal()) {\n      LOG.info(\"Host is used for schema canary\");\n      hostIsCanarySchema.set(1);\n    } else if (updateSchemaFromZooKeeper(zkClientToUse)) {\n      // Host is using schema file from zookeeper\n      hostIsCanarySchema.set(0);\n    } else {\n      // Schema update failed. Please check schema file exists on zookeeper and make sure\n      // rollback after canary. Current version: {}.{}\n      return;\n    }\n  }\n\n  private boolean updateSchemaFromLocal() {\n    ThriftSchema thriftSchema =\n        loadCanaryThriftSchemaFromLocal(getCanarySchemaFileOnLocal());\n    if (thriftSchema == null) {\n      // It is expected to not find a local schema file. The schema file only exists when the host\n      // is used as canary for schema updates\n      return false;\n    }\n    return updateSchemaFromThriftSchema(thriftSchema);\n  }\n\n  private boolean updateSchemaFromZooKeeper(ZooKeeperProxy zkClientToUse) {\n    ThriftSchema thriftSchema = loadThriftSchemaFromZooKeeper(zkClientToUse);\n    if (thriftSchema == null) {\n      // It is expected to usually not find a schema file on ZooKeeper; one is only uploaded if the\n      // schema changes after the package has been compiled. All the relevant error handling and\n      // logging is expected to be handled by loadThriftSchemaFromZooKeeper().\n      failToLoadSchemaCount.increment();\n      return false;\n    }\n    return updateSchemaFromThriftSchema(thriftSchema);\n  }\n\n  private boolean updateSchemaFromThriftSchema(ThriftSchema thriftSchema) {\n    Schema currentSchema = indexConfig.getSchema();\n    if (thriftSchema.getMajorVersionNumber() != currentSchema.getMajorVersionNumber()) {\n      LOG.warn(\n          \"Major version updates are not allowed. Current major version {}, try to update to {}\",\n          currentSchema.getMajorVersionNumber(), thriftSchema.getMajorVersionNumber());\n      return false;\n    }\n    if (thriftSchema.getMinorVersionNumber() > currentSchema.getMinorVersionNumber()) {\n      try {\n        updateSchemaVersionWithThriftSchema(thriftSchema);\n      } catch (Schema.SchemaValidationException | DynamicSchema.SchemaUpdateException e) {\n        LOG.warn(\"Exception while updating schema: \", e);\n        return false;\n      }\n      return true;\n    } else if (thriftSchema.getMinorVersionNumber() == currentSchema.getMinorVersionNumber()) {\n      LOG.info(\"Schema version to update is same as current one: {}.{}\",\n          currentSchema.getMajorVersionNumber(), currentSchema.getMinorVersionNumber());\n      return true;\n    } else {\n      LOG.info(\"Found schema to update, but not eligible for dynamic update. \"\n              + \"Current Version: {}.{};  Schema Version for updates: {}.{}\",\n          currentSchema.getMajorVersionNumber(),\n          currentSchema.getMinorVersionNumber(),\n          thriftSchema.getMajorVersionNumber(),\n          thriftSchema.getMinorVersionNumber());\n      return false;\n    }\n  }\n\n  void updatePartitionConfig(@Nullable AuroraSchedulerClient schedulerClientToUse) {\n    try {\n      if (schedulerClientToUse == null) {\n        NonPagingAssert.assertFailed(\"aurora_scheduler_client_is_null\");\n        throw new PartitionConfigLoadingException(\"AuroraSchedulerClient can not be null.\");\n      }\n\n      PartitionConfig newPartitionConfig =\n          PartitionConfigLoader.getPartitionInfoForMesosConfig(schedulerClientToUse);\n      partitionConfig.setCurrentPartitionConfig(newPartitionConfig);\n      lastSuccessfulPartitionConfigReloadTimeMillis.set(System.currentTimeMillis());\n      hostIsInLayout.set(1);\n    } catch (PartitionConfigLoadingException e) {\n      // Do not change hostIsInLayout's value if we could not load the layout.\n      LOG.warn(\"Failed to load partition config from ZooKeeper.\", e);\n      failToLoadPartitionConfigCount.increment();\n    }\n  }\n\n  @Nullable\n  private ThriftSchema loadCanaryThriftSchemaFromLocal(LocalFile schemaFile) {\n    String schemaString;\n    if (!schemaFile.getFile().exists()) {\n      return null;\n    }\n    try {\n      schemaString = schemaFile.getCharSource().read();\n    } catch (IOException e) {\n      LOG.warn(\"Fail to read from local schema file.\");\n      return null;\n    }\n    ThriftSchema thriftSchema = new ThriftSchema();\n    try {\n      ThriftUtils.fromTextFormat(schemaString, thriftSchema);\n      return thriftSchema;\n    } catch (TException e) {\n      LOG.warn(\"Unable to deserialize ThriftSchema loaded locally from {}.\\n{}\",\n          schemaFile.getName(), e);\n      return null;\n    }\n  }\n\n  @Nullable\n  private ThriftSchema loadThriftSchemaFromZooKeeper(ZooKeeperProxy zkClientToUse) {\n    String schemaPathOnZk = getFullSchemaPathOnZK();\n    byte[] rawBytes;\n    try {\n      rawBytes = zkClientToUse.getData(schemaPathOnZk, false, null);\n    } catch (KeeperException.NoNodeException e) {\n      didNotFindSchemaCount.increment();\n      return null;\n    } catch (KeeperException e) {\n      LOG.warn(\"Exception while loading schema from ZK at {}.\\n{}\", schemaPathOnZk, e);\n      return null;\n    } catch (InterruptedException e) {\n      Thread.currentThread().interrupt();\n      LOG.warn(\"Interrupted while loading schema from ZK at {}.\\n{}\", schemaPathOnZk, e);\n      return null;\n    } catch (ZooKeeperClient.ZooKeeperConnectionException e) {\n      LOG.warn(\"Exception while loading schema from ZK at {}.\\n{}\", schemaPathOnZk, e);\n      return null;\n    }\n    if (rawBytes == null) {\n      LOG.warn(\"Got null schema from ZooKeeper at {}.\", schemaPathOnZk);\n      return null;\n    }\n    String schemaString = new String(rawBytes, Charsets.UTF_8);\n    ThriftSchema thriftSchema = new ThriftSchema();\n    try {\n      ThriftUtils.fromTextFormat(schemaString, thriftSchema);\n      return thriftSchema;\n    } catch (TException e) {\n      LOG.warn(\"Unable to deserialize ThriftSchema loaded from ZK at {}.\\n{}\", schemaPathOnZk, e);\n      return null;\n    }\n  }\n\n  @VisibleForTesting\n  protected String getSchemaFileName() {\n    return indexConfig.getCluster().name().toLowerCase()\n        + UpdateableEarlybirdStateManager.SCHEMA_SUFFIX\n        + indexConfig.getSchema().getMajorVersionNumber();\n  }\n\n  @VisibleForTesting\n  protected String getFullSchemaPathOnZK() {\n    return String.format(\"%s/%s\", schemaLocationOnZK, getSchemaFileName());\n  }\n\n  LocalFile getCanarySchemaFileOnLocal() {\n    String canarySchemaFilePath =\n        String.format(\"%s/%s\", schemaLocationOnLocal, getSchemaFileName());\n    return new LocalFile(new File(canarySchemaFilePath));\n  }\n\n  void setNoShutdownWhenNotInLayout(boolean noShutdown) {\n    noShutdownWhenNotInLayout.set(noShutdown ? 1 : 0);\n  }\n\n  @Override\n  protected void runOneIteration() {\n    updateSchema(zkClient);\n    updatePartitionConfig(schedulerClient);\n\n    LOG.info(\"Reloading models.\");\n    scoringModelsManager.reload();\n    tensorflowModelsManager.run();\n\n    Random random = new Random();\n\n    try {\n      // We had an issue where HDFS operations were blocking, so reloading these models\n      // was finishing at the same time on each instance and after that every time an instance\n      // was reloading models, it was happening at the same time. This caused issues with HDFS\n      // load. We now place a \"guard\" waiting time after each reload so that the execution time\n      // on every instance is different and these calls can't easily sync to the same point in time.\n      int sleepSeconds = random.nextInt(30 * 60);\n      LOG.info(\"Sleeping for {} seconds\", sleepSeconds);\n      clock.waitFor(sleepSeconds * 1000);\n    } catch (InterruptedException ex) {\n      LOG.info(\"Interrupted while sleeping\");\n    }\n  }\n\n  public void setEarlybirdServer(EarlybirdServer earlybirdServer) {\n    this.earlybirdServer = earlybirdServer;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/ArchiveEarlybirdIndexConfig.java",
    "content": "package com.twitter.search.earlybird.archive;\n\nimport java.io.IOException;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.IndexWriterConfig;\nimport org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy;\nimport org.apache.lucene.index.LogByteSizeMergePolicy;\nimport org.apache.lucene.index.SerialMergeScheduler;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.schema.SearchWhitespaceAnalyzer;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.util.CloseResourceUtil;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentData;\nimport com.twitter.search.core.earlybird.index.EarlybirdLuceneIndexSegmentData;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\n\n/**\n * Base config for the top archive tweet clusters.\n */\npublic abstract class ArchiveEarlybirdIndexConfig extends EarlybirdIndexConfig {\n\n  private final CloseResourceUtil resourceCloser = new CloseResourceUtil();\n\n  public ArchiveEarlybirdIndexConfig(\n      EarlybirdCluster cluster, Decider decider, SearchIndexingMetricSet searchIndexingMetricSet,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    super(cluster, decider, searchIndexingMetricSet, criticalExceptionHandler);\n  }\n\n  @Override\n  public IndexWriterConfig newIndexWriterConfig() {\n    return new IndexWriterConfig(new SearchWhitespaceAnalyzer())\n        .setIndexDeletionPolicy(new KeepOnlyLastCommitDeletionPolicy())\n        .setMergeScheduler(new SerialMergeScheduler())\n        .setMergePolicy(new LogByteSizeMergePolicy())\n        .setRAMBufferSizeMB(IndexWriterConfig.DEFAULT_RAM_PER_THREAD_HARD_LIMIT_MB)\n        .setMaxBufferedDocs(IndexWriterConfig.DISABLE_AUTO_FLUSH)\n        .setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);\n  }\n\n  @Override\n  public CloseResourceUtil getResourceCloser() {\n    return resourceCloser;\n  }\n\n  @Override\n  public EarlybirdIndexSegmentData optimize(\n      EarlybirdIndexSegmentData segmentData) throws IOException {\n    Preconditions.checkArgument(\n        segmentData instanceof EarlybirdLuceneIndexSegmentData,\n        \"Expected EarlybirdLuceneIndexSegmentData but got %s\",\n        segmentData.getClass());\n    EarlybirdLuceneIndexSegmentData data = (EarlybirdLuceneIndexSegmentData) segmentData;\n\n    return new EarlybirdLuceneIndexSegmentData(\n        data.getLuceneDirectory(),\n        data.getMaxSegmentSize(),\n        data.getTimeSliceID(),\n        data.getSchema(),\n        true, // isOptimized\n        data.getSyncData().getSmallestDocID(),\n        new ConcurrentHashMap<>(data.getPerFieldMap()),\n        data.getFacetCountingArray(),\n        data.getDocValuesManager(),\n        data.getDocIDToTweetIDMapper(),\n        data.getTimeMapper(),\n        data.getIndexExtensionsData());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/ArchiveHDFSUtils.java",
    "content": "package com.twitter.search.earlybird.archive;\n\nimport java.io.IOException;\nimport java.util.Calendar;\nimport java.util.Date;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\n\nimport org.apache.commons.io.IOUtils;\nimport org.apache.hadoop.fs.FileStatus;\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.Path;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.partitioning.base.Segment;\nimport com.twitter.search.earlybird.partition.HdfsUtil;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\n\n\npublic final class ArchiveHDFSUtils {\n  private static final Logger LOG = LoggerFactory.getLogger(ArchiveHDFSUtils.class);\n\n  private static final Pattern SEGMENT_NAME_PATTERN =\n      Pattern.compile(\"_start_([0-9]+)_p_([0-9]+)_of_([0-9]+)_([0-9]{14}+)_\");\n  private static final int MATCHER_GROUP_END_DATE = 4;\n\n  private ArchiveHDFSUtils() {\n  }\n\n  /**\n   * Check if a given segment already has its indices built on hdfs.\n   * @return true if the indices exist on hdfs; otherwise, false.\n   */\n  public static boolean hasSegmentIndicesOnHDFS(SegmentSyncConfig sync, SegmentInfo segment) {\n    LOG.info(\"checking segment on hdfs: \" + segment\n        + \" enabled: \" + sync.isSegmentLoadFromHdfsEnabled());\n    FileSystem fs = null;\n    try {\n      fs = HdfsUtil.getHdfsFileSystem();\n      String hdfsBaseDirPrefix = segment.getSyncInfo()\n          .getHdfsSyncDirPrefix();\n      FileStatus[] statuses = fs.globStatus(new Path(hdfsBaseDirPrefix));\n      return statuses != null && statuses.length > 0;\n    } catch (IOException ex) {\n      LOG.error(\"Failed checking segment on hdfs: \" + segment, ex);\n      return false;\n    } finally {\n      IOUtils.closeQuietly(fs);\n    }\n  }\n\n  /**\n   * Delete the segment index directories on the HDFS. If 'deleteCurrentDir' is true, the\n   * index directory with the end date matching 'segment' will be deleted. If 'deleteOlderDirs',\n   * the index directories with the end date earlier than the the segment enddate will be deleted.\n   *\n   */\n  public static void deleteHdfsSegmentDir(SegmentSyncConfig sync, SegmentInfo segment,\n                                          boolean deleteCurrentDir, boolean deleteOlderDirs) {\n    FileSystem fs = null;\n    try {\n      fs = HdfsUtil.getHdfsFileSystem();\n      String hdfsFlushDir = segment.getSyncInfo().getHdfsFlushDir();\n      String hdfsBaseDirPrefix = segment.getSyncInfo()\n          .getHdfsSyncDirPrefix();\n      String endDateStr = extractEndDate(hdfsBaseDirPrefix);\n      if (endDateStr != null) {\n        hdfsBaseDirPrefix = hdfsBaseDirPrefix.replace(endDateStr, \"*\");\n      }\n      String[] hdfsDirs = {segment.getSyncInfo().getHdfsTempFlushDir(),\n          hdfsBaseDirPrefix};\n      for (String hdfsDir : hdfsDirs) {\n        FileStatus[] statuses = fs.globStatus(new Path(hdfsDir));\n        if (statuses != null && statuses.length > 0) {\n          for (FileStatus status : statuses) {\n            if (status.getPath().toString().endsWith(hdfsFlushDir)) {\n              if (deleteCurrentDir) {\n                fs.delete(status.getPath(), true);\n                LOG.info(\"Deleted segment: \" + status.getPath());\n              }\n            } else {\n              if (deleteOlderDirs) {\n                fs.delete(status.getPath(), true);\n                LOG.info(\"Deleted segment: \" + status.getPath());\n              }\n            }\n          }\n        }\n      }\n    } catch (IOException e) {\n      LOG.error(\"Error delete Segment Dir :\" + segment, e);\n    } finally {\n      IOUtils.closeQuietly(fs);\n    }\n  }\n\n  /**\n   * Given a segment, check if there is any indices built on HDFS; if yes, return the end date\n   * of the index built on HDFS; otherwise, return null.\n   */\n  public static Date getSegmentEndDateOnHdfs(SegmentSyncConfig sync, SegmentInfo segment) {\n    if (sync.isSegmentLoadFromHdfsEnabled()) {\n      LOG.info(\"About to check segment on hdfs: \" + segment\n          + \" enabled: \" + sync.isSegmentLoadFromHdfsEnabled());\n\n      FileSystem fs = null;\n      try {\n        String hdfsBaseDirPrefix = segment.getSyncInfo()\n            .getHdfsSyncDirPrefix();\n        String endDateStr = extractEndDate(hdfsBaseDirPrefix);\n        if (endDateStr == null) {\n          return null;\n        }\n        hdfsBaseDirPrefix = hdfsBaseDirPrefix.replace(endDateStr, \"*\");\n\n        fs = HdfsUtil.getHdfsFileSystem();\n        FileStatus[] statuses = fs.globStatus(new Path(hdfsBaseDirPrefix));\n        if (statuses != null && statuses.length > 0) {\n          Path hdfsSyncPath = statuses[statuses.length - 1].getPath();\n          String hdfsSyncPathName = hdfsSyncPath.getName();\n          endDateStr = extractEndDate(hdfsSyncPathName);\n          return Segment.getSegmentEndDate(endDateStr);\n        }\n      } catch (Exception ex) {\n        LOG.error(\"Failed getting segment from hdfs: \" + segment, ex);\n        return null;\n      } finally {\n        IOUtils.closeQuietly(fs);\n      }\n    }\n    return null;\n  }\n\n  private static String extractEndDate(String segmentDirPattern) {\n    Matcher matcher = SEGMENT_NAME_PATTERN.matcher(segmentDirPattern);\n    if (!matcher.find()) {\n      return null;\n    }\n\n    try {\n      return matcher.group(MATCHER_GROUP_END_DATE);\n    } catch (IllegalStateException e) {\n      LOG.error(\"Match operation failed: \" + segmentDirPattern, e);\n      return null;\n    } catch (IndexOutOfBoundsException e) {\n      LOG.error(\" No group in the pattern with the given index : \" + segmentDirPattern, e);\n      return null;\n    }\n  }\n\n  /**\n   * Converts the given date to a path, using the given separator. For example, if the sate is\n   * January 5, 2019, and the separator is \"/\", this method will return \"2019/01/05\".\n   */\n  public static String dateToPath(Date date, String separator) {\n    StringBuilder builder = new StringBuilder();\n    Calendar cal = Calendar.getInstance();\n    cal.setTime(date);\n    builder.append(cal.get(Calendar.YEAR))\n           .append(separator)\n           .append(padding(cal.get(Calendar.MONTH) + 1, 2))\n           .append(separator)\n           .append(padding(cal.get(Calendar.DAY_OF_MONTH), 2));\n    return builder.toString();\n  }\n\n  private static String padding(int value, int len) {\n    return String.format(\"%0\" + len + \"d\", value);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/ArchiveOnDiskEarlybirdIndexConfig.java",
    "content": "package com.twitter.search.earlybird.archive;\n\nimport java.io.File;\nimport java.io.IOException;\n\nimport org.apache.lucene.store.Directory;\nimport org.apache.lucene.store.FSDirectory;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentData;\nimport com.twitter.search.core.earlybird.index.EarlybirdLuceneIndexSegmentData;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdIndexExtensionsFactory;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.index.DocValuesBasedTimeMapper;\nimport com.twitter.search.earlybird.index.DocValuesBasedTweetIDMapper;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\nimport com.twitter.search.earlybird.partition.SegmentSyncInfo;\n\n/**\n * Index config for the on-disk Tweet clusters.\n */\npublic class ArchiveOnDiskEarlybirdIndexConfig extends ArchiveEarlybirdIndexConfig {\n  public ArchiveOnDiskEarlybirdIndexConfig(\n      Decider decider, SearchIndexingMetricSet searchIndexingMetricSet,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    super(EarlybirdCluster.FULL_ARCHIVE, decider, searchIndexingMetricSet,\n        criticalExceptionHandler);\n  }\n\n  @Override\n  public boolean isIndexStoredOnDisk() {\n    return true;\n  }\n\n  @Override\n  public Directory newLuceneDirectory(SegmentSyncInfo segmentSyncInfo) throws IOException {\n    File dirPath = new File(segmentSyncInfo.getLocalLuceneSyncDir());\n    return FSDirectory.open(dirPath.toPath());\n  }\n\n  @Override\n  public EarlybirdIndexSegmentData newSegmentData(\n      int maxSegmentSize,\n      long timeSliceID,\n      Directory dir,\n      EarlybirdIndexExtensionsFactory extensionsFactory) {\n    return new EarlybirdLuceneIndexSegmentData(\n        dir,\n        maxSegmentSize,\n        timeSliceID,\n        getSchema(),\n        new DocValuesBasedTweetIDMapper(),\n        new DocValuesBasedTimeMapper(),\n        extensionsFactory);\n  }\n\n  @Override\n  public EarlybirdIndexSegmentData loadSegmentData(\n      FlushInfo flushInfo,\n      DataDeserializer dataInputStream,\n      Directory dir,\n      EarlybirdIndexExtensionsFactory extensionsFactory) throws IOException {\n    // IO Exception will be thrown if there's an error during load\n    return (new EarlybirdLuceneIndexSegmentData.OnDiskSegmentDataFlushHandler(\n        getSchema(),\n        dir,\n        extensionsFactory,\n        new DocValuesBasedTweetIDMapper.FlushHandler(),\n        new DocValuesBasedTimeMapper.FlushHandler())).load(flushInfo, dataInputStream);\n  }\n\n  @Override\n  public boolean supportOutOfOrderIndexing() {\n    return false;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/ArchiveSearchPartitionManager.java",
    "content": "package com.twitter.search.earlybird.archive;\n\nimport java.io.IOException;\nimport java.util.Date;\nimport java.util.List;\nimport java.util.concurrent.TimeUnit;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Predicate;\nimport com.google.common.collect.Lists;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.concurrent.ScheduledExecutorServiceFactory;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.util.GCUtil;\nimport com.twitter.search.common.util.io.recordreader.RecordReader;\nimport com.twitter.search.common.util.zktrylock.ZooKeeperTryLockFactory;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.EarlybirdStatus;\nimport com.twitter.search.earlybird.ServerSetMember;\nimport com.twitter.search.earlybird.archive.ArchiveTimeSlicer.ArchiveTimeSlice;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.util.ScrubGenUtil;\nimport com.twitter.search.earlybird.document.TweetDocument;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.partition.CompleteSegmentManager;\nimport com.twitter.search.earlybird.partition.DynamicPartitionConfig;\nimport com.twitter.search.earlybird.partition.MultiSegmentTermDictionaryManager;\nimport com.twitter.search.earlybird.partition.PartitionConfig;\nimport com.twitter.search.earlybird.partition.PartitionManager;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\nimport com.twitter.search.earlybird.partition.SegmentHdfsFlusher;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentLoader;\nimport com.twitter.search.earlybird.partition.SegmentManager;\nimport com.twitter.search.earlybird.partition.SegmentManager.Filter;\nimport com.twitter.search.earlybird.partition.SegmentManager.Order;\nimport com.twitter.search.earlybird.partition.SegmentOptimizer;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\nimport com.twitter.search.earlybird.partition.SegmentWarmer;\nimport com.twitter.search.earlybird.partition.SimpleSegmentIndexer;\nimport com.twitter.search.earlybird.partition.UserScrubGeoEventStreamIndexer;\nimport com.twitter.search.earlybird.partition.UserUpdatesStreamIndexer;\nimport com.twitter.search.earlybird.querycache.QueryCacheManager;\nimport com.twitter.search.earlybird.segment.SegmentDataProvider;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusCode;\nimport com.twitter.search.earlybird.util.CoordinatedEarlybirdAction;\nimport com.twitter.search.earlybird.util.CoordinatedEarlybirdActionInterface;\nimport com.twitter.search.earlybird.util.CoordinatedEarlybirdActionLockFailed;\n\npublic class ArchiveSearchPartitionManager extends PartitionManager {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(ArchiveSearchPartitionManager.class);\n\n  public static final String CONFIG_NAME = \"archive\";\n\n  private static final long ONE_DAY_MILLIS = TimeUnit.DAYS.toMillis(1);\n\n  private final ArchiveTimeSlicer timeSlicer;\n  private final ArchiveSegmentDataProvider segmentDataProvider;\n\n  private final UserUpdatesStreamIndexer userUpdatesStreamIndexer;\n  private final UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer;\n\n  private final SegmentWarmer segmentWarmer;\n  private final EarlybirdIndexConfig earlybirdIndexConfig;\n  private final ZooKeeperTryLockFactory zkTryLockFactory;\n  private final Clock clock;\n  private final SegmentSyncConfig segmentSyncConfig;\n  protected final SearchCounter gcAfterIndexing;\n\n  // Used for coordinating daily updated across different replicas on the same hash partition,\n  // to run them one at a time, and minimize the impact on query latencies.\n  private final CoordinatedEarlybirdActionInterface coordinatedDailyUpdate;\n\n  private final SearchIndexingMetricSet indexingMetricSet;\n\n  // This is only used in tests where no coordination is needed.\n  @VisibleForTesting\n  public ArchiveSearchPartitionManager(\n      ZooKeeperTryLockFactory zooKeeperTryLockFactory,\n      QueryCacheManager queryCacheManager,\n      SegmentManager segmentManager,\n      DynamicPartitionConfig dynamicPartitionConfig,\n      UserUpdatesStreamIndexer userUpdatesStreamIndexer,\n      UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer,\n      SearchStatsReceiver searchStatsReceiver,\n      ArchiveEarlybirdIndexConfig earlybirdIndexConfig,\n      ScheduledExecutorServiceFactory executorServiceFactory,\n      ScheduledExecutorServiceFactory userUpdateIndexerScheduledExecutorFactory,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      SegmentSyncConfig syncConfig,\n      Clock clock,\n      CriticalExceptionHandler criticalExceptionHandler)\n      throws IOException {\n    this(\n        zooKeeperTryLockFactory,\n        queryCacheManager,\n        segmentManager,\n        dynamicPartitionConfig,\n        userUpdatesStreamIndexer,\n        userScrubGeoEventStreamIndexer,\n        searchStatsReceiver,\n        earlybirdIndexConfig,\n        null,\n        executorServiceFactory,\n        userUpdateIndexerScheduledExecutorFactory,\n        searchIndexingMetricSet,\n        syncConfig,\n        clock,\n        criticalExceptionHandler);\n  }\n\n  public ArchiveSearchPartitionManager(\n      ZooKeeperTryLockFactory zooKeeperTryLockFactory,\n      QueryCacheManager queryCacheManager,\n      SegmentManager segmentManager,\n      DynamicPartitionConfig dynamicPartitionConfig,\n      UserUpdatesStreamIndexer userUpdatesStreamIndexer,\n      UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer,\n      SearchStatsReceiver searchStatsReceiver,\n      ArchiveEarlybirdIndexConfig earlybirdIndexConfig,\n      ServerSetMember serverSetMember,\n      ScheduledExecutorServiceFactory executorServiceFactory,\n      ScheduledExecutorServiceFactory userUpdateIndexerExecutorFactory,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      SegmentSyncConfig syncConfig,\n      Clock clock,\n      CriticalExceptionHandler criticalExceptionHandler) throws IOException {\n    super(queryCacheManager, segmentManager, dynamicPartitionConfig, executorServiceFactory,\n        searchIndexingMetricSet, searchStatsReceiver, criticalExceptionHandler);\n\n    Preconditions.checkState(syncConfig.getScrubGen().isPresent());\n    Date scrubGen = ScrubGenUtil.parseScrubGenToDate(syncConfig.getScrubGen().get());\n\n    this.zkTryLockFactory = zooKeeperTryLockFactory;\n    final DailyStatusBatches dailyStatusBatches = new DailyStatusBatches(\n        zkTryLockFactory,\n        scrubGen);\n    this.earlybirdIndexConfig = earlybirdIndexConfig;\n    PartitionConfig curPartitionConfig = dynamicPartitionConfig.getCurrentPartitionConfig();\n\n    this.indexingMetricSet = searchIndexingMetricSet;\n\n    this.timeSlicer = new ArchiveTimeSlicer(\n        EarlybirdConfig.getMaxSegmentSize(), dailyStatusBatches,\n        curPartitionConfig.getTierStartDate(), curPartitionConfig.getTierEndDate(),\n        earlybirdIndexConfig);\n    this.segmentDataProvider =\n        new ArchiveSegmentDataProvider(\n            dynamicPartitionConfig,\n            timeSlicer,\n            this.earlybirdIndexConfig);\n\n    this.userUpdatesStreamIndexer = userUpdatesStreamIndexer;\n    this.userScrubGeoEventStreamIndexer = userScrubGeoEventStreamIndexer;\n\n    this.coordinatedDailyUpdate = new CoordinatedEarlybirdAction(\n        zkTryLockFactory,\n        \"archive_daily_update\",\n        dynamicPartitionConfig,\n        serverSetMember,\n        criticalExceptionHandler,\n        syncConfig);\n\n    this.segmentWarmer = new SegmentWarmer(criticalExceptionHandler);\n    this.clock = clock;\n    this.segmentSyncConfig = syncConfig;\n    this.gcAfterIndexing = SearchCounter.export(\"gc_after_indexing\");\n  }\n\n  @Override\n  public SegmentDataProvider getSegmentDataProvider() {\n    return segmentDataProvider;\n  }\n\n  @Override\n  protected void startUp() throws Exception {\n    LOG.info(\"Using CompleteSegmentManager to index complete segments.\");\n\n    // deferring handling of multi-segment term dictionary for the archive.\n    // SEARCH-11952\n    CompleteSegmentManager completeSegmentManager = new CompleteSegmentManager(\n        zkTryLockFactory,\n        segmentDataProvider,\n        userUpdatesStreamIndexer,\n        userScrubGeoEventStreamIndexer,\n        segmentManager,\n        null,\n        indexingMetricSet,\n        clock,\n        MultiSegmentTermDictionaryManager.NOOP_INSTANCE,\n        segmentSyncConfig,\n        criticalExceptionHandler);\n\n    completeSegmentManager.indexUserEvents();\n    completeSegmentManager.indexCompleteSegments(\n        () -> segmentManager.getSegmentInfos(Filter.NeedsIndexing, Order.OLD_TO_NEW));\n\n    // In the archive cluster, the current segment needs to be loaded too.\n    List<SegmentInfo> allSegments =\n        Lists.newArrayList(segmentManager.getSegmentInfos(Filter.All, Order.OLD_TO_NEW));\n    completeSegmentManager.loadCompleteSegments(allSegments);\n\n    completeSegmentManager.buildMultiSegmentTermDictionary();\n\n    completeSegmentManager.warmSegments(allSegments);\n\n    LOG.info(\"Starting to run UserUpdatesKafkaConsumer\");\n    new Thread(userUpdatesStreamIndexer::run, \"userupdates-stream-indexer\").start();\n\n    if (EarlybirdConfig.consumeUserScrubGeoEvents()) {\n      LOG.info(\"Starting to run UserScrubGeoEventKafkaConsumer\");\n      new Thread(userScrubGeoEventStreamIndexer::run,\n          \"userScrubGeoEvent-stream-indexer\").start();\n    }\n  }\n\n  private static List<ArchiveTimeSlice> truncateSegmentList(List<ArchiveTimeSlice> segmentList,\n                                                            int maxNumSegments) {\n    // Maybe cut-off the beginning of the sorted list of IDs.\n    if (maxNumSegments > 0 && maxNumSegments < segmentList.size()) {\n      return segmentList.subList(segmentList.size() - maxNumSegments, segmentList.size());\n    } else {\n      return segmentList;\n    }\n  }\n\n\n  @Override\n  protected void indexingLoop(boolean firstLoop) throws Exception {\n    if (firstLoop) {\n      EarlybirdStatus.beginEvent(\n          INDEX_CURRENT_SEGMENT, getSearchIndexingMetricSet().startupInCurrentSegment);\n    }\n\n    List<ArchiveTimeSlice> timeSlices = timeSlicer.getTimeSlicesInTierRange();\n    PartitionConfig curPartitionConfig = dynamicPartitionConfig.getCurrentPartitionConfig();\n    timeSlices = truncateSegmentList(timeSlices, curPartitionConfig.getMaxEnabledLocalSegments());\n\n    for (final ArchiveTimeSlice timeSlice : timeSlices) {\n      // If any timeslice build failed, do not try to build timeslice after that to prevent\n      // possible holes between timeslices.\n      try {\n        if (!processArchiveTimeSlice(timeSlice)) {\n          LOG.warn(\"Building timeslice {} has failed, stopping future builds.\",\n              timeSlice.getDescription());\n          indexingMetricSet.archiveTimeSliceBuildFailedCounter.increment();\n          return;\n        }\n      } catch (CoordinatedEarlybirdActionLockFailed e) {\n        // If the timeslice build failed because of lock coordination, we can wait for the next\n        // iteration to build again.\n        return;\n      }\n    }\n\n    if (firstLoop) {\n      EarlybirdStatus.endEvent(\n          INDEX_CURRENT_SEGMENT, getSearchIndexingMetricSet().startupInCurrentSegment);\n      LOG.info(\"First indexing loop complete. Setting up query cache...\");\n      EarlybirdStatus.beginEvent(\n          SETUP_QUERY_CACHE, getSearchIndexingMetricSet().startupInQueryCacheUpdates);\n    }\n    setupQueryCacheIfNeeded();\n\n    if (EarlybirdStatus.isStarting() && queryCacheManager.allTasksRan()) {\n      LOG.info(\"Query cache setup complete. Becoming current now...\");\n      EarlybirdStatus.endEvent(\n          SETUP_QUERY_CACHE, getSearchIndexingMetricSet().startupInQueryCacheUpdates);\n\n      becomeCurrent();\n      EarlybirdStatus.recordEarlybirdEvent(\"Archive Earlybird is current\");\n    }\n\n    updateIndexFreshnessStats(timeSlices);\n  }\n\n  @VisibleForTesting\n  protected boolean processArchiveTimeSlice(final ArchiveTimeSlice timeSlice)\n      throws CoordinatedEarlybirdActionLockFailed, IOException {\n    PartitionConfig curPartitionConfig = dynamicPartitionConfig.getCurrentPartitionConfig();\n    long minStatusID = timeSlice.getMinStatusID(curPartitionConfig.getIndexingHashPartitionID());\n    SegmentInfo segmentInfo = segmentManager.getSegmentInfo(minStatusID);\n    if (segmentInfo == null) {\n      return indexSegmentFromScratch(timeSlice);\n    } else if (existingSegmentNeedsUpdating(timeSlice, segmentInfo)) {\n      return indexNewDayAndAppendExistingSegment(timeSlice, segmentInfo);\n    }\n    return true;\n  }\n\n\n  @VisibleForTesting\n  SegmentInfo newSegmentInfo(ArchiveTimeSlice timeSlice) throws IOException {\n    return new SegmentInfo(segmentDataProvider.newArchiveSegment(timeSlice),\n        segmentManager.getEarlybirdSegmentFactory(), segmentSyncConfig);\n  }\n\n  private boolean indexNewDayAndAppendExistingSegment(final ArchiveTimeSlice timeSlice,\n                                                      SegmentInfo segmentInfo)\n      throws CoordinatedEarlybirdActionLockFailed, IOException {\n\n    LOG.info(\"Updating segment: {}; new endDate will be {} segmentInfo: {}\",\n        segmentInfo.getSegment().getTimeSliceID(), timeSlice.getEndDate(), segmentInfo);\n\n    // Create another new SegmentInfo for indexing\n    final SegmentInfo newSegmentInfoForIndexing = newSegmentInfo(timeSlice);\n    // make a final reference of the old segment info to be passed into closure.\n    final SegmentInfo oldSegmentInfo = segmentInfo;\n\n    // Sanity check: the old and new segment should not share the same lucene directory.\n    Preconditions.checkState(\n        !newSegmentInfoForIndexing.getSyncInfo().getLocalLuceneSyncDir().equals(\n            oldSegmentInfo.getSyncInfo().getLocalLuceneSyncDir()));\n\n    Preconditions.checkState(\n        !newSegmentInfoForIndexing.getSyncInfo().getLocalSyncDir().equals(\n            oldSegmentInfo.getSyncInfo().getLocalSyncDir()));\n\n    final ArchiveSegment oldSegment = (ArchiveSegment) segmentInfo.getSegment();\n\n    return indexSegment(newSegmentInfoForIndexing, oldSegmentInfo, input -> {\n      // we're updating the segment - only index days after the old end date, but only if\n      // we're in the on-disk archive, and we're sure that the previous days have already\n      // been indexed.\n      return !earlybirdIndexConfig.isIndexStoredOnDisk()\n          // First time around, and the segment has not been indexed and optimized yet,\n          // we will want to add all the days\n          || !oldSegmentInfo.isOptimized()\n          || oldSegmentInfo.getIndexSegment().getIndexStats().getStatusCount() == 0\n          || !oldSegment.getDataEndDate().before(timeSlice.getEndDate())\n          // Index any new days\n          || input.after(oldSegment.getDataEndDate());\n    });\n  }\n\n  private boolean existingSegmentNeedsUpdating(ArchiveTimeSlice timeSlice,\n                                               SegmentInfo segmentInfo) {\n    return ((ArchiveSegment) segmentInfo.getSegment())\n        .getDataEndDate().before(timeSlice.getEndDate())\n        // First time around, the end date is the same as the timeSlice end date, but\n        // the segment has not been indexed and optimized yet\n        || (!segmentInfo.isOptimized() && !segmentInfo.wasIndexed())\n        // If indexing failed, this index will not be marked as complete, and we will want\n        // to reindex\n        || !segmentInfo.isComplete();\n  }\n\n  private boolean indexSegmentFromScratch(ArchiveTimeSlice timeSlice) throws\n      CoordinatedEarlybirdActionLockFailed, IOException {\n\n    SegmentInfo segmentInfo = newSegmentInfo(timeSlice);\n    LOG.info(\"Creating segment: \" + segmentInfo.getSegment().getTimeSliceID()\n        + \"; new endDate will be \" + timeSlice.getEndDate() + \" segmentInfo: \" + segmentInfo);\n\n    return indexSegment(segmentInfo, null, ArchiveSegment.MATCH_ALL_DATE_PREDICATE);\n  }\n\n  private void updateIndexFreshnessStats(List<ArchiveTimeSlice> timeSlices) {\n    if (!timeSlices.isEmpty()) {\n      ArchiveTimeSlice lastTimeslice = timeSlices.get(timeSlices.size() - 1);\n\n      // Add ~24 hours to start of end date to estimate freshest tweet time.\n      indexingMetricSet.freshestTweetTimeMillis.set(\n          lastTimeslice.getEndDate().getTime() + ONE_DAY_MILLIS);\n\n      PartitionConfig curPartitionConfig = dynamicPartitionConfig.getCurrentPartitionConfig();\n      long maxStatusId = lastTimeslice.getMaxStatusID(\n          curPartitionConfig.getIndexingHashPartitionID());\n      if (maxStatusId > indexingMetricSet.highestStatusId.get()) {\n        indexingMetricSet.highestStatusId.set(maxStatusId);\n      }\n    }\n  }\n\n  @Override\n  public void shutDownIndexing() {\n    LOG.info(\"Shutting down.\");\n    userUpdatesStreamIndexer.close();\n    userScrubGeoEventStreamIndexer.close();\n    LOG.info(\"Closed User Event Kafka Consumers. Now Shutting down reader set.\");\n    getSegmentDataProvider().getSegmentDataReaderSet().stopAll();\n  }\n\n  /**\n   * Attempts to index new days of data into the provided segment, indexing only the days that\n   * match the \"dateFilter\" predicate.\n   * @return true iff indexing succeeded, false otherwise.\n   */\n  @VisibleForTesting\n  protected boolean indexSegment(final SegmentInfo segmentInfo,\n                                 @Nullable final SegmentInfo segmentToAppend,\n                                 final Predicate<Date> dateFilter)\n      throws CoordinatedEarlybirdActionLockFailed, IOException {\n    // Don't coordinate while we're starting up\n    if (!EarlybirdStatus.isStarting()) {\n      return coordinatedDailyUpdate.execute(segmentInfo.getSegmentName(),\n          isCoordinated -> innerIndexSegment(segmentInfo, segmentToAppend, dateFilter));\n    } else {\n      return innerIndexSegment(segmentInfo, segmentToAppend, dateFilter);\n    }\n  }\n\n  private boolean innerIndexSegment(SegmentInfo segmentInfo,\n                                    @Nullable SegmentInfo segmentToAppend,\n                                    Predicate<Date> dateFilter)\n      throws IOException {\n\n    // First try to load the new day from HDFS / Local disk\n    if (new SegmentLoader(segmentSyncConfig, criticalExceptionHandler).load(segmentInfo)) {\n      LOG.info(\"Successful loaded segment for new day: \" + segmentInfo);\n      segmentManager.putSegmentInfo(segmentInfo);\n      gcAfterIndexing.increment();\n      GCUtil.runGC();\n      return true;\n    }\n\n    LOG.info(\"Failed to load segment for new day. Will index segment: \" + segmentInfo);\n    RecordReader<TweetDocument> tweetReader = ((ArchiveSegment) segmentInfo.getSegment())\n        .getStatusRecordReader(earlybirdIndexConfig.createDocumentFactory(), dateFilter);\n    try {\n      // Read and index the statuses\n      boolean success = newSimpleSegmentIndexer(tweetReader, segmentToAppend)\n          .indexSegment(segmentInfo);\n      if (!success) {\n        return false;\n      }\n    } finally {\n      tweetReader.stop();\n    }\n\n    if (!SegmentOptimizer.optimize(segmentInfo)) {\n      // We consider the whole indexing event as failed if we fail to optimize.\n      LOG.error(\"Failed to optimize segment: \" + segmentInfo);\n      segmentInfo.deleteLocalIndexedSegmentDirectoryImmediately();\n      return false;\n    }\n\n    if (!segmentWarmer.warmSegmentIfNecessary(segmentInfo)) {\n      // We consider the whole indexing event as failed if we failed to warm (because we open\n      // index readers in the warmer).\n      LOG.error(\"Failed to warm segment: \" + segmentInfo);\n      segmentInfo.deleteLocalIndexedSegmentDirectoryImmediately();\n      return false;\n    }\n\n    // Flush and upload segment to HDFS. If this fails, we just log a warning and return true.\n    boolean success = new SegmentHdfsFlusher(zkTryLockFactory, segmentSyncConfig)\n        .flushSegmentToDiskAndHDFS(segmentInfo);\n    if (!success) {\n      LOG.warn(\"Failed to flush segment to HDFS: \" + segmentInfo);\n    }\n\n    segmentManager.putSegmentInfo(segmentInfo);\n    gcAfterIndexing.increment();\n    GCUtil.runGC();\n    return true;\n  }\n\n  @VisibleForTesting\n  protected SimpleSegmentIndexer newSimpleSegmentIndexer(\n      RecordReader<TweetDocument> tweetReader, SegmentInfo segmentToAppend) {\n    return new SimpleSegmentIndexer(tweetReader, indexingMetricSet, segmentToAppend);\n  }\n\n  @Override\n  public boolean isCaughtUpForTests() {\n    return EarlybirdStatus.getStatusCode() == EarlybirdStatusCode.CURRENT;\n  }\n\n  public CoordinatedEarlybirdActionInterface getCoordinatedOptimizer() {\n    return this.coordinatedDailyUpdate;\n  }\n\n  public ArchiveTimeSlicer getTimeSlicer() {\n    return timeSlicer;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/ArchiveSegment.java",
    "content": "package com.twitter.search.earlybird.archive;\n\nimport java.io.IOException;\nimport java.util.Date;\n\nimport com.google.common.base.Predicate;\nimport com.google.common.base.Predicates;\n\nimport com.twitter.search.common.partitioning.base.Segment;\nimport com.twitter.search.common.partitioning.base.TimeSlice;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.util.io.recordreader.RecordReader;\nimport com.twitter.search.earlybird.archive.ArchiveTimeSlicer.ArchiveTimeSlice;\nimport com.twitter.search.earlybird.document.DocumentFactory;\nimport com.twitter.search.earlybird.document.TweetDocument;\n\npublic class ArchiveSegment extends Segment {\n  private final ArchiveTimeSlice archiveTimeSlice;\n\n  public static final Predicate<Date> MATCH_ALL_DATE_PREDICATE = input -> true;\n\n  // Constructor used for indexing an archive segment\n  public ArchiveSegment(ArchiveTimeSlice archiveTimeSlice,\n                        int hashPartitionID,\n                        int maxSegmentSize) {\n    super(new TimeSlice(archiveTimeSlice.getMinStatusID(hashPartitionID),\n            maxSegmentSize, hashPartitionID,\n            archiveTimeSlice.getNumHashPartitions()),\n        archiveTimeSlice.getEndDate().getTime());\n    this.archiveTimeSlice = archiveTimeSlice;\n  }\n\n  /**\n   * Constructor used for loading a flushed segment. Only be used by SegmentBuilder; Earlybird\n   * does not use this.\n   */\n  ArchiveSegment(long timeSliceId,\n                 int maxSegmentSize,\n                 int partitions,\n                 int hashPartitionID,\n                 Date dataEndDate) {\n    super(new TimeSlice(timeSliceId, maxSegmentSize, hashPartitionID, partitions),\n        dataEndDate.getTime());\n    // No archive timeslice is needed for loading.\n    this.archiveTimeSlice = null;\n  }\n\n  /**\n   * Returns the tweets reader for this segment.\n   *\n   * @param documentFactory The factory that converts ThriftDocuments to Lucene documents.\n   */\n  public RecordReader<TweetDocument> getStatusRecordReader(\n      DocumentFactory<ThriftIndexingEvent> documentFactory) throws IOException {\n    return getStatusRecordReader(documentFactory, Predicates.<Date>alwaysTrue());\n  }\n\n  /**\n   * Returns the tweets reader for this segment.\n   *\n   * @param documentFactory The factory that converts ThriftDocuments to Lucene documents.\n   * @param filter A predicate that filters tweets based on the date they were created on.\n   */\n  public RecordReader<TweetDocument> getStatusRecordReader(\n      DocumentFactory<ThriftIndexingEvent> documentFactory,\n      Predicate<Date> filter) throws IOException {\n    if (archiveTimeSlice != null) {\n      return archiveTimeSlice.getStatusReader(this, documentFactory, filter);\n    } else {\n      throw new IllegalStateException(\"ArchiveSegment has no associated ArchiveTimeslice.\"\n          + \"This ArchiveSegment can only be used for loading flushed segments.\");\n    }\n  }\n\n  public Date getDataEndDate() {\n    return archiveTimeSlice == null\n        ? new Date(getDataEndDateInclusiveMillis()) : archiveTimeSlice.getEndDate();\n  }\n\n  public ArchiveTimeSlice getArchiveTimeSlice() {\n    return archiveTimeSlice;\n  }\n\n  @Override\n  public String toString() {\n    return super.toString() + \" \" + archiveTimeSlice.getDescription();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/ArchiveSegmentDataProvider.java",
    "content": "package com.twitter.search.earlybird.archive;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\n\nimport com.twitter.search.common.partitioning.base.Segment;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.util.io.recordreader.RecordReader;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.archive.ArchiveTimeSlicer.ArchiveTimeSlice;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.document.DocumentFactory;\nimport com.twitter.search.earlybird.document.TweetDocument;\nimport com.twitter.search.earlybird.partition.DynamicPartitionConfig;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.segment.EmptySegmentDataReaderSet;\nimport com.twitter.search.earlybird.segment.SegmentDataProvider;\nimport com.twitter.search.earlybird.segment.SegmentDataReaderSet;\n\npublic class ArchiveSegmentDataProvider implements SegmentDataProvider {\n  private static final org.slf4j.Logger LOG =\n      org.slf4j.LoggerFactory.getLogger(ArchiveSegmentDataProvider.class);\n\n  private DynamicPartitionConfig dynamicPartitionConfig;\n  private final ArchiveTimeSlicer timeSlicer;\n\n  private final DocumentFactory<ThriftIndexingEvent> documentFactory;\n\n  private final SegmentDataReaderSet readerSet;\n\n  public ArchiveSegmentDataProvider(\n      DynamicPartitionConfig dynamicPartitionConfig,\n      ArchiveTimeSlicer timeSlicer,\n      EarlybirdIndexConfig earlybirdIndexConfig) throws IOException {\n    this.dynamicPartitionConfig = dynamicPartitionConfig;\n    this.timeSlicer = timeSlicer;\n    this.readerSet = createSegmentDataReaderSet();\n    this.documentFactory = earlybirdIndexConfig.createDocumentFactory();\n  }\n\n  @Override\n  public List<Segment> newSegmentList() throws IOException {\n    List<ArchiveTimeSlice> timeSlices = timeSlicer.getTimeSlicesInTierRange();\n    if (timeSlices == null || timeSlices.isEmpty()) {\n      return Lists.newArrayList();\n    }\n    List<Segment> segments = Lists.newArrayListWithCapacity(timeSlices.size());\n    for (ArchiveTimeSlice timeSlice : timeSlices) {\n      segments.add(newArchiveSegment(timeSlice));\n    }\n    return segments;\n  }\n\n  /**\n   * Creates a new Segment instance for the given timeslice.\n   */\n  public ArchiveSegment newArchiveSegment(ArchiveTimeSlice archiveTimeSlice) {\n    return new ArchiveSegment(\n        archiveTimeSlice,\n        dynamicPartitionConfig.getCurrentPartitionConfig().getIndexingHashPartitionID(),\n        EarlybirdConfig.getMaxSegmentSize());\n  }\n\n  @Override\n  public SegmentDataReaderSet getSegmentDataReaderSet() {\n    return readerSet;\n  }\n\n  private EmptySegmentDataReaderSet createSegmentDataReaderSet() throws IOException {\n    return new EmptySegmentDataReaderSet() {\n\n      @Override\n      public RecordReader<TweetDocument> newDocumentReader(SegmentInfo segmentInfo)\n          throws IOException {\n        Segment segment = segmentInfo.getSegment();\n        Preconditions.checkArgument(segment instanceof ArchiveSegment);\n        return ((ArchiveSegment) segment).getStatusRecordReader(documentFactory);\n      }\n    };\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/ArchiveSegmentUpdater.java",
    "content": "package com.twitter.search.earlybird.archive;\n\nimport java.io.IOException;\nimport java.util.Date;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Predicate;\n\nimport org.apache.commons.lang.time.FastDateFormat;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.metrics.SearchStatsReceiverImpl;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.util.io.recordreader.RecordReader;\nimport com.twitter.search.common.util.zktrylock.ZooKeeperTryLockFactory;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.document.DocumentFactory;\nimport com.twitter.search.earlybird.document.TweetDocument;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.index.EarlybirdSegmentFactory;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\nimport com.twitter.search.earlybird.partition.SegmentHdfsFlusher;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentLoader;\nimport com.twitter.search.earlybird.partition.SegmentOptimizer;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\nimport com.twitter.search.earlybird.partition.SimpleSegmentIndexer;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\n\n/**\n * Given a segment, this class checks if the segment has an index built on HDFS:\n *   if not, use SimpleSegmentIndexer to build an index\n *   if yes, load the HDFS index, build a new index for the new status data which has dates newer\n *   than the HDFS index, then append the loaded HDFS index.\n */\npublic class ArchiveSegmentUpdater {\n  private static final Logger LOG = LoggerFactory.getLogger(ArchiveSegmentUpdater.class);\n\n  private final SegmentSyncConfig sync;\n  private final EarlybirdIndexConfig earlybirdIndexConfig;\n  private final ZooKeeperTryLockFactory zkTryLockFactory;\n  private final SearchStatsReceiver statsReceiver = new SearchStatsReceiverImpl();\n  private final SearchIndexingMetricSet searchIndexingMetricSet =\n      new SearchIndexingMetricSet(statsReceiver);\n  private final EarlybirdSearcherStats searcherStats =\n      new EarlybirdSearcherStats(statsReceiver);\n  private final SearchRateCounter indexNewSegment =\n      new SearchRateCounter(\"index_new_segment\");\n  private final SearchRateCounter updateExistingSegment =\n      new SearchRateCounter(\"update_existing_segment\");\n  private final SearchRateCounter skipExistingSegment =\n      new SearchRateCounter(\"skip_existing_segment\");\n  private Clock clock;\n\n  public ArchiveSegmentUpdater(ZooKeeperTryLockFactory zooKeeperTryLockFactory,\n                               SegmentSyncConfig sync,\n                               EarlybirdIndexConfig earlybirdIndexConfig,\n                               Clock clock) {\n    this.sync = sync;\n    this.earlybirdIndexConfig = earlybirdIndexConfig;\n    this.zkTryLockFactory = zooKeeperTryLockFactory;\n    this.clock = clock;\n  }\n\n  private boolean canUpdateSegment(SegmentInfo segmentInfo) {\n    if (!(segmentInfo.getSegment() instanceof ArchiveSegment)) {\n      LOG.info(\"only ArchiveSegment is available for updating now: \"\n          + segmentInfo);\n      return false;\n    }\n\n    if (!segmentInfo.isEnabled()) {\n      LOG.debug(\"Segment is disabled: \" + segmentInfo);\n      return false;\n    }\n\n    if (segmentInfo.isComplete() || segmentInfo.isIndexing()\n        || segmentInfo.getSyncInfo().isLoaded()) {\n      LOG.debug(\"Cannot update already indexed segment: \" + segmentInfo);\n      return false;\n    }\n\n    return true;\n  }\n\n  /**\n   * Given a segment, checks if the segment has an index built on HDFS:\n   *   if not, use SimpleSegmentIndexer to build an index\n   *   if yes, load the HDFS index, build a new index for the new status data which has dates newer\n   *   than the HDFS index, then append the loaded HDFS index.\n   *\n   * Returns whether the segment was successfully updated.\n   */\n  public boolean updateSegment(SegmentInfo segmentInfo) {\n    Preconditions.checkArgument(segmentInfo.getSegment() instanceof ArchiveSegment);\n    if (!canUpdateSegment(segmentInfo)) {\n      return false;\n    }\n\n    if (segmentInfo.isIndexing()) {\n      LOG.error(\"Segment is already being indexed: \" + segmentInfo);\n      return false;\n    }\n\n    final Date hdfsEndDate = ArchiveHDFSUtils.getSegmentEndDateOnHdfs(sync, segmentInfo);\n    if (hdfsEndDate == null) {\n      indexNewSegment.increment();\n      if (!indexSegment(segmentInfo, ArchiveSegment.MATCH_ALL_DATE_PREDICATE)) {\n        return false;\n      }\n    } else {\n      final Date curEndDate = ((ArchiveSegment) segmentInfo.getSegment()).getDataEndDate();\n      if (!hdfsEndDate.before(curEndDate)) {\n        skipExistingSegment.increment();\n        LOG.info(\"Segment is up-to-date: \" + segmentInfo.getSegment().getTimeSliceID()\n            + \" Found flushed segment on HDFS with end date: \"\n            + FastDateFormat.getInstance(\"yyyyMMdd\").format(hdfsEndDate));\n        segmentInfo.setComplete(true);\n        segmentInfo.getSyncInfo().setFlushed(true);\n        return true;\n      }\n\n      updateExistingSegment.increment();\n      LOG.info(\"Updating segment: \" + segmentInfo.getSegment().getTimeSliceID()\n          + \"; new endDate will be \" + FastDateFormat.getInstance(\"yyyyMMdd\").format(curEndDate));\n\n      if (!updateSegment(segmentInfo, hdfsEndDate)) {\n        return false;\n      }\n    }\n\n    boolean success = SegmentOptimizer.optimize(segmentInfo);\n    if (!success) {\n      // Clean up the segment dir on local disk\n      segmentInfo.deleteLocalIndexedSegmentDirectoryImmediately();\n      LOG.info(\"Error optimizing segment: \" + segmentInfo);\n      return false;\n    }\n\n    // Verify segment before uploading.\n    success = ArchiveSegmentVerifier.verifySegment(segmentInfo);\n    if (!success) {\n      segmentInfo.deleteLocalIndexedSegmentDirectoryImmediately();\n      LOG.info(\"Segment not uploaded to HDFS because it did not pass verification: \" + segmentInfo);\n      return false;\n    }\n\n    // upload the index to HDFS\n    success = new SegmentHdfsFlusher(zkTryLockFactory, sync, false)\n        .flushSegmentToDiskAndHDFS(segmentInfo);\n    if (success) {\n      ArchiveHDFSUtils.deleteHdfsSegmentDir(sync, segmentInfo, false, true);\n    } else {\n      // Clean up the segment dir on hdfs\n      ArchiveHDFSUtils.deleteHdfsSegmentDir(sync, segmentInfo, true, false);\n      LOG.info(\"Error uploading segment to HDFS: \" + segmentInfo);\n    }\n    segmentInfo.deleteLocalIndexedSegmentDirectoryImmediately();\n\n    return success;\n  }\n\n  /**\n   * Build index for the given segmentInfo. Only those statuses passing the dateFilter are indexed.\n   */\n  private boolean indexSegment(final SegmentInfo segmentInfo, Predicate<Date> dateFilter) {\n    Preconditions.checkArgument(segmentInfo.getSegment() instanceof ArchiveSegment);\n\n    RecordReader<TweetDocument> documentReader = null;\n    try {\n      ArchiveSegment archiveSegment = (ArchiveSegment) segmentInfo.getSegment();\n      DocumentFactory<ThriftIndexingEvent> documentFactory =\n          earlybirdIndexConfig.createDocumentFactory();\n      documentReader = archiveSegment.getStatusRecordReader(documentFactory, dateFilter);\n\n      // Read and index the statuses\n      boolean success = new SimpleSegmentIndexer(documentReader, searchIndexingMetricSet)\n          .indexSegment(segmentInfo);\n      if (!success) {\n        // Clean up segment dir on local disk\n        segmentInfo.deleteLocalIndexedSegmentDirectoryImmediately();\n        LOG.info(\"Error indexing segment: \" + segmentInfo);\n      }\n\n      return success;\n    } catch (IOException e) {\n      segmentInfo.deleteLocalIndexedSegmentDirectoryImmediately();\n      LOG.info(\"Exception while indexing segment: \" + segmentInfo, e);\n      return false;\n    } finally {\n      if (documentReader != null) {\n        documentReader.stop();\n      }\n    }\n  }\n\n  /**\n   * Load the index built on HDFS for the given segmentInfo, index the new data and append the\n   * HDFS index to the new indexed segment\n   */\n  private boolean updateSegment(final SegmentInfo segmentInfo, final Date hdfsEndDate) {\n    SegmentInfo hdfsSegmentInfo = loadSegmentFromHdfs(segmentInfo, hdfsEndDate);\n    if (hdfsSegmentInfo == null) {\n      return indexSegment(segmentInfo, ArchiveSegment.MATCH_ALL_DATE_PREDICATE);\n    }\n\n    boolean success = indexSegment(segmentInfo, input -> {\n      // we're updating the segment - only index days after the old end date,\n      // and we're sure that the previous days have already been indexed.\n      return input.after(hdfsEndDate);\n    });\n    if (!success) {\n      LOG.error(\"Error indexing new data: \" + segmentInfo);\n      return indexSegment(segmentInfo, ArchiveSegment.MATCH_ALL_DATE_PREDICATE);\n    }\n\n    // Now, append the index loaded from hdfs\n    try {\n      segmentInfo.getIndexSegment().append(hdfsSegmentInfo.getIndexSegment());\n      hdfsSegmentInfo.deleteLocalIndexedSegmentDirectoryImmediately();\n      LOG.info(\"Deleted local segment directories with end date \" + hdfsEndDate + \" : \"\n          + segmentInfo);\n    } catch (IOException e) {\n      LOG.warn(\"Caught IOException while appending segment \" + hdfsSegmentInfo.getSegmentName(), e);\n      hdfsSegmentInfo.deleteLocalIndexedSegmentDirectoryImmediately();\n      segmentInfo.deleteLocalIndexedSegmentDirectoryImmediately();\n      return false;\n    }\n\n    segmentInfo.setComplete(true);\n    return true;\n  }\n\n  /**\n   * Load the index built on HDFS for the given segmentInfo and end date\n   */\n  private SegmentInfo loadSegmentFromHdfs(final SegmentInfo segmentInfo, final Date hdfsEndDate) {\n    Preconditions.checkArgument(segmentInfo.getSegment() instanceof ArchiveSegment);\n\n    ArchiveSegment segment = new ArchiveSegment(\n        segmentInfo.getTimeSliceID(),\n        EarlybirdConfig.getMaxSegmentSize(),\n        segmentInfo.getNumPartitions(),\n        segmentInfo.getSegment().getHashPartitionID(),\n        hdfsEndDate);\n    EarlybirdSegmentFactory factory = new EarlybirdSegmentFactory(\n        earlybirdIndexConfig,\n        searchIndexingMetricSet,\n        searcherStats,\n        clock);\n\n    SegmentInfo hdfsSegmentInfo;\n\n    try {\n      hdfsSegmentInfo = new SegmentInfo(segment,  factory, sync);\n      CriticalExceptionHandler criticalExceptionHandler =\n          new CriticalExceptionHandler();\n\n      boolean success = new SegmentLoader(sync, criticalExceptionHandler)\n          .load(hdfsSegmentInfo);\n      if (!success) {\n        // If not successful, segmentLoader has already cleaned up the local dir.\n        LOG.info(\"Error loading hdfs segment \" + hdfsSegmentInfo\n            + \", building segment from scratch.\");\n        hdfsSegmentInfo = null;\n      }\n    } catch (IOException e) {\n      LOG.error(\"Exception while loading segment from hdfs: \" + segmentInfo, e);\n      hdfsSegmentInfo = null;\n    }\n\n    return hdfsSegmentInfo;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/ArchiveSegmentVerifier.java",
    "content": "package com.twitter.search.earlybird.archive;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.lucene.index.DirectoryReader;\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.store.Directory;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.earlybird.partition.SegmentInfo;\n\npublic final class ArchiveSegmentVerifier {\n  private static final Logger LOG = LoggerFactory.getLogger(ArchiveSegmentVerifier.class);\n\n  private ArchiveSegmentVerifier() {\n  }\n\n  @VisibleForTesting\n  static boolean shouldVerifySegment(SegmentInfo segmentInfo) {\n    if (segmentInfo.isIndexing()) {\n      LOG.warn(\"ArchiveSegmentVerifier got segment still indexing.\");\n      return false;\n    }\n\n    if (!segmentInfo.isComplete()) {\n      LOG.warn(\"ArchiveSegmentVerifyer got incomplete segment.\");\n      return false;\n    }\n\n    if (!segmentInfo.isOptimized()) {\n      LOG.warn(\"ArchiveSegmentVerifyer got unoptimized segment.\");\n      return false;\n    }\n\n    return true;\n  }\n\n  /**\n   * Verifies an archive segment has a sane number of leaves.\n   */\n  public static boolean verifySegment(SegmentInfo segmentInfo) {\n    if (!shouldVerifySegment(segmentInfo)) {\n      return false;\n    }\n    Directory directory = segmentInfo.getIndexSegment().getLuceneDirectory();\n    return verifyLuceneIndex(directory);\n  }\n\n  private static boolean verifyLuceneIndex(Directory directory) {\n    try {\n      DirectoryReader indexerReader = DirectoryReader.open(directory);\n      List<LeafReaderContext> leaves = indexerReader.getContext().leaves();\n      if (leaves.size() != 1) {\n        LOG.warn(\"Lucene index does not have exactly one segment: \" + leaves.size() + \" != 1. \"\n            + \"Lucene segments should have been merged during optimization.\");\n        return false;\n      }\n\n      LeafReader reader = leaves.get(0).reader();\n      if (reader.numDocs() <= 0) {\n        LOG.warn(\"Lucene index has no document: \" + reader);\n        return false;\n      }\n      return true;\n    } catch (IOException e) {\n      LOG.warn(\"Found bad lucene index at: \" + directory);\n      return false;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/ArchiveTimeSlicer.java",
    "content": "package com.twitter.search.earlybird.archive;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Calendar;\nimport java.util.Collections;\nimport java.util.Comparator;\nimport java.util.Date;\nimport java.util.List;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Predicate;\nimport com.google.common.collect.Lists;\n\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.util.io.MergingSortedRecordReader;\nimport com.twitter.search.common.util.io.recordreader.RecordReader;\nimport com.twitter.search.earlybird.config.TierConfig;\nimport com.twitter.search.earlybird.document.DocumentFactory;\nimport com.twitter.search.earlybird.document.ThriftIndexingEventDocumentFactory;\nimport com.twitter.search.earlybird.document.TweetDocument;\n\n\n/**\n * Responsible for taking a number of daily status batches and partitioning them into time slices\n * which will be used to build segments.\n *\n * We try to put at most N number of tweets into a time slice.\n */\npublic class ArchiveTimeSlicer {\n  private static final Logger LOG = LoggerFactory.getLogger(ArchiveTimeSlicer.class);\n\n  private static final Comparator<TweetDocument> ASCENDING =\n      (o1, o2) -> Long.compare(o1.getTweetID(), o2.getTweetID());\n\n  private static final Comparator<TweetDocument> DESCENDING =\n      (o1, o2) -> Long.compare(o2.getTweetID(), o1.getTweetID());\n\n  // Represents a number of daily batches which will go into a segment.\n  public static final class ArchiveTimeSlice {\n    private Date startDate;\n    private Date endDate;\n    private int statusCount;\n    private final DailyStatusBatches directory;\n    private final ArchiveEarlybirdIndexConfig earlybirdIndexConfig;\n\n    // This list is always ordered from oldest day, to the newest day.\n    // For the on-disk archive, we reverse the days in getTweetReaders().\n    private final List<DailyStatusBatch> batches = Lists.newArrayList();\n\n    private ArchiveTimeSlice(DailyStatusBatches directory,\n                             ArchiveEarlybirdIndexConfig earlybirdIndexConfig) {\n      this.directory = directory;\n      this.earlybirdIndexConfig = earlybirdIndexConfig;\n    }\n\n    public Date getEndDate() {\n      return endDate;\n    }\n\n    public int getStatusCount() {\n      return statusCount;\n    }\n\n    public int getNumHashPartitions() {\n      return batches.isEmpty() ? 0 : batches.get(0).getNumHashPartitions();\n    }\n\n    /**\n     * Returns a reader for reading tweets from this timeslice.\n     *\n     * @param archiveSegment The segment to which the timeslice belongs.\n     * @param documentFactory The ThriftIndexingEvent to TweetDocument converter.\n     * @param filter A filter that determines what dates should be read.\n     */\n    public RecordReader<TweetDocument> getStatusReader(\n        ArchiveSegment archiveSegment,\n        DocumentFactory<ThriftIndexingEvent> documentFactory,\n        Predicate<Date> filter) throws IOException {\n      // We no longer support ThriftStatus based document factories.\n      Preconditions.checkState(documentFactory instanceof ThriftIndexingEventDocumentFactory);\n\n      final int hashPartitionID = archiveSegment.getHashPartitionID();\n      List<RecordReader<TweetDocument>> readers = new ArrayList<>(batches.size());\n      List<DailyStatusBatch> orderedForReading = orderBatchesForReading(batches);\n      LOG.info(\"Creating new status reader for hashPartition: \"\n          + hashPartitionID + \" timeslice: \" + getDescription());\n\n      for (DailyStatusBatch batch : orderedForReading) {\n        if (filter.apply(batch.getDate())) {\n          LOG.info(\"Adding reader for \" + batch.getDate() + \" \" + getDescription());\n          PartitionedBatch partitionedBatch = batch.getPartition(hashPartitionID);\n          // Don't even try to create a reader if the partition is empty.\n          // There does not seem to be any problem in production now, but HDFS FileSystem's javadoc\n          // does indicate that listStatus() is allowed to throw a FileNotFoundException if the\n          // partition does not exist. This check makes the code more robust against future\n          // HDFS FileSystem implementation changes.\n          if (partitionedBatch.getStatusCount() > 0) {\n            RecordReader<TweetDocument> tweetReaders = partitionedBatch.getTweetReaders(\n                archiveSegment,\n                directory.getStatusPathToUseForDay(batch.getDate()),\n                documentFactory);\n            readers.add(tweetReaders);\n          }\n        } else {\n          LOG.info(\"Filtered reader for \" + batch.getDate() + \" \" + getDescription());\n        }\n      }\n\n      LOG.info(\"Creating reader for timeslice: \" + getDescription()\n          + \" with \" + readers.size() + \" readers\");\n\n      return new MergingSortedRecordReader<TweetDocument>(getMergingComparator(), readers);\n    }\n\n    private List<DailyStatusBatch> orderBatchesForReading(List<DailyStatusBatch> orderedBatches) {\n      // For the index formats using stock lucene, we want the most recent days to be indexed first.\n      // In the twitter in-memory optimized indexes, older tweets will be added first, and\n      // optimization will reverse the documents to make most recent tweets be first.\n      return this.earlybirdIndexConfig.isUsingLIFODocumentOrdering()\n          ? orderedBatches : Lists.reverse(orderedBatches);\n    }\n\n    private Comparator<TweetDocument> getMergingComparator() {\n      // We always want to retrieve larger tweet ids first.\n      // LIFO means that the smaller ids get inserted first --> ASCENDING order.\n      // FIFO would mean that we want to first insert the larger ids --> DESCENDING order.\n      return this.earlybirdIndexConfig.isUsingLIFODocumentOrdering()\n          ? ASCENDING : DESCENDING;\n    }\n\n    /**\n     * Returns the smallest indexed tweet ID in this timeslice for the given partition.\n     *\n     * @param hashPartitionID The partition.\n     */\n    public long getMinStatusID(int hashPartitionID) {\n      if (batches.isEmpty()) {\n        return 0;\n      }\n\n      for (int i = 0; i < batches.size(); i++) {\n        long minStatusID = batches.get(i).getPartition(hashPartitionID).getMinStatusID();\n        if (minStatusID != DailyStatusBatch.EMPTY_BATCH_STATUS_ID) {\n          return minStatusID;\n        }\n      }\n\n      return 0;\n    }\n\n    /**\n     * Returns the highest indexed tweet ID in this timeslice for the given partition.\n     *\n     * @param hashPartitionID The partition.\n     */\n    public long getMaxStatusID(int hashPartitionID) {\n      if (batches.isEmpty()) {\n        return Long.MAX_VALUE;\n      }\n\n      for (int i = batches.size() - 1; i >= 0; i--) {\n        long maxStatusID = batches.get(i).getPartition(hashPartitionID).getMaxStatusID();\n        if (maxStatusID != DailyStatusBatch.EMPTY_BATCH_STATUS_ID) {\n          return maxStatusID;\n        }\n      }\n\n      return Long.MAX_VALUE;\n    }\n\n    /**\n     * Returns a string with some information for this timeslice.\n     */\n    public String getDescription() {\n      StringBuilder builder = new StringBuilder();\n      builder.append(\"TimeSlice[start date=\");\n      builder.append(DailyStatusBatches.DATE_FORMAT.format(startDate));\n      builder.append(\", end date=\");\n      builder.append(DailyStatusBatches.DATE_FORMAT.format(endDate));\n      builder.append(\", status count=\");\n      builder.append(statusCount);\n      builder.append(\", days count=\");\n      builder.append(batches.size());\n      builder.append(\"]\");\n      return builder.toString();\n    }\n  }\n\n  private final int maxSegmentSize;\n  private final DailyStatusBatches dailyStatusBatches;\n  private final Date tierStartDate;\n  private final Date tierEndDate;\n  private final ArchiveEarlybirdIndexConfig earlybirdIndexConfig;\n\n  private List<ArchiveTimeSlice> lastCachedTimeslices = null;\n\n  public ArchiveTimeSlicer(int maxSegmentSize,\n                           DailyStatusBatches dailyStatusBatches,\n                           ArchiveEarlybirdIndexConfig earlybirdIndexConfig) {\n    this(maxSegmentSize, dailyStatusBatches, TierConfig.DEFAULT_TIER_START_DATE,\n        TierConfig.DEFAULT_TIER_END_DATE, earlybirdIndexConfig);\n  }\n\n  public ArchiveTimeSlicer(int maxSegmentSize,\n                           DailyStatusBatches dailyStatusBatches,\n                           Date tierStartDate,\n                           Date tierEndDate,\n                           ArchiveEarlybirdIndexConfig earlybirdIndexConfig) {\n    this.maxSegmentSize = maxSegmentSize;\n    this.dailyStatusBatches = dailyStatusBatches;\n    this.tierStartDate = tierStartDate;\n    this.tierEndDate = tierEndDate;\n    this.earlybirdIndexConfig = earlybirdIndexConfig;\n  }\n\n  private boolean cacheIsValid() throws IOException {\n    return lastCachedTimeslices != null\n        && !lastCachedTimeslices.isEmpty()\n        && cacheIsValid(lastCachedTimeslices.get(lastCachedTimeslices.size() - 1).endDate);\n  }\n\n  private boolean cacheIsValid(Date lastDate) throws IOException {\n    if (lastCachedTimeslices == null || lastCachedTimeslices.isEmpty()) {\n      return false;\n    }\n\n    // Check if we have a daily batch newer than the last batch used for the newest timeslice.\n    Calendar cal = Calendar.getInstance();\n    cal.setTime(lastDate);\n    cal.add(Calendar.DATE, 1);\n    Date nextDate = cal.getTime();\n\n    boolean foundBatch = dailyStatusBatches.hasValidBatchForDay(nextDate);\n\n    LOG.info(\"Checking cache: Looked for valid batch for day {}. Found: {}\",\n        DailyStatusBatches.DATE_FORMAT.format(nextDate), foundBatch);\n\n    return !foundBatch;\n  }\n\n  private boolean timesliceIsFull(ArchiveTimeSlice timeSlice, DailyStatusBatch batch) {\n    return timeSlice.statusCount + batch.getMaxPerPartitionStatusCount() > maxSegmentSize;\n  }\n\n  private void doTimeSlicing() throws IOException {\n    dailyStatusBatches.refresh();\n\n    lastCachedTimeslices = Lists.newArrayList();\n    ArchiveTimeSlice currentTimeSlice = null;\n\n    // Iterate over each day and add it to the current timeslice, until it gets full.\n    for (DailyStatusBatch batch : dailyStatusBatches.getStatusBatches()) {\n      if (!batch.isValid()) {\n        LOG.warn(\"Skipping hole: \" + batch.getDate());\n        continue;\n      }\n\n      if (currentTimeSlice == null || timesliceIsFull(currentTimeSlice, batch)) {\n        if (currentTimeSlice != null) {\n          LOG.info(\"Filled timeslice: \" + currentTimeSlice.getDescription());\n        }\n        currentTimeSlice = new ArchiveTimeSlice(dailyStatusBatches, earlybirdIndexConfig);\n        currentTimeSlice.startDate = batch.getDate();\n        lastCachedTimeslices.add(currentTimeSlice);\n      }\n\n      currentTimeSlice.endDate = batch.getDate();\n      currentTimeSlice.statusCount += batch.getMaxPerPartitionStatusCount();\n      currentTimeSlice.batches.add(batch);\n    }\n    LOG.info(\"Last timeslice: {}\", currentTimeSlice.getDescription());\n\n    LOG.info(\"Done with time slicing. Number of timeslices: {}\",\n        lastCachedTimeslices.size());\n  }\n\n  /**\n   * Returns all timeslices for this earlybird.\n   */\n  public List<ArchiveTimeSlice> getTimeSlices() throws IOException {\n    if (cacheIsValid()) {\n      return lastCachedTimeslices;\n    }\n\n    LOG.info(\"Cache is outdated. Loading new daily batches now...\");\n\n    doTimeSlicing();\n\n    return lastCachedTimeslices != null ? Collections.unmodifiableList(lastCachedTimeslices) : null;\n  }\n\n  /**\n   * Return the timeslices that overlap the tier start/end date ranges if they are specified\n   */\n  public List<ArchiveTimeSlice> getTimeSlicesInTierRange() throws IOException {\n    List<ArchiveTimeSlice> timeSlices = getTimeSlices();\n    if (tierStartDate == TierConfig.DEFAULT_TIER_START_DATE\n        && tierEndDate == TierConfig.DEFAULT_TIER_END_DATE) {\n      return timeSlices;\n    }\n\n    List<ArchiveTimeSlice> filteredTimeSlice = Lists.newArrayList();\n    for (ArchiveTimeSlice timeSlice : timeSlices) {\n      if (timeSlice.startDate.before(tierEndDate) && !timeSlice.endDate.before(tierStartDate)) {\n        filteredTimeSlice.add(timeSlice);\n      }\n    }\n\n    return filteredTimeSlice;\n  }\n\n  @VisibleForTesting\n  protected DailyStatusBatches getDailyStatusBatches() {\n    return dailyStatusBatches;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/DailyStatusBatch.java",
    "content": "package com.twitter.search.earlybird.archive;\n\nimport java.io.IOException;\nimport java.util.Date;\nimport java.util.Map;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.Maps;\nimport com.google.gson.Gson;\nimport com.google.gson.JsonParseException;\n\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.Path;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\n/**\n * Represents a day's worth of statuses (tweets) for multiple hash partitions.\n *\n * Note that what this class contains is not the data, but metadata.\n *\n * A day of tweets will come from:\n * - A scrubgen, if it has happened before the scrubgen date.\n * - Our daily jobs pipeline, if it has happened after that.\n *\n * This class checks the _SUCCESS file exists in the \"statuses\" subdirectory and extracts the status\n * count, min status id and max status id.\n */\npublic class DailyStatusBatch implements Comparable<DailyStatusBatch> {\n  private static final Logger LOG = LoggerFactory.getLogger(DailyStatusBatch.class);\n\n  public static final long EMPTY_BATCH_STATUS_ID = -1;\n  private static final String PARTITION_FORMAT = \"p_%d_of_%d\";\n  private static final String SUCCESS_FILE_NAME = \"_SUCCESS\";\n\n  private final Map<Integer, PartitionedBatch> hashPartitionToStatuses = Maps.newHashMap();\n\n  private final Date date;\n  private final int numHashPartitions;\n  private final boolean hasSuccessFiles;\n\n  public DailyStatusBatch(Date date, int numHashPartitions, Path statusPath, FileSystem hdfs) {\n    this.date = date;\n    this.numHashPartitions = numHashPartitions;\n    this.hasSuccessFiles = checkForSuccessFile(hdfs, date, statusPath);\n  }\n\n  public Date getDate() {\n    return date;\n  }\n\n  /**\n   * Check for the presence of the _SUCCESS file for the given day's path on HDFS for the statuses\n   * field group.\n   */\n  private boolean checkForSuccessFile(FileSystem hdfs, Date inputDate, Path statusPath) {\n    Path dayPath = new Path(statusPath, ArchiveHDFSUtils.dateToPath(inputDate, \"/\"));\n    Path successFilePath = new Path(dayPath, SUCCESS_FILE_NAME);\n    try {\n      return hdfs.getFileStatus(successFilePath).isFile();\n    } catch (IOException e) {\n      LOG.error(\"Could not verify existence of the _SUCCESS file. Assuming it doesn't exist.\", e);\n    }\n    return false;\n  }\n\n  /**\n   * Loads the data for this day for the given partition.\n   */\n  public PartitionedBatch addPartition(FileSystem hdfs, Path dayPath, int hashPartitionID)\n      throws IOException {\n    String partitionDir = String.format(PARTITION_FORMAT, hashPartitionID, numHashPartitions);\n    Path path = new Path(dayPath, partitionDir);\n    PartitionedBatch batch =\n        new PartitionedBatch(path, hashPartitionID, numHashPartitions, date);\n    batch.load(hdfs);\n    hashPartitionToStatuses.put(hashPartitionID, batch);\n    return batch;\n  }\n\n  public PartitionedBatch getPartition(int hashPartitionID) {\n    return hashPartitionToStatuses.get(hashPartitionID);\n  }\n\n  /**\n   * Returns the greatest status count in all partitions belonging to this batch.\n   */\n  public int getMaxPerPartitionStatusCount() {\n    int maxPerPartitionStatusCount = 0;\n    for (PartitionedBatch batch : hashPartitionToStatuses.values()) {\n      maxPerPartitionStatusCount = Math.max(batch.getStatusCount(), maxPerPartitionStatusCount);\n    }\n    return maxPerPartitionStatusCount;\n  }\n\n  public int getNumHashPartitions() {\n    return numHashPartitions;\n  }\n\n  @VisibleForTesting\n  boolean hasSuccessFiles() {\n    return hasSuccessFiles;\n  }\n\n  /**\n   * Returns true if the _status_counts files could be found in each\n   * hash partition subfolder that belongs to this timeslice\n   * AND the _SUCCESS file can be found at the root folder for day\n   */\n  public boolean isValid() {\n    // make sure we have data for all hash partitions\n    for (int i = 0; i < numHashPartitions; i++) {\n      PartitionedBatch day = hashPartitionToStatuses.get(i);\n      if (day == null || !day.hasStatusCount() || day.isDisallowedEmptyPartition()) {\n        return false;\n      }\n    }\n    return hasSuccessFiles;\n  }\n\n  @Override\n  public String toString() {\n    StringBuilder builder = new StringBuilder();\n    builder.append(\"DailyStatusBatch[date=\").append(date)\n           .append(\",valid=\").append(isValid())\n           .append(\",hasSuccessFiles=\").append(hasSuccessFiles)\n           .append(\",numHashPartitions=\").append(numHashPartitions)\n           .append(\"]:\\n\");\n    for (int i = 0; i < numHashPartitions; i++) {\n      builder.append('\\t').append(hashPartitionToStatuses.get(i).toString()).append('\\n');\n    }\n    return builder.toString();\n  }\n\n  @Override\n  public int compareTo(DailyStatusBatch o) {\n    return date.compareTo(o.date);\n  }\n\n  /**\n   * Serialize DailyStatusBatch to a json string.\n   */\n  public String serializeToJson() {\n    return serializeToJson(new Gson());\n  }\n\n  @VisibleForTesting\n  String serializeToJson(Gson gson) {\n    return gson.toJson(this);\n  }\n\n  /**\n   * Given a json string, parse its fields and construct a daily status batch.\n   * @param batchStr the json string representation of a daily status batch.\n   * @return the daily status batch constructed; if the string is of invalid format, null will be\n   *         returned.\n   */\n  static DailyStatusBatch deserializeFromJson(String batchStr) {\n    try {\n      return new Gson().fromJson(batchStr, DailyStatusBatch.class);\n    } catch (JsonParseException e) {\n      LOG.error(\"Error parsing json string: \" + batchStr, e);\n      return null;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/DailyStatusBatches.java",
    "content": "package com.twitter.search.earlybird.archive;\n\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.util.Calendar;\nimport java.util.Collection;\nimport java.util.Date;\nimport java.util.NavigableMap;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.atomic.AtomicBoolean;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Stopwatch;\nimport com.google.common.collect.Maps;\n\nimport org.apache.commons.io.IOUtils;\nimport org.apache.commons.lang3.time.FastDateFormat;\nimport org.apache.hadoop.fs.FileStatus;\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.Path;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.quantity.Amount;\nimport com.twitter.common.quantity.Time;\nimport com.twitter.search.common.database.DatabaseConfig;\nimport com.twitter.search.common.util.date.DateUtil;\nimport com.twitter.search.common.util.io.LineRecordFileReader;\nimport com.twitter.search.common.util.zktrylock.TryLock;\nimport com.twitter.search.common.util.zktrylock.ZooKeeperTryLockFactory;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.partition.HdfsUtil;\nimport com.twitter.search.earlybird.partition.StatusBatchFlushVersion;\n\n/**\n * Provides access to preprocessed statuses (tweets) to be indexed by archive search earlybirds.\n *\n * These tweets can be coming from a scrub gen or from the output of the daily jobs.\n */\npublic class DailyStatusBatches {\n  private static final Logger LOG = LoggerFactory.getLogger(DailyStatusBatches.class);\n\n  // Maximum time to spend on obtaining daily status batches by computing or loading from HDFS\n  private static final Amount<Long, Time> MAX_TIME_ALLOWED_DAILY_STATUS_BATCHES_MINUTES =\n      Amount.of(EarlybirdConfig.getLong(\"daily_status_batches_max_initial_load_time_minutes\"),\n          Time.MINUTES);\n  // Time to wait before trying again when obtaining daily status batches fails\n  private static final Amount<Long, Time> DAILY_STATUS_BATCHES_WAITING_TIME_MINUTES =\n      Amount.of(EarlybirdConfig.getLong(\"daily_status_batches_waiting_time_minutes\"),\n          Time.MINUTES);\n  private static final String DAILY_STATUS_BATCHES_SYNC_PATH =\n      EarlybirdProperty.ZK_APP_ROOT.get() + \"/daily_batches_sync\";\n  private static final String DAILY_BATCHES_ZK_LOCK = \"daily_batches_zk_lock\";\n  private static final Amount<Long, Time> DAILY_STATUS_BATCHES_ZK_LOCK_EXPIRATION_MINUTES =\n      Amount.of(EarlybirdConfig.getLong(\"daily_status_batches_zk_lock_expiration_minutes\"),\n          Time.MINUTES);\n\n  static final FastDateFormat DATE_FORMAT = FastDateFormat.getInstance(\"yyyyMMdd\");\n\n  // before this date, there was no twitter\n  private static final Date FIRST_TWITTER_DAY = DateUtil.toDate(2006, 2, 1);\n\n  private static final String STATUS_BATCHES_PREFIX = \"status_batches\";\n\n  private final String rootDir =\n      EarlybirdConfig.getString(\"hdfs_offline_segment_sync_dir\", \"top_archive_statuses\");\n\n  private final String buildGen =\n      EarlybirdConfig.getString(\"offline_segment_build_gen\", \"bg_1\");\n\n  public static final String STATUS_SUBDIR_NAME = \"statuses\";\n  public static final String LAYOUT_SUBDIR_NAME = \"layouts\";\n  public static final String SCRUB_GEN_SUFFIX_PATTERN = \"scrubbed/%s\";\n\n  private static final String INTERMEDIATE_COUNTS_SUBDIR_NAME = \"counts\";\n  private static final String SUCCESS_FILE_NAME = \"_SUCCESS\";\n  private static final Pattern HASH_PARTITION_PATTERN = Pattern.compile(\"p_(\\\\d+)_of_(\\\\d+)\");\n  private static final Date FIRST_TWEET_DAY = DateUtil.toDate(2006, 3, 21);\n\n  private final Path rootPath = new Path(rootDir);\n  private final Path buildGenPath = new Path(rootPath, buildGen);\n  private final Path statusPath = new Path(buildGenPath, STATUS_SUBDIR_NAME);\n\n  private final NavigableMap<Date, DailyStatusBatch> statusBatches = Maps.newTreeMap();\n\n  private Date firstValidDay = null;\n  private Date lastValidDay = null;\n\n  private final ZooKeeperTryLockFactory zkTryLockFactory;\n  private final Date scrubGenDay;\n  private long numberOfDaysWithValidScrubGenData;\n\n  public DailyStatusBatches(\n      ZooKeeperTryLockFactory zooKeeperTryLockFactory, Date scrubGenDay) throws IOException {\n    this.zkTryLockFactory = zooKeeperTryLockFactory;\n    this.scrubGenDay = scrubGenDay;\n\n    FileSystem hdfs = null;\n    try {\n      hdfs = HdfsUtil.getHdfsFileSystem();\n      verifyDirectory(hdfs);\n    } finally {\n      IOUtils.closeQuietly(hdfs);\n    }\n  }\n\n  @VisibleForTesting\n  public Date getScrubGenDay() {\n    return scrubGenDay;\n  }\n\n  public Collection<DailyStatusBatch> getStatusBatches() {\n    return statusBatches.values();\n  }\n\n  /**\n   * Reset the states of the directory\n   */\n  private void resetDirectory() {\n    statusBatches.clear();\n    firstValidDay = null;\n    lastValidDay = null;\n  }\n\n  /**\n   *  Indicate whether the directory has been initialized\n   */\n  private boolean isInitialized() {\n    return lastValidDay != null;\n  }\n\n  /**\n   * Load the daily status batches from HDFS; return true if one or more batches could be loaded.\n   **/\n  private boolean refreshByLoadingHDFSStatusBatches(final FileSystem fs) throws IOException {\n    // first find the latest valid end date of statuses\n    final Date lastValidStatusDay = getLastValidInputDateFromNow(fs);\n    if (lastValidStatusDay != null) {\n      if (hasStatusBatchesOnHdfs(fs, lastValidStatusDay)) {\n        if (loadStatusBatchesFromHdfs(fs, lastValidStatusDay)) {\n          return true;\n        }\n      }\n    }\n\n    resetDirectory();\n    return false;\n  }\n\n  /**\n   * Checks the directory for new data and returns true, if one or more new batches could be loaded.\n   */\n  public void refresh() throws IOException {\n    final FileSystem hdfs = HdfsUtil.getHdfsFileSystem();\n\n    final Stopwatch stopwatch = Stopwatch.createStarted();\n    try {\n      if (!isInitialized()) {\n        if (initializeDailyStatusBatches(hdfs, stopwatch)) {\n          LOG.info(\"Successfully obtained daily status batches after {}\", stopwatch);\n        } else {\n          String errMsg = \"Failed to load or compute daily status batches after \"\n              + stopwatch.toString();\n          LOG.error(errMsg);\n          throw new IOException(errMsg);\n        }\n      } else {\n        loadNewDailyBatches(hdfs);\n      }\n    } finally {\n      IOUtils.closeQuietly(hdfs);\n    }\n  }\n\n  private boolean initializeDailyStatusBatches(final FileSystem hdfs,\n                                               final Stopwatch stopwatch) throws IOException {\n    long timeSpentOnDailyBatches = 0L;\n    long maxAllowedTimeMs = MAX_TIME_ALLOWED_DAILY_STATUS_BATCHES_MINUTES.as(Time.MILLISECONDS);\n    long waitingTimeMs = DAILY_STATUS_BATCHES_WAITING_TIME_MINUTES.as(Time.MILLISECONDS);\n    boolean firstLoop = true;\n    LOG.info(\"Starting to load or compute daily status batches for the first time.\");\n    while (timeSpentOnDailyBatches <= maxAllowedTimeMs && !Thread.currentThread().isInterrupted()) {\n      if (!firstLoop) {\n        try {\n          LOG.info(\"Sleeping \" + waitingTimeMs\n              + \" millis before trying to obtain daily batches again\");\n          Thread.sleep(waitingTimeMs);\n        } catch (InterruptedException e) {\n          LOG.warn(\"Interrupted while waiting to load daily batches\", e);\n          Thread.currentThread().interrupt();\n          break;\n        }\n      }\n\n      if (isStatusBatchLoadingEnabled() && refreshByLoadingHDFSStatusBatches(hdfs)) {\n        LOG.info(\"Successfully loaded daily status batches after {}\", stopwatch);\n        return true;\n      }\n\n      final AtomicBoolean successRef = new AtomicBoolean(false);\n      if (computeDailyBatchesWithZKLock(hdfs, successRef, stopwatch)) {\n        return successRef.get();\n      }\n\n      timeSpentOnDailyBatches = stopwatch.elapsed(TimeUnit.MILLISECONDS);\n      firstLoop = false;\n    }\n\n    return false;\n  }\n\n  private boolean computeDailyBatchesWithZKLock(final FileSystem hdfs,\n                                                final AtomicBoolean successRef,\n                                                final Stopwatch stopwatch) throws IOException {\n    // Using a global lock to coordinate among earlybirds and segment builders so that only\n    // one instance would hit the HDFS name node to query the daily status directories\n    TryLock lock = zkTryLockFactory.createTryLock(\n        DatabaseConfig.getLocalHostname(),\n        DAILY_STATUS_BATCHES_SYNC_PATH,\n        DAILY_BATCHES_ZK_LOCK,\n        DAILY_STATUS_BATCHES_ZK_LOCK_EXPIRATION_MINUTES);\n\n    return lock.tryWithLock(() -> {\n      LOG.info(\"Obtained ZK lock to compute daily status batches after {}\", stopwatch);\n      successRef.set(initialLoadDailyBatchInfos(hdfs));\n      if (successRef.get()) {\n        LOG.info(\"Successfully computed daily status batches after {}\", stopwatch);\n        if (isStatusBatchFlushingEnabled()) {\n          LOG.info(\"Starting to store daily status batches to HDFS\");\n          if (storeStatusBatchesToHdfs(hdfs, lastValidDay)) {\n            LOG.info(\"Successfully stored daily status batches to HDFS\");\n          } else {\n            LOG.warn(\"Failed storing daily status batches to HDFS\");\n          }\n        }\n      } else {\n        LOG.info(\"Failed loading daily status info\");\n      }\n    });\n  }\n\n  private void verifyDirectory(FileSystem hdfs) throws IOException {\n    if (!hdfs.exists(rootPath)) {\n      throw new IOException(\"Root dir '\" + rootPath + \"' does not exist.\");\n    }\n\n    if (!hdfs.exists(buildGenPath)) {\n      throw new IOException(\"Build gen dir '\" + buildGenPath + \"' does not exist.\");\n    }\n\n    if (!hdfs.exists(statusPath)) {\n      throw new IOException(\"Status dir '\" + statusPath + \"' does not exist.\");\n    }\n  }\n\n  private void loadNewDailyBatches(FileSystem hdfs) throws IOException {\n    Preconditions.checkNotNull(lastValidDay);\n\n    Calendar day = Calendar.getInstance();\n    day.setTime(lastValidDay);\n    day.add(Calendar.DATE, 1);\n\n    while (loadDay(hdfs, day.getTime()) != null) {\n      lastValidDay = day.getTime();\n      day.add(Calendar.DATE, 1);\n    }\n  }\n\n  private boolean initialLoadDailyBatchInfos(FileSystem hdfs) throws IOException {\n    LOG.info(\"Starting to build timeslice map from scratch.\");\n\n    final Date lastValidStatusDay = getLastValidInputDateFromNow(hdfs);\n\n    if (lastValidStatusDay == null) {\n      LOG.warn(\"No data found in \" + statusPath + \" and scrubbed path\");\n      return false;\n    }\n    int mostRecentYear = DateUtil.getCalendar(lastValidStatusDay).get(Calendar.YEAR);\n    for (int year = 2006; year <= mostRecentYear; ++year) {\n      // construct path to avoid hdfs.listStatus() calls\n      Calendar day = Calendar.getInstance();\n      day.set(year, Calendar.JANUARY, 1, 0, 0, 0);\n      day.set(Calendar.MILLISECOND, 0);\n\n      Calendar yearEnd = Calendar.getInstance();\n      yearEnd.set(year, Calendar.DECEMBER, 31, 0, 0, 0);\n      yearEnd.set(Calendar.MILLISECOND, 0);\n\n      if (lastValidDay != null) {\n        // We're updating.\n        if (lastValidDay.after(yearEnd.getTime())) {\n          // This year was already loaded.\n          continue;\n        }\n        if (lastValidDay.after(day.getTime())) {\n          // Start one day after last valid date.\n          day.setTime(lastValidDay);\n          day.add(Calendar.DATE, 1);\n        }\n      }\n\n      for (; !day.after(yearEnd); day.add(Calendar.DATE, 1)) {\n        loadDay(hdfs, day.getTime());\n      }\n    }\n\n    boolean updated = false;\n    numberOfDaysWithValidScrubGenData = 0;\n\n    // Iterate batches in sorted order.\n    for (DailyStatusBatch batch : statusBatches.values()) {\n      if (!batch.isValid()) {\n        break;\n      }\n      if (batch.getDate().before(scrubGenDay)) {\n        numberOfDaysWithValidScrubGenData++;\n      }\n      if (firstValidDay == null) {\n        firstValidDay = batch.getDate();\n      }\n      if (lastValidDay == null || lastValidDay.before(batch.getDate())) {\n        lastValidDay = batch.getDate();\n        updated = true;\n      }\n    }\n\n    LOG.info(\"Number of statusBatches: {}\", statusBatches.size());\n    return updated;\n  }\n\n  private static String filesToString(FileStatus[] files) {\n    if (files == null) {\n      return \"null\";\n    }\n    StringBuilder b = new StringBuilder();\n    for (FileStatus s : files) {\n      b.append(s.getPath().toString()).append(\", \");\n    }\n    return b.toString();\n  }\n\n  @VisibleForTesting\n  protected DailyStatusBatch loadDay(FileSystem hdfs, Date day) throws IOException {\n    Path dayPath = new Path(getStatusPathToUseForDay(day), ArchiveHDFSUtils.dateToPath(day, \"/\"));\n    LOG.debug(\"Looking for batch in \" + dayPath.toString());\n    DailyStatusBatch result = this.statusBatches.get(day);\n    if (result != null) {\n      return result;\n    }\n\n    final FileStatus[] files;\n    try {\n      files = hdfs.listStatus(dayPath);\n      LOG.debug(\"Files found:  \" + filesToString(files));\n    } catch (FileNotFoundException e) {\n      LOG.debug(\"loadDay() called, but directory does not exist for day: \" + day\n          + \" in: \" + dayPath);\n      return null;\n    }\n\n    if (files != null && files.length > 0) {\n      for (FileStatus file : files) {\n        Matcher matcher = HASH_PARTITION_PATTERN.matcher(file.getPath().getName());\n        if (matcher.matches()) {\n          int numHashPartitions = Integer.parseInt(matcher.group(2));\n          result = new DailyStatusBatch(\n              day, numHashPartitions, getStatusPathToUseForDay(day), hdfs);\n\n          for (int partitionID = 0; partitionID < numHashPartitions; partitionID++) {\n            result.addPartition(hdfs, dayPath, partitionID);\n          }\n\n          if (result.isValid()) {\n            statusBatches.put(day, result);\n            return result;\n          } else {\n            LOG.info(\"Invalid batch found for day: \" + day + \", batch: \" + result);\n          }\n        } else {\n          // skip logging the intermediate count subdirectories or _SUCCESS files.\n          if (!INTERMEDIATE_COUNTS_SUBDIR_NAME.equals(file.getPath().getName())\n              && !SUCCESS_FILE_NAME.equals(file.getPath().getName())) {\n            LOG.warn(\"Path does not match hash partition pattern: \" + file.getPath());\n          }\n        }\n      }\n    } else {\n      LOG.warn(\"No data found for day: \" + day + \" in: \" + dayPath\n              + \" files null: \" + (files == null));\n    }\n\n    return null;\n  }\n\n  /**\n   * Determines if this directory has a valid batch for the given day.\n   */\n  public boolean hasValidBatchForDay(Date day) throws IOException {\n    FileSystem hdfs = null;\n    try {\n      hdfs = HdfsUtil.getHdfsFileSystem();\n      return hasValidBatchForDay(hdfs, day);\n    } finally {\n      IOUtils.closeQuietly(hdfs);\n    }\n  }\n\n  private boolean hasValidBatchForDay(FileSystem fs, Date day) throws IOException {\n    DailyStatusBatch batch = loadDay(fs, day);\n\n    return batch != null && batch.isValid();\n  }\n\n  @VisibleForTesting\n  Date getFirstValidDay() {\n    return firstValidDay;\n  }\n\n  @VisibleForTesting\n  Date getLastValidDay() {\n    return lastValidDay;\n  }\n\n  private Date getLastValidInputDateFromNow(FileSystem hdfs) throws IOException {\n    Calendar cal = Calendar.getInstance();\n    cal.setTime(new Date()); // current date\n    return getLastValidInputDate(hdfs, cal);\n  }\n\n  /**\n   * Starting from current date, probe backward till we find a valid input Date\n   */\n  @VisibleForTesting\n  Date getLastValidInputDate(FileSystem hdfs, Calendar cal) throws IOException {\n    cal.set(Calendar.MILLISECOND, 0);\n    cal.set(Calendar.HOUR_OF_DAY, 0);\n    cal.set(Calendar.MINUTE, 0);\n    cal.set(Calendar.SECOND, 0);\n    cal.set(Calendar.MILLISECOND, 0);\n    Date lastValidInputDate = cal.getTime();\n    LOG.info(\"Probing backwards for last valid data date from \" + lastValidInputDate);\n    while (lastValidInputDate.after(FIRST_TWITTER_DAY)) {\n      if (hasValidBatchForDay(hdfs, lastValidInputDate)) {\n        LOG.info(\"Found latest valid data on date \" + lastValidInputDate);\n        LOG.info(\"  Used path: {}\", getStatusPathToUseForDay(lastValidInputDate));\n        return lastValidInputDate;\n      }\n      cal.add(Calendar.DATE, -1);\n      lastValidInputDate = cal.getTime();\n    }\n\n    return null;\n  }\n\n  /**\n   * Check if the daily status batches are already on HDFS\n   */\n  @VisibleForTesting\n  boolean hasStatusBatchesOnHdfs(FileSystem fs, Date lastDataDay) {\n    String hdfsFileName = getHdfsStatusBatchSyncFileName(lastDataDay);\n    try {\n      return fs.exists(new Path(hdfsFileName));\n    } catch (IOException ex) {\n      LOG.error(\"Failed checking status batch file on HDFS: \" + hdfsFileName, ex);\n      return false;\n    }\n  }\n\n  /**\n   * Load the daily status batches from HDFS by first copying the file from HDFS to local disk\n   * and then reading from the local disk.\n   *\n   * @param day the latest day of valid statuses.\n   * @return true if the loading is successful.\n   */\n  @VisibleForTesting\n  boolean loadStatusBatchesFromHdfs(FileSystem fs, Date day) {\n    // set the directory state to initial state\n    resetDirectory();\n\n    String fileHdfsPath = getHdfsStatusBatchSyncFileName(day);\n    String fileLocalPath = getLocalStatusBatchSyncFileName(day);\n\n    LOG.info(\"Using \" + fileHdfsPath + \" as the HDFS batch summary load path.\");\n    LOG.info(\"Using \" + fileLocalPath + \" as the local batch summary sync path.\");\n\n    LineRecordFileReader lineReader = null;\n    try {\n      fs.copyToLocalFile(new Path(fileHdfsPath), new Path(fileLocalPath));\n\n      lineReader = new LineRecordFileReader(fileLocalPath);\n      String batchLine;\n      while ((batchLine = lineReader.readNext()) != null) {\n        DailyStatusBatch batch = DailyStatusBatch.deserializeFromJson(batchLine);\n        if (batch == null) {\n          LOG.error(\"Invalid daily status batch constructed from line: \" + batchLine);\n          resetDirectory();\n          return false;\n        }\n        Date date = batch.getDate();\n        if (firstValidDay == null || firstValidDay.after(date)) {\n          firstValidDay = date;\n        }\n        if (lastValidDay == null || lastValidDay.before(date)) {\n          lastValidDay = date;\n        }\n        statusBatches.put(date, batch);\n      }\n      LOG.info(\"Loaded {} status batches from HDFS: {}\",\n          statusBatches.size(), fileHdfsPath);\n      LOG.info(\"First entry: {}\", statusBatches.firstEntry().getValue().toString());\n      LOG.info(\"Last entry: {}\", statusBatches.lastEntry().getValue().toString());\n\n      return true;\n    } catch (IOException ex) {\n      LOG.error(\"Failed loading time slices from HDFS: \" + fileHdfsPath, ex);\n      resetDirectory();\n      return false;\n    } finally {\n      if (lineReader != null) {\n        lineReader.stop();\n      }\n    }\n  }\n\n  /**\n   * Flush the daily status batches to local disk and then upload to HDFS.\n   */\n  private boolean storeStatusBatchesToHdfs(FileSystem fs, Date day) {\n    Preconditions.checkNotNull(lastValidDay);\n\n    if (!StatusBatchFlushVersion.CURRENT_FLUSH_VERSION.isOfficial()) {\n      LOG.info(\"Status batch flush version is not official, no batches will be flushed to HDFS\");\n      return true;\n    }\n\n    String fileLocalPath = getLocalStatusBatchSyncFileName(day);\n\n    // Flush to local disk\n    File outputFile = null;\n    FileWriter fileWriter = null;\n    try {\n      LOG.info(\"Flushing daily status batches into: \" + fileLocalPath);\n      outputFile = new File(fileLocalPath);\n      outputFile.getParentFile().mkdirs();\n      if (!outputFile.getParentFile().exists()) {\n        LOG.error(\"Cannot create directory: \" + outputFile.getParentFile().toString());\n        return false;\n      }\n      fileWriter = new FileWriter(outputFile, false);\n      for (Date date : statusBatches.keySet()) {\n        fileWriter.write(statusBatches.get(date).serializeToJson());\n        fileWriter.write(\"\\n\");\n      }\n      fileWriter.flush();\n\n      // Upload the file to HDFS\n      return uploadStatusBatchesToHdfs(fs, day);\n    } catch (IOException e) {\n      String fileHdfsPath = getHdfsStatusBatchSyncFileName(day);\n      LOG.error(\"Failed storing status batches to HDFS: \" + fileHdfsPath, e);\n      return false;\n    } finally {\n      try {\n        if (fileWriter != null) {\n          fileWriter.close();\n        }\n      } catch (IOException e) {\n        LOG.error(\"Error to close fileWrite.\", e);\n      }\n      if (outputFile != null) {\n        // Delete the local file\n        outputFile.delete();\n      }\n    }\n  }\n\n  /**\n   * Upload the status batches to HDFS.\n   */\n  @VisibleForTesting\n  boolean uploadStatusBatchesToHdfs(FileSystem fs, Date day) {\n    String localFileName = getLocalStatusBatchSyncFileName(day);\n    String hdfsFileName = getHdfsStatusBatchSyncFileName(day);\n\n    LOG.info(\"Using \" + hdfsFileName + \" as the HDFS batch summary upload path.\");\n    LOG.info(\"Using \" + localFileName + \" as the local batch summary sync path.\");\n\n    try {\n      Path hdfsFilePath = new Path(hdfsFileName);\n      if (fs.exists(hdfsFilePath)) {\n        LOG.warn(\"Found status batch file on HDFS: \" + hdfsFileName);\n        return true;\n      }\n\n      String hdfsTempName = getHdfsStatusBatchTempSyncFileName(day);\n      Path hdfsTempPath = new Path(hdfsTempName);\n      if (fs.exists(hdfsTempPath)) {\n        LOG.info(\"Found existing temporary status batch file on HDFS, removing: \" + hdfsTempName);\n        if (!fs.delete(hdfsTempPath, false)) {\n          LOG.error(\"Failed to delete temporary file: \" + hdfsTempName);\n          return false;\n        }\n      }\n      fs.copyFromLocalFile(new Path(localFileName), hdfsTempPath);\n\n      if (fs.rename(hdfsTempPath, hdfsFilePath)) {\n        LOG.debug(\"Renamed \" + hdfsTempName + \" on HDFS to: \" + hdfsFileName);\n        return true;\n      } else {\n        LOG.error(\"Failed to rename \" + hdfsTempName + \" on HDFS to: \" + hdfsFileName);\n        return false;\n      }\n    } catch (IOException ex) {\n      LOG.error(\"Failed uploading status batch file to HDFS: \" + hdfsFileName, ex);\n      return false;\n    }\n  }\n\n  private static boolean isStatusBatchFlushingEnabled() {\n    return EarlybirdProperty.ARCHIVE_DAILY_STATUS_BATCH_FLUSHING_ENABLED.get(false);\n  }\n\n  private static boolean isStatusBatchLoadingEnabled() {\n    return EarlybirdConfig.getBool(\"archive_daily_status_batch_loading_enabled\", false);\n  }\n\n  private static String getVersionFileExtension() {\n    return StatusBatchFlushVersion.CURRENT_FLUSH_VERSION.getVersionFileExtension();\n  }\n\n  String getStatusBatchSyncRootDir() {\n    return EarlybirdConfig.getString(\"archive_daily_status_batch_sync_dir\",\n        \"daily_status_batches\") + \"/\" + scrubGenSuffix();\n  }\n\n  @VisibleForTesting\n  String getLocalStatusBatchSyncFileName(Date day) {\n    return  getStatusBatchSyncRootDir() + \"/\" + STATUS_BATCHES_PREFIX + \"_\"\n        + DATE_FORMAT.format(day) + getVersionFileExtension();\n  }\n\n  String getHdfsStatusBatchSyncRootDir() {\n    return EarlybirdConfig.getString(\"hdfs_archive_daily_status_batch_sync_dir\",\n        \"daily_status_batches\") + \"/\" + scrubGenSuffix();\n  }\n\n  @VisibleForTesting\n  String getHdfsStatusBatchSyncFileName(Date day) {\n    return getHdfsStatusBatchSyncRootDir() + \"/\" + STATUS_BATCHES_PREFIX + \"_\"\n        + DATE_FORMAT.format(day) + getVersionFileExtension();\n  }\n\n  private String getHdfsStatusBatchTempSyncFileName(Date day) {\n    return getHdfsStatusBatchSyncRootDir() + \"/\" + DatabaseConfig.getLocalHostname() + \"_\"\n        + STATUS_BATCHES_PREFIX + \"_\" + DATE_FORMAT.format(day) + getVersionFileExtension();\n  }\n\n  private String scrubGenSuffix() {\n    return String.format(SCRUB_GEN_SUFFIX_PATTERN, DATE_FORMAT.format(scrubGenDay));\n  }\n\n  /**\n   * Returns the path to the directory that stores the statuses for the given day.\n   */\n  public Path getStatusPathToUseForDay(Date day) {\n    if (!day.before(scrubGenDay)) {\n      return statusPath;\n    }\n\n    String suffix = scrubGenSuffix();\n    Preconditions.checkArgument(!suffix.isEmpty());\n    Path scrubPath = new Path(buildGenPath, suffix);\n    return new Path(scrubPath, STATUS_SUBDIR_NAME);\n  }\n\n  /**\n   * Determines if the data for the specified scrub gen was fully built, by checking the number of\n   * days for which data was built against the expected number of days extracted from the specified\n   * scrub gen date.\n   */\n  public boolean isScrubGenDataFullyBuilt(FileSystem hdfs) throws IOException {\n    initialLoadDailyBatchInfos(hdfs);\n    if (numberOfDaysWithValidScrubGenData == 0) {\n      LOG.warn(\"numberOfDaysWithValidScrubGenData is 0\");\n    }\n    long expectedDays = getDiffBetweenDays(scrubGenDay);\n    return expectedDays == numberOfDaysWithValidScrubGenData;\n  }\n\n  @VisibleForTesting\n  long getDiffBetweenDays(Date day) {\n    long diff = day.getTime() - FIRST_TWEET_DAY.getTime();\n    return TimeUnit.DAYS.convert(diff, TimeUnit.MILLISECONDS);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/PartitionedBatch.java",
    "content": "package com.twitter.search.earlybird.archive;\n\nimport java.io.FileNotFoundException;\nimport java.io.IOException;\nimport java.util.Comparator;\nimport java.util.Date;\nimport java.util.List;\nimport java.util.concurrent.TimeUnit;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Function;\nimport com.google.common.base.Predicate;\nimport com.google.common.collect.ComparisonChain;\nimport com.google.common.collect.Lists;\n\nimport org.apache.commons.io.IOUtils;\nimport org.apache.hadoop.fs.FileStatus;\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.Path;\nimport org.apache.hadoop.fs.PathFilter;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.search.common.schema.earlybird.EarlybirdThriftDocumentUtil;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.util.date.DateUtil;\nimport com.twitter.search.common.util.io.EmptyRecordReader;\nimport com.twitter.search.common.util.io.LzoThriftBlockFileReader;\nimport com.twitter.search.common.util.io.MergingSortedRecordReader;\nimport com.twitter.search.common.util.io.TransformingRecordReader;\nimport com.twitter.search.common.util.io.recordreader.RecordReader;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.document.DocumentFactory;\nimport com.twitter.search.earlybird.document.TweetDocument;\nimport com.twitter.search.earlybird.partition.HdfsUtil;\n\n/**\n * A batch of pre-processed tweets for a single hash partition from a particular day.\n */\npublic class PartitionedBatch {\n  private static final Logger LOG = LoggerFactory.getLogger(PartitionedBatch.class);\n  private static final Date START_DATE_INCLUSIVE = DateUtil.toDate(2006, 03, 21);\n  private static final String STATUS_COUNT_FILE_PREFIX = \"_status_count_\";\n  private static final Pattern STATUS_COUNT_FILE_PATTERN =\n      Pattern.compile(STATUS_COUNT_FILE_PREFIX + \"(\\\\d+)_minid_(\\\\d+)_maxid_(\\\\d+)\");\n  private static final int MAXIMUM_OUT_OF_ORDER_TOLERANCE_HOURS =\n      EarlybirdConfig.getInt(\"archive_max_out_of_order_tolerance_hours\", 12);\n  private static final int READER_INIT_IOEXCEPTION_RETRIES = 20;\n  private static final PathFilter LZO_DATA_FILES_FILTER = file -> file.getName().endsWith(\".lzo\");\n  private static final PathFilter TXT_DATA_FILES_FILTER = file -> file.getName().endsWith(\".txt\");\n\n  private static final Comparator<ThriftIndexingEvent> DESC_THRIFT_INDEXING_EVENT_COMPARATOR =\n      (o1, o2) -> ComparisonChain.start()\n          .compare(o2.getSortId(), o1.getSortId())\n          .compare(o2.getUid(), o1.getUid())\n          .result();\n\n  // Number archive tweets skipped because they are too out-of-order.\n  private static final SearchCounter OUT_OF_ORDER_STATUSES_SKIPPED =\n      SearchCounter.export(\"out_of_order_archive_statuses_skipped\");\n\n  @VisibleForTesting\n  protected static final long MAXIMUM_OUT_OF_ORDER_TOLERANCE_MILLIS =\n      TimeUnit.HOURS.toMillis(MAXIMUM_OUT_OF_ORDER_TOLERANCE_HOURS);\n\n  private final Date date;\n  private final Path path;\n  private int statusCount;\n  private long minStatusID;\n  private long maxStatusID;\n  private final int hashPartitionID;\n  private boolean hasStatusCountFile;\n  private final int numHashPartitions;\n\n  @VisibleForTesting\n  public PartitionedBatch(\n      Path path,\n      int hashPartitionID,\n      int numHashPartitions,\n      Date date) {\n    this.path = path;\n    this.hashPartitionID = hashPartitionID;\n    this.numHashPartitions = numHashPartitions;\n    this.date = date;\n  }\n\n  /**\n   * Loads all the information (tweet count, etc.) for this partition and day from HDFS.\n   */\n  public void load(FileSystem hdfs) throws IOException {\n    FileStatus[] dailyBatchFiles = null;\n    try {\n      // listStatus() javadoc says it throws FileNotFoundException when path does not exist.\n      // However, the actual implementations return null or an empty array instead.\n      // We handle all 3 cases: null, empty array, or FileNotFoundException.\n      dailyBatchFiles = hdfs.listStatus(path);\n    } catch (FileNotFoundException e) {\n      // don't do anything here and the day will be handled as empty.\n    }\n\n    if (dailyBatchFiles != null && dailyBatchFiles.length > 0) {\n      for (FileStatus file : dailyBatchFiles) {\n        String fileName = file.getPath().getName();\n        if (fileName.equals(STATUS_COUNT_FILE_PREFIX)) {\n          // zero tweets in this partition - this can happen for early days in 2006\n          handleEmptyPartition();\n        } else {\n          Matcher matcher = STATUS_COUNT_FILE_PATTERN.matcher(fileName);\n          if (matcher.matches()) {\n            try {\n              statusCount = Integer.parseInt(matcher.group(1));\n              // Only adjustMinStatusId in production. For tests, this makes the tests harder to\n              // understand.\n              minStatusID = Config.environmentIsTest() ? Long.parseLong(matcher.group(2))\n                  : adjustMinStatusId(Long.parseLong(matcher.group(2)), date);\n              maxStatusID = Long.parseLong(matcher.group(3));\n              hasStatusCountFile = true;\n            } catch (NumberFormatException e) {\n              // invalid file - ignore\n              LOG.warn(\"Could not parse status count file name.\", e);\n            }\n          }\n        }\n      }\n    } else {\n      // Partition folder does not exist. This case can happen for early days of twitter\n      // where some partitions are empty. Set us to having a status count file, the validity of\n      // the parent DailyStatusBatch will still be determined by whether there was a _SUCCESS file\n      // in the day root.\n      handleEmptyPartition();\n\n      if (date.after(getEarliestDenseDay())) {\n        LOG.error(\"Unexpected empty directory {} for {}\", path, date);\n      }\n    }\n  }\n\n  private void handleEmptyPartition() {\n    statusCount = 0;\n    minStatusID = DailyStatusBatch.EMPTY_BATCH_STATUS_ID;\n    maxStatusID = DailyStatusBatch.EMPTY_BATCH_STATUS_ID;\n    hasStatusCountFile = true;\n  }\n\n  /**\n   * Sometimes tweets are out-of-order (E.g. a tweet from Sep 2012 got into a\n   * batch in July 2013). See SEARCH-1750 for more details.\n   * This adjust the minStatusID if it is badly out-of-order.\n   */\n  @VisibleForTesting\n  protected static long adjustMinStatusId(long minStatusID, Date date) {\n    long dateTime = date.getTime();\n    // If the daily batch is for a day before we started using snow flake IDs. Never adjust.\n    if (!SnowflakeIdParser.isUsableSnowflakeTimestamp(dateTime)) {\n      return minStatusID;\n    }\n\n    long earliestStartTime = dateTime - MAXIMUM_OUT_OF_ORDER_TOLERANCE_MILLIS;\n    long minStatusTime = SnowflakeIdParser.getTimestampFromTweetId(minStatusID);\n    if (minStatusTime < earliestStartTime) {\n      long newMinId =  SnowflakeIdParser.generateValidStatusId(earliestStartTime, 0);\n      LOG.info(\"Daily batch for \" + date + \" has badly out of order tweet: \" + minStatusID\n          + \". The minStatusID for the day this batch is adjusted to \" + newMinId);\n      return newMinId;\n    } else {\n      return minStatusID;\n    }\n  }\n\n  /**\n   * Returns a reader that reads tweets from the given directory.\n   *\n   * @param archiveSegment Determines the timeslice ID of all read tweets.\n   * @param tweetsPath The path to the directory where the tweets for this day are stored.\n   * @param documentFactory The ThriftIndexingEvent to TweetDocument converter.\n   */\n  public RecordReader<TweetDocument> getTweetReaders(\n      ArchiveSegment archiveSegment,\n      Path tweetsPath,\n      DocumentFactory<ThriftIndexingEvent> documentFactory) throws IOException {\n    RecordReader<TweetDocument> tweetDocumentReader =\n        new TransformingRecordReader<>(\n            createTweetReader(tweetsPath), new Function<ThriftIndexingEvent, TweetDocument>() {\n          @Override\n          public TweetDocument apply(ThriftIndexingEvent event) {\n            return new TweetDocument(\n                event.getSortId(),\n                archiveSegment.getTimeSliceID(),\n                EarlybirdThriftDocumentUtil.getCreatedAtMs(event.getDocument()),\n                documentFactory.newDocument(event)\n            );\n          }\n        });\n\n    tweetDocumentReader.setExhaustStream(true);\n    return tweetDocumentReader;\n  }\n\n  private RecordReader<ThriftIndexingEvent> createTweetReader(Path tweetsPath) throws IOException {\n    if (date.before(START_DATE_INCLUSIVE)) {\n      return new EmptyRecordReader<>();\n    }\n\n    List<RecordReader<ThriftIndexingEvent>> readers = Lists.newArrayList();\n    FileSystem hdfs = HdfsUtil.getHdfsFileSystem();\n    try {\n      Path dayPath = new Path(tweetsPath, ArchiveHDFSUtils.dateToPath(date, \"/\"));\n      Path partitionPath =\n          new Path(dayPath, String.format(\"p_%d_of_%d\", hashPartitionID, numHashPartitions));\n      PathFilter pathFilter =\n          Config.environmentIsTest() ? TXT_DATA_FILES_FILTER : LZO_DATA_FILES_FILTER;\n      FileStatus[] files = hdfs.listStatus(partitionPath, pathFilter);\n      for (FileStatus fileStatus : files) {\n        String fileStatusPath = fileStatus.getPath().toString().replaceAll(\"file:/\", \"/\");\n        RecordReader<ThriftIndexingEvent> reader = createRecordReaderWithRetries(fileStatusPath);\n        readers.add(reader);\n      }\n    } finally {\n      IOUtils.closeQuietly(hdfs);\n    }\n\n    if (readers.isEmpty()) {\n      return new EmptyRecordReader<>();\n    }\n\n    return new MergingSortedRecordReader<>(DESC_THRIFT_INDEXING_EVENT_COMPARATOR, readers);\n  }\n\n  private RecordReader<ThriftIndexingEvent> createRecordReaderWithRetries(String filePath)\n      throws IOException {\n    Predicate<ThriftIndexingEvent> recordFilter = getRecordFilter();\n    int numTries = 0;\n    while (true) {\n      try {\n        ++numTries;\n        return new LzoThriftBlockFileReader<>(filePath, ThriftIndexingEvent.class, recordFilter);\n      } catch (IOException e) {\n        if (numTries < READER_INIT_IOEXCEPTION_RETRIES) {\n          LOG.warn(\"Failed to open LzoThriftBlockFileReader for \" + filePath + \". Will retry.\", e);\n        } else {\n          LOG.error(\"Failed to open LzoThriftBlockFileReader for \" + filePath\n              + \" after too many retries.\", e);\n          throw e;\n        }\n      }\n    }\n  }\n\n  private Predicate<ThriftIndexingEvent> getRecordFilter() {\n    return Config.environmentIsTest() ? null : input -> {\n      if (input == null) {\n        return false;\n      }\n      // We only guard against status IDs that are too small, because it is possible\n      // for a very old tweet to get into today's batch, but not possible for a very\n      // large ID (a future tweet ID that is not yet published) to get in today's\n      // batch, unless tweet ID generation messed up.\n      long statusId = input.getSortId();\n      boolean keep = statusId >= minStatusID;\n      if (!keep) {\n        LOG.debug(\"Out of order documentId: {} minStatusID: {} Date: {} Path: {}\",\n            statusId, minStatusID, date, path);\n        OUT_OF_ORDER_STATUSES_SKIPPED.increment();\n      }\n      return keep;\n    };\n  }\n\n  /**\n   * Returns the number of statuses in this batch\n   */\n  public int getStatusCount() {\n    return statusCount;\n  }\n\n  /**\n   * Was the _status_count file was found in this folder.\n   */\n  public boolean hasStatusCount() {\n    return hasStatusCountFile;\n  }\n\n  public long getMinStatusID() {\n    return minStatusID;\n  }\n\n  public long getMaxStatusID() {\n    return maxStatusID;\n  }\n\n  public Date getDate() {\n    return date;\n  }\n\n  public Path getPath() {\n    return path;\n  }\n\n  /**\n   * Check whether the partition is\n   * . empty and\n   * . it is disallowed (empty partition can only happen before 2010)\n   * (Empty partition means that the directory is missing when scan happens.)\n   *\n   * @return true if the partition has no documents and it is not allowed.\n   */\n  public boolean isDisallowedEmptyPartition() {\n    return hasStatusCountFile\n        && statusCount == 0\n        && minStatusID == DailyStatusBatch.EMPTY_BATCH_STATUS_ID\n        && maxStatusID == DailyStatusBatch.EMPTY_BATCH_STATUS_ID\n        && date.after(getEarliestDenseDay());\n  }\n\n  @Override\n  public String toString() {\n    return \"PartitionedBatch[hashPartitionId=\" + hashPartitionID\n        + \",numHashPartitions=\" + numHashPartitions\n        + \",date=\" + date\n        + \",path=\" + path\n        + \",hasStatusCountFile=\" + hasStatusCountFile\n        + \",statusCount=\" + statusCount + \"]\";\n  }\n\n  private Date getEarliestDenseDay() {\n    return EarlybirdConfig.getDate(\"archive_search_earliest_dense_day\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/BUILD.bazel",
    "content": "java_library(\n    name = \"segment_builder_lib\",\n    sources = [\"**/*.java\"],\n    platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/org/apache/bookkeeper:bookkeeper-server\",\n        \"3rdparty/jvm/org/apache/bookkeeper:bookkeeper-twitter-science-provider\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"decider/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-server/src/main/scala/com/twitter/inject/server\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/quantity\",\n        \"src/java/com/twitter/common/util:system-mocks\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/search/common/config\",\n        \"src/java/com/twitter/search/common/database\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/partitioning/base\",\n        \"src/java/com/twitter/search/common/partitioning/zookeeper\",\n        \"src/java/com/twitter/search/common/schema\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/util:closeresourceutil\",\n        \"src/java/com/twitter/search/common/util:gcutil\",\n        \"src/java/com/twitter/search/common/util:kerberos\",\n        \"src/java/com/twitter/search/common/util/date\",\n        \"src/java/com/twitter/search/common/util/io:flushable\",\n        \"src/java/com/twitter/search/common/util/zktrylock\",\n        \"src/java/com/twitter/search/common/util/zookeeper\",\n        \"src/java/com/twitter/search/earlybird:earlybird-lib\",\n        \"src/java/com/twitter/search/earlybird/common\",\n        \"src/java/com/twitter/search/earlybird/common/config\",\n        \"src/java/com/twitter/search/earlybird/common/userupdates\",\n        \"util/util-core:scala\",\n    ],\n)\n\n# Using hadoop_binary target can automatically exclude hadoop related jars in the built jar\n# and load in the right jars based on hadoop config.\nhadoop_binary(\n    name = \"segment_builder_binary\",\n    basename = \"segment_builder\",\n    main = \"com.twitter.search.earlybird.archive.segmentbuilder.SegmentBuilderMain\",\n    platform = \"java8\",\n    runtime_platform = \"java8\",\n    tags = [\n        \"bazel-compatible\",\n        \"bazel-compatible:migrated\",\n        \"bazel-only\",\n    ],\n    dependencies = [\n        \":segment_builder_lib\",\n        \"src/java/com/twitter/search/common/logging:search-log4j\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/BuiltAndFinalizedSegment.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\nimport com.twitter.search.earlybird.index.EarlybirdSegmentFactory;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\n\npublic class BuiltAndFinalizedSegment extends SegmentBuilderSegment {\n  public BuiltAndFinalizedSegment(\n      SegmentInfo segmentInfo,\n      SegmentConfig segmentConfig,\n      EarlybirdSegmentFactory earlybirdSegmentFactory,\n      int alreadyRetriedCount,\n      SegmentSyncConfig sync) {\n\n    super(segmentInfo, segmentConfig, earlybirdSegmentFactory, alreadyRetriedCount, sync);\n  }\n\n  @Override\n  public SegmentBuilderSegment handle() throws SegmentInfoConstructionException,\n      SegmentUpdaterException {\n\n    throw new IllegalStateException(\"Should not handle a BuildAndFinalizedSegment.\");\n  }\n\n  @Override\n  public boolean isBuilt() {\n    return true;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/NotYetBuiltSegment.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\nimport java.util.concurrent.atomic.AtomicBoolean;\n\nimport com.google.common.base.Stopwatch;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.util.GCUtil;\nimport com.twitter.search.common.util.zktrylock.TryLock;\nimport com.twitter.search.earlybird.archive.ArchiveSegmentUpdater;\nimport com.twitter.search.earlybird.index.EarlybirdSegmentFactory;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\n\npublic class NotYetBuiltSegment extends SegmentBuilderSegment {\n  private static final Logger LOG = LoggerFactory.getLogger(NotYetBuiltSegment.class);\n\n  public NotYetBuiltSegment(\n      SegmentInfo segmentInfo,\n      SegmentConfig segmentConfig,\n      EarlybirdSegmentFactory earlybirdSegmentFactory,\n      int alreadyRetriedCount,\n      SegmentSyncConfig sync) {\n\n    super(segmentInfo, segmentConfig, earlybirdSegmentFactory, alreadyRetriedCount, sync);\n  }\n\n  /**\n   * 1. Grab the ZK lock for this segment.\n   *   2a. if lock fails, another host is updating; return the SOMEONE_ELSE_IS_BUILDING state.\n   *   2b. if lock succeeds, check again if the updated segment exists on HDFS.\n   *     3a. if so, just move on.\n   *     3b. if not, update the segment.\n   *     In both cases, we need to check if the segment can now be marked as BUILT_AND_FINALIZED.\n   */\n  @Override\n  public SegmentBuilderSegment handle()\n      throws SegmentUpdaterException, SegmentInfoConstructionException {\n    LOG.info(\"Handling a not yet built segment: {}\", this.getSegmentName());\n    Stopwatch stopwatch = Stopwatch.createStarted();\n    TryLock lock = getZooKeeperTryLock();\n\n    // The tryWithLock can only access variables from parent class that are final. However, we\n    // would like to pass the process() return value to the parent class. So here we use\n    // AtomicBoolean reference instead of Boolean.\n    final AtomicBoolean successRef = new AtomicBoolean(false);\n    boolean gotLock = lock.tryWithLock(() -> {\n      ArchiveSegmentUpdater updater = new ArchiveSegmentUpdater(\n          segmentConfig.getTryLockFactory(),\n          sync,\n          segmentConfig.getEarlybirdIndexConfig(),\n          Clock.SYSTEM_CLOCK);\n\n      boolean success = updater.updateSegment(segmentInfo);\n      successRef.set(success);\n    });\n\n    if (!gotLock) {\n      LOG.info(\"cannot acquire zookeeper lock for: \" + segmentInfo);\n      return new SomeoneElseIsBuildingSegment(\n          segmentInfo,\n          segmentConfig,\n          earlybirdSegmentFactory,\n          alreadyRetriedCount,\n          sync);\n    }\n\n    // 1. we want to make sure the heap is clean right after building a segment so that it's ready\n    //   for us to start allocations for a new segment\n    // — I think we've had cases where we were seeing OOM's while building\n    // 2. the thing that I think it helps with is compaction (vs just organically running CMS)\n    // — which would clean up the heap, but may leave it in a fragmented state\n    // — and running a Full GC is supposed to compact the remaining tenured space.\n    GCUtil.runGC();\n\n    if (successRef.get()) {\n      LOG.info(\"Indexing segment {} took {}\", segmentInfo, stopwatch);\n      LOG.info(\"Finished building {}\", segmentInfo.getSegment().getSegmentName());\n      return new BuiltAndFinalizedSegment(\n          segmentInfo, segmentConfig, earlybirdSegmentFactory, 0, sync);\n    } else {\n      int alreadyTried = alreadyRetriedCount + 1;\n      String errMsg = \"failed updating segment for: \" + segmentInfo\n          + \" for \" + alreadyTried + \" times\";\n      LOG.error(errMsg);\n      if (alreadyTried < segmentConfig.getMaxRetriesOnFailure()) {\n        return new NotYetBuiltSegment(\n            createNewSegmentInfo(segmentInfo),\n            segmentConfig,\n            earlybirdSegmentFactory,\n            alreadyTried,\n            sync);\n      } else {\n        throw new SegmentUpdaterException(errMsg);\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/RateLimitingSegmentHandler.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\nimport java.util.HashMap;\nimport java.util.Map;\n\nimport com.twitter.common.util.Clock;\n\n/**\n * A class that prevents handling a given segment more than once every hdfsCheckIntervalMillis\n */\npublic class RateLimitingSegmentHandler {\n  private final long hdfsCheckIntervalMillis;\n  private final Clock clock;\n  private final Map<String, Long> segmentNameToLastUpdatedTimeMillis = new HashMap<>();\n\n  RateLimitingSegmentHandler(long hdfsCheckIntervalMillis, Clock clock) {\n    this.hdfsCheckIntervalMillis = hdfsCheckIntervalMillis;\n    this.clock = clock;\n  }\n\n  SegmentBuilderSegment processSegment(SegmentBuilderSegment segment)\n      throws SegmentUpdaterException, SegmentInfoConstructionException {\n\n    String segmentName = segment.getSegmentName();\n\n    Long lastUpdatedMillis = segmentNameToLastUpdatedTimeMillis.get(segmentName);\n    if (lastUpdatedMillis == null) {\n      lastUpdatedMillis = 0L;\n    }\n\n    long nowMillis = clock.nowMillis();\n    if (nowMillis - lastUpdatedMillis < hdfsCheckIntervalMillis) {\n      return segment;\n    }\n    segmentNameToLastUpdatedTimeMillis.put(segmentName, nowMillis);\n\n    return segment.handle();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/SegmentBuilder.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.Date;\nimport java.util.HashMap;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Optional;\nimport java.util.Random;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Stopwatch;\nimport com.google.common.collect.ComparisonChain;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.util.concurrent.Uninterruptibles;\nimport com.google.inject.Inject;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.quantity.Amount;\nimport com.twitter.common.quantity.Time;\nimport com.twitter.common.util.Clock;\nimport com.twitter.decider.Decider;\nimport com.twitter.inject.annotations.Flag;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.metrics.SearchStatsReceiverImpl;\nimport com.twitter.search.common.partitioning.zookeeper.SearchZkClient;\nimport com.twitter.search.common.util.Kerberos;\nimport com.twitter.search.common.util.zktrylock.ZooKeeperTryLockFactory;\nimport com.twitter.search.earlybird.archive.ArchiveOnDiskEarlybirdIndexConfig;\nimport com.twitter.search.earlybird.archive.ArchiveSegment;\nimport com.twitter.search.earlybird.archive.DailyStatusBatches;\nimport com.twitter.search.earlybird.archive.ArchiveTimeSlicer;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.util.ScrubGenUtil;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.index.EarlybirdSegmentFactory;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\n\n/**\n * This class provides the core logic to build segment indices offline.\n * For each server, it coordinate via zookeeper to pick the next segment, build the indices for it\n * and upload them to HDFS. A state machine is used to handle the build state transitions. There\n * are three states:\n *  NOT_BUILD_YET: a segment that needs to be built\n *  SOMEONE_ELSE_IS_BUILDING: another server is building the segment.\n *  BUILT_AND_FINALIZED: the indices of this segment have already been built.\n */\npublic class SegmentBuilder {\n  private static final Logger LOG = LoggerFactory.getLogger(SegmentBuilder.class);\n\n  private final boolean onlyRunOnce;\n  private final int waitBetweenLoopsMins;\n  private final int startUpBatchSize;\n  private final int instance;\n  private final int waitBetweenSegmentsSecs;\n  private final int waitBeforeQuitMins;\n\n  // When multiple segment builders start simultaneously, they might make the HDFS name node and\n  // zookeeper overwhelmed. So, we let some instances sleep sometimes before they start to avoid\n  // the issues.\n  private final long startUpSleepMins;\n\n  // If no more segments to built, wait this interval before checking again.\n  private final long processWaitingInterval = TimeUnit.MINUTES.toMillis(10);\n\n  // The hash partitions that segments will be built.\n  private final ImmutableList<Integer> hashPartitions;\n\n  private final SearchStatsReceiver statsReceiver = new SearchStatsReceiverImpl();\n  private final SearchIndexingMetricSet searchIndexingMetricSet =\n      new SearchIndexingMetricSet(statsReceiver);\n  private final EarlybirdSearcherStats searcherStats =\n      new EarlybirdSearcherStats(statsReceiver);\n\n  private final ArchiveOnDiskEarlybirdIndexConfig earlybirdIndexConfig;\n\n  private final ZooKeeperTryLockFactory zkTryLockFactory;\n  private final RateLimitingSegmentHandler segmentHandler;\n  private final Clock clock;\n  private final int numSegmentBuilderPartitions;\n  private final int myPartitionId;\n  private final SegmentConfig segmentConfig;\n  private final EarlybirdSegmentFactory segmentFactory;\n  private final SegmentBuilderCoordinator segmentBuilderCoordinator;\n  private final SegmentSyncConfig segmentSyncConfig;\n  private final Random random = new Random();\n\n  private static final double SLEEP_RANDOMIZATION_RATIO = .2;\n\n  // Stats\n  // The flush version used to build segments\n  private static final SearchLongGauge CURRENT_FLUSH_VERSION =\n      SearchLongGauge.export(\"current_flush_version\");\n\n  // Accumulated number and time in seconds spent on building segments locally\n  private static SearchCounter segmentsBuiltLocally =\n      SearchCounter.export(\"segments_built_locally\");\n  private static SearchCounter timeSpentOnSuccessfulBuildSecs =\n      SearchCounter.export(\"time_spent_on_successful_build_secs\");\n\n  // The total number of segments to be built\n  private static final SearchLongGauge SEGMENTS_TO_BUILD =\n      SearchLongGauge.export(\"segments_to_build\");\n\n  // How many segments failed locally\n  private static final SearchCounter FAILED_SEGMENTS =\n      SearchCounter.export(\"failed_segments\");\n\n  @Inject\n  protected SegmentBuilder(@Flag(\"onlyRunOnce\") boolean onlyRunOnceFlag,\n                           @Flag(\"waitBetweenLoopsMins\") int waitBetweenLoopsMinsFlag,\n                           @Flag(\"startup_batch_size\") int startUpBatchSizeFlag,\n                           @Flag(\"instance\") int instanceFlag,\n                           @Flag(\"segmentZkLockExpirationHours\")\n                                 int segmentZkLockExpirationHoursFlag,\n                           @Flag(\"startupSleepMins\") long startupSleepMinsFlag,\n                           @Flag(\"maxRetriesOnFailure\") int maxRetriesOnFailureFlag,\n                           @Flag(\"hash_partitions\") List<Integer> hashPartitionsFlag,\n                           @Flag(\"numSegmentBuilderPartitions\") int numSegmentBuilderPartitionsFlag,\n                           @Flag(\"waitBetweenSegmentsSecs\") int waitBetweenSegmentsSecsFlag,\n                           @Flag(\"waitBeforeQuitMins\") int waitBeforeQuitMinsFlag,\n                           @Flag(\"scrubGen\") String scrubGen,\n                           Decider decider) {\n    this(onlyRunOnceFlag,\n        waitBetweenLoopsMinsFlag,\n        startUpBatchSizeFlag,\n        instanceFlag,\n        segmentZkLockExpirationHoursFlag,\n        startupSleepMinsFlag,\n        hashPartitionsFlag,\n        maxRetriesOnFailureFlag,\n        waitBetweenSegmentsSecsFlag,\n        waitBeforeQuitMinsFlag,\n        SearchZkClient.getSZooKeeperClient().createZooKeeperTryLockFactory(),\n        new RateLimitingSegmentHandler(TimeUnit.MINUTES.toMillis(10), Clock.SYSTEM_CLOCK),\n        Clock.SYSTEM_CLOCK,\n        numSegmentBuilderPartitionsFlag,\n        decider,\n        getSyncConfig(scrubGen));\n  }\n\n  @VisibleForTesting\n  protected SegmentBuilder(boolean onlyRunOnceFlag,\n                           int waitBetweenLoopsMinsFlag,\n                           int startUpBatchSizeFlag,\n                           int instanceFlag,\n                           int segmentZkLockExpirationHoursFlag,\n                           long startupSleepMinsFlag,\n                           List<Integer> hashPartitions,\n                           int maxRetriesOnFailure,\n                           int waitBetweenSegmentsSecsFlag,\n                           int waitBeforeQuitMinsFlag,\n                           ZooKeeperTryLockFactory zooKeeperTryLockFactory,\n                           RateLimitingSegmentHandler segmentHandler,\n                           Clock clock,\n                           int numSegmentBuilderPartitions,\n                           Decider decider,\n                           SegmentSyncConfig syncConfig) {\n    LOG.info(\"Creating SegmentBuilder\");\n    LOG.info(\"Penguin version in use: \" + EarlybirdConfig.getPenguinVersion());\n\n    // Set command line flag values\n    this.onlyRunOnce = onlyRunOnceFlag;\n    this.waitBetweenLoopsMins = waitBetweenLoopsMinsFlag;\n    this.startUpBatchSize = startUpBatchSizeFlag;\n    this.instance = instanceFlag;\n    this.waitBetweenSegmentsSecs = waitBetweenSegmentsSecsFlag;\n    this.waitBeforeQuitMins = waitBeforeQuitMinsFlag;\n\n    this.segmentHandler = segmentHandler;\n    this.zkTryLockFactory = zooKeeperTryLockFactory;\n    this.segmentSyncConfig = syncConfig;\n    this.startUpSleepMins = startupSleepMinsFlag;\n\n    if (!hashPartitions.isEmpty()) {\n      this.hashPartitions = ImmutableList.copyOf(hashPartitions);\n    } else {\n      this.hashPartitions = null;\n    }\n\n    Amount<Long, Time> segmentZKLockExpirationTime = Amount.of((long)\n        segmentZkLockExpirationHoursFlag, Time.HOURS);\n\n    this.earlybirdIndexConfig =\n        new ArchiveOnDiskEarlybirdIndexConfig(decider, searchIndexingMetricSet,\n            new CriticalExceptionHandler());\n\n    this.segmentConfig = new SegmentConfig(\n        earlybirdIndexConfig,\n        segmentZKLockExpirationTime,\n        maxRetriesOnFailure,\n        zkTryLockFactory);\n    this.segmentFactory = new EarlybirdSegmentFactory(\n        earlybirdIndexConfig,\n        searchIndexingMetricSet,\n        searcherStats,\n        clock);\n    this.segmentBuilderCoordinator = new SegmentBuilderCoordinator(\n        zkTryLockFactory, syncConfig, clock);\n\n    this.clock = clock;\n\n    this.numSegmentBuilderPartitions = numSegmentBuilderPartitions;\n    this.myPartitionId = instance % numSegmentBuilderPartitions;\n    SearchLongGauge.export(\"segment_builder_partition_id_\" + myPartitionId).set(1);\n\n    CURRENT_FLUSH_VERSION.set(earlybirdIndexConfig.getSchema().getMajorVersionNumber());\n  }\n\n  void run() {\n    LOG.info(\"Config values: {}\", EarlybirdConfig.allValuesAsString());\n\n    // Sleep some time uninterruptibly before get started so that if multiple instances are running,\n    // the HDFS name node and zookeeper wont be overwhelmed\n    // Say, we have 100 instances (instance_arg will have value from 0 - 99, our\n    // STARTUP_BATCH_SIZE_ARG is 20 and startUpSleepMins is 3 mins. Then the first 20 instances\n    // will not sleep, but start immediately. then instance 20 - 39 will sleep 3 mins and then\n    // start to run. instance 40 - 59 will sleep 6 mins then start to run. instances 60 - 79 will\n    // sleep 9 mins and then start to run and so forth.\n    long sleepTime = instance / startUpBatchSize * startUpSleepMins;\n    LOG.info(\"Instance={}, Start up batch size={}\", instance, startUpBatchSize);\n    LOG.info(\"Sleep {} minutes to void HDFS name node and ZooKeeper overwhelmed.\", sleepTime);\n    Uninterruptibles.sleepUninterruptibly(sleepTime, TimeUnit.MINUTES);\n\n    // Kinit here.\n    Kerberos.kinit(\n        EarlybirdConfig.getString(\"kerberos_user\", \"\"),\n        EarlybirdConfig.getString(\"kerberos_keytab_path\", \"\")\n    );\n\n    long waitBetweenLoopsMs = TimeUnit.MINUTES.toMillis(waitBetweenLoopsMins);\n    if (onlyRunOnce) {\n      LOG.info(\"This segment builder will run the full rebuild of all the segments\");\n    } else {\n      LOG.info(\"This segment builder will incrementally check for new data and rebuilt \"\n          + \"current segments as needed.\");\n      LOG.info(\"The waiting interval between two new data checking is: \"\n          + waitBetweenLoopsMs + \" ms.\");\n    }\n\n    boolean scrubGenPresent = segmentSyncConfig.getScrubGen().isPresent();\n    LOG.info(\"Scrub gen present: {}\", scrubGenPresent);\n    boolean scrubGenDataFullyBuilt = segmentBuilderCoordinator.isScrubGenDataFullyBuilt(instance);\n    LOG.info(\"Scrub gen data fully built: {}\", scrubGenDataFullyBuilt);\n\n    if (!scrubGenPresent || scrubGenDataFullyBuilt) {\n      LOG.info(\"Starting segment building loop...\");\n      while (!Thread.currentThread().isInterrupted()) {\n        try {\n          indexingLoop();\n          if (onlyRunOnce) {\n            LOG.info(\"only run once is true, breaking\");\n            break;\n          }\n          clock.waitFor(waitBetweenLoopsMs);\n        } catch (InterruptedException e) {\n          LOG.info(\"Interrupted, quitting segment builder\");\n          Thread.currentThread().interrupt();\n        } catch (SegmentInfoConstructionException e) {\n          LOG.error(\"Error creating new segmentInfo, quitting segment builder: \", e);\n          break;\n        } catch (SegmentUpdaterException e) {\n          FAILED_SEGMENTS.increment();\n          // Before the segment builder quits, sleep for WAIT_BEFORE_QUIT_MINS minutes so that the\n          // FAILED_SEGMENTS stat can be exported.\n          try {\n            clock.waitFor(TimeUnit.MINUTES.toMillis(waitBeforeQuitMins));\n          } catch (InterruptedException ex) {\n            LOG.info(\"Interrupted, quitting segment builder\");\n            Thread.currentThread().interrupt();\n          }\n          LOG.error(\"SegmentUpdater processing segment error, quitting segment builder: \", e);\n          break;\n        }\n      }\n    } else {\n      LOG.info(\"Cannot build the segments for scrub gen yet.\");\n    }\n  }\n\n  // Refactoring the run loop to here for unittest\n  @VisibleForTesting\n  void indexingLoop()\n      throws SegmentInfoConstructionException, InterruptedException, SegmentUpdaterException {\n    // This map contains all the segments to be processed; if a segment is built, it will be removed\n    // from the map.\n    Map<String, SegmentBuilderSegment> buildableSegmentInfoMap;\n    try {\n      buildableSegmentInfoMap = createSegmentInfoMap();\n      printSegmentInfoMap(buildableSegmentInfoMap);\n    } catch (IOException e) {\n      LOG.error(\"Error creating segmentInfoMap: \", e);\n      return;\n    }\n\n    while (!buildableSegmentInfoMap.isEmpty()) {\n      boolean hasBuiltSegment = processSegments(buildableSegmentInfoMap);\n\n      if (!hasBuiltSegment) {\n        // If we successfully built a segment, no need to sleep since building a segment takes a\n        // long time\n        clock.waitFor(processWaitingInterval);\n      }\n    }\n  }\n\n  // Actual shutdown.\n  protected void doShutdown() {\n    LOG.info(\"doShutdown()...\");\n    try {\n      earlybirdIndexConfig.getResourceCloser().shutdownExecutor();\n    } catch (InterruptedException e) {\n      LOG.error(\"Interrupted during shutdown. \", e);\n    }\n\n    LOG.info(\"Segment builder stopped!\");\n  }\n\n  private List<ArchiveTimeSlicer.ArchiveTimeSlice> createTimeSlices() throws IOException {\n    Preconditions.checkState(segmentSyncConfig.getScrubGen().isPresent());\n    Date scrubGen = ScrubGenUtil.parseScrubGenToDate(segmentSyncConfig.getScrubGen().get());\n\n    final DailyStatusBatches dailyStatusBatches =\n        new DailyStatusBatches(zkTryLockFactory, scrubGen);\n    final ArchiveTimeSlicer archiveTimeSlicer = new ArchiveTimeSlicer(\n        EarlybirdConfig.getMaxSegmentSize(), dailyStatusBatches, earlybirdIndexConfig);\n\n    Stopwatch stopwatch = Stopwatch.createStarted();\n    List<ArchiveTimeSlicer.ArchiveTimeSlice> timeSlices = archiveTimeSlicer.getTimeSlices();\n\n    if (timeSlices == null) {\n      LOG.error(\"Failed to load timeslice map after {}\", stopwatch);\n      return Collections.emptyList();\n    }\n\n    LOG.info(\"Took {} to get timeslices\", stopwatch);\n    return timeSlices;\n  }\n\n  private static class TimeSliceAndHashPartition implements Comparable<TimeSliceAndHashPartition> {\n    public final ArchiveTimeSlicer.ArchiveTimeSlice timeSlice;\n    public final Integer hashPartition;\n\n    public TimeSliceAndHashPartition(\n        ArchiveTimeSlicer.ArchiveTimeSlice timeSlice,\n        Integer hashPartition) {\n      this.timeSlice = timeSlice;\n      this.hashPartition = hashPartition;\n    }\n\n    @Override\n    public int compareTo(TimeSliceAndHashPartition o) {\n      Integer myHashPartition = this.hashPartition;\n      Integer otherHashPartition = o.hashPartition;\n\n      long myTimeSliceId = this.timeSlice.getMinStatusID(myHashPartition);\n      long otherTimeSliceId = o.timeSlice.getMinStatusID(otherHashPartition);\n\n      return ComparisonChain.start()\n          .compare(myHashPartition, otherHashPartition)\n          .compare(myTimeSliceId, otherTimeSliceId)\n          .result();\n    }\n  }\n\n  /**\n   * For all the timeslices, create the corresponding SegmentInfo and store in a map\n   */\n  @VisibleForTesting\n  Map<String, SegmentBuilderSegment> createSegmentInfoMap() throws IOException {\n    final List<ArchiveTimeSlicer.ArchiveTimeSlice> timeSlices = createTimeSlices();\n\n    List<TimeSliceAndHashPartition> timeSlicePairs = createPairs(timeSlices);\n    // Export how many segments should be built\n    SEGMENTS_TO_BUILD.set(timeSlicePairs.size());\n    LOG.info(\"Total number of segments to be built across all segment builders: {}\",\n        timeSlicePairs.size());\n\n    List<TimeSliceAndHashPartition> mySegments = getSegmentsForMyPartition(timeSlicePairs);\n\n    Map<String, SegmentBuilderSegment> segmentInfoMap = new HashMap<>();\n    for (TimeSliceAndHashPartition mySegment : mySegments) {\n      ArchiveSegment segment = new ArchiveSegment(mySegment.timeSlice, mySegment.hashPartition,\n          EarlybirdConfig.getMaxSegmentSize());\n      SegmentInfo segmentInfo = new SegmentInfo(segment, segmentFactory, segmentSyncConfig);\n\n      segmentInfoMap.put(segmentInfo.getSegment().getSegmentName(), new NotYetBuiltSegment(\n          segmentInfo, segmentConfig, segmentFactory, 0, segmentSyncConfig));\n    }\n\n    return segmentInfoMap;\n  }\n\n  private List<TimeSliceAndHashPartition> createPairs(\n      List<ArchiveTimeSlicer.ArchiveTimeSlice> timeSlices) {\n\n    List<TimeSliceAndHashPartition> timeSlicePairs = new ArrayList<>();\n\n    for (ArchiveTimeSlicer.ArchiveTimeSlice slice : timeSlices) {\n      List<Integer> localPartitions = hashPartitions;\n      if (localPartitions == null) {\n        localPartitions = range(slice.getNumHashPartitions());\n      }\n\n      for (Integer partition : localPartitions) {\n        timeSlicePairs.add(new TimeSliceAndHashPartition(slice, partition));\n      }\n    }\n    return timeSlicePairs;\n  }\n\n  private List<TimeSliceAndHashPartition> getSegmentsForMyPartition(\n      List<TimeSliceAndHashPartition> timeSlicePairs) {\n\n    Collections.sort(timeSlicePairs);\n\n    List<TimeSliceAndHashPartition> myTimeSlices = new ArrayList<>();\n    for (int i = myPartitionId; i < timeSlicePairs.size(); i += numSegmentBuilderPartitions) {\n      myTimeSlices.add(timeSlicePairs.get(i));\n    }\n\n    LOG.info(\"Getting segments to be built for partition: {}\", myPartitionId);\n    LOG.info(\"Total number of partitions: {}\", numSegmentBuilderPartitions);\n    LOG.info(\"Number of segments picked: {}\", myTimeSlices.size());\n    return myTimeSlices;\n  }\n\n  /**\n   * Print out the segmentInfo Map for debugging\n   */\n  private void printSegmentInfoMap(Map<String, SegmentBuilderSegment> segmentInfoMap) {\n    LOG.info(\"SegmentInfoMap: \");\n    for (Map.Entry<String, SegmentBuilderSegment> entry : segmentInfoMap.entrySet()) {\n      LOG.info(entry.getValue().toString());\n    }\n    LOG.info(\"Total SegmentInfoMap size: \" + segmentInfoMap.size() + \". done.\");\n  }\n\n  /**\n   * Build indices or refresh state for the segments in the specified segmentInfoMap, which only\n   * contains the segments that need to build or are building. When a segment has not been built,\n   * it is built here. If built successfully, it will be removed from the map; otherwise, its\n   * state will be updated in the map.\n   *\n   * Returns true iff this process has built a segment.\n   */\n  @VisibleForTesting\n  boolean processSegments(Map<String, SegmentBuilderSegment> segmentInfoMap)\n      throws SegmentInfoConstructionException, SegmentUpdaterException, InterruptedException {\n\n    boolean hasBuiltSegment = false;\n\n    Iterator<Map.Entry<String, SegmentBuilderSegment>> iter =\n        segmentInfoMap.entrySet().iterator();\n    while (iter.hasNext()) {\n      Map.Entry<String, SegmentBuilderSegment> entry = iter.next();\n      SegmentBuilderSegment originalSegment = entry.getValue();\n\n      LOG.info(\"About to process segment: {}\", originalSegment.getSegmentName());\n      long startMillis = System.currentTimeMillis();\n      SegmentBuilderSegment updatedSegment = segmentHandler.processSegment(originalSegment);\n\n      if (updatedSegment.isBuilt()) {\n        iter.remove();\n        hasBuiltSegment = true;\n\n        if (originalSegment instanceof NotYetBuiltSegment) {\n          // Record the total time spent on successfully building a semgent, used to compute the\n          // average segment building time.\n          long timeSpent = System.currentTimeMillis() - startMillis;\n          segmentsBuiltLocally.increment();\n          timeSpentOnSuccessfulBuildSecs.add(timeSpent / 1000);\n        }\n      } else {\n        entry.setValue(updatedSegment);\n      }\n\n      clock.waitFor(getSegmentSleepTime());\n    }\n\n    return hasBuiltSegment;\n  }\n\n  private long getSegmentSleepTime() {\n    // The Hadoop name node can handle only about 200 requests/sec before it gets overloaded.\n    // Updating the state of a node that has been built takes about 1 second.  In the worst case\n    // scenario with 800 segment builders, we end up with about 800 requests/sec.  Adding a 10\n    // second sleep lowers the worst case to about 80 requests/sec.\n\n    long sleepMillis = TimeUnit.SECONDS.toMillis(waitBetweenSegmentsSecs);\n\n    // Use randomization so that we can't get all segment builders hitting it at the exact same time\n\n    int lowerSleepBoundMillis = (int) (sleepMillis * (1.0 - SLEEP_RANDOMIZATION_RATIO));\n    int upperSleepBoundMillis = (int) (sleepMillis * (1.0 + SLEEP_RANDOMIZATION_RATIO));\n    return randRange(lowerSleepBoundMillis, upperSleepBoundMillis);\n  }\n\n  /**\n   * Returns a pseudo-random number between min and max, inclusive.\n   */\n  private int randRange(int min, int max) {\n    return random.nextInt((max - min) + 1) + min;\n  }\n\n  /**\n   * Returns list of integers 0, 1, 2, ..., count-1.\n   */\n  private static List<Integer> range(int count) {\n    List<Integer> nums = new ArrayList<>(count);\n\n    for (int i = 0; i < count; i++) {\n      nums.add(i);\n    }\n\n    return nums;\n  }\n\n  private static SegmentSyncConfig getSyncConfig(String scrubGen) {\n    if (scrubGen == null || scrubGen.isEmpty()) {\n      throw new RuntimeException(\n          \"Scrub gen expected, but could not get it from the arguments.\");\n    }\n\n    LOG.info(\"Scrub gen: \" + scrubGen);\n    return new SegmentSyncConfig(Optional.of(scrubGen));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/SegmentBuilderApp.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\nimport java.util.Collection;\n\nimport com.google.common.collect.ImmutableList;\nimport com.google.inject.Module;\n\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.app.Flaggable;\nimport com.twitter.inject.server.AbstractTwitterServer;\nimport com.twitter.util.Future;\nimport com.twitter.util.Time;\n\npublic class SegmentBuilderApp extends AbstractTwitterServer {\n  private static final Logger LOG = LoggerFactory.getLogger(SegmentBuilderApp.class);\n\n  public SegmentBuilderApp() {\n    createFlag(\"onlyRunOnce\",\n        true,\n        \"whether to stop segment builder after one loop\",\n        Flaggable.ofBoolean());\n\n    createFlag(\"waitBetweenLoopsMins\",\n        60,\n        \"how many minutes to wait between building loops\",\n        Flaggable.ofInt());\n\n    createFlag(\"startup_batch_size\",\n        30,\n        \"How many instances can start and read timeslice info from HDFS at the same time. \"\n            + \"If you don't know what this parameter is, please do not change this parameter.\",\n        Flaggable.ofInt());\n\n    createFlag(\"instance\",\n        20,\n        \"the job instance number\",\n        Flaggable.ofInt());\n\n    createFlag(\"segmentZkLockExpirationHours\",\n        0,\n        \"max hours to hold the zookeeper lock while building segment\",\n        Flaggable.ofInt());\n\n    createFlag(\"startupSleepMins\",\n        2L,\n        \"sleep multiplier of startupSleepMins before job runs\",\n        Flaggable.ofLong());\n\n    createFlag(\"maxRetriesOnFailure\",\n        3,\n        \"how many times we should try to rebuild a segment when failure happens\",\n        Flaggable.ofInt());\n\n    createFlag(\"hash_partitions\",\n        ImmutableList.of(),\n        \"comma separated hash partition ids, e.g., 0,1,3,4. \"\n            + \"If not specified, all the partitions will be built.\",\n        Flaggable.ofJavaList(Flaggable.ofInt()));\n\n    createFlag(\"numSegmentBuilderPartitions\",\n        100,\n        \"Number of partitions for dividing up all segment builder work\",\n        Flaggable.ofInt());\n\n    createFlag(\"waitBetweenSegmentsSecs\",\n        10,\n        \"Time to sleep between processing segments.\",\n        Flaggable.ofInt());\n\n    createFlag(\"waitBeforeQuitMins\",\n        2,\n        \"How many minutes to sleep before quitting.\",\n        Flaggable.ofInt());\n\n    createFlag(\"scrubGen\",\n        \"\",\n        \"Scrub gen for which segment builders should be run.\",\n        Flaggable.ofString());\n  }\n\n  @Override\n  public void start() {\n    SegmentBuilder segmentBuilder = injector().instance(SegmentBuilder.class);\n    closeOnExit((Time time) -> {\n      segmentBuilder.doShutdown();\n      return Future.Unit();\n    });\n\n    LOG.info(\"Starting run()\");\n    segmentBuilder.run();\n    LOG.info(\"run() complete\");\n\n    // Now shutdown\n    shutdown();\n  }\n\n  protected void shutdown() {\n    LOG.info(\"Calling close() to initiate shutdown\");\n    close();\n  }\n\n  @Override\n  public Collection<Module> javaModules() {\n    return ImmutableList.of(new SegmentBuilderModule());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/SegmentBuilderCoordinator.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\nimport java.io.IOException;\nimport java.util.Date;\nimport java.util.Optional;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.Path;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.quantity.Amount;\nimport com.twitter.common.quantity.Time;\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.database.DatabaseConfig;\nimport com.twitter.search.common.util.zktrylock.TryLock;\nimport com.twitter.search.common.util.zktrylock.ZooKeeperTryLockFactory;\nimport com.twitter.search.earlybird.archive.DailyStatusBatches;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.util.ScrubGenUtil;\nimport com.twitter.search.earlybird.partition.HdfsUtil;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\nimport com.twitter.util.Duration;\n\n/**\n * Coordinate between segment builders for scrubbing pipeline.\n * When segment builder is running, all of them will try to find a HDFS file indicating if data is\n * ready. If the file does not exist, only one of them will go through the files and see if\n * scrubbing pipeline has generated all data for this scrub gen.\n *\n * If the instance that got the lock found all data, it still exists, because otherwise we will\n * have one single segmentbuilder instance trying to build all segments, which is not what we want.\n * But if it exists, then the next time all segmentbuilder instances are scheduled, they will all\n * find the file, and will start building segments.\n */\nclass SegmentBuilderCoordinator {\n  private static final Logger LOG = LoggerFactory.getLogger(SegmentBuilderCoordinator.class);\n\n  private static final Amount<Long, Time> ZK_LOCK_EXPIRATION_MIN = Amount.of(5L, Time.MINUTES);\n  private static final String SEGMENT_BUILDER_SYNC_NODE = \"scrub_gen_data_sync\";\n  private static final String SEGMENT_BUILDER_SYNC_ZK_PATH =\n      EarlybirdProperty.ZK_APP_ROOT.get() + \"/segment_builder_sync\";\n  private static final String DATA_FULLY_BUILT_FILE = \"_data_fully_built\";\n  static final int FIRST_INSTANCE = 0;\n\n  private static final long NON_FIRST_INSTANCE_SLEEP_BEFORE_RETRY_DURATION_MS =\n      Duration.fromHours(1).inMillis();\n\n  private final ZooKeeperTryLockFactory zkTryLockFactory;\n  private final SegmentSyncConfig syncConfig;\n  private final Optional<Date> scrubGenDayOpt;\n  private final Optional<String> scrubGenOpt;\n  private final Clock clock;\n\n  SegmentBuilderCoordinator(\n      ZooKeeperTryLockFactory zkTryLockFactory, SegmentSyncConfig syncConfig, Clock clock) {\n    this.zkTryLockFactory = zkTryLockFactory;\n    this.syncConfig = syncConfig;\n    this.scrubGenOpt = syncConfig.getScrubGen();\n    this.scrubGenDayOpt = scrubGenOpt.map(ScrubGenUtil::parseScrubGenToDate);\n    this.clock = clock;\n  }\n\n\n  public boolean isScrubGenDataFullyBuilt(int instanceNumber) {\n    // Only segment builder that takes scrub gen should use isPartitioningOutputReady to coordinate\n    Preconditions.checkArgument(scrubGenDayOpt.isPresent());\n\n    final FileSystem hdfs;\n    try {\n      hdfs = HdfsUtil.getHdfsFileSystem();\n    } catch (IOException e) {\n      LOG.error(\"Could not create HDFS file system.\", e);\n      return false;\n    }\n\n    return isScrubGenDataFullyBuilt(\n        instanceNumber,\n        scrubGenDayOpt.get(),\n        NON_FIRST_INSTANCE_SLEEP_BEFORE_RETRY_DURATION_MS,\n        hdfs\n    );\n  }\n\n  @VisibleForTesting\n  boolean isScrubGenDataFullyBuilt(\n      int instanceNumber,\n      Date scrubGenDay,\n      long nonFirstInstanceSleepBeforeRetryDuration,\n      FileSystem hdfs) {\n    // Check if the scrub gen has been fully built file exists.\n    if (checkHaveScrubGenDataFullyBuiltFileOnHdfs(hdfs)) {\n      return true;\n    }\n\n    // If it doesn't exist, let first instance see if scrub gen has been fully built and create the\n    // file.\n    if (instanceNumber == FIRST_INSTANCE) {\n      // We were missing some data on HDFS for this scrub gen in previous run,\n      // but we might've gotten more data in the meantime, check again.\n      // Only allow instance 0 to do this mainly for 2 reasons:\n      // 1) Since instances are scheduled in batches, it's possible that a instance from latter\n      // batch find the fully built file in hdfs and start processing. We end up doing work with\n      // only partial instances.\n      // 2) If we sleep before we release lock, it's hard to estimate how long a instance will\n      // be scheduled.\n      // For deterministic reason, we simplify a bit and only allow instance 0 to check and write\n      // data is fully build file to hdfs.\n      try {\n        checkIfScrubGenDataIsFullyBuilt(hdfs, scrubGenDay);\n      } catch (IOException e) {\n        LOG.error(\"Failed to grab lock and check scrub gen data.\", e);\n      }\n    } else {\n      // for all other instances, sleep for a bit to give time for first instance to check if scrub\n      // gen has been fully built and create the file, then check again.\n      try {\n        LOG.info(\n            \"Sleeping for {} ms before re-checking if scrub gen has been fully built file exists\",\n            nonFirstInstanceSleepBeforeRetryDuration);\n        clock.waitFor(nonFirstInstanceSleepBeforeRetryDuration);\n        return checkHaveScrubGenDataFullyBuiltFileOnHdfs(hdfs);\n      } catch (InterruptedException e) {\n        LOG.warn(\"Interrupted when sleeping before re-checking if scrub gen has been fully built \"\n            + \"file exists\", e);\n      }\n    }\n\n    // if hasSuccessFileToHdfs returns false, then should always return false in the end.\n    // next run will find success file for this scrub gen and move forward.\n    return false;\n  }\n\n  private void checkIfScrubGenDataIsFullyBuilt(\n      FileSystem hdfs, Date scrubGenDay) throws IOException {\n    // Build the lock, try to acquire it, and check the data on HDFS\n    TryLock lock = zkTryLockFactory.createTryLock(\n        DatabaseConfig.getLocalHostname(),\n        SEGMENT_BUILDER_SYNC_ZK_PATH,\n        SEGMENT_BUILDER_SYNC_NODE,\n        ZK_LOCK_EXPIRATION_MIN);\n    Preconditions.checkState(scrubGenOpt.isPresent());\n    String scrubGen = scrubGenOpt.get();\n\n    lock.tryWithLock(() -> {\n      LOG.info(String.format(\n          \"Obtained ZK lock to check if data for scrub gen %s is ready.\", scrubGen));\n      final DailyStatusBatches directory =\n          new DailyStatusBatches(zkTryLockFactory, scrubGenDay);\n      if (directory.isScrubGenDataFullyBuilt(hdfs)\n          && createScrubGenDataFullyBuiltFileOnHdfs(hdfs)) {\n        LOG.info(String.format(\"All data for scrub gen %s is ready.\", scrubGen));\n      } else {\n        LOG.info(String.format(\"Data for scrub gen %s is not ready yet.\", scrubGen));\n      }\n    });\n  }\n\n  private boolean createScrubGenDataFullyBuiltFileOnHdfs(FileSystem fs) {\n    Path path = getScrubGenDataFullyBuiltFilePath();\n    try {\n      fs.mkdirs(new Path(statusReadyHDFSPath()));\n      if (fs.createNewFile(path)) {\n        LOG.info(\"Successfully created file \" + path + \" on HDFS.\");\n        return true;\n      } else {\n        LOG.warn(\"Failed to create file \" + path + \" on HDFS.\");\n      }\n    } catch (IOException e) {\n      LOG.error(\"Failed to create file on HDFS \" + path.toString(), e);\n    }\n    return false;\n  }\n\n  private boolean checkHaveScrubGenDataFullyBuiltFileOnHdfs(FileSystem fs) {\n    Path path = getScrubGenDataFullyBuiltFilePath();\n    try {\n      boolean ret = fs.exists(path);\n      LOG.info(\"Checking if file exists showing scrubgen is fully built.\");\n      LOG.info(\"Path checked: {}, Exist check: {}\", path, ret);\n      return ret;\n    } catch (IOException e) {\n      LOG.error(\"Failed to check file on HDFS \" + path.toString(), e);\n      return false;\n    }\n  }\n\n  @VisibleForTesting\n  Path getScrubGenDataFullyBuiltFilePath() {\n    return new Path(statusReadyHDFSPath(), DATA_FULLY_BUILT_FILE);\n  }\n\n  @VisibleForTesting\n  String statusReadyHDFSPath() {\n    return syncConfig.getHdfsSegmentSyncRootDir() + \"/segment_builder_sync\";\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/SegmentBuilderMain.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\npublic final class SegmentBuilderMain {\n\n  private SegmentBuilderMain() { }\n\n  public static void main(String[] args) {\n    new SegmentBuilderApp().main(args);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/SegmentBuilderModule.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\nimport java.io.File;\n\nimport com.google.inject.Provides;\nimport com.google.inject.Singleton;\n\nimport com.twitter.app.Flaggable;\nimport com.twitter.decider.Decider;\nimport com.twitter.inject.TwitterModule;\nimport com.twitter.inject.annotations.Flag;\nimport com.twitter.search.common.config.LoggerConfiguration;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.util.EarlybirdDecider;\n\npublic class SegmentBuilderModule extends TwitterModule {\n\n  private static final String CONFIG_FILE_FLAG_NAME = \"config_file\";\n  private static final String SEGMENT_LOG_DIR_FLAG_NAME = \"segment_log_dir\";\n\n  public SegmentBuilderModule() {\n    createFlag(CONFIG_FILE_FLAG_NAME,\n            new File(\"earlybird-search.yml\"),\n            \"specify config file\",\n            Flaggable.ofFile());\n\n    createFlag(SEGMENT_LOG_DIR_FLAG_NAME,\n            \"\",\n            \"override log dir from config file\",\n            Flaggable.ofString());\n  }\n\n  /**\n   * Initializes the Earlybird config and the log configuration, and returns an EarlybirdDecider\n   * object, which will be injected into the SegmentBuilder instance.\n   *\n   * @param configFile The config file to use to initialize EarlybirdConfig\n   * @param segmentLogDir If not empty, used to override the log directory from the config file\n   * @return An initialized EarlybirdDecider\n   */\n  @Provides\n  @Singleton\n  public Decider provideDecider(@Flag(CONFIG_FILE_FLAG_NAME) File configFile,\n                                @Flag(SEGMENT_LOG_DIR_FLAG_NAME) String segmentLogDir) {\n    // By default Guice will build singletons eagerly:\n    //    https://github.com/google/guice/wiki/Scopes#eager-singletons\n    // So in order to ensure that the EarlybirdConfig and LoggerConfiguration initializations occur\n    // before the EarlybirdDecider initialization, we place them here.\n    EarlybirdConfig.init(configFile.getName());\n    if (!segmentLogDir.isEmpty()) {\n      EarlybirdConfig.overrideLogDir(segmentLogDir);\n    }\n    new LoggerConfiguration(EarlybirdConfig.getLogPropertiesFile(), EarlybirdConfig.getLogDir())\n            .configure();\n\n    return EarlybirdDecider.initialize();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/SegmentBuilderSegment.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.quantity.Amount;\nimport com.twitter.common.quantity.Time;\nimport com.twitter.search.common.database.DatabaseConfig;\nimport com.twitter.search.common.util.zktrylock.TryLock;\nimport com.twitter.search.common.util.zktrylock.ZooKeeperTryLockFactory;\nimport com.twitter.search.earlybird.archive.ArchiveSegment;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.index.EarlybirdSegmentFactory;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\n\npublic abstract class SegmentBuilderSegment {\n  protected final SegmentInfo segmentInfo;\n  protected final SegmentConfig segmentConfig;\n  protected final EarlybirdSegmentFactory earlybirdSegmentFactory;\n  protected final int alreadyRetriedCount;\n  protected final SegmentSyncConfig sync;\n\n  public SegmentBuilderSegment(SegmentInfo segmentInfo,\n                               SegmentConfig segmentConfig,\n                               EarlybirdSegmentFactory earlybirdSegmentFactory,\n                               int alreadyRetriedCount,\n                               SegmentSyncConfig segmentSyncConfig) {\n    this.segmentConfig = segmentConfig;\n    this.earlybirdSegmentFactory = earlybirdSegmentFactory;\n    this.alreadyRetriedCount = alreadyRetriedCount;\n    this.sync = segmentSyncConfig;\n    Preconditions.checkState(segmentInfo.getSegment() instanceof ArchiveSegment);\n    this.segmentInfo = Preconditions.checkNotNull(segmentInfo);\n  }\n\n  public SegmentInfo getSegmentInfo() {\n    return segmentInfo;\n  }\n\n  public String getSegmentName() {\n    return segmentInfo.getSegmentName();\n  }\n\n  public int getAlreadyRetriedCount() {\n    return alreadyRetriedCount;\n  }\n\n  /**\n   * Handle the segment, potentially transitioning to a new state.\n   * @return The state after handling.\n   */\n  public abstract SegmentBuilderSegment handle()\n      throws SegmentInfoConstructionException, SegmentUpdaterException;\n\n  public boolean isBuilt() {\n    return false;\n  }\n\n  @Override\n  public String toString() {\n    return \"SegmentBuilderSegment{\"\n        + \"segmentInfo=\" + segmentInfo\n        + \", state=\" + this.getClass().getSimpleName()\n        + \", alreadyRetriedCount=\" + alreadyRetriedCount + '}';\n  }\n\n  /**\n   * Given a SegmentInfo, create a new one with the same time slice and partitionID but clean\n   * internal state.\n   */\n  protected SegmentInfo createNewSegmentInfo(SegmentInfo oldSegmentInfo)\n      throws SegmentInfoConstructionException {\n    Preconditions.checkArgument(oldSegmentInfo.getSegment() instanceof ArchiveSegment);\n    ArchiveSegment archiveSegment = (ArchiveSegment) oldSegmentInfo.getSegment();\n\n    try {\n      ArchiveSegment segment = new ArchiveSegment(archiveSegment.getArchiveTimeSlice(),\n          archiveSegment.getHashPartitionID(), EarlybirdConfig.getMaxSegmentSize());\n\n      return new SegmentInfo(segment, earlybirdSegmentFactory, sync);\n    } catch (IOException e) {\n      throw new SegmentInfoConstructionException(\"Error creating new segments\", e);\n    }\n  }\n\n  protected TryLock getZooKeeperTryLock() {\n    ZooKeeperTryLockFactory tryLockFactory = segmentConfig.getTryLockFactory();\n    String zkRootPath = sync.getZooKeeperSyncFullPath();\n    String nodeName = segmentInfo.getZkNodeName();\n    Amount<Long, Time> expirationTime = segmentConfig.getSegmentZKLockExpirationTime();\n\n    return tryLockFactory.createTryLock(\n        DatabaseConfig.getLocalHostname(),\n        zkRootPath,\n        nodeName,\n        expirationTime);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/SegmentConfig.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\nimport com.twitter.common.quantity.Amount;\nimport com.twitter.common.quantity.Time;\nimport com.twitter.search.common.util.zktrylock.ZooKeeperTryLockFactory;\nimport com.twitter.search.earlybird.archive.ArchiveOnDiskEarlybirdIndexConfig;\n\npublic class SegmentConfig {\n  private final ArchiveOnDiskEarlybirdIndexConfig earlybirdIndexConfig;\n  private final Amount<Long, Time> segmentZKLockExpirationTime;\n  private final int maxRetriesOnFailure;\n  private final ZooKeeperTryLockFactory tryLockFactory;\n\n  public SegmentConfig(\n      ArchiveOnDiskEarlybirdIndexConfig earlybirdIndexConfig,\n      Amount<Long, Time> segmentZKLockExpirationTime,\n      int maxRetriesOnFailure,\n      ZooKeeperTryLockFactory tryLockFactory) {\n\n    this.earlybirdIndexConfig = earlybirdIndexConfig;\n    this.segmentZKLockExpirationTime = segmentZKLockExpirationTime;\n    this.maxRetriesOnFailure = maxRetriesOnFailure;\n    this.tryLockFactory = tryLockFactory;\n  }\n\n  public ArchiveOnDiskEarlybirdIndexConfig getEarlybirdIndexConfig() {\n    return earlybirdIndexConfig;\n  }\n\n  public Amount<Long, Time> getSegmentZKLockExpirationTime() {\n    return segmentZKLockExpirationTime;\n  }\n\n  public int getMaxRetriesOnFailure() {\n    return maxRetriesOnFailure;\n  }\n\n  public ZooKeeperTryLockFactory getTryLockFactory() {\n    return tryLockFactory;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/SegmentInfoConstructionException.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\nimport java.io.IOException;\n\n/**\n * Used if exceptions are thrown during creating new SegmentInfo during the indexing loop\n */\nclass SegmentInfoConstructionException extends Exception {\n  SegmentInfoConstructionException(String msg, IOException e) {\n    super(msg, e);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/SegmentUpdaterException.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\nimport com.google.common.annotations.VisibleForTesting;\n\n/**\n * Used when when SegmentUpdater fails processing segments.\n */\n@VisibleForTesting\nclass SegmentUpdaterException extends Exception {\n  SegmentUpdaterException(String msg) {\n    super(msg);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/archive/segmentbuilder/SomeoneElseIsBuildingSegment.java",
    "content": "package com.twitter.search.earlybird.archive.segmentbuilder;\n\nimport java.util.concurrent.atomic.AtomicBoolean;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport com.twitter.common.base.Command;\nimport com.twitter.search.common.util.zktrylock.TryLock;\nimport com.twitter.search.earlybird.archive.ArchiveHDFSUtils;\nimport com.twitter.search.earlybird.index.EarlybirdSegmentFactory;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\n\npublic class SomeoneElseIsBuildingSegment extends SegmentBuilderSegment {\n  public SomeoneElseIsBuildingSegment(\n      SegmentInfo segmentInfo,\n      SegmentConfig segmentConfig,\n      EarlybirdSegmentFactory earlybirdSegmentFactory,\n      int alreadyRetriedCount,\n      SegmentSyncConfig sync) {\n\n    super(segmentInfo, segmentConfig, earlybirdSegmentFactory, alreadyRetriedCount, sync);\n  }\n\n  /**\n   * This method refreshes local state of a segment.\n   * 1. Try to grab the ZK lock\n   *   2a. if got the lock, the segment is not being built; mark segment as NOT_BUILT_YET.\n   *   2b. otherwise, the segment is being built; keep the SOMEONE_ELSE_IS_BUILDING state\n   */\n  @Override\n  public SegmentBuilderSegment handle()\n      throws SegmentInfoConstructionException, SegmentUpdaterException {\n\n    TryLock lock = getZooKeeperTryLock();\n\n    final AtomicBoolean alreadyBuilt = new AtomicBoolean(false);\n    boolean gotLock = lock.tryWithLock((Command) () -> {\n      // The segment might have already finished built by others\n      if (segmentExistsOnHdfs()) {\n        alreadyBuilt.set(true);\n      }\n    });\n\n    if (!gotLock) {\n      return this;\n    }\n\n    if (alreadyBuilt.get()) {\n      return new BuiltAndFinalizedSegment(\n          segmentInfo, segmentConfig, earlybirdSegmentFactory, 0, sync);\n    } else {\n      // When a segment failed building, its state might not be clean. So, it is necessary to\n      // create a new SegmentInfo with a clean state\n      SegmentInfo newSegmentInfo = createNewSegmentInfo(segmentInfo);\n      return new NotYetBuiltSegment(\n          newSegmentInfo,\n          segmentConfig,\n          earlybirdSegmentFactory,\n          alreadyRetriedCount + 1,\n          sync);\n    }\n  }\n\n  @VisibleForTesting\n  boolean segmentExistsOnHdfs() {\n    return ArchiveHDFSUtils.hasSegmentIndicesOnHDFS(sync, segmentInfo);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/twitter/elephantbird:core\",\n        \"3rdparty/jvm/commons-codec\",\n        \"3rdparty/jvm/commons-httpclient\",\n        \"3rdparty/jvm/geo/google:geoGoogle\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"decider/src/main/scala\",\n        \"finagle/finagle-core/src/main\",\n        \"finagle/finagle-thrift/src/main/java\",\n        \"finagle/finagle-thrift/src/main/scala\",\n        \"scrooge/scrooge-core/src/main/scala\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/optional\",\n        \"src/java/com/twitter/search/common/decider\",\n        \"src/java/com/twitter/search/common/logging\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/util:finagleutil\",\n        \"src/java/com/twitter/search/common/util/earlybird\",\n        \"src/java/com/twitter/search/common/util/thrift:thrift-utils\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/thrift/com/twitter/context:twitter-context-scala\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n        \"src/thrift/com/twitter/search/common:caching-java\",\n        \"src/thrift/com/twitter/search/common:constants-java\",\n        \"src/thrift/com/twitter/search/common:query-java\",\n        \"strato/src/main/scala/com/twitter/strato/opcontext\",\n        \"twitter-context/src/main/scala\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/Base64RequestResponseForLogging.java",
    "content": "package com.twitter.search.earlybird.common;\n\nimport org.apache.commons.codec.binary.Base64;\nimport org.apache.thrift.TException;\nimport org.apache.thrift.TSerializer;\nimport org.apache.thrift.protocol.TBinaryProtocol;\nimport org.slf4j.Logger;\n\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\n\npublic final class Base64RequestResponseForLogging {\n  private static final Logger GENERAL_LOG = org.slf4j.LoggerFactory.getLogger(\n      Base64RequestResponseForLogging.class);\n  private static final Logger FAILED_REQUEST_LOG = org.slf4j.LoggerFactory.getLogger(\n      Base64RequestResponseForLogging.class.getName() + \".FailedRequests\");\n  private static final Logger RANDOM_REQUEST_LOG = org.slf4j.LoggerFactory.getLogger(\n      Base64RequestResponseForLogging.class.getName() + \".RandomRequests\");\n  private static final Logger SLOW_REQUEST_LOG = org.slf4j.LoggerFactory.getLogger(\n      Base64RequestResponseForLogging.class.getName() + \".SlowRequests\");\n\n  private enum LogType {\n    FAILED,\n    RANDOM,\n    SLOW,\n  };\n\n  private final LogType logtype;\n  private final String logLine;\n  private final EarlybirdRequest request;\n  private final EarlybirdResponse response;\n  private final Base64 base64 = new Base64();\n\n  // TSerializer is not threadsafe, so create a new one for each request\n  private final TSerializer serializer = new TSerializer(new TBinaryProtocol.Factory());\n\n  private Base64RequestResponseForLogging(\n      LogType logType, String logLine, EarlybirdRequest request, EarlybirdResponse response) {\n    this.logtype = logType;\n    this.logLine = logLine;\n    this.request = request;\n    this.response = response;\n  }\n\n  public static Base64RequestResponseForLogging randomRequest(\n      String logLine, EarlybirdRequest request, EarlybirdResponse response) {\n    return new Base64RequestResponseForLogging(LogType.RANDOM, logLine, request, response);\n  }\n\n  public static Base64RequestResponseForLogging failedRequest(\n      String logLine, EarlybirdRequest request, EarlybirdResponse response) {\n    return new Base64RequestResponseForLogging(LogType.FAILED, logLine, request, response);\n  }\n\n  public static Base64RequestResponseForLogging slowRequest(\n      String logLine, EarlybirdRequest request, EarlybirdResponse response) {\n    return new Base64RequestResponseForLogging(LogType.SLOW, logLine, request, response);\n  }\n\n  private String asBase64(EarlybirdRequest clearedRequest) {\n    try {\n      // The purpose of this log is to make it easy to re-issue requests in formz to reproduce\n      // issues. If queries are re-issued as is they will be treated as late-arriving queries and\n      // dropped due to the clientRequestTimeMs being set to the original query time. For ease of\n      // use purposes we clear clientRequestTimeMs and log it out separately for the rare case it\n      // is needed.\n      clearedRequest.unsetClientRequestTimeMs();\n      return base64.encodeToString(serializer.serialize(clearedRequest));\n    } catch (TException e) {\n      GENERAL_LOG.error(\"Failed to serialize request for logging.\", e);\n      return \"failed_to_serialize\";\n    }\n  }\n\n  private String asBase64(EarlybirdResponse earlybirdResponse) {\n    try {\n      return base64.encodeToString(serializer.serialize(earlybirdResponse));\n    } catch (TException e) {\n      GENERAL_LOG.error(\"Failed to serialize response for logging.\", e);\n      return \"failed_to_serialize\";\n    }\n  }\n\n  private String getFormattedMessage() {\n    String base64Request = asBase64(\n        EarlybirdRequestUtil.copyAndClearUnnecessaryValuesForLogging(request));\n    String base64Response = asBase64(response);\n    return logLine + \", clientRequestTimeMs: \" + request.getClientRequestTimeMs()\n        + \", \" + base64Request + \", \" + base64Response;\n  }\n\n  /**\n   * Logs the Base64-encoded request and response to the success or failure log.\n   */\n  public void log() {\n    // Do the serializing/concatting this way so it happens on the background thread for\n    // async logging\n    Object logObject = new Object() {\n      @Override\n      public String toString() {\n        return getFormattedMessage();\n      }\n    };\n\n    switch (logtype) {\n      case FAILED:\n        FAILED_REQUEST_LOG.info(\"{}\", logObject);\n        break;\n      case RANDOM:\n        RANDOM_REQUEST_LOG.info(\"{}\", logObject);\n        break;\n      case SLOW:\n        SLOW_REQUEST_LOG.info(\"{}\", logObject);\n        break;\n      default:\n        // Not logging anything for other log types.\n        break;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/CaughtUpMonitor.java",
    "content": "package com.twitter.search.earlybird.common;\n\nimport java.util.concurrent.atomic.AtomicBoolean;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchCustomGauge;\n\n/**\n * A monitor which enforces the condition that a single thread's work is caught up, and allows\n * other threads to wait to be notified when the work is complete. An AtomicBoolean ensures the\n * current status is visible to all threads.\n */\npublic class CaughtUpMonitor {\n  private static final Logger LOG = LoggerFactory.getLogger(CaughtUpMonitor.class);\n\n  protected final AtomicBoolean isCaughtUp = new AtomicBoolean(false);\n\n  public CaughtUpMonitor(String statPrefix) {\n    SearchCustomGauge.export(statPrefix + \"_is_caught_up\", () -> isCaughtUp() ? 1 : 0);\n  }\n\n  public boolean isCaughtUp() {\n    return isCaughtUp.get();\n  }\n\n  /**\n   * Set caught up state, and notify waiting threads if caught up.\n   */\n  public synchronized void setAndNotify(boolean caughtUp) {\n    isCaughtUp.set(caughtUp);\n    if (caughtUp) {\n      // Readers are caught up, notify waiting threads\n      notifyAll();\n    }\n  }\n\n  /**\n   * Wait using Object.wait() until caught up or until thread is interrupted.\n   */\n  public synchronized void resetAndWaitUntilCaughtUp() {\n    LOG.info(\"Waiting to catch up.\");\n    // Explicitly set isCaughtUp to false before waiting\n    isCaughtUp.set(false);\n    try {\n      while (!isCaughtUp()) {\n        wait();\n      }\n    } catch (InterruptedException e) {\n      LOG.error(\"{} was interrupted while waiting to catch up\", Thread.currentThread());\n    }\n    LOG.info(\"Caught up.\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/ClientIdUtil.java",
    "content": "package com.twitter.search.earlybird.common;\n\nimport java.util.Optional;\n\nimport com.twitter.common.optional.Optionals;\nimport com.twitter.search.common.util.FinagleUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.strato.opcontext.Attribution;\nimport com.twitter.strato.opcontext.HttpEndpoint;\n\npublic final class ClientIdUtil {\n  // Blenders should always set the EarlybirdRequest.clientId field. It should be set to the Finagle\n  // client ID of the client that caused the blender to send this request to the roots. If the\n  // Finagle ID of the blender's client cannot be determined, it will be set to \"unknown\" (see\n  // com.twitter.search.common.util.FinagleUtil.UNKNOWN_CLIENT_NAME). However, other services that\n  // send requests to roots might not set EarlybirdRequest.clientId.\n  //\n  // So an \"unset\" clientId means: EarlybirdRequest.clientId was null.\n  // An \"unknown\" clientId means: the client that sent us the request\n  // tried setting EarlybirdRequest.clientId, but couldn't figure out a good value for it.\n  public static final String UNSET_CLIENT_ID = \"unset\";\n\n  private static final String CLIENT_ID_FOR_UNKNOWN_CLIENTS = \"unknown_client_id\";\n\n  private static final String CLIENT_ID_PREFIX = \"client_id_\";\n\n  private static final String FINAGLE_CLIENT_ID_AND_CLIENT_ID_PATTERN =\n      \"finagle_id_%s_and_client_id_%s\";\n\n  private static final String CLIENT_ID_AND_REQUEST_TYPE = \"client_id_%s_and_type_%s\";\n\n  private ClientIdUtil() {\n  }\n\n  /** Returns the ID of the client that initiated this request or UNSET_CLIENT_ID if not set. */\n  public static String getClientIdFromRequest(EarlybirdRequest request) {\n    return Optional\n        .ofNullable(request.getClientId())\n        .map(String::toLowerCase)\n        .orElse(UNSET_CLIENT_ID);\n  }\n\n  /**\n   * Returns the Strato http endpoint attribution as an Optional.\n   */\n  public static Optional<String> getClientIdFromHttpEndpointAttribution() {\n    return Optionals\n        .optional(Attribution.httpEndpoint())\n        .map(HttpEndpoint::name)\n        .map(String::toLowerCase);\n  }\n\n  /** Formats the given clientId into a string that can be used for stats. */\n  public static String formatClientId(String clientId) {\n    return CLIENT_ID_PREFIX + clientId;\n  }\n\n  /**\n   * Formats the given Finagle clientId and the given clientId into a single string that can be used\n   * for stats, or other purposes where the two IDs need to be combined.\n   */\n  public static String formatFinagleClientIdAndClientId(String finagleClientId, String clientId) {\n    return String.format(FINAGLE_CLIENT_ID_AND_CLIENT_ID_PATTERN, finagleClientId, clientId);\n  }\n\n  /**\n   * Formats the given clientId and requestType into a single string that can be used\n   * for stats or other purposes.\n   */\n  public static String formatClientIdAndRequestType(\n      String clientId, String requestType) {\n    return String.format(CLIENT_ID_AND_REQUEST_TYPE, clientId, requestType);\n  }\n\n  /**\n   * Format the quota client id\n   */\n  public static String getQuotaClientId(String clientId) {\n    if (FinagleUtil.UNKNOWN_CLIENT_NAME.equals(clientId) || UNSET_CLIENT_ID.equals(clientId)) {\n      return CLIENT_ID_FOR_UNKNOWN_CLIENTS;\n    }\n\n    return clientId;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/EarlybirdRequestLogger.java",
    "content": "package com.twitter.search.earlybird.common;\n\nimport java.util.EnumMap;\nimport java.util.Map;\n\nimport scala.Option;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.Maps;\n\nimport com.twitter.context.TwitterContext;\nimport com.twitter.context.thriftscala.Viewer;\nimport com.twitter.decider.Decider;\nimport com.twitter.finagle.thrift.ClientId;\nimport com.twitter.finagle.thrift.ClientId$;\nimport com.twitter.search.TwitterContextPermit;\nimport com.twitter.search.common.constants.thriftjava.ThriftQuerySource;\nimport com.twitter.search.common.decider.DeciderUtil;\nimport com.twitter.search.common.logging.RPCLogger;\nimport com.twitter.search.common.metrics.FailureRatioCounter;\nimport com.twitter.search.common.metrics.Timer;\nimport com.twitter.search.common.util.earlybird.TermStatisticsUtil;\nimport com.twitter.search.common.util.earlybird.ThriftSearchResultUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.ThriftFacetFieldRequest;\nimport com.twitter.search.earlybird.thrift.ThriftHistogramSettings;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftTermStatisticsRequest;\n\nimport static com.twitter.search.common.util.earlybird.EarlybirdResponseUtil\n    .responseConsideredFailed;\n\n\npublic class EarlybirdRequestLogger extends RPCLogger {\n  protected enum ExtraFields {\n    QUERY_MAX_HITS_TO_PROCESS,\n    COLLECTOR_PARAMS_MAX_HITS_TO_PROCESS,\n    RELEVANCE_OPTIONS_MAX_HITS_TO_PROCESS,\n    NUM_HITS_PROCESSED,\n    QUERY_COST,\n    CPU_TOTAL,\n    QUERY_SOURCE,\n    CLIENT_ID,\n    FINAGLE_CLIENT_ID\n  }\n\n  protected enum ShardOnlyExtraFields {\n    NUM_SEARCHED_SEGMENTS,\n    SCORING_TIME_NANOS\n  }\n\n  protected enum RootOnlyExtraFields {\n    CACHING_ALLOWED,\n    DEBUG_MODE,\n    CACHE_HIT,\n    USER_AGENT,\n    // See JIRA APPSEC-2303 for IP addresses logging\n  }\n\n  private static final String LOG_FULL_REQUEST_DETAILS_ON_ERROR_DECIDER_KEY =\n      \"log_full_request_details_on_error\";\n  private static final String LOG_FULL_REQUEST_DETAILS_RANDOM_FRACTION_DECIDER_KEY =\n      \"log_full_request_details_random_fraction\";\n  private static final String LOG_FULL_SLOW_REQUEST_DETAILS_RANDOM_FRACTION_DECIDER_KEY =\n      \"log_full_slow_request_details_random_fraction\";\n  private static final String SLOW_REQUEST_LATENCY_THRESHOLD_MS_DECIDER_KEY =\n      \"slow_request_latency_threshold_ms\";\n\n  private final Decider decider;\n  private final boolean enableLogUnknownClientRequests;\n\n  private static final Map<ThriftQuerySource, FailureRatioCounter>\n      FAILURE_RATIO_COUNTER_BY_QUERY_SOURCE = preBuildFailureRatioCounters();\n  private static final FailureRatioCounter NO_QUERY_SOURCE_FAILURE_RATIO_COUNTER =\n      new FailureRatioCounter(\"earlybird_logger\", \"query_source\", \"not_set\");\n\n  static EarlybirdRequestLogger buildForRoot(\n      String loggerName, int latencyWarnThreshold, Decider decider) {\n\n    return new EarlybirdRequestLogger(loggerName, latencyWarnThreshold,\n        decider, true, RPCLogger.Fields.values(), ExtraFields.values(),\n        RootOnlyExtraFields.values());\n  }\n\n  static EarlybirdRequestLogger buildForShard(\n      String loggerName, int latencyWarnThreshold, Decider decider) {\n\n    return new EarlybirdRequestLogger(loggerName, latencyWarnThreshold,\n        decider, false, RPCLogger.Fields.values(), ExtraFields.values(),\n        ShardOnlyExtraFields.values());\n  }\n\n  @VisibleForTesting\n  EarlybirdRequestLogger(String loggerName, int latencyWarnThreshold, Decider decider) {\n    this(loggerName, latencyWarnThreshold, decider, false, RPCLogger.Fields.values(),\n        ExtraFields.values(), RootOnlyExtraFields.values(), ShardOnlyExtraFields.values());\n  }\n\n  private EarlybirdRequestLogger(String loggerName, int latencyWarnThreshold, Decider decider,\n                                 boolean enableLogUnknownClientRequests, Enum[]... fieldEnums) {\n    super(loggerName, fieldEnums);\n    this.decider = decider;\n    this.enableLogUnknownClientRequests = enableLogUnknownClientRequests;\n    setLatencyWarnThreshold(latencyWarnThreshold);\n  }\n\n  /**\n   * Logs the given earlybird request and response.\n   *\n   * @param request The earlybird request.\n   * @param response The earlybird response.\n   * @param timer The time it took to process this request.\n   */\n  public void logRequest(EarlybirdRequest request, EarlybirdResponse response, Timer timer) {\n    try {\n      LogEntry entry = newLogEntry();\n\n      setRequestLogEntries(entry, request);\n      setResponseLogEntries(entry, response);\n      if (timer != null) {\n        entry.setField(ExtraFields.CPU_TOTAL, Long.toString(timer.getElapsedCpuTotal()));\n      }\n\n      boolean wasError = response != null && responseConsideredFailed(response.getResponseCode());\n\n      long responseTime = response != null ? response.getResponseTime() : 0L;\n\n      String logLine = writeLogLine(entry, responseTime, wasError);\n\n      // This code path is called for pre/post logging\n      // Prevent same request showing up twice by only logging on post logging\n      if (response != null && DeciderUtil.isAvailableForRandomRecipient(\n          decider, LOG_FULL_REQUEST_DETAILS_RANDOM_FRACTION_DECIDER_KEY)) {\n        Base64RequestResponseForLogging.randomRequest(logLine, request, response).log();\n      }\n\n      // Unknown client request logging only applies to pre-logging.\n      if (enableLogUnknownClientRequests && response == null) {\n        UnknownClientRequestForLogging unknownClientRequestLogger =\n            UnknownClientRequestForLogging.unknownClientRequest(logLine, request);\n        if (unknownClientRequestLogger != null) {\n          unknownClientRequestLogger.log();\n        }\n      }\n\n      if (wasError\n          && DeciderUtil.isAvailableForRandomRecipient(\n          decider, LOG_FULL_REQUEST_DETAILS_ON_ERROR_DECIDER_KEY)) {\n        new RequestResponseForLogging(request, response).logFailedRequest();\n        Base64RequestResponseForLogging.failedRequest(logLine, request, response).log();\n      }\n\n      boolean wasSlow = response != null\n          && responseTime >= DeciderUtil.getAvailability(\n              decider, SLOW_REQUEST_LATENCY_THRESHOLD_MS_DECIDER_KEY);\n      if (wasSlow\n          && DeciderUtil.isAvailableForRandomRecipient(\n              decider, LOG_FULL_SLOW_REQUEST_DETAILS_RANDOM_FRACTION_DECIDER_KEY)) {\n        Base64RequestResponseForLogging.slowRequest(logLine, request, response).log();\n      }\n\n      FailureRatioCounter failureRatioCounter =\n          FAILURE_RATIO_COUNTER_BY_QUERY_SOURCE.get(request.getQuerySource());\n      if (failureRatioCounter != null) {\n        failureRatioCounter.requestFinished(!wasError);\n      } else {\n        NO_QUERY_SOURCE_FAILURE_RATIO_COUNTER.requestFinished(!wasError);\n      }\n\n    } catch (Exception e) {\n      LOG.error(\"Exception building log entry \", e);\n    }\n  }\n\n  private void setRequestLogEntries(LogEntry entry, EarlybirdRequest request) {\n    entry.setField(Fields.CLIENT_HOST, request.getClientHost());\n    entry.setField(Fields.CLIENT_REQUEST_ID, request.getClientRequestID());\n    entry.setField(Fields.REQUEST_TYPE, requestTypeForLog(request));\n\n    if (request.isSetSearchQuery()) {\n      ThriftSearchQuery searchQuery = request.getSearchQuery();\n      entry.setField(Fields.QUERY, searchQuery.getSerializedQuery());\n\n      if (searchQuery.isSetMaxHitsToProcess()) {\n        entry.setField(ExtraFields.QUERY_MAX_HITS_TO_PROCESS,\n                       Integer.toString(searchQuery.getMaxHitsToProcess()));\n      }\n\n      if (searchQuery.isSetCollectorParams()\n          && searchQuery.getCollectorParams().isSetTerminationParams()\n          && searchQuery.getCollectorParams().getTerminationParams().isSetMaxHitsToProcess()) {\n        entry.setField(ExtraFields.COLLECTOR_PARAMS_MAX_HITS_TO_PROCESS,\n                       Integer.toString(searchQuery.getCollectorParams().getTerminationParams()\n                                        .getMaxHitsToProcess()));\n      }\n\n      if (searchQuery.isSetRelevanceOptions()\n          && searchQuery.getRelevanceOptions().isSetMaxHitsToProcess()) {\n        entry.setField(ExtraFields.RELEVANCE_OPTIONS_MAX_HITS_TO_PROCESS,\n                       Integer.toString(searchQuery.getRelevanceOptions().getMaxHitsToProcess()));\n      }\n    }\n\n    entry.setField(Fields.NUM_REQUESTED, Integer.toString(numRequestedForLog(request)));\n\n    if (request.isSetQuerySource()) {\n      entry.setField(ExtraFields.QUERY_SOURCE, request.getQuerySource().name());\n    }\n\n    if (request.isSetClientId()) {\n      entry.setField(ExtraFields.CLIENT_ID, request.getClientId());\n    }\n\n    entry.setField(RootOnlyExtraFields.CACHING_ALLOWED,\n                   Boolean.toString(EarlybirdRequestUtil.isCachingAllowed(request)));\n\n    entry.setField(RootOnlyExtraFields.DEBUG_MODE, Byte.toString(request.getDebugMode()));\n\n    Option<ClientId> clientIdOption = ClientId$.MODULE$.current();\n    if (clientIdOption.isDefined()) {\n      entry.setField(ExtraFields.FINAGLE_CLIENT_ID, clientIdOption.get().name());\n    }\n\n    setLogEntriesFromTwitterContext(entry);\n  }\n\n  @VisibleForTesting\n  Option<Viewer> getTwitterContext() {\n    return TwitterContext.acquire(TwitterContextPermit.get()).apply();\n  }\n\n  private void setLogEntriesFromTwitterContext(LogEntry entry) {\n    Option<Viewer> viewerOption = getTwitterContext();\n    if (viewerOption.nonEmpty()) {\n      Viewer viewer = viewerOption.get();\n\n      if (viewer.userAgent().nonEmpty()) {\n        String userAgent = viewer.userAgent().get();\n\n        // we only replace the comma in the user-agent with %2C to make it easily parseable,\n        // specially with command line tools like cut/sed/awk\n        userAgent = userAgent.replace(\",\", \"%2C\");\n\n        entry.setField(RootOnlyExtraFields.USER_AGENT, userAgent);\n      }\n    }\n  }\n\n  private void setResponseLogEntries(LogEntry entry, EarlybirdResponse response) {\n    if (response != null) {\n      entry.setField(Fields.NUM_RETURNED, Integer.toString(numResultsForLog(response)));\n      entry.setField(Fields.RESPONSE_CODE, String.valueOf(response.getResponseCode()));\n      entry.setField(Fields.RESPONSE_TIME_MICROS, Long.toString(response.getResponseTimeMicros()));\n      if (response.isSetSearchResults()) {\n        entry.setField(ExtraFields.NUM_HITS_PROCESSED,\n            Integer.toString(response.getSearchResults().getNumHitsProcessed()));\n        entry.setField(ExtraFields.QUERY_COST,\n            Double.toString(response.getSearchResults().getQueryCost()));\n        if (response.getSearchResults().isSetScoringTimeNanos()) {\n          entry.setField(ShardOnlyExtraFields.SCORING_TIME_NANOS,\n              Long.toString(response.getSearchResults().getScoringTimeNanos()));\n        }\n      }\n      if (response.isSetCacheHit()) {\n        entry.setField(RootOnlyExtraFields.CACHE_HIT, String.valueOf(response.isCacheHit()));\n      }\n      if (response.isSetNumSearchedSegments()) {\n        entry.setField(ShardOnlyExtraFields.NUM_SEARCHED_SEGMENTS,\n            Integer.toString(response.getNumSearchedSegments()));\n      }\n    }\n  }\n\n  private static int numRequestedForLog(EarlybirdRequest request) {\n    int num = 0;\n    if (request.isSetFacetRequest() && request.getFacetRequest().isSetFacetFields()) {\n      for (ThriftFacetFieldRequest field : request.getFacetRequest().getFacetFields()) {\n        num += field.getNumResults();\n      }\n    } else if (request.isSetTermStatisticsRequest()) {\n      num = request.getTermStatisticsRequest().getTermRequestsSize();\n    } else if (request.isSetSearchQuery()) {\n      num =  request.getSearchQuery().isSetCollectorParams()\n          ? request.getSearchQuery().getCollectorParams().getNumResultsToReturn() : 0;\n      if (request.getSearchQuery().getSearchStatusIdsSize() > 0) {\n        num = Math.max(num, request.getSearchQuery().getSearchStatusIdsSize());\n      }\n    }\n    return num;\n  }\n\n  /**\n   * Returns the number of results in the given response. If the response is a term stats response,\n   * then the returned value will be the number of term results. If the response is a facet\n   * response, then the returned value will be the number of facet results. Otherwise, the returned\n   * value will be the number of search results.\n   */\n  public static int numResultsForLog(EarlybirdResponse response) {\n    if (response == null) {\n      return 0;\n    } else if (response.isSetFacetResults()) {\n      return ThriftSearchResultUtil.numFacetResults(response.getFacetResults());\n    } else if (response.isSetTermStatisticsResults()) {\n      return response.getTermStatisticsResults().getTermResultsSize();\n    } else {\n      return ThriftSearchResultUtil.numResults(response.getSearchResults());\n    }\n  }\n\n  private static String requestTypeForLog(EarlybirdRequest request) {\n    StringBuilder requestType = new StringBuilder(64);\n    if (request.isSetFacetRequest()) {\n      requestType.append(\"FACETS\");\n      int numFields = request.getFacetRequest().getFacetFieldsSize();\n      if (numFields > 0) {\n        // For 1 or 2 fields, just put them in the request type.  For more, just log the number.\n        if (numFields <= 2) {\n          for (ThriftFacetFieldRequest field : request.getFacetRequest().getFacetFields()) {\n            requestType.append(\":\").append(field.getFieldName().toUpperCase());\n          }\n        } else {\n          requestType.append(\":MULTI-\").append(numFields);\n        }\n      }\n    } else if (request.isSetTermStatisticsRequest()) {\n      ThriftTermStatisticsRequest termStatsRequest = request.getTermStatisticsRequest();\n      requestType.append(\"TERMSTATS-\")\n          .append(termStatsRequest.getTermRequestsSize());\n\n      ThriftHistogramSettings histoSettings = termStatsRequest.getHistogramSettings();\n      if (histoSettings != null) {\n        String binSizeVal = String.valueOf(TermStatisticsUtil.determineBinSize(histoSettings));\n        String numBinsVal = String.valueOf(histoSettings.getNumBins());\n        requestType.append(\":NUMBINS-\").append(numBinsVal).append(\":BINSIZE-\").append(binSizeVal);\n      }\n    } else if (request.isSetSearchQuery()) {\n      requestType.append(\"SEARCH:\");\n      requestType.append(request.getSearchQuery().getRankingMode().name());\n      // Denote when a from user id is present.\n      if (request.getSearchQuery().isSetFromUserIDFilter64()) {\n        requestType.append(\":NETWORK-\")\n            .append(request.getSearchQuery().getFromUserIDFilter64Size());\n      }\n      // Denote when required status ids are present.\n      if (request.getSearchQuery().getSearchStatusIdsSize() > 0) {\n        requestType.append(\":IDS-\").append(request.getSearchQuery().getSearchStatusIdsSize());\n      }\n    }\n    return requestType.toString();\n  }\n\n  private static Map<ThriftQuerySource, FailureRatioCounter> preBuildFailureRatioCounters() {\n    Map<ThriftQuerySource, FailureRatioCounter> counterByQuerySource =\n        new EnumMap<>(ThriftQuerySource.class);\n\n    for (ThriftQuerySource thriftQuerySource : ThriftQuerySource.values()) {\n      FailureRatioCounter counter = new FailureRatioCounter(\"earlybird_logger\", \"query_source\",\n          thriftQuerySource.toString());\n      counterByQuerySource.put(thriftQuerySource, counter);\n    }\n\n    return Maps.immutableEnumMap(counterByQuerySource);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/EarlybirdRequestPostLogger.java",
    "content": "package com.twitter.search.earlybird.common;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.metrics.Timer;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\n\npublic final class EarlybirdRequestPostLogger {\n  private final EarlybirdRequestLogger logger;\n\n  public static EarlybirdRequestPostLogger buildForRoot(\n      int latencyWarnThreshold, Decider decider) {\n\n    EarlybirdRequestLogger requestLogger = EarlybirdRequestLogger.buildForRoot(\n        EarlybirdRequestPostLogger.class.getName(), latencyWarnThreshold, decider);\n\n    return new EarlybirdRequestPostLogger(requestLogger);\n  }\n\n  public static EarlybirdRequestPostLogger buildForShard(\n      int latencyWarnThreshold, Decider decider) {\n\n    EarlybirdRequestLogger requestLogger = EarlybirdRequestLogger.buildForShard(\n        EarlybirdRequestPostLogger.class.getName(), latencyWarnThreshold, decider);\n\n    return new EarlybirdRequestPostLogger(requestLogger);\n  }\n\n  private EarlybirdRequestPostLogger(EarlybirdRequestLogger logger) {\n    this.logger = logger;\n  }\n\n  public void logRequest(EarlybirdRequest request, EarlybirdResponse response, Timer timer) {\n    EarlybirdRequestUtil.updateHitsCounters(request);\n    logger.logRequest(request, response, timer);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/EarlybirdRequestPreLogger.java",
    "content": "package com.twitter.search.earlybird.common;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\n\npublic final class EarlybirdRequestPreLogger {\n  private final EarlybirdRequestLogger logger;\n\n  public static EarlybirdRequestPreLogger buildForRoot(Decider decider) {\n    EarlybirdRequestLogger requestLogger = EarlybirdRequestLogger.buildForRoot(\n        EarlybirdRequestPreLogger.class.getName(), Integer.MAX_VALUE, decider);\n\n    return new EarlybirdRequestPreLogger(requestLogger);\n  }\n\n  public static EarlybirdRequestPreLogger buildForShard(\n      int latencyWarnThreshold, Decider decider) {\n\n    EarlybirdRequestLogger requestLogger = EarlybirdRequestLogger.buildForShard(\n        EarlybirdRequestPreLogger.class.getName(), latencyWarnThreshold, decider);\n\n    return new EarlybirdRequestPreLogger(requestLogger);\n  }\n\n  private EarlybirdRequestPreLogger(EarlybirdRequestLogger logger) {\n    this.logger = logger;\n  }\n\n  public void logRequest(EarlybirdRequest request) {\n    logger.logRequest(request, null, null);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/EarlybirdRequestUtil.java",
    "content": "package com.twitter.search.earlybird.common;\n\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchMovingAverage;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.query.thriftjava.CollectorParams;\nimport com.twitter.search.common.query.thriftjava.CollectorTerminationParams;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchRelevanceOptions;\n\npublic final class EarlybirdRequestUtil {\n  // This logger is setup to log to a separate set of log files (request_info) and use an\n  // async logger so as to not block the searcher thread. See search/earlybird/config/log4j.xml\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdRequestUtil.class);\n\n  @VisibleForTesting\n  static final SearchMovingAverage REQUESTED_NUM_RESULTS_STAT =\n      SearchMovingAverage.export(\"requested_num_results\");\n\n  @VisibleForTesting\n  static final SearchMovingAverage REQUESTED_MAX_HITS_TO_PROCESS_STAT =\n      SearchMovingAverage.export(\"requested_max_hits_to_process\");\n\n  @VisibleForTesting\n  static final SearchMovingAverage REQUESTED_COLLECTOR_PARAMS_MAX_HITS_TO_PROCESS_STAT =\n      SearchMovingAverage.export(\"requested_collector_params_max_hits_to_process\");\n\n  @VisibleForTesting\n  static final SearchMovingAverage REQUESTED_RELEVANCE_OPTIONS_MAX_HITS_TO_PROCESS_STAT =\n      SearchMovingAverage.export(\"requested_relevance_options_max_hits_to_process\");\n\n  @VisibleForTesting\n  static final SearchCounter REQUESTED_MAX_HITS_TO_PROCESS_ARE_DIFFERENT_STAT =\n      SearchCounter.export(\"requested_max_hits_to_process_are_different\");\n\n  private static final SearchRateCounter REQUEST_WITH_MORE_THAN_2K_NUM_RESULTS_STAT =\n      SearchRateCounter.export(\"request_with_more_than_2k_num_result\");\n  private static final SearchRateCounter REQUEST_WITH_MORE_THAN_4K_NUM_RESULTS_STAT =\n      SearchRateCounter.export(\"request_with_more_than_4k_num_result\");\n\n  // Stats for tracking clock skew between earlybird and the client-specified request timestamp.\n  @VisibleForTesting\n  public static final SearchTimerStats CLIENT_CLOCK_DIFF_ABS =\n      SearchTimerStats.export(\"client_clock_diff_abs\", TimeUnit.MILLISECONDS, false, true);\n  @VisibleForTesting\n  public static final SearchTimerStats CLIENT_CLOCK_DIFF_POS =\n      SearchTimerStats.export(\"client_clock_diff_pos\", TimeUnit.MILLISECONDS, false, true);\n  @VisibleForTesting\n  public static final SearchTimerStats CLIENT_CLOCK_DIFF_NEG =\n      SearchTimerStats.export(\"client_clock_diff_neg\", TimeUnit.MILLISECONDS, false, true);\n  @VisibleForTesting\n  public static final SearchRateCounter CLIENT_CLOCK_DIFF_MISSING =\n      SearchRateCounter.export(\"client_clock_diff_missing\");\n\n  private static final int MAX_NUM_RESULTS = 4000;\n  private static final int OLD_MAX_NUM_RESULTS = 2000;\n\n  private EarlybirdRequestUtil() {\n  }\n\n  /**\n   * Logs and fixes some potentially excessive values in the given request.\n   */\n  public static void logAndFixExcessiveValues(EarlybirdRequest request) {\n    ThriftSearchQuery searchQuery = request.getSearchQuery();\n    if (searchQuery != null) {\n      int maxHitsToProcess = 0;\n      int numResultsToReturn = 0;\n\n      if (searchQuery.isSetCollectorParams()) {\n        numResultsToReturn = searchQuery.getCollectorParams().getNumResultsToReturn();\n\n        if (searchQuery.getCollectorParams().isSetTerminationParams()) {\n          maxHitsToProcess =\n              searchQuery.getCollectorParams().getTerminationParams().getMaxHitsToProcess();\n        }\n      }\n\n      if (maxHitsToProcess > 50000) {\n        LOG.warn(\"Excessive max hits in \" + request.toString());\n      }\n\n      // We used to limit number of results to 2000. These two counters help us track if we receive\n      // too many requests with large number of results set.\n      String warningMessageTemplate = \"Exceed %d num result in %s\";\n      if (numResultsToReturn > MAX_NUM_RESULTS) {\n        LOG.warn(String.format(warningMessageTemplate, MAX_NUM_RESULTS, request.toString()));\n        REQUEST_WITH_MORE_THAN_4K_NUM_RESULTS_STAT.increment();\n        searchQuery.getCollectorParams().setNumResultsToReturn(MAX_NUM_RESULTS);\n      } else if (numResultsToReturn > OLD_MAX_NUM_RESULTS) {\n        LOG.warn(String.format(warningMessageTemplate, OLD_MAX_NUM_RESULTS, request.toString()));\n        REQUEST_WITH_MORE_THAN_2K_NUM_RESULTS_STAT.increment();\n      }\n\n      ThriftSearchRelevanceOptions options = searchQuery.getRelevanceOptions();\n      if (options != null) {\n        if (options.getMaxHitsToProcess() > 50000) {\n          LOG.warn(\"Excessive max hits in \" + request.toString());\n        }\n      }\n    }\n  }\n\n  /**\n   * Sets {@code request.searchQuery.collectorParams} if they are not already set.\n   */\n  public static void checkAndSetCollectorParams(EarlybirdRequest request) {\n    ThriftSearchQuery searchQuery = request.getSearchQuery();\n    if (searchQuery == null) {\n      return;\n    }\n\n    if (!searchQuery.isSetCollectorParams()) {\n      searchQuery.setCollectorParams(new CollectorParams());\n    }\n    if (!searchQuery.getCollectorParams().isSetNumResultsToReturn()) {\n      searchQuery.getCollectorParams().setNumResultsToReturn(searchQuery.getNumResults());\n    }\n    if (!searchQuery.getCollectorParams().isSetTerminationParams()) {\n      CollectorTerminationParams terminationParams = new CollectorTerminationParams();\n      if (request.isSetTimeoutMs()) {\n        terminationParams.setTimeoutMs(request.getTimeoutMs());\n      }\n      if (request.isSetMaxQueryCost()) {\n        terminationParams.setMaxQueryCost(request.getMaxQueryCost());\n      }\n      searchQuery.getCollectorParams().setTerminationParams(terminationParams);\n    }\n    setMaxHitsToProcess(searchQuery);\n  }\n\n  // Early birds will only look for maxHitsToProcess in CollectorParameters.TerminationParameters.\n  // Priority to set  CollectorParameters.TerminationParameters.maxHitsToProcess is\n  // 1 Collector parameters\n  // 2 RelevanceParameters\n  // 3 ThrfitQuery.maxHitsToProcess\n  private static void setMaxHitsToProcess(ThriftSearchQuery thriftSearchQuery) {\n    CollectorTerminationParams terminationParams = thriftSearchQuery\n        .getCollectorParams().getTerminationParams();\n    if (!terminationParams.isSetMaxHitsToProcess()) {\n      if (thriftSearchQuery.isSetRelevanceOptions()\n          && thriftSearchQuery.getRelevanceOptions().isSetMaxHitsToProcess()) {\n        terminationParams.setMaxHitsToProcess(\n            thriftSearchQuery.getRelevanceOptions().getMaxHitsToProcess());\n      } else {\n        terminationParams.setMaxHitsToProcess(thriftSearchQuery.getMaxHitsToProcess());\n      }\n    }\n  }\n\n  /**\n   * Creates a copy of the given request and unsets the binary fields to make the logged line for\n   * this request look nicer.\n   */\n  public static EarlybirdRequest copyAndClearUnnecessaryValuesForLogging(EarlybirdRequest request) {\n    EarlybirdRequest copiedRequest = request.deepCopy();\n\n    if (copiedRequest.isSetSearchQuery()) {\n      // These fields are very large and the binary data doesn't play well with formz\n      copiedRequest.getSearchQuery().unsetTrustedFilter();\n      copiedRequest.getSearchQuery().unsetDirectFollowFilter();\n    }\n\n    return copiedRequest;\n  }\n\n  /**\n   * Updates some hit-related stats based on the parameters in the given request.\n   */\n  public static void updateHitsCounters(EarlybirdRequest request) {\n    if ((request == null) || !request.isSetSearchQuery()) {\n      return;\n    }\n\n    ThriftSearchQuery searchQuery = request.getSearchQuery();\n\n    if (searchQuery.isSetNumResults()) {\n      REQUESTED_NUM_RESULTS_STAT.addSample(searchQuery.getNumResults());\n    }\n\n    if (searchQuery.isSetMaxHitsToProcess()) {\n      REQUESTED_MAX_HITS_TO_PROCESS_STAT.addSample(searchQuery.getMaxHitsToProcess());\n    }\n\n    Integer collectorParamsMaxHitsToProcess = null;\n    if (searchQuery.isSetCollectorParams()\n        && searchQuery.getCollectorParams().isSetTerminationParams()\n        && searchQuery.getCollectorParams().getTerminationParams().isSetMaxHitsToProcess()) {\n      collectorParamsMaxHitsToProcess =\n          searchQuery.getCollectorParams().getTerminationParams().getMaxHitsToProcess();\n      REQUESTED_COLLECTOR_PARAMS_MAX_HITS_TO_PROCESS_STAT\n          .addSample(collectorParamsMaxHitsToProcess);\n    }\n\n    Integer relevanceOptionsMaxHitsToProcess = null;\n    if (searchQuery.isSetRelevanceOptions()\n        && searchQuery.getRelevanceOptions().isSetMaxHitsToProcess()) {\n      relevanceOptionsMaxHitsToProcess = searchQuery.getRelevanceOptions().getMaxHitsToProcess();\n      REQUESTED_RELEVANCE_OPTIONS_MAX_HITS_TO_PROCESS_STAT\n          .addSample(relevanceOptionsMaxHitsToProcess);\n    }\n\n    if ((collectorParamsMaxHitsToProcess != null)\n        && (relevanceOptionsMaxHitsToProcess != null)\n        && (collectorParamsMaxHitsToProcess != relevanceOptionsMaxHitsToProcess)) {\n      REQUESTED_MAX_HITS_TO_PROCESS_ARE_DIFFERENT_STAT.increment();\n    }\n  }\n\n  public static boolean isCachingAllowed(EarlybirdRequest request) {\n    return !request.isSetCachingParams() || request.getCachingParams().isCache();\n  }\n\n  /**\n   * Track the clock difference between this server and its client's specified request time.\n   * When there is no clock drift between machines, this will record the inflight time between this\n   * server and the client.\n   *\n   * @param request the incoming earlybird request.\n   */\n  public static void recordClientClockDiff(EarlybirdRequest request) {\n    if (request.isSetClientRequestTimeMs()) {\n      final long timeDiff = System.currentTimeMillis() - request.getClientRequestTimeMs();\n      final long timeDiffAbs = Math.abs(timeDiff);\n      if (timeDiff >= 0) {\n        CLIENT_CLOCK_DIFF_POS.timerIncrement(timeDiffAbs);\n      } else {\n        CLIENT_CLOCK_DIFF_NEG.timerIncrement(timeDiffAbs);\n      }\n      CLIENT_CLOCK_DIFF_ABS.timerIncrement(timeDiffAbs);\n    } else {\n      CLIENT_CLOCK_DIFF_MISSING.increment();\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/EarlybirdThriftBackend.java",
    "content": "package com.twitter.search.earlybird.common;\n\nimport javax.inject.Inject;\nimport javax.inject.Singleton;\n\nimport org.apache.thrift.protocol.TProtocolFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.search.common.util.thrift.ThriftToBytesFilter;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\n\n@Singleton\npublic class EarlybirdThriftBackend extends EarlybirdService.ServiceToClient {\n\n  /**\n   * Wrapping the bytes svc back to a EarlybirdService.ServiceToClient, which\n   * is a EarlybirdService.ServiceIface again.\n   */\n  @Inject\n  public EarlybirdThriftBackend(\n      ThriftToBytesFilter thriftToBytesFilter,\n      Service<byte[], byte[]> byteService,\n      TProtocolFactory protocolFactory) {\n\n    super(thriftToBytesFilter.andThen(byteService), protocolFactory);\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/NonPagingAssert.java",
    "content": "package com.twitter.search.earlybird.common;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchRateCounter;\n\n/**\n * When incremented, a non-paging alert will be triggered. Use this to assert for bad conditions\n * that should generally never happen.\n */\npublic class NonPagingAssert {\n    private static final Logger LOG = LoggerFactory.getLogger(NonPagingAssert.class);\n\n    private static final String ASSERT_STAT_PREFIX = \"non_paging_assert_\";\n\n    private final String name;\n    private final SearchRateCounter assertCounter;\n\n    public NonPagingAssert(String name) {\n        this.name = name;\n        this.assertCounter = SearchRateCounter.export(ASSERT_STAT_PREFIX + name);\n    }\n\n    public void assertFailed() {\n        LOG.error(\"NonPagingAssert failed: {}\", name);\n        assertCounter.increment();\n    }\n\n    public static void assertFailed(String name) {\n        NonPagingAssert nonPagingAssert = new NonPagingAssert(name);\n        nonPagingAssert.assertFailed();\n    }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/RequestResponseForLogging.java",
    "content": "package com.twitter.search.earlybird.common;\n\n\nimport org.apache.thrift.TException;\nimport org.apache.thrift.TSerializer;\nimport org.apache.thrift.protocol.TSimpleJSONProtocol;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\n\npublic class RequestResponseForLogging {\n  private static final Logger LOG = LoggerFactory.getLogger(\n      RequestResponseForLogging.class);\n\n  private static final Logger FAILED_REQUEST_LOG = LoggerFactory.getLogger(\n      RequestResponseForLogging.class.getName() + \".FailedRequests\");\n\n  private final EarlybirdRequest request;\n  private final EarlybirdResponse response;\n\n  public RequestResponseForLogging(EarlybirdRequest request,\n                                   EarlybirdResponse response) {\n    this.request = request;\n    this.response = response;\n  }\n\n  private String serialize(EarlybirdRequest clearedRequest, EarlybirdResponse theResponse) {\n    TSerializer serializer = new TSerializer(new TSimpleJSONProtocol.Factory());\n    try {\n      String requestJson = serializer.toString(clearedRequest);\n      String responseJson = serializer.toString(theResponse);\n      return \"{\\\"request\\\":\" + requestJson + \", \\\"response\\\":\" + responseJson + \"}\";\n    } catch (TException e) {\n      LOG.error(\"Failed to serialize request/response for logging.\", e);\n      return \"\";\n    }\n  }\n\n  /**\n   * Logs the request and response stored in this instance to the failure log file.\n   */\n  public void logFailedRequest() {\n    // Do the serializing/concatting this way so it happens on the background thread for\n    // async logging\n    FAILED_REQUEST_LOG.info(\"{}\", new Object() {\n      @Override\n      public String toString() {\n        return serialize(\n            EarlybirdRequestUtil.copyAndClearUnnecessaryValuesForLogging(request), response);\n      }\n    });\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/RequestResponsePair.java",
    "content": "package com.twitter.search.earlybird.common;\n\nimport org.apache.lucene.search.Query;\n\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\n\npublic class RequestResponsePair {\n  private final EarlybirdRequest request;\n  private final EarlybirdResponse response;\n  private final org.apache.lucene.search.Query luceneQuery;\n\n  // The serialized query in its final form, after various modifications have been applied to it.\n  // As a note, we have some code paths in which this can be null, but I don't really see them\n  // triggered in production right now.\n  private final com.twitter.search.queryparser.query.Query finalSerializedQuery;\n\n  public RequestResponsePair(\n      EarlybirdRequest request,\n      com.twitter.search.queryparser.query.Query finalSerializedQuery,\n      org.apache.lucene.search.Query luceneQuery,\n      EarlybirdResponse response) {\n    this.request = request;\n    this.luceneQuery = luceneQuery;\n    this.response = response;\n    this.finalSerializedQuery = finalSerializedQuery;\n  }\n\n  public String getFinalSerializedQuery() {\n    return finalSerializedQuery != null ? finalSerializedQuery.serialize() : \"N/A\";\n  }\n\n  public EarlybirdRequest getRequest() {\n    return request;\n  }\n\n  public EarlybirdResponse getResponse() {\n    return response;\n  }\n\n  public Query getLuceneQuery() {\n    return luceneQuery;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/UnknownClientRequestForLogging.java",
    "content": "package com.twitter.search.earlybird.common;\n\nimport org.apache.commons.codec.binary.Base64;\nimport org.apache.thrift.TException;\nimport org.apache.thrift.TSerializer;\nimport org.apache.thrift.protocol.TBinaryProtocol;\nimport org.slf4j.Logger;\n\nimport com.twitter.search.common.util.FinagleUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\n\n/**\n * This class logs all requests that misses either the finagle Id or the client Id.\n */\npublic final class UnknownClientRequestForLogging {\n  private static final Logger GENERAL_LOG = org.slf4j.LoggerFactory.getLogger(\n      UnknownClientRequestForLogging.class);\n  private static final Logger LOG = org.slf4j.LoggerFactory.getLogger(\n      UnknownClientRequestForLogging.class.getName() + \".unknownClientRequests\");\n\n  private final String logLine;\n  private final EarlybirdRequest request;\n  private final String clientId;\n  private final String finagleId;\n\n  private final Base64 base64 = new Base64();\n  private final TSerializer serializer = new TSerializer(new TBinaryProtocol.Factory());\n\n  private UnknownClientRequestForLogging(\n      String logLine,\n      EarlybirdRequest request,\n      String clientId,\n      String finagleId) {\n\n    this.logLine = logLine;\n    this.request = request;\n    this.clientId = clientId;\n    this.finagleId = finagleId;\n  }\n\n  /**\n   * Returns an UnknownClientRequestForLogging instance if a client ID is not set on the given\n   * earlybird request. If the request has a client ID set, {@code null} is returned.\n   *\n   * @param logLine Additional information to propagate to the log file, when logging this request.\n   * @param request The earlybird request.\n   */\n  public static UnknownClientRequestForLogging unknownClientRequest(\n      String logLine, EarlybirdRequest request) {\n    String clientId = ClientIdUtil.getClientIdFromRequest(request);\n    String finagleId = FinagleUtil.getFinagleClientName();\n\n    if (clientId.equals(ClientIdUtil.UNSET_CLIENT_ID)) {\n      return new UnknownClientRequestForLogging(logLine, request, clientId, finagleId);\n    } else {\n      return null;\n    }\n  }\n\n  private String asBase64() {\n    try {\n      // Need to make a deepCopy() here, because the request may still be in use (e.g. if we are\n      // doing this in the pre-logger), and we should not be modifying crucial fields on the\n      // EarlybirdRequest in place.\n      EarlybirdRequest clearedRequest = request.deepCopy();\n      clearedRequest.unsetClientRequestTimeMs();\n      return base64.encodeToString(serializer.serialize(clearedRequest));\n    } catch (TException e) {\n      GENERAL_LOG.error(\"Failed to serialize request for logging.\", e);\n      return \"failed_to_serialize\";\n    }\n  }\n\n  public void log() {\n    LOG.info(\"{},{},{},{}\", clientId, finagleId, logLine, asBase64());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/config/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/code/findbugs:jsr305\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/org/apache/commons:commons-lang3\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"3rdparty/jvm/org/yaml:snakeyaml\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/search/common/aurora\",\n        \"src/java/com/twitter/search/common/config\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/util/zookeeper\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/config/EarlybirdConfig.java",
    "content": "package com.twitter.search.earlybird.common.config;\n\nimport java.util.Date;\nimport java.util.List;\nimport java.util.Map;\nimport javax.annotation.Nullable;\n\nimport com.google.common.collect.ImmutableMap;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.search.common.aurora.AuroraInstanceKey;\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.common.config.ConfigFile;\nimport com.twitter.search.common.config.ConfigurationException;\nimport com.twitter.search.common.config.SearchPenguinVersionsConfig;\n\npublic final class EarlybirdConfig {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdConfig.class);\n\n  private static final String DEFAULT_CONFIG_FILE = \"earlybird-search.yml\";\n  private static final String LATE_TWEET_BUFFER_KEY = \"late_tweet_buffer\";\n\n  public static final String EARLYBIRD_ZK_CONFIG_DIR = \"/twitter/search/production/earlybird/\";\n  public static final String EARLYBIRD_CONFIG_DIR = \"earlybird/config\";\n\n  public static final String USER_SNAPSHOT_BASE_DIR = \"user_snapshot_base_dir\";\n\n  private static volatile ConfigFile earlybirdConfig = null;\n  private static volatile Map<String, Object> overrideValueMap = ImmutableMap.of();\n\n  private static String logDirOverride = null;\n  private static AuroraInstanceKey auroraInstanceKey = null;\n\n  private static int adminPort;\n\n  private EarlybirdConfig() { }\n\n  private static final class PenguinVersionHolder {\n    private static final PenguinVersion PENGUIN_VERSION_SINGLETON =\n        SearchPenguinVersionsConfig.getSingleSupportedVersion(\n            EarlybirdProperty.PENGUIN_VERSION.get());\n    private static final byte PENGUIN_VERSION_BYTE_VALUE =\n        PENGUIN_VERSION_SINGLETON.getByteValue();\n  }\n\n  public static byte getPenguinVersionByte() {\n    return PenguinVersionHolder.PENGUIN_VERSION_BYTE_VALUE;\n  }\n\n  public static PenguinVersion getPenguinVersion() {\n    return PenguinVersionHolder.PENGUIN_VERSION_SINGLETON;\n  }\n\n  /**\n   * Reads the earlybird configuration from the given file.\n   */\n  public static synchronized void init(@Nullable String configFile) {\n    if (earlybirdConfig == null) {\n      String file = configFile == null ? DEFAULT_CONFIG_FILE : configFile;\n      earlybirdConfig = new ConfigFile(EARLYBIRD_CONFIG_DIR, file);\n    }\n  }\n\n  public static synchronized void setOverrideValues(Map<String, Object> overrideValues) {\n    overrideValueMap = ImmutableMap.copyOf(overrideValues);\n  }\n\n  /**\n   * Pack all values in a string that can be printed for informational purposes.\n   * @return the string.\n   */\n  public static String allValuesAsString() {\n    Map<String, String> stringMap = earlybirdConfig.getStringMap();\n\n    StringBuilder stringBuilder = new StringBuilder();\n\n    stringBuilder.append(\"Config environment: \" + Config.getEnvironment() + \"\\n\\n\");\n    stringBuilder.append(\n        String.format(\"Values from earlybird-search.yml (total %d):\\n\", stringMap.size()));\n\n    stringMap.forEach((key, value) -> {\n      stringBuilder.append(String.format(\"  %s: %s\\n\", key, value.toString()));\n      if (overrideValueMap.containsKey(key)) {\n        stringBuilder.append(String.format(\n          \"    override value: %s\\n\", overrideValueMap.get(key).toString()));\n      }\n    });\n\n    stringBuilder.append(String.format(\n        \"\\n\\nAll command-line overrides (total: %d):\\n\", overrideValueMap.size()));\n    overrideValueMap.forEach((key, value) -> {\n      stringBuilder.append(String.format(\"  %s: %s\\n\", key, value.toString()));\n    });\n\n    return stringBuilder.toString();\n  }\n\n  /**\n   * Returns the value of the given property as a string. If the property is not set, a runtime\n   * exception is thrown.\n   */\n  public static String getString(String property) {\n    Object overrideValue = overrideValueMap.get(property);\n    if (overrideValue != null) {\n      return (String) overrideValue;\n    }\n\n    try {\n      return earlybirdConfig.getString(property);\n    } catch (ConfigurationException e) {\n      LOG.error(\"Fatal error: could not get config string \" + property, e);\n      throw new RuntimeException(e);\n    }\n  }\n\n  /**\n   * Returns the value of the given property as a string.\n   */\n  public static String getString(String property, String defaultValue) {\n    Object overrideValue = overrideValueMap.get(property);\n    if (overrideValue != null) {\n      return (String) overrideValue;\n    }\n\n    return earlybirdConfig.getString(property, defaultValue);\n  }\n\n  /**\n   * Returns the value of the given property as an integer. If the property is not set, a runtime\n   * exception is thrown.\n   */\n  public static int getInt(String property) {\n    Object overrideValue = overrideValueMap.get(property);\n    if (overrideValue != null) {\n      return (int) overrideValue;\n    }\n\n    try {\n      return earlybirdConfig.getInt(property);\n    } catch (ConfigurationException e) {\n      LOG.error(\"Fatal error: could not get config int \" + property, e);\n      throw new RuntimeException(e);\n    }\n  }\n\n  /**\n   * Returns the value of the given property as an integer.\n   */\n  public static int getInt(String property, int defaultValue) {\n    Object overrideValue = overrideValueMap.get(property);\n    if (overrideValue != null) {\n      return (int) overrideValue;\n    }\n\n    return earlybirdConfig.getInt(property, defaultValue);\n  }\n\n  /**\n   * Returns the value of the given property as a double.\n   */\n  public static double getDouble(String property, double defaultValue) {\n    Object overrideValue = overrideValueMap.get(property);\n    if (overrideValue != null) {\n      return (double) overrideValue;\n    }\n\n    return earlybirdConfig.getDouble(property, defaultValue);\n  }\n\n  /**\n   * Returns the value of the given property as a long. If the property is not set, a runtime\n   * exception is thrown.\n   */\n  public static long getLong(String property) {\n    Object overrideValue = overrideValueMap.get(property);\n    if (overrideValue != null) {\n      return (long) overrideValue;\n    }\n\n    try {\n      return earlybirdConfig.getLong(property);\n    } catch (ConfigurationException e) {\n      LOG.error(\"Fatal error: could not get config long \" + property, e);\n      throw new RuntimeException(e);\n    }\n  }\n\n  /**\n   * Returns the value of the given property as a long.\n   */\n  public static long getLong(String property, long defaultValue) {\n    Object overrideValue = overrideValueMap.get(property);\n    if (overrideValue != null) {\n      return (long) overrideValue;\n    }\n\n    return earlybirdConfig.getLong(property, defaultValue);\n  }\n\n  /**\n   * Returns the value of the given property as a boolean. If the property is not set, a runtime\n   * exception is thrown.\n   */\n  public static boolean getBool(String property) {\n    Object overrideValue = overrideValueMap.get(property);\n    if (overrideValue != null) {\n      return (boolean) overrideValue;\n    }\n\n    try {\n      return earlybirdConfig.getBool(property);\n    } catch (ConfigurationException e) {\n      LOG.error(\"Fatal error: could not get config boolean \" + property, e);\n      throw new RuntimeException(e);\n    }\n  }\n\n  /**\n   * Returns the value of the given property as a boolean.\n   */\n  public static boolean getBool(String property, boolean defaultValue) {\n    Object overrideValue = overrideValueMap.get(property);\n    if (overrideValue != null) {\n      return (boolean) overrideValue;\n    }\n\n    return earlybirdConfig.getBool(property, defaultValue);\n  }\n\n  /**\n   * Returns the value of the given property as a date.\n   */\n  public static Date getDate(String property) {\n    Object overrideValue = overrideValueMap.get(property);\n    if (overrideValue != null) {\n      return (Date) overrideValue;\n    }\n\n    Date date = (Date) earlybirdConfig.getObject(property, null);\n    if (date == null) {\n      throw new RuntimeException(\"Could not get config date: \" + property);\n    }\n    return date;\n  }\n\n  /**\n   * Returns the value of the given property as a list of strings.\n   */\n  public static List<String> getListOfStrings(String property) {\n    Object overrideValue = overrideValueMap.get(property);\n    if (overrideValue != null) {\n      return (List<String>) overrideValue;\n    }\n\n    List<String> list = (List<String>) earlybirdConfig.getObject(property, null);\n    if (list == null) {\n      throw new RuntimeException(\"Could not get list of strings: \" + property);\n    }\n    return list;\n  }\n\n  /**\n   * Returns the value of the given property as a map.\n   */\n  @SuppressWarnings(\"unchecked\")\n  public static Map<String, Object> getMap(String property) {\n    Map<String, Object> map = (Map<String, Object>) earlybirdConfig.getObject(property, null);\n    if (map == null) {\n      throw new RuntimeException(\"Could not find config property: \" + property);\n    }\n    return map;\n  }\n\n  public static int getMaxSegmentSize() {\n    return EarlybirdConfig.getInt(\"max_segment_size\", 1 << 16);\n  }\n\n  /**\n   * Returns the log properties file.\n   */\n  public static String getLogPropertiesFile() {\n    try {\n      String filename = earlybirdConfig.getString(\"log_properties_filename\");\n      return earlybirdConfig.getConfigFilePath(filename);\n    } catch (ConfigurationException e) {\n      // Print here rather than use LOG - log was probably not initialized yet.\n      LOG.error(\"Fatal error: could not get log properties file\", e);\n      throw new RuntimeException(e);\n    }\n  }\n\n  /**\n   * Returns the log directory.\n   */\n  public static String getLogDir() {\n    if (logDirOverride != null) {\n      return logDirOverride;\n    } else {\n      return EarlybirdConfig.getString(\"log_dir\");\n    }\n  }\n\n  public static void overrideLogDir(String logDir) {\n    EarlybirdConfig.logDirOverride = logDir;\n  }\n\n  public static int getThriftPort() {\n    return EarlybirdProperty.THRIFT_PORT.get();\n  }\n\n  public static int getWarmUpThriftPort() {\n    return EarlybirdProperty.WARMUP_THRIFT_PORT.get();\n  }\n\n  public static int getSearcherThreads() {\n    return EarlybirdProperty.SEARCHER_THREADS.get();\n  }\n\n  public static int getLateTweetBuffer() {\n    return getInt(LATE_TWEET_BUFFER_KEY);\n  }\n\n  public static int getAdminPort() {\n    return adminPort;\n  }\n\n  public static void setAdminPort(int adminPort) {\n    EarlybirdConfig.adminPort = adminPort;\n  }\n\n  public static boolean isRealtimeOrProtected() {\n    String earlybirdName = EarlybirdProperty.EARLYBIRD_NAME.get();\n    return earlybirdName.contains(\"realtime\") || earlybirdName.contains(\"protected\");\n  }\n\n  public static boolean consumeUserScrubGeoEvents() {\n    return EarlybirdProperty.CONSUME_GEO_SCRUB_EVENTS.get();\n  }\n\n  @Nullable\n  public static AuroraInstanceKey getAuroraInstanceKey() {\n    return auroraInstanceKey;\n  }\n\n  public static void setAuroraInstanceKey(AuroraInstanceKey auroraInstanceKey) {\n    EarlybirdConfig.auroraInstanceKey = auroraInstanceKey;\n  }\n\n  public static boolean isAurora() {\n    return auroraInstanceKey != null;\n  }\n\n  public static void setForTests(String property, Object value) {\n    earlybirdConfig.setForTests(DEFAULT_CONFIG_FILE, property, value);\n  }\n\n  public static synchronized void clearForTests() {\n    earlybirdConfig = new ConfigFile(EARLYBIRD_CONFIG_DIR, DEFAULT_CONFIG_FILE);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/config/EarlybirdProperty.java",
    "content": "package com.twitter.search.earlybird.common.config;\n\nimport java.lang.reflect.Modifier;\nimport java.util.Arrays;\nimport java.util.List;\nimport java.util.function.BiFunction;\nimport java.util.function.Function;\nimport java.util.stream.Collectors;\n\nimport com.google.common.collect.ImmutableList;\n\nimport com.twitter.app.Flag;\nimport com.twitter.app.Flaggable;\nimport com.twitter.app.Flags;\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier;\n\n/**\n * Stateless class that represents an Earlybird property that can be specified by a command line\n * flag.\n * <p>\n * This is a regular Java class instead of enum to have a generic type.\n *\n * @param <T>\n */\npublic final class EarlybirdProperty<T> {\n\n  private static final class PropertyType<T> {\n\n    private static final PropertyType<Boolean> BOOLEAN = new PropertyType<>(\n        Flaggable.ofJavaBoolean(), EarlybirdConfig::getBool, EarlybirdConfig::getBool);\n\n    private static final PropertyType<Integer> INT = new PropertyType<>(\n        Flaggable.ofJavaInteger(), EarlybirdConfig::getInt, EarlybirdConfig::getInt);\n\n    private static final PropertyType<String> STRING = new PropertyType<>(\n        Flaggable.ofString(), EarlybirdConfig::getString, EarlybirdConfig::getString);\n\n    private final Flaggable<T> flaggable;\n    private final Function<String, T> getter;\n    private final BiFunction<String, T, T> getterWithDefault;\n\n    private PropertyType(Flaggable<T> flaggable, Function<String, T> getter,\n                         BiFunction<String, T, T> getterWithDefault) {\n      this.flaggable = flaggable;\n      this.getter = getter;\n      this.getterWithDefault = getterWithDefault;\n    }\n  }\n\n  public static final EarlybirdProperty<String> PENGUIN_VERSION =\n      new EarlybirdProperty<>(\n          \"penguin_version\",\n          \"The penguin version to index.\",\n          PropertyType.STRING,\n          false);\n\n  public static final EarlybirdProperty<Integer> THRIFT_PORT = new EarlybirdProperty<>(\n      \"thrift_port\",\n      \"override thrift port from config file\",\n      PropertyType.INT,\n      false);\n\n  public static final EarlybirdProperty<Integer> WARMUP_THRIFT_PORT = new EarlybirdProperty<>(\n      \"warmup_thrift_port\",\n      \"override warmup thrift port from config file\",\n      PropertyType.INT,\n      false);\n\n  public static final EarlybirdProperty<Integer> SEARCHER_THREADS = new EarlybirdProperty<>(\n      \"searcher_threads\",\n      \"override number of searcher threads from config file\",\n      PropertyType.INT,\n      false);\n\n  public static final EarlybirdProperty<String> EARLYBIRD_TIER = new EarlybirdProperty<>(\n      \"earlybird_tier\",\n      \"the earlybird tier (e.g. tier1), used on Aurora\",\n      PropertyType.STRING,\n      true);\n\n  public static final EarlybirdProperty<Integer> REPLICA_ID = new EarlybirdProperty<>(\n      \"replica_id\",\n      \"the ID in a partition, used on Aurora\",\n      PropertyType.INT,\n      true);\n\n  public static final EarlybirdProperty<Integer> PARTITION_ID = new EarlybirdProperty<>(\n      \"partition_id\",\n      \"partition ID, used on Aurora\",\n      PropertyType.INT,\n      true);\n\n  public static final EarlybirdProperty<Integer> NUM_PARTITIONS = new EarlybirdProperty<>(\n      \"num_partitions\",\n      \"number of partitions, used on Aurora\",\n      PropertyType.INT,\n      true);\n\n  public static final EarlybirdProperty<Integer> NUM_INSTANCES = new EarlybirdProperty<>(\n      \"num_instances\",\n      \"number of instances in the job, used on Aurora\",\n      PropertyType.INT,\n      true);\n\n  public static final EarlybirdProperty<Integer> SERVING_TIMESLICES = new EarlybirdProperty<>(\n      \"serving_timeslices\",\n      \"number of time slices to serve, used on Aurora\",\n      PropertyType.INT,\n      true);\n\n  public static final EarlybirdProperty<String> ROLE = new EarlybirdProperty<>(\n      \"role\",\n      \"Role in the service path of Earlybird\",\n      PropertyType.STRING,\n      true,\n      true);\n\n  public static final EarlybirdProperty<String> EARLYBIRD_NAME = new EarlybirdProperty<>(\n      \"earlybird_name\",\n      \"Name in the service path of Earlybird without hash partition suffix\",\n      PropertyType.STRING,\n      true,\n      true);\n\n  public static final EarlybirdProperty<String> ENV = new EarlybirdProperty<>(\n      \"env\",\n      \"Environment in the service path of Earlybird\",\n      PropertyType.STRING,\n      true,\n      true);\n\n  public static final EarlybirdProperty<String> ZONE = new EarlybirdProperty<>(\n      \"zone\",\n      \"Zone (data center) in the service path of Earlybird\",\n      PropertyType.STRING,\n      true,\n      true);\n\n  public static final EarlybirdProperty<String> DL_URI = new EarlybirdProperty<>(\n      \"dl_uri\",\n      \"DistributedLog URI for default DL reader\",\n      PropertyType.STRING,\n      false);\n\n  public static final EarlybirdProperty<String> USER_UPDATES_DL_URI = new EarlybirdProperty<>(\n      \"user_updates_dl_uri\",\n      \"DistributedLog URI for user updates DL reader\",\n      PropertyType.STRING,\n      false);\n\n  public static final EarlybirdProperty<String> ANTISOCIAL_USERUPDATES_DL_STREAM =\n      new EarlybirdProperty<>(\n          \"antisocial_userupdates_dl_stream\",\n          \"DL stream name for antisocial user updates without DL version suffix\",\n          PropertyType.STRING,\n          false);\n\n  public static final EarlybirdProperty<String> ZK_APP_ROOT = new EarlybirdProperty<>(\n      \"zk_app_root\",\n      \"SZooKeeper base root path for this application\",\n      PropertyType.STRING,\n      true);\n\n  public static final EarlybirdProperty<Boolean> SEGMENT_LOAD_FROM_HDFS_ENABLED =\n      new EarlybirdProperty<>(\n          \"segment_load_from_hdfs_enabled\",\n          \"Whether to load segment data from HDFS\",\n          PropertyType.BOOLEAN,\n          false);\n\n  public static final EarlybirdProperty<Boolean> SEGMENT_FLUSH_TO_HDFS_ENABLED =\n      new EarlybirdProperty<>(\n          \"segment_flush_to_hdfs_enabled\",\n          \"Whether to flush segment data to HDFS\",\n          PropertyType.BOOLEAN,\n          false);\n\n  public static final EarlybirdProperty<String> HDFS_SEGMENT_SYNC_DIR = new EarlybirdProperty<>(\n      \"hdfs_segment_sync_dir\",\n      \"HDFS directory to sync segment data\",\n      PropertyType.STRING,\n      false);\n\n  public static final EarlybirdProperty<String> HDFS_SEGMENT_UPLOAD_DIR = new EarlybirdProperty<>(\n      \"hdfs_segment_upload_dir\",\n      \"HDFS directory to upload segment data\",\n      PropertyType.STRING,\n      false);\n\n  public static final EarlybirdProperty<Boolean> ARCHIVE_DAILY_STATUS_BATCH_FLUSHING_ENABLED =\n      new EarlybirdProperty<>(\n          \"archive_daily_status_batch_flushing_enabled\",\n          \"Whether to enable archive daily status batch flushing\",\n          PropertyType.BOOLEAN,\n          false);\n\n  public static final EarlybirdProperty<String> HDFS_INDEX_SYNC_DIR = new EarlybirdProperty<>(\n      \"hdfs_index_sync_dir\",\n      \"HDFS directory to sync index data\",\n      PropertyType.STRING,\n      true);\n\n  public static final EarlybirdProperty<Boolean> READ_INDEX_FROM_PROD_LOCATION =\n      new EarlybirdProperty<>(\n      \"read_index_from_prod_location\",\n      \"Read index from prod to speed up startup on staging / loadtest\",\n      PropertyType.BOOLEAN,\n      false);\n\n  public static final EarlybirdProperty<Boolean> USE_DECIDER_OVERLAY = new EarlybirdProperty<>(\n      \"use_decider_overlay\",\n      \"Whether to use decider overlay\",\n      PropertyType.BOOLEAN,\n      false);\n\n  public static final EarlybirdProperty<String> DECIDER_OVERLAY_CONFIG = new EarlybirdProperty<>(\n      \"decider_overlay_config\",\n      \"Path to decider overlay config\",\n      PropertyType.STRING,\n      false);\n\n  public static final EarlybirdProperty<Integer> MAX_CONCURRENT_SEGMENT_INDEXERS =\n      new EarlybirdProperty<>(\n        \"max_concurrent_segment_indexers\",\n        \"Maximum number of segments indexed concurrently\",\n        PropertyType.INT,\n        false);\n\n  public static final EarlybirdProperty<Boolean> TF_MODELS_ENABLED =\n      new EarlybirdProperty<>(\n        \"tf_models_enabled\",\n        \"Whether tensorflow models should be loaded\",\n        PropertyType.BOOLEAN,\n        false);\n\n  public static final EarlybirdProperty<String> TF_MODELS_CONFIG_PATH =\n      new EarlybirdProperty<>(\n        \"tf_models_config_path\",\n        \"The configuration path of the yaml file containing the list of tensorflow models to load.\",\n        PropertyType.STRING,\n        false);\n\n  public static final EarlybirdProperty<Integer> TF_INTER_OP_THREADS =\n      new EarlybirdProperty<>(\n        \"tf_inter_op_threads\",\n        \"How many tensorflow inter op threads to use. See TF documentation for more information.\",\n        PropertyType.INT,\n        false);\n\n  public static final EarlybirdProperty<Integer> TF_INTRA_OP_THREADS =\n      new EarlybirdProperty<>(\n        \"tf_intra_op_threads\",\n        \"How many tensorflow intra op threads to use. See TF documentation for more information.\",\n        PropertyType.INT,\n        false);\n\n  public static final EarlybirdProperty<Integer> MAX_ALLOWED_REPLICAS_NOT_IN_SERVER_SET =\n      new EarlybirdProperty<>(\n          \"max_allowed_replicas_not_in_server_set\",\n          \"How many replicas are allowed to be missing from the Earlybird server set.\",\n          PropertyType.INT,\n          false);\n\n  public static final EarlybirdProperty<Boolean> CHECK_NUM_REPLICAS_IN_SERVER_SET =\n      new EarlybirdProperty<>(\n          \"check_num_replicas_in_server_set\",\n          \"Whether CoordinatedEarlybirdActions should check the number of alive replicas\",\n          PropertyType.BOOLEAN,\n          false);\n\n  public static final EarlybirdProperty<Integer> MAX_QUEUE_SIZE =\n      new EarlybirdProperty<>(\n          \"max_queue_size\",\n          \"Maximum size of searcher worker executor queue. If <= 0 queue is unbounded.\",\n          PropertyType.INT,\n          false);\n\n  public static final EarlybirdProperty<String> KAFKA_ENV =\n      new EarlybirdProperty<>(\n          \"kafka_env\",\n          \"The environment to use for kafka topics.\",\n          PropertyType.STRING,\n          false);\n  public static final EarlybirdProperty<String> KAFKA_PATH =\n      new EarlybirdProperty<>(\n          \"kafka_path\",\n          \"Wily path to the Search kafka cluster.\",\n          PropertyType.STRING,\n          false);\n  public static final EarlybirdProperty<String> TWEET_EVENTS_KAFKA_PATH =\n      new EarlybirdProperty<>(\n          \"tweet_events_kafka_path\",\n          \"Wily path to the tweet-events kafka cluster.\",\n          PropertyType.STRING,\n          false);\n  public static final EarlybirdProperty<String> USER_UPDATES_KAFKA_TOPIC =\n      new EarlybirdProperty<>(\n          \"user_updates_topic\",\n          \"Name of the Kafka topic that contain user updates.\",\n          PropertyType.STRING,\n          false);\n  public static final EarlybirdProperty<String> USER_SCRUB_GEO_KAFKA_TOPIC =\n      new EarlybirdProperty<>(\n          \"user_scrub_geo_topic\",\n          \"Name of the Kafka topic that contain UserScrubGeoEvents.\",\n          PropertyType.STRING,\n          false);\n  public static final EarlybirdProperty<String> EARLYBIRD_SCRUB_GEN =\n      new EarlybirdProperty<>(\n          \"earlybird_scrub_gen\",\n          \"SCRUB_GEN TO DEPLOY\",\n          PropertyType.STRING,\n          false);\n  public static final EarlybirdProperty<Boolean> CONSUME_GEO_SCRUB_EVENTS =\n      new EarlybirdProperty<>(\n        \"consume_geo_scrub_events\",\n        \"Whether to consume user scrub geo events or not\",\n        PropertyType.BOOLEAN,\n        false);\n\n  private static final List<EarlybirdProperty<?>> ALL_PROPERTIES =\n      Arrays.stream(EarlybirdProperty.class.getDeclaredFields())\n          .filter(field ->\n              (field.getModifiers() & Modifier.STATIC) > 0\n                && field.getType() == EarlybirdProperty.class)\n          .map(field -> {\n            try {\n              return (EarlybirdProperty<?>) field.get(EarlybirdProperty.class);\n            } catch (Exception e) {\n              throw new RuntimeException(e);\n            }\n          })\n          .collect(Collectors.collectingAndThen(Collectors.toList(), ImmutableList::copyOf));\n\n  public static ServiceIdentifier getServiceIdentifier() {\n    return new ServiceIdentifier(\n        ROLE.get(),\n        EARLYBIRD_NAME.get(),\n        ENV.get(),\n        ZONE.get());\n  }\n\n  private final String name;\n  private final String help;\n  private final PropertyType<T> type;\n  private final boolean requiredOnAurora;\n  private final boolean requiredOnDedicated;\n\n  private EarlybirdProperty(String name, String help, PropertyType<T> type,\n                            boolean requiredOnAurora) {\n    this(name, help, type, requiredOnAurora, false);\n  }\n\n  private EarlybirdProperty(String name, String help, PropertyType<T> type,\n                            boolean requiredOnAurora, boolean requiredOnDedicated) {\n    this.name = name;\n    this.help = help;\n    this.type = type;\n    this.requiredOnAurora = requiredOnAurora;\n    this.requiredOnDedicated = requiredOnDedicated;\n  }\n\n  public String name() {\n    return name;\n  }\n\n  public boolean isRequiredOnAurora() {\n    return requiredOnAurora;\n  }\n\n  public boolean isRequiredOnDedicated() {\n    return requiredOnDedicated;\n  }\n\n  public Flag<T> createFlag(Flags flags) {\n    return flags.createMandatory(name, help, null, type.flaggable);\n  }\n\n  public T get() {\n    return type.getter.apply(name);\n  }\n\n  public T get(T devaultValue) {\n    return type.getterWithDefault.apply(name, devaultValue);\n  }\n\n  public static EarlybirdProperty[] values() {\n    return ALL_PROPERTIES.toArray(new EarlybirdProperty[0]);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/userupdates/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/commons-io\",\n        \"3rdparty/jvm/geo/google:geoGoogle\",\n        \"3rdparty/jvm/org/apache/bookkeeper:bookkeeper-server\",\n        \"3rdparty/jvm/org/apache/bookkeeper:bookkeeper-twitter-science-provider\",\n        \"3rdparty/jvm/org/apache/hadoop:hadoop-client-default\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-common\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-smartcn\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-core\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-facet\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"3rdparty/src/jvm/com/twitter/scalding:core\",\n        \"3rdparty/src/jvm/com/twitter/scalding:date\",\n        \"3rdparty/src/jvm/com/twitter/scalding:parquet\",\n        \"decider/src/main/scala\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/util:system-mocks\",\n        \"src/java/com/twitter/common_internal/hadoop\",\n        \"src/java/com/twitter/search/common/logging\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/partitioning/snowflakeparser\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/common/util/hash\",\n        \"src/java/com/twitter/search/common/util/io\",\n        \"src/java/com/twitter/search/common/util/io:dl-reader-writer\",\n        \"src/java/com/twitter/search/common/util/io:flushable\",\n        \"src/java/com/twitter/search/common/util/io:record-reader-api\",\n        \"src/java/com/twitter/search/earlybird/common/config\",\n        \"src/scala/com/twitter/scalding_internal/error_handling\",\n        \"src/scala/com/twitter/scalding_internal/multiformat\",\n        \"src/scala/com/twitter/scalding_internal/source\",\n        \"src/scala/com/twitter/search/user_table/sources\",\n        \"src/thrift/com/twitter/search/common:indexing-java\",\n        \"src/thrift/com/twitter/tweetypie:events-java\",\n        \"util/util-core:scala\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/userupdates/UserScrubGeoMap.java",
    "content": "package com.twitter.search.earlybird.common.userupdates;\n\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.TimeUnit;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchCustomGauge;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.tweetypie.thriftjava.UserScrubGeoEvent;\n\n/**\n * Map of users who have actioned to delete location data from their tweets. UserID's are mapped\n * to the maxTweetId that will eventually be scrubbed from the index (userId -> maxTweetId).\n *\n * ConcurrentHashMap is thread safe without synchronizing the whole map. Reads can happen very fast\n * while writes are done with a lock. This is ideal since many Earlybird Searcher threads could\n * be reading from the map at once, whereas we will only be adding to the map via kafka.\n *\n * This map is checked against to filter out tweets that should not be returned to geo queries.\n * See: go/realtime-geo-filtering\n */\npublic class UserScrubGeoMap {\n  // The number of geo events that contain a user ID already present in the map. This count is used\n  // to verify the number of users in the map against the number of events consumed from kafka.\n  private static final SearchCounter USER_SCRUB_GEO_EVENT_EXISTING_USER_COUNT =\n      SearchCounter.export(\"user_scrub_geo_event_existing_user_count\");\n  public static final SearchTimerStats USER_SCRUB_GEO_EVENT_LAG_STAT =\n      SearchTimerStats.export(\"user_scrub_geo_event_lag\",\n          TimeUnit.MILLISECONDS,\n          false,\n          true);\n  private ConcurrentHashMap<Long, Long> map;\n\n  public UserScrubGeoMap() {\n    map = new ConcurrentHashMap<>();\n    SearchCustomGauge.export(\"num_users_in_geo_map\", this::getNumUsersInMap);\n  }\n\n  /**\n   * Ensure that the max_tweet_id in the userScrubGeoEvent is greater than the one already stored\n   * in the map for the given user id (if any) before updating the entry for this user.\n   * This will protect Earlybirds from potential issues where out of date UserScrubGeoEvents\n   * appear in the incoming Kafka stream.\n   *\n   * @param userScrubGeoEvent\n   */\n  public void indexUserScrubGeoEvent(UserScrubGeoEvent userScrubGeoEvent) {\n    long userId = userScrubGeoEvent.getUser_id();\n    long newMaxTweetId = userScrubGeoEvent.getMax_tweet_id();\n    long oldMaxTweetId = map.getOrDefault(userId, 0L);\n    if (map.containsKey(userId)) {\n      USER_SCRUB_GEO_EVENT_EXISTING_USER_COUNT.increment();\n    }\n    map.put(userId, Math.max(oldMaxTweetId, newMaxTweetId));\n    USER_SCRUB_GEO_EVENT_LAG_STAT.timerIncrement(computeEventLag(newMaxTweetId));\n  }\n\n  /**\n   * A tweet is geo scrubbed if it is older than the max tweet id that is scrubbed for the tweet's\n   * author.\n   * If there is no entry for the tweet's author in the map, then the tweet is not geo scrubbed.\n   *\n   * @param tweetId\n   * @param fromUserId\n   * @return\n   */\n  public boolean isTweetGeoScrubbed(long tweetId, long fromUserId) {\n    return tweetId <= map.getOrDefault(fromUserId, 0L);\n  }\n\n  /**\n   * The lag (in milliseconds) from when a UserScrubGeoEvent is created, until it is applied to the\n   * UserScrubGeoMap. Take the maxTweetId found in the current event and convert it to a timestamp.\n   * The maxTweetId will give us a timestamp closest to when Tweetypie processes macaw-geo requests.\n   *\n   * @param maxTweetId\n   * @return\n   */\n  private long computeEventLag(long maxTweetId) {\n    long eventCreatedAtTime = SnowflakeIdParser.getTimestampFromTweetId(maxTweetId);\n    return System.currentTimeMillis() - eventCreatedAtTime;\n  }\n\n  public long getNumUsersInMap() {\n    return map.size();\n  }\n\n  public ConcurrentHashMap<Long, Long> getMap() {\n    return map;\n  }\n\n  public boolean isEmpty() {\n    return map.isEmpty();\n  }\n\n  public boolean isSet(long userId) {\n    return map.containsKey(userId);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/userupdates/UserTable.java",
    "content": "package com.twitter.search.earlybird.common.userupdates;\n\nimport java.util.Iterator;\nimport java.util.concurrent.atomic.AtomicReference;\nimport java.util.function.Predicate;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.util.hash.GeneralLongHashFunction;\n\n/**\n * Table containing metadata about users, like NSFW or Antisocial status.\n * Used for result filtering.\n */\npublic class UserTable {\n  private static final Logger LOG = LoggerFactory.getLogger(UserTable.class);\n\n  @VisibleForTesting // Not final for testing.\n  protected static long userUpdateTableMaxCapacity = 1L << 30;\n\n  private static final int DEFAULT_INITIAL_CAPACITY = 1024;\n  private static final int BYTE_WIDTH = 8;\n\n  private static final String USER_TABLE_CAPACITY = \"user_table_capacity\";\n  private static final String USER_TABLE_SIZE = \"user_table_size\";\n  private static final String\n      USER_NUM_USERS_WITH_NO_BITS_SET = \"user_table_users_with_no_bits_set\";\n  private static final String USER_TABLE_ANTISOCIAL_USERS = \"user_table_antisocial_users\";\n  private static final String USER_TABLE_OFFENSIVE_USERS = \"user_table_offensive_users\";\n  private static final String USER_TABLE_NSFW_USERS = \"user_table_nsfw_users\";\n  private static final String USER_TABLE_IS_PROTECTED_USERS = \"user_table_is_protected_users\";\n\n  /**\n   * number of users filtered\n   */\n  private static final SearchRateCounter USER_TABLE_USERS_FILTERED_COUNTER =\n      new SearchRateCounter(\"user_table_users_filtered\");\n\n  private SearchLongGauge userTableCapacity;\n  private SearchLongGauge userTableSize;\n  private SearchLongGauge userTableNumUsersWithNoBitsSet;\n  private SearchLongGauge userTableAntisocialUsers;\n  private SearchLongGauge userTableOffensiveUsers;\n  private SearchLongGauge userTableNsfwUsers;\n  private SearchLongGauge userTableIsProtectedUsers;\n\n  private final Predicate<Long> userIdFilter;\n  private long lastRecordTimestamp;\n\n  private static final class HashTable {\n    private int numUsersInTable;\n    private int numUsersWithNoBitsSet;\n    // size 8 array contains the number of users who have the bit set at the index (0-7) position\n    // e.g. setBitCounts[0] stores the number of users who have the 0 bit set in their bytes\n    private long[] setBitCounts;\n\n    private final long[] hash;\n    private final byte[] bits;\n\n    private final int hashMask;\n\n    HashTable(int size) {\n      this.hash = new long[size];\n      this.bits = new byte[size];\n      this.hashMask = size - 1;\n      this.numUsersInTable = 0;\n      this.setBitCounts = new long[BYTE_WIDTH];\n    }\n\n    protected int hashSize() {\n      return hash.length;\n    }\n\n    // If we want to decrease the number of users in the table, we can delete as many users\n    // as this table returns, by calling filterTableAndCountValidItems.\n    public void setCountOfNumUsersWithNoBitsSet() {\n      int count = 0;\n      for (int i = 0; i < hash.length; i++) {\n        if ((hash[i] > 0) && (bits[i] == 0)) {\n          count++;\n        }\n      }\n\n      numUsersWithNoBitsSet = count;\n    }\n\n    public void setSetBitCounts() {\n      long[] counts = new long[BYTE_WIDTH];\n      for (int i = 0; i < hash.length; i++) {\n        if (hash[i] > 0) {\n          int tempBits = bits[i] & 0xff;\n          int curBitPos = 0;\n          while (tempBits != 0) {\n            if ((tempBits & 1) != 0) {\n              counts[curBitPos]++;\n            }\n            tempBits = tempBits >>> 1;\n            curBitPos++;\n          }\n        }\n      }\n      setBitCounts = counts;\n    }\n  }\n\n  public static final int ANTISOCIAL_BIT = 1;\n  public static final int OFFENSIVE_BIT = 1 << 1;\n  public static final int NSFW_BIT = 1 << 2;\n  public static final int IS_PROTECTED_BIT = 1 << 3;\n\n  public long getLastRecordTimestamp() {\n    return this.lastRecordTimestamp;\n  }\n\n  public void setLastRecordTimestamp(long lastRecordTimestamp) {\n    this.lastRecordTimestamp = lastRecordTimestamp;\n  }\n\n  public void setOffensive(long userID, boolean offensive) {\n    set(userID, OFFENSIVE_BIT, offensive);\n  }\n\n  public void setAntisocial(long userID, boolean antisocial) {\n    set(userID, ANTISOCIAL_BIT, antisocial);\n  }\n\n  public void setNSFW(long userID, boolean nsfw) {\n    set(userID, NSFW_BIT, nsfw);\n  }\n\n  public void setIsProtected(long userID, boolean isProtected) {\n    set(userID, IS_PROTECTED_BIT, isProtected);\n  }\n\n  /**\n   * Adds the given user update to this table.\n   */\n  public boolean indexUserUpdate(UserUpdatesChecker checker, UserUpdate userUpdate) {\n    if (checker.skipUserUpdate(userUpdate)) {\n      return false;\n    }\n\n    switch (userUpdate.updateType) {\n      case ANTISOCIAL:\n        setAntisocial(userUpdate.twitterUserID, userUpdate.updateValue != 0);\n        break;\n      case NSFW:\n        setNSFW(userUpdate.twitterUserID, userUpdate.updateValue != 0);\n        break;\n      case OFFENSIVE:\n        setOffensive(userUpdate.twitterUserID, userUpdate.updateValue != 0);\n        break;\n      case PROTECTED:\n        setIsProtected(userUpdate.twitterUserID, userUpdate.updateValue != 0);\n        break;\n      default:\n        return false;\n    }\n\n    return true;\n  }\n\n  private final AtomicReference<HashTable> hashTable = new AtomicReference<>();\n\n  private int hashCode(long userID) {\n    return (int) GeneralLongHashFunction.hash(userID);\n  }\n\n  /**\n   * Returns an iterator for user IDs that have at least one of the bits set.\n   */\n  public Iterator<Long> getFlaggedUserIdIterator() {\n    HashTable table = hashTable.get();\n\n    final long[] currUserIdTable = table.hash;\n    final byte[] currBitsTable = table.bits;\n    return new Iterator<Long>() {\n      private int index = findNext(0);\n\n      private int findNext(int index) {\n        int startingIndex = index;\n        while (startingIndex < currUserIdTable.length) {\n          if (currUserIdTable[startingIndex] != 0 && currBitsTable[startingIndex] != 0) {\n            break;\n          }\n          ++startingIndex;\n        }\n        return startingIndex;\n      }\n\n      @Override\n      public boolean hasNext() {\n        return index < currUserIdTable.length;\n      }\n\n      @Override\n      public Long next() {\n        Long r = currUserIdTable[index];\n        index = findNext(index + 1);\n        return r;\n      }\n\n      @Override\n      public void remove() {\n        throw new UnsupportedOperationException();\n      }\n    };\n  }\n\n  /**\n   * Constructs an UserUpdatesTable with an given HashTable instance.\n   * Use <code>useIdFilter</code> as a Predicate that returns true for the elements\n   * needed to be kept in the table.\n   * Use shouldRehash to force a rehasing on the given HashTable.\n   */\n  private UserTable(HashTable hashTable, Predicate<Long> userIdFilter,\n                    boolean shouldRehash) {\n\n    Preconditions.checkNotNull(userIdFilter);\n\n    this.hashTable.set(hashTable);\n    this.userIdFilter = userIdFilter;\n\n    exportUserUpdatesTableStats();\n\n    LOG.info(\"User table num users: {}. Users with no bits set: {}. \"\n            + \"Antisocial users: {}. Offensive users: {}. Nsfw users: {}. IsProtected users: {}.\",\n        this.getNumUsersInTable(),\n        this.getNumUsersWithNoBitsSet(),\n        this.getSetBitCount(ANTISOCIAL_BIT),\n        this.getSetBitCount(OFFENSIVE_BIT),\n        this.getSetBitCount(NSFW_BIT),\n        this.getSetBitCount(IS_PROTECTED_BIT));\n\n    if (shouldRehash) {\n      int filteredTableSize = filterTableAndCountValidItems();\n      // Having exactly 100% usage can impact lookup. Maintain the table at under 50% usage.\n      int newTableCapacity = computeDesiredHashTableCapacity(filteredTableSize * 2);\n\n      rehash(newTableCapacity);\n\n      LOG.info(\"User table num users after rehash: {}. Users with no bits set: {}. \"\n              + \"Antisocial users: {}. Offensive users: {}. Nsfw users: {}. IsProtected users: {}.\",\n          this.getNumUsersInTable(),\n          this.getNumUsersWithNoBitsSet(),\n          this.getSetBitCount(ANTISOCIAL_BIT),\n          this.getSetBitCount(OFFENSIVE_BIT),\n          this.getSetBitCount(NSFW_BIT),\n          this.getSetBitCount(IS_PROTECTED_BIT));\n    }\n  }\n\n  private UserTable(int initialSize, Predicate<Long> userIdFilter) {\n    this(new HashTable(computeDesiredHashTableCapacity(initialSize)), userIdFilter, false);\n  }\n\n  @VisibleForTesting\n  public UserTable(int initialSize) {\n    this(initialSize, userId -> true);\n  }\n\n  public static UserTable\n    newTableWithDefaultCapacityAndPredicate(Predicate<Long> userIdFilter) {\n\n    return new UserTable(DEFAULT_INITIAL_CAPACITY, userIdFilter);\n  }\n\n  public static UserTable newTableNonFilteredWithDefaultCapacity() {\n    return newTableWithDefaultCapacityAndPredicate(userId -> true);\n  }\n\n  private void exportUserUpdatesTableStats() {\n    userTableSize = SearchLongGauge.export(USER_TABLE_SIZE);\n    userTableCapacity = SearchLongGauge.export(USER_TABLE_CAPACITY);\n    userTableNumUsersWithNoBitsSet = SearchLongGauge.export(\n        USER_NUM_USERS_WITH_NO_BITS_SET\n    );\n    userTableAntisocialUsers = SearchLongGauge.export(USER_TABLE_ANTISOCIAL_USERS);\n    userTableOffensiveUsers = SearchLongGauge.export(USER_TABLE_OFFENSIVE_USERS);\n    userTableNsfwUsers = SearchLongGauge.export(USER_TABLE_NSFW_USERS);\n    userTableIsProtectedUsers = SearchLongGauge.export(USER_TABLE_IS_PROTECTED_USERS);\n\n    LOG.info(\n        \"Exporting stats for user table. Starting with numUsersInTable={}, usersWithZeroBits={}, \"\n            + \"antisocialUsers={}, offensiveUsers={}, nsfwUsers={}, isProtectedUsers={}.\",\n        getNumUsersInTable(),\n        getNumUsersWithNoBitsSet(),\n        getSetBitCount(ANTISOCIAL_BIT),\n        getSetBitCount(OFFENSIVE_BIT),\n        getSetBitCount(NSFW_BIT),\n        getSetBitCount(IS_PROTECTED_BIT));\n    updateStats();\n  }\n\n  private void updateStats() {\n    HashTable table = this.hashTable.get();\n    userTableSize.set(table.numUsersInTable);\n    userTableNumUsersWithNoBitsSet.set(table.numUsersWithNoBitsSet);\n    userTableCapacity.set(table.hashSize());\n    userTableAntisocialUsers.set(getSetBitCount(ANTISOCIAL_BIT));\n    userTableOffensiveUsers.set(getSetBitCount(OFFENSIVE_BIT));\n    userTableNsfwUsers.set(getSetBitCount(NSFW_BIT));\n    userTableIsProtectedUsers.set(getSetBitCount(IS_PROTECTED_BIT));\n  }\n\n  /**\n   * Computes the size of the hashtable as the first power of two greater than or equal to initialSize\n   */\n  private static int computeDesiredHashTableCapacity(int initialSize) {\n    long powerOfTwoSize = 2;\n    while (initialSize > powerOfTwoSize) {\n      powerOfTwoSize *= 2;\n    }\n    if (powerOfTwoSize > Integer.MAX_VALUE) {\n      LOG.error(\"Error: powerOfTwoSize overflowed Integer.MAX_VALUE! Initial size: \" + initialSize);\n      powerOfTwoSize = 1 << 30;  // max power of 2\n    }\n\n    return (int) powerOfTwoSize;\n  }\n\n  public int getNumUsersInTable() {\n    return hashTable.get().numUsersInTable;\n  }\n\n  /**\n   * Get the number of users who have the bit set at the `userStateBit` position\n   */\n  public long getSetBitCount(int userStateBit) {\n    int bit = userStateBit;\n    int bitPosition = 0;\n    while (bit != 0 && (bit & 1) == 0) {\n      bit = bit >>> 1;\n      bitPosition++;\n    }\n    return hashTable.get().setBitCounts[bitPosition];\n  }\n\n  public Predicate<Long> getUserIdFilter() {\n    return userIdFilter::test;\n  }\n\n  /**\n   * Updates a user flag in this table.\n   */\n  public final void set(long userID, int bit, boolean value) {\n    // if userID is filtered return immediately\n    if (!shouldKeepUser(userID)) {\n      USER_TABLE_USERS_FILTERED_COUNTER.increment();\n      return;\n    }\n\n    HashTable table = this.hashTable.get();\n\n    int hashPos = findHashPosition(table, userID);\n    long item = table.hash[hashPos];\n    byte bits = 0;\n    int bitsDiff = 0;\n\n    if (item != 0) {\n      byte bitsOriginally = bits = table.bits[hashPos];\n      if (value) {\n        bits |= bit;\n      } else {\n        // AND'ing with the inverse map clears the desired bit, but\n        // doesn't change any of the other bits\n        bits &= ~bit;\n      }\n\n      // Find the changed bits after the above operation, it is possible that no bit is changed if\n      // the input 'bit' is already set/unset in the table.\n      // Since bitwise operators cannot be directly applied on Byte, Byte is promoted into int to\n      // apply the operators. When that happens, if the most significant bit of the Byte is set,\n      // the promoted int has all significant bits set to 1. 0xff bitmask is applied here to make\n      // sure only the last 8 bits are considered.\n      bitsDiff = (bitsOriginally & 0xff) ^ (bits & 0xff);\n\n      if (bitsOriginally > 0 && bits == 0) {\n        table.numUsersWithNoBitsSet++;\n      } else if (bitsOriginally == 0 && bits > 0) {\n        table.numUsersWithNoBitsSet--;\n      }\n    } else {\n      if (!value) {\n        // no need to add this user, since all bits would be false anyway\n        return;\n      }\n\n      // New user string.\n      if (table.numUsersInTable + 1 >= (table.hashSize() >> 1)\n          && table.hashSize() != userUpdateTableMaxCapacity) {\n        if (2L * (long) table.hashSize() < userUpdateTableMaxCapacity) {\n          rehash(2 * table.hashSize());\n          table = this.hashTable.get();\n        } else {\n          if (table.hashSize() < (int) userUpdateTableMaxCapacity) {\n            rehash((int) userUpdateTableMaxCapacity);\n            table = this.hashTable.get();\n            LOG.warn(\"User update table size reached Integer.MAX_VALUE, performance will degrade.\");\n          }\n        }\n\n        // Must repeat this operation with the resized hashTable.\n        hashPos = findHashPosition(table, userID);\n      }\n\n      item = userID;\n      bits |= bit;\n      bitsDiff = bit & 0xff;\n\n      table.numUsersInTable++;\n    }\n\n    table.hash[hashPos] = item;\n    table.bits[hashPos] = bits;\n\n    // update setBitCounts for the changed bits after applying the input 'bit'\n    int curBitsDiffPos = 0;\n    while (bitsDiff != 0) {\n      if ((bitsDiff & 1) != 0) {\n        if (value) {\n          table.setBitCounts[curBitsDiffPos]++;\n        } else {\n          table.setBitCounts[curBitsDiffPos]--;\n        }\n      }\n      bitsDiff = bitsDiff >>> 1;\n      curBitsDiffPos++;\n    }\n\n    updateStats();\n  }\n\n  public final boolean isSet(long userID, int bits) {\n    HashTable table = hashTable.get();\n    int hashPos = findHashPosition(table, userID);\n    return table.hash[hashPos] != 0 && (table.bits[hashPos] & bits) != 0;\n  }\n\n  /**\n   * Returns true when userIdFilter condition is being met.\n   * If filter is not present returns true\n   */\n  private boolean shouldKeepUser(long userID) {\n    return userIdFilter.test(userID);\n  }\n\n  private int findHashPosition(final HashTable table, final long userID) {\n    int code = hashCode(userID);\n    int hashPos = code & table.hashMask;\n\n    // Locate user in hash\n    long item = table.hash[hashPos];\n\n    if (item != 0 && item != userID) {\n      // Conflict: keep searching different locations in\n      // the hash table.\n      final int inc = ((code >> 8) + code) | 1;\n      do {\n        code += inc;\n        hashPos = code & table.hashMask;\n        item = table.hash[hashPos];\n      } while (item != 0 && item != userID);\n    }\n\n    return hashPos;\n  }\n\n  /**\n   * Applies the filtering predicate and returns the size of the filtered table.\n   */\n  private synchronized int filterTableAndCountValidItems() {\n    final HashTable oldTable = this.hashTable.get();\n    int newSize = 0;\n\n    int clearNoItemSet = 0;\n    int clearNoBitsSet = 0;\n    int clearDontKeepUser = 0;\n\n    for (int i = 0; i < oldTable.hashSize(); i++) {\n      final long item = oldTable.hash[i]; // this is the userID\n      final byte bits = oldTable.bits[i];\n\n      boolean clearSlot = false;\n      if (item == 0) {\n        clearSlot = true;\n        clearNoItemSet++;\n      } else if (bits == 0) {\n        clearSlot = true;\n        clearNoBitsSet++;\n      } else if (!shouldKeepUser(item)) {\n        clearSlot = true;\n        clearDontKeepUser++;\n      }\n\n      if (clearSlot) {\n        oldTable.hash[i] = 0;\n        oldTable.bits[i] = 0;\n      } else {\n        newSize += 1;\n      }\n    }\n\n    oldTable.setCountOfNumUsersWithNoBitsSet();\n    oldTable.setSetBitCounts();\n\n    LOG.info(\"Done filtering table: clearNoItemSet={}, clearNoBitsSet={}, clearDontKeepUser={}\",\n        clearNoItemSet, clearNoBitsSet, clearDontKeepUser);\n\n    return newSize;\n  }\n\n  /**\n   * Called when hash is too small (> 50% occupied)\n   */\n  private void rehash(final int newSize) {\n    final HashTable oldTable = this.hashTable.get();\n    final HashTable newTable = new HashTable(newSize);\n\n    final int newMask = newTable.hashMask;\n    final long[] newHash = newTable.hash;\n    final byte[] newBits = newTable.bits;\n\n    for (int i = 0; i < oldTable.hashSize(); i++) {\n      final long item = oldTable.hash[i];\n      final byte bits = oldTable.bits[i];\n      if (item != 0 && bits != 0) {\n        int code = hashCode(item);\n\n        int hashPos = code & newMask;\n        assert hashPos >= 0;\n        if (newHash[hashPos] != 0) {\n          final int inc = ((code >> 8) + code) | 1;\n          do {\n            code += inc;\n            hashPos = code & newMask;\n          } while (newHash[hashPos] != 0);\n        }\n        newHash[hashPos] = item;\n        newBits[hashPos] = bits;\n        newTable.numUsersInTable++;\n      }\n    }\n\n    newTable.setCountOfNumUsersWithNoBitsSet();\n    newTable.setSetBitCounts();\n    this.hashTable.set(newTable);\n\n    updateStats();\n  }\n\n  public void setTable(UserTable newTable) {\n    hashTable.set(newTable.hashTable.get());\n    updateStats();\n  }\n\n  @VisibleForTesting\n  protected int getHashTableCapacity() {\n    return hashTable.get().hashSize();\n  }\n\n  @VisibleForTesting\n  protected int getNumUsersWithNoBitsSet() {\n    return hashTable.get().numUsersWithNoBitsSet;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/userupdates/UserTableBuilderFromSnapshot.java",
    "content": "package com.twitter.search.earlybird.common.userupdates;\n\nimport java.io.BufferedReader;\nimport java.io.IOException;\nimport java.io.InputStreamReader;\nimport java.util.Arrays;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.NoSuchElementException;\nimport java.util.Optional;\nimport java.util.Spliterator;\nimport java.util.Spliterators;\nimport java.util.concurrent.TimeUnit;\nimport java.util.function.Predicate;\nimport java.util.stream.Collectors;\nimport java.util.stream.Stream;\nimport java.util.stream.StreamSupport;\nimport javax.annotation.Nullable;\n\nimport org.apache.hadoop.conf.Configuration;\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.Path;\nimport org.apache.hadoop.hdfs.HdfsConfiguration;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common_internal.hadoop.HdfsUtils;\nimport com.twitter.scalding.DateRange;\nimport com.twitter.scalding.Hours;\nimport com.twitter.scalding.RichDate;\nimport com.twitter.search.user_table.sources.MostRecentGoodSafetyUserStateSource;\nimport com.twitter.search.common.indexing.thriftjava.SafetyUserState;\nimport com.twitter.search.common.util.io.LzoThriftBlockFileReader;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.util.Duration;\nimport com.twitter.util.Time;\n\n/**\n * Builds a user table from a user safety snapshot on HDFS.\n */\npublic class UserTableBuilderFromSnapshot {\n  private static final Logger LOG = LoggerFactory.getLogger(UserTableBuilderFromSnapshot.class);\n\n  private static final int MAX_DAYS_TO_CHECK = 7;\n  public static final String DATA_DIR = \"user_states\";\n  public static final String METADATA_DIR = \"last_updated_ms\";\n\n  private final String snapshotBaseDir;\n\n  private String snapshotDataPath;\n  private String snapshotMetaDataPath;\n  private UserTable userTable;\n\n  private long nsfwCount;\n  private long antisocialCount;\n  private long isProtectedCount;\n\n  public UserTableBuilderFromSnapshot() {\n    snapshotBaseDir =\n        EarlybirdConfig.getString(EarlybirdConfig.USER_SNAPSHOT_BASE_DIR, null);\n\n    LOG.info(\"Configured user snapshot directory: \" + snapshotBaseDir);\n  }\n\n  private static final class UserUpdate {\n    public final long userId;\n    @Nullable public final Boolean antisocial;\n    @Nullable public final Boolean nsfw;\n    @Nullable public final Boolean isProtected;\n\n    private UserUpdate(long userId,\n                       @Nullable Boolean antisocial,\n                       @Nullable Boolean nsfw,\n                       @Nullable Boolean isProtected) {\n      this.userId = userId;\n      this.antisocial = antisocial;\n      this.nsfw = nsfw;\n      this.isProtected = isProtected;\n    }\n\n    public static UserUpdate fromUserState(SafetyUserState safetyUserState) {\n      long userId = safetyUserState.getUserID();\n      @Nullable Boolean antisocial = null;\n      @Nullable Boolean nsfw = null;\n      @Nullable Boolean isProtected = null;\n\n      if (safetyUserState.isIsAntisocial()) {\n        antisocial = true;\n      }\n      if (safetyUserState.isIsNsfw()) {\n        nsfw = true;\n      }\n      if (safetyUserState.isSetIsProtected() && safetyUserState.isIsProtected()) {\n        isProtected = true;\n      }\n\n      return new UserUpdate(userId, antisocial, nsfw, isProtected);\n    }\n  }\n\n  /**\n   * Builds a user table from an HDFS user snapshot.\n   * @return The table, or nothing if something went wrong.\n   */\n  public Optional<UserTable> build(Predicate<Long> userFilter) {\n    userTable = UserTable.newTableWithDefaultCapacityAndPredicate(userFilter);\n    nsfwCount = 0;\n    antisocialCount = 0;\n    isProtectedCount = 0;\n\n    if (snapshotBaseDir == null || snapshotBaseDir.isEmpty()) {\n      LOG.info(\"No snapshot directory. Can't build user table.\");\n      return Optional.empty();\n    }\n\n    LOG.info(\"Starting to build user table.\");\n\n    Stream<UserUpdate> stream = null;\n\n    try {\n      setSnapshotPath();\n\n      stream = getUserUpdates();\n      stream.forEach(this::insertUser);\n    } catch (IOException e) {\n      LOG.error(\"IOException while building table: {}\", e.getMessage(), e);\n\n      return Optional.empty();\n    } finally {\n      if (stream != null) {\n        stream.close();\n      }\n    }\n\n    LOG.info(\"Built user table with {} users, {} nsfw, {} antisocial and {} protected.\",\n        userTable.getNumUsersInTable(),\n        nsfwCount,\n        antisocialCount,\n        isProtectedCount);\n\n    try {\n      userTable.setLastRecordTimestamp(readTimestampOfLastSeenUpdateFromSnapshot());\n    } catch (IOException e) {\n      LOG.error(\"IOException reading timestamp of last update: {}\", e.getMessage(), e);\n      return Optional.empty();\n    }\n\n    LOG.info(\"Setting last record timestamp to {}.\", userTable.getLastRecordTimestamp());\n\n    return Optional.of(userTable);\n  }\n\n  private void setSnapshotPath() {\n    snapshotDataPath =\n        new MostRecentGoodSafetyUserStateSource(\n            snapshotBaseDir,\n            DATA_DIR,\n            METADATA_DIR,\n            DateRange.apply(\n                RichDate.now().$minus(Hours.apply(MAX_DAYS_TO_CHECK * 24)),\n                RichDate.now())\n        ).partitionHdfsPaths(new HdfsConfiguration())\n         ._1()\n         .head()\n         .replaceAll(\"\\\\*$\", \"\");\n    snapshotMetaDataPath = snapshotDataPath.replace(DATA_DIR, METADATA_DIR);\n\n    LOG.info(\"Snapshot data path: {}\", snapshotDataPath);\n    LOG.info(\"Snapshot metadata path: {}\", snapshotMetaDataPath);\n  }\n\n  private Stream<UserUpdate> getUserUpdates() throws IOException {\n    FileSystem fs = FileSystem.get(new Configuration());\n    List<String> lzoFiles =\n        Arrays.stream(fs.listStatus(new Path(snapshotDataPath),\n                                    path -> path.getName().startsWith(\"part-\")))\n              .map(fileStatus -> Path.getPathWithoutSchemeAndAuthority(fileStatus.getPath())\n                                     .toString())\n              .collect(Collectors.toList());\n\n    final LzoThriftBlockFileReader<SafetyUserState> thriftReader =\n        new LzoThriftBlockFileReader<>(lzoFiles, SafetyUserState.class, null);\n\n    Iterator<UserUpdate> iter = new Iterator<UserUpdate>() {\n      private SafetyUserState next;\n\n      @Override\n      public boolean hasNext() {\n        if (next != null) {\n          return true;\n        }\n\n        do {\n          try {\n            next = thriftReader.readNext();\n          } catch (IOException e) {\n            throw new RuntimeException(e);\n          }\n        } while (next == null && !thriftReader.isExhausted());\n        return next != null;\n      }\n\n      @Override\n      public UserUpdate next() {\n        if (next != null || hasNext()) {\n          UserUpdate userUpdate = UserUpdate.fromUserState(next);\n          next = null;\n          return userUpdate;\n        }\n        throw new NoSuchElementException();\n      }\n    };\n\n    return StreamSupport\n        .stream(\n            Spliterators.spliteratorUnknownSize(iter, Spliterator.ORDERED | Spliterator.NONNULL),\n            false)\n        .onClose(thriftReader::stop);\n  }\n\n  private long readTimestampOfLastSeenUpdateFromSnapshot() throws IOException {\n    String timestampFile = snapshotMetaDataPath + \"part-00000\";\n    BufferedReader buffer = new BufferedReader(new InputStreamReader(\n        HdfsUtils.getInputStreamSupplier(timestampFile).openStream()));\n\n    long timestampMillis = Long.parseLong(buffer.readLine());\n    LOG.info(\"read timestamp {} from HDFS:{}\", timestampMillis, timestampFile);\n\n    Time time = Time.fromMilliseconds(timestampMillis)\n                    .minus(Duration.fromTimeUnit(10, TimeUnit.MINUTES));\n    return time.inMilliseconds();\n  }\n\n  private void insertUser(UserUpdate userUpdate) {\n    if (userUpdate == null) {\n      return;\n    }\n\n    if (userUpdate.antisocial != null) {\n      userTable.set(\n          userUpdate.userId,\n          UserTable.ANTISOCIAL_BIT,\n          userUpdate.antisocial);\n      antisocialCount++;\n    }\n\n    if (userUpdate.nsfw != null) {\n      userTable.set(\n          userUpdate.userId,\n          UserTable.NSFW_BIT,\n          userUpdate.nsfw);\n      nsfwCount++;\n    }\n\n    if (userUpdate.isProtected != null) {\n      userTable.set(\n          userUpdate.userId,\n          UserTable.IS_PROTECTED_BIT,\n          userUpdate.isProtected);\n      isProtectedCount++;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/userupdates/UserUpdate.java",
    "content": "package com.twitter.search.earlybird.common.userupdates;\n\nimport java.util.Date;\n\nimport com.twitter.search.common.indexing.thriftjava.UserUpdateType;\n\n/**\n * Contains an update for a user.\n */\npublic class UserUpdate {\n  public final long twitterUserID;\n  public final UserUpdateType updateType;\n  public final int updateValue;\n  private final Date updatedAt;\n\n  public UserUpdate(long twitterUserID,\n                    UserUpdateType updateType,\n                    int updateValue,\n                    Date updatedAt) {\n\n    this.twitterUserID = twitterUserID;\n    this.updateType = updateType;\n    this.updateValue = updateValue;\n    this.updatedAt = (Date) updatedAt.clone();\n  }\n\n  @Override public String toString() {\n    return \"UserInfoUpdate[userID=\" + twitterUserID + \",updateType=\" + updateType\n           + \",updateValue=\" + updateValue + \",updatedAt=\" + getUpdatedAt() + \"]\";\n  }\n\n  /**\n   * Returns a copy of the updated-at date.\n   */\n  public Date getUpdatedAt() {\n    return (Date) updatedAt.clone();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/common/userupdates/UserUpdatesChecker.java",
    "content": "package com.twitter.search.earlybird.common.userupdates;\n\nimport java.util.Date;\nimport java.util.concurrent.TimeUnit;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.indexing.thriftjava.UserUpdateType;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\n\n/**\n * Contains logic for deciding whether to apply a certain user update to the {@link UserTable}.\n */\npublic class UserUpdatesChecker {\n  private final Date antisocialStartDate;\n  private final Decider decider;\n  private final boolean isFullArchiveCluster;\n\n  public UserUpdatesChecker(Clock clock, Decider decider, EarlybirdCluster cluster) {\n    // How many days of antisocial users to keep. A value of -1 means keeping all user updates.\n    long antisocialRecordDays =\n        EarlybirdConfig.getLong(\"keep_recent_antisocial_user_updates_days\", 30);\n    this.antisocialStartDate = antisocialRecordDays > 0\n        ? new Date(clock.nowMillis() - TimeUnit.DAYS.toMillis(antisocialRecordDays)) : null;\n    this.decider = decider;\n    this.isFullArchiveCluster = cluster == EarlybirdCluster.FULL_ARCHIVE;\n  }\n\n  /**\n   * Decides whether to skip the given UserInfoUpdate.\n   */\n  public boolean skipUserUpdate(UserUpdate userUpdate) {\n    if (userUpdate == null) { // always skip null updates\n      return true;\n    }\n\n    UserUpdateType type = userUpdate.updateType;\n\n    if (type == UserUpdateType.PROTECTED && skipProtectedUserUpdate()) {\n      return true;\n    }\n\n    if (type == UserUpdateType.ANTISOCIAL && skipAntisocialUserUpdate(userUpdate)) {\n      return true;\n    }\n\n    // NSFW users can continue to tweet even after they are marked as NSFW. That means\n    // that the snapshot needs to have all NSFW users from the beginning of time. Hence, no NSFW\n    // users updates check here.\n\n    // pass all checks, do not skip this user update\n    return false;\n  }\n\n  // Antisocial/suspended users can't tweet after they are suspended. Thus if our index stores\n  // tweets from the last 10 days, and they were suspended 60 days ago, we don't need them since\n  // there will be no tweets from them. We can save space by not storing info about those users.\n\n  // (For archive, at rebuild time we filter out all suspended users tweets, so for a user that\n  // was suspended before a rebuild, no need to use space to store that the user is suspended)\n  private boolean skipAntisocialUserUpdate(UserUpdate userUpdate) {\n    return antisocialStartDate != null && userUpdate.getUpdatedAt().before(antisocialStartDate);\n  }\n\n  // skip protected user updates for realtime and protected clusters\n  private boolean skipProtectedUserUpdate() {\n    return !isFullArchiveCluster;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/config/BUILD",
    "content": "java_library(\n    sources = [\"**/*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/code/findbugs:jsr305\",\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/org/apache/thrift:libthrift\",\n        \"3rdparty/jvm/org/apache/zookeeper:zookeeper-client\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/base\",\n        \"src/java/com/twitter/common/util:system-mocks\",\n        \"src/java/com/twitter/search/common/config\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/partitioning/snowflakeparser\",\n        \"src/java/com/twitter/search/common/util/date\",\n        \"src/java/com/twitter/search/common/util/zookeeper\",\n        \"src/java/com/twitter/search/earlybird/common/config\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/config/ServingRange.java",
    "content": "package com.twitter.search.earlybird.config;\n\n/**\n * An interface for abstracting a tier's serving range.\n */\npublic interface ServingRange {\n  /**\n   * Returns the serving range's lowest tweet ID.\n   */\n  long getServingRangeSinceId();\n\n  /**\n   * Returns the serving range's highest tweet ID.\n   */\n  long getServingRangeMaxId();\n\n  /**\n   * Returns the serving range's earliest time, in seconds since epoch.\n   */\n  long getServingRangeSinceTimeSecondsFromEpoch();\n\n  /**\n   * Returns the serving range's latest time, in seconds since epoch.\n   */\n  long getServingRangeUntilTimeSecondsFromEpoch();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/config/TierConfig.java",
    "content": "package com.twitter.search.earlybird.config;\n\nimport java.util.Date;\nimport java.util.Map;\nimport java.util.Set;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.common.config.ConfigFile;\nimport com.twitter.search.common.config.ConfigurationException;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.util.date.DateUtil;\n\n/**\n * This class provides APIs to access the tier configurations for a cluster.\n * Each tier has tier name, number of partitions, tier start time and end time.\n */\npublic final class TierConfig {\n  private static final org.slf4j.Logger LOG = org.slf4j.LoggerFactory.getLogger(TierConfig.class);\n\n  private static final String DEFAULT_CONFIG_DIR = \"common/config\";\n  public static final String DEFAULT_TIER_FILE = \"earlybird-tiers.yml\";\n\n  public static final Date DEFAULT_TIER_START_DATE = DateUtil.toDate(2006, 3, 21);\n  // It's convenient for DEFAULT_TIER_END_DATE to be before ~2100, because then the output of\n  // FieldTermCounter.getHourValue(DEFAULT_TIER_END_END_DATE) can still fit into an integer.\n  public static final Date DEFAULT_TIER_END_DATE = DateUtil.toDate(2099, 1, 1);\n\n  public static final String DEFAULT_TIER_NAME = \"all\";\n  public static final boolean DEFAULT_ENABLED = true;\n  public static final TierInfo.RequestReadType DEFAULT_READ_TYPE = TierInfo.RequestReadType.LIGHT;\n\n  private static ConfigFile tierConfigFile = null;\n  private static ConfigSource tierConfigSource = null;\n\n  public enum ConfigSource {\n    LOCAL,\n    ZOOKEEPER\n  }\n\n  private TierConfig() { }\n\n  private static synchronized void init() {\n    if (tierConfigFile == null) {\n      tierConfigFile = new ConfigFile(DEFAULT_CONFIG_DIR, DEFAULT_TIER_FILE);\n      tierConfigSource = ConfigSource.LOCAL;\n      SearchLongGauge.export(\"tier_config_source_\" + tierConfigSource.name()).set(1);\n      LOG.info(\"Tier config file \" + DEFAULT_TIER_FILE + \" is successfully loaded from bundle.\");\n    }\n  }\n\n  public static ConfigFile getConfigFile() {\n    init();\n    return tierConfigFile;\n  }\n\n  public static String getConfigFileName() {\n    return getConfigFile().getConfigFileName();\n  }\n\n  /**\n   * Return all the tier names specified in the config file.\n   */\n  public static Set<String> getTierNames() {\n    return Config.getConfig().getMapCopy(getConfigFileName()).keySet();\n  }\n\n  /**\n   * Sets the value of the given tier config property to the given value.\n   */\n  public static void setForTests(String property, Object value) {\n    Config.getConfig().setForTests(DEFAULT_TIER_FILE, property, value);\n  }\n\n  /**\n   * Returns the config info for the specified tier.\n   */\n  public static TierInfo getTierInfo(String tierName) {\n    return getTierInfo(tierName, null /* use current environment */);\n  }\n\n  /**\n   * Returns the config info for the specified tier and environment.\n   */\n  public static TierInfo getTierInfo(String tierName, @Nullable String environment) {\n    String tierConfigFileType = getConfigFileName();\n    Map<String, Object> tierInfo;\n    try {\n      tierInfo = (Map<String, Object>) Config.getConfig()\n          .getFromEnvironment(environment, tierConfigFileType, tierName);\n    } catch (ConfigurationException e) {\n      throw new RuntimeException(e);\n    }\n    if (tierInfo == null) {\n      LOG.error(\"Cannot find tier config for \"\n          + tierName + \"in config file: \" + tierConfigFileType);\n      throw new RuntimeException(\"Configuration error: \" + tierConfigFileType);\n    }\n\n    Long partitions = (Long) tierInfo.get(\"number_of_partitions\");\n    if (partitions == null) {\n      LOG.error(\"No number of partition is specified for tier \"\n          + tierName + \" in tier config file \" + tierConfigFileType);\n      throw new RuntimeException(\"Configuration error: \" + tierConfigFileType);\n    }\n\n    Long numTimeslices = (Long) tierInfo.get(\"serving_timeslices\");\n    if (numTimeslices == null) {\n      LOG.info(\"No max timeslices is specified for tier \"\n          + tierName + \" in tier config file \" + tierConfigFileType\n          + \", not setting a cap on number of serving timeslices\");\n      // NOTE: we use max int32 here because it will ultimately be cast to an int, but the config\n      // map expects Longs for all integral types.  Using Long.MAX_VALUE leads to max serving\n      // timeslices being set to -1 when it is truncated to an int.\n      numTimeslices = (long) Integer.MAX_VALUE;\n    }\n\n    Date tierStartDate = (Date) tierInfo.get(\"data_range_start_date_inclusive\");\n    if (tierStartDate == null) {\n      tierStartDate = DEFAULT_TIER_START_DATE;\n    }\n    Date tierEndDate = (Date) tierInfo.get(\"data_range_end_date_exclusive\");\n    if (tierEndDate == null) {\n      tierEndDate = DEFAULT_TIER_END_DATE;\n    }\n\n    Boolean tierEnabled = (Boolean) tierInfo.get(\"tier_enabled\");\n    if (tierEnabled == null) {\n      tierEnabled = DEFAULT_ENABLED;\n    }\n\n    TierInfo.RequestReadType readType =\n      getRequestReadType((String) tierInfo.get(\"tier_read_type\"), DEFAULT_READ_TYPE);\n    TierInfo.RequestReadType readTypeOverride =\n      getRequestReadType((String) tierInfo.get(\"tier_read_type_override\"), readType);\n\n    return new TierInfo(\n        tierName,\n        tierStartDate,\n        tierEndDate,\n        partitions.intValue(),\n        numTimeslices.intValue(),\n        tierEnabled,\n        (String) tierInfo.get(\"serving_range_since_id_exclusive\"),\n        (String) tierInfo.get(\"serving_range_max_id_inclusive\"),\n        (Date) tierInfo.get(\"serving_range_start_date_inclusive_override\"),\n        (Date) tierInfo.get(\"serving_range_end_date_exclusive_override\"),\n        readType,\n        readTypeOverride,\n        Clock.SYSTEM_CLOCK);\n  }\n\n  public static synchronized void clear() {\n    tierConfigFile = null;\n    tierConfigSource = null;\n  }\n\n  protected static synchronized ConfigSource getTierConfigSource() {\n    return tierConfigSource;\n  }\n\n  private static TierInfo.RequestReadType getRequestReadType(\n      String readTypeEnumName, TierInfo.RequestReadType defaultReadType) {\n    TierInfo.RequestReadType readType = defaultReadType;\n    if (readTypeEnumName != null) {\n      readType = TierInfo.RequestReadType.valueOf(readTypeEnumName.trim().toUpperCase());\n      Preconditions.checkState(readType != null);\n    }\n    return readType;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/config/TierInfo.java",
    "content": "package com.twitter.search.earlybird.config;\n\nimport java.util.Date;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.util.Clock;\n\n/**\n * Properties of a single tier.\n */\npublic class TierInfo implements ServingRange {\n  // What I'm seeing historically is that this has been used when adding a new tier. First you\n  // add it and send dark traffic to it, then possibly grey and then you launch it by turning on\n  // light traffic.\n  public static enum RequestReadType {\n    // Light read: send request, wait for results, and results are returned\n    LIGHT,\n    // Dark read: send request, do not wait for results, and results are discarded\n    DARK,\n    // Grey read: send request, wait for results, but discard after results come back.\n    // Same results as dark read; similar latency as light read.\n    GREY,\n  }\n\n  private final String tierName;\n  private final Date dataStartDate;\n  private final Date dataEndDate;\n  private final int numPartitions;\n  private final int maxTimeslices;\n  private final TierServingBoundaryEndPoint servingRangeSince;\n  private final TierServingBoundaryEndPoint servingRangeMax;\n  private final TierServingBoundaryEndPoint servingRangeSinceOverride;\n  private final TierServingBoundaryEndPoint servingRangeMaxOverride;\n\n  // These two properties are only used by clients of Earlybird (E.g. roots),\n  // but not by Earlybirds.\n  private final boolean enabled;\n  private final RequestReadType readType;\n  private final RequestReadType readTypeOverride;\n\n  public TierInfo(String tierName,\n                  Date dataStartDate,\n                  Date dataEndDate,\n                  int numPartitions,\n                  int maxTimeslices,\n                  boolean enabled,\n                  String sinceIdString,\n                  String maxIdString,\n                  Date servingStartDateOverride,\n                  Date servingEndDateOverride,\n                  RequestReadType readType,\n                  RequestReadType readTypeOverride,\n                  Clock clock) {\n    Preconditions.checkArgument(numPartitions > 0);\n    Preconditions.checkArgument(maxTimeslices > 0);\n    this.tierName = tierName;\n    this.dataStartDate = dataStartDate;\n    this.dataEndDate = dataEndDate;\n    this.numPartitions = numPartitions;\n    this.maxTimeslices = maxTimeslices;\n    this.enabled = enabled;\n    this.readType = readType;\n    this.readTypeOverride = readTypeOverride;\n    this.servingRangeSince = TierServingBoundaryEndPoint\n        .newTierServingBoundaryEndPoint(sinceIdString, dataStartDate, clock);\n    this.servingRangeMax = TierServingBoundaryEndPoint\n        .newTierServingBoundaryEndPoint(maxIdString, dataEndDate, clock);\n    if (servingStartDateOverride != null) {\n      this.servingRangeSinceOverride = TierServingBoundaryEndPoint.newTierServingBoundaryEndPoint(\n          TierServingBoundaryEndPoint.INFERRED_FROM_DATA_RANGE, servingStartDateOverride, clock);\n    } else {\n      this.servingRangeSinceOverride = servingRangeSince;\n    }\n\n    if (servingEndDateOverride != null) {\n      this.servingRangeMaxOverride = TierServingBoundaryEndPoint.newTierServingBoundaryEndPoint(\n          TierServingBoundaryEndPoint.INFERRED_FROM_DATA_RANGE, servingEndDateOverride, clock);\n    } else {\n      this.servingRangeMaxOverride = servingRangeMax;\n    }\n  }\n\n  @VisibleForTesting\n  public TierInfo(String tierName,\n                  Date dataStartDate,\n                  Date dataEndDate,\n                  int numPartitions,\n                  int maxTimeslices,\n                  boolean enabled,\n                  String sinceIdString,\n                  String maxIdString,\n                  RequestReadType readType,\n                  Clock clock) {\n    // No overrides:\n    //   servingRangeSinceOverride == servingRangeSince\n    //   servingRangeMaxOverride == servingRangeMax\n    //   readTypeOverride == readType\n    this(tierName, dataStartDate, dataEndDate, numPartitions, maxTimeslices, enabled, sinceIdString,\n         maxIdString, null, null, readType, readType, clock);\n  }\n\n  @Override\n  public String toString() {\n    return tierName;\n  }\n\n  public String getTierName() {\n    return tierName;\n  }\n\n  public Date getDataStartDate() {\n    return dataStartDate;\n  }\n\n  public Date getDataEndDate() {\n    return dataEndDate;\n  }\n\n  public int getNumPartitions() {\n    return numPartitions;\n  }\n\n  public int getMaxTimeslices() {\n    return maxTimeslices;\n  }\n\n  public TierConfig.ConfigSource getSource() {\n    return TierConfig.getTierConfigSource();\n  }\n\n  public boolean isEnabled() {\n    return enabled;\n  }\n\n  public boolean isDarkRead() {\n    return readType == RequestReadType.DARK;\n  }\n\n  public RequestReadType getReadType() {\n    return readType;\n  }\n\n  public RequestReadType getReadTypeOverride() {\n    return readTypeOverride;\n  }\n\n  public long getServingRangeSinceId() {\n    return servingRangeSince.getBoundaryTweetId();\n  }\n\n  public long getServingRangeMaxId() {\n    return servingRangeMax.getBoundaryTweetId();\n  }\n\n  long getServingRangeOverrideSinceId() {\n    return servingRangeSinceOverride.getBoundaryTweetId();\n  }\n\n  long getServingRangeOverrideMaxId() {\n    return servingRangeMaxOverride.getBoundaryTweetId();\n  }\n\n  public long getServingRangeSinceTimeSecondsFromEpoch() {\n    return servingRangeSince.getBoundaryTimeSecondsFromEpoch();\n  }\n\n  public long getServingRangeUntilTimeSecondsFromEpoch() {\n    return servingRangeMax.getBoundaryTimeSecondsFromEpoch();\n  }\n\n  long getServingRangeOverrideSinceTimeSecondsFromEpoch() {\n    return servingRangeSinceOverride.getBoundaryTimeSecondsFromEpoch();\n  }\n\n  long getServingRangeOverrideUntilTimeSecondsFromEpoch() {\n    return servingRangeMaxOverride.getBoundaryTimeSecondsFromEpoch();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/config/TierInfoSource.java",
    "content": "package com.twitter.search.earlybird.config;\n\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Set;\n\nimport javax.inject.Inject;\n\nimport com.twitter.search.common.util.zookeeper.ZooKeeperProxy;\n\npublic class TierInfoSource {\n  private final ZooKeeperProxy zkClient;\n\n  @Inject\n  public TierInfoSource(ZooKeeperProxy sZooKeeperClient) {\n    this.zkClient = sZooKeeperClient;\n  }\n\n  public List<TierInfo> getTierInformation() {\n    return getTierInfoWithPrefix(\"tier\");\n  }\n\n  public String getConfigFileType() {\n    return TierConfig.getConfigFileName();\n  }\n\n  private List<TierInfo> getTierInfoWithPrefix(String tierPrefix) {\n    Set<String> tierNames = TierConfig.getTierNames();\n    List<TierInfo> tierInfos = new ArrayList<>();\n    for (String name : tierNames) {\n      if (name.startsWith(tierPrefix)) {\n        TierInfo tierInfo = TierConfig.getTierInfo(name);\n        tierInfos.add(tierInfo);\n      }\n    }\n    return tierInfos;\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/config/TierInfoUtil.java",
    "content": "package com.twitter.search.earlybird.config;\n\nimport java.util.Comparator;\nimport java.util.SortedSet;\n\nimport com.google.common.base.Preconditions;\n\npublic final class TierInfoUtil {\n  public static final Comparator<TierInfo> TIER_COMPARATOR = (t1, t2) -> {\n    // Reverse sort order based on date.\n    return t2.getDataStartDate().compareTo(t1.getDataStartDate());\n  };\n\n  private TierInfoUtil() {\n  }\n\n  /**\n   * Checks that the serving ranges and the override serving ranges of the given tiers do not\n   * overlap, and do not have gaps. Dark reads tiers are ignored.\n   */\n  public static void checkTierServingRanges(SortedSet<TierInfo> tierInfos) {\n    boolean tierServingRangesOverlap = false;\n    boolean tierOverrideServingRangesOverlap = false;\n    boolean tierServingRangesHaveGaps = false;\n    boolean tierOverrideServingRangesHaveGaps = false;\n\n    TierInfoWrapper previousTierInfoWrapper = null;\n    TierInfoWrapper previousOverrideTierInfoWrapper = null;\n    for (TierInfo tierInfo : tierInfos) {\n      TierInfoWrapper tierInfoWrapper = new TierInfoWrapper(tierInfo, false);\n      TierInfoWrapper overrideTierInfoWrapper = new TierInfoWrapper(tierInfo, true);\n\n      // Check only the tiers to which we send light reads.\n      if (!tierInfoWrapper.isDarkRead()) {\n        if (previousTierInfoWrapper != null) {\n          if (TierInfoWrapper.servingRangesOverlap(previousTierInfoWrapper, tierInfoWrapper)) {\n            // In case of rebalancing, we may have an overlap data range while\n            // overriding with a good serving range.\n            if (previousOverrideTierInfoWrapper == null\n                || TierInfoWrapper.servingRangesOverlap(\n                       previousOverrideTierInfoWrapper, overrideTierInfoWrapper)) {\n              tierServingRangesOverlap = true;\n            }\n          }\n          if (TierInfoWrapper.servingRangesHaveGap(previousTierInfoWrapper, tierInfoWrapper)) {\n            tierServingRangesHaveGaps = true;\n          }\n        }\n\n        previousTierInfoWrapper = tierInfoWrapper;\n      }\n\n      if (!overrideTierInfoWrapper.isDarkRead()) {\n        if (previousOverrideTierInfoWrapper != null) {\n          if (TierInfoWrapper.servingRangesOverlap(previousOverrideTierInfoWrapper,\n                                                   overrideTierInfoWrapper)) {\n            tierOverrideServingRangesOverlap = true;\n          }\n          if (TierInfoWrapper.servingRangesHaveGap(previousOverrideTierInfoWrapper,\n                                                   overrideTierInfoWrapper)) {\n            tierOverrideServingRangesHaveGaps = true;\n          }\n        }\n\n        previousOverrideTierInfoWrapper = overrideTierInfoWrapper;\n      }\n    }\n\n    Preconditions.checkState(!tierServingRangesOverlap,\n                             \"Serving ranges of light reads tiers must not overlap.\");\n    Preconditions.checkState(!tierServingRangesHaveGaps,\n                             \"Serving ranges of light reads tiers must not have gaps.\");\n    Preconditions.checkState(!tierOverrideServingRangesOverlap,\n                             \"Override serving ranges of light reads tiers must not overlap.\");\n    Preconditions.checkState(!tierOverrideServingRangesHaveGaps,\n                             \"Override serving ranges of light reads tiers must not have gaps.\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/config/TierInfoWrapper.java",
    "content": "package com.twitter.search.earlybird.config;\n\nimport java.util.Date;\n\nimport com.google.common.base.Preconditions;\n\n/**\n * A simple wrapper around TierInfo that returns the \"real\" or the \"overriden\" values from the given\n * {@code TierInfo} instance, based on the given {@code useOverrideTierConfig} flag.\n */\npublic class TierInfoWrapper implements ServingRange {\n  private final TierInfo tierInfo;\n  private final boolean useOverrideTierConfig;\n\n  public TierInfoWrapper(TierInfo tierInfo, boolean useOverrideTierConfig) {\n    this.tierInfo = Preconditions.checkNotNull(tierInfo);\n    this.useOverrideTierConfig = useOverrideTierConfig;\n  }\n\n  public String getTierName() {\n    return tierInfo.getTierName();\n  }\n\n  public Date getDataStartDate() {\n    return tierInfo.getDataStartDate();\n  }\n\n  public Date getDataEndDate() {\n    return tierInfo.getDataEndDate();\n  }\n\n  public int getNumPartitions() {\n    return tierInfo.getNumPartitions();\n  }\n\n  public int getMaxTimeslices() {\n    return tierInfo.getMaxTimeslices();\n  }\n\n  public TierConfig.ConfigSource getSource() {\n    return tierInfo.getSource();\n  }\n\n  public boolean isEnabled() {\n    return tierInfo.isEnabled();\n  }\n\n  public boolean isDarkRead() {\n    return getReadType() == TierInfo.RequestReadType.DARK;\n  }\n\n  public TierInfo.RequestReadType getReadType() {\n    return useOverrideTierConfig ? tierInfo.getReadTypeOverride() : tierInfo.getReadType();\n  }\n\n  public long getServingRangeSinceId() {\n    return useOverrideTierConfig\n      ? tierInfo.getServingRangeOverrideSinceId()\n      : tierInfo.getServingRangeSinceId();\n  }\n\n  public long getServingRangeMaxId() {\n    return useOverrideTierConfig\n      ? tierInfo.getServingRangeOverrideMaxId()\n      : tierInfo.getServingRangeMaxId();\n  }\n\n  public long getServingRangeSinceTimeSecondsFromEpoch() {\n    return useOverrideTierConfig\n      ? tierInfo.getServingRangeOverrideSinceTimeSecondsFromEpoch()\n      : tierInfo.getServingRangeSinceTimeSecondsFromEpoch();\n  }\n\n  public long getServingRangeUntilTimeSecondsFromEpoch() {\n    return useOverrideTierConfig\n      ? tierInfo.getServingRangeOverrideUntilTimeSecondsFromEpoch()\n      : tierInfo.getServingRangeUntilTimeSecondsFromEpoch();\n  }\n\n  public static boolean servingRangesOverlap(TierInfoWrapper tier1, TierInfoWrapper tier2) {\n    return (tier1.getServingRangeMaxId() > tier2.getServingRangeSinceId())\n      && (tier2.getServingRangeMaxId() > tier1.getServingRangeSinceId());\n  }\n\n  public static boolean servingRangesHaveGap(TierInfoWrapper tier1, TierInfoWrapper tier2) {\n    return (tier1.getServingRangeMaxId() < tier2.getServingRangeSinceId())\n      || (tier2.getServingRangeMaxId() < tier1.getServingRangeSinceId());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/config/TierServingBoundaryEndPoint.java",
    "content": "package com.twitter.search.earlybird.config;\n\nimport java.util.Date;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\n\n/**\n * The start or end boundary of a tier's serving range.\n * This is used to add since_id and max_id operators onto search queries.\n */\npublic class TierServingBoundaryEndPoint {\n  @VisibleForTesting\n  public static final String INFERRED_FROM_DATA_RANGE = \"inferred_from_data_range\";\n  public static final String RELATIVE_TO_CURRENT_TIME_MS = \"relative_to_current_time_ms\";\n\n  // Either offsetToCurrentTimeMillis is set or (absoluteTweetId and timeBoundarySecondsFromEpoch)\n  // are set.\n  @Nullable\n  private final Long offsetToCurrentTimeMillis;\n  @Nullable\n  private final Long absoluteTweetId;\n  @Nullable\n  private final Long timeBoundarySecondsFromEpoch;\n  private final Clock clock;\n\n  TierServingBoundaryEndPoint(Long absoluteTweetId,\n                              Long timeBoundarySecondsFromEpoch,\n                              Long offsetToCurrentTimeMillis,\n                              Clock clock) {\n    this.offsetToCurrentTimeMillis = offsetToCurrentTimeMillis;\n    this.absoluteTweetId = absoluteTweetId;\n    this.timeBoundarySecondsFromEpoch = timeBoundarySecondsFromEpoch;\n    this.clock = clock;\n  }\n\n  /**\n   * Parse the boundary string and construct a TierServingBoundaryEndPoint instance.\n   * @param boundaryString boundary configuration string. Valid values are:\n   * <li>\n   * \"inferred_from_data_range\" infers serving range from data range. This only works after\n   *                               Nov 2010 when Twitter switched to snowflake IDs.\n   *                               This is the default value.\n   * </li>\n   * <li>\n   * \"absolute_tweet_id_and_timestamp_millis:id:timestamp\" a tweet ID/timestamp is given\n   *                                                       explicitly as the serving range\n   *                                                       boundary.\n   * </li>\n   * <li>\n   * \"relative_to_current_time_ms:offset\" adds offset onto current timestamp in millis to\n   *                                         compute serving range.\n   * </li>\n   *\n   * @param boundaryDate the data boundary. This is used in conjunction with\n   * inferred_from_data_date to determine the serving boundary.\n   * @param clock  Clock used to obtain current time, when relative_to_current_time_ms is used.\n   *               Tests pass in a FakeClock.\n   */\n  public static TierServingBoundaryEndPoint newTierServingBoundaryEndPoint(String boundaryString,\n      Date boundaryDate,\n      Clock clock) {\n    if (boundaryString == null || boundaryString.trim().equals(\n        INFERRED_FROM_DATA_RANGE)) {\n      return inferBoundaryFromDataRange(boundaryDate, clock);\n    } else if (boundaryString.trim().startsWith(RELATIVE_TO_CURRENT_TIME_MS)) {\n      return getRelativeBoundary(boundaryString, clock);\n    } else {\n      throw new IllegalStateException(\"Cannot parse serving range string: \" + boundaryString);\n    }\n  }\n\n  private static TierServingBoundaryEndPoint inferBoundaryFromDataRange(Date boundaryDate,\n                                                                        Clock clock) {\n    // infer from data range\n    // handle default start date and end date, in case the dates are not specified in the config\n    if (boundaryDate.equals(TierConfig.DEFAULT_TIER_START_DATE)) {\n      return new TierServingBoundaryEndPoint(\n          -1L, TierConfig.DEFAULT_TIER_START_DATE.getTime() / 1000, null, clock);\n    } else if (boundaryDate.equals(TierConfig.DEFAULT_TIER_END_DATE)) {\n      return new TierServingBoundaryEndPoint(\n          Long.MAX_VALUE, TierConfig.DEFAULT_TIER_END_DATE.getTime() / 1000, null, clock);\n    } else {\n      // convert data start / end dates into since / max ID.\n      long boundaryTimeMillis = boundaryDate.getTime();\n      if (!SnowflakeIdParser.isUsableSnowflakeTimestamp(boundaryTimeMillis)) {\n        throw new IllegalStateException(\"Serving time range can not be determined, because \"\n            + boundaryDate + \" is before Twitter switched to snowflake tweet IDs.\");\n      }\n      // Earlybird since_id is inclusive and max_id is exclusive. We substract 1 here.\n      // Consider example:\n      //   full0:  5000 (inclusive) - 6000 (exclusive)\n      //   full1:  6000 (inclusive) - 7000 (exclusive)\n      // For tier full0, we should use max_id 5999 instead of 6000.\n      // For tier full1, we should use since_id 5999 instead of 6000.\n      // Hence we substract 1 here.\n      long adjustedTweetId =\n        SnowflakeIdParser.generateValidStatusId(boundaryTimeMillis, 0) - 1;\n      Preconditions.checkState(adjustedTweetId >= 0, \"boundary tweet ID must be non-negative\");\n      return new TierServingBoundaryEndPoint(\n          adjustedTweetId, boundaryTimeMillis / 1000, null, clock);\n    }\n  }\n\n  private static TierServingBoundaryEndPoint getRelativeBoundary(String boundaryString,\n                                                                 Clock clock) {\n    // An offset relative to current time is given\n    String[] parts = boundaryString.split(\":\");\n    Preconditions.checkState(parts.length == 2);\n    long offset = Long.parseLong(parts[1]);\n    return new TierServingBoundaryEndPoint(null, null, offset, clock);\n  }\n\n  /**\n   * Returns the tweet ID for this tier boundary. If the tier boundary was created using a tweet ID,\n   * that tweet ID is returned. Otherwise, a tweet ID is derived from the time boundary.\n   */\n  @VisibleForTesting\n  public long getBoundaryTweetId() {\n    // If absoluteTweetId is available, use it.\n    if (absoluteTweetId != null) {\n      return absoluteTweetId;\n    } else {\n      Preconditions.checkNotNull(offsetToCurrentTimeMillis);\n      long boundaryTime = clock.nowMillis() + offsetToCurrentTimeMillis;\n      return SnowflakeIdParser.generateValidStatusId(boundaryTime, 0);\n    }\n  }\n\n  /**\n   * Returns the time boundary for this tier boundary, in seconds since epoch.\n   */\n  public long getBoundaryTimeSecondsFromEpoch() {\n    if (timeBoundarySecondsFromEpoch != null) {\n      return timeBoundarySecondsFromEpoch;\n    } else {\n      Preconditions.checkNotNull(offsetToCurrentTimeMillis);\n      return (clock.nowMillis() + offsetToCurrentTimeMillis) / 1000;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/document/DeletedStatus.java",
    "content": "package com.twitter.search.earlybird.document;\n\n/**\n * DeletedStatus is a marker indicating that the specified tweet in the specified\n * timeslice has been deleted.\n */\npublic final class DeletedStatus {\n  public final long timeSliceID;\n  public final long statusID;\n\n  public DeletedStatus(long timeSliceID, long statusID) {\n    this.timeSliceID = timeSliceID;\n    this.statusID = statusID;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/document/DocumentFactory.java",
    "content": "package com.twitter.search.earlybird.document;\n\nimport java.io.IOException;\nimport javax.annotation.Nullable;\n\nimport org.apache.commons.codec.binary.Base64;\nimport org.apache.lucene.document.Document;\nimport org.apache.lucene.document.Field;\nimport org.apache.lucene.document.FieldType;\nimport org.apache.lucene.index.IndexableField;\nimport org.apache.thrift.TBase;\nimport org.apache.thrift.TException;\nimport org.apache.thrift.TSerializer;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.util.text.OmitNormTextField;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\n\n/**\n * Factory that constructs a Lucene document from a thrift object stored in T format.\n *\n * @param <T> ThriftStatus or ThriftIndexingEvent, to be converted to a Lucene Document.\n */\npublic abstract class DocumentFactory<T extends TBase<T, ?>> {\n  private static final Logger LOG = LoggerFactory.getLogger(DocumentFactory.class);\n  private static final int MAX_ALLOWED_INVALID_DOCUMENTS = 100;\n\n  private static final SearchCounter INVALID_DOCUMENTS_COUNTER =\n      SearchCounter.export(\"invalid_documents\");\n\n  private final CriticalExceptionHandler criticalExceptionHandler;\n\n  public DocumentFactory(CriticalExceptionHandler criticalExceptionHandler) {\n    this.criticalExceptionHandler = criticalExceptionHandler;\n  }\n\n  /**\n   * Given the thrift representation of a tweet, returns the associated tweetId.\n   */\n  public abstract long getStatusId(T thriftObject);\n\n  /**\n   * Given the thrift representation of a tweet, returns a Lucene Document with all the fields\n   * that need to be indexed.\n   */\n  @Nullable\n  public final Document newDocument(T thriftObject) {\n    try {\n      return innerNewDocument(thriftObject);\n    } catch (Exception e) {\n      String statusId = \"Not available\";\n      if (thriftObject != null) {\n        try {\n          statusId = Long.toString(getStatusId(thriftObject));\n        } catch (Exception ex) {\n          LOG.error(\"Unable to get tweet id for document\", ex);\n          statusId = \"Not parsable\";\n        }\n      }\n      LOG.error(\"Unexpected exception while indexing. Status id: \" + statusId, e);\n\n      if (thriftObject != null) {\n        // Log the status in base64 for debugging\n        try {\n          LOG.warn(\"Bad ThriftStatus. Id: \" + statusId + \" base 64: \"\n              + Base64.encodeBase64String(new TSerializer().serialize(thriftObject)));\n        } catch (TException e1) {\n          // Ignored since this is logging for debugging.\n        }\n      }\n      INVALID_DOCUMENTS_COUNTER.increment();\n      if (INVALID_DOCUMENTS_COUNTER.get() > MAX_ALLOWED_INVALID_DOCUMENTS) {\n        criticalExceptionHandler.handle(this, e);\n      }\n      return new Document();\n    }\n  }\n\n  /**\n   * Given the thrift representation of a tweet, returns a Lucene Document with all the fields\n   * that need to be indexed.\n   *\n   * Return null if the given thrift object is invalid.\n   *\n   * @throws IOException if there are problems reading the input of producing the output. Exception\n   *         is handled in {@link #newDocument(TBase)}.\n   */\n  @Nullable\n  protected abstract Document innerNewDocument(T thriftObject) throws IOException;\n\n  // Helper methods that prevent us from adding null fields to the lucene index\n  protected void addField(Document document, IndexableField field) {\n    if (field != null) {\n      document.add(field);\n    }\n  }\n\n  protected Field newField(String data, String fieldName) {\n    return newField(data, fieldName, OmitNormTextField.TYPE_NOT_STORED);\n  }\n\n  protected Field newField(String data, String fieldName, FieldType fieldType) {\n    if (data != null) {\n      return new Field(fieldName, data, fieldType);\n    }\n    return null;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/document/ThriftDocumentPreprocessor.java",
    "content": "package com.twitter.search.earlybird.document;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchTruthTableCounter;\nimport com.twitter.search.common.schema.base.FieldNameToIdMapping;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.ThriftDocumentUtil;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdEncodedFeatures;\nimport com.twitter.search.common.schema.earlybird.EarlybirdEncodedFeaturesUtil;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.schema.earlybird.EarlybirdThriftDocumentUtil;\nimport com.twitter.search.common.schema.thriftjava.ThriftDocument;\nimport com.twitter.search.common.schema.thriftjava.ThriftField;\n\nimport geo.google.datamodel.GeoAddressAccuracy;\n\n/**\n * Used to preprocess a ThriftDocument before indexing.\n */\npublic final class ThriftDocumentPreprocessor {\n  private static final FieldNameToIdMapping ID_MAP = new EarlybirdFieldConstants();\n  private static final String FILTER_LINK_VALUE = EarlybirdThriftDocumentUtil.formatFilter(\n      EarlybirdFieldConstant.LINKS_FIELD.getFieldName());\n  private static final String HAS_LINK_VALUE = EarlybirdFieldConstant.getFacetSkipFieldName(\n      EarlybirdFieldConstant.LINKS_FIELD.getFieldName());\n\n  private ThriftDocumentPreprocessor() {\n  }\n\n  /**\n   * Processes the given document.\n   */\n  public static ThriftDocument preprocess(\n      ThriftDocument doc, EarlybirdCluster cluster, ImmutableSchemaInterface schema)\n      throws IOException {\n    patchArchiveThriftDocumentAccuracy(doc, cluster);\n    patchArchiveHasLinks(doc, cluster);\n    addAllMissingMinEngagementFields(doc, cluster, schema);\n    return doc;\n  }\n\n  private static final SearchCounter GEO_SCRUBBED_COUNT =\n      SearchCounter.export(\"geo_scrubbed_count\");\n  private static final SearchCounter GEO_ARCHIVE_PATCHED_ACCURACY_COUNT =\n      SearchCounter.export(\"geo_archive_patched_accuracy_count\");\n  private static final SearchCounter GEO_MISSING_COORDINATE_COUNT =\n      SearchCounter.export(\"geo_missing_coordinate_count\");\n  private static final SearchCounter ARCHIVED_LINKS_FIELD_PATCHED_COUNT =\n      SearchCounter.export(\"links_field_patched_count\");\n\n  /**\n   * Counter for all the combinations of nullcast bit set and nullcast filter set.\n   *\n   * Sum over `ThriftDocumentPreprocessor_nullcast_doc_stats__nullcastBitSet_true_*` to get all docs\n   * with nullcast bit set to true.\n   */\n  private static final SearchTruthTableCounter NULLCAST_DOC_STATS =\n      SearchTruthTableCounter.export(\n          \"ThriftDocumentPreprocessor_nullcast_doc_stats\",\n          \"nullcastBitSet\",\n          \"nullcastFilterSet\");\n\n  /***\n   * See JIRA SEARCH-7329\n   */\n  private static void patchArchiveThriftDocumentAccuracy(ThriftDocument doc,\n                                                         EarlybirdCluster cluster) {\n    ThriftField geoField = ThriftDocumentUtil.getField(\n        doc,\n        EarlybirdFieldConstant.GEO_HASH_FIELD.getFieldName(),\n        ID_MAP);\n    if (geoField != null) {\n      if (!geoField.getFieldData().isSetGeoCoordinate()) {\n        GEO_MISSING_COORDINATE_COUNT.increment();\n        return;\n      }\n\n      // -1 means that the data is geo scrubbed.\n      if (geoField.getFieldData().getGeoCoordinate().getAccuracy() == -1) {\n        doc.getFields().remove(geoField);\n        GEO_SCRUBBED_COUNT.increment();\n      } else if (EarlybirdCluster.isArchive(cluster)) {\n        // In archive indexing, we base precision on SearchArchiveStatus.getPrecision, which is not\n        // in the scale we want.  We always use POINT_LEVEL scale for now.\n        geoField.getFieldData().getGeoCoordinate().setAccuracy(\n            GeoAddressAccuracy.POINT_LEVEL.getCode());\n        GEO_ARCHIVE_PATCHED_ACCURACY_COUNT.increment();\n      }\n    }\n  }\n\n  /**\n   * See SEARCH-9635\n   * This patch is used to replace\n   *   (\"field\":\"internal\",\"term\":\"__filter_links\") with\n   *   (\"field\":\"internal\",\"term\":\"__has_links\").\n   */\n  private static void patchArchiveHasLinks(ThriftDocument doc, EarlybirdCluster cluster) {\n    if (!EarlybirdCluster.isArchive(cluster)) {\n      return;\n    }\n\n    List<ThriftField> fieldList = ThriftDocumentUtil.getFields(doc,\n        EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n        ID_MAP);\n    for (ThriftField field : fieldList) {\n      if (field.getFieldData().getStringValue().equals(FILTER_LINK_VALUE)) {\n        field.getFieldData().setStringValue(HAS_LINK_VALUE);\n        ARCHIVED_LINKS_FIELD_PATCHED_COUNT.increment();\n        break;\n      }\n    }\n  }\n\n  /**\n   * Check whether the nullcast bit and nullcast filter are consistent in the given doc.\n   */\n  public static boolean isNullcastBitAndFilterConsistent(ThriftDocument doc,\n                                                         ImmutableSchemaInterface schema) {\n    return isNullcastBitAndFilterConsistent(doc, schema, NULLCAST_DOC_STATS);\n  }\n\n  @VisibleForTesting\n  static boolean isNullcastBitAndFilterConsistent(\n      ThriftDocument doc, ImmutableSchemaInterface schema, SearchTruthTableCounter nullCastStats) {\n    final boolean isNullcastBitSet = EarlybirdThriftDocumentUtil.isNullcastBitSet(schema, doc);\n    final boolean isNullcastFilterSet = EarlybirdThriftDocumentUtil.isNullcastFilterSet(doc);\n\n    // Track stats.\n    nullCastStats.record(isNullcastBitSet, isNullcastFilterSet);\n\n    return isNullcastBitSet == isNullcastFilterSet;\n  }\n\n  @VisibleForTesting\n  static void addAllMissingMinEngagementFields(\n      ThriftDocument doc, EarlybirdCluster cluster, ImmutableSchemaInterface schema\n  ) throws IOException {\n    if (!EarlybirdCluster.isArchive(cluster)) {\n      return;\n    }\n    EarlybirdFieldConstants.EarlybirdFieldConstant encodedFeatureFieldConstant =\n        EarlybirdFieldConstant.ENCODED_TWEET_FEATURES_FIELD;\n    byte[] encodedFeaturesBytes = ThriftDocumentUtil.getBytesValue(doc,\n        encodedFeatureFieldConstant.getFieldName(), ID_MAP);\n    if (encodedFeaturesBytes == null) {\n      return;\n    }\n    EarlybirdEncodedFeatures encodedFeatures = EarlybirdEncodedFeaturesUtil.fromBytes(\n        schema,\n        EarlybirdFieldConstant.ENCODED_TWEET_FEATURES_FIELD,\n        encodedFeaturesBytes,\n        0);\n    for (String field: EarlybirdFieldConstants.MIN_ENGAGEMENT_FIELD_TO_CSF_NAME_MAP.keySet()) {\n      EarlybirdFieldConstant csfEngagementField = EarlybirdFieldConstants\n          .MIN_ENGAGEMENT_FIELD_TO_CSF_NAME_MAP.get(field);\n      Preconditions.checkState(csfEngagementField != null);\n      int engagementCounter = encodedFeatures.getFeatureValue(csfEngagementField);\n      EarlybirdThriftDocumentUtil.addNormalizedMinEngagementField(doc, field, engagementCounter);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/document/ThriftIndexingEventDocumentFactory.java",
    "content": "package com.twitter.search.earlybird.document;\n\nimport java.io.IOException;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\n\nimport org.apache.lucene.document.Document;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.decider.DeciderUtil;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.search.common.schema.SchemaDocumentFactory;\nimport com.twitter.search.common.schema.base.FieldNameToIdMapping;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.base.ThriftDocumentUtil;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.schema.earlybird.EarlybirdThriftDocumentUtil;\nimport com.twitter.search.common.schema.thriftjava.ThriftDocument;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.util.text.filter.NormalizedTokenFilter;\nimport com.twitter.search.common.util.text.splitter.HashtagMentionPunctuationSplitter;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\n\npublic class ThriftIndexingEventDocumentFactory extends DocumentFactory<ThriftIndexingEvent> {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(ThriftIndexingEventDocumentFactory.class);\n\n  private static final FieldNameToIdMapping ID_MAPPING = new EarlybirdFieldConstants();\n  private static final long TIMESTAMP_ALLOWED_FUTURE_DELTA_MS = TimeUnit.SECONDS.toMillis(60);\n  private static final String FILTER_TWEETS_WITH_FUTURE_TWEET_ID_AND_CREATED_AT_DECIDER_KEY =\n      \"filter_tweets_with_future_tweet_id_and_created_at\";\n\n  private static final SearchCounter NUM_TWEETS_WITH_FUTURE_TWEET_ID_AND_CREATED_AT_MS =\n      SearchCounter.export(\"num_tweets_with_future_tweet_id_and_created_at_ms\");\n  private static final SearchCounter NUM_TWEETS_WITH_INCONSISTENT_TWEET_ID_AND_CREATED_AT_MS_FOUND =\n      SearchCounter.export(\"num_tweets_with_inconsistent_tweet_id_and_created_at_ms_found\");\n  private static final SearchCounter\n    NUM_TWEETS_WITH_INCONSISTENT_TWEET_ID_AND_CREATED_AT_MS_ADJUSTED =\n      SearchCounter.export(\"num_tweets_with_inconsistent_tweet_id_and_created_at_ms_adjusted\");\n  private static final SearchCounter NUM_TWEETS_WITH_INCONSISTENT_TWEET_ID_AND_CREATED_AT_MS_DROPPED\n    = SearchCounter.export(\"num_tweets_with_inconsistent_tweet_id_and_created_at_ms_dropped\");\n\n  @VisibleForTesting\n  static final String ENABLE_ADJUST_CREATED_AT_TIME_IF_MISMATCH_WITH_SNOWFLAKE =\n      \"enable_adjust_created_at_time_if_mismatch_with_snowflake\";\n\n  @VisibleForTesting\n  static final String ENABLE_DROP_CREATED_AT_TIME_IF_MISMATCH_WITH_SNOWFLAKE =\n      \"enable_drop_created_at_time_if_mismatch_with_snowflake\";\n\n  private final SchemaDocumentFactory schemaDocumentFactory;\n  private final EarlybirdCluster cluster;\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n  private final Decider decider;\n  private final Schema schema;\n  private final Clock clock;\n\n  public ThriftIndexingEventDocumentFactory(\n      Schema schema,\n      EarlybirdCluster cluster,\n      Decider decider,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    this(\n        schema,\n        getSchemaDocumentFactory(schema, cluster, decider),\n        cluster,\n        searchIndexingMetricSet,\n        decider,\n        Clock.SYSTEM_CLOCK,\n        criticalExceptionHandler\n    );\n  }\n\n  /**\n   * Returns a document factory that knows how to convert ThriftDocuments to Documents based on the\n   * provided schema.\n   */\n  public static SchemaDocumentFactory getSchemaDocumentFactory(\n      Schema schema,\n      EarlybirdCluster cluster,\n      Decider decider) {\n    return new SchemaDocumentFactory(schema,\n        Lists.newArrayList(\n            new TruncationTokenStreamWriter(cluster, decider),\n            (fieldInfo, stream) -> {\n              // Strip # @ $ symbols, and break up underscore connected tokens.\n              if (fieldInfo.getFieldType().useTweetSpecificNormalization()) {\n                return new HashtagMentionPunctuationSplitter(new NormalizedTokenFilter(stream));\n              }\n\n              return stream;\n            }));\n  }\n\n  @VisibleForTesting\n  protected ThriftIndexingEventDocumentFactory(\n      Schema schema,\n      SchemaDocumentFactory schemaDocumentFactory,\n      EarlybirdCluster cluster,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      Decider decider,\n      Clock clock,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    super(criticalExceptionHandler);\n    this.schema = schema;\n    this.schemaDocumentFactory = schemaDocumentFactory;\n    this.cluster = cluster;\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n    this.decider = decider;\n    this.clock = clock;\n  }\n\n  @Override\n  public long getStatusId(ThriftIndexingEvent event) {\n    Preconditions.checkNotNull(event);\n    if (event.isSetDocument() && event.getDocument() != null) {\n      ThriftDocument thriftDocument = event.getDocument();\n      try {\n        // Ideally, we should not call getSchemaSnapshot() here.  But, as this is called only to\n        // retrieve status id and the ID field is static, this is fine for the purpose.\n        thriftDocument = ThriftDocumentPreprocessor.preprocess(\n            thriftDocument, cluster, schema.getSchemaSnapshot());\n      } catch (IOException e) {\n        throw new IllegalStateException(\"Unable to obtain tweet ID from ThriftDocument\", e);\n      }\n      return ThriftDocumentUtil.getLongValue(\n          thriftDocument, EarlybirdFieldConstant.ID_FIELD.getFieldName(), ID_MAPPING);\n    } else {\n      throw new IllegalArgumentException(\"ThriftDocument is null inside ThriftIndexingEvent.\");\n    }\n  }\n\n  @Override\n  protected Document innerNewDocument(ThriftIndexingEvent event) throws IOException {\n    Preconditions.checkNotNull(event);\n    Preconditions.checkNotNull(event.getDocument());\n\n    ImmutableSchemaInterface schemaSnapshot = schema.getSchemaSnapshot();\n\n    // If the tweet id and create_at are in the future, do not index it.\n    if (areTweetIDAndCreateAtInTheFuture(event)\n        && DeciderUtil.isAvailableForRandomRecipient(decider,\n        FILTER_TWEETS_WITH_FUTURE_TWEET_ID_AND_CREATED_AT_DECIDER_KEY)) {\n      NUM_TWEETS_WITH_FUTURE_TWEET_ID_AND_CREATED_AT_MS.increment();\n      return null;\n    }\n\n    if (isNullcastBitAndFilterConsistent(schemaSnapshot, event)) {\n      ThriftDocument thriftDocument =\n          adjustOrDropIfTweetIDAndCreatedAtAreInconsistent(\n              ThriftDocumentPreprocessor.preprocess(event.getDocument(), cluster, schemaSnapshot));\n\n      if (thriftDocument != null) {\n        return schemaDocumentFactory.newDocument(thriftDocument);\n      } else {\n        return null;\n      }\n    } else {\n      return null;\n    }\n  }\n\n  private ThriftDocument adjustOrDropIfTweetIDAndCreatedAtAreInconsistent(ThriftDocument document) {\n    final long tweetID = EarlybirdThriftDocumentUtil.getID(document);\n    // Thrift document is storing created at in seconds.\n    final long createdAtMs = EarlybirdThriftDocumentUtil.getCreatedAtMs(document);\n\n    if (!SnowflakeIdParser.isTweetIDAndCreatedAtConsistent(tweetID, createdAtMs)) {\n      // Increment found counter.\n      NUM_TWEETS_WITH_INCONSISTENT_TWEET_ID_AND_CREATED_AT_MS_FOUND.increment();\n      LOG.error(\n          \"Found inconsistent tweet ID and created at timestamp: [tweetID={}], [createdAtMs={}]\",\n          tweetID, createdAtMs);\n\n      if (DeciderUtil.isAvailableForRandomRecipient(\n          decider, ENABLE_ADJUST_CREATED_AT_TIME_IF_MISMATCH_WITH_SNOWFLAKE)) {\n        // Update created at (and csf) with the time stamp in snow flake ID.\n        final long createdAtMsInID = SnowflakeIdParser.getTimestampFromTweetId(tweetID);\n        EarlybirdThriftDocumentUtil.replaceCreatedAtAndCreatedAtCSF(\n            document, (int) (createdAtMsInID / 1000));\n\n        // Increment adjusted counter.\n        NUM_TWEETS_WITH_INCONSISTENT_TWEET_ID_AND_CREATED_AT_MS_ADJUSTED.increment();\n        LOG.error(\n            \"Updated created at to match tweet ID: createdAtMs={}, tweetID={}, createdAtMsInID={}\",\n            createdAtMs, tweetID, createdAtMsInID);\n      } else if (DeciderUtil.isAvailableForRandomRecipient(\n          decider, ENABLE_DROP_CREATED_AT_TIME_IF_MISMATCH_WITH_SNOWFLAKE)) {\n        // Drop and increment counter!\n        NUM_TWEETS_WITH_INCONSISTENT_TWEET_ID_AND_CREATED_AT_MS_DROPPED.increment();\n        LOG.error(\n            \"Dropped tweet with inconsistent ID and timestamp: createdAtMs={}, tweetID={}\",\n            createdAtMs, tweetID);\n        return null;\n      }\n    }\n\n    return document;\n  }\n\n  private boolean isNullcastBitAndFilterConsistent(\n      ImmutableSchemaInterface schemaSnapshot,\n      ThriftIndexingEvent event) {\n    return ThriftDocumentPreprocessor.isNullcastBitAndFilterConsistent(\n        event.getDocument(), schemaSnapshot);\n  }\n\n  /**\n   * Check if the tweet ID and create_at are in the future and beyond the allowed\n   * TIMESTAMP_ALLOWED_FUTURE_DELTA_MS range from current time stamp.\n   */\n  private boolean areTweetIDAndCreateAtInTheFuture(ThriftIndexingEvent event) {\n    ThriftDocument document = event.getDocument();\n\n    final long tweetID = EarlybirdThriftDocumentUtil.getID(document);\n    if (tweetID < SnowflakeIdParser.SNOWFLAKE_ID_LOWER_BOUND) {\n      return false;\n    }\n\n    final long tweetIDTimestampMs = SnowflakeIdParser.getTimestampFromTweetId(tweetID);\n    final long allowedFutureTimestampMs = clock.nowMillis() + TIMESTAMP_ALLOWED_FUTURE_DELTA_MS;\n\n    final long createdAtMs = EarlybirdThriftDocumentUtil.getCreatedAtMs(document);\n    if (tweetIDTimestampMs > allowedFutureTimestampMs && createdAtMs > allowedFutureTimestampMs) {\n      LOG.error(\n          \"Found future tweet ID and created at timestamp: \"\n              + \"[tweetID={}], [createdAtMs={}], [compareDeltaMs={}]\",\n          tweetID, createdAtMs, TIMESTAMP_ALLOWED_FUTURE_DELTA_MS);\n      return true;\n    }\n\n    return false;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/document/ThriftIndexingEventUpdateFactory.java",
    "content": "package com.twitter.search.earlybird.document;\n\nimport java.io.IOException;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.document.Document;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.schema.SchemaDocumentFactory;\nimport com.twitter.search.common.schema.base.FieldNameToIdMapping;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.base.ThriftDocumentUtil;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.schema.thriftjava.ThriftDocument;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\n\n/**\n * Builds a Lucene Document from a ThriftIndexingEvent. A simplified version of\n * {@link ThriftIndexingEventDocumentFactory} that can be used for update events, which exclude\n * many fields that the tweet indexing events contain.\n */\npublic class ThriftIndexingEventUpdateFactory extends DocumentFactory<ThriftIndexingEvent> {\n  private static final FieldNameToIdMapping ID_MAPPING = new EarlybirdFieldConstants();\n\n  private final SchemaDocumentFactory schemaDocumentFactory;\n  private final EarlybirdCluster cluster;\n  private final Schema schema;\n\n  public ThriftIndexingEventUpdateFactory(\n      Schema schema,\n      EarlybirdCluster cluster,\n      Decider decider,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    this(\n        schema,\n        ThriftIndexingEventDocumentFactory.getSchemaDocumentFactory(schema, cluster, decider),\n        cluster,\n        criticalExceptionHandler\n    );\n  }\n\n  @VisibleForTesting\n  protected ThriftIndexingEventUpdateFactory(\n      Schema schema,\n      SchemaDocumentFactory schemaDocumentFactory,\n      EarlybirdCluster cluster,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    super(criticalExceptionHandler);\n    this.schema = schema;\n    this.schemaDocumentFactory = schemaDocumentFactory;\n    this.cluster = cluster;\n  }\n\n  @Override\n  public long getStatusId(ThriftIndexingEvent event) {\n    Preconditions.checkNotNull(event);\n    Preconditions.checkState(\n        event.isSetDocument(), \"ThriftDocument is null inside ThriftIndexingEvent.\");\n\n    ThriftDocument thriftDocument;\n    try {\n      // Ideally, we should not call getSchemaSnapshot() here.  But, as this is called only to\n      // retrieve status id and the ID field is static, this is fine for the purpose.\n      thriftDocument = ThriftDocumentPreprocessor.preprocess(\n          event.getDocument(), cluster, schema.getSchemaSnapshot());\n    } catch (IOException e) {\n      throw new IllegalStateException(\"Unable to obtain tweet ID from ThriftDocument: \" + event, e);\n    }\n    return ThriftDocumentUtil.getLongValue(\n        thriftDocument, EarlybirdFieldConstant.ID_FIELD.getFieldName(), ID_MAPPING);\n  }\n\n  @Override\n  protected Document innerNewDocument(ThriftIndexingEvent event) throws IOException {\n    Preconditions.checkNotNull(event);\n    Preconditions.checkNotNull(event.getDocument());\n\n    ImmutableSchemaInterface schemaSnapshot = schema.getSchemaSnapshot();\n\n    ThriftDocument document = ThriftDocumentPreprocessor.preprocess(\n        event.getDocument(), cluster, schemaSnapshot);\n\n    return schemaDocumentFactory.newDocument(document);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/document/TimeSlicedThriftIndexingEvent.java",
    "content": "package com.twitter.search.earlybird.document;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\n\n/**\n * Object to encapsulate {@link ThriftIndexingEvent} with a time slice ID.\n */\npublic class TimeSlicedThriftIndexingEvent {\n  private final long timeSliceID;\n  private final ThriftIndexingEvent thriftIndexingEvent;\n\n  public TimeSlicedThriftIndexingEvent(long timeSliceID, ThriftIndexingEvent thriftIndexingEvent) {\n    Preconditions.checkNotNull(thriftIndexingEvent);\n\n    this.timeSliceID = timeSliceID;\n    this.thriftIndexingEvent = thriftIndexingEvent;\n  }\n\n  public long getStatusID() {\n    return thriftIndexingEvent.getUid();\n  }\n\n  public long getTimeSliceID() {\n    return timeSliceID;\n  }\n\n  public ThriftIndexingEvent getThriftIndexingEvent() {\n    return thriftIndexingEvent;\n  }\n\n  @Override\n  public String toString() {\n    return \"TimeSlicedThriftIndexingEvent{\"\n        + \"timeSliceID=\" + timeSliceID\n        + \", thriftIndexingEvent=\" + thriftIndexingEvent\n        + '}';\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/document/TruncationTokenStreamWriter.java",
    "content": "package com.twitter.search.earlybird.document;\n\nimport com.twitter.common.text.token.TokenProcessor;\nimport com.twitter.common.text.token.TwitterTokenStream;\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.decider.DeciderUtil;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.schema.SchemaDocumentFactory;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\n\npublic class TruncationTokenStreamWriter implements SchemaDocumentFactory.TokenStreamRewriter {\n  private static final int NEVER_TRUNCATE_CHARS_BELOW_POSITION = 140;\n  private static final String TRUNCATE_LONG_TWEETS_DECIDER_KEY_PREFIX =\n      \"truncate_long_tweets_in_\";\n  private static final String NUM_TWEET_CHARACTERS_SUPPORTED_DECIDER_KEY_PREFIX =\n      \"num_tweet_characters_supported_in_\";\n\n  private static final SearchCounter NUM_TWEETS_TRUNCATED =\n      SearchCounter.export(\"num_tweets_truncated\");\n  private static final SearchLongGauge NUM_TWEET_CHARACTERS_SUPPORTED =\n      SearchLongGauge.export(\"num_tweet_characters_supported\");\n\n  private final Decider decider;\n  private final String truncateLongTweetsDeciderKey;\n  private final String numCharsSupportedDeciderKey;\n\n  /**\n   * Creates a TruncationTokenStreamWriter\n   */\n  public TruncationTokenStreamWriter(EarlybirdCluster cluster, Decider decider) {\n    this.decider = decider;\n\n    this.truncateLongTweetsDeciderKey =\n        TRUNCATE_LONG_TWEETS_DECIDER_KEY_PREFIX + cluster.name().toLowerCase();\n    this.numCharsSupportedDeciderKey =\n        NUM_TWEET_CHARACTERS_SUPPORTED_DECIDER_KEY_PREFIX + cluster.name().toLowerCase();\n  }\n\n  @Override\n  public TwitterTokenStream rewrite(Schema.FieldInfo fieldInfo, TwitterTokenStream stream) {\n    if (EarlybirdFieldConstant.TEXT_FIELD.getFieldName().equals(fieldInfo.getName())) {\n      final int maxPosition = getTruncatePosition();\n      NUM_TWEET_CHARACTERS_SUPPORTED.set(maxPosition);\n      if (maxPosition >= NEVER_TRUNCATE_CHARS_BELOW_POSITION) {\n        return new TokenProcessor(stream) {\n          @Override\n          public final boolean incrementToken() {\n            if (incrementInputStream()) {\n              if (offset() < maxPosition) {\n                return true;\n              }\n              NUM_TWEETS_TRUNCATED.increment();\n            }\n\n            return false;\n          }\n        };\n      }\n    }\n\n    return stream;\n  }\n\n  /**\n   * Get the truncation position.\n   *\n   * @return the truncation position or -1 if truncation is disabled.\n   */\n  private int getTruncatePosition() {\n    int maxPosition;\n    if (!DeciderUtil.isAvailableForRandomRecipient(decider, truncateLongTweetsDeciderKey)) {\n      return -1;\n    }\n    maxPosition = DeciderUtil.getAvailability(decider, numCharsSupportedDeciderKey);\n\n    if (maxPosition < NEVER_TRUNCATE_CHARS_BELOW_POSITION) {\n      // Never truncate below NEVER_TRUNCATE_CHARS_BELOW_POSITION chars\n      maxPosition = NEVER_TRUNCATE_CHARS_BELOW_POSITION;\n    }\n\n    return maxPosition;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/document/TweetDocument.java",
    "content": "package com.twitter.search.earlybird.document;\n\nimport org.apache.lucene.document.Document;\n\n/**\n * TweetDocument is a record produced by DocumentReader and TweetIndexUpdateReader\n * for consumption by the partition indexer.\n */\npublic final class TweetDocument {\n  private final long tweetID;\n  private final long timeSliceID;\n  private final long eventTimeMs;\n  private final Document document;\n\n  public TweetDocument(\n      long tweetID,\n      long timeSliceID,\n      long eventTimeMs,\n      Document document\n  ) {\n    this.tweetID = tweetID;\n    this.timeSliceID = timeSliceID;\n    this.eventTimeMs = eventTimeMs;\n    this.document = document;\n  }\n\n  public long getTweetID() {\n    return tweetID;\n  }\n\n  public long getTimeSliceID() {\n    return timeSliceID;\n  }\n\n  public long getEventTimeMs() {\n    return eventTimeMs;\n  }\n\n  public Document getDocument() {\n    return document;\n  }\n\n  @Override\n  public String toString() {\n    return \"TweetDocument{\"\n        + \"tweetID=\" + tweetID\n        + \", timeSliceID=\" + timeSliceID\n        + \", eventTimeMs=\" + eventTimeMs\n        + \", document=\" + document\n        + '}';\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/AlreadyInServerSetUpdateException.java",
    "content": "package com.twitter.search.earlybird.exception;\n\nimport com.twitter.common.zookeeper.ServerSet;\n\n/**\n * Used when trying to join a server set when this earlybird is already in a server set.\n */\npublic class AlreadyInServerSetUpdateException extends ServerSet.UpdateException {\n  public AlreadyInServerSetUpdateException(String message) {\n    super(message);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/BadRequestException.java",
    "content": "package com.twitter.search.earlybird.exception;\n\npublic class BadRequestException extends Exception {\n  public BadRequestException(String message, Throwable cause) {\n    super(message, cause);\n  }\n\n  public BadRequestException(String message) {\n    super(message);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/ClientException.java",
    "content": "package com.twitter.search.earlybird.exception;\n\npublic class ClientException extends Exception {\n  public ClientException(Throwable t) {\n    super(t);\n  }\n\n  public ClientException(String message) {\n    super(message);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/CriticalExceptionHandler.java",
    "content": "package com.twitter.search.earlybird.exception;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport org.slf4j.Marker;\nimport org.slf4j.MarkerFactory;\n\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.EarlybirdStatus;\n\n/**\n * Used for handling exceptions considered critical.\n *\n * When you handle an exception with this class, two things might happen.\n * 1. If earlybirds are still starting, we'll shut them down.\n * 2. If earlybirds have started, we'll increment a counter that will cause alerts.\n *\n * If you want to verify that your code handles exceptions as you expect, you can use the\n * helper class ExceptionCauser.\n */\npublic class CriticalExceptionHandler {\n  private static final Logger LOG = LoggerFactory.getLogger(CriticalExceptionHandler.class);\n  private static final Marker FATAL = MarkerFactory.getMarker(\"FATAL\");\n\n  // This stat should remain at 0 during normal operations.\n  // This stat being non-zero should trigger alerts.\n  public static final SearchCounter CRITICAL_EXCEPTION_COUNT =\n      SearchCounter.export(\"fatal_exception_count\");\n\n  public static final SearchCounter UNSAFE_MEMORY_ACCESS =\n      SearchCounter.export(\"unsafe_memory_access\");\n\n  private Runnable shutdownHook;\n\n  public void setShutdownHook(Runnable shutdownHook) {\n    this.shutdownHook = shutdownHook;\n  }\n\n  /**\n   * Handle a critical exception.\n   *\n   * @param thrower Instance of the class where the exception was thrown.\n   * @param thrown The exception.\n   */\n  public void handle(Object thrower, Throwable thrown) {\n    if (thrown == null) {\n      return;\n    }\n\n    try {\n      handleFatalException(thrower, thrown);\n    } catch (Throwable e) {\n      LOG.error(\"Unexpected exception in EarlybirdExceptionHandler.handle() while handling an \"\n                + \"unexpected exception from \" + thrower.getClass(), e);\n    }\n  }\n\n  @VisibleForTesting\n  boolean shouldIncrementFatalExceptionCounter(Throwable thrown) {\n    // See D212952\n    // We don't want to get pages when this happens.\n    for (Throwable t = thrown; t != null; t = t.getCause()) {\n      if (t instanceof InternalError && t.getMessage() != null\n          && t.getMessage().contains(\"unsafe memory access operation\")) {\n        // Don't treat InternalError caused by unsafe memory access operation which is usually\n        // triggered by SIGBUS for accessing a corrupted memory block.\n        UNSAFE_MEMORY_ACCESS.increment();\n        return false;\n      }\n    }\n\n    return true;\n  }\n\n  /**\n   * Handle an exception that's considered fatal.\n   *\n   * @param thrower instance of the class where the exception was thrown.\n   * @param thrown The Error or Exception.\n   */\n  private void handleFatalException(Object thrower, Throwable thrown) {\n    LOG.error(FATAL, \"Fatal exception in \" + thrower.getClass() + \":\", thrown);\n\n    if (shouldIncrementFatalExceptionCounter(thrown)) {\n      CRITICAL_EXCEPTION_COUNT.increment();\n    }\n\n    if (EarlybirdStatus.isStarting()) {\n      LOG.error(FATAL, \"Got fatal exception while starting up, exiting ...\");\n      if (this.shutdownHook != null) {\n        this.shutdownHook.run();\n      } else {\n        LOG.error(\"earlybirdServer not set, can't shut down.\");\n      }\n\n      if (!Config.environmentIsTest()) {\n        // Sleep for 3 minutes to allow the fatal exception to be caught by observability.\n        try {\n          Thread.sleep(3 * 60 * 1000);\n        } catch (InterruptedException e) {\n          LOG.error(FATAL, \"interupted sleep while shutting down.\");\n        }\n        LOG.info(\"Terminate JVM.\");\n        //CHECKSTYLE:OFF RegexpSinglelineJava\n        // See SEARCH-15256\n        System.exit(-1);\n        //CHECKSTYLE:ON RegexpSinglelineJava\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/EarlybirdException.java",
    "content": "package com.twitter.search.earlybird.exception;\n\n/**\n * General Earlybird exception class to use instead of the Java exception class.\n */\npublic class EarlybirdException extends Exception {\n  public EarlybirdException(Throwable cause) {\n    super(cause);\n  }\n\n  public EarlybirdException(String message) {\n    super(message);\n  }\n\n  public EarlybirdException(String message, Throwable cause) {\n    super(message, cause);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/EarlybirdFinagleServerMonitor.java",
    "content": "package com.twitter.search.earlybird.exception;\n\nimport com.twitter.finagle.Failure;\nimport com.twitter.util.AbstractMonitor;\n\npublic class EarlybirdFinagleServerMonitor extends AbstractMonitor {\n  private final CriticalExceptionHandler criticalExceptionHandler;\n\n  public EarlybirdFinagleServerMonitor(CriticalExceptionHandler criticalExceptionHandler) {\n    this.criticalExceptionHandler = criticalExceptionHandler;\n  }\n\n  @Override\n  public boolean handle(Throwable e) {\n    if (e instanceof Failure) {\n      // skip Finagle failure\n      return true;\n    }\n\n    criticalExceptionHandler.handle(this, e);\n\n    // We return true here because we handle all exceptions.\n    return true;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/EarlybirdRuntimeException.java",
    "content": "package com.twitter.search.earlybird.exception;\n\npublic class EarlybirdRuntimeException extends RuntimeException {\n  public EarlybirdRuntimeException(Throwable cause) {\n    super(cause);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/EarlybirdStartupException.java",
    "content": "package com.twitter.search.earlybird.exception;\n\n/**\n * Thrown by code that is executed during startup and used to communicate to caller that startup\n * has failed. Generally results in shutting down of the server, but check on your own if you\n * need to.\n */\npublic class EarlybirdStartupException extends Exception {\n  public EarlybirdStartupException(Throwable cause) {\n    super(cause);\n  }\n\n  public EarlybirdStartupException(String message) {\n    super(message);\n  }\n\n  public EarlybirdStartupException(String message, Throwable cause) {\n    super(message, cause);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/FlushVersionMismatchException.java",
    "content": "package com.twitter.search.earlybird.exception;\n\nimport java.io.IOException;\n\npublic class FlushVersionMismatchException extends IOException {\n  public FlushVersionMismatchException(Throwable cause) {\n    super(cause);\n  }\n\n  public FlushVersionMismatchException(String message) {\n    super(message);\n  }\n\n  public FlushVersionMismatchException(String message, Throwable cause) {\n    super(message, cause);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/MissingKafkaTopicException.java",
    "content": "package com.twitter.search.earlybird.exception;\n\npublic class MissingKafkaTopicException extends Exception {\n  public MissingKafkaTopicException(String message) {\n    super(message);\n  }\n\n  public MissingKafkaTopicException(String message, Throwable cause) {\n    super(message, cause);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/MissingUserException.java",
    "content": "package com.twitter.search.earlybird.exception;\n\npublic class MissingUserException extends Exception {\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/NotInServerSetUpdateException.java",
    "content": "package com.twitter.search.earlybird.exception;\n\nimport com.twitter.common.zookeeper.ServerSet;\n\n/**\n * Used when trying to leave a server set when this earlybird is already out of the server set.\n */\npublic class NotInServerSetUpdateException extends ServerSet.UpdateException {\n  public NotInServerSetUpdateException(String message) {\n    super(message);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/TransientException.java",
    "content": "package com.twitter.search.earlybird.exception;\n\npublic class TransientException extends Exception {\n  public TransientException(Throwable t) {\n    super(t);\n  }\n\n  public TransientException(String message, Throwable cause) {\n    super(message, cause);\n  }\n\n  public TransientException(String message) {\n    super(message);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/UncaughtExceptionHandler.java",
    "content": "package com.twitter.search.earlybird.exception;\n\nimport com.twitter.util.AbstractMonitor;\n\npublic class UncaughtExceptionHandler extends AbstractMonitor {\n  private final CriticalExceptionHandler criticalExceptionHandler;\n\n  public UncaughtExceptionHandler() {\n    this.criticalExceptionHandler = new CriticalExceptionHandler();\n  }\n\n  public void setShutdownHook(Runnable shutdown) {\n    this.criticalExceptionHandler.setShutdownHook(shutdown);\n  }\n\n  @Override\n  public boolean handle(Throwable e) {\n    criticalExceptionHandler.handle(this, e);\n\n    // We return true here because we handle all exceptions.\n    return true;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/exception/WrappedKafkaApiException.java",
    "content": "package com.twitter.search.earlybird.exception;\n\nimport org.apache.kafka.common.errors.ApiException;\n\n/**\n * Kafka's ApiException class doesn't retain its stack trace (see its source code).\n * As a result a kafka exception that propagates up the call chain can't point to where exactly\n * did the exception happen in our code. As a solution, use this class when calling kafka API\n * methods.\n */\npublic class WrappedKafkaApiException extends RuntimeException {\n  public WrappedKafkaApiException(ApiException cause) {\n    super(cause);\n  }\n\n  public WrappedKafkaApiException(String message, ApiException cause) {\n    super(message, cause);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/factory/EarlybirdIndexConfigUtil.java",
    "content": "package com.twitter.search.earlybird.factory;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.RealtimeEarlybirdIndexConfig;\nimport com.twitter.search.earlybird.archive.ArchiveOnDiskEarlybirdIndexConfig;\nimport com.twitter.search.earlybird.archive.ArchiveSearchPartitionManager;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\n\npublic final class EarlybirdIndexConfigUtil {\n  private EarlybirdIndexConfigUtil() {\n  }\n\n  /**\n   * Creates the index config for this earlybird.\n   */\n  public static EarlybirdIndexConfig createEarlybirdIndexConfig(\n      Decider decider, SearchIndexingMetricSet searchIndexingMetricSet,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    if (isArchiveSearch()) {\n      return new ArchiveOnDiskEarlybirdIndexConfig(decider, searchIndexingMetricSet,\n          criticalExceptionHandler);\n    } else if (isProtectedSearch()) {\n      return new RealtimeEarlybirdIndexConfig(\n          EarlybirdCluster.PROTECTED, decider, searchIndexingMetricSet, criticalExceptionHandler);\n    } else if (isRealtimeCG()) {\n      return new RealtimeEarlybirdIndexConfig(\n          EarlybirdCluster.REALTIME_CG, decider, searchIndexingMetricSet, criticalExceptionHandler);\n    } else {\n      return new RealtimeEarlybirdIndexConfig(\n          EarlybirdCluster.REALTIME, decider, searchIndexingMetricSet, criticalExceptionHandler);\n    }\n  }\n\n  public static boolean isArchiveSearch() {\n    // Re-reading config on each call so that tests can reliably overwrite this\n    return EarlybirdConfig.getString(\"partition_manager\", \"realtime\")\n        .equals(ArchiveSearchPartitionManager.CONFIG_NAME);\n  }\n\n  private static boolean isProtectedSearch() {\n    // Re-reading config on each call so that tests can reliably overwrite this\n    return EarlybirdConfig.getBool(\"protected_index\", false);\n  }\n\n  private static boolean isRealtimeCG() {\n    // Re-reading config on each call so that tests can reliably overwrite this\n    return EarlybirdConfig.getBool(\"realtime_cg_index\", false);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/factory/EarlybirdKafkaConsumersFactory.java",
    "content": "package com.twitter.search.earlybird.factory;\n\nimport org.apache.kafka.clients.consumer.KafkaConsumer;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\n\npublic interface EarlybirdKafkaConsumersFactory {\n  /**\n   * Create a kafka consumer with default records to be polled.\n   */\n  KafkaConsumer<Long, ThriftVersionedEvents> createKafkaConsumer(\n      String clientID);\n\n  /**\n   * Create a kafka consumer with a set number of records to be polled.\n   */\n  KafkaConsumer<Long, ThriftVersionedEvents> createKafkaConsumer(\n      String clientID, int maxPollRecords);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/factory/EarlybirdServerFactory.java",
    "content": "package com.twitter.search.earlybird.factory;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.aurora.AuroraInstanceKey;\nimport com.twitter.search.common.aurora.AuroraSchedulerClient;\nimport com.twitter.search.common.concurrent.ScheduledExecutorServiceFactory;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.util.ml.tensorflow_engine.TensorflowModelsManager;\nimport com.twitter.search.common.util.zktrylock.ZooKeeperTryLockFactory;\nimport com.twitter.search.earlybird.EarlybirdDarkProxy;\nimport com.twitter.search.earlybird.EarlybirdFinagleServerManager;\nimport com.twitter.search.earlybird.EarlybirdFuturePoolManager;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.EarlybirdServer;\nimport com.twitter.search.earlybird.EarlybirdServerSetManager;\nimport com.twitter.search.earlybird.EarlybirdWarmUpManager;\nimport com.twitter.search.earlybird.QualityFactor;\nimport com.twitter.search.earlybird.UpdateableEarlybirdStateManager;\nimport com.twitter.search.earlybird.archive.ArchiveEarlybirdIndexConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.userupdates.UserScrubGeoMap;\nimport com.twitter.search.earlybird.common.userupdates.UserUpdatesChecker;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.index.EarlybirdSegmentFactory;\nimport com.twitter.search.earlybird.ml.ScoringModelsManager;\nimport com.twitter.search.earlybird.partition.AudioSpaceEventsStreamIndexer;\nimport com.twitter.search.earlybird.partition.AudioSpaceTable;\nimport com.twitter.search.earlybird.partition.DynamicPartitionConfig;\nimport com.twitter.search.earlybird.partition.EarlybirdStartup;\nimport com.twitter.search.earlybird.partition.MultiSegmentTermDictionaryManager;\nimport com.twitter.search.earlybird.partition.PartitionConfig;\nimport com.twitter.search.earlybird.partition.PartitionManager;\nimport com.twitter.search.earlybird.partition.SegmentManager;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\nimport com.twitter.search.earlybird.partition.UserScrubGeoEventStreamIndexer;\nimport com.twitter.search.earlybird.partition.UserUpdatesStreamIndexer;\nimport com.twitter.search.earlybird.querycache.QueryCacheConfig;\nimport com.twitter.search.earlybird.querycache.QueryCacheManager;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.util.TermCountMonitor;\nimport com.twitter.search.earlybird.util.TweetCountMonitor;\n\n/**\n * This is the wiring file that builds EarlybirdServers.\n * Production and test code share this same wiring file.\n * <p/>\n * To supply mocks for testing, one can do so by supplying a different\n * EarlybirdWiringModule to this wiring file.\n */\npublic final class EarlybirdServerFactory {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdServerFactory.class);\n\n  /**\n   * Creates the EarlybirdServer based on the bindings in the given wire module.\n   *\n   * @param earlybirdWireModule The wire module that specifies all required bindings.\n   */\n  public EarlybirdServer makeEarlybirdServer(EarlybirdWireModule earlybirdWireModule)\n      throws IOException {\n    LOG.info(\"Started making an Earlybird server\");\n    CriticalExceptionHandler criticalExceptionHandler = new CriticalExceptionHandler();\n    Decider decider = earlybirdWireModule.provideDecider();\n    SearchDecider searchDecider = new SearchDecider(decider);\n\n    EarlybirdWireModule.ZooKeeperClients zkClients = earlybirdWireModule.provideZooKeeperClients();\n    ZooKeeperTryLockFactory zkTryLockFactory =\n        zkClients.stateClient.createZooKeeperTryLockFactory();\n\n    EarlybirdIndexConfig earlybirdIndexConfig =\n        earlybirdWireModule.provideEarlybirdIndexConfig(\n            decider, earlybirdWireModule.provideSearchIndexingMetricSet(),\n            criticalExceptionHandler);\n\n    SearchStatsReceiver earlybirdServerStats =\n        earlybirdWireModule.provideEarlybirdServerStatsReceiver();\n\n    EarlybirdSearcherStats tweetsSearcherStats =\n        earlybirdWireModule.provideTweetsSearcherStats();\n\n    DynamicPartitionConfig dynamicPartitionConfig =\n        earlybirdWireModule.provideDynamicPartitionConfig();\n\n    PartitionConfig partitionConfig = dynamicPartitionConfig.getCurrentPartitionConfig();\n    LOG.info(\"Partition config info [Cluster: {}, Tier: {}, Partition: {}, Replica: {}]\",\n            partitionConfig.getClusterName(),\n            partitionConfig.getTierName(),\n            partitionConfig.getIndexingHashPartitionID(),\n            partitionConfig.getHostPositionWithinHashPartition());\n\n    Clock clock = earlybirdWireModule.provideClock();\n    UserUpdatesChecker userUpdatesChecker =\n        new UserUpdatesChecker(clock, decider, earlybirdIndexConfig.getCluster());\n\n    UserTable userTable = UserTable.newTableWithDefaultCapacityAndPredicate(\n        earlybirdIndexConfig.getUserTableFilter(partitionConfig)::apply);\n\n    UserScrubGeoMap userScrubGeoMap = new UserScrubGeoMap();\n\n    AudioSpaceTable audioSpaceTable = new AudioSpaceTable(clock);\n\n    SegmentSyncConfig segmentSyncConfig =\n        earlybirdWireModule.provideSegmentSyncConfig(earlybirdIndexConfig.getCluster());\n\n    SegmentManager segmentManager = earlybirdWireModule.provideSegmentManager(\n        dynamicPartitionConfig,\n        earlybirdIndexConfig,\n        earlybirdWireModule.provideSearchIndexingMetricSet(),\n        tweetsSearcherStats,\n        earlybirdServerStats,\n        userUpdatesChecker,\n        segmentSyncConfig,\n        userTable,\n        userScrubGeoMap,\n        clock,\n        criticalExceptionHandler);\n\n    QueryCacheConfig config = earlybirdWireModule.provideQueryCacheConfig(earlybirdServerStats);\n\n    QueryCacheManager queryCacheManager = earlybirdWireModule.provideQueryCacheManager(\n        config,\n        earlybirdIndexConfig,\n        partitionConfig.getMaxEnabledLocalSegments(),\n        userTable,\n        userScrubGeoMap,\n        earlybirdWireModule.provideQueryCacheUpdateTaskScheduledExecutorFactory(),\n        earlybirdServerStats,\n        tweetsSearcherStats,\n        decider,\n        criticalExceptionHandler,\n        clock);\n\n    EarlybirdServerSetManager serverSetManager = earlybirdWireModule.provideServerSetManager(\n        zkClients.discoveryClient,\n        dynamicPartitionConfig,\n        earlybirdServerStats,\n        EarlybirdConfig.getThriftPort(),\n        \"\");\n\n    EarlybirdWarmUpManager warmUpManager =\n        earlybirdWireModule.provideWarmUpManager(zkClients.discoveryClient,\n                                                 dynamicPartitionConfig,\n                                                 earlybirdServerStats,\n                                                 decider,\n                                                 clock,\n                                                 EarlybirdConfig.getWarmUpThriftPort(),\n                                                 \"warmup_\");\n\n    EarlybirdDarkProxy earlybirdDarkProxy = earlybirdWireModule.provideEarlybirdDarkProxy(\n        new SearchDecider(decider),\n        earlybirdWireModule.provideFinagleStatsReceiver(),\n        serverSetManager,\n        warmUpManager,\n        partitionConfig.getClusterName());\n\n    UserUpdatesStreamIndexer userUpdatesStreamIndexer =\n        earlybirdWireModule.provideUserUpdatesKafkaConsumer(segmentManager);\n\n    UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer =\n        earlybirdWireModule.provideUserScrubGeoEventKafkaConsumer(segmentManager);\n\n    AudioSpaceEventsStreamIndexer audioSpaceEventsStreamIndexer =\n        earlybirdWireModule.provideAudioSpaceEventsStreamIndexer(audioSpaceTable, clock);\n\n    MultiSegmentTermDictionaryManager.Config termDictionaryConfig =\n        earlybirdWireModule.provideMultiSegmentTermDictionaryManagerConfig();\n    MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager =\n        earlybirdWireModule.provideMultiSegmentTermDictionaryManager(\n            termDictionaryConfig,\n            segmentManager,\n            earlybirdServerStats,\n            decider,\n            earlybirdIndexConfig.getCluster());\n\n    TermCountMonitor termCountMonitor =\n        earlybirdWireModule.provideTermCountMonitor(\n            segmentManager, earlybirdWireModule.provideTermCountMonitorScheduledExecutorFactory(),\n            earlybirdServerStats,\n            criticalExceptionHandler);\n    TweetCountMonitor tweetCountMonitor =\n        earlybirdWireModule.provideTweetCountMonitor(\n            segmentManager, earlybirdWireModule.provideTweetCountMonitorScheduledExecutorFactory(),\n            earlybirdServerStats,\n            criticalExceptionHandler);\n\n    ScoringModelsManager scoringModelsManager = earlybirdWireModule.provideScoringModelsManager(\n        earlybirdServerStats,\n        earlybirdIndexConfig\n    );\n\n    TensorflowModelsManager tensorflowModelsManager =\n        earlybirdWireModule.provideTensorflowModelsManager(\n            earlybirdServerStats,\n            \"tf_loader\",\n            decider,\n            earlybirdIndexConfig\n        );\n\n    AuroraSchedulerClient schedulerClient = null;\n    AuroraInstanceKey auroraInstanceKey = EarlybirdConfig.getAuroraInstanceKey();\n    if (auroraInstanceKey != null) {\n      schedulerClient = new AuroraSchedulerClient(auroraInstanceKey.getCluster());\n    }\n\n    UpdateableEarlybirdStateManager earlybirdStateManager =\n        earlybirdWireModule.provideUpdateableEarlybirdStateManager(\n            earlybirdIndexConfig,\n            dynamicPartitionConfig,\n            zkClients.stateClient,\n            schedulerClient,\n            earlybirdWireModule.provideStateUpdateManagerExecutorFactory(),\n            scoringModelsManager,\n            tensorflowModelsManager,\n            earlybirdServerStats,\n            new SearchDecider(decider),\n            criticalExceptionHandler);\n\n    EarlybirdFuturePoolManager futurePoolManager = earlybirdWireModule.provideFuturePoolManager();\n    EarlybirdFinagleServerManager finagleServerManager =\n        earlybirdWireModule.provideFinagleServerManager(criticalExceptionHandler);\n\n    PartitionManager partitionManager = null;\n    if (EarlybirdIndexConfigUtil.isArchiveSearch()) {\n      partitionManager = buildArchivePartitionManager(\n          earlybirdWireModule,\n          userUpdatesStreamIndexer,\n          userScrubGeoEventStreamIndexer,\n          zkTryLockFactory,\n          earlybirdIndexConfig,\n          dynamicPartitionConfig,\n          segmentManager,\n          queryCacheManager,\n          earlybirdServerStats,\n          serverSetManager,\n          earlybirdWireModule.providePartitionManagerExecutorFactory(),\n          earlybirdWireModule.provideSimpleUserUpdateIndexerScheduledExecutorFactory(),\n          clock,\n          segmentSyncConfig,\n          criticalExceptionHandler);\n    } else {\n      LOG.info(\"Not creating PartitionManager\");\n    }\n\n    EarlybirdSegmentFactory earlybirdSegmentFactory = new EarlybirdSegmentFactory(\n        earlybirdIndexConfig,\n        earlybirdWireModule.provideSearchIndexingMetricSet(),\n        tweetsSearcherStats,\n        clock);\n\n    EarlybirdStartup earlybirdStartup = earlybirdWireModule.provideEarlybirdStartup(\n        partitionManager,\n        userUpdatesStreamIndexer,\n        userScrubGeoEventStreamIndexer,\n        audioSpaceEventsStreamIndexer,\n        dynamicPartitionConfig,\n        criticalExceptionHandler,\n        segmentManager,\n        multiSegmentTermDictionaryManager,\n        queryCacheManager,\n        zkTryLockFactory,\n        serverSetManager,\n        clock,\n        segmentSyncConfig,\n        earlybirdSegmentFactory,\n        earlybirdIndexConfig.getCluster(),\n        searchDecider);\n\n    QualityFactor qualityFactor = earlybirdWireModule.provideQualityFactor(\n        decider,\n        earlybirdServerStats);\n\n    EarlybirdServer earlybirdServer = new EarlybirdServer(\n        queryCacheManager,\n        zkClients.stateClient,\n        decider,\n        earlybirdIndexConfig,\n        dynamicPartitionConfig,\n        partitionManager,\n        segmentManager,\n        audioSpaceTable,\n        termCountMonitor,\n        tweetCountMonitor,\n        earlybirdStateManager,\n        futurePoolManager,\n        finagleServerManager,\n        serverSetManager,\n        warmUpManager,\n        earlybirdServerStats,\n        tweetsSearcherStats,\n        scoringModelsManager,\n        tensorflowModelsManager,\n        clock,\n        multiSegmentTermDictionaryManager,\n        earlybirdDarkProxy,\n        segmentSyncConfig,\n        earlybirdWireModule.provideQueryTimeoutFactory(),\n        earlybirdStartup,\n        qualityFactor,\n        earlybirdWireModule.provideSearchIndexingMetricSet());\n\n    earlybirdStateManager.setEarlybirdServer(earlybirdServer);\n    criticalExceptionHandler.setShutdownHook(earlybirdServer::shutdown);\n\n    return earlybirdServer;\n  }\n\n  private PartitionManager buildArchivePartitionManager(\n      EarlybirdWireModule earlybirdWireModule,\n      UserUpdatesStreamIndexer userUpdatesStreamIndexer,\n      UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer,\n      ZooKeeperTryLockFactory zkTryLockFactory,\n      EarlybirdIndexConfig earlybirdIndexConfig,\n      DynamicPartitionConfig dynamicPartitionConfig,\n      SegmentManager segmentManager,\n      QueryCacheManager queryCacheManager,\n      SearchStatsReceiver searchStatsReceiver,\n      EarlybirdServerSetManager serverSetManager,\n      ScheduledExecutorServiceFactory partitionManagerExecutorServiceFactory,\n      ScheduledExecutorServiceFactory simpleUserUpdateIndexerExecutorFactory,\n      Clock clock,\n      SegmentSyncConfig segmentSyncConfig,\n      CriticalExceptionHandler criticalExceptionHandler)\n      throws IOException {\n\n      Preconditions.checkState(earlybirdIndexConfig instanceof ArchiveEarlybirdIndexConfig);\n      LOG.info(\"Creating ArchiveSearchPartitionManager\");\n      return earlybirdWireModule.provideFullArchivePartitionManager(\n          zkTryLockFactory,\n          queryCacheManager,\n          segmentManager,\n          dynamicPartitionConfig,\n          userUpdatesStreamIndexer,\n          userScrubGeoEventStreamIndexer,\n          searchStatsReceiver,\n          (ArchiveEarlybirdIndexConfig) earlybirdIndexConfig,\n          serverSetManager,\n          partitionManagerExecutorServiceFactory,\n          simpleUserUpdateIndexerExecutorFactory,\n          earlybirdWireModule.provideSearchIndexingMetricSet(),\n          clock,\n          segmentSyncConfig,\n          criticalExceptionHandler);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/factory/EarlybirdWireModule.java",
    "content": "package com.twitter.search.earlybird.factory;\n\nimport java.io.IOException;\nimport java.lang.management.ManagementFactory;\nimport java.util.Optional;\nimport java.util.concurrent.ScheduledThreadPoolExecutor;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\nimport com.sun.management.OperatingSystemMXBean;\n\nimport org.apache.directory.api.util.Strings;\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.kafka.clients.consumer.KafkaConsumer;\nimport org.apache.kafka.common.TopicPartition;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.decider.Decider;\nimport com.twitter.finagle.stats.MetricsStatsReceiver;\nimport com.twitter.finagle.stats.StatsReceiver;\nimport com.twitter.search.common.aurora.AuroraSchedulerClient;\nimport com.twitter.search.common.concurrent.ScheduledExecutorServiceFactory;\nimport com.twitter.search.common.decider.DeciderUtil;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.file.FileUtils;\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.metrics.SearchStatsReceiverImpl;\nimport com.twitter.search.common.partitioning.zookeeper.SearchZkClient;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.search.termination.QueryTimeoutFactory;\nimport com.twitter.search.common.util.io.kafka.FinagleKafkaClientUtils;\nimport com.twitter.search.common.util.io.kafka.ThriftDeserializer;\nimport com.twitter.search.common.util.ml.tensorflow_engine.TensorflowModelsManager;\nimport com.twitter.search.common.util.zktrylock.ZooKeeperTryLockFactory;\nimport com.twitter.search.common.util.zookeeper.ZooKeeperProxy;\nimport com.twitter.search.earlybird.EarlybirdCPUQualityFactor;\nimport com.twitter.search.earlybird.EarlybirdDarkProxy;\nimport com.twitter.search.earlybird.EarlybirdFinagleServerManager;\nimport com.twitter.search.earlybird.EarlybirdFuturePoolManager;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.EarlybirdProductionFinagleServerManager;\nimport com.twitter.search.earlybird.EarlybirdServerSetManager;\nimport com.twitter.search.earlybird.EarlybirdWarmUpManager;\nimport com.twitter.search.earlybird.QualityFactor;\nimport com.twitter.search.earlybird.ServerSetMember;\nimport com.twitter.search.earlybird.UpdateableEarlybirdStateManager;\nimport com.twitter.search.earlybird.archive.ArchiveEarlybirdIndexConfig;\nimport com.twitter.search.earlybird.archive.ArchiveSearchPartitionManager;\nimport com.twitter.search.earlybird.common.CaughtUpMonitor;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.common.userupdates.UserScrubGeoMap;\nimport com.twitter.search.earlybird.common.userupdates.UserUpdatesChecker;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.exception.MissingKafkaTopicException;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.index.EarlybirdSegmentFactory;\nimport com.twitter.search.earlybird.ml.ScoringModelsManager;\nimport com.twitter.search.earlybird.partition.AudioSpaceEventsStreamIndexer;\nimport com.twitter.search.earlybird.partition.AudioSpaceTable;\nimport com.twitter.search.earlybird.partition.DynamicPartitionConfig;\nimport com.twitter.search.earlybird.partition.EarlybirdIndexFlusher;\nimport com.twitter.search.earlybird.partition.EarlybirdIndexLoader;\nimport com.twitter.search.earlybird.partition.EarlybirdKafkaConsumer;\nimport com.twitter.search.earlybird.partition.EarlybirdStartup;\nimport com.twitter.search.earlybird.partition.OptimizationAndFlushingCoordinationLock;\nimport com.twitter.search.earlybird.partition.TimeLimitedHadoopExistsCall;\nimport com.twitter.search.earlybird.partition.UserScrubGeoEventStreamIndexer;\nimport com.twitter.search.earlybird.partition.freshstartup.FreshStartupHandler;\nimport com.twitter.search.earlybird.partition.HdfsUtil;\nimport com.twitter.search.earlybird.partition.KafkaStartup;\nimport com.twitter.search.earlybird.partition.MultiSegmentTermDictionaryManager;\nimport com.twitter.search.earlybird.partition.PartitionManager;\nimport com.twitter.search.earlybird.partition.PartitionManagerStartup;\nimport com.twitter.search.earlybird.partition.PartitionWriter;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\nimport com.twitter.search.earlybird.partition.SegmentManager;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\nimport com.twitter.search.earlybird.partition.StartupUserEventIndexer;\nimport com.twitter.search.earlybird.partition.TweetCreateHandler;\nimport com.twitter.search.earlybird.partition.TweetUpdateHandler;\nimport com.twitter.search.earlybird.partition.UserUpdatesStreamIndexer;\nimport com.twitter.search.earlybird.querycache.QueryCacheConfig;\nimport com.twitter.search.earlybird.querycache.QueryCacheManager;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.util.CoordinatedEarlybirdAction;\nimport com.twitter.search.earlybird.util.EarlybirdDecider;\nimport com.twitter.search.earlybird.util.TermCountMonitor;\nimport com.twitter.search.earlybird.util.TweetCountMonitor;\nimport com.twitter.ubs.thriftjava.AudioSpaceBaseEvent;\n\n/**\n * Production module that provides Earlybird components.\n */\npublic class EarlybirdWireModule {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdWireModule.class);\n  private static final int MAX_POLL_RECORDS = 1000;\n\n  /**\n   * How many threads we will use for building up the query cache during startup.\n   * The number of threads will be set to 1 after this earlybird is current.\n   */\n  private static final int QUERY_CACHE_NUM_WORKER_THREADS_AT_STARTUP =\n      EarlybirdConfig.getInt(\"query_cache_updater_startup_threads\", 1);\n\n  /**\n   * Scheduled executor service factory can be re-used in production.\n   * All the managers can share the same executor service factory.\n   */\n  private final ScheduledExecutorServiceFactory sharedExecutorServiceFactory =\n      new ScheduledExecutorServiceFactory();\n\n  private final SearchStatsReceiver sharedSearchStatsReceiver = new SearchStatsReceiverImpl();\n  private final StatsReceiver sharedFinagleStatsReceiver = new MetricsStatsReceiver();\n\n  private final SearchIndexingMetricSet searchIndexingMetricSet =\n      new SearchIndexingMetricSet(sharedSearchStatsReceiver);\n\n  private final EarlybirdSearcherStats tweetsSearcherStats =\n      new EarlybirdSearcherStats(sharedSearchStatsReceiver);\n\n  private final CaughtUpMonitor indexCaughtUpMonitor = new CaughtUpMonitor(\"dl_index\");\n\n  public CaughtUpMonitor provideIndexCaughtUpMonitor() {\n    return indexCaughtUpMonitor;\n  }\n\n  private final CaughtUpMonitor kafkaIndexCaughtUpMonitor = new CaughtUpMonitor(\"kafka_index\");\n\n  public CaughtUpMonitor provideKafkaIndexCaughtUpMonitor() {\n    return kafkaIndexCaughtUpMonitor;\n  }\n\n  private final OptimizationAndFlushingCoordinationLock optimizationAndFlushingCoordinationLock =\n      new OptimizationAndFlushingCoordinationLock();\n\n  public OptimizationAndFlushingCoordinationLock provideOptimizationAndFlushingCoordinationLock() {\n    return optimizationAndFlushingCoordinationLock;\n  }\n\n  public QueryTimeoutFactory provideQueryTimeoutFactory() {\n    return new QueryTimeoutFactory();\n  }\n\n  public static class ZooKeeperClients {\n    public ZooKeeperProxy discoveryClient;\n    public ZooKeeperProxy stateClient;\n\n    public ZooKeeperClients() {\n      this(\n          SearchZkClient.getServiceDiscoveryZooKeeperClient(),\n          SearchZkClient.getSZooKeeperClient());\n    }\n\n    public ZooKeeperClients(ZooKeeperProxy discoveryClient, ZooKeeperProxy stateClient) {\n      this.discoveryClient = discoveryClient;\n      this.stateClient = stateClient;\n    }\n  }\n\n  /**\n   * Provides the earlybird decider.\n   */\n  public Decider provideDecider() {\n    return EarlybirdDecider.initialize();\n  }\n\n  /**\n   * Provides the set of ZooKeeper clients to be used by earlybird.\n   */\n  public ZooKeeperClients provideZooKeeperClients() {\n    return new ZooKeeperClients();\n  }\n\n  /**\n   * Provides the query cache config.\n   */\n  public QueryCacheConfig provideQueryCacheConfig(SearchStatsReceiver searchStatsReceiver) {\n    return new QueryCacheConfig(searchStatsReceiver);\n  }\n\n  /**\n   * Provides the earlybird index config.\n   */\n  public EarlybirdIndexConfig provideEarlybirdIndexConfig(\n      Decider decider, SearchIndexingMetricSet indexingMetricSet,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    return EarlybirdIndexConfigUtil.createEarlybirdIndexConfig(decider, indexingMetricSet,\n        criticalExceptionHandler);\n  }\n\n  public DynamicPartitionConfig provideDynamicPartitionConfig() {\n    return new DynamicPartitionConfig(PartitionConfigUtil.initPartitionConfig());\n  }\n\n  /**\n   * Provides the segment manager to be used by this earlybird.\n   */\n  public SegmentManager provideSegmentManager(\n      DynamicPartitionConfig dynamicPartitionConfig,\n      EarlybirdIndexConfig earlybirdIndexConfig,\n      SearchIndexingMetricSet partitionIndexingMetricSet,\n      EarlybirdSearcherStats searcherStats,\n      SearchStatsReceiver earlybirdServerStats,\n      UserUpdatesChecker userUpdatesChecker,\n      SegmentSyncConfig segmentSyncConfig,\n      UserTable userTable,\n      UserScrubGeoMap userScrubGeoMap,\n      Clock clock,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    return new SegmentManager(\n        dynamicPartitionConfig,\n        earlybirdIndexConfig,\n        partitionIndexingMetricSet,\n        searcherStats,\n        earlybirdServerStats,\n        userUpdatesChecker,\n        segmentSyncConfig,\n        userTable,\n        userScrubGeoMap,\n        clock,\n        EarlybirdConfig.getMaxSegmentSize(),\n        criticalExceptionHandler,\n        provideKafkaIndexCaughtUpMonitor());\n  }\n\n  public QueryCacheManager provideQueryCacheManager(\n      QueryCacheConfig config,\n      EarlybirdIndexConfig indexConfig,\n      int maxEnabledSegments,\n      UserTable userTable,\n      UserScrubGeoMap userScrubGeoMap,\n      ScheduledExecutorServiceFactory queryCacheUpdaterScheduledExecutorFactory,\n      SearchStatsReceiver searchStatsReceiver,\n      EarlybirdSearcherStats searcherStats,\n      Decider decider,\n      CriticalExceptionHandler criticalExceptionHandler,\n      Clock clock) {\n    return new QueryCacheManager(config, indexConfig, maxEnabledSegments, userTable,\n        userScrubGeoMap, queryCacheUpdaterScheduledExecutorFactory, searchStatsReceiver,\n        searcherStats, decider, criticalExceptionHandler, clock);\n  }\n\n  public TermCountMonitor provideTermCountMonitor(\n      SegmentManager segmentManager, ScheduledExecutorServiceFactory executorServiceFactory,\n      SearchStatsReceiver searchStatsReceiver,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    return new TermCountMonitor(segmentManager, executorServiceFactory, 500, TimeUnit.MILLISECONDS,\n        searchStatsReceiver, criticalExceptionHandler);\n  }\n\n  public TweetCountMonitor provideTweetCountMonitor(\n      SegmentManager segmentManager,\n      ScheduledExecutorServiceFactory executorServiceFactory,\n      SearchStatsReceiver searchStatsReceiver,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    return new TweetCountMonitor(segmentManager, executorServiceFactory, 500,\n        TimeUnit.MILLISECONDS, searchStatsReceiver, criticalExceptionHandler);\n  }\n\n  /**\n   * Returns a manager that keeps track of earlybird's global state while it runs.\n   */\n  public UpdateableEarlybirdStateManager provideUpdateableEarlybirdStateManager(\n      EarlybirdIndexConfig earlybirdIndexConfig,\n      DynamicPartitionConfig dynamicPartitionConfig,\n      ZooKeeperProxy zooKeeperClient,\n      AuroraSchedulerClient schedulerClient,\n      ScheduledExecutorServiceFactory executorServiceFactory,\n      ScoringModelsManager scoringModelsManager,\n      TensorflowModelsManager tensorflowModelsManager,\n      SearchStatsReceiver searchStatsReceiver,\n      SearchDecider searchDecider,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    Clock clock = provideClockForStateManager();\n\n    return new UpdateableEarlybirdStateManager(\n        earlybirdIndexConfig, dynamicPartitionConfig, zooKeeperClient, schedulerClient,\n        executorServiceFactory, scoringModelsManager, tensorflowModelsManager, searchStatsReceiver,\n        searchDecider, criticalExceptionHandler,\n        clock);\n  }\n\n  public Clock provideClockForStateManager() {\n    return this.provideClock();\n  }\n\n  public ScheduledExecutorServiceFactory providePartitionManagerExecutorFactory() {\n    return sharedExecutorServiceFactory;\n  }\n\n  public ScheduledExecutorServiceFactory provideStateUpdateManagerExecutorFactory() {\n    return sharedExecutorServiceFactory;\n  }\n\n  public ScheduledExecutorServiceFactory provideTermCountMonitorScheduledExecutorFactory() {\n    return sharedExecutorServiceFactory;\n  }\n\n  public ScheduledExecutorServiceFactory provideTweetCountMonitorScheduledExecutorFactory() {\n    return sharedExecutorServiceFactory;\n  }\n\n  /**\n   * Provides the ScheduledExecutorServiceFactory that will be used to schedule all query cache\n   * update tasks.\n   */\n  public ScheduledExecutorServiceFactory provideQueryCacheUpdateTaskScheduledExecutorFactory() {\n    return new ScheduledExecutorServiceFactory() {\n      @Override\n      public QueryCacheUpdaterScheduledExecutorService<ScheduledThreadPoolExecutor> build(\n          String threadNameFormat, boolean isDaemon) {\n        ScheduledThreadPoolExecutor threadpoolExecutor =\n            new ScheduledThreadPoolExecutor(QUERY_CACHE_NUM_WORKER_THREADS_AT_STARTUP,\n                buildThreadFactory(threadNameFormat, isDaemon));\n        threadpoolExecutor.setMaximumPoolSize(QUERY_CACHE_NUM_WORKER_THREADS_AT_STARTUP);\n        threadpoolExecutor.setCorePoolSize(QUERY_CACHE_NUM_WORKER_THREADS_AT_STARTUP);\n        threadpoolExecutor.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);\n        threadpoolExecutor.setContinueExistingPeriodicTasksAfterShutdownPolicy(false);\n        threadpoolExecutor.setRemoveOnCancelPolicy(true);\n        LOG.info(\"Starting query cache executor with {} thread.\",\n            QUERY_CACHE_NUM_WORKER_THREADS_AT_STARTUP);\n\n        return new QueryCacheUpdaterScheduledExecutorService<ScheduledThreadPoolExecutor>(\n            threadpoolExecutor) {\n          @Override public void setWorkerPoolSizeAfterStartup() {\n            delegate.setCorePoolSize(1);\n            delegate.setMaximumPoolSize(1);\n            LOG.info(\"Reset query cache executor to be single threaded.\");\n          }\n        };\n      }\n    };\n  }\n\n  public ScheduledExecutorServiceFactory provideSimpleUserUpdateIndexerScheduledExecutorFactory() {\n    return sharedExecutorServiceFactory;\n  }\n\n  /**\n   * Returns the manager that manages the pool of searcher threads.\n   */\n  public EarlybirdFuturePoolManager provideFuturePoolManager() {\n    return new EarlybirdFuturePoolManager(\"SearcherWorker\");\n  }\n\n  /**\n   * Returns the manager that manages all earlybird finagle servers (warm up and production).\n   */\n  public EarlybirdFinagleServerManager provideFinagleServerManager(\n      CriticalExceptionHandler criticalExceptionHandler) {\n    return new EarlybirdProductionFinagleServerManager(criticalExceptionHandler);\n  }\n\n  /**\n   * Creates the production serverset manager.\n   */\n  public EarlybirdServerSetManager provideServerSetManager(\n      ZooKeeperProxy discoveryClient,\n      DynamicPartitionConfig dynamicPartitionConfig,\n      SearchStatsReceiver searchStatsReceiver,\n      int port,\n      String serverSetNamePrefix) {\n    return new EarlybirdServerSetManager(\n        searchStatsReceiver,\n        discoveryClient,\n        dynamicPartitionConfig.getCurrentPartitionConfig(),\n        port,\n        serverSetNamePrefix);\n  }\n\n  /**\n   * Creates the warm up serverset manager.\n   */\n  public EarlybirdWarmUpManager provideWarmUpManager(\n      ZooKeeperProxy discoveryClient,\n      DynamicPartitionConfig dynamicPartitionConfig,\n      SearchStatsReceiver searchStatsReceiver,\n      Decider decider,\n      Clock clock,\n      int port,\n      String serverSetNamePrefix) {\n    return new EarlybirdWarmUpManager(\n        new EarlybirdServerSetManager(\n            searchStatsReceiver,\n            discoveryClient,\n            dynamicPartitionConfig.getCurrentPartitionConfig(),\n            port,\n            serverSetNamePrefix),\n        dynamicPartitionConfig.getCurrentPartitionConfig(),\n        searchIndexingMetricSet,\n        decider,\n        clock);\n  }\n\n  /**\n   * Returns a dark proxy that knows how to send dark traffic to the warm up earlybird serverset.\n   */\n  public EarlybirdDarkProxy provideEarlybirdDarkProxy(\n      SearchDecider searchDecider,\n      StatsReceiver finagleStatsReceiver,\n      EarlybirdServerSetManager earlybirdServerSetManager,\n      EarlybirdWarmUpManager earlybirdWarmUpManager,\n      String clusterName) {\n    return new EarlybirdDarkProxy(searchDecider,\n                                  finagleStatsReceiver.scope(\"dark_proxy\"),\n                                  earlybirdServerSetManager,\n                                  earlybirdWarmUpManager,\n                                  clusterName);\n  }\n\n\n  /**\n   * Returns the manager for all (non-Tensorflow) scoring models.\n   */\n  public ScoringModelsManager provideScoringModelsManager(\n      SearchStatsReceiver serverStats,\n      EarlybirdIndexConfig earlybirdIndexConfig) {\n    boolean modelsEnabled = EarlybirdConfig.getBool(\"scoring_models_enabled\", false);\n    if (!modelsEnabled) {\n      LOG.info(\"Scoring Models - Disabled in the config. Not loading any models.\");\n      serverStats.getCounter(\"scoring_models_disabled_in_config\").increment();\n      return ScoringModelsManager.NO_OP_MANAGER;\n    }\n\n    String hdfsNameNode = EarlybirdConfig.getString(\"scoring_models_namenode\");\n    String hdfsModelsPath = EarlybirdConfig.getString(\"scoring_models_basedir\");\n    try {\n      return ScoringModelsManager.create(\n          serverStats, hdfsNameNode, hdfsModelsPath, earlybirdIndexConfig.getSchema());\n    } catch (IOException e) {\n      LOG.error(\"Scoring Models - Error creating ScoringModelsManager\", e);\n      serverStats.getCounter(\"scoring_models_initialization_errors\").increment();\n      return ScoringModelsManager.NO_OP_MANAGER;\n    }\n  }\n\n  /**\n   * Provides the manager for all Tensorflow models.\n   */\n  public TensorflowModelsManager provideTensorflowModelsManager(\n      SearchStatsReceiver serverStats,\n      String statsPrefix,\n      Decider decider,\n      EarlybirdIndexConfig earlybirdIndexConfig) {\n\n    boolean modelsEnabled = EarlybirdProperty.TF_MODELS_ENABLED.get(false);\n\n    if (!modelsEnabled) {\n      LOG.info(\"Tensorflow Models - Disabled in the config. Not loading any models.\");\n      serverStats.getCounter(\"tf_models_disabled_in_config\").increment();\n      return TensorflowModelsManager.createNoOp(statsPrefix);\n    }\n\n    String modelsConfigPath =\n        Preconditions.checkNotNull(EarlybirdProperty.TF_MODELS_CONFIG_PATH.get());\n\n\n    int intraOpThreads = Preconditions.checkNotNull(EarlybirdProperty.TF_INTRA_OP_THREADS.get(0));\n    int interOpThreads = Preconditions.checkNotNull(EarlybirdProperty.TF_INTER_OP_THREADS.get(0));\n\n    TensorflowModelsManager.initTensorflowThreadPools(intraOpThreads, interOpThreads);\n\n    return TensorflowModelsManager.createUsingConfigFile(\n        FileUtils.getFileHandle(modelsConfigPath),\n        true,\n        statsPrefix,\n        () -> DeciderUtil.isAvailableForRandomRecipient(\n          decider, \"enable_tf_serve_models\"),\n        () -> decider.isAvailable(\"enable_tf_load_models\"),\n        earlybirdIndexConfig.getSchema());\n  }\n\n  public SearchStatsReceiver provideEarlybirdServerStatsReceiver() {\n    return sharedSearchStatsReceiver;\n  }\n\n  public StatsReceiver provideFinagleStatsReceiver() {\n    return sharedFinagleStatsReceiver;\n  }\n\n  public SearchIndexingMetricSet provideSearchIndexingMetricSet() {\n    return searchIndexingMetricSet;\n  }\n\n  public EarlybirdSearcherStats provideTweetsSearcherStats() {\n    return tweetsSearcherStats;\n  }\n\n  /**\n   * Provides the clock to be used by this earlybird.\n   */\n  public Clock provideClock() {\n    return Clock.SYSTEM_CLOCK;\n  }\n\n  /**\n   * Provides the config for the multi-segment term dictionary manager.\n   */\n  public MultiSegmentTermDictionaryManager.Config provideMultiSegmentTermDictionaryManagerConfig() {\n    return new MultiSegmentTermDictionaryManager.Config(\n        Lists.newArrayList(\n            EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName()));\n  }\n\n  /**\n   * Provides the manager for the term dictionary that spans all segments.\n   */\n  public MultiSegmentTermDictionaryManager provideMultiSegmentTermDictionaryManager(\n      MultiSegmentTermDictionaryManager.Config termDictionaryConfig,\n      SegmentManager segmentManager,\n      SearchStatsReceiver statsReceiver,\n      Decider decider,\n      EarlybirdCluster earlybirdCluster) {\n    return new MultiSegmentTermDictionaryManager(\n        termDictionaryConfig, segmentManager, statsReceiver, decider, earlybirdCluster);\n  }\n\n  /**\n   * Returns the partition manager to be used by the archive earlybirds.\n   */\n  public PartitionManager provideFullArchivePartitionManager(\n      ZooKeeperTryLockFactory zooKeeperTryLockFactory,\n      QueryCacheManager queryCacheManager,\n      SegmentManager segmentManager,\n      DynamicPartitionConfig dynamicPartitionConfig,\n      UserUpdatesStreamIndexer userUpdatesStreamIndexer,\n      UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer,\n      SearchStatsReceiver searchStatsReceiver,\n      ArchiveEarlybirdIndexConfig earlybirdIndexConfig,\n      ServerSetMember serverSetMember,\n      ScheduledExecutorServiceFactory executorServiceFactory,\n      ScheduledExecutorServiceFactory userUpdateIndexerExecutorFactory,\n      SearchIndexingMetricSet earlybirdSearchIndexingMetricSet,\n      Clock clock,\n      SegmentSyncConfig segmentSyncConfig,\n      CriticalExceptionHandler criticalExceptionHandler) throws IOException {\n\n    return new ArchiveSearchPartitionManager(\n        zooKeeperTryLockFactory,\n        queryCacheManager,\n        segmentManager,\n        dynamicPartitionConfig,\n        userUpdatesStreamIndexer,\n        userScrubGeoEventStreamIndexer,\n        searchStatsReceiver,\n        earlybirdIndexConfig,\n        serverSetMember,\n        executorServiceFactory,\n        userUpdateIndexerExecutorFactory,\n        earlybirdSearchIndexingMetricSet,\n        segmentSyncConfig,\n        clock,\n        criticalExceptionHandler);\n  }\n\n  /**\n   * Provides the SegmentSyncConfig instance to be used by earlybird.\n   */\n  public SegmentSyncConfig provideSegmentSyncConfig(EarlybirdCluster cluster) {\n    String scrubGen = null;\n    if (cluster == EarlybirdCluster.FULL_ARCHIVE) {\n      scrubGen = EarlybirdProperty.EARLYBIRD_SCRUB_GEN.get();\n      LOG.info(\"The scrubGen provided from Aurora is: {}\", scrubGen);\n      Preconditions.checkState(Strings.isNotEmpty(scrubGen));\n    }\n    return new SegmentSyncConfig(Optional.ofNullable(scrubGen));\n  }\n\n  protected void storeEarlybirdStartupProducts(\n      TweetCreateHandler tweetCreateHandler,\n      PartitionWriter partitionWriter,\n      EarlybirdIndexFlusher earlybirdIndexFlusher\n  ) {\n    // TestWireModule wants to store these for further use.\n  }\n\n  /**\n   * What directory are we going to load segments from on startup.\n   *\n   * When you're running loadtests or stagingN instances and they don't have a recent index\n   * flushed, it can take hours to generate a new index with a fresh startup. This slows\n   * down development. If the read_index_from_prod_location flag is set to true, we will read\n   * the index from the location where prod instances are flushing their index to.\n   * Unset it if you want to generate your own index.\n   *\n   * @return a string with the directory.\n   */\n  public String getIndexLoadingDirectory() {\n    boolean readIndexFromProdLocation = EarlybirdProperty.READ_INDEX_FROM_PROD_LOCATION.get(false);\n    String environment = EarlybirdProperty.ENV.get(\"no_env_specified\"); // default value for tests.\n    String readIndexDir = EarlybirdProperty.HDFS_INDEX_SYNC_DIR.get();\n\n    if (readIndexFromProdLocation) {\n      LOG.info(\"Will attempt to read index from prod locations\");\n      LOG.info(\"Index directory provided: {}\", readIndexDir);\n      // Replacing the path is a bit hacky, but it works ok.\n      readIndexDir = readIndexDir.replace(\"/\" + environment + \"/\", \"/prod/\");\n      LOG.info(\"Will instead use index directory: {}\", readIndexDir);\n    }\n\n    return readIndexDir;\n  }\n\n  /**\n   * Indexer for audio space events.\n   */\n  public AudioSpaceEventsStreamIndexer provideAudioSpaceEventsStreamIndexer(\n      AudioSpaceTable audioSpaceTable,\n      Clock clock) {\n    try {\n      return new AudioSpaceEventsStreamIndexer(\n          FinagleKafkaClientUtils.newKafkaConsumerForAssigning(\n              \"\",\n              new ThriftDeserializer<>(AudioSpaceBaseEvent.class),\n              \"\",\n              20\n          ), audioSpaceTable, clock);\n    } catch (MissingKafkaTopicException ex) {\n      LOG.error(\"Missing kafka stream\", ex);\n      return null;\n    }\n  }\n\n  /**\n   * Returns a class to start the Earlybird. See {@link EarlybirdStartup}.\n   */\n  public EarlybirdStartup provideEarlybirdStartup(\n      PartitionManager partitionManager,\n      UserUpdatesStreamIndexer userUpdatesStreamIndexer,\n      UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer,\n      AudioSpaceEventsStreamIndexer audioSpaceEventsStreamIndexer,\n      DynamicPartitionConfig dynamicPartitionConfig,\n      CriticalExceptionHandler criticalExceptionHandler,\n      SegmentManager segmentManager,\n      MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager,\n      QueryCacheManager queryCacheManager,\n      ZooKeeperTryLockFactory zooKeeperTryLockFactory,\n      ServerSetMember serverSetMember,\n      Clock clock,\n      SegmentSyncConfig segmentSyncConfig,\n      EarlybirdSegmentFactory earlybirdSegmentFactory,\n      EarlybirdCluster cluster,\n      SearchDecider decider) throws IOException {\n    if (cluster == EarlybirdCluster.FULL_ARCHIVE) {\n      return new PartitionManagerStartup(clock, partitionManager);\n    }\n\n    // Check that the earlybird name is what we're expecting so we can build the kafka topics.\n    String earlybirdName = EarlybirdProperty.EARLYBIRD_NAME.get();\n    Preconditions.checkArgument(\"earlybird-realtime\".equals(earlybirdName)\n        || \"earlybird-protected\".equals(earlybirdName)\n        || \"earlybird-realtime-exp0\".equals(earlybirdName)\n        || \"earlybird-realtime_cg\".equals(earlybirdName));\n\n    StartupUserEventIndexer startupUserEventIndexer = new StartupUserEventIndexer(\n        provideSearchIndexingMetricSet(),\n        userUpdatesStreamIndexer,\n        userScrubGeoEventStreamIndexer,\n        segmentManager,\n        clock);\n\n    // Coordinate leaving the serverset to flush segments to HDFS.\n    CoordinatedEarlybirdAction actionCoordinator = new CoordinatedEarlybirdAction(\n        zooKeeperTryLockFactory,\n        \"segment_flusher\",\n        dynamicPartitionConfig,\n        serverSetMember,\n        criticalExceptionHandler,\n        segmentSyncConfig);\n    actionCoordinator.setShouldSynchronize(true);\n\n    FileSystem hdfsFileSystem = HdfsUtil.getHdfsFileSystem();\n    EarlybirdIndexFlusher earlybirdIndexFlusher = new EarlybirdIndexFlusher(\n        actionCoordinator,\n        hdfsFileSystem,\n        EarlybirdProperty.HDFS_INDEX_SYNC_DIR.get(),\n        segmentManager,\n        dynamicPartitionConfig.getCurrentPartitionConfig(),\n        clock,\n        new TimeLimitedHadoopExistsCall(hdfsFileSystem),\n        provideOptimizationAndFlushingCoordinationLock());\n\n    String baseTopicName = \"search_ingester_%s_events_%s_%s\";\n\n    String earlybirdType;\n\n    if (\"earlybird-protected\".equals(earlybirdName)) {\n      earlybirdType = \"protected\";\n    } else if (\"earlybird-realtime_cg\".equals(earlybirdName)) {\n      earlybirdType = \"realtime_cg\";\n    } else {\n      earlybirdType = \"realtime\";\n    }\n\n    String tweetTopicName = String.format(\n        baseTopicName,\n        \"indexing\",\n        earlybirdType,\n        EarlybirdProperty.KAFKA_ENV.get());\n\n    String updateTopicName = String.format(\n        baseTopicName,\n        \"update\",\n        earlybirdType,\n        EarlybirdProperty.KAFKA_ENV.get());\n\n    LOG.info(\"Tweet topic: {}\", tweetTopicName);\n    LOG.info(\"Update topic: {}\", updateTopicName);\n\n    TopicPartition tweetTopic = new TopicPartition(\n        tweetTopicName,\n        dynamicPartitionConfig.getCurrentPartitionConfig().getIndexingHashPartitionID());\n    TopicPartition updateTopic = new TopicPartition(\n        updateTopicName,\n        dynamicPartitionConfig.getCurrentPartitionConfig().getIndexingHashPartitionID());\n\n    EarlybirdKafkaConsumersFactory earlybirdKafkaConsumersFactory =\n        provideEarlybirdKafkaConsumersFactory();\n    FreshStartupHandler freshStartupHandler = new FreshStartupHandler(\n        clock,\n        earlybirdKafkaConsumersFactory,\n        tweetTopic,\n        updateTopic,\n        segmentManager,\n        EarlybirdConfig.getMaxSegmentSize(),\n        EarlybirdConfig.getLateTweetBuffer(),\n        criticalExceptionHandler\n    );\n\n    TweetUpdateHandler updateHandler = new TweetUpdateHandler(segmentManager);\n\n    CoordinatedEarlybirdAction postOptimizationRebuilds = new CoordinatedEarlybirdAction(\n            zooKeeperTryLockFactory,\n            \"post_optimization_rebuilds\",\n            dynamicPartitionConfig,\n            serverSetMember,\n            criticalExceptionHandler,\n            segmentSyncConfig\n    );\n    postOptimizationRebuilds.setShouldSynchronize(true);\n    CoordinatedEarlybirdAction gcAction = new CoordinatedEarlybirdAction(\n            zooKeeperTryLockFactory,\n            \"gc_before_optimization\",\n            dynamicPartitionConfig,\n            serverSetMember,\n            criticalExceptionHandler,\n            segmentSyncConfig\n    );\n    gcAction.setShouldSynchronize(true);\n\n    TweetCreateHandler createHandler = new TweetCreateHandler(\n        segmentManager,\n        provideSearchIndexingMetricSet(),\n        criticalExceptionHandler,\n        multiSegmentTermDictionaryManager,\n        queryCacheManager,\n        postOptimizationRebuilds,\n        gcAction,\n        EarlybirdConfig.getLateTweetBuffer(),\n        EarlybirdConfig.getMaxSegmentSize(),\n        provideKafkaIndexCaughtUpMonitor(),\n        provideOptimizationAndFlushingCoordinationLock());\n\n    PartitionWriter partitionWriter = new PartitionWriter(\n        createHandler,\n        updateHandler,\n        criticalExceptionHandler,\n        PenguinVersion.versionFromByteValue(EarlybirdConfig.getPenguinVersionByte()),\n        clock);\n\n    KafkaConsumer<Long, ThriftVersionedEvents> rawKafkaConsumer =\n        earlybirdKafkaConsumersFactory.createKafkaConsumer(\n            \"earlybird_tweet_kafka_consumer\");\n\n    EarlybirdKafkaConsumer earlybirdKafkaConsumer = provideKafkaConsumer(\n        criticalExceptionHandler,\n        rawKafkaConsumer,\n        tweetTopic,\n        updateTopic,\n        partitionWriter,\n        earlybirdIndexFlusher);\n\n    EarlybirdIndexLoader earlybirdIndexLoader = new EarlybirdIndexLoader(\n        hdfsFileSystem,\n        getIndexLoadingDirectory(), // See SEARCH-32839\n        EarlybirdProperty.ENV.get(\"default_env_value\"),\n        dynamicPartitionConfig.getCurrentPartitionConfig(),\n        earlybirdSegmentFactory,\n        segmentSyncConfig,\n        clock);\n\n    this.storeEarlybirdStartupProducts(\n        createHandler,\n        partitionWriter,\n        earlybirdIndexFlusher\n    );\n\n    return new KafkaStartup(\n        segmentManager,\n        earlybirdKafkaConsumer,\n        startupUserEventIndexer,\n        userUpdatesStreamIndexer,\n        userScrubGeoEventStreamIndexer,\n        audioSpaceEventsStreamIndexer,\n        queryCacheManager,\n        earlybirdIndexLoader,\n        freshStartupHandler,\n        provideSearchIndexingMetricSet(),\n        multiSegmentTermDictionaryManager,\n        criticalExceptionHandler,\n        decider\n    );\n  }\n\n  public QualityFactor provideQualityFactor(\n      Decider decider,\n      SearchStatsReceiver searchStatsReceiver\n  ) {\n    return new EarlybirdCPUQualityFactor(decider,\n        ManagementFactory.getPlatformMXBean(OperatingSystemMXBean.class),\n        searchStatsReceiver);\n  }\n\n  /**\n   * Returns a new UserUpdatesKafkaConsumer to read user updates.\n   */\n  public UserUpdatesStreamIndexer provideUserUpdatesKafkaConsumer(\n      SegmentManager segmentManager) {\n    try {\n      return new UserUpdatesStreamIndexer(\n          UserUpdatesStreamIndexer.provideKafkaConsumer(),\n          EarlybirdProperty.USER_UPDATES_KAFKA_TOPIC.get(),\n          provideSearchIndexingMetricSet(),\n          segmentManager);\n    } catch (MissingKafkaTopicException ex) {\n      // Yes, it will crash the server. We've never seen this topic missing, but\n      // we've seen some others, so we had to build this functionality in the\n      // constructor. If one day this one goes missing, we'll have to figure out\n      // how to handle it. For now, we crash.\n      throw new RuntimeException(ex);\n    }\n  }\n\n  /**\n   * Returns a new UserScrubGeosKafkaConsumer to read geo scrubbing events.\n   */\n  public UserScrubGeoEventStreamIndexer provideUserScrubGeoEventKafkaConsumer(\n      SegmentManager segmentManager) {\n    try {\n      return new UserScrubGeoEventStreamIndexer(\n          UserScrubGeoEventStreamIndexer.provideKafkaConsumer(),\n          EarlybirdProperty.USER_SCRUB_GEO_KAFKA_TOPIC.get(),\n          provideSearchIndexingMetricSet(),\n          segmentManager);\n    } catch (MissingKafkaTopicException ex) {\n      /**\n       * See {@link #provideUserUpdatesKafkaConsumer}\n       */\n      throw new RuntimeException(ex);\n    }\n  }\n\n  /**\n   * Returns a new ProductionEarlybirdKafkaConsumer to read ThriftVersionedEvents.\n   */\n  public EarlybirdKafkaConsumersFactory provideEarlybirdKafkaConsumersFactory() {\n    return new ProductionEarlybirdKafkaConsumersFactory(\n        EarlybirdProperty.KAFKA_PATH.get(),\n        MAX_POLL_RECORDS\n    );\n  }\n\n  /**\n   * Returns a class to read Tweets in the Earlybird. See {@link EarlybirdKafkaConsumer}.\n   */\n  public EarlybirdKafkaConsumer provideKafkaConsumer(\n      CriticalExceptionHandler criticalExceptionHandler,\n      KafkaConsumer<Long, ThriftVersionedEvents> rawKafkaConsumer,\n      TopicPartition tweetTopic,\n      TopicPartition updateTopic,\n      PartitionWriter partitionWriter,\n      EarlybirdIndexFlusher earlybirdIndexFlusher\n  ) {\n    return new EarlybirdKafkaConsumer(\n        rawKafkaConsumer,\n        provideSearchIndexingMetricSet(),\n        criticalExceptionHandler,\n        partitionWriter,\n        tweetTopic,\n        updateTopic,\n        earlybirdIndexFlusher,\n        provideKafkaIndexCaughtUpMonitor());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/factory/PartitionConfigUtil.java",
    "content": "package com.twitter.search.earlybird.factory;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.config.TierConfig;\nimport com.twitter.search.earlybird.config.TierInfo;\nimport com.twitter.search.earlybird.partition.PartitionConfig;\n\npublic final class PartitionConfigUtil {\n  private static final Logger LOG = LoggerFactory.getLogger(PartitionConfigUtil.class);\n\n  private PartitionConfigUtil() {\n  }\n\n  /**\n   * Initiate PartitionConfig for earlybirds running on Aurora\n   */\n  public static PartitionConfig initPartitionConfigForAurora(int numOfInstances) {\n    String tier = EarlybirdProperty.EARLYBIRD_TIER.get();\n    int partitionId = EarlybirdProperty.PARTITION_ID.get();\n    int replicaId = EarlybirdProperty.REPLICA_ID.get();\n    if (tier.equals(PartitionConfig.DEFAULT_TIER_NAME)) {\n      // realtime or protected earlybird\n      return new PartitionConfig(\n          partitionId,\n          EarlybirdProperty.SERVING_TIMESLICES.get(),\n          replicaId,\n          numOfInstances,\n          EarlybirdProperty.NUM_PARTITIONS.get());\n    } else {\n      // archive earlybird\n      TierInfo tierInfo = TierConfig.getTierInfo(tier);\n      return new PartitionConfig(tier, tierInfo.getDataStartDate(), tierInfo.getDataEndDate(),\n          partitionId, tierInfo.getMaxTimeslices(), replicaId, numOfInstances,\n          tierInfo.getNumPartitions());\n    }\n  }\n\n  /**\n   * Tries to create a new PartitionConfig instance based on the Aurora flags\n   */\n  public static PartitionConfig initPartitionConfig() {\n    return initPartitionConfigForAurora(EarlybirdProperty.NUM_INSTANCES.get());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/factory/ProductionEarlybirdKafkaConsumersFactory.java",
    "content": "package com.twitter.search.earlybird.factory;\n\nimport org.apache.kafka.clients.consumer.KafkaConsumer;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.util.io.kafka.CompactThriftDeserializer;\nimport com.twitter.search.common.util.io.kafka.FinagleKafkaClientUtils;\n\n/**\n * Responsible for creating kafka consumers.\n */\npublic class ProductionEarlybirdKafkaConsumersFactory implements EarlybirdKafkaConsumersFactory {\n  private final String kafkaPath;\n  private final int defaultMaxPollRecords;\n\n  ProductionEarlybirdKafkaConsumersFactory(String kafkaPath, int defaultMaxPollRecords) {\n    this.kafkaPath = kafkaPath;\n    this.defaultMaxPollRecords = defaultMaxPollRecords;\n  }\n\n  /**\n   * Create a kafka consumer with set maximum of records to be polled.\n   */\n  @Override\n  public KafkaConsumer<Long, ThriftVersionedEvents> createKafkaConsumer(\n      String clientID, int maxPollRecords) {\n    return FinagleKafkaClientUtils.newKafkaConsumerForAssigning(\n        kafkaPath,\n        new CompactThriftDeserializer<>(ThriftVersionedEvents.class),\n        clientID,\n        maxPollRecords);\n  }\n\n  /**\n   * Create a kafka consumer with default records to be polled.\n   */\n  @Override\n  public KafkaConsumer<Long, ThriftVersionedEvents> createKafkaConsumer(String clientID) {\n    return createKafkaConsumer(clientID, defaultMaxPollRecords);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/factory/QueryCacheUpdaterScheduledExecutorService.java",
    "content": "package com.twitter.search.earlybird.factory;\n\nimport java.util.concurrent.Callable;\nimport java.util.concurrent.ScheduledExecutorService;\nimport java.util.concurrent.ScheduledFuture;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport com.twitter.common.util.concurrent.ForwardingExecutorService;\n\n/**\n * This delegate type is intended for QueryCacheUpdater because it uses multiple threads to\n * create query cache during startup and then switch later to use single thread to update the\n * cache.\n */\npublic abstract class QueryCacheUpdaterScheduledExecutorService<T extends ScheduledExecutorService>\n  extends ForwardingExecutorService<T> implements ScheduledExecutorService {\n  public QueryCacheUpdaterScheduledExecutorService(T executor) {\n    super(executor);\n  }\n\n  /**\n   * Sets the number of worker threads in this executor service to an appropriate value after the\n   * earlybird startup has finished. While earlybird is starting up, we might want this executor\n   * service to have more threads, in order to parallelize more some start up tasks. But once\n   * earlybird is up, it might make sense to lower the number of worker threads.\n   */\n  public abstract void setWorkerPoolSizeAfterStartup();\n\n  @Override\n  public ScheduledFuture<?> schedule(Runnable command, long delay, TimeUnit unit) {\n    return delegate.schedule(command, delay, unit);\n  }\n\n  @Override\n  public ScheduledFuture<?> scheduleAtFixedRate(\n      Runnable command, long initialDelay, long period, TimeUnit unit) {\n    return delegate.scheduleAtFixedRate(command, initialDelay, period, unit);\n  }\n\n  @Override\n  public ScheduledFuture<?> scheduleWithFixedDelay(\n      Runnable command, long initialDelay, long delay, TimeUnit unit) {\n    return delegate.scheduleWithFixedDelay(command, initialDelay, delay, unit);\n  }\n\n  @Override\n  public <V> ScheduledFuture<V> schedule(Callable<V> callable, long delay, TimeUnit unit) {\n    return delegate.schedule(callable, delay, unit);\n  }\n\n  @VisibleForTesting\n  public T getDelegate() {\n    return delegate;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/AbstractInMemoryTimeMapper.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport com.twitter.search.core.earlybird.index.TimeMapper;\nimport com.twitter.search.core.earlybird.index.inverted.IntBlockPool;\nimport com.twitter.search.core.earlybird.index.util.SearchSortUtils;\nimport com.twitter.search.earlybird.search.queries.SinceUntilFilter;\n\npublic abstract class AbstractInMemoryTimeMapper implements TimeMapper {\n  // Reverse map: timestamp to first doc ID seen with that timestamp.\n  // This is two arrays: the timestamps (sorted), and the doc ids.\n  protected final IntBlockPool reverseMapTimes;\n  protected final IntBlockPool reverseMapIds;\n  protected volatile int reverseMapLastIndex;\n\n  public AbstractInMemoryTimeMapper() {\n    this.reverseMapTimes = new IntBlockPool(ILLEGAL_TIME, \"time_mapper_times\");\n    this.reverseMapIds = new IntBlockPool(ILLEGAL_TIME, \"time_mapper_ids\");\n    this.reverseMapLastIndex = -1;\n  }\n\n  protected AbstractInMemoryTimeMapper(int reverseMapLastIndex,\n                                       IntBlockPool reverseMapTimes,\n                                       IntBlockPool reverseMapIds) {\n    this.reverseMapTimes = reverseMapTimes;\n    this.reverseMapIds = reverseMapIds;\n    this.reverseMapLastIndex = reverseMapLastIndex;\n  }\n\n  @Override\n  public final int getLastTime() {\n    return reverseMapLastIndex == -1 ? ILLEGAL_TIME : reverseMapTimes.get(reverseMapLastIndex);\n  }\n\n  @Override\n  public final int getFirstTime() {\n    return reverseMapLastIndex == -1 ? ILLEGAL_TIME : reverseMapTimes.get(0);\n  }\n\n  @Override\n  public final int findFirstDocId(int timeSeconds, int smallestDocID) {\n    if (timeSeconds == SinceUntilFilter.NO_FILTER || reverseMapLastIndex == -1) {\n      return smallestDocID;\n    }\n\n    final int index = SearchSortUtils.binarySearch(\n        new IntArrayComparator(), 0, reverseMapLastIndex, timeSeconds, false);\n\n    if (index == reverseMapLastIndex && reverseMapTimes.get(index) < timeSeconds) {\n      // Special case for out of bounds time.\n      return smallestDocID;\n    }\n\n    return reverseMapIds.get(index);\n  }\n\n  protected abstract void setTime(int docID, int timeSeconds);\n\n  protected void doAddMapping(int docID, int timeSeconds) {\n    setTime(docID, timeSeconds);\n    int lastTime = getLastTime();\n    if (timeSeconds > lastTime) {\n      // Found a timestamp newer than any timestamp we've seen before.\n      // Add a reverse mapping to this tweet (the first seen with this timestamp).\n      //\n      // When indexing out of order tweets, we could have gaps in the timestamps recorded in\n      // reverseMapTimes. For example, if we get 3 tweets with timestamp T0, T0 + 5, T0 + 3, then we\n      // will only record T0 and T0 + 5 in reverseMapTimes. However, this should not be an issue,\n      // because reverseMapTimes is only used by findFirstDocId(), and it's OK for that method to\n      // return a smaller doc ID than strictly necessary (in this case, findFirstDocId(T0 + 3) will\n      // return the doc ID of the second tweet, instead of returning the doc ID of the third tweet).\n      reverseMapTimes.add(timeSeconds);\n      reverseMapIds.add(docID);\n      reverseMapLastIndex++;\n    }\n  }\n\n  private class IntArrayComparator implements SearchSortUtils.Comparator<Integer> {\n    @Override\n    public int compare(int index, Integer value) {\n      return Integer.compare(reverseMapTimes.get(index), value);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/DocValuesBasedTimeMapper.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.search.DocIdSetIterator;\n\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.common.util.analysis.IntTermAttributeImpl;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.TimeMapper;\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideFieldIndex;\n\n/**\n * A few caveats when using this class:\n *   - This class only supports in-order createdAt!\n *   - Before actually using this class, one must call prepareToRead() with a Lucene AtomicReader\n *   - prepareToRead() will load docID to createdAt mapping into memory, if not already done.\n */\npublic class DocValuesBasedTimeMapper implements TimeMapper {\n  private LeafReader reader;\n  private ColumnStrideFieldIndex docValues;\n\n  protected int minTimestamp = ILLEGAL_TIME;\n  protected int maxTimestamp = ILLEGAL_TIME;\n\n  /**\n   * When indexing finishes, this method should be called with a index reader that\n   * can see all documents.\n   * @param leafReader Lucene index reader used to access \"TweetID\" to \"createdAt\" mapping.\n   */\n  public void initializeWithLuceneReader(LeafReader leafReader, ColumnStrideFieldIndex csf)\n      throws IOException {\n    reader = Preconditions.checkNotNull(leafReader);\n    docValues = Preconditions.checkNotNull(csf);\n\n    // Find the min and max timestamps.\n    // See SEARCH-5534\n    // In the archive, tweets are always sorted in descending order by tweet ID, but\n    // that does not mean that the documents are necessarily sorted by time. We've observed tweet ID\n    // generation be decoupled from timestamp creation (i.e. a larger tweet ID having a smaller\n    // created_at time).\n    minTimestamp = Integer.MAX_VALUE;\n    maxTimestamp = Integer.MIN_VALUE;\n\n    NumericDocValues onDiskDocValues = reader.getNumericDocValues(\n        EarlybirdFieldConstants.EarlybirdFieldConstant.CREATED_AT_CSF_FIELD.getFieldName());\n    for (int i = 0; i < reader.maxDoc(); ++i) {\n      Preconditions.checkArgument(onDiskDocValues.advanceExact(i));\n      int timestamp = (int) onDiskDocValues.longValue();\n      docValues.setValue(i, timestamp);\n\n      if (timestamp < minTimestamp) {\n        minTimestamp = timestamp;\n      }\n      if (timestamp > maxTimestamp) {\n        maxTimestamp = timestamp;\n      }\n    }\n  }\n\n  @Override\n  public int getLastTime() {\n    return maxTimestamp;\n  }\n\n  @Override\n  public int getFirstTime() {\n    return minTimestamp;\n  }\n\n  @Override\n  public int getTime(int docID) {\n    if (docID < 0 || docID > reader.maxDoc()) {\n      return ILLEGAL_TIME;\n    }\n    return (int) docValues.get(docID);\n  }\n\n  @Override\n  public int findFirstDocId(int timeSeconds, int smallestDocID) throws IOException {\n    // In the full archive, the smallest doc id corresponds to largest timestamp.\n    if (timeSeconds > maxTimestamp) {\n      return smallestDocID;\n    }\n    if (timeSeconds < minTimestamp) {\n      return reader.maxDoc() - 1;\n    }\n\n    int docId = DocValuesHelper.getLargestDocIdWithCeilOfValue(\n        reader,\n        EarlybirdFieldConstants.EarlybirdFieldConstant.CREATED_AT_FIELD.getFieldName(),\n        IntTermAttributeImpl.copyIntoNewBytesRef(timeSeconds));\n    if (docId == DocIdSetIterator.NO_MORE_DOCS) {\n      return ILLEGAL_TIME;\n    }\n\n    return docId;\n  }\n\n  @Override\n  public TimeMapper optimize(DocIDToTweetIDMapper originalTweetIdMapper,\n                             DocIDToTweetIDMapper optimizedTweetIdMapper) {\n    // DocValuesBasedTimerMapper instances are not flushed or loaded,\n    // so their optimization is a no-op.\n    return this;\n  }\n\n  @Override\n  public Flushable.Handler<DocValuesBasedTimeMapper> getFlushHandler() {\n    // EarlybirdIndexSegmentData will still try to flush the DocValuesBasedTimeMapper for the\n    // respective segment, so we need to pass in a DocValuesBasedTimeMapper instance to this\n    // flusher: otherwise, Flushable.Handler.flush() will throw a NullPointerException.\n    return new FlushHandler(new DocValuesBasedTimeMapper());\n  }\n\n  // Full archive earlybirds don't actually flush or load the DocValuesBasedTimeMapper. This is\n  // why doFlush() is a no-op, and doLoad() returns a new DocValuesBasedTimeMapper instance\n  // (initializeWithLuceneReader() will be called at load time to initialize this new\n  // DocValuesBasedTimeMapper instance).\n  public static class FlushHandler extends Flushable.Handler<DocValuesBasedTimeMapper> {\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(DocValuesBasedTimeMapper objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) {\n    }\n\n    @Override\n    protected DocValuesBasedTimeMapper doLoad(FlushInfo flushInfo, DataDeserializer in) {\n      return new DocValuesBasedTimeMapper();\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/DocValuesBasedTweetIDMapper.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.search.DocIdSetIterator;\n\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.common.util.analysis.SortableLongTermAttributeImpl;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideFieldIndex;\n\n/**\n * A few caveats when using this class:\n *   - Before actually using this class, one must call prepareToRead() with a Lucene AtomicReader\n *   - prepareToRead() will load docID to tweetID mapping into memory, if not already done.\n */\npublic class DocValuesBasedTweetIDMapper extends TweetIDMapper implements Flushable {\n  private LeafReader reader;\n  private ColumnStrideFieldIndex docValues;\n\n  /**\n   * When indexing finishes, this method should be called with a index reader that\n   * can see all documents.\n   * @param leafReader Lucene index reader used to access TweetID to internal ID mapping\n   */\n  public void initializeWithLuceneReader(LeafReader leafReader, ColumnStrideFieldIndex csf)\n      throws IOException {\n    reader = Preconditions.checkNotNull(leafReader);\n    docValues = Preconditions.checkNotNull(csf);\n\n    NumericDocValues onDiskDocValues = reader.getNumericDocValues(\n        EarlybirdFieldConstants.EarlybirdFieldConstant.ID_CSF_FIELD.getFieldName());\n    for (int i = 0; i < reader.maxDoc(); ++i) {\n      Preconditions.checkArgument(onDiskDocValues.advanceExact(i));\n      docValues.setValue(i, onDiskDocValues.longValue());\n    }\n\n    // In the archive, tweets are always sorted in descending order of tweet ID.\n    setMinTweetID(docValues.get(reader.maxDoc() - 1));\n    setMaxTweetID(docValues.get(0));\n    setMinDocID(0);\n    setMaxDocID(reader.maxDoc() - 1);\n    setNumDocs(reader.maxDoc());\n  }\n\n  @Override\n  public int getDocID(long tweetID) throws IOException {\n    int docId = DocValuesHelper.getFirstDocIdWithValue(\n        reader,\n        EarlybirdFieldConstants.EarlybirdFieldConstant.ID_FIELD.getFieldName(),\n        SortableLongTermAttributeImpl.copyIntoNewBytesRef(tweetID));\n    if (docId == DocIdSetIterator.NO_MORE_DOCS) {\n      return ID_NOT_FOUND;\n    }\n    return docId;\n  }\n\n  @Override\n  protected int getNextDocIDInternal(int docID) {\n    // The doc IDs are consecutive and TweetIDMapper already checked the boundary conditions.\n    return docID + 1;\n  }\n\n  @Override\n  protected int getPreviousDocIDInternal(int docID) {\n    // The doc IDs are consecutive and TweetIDMapper already checked the boundary conditions.\n    return docID - 1;\n  }\n\n  @Override\n  public long getTweetID(int internalID) {\n    if (internalID < 0 || internalID > getMaxDocID()) {\n      return ID_NOT_FOUND;\n    }\n    return docValues.get(internalID);\n  }\n\n  @Override\n  protected int addMappingInternal(long tweetID) {\n    throw new UnsupportedOperationException(\n        \"ArchiveTweetIDMapper should be written through Lucene instead of TweetIDMappingWriter\");\n  }\n\n  @Override\n  protected final int findDocIDBoundInternal(long tweetID,\n                                             boolean findMaxDocID) throws IOException {\n    // TermsEnum has a seekCeil() method, but doesn't have a seekFloor() method, so the best we can\n    // do here is ignore findLow and always return the ceiling if the tweet ID cannot be found.\n    // However, in practice, we do a seekExact() in both cases: see the inner classes in\n    // com.twitter.search.core.earlybird.index.inverted.RealtimeIndexTerms.\n    int docId = DocValuesHelper.getLargestDocIdWithCeilOfValue(\n        reader,\n        EarlybirdFieldConstants.EarlybirdFieldConstant.ID_FIELD.getFieldName(),\n        SortableLongTermAttributeImpl.copyIntoNewBytesRef(tweetID));\n    if (docId == DocIdSetIterator.NO_MORE_DOCS) {\n      return ID_NOT_FOUND;\n    }\n\n    // The docId is the upper bound of the search, so if we want the lower bound,\n    // because doc IDs are dense, we subtract one.\n    return findMaxDocID ? docId : docId - 1;\n  }\n\n  @Override\n  public DocIDToTweetIDMapper optimize() {\n    // DocValuesBasedTweetIDMapper instances are not flushed or loaded,\n    // so their optimization is a no-op.\n    return this;\n  }\n\n  @Override\n  public Flushable.Handler<DocValuesBasedTweetIDMapper> getFlushHandler() {\n    // EarlybirdIndexSegmentData will still try to flush the DocValuesBasedTweetIDMapper\n    // for the respective segment, so we need to pass in a DocValuesBasedTweetIDMapper instance to\n    // this flusher: otherwise, Flushable.Handler.flush() will throw a NullPointerException.\n    return new FlushHandler(new DocValuesBasedTweetIDMapper());\n  }\n\n  // Full archive earlybirds don't actually flush or load the DocValuesBasedTweetIDMapper. This is\n  // why doFlush() is a no-op, and doLoad() returns a new DocValuesBasedTweetIDMapper instance\n  // (initializeWithLuceneReader() will be called at load time to initialize this new\n  // DocValuesBasedTweetIDMapper instance).\n  public static class FlushHandler extends Flushable.Handler<DocValuesBasedTweetIDMapper> {\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(DocValuesBasedTweetIDMapper objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) {\n    }\n\n    @Override\n    protected DocValuesBasedTweetIDMapper doLoad(FlushInfo flushInfo, DataDeserializer in) {\n      return new DocValuesBasedTweetIDMapper();\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/DocValuesHelper.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.util.BytesRef;\n\npublic final class DocValuesHelper {\n  private DocValuesHelper() {\n  }\n\n  /**\n   * Reverse lookup. Given a value, returns the first doc ID with this value. This requires a field\n   * that indexes the values.\n   *\n   * @param reader The reader to use to look up field values.\n   * @param value The value to lookup.\n   * @param indexField The field containing an index of the values.\n   */\n  public static int getFirstDocIdWithValue(\n      LeafReader reader, String indexField, BytesRef value) throws IOException {\n    TermsEnum termsEnum = getTermsEnum(reader, indexField);\n    if (termsEnum == null || !termsEnum.seekExact(value)) {\n      return DocIdSetIterator.NO_MORE_DOCS;\n    }\n\n    DocIdSetIterator docsIterator = termsEnum.postings(null);\n    return docsIterator.nextDoc();\n  }\n\n  /**\n   * Reverse lookup. Same as getFirstDocIdWithValue(), but if no document with the given value\n   * exists, the next bigger value is used for looking up the first doc ID.\n   *\n   * If there are multiple documents that match the value, all documents will be scanned, and the\n   * largest doc ID that matches will be returned.\n   *\n   * @param reader The reader to use to look up field values.\n   * @param value The value to lookup.\n   * @param indexField The field containing an index of the values.\n   */\n  public static int getLargestDocIdWithCeilOfValue(\n      LeafReader reader, String indexField, BytesRef value) throws IOException {\n    TermsEnum termsEnum = getTermsEnum(reader, indexField);\n    if (termsEnum == null) {\n      return DocIdSetIterator.NO_MORE_DOCS;\n    }\n    if (termsEnum.seekCeil(value) == TermsEnum.SeekStatus.END) {\n      return DocIdSetIterator.NO_MORE_DOCS;\n    }\n\n    DocIdSetIterator docsIterator = termsEnum.postings(null);\n    int docId = docsIterator.nextDoc();\n    while (docsIterator.nextDoc() != DocIdSetIterator.NO_MORE_DOCS) {\n      docId = docsIterator.docID();\n    }\n    return docId;\n  }\n\n  private static TermsEnum getTermsEnum(LeafReader reader, String indexField) throws IOException {\n    Terms terms = reader.terms(indexField);\n    if (terms == null) {\n      return null;\n    }\n    return terms.iterator();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/EarlybirdSegment.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.Closeable;\nimport java.io.File;\nimport java.io.IOException;\nimport java.time.Instant;\nimport java.time.ZoneOffset;\nimport java.time.ZonedDateTime;\nimport java.time.format.DateTimeFormatter;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Objects;\nimport java.util.concurrent.atomic.AtomicReference;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.HashBasedTable;\nimport com.google.common.collect.Table;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.apache.commons.io.FileUtils;\nimport org.apache.lucene.document.Document;\nimport org.apache.lucene.index.DirectoryReader;\nimport org.apache.lucene.index.IndexWriterConfig;\nimport org.apache.lucene.index.IndexableField;\nimport org.apache.lucene.store.Directory;\nimport org.apache.lucene.store.FSDirectory;\nimport org.apache.lucene.store.IOContext;\nimport org.apache.lucene.store.IndexOutput;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.schema.base.FeatureConfiguration;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.ThriftDocumentUtil;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdEncodedFeatures;\nimport com.twitter.search.common.schema.earlybird.EarlybirdEncodedFeaturesUtil;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.schema.thriftjava.ThriftDocument;\nimport com.twitter.search.common.schema.thriftjava.ThriftField;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEventType;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentData;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentWriter;\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideFieldIndex;\nimport com.twitter.search.core.earlybird.index.column.DocValuesUpdate;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdIndexExtensionsFactory;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.document.TweetDocument;\nimport com.twitter.search.earlybird.exception.FlushVersionMismatchException;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\nimport com.twitter.search.earlybird.partition.SegmentIndexStats;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.snowflake.id.SnowflakeId;\n\npublic class EarlybirdSegment {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdSegment.class);\n  private static final Logger UPDATES_ERRORS_LOG =\n      LoggerFactory.getLogger(EarlybirdSegment.class.getName() + \".UpdatesErrors\");\n  private static final String SUCCESS_FILE = \"EARLYBIRD_SUCCESS\";\n  private static final DateTimeFormatter HOURLY_COUNT_DATE_TIME_FORMATTER =\n      DateTimeFormatter.ofPattern(\"yyyy_MM_dd_HH\");\n\n  @VisibleForTesting\n  public static final String NUM_TWEETS_CREATED_AT_PATTERN = \"num_tweets_%s_%s_created_at_%s\";\n\n  private static final String INVALID_FEATURE_UPDATES_DROPPED_PREFIX =\n      \"invalid_index_feature_update_dropped_\";\n\n  // The number of tweets not indexed because they have been previously indexed.\n  private static final SearchCounter DUPLICATE_TWEET_SKIPPED_COUNTER =\n      SearchCounter.export(\"duplicate_tweet_skipped\");\n\n  // The number of tweets that came out of order.\n  private static final SearchCounter OUT_OF_ORDER_TWEET_COUNTER =\n      SearchCounter.export(\"out_of_order_tweet\");\n\n  // The number partial updates dropped because the field could not be found in the schema.\n  // This counter is incremented once per field rather than once per partial update event.\n  // Note: caller may retry update, this counter will be incremented multiple times for same update.\n  private static final SearchCounter INVALID_FIELDS_IN_PARTIAL_UPDATES =\n      SearchCounter.export(\"invalid_fields_in_partial_updates\");\n\n  // The number partial updates dropped because the tweet id could not be found in the segment.\n  // Note: caller may retry update, this counter will be incremented multiple times for same update.\n  private static final SearchCounter PARTIAL_UPDATE_FOR_TWEET_NOT_IN_INDEX =\n      SearchCounter.export(\"partial_update_for_tweet_id_not_in_index\");\n\n  // The number of partial updates that were applied only partially, because the update could not\n  // be applied for at least one of the fields.\n  private static final SearchCounter PARTIAL_UPDATE_PARTIAL_FAILURE =\n      SearchCounter.export(\"partial_update_partial_failure\");\n\n  // Both the indexing chain and the index writer are lazily initialized when adding docs for\n  // the first time.\n  private final AtomicReference<EarlybirdIndexSegmentWriter> segmentWriterReference =\n      new AtomicReference<>();\n\n  // Stats from the PartitionIndexer / SimpleSegmentIndexer.\n  private final SegmentIndexStats indexStats;\n  private final String segmentName;\n  private final int maxSegmentSize;\n  private final long timeSliceID;\n  private final AtomicReference<EarlybirdIndexSegmentAtomicReader> luceneIndexReader =\n      new AtomicReference<>();\n  private final Directory luceneDir;\n  private final File luceneDirFile;\n  private final EarlybirdIndexConfig indexConfig;\n  private final List<Closeable> closableResources = Lists.newArrayList();\n  private long lastInOrderTweetId = 0;\n\n  private final EarlybirdIndexExtensionsFactory extensionsFactory;\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n  private final EarlybirdSearcherStats searcherStats;\n\n  private final Map<String, SearchCounter> indexedTweetsCounters = Maps.newHashMap();\n  private final PerFieldCounters perFieldCounters;\n  private final Clock clock;\n\n  @VisibleForTesting\n  public volatile boolean appendedLuceneIndex = false;\n\n  public EarlybirdSegment(\n      String segmentName,\n      long timeSliceID,\n      int maxSegmentSize,\n      Directory luceneDir,\n      EarlybirdIndexConfig indexConfig,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      EarlybirdSearcherStats searcherStats,\n      Clock clock) {\n    this.segmentName = segmentName;\n    this.maxSegmentSize = maxSegmentSize;\n    this.timeSliceID = timeSliceID;\n    this.luceneDir = luceneDir;\n    this.indexConfig = indexConfig;\n    this.indexStats = new SegmentIndexStats();\n    this.perFieldCounters = new PerFieldCounters();\n    this.extensionsFactory = new TweetSearchIndexExtensionsFactory();\n\n    if (luceneDir != null && luceneDir instanceof FSDirectory) {\n      // getDirectory() throws if the luceneDir is already closed.\n      // To delete a directory, we need to close it first.\n      // Obtain a reference to the File now, so we can delete it later.\n      // See SEARCH-5281\n      this.luceneDirFile = ((FSDirectory) luceneDir).getDirectory().toFile();\n    } else {\n      this.luceneDirFile = null;\n    }\n    this.searchIndexingMetricSet = Preconditions.checkNotNull(searchIndexingMetricSet);\n    this.searcherStats = searcherStats;\n    this.clock = clock;\n  }\n\n  @VisibleForTesting\n  public Directory getLuceneDirectory() {\n    return luceneDir;\n  }\n\n  public SegmentIndexStats getIndexStats() {\n    return indexStats;\n  }\n\n  /**\n   * Returns the smallest tweet ID in this segment. If the segment is not loaded yet, or is empty,\n   * DocIDToTweetIDMapper.ID_NOT_FOUND is returned (-1).\n   *\n   * @return The smallest tweet ID in this segment.\n   */\n  public long getLowestTweetId() {\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    if (segmentWriter == null) {\n      return DocIDToTweetIDMapper.ID_NOT_FOUND;\n    }\n\n    DocIDToTweetIDMapper mapper = segmentWriter.getSegmentData().getDocIDToTweetIDMapper();\n    int highestDocID = mapper.getPreviousDocID(Integer.MAX_VALUE);\n    return mapper.getTweetID(highestDocID);\n  }\n\n  /**\n   * Returns the cardinality (size) sum of the cardinality of each\n   * query cache set.\n   */\n  public long getQueryCachesCardinality() {\n    EarlybirdIndexSegmentWriter writer = getIndexSegmentWriter();\n    if (writer == null) {\n      // The segment is not loaded yet, or the query caches for this segment are not built yet.\n      return -1;\n    }\n\n    EarlybirdIndexSegmentData earlybirdIndexSegmentData = writer.getSegmentData();\n    return earlybirdIndexSegmentData.getQueryCachesCardinality();\n  }\n\n  public List<Pair<String, Long>> getQueryCachesData() {\n    return getIndexSegmentWriter().getSegmentData().getPerQueryCacheCardinality();\n  }\n\n\n  /**\n   * Returns the highest tweet ID in this segment. If the segment is not loaded yet, or is empty,\n   * DocIDToTweetIDMapper.ID_NOT_FOUND is returned (-1).\n   *\n   * @return The highest tweet ID in this segment.\n   */\n  public long getHighestTweetId() {\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    if (segmentWriter == null) {\n      return DocIDToTweetIDMapper.ID_NOT_FOUND;\n    }\n\n    DocIDToTweetIDMapper mapper = segmentWriter.getSegmentData().getDocIDToTweetIDMapper();\n    int lowestDocID = mapper.getNextDocID(-1);\n    return mapper.getTweetID(lowestDocID);\n  }\n\n  /**\n   * Optimizes the underlying segment data.\n   */\n  public void optimizeIndexes() throws IOException {\n    EarlybirdIndexSegmentWriter unoptimizedWriter = segmentWriterReference.get();\n    Preconditions.checkNotNull(unoptimizedWriter);\n\n    unoptimizedWriter.forceMerge();\n    unoptimizedWriter.close();\n\n    // Optimize our own data structures in the indexing chain\n    // In the archive this is pretty much a no-op.\n    // The indexWriter in writeableSegment should no longer be used and referenced, and\n    // writeableSegment.writer can be garbage collected at this point.\n    EarlybirdIndexSegmentData optimized = indexConfig.optimize(unoptimizedWriter.getSegmentData());\n    resetSegmentWriterReference(newWriteableSegment(optimized), true);\n\n    addSuccessFile();\n  }\n\n  /**\n   * Returns a new, optimized, realtime segment, by copying the data in this segment.\n   */\n  public EarlybirdSegment makeOptimizedSegment() throws IOException {\n    EarlybirdIndexSegmentWriter unoptimizedWriter = segmentWriterReference.get();\n    Preconditions.checkNotNull(unoptimizedWriter);\n    EarlybirdSegment optimizedSegment = new EarlybirdSegment(\n        segmentName,\n        timeSliceID,\n        maxSegmentSize,\n        luceneDir,\n        indexConfig,\n        searchIndexingMetricSet,\n        searcherStats,\n        clock);\n\n    EarlybirdIndexSegmentData optimizedSegmentData =\n        indexConfig.optimize(unoptimizedWriter.getSegmentData());\n    LOG.info(\"Done optimizing, setting segment data\");\n\n    optimizedSegment.setSegmentData(\n        optimizedSegmentData,\n        indexStats.getPartialUpdateCount(),\n        indexStats.getOutOfOrderUpdateCount());\n    return optimizedSegment;\n  }\n\n  public String getSegmentName() {\n    return segmentName;\n  }\n\n  public boolean isOptimized() {\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    return segmentWriter != null && segmentWriter.getSegmentData().isOptimized();\n  }\n\n  /**\n   * Removes the document for the given tweet ID from this segment, if this segment contains a\n   * document for this tweet ID.\n   */\n  public boolean delete(long tweetID) throws IOException {\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    if (!hasDocument(tweetID)) {\n      return false;\n    }\n\n    segmentWriter.deleteDocuments(new TweetIDQuery(tweetID));\n    return true;\n  }\n\n  protected void updateDocValues(long tweetID, String field, DocValuesUpdate update)\n      throws IOException {\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    segmentWriter.updateDocValues(new TweetIDQuery(tweetID), field, update);\n  }\n\n  /**\n   * Appends the Lucene index from another segment to this segment.\n   */\n  public void append(EarlybirdSegment otherSegment) throws IOException {\n    if (indexConfig.isIndexStoredOnDisk()) {\n      EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n      Preconditions.checkNotNull(segmentWriter);\n      EarlybirdIndexSegmentWriter otherSegmentWriter = otherSegment.segmentWriterReference.get();\n      if (otherSegmentWriter != null) {\n        otherSegmentWriter.close();\n      }\n      segmentWriter.addIndexes(otherSegment.luceneDir);\n      LOG.info(\"Calling forceMerge now after appending segment.\");\n      segmentWriter.forceMerge();\n      appendedLuceneIndex = true;\n      LOG.info(\"Appended {} docs to segment {}. New doc count = {}\",\n               otherSegment.indexStats.getStatusCount(), luceneDir.toString(),\n               indexStats.getStatusCount());\n\n      indexStats.setIndexSizeOnDiskInBytes(getSegmentSizeOnDisk());\n    }\n  }\n\n  /**\n   * Only needed for the on disk archive.\n   * Creates TwitterIndexReader used for searching. This is shared by all Searchers.\n   * This method also initializes the Lucene based mappers and CSF for the on disk archive.\n   *\n   * This method should be called after optimizing/loading a segment, but before the segment starts\n   * to serve search queries.\n   */\n  public void warmSegment() throws IOException {\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    Preconditions.checkNotNull(segmentWriter);\n\n    // only need to pre-create reader and initialize mappers and CSF in the on disk archive cluster\n    if (indexConfig.isIndexStoredOnDisk() && luceneIndexReader.get() == null) {\n      EarlybirdIndexSegmentAtomicReader luceneAtomicReader =\n          segmentWriter.getSegmentData().createAtomicReader();\n\n      luceneIndexReader.set(luceneAtomicReader);\n      closableResources.add(luceneAtomicReader);\n      closableResources.add(luceneDir);\n    }\n  }\n\n  /**\n   * Create a tweet index searcher on the segment.\n   *\n   * For production search session, the schema snapshot should be always passed in to make sure\n   * that the schema usage inside scoring is consistent.\n   *\n   * For non-production usage, like one-off debugging search, you can use the function call without\n   * the schema snapshot.\n   */\n  @Nullable\n  public EarlybirdSingleSegmentSearcher getSearcher(\n      UserTable userTable,\n      ImmutableSchemaInterface schemaSnapshot) throws IOException {\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    if (segmentWriter == null) {\n      return null;\n    }\n    return new EarlybirdSingleSegmentSearcher(\n        schemaSnapshot, getIndexReader(segmentWriter), userTable, searcherStats, clock);\n  }\n\n  /**\n   * Returns a new searcher for this segment.\n   */\n  @Nullable\n  public EarlybirdSingleSegmentSearcher getSearcher(\n      UserTable userTable) throws IOException {\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    if (segmentWriter == null) {\n      return null;\n    }\n    return new EarlybirdSingleSegmentSearcher(\n        segmentWriter.getSegmentData().getSchema().getSchemaSnapshot(),\n        getIndexReader(segmentWriter),\n        userTable,\n        searcherStats,\n        clock);\n  }\n\n  /**\n   * Returns a new reader for this segment.\n   */\n  @Nullable\n  public EarlybirdIndexSegmentAtomicReader getIndexReader() throws IOException {\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    if (segmentWriter == null) {\n      return null;\n    }\n    return getIndexReader(segmentWriter);\n  }\n\n  private EarlybirdIndexSegmentAtomicReader getIndexReader(\n      EarlybirdIndexSegmentWriter segmentWriter\n  ) throws IOException {\n    EarlybirdIndexSegmentAtomicReader reader = luceneIndexReader.get();\n    if (reader != null) {\n      return reader;\n    }\n    Preconditions.checkState(!indexConfig.isIndexStoredOnDisk());\n\n    // Realtime EB mode.\n    return segmentWriter.getSegmentData().createAtomicReader();\n  }\n\n  /**\n   * Gets max tweet id in this segment.\n   *\n   * @return the tweet id or -1 if not found.\n   */\n  public long getMaxTweetId() {\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    if (segmentWriter == null) {\n      return -1;\n    } else {\n      TweetIDMapper tweetIDMapper =\n          (TweetIDMapper) segmentWriter.getSegmentData().getDocIDToTweetIDMapper();\n      return tweetIDMapper.getMaxTweetID();\n    }\n  }\n\n  private EarlybirdIndexSegmentWriter newWriteableSegment(EarlybirdIndexSegmentData segmentData)\n      throws IOException {\n    EarlybirdIndexSegmentWriter old = segmentWriterReference.get();\n    if (old != null) {\n      old.close();\n    }\n\n    LOG.info(\"Creating new segment writer for {} on {}\", segmentName, luceneDir);\n    IndexWriterConfig indexWriterConfig = indexConfig.newIndexWriterConfig();\n    return segmentData.createEarlybirdIndexSegmentWriter(indexWriterConfig);\n  }\n\n  private void resetSegmentWriterReference(\n      EarlybirdIndexSegmentWriter segmentWriter, boolean previousSegmentWriterAllowed) {\n    EarlybirdIndexSegmentWriter previousSegmentWriter =\n        segmentWriterReference.getAndSet(segmentWriter);\n    if (!previousSegmentWriterAllowed) {\n      Preconditions.checkState(\n          previousSegmentWriter == null,\n          \"A previous segment writer must have been set for segment \" + segmentName);\n    }\n\n    // Reset the stats for the number of indexed tweets per hour and recompute them.\n    // See SEARCH-23619\n    for (SearchCounter indexedTweetsCounter : indexedTweetsCounters.values()) {\n      indexedTweetsCounter.reset();\n    }\n\n    if (segmentWriter != null) {\n      indexStats.setSegmentData(segmentWriter.getSegmentData());\n\n      if (indexConfig.getCluster() != EarlybirdCluster.FULL_ARCHIVE) {\n        initHourlyTweetCounts(segmentWriterReference.get());\n      }\n    } else {\n      // It's important to unset segment data so that there are no references to it\n      // and it can be GC-ed.\n      indexStats.unsetSegmentDataAndSaveCounts();\n    }\n  }\n\n  /**\n   * Add a document if it is not already in segment.\n   */\n  public void addDocument(TweetDocument doc) throws IOException {\n    if (indexConfig.isIndexStoredOnDisk()) {\n      addDocumentToArchiveSegment(doc);\n    } else {\n      addDocumentToRealtimeSegment(doc);\n    }\n  }\n\n  private void addDocumentToArchiveSegment(TweetDocument doc) throws IOException {\n    // For archive, the document id should come in order, to drop duplicates, only need to\n    // compare current id with last one.\n    long tweetId = doc.getTweetID();\n    if (tweetId == lastInOrderTweetId) {\n      LOG.warn(\"Dropped duplicate tweet for archive: {}\", tweetId);\n      DUPLICATE_TWEET_SKIPPED_COUNTER.increment();\n      return;\n    }\n\n    if (tweetId > lastInOrderTweetId && lastInOrderTweetId != 0) {\n      // Archive orders document from newest to oldest, so this shouldn't happen\n      LOG.warn(\"Encountered out-of-order tweet for archive: {}\", tweetId);\n      OUT_OF_ORDER_TWEET_COUNTER.increment();\n    } else {\n      lastInOrderTweetId = tweetId;\n    }\n\n    addDocumentInternal(doc);\n  }\n\n  private void addDocumentToRealtimeSegment(TweetDocument doc) throws IOException {\n    long tweetId = doc.getTweetID();\n    boolean outOfOrder = tweetId <= lastInOrderTweetId;\n    if (outOfOrder) {\n      OUT_OF_ORDER_TWEET_COUNTER.increment();\n    } else {\n      lastInOrderTweetId = tweetId;\n    }\n\n    // We only need to call hasDocument() for out-of-order tweets.\n    if (outOfOrder && hasDocument(tweetId)) {\n      // We do get duplicates sometimes so you'll see some amount of these.\n      DUPLICATE_TWEET_SKIPPED_COUNTER.increment();\n    } else {\n      addDocumentInternal(doc);\n      incrementHourlyTweetCount(doc.getTweetID());\n    }\n  }\n\n  private void addDocumentInternal(TweetDocument tweetDocument) throws IOException {\n    Document doc = tweetDocument.getDocument();\n\n    // Never write blank documents into the index.\n    if (doc == null || doc.getFields() == null || doc.getFields().size() == 0) {\n      return;\n    }\n\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    if (segmentWriter == null) {\n      EarlybirdIndexSegmentData segmentData = indexConfig.newSegmentData(\n          maxSegmentSize,\n          timeSliceID,\n          luceneDir,\n          extensionsFactory);\n      segmentWriter = newWriteableSegment(segmentData);\n      resetSegmentWriterReference(segmentWriter, false);\n    }\n\n    Preconditions.checkState(segmentWriter.numDocs() < maxSegmentSize,\n                             \"Reached max segment size %s\", maxSegmentSize);\n\n    IndexableField[] featuresField = doc.getFields(\n        EarlybirdFieldConstants.ENCODED_TWEET_FEATURES_FIELD_NAME);\n    Preconditions.checkState(featuresField.length == 1,\n            \"featuresField.length should be 1, but is %s\", featuresField.length);\n\n    // We require the createdAt field to be set so we can properly filter tweets based on time.\n    IndexableField[] createdAt =\n        doc.getFields(EarlybirdFieldConstant.CREATED_AT_FIELD.getFieldName());\n    Preconditions.checkState(createdAt.length == 1);\n\n    EarlybirdEncodedFeatures features = EarlybirdEncodedFeaturesUtil.fromBytes(\n        indexConfig.getSchema().getSchemaSnapshot(),\n        EarlybirdFieldConstant.ENCODED_TWEET_FEATURES_FIELD,\n        featuresField[0].binaryValue().bytes,\n        featuresField[0].binaryValue().offset);\n    boolean currentDocIsOffensive = features.isFlagSet(EarlybirdFieldConstant.IS_OFFENSIVE_FLAG);\n    perFieldCounters.increment(ThriftIndexingEventType.INSERT, doc);\n    segmentWriter.addTweet(doc, tweetDocument.getTweetID(), currentDocIsOffensive);\n  }\n\n  private void incrementHourlyTweetCount(long tweetId) {\n    // SEARCH-23619, We won't attempt to increment the count for pre-snowflake IDs, since\n    // extracting an exact create time is pretty tricky at this point, and the stat is mostly\n    // useful for checking realtime tweet indexing.\n    if (SnowflakeId.isSnowflakeId(tweetId)) {\n      long tweetCreateTime = SnowflakeId.unixTimeMillisFromId(tweetId);\n      String tweetHour = HOURLY_COUNT_DATE_TIME_FORMATTER.format(\n          ZonedDateTime.ofInstant(Instant.ofEpochMilli(tweetCreateTime), ZoneOffset.UTC));\n\n      String segmentOptimizedSuffix = isOptimized() ? \"optimized\" : \"unoptimized\";\n      SearchCounter indexedTweetsCounter = indexedTweetsCounters.computeIfAbsent(\n          tweetHour + \"_\" + segmentOptimizedSuffix,\n          (tweetHourKey) -> SearchCounter.export(String.format(\n              NUM_TWEETS_CREATED_AT_PATTERN, segmentOptimizedSuffix, segmentName, tweetHour)));\n      indexedTweetsCounter.increment();\n    }\n  }\n\n  private void initHourlyTweetCounts(EarlybirdIndexSegmentWriter segmentWriter) {\n    DocIDToTweetIDMapper mapper = segmentWriter.getSegmentData().getDocIDToTweetIDMapper();\n    int docId = Integer.MIN_VALUE;\n    while ((docId = mapper.getNextDocID(docId)) != DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      incrementHourlyTweetCount(mapper.getTweetID(docId));\n    }\n  }\n\n  /**\n   * Adds the given document for the given tweet ID to the segment, potentially out of order.\n   */\n  public boolean appendOutOfOrder(Document doc, long tweetID) throws IOException {\n    // Never write blank documents into the index.\n    if (doc == null || doc.getFields() == null || doc.getFields().size() == 0) {\n      return false;\n    }\n\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    if (segmentWriter == null) {\n      logAppendOutOfOrderFailure(tweetID, doc, \"segment is null\");\n      return false;\n    }\n\n    if (!indexConfig.supportOutOfOrderIndexing()) {\n      logAppendOutOfOrderFailure(tweetID, doc, \"out of order indexing not supported\");\n      return false;\n    }\n\n    if (!hasDocument(tweetID)) {\n      logAppendOutOfOrderFailure(tweetID, doc, \"tweet ID index lookup failed\");\n      searchIndexingMetricSet.updateOnMissingTweetCounter.increment();\n      perFieldCounters.incrementTweetNotInIndex(ThriftIndexingEventType.OUT_OF_ORDER_APPEND, doc);\n      return false;\n    }\n\n    perFieldCounters.increment(ThriftIndexingEventType.OUT_OF_ORDER_APPEND, doc);\n    segmentWriter.appendOutOfOrder(new TweetIDQuery(tweetID), doc);\n    indexStats.incrementOutOfOrderUpdateCount();\n    return true;\n  }\n\n  private void logAppendOutOfOrderFailure(long tweetID, Document doc, String reason) {\n    UPDATES_ERRORS_LOG.debug(\n        \"appendOutOfOrder() failed to apply update document with hash {} on tweet ID {}: {}\",\n        Objects.hashCode(doc), tweetID, reason);\n  }\n\n  /**\n   * Determines if this segment contains the given tweet ID.\n   */\n  public boolean hasDocument(long tweetID) throws IOException {\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    if (segmentWriter == null) {\n      return false;\n    }\n\n    return segmentWriter.getSegmentData().getDocIDToTweetIDMapper().getDocID(tweetID)\n        != DocIDToTweetIDMapper.ID_NOT_FOUND;\n  }\n\n  private static final String VERSION_PROP_NAME = \"version\";\n  private static final String VERSION_DESC_PROP_NAME = \"versionDescription\";\n  private static final String PARTIAL_UPDATES_COUNT = \"partialUpdatesCount\";\n  private static final String OUT_OF_ORDER_UPDATES_COUNT = \"outOfOrderUpdatesCount\";\n\n  private void checkIfFlushedDataVersionMatchesExpected(FlushInfo flushInfo) throws IOException {\n    int expectedVersionNumber = indexConfig.getSchema().getMajorVersionNumber();\n    String expectedVersionDesc = indexConfig.getSchema().getVersionDescription();\n    int version = flushInfo.getIntProperty(VERSION_PROP_NAME);\n    final String versionDesc = flushInfo.getStringProperty(VERSION_DESC_PROP_NAME);\n\n    if (version != expectedVersionNumber) {\n      throw new FlushVersionMismatchException(\"Flushed version mismatch. Expected: \"\n          + expectedVersionNumber + \", but was: \" + version);\n    }\n\n    if (!expectedVersionDesc.equals(versionDesc)) {\n      final String message = \"Flush version \" + expectedVersionNumber + \" is ambiguous\"\n          + \"  Expected: \" + expectedVersionDesc\n          + \"  Found:  \"  + versionDesc\n          + \"  Please clean up segments with bad flush version from HDFS and Earlybird local disk.\";\n      throw new FlushVersionMismatchException(message);\n    }\n  }\n\n  /**\n   * Loads the segment data and properties from the given deserializer and flush info.\n   *\n   * @param in The deserializer from which the segment's data will be read.\n   * @param flushInfo The flush info from which the segment's properties will be read.\n   */\n  public void load(DataDeserializer in, FlushInfo flushInfo) throws IOException {\n    checkIfFlushedDataVersionMatchesExpected(flushInfo);\n\n    int partialUpdatesCount = flushInfo.getIntProperty(PARTIAL_UPDATES_COUNT);\n    int outOfOrderUpdatesCount = flushInfo.getIntProperty(OUT_OF_ORDER_UPDATES_COUNT);\n\n    EarlybirdIndexSegmentData loadedSegmentData = indexConfig.loadSegmentData(\n        flushInfo, in, luceneDir, extensionsFactory);\n\n    setSegmentData(loadedSegmentData, partialUpdatesCount, outOfOrderUpdatesCount);\n  }\n\n  /**\n   * Update the data backing this EarlyirdSegment.\n   */\n  public void setSegmentData(\n      EarlybirdIndexSegmentData segmentData,\n      int partialUpdatesCount,\n      int outOfOrderUpdatesCount) throws IOException {\n    resetSegmentWriterReference(newWriteableSegment(segmentData), false);\n    try {\n      warmSegment();\n    } catch (IOException e) {\n      LOG.error(\"Failed to create IndexReader for segment {}. Will destroy unreadable segment.\",\n          segmentName, e);\n      destroyImmediately();\n      throw e;\n    }\n\n    LOG.info(\"Starting segment {} with {} partial updates, {} out of order updates and {} deletes.\",\n        segmentName, partialUpdatesCount, outOfOrderUpdatesCount, indexStats.getDeleteCount());\n    indexStats.setPartialUpdateCount(partialUpdatesCount);\n    indexStats.setOutOfOrderUpdateCount(outOfOrderUpdatesCount);\n    indexStats.setIndexSizeOnDiskInBytes(getSegmentSizeOnDisk());\n  }\n\n  /**\n   * Flushes the this segment's properties to the given FlushInfo instance, and this segment's data\n   * to the given DataSerializer instance.\n   *\n   * @param flushInfo The FlushInfo instance where all segment properties should be added.\n   * @param out The serializer to which all segment data should be flushed.\n   */\n  public void flush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n    flushInfo.addIntProperty(VERSION_PROP_NAME, indexConfig.getSchema().getMajorVersionNumber());\n    flushInfo.addStringProperty(VERSION_DESC_PROP_NAME,\n        indexConfig.getSchema().getVersionDescription());\n    flushInfo.addIntProperty(PARTIAL_UPDATES_COUNT, indexStats.getPartialUpdateCount());\n    flushInfo.addIntProperty(OUT_OF_ORDER_UPDATES_COUNT, indexStats.getOutOfOrderUpdateCount());\n    if (segmentWriterReference.get() == null) {\n      LOG.warn(\"Segment writer is null. flushInfo: {}\", flushInfo);\n    } else if (segmentWriterReference.get().getSegmentData() == null) {\n      LOG.warn(\"Segment data is null. segment writer: {}, flushInfo: {}\",\n          segmentWriterReference.get(), flushInfo);\n    }\n    segmentWriterReference.get().getSegmentData().flushSegment(flushInfo, out);\n    indexStats.setIndexSizeOnDiskInBytes(getSegmentSizeOnDisk());\n  }\n\n  /**\n   * Check to see if this segment can be loaded from an on-disk index, and load it if it can be.\n   *\n   * This should only be applicable to the current segment for the on-disk archive. It's not\n   * fully flushed until it's full, but we do have a lucene index on local disk which can be\n   * used at startup (rather than have to reindex all the current timeslice documents again).\n   *\n   * If loaded, the index reader will be pre-created, and the segment will be marked as\n   * optimized.\n   *\n   * If the index directory exists but it cannot be loaded, the index directory will be deleted.\n   *\n   * @return true if the index exists on disk, and was loaded.\n   */\n  public boolean tryToLoadExistingIndex() throws IOException {\n    Preconditions.checkState(segmentWriterReference.get() == null);\n    if (indexConfig.isIndexStoredOnDisk()) {\n      if (DirectoryReader.indexExists(luceneDir) && checkSuccessFile()) {\n        LOG.info(\"Index directory already exists for {} at {}\", segmentName, luceneDir);\n\n        // set the optimized flag, since we don't need to optimize any more, and pre-create\n        // the index reader (for the on-disk index optimize() is a noop that just sets the\n        // optimized flag).\n        EarlybirdIndexSegmentData earlybirdIndexSegmentData = indexConfig.newSegmentData(\n            maxSegmentSize,\n            timeSliceID,\n            luceneDir,\n            extensionsFactory);\n        EarlybirdIndexSegmentData optimizedEarlybirdIndexSegmentData =\n            indexConfig.optimize(earlybirdIndexSegmentData);\n        resetSegmentWriterReference(newWriteableSegment(optimizedEarlybirdIndexSegmentData), false);\n\n        warmSegment();\n\n        LOG.info(\"Used existing lucene index for {} with {} documents\",\n                 segmentName, indexStats.getStatusCount());\n\n        indexStats.setIndexSizeOnDiskInBytes(getSegmentSizeOnDisk());\n\n        return true;\n      } else {\n        // Check if there is an existing lucene dir without a SUCCESS file on disk.\n        // If so, we will remove it and reindex from scratch.\n        if (moveFSDirectoryIfExists(luceneDir)) {\n          // Throw here to be cleaned up and retried by SimpleSegmentIndexer.\n          throw new IOException(\"Found invalid existing lucene directory at: \" + luceneDir);\n        }\n      }\n    }\n    return false;\n  }\n\n  /**\n   * Partially updates a document with the field value(s) specified by event.\n   * Returns true if all writes were successful and false if one or more writes fail or if\n   * tweet id isn't found in the segment.\n   */\n  public boolean applyPartialUpdate(ThriftIndexingEvent event) throws IOException {\n    Preconditions.checkArgument(event.getEventType() == ThriftIndexingEventType.PARTIAL_UPDATE);\n    Preconditions.checkArgument(event.isSetUid());\n    Preconditions.checkArgument(!ThriftDocumentUtil.hasDuplicateFields(event.getDocument()));\n    ImmutableSchemaInterface schemaSnapshot = indexConfig.getSchema().getSchemaSnapshot();\n\n    long tweetId = event.getUid();\n    ThriftDocument doc = event.getDocument();\n\n    if (!hasDocument(tweetId)) {\n      // no need to attempt field writes, fail early\n      PARTIAL_UPDATE_FOR_TWEET_NOT_IN_INDEX.increment();\n       perFieldCounters.incrementTweetNotInIndex(\n           ThriftIndexingEventType.PARTIAL_UPDATE, doc);\n      return false;\n    }\n\n    int invalidFields = 0;\n    for (ThriftField field : doc.getFields()) {\n      String featureName = schemaSnapshot.getFieldName(field.getFieldConfigId());\n      FeatureConfiguration featureConfig =\n          schemaSnapshot.getFeatureConfigurationByName(featureName);\n      if (featureConfig == null) {\n        INVALID_FIELDS_IN_PARTIAL_UPDATES.increment();\n        invalidFields++;\n        continue;\n      }\n\n      perFieldCounters.increment(ThriftIndexingEventType.PARTIAL_UPDATE, featureName);\n\n      updateDocValues(\n          tweetId,\n          featureName,\n          (docValues, docID) -> updateFeatureValue(docID, featureConfig, docValues, field));\n    }\n\n    if (invalidFields > 0 && invalidFields != doc.getFieldsSize()) {\n      PARTIAL_UPDATE_PARTIAL_FAILURE.increment();\n    }\n\n    if (invalidFields == 0) {\n      indexStats.incrementPartialUpdateCount();\n    } else {\n      UPDATES_ERRORS_LOG.warn(\"Failed to apply update for tweetID {}, found {} invalid fields: {}\",\n          tweetId, invalidFields, event);\n    }\n\n    return invalidFields == 0;\n  }\n\n  @VisibleForTesting\n  static void updateFeatureValue(int docID,\n                                 FeatureConfiguration featureConfig,\n                                 ColumnStrideFieldIndex docValues,\n                                 ThriftField updateField) {\n    int oldValue = Math.toIntExact(docValues.get(docID));\n    int newValue = updateField.getFieldData().getIntValue();\n\n    if (!featureConfig.validateFeatureUpdate(oldValue, newValue)) {\n      // Counter values can only increase\n      SearchCounter.export(\n          INVALID_FEATURE_UPDATES_DROPPED_PREFIX + featureConfig.getName()).increment();\n    } else {\n      docValues.setValue(docID, newValue);\n    }\n  }\n\n  /**\n   * Checks if the provided directory exists and is not empty,\n   * and if it does moves it out to a diff directory for later inspection.\n   * @param luceneDirectory the dir to move if it exists.\n   * @return true iff we found an existing directory.\n   */\n  private static boolean moveFSDirectoryIfExists(Directory luceneDirectory) {\n    Preconditions.checkState(luceneDirectory instanceof FSDirectory);\n    File directory = ((FSDirectory) luceneDirectory).getDirectory().toFile();\n    if (directory != null && directory.exists() && directory.list().length > 0) {\n      // Save the bad lucene index by moving it out, for later inspection.\n      File movedDir = new File(directory.getParent(),\n          directory.getName() + \".failed.\" + System.currentTimeMillis());\n      LOG.warn(\"Moving existing non-successful index for {} from {} to {}\",\n               luceneDirectory, directory, movedDir);\n      boolean success = directory.renameTo(movedDir);\n      if (!success) {\n        LOG.warn(\"Unable to rename non-successful index: {}\", luceneDirectory);\n      }\n      return true;\n    }\n    return false;\n  }\n\n  /**\n   * For the on-disk archive, if we were able to successfully merge and flush the Lucene index to\n   * disk, we mark it explicitly with a SUCCESS file, so that it can be safely reused.\n   */\n  private void addSuccessFile() throws IOException {\n    if (indexConfig.isIndexStoredOnDisk()) {\n      IndexOutput successFile = luceneDir.createOutput(SUCCESS_FILE, IOContext.DEFAULT);\n      successFile.close();\n    }\n  }\n\n  /**\n   * Returns the current number of documents in this segment.\n   */\n  public int getNumDocs() throws IOException {\n    return indexStats.getStatusCount();\n  }\n\n  /**\n   * Reclaim resources used by this segment (E.g. closing lucene index reader).\n   * Resources will be reclaimed within the calling thread with no delay.\n   */\n  public void destroyImmediately() {\n    try {\n      closeSegmentWriter();\n      maybeDeleteSegmentOnDisk();\n      unloadSegmentFromMemory();\n    } finally {\n      indexConfig.getResourceCloser().closeResourcesImmediately(closableResources);\n    }\n  }\n\n  /**\n   * Close the in-memory resources belonging to this segment. This should allow the in-memory\n   * segment data to be garbage collected. After closing, the segment is not writable.\n   */\n  public void close() {\n    if (segmentWriterReference.get() == null) {\n      LOG.info(\"Segment {} already closed.\", segmentName);\n      return;\n    }\n\n    LOG.info(\"Closing segment {}.\", segmentName);\n    try {\n      closeSegmentWriter();\n      unloadSegmentFromMemory();\n    } finally {\n      indexConfig.getResourceCloser().closeResourcesImmediately(closableResources);\n    }\n  }\n\n  private void closeSegmentWriter() {\n    EarlybirdIndexSegmentWriter segmentWriter = segmentWriterReference.get();\n    if (segmentWriter != null) {\n      closableResources.add(() -> {\n          LOG.info(\"Closing writer for segment: {}\", segmentName);\n          segmentWriter.close();\n      });\n    }\n  }\n\n  private void maybeDeleteSegmentOnDisk() {\n    if (indexConfig.isIndexStoredOnDisk()) {\n      Preconditions.checkState(\n          luceneDir instanceof FSDirectory,\n          \"On-disk indexes should have an underlying directory that we can close and remove.\");\n      closableResources.add(luceneDir);\n\n      if (luceneDirFile != null && luceneDirFile.exists()) {\n        closableResources.add(new Closeable() {\n          @Override\n          public void close() throws IOException {\n            FileUtils.deleteDirectory(luceneDirFile);\n          }\n\n          @Override\n          public String toString() {\n            return \"delete {\" + luceneDirFile + \"}\";\n          }\n        });\n      }\n    }\n  }\n\n  private void unloadSegmentFromMemory() {\n    // Make sure we don't retain a reference to the IndexWriter or SegmentData.\n    resetSegmentWriterReference(null, true);\n  }\n\n  private long getSegmentSizeOnDisk() throws IOException {\n    searchIndexingMetricSet.segmentSizeCheckCount.increment();\n\n    long totalSize = 0;\n    if (luceneDir != null) {\n      for (String file : luceneDir.listAll()) {\n        totalSize += luceneDir.fileLength(file);\n      }\n    }\n    return totalSize;\n  }\n\n  //////////////////////////\n  // for unit tests only\n  //////////////////////////\n\n  public EarlybirdIndexConfig getEarlybirdIndexConfig() {\n    return indexConfig;\n  }\n\n  @VisibleForTesting\n  public boolean checkSuccessFile() {\n    return new File(luceneDirFile, SUCCESS_FILE).exists();\n  }\n\n  @VisibleForTesting\n  EarlybirdIndexSegmentWriter getIndexSegmentWriter() {\n    return segmentWriterReference.get();\n  }\n\n  // Helper class to encapsulate counter tables, patterns and various ways to increment\n  private class PerFieldCounters {\n    // The number of update/append events for each field in the schema.\n    private static final String PER_FIELD_EVENTS_COUNTER_PATTERN = \"%s_for_field_%s\";\n    // The number of dropped update/append events for each field due to tweetId not found\n    private static final String TWEET_NOT_IN_INDEX_PER_FIELD_EVENTS_COUNTER_PATTERN =\n        \"%s_for_tweet_id_not_in_index_for_field_%s\";\n    private final Table<ThriftIndexingEventType, String, SearchCounter> perFieldTable =\n        HashBasedTable.create();\n    private final Table<ThriftIndexingEventType, String, SearchCounter> notInIndexPerFieldTable =\n        HashBasedTable.create();\n\n    public void increment(\n        ThriftIndexingEventType eventType, ThriftDocument doc) {\n      ImmutableSchemaInterface schemaSnapshot = indexConfig.getSchema().getSchemaSnapshot();\n      for (ThriftField field : doc.getFields()) {\n        String fieldName = schemaSnapshot.getFieldName(field.getFieldConfigId());\n        incrementForPattern(\n            eventType, fieldName, perFieldTable, PER_FIELD_EVENTS_COUNTER_PATTERN);\n      }\n    }\n\n    public void incrementTweetNotInIndex(\n        ThriftIndexingEventType eventType, ThriftDocument doc) {\n      ImmutableSchemaInterface schemaSnapshot = indexConfig.getSchema().getSchemaSnapshot();\n      for (ThriftField field : doc.getFields()) {\n        String fieldName = schemaSnapshot.getFieldName(field.getFieldConfigId());\n        incrementForPattern(\n            eventType, fieldName, notInIndexPerFieldTable,\n            TWEET_NOT_IN_INDEX_PER_FIELD_EVENTS_COUNTER_PATTERN);\n      }\n    }\n\n    public void increment(ThriftIndexingEventType eventType, Document doc) {\n      for (IndexableField field : doc.getFields()) {\n        incrementForPattern(\n            eventType, field.name(),\n            perFieldTable, PER_FIELD_EVENTS_COUNTER_PATTERN);\n      }\n    }\n\n    public void increment(ThriftIndexingEventType eventType, String fieldName) {\n      incrementForPattern(eventType, fieldName, perFieldTable, PER_FIELD_EVENTS_COUNTER_PATTERN);\n    }\n\n    public void incrementTweetNotInIndex(ThriftIndexingEventType eventType, Document doc) {\n      for (IndexableField field : doc.getFields()) {\n        incrementForPattern(\n            eventType, field.name(),\n            notInIndexPerFieldTable,\n            TWEET_NOT_IN_INDEX_PER_FIELD_EVENTS_COUNTER_PATTERN);\n      }\n    }\n\n    private void incrementForPattern(\n        ThriftIndexingEventType eventType, String fieldName,\n        Table<ThriftIndexingEventType, String, SearchCounter> counterTable, String pattern) {\n\n      SearchCounter stat;\n      if (counterTable.contains(eventType, fieldName)) {\n        stat = counterTable.get(eventType, fieldName);\n      } else {\n        stat = SearchCounter.export(String.format(pattern, eventType, fieldName).toLowerCase());\n        counterTable.put(eventType, fieldName, stat);\n      }\n      stat.increment();\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/EarlybirdSegmentFactory.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.store.Directory;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.partitioning.base.Segment;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.partition.SearchIndexingMetricSet;\nimport com.twitter.search.earlybird.partition.SegmentSyncInfo;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\n\npublic class EarlybirdSegmentFactory {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdSegmentFactory.class);\n\n  private final EarlybirdIndexConfig earlybirdIndexConfig;\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n  private final EarlybirdSearcherStats searcherStats;\n  private Clock clock;\n\n  public EarlybirdSegmentFactory(\n      EarlybirdIndexConfig earlybirdIndexConfig,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      EarlybirdSearcherStats searcherStats,\n      Clock clock) {\n    this.earlybirdIndexConfig = earlybirdIndexConfig;\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n    this.searcherStats = searcherStats;\n    this.clock = clock;\n  }\n\n  public EarlybirdIndexConfig getEarlybirdIndexConfig() {\n    return earlybirdIndexConfig;\n  }\n\n  /**\n   * Creates a new earlybird segment.\n   */\n  public EarlybirdSegment newEarlybirdSegment(Segment segment, SegmentSyncInfo segmentSyncInfo)\n      throws IOException {\n    Directory dir = earlybirdIndexConfig.newLuceneDirectory(segmentSyncInfo);\n\n    LOG.info(\"Creating EarlybirdSegment on \" + dir.toString());\n\n    return new EarlybirdSegment(\n        segment.getSegmentName(),\n        segment.getTimeSliceID(),\n        segment.getMaxSegmentSize(),\n        dir,\n        earlybirdIndexConfig,\n        searchIndexingMetricSet,\n        searcherStats,\n        clock);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/EarlybirdSingleSegmentSearcher.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\nimport java.util.List;\nimport java.util.Locale;\nimport java.util.Map;\nimport java.util.Map.Entry;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.CollectionStatistics;\nimport org.apache.lucene.search.Collector;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.Explanation;\nimport org.apache.lucene.search.LeafCollector;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.TermStatistics;\nimport org.apache.lucene.search.Weight;\nimport org.apache.lucene.util.BytesRef;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.relevance.features.EarlybirdDocumentFeatures;\nimport com.twitter.search.common.results.thriftjava.FieldHitAttribution;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.search.TwitterCollector;\nimport com.twitter.search.common.search.TwitterIndexSearcher;\nimport com.twitter.search.common.util.analysis.LongTermAttributeImpl;\nimport com.twitter.search.common.util.lang.ThriftLanguageUtil;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentData;\nimport com.twitter.search.earlybird.EarlybirdSearcher;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.search.EarlybirdLuceneSearcher;\nimport com.twitter.search.earlybird.search.Hit;\nimport com.twitter.search.earlybird.search.SearchRequestInfo;\nimport com.twitter.search.earlybird.search.SimpleSearchResults;\nimport com.twitter.search.earlybird.search.facets.AbstractFacetTermCollector;\nimport com.twitter.search.earlybird.search.facets.TermStatisticsCollector;\nimport com.twitter.search.earlybird.search.facets.TermStatisticsRequestInfo;\nimport com.twitter.search.earlybird.search.relevance.scoring.RelevanceQuery;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCount;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCountMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird.thrift.ThriftTermRequest;\nimport com.twitter.search.earlybird.thrift.ThriftTermResults;\nimport com.twitter.search.earlybird.thrift.ThriftTermStatisticsResults;\n\npublic class EarlybirdSingleSegmentSearcher extends EarlybirdLuceneSearcher {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdSingleSegmentSearcher.class);\n\n  private final EarlybirdIndexSegmentAtomicReader twitterReader;\n  private final ImmutableSchemaInterface schema;\n  private final UserTable userTable;\n  private final long timeSliceID;\n\n  private final EarlybirdSearcherStats searcherStats;\n  private Clock clock;\n\n  public EarlybirdSingleSegmentSearcher(\n      ImmutableSchemaInterface schema,\n      EarlybirdIndexSegmentAtomicReader reader,\n      UserTable userTable,\n      EarlybirdSearcherStats searcherStats,\n      Clock clock) {\n    super(reader);\n    this.schema = schema;\n    this.twitterReader = reader;\n    this.userTable = userTable;\n    this.timeSliceID = reader.getSegmentData().getTimeSliceID();\n    this.searcherStats = searcherStats;\n    this.clock = clock;\n  }\n\n  public final long getTimeSliceID() {\n    return timeSliceID;\n  }\n\n  public EarlybirdIndexSegmentAtomicReader getTwitterIndexReader() {\n    return twitterReader;\n  }\n\n  /**\n   * search() main loop.\n   * This behaves exactly like IndexSearcher.search() if a stock Lucene collector passed in.\n   * However, if a TwitterCollector is passed in, this class performs Twitter style early\n   * termination without relying on\n   * {@link org.apache.lucene.search.CollectionTerminatedException}.\n   * This method is nearly identical to TwitterIndexSearcher.search() with two differences:\n   *  1) advances to smallest docID before searching.  Important to skip incomplete docs in\n   *     realtime segments.\n   *  2) skips deletes using twitterReader\n   */\n  @Override\n  protected void search(List<LeafReaderContext> leaves, Weight weight, Collector coll)\n      throws IOException {\n    // If an TwitterCollector is passed in, we can do a few extra things in here, such\n    // as early termination.  Otherwise we can just fall back to IndexSearcher.search().\n    if (!(coll instanceof TwitterCollector)) {\n      super.search(leaves, weight, coll);\n      return;\n    }\n\n    TwitterCollector collector = (TwitterCollector) coll;\n    if (collector.isTerminated()) {\n      return;\n    }\n\n    LOG.debug(\"Starting segment {}\", timeSliceID);\n\n    // Notify the collector that we're starting this segment, and check for early\n    // termination criteria again.  setNextReader() performs 'expensive' early\n    // termination checks in some implementations such as TwitterEarlyTerminationCollector.\n    LeafCollector leafCollector = collector.getLeafCollector(twitterReader.getContext());\n    if (collector.isTerminated()) {\n      return;\n    }\n\n    // Initialize the scorer:\n    // Note that constructing the scorer may actually do real work, such as advancing to the\n    // first hit.\n    // The scorer may be null if we can tell right away that the query has no hits: e.g. if the\n    // first hit does not actually exist.\n    Scorer scorer = weight.scorer(twitterReader.getContext());\n    if (scorer == null) {\n      LOG.debug(\"Scorer was null, not searching segment {}\", timeSliceID);\n      collector.finishSegment(DocIdSetIterator.NO_MORE_DOCS);\n      return;\n    }\n    leafCollector.setScorer(scorer);\n\n    // Make sure to start searching at the smallest docID.\n    DocIdSetIterator docIdSetIterator = scorer.iterator();\n    int smallestDocId = twitterReader.getSmallestDocID();\n    int docID = docIdSetIterator.advance(smallestDocId);\n\n    // Collect results.\n    while (docID != DocIdSetIterator.NO_MORE_DOCS) {\n      // Exclude deleted docs.\n      if (!twitterReader.getDeletesView().isDeleted(docID)) {\n        leafCollector.collect(docID);\n      }\n\n      // Check if we're done after we consumed the document.\n      if (collector.isTerminated()) {\n        break;\n      }\n\n      docID = docIdSetIterator.nextDoc();\n    }\n\n    // Always finish the segment, providing the last docID advanced to.\n    collector.finishSegment(docID);\n  }\n\n  @Override\n  public void fillFacetResults(\n      AbstractFacetTermCollector collector, ThriftSearchResults searchResults)\n      throws IOException {\n    if (searchResults == null || searchResults.getResultsSize() == 0) {\n      return;\n    }\n\n    EarlybirdIndexSegmentData segmentData = twitterReader.getSegmentData();\n    collector.resetFacetLabelProviders(\n        segmentData.getFacetLabelProviders(), segmentData.getFacetIDMap());\n    DocIDToTweetIDMapper docIdMapper = segmentData.getDocIDToTweetIDMapper();\n    for (ThriftSearchResult result : searchResults.getResults()) {\n      int docId = docIdMapper.getDocID(result.getId());\n      if (docId < 0) {\n        continue;\n      }\n\n      segmentData.getFacetCountingArray().collectForDocId(docId, collector);\n      collector.fillResultAndClear(result);\n    }\n  }\n\n  @Override\n  public TermStatisticsCollector.TermStatisticsSearchResults collectTermStatistics(\n      TermStatisticsRequestInfo searchRequestInfo,\n      EarlybirdSearcher searcher, int requestDebugMode) throws IOException {\n    TermStatisticsCollector collector = new TermStatisticsCollector(\n        schema, searchRequestInfo, searcherStats, clock, requestDebugMode);\n\n    search(searchRequestInfo.getLuceneQuery(), collector);\n    searcher.maybeSetCollectorDebugInfo(collector);\n    return collector.getResults();\n  }\n\n  /** This method is only used for debugging, so it's not optimized for speed */\n  @Override\n  public void explainSearchResults(SearchRequestInfo searchRequestInfo,\n                                   SimpleSearchResults hits,\n                                   ThriftSearchResults searchResults) throws IOException {\n    Weight weight =\n        createWeight(rewrite(searchRequestInfo.getLuceneQuery()), ScoreMode.COMPLETE, 1.0f);\n\n    DocIDToTweetIDMapper docIdMapper = twitterReader.getSegmentData().getDocIDToTweetIDMapper();\n    for (int i = 0; i < hits.numHits(); i++) {\n      final Hit hit = hits.getHit(i);\n      Preconditions.checkState(hit.getTimeSliceID() == timeSliceID,\n          \"hit: \" + hit.toString() + \" is not in timeslice: \" + timeSliceID);\n      final ThriftSearchResult result = searchResults.getResults().get(i);\n      if (!result.isSetMetadata()) {\n        result.setMetadata(new ThriftSearchResultMetadata()\n            .setPenguinVersion(EarlybirdConfig.getPenguinVersionByte()));\n      }\n\n      final int docIdToExplain = docIdMapper.getDocID(hit.getStatusID());\n      if (docIdToExplain == DocIDToTweetIDMapper.ID_NOT_FOUND) {\n        result.getMetadata().setExplanation(\n            \"ERROR: Could not find doc ID to explain for \" + hit.toString());\n      } else {\n        Explanation explanation;\n        FieldHitAttribution fieldHitAttribution = result.getMetadata().getFieldHitAttribution();\n        if (weight instanceof RelevanceQuery.RelevanceWeight && fieldHitAttribution != null) {\n          RelevanceQuery.RelevanceWeight relevanceWeight =\n              (RelevanceQuery.RelevanceWeight) weight;\n\n          explanation = relevanceWeight.explain(\n              twitterReader.getContext(), docIdToExplain, fieldHitAttribution);\n        } else {\n          explanation = weight.explain(twitterReader.getContext(), docIdToExplain);\n        }\n        hit.setHasExplanation(true);\n        result.getMetadata().setExplanation(explanation.toString());\n      }\n    }\n  }\n\n  @Override\n  public void fillFacetResultMetadata(Map<Term, ThriftFacetCount> facetResults,\n                                      ImmutableSchemaInterface documentSchema,\n                                      byte debugMode) throws IOException {\n    FacetLabelProvider provider = twitterReader.getFacetLabelProviders(\n            documentSchema.getFacetFieldByFacetName(EarlybirdFieldConstant.TWIMG_FACET));\n\n    FacetLabelProvider.FacetLabelAccessor photoAccessor = null;\n\n    if (provider != null) {\n      photoAccessor = provider.getLabelAccessor();\n    }\n\n    for (Entry<Term, ThriftFacetCount> facetResult : facetResults.entrySet()) {\n      Term term = facetResult.getKey();\n      ThriftFacetCount facetCount = facetResult.getValue();\n\n      ThriftFacetCountMetadata metadata = facetCount.getMetadata();\n      if (metadata == null) {\n        metadata = new ThriftFacetCountMetadata();\n        facetCount.setMetadata(metadata);\n      }\n\n      fillTermMetadata(term, metadata, photoAccessor, debugMode);\n    }\n  }\n\n  @Override\n  public void fillTermStatsMetadata(ThriftTermStatisticsResults termStatsResults,\n                                    ImmutableSchemaInterface documentSchema,\n                                    byte debugMode) throws IOException {\n\n    FacetLabelProvider provider = twitterReader.getFacetLabelProviders(\n        documentSchema.getFacetFieldByFacetName(EarlybirdFieldConstant.TWIMG_FACET));\n\n    FacetLabelProvider.FacetLabelAccessor photoAccessor = null;\n\n    if (provider != null) {\n      photoAccessor = provider.getLabelAccessor();\n    }\n\n    for (Map.Entry<ThriftTermRequest, ThriftTermResults> entry\n         : termStatsResults.termResults.entrySet()) {\n\n      ThriftTermRequest termRequest = entry.getKey();\n      if (termRequest.getFieldName().isEmpty()) {\n        continue;\n      }\n      Schema.FieldInfo facetField = schema.getFacetFieldByFacetName(termRequest.getFieldName());\n      Term term = null;\n      if (facetField != null) {\n        term = new Term(facetField.getName(), termRequest.getTerm());\n      }\n      if (term == null) {\n        continue;\n      }\n\n      ThriftFacetCountMetadata metadata = entry.getValue().getMetadata();\n      if (metadata == null) {\n        metadata = new ThriftFacetCountMetadata();\n        entry.getValue().setMetadata(metadata);\n      }\n\n      fillTermMetadata(term, metadata, photoAccessor, debugMode);\n    }\n  }\n\n  private void fillTermMetadata(Term term, ThriftFacetCountMetadata metadata,\n                                FacetLabelProvider.FacetLabelAccessor photoAccessor,\n                                byte debugMode) throws IOException {\n    boolean isTwimg = term.field().equals(EarlybirdFieldConstant.TWIMG_LINKS_FIELD.getFieldName());\n    int internalDocID = DocIDToTweetIDMapper.ID_NOT_FOUND;\n    long statusID = -1;\n    long userID = -1;\n    Term facetTerm = term;\n\n    // Deal with the from_user_id facet.\n    if (term.field().equals(EarlybirdFieldConstant.FROM_USER_ID_CSF.getFieldName())) {\n      userID = Long.parseLong(term.text());\n      facetTerm = new Term(EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName(),\n          LongTermAttributeImpl.copyIntoNewBytesRef(userID));\n    } else if (isTwimg) {\n      statusID = Long.parseLong(term.text());\n      internalDocID = twitterReader.getSegmentData().getDocIDToTweetIDMapper().getDocID(statusID);\n    }\n\n    if (internalDocID == DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      // If this is not a twimg, this is how statusID should be looked up\n      //\n      // If this is a twimg but we couldn't find the internalDocID, that means this segment,\n      // or maybe even this earlybird, does not contain the original tweet. Then we treat this as\n      // a normal facet for now\n      internalDocID = twitterReader.getOldestDocID(facetTerm);\n      if (internalDocID >= 0) {\n        statusID =\n            twitterReader.getSegmentData().getDocIDToTweetIDMapper().getTweetID(internalDocID);\n      } else {\n        statusID = -1;\n      }\n    }\n\n    // make sure tweet is not deleted\n    if (internalDocID < 0 || twitterReader.getDeletesView().isDeleted(internalDocID)) {\n      return;\n    }\n\n    if (metadata.isSetStatusId()\n        && metadata.getStatusId() > 0\n        && metadata.getStatusId() <= statusID) {\n      // we already have the metadata for this facet from an earlier tweet\n      return;\n    }\n\n    // now check if this tweet is offensive, e.g. antisocial, nsfw, sensitive\n    EarlybirdDocumentFeatures documentFeatures = new EarlybirdDocumentFeatures(twitterReader);\n    documentFeatures.advance(internalDocID);\n    boolean isOffensiveFlagSet =\n        documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_OFFENSIVE_FLAG);\n    boolean isSensitiveFlagSet =\n        documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_SENSITIVE_CONTENT);\n    boolean offensive = isOffensiveFlagSet || isSensitiveFlagSet;\n\n    // also, user should not be marked as antisocial, nsfw or offensive\n    if (userID < 0) {\n      userID = documentFeatures.getFeatureValue(EarlybirdFieldConstant.FROM_USER_ID_CSF);\n    }\n    offensive |= userTable.isSet(userID,\n        UserTable.ANTISOCIAL_BIT\n        | UserTable.OFFENSIVE_BIT\n        | UserTable.NSFW_BIT);\n\n    metadata.setStatusId(statusID);\n    metadata.setTwitterUserId(userID);\n    metadata.setCreated_at(twitterReader.getSegmentData().getTimeMapper().getTime(internalDocID));\n    int langId = (int) documentFeatures.getFeatureValue(EarlybirdFieldConstant.LANGUAGE);\n    Locale lang = ThriftLanguageUtil.getLocaleOf(ThriftLanguage.findByValue(langId));\n    metadata.setStatusLanguage(ThriftLanguageUtil.getThriftLanguageOf(lang));\n    metadata.setStatusPossiblySensitive(offensive);\n    if (isTwimg && photoAccessor != null && !metadata.isSetNativePhotoUrl()) {\n      int termID = twitterReader.getTermID(term);\n      if (termID != EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND) {\n        BytesRef termPayload = photoAccessor.getTermPayload(termID);\n        if (termPayload != null) {\n          metadata.setNativePhotoUrl(termPayload.utf8ToString());\n        }\n      }\n    }\n\n    if (debugMode > 3) {\n      StringBuilder sb = new StringBuilder(256);\n      if (metadata.isSetExplanation()) {\n        sb.append(metadata.getExplanation());\n      }\n      sb.append(String.format(\"TweetId=%d (%s %s), UserId=%d (%s %s), Term=%s\\n\",\n          statusID,\n          isOffensiveFlagSet ? \"OFFENSIVE\" : \"\",\n          isSensitiveFlagSet ? \"SENSITIVE\" : \"\",\n          userID,\n          userTable.isSet(userID, UserTable.ANTISOCIAL_BIT) ? \"ANTISOCIAL\" : \"\",\n          userTable.isSet(userID, UserTable.NSFW_BIT) ? \"NSFW\" : \"\",\n          term.toString()));\n      metadata.setExplanation(sb.toString());\n    }\n  }\n\n  public ImmutableSchemaInterface getSchemaSnapshot() {\n    return schema;\n  }\n\n  @Override\n  public CollectionStatistics collectionStatistics(String field) throws IOException {\n    return TwitterIndexSearcher.collectionStatistics(field, getIndexReader());\n  }\n\n  @Override\n  public TermStatistics termStatistics(Term term, int docFreq, long totalTermFreq) {\n    return TwitterIndexSearcher.termStats(term, docFreq, totalTermFreq);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/OptimizedTimeMapper.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\nimport java.util.Arrays;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.TimeMapper;\nimport com.twitter.search.core.earlybird.index.inverted.IntBlockPool;\n\n/**\n * A TimeMapper implementation that stores the timestamps associated with the doc IDs in an array.\n */\npublic class OptimizedTimeMapper extends AbstractInMemoryTimeMapper implements Flushable {\n  // Doc id to timestamp map. Timestamps that are negative are out-of-order.\n  protected final int[] timeMap;\n\n  // Size must be greater than the max doc ID stored in the optimized tweet ID mapper.\n  public OptimizedTimeMapper(RealtimeTimeMapper realtimeTimeMapper,\n                             DocIDToTweetIDMapper originalTweetIdMapper,\n                             DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    super();\n    int maxDocId = optimizedTweetIdMapper.getPreviousDocID(Integer.MAX_VALUE);\n    timeMap = new int[maxDocId + 1];\n    Arrays.fill(timeMap, ILLEGAL_TIME);\n\n    int docId = maxDocId;\n    while (docId != DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      int originalDocId = originalTweetIdMapper.getDocID(optimizedTweetIdMapper.getTweetID(docId));\n      Preconditions.checkState(originalDocId != DocIDToTweetIDMapper.ID_NOT_FOUND);\n\n      int docIdTimestamp = realtimeTimeMapper.getTime(originalDocId);\n      Preconditions.checkState(docIdTimestamp != TimeMapper.ILLEGAL_TIME);\n\n      doAddMapping(docId, docIdTimestamp);\n\n      docId = optimizedTweetIdMapper.getPreviousDocID(docId);\n    }\n  }\n\n  private OptimizedTimeMapper(int[] timeMap,\n                              int reverseMapLastIndex,\n                              IntBlockPool reverseMapTimes,\n                              IntBlockPool reverseMapIds) {\n    super(reverseMapLastIndex, reverseMapTimes, reverseMapIds);\n    this.timeMap = timeMap;\n  }\n\n  @Override\n  public int getTime(int docID) {\n    return timeMap[docID];\n  }\n\n  @Override\n  protected void setTime(int docID, int timeSeconds) {\n    timeMap[docID] = timeSeconds;\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<OptimizedTimeMapper> {\n    private static final String REVERSE_MAP_LAST_INDEX_PROP = \"reverseMapLastIndex\";\n    private static final String TIMES_SUB_PROP = \"times\";\n    private static final String IDS_SUB_PROP = \"ids\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(OptimizedTimeMapper objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      OptimizedTimeMapper mapper = getObjectToFlush();\n      out.writeIntArray(mapper.timeMap);\n      flushInfo.addIntProperty(REVERSE_MAP_LAST_INDEX_PROP, mapper.reverseMapLastIndex);\n      mapper.reverseMapTimes.getFlushHandler().flush(\n          flushInfo.newSubProperties(TIMES_SUB_PROP), out);\n      mapper.reverseMapIds.getFlushHandler().flush(\n          flushInfo.newSubProperties(IDS_SUB_PROP), out);\n    }\n\n    @Override\n    protected OptimizedTimeMapper doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      return new OptimizedTimeMapper(\n          in.readIntArray(),\n          flushInfo.getIntProperty(REVERSE_MAP_LAST_INDEX_PROP),\n          new IntBlockPool.FlushHandler().load(flushInfo.getSubProperties(TIMES_SUB_PROP), in),\n          new IntBlockPool.FlushHandler().load(flushInfo.getSubProperties(IDS_SUB_PROP), in));\n    }\n  }\n\n  @Override\n  public TimeMapper optimize(DocIDToTweetIDMapper originalTweetIdMapper,\n                             DocIDToTweetIDMapper optimizedTweetIdMapper) {\n    throw new UnsupportedOperationException(\"OptimizedTimeMapper instances are already optimized.\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/OptimizedTweetIDMapper.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\nimport it.unimi.dsi.fastutil.longs.Long2IntMap;\nimport it.unimi.dsi.fastutil.longs.Long2IntOpenHashMap;\nimport it.unimi.dsi.fastutil.longs.LongArrays;\n\n/**\n * After a segment is complete, we call {@link EarlybirdSegment#optimizeIndexes()} to compact the\n * doc IDs assigned to the tweets in this segment, so that we can do faster ceil and floor lookups.\n */\npublic class OptimizedTweetIDMapper extends TweetIDMapper {\n  // Maps doc IDs to tweet IDs. Therefore, it should be sorted in descending order of tweet IDs.\n  protected final long[] inverseMap;\n  private final Long2IntMap tweetIdToDocIdMap;\n\n  private OptimizedTweetIDMapper(long[] inverseMap,\n                                 long minTweetID,\n                                 long maxTweetID,\n                                 int minDocID,\n                                 int maxDocID) {\n    super(minTweetID, maxTweetID, minDocID, maxDocID, inverseMap.length);\n    this.inverseMap = inverseMap;\n    this.tweetIdToDocIdMap = buildTweetIdToDocIdMap();\n  }\n\n  public OptimizedTweetIDMapper(OutOfOrderRealtimeTweetIDMapper source) throws IOException {\n    super(source.getMinTweetID(),\n          source.getMaxTweetID(),\n          0,\n          source.getNumDocs() - 1,\n          source.getNumDocs());\n    inverseMap = source.sortTweetIds();\n    tweetIdToDocIdMap = buildTweetIdToDocIdMap();\n  }\n\n  private Long2IntMap buildTweetIdToDocIdMap() {\n    int[] values = new int[inverseMap.length];\n    for (int i = 0; i < values.length; i++) {\n      values[i] = i;\n    }\n\n    Long2IntMap map = new Long2IntOpenHashMap(inverseMap, values);\n    map.defaultReturnValue(-1);\n    return map;\n  }\n\n  @Override\n  public int getDocID(long tweetID) {\n    return tweetIdToDocIdMap.getOrDefault(tweetID, ID_NOT_FOUND);\n  }\n\n  @Override\n  protected int getNextDocIDInternal(int docID) {\n    // The doc IDs are consecutive and TweetIDMapper already checked the boundary conditions.\n    return docID + 1;\n  }\n\n  @Override\n  protected int getPreviousDocIDInternal(int docID) {\n    // The doc IDs are consecutive and TweetIDMapper already checked the boundary conditions.\n    return docID - 1;\n  }\n\n  @Override\n  public long getTweetID(int internalID) {\n    return inverseMap[internalID];\n  }\n\n  @Override\n  protected int findDocIDBoundInternal(long tweetID, boolean findMaxDocID) {\n    int docId = tweetIdToDocIdMap.get(tweetID);\n    if (docId >= 0) {\n      return docId;\n    }\n\n    int binarySearchResult =\n        LongArrays.binarySearch(inverseMap, tweetID, (k1, k2) -> -Long.compare(k1, k2));\n    // Since the tweet ID is not present in this mapper, the binary search should return a negative\n    // value (-insertionPoint - 1). And since TweetIDMapper.findDocIdBound() already verified that\n    // tweetID is not smaller than all tweet IDs in this mapper, and not larger than all tweet IDs\n    // in this mapper, the insertionPoint should never be 0 or inverseMap.length.\n    int insertionPoint = -binarySearchResult - 1;\n    // The insertion point is the index in the tweet array of the upper bound of the search, so if\n    // we want the lower bound, because doc IDs are dense, we subtract one.\n    return findMaxDocID ? insertionPoint : insertionPoint - 1;\n  }\n\n  @Override\n  protected final int addMappingInternal(final long tweetID) {\n    throw new UnsupportedOperationException(\"The OptimizedTweetIDMapper is immutable.\");\n  }\n\n  @Override\n  public DocIDToTweetIDMapper optimize() {\n    throw new UnsupportedOperationException(\"OptimizedTweetIDMapper is already optimized.\");\n  }\n\n  @Override\n  public FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static class FlushHandler extends Flushable.Handler<OptimizedTweetIDMapper> {\n    private static final String MIN_TWEET_ID_PROP_NAME = \"MinTweetID\";\n    private static final String MAX_TWEET_ID_PROP_NAME = \"MaxTweetID\";\n    private static final String MIN_DOC_ID_PROP_NAME = \"MinDocID\";\n    private static final String MAX_DOC_ID_PROP_NAME = \"MaxDocID\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(OptimizedTweetIDMapper objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out) throws IOException {\n      OptimizedTweetIDMapper objectToFlush = getObjectToFlush();\n      flushInfo.addLongProperty(MIN_TWEET_ID_PROP_NAME, objectToFlush.getMinTweetID());\n      flushInfo.addLongProperty(MAX_TWEET_ID_PROP_NAME, objectToFlush.getMaxTweetID());\n      flushInfo.addIntProperty(MIN_DOC_ID_PROP_NAME, objectToFlush.getMinDocID());\n      flushInfo.addIntProperty(MAX_DOC_ID_PROP_NAME, objectToFlush.getMaxDocID());\n      out.writeLongArray(objectToFlush.inverseMap);\n    }\n\n    @Override\n    protected OptimizedTweetIDMapper doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      return new OptimizedTweetIDMapper(in.readLongArray(),\n                                        flushInfo.getLongProperty(MIN_TWEET_ID_PROP_NAME),\n                                        flushInfo.getLongProperty(MAX_TWEET_ID_PROP_NAME),\n                                        flushInfo.getIntProperty(MIN_DOC_ID_PROP_NAME),\n                                        flushInfo.getIntProperty(MAX_DOC_ID_PROP_NAME));\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/OutOfOrderRealtimeTweetIDMapper.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\nimport java.util.Arrays;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\nimport it.unimi.dsi.fastutil.ints.Int2ByteOpenHashMap;\nimport it.unimi.dsi.fastutil.ints.Int2LongMap;\nimport it.unimi.dsi.fastutil.ints.Int2LongOpenHashMap;\n\n/**\n * A mapper that maps tweet IDs to doc IDs based on the tweet timestamps. This mapper guarantees\n * that if creationTime(A) > creationTime(B), then docId(A) < docId(B), no matter in which order\n * the tweets are added to this mapper. However, if creationTime(A) == creationTime(B), then there\n * is no guarantee on the order between docId(A) and docId(B).\n *\n * Essentially, this mapper guarantees that tweets with a later creation time are mapped to smaller\n * doc IDs, but it does not provide any ordering for tweets with the same timestamp (down to\n * millisecond granularity, which is what Snowflake provides). Our claim is that ordering tweets\n * with the same timestamp is not needed, because for the purposes of realtime search, the only\n * significant part of the tweet ID is the timestamp. So any such ordering would just be an ordering\n * for the Snowflake shards and/or sequence numbers, rather than a time based ordering for tweets.\n *\n * The mapper uses the following scheme to assign docIDs to tweets:\n *   +----------+-----------------------------+------------------------------+\n *   | Bit 0    | Bits 1 - 27                 | Bits 28 - 31                 |\n *   + ---------+-----------------------------+------------------------------+\n *   | sign     | tweet ID timestamp -        | Allow 16 tweets to be posted |\n *   | always 0 | segment boundary timestamp  | on the same millisecond      |\n *   + ---------+-----------------------------+------------------------------+\n *\n * Important assumptions:\n *   * Snowflake IDs have millisecond granularity. Therefore, 27 bits is enough to represent a time\n *     period of 2^27 / (3600 * 100) = ~37 hours, which is more than enough to cover one realtime\n *     segment (our realtime segments currently span ~13 hours).\n *   * At peak times, the tweet posting rate is less than 10,000 tps. Given our current partitioning\n *     scheme (22 partitions), each realtime earlybird should expect to get less than 500 tweets per\n *     second, which comes down to less than 1 tweet per millisecond, assuming the partitioning hash\n *     function distributes the tweets fairly randomly independent of their timestamps. Therefore,\n *     providing space for 16 tweets (4 bits) in every millisecond should be more than enough to\n *     accommodate the current requirements, and any potential future changes (higher tweet rate,\n *     fewer partitions, etc.).\n *\n * How the mapper works:\n *   * The tweetId -> docId conversion is implicit (using the tweet's timestamp).\n *   * We use a IntToByteMap to store the number of tweets for each timestamp, so that we can\n *     allocate different doc IDs to tweets posted on the same millisecond. The size of this map is:\n *         segmentSize * 2 (load factor) * 1 (size of byte) = 16MB\n *   * The docId -> tweetId mappings are stored in an IntToLongMap. The size of this map is:\n *         segmentSize * 2 (load factor) * 8 (size of long) = 128MB\n *   * The mapper takes the \"segment boundary\" (the timestamp of the timeslice ID) as a parameter.\n *     This segment boundary determines the earliest tweet that this mapper can correctly index\n *     (it is subtracted from the timestamp of all tweets added to the mapper). Therefore, in order\n *     to correctly handle late tweets, we move back this segment boundary by twelve hour.\n *   * Tweets created before (segment boundary - 12 hours) are stored as if their timestamp was the\n *     segment boundary.\n *   * The largest timestamp that the mapper can store is:\n *         LARGEST_RELATIVE_TIMESTAMP = (1 << TIMESTAMP_BITS) - LUCENE_TIMESTAMP_BUFFER.\n *     Tweets created after (segmentBoundaryTimestamp + LARGEST_RELATIVE_TIMESTAMP) are stored as if\n *     their timestamp was (segmentBoundaryTimestamp + LARGEST_RELATIVE_TIMESTAMP).\n *   * When a tweet is added, we compute its doc ID as:\n *         int relativeTimestamp = tweetTimestamp - segmentBoundaryTimestamp;\n *         int docIdTimestamp = LARGEST_RELATIVE_TIMESTAMP - relativeTimestamp;\n *         int numTweetsForTimestamp = tweetsPerTimestamp.get(docIdTimestamp);\n *         int docId = (docIdTimestamp << DOC_ID_BITS)\n *             + MAX_DOCS_PER_TIMESTAMP - numTweetsForTimestamp - 1\n *\n * This doc ID distribution scheme guarantees that tweets created later will be assigned smaller doc\n * IDs (as long as we don't have more than 16 tweets created in the same millisecond). However,\n * there is no ordering guarantee for tweets created at the same timestamp -- they are assigned doc\n * IDs in the order in which they're added to the mapper.\n *\n * If we have more than 16 tweets created at time T, the mapper will still gracefully handle that\n * case: the \"extra\" tweets will be assigned doc IDs from the pool of doc IDs for timestamp (T + 1).\n * However, the ordering guarantee might no longer hold for those \"extra\" tweets. Also, the \"extra\"\n * tweets might be missed by certain since_id/max_id queries (the findDocIdBound() method might not\n * be able to correctly work for these tweet IDs).\n */\npublic class OutOfOrderRealtimeTweetIDMapper extends TweetIDMapper {\n  private static final Logger LOG = LoggerFactory.getLogger(OutOfOrderRealtimeTweetIDMapper.class);\n\n  // The number of bits used to represent the tweet timestamp.\n  private static final int TIMESTAMP_BITS = 27;\n\n  // The number of bits used to represent the number of tweets with a certain timestamp.\n  @VisibleForTesting\n  static final int DOC_ID_BITS = Integer.SIZE - TIMESTAMP_BITS - 1;\n\n  // The maximum number of tweets/docs that we can store per timestamp.\n  @VisibleForTesting\n  static final int MAX_DOCS_PER_TIMESTAMP = 1 << DOC_ID_BITS;\n\n  // Lucene has some logic that doesn't deal well with doc IDs close to Integer.MAX_VALUE.\n  // For example, BooleanScorer has a SIZE constant set to 2048, which gets added to the doc IDs\n  // inside the score() method. So when the doc IDs are close to Integer.MAX_VALUE, this causes an\n  // overflow, which can send Lucene into an infinite loop. Therefore, we need to make sure that\n  // we do not assign doc IDs close to Integer.MAX_VALUE.\n  private static final int LUCENE_TIMESTAMP_BUFFER = 1 << 16;\n\n  @VisibleForTesting\n  public static final int LATE_TWEETS_TIME_BUFFER_MILLIS = 12 * 3600 * 1000;  // 12 hours\n\n  // The largest relative timestamp that this mapper can store.\n  @VisibleForTesting\n  static final int LARGEST_RELATIVE_TIMESTAMP = (1 << TIMESTAMP_BITS) - LUCENE_TIMESTAMP_BUFFER;\n\n  private final long segmentBoundaryTimestamp;\n  private final int segmentSize;\n\n  private final Int2LongOpenHashMap tweetIds;\n  private final Int2ByteOpenHashMap tweetsPerTimestamp;\n\n  private static final SearchRateCounter BAD_BUCKET_RATE =\n      SearchRateCounter.export(\"tweets_assigned_to_bad_timestamp_bucket\");\n  private static final SearchRateCounter TWEETS_NOT_ASSIGNED_RATE =\n      SearchRateCounter.export(\"tweets_not_assigned\");\n  private static final SearchRateCounter OLD_TWEETS_DROPPED =\n      SearchRateCounter.export(\"old_tweets_dropped\");\n\n  public OutOfOrderRealtimeTweetIDMapper(int segmentSize, long timesliceID) {\n    long firstTimestamp = SnowflakeIdParser.getTimestampFromTweetId(timesliceID);\n    // Leave a buffer so that we can handle tweets that are up to twelve hours late.\n    this.segmentBoundaryTimestamp = firstTimestamp - LATE_TWEETS_TIME_BUFFER_MILLIS;\n    this.segmentSize = segmentSize;\n\n    tweetIds = new Int2LongOpenHashMap(segmentSize);\n    tweetIds.defaultReturnValue(ID_NOT_FOUND);\n\n    tweetsPerTimestamp = new Int2ByteOpenHashMap(segmentSize);\n    tweetsPerTimestamp.defaultReturnValue((byte) ID_NOT_FOUND);\n  }\n\n  @VisibleForTesting\n  int getDocIdTimestamp(long tweetId) {\n    long tweetTimestamp = SnowflakeIdParser.getTimestampFromTweetId(tweetId);\n    if (tweetTimestamp < segmentBoundaryTimestamp) {\n      return ID_NOT_FOUND;\n    }\n\n    long relativeTimestamp = tweetTimestamp - segmentBoundaryTimestamp;\n    if (relativeTimestamp > LARGEST_RELATIVE_TIMESTAMP) {\n      relativeTimestamp = LARGEST_RELATIVE_TIMESTAMP;\n    }\n\n    return LARGEST_RELATIVE_TIMESTAMP - (int) relativeTimestamp;\n  }\n\n  private int getDocIdForTimestamp(int docIdTimestamp, byte docIndexInTimestamp) {\n    return (docIdTimestamp << DOC_ID_BITS) + MAX_DOCS_PER_TIMESTAMP - docIndexInTimestamp;\n  }\n\n  @VisibleForTesting\n  long[] getTweetsForDocIdTimestamp(int docIdTimestamp) {\n    byte numDocsForTimestamp = tweetsPerTimestamp.get(docIdTimestamp);\n    if (numDocsForTimestamp == ID_NOT_FOUND) {\n      // This should never happen in prod, but better to be safe.\n      return new long[0];\n    }\n\n    long[] tweetIdsInBucket = new long[numDocsForTimestamp];\n    int startingDocId = (docIdTimestamp << DOC_ID_BITS) + MAX_DOCS_PER_TIMESTAMP - 1;\n    for (int i = 0; i < numDocsForTimestamp; ++i) {\n      tweetIdsInBucket[i] = tweetIds.get(startingDocId - i);\n    }\n    return tweetIdsInBucket;\n  }\n\n  private int newDocId(long tweetId) {\n    int expectedDocIdTimestamp = getDocIdTimestamp(tweetId);\n    if (expectedDocIdTimestamp == ID_NOT_FOUND) {\n      LOG.info(\"Dropping tweet {} because it is from before the segment boundary timestamp {}\",\n          tweetId,\n          segmentBoundaryTimestamp);\n      OLD_TWEETS_DROPPED.increment();\n      return ID_NOT_FOUND;\n    }\n\n    int docIdTimestamp = expectedDocIdTimestamp;\n    byte numDocsForTimestamp = tweetsPerTimestamp.get(docIdTimestamp);\n\n    if (numDocsForTimestamp == MAX_DOCS_PER_TIMESTAMP) {\n      BAD_BUCKET_RATE.increment();\n    }\n\n    while ((docIdTimestamp > 0) && (numDocsForTimestamp == MAX_DOCS_PER_TIMESTAMP)) {\n      --docIdTimestamp;\n      numDocsForTimestamp = tweetsPerTimestamp.get(docIdTimestamp);\n    }\n\n    if (numDocsForTimestamp == MAX_DOCS_PER_TIMESTAMP) {\n      // The relative timestamp 0 already has MAX_DOCS_PER_TIMESTAMP. Can't add more docs.\n      LOG.error(\"Tweet {} could not be assigned a doc ID in any bucket, because the bucket for \"\n          + \"timestamp 0 is already full: {}\",\n          tweetId, Arrays.toString(getTweetsForDocIdTimestamp(0)));\n      TWEETS_NOT_ASSIGNED_RATE.increment();\n      return ID_NOT_FOUND;\n    }\n\n    if (docIdTimestamp != expectedDocIdTimestamp) {\n      LOG.warn(\"Tweet {} could not be assigned a doc ID in the bucket for its timestamp {}, \"\n               + \"because this bucket is full. Instead, it was assigned a doc ID in the bucket for \"\n               + \"timestamp {}. The tweets in the correct bucket are: {}\",\n               tweetId,\n               expectedDocIdTimestamp,\n               docIdTimestamp,\n               Arrays.toString(getTweetsForDocIdTimestamp(expectedDocIdTimestamp)));\n    }\n\n    if (numDocsForTimestamp == ID_NOT_FOUND) {\n      numDocsForTimestamp = 0;\n    }\n    ++numDocsForTimestamp;\n    tweetsPerTimestamp.put(docIdTimestamp, numDocsForTimestamp);\n\n    return getDocIdForTimestamp(docIdTimestamp, numDocsForTimestamp);\n  }\n\n  @Override\n  public int getDocID(long tweetId) {\n    int docIdTimestamp = getDocIdTimestamp(tweetId);\n    while (docIdTimestamp >= 0) {\n      int numDocsForTimestamp = tweetsPerTimestamp.get(docIdTimestamp);\n      int startingDocId = (docIdTimestamp << DOC_ID_BITS) + MAX_DOCS_PER_TIMESTAMP - 1;\n      for (int docId = startingDocId; docId > startingDocId - numDocsForTimestamp; --docId) {\n        if (tweetIds.get(docId) == tweetId) {\n          return docId;\n        }\n      }\n\n      // If we have MAX_DOCS_PER_TIMESTAMP docs with this timestamp, then we might've mis-assigned\n      // a tweet to the previous docIdTimestamp bucket. In that case, we need to keep searching.\n      // Otherwise, the tweet is not in the index.\n      if (numDocsForTimestamp < MAX_DOCS_PER_TIMESTAMP) {\n        break;\n      }\n\n      --docIdTimestamp;\n    }\n\n    return ID_NOT_FOUND;\n  }\n\n  @Override\n  protected int getNextDocIDInternal(int docId) {\n    // Check if docId + 1 is an assigned doc ID in this mapper. This might be the case when we have\n    // multiple tweets posted on the same millisecond.\n    if (tweetIds.get(docId + 1) != ID_NOT_FOUND) {\n      return docId + 1;\n    }\n\n    // If (docId + 1) is not assigned, then it means we do not have any more tweets posted at the\n    // timestamp corresponding to docId. We need to find the next relative timestamp for which this\n    // mapper has tweets, and return the first tweet for that timestamp. Note that iterating over\n    // the space of all possible timestamps is faster than iterating over the space of all possible\n    // doc IDs (it's MAX_DOCS_PER_TIMESTAMP times faster).\n    int nextDocIdTimestamp = (docId >> DOC_ID_BITS) + 1;\n    byte numDocsForTimestamp = tweetsPerTimestamp.get(nextDocIdTimestamp);\n    int maxDocIdTimestamp = getMaxDocID() >> DOC_ID_BITS;\n    while ((nextDocIdTimestamp <= maxDocIdTimestamp)\n           && (numDocsForTimestamp == ID_NOT_FOUND)) {\n      ++nextDocIdTimestamp;\n      numDocsForTimestamp = tweetsPerTimestamp.get(nextDocIdTimestamp);\n    }\n\n    if (numDocsForTimestamp != ID_NOT_FOUND) {\n      return getDocIdForTimestamp(nextDocIdTimestamp, numDocsForTimestamp);\n    }\n\n    return ID_NOT_FOUND;\n  }\n\n  @Override\n  protected int getPreviousDocIDInternal(int docId) {\n    // Check if docId - 1 is an assigned doc ID in this mapper. This might be the case when we have\n    // multiple tweets posted on the same millisecond.\n    if (tweetIds.get(docId - 1) != ID_NOT_FOUND) {\n      return docId - 1;\n    }\n\n    // If (docId - 1) is not assigned, then it means we do not have any more tweets posted at the\n    // timestamp corresponding to docId. We need to find the previous relative timestamp for which\n    // this mapper has tweets, and return the first tweet for that timestamp. Note that iterating\n    // over the space of all possible timestamps is faster than iterating over the space of all\n    // possible doc IDs (it's MAX_DOCS_PER_TIMESTAMP times faster).\n    int previousDocIdTimestamp = (docId >> DOC_ID_BITS) - 1;\n    byte numDocsForTimestamp = tweetsPerTimestamp.get(previousDocIdTimestamp);\n    int minDocIdTimestamp = getMinDocID() >> DOC_ID_BITS;\n    while ((previousDocIdTimestamp >= minDocIdTimestamp)\n           && (numDocsForTimestamp == ID_NOT_FOUND)) {\n      --previousDocIdTimestamp;\n      numDocsForTimestamp = tweetsPerTimestamp.get(previousDocIdTimestamp);\n    }\n\n    if (numDocsForTimestamp != ID_NOT_FOUND) {\n      return getDocIdForTimestamp(previousDocIdTimestamp, (byte) 1);\n    }\n\n    return ID_NOT_FOUND;\n  }\n\n  @Override\n  public long getTweetID(int docId) {\n    return tweetIds.get(docId);\n  }\n\n  @Override\n  protected int addMappingInternal(long tweetId) {\n    int docId = newDocId(tweetId);\n    if (docId == ID_NOT_FOUND) {\n      return ID_NOT_FOUND;\n    }\n\n    tweetIds.put(docId, tweetId);\n    return docId;\n  }\n\n  @Override\n  protected int findDocIDBoundInternal(long tweetId, boolean findMaxDocId) {\n    // Note that it would be incorrect to lookup the doc ID for the given tweet ID and return that\n    // doc ID, as we would skip over tweets created in the same millisecond but with a lower doc ID.\n    int docIdTimestamp = getDocIdTimestamp(tweetId);\n\n    // The docIdTimestamp is ID_NOT_FOUND only if the tweet is from before the segment boundary and\n    // this should never happen here because TweetIDMapper.findDocIdBound ensures that the tweet id\n    // passed into this method is >= minTweetID which means the tweet is from after the segment\n    // boundary.\n    Preconditions.checkState(\n        docIdTimestamp != ID_NOT_FOUND,\n        \"Tried to find doc id bound for tweet %d which is from before the segment boundary %d\",\n        tweetId,\n        segmentBoundaryTimestamp);\n\n    // It's OK to return a doc ID that doesn't correspond to any tweet ID in the index,\n    // as the doc ID is simply used as a starting point and ending point for range queries,\n    // not a source of truth.\n    if (findMaxDocId) {\n      // Return the largest possible doc ID for the timestamp.\n      return getDocIdForTimestamp(docIdTimestamp, (byte) 1);\n    } else {\n      // Return the smallest possible doc ID for the timestamp.\n      byte tweetsInTimestamp = tweetsPerTimestamp.getOrDefault(docIdTimestamp, (byte) 0);\n      return getDocIdForTimestamp(docIdTimestamp, tweetsInTimestamp);\n    }\n  }\n\n  /**\n   * Returns the array of all tweet IDs stored in this mapper in a sorted (descending) order.\n   * Essentially, this method remaps all tweet IDs stored in this mapper to a compressed doc ID\n   * space of [0, numDocs).\n   *\n   * Note that this method is not thread safe, and it's meant to be called only at segment\n   * optimization time. If addMappingInternal() is called during the execution of this method,\n   * the behavior is undefined (it will most likely return bad results or throw an exception).\n   *\n   * @return An array of all tweet IDs stored in this mapper, in a sorted (descending) order.\n   */\n  public long[] sortTweetIds() {\n    int numDocs = getNumDocs();\n    if (numDocs == 0) {\n      return new long[0];\n    }\n\n    // Add all tweets stored in this mapper to sortTweetIds.\n    long[] sortedTweetIds = new long[numDocs];\n    int sortedTweetIdsIndex = 0;\n    for (int docId = getMinDocID(); docId != ID_NOT_FOUND; docId = getNextDocID(docId)) {\n      sortedTweetIds[sortedTweetIdsIndex++] = getTweetID(docId);\n    }\n    Preconditions.checkState(sortedTweetIdsIndex == numDocs,\n                             \"Could not traverse all documents in the mapper. Expected to find \"\n                             + numDocs + \" docs, but found only \" + sortedTweetIdsIndex);\n\n    // Sort sortedTweetIdsIndex in descending order. There's no way to sort a primitive array in\n    // descending order, so we have to sort it in ascending order and then reverse it.\n    Arrays.sort(sortedTweetIds);\n    for (int i = 0; i < numDocs / 2; ++i) {\n      long tmp = sortedTweetIds[i];\n      sortedTweetIds[i] = sortedTweetIds[numDocs - 1 - i];\n      sortedTweetIds[numDocs - 1 - i] = tmp;\n    }\n\n    return sortedTweetIds;\n  }\n\n  @Override\n  public DocIDToTweetIDMapper optimize() throws IOException {\n    return new OptimizedTweetIDMapper(this);\n  }\n\n  /**\n   * Returns the largest Tweet ID that this doc ID mapper could handle. The returned Tweet ID\n   * would be safe to put into the mapper, but any larger ones would not be correctly handled.\n   */\n  public static long calculateMaxTweetID(long timesliceID) {\n    long numberOfUsableTimestamps = LARGEST_RELATIVE_TIMESTAMP - LATE_TWEETS_TIME_BUFFER_MILLIS;\n    long firstTimestamp = SnowflakeIdParser.getTimestampFromTweetId(timesliceID);\n    long lastTimestamp = firstTimestamp + numberOfUsableTimestamps;\n    return SnowflakeIdParser.generateValidStatusId(\n        lastTimestamp, SnowflakeIdParser.RESERVED_BITS_MASK);\n  }\n\n  /**\n   * Evaluates whether two instances of OutOfOrderRealtimeTweetIDMapper are equal by value. It is\n   * slow because it has to check every tweet ID/doc ID in the map.\n   */\n  @VisibleForTesting\n  boolean verySlowEqualsForTests(OutOfOrderRealtimeTweetIDMapper that) {\n    return getMinTweetID() == that.getMinTweetID()\n        && getMaxTweetID() == that.getMaxTweetID()\n        && getMinDocID() == that.getMinDocID()\n        && getMaxDocID() == that.getMaxDocID()\n        && segmentBoundaryTimestamp == that.segmentBoundaryTimestamp\n        && segmentSize == that.segmentSize\n        && tweetsPerTimestamp.equals(that.tweetsPerTimestamp)\n        && tweetIds.equals(that.tweetIds);\n  }\n\n  @Override\n  public OutOfOrderRealtimeTweetIDMapper.FlushHandler getFlushHandler() {\n    return new OutOfOrderRealtimeTweetIDMapper.FlushHandler(this);\n  }\n\n  private OutOfOrderRealtimeTweetIDMapper(\n    long minTweetID,\n    long maxTweetID,\n    int minDocID,\n    int maxDocID,\n    long segmentBoundaryTimestamp,\n    int segmentSize,\n    int[] docIDs,\n    long[] tweetIDList\n  ) {\n    super(minTweetID, maxTweetID, minDocID, maxDocID, docIDs.length);\n\n    Preconditions.checkState(docIDs.length == tweetIDList.length);\n\n    this.segmentBoundaryTimestamp = segmentBoundaryTimestamp;\n    this.segmentSize = segmentSize;\n\n    tweetIds = new Int2LongOpenHashMap(segmentSize);\n    tweetIds.defaultReturnValue(ID_NOT_FOUND);\n\n    tweetsPerTimestamp = new Int2ByteOpenHashMap(segmentSize);\n    tweetsPerTimestamp.defaultReturnValue((byte) ID_NOT_FOUND);\n\n    for (int i = 0; i < docIDs.length; i++) {\n      int docID = docIDs[i];\n      long tweetID = tweetIDList[i];\n      tweetIds.put(docID, tweetID);\n\n      int timestampBucket = docID >> DOC_ID_BITS;\n      if (tweetsPerTimestamp.containsKey(timestampBucket)) {\n        tweetsPerTimestamp.addTo(timestampBucket, (byte) 1);\n      } else {\n        tweetsPerTimestamp.put(timestampBucket, (byte) 1);\n      }\n    }\n  }\n\n  public static class FlushHandler extends Flushable.Handler<OutOfOrderRealtimeTweetIDMapper> {\n    private static final String MIN_TWEET_ID_PROP_NAME = \"MinTweetID\";\n    private static final String MAX_TWEET_ID_PROP_NAME = \"MaxTweetID\";\n    private static final String MIN_DOC_ID_PROP_NAME = \"MinDocID\";\n    private static final String MAX_DOC_ID_PROP_NAME = \"MaxDocID\";\n    private static final String SEGMENT_BOUNDARY_TIMESTAMP_PROP_NAME = \"SegmentBoundaryTimestamp\";\n    private static final String SEGMENT_SIZE_PROP_NAME = \"SegmentSize\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(OutOfOrderRealtimeTweetIDMapper objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer serializer) throws IOException {\n      OutOfOrderRealtimeTweetIDMapper mapper = getObjectToFlush();\n\n      flushInfo.addLongProperty(MIN_TWEET_ID_PROP_NAME, mapper.getMinTweetID());\n      flushInfo.addLongProperty(MAX_TWEET_ID_PROP_NAME, mapper.getMaxTweetID());\n      flushInfo.addIntProperty(MIN_DOC_ID_PROP_NAME, mapper.getMinDocID());\n      flushInfo.addIntProperty(MAX_DOC_ID_PROP_NAME, mapper.getMaxDocID());\n      flushInfo.addLongProperty(SEGMENT_BOUNDARY_TIMESTAMP_PROP_NAME,\n          mapper.segmentBoundaryTimestamp);\n      flushInfo.addIntProperty(SEGMENT_SIZE_PROP_NAME, mapper.segmentSize);\n\n      serializer.writeInt(mapper.tweetIds.size());\n      for (Int2LongMap.Entry entry : mapper.tweetIds.int2LongEntrySet()) {\n        serializer.writeInt(entry.getIntKey());\n        serializer.writeLong(entry.getLongValue());\n      }\n    }\n\n    @Override\n    protected OutOfOrderRealtimeTweetIDMapper doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n\n      int size = in.readInt();\n      int[] docIds = new int[size];\n      long[] tweetIds = new long[size];\n      for (int i = 0; i < size; i++) {\n        docIds[i] = in.readInt();\n        tweetIds[i] = in.readLong();\n      }\n\n      return new OutOfOrderRealtimeTweetIDMapper(\n          flushInfo.getLongProperty(MIN_TWEET_ID_PROP_NAME),\n          flushInfo.getLongProperty(MAX_TWEET_ID_PROP_NAME),\n          flushInfo.getIntProperty(MIN_DOC_ID_PROP_NAME),\n          flushInfo.getIntProperty(MAX_DOC_ID_PROP_NAME),\n          flushInfo.getLongProperty(SEGMENT_BOUNDARY_TIMESTAMP_PROP_NAME),\n          flushInfo.getIntProperty(SEGMENT_SIZE_PROP_NAME),\n          docIds,\n          tweetIds);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/RealtimeTimeMapper.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.TimeMapper;\nimport com.twitter.search.core.earlybird.index.inverted.IntBlockPool;\n\nimport it.unimi.dsi.fastutil.ints.Int2IntMap;\nimport it.unimi.dsi.fastutil.ints.Int2IntOpenHashMap;\n\n/**\n * Maps 32-bit document IDs to seconds-since-epoch timestamps.\n */\npublic class RealtimeTimeMapper extends AbstractInMemoryTimeMapper {\n  // Doc id to timestamp map. Timestamps that are negative are out-of-order.\n  protected final Int2IntOpenHashMap timeMap;\n  private final int capacity;\n\n  public RealtimeTimeMapper(int capacity) {\n    super();\n    this.capacity = capacity;\n\n    timeMap = new Int2IntOpenHashMap(capacity);\n    timeMap.defaultReturnValue(ILLEGAL_TIME);\n  }\n\n  @Override\n  public int getTime(int docID) {\n    return timeMap.get(docID);\n  }\n\n  @Override\n  protected void setTime(int docID, int timeSeconds) {\n    timeMap.put(docID, timeSeconds);\n  }\n\n  public final void addMapping(int docID, int timeSeconds) {\n    doAddMapping(docID, timeSeconds);\n  }\n\n  @Override\n  public TimeMapper optimize(DocIDToTweetIDMapper originalTweetIdMapper,\n                             DocIDToTweetIDMapper optimizedTweetIdMapper) throws IOException {\n    return new OptimizedTimeMapper(this, originalTweetIdMapper, optimizedTweetIdMapper);\n  }\n\n  /**\n   * Evaluates whether two instances of RealtimeTimeMapper are equal by value. It is\n   * slow because it has to check every tweet ID/timestamp in the map.\n   */\n  @VisibleForTesting\n  boolean verySlowEqualsForTests(RealtimeTimeMapper that) {\n    return reverseMapLastIndex == that.reverseMapLastIndex\n        && reverseMapIds.verySlowEqualsForTests(that.reverseMapIds)\n        && reverseMapTimes.verySlowEqualsForTests(that.reverseMapTimes)\n        && capacity == that.capacity\n        && timeMap.equals(that.timeMap);\n  }\n\n  private RealtimeTimeMapper(\n      int capacity,\n      int reverseMapLastIndex,\n      int[] docIds,\n      int[] timestamps,\n      IntBlockPool reverseMapTimes,\n      IntBlockPool reverseMapIds\n  ) {\n    super(reverseMapLastIndex, reverseMapTimes, reverseMapIds);\n\n    this.capacity = capacity;\n\n    timeMap = new Int2IntOpenHashMap(capacity);\n    timeMap.defaultReturnValue(ILLEGAL_TIME);\n\n    Preconditions.checkState(docIds.length == timestamps.length);\n\n    for (int i = 0; i < docIds.length; i++) {\n      timeMap.put(docIds[i], timestamps[i]);\n    }\n  }\n\n  @Override\n  public RealtimeTimeMapper.FlushHandler getFlushHandler() {\n    return new RealtimeTimeMapper.FlushHandler(this);\n  }\n\n  public static class FlushHandler extends Flushable.Handler<RealtimeTimeMapper> {\n    private static final String REVERSE_MAP_LAST_INDEX_PROP = \"reverseMapLastIndex\";\n    private static final String TIMES_SUB_PROP = \"times\";\n    private static final String IDS_SUB_PROP = \"ids\";\n    private static final String CAPACITY_PROP = \"capacity\";\n\n    public FlushHandler() {\n      super();\n    }\n\n    public FlushHandler(RealtimeTimeMapper objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer serializer) throws IOException {\n      RealtimeTimeMapper mapper = getObjectToFlush();\n\n      flushInfo.addIntProperty(CAPACITY_PROP, mapper.capacity);\n      flushInfo.addIntProperty(REVERSE_MAP_LAST_INDEX_PROP, mapper.reverseMapLastIndex);\n\n      serializer.writeInt(mapper.timeMap.size());\n      for (Int2IntMap.Entry entry : mapper.timeMap.int2IntEntrySet()) {\n        serializer.writeInt(entry.getIntKey());\n        serializer.writeInt(entry.getIntValue());\n      }\n\n      mapper.reverseMapTimes.getFlushHandler().flush(\n          flushInfo.newSubProperties(TIMES_SUB_PROP), serializer);\n      mapper.reverseMapIds.getFlushHandler().flush(\n          flushInfo.newSubProperties(IDS_SUB_PROP), serializer);\n    }\n\n    @Override\n    protected RealtimeTimeMapper doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n\n      int size = in.readInt();\n      int[] docIds = new int[size];\n      int[] timestamps = new int[size];\n      for (int i = 0; i < size; i++) {\n        docIds[i] = in.readInt();\n        timestamps[i] = in.readInt();\n      }\n\n      return new RealtimeTimeMapper(\n          flushInfo.getIntProperty(CAPACITY_PROP),\n          flushInfo.getIntProperty(REVERSE_MAP_LAST_INDEX_PROP),\n          docIds,\n          timestamps,\n          new IntBlockPool.FlushHandler().load(flushInfo.getSubProperties(TIMES_SUB_PROP), in),\n          new IntBlockPool.FlushHandler().load(flushInfo.getSubProperties(IDS_SUB_PROP), in));\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/TimeMappingWriter.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.util.AttributeSource;\n\nimport com.twitter.search.common.util.analysis.IntTermAttribute;\nimport com.twitter.search.core.earlybird.index.EarlybirdRealtimeIndexSegmentWriter;\n\npublic class TimeMappingWriter implements EarlybirdRealtimeIndexSegmentWriter.InvertedDocConsumer {\n  private IntTermAttribute termAtt;\n  private final RealtimeTimeMapper mapper;\n\n  public TimeMappingWriter(RealtimeTimeMapper mapper) {\n    this.mapper = mapper;\n  }\n\n  @Override\n  public final void start(AttributeSource attributeSource, boolean currentDocIsOffensive) {\n    termAtt = attributeSource.addAttribute(IntTermAttribute.class);\n  }\n\n  @Override\n  public final void add(int docId, int position) throws IOException {\n    final int timeSec = termAtt.getTerm();\n    mapper.addMapping(docId, timeSec);\n  }\n\n  @Override\n  public void finish() {\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/TweetIDMapper.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\npublic abstract class TweetIDMapper implements DocIDToTweetIDMapper, Flushable {\n  private long minTweetID;\n  private long maxTweetID;\n  private int minDocID;\n  private int maxDocID;\n  private int numDocs;\n\n  protected TweetIDMapper() {\n    this(Long.MAX_VALUE, Long.MIN_VALUE, Integer.MAX_VALUE, Integer.MIN_VALUE, 0);\n  }\n\n  protected TweetIDMapper(\n      long minTweetID, long maxTweetID, int minDocID, int maxDocID, int numDocs) {\n    this.minTweetID = minTweetID;\n    this.maxTweetID = maxTweetID;\n    this.minDocID = minDocID;\n    this.maxDocID = maxDocID;\n    this.numDocs = numDocs;\n  }\n\n  // Realtime updates minTweetID and maxTweetID in addMapping.\n  // Archives updates minTweetID and maxTweetID in prepareToRead.\n  protected void setMinTweetID(long minTweetID) {\n    this.minTweetID = minTweetID;\n  }\n\n  protected void setMaxTweetID(long maxTweetID) {\n    this.maxTweetID = maxTweetID;\n  }\n\n  protected void setMinDocID(int minDocID) {\n    this.minDocID = minDocID;\n  }\n\n  protected void setMaxDocID(int maxDocID) {\n    this.maxDocID = maxDocID;\n  }\n\n  protected void setNumDocs(int numDocs) {\n    this.numDocs = numDocs;\n  }\n\n  public long getMinTweetID() {\n    return this.minTweetID;\n  }\n\n  public long getMaxTweetID() {\n    return this.maxTweetID;\n  }\n\n  public int getMinDocID() {\n    return minDocID;\n  }\n\n  public int getMaxDocID() {\n    return maxDocID;\n  }\n\n  @Override\n  public int getNumDocs() {\n    return numDocs;\n  }\n\n  /**\n   * Given a tweetId, find the corresponding doc ID to start, or end, a search.\n   *\n   * In the ordered, dense doc ID mappers, this returns either the doc ID assigned to the tweet ID,\n   * or doc ID of the next lowest tweet ID, if the tweet is not in the index. In this case\n   * findMaxDocID is ignored.\n   *\n   * In {@link OutOfOrderRealtimeTweetIDMapper}, doc IDs are not ordered within a millisecond, so we\n   * want to search the entire millisecond bucket for a filter. To accomplish this,\n   * if findMaxDocId is true we return the largest possible doc ID for that millisecond.\n   * If findMaxDocId is false, we return the smallest possible doc ID for that millisecond.\n   *\n   * The returned doc ID will be between smallestDocID and largestDocID (inclusive).\n   * The returned doc ID may not be in the index.\n   */\n  public int findDocIdBound(long tweetID,\n                            boolean findMaxDocID,\n                            int smallestDocID,\n                            int largestDocID) throws IOException {\n    if (tweetID > maxTweetID) {\n      return smallestDocID;\n    }\n    if (tweetID < minTweetID) {\n      return largestDocID;\n    }\n\n    int internalID = findDocIDBoundInternal(tweetID, findMaxDocID);\n\n    return Math.max(smallestDocID, Math.min(largestDocID, internalID));\n  }\n\n  @Override\n  public final int getNextDocID(int docID) {\n    if (numDocs <= 0) {\n      return ID_NOT_FOUND;\n    }\n    if (docID < minDocID) {\n      return minDocID;\n    }\n    if (docID >= maxDocID) {\n      return ID_NOT_FOUND;\n    }\n    return getNextDocIDInternal(docID);\n  }\n\n  @Override\n  public final int getPreviousDocID(int docID) {\n    if (numDocs <= 0) {\n      return ID_NOT_FOUND;\n    }\n    if (docID <= minDocID) {\n      return ID_NOT_FOUND;\n    }\n    if (docID > maxDocID) {\n      return maxDocID;\n    }\n    return getPreviousDocIDInternal(docID);\n  }\n\n  @Override\n  public int addMapping(final long tweetID) {\n    int docId = addMappingInternal(tweetID);\n    if (docId != ID_NOT_FOUND) {\n      ++numDocs;\n      if (tweetID > maxTweetID) {\n        maxTweetID = tweetID;\n      }\n      if (tweetID < minTweetID) {\n        minTweetID = tweetID;\n      }\n      if (docId > maxDocID) {\n        maxDocID = docId;\n      }\n      if (docId < minDocID) {\n        minDocID = docId;\n      }\n    }\n\n    return docId;\n  }\n\n  /**\n   * Returns the smallest valid doc ID in this mapper that's strictly higher than the given doc ID.\n   * If no such doc ID exists, ID_NOT_FOUND must be returned.\n   *\n   * The given docID is guaranteed to be in the range [minDocID, maxDocID).\n   *\n   * @param docID The current doc ID.\n   * @return The smallest valid doc ID in this mapper that's strictly higher than the given doc ID,\n   *         or a negative number, if no such doc ID exists.\n   */\n  protected abstract int getNextDocIDInternal(int docID);\n\n  /**\n   * Returns the smallest valid doc ID in this mapper that's strictly higher than the given doc ID.\n   * If no such doc ID exists, ID_NOT_FOUND must be returned.\n   *\n   * The given docID is guaranteed to be in the range (minDocID, maxDocID].\n   *\n   * @param docID The current doc ID.\n   * @return The smallest valid doc ID in this mapper that's strictly higher than the given doc ID,\n   *         or a negative number, if no such doc ID exists.\n   */\n  protected abstract int getPreviousDocIDInternal(int docID);\n\n  protected abstract int addMappingInternal(final long tweetID);\n\n  /**\n   * See {@link TweetIDMapper#findDocIdBound}.\n   */\n  protected abstract int findDocIDBoundInternal(long tweetID,\n                                                boolean findMaxDocID) throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/TweetIDQuery.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\nimport java.util.Arrays;\nimport java.util.Set;\n\nimport com.google.common.collect.Sets;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.query.DefaultFilterWeight;\nimport com.twitter.search.common.search.IntArrayDocIdSetIterator;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentData;\n\npublic class TweetIDQuery extends Query {\n  private final Set<Long> tweetIDs = Sets.newHashSet();\n\n  public TweetIDQuery(long... tweetIDs) {\n    for (long tweetID : tweetIDs) {\n      this.tweetIDs.add(tweetID);\n    }\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n    return new DefaultFilterWeight(this) {\n      @Override\n      protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context) throws IOException {\n        EarlybirdIndexSegmentData segmentData =\n            ((EarlybirdIndexSegmentAtomicReader) context.reader()).getSegmentData();\n        DocIDToTweetIDMapper docIdToTweetIdMapper = segmentData.getDocIDToTweetIDMapper();\n\n        Set<Integer> set = Sets.newHashSet();\n        for (long tweetID : tweetIDs) {\n          int docID = docIdToTweetIdMapper.getDocID(tweetID);\n          if (docID != DocIDToTweetIDMapper.ID_NOT_FOUND) {\n            set.add(docID);\n          }\n        }\n\n        if (set.isEmpty()) {\n          return DocIdSetIterator.empty();\n        }\n\n        int[] docIDs = new int[set.size()];\n        int i = 0;\n        for (int docID : set) {\n          docIDs[i++] = docID;\n        }\n        Arrays.sort(docIDs);\n        return new IntArrayDocIdSetIterator(docIDs);\n      }\n    };\n  }\n\n  @Override\n  public int hashCode() {\n    return tweetIDs.hashCode();\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof TweetIDQuery)) {\n      return false;\n    }\n\n    return tweetIDs.equals(TweetIDQuery.class.cast(obj).tweetIDs);\n  }\n\n  @Override\n  public String toString(String field) {\n    return \"TWEET_ID_QUERY: \" + tweetIDs;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/TweetIDToInternalIDMap.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\nimport java.util.Arrays;\n\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.Flushable;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\n\npublic final class TweetIDToInternalIDMap implements Flushable {\n  private final int   size;\n  private final int[] hash;\n  public final int   halfSize;\n  private final int   mask;\n  public int         numMappings;\n\n  static final int PRIME_NUMBER = 37;\n\n  // For FlushHandler.load() use only\n  private TweetIDToInternalIDMap(final int[] hash,\n                                 final int numMappings) {\n    this.hash        = hash;\n    this.size        = hash.length;\n    this.halfSize    = size >> 1;\n    this.mask        = size - 1;\n    this.numMappings = numMappings;\n  }\n\n  TweetIDToInternalIDMap(final int size) {\n    this.hash = new int[size];\n    Arrays.fill(hash, DocIDToTweetIDMapper.ID_NOT_FOUND);\n    this.size = size;\n    this.halfSize = size >> 1;\n    this.mask = size - 1;\n    this.numMappings = 0;\n  }\n\n  // Slightly different hash function from the one used to partition tweets to Earlybirds.\n  protected static int hashCode(final long tweetID) {\n    long timestamp = SnowflakeIdParser.getTimestampFromTweetId(tweetID);\n    int code = (int) ((timestamp - 1) ^ (timestamp >>> 32));\n    code = PRIME_NUMBER * (int) (tweetID & SnowflakeIdParser.RESERVED_BITS_MASK) + code;\n    return code;\n  }\n\n  protected static int incrementHashCode(int code) {\n    return ((code >> 8) + code) | 1;\n  }\n\n  private int hashPos(int code) {\n    return code & mask;\n  }\n\n  /**\n   * Associates the given tweet ID with the given internal doc ID.\n   *\n   * @param tweetID The tweet ID.\n   * @param internalID The doc ID that should be associated with this tweet ID.\n   * @param inverseMap The map that stores the doc ID to tweet ID associations.\n   */\n  public void add(final long tweetID, final int internalID, final long[] inverseMap) {\n    int code = hashCode(tweetID);\n    int hashPos = hashPos(code);\n    int value = hash[hashPos];\n    assert inverseMap[internalID] == tweetID;\n\n    if (value != DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      final int inc = incrementHashCode(code);\n      do {\n        code += inc;\n        hashPos = hashPos(code);\n        value = hash[hashPos];\n      } while (value != DocIDToTweetIDMapper.ID_NOT_FOUND);\n    }\n\n    assert value == DocIDToTweetIDMapper.ID_NOT_FOUND;\n\n    hash[hashPos] = internalID;\n    numMappings++;\n  }\n\n  /**\n   * Returns the doc ID corresponding to the given tweet ID.\n   *\n   * @param tweetID The tweet ID.\n   * @param inverseMap The map that stores the doc ID to tweet ID associations.\n   * @return The doc ID corresponding to the given tweet ID.\n   */\n  public int get(long tweetID, final long[] inverseMap) {\n    int code = hashCode(tweetID);\n    int hashPos = hashPos(code);\n    int value = hash[hashPos];\n\n    if (value != DocIDToTweetIDMapper.ID_NOT_FOUND && inverseMap[value] != tweetID) {\n      final int inc = incrementHashCode(code);\n\n      do {\n        code += inc;\n        hashPos = hashPos(code);\n        value = hash[hashPos];\n      } while (value != DocIDToTweetIDMapper.ID_NOT_FOUND && inverseMap[value] != tweetID);\n    }\n\n    if (hashPos == -1) {\n      return DocIDToTweetIDMapper.ID_NOT_FOUND;\n    }\n    return hash[hashPos];\n  }\n\n  @Override\n  public TweetIDToInternalIDMap.FlushHandler getFlushHandler() {\n    return new FlushHandler(this);\n  }\n\n  public static final class FlushHandler extends Flushable.Handler<TweetIDToInternalIDMap> {\n    public FlushHandler() {\n      super();\n    }\n\n    private static final String HASH_ARRAY_SIZE_PROP_NAME = \"HashArraySize\";\n    private static final String MASK_PROP_NAME = \"Mask\";\n    private static final String NUM_MAPPINGS_PROP_NAME = \"NumMappings\";\n\n    public FlushHandler(TweetIDToInternalIDMap objectToFlush) {\n      super(objectToFlush);\n    }\n\n    @Override\n    protected void doFlush(FlushInfo flushInfo, DataSerializer out)\n      throws IOException {\n      TweetIDToInternalIDMap mapper = getObjectToFlush();\n\n      flushInfo\n          .addIntProperty(HASH_ARRAY_SIZE_PROP_NAME, mapper.hash.length)\n          .addIntProperty(MASK_PROP_NAME, mapper.mask)\n          .addIntProperty(NUM_MAPPINGS_PROP_NAME, mapper.numMappings);\n\n      out.writeIntArray(mapper.hash);\n    }\n\n    @Override\n    protected TweetIDToInternalIDMap doLoad(FlushInfo flushInfo, DataDeserializer in)\n        throws IOException {\n      final int[] hash = in.readIntArray();\n\n      final int numMappings = flushInfo.getIntProperty(NUM_MAPPINGS_PROP_NAME);\n\n      return new TweetIDToInternalIDMap(hash, numMappings);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/TweetSearchIndexExtensionsFactory.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdIndexExtensionsData;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdIndexExtensionsFactory;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdRealtimeIndexExtensionsData;\n\npublic class TweetSearchIndexExtensionsFactory extends EarlybirdIndexExtensionsFactory {\n  @Override\n  public EarlybirdRealtimeIndexExtensionsData newRealtimeIndexExtensionsData() {\n    return new TweetSearchRealtimeIndexExtensionsData();\n  }\n\n  @Override\n  public EarlybirdIndexExtensionsData newLuceneIndexExtensionsData() {\n    return new TweetSearchLuceneIndexExtensionsData();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/TweetSearchLuceneIndexExtensionsData.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.schema.base.EarlybirdFieldType;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentData;\nimport com.twitter.search.core.earlybird.index.column.ColumnStrideFieldIndex;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdIndexExtensionsData;\n\npublic class TweetSearchLuceneIndexExtensionsData implements EarlybirdIndexExtensionsData {\n  @Override\n  public void setupExtensions(EarlybirdIndexSegmentAtomicReader atomicReader) throws IOException {\n    // If we use stock lucene to back the mappers and column stride fields,\n    // we need to initialize them\n    EarlybirdIndexSegmentData segmentData = atomicReader.getSegmentData();\n    DocValuesBasedTweetIDMapper tweetIDMapper =\n        (DocValuesBasedTweetIDMapper) segmentData.getDocIDToTweetIDMapper();\n    tweetIDMapper.initializeWithLuceneReader(\n        atomicReader,\n        getColumnStrideFieldIndex(segmentData, EarlybirdFieldConstant.ID_CSF_FIELD));\n\n    DocValuesBasedTimeMapper timeMapper =\n        (DocValuesBasedTimeMapper) segmentData.getTimeMapper();\n    timeMapper.initializeWithLuceneReader(\n        atomicReader,\n        getColumnStrideFieldIndex(segmentData, EarlybirdFieldConstant.CREATED_AT_CSF_FIELD));\n  }\n\n  private ColumnStrideFieldIndex getColumnStrideFieldIndex(\n      EarlybirdIndexSegmentData segmentData, EarlybirdFieldConstant csfField) {\n    String csfFieldName = csfField.getFieldName();\n    EarlybirdFieldType fieldType =\n        segmentData.getSchema().getFieldInfo(csfFieldName).getFieldType();\n    Preconditions.checkState(fieldType.isCsfLoadIntoRam());\n    return segmentData.getDocValuesManager().addColumnStrideField(csfFieldName, fieldType);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/TweetSearchRealtimeIndexExtensionsData.java",
    "content": "package com.twitter.search.earlybird.index;\n\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.EarlybirdRealtimeIndexSegmentWriter.InvertedDocConsumerBuilder;\nimport com.twitter.search.core.earlybird.index.EarlybirdRealtimeIndexSegmentWriter.StoredFieldsConsumerBuilder;\nimport com.twitter.search.core.earlybird.index.extensions.EarlybirdRealtimeIndexExtensionsData;\n\npublic class TweetSearchRealtimeIndexExtensionsData\n    implements EarlybirdRealtimeIndexExtensionsData {\n  @Override\n  public void createStoredFieldsConsumer(StoredFieldsConsumerBuilder builder) {\n    // no extensions necessary here\n  }\n\n  @Override\n  public void createInvertedDocConsumer(InvertedDocConsumerBuilder builder) {\n    if (EarlybirdFieldConstant.ID_FIELD.getFieldName().equals(builder.getFieldName())) {\n      // The tweet ID should've already been added to the tweet ID <-> doc ID mapper.\n      builder.setUseDefaultConsumer(false);\n    }\n\n    if (EarlybirdFieldConstant.CREATED_AT_FIELD.getFieldName().equals(builder.getFieldName())) {\n      RealtimeTimeMapper timeMapper = (RealtimeTimeMapper) builder.getSegmentData().getTimeMapper();\n      builder.addConsumer(new TimeMappingWriter(timeMapper));\n      builder.setUseDefaultConsumer(false);\n    }\n  }\n\n  @Override\n  public void setupExtensions(EarlybirdIndexSegmentAtomicReader atomicReader) {\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/facets/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-common\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-analyzers-smartcn\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-facet\",\n        \"3rdparty/jvm/org/apache/lucene:lucene-queries\",\n        \"src/java/com/twitter/search/common/constants\",\n        \"src/java/com/twitter/search/common/schema/base\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/core/earlybird\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/index/facets/FacetSkipList.java",
    "content": "package com.twitter.search.earlybird.index.facets;\n\nimport java.io.IOException;\nimport java.util.HashSet;\nimport java.util.Iterator;\nimport java.util.Set;\n\nimport org.apache.lucene.analysis.TokenStream;\nimport org.apache.lucene.analysis.tokenattributes.CharTermAttribute;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.TermQuery;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.facets.FacetCountState;\nimport com.twitter.search.earlybird.thrift.ThriftTermRequest;\n\npublic abstract class FacetSkipList {\n  public static class SkipTokenStream extends TokenStream {\n    private CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);\n\n    private Iterator<Schema.FieldInfo> iterator;\n    private Set<Schema.FieldInfo> facetFields = new HashSet<>();\n\n    public void add(Schema.FieldInfo field) {\n      this.facetFields.add(field);\n    }\n\n    @Override\n    public final boolean incrementToken() throws IOException {\n      if (iterator == null) {\n        iterator = facetFields.iterator();\n      }\n\n      while (iterator.hasNext()) {\n        Schema.FieldInfo field = iterator.next();\n        if (field.getFieldType().isStoreFacetSkiplist()) {\n          termAtt.setEmpty();\n          termAtt.append(EarlybirdFieldConstant.getFacetSkipFieldName(field.getName()));\n\n          return true;\n        }\n      }\n\n      return false;\n    }\n  }\n\n  /**\n   * Returns a Term query to search in the given facet field.\n   */\n  public static Term getSkipListTerm(Schema.FieldInfo facetField) {\n    if (facetField.getFieldType().isStoreFacetSkiplist()) {\n      return new Term(EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n                      EarlybirdFieldConstant.getFacetSkipFieldName(facetField.getName()));\n    }\n    return null;\n  }\n\n  /**\n   * Returns a disjunction query that searches in all facet fields in the given facet count state.\n   */\n  public static Query getSkipListQuery(FacetCountState facetCountState) {\n    Set<Schema.FieldInfo> fieldsWithSkipLists =\n        facetCountState.getFacetFieldsToCountWithSkipLists();\n\n    if (fieldsWithSkipLists == null || fieldsWithSkipLists.isEmpty()) {\n      return null;\n    }\n\n    Query skipLists;\n\n    if (fieldsWithSkipLists.size() == 1) {\n      skipLists = new TermQuery(getSkipListTerm(fieldsWithSkipLists.iterator().next()));\n    } else {\n      BooleanQuery.Builder disjunctionBuilder = new BooleanQuery.Builder();\n      for (Schema.FieldInfo facetField : fieldsWithSkipLists) {\n        disjunctionBuilder.add(\n            new TermQuery(new Term(\n                EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n                EarlybirdFieldConstant.getFacetSkipFieldName(facetField.getName()))),\n            BooleanClause.Occur.SHOULD);\n      }\n      skipLists = disjunctionBuilder.build();\n    }\n\n    return skipLists;\n  }\n\n  /**\n   * Returns a term request that can be used to get term statistics for the skip list term\n   * associated with the provided facet. Returns null, if this FacetField is configured to not\n   * store a skiplist.\n   */\n  public static ThriftTermRequest getSkipListTermRequest(Schema schema, String facetName) {\n    return getSkipListTermRequest(schema.getFacetFieldByFacetName(facetName));\n  }\n\n  /**\n   * Returns a term request that can be used to get term statistics for the skip list term\n   * associated with the provided facet. Returns null, if this FacetField is configured to not\n   * store a skiplist.\n   */\n  public static ThriftTermRequest getSkipListTermRequest(Schema.FieldInfo facetField) {\n    return facetField != null && facetField.getFieldType().isStoreFacetSkiplist()\n           ? new ThriftTermRequest(\n                EarlybirdFieldConstant.getFacetSkipFieldName(facetField.getName()))\n             .setFieldName(EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName())\n           : null;\n  }\n\n  /**\n   * Returns a term request using the specified fieldName. This is only a temporary solution until\n   * Blender can access the Schema to pass the FacetIDMap into the method above.\n   *\n   * @deprecated Temporary solution until Blender\n   */\n  @Deprecated\n  public static ThriftTermRequest getSkipListTermRequest(String fieldName) {\n    return new ThriftTermRequest(EarlybirdFieldConstant.getFacetSkipFieldName(fieldName))\n        .setFieldName(EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/ml/ScoringModelsManager.java",
    "content": "package com.twitter.search.earlybird.ml;\n\nimport java.io.IOException;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Optional;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.file.AbstractFile;\nimport com.twitter.search.common.file.FileUtils;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.schema.DynamicSchema;\nimport com.twitter.search.common.util.ml.prediction_engine.CompositeFeatureContext;\nimport com.twitter.search.common.util.ml.prediction_engine.LightweightLinearModel;\nimport com.twitter.search.common.util.ml.prediction_engine.ModelLoader;\n\nimport static com.twitter.search.modeling.tweet_ranking.TweetScoringFeatures.CONTEXT;\nimport static com.twitter.search.modeling.tweet_ranking.TweetScoringFeatures.FeatureContextVersion.CURRENT_VERSION;\n\n/**\n * Loads the scoring models for tweets and provides access to them.\n *\n * This class relies on a list of ModelLoader objects to retrieve the objects from them. It will\n * return the first model found according to the order in the list.\n *\n * For production, we load models from 2 sources: classpath and HDFS. If a model is available\n * from HDFS, we return it, otherwise we use the model from the classpath.\n *\n * The models used for default requests (i.e. not experiments) MUST be present in the\n * classpath, this allows us to avoid errors if they can't be loaded from HDFS.\n * Models for experiments can live only in HDFS, so we don't need to redeploy Earlybird if we\n * want to test them.\n */\npublic class ScoringModelsManager {\n\n  private static final Logger LOG = LoggerFactory.getLogger(ScoringModelsManager.class);\n\n  /**\n   * Used when\n   * 1. Testing\n   * 2. The scoring models are disabled in the config\n   * 3. Exceptions thrown during loading the scoring models\n   */\n  public static final ScoringModelsManager NO_OP_MANAGER = new ScoringModelsManager() {\n    @Override\n    public boolean isEnabled() {\n      return false;\n    }\n  };\n\n  private final ModelLoader[] loaders;\n  private final DynamicSchema dynamicSchema;\n\n  public ScoringModelsManager(ModelLoader... loaders) {\n    this.loaders = loaders;\n    this.dynamicSchema = null;\n  }\n\n  public ScoringModelsManager(DynamicSchema dynamicSchema, ModelLoader... loaders) {\n    this.loaders = loaders;\n    this.dynamicSchema = dynamicSchema;\n  }\n\n  /**\n   * Indicates that the scoring models were enabled in the config and were loaded successfully\n   */\n  public boolean isEnabled() {\n    return true;\n  }\n\n  public void reload() {\n    for (ModelLoader loader : loaders) {\n      loader.run();\n    }\n  }\n\n  /**\n   * Loads and returns the model with the given name, if one exists.\n   */\n  public Optional<LightweightLinearModel> getModel(String modelName) {\n    for (ModelLoader loader : loaders) {\n      Optional<LightweightLinearModel> model = loader.getModel(modelName);\n      if (model.isPresent()) {\n        return model;\n      }\n    }\n    return Optional.absent();\n  }\n\n  /**\n   * Creates an instance that loads models first from HDFS and the classpath resources.\n   *\n   * If the models are not found in HDFS, it uses the models from the classpath as fallback.\n   */\n  public static ScoringModelsManager create(\n      SearchStatsReceiver serverStats,\n      String hdfsNameNode,\n      String hdfsBasedPath,\n      DynamicSchema dynamicSchema) throws IOException {\n    // Create a composite feature context so we can load both legacy and schema-based models\n    CompositeFeatureContext featureContext = new CompositeFeatureContext(\n        CONTEXT, dynamicSchema::getSearchFeatureSchema);\n    ModelLoader hdfsLoader = createHdfsLoader(\n        serverStats, hdfsNameNode, hdfsBasedPath, featureContext);\n    ModelLoader classpathLoader = createClasspathLoader(\n        serverStats, featureContext);\n\n    // Explicitly load the models from the classpath\n    classpathLoader.run();\n\n    ScoringModelsManager manager = new ScoringModelsManager(hdfsLoader, classpathLoader);\n    LOG.info(\"Initialized ScoringModelsManager for loading models from HDFS and the classpath\");\n    return manager;\n  }\n\n  protected static ModelLoader createHdfsLoader(\n      SearchStatsReceiver serverStats,\n      String hdfsNameNode,\n      String hdfsBasedPath,\n      CompositeFeatureContext featureContext) {\n    String hdfsVersionedPath = hdfsBasedPath + \"/\" + CURRENT_VERSION.getVersionDirectory();\n    LOG.info(\"Starting to load scoring models from HDFS: {}:{}\",\n        hdfsNameNode, hdfsVersionedPath);\n    return ModelLoader.forHdfsDirectory(\n        hdfsNameNode,\n        hdfsVersionedPath,\n        featureContext,\n        \"scoring_models_hdfs_\",\n        serverStats);\n  }\n\n  /**\n   * Creates a loader that loads models from a default location in the classpath.\n   */\n  @VisibleForTesting\n  public static ModelLoader createClasspathLoader(\n      SearchStatsReceiver serverStats, CompositeFeatureContext featureContext)\n      throws IOException {\n    AbstractFile defaultModelsBaseDir = FileUtils.getTmpDirHandle(\n        ScoringModelsManager.class,\n        \"/com/twitter/search/earlybird/ml/default_models\");\n    AbstractFile defaultModelsDir = defaultModelsBaseDir.getChild(\n        CURRENT_VERSION.getVersionDirectory());\n\n    LOG.info(\"Starting to load scoring models from the classpath: {}\",\n        defaultModelsDir.getPath());\n    return ModelLoader.forDirectory(\n        defaultModelsDir,\n        featureContext,\n        \"scoring_models_classpath_\",\n        serverStats);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/AudioSpaceEventsStreamIndexer.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.kafka.clients.consumer.ConsumerRecord;\nimport org.apache.kafka.clients.consumer.KafkaConsumer;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.earlybird.exception.MissingKafkaTopicException;\nimport com.twitter.ubs.thriftjava.AudioSpaceBaseEvent;\nimport com.twitter.ubs.thriftjava.AudioSpaceEvent;\nimport com.twitter.util.Duration;\n\n/**\n *\n * An example publish event looks like this:\n *  <AudioBaseSpaceEvent space_publish_event:SpacePublishEvent(\n *    time_stamp_millis:1616430926899,\n *    user_id:123456,\n *    broadcast_id:123456789)>\n */\npublic class AudioSpaceEventsStreamIndexer extends SimpleStreamIndexer<Long, AudioSpaceBaseEvent> {\n  private static final Logger LOG =  LoggerFactory.getLogger(AudioSpaceEventsStreamIndexer.class);\n\n  private static final String AUDIO_SPACE_EVENTS_TOPIC = \"audio_space_events_v1\";\n\n  @VisibleForTesting\n  // We use this to filter out old space publish events so as to avoid the risk of processing\n  // old space publish events whose corresponding finish events are no longer in the stream.\n  // It's unlikely that spaces would last longer than this constant so it should be safe to assume\n  // that the space whose publish event is older than this age is finished.\n  protected static final long MAX_PUBLISH_EVENTS_AGE_MS =\n      Duration.fromHours(11).inMillis();\n\n  private final AudioSpaceTable audioSpaceTable;\n  private final Clock clock;\n\n  public AudioSpaceEventsStreamIndexer(\n      KafkaConsumer<Long, AudioSpaceBaseEvent> kafkaConsumer,\n      AudioSpaceTable audioSpaceTable,\n      Clock clock) throws MissingKafkaTopicException {\n    super(kafkaConsumer, AUDIO_SPACE_EVENTS_TOPIC);\n    this.audioSpaceTable = audioSpaceTable;\n    this.clock = clock;\n  }\n\n  @Override\n  protected void validateAndIndexRecord(ConsumerRecord<Long, AudioSpaceBaseEvent> record) {\n    AudioSpaceBaseEvent baseEvent = record.value();\n\n    if (baseEvent != null && baseEvent.isSetBroadcast_id() && baseEvent.isSetEvent_metadata()) {\n      AudioSpaceEvent event = baseEvent.getEvent_metadata();\n      String spaceId = baseEvent.getBroadcast_id();\n      if (event != null && event.isSet(AudioSpaceEvent._Fields.SPACE_PUBLISH_EVENT)) {\n        long publishEventAgeMs = clock.nowMillis() - baseEvent.getTime_stamp_millis();\n        if (publishEventAgeMs < MAX_PUBLISH_EVENTS_AGE_MS) {\n          audioSpaceTable.audioSpaceStarts(spaceId);\n        }\n      } else if (event != null && event.isSet(AudioSpaceEvent._Fields.SPACE_END_EVENT)) {\n        audioSpaceTable.audioSpaceFinishes(spaceId);\n      }\n    }\n  }\n\n  @VisibleForTesting\n  public AudioSpaceTable getAudioSpaceTable() {\n    return audioSpaceTable;\n  }\n\n  void printSummary() {\n    LOG.info(audioSpaceTable.toString());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/AudioSpaceTable.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.util.ArrayDeque;\nimport java.util.Queue;\nimport java.util.Set;\nimport java.util.concurrent.ConcurrentSkipListSet;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchCustomGauge;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.util.Duration;\nimport com.twitter.util.Time;\n\npublic class AudioSpaceTable {\n  private static final String STATS_PREFIX = \"audio_space_\";\n  private static final Duration AUDIO_EVENT_EXPIRATION_DURATION =\n      Duration.fromHours(12);\n\n  private final Set<String> startedSpaces;\n  private final Set<String> finishedSpaces;\n  /**\n   * timestampedSpaceEvents contains both start and finish events.\n   * This is to aid in the case in which we receive only on or the other for a spaceId -- start or finish\n   * without doing this, we could potentially never purge from the sets.\n   */\n  private final Queue<Pair<Time, String>> timestampedSpaceEvents;\n  private final Clock clock;\n\n  private final SearchRateCounter audioSpaceStarts =\n      SearchRateCounter.export(STATS_PREFIX + \"stream_starts\");\n  private final SearchRateCounter audioSpaceFinishes =\n      SearchRateCounter.export(STATS_PREFIX + \"stream_finishes\");\n  private final SearchRateCounter isRunningCalls =\n      SearchRateCounter.export(STATS_PREFIX + \"is_running_calls\");\n  private final SearchRateCounter audioSpaceDuplicateStarts =\n      SearchRateCounter.export(STATS_PREFIX + \"duplicate_start_events\");\n  private final SearchRateCounter audioSpaceDuplicateFinishes =\n      SearchRateCounter.export(STATS_PREFIX + \"duplicate_finish_events\");\n  private final SearchRateCounter startsProcessedAfterCorrespondingFinishes =\n      SearchRateCounter.export(STATS_PREFIX + \"starts_processed_after_corresponding_finishes\");\n  private final SearchRateCounter finishesProcessedWithoutCorrespondingStarts =\n      SearchRateCounter.export(STATS_PREFIX + \"finishes_processed_without_corresponding_starts\");\n\n  public AudioSpaceTable(Clock clock) {\n    // We read and write from different threads, so we need a thread-safe set implementation.\n    startedSpaces = new ConcurrentSkipListSet<>();\n    finishedSpaces = new ConcurrentSkipListSet<>();\n    timestampedSpaceEvents = new ArrayDeque<>();\n    this.clock = clock;\n    SearchCustomGauge.export(STATS_PREFIX + \"live\", this::getNumberOfLiveAudioSpaces);\n    SearchCustomGauge.export(STATS_PREFIX + \"retained_starts\", startedSpaces::size);\n    SearchCustomGauge.export(STATS_PREFIX + \"retained_finishes\", finishedSpaces::size);\n  }\n\n  private int getNumberOfLiveAudioSpaces() {\n    // This call is a bit expensive, but I logged it and it's getting called once a minute, at\n    // the beginning of the minute, so it's fine.\n    int count = 0;\n    for (String startedSpace : startedSpaces) {\n      count += finishedSpaces.contains(startedSpace) ? 0 : 1;\n    }\n    return count;\n  }\n\n  /**\n   * We keep spaces that have started in the last 12 hours.\n   * This is called on every start space event received, and cleans up\n   * the retained spaces so memory usage does not become too high\n   */\n  private void purgeOldSpaces() {\n    Pair<Time, String> oldest = timestampedSpaceEvents.peek();\n    Time now = Time.fromMilliseconds(clock.nowMillis());\n    while (oldest != null) {\n      Duration durationSinceInsert = now.minus(oldest.getFirst());\n      if (durationSinceInsert.compareTo(AUDIO_EVENT_EXPIRATION_DURATION) > 0) {\n        // This event has expired, so we purge it and move on to the next.\n        String oldSpaceId = oldest.getSecond();\n        startedSpaces.remove(oldSpaceId);\n        finishedSpaces.remove(oldSpaceId);\n        oldest = timestampedSpaceEvents.poll();\n      } else {\n        // Oldest event is not old enough so quit purging\n        break;\n      }\n    }\n  }\n\n  /**\n  * Record AudioSpace start event\n   */\n  public void audioSpaceStarts(String spaceId) {\n    audioSpaceStarts.increment();\n    boolean spaceSeenBefore = !startedSpaces.add(spaceId);\n    if (spaceSeenBefore) {\n      audioSpaceDuplicateStarts.increment();\n    }\n\n    if (finishedSpaces.contains(spaceId)) {\n      startsProcessedAfterCorrespondingFinishes.increment();\n    }\n\n    timestampedSpaceEvents.add(new Pair(Time.fromMilliseconds(clock.nowMillis()), spaceId));\n    purgeOldSpaces();\n  }\n\n  /**\n   * Record AudioSpace finish event\n   */\n  public void audioSpaceFinishes(String spaceId) {\n    audioSpaceFinishes.increment();\n    boolean spaceSeenBefore = !finishedSpaces.add(spaceId);\n    if (spaceSeenBefore) {\n      audioSpaceDuplicateFinishes.increment();\n    }\n\n    if (!startedSpaces.contains(spaceId)) {\n      finishesProcessedWithoutCorrespondingStarts.increment();\n    }\n\n    timestampedSpaceEvents.add(new Pair(Time.fromMilliseconds(clock.nowMillis()), spaceId));\n    purgeOldSpaces();\n  }\n\n  public boolean isRunning(String spaceId) {\n    isRunningCalls.increment();\n    return startedSpaces.contains(spaceId) && !finishedSpaces.contains(spaceId);\n  }\n\n  /**\n   * Print stats on this AudioSpaceTable\n   * @return Stats string\n   */\n  public String toString() {\n    return \"AudioSpaceTable: Starts: \" + audioSpaceStarts.getCounter().get()\n        + \", Finishes: \" + audioSpaceFinishes.getCounter().get()\n        + \", Retained starts: \" + startedSpaces.size()\n        + \", Retained finishes: \" + finishedSpaces.size()\n        + \", Currently live: \" + getNumberOfLiveAudioSpaces();\n  }\n\n  public Set<String> getStartedSpaces() {\n    return startedSpaces;\n  }\n\n  public Set<String> getFinishedSpaces() {\n    return finishedSpaces;\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/BalancingKafkaConsumer.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.time.Duration;\nimport java.util.Arrays;\nimport java.util.Collections;\n\nimport org.apache.kafka.clients.consumer.ConsumerRecord;\nimport org.apache.kafka.clients.consumer.ConsumerRecords;\nimport org.apache.kafka.clients.consumer.KafkaConsumer;\nimport org.apache.kafka.common.TopicPartition;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.metrics.SearchRateCounter;\n\n/**\n * BalancingKafkaConsumer is designed to read from the tweets and updates streams in proportion to\n * the rates that those streams are written to, i.e. both topics should have nearly the same amount\n * of lag. This is important because if one stream gets too far ahead of the other, we could end up\n * in a situation where:\n * 1. If the tweet stream is ahead of the updates stream, we couldn't apply an update because a\n *    segment has been optimized, and one of those fields became frozen.\n * 2. If the updates stream is ahead of the tweet stream, we might drop updates because they are\n *    more than a minute old, but the tweets might still not be indexed.\n *\n * Also see 'Consumption Flow Control' in\n * https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html\n */\npublic class BalancingKafkaConsumer {\n  // If one of the topic-partitions lags the other by more than 10 seconds,\n  // it's worth it to pause the faster one and let the slower one catch up.\n  private static final long BALANCE_THRESHOLD_MS = Duration.ofSeconds(10).toMillis();\n  private final KafkaConsumer<Long, ThriftVersionedEvents> kafkaConsumer;\n  private final TopicPartition tweetTopic;\n  private final TopicPartition updateTopic;\n  private final SearchRateCounter tweetsPaused;\n  private final SearchRateCounter updatesPaused;\n  private final SearchRateCounter resumed;\n\n  private long tweetTimestamp = 0;\n  private long updateTimestamp = 0;\n  private long pausedAt = 0;\n  private boolean paused = false;\n\n  public BalancingKafkaConsumer(\n      KafkaConsumer<Long, ThriftVersionedEvents> kafkaConsumer,\n      TopicPartition tweetTopic,\n      TopicPartition updateTopic\n  ) {\n    this.kafkaConsumer = kafkaConsumer;\n    this.tweetTopic = tweetTopic;\n    this.updateTopic = updateTopic;\n\n    String prefix = \"balancing_kafka_\";\n    String suffix = \"_topic_paused\";\n\n    tweetsPaused = SearchRateCounter.export(prefix + tweetTopic.topic() + suffix);\n    updatesPaused = SearchRateCounter.export(prefix + updateTopic.topic() + suffix);\n    resumed = SearchRateCounter.export(prefix + \"topics_resumed\");\n  }\n\n  /**\n   * Calls poll on the underlying consumer and pauses topics as necessary.\n   */\n  public ConsumerRecords<Long, ThriftVersionedEvents> poll(Duration timeout) {\n    ConsumerRecords<Long, ThriftVersionedEvents> records = kafkaConsumer.poll(timeout);\n    topicFlowControl(records);\n    return records;\n  }\n\n  private void topicFlowControl(ConsumerRecords<Long, ThriftVersionedEvents> records) {\n    for (ConsumerRecord<Long, ThriftVersionedEvents> record : records) {\n      long timestamp = record.timestamp();\n\n      if (updateTopic.topic().equals(record.topic())) {\n        updateTimestamp = Math.max(updateTimestamp, timestamp);\n      } else if (tweetTopic.topic().equals(record.topic())) {\n        tweetTimestamp = Math.max(tweetTimestamp, timestamp);\n      } else {\n        throw new IllegalStateException(\n            \"Unexpected partition \" + record.topic() + \" in BalancingKafkaConsumer\");\n      }\n    }\n\n    if (paused) {\n      // If we paused and one of the streams is still below the pausedAt point, we want to continue\n      // reading from just the lagging stream.\n      if (tweetTimestamp >= pausedAt && updateTimestamp >= pausedAt) {\n        // We caught up, resume reading from both topics.\n        paused = false;\n        kafkaConsumer.resume(Arrays.asList(tweetTopic, updateTopic));\n        resumed.increment();\n      }\n    } else {\n      long difference = Math.abs(tweetTimestamp - updateTimestamp);\n\n      if (difference < BALANCE_THRESHOLD_MS) {\n        // The streams have approximately the same lag, so no need to pause anything.\n        return;\n      }\n      // The difference is too great, one of the streams is lagging behind the other so we need to\n      // pause one topic so the other can catch up.\n      paused = true;\n      pausedAt = Math.max(updateTimestamp, tweetTimestamp);\n      if (tweetTimestamp > updateTimestamp) {\n        kafkaConsumer.pause(Collections.singleton(tweetTopic));\n        tweetsPaused.increment();\n      } else {\n        kafkaConsumer.pause(Collections.singleton(updateTopic));\n        updatesPaused.increment();\n      }\n    }\n  }\n\n  public void close() {\n    kafkaConsumer.close();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/CompleteSegmentManager.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.function.Supplier;\n\nimport com.google.common.collect.Lists;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.util.io.recordreader.RecordReader;\nimport com.twitter.search.common.util.zktrylock.ZooKeeperTryLockFactory;\nimport com.twitter.search.earlybird.EarlybirdStatus;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.document.TweetDocument;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.segment.SegmentDataProvider;\n\n/**\n * CompleteSegmentManager is used to parallelize indexing of complete (not partial) segments\n * on startup.  It also populates the fields used by the PartitionManager.\n */\npublic class CompleteSegmentManager {\n  private static final Logger LOG = LoggerFactory.getLogger(CompleteSegmentManager.class);\n\n  private static final String INDEX_COMPLETED_SEGMENTS =\n      \"indexing, optimizing and flushing complete segments\";\n  private static final String LOAD_COMPLETED_SEGMENTS = \"loading complete segments\";\n  private static final String INDEX_UPDATES_FOR_COMPLETED_SEGMENTS =\n      \"indexing updates for complete segments\";\n  private static final String BUILD_MULTI_SEGMENT_TERM_DICT =\n      \"build multi segment term dictionaries\";\n\n  // Max number of segments being loaded / indexed concurrently.\n  private final int maxConcurrentSegmentIndexers =\n      EarlybirdProperty.MAX_CONCURRENT_SEGMENT_INDEXERS.get(3);\n\n  // The state we are building.\n  protected final SegmentDataProvider segmentDataProvider;\n  private final InstrumentedQueue<ThriftVersionedEvents> retryQueue;\n\n  private final UserUpdatesStreamIndexer userUpdatesStreamIndexer;\n  private final UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer;\n\n  private final SegmentManager segmentManager;\n  private final ZooKeeperTryLockFactory zkTryLockFactory;\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n  private final Clock clock;\n  private MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager;\n  private final SegmentSyncConfig segmentSyncConfig;\n\n  private final CriticalExceptionHandler criticalExceptionHandler;\n\n  private boolean interrupted = false;\n\n  public CompleteSegmentManager(\n      ZooKeeperTryLockFactory zooKeeperTryLockFactory,\n      SegmentDataProvider segmentDataProvider,\n      UserUpdatesStreamIndexer userUpdatesStreamIndexer,\n      UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer,\n      SegmentManager segmentManager,\n      InstrumentedQueue<ThriftVersionedEvents> retryQueue,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      Clock clock,\n      MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager,\n      SegmentSyncConfig segmentSyncConfig,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    this.zkTryLockFactory = zooKeeperTryLockFactory;\n    this.segmentDataProvider = segmentDataProvider;\n    this.userUpdatesStreamIndexer = userUpdatesStreamIndexer;\n    this.userScrubGeoEventStreamIndexer = userScrubGeoEventStreamIndexer;\n    this.segmentManager = segmentManager;\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n    this.clock = clock;\n    this.multiSegmentTermDictionaryManager = multiSegmentTermDictionaryManager;\n    this.segmentSyncConfig = segmentSyncConfig;\n    this.retryQueue = retryQueue;\n    this.criticalExceptionHandler = criticalExceptionHandler;\n  }\n\n  /**\n   * Indexes all user events.\n   */\n  public void indexUserEvents() {\n    LOG.info(\"Loading/indexing user events.\");\n    StartupUserEventIndexer startupUserEventIndexer = new StartupUserEventIndexer(\n        searchIndexingMetricSet,\n        userUpdatesStreamIndexer,\n        userScrubGeoEventStreamIndexer,\n        segmentManager,\n        clock\n    );\n\n    startupUserEventIndexer.indexAllEvents();\n    LOG.info(\"Finished loading/indexing user events.\");\n  }\n\n  /**\n   * Loads or indexes from scratch all complete segments.\n   *\n   * @param segmentsToIndexProvider A supplier that provides the list of all complete segments.\n   */\n  public void indexCompleteSegments(\n      Supplier<Iterable<SegmentInfo>> segmentsToIndexProvider) throws Exception {\n    List<Thread> segmentIndexers = Lists.newArrayList();\n\n    EarlybirdStatus.beginEvent(\n        INDEX_COMPLETED_SEGMENTS, searchIndexingMetricSet.startupInIndexCompletedSegments);\n    while (!interrupted && !Thread.currentThread().isInterrupted()) {\n      try {\n        // Get the refreshed list of local segment databases.\n        segmentManager.updateSegments(segmentDataProvider.newSegmentList());\n        Iterator<SegmentInfo> segmentsToIndex = segmentsToIndexProvider.get().iterator();\n\n        // Start up to max concurrent segment indexers.\n        segmentIndexers.clear();\n        while (segmentsToIndex.hasNext() && segmentIndexers.size() < maxConcurrentSegmentIndexers) {\n          SegmentInfo nextSegment = segmentsToIndex.next();\n          if (!nextSegment.isComplete()) {\n            Thread thread = new Thread(new SingleSegmentIndexer(nextSegment),\n                                       \"startup-segment-indexer-\" + nextSegment.getSegmentName());\n            thread.start();\n            segmentIndexers.add(thread);\n          }\n        }\n\n        // No remaining indexer threads, we're done.\n        if (segmentIndexers.size() == 0) {\n          LOG.info(\"Finished indexing complete segments\");\n          EarlybirdStatus.endEvent(\n              INDEX_COMPLETED_SEGMENTS, searchIndexingMetricSet.startupInIndexCompletedSegments);\n          break;\n        }\n\n        // Wait for threads to complete fully.\n        LOG.info(\"Started {} indexing threads\", segmentIndexers.size());\n        for (Thread thread : segmentIndexers) {\n          thread.join();\n        }\n        LOG.info(\"Joined all {} indexing threads\", segmentIndexers.size());\n      } catch (IOException e) {\n        LOG.error(\"IOException in SegmentStartupManager loop\", e);\n      } catch (InterruptedException e) {\n        interrupted = true;\n        LOG.error(\"Interrupted joining segment indexer thread\", e);\n      }\n    }\n  }\n\n  /**\n   * Loads all given complete segments.\n   *\n   * @param completeSegments The list of all complete segments to be loaded.\n   */\n  public void loadCompleteSegments(List<SegmentInfo> completeSegments) throws Exception {\n    if (!interrupted && !Thread.currentThread().isInterrupted()) {\n      LOG.info(\"Starting to load {} complete segments.\", completeSegments.size());\n      EarlybirdStatus.beginEvent(\n          LOAD_COMPLETED_SEGMENTS, searchIndexingMetricSet.startupInLoadCompletedSegments);\n\n      List<Thread> segmentThreads = Lists.newArrayList();\n      List<SegmentInfo> segmentsToBeLoaded = Lists.newArrayList();\n      for (SegmentInfo segmentInfo : completeSegments) {\n        if (segmentInfo.isEnabled()) {\n          segmentsToBeLoaded.add(segmentInfo);\n          Thread segmentLoaderThread = new Thread(\n              () -> new SegmentLoader(segmentSyncConfig, criticalExceptionHandler)\n                  .load(segmentInfo),\n              \"startup-segment-loader-\" + segmentInfo.getSegmentName());\n          segmentThreads.add(segmentLoaderThread);\n          segmentLoaderThread.start();\n        } else {\n          LOG.info(\"Will not load segment {} because it's disabled.\", segmentInfo.getSegmentName());\n        }\n      }\n\n      for (Thread segmentLoaderThread : segmentThreads) {\n        segmentLoaderThread.join();\n      }\n\n      for (SegmentInfo segmentInfo : segmentsToBeLoaded) {\n        if (!segmentInfo.getSyncInfo().isLoaded()) {\n          // Throw an exception if a segment could not be loaded: We do not want earlybirds to\n          // startup with missing segments.\n          throw new RuntimeException(\"Could not load segment \" + segmentInfo.getSegmentName());\n        }\n      }\n\n      LOG.info(\"Loaded all complete segments, starting indexing all updates.\");\n      EarlybirdStatus.beginEvent(\n          INDEX_UPDATES_FOR_COMPLETED_SEGMENTS,\n          searchIndexingMetricSet.startupInIndexUpdatesForCompletedSegments);\n\n      // Index all updates for all complete segments until we're fully caught up.\n      if (!EarlybirdCluster.isArchive(segmentManager.getEarlybirdIndexConfig().getCluster())) {\n        segmentThreads.clear();\n        for (SegmentInfo segmentInfo : completeSegments) {\n          if (segmentInfo.isEnabled()) {\n            Thread segmentUpdatesThread = new Thread(\n                () -> new SimpleUpdateIndexer(\n                    segmentDataProvider.getSegmentDataReaderSet(),\n                    searchIndexingMetricSet,\n                    retryQueue,\n                    criticalExceptionHandler).indexAllUpdates(segmentInfo),\n                \"startup-complete-segment-update-indexer-\" + segmentInfo.getSegmentName());\n            segmentThreads.add(segmentUpdatesThread);\n            segmentUpdatesThread.start();\n          } else {\n            LOG.info(\"Will not index updates for segment {} because it's disabled.\",\n                     segmentInfo.getSegmentName());\n          }\n        }\n\n        for (Thread segmentUpdatesThread : segmentThreads) {\n          segmentUpdatesThread.join();\n        }\n      }\n      LOG.info(\"Indexed updates for all complete segments.\");\n      EarlybirdStatus.endEvent(\n          INDEX_UPDATES_FOR_COMPLETED_SEGMENTS,\n          searchIndexingMetricSet.startupInIndexUpdatesForCompletedSegments);\n\n      EarlybirdStatus.endEvent(\n          LOAD_COMPLETED_SEGMENTS, searchIndexingMetricSet.startupInLoadCompletedSegments);\n    }\n  }\n\n  /**\n   * Builds the term dictionary that spans all earlybird segments. Some fields share the term\n   * dictionary across segments as an optimization.\n   */\n  public void buildMultiSegmentTermDictionary() {\n    EarlybirdStatus.beginEvent(\n        BUILD_MULTI_SEGMENT_TERM_DICT,\n        searchIndexingMetricSet.startupInMultiSegmentTermDictionaryUpdates);\n    if (!interrupted && !Thread.currentThread().isInterrupted()) {\n      LOG.info(\"Building multi segment term dictionaries.\");\n      boolean built = multiSegmentTermDictionaryManager.buildDictionary();\n      LOG.info(\"Done building multi segment term dictionaries, result: {}\", built);\n    }\n    EarlybirdStatus.endEvent(\n        BUILD_MULTI_SEGMENT_TERM_DICT,\n        searchIndexingMetricSet.startupInMultiSegmentTermDictionaryUpdates);\n  }\n\n  /**\n   * Warms up the data in the given segments. The warm up will usually make sure that all necessary\n   * is loaded in RAM and all relevant data structures are created before the segments starts\n   * serving real requests.\n   *\n   * @param segments The list of segments to warm up.\n   */\n  public final void warmSegments(Iterable<SegmentInfo> segments) throws InterruptedException {\n    int threadId = 1;\n    Iterator<SegmentInfo> it = segments.iterator();\n\n    try {\n      List<Thread> segmentWarmers = Lists.newLinkedList();\n      while (it.hasNext()) {\n\n        segmentWarmers.clear();\n        while (it.hasNext() && segmentWarmers.size() < maxConcurrentSegmentIndexers) {\n          final SegmentInfo segment = it.next();\n          Thread t = new Thread(() ->\n            new SegmentWarmer(criticalExceptionHandler).warmSegmentIfNecessary(segment),\n              \"startup-warmer-\" + threadId++);\n\n          t.start();\n          segmentWarmers.add(t);\n        }\n\n        for (Thread t : segmentWarmers) {\n          t.join();\n        }\n      }\n    } catch (InterruptedException e) {\n      LOG.error(\"Interrupted segment warmer thread\", e);\n      Thread.currentThread().interrupt();\n      throw e;\n    }\n  }\n\n  /**\n   * Indexes a complete segment.\n   */\n  private class SingleSegmentIndexer implements Runnable {\n    private final SegmentInfo segmentInfo;\n\n    public SingleSegmentIndexer(SegmentInfo segmentInfo) {\n      this.segmentInfo = segmentInfo;\n    }\n\n    @Override\n    public void run() {\n      // 0) Check if the segment can be loaded. This might copy the segment from HDFS.\n      if (new SegmentLoader(segmentSyncConfig, criticalExceptionHandler)\n          .downloadSegment(segmentInfo)) {\n        LOG.info(\"Will not index segment {} because it was downloaded from HDFS.\",\n                 segmentInfo.getSegmentName());\n        segmentInfo.setComplete(true);\n        return;\n      }\n\n      LOG.info(\"SingleSegmentIndexer starting for segment: \" + segmentInfo);\n\n      // 1) Index all tweets in this segment.\n      RecordReader<TweetDocument> tweetReader;\n      try {\n        tweetReader = segmentDataProvider.getSegmentDataReaderSet().newDocumentReader(segmentInfo);\n        if (tweetReader != null) {\n          tweetReader.setExhaustStream(true);\n        }\n      } catch (Exception e) {\n        throw new RuntimeException(\"Could not create tweet reader for segment: \" + segmentInfo, e);\n      }\n\n      new SimpleSegmentIndexer(tweetReader, searchIndexingMetricSet).indexSegment(segmentInfo);\n\n      if (!segmentInfo.isComplete() || segmentInfo.isIndexing()) {\n        throw new RuntimeException(\"Segment does not appear to be complete: \" + segmentInfo);\n      }\n\n      // 2) Index all updates in this segment (archive earlybirds don't have updates).\n      if (!EarlybirdCluster.isArchive(segmentManager.getEarlybirdIndexConfig().getCluster())) {\n        new SimpleUpdateIndexer(\n            segmentDataProvider.getSegmentDataReaderSet(),\n            searchIndexingMetricSet,\n            retryQueue,\n            criticalExceptionHandler).indexAllUpdates(segmentInfo);\n      }\n\n      // 3) Optimize the segment.\n      SegmentOptimizer.optimize(segmentInfo);\n\n      // 4) Flush to HDFS if necessary.\n      new SegmentHdfsFlusher(zkTryLockFactory, segmentSyncConfig)\n          .flushSegmentToDiskAndHDFS(segmentInfo);\n\n      // 5) Unload the segment from memory.\n      segmentInfo.getIndexSegment().close();\n    }\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/DynamicPartitionConfig.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\n\n/**\n * Keeps track of an up-to-date PartitionConfig. The PartitionConfig may be periodically reloaded\n * from ZooKeeper. If you need a consistent view of the current partition configuration, make sure\n * to grab a reference to a single PartitionConfig using getCurrentPartitionConfig() and reuse that\n * object.\n */\npublic class DynamicPartitionConfig {\n  private static final Logger LOG = LoggerFactory.getLogger(DynamicPartitionConfig.class);\n  private static final SearchCounter FAILED_UPDATE_COUNTER_NAME =\n      SearchCounter.export(\"dynamic_partition_config_failed_update\");\n  private static final SearchCounter SUCCESSFUL_UPDATE_COUNTER =\n      SearchCounter.export(\"dynamic_partition_config_successful_update\");\n  // We assume that DynamicPartitionConfig is practically a singleton in Earlybird app.\n  private static final SearchLongGauge NUM_REPLICAS_IN_HASH_PARTITION =\n      SearchLongGauge.export(\"dynamic_partition_config_num_replicas_in_hash_partition\");\n\n  private final PartitionConfig curPartitionConfig;\n\n  public DynamicPartitionConfig(PartitionConfig initialConfig) {\n    this.curPartitionConfig = initialConfig;\n    NUM_REPLICAS_IN_HASH_PARTITION.set(initialConfig.getNumReplicasInHashPartition());\n  }\n\n  public PartitionConfig getCurrentPartitionConfig() {\n    return curPartitionConfig;\n  }\n\n  /**\n   * Verifies that the new partition config is compatible with the old one, and if it is, updates\n   * the number of replicas per partition based on the new partition config.\n   */\n  public void setCurrentPartitionConfig(PartitionConfig partitionConfig) {\n    Preconditions.checkNotNull(partitionConfig);\n    // For now, we only allow the number of replicas in this partition to be dynamically updated.\n    // Ensure that the only things that have changed between the previous\n    if (curPartitionConfig.getClusterName().equals(partitionConfig.getClusterName())\n        && (curPartitionConfig.getMaxEnabledLocalSegments()\n            == partitionConfig.getMaxEnabledLocalSegments())\n        && (curPartitionConfig.getNumPartitions() == partitionConfig.getNumPartitions())\n        && (curPartitionConfig.getTierStartDate().equals(partitionConfig.getTierStartDate()))\n        && (curPartitionConfig.getTierEndDate().equals(partitionConfig.getTierEndDate()))\n        && (curPartitionConfig.getTierName().equals(partitionConfig.getTierName()))) {\n\n      if (curPartitionConfig.getNumReplicasInHashPartition()\n          != partitionConfig.getNumReplicasInHashPartition()) {\n        SUCCESSFUL_UPDATE_COUNTER.increment();\n        curPartitionConfig.setNumReplicasInHashPartition(\n            partitionConfig.getNumReplicasInHashPartition());\n        NUM_REPLICAS_IN_HASH_PARTITION.set(partitionConfig.getNumReplicasInHashPartition());\n      }\n    } else {\n      FAILED_UPDATE_COUNTER_NAME.increment();\n      LOG.warn(\n          \"Attempted to update partition config with inconsistent layout.\\n\"\n          + \"Current: \" + curPartitionConfig.getPartitionConfigDescription() + \"\\n\"\n          + \"New: \" + partitionConfig.getPartitionConfigDescription());\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/EarlybirdIndex.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.List;\n\npublic class EarlybirdIndex {\n  private final List<SegmentInfo> segmentInfoList;\n\n  public static final int MAX_NUM_OF_NON_OPTIMIZED_SEGMENTS = 2;\n\n  // The Kafka offsets for the tweet create stream and the tweet update stream. Indexing should\n  // start from these offsets when it resumes.\n  private final long tweetOffset;\n  private final long updateOffset;\n  private final long maxIndexedTweetId;\n\n  public EarlybirdIndex(\n      List<SegmentInfo> segmentInfoList,\n      long tweetOffset,\n      long updateOffset,\n      long maxIndexedTweetId\n  ) {\n    List<SegmentInfo> segmentInfos = new ArrayList<>(segmentInfoList);\n    Collections.sort(segmentInfos);\n    this.segmentInfoList = segmentInfos;\n    this.tweetOffset = tweetOffset;\n    this.updateOffset = updateOffset;\n    this.maxIndexedTweetId = maxIndexedTweetId;\n  }\n\n  public EarlybirdIndex(List<SegmentInfo> segmentInfoList, long tweetOffset, long updateOffset) {\n    this(segmentInfoList, tweetOffset, updateOffset, -1);\n  }\n\n  public List<SegmentInfo> getSegmentInfoList() {\n    return segmentInfoList;\n  }\n\n  public long getTweetOffset() {\n    return tweetOffset;\n  }\n\n  public long getUpdateOffset() {\n    return updateOffset;\n  }\n\n  public long getMaxIndexedTweetId() {\n    return maxIndexedTweetId;\n  }\n\n  /**\n   * Returns the number of non-optimized segments in this index.\n   * @return the number of non-optimized segments in this index.\n   */\n  public int numOfNonOptimizedSegments() {\n    int numNonOptimized = 0;\n    for (SegmentInfo segmentInfo : segmentInfoList) {\n      if (!segmentInfo.isOptimized()) {\n        numNonOptimized++;\n      }\n    }\n    return numNonOptimized;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/EarlybirdIndexFlusher.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.OutputStreamWriter;\nimport java.text.DateFormat;\nimport java.text.ParseException;\nimport java.text.SimpleDateFormat;\nimport java.time.Duration;\nimport java.util.ArrayList;\nimport java.util.Date;\nimport java.util.SortedMap;\nimport java.util.TreeMap;\nimport java.util.concurrent.TimeoutException;\n\nimport scala.runtime.BoxedUnit;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.commons.compress.utils.Lists;\nimport org.apache.commons.lang.RandomStringUtils;\nimport org.apache.hadoop.fs.FSDataOutputStream;\nimport org.apache.hadoop.fs.FileStatus;\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.Path;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.schema.earlybird.FlushVersion;\nimport com.twitter.search.common.util.io.flushable.DataSerializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.earlybird.common.NonPagingAssert;\nimport com.twitter.search.earlybird.util.ActionLogger;\nimport com.twitter.search.earlybird.util.CoordinatedEarlybirdActionInterface;\nimport com.twitter.search.earlybird.util.CoordinatedEarlybirdActionLockFailed;\nimport com.twitter.search.earlybird.util.ParallelUtil;\n\n/**\n * Flushes an EarlybirdIndex to HDFS, so that when Earlybird starts, it can read the index from\n * HDFS instead of indexing from scratch.\n *\n * The path looks like:\n * /smf1/rt2/user/search/earlybird/loadtest/realtime/indexes/flush_version_158/partition_8/index_2020_02_25_02\n */\npublic class EarlybirdIndexFlusher {\n  public enum FlushAttemptResult {\n    CHECKED_RECENTLY,\n    FOUND_INDEX,\n    FLUSH_ATTEMPT_MADE,\n    FAILED_LOCK_ATTEMPT,\n    HADOOP_TIMEOUT\n  }\n\n  @FunctionalInterface\n  public interface PostFlushOperation {\n    /**\n     * Run this after we finish flushing an index, before we rejoin the serverset.\n     */\n    void execute();\n  }\n\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdIndexFlusher.class);\n\n  private static final SearchCounter FLUSH_SUCCESS_COUNTER =\n      SearchCounter.export(\"successfully_flushed_index\");\n\n  public static final String TWEET_KAFKA_OFFSET = \"tweet_kafka_offset\";\n  public static final String UPDATE_KAFKA_OFFSET = \"update_kafka_offset\";\n  public static final String FLUSHED_FROM_REPLICA = \"flushed_from_replica\";\n  public static final String SEGMENTS = \"segments\";\n  public static final String TIMESLICE_ID = \"timeslice_id\";\n\n  public static final String DATA_SUFFIX = \".data\";\n  public static final String INFO_SUFFIX = \".info\";\n  public static final String INDEX_INFO = \"earlybird_index.info\";\n\n  private static final String INDEX_PATH_FORMAT = \"%s/flush_version_%d/partition_%d\";\n  public static final DateFormat INDEX_DATE_SUFFIX = new SimpleDateFormat(\"yyyy_MM_dd_HH\");\n  public static final String INDEX_PREFIX = \"index_\";\n  public static final String TMP_PREFIX = \"tmp_\";\n\n  // Check if we need to flush every five minutes.\n  private static final long FLUSH_CHECK_PERIOD = Duration.ofMinutes(5).toMillis();\n\n  // Make sure we don't keep more than 3 copies of the index in HDFS, so that we don't run out of\n  // HDFS space.\n  private static final int INDEX_COPIES = 3;\n\n  private static final NonPagingAssert FLUSHING_TOO_MANY_NON_OPTIMIZED_SEGMENTS =\n          new NonPagingAssert(\"flushing_too_many_non_optimized_segments\");\n\n  private final CoordinatedEarlybirdActionInterface actionCoordinator;\n  private final FileSystem fileSystem;\n  private final Path indexPath;\n  private final Clock clock;\n  private final SegmentManager segmentManager;\n  private final int replicaId;\n  private final TimeLimitedHadoopExistsCall timeLimitedHadoopExistsCall;\n  private final OptimizationAndFlushingCoordinationLock optimizationAndFlushingCoordinationLock;\n\n  private long checkedAt = 0;\n\n  public EarlybirdIndexFlusher(\n      CoordinatedEarlybirdActionInterface actionCoordinator,\n      FileSystem fileSystem,\n      String indexHDFSPath,\n      SegmentManager segmentManager,\n      PartitionConfig partitionConfig,\n      Clock clock,\n      TimeLimitedHadoopExistsCall timeLimitedHadoopExistsCall,\n      OptimizationAndFlushingCoordinationLock optimizationAndFlushingCoordinationLock\n  ) {\n    this.actionCoordinator = actionCoordinator;\n    this.fileSystem = fileSystem;\n    this.indexPath = buildPathToIndexes(indexHDFSPath, partitionConfig);\n    this.segmentManager = segmentManager;\n    this.clock = clock;\n    this.replicaId = partitionConfig.getHostPositionWithinHashPartition();\n    this.timeLimitedHadoopExistsCall = timeLimitedHadoopExistsCall;\n    this.optimizationAndFlushingCoordinationLock = optimizationAndFlushingCoordinationLock;\n  }\n\n  /**\n   * Periodically checks if an index needs to be uploaded to HDFS, and uploads it if necessary.\n   * Skips flush if unable to acquire the optimizationAndFlushingCoordinationLock.\n   */\n  public FlushAttemptResult flushIfNecessary(\n      long tweetOffset,\n      long updateOffset,\n      PostFlushOperation postFlushOperation) throws Exception {\n    long now = clock.nowMillis();\n    if (now - checkedAt < FLUSH_CHECK_PERIOD) {\n      return FlushAttemptResult.CHECKED_RECENTLY;\n    }\n\n    checkedAt = now;\n\n    // Try to aqcuire lock to ensure that we are not in the gc_before_optimization or the\n    // post_optimization_rebuilds step of optimization. If the lock is not available, then skip\n    // flushing.\n    if (!optimizationAndFlushingCoordinationLock.tryLock()) {\n      return FlushAttemptResult.FAILED_LOCK_ATTEMPT;\n    }\n    // Acquired the lock, so wrap the flush in a try/finally block to ensure we release the lock\n    try {\n      Path flushPath = pathForHour();\n\n      try {\n        // If this doesn't execute on time, it will throw an exception and this function\n        // finishes its execution.\n        boolean result = timeLimitedHadoopExistsCall.exists(flushPath);\n\n        if (result) {\n          return FlushAttemptResult.FOUND_INDEX;\n        }\n      } catch (TimeoutException e) {\n        LOG.warn(\"Timeout while calling hadoop\", e);\n        return FlushAttemptResult.HADOOP_TIMEOUT;\n      }\n\n      boolean flushedIndex = false;\n      try {\n        // this function returns a boolean.\n        actionCoordinator.execute(\"index_flushing\", isCoordinated ->\n            flushIndex(flushPath, isCoordinated, tweetOffset, updateOffset, postFlushOperation));\n        flushedIndex = true;\n      } catch (CoordinatedEarlybirdActionLockFailed e) {\n        // This only happens when we fail to grab the lock, which is fine because another Earlybird\n        // is already working on flushing this index, so we don't need to.\n        LOG.debug(\"Failed to grab lock\", e);\n      }\n\n      if (flushedIndex) {\n        // We don't return with a guarantee that we actually flushed something. It's possible\n        // that the .execute() function above was not able to leave the server set to flush.\n        return FlushAttemptResult.FLUSH_ATTEMPT_MADE;\n      } else {\n        return FlushAttemptResult.FAILED_LOCK_ATTEMPT;\n      }\n    } finally {\n      optimizationAndFlushingCoordinationLock.unlock();\n    }\n  }\n\n  /**\n   * Create a subpath to the directory with many indexes in it. Will have an index for each hour.\n   */\n  public static Path buildPathToIndexes(String root, PartitionConfig partitionConfig) {\n    return new Path(String.format(\n        INDEX_PATH_FORMAT,\n        root,\n        FlushVersion.CURRENT_FLUSH_VERSION.getVersionNumber(),\n        partitionConfig.getIndexingHashPartitionID()));\n  }\n\n\n  /**\n   * Returns a sorted map from the unix time in millis an index was flushed to the path of an index.\n   * The last element will be the path of the most recent index.\n   */\n  public static SortedMap<Long, Path> getIndexPathsByTime(\n      Path indexPath,\n      FileSystem fileSystem\n  ) throws IOException, ParseException {\n    LOG.info(\"Getting index paths from file system: {}\", fileSystem.getUri().toASCIIString());\n\n    SortedMap<Long, Path> pathByTime = new TreeMap<>();\n    Path globPattern = indexPath.suffix(\"/\" + EarlybirdIndexFlusher.INDEX_PREFIX + \"*\");\n    LOG.info(\"Lookup glob pattern: {}\", globPattern);\n\n    for (FileStatus indexDir : fileSystem.globStatus(globPattern)) {\n      String name = new File(indexDir.getPath().toString()).getName();\n      String dateString = name.substring(EarlybirdIndexFlusher.INDEX_PREFIX.length());\n      Date date = EarlybirdIndexFlusher.INDEX_DATE_SUFFIX.parse(dateString);\n      pathByTime.put(date.getTime(), indexDir.getPath());\n    }\n    LOG.info(\"Found {} files matching the pattern.\", pathByTime.size());\n\n    return pathByTime;\n  }\n\n  private boolean flushIndex(\n      Path flushPath,\n      boolean isCoordinated,\n      long tweetOffset,\n      long updateOffset,\n      PostFlushOperation postFlushOperation\n  ) throws Exception {\n    Preconditions.checkState(isCoordinated);\n\n    if (fileSystem.exists(flushPath)) {\n      return false;\n    }\n\n    LOG.info(\"Starting index flush\");\n\n    // In case the process is killed suddenly, we wouldn't be able to clean up the temporary\n    // directory, and we don't want other processes to reuse it, so add some randomness.\n    Path tmpPath = indexPath.suffix(\"/\" + TMP_PREFIX + RandomStringUtils.randomAlphabetic(8));\n    boolean creationSucceed = fileSystem.mkdirs(tmpPath);\n    if (!creationSucceed) {\n      throw new IOException(\"Couldn't create HDFS directory at \" + flushPath);\n    }\n\n    LOG.info(\"Temp path: {}\", tmpPath);\n    try {\n      ArrayList<SegmentInfo> segmentInfos = Lists.newArrayList(segmentManager.getSegmentInfos(\n          SegmentManager.Filter.Enabled, SegmentManager.Order.NEW_TO_OLD).iterator());\n      segmentManager.logState(\"Before flushing\");\n      EarlybirdIndex index = new EarlybirdIndex(segmentInfos, tweetOffset, updateOffset);\n      ActionLogger.run(\n          \"Flushing index to \" + tmpPath,\n          () -> flushIndex(tmpPath, index));\n    } catch (Exception e) {\n      LOG.error(\"Exception while flushing index. Rethrowing.\");\n\n      if (fileSystem.delete(tmpPath, true)) {\n        LOG.info(\"Successfully deleted temp output\");\n      } else {\n        LOG.error(\"Couldn't delete temp output\");\n      }\n\n      throw e;\n    }\n\n    // We flush it to a temporary directory, then rename the temporary directory so that it the\n    // change is atomic, and other Earlybirds will either see the old indexes, or the new, complete\n    // index, but never an in progress index.\n    boolean renameSucceeded = fileSystem.rename(tmpPath, flushPath);\n    if (!renameSucceeded) {\n      throw new IOException(\"Couldn't rename HDFS from \" + tmpPath + \" to \" + flushPath);\n    }\n    LOG.info(\"Flushed index to {}\", flushPath);\n\n    cleanupOldIndexes();\n\n    FLUSH_SUCCESS_COUNTER.increment();\n\n    LOG.info(\"Executing post flush operation...\");\n    postFlushOperation.execute();\n\n    return true;\n  }\n\n  private void cleanupOldIndexes() throws Exception {\n    LOG.info(\"Looking up whether we need to clean up old indexes...\");\n    SortedMap<Long, Path> pathsByTime =\n        EarlybirdIndexFlusher.getIndexPathsByTime(indexPath, fileSystem);\n\n    while (pathsByTime.size() > INDEX_COPIES) {\n      Long key = pathsByTime.firstKey();\n      Path oldestHourPath = pathsByTime.remove(key);\n      LOG.info(\"Deleting old index at path '{}'.\", oldestHourPath);\n\n      if (fileSystem.delete(oldestHourPath, true)) {\n        LOG.info(\"Successfully deleted old index\");\n      } else {\n        LOG.error(\"Couldn't delete old index\");\n      }\n    }\n  }\n\n  private Path pathForHour() {\n    Date date = new Date(clock.nowMillis());\n    String time = INDEX_DATE_SUFFIX.format(date);\n    return indexPath.suffix(\"/\" + INDEX_PREFIX + time);\n  }\n\n  private void flushIndex(Path flushPath, EarlybirdIndex index) throws Exception {\n    int numOfNonOptimized = index.numOfNonOptimizedSegments();\n    if (numOfNonOptimized > EarlybirdIndex.MAX_NUM_OF_NON_OPTIMIZED_SEGMENTS) {\n      LOG.error(\n              \"Found {} non-optimized segments when flushing to disk!\", numOfNonOptimized);\n      FLUSHING_TOO_MANY_NON_OPTIMIZED_SEGMENTS.assertFailed();\n    }\n\n    int numSegments = index.getSegmentInfoList().size();\n    int flushingThreadPoolSize = numSegments;\n\n    if (Config.environmentIsTest()) {\n      // SEARCH-33763: Limit the thread pool size for tests to avoid using too much memory on scoot.\n      flushingThreadPoolSize = 2;\n    }\n\n    LOG.info(\"Flushing index using a thread pool size of {}\", flushingThreadPoolSize);\n\n    ParallelUtil.parmap(\"flush-index\", flushingThreadPoolSize, si -> ActionLogger.call(\n        \"Flushing segment \" + si.getSegmentName(),\n        () -> flushSegment(flushPath, si)), index.getSegmentInfoList());\n\n    FlushInfo indexInfo = new FlushInfo();\n    indexInfo.addLongProperty(UPDATE_KAFKA_OFFSET, index.getUpdateOffset());\n    indexInfo.addLongProperty(TWEET_KAFKA_OFFSET, index.getTweetOffset());\n    indexInfo.addIntProperty(FLUSHED_FROM_REPLICA, replicaId);\n\n    FlushInfo segmentFlushInfos = indexInfo.newSubProperties(SEGMENTS);\n    for (SegmentInfo segmentInfo : index.getSegmentInfoList()) {\n      FlushInfo segmentFlushInfo = segmentFlushInfos.newSubProperties(segmentInfo.getSegmentName());\n      segmentFlushInfo.addLongProperty(TIMESLICE_ID, segmentInfo.getTimeSliceID());\n    }\n\n    Path indexInfoPath = flushPath.suffix(\"/\" + INDEX_INFO);\n    try (FSDataOutputStream infoOutputStream = fileSystem.create(indexInfoPath)) {\n      OutputStreamWriter infoFileWriter = new OutputStreamWriter(infoOutputStream);\n      FlushInfo.flushAsYaml(indexInfo, infoFileWriter);\n    }\n  }\n\n  private BoxedUnit flushSegment(Path flushPath, SegmentInfo segmentInfo) throws Exception {\n    Path segmentPrefix = flushPath.suffix(\"/\" + segmentInfo.getSegmentName());\n    Path segmentPath = segmentPrefix.suffix(DATA_SUFFIX);\n\n    FlushInfo flushInfo = new FlushInfo();\n\n    try (FSDataOutputStream outputStream = fileSystem.create(segmentPath)) {\n      DataSerializer out = new DataSerializer(segmentPath.toString(), outputStream);\n      segmentInfo.getIndexSegment().flush(flushInfo, out);\n    }\n\n    Path infoPath = segmentPrefix.suffix(INFO_SUFFIX);\n\n    try (FSDataOutputStream infoOutputStream = fileSystem.create(infoPath)) {\n      OutputStreamWriter infoFileWriter = new OutputStreamWriter(infoOutputStream);\n      FlushInfo.flushAsYaml(flushInfo, infoFileWriter);\n    }\n    return BoxedUnit.UNIT;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/EarlybirdIndexLoader.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.BufferedInputStream;\nimport java.io.IOException;\nimport java.time.Duration;\nimport java.util.List;\nimport java.util.Optional;\nimport java.util.SortedMap;\n\nimport com.google.common.base.Stopwatch;\n\nimport org.apache.commons.compress.utils.Lists;\nimport org.apache.hadoop.fs.FSDataInputStream;\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.Path;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.partitioning.base.TimeSlice;\nimport com.twitter.search.common.util.io.flushable.DataDeserializer;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.earlybird.common.NonPagingAssert;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.index.EarlybirdSegmentFactory;\nimport com.twitter.search.earlybird.util.ActionLogger;\nimport com.twitter.search.earlybird.util.ParallelUtil;\n\n/**\n * Loads an index from HDFS, if possible, or indexes all tweets from scratch using a\n * FreshStartupHandler.\n */\npublic class EarlybirdIndexLoader {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdIndexLoader.class);\n\n  public static final String ENV_FOR_TESTS = \"test_env\";\n\n  // To determine whether we should or should not load the most recent index from HDFS if available.\n  public static final long INDEX_FRESHNESS_THRESHOLD_MILLIS = Duration.ofDays(1).toMillis();\n\n  private static final NonPagingAssert LOADING_TOO_MANY_NON_OPTIMIZED_SEGMENTS =\n          new NonPagingAssert(\"loading_too_many_non_optimized_segments\");\n\n  private final FileSystem fileSystem;\n  private final Path indexPath;\n  private final PartitionConfig partitionConfig;\n  private final EarlybirdSegmentFactory earlybirdSegmentFactory;\n  private final SegmentSyncConfig segmentSyncConfig;\n  private final Clock clock;\n  // Aurora environment we're running in: \"prod\", \"loadtest\", \"staging2\" etc. etc\n  private final String environment;\n\n  public EarlybirdIndexLoader(\n      FileSystem fileSystem,\n      String indexHDFSPath,\n      String environment,\n      PartitionConfig partitionConfig,\n      EarlybirdSegmentFactory earlybirdSegmentFactory,\n      SegmentSyncConfig segmentSyncConfig,\n      Clock clock\n  ) {\n    this.fileSystem = fileSystem;\n    this.partitionConfig = partitionConfig;\n    this.earlybirdSegmentFactory = earlybirdSegmentFactory;\n    this.segmentSyncConfig = segmentSyncConfig;\n    this.indexPath = EarlybirdIndexFlusher.buildPathToIndexes(indexHDFSPath, partitionConfig);\n    this.clock = clock;\n    this.environment = environment;\n  }\n\n  /**\n   * Tries to load an index from HDFS for this FlushVersion/Partition/Cluster. Returns an empty\n   * option if there is no index found.\n   */\n  public Optional<EarlybirdIndex> loadIndex() {\n    try {\n      Optional<EarlybirdIndex> loadedIndex =\n          ActionLogger.call(\"Load index from HDFS.\", this::loadFromHDFS);\n\n      if (loadedIndex.isPresent()) {\n        EarlybirdIndex index = loadedIndex.get();\n        int numOfNonOptimized = index.numOfNonOptimizedSegments();\n        if (numOfNonOptimized > EarlybirdIndex.MAX_NUM_OF_NON_OPTIMIZED_SEGMENTS) {\n          // We should never have too many unoptimized segments. If this happens we likely have a\n          // bug somewhere that caused another Earlybird to flush too many unoptimized segments.\n          // Use NonPagingAssert to alert the oncall if this happens so they can look into it.\n          LOG.error(\"Found {} non-optimized segments when loading from disk!\", numOfNonOptimized);\n          LOADING_TOO_MANY_NON_OPTIMIZED_SEGMENTS.assertFailed();\n\n          // If there are too many unoptimized segments, optimize the older ones until there are\n          // only MAX_NUM_OF_NON_OPTIMIZED_SEGMENTS left in the unoptimized state. The segment info\n          // list is always in order, so we will never try to optimize the most recent segments\n          // here.\n          int numSegmentsToOptimize =\n              numOfNonOptimized - EarlybirdIndex.MAX_NUM_OF_NON_OPTIMIZED_SEGMENTS;\n          LOG.info(\"Will try to optimize {} segments\", numSegmentsToOptimize);\n          for (SegmentInfo segmentInfo : index.getSegmentInfoList()) {\n            if (numSegmentsToOptimize > 0 && !segmentInfo.isOptimized()) {\n              Stopwatch optimizationStopwatch = Stopwatch.createStarted();\n              LOG.info(\"Starting to optimize segment: {}\", segmentInfo.getSegmentName());\n              segmentInfo.getIndexSegment().optimizeIndexes();\n              numSegmentsToOptimize--;\n              LOG.info(\"Optimization of segment {} finished in {}.\",\n                  segmentInfo.getSegmentName(), optimizationStopwatch);\n            }\n          }\n        }\n\n        int newNumOfNonOptimized = index.numOfNonOptimizedSegments();\n        LOG.info(\"Loaded {} segments. {} are unoptimized.\",\n                index.getSegmentInfoList().size(),\n                newNumOfNonOptimized);\n\n        return loadedIndex;\n      }\n    } catch (Throwable e) {\n      LOG.error(\"Error loading index from HDFS, will index from scratch.\", e);\n    }\n\n    return Optional.empty();\n  }\n\n  private Optional<EarlybirdIndex> loadFromHDFS() throws Exception {\n    SortedMap<Long, Path> pathsByTime =\n        EarlybirdIndexFlusher.getIndexPathsByTime(indexPath, fileSystem);\n\n    if (pathsByTime.isEmpty()) {\n      LOG.info(\"Could not load index from HDFS (path: {}), will index from scratch.\", indexPath);\n      return Optional.empty();\n    }\n\n    long mostRecentIndexTimeMillis = pathsByTime.lastKey();\n    Path mostRecentIndexPath = pathsByTime.get(mostRecentIndexTimeMillis);\n\n    if (clock.nowMillis() - mostRecentIndexTimeMillis > INDEX_FRESHNESS_THRESHOLD_MILLIS) {\n      LOG.info(\"Most recent index in HDFS (path: {}) is old, will do a fresh startup.\",\n              mostRecentIndexPath);\n      return Optional.empty();\n    }\n\n    EarlybirdIndex index = ActionLogger.call(\n        \"loading index from \" + mostRecentIndexPath,\n        () -> loadIndex(mostRecentIndexPath));\n\n    return Optional.of(index);\n  }\n\n  private EarlybirdIndex loadIndex(Path flushPath) throws Exception {\n    Path indexInfoPath = flushPath.suffix(\"/\" + EarlybirdIndexFlusher.INDEX_INFO);\n\n    FlushInfo indexInfo;\n    try (FSDataInputStream infoInputStream = fileSystem.open(indexInfoPath)) {\n      indexInfo = FlushInfo.loadFromYaml(infoInputStream);\n    }\n\n    FlushInfo segmentsFlushInfo = indexInfo.getSubProperties(EarlybirdIndexFlusher.SEGMENTS);\n    List<String> segmentNames = Lists.newArrayList(segmentsFlushInfo.getKeyIterator());\n\n    // This should only happen if you're running in stagingN and loading a prod index through\n    // the read_index_from_prod_location flag. In this case, we point to a directory that has\n    // a lot more than the number of segments we want in staging and we trim this list to the\n    // desired number.\n    if (environment.matches(\"staging\\\\d\")) {\n      if (segmentNames.size() > partitionConfig.getMaxEnabledLocalSegments()) {\n        LOG.info(\"Trimming list of loaded segments from size {} to size {}.\",\n            segmentNames.size(), partitionConfig.getMaxEnabledLocalSegments());\n        segmentNames = segmentNames.subList(\n            segmentNames.size() - partitionConfig.getMaxEnabledLocalSegments(),\n            segmentNames.size());\n      }\n    }\n\n    List<SegmentInfo> segmentInfoList = ParallelUtil.parmap(\"load-index\", name -> {\n      FlushInfo subProperties = segmentsFlushInfo.getSubProperties(name);\n      long timesliceID = subProperties.getLongProperty(EarlybirdIndexFlusher.TIMESLICE_ID);\n      return ActionLogger.call(\n          \"loading segment \" + name,\n          () -> loadSegment(flushPath, name, timesliceID));\n    }, segmentNames);\n\n    return new EarlybirdIndex(\n        segmentInfoList,\n        indexInfo.getLongProperty(EarlybirdIndexFlusher.TWEET_KAFKA_OFFSET),\n        indexInfo.getLongProperty(EarlybirdIndexFlusher.UPDATE_KAFKA_OFFSET));\n  }\n\n  private SegmentInfo loadSegment(\n      Path flushPath,\n      String segmentName,\n      long timesliceID\n  ) throws IOException {\n    Path segmentPrefix = flushPath.suffix(\"/\" + segmentName);\n    Path segmentPath = segmentPrefix.suffix(EarlybirdIndexFlusher.DATA_SUFFIX);\n\n    TimeSlice timeSlice = new TimeSlice(\n        timesliceID,\n        EarlybirdConfig.getMaxSegmentSize(),\n        partitionConfig.getIndexingHashPartitionID(),\n        partitionConfig.getNumPartitions());\n\n    SegmentInfo segmentInfo = new SegmentInfo(\n        timeSlice.getSegment(),\n        earlybirdSegmentFactory,\n        segmentSyncConfig);\n\n    Path infoPath = segmentPrefix.suffix(EarlybirdIndexFlusher.INFO_SUFFIX);\n    FlushInfo flushInfo;\n    try (FSDataInputStream infoInputStream = fileSystem.open(infoPath)) {\n      flushInfo = FlushInfo.loadFromYaml(infoInputStream);\n    }\n\n    FSDataInputStream inputStream = fileSystem.open(segmentPath);\n\n    // It's significantly slower to read from the FSDataInputStream on demand, so we\n    // use a buffered reader to pre-read bigger chunks.\n    int bufferSize = 1 << 22; // 4MB\n    BufferedInputStream bufferedInputStream = new BufferedInputStream(inputStream, bufferSize);\n\n    DataDeserializer in = new DataDeserializer(bufferedInputStream, segmentName);\n    segmentInfo.getIndexSegment().load(in, flushInfo);\n\n    return segmentInfo;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/EarlybirdKafkaConsumer.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.Closeable;\nimport java.time.Duration;\nimport java.util.Map;\nimport java.util.concurrent.atomic.AtomicBoolean;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Stopwatch;\nimport com.google.common.collect.ImmutableList;\n\nimport org.apache.kafka.clients.consumer.ConsumerRecords;\nimport org.apache.kafka.clients.consumer.KafkaConsumer;\nimport org.apache.kafka.common.TopicPartition;\nimport org.apache.kafka.common.errors.ApiException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.util.LogFormatUtil;\nimport com.twitter.search.earlybird.EarlybirdStatus;\nimport com.twitter.search.earlybird.common.CaughtUpMonitor;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.exception.WrappedKafkaApiException;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusCode;\n\n/**\n * Reads TVEs from Kafka and writes them to a PartitionWriter.\n */\npublic class EarlybirdKafkaConsumer implements Closeable {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdKafkaConsumer.class);\n\n  private static final Duration POLL_TIMEOUT = Duration.ofSeconds(1);\n  private static final String STATS_PREFIX = \"earlybird_kafka_consumer_\";\n\n  // See SEARCH-31827\n  private static final SearchCounter INGESTING_DONE =\n      SearchCounter.export(STATS_PREFIX + \"ingesting_done\");\n  private static final SearchRateCounter POLL_LOOP_EXCEPTIONS =\n      SearchRateCounter.export(STATS_PREFIX + \"poll_loop_exceptions\");\n  private static final SearchRateCounter FLUSHING_EXCEPTIONS =\n      SearchRateCounter.export(STATS_PREFIX + \"flushing_exceptions\");\n\n  private static final SearchTimerStats TIMED_POLLS =\n      SearchTimerStats.export(STATS_PREFIX + \"timed_polls\");\n  private static final SearchTimerStats TIMED_INDEX_EVENTS =\n      SearchTimerStats.export(STATS_PREFIX + \"timed_index_events\");\n\n  private final AtomicBoolean running = new AtomicBoolean(true);\n  private final BalancingKafkaConsumer balancingKafkaConsumer;\n  private final PartitionWriter partitionWriter;\n  protected final TopicPartition tweetTopic;\n  protected final TopicPartition updateTopic;\n  private final KafkaConsumer<Long, ThriftVersionedEvents> underlyingKafkaConsumer;\n  private final CriticalExceptionHandler criticalExceptionHandler;\n  private final EarlybirdIndexFlusher earlybirdIndexFlusher;\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n  private boolean finishedIngestUntilCurrent;\n  private final CaughtUpMonitor indexCaughtUpMonitor;\n\n  protected class ConsumeBatchResult {\n    private boolean isCaughtUp;\n    private long readRecordsCount;\n\n    public ConsumeBatchResult(boolean isCaughtUp, long readRecordsCount) {\n      this.isCaughtUp = isCaughtUp;\n      this.readRecordsCount = readRecordsCount;\n    }\n\n    public boolean isCaughtUp() {\n      return isCaughtUp;\n    }\n\n    public long getReadRecordsCount() {\n      return readRecordsCount;\n    }\n  }\n\n  public EarlybirdKafkaConsumer(\n      KafkaConsumer<Long, ThriftVersionedEvents> underlyingKafkaConsumer,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      CriticalExceptionHandler criticalExceptionHandler,\n      PartitionWriter partitionWriter,\n      TopicPartition tweetTopic,\n      TopicPartition updateTopic,\n      EarlybirdIndexFlusher earlybirdIndexFlusher,\n      CaughtUpMonitor kafkaIndexCaughtUpMonitor\n  ) {\n    this.partitionWriter = partitionWriter;\n    this.underlyingKafkaConsumer = underlyingKafkaConsumer;\n    this.criticalExceptionHandler = criticalExceptionHandler;\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n    this.tweetTopic = tweetTopic;\n    this.updateTopic = updateTopic;\n    this.earlybirdIndexFlusher = earlybirdIndexFlusher;\n\n    LOG.info(\"Reading from Kafka topics: tweetTopic={}, updateTopic={}\", tweetTopic, updateTopic);\n    underlyingKafkaConsumer.assign(ImmutableList.of(updateTopic, tweetTopic));\n\n    this.balancingKafkaConsumer =\n        new BalancingKafkaConsumer(underlyingKafkaConsumer, tweetTopic, updateTopic);\n    this.finishedIngestUntilCurrent = false;\n    this.indexCaughtUpMonitor = kafkaIndexCaughtUpMonitor;\n  }\n\n  /**\n   * Run the consumer, indexing from Kafka.\n   */\n  @VisibleForTesting\n  public void run() {\n    while (isRunning()) {\n      ConsumeBatchResult result = consumeBatch(true);\n      indexCaughtUpMonitor.setAndNotify(result.isCaughtUp());\n    }\n  }\n\n  /**\n   * Reads from Kafka, starting at the given offsets, and applies the events until we are caught up\n   * with the current streams.\n   */\n  public void ingestUntilCurrent(long tweetOffset, long updateOffset) {\n    Preconditions.checkState(!finishedIngestUntilCurrent);\n    Stopwatch stopwatch = Stopwatch.createStarted();\n    LOG.info(\"Ingest until current: seeking to Kafka offset {} for tweets and {} for updates.\",\n        tweetOffset, updateOffset);\n\n    try {\n      underlyingKafkaConsumer.seek(tweetTopic, tweetOffset);\n      underlyingKafkaConsumer.seek(updateTopic, updateOffset);\n    } catch (ApiException kafkaApiException) {\n      throw new WrappedKafkaApiException(\"Can't seek to tweet and update offsets\",\n          kafkaApiException);\n    }\n\n    Map<TopicPartition, Long> endOffsets;\n    try {\n      endOffsets = underlyingKafkaConsumer.endOffsets(ImmutableList.of(tweetTopic, updateTopic));\n    } catch (ApiException kafkaApiException) {\n      throw new WrappedKafkaApiException(\"Can't find end offsets\",\n          kafkaApiException);\n    }\n\n    if (endOffsets.size() > 0) {\n      LOG.info(String.format(\"Records until current: tweets=%,d, updates=%,d\",\n          endOffsets.get(tweetTopic) - tweetOffset + 1,\n          endOffsets.get(updateTopic) - updateOffset + 1));\n    }\n\n    consumeBatchesUntilCurrent(true);\n\n    LOG.info(\"ingestUntilCurrent finished in {}.\", stopwatch);\n\n    partitionWriter.logState();\n    INGESTING_DONE.increment();\n    finishedIngestUntilCurrent = true;\n  }\n\n  /**\n   * Consume tweets and updates from streams until we're up to date.\n   *\n   * @return total number of read records.\n   */\n  private long consumeBatchesUntilCurrent(boolean flushingEnabled) {\n    long totalRecordsRead = 0;\n    long batchesConsumed = 0;\n\n    while (isRunning()) {\n      ConsumeBatchResult result = consumeBatch(flushingEnabled);\n      batchesConsumed++;\n      totalRecordsRead += result.getReadRecordsCount();\n      if (isCurrent(result.isCaughtUp())) {\n        break;\n      }\n    }\n\n    LOG.info(\"Processed batches: {}\", batchesConsumed);\n\n    return totalRecordsRead;\n  }\n\n  // This method is overriden in MockEarlybirdKafkaConsumer.\n  public boolean isCurrent(boolean current) {\n    return current;\n  }\n\n  /**\n   * We don't index during flushing, so after the flush is done, the index is stale.\n   * We need to get to current, before we rejoin the serverset so that upon rejoining we're\n   * not serving a stale index.\n   */\n  @VisibleForTesting\n  void getToCurrentPostFlush() {\n    LOG.info(\"Getting to current post flush\");\n    Stopwatch stopwatch = Stopwatch.createStarted();\n\n    long totalRecordsRead = consumeBatchesUntilCurrent(false);\n\n    LOG.info(\"Post flush, became current in: {}, after reading {} records.\",\n        stopwatch, LogFormatUtil.formatInt(totalRecordsRead));\n  }\n\n  /*\n   * @return true if we are current after indexing this batch.\n   */\n  @VisibleForTesting\n  protected ConsumeBatchResult consumeBatch(boolean flushingEnabled) {\n    long readRecordsCount = 0;\n    boolean isCaughtUp = false;\n\n    try {\n      // Poll.\n      SearchTimer pollTimer = TIMED_POLLS.startNewTimer();\n      ConsumerRecords<Long, ThriftVersionedEvents> records =\n          balancingKafkaConsumer.poll(POLL_TIMEOUT);\n      readRecordsCount += records.count();\n      TIMED_POLLS.stopTimerAndIncrement(pollTimer);\n\n      // Index.\n      SearchTimer indexTimer = TIMED_INDEX_EVENTS.startNewTimer();\n      isCaughtUp = partitionWriter.indexBatch(records);\n      TIMED_INDEX_EVENTS.stopTimerAndIncrement(indexTimer);\n    } catch (Exception ex) {\n      POLL_LOOP_EXCEPTIONS.increment();\n      LOG.error(\"Exception in poll loop\", ex);\n    }\n\n    try {\n      // Possibly flush the index.\n      if (isCaughtUp && flushingEnabled) {\n        long tweetOffset = 0;\n        long updateOffset = 0;\n\n        try {\n          tweetOffset = underlyingKafkaConsumer.position(tweetTopic);\n          updateOffset = underlyingKafkaConsumer.position(updateTopic);\n        } catch (ApiException kafkaApiException) {\n          throw new WrappedKafkaApiException(\"can't get topic positions\", kafkaApiException);\n        }\n\n        EarlybirdIndexFlusher.FlushAttemptResult flushAttemptResult =\n            earlybirdIndexFlusher.flushIfNecessary(\n                tweetOffset, updateOffset, this::getToCurrentPostFlush);\n\n        if (flushAttemptResult == EarlybirdIndexFlusher.FlushAttemptResult.FLUSH_ATTEMPT_MADE) {\n          // Viz might show this as a fairly high number, so we're printing it here to confirm\n          // the value on the server.\n          LOG.info(\"Finished flushing. Index freshness in ms: {}\",\n              LogFormatUtil.formatInt(searchIndexingMetricSet.getIndexFreshnessInMillis()));\n        }\n\n        if (!finishedIngestUntilCurrent) {\n          LOG.info(\"Became current on startup. Tried to flush with result: {}\",\n              flushAttemptResult);\n        }\n      }\n    } catch (Exception ex) {\n      FLUSHING_EXCEPTIONS.increment();\n      LOG.error(\"Exception while flushing\", ex);\n    }\n\n    return new ConsumeBatchResult(isCaughtUp, readRecordsCount);\n  }\n\n  public boolean isRunning() {\n    return running.get() && EarlybirdStatus.getStatusCode() != EarlybirdStatusCode.STOPPING;\n  }\n\n  public void prepareAfterStartingWithIndex(long maxIndexedTweetId) {\n    partitionWriter.prepareAfterStartingWithIndex(maxIndexedTweetId);\n  }\n\n  public void close() {\n    balancingKafkaConsumer.close();\n    running.set(false);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/EarlybirdStartup.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.Closeable;\n\nimport com.twitter.search.earlybird.exception.EarlybirdStartupException;\n\n/**\n * Handles starting and indexing data for an Earlybird.\n */\n@FunctionalInterface\npublic interface EarlybirdStartup {\n  /**\n   * Handles indexing Tweets, Tweet Updates and user updates. Blocks until current, and forks a\n   * thread to keep the index current.\n   */\n  Closeable start() throws EarlybirdStartupException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/FlowControlException.java",
    "content": "package com.twitter.search.earlybird.partition;\n\n/**\n * Exception used to cause a ScheduledExecutorService to stop executing. Used when the\n * success condition of the class has been achieved.\n */\npublic class FlowControlException extends RuntimeException {\n\n  public FlowControlException() {\n    super();\n  }\n\n  public FlowControlException(String message) {\n    super(message);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/HdfsUtil.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\n\nimport org.apache.hadoop.conf.Configuration;\nimport org.apache.hadoop.fs.FileStatus;\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.Path;\n\npublic final class HdfsUtil {\n  private HdfsUtil() {\n  }\n\n  public static FileSystem getHdfsFileSystem() throws IOException {\n    Configuration config = new Configuration();\n    // Since earlybird uses hdfs from different threads, and closes the FileSystem from\n    // them independently, we want each thread to have its own, new FileSystem.\n    return FileSystem.newInstance(config);\n  }\n\n  /**\n   * Checks if the given segment is present on HDFS\n   */\n  public static boolean segmentExistsOnHdfs(FileSystem fs, SegmentInfo segmentInfo)\n      throws IOException {\n    String hdfsBaseDirPrefix = segmentInfo.getSyncInfo().getHdfsUploadDirPrefix();\n    FileStatus[] statuses = fs.globStatus(new Path(hdfsBaseDirPrefix));\n    return statuses != null && statuses.length > 0;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/ISegmentWriter.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\n\npublic interface ISegmentWriter {\n  enum Result {\n    SUCCESS,\n    FAILURE_RETRYABLE,\n    FAILURE_NOT_RETRYABLE,\n  }\n\n  /**\n   * Indexes the given ThriftVersionedEvents instance (adds it to the segment associated with this\n   * SegmentWriter instance).\n   */\n  Result indexThriftVersionedEvents(ThriftVersionedEvents tve) throws IOException;\n\n  /**\n   * Returns the segment info for this segment writer.\n   */\n  SegmentInfo getSegmentInfo();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/IndexingResultCounts.java",
    "content": "package com.twitter.search.earlybird.partition;\n\n/**\n * Helper class used to store counts to be logged.\n */\npublic class IndexingResultCounts {\n  private int indexingCalls;\n  private int failureRetriable;\n  private int failureNotRetriable;\n  private int indexingSuccess;\n\n  public IndexingResultCounts() {\n  }\n\n  /**\n   * Updates the internal counts with a single result.\n   */\n  public void countResult(ISegmentWriter.Result result) {\n    indexingCalls++;\n    if (result == ISegmentWriter.Result.FAILURE_NOT_RETRYABLE) {\n      failureNotRetriable++;\n    } else if (result == ISegmentWriter.Result.FAILURE_RETRYABLE) {\n      failureRetriable++;\n    } else if (result == ISegmentWriter.Result.SUCCESS) {\n      indexingSuccess++;\n    }\n  }\n\n  int getIndexingCalls() {\n    return indexingCalls;\n  }\n\n  int getFailureRetriable() {\n    return failureRetriable;\n  }\n\n  int getFailureNotRetriable() {\n    return failureNotRetriable;\n  }\n\n  int getIndexingSuccess() {\n    return indexingSuccess;\n  }\n\n  @Override\n  public String toString() {\n    return String.format(\"[calls: %,d, success: %,d, fail not-retryable: %,d, fail retryable: %,d]\",\n        indexingCalls, indexingSuccess, failureNotRetriable, failureRetriable);\n  }\n}\n\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/InstrumentedQueue.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.util.concurrent.ConcurrentLinkedDeque;\nimport java.util.concurrent.atomic.AtomicLong;\n\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchRateCounter;\n\n/**\n * A queue with metrics on size, enqueue rate and dequeue rate.\n */\npublic class InstrumentedQueue<T> {\n  private final SearchRateCounter enqueueRate;\n  private final SearchRateCounter dequeueRate;\n  private final AtomicLong queueSize = new AtomicLong();\n\n  private final ConcurrentLinkedDeque<T> queue;\n\n  public InstrumentedQueue(String statsPrefix) {\n    SearchLongGauge.export(statsPrefix + \"_size\", queueSize);\n    enqueueRate = SearchRateCounter.export(statsPrefix + \"_enqueue\");\n    dequeueRate = SearchRateCounter.export(statsPrefix + \"_dequeue\");\n\n    queue = new ConcurrentLinkedDeque<>();\n  }\n\n  /**\n   * Adds a new element to the queue.\n   */\n  public void add(T tve) {\n    queue.add(tve);\n    enqueueRate.increment();\n    queueSize.incrementAndGet();\n  }\n\n  /**\n   * Returns the first element in the queue. If the queue is empty, {@code null} is returned.\n   */\n  public T poll() {\n    T tve = queue.poll();\n    if (tve != null) {\n      dequeueRate.increment();\n      queueSize.decrementAndGet();\n    }\n    return tve;\n  }\n\n  public long getQueueSize() {\n    return queueSize.get();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/KafkaStartup.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.Closeable;\nimport java.util.Optional;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Stopwatch;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.earlybird.EarlybirdStatus;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.exception.EarlybirdStartupException;\nimport com.twitter.search.earlybird.partition.freshstartup.FreshStartupHandler;\nimport com.twitter.search.earlybird.querycache.QueryCacheManager;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusCode;\nimport com.twitter.search.queryparser.query.QueryParserException;\n\n/**\n * Handles starting an Earlybird from Kafka topics.\n *\n * Currently very unoptimized -- future versions will implement parallel indexing and loading\n * serialized data from HDFS. See http://go/removing-dl-tdd.\n */\npublic class KafkaStartup implements EarlybirdStartup {\n  private static final Logger LOG = LoggerFactory.getLogger(KafkaStartup.class);\n\n  private final EarlybirdKafkaConsumer earlybirdKafkaConsumer;\n  private final StartupUserEventIndexer startupUserEventIndexer;\n  private final QueryCacheManager queryCacheManager;\n  private final SegmentManager segmentManager;\n  private final EarlybirdIndexLoader earlybirdIndexLoader;\n  private final FreshStartupHandler freshStartupHandler;\n  private final UserUpdatesStreamIndexer userUpdatesStreamIndexer;\n  private final UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer;\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n  private final SearchLongGauge loadedIndex;\n  private final SearchLongGauge freshStartup;\n  private final MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager;\n  private final AudioSpaceEventsStreamIndexer audioSpaceEventsStreamIndexer;\n  private final CriticalExceptionHandler earlybirdExceptionHandler;\n  private final SearchDecider decider;\n\n  private static final String FRESH_STARTUP = \"fresh startup\";\n  private static final String INGEST_UNTIL_CURRENT = \"ingest until current\";\n  private static final String LOAD_FLUSHED_INDEX = \"load flushed index\";\n  private static final String SETUP_QUERY_CACHE = \"setting up query cache\";\n  private static final String USER_UPDATES_STARTUP = \"user updates startup\";\n  private static final String AUDIO_SPACES_STARTUP = \"audio spaces startup\";\n  private static final String BUILD_MULTI_SEGMENT_TERM_DICTIONARY =\n          \"build multi segment term dictionary\";\n\n  public KafkaStartup(\n      SegmentManager segmentManager,\n      EarlybirdKafkaConsumer earlybirdKafkaConsumer,\n      StartupUserEventIndexer startupUserEventIndexer,\n      UserUpdatesStreamIndexer userUpdatesStreamIndexer,\n      UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer,\n      AudioSpaceEventsStreamIndexer audioSpaceEventsStreamIndexer,\n      QueryCacheManager queryCacheManager,\n      EarlybirdIndexLoader earlybirdIndexLoader,\n      FreshStartupHandler freshStartupHandler,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager,\n      CriticalExceptionHandler earlybirdExceptionHandler,\n      SearchDecider decider\n  ) {\n    this.segmentManager = segmentManager;\n    this.earlybirdKafkaConsumer = earlybirdKafkaConsumer;\n    this.startupUserEventIndexer = startupUserEventIndexer;\n    this.queryCacheManager = queryCacheManager;\n    this.earlybirdIndexLoader = earlybirdIndexLoader;\n    this.freshStartupHandler = freshStartupHandler;\n    this.userUpdatesStreamIndexer = userUpdatesStreamIndexer;\n    this.userScrubGeoEventStreamIndexer = userScrubGeoEventStreamIndexer;\n    this.audioSpaceEventsStreamIndexer = audioSpaceEventsStreamIndexer;\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n    this.loadedIndex = SearchLongGauge.export(\"kafka_startup_loaded_index\");\n    this.freshStartup = SearchLongGauge.export(\"fresh_startup\");\n    this.multiSegmentTermDictionaryManager = multiSegmentTermDictionaryManager;\n    this.earlybirdExceptionHandler = earlybirdExceptionHandler;\n    this.decider = decider;\n    freshStartup.set(0);\n  }\n\n  private void userEventsStartup() {\n    LOG.info(\"Start indexing user events.\");\n\n    startupUserEventIndexer.indexAllEvents();\n\n    LOG.info(\"Finished loading/indexing user events.\");\n\n    // User updates are now current, keep them current by continuing to index from the stream.\n    LOG.info(\"Starting to run UserUpdatesStreamIndexer\");\n    new Thread(userUpdatesStreamIndexer::run, \"userupdates-stream-indexer\").start();\n\n    if (EarlybirdConfig.consumeUserScrubGeoEvents()) {\n      // User scrub geo events are now current,\n      // keep them current by continuing to index from the stream.\n      LOG.info(\"Starting to run UserScrubGeoEventsStreamIndexer\");\n      new Thread(userScrubGeoEventStreamIndexer::run,\n          \"userScrubGeoEvents-stream-indexer\").start();\n    }\n  }\n\n  private void loadAudioSpaceEvents() {\n    LOG.info(\"Index audio space events...\");\n    EarlybirdStatus.beginEvent(AUDIO_SPACES_STARTUP,\n        searchIndexingMetricSet.startupInAudioSpaceEventIndexer);\n\n    if (audioSpaceEventsStreamIndexer == null) {\n      LOG.error(\"Null audioSpaceEventsStreamIndexer\");\n      return;\n    }\n\n    if (decider.isAvailable(\"enable_reading_audio_space_events\")) {\n      Stopwatch stopwatch = Stopwatch.createStarted();\n      audioSpaceEventsStreamIndexer.seekToBeginning();\n      audioSpaceEventsStreamIndexer.readRecordsUntilCurrent();\n      LOG.info(\"Finished reading audio spaces in {}\", stopwatch);\n      audioSpaceEventsStreamIndexer.printSummary();\n\n      new Thread(audioSpaceEventsStreamIndexer::run,\n          \"audioSpaceEvents-stream-indexer\").start();\n    } else {\n      LOG.info(\"Reading audio space events not enabled\");\n    }\n\n    EarlybirdStatus.endEvent(AUDIO_SPACES_STARTUP,\n        searchIndexingMetricSet.startupInAudioSpaceEventIndexer);\n  }\n\n  private void tweetsAndUpdatesStartup() throws EarlybirdStartupException {\n    LOG.info(\"Index tweets and updates...\");\n    EarlybirdStatus.beginEvent(LOAD_FLUSHED_INDEX,\n        searchIndexingMetricSet.startupInLoadFlushedIndex);\n    EarlybirdIndex index;\n\n    // Set when you want to get a server from starting to ready quickly for development\n    // purposes.\n    boolean fastDevStartup = EarlybirdConfig.getBool(\"fast_dev_startup\");\n\n    Optional<EarlybirdIndex> optIndex = Optional.empty();\n    if (!fastDevStartup) {\n      optIndex = earlybirdIndexLoader.loadIndex();\n    }\n\n    if (optIndex.isPresent()) {\n      loadedIndex.set(1);\n      LOG.info(\"Loaded an index.\");\n      index = optIndex.get();\n      EarlybirdStatus.endEvent(LOAD_FLUSHED_INDEX,\n          searchIndexingMetricSet.startupInLoadFlushedIndex);\n    } else {\n      LOG.info(\"Didn't load an index, indexing from scratch.\");\n      freshStartup.set(1);\n      boolean parallelIndexFromScratch = EarlybirdConfig.getBool(\n          \"parallel_index_from_scratch\");\n      LOG.info(\"parallel_index_from_scratch: {}\", parallelIndexFromScratch);\n      EarlybirdStatus.beginEvent(FRESH_STARTUP,\n          searchIndexingMetricSet.startupInFreshStartup);\n      try {\n        if (fastDevStartup) {\n          index = freshStartupHandler.fastIndexFromScratchForDevelopment();\n        } else if (parallelIndexFromScratch) {\n          index = freshStartupHandler.parallelIndexFromScratch();\n        } else {\n          index = freshStartupHandler.indexFromScratch();\n        }\n      } catch (Exception ex) {\n        throw new EarlybirdStartupException(ex);\n      } finally {\n        EarlybirdStatus.endEvent(FRESH_STARTUP,\n            searchIndexingMetricSet.startupInFreshStartup);\n      }\n    }\n\n    LOG.info(\"Index has {} segments.\", index.getSegmentInfoList().size());\n    if (index.getSegmentInfoList().size() > 0) {\n      LOG.info(\"Inserting segments into SegmentManager\");\n      for (SegmentInfo segmentInfo : index.getSegmentInfoList()) {\n        segmentManager.putSegmentInfo(segmentInfo);\n      }\n\n      earlybirdKafkaConsumer.prepareAfterStartingWithIndex(\n          index.getMaxIndexedTweetId()\n      );\n    }\n\n    // Build the Multi segment term dictionary before catching up on indexing to ensure that the\n    // segments won't roll and delete the oldest segment while a multi segment term dictionary that\n    // includes that segment is being built.\n    buildMultiSegmentTermDictionary();\n\n    segmentManager.logState(\"Starting ingestUntilCurrent\");\n    LOG.info(\"partial updates indexed: {}\", segmentManager.getNumPartialUpdates());\n    EarlybirdStatus.beginEvent(INGEST_UNTIL_CURRENT,\n        searchIndexingMetricSet.startupInIngestUntilCurrent);\n\n    earlybirdKafkaConsumer.ingestUntilCurrent(index.getTweetOffset(), index.getUpdateOffset());\n\n    validateSegments();\n    segmentManager.logState(\"ingestUntilCurrent is done\");\n    LOG.info(\"partial updates indexed: {}\", segmentManager.getNumPartialUpdates());\n    EarlybirdStatus.endEvent(INGEST_UNTIL_CURRENT,\n        searchIndexingMetricSet.startupInIngestUntilCurrent);\n    new Thread(earlybirdKafkaConsumer::run, \"earlybird-kafka-consumer\").start();\n  }\n\n  protected void validateSegments() throws EarlybirdStartupException {\n    if (!Config.environmentIsTest()) {\n      // Unfortunately, many tests start Earlybirds with 0 indexed documents, so we disable this\n      // check in tests.\n      validateSegmentsForNonTest();\n    }\n  }\n\n  protected void validateSegmentsForNonTest() throws EarlybirdStartupException {\n    // SEARCH-24123: Prevent Earlybird from starting if there are no indexed documents.\n    if (segmentManager.getNumIndexedDocuments() == 0) {\n      throw new EarlybirdStartupException(\"Earlybird has zero indexed documents.\");\n    }\n  }\n\n  private void queryCacheStartup() throws EarlybirdStartupException {\n    EarlybirdStatus.beginEvent(SETUP_QUERY_CACHE,\n        searchIndexingMetricSet.startupInQueryCacheUpdates);\n    try {\n      queryCacheManager.setupTasksIfNeeded(segmentManager);\n    } catch (QueryParserException e) {\n      LOG.error(\"Exception when setting up query cache tasks\");\n      throw new EarlybirdStartupException(e);\n    }\n\n    queryCacheManager.waitUntilAllQueryCachesAreBuilt();\n\n    // Print the sizes of the query caches so that we can see that they're built.\n    Iterable<SegmentInfo> segmentInfos =\n        segmentManager.getSegmentInfos(SegmentManager.Filter.All, SegmentManager.Order.OLD_TO_NEW);\n    segmentManager.logState(\"After building query caches\");\n    for (SegmentInfo segmentInfo : segmentInfos) {\n      LOG.info(\"Segment: {}, Total cardinality: {}\", segmentInfo.getSegmentName(),\n          segmentInfo.getIndexSegment().getQueryCachesCardinality());\n    }\n\n    // We're done building the query caches for all segments, and the earlybird is ready to become\n    // current. Restrict all future query cache task runs to one single core, to make sure our\n    // searcher threads are not impacted.\n    queryCacheManager.setWorkerPoolSizeAfterStartup();\n    EarlybirdStatus.endEvent(SETUP_QUERY_CACHE,\n        searchIndexingMetricSet.startupInQueryCacheUpdates);\n  }\n\n  /**\n   * Closes all currently running Indexers.\n   */\n  @VisibleForTesting\n  public void shutdownIndexing() {\n    LOG.info(\"Shutting down KafkaStartup.\");\n\n    earlybirdKafkaConsumer.close();\n    userUpdatesStreamIndexer.close();\n    userScrubGeoEventStreamIndexer.close();\n    // Note that the QueryCacheManager is shut down in EarlybirdServer::shutdown.\n  }\n\n  private void buildMultiSegmentTermDictionary() {\n    EarlybirdStatus.beginEvent(BUILD_MULTI_SEGMENT_TERM_DICTIONARY,\n            searchIndexingMetricSet.startupInMultiSegmentTermDictionaryUpdates);\n    Stopwatch stopwatch = Stopwatch.createStarted();\n    LOG.info(\"Building multi segment term dictionary\");\n    multiSegmentTermDictionaryManager.buildDictionary();\n    LOG.info(\"Done with building multi segment term dictionary in {}\", stopwatch);\n    EarlybirdStatus.endEvent(BUILD_MULTI_SEGMENT_TERM_DICTIONARY,\n            searchIndexingMetricSet.startupInMultiSegmentTermDictionaryUpdates);\n  }\n\n  private void parallelIndexingStartup() throws EarlybirdStartupException {\n    Thread userEventsThread = new Thread(this::userEventsStartup, \"index-user-events-startup\");\n    Thread tweetsAndUpdatesThread = new Thread(() -> {\n      try {\n        tweetsAndUpdatesStartup();\n      } catch (EarlybirdStartupException e) {\n        earlybirdExceptionHandler.handle(this, e);\n      }\n    }, \"index-tweets-and-updates-startup\");\n    Thread audioSpaceEventsThread = new Thread(this::loadAudioSpaceEvents,\n        \"index-audio-space-events-startup\");\n    userEventsThread.start();\n    tweetsAndUpdatesThread.start();\n    audioSpaceEventsThread.start();\n\n    try {\n      userEventsThread.join();\n    } catch (InterruptedException e) {\n      throw new EarlybirdStartupException(\"Interrupted while indexing user events\");\n    }\n    try {\n      tweetsAndUpdatesThread.join();\n    } catch (InterruptedException e) {\n      throw new EarlybirdStartupException(\"Interrupted while indexing tweets and updates\");\n    }\n    try {\n      audioSpaceEventsThread.join();\n    } catch (InterruptedException e) {\n      throw new EarlybirdStartupException(\"Interrupted while indexing audio space events\");\n    }\n  }\n\n  /**\n   * Does startups and starts indexing. Returns when the earlybird\n   * is current.\n   */\n  @Override\n  public Closeable start() throws EarlybirdStartupException {\n    parallelIndexingStartup();\n    queryCacheStartup();\n\n    EarlybirdStatus.setStatus(EarlybirdStatusCode.CURRENT);\n\n    return this::shutdownIndexing;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/MultiSegmentTermDictionaryManager.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.concurrent.TimeUnit;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.decider.DeciderUtil;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.core.earlybird.index.inverted.InvertedIndex;\nimport com.twitter.search.core.earlybird.index.inverted.MultiSegmentTermDictionary;\nimport com.twitter.search.core.earlybird.index.inverted.MultiSegmentTermDictionaryWithFastutil;\nimport com.twitter.search.core.earlybird.index.inverted.OptimizedMemoryIndex;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.index.EarlybirdSegment;\nimport com.twitter.search.earlybird.partition.SegmentManager.Filter;\nimport com.twitter.search.earlybird.partition.SegmentManager.Order;\n\n/**\n * Manages MultiSegmentTermDictionary's for specific fields on this earlybird. Only manages them\n * for optimized segments, and should only regenerate new dictionaries when the list of optimized\n * segments changes. See SEARCH-10836\n */\npublic class MultiSegmentTermDictionaryManager {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(MultiSegmentTermDictionaryManager.class);\n\n  @VisibleForTesting\n  public static final SearchTimerStats TERM_DICTIONARY_CREATION_STATS =\n      SearchTimerStats.export(\"multi_segment_term_dictionary_manager_build_dictionary\",\n          TimeUnit.MILLISECONDS, false);\n\n  public static final MultiSegmentTermDictionaryManager NOOP_INSTANCE =\n      new MultiSegmentTermDictionaryManager(\n          new Config(Collections.emptyList()), null, null, null, null) {\n        @Override\n        public boolean buildDictionary() {\n          return false;\n        }\n      };\n\n  private static final String MANAGER_DISABLED_DECIDER_KEY_PREFIX =\n      \"multi_segment_term_dictionary_manager_disabled_in_\";\n\n  public static class Config {\n    private final ImmutableList<String> fieldNames;\n\n    public Config(List<String> fieldNames) {\n      Preconditions.checkNotNull(fieldNames);\n      this.fieldNames = ImmutableList.copyOf(fieldNames);\n    }\n\n    public List<String> managedFieldNames() {\n      return fieldNames;\n    }\n\n    public boolean isEnabled() {\n      return EarlybirdConfig.getBool(\"multi_segment_term_dictionary_enabled\", false);\n    }\n  }\n\n  @VisibleForTesting\n  public static String getManagerDisabledDeciderName(EarlybirdCluster earlybirdCluster) {\n    return MANAGER_DISABLED_DECIDER_KEY_PREFIX + earlybirdCluster.name().toLowerCase();\n  }\n\n  private static final class FieldStats {\n    private final SearchTimerStats buildTime;\n    private final SearchLongGauge numTerms;\n    private final SearchLongGauge numTermEntries;\n\n    private FieldStats(SearchStatsReceiver statsReceiver, String fieldName) {\n      Preconditions.checkNotNull(fieldName);\n      Preconditions.checkNotNull(statsReceiver);\n\n      String timerName = String.format(\n          \"multi_segment_term_dictionary_manager_field_%s_build_dictionary\", fieldName);\n      this.buildTime = statsReceiver.getTimerStats(\n          timerName, TimeUnit.MILLISECONDS, false, false, false);\n\n      String numTermsName = String.format(\n          \"multi_segment_term_dictionary_manager_field_%s_num_terms\", fieldName);\n      this.numTerms = statsReceiver.getLongGauge(numTermsName);\n\n      String numTermEntriesName = String.format(\n          \"multi_segment_term_dictionary_manager_field_%s_num_term_entries\", fieldName);\n      this.numTermEntries = statsReceiver.getLongGauge(numTermEntriesName);\n    }\n  }\n\n  private final Config config;\n  @Nullable private final SegmentManager segmentManager;\n  @Nullable private final Decider decider;\n  @Nullable private final EarlybirdCluster earlybirdCluster;\n  private final ImmutableMap<String, FieldStats> fieldTimerStats;\n  // A per-field map of multi-segment term dictionaries. Each key is a field. The values are the\n  // multi-segment term dictionaries for that field.\n  private volatile ImmutableMap<String, MultiSegmentTermDictionary> multiSegmentTermDictionaryMap;\n  private List<SegmentInfo> previousSegmentsToMerge;\n\n  public MultiSegmentTermDictionaryManager(\n      Config config,\n      SegmentManager segmentManager,\n      SearchStatsReceiver statsReceiver,\n      Decider decider,\n      EarlybirdCluster earlybirdCluster) {\n    this.config = config;\n    this.segmentManager = segmentManager;\n    this.decider = decider;\n    this.earlybirdCluster = earlybirdCluster;\n\n    this.multiSegmentTermDictionaryMap = ImmutableMap.of();\n    this.previousSegmentsToMerge = Lists.newArrayList();\n\n    ImmutableMap.Builder<String, FieldStats> builder = ImmutableMap.builder();\n    if (statsReceiver != null) {\n      for (String fieldName : config.managedFieldNames()) {\n        builder.put(fieldName, new FieldStats(statsReceiver, fieldName));\n      }\n    }\n    this.fieldTimerStats = builder.build();\n  }\n\n  /**\n   * Return the most recently built MultiSegmentTermDictionary for the given field.\n   * Will return null if the field is not supported by this manager.\n   */\n  @Nullable\n  public MultiSegmentTermDictionary getMultiSegmentTermDictionary(String fieldName) {\n    return this.multiSegmentTermDictionaryMap.get(fieldName);\n  }\n\n  /**\n   * Build new versions of multi-segment term dictionaries if the manager is enabled, and new\n   * segments are available.\n   * @return true if the manager actually ran, and generated new versions of multi-segment term\n   * dictionaries.\n   *\n   * We synchronize this method because it would be a logic error to modify the variables from\n   * multiple threads simultaneously, and it is possible for two segments to finish optimizing at\n   * the same time and try to run it.\n   */\n  public synchronized boolean buildDictionary() {\n    if (!config.isEnabled()) {\n      return false;\n    }\n\n    Preconditions.checkNotNull(decider);\n    Preconditions.checkNotNull(earlybirdCluster);\n    if (DeciderUtil.isAvailableForRandomRecipient(decider,\n        getManagerDisabledDeciderName(earlybirdCluster))) {\n      LOG.info(\"Multi segment term dictionary manager is disabled via decider for cluster {}.\",\n          earlybirdCluster);\n      this.multiSegmentTermDictionaryMap = ImmutableMap.of();\n      this.previousSegmentsToMerge = Lists.newArrayList();\n      return false;\n    }\n\n    List<SegmentInfo> segmentsToMerge = getSegmentsToMerge();\n\n    if (differentFromPreviousList(segmentsToMerge)) {\n       long start = System.currentTimeMillis();\n       try {\n         this.multiSegmentTermDictionaryMap = createNewDictionaries(segmentsToMerge);\n         this.previousSegmentsToMerge = segmentsToMerge;\n         return true;\n       } catch (IOException e) {\n         LOG.error(\"Unable to build multi segment term dictionaries\", e);\n         return false;\n       } finally {\n         long elapsed = System.currentTimeMillis() - start;\n         TERM_DICTIONARY_CREATION_STATS.timerIncrement(elapsed);\n       }\n    } else {\n      LOG.warn(\"No-op for buildDictionary()\");\n      return false;\n    }\n  }\n\n  /**\n   * Only merge terms from enabled and optimized segments. No need to look at non-enabled segments,\n   * and we also don't want to use un-optimized segments as their term dictionaries are still\n   * changing.\n   */\n  private List<SegmentInfo> getSegmentsToMerge() {\n    Iterable<SegmentInfo> segmentInfos =\n        segmentManager.getSegmentInfos(Filter.Enabled, Order.OLD_TO_NEW);\n\n    List<SegmentInfo> segmentsToMerge = Lists.newArrayList();\n    for (SegmentInfo segmentInfo : segmentInfos) {\n      if (segmentInfo.getIndexSegment().isOptimized()) {\n        segmentsToMerge.add(segmentInfo);\n      }\n    }\n    return segmentsToMerge;\n  }\n\n  private boolean differentFromPreviousList(List<SegmentInfo> segmentsToMerge) {\n    // there is a potentially different approach here to only check if the\n    // segmentsToMerge is subsumed by the previousSegmentsToMerge list, and not recompute\n    // the multi segment term dictionary if so.\n    // There is a case where a new segment is added, the previously current segment is not yet\n    // optimized, but the oldest segment is dropped. With this impl, we will recompute to remove\n    // the dropped segment, however, we will recompute soon again when the\n    // \"previously current segment\" is actually optimized. We can potentially delay the first\n    // merging before the optimization.\n    if (this.previousSegmentsToMerge.size() == segmentsToMerge.size()) {\n      for (int i = 0; i < this.previousSegmentsToMerge.size(); i++) {\n        if (previousSegmentsToMerge.get(i).compareTo(segmentsToMerge.get(i)) != 0) {\n          return true;\n        }\n      }\n      return false;\n    }\n    return true;\n  }\n\n  /**\n   * Rebuild the term dictionaries from scratch for all the managed fields.\n   * Returning a brand new map here with all the fields' term dictionaries so that we can isolate\n   * failures to build, and only replace the entire map of all the fields are built successfully.\n   */\n  private ImmutableMap<String, MultiSegmentTermDictionary> createNewDictionaries(\n      List<SegmentInfo> segments) throws IOException {\n\n    Map<String, MultiSegmentTermDictionary> map = Maps.newHashMap();\n\n    for (String field : config.managedFieldNames()) {\n      LOG.info(\"Merging term dictionaries for field {}\", field);\n\n      List<OptimizedMemoryIndex> indexesToMerge = findFieldIndexesToMerge(segments, field);\n\n      if (indexesToMerge.isEmpty()) {\n        LOG.info(\"No indexes to merge for field {}\", field);\n      } else {\n        long start = System.currentTimeMillis();\n\n        MultiSegmentTermDictionary multiSegmentTermDictionary =\n            mergeDictionaries(field, indexesToMerge);\n\n        map.put(field, multiSegmentTermDictionary);\n\n        long elapsed = System.currentTimeMillis() - start;\n        LOG.info(\"Done merging term dictionary for field {}, for {} segments in {}ms\",\n            field, indexesToMerge.size(), elapsed);\n\n        FieldStats fieldStats = fieldTimerStats.get(field);\n        fieldStats.buildTime.timerIncrement(elapsed);\n        fieldStats.numTerms.set(multiSegmentTermDictionary.getNumTerms());\n        fieldStats.numTermEntries.set(multiSegmentTermDictionary.getNumTermEntries());\n      }\n    }\n    return ImmutableMap.copyOf(map);\n  }\n\n  private List<OptimizedMemoryIndex> findFieldIndexesToMerge(\n      List<SegmentInfo> segments, String field) throws IOException {\n\n    List<OptimizedMemoryIndex> indexesToMerge = Lists.newArrayList();\n\n    for (SegmentInfo segment : segments) {\n      EarlybirdSegment indexSegment = segment.getIndexSegment();\n      Preconditions.checkState(indexSegment.isOptimized(),\n          \"Expect segment to be optimized: %s\", segment);\n\n      InvertedIndex fieldIndex = Preconditions.checkNotNull(indexSegment.getIndexReader())\n          .getSegmentData().getFieldIndex(field);\n\n      // See SEARCH-11952\n      // We will only have a InvertedIndex/OptimizedMemoryIndex here\n      // in the in-memory non-lucene-based indexes, and not in the archive. We can somewhat\n      // reasonably extend this to work with the archive by making the dictionaries work with\n      // TermsEnum's directly instead of OptimizedMemoryIndex's. Leaving this as a further\n      // extension for now.\n      if (fieldIndex != null) {\n        if (fieldIndex instanceof OptimizedMemoryIndex) {\n          indexesToMerge.add((OptimizedMemoryIndex) fieldIndex);\n        } else {\n          LOG.info(\"Found field index for field {} in segment {} of type {}\",\n              field, segment, fieldIndex.getClass());\n        }\n      } else {\n        LOG.info(\"Found null field index for field {} in segment {}\", field, segment);\n      }\n    }\n    LOG.info(\"Found good fields for {} out of {} segments\", indexesToMerge.size(),\n            segments.size());\n\n    return indexesToMerge;\n  }\n\n  private MultiSegmentTermDictionary mergeDictionaries(\n      String field,\n      List<OptimizedMemoryIndex> indexes) {\n    // May change this if we get a better implementation in the future.\n    return new MultiSegmentTermDictionaryWithFastutil(field, indexes);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/OptimizationAndFlushingCoordinationLock.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.util.concurrent.locks.ReentrantLock;\n\nimport com.google.common.annotations.VisibleForTesting;\n\n/**\n * Lock used to ensure that flushing does not occur concurrently with the gc_before_optimization\n * and post_optimization_rebuilds actions - see where we call the \"lock\" method of this class.\n *\n * Both coordinated actions include a full GC in them, for reasons described in that part\n * of the code. After the GC, they wait until indexing has caught up before rejoining the serverset.\n *\n * If we flush concurrently with these actions, we can pause indexing for a while and waiting\n * until we're caught up can take some time, which can affect the memory state negatively.\n * For example, the first GC (before optimization) we do so that we have a clean state of memory\n * before optimization.\n *\n * The other reason we lock before executing the actions is because if we have flushing that's\n * currently running, once it finishes, we will rejoin the serverset and that can be followed by\n * a stop-the-world GC from the actions, which will affect our success rate.\n */\npublic class OptimizationAndFlushingCoordinationLock {\n  private final ReentrantLock lock;\n\n  public OptimizationAndFlushingCoordinationLock() {\n    this.lock = new ReentrantLock();\n  }\n\n  public void lock() {\n    lock.lock();\n  }\n\n  public void unlock() {\n    lock.unlock();\n  }\n\n  public boolean tryLock() {\n    return lock.tryLock();\n  }\n\n  @VisibleForTesting\n  public boolean hasQueuedThreads() {\n    return lock.hasQueuedThreads();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/OptimizingSegmentWriter.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\nimport java.util.concurrent.ConcurrentLinkedQueue;\nimport java.util.concurrent.atomic.AtomicReference;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Stopwatch;\nimport com.google.common.base.Verify;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.util.GCUtil;\nimport com.twitter.search.earlybird.EarlybirdStatus;\nimport com.twitter.search.earlybird.common.CaughtUpMonitor;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.index.EarlybirdSegment;\nimport com.twitter.search.earlybird.util.CoordinatedEarlybirdActionInterface;\nimport com.twitter.util.Future;\nimport com.twitter.util.Promise;\n\n/**\n * This class optimizes a segment without blocking reads or writes.\n *\n * In steady state operation (Indexing or Optimized), it delegates operations directly to a\n * SegmentWriter.\n *\n * Optimization is naturally a copying operation -- we don't need to mutate anything internally.\n * We need to be able to apply updates to the unoptimized segment while we are creating\n * the optimized segment. We also need to be able to apply these updates to the optimized segment,\n * but we can't apply updates while a segment is being optimized, because document IDs will be\n * changing internally and posting lists could be any state. To deal with this, we queue updates\n * that occur during optimization, and then apply them as the last step of optimization. At that\n * point, the segment will be optimized and up to date, so we can swap the unoptimized segment for\n * the optimized one.\n */\npublic class OptimizingSegmentWriter implements ISegmentWriter {\n  private static final Logger LOG = LoggerFactory.getLogger(OptimizingSegmentWriter.class);\n\n  private final AtomicReference<State> state = new AtomicReference<>(State.Indexing);\n  private final ConcurrentLinkedQueue<ThriftVersionedEvents> queuedEvents =\n      new ConcurrentLinkedQueue<>();\n\n  private final CriticalExceptionHandler criticalExceptionHandler;\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n  private final String segmentName;\n  private final Promise<SegmentInfo> optimizationPromise = new Promise<>();\n\n  // We use the lock to ensure that the optimizing thread and the writer thread do not attempt\n  // to call indexThriftVersionedEvents on the underlying writer simultaneously.\n  private final Object lock = new Object();\n  // The reference to the current writer. Protected by lock.\n  private final AtomicReference<SegmentWriter> segmentWriterReference;\n\n  private final CaughtUpMonitor indexCaughtUpMonitor;\n\n  /**\n   * The state flow:\n   * Indexing -> Optimizing ->\n   * ONE OF:\n   * - Optimized\n   * - FailedToOptimize\n   */\n  @VisibleForTesting\n  enum State {\n    Indexing,\n    Optimizing,\n    FailedToOptimize,\n    Optimized,\n  }\n\n  public OptimizingSegmentWriter(\n      SegmentWriter segmentWriter,\n      CriticalExceptionHandler criticalExceptionHandler,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      CaughtUpMonitor indexCaughtUpMonitor\n  ) {\n    Preconditions.checkState(!segmentWriter.getSegmentInfo().isOptimized());\n    segmentWriterReference = new AtomicReference<>(segmentWriter);\n\n    this.criticalExceptionHandler = criticalExceptionHandler;\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n    this.segmentName = segmentWriter.getSegmentInfo().getSegmentName();\n    this.indexCaughtUpMonitor = indexCaughtUpMonitor;\n  }\n\n  /**\n   * Start optimizing this segment in the background. Returns a Future that will complete when\n   * the optimization is complete.\n   * Acquires the optimizationAndFlushingCoordinationLock before attempting to optimize.\n   */\n  public Future<SegmentInfo> startOptimization(\n      CoordinatedEarlybirdActionInterface gcAction,\n      OptimizationAndFlushingCoordinationLock optimizationAndFlushingCoordinationLock) {\n    new Thread(() -> {\n      // Acquire lock to ensure that flushing is not in progress. If the lock is not available,\n      // then wait until it is.\n      LOG.info(\"Acquire coordination lock before beginning gc_before_optimization action.\");\n      try {\n        optimizationAndFlushingCoordinationLock.lock();\n        LOG.info(\"Successfully acquired coordination lock for gc_before_optimization action.\");\n        gcAction.retryActionUntilRan(\"gc before optimization\", () -> {\n          LOG.info(\"Run GC before optimization\");\n          GCUtil.runGC();\n          // Wait for indexing to catch up before gcAction rejoins the serverset. We only need to do\n          // this if the host has already finished startup.\n          if (EarlybirdStatus.hasStarted()) {\n            indexCaughtUpMonitor.resetAndWaitUntilCaughtUp();\n          }\n        });\n      } finally {\n        LOG.info(\"Finished gc_before_optimization action. \"\n            + \"Releasing coordination lock and beginning optimization.\");\n        optimizationAndFlushingCoordinationLock.unlock();\n      }\n\n      transition(State.Indexing, State.Optimizing);\n\n      SegmentInfo unoptimizedSegmentInfo = null;\n      try {\n        unoptimizedSegmentInfo = segmentWriterReference.get().getSegmentInfo();\n        Preconditions.checkState(!unoptimizedSegmentInfo.isOptimized());\n\n        Stopwatch stopwatch = Stopwatch.createStarted();\n        LOG.info(\"Started optimizing segment data {}.\", segmentName);\n        EarlybirdSegment optimizedSegment =\n            unoptimizedSegmentInfo.getIndexSegment().makeOptimizedSegment();\n        LOG.info(\"Finished optimizing segment data {} in {}.\", segmentName, stopwatch);\n\n        SegmentInfo newSegmentInfo = unoptimizedSegmentInfo\n            .copyWithEarlybirdSegment(optimizedSegment);\n\n        SegmentWriter optimizedWriter =\n            new SegmentWriter(newSegmentInfo, searchIndexingMetricSet.updateFreshness);\n        Verify.verify(optimizedWriter.getSegmentInfo().isOptimized());\n\n        // We want to apply all updates to the new segment twice, because this first call may apply\n        // many thousands of updates and take a while to complete.\n        applyAllPendingUpdates(optimizedWriter);\n\n        // We try to do as little as possible while holding the lock, so the writer can continue\n        // to make progress. First we apply all the updates that have been queued up before we\n        // grabbed the lock, then we need to swap the new writer for the old one.\n        synchronized (lock) {\n          applyAllPendingUpdates(optimizedWriter);\n          segmentWriterReference.getAndSet(optimizedWriter);\n          transition(State.Optimizing, State.Optimized);\n        }\n\n        if (!unoptimizedSegmentInfo.isEnabled()) {\n          LOG.info(\"Disabling segment: {}\", unoptimizedSegmentInfo.getSegmentName());\n          newSegmentInfo.setIsEnabled(false);\n        }\n\n        optimizationPromise.setValue(newSegmentInfo);\n      } catch (Throwable e) {\n        if (unoptimizedSegmentInfo != null) {\n          unoptimizedSegmentInfo.setFailedOptimize();\n        }\n\n        transition(State.Optimizing, State.FailedToOptimize);\n        optimizationPromise.setException(e);\n      }\n    }, \"optimizing-segment-writer\").start();\n\n    return optimizationPromise;\n  }\n\n  private void applyAllPendingUpdates(SegmentWriter segmentWriter) throws IOException {\n    LOG.info(\"Applying {} queued updates to segment {}.\", queuedEvents.size(), segmentName);\n    // More events can be enqueued while this method is running, so we track the total applied too.\n    long eventCount = 0;\n    Stopwatch stopwatch = Stopwatch.createStarted();\n    ThriftVersionedEvents update;\n    while ((update = queuedEvents.poll()) != null) {\n      segmentWriter.indexThriftVersionedEvents(update);\n      eventCount++;\n    }\n    LOG.info(\"Applied {} queued updates to segment {} in {}.\",\n        eventCount, segmentName, stopwatch);\n  }\n\n  @Override\n  public Result indexThriftVersionedEvents(ThriftVersionedEvents tve) throws IOException {\n    synchronized (lock) {\n      if (state.get() == State.Optimizing) {\n        queuedEvents.add(tve);\n      }\n      return segmentWriterReference.get().indexThriftVersionedEvents(tve);\n    }\n  }\n\n  @Override\n  public SegmentInfo getSegmentInfo() {\n    return segmentWriterReference.get().getSegmentInfo();\n  }\n\n  private void transition(State from, State to) {\n    Preconditions.checkState(state.compareAndSet(from, to));\n    LOG.info(\"Transitioned from {} to {} for segment {}.\", from, to, segmentName);\n  }\n\n  @VisibleForTesting\n  public Future<SegmentInfo> getOptimizationPromise() {\n    return optimizationPromise;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/PartitionConfig.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.util.Date;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.apache.commons.lang3.builder.ToStringBuilder;\n\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.config.TierConfig;\n\npublic class PartitionConfig {\n  // Which sub-cluster this host belongs to\n  private final String tierName;\n\n  // Which cluster this host belongs to\n  private final String clusterName;\n\n  public static final String DEFAULT_TIER_NAME = \"all\";\n\n  // the date range of the timeslices this tier will load. The start date is inclusive, while\n  // the end date is exclusive.\n  private final Date tierStartDate;\n  private final Date tierEndDate;\n\n  private final int indexingHashPartitionID; // Hash Partition ID assigned for this EB\n  private final int maxEnabledLocalSegments; // Number of segments to keep\n  // The position of this host in the ordered list of hosts serving this hash partition\n  private final int hostPositionWithinHashPartition;\n  private volatile int numReplicasInHashPartition;\n\n  private final int numPartitions; // Total number of partitions in the current cluster\n\n  public PartitionConfig(\n      int indexingHashPartitionID,\n      int maxEnabledLocalSegments,\n      int hostPositionWithinHashPartition,\n      int numReplicasInHashPartition,\n      int numPartitions) {\n    this(DEFAULT_TIER_NAME,\n        TierConfig.DEFAULT_TIER_START_DATE,\n        TierConfig.DEFAULT_TIER_END_DATE,\n        indexingHashPartitionID,\n        maxEnabledLocalSegments,\n        hostPositionWithinHashPartition,\n        numReplicasInHashPartition,\n        numPartitions);\n  }\n\n  public PartitionConfig(String tierName,\n                         Date tierStartDate,\n                         Date tierEndDate,\n                         int indexingHashPartitionID,\n                         int maxEnabledLocalSegments,\n                         int hostPositionWithinHashPartition,\n                         int numReplicasInHashPartition,\n                         int numPartitions) {\n    this(tierName, tierStartDate, tierEndDate, indexingHashPartitionID, maxEnabledLocalSegments,\n        hostPositionWithinHashPartition, numReplicasInHashPartition, Config.getEnvironment(),\n        numPartitions);\n  }\n\n  public PartitionConfig(String tierName,\n                         Date tierStartDate,\n                         Date tierEndDate,\n                         int indexingHashPartitionID,\n                         int maxEnabledLocalSegments,\n                         int hostPositionWithinHashPartition,\n                         int numReplicasInHashPartition,\n                         String clusterName,\n                         int numPartitions) {\n    this.tierName = Preconditions.checkNotNull(tierName);\n    this.clusterName = Preconditions.checkNotNull(clusterName);\n    this.tierStartDate = Preconditions.checkNotNull(tierStartDate);\n    this.tierEndDate = Preconditions.checkNotNull(tierEndDate);\n    this.indexingHashPartitionID = indexingHashPartitionID;\n    this.maxEnabledLocalSegments = maxEnabledLocalSegments;\n    this.hostPositionWithinHashPartition = hostPositionWithinHashPartition;\n    this.numReplicasInHashPartition = numReplicasInHashPartition;\n    this.numPartitions = numPartitions;\n  }\n\n  public String getTierName() {\n    return tierName;\n  }\n\n  public String getClusterName() {\n    return clusterName;\n  }\n\n  public Date getTierStartDate() {\n    return tierStartDate;\n  }\n\n  public Date getTierEndDate() {\n    return tierEndDate;\n  }\n\n  public int getIndexingHashPartitionID() {\n    return indexingHashPartitionID;\n  }\n\n  public int getMaxEnabledLocalSegments() {\n    return maxEnabledLocalSegments;\n  }\n\n  public int getHostPositionWithinHashPartition() {\n    return hostPositionWithinHashPartition;\n  }\n\n  public int getNumReplicasInHashPartition() {\n    return numReplicasInHashPartition;\n  }\n\n  /**\n   * The number of ways the Tweet and/or user data is partitioned (or sharded) in this Earlybird, in\n   * this tier.\n   */\n  public int getNumPartitions() {\n    return numPartitions;\n  }\n\n  public String getPartitionConfigDescription() {\n    return ToStringBuilder.reflectionToString(this);\n  }\n\n  public void setNumReplicasInHashPartition(int numReplicas) {\n    numReplicasInHashPartition = numReplicas;\n  }\n\n  public static final int DEFAULT_NUM_SERVING_TIMESLICES_FOR_TEST = 18;\n  public static PartitionConfig getPartitionConfigForTests() {\n    return getPartitionConfigForTests(\n        TierConfig.DEFAULT_TIER_START_DATE,\n        TierConfig.DEFAULT_TIER_END_DATE);\n  }\n\n  public static PartitionConfig getPartitionConfigForTests(Date tierStartDate, Date tierEndDate) {\n    return getPartitionConfigForTests(\n        DEFAULT_NUM_SERVING_TIMESLICES_FOR_TEST, tierStartDate, tierEndDate, 1);\n  }\n\n  /**\n   * Returns a PartitionConfig instance configured for tests.\n   *\n   * @param numServingTimeslices The number of timeslices that should be served.\n   * @param tierStartDate The tier's start date. Used only in the full archive earlybirds.\n   * @param tierEndDate The tier's end date. Used only by in the full archive earlybirds.\n   * @param numReplicasInHashPartition The number of replicas for each partition.\n   * @return A PartitionConfig instance configured for tests.\n   */\n  @VisibleForTesting\n  public static PartitionConfig getPartitionConfigForTests(\n      int numServingTimeslices,\n      Date tierStartDate,\n      Date tierEndDate,\n      int numReplicasInHashPartition) {\n    return new PartitionConfig(\n        EarlybirdConfig.getString(\"sub_tiers_for_tests\", \"test\"),\n        tierStartDate,\n        tierEndDate,\n        EarlybirdConfig.getInt(\"hash_partition_for_tests\", -1),\n        numServingTimeslices,\n        0, // hostPositionWithinHashPartition\n        numReplicasInHashPartition,\n        EarlybirdConfig.getInt(\"num_partitions_for_tests\", -1)\n    );\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/PartitionConfigLoader.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.aurora.AuroraInstanceKey;\nimport com.twitter.search.common.aurora.AuroraSchedulerClient;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.factory.PartitionConfigUtil;\n\npublic final class PartitionConfigLoader {\n  private static final Logger LOG = LoggerFactory.getLogger(PartitionConfigLoader.class);\n\n  private PartitionConfigLoader() {\n    // this never gets called\n  }\n\n  /**\n   * Load partition information from the command line arguments and Aurora scheduler.\n   *\n   * @return The new PartitionConfig object for this host\n   */\n  public static PartitionConfig getPartitionInfoForMesosConfig(\n      AuroraSchedulerClient schedulerClient) throws PartitionConfigLoadingException {\n    AuroraInstanceKey instanceKey =\n        Preconditions.checkNotNull(EarlybirdConfig.getAuroraInstanceKey());\n    int numTasks;\n\n    try {\n      numTasks = schedulerClient.getActiveTasks(\n          instanceKey.getRole(), instanceKey.getEnv(), instanceKey.getJobName()).size();\n      LOG.info(\"Found {} active tasks\", numTasks);\n    } catch (IOException e) {\n      // This can happen when Aurora Scheduler is holding a conclave to elect a new reader.\n      LOG.warn(\"Failed to get tasks from Aurora scheduler.\", e);\n      throw new PartitionConfigLoadingException(\"Failed to get tasks from Aurora scheduler.\");\n    }\n\n    return PartitionConfigUtil.initPartitionConfigForAurora(numTasks);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/PartitionConfigLoadingException.java",
    "content": "package com.twitter.search.earlybird.partition;\n\n/**\n * An exception thrown when the earlybird layout could not be loaded, or when a host cannot find\n * itself in the layout, and the layout has errors (which might be the reason why the host could not\n * find itself in the layout).\n */\npublic class PartitionConfigLoadingException extends Exception {\n  public PartitionConfigLoadingException(String message) {\n    super(message);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/PartitionManager.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.concurrent.ScheduledExecutorServiceFactory;\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.earlybird.EarlybirdStatus;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.exception.EarlybirdStartupException;\nimport com.twitter.search.earlybird.querycache.QueryCacheManager;\nimport com.twitter.search.earlybird.segment.SegmentDataProvider;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusCode;\nimport com.twitter.search.earlybird.util.OneTaskScheduledExecutorManager;\nimport com.twitter.search.earlybird.util.PeriodicActionParams;\nimport com.twitter.search.earlybird.util.ShutdownWaitTimeParams;\nimport com.twitter.search.queryparser.query.QueryParserException;\n\n/**\n * PartitionManager is responsible for indexing data for a partition, including Tweets and Users.\n */\npublic abstract class PartitionManager extends OneTaskScheduledExecutorManager {\n  private static final Logger LOG = LoggerFactory.getLogger(PartitionManager.class);\n\n  private static final SearchCounter IGNORED_EXCEPTIONS =\n      SearchCounter.export(\"partition_manager_ignored_exceptions\");\n\n  private static final String PARTITION_MANAGER_THREAD_NAME = \"PartitionManager\";\n  private static final boolean THREAD_IS_DAEMON = true;\n  protected static final String INDEX_CURRENT_SEGMENT = \"indexing the current segment\";\n  protected static final String SETUP_QUERY_CACHE = \"setting up query cache\";\n\n  protected final SegmentManager segmentManager;\n  protected final QueryCacheManager queryCacheManager;\n  // Should be updated by info read from ZK\n  protected final DynamicPartitionConfig dynamicPartitionConfig;\n\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n\n  private boolean partitionManagerFirstLoop = true;\n\n  public PartitionManager(QueryCacheManager queryCacheManager,\n                          SegmentManager segmentManager,\n                          DynamicPartitionConfig dynamicPartitionConfig,\n                          ScheduledExecutorServiceFactory executorServiceFactory,\n                          SearchIndexingMetricSet searchIndexingMetricSet,\n                          SearchStatsReceiver searchStatsReceiver,\n                          CriticalExceptionHandler criticalExceptionHandler) {\n    super(\n        executorServiceFactory,\n        PARTITION_MANAGER_THREAD_NAME,\n        THREAD_IS_DAEMON,\n        PeriodicActionParams.withFixedDelay(\n          EarlybirdConfig.getInt(\"time_slice_roll_check_interval_ms\", 500),\n          TimeUnit.MILLISECONDS),\n        ShutdownWaitTimeParams.indefinitely(),\n        searchStatsReceiver,\n        criticalExceptionHandler);\n\n    this.segmentManager = segmentManager;\n    this.queryCacheManager = queryCacheManager;\n    this.dynamicPartitionConfig = dynamicPartitionConfig;\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n  }\n\n  /**\n   * Runs the partition manager.\n   */\n  public final void runImpl() {\n    if (partitionManagerFirstLoop) {\n      try {\n        testHookBeforeStartUp();\n        startUp();\n        validateSegments();\n        segmentManager.logState(\"After startUp\");\n      } catch (Throwable t) {\n        criticalExceptionHandler.handle(this, t);\n        shutDownIndexing();\n        throw new RuntimeException(\"PartitionManager unhandled exception, stopping scheduler\", t);\n      }\n    }\n\n    try {\n      testHookAfterSleep();\n      indexingLoop(partitionManagerFirstLoop);\n    } catch (InterruptedException e) {\n      LOG.warn(\"PartitionManager thread interrupted, stoping scheduler\", e);\n      shutDownIndexing();\n      throw new RuntimeException(\"PartitionManager thread interrupted\", e);\n    } catch (Exception e) {\n      LOG.error(\"Exception in indexing PartitionManager loop\", e);\n      IGNORED_EXCEPTIONS.increment();\n    } catch (Throwable t) {\n      LOG.error(\"Unhandled exception in indexing PartitionManager loop\", t);\n      criticalExceptionHandler.handle(this, t);\n      shutDownIndexing();\n      throw new RuntimeException(\"PartitionManager unhandled exception, stopping scheduler\", t);\n    } finally {\n      partitionManagerFirstLoop = false;\n    }\n  }\n\n  /**\n   * Returns the SegmentDataProvider instance that will be used to fetch the information for all\n   * segments.\n   */\n  public abstract SegmentDataProvider getSegmentDataProvider();\n\n  /**\n   * Starts up this partition manager.\n   */\n  protected abstract void startUp() throws Exception;\n\n  /**\n   * Runs one indexing iteration.\n   *\n   * @param firstLoop Determines if this is the first time the indexing loop is running.\n   */\n  protected abstract void indexingLoop(boolean firstLoop) throws Exception;\n\n  /**\n   * Shuts down all indexing.\n   */\n  protected abstract void shutDownIndexing();\n\n  @Override\n  public void shutdownComponent() {\n    shutDownIndexing();\n  }\n\n  /**\n   * Notifies all other threads that the partition manager has become current (ie. has indexed all\n   * available events).\n   */\n  public void becomeCurrent() {\n    LOG.info(\"PartitionManager became current\");\n    if (EarlybirdStatus.isStarting()) {\n      EarlybirdStatus.setStatus(EarlybirdStatusCode.CURRENT);\n    } else {\n      LOG.warn(\"Could not set statusCode to CURRENT from \" + EarlybirdStatus.getStatusCode());\n    }\n\n    // Now that we're done starting up, set the query cache thread pool size to one.\n    queryCacheManager.setWorkerPoolSizeAfterStartup();\n  }\n\n  protected void setupQueryCacheIfNeeded() throws QueryParserException {\n    queryCacheManager.setupTasksIfNeeded(segmentManager);\n  }\n\n  // Only for tests, used for testing exception handling\n  private static TestHook testHookBeforeStartUp;\n  private static TestHook testHookAfterSleep;\n\n  private static void testHookBeforeStartUp() throws Exception {\n    if (Config.environmentIsTest() && testHookBeforeStartUp != null) {\n      testHookBeforeStartUp.run();\n    }\n  }\n\n  private static void testHookAfterSleep() throws Exception {\n    if (Config.environmentIsTest() && testHookAfterSleep != null) {\n      testHookAfterSleep.run();\n    }\n  }\n\n  @Override\n  protected void runOneIteration() {\n    try {\n      runImpl();\n    } catch (Throwable t) {\n      LOG.error(\"Unhandled exception in PartitionManager loop\", t);\n      throw new RuntimeException(t.getMessage());\n    }\n  }\n\n  public SearchIndexingMetricSet getSearchIndexingMetricSet() {\n    return searchIndexingMetricSet;\n  }\n\n  /**\n   * Allows tests to run code before the partition manager starts up.\n   *\n   * @param testHook The code to run before the start up.\n   */\n  @VisibleForTesting\n  public static void setTestHookBeforeStartUp(TestHook testHook) {\n    if (Config.environmentIsTest()) {\n      testHookBeforeStartUp = testHook;\n    } else {\n      throw new RuntimeException(\"Trying to set startup test hook in non-test code!!\");\n    }\n  }\n\n  /**\n   * Allows tests to run code before the indexing loop.\n   *\n   * @param testHook The code to run before the indexing loop.\n   */\n  @VisibleForTesting\n  public static void setTestHookAfterSleep(TestHook testHook) {\n    if (Config.environmentIsTest()) {\n      testHookAfterSleep = testHook;\n    } else {\n      throw new RuntimeException(\"Trying to set test hook in non-test code!!\");\n    }\n  }\n\n  /**\n   * An interface that allows tests to run code at various points in the PartitionManager's\n   * lyfecycle.\n   */\n  @VisibleForTesting\n  public interface TestHook {\n    /**\n     * Defines the code that should be run.\n     */\n    void run() throws Exception;\n  }\n\n  /**\n   * Allows tests to determine if this partition manager is all caught up.\n   *\n   * @return {@code true} if this partition manager is caught up, {@code false} otherwise.\n   */\n  @VisibleForTesting\n  public abstract boolean isCaughtUpForTests();\n\n  @VisibleForTesting\n  protected void validateSegments() throws EarlybirdStartupException {\n    // This is necessary because many tests rely on starting partition manager but not indexing any\n    // tweets. However, we do not want Earlybirds to start in production if they are not serving any\n    // tweets. (SEARCH-24238)\n    if (Config.environmentIsTest()) {\n      return;\n    }\n    validateSegmentsForNonTest();\n  }\n\n  @VisibleForTesting\n  protected void validateSegmentsForNonTest() throws EarlybirdStartupException {\n    // Subclasses can override this and provide additional checks.\n    if (segmentManager.getNumIndexedDocuments() == 0) {\n      throw new EarlybirdStartupException(\"Earlybird has zero indexed documents.\");\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/PartitionManagerStartup.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.Closeable;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.earlybird.EarlybirdServer;\nimport com.twitter.search.earlybird.EarlybirdStatus;\nimport com.twitter.search.earlybird.exception.EarlybirdStartupException;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusCode;\n\n/**\n * Handles starting and indexing data for a partition, using a PartitionManager.\n */\npublic class PartitionManagerStartup implements EarlybirdStartup {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdServer.class);\n\n  private final Clock clock;\n  private final PartitionManager partitionManager;\n\n  public PartitionManagerStartup(\n      Clock clock,\n      PartitionManager partitionManager\n  ) {\n    this.clock = clock;\n    this.partitionManager = partitionManager;\n  }\n\n  @Override\n  public Closeable start() throws EarlybirdStartupException {\n    partitionManager.schedule();\n\n    int count = 0;\n\n    while (EarlybirdStatus.getStatusCode() != EarlybirdStatusCode.CURRENT) {\n      if (EarlybirdStatus.getStatusCode() == EarlybirdStatusCode.STOPPING) {\n        return partitionManager;\n      }\n\n      try {\n        clock.waitFor(1000);\n      } catch (InterruptedException e) {\n        LOG.info(\"Sleep interrupted, quitting earlybird\");\n        throw new EarlybirdStartupException(\"Sleep interrupted\");\n      }\n\n      // Log every 120 seconds.\n      if (count++ % 120 == 0) {\n        LOG.info(\"Thrift port closed until Earlybird, both indexing and query cache, is current\");\n      }\n    }\n\n    return partitionManager;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/PartitionWriter.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\nimport java.time.Duration;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.kafka.clients.consumer.ConsumerRecord;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEventType;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\n\n/**\n * PartitionWriter writes Tweet events and Tweet update events to an Earlybird index. It is\n * responsible for creating new segments, adding Tweets to the correct segment, and applying updates\n * to the correct segment.\n */\npublic class PartitionWriter {\n  private static final Logger LOG = LoggerFactory.getLogger(PartitionWriter.class);\n  private static final String STATS_PREFIX = \"partition_writer_\";\n\n  private static final SearchRateCounter MISSING_PENGUIN_VERSION =\n      SearchRateCounter.export(STATS_PREFIX + \"missing_penguin_version\");\n  private static final Duration CAUGHT_UP_FRESHNESS = Duration.ofSeconds(5);\n  private static final SearchRateCounter EVENTS_CONSUMED =\n      SearchRateCounter.export(STATS_PREFIX + \"events_consumed\");\n\n  private final PenguinVersion penguinVersion;\n  private final TweetUpdateHandler updateHandler;\n  private final TweetCreateHandler createHandler;\n  private final Clock clock;\n  private final CriticalExceptionHandler criticalExceptionHandler;\n\n\n\n  public PartitionWriter(\n      TweetCreateHandler tweetCreateHandler,\n      TweetUpdateHandler tweetUpdateHandler,\n      CriticalExceptionHandler criticalExceptionHandler,\n      PenguinVersion penguinVersion,\n      Clock clock\n  ) {\n    LOG.info(\"Creating PartitionWriter.\");\n    this.createHandler = tweetCreateHandler;\n    this.updateHandler = tweetUpdateHandler;\n    this.criticalExceptionHandler = criticalExceptionHandler;\n    this.penguinVersion = penguinVersion;\n    this.clock = clock;\n  }\n\n  /**\n   * Index a batch of TVE records.\n   */\n  public boolean indexBatch(Iterable<ConsumerRecord<Long, ThriftVersionedEvents>> records)\n      throws Exception {\n    long minTweetAge = Long.MAX_VALUE;\n    for (ConsumerRecord<Long, ThriftVersionedEvents> record : records) {\n      ThriftVersionedEvents tve = record.value();\n      indexTVE(tve);\n      EVENTS_CONSUMED.increment();\n      long tweetAgeInMs = SnowflakeIdParser.getTweetAgeInMs(clock.nowMillis(), tve.getId());\n      minTweetAge = Math.min(tweetAgeInMs, minTweetAge);\n    }\n\n    return minTweetAge < CAUGHT_UP_FRESHNESS.toMillis();\n  }\n\n  /**\n   * Index a ThriftVersionedEvents struct.\n   */\n  @VisibleForTesting\n  public void indexTVE(ThriftVersionedEvents tve) throws IOException {\n    ThriftIndexingEvent tie = tve.getVersionedEvents().get(penguinVersion.getByteValue());\n    if (tie == null) {\n      LOG.error(\"Could not find a ThriftIndexingEvent for PenguinVersion {} in \"\n          + \"ThriftVersionedEvents: {}\", penguinVersion, tve);\n      MISSING_PENGUIN_VERSION.increment();\n      return;\n    }\n\n    // An `INSERT` event is used for new Tweets. These are generated from Tweet Create Events from\n    // TweetyPie.\n    if (tie.getEventType() == ThriftIndexingEventType.INSERT) {\n      createHandler.handleTweetCreate(tve);\n      updateHandler.retryPendingUpdates(tve.getId());\n    } else {\n      updateHandler.handleTweetUpdate(tve, false);\n    }\n  }\n\n  public void prepareAfterStartingWithIndex(long maxIndexedTweetId) {\n    createHandler.prepareAfterStartingWithIndex(maxIndexedTweetId);\n  }\n\n  void logState() {\n    LOG.info(\"PartitionWriter state:\");\n    LOG.info(String.format(\"  Events indexed: %,d\", EVENTS_CONSUMED.getCount()));\n    createHandler.logState();\n    updateHandler.logState();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SearchIndexingMetricSet.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.util.EnumMap;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.atomic.AtomicLong;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEventType;\nimport com.twitter.search.earlybird.util.ScheduledExecutorManager;\n\n/**\n * Collection of common metrics used in the indexing, and related code.\n * We create a set/holder for them as we want to create all counters only one time, and these\n * counters can be used by both SimpleUpdateIndexer, PartitionIndexer, EarlybirdSegment, and others.\n */\npublic class SearchIndexingMetricSet {\n  /**\n   * A proxy for the creation time of the \"freshest\" tweet that we have in the index.\n   * It is used in computing the index freshness stat \"earlybird_index_freshness_millis\".\n   * - In the realtme clusters, this should match the creation time of highestStatusId.\n   * - In the archive clusters, this should match the timestamp of the latest indexed day.\n   */\n  public final SearchLongGauge freshestTweetTimeMillis;\n\n  /** The highest indexed tweet ID. Used to compute index freshness. */\n  public final SearchLongGauge highestStatusId;\n\n  /**\n   * The current timeslice's ID. We can compare this to indexer's exported current timeslice ID to\n   * identify stuck timeslice rolls.\n   */\n  public final SearchLongGauge currentTimesliceId;\n\n  /** The number of archive timeslices that we failed to process. */\n  public final SearchCounter archiveTimeSliceBuildFailedCounter;\n\n  /** The number of times we checked a segment's size on disk. */\n  public final SearchCounter segmentSizeCheckCount;\n\n  /** The number of segments that have reached their max size. */\n  public final SearchCounter maxSegmentSizeReachedCounter;\n\n  /** The number of indexed tweets and the aggregate indexing latencies in microseconds. */\n  public final SearchTimerStats statusStats;\n  /** The number of applied updates and the aggregate indexing latencies in microseconds. */\n  public final SearchTimerStats updateStats;\n  /** The number of retried updates and the aggregate indexing latencies in microseconds. */\n  public final SearchTimerStats updateRetryStats;\n  /** The number of applied user updates and the aggregate indexing latencies in microseconds. */\n  public final SearchTimerStats userUpdateIndexingStats;\n  /** The number of applied userGeoScrubEvents and the aggregate indexing latencies in\n   * microseconds. */\n  public final SearchTimerStats userScrubGeoIndexingStats;\n  /** The number of updates attempted on missing tweets. */\n  public final SearchRateCounter updateOnMissingTweetCounter;\n  /** The number of updates dropped. */\n  public final SearchRateCounter droppedUpdateEvent;\n\n  /** The latencies in microseconds of the PartitionIndexer loop. */\n  public final SearchTimerStats partitionIndexerRunLoopCounter;\n  /** The latencies in microseconds of the PartitionIndexer.indexFromReaders() calls. */\n  public final SearchTimerStats partitionIndexerIndexFromReadersCounter;\n  /** The number of invocations of the PartitionIndexer task. */\n  public final SearchCounter partitionIndexerIterationCounter;\n\n  /** The number of unsorted updates handled by SimpleUpdateIndexer. */\n  public final SearchCounter simpleUpdateIndexerUnsortedUpdateCounter;\n  /** The number of unsorted updates with the wrong segment handled by SimpleUpdateIndexer. */\n  public final SearchCounter simpleUpdateIndexerUnsortedUpdateWithWrongSegmentCounter;\n\n  /** The number of invocations of the SimpleUserUpdateIndexer task. */\n  public final SearchCounter simpleUserUpdateIndexerIterationCounter;\n\n  /** The number of exceptions encountered by SimpleSegmentIndexer while indexing a segment. */\n  public final SearchCounter simpleSegmentIndexerExceptionCounter;\n\n  /**\n   * A map from TIE update type to the creation time of the updated tweet in milliseconds of the\n   * freshest update we have indexed.\n   */\n  public final EnumMap<ThriftIndexingEventType, AtomicLong> updateFreshness =\n      new EnumMap<>(ThriftIndexingEventType.class);\n\n  public final SearchStatsReceiver searchStatsReceiver;\n\n  public static class StartupMetric {\n    // Switched from 0 to 1 during the event.\n    private SearchLongGauge duringGauge;\n    // Switched from 0 to time it takes, in milliseconds.\n    private SearchLongGauge durationMillisGauge;\n\n    StartupMetric(String name) {\n      this.duringGauge = SearchLongGauge.export(name);\n      this.durationMillisGauge = SearchLongGauge.export(\"duration_of_\" + name);\n    }\n\n    public void begin() {\n      duringGauge.set(1);\n    }\n\n    public void end(long durationInMillis) {\n      duringGauge.set(0);\n      durationMillisGauge.set(durationInMillis);\n    }\n  }\n\n  public final StartupMetric startupInProgress;\n  public final StartupMetric startupInIndexCompletedSegments;\n  public final StartupMetric startupInLoadCompletedSegments;\n  public final StartupMetric startupInIndexUpdatesForCompletedSegments;\n  public final StartupMetric startupInCurrentSegment;\n  public final StartupMetric startupInUserUpdates;\n  public final StartupMetric startupInQueryCacheUpdates;\n  public final StartupMetric startupInMultiSegmentTermDictionaryUpdates;\n  public final StartupMetric startupInWarmUp;\n\n  // Kafka metrics\n  public final StartupMetric startupInLoadFlushedIndex;\n  public final StartupMetric startupInFreshStartup;\n  public final StartupMetric startupInIngestUntilCurrent;\n  public final StartupMetric startupInUserUpdatesStartup;\n  public final StartupMetric startupInUserEventIndexer;\n  public final StartupMetric startupInAudioSpaceEventIndexer;\n\n  public SearchIndexingMetricSet(SearchStatsReceiver searchStatsReceiver) {\n    this.freshestTweetTimeMillis = searchStatsReceiver.getLongGauge(\n        \"earlybird_freshest_tweet_timestamp_millis\");\n    this.highestStatusId = searchStatsReceiver.getLongGauge(\"highest_indexed_status_id\");\n    this.currentTimesliceId = searchStatsReceiver.getLongGauge(\"earlybird_current_timeslice_id\");\n    this.archiveTimeSliceBuildFailedCounter = searchStatsReceiver.getCounter(\n        \"archive_time_slice_build_failed\");\n    this.segmentSizeCheckCount = searchStatsReceiver.getCounter(\"segment_size_check_count\");\n    this.maxSegmentSizeReachedCounter = searchStatsReceiver.getCounter(\"max_segment_reached\");\n\n    this.statusStats = searchStatsReceiver.getTimerStats(\n        \"index_status\", TimeUnit.MICROSECONDS, false, false, false);\n    this.updateStats = searchStatsReceiver.getTimerStats(\n        \"updates\", TimeUnit.MICROSECONDS, false, false, false);\n    this.updateRetryStats = searchStatsReceiver.getTimerStats(\n        \"update_retries\", TimeUnit.MICROSECONDS, false, false, false);\n    this.userUpdateIndexingStats = searchStatsReceiver.getTimerStats(\n        \"user_updates\", TimeUnit.MICROSECONDS, false, false, false);\n    this.userScrubGeoIndexingStats = searchStatsReceiver.getTimerStats(\n        \"user_scrub_geo\", TimeUnit.MICROSECONDS, false, false, false);\n    this.updateOnMissingTweetCounter = searchStatsReceiver.getRateCounter(\n        \"index_update_on_missing_tweet\");\n    this.droppedUpdateEvent = searchStatsReceiver.getRateCounter(\"dropped_update_event\");\n\n    this.partitionIndexerRunLoopCounter = searchStatsReceiver.getTimerStats(\n        \"partition_indexer_run_loop\", TimeUnit.MICROSECONDS, false, true, false);\n    this.partitionIndexerIndexFromReadersCounter = searchStatsReceiver.getTimerStats(\n        \"partition_indexer_indexFromReaders\", TimeUnit.MICROSECONDS, false, true, false);\n    this.partitionIndexerIterationCounter = searchStatsReceiver.getCounter(\n        ScheduledExecutorManager.SCHEDULED_EXECUTOR_TASK_PREFIX + \"PartitionIndexer\");\n\n    this.simpleUpdateIndexerUnsortedUpdateCounter = searchStatsReceiver.getCounter(\n        \"simple_update_indexer_unsorted_update_count\");\n    this.simpleUpdateIndexerUnsortedUpdateWithWrongSegmentCounter = searchStatsReceiver.getCounter(\n        \"simple_update_indexer_unsorted_update_with_wrong_segment_count\");\n\n    this.simpleUserUpdateIndexerIterationCounter = searchStatsReceiver.getCounter(\n        ScheduledExecutorManager.SCHEDULED_EXECUTOR_TASK_PREFIX + \"SimpleUserUpdateIndexer\");\n\n    this.simpleSegmentIndexerExceptionCounter = searchStatsReceiver.getCounter(\n        \"exception_while_indexing_segment\");\n\n    for (ThriftIndexingEventType type : ThriftIndexingEventType.values()) {\n      AtomicLong freshness = new AtomicLong(0);\n      updateFreshness.put(type, freshness);\n      String statName = (\"index_freshness_\" + type + \"_age_millis\").toLowerCase();\n      searchStatsReceiver.getCustomGauge(statName,\n          () -> System.currentTimeMillis() - freshness.get());\n    }\n\n    this.startupInProgress = new StartupMetric(\"startup_in_progress\");\n    this.startupInIndexCompletedSegments = new StartupMetric(\"startup_in_index_completed_segments\");\n    this.startupInLoadCompletedSegments = new StartupMetric(\"startup_in_load_completed_segments\");\n    this.startupInIndexUpdatesForCompletedSegments =\n        new StartupMetric(\"startup_in_index_updates_for_completed_segments\");\n    this.startupInCurrentSegment = new StartupMetric(\"startup_in_current_segment\");\n    this.startupInUserUpdates = new StartupMetric(\"startup_in_user_updates\");\n    this.startupInQueryCacheUpdates = new StartupMetric(\"startup_in_query_cache_updates\");\n    this.startupInMultiSegmentTermDictionaryUpdates =\n        new StartupMetric(\"startup_in_multi_segment_dictionary_updates\");\n    this.startupInWarmUp = new StartupMetric(\"startup_in_warm_up\");\n\n    this.startupInLoadFlushedIndex = new StartupMetric(\"startup_in_load_flushed_index\");\n    this.startupInFreshStartup = new StartupMetric(\"startup_in_fresh_startup\");\n    this.startupInIngestUntilCurrent = new StartupMetric(\"startup_in_ingest_until_current\");\n    this.startupInUserUpdatesStartup = new StartupMetric(\"startup_in_user_updates_startup\");\n    this.startupInUserEventIndexer = new StartupMetric(\"startup_in_user_events_indexer\");\n    this.startupInAudioSpaceEventIndexer =\n        new StartupMetric(\"startup_in_audio_space_events_indexer\");\n\n    searchStatsReceiver.getCustomGauge(\"earlybird_index_freshness_millis\",\n        this::getIndexFreshnessInMillis);\n\n    this.searchStatsReceiver = searchStatsReceiver;\n  }\n\n  long getIndexFreshnessInMillis() {\n    return System.currentTimeMillis() - freshestTweetTimeMillis.get();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SegmentHdfsFlusher.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.util.concurrent.TimeUnit;\n\nimport org.apache.commons.io.FileUtils;\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.Path;\nimport org.apache.lucene.store.Directory;\nimport org.apache.lucene.store.FSDirectory;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.base.Command;\nimport com.twitter.common.quantity.Amount;\nimport com.twitter.common.quantity.Time;\nimport com.twitter.search.common.database.DatabaseConfig;\nimport com.twitter.search.common.metrics.Timer;\nimport com.twitter.search.common.util.io.flushable.PersistentFile;\nimport com.twitter.search.common.util.zktrylock.TryLock;\nimport com.twitter.search.common.util.zktrylock.ZooKeeperTryLockFactory;\n\n/**\n * Flush segments to disk and upload them to HDFS.\n */\npublic class SegmentHdfsFlusher {\n  private static final Logger LOG = LoggerFactory.getLogger(SegmentHdfsFlusher.class);\n  private static final Amount<Long, Time> HDFS_UPLOADER_TRY_LOCK_NODE_EXPIRATION_TIME_MILLIS =\n      Amount.of(1L, Time.HOURS);\n\n  private final SegmentSyncConfig sync;\n  private final boolean holdLockWhileUploading;\n  private final ZooKeeperTryLockFactory zkTryLockFactory;\n\n  public SegmentHdfsFlusher(ZooKeeperTryLockFactory zooKeeperTryLockFactory,\n                            SegmentSyncConfig sync,\n                            boolean holdLockWhileUploading) {\n    this.zkTryLockFactory = zooKeeperTryLockFactory;\n    this.sync = sync;\n    this.holdLockWhileUploading = holdLockWhileUploading;\n  }\n\n  public SegmentHdfsFlusher(\n      ZooKeeperTryLockFactory zooKeeperTryLockFactory,\n      SegmentSyncConfig sync) {\n    this(zooKeeperTryLockFactory, sync, true);\n  }\n\n  private boolean shouldFlushSegment(SegmentInfo segmentInfo) {\n    return segmentInfo.isEnabled()\n        && !segmentInfo.getSyncInfo().isFlushed()\n        && segmentInfo.isComplete()\n        && segmentInfo.isOptimized()\n        && !segmentInfo.isFailedOptimize()\n        && !segmentInfo.getSyncInfo().isLoaded();\n  }\n\n  /**\n   * Flushes a segment to local disk and to HDFS.\n   */\n  public boolean flushSegmentToDiskAndHDFS(SegmentInfo segmentInfo) {\n    if (!shouldFlushSegment(segmentInfo)) {\n      return false;\n    }\n    try {\n      if (segmentInfo.isIndexing()) {\n        LOG.error(\"Tried to flush current segment!\");\n        return false;\n      }\n\n      // Check-and-set the beingUploaded flag from false to true. If the CAS fails, it means the\n      // segment is being flushed already, or being deleted. In this case, we can just return false.\n      if (!segmentInfo.casBeingUploaded(false, true)) {\n        LOG.warn(\"Tried to flush a segment that's being flushed or deleted.\");\n        return false;\n      }\n\n      // At this point, the above CAS must have returned false. This mean the beingUploaded flag\n      // was false, and set to true now. We can proceed with flushing the segment.\n      try {\n        checkAndFlushSegmentToHdfs(segmentInfo);\n      } finally {\n        segmentInfo.setBeingUploaded(false);\n      }\n      return true;\n    } catch (Exception e) {\n      LOG.error(\"Exception while flushing IndexSegment to \"\n          + segmentInfo.getSyncInfo().getHdfsFlushDir(), e);\n      return false;\n    }\n  }\n\n  /**\n   * First try to acquire a lock in Zookeeper for this segment, so multiple Earlybirds in the same\n   * partition don't flush or upload the segment at the same time. When the lock is acquired, check\n   * for the segment in HDFS. If the data already exists, don't flush to disk.\n   */\n  private void checkAndFlushSegmentToHdfs(final SegmentInfo segment) {\n    LOG.info(\"Checking and flushing segment {}\", segment);\n\n    try {\n      // Always flush the segment locally.\n      Directory dir = FSDirectory.open(createFlushDir(segment).toPath());\n      segment.flush(dir);\n      LOG.info(\"Completed local flush of segment {}. Flush to HDFS enabled: {}\",\n               segment, sync.isFlushToHdfsEnabled());\n    } catch (IOException e) {\n      LOG.error(\"Failed to flush segment \" + segment + \" locally\", e);\n      return;\n    }\n\n    if (!holdLockWhileUploading) {\n      flushToHdfsIfNecessary(segment);\n    } else {\n      TryLock lock = zkTryLockFactory.createTryLock(\n          DatabaseConfig.getLocalHostname(),\n          sync.getZooKeeperSyncFullPath(),\n          sync.getVersionedName(segment.getSegment()),\n          HDFS_UPLOADER_TRY_LOCK_NODE_EXPIRATION_TIME_MILLIS\n      );\n\n      boolean gotLock = lock.tryWithLock((Command) () -> flushToHdfsIfNecessary(segment));\n      if (!gotLock) {\n        LOG.info(\"Failed to get zk upload lock for segment {}\", segment);\n      }\n    }\n  }\n\n  /**\n   * Check whether the segment has already been flushed to HDFS. If not, flush the segment to disk\n   * and upload the files to HDFS.\n   *\n   * If the ZK lock isn't used, there is a race between the existence check and the upload (in\n   * which another Earlybird can sneak in and upload the segment), so we will potentially upload\n   * the same segment from different hosts. Thus, the Earlybird hostname is part of the segment's\n   * path on HDFS.\n   */\n  private void flushToHdfsIfNecessary(SegmentInfo segmentInfo) {\n    Timer timer = new Timer(TimeUnit.MILLISECONDS);\n    String status = \"flushed\";\n    try (FileSystem fs = HdfsUtil.getHdfsFileSystem()) {\n      // If we can't load segments from HDFS, don't bother checking HDFS for the segment\n      if (sync.isSegmentLoadFromHdfsEnabled()\n          && (segmentInfo.getSyncInfo().isFlushed()\n              || HdfsUtil.segmentExistsOnHdfs(fs, segmentInfo))) {\n        status = \"existing\";\n      } else if (sync.isFlushToHdfsEnabled()) {\n        copyLocalFilesToHdfs(fs, segmentInfo);\n        status = \"uploaded\";\n      }\n\n      // whether we uploaded, or someone else did, this segment should now be on HDFS. If\n      // uploading to HDFS is disabled, we still consider it complete.\n      segmentInfo.getSyncInfo().setFlushed(true);\n    } catch (IOException e) {\n      LOG.error(\"Failed copying segment {} to HDFS after {} ms\", segmentInfo, timer.stop(), e);\n      status = \"exception\";\n    } finally {\n      if (timer.running()) {\n        timer.stop();\n      }\n      LOG.info(\"Flush of segment {} to HDFS completed in {} milliseconds. Status: {}\",\n          segmentInfo, timer.getElapsed(), status);\n    }\n  }\n\n  /**\n   * Copy local segment files to HDFS. Files are first copied into a temporary directory\n   * in the form <hostname>_<segmentname> and when all the files are written out to HDFS,\n   * the dir is renamed to <segmentname>_<hostname>, where it is accessible to other Earlybirds.\n   */\n  private void copyLocalFilesToHdfs(FileSystem fs, SegmentInfo segment) throws IOException {\n    String hdfsTempBaseDir = segment.getSyncInfo().getHdfsTempFlushDir();\n\n    // If the temp dir already exists on HDFS, a prior flush must have been interrupted.\n    // Delete it and start fresh.\n    removeHdfsTempDir(fs, hdfsTempBaseDir);\n\n    for (String fileName : sync.getAllSyncFileNames(segment)) {\n      String hdfsFileName = hdfsTempBaseDir + \"/\" + fileName;\n      String localBaseDir = segment.getSyncInfo().getLocalSyncDir();\n      String localFileName = localBaseDir + \"/\" + fileName;\n\n      LOG.debug(\"About to start copying {} to HDFS, from {} to {}\",\n          fileName, localFileName, hdfsFileName);\n      Timer timer = new Timer(TimeUnit.MILLISECONDS);\n      fs.copyFromLocalFile(new Path(localFileName), new Path(hdfsFileName));\n      LOG.debug(\"Completed copying {} to HDFS, from {} to {}, in {} ms\",\n          fileName, localFileName, hdfsFileName, timer.stop());\n    }\n\n    // now let's rename the dir into its proper form.\n    String hdfsBaseDir = segment.getSyncInfo().getHdfsFlushDir();\n    if (fs.rename(new Path(hdfsTempBaseDir), new Path(hdfsBaseDir))) {\n      LOG.info(\"Renamed segment dir on HDFS from {} to {}\", hdfsTempBaseDir, hdfsBaseDir);\n    } else {\n      String errorMessage = String.format(\"Failed to rename segment dir on HDFS from %s to %s\",\n          hdfsTempBaseDir, hdfsBaseDir);\n      LOG.error(errorMessage);\n\n      removeHdfsTempDir(fs, hdfsTempBaseDir);\n\n      // Throw an IOException so the calling code knows that the copy failed\n      throw new IOException(errorMessage);\n    }\n  }\n\n  private void removeHdfsTempDir(FileSystem fs, String tempDir) throws IOException {\n    Path tempDirPath = new Path(tempDir);\n    if (fs.exists(tempDirPath)) {\n      LOG.info(\"Found existing temporary flush dir {} on HDFS, removing\", tempDir);\n      if (!fs.delete(tempDirPath, true /* recursive */)) {\n        LOG.error(\"Failed to delete temp dir {}\", tempDir);\n      }\n    }\n  }\n\n  // Create or replace the local flush directory\n  private File createFlushDir(SegmentInfo segmentInfo) throws IOException {\n    final String flushDirStr = segmentInfo.getSyncInfo().getLocalSyncDir();\n\n    File flushDir = new File(flushDirStr);\n    if (flushDir.exists()) {\n      // Delete just the flushed persistent files if they are there.\n      // We may also have the lucene on-disk indexed in the same dir here,\n      // that we do not want to delete.\n      for (String persistentFile : sync.getPersistentFileNames(segmentInfo)) {\n        for (String fileName : PersistentFile.getAllFileNames(persistentFile)) {\n          File file = new File(flushDir, fileName);\n          if (file.exists()) {\n            LOG.info(\"Deleting incomplete flush file {}\", file.getAbsolutePath());\n            FileUtils.forceDelete(file);\n          }\n        }\n      }\n      return flushDir;\n    }\n\n    // Try to create the flush directory\n    if (!flushDir.mkdirs()) {\n      throw new IOException(\"Not able to create segment flush directory \\\"\" + flushDirStr + \"\\\"\");\n    }\n\n    return flushDir;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SegmentIndexStats.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.util.Optional;\nimport java.util.concurrent.atomic.AtomicInteger;\nimport java.util.concurrent.atomic.AtomicLong;\n\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentData;\n\npublic class SegmentIndexStats {\n  private EarlybirdIndexSegmentData segmentData;\n\n  private final AtomicLong indexSizeOnDiskInBytes = new AtomicLong(0);\n  private final AtomicInteger partialUpdateCount = new AtomicInteger(0);\n  private final AtomicInteger outOfOrderUpdateCount = new AtomicInteger(0);\n\n  private Optional<Integer> savedStatusCount = Optional.empty();\n  private Optional<Integer> savedDeletesCount = Optional.empty();\n\n  public void setSegmentData(EarlybirdIndexSegmentData segmentData) {\n    this.segmentData = segmentData;\n  }\n\n  /**\n   * We'd like to be able to return the last counts after we unload a segment from memory.\n   */\n  public void unsetSegmentDataAndSaveCounts() {\n    savedStatusCount = Optional.of(getStatusCount());\n    savedDeletesCount = Optional.of(getDeleteCount());\n    segmentData = null;\n  }\n\n  /**\n   * Returns the number of deletes processed by this segment.\n   */\n  public int getDeleteCount() {\n    if (segmentData != null) {\n      return segmentData.getDeletedDocs().numDeletions();\n    } else {\n      return savedDeletesCount.orElse(0);\n    }\n  }\n\n  /**\n   * Return the number of documents in this segment.\n   */\n  public int getStatusCount() {\n    if (segmentData != null) {\n      return segmentData.numDocs();\n    } else {\n      return savedStatusCount.orElse(0);\n    }\n  }\n\n  public long getIndexSizeOnDiskInBytes() {\n    return indexSizeOnDiskInBytes.get();\n  }\n\n  public void setIndexSizeOnDiskInBytes(long value) {\n    indexSizeOnDiskInBytes.set(value);\n  }\n\n  public int getPartialUpdateCount() {\n    return partialUpdateCount.get();\n  }\n\n  public void incrementPartialUpdateCount() {\n    partialUpdateCount.incrementAndGet();\n  }\n\n  public void setPartialUpdateCount(int value) {\n    partialUpdateCount.set(value);\n  }\n\n  public int getOutOfOrderUpdateCount() {\n    return outOfOrderUpdateCount.get();\n  }\n\n  public void incrementOutOfOrderUpdateCount() {\n    outOfOrderUpdateCount.incrementAndGet();\n  }\n\n  public void setOutOfOrderUpdateCount(int value) {\n    outOfOrderUpdateCount.set(value);\n  }\n\n  @Override\n  public String toString() {\n    StringBuilder sb = new StringBuilder();\n    sb.append(\"Indexed \").append(getStatusCount()).append(\" documents, \");\n    sb.append(getDeleteCount()).append(\" deletes, \");\n    sb.append(getPartialUpdateCount()).append(\" partial updates, \");\n    sb.append(getOutOfOrderUpdateCount()).append(\" out of order udpates. \");\n    sb.append(\"Index size: \").append(getIndexSizeOnDiskInBytes());\n    return sb.toString();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SegmentIndexStatsExporter.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport com.twitter.common.base.Supplier;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchMetric;\nimport com.twitter.search.common.metrics.SearchMetricsRegistry;\n\n/**\n * Exporting per-segment stats collected in {@link SegmentIndexStats}.\n *\n * This class tries to reuse stat prefixes of \"segment_stats_[0-N]_*\" where N is the number\n * of segments managed by this earlybird.\n * For example, stats prefixed with \"segment_stats_0_*\" always represent the most recent segment.\n * As we add more segments (and drop older ones), the same \"segment_stats_*\" stats end up exporting\n * data for different underlying segments.\n *\n * This is done as an alternative to exporting stats that have the timesliceId in them, which\n * would avoid the need for reusing the same stat names, but would create an ever-increasing set\n * of unique stats exported by earlybirds.\n */\npublic final class SegmentIndexStatsExporter {\n  private static final class StatReader extends SearchMetric<Long> {\n    private volatile Supplier<Number> counter = () -> 0;\n\n    private StatReader(String name) {\n      super(name);\n    }\n\n    @Override\n    public Long read() {\n      return counter.get().longValue();\n    }\n\n    @Override\n    public void reset() {\n      counter = () -> 0;\n    }\n  }\n\n  private SegmentIndexStatsExporter() {\n  }\n\n  private static final String NAME_PREFIX = \"segment_stats_\";\n\n  /**\n   * Exports stats for some counts for the given segment:\n   *  - status_count: number of tweets indexed\n   *  - delete_count: number of deletes indexed\n   *  - partial_update_count: number of partial updates indexed\n   *  - out_of_order_update_count: number of out of order updates indexed\n   *  - segment_size_bytes: the segment size in bytes\n   *\n   * @param segmentInfo The segment for which these stats should be exported.\n   * @param segmentIndex The index of this segment in the list of all segments.\n   */\n  public static void export(SegmentInfo segmentInfo, int segmentIndex) {\n    exportStat(segmentIndex, \"status_count\",\n        () -> segmentInfo.getIndexStats().getStatusCount());\n    exportStat(segmentIndex, \"delete_count\",\n        () -> segmentInfo.getIndexStats().getDeleteCount());\n    exportStat(segmentIndex, \"partial_update_count\",\n        () -> segmentInfo.getIndexStats().getPartialUpdateCount());\n    exportStat(segmentIndex, \"out_of_order_update_count\",\n        () -> segmentInfo.getIndexStats().getOutOfOrderUpdateCount());\n    exportStat(segmentIndex, \"segment_size_bytes\",\n        () -> segmentInfo.getIndexStats().getIndexSizeOnDiskInBytes());\n\n    SearchLongGauge timeSliceIdStat =\n        SearchLongGauge.export(NAME_PREFIX + segmentIndex + \"_timeslice_id\");\n    timeSliceIdStat.set(segmentInfo.getTimeSliceID());\n  }\n\n  private static void exportStat(final int segmentIndex,\n                                 final String nameSuffix,\n                                 Supplier<Number> counter) {\n    final String name = getName(segmentIndex, nameSuffix);\n    StatReader statReader = SearchMetricsRegistry.registerOrGet(\n        () -> new StatReader(name), name, StatReader.class);\n    statReader.counter = counter;\n  }\n\n  private static String getName(final int segmentIndex, final String nameSuffix) {\n    return NAME_PREFIX + segmentIndex + \"_\" + nameSuffix;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SegmentInfo.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.OutputStreamWriter;\nimport java.util.concurrent.atomic.AtomicBoolean;\nimport java.util.concurrent.atomic.AtomicInteger;\nimport java.util.concurrent.atomic.AtomicLong;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.apache.commons.io.FileUtils;\nimport org.apache.lucene.store.Directory;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.search.common.partitioning.base.Segment;\nimport com.twitter.search.common.partitioning.base.TimeSlice;\nimport com.twitter.search.common.schema.earlybird.FlushVersion;\nimport com.twitter.search.common.util.LogFormatUtil;\nimport com.twitter.search.common.util.io.flushable.FlushInfo;\nimport com.twitter.search.common.util.io.flushable.PersistentFile;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.index.EarlybirdSegment;\nimport com.twitter.search.earlybird.index.EarlybirdSegmentFactory;\n\npublic class SegmentInfo implements Comparable<SegmentInfo> {\n  private static final Logger LOG = LoggerFactory.getLogger(SegmentInfo.class);\n\n  private static final String UPDATE_STREAM_OFFSET_TIMESTAMP = \"updateStreamOffsetTimestamp\";\n  public static final int INVALID_ID = -1;\n\n  // Delay before deleting a segment\n  private final long timeToWaitBeforeClosingMillis = EarlybirdConfig.getLong(\n      \"defer_index_closing_time_millis\", 600000L);\n  // How many times deletions are retired.\n  private final AtomicInteger deletionRetries = new AtomicInteger(5);\n\n  // Base segment information, including database name, minStatusId.\n  private final Segment segment;\n\n  // Bits managed by various SegmentProcessors and PartitionManager.\n  private volatile boolean isEnabled   = true;   // True if the segment is enabled.\n  private volatile boolean isIndexing  = false;  // True during indexing.\n  private volatile boolean isComplete  = false;  // True when indexing is complete.\n  private volatile boolean isClosed    = false;  // True if indexSegment is closed.\n  private volatile boolean wasIndexed  = false;  // True if the segment was indexed from scratch.\n  private volatile boolean failedOptimize = false;  // optimize attempt failed.\n  private AtomicBoolean beingUploaded = new AtomicBoolean();  // segment is being copied to HDFS\n\n  private final SegmentSyncInfo segmentSyncInfo;\n  private final EarlybirdIndexConfig earlybirdIndexConfig;\n\n  private final EarlybirdSegment indexSegment;\n\n  private final AtomicLong updatesStreamOffsetTimestamp = new AtomicLong(0);\n\n  public SegmentInfo(Segment segment,\n                     EarlybirdSegmentFactory earlybirdSegmentFactory,\n                     SegmentSyncConfig syncConfig) throws IOException {\n    this(segment, earlybirdSegmentFactory, new SegmentSyncInfo(syncConfig, segment));\n  }\n\n  @VisibleForTesting\n  public SegmentInfo(Segment segment,\n                     EarlybirdSegmentFactory earlybirdSegmentFactory,\n                     SegmentSyncInfo segmentSyncInfo) throws IOException {\n    this(earlybirdSegmentFactory.newEarlybirdSegment(segment, segmentSyncInfo),\n        segmentSyncInfo,\n        segment,\n        earlybirdSegmentFactory.getEarlybirdIndexConfig());\n  }\n\n  public SegmentInfo(\n      EarlybirdSegment earlybirdSegment,\n      SegmentSyncInfo segmentSyncInfo,\n      Segment segment,\n      EarlybirdIndexConfig earlybirdIndexConfig\n  ) {\n    this.indexSegment = earlybirdSegment;\n    this.segmentSyncInfo = segmentSyncInfo;\n    this.earlybirdIndexConfig = earlybirdIndexConfig;\n    this.segment = segment;\n  }\n\n  public EarlybirdSegment getIndexSegment() {\n    return indexSegment;\n  }\n\n  public SegmentIndexStats getIndexStats() {\n    return indexSegment.getIndexStats();\n  }\n\n  public EarlybirdIndexConfig getEarlybirdIndexConfig() {\n    return earlybirdIndexConfig;\n  }\n\n  public long getTimeSliceID() {\n    return segment.getTimeSliceID();\n  }\n\n  public String getSegmentName() {\n    return segment.getSegmentName();\n  }\n\n  public int getNumPartitions() {\n    return segment.getNumHashPartitions();\n  }\n\n  public boolean isEnabled() {\n    return isEnabled;\n  }\n\n  public void setIsEnabled(boolean isEnabled) {\n    this.isEnabled = isEnabled;\n  }\n\n  public boolean isOptimized() {\n    return indexSegment.isOptimized();\n  }\n\n  public boolean wasIndexed() {\n    return wasIndexed;\n  }\n\n  public void setWasIndexed(boolean wasIndexed) {\n    this.wasIndexed = wasIndexed;\n  }\n\n  public boolean isFailedOptimize() {\n    return failedOptimize;\n  }\n\n  public void setFailedOptimize() {\n    this.failedOptimize = true;\n  }\n\n  public boolean isIndexing() {\n    return isIndexing;\n  }\n\n  public void setIndexing(boolean indexing) {\n    this.isIndexing = indexing;\n  }\n\n  public boolean isComplete() {\n    return isComplete;\n  }\n\n  public boolean isClosed() {\n    return isClosed;\n  }\n\n  public boolean isBeingUploaded() {\n    return beingUploaded.get();\n  }\n\n  public void setBeingUploaded(boolean beingUploaded) {\n    this.beingUploaded.set(beingUploaded);\n  }\n\n  public boolean casBeingUploaded(boolean expectation, boolean updateValue) {\n    return beingUploaded.compareAndSet(expectation, updateValue);\n  }\n\n  @VisibleForTesting\n  public void setComplete(boolean complete) {\n    this.isComplete = complete;\n  }\n\n  public boolean needsIndexing() {\n    return isEnabled && !isIndexing && !isComplete;\n  }\n\n  @Override\n  public int compareTo(SegmentInfo other) {\n    return Long.compare(getTimeSliceID(), other.getTimeSliceID());\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    return obj instanceof SegmentInfo && compareTo((SegmentInfo) obj) == 0;\n  }\n\n  @Override\n  public int hashCode() {\n    return new Long(getTimeSliceID()).hashCode();\n  }\n\n  public long getUpdatesStreamOffsetTimestamp() {\n    return updatesStreamOffsetTimestamp.get();\n  }\n\n  public void setUpdatesStreamOffsetTimestamp(long timestamp) {\n    updatesStreamOffsetTimestamp.set(timestamp);\n  }\n\n  @Override\n  public String toString() {\n    StringBuilder builder = new StringBuilder();\n    builder.append(getSegmentName()).append(\" [\");\n    builder.append(isEnabled ? \"enabled, \" : \"disabled, \");\n\n    if (isIndexing) {\n      builder.append(\"indexing, \");\n    }\n\n    if (isComplete) {\n      builder.append(\"complete, \");\n    }\n\n    if (isOptimized()) {\n      builder.append(\"optimized, \");\n    }\n\n    if (wasIndexed) {\n      builder.append(\"wasIndexed, \");\n    }\n\n    builder.append(\"IndexSync:\");\n    this.segmentSyncInfo.addDebugInfo(builder);\n\n    return builder.append(\"]\").toString();\n  }\n\n  public Segment getSegment() {\n    return segment;\n  }\n\n  /**\n   * Delete the index segment directory corresponding to this segment info. Return true if deleted\n   * successfully; otherwise, false.\n   */\n  public boolean deleteLocalIndexedSegmentDirectoryImmediately() {\n    if (isClosed) {\n      LOG.info(\"SegmentInfo is already closed: \" + toString());\n      return true;\n    }\n\n    Preconditions.checkNotNull(indexSegment, \"indexSegment should never be null.\");\n    isClosed = true;\n    indexSegment.destroyImmediately();\n\n    SegmentSyncConfig sync = getSyncInfo().getSegmentSyncConfig();\n    try {\n      String dirToClear = sync.getLocalSyncDirName(segment);\n      FileUtils.forceDelete(new File(dirToClear));\n      LOG.info(\"Deleted segment directory: \" + toString());\n      return true;\n    } catch (IOException e) {\n      LOG.error(\"Cannot clean up segment directory for segment: \" + toString(), e);\n      return false;\n    }\n  }\n\n  /**\n   * Delete the index segment directory after some configured delay.\n   * Note that we don't delete segments that are being uploaded.\n   * If a segment is being uploaded when we try to delete, close() retries the deletion later.\n   */\n  public void deleteIndexSegmentDirectoryAfterDelay() {\n    LOG.info(\"Scheduling SegmentInfo for deletion: \" + toString());\n    getEarlybirdIndexConfig().getResourceCloser().closeResourceQuietlyAfterDelay(\n        timeToWaitBeforeClosingMillis, () -> {\n          // Atomically check and set the being uploaded flag, if it is not set.\n          if (beingUploaded.compareAndSet(false, true)) {\n            // If successfully set the flag to true, we can delete immediately\n            setIsEnabled(false);\n            deleteLocalIndexedSegmentDirectoryImmediately();\n            LOG.info(\"Deleted index segment dir for segment: \"\n                + getSegment().getSegmentName());\n          } else {\n            // If the flag is already true (compareAndSet fails), we need to reschedule.\n            if (deletionRetries.decrementAndGet() > 0) {\n              LOG.warn(\"Segment is being uploaded, will retry deletion later. SegmentInfo: \"\n                  + getSegment().getSegmentName());\n              deleteIndexSegmentDirectoryAfterDelay();\n            } else {\n              LOG.warn(\"Failed to cleanup index segment dir for segment: \"\n                  + getSegment().getSegmentName());\n            }\n          }\n        });\n  }\n\n  public SegmentSyncInfo getSyncInfo() {\n    return segmentSyncInfo;\n  }\n\n  public FlushVersion getFlushVersion() {\n    return FlushVersion.CURRENT_FLUSH_VERSION;\n  }\n\n  public String getZkNodeName() {\n    return getSegmentName() + getFlushVersion().getVersionFileExtension();\n  }\n\n  static String getSyncDirName(String parentDir, String dbName, String version) {\n    return parentDir + \"/\" + dbName + version;\n  }\n\n  /**\n   * Parses the segment name from the name of the flushed directory.\n   */\n  public static String getSegmentNameFromFlushedDir(String flushedDir) {\n    String segmentName = null;\n    String[] fields = flushedDir.split(\"/\");\n    if (fields.length > 0) {\n      segmentName = fields[fields.length - 1];\n      segmentName = segmentName.replaceAll(FlushVersion.DELIMITER + \".*\", \"\");\n    }\n    return segmentName;\n  }\n\n  /**\n   * Flushes this segment to the given directory.\n   *\n   * @param dir The directory to flush the segment to.\n   * @throws IOException If the segment could not be flushed.\n   */\n  public void flush(Directory dir) throws IOException {\n    LOG.info(\"Flushing segment: {}\", getSegmentName());\n    try (PersistentFile.Writer writer = PersistentFile.getWriter(dir, getSegmentName())) {\n      FlushInfo flushInfo = new FlushInfo();\n      flushInfo.addLongProperty(UPDATE_STREAM_OFFSET_TIMESTAMP, getUpdatesStreamOffsetTimestamp());\n      getIndexSegment().flush(flushInfo, writer.getDataSerializer());\n\n      OutputStreamWriter infoFileWriter = new OutputStreamWriter(writer.getInfoFileOutputStream());\n      FlushInfo.flushAsYaml(flushInfo, infoFileWriter);\n    }\n  }\n\n  /**\n   * Makes a new SegmentInfo out of the current segment info, except that we switch the underlying\n   * segment.\n   */\n  public SegmentInfo copyWithEarlybirdSegment(EarlybirdSegment optimizedSegment) {\n    // Take everything from the current segment info that doesn't change for the new segment\n    // info and rebuild everything that can change.\n    TimeSlice newTimeSlice = new TimeSlice(\n      getTimeSliceID(),\n      EarlybirdConfig.getMaxSegmentSize(),\n      segment.getHashPartitionID(),\n      segment.getNumHashPartitions()\n    );\n    Segment newSegment = newTimeSlice.getSegment();\n\n    return new SegmentInfo(\n        optimizedSegment,\n        new SegmentSyncInfo(\n            segmentSyncInfo.getSegmentSyncConfig(),\n            newSegment),\n        newSegment,\n        earlybirdIndexConfig\n    );\n  }\n\n  /**\n   * Loads the segment from the given directory.\n   *\n   * @param dir The directory to load the segment from.\n   * @throws IOException If the segment could not be loaded.\n   */\n  public void load(Directory dir) throws IOException {\n    LOG.info(\"Loading segment: {}\", getSegmentName());\n    try (PersistentFile.Reader reader = PersistentFile.getReader(dir, getSegmentName())) {\n      FlushInfo flushInfo = FlushInfo.loadFromYaml(reader.getInfoInputStream());\n      setUpdatesStreamOffsetTimestamp(flushInfo.getLongProperty(UPDATE_STREAM_OFFSET_TIMESTAMP));\n      getIndexSegment().load(reader.getDataInputStream(), flushInfo);\n    }\n  }\n\n  private String getShortStatus() {\n    if (!isEnabled()) {\n      return \"disabled\";\n    }\n\n    if (isIndexing()) {\n      return \"indexing\";\n    }\n\n    if (isComplete()) {\n      return \"indexed\";\n    }\n\n    return \"pending\";\n  }\n\n  /**\n   * Get a string to be shown in admin commands which shows the query caches' sizes for this\n   * segment.\n   */\n  public String getQueryCachesData() {\n    StringBuilder out = new StringBuilder();\n    out.append(\"Segment: \" + getSegmentName() + \"\\n\");\n    out.append(\"Total documents: \" + LogFormatUtil.formatInt(\n        getIndexStats().getStatusCount()) + \"\\n\");\n    out.append(\"Query caches:\\n\");\n    for (Pair<String, Long> data : indexSegment.getQueryCachesData()) {\n      out.append(\"  \" + data.getFirst());\n      out.append(\": \");\n      out.append(LogFormatUtil.formatInt(data.getSecond()));\n      out.append(\"\\n\");\n    }\n    return out.toString();\n  }\n\n  public String getSegmentMetadata() {\n    return \"status: \" + getShortStatus() + \"\\n\"\n        + \"id: \" + getTimeSliceID() + \"\\n\"\n        + \"name: \" + getSegmentName() + \"\\n\"\n        + \"statusCount: \" + getIndexStats().getStatusCount() + \"\\n\"\n        + \"deleteCount: \" + getIndexStats().getDeleteCount() + \"\\n\"\n        + \"partialUpdateCount: \" + getIndexStats().getPartialUpdateCount() + \"\\n\"\n        + \"outOfOrderUpdateCount: \" + getIndexStats().getOutOfOrderUpdateCount() + \"\\n\"\n        + \"isEnabled: \" + isEnabled() + \"\\n\"\n        + \"isIndexing: \" + isIndexing() + \"\\n\"\n        + \"isComplete: \" + isComplete() + \"\\n\"\n        + \"isFlushed: \" + getSyncInfo().isFlushed() + \"\\n\"\n        + \"isOptimized: \" + isOptimized() + \"\\n\"\n        + \"isLoaded: \" + getSyncInfo().isLoaded() + \"\\n\"\n        + \"wasIndexed: \" + wasIndexed() + \"\\n\"\n        + \"queryCachesCardinality: \" + indexSegment.getQueryCachesCardinality() + \"\\n\";\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SegmentLoader.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.util.concurrent.TimeUnit;\n\nimport org.apache.commons.io.FileUtils;\nimport org.apache.commons.io.IOUtils;\nimport org.apache.hadoop.fs.FileStatus;\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.Path;\nimport org.apache.lucene.store.Directory;\nimport org.apache.lucene.store.FSDirectory;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.Timer;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.search.common.util.io.flushable.PersistentFile;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.exception.FlushVersionMismatchException;\nimport com.twitter.search.earlybird.stats.SegmentSyncStats;\n\npublic class SegmentLoader {\n  private static final Logger LOG = LoggerFactory.getLogger(SegmentLoader.class);\n  private static final SegmentSyncStats SEGMENT_LOAD_FROM_HDFS_STATS =\n      new SegmentSyncStats(\"load_from_hdfs\");\n\n  private final CriticalExceptionHandler criticalExceptionHandler;\n  private final SegmentSyncConfig segmentSyncConfig;\n\n  private final Clock clock;\n\n  public SegmentLoader(SegmentSyncConfig sync,\n                       CriticalExceptionHandler criticalExceptionHandler) {\n    this(sync, criticalExceptionHandler, Clock.SYSTEM_CLOCK);\n  }\n\n  public SegmentLoader(SegmentSyncConfig sync,\n                       CriticalExceptionHandler criticalExceptionHandler,\n                       Clock clock) {\n    this.criticalExceptionHandler = criticalExceptionHandler;\n    this.segmentSyncConfig = sync;\n    this.clock = clock;\n  }\n\n  public boolean load(SegmentInfo segmentInfo) {\n    return downloadSegment(segmentInfo) && loadSegmentFromDisk(segmentInfo);\n  }\n\n  /**\n   * Determines if the Earlybird should attempt to download the given segment from HDFS. This\n   * returns true if the segment is not already present on local disk, and the segment does exist\n   * on HDFS.\n   */\n  public boolean shouldDownloadSegmentWhileInServerSet(SegmentInfo segmentInfo) {\n    if (isValidSegmentOnDisk(segmentInfo)) {\n      return false;\n    }\n    try (FileSystem fs = HdfsUtil.getHdfsFileSystem()) {\n      return HdfsUtil.segmentExistsOnHdfs(fs, segmentInfo);\n    } catch (IOException e) {\n      LOG.error(\"Failed to check HDFS for segment \" + segmentInfo, e);\n      return false;\n    }\n  }\n\n  /**\n   * Verifies if the data for the given segment is present on the local disk, and if it's not,\n   * downloads it from HDFS.\n   */\n  public boolean downloadSegment(SegmentInfo segmentInfo) {\n    if (!segmentInfo.isEnabled()) {\n      LOG.debug(\"Segment is disabled: \" + segmentInfo);\n      return false;\n    }\n\n    if (segmentInfo.isIndexing() || segmentInfo.getSyncInfo().isLoaded()) {\n      LOG.debug(\"Cannot load indexing or loaded segment: \" + segmentInfo);\n      return false;\n    }\n\n    // Return whether the appropriate version is on disk, and if not, download it from HDFS.\n    return isValidSegmentOnDisk(segmentInfo) || checkSegmentOnHdfsAndCopyLocally(segmentInfo);\n  }\n\n  /**\n   * Loads the data for the given segment from the local disk.\n   */\n  public boolean loadSegmentFromDisk(SegmentInfo segmentInfo) {\n    if (segmentInfo.isIndexing()) {\n      LOG.error(\"Tried to load current segment!\");\n      return false;\n    }\n\n    segmentInfo.setIndexing(true);\n    try {\n      File flushDir = new File(segmentInfo.getSyncInfo().getLocalSyncDir());\n      Directory loadDir = FSDirectory.open(flushDir.toPath());\n\n      segmentInfo.load(loadDir);\n\n      if (!verifySegmentStatusCountLargeEnough(segmentInfo)) {\n        SearchRateCounter.export(\n            \"segment_loader_failed_too_few_tweets_in_segment_\" + segmentInfo.getSegmentName())\n            .increment();\n        return false;\n      }\n\n      segmentInfo.setIndexing(false);\n      segmentInfo.setComplete(true);\n      segmentInfo.getSyncInfo().setLoaded(true);\n      return true;\n    } catch (FlushVersionMismatchException e) {\n      handleException(segmentInfo, e);\n      // If earlybird is in starting state, handler will terminate it\n      criticalExceptionHandler.handle(this, e);\n    } catch (Exception e) {\n      handleException(segmentInfo, e);\n    }\n\n    SearchRateCounter.export(\"segment_loader_failed_\" + segmentInfo.getSegmentName()).increment();\n    return false;\n  }\n\n  // Check to see if the segment exists on disk, and its checksum passes.\n  private boolean isValidSegmentOnDisk(SegmentInfo segment) {\n    String loadDirStr = segment.getSyncInfo().getLocalSyncDir();\n    File loadDir = new File(loadDirStr);\n\n    if (!loadDir.exists()) {\n      return false;\n    }\n\n    for (String persistentFileName : segmentSyncConfig.getPersistentFileNames(segment)) {\n      if (!verifyInfoChecksum(loadDir, persistentFileName)) {\n        return false;\n      }\n    }\n\n    return true;\n  }\n\n  private static boolean verifyInfoChecksum(File loadDir, String databaseName) {\n    if (checksumFileExists(loadDir, databaseName)) {\n      try {\n        Directory dir = FSDirectory.open(loadDir.toPath());\n        PersistentFile.Reader reader = PersistentFile.getReader(dir, databaseName);\n        try {\n          reader.verifyInfoChecksum();\n          return true;\n        } finally {\n          IOUtils.closeQuietly(reader);\n          IOUtils.closeQuietly(dir);\n        }\n      } catch (PersistentFile.CorruptFileException e) {\n        LOG.error(\"Failed checksum verification.\", e);\n      } catch (IOException e) {\n        LOG.error(\"Error while trying to read checksum file\", e);\n      }\n    }\n    return false;\n  }\n\n  // Check that the loaded segment's status count is higher than the configured threshold\n  private boolean verifySegmentStatusCountLargeEnough(SegmentInfo segmentInfo) {\n    long segmentStatusCount = segmentInfo.getIndexStats().getStatusCount();\n    if (segmentStatusCount > segmentSyncConfig.getMinSegmentStatusCountThreshold()) {\n      return true;\n    } else if (segmentInfo.getEarlybirdIndexConfig().isIndexStoredOnDisk()\n        && couldBeMostRecentArchiveSegment(segmentInfo)) {\n      // The most recent archive earlybird segment is expected to be incomplete\n      LOG.info(\"Segment status count (\" + segmentStatusCount + \") is below the threshold of \"\n          + segmentSyncConfig.getMinSegmentStatusCountThreshold()\n          + \", but this is expected because the most recent segment is expected to be incomplete: \"\n          + segmentInfo);\n      return true;\n    } else {\n      // The segment status count is small so the segment is likely incomplete.\n      LOG.error(\"Segment status count (\" + segmentStatusCount + \") is below the threshold of \"\n          + segmentSyncConfig.getMinSegmentStatusCountThreshold() + \": \" + segmentInfo);\n      segmentInfo.setIndexing(false);\n      segmentInfo.getSyncInfo().setLoaded(false);\n\n      // Remove segment from local disk\n      if (!segmentInfo.deleteLocalIndexedSegmentDirectoryImmediately()) {\n        LOG.error(\"Failed to cleanup unloadable segment directory.\");\n      }\n\n      return false;\n    }\n  }\n\n  // Check if this segment could be the most recent archive earlybird segment (would be on the\n  // latest tier). Archive segments tend to span around 12 days, so using a conservative threshold\n  // of 20 days.\n  private boolean couldBeMostRecentArchiveSegment(SegmentInfo segmentInfo) {\n    long timesliceAgeMs =\n        SnowflakeIdParser.getTweetAgeInMs(clock.nowMillis(), segmentInfo.getTimeSliceID());\n    return (timesliceAgeMs / 1000 / 60 / 60 / 24) <= 20;\n  }\n\n  /**\n   * Check to see if the segment exists on hdfs. Will look for the correct segment version\n   * uploaded by any of the hosts.\n   * If the segment exists on hdfs, the segment will be copied from hdfs to the local file\n   * system, and we will verify the checksum against the copied version.\n   * @return true iff the segment was copied to local disk, and the checksum is verified.\n   */\n  private boolean checkSegmentOnHdfsAndCopyLocally(SegmentInfo segment) {\n    if (!segmentSyncConfig.isSegmentLoadFromHdfsEnabled()) {\n      return isValidSegmentOnDisk(segment);\n    }\n\n    LOG.info(\"About to start downloading segment from hdfs: \" + segment);\n    Timer timer = new Timer(TimeUnit.MILLISECONDS);\n    String status = null;\n    String localBaseDir = segment.getSyncInfo().getLocalSyncDir();\n    FileSystem fs = null;\n    try {\n      fs = HdfsUtil.getHdfsFileSystem();\n\n      String hdfsBaseDirPrefix = segment.getSyncInfo().getHdfsSyncDirPrefix();\n      FileStatus[] statuses = fs.globStatus(new Path(hdfsBaseDirPrefix));\n      if (statuses != null && statuses.length > 0) {\n        Path hdfsSyncPath = statuses[0].getPath();\n        copySegmentFilesFromHdfs(segment, segmentSyncConfig, fs, hdfsSyncPath);\n        status = \"loaded\";\n      } else {\n        LOG.info(\"No segments found in hdfs under: \" + hdfsBaseDirPrefix);\n        status = \"notloaded\";\n      }\n      fs.close();\n    } catch (IOException ex) {\n      LOG.error(\"Failed copying segment from hdfs: \" + segment + \" after: \"\n                + timer.stop() + \" ms\", ex);\n      status = \"exception\";\n      SEGMENT_LOAD_FROM_HDFS_STATS.recordError();\n      try {\n        FileUtils.deleteDirectory(new File(localBaseDir));\n      } catch (IOException e) {\n        LOG.error(\"Error cleaning up local segment directory: \" + segment, e);\n      }\n    } finally {\n      timer.stop();\n      SEGMENT_LOAD_FROM_HDFS_STATS.actionComplete(timer);\n      LOG.info(\"Download from hdfs completed in \"\n          + timer.getElapsed() + \" milliseconds: \" + segment + \" status: \" + status);\n      IOUtils.closeQuietly(fs);\n    }\n\n    // now check to see if we have successfully copied the segment\n    return isValidSegmentOnDisk(segment);\n  }\n\n  private static void copySegmentFilesFromHdfs(SegmentInfo segment,\n                                               SegmentSyncConfig syncConfig,\n                                               FileSystem fs,\n                                               Path hdfsSyncPath) throws IOException {\n    String localBaseDir = segment.getSyncInfo().getLocalSyncDir();\n    File localBaseDirFile = new File(localBaseDir);\n    FileUtils.deleteQuietly(localBaseDirFile);\n    if (localBaseDirFile.exists()) {\n      LOG.warn(\"Cannot delete the existing path: \" + localBaseDir);\n    }\n    for (String fileName : syncConfig.getAllSyncFileNames(segment)) {\n      Path hdfsFilePath = new Path(hdfsSyncPath, fileName);\n      String localFileName = localBaseDir + \"/\" + fileName;\n      LOG.debug(\"About to start loading from hdfs: \" + fileName + \" from: \"\n                + hdfsFilePath + \" to: \" + localFileName);\n\n      Timer timer = new Timer(TimeUnit.MILLISECONDS);\n      fs.copyToLocalFile(hdfsFilePath, new Path(localFileName));\n      LOG.debug(\"Loaded segment file from hdfs: \" + fileName + \" from: \"\n                + hdfsFilePath + \" to: \" + localFileName + \" in: \" + timer.stop() + \" ms.\");\n    }\n\n    LOG.info(\"Finished downloading segments from \" + hdfsSyncPath);\n  }\n\n  private static boolean checksumFileExists(File loadDir, String databaseName) {\n    String checksumFileName = PersistentFile.genChecksumFileName(databaseName);\n    File checksumFile = new File(loadDir, checksumFileName);\n\n    return checksumFile.exists();\n  }\n\n  private void handleException(SegmentInfo segmentInfo, Exception e) {\n    LOG.error(\"Exception while loading IndexSegment from \"\n        + segmentInfo.getSyncInfo().getLocalSyncDir(), e);\n\n    segmentInfo.setIndexing(false);\n    segmentInfo.getSyncInfo().setLoaded(false);\n    if (!segmentInfo.deleteLocalIndexedSegmentDirectoryImmediately()) {\n      LOG.error(\"Failed to cleanup unloadable segment directory.\");\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SegmentManager.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.Comparator;\nimport java.util.HashSet;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.concurrent.ConcurrentSkipListMap;\nimport java.util.stream.Collectors;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Predicate;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.partitioning.base.Segment;\nimport com.twitter.search.common.partitioning.base.TimeSlice;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.common.CaughtUpMonitor;\nimport com.twitter.search.earlybird.common.userupdates.UserScrubGeoMap;\nimport com.twitter.search.earlybird.common.userupdates.UserUpdate;\nimport com.twitter.search.earlybird.common.userupdates.UserUpdatesChecker;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.index.EarlybirdSegmentFactory;\nimport com.twitter.search.earlybird.index.EarlybirdSingleSegmentSearcher;\nimport com.twitter.search.earlybird.search.EarlybirdLuceneSearcher;\nimport com.twitter.search.earlybird.search.EarlybirdMultiSegmentSearcher;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.tweetypie.thriftjava.UserScrubGeoEvent;\n\npublic class SegmentManager {\n  private static final Logger LOG = LoggerFactory.getLogger(SegmentManager.class);\n  private final Clock clock;\n  private static final String STATS_PREFIX = \"segment_manager_\";\n  private static final SearchLongGauge SEGMENT_COUNT_STATS =\n          SearchLongGauge.export(STATS_PREFIX + \"total_segments\");\n  private static final SearchCounter OPTIMIZED_SEGMENTS =\n          SearchCounter.export(STATS_PREFIX + \"optimized_segments\");\n  private static final SearchCounter UNOPTIMIZED_SEGMENTS =\n          SearchCounter.export(STATS_PREFIX + \"unoptimized_segments\");\n\n  public enum Filter {\n    All(info -> true),\n    Enabled(SegmentInfo::isEnabled),\n    NeedsIndexing(SegmentInfo::needsIndexing),\n    Complete(SegmentInfo::isComplete);\n\n    private final Predicate<SegmentInfo> predicate;\n\n    Filter(Predicate<SegmentInfo> predicate) {\n      this.predicate = predicate;\n    }\n\n    private static final Map<String, Filter> NAME_INDEX =\n        Maps.newHashMapWithExpectedSize(Filter.values().length);\n\n    static {\n      for (Filter filter : Filter.values()) {\n        NAME_INDEX.put(filter.name().toLowerCase(), filter);\n      }\n    }\n\n    /**\n     * Parses the filter from the given string, based on the filter name.\n     */\n    public static Filter fromStringIgnoreCase(String str) {\n      if (str == null) {\n        return null;\n      }\n\n      return NAME_INDEX.get(str.toLowerCase());\n    }\n  }\n\n  public enum Order {\n    OLD_TO_NEW,\n    NEW_TO_OLD,\n  }\n\n  /**\n   * A listener that gets notified when the list of segments changes.\n   */\n  public interface SegmentUpdateListener {\n    /**\n     * Called with the new list of segments when it changes.\n     *\n     * @param segments The new list of segments.\n     */\n    void update(Collection<SegmentInfo> segments, String message);\n  }\n\n  private final List<SegmentUpdateListener> updateListeners =\n          Collections.synchronizedList(Lists.newLinkedList());\n\n  private final ConcurrentSkipListMap<Long, ISegmentWriter> segmentWriters =\n      new ConcurrentSkipListMap<>();\n\n  private final Set<Long> badTimesliceIds = new HashSet<>();\n\n  private final int maxEnabledSegments;\n  private final int maxSegmentSize;\n  private final EarlybirdSegmentFactory earlybirdSegmentFactory;\n  private final UserTable userTable;\n  private final UserScrubGeoMap userScrubGeoMap;\n  private final EarlybirdIndexConfig earlybirdIndexConfig;\n  private final DynamicPartitionConfig dynamicPartitionConfig;\n  private final UserUpdatesChecker userUpdatesChecker;\n  private final SegmentSyncConfig segmentSyncConfig;\n  private final EarlybirdSearcherStats searcherStats;\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n  private final CriticalExceptionHandler criticalExceptionHandler;\n  private final CaughtUpMonitor indexCaughtUpMonitor;\n\n  public SegmentManager(\n      DynamicPartitionConfig dynamicPartitionConfig,\n      EarlybirdIndexConfig earlybirdIndexConfig,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      EarlybirdSearcherStats searcherStats,\n      SearchStatsReceiver earlybirdStatsReceiver,\n      UserUpdatesChecker userUpdatesChecker,\n      SegmentSyncConfig segmentSyncConfig,\n      UserTable userTable,\n      UserScrubGeoMap userScrubGeoMap,\n      Clock clock,\n      int maxSegmentSize,\n      CriticalExceptionHandler criticalExceptionHandler,\n      CaughtUpMonitor indexCaughtUpMonitor) {\n\n    PartitionConfig curPartitionConfig = dynamicPartitionConfig.getCurrentPartitionConfig();\n\n    this.userTable = userTable;\n    this.userScrubGeoMap = userScrubGeoMap;\n\n    this.earlybirdSegmentFactory = new EarlybirdSegmentFactory(\n        earlybirdIndexConfig,\n        searchIndexingMetricSet,\n        searcherStats,\n        clock);\n    this.earlybirdIndexConfig = earlybirdIndexConfig;\n    this.maxEnabledSegments = curPartitionConfig.getMaxEnabledLocalSegments();\n    this.dynamicPartitionConfig = dynamicPartitionConfig;\n    this.userUpdatesChecker = userUpdatesChecker;\n    this.segmentSyncConfig = segmentSyncConfig;\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n    this.searcherStats = searcherStats;\n    this.clock = clock;\n    this.maxSegmentSize = maxSegmentSize;\n    this.criticalExceptionHandler = criticalExceptionHandler;\n    this.indexCaughtUpMonitor = indexCaughtUpMonitor;\n\n    earlybirdStatsReceiver.getCustomGauge(\"total_loaded_segments\",\n        segmentWriters::size);\n    earlybirdStatsReceiver.getCustomGauge(\"total_indexed_documents\",\n        this::getNumIndexedDocuments);\n    earlybirdStatsReceiver.getCustomGauge(\"total_segment_size_bytes\",\n        this::getTotalSegmentSizeOnDisk);\n    earlybirdStatsReceiver.getCustomGauge(\"earlybird_index_depth_millis\",\n        this::getIndexDepthMillis);\n  }\n\n  /**\n   * Logs the current state of this segment manager.\n   *\n   * @param label A label that should identify the segment manager.\n   */\n  public void logState(String label) {\n    StringBuilder sb = new StringBuilder();\n    sb.append(\"State of SegmentManager (\" + label + \"):\\n\");\n    sb.append(\"Number of segments: \" + segmentWriters.size());\n    boolean hasSegments = false;\n    for (Map.Entry<Long, ISegmentWriter> entry : this.segmentWriters.entrySet()) {\n      SegmentInfo segmentInfo = entry.getValue().getSegmentInfo();\n      hasSegments = true;\n\n      sb.append(String.format(\"\\nSegment (%s): isClosed: %5s, isComplete: %5s, \"\n              + \"isEnabled: %5s, isIndexing: %5s, isOptimized: %5s, wasIndexed: %5s\",\n          segmentInfo.getSegmentName(),\n          segmentInfo.isClosed(),\n          segmentInfo.isComplete(),\n          segmentInfo.isEnabled(),\n          segmentInfo.isIndexing(),\n          segmentInfo.isOptimized(),\n          segmentInfo.wasIndexed()\n      ));\n\n      sb.append(String.format(\" | Index stats: %s\", segmentInfo.getIndexStats().toString()));\n    }\n    if (!hasSegments) {\n      sb.append(\" No segments.\");\n    }\n    LOG.info(sb.toString());\n  }\n\n\n  public PartitionConfig getPartitionConfig() {\n    return dynamicPartitionConfig.getCurrentPartitionConfig();\n  }\n\n  public int getMaxEnabledSegments() {\n    return maxEnabledSegments;\n  }\n\n  public EarlybirdSegmentFactory getEarlybirdSegmentFactory() {\n    return earlybirdSegmentFactory;\n  }\n\n  public EarlybirdIndexConfig getEarlybirdIndexConfig() {\n    return earlybirdIndexConfig;\n  }\n\n  public UserTable getUserTable() {\n    return userTable;\n  }\n\n  public UserScrubGeoMap getUserScrubGeoMap() {\n    return userScrubGeoMap;\n  }\n\n  @VisibleForTesting\n  public void reset() {\n    segmentWriters.clear();\n  }\n\n  /**\n   * Returns the list of all segments that match the given filter, in the given order.\n   */\n  public Iterable<SegmentInfo> getSegmentInfos(Filter filter, Order order) {\n    Comparator<SegmentInfo> comparator;\n\n    if (order == Order.OLD_TO_NEW) {\n      comparator = Comparator.naturalOrder();\n    } else {\n      comparator = Comparator.reverseOrder();\n    }\n\n    return () -> segmentWriters.values().stream()\n        .map(ISegmentWriter::getSegmentInfo)\n        .filter(filter.predicate::apply)\n        .sorted(comparator)\n        .iterator();\n  }\n\n  private void createAndPutSegmentInfo(Segment segment) throws IOException {\n    LOG.info(\"Creating new SegmentInfo for segment \" + segment.getSegmentName());\n    putSegmentInfo(new SegmentInfo(segment, earlybirdSegmentFactory, segmentSyncConfig));\n  }\n\n  /**\n   * Updates the list of segments managed by this manager, based on the given list.\n   */\n  public void updateSegments(List<Segment> segmentsList) throws IOException {\n    // Truncate to the amount of segments we want to keep enabled.\n    List<Segment> truncatedSegmentList =\n        SegmentManager.truncateSegmentList(segmentsList, maxEnabledSegments);\n\n    final long newestTimeSliceID = getNewestTimeSliceID();\n    final Set<Long> segmentsToDisable = new HashSet<>(segmentWriters.keySet());\n\n    for (Segment segment : truncatedSegmentList) {\n      final long timeSliceID = segment.getTimeSliceID();\n      segmentsToDisable.remove(timeSliceID);\n\n      // On the first loop iteration of the first call to updateSegments(), newestTimeSliceID should\n      // be set to -1, so the condition should be false. After that, all segments should either be\n      // newer than the latest process segment, or if we're replacing an old segment, it should have\n      // a SegmentInfo instance associated with it.\n      if (timeSliceID <= newestTimeSliceID) {\n        ISegmentWriter segmentWriter = segmentWriters.get(timeSliceID);\n        // Old time slice ID. It should have a SegmentInfo instance associated with it.\n        if (segmentWriter == null) {\n          if (!badTimesliceIds.contains(timeSliceID)) {\n            // We're dealing with a bad timeslice. Log an error, but do it only once per timeslice.\n            LOG.error(\"The SegmentInfo instance associated with an old timeSliceID should never be \"\n                      + \"null. TimeSliceID: {}\", timeSliceID);\n            badTimesliceIds.add(timeSliceID);\n          }\n        } else if (segmentWriter.getSegmentInfo().isClosed()) {\n          // If the SegmentInfo was closed, create a new one.\n          LOG.info(\"SegmentInfo for segment {} is closed.\", segment.getSegmentName());\n          createAndPutSegmentInfo(segment);\n        }\n      } else {\n        // New time slice ID: create a SegmentInfo instance for it.\n        createAndPutSegmentInfo(segment);\n      }\n    }\n\n    // Anything we didn't see locally can be disabled.\n    for (Long segmentID : segmentsToDisable) {\n      disableSegment(segmentID);\n    }\n\n    // Update segment stats and other exported variables.\n    updateStats();\n  }\n\n  /**\n   * Re-export stats after a segment has changed, or the set of segments has changed.\n   */\n  public void updateStats() {\n    // Update the partition count stats.\n    SEGMENT_COUNT_STATS.set(segmentWriters.size());\n\n    OPTIMIZED_SEGMENTS.reset();\n    UNOPTIMIZED_SEGMENTS.reset();\n    for (ISegmentWriter writer : segmentWriters.values()) {\n      if (writer.getSegmentInfo().isOptimized()) {\n        OPTIMIZED_SEGMENTS.increment();\n      } else {\n        UNOPTIMIZED_SEGMENTS.increment();\n      }\n    }\n  }\n\n  private long getIndexDepthMillis() {\n    long oldestTimeSliceID = getOldestEnabledTimeSliceID();\n    if (oldestTimeSliceID == SegmentInfo.INVALID_ID) {\n      return 0;\n    } else {\n      // Compute timestamp from timesliceId, which is also a snowflake tweetId\n      long timestamp = SnowflakeIdParser.getTimestampFromTweetId(oldestTimeSliceID);\n      // Set current index depth in milliseconds\n      long indexDepthInMillis = System.currentTimeMillis() - timestamp;\n      // Index depth should never be negative.\n      if (indexDepthInMillis < 0) {\n        LOG.warn(\"Negative index depth. Large time skew on this Earlybird?\");\n        return 0;\n      } else {\n        return indexDepthInMillis;\n      }\n    }\n  }\n\n  private void updateExportedSegmentStats() {\n    int index = 0;\n    for (SegmentInfo segmentInfo : getSegmentInfos(Filter.Enabled, Order.NEW_TO_OLD)) {\n      SegmentIndexStatsExporter.export(segmentInfo, index++);\n    }\n  }\n\n  // Marks the SegmentInfo object matching this time slice as disabled.\n  private void disableSegment(long timeSliceID) {\n    SegmentInfo info = getSegmentInfo(timeSliceID);\n    if (info == null) {\n      LOG.warn(\"Tried to disable missing segment \" + timeSliceID);\n      return;\n    }\n    info.setIsEnabled(false);\n    LOG.info(\"Disabled segment \" + info);\n  }\n\n  public long getNewestTimeSliceID() {\n    final Iterator<SegmentInfo> segments = getSegmentInfos(Filter.All, Order.NEW_TO_OLD).iterator();\n    return segments.hasNext() ? segments.next().getTimeSliceID() : SegmentInfo.INVALID_ID;\n  }\n\n  /**\n   * Returns the timeslice ID of the oldest enabled segment.\n   */\n  public long getOldestEnabledTimeSliceID() {\n    if (segmentWriters.size() == 0) {\n      return SegmentInfo.INVALID_ID;\n    }\n    ISegmentWriter segmentWriter = segmentWriters.firstEntry().getValue();\n    return segmentWriter.getSegmentInfo().getTimeSliceID();\n  }\n\n  /**\n   * Returns the SegmentInfo for the given timeSliceID.\n   */\n  public final SegmentInfo getSegmentInfo(long timeSliceID) {\n    ISegmentWriter segmentWriter = segmentWriters.get(timeSliceID);\n    return segmentWriter == null ? null : segmentWriter.getSegmentInfo();\n  }\n\n  /**\n   * Returns the segment info for the segment that should contain the given tweet ID.\n   */\n  public final SegmentInfo getSegmentInfoFromStatusID(long tweetID) {\n    for (SegmentInfo segmentInfo : getSegmentInfos(Filter.All, Order.NEW_TO_OLD)) {\n      if (tweetID >= segmentInfo.getTimeSliceID()) {\n        return segmentInfo;\n      }\n    }\n\n    return null;\n  }\n\n  /**\n   * Removes the segment associated with the given timeslice ID from the segment manager. This will\n   * also take care of all required clean up related to the segment being removed, such as closing\n   * its writer.\n   */\n  public boolean removeSegmentInfo(long timeSliceID) {\n    if (timeSliceID == getNewestTimeSliceID()) {\n      throw new RuntimeException(\"Cannot drop segment of current time-slice \" + timeSliceID);\n    }\n\n    ISegmentWriter removed = segmentWriters.get(timeSliceID);\n    if (removed == null) {\n      return false;\n    }\n\n    LOG.info(\"Removing segment {}\", removed.getSegmentInfo());\n    Preconditions.checkState(!removed.getSegmentInfo().isEnabled());\n    removed.getSegmentInfo().getIndexSegment().close();\n    segmentWriters.remove(timeSliceID);\n\n    String segmentName = removed.getSegmentInfo().getSegmentName();\n    updateAllListeners(\"Removed segment \" + segmentName);\n    LOG.info(\"Removed segment \" + segmentName);\n    updateExportedSegmentStats();\n    updateStats();\n    return true;\n  }\n\n  /**\n   * Add the given SegmentWriter into the segmentWriters map.\n   * If a segment with the same timesliceID already exists in the map, the old one is replaced\n   * with the new one; this should only happen in the archive.\n   *\n   * The replaced segment is destroyed after a delay to allow in-flight requests to finish.\n   */\n  public ISegmentWriter putSegmentInfo(SegmentInfo info) {\n    ISegmentWriter usedSegmentWriter;\n\n    SegmentWriter segmentWriter\n        = new SegmentWriter(info, searchIndexingMetricSet.updateFreshness);\n\n    if (!info.isOptimized()) {\n      LOG.info(\"Inserting an optimizing segment writer for segment: {}\",\n          info.getSegmentName());\n\n      usedSegmentWriter = new OptimizingSegmentWriter(\n          segmentWriter,\n          criticalExceptionHandler,\n          searchIndexingMetricSet,\n          indexCaughtUpMonitor);\n    } else {\n      usedSegmentWriter = segmentWriter;\n    }\n\n    putSegmentWriter(usedSegmentWriter);\n    return usedSegmentWriter;\n  }\n\n  private void putSegmentWriter(ISegmentWriter segmentWriter) {\n    SegmentInfo newSegmentInfo = segmentWriter.getSegmentInfo();\n    SegmentInfo oldSegmentInfo = getSegmentInfo(newSegmentInfo.getTimeSliceID());\n\n    // Some sanity checks.\n    if (oldSegmentInfo != null) {\n      // This map is thread safe, so this put can be considered atomic.\n      segmentWriters.put(newSegmentInfo.getTimeSliceID(), segmentWriter);\n      LOG.info(\"Replaced SegmentInfo with a new one in segmentWriters map. \"\n          + \"Old SegmentInfo: {} New SegmentInfo: {}\", oldSegmentInfo, newSegmentInfo);\n\n      if (!oldSegmentInfo.isClosed()) {\n        oldSegmentInfo.deleteIndexSegmentDirectoryAfterDelay();\n      }\n    } else {\n      long newestTimeSliceID = getNewestTimeSliceID();\n      if (newestTimeSliceID != SegmentInfo.INVALID_ID\n          && newestTimeSliceID > newSegmentInfo.getTimeSliceID()) {\n        LOG.error(\"Not adding out-of-order segment \" + newSegmentInfo);\n        return;\n      }\n\n      segmentWriters.put(newSegmentInfo.getTimeSliceID(), segmentWriter);\n      LOG.info(\"Added segment \" + newSegmentInfo);\n    }\n\n    updateAllListeners(\"Added segment \" + newSegmentInfo.getTimeSliceID());\n    updateExportedSegmentStats();\n    updateStats();\n  }\n\n  private SegmentInfo createSegmentInfo(long timesliceID) throws IOException {\n    PartitionConfig partitionConfig = dynamicPartitionConfig.getCurrentPartitionConfig();\n\n    TimeSlice timeSlice = new TimeSlice(\n        timesliceID,\n        maxSegmentSize,\n        partitionConfig.getIndexingHashPartitionID(),\n        partitionConfig.getNumPartitions());\n\n    SegmentInfo segmentInfo =\n        new SegmentInfo(timeSlice.getSegment(), earlybirdSegmentFactory, segmentSyncConfig);\n\n    return segmentInfo;\n  }\n\n  /**\n   * Create a new optimizing segment writer and add it to the map.\n   */\n  public OptimizingSegmentWriter createAndPutOptimizingSegmentWriter(\n      long timesliceID) throws IOException {\n    SegmentInfo segmentInfo = createSegmentInfo(timesliceID);\n\n    OptimizingSegmentWriter writer = new OptimizingSegmentWriter(\n        new SegmentWriter(segmentInfo, searchIndexingMetricSet.updateFreshness),\n        criticalExceptionHandler,\n        searchIndexingMetricSet,\n        indexCaughtUpMonitor);\n\n    putSegmentWriter(writer);\n    return writer;\n  }\n\n  /**\n   * Create a new segment writer.\n   */\n  public SegmentWriter createSegmentWriter(long timesliceID) throws IOException {\n    SegmentInfo segmentInfo = createSegmentInfo(timesliceID);\n\n    SegmentWriter writer = new SegmentWriter(\n        segmentInfo, searchIndexingMetricSet.updateFreshness);\n\n    return writer;\n  }\n\n  private void updateAllListeners(String message) {\n    List<SegmentInfo> segmentInfos = segmentWriters.values().stream()\n        .map(ISegmentWriter::getSegmentInfo)\n        .collect(Collectors.toList());\n    for (SegmentUpdateListener listener : updateListeners) {\n      try {\n        listener.update(segmentInfos, message);\n      } catch (Exception e) {\n        LOG.warn(\"SegmentManager: Unable to call update() on listener.\", e);\n      }\n    }\n  }\n\n  // Returns true if the map contains a SegmentInfo matching the given time slice.\n  public final boolean hasSegmentInfo(long timeSliceID) {\n    return segmentWriters.containsKey(timeSliceID);\n  }\n\n  public void addUpdateListener(SegmentUpdateListener listener) {\n    updateListeners.add(listener);\n  }\n\n  /**\n   * Look up the segment containing the given status id.\n   * If found, its timeslice id is returned.\n   * If none found, -1 is returned.\n   */\n  public long lookupTimeSliceID(long statusID) throws IOException {\n    SegmentInfo segmentInfo = getSegmentInfoForID(statusID);\n    if (segmentInfo == null) {\n      return -1;\n    }\n    if (!segmentInfo.getIndexSegment().hasDocument(statusID)) {\n        return -1;\n    }\n\n    return segmentInfo.getTimeSliceID();\n  }\n\n  /**\n   * Truncates the given segment list to the specified number of segments, by keeping the newest\n   * segments.\n   */\n  @VisibleForTesting\n  public static List<Segment> truncateSegmentList(List<Segment> segmentList, int maxNumSegments) {\n    // Maybe cut-off the beginning of the sorted list of IDs.\n    if (maxNumSegments > 0 && maxNumSegments < segmentList.size()) {\n      return segmentList.subList(segmentList.size() - maxNumSegments, segmentList.size());\n    } else {\n      return segmentList;\n    }\n  }\n\n  @VisibleForTesting\n  public void setOffensive(long userID, boolean offensive) {\n    userTable.setOffensive(userID, offensive);\n  }\n\n  @VisibleForTesting\n  public void setAntisocial(long userID, boolean antisocial) {\n    userTable.setAntisocial(userID, antisocial);\n  }\n\n  /**\n   * Returns a searcher for all segments.\n   */\n  public EarlybirdMultiSegmentSearcher getMultiSearcher(ImmutableSchemaInterface schemaSnapshot)\n      throws IOException {\n    return new EarlybirdMultiSegmentSearcher(\n        schemaSnapshot,\n        getSearchers(schemaSnapshot, Filter.All, Order.NEW_TO_OLD),\n        searcherStats,\n        clock);\n  }\n\n  /**\n   * Returns a new searcher for the given segment.\n   */\n  @Nullable\n  public EarlybirdLuceneSearcher getSearcher(\n      Segment segment,\n      ImmutableSchemaInterface schemaSnapshot) throws IOException {\n    return getSearcher(segment.getTimeSliceID(), schemaSnapshot);\n  }\n\n  /**\n   * Get max tweet id across all enabled segments.\n   * @return max tweet id or -1 if none found\n   */\n  public long getMaxTweetIdFromEnabledSegments() {\n    for (SegmentInfo segmentInfo : getSegmentInfos(Filter.Enabled, Order.NEW_TO_OLD)) {\n      long maxTweetId = segmentInfo.getIndexSegment().getMaxTweetId();\n      if (maxTweetId != -1) {\n        return maxTweetId;\n      }\n    }\n\n    return -1;\n  }\n\n  /**\n   * Create a tweet index searcher on the segment represented by the timeslice id.  For production\n   * search session, the schema snapshot should be always passed in to make sure that the schema\n   * usage inside scoring is consistent.\n   *\n   * For non-production usage, like one-off debugging search, you can use the function call without\n   * the schema snapshot.\n   *\n   * @param timeSliceID the timeslice id, which represents the index segment\n   * @param schemaSnapshot the schema snapshot\n   * @return the tweet index searcher\n   */\n  @Nullable\n  public EarlybirdSingleSegmentSearcher getSearcher(\n      long timeSliceID,\n      ImmutableSchemaInterface schemaSnapshot) throws IOException {\n    SegmentInfo segmentInfo = getSegmentInfo(timeSliceID);\n    if (segmentInfo == null) {\n      return null;\n    }\n    return segmentInfo.getIndexSegment().getSearcher(userTable, schemaSnapshot);\n  }\n\n  /**\n   * Returns a new searcher for the segment with the given timeslice ID. If the given timeslice ID\n   * does not correspond to any active segment, {@code null} is returned.\n   *\n   * @param timeSliceID The segment's timeslice ID.\n   * @return A new searcher for the segment with the given timeslice ID.\n   */\n  @Nullable\n  public EarlybirdSingleSegmentSearcher getSearcher(long timeSliceID) throws IOException {\n    SegmentInfo segmentInfo = getSegmentInfo(timeSliceID);\n    if (segmentInfo == null) {\n      return null;\n    }\n    return segmentInfo.getIndexSegment().getSearcher(userTable);\n  }\n\n  @Nullable\n  public EarlybirdResponseCode checkSegment(Segment segment) {\n    return checkSegmentInternal(getSegmentInfo(segment.getTimeSliceID()));\n  }\n\n  private static EarlybirdResponseCode checkSegmentInternal(SegmentInfo info) {\n    if (info == null) {\n      return EarlybirdResponseCode.PARTITION_NOT_FOUND;\n    } else if (info.isEnabled()) {\n      return EarlybirdResponseCode.SUCCESS;\n    } else {\n      return EarlybirdResponseCode.PARTITION_DISABLED;\n    }\n  }\n\n  private List<EarlybirdSingleSegmentSearcher> getSearchers(\n      ImmutableSchemaInterface schemaSnapshot,\n      Filter filter,\n      Order order) throws IOException {\n    List<EarlybirdSingleSegmentSearcher> searchers = Lists.newArrayList();\n    for (SegmentInfo segmentInfo : getSegmentInfos(filter, order)) {\n      EarlybirdSingleSegmentSearcher searcher =\n          segmentInfo.getIndexSegment().getSearcher(userTable, schemaSnapshot);\n      if (searcher != null) {\n        searchers.add(searcher);\n      }\n    }\n    return searchers;\n  }\n\n  /**\n   * Gets metadata for segments for debugging purposes.\n   */\n  public List<String> getSegmentMetadata() {\n    List<String> segmentMetadata = new ArrayList<>();\n    for (SegmentInfo segment : getSegmentInfos(Filter.All, Order.OLD_TO_NEW)) {\n      segmentMetadata.add(segment.getSegmentMetadata());\n    }\n    return segmentMetadata;\n  }\n\n  /**\n   * Gets info for query caches to be displayed in an admin page.\n   */\n  public String getQueryCachesData() {\n    StringBuilder output = new StringBuilder();\n    for (SegmentInfo segment : getSegmentInfos(Filter.All, Order.OLD_TO_NEW)) {\n      output.append(segment.getQueryCachesData() + \"\\n\");\n    }\n    return output.toString();\n  }\n\n  /**\n   * Index the given user update. Returns false if the given update is skipped.\n   */\n  public boolean indexUserUpdate(UserUpdate userUpdate) {\n    return userTable.indexUserUpdate(userUpdatesChecker, userUpdate);\n  }\n\n  /**\n   * Index the given UserScrubGeoEvent.\n   * @param userScrubGeoEvent\n   */\n  public void indexUserScrubGeoEvent(UserScrubGeoEvent userScrubGeoEvent) {\n    userScrubGeoMap.indexUserScrubGeoEvent(userScrubGeoEvent);\n  }\n\n  /**\n   * Return how many documents this segment manager has indexed in all of its enabled segments.\n   */\n  public long getNumIndexedDocuments() {\n    // Order here doesn't matter, we just want all enabled segments, and allocate\n    // as little as needed.\n    long indexedDocs = 0;\n    for (SegmentInfo segmentInfo : getSegmentInfos(Filter.Enabled, Order.OLD_TO_NEW)) {\n      indexedDocs += segmentInfo.getIndexSegment().getIndexStats().getStatusCount();\n    }\n    return indexedDocs;\n  }\n\n  /**\n   * Return how many partial updates this segment manager has applied\n   * in all of its enabled segments.\n   */\n  public long getNumPartialUpdates() {\n    long partialUpdates = 0;\n    for (SegmentInfo segmentInfo : getSegmentInfos(Filter.Enabled, Order.OLD_TO_NEW)) {\n      partialUpdates += segmentInfo.getIndexSegment().getIndexStats().getPartialUpdateCount();\n    }\n    return partialUpdates;\n  }\n\n  /**\n   * Returns the segment info for the segment containing the given tweet ID.\n   */\n  public SegmentInfo getSegmentInfoForID(long tweetID) {\n    ISegmentWriter segmentWriter = getSegmentWriterForID(tweetID);\n    return segmentWriter == null ? null : segmentWriter.getSegmentInfo();\n  }\n\n  /**\n   * Returns the segment writer for the segment containing the given tweet ID.\n   */\n  @Nullable\n  public ISegmentWriter getSegmentWriterForID(long tweetID) {\n    Map.Entry<Long, ISegmentWriter> entry = segmentWriters.floorEntry(tweetID);\n    return entry == null ? null : entry.getValue();\n  }\n\n  /**\n   * Remove old segments until we have less than or equal to the number of max enabled segments.\n   */\n  public void removeExcessSegments() {\n    int removedSegmentCount = 0;\n    while (segmentWriters.size() > getMaxEnabledSegments()) {\n      long timesliceID = getOldestEnabledTimeSliceID();\n      disableSegment(timesliceID);\n      removeSegmentInfo(timesliceID);\n      removedSegmentCount += 1;\n    }\n    LOG.info(\"Segment manager removed {} excess segments\", removedSegmentCount);\n  }\n\n  /**\n   * Returns total index size on disk across all enabled segments in this segment manager.\n   */\n  private long getTotalSegmentSizeOnDisk() {\n    long totalIndexSize = 0;\n    for (SegmentInfo segmentInfo : getSegmentInfos(Filter.Enabled, Order.OLD_TO_NEW)) {\n      totalIndexSize += segmentInfo.getIndexSegment().getIndexStats().getIndexSizeOnDiskInBytes();\n    }\n    return totalIndexSize;\n  }\n\n  @VisibleForTesting\n  ISegmentWriter getSegmentWriterWithoutCreationForTests(long timesliceID) {\n    return segmentWriters.get(timesliceID);\n  }\n\n  @VisibleForTesting\n  ArrayList<Long> getTimeSliceIdsForTests() {\n    return new ArrayList<Long>(segmentWriters.keySet());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SegmentOptimizer.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.earlybird.EarlybirdStatus;\n\npublic final class SegmentOptimizer {\n  private static final Logger LOG = LoggerFactory.getLogger(SegmentOptimizer.class);\n\n  private static final String OPTIMIZING_SEGMENT_EVENT_PATTERN = \"optimizing segment %s\";\n  private static final String OPTIMIZING_SEGMENT_GAUGE_PATTERN = \"optimizing_segment_%s\";\n\n  private SegmentOptimizer() {\n  }\n\n  /**\n   * Optimize a segment. Returns whether optimization was successful.\n   */\n  public static boolean optimize(SegmentInfo segmentInfo) {\n    try {\n      return optimizeThrowing(segmentInfo);\n    } catch (Exception e) {\n      // This is a bad situation, as earlybird can't run with too many un-optimized\n      // segments in memory.\n      LOG.error(\"Exception while optimizing segment \" + segmentInfo.getSegmentName() + \": \", e);\n      segmentInfo.setFailedOptimize();\n      return false;\n    }\n  }\n\n  public static boolean needsOptimization(SegmentInfo segmentInfo) {\n    return segmentInfo.isComplete() && !segmentInfo.isOptimized()\n        && !segmentInfo.isFailedOptimize() && !segmentInfo.isIndexing();\n  }\n\n  private static boolean optimizeThrowing(SegmentInfo segmentInfo) throws IOException {\n    if (!needsOptimization(segmentInfo)) {\n      return false;\n    }\n\n    String gaugeName =\n        String.format(OPTIMIZING_SEGMENT_GAUGE_PATTERN, segmentInfo.getSegmentName());\n    SearchIndexingMetricSet.StartupMetric metric =\n        new SearchIndexingMetricSet.StartupMetric(gaugeName);\n\n    String eventName =\n        String.format(OPTIMIZING_SEGMENT_EVENT_PATTERN, segmentInfo.getSegmentName());\n    EarlybirdStatus.beginEvent(eventName, metric);\n    try {\n      segmentInfo.getIndexSegment().optimizeIndexes();\n    } finally {\n      EarlybirdStatus.endEvent(eventName, metric);\n    }\n\n    return true;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SegmentSyncConfig.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.Date;\nimport java.util.Optional;\nimport java.util.concurrent.TimeUnit;\n\nimport com.twitter.search.common.database.DatabaseConfig;\nimport com.twitter.search.common.metrics.SearchCustomGauge;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.partitioning.base.Segment;\nimport com.twitter.search.common.schema.earlybird.FlushVersion;\nimport com.twitter.search.common.util.io.flushable.PersistentFile;\nimport com.twitter.search.earlybird.archive.ArchiveSegment;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.util.ScrubGenUtil;\nimport com.twitter.util.TwitterDateFormat;\n\n/**\n * Encapsulates config information related to reading and writing segments to local filesystem or\n * HDFS.\n */\npublic class SegmentSyncConfig {\n  public static final String LUCENE_DIR_PREFIX = \"lucene_\";\n\n  private final Optional<String> scrubGen;\n\n  public SegmentSyncConfig(Optional<String> scrubGen) {\n    this.scrubGen = scrubGen;\n    String scrubGenStat = scrubGen.orElse(\"unset\");\n    SearchLongGauge.export(\"scrub_gen_\" + scrubGenStat).set(1);\n    if (scrubGen.isPresent()) {\n      // Export a stat for the number of days between the scrub gen date and now\n      SearchCustomGauge.export(\"scrub_gen_age_in_days\", () -> {\n        long scrubGenMillis = ScrubGenUtil.parseScrubGenToDate(scrubGen.get()).getTime();\n        return TimeUnit.MILLISECONDS.toDays(System.currentTimeMillis() - scrubGenMillis);\n      });\n    }\n  }\n\n  /**\n   * Returns the file extension to be used for the current flush version.\n   */\n  public String getVersionFileExtension() {\n    return FlushVersion.CURRENT_FLUSH_VERSION.getVersionFileExtension();\n  }\n\n  /**\n   * Returns the threshold for how large a segment's status count must be at load time to be\n   * considered valid.\n   */\n  public int getMinSegmentStatusCountThreshold() {\n    double minSegmentTweetCountProportionThreshold =\n        EarlybirdConfig.getDouble(\"min_segment_tweet_count_percentage_threshold\", 0) / 100;\n    return (int) (EarlybirdConfig.getMaxSegmentSize() * minSegmentTweetCountProportionThreshold);\n  }\n\n  /**\n   * Determines if this earlybird is allowed to flush segments to HDFS.\n   */\n  public boolean isFlushToHdfsEnabled() {\n    return EarlybirdProperty.SEGMENT_FLUSH_TO_HDFS_ENABLED.get(false)\n        // Flush to HDFS is always disabled if FlushVersion is not official.\n        && FlushVersion.CURRENT_FLUSH_VERSION.isOfficial();\n  }\n\n  /**\n   * Determines if this earlybird is allowed to load segments from HDFS.\n   */\n  public boolean isSegmentLoadFromHdfsEnabled() {\n    return EarlybirdProperty.SEGMENT_LOAD_FROM_HDFS_ENABLED.get(false);\n  }\n\n  /**\n   * Determines if this earlybird is allowed to delete flushed segments.\n   */\n  public boolean isDeleteFlushedSegmentsEnabled() {\n    return EarlybirdConfig.getBool(\"segment_dropper_delete_flushed\", true);\n  }\n\n  /**\n   * Returns the root of the segment directory on the local disk.\n   */\n  public String getLocalSegmentSyncRootDir() {\n    return EarlybirdConfig.getString(\"segment_sync_dir\", \"partitions\")\n        + getScrubGenFlushDirSuffix();\n  }\n\n  /**\n   * Returns the root of the segment directory on HDFS.\n   */\n  public String getHdfsSegmentSyncRootDir() {\n    return EarlybirdProperty.HDFS_SEGMENT_SYNC_DIR.get(\"partitions\")\n        + getScrubGenFlushDirSuffix();\n  }\n\n  /**\n   * Returns the HDFS root directory where all segments should be uploaded.\n   */\n  public String getHdfsSegmentUploadRootDir() {\n    String hdfsSegmentUploadDir = EarlybirdProperty.HDFS_SEGMENT_UPLOAD_DIR.get(null);\n    return hdfsSegmentUploadDir != null\n        ? hdfsSegmentUploadDir + getScrubGenFlushDirSuffix()\n        : getHdfsSegmentSyncRootDir();\n  }\n\n  /**\n   * Returns the ZooKeeper path used for segment sync'ing.\n   */\n  public String getZooKeeperSyncFullPath() {\n    return EarlybirdProperty.ZK_APP_ROOT.get() + \"/\"\n        + EarlybirdConfig.getString(\"segment_flush_sync_relative_path\", \"segment_flush_sync\");\n  }\n\n  /**\n   * Returns the list of directories that should be persisted for this segment.\n   */\n  public Collection<String> getPersistentFileNames(SegmentInfo segment) {\n    return Collections.singleton(segment.getSegmentName());\n  }\n\n  /**\n   * Returns the list of all files that should be sync'ed for this segment.\n   */\n  public Collection<String> getAllSyncFileNames(SegmentInfo segment) {\n    Collection<String> allFileNames = PersistentFile.getAllFileNames(segment.getSegmentName());\n    if (segment.getEarlybirdIndexConfig().isIndexStoredOnDisk()) {\n      allFileNames = new ArrayList<>(allFileNames);\n      // Just the file name, not the full path\n      allFileNames.add(getLocalLuceneSyncDirFileName(segment.getSegment()));\n    }\n    return allFileNames;\n  }\n\n  /**\n   * Returns the local sync directory for the given segment.\n   */\n  public String getLocalSyncDirName(Segment segment) {\n    return getLocalSegmentSyncRootDir() + \"/\" + segment.getSegmentName()\n        + getVersionFileExtension();\n  }\n\n  /**\n   * Returns the local Lucene directory for the given segment.\n   */\n  public String getLocalLuceneSyncDirName(Segment segment) {\n    return getLocalSyncDirName(segment) + \"/\" + getLocalLuceneSyncDirFileName(segment);\n  }\n\n  /**\n   * Returns the name (not the path) of the Lucene directory for the given segment.\n   */\n  private String getLocalLuceneSyncDirFileName(Segment segment) {\n    if (segment instanceof ArchiveSegment) {\n      Date endDate = ((ArchiveSegment) segment).getDataEndDate();\n      String endDateString = TwitterDateFormat.apply(\"yyyyMMdd\").format(endDate);\n      return LUCENE_DIR_PREFIX + endDateString;\n    } else {\n      return LUCENE_DIR_PREFIX + \"realtime\";\n    }\n  }\n\n  /**\n   * Returns the HDFS sync directory for the given segment.\n   */\n  public String getHdfsSyncDirNamePrefix(Segment segment) {\n    return getHdfsSegmentSyncRootDir() + \"/\" + segment.getSegmentName()\n        + getVersionFileExtension() + \"*\";\n  }\n\n  /**\n   * Returns the prefix of the HDFS directory where the files for this segment should be uploaded.\n   */\n  public String getHdfsUploadDirNamePrefix(Segment segment) {\n    return getHdfsSegmentUploadRootDir() + \"/\" + segment.getSegmentName()\n        + getVersionFileExtension() + \"*\";\n  }\n\n  /**\n   * Returns the HDFS directory where the files for this segment should be uploaded.\n   */\n  public String getHdfsFlushDirName(Segment segment) {\n    return getHdfsSegmentUploadRootDir() + \"/\" + segment.getSegmentName()\n        + getVersionFileExtension() + \"_\" + DatabaseConfig.getLocalHostname();\n  }\n\n  /**\n   * Returns a temp HDFS directory to be used for this segment.\n   */\n  public String getHdfsTempFlushDirName(Segment segment) {\n    return getHdfsSegmentUploadRootDir() + \"/temp_\"\n        + DatabaseConfig.getLocalHostname() + \"_\" + segment.getSegmentName()\n        + getVersionFileExtension();\n  }\n\n  /**\n   * Concatenates the name of this segment with the flush version extension.\n   */\n  public String getVersionedName(Segment segment) {\n    return segment.getSegmentName() + getVersionFileExtension();\n  }\n\n  private String getScrubGenFlushDirSuffix() {\n    return scrubGen\n        .map(s -> \"/scrubbed/\" + s)\n        .orElse(\"\");\n  }\n\n  /**\n   * Returns the scrub gen set for this earlybird.\n   */\n  public Optional<String> getScrubGen() {\n    return scrubGen;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SegmentSyncInfo.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport com.twitter.search.common.partitioning.base.Segment;\n\n/**\n * Representation for segment sync state, the local and hdfs file locations, as well as the\n * current in-memory sync states maintained by earlybirds.\n */\npublic class SegmentSyncInfo {\n  // Is this segment loaded from disk?\n  private volatile boolean loaded = false;\n  // Has this segment been flushed to disk, and uploaded to HDFS if uploading is enabled?\n  private volatile boolean flushed = false;\n  // Time when the segment was flushed to local disk\n  private volatile long flushTimeMillis = 0;\n\n  private final Segment segment;\n  private final SegmentSyncConfig syncConfig;\n  private final String localSyncDir;\n  private final String hdfsFlushDir;\n  private final String hdfsSyncDirPrefix;\n  private final String hdfsUploadDirPrefix;\n  private final String hdfsTempFlushDir;\n\n  @VisibleForTesting\n  public SegmentSyncInfo(SegmentSyncConfig syncConfig, Segment segment) {\n    this.segment = segment;\n    this.syncConfig = syncConfig;\n    this.localSyncDir = syncConfig.getLocalSyncDirName(segment);\n    this.hdfsSyncDirPrefix = syncConfig.getHdfsSyncDirNamePrefix(segment);\n    this.hdfsUploadDirPrefix = syncConfig.getHdfsUploadDirNamePrefix(segment);\n    this.hdfsFlushDir = syncConfig.getHdfsFlushDirName(segment);\n    this.hdfsTempFlushDir = syncConfig.getHdfsTempFlushDirName(segment);\n  }\n\n  public boolean isLoaded() {\n    return loaded;\n  }\n\n  public boolean isFlushed() {\n    return flushed;\n  }\n\n  public long getFlushTimeMillis() {\n    return flushTimeMillis;\n  }\n\n  public String getLocalSyncDir() {\n    return localSyncDir;\n  }\n\n  public SegmentSyncConfig getSegmentSyncConfig() {\n    return syncConfig;\n  }\n\n  public String getLocalLuceneSyncDir() {\n    // For archive search this name depends on the end date of the segment, which can change,\n    // so we cannot pre-compute this in the constructor.\n    // This should only be used in the on-disk archive.\n    return syncConfig.getLocalLuceneSyncDirName(segment);\n  }\n\n  public String getHdfsFlushDir() {\n    return hdfsFlushDir;\n  }\n\n  public String getHdfsSyncDirPrefix() {\n    return hdfsSyncDirPrefix;\n  }\n\n  public String getHdfsUploadDirPrefix() {\n    return hdfsUploadDirPrefix;\n  }\n\n  public String getHdfsTempFlushDir() {\n    return hdfsTempFlushDir;\n  }\n\n  public void setLoaded(boolean isLoaded) {\n    this.loaded = isLoaded;\n  }\n\n  /**\n   * Stores the flushing state for this segment.\n   */\n  public void setFlushed(boolean isFlushed) {\n    if (isFlushed) {\n      this.flushTimeMillis = System.currentTimeMillis();\n    }\n    this.flushed = isFlushed;\n  }\n\n  /**\n   * Adds debug information about the loaded and flushed status of this segment to the given\n   * StringBuilder.\n   */\n  public void addDebugInfo(StringBuilder builder) {\n    builder.append(\"[\");\n    int startLength = builder.length();\n    if (loaded) {\n      builder.append(\"loaded, \");\n    }\n    if (flushed) {\n      builder.append(\"flushed, \");\n    }\n    if (startLength < builder.length()) {\n      builder.setLength(builder.length() - 2);\n    }\n    builder.append(\"]\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SegmentVulture.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.util.List;\nimport java.util.Set;\nimport java.util.SortedSet;\nimport java.util.TreeSet;\n\nimport javax.annotation.Nonnull;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Sets;\n\nimport org.apache.commons.io.FileUtils;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.partitioning.base.Segment;\nimport com.twitter.search.common.schema.earlybird.FlushVersion;\nimport com.twitter.search.earlybird.archive.ArchiveSearchPartitionManager;\nimport com.twitter.search.earlybird.archive.ArchiveTimeSlicer;\nimport com.twitter.search.earlybird.archive.ArchiveTimeSlicer.ArchiveTimeSlice;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.factory.EarlybirdIndexConfigUtil;\n\n/**\n * This class removes older flush version segments.\n * Considering that we almost never increase status flush versions, old statuses are not cleaned up\n * automatically.\n */\npublic final class SegmentVulture {\n  private static final Logger LOG = LoggerFactory.getLogger(SegmentVulture.class);\n  @VisibleForTesting // Not final for testing.\n  protected static int numIndexFlushVersionsToKeep =\n      EarlybirdConfig.getInt(\"number_of_flush_versions_to_keep\", 2);\n\n  private SegmentVulture() {\n    // this never gets called\n  }\n\n  /**\n   * Delete old build generations, keep currentGeneration.\n   */\n  @VisibleForTesting\n  static void removeOldBuildGenerations(String rootDirPath, String currentGeneration) {\n    File rootDir = new File(rootDirPath);\n\n    if (!rootDir.exists() || !rootDir.isDirectory()) {\n      LOG.error(\"Root directory is invalid: \" + rootDirPath);\n      return;\n    }\n\n    File[] buildGenerations = rootDir.listFiles();\n\n    for (File generation : buildGenerations) {\n      if (generation.getName().equals(currentGeneration)) {\n        LOG.info(\"Skipping current generation: \" + generation.getAbsoluteFile());\n        continue;\n      }\n\n      try {\n        FileUtils.deleteDirectory(generation);\n        LOG.info(\"Deleted old build generation: \" + generation.getAbsolutePath());\n      } catch (IOException e) {\n        LOG.error(\"Failed to delete old build generation at: \" + generation.getAbsolutePath(), e);\n      }\n    }\n    LOG.info(\"Successfully deleted all old generations\");\n  }\n\n  /**\n   * Delete all the timeslice data outside the serving range.\n   */\n  @VisibleForTesting\n  static void removeArchiveTimesliceOutsideServingRange(PartitionConfig partitionConfig,\n      ArchiveTimeSlicer timeSlicer, SegmentSyncConfig segmentSyncConfig) {\n    try {\n      long servingStartTimesliceId = Long.MAX_VALUE;\n      long servingEndTimesliceId = 0;\n      int partitionID = partitionConfig.getIndexingHashPartitionID();\n      List<ArchiveTimeSlice> timeSliceList = timeSlicer.getTimeSlicesInTierRange();\n      for (ArchiveTimeSlice timeSlice : timeSliceList) {\n        if (timeSlice.getMinStatusID(partitionID) < servingStartTimesliceId) {\n          servingStartTimesliceId = timeSlice.getMinStatusID(partitionID);\n        }\n        if (timeSlice.getMaxStatusID(partitionID) > servingEndTimesliceId) {\n          servingEndTimesliceId = timeSlice.getMaxStatusID(partitionID);\n        }\n      }\n      LOG.info(\"Got the serving range: [\" + servingStartTimesliceId + \", \"\n          + servingEndTimesliceId + \"], \" + \"[\" + partitionConfig.getTierStartDate() + \", \"\n          + partitionConfig.getTierEndDate() + \") for tier: \" + partitionConfig.getTierName());\n\n      // The tier configuration does not have valid serving range: do not do anything.\n      if (servingEndTimesliceId <= servingStartTimesliceId) {\n        LOG.error(\"Invalid serving range [\" + partitionConfig.getTierStartDate() + \", \"\n            + partitionConfig.getTierEndDate() + \"] for tier: \" + partitionConfig.getTierName());\n        return;\n      }\n\n      int numDeleted = 0;\n      File[] segments = getSegmentsOnRootDir(segmentSyncConfig);\n      for (File segment : segments) {\n        String segmentName = SegmentInfo.getSegmentNameFromFlushedDir(segment.getName());\n        if (segmentName == null) {\n          LOG.error(\"Invalid directory for segments: \" + segment.getAbsolutePath());\n          continue;\n        }\n        long timesliceId = Segment.getTimeSliceIdFromName(segmentName);\n        if (timesliceId < 0) {\n          LOG.error(\"Unknown dir/file found: \" + segment.getAbsolutePath());\n          continue;\n        }\n\n        if (timesliceId < servingStartTimesliceId || timesliceId > servingEndTimesliceId) {\n          LOG.info(segment.getAbsolutePath() + \" will be deleted for outside serving Range[\"\n              + partitionConfig.getTierStartDate() + \", \" + partitionConfig.getTierEndDate() + \")\");\n          if (deleteSegment(segment)) {\n            numDeleted++;\n          }\n        }\n      }\n      LOG.info(\"Deleted \" + numDeleted + \" segments out of \" + segments.length + \" segments\");\n    } catch (IOException e) {\n      LOG.error(\"Can not timeslice based on the document data: \", e);\n      throw new RuntimeException(e);\n    }\n  }\n\n  /**\n   * Deleted segments from other partitions. When boxes are moved between\n   * partitions, segments from other partitions may stay, we will have to\n   * delete them.\n   */\n  @VisibleForTesting\n  static void removeIndexesFromOtherPartitions(int myPartition, int numPartitions,\n        SegmentSyncConfig segmentSyncConfig) {\n    File[] segments = getSegmentsOnRootDir(segmentSyncConfig);\n    int numDeleted = 0;\n    for (File segment : segments) {\n      int segmentNumPartitions = Segment.numPartitionsFromName(segment.getName());\n      int segmentPartition = Segment.getPartitionFromName(segment.getName());\n\n      if (segmentNumPartitions < 0 || segmentPartition < 0) { // Not a segment file, ignoring\n        LOG.info(\"Unknown dir/file found: \" + segment.getAbsolutePath());\n        continue;\n      }\n\n      if (segmentNumPartitions != numPartitions || segmentPartition != myPartition) {\n        if (deleteSegment(segment)) {\n          numDeleted++;\n        }\n      }\n    }\n    LOG.info(\"Deleted \" + numDeleted + \" segments out of \" + segments.length + \" segments\");\n  }\n\n  /**\n   * Delete flushed segments of older flush versions.\n   */\n  @VisibleForTesting\n  static void removeOldFlushVersionIndexes(int currentFlushVersion,\n                                           SegmentSyncConfig segmentSyncConfig) {\n    SortedSet<Integer> indexFlushVersions =\n        listFlushVersions(segmentSyncConfig, currentFlushVersion);\n\n    if (indexFlushVersions == null\n        || indexFlushVersions.size() <= numIndexFlushVersionsToKeep) {\n      return;\n    }\n\n    Set<String> suffixesToKeep = Sets.newHashSetWithExpectedSize(numIndexFlushVersionsToKeep);\n    int flushVersionsToKeep = numIndexFlushVersionsToKeep;\n    while (flushVersionsToKeep > 0 && !indexFlushVersions.isEmpty()) {\n      Integer oldestFlushVersion = indexFlushVersions.last();\n      String flushFileExtension = FlushVersion.getVersionFileExtension(oldestFlushVersion);\n      if (flushFileExtension != null) {\n        suffixesToKeep.add(flushFileExtension);\n        flushVersionsToKeep--;\n      } else {\n        LOG.warn(\"Found unknown flush versions: \" + oldestFlushVersion\n            + \" Segments with this flush version will be deleted to recover disk space.\");\n      }\n      indexFlushVersions.remove(oldestFlushVersion);\n    }\n\n    String segmentSyncRootDir = segmentSyncConfig.getLocalSegmentSyncRootDir();\n    File dir = new File(segmentSyncRootDir);\n    File[] segments = dir.listFiles();\n\n    for (File segment : segments) {\n      boolean keepSegment = false;\n      for (String suffix : suffixesToKeep) {\n        if (segment.getName().endsWith(suffix)) {\n          keepSegment = true;\n          break;\n        }\n      }\n      if (!keepSegment) {\n        try {\n          FileUtils.deleteDirectory(segment);\n          LOG.info(\"Deleted old flushed segment: \" + segment.getAbsolutePath());\n        } catch (IOException e) {\n          LOG.error(\"Failed to delete old flushed segment.\", e);\n        }\n      }\n    }\n  }\n\n  private static File[] getSegmentsOnRootDir(SegmentSyncConfig segmentSyncConfig) {\n    String segmentSyncRootDir = segmentSyncConfig.getLocalSegmentSyncRootDir();\n    File dir = new File(segmentSyncRootDir);\n    File[] segments = dir.listFiles();\n    if (segments == null) {\n      return new File[0];\n    } else {\n      return segments;\n    }\n  }\n\n  private static boolean deleteSegment(File segment) {\n    try {\n      FileUtils.deleteDirectory(segment);\n      LOG.info(\"Deleted segment from other partition: \" + segment.getAbsolutePath());\n      return true;\n    } catch (IOException e) {\n      LOG.error(\"Failed to delete segment from other partition.\", e);\n      return false;\n    }\n  }\n\n  // Returns FlushVersions found on disk.\n  // Current FlushVersion is always added into the list, even if segments are not found on disk,\n  // because they may not have appeared yet.\n  @Nonnull\n  @VisibleForTesting\n  static SortedSet<Integer> listFlushVersions(SegmentSyncConfig sync, int currentFlushVersion) {\n    TreeSet<Integer> flushVersions = Sets.newTreeSet();\n\n    // Always add current flush version.\n    // It is possible that on startup when this is run, the current flush version\n    // segments have not appeared yet.\n    flushVersions.add(currentFlushVersion);\n\n    String segmentSyncRootDir = sync.getLocalSegmentSyncRootDir();\n    File dir = new File(segmentSyncRootDir);\n    if (!dir.exists()) {\n      LOG.info(\"segmentSyncRootDir [\" + segmentSyncRootDir\n          + \"] does not exist\");\n      return flushVersions;\n    }\n    if (!dir.isDirectory()) {\n      LOG.error(\"segmentSyncRootDir [\" + segmentSyncRootDir\n          + \"] does not point to a directory\");\n      return flushVersions;\n    }\n    if (!dir.canRead()) {\n      LOG.error(\"No permission to read from segmentSyncRootDir [\"\n          + segmentSyncRootDir + \"]\");\n      return flushVersions;\n    }\n    if (!dir.canWrite()) {\n      LOG.error(\"No permission to write to segmentSyncRootDir [\"\n          + segmentSyncRootDir + \"]\");\n      return flushVersions;\n    }\n\n    File[] segments = dir.listFiles();\n    for (File segment : segments) {\n      String name = segment.getName();\n      if (!name.contains(FlushVersion.DELIMITER)) {\n        // This is a not a segment with a FlushVersion, skip.\n        LOG.info(\"Found segment directory without a flush version: \" + name);\n        continue;\n      }\n      String[] nameSplits = name.split(FlushVersion.DELIMITER);\n      if (nameSplits.length != 2) {\n        LOG.warn(\"Found segment with bad name: \" + segment.getAbsolutePath());\n        continue;\n      }\n\n      // Second half contains flush version\n      try {\n        int flushVersion = Integer.parseInt(nameSplits[1]);\n        flushVersions.add(flushVersion);\n      } catch (NumberFormatException e) {\n        LOG.warn(\"Bad flush version number in segment name: \" + segment.getAbsolutePath());\n      }\n    }\n    return flushVersions;\n  }\n\n  /**\n   * Removes old segments in the current build gen.\n   */\n  @VisibleForTesting\n  static void removeOldSegments(SegmentSyncConfig sync) {\n    if (!sync.getScrubGen().isPresent()) {\n      return;\n    }\n\n    File currentScrubGenSegmentDir = new File(sync.getLocalSegmentSyncRootDir());\n\n    // The unscrubbed segment root directory, used for rebuilds and for segments created before\n    // we introduced scrub gens. The getLocalSegmentSyncRootDir should be something like:\n    // $unscrubbedSegmentDir/scrubbed/$scrub_gen/,\n    // get unscrubbedSegmentDir from string name here in case scrubbed dir does not exist yet\n    File unscrubbedSegmentDir = new File(sync.getLocalSegmentSyncRootDir().split(\"scrubbed\")[0]);\n    if (!unscrubbedSegmentDir.exists()) {\n      // For a new host that swapped in, it might not have flushed_segment dir yet.\n      // return directly in that case.\n      LOG.info(unscrubbedSegmentDir.getAbsoluteFile() + \"does not exist, nothing to remove.\");\n      return;\n    }\n    Preconditions.checkArgument(unscrubbedSegmentDir.exists());\n    for (File file : unscrubbedSegmentDir.listFiles()) {\n      if (file.getName().matches(\"scrubbed\")) {\n        continue;\n      }\n      try {\n        LOG.info(\"Deleting old unscrubbed segment: \" + file.getAbsolutePath());\n        FileUtils.deleteDirectory(file);\n      } catch (IOException e) {\n        LOG.error(\"Failed to delete directory: \" + file.getPath(), e);\n      }\n    }\n\n    // Delete all segments from previous scrub generations.\n    File allScrubbedSegmentsDir = currentScrubGenSegmentDir.getParentFile();\n    if (allScrubbedSegmentsDir.exists()) {\n      for (File file : allScrubbedSegmentsDir.listFiles()) {\n        if (file.getPath().equals(currentScrubGenSegmentDir.getPath())) {\n          continue;\n        }\n        try {\n          LOG.info(\"Deleting old scrubbed segment: \" + file.getAbsolutePath());\n          FileUtils.deleteDirectory(file);\n        } catch (IOException e) {\n          LOG.error(\"Failed to delete directory: \" + file.getPath(), e);\n        }\n      }\n    }\n  }\n\n  /**\n   * Removes the data for all unused segments from the local disk. This includes:\n   *  - data for old segments\n   *  - data for segments belonging to another partition\n   *  - data for segments belonging to a different flush version.\n   */\n  public static void removeUnusedSegments(\n      PartitionManager partitionManager,\n      PartitionConfig partitionConfig,\n      int schemaMajorVersion,\n      SegmentSyncConfig segmentSyncConfig) {\n\n    if (EarlybirdIndexConfigUtil.isArchiveSearch()) {\n      removeOldBuildGenerations(\n          EarlybirdConfig.getString(\"root_dir\"),\n          EarlybirdConfig.getString(\"offline_segment_build_gen\")\n      );\n      removeOldSegments(segmentSyncConfig);\n\n      Preconditions.checkState(partitionManager instanceof ArchiveSearchPartitionManager);\n      removeArchiveTimesliceOutsideServingRange(\n          partitionConfig,\n          ((ArchiveSearchPartitionManager) partitionManager).getTimeSlicer(), segmentSyncConfig);\n    }\n\n    // Remove segments from other partitions\n    removeIndexesFromOtherPartitions(\n        partitionConfig.getIndexingHashPartitionID(),\n        partitionConfig.getNumPartitions(), segmentSyncConfig);\n\n    // Remove old flushed segments\n    removeOldFlushVersionIndexes(schemaMajorVersion, segmentSyncConfig);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SegmentWarmer.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\n\npublic class SegmentWarmer {\n  private static final Logger LOG = LoggerFactory.getLogger(SegmentWarmer.class);\n\n  private final CriticalExceptionHandler criticalExceptionHandler;\n\n  public SegmentWarmer(CriticalExceptionHandler criticalExceptionHandler) {\n    this.criticalExceptionHandler = criticalExceptionHandler;\n  }\n\n  private boolean shouldWarmSegment(SegmentInfo segmentInfo) {\n    return segmentInfo.isEnabled()\n        && segmentInfo.isComplete()\n        && segmentInfo.isOptimized()\n        && !segmentInfo.isIndexing();\n  }\n\n  /**\n   * Warms a segment if it is ready to be warmed. Only has an affect on Archive Lucene segments.\n   */\n  public boolean warmSegmentIfNecessary(SegmentInfo segmentInfo) {\n    if (!shouldWarmSegment(segmentInfo)) {\n      return false;\n    }\n    try {\n      segmentInfo.getIndexSegment().warmSegment();\n      return true;\n    } catch (IOException e) {\n      // This is a bad situation, as earlybird can't search a segment that hasn't been warmed up\n      // So we delete the bad segment, and restart the earlybird if it's in starting phrase,\n      // otherwise alert.\n      LOG.error(\"Failed to warmup segment \" + segmentInfo.getSegmentName()\n          + \". Will destroy local unreadable segment.\", e);\n      segmentInfo.deleteLocalIndexedSegmentDirectoryImmediately();\n\n      criticalExceptionHandler.handle(this, e);\n\n      return false;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SegmentWriter.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\nimport java.util.EnumMap;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.atomic.AtomicLong;\n\nimport com.google.common.collect.HashBasedTable;\nimport com.google.common.collect.Table;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.metrics.Percentile;\nimport com.twitter.search.common.metrics.PercentileUtil;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEventType;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.document.DocumentFactory;\nimport com.twitter.search.earlybird.document.TweetDocument;\nimport com.twitter.search.earlybird.index.EarlybirdSegment;\nimport com.twitter.util.Time;\n\npublic class SegmentWriter implements ISegmentWriter {\n\n  // helper, used for collecting stats\n  enum FailureReason {\n    FAILED_INSERT,\n    FAILED_FOR_TWEET_IN_INDEX,\n    FAILED_FOR_COMPLETE_SEGMENT\n  }\n\n  private static final String STAT_PREFIX = \"segment_writer_\";\n  private static final String EVENT_COUNTER = STAT_PREFIX + \"%s_%s_segment_%s\";\n  private static final String EVENT_COUNTER_ALL_SEGMENTS = STAT_PREFIX + \"%s_%s_all_segments\";\n  private static final String EVENT_TIMERS = STAT_PREFIX + \"%s_timing\";\n  private static final String DROPPED_UPDATES_FOR_DISABLED_SEGMENTS =\n      STAT_PREFIX + \"%s_dropped_updates_for_disabled_segments\";\n  private static final String INDEXING_LATENCY =\n      STAT_PREFIX + \"%s_indexing_latency_ms\";\n\n  private final byte penguinVersion;\n  private final DocumentFactory<ThriftIndexingEvent> updateFactory;\n  private final DocumentFactory<ThriftIndexingEvent> documentFactory;\n  private final SearchRateCounter missingPenguinVersion;\n  private final EarlybirdSegment earlybirdSegment;\n  private final SegmentInfo segmentInfo;\n  // Stores per segment counters for each (indexing event type, result) pair\n  // Example stat name\n  // \"segment_writer_partial_update_success_segment_twttr_search_test_start_%d_p_0_of_1\"\n  private final Table<ThriftIndexingEventType, Result, SearchRateCounter> statsForUpdateType =\n      HashBasedTable.create();\n  // Stores aggregated counters for each (indexing event type, result) pair across all segments\n  // Example stat name\n  // \"segment_writer_partial_update_success_all_segments\"\n  private final Table<ThriftIndexingEventType, Result, SearchRateCounter>\n      aggregateStatsForUpdateType = HashBasedTable.create();\n  // Stores per segment counters for each (indexing event type, non-retryable failure reason) pair\n  // Example stat name\n  // \"segment_writer_partial_update_failed_for_tweet_in_index_segment_twttr_search_t_%d_p_0_of_1\"\n  private final Table<ThriftIndexingEventType, FailureReason, SearchRateCounter>\n      failureStatsForUpdateType = HashBasedTable.create();\n  // Stores aggregated counters for each (indexing event type, non-retryable failure reason) pair\n  // Example stat name\n  // \"segment_writer_partial_update_failed_for_tweet_in_index_all_segments\"\n  private final Table<ThriftIndexingEventType, FailureReason, SearchRateCounter>\n      aggregateFailureStatsForUpdateType = HashBasedTable.create();\n  private final EnumMap<ThriftIndexingEventType, SearchTimerStats> eventTimers =\n      new EnumMap<>(ThriftIndexingEventType.class);\n  private final EnumMap<ThriftIndexingEventType, SearchRateCounter>\n    droppedUpdatesForDisabledSegments = new EnumMap<>(ThriftIndexingEventType.class);\n  // We pass this stat from the SearchIndexingMetricSet so that we can share the atomic longs\n  // between all SegmentWriters and export the largest freshness value across all segments.\n  private final EnumMap<ThriftIndexingEventType, AtomicLong> updateFreshness;\n  private final EnumMap<ThriftIndexingEventType, Percentile<Long>> indexingLatency =\n      new EnumMap<>(ThriftIndexingEventType.class);\n\n  public SegmentWriter(\n      SegmentInfo segmentInfo,\n      EnumMap<ThriftIndexingEventType, AtomicLong> updateFreshness\n  ) {\n    this.segmentInfo = segmentInfo;\n    this.updateFreshness = updateFreshness;\n    this.earlybirdSegment = segmentInfo.getIndexSegment();\n    this.penguinVersion = EarlybirdConfig.getPenguinVersionByte();\n    this.updateFactory = segmentInfo.getEarlybirdIndexConfig().createUpdateFactory();\n    this.documentFactory = segmentInfo.getEarlybirdIndexConfig().createDocumentFactory();\n\n    String segmentName = segmentInfo.getSegmentName();\n    for (ThriftIndexingEventType type : ThriftIndexingEventType.values()) {\n      for (Result result : Result.values()) {\n        String stat = String.format(EVENT_COUNTER, type, result, segmentName).toLowerCase();\n        statsForUpdateType.put(type, result, SearchRateCounter.export(stat));\n\n        String aggregateStat =\n            String.format(EVENT_COUNTER_ALL_SEGMENTS, type, result).toLowerCase();\n        aggregateStatsForUpdateType.put(type, result, SearchRateCounter.export(aggregateStat));\n      }\n\n      for (FailureReason reason : FailureReason.values()) {\n        String stat = String.format(EVENT_COUNTER, type, reason, segmentName).toLowerCase();\n        failureStatsForUpdateType.put(type, reason, SearchRateCounter.export(stat));\n\n        String aggregateStat =\n            String.format(EVENT_COUNTER_ALL_SEGMENTS, type, reason).toLowerCase();\n        aggregateFailureStatsForUpdateType.put(\n            type, reason, SearchRateCounter.export(aggregateStat));\n      }\n\n      eventTimers.put(type, SearchTimerStats.export(\n          String.format(EVENT_TIMERS, type).toLowerCase(),\n          TimeUnit.MICROSECONDS,\n          false));\n      droppedUpdatesForDisabledSegments.put(\n          type,\n          SearchRateCounter.export(\n              String.format(DROPPED_UPDATES_FOR_DISABLED_SEGMENTS, type).toLowerCase()));\n      indexingLatency.put(\n          type,\n           PercentileUtil.createPercentile(\n              String.format(INDEXING_LATENCY, type).toLowerCase()));\n    }\n\n    this.missingPenguinVersion = SearchRateCounter.export(\n        \"documents_without_current_penguin_version_\" + penguinVersion + \"_\" + segmentName);\n  }\n\n  @Override\n  public synchronized Result indexThriftVersionedEvents(ThriftVersionedEvents tve)\n      throws IOException {\n    if (!tve.getVersionedEvents().containsKey(penguinVersion)) {\n      missingPenguinVersion.increment();\n      return Result.FAILURE_NOT_RETRYABLE;\n    }\n\n    ThriftIndexingEvent tie = tve.getVersionedEvents().get(penguinVersion);\n    ThriftIndexingEventType eventType = tie.getEventType();\n\n    if (!segmentInfo.isEnabled()) {\n      droppedUpdatesForDisabledSegments.get(eventType).increment();\n      return Result.SUCCESS;\n    }\n\n    SearchTimerStats timerStats = eventTimers.get(eventType);\n    SearchTimer timer = timerStats.startNewTimer();\n\n    long tweetId = tve.getId();\n    Result result = tryApplyIndexingEvent(tweetId, tie);\n\n    if (result == Result.SUCCESS) {\n      long tweetAgeInMs = SnowflakeIdParser.getTimestampFromTweetId(tweetId);\n\n      AtomicLong freshness = updateFreshness.get(tie.getEventType());\n      // Note that this is racy at startup because we don't do an atomic swap, but it will be\n      // approximately accurate, and this stat doesn't matter until we are current.\n      if (freshness.get() < tweetAgeInMs) {\n        freshness.set(tweetAgeInMs);\n      }\n\n      if (tie.isSetCreateTimeMillis()) {\n        long age = Time.now().inMillis() - tie.getCreateTimeMillis();\n        indexingLatency.get(tie.getEventType()).record(age);\n      }\n    }\n\n    statsForUpdateType.get(eventType, result).increment();\n    aggregateStatsForUpdateType.get(eventType, result).increment();\n    timerStats.stopTimerAndIncrement(timer);\n\n    return result;\n  }\n\n  public SegmentInfo getSegmentInfo() {\n    return segmentInfo;\n  }\n\n  public boolean hasTweet(long tweetId) throws IOException {\n    return earlybirdSegment.hasDocument(tweetId);\n  }\n\n  private Result tryApplyIndexingEvent(long tweetId, ThriftIndexingEvent tie) throws IOException {\n    if (applyIndexingEvent(tie, tweetId)) {\n      return Result.SUCCESS;\n    }\n\n    if (tie.getEventType() == ThriftIndexingEventType.INSERT) {\n      // We don't retry inserts\n      incrementFailureStats(tie, FailureReason.FAILED_INSERT);\n      return Result.FAILURE_NOT_RETRYABLE;\n    }\n\n    if (earlybirdSegment.hasDocument(tweetId)) {\n      // An update fails to be applied for a tweet that is in the index.\n      incrementFailureStats(tie, FailureReason.FAILED_FOR_TWEET_IN_INDEX);\n      return Result.FAILURE_NOT_RETRYABLE;\n    }\n\n    if (segmentInfo.isComplete()) {\n      // An update is directed at a tweet that is not in the segment (hasDocument(tweetId) failed),\n      // and the segment is complete (i.e. there will never be new tweets for this segment).\n      incrementFailureStats(tie, FailureReason.FAILED_FOR_COMPLETE_SEGMENT);\n      return Result.FAILURE_NOT_RETRYABLE;\n    }\n\n    // The tweet may arrive later for this event, so it's possible a later try will succeed\n    return Result.FAILURE_RETRYABLE;\n  }\n\n  private void incrementFailureStats(ThriftIndexingEvent tie, FailureReason failureReason) {\n    failureStatsForUpdateType.get(tie.getEventType(), failureReason).increment();\n    aggregateFailureStatsForUpdateType.get(tie.getEventType(), failureReason).increment();\n  }\n\n  private boolean applyIndexingEvent(ThriftIndexingEvent tie, long tweetId) throws IOException {\n    switch (tie.getEventType()) {\n      case OUT_OF_ORDER_APPEND:\n        return earlybirdSegment.appendOutOfOrder(updateFactory.newDocument(tie), tweetId);\n      case PARTIAL_UPDATE:\n        return earlybirdSegment.applyPartialUpdate(tie);\n      case DELETE:\n        return earlybirdSegment.delete(tweetId);\n      case INSERT:\n        earlybirdSegment.addDocument(buildInsertDocument(tie, tweetId));\n        return true;\n      default:\n        throw new IllegalArgumentException(\"Unexpected update type: \" + tie.getEventType());\n    }\n  }\n\n  private TweetDocument buildInsertDocument(ThriftIndexingEvent tie, long tweetId) {\n    return new TweetDocument(\n        tweetId,\n        segmentInfo.getTimeSliceID(),\n        tie.getCreateTimeMillis(),\n        documentFactory.newDocument(tie));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SimpleSegmentIndexer.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\nimport java.util.concurrent.TimeUnit;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Stopwatch;\n\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.util.io.recordreader.RecordReader;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.document.TweetDocument;\nimport com.twitter.search.earlybird.index.EarlybirdSegment;\n\n/**\n * SimpleSegmentIndex indexes all Tweets for a *complete* segment. It does not index any updates or\n * deletes.\n */\npublic class SimpleSegmentIndexer {\n  private static final Logger LOG = LoggerFactory.getLogger(SimpleSegmentIndexer.class);\n\n  /**\n   * If not null, this segment is appended at the end after indexing finishes.\n   */\n  @Nullable\n  private final SegmentInfo segmentToAppend;\n\n  private final RecordReader<TweetDocument> tweetReader;\n  private final SearchIndexingMetricSet partitionIndexingMetricSet;\n\n  // Segment we are indexing.\n  private EarlybirdSegment indexingSegment;\n\n  // Total number of statuses indexed in this segment.\n  private long segmentSize = 0;\n\n  public SimpleSegmentIndexer(\n      RecordReader<TweetDocument> tweetReader,\n      SearchIndexingMetricSet partitionIndexingMetricSet) {\n    this(tweetReader, partitionIndexingMetricSet, null);\n  }\n\n  public SimpleSegmentIndexer(RecordReader<TweetDocument> tweetReader,\n                              SearchIndexingMetricSet partitionIndexingMetricSet,\n                              @Nullable SegmentInfo segmentToAppend) {\n    this.tweetReader = tweetReader;\n    this.segmentToAppend = segmentToAppend;\n    this.partitionIndexingMetricSet = partitionIndexingMetricSet;\n  }\n\n  private boolean shouldIndexSegment(SegmentInfo segmentInfo) {\n    if (!segmentInfo.isEnabled()) {\n      return false;\n    }\n\n    if (segmentToAppend != null) {\n      return true;\n    }\n\n    return !segmentInfo.isComplete()\n        && !segmentInfo.isIndexing()\n        && !segmentInfo.getSyncInfo().isLoaded();\n  }\n\n  /**\n   * Indexes all tweets for a complete segment.\n   */\n  public boolean indexSegment(SegmentInfo segmentInfo) {\n    LOG.info(\"Indexing segment \" + segmentInfo.getSegmentName());\n    if (!shouldIndexSegment(segmentInfo)) {\n      return false;\n    }\n\n    // If we're starting to index, we're not complete, will become complete if we\n    // were successful here.\n    segmentInfo.setComplete(false);\n\n    try {\n      segmentInfo.setIndexing(true);\n      indexingSegment = segmentInfo.getIndexSegment();\n\n      // if we're updating the segment, then we'll index only the new available days\n      // and then append the lucene index from the old segment\n      // If segmentToAppend is not null, it means we are updating a segment.\n      if (indexingSegment.tryToLoadExistingIndex()) {\n        segmentInfo.getSyncInfo().setLoaded(true);\n        LOG.info(\"Loaded existing index for \" + segmentInfo + \", not indexing.\");\n      } else {\n        indexingLoop();\n        if (segmentToAppend != null) {\n          indexingSegment.append(segmentToAppend.getIndexSegment());\n        }\n      }\n\n      segmentInfo.setIndexing(false);\n      segmentInfo.setComplete(true);\n      segmentInfo.setWasIndexed(true);\n      LOG.info(\"Successfully indexed segment \" + segmentInfo.getSegmentName());\n      return true;\n    } catch (Exception e) {\n      LOG.error(\"Exception while indexing IndexSegment \" + segmentInfo\n          + \" after \" + indexingSegment.getIndexStats().getStatusCount() + \" documents.\", e);\n      partitionIndexingMetricSet.simpleSegmentIndexerExceptionCounter.increment();\n\n      LOG.warn(\"Failed to load a new day into full archive. Cleaning up segment: \"\n          + indexingSegment.getSegmentName());\n\n      // Clean up the lucene dir if it exists. Earlybird will retry loading the new day again later.\n      if (!segmentInfo.deleteLocalIndexedSegmentDirectoryImmediately()) {\n        LOG.error(\"Failed to clean up index segment folder after indexing failures.\");\n      }\n\n      return false;\n    } finally {\n      if (tweetReader != null) {\n        tweetReader.stop();\n      }\n      segmentInfo.setIndexing(false);\n    }\n  }\n\n  // Indexes a document if available.  Returns true if index was updated.\n  protected boolean indexDocument(TweetDocument tweetDocument) throws IOException {\n    if (tweetDocument == null) {\n      return false;\n    }\n\n    SearchTimer timer = partitionIndexingMetricSet.statusStats.startNewTimer();\n    indexingSegment.addDocument(tweetDocument);\n    partitionIndexingMetricSet.statusStats.stopTimerAndIncrement(timer);\n    segmentSize++;\n    return true;\n  }\n\n  /**\n   * Indexes all tweets for this segment, until no more tweets are available.\n   *\n   * @throws InterruptedException If the thread is interrupted while indexing tweets.\n   * @throws IOException If there's a problem reading or indexing tweets.\n   */\n  public void indexingLoop() throws InterruptedException, IOException {\n    Stopwatch stopwatch = Stopwatch.createStarted();\n\n    Stopwatch readingStopwatch = Stopwatch.createUnstarted();\n    Stopwatch indexingStopwatch = Stopwatch.createUnstarted();\n\n    int indexedDocumentsCount = 0;\n    SearchLongGauge timeToIndexSegment = SearchLongGauge.export(\"time_to_index_segment\");\n    timeToIndexSegment.set(0);\n    if (tweetReader != null) {\n      while (!tweetReader.isExhausted() && !Thread.currentThread().isInterrupted()) {\n        readingStopwatch.start();\n        TweetDocument tweetDocument = tweetReader.readNext();\n        readingStopwatch.stop();\n\n        indexingStopwatch.start();\n        boolean documentIndexed = indexDocument(tweetDocument);\n        indexingStopwatch.stop();\n\n        if (!documentIndexed) {\n          // No documents waiting to be indexed.  Take a nap.\n          Thread.sleep(10);\n        } else {\n          indexedDocumentsCount++;\n        }\n\n        if (segmentSize >= EarlybirdConfig.getMaxSegmentSize()) {\n          LOG.error(\"Reached max segment size \" + segmentSize + \", stopping indexer\");\n          partitionIndexingMetricSet.maxSegmentSizeReachedCounter.increment();\n          tweetReader.stop();\n          break;\n        }\n      }\n    }\n\n    timeToIndexSegment.set(stopwatch.elapsed(TimeUnit.MILLISECONDS));\n\n    LOG.info(\"SimpleSegmentIndexer finished: {}. Documents: {}\",\n        indexingSegment.getSegmentName(), indexedDocumentsCount);\n    LOG.info(\"Time taken: {}, Reading time: {}, Indexing time: {}\",\n        stopwatch, readingStopwatch, indexingStopwatch);\n    LOG.info(\"Total Memory: {}, Free Memory: {}\",\n        Runtime.getRuntime().totalMemory(), Runtime.getRuntime().freeMemory());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SimpleStreamIndexer.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.time.Duration;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.concurrent.atomic.AtomicBoolean;\nimport java.util.stream.Collectors;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Verify;\n\nimport org.apache.kafka.clients.consumer.ConsumerRecord;\nimport org.apache.kafka.clients.consumer.ConsumerRecords;\nimport org.apache.kafka.clients.consumer.KafkaConsumer;\nimport org.apache.kafka.clients.consumer.OffsetAndTimestamp;\nimport org.apache.kafka.common.PartitionInfo;\nimport org.apache.kafka.common.TopicPartition;\nimport org.apache.kafka.common.errors.WakeupException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.earlybird.common.NonPagingAssert;\nimport com.twitter.search.earlybird.exception.MissingKafkaTopicException;\n\n/**\n * Abstract base class for processing events from Kafka with the goal of indexing them and\n * keeping Earlybirds up to date with the latest events. Indexing is defined by the\n * implementation.\n *\n * NOTE: {@link EarlybirdKafkaConsumer} (tweet/tweet events consumer) is doing this in its\n * own way, we might merge in the future.\n *\n * @param <K> (Long)\n * @param <V> (Event/Thrift type to be consumed)\n */\npublic abstract class SimpleStreamIndexer<K, V> {\n  private static final Logger LOG = LoggerFactory.getLogger(SimpleStreamIndexer.class);\n\n  private static final Duration POLL_TIMEOUT = Duration.ofMillis(250);\n  private static final Duration CAUGHT_UP_FRESHNESS = Duration.ofSeconds(5);\n\n  protected static final int MAX_POLL_RECORDS = 1000;\n\n  private final SearchCounter numPollErrors;\n  protected SearchRateCounter indexingSuccesses;\n  protected SearchRateCounter indexingFailures;\n\n  protected List<TopicPartition> topicPartitionList;\n  protected final KafkaConsumer<K, V> kafkaConsumer;\n  private final AtomicBoolean running = new AtomicBoolean(true);\n  private final String topic;\n\n  private boolean isCaughtUp = false;\n\n  /**\n   * Create a simple stream indexer.\n   *\n   * @throws MissingKafkaTopicException - this shouldn't happen, but in case some\n   * external stream is not present, we want to have the caller decide how to\n   * handle it. Some missing streams might be fatal, for others it might not be\n   * justified to block startup. There's no point in constructing this object if\n   * a stream is missing, so we don't allow that to happen.\n   */\n  public SimpleStreamIndexer(KafkaConsumer<K, V> kafkaConsumer,\n                             String topic) throws MissingKafkaTopicException {\n    this.kafkaConsumer = kafkaConsumer;\n    this.topic = topic;\n    List<PartitionInfo> partitionInfos = this.kafkaConsumer.partitionsFor(topic);\n\n    if (partitionInfos == null) {\n      LOG.error(\"Ooops, no partitions for {}\", topic);\n      NonPagingAssert.assertFailed(\"missing_topic_\" + topic);\n      throw new MissingKafkaTopicException(topic);\n    }\n    LOG.info(\"Discovered {} partitions for topic: {}\", partitionInfos.size(), topic);\n\n    numPollErrors = SearchCounter.export(\"stream_indexer_poll_errors_\" + topic);\n\n    this.topicPartitionList = partitionInfos\n        .stream()\n        .map(info -> new TopicPartition(topic, info.partition()))\n        .collect(Collectors.toList());\n    this.kafkaConsumer.assign(topicPartitionList);\n  }\n\n  /**\n   * Consume updates on startup until current (eg. until we've seen a record within 5 seconds\n   * of current time.)\n   */\n  public void readRecordsUntilCurrent() {\n    do {\n      ConsumerRecords<K, V> records = poll();\n\n      for (ConsumerRecord<K, V> record : records) {\n        if (record.timestamp() > System.currentTimeMillis() - CAUGHT_UP_FRESHNESS.toMillis()) {\n          isCaughtUp = true;\n        }\n        validateAndIndexRecord(record);\n      }\n    } while (!isCaughtUp());\n  }\n\n  /**\n   * Run the consumer, indexing record values directly into their respective structures.\n   */\n  public void run() {\n    try {\n      while (running.get()) {\n        for (ConsumerRecord<K, V> record : poll()) {\n          validateAndIndexRecord(record);\n        }\n      }\n    } catch (WakeupException e) {\n      if (running.get()) {\n        LOG.error(\"Caught wakeup exception while running\", e);\n      }\n    } finally {\n      kafkaConsumer.close();\n      LOG.info(\"Consumer closed.\");\n    }\n  }\n\n  public boolean isCaughtUp() {\n    return isCaughtUp;\n  }\n\n  /**\n   * For every partition in the topic, seek to an offset that has a timestamp greater\n   * than or equal to the given timestamp.\n   * @param timestamp\n   */\n  public void seekToTimestamp(Long timestamp) {\n    Map<TopicPartition, Long> partitionTimestampMap = topicPartitionList.stream()\n        .collect(Collectors.toMap(tp -> tp, tp -> timestamp));\n    Map<TopicPartition, OffsetAndTimestamp> partitionOffsetMap =\n        kafkaConsumer.offsetsForTimes(partitionTimestampMap);\n\n    partitionOffsetMap.forEach((tp, offsetAndTimestamp) -> {\n      Verify.verify(offsetAndTimestamp != null,\n        \"Couldn't find records after timestamp: \" + timestamp);\n\n      kafkaConsumer.seek(tp, offsetAndTimestamp.offset());\n    });\n  }\n\n  /**\n   * Seeks the kafka consumer to the beginning.\n   */\n  public void seekToBeginning() {\n    kafkaConsumer.seekToBeginning(topicPartitionList);\n  }\n\n  /**\n   * Polls and returns at most MAX_POLL_RECORDS records.\n   * @return\n   */\n  @VisibleForTesting\n  protected ConsumerRecords<K, V> poll() {\n    ConsumerRecords<K, V> records;\n    try {\n      records = kafkaConsumer.poll(POLL_TIMEOUT);\n    } catch (Exception e) {\n      records = ConsumerRecords.empty();\n      if (e instanceof WakeupException) {\n        throw e;\n      } else {\n        LOG.warn(\"Error polling from {} kafka topic.\", topic, e);\n        numPollErrors.increment();\n      }\n    }\n    return records;\n  }\n\n  protected abstract void validateAndIndexRecord(ConsumerRecord<K, V> record);\n\n  // Shutdown hook which can be called from a seperate thread. Calling consumer.wakeup() interrupts\n  // the running indexer and causes it to first stop polling for new records before gracefully\n  // closing the consumer.\n  public void close() {\n    LOG.info(\"Shutting down stream indexer for topic {}\", topic);\n    running.set(false);\n    kafkaConsumer.wakeup();\n  }\n}\n\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/SimpleUpdateIndexer.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\nimport java.util.Optional;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Stopwatch;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.util.io.dl.DLRecordTimestampUtil;\nimport com.twitter.search.common.util.io.recordreader.RecordReader;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.segment.SegmentDataReaderSet;\n\n/**\n * Indexes all updates for a complete segment at startup.\n */\npublic class SimpleUpdateIndexer {\n  private static final Logger LOG = LoggerFactory.getLogger(SimpleUpdateIndexer.class);\n\n  private final SegmentDataReaderSet readerSet;\n  private final SearchIndexingMetricSet partitionIndexingMetricSet;\n  private final InstrumentedQueue<ThriftVersionedEvents> retryQueue;\n  private final CriticalExceptionHandler criticalExceptionHandler;\n\n  public SimpleUpdateIndexer(SegmentDataReaderSet readerSet,\n                             SearchIndexingMetricSet partitionIndexingMetricSet,\n                             InstrumentedQueue<ThriftVersionedEvents> retryQueue,\n                             CriticalExceptionHandler criticalExceptionHandler) {\n    this.readerSet = readerSet;\n    this.partitionIndexingMetricSet = partitionIndexingMetricSet;\n    this.retryQueue = retryQueue;\n    this.criticalExceptionHandler = criticalExceptionHandler;\n  }\n\n  /**\n   * Indexes all updates for the given segment.\n   */\n  public void indexAllUpdates(SegmentInfo segmentInfo) {\n    Preconditions.checkState(\n        segmentInfo.isEnabled() && segmentInfo.isComplete() && !segmentInfo.isIndexing());\n\n    try {\n      readerSet.attachUpdateReaders(segmentInfo);\n    } catch (IOException e) {\n      throw new RuntimeException(\"Could not attach readers for segment: \" + segmentInfo, e);\n    }\n\n    RecordReader<ThriftVersionedEvents> reader =\n        readerSet.getUpdateEventsReaderForSegment(segmentInfo);\n    if (reader == null) {\n      return;\n    }\n\n    LOG.info(\"Got updates reader (starting timestamp = {}) for segment {}: {}\",\n             DLRecordTimestampUtil.recordIDToTimestamp(reader.getOffset()),\n             segmentInfo.getSegmentName(),\n             reader);\n\n    // The segment is complete (we check this in indexAllUpdates()), so we can safely get\n    // the smallest and largest tweet IDs in this segment.\n    long lowestTweetId = segmentInfo.getIndexSegment().getLowestTweetId();\n    long highestTweetId = segmentInfo.getIndexSegment().getHighestTweetId();\n    Preconditions.checkArgument(\n        lowestTweetId > 0,\n        \"Could not get the lowest tweet ID in segment \" + segmentInfo.getSegmentName());\n    Preconditions.checkArgument(\n        highestTweetId > 0,\n        \"Could not get the highest tweet ID in segment \" + segmentInfo.getSegmentName());\n\n    SegmentWriter segmentWriter =\n        new SegmentWriter(segmentInfo, partitionIndexingMetricSet.updateFreshness);\n\n    LOG.info(\"Starting to index updates for segment: {}\", segmentInfo.getSegmentName());\n    Stopwatch stopwatch = Stopwatch.createStarted();\n\n    while (!Thread.currentThread().isInterrupted() && !reader.isCaughtUp()) {\n      applyUpdate(segmentInfo, reader, segmentWriter, lowestTweetId, highestTweetId);\n    }\n\n    LOG.info(\"Finished indexing updates for segment {} in {} seconds.\",\n             segmentInfo.getSegmentName(),\n             stopwatch.elapsed(TimeUnit.SECONDS));\n  }\n\n  private void applyUpdate(SegmentInfo segmentInfo,\n                           RecordReader<ThriftVersionedEvents> reader,\n                           SegmentWriter segmentWriter,\n                           long lowestTweetId,\n                           long highestTweetId) {\n    ThriftVersionedEvents update;\n    try {\n      update = reader.readNext();\n    } catch (IOException e) {\n      LOG.error(\"Exception while reading update for segment: \" + segmentInfo.getSegmentName(), e);\n      criticalExceptionHandler.handle(this, e);\n      return;\n    }\n    if (update == null) {\n      LOG.warn(\"Update is not available but reader was not caught up. Segment: {}\",\n               segmentInfo.getSegmentName());\n      return;\n    }\n\n    try {\n      // If the indexer put this update in the wrong timeslice, add it to the retry queue, and\n      // let PartitionIndexer retry it (it has logic to apply it to the correct segment).\n      if ((update.getId() < lowestTweetId) || (update.getId() > highestTweetId)) {\n        retryQueue.add(update);\n        return;\n      }\n\n      // At this point, we are updating a segment that has every tweet it will ever have,\n      // (the segment is complete), so there is no point queueing an update to retry it.\n      SearchTimer timer = partitionIndexingMetricSet.updateStats.startNewTimer();\n      segmentWriter.indexThriftVersionedEvents(update);\n      partitionIndexingMetricSet.updateStats.stopTimerAndIncrement(timer);\n\n      updateUpdatesStreamTimestamp(segmentInfo);\n    } catch (IOException e) {\n      LOG.error(\"Exception while indexing updates for segment: \" + segmentInfo.getSegmentName(), e);\n      criticalExceptionHandler.handle(this, e);\n    }\n  }\n\n  private void updateUpdatesStreamTimestamp(SegmentInfo segmentInfo) {\n    Optional<Long> offset = readerSet.getUpdateEventsStreamOffsetForSegment(segmentInfo);\n    if (!offset.isPresent()) {\n      LOG.info(\"Unable to get updates stream offset for segment: {}\", segmentInfo.getSegmentName());\n    } else {\n      long offsetTimeMillis = DLRecordTimestampUtil.recordIDToTimestamp(offset.get());\n      segmentInfo.setUpdatesStreamOffsetTimestamp(offsetTimeMillis);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/StartupUserEventIndexer.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.sql.Timestamp;\nimport java.text.DateFormat;\nimport java.text.SimpleDateFormat;\nimport java.time.Duration;\nimport java.util.Date;\nimport java.util.Optional;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.earlybird.EarlybirdStatus;\nimport com.twitter.search.earlybird.common.NonPagingAssert;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.common.userupdates.UserScrubGeoMap;\nimport com.twitter.search.earlybird.common.userupdates.UserTableBuilderFromSnapshot;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.factory.EarlybirdIndexConfigUtil;\n\n/**\n * Indexer class responsible for getting the the {@link UserTable} and {@link UserScrubGeoMap}\n * indexed up until the current moment.\n */\npublic class StartupUserEventIndexer {\n  private static final Logger LOG = LoggerFactory.getLogger(StartupUserEventIndexer.class);\n  private static final String LOAD_USER_UPDATE_SNAPSHOT =\n      \"loading user update snapshot\";\n  private static final String INDEX_ALL_USER_EVENTS =\n      \"indexing all user events\";\n  private static final NonPagingAssert FAILED_USER_TABLE_HDFS_LOAD\n      = new NonPagingAssert(\"failed_user_table_hdfs_load\");\n\n  private static final long MAX_RETRY_MILLIS_FOR_SEEK_TO_TIMESTAMP =\n      Duration.ofMinutes(1).toMillis();\n  private static final long SLEEP_MILLIS_BETWEEN_RETRIES_FOR_SEEK_TO_TIMESTAMP =\n      Duration.ofSeconds(1).toMillis();\n\n  private static final long MILLIS_IN_FOURTEEN_DAYS = 1209600000;\n  private static final long MILLIS_IN_ONE_DAY = 86400000;\n\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n  private final UserUpdatesStreamIndexer userUpdatesStreamIndexer;\n  private final UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer;\n  private final SegmentManager segmentManager;\n  private final Clock clock;\n\n  public StartupUserEventIndexer(\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      UserUpdatesStreamIndexer userUpdatesStreamIndexer,\n      UserScrubGeoEventStreamIndexer userScrubGeoEventStreamIndexer,\n      SegmentManager segmentManager,\n      Clock clock) {\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n    this.userUpdatesStreamIndexer = userUpdatesStreamIndexer;\n    this.userScrubGeoEventStreamIndexer = userScrubGeoEventStreamIndexer;\n    this.segmentManager = segmentManager;\n    this.clock = clock;\n  }\n\n  /**\n   * Index all user events.\n   */\n  public void indexAllEvents() {\n    EarlybirdStatus.beginEvent(\n        INDEX_ALL_USER_EVENTS, searchIndexingMetricSet.startupInUserEventIndexer);\n\n    indexUserUpdates();\n    if (EarlybirdConfig.consumeUserScrubGeoEvents()) {\n      indexUserScrubGeoEvents();\n    }\n\n    EarlybirdStatus.endEvent(\n        INDEX_ALL_USER_EVENTS, searchIndexingMetricSet.startupInUserEventIndexer);\n  }\n\n  /**\n   * Index user updates until current.\n   */\n  public void indexUserUpdates() {\n    EarlybirdStatus.beginEvent(\n        LOAD_USER_UPDATE_SNAPSHOT, searchIndexingMetricSet.startupInUserUpdates);\n\n    Optional<UserTable> userTable = buildUserTable();\n    if (userTable.isPresent()) {\n      segmentManager.getUserTable().setTable(userTable.get());\n      LOG.info(\"Set new user table.\");\n\n      if (!seekToTimestampWithRetriesIfNecessary(\n          userTable.get().getLastRecordTimestamp(),\n          userUpdatesStreamIndexer)) {\n        LOG.error(\"User Updates stream indexer unable to seek to timestamp. \"\n            + \"Will seek to beginning.\");\n        userUpdatesStreamIndexer.seekToBeginning();\n      }\n    } else {\n      LOG.info(\"Failed to load user update snapshot. Will reindex user updates from scratch.\");\n      FAILED_USER_TABLE_HDFS_LOAD.assertFailed();\n      userUpdatesStreamIndexer.seekToBeginning();\n    }\n\n    userUpdatesStreamIndexer.readRecordsUntilCurrent();\n    LOG.info(\"Finished catching up on user updates via Kafka\");\n\n    EarlybirdStatus.endEvent(\n        LOAD_USER_UPDATE_SNAPSHOT, searchIndexingMetricSet.startupInUserUpdates);\n  }\n\n  /**\n   * Index UserScrubGeoEvents until current.\n   */\n  public void indexUserScrubGeoEvents() {\n    seekUserScrubGeoEventKafkaConsumer();\n\n    SearchTimer timer = new SearchTimer();\n    timer.start();\n    userScrubGeoEventStreamIndexer.readRecordsUntilCurrent();\n    timer.stop();\n\n    LOG.info(\"Finished catching up on user scrub geo events via Kafka\");\n    LOG.info(\"UserScrubGeoMap contains {} users and finished in {} milliseconds\",\n        segmentManager.getUserScrubGeoMap().getNumUsersInMap(), timer.getElapsed());\n  }\n\n  /**\n   * Seeks UserScrubGeoEventKafkaConsumer using timestamp derived from\n   * getTimestampForUserScrubGeoEventKafkaConsumer().\n   */\n  @VisibleForTesting\n  public void seekUserScrubGeoEventKafkaConsumer() {\n    long seekTimestamp = getTimestampForUserScrubGeoEventKafkaConsumer();\n    if (seekTimestamp == -1) {\n      userScrubGeoEventStreamIndexer.seekToBeginning();\n    } else {\n      if (!seekToTimestampWithRetriesIfNecessary(seekTimestamp, userScrubGeoEventStreamIndexer)) {\n        LOG.error(\"User Scrub Geo stream indexer unable to seek to timestamp. \"\n            + \"Will seek to beginning.\");\n        userScrubGeoEventStreamIndexer.seekToBeginning();\n      }\n    }\n  }\n\n  /**\n   * Get timestamp to seek UserScrubGeoEventKafkaConsumer to.\n   * @return\n   */\n  public long getTimestampForUserScrubGeoEventKafkaConsumer() {\n    if (EarlybirdIndexConfigUtil.isArchiveSearch()) {\n      return getTimestampForArchive();\n    } else {\n      return getTimestampForRealtime();\n    }\n  }\n\n  /**\n   * For archive: grab scrub gen from config file and convert date into a timestamp. Add buffer of\n   * one day. We need all UserScrubGeoEvents since the date of the current scrub gen.\n   *\n   * See go/realtime-geo-filtering\n   * @return\n   */\n  public long getTimestampForArchive() {\n    try {\n      String scrubGenString = EarlybirdProperty.EARLYBIRD_SCRUB_GEN.get();\n\n      DateFormat dateFormat = new SimpleDateFormat(\"yyyyMMdd\");\n      Date date = dateFormat.parse(scrubGenString);\n      return new Timestamp(date.getTime()).getTime() - MILLIS_IN_ONE_DAY;\n\n    } catch (Exception e) {\n      LOG.error(\"Could not derive timestamp from scrub gen. \"\n          + \"Will seek User Scrub Geo Kafka consumer to beginning of topic\");\n    }\n    return -1;\n  }\n\n  /**\n   * For realtime/protected: Compute the timestamp 14 days from the current time. This will account\n   * for all events that have occurred during the lifecylce of the current index.\n   *\n   * See go/realtime-geo-filtering\n   */\n  public long getTimestampForRealtime() {\n   return System.currentTimeMillis() - MILLIS_IN_FOURTEEN_DAYS;\n  }\n\n  private boolean seekToTimestampWithRetriesIfNecessary(\n      long lastRecordTimestamp,\n      SimpleStreamIndexer streamIndexer) {\n    long initialTimeMillis = clock.nowMillis();\n    int numFailures = 0;\n    while (shouldTrySeekToTimestamp(initialTimeMillis, numFailures)) {\n      try {\n        streamIndexer.seekToTimestamp(lastRecordTimestamp);\n        LOG.info(\"Seeked consumer to timestamp {} after {} failures\",\n            lastRecordTimestamp, numFailures);\n        return true;\n      } catch (Exception e) {\n        numFailures++;\n        LOG.info(\"Caught exception when seeking to timestamp. Num failures: {}. Exception: {}\",\n            numFailures, e);\n        // Sleep before attempting to retry\n        try {\n          clock.waitFor(SLEEP_MILLIS_BETWEEN_RETRIES_FOR_SEEK_TO_TIMESTAMP);\n        } catch (InterruptedException interruptedException) {\n          LOG.warn(\"Interrupted while sleeping between seekToTimestamp retries\",\n              interruptedException);\n          // Preserve interrupt status.\n          Thread.currentThread().interrupt();\n          break;\n        }\n      }\n    }\n    // Failed to seek to timestamp\n    return false;\n  }\n\n  private boolean shouldTrySeekToTimestamp(long initialTimeMillis, int numFailures) {\n    if (numFailures == 0) {\n      // no attempts have been made yet, so we should try to seek to timestamp\n      return true;\n    } else {\n      return clock.nowMillis() - initialTimeMillis < MAX_RETRY_MILLIS_FOR_SEEK_TO_TIMESTAMP;\n    }\n  }\n\n  protected Optional<UserTable> buildUserTable() {\n    UserTableBuilderFromSnapshot builder = new UserTableBuilderFromSnapshot();\n    return builder.build(segmentManager.getUserTable().getUserIdFilter());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/StatusBatchFlushVersion.java",
    "content": "package com.twitter.search.earlybird.partition;\n\n/**\n * Keeps track of versioning for flushed status batch data.\n */\npublic enum StatusBatchFlushVersion {\n\n  VERSION_0(\"Initial version of status batch flushing\", true),\n  VERSION_1(\"Switching to use field groups (contains changes to PartitionedBatch)\", true),\n  VERSION_2(\"Removing support for per-partition _SUCCESS markers\", true),\n  /* Put the semi colon on a separate line to avoid polluting git blame history */;\n\n  public static final StatusBatchFlushVersion CURRENT_FLUSH_VERSION =\n      StatusBatchFlushVersion.values()[StatusBatchFlushVersion.values().length - 1];\n\n  public static final String DELIMITER = \"_v_\";\n\n  private final String description;\n  private final boolean isOfficial;\n\n  private StatusBatchFlushVersion(String description, boolean official) {\n    this.description = description;\n    isOfficial = official;\n  }\n\n  public int getVersionNumber() {\n    return this.ordinal();\n  }\n\n  public String getVersionFileExtension() {\n      return DELIMITER + ordinal();\n  }\n\n  public boolean isOfficial() {\n    return isOfficial;\n  }\n\n  public String getDescription() {\n    return description;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/TimeLimitedHadoopExistsCall.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\nimport java.util.concurrent.Callable;\nimport java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.util.concurrent.SimpleTimeLimiter;\nimport com.google.common.util.concurrent.TimeLimiter;\n\nimport org.apache.hadoop.fs.FileSystem;\nimport org.apache.hadoop.fs.Path;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.metrics.SearchTimerStats;\n\n/**\n * Abstracts details of making time limited calls to hadoop.\n *\n * During IM-3556 we discovered that hadoop API calls can take a long time (seconds, minutes)\n * if the Hadoop clsuter is in a bad state. Our code was generally not prepared for that and\n * this caused various issues. This class is a fix on top of the Hadoop API's exists call and\n * it introduces a timeout.\n *\n * The main motivation for having this as an external class is for testability.\n */\npublic class TimeLimitedHadoopExistsCall {\n  private final TimeLimiter hadoopCallsTimeLimiter;\n  private final FileSystem fileSystem;\n  private final int timeLimitInSeconds;\n\n  private static final SearchTimerStats EXISTS_CALLS_TIMER =\n      SearchTimerStats.export(\"hadoop_exists_calls\");\n\n  private static final SearchCounter EXISTS_CALLS_EXCEPTION =\n      SearchCounter.export(\"hadoop_exists_calls_exception\");\n\n  public TimeLimitedHadoopExistsCall(FileSystem fileSystem) {\n    // This times varies. Sometimes it's very quick, sometimes it takes some amount of seconds.\n    // Do a rate on hadoop_exists_calls_latency_ms to see for yourself.\n    this(fileSystem, 30);\n  }\n\n  public TimeLimitedHadoopExistsCall(FileSystem fileSystem, int timeLimitInSeconds) {\n    // We do hadoop calls once every \"FLUSH_CHECK_PERIOD\" minutes. If a call takes\n    // a long time (say 10 minutes), we'll use a new thread for the next call, to give it\n    // a chance to complete.\n    //\n    // Let's say every call takes 2 hours. After 5 calls, the 6th call won't be able\n    // to take a thread out of the thread pool and it will time out. That's fair, we don't\n    // want to keep sending requests to Hadoop if the situation is so dire.\n    ExecutorService executorService = Executors.newFixedThreadPool(5);\n    this.hadoopCallsTimeLimiter = SimpleTimeLimiter.create(executorService);\n    this.fileSystem = fileSystem;\n    this.timeLimitInSeconds = timeLimitInSeconds;\n  }\n\n\n  protected boolean hadoopExistsCall(Path path) throws IOException {\n    SearchTimer timer = EXISTS_CALLS_TIMER.startNewTimer();\n    boolean res =  fileSystem.exists(path);\n    EXISTS_CALLS_TIMER.stopTimerAndIncrement(timer);\n    return res;\n  }\n\n  /**\n   * Checks if a path exists on Hadoop.\n   *\n   * @return true if the path exists.\n   * @throws Exception see exceptions thrown by callWithTimeout\n   */\n  boolean exists(Path path) throws Exception {\n    try {\n      boolean result = hadoopCallsTimeLimiter.callWithTimeout(new Callable<Boolean>() {\n        @Override\n        public Boolean call() throws Exception {\n          return hadoopExistsCall(path);\n        }\n      }, timeLimitInSeconds, TimeUnit.SECONDS);\n\n      return result;\n    } catch (Exception ex) {\n      EXISTS_CALLS_EXCEPTION.increment();\n      // No need to print and rethrow, it will be printed when caught upstream.\n      throw ex;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/TweetCreateHandler.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\nimport java.util.Iterator;\n\nimport scala.runtime.BoxedUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Stopwatch;\nimport com.google.common.base.Verify;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.search.common.util.GCUtil;\nimport com.twitter.search.earlybird.EarlybirdStatus;\nimport com.twitter.search.earlybird.common.CaughtUpMonitor;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.index.OutOfOrderRealtimeTweetIDMapper;\nimport com.twitter.search.earlybird.querycache.QueryCacheManager;\nimport com.twitter.search.earlybird.util.CoordinatedEarlybirdActionInterface;\nimport com.twitter.util.Await;\nimport com.twitter.util.Duration;\nimport com.twitter.util.Future;\nimport com.twitter.util.TimeoutException;\n\n/**\n * This class handles incoming new Tweets. It is responsible for creating segments for the incoming\n * Tweets when necessary, triggering optimization on those segments, and writing Tweets to the\n * correct segment.\n */\npublic class TweetCreateHandler {\n  private static final Logger LOG = LoggerFactory.getLogger(TweetCreateHandler.class);\n\n  public static final long LATE_TWEET_TIME_BUFFER_MS = Duration.fromMinutes(1).inMilliseconds();\n\n  private static final String STATS_PREFIX = \"tweet_create_handler_\";\n\n  // To get a better idea of which of these succeeded and so on, see stats in SegmentManager.\n  private IndexingResultCounts indexingResultCounts;\n  private static final SearchRateCounter TWEETS_IN_WRONG_SEGMENT =\n      SearchRateCounter.export(STATS_PREFIX + \"tweets_in_wrong_segment\");\n  private static final SearchRateCounter SEGMENTS_CLOSED_EARLY =\n      SearchRateCounter.export(STATS_PREFIX + \"segments_closed_early\");\n  private static final SearchRateCounter INSERTED_IN_CURRENT_SEGMENT =\n      SearchRateCounter.export(STATS_PREFIX + \"inserted_in_current_segment\");\n  private static final SearchRateCounter INSERTED_IN_PREVIOUS_SEGMENT =\n      SearchRateCounter.export(STATS_PREFIX + \"inserted_in_previous_segment\");\n  private static final NewSegmentStats NEW_SEGMENT_STATS = new NewSegmentStats();\n  private static final SearchCounter CREATED_SEGMENTS =\n      SearchCounter.export(STATS_PREFIX + \"created_segments\");\n  private static final SearchRateCounter INCOMING_TWEETS =\n          SearchRateCounter.export(STATS_PREFIX + \"incoming_tweets\");\n  private static final SearchRateCounter INDEXING_SUCCESS =\n          SearchRateCounter.export(STATS_PREFIX + \"indexing_success\");\n  private static final SearchRateCounter INDEXING_FAILURE =\n          SearchRateCounter.export(STATS_PREFIX + \"indexing_failure\");\n\n  // Various stats and logging around creation of new segments, put in this\n  // class so that the code is not watered down too much by this.\n  private static class NewSegmentStats {\n    private static final String NEW_SEGMENT_STATS_PREFIX =\n      STATS_PREFIX + \"new_segment_\";\n\n    private static final SearchCounter START_NEW_AFTER_REACHING_LIMIT =\n        SearchCounter.export(NEW_SEGMENT_STATS_PREFIX + \"start_after_reaching_limit\");\n    private static final SearchCounter START_NEW_AFTER_EXCEEDING_MAX_ID =\n        SearchCounter.export(NEW_SEGMENT_STATS_PREFIX + \"start_after_exceeding_max_id\");\n    private static final SearchCounter TIMESLICE_SET_TO_CURRENT_ID =\n        SearchCounter.export(NEW_SEGMENT_STATS_PREFIX + \"timeslice_set_to_current_id\");\n    private static final SearchCounter TIMESLICE_SET_TO_MAX_ID =\n        SearchCounter.export(NEW_SEGMENT_STATS_PREFIX + \"timeslice_set_to_max_id\");\n    private static final SearchLongGauge TIMESPAN_BETWEEN_MAX_AND_CURRENT =\n        SearchLongGauge.export(NEW_SEGMENT_STATS_PREFIX + \"timespan_between_id_and_max\");\n\n    void recordCreateNewSegment() {\n      CREATED_SEGMENTS.increment();\n    }\n\n    void recordStartAfterReachingTweetsLimit(int numDocs, int numDocsCutoff,\n                                             int maxSegmentSize, int lateTweetBuffer) {\n      START_NEW_AFTER_REACHING_LIMIT.increment();\n      LOG.info(String.format(\n          \"Will create new segment: numDocs=%,d, numDocsCutoff=%,d\"\n              + \" | maxSegmentSize=%,d, lateTweetBuffer=%,d\",\n          numDocs, numDocsCutoff, maxSegmentSize, lateTweetBuffer));\n    }\n\n    void recordStartAfterExceedingLargestValidTweetId(long tweetId, long largestValidTweetId) {\n      START_NEW_AFTER_EXCEEDING_MAX_ID.increment();\n      LOG.info(String.format(\n          \"Will create new segment: tweetDd=%,d, largestValidTweetID for segment=%,d\",\n          tweetId, largestValidTweetId));\n    }\n\n    void recordSettingTimesliceToCurrentTweet(long tweetID) {\n      TIMESLICE_SET_TO_CURRENT_ID.increment();\n      LOG.info(\"Creating new segment: tweet that triggered it has the largest id we've seen. \"\n          + \" id={}\", tweetID);\n    }\n\n    void recordSettingTimesliceToMaxTweetId(long tweetID, long maxTweetID) {\n      TIMESLICE_SET_TO_MAX_ID.increment();\n      LOG.info(\"Creating new segment: tweet that triggered it doesn't have the largest id\"\n          + \" we've seen. tweetId={}, maxTweetId={}\",\n          tweetID, maxTweetID);\n      long timeDifference =\n          SnowflakeIdParser.getTimeDifferenceBetweenTweetIDs(maxTweetID, tweetID);\n      LOG.info(\"Time difference between max seen and last seen: {} ms\", timeDifference);\n      TIMESPAN_BETWEEN_MAX_AND_CURRENT.set(timeDifference);\n    }\n\n    void wrapNewSegmentCreation(long tweetID, long maxTweetID,\n                                long currentSegmentTimesliceBoundary,\n                                long largestValidTweetIDForCurrentSegment) {\n      long timeDifferenceStartToMax = SnowflakeIdParser.getTimeDifferenceBetweenTweetIDs(\n          largestValidTweetIDForCurrentSegment,\n          currentSegmentTimesliceBoundary);\n      LOG.info(\"Time between timeslice boundary and largest valid tweet id: {} ms\",\n          timeDifferenceStartToMax);\n\n      LOG.info(\"Created new segment: (tweetId={}, maxTweetId={}, maxTweetId-tweetId={} \"\n              + \" | currentSegmentTimesliceBoundary={}, largestValidTweetIDForSegment={})\",\n          tweetID, maxTweetID, maxTweetID - tweetID, currentSegmentTimesliceBoundary,\n          largestValidTweetIDForCurrentSegment);\n    }\n  }\n\n\n  private final SegmentManager segmentManager;\n  private final MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager;\n  private final int maxSegmentSize;\n  private final int lateTweetBuffer;\n\n  private long maxTweetID = Long.MIN_VALUE;\n\n  private long largestValidTweetIDForCurrentSegment;\n  private long currentSegmentTimesliceBoundary;\n  private OptimizingSegmentWriter currentSegment;\n  private OptimizingSegmentWriter previousSegment;\n  private final QueryCacheManager queryCacheManager;\n  private final CriticalExceptionHandler criticalExceptionHandler;\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n  private final CoordinatedEarlybirdActionInterface postOptimizationRebuildsAction;\n  private final CoordinatedEarlybirdActionInterface gcAction;\n  private final CaughtUpMonitor indexCaughtUpMonitor;\n  private final OptimizationAndFlushingCoordinationLock optimizationAndFlushingCoordinationLock;\n\n  public TweetCreateHandler(\n      SegmentManager segmentManager,\n      SearchIndexingMetricSet searchIndexingMetricSet,\n      CriticalExceptionHandler criticalExceptionHandler,\n      MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager,\n      QueryCacheManager queryCacheManager,\n      CoordinatedEarlybirdActionInterface postOptimizationRebuildsAction,\n      CoordinatedEarlybirdActionInterface gcAction,\n      int lateTweetBuffer,\n      int maxSegmentSize,\n      CaughtUpMonitor indexCaughtUpMonitor,\n      OptimizationAndFlushingCoordinationLock optimizationAndFlushingCoordinationLock\n  ) {\n    this.segmentManager = segmentManager;\n    this.criticalExceptionHandler = criticalExceptionHandler;\n    this.multiSegmentTermDictionaryManager = multiSegmentTermDictionaryManager;\n    this.queryCacheManager = queryCacheManager;\n    this.indexingResultCounts = new IndexingResultCounts();\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n    this.postOptimizationRebuildsAction = postOptimizationRebuildsAction;\n    this.gcAction = gcAction;\n    this.indexCaughtUpMonitor = indexCaughtUpMonitor;\n\n    Preconditions.checkState(lateTweetBuffer < maxSegmentSize);\n    this.lateTweetBuffer = lateTweetBuffer;\n    this.maxSegmentSize = maxSegmentSize;\n    this.optimizationAndFlushingCoordinationLock = optimizationAndFlushingCoordinationLock;\n  }\n\n  void prepareAfterStartingWithIndex(long maxIndexedTweetId) {\n    LOG.info(\"Preparing after starting with an index.\");\n\n    Iterator<SegmentInfo> segmentInfosIterator =\n        segmentManager\n            .getSegmentInfos(SegmentManager.Filter.All, SegmentManager.Order.NEW_TO_OLD)\n            .iterator();\n\n    // Setup the last segment.\n    Verify.verify(segmentInfosIterator.hasNext(), \"at least one segment expected\");\n    ISegmentWriter lastWriter = segmentManager.getSegmentWriterForID(\n        segmentInfosIterator.next().getTimeSliceID());\n    Verify.verify(lastWriter != null);\n\n    LOG.info(\"TweetCreateHandler found last writer: {}\", lastWriter.getSegmentInfo().toString());\n    this.currentSegmentTimesliceBoundary = lastWriter.getSegmentInfo().getTimeSliceID();\n    this.largestValidTweetIDForCurrentSegment =\n        OutOfOrderRealtimeTweetIDMapper.calculateMaxTweetID(currentSegmentTimesliceBoundary);\n    this.currentSegment = (OptimizingSegmentWriter) lastWriter;\n\n    if (maxIndexedTweetId == -1) {\n      maxTweetID = lastWriter.getSegmentInfo().getIndexSegment().getMaxTweetId();\n      LOG.info(\"Max tweet id = {}\", maxTweetID);\n    } else {\n      // See SEARCH-31032\n      maxTweetID = maxIndexedTweetId;\n    }\n\n    // If we have a previous segment that's not optimized, set it up too, we still need to pick\n    // it up for optimization and we might still be able to add tweets to it.\n    if (segmentInfosIterator.hasNext()) {\n      SegmentInfo previousSegmentInfo = segmentInfosIterator.next();\n      if (!previousSegmentInfo.isOptimized()) {\n        ISegmentWriter previousSegmentWriter = segmentManager.getSegmentWriterForID(\n            previousSegmentInfo.getTimeSliceID());\n\n        if (previousSegmentWriter != null) {\n          LOG.info(\"Picked previous segment\");\n          this.previousSegment = (OptimizingSegmentWriter) previousSegmentWriter;\n        } else {\n          // Should not happen.\n          LOG.error(\"Not found previous segment writer\");\n        }\n      } else {\n        LOG.info(\"Previous segment info is optimized\");\n      }\n    } else {\n      LOG.info(\"Previous segment info not found, we only have one segment\");\n    }\n  }\n\n  private void updateIndexFreshness() {\n    searchIndexingMetricSet.highestStatusId.set(maxTweetID);\n\n    long tweetTimestamp = SnowflakeIdParser.getTimestampFromTweetId(\n        searchIndexingMetricSet.highestStatusId.get());\n    searchIndexingMetricSet.freshestTweetTimeMillis.set(tweetTimestamp);\n  }\n\n  /**\n   * Index a new TVE representing a Tweet create event.\n   */\n  public void handleTweetCreate(ThriftVersionedEvents tve) throws IOException {\n    INCOMING_TWEETS.increment();\n    long id = tve.getId();\n    maxTweetID = Math.max(id, maxTweetID);\n\n    updateIndexFreshness();\n\n    boolean shouldCreateNewSegment = false;\n\n    if (currentSegment == null) {\n      shouldCreateNewSegment = true;\n      LOG.info(\"Will create new segment: current segment is null\");\n    } else {\n      int numDocs = currentSegment.getSegmentInfo().getIndexSegment().getNumDocs();\n      int numDocsCutoff = maxSegmentSize - lateTweetBuffer;\n      if (numDocs >= numDocsCutoff) {\n        NEW_SEGMENT_STATS.recordStartAfterReachingTweetsLimit(numDocs, numDocsCutoff,\n            maxSegmentSize, lateTweetBuffer);\n        shouldCreateNewSegment = true;\n      } else if (id > largestValidTweetIDForCurrentSegment) {\n        NEW_SEGMENT_STATS.recordStartAfterExceedingLargestValidTweetId(id,\n            largestValidTweetIDForCurrentSegment);\n        shouldCreateNewSegment = true;\n      }\n    }\n\n    if (shouldCreateNewSegment) {\n      createNewSegment(id);\n    }\n\n    if (previousSegment != null) {\n      // Inserts and some updates can't be applied to an optimized segment, so we want to wait at\n      // least LATE_TWEET_TIME_BUFFER between when we created the new segment and when we optimize\n      // the previous segment, in case there are late tweets.\n      // We leave a large (150k, typically) buffer in the segment so that we don't have to close\n      // the previousSegment before LATE_TWEET_TIME_BUFFER has passed, but if we index\n      // lateTweetBuffer Tweets before optimizing, then we must optimize,\n      // so that we don't insert more than max segment size tweets into the previous segment.\n      long relativeTweetAgeMs =\n          SnowflakeIdParser.getTimeDifferenceBetweenTweetIDs(id, currentSegmentTimesliceBoundary);\n\n      boolean needToOptimize = false;\n      int numDocs = previousSegment.getSegmentInfo().getIndexSegment().getNumDocs();\n      String previousSegmentName = previousSegment.getSegmentInfo().getSegmentName();\n      if (numDocs >= maxSegmentSize) {\n        LOG.info(String.format(\"Previous segment (%s) reached maxSegmentSize, need to optimize it.\"\n            + \" numDocs=%,d, maxSegmentSize=%,d\", previousSegmentName, numDocs, maxSegmentSize));\n        needToOptimize = true;\n      } else if (relativeTweetAgeMs > LATE_TWEET_TIME_BUFFER_MS) {\n        LOG.info(String.format(\"Previous segment (%s) is old enough, we can optimize it.\"\n            + \" Got tweet past time buffer of %,d ms by: %,d ms\", previousSegmentName,\n            LATE_TWEET_TIME_BUFFER_MS, relativeTweetAgeMs - LATE_TWEET_TIME_BUFFER_MS));\n        needToOptimize = true;\n      }\n\n      if (needToOptimize) {\n        optimizePreviousSegment();\n      }\n    }\n\n    ISegmentWriter segmentWriter;\n    if (id >= currentSegmentTimesliceBoundary) {\n      INSERTED_IN_CURRENT_SEGMENT.increment();\n      segmentWriter = currentSegment;\n    } else if (previousSegment != null) {\n      INSERTED_IN_PREVIOUS_SEGMENT.increment();\n      segmentWriter = previousSegment;\n    } else {\n      TWEETS_IN_WRONG_SEGMENT.increment();\n      LOG.info(\"Inserting TVE ({}) into the current segment ({}) even though it should have gone \"\n          + \"in a previous segment.\", id, currentSegmentTimesliceBoundary);\n      segmentWriter = currentSegment;\n    }\n\n    SearchTimer timer = searchIndexingMetricSet.statusStats.startNewTimer();\n    ISegmentWriter.Result result = segmentWriter.indexThriftVersionedEvents(tve);\n    searchIndexingMetricSet.statusStats.stopTimerAndIncrement(timer);\n\n    if (result == ISegmentWriter.Result.SUCCESS) {\n      INDEXING_SUCCESS.increment();\n    } else {\n      INDEXING_FAILURE.increment();\n    }\n\n    indexingResultCounts.countResult(result);\n  }\n\n  /**\n   * Many tests need to verify behavior with segments optimized & unoptimized, so we need to expose\n   * this.\n   */\n  @VisibleForTesting\n  public Future<SegmentInfo> optimizePreviousSegment() {\n    String segmentName = previousSegment.getSegmentInfo().getSegmentName();\n    previousSegment.getSegmentInfo().setIndexing(false);\n    LOG.info(\"Optimizing previous segment: {}\", segmentName);\n    segmentManager.logState(\"Starting optimization for segment: \" + segmentName);\n\n    Future<SegmentInfo> future = previousSegment\n        .startOptimization(gcAction, optimizationAndFlushingCoordinationLock)\n        .map(this::postOptimizationSteps)\n        .onFailure(t -> {\n          criticalExceptionHandler.handle(this, t);\n          return BoxedUnit.UNIT;\n        });\n\n    waitForOptimizationIfInTest(future);\n\n    previousSegment = null;\n    return future;\n  }\n\n  /**\n   * In tests, it's easier if when a segment starts optimizing, we know that it will finish\n   * optimizing. This way we have no race condition where we're surprised that something that\n   * started optimizing is not ready.\n   *\n   * In prod we don't have this problem. Segments run for 10 hours and optimization is 20 minutes\n   * so there's no need for extra synchronization.\n   */\n  private void waitForOptimizationIfInTest(Future<SegmentInfo> future) {\n    if (Config.environmentIsTest()) {\n      try {\n        Await.ready(future);\n        LOG.info(\"Optimizing is done\");\n      } catch (InterruptedException | TimeoutException ex) {\n        LOG.info(\"Exception while optimizing\", ex);\n      }\n    }\n  }\n\n  private SegmentInfo postOptimizationSteps(SegmentInfo optimizedSegmentInfo) {\n    segmentManager.updateStats();\n    // See SEARCH-32175\n    optimizedSegmentInfo.setComplete(true);\n\n    String segmentName = optimizedSegmentInfo.getSegmentName();\n    LOG.info(\"Finished optimization for segment: \" + segmentName);\n    segmentManager.logState(\n            \"Finished optimization for segment: \" + segmentName);\n\n    /*\n     * Building the multi segment term dictionary causes GC pauses. The reason for this is because\n     * it's pretty big (possible ~15GB). When it's allocated, we have to copy a lot of data from\n     * survivor space to old gen. That causes several GC pauses. See SEARCH-33544\n     *\n     * GC pauses are in general not fatal, but since all instances finish a segment at roughly the\n     * same time, they might happen at the same time and then it's a problem.\n     *\n     * Some possible solutions to this problem would be to build this dictionary in some data\n     * structures that are pre-allocated or to build only the part for the last segment, as\n     * everything else doesn't change. These solutions are a bit difficult to implement and this\n     * here is an easy workaround.\n     *\n     * Note that we might finish optimizing a segment and then it might take ~60+ minutes until it's\n     * a particular Earlybird's turn to run this code. The effect of this is going to be that we\n     * are not going to use the multi segment dictionary for the last two segments, one of which is\n     * still pretty small. That's not terrible, since right before optimization we're not using\n     * the dictionary for the last segment anyways, since it's still not optimized.\n     */\n    try {\n      LOG.info(\"Acquire coordination lock before beginning post_optimization_rebuilds action.\");\n      optimizationAndFlushingCoordinationLock.lock();\n      LOG.info(\"Successfully acquired coordination lock for post_optimization_rebuilds action.\");\n      postOptimizationRebuildsAction.retryActionUntilRan(\n          \"post optimization rebuilds\", () -> {\n            Stopwatch stopwatch = Stopwatch.createStarted();\n            LOG.info(\"Starting to build multi term dictionary for {}\", segmentName);\n            boolean result = multiSegmentTermDictionaryManager.buildDictionary();\n            LOG.info(\"Done building multi term dictionary for {} in {}, result: {}\",\n                segmentName, stopwatch, result);\n            queryCacheManager.rebuildQueryCachesAfterSegmentOptimization(\n                optimizedSegmentInfo);\n\n            // This is a serial full GC and it defragments the memory so things can run smoothly\n            // until the next segment rolls. What we have observed is that if we don't do that\n            // later on some earlybirds can have promotion failures on an old gen that hasn't\n            // reached the initiating occupancy limit and these promotions failures can trigger a\n            // long (1.5 min) full GC. That usually happens because of fragmentation issues.\n            GCUtil.runGC();\n            // Wait for indexing to catch up before rejoining the serverset. We only need to do\n            // this if the host has already finished startup.\n            if (EarlybirdStatus.hasStarted()) {\n              indexCaughtUpMonitor.resetAndWaitUntilCaughtUp();\n            }\n          });\n    } finally {\n      LOG.info(\"Finished post_optimization_rebuilds action. Releasing coordination lock.\");\n      optimizationAndFlushingCoordinationLock.unlock();\n    }\n\n    return optimizedSegmentInfo;\n  }\n\n  /**\n   * Many tests rely on precise segment boundaries, so we expose this to allow them to create a\n   * particular segment.\n   */\n  @VisibleForTesting\n  public void createNewSegment(long tweetID) throws IOException {\n    NEW_SEGMENT_STATS.recordCreateNewSegment();\n\n    if (previousSegment != null) {\n      // We shouldn't have more than one unoptimized segment, so if we get to this point and the\n      // previousSegment has not been optimized and set to null, start optimizing it before\n      // creating the next one. Note that this is a weird case and would only happen if we get\n      // Tweets with drastically different IDs than we expect, or there is a large amount of time\n      // where no Tweets are created in this partition.\n      LOG.error(\"Creating new segment for Tweet {} when the previous segment {} was not sealed. \"\n          + \"Current segment: {}. Documents: {}. largestValidTweetIDForSegment: {}.\",\n          tweetID,\n          previousSegment.getSegmentInfo().getTimeSliceID(),\n          currentSegment.getSegmentInfo().getTimeSliceID(),\n          currentSegment.getSegmentInfo().getIndexSegment().getNumDocs(),\n          largestValidTweetIDForCurrentSegment);\n      optimizePreviousSegment();\n      SEGMENTS_CLOSED_EARLY.increment();\n    }\n\n    previousSegment = currentSegment;\n\n    // We have two cases:\n    //\n    // Case 1:\n    // If the greatest Tweet ID we have seen is tweetID, then when we want to create a new segment\n    // with that ID, so the Tweet being processed goes into the new segment.\n    //\n    // Case 2:\n    // If the tweetID is bigger than the max tweetID, then this method is being called directly from\n    // tests, so we didn't update the maxTweetID, so we can create a new segment with the new\n    // Tweet ID.\n    //\n    // Case 3:\n    // If it's not the greatest Tweet ID we have seen, then we don't want to create a\n    // segment boundary that is lower than any Tweet IDs in the current segment, because then\n    // some tweets from the previous segment would be in the wrong segment, so create a segment\n    // that has a greater ID than any Tweets that we have seen.\n    //\n    //   Example:\n    //     - We have seen tweets 3, 10, 5, 6.\n    //     - We now see tweet 7 and we decide it's time to create a new segment.\n    //     - The new segment will start at tweet 11. It can't start at tweet 7, because\n    //       tweet 10 will be in the wrong segment.\n    //     - Tweet 7 that we just saw will end up in the previous segment.\n    if (maxTweetID <= tweetID) {\n      currentSegmentTimesliceBoundary = tweetID;\n      NEW_SEGMENT_STATS.recordSettingTimesliceToCurrentTweet(tweetID);\n    } else {\n      currentSegmentTimesliceBoundary = maxTweetID + 1;\n      NEW_SEGMENT_STATS.recordSettingTimesliceToMaxTweetId(tweetID, maxTweetID);\n    }\n    currentSegment = segmentManager.createAndPutOptimizingSegmentWriter(\n        currentSegmentTimesliceBoundary);\n\n    currentSegment.getSegmentInfo().setIndexing(true);\n\n    largestValidTweetIDForCurrentSegment =\n        OutOfOrderRealtimeTweetIDMapper.calculateMaxTweetID(currentSegmentTimesliceBoundary);\n\n    NEW_SEGMENT_STATS.wrapNewSegmentCreation(tweetID, maxTweetID,\n        currentSegmentTimesliceBoundary, largestValidTweetIDForCurrentSegment);\n\n    segmentManager.removeExcessSegments();\n  }\n\n  void logState() {\n    LOG.info(\"TweetCreateHandler:\");\n    LOG.info(String.format(\"  tweets sent for indexing: %,d\",\n        indexingResultCounts.getIndexingCalls()));\n    LOG.info(String.format(\"  non-retriable failure: %,d\",\n        indexingResultCounts.getFailureNotRetriable()));\n    LOG.info(String.format(\"  retriable failure: %,d\",\n        indexingResultCounts.getFailureRetriable()));\n    LOG.info(String.format(\"  successfully indexed: %,d\",\n        indexingResultCounts.getIndexingSuccess()));\n    LOG.info(String.format(\"  tweets in wrong segment: %,d\", TWEETS_IN_WRONG_SEGMENT.getCount()));\n    LOG.info(String.format(\"  segments closed early: %,d\", SEGMENTS_CLOSED_EARLY.getCount()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/TweetUpdateHandler.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.SortedMap;\nimport java.util.TreeMap;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\n\n/**\n * This class handles incoming updates to Tweets in the index.\n *\n * Much of the logic deals with retries. It is very common to get an update before we have gotten\n * the Tweet that the update should be applied to. In this case, we queue the update for up to a\n * minute, so that we give the original Tweet the chance to be written to the index.\n */\npublic class TweetUpdateHandler {\n  private static final Logger LOG = LoggerFactory.getLogger(TweetUpdateHandler.class);\n  private static final Logger UPDATES_ERRORS_LOG =\n          LoggerFactory.getLogger(TweetUpdateHandler.class.getName() + \".UpdatesErrors\");\n\n  private static final String STATS_PREFIX = \"tweet_update_handler_\";\n\n  private IndexingResultCounts indexingResultCounts;\n  private static final SearchRateCounter INCOMING_EVENT =\n          SearchRateCounter.export(STATS_PREFIX + \"incoming_event\");\n  private static final SearchRateCounter QUEUED_FOR_RETRY =\n      SearchRateCounter.export(STATS_PREFIX + \"queued_for_retry\");\n  private static final SearchRateCounter DROPPED_OLD_EVENT =\n      SearchRateCounter.export(STATS_PREFIX + \"dropped_old_event\");\n  private static final SearchRateCounter DROPPED_INCOMING_EVENT =\n      SearchRateCounter.export(STATS_PREFIX + \"dropped_incoming_event\");\n  private static final SearchRateCounter DROPPED_CLEANUP_EVENT =\n      SearchRateCounter.export(STATS_PREFIX + \"dropped_cleanup_event\");\n  private static final SearchRateCounter DROPPED_NOT_RETRYABLE_EVENT =\n          SearchRateCounter.export(STATS_PREFIX + \"dropped_not_retryable_event\");\n  private static final SearchRateCounter PICKED_TO_RETRY =\n      SearchRateCounter.export(STATS_PREFIX + \"picked_to_retry\");\n  private static final SearchRateCounter INDEXED_EVENT =\n          SearchRateCounter.export(STATS_PREFIX + \"indexed_event\");\n\n  private static final long RETRY_TIME_THRESHOLD_MS = 60_000; // one minute.\n\n  private final SortedMap<Long, List<ThriftVersionedEvents>> pendingUpdates = new TreeMap<>();\n  private final SegmentManager segmentManager;\n\n  /**\n   * At this time we cleaned all updates that are more than RETRY_TIME_THRESHOLD_MS old.\n   */\n  private long lastCleanedUpdatesTime = 0;\n\n  /**\n   * The time of the most recent Tweet that we have applied an update for. We use this to\n   * determine when we should give up on retrying an update, instead of using the system clock,\n   * because we may be processing the stream from a long time ago if we are starting up or if\n   * there is lag in the Kafka topics and we want to let each update get a fair shot at being\n   * applied.\n   */\n  private long mostRecentUpdateTime = 0;\n\n  public TweetUpdateHandler(SegmentManager segmentManager) {\n    this.segmentManager = segmentManager;\n    this.indexingResultCounts = new IndexingResultCounts();\n  }\n\n  /**\n   * Index an update to a Tweet.\n   */\n  public void handleTweetUpdate(ThriftVersionedEvents tve, boolean isRetry) throws IOException {\n    if (!isRetry) {\n      INCOMING_EVENT.increment();\n    }\n    long id = tve.getId();\n\n    mostRecentUpdateTime =\n        Math.max(SnowflakeIdParser.getTimestampFromTweetId(id), mostRecentUpdateTime);\n    cleanStaleUpdates();\n\n    ISegmentWriter writer = segmentManager.getSegmentWriterForID(id);\n    if (writer == null) {\n      if (segmentManager.getNumIndexedDocuments() == 0) {\n        // If we haven't indexed any tweets at all, then we shouldn't drop this update, because it\n        // might be applied to a Tweet we haven't indexed yet so queue it up for retry.\n        queueForRetry(id, tve);\n      } else {\n        DROPPED_OLD_EVENT.increment();\n      }\n      return;\n    }\n\n    SegmentWriter.Result result = writer.indexThriftVersionedEvents(tve);\n    indexingResultCounts.countResult(result);\n\n    if (result == ISegmentWriter.Result.FAILURE_RETRYABLE) {\n      // If the tweet hasn't arrived yet.\n      queueForRetry(id, tve);\n    } else if (result == ISegmentWriter.Result.FAILURE_NOT_RETRYABLE) {\n      DROPPED_NOT_RETRYABLE_EVENT.increment();\n      UPDATES_ERRORS_LOG.warn(\"Failed to apply update for tweetID {}: {}\", id, tve);\n    } else if (result == ISegmentWriter.Result.SUCCESS) {\n      INDEXED_EVENT.increment();\n    }\n  }\n\n  private void queueForRetry(long id, ThriftVersionedEvents tve) {\n    long ageMillis = mostRecentUpdateTime - SnowflakeIdParser.getTimestampFromTweetId(id);\n    if (ageMillis > RETRY_TIME_THRESHOLD_MS) {\n      DROPPED_INCOMING_EVENT.increment();\n      UPDATES_ERRORS_LOG.warn(\n              \"Giving up retrying update for tweetID {}: {} because the retry time has elapsed\",\n              id, tve);\n      return;\n    }\n\n    pendingUpdates.computeIfAbsent(id, i -> new ArrayList<>()).add(tve);\n    QUEUED_FOR_RETRY.increment();\n  }\n\n  // Every time we have processed a minute's worth of updates, remove all pending updates that are\n  // more than a minute old, relative to the most recent Tweet we have seen.\n  private void cleanStaleUpdates() {\n    long oldUpdatesThreshold = mostRecentUpdateTime - RETRY_TIME_THRESHOLD_MS;\n    if (lastCleanedUpdatesTime < oldUpdatesThreshold) {\n      SortedMap<Long, List<ThriftVersionedEvents>> droppedUpdates = pendingUpdates\n          .headMap(SnowflakeIdParser.generateValidStatusId(oldUpdatesThreshold, 0));\n      for (List<ThriftVersionedEvents> events : droppedUpdates.values()) {\n        for (ThriftVersionedEvents event : events) {\n          UPDATES_ERRORS_LOG.warn(\n                  \"Giving up retrying update for tweetID {}: {} because the retry time has elapsed\",\n                  event.getId(), event);\n        }\n        DROPPED_CLEANUP_EVENT.increment(events.size());\n      }\n      droppedUpdates.clear();\n\n      lastCleanedUpdatesTime = mostRecentUpdateTime;\n    }\n  }\n\n  /**\n   * After we successfully indexed tweetID, if we have any pending updates for that tweetID, try to\n   * apply them again.\n   */\n  public void retryPendingUpdates(long tweetID) throws IOException {\n    if (pendingUpdates.containsKey(tweetID)) {\n      for (ThriftVersionedEvents update : pendingUpdates.remove(tweetID)) {\n        PICKED_TO_RETRY.increment();\n        handleTweetUpdate(update, true);\n      }\n    }\n  }\n\n  void logState() {\n    LOG.info(\"TweetUpdateHandler:\");\n    LOG.info(String.format(\"  tweets sent for indexing: %,d\",\n        indexingResultCounts.getIndexingCalls()));\n    LOG.info(String.format(\"  non-retriable failure: %,d\",\n        indexingResultCounts.getFailureNotRetriable()));\n    LOG.info(String.format(\"  retriable failure: %,d\",\n        indexingResultCounts.getFailureRetriable()));\n    LOG.info(String.format(\"  successfully indexed: %,d\",\n        indexingResultCounts.getIndexingSuccess()));\n    LOG.info(String.format(\"  queued for retry: %,d\", QUEUED_FOR_RETRY.getCount()));\n    LOG.info(String.format(\"  dropped old events: %,d\", DROPPED_OLD_EVENT.getCount()));\n    LOG.info(String.format(\"  dropped incoming events: %,d\", DROPPED_INCOMING_EVENT.getCount()));\n    LOG.info(String.format(\"  dropped cleanup events: %,d\", DROPPED_CLEANUP_EVENT.getCount()));\n    LOG.info(String.format(\"  picked events to retry: %,d\", PICKED_TO_RETRY.getCount()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/UserPartitionUtil.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport com.google.common.base.Predicate;\n\nimport com.twitter.search.common.util.hash.EarlybirdPartitioningFunction;\nimport com.twitter.search.common.util.hash.GeneralEarlybirdPartitioningFunction;\n\npublic final class UserPartitionUtil {\n  private UserPartitionUtil() {\n  }\n\n  /**\n   * Filter out the users that are not present in this partition.\n   */\n  public static Predicate<Long> filterUsersByPartitionPredicate(final PartitionConfig config) {\n    return new Predicate<Long>() {\n\n      private final int partitionID = config.getIndexingHashPartitionID();\n      private final int numPartitions = config.getNumPartitions();\n      private final EarlybirdPartitioningFunction partitioner =\n          new GeneralEarlybirdPartitioningFunction();\n\n      @Override\n      public boolean apply(Long userId) {\n        // See SEARCH-6675\n        // Right now if the partitioning logic changes in ArchivePartitioning this logic\n        // needs to be updated too.\n        return partitioner.getPartition(userId, numPartitions) == partitionID;\n      }\n    };\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/UserScrubGeoEventStreamIndexer.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.kafka.clients.consumer.ConsumerRecord;\nimport org.apache.kafka.clients.consumer.KafkaConsumer;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.util.io.kafka.FinagleKafkaClientUtils;\nimport com.twitter.search.common.util.io.kafka.ThriftDeserializer;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.exception.MissingKafkaTopicException;\nimport com.twitter.tweetypie.thriftjava.TweetEvent;\nimport com.twitter.tweetypie.thriftjava.UserScrubGeoEvent;\n\npublic class UserScrubGeoEventStreamIndexer extends SimpleStreamIndexer<Long, TweetEvent> {\n  private static final Logger LOG = LoggerFactory.getLogger(UserScrubGeoEventStreamIndexer.class);\n\n  protected static String kafkaClientId = \"earlybird_user_scrub_geo_kafka_consumer\";\n  private static final SearchCounter NUM_MISSING_DATA_ERRORS =\n      SearchCounter.export(\"num_user_scrub_geo_event_kafka_consumer_num_missing_data_errors\");\n\n  private final SegmentManager segmentManager;\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n\n  public UserScrubGeoEventStreamIndexer(KafkaConsumer<Long, TweetEvent> kafkaConsumer,\n                                        String topic,\n                                        SearchIndexingMetricSet searchIndexingMetricSet,\n                                        SegmentManager segmentManager)\n      throws MissingKafkaTopicException {\n    super(kafkaConsumer, topic);\n\n    this.segmentManager = segmentManager;\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n\n    indexingSuccesses = SearchRateCounter.export(\"user_scrub_geo_indexing_successes\");\n    indexingFailures = SearchRateCounter.export(\"user_scrub_geo_indexing_failures\");\n  }\n\n  /**\n   * Provides UserScrubGeoEvent Kafka Consumer to EarlybirdWireModule.\n   * @return\n   */\n  public static KafkaConsumer<Long, TweetEvent> provideKafkaConsumer() {\n    return FinagleKafkaClientUtils.newKafkaConsumerForAssigning(\n        EarlybirdProperty.TWEET_EVENTS_KAFKA_PATH.get(),\n        new ThriftDeserializer<>(TweetEvent.class),\n        kafkaClientId,\n        MAX_POLL_RECORDS);\n  }\n\n  @VisibleForTesting\n  protected void validateAndIndexRecord(ConsumerRecord<Long, TweetEvent> record) {\n    TweetEvent event = record.value();\n    UserScrubGeoEvent geoEvent;\n    try {\n     geoEvent = event.getData().getUser_scrub_geo_event();\n    } catch (Exception e) {\n      LOG.warn(\"TweetEventData is null for TweetEvent: \" + event.toString());\n      indexingFailures.increment();\n      return;\n    }\n\n    if (geoEvent == null) {\n      LOG.warn(\"UserScrubGeoEvent is null\");\n      indexingFailures.increment();\n\n    } else if (!geoEvent.isSetMax_tweet_id() || !geoEvent.isSetUser_id()) {\n      // We should not consume an event that does not contain both a maxTweetId & userId since we\n      // we won't have enough data to properly store them in the map. We should, however, keep\n      // track of these cases since we don't want to miss out on users who have scrubbed their\n      // geo data from their tweets when applying the UserScrubGeoFilter.\n      LOG.warn(\"UserScrubGeoEvent is missing fields: \" + geoEvent.toString());\n      indexingFailures.increment();\n      NUM_MISSING_DATA_ERRORS.increment();\n\n    } else {\n      SearchTimer timer = searchIndexingMetricSet.userScrubGeoIndexingStats.startNewTimer();\n      segmentManager.indexUserScrubGeoEvent(geoEvent);\n      indexingSuccesses.increment();\n      searchIndexingMetricSet.userScrubGeoIndexingStats.stopTimerAndIncrement(timer);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/UserUpdatesStreamIndexer.java",
    "content": "package com.twitter.search.earlybird.partition;\n\nimport java.util.Date;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.kafka.clients.consumer.ConsumerRecord;\nimport org.apache.kafka.clients.consumer.KafkaConsumer;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.indexing.thriftjava.AntisocialUserUpdate;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.util.io.kafka.CompactThriftDeserializer;\nimport com.twitter.search.common.util.io.kafka.FinagleKafkaClientUtils;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.common.userupdates.UserUpdate;\nimport com.twitter.search.earlybird.exception.MissingKafkaTopicException;\n\npublic class UserUpdatesStreamIndexer extends SimpleStreamIndexer<Long, AntisocialUserUpdate> {\n  private static final Logger LOG = LoggerFactory.getLogger(UserUpdatesStreamIndexer.class);\n\n  private static final SearchCounter NUM_CORRUPT_DATA_ERRORS =\n      SearchCounter.export(\"num_user_updates_kafka_consumer_corrupt_data_errors\");\n  protected static String kafkaClientId = \"\";\n\n  private final SegmentManager segmentManager;\n  private final SearchIndexingMetricSet searchIndexingMetricSet;\n\n  public UserUpdatesStreamIndexer(KafkaConsumer<Long, AntisocialUserUpdate> kafkaConsumer,\n                                  String topic,\n                                  SearchIndexingMetricSet searchIndexingMetricSet,\n                                  SegmentManager segmentManager)\n      throws MissingKafkaTopicException {\n    super(kafkaConsumer, topic);\n    this.segmentManager = segmentManager;\n    this.searchIndexingMetricSet = searchIndexingMetricSet;\n\n    indexingSuccesses = SearchRateCounter.export(\"user_update_indexing_successes\");\n    indexingFailures = SearchRateCounter.export(\"user_update_indexing_failures\");\n  }\n\n  /**\n   * Provides user updates kafka consumer to EarlybirdWireModule.\n   * @return\n   */\n  public static KafkaConsumer<Long, AntisocialUserUpdate> provideKafkaConsumer() {\n    return FinagleKafkaClientUtils.newKafkaConsumerForAssigning(\n        EarlybirdProperty.KAFKA_PATH.get(),\n        new CompactThriftDeserializer<>(AntisocialUserUpdate.class),\n        kafkaClientId,\n        MAX_POLL_RECORDS);\n  }\n\n  UserUpdate convertToUserInfoUpdate(AntisocialUserUpdate update) {\n    return new UserUpdate(\n        update.getUserID(),\n        update.getType(),\n        update.isValue() ? 1 : 0,\n        new Date(update.getUpdatedAt()));\n  }\n\n  @VisibleForTesting\n  protected void validateAndIndexRecord(ConsumerRecord<Long, AntisocialUserUpdate> record) {\n    AntisocialUserUpdate update = record.value();\n    if (update == null) {\n      LOG.warn(\"null value returned from poll\");\n      return;\n    }\n    if (update.getType() == null) {\n      LOG.error(\"User update does not have type set: \" + update);\n      NUM_CORRUPT_DATA_ERRORS.increment();\n      return;\n    }\n\n    SearchTimer timer = searchIndexingMetricSet.userUpdateIndexingStats.startNewTimer();\n    boolean isUpdateIndexed = segmentManager.indexUserUpdate(\n        convertToUserInfoUpdate(update));\n    searchIndexingMetricSet.userUpdateIndexingStats.stopTimerAndIncrement(timer);\n\n    if (isUpdateIndexed) {\n      indexingSuccesses.increment();\n    } else {\n      indexingFailures.increment();\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/freshstartup/FreshStartupHandler.java",
    "content": "package com.twitter.search.earlybird.partition.freshstartup;\n\nimport java.io.IOException;\nimport java.time.Duration;\nimport java.util.ArrayList;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\n\nimport com.google.common.base.Stopwatch;\nimport com.google.common.base.Verify;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Lists;\n\nimport org.apache.kafka.clients.consumer.ConsumerRecord;\nimport org.apache.kafka.clients.consumer.ConsumerRecords;\nimport org.apache.kafka.clients.consumer.KafkaConsumer;\nimport org.apache.kafka.clients.consumer.OffsetAndTimestamp;\nimport org.apache.kafka.common.TopicPartition;\nimport org.apache.kafka.common.errors.ApiException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport static com.twitter.search.common.util.LogFormatUtil.formatInt;\n\nimport com.twitter.search.common.util.GCUtil;\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.util.LogFormatUtil;\nimport com.twitter.search.earlybird.common.NonPagingAssert;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.exception.EarlybirdStartupException;\nimport com.twitter.search.earlybird.exception.WrappedKafkaApiException;\nimport com.twitter.search.earlybird.factory.EarlybirdKafkaConsumersFactory;\nimport com.twitter.search.earlybird.partition.EarlybirdIndex;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentManager;\nimport com.twitter.search.earlybird.util.ParallelUtil;\n\n/**\n * Bootstraps an index by indexing tweets and updates in parallel.\n *\n * DEVELOPMENT\n * ===========\n *\n * 1. In earlybird-search.yml, set the following values in the \"production\" section:\n *   - max_segment_size to 200000\n *   - late_tweet_buffer to 10000\n *\n * 2. In KafkaStartup, don't load the index, replace the .loadIndex call as instructed\n *  in the file.\n *\n * 3. In the aurora configs, set serving_timeslices to a low number (like 5) for staging.\n */\npublic class FreshStartupHandler {\n  private static final Logger LOG = LoggerFactory.getLogger(FreshStartupHandler.class);\n  private static final NonPagingAssert BUILDING_FEWER_THAN_SPECIFIED_SEGMENTS =\n          new NonPagingAssert(\"building_fewer_than_specified_segments\");\n\n  private final Clock clock;\n  private final TopicPartition tweetTopic;\n  private final TopicPartition updateTopic;\n  private final SegmentManager segmentManager;\n  private final int maxSegmentSize;\n  private final int lateTweetBuffer;\n  private final EarlybirdKafkaConsumersFactory earlybirdKafkaConsumersFactory;\n  private final CriticalExceptionHandler criticalExceptionHandler;\n\n  public FreshStartupHandler(\n    Clock clock,\n    EarlybirdKafkaConsumersFactory earlybirdKafkaConsumersFactory,\n    TopicPartition tweetTopic,\n    TopicPartition updateTopic,\n    SegmentManager segmentManager,\n    int maxSegmentSize,\n    int lateTweetBuffer,\n    CriticalExceptionHandler criticalExceptionHandler\n  ) {\n    this.clock = clock;\n    this.earlybirdKafkaConsumersFactory = earlybirdKafkaConsumersFactory;\n    this.tweetTopic = tweetTopic;\n    this.updateTopic = updateTopic;\n    this.segmentManager = segmentManager;\n    this.maxSegmentSize = maxSegmentSize;\n    this.criticalExceptionHandler = criticalExceptionHandler;\n    this.lateTweetBuffer = lateTweetBuffer;\n  }\n\n  /**\n   * Don't index in parallel, just pass some time back that the EarlybirdKafkaConsumer\n   * can start indexing from.\n   */\n  public EarlybirdIndex indexFromScratch() {\n    long indexTimePeriod = Duration.ofHours(\n        EarlybirdConfig.getInt(\"index_from_scratch_hours\", 12)\n    ).toMillis();\n\n    return runIndexFromScratch(indexTimePeriod);\n  }\n\n  public EarlybirdIndex fastIndexFromScratchForDevelopment() {\n    LOG.info(\"Running fast index from scratch...\");\n    return runIndexFromScratch(Duration.ofMinutes(10).toMillis());\n  }\n\n  private EarlybirdIndex runIndexFromScratch(long indexTimePeriodMs) {\n    KafkaConsumer<Long, ThriftVersionedEvents> consumerForFindingOffsets =\n        earlybirdKafkaConsumersFactory.createKafkaConsumer(\"consumer_for_offsets\");\n\n    long timestamp = clock.nowMillis() - indexTimePeriodMs;\n\n    Map<TopicPartition, OffsetAndTimestamp> offsets;\n    try {\n      offsets = consumerForFindingOffsets\n          .offsetsForTimes(ImmutableMap.of(tweetTopic, timestamp, updateTopic, timestamp));\n    } catch (ApiException kafkaApiException) {\n      throw new WrappedKafkaApiException(kafkaApiException);\n    }\n\n    return new EarlybirdIndex(\n        Lists.newArrayList(),\n        offsets.get(tweetTopic).offset(),\n        offsets.get(updateTopic).offset());\n  }\n\n\n  /**\n   * Index Tweets and updates from scratch, without relying on a serialized index in HDFS.\n   *\n   * This function indexes the segments in parallel, limiting the number of segments that\n   * are currently indexed, due to memory limitations. That's followed by another pass to index\n   * some updates - see the implementation for more details.\n   *\n   * The index this function outputs contains N segments, where the first N-1 are optimized and\n   * the last one is not.\n   */\n  public EarlybirdIndex parallelIndexFromScratch() throws Exception {\n    Stopwatch parallelIndexStopwatch = Stopwatch.createStarted();\n\n    LOG.info(\"Starting parallel fresh startup.\");\n    LOG.info(\"Max segment size: {}\", maxSegmentSize);\n    LOG.info(\"Late tweet buffer size: {}\", lateTweetBuffer);\n\n    // Once we finish fresh startup and proceed to indexing from the streams, we'll immediately\n    // start a new segment, since the output of the fresh startup is full segments.\n    //\n    // That's why we index max_segments-1 segments here instead of indexing max_segments segments\n    // and discarding the first one later.\n    int numSegments = segmentManager.getMaxEnabledSegments() - 1;\n    LOG.info(\"Number of segments to build: {}\", numSegments);\n\n    // Find end offsets.\n    KafkaOffsetPair tweetsOffsetRange = findOffsetRangeForTweetsKafkaTopic();\n\n    ArrayList<SegmentBuildInfo> segmentBuildInfos = makeSegmentBuildInfos(\n        numSegments, tweetsOffsetRange);\n\n    segmentManager.logState(\"Before starting fresh startup\");\n\n    // Index tweets and events.\n    Stopwatch initialIndexStopwatch = Stopwatch.createStarted();\n\n    // We index at most `MAX_PARALLEL_INDEXED` (MPI) segments at the same time. If we need to\n    // produce 20 segments here, we'd need memory for MPI unoptimized and 20-MPI optimized segments.\n    //\n    // For back of envelope calculations you can assume optimized segments take ~6GB and unoptimized\n    // ones ~12GB.\n    final int MAX_PARALLEL_INDEXED = 8;\n\n    List<SegmentInfo> segmentInfos = ParallelUtil.parmap(\n        \"fresh-startup\",\n        MAX_PARALLEL_INDEXED,\n        segmentBuildInfo -> indexTweetsAndUpdatesForSegment(segmentBuildInfo, segmentBuildInfos),\n        segmentBuildInfos\n    );\n\n    LOG.info(\"Finished indexing tweets and updates in {}\", initialIndexStopwatch);\n\n    PostOptimizationUpdatesIndexer postOptimizationUpdatesIndexer =\n        new PostOptimizationUpdatesIndexer(\n            segmentBuildInfos,\n            earlybirdKafkaConsumersFactory,\n            updateTopic);\n\n    postOptimizationUpdatesIndexer.indexRestOfUpdates();\n\n    // Finished indexing tweets and updates.\n    LOG.info(\"Segment build infos after we're done:\");\n    for (SegmentBuildInfo segmentBuildInfo : segmentBuildInfos) {\n      segmentBuildInfo.logState();\n    }\n\n    segmentManager.logState(\"After finishing fresh startup\");\n\n    LOG.info(\"Collected {} segment infos\", segmentInfos.size());\n    LOG.info(\"Segment names:\");\n    for (SegmentInfo segmentInfo : segmentInfos) {\n      LOG.info(segmentInfo.getSegmentName());\n    }\n\n    SegmentBuildInfo lastSegmentBuildInfo = segmentBuildInfos.get(segmentBuildInfos.size() - 1);\n    long finishedUpdatesAtOffset = lastSegmentBuildInfo.getUpdateKafkaOffsetPair().getEndOffset();\n    long maxIndexedTweetId = lastSegmentBuildInfo.getMaxIndexedTweetId();\n\n    LOG.info(\"Max indexed tweet id: {}\", maxIndexedTweetId);\n    LOG.info(\"Parallel startup finished in {}\", parallelIndexStopwatch);\n\n    // verifyConstructedIndex(segmentBuildInfos);\n    // Run a GC to free up some memory after the fresh startup.\n    GCUtil.runGC();\n    logMemoryStats();\n\n    return new EarlybirdIndex(\n        segmentInfos,\n        tweetsOffsetRange.getEndOffset() + 1,\n        finishedUpdatesAtOffset + 1,\n        maxIndexedTweetId\n    );\n  }\n\n  private void logMemoryStats() {\n    double toGB = 1024 * 1024 * 1024;\n    double totalMemoryGB = Runtime.getRuntime().totalMemory() / toGB;\n    double freeMemoryGB = Runtime.getRuntime().freeMemory() / toGB;\n    LOG.info(\"Memory stats: Total memory GB: {}, Free memory GB: {}\",\n        totalMemoryGB, freeMemoryGB);\n  }\n\n  /**\n   * Prints statistics about the constructed index compared to all tweets in the\n   * tweets stream.\n   *\n   * Only run this for testing and debugging purposes, never in prod environment.\n   */\n  private void verifyConstructedIndex(List<SegmentBuildInfo> segmentBuildInfos)\n      throws IOException {\n    LOG.info(\"Verifying constructed index...\");\n    // Read every tweet from the offset range that we're constructing an index for.\n    KafkaConsumer<Long, ThriftVersionedEvents> tweetsKafkaConsumer =\n        earlybirdKafkaConsumersFactory.createKafkaConsumer(\"tweets_verify\");\n    try {\n      tweetsKafkaConsumer.assign(ImmutableList.of(tweetTopic));\n      tweetsKafkaConsumer.seek(tweetTopic, segmentBuildInfos.get(0).getTweetStartOffset());\n    } catch (ApiException apiException) {\n      throw new WrappedKafkaApiException(apiException);\n    }\n    long finalTweetOffset = segmentBuildInfos.get(segmentBuildInfos.size() - 1).getTweetEndOffset();\n    boolean done = false;\n    Set<Long> uniqueTweetIds = new HashSet<>();\n    long readTweetsCount = 0;\n    do {\n      for (ConsumerRecord<Long, ThriftVersionedEvents> record\n          : tweetsKafkaConsumer.poll(Duration.ofSeconds(1))) {\n        if (record.offset() > finalTweetOffset) {\n          done = true;\n          break;\n        }\n        readTweetsCount++;\n        uniqueTweetIds.add(record.value().getId());\n      }\n    } while (!done);\n\n    LOG.info(\"Total amount of read tweets: {}\", formatInt(readTweetsCount));\n    // Might be less, due to duplicates.\n    LOG.info(\"Unique tweet ids : {}\", LogFormatUtil.formatInt(uniqueTweetIds.size()));\n\n    int notFoundInIndex = 0;\n    for (Long tweetId : uniqueTweetIds) {\n      boolean found = false;\n      for (SegmentBuildInfo segmentBuildInfo : segmentBuildInfos) {\n        if (segmentBuildInfo.getSegmentWriter().hasTweet(tweetId)) {\n          found = true;\n          break;\n        }\n      }\n      if (!found) {\n        notFoundInIndex++;\n      }\n    }\n\n    LOG.info(\"Tweets not found in the index: {}\", LogFormatUtil.formatInt(notFoundInIndex));\n\n    long totalIndexedTweets = 0;\n    for (SegmentBuildInfo segmentBuildInfo : segmentBuildInfos) {\n      SegmentInfo si = segmentBuildInfo.getSegmentWriter().getSegmentInfo();\n      totalIndexedTweets += si.getIndexStats().getStatusCount();\n    }\n\n    LOG.info(\"Total indexed tweets: {}\", formatInt(totalIndexedTweets));\n  }\n\n  /**\n   * Find the end offsets for the tweets Kafka topic this partition is reading\n   * from.\n   */\n  private KafkaOffsetPair findOffsetRangeForTweetsKafkaTopic() {\n    KafkaConsumer<Long, ThriftVersionedEvents> consumerForFindingOffsets =\n        earlybirdKafkaConsumersFactory.createKafkaConsumer(\"consumer_for_end_offsets\");\n\n    Map<TopicPartition, Long> endOffsets;\n    Map<TopicPartition, Long> beginningOffsets;\n\n    try {\n      endOffsets = consumerForFindingOffsets.endOffsets(ImmutableList.of(tweetTopic));\n      beginningOffsets = consumerForFindingOffsets.beginningOffsets(ImmutableList.of(tweetTopic));\n    } catch (ApiException kafkaApiException) {\n      throw new WrappedKafkaApiException(kafkaApiException);\n    } finally {\n      consumerForFindingOffsets.close();\n    }\n\n    long tweetsBeginningOffset = beginningOffsets.get(tweetTopic);\n    long tweetsEndOffset = endOffsets.get(tweetTopic);\n    LOG.info(String.format(\"Tweets beginning offset: %,d\", tweetsBeginningOffset));\n    LOG.info(String.format(\"Tweets end offset: %,d\", tweetsEndOffset));\n    LOG.info(String.format(\"Total amount of records in the stream: %,d\",\n        tweetsEndOffset - tweetsBeginningOffset + 1));\n\n    return new KafkaOffsetPair(tweetsBeginningOffset, tweetsEndOffset);\n  }\n\n  /**\n   * For each segment, we know what offset it begins at. This function finds the tweet ids\n   * for these offsets.\n   */\n  private void fillTweetIdsForSegmentStarts(List<SegmentBuildInfo> segmentBuildInfos)\n      throws EarlybirdStartupException {\n    KafkaConsumer<Long, ThriftVersionedEvents> consumerForTweetIds =\n        earlybirdKafkaConsumersFactory.createKafkaConsumer(\"consumer_for_tweet_ids\", 1);\n    consumerForTweetIds.assign(ImmutableList.of(tweetTopic));\n\n    // Find first tweet ids for each segment.\n    for (SegmentBuildInfo buildInfo : segmentBuildInfos) {\n      long tweetOffset = buildInfo.getTweetStartOffset();\n      ConsumerRecords<Long, ThriftVersionedEvents> records;\n      try {\n        consumerForTweetIds.seek(tweetTopic, tweetOffset);\n        records = consumerForTweetIds.poll(Duration.ofSeconds(1));\n      } catch (ApiException kafkaApiException) {\n        throw new WrappedKafkaApiException(kafkaApiException);\n      }\n\n      if (records.count() > 0) {\n        ConsumerRecord<Long, ThriftVersionedEvents> recordAtOffset = records.iterator().next();\n        if (recordAtOffset.offset() != tweetOffset) {\n          LOG.error(String.format(\"We were looking for offset %,d. Found a record at offset %,d\",\n              tweetOffset, recordAtOffset.offset()));\n        }\n\n        buildInfo.setStartTweetId(recordAtOffset.value().getId());\n      } else {\n        throw new EarlybirdStartupException(\"Didn't get any tweets back for an offset\");\n      }\n    }\n\n    // Check that something weird didn't happen where we end up with segment ids\n    // which are in non-incresing order.\n    // Goes from oldest to newest.\n    for (int i = 1; i < segmentBuildInfos.size(); i++) {\n      long startTweetId = segmentBuildInfos.get(i).getStartTweetId();\n      long prevStartTweetId = segmentBuildInfos.get(i - 1).getStartTweetId();\n      Verify.verify(prevStartTweetId < startTweetId);\n    }\n  }\n\n  /**\n   * Generate the offsets at which tweets begin and end for each segment that we want\n   * to create.\n   */\n  private ArrayList<SegmentBuildInfo> makeSegmentBuildInfos(\n      int numSegments, KafkaOffsetPair tweetsOffsets) throws EarlybirdStartupException {\n    ArrayList<SegmentBuildInfo> segmentBuildInfos = new ArrayList<>();\n\n    // If we have 3 segments, the starting tweet offsets are:\n    // end-3N, end-2N, end-N\n    int segmentSize = maxSegmentSize - lateTweetBuffer;\n    LOG.info(\"Segment size: {}\", segmentSize);\n\n    long tweetsInStream = tweetsOffsets.getEndOffset() - tweetsOffsets.getBeginOffset() + 1;\n    double numBuildableSegments = ((double) tweetsInStream) / segmentSize;\n\n    LOG.info(\"Number of segments we can build: {}\", numBuildableSegments);\n\n    int numSegmentsToBuild = numSegments;\n    int numBuildableSegmentsInt = (int) numBuildableSegments;\n\n    if (numBuildableSegmentsInt < numSegmentsToBuild) {\n      // This can happen if we get a low amount of tweets such that the ~10 days of tweets stored in\n      // Kafka are not enough to build the specified number of segments.\n      LOG.warn(\"Building {} segments instead of the specified {} segments because there are not \"\n              + \"enough tweets\", numSegmentsToBuild, numSegments);\n      BUILDING_FEWER_THAN_SPECIFIED_SEGMENTS.assertFailed();\n      numSegmentsToBuild = numBuildableSegmentsInt;\n    }\n\n    for (int rewind = numSegmentsToBuild; rewind >= 1; rewind--) {\n      long tweetStartOffset = (tweetsOffsets.getEndOffset() + 1) - (rewind * segmentSize);\n      long tweetEndOffset = tweetStartOffset + segmentSize - 1;\n\n      int index = segmentBuildInfos.size();\n\n      segmentBuildInfos.add(new SegmentBuildInfo(\n          tweetStartOffset,\n          tweetEndOffset,\n          index,\n          rewind == 1\n      ));\n    }\n\n    Verify.verify(segmentBuildInfos.get(segmentBuildInfos.size() - 1)\n        .getTweetEndOffset() == tweetsOffsets.getEndOffset());\n\n    LOG.info(\"Filling start tweet ids ...\");\n    fillTweetIdsForSegmentStarts(segmentBuildInfos);\n\n    return segmentBuildInfos;\n  }\n\n  private SegmentInfo indexTweetsAndUpdatesForSegment(\n      SegmentBuildInfo segmentBuildInfo,\n      ArrayList<SegmentBuildInfo> segmentBuildInfos) throws Exception {\n\n    PreOptimizationSegmentIndexer preOptimizationSegmentIndexer =\n        new PreOptimizationSegmentIndexer(\n            segmentBuildInfo,\n            segmentBuildInfos,\n            this.segmentManager,\n            this.tweetTopic,\n            this.updateTopic,\n            this.earlybirdKafkaConsumersFactory,\n            this.lateTweetBuffer\n        );\n\n    return preOptimizationSegmentIndexer.runIndexing();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/freshstartup/KafkaOffsetPair.java",
    "content": "package com.twitter.search.earlybird.partition.freshstartup;\n\nclass KafkaOffsetPair {\n  private final long beginOffset;\n  private final long endOffset;\n\n  public KafkaOffsetPair(long beginOffset, long endOffset) {\n    this.beginOffset = beginOffset;\n    this.endOffset = endOffset;\n  }\n\n  public boolean includes(long offset) {\n    return beginOffset <= offset && offset <= endOffset;\n  }\n\n  public long getBeginOffset() {\n    return beginOffset;\n  }\n\n  public long getEndOffset() {\n    return endOffset;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/freshstartup/PostOptimizationUpdatesIndexer.java",
    "content": "package com.twitter.search.earlybird.partition.freshstartup;\n\nimport java.io.IOException;\nimport java.time.Duration;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.base.Stopwatch;\nimport com.google.common.collect.ImmutableList;\n\nimport org.apache.kafka.clients.consumer.ConsumerRecord;\nimport org.apache.kafka.clients.consumer.ConsumerRecords;\nimport org.apache.kafka.clients.consumer.KafkaConsumer;\nimport org.apache.kafka.common.TopicPartition;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.earlybird.factory.EarlybirdKafkaConsumersFactory;\nimport com.twitter.search.earlybird.partition.IndexingResultCounts;\n\n/**\n * Indexes updates for all segments after they have been optimized. Some of the updates have been\n * indexed before in the PreOptimizationSegmentIndexer, but the rest are indexed here.\n */\nclass PostOptimizationUpdatesIndexer {\n  private static final Logger LOG = LoggerFactory.getLogger(PostOptimizationUpdatesIndexer.class);\n\n  private static final String STAT_PREFIX = \"post_optimization_\";\n  private static final String READ_STAT_PREFIX = STAT_PREFIX + \"read_updates_for_segment_\";\n  private static final String APPLIED_STAT_PREFIX = STAT_PREFIX + \"applied_updates_for_segment_\";\n\n  private final ArrayList<SegmentBuildInfo> segmentBuildInfos;\n  private final EarlybirdKafkaConsumersFactory earlybirdKafkaConsumersFactory;\n  private final TopicPartition updateTopic;\n\n  PostOptimizationUpdatesIndexer(\n      ArrayList<SegmentBuildInfo> segmentBuildInfos,\n      EarlybirdKafkaConsumersFactory earlybirdKafkaConsumersFactory,\n      TopicPartition updateTopic) {\n    this.segmentBuildInfos = segmentBuildInfos;\n    this.earlybirdKafkaConsumersFactory = earlybirdKafkaConsumersFactory;\n    this.updateTopic = updateTopic;\n  }\n\n  void indexRestOfUpdates() throws IOException {\n    LOG.info(\"Indexing rest of updates.\");\n\n    long updatesStartOffset = segmentBuildInfos.get(0)\n        .getUpdateKafkaOffsetPair().getBeginOffset();\n    long updatesEndOffset = segmentBuildInfos.get(segmentBuildInfos.size() - 1)\n        .getUpdateKafkaOffsetPair().getEndOffset();\n\n    LOG.info(String.format(\"Total updates to go through: %,d\",\n        updatesEndOffset - updatesStartOffset + 1));\n\n    KafkaConsumer<Long, ThriftVersionedEvents> kafkaConsumer =\n        earlybirdKafkaConsumersFactory.createKafkaConsumer(\"index_rest_of_updates\");\n    kafkaConsumer.assign(ImmutableList.of(updateTopic));\n    kafkaConsumer.seek(updateTopic, updatesStartOffset);\n\n    long readEvents = 0;\n    long foundSegment = 0;\n    long applied = 0;\n\n    Map<Integer, SearchRateCounter> perSegmentReadUpdates = new HashMap<>();\n    Map<Integer, SearchRateCounter> perSegmentAppliedUpdates = new HashMap<>();\n    Map<Integer, IndexingResultCounts> perSegmentIndexingResultCounts = new HashMap<>();\n\n    for (int i = 0; i < segmentBuildInfos.size(); i++) {\n      perSegmentReadUpdates.put(i, SearchRateCounter.export(READ_STAT_PREFIX + i));\n      perSegmentAppliedUpdates.put(i, SearchRateCounter.export(APPLIED_STAT_PREFIX + i));\n      perSegmentIndexingResultCounts.put(i, new IndexingResultCounts());\n    }\n\n    SearchTimerStats pollStats = SearchTimerStats.export(\n        \"final_pass_polls\", TimeUnit.NANOSECONDS, false);\n    SearchTimerStats indexStats = SearchTimerStats.export(\n        \"final_pass_index\", TimeUnit.NANOSECONDS, false);\n\n    Stopwatch totalTime = Stopwatch.createStarted();\n\n    boolean done = false;\n    do {\n      // Poll events.\n      SearchTimer pt = pollStats.startNewTimer();\n      ConsumerRecords<Long, ThriftVersionedEvents> records =\n          kafkaConsumer.poll(Duration.ofSeconds(1));\n      pollStats.stopTimerAndIncrement(pt);\n\n      // Index events.\n      SearchTimer it = indexStats.startNewTimer();\n      for (ConsumerRecord<Long, ThriftVersionedEvents> record : records) {\n        if (record.offset() >= updatesEndOffset) {\n          done = true;\n        }\n\n        readEvents++;\n\n        ThriftVersionedEvents tve = record.value();\n        long tweetId = tve.getId();\n\n        // Find segment to apply to. If we can't find a segment, this is an\n        // update for an old tweet that's not in the index.\n        int segmentIndex = -1;\n        for (int i = segmentBuildInfos.size() - 1; i >= 0; i--) {\n          if (segmentBuildInfos.get(i).getStartTweetId() <= tweetId) {\n            segmentIndex = i;\n            foundSegment++;\n            break;\n          }\n        }\n\n        if (segmentIndex != -1) {\n          SegmentBuildInfo segmentBuildInfo = segmentBuildInfos.get(segmentIndex);\n\n          perSegmentReadUpdates.get(segmentIndex).increment();\n\n          // Not already applied?\n          if (!segmentBuildInfo.getUpdateKafkaOffsetPair().includes(record.offset())) {\n            applied++;\n\n            // Index the update.\n            //\n            // IMPORTANT: Note that there you'll see about 2-3% of updates that\n            // fail as \"retryable\". This type of failure happens when the update is\n            // for a tweet that's not found in the index. We found out that we are\n            // receiving some updates for protected tweets and these are not in the\n            // realtime index - they are the source of this error.\n            perSegmentIndexingResultCounts.get(segmentIndex).countResult(\n                segmentBuildInfo.getSegmentWriter().indexThriftVersionedEvents(tve)\n            );\n\n            perSegmentAppliedUpdates.get(segmentIndex).increment();\n          }\n        }\n        if (record.offset() >= updatesEndOffset) {\n          break;\n        }\n      }\n      indexStats.stopTimerAndIncrement(it);\n\n    } while (!done);\n\n    LOG.info(String.format(\"Done in: %s, read %,d events, found segment for %,d, applied %,d\",\n        totalTime, readEvents, foundSegment, applied));\n\n    LOG.info(\"Indexing time: {}\", indexStats.getElapsedTimeAsString());\n    LOG.info(\"Polling time: {}\", pollStats.getElapsedTimeAsString());\n\n    LOG.info(\"Per segment indexing result counts:\");\n    for (int i = 0; i < segmentBuildInfos.size(); i++) {\n      LOG.info(\"{} : {}\", i, perSegmentIndexingResultCounts.get(i));\n    }\n\n    LOG.info(\"Found and applied per segment:\");\n    for (int i = 0; i < segmentBuildInfos.size(); i++) {\n      LOG.info(\"{}: found: {}, applied: {}\",\n          i,\n          perSegmentReadUpdates.get(i).getCount(),\n          perSegmentAppliedUpdates.get(i).getCount());\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/freshstartup/PreOptimizationSegmentIndexer.java",
    "content": "package com.twitter.search.earlybird.partition.freshstartup;\n\nimport java.io.IOException;\nimport java.time.Duration;\nimport java.util.ArrayList;\nimport java.util.Optional;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Stopwatch;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.ImmutableMap;\n\nimport org.apache.kafka.clients.consumer.ConsumerRecord;\nimport org.apache.kafka.clients.consumer.ConsumerRecords;\nimport org.apache.kafka.clients.consumer.KafkaConsumer;\nimport org.apache.kafka.clients.consumer.OffsetAndTimestamp;\nimport org.apache.kafka.common.TopicPartition;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.earlybird.factory.EarlybirdKafkaConsumersFactory;\nimport com.twitter.search.earlybird.partition.IndexingResultCounts;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentManager;\nimport com.twitter.search.earlybird.partition.SegmentWriter;\n\n/**\n * Responsible for indexing the tweets and updates that need to be applied to a single segment\n * before it gets optimized and then optimizing the segment (except if it's the last one).\n *\n * After that, no more tweets are added to the segment and the rest of the updates are added\n * in PostOptimizationUpdatesIndexer.\n */\nclass PreOptimizationSegmentIndexer {\n  private static final Logger LOG = LoggerFactory.getLogger(PreOptimizationSegmentIndexer.class);\n\n  private SegmentBuildInfo segmentBuildInfo;\n  private final ArrayList<SegmentBuildInfo> segmentBuildInfos;\n  private SegmentManager segmentManager;\n  private final TopicPartition tweetTopic;\n  private final TopicPartition updateTopic;\n  private final EarlybirdKafkaConsumersFactory earlybirdKafkaConsumersFactory;\n  private final long lateTweetBuffer;\n\n  public PreOptimizationSegmentIndexer(\n      SegmentBuildInfo segmentBuildInfo,\n      ArrayList<SegmentBuildInfo> segmentBuildInfos,\n      SegmentManager segmentManager,\n      TopicPartition tweetTopic,\n      TopicPartition updateTopic,\n      EarlybirdKafkaConsumersFactory earlybirdKafkaConsumersFactory,\n      long lateTweetBuffer) {\n    this.segmentBuildInfo = segmentBuildInfo;\n    this.segmentBuildInfos = segmentBuildInfos;\n    this.segmentManager = segmentManager;\n    this.tweetTopic = tweetTopic;\n    this.updateTopic = updateTopic;\n    this.earlybirdKafkaConsumersFactory = earlybirdKafkaConsumersFactory;\n    this.lateTweetBuffer = lateTweetBuffer;\n  }\n\n  SegmentInfo runIndexing() throws IOException {\n    LOG.info(String.format(\"Starting segment building for segment %d. \"\n            + \"Tweet offset range [ %,d, %,d ]\",\n        segmentBuildInfo.getIndex(),\n        segmentBuildInfo.getTweetStartOffset(),\n        segmentBuildInfo.getTweetEndOffset()));\n\n    Optional<Long> firstTweetIdInNextSegment = Optional.empty();\n    int index = segmentBuildInfo.getIndex();\n    if (index + 1 < segmentBuildInfos.size()) {\n      firstTweetIdInNextSegment = Optional.of(\n          segmentBuildInfos.get(index + 1).getStartTweetId());\n    }\n\n    // Index tweets.\n    SegmentTweetsIndexingResult tweetIndexingResult = indexSegmentTweetsFromStream(\n        tweetTopic,\n        String.format(\"tweet_consumer_for_segment_%d\", segmentBuildInfo.getIndex()),\n        firstTweetIdInNextSegment\n    );\n\n    // Index updates.\n    KafkaOffsetPair updatesIndexingOffsets = findUpdateStreamOffsetRange(tweetIndexingResult);\n\n    String updatesConsumerClientId =\n        String.format(\"update_consumer_for_segment_%d\", segmentBuildInfo.getIndex());\n\n    LOG.info(String.format(\"Consumer: %s :: Tweets start time: %d, end time: %d ==> \"\n            + \"Updates start offset: %,d, end offset: %,d\",\n        updatesConsumerClientId,\n        tweetIndexingResult.getMinRecordTimestampMs(),\n        tweetIndexingResult.getMaxRecordTimestampMs(),\n        updatesIndexingOffsets.getBeginOffset(),\n        updatesIndexingOffsets.getEndOffset()));\n\n    indexUpdatesFromStream(\n        updateTopic,\n        updatesConsumerClientId,\n        updatesIndexingOffsets.getBeginOffset(),\n        updatesIndexingOffsets.getEndOffset(),\n        tweetIndexingResult.getSegmentWriter()\n    );\n\n    if (segmentBuildInfo.isLastSegment()) {\n      /*\n       * We don't optimize the last segment for a few reasons:\n       *\n       * 1. We might have tweets coming next in the stream, which are supposed to end\n       *    up in this segment.\n       *\n       * 2. We might have updates coming next in the stream, which need to be applied to\n       *    this segment before it's optimized.\n       *\n       * So the segment is kept unoptimized and later we take care of setting up things\n       * so that PartitionWriter and the tweet create/update handlers can start correctly.\n       */\n      LOG.info(\"Not optimizing the last segment ({})\", segmentBuildInfo.getIndex());\n    } else {\n      Stopwatch optimizationStopwatch = Stopwatch.createStarted();\n      try {\n        LOG.info(\"Starting to optimize segment: {}\", segmentBuildInfo.getIndex());\n        tweetIndexingResult.getSegmentWriter().getSegmentInfo()\n            .getIndexSegment().optimizeIndexes();\n      } finally {\n        LOG.info(\"Optimization of segment {} finished in {}.\",\n            segmentBuildInfo.getIndex(), optimizationStopwatch);\n      }\n    }\n\n    segmentBuildInfo.setUpdateKafkaOffsetPair(updatesIndexingOffsets);\n    segmentBuildInfo.setMaxIndexedTweetId(tweetIndexingResult.getMaxIndexedTweetId());\n    segmentBuildInfo.setSegmentWriter(tweetIndexingResult.getSegmentWriter());\n\n    return tweetIndexingResult.getSegmentWriter().getSegmentInfo();\n  }\n\n  private SegmentTweetsIndexingResult indexSegmentTweetsFromStream(\n      TopicPartition topicPartition,\n      String consumerClientId,\n      Optional<Long> firstTweetIdInNextSegment) throws IOException {\n    long startOffset = segmentBuildInfo.getTweetStartOffset();\n    long endOffset = segmentBuildInfo.getTweetEndOffset();\n    long marginSize = lateTweetBuffer / 2;\n\n    boolean isFirstSegment = segmentBuildInfo.getIndex() == 0;\n\n    long startReadingAtOffset = startOffset;\n    if (!isFirstSegment) {\n      startReadingAtOffset -= marginSize;\n    } else {\n      LOG.info(\"Not moving start offset backwards for segment {}.\", segmentBuildInfo.getIndex());\n    }\n\n    long endReadingAtOffset = endOffset;\n    if (firstTweetIdInNextSegment.isPresent()) {\n      endReadingAtOffset += marginSize;\n    } else {\n      LOG.info(\"Not moving end offset forwards for segment {}.\", segmentBuildInfo.getIndex());\n    }\n\n    KafkaConsumer<Long, ThriftVersionedEvents> tweetsKafkaConsumer =\n        makeKafkaConsumerForIndexing(consumerClientId,\n            topicPartition, startReadingAtOffset);\n\n    boolean done = false;\n    long minIndexedTimestampMs = Long.MAX_VALUE;\n    long maxIndexedTimestampMs = Long.MIN_VALUE;\n    int indexedEvents = 0;\n\n    Stopwatch stopwatch = Stopwatch.createStarted();\n\n    LOG.info(\"Creating segment writer for timeslice ID {}.\", segmentBuildInfo.getStartTweetId());\n    SegmentWriter segmentWriter = segmentManager.createSegmentWriter(\n        segmentBuildInfo.getStartTweetId());\n\n    /*\n     * We don't have a guarantee that tweets come in sorted order, so when we're building segment\n     * X', we try to pick some tweets from the previous and next ranges we're going to index.\n     *\n     * We also ignore tweets in the beginning and the end of our tweets range, which are picked\n     * by the previous or following segment.\n     *\n     *   Segment X        Segment X'                              Segment X''\n     * -------------- o ----------------------------------------- o ---------------\n     *        [~~~~~] ^ [~~~~~]                           [~~~~~] | [~~~~~]\n     *           |    |    |                                 |    |    |\n     *  front margin  |    front padding (size K)   back padding  |   back margin\n     *                |                                           |\n     *                segment boundary at offset B' (1)           B''\n     *\n     * (1) This is at a predetermined tweet offset / tweet id.\n     *\n     * For segment X', we start to read tweets at offset B'-K and finish reading\n     * tweets at offset B''+K. K is a constant.\n     *\n     * For middle segments X'\n     * ======================\n     * We move some tweets from the front margin and back margin into segment X'.\n     * Some tweets from the front and back padding are ignored, as they are moved\n     * into the previous and next segments.\n     *\n     * For the first segment\n     * =====================\n     * No front margin, no front padding. We just read from the beginning offset\n     * and insert everything.\n     *\n     * For the last segment\n     * ====================\n     * No back margin, no back padding. We just read until the end.\n     */\n\n    SkippedPickedCounter frontMargin = new SkippedPickedCounter(\"front margin\");\n    SkippedPickedCounter backMargin = new SkippedPickedCounter(\"back margin\");\n    SkippedPickedCounter frontPadding = new SkippedPickedCounter(\"front padding\");\n    SkippedPickedCounter backPadding = new SkippedPickedCounter(\"back padding\");\n    SkippedPickedCounter regular = new SkippedPickedCounter(\"regular\");\n    int totalRead = 0;\n    long maxIndexedTweetId = -1;\n\n    Stopwatch pollTimer = Stopwatch.createUnstarted();\n    Stopwatch indexTimer = Stopwatch.createUnstarted();\n\n    do {\n      // This can cause an exception, See P33896\n      pollTimer.start();\n      ConsumerRecords<Long, ThriftVersionedEvents> records =\n          tweetsKafkaConsumer.poll(Duration.ofSeconds(1));\n      pollTimer.stop();\n\n      indexTimer.start();\n      for (ConsumerRecord<Long, ThriftVersionedEvents> record : records) {\n        // Done reading?\n        if (record.offset() >= endReadingAtOffset) {\n          done = true;\n        }\n\n        ThriftVersionedEvents tve = record.value();\n        boolean indexTweet = false;\n        SkippedPickedCounter skippedPickedCounter;\n\n        if (record.offset() < segmentBuildInfo.getTweetStartOffset()) {\n          // Front margin.\n          skippedPickedCounter = frontMargin;\n          if (tve.getId() > segmentBuildInfo.getStartTweetId()) {\n            indexTweet = true;\n          }\n        } else if (record.offset() > segmentBuildInfo.getTweetEndOffset()) {\n          // Back margin.\n          skippedPickedCounter = backMargin;\n          if (firstTweetIdInNextSegment.isPresent()\n              && tve.getId() < firstTweetIdInNextSegment.get()) {\n            indexTweet = true;\n          }\n        } else if (record.offset() < segmentBuildInfo.getTweetStartOffset() + marginSize) {\n          // Front padding.\n          skippedPickedCounter = frontPadding;\n          if (tve.getId() >= segmentBuildInfo.getStartTweetId()) {\n            indexTweet = true;\n          }\n        } else if (firstTweetIdInNextSegment.isPresent()\n            && record.offset() > segmentBuildInfo.getTweetEndOffset() - marginSize) {\n          // Back padding.\n          skippedPickedCounter = backPadding;\n          if (tve.getId() < firstTweetIdInNextSegment.get()) {\n            indexTweet = true;\n          }\n        } else {\n          skippedPickedCounter = regular;\n          // These we just pick. A tweet that came very late can end up in the wrong\n          // segment, but it's better for it to be present in a segment than dropped.\n          indexTweet = true;\n        }\n\n        if (indexTweet) {\n          skippedPickedCounter.incrementPicked();\n          segmentWriter.indexThriftVersionedEvents(tve);\n          maxIndexedTweetId = Math.max(maxIndexedTweetId, tve.getId());\n          indexedEvents++;\n\n          // Note that records don't necessarily have increasing timestamps.\n          // Why? The timestamps whatever timestamp we picked when creating the record\n          // in ingesters and there are many ingesters.\n          minIndexedTimestampMs = Math.min(minIndexedTimestampMs, record.timestamp());\n          maxIndexedTimestampMs = Math.max(maxIndexedTimestampMs, record.timestamp());\n        } else {\n          skippedPickedCounter.incrementSkipped();\n        }\n        totalRead++;\n\n        if (record.offset() >= endReadingAtOffset) {\n          break;\n        }\n      }\n      indexTimer.stop();\n    } while (!done);\n\n    tweetsKafkaConsumer.close();\n\n    SegmentTweetsIndexingResult result = new SegmentTweetsIndexingResult(\n        minIndexedTimestampMs, maxIndexedTimestampMs, maxIndexedTweetId, segmentWriter);\n\n    LOG.info(\"Finished indexing {} tweets for {} in {}. Read {} tweets. Result: {}.\"\n            + \" Time polling: {}, Time indexing: {}.\",\n        indexedEvents, consumerClientId, stopwatch, totalRead, result,\n        pollTimer, indexTimer);\n\n    // In normal conditions, expect to pick just a few in front and in the back.\n    LOG.info(\"SkippedPicked ({}) -- {}, {}, {}, {}, {}\",\n        consumerClientId, frontMargin, frontPadding, backPadding, backMargin, regular);\n\n    return result;\n  }\n\n\n  /**\n   * After indexing all the tweets for a segment, index updates that need to be applied before\n   * the segment is optimized.\n   *\n   * This is required because some updates (URL updates, cards and Named Entities) can only be\n   * applied to an unoptimized segment. Luckily, all of these updates should arrive close to when\n   * the Tweet is created.\n   */\n  private KafkaOffsetPair findUpdateStreamOffsetRange(\n      SegmentTweetsIndexingResult tweetsIndexingResult) {\n    KafkaConsumer<Long, ThriftVersionedEvents> offsetsConsumer =\n        earlybirdKafkaConsumersFactory.createKafkaConsumer(\n            \"consumer_for_update_offsets_\" + segmentBuildInfo.getIndex());\n\n    // Start one minute before the first indexed tweet. One minute is excessive, but\n    // we need to start a bit earlier in case the first tweet we indexed came in\n    // later than some of its updates.\n    long updatesStartOffset = offsetForTime(offsetsConsumer, updateTopic,\n        tweetsIndexingResult.getMinRecordTimestampMs() - Duration.ofMinutes(1).toMillis());\n\n    // Two cases:\n    //\n    // 1. If we're not indexing the last segment, end 10 minutes after the last tweet. So for\n    //    example if we resolve an url in a tweet 3 minutes after the tweet is published,\n    //    we'll apply that update before the segment is optimized. 10 minutes is a bit too\n    //    much, but that doesn't matter a whole lot, since we're indexing about ~10 hours of\n    //    updates.\n    //\n    // 2. If we're indexing the last segment, end a bit before the last indexed tweet. We might\n    //    have incoming tweets that are a bit late. In fresh startup, we don't have a mechanism\n    //    to store these tweets to be applied when the tweet arrives, as in TweetUpdateHandler,\n    //    so just stop a bit earlier and let TweetCreateHandler and TweetUpdateHandler deal with\n    //    that.\n    long millisAdjust;\n    if (segmentBuildInfo.getIndex() == segmentBuildInfos.size() - 1) {\n      millisAdjust = -Duration.ofMinutes(1).toMillis();\n    } else {\n      millisAdjust = Duration.ofMinutes(10).toMillis();\n    }\n    long updatesEndOffset = offsetForTime(offsetsConsumer, updateTopic,\n        tweetsIndexingResult.getMaxRecordTimestampMs() + millisAdjust);\n\n    offsetsConsumer.close();\n\n    return new KafkaOffsetPair(updatesStartOffset, updatesEndOffset);\n  }\n\n  /**\n   * Get the earliest offset with a timestamp >= $timestamp.\n   *\n   * The guarantee we get is that if we start reading from here on, we will get\n   * every single message that came in with a timestamp >= $timestamp.\n   */\n  private long offsetForTime(KafkaConsumer<Long, ThriftVersionedEvents> kafkaConsumer,\n                             TopicPartition partition,\n                             long timestamp) {\n    Preconditions.checkNotNull(kafkaConsumer);\n    Preconditions.checkNotNull(partition);\n\n    OffsetAndTimestamp offsetAndTimestamp = kafkaConsumer\n        .offsetsForTimes(ImmutableMap.of(partition, timestamp))\n        .get(partition);\n    if (offsetAndTimestamp == null) {\n      return -1;\n    } else {\n      return offsetAndTimestamp.offset();\n    }\n  }\n\n  private void indexUpdatesFromStream(\n      TopicPartition topicPartition,\n      String consumerClientId,\n      long startOffset,\n      long endOffset,\n      SegmentWriter segmentWriter) throws IOException {\n    KafkaConsumer<Long, ThriftVersionedEvents> kafkaConsumer =\n        makeKafkaConsumerForIndexing(consumerClientId, topicPartition, startOffset);\n\n    // Index TVEs.\n    boolean done = false;\n\n    Stopwatch pollTimer = Stopwatch.createUnstarted();\n    Stopwatch indexTimer = Stopwatch.createUnstarted();\n\n    SkippedPickedCounter updatesSkippedPicked = new SkippedPickedCounter(\"streamed_updates\");\n    IndexingResultCounts indexingResultCounts = new IndexingResultCounts();\n\n    long segmentTimesliceId = segmentWriter.getSegmentInfo().getTimeSliceID();\n\n    Stopwatch totalTime = Stopwatch.createStarted();\n\n    do {\n      pollTimer.start();\n      ConsumerRecords<Long, ThriftVersionedEvents> records =\n          kafkaConsumer.poll(Duration.ofSeconds(1));\n      pollTimer.stop();\n\n      indexTimer.start();\n      for (ConsumerRecord<Long, ThriftVersionedEvents> record : records) {\n        if (record.value().getId() < segmentTimesliceId) {\n          // Doesn't apply to this segment, can be skipped instead of skipping it\n          // inside the more costly segmentWriter.indexThriftVersionedEvents call.\n          updatesSkippedPicked.incrementSkipped();\n        } else {\n          if (record.offset() >= endOffset) {\n            done = true;\n          }\n\n          updatesSkippedPicked.incrementPicked();\n          indexingResultCounts.countResult(\n              segmentWriter.indexThriftVersionedEvents(record.value()));\n        }\n\n        if (record.offset() >= endOffset) {\n          break;\n        }\n      }\n      indexTimer.stop();\n    } while (!done);\n\n    // Note that there'll be a decent amount of failed retryable updates. Since we index\n    // updates in a range that's a bit wider, they can't be applied here.\n    LOG.info(\"Client: {}, Finished indexing updates: {}. \"\n            + \"Times -- total: {}. polling: {}, indexing: {}. Indexing result counts: {}\",\n        consumerClientId, updatesSkippedPicked,\n        totalTime, pollTimer, indexTimer, indexingResultCounts);\n  }\n\n  /**\n   * Make a consumer that reads from a single partition, starting at some offset.\n   */\n  private KafkaConsumer<Long, ThriftVersionedEvents> makeKafkaConsumerForIndexing(\n      String consumerClientId,\n      TopicPartition topicPartition,\n      long offset) {\n    KafkaConsumer<Long, ThriftVersionedEvents> kafkaConsumer =\n        earlybirdKafkaConsumersFactory.createKafkaConsumer(consumerClientId);\n    kafkaConsumer.assign(ImmutableList.of(topicPartition));\n    kafkaConsumer.seek(topicPartition, offset);\n    LOG.info(\"Indexing TVEs. Kafka consumer: {}\", consumerClientId);\n    return kafkaConsumer;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/freshstartup/SegmentBuildInfo.java",
    "content": "package com.twitter.search.earlybird.partition.freshstartup;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.earlybird.partition.SegmentWriter;\n\n// Data collected and produced while building a segment.\nclass SegmentBuildInfo {\n  private static final Logger LOG = LoggerFactory.getLogger(SegmentBuildInfo.class);\n\n  // Inclusive boundaries. [start, end].\n  private final long tweetStartOffset;\n  private final long tweetEndOffset;\n  private final int index;\n  private final boolean lastSegment;\n\n  private long startTweetId;\n  private long maxIndexedTweetId;\n  private KafkaOffsetPair updateKafkaOffsetPair;\n  private SegmentWriter segmentWriter;\n\n  public SegmentBuildInfo(long tweetStartOffset,\n                          long tweetEndOffset,\n                          int index,\n                          boolean lastSegment) {\n    this.tweetStartOffset = tweetStartOffset;\n    this.tweetEndOffset = tweetEndOffset;\n    this.index = index;\n    this.lastSegment = lastSegment;\n\n    this.startTweetId = -1;\n    this.updateKafkaOffsetPair = null;\n    this.maxIndexedTweetId = -1;\n    this.segmentWriter = null;\n  }\n\n  public void setUpdateKafkaOffsetPair(KafkaOffsetPair updateKafkaOffsetPair) {\n    this.updateKafkaOffsetPair = updateKafkaOffsetPair;\n  }\n\n  public KafkaOffsetPair getUpdateKafkaOffsetPair() {\n    return updateKafkaOffsetPair;\n  }\n\n  public boolean isLastSegment() {\n    return lastSegment;\n  }\n\n  public void setStartTweetId(long startTweetId) {\n    this.startTweetId = startTweetId;\n  }\n\n  public long getTweetStartOffset() {\n    return tweetStartOffset;\n  }\n\n  public long getTweetEndOffset() {\n    return tweetEndOffset;\n  }\n\n  public long getStartTweetId() {\n    return startTweetId;\n  }\n\n  public int getIndex() {\n    return index;\n  }\n\n  public void setMaxIndexedTweetId(long maxIndexedTweetId) {\n    this.maxIndexedTweetId = maxIndexedTweetId;\n  }\n\n  public long getMaxIndexedTweetId() {\n    return maxIndexedTweetId;\n  }\n\n  public SegmentWriter getSegmentWriter() {\n    return segmentWriter;\n  }\n\n  public void setSegmentWriter(SegmentWriter segmentWriter) {\n    this.segmentWriter = segmentWriter;\n  }\n\n  public void logState() {\n    LOG.info(\"SegmentBuildInfo (index:{})\", index);\n    LOG.info(String.format(\"  Start offset: %,d\", tweetStartOffset));\n    LOG.info(String.format(\"  End offset: %,d\", tweetEndOffset));\n    LOG.info(String.format(\"  Start tweet id: %d\", startTweetId));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/freshstartup/SegmentTweetsIndexingResult.java",
    "content": "package com.twitter.search.earlybird.partition.freshstartup;\n\nimport com.twitter.search.earlybird.partition.SegmentWriter;\n\n/**\n   * Data collected and created while indexing tweets for a single segment.\n   */\nclass SegmentTweetsIndexingResult {\n  private final long minRecordTimestampMs;\n  private final long maxRecordTimestampMs;\n  private final long maxIndexedTweetId;\n  private final SegmentWriter segmentWriter;\n\n  public SegmentTweetsIndexingResult(long minRecordTimestampMs, long maxRecordTimestampMs,\n                                     long maxIndexedTweetId,\n                                     SegmentWriter segmentWriter) {\n    this.minRecordTimestampMs = minRecordTimestampMs;\n    this.maxRecordTimestampMs = maxRecordTimestampMs;\n    this.maxIndexedTweetId = maxIndexedTweetId;\n    this.segmentWriter = segmentWriter;\n  }\n\n  public long getMinRecordTimestampMs() {\n    return minRecordTimestampMs;\n  }\n\n  public long getMaxRecordTimestampMs() {\n    return maxRecordTimestampMs;\n  }\n\n  public SegmentWriter getSegmentWriter() {\n    return segmentWriter;\n  }\n\n  public long getMaxIndexedTweetId() {\n    return maxIndexedTweetId;\n  }\n\n  @Override\n  public String toString() {\n    return String.format(\"Start time: %d, end time: %d, segment name: %s, max indexed: %d\",\n        minRecordTimestampMs, maxRecordTimestampMs,\n        segmentWriter.getSegmentInfo().getSegmentName(),\n        maxIndexedTweetId);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/partition/freshstartup/SkippedPickedCounter.java",
    "content": "package com.twitter.search.earlybird.partition.freshstartup;\n\nclass SkippedPickedCounter {\n  private long skipped;\n  private long picked;\n  private String name;\n\n  public SkippedPickedCounter(String name) {\n    this.skipped = 0;\n    this.picked = 0;\n    this.name = name;\n  }\n\n  @Override\n  public String toString() {\n    return String.format(\"[%s - picked: %,d, skipped: %,d]\",\n        name, picked, skipped);\n  }\n\n  void incrementSkipped() {\n    skipped++;\n  }\n  void incrementPicked() {\n    picked++;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/querycache/CachedFilterQuery.java",
    "content": "package com.twitter.search.earlybird.querycache;\n\nimport java.io.IOException;\nimport java.util.Objects;\nimport java.util.Set;\n\nimport org.apache.lucene.index.IndexReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.ConstantScoreScorer;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.Explanation;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.query.DefaultFilterWeight;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.QueryCacheResultForSegment;\n\n/**\n * Query to iterate QueryCache result (the cache)\n */\npublic final class CachedFilterQuery extends Query {\n  private static final String STAT_PREFIX = \"querycache_serving_\";\n  private static final SearchCounter REWRITE_CALLS = SearchCounter.export(\n      STAT_PREFIX + \"rewrite_calls\");\n  private static final SearchCounter NO_CACHE_FOUND = SearchCounter.export(\n      STAT_PREFIX + \"no_cache_found\");\n  private static final SearchCounter USED_CACHE_AND_FRESH_DOCS = SearchCounter.export(\n      STAT_PREFIX + \"used_cache_and_fresh_docs\");\n  private static final SearchCounter USED_CACHE_ONLY = SearchCounter.export(\n      STAT_PREFIX + \"used_cache_only\");\n\n\n  public static class NoSuchFilterException extends Exception {\n    NoSuchFilterException(String filterName) {\n      super(\"Filter [\" + filterName + \"] does not exists\");\n    }\n  }\n\n  private static class CachedResultQuery extends Query {\n    private final QueryCacheResultForSegment cachedResult;\n\n    public CachedResultQuery(QueryCacheResultForSegment cachedResult) {\n      this.cachedResult = cachedResult;\n    }\n\n    @Override\n    public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n      return new DefaultFilterWeight(this) {\n        @Override\n        protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context)\n            throws IOException {\n          return cachedResult.getDocIdSet().iterator();\n        }\n      };\n    }\n\n    @Override\n    public int hashCode() {\n      return cachedResult == null ? 0 : cachedResult.hashCode();\n    }\n\n    @Override\n    public boolean equals(Object obj) {\n      if (!(obj instanceof CachedResultQuery)) {\n        return false;\n      }\n\n      CachedResultQuery query = (CachedResultQuery) obj;\n      return Objects.equals(cachedResult, query.cachedResult);\n    }\n\n    @Override\n    public String toString(String field) {\n      return \"CACHED_RESULT\";\n    }\n  }\n\n  private static class CachedResultAndFreshDocsQuery extends Query {\n    private final Query cacheLuceneQuery;\n    private final QueryCacheResultForSegment cachedResult;\n\n    public CachedResultAndFreshDocsQuery(\n        Query cacheLuceneQuery, QueryCacheResultForSegment cachedResult) {\n      this.cacheLuceneQuery = cacheLuceneQuery;\n      this.cachedResult = cachedResult;\n    }\n\n    @Override\n    public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n      return new Weight(this) {\n        @Override\n        public void extractTerms(Set<Term> terms) {\n        }\n\n        @Override\n        public Explanation explain(LeafReaderContext context, int doc) throws IOException {\n          Scorer scorer = scorer(context);\n          if ((scorer != null) && (scorer.iterator().advance(doc) == doc)) {\n            return Explanation.match(0f, \"Match on id \" + doc);\n          }\n          return Explanation.match(0f, \"No match on id \" + doc);\n        }\n\n        @Override\n        public Scorer scorer(LeafReaderContext context) throws IOException {\n          Weight luceneWeight;\n          try  {\n            luceneWeight = cacheLuceneQuery.createWeight(searcher, scoreMode, boost);\n          } catch (UnsupportedOperationException e) {\n            // Some queries do not support weights. This is fine, it simply means the query has\n            // no docs, and means the same thing as a null scorer.\n            return null;\n          }\n\n          Scorer luceneScorer = luceneWeight.scorer(context);\n          if (luceneScorer == null) {\n            return null;\n          }\n\n          DocIdSetIterator iterator = new CachedResultDocIdSetIterator(\n              cachedResult.getSmallestDocID(),\n              luceneScorer.iterator(),\n              cachedResult.getDocIdSet().iterator());\n          return new ConstantScoreScorer(luceneWeight, 0.0f, scoreMode, iterator);\n        }\n\n        @Override\n        public boolean isCacheable(LeafReaderContext ctx) {\n          return true;\n        }\n      };\n    }\n\n    @Override\n    public int hashCode() {\n      return (cacheLuceneQuery == null ? 0 : cacheLuceneQuery.hashCode()) * 13\n          + (cachedResult == null ? 0 : cachedResult.hashCode());\n    }\n\n    @Override\n    public boolean equals(Object obj) {\n      if (!(obj instanceof CachedResultAndFreshDocsQuery)) {\n        return false;\n      }\n\n      CachedResultAndFreshDocsQuery query = (CachedResultAndFreshDocsQuery) obj;\n      return Objects.equals(cacheLuceneQuery, query.cacheLuceneQuery)\n          && Objects.equals(cachedResult, query.cachedResult);\n    }\n\n    @Override\n    public String toString(String field) {\n      return \"CACHED_RESULT_AND_FRESH_DOCS\";\n    }\n  }\n\n  private static final Query DUMMY_FILTER = wrapFilter(new Query() {\n    @Override\n    public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n      return new DefaultFilterWeight(this) {\n        @Override\n        protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context) {\n          return null;\n        }\n      };\n    }\n\n    @Override\n    public int hashCode() {\n      return System.identityHashCode(this);\n    }\n\n    @Override\n    public boolean equals(Object obj) {\n      return this == obj;\n    }\n\n    @Override\n    public String toString(String field) {\n      return \"DUMMY_FILTER\";\n    }\n  });\n\n  private final QueryCacheFilter queryCacheFilter;\n\n  // Lucene Query used to fill the cache\n  private final Query cacheLuceneQuery;\n\n  public static Query getCachedFilterQuery(String filterName, QueryCacheManager queryCacheManager)\n      throws NoSuchFilterException {\n    return wrapFilter(new CachedFilterQuery(filterName, queryCacheManager));\n  }\n\n  private static Query wrapFilter(Query filter) {\n    return new BooleanQuery.Builder()\n        .add(filter, BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  private CachedFilterQuery(String filterName, QueryCacheManager queryCacheManager)\n      throws NoSuchFilterException {\n    queryCacheFilter = queryCacheManager.getFilter(filterName);\n    if (queryCacheFilter == null) {\n      throw new NoSuchFilterException(filterName);\n    }\n    queryCacheFilter.incrementUsageStat();\n\n    // retrieve the query that was used to populate the cache\n    cacheLuceneQuery = queryCacheFilter.getLuceneQuery();\n  }\n\n  /**\n   * Creates a query base on the cache situation\n   */\n  @Override\n  public Query rewrite(IndexReader reader) {\n    EarlybirdIndexSegmentAtomicReader twitterReader = (EarlybirdIndexSegmentAtomicReader) reader;\n    QueryCacheResultForSegment cachedResult =\n        twitterReader.getSegmentData().getQueryCacheResult(queryCacheFilter.getFilterName());\n    REWRITE_CALLS.increment();\n\n    if (cachedResult == null || cachedResult.getSmallestDocID() == -1) {\n      // No cached result, or cache has never been updated\n      // This happens to the newly created segment, between the segment creation and first\n      // query cache update\n      NO_CACHE_FOUND.increment();\n\n      if (queryCacheFilter.getCacheModeOnly()) {\n        // since this query cache filter allows cache mode only, we return a query that\n        // matches no doc\n        return DUMMY_FILTER;\n      }\n\n      return wrapFilter(cacheLuceneQuery);\n    }\n\n    if (!queryCacheFilter.getCacheModeOnly() && // is this a cache mode only filter?\n        // the following check is only necessary for the realtime segment, which\n        // grows. Since we decrement docIds in the realtime segment, a reader\n        // having a smallestDocID less than the one in the cachedResult indicates\n        // that the segment/reader has new documents.\n        cachedResult.getSmallestDocID() > twitterReader.getSmallestDocID()) {\n      // The segment has more documents than the cached result. IOW, there are new\n      // documents that are not cached. This happens to latest segment that we're indexing to.\n      USED_CACHE_AND_FRESH_DOCS.increment();\n      return wrapFilter(new CachedResultAndFreshDocsQuery(cacheLuceneQuery, cachedResult));\n    }\n\n    // The segment has not grown since the cache was last updated.\n    // This happens mostly to old segments that we're no longer indexing to.\n    USED_CACHE_ONLY.increment();\n    return wrapFilter(new CachedResultQuery(cachedResult));\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost)\n      throws IOException {\n    final Weight luceneWeight = cacheLuceneQuery.createWeight(searcher, scoreMode, boost);\n\n    return new Weight(this) {\n      @Override\n      public Scorer scorer(LeafReaderContext context) throws IOException {\n        return luceneWeight.scorer(context);\n      }\n\n      @Override\n      public void extractTerms(Set<Term> terms) {\n        luceneWeight.extractTerms(terms);\n      }\n\n      @Override\n      public Explanation explain(LeafReaderContext context, int doc) throws IOException {\n        return luceneWeight.explain(context, doc);\n      }\n\n      @Override\n      public boolean isCacheable(LeafReaderContext ctx) {\n        return luceneWeight.isCacheable(ctx);\n      }\n    };\n  }\n\n  @Override\n  public int hashCode() {\n    return cacheLuceneQuery == null ? 0 : cacheLuceneQuery.hashCode();\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof CachedFilterQuery)) {\n      return false;\n    }\n\n    CachedFilterQuery filter = (CachedFilterQuery) obj;\n    return Objects.equals(cacheLuceneQuery, filter.cacheLuceneQuery);\n  }\n\n  @Override\n  public String toString(String s) {\n    return \"CachedFilterQuery[\" + queryCacheFilter.getFilterName() + \"]\";\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/querycache/CachedResultDocIdSetIterator.java",
    "content": "package com.twitter.search.earlybird.querycache;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.search.DocIdSetIterator;\n\npublic class CachedResultDocIdSetIterator extends DocIdSetIterator {\n  // With the realtime index, we grow the doc id negatively.\n  // Hence the smallest doc id is the ID the latest/newest document in the cache.\n  private final int cachedSmallestDocID;\n\n  // Documents that were indexed after the last cache update\n  private final DocIdSetIterator freshDocIdIterator;\n  // Documents that were cached\n  private final DocIdSetIterator cachedDocIdIterator;\n\n  private int currentDocId;\n  private boolean initialized = false;\n\n  public CachedResultDocIdSetIterator(int cachedSmallestDocID,\n                                      DocIdSetIterator freshDocIdIterator,\n                                      DocIdSetIterator cachedDocIdIterator) {\n    this.cachedSmallestDocID = cachedSmallestDocID;\n\n    this.freshDocIdIterator = freshDocIdIterator;\n    this.cachedDocIdIterator = cachedDocIdIterator;\n    this.currentDocId = -1;\n  }\n\n  @Override\n  public int docID() {\n    return currentDocId;\n  }\n\n  @Override\n  public int nextDoc() throws IOException {\n    if (currentDocId < cachedSmallestDocID) {\n      currentDocId = freshDocIdIterator.nextDoc();\n    } else if (currentDocId != NO_MORE_DOCS) {\n      if (!initialized) {\n        // the first time we come in here, currentDocId should be pointing to\n        // something >= cachedMinDocID. We need to go to the doc after cachedMinDocID.\n        currentDocId = cachedDocIdIterator.advance(currentDocId + 1);\n        initialized = true;\n      } else {\n        currentDocId = cachedDocIdIterator.nextDoc();\n      }\n    }\n    return currentDocId;\n  }\n\n  @Override\n  public int advance(int target) throws IOException {\n    if (target < cachedSmallestDocID) {\n      currentDocId = freshDocIdIterator.advance(target);\n    } else if (currentDocId != NO_MORE_DOCS) {\n      initialized = true;\n      currentDocId = cachedDocIdIterator.advance(target);\n    }\n\n    return currentDocId;\n  }\n\n  @Override\n  public long cost() {\n    if (currentDocId < cachedSmallestDocID) {\n      return freshDocIdIterator.cost();\n    } else {\n      return cachedDocIdIterator.cost();\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/querycache/QueryCacheConfig.java",
    "content": "package com.twitter.search.earlybird.querycache;\n\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.Reader;\nimport java.util.ArrayList;\nimport java.util.List;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport org.yaml.snakeyaml.TypeDescription;\nimport org.yaml.snakeyaml.Yaml;\nimport org.yaml.snakeyaml.constructor.Constructor;\n\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\n\n// QueryCacheConfig is not thread safe. *Do not* attempt to create multiple QueryCacheConfig\n// in different threads\npublic class QueryCacheConfig {\n  private static final Logger LOG = LoggerFactory.getLogger(QueryCacheConfig.class);\n  private static final String DEFAULT_CONFIG_FILE = \"querycache.yml\";\n  private final SearchStatsReceiver statsReceiver;\n\n  private List<QueryCacheFilter> filters;\n\n  public QueryCacheConfig(SearchStatsReceiver statsReceiver) {\n    this(locateConfigFile(EarlybirdConfig.getString(\"query_cache_config_file_name\",\n                                                    DEFAULT_CONFIG_FILE)), statsReceiver);\n  }\n\n  // package protected constructor for unit test only\n  QueryCacheConfig(Reader reader, SearchStatsReceiver statsReceiver) {\n    this.statsReceiver = statsReceiver;\n    if (reader == null) {\n      throw new RuntimeException(\"Query cache config not loaded\");\n    }\n    loadConfig(reader);\n  }\n\n  public List<QueryCacheFilter> filters() {\n    return filters;\n  }\n\n  int getFilterSize() {\n    return filters.size();\n  }\n\n  private static FileReader locateConfigFile(String configFileName) {\n    File configFile = null;\n    String dir = Config.locateSearchConfigDir(EarlybirdConfig.EARLYBIRD_CONFIG_DIR, configFileName);\n    if (dir != null) {\n      configFile = openConfigFile(dir + \"/\" + configFileName);\n    }\n    if (configFile != null) {\n      try {\n        return new FileReader(configFile);\n      } catch (FileNotFoundException e) {\n        // This should not happen as the caller should make sure that the file exists before\n        // calling this function.\n        LOG.error(\"Unexpected exception\", e);\n        throw new RuntimeException(\"Query cache config file not loaded!\", e);\n      }\n    }\n    return null;\n  }\n\n  private static File openConfigFile(String configFilePath) {\n    File configFile = new File(configFilePath);\n    if (!configFile.exists()) {\n      LOG.warn(\"QueryCache config file [\" + configFile + \"] not found\");\n      configFile = null;\n    } else {\n      LOG.info(\"Opened QueryCacheFilter config file [\" + configFile + \"]\");\n    }\n    return configFile;\n  }\n\n  private void loadConfig(Reader reader) {\n    TypeDescription qcEntryDescription = new TypeDescription(QueryCacheFilter.class);\n    Constructor constructor = new Constructor(qcEntryDescription);\n    Yaml yaml = new Yaml(constructor);\n\n    filters = new ArrayList<>();\n\n    for (Object data : yaml.loadAll(reader)) {\n      QueryCacheFilter cacheFilter = (QueryCacheFilter) data;\n      try {\n        cacheFilter.sanityCheck();\n      } catch (QueryCacheFilter.InvalidEntryException e) {\n        throw new RuntimeException(e);\n      }\n      cacheFilter.createQueryCounter(statsReceiver);\n      filters.add(cacheFilter);\n      LOG.info(\"Loaded filter from config {}\", cacheFilter.toString());\n    }\n    LOG.info(\"Total filters loaded: {}\", filters.size());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/querycache/QueryCacheConversionRules.java",
    "content": "package com.twitter.search.earlybird.querycache;\n\nimport java.util.Arrays;\nimport java.util.List;\nimport java.util.Set;\n\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.Sets;\n\nimport com.twitter.search.common.constants.QueryCacheConstants;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\nimport com.twitter.search.queryparser.query.search.SearchOperatorConstants;\n\nimport static com.twitter.search.common.util.RuleBasedConverter.Rule;\n\n/**\n * Rules to convert exclude operators into cached filters and consolidate them.\n * NOTE: this is copied from blender/core/parser/service/queryparser/QueryCacheConversionRules.java\n * We should remove the blender one once this is in production.\n */\npublic final class QueryCacheConversionRules {\n  static final SearchOperator EXCLUDE_ANTISOCIAL =\n      new SearchOperator(SearchOperator.Type.EXCLUDE, SearchOperatorConstants.ANTISOCIAL);\n  static final SearchOperator EXCLUDE_SPAM =\n      new SearchOperator(SearchOperator.Type.EXCLUDE, SearchOperatorConstants.SPAM);\n  static final SearchOperator EXCLUDE_REPLIES =\n      new SearchOperator(SearchOperator.Type.EXCLUDE, SearchOperatorConstants.REPLIES);\n  static final SearchOperator EXCLUDE_NATIVERETWEETS =\n      new SearchOperator(SearchOperator.Type.EXCLUDE, SearchOperatorConstants.NATIVE_RETWEETS);\n\n  public static final SearchOperator CACHED_EXCLUDE_ANTISOCIAL =\n      new SearchOperator(SearchOperator.Type.CACHED_FILTER,\n                         QueryCacheConstants.EXCLUDE_ANTISOCIAL);\n  static final SearchOperator CACHED_EXCLUDE_NATIVERETWEETS =\n      new SearchOperator(SearchOperator.Type.CACHED_FILTER,\n                         QueryCacheConstants.EXCLUDE_ANTISOCIAL_AND_NATIVERETWEETS);\n  static final SearchOperator CACHED_EXCLUDE_SPAM =\n      new SearchOperator(SearchOperator.Type.CACHED_FILTER,\n                         QueryCacheConstants.EXCLUDE_SPAM);\n  static final SearchOperator CACHED_EXCLUDE_SPAM_AND_NATIVERETWEETS =\n      new SearchOperator(SearchOperator.Type.CACHED_FILTER,\n                         QueryCacheConstants.EXCLUDE_SPAM_AND_NATIVERETWEETS);\n  static final SearchOperator CACHED_EXCLUDE_REPLIES =\n      new SearchOperator(SearchOperator.Type.CACHED_FILTER,\n                         QueryCacheConstants.EXCLUDE_REPLIES);\n\n  private QueryCacheConversionRules() {\n  }\n\n  public static final List<Rule<Query>> DEFAULT_RULES = ImmutableList.of(\n      // basic translation from exclude:filter to cached filter\n      new Rule<>(new Query[]{EXCLUDE_ANTISOCIAL},\n                 new Query[]{CACHED_EXCLUDE_ANTISOCIAL}),\n\n      new Rule<>(new Query[]{EXCLUDE_SPAM},\n                 new Query[]{CACHED_EXCLUDE_SPAM}),\n\n      new Rule<>(new Query[]{EXCLUDE_NATIVERETWEETS},\n                 new Query[]{CACHED_EXCLUDE_NATIVERETWEETS}),\n\n      new Rule<>(new Query[]{EXCLUDE_REPLIES},\n                 new Query[]{CACHED_EXCLUDE_REPLIES}),\n\n      // combine two cached filter to a new one\n      new Rule<>(new Query[]{CACHED_EXCLUDE_SPAM, CACHED_EXCLUDE_NATIVERETWEETS},\n                 new Query[]{CACHED_EXCLUDE_SPAM_AND_NATIVERETWEETS}),\n\n      // Remove redundant filters. A cached filter is redundant when it coexist with a\n      // more strict filter. Note all the filter will filter out antisocial.\n      new Rule<>(\n          new Query[]{CACHED_EXCLUDE_SPAM, CACHED_EXCLUDE_ANTISOCIAL},\n          new Query[]{CACHED_EXCLUDE_SPAM}),\n\n      new Rule<>(\n          new Query[]{CACHED_EXCLUDE_NATIVERETWEETS, CACHED_EXCLUDE_ANTISOCIAL},\n          new Query[]{CACHED_EXCLUDE_NATIVERETWEETS}),\n\n      new Rule<>(\n          new Query[]{CACHED_EXCLUDE_SPAM_AND_NATIVERETWEETS, CACHED_EXCLUDE_ANTISOCIAL},\n          new Query[]{CACHED_EXCLUDE_SPAM_AND_NATIVERETWEETS}),\n\n      new Rule<>(\n          new Query[]{CACHED_EXCLUDE_SPAM_AND_NATIVERETWEETS, CACHED_EXCLUDE_SPAM},\n          new Query[]{CACHED_EXCLUDE_SPAM_AND_NATIVERETWEETS}),\n\n      new Rule<>(\n          new Query[]{CACHED_EXCLUDE_SPAM_AND_NATIVERETWEETS, CACHED_EXCLUDE_NATIVERETWEETS},\n          new Query[]{CACHED_EXCLUDE_SPAM_AND_NATIVERETWEETS})\n  );\n\n  public static final List<Query> STRIP_ANNOTATIONS_QUERIES;\n  static {\n    Set<Query> stripAnnotationsQueries = Sets.newHashSet();\n    for (Rule<Query> rule : DEFAULT_RULES) {\n      stripAnnotationsQueries.addAll(Arrays.asList(rule.getSources()));\n    }\n    STRIP_ANNOTATIONS_QUERIES = ImmutableList.copyOf(stripAnnotationsQueries);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/querycache/QueryCacheFilter.java",
    "content": "package com.twitter.search.earlybird.querycache;\n\nimport java.util.List;\nimport java.util.TreeMap;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.search.Query;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.common.quantity.Amount;\nimport com.twitter.common.quantity.Time;\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.query.thriftjava.CollectorParams;\nimport com.twitter.search.common.query.thriftjava.CollectorTerminationParams;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.common.util.text.regex.Regex;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.queryparser.EarlybirdLuceneQueryVisitor;\nimport com.twitter.search.earlybird.search.SearchRequestInfo;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.queryparser.parser.SerializedQueryParser;\nimport com.twitter.search.queryparser.query.QueryParserException;\n\n/**\n * The definition of a QueryCache filter/entry, like the name of the filter, the query used\n * to populate the cache, update schedule, etc..\n *\n * Instances of this class are created by the YAML loader when loading the config file. Most\n * members are populated by YAML using setters through reflection.\n */\npublic class QueryCacheFilter {\n  // Data structure type supported as cache result holder\n  public enum ResultSetType {\n    FixedBitSet,\n    SparseFixedBitSet\n  }\n\n  // Fields set directly from YML config file.\n  private String filterName;           // unique name for cached filter\n  private String query;                // serialized query string\n  private ResultSetType resultType;\n  private boolean cacheModeOnly;\n  private List<UpdateInterval> schedule;\n  private SearchCounter queries;\n\n  // Fields generated based on config (but not directly).\n  private volatile Pair<ThriftSearchQuery, Query> queryPair;\n  private TreeMap<Integer, UpdateInterval> scheduleMap;  // tree map from index to interval\n\n  public class InvalidEntryException extends Exception {\n    public InvalidEntryException(String message) {\n      super(\"Filter [\" + filterName + \"]: \" + message);\n    }\n  }\n\n  public static class UpdateInterval {\n    // Overrides *all* query cache update frequencies to be this value, in seconds.\n    private final int overrideSecondsForTests = EarlybirdConfig.getInt(\n        \"override_query_cache_update_frequency\", -1);\n\n    // Fields set directly from YML config file.\n    private int segment;\n    private long seconds;\n\n    public void setSegment(int segment) {\n      this.segment = segment;\n    }\n\n    /**\n     * Sets the update period in seconds. If the override_query_cache_update_frequency parameter is\n     * specified in the earlybird configuration, its value is used instead (the value passed to this\n     * method is ignored).\n     */\n    public void setSeconds(long seconds) {\n      if (overrideSecondsForTests != -1) {\n        this.seconds = overrideSecondsForTests;\n      } else {\n        this.seconds = seconds;\n      }\n    }\n\n    public int getSegment() {\n      return segment;\n    }\n\n    public long getSeconds() {\n      return seconds;\n    }\n  }\n\n  public void setFilterName(String filterName) throws InvalidEntryException {\n    sanityCheckFilterName(filterName);\n    this.filterName = filterName;\n  }\n\n  /**\n   * Sets the driving query for this query cache filter.\n   */\n  public void setQuery(String query) throws InvalidEntryException {\n    if (query == null || query.isEmpty()) {\n      throw new InvalidEntryException(\"Empty query string\");\n    }\n\n    this.query = query;\n  }\n\n  /**\n   * Sets the type of the results that will be generated by this query cache filter.\n   */\n  public void setResultType(String resultType) throws InvalidEntryException {\n    if (ResultSetType.FixedBitSet.toString().equalsIgnoreCase(resultType)) {\n      this.resultType = ResultSetType.FixedBitSet;\n    } else if (ResultSetType.SparseFixedBitSet.toString().equalsIgnoreCase(resultType)) {\n      this.resultType = ResultSetType.SparseFixedBitSet;\n    } else {\n      throw new InvalidEntryException(\"Unregconized result type [\" + resultType + \"]\");\n    }\n  }\n\n  public void setCacheModeOnly(boolean cacheModeOnly) {\n    this.cacheModeOnly = cacheModeOnly;\n  }\n\n  public void setSchedule(List<UpdateInterval> schedule)\n      throws QueryCacheFilter.InvalidEntryException {\n    sanityCheckSchedule(schedule);\n    this.schedule = schedule;\n    this.scheduleMap = createScheduleMap(schedule);\n  }\n\n  public void createQueryCounter(SearchStatsReceiver statsReceiver) {\n    queries = statsReceiver.getCounter(\"cached_filter_\" + filterName + \"_queries\");\n  }\n\n  public void incrementUsageStat() {\n    queries.increment();\n  }\n\n  public String getFilterName() {\n    return filterName;\n  }\n\n  public String getQueryString() {\n    return query;\n  }\n\n  // snakeyaml does not like a getter named getResultType() that does not return a string\n  public ResultSetType getResultSetType() {\n    return resultType;\n  }\n\n  public boolean getCacheModeOnly() {\n    return cacheModeOnly;\n  }\n\n  public Query getLuceneQuery() {\n    return queryPair.getSecond();\n  }\n\n  public ThriftSearchQuery getSearchQuery() {\n    return queryPair.getFirst();\n  }\n\n  /**\n   * Create a new {@link SearchRequestInfo} using {@link #queryPair}.\n   *\n   * @return a new {@link SearchRequestInfo}\n   */\n  public SearchRequestInfo createSearchRequestInfo() {\n    ThriftSearchQuery searchQuery = Preconditions.checkNotNull(queryPair.getFirst());\n    Query luceneQuery = Preconditions.checkNotNull(queryPair.getSecond());\n\n    return new SearchRequestInfo(\n        searchQuery, luceneQuery, new TerminationTracker(Clock.SYSTEM_CLOCK));\n  }\n\n  public void setup(\n      QueryCacheManager queryCacheManager,\n      UserTable userTable,\n      EarlybirdCluster earlybirdCluster) throws QueryParserException {\n    createQuery(queryCacheManager, userTable, earlybirdCluster);\n  }\n\n  // index corresponds to 'segment' from the config file.  this is the index of the\n  // segment, starting with the current segment (0) and counting backwards in time.\n  public Amount<Long, Time> getUpdateInterval(int index) {\n    long seconds = scheduleMap.floorEntry(index).getValue().getSeconds();\n    return Amount.of(seconds, Time.SECONDS);\n  }\n\n  private TreeMap<Integer, UpdateInterval> createScheduleMap(List<UpdateInterval> scheduleToUse) {\n    TreeMap<Integer, UpdateInterval> map = new TreeMap<>();\n    for (UpdateInterval interval : scheduleToUse) {\n      map.put(interval.segment, interval);\n    }\n    return map;\n  }\n\n  private void createQuery(\n      QueryCacheManager queryCacheManager,\n      UserTable userTable,\n      EarlybirdCluster earlybirdCluster) throws QueryParserException {\n\n    int maxSegmentSize = EarlybirdConfig.getMaxSegmentSize();\n    CollectorParams collectionParams = new CollectorParams();\n    collectionParams.setNumResultsToReturn(maxSegmentSize);\n    CollectorTerminationParams terminationParams = new CollectorTerminationParams();\n    terminationParams.setMaxHitsToProcess(maxSegmentSize);\n    collectionParams.setTerminationParams(terminationParams);\n\n    ThriftSearchQuery searchQuery = new ThriftSearchQuery();\n    searchQuery.setMaxHitsPerUser(maxSegmentSize);\n    searchQuery.setCollectorParams(collectionParams);\n    searchQuery.setSerializedQuery(query);\n\n    final SerializedQueryParser parser = new SerializedQueryParser(\n        EarlybirdConfig.getPenguinVersion());\n\n    Query luceneQuery = parser.parse(query).simplify().accept(\n        new EarlybirdLuceneQueryVisitor(\n            queryCacheManager.getIndexConfig().getSchema().getSchemaSnapshot(),\n            queryCacheManager,\n            userTable,\n            queryCacheManager.getUserScrubGeoMap(),\n            earlybirdCluster,\n            queryCacheManager.getDecider()));\n    if (luceneQuery == null) {\n      throw new QueryParserException(\"Unable to create lucene query from \" + query);\n    }\n\n    queryPair = new Pair<>(searchQuery, luceneQuery);\n  }\n\n  private void sanityCheckFilterName(String filter) throws InvalidEntryException {\n    if (filter == null || filter.isEmpty()) {\n      throw new InvalidEntryException(\"Missing filter name\");\n    }\n    if (Regex.FILTER_NAME_CHECK.matcher(filter).find()) {\n      throw new InvalidEntryException(\n          \"Invalid character in filter name. Chars allowed [a-zA-Z_0-9]\");\n    }\n  }\n\n  private void sanityCheckSchedule(List<UpdateInterval> intervals)\n      throws InvalidEntryException {\n    // Make sure there's at least 1 interval defined\n    if (intervals == null || intervals.isEmpty()) {\n      throw new InvalidEntryException(\"No schedule defined\");\n    }\n\n    // Make sure the first interval starts with segment 0\n    if (intervals.get(0).getSegment() != 0) {\n      throw new InvalidEntryException(\n          \"The first interval in the schedule must start from segment 0\");\n    }\n\n    // Make sure segments are defined in order, and no segment is defined more than twice\n    int prevSegment = intervals.get(0).getSegment();\n    for (int i = 1; i < intervals.size(); ++i) {\n      int currentSegment = intervals.get(i).getSegment();\n      if (prevSegment > currentSegment) {\n        throw new InvalidEntryException(\"Segment intervals out of order. Segment \" + prevSegment\n            + \" is defined before segment \" + currentSegment);\n      }\n\n      if (prevSegment == intervals.get(i).getSegment()) {\n        throw new InvalidEntryException(\"Segment \" + prevSegment + \" is defined twice\");\n      }\n\n      prevSegment = currentSegment;\n    }\n  }\n\n  protected void sanityCheck() throws InvalidEntryException {\n    sanityCheckFilterName(filterName);\n    if (query == null || query.isEmpty()) {\n      throw new InvalidEntryException(\"Missing query\");\n    }\n    if (resultType == null) {\n      throw new InvalidEntryException(\"Missing result type\");\n    }\n    if (schedule == null || schedule.size() == 0) {\n      throw new InvalidEntryException(\"Missing update schedule\");\n    }\n    if (scheduleMap == null || scheduleMap.size() == 0) {\n      throw new InvalidEntryException(\"Missing update schedule map\");\n    }\n  }\n\n  @Override\n  public String toString() {\n    return \"filterName: [\" + getFilterName()\n        + \"] query: [\" + getQueryString()\n        + \"] result type [\" + getResultSetType()\n        + \"] schedule: \" + schedule;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/querycache/QueryCacheManager.java",
    "content": "package com.twitter.search.earlybird.querycache;\n\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Stopwatch;\nimport com.google.common.collect.Lists;\nimport com.google.common.primitives.Longs;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.quantity.Amount;\nimport com.twitter.common.quantity.Time;\nimport com.twitter.common.util.Clock;\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.concurrent.ScheduledExecutorServiceFactory;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.EarlybirdStatus;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.userupdates.UserScrubGeoMap;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentManager;\nimport com.twitter.search.earlybird.partition.SegmentManager.Filter;\nimport com.twitter.search.earlybird.partition.SegmentManager.Order;\nimport com.twitter.search.earlybird.partition.SegmentManager.SegmentUpdateListener;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusCode;\nimport com.twitter.search.queryparser.query.QueryParserException;\n\n/**\n * Main class to manage Earlybird's QueryCache.\n *\n * Initialize the QueryCache and new segments are notified to the QueryCache subsystem\n * through this class.\n *\n * This class is thread-safe when calling methods that modify the list of tasks that\n * we're executing or when we need to traverse all tasks and check something. The way\n * thread-safety is achieved here right now is through making methods synchronized.\n */\npublic class QueryCacheManager implements SegmentUpdateListener {\n  private static final Logger LOG = LoggerFactory.getLogger(QueryCacheManager.class);\n\n  private static final Amount<Long, Time> ZERO_SECONDS = Amount.of(0L, Time.SECONDS);\n\n  private final boolean enabled = EarlybirdConfig.getBool(\"querycache\", false);\n\n  // segments are removed from SegmentInfoMap lazily, and there may be a wait time.\n  // So, beware that there's short period of time where there's more segments than\n  // maxEnabledSegments.\n  private final int maxEnabledSegments;\n\n  private final UserTable userTable;\n  private final UserScrubGeoMap userScrubGeoMap;\n  private final EarlybirdIndexConfig indexConfig;\n  private QueryCacheUpdater updater;\n  private final Map<String, QueryCacheFilter> filters;\n  private final ScheduledExecutorServiceFactory updaterScheduledExecutorServiceFactory;\n\n  private final SearchStatsReceiver searchStatsReceiver;\n\n  private static final SearchLongGauge NUM_CACHE_ENTRY_STAT =\n      SearchLongGauge.export(\"querycache_num_entries\");\n\n  private static final SearchCounter NUM_UPDATE_SEGMENTS_CALLS =\n      SearchCounter.export(\"querycache_num_update_segments_calls\");\n\n  private volatile boolean didSetup = false;\n\n  private final EarlybirdSearcherStats searcherStats;\n  private final Decider decider;\n  private final CriticalExceptionHandler criticalExceptionHandler;\n  private final Clock clock;\n\n  public QueryCacheManager(\n      QueryCacheConfig config,\n      EarlybirdIndexConfig indexConfig,\n      int maxEnabledSegments,\n      UserTable userTable,\n      UserScrubGeoMap userScrubGeoMap,\n      ScheduledExecutorServiceFactory updaterScheduledExecutorServiceFactory,\n      SearchStatsReceiver searchStatsReceiver,\n      EarlybirdSearcherStats searcherStats,\n      Decider decider,\n      CriticalExceptionHandler criticalExceptionHandler,\n      Clock clock) {\n\n    Preconditions.checkArgument(maxEnabledSegments > 0);\n\n    QueryCacheConfig queryCacheConfig = config;\n    if (queryCacheConfig == null) {\n      queryCacheConfig = new QueryCacheConfig(searchStatsReceiver);\n    }\n    this.indexConfig = indexConfig;\n    this.maxEnabledSegments = maxEnabledSegments;\n    this.userTable = userTable;\n    this.userScrubGeoMap = userScrubGeoMap;\n    this.updaterScheduledExecutorServiceFactory = updaterScheduledExecutorServiceFactory;\n    this.searchStatsReceiver = searchStatsReceiver;\n    this.searcherStats = searcherStats;\n    this.filters = new HashMap<>();\n    this.decider = decider;\n    this.criticalExceptionHandler = criticalExceptionHandler;\n    this.clock = clock;\n    for (QueryCacheFilter filter : queryCacheConfig.filters()) {\n      filters.put(filter.getFilterName(), filter);\n    }\n    NUM_CACHE_ENTRY_STAT.set(filters.size());\n  }\n\n  public EarlybirdIndexConfig getIndexConfig() {\n    return indexConfig;\n  }\n\n  public UserScrubGeoMap getUserScrubGeoMap() {\n    return userScrubGeoMap;\n  }\n\n  /** Setup all update tasks at once, should only be called after Earlybird has loaded/indexed all\n   * segments during start-up\n   *\n   * Only the first call to the function has effect, subsequent calls are no-ops\n   */\n  public void setupTasksIfNeeded(SegmentManager segmentManager)\n      throws QueryParserException {\n    setupTasks(\n        segmentManager.getSegmentInfos(Filter.All, Order.OLD_TO_NEW),\n        segmentManager.getEarlybirdIndexConfig().getCluster());\n  }\n\n  @VisibleForTesting\n  synchronized void setupTasks(\n      Iterable<SegmentInfo> newSegments,\n      EarlybirdCluster earlybirdCluster) throws QueryParserException {\n    // Setup needs to be done only once after all index caught up.\n    if (didSetup) {\n      return;\n    }\n\n    LOG.info(\"Setting up {} query cache tasks\", filters.values().size());\n\n    for (QueryCacheFilter filter : filters.values()) {\n      filter.setup(this, userTable, earlybirdCluster);\n    }\n\n    if (!enabled()) {\n      // Note that the definition of disabling the query caches here is \"don't compute the caches\".\n      // We still load the queries from the .yml, we still rewrite search queries to use\n      // cached queries. The reason we are choosing this definition is that it's somewhat simpler\n      // to implement (no need to turn off rewriting) and because we might get external queries that\n      // contain cached filters (they're listed in go/searchsyntax).\n      //\n      // If we need a stricter definition of turning off query caches, we can implement it too, or\n      // just tighten this one.\n      return;\n    }\n\n    Preconditions.checkState(updater == null);\n    updater = new QueryCacheUpdater(\n        filters.values(),\n        updaterScheduledExecutorServiceFactory,\n        userTable,\n        searchStatsReceiver,\n        searcherStats,\n        decider,\n        criticalExceptionHandler,\n        clock);\n\n    LOG.info(\"Finished setting up query cache updater.\");\n\n    scheduleTasks(newSegments, false);\n\n    didSetup = true;\n  }\n\n  private void scheduleTasks(Iterable<SegmentInfo> segments, boolean isCurrent) {\n    List<SegmentInfo> sortedSegments = Lists.newArrayList(segments);\n    Collections.sort(sortedSegments, (o1, o2) -> {\n      // sort new to old (o2 and o1 are reversed here)\n      return Longs.compare(o2.getTimeSliceID(), o1.getTimeSliceID());\n    });\n\n    LOG.info(\"Scheduling tasks for {} segments.\", sortedSegments.size());\n\n    for (int segmentIndex = 0; segmentIndex < sortedSegments.size(); ++segmentIndex) {\n      SegmentInfo segmentInfo = sortedSegments.get(segmentIndex);\n      if (segmentIndex == maxEnabledSegments) {\n        LOG.warn(\"Tried to add more segments than MaxEnabledSegments (\" + maxEnabledSegments\n            + \"). Removed oldest segment \" + segmentInfo.getTimeSliceID());\n        continue;\n      }\n      addQueryCacheTasksForSegment(segmentInfo, segmentIndex, !isCurrent);\n    }\n  }\n\n  /**\n   * Rebuilds the query cache for the given segment after it was optimized.\n   */\n  public synchronized void rebuildQueryCachesAfterSegmentOptimization(\n      SegmentInfo optimizedSegment) {\n    Preconditions.checkState(optimizedSegment.getIndexSegment().isOptimized(),\n                             \"Segment \" + optimizedSegment.getSegmentName() + \" is not optimized.\");\n\n    if (!didSetup) {\n      // Once our indexing is current, we'll just start tasks for all segments, optimized or not.\n      // Before that event, we don't do anything query cache related.\n      LOG.info(\"Haven't done initial setup, returning.\");\n      return;\n    }\n\n    LOG.info(\"Rebuilding query caches for optimized segment {}\",\n        optimizedSegment.getSegmentName());\n\n    // The optimized segment should always be the 1st segment (the current segment has index 0).\n    Stopwatch stopwatch = Stopwatch.createStarted();\n    updater.removeAllTasksForSegment(optimizedSegment);\n    addQueryCacheTasksForSegment(optimizedSegment, 1, true);\n\n    while (!updater.allTasksRanForSegment(optimizedSegment)) {\n      try {\n        Thread.sleep(1000);\n      } catch (InterruptedException e) {\n        // Ignore\n      }\n    }\n\n    LOG.info(\"Rebuilding all query caches for the optimized segment {} took {}.\",\n             optimizedSegment.getSegmentName(), stopwatch);\n  }\n\n  /**\n   * Block until all the tasks inside this manager have ran at least once.\n   */\n  public void waitUntilAllQueryCachesAreBuilt() {\n    LOG.info(\"Waiting until all query caches are built...\");\n\n    Stopwatch stopwatch = Stopwatch.createStarted();\n    while (!allTasksRan()) {\n      try {\n        Thread.sleep(1000);\n      } catch (InterruptedException ex) {\n        Thread.currentThread().interrupt();\n      }\n    }\n\n    LOG.info(\"Ran query cache tasks in: {}\", stopwatch);\n  }\n\n  private void addQueryCacheTasksForSegment(\n      SegmentInfo segmentInfo, int segmentIndex, boolean scheduleImmediately) {\n    LOG.info(\"Adding query cache tasks for segment {}.\", segmentInfo.getTimeSliceID());\n    double updateIntervalMultiplier =\n        EarlybirdConfig.getDouble(\"query_cache_update_interval_multiplier\", 1.0);\n    for (QueryCacheFilter filter : filters.values()) {\n      Amount<Long, Time> updateIntervalFromConfig = filter.getUpdateInterval(segmentIndex);\n      Amount<Long, Time> updateInterval = Amount.of(\n          (long) (updateIntervalFromConfig.getValue() * updateIntervalMultiplier),\n          updateIntervalFromConfig.getUnit());\n\n      Amount<Long, Time> initialDelay = scheduleImmediately ? ZERO_SECONDS : updateInterval;\n      updater.addTask(filter, segmentInfo, updateInterval, initialDelay);\n    }\n  }\n\n  /**\n   * Notify QueryCacheManager of a new list of segments we currently have, so that cache tasks\n   * can be updated.\n   *\n   * @param segments fresh list of all segments\n   *\n   * All existing tasks will be canceled/removed/destroyed, new tasks will be created for all\n   * segments.\n   */\n  @Override\n  public synchronized void update(Collection<SegmentInfo> segments, String message) {\n    if (!enabled()) {\n      return;\n    }\n\n    // This manager is created right at the beginning of a startup. Before we set it up,\n    // we'll read tweets and create segments and therefore this method will be called.\n    // We don't want to start computing query caches during that time, so we just return.\n    if (!didSetup) {\n      return;\n    }\n\n    NUM_UPDATE_SEGMENTS_CALLS.increment();\n\n    LOG.info(\"Rescheduling all query cache tasks ({}). Number of segments received = {}.\",\n        message, segments.size());\n    updater.clearTasks(); // cancel and remove all scheduled tasks\n\n    // If Earlybird is still starting up, and we get a partition roll, don't delay rebuilding\n    // the query cache.\n    boolean isCurrent = EarlybirdStatus.getStatusCode() == EarlybirdStatusCode.CURRENT;\n    scheduleTasks(segments, isCurrent);\n  }\n\n  /**\n   * Determines if all query cache tasks ran at least once (even if they failed).\n   */\n  public synchronized boolean allTasksRan() {\n    return (!(enabled() && didSetup)) || updater.allTasksRan();\n  }\n\n  /**\n   * Determines if the query cache manager is enabled.\n   */\n  public boolean enabled() {\n    return enabled;\n  }\n\n  /**\n   * Returns the query cache filter with the given name.\n   */\n  public QueryCacheFilter getFilter(String filterName) {\n    return filters.get(filterName);\n  }\n\n  /**\n   * Shuts down the query cache manager.\n   */\n  public synchronized void shutdown() throws InterruptedException {\n    LOG.info(\"Shutting down QueryCacheManager\");\n    if (updater != null) {\n      updater.shutdown();\n      updater = null;\n    }\n    didSetup = false; // needed for unit test\n  }\n\n  /**\n   * After startup, we want only one thread to update the query cache.\n   */\n  public void setWorkerPoolSizeAfterStartup() {\n    if (this.updater != null) {\n      this.updater.setWorkerPoolSizeAfterStartup();\n    }\n  }\n\n  public Decider getDecider() {\n    return this.decider;\n  }\n\n  //////////////////////////\n  // for unit tests only\n  //////////////////////////\n  QueryCacheUpdater getUpdaterForTest() {\n    return updater;\n  }\n  Map<String, QueryCacheFilter> getCacheMapForTest() {\n    return filters;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/querycache/QueryCacheResultCollector.java",
    "content": "package com.twitter.search.earlybird.querycache;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.util.BitDocIdSet;\nimport org.apache.lucene.util.BitSet;\nimport org.apache.lucene.util.FixedBitSet;\nimport org.apache.lucene.util.SparseFixedBitSet;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.core.earlybird.index.QueryCacheResultForSegment;\nimport com.twitter.search.earlybird.RecentTweetRestriction;\nimport com.twitter.search.earlybird.search.AbstractResultsCollector;\nimport com.twitter.search.earlybird.search.SearchRequestInfo;\nimport com.twitter.search.earlybird.search.SearchResultsInfo;\nimport com.twitter.search.earlybird.search.queries.SinceUntilFilter;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\n\nimport static org.apache.lucene.search.DocIdSetIterator.NO_MORE_DOCS;\n\nimport static com.twitter.search.core.earlybird.index.TimeMapper.ILLEGAL_TIME;\n\n/**\n * Collector to update the query cache (one segment for a filter)\n */\npublic class QueryCacheResultCollector\n    extends AbstractResultsCollector<SearchRequestInfo, SearchResultsInfo> {\n  private static final int UNSET = -1;\n\n  private final QueryCacheFilter queryCacheFilter;\n  private final Decider decider;\n\n  private BitSet bitSet;\n  private long cardinality = 0L;\n  private int startingDocID = UNSET;\n\n  public QueryCacheResultCollector(\n      ImmutableSchemaInterface schema,\n      QueryCacheFilter queryCacheFilter,\n      EarlybirdSearcherStats searcherStats,\n      Decider decider,\n      Clock clock,\n      int requestDebugMode) {\n    super(schema,\n        queryCacheFilter.createSearchRequestInfo(),\n        clock,\n        searcherStats,\n        requestDebugMode);\n    this.queryCacheFilter = queryCacheFilter;\n    this.decider = decider;\n  }\n\n  @Override\n  public void startSegment() throws IOException {\n    // The doc IDs in the optimized segments are always in the 0 .. (segmentSize - 1) range, so we\n    // can use a dense bitset to collect the hits. However, unoptimized segments can use any int\n    // doc IDs, so we have to use a sparse bitset to collect the hits in those segments.\n    if (currTwitterReader.getSegmentData().isOptimized()) {\n      switch (queryCacheFilter.getResultSetType()) {\n        case FixedBitSet:\n          bitSet = new FixedBitSet(currTwitterReader.maxDoc());\n          break;\n        case SparseFixedBitSet:\n          bitSet = new SparseFixedBitSet(currTwitterReader.maxDoc());\n          break;\n        default:\n          throw new IllegalStateException(\n              \"Unknown ResultSetType: \" + queryCacheFilter.getResultSetType().name());\n      }\n    } else {\n      bitSet = new SparseFixedBitSet(currTwitterReader.maxDoc());\n    }\n\n    startingDocID = findStartingDocID();\n    cardinality = 0;\n  }\n\n  @Override\n  protected void doCollect(long tweetID)  {\n    bitSet.set(curDocId);\n    cardinality++;\n  }\n\n  @Override\n  protected SearchResultsInfo doGetResults() {\n    return new SearchResultsInfo();\n  }\n\n  public QueryCacheResultForSegment getCachedResult() {\n    // Note that BitSet.cardinality takes linear time in the size of the maxDoc, so we track\n    // cardinality separately.\n    return new QueryCacheResultForSegment(new BitDocIdSet(bitSet, cardinality),\n        cardinality, startingDocID);\n  }\n\n  /**\n   * We don't want to return results less than 15 seconds older than the most recently indexed tweet,\n   * as they might not be completely indexed.\n   * We can't simply use the first hit, as some cached filters might not have any hits,\n   * e.g. has_engagement in the protected cluster.\n   * We can't use a clock because streams can lag.\n   */\n  private int findStartingDocID() throws IOException {\n    int lastTime = currTwitterReader.getSegmentData().getTimeMapper().getLastTime();\n    if (lastTime == ILLEGAL_TIME) {\n      return NO_MORE_DOCS;\n    }\n\n    int untilTime = RecentTweetRestriction.queryCacheUntilTime(decider, lastTime);\n    if (untilTime == 0) {\n      return currTwitterReader.getSmallestDocID();\n    }\n\n    return SinceUntilFilter.getUntilQuery(untilTime)\n        .createWeight(new IndexSearcher(currTwitterReader), ScoreMode.COMPLETE_NO_SCORES, 1.0f)\n        .scorer(currTwitterReader.getContext())\n        .iterator()\n        .nextDoc();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/querycache/QueryCacheUpdateTask.java",
    "content": "package com.twitter.search.earlybird.querycache;\n\nimport java.io.IOException;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.cache.CacheBuilder;\nimport com.google.common.cache.CacheLoader;\nimport com.google.common.cache.LoadingCache;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.quantity.Amount;\nimport com.twitter.common.quantity.Time;\nimport com.twitter.common.util.Clock;\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.Timer;\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.core.earlybird.index.QueryCacheResultForSegment;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.exception.EarlybirdException;\nimport com.twitter.search.earlybird.index.EarlybirdSegment;\nimport com.twitter.search.earlybird.index.EarlybirdSingleSegmentSearcher;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.search.SearchResultsInfo;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.util.ScheduledExecutorTask;\n\n/**\n * Each task is responsible for one filter on one segment. We should have a total\n * of num_of_filter * num_of_segments tasks\n */\n@VisibleForTesting\nclass QueryCacheUpdateTask extends ScheduledExecutorTask {\n  private static final Logger LOG =  LoggerFactory.getLogger(QueryCacheUpdateTask.class);\n\n  // See OBSERVE-10347\n  private static final boolean EXPORT_STATS =\n      EarlybirdConfig.getBool(\"export_query_cache_update_task_stats\", false);\n\n  private static final LoadingCache<String, TaskStats> TASK_STATS =\n      CacheBuilder.newBuilder().build(new CacheLoader<String, TaskStats>() {\n        @Override\n        public TaskStats load(String statNamePrefix) {\n          return new TaskStats(statNamePrefix, EXPORT_STATS);\n        }\n      });\n\n  private static final SearchCounter FINISHED_TASKS = SearchCounter.export(\n      \"querycache_finished_tasks\");\n\n  private final QueryCacheFilter filter;\n\n  // Info/data of the segment this task is responsible for\n  private final SegmentInfo segmentInfo;\n\n  private final UserTable userTable;\n\n  private volatile boolean ranOnce;\n  private final TaskStats stats;\n  private Amount<Long, Time> lastRunFinishTime;\n\n  // See SEARCH-4346\n  private final String filterAndSegment;\n\n  private final Decider decider;\n\n  private static final class TaskStats {\n    private final SearchLongGauge numHitsStat;\n    private final SearchLongGauge updateLatencyStat;\n    private final SearchCounter updateSuccessCountStat;\n    private final SearchCounter updateFailureCountStat;\n\n    private TaskStats(String statNamePrefix, boolean exportStats) {\n      // See SEARCH-3698\n      numHitsStat = exportStats ? SearchLongGauge.export(statNamePrefix + \"numhit\")\n          : new SearchLongGauge(statNamePrefix + \"numhit\");\n      updateLatencyStat = exportStats\n          ? SearchLongGauge.export(statNamePrefix + \"update_latency_ms\")\n          : new SearchLongGauge(statNamePrefix + \"update_latency_ms\");\n      updateSuccessCountStat = exportStats\n          ? SearchCounter.export(statNamePrefix + \"update_success_count\")\n          : SearchCounter.create(statNamePrefix + \"update_success_count\");\n      updateFailureCountStat = exportStats\n          ? SearchCounter.export(statNamePrefix + \"update_failure_count\")\n          : SearchCounter.create(statNamePrefix + \"update_failure_count\");\n    }\n  }\n\n  private final Amount<Long, Time> updateInterval;\n  private final Amount<Long, Time> initialDelay;\n\n  private final EarlybirdSearcherStats searcherStats;\n  private final CriticalExceptionHandler criticalExceptionHandler;\n\n  /**\n   * Constructor\n   * @param filter Filter to be used to populate the cache\n   * @param segmentInfo Segment this task is responsible for\n   * @param updateInterval Time between successive updates\n   * @param initialDelay Time before the first update\n   * @param updateIterationCounter\n   * @param decider\n   */\n  public QueryCacheUpdateTask(QueryCacheFilter filter,\n                              SegmentInfo segmentInfo,\n                              UserTable userTable,\n                              Amount<Long, Time> updateInterval,\n                              Amount<Long, Time> initialDelay,\n                              SearchCounter updateIterationCounter,\n                              EarlybirdSearcherStats searcherStats,\n                              Decider decider,\n                              CriticalExceptionHandler criticalExceptionHandler,\n                              Clock clock) {\n    super(updateIterationCounter, clock);\n    this.filter = filter;\n    this.segmentInfo = segmentInfo;\n    this.userTable = userTable;\n    this.ranOnce = false;\n    this.updateInterval = updateInterval;\n    this.initialDelay = initialDelay;\n    this.stats = setupStats();\n    this.filterAndSegment = String.format(\n        \"QueryCacheFilter: %s | Segment: %d\",\n        filter.getFilterName(), segmentInfo.getTimeSliceID());\n    this.searcherStats = searcherStats;\n    this.criticalExceptionHandler = criticalExceptionHandler;\n    this.decider = decider;\n  }\n\n  @Override\n  protected void runOneIteration() {\n    try {\n      if (LOG.isDebugEnabled()) {\n        LOG.debug(\n            \"[{}] Updating with query [{}] for the {} th time.\",\n            filterAndSegment,\n            filter.getQueryString(),\n            stats.updateSuccessCountStat.get() + stats.updateFailureCountStat.get() + 1\n        );\n        if (lastRunFinishTime != null) {\n          LOG.debug(\n              \"[{}] Last run, {} th time, finished {} secs ago. Should run every {} secs\",\n              filterAndSegment,\n              stats.updateSuccessCountStat.get() + stats.updateFailureCountStat.get(),\n              TimeUnit.NANOSECONDS.toSeconds(\n                  System.nanoTime() - lastRunFinishTime.as(Time.NANOSECONDS)),\n              updateInterval.as(Time.SECONDS)\n          );\n        }\n      }\n\n      Timer timer = new Timer(TimeUnit.MILLISECONDS);\n      SearchResultsInfo result = null;\n      try {\n        result = update();\n      } catch (Exception e) {\n        String msg = \"Failed to update query cache entry [\" + filter.getFilterName()\n            + \"] on segment [\" + segmentInfo.getTimeSliceID() + \"]\";\n        LOG.warn(msg, e);\n      }\n\n      long endTime = timer.stop();\n      updateStats(result, endTime);\n\n      if (LOG.isDebugEnabled()) {\n        LOG.debug(\"[{}] Updated in {} ms, hit {} docs.\",\n            filterAndSegment, endTime, stats.numHitsStat.read());\n      }\n      // Need to catch throwable here instead of exception so we handle errors like OutOfMemory\n      // See RB=528695 and SEARCH-4402\n    } catch (Throwable t) {\n      String message = String.format(\"Got unexpected throwable in %s\", getClass().getName());\n      LOG.error(message, t);\n\n      // Wrap the Throwable in a FatalEarlybirdException to categorize it and ensure it's\n      // handled as a fatal exception\n      criticalExceptionHandler.handle(this,\n          new EarlybirdException(message, t));\n    } finally {\n      // Earlybird won't become CURRENT until all tasks are run at least once. We don't want\n      // failed \"run\" (update) to prevent Earlybird from becoming CURRENT. As long as all tasks\n      // got a chance to run at least once, we are good to go.\n      ranOnce = true;\n\n      lastRunFinishTime = Amount.of(System.nanoTime(), Time.NANOSECONDS);\n    }\n  }\n\n  public boolean ranOnce() {\n    return ranOnce;\n  }\n\n  private TaskStats setupStats() {\n    return TASK_STATS.getUnchecked(statNamePrefix());\n  }\n\n  private SearchResultsInfo update() throws IOException {\n    // There's a chance that the EarlybirdSegment of a SegmentInfo to change at any\n    // time. Therefore, it's not safe to operate segments on the SegmentInfo level.\n    // On the archive clusters we create a new EarlybirdSegment and then swap it in when there's\n    // new data instead of appending to an existing EarlybirdSegment.\n    EarlybirdSegment earlybirdSegment = segmentInfo.getIndexSegment();\n\n    EarlybirdSingleSegmentSearcher searcher = earlybirdSegment.getSearcher(userTable);\n    if (searcher == null) {\n      LOG.warn(\"Unable to get searcher from TwitterIndexManager for segment [\"\n          + segmentInfo.getTimeSliceID() + \"]. Has it been dropped?\");\n      return null;\n    }\n\n    QueryCacheResultCollector collector = new QueryCacheResultCollector(\n        searcher.getSchemaSnapshot(), filter, searcherStats, decider, clock, 0);\n    searcher.search(filter.getLuceneQuery(), collector);\n\n    QueryCacheResultForSegment cacheResult = collector.getCachedResult();\n    searcher.getTwitterIndexReader().getSegmentData().updateQueryCacheResult(\n        filter.getFilterName(), cacheResult);\n\n    FINISHED_TASKS.increment();\n\n    if (LOG.isDebugEnabled()) {\n      TerminationTracker tracker = collector.getSearchRequestInfo().getTerminationTracker();\n      LOG.debug(\n          \"[{}] Updating query finished, start time ms is {}, termination reason is {}\",\n          filterAndSegment,\n          tracker.getLocalStartTimeMillis(),\n          tracker.getEarlyTerminationState().getTerminationReason());\n    }\n\n    return collector.getResults();\n  }\n\n  private void updateStats(SearchResultsInfo result, long endTime) {\n    if (result != null) {\n      stats.numHitsStat.set(result.getNumHitsProcessed());\n      stats.updateSuccessCountStat.increment();\n    } else {\n      stats.updateFailureCountStat.increment();\n    }\n    stats.updateLatencyStat.set(endTime);\n  }\n\n  @VisibleForTesting\n  String statNamePrefix() {\n    // If we use this and try to display in monviz \"ts(partition, single_instance, querycache*)\",\n    // the UI shows \"Really expensive query\" message. We can keep this around for times when we\n    // want to start things manually and debug.\n    return \"querycache_\" + filter.getFilterName() + \"_\" + segmentInfo.getTimeSliceID() + \"_\";\n  }\n\n  public long getTimeSliceID() {\n    return segmentInfo.getTimeSliceID();\n  }\n\n  //////////////////////////\n  // for unit tests only\n  //////////////////////////\n  @VisibleForTesting\n  String getFilterNameForTest() {\n    return filter.getFilterName();\n  }\n\n  @VisibleForTesting\n  Amount<Long, Time> getUpdateIntervalForTest() {\n    return updateInterval;\n  }\n\n  @VisibleForTesting\n  Amount<Long, Time> getInitialDelayForTest() {\n    return initialDelay;\n  }\n\n  @VisibleForTesting\n  TaskStats getTaskStatsForTest() {\n    return stats;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/querycache/QueryCacheUpdater.java",
    "content": "package com.twitter.search.earlybird.querycache;\n\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.concurrent.ScheduledExecutorService;\nimport java.util.concurrent.ScheduledFuture;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.quantity.Amount;\nimport com.twitter.common.quantity.Time;\nimport com.twitter.common.util.Clock;\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.concurrent.ScheduledExecutorServiceFactory;\nimport com.twitter.search.common.metrics.SearchCustomGauge;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.factory.QueryCacheUpdaterScheduledExecutorService;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.util.PeriodicActionParams;\nimport com.twitter.search.earlybird.util.ScheduledExecutorManager;\nimport com.twitter.search.earlybird.util.ShutdownWaitTimeParams;\n\n/**\n * Class to manage the scheduler service and all the update tasks. Through this\n * class, update tasks are created and scheduled, canceled and removed.\n *\n * This class is not thread-safe.\n */\n@VisibleForTesting\nfinal class QueryCacheUpdater extends ScheduledExecutorManager {\n  private static final Logger LOG = LoggerFactory.getLogger(QueryCacheUpdater.class);\n\n  private final List<Task> tasks;\n  private final EarlybirdSearcherStats searcherStats;\n  private final Decider decider;\n  private final UserTable userTable;\n  private final Clock clock;\n\n  @VisibleForTesting\n  static final class Task {\n    @VisibleForTesting public final QueryCacheUpdateTask updateTask;\n    @VisibleForTesting public final ScheduledFuture future;\n\n    private Task(QueryCacheUpdateTask updateTask, ScheduledFuture future) {\n      this.updateTask = updateTask;\n      this.future = future;\n    }\n  }\n\n  public QueryCacheUpdater(Collection<QueryCacheFilter> cacheFilters,\n                           ScheduledExecutorServiceFactory updaterScheduledExecutorServiceFactory,\n                           UserTable userTable,\n                           SearchStatsReceiver searchStatsReceiver,\n                           EarlybirdSearcherStats searcherStats,\n                           Decider decider,\n                           CriticalExceptionHandler criticalExceptionHandler,\n                           Clock clock) {\n    super(updaterScheduledExecutorServiceFactory.build(\"QueryCacheUpdateThread-%d\", true),\n        ShutdownWaitTimeParams.immediately(), searchStatsReceiver,\n        criticalExceptionHandler, clock);\n    Preconditions.checkNotNull(cacheFilters);\n    Preconditions.checkArgument(getExecutor() instanceof QueryCacheUpdaterScheduledExecutorService,\n        getExecutor().getClass());\n\n    this.searcherStats = searcherStats;\n    this.decider = decider;\n    this.userTable = userTable;\n    this.clock = clock;\n\n    shouldLog = false;\n    // One update task per <query, segment>\n    tasks = Lists.newArrayListWithCapacity(cacheFilters.size() * 20);\n\n    SearchCustomGauge.export(\n        \"querycache_num_tasks\",\n        tasks::size\n    );\n  }\n\n  /**\n   * Create an update task and add it to the executor\n   *\n   * @param filter The filter the task should execute\n   * @param segmentInfo The segment that this task would be responsible for\n   * @param updateInterval time in milliseconds between successive updates\n   * @param initialDelay Introduce a delay when adding the task to the executor\n   */\n  void addTask(QueryCacheFilter filter, SegmentInfo segmentInfo,\n               Amount<Long, Time> updateInterval, Amount<Long, Time> initialDelay) {\n    String filterName = filter.getFilterName();\n    String query = filter.getQueryString();\n\n    // Create the task.\n    QueryCacheUpdateTask qcTask = new QueryCacheUpdateTask(\n        filter,\n        segmentInfo,\n        userTable,\n        updateInterval,\n        initialDelay,\n        getIterationCounter(),\n        searcherStats,\n        decider,\n        criticalExceptionHandler,\n        clock);\n\n    long initialDelayAsMS = initialDelay.as(Time.MILLISECONDS);\n    long updateIntervalAsMS = updateInterval.as(Time.MILLISECONDS);\n    Preconditions.checkArgument(\n        initialDelayAsMS >= initialDelay.getValue(), \"initial delay unit granularity too small\");\n    Preconditions.checkArgument(\n        updateIntervalAsMS >= updateInterval.getValue(),\n        \"update interval unit granularity too small\");\n\n    // Schedule the task.\n    ScheduledFuture future = scheduleNewTask(qcTask,\n        PeriodicActionParams.withIntialWaitAndFixedDelay(\n            initialDelayAsMS, updateIntervalAsMS, TimeUnit.MILLISECONDS\n        )\n    );\n\n    tasks.add(new Task(qcTask, future));\n\n    LOG.debug(\"Added a task for filter [\" + filterName\n            + \"] for segment [\" + segmentInfo.getTimeSliceID()\n            + \"] with query [\" + query\n            + \"] update interval \" + updateInterval + \" \"\n            + (initialDelay.getValue() == 0 ? \"without\" : \"with \" + initialDelay)\n            + \" initial delay\");\n\n  }\n\n  void removeAllTasksForSegment(SegmentInfo segmentInfo) {\n    int removedTasksCount = 0;\n    for (Iterator<Task> it = tasks.iterator(); it.hasNext();) {\n      Task task = it.next();\n      if (task.updateTask.getTimeSliceID() == segmentInfo.getTimeSliceID()) {\n        task.future.cancel(true);\n        it.remove();\n        removedTasksCount += 1;\n      }\n    }\n\n    LOG.info(\"Removed {} update tasks for segment {}.\", removedTasksCount,\n        segmentInfo.getTimeSliceID());\n  }\n\n  public void clearTasks() {\n    int totalTasks = tasks.size();\n    LOG.info(\"Removing {} update tasks for all segments.\", totalTasks);\n    for (Task task : tasks) {\n      task.future.cancel(true);\n    }\n    tasks.clear();\n    LOG.info(\"Canceled {} QueryCache update tasks\", totalTasks);\n  }\n\n  // Have all tasks run at least once (even if they failed)?\n  public boolean allTasksRan() {\n    boolean allTasksRan = true;\n    for (Task task : tasks) {\n      if (!task.updateTask.ranOnce()) {\n        allTasksRan = false;\n        break;\n      }\n    }\n\n    return allTasksRan;\n  }\n\n  // Have all tasks for this run at least once (even if they failed)?\n  public boolean allTasksRanForSegment(SegmentInfo segmentInfo) {\n    boolean allTasksRanForSegment = true;\n    for (Task task : tasks) {\n      if ((task.updateTask.getTimeSliceID() == segmentInfo.getTimeSliceID())\n          && !task.updateTask.ranOnce()) {\n        allTasksRanForSegment = false;\n        break;\n      }\n    }\n\n    return allTasksRanForSegment;\n  }\n\n  /**\n   * After startup, we want only one thread to update the query cache.\n   */\n  void setWorkerPoolSizeAfterStartup() {\n    QueryCacheUpdaterScheduledExecutorService executor =\n        (QueryCacheUpdaterScheduledExecutorService) getExecutor();\n    executor.setWorkerPoolSizeAfterStartup();\n    LOG.info(\"Done setting executor core pool size to one\");\n  }\n\n  @Override\n  protected void shutdownComponent() {\n    clearTasks();\n  }\n\n  //////////////////////////\n  // for unit tests only\n  //////////////////////////\n\n  /**\n   * Returns the list of all query cache updater tasks. This method should be used only in tests.\n   */\n  @VisibleForTesting\n  List<Task> getTasksForTest() {\n    synchronized (tasks) {\n      return new ArrayList<>(tasks);\n    }\n  }\n\n  @VisibleForTesting\n  int getTasksSize() {\n    synchronized (tasks) {\n      return tasks.size();\n    }\n  }\n\n  @VisibleForTesting\n  boolean tasksContains(Task task) {\n    synchronized (tasks) {\n      return tasks.contains(task);\n    }\n  }\n\n  @VisibleForTesting\n  public ScheduledExecutorService getExecutorForTest() {\n    return getExecutor();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/queryparser/DetectAntisocialVisitor.java",
    "content": "package com.twitter.search.earlybird.queryparser;\n\nimport com.twitter.search.common.constants.QueryCacheConstants;\nimport com.twitter.search.queryparser.query.Conjunction;\nimport com.twitter.search.queryparser.query.Disjunction;\nimport com.twitter.search.queryparser.query.Phrase;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.SpecialTerm;\nimport com.twitter.search.queryparser.query.Term;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\nimport com.twitter.search.queryparser.query.search.SearchOperatorConstants;\nimport com.twitter.search.queryparser.query.search.SearchQueryVisitor;\n\n/**\n * Visitor to detect presence of any antisocial / spam operator in a Query.\n * Visitor returns true if any operators it detects were found.\n */\npublic class DetectAntisocialVisitor extends SearchQueryVisitor<Boolean> {\n  // True if the query contains any operator to include antisocial tweets.\n  private boolean includeAntisocial = false;\n\n  // True if the query contains any operator to exclude antisocial/spam tweets.\n  private boolean excludeAntisocial = false;\n\n  // True if the query contains an antisocial tweets filter.\n  private boolean filterAntisocial = false;\n\n  public boolean hasIncludeAntisocial() {\n    return includeAntisocial;\n  }\n\n  public boolean hasExcludeAntisocial() {\n    return excludeAntisocial;\n  }\n\n  public boolean hasFilterAntisocial() {\n    return filterAntisocial;\n  }\n\n  public boolean hasAnyAntisocialOperator() {\n    // Top tweets is considered an antisocial operator due to scoring also excluding\n    // spam tweets.\n    return hasIncludeAntisocial() || hasExcludeAntisocial() || hasFilterAntisocial();\n  }\n\n  @Override public Boolean visit(Disjunction disjunction) throws QueryParserException {\n    boolean found = false;\n    for (com.twitter.search.queryparser.query.Query node : disjunction.getChildren()) {\n      if (node.accept(this)) {\n        found = true;\n      }\n    }\n    return found;\n  }\n\n  @Override public Boolean visit(Conjunction conjunction) throws QueryParserException {\n    boolean found = false;\n    for (com.twitter.search.queryparser.query.Query node : conjunction.getChildren()) {\n      if (node.accept(this)) {\n        found = true;\n      }\n    }\n    return found;\n  }\n\n  @Override public Boolean visit(SearchOperator operator) throws QueryParserException {\n    boolean found = false;\n    switch (operator.getOperatorType()) {\n      case INCLUDE:\n        if (SearchOperatorConstants.ANTISOCIAL.equals(operator.getOperand())) {\n          if (operator.mustNotOccur()) {\n            excludeAntisocial = true;\n          } else {\n            includeAntisocial = true;\n          }\n          found = true;\n        }\n        break;\n      case EXCLUDE:\n        if (SearchOperatorConstants.ANTISOCIAL.equals(operator.getOperand())) {\n          if (operator.mustNotOccur()) {\n            includeAntisocial = true;\n          } else {\n            excludeAntisocial = true;\n          }\n          found = true;\n        }\n        break;\n      case FILTER:\n        if (SearchOperatorConstants.ANTISOCIAL.equals(operator.getOperand())) {\n          if (operator.mustNotOccur()) {\n            excludeAntisocial = true;\n          } else {\n            filterAntisocial = true;\n          }\n          found = true;\n        }\n        break;\n      case CACHED_FILTER:\n        if (QueryCacheConstants.EXCLUDE_SPAM.equals(operator.getOperand())\n            || QueryCacheConstants.EXCLUDE_SPAM_AND_NATIVERETWEETS.equals(operator.getOperand())\n            || QueryCacheConstants.EXCLUDE_ANTISOCIAL.equals(operator.getOperand())\n            || QueryCacheConstants.EXCLUDE_ANTISOCIAL_AND_NATIVERETWEETS\n                .equals(operator.getOperand())) {\n\n          excludeAntisocial = true;\n          found = true;\n        }\n        break;\n      default:\n        break;\n    }\n\n    return found;\n  }\n\n  @Override\n  public Boolean visit(SpecialTerm special) throws QueryParserException {\n    return false;\n  }\n\n  @Override\n  public Boolean visit(Phrase phrase) throws QueryParserException {\n    return false;\n  }\n\n  @Override\n  public Boolean visit(Term term) throws QueryParserException {\n    return false;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/queryparser/DetectFieldAnnotationVisitor.java",
    "content": "package com.twitter.search.earlybird.queryparser;\n\nimport java.util.Set;\n\nimport com.google.common.collect.ImmutableSet;\n\nimport com.twitter.search.queryparser.query.BooleanQuery;\nimport com.twitter.search.queryparser.query.Conjunction;\nimport com.twitter.search.queryparser.query.Disjunction;\nimport com.twitter.search.queryparser.query.Operator;\nimport com.twitter.search.queryparser.query.Phrase;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.QueryVisitor;\nimport com.twitter.search.queryparser.query.SpecialTerm;\nimport com.twitter.search.queryparser.query.Term;\nimport com.twitter.search.queryparser.query.annotation.Annotation;\nimport com.twitter.search.queryparser.query.annotation.FieldNameWithBoost;\n\n/**\n * Detects whether the query tree has certain field annotations.\n */\npublic class DetectFieldAnnotationVisitor extends QueryVisitor<Boolean> {\n  private final ImmutableSet<String> fieldNames;\n\n  /**\n   * This visitor will return true if the query tree has a FIELD annotation with any of the given\n   * field names. If the set is empty, any FIELD annotation will match.\n   */\n  public DetectFieldAnnotationVisitor(Set<String> fieldNames) {\n    this.fieldNames = ImmutableSet.copyOf(fieldNames);\n  }\n\n  /**\n   * This visitor will return true if the query tree has a FIELD annotation.\n   */\n  public DetectFieldAnnotationVisitor() {\n    this.fieldNames = ImmutableSet.of();\n  }\n\n  @Override\n  public Boolean visit(Disjunction disjunction) throws QueryParserException {\n    return visitQuery(disjunction) || visitBooleanQuery(disjunction);\n  }\n\n  @Override\n  public Boolean visit(Conjunction conjunction) throws QueryParserException {\n    return visitQuery(conjunction) || visitBooleanQuery(conjunction);\n  }\n\n  @Override\n  public Boolean visit(Phrase phrase) throws QueryParserException {\n    return visitQuery(phrase);\n  }\n\n  @Override\n  public Boolean visit(Term term) throws QueryParserException {\n    return visitQuery(term);\n  }\n\n  @Override\n  public Boolean visit(Operator operator) throws QueryParserException {\n    return visitQuery(operator);\n  }\n\n  @Override\n  public Boolean visit(SpecialTerm special) throws QueryParserException {\n    return visitQuery(special);\n  }\n\n  private Boolean visitQuery(Query query) throws QueryParserException {\n    if (query.hasAnnotations()) {\n      for (Annotation annotation : query.getAnnotations()) {\n        if (!Annotation.Type.FIELD.equals(annotation.getType())) {\n          continue;\n        }\n        if (fieldNames.isEmpty()) {\n          return true;\n        }\n        FieldNameWithBoost value = (FieldNameWithBoost) annotation.getValue();\n        if (fieldNames.contains(value.getFieldName())) {\n          return true;\n        }\n      }\n    }\n\n    return false;\n  }\n\n  private boolean visitBooleanQuery(BooleanQuery query) throws QueryParserException {\n    for (Query subQuery : query.getChildren()) {\n      if (subQuery.accept(this)) {\n        return true;\n      }\n    }\n\n    return false;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/queryparser/EarlybirdLuceneQueryVisitor.java",
    "content": "package com.twitter.search.earlybird.queryparser;\n\nimport java.util.Arrays;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.TreeSet;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Functions;\nimport com.google.common.base.Optional;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Sets;\n\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanClause.Occur;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.BoostQuery;\nimport org.apache.lucene.search.MatchNoDocsQuery;\nimport org.apache.lucene.search.PhraseQuery;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.TermQuery;\nimport org.locationtech.spatial4j.shape.Point;\nimport org.locationtech.spatial4j.shape.Rectangle;\nimport org.locationtech.spatial4j.shape.impl.PointImpl;\nimport org.locationtech.spatial4j.shape.impl.RectangleImpl;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.constants.QueryCacheConstants;\nimport com.twitter.search.common.decider.DeciderUtil;\nimport com.twitter.search.common.encoding.features.ByteNormalizer;\nimport com.twitter.search.common.indexing.thriftjava.ThriftGeoLocationSource;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.query.BoostUtils;\nimport com.twitter.search.common.query.FieldWeightUtil;\nimport com.twitter.search.common.query.FilteredQuery;\nimport com.twitter.search.common.query.HitAttributeHelper;\nimport com.twitter.search.common.query.MappableField;\nimport com.twitter.search.common.schema.ImmutableSchema;\nimport com.twitter.search.common.schema.SchemaUtil;\nimport com.twitter.search.common.schema.base.FieldWeightDefault;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.schema.earlybird.EarlybirdThriftDocumentBuilder;\nimport com.twitter.search.common.schema.earlybird.EarlybirdThriftDocumentUtil;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFType;\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.common.search.termination.QueryTimeout;\nimport com.twitter.search.common.util.analysis.IntTermAttributeImpl;\nimport com.twitter.search.common.util.analysis.LongTermAttributeImpl;\nimport com.twitter.search.common.util.spatial.GeohashChunkImpl;\nimport com.twitter.search.common.util.text.HighFrequencyTermPairs;\nimport com.twitter.search.common.util.text.NormalizerHelper;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.userupdates.UserScrubGeoMap;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.partition.MultiSegmentTermDictionaryManager;\nimport com.twitter.search.earlybird.querycache.CachedFilterQuery;\nimport com.twitter.search.earlybird.querycache.QueryCacheManager;\nimport com.twitter.search.earlybird.search.queries.CSFDisjunctionFilter;\nimport com.twitter.search.earlybird.search.queries.DocValRangeFilter;\nimport com.twitter.search.earlybird.search.queries.FeatureValueInAcceptListOrUnsetFilter;\nimport com.twitter.search.earlybird.search.GeoQuadTreeQueryBuilder;\nimport com.twitter.search.earlybird.search.queries.MatchAllDocsQuery;\nimport com.twitter.search.earlybird.search.queries.RequiredStatusIDsFilter;\nimport com.twitter.search.earlybird.search.queries.SinceMaxIDFilter;\nimport com.twitter.search.earlybird.search.queries.SinceUntilFilter;\nimport com.twitter.search.earlybird.search.queries.TermQueryWithSafeToString;\nimport com.twitter.search.earlybird.search.queries.UserFlagsExcludeFilter;\nimport com.twitter.search.earlybird.search.queries.UserScrubGeoFilter;\nimport com.twitter.search.earlybird.search.queries.UserIdMultiSegmentQuery;\nimport com.twitter.search.earlybird.search.relevance.MinFeatureValueFilter;\nimport com.twitter.search.earlybird.search.relevance.ScoreFilterQuery;\nimport com.twitter.search.earlybird.search.relevance.scoring.ScoringFunctionProvider;\nimport com.twitter.search.queryparser.query.Conjunction;\nimport com.twitter.search.queryparser.query.Disjunction;\nimport com.twitter.search.queryparser.query.Phrase;\nimport com.twitter.search.queryparser.query.QueryNodeUtils;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.SpecialTerm;\nimport com.twitter.search.queryparser.query.Term;\nimport com.twitter.search.queryparser.query.annotation.Annotation;\nimport com.twitter.search.queryparser.query.annotation.FloatAnnotation;\nimport com.twitter.search.queryparser.query.search.Link;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\nimport com.twitter.search.queryparser.query.search.SearchOperatorConstants;\nimport com.twitter.search.queryparser.query.search.SearchQueryVisitor;\nimport com.twitter.search.queryparser.util.GeoCode;\nimport com.twitter.service.spiderduck.gen.LinkCategory;\nimport com.twitter.tweetypie.thriftjava.ComposerSource;\n\n/**\n * Visitor for {@link com.twitter.search.queryparser.query.Query}, which produces a Lucene\n * Query ({@link Query}).\n */\npublic class EarlybirdLuceneQueryVisitor extends SearchQueryVisitor<Query> {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdLuceneQueryVisitor.class);\n\n  @VisibleForTesting\n  static final String UNSUPPORTED_OPERATOR_PREFIX = \"unsupported_query_operator_\";\n\n  private static final String SMILEY_FORMAT_STRING = \"__has_%s_smiley\";\n  private static final String PHRASE_WILDCARD = \"*\";\n  private static final float DEFAULT_FIELD_WEIGHT = 1.0f;\n\n  private static final SearchCounter SINCE_TIME_INVALID_INT_COUNTER =\n      SearchCounter.export(\"EarlybirdLuceneQueryVisitor_since_time_invalid_int\");\n  private static final SearchCounter UNTIL_TIME_INVALID_INT_COUNTER =\n      SearchCounter.export(\"EarlybirdLuceneQueryVisitor_until_time_invalid_int\");\n\n  private static final SearchCounter NUM_QUERIES_BELOW_MIN_ENGAGEMENT_THRESHOLD =\n      SearchCounter.export(\n          \"EarlybirdLuceneQueryVisitor_num_queries_below_min_engagement_threshold\");\n  private static final SearchCounter NUM_QUERIES_ABOVE_MIN_ENGAGEMENT_THRESHOLD =\n      SearchCounter.export(\n          \"EarlybirdLuceneQueryVisitor_num_queries_above_min_engagement_threshold\");\n\n  private static final SearchOperator OPERATOR_CACHED_EXCLUDE_ANTISOCIAL_AND_NATIVERETWEETS =\n      new SearchOperator(SearchOperator.Type.CACHED_FILTER,\n          QueryCacheConstants.EXCLUDE_ANTISOCIAL_AND_NATIVERETWEETS);\n\n  private static final Map<String, List<SearchOperator>> OPERATORS_BY_SAFE_EXCLUDE_OPERAND =\n      ImmutableMap.of(\n          SearchOperatorConstants.TWEET_SPAM, ImmutableList.of(\n              new SearchOperator(SearchOperator.Type.DOCVAL_RANGE_FILTER,\n                  \"extended_encoded_tweet_features.label_spam_flag\", \"0\", \"1\"),\n              new SearchOperator(SearchOperator.Type.DOCVAL_RANGE_FILTER,\n                  \"extended_encoded_tweet_features.label_spam_hi_rcl_flag\", \"0\", \"1\"),\n              new SearchOperator(SearchOperator.Type.DOCVAL_RANGE_FILTER,\n                  \"extended_encoded_tweet_features.label_dup_content_flag\", \"0\", \"1\")),\n\n          SearchOperatorConstants.TWEET_ABUSIVE, ImmutableList.of(\n              new SearchOperator(SearchOperator.Type.DOCVAL_RANGE_FILTER,\n                  \"extended_encoded_tweet_features.label_abusive_flag\", \"0\", \"1\")),\n\n          SearchOperatorConstants.TWEET_UNSAFE, ImmutableList.of(\n              new SearchOperator(SearchOperator.Type.DOCVAL_RANGE_FILTER,\n                  \"extended_encoded_tweet_features.label_nsfw_hi_prc_flag\", \"0\", \"1\"))\n      );\n\n  private static final ImmutableMap<String, FieldWeightDefault> DEFAULT_FIELDS =\n      ImmutableMap.of(EarlybirdFieldConstant.TEXT_FIELD.getFieldName(),\n                      new FieldWeightDefault(true, DEFAULT_FIELD_WEIGHT));\n\n  // All Earlybird fields that should have geo scrubbed tweets filtered out when searched.\n  // See go/realtime-geo-filtering\n  @VisibleForTesting\n  public static final List<String> GEO_FIELDS_TO_BE_SCRUBBED = Arrays.asList(\n      EarlybirdFieldConstant.GEO_HASH_FIELD.getFieldName(),\n      EarlybirdFieldConstant.PLACE_FIELD.getFieldName(),\n      EarlybirdFieldConstant.PLACE_ID_FIELD.getFieldName(),\n      EarlybirdFieldConstant.PLACE_FULL_NAME_FIELD.getFieldName(),\n      EarlybirdFieldConstant.PLACE_COUNTRY_CODE_FIELD.getFieldName());\n\n  // Geo scrubbing doesn't remove user profile location, so when using the geo location type filters\n  // we only need to filter out geo scrubbed tweets for the geo location types other than\n  // ThriftGeoLocationSource.USER_PROFILE.\n  // Separately, we also need to filter out geo scrubbed tweets for the place_id filter.\n  private static final List<String> GEO_FILTERS_TO_BE_SCRUBBED = Arrays.asList(\n      EarlybirdFieldConstants.formatGeoType(ThriftGeoLocationSource.GEOTAG),\n      EarlybirdFieldConstants.formatGeoType(ThriftGeoLocationSource.TWEET_TEXT),\n      EarlybirdThriftDocumentUtil.formatFilter(\n          EarlybirdFieldConstant.PLACE_ID_FIELD.getFieldName()));\n\n  // queries whose parents are negated.\n  // used to decide if a negated query is within a negated parent or not.\n  private final Set<com.twitter.search.queryparser.query.Query> parentNegatedQueries =\n      Sets.newIdentityHashSet();\n\n  private final ImmutableSchemaInterface schemaSnapshot;\n  private final ImmutableMap<String, FieldWeightDefault> defaultFieldWeightMap;\n  private final QueryCacheManager queryCacheManager;\n  private final UserTable userTable;\n  private final UserScrubGeoMap userScrubGeoMap;\n\n  @Nullable\n  private final TerminationTracker terminationTracker;\n  private final Map<MappableField, String> mappableFieldMap;\n  private final MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager;\n  private final Decider decider;\n  private final EarlybirdCluster earlybirdCluster;\n\n  private float proximityPhraseWeight = 1.0f;\n  private int proximityPhraseSlop = 255;\n  private ImmutableMap<String, Float> enabledFieldWeightMap;\n  private Set<String> queriedFields;\n\n  // If we need to accumulate and collect per-field and per query node hit attribution information,\n  // this will have a mapping between the query nodes and their unique ranks, as well as the\n  // attribute collector.\n  @Nullable\n  private HitAttributeHelper hitAttributeHelper;\n\n  @Nullable\n  private QueryTimeout queryTimeout;\n\n  public EarlybirdLuceneQueryVisitor(\n      ImmutableSchemaInterface schemaSnapshot,\n      QueryCacheManager queryCacheManager,\n      UserTable userTable,\n      UserScrubGeoMap userScrubGeoMap,\n      EarlybirdCluster earlybirdCluster,\n      Decider decider) {\n    this(schemaSnapshot, queryCacheManager, userTable, userScrubGeoMap, null, DEFAULT_FIELDS,\n         Collections.emptyMap(), null, decider, earlybirdCluster, null);\n  }\n\n  public EarlybirdLuceneQueryVisitor(\n      ImmutableSchemaInterface schemaSnapshot,\n      QueryCacheManager queryCacheManager,\n      UserTable userTable,\n      UserScrubGeoMap userScrubGeoMap,\n      @Nullable TerminationTracker terminationTracker,\n      Map<String, FieldWeightDefault> fieldWeightMap,\n      Map<MappableField, String> mappableFieldMap,\n      MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager,\n      Decider decider,\n      EarlybirdCluster earlybirdCluster,\n      QueryTimeout queryTimeout) {\n    this.schemaSnapshot = schemaSnapshot;\n    this.defaultFieldWeightMap = ImmutableMap.copyOf(fieldWeightMap);\n    this.enabledFieldWeightMap = FieldWeightDefault.getOnlyEnabled(defaultFieldWeightMap);\n    this.queryCacheManager = queryCacheManager;\n    this.userTable = userTable;\n    this.userScrubGeoMap = userScrubGeoMap;\n    this.mappableFieldMap = Preconditions.checkNotNull(mappableFieldMap);\n    this.terminationTracker = terminationTracker;\n    this.multiSegmentTermDictionaryManager = multiSegmentTermDictionaryManager;\n    this.decider = decider;\n    this.earlybirdCluster = earlybirdCluster;\n    this.queryTimeout = queryTimeout;\n    this.queriedFields = new TreeSet<>();\n  }\n\n  public ImmutableMap<String, Float> getEnabledFieldWeightMap() {\n    return enabledFieldWeightMap;\n  }\n\n  public ImmutableMap<String, FieldWeightDefault> getDefaultFieldWeightMap() {\n    return defaultFieldWeightMap;\n  }\n\n  public EarlybirdLuceneQueryVisitor setProximityPhraseWeight(float weight) {\n    this.proximityPhraseWeight = weight;\n    return this;\n  }\n\n  public EarlybirdLuceneQueryVisitor setProximityPhraseSlop(int slop) {\n    this.proximityPhraseSlop = slop;\n    return this;\n  }\n\n  public void setFieldHitAttributeHelper(HitAttributeHelper newHitAttributeHelper) {\n    this.hitAttributeHelper = newHitAttributeHelper;\n  }\n\n  @Override\n  public final Query visit(Disjunction disjunction) throws QueryParserException {\n    BooleanQuery.Builder bqBuilder = new BooleanQuery.Builder();\n    List<com.twitter.search.queryparser.query.Query> children = disjunction.getChildren();\n    // Do a final round of check, if all nodes under a disjunction are MUST,\n    // treat them all as DEFAULT (SHOULD in Lucene).\n    boolean allMust = true;\n    for (com.twitter.search.queryparser.query.Query child : children) {\n      if (!child.mustOccur()) {\n        allMust = false;\n        break;\n      }\n    }\n    if (allMust) {\n      children = Lists.transform(children, QueryNodeUtils.MAKE_QUERY_DEFAULT);\n    }\n    // Actually converting all children now.\n    for (com.twitter.search.queryparser.query.Query child : children) {\n      final Query q = child.accept(this);\n      if (q != null) {\n        // if a node is marked with MUSTHAVE annotation, we set it to must even if it's a\n        // disjunction.\n        if (child.mustOccur()) {\n          bqBuilder.add(q, Occur.MUST);\n        } else {\n          bqBuilder.add(q, Occur.SHOULD);\n        }\n      }\n    }\n\n    Query bq = bqBuilder.build();\n    float boost = (float) getBoostFromAnnotations(disjunction.getAnnotations());\n    if (boost >= 0) {\n      bq = BoostUtils.maybeWrapInBoostQuery(bq, boost);\n    }\n    return bq;\n  }\n\n  @Override\n  public Query visit(Conjunction conjunction) throws QueryParserException {\n    BooleanQuery.Builder bqBuilder = new BooleanQuery.Builder();\n    List<com.twitter.search.queryparser.query.Query> children = conjunction.getChildren();\n    boolean hasPositiveTerms = false;\n    for (com.twitter.search.queryparser.query.Query child : children) {\n      boolean childMustNotOccur = child.mustNotOccur();\n      boolean childAdded = addQuery(bqBuilder, child);\n      if (childAdded && !childMustNotOccur) {\n        hasPositiveTerms = true;\n      }\n    }\n    if (!children.isEmpty() && !hasPositiveTerms) {\n      bqBuilder.add(new MatchAllDocsQuery(), Occur.MUST);\n    }\n\n    Query bq = bqBuilder.build();\n    float boost = (float) getBoostFromAnnotations(conjunction.getAnnotations());\n    if (boost >= 0) {\n      bq = BoostUtils.maybeWrapInBoostQuery(bq, boost);\n    }\n    return bq;\n  }\n\n  @Override\n  public Query visit(Phrase phrase) throws QueryParserException {\n    return visit(phrase, false);\n  }\n\n  @Override\n  public Query visit(Term term) throws QueryParserException {\n    return finalizeQuery(createTermQueryDisjunction(term), term);\n  }\n\n  @Override\n  public Query visit(SpecialTerm special) throws QueryParserException {\n    String field;\n\n    switch (special.getType()) {\n      case HASHTAG:\n        field = EarlybirdFieldConstant.HASHTAGS_FIELD.getFieldName();\n        break;\n      case STOCK:\n        field = EarlybirdFieldConstant.STOCKS_FIELD.getFieldName();\n        break;\n      case MENTION:\n        field = EarlybirdFieldConstant.MENTIONS_FIELD.getFieldName();\n        break;\n      default:\n        field = EarlybirdFieldConstant.TEXT_FIELD.getFieldName();\n    }\n\n    String termText = special.getSpecialChar() + special.getValue();\n    Query q = createSimpleTermQuery(special, field, termText);\n\n    float boost = (float) getBoostFromAnnotations(special.getAnnotations());\n    if (boost >= 0) {\n      q = BoostUtils.maybeWrapInBoostQuery(q, boost);\n    }\n\n    return negateQueryIfNodeNegated(special, q);\n  }\n\n  @Override\n  public Query visit(Link link) throws QueryParserException {\n    Query q = createSimpleTermQuery(\n        link, EarlybirdFieldConstant.LINKS_FIELD.getFieldName(), link.getOperand());\n\n    float boost = (float) getBoostFromAnnotations(link.getAnnotations());\n    if (boost >= 0) {\n      q = BoostUtils.maybeWrapInBoostQuery(q, boost);\n    }\n\n    return negateQueryIfNodeNegated(link, q);\n  }\n\n  @Override\n  public Query visit(final SearchOperator op) throws QueryParserException {\n    final Query query;\n    SearchOperator.Type type = op.getOperatorType();\n\n    switch (type) {\n      case TO:\n        query = visitToOperator(op);\n        break;\n\n      case FROM:\n        query = visitFromOperator(op);\n        break;\n\n      case FILTER:\n        query = visitFilterOperator(op);\n        break;\n\n      case INCLUDE:\n        query = visitIncludeOperator(op);\n        break;\n\n      case EXCLUDE:\n        query = visitExcludeOperator(op);\n        break;\n\n      case LANG:\n        query = visitLangOperator(op);\n        break;\n\n      case SOURCE:\n        query = visitSourceOperator(op);\n        break;\n\n      case SMILEY:\n        query = visitSmileyOperator(op);\n        break;\n\n      case DOCVAL_RANGE_FILTER:\n        query = visitDocValRangeFilterOperator(op);\n        break;\n\n      case CACHED_FILTER:\n        query = visitCachedFilterOperator(op);\n        break;\n\n      case SCORE_FILTER:\n        query = visitScoredFilterOperator(op);\n        break;\n\n      case SINCE_TIME:\n        query = visitSinceTimeOperator(op);\n        break;\n\n      case UNTIL_TIME:\n        query = visitUntilTimeOperator(op);\n        break;\n\n      case SINCE_ID:\n        query = visitSinceIDOperator(op);\n        break;\n\n      case MAX_ID:\n        query = visitMaxIDOperator(op);\n        break;\n\n      case GEOLOCATION_TYPE:\n        query = visitGeoLocationTypeOperator(op);\n        break;\n\n      case GEOCODE:\n        query = visitGeocodeOperator(op);\n        break;\n\n      case GEO_BOUNDING_BOX:\n        query = visitGeoBoundingBoxOperator(op);\n        break;\n\n      case PLACE:\n        query = visitPlaceOperator(op);\n        break;\n\n      case LINK:\n        // This should never be called - the Link visitor (visitor(Link link)) should be.\n        query = visitLinkOperator(op);\n        break;\n\n      case ENTITY_ID:\n        query = visitEntityIdOperator(op);\n        break;\n\n      case FROM_USER_ID:\n        query = visitFromUserIDOperator(op);\n        break;\n\n      case IN_REPLY_TO_TWEET_ID:\n        query = visitInReplyToTweetIdOperator(op);\n        break;\n\n      case IN_REPLY_TO_USER_ID:\n        query = visitInReplyToUserIdOperator(op);\n        break;\n\n      case LIKED_BY_USER_ID:\n        query = visitLikedByUserIdOperator(op);\n        break;\n\n      case RETWEETED_BY_USER_ID:\n        query = visitRetweetedByUserIdOperator(op);\n        break;\n\n      case REPLIED_TO_BY_USER_ID:\n        query = visitRepliedToByUserIdOperator(op);\n        break;\n\n      case QUOTED_USER_ID:\n        query = visitQuotedUserIdOperator(op);\n        break;\n\n      case QUOTED_TWEET_ID:\n        query = visitQuotedTweetIdOperator(op);\n        break;\n\n      case DIRECTED_AT_USER_ID:\n        query = visitDirectedAtUserIdOperator(op);\n        break;\n\n      case CONVERSATION_ID:\n        query = visitConversationIdOperator(op);\n        break;\n\n      case COMPOSER_SOURCE:\n        query = visitComposerSourceOperator(op);\n        break;\n\n      case RETWEETS_OF_TWEET_ID:\n        query = visitRetweetsOfTweetIdOperator(op);\n        break;\n\n      case RETWEETS_OF_USER_ID:\n        query = visitRetweetsOfUserIdOperator(op);\n        break;\n\n      case LINK_CATEGORY:\n        query = visitLinkCategoryOperator(op);\n        break;\n\n      case CARD_NAME:\n        query = visitCardNameOperator(op);\n        break;\n\n      case CARD_DOMAIN:\n        query = visitCardDomainOperator(op);\n        break;\n\n      case CARD_LANG:\n        query = visitCardLangOperator(op);\n        break;\n\n      case HF_TERM_PAIR:\n        query = visitHFTermPairOperator(op);\n        break;\n\n      case HF_PHRASE_PAIR:\n        query = visitHFTermPhrasePairOperator(op);\n        break;\n\n      case PROXIMITY_GROUP:\n        Phrase phrase = new Phrase(\n            Lists.transform(op.getOperands(),\n                            s -> NormalizerHelper.normalizeWithUnknownLocale(\n                                s, EarlybirdConfig.getPenguinVersion())));\n\n        query = visit(phrase, true);\n        break;\n\n      case MULTI_TERM_DISJUNCTION:\n        query = visitMultiTermDisjunction(op);\n        break;\n\n      case CSF_DISJUNCTION_FILTER:\n        query = visitCSFDisjunctionFilter(op);\n        break;\n\n      case SAFETY_EXCLUDE:\n        query = visitSafetyExclude(op);\n        break;\n\n      case SPACE_ID:\n        query = visitSpaceId(op);\n        break;\n\n      case NAMED_ENTITY:\n        query = visitNamedEntity(op);\n        break;\n\n      case NAMED_ENTITY_WITH_TYPE:\n        query = visitNamedEntityWithType(op);\n        break;\n\n      case MIN_FAVES:\n      case MIN_QUALITY_SCORE:\n      case MIN_REPLIES:\n      case MIN_RETWEETS:\n      case MIN_REPUTATION:\n        query = visitMinFeatureValueOperator(type, op);\n        break;\n\n      case FEATURE_VALUE_IN_ACCEPT_LIST_OR_UNSET:\n        query = visitFeatureValueInAcceptListOrUnsetFilterOperator(op);\n        break;\n\n      case NEAR:\n      case RELATED_TO_TWEET_ID:\n      case SINCE:\n      case SITE:\n      case UNTIL:\n      case WITHIN:\n      case WITHIN_TIME:\n        query = createUnsupportedOperatorQuery(op);\n        break;\n\n      case NAMED_CSF_DISJUNCTION_FILTER:\n      case NAMED_MULTI_TERM_DISJUNCTION:\n        query = logAndThrowQueryParserException(\n            \"Named disjunction operator could not be converted to a disjunction operator.\");\n        break;\n\n      default:\n        query = logAndThrowQueryParserException(\"Unknown operator \" + op.toString());\n    }\n\n    return negateQueryIfNodeNegated(op, query);\n  }\n\n  protected Query visitToOperator(SearchOperator op) throws QueryParserException {\n    return createNormalizedTermQuery(\n        op, EarlybirdFieldConstant.TO_USER_FIELD.getFieldName(), op.getOperand());\n  }\n\n  protected Query visitFromOperator(SearchOperator op) throws QueryParserException {\n    return createNormalizedTermQuery(\n        op, EarlybirdFieldConstant.FROM_USER_FIELD.getFieldName(), op.getOperand());\n  }\n\n  protected Query visitFilterOperator(SearchOperator op) throws QueryParserException {\n    return visitFilterOperator(op, false);\n  }\n\n  protected Query visitIncludeOperator(SearchOperator op) throws QueryParserException {\n    // Include is a bit funny.  If we have [include retweets] we are saying\n    // do include retweets, which is the default.  Also conjunctions re-negate\n    // whatever node we emit from the visitor.\n    if (!isParentNegated(op) && !nodeIsNegated(op)) {\n      // positive include - no-op.\n      return null;\n    }\n    return visitFilterOperator(op, false);\n  }\n\n  protected Query visitExcludeOperator(SearchOperator op) throws QueryParserException {\n    // Exclude is a bit funny.  If we have -[exclude retweets] we are saying\n    // dont exclude retweets, which is the default.\n    if (isParentNegated(op) || nodeIsNegated(op)) {\n      // Negative exclude.  Do nothing - parent will not add this to the list of children.\n      return null;\n    } else {\n      // Positive exclude.\n      return visitFilterOperator(op, true);\n    }\n  }\n\n  protected Query visitFilterOperator(SearchOperator op, boolean negate)\n      throws QueryParserException {\n    Query q;\n    boolean negateQuery = negate;\n\n    if (op.getOperand().equals(SearchOperatorConstants.ANTISOCIAL)) {\n      // Since the object we use to implement these filters is actually an\n      // EXCLUDE filter, we need to negate it to get it to work as a regular filter.\n      q = UserFlagsExcludeFilter.getUserFlagsExcludeFilter(userTable, true, false, false);\n      negateQuery = !negateQuery;\n    } else if (op.getOperand().equals(SearchOperatorConstants.OFFENSIVE_USER)) {\n      q = UserFlagsExcludeFilter.getUserFlagsExcludeFilter(userTable, false, true, false);\n      negateQuery = !negateQuery;\n    } else if (op.getOperand().equals(SearchOperatorConstants.ANTISOCIAL_OFFENSIVE_USER)) {\n      q = UserFlagsExcludeFilter.getUserFlagsExcludeFilter(userTable, true, true, false);\n      negateQuery = !negateQuery;\n    } else if (op.getOperand().equals(SearchOperatorConstants.PROTECTED)) {\n      q = UserFlagsExcludeFilter.getUserFlagsExcludeFilter(userTable, false, false, true);\n      negateQuery = !negateQuery;\n    } else if (op.getOperand().equals(SearchOperatorConstants.HAS_ENGAGEMENT)) {\n      return buildHasEngagementsQuery();\n    } else if (op.getOperand().equals(SearchOperatorConstants.SAFE_SEARCH_FILTER)) {\n      BooleanQuery.Builder bqBuilder = new BooleanQuery.Builder();\n      bqBuilder.add(\n          createNoScoreTermQuery(\n              op,\n              EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n              EarlybirdFieldConstant.IS_OFFENSIVE),\n          Occur.SHOULD);\n\n      // The following internal field __filter_sensitive_content\n      // is not currently built by earlybird.\n      // This means the safe search filter soley operates on the is_offensive bit\n      bqBuilder.add(\n          createNoScoreTermQuery(\n              op,\n              EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n              EarlybirdThriftDocumentUtil.formatFilter(SearchOperatorConstants.SENSITIVE_CONTENT)),\n          Occur.SHOULD);\n      q = bqBuilder.build();\n      negateQuery = !negateQuery;\n    } else if (op.getOperand().equals(SearchOperatorConstants.RETWEETS)) {\n      // Special case for filter:retweets - we use the text field search \"-rt\"\n      // mostly for legacy reasons.\n      q = createSimpleTermQuery(\n          op,\n          EarlybirdFieldConstant.TEXT_FIELD.getFieldName(),\n          EarlybirdThriftDocumentBuilder.RETWEET_TERM);\n    } else if (schemaSnapshot.getFacetFieldByFacetName(op.getOperand()) != null) {\n      Schema.FieldInfo facetField = schemaSnapshot.getFacetFieldByFacetName(op.getOperand());\n      if (facetField.getFieldType().isStoreFacetSkiplist()) {\n        q = createSimpleTermQuery(\n            op,\n            EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n            EarlybirdFieldConstant.getFacetSkipFieldName(facetField.getName()));\n      } else {\n        // return empty BQ that doesn't match anything\n        q = new BooleanQuery.Builder().build();\n      }\n    } else if (op.getOperand().equals(SearchOperatorConstants.VINE_LINK)) {\n      // Temporary special case for filter:vine_link. The filter is called \"vine_link\", but it\n      // should use the internal field \"__filter_vine\". We need this special case because otherwise\n      // it would look for the non-existing \"__filter_vine_link\" field. See SEARCH-9390\n      q = createNoScoreTermQuery(\n          op,\n          EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n          EarlybirdThriftDocumentUtil.formatFilter(\"vine\"));\n    } else {\n      // The default vanilla filters just uses the filter format string and the\n      // operand text.\n      q = createNoScoreTermQuery(\n          op,\n          EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n          EarlybirdThriftDocumentUtil.formatFilter(op.getOperand()));\n    }\n    // Double check: no filters should have any score contribution.\n    q = new BoostQuery(q, 0.0f);\n    return negateQuery ? negateQuery(q) : q;\n  }\n\n  private Query buildHasEngagementsQuery() {\n    if (earlybirdCluster == EarlybirdCluster.PROTECTED) {\n      // Engagements and engagement counts are not indexed on Earlybirds, so there is no need to\n      // traverse the entire segment with the MinFeatureValueFilter. See SEARCH-28120\n      return new MatchNoDocsQuery();\n    }\n\n    Query favFilter = MinFeatureValueFilter.getMinFeatureValueFilter(\n        EarlybirdFieldConstant.FAVORITE_COUNT.getFieldName(), 1);\n    Query retweetFilter = MinFeatureValueFilter.getMinFeatureValueFilter(\n        EarlybirdFieldConstant.RETWEET_COUNT.getFieldName(), 1);\n    Query replyFilter = MinFeatureValueFilter.getMinFeatureValueFilter(\n        EarlybirdFieldConstant.REPLY_COUNT.getFieldName(), 1);\n    return new BooleanQuery.Builder()\n        .add(favFilter, Occur.SHOULD)\n        .add(retweetFilter, Occur.SHOULD)\n        .add(replyFilter, Occur.SHOULD)\n        .build();\n  }\n\n  protected Query visitLangOperator(SearchOperator op) throws QueryParserException {\n    return createNoScoreTermQuery(\n        op, EarlybirdFieldConstant.ISO_LANGUAGE_FIELD.getFieldName(), op.getOperand());\n  }\n\n  protected Query visitSourceOperator(SearchOperator op) throws QueryParserException {\n    return createNoScoreTermQuery(\n        op, EarlybirdFieldConstant.NORMALIZED_SOURCE_FIELD.getFieldName(), op.getOperand());\n  }\n\n  protected Query visitSmileyOperator(SearchOperator op) throws QueryParserException {\n    return createSimpleTermQuery(\n        op,\n        EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n        String.format(SMILEY_FORMAT_STRING, op.getOperand()));\n  }\n\n  protected Query visitDocValRangeFilterOperator(SearchOperator op) throws QueryParserException {\n    String csfFieldName = op.getOperands().get(0).toLowerCase();\n\n    ThriftCSFType csfFieldType = schemaSnapshot.getCSFFieldType(csfFieldName);\n    if (csfFieldType == null) {\n      throw new QueryParserException(\"invalid csf field name \" + op.getOperands().get(0)\n          + \" used in \" + op.serialize());\n    }\n\n    try {\n      if (csfFieldType == ThriftCSFType.DOUBLE\n          || csfFieldType == ThriftCSFType.FLOAT) {\n        return DocValRangeFilter.getDocValRangeQuery(csfFieldName, csfFieldType,\n            Double.parseDouble(op.getOperands().get(1)),\n            Double.parseDouble(op.getOperands().get(2)));\n      } else if (csfFieldType == ThriftCSFType.LONG\n          || csfFieldType == ThriftCSFType.INT\n          || csfFieldType == ThriftCSFType.BYTE) {\n        Query query = DocValRangeFilter.getDocValRangeQuery(csfFieldName, csfFieldType,\n            Long.parseLong(op.getOperands().get(1)),\n            Long.parseLong(op.getOperands().get(2)));\n        if (csfFieldName.equals(EarlybirdFieldConstant.LAT_LON_CSF_FIELD.getFieldName())) {\n          return wrapQueryInUserScrubGeoFilter(query);\n        }\n        return query;\n      } else {\n        throw new QueryParserException(\"invalid ThriftCSFType. drop this op: \" + op.serialize());\n      }\n    } catch (NumberFormatException e) {\n      throw new QueryParserException(\"invalid range numeric type used in \" + op.serialize());\n    }\n  }\n\n  protected final Query visitCachedFilterOperator(SearchOperator op) throws QueryParserException {\n    try {\n      return CachedFilterQuery.getCachedFilterQuery(op.getOperand(), queryCacheManager);\n    } catch (CachedFilterQuery.NoSuchFilterException e) {\n      throw new QueryParserException(e.getMessage(), e);\n    }\n  }\n\n  protected final Query visitScoredFilterOperator(SearchOperator op) throws QueryParserException {\n    final List<String> operands = op.getOperands();\n    final String scoreFunction = operands.get(0);\n    ScoringFunctionProvider.NamedScoringFunctionProvider scoringFunctionProvider =\n      ScoringFunctionProvider.getScoringFunctionProviderByName(scoreFunction, schemaSnapshot);\n    if (scoringFunctionProvider == null) {\n      throw new QueryParserException(\"Unknown scoring function name [\" + scoreFunction\n          + \" ] used as score_filter's operand\");\n    }\n\n    return ScoreFilterQuery.getScoreFilterQuery(\n        schemaSnapshot,\n        scoringFunctionProvider,\n        Float.parseFloat(operands.get(1)),\n        Float.parseFloat(operands.get(2)));\n  }\n\n  protected Query visitSinceTimeOperator(SearchOperator op) {\n    try {\n      return SinceUntilFilter.getSinceQuery(Integer.parseInt(op.getOperand()));\n    } catch (NumberFormatException e) {\n      LOG.warn(\"since time is not a valid integer, the date isn't reasonable. drop this op: \"\n          + op.serialize());\n      SINCE_TIME_INVALID_INT_COUNTER.increment();\n      return null;\n    }\n  }\n\n  protected Query visitUntilTimeOperator(SearchOperator op) {\n    try {\n      return SinceUntilFilter.getUntilQuery(Integer.parseInt(op.getOperand()));\n    } catch (NumberFormatException e) {\n      LOG.warn(\"until time is not a valid integer, the date isn't reasonable. drop this op: \"\n          + op.serialize());\n      UNTIL_TIME_INVALID_INT_COUNTER.increment();\n      return null;\n    }\n  }\n\n  protected Query visitSinceIDOperator(SearchOperator op) {\n    long id = Long.parseLong(op.getOperand());\n    return SinceMaxIDFilter.getSinceIDQuery(id);\n  }\n\n  protected Query visitMaxIDOperator(SearchOperator op) {\n    long id = Long.parseLong(op.getOperand());\n    return SinceMaxIDFilter.getMaxIDQuery(id);\n  }\n\n  protected Query visitGeoLocationTypeOperator(SearchOperator op) throws QueryParserException {\n    String operand = op.getOperand();\n    ThriftGeoLocationSource source = ThriftGeoLocationSource.valueOf(operand.toUpperCase());\n    // If necessary, this query will be wrapped by the UserScrubGeoFilter within\n    // the createSimpleTermQuery() helper method\n    return createNoScoreTermQuery(\n        op,\n        EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName(),\n        EarlybirdFieldConstants.formatGeoType(source));\n  }\n\n  protected Query visitGeocodeOperator(SearchOperator op) throws QueryParserException {\n    return visitGeocodeOrGeocodePrivateOperator(op);\n  }\n\n  protected Query visitGeoBoundingBoxOperator(SearchOperator op) throws QueryParserException {\n    Rectangle rectangle = boundingBoxFromSearchOperator(op);\n    return wrapQueryInUserScrubGeoFilter(\n        GeoQuadTreeQueryBuilder.buildGeoQuadTreeQuery(rectangle, terminationTracker));\n  }\n\n  protected Query visitPlaceOperator(SearchOperator op) throws QueryParserException {\n    // This query will be wrapped by the UserScrubGeoFilter within the createSimpleTermQuery()\n    // helper method\n    return createSimpleTermQuery(\n        op, EarlybirdFieldConstant.PLACE_FIELD.getFieldName(), op.getOperand());\n  }\n\n  protected Query visitLinkOperator(SearchOperator op) throws QueryParserException {\n    // This should never be called - the Link visitor (visitor(Link link)) should be.\n    if (op instanceof Link) {\n      LOG.warn(\"Unexpected Link operator \" + op.serialize());\n      return visit((Link) op);\n    } else {\n      throw new QueryParserException(\"Operator type set to \" + op.getOperatorName()\n          + \" but it is not an instance of Link [\" + op.toString() + \"]\");\n    }\n  }\n\n  protected Query visitEntityIdOperator(SearchOperator op) throws QueryParserException {\n    return createSimpleTermQuery(\n        op, EarlybirdFieldConstant.ENTITY_ID_FIELD.getFieldName(), op.getOperand());\n  }\n\n  protected Query visitFromUserIDOperator(SearchOperator op) {\n    return buildLongTermAttributeQuery(\n        op, EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName());\n  }\n\n  protected Query visitInReplyToTweetIdOperator(SearchOperator op) {\n    return buildLongTermAttributeQuery(\n        op, EarlybirdFieldConstant.IN_REPLY_TO_TWEET_ID_FIELD.getFieldName());\n  }\n\n  protected Query visitInReplyToUserIdOperator(SearchOperator op) {\n    return buildLongTermAttributeQuery(\n        op, EarlybirdFieldConstant.IN_REPLY_TO_USER_ID_FIELD.getFieldName());\n  }\n\n  protected Query visitLikedByUserIdOperator(SearchOperator op) throws QueryParserException {\n    return buildLongTermAttributeQuery(op,\n        EarlybirdFieldConstant.LIKED_BY_USER_ID_FIELD.getFieldName());\n  }\n\n  protected Query visitRetweetedByUserIdOperator(SearchOperator op) throws QueryParserException {\n    return buildLongTermAttributeQuery(op,\n        EarlybirdFieldConstant.RETWEETED_BY_USER_ID.getFieldName());\n  }\n\n  protected Query visitRepliedToByUserIdOperator(SearchOperator op) throws QueryParserException {\n    return buildLongTermAttributeQuery(op,\n        EarlybirdFieldConstant.REPLIED_TO_BY_USER_ID.getFieldName());\n  }\n\n  protected Query visitQuotedUserIdOperator(SearchOperator op) throws QueryParserException {\n    return buildLongTermAttributeQuery(op,\n        EarlybirdFieldConstant.QUOTED_USER_ID_FIELD.getFieldName());\n  }\n\n  protected Query visitQuotedTweetIdOperator(SearchOperator op) throws QueryParserException {\n    return buildLongTermAttributeQuery(op,\n        EarlybirdFieldConstant.QUOTED_TWEET_ID_FIELD.getFieldName());\n  }\n\n  protected Query visitDirectedAtUserIdOperator(SearchOperator op) throws QueryParserException {\n    return buildLongTermAttributeQuery(op,\n        EarlybirdFieldConstant.DIRECTED_AT_USER_ID_FIELD.getFieldName());\n  }\n\n  protected Query visitConversationIdOperator(SearchOperator op) throws QueryParserException {\n    return buildLongTermAttributeQuery(\n        op, EarlybirdFieldConstant.CONVERSATION_ID_FIELD.getFieldName());\n  }\n\n  protected Query visitComposerSourceOperator(SearchOperator op) throws QueryParserException {\n    Preconditions.checkNotNull(op.getOperand(), \"composer_source requires operand\");\n    try {\n      ComposerSource composerSource = ComposerSource.valueOf(op.getOperand().toUpperCase());\n      return buildNoScoreIntTermQuery(\n          op, EarlybirdFieldConstant.COMPOSER_SOURCE, composerSource.getValue());\n    } catch (IllegalArgumentException e) {\n      throw new QueryParserException(\"Invalid operand for composer_source: \" + op.getOperand(), e);\n    }\n  }\n\n  protected Query visitRetweetsOfTweetIdOperator(SearchOperator op) {\n    return buildLongTermAttributeQuery(\n        op, EarlybirdFieldConstant.RETWEET_SOURCE_TWEET_ID_FIELD.getFieldName());\n  }\n\n  protected Query visitRetweetsOfUserIdOperator(SearchOperator op) {\n    return buildLongTermAttributeQuery(\n        op, EarlybirdFieldConstant.RETWEET_SOURCE_USER_ID_FIELD.getFieldName());\n  }\n\n  protected Query visitLinkCategoryOperator(SearchOperator op) {\n    int linkCategory;\n    try {\n      linkCategory = LinkCategory.valueOf(op.getOperand()).getValue();\n    } catch (IllegalArgumentException e) {\n      linkCategory = Integer.parseInt(op.getOperand());\n    }\n\n    String fieldName = EarlybirdFieldConstant.LINK_CATEGORY_FIELD.getFieldName();\n    org.apache.lucene.index.Term term = new org.apache.lucene.index.Term(\n        fieldName, IntTermAttributeImpl.copyIntoNewBytesRef(linkCategory));\n    return wrapQuery(\n        new TermQueryWithSafeToString(term, Integer.toString(linkCategory)), op, fieldName);\n  }\n\n  protected Query visitCardNameOperator(SearchOperator op) throws QueryParserException {\n    return createNoScoreTermQuery(\n        op, EarlybirdFieldConstant.CARD_NAME_FIELD.getFieldName(), op.getOperand());\n  }\n\n  protected Query visitCardDomainOperator(SearchOperator op) throws QueryParserException {\n    return createNoScoreTermQuery(\n        op, EarlybirdFieldConstant.CARD_DOMAIN_FIELD.getFieldName(), op.getOperand());\n  }\n\n  protected Query visitCardLangOperator(SearchOperator op) throws QueryParserException {\n    return createNoScoreTermQuery(\n        op, EarlybirdFieldConstant.CARD_LANG.getFieldName(), op.getOperand());\n  }\n\n  protected Query visitHFTermPairOperator(SearchOperator op) throws QueryParserException {\n    final List<String> operands = op.getOperands();\n    String termPair = HighFrequencyTermPairs.createPair(op.getOperands().get(0),\n        op.getOperands().get(1));\n    Query q = createSimpleTermQuery(op, ImmutableSchema.HF_TERM_PAIRS_FIELD, termPair);\n    float boost = Float.parseFloat(operands.get(2));\n    if (boost >= 0) {\n      q = BoostUtils.maybeWrapInBoostQuery(q, boost);\n    }\n    return q;\n  }\n\n  protected Query visitHFTermPhrasePairOperator(SearchOperator op) throws QueryParserException {\n    final List<String> operands = op.getOperands();\n    String termPair = HighFrequencyTermPairs.createPhrasePair(op.getOperands().get(0),\n                                                              op.getOperands().get(1));\n    Query q = createSimpleTermQuery(op, ImmutableSchema.HF_PHRASE_PAIRS_FIELD, termPair);\n    float boost = Float.parseFloat(operands.get(2));\n    if (boost >= 0) {\n      q = BoostUtils.maybeWrapInBoostQuery(q, boost);\n    }\n    return q;\n  }\n\n  private Query logAndThrowQueryParserException(String message) throws QueryParserException {\n    LOG.error(message);\n    throw new QueryParserException(message);\n  }\n\n  private Query logMissingEntriesAndThrowQueryParserException(String field, SearchOperator op)\n      throws QueryParserException {\n    return logAndThrowQueryParserException(\n        String.format(\"Missing required %s entries for %s\", field, op.serialize()));\n  }\n\n  // previous implementation of this operator allowed insertion of\n  // operands from the thrift search query.  This was reverted to ensure simplicity\n  // of the api, and to keep the serialized query self contained.\n  protected final Query visitMultiTermDisjunction(SearchOperator op) throws QueryParserException {\n    final List<String> operands = op.getOperands();\n    final String field = operands.get(0);\n\n    if (isUserIdField(field)) {\n      List<Long> ids = Lists.newArrayList();\n      parseLongArgs(operands.subList(1, operands.size()), ids, op);\n      if (ids.size() > 0) {\n        // Try to get ranks for ids if exist from hitAttributeHelper.\n        // Otherwise just pass in a empty list.\n        List<Integer> ranks;\n        if (hitAttributeHelper != null\n            && hitAttributeHelper.getExpandedNodeToRankMap().containsKey(op)) {\n          ranks = hitAttributeHelper.getExpandedNodeToRankMap().get(op);\n        } else {\n          ranks = Lists.newArrayList();\n        }\n        return UserIdMultiSegmentQuery.createIdDisjunctionQuery(\n            \"multi_term_disjunction_\" + field,\n            ids,\n            field,\n            schemaSnapshot,\n            multiSegmentTermDictionaryManager,\n            decider,\n            earlybirdCluster,\n            ranks,\n            hitAttributeHelper,\n            queryTimeout);\n      } else {\n        return logMissingEntriesAndThrowQueryParserException(field, op);\n      }\n    } else if (EarlybirdFieldConstant.ID_FIELD.getFieldName().equals(field)) {\n      List<Long> ids = Lists.newArrayList();\n      parseLongArgs(operands.subList(1, operands.size()), ids, op);\n      if (ids.size() > 0) {\n        return RequiredStatusIDsFilter.getRequiredStatusIDsQuery(ids);\n      } else {\n        return logMissingEntriesAndThrowQueryParserException(field, op);\n      }\n    } else if (isTweetIdField(field)) {\n      List<Long> ids = Lists.newArrayList();\n      parseLongArgs(operands.subList(1, operands.size()), ids, op);\n      if (ids.size() > 0) {\n        BooleanQuery.Builder bqBuilder = new BooleanQuery.Builder();\n        int numClauses = 0;\n        for (long id : ids) {\n          if (numClauses >= BooleanQuery.getMaxClauseCount()) {\n            BooleanQuery saved = bqBuilder.build();\n            bqBuilder = new BooleanQuery.Builder();\n            bqBuilder.add(saved, BooleanClause.Occur.SHOULD);\n            numClauses = 1;\n          }\n          bqBuilder.add(buildLongTermAttributeQuery(op, field, id), Occur.SHOULD);\n          ++numClauses;\n        }\n        return bqBuilder.build();\n      } else {\n        return logMissingEntriesAndThrowQueryParserException(field, op);\n      }\n    } else {\n      return createUnsupportedOperatorQuery(op);\n    }\n  }\n\n  protected final Query visitCSFDisjunctionFilter(SearchOperator op)\n      throws QueryParserException {\n    List<String> operands = op.getOperands();\n    String field = operands.get(0);\n\n    ThriftCSFType csfType = schemaSnapshot.getCSFFieldType(field);\n    if (csfType == null) {\n      throw new QueryParserException(\"Field must be a CSF\");\n    }\n\n    if (csfType != ThriftCSFType.LONG) {\n      throw new QueryParserException(\"csf_disjunction_filter only works with long fields\");\n    }\n\n    Set<Long> values = new HashSet<>();\n    parseLongArgs(operands.subList(1, operands.size()), values, op);\n\n    Query query = CSFDisjunctionFilter.getCSFDisjunctionFilter(field, values);\n    if (field.equals(EarlybirdFieldConstant.LAT_LON_CSF_FIELD.getFieldName())) {\n      return wrapQueryInUserScrubGeoFilter(query);\n    }\n    return query;\n  }\n\n  protected Query visitSafetyExclude(SearchOperator op) throws QueryParserException {\n    // We do not allow negating safety_exclude operator. Note the operator is internal so if we\n    // get here, it means there's a bug in the query construction side.\n    if (isParentNegated(op) || nodeIsNegated(op)) {\n      throw new QueryParserException(\"Negating safety_exclude operator is not allowed: \" + op);\n    }\n\n    // Convert the safety filter to other operators depending on cluster setting\n    // The safety filter is interpreted differently on archive because the underlying safety labels\n    // in extended encoded field are not available on archive.\n    if (EarlybirdCluster.isArchive(earlybirdCluster)) {\n      return visit(OPERATOR_CACHED_EXCLUDE_ANTISOCIAL_AND_NATIVERETWEETS);\n    } else {\n      List<com.twitter.search.queryparser.query.Query> children = Lists.newArrayList();\n      for (String filterName : op.getOperands()) {\n        children.addAll(\n            OPERATORS_BY_SAFE_EXCLUDE_OPERAND.getOrDefault(filterName, ImmutableList.of()));\n      }\n      return visit(new Conjunction(children));\n    }\n  }\n\n  protected Query visitNamedEntity(SearchOperator op) throws QueryParserException {\n    List<String> operands = op.getOperands();\n    Preconditions.checkState(operands.size() == 1,\n        \"named_entity: wrong number of operands\");\n\n    return createDisjunction(\n        operands.get(0).toLowerCase(),\n        op,\n        EarlybirdFieldConstant.NAMED_ENTITY_FROM_TEXT_FIELD,\n        EarlybirdFieldConstant.NAMED_ENTITY_FROM_URL_FIELD);\n  }\n\n  protected Query visitSpaceId(SearchOperator op) throws QueryParserException {\n    List<String> operands = op.getOperands();\n    Preconditions.checkState(operands.size() == 1,\n        \"space_id: wrong number of operands\");\n\n    return createSimpleTermQuery(\n        op,\n        EarlybirdFieldConstant.SPACE_ID_FIELD.getFieldName(),\n        op.getOperand()\n    );\n  }\n\n  protected Query visitNamedEntityWithType(SearchOperator op) throws QueryParserException {\n    List<String> operands = op.getOperands();\n    Preconditions.checkState(operands.size() == 2,\n        \"named_entity_with_type: wrong number of operands\");\n\n    String name = operands.get(0);\n    String type = operands.get(1);\n    return createDisjunction(\n        String.format(\"%s:%s\", name, type).toLowerCase(),\n        op,\n        EarlybirdFieldConstant.NAMED_ENTITY_WITH_TYPE_FROM_TEXT_FIELD,\n        EarlybirdFieldConstant.NAMED_ENTITY_WITH_TYPE_FROM_URL_FIELD);\n  }\n\n  // Create a disjunction query for a given value in one of the given fields\n  private Query createDisjunction(\n      String value, SearchOperator operator, EarlybirdFieldConstant... fields)\n      throws QueryParserException {\n    BooleanQuery.Builder booleanQueryBuilder = new BooleanQuery.Builder();\n    for (EarlybirdFieldConstant field : fields) {\n      booleanQueryBuilder.add(\n          createSimpleTermQuery(operator, field.getFieldName(), value), Occur.SHOULD);\n    }\n    return booleanQueryBuilder.build();\n  }\n\n  protected Query visitMinFeatureValueOperator(SearchOperator.Type type, SearchOperator op) {\n    final List<String> operands = op.getOperands();\n\n    String featureName;\n    switch (type) {\n      case MIN_FAVES:\n        featureName = EarlybirdFieldConstant.FAVORITE_COUNT.getFieldName();\n        break;\n      case MIN_QUALITY_SCORE:\n        featureName = EarlybirdFieldConstant.PARUS_SCORE.getFieldName();\n        break;\n      case MIN_REPLIES:\n        featureName = EarlybirdFieldConstant.REPLY_COUNT.getFieldName();\n        break;\n      case MIN_REPUTATION:\n        featureName = EarlybirdFieldConstant.USER_REPUTATION.getFieldName();\n        break;\n      case MIN_RETWEETS:\n        featureName = EarlybirdFieldConstant.RETWEET_COUNT.getFieldName();\n        break;\n      default:\n        throw new IllegalArgumentException(\"Unknown min feature type \" + type);\n    }\n\n    double operand = Double.parseDouble(operands.get(0));\n\n    // SEARCH-16751: Because we use QueryCacheConstants.HAS_ENGAGEMENT as a driving query below, we\n    // won't return tweets with 0 engagements when we handle a query with a [min_X 0] filter (e.g.\n    // (* cat [min_faves 0] ). Thus we need to return a MatchAllDocsQuery in that case.\n    if (operand == 0) {\n      return new MatchAllDocsQuery();\n    }\n\n    // Only perform the rewrite if the operator is a min engagement operator.\n    if (isOperatorTypeEngagementFilter(type)) {\n      return buildQueryForEngagementOperator(op, operands, featureName);\n    }\n\n    if (type == SearchOperator.Type.MIN_REPUTATION) {\n      return buildQueryForMinReputationOperator(operands, featureName);\n    }\n\n    return MinFeatureValueFilter.getMinFeatureValueFilter(\n        featureName, Double.parseDouble(operands.get(0)));\n  }\n\n  protected Query visitFeatureValueInAcceptListOrUnsetFilterOperator(SearchOperator op)\n      throws QueryParserException {\n    final List<String> operands = op.getOperands();\n    final String field = operands.get(0);\n\n    if (isIdCSFField(field)) {\n      Set<Long> ids = Sets.newHashSet();\n      parseLongArgs(operands.subList(1, operands.size()), ids, op);\n      return FeatureValueInAcceptListOrUnsetFilter.getFeatureValueInAcceptListOrUnsetFilter(\n          field, ids);\n    } else {\n      return logAndThrowQueryParserException(\n          \"Invalid CSF field passed to operator \" + op.toString());\n    }\n  }\n\n  /**\n   * Creates a Lucene query for an operator that's not supported by the search service.\n   *\n   * NOTE: Developer, if you are writing a class to extends this class, make sure the\n   * behaviour of this function makes sense for your search service.\n   *\n   * @param op The operator that's not supported by the search service.\n   * @return The Lucene query for this operator\n   */\n  protected Query createUnsupportedOperatorQuery(SearchOperator op) throws QueryParserException {\n    SearchCounter\n        .export(UNSUPPORTED_OPERATOR_PREFIX + op.getOperatorType().getOperatorName())\n        .increment();\n    return visit(op.toPhrase());\n  }\n\n  private Query buildNoScoreIntTermQuery(\n      SearchOperator op,\n      EarlybirdFieldConstant field,\n      int termValue) {\n    org.apache.lucene.index.Term term = new org.apache.lucene.index.Term(\n        field.getFieldName(), IntTermAttributeImpl.copyIntoNewBytesRef(termValue));\n    return wrapQuery(\n        new TermQueryWithSafeToString(term, Integer.toString(termValue)), op, field.getFieldName());\n  }\n\n  private Query buildQueryForMinReputationOperator(List<String> operands, String featureName) {\n    int operand = (int) Double.parseDouble(operands.get(0));\n    // Driving by MinFeatureValueFilter's DocIdSetIterator is very slow, because we have to\n    // perform an expensive check for all doc IDs in the segment, so we use a cached result to\n    // drive the query, and use MinFeatureValueFilter as a secondary filter.\n    String queryCacheFilterName;\n    if (operand >= 50) {\n      queryCacheFilterName = QueryCacheConstants.MIN_REPUTATION_50;\n    } else if (operand >= 36) {\n      queryCacheFilterName = QueryCacheConstants.MIN_REPUTATION_36;\n    } else if (operand >= 30) {\n      queryCacheFilterName = QueryCacheConstants.MIN_REPUTATION_30;\n    } else {\n      return MinFeatureValueFilter.getMinFeatureValueFilter(featureName, operand);\n    }\n\n    try {\n      Query drivingQuery = CachedFilterQuery.getCachedFilterQuery(\n          queryCacheFilterName, queryCacheManager);\n      return new FilteredQuery(\n          drivingQuery, MinFeatureValueFilter.getDocIdFilterFactory(featureName, operand));\n    } catch (Exception e) {\n      // If the filter is not found, that's OK, it might be our first time running the query cache,\n      // or there may be no tweets with that high reputation.\n      return MinFeatureValueFilter.getMinFeatureValueFilter(featureName, operand);\n    }\n  }\n\n  private Query buildQueryForEngagementOperator(\n      SearchOperator op, List<String> operands, String featureName) {\n    // Engagements and engagement counts are not indexed on Protected Earlybirds, so there is no\n    // need to traverse the entire segment with the MinFeatureValueFilter. SEARCH-28120\n    if (earlybirdCluster == EarlybirdCluster.PROTECTED) {\n      return new MatchNoDocsQuery();\n    }\n\n    EarlybirdFieldConstant field =\n        EarlybirdFieldConstants.CSF_NAME_TO_MIN_ENGAGEMENT_FIELD_MAP.get(featureName);\n    if (field == null) {\n      throw new IllegalArgumentException(String.format(\"Expected the feature to be \"\n          + \"FAVORITE_COUNT, REPLY_COUNT, or RETWEET_COUNT. Got %s.\", featureName));\n    }\n    int operand = (int) Double.parseDouble(operands.get(0));\n    ByteNormalizer normalizer = MinFeatureValueFilter.getMinFeatureValueNormalizer(featureName);\n    int minValue = normalizer.unsignedByteToInt(normalizer.normalize(operand));\n\n    // We default to the old behavior of filtering posts instead of consulting the min engagement\n    // field if the operand is less than some threshold value because it seems, empirically, that\n    // the old method results in lower query latencies for lower values of the filter operand.\n    // This threshold can be controlled by the \"use_min_engagement_field_threshold\" decider. The\n    // current default value is 90. SEARCH-16102\n    int useMinEngagementFieldThreshold = decider.getAvailability(\n        \"use_min_engagement_field_threshold\").getOrElse(() -> 0);\n    if (operand >= useMinEngagementFieldThreshold) {\n      NUM_QUERIES_ABOVE_MIN_ENGAGEMENT_THRESHOLD.increment();\n    } else {\n      NUM_QUERIES_BELOW_MIN_ENGAGEMENT_THRESHOLD.increment();\n    }\n    if (schemaHasField(field) && operand >= useMinEngagementFieldThreshold) {\n      return buildNoScoreIntTermQuery(op, field, minValue);\n    }\n    // Driving by MinFeatureValueFilter's DocIdSetIterator is very slow, because we have to\n    // perform an expensive check for all doc IDs in the segment, so we use a cached result to\n    // drive the query, and use MinFeatureValueFilter as a secondary filter.\n    try {\n      Query drivingQuery = minEngagmentsDrivingQuery(op, operand);\n      return new FilteredQuery(\n          drivingQuery, MinFeatureValueFilter.getDocIdFilterFactory(featureName, operand));\n    } catch (Exception e) {\n      // If the filter is not found, that's OK, it might be our first time running the query cache,\n      // or there may be no Tweets with that many engagements (we would only expect this in tests).\n      return MinFeatureValueFilter.getMinFeatureValueFilter(featureName, operand);\n    }\n  }\n\n  private Query minEngagmentsDrivingQuery(SearchOperator operator, int minValue)\n          throws CachedFilterQuery.NoSuchFilterException, QueryParserException {\n    // If the min engagements value is large, then many of the hits that have engagement will still\n    // not match the query, leading to extremely slow queries. Therefore, if there is more than 100\n    // engagements, we drive by a more restricted filter. See SEARCH-33740\n    String filter;\n    if (minValue < 100) {\n      filter = QueryCacheConstants.HAS_ENGAGEMENT;\n    } else if (operator.getOperatorType() == SearchOperator.Type.MIN_FAVES) {\n      filter = QueryCacheConstants.MIN_FAVES_100;\n    } else if (operator.getOperatorType() == SearchOperator.Type.MIN_REPLIES) {\n      filter = QueryCacheConstants.MIN_REPLIES_100;\n    } else if (operator.getOperatorType() == SearchOperator.Type.MIN_RETWEETS) {\n      filter = QueryCacheConstants.MIN_RETWEETS_100;\n    } else {\n      throw new QueryParserException(\"Missing engagement filter.\");\n    }\n    return CachedFilterQuery.getCachedFilterQuery(filter, queryCacheManager);\n  }\n\n  private boolean isOperatorTypeEngagementFilter(SearchOperator.Type type) {\n    return type == SearchOperator.Type.MIN_FAVES\n        || type == SearchOperator.Type.MIN_RETWEETS\n        || type == SearchOperator.Type.MIN_REPLIES;\n  }\n\n  private boolean schemaHasField(EarlybirdFieldConstant field) {\n    return schemaSnapshot.hasField(field.getFieldId());\n  }\n\n  // Helper functions\n  private Query createSimpleTermQuery(\n      com.twitter.search.queryparser.query.Query node, String field, String text)\n      throws QueryParserException {\n    Query baseQuery = new TermQuery(createTerm(field, text));\n    if (isGeoFieldThatShouldBeScrubbed(field, text)) {\n      baseQuery = wrapQueryInUserScrubGeoFilter(baseQuery);\n    }\n    return wrapQuery(baseQuery, node, field);\n  }\n\n  private boolean isGeoFieldThatShouldBeScrubbed(String field, String text) {\n    if (field.equals(EarlybirdFieldConstant.INTERNAL_FIELD.getFieldName())) {\n      // the internal field is used for the place id filter and the geo location type filters, some\n      // of which should be scrubbed\n      return GEO_FILTERS_TO_BE_SCRUBBED.contains(text);\n    }\n    return GEO_FIELDS_TO_BE_SCRUBBED.contains(field);\n  }\n\n  // Like above, but sets boost to 0 to disable scoring component.  This should be used\n  // for filters that do not impact scoring (such as filter:images).\n  private Query createNoScoreTermQuery(com.twitter.search.queryparser.query.Query node,\n                                             String field, String text)\n      throws QueryParserException {\n    Query query = createSimpleTermQuery(node, field, text);\n    return new BoostQuery(query, 0.0f);  // No score contribution.\n  }\n\n  private Query createNormalizedTermQuery(com.twitter.search.queryparser.query.Query node,\n                                                String field, String text)\n      throws QueryParserException {\n    return createSimpleTermQuery(\n        node,\n        field,\n        NormalizerHelper.normalizeWithUnknownLocale(text, EarlybirdConfig.getPenguinVersion()));\n  }\n\n  /**\n   * Get the boost from the annotation list of a query node.\n   * Right now this is very simple, we simple extract the value of some annotations and ignore all\n   * others, also, if there are multiple annotations that have values, we only use the first one we\n   * see in the list (although the rewritten query EB receives should have this).\n   * NOTE: we use simple weight selection logic here based on the assumption that the annotator\n   * and rewriter will not produce ambiguous weight information. There should always be only one\n   * weight-bearing annotation for a specific node.\n   *\n   * @param annotations The list of annotations of the query node.\n   * @return The boost for this query node, 0 if there is no boost, in which case you shouldn't\n   *         apply it at all.\n   */\n  private static double getBoostFromAnnotations(List<Annotation> annotations) {\n    if (annotations != null) {\n      for (Annotation anno : annotations) {\n        switch (anno.getType()) {\n          case VARIANT:\n          case SPELLING:\n          case WEIGHT:\n          case OPTIONAL:\n            return ((FloatAnnotation) anno).getValue();\n          default:\n        }\n      }\n    }\n    return -1;\n  }\n\n  private static double getPhraseProximityFromAnnotations(List<Annotation> annotations) {\n    if (annotations != null) {\n      for (Annotation anno : annotations) {\n        if (anno.getType() == Annotation.Type.PROXIMITY) {\n          return ((FloatAnnotation) anno).getValue();\n        }\n      }\n    }\n    return -1;\n  }\n\n  private static boolean isOptional(com.twitter.search.queryparser.query.Query node) {\n    return node.hasAnnotationType(Annotation.Type.OPTIONAL);\n  }\n\n  private static boolean isProximityGroup(com.twitter.search.queryparser.query.Query node) {\n    if (node.isTypeOf(com.twitter.search.queryparser.query.Query.QueryType.OPERATOR)) {\n      SearchOperator op = (SearchOperator) node;\n      if (op.getOperatorType() == SearchOperator.Type.PROXIMITY_GROUP) {\n        return true;\n      }\n    }\n    return false;\n  }\n\n  private final Query simplifyBooleanQuery(BooleanQuery q) {\n    if (q.clauses() == null || q.clauses().size() != 1) {\n      return q;\n    }\n\n    return q.clauses().get(0).getQuery();\n  }\n\n  private Query visit(final Phrase phrase, boolean sloppy) throws QueryParserException {\n    Optional<Annotation> fieldOpt = phrase.getAnnotationOf(Annotation.Type.FIELD);\n    if (fieldOpt.isPresent()) {\n      String field = fieldOpt.get().valueToString();\n      Schema.FieldInfo fieldInfo = schemaSnapshot.getFieldInfo(field);\n      if (fieldInfo != null && !fieldInfo.getFieldType().hasPositions()) {\n        throw new QueryParserException(String.format(\"Field %s does not support phrase queries \"\n            + \"because it does not have position information.\", field));\n      }\n    }\n    BooleanQuery.Builder queryBuilder = new BooleanQuery.Builder();\n    Map<String, Float> actualFieldWeights = getFieldWeightMapForNode(phrase);\n    for (Map.Entry<String, Float> entry : actualFieldWeights.entrySet()) {\n      PhraseQuery.Builder phraseQueryBuilder = new PhraseQuery.Builder();\n      int curPos = 0;\n      for (String term : phrase.getTerms()) {\n        if (!term.equals(PHRASE_WILDCARD)) {\n          phraseQueryBuilder.add(createTerm(entry.getKey(), term), curPos);\n          curPos++;\n        } else if (curPos != 0) { //\"*\" at the beggining of a phrase has no effect/meaning\n          curPos++;\n        }\n      }\n\n      // No actual terms added to query\n      if (curPos == 0) {\n        break;\n      }\n      int annotatedSloppiness = (int) getPhraseProximityFromAnnotations(phrase.getAnnotations());\n      if (annotatedSloppiness > 0) {\n        phraseQueryBuilder.setSlop(annotatedSloppiness);\n      } else if (sloppy) {\n        phraseQueryBuilder.setSlop(proximityPhraseSlop);\n      }\n      float fieldWeight = entry.getValue();\n      float boost = (float) getBoostFromAnnotations(phrase.getAnnotations());\n      Query query = phraseQueryBuilder.build();\n      if (boost >= 0) {\n        query = BoostUtils.maybeWrapInBoostQuery(query, boost * fieldWeight);\n      } else if (fieldWeight != DEFAULT_FIELD_WEIGHT) {\n        query = BoostUtils.maybeWrapInBoostQuery(query, fieldWeight);\n      } else {\n        query = BoostUtils.maybeWrapInBoostQuery(query, proximityPhraseWeight);\n      }\n      Occur occur = actualFieldWeights.size() > 1 ? Occur.SHOULD : Occur.MUST;\n      queryBuilder.add(wrapQuery(query, phrase, entry.getKey()), occur);\n    }\n    Query q = simplifyBooleanQuery(queryBuilder.build());\n    return negateQueryIfNodeNegated(phrase, q);\n  }\n\n  private Query wrapQuery(\n      org.apache.lucene.search.Query query,\n      com.twitter.search.queryparser.query.Query node,\n      String fieldName) {\n    return EarlybirdQueryHelper.maybeWrapWithTimeout(\n        EarlybirdQueryHelper.maybeWrapWithHitAttributionCollector(\n            query, node, schemaSnapshot.getFieldInfo(fieldName), hitAttributeHelper),\n        node, queryTimeout);\n  }\n\n  private final boolean nodeIsNegated(com.twitter.search.queryparser.query.Query node) {\n    if (isParentNegated(node)) {\n      return !node.mustNotOccur();\n    } else {\n      return node.mustNotOccur();\n    }\n  }\n\n  private final Query negateQuery(Query q) {\n    return new BooleanQuery.Builder()\n        .add(q, Occur.MUST_NOT)\n        .add(new MatchAllDocsQuery(), Occur.MUST)\n        .build();\n  }\n\n  // Simple helper to examine node, and negate the lucene query if necessary.\n  private final Query negateQueryIfNodeNegated(com.twitter.search.queryparser.query.Query node,\n                                                 Query query) {\n    if (query == null) {\n      return null;\n    }\n    return nodeIsNegated(node) ? negateQuery(query) : query;\n  }\n\n  private boolean isParentNegated(com.twitter.search.queryparser.query.Query query) {\n    return parentNegatedQueries.contains(query);\n  }\n\n  private org.apache.lucene.index.Term createTerm(String field, String text)\n      throws QueryParserException {\n    Schema.FieldInfo fieldInfo = schemaSnapshot.getFieldInfo(field);\n    if (fieldInfo == null) {\n      throw new QueryParserException(\"Unknown field: \" + field);\n    }\n\n    queriedFields.add(field);\n\n    try {\n      return new org.apache.lucene.index.Term(field, SchemaUtil.toBytesRef(fieldInfo, text));\n    } catch (UnsupportedOperationException e) {\n      throw new QueryParserException(e.getMessage(), e.getCause());\n    }\n  }\n\n  /**\n   * Get field weight map for a node, combing default values and its annotations.\n   */\n  private Map<String, Float> getFieldWeightMapForNode(\n      com.twitter.search.queryparser.query.Query query) throws QueryParserException {\n    return FieldWeightUtil.combineDefaultWithAnnotation(\n        query,\n        defaultFieldWeightMap,\n        enabledFieldWeightMap,\n        Functions.<String>identity(),\n        mappableFieldMap,\n        Functions.<String>identity());\n  }\n\n  private boolean addQuery(\n      BooleanQuery.Builder bqBuilder,\n      com.twitter.search.queryparser.query.Query child) throws QueryParserException {\n    Occur occur = Occur.MUST;\n    if (child.mustNotOccur()) {\n      // To build a conjunction, we will not rely on the negation in the child visitor.\n      // Instead we will add the term as MUST_NOT occur.\n      // Store this in parentNegatedQueries so the child visitor can do the right thing.\n      occur = Occur.MUST_NOT;\n      parentNegatedQueries.add(child);\n    } else if (isOptional(child) || isProximityGroup(child)) {\n      occur = Occur.SHOULD;\n    }\n\n    Query q = child.accept(this);\n    if (q != null) {\n      bqBuilder.add(q, occur);\n      return true;\n    }\n    return false;\n  }\n\n  /**\n   * Constructs a BooleanQuery from a queryparser Query node.\n   * Adds fields as configured in the fieldWeightMap and specified by termQueryDisjunctionType\n   *  - TermQueryDisjunctionType.ONLY_OPTIONALIZED adds optional fields\n   *  (only resolved_links_text for now),\n   *  - TermQueryDisjunctionType.DROP_OPTIONALIZED adds all other valid fields expect\n   *  resolved_links_text (for now),\n   *  - TermQueryDisjunctionType.NORMAL adds all valid fields\n   * @param query an instance of com.twitter.search.queryparser.query.Query or\n   * com.twitter.search.queryparser.query.Term\n   * @return a BooleanQuery consists of fields from query\n   */\n  private BooleanQuery createTermQueryDisjunction(\n      com.twitter.search.queryparser.query.Query query) throws QueryParserException {\n    String normTerm = query.isTypeOf(com.twitter.search.queryparser.query.Query.QueryType.TERM)\n        ? ((Term) query).getValue() : query.toString(false);\n    BooleanQuery.Builder booleanQueryBuilder = new BooleanQuery.Builder();\n    Map<String, Float> actualFieldWeightMap = getFieldWeightMapForNode(query);\n    Set<String> fieldsToUse = Sets.newLinkedHashSet(actualFieldWeightMap.keySet());\n    Occur occur = fieldsToUse.size() > 1 ? Occur.SHOULD : Occur.MUST;\n    for (String field : fieldsToUse) {\n      addTermQueryWithField(booleanQueryBuilder, query, normTerm, field, occur,\n          actualFieldWeightMap.get(field));\n    }\n    return booleanQueryBuilder.build();\n  }\n\n  private void addTermQueryWithField(\n      BooleanQuery.Builder bqBuilder,\n      com.twitter.search.queryparser.query.Query term,\n      String normTerm,\n      String fieldName,\n      Occur occur,\n      float fieldWeight) throws QueryParserException {\n    float boost = (float) getBoostFromAnnotations(term.getAnnotations());\n    Query query = createSimpleTermQuery(term, fieldName, normTerm);\n    if (boost >= 0) {\n      query = BoostUtils.maybeWrapInBoostQuery(query, boost * fieldWeight);\n    } else {\n      query = BoostUtils.maybeWrapInBoostQuery(query, fieldWeight);\n    }\n    bqBuilder.add(query, occur);\n  }\n\n  private Query finalizeQuery(BooleanQuery bq, Term term) {\n    Query q = simplifyBooleanQuery(bq);\n    return negateQueryIfNodeNegated(term, q);\n  }\n\n  private Rectangle boundingBoxFromSearchOperator(SearchOperator op) throws QueryParserException {\n    Preconditions.checkArgument(op.getOperatorType() == SearchOperator.Type.GEO_BOUNDING_BOX);\n    Preconditions.checkNotNull(op.getOperands());\n    Preconditions.checkState(op.getOperands().size() == 4);\n\n    List<String> operands = op.getOperands();\n    try {\n      // Unfortunately, we store coordinates as floats in our index, which causes a lot of precision\n      // loss. On the query side, we have to cast into floats to match.\n      float minLat = (float) Double.parseDouble(operands.get(0));\n      float minLon = (float) Double.parseDouble(operands.get(1));\n      float maxLat = (float) Double.parseDouble(operands.get(2));\n      float maxLon = (float) Double.parseDouble(operands.get(3));\n\n      Point lowerLeft = new PointImpl(minLon, minLat, GeohashChunkImpl.getSpatialContext());\n      Point upperRight = new PointImpl(maxLon, maxLat, GeohashChunkImpl.getSpatialContext());\n      return new RectangleImpl(lowerLeft, upperRight, GeohashChunkImpl.getSpatialContext());\n    } catch (NumberFormatException e) {\n      // consider operator invalid if any of the coordinate cannot be parsed.\n      throw new QueryParserException(\"Malformed bounding box operator.\" + op.serialize());\n    }\n  }\n\n  private Query visitGeocodeOrGeocodePrivateOperator(SearchOperator op)\n      throws QueryParserException {\n\n    GeoCode geoCode = GeoCode.fromOperator(op);\n    if (geoCode == null) {\n      throw new QueryParserException(\"Invalid GeoCode operator:\" + op.serialize());\n    }\n\n    return wrapQueryInUserScrubGeoFilter(\n        GeoQuadTreeQueryBuilder.buildGeoQuadTreeQuery(geoCode, terminationTracker));\n  }\n\n  private Query wrapQueryInUserScrubGeoFilter(Query baseQuery) {\n    if (DeciderUtil.isAvailableForRandomRecipient(\n        decider, \"filter_out_geo_scrubbed_tweets_\" + earlybirdCluster.getNameForStats())) {\n      return new FilteredQuery(\n          baseQuery,\n          UserScrubGeoFilter.getDocIdFilterFactory(userScrubGeoMap));\n    } else {\n      return baseQuery;\n    }\n  }\n\n  private Query buildLongTermAttributeQuery(SearchOperator op, String fieldName) {\n    return buildLongTermAttributeQuery(op, fieldName, Long.parseLong(op.getOperand()));\n  }\n\n  private Query buildLongTermAttributeQuery(SearchOperator op, String fieldName, long argValue) {\n    org.apache.lucene.index.Term term = new org.apache.lucene.index.Term(\n        fieldName, LongTermAttributeImpl.copyIntoNewBytesRef(argValue));\n    return wrapQuery(new TermQueryWithSafeToString(term, Long.toString(argValue)), op, fieldName);\n  }\n\n  private static void parseLongArgs(List<String> operands,\n                                    Collection<Long> arguments,\n                                    SearchOperator op) throws QueryParserException {\n    for (String operand : operands) {\n      try {\n        arguments.add(Long.parseLong(operand));\n      } catch (NumberFormatException e) {\n        throw new QueryParserException(\"Invalid long operand in \" + op.serialize(), e);\n      }\n    }\n  }\n\n  private static boolean isUserIdField(String field) {\n    return EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName().equals(field)\n        || EarlybirdFieldConstant.IN_REPLY_TO_USER_ID_FIELD.getFieldName().equals(field)\n        || EarlybirdFieldConstant.RETWEET_SOURCE_USER_ID_FIELD.getFieldName().equals(field)\n        || EarlybirdFieldConstant.LIKED_BY_USER_ID_FIELD.getFieldName().equals(field)\n        || EarlybirdFieldConstant.RETWEETED_BY_USER_ID.getFieldName().equals(field)\n        || EarlybirdFieldConstant.REPLIED_TO_BY_USER_ID.getFieldName().equals(field)\n        || EarlybirdFieldConstant.QUOTED_USER_ID_FIELD.getFieldName().equals(field)\n        || EarlybirdFieldConstant.DIRECTED_AT_USER_ID_FIELD.getFieldName().equals(field);\n  }\n\n  private static boolean isTweetIdField(String field) {\n    return EarlybirdFieldConstant.IN_REPLY_TO_TWEET_ID_FIELD.getFieldName().equals(field)\n        || EarlybirdFieldConstant.RETWEET_SOURCE_TWEET_ID_FIELD.getFieldName().equals(field)\n        || EarlybirdFieldConstant.QUOTED_TWEET_ID_FIELD.getFieldName().equals(field)\n        || EarlybirdFieldConstant.CONVERSATION_ID_FIELD.getFieldName().equals(field);\n  }\n\n  private static boolean isIdCSFField(String field) {\n    return EarlybirdFieldConstant.DIRECTED_AT_USER_ID_CSF.getFieldName().equals(field);\n  }\n\n  public Set<String> getQueriedFields() {\n    return queriedFields;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/queryparser/EarlybirdQueryHelper.java",
    "content": "package com.twitter.search.earlybird.queryparser;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Optional;\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.constants.QueryCacheConstants;\nimport com.twitter.search.common.query.HitAttributeCollector;\nimport com.twitter.search.common.query.HitAttributeHelper;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.search.termination.QueryTimeout;\nimport com.twitter.search.common.search.termination.TerminationQuery;\nimport com.twitter.search.earlybird.querycache.QueryCacheManager;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryNodeUtils;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.annotation.Annotation;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\nimport com.twitter.search.queryparser.query.search.SearchOperatorConstants;\n\npublic abstract class EarlybirdQueryHelper {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdQueryHelper.class);\n\n  /**\n   * Wraps the given query and some clauses to exclude antisocial tweets into a conjunction.\n   */\n  public static Query requireExcludeAntisocial(\n      Query basicQuery,\n      QueryCacheManager queryCacheManager) throws QueryParserException {\n    // Do not set exclude antisocial if they have any other antisocial filters set\n    Query query = basicQuery;\n    DetectAntisocialVisitor detectAntisocialVisitor = new DetectAntisocialVisitor();\n    query.accept(detectAntisocialVisitor);\n    if (detectAntisocialVisitor.hasAnyAntisocialOperator()) {\n      return query;\n    }\n\n    // No operator found, force antisocial filter.\n    if (queryCacheManager.enabled()) {\n      SearchOperator filter =\n          new SearchOperator(SearchOperator.Type.CACHED_FILTER,\n              QueryCacheConstants.EXCLUDE_ANTISOCIAL);\n\n      query = QueryNodeUtils.appendAsConjunction(query, filter);\n    } else {\n      SearchOperator filter = new SearchOperator(SearchOperator.Type.EXCLUDE,\n          SearchOperatorConstants.ANTISOCIAL);\n\n      query = QueryNodeUtils.appendAsConjunction(query, filter);\n    }\n    return query;\n  }\n\n  /**\n   * Wraps the given query into an equivalent query that will also collect hit attribution data.\n   *\n   * @param query The original query.\n   * @param node The query parser node storing this query.\n   * @param fieldInfo The field in which the given query will be searching.\n   * @param hitAttributeHelper The helper that will collect all hit attribution data.\n   * @return An equivalent query that will also collect hit attribution data.\n   */\n  public static final org.apache.lucene.search.Query maybeWrapWithHitAttributionCollector(\n      org.apache.lucene.search.Query query,\n      @Nullable com.twitter.search.queryparser.query.Query node,\n      Schema.FieldInfo fieldInfo,\n      @Nullable HitAttributeHelper hitAttributeHelper) {\n    // Prevents lint error for assigning to a function parameter.\n    org.apache.lucene.search.Query luceneQuery = query;\n    if (hitAttributeHelper != null && node != null) {\n      Optional<Annotation> annotation = node.getAnnotationOf(Annotation.Type.NODE_RANK);\n\n      if (annotation.isPresent()) {\n        Integer nodeRank = (Integer) annotation.get().getValue();\n        luceneQuery = wrapWithHitAttributionCollector(\n            luceneQuery,\n            fieldInfo,\n            nodeRank,\n            hitAttributeHelper.getFieldRankHitAttributeCollector());\n      }\n    }\n\n    return luceneQuery;\n  }\n\n  /**\n   * Wraps the given query into an equivalent query that will also collect hit attribution data.\n   *\n   * @param query The original query.\n   * @param nodeRank The rank of the given query in the overall request query.\n   * @param fieldInfo The field in which the given query will be searching.\n   * @param hitAttributeHelper The helper that will collect all hit attribution data.\n   * @return An equivalent query that will also collect hit attribution data.\n   */\n  public static final org.apache.lucene.search.Query maybeWrapWithHitAttributionCollector(\n      org.apache.lucene.search.Query query,\n      int nodeRank,\n      Schema.FieldInfo fieldInfo,\n      @Nullable HitAttributeHelper hitAttributeHelper) {\n\n    org.apache.lucene.search.Query luceneQuery = query;\n    if (hitAttributeHelper != null && nodeRank != -1) {\n      Preconditions.checkArgument(nodeRank > 0);\n      luceneQuery = wrapWithHitAttributionCollector(\n          luceneQuery, fieldInfo, nodeRank, hitAttributeHelper.getFieldRankHitAttributeCollector());\n    }\n    return luceneQuery;\n  }\n\n  private static final org.apache.lucene.search.Query wrapWithHitAttributionCollector(\n      org.apache.lucene.search.Query luceneQuery,\n      Schema.FieldInfo fieldInfo,\n      int nodeRank,\n      HitAttributeCollector hitAttributeCollector) {\n    Preconditions.checkNotNull(fieldInfo,\n        \"Tried collecting hit attribution for unknown field: \" + fieldInfo.getName()\n            + \" luceneQuery: \" + luceneQuery);\n    return hitAttributeCollector.newIdentifiableQuery(\n        luceneQuery, fieldInfo.getFieldId(), nodeRank);\n  }\n\n  /**\n   * Returns a query equivalent to the given query, and with the given timeout enforced.\n   */\n  public static org.apache.lucene.search.Query maybeWrapWithTimeout(\n      org.apache.lucene.search.Query query,\n      QueryTimeout timeout) {\n    if (timeout != null) {\n      return new TerminationQuery(query, timeout);\n    }\n    return query;\n  }\n\n  /**\n   * Returns a query equivalent to the given query, and with the given timeout enforced. If the\n   * given query is negated, it is returned without any modifications.\n   */\n  public static org.apache.lucene.search.Query maybeWrapWithTimeout(\n      org.apache.lucene.search.Query query,\n      @Nullable com.twitter.search.queryparser.query.Query node,\n      QueryTimeout timeout) {\n    // If the node is looking for negation of something, we don't want to include it in node-level\n    // timeout checks. In general, nodes keep track of the last doc seen, but non-matching docs\n    // encountered by \"must not occur\" node do not reflect overall progress in the index.\n    if (node != null && node.mustNotOccur()) {\n      return query;\n    }\n    return maybeWrapWithTimeout(query, timeout);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/queryparser/HighFrequencyTermPairExtractor.java",
    "content": "package com.twitter.search.earlybird.queryparser;\n\nimport java.util.ArrayList;\nimport java.util.IdentityHashMap;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.util.text.HighFrequencyTermPairs;\nimport com.twitter.search.queryparser.query.BooleanQuery;\nimport com.twitter.search.queryparser.query.Conjunction;\nimport com.twitter.search.queryparser.query.Disjunction;\nimport com.twitter.search.queryparser.query.Operator;\nimport com.twitter.search.queryparser.query.Phrase;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.QueryVisitor;\nimport com.twitter.search.queryparser.query.SpecialTerm;\nimport com.twitter.search.queryparser.query.Term;\nimport com.twitter.search.queryparser.query.annotation.Annotation;\n\n/**\n * Iterates over the Query, populating information of an ArrayList of HighFrequencyTermQueryGroup so that\n * HighFrequencyTermPairRewriteVisitor can rewrite the query to use hf term pairs. Returns the\n * (approximate) number of high frequency terms it has detected. Iff that number is greater than 1\n * it MAY be able to rewrite the query to use the hf_term_pairs field.\n *\n * The key to HF Term Pair rewriting is understanding which nodes can be combined. This extractor\n * accomplishes this job by grouping nodes of the query together. All positive children of a\n * conjunction are grouped together, and all negative children of a disjunction are grouped\n * together. The end result is a tree of groups, where every child of a single group will have the\n * opposite value of isPositive of the parent group.\n *\n * I'll try to break it down a bit further. Let's assume \"a\" and \"b\" are hf terms, and '\n * \"[hf_term_pair a b]\" represents querying their co-occurence.\n * Query (* a b not_hf) can become (* [hf_term_pair a b] not_hf)\n * Query (+ -a -b -not_hf) can become (+ -[hf_term_pair a b] -not_hf)\n * These two rules represent the bulk of the rewrites that this class makes.\n *\n * We also keep track of another form of rewrite. A member of a group can be paired up with a member\n * of any of its parent groups as long as both groups have the same isPositive value. This\n * operation mimics boolean distribution. As this is probably better explained with an example:\n * Query (* a (+ not_hf (* b not_hf2))) can become (* a (+ not_hf (* [hf_term_pair a b ] not_hf2)))\n * Query (+ -a (* not_hf (+ -b not_hf2))) can become (+ -a (* not_hf (+ -[hf_term_pair a b] not_hf2)))\n */\npublic class HighFrequencyTermPairExtractor extends QueryVisitor<Integer> {\n\n  private final ArrayList<HighFrequencyTermQueryGroup> groupList;\n  private final IdentityHashMap<Query, Integer> groupIds;\n\n  public HighFrequencyTermPairExtractor(ArrayList<HighFrequencyTermQueryGroup> groupList,\n                                        IdentityHashMap<Query, Integer> groupIds) {\n    Preconditions.checkNotNull(groupList);\n    Preconditions.checkArgument(groupList.isEmpty());\n    this.groupList = groupList;\n    this.groupIds = groupIds;\n  }\n\n  @Override\n  public Integer visit(Disjunction disjunction) throws QueryParserException {\n    return visit((BooleanQuery) disjunction);\n  }\n\n  @Override\n  public Integer visit(Conjunction conjunction) throws QueryParserException {\n    return visit((BooleanQuery) conjunction);\n  }\n\n  /**\n   * All positive children under a conjunction (negative children under disjunction) belong in the\n   * same group as booleanQuery. All other children belong in their own, separate, new groups.\n   * @param booleanQuery\n   * @return Number of high frequency terms seen by this node and its children\n   * @throws QueryParserException\n   */\n  private Integer visit(BooleanQuery booleanQuery) throws QueryParserException {\n    HighFrequencyTermQueryGroup group = getGroupForQuery(booleanQuery);\n    int numHits = 0;\n\n    for (Query node : booleanQuery.getChildren()) {\n      boolean neg = node.mustNotOccur();\n      if (node.isTypeOf(Query.QueryType.DISJUNCTION)) {\n        // Disjunctions, being negative conjunctions, are inherently negative nodes. In terms of\n        // being in a positive or negative group, we must flip their Occur value.\n        neg = !neg;\n      }\n\n      if (booleanQuery.isTypeOf(Query.QueryType.DISJUNCTION) && node.mustOccur()) {\n        // Potential Example: (* a (+ +b not_c)) => (* (+ +b not_c) [hf_term_pair a b 0.05])\n        // Implementation is too difficult and would make this rewriter even MORE complicated for\n        // a rarely used query. For now, we ignore it completely. We might gain some benefit in the\n        // future if we decide to create a new extractor and rewriter and rewrite this subquery, and\n        // that wouldn't complicate things too much.\n        continue;\n      }\n\n      if (booleanQuery.isTypeOf(Query.QueryType.CONJUNCTION) != neg) { // Add node to current group\n        groupIds.put(node, group.groupIdx);\n        group.numMembers++;\n      } else { // Create a new group\n        HighFrequencyTermQueryGroup newGroup =\n            new HighFrequencyTermQueryGroup(groupList.size(), group.groupIdx, !group.isPositive);\n        newGroup.numMembers++;\n        groupIds.put(node, newGroup.groupIdx);\n        groupList.add(newGroup);\n      }\n      numHits += node.accept(this);\n    }\n\n    return numHits;\n  }\n\n  @Override\n  public Integer visit(Phrase phrase) throws QueryParserException {\n    HighFrequencyTermQueryGroup group = getGroupForQuery(phrase);\n\n    int numFound = 0;\n    if (!phrase.hasAnnotationType(Annotation.Type.OPTIONAL)) {\n      boolean canBeRewritten = false;\n\n      // Special case: phrases with exactly 2 terms that are both high frequency can be\n      // rewritten. In all other cases terms will be treated as pre-used hf term phrases.\n      if (!phrase.hasAnnotations() && phrase.size() == 2\n          && HighFrequencyTermPairs.HF_TERM_SET.contains(phrase.getTerms().get(0))\n          && HighFrequencyTermPairs.HF_TERM_SET.contains(phrase.getTerms().get(1))) {\n        canBeRewritten = true;\n      }\n\n      // Special case: do not treat phrase containing :prox annotation as a real phrase.\n      boolean proximityPhrase = phrase.hasAnnotationType(Annotation.Type.PROXIMITY);\n\n      String lastHFToken = null;\n      for (String token : phrase.getTerms()) {\n        if (HighFrequencyTermPairs.HF_TERM_SET.contains(token)) {\n          group.preusedHFTokens.add(token);\n          if (group.distributiveToken == null) {\n            group.distributiveToken = token;\n          }\n          if (lastHFToken != null && !proximityPhrase) {\n            if (canBeRewritten) {\n              group.hfPhrases.add(lastHFToken + \" \" + token);\n            } else {\n              group.preusedHFPhrases.add(lastHFToken + \" \" + token);\n            }\n          }\n          lastHFToken = token;\n          numFound++;\n        } else {\n          lastHFToken = null;\n        }\n      }\n    }\n\n    return numFound;\n  }\n\n  @Override\n  public Integer visit(Term term) throws QueryParserException {\n    if (groupList.isEmpty()) { // Shortcut for 1 term queries.\n      return 0;\n    }\n\n    HighFrequencyTermQueryGroup group = getGroupForQuery(term);\n\n    if (!term.hasAnnotationType(Annotation.Type.OPTIONAL)\n        && HighFrequencyTermPairs.HF_TERM_SET.contains(term.getValue())) {\n      if (!term.hasAnnotations()) {\n        group.hfTokens.add(term.getValue());\n      } else { // Should not remove the annotated term.\n        group.preusedHFTokens.add(term.getValue());\n      }\n\n      if (group.distributiveToken == null) {\n        group.distributiveToken = term.getValue();\n      }\n      return 1;\n    }\n\n    return 0;\n  }\n\n  @Override\n  public Integer visit(Operator operator) throws QueryParserException {\n    return 0;\n  }\n\n  @Override\n  public Integer visit(SpecialTerm special) throws QueryParserException {\n    return 0;\n  }\n\n  /**\n   * Uses the query's visitor data as an index and returns the group it belongs to. If groupList is\n   * empty, create a new group and set this group's visitor data to be index 0.\n   * @param query\n   * @return the group which query belongs to.\n   */\n  private HighFrequencyTermQueryGroup getGroupForQuery(Query query) {\n    if (groupList.isEmpty()) {\n      boolean pos = !query.mustNotOccur();\n      if (query instanceof Disjunction) {\n        pos = !pos;\n      }\n      HighFrequencyTermQueryGroup group = new HighFrequencyTermQueryGroup(0, pos);\n      group.numMembers++;\n      groupList.add(group);\n      groupIds.put(query, 0);\n    }\n\n    return groupList.get(groupIds.get(query));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/queryparser/HighFrequencyTermPairRewriteVisitor.java",
    "content": "package com.twitter.search.earlybird.queryparser;\n\nimport java.util.ArrayList;\nimport java.util.IdentityHashMap;\nimport java.util.List;\nimport java.util.Set;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.util.text.HighFrequencyTermPairs;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.queryparser.parser.SerializedQueryParser;\nimport com.twitter.search.queryparser.query.BooleanQuery;\nimport com.twitter.search.queryparser.query.Conjunction;\nimport com.twitter.search.queryparser.query.Disjunction;\nimport com.twitter.search.queryparser.query.Operator;\nimport com.twitter.search.queryparser.query.Phrase;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryNodeUtils;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.QueryVisitor;\nimport com.twitter.search.queryparser.query.SpecialTerm;\nimport com.twitter.search.queryparser.query.Term;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\n\n/**\n * Iterates over the Query, modifying it to include high frequency term pairs, replacing\n * singular high frequency terms where possible.\n *\n * Assumes that this will be used IMMEDIATELY after using HighFrequencyTermPairExtractor\n *\n * There are two primary functions of this visitor:\n *  1. Append hf_term_pairs to each group's root node.\n *  2. Remove all unnecessary term queries (unnecessary as they are captured by an hf_term_pair)\n *\n * Every time the visitor finishes visiting a node, HighFrequencyTermQueryGroup.numVisits will be\n * incremented for that node's group. When numVisits == numChildren, we know we have just finished\n * processing the root of the group. At this point, we must append relevant hf_term_pairs to this\n * node.\n */\npublic class HighFrequencyTermPairRewriteVisitor extends QueryVisitor<Query> {\n  private static final Logger LOG = LoggerFactory.getLogger(\n      HighFrequencyTermPairRewriteVisitor.class);\n  private static final SearchRateCounter SEARCH_HF_PAIR_COUNTER =\n      SearchRateCounter.export(\"hf_pair_rewrite\");\n\n  private final ArrayList<HighFrequencyTermQueryGroup> groupList;\n  private final IdentityHashMap<Query, Integer> groupIds;\n  private final boolean allowNegativeOrRewrite;\n\n  /**\n   * Creates a new HighFrequencyTermPairRewriteVisitor. Should be used only IMMEDIATELY after using\n   * a HighFrequencyTermPairExtractor\n   * @param groupList The groups extracted using HighFrequencyTermPairExtractor\n   * @param groupIds the mapping from query to the HF term query group\n   */\n  public HighFrequencyTermPairRewriteVisitor(ArrayList<HighFrequencyTermQueryGroup> groupList,\n                                             IdentityHashMap<Query, Integer> groupIds) {\n    this(groupList, groupIds, true);\n  }\n\n  /**\n   * Creates a new HighFrequencyTermPairRewriteVisitor. Should be used only IMMEDIATELY after using\n   * a HighFrequencyTermPairExtractor\n   * @param groupList The groups extracted using HighFrequencyTermPairExtractor\n   * @param groupIds the mapping from query to the HF term query group\n   * @param allowNegativeOrRewrite whether to allow rewrite for 'or (-terms)'\n   */\n  public HighFrequencyTermPairRewriteVisitor(ArrayList<HighFrequencyTermQueryGroup> groupList,\n                                             IdentityHashMap<Query, Integer> groupIds,\n                                             boolean allowNegativeOrRewrite) {\n    this.groupList = groupList;\n    this.groupIds = groupIds;\n    this.allowNegativeOrRewrite = allowNegativeOrRewrite;\n  }\n\n  /**\n   * This method logs successful rewrites, and protects against unsuccessful ones by\n   * catching all exceptions and restoring the previous query. \n   */\n  public static Query safeRewrite(Query safeQuery, boolean allowNegativeOrRewrite)\n      throws QueryParserException {\n    Query query = safeQuery;\n\n    ArrayList<HighFrequencyTermQueryGroup> groups = Lists.newArrayList();\n    IdentityHashMap<Query, Integer> groupIds = Maps.newIdentityHashMap();\n\n    // Step 1: extract high frequency term pairs and phrases.\n    try {\n      int hfTermsFound = query.accept(new HighFrequencyTermPairExtractor(groups, groupIds));\n      if (hfTermsFound < 2) {\n        return query;\n      }\n    } catch (Exception e) {\n      LOG.error(\"Exception while extracting high frequency term pairs\", e);\n      return query;\n    }\n\n    // Step 2: rewrite (safely).\n    String original = query.serialize();\n    try {\n      query = query.accept(\n          new HighFrequencyTermPairRewriteVisitor(groups, groupIds, allowNegativeOrRewrite))\n          .simplify();\n      String rewrite = query.serialize();\n      if (LOG.isDebugEnabled()) {\n        LOG.debug(\"Optimized query: \" + original + \" -> \" + rewrite);\n      }\n      SEARCH_HF_PAIR_COUNTER.increment();\n      return query;\n    } catch (Exception e) {\n      LOG.error(\"Exception rewriting high frequency term pairs\", e);\n      return new SerializedQueryParser(EarlybirdConfig.getPenguinVersion()).parse(original);\n    }\n  }\n\n  /**\n   * The rewritten query to use the hf_term_pair operators.\n   *\n   * @param disjunction query node which must have been previously visited by\n   *                    HighFrequencyTermPairExtractor and not had its visitor data cleared.\n   */\n  @Override\n  public Query visit(Disjunction disjunction) throws QueryParserException {\n    return visit((BooleanQuery) disjunction);\n  }\n\n  /**\n   * The rewritten query to use the hf_term_pair operators.\n   *\n   * @param conjunction query node which must have been previously visited by\n   *                    HighFrequencyTermPairExtractor and not had its visitor data cleared.\n   */\n  @Override\n  public Query visit(Conjunction conjunction) throws QueryParserException {\n    return visit((BooleanQuery) conjunction);\n  }\n\n  /**\n   * Applies this visitor to a BooleanQuery.\n   */\n  public Query visit(BooleanQuery booleanQuery) throws QueryParserException {\n    HighFrequencyTermQueryGroup group = groupList.get(groupIds.get(booleanQuery));\n    queryPreprocess(group);\n\n    ArrayList<Query> children = Lists.newArrayList();\n    for (Query node : booleanQuery.getChildren()) {\n      if (booleanQuery.isTypeOf(Query.QueryType.DISJUNCTION) && node.mustOccur()) {\n        // Potential Example: (* a (+ +b not_c)) => (* (+ +b not_c) [hf_term_pair a b 0.05])\n        // Implementation is too difficult and would make this rewriter even MORE complicated for\n        // a rarely used query. For now, we ignore it completely. We might gain some benefit in the\n        // future if we decide to create a new extractor and rewriter and rewrite this subquery, and\n        // that wouldn't complicate things too much.\n        children.add(node);\n        continue;\n      }\n      Query child = node.accept(this);\n      if (child != null) {\n        children.add(child);\n      }\n    }\n\n    Query newBooleanQuery = booleanQuery.newBuilder().setChildren(children).build();\n\n    return queryPostprocess(newBooleanQuery, group);\n  }\n\n  /**\n   * The rewritten query to use the hf_term_pair operators.\n   *\n   * @param phraseToVisit query node which must have been previously visited by\n   *               HighFrequencyTermPairExtractor and not had its visitor data cleared.\n   */\n  @Override\n  public Query visit(Phrase phraseToVisit) throws QueryParserException {\n    Phrase phrase = phraseToVisit;\n\n    HighFrequencyTermQueryGroup group = groupList.get(groupIds.get(phrase));\n    queryPreprocess(group);\n\n    // Remove all high frequency phrases from the query that do not have any annotations.\n    // This will cause phrase de-duping, which we probably don't care about.\n    if (!hasAnnotations(phrase) && (\n        group.hfPhrases.contains(phrase.getPhraseValue())\n        || group.preusedHFPhrases.contains(phrase.getPhraseValue()))) {\n      // This term will be appended to the end of the query in the form of a pair.\n      phrase = null;\n    }\n\n    return queryPostprocess(phrase, group);\n  }\n\n  /**\n   * The rewritten query to use the hf_term_pair operators.\n   *\n   * @param termToVisit query node which must have been previously visited by\n   *             HighFrequencyTermPairExtractor and not had its visitor data cleared.\n   */\n  @Override\n  public Query visit(Term termToVisit) throws QueryParserException {\n    Term term = termToVisit;\n\n    HighFrequencyTermQueryGroup group = groupList.get(groupIds.get(term));\n    queryPreprocess(group);\n\n    // Remove all high frequency terms from the query that do not have any annotations. This will\n    // do term de-duping within a group, which may effect scoring, but since these are high df\n    // terms, they don't have much of an impact anyways.\n    if (!hasAnnotations(term)\n        && (group.preusedHFTokens.contains(term.getValue())\n            || group.hfTokens.contains(term.getValue()))) {\n      // This term will be appended to the end of the query in the form of a pair.\n      term = null;\n    }\n\n    return queryPostprocess(term, group);\n  }\n\n  /**\n   * The rewritten query to use the hf_term_pair operators.\n   *\n   * @param operator query node which must have been previously visited by\n   *                 HighFrequencyTermPairExtractor and not had its visitor data cleared.\n   */\n  @Override\n  public Query visit(Operator operator) throws QueryParserException {\n    HighFrequencyTermQueryGroup group = groupList.get(groupIds.get(operator));\n    queryPreprocess(group);\n\n    return queryPostprocess(operator, group);\n  }\n\n  /**\n   * The rewritten query to use the hf_term_pair operators.\n   *\n   * @param special query node which must have been previously visited by\n   *                HighFrequencyTermPairExtractor and not had its visitor data cleared.\n   */\n  @Override\n  public Query visit(SpecialTerm special) throws QueryParserException {\n    HighFrequencyTermQueryGroup group = groupList.get(groupIds.get(special));\n    queryPreprocess(group);\n\n    return queryPostprocess(special, group);\n  }\n\n  /**\n   * Before visiting a node's children, we must process its group's distributiveToken. This way, a\n   * node only has to check its grandparent group for a distributiveToken instead of recursing all\n   * of the way up to the root of the tree.\n   */\n  private void queryPreprocess(HighFrequencyTermQueryGroup group) {\n    if (group.distributiveToken == null) {\n      group.distributiveToken = getAncestorDistributiveToken(group);\n    }\n  }\n\n  /**\n   * If the query isn't the root of the group, returns the query. Otherwise, if the query's\n   * group has at most one hf term, return the query. Otherwise, returns the query with hf_term_pair\n   * operators created from the group's hf terms appended to it.\n   */\n  private Query queryPostprocess(@Nullable Query query, HighFrequencyTermQueryGroup group)\n      throws QueryParserException {\n\n    group.numVisits++;\n    if (group.numMembers == group.numVisits\n        && (!group.hfTokens.isEmpty() || !group.preusedHFTokens.isEmpty()\n        || group.hasPhrases())) {\n\n      group.removePreusedTokens();\n      String ancestorDistributiveToken = getAncestorDistributiveToken(group);\n\n      // Need at least 2 tokens to perform a pair rewrite.  Try to get one\n      // additional token from ancestors, and if that fails, from phrases.\n      if ((group.hfTokens.size() + group.preusedHFTokens.size()) == 1\n          && ancestorDistributiveToken != null) {\n        group.preusedHFTokens.add(ancestorDistributiveToken);\n      }\n      if ((group.hfTokens.size() + group.preusedHFTokens.size()) == 1) {\n        String tokenFromPhrase = group.getTokenFromPhrase();\n        if (tokenFromPhrase != null) {\n          group.preusedHFTokens.add(tokenFromPhrase);\n        }\n      }\n\n      return appendPairs(query, group);\n    }\n\n    return query;\n  }\n\n  /**\n   * Returns the distributiveToken of group's grandparent.\n   */\n  private String getAncestorDistributiveToken(HighFrequencyTermQueryGroup group) {\n    String ancestorDistributiveToken = null;\n    if (group.parentGroupIdx >= 0 && groupList.get(group.parentGroupIdx).parentGroupIdx >= 0) {\n      ancestorDistributiveToken =\n              groupList.get(groupList.get(group.parentGroupIdx).parentGroupIdx).distributiveToken;\n    }\n    return ancestorDistributiveToken;\n  }\n\n  /**\n   * Returns the hf_term_pair operators created using the hf terms of the group appended to query.\n   *\n   * @param query The query which the new hf_term_pair operators will be appended to.\n   * @param group The group which this query belongs to.\n   * @return The hf_term_pair operators created using the hf terms of the group appended to query.\n   */\n  private Query appendPairs(@Nullable Query query, HighFrequencyTermQueryGroup group)\n      throws QueryParserException {\n\n    BooleanQuery query2 = createQueryFromGroup(group);\n\n    // If either of the queries are null, do not have to worry about combining them.\n    if (query2 == null) {\n      return query;\n    } else if (query == null) {\n      return query2;\n    }\n\n    Query newQuery;\n\n    if (query.isTypeOf(Query.QueryType.CONJUNCTION)\n        || query.isTypeOf(Query.QueryType.DISJUNCTION)) {\n      // Adding children in this way is safer when its query is a conjunction or disjunction\n      // ex. Other way: (+ +de -la -the) => (+ (+ +de -la -the) -[hf_term_pair la the 0.005])\n      //     This way: (+ +de -la -the) => (+ +de -la -the -[hf_term_pair la the 0.005])\n      return ((BooleanQuery.Builder) query.newBuilder()).addChildren(query2.getChildren()).build();\n    } else if (!group.isPositive) {\n      // In lucene, [+ (-term1, -term2, ...)] has non-deterministic behavior and the rewrite is not\n      // efficient from query execution perspective.  So, we will not do this rewrite if it is\n      // configured that way.\n      if (!allowNegativeOrRewrite) {\n        return query;\n      }\n\n      // Negate both queries to combine, and the append as a conjunction, followed by negating\n      // whole query. Equivalent to appending as a disjunction.\n      newQuery = QueryNodeUtils.appendAsConjunction(\n          query.negate(),\n          query2.negate()\n      );\n      newQuery = newQuery.makeMustNot();\n    } else {\n      newQuery = QueryNodeUtils.appendAsConjunction(query, query2);\n      newQuery = newQuery.makeDefault();\n    }\n\n    return newQuery;\n  }\n\n  /**\n   * Creates a conjunction of term_pairs using the sets of hf terms in HighFrequencyTermQueryGroup\n   * group. If !group.isPositive, will return a disjunction of negated pairs. If there aren't enough\n   * hfTokens, will return null.\n   */\n  private BooleanQuery createQueryFromGroup(HighFrequencyTermQueryGroup group)\n      throws QueryParserException {\n\n    if (!group.hfTokens.isEmpty() || group.preusedHFTokens.size() > 1 || group.hasPhrases()) {\n      List<Query>  terms = createTermPairsForGroup(group.hfTokens,\n                                                   group.preusedHFTokens,\n                                                   group.hfPhrases,\n                                                   group.preusedHFPhrases);\n\n      if (group.isPositive) {\n        return new Conjunction(terms);\n      } else {\n        return new Disjunction(Lists.transform(terms, QueryNodeUtils.NEGATE_QUERY));\n      }\n    }\n\n    return null;\n  }\n\n  /**\n   * Creates HF_TERM_PAIR terms out of hfTokens and optHFTokens. Attempts to create the minimal\n   * amount of tokens necessary. optHFToken pairs should be given a weight of 0.0 and not be scored,\n   * as they are likely already included in the query in a phrase or an annotated term.\n   * @param hfTokens\n   * @param optHFTokens\n   * @return A list of hf_term_pair operators.\n   */\n  private List<Query> createTermPairsForGroup(Set<String> hfTokens,\n                                              Set<String> optHFTokens,\n                                              Set<String> hfPhrases,\n                                              Set<String> optHFPhrases) {\n    // Handle sets with only one token.\n    if (optHFTokens.size() == 1 && hfTokens.size() > 0) {\n      // (* \"a not_hf\" b c) => (* \"a not_hf\" [hf_term_pair a b 0.05] [hf_term_pair b c 0.05])\n      // optHFTokens: [a] hfTokens: [b, c] => optHFTokens: [] hfTokens: [a, b, c]\n      hfTokens.addAll(optHFTokens);\n      optHFTokens.clear();\n    } else if (hfTokens.size() == 1 && optHFTokens.size() > 0) {\n      // (* \"a b\" not_hf c) => (* \"a b\" not_hf [hf_term_pair a b 0.0] [hf_term_pair a c 0.005])\n      // optHFTokens: [a, b] hfTokens: [c] => optHFTokens: [a, b] hfTokens: [a, c]\n      String term = optHFTokens.iterator().next();\n      hfTokens.add(term);\n    }\n\n    List<Query> terms = createTermPairs(hfTokens, true, HighFrequencyTermPairs.HF_DEFAULT_WEIGHT);\n    terms.addAll(createTermPairs(optHFTokens, false, 0));\n    terms.addAll(createPhrasePairs(hfPhrases, HighFrequencyTermPairs.HF_DEFAULT_WEIGHT));\n    terms.addAll(createPhrasePairs(optHFPhrases, 0));\n\n    return terms;\n  }\n\n  /**\n   * Turns a set of hf terms into a list of hf_term_pair operators. Each term will be used at least\n   * once in as few pairs as possible.\n   * @param tokens\n   * @param createSingle If the set contains only one query, the returned list will contain a single\n   *                     Term for that query if createSingle is true, and an empty list otherwise.\n   * @param weight Each term pair will be given a score boost of serializedWeight.\n   * @return\n   */\n  private static List<Query> createTermPairs(Set<String> tokens, boolean createSingle,\n      double weight) {\n\n    List<Query> terms = Lists.newArrayList();\n    if (tokens.size() >= 2) {\n      int tokensLeft = tokens.size();\n      String token1 = null;\n      for (String token2 : tokens) {\n        if (token1 == null) {\n          token1 = token2;\n        } else {\n          terms.add(createHFTermPair(token1, token2, weight));\n\n          if (tokensLeft > 2) { // Only reset if there is more than one token remaining.\n            token1 = null;\n          }\n        }\n        tokensLeft--;\n      }\n    } else if (createSingle && !tokens.isEmpty()) { // Only one high frequency token\n      // Need to add token as a term because it was removed from the query earlier in rewriting.\n      Term newTerm = new Term(tokens.iterator().next());\n      terms.add(newTerm);\n    }\n\n    return terms;\n  }\n\n  private static List<Query> createPhrasePairs(Set<String> phrases, double weight) {\n    List<Query> ops = Lists.newArrayList();\n    for (String phrase : phrases) {\n      String[] terms = phrase.split(\" \");\n      assert terms.length == 2;\n      SearchOperator op = new SearchOperator(SearchOperator.Type.HF_PHRASE_PAIR,\n          terms[0], terms[1], Double.toString(weight));\n      ops.add(op);\n    }\n    return ops;\n  }\n\n  private static SearchOperator createHFTermPair(String token1, String token2, double weight) {\n    SearchOperator op = new SearchOperator(SearchOperator.Type.HF_TERM_PAIR,\n        token1, token2, Double.toString(weight));\n    return op;\n  }\n\n  private static boolean hasAnnotations(com.twitter.search.queryparser.query.Query node) {\n    return node.hasAnnotations();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/queryparser/HighFrequencyTermQueryGroup.java",
    "content": "package com.twitter.search.earlybird.queryparser;\n\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Set;\n\nimport com.google.common.collect.Sets;\n\n/**\n * Used to store information relevant to processing query groups for HighFrequencyTermPairExtractor\n * and HighFrequencyTermPairRewriter\n */\npublic class HighFrequencyTermQueryGroup {\n  protected final int groupIdx;\n  protected final int parentGroupIdx;\n  // The number of nodes in this group.\n  protected int numMembers = 0;\n  // For the rewrite visitor: Incremented once at the end of each of this group's nodes' visits.\n  protected int numVisits = 0;\n\n  // The set of tokens that should be removed from the query if seen as an individual term and\n  // rewritten in the query as a hf term pair.\n  protected final Set<String> hfTokens = Sets.newTreeSet();\n\n  // Tokens that can be used to restrict searches but should not be scored. They will be given a\n  // weight of 0.\n  protected final Set<String> preusedHFTokens = Sets.newTreeSet();\n\n  // Set of phrases that should be removed from the query if seen as an individual phrase and\n  // rewritten in the query as a hf term phrase pair.\n  protected final Set<String> hfPhrases = Sets.newTreeSet();\n\n  // Phrases that can be used to restrict searches but should not be scored. They will be given a\n  // weight of 0.\n  protected final Set<String> preusedHFPhrases = Sets.newTreeSet();\n\n  // The first found hf_term, or the hf_term of an ancestor with the same isPositive value.\n  protected String distributiveToken = null;\n\n  // If it is a single node group, isPositive is true iff that node is true.\n  // Otherwise, isPositive is false iff the root of the group is a disjunction.\n  protected final boolean isPositive;\n\n  public HighFrequencyTermQueryGroup(int groupIdx, boolean positive) {\n    this(groupIdx, -1, positive);\n  }\n\n  public HighFrequencyTermQueryGroup(int groupIdx, int parentGroupIdx, boolean positive) {\n    this.groupIdx = groupIdx;\n    this.parentGroupIdx = parentGroupIdx;\n    isPositive = positive;\n  }\n\n  public boolean hasPhrases() {\n    return !hfPhrases.isEmpty() || !preusedHFPhrases.isEmpty();\n  }\n\n  protected List<String> tokensFromPhrases() {\n    if (!hasPhrases()) {\n      return null;\n    }\n    List<String> tokens = new ArrayList<>();\n    for (String phrase : hfPhrases) {\n      for (String term : phrase.split(\" \")) {\n        tokens.add(term);\n      }\n    }\n    for (String phrase : preusedHFPhrases) {\n      for (String term : phrase.split(\" \")) {\n        tokens.add(term);\n      }\n    }\n    return tokens;\n  }\n\n  protected void removePreusedTokens() {\n    hfTokens.removeAll(preusedHFTokens);\n    List<String> phraseTokens = tokensFromPhrases();\n    if (phraseTokens != null) {\n      hfTokens.removeAll(phraseTokens);\n      preusedHFTokens.removeAll(phraseTokens);\n    }\n    hfPhrases.removeAll(preusedHFPhrases);\n  }\n\n  protected String getTokenFromPhrase() {\n    List<String> phraseTokens = tokensFromPhrases();\n    if (phraseTokens != null) {\n      return phraseTokens.get(0);\n    } else {\n      return null;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/queryparser/LuceneRelevanceQueryVisitor.java",
    "content": "package com.twitter.search.earlybird.queryparser;\n\nimport java.util.Map;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.lucene.search.Query;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.query.MappableField;\nimport com.twitter.search.common.schema.base.FieldWeightDefault;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.common.search.termination.QueryTimeout;\nimport com.twitter.search.earlybird.common.userupdates.UserScrubGeoMap;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.partition.MultiSegmentTermDictionaryManager;\nimport com.twitter.search.earlybird.querycache.QueryCacheManager;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\n\npublic class LuceneRelevanceQueryVisitor extends EarlybirdLuceneQueryVisitor {\n  public LuceneRelevanceQueryVisitor(\n      ImmutableSchemaInterface schema,\n      QueryCacheManager queryCacheManager,\n      UserTable userTable,\n      UserScrubGeoMap userScrubGeoMap,\n      TerminationTracker terminationTracker,\n      Map<String, FieldWeightDefault> fieldWeightMap,\n      Map<MappableField, String> mappableFieldMap,\n      MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager,\n      Decider decider,\n      EarlybirdCluster earlybirdCluster,\n      QueryTimeout queryTimeout) {\n    super(\n        schema,\n        queryCacheManager,\n        userTable,\n        userScrubGeoMap,\n        terminationTracker,\n        fieldWeightMap,\n        mappableFieldMap,\n        multiSegmentTermDictionaryManager,\n        decider,\n        earlybirdCluster,\n        queryTimeout);\n  }\n\n  @VisibleForTesting\n  protected LuceneRelevanceQueryVisitor(\n      ImmutableSchemaInterface schema,\n      QueryCacheManager queryCacheManager,\n      UserTable userTable,\n      UserScrubGeoMap userScrubGeoMap,\n      EarlybirdCluster earlybirdCluster) {\n    super(schema,\n          queryCacheManager,\n          userTable,\n          userScrubGeoMap,\n          earlybirdCluster,\n          queryCacheManager.getDecider());\n  }\n\n  @Override\n  protected Query visitSinceIDOperator(SearchOperator op) {\n    // since_id is handled by the blender for relevance queries, so don't filter on it.\n    return null;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/queryparser/ProtectedOperatorQueryRewriter.java",
    "content": "package com.twitter.search.earlybird.queryparser;\n\nimport java.util.List;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableList;\n\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.queryparser.query.Conjunction;\nimport com.twitter.search.queryparser.query.Disjunction;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\nimport com.twitter.search.queryparser.query.search.SearchOperatorConstants;\n\npublic class ProtectedOperatorQueryRewriter {\n  private static final String ERROR_MESSAGE = \"Positive 'protected' operator must be in the root\"\n      + \" query node and the root query node must be a Conjunction.\";\n  private static final Query EXCLUDE_PROTECTED_OPERATOR =\n      new SearchOperator(SearchOperator.Type.EXCLUDE, SearchOperatorConstants.PROTECTED);\n\n  /**\n   * Rewrite a query with positive 'protected' operator into an equivalent query without the positive\n   * 'protected' operator. This method assumes the following preconditions hold:\n   *  1. 'followedUserIds' is not empty\n   *  2. the query's root node is of type Conjunction\n   *  3. the query's root node is not negated\n   *  4. there is one positive 'protected' operator in the root node\n   *  5. there is only one 'protected' operator in the whole query\n   *\n   *  Query with '[include protected]' operator is rewritten into a Disjunction of a query with\n   *  protected Tweets only and a query with public Tweets only.\n   *  For example,\n   *    Original query:\n   *      (* \"cat\" [include protected])\n   *        with followedUserIds=[1, 7, 12] where 1 and 7 are protected users\n   *    Rewritten query:\n   *      (+\n   *        (* \"cat\" [multi_term_disjunction from_user_id 1 7])\n   *        (* \"cat\" [exclude protected])\n   *      )\n   *\n   *  Query with '[filter protected]' operator is rewritten with multi_term_disjunction from_user_id\n   *  operator.\n   *  For example,\n   *    Original query:\n   *      (* \"cat\" [filter protected])\n   *        with followedUserIds=[1, 7, 12] where 1 and 7 are protected users\n   *    Rewritten query:\n   *      (* \"cat\" [multi_term_disjunction from_user_id 1 7])\n   */\n  public Query rewrite(Query parsedQuery, List<Long> followedUserIds, UserTable userTable) {\n    Preconditions.checkState(followedUserIds != null && !followedUserIds.isEmpty(),\n        \"'followedUserIds' should not be empty when positive 'protected' operator exists.\");\n    Preconditions.checkState(\n        parsedQuery.isTypeOf(com.twitter.search.queryparser.query.Query.QueryType.CONJUNCTION),\n        ERROR_MESSAGE);\n    Conjunction parsedConjQuery = (Conjunction) parsedQuery;\n    List<Query> children = parsedConjQuery.getChildren();\n    int opIndex = findPositiveProtectedOperatorIndex(children);\n    Preconditions.checkState(opIndex >= 0, ERROR_MESSAGE);\n    SearchOperator protectedOp = (SearchOperator) children.get(opIndex);\n\n    ImmutableList.Builder<Query> otherChildrenBuilder = ImmutableList.builder();\n    otherChildrenBuilder.addAll(children.subList(0, opIndex));\n    if (opIndex + 1 < children.size()) {\n      otherChildrenBuilder.addAll(children.subList(opIndex + 1, children.size()));\n    }\n    List<Query> otherChildren = otherChildrenBuilder.build();\n\n    List<Long> protectedUserIds = getProtectedUserIds(followedUserIds, userTable);\n    if (protectedOp.getOperatorType() == SearchOperator.Type.FILTER) {\n      if (protectedUserIds.isEmpty()) {\n        // match none query\n        return Disjunction.EMPTY_DISJUNCTION;\n      } else {\n        return parsedConjQuery.newBuilder()\n            .setChildren(otherChildren)\n            .addChild(createFromUserIdMultiTermDisjunctionQuery(protectedUserIds))\n            .build();\n      }\n    } else {\n      // 'include' or negated 'exclude' operator\n      // negated 'exclude' is considered the same as 'include' to be consistent with the logic in\n      // EarlybirdLuceneQueryVisitor\n      if (protectedUserIds.isEmpty()) {\n        // return public only query\n        return parsedConjQuery.newBuilder()\n            .setChildren(otherChildren)\n            .addChild(EXCLUDE_PROTECTED_OPERATOR)\n            .build();\n      } else {\n        // build a disjunction of protected only query and public only query\n        Query protectedOnlyQuery = parsedConjQuery.newBuilder()\n            .setChildren(otherChildren)\n            .addChild(createFromUserIdMultiTermDisjunctionQuery(protectedUserIds))\n            .build();\n        Query publicOnlyQuery = parsedConjQuery.newBuilder()\n            .setChildren(otherChildren)\n            .addChild(EXCLUDE_PROTECTED_OPERATOR)\n            .build();\n        return new Disjunction(protectedOnlyQuery, publicOnlyQuery);\n      }\n    }\n  }\n\n  private Query createFromUserIdMultiTermDisjunctionQuery(List<Long> userIds) {\n    ImmutableList.Builder<String> operandsBuilder = ImmutableList.builder();\n    operandsBuilder\n        .add(EarlybirdFieldConstants.EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName());\n    for (Long userId : userIds) {\n      operandsBuilder.add(userId.toString());\n    }\n    List<String> operands = operandsBuilder.build();\n    return new SearchOperator(SearchOperator.Type.MULTI_TERM_DISJUNCTION, operands);\n  }\n\n  private List<Long> getProtectedUserIds(List<Long> followedUserIds, UserTable userTable) {\n    ImmutableList.Builder<Long> protectedUserIds = ImmutableList.builder();\n    for (Long userId : followedUserIds) {\n      if (userTable.isSet(userId, UserTable.IS_PROTECTED_BIT)) {\n        protectedUserIds.add(userId);\n      }\n    }\n    return protectedUserIds.build();\n  }\n\n  private int findPositiveProtectedOperatorIndex(List<Query> children) {\n    for (int i = 0; i < children.size(); i++) {\n      Query child = children.get(i);\n      if (child instanceof SearchOperator) {\n        SearchOperator searchOp = (SearchOperator) child;\n        if (SearchOperatorConstants.PROTECTED.equals(searchOp.getOperand())\n            && (isNegateExclude(searchOp) || isPositive(searchOp))) {\n          return i;\n        }\n      }\n    }\n\n    return -1;\n  }\n\n  private boolean isNegateExclude(SearchOperator searchOp) {\n    return searchOp.mustNotOccur()\n        && searchOp.getOperatorType() == SearchOperator.Type.EXCLUDE;\n  }\n\n  private boolean isPositive(SearchOperator searchOp) {\n    return !searchOp.mustNotOccur()\n        && (searchOp.getOperatorType() == SearchOperator.Type.INCLUDE\n        || searchOp.getOperatorType() == SearchOperator.Type.FILTER);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/AbstractResultsCollector.java",
    "content": "package com.twitter.search.earlybird.search;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Optional;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Maps;\nimport com.google.common.collect.Sets;\n\nimport org.apache.commons.collections.CollectionUtils;\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.ScoreMode;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.search.common.relevance.features.EarlybirdDocumentFeatures;\nimport com.twitter.search.common.results.thriftjava.FieldHitAttribution;\nimport com.twitter.search.common.results.thriftjava.FieldHitList;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.search.TwitterEarlyTerminationCollector;\nimport com.twitter.search.common.util.spatial.GeoUtil;\nimport com.twitter.search.core.earlybird.facets.AbstractFacetCountingArray;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentData;\nimport com.twitter.search.core.earlybird.index.TimeMapper;\nimport com.twitter.search.core.earlybird.index.inverted.QueryCostTracker;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.index.EarlybirdSingleSegmentSearcher;\nimport com.twitter.search.earlybird.index.TweetIDMapper;\nimport com.twitter.search.earlybird.search.facets.FacetLabelCollector;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.ThriftFacetLabel;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultExtraMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultGeoLocation;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.queryparser.util.IdTimeRanges;\n\nimport geo.google.datamodel.GeoCoordinate;\n\n/**\n * Abstract parent class for all results collectors in earlybird.\n * This collector should be able to handle both single-segment and\n * multi-segment collection.\n */\npublic abstract class AbstractResultsCollector<R extends SearchRequestInfo,\n    S extends SearchResultsInfo>\n    extends TwitterEarlyTerminationCollector {\n  enum IdAndRangeUpdateType {\n    BEGIN_SEGMENT,\n    END_SEGMENT,\n    HIT\n  }\n\n  // Earlybird used to have a special early termination logic: at segment boundaries\n  // the collector estimates how much time it'll take to search the next segment.\n  // If this estimate * 1.5 will cause the request to timeout, the search early terminates.\n  // That logic is removed in favor of more fine grained checks---now we check timeout\n  // within a segment, every 2,000,000 docs processed.\n  private static final int EXPENSIVE_TERMINATION_CHECK_INTERVAL =\n      EarlybirdConfig.getInt(\"expensive_termination_check_interval\", 2000000);\n\n  private static final long NO_TIME_SLICE_ID = -1;\n\n  protected final R searchRequestInfo;\n\n  // Sometimes maxHitsToProcess can also come from places other than collector params.\n  // E.g. from searchQuery.getRelevanceOptions(). This provides a way to allow\n  // subclasses to override the maxHitsToProcess on collector params.\n  private final long maxHitsToProcessOverride;\n\n  // min and max status id actually considered in the search (may not be a hit)\n  private long minSearchedStatusID = Long.MAX_VALUE;\n  private long maxSearchedStatusID = Long.MIN_VALUE;\n\n  private int minSearchedTime = Integer.MAX_VALUE;\n  private int maxSearchedTime = Integer.MIN_VALUE;\n\n  // per-segment start time. Will be re-started in setNextReader().\n  private long segmentStartTime;\n\n  // Current segment being searched.\n  protected EarlybirdIndexSegmentAtomicReader currTwitterReader;\n  protected TweetIDMapper tweetIdMapper;\n  protected TimeMapper timeMapper;\n  protected long currTimeSliceID = NO_TIME_SLICE_ID;\n\n  private final long queryTime;\n\n  // Time periods, in milliseconds, for which hits are counted.\n  private final List<Long> hitCountsThresholdsMsec;\n\n  // hitCounts[i] is the number of hits that are more recent than hitCountsThresholdsMsec[i]\n  private final int[] hitCounts;\n\n  private final ImmutableSchemaInterface schema;\n\n  private final EarlybirdSearcherStats searcherStats;\n  // For collectors that fill in the results' geo locations, this will be used to retrieve the\n  // documents' lat/lon coordinates.\n  private GeoCoordinate resultGeoCoordinate;\n  protected final boolean fillInLatLonForHits;\n\n  protected EarlybirdDocumentFeatures documentFeatures;\n  protected boolean featuresRequested = false;\n\n  private final FacetLabelCollector facetCollector;\n\n  // debugMode set in request to determine debugging level.\n  private int requestDebugMode;\n\n  // debug info to be returned in earlybird response\n  protected List<String> debugInfo;\n\n  private int numHitsCollectedPerSegment;\n\n  public AbstractResultsCollector(\n      ImmutableSchemaInterface schema,\n      R searchRequestInfo,\n      Clock clock,\n      EarlybirdSearcherStats searcherStats,\n      int requestDebugMode) {\n    super(searchRequestInfo.getSearchQuery().getCollectorParams(),\n        searchRequestInfo.getTerminationTracker(),\n        QueryCostTracker.getTracker(),\n        EXPENSIVE_TERMINATION_CHECK_INTERVAL,\n        clock);\n\n    this.schema = schema;\n    this.searchRequestInfo = searchRequestInfo;\n    ThriftSearchQuery thriftSearchQuery = searchRequestInfo.getSearchQuery();\n    this.maxHitsToProcessOverride = searchRequestInfo.getMaxHitsToProcess();\n    this.facetCollector = buildFacetCollector(searchRequestInfo, schema);\n\n    if (searchRequestInfo.getTimestamp() > 0) {\n      queryTime = searchRequestInfo.getTimestamp();\n    } else {\n      queryTime = System.currentTimeMillis();\n    }\n    hitCountsThresholdsMsec = thriftSearchQuery.getHitCountBuckets();\n    hitCounts = hitCountsThresholdsMsec == null || hitCountsThresholdsMsec.size() == 0\n        ? null\n        : new int[hitCountsThresholdsMsec.size()];\n\n    this.searcherStats = searcherStats;\n\n    Schema.FieldInfo latLonCSFField =\n        schema.hasField(EarlybirdFieldConstant.LAT_LON_CSF_FIELD.getFieldName())\n            ? schema.getFieldInfo(EarlybirdFieldConstant.LAT_LON_CSF_FIELD.getFieldName())\n            : null;\n    boolean loadLatLonMapperIntoRam = true;\n    if (latLonCSFField != null) {\n      // If the latlon_csf field is explicitly defined, then take the config from the schema.\n      // If it's not defined, we assume that the latlon mapper is stored in memory.\n      loadLatLonMapperIntoRam = latLonCSFField.getFieldType().isCsfLoadIntoRam();\n    }\n    // Default to not fill in lat/lon if the lat/lon CSF field is not loaded into RAM\n    this.fillInLatLonForHits = EarlybirdConfig.getBool(\"fill_in_lat_lon_for_hits\",\n        loadLatLonMapperIntoRam);\n    this.requestDebugMode = requestDebugMode;\n\n    if (shouldCollectDetailedDebugInfo()) {\n      this.debugInfo = new ArrayList<>();\n      debugInfo.add(\"Starting Search\");\n    }\n  }\n\n  private static FacetLabelCollector buildFacetCollector(\n      SearchRequestInfo request,\n      ImmutableSchemaInterface schema) {\n    if (CollectionUtils.isEmpty(request.getFacetFieldNames())) {\n      return null;\n    }\n\n    // Get all facet field ids requested.\n    Set<String> requiredFields = Sets.newHashSet();\n    for (String fieldName : request.getFacetFieldNames()) {\n      Schema.FieldInfo field = schema.getFacetFieldByFacetName(fieldName);\n      if (field != null) {\n        requiredFields.add(field.getFieldType().getFacetName());\n      }\n    }\n\n    if (requiredFields.size() > 0) {\n      return new FacetLabelCollector(requiredFields);\n    } else {\n      return null;\n    }\n  }\n\n  /**\n   * Subclasses should implement the following methods.\n   */\n\n  // Subclasses should process collected hits and construct a final\n  // AbstractSearchResults object.\n  protected abstract S doGetResults() throws IOException;\n\n  // Subclasses can override this method to add more collection logic.\n  protected abstract void doCollect(long tweetID) throws IOException;\n\n  public final ImmutableSchemaInterface getSchema() {\n    return schema;\n  }\n\n  // Updates the hit count array - each result only increments the first qualifying bucket.\n  protected final void updateHitCounts(long statusId) {\n    if (hitCounts == null) {\n      return;\n    }\n\n    long delta = queryTime - SnowflakeIdParser.getTimestampFromTweetId(statusId);\n    for (int i = 0; i < hitCountsThresholdsMsec.size(); ++i) {\n      if (delta >= 0 && delta < hitCountsThresholdsMsec.get(i)) {\n        hitCounts[i]++;\n        // Increments to the rest of the count array are implied, and aggregated later, since the\n        // array is sorted.\n        break;\n      }\n    }\n  }\n\n  private boolean searchedStatusIDsAndTimesInitialized() {\n    return maxSearchedStatusID != Long.MIN_VALUE;\n  }\n\n  // Updates the first searched status ID when starting to search a new segment.\n  private void updateFirstSearchedStatusID() {\n    // Only try to update the min/max searched ids, if this segment/reader actually has documents\n    // See SEARCH-4535\n    int minDocID = currTwitterReader.getSmallestDocID();\n    if (currTwitterReader.hasDocs() && minDocID >= 0 && !searchedStatusIDsAndTimesInitialized()) {\n      final long firstStatusID = tweetIdMapper.getTweetID(minDocID);\n      final int firstStatusTime = timeMapper.getTime(minDocID);\n      if (shouldCollectDetailedDebugInfo()) {\n        debugInfo.add(\n            \"updateFirstSearchedStatusID. minDocId=\" + minDocID + \", firstStatusID=\"\n                + firstStatusID + \", firstStatusTime=\" + firstStatusTime);\n      }\n      updateIDandTimeRanges(firstStatusID, firstStatusTime, IdAndRangeUpdateType.BEGIN_SEGMENT);\n    }\n  }\n\n  public final R getSearchRequestInfo() {\n    return searchRequestInfo;\n  }\n\n  public final long getMinSearchedStatusID() {\n    return minSearchedStatusID;\n  }\n\n  public final long getMaxSearchedStatusID() {\n    return maxSearchedStatusID;\n  }\n\n  public final int getMinSearchedTime() {\n    return minSearchedTime;\n  }\n\n  public boolean isSetMinSearchedTime() {\n    return minSearchedTime != Integer.MAX_VALUE;\n  }\n\n  public final int getMaxSearchedTime() {\n    return maxSearchedTime;\n  }\n\n  @Override\n  public final long getMaxHitsToProcess() {\n    return maxHitsToProcessOverride;\n  }\n\n  // Notifies classes that a new index segment is about to be searched.\n  @Override\n  public final void setNextReader(LeafReaderContext context) throws IOException {\n    super.setNextReader(context);\n    setNextReader(context.reader());\n  }\n\n  /**\n   * Notifies the collector that a new segment is about to be searched.\n   *\n   * It's easier to use this method from tests, because LeafReader is not a final class, so it can\n   * be mocked (unlike LeafReaderContext).\n   */\n  @VisibleForTesting\n  public final void setNextReader(LeafReader reader) throws IOException {\n    if (!(reader instanceof EarlybirdIndexSegmentAtomicReader)) {\n      throw new RuntimeException(\"IndexReader type not supported: \" + reader.getClass());\n    }\n\n    currTwitterReader = (EarlybirdIndexSegmentAtomicReader) reader;\n    documentFeatures = new EarlybirdDocumentFeatures(currTwitterReader);\n    tweetIdMapper = (TweetIDMapper) currTwitterReader.getSegmentData().getDocIDToTweetIDMapper();\n    timeMapper = currTwitterReader.getSegmentData().getTimeMapper();\n    currTimeSliceID = currTwitterReader.getSegmentData().getTimeSliceID();\n    updateFirstSearchedStatusID();\n    if (shouldCollectDetailedDebugInfo()) {\n      debugInfo.add(\"Starting search in segment with timeslice ID: \" + currTimeSliceID);\n    }\n\n    segmentStartTime = getClock().nowMillis();\n    startSegment();\n  }\n\n  protected abstract void startSegment() throws IOException;\n\n  @Override\n  protected final void doCollect() throws IOException {\n    documentFeatures.advance(curDocId);\n    long tweetID = tweetIdMapper.getTweetID(curDocId);\n    updateIDandTimeRanges(tweetID, timeMapper.getTime(curDocId), IdAndRangeUpdateType.HIT);\n    doCollect(tweetID);\n    numHitsCollectedPerSegment++;\n  }\n\n  protected void collectFeatures(ThriftSearchResultMetadata metadata) throws IOException {\n    if (featuresRequested) {\n      ensureExtraMetadataIsSet(metadata);\n\n      metadata.getExtraMetadata().setDirectedAtUserId(\n          documentFeatures.getFeatureValue(EarlybirdFieldConstant.DIRECTED_AT_USER_ID_CSF));\n      metadata.getExtraMetadata().setQuotedTweetId(\n          documentFeatures.getFeatureValue(EarlybirdFieldConstant.QUOTED_TWEET_ID_CSF));\n      metadata.getExtraMetadata().setQuotedUserId(\n          documentFeatures.getFeatureValue(EarlybirdFieldConstant.QUOTED_USER_ID_CSF));\n\n      int cardLangValue =\n          (int) documentFeatures.getFeatureValue(EarlybirdFieldConstant.CARD_LANG_CSF);\n      ThriftLanguage thriftLanguage = ThriftLanguage.findByValue(cardLangValue);\n      metadata.getExtraMetadata().setCardLang(thriftLanguage);\n\n      long cardNumericUri =\n          (long) documentFeatures.getFeatureValue(EarlybirdFieldConstant.CARD_URI_CSF);\n      if (cardNumericUri > 0) {\n        metadata.getExtraMetadata().setCardUri(String.format(\"card://%s\", cardNumericUri));\n      }\n    }\n  }\n\n  protected void collectIsProtected(\n      ThriftSearchResultMetadata metadata, EarlybirdCluster cluster, UserTable userTable)\n      throws IOException {\n    // 'isUserProtected' field is only set for archive cluster because only archive cluster user\n    // table has IS_PROTECTED_BIT populated.\n    // Since this bit is checked after UserFlagsExcludeFilter checked this bit, there is a slight\n    // chance that this bit is updated in-between. When that happens, it is possible that we will\n    // see a small number of protected Tweets in the response when we meant to exclude them.\n    if (cluster == EarlybirdCluster.FULL_ARCHIVE) {\n      ensureExtraMetadataIsSet(metadata);\n      long userId = documentFeatures.getFeatureValue(EarlybirdFieldConstant.FROM_USER_ID_CSF);\n      boolean isProtected = userTable.isSet(userId, UserTable.IS_PROTECTED_BIT);\n      metadata.getExtraMetadata().setIsUserProtected(isProtected);\n    }\n  }\n\n  protected void collectExclusiveConversationAuthorId(ThriftSearchResultMetadata metadata)\n      throws IOException {\n    if (searchRequestInfo.isCollectExclusiveConversationAuthorId()) {\n      long exclusiveConversationAuthorId = documentFeatures.getFeatureValue(\n          EarlybirdFieldConstant.EXCLUSIVE_CONVERSATION_AUTHOR_ID_CSF);\n      if (exclusiveConversationAuthorId != 0L) {\n        ensureExtraMetadataIsSet(metadata);\n        metadata.getExtraMetadata().setExclusiveConversationAuthorId(exclusiveConversationAuthorId);\n      }\n    }\n  }\n\n  // It only makes sense to collectFacets for search types that return individual results (recency,\n  // relevance and top_tweets), which use the AbstractRelevanceCollector and SearchResultsCollector,\n  // so this method should only be called from these classes.\n  protected void collectFacets(ThriftSearchResultMetadata metadata) {\n    if (currTwitterReader == null) {\n      return;\n    }\n\n    AbstractFacetCountingArray facetCountingArray = currTwitterReader.getFacetCountingArray();\n    EarlybirdIndexSegmentData segmentData = currTwitterReader.getSegmentData();\n\n    if (facetCountingArray == null || facetCollector == null) {\n      return;\n    }\n\n    facetCollector.resetFacetLabelProviders(\n        segmentData.getFacetLabelProviders(),\n        segmentData.getFacetIDMap());\n\n    facetCountingArray.collectForDocId(curDocId, facetCollector);\n\n    List<ThriftFacetLabel> labels = facetCollector.getLabels();\n    if (labels.size() > 0) {\n      metadata.setFacetLabels(labels);\n    }\n  }\n\n  protected void ensureExtraMetadataIsSet(ThriftSearchResultMetadata metadata) {\n    if (!metadata.isSetExtraMetadata()) {\n      metadata.setExtraMetadata(new ThriftSearchResultExtraMetadata());\n    }\n  }\n\n  @Override\n  protected final void doFinishSegment(int lastSearchedDocID) {\n    if (shouldCollectDetailedDebugInfo()) {\n      long timeSpentSearchingSegmentInMillis = getClock().nowMillis() - segmentStartTime;\n      debugInfo.add(\"Finished segment at doc id: \" + lastSearchedDocID);\n      debugInfo.add(\"Time spent searching \" + currTimeSliceID\n        + \": \" + timeSpentSearchingSegmentInMillis + \"ms\");\n      debugInfo.add(\"Number of hits collected in segment \" + currTimeSliceID + \": \"\n          + numHitsCollectedPerSegment);\n    }\n\n    if (!currTwitterReader.hasDocs()) {\n      // Due to race between the reader and the indexing thread, a seemingly empty segment that\n      // does not have document committed in the posting lists, might already have a document\n      // inserted into the id/time mappers, which we do not want to take into account.\n      // If there are no documents in the segment, we don't update searched min/max ids to\n      // anything.\n      return;\n    } else if (lastSearchedDocID == DocIdSetIterator.NO_MORE_DOCS) {\n      // Segment exhausted.\n      if (shouldCollectDetailedDebugInfo()) {\n        debugInfo.add(\"Segment exhausted\");\n      }\n      updateIDandTimeRanges(tweetIdMapper.getMinTweetID(), timeMapper.getFirstTime(),\n          IdAndRangeUpdateType.END_SEGMENT);\n    } else if (lastSearchedDocID >= 0) {\n      long lastSearchedTweetID = tweetIdMapper.getTweetID(lastSearchedDocID);\n      int lastSearchTweetTime = timeMapper.getTime(lastSearchedDocID);\n      if (shouldCollectDetailedDebugInfo()) {\n        debugInfo.add(\"lastSearchedDocId=\" + lastSearchedDocID);\n      }\n      updateIDandTimeRanges(lastSearchedTweetID, lastSearchTweetTime,\n          IdAndRangeUpdateType.END_SEGMENT);\n    }\n\n    numHitsCollectedPerSegment = 0;\n  }\n\n  private void updateIDandTimeRanges(long tweetID, int time, IdAndRangeUpdateType updateType) {\n    // We need to update minSearchedStatusID/maxSearchedStatusID and\n    // minSearchedTime/maxSearchedTime independently: SEARCH-6139\n    minSearchedStatusID = Math.min(minSearchedStatusID, tweetID);\n    maxSearchedStatusID = Math.max(maxSearchedStatusID, tweetID);\n    if (time > 0) {\n      minSearchedTime = Math.min(minSearchedTime, time);\n      maxSearchedTime = Math.max(maxSearchedTime, time);\n    }\n    if (shouldCollectVerboseDebugInfo()) {\n      debugInfo.add(\n          String.format(\"call to updateIDandTimeRanges(%d, %d, %s)\"\n                  + \" set minSearchStatusID=%d, maxSearchedStatusID=%d,\"\n                  + \" minSearchedTime=%d, maxSearchedTime=%d)\",\n              tweetID, time, updateType.toString(),\n              minSearchedStatusID, maxSearchedStatusID,\n              minSearchedTime, maxSearchedTime));\n    }\n  }\n\n  /**\n   * This is called when a segment is skipped but we would want to do accounting\n   * for minSearchDocId as well as numDocsProcessed.\n   */\n  public void skipSegment(EarlybirdSingleSegmentSearcher searcher) throws IOException {\n    setNextReader(searcher.getTwitterIndexReader().getContext());\n    trackCompleteSegment(DocIdSetIterator.NO_MORE_DOCS);\n    if (shouldCollectDetailedDebugInfo()) {\n      debugInfo.add(\"Skipping segment: \" + currTimeSliceID);\n    }\n  }\n\n  /**\n   * Returns the results collected by this collector.\n   */\n  public final S getResults() throws IOException {\n    // In order to make pagination work, if minSearchedStatusID is greater than the asked max_id.\n    // We force the minSearchedStatusID to be max_id + 1.\n    IdTimeRanges idTimeRanges = searchRequestInfo.getIdTimeRanges();\n    if (idTimeRanges != null) {\n      Optional<Long> maxIDInclusive = idTimeRanges.getMaxIDInclusive();\n      if (maxIDInclusive.isPresent() && minSearchedStatusID > maxIDInclusive.get()) {\n        searcherStats.numCollectorAdjustedMinSearchedStatusID.increment();\n        minSearchedStatusID = maxIDInclusive.get() + 1;\n      }\n    }\n\n    S results = doGetResults();\n    results.setNumHitsProcessed((int) getNumHitsProcessed());\n    results.setNumSearchedSegments(getNumSearchedSegments());\n    if (searchedStatusIDsAndTimesInitialized()) {\n      results.setMaxSearchedStatusID(maxSearchedStatusID);\n      results.setMinSearchedStatusID(minSearchedStatusID);\n      results.setMaxSearchedTime(maxSearchedTime);\n      results.setMinSearchedTime(minSearchedTime);\n    }\n    results.setEarlyTerminated(getEarlyTerminationState().isTerminated());\n    if (getEarlyTerminationState().isTerminated()) {\n      results.setEarlyTerminationReason(getEarlyTerminationState().getTerminationReason());\n    }\n    Map<Long, Integer> counts = getHitCountMap();\n    if (counts != null) {\n      results.hitCounts.putAll(counts);\n    }\n    return results;\n  }\n\n  /**\n   * Returns a map of timestamps (specified in the query) to the number of hits that are more recent\n   * that the respective timestamps.\n   */\n  public final Map<Long, Integer> getHitCountMap() {\n    int total = 0;\n    if (hitCounts == null) {\n      return null;\n    }\n    Map<Long, Integer> map = Maps.newHashMap();\n    // since the array is incremental, need to aggregate here.\n    for (int i = 0; i < hitCounts.length; ++i) {\n      map.put(hitCountsThresholdsMsec.get(i), total += hitCounts[i]);\n    }\n    return map;\n  }\n\n  /**\n   * Common helper for collecting per-field hit attribution data (if it's available).\n   *\n   * @param metadata the metadata to fill for this hit.\n   */\n  protected final void fillHitAttributionMetadata(ThriftSearchResultMetadata metadata) {\n    if (searchRequestInfo.getHitAttributeHelper() == null) {\n      return;\n    }\n\n    Map<Integer, List<String>> hitAttributeMapping =\n        searchRequestInfo.getHitAttributeHelper().getHitAttribution(curDocId);\n    Preconditions.checkNotNull(hitAttributeMapping);\n\n    FieldHitAttribution fieldHitAttribution = new FieldHitAttribution();\n    for (Map.Entry<Integer, List<String>> entry : hitAttributeMapping.entrySet()) {\n      FieldHitList fieldHitList = new FieldHitList();\n      fieldHitList.setHitFields(entry.getValue());\n\n      fieldHitAttribution.putToHitMap(entry.getKey(), fieldHitList);\n    }\n    metadata.setFieldHitAttribution(fieldHitAttribution);\n  }\n\n  /**\n   * Fill the geo location of the given document in metadata, if we have the lat/lon for it.\n   * For queries that specify a geolocation, this will also have the distance from\n   * the location specified in the query, and the location of this document.\n   */\n  protected final void fillResultGeoLocation(ThriftSearchResultMetadata metadata)\n      throws IOException {\n    Preconditions.checkNotNull(metadata);\n    if (currTwitterReader != null && fillInLatLonForHits) {\n      // See if we can have a lat/lon for this doc.\n      if (resultGeoCoordinate == null) {\n        resultGeoCoordinate = new GeoCoordinate();\n      }\n      // Only fill if necessary\n      if (searchRequestInfo.isCollectResultLocation()\n          && GeoUtil.decodeLatLonFromInt64(\n              documentFeatures.getFeatureValue(EarlybirdFieldConstant.LAT_LON_CSF_FIELD),\n              resultGeoCoordinate)) {\n        ThriftSearchResultGeoLocation resultLocation = new ThriftSearchResultGeoLocation();\n        resultLocation.setLatitude(resultGeoCoordinate.getLatitude());\n        resultLocation.setLongitude(resultGeoCoordinate.getLongitude());\n        metadata.setResultLocation(resultLocation);\n      }\n    }\n  }\n\n  @Override\n  public ScoreMode scoreMode() {\n    return ScoreMode.COMPLETE;\n  }\n\n  private int terminationDocID = -1;\n\n  @Override\n  protected void collectedEnoughResults() throws IOException {\n    // We find 'terminationDocID' once we collect enough results, so that we know the point at which\n    // we can stop searching. We must do this because with the unordered doc ID mapper, tweets\n    // are not ordered within a millisecond, so we must search the entire millisecond bucket before\n    // terminating the search, otherwise we could skip over tweets and have an incorrect\n    // minSearchedStatusID.\n    if (curDocId != -1 && terminationDocID == -1) {\n      long tweetId = tweetIdMapper.getTweetID(curDocId);\n      // We want to find the highest possible doc ID for this tweetId, so pass true.\n      boolean findMaxDocID = true;\n      terminationDocID = tweetIdMapper.findDocIdBound(tweetId,\n          findMaxDocID,\n          curDocId,\n          curDocId);\n    }\n  }\n\n  @Override\n  protected boolean shouldTerminate() {\n    return curDocId >= terminationDocID;\n  }\n\n  @Override\n  public List<String> getDebugInfo() {\n    return debugInfo;\n  }\n\n  protected boolean shouldCollectDetailedDebugInfo() {\n    return requestDebugMode >= 5;\n  }\n\n  // Use this for per-result debug info. Useful for queries with no results\n  // or a very small number of results.\n  protected boolean shouldCollectVerboseDebugInfo() {\n    return requestDebugMode >= 6;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/AntiGamingFilter.java",
    "content": "package com.twitter.search.earlybird.search;\n\nimport java.io.IOException;\nimport java.util.Comparator;\nimport java.util.HashSet;\nimport java.util.Set;\nimport java.util.SortedSet;\nimport java.util.TreeSet;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.commons.lang.mutable.MutableInt;\nimport org.apache.lucene.index.IndexReader;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\n\nimport com.twitter.common_internal.collections.RandomAccessPriorityQueue;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.search.TwitterIndexSearcher;\nimport com.twitter.search.common.util.analysis.LongTermAttributeImpl;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\npublic class AntiGamingFilter {\n  private interface Acceptor {\n    boolean accept(int internalDocID) throws IOException;\n  }\n\n  private NumericDocValues userReputation;\n  private NumericDocValues fromUserIDs;\n\n  private final Query luceneQuery;\n\n  private boolean termsExtracted = false;\n  private final Set<Term> queryTerms;\n\n  // we ignore these user ids for anti-gaming filtering, because they were explicitly queried for\n  private Set<Long> segmentUserIDWhitelist = null;\n  // we gather the whitelisted userIDs from all segments here\n  private Set<Long> globalUserIDWhitelist = null;\n\n  /**\n   * Used to track the number of occurrences of a particular user.\n   */\n  private static final class UserCount\n      implements RandomAccessPriorityQueue.SignatureProvider<Long> {\n    private long userID;\n    private int count;\n\n    @Override\n    public Long getSignature() {\n      return userID;\n    }\n\n    @Override\n    public void clear() {\n      userID = 0;\n      count = 0;\n    }\n  }\n\n  private static final Comparator<UserCount> USER_COUNT_COMPARATOR =\n      (d1, d2) -> d1.count == d2.count ? Long.compare(d1.userID, d2.userID) : d1.count - d2.count;\n\n  private final RandomAccessPriorityQueue<UserCount, Long> priorityQueue =\n      new RandomAccessPriorityQueue<UserCount, Long>(1024, USER_COUNT_COMPARATOR) {\n    @Override\n    protected UserCount getSentinelObject() {\n      return new UserCount();\n    }\n  };\n\n  private final Acceptor acceptor;\n  private final int maxHitsPerUser;\n\n  /**\n   * Creates an AntiGamingFilter that either accepts or rejects tweets from all users.\n   * This method should only be called in tests.\n   *\n   * @param alwaysValue Determines if tweets should always be accepted or rejected.\n   * @return An AntiGamingFilter that either accepts or rejects tweets from all users.\n   */\n  @VisibleForTesting\n  public static AntiGamingFilter newMock(boolean alwaysValue) {\n    return new AntiGamingFilter(alwaysValue) {\n      @Override\n      public void startSegment(EarlybirdIndexSegmentAtomicReader reader) {\n      }\n    };\n  }\n\n  private AntiGamingFilter(boolean alwaysValue) {\n    acceptor = internalDocID -> alwaysValue;\n    maxHitsPerUser = Integer.MAX_VALUE;\n    termsExtracted = true;\n    luceneQuery = null;\n    queryTerms = null;\n  }\n\n  public AntiGamingFilter(int maxHitsPerUser, int maxTweepCred, Query luceneQuery) {\n    this.maxHitsPerUser = maxHitsPerUser;\n    this.luceneQuery = luceneQuery;\n\n    if (maxTweepCred != -1) {\n      this.acceptor = internalDocID -> {\n        long userReputationVal =\n            userReputation.advanceExact(internalDocID) ? userReputation.longValue() : 0L;\n        return ((byte) userReputationVal > maxTweepCred) || acceptUser(internalDocID);\n      };\n    } else {\n      this.acceptor = this::acceptUser;\n    }\n\n    this.queryTerms = new HashSet<>();\n  }\n\n  public Set<Long> getUserIDWhitelist() {\n    return globalUserIDWhitelist;\n  }\n\n  private boolean acceptUser(int internalDocID) throws IOException {\n    final long fromUserID = getUserId(internalDocID);\n    final MutableInt freq = new MutableInt();\n    // try to increment UserCount for an user already exist in the priority queue.\n    boolean incremented = priorityQueue.incrementElement(\n        fromUserID, element -> freq.setValue(++element.count));\n\n    // If not incremented, it means the user node does not exist in the priority queue yet.\n    if (!incremented) {\n      priorityQueue.updateTop(element -> {\n        element.userID = fromUserID;\n        element.count = 1;\n        freq.setValue(element.count);\n      });\n    }\n\n    if (freq.intValue() <= maxHitsPerUser) {\n      return true;\n    } else if (segmentUserIDWhitelist == null) {\n      return false;\n    }\n    return segmentUserIDWhitelist.contains(fromUserID);\n  }\n\n  /**\n   * Initializes this filter with the new feature source. This method should be called every time an\n   * earlybird searcher starts searching in a new segment.\n   *\n   * @param reader The reader for the new segment.\n   */\n  public void startSegment(EarlybirdIndexSegmentAtomicReader reader) throws IOException {\n    if (!termsExtracted) {\n      extractTerms(reader);\n    }\n\n    fromUserIDs =\n        reader.getNumericDocValues(EarlybirdFieldConstant.FROM_USER_ID_CSF.getFieldName());\n\n    // fill the id whitelist for the current segment.  initialize lazily.\n    segmentUserIDWhitelist = null;\n\n    SortedSet<Integer> sortedFromUserDocIds = new TreeSet<>();\n    for (Term t : queryTerms) {\n      if (t.field().equals(EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName())) {\n        // Add the operand of the from_user_id operator to the whitelist\n        long fromUserID = LongTermAttributeImpl.copyBytesRefToLong(t.bytes());\n        addUserToWhitelists(fromUserID);\n      } else if (t.field().equals(EarlybirdFieldConstant.FROM_USER_FIELD.getFieldName())) {\n        // For a [from X] filter, we need to find a document that has the from_user field set to X,\n        // and then we need to get the value of the from_user_id field for that document and add it\n        // to the whitelist. We can get the from_user_id value from the fromUserIDs NumericDocValues\n        // instance, but we need to traverse it in increasing order of doc IDs. So we add a doc ID\n        // for each term to a sorted set for now, and then we traverse it in increasing doc ID order\n        // and add the from_user_id values for those docs to the whitelist.\n        int firstInternalDocID = reader.getNewestDocID(t);\n        if (firstInternalDocID != EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND) {\n          sortedFromUserDocIds.add(firstInternalDocID);\n        }\n      }\n    }\n\n    for (int fromUserDocId : sortedFromUserDocIds) {\n      addUserToWhitelists(getUserId(fromUserDocId));\n    }\n\n    userReputation =\n        reader.getNumericDocValues(EarlybirdFieldConstant.USER_REPUTATION.getFieldName());\n\n    // Reset the fromUserIDs NumericDocValues so that the acceptor can use it to iterate over docs.\n    fromUserIDs =\n        reader.getNumericDocValues(EarlybirdFieldConstant.FROM_USER_ID_CSF.getFieldName());\n  }\n\n  private void extractTerms(IndexReader reader) throws IOException {\n    Query query = luceneQuery;\n    for (Query rewrittenQuery = query.rewrite(reader); rewrittenQuery != query;\n         rewrittenQuery = query.rewrite(reader)) {\n      query = rewrittenQuery;\n    }\n\n    // Create a new TwitterIndexSearcher instance here instead of an IndexSearcher instance, to use\n    // the TwitterIndexSearcher.collectionStatistics() implementation.\n    query.createWeight(new TwitterIndexSearcher(reader), ScoreMode.COMPLETE, 1.0f)\n        .extractTerms(queryTerms);\n    termsExtracted = true;\n  }\n\n  public boolean accept(int internalDocID) throws IOException {\n    return acceptor.accept(internalDocID);\n  }\n\n  private void addUserToWhitelists(long userID) {\n    if (this.segmentUserIDWhitelist == null) {\n      this.segmentUserIDWhitelist = new HashSet<>();\n    }\n    if (this.globalUserIDWhitelist == null) {\n      this.globalUserIDWhitelist = new HashSet<>();\n    }\n    this.segmentUserIDWhitelist.add(userID);\n    this.globalUserIDWhitelist.add(userID);\n  }\n\n  @VisibleForTesting\n  protected long getUserId(int internalDocId) throws IOException {\n    return fromUserIDs.advanceExact(internalDocId) ? fromUserIDs.longValue() : 0L;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/EarlybirdLuceneSearcher.java",
    "content": "package com.twitter.search.earlybird.search;\n\nimport java.io.IOException;\nimport java.util.Map;\n\nimport org.apache.lucene.index.IndexReader;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.IndexSearcher;\n\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.earlybird.EarlybirdSearcher;\nimport com.twitter.search.earlybird.search.facets.AbstractFacetTermCollector;\nimport com.twitter.search.earlybird.search.facets.FacetResultsCollector;\nimport com.twitter.search.earlybird.search.facets.TermStatisticsCollector.TermStatisticsSearchResults;\nimport com.twitter.search.earlybird.search.facets.TermStatisticsRequestInfo;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCount;\nimport com.twitter.search.earlybird.thrift.ThriftFacetFieldResults;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird.thrift.ThriftTermStatisticsResults;\n\npublic abstract class EarlybirdLuceneSearcher extends IndexSearcher {\n  public EarlybirdLuceneSearcher(IndexReader r) {\n    super(r);\n  }\n\n  /**\n   * Fills facet information for all given search results.\n   *\n   * @param collector A collector that knows how collect facet information.\n   * @param searchResults The search results.\n   */\n  public abstract void fillFacetResults(\n      AbstractFacetTermCollector collector, ThriftSearchResults searchResults)\n      throws IOException;\n\n  /**\n   * Fills metadata for all given facet results.\n   *\n   * @param facetResults The facet results.\n   * @param schema The earlybird schema.\n   * @param debugMode The debug mode for the request that yielded these results.\n   */\n  public abstract void fillFacetResultMetadata(\n      Map<Term, ThriftFacetCount> facetResults,\n      ImmutableSchemaInterface schema,\n      byte debugMode) throws IOException;\n\n  /**\n   * Fills metadata for all given term stats results.\n   *\n   * @param termStatsResults The term stats results.\n   * @param schema The earlybird schema.\n   * @param debugMode The debug mode for the request that yielded these results.\n   */\n  public abstract void fillTermStatsMetadata(\n      ThriftTermStatisticsResults termStatsResults,\n      ImmutableSchemaInterface schema,\n      byte debugMode) throws IOException;\n\n  /**\n   * Returns the results for the given term stats request.\n   *\n   * @param searchRequestInfo Stores the original term stats request and some other useful request\n   *                          information.\n   * @param searcher The searcher that should be used to execute the request.\n   * @param requestDebugMode The debug mode for this request.\n   * @return The term stats results for the given request.\n   */\n  public abstract TermStatisticsSearchResults collectTermStatistics(\n      TermStatisticsRequestInfo searchRequestInfo,\n      EarlybirdSearcher searcher,\n      int requestDebugMode) throws IOException;\n\n  /**\n   * Writes an explanation for the given hits into the given ThriftSearchResults instance.\n   *\n   * @param searchRequestInfo Stores the original request and some other useful request context.\n   * @param hits The hits.\n   * @param searchResults The ThriftSearchResults where the explanation for the given hits will be\n   *                      stored.\n   */\n  // Writes explanations into the searchResults thrift.\n  public abstract void explainSearchResults(SearchRequestInfo searchRequestInfo,\n                                            SimpleSearchResults hits,\n                                            ThriftSearchResults searchResults) throws IOException;\n\n  public static class FacetSearchResults extends SearchResultsInfo {\n    private FacetResultsCollector collector;\n\n    public FacetSearchResults(FacetResultsCollector collector) {\n      this.collector = collector;\n    }\n\n    public ThriftFacetFieldResults getFacetResults(String facetName, int topK) {\n      return collector.getFacetResults(facetName, topK);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/EarlybirdMultiSegmentSearcher.java",
    "content": "package com.twitter.search.earlybird.search;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.HashSet;\nimport java.util.LinkedHashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.stream.Collectors;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.MultiReader;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.Collector;\nimport org.apache.lucene.search.Explanation;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentData;\nimport com.twitter.search.earlybird.EarlybirdSearcher;\nimport com.twitter.search.earlybird.index.EarlybirdSingleSegmentSearcher;\nimport com.twitter.search.earlybird.index.TweetIDMapper;\nimport com.twitter.search.earlybird.search.facets.AbstractFacetTermCollector;\nimport com.twitter.search.earlybird.search.facets.TermStatisticsCollector;\nimport com.twitter.search.earlybird.search.facets.TermStatisticsCollector.TermStatisticsSearchResults;\nimport com.twitter.search.earlybird.search.facets.TermStatisticsRequestInfo;\nimport com.twitter.search.earlybird.search.queries.SinceMaxIDFilter;\nimport com.twitter.search.earlybird.search.queries.SinceUntilFilter;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCount;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird.thrift.ThriftTermStatisticsResults;\nimport com.twitter.search.queryparser.util.IdTimeRanges;\n\npublic class EarlybirdMultiSegmentSearcher extends EarlybirdLuceneSearcher {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdMultiSegmentSearcher.class);\n\n  private final ImmutableSchemaInterface schema;\n  private final Map<Long, EarlybirdSingleSegmentSearcher> segmentSearchers;\n  protected final int numSegments;\n  private final Clock clock;\n\n  // This will prevent us from even considering segments that are out of range.\n  // It's an important optimization for a certain class of queries.\n  protected IdTimeRanges idTimeRanges = null;\n\n  private final EarlybirdSearcherStats searcherStats;\n\n  public EarlybirdMultiSegmentSearcher(\n      ImmutableSchemaInterface schema,\n      List<EarlybirdSingleSegmentSearcher> searchers,\n      EarlybirdSearcherStats searcherStats,\n      Clock clock) throws IOException {\n    // NOTE: We pass in an empty MultiReader to super and retain the list of searchers in this\n    // class since MultiReader does not allow an aggregate of more than Integer.MAX_VALUE docs,\n    // which some of our larger archive indexes may have.\n    super(new MultiReader());\n    // segmentSearchers are mapped from time slice IDs to searchers so that we can quickly\n    // find the correct searcher for a given time slice ID (see fillPayload).\n    // make sure we maintain order of segments, hence a LinkedHashMap instead of just a HashMap\n    this.segmentSearchers = new LinkedHashMap<>();\n    this.schema = schema;\n    for (EarlybirdSingleSegmentSearcher searcher : searchers) {\n      if (searcher != null) {\n        long timeSliceID = searcher.getTimeSliceID();\n        this.segmentSearchers.put(timeSliceID, searcher);\n      }\n    }\n    // initializing this after populating the list.  previously initialized before, and\n    // this may have lead to a race condition, although this doesn't seem possible given\n    // that segments should be an immutable cloned list.\n    this.numSegments = segmentSearchers.size();\n\n    this.searcherStats = searcherStats;\n    this.clock = clock;\n  }\n\n  public void setIdTimeRanges(IdTimeRanges idTimeRanges) {\n    this.idTimeRanges = idTimeRanges;\n  }\n\n  @Override\n  protected void search(List<LeafReaderContext> unusedLeaves, Weight weight, Collector coll)\n      throws IOException {\n    Preconditions.checkState(coll instanceof AbstractResultsCollector);\n    AbstractResultsCollector<?, ?> collector = (AbstractResultsCollector<?, ?>) coll;\n\n    for (EarlybirdSingleSegmentSearcher segmentSearcher : segmentSearchers.values()) {\n      if (shouldSkipSegment(segmentSearcher)) {\n        collector.skipSegment(segmentSearcher);\n      } else {\n        segmentSearcher.search(weight.getQuery(), collector);\n        if (collector.isTerminated()) {\n          break;\n        }\n      }\n    }\n  }\n\n  @VisibleForTesting\n  protected boolean shouldSkipSegment(EarlybirdSingleSegmentSearcher segmentSearcher) {\n    EarlybirdIndexSegmentData segmentData =\n        segmentSearcher.getTwitterIndexReader().getSegmentData();\n    if (idTimeRanges != null) {\n      if (!SinceMaxIDFilter.sinceMaxIDsInRange(\n              (TweetIDMapper) segmentData.getDocIDToTweetIDMapper(),\n              idTimeRanges.getSinceIDExclusive().or(SinceMaxIDFilter.NO_FILTER),\n              idTimeRanges.getMaxIDInclusive().or(SinceMaxIDFilter.NO_FILTER))\n          || !SinceUntilFilter.sinceUntilTimesInRange(\n              segmentData.getTimeMapper(),\n              idTimeRanges.getSinceTimeInclusive().or(SinceUntilFilter.NO_FILTER),\n              idTimeRanges.getUntilTimeExclusive().or(SinceUntilFilter.NO_FILTER))) {\n        return true;\n      }\n    }\n    return false;\n  }\n\n  @Override\n  public void fillFacetResults(\n      AbstractFacetTermCollector collector, ThriftSearchResults searchResults) throws IOException {\n    for (EarlybirdSingleSegmentSearcher segmentSearcher : segmentSearchers.values()) {\n      segmentSearcher.fillFacetResults(collector, searchResults);\n    }\n  }\n\n  @Override\n  public TermStatisticsSearchResults collectTermStatistics(\n      TermStatisticsRequestInfo searchRequestInfo,\n      EarlybirdSearcher searcher,\n      int requestDebugMode) throws IOException {\n    TermStatisticsCollector collector = new TermStatisticsCollector(\n        schema, searchRequestInfo, searcherStats, clock, requestDebugMode);\n    search(collector.getSearchRequestInfo().getLuceneQuery(), collector);\n    searcher.maybeSetCollectorDebugInfo(collector);\n    return collector.getResults();\n  }\n\n  @Override\n  public void explainSearchResults(SearchRequestInfo searchRequestInfo,\n      SimpleSearchResults hits, ThriftSearchResults searchResults) throws IOException {\n    for (EarlybirdSingleSegmentSearcher segmentSearcher : segmentSearchers.values()) {\n      // the hits that are getting passed into this method are hits across\n      // all searched segments. We need to get the per segment hits and\n      // generate explanations one segment at a time.\n      List<Hit> hitsForCurrentSegment = new ArrayList<>();\n      Set<Long> tweetIdsForCurrentSegment = new HashSet<>();\n      List<ThriftSearchResult> hitResultsForCurrentSegment = new ArrayList<>();\n\n      for (Hit hit : hits.hits) {\n        if (hit.getTimeSliceID() == segmentSearcher.getTimeSliceID()) {\n          hitsForCurrentSegment.add(hit);\n          tweetIdsForCurrentSegment.add(hit.statusID);\n        }\n      }\n      for (ThriftSearchResult result : searchResults.getResults()) {\n        if (tweetIdsForCurrentSegment.contains(result.id)) {\n          hitResultsForCurrentSegment.add(result);\n        }\n      }\n      ThriftSearchResults resultsForSegment = new ThriftSearchResults()\n          .setResults(hitResultsForCurrentSegment);\n\n      SimpleSearchResults finalHits = new SimpleSearchResults(hitsForCurrentSegment);\n      segmentSearcher.explainSearchResults(searchRequestInfo, finalHits, resultsForSegment);\n    }\n    // We should not see hits that are not associated with an active segment\n    List<Hit> hitsWithUnknownSegment =\n        Arrays.stream(hits.hits()).filter(hit -> !hit.isHasExplanation())\n            .collect(Collectors.toList());\n    for (Hit hit : hitsWithUnknownSegment) {\n      LOG.error(\"Unable to find segment associated with hit: \" + hit.toString());\n    }\n  }\n\n  @Override\n  public void fillFacetResultMetadata(Map<Term, ThriftFacetCount> facetResults,\n                                      ImmutableSchemaInterface documentSchema, byte debugMode)\n      throws IOException {\n    for (EarlybirdSingleSegmentSearcher segmentSearcher : segmentSearchers.values()) {\n      segmentSearcher.fillFacetResultMetadata(facetResults, documentSchema, debugMode);\n    }\n  }\n\n  @Override\n  public void fillTermStatsMetadata(ThriftTermStatisticsResults termStatsResults,\n                                    ImmutableSchemaInterface documentSchema, byte debugMode)\n      throws IOException {\n    for (EarlybirdSingleSegmentSearcher segmentSearcher : segmentSearchers.values()) {\n      segmentSearcher.fillTermStatsMetadata(termStatsResults, documentSchema, debugMode);\n    }\n  }\n\n  /**\n   * The searchers for individual segments will rewrite the query as they see fit, so the multi\n   * segment searcher does not need to rewrite it. In fact, not rewriting the query here improves\n   * the request latency by ~5%.\n   */\n  @Override\n  public Query rewrite(Query original) {\n    return original;\n  }\n\n  /**\n   * The searchers for individual segments will create their own weights. This method only creates\n   * a dummy weight to pass the Lucene query to the search() method of these individual segment\n   * searchers.\n   */\n  @Override\n  public Weight createWeight(Query query, ScoreMode scoreMode, float boost) {\n    return new DummyWeight(query);\n  }\n\n  /**\n   * Dummy weight used solely to pass Lucene Query around.\n   */\n  private static final class DummyWeight extends Weight {\n    private DummyWeight(Query luceneQuery) {\n      super(luceneQuery);\n    }\n\n    @Override\n    public Explanation explain(LeafReaderContext context, int doc) {\n      throw new UnsupportedOperationException();\n    }\n\n    @Override\n    public Scorer scorer(LeafReaderContext context) {\n      throw new UnsupportedOperationException();\n    }\n\n    @Override\n    public void extractTerms(Set<Term> terms) {\n      throw new UnsupportedOperationException();\n    }\n\n    @Override\n    public boolean isCacheable(LeafReaderContext context) {\n      return true;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/GeoQuadTreeQueryBuilder.java",
    "content": "package com.twitter.search.earlybird.search;\n\nimport java.io.IOException;\nimport java.util.LinkedHashSet;\nimport java.util.Set;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.spatial.prefix.tree.Cell;\nimport org.apache.lucene.spatial.prefix.tree.CellIterator;\nimport org.apache.lucene.util.BytesRef;\nimport org.locationtech.spatial4j.shape.Rectangle;\n\nimport com.twitter.search.common.query.MultiTermDisjunctionQuery;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.search.GeoQuadTreeQueryBuilderUtil;\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.common.util.spatial.BoundingBox;\nimport com.twitter.search.common.util.spatial.GeoUtil;\nimport com.twitter.search.common.util.spatial.GeohashChunkImpl;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.earlybird.search.queries.GeoTwoPhaseQuery;\nimport com.twitter.search.earlybird.search.queries.GeoTwoPhaseQuery.SecondPhaseDocAccepter;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.util.GeoCode;\n\nimport geo.google.datamodel.GeoCoordinate;\n\n/**\n * A class that builds queries to query the quadtree.\n */\npublic final class GeoQuadTreeQueryBuilder {\n  private GeoQuadTreeQueryBuilder() {\n  }\n\n  /**\n   * Returns a GeoTwoPhaseQuery for the given geocode.\n   */\n  public static Query buildGeoQuadTreeQuery(final GeoCode geocode) {\n    return buildGeoQuadTreeQuery(geocode, null);\n  }\n\n  /**\n   * Returns a GeoTwoPhaseQuery for the given geocode.\n   *\n   * @param geocode The geocode.\n   * @param terminationTracker The tracker that determines when the query needs to terminate.\n   */\n  public static Query buildGeoQuadTreeQuery(GeoCode geocode,\n                                            TerminationTracker terminationTracker) {\n    Query geoHashDisjuntiveQuery = GeoQuadTreeQueryBuilderUtil.buildGeoQuadTreeQuery(\n        geocode, EarlybirdFieldConstant.GEO_HASH_FIELD.getFieldName());\n\n    // 5. Create post filtering accepter\n    final SecondPhaseDocAccepter accepter = (geocode.distanceKm != GeoCode.DOUBLE_DISTANCE_NOT_SET)\n            ? new CenterRadiusAccepter(geocode.latitude, geocode.longitude, geocode.distanceKm)\n            : GeoTwoPhaseQuery.ALL_DOCS_ACCEPTER;\n\n    return new GeoTwoPhaseQuery(geoHashDisjuntiveQuery, accepter, terminationTracker);\n  }\n\n  /**\n   * Construct a query as below:\n   *   1. Compute all quadtree cells that intersects the bounding box.\n   *   2. Create a disjunction of the geohashes of all the intersecting cells.\n   *   3. Add a filter to only keep points inside the giving bounding box.\n   */\n  public static Query buildGeoQuadTreeQuery(final Rectangle boundingBox,\n                                            final TerminationTracker terminationTracker)\n      throws QueryParserException {\n    // 1. Locate the main quadtree cell---the cell containing the bounding box's center point whose\n    // diagonal is just longer than the bounding box's diagonal.\n    final Cell centerCell = GeohashChunkImpl.getGeoNodeByBoundingBox(boundingBox);\n\n    // 2. Determine quadtree level to search.\n    int treeLevel = -1;\n    if (centerCell != null) {\n      treeLevel = centerCell.getLevel();\n    } else {\n      // This should not happen.\n      throw new QueryParserException(\n          \"Unable to locate quadtree cell containing the given bounding box.\"\n          + \"Bounding box is: \" + boundingBox);\n    }\n\n    // 3. get all quadtree cells at treeLevel that intersects the given bounding box.\n    CellIterator intersectingCells =\n        GeohashChunkImpl.getNodesIntersectingBoundingBox(boundingBox, treeLevel);\n\n    // 4. Construct disjunction query\n    final Set<BytesRef> geoHashSet = new LinkedHashSet<>();\n\n    // Add center node\n    geoHashSet.add(centerCell.getTokenBytesNoLeaf(new BytesRef()));\n    // If there are other nodes intersecting query circle, also add them in.\n    if (intersectingCells != null) {\n      while (intersectingCells.hasNext()) {\n        geoHashSet.add(intersectingCells.next().getTokenBytesNoLeaf(new BytesRef()));\n      }\n    }\n    MultiTermDisjunctionQuery geoHashDisjuntiveQuery = new MultiTermDisjunctionQuery(\n        EarlybirdFieldConstant.GEO_HASH_FIELD.getFieldName(), geoHashSet);\n\n    // 5. Create post filtering accepter\n    final GeoDocAccepter accepter = new BoundingBoxAccepter(boundingBox);\n\n    return new GeoTwoPhaseQuery(geoHashDisjuntiveQuery, accepter, terminationTracker);\n  }\n\n  private abstract static class GeoDocAccepter extends SecondPhaseDocAccepter {\n    private NumericDocValues latLonDocValues;\n    private final GeoCoordinate geoCoordReuse = new GeoCoordinate();\n\n    @Override\n    public void initialize(LeafReaderContext context) throws IOException {\n      final EarlybirdIndexSegmentAtomicReader reader =\n          (EarlybirdIndexSegmentAtomicReader) context.reader();\n      latLonDocValues =\n          reader.getNumericDocValues(EarlybirdFieldConstant.LAT_LON_CSF_FIELD.getFieldName());\n    }\n\n    // Decides whether a point should be accepted.\n    protected abstract boolean acceptPoint(double lat, double lon);\n\n    // Decides whether a document should be accepted based on its geo coordinates.\n    @Override\n    public final boolean accept(int internalDocId) throws IOException {\n      // Cannot obtain valid geo coordinates for the document. Not acceptable.\n      if (latLonDocValues == null\n          || !latLonDocValues.advanceExact(internalDocId)\n          || !GeoUtil.decodeLatLonFromInt64(latLonDocValues.longValue(), geoCoordReuse)) {\n        return false;\n      }\n\n      return acceptPoint(geoCoordReuse.getLatitude(), geoCoordReuse.getLongitude());\n    }\n  }\n\n  // Accepts points within a circle defined by a center point and a radius.\n  private static final class CenterRadiusAccepter extends GeoDocAccepter {\n    private final double centerLat;\n    private final double centerLon;\n    private final double radiusKm;\n\n    public CenterRadiusAccepter(double centerLat, double centerLon, double radiusKm) {\n      this.centerLat = centerLat;\n      this.centerLon = centerLon;\n      this.radiusKm = radiusKm;\n    }\n\n    @Override\n    protected boolean acceptPoint(double lat, double lon) {\n      double actualDistance =\n          BoundingBox.approxDistanceC(centerLat, centerLon, lat, lon);\n      if (actualDistance < radiusKm) {\n        return true;\n      } else if (Double.isNaN(actualDistance)) {\n        // There seems to be a rare bug in GeoUtils that computes NaN\n        // for two identical lat/lon pairs on occasion. Check for that here.\n        if (lat == centerLat && lon == centerLon) {\n          return true;\n        }\n      }\n\n      return false;\n    }\n\n    @Override\n    public String toString() {\n      return String.format(\"CenterRadiusAccepter(Center: %.4f, %.4f Radius (km): %.4f)\",\n              centerLat, centerLon, radiusKm);\n    }\n  }\n\n  // Accepts points within a BoundingBox\n  private static final class BoundingBoxAccepter extends GeoDocAccepter {\n    private final Rectangle boundingBox;\n\n    public BoundingBoxAccepter(Rectangle boundingBox)  {\n      this.boundingBox = boundingBox;\n    }\n\n    @Override\n    protected boolean acceptPoint(double lat, double lon) {\n      return GeohashChunkImpl.isPointInBoundingBox(lat, lon, boundingBox);\n\n    }\n\n    @Override\n    public String toString() {\n      return String.format(\"PointInBoundingBoxAccepter((%.4f, %.4f), (%.4f, %.4f), \"\n              + \"crossesDateLine=%b)\",\n              boundingBox.getMinY(), boundingBox.getMinX(),\n              boundingBox.getMaxY(), boundingBox.getMaxX(),\n              boundingBox.getCrossesDateLine());\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/Hit.java",
    "content": "package com.twitter.search.earlybird.search;\n\nimport javax.annotation.Nullable;\n\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\n\n/**\n * Class that abstracts a document that matches a query we're processing in Earlybird.\n */\npublic class Hit implements Comparable<Hit> {\n  protected long timeSliceID;\n  protected long statusID;\n  private boolean hasExplanation;\n\n  @Nullable\n  protected ThriftSearchResultMetadata metadata;\n\n  public Hit(long timeSliceID, long statusID) {\n    this.timeSliceID = timeSliceID;\n    this.statusID = statusID;\n    this.metadata = null;\n  }\n\n  public long getTimeSliceID() {\n    return timeSliceID;\n  }\n\n  public long getStatusID() {\n    return statusID;\n  }\n\n  @Nullable\n  public ThriftSearchResultMetadata getMetadata() {\n    return metadata;\n  }\n\n  public void setMetadata(ThriftSearchResultMetadata metadata) {\n    this.metadata = metadata;\n  }\n\n  @Override\n  public int compareTo(Hit other) {\n    return -Long.compare(this.statusID, other.statusID);\n  }\n\n  @Override\n  public String toString() {\n    return \"Hit[tweetID=\" + statusID + \",timeSliceID=\" + timeSliceID\n        + \",score=\" + (metadata == null ? \"null\" : metadata.getScore()) + \"]\";\n  }\n\n  public boolean isHasExplanation() {\n    return hasExplanation;\n  }\n\n  public void setHasExplanation(boolean hasExplanation) {\n    this.hasExplanation = hasExplanation;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/SearchRequestInfo.java",
    "content": "package com.twitter.search.earlybird.search;\n\nimport java.util.List;\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.search.Query;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.query.HitAttributeHelper;\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.earlybird.QualityFactor;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.queryparser.util.IdTimeRanges;\n\npublic class SearchRequestInfo {\n  private final ThriftSearchQuery searchQuery;\n  private final Query luceneQuery;\n  private final boolean collectConversationId;\n  private final boolean collectResultLocation;\n  private final boolean getInReplyToStatusId;\n  private final boolean getReferenceAuthorId;\n  private final boolean getFromUserId;\n  private final boolean collectExclusiveConversationAuthorId;\n\n  private final int numResultsRequested;\n  private final int maxHitsToProcess;\n  private final List<String> facetFieldNames;\n  private long timestamp;\n\n  private final TerminationTracker terminationTracker;\n\n  protected final QualityFactor qualityFactor;\n\n  // Set if we want to collect per-field hit attributes for this request.\n  @Nullable\n  private HitAttributeHelper hitAttributeHelper;\n\n  private IdTimeRanges idTimeRanges;\n\n  private static final int DEFAULT_MAX_HITS = 1000;\n\n  private static final SearchCounter RESET_MAX_HITS_TO_PROCESS_COUNTER =\n      SearchCounter.export(\"search_request_info_reset_max_hits_to_process\");\n\n  public SearchRequestInfo(\n      ThriftSearchQuery searchQuery,\n      Query luceneQuery,\n      TerminationTracker terminationTracker) {\n    this(searchQuery, luceneQuery, terminationTracker, null);\n  }\n\n  public SearchRequestInfo(\n      ThriftSearchQuery searchQuery,\n      Query luceneQuery,\n      TerminationTracker terminationTracker,\n      QualityFactor qualityFactor) {\n    Preconditions.checkNotNull(searchQuery.getCollectorParams());\n    Preconditions.checkNotNull(terminationTracker);\n\n    this.searchQuery = searchQuery;\n    this.luceneQuery = luceneQuery;\n    this.collectConversationId = searchQuery.isCollectConversationId();\n    if (searchQuery.isSetResultMetadataOptions()) {\n      this.collectResultLocation = searchQuery.getResultMetadataOptions().isGetResultLocation();\n      this.getInReplyToStatusId = searchQuery.getResultMetadataOptions().isGetInReplyToStatusId();\n      this.getReferenceAuthorId =\n          searchQuery.getResultMetadataOptions().isGetReferencedTweetAuthorId();\n      this.getFromUserId = searchQuery.getResultMetadataOptions().isGetFromUserId();\n      this.collectExclusiveConversationAuthorId =\n          searchQuery.getResultMetadataOptions().isGetExclusiveConversationAuthorId();\n    } else {\n      this.collectResultLocation = false;\n      this.getInReplyToStatusId = false;\n      this.getReferenceAuthorId = false;\n      this.getFromUserId = false;\n      this.collectExclusiveConversationAuthorId = false;\n    }\n\n    this.qualityFactor = qualityFactor;\n\n    this.numResultsRequested = searchQuery.getCollectorParams().getNumResultsToReturn();\n    this.maxHitsToProcess = calculateMaxHitsToProcess(searchQuery);\n    this.terminationTracker = terminationTracker;\n    this.facetFieldNames = searchQuery.getFacetFieldNames();\n  }\n\n  /**\n   * Gets the value to be used as max hits to process for this query. The base class gets it from\n   * the searchQuery directly, and uses a default if that's not set.\n   *\n   * Subclasses can override this to compute a different value for max hits to process.\n   */\n  protected int calculateMaxHitsToProcess(ThriftSearchQuery thriftSearchQuery) {\n    int maxHits = thriftSearchQuery.getCollectorParams().isSetTerminationParams()\n        ? thriftSearchQuery.getCollectorParams().getTerminationParams().getMaxHitsToProcess() : 0;\n\n    if (maxHits <= 0) {\n      maxHits = DEFAULT_MAX_HITS;\n      RESET_MAX_HITS_TO_PROCESS_COUNTER.increment();\n    }\n    return maxHits;\n  }\n\n  public final ThriftSearchQuery getSearchQuery() {\n    return this.searchQuery;\n  }\n\n  public Query getLuceneQuery() {\n    return luceneQuery;\n  }\n\n  public final int getNumResultsRequested() {\n    return numResultsRequested;\n  }\n\n  public final int getMaxHitsToProcess() {\n    return maxHitsToProcess;\n  }\n\n  public boolean isCollectConversationId() {\n    return collectConversationId;\n  }\n\n  public boolean isCollectResultLocation() {\n    return collectResultLocation;\n  }\n\n  public boolean isGetInReplyToStatusId() {\n    return getInReplyToStatusId;\n  }\n\n  public boolean isGetReferenceAuthorId() {\n    return getReferenceAuthorId;\n  }\n\n  public boolean isCollectExclusiveConversationAuthorId() {\n    return collectExclusiveConversationAuthorId;\n  }\n\n  public final IdTimeRanges getIdTimeRanges() {\n    return idTimeRanges;\n  }\n\n  public SearchRequestInfo setIdTimeRanges(IdTimeRanges newIdTimeRanges) {\n    this.idTimeRanges = newIdTimeRanges;\n    return this;\n  }\n\n  public SearchRequestInfo setTimestamp(long newTimestamp) {\n    this.timestamp = newTimestamp;\n    return this;\n  }\n\n  public long getTimestamp() {\n    return timestamp;\n  }\n\n  public TerminationTracker getTerminationTracker() {\n    return this.terminationTracker;\n  }\n\n  @Nullable\n  public HitAttributeHelper getHitAttributeHelper() {\n    return hitAttributeHelper;\n  }\n\n  public void setHitAttributeHelper(@Nullable HitAttributeHelper hitAttributeHelper) {\n    this.hitAttributeHelper = hitAttributeHelper;\n  }\n\n  public List<String> getFacetFieldNames() {\n    return facetFieldNames;\n  }\n\n  public boolean isGetFromUserId() {\n    return getFromUserId;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/SearchResultsCollector.java",
    "content": "package com.twitter.search.earlybird.search;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Set;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.features.thrift.ThriftSearchResultFeatures;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.search.EarlyTerminationState;\nimport com.twitter.search.common.util.LongIntConverter;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadataOptions;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultType;\n\n/**\n * This class collects results for Recency queries for delegation to collectors based on query mode\n */\npublic class SearchResultsCollector\n    extends AbstractResultsCollector<SearchRequestInfo, SimpleSearchResults> {\n  private static final EarlyTerminationState TERMINATED_COLLECTED_ENOUGH_RESULTS =\n      new EarlyTerminationState(\"terminated_collected_enough_results\", true);\n\n  protected final List<Hit> results;\n  private final Set<Integer> requestedFeatureIds;\n  private final EarlybirdCluster cluster;\n  private final UserTable userTable;\n\n  public SearchResultsCollector(\n      ImmutableSchemaInterface schema,\n      SearchRequestInfo searchRequestInfo,\n      Clock clock,\n      EarlybirdSearcherStats searcherStats,\n      EarlybirdCluster cluster,\n      UserTable userTable,\n      int requestDebugMode) {\n    super(schema, searchRequestInfo, clock, searcherStats, requestDebugMode);\n    results = new ArrayList<>();\n    this.cluster = cluster;\n    this.userTable = userTable;\n\n    ThriftSearchResultMetadataOptions options =\n        searchRequestInfo.getSearchQuery().getResultMetadataOptions();\n    if (options != null && options.isReturnSearchResultFeatures()) {\n      requestedFeatureIds = schema.getSearchFeatureSchema().getEntries().keySet();\n    } else if (options != null && options.isSetRequestedFeatureIDs()) {\n      requestedFeatureIds = new HashSet<>(options.getRequestedFeatureIDs());\n    } else {\n      requestedFeatureIds = null;\n    }\n  }\n\n  @Override\n  public void startSegment() throws IOException {\n    featuresRequested = requestedFeatureIds != null;\n  }\n\n  @Override\n  public void doCollect(long tweetID) throws IOException {\n    Hit hit = new Hit(currTimeSliceID, tweetID);\n    ThriftSearchResultMetadata metadata =\n        new ThriftSearchResultMetadata(ThriftSearchResultType.RECENCY)\n            .setPenguinVersion(EarlybirdConfig.getPenguinVersionByte());\n\n    // Set tweet language in metadata\n    ThriftLanguage thriftLanguage = ThriftLanguage.findByValue(\n        (int) documentFeatures.getFeatureValue(EarlybirdFieldConstant.LANGUAGE));\n    metadata.setLanguage(thriftLanguage);\n\n    // Check and collect hit attribution data, if it's available.\n    fillHitAttributionMetadata(metadata);\n\n    // Set the nullcast flag in metadata\n    metadata.setIsNullcast(documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_NULLCAST_FLAG));\n\n    if (searchRequestInfo.isCollectConversationId()) {\n      long conversationId =\n          documentFeatures.getFeatureValue(EarlybirdFieldConstant.CONVERSATION_ID_CSF);\n      if (conversationId != 0) {\n        ensureExtraMetadataIsSet(metadata);\n        metadata.getExtraMetadata().setConversationId(conversationId);\n      }\n    }\n\n    fillResultGeoLocation(metadata);\n    collectRetweetAndReplyMetadata(metadata);\n\n    long fromUserId = documentFeatures.getFeatureValue(EarlybirdFieldConstant.FROM_USER_ID_CSF);\n    if (requestedFeatureIds != null) {\n      ThriftSearchResultFeatures features = documentFeatures.getSearchResultFeatures(\n          getSchema(), requestedFeatureIds::contains);\n      ensureExtraMetadataIsSet(metadata);\n      metadata.getExtraMetadata().setFeatures(features);\n      metadata.setFromUserId(fromUserId);\n      if (documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_CARD_FLAG)) {\n        metadata.setCardType(\n            (byte) documentFeatures.getFeatureValue(EarlybirdFieldConstant.CARD_TYPE_CSF_FIELD));\n      }\n    }\n    if (searchRequestInfo.isGetFromUserId()) {\n      metadata.setFromUserId(fromUserId);\n    }\n\n    collectExclusiveConversationAuthorId(metadata);\n    collectFacets(metadata);\n    collectFeatures(metadata);\n    collectIsProtected(metadata, cluster, userTable);\n    hit.setMetadata(metadata);\n    results.add(hit);\n    updateHitCounts(tweetID);\n  }\n\n  private final void collectRetweetAndReplyMetadata(ThriftSearchResultMetadata metadata)\n      throws IOException {\n    if (searchRequestInfo.isGetInReplyToStatusId() || searchRequestInfo.isGetReferenceAuthorId()) {\n      boolean isRetweet = documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_RETWEET_FLAG);\n      boolean isReply = documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_REPLY_FLAG);\n      // Set the isRetweet and isReply metadata so that clients who request retweet and reply\n      // metadata know whether a result is a retweet or reply or neither.\n      metadata.setIsRetweet(isRetweet);\n      metadata.setIsReply(isReply);\n\n      // Only store the shared status id if the hit is a reply or a retweet and\n      // the getInReplyToStatusId flag is set.\n      if (searchRequestInfo.isGetInReplyToStatusId() && (isReply || isRetweet)) {\n        long sharedStatusID =\n            documentFeatures.getFeatureValue(EarlybirdFieldConstant.SHARED_STATUS_ID_CSF);\n        if (sharedStatusID != 0) {\n          metadata.setSharedStatusId(sharedStatusID);\n        }\n      }\n\n      // Only store the reference tweet author ID if the hit is a reply or a retweet and the\n      // getReferenceAuthorId flag is set.\n      if (searchRequestInfo.isGetReferenceAuthorId() && (isReply || isRetweet)) {\n        // the REFERENCE_AUTHOR_ID_CSF stores the source tweet author id for all retweets\n        long referenceAuthorId =\n            documentFeatures.getFeatureValue(EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_CSF);\n        if (referenceAuthorId != 0) {\n          metadata.setReferencedTweetAuthorId(referenceAuthorId);\n        } else if (cluster != EarlybirdCluster.FULL_ARCHIVE) {\n          // we also store the reference author id for retweets, directed at tweets, and self\n          // threaded tweets separately on Realtime/Protected Earlybirds. This data will be moved to\n          // the REFERENCE_AUTHOR_ID_CSF and these fields will be deprecated in SEARCH-34958.\n          referenceAuthorId = LongIntConverter.convertTwoIntToOneLong(\n              (int) documentFeatures.getFeatureValue(\n                  EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_MOST_SIGNIFICANT_INT),\n              (int) documentFeatures.getFeatureValue(\n                  EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_LEAST_SIGNIFICANT_INT));\n          if (referenceAuthorId > 0) {\n            metadata.setReferencedTweetAuthorId(referenceAuthorId);\n          }\n        }\n      }\n    }\n  }\n\n  /**\n   * This differs from base class because we check against num results collected instead of\n   * num hits collected.\n   */\n  @Override\n  public EarlyTerminationState innerShouldCollectMore() throws IOException {\n    if (results.size() >= searchRequestInfo.getNumResultsRequested()) {\n      collectedEnoughResults();\n      if (shouldTerminate()) {\n        return setEarlyTerminationState(TERMINATED_COLLECTED_ENOUGH_RESULTS);\n      }\n    }\n    return EarlyTerminationState.COLLECTING;\n  }\n\n  @Override\n  public SimpleSearchResults doGetResults() {\n    // Sort hits by tweet id.\n    Collections.sort(results);\n    return new SimpleSearchResults(results);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/SearchResultsInfo.java",
    "content": "package com.twitter.search.earlybird.search;\n\nimport java.util.Map;\n\nimport com.google.common.collect.Maps;\n\nimport com.twitter.search.earlybird.search.queries.SinceMaxIDFilter;\n\npublic class SearchResultsInfo {\n  public static final long NO_ID = SinceMaxIDFilter.NO_FILTER;\n  public static final int NO_TIME = -1;\n\n  private int numHitsProcessed = 0;\n  private int numSearchedSegments = 0;\n\n  private boolean earlyTerminated = false;\n  private String earlyTerminationReason = null;\n\n  private long maxSearchedStatusID = NO_ID;\n  private long minSearchedStatusID = NO_ID;\n\n  private int maxSearchedTime = NO_TIME;\n  private int minSearchedTime = NO_TIME;\n\n  // Map from time thresholds (in milliseconds) to number of results more recent than this period.\n  protected final Map<Long, Integer> hitCounts = Maps.newHashMap();\n\n  public final int getNumHitsProcessed() {\n    return numHitsProcessed;\n  }\n\n  public final void setNumHitsProcessed(int numHitsProcessed) {\n    this.numHitsProcessed = numHitsProcessed;\n  }\n\n  public final int getNumSearchedSegments() {\n    return numSearchedSegments;\n  }\n\n  public final void setNumSearchedSegments(int numSearchedSegments) {\n    this.numSearchedSegments = numSearchedSegments;\n  }\n\n  public final long getMaxSearchedStatusID() {\n    return maxSearchedStatusID;\n  }\n\n  public final long getMinSearchedStatusID() {\n    return minSearchedStatusID;\n  }\n\n  public final int getMaxSearchedTime() {\n    return maxSearchedTime;\n  }\n\n  public final int getMinSearchedTime() {\n    return minSearchedTime;\n  }\n\n  public boolean isSetSearchedStatusIDs() {\n    return maxSearchedStatusID != NO_ID && minSearchedStatusID != NO_ID;\n  }\n\n  public boolean isSetSearchedTimes() {\n    return maxSearchedTime != NO_TIME && minSearchedTime != NO_TIME;\n  }\n\n  public void setMaxSearchedStatusID(long maxSearchedStatusID) {\n    this.maxSearchedStatusID = maxSearchedStatusID;\n  }\n\n  public void setMinSearchedStatusID(long minSearchedStatusID) {\n    this.minSearchedStatusID = minSearchedStatusID;\n  }\n\n  public void setMaxSearchedTime(int maxSearchedTime) {\n    this.maxSearchedTime = maxSearchedTime;\n  }\n\n  public void setMinSearchedTime(int minSearchedTime) {\n    this.minSearchedTime = minSearchedTime;\n  }\n\n  public void setEarlyTerminated(boolean earlyTerminated) {\n    this.earlyTerminated = earlyTerminated;\n  }\n\n  public boolean isEarlyTerminated() {\n    return earlyTerminated;\n  }\n\n  public String getEarlyTerminationReason() {\n    return earlyTerminationReason;\n  }\n\n  public void setEarlyTerminationReason(String earlyTerminationReason) {\n    this.earlyTerminationReason = earlyTerminationReason;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/SimpleSearchResults.java",
    "content": "package com.twitter.search.earlybird.search;\n\nimport java.util.List;\n\npublic class SimpleSearchResults extends SearchResultsInfo {\n  protected Hit[] hits;\n  protected int numHits;\n\n  public SimpleSearchResults(int size) {\n    this.hits = new Hit[size];\n    this.numHits = 0;\n  }\n\n  public SimpleSearchResults(List<Hit> hits) {\n    this.hits = new Hit[hits.size()];\n    this.numHits = hits.size();\n    hits.toArray(this.hits);\n  }\n\n  public Hit[] hits() {\n    return hits;\n  }\n\n  public int numHits() {\n    return numHits;\n  }\n\n  public void setNumHits(int numHits) {\n    this.numHits = numHits;\n  }\n\n  public Hit getHit(int hitIndex) {\n    return hits[hitIndex];\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/SocialFilter.java",
    "content": "package com.twitter.search.earlybird.search;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.primitives.Longs;\n\nimport org.apache.lucene.index.NumericDocValues;\n\nimport com.twitter.common_internal.bloomfilter.BloomFilter;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.earlybird.thrift.ThriftSocialFilterType;\n\n/**\n * Filter class used by the SearchResultsCollector to filter social tweets\n * from the hits.\n */\npublic class SocialFilter {\n  private interface Acceptor {\n    boolean accept(long fromUserLong, byte[] userIDInBytes);\n  }\n\n  private NumericDocValues fromUserID;\n  private final Acceptor acceptor;\n  private final long searcherId;\n  private final BloomFilter trustedFilter;\n  private final BloomFilter followFilter;\n\n  private class FollowsAcceptor implements Acceptor {\n    @Override\n    public boolean accept(long fromUserLong, byte[] userIdInBytes) {\n      return followFilter.contains(userIdInBytes);\n    }\n  }\n\n  private class TrustedAcceptor implements Acceptor {\n    @Override\n    public boolean accept(long fromUserLong, byte[] userIdInBytes) {\n      return trustedFilter.contains(userIdInBytes);\n    }\n  }\n\n  private class AllAcceptor implements Acceptor {\n    @Override\n    public boolean accept(long fromUserLong, byte[] userIdInBytes) {\n      return trustedFilter.contains(userIdInBytes)\n          || followFilter.contains(userIdInBytes)\n          || fromUserLong == searcherId;\n    }\n  }\n\n  public SocialFilter(\n      ThriftSocialFilterType socialFilterType,\n      final long searcherId,\n      final byte[] trustedFilter,\n      final byte[] followFilter) throws IOException {\n    Preconditions.checkNotNull(socialFilterType);\n    Preconditions.checkNotNull(trustedFilter);\n    Preconditions.checkNotNull(followFilter);\n    this.searcherId = searcherId;\n    this.trustedFilter = new BloomFilter(trustedFilter);\n    this.followFilter = new BloomFilter(followFilter);\n\n\n    switch (socialFilterType) {\n      case FOLLOWS:\n        this.acceptor = new FollowsAcceptor();\n        break;\n      case TRUSTED:\n        this.acceptor = new TrustedAcceptor();\n        break;\n      case ALL:\n        this.acceptor = new AllAcceptor();\n        break;\n      default:\n        throw new UnsupportedOperationException(\"Invalid social filter type passed\");\n    }\n  }\n\n  public void startSegment(EarlybirdIndexSegmentAtomicReader indexReader) throws IOException {\n    fromUserID =\n        indexReader.getNumericDocValues(EarlybirdFieldConstant.FROM_USER_ID_CSF.getFieldName());\n  }\n\n  /**\n   * Determines if the given doc ID should be accepted.\n   */\n  public boolean accept(int internalDocID) throws IOException {\n    if (!fromUserID.advanceExact(internalDocID)) {\n      return false;\n    }\n\n    long fromUserLong = fromUserID.longValue();\n    byte[] userIDInBytes = Longs.toByteArray(fromUserLong);\n    return acceptor.accept(fromUserLong, userIDInBytes);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/SocialSearchResultsCollector.java",
    "content": "package com.twitter.search.earlybird.search;\n\nimport java.io.IOException;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\n\n/**\n * Created with IntelliJ IDEA.\n * Date: 6/20/12\n * Time: 12:06 PM\n * To change this template use File | Settings | File Templates.\n */\npublic class SocialSearchResultsCollector extends SearchResultsCollector {\n\n  private final SocialFilter socialFilter;\n\n  public SocialSearchResultsCollector(\n      ImmutableSchemaInterface schema,\n      SearchRequestInfo searchRequestInfo,\n      SocialFilter socialFilter,\n      EarlybirdSearcherStats searcherStats,\n      EarlybirdCluster cluster,\n      UserTable userTable,\n      int requestDebugMode) {\n    super(schema, searchRequestInfo, Clock.SYSTEM_CLOCK, searcherStats, cluster, userTable,\n        requestDebugMode);\n    this.socialFilter = socialFilter;\n  }\n\n  @Override\n  public final void doCollect(long tweetID) throws IOException {\n    if (socialFilter == null || socialFilter.accept(curDocId)) {\n      results.add(new Hit(currTimeSliceID, tweetID));\n    }\n  }\n\n  @Override\n  public void startSegment() throws IOException {\n    if (socialFilter != null) {\n      socialFilter.startSegment(currTwitterReader);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/AbstractFacetTermCollector.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.util.Map;\nimport java.util.Set;\n\nimport com.twitter.search.core.earlybird.facets.FacetIDMap;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.facets.FacetTermCollector;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultExtraMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\n\npublic abstract class AbstractFacetTermCollector implements FacetTermCollector {\n  private Map<String, FacetLabelProvider> facetLabelProviders;\n  private FacetIDMap facetIdMap;\n\n  /**\n   * Populates the given ThriftSearchResult instance with the results collected by this collector\n   * and clears all collected results in this collector.\n   *\n   * @param result The ThriftSearchResult instance to be populated with the results collected in\n   *               this collector.\n   */\n  public abstract void fillResultAndClear(ThriftSearchResult result);\n\n  public void resetFacetLabelProviders(\n      Map<String, FacetLabelProvider> facetLabelProvidersToReset, FacetIDMap facetIdMapToReset) {\n    this.facetLabelProviders = facetLabelProvidersToReset;\n    this.facetIdMap = facetIdMapToReset;\n  }\n\n  String findFacetName(int fieldId) {\n    return fieldId < 0 ? null : facetIdMap.getFacetFieldByFacetID(fieldId).getFacetName();\n  }\n\n  protected ThriftSearchResultExtraMetadata getExtraMetadata(ThriftSearchResult result) {\n    ThriftSearchResultMetadata metadata = result.getMetadata();\n    if (!metadata.isSetExtraMetadata()) {\n      metadata.setExtraMetadata(new ThriftSearchResultExtraMetadata());\n    }\n    return metadata.getExtraMetadata();\n  }\n\n  protected String getTermFromProvider(\n      String facetName, long termID, FacetLabelProvider provider) {\n    return provider.getLabelAccessor().getTermText(termID);\n  }\n\n  protected String getTermFromFacet(long termID, int fieldID, Set<String> facetsToCollectFrom) {\n    if (termID == EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND) {\n      return null;\n    }\n\n    String facetName = findFacetName(fieldID);\n    if (!facetsToCollectFrom.contains(facetName)) {\n      return null;\n    }\n\n    final FacetLabelProvider provider = facetLabelProviders.get(facetName);\n    if (provider == null) {\n      return null;\n    }\n\n    return getTermFromProvider(facetName, termID, provider);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/DefaultFacetScorer.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.io.IOException;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.ranking.thriftjava.ThriftFacetEarlybirdSortingMode;\nimport com.twitter.search.common.ranking.thriftjava.ThriftFacetRankingOptions;\nimport com.twitter.search.common.relevance.features.EarlybirdDocumentFeatures;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.util.lang.ThriftLanguageUtil;\nimport com.twitter.search.core.earlybird.facets.FacetAccumulator;\nimport com.twitter.search.core.earlybird.facets.FacetCountIterator;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.earlybird.search.AntiGamingFilter;\nimport com.twitter.search.earlybird.search.facets.FacetResultsCollector.Accumulator;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\n\npublic class DefaultFacetScorer extends FacetScorer {\n  private static final Logger LOG = LoggerFactory.getLogger(FacetScorer.class.getName());\n  private static final double DEFAULT_FEATURE_WEIGHT = 0.0;\n  private static final byte DEFAULT_PENALTY = 1;\n\n  private static final byte DEFAULT_REPUTATION_MIN = 45;\n\n  private final AntiGamingFilter antiGamingFilter;\n\n  // tweepcreds below this value will not be counted at all\n  private final byte reputationMinFilterThresholdVal;\n\n  // tweepcreds between reputationMinFilterThresholdVal and this value will be counted\n  // with a score of 1\n  private final byte reputationMinScoreVal;\n\n  private final double userRepWeight;\n  private final double favoritesWeight;\n  private final double parusWeight;\n  private final double parusBase;\n  private final double queryIndependentPenaltyWeight;\n\n  private final ThriftLanguage uiLang;\n  private final double langEnglishUIBoost;\n  private final double langEnglishFacetBoost;\n  private final double langDefaultBoost;\n\n  private final int antigamingPenalty;\n  private final int offensiveTweetPenalty;\n  private final int multipleHashtagsOrTrendsPenalty;\n\n  private final int maxScorePerTweet;\n  private final ThriftFacetEarlybirdSortingMode sortingMode;\n\n  private EarlybirdIndexSegmentAtomicReader reader;\n  private EarlybirdDocumentFeatures features;\n\n  /**\n   * Creates a new facet scorer.\n   */\n  public DefaultFacetScorer(ThriftSearchQuery searchQuery,\n                            ThriftFacetRankingOptions rankingOptions,\n                            AntiGamingFilter antiGamingFilter,\n                            ThriftFacetEarlybirdSortingMode sortingMode) {\n    this.sortingMode = sortingMode;\n    this.antiGamingFilter = antiGamingFilter;\n\n    maxScorePerTweet =\n        rankingOptions.isSetMaxScorePerTweet()\n        ? rankingOptions.getMaxScorePerTweet()\n        : Integer.MAX_VALUE;\n\n    // filters\n    reputationMinFilterThresholdVal =\n        rankingOptions.isSetMinTweepcredFilterThreshold()\n        ? (byte) (rankingOptions.getMinTweepcredFilterThreshold() & 0xFF)\n        : DEFAULT_REPUTATION_MIN;\n\n    // weights\n    // reputationMinScoreVal must be >= reputationMinFilterThresholdVal\n    reputationMinScoreVal =\n        (byte) Math.max(rankingOptions.isSetReputationParams()\n        ? (byte) rankingOptions.getReputationParams().getMin()\n        : DEFAULT_REPUTATION_MIN, reputationMinFilterThresholdVal);\n\n    parusWeight =\n        rankingOptions.isSetParusScoreParams() && rankingOptions.getParusScoreParams().isSetWeight()\n        ? rankingOptions.getParusScoreParams().getWeight()\n        : DEFAULT_FEATURE_WEIGHT;\n    // compute this once so that base ** parusScore is backwards-compatible\n    parusBase = Math.sqrt(1 + parusWeight);\n\n    userRepWeight =\n        rankingOptions.isSetReputationParams() && rankingOptions.getReputationParams().isSetWeight()\n        ? rankingOptions.getReputationParams().getWeight()\n        : DEFAULT_FEATURE_WEIGHT;\n\n    favoritesWeight =\n        rankingOptions.isSetFavoritesParams() && rankingOptions.getFavoritesParams().isSetWeight()\n        ? rankingOptions.getFavoritesParams().getWeight()\n        : DEFAULT_FEATURE_WEIGHT;\n\n    queryIndependentPenaltyWeight =\n        rankingOptions.isSetQueryIndependentPenaltyWeight()\n        ? rankingOptions.getQueryIndependentPenaltyWeight()\n        : DEFAULT_FEATURE_WEIGHT;\n\n    // penalty increment\n    antigamingPenalty =\n        rankingOptions.isSetAntigamingPenalty()\n        ? rankingOptions.getAntigamingPenalty()\n        : DEFAULT_PENALTY;\n\n    offensiveTweetPenalty =\n        rankingOptions.isSetOffensiveTweetPenalty()\n        ? rankingOptions.getOffensiveTweetPenalty()\n        : DEFAULT_PENALTY;\n\n    multipleHashtagsOrTrendsPenalty =\n        rankingOptions.isSetMultipleHashtagsOrTrendsPenalty()\n        ? rankingOptions.getMultipleHashtagsOrTrendsPenalty()\n        : DEFAULT_PENALTY;\n\n    // query information\n    if (!searchQuery.isSetUiLang() || searchQuery.getUiLang().isEmpty()) {\n      uiLang = ThriftLanguage.UNKNOWN;\n    } else {\n      uiLang = ThriftLanguageUtil.getThriftLanguageOf(searchQuery.getUiLang());\n    }\n    langEnglishUIBoost = rankingOptions.getLangEnglishUIBoost();\n    langEnglishFacetBoost = rankingOptions.getLangEnglishFacetBoost();\n    langDefaultBoost = rankingOptions.getLangDefaultBoost();\n  }\n\n  @Override\n  protected void startSegment(EarlybirdIndexSegmentAtomicReader segmentReader) throws IOException {\n    reader = segmentReader;\n    features = new EarlybirdDocumentFeatures(reader);\n    if (antiGamingFilter != null) {\n      antiGamingFilter.startSegment(reader);\n    }\n  }\n\n  @Override\n  public void incrementCounts(Accumulator accumulator, int internalDocID) throws IOException {\n    FacetCountIterator.IncrementData data = accumulator.accessor.incrementData;\n    data.accumulators = accumulator.accumulators;\n    features.advance(internalDocID);\n\n    // Also keep track of the tweet language of tweet themselves.\n    data.languageId = (int) features.getFeatureValue(EarlybirdFieldConstant.LANGUAGE);\n\n    if (antigamingPenalty > 0\n        && antiGamingFilter != null\n        && !antiGamingFilter.accept(internalDocID)) {\n      data.weightedCountIncrement = 0;\n      data.penaltyIncrement = antigamingPenalty;\n      data.tweepCred = 0;\n      accumulator.accessor.collect(internalDocID);\n      return;\n    }\n\n    if (offensiveTweetPenalty > 0 && features.isFlagSet(EarlybirdFieldConstant.IS_OFFENSIVE_FLAG)) {\n      data.weightedCountIncrement = 0;\n      data.penaltyIncrement = offensiveTweetPenalty;\n      data.tweepCred = 0;\n      accumulator.accessor.collect(internalDocID);\n      return;\n    }\n\n    byte userRep = (byte) features.getFeatureValue(EarlybirdFieldConstant.USER_REPUTATION);\n\n    if (userRep < reputationMinFilterThresholdVal) {\n      // don't penalize\n      data.weightedCountIncrement = 0;\n      data.penaltyIncrement = 0;\n      data.tweepCred = 0;\n      accumulator.accessor.collect(internalDocID);\n      return;\n    }\n\n    // Other non-terminating penalties\n    int penalty = 0;\n    if (multipleHashtagsOrTrendsPenalty > 0\n        && features.isFlagSet(EarlybirdFieldConstant.HAS_MULTIPLE_HASHTAGS_OR_TRENDS_FLAG)) {\n      penalty += multipleHashtagsOrTrendsPenalty;\n    }\n\n    double parus = 0xFF & (byte) features.getFeatureValue(EarlybirdFieldConstant.PARUS_SCORE);\n\n    double score = Math.pow(1 + userRepWeight, Math.max(0, userRep - reputationMinScoreVal));\n\n    if (parus > 0) {\n      score += Math.pow(parusBase, parus);\n    }\n\n    int favoriteCount =\n        (int) features.getUnnormalizedFeatureValue(EarlybirdFieldConstant.FAVORITE_COUNT);\n    if (favoriteCount > 0) {\n      score += favoriteCount * favoritesWeight;\n    }\n\n    // Language preferences\n    int tweetLinkLangId = (int) features.getFeatureValue(EarlybirdFieldConstant.LINK_LANGUAGE);\n    if (tweetLinkLangId == ThriftLanguage.UNKNOWN.getValue()) {\n      // fall back to use the tweet language itself.\n      tweetLinkLangId = (int) features.getFeatureValue(EarlybirdFieldConstant.LANGUAGE);\n    }\n    if (uiLang != ThriftLanguage.UNKNOWN && uiLang.getValue() != tweetLinkLangId) {\n      if (uiLang == ThriftLanguage.ENGLISH) {\n        score *= langEnglishUIBoost;\n      } else if (tweetLinkLangId == ThriftLanguage.ENGLISH.getValue()) {\n        score *= langEnglishFacetBoost;\n      } else {\n        score *= langDefaultBoost;\n      }\n    }\n\n    // make sure a single tweet can't contribute too high a score\n    if (score > maxScorePerTweet) {\n      score = maxScorePerTweet;\n    }\n\n    data.weightedCountIncrement = (int) score;\n    data.penaltyIncrement = penalty;\n    data.tweepCred = userRep & 0xFF;\n    accumulator.accessor.collect(internalDocID);\n  }\n\n  @Override\n  public FacetAccumulator getFacetAccumulator(FacetLabelProvider labelProvider) {\n    return new HashingAndPruningFacetAccumulator(labelProvider, queryIndependentPenaltyWeight,\n            HashingAndPruningFacetAccumulator.getComparator(sortingMode));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/EntityAnnotationCollector.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.util.List;\n\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Sets;\n\nimport org.apache.commons.lang.StringUtils;\n\nimport com.twitter.escherbird.thriftjava.TweetEntityAnnotation;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\n\npublic class EntityAnnotationCollector extends AbstractFacetTermCollector {\n  private List<TweetEntityAnnotation> annotations = Lists.newArrayList();\n\n  @Override\n  public boolean collect(int docID, long termID, int fieldID) {\n\n    String term = getTermFromFacet(termID, fieldID,\n        Sets.newHashSet(EarlybirdFieldConstant.ENTITY_ID_FIELD.getFieldName()));\n    if (StringUtils.isEmpty(term)) {\n      return false;\n    }\n\n    String[] idParts = term.split(\"\\\\.\");\n\n    // Only include the full three-part form of the entity ID: \"groupId.domainId.entityId\"\n    // Exclude the less-specific forms we index: \"domainId.entityId\" and \"entityId\"\n    if (idParts.length < 3) {\n      return false;\n    }\n\n    annotations.add(new TweetEntityAnnotation(\n        Long.valueOf(idParts[0]),\n        Long.valueOf(idParts[1]),\n        Long.valueOf(idParts[2])));\n\n    return true;\n  }\n\n  @Override\n  public void fillResultAndClear(ThriftSearchResult result) {\n    getExtraMetadata(result).setEntityAnnotations(ImmutableList.copyOf(annotations));\n    annotations.clear();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/ExpandedUrlCollector.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.util.LinkedHashMap;\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.ImmutableSet;\n\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultUrl;\nimport com.twitter.service.spiderduck.gen.MediaTypes;\n\n/**\n * A collector for collecting expanded urls from facets. Note that the only thing connecting this\n * collector with expanded URLs is the fact that we only store the expanded url in the facet fields.\n */\npublic class ExpandedUrlCollector extends AbstractFacetTermCollector {\n  private static final ImmutableSet<String> FACET_CONTAINS_URL = ImmutableSet.of(\n      EarlybirdFieldConstant.VIDEOS_FACET,\n      EarlybirdFieldConstant.IMAGES_FACET,\n      EarlybirdFieldConstant.NEWS_FACET,\n      EarlybirdFieldConstant.LINKS_FACET,\n      EarlybirdFieldConstant.TWIMG_FACET);\n\n  private final Map<String, ThriftSearchResultUrl> dedupedUrls = new LinkedHashMap<>();\n\n\n  @Override\n  protected String getTermFromProvider(\n      String facetName,\n      long termID,\n      FacetLabelProvider provider) {\n    String url = null;\n    if (EarlybirdFieldConstant.TWIMG_FACET.equals(facetName)) {\n      // Special case extraction of media url for twimg.\n      FacetLabelProvider.FacetLabelAccessor photoAccessor = provider.getLabelAccessor();\n      BytesRef termPayload = photoAccessor.getTermPayload(termID);\n      if (termPayload != null) {\n        url = termPayload.utf8ToString();\n      }\n    } else {\n      url = provider.getLabelAccessor().getTermText(termID);\n    }\n    return url;\n  }\n\n  @Override\n  public boolean collect(int docID, long termID, int fieldID) {\n\n    String url = getTermFromFacet(termID, fieldID, FACET_CONTAINS_URL);\n    if (url == null || url.isEmpty()) {\n      return false;\n    }\n\n    ThriftSearchResultUrl resultUrl = new ThriftSearchResultUrl();\n    resultUrl.setOriginalUrl(url);\n    MediaTypes mediaType = getMediaType(findFacetName(fieldID));\n    resultUrl.setMediaType(mediaType);\n\n    // Media links will show up twice:\n    //   - once in image/native_image/video/news facets\n    //   - another time in the links facet\n    //\n    // For those urls, we only want to return the media version. If it is non-media version, only\n    // write to map if doesn't exist already, if media version, overwrite any previous entries.\n    if (mediaType == MediaTypes.UNKNOWN) {\n      if (!dedupedUrls.containsKey(url)) {\n        dedupedUrls.put(url, resultUrl);\n      }\n    } else {\n      dedupedUrls.put(url, resultUrl);\n    }\n\n    return true;\n  }\n\n  @Override\n  public void fillResultAndClear(ThriftSearchResult result) {\n    result.getMetadata().setTweetUrls(getExpandedUrls());\n    dedupedUrls.clear();\n  }\n\n  @VisibleForTesting\n  List<ThriftSearchResultUrl> getExpandedUrls() {\n    return ImmutableList.copyOf(dedupedUrls.values());\n  }\n\n  /**\n   * Gets the Spiderduck media type for a given facet name.\n   *\n   * @param facetName A given facet name.\n   * @return {@code MediaTypes} enum corresponding to the facet name.\n   */\n  private static MediaTypes getMediaType(String facetName) {\n    if (facetName == null) {\n      return MediaTypes.UNKNOWN;\n    }\n\n    switch (facetName) {\n      case EarlybirdFieldConstant.TWIMG_FACET:\n        return MediaTypes.NATIVE_IMAGE;\n      case EarlybirdFieldConstant.IMAGES_FACET:\n        return MediaTypes.IMAGE;\n      case EarlybirdFieldConstant.VIDEOS_FACET:\n        return MediaTypes.VIDEO;\n      case EarlybirdFieldConstant.NEWS_FACET:\n        return MediaTypes.NEWS;\n      default:\n        return MediaTypes.UNKNOWN;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/ExplainFacetResultsCollector.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.facets.FacetIDMap;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.earlybird.search.AntiGamingFilter;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCount;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCountMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftFacetFieldResults;\nimport com.twitter.search.earlybird.thrift.ThriftFacetResults;\n\npublic class ExplainFacetResultsCollector extends FacetResultsCollector {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(ExplainFacetResultsCollector.class.getName());\n\n  protected final List<Pair<Integer, Long>> proofs;\n  protected final Map<String, Map<String, List<Long>>> proofAccumulators;\n\n  protected Map<String, FacetLabelProvider> facetLabelProviders;\n  private FacetIDMap facetIDMap;\n\n  /**\n   * Creates a new facet collector with the ability to provide explanations for the search results.\n   */\n  public ExplainFacetResultsCollector(\n      ImmutableSchemaInterface schema,\n      FacetSearchRequestInfo searchRequestInfo,\n      AntiGamingFilter antiGamingFilter,\n      EarlybirdSearcherStats searcherStats,\n      Clock clock,\n      int requestDebugMode) throws IOException {\n    super(schema, searchRequestInfo, antiGamingFilter, searcherStats, clock, requestDebugMode);\n\n    proofs = new ArrayList<>(128);\n\n    proofAccumulators = Maps.newHashMap();\n    for (Schema.FieldInfo facetField : schema.getFacetFields()) {\n      HashMap<String, List<Long>> fieldLabelToTweetIdsMap = new HashMap<>();\n      proofAccumulators.put(facetField.getFieldType().getFacetName(), fieldLabelToTweetIdsMap);\n    }\n  }\n\n  @Override\n  protected Accumulator newPerSegmentAccumulator(EarlybirdIndexSegmentAtomicReader indexReader) {\n    Accumulator accumulator = super.newPerSegmentAccumulator(indexReader);\n    accumulator.accessor.setProofs(proofs);\n    facetLabelProviders = indexReader.getFacetLabelProviders();\n    facetIDMap = indexReader.getFacetIDMap();\n\n    return accumulator;\n  }\n\n  @Override\n  public void doCollect(long tweetID) throws IOException {\n    proofs.clear();\n\n    // FacetResultsCollector.doCollect() calls FacetScorer.incrementCounts(),\n    // FacetResultsCollector.doCollect() creates a FacetResultsCollector.Accumulator, if\n    // necessary, which contains the accessor (a CompositeFacetIterator) and accumulators\n    // (FacetAccumulator of each field)\n    super.doCollect(tweetID);\n\n    for (Pair<Integer, Long> fieldIdTermIdPair : proofs) {\n      int fieldID = fieldIdTermIdPair.getFirst();\n      long termID = fieldIdTermIdPair.getSecond();\n\n      // Convert term ID to the term text, a.k.a. facet label\n      String facetName = facetIDMap.getFacetFieldByFacetID(fieldID).getFacetName();\n      if (facetName != null) {\n        String facetLabel = facetLabelProviders.get(facetName)\n                .getLabelAccessor().getTermText(termID);\n\n        List<Long> tweetIDs = proofAccumulators.get(facetName).get(facetLabel);\n        if (tweetIDs == null) {\n          tweetIDs = new ArrayList<>();\n          proofAccumulators.get(facetName).put(facetLabel, tweetIDs);\n        }\n\n        tweetIDs.add(tweetID);\n      }\n    }\n\n    // clear it again just to be sure\n    proofs.clear();\n  }\n\n  /**\n   * Sets explanations for the facet results.\n   */\n  public void setExplanations(ThriftFacetResults facetResults) {\n    StringBuilder explanation = new StringBuilder();\n\n    for (Map.Entry<String, ThriftFacetFieldResults> facetFieldResultsEntry\n            : facetResults.getFacetFields().entrySet()) {\n      String facetName = facetFieldResultsEntry.getKey();\n      ThriftFacetFieldResults facetFieldResults = facetFieldResultsEntry.getValue();\n\n      Map<String, List<Long>> proofAccumulator = proofAccumulators.get(facetName);\n\n      if (proofAccumulator == null) {\n        // did not accumulate explanation for this facet type? a bug?\n        LOG.warn(\"No explanation accumulated for facet type \" + facetName);\n        continue;\n      }\n\n      for (ThriftFacetCount facetCount : facetFieldResults.getTopFacets()) {\n        String facetLabel = facetCount.getFacetLabel(); // a.k.a. term text\n        ThriftFacetCountMetadata metadata = facetCount.getMetadata();\n\n        List<Long> tweetIDs = proofAccumulator.get(facetLabel);\n        if (tweetIDs == null) {\n          // did not accumulate explanation for this facet label? a bug?\n          LOG.warn(\"No explanation accumulated for \" + facetLabel + \" of facet type \" + facetName);\n          continue;\n        }\n\n        explanation.setLength(0);\n        String oldExplanation = null;\n        if (metadata.isSetExplanation()) {\n          // save the old explanation from TwitterInMemoryIndexSearcher.fillTermMetadata()\n          oldExplanation = metadata.getExplanation();\n          // as of 2012/05/29, we have 18 digits tweet IDs\n          explanation.ensureCapacity(oldExplanation.length() + (18 + 2) + 10);\n        } else {\n          // as of 2012/05/29, we have 18 digits tweet IDs\n          explanation.ensureCapacity(tweetIDs.size() * (18 + 2) + 10);\n        }\n\n        explanation.append(\"[\");\n        for (Long tweetID : tweetIDs) {\n          explanation.append(tweetID)\n                  .append(\", \");\n        }\n        explanation.setLength(explanation.length() - 2); // remove the last \", \"\n        explanation.append(\"]\\n\");\n        if (oldExplanation != null) {\n          explanation.append(oldExplanation);\n        }\n        metadata.setExplanation(explanation.toString());\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/FacetLabelCollector.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\n\nimport com.twitter.search.core.earlybird.facets.FacetIDMap;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.facets.FacetTermCollector;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.earlybird.thrift.ThriftFacetLabel;\n\n/**\n * A collector for facet labels of given fields.\n */\npublic class FacetLabelCollector implements FacetTermCollector {\n\n  private final Set<String> requiredFields;\n  private FacetIDMap facetIDMap;\n  private Map<String, FacetLabelProvider> facetLabelProviders;\n\n  private final List<ThriftFacetLabel> labels = new ArrayList<>();\n\n  public FacetLabelCollector(Set<String> requiredFields) {\n    this.requiredFields = requiredFields;\n  }\n\n  public void resetFacetLabelProviders(Map<String, FacetLabelProvider> facetLabelProvidersToReset,\n                                       FacetIDMap facetIDMapToReset) {\n    this.facetLabelProviders = facetLabelProvidersToReset;\n    this.facetIDMap = facetIDMapToReset;\n    labels.clear();\n  }\n\n  @Override\n  public boolean collect(int docID, long termID, int fieldID) {\n    String facetName = facetIDMap.getFacetFieldByFacetID(fieldID).getFacetName();\n    if (facetName == null || !requiredFields.contains(facetName)) {\n      return false;\n    }\n    if (termID != EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND && fieldID >= 0) {\n      final FacetLabelProvider provider = facetLabelProviders.get(facetName);\n      if (provider != null) {\n        FacetLabelProvider.FacetLabelAccessor labelAccessor = provider.getLabelAccessor();\n        String label = labelAccessor.getTermText(termID);\n        int offensiveCount = labelAccessor.getOffensiveCount(termID);\n        labels.add(new ThriftFacetLabel()\n            .setFieldName(facetName)\n            .setLabel(label)\n            .setOffensiveCount(offensiveCount));\n        return true;\n      }\n    }\n    return false;\n  }\n\n  public List<ThriftFacetLabel> getLabels() {\n    // Make a copy\n    return new ArrayList<>(labels);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/FacetRankingModule.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.util.ArrayList;\nimport java.util.List;\n\nimport com.twitter.search.core.earlybird.facets.FacetCountState;\nimport com.twitter.search.earlybird.search.EarlybirdLuceneSearcher;\nimport com.twitter.search.earlybird.thrift.ThriftFacetFieldResults;\n\npublic abstract class FacetRankingModule {\n  public static final List<FacetRankingModule> REGISTERED_RANKING_MODULES =\n      new ArrayList<>();\n\n  static {\n    REGISTERED_RANKING_MODULES.add(new SimpleCountRankingModule());\n  }\n\n  /**\n   * Prepares the {@link com.twitter.search.earlybird.thrift.ThriftFacetFieldResults}\n   * in {@link FacetCountState} before they're returned. This extension point therefore allows\n   * post-processing the facet results, e.g. for re-ranking or sorting purposes.\n   */\n  public abstract void prepareResults(\n      EarlybirdLuceneSearcher.FacetSearchResults hits,\n      FacetCountState<ThriftFacetFieldResults> facetCountState);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/FacetResultsCollector.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.PriorityQueue;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.ranking.thriftjava.ThriftFacetEarlybirdSortingMode;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.core.earlybird.facets.DummyFacetAccumulator;\nimport com.twitter.search.core.earlybird.facets.FacetAccumulator;\nimport com.twitter.search.core.earlybird.facets.FacetCountIterator;\nimport com.twitter.search.core.earlybird.facets.FacetIDMap;\nimport com.twitter.search.core.earlybird.facets.FacetIDMap.FacetField;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.facets.LanguageHistogram;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.earlybird.search.AbstractResultsCollector;\nimport com.twitter.search.earlybird.search.AntiGamingFilter;\nimport com.twitter.search.earlybird.search.EarlybirdLuceneSearcher.FacetSearchResults;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCount;\nimport com.twitter.search.earlybird.thrift.ThriftFacetFieldResults;\n\npublic class FacetResultsCollector extends\n    AbstractResultsCollector<FacetSearchRequestInfo, FacetSearchResults> {\n\n  private final FacetScorer facetScorer;\n  private final ThriftFacetEarlybirdSortingMode sortingMode;\n\n  static class Accumulator {\n    protected final FacetAccumulator<ThriftFacetFieldResults>[] accumulators;\n    protected final FacetCountIterator accessor;\n    protected final FacetIDMap facetIDMap;\n\n    Accumulator(FacetAccumulator<ThriftFacetFieldResults>[] accumulators,\n                FacetCountIterator accessor,\n                FacetIDMap facetIDMap) {\n      this.accumulators = accumulators;\n      this.accessor = accessor;\n      this.facetIDMap = facetIDMap;\n    }\n\n    FacetAccumulator<ThriftFacetFieldResults> getFacetAccumulator(String facetName) {\n      FacetField facet = facetIDMap.getFacetFieldByFacetName(facetName);\n      return accumulators[facet.getFacetId()];\n    }\n  }\n\n  private Accumulator currentAccumulator;\n  private List<Accumulator> segAccumulators;\n  private final HashingAndPruningFacetAccumulator.FacetComparator facetComparator;\n\n  /**\n   * Creates a new FacetResultsCollector for the given facet search request.\n   */\n  public FacetResultsCollector(\n      ImmutableSchemaInterface schema,\n      FacetSearchRequestInfo searchRequestInfo,\n      AntiGamingFilter antiGamingFilter,\n      EarlybirdSearcherStats searcherStats,\n      Clock clock,\n      int requestDebugInfo) {\n    super(schema, searchRequestInfo, clock, searcherStats, requestDebugInfo);\n\n    if (searchRequestInfo.rankingOptions != null\n        && searchRequestInfo.rankingOptions.isSetSortingMode()) {\n      this.sortingMode = searchRequestInfo.rankingOptions.getSortingMode();\n    } else {\n      this.sortingMode = ThriftFacetEarlybirdSortingMode.SORT_BY_WEIGHTED_COUNT;\n    }\n\n    this.facetComparator = HashingAndPruningFacetAccumulator.getComparator(sortingMode);\n    this.facetScorer = createScorer(antiGamingFilter);\n    this.segAccumulators = new ArrayList<>();\n  }\n\n  @Override\n  public void startSegment() {\n    currentAccumulator = null;\n  }\n\n  @Override\n  public void doCollect(long tweetID) throws IOException {\n    if (currentAccumulator == null) {\n      // Lazily create accumulators.  Most segment / query / facet combinations have no hits.\n      currentAccumulator = newPerSegmentAccumulator(currTwitterReader);\n      segAccumulators.add(currentAccumulator);\n      facetScorer.startSegment(currTwitterReader);\n    }\n    facetScorer.incrementCounts(currentAccumulator, curDocId);\n  }\n\n  @Override\n  public FacetSearchResults doGetResults() {\n    return new FacetSearchResults(this);\n  }\n\n  /**\n   * Returns the top-k facet results for the requested facetName.\n   */\n  public ThriftFacetFieldResults getFacetResults(String facetName, int topK) {\n    int totalCount = 0;\n    final Map<String, ThriftFacetCount> map = new HashMap<>();\n\n    LanguageHistogram languageHistogram = new LanguageHistogram();\n\n    for (Accumulator segAccumulator : segAccumulators) {\n      FacetAccumulator<ThriftFacetFieldResults> accumulator =\n          segAccumulator.getFacetAccumulator(facetName);\n      Preconditions.checkNotNull(accumulator);\n\n      ThriftFacetFieldResults results = accumulator.getAllFacets();\n      if (results == null) {\n        continue;\n      }\n\n      totalCount += results.totalCount;\n\n      // merge language histograms from different segments\n      languageHistogram.addAll(accumulator.getLanguageHistogram());\n\n      for (ThriftFacetCount facetCount : results.getTopFacets()) {\n        String label = facetCount.getFacetLabel();\n        ThriftFacetCount oldCount = map.get(label);\n        if (oldCount != null) {\n          oldCount.setSimpleCount(oldCount.getSimpleCount() + facetCount.getSimpleCount());\n          oldCount.setWeightedCount(oldCount.getWeightedCount() + facetCount.getWeightedCount());\n\n          oldCount.setFacetCount(oldCount.getFacetCount() + facetCount.getFacetCount());\n          oldCount.setPenaltyCount(oldCount.getPenaltyCount() + facetCount.getPenaltyCount());\n        } else {\n          map.put(label, facetCount);\n        }\n      }\n    }\n\n    if (map.size() == 0 || totalCount == 0) {\n      // No results.\n      return null;\n    }\n\n    // sort table wrt percentage\n    PriorityQueue<ThriftFacetCount> pq =\n        new PriorityQueue<>(map.size(), facetComparator.getThriftComparator(true));\n    pq.addAll(map.values());\n\n    ThriftFacetFieldResults results = new ThriftFacetFieldResults();\n    results.setTopFacets(new ArrayList<>());\n    results.setTotalCount(totalCount);\n\n    // Store merged language histogram into thrift object\n    for (Map.Entry<ThriftLanguage, Integer> entry\n        : languageHistogram.getLanguageHistogramAsMap().entrySet()) {\n      results.putToLanguageHistogram(entry.getKey(), entry.getValue());\n    }\n\n    // Get top facets.\n    for (int i = 0; i < topK && i < map.size(); i++) {\n      ThriftFacetCount facetCount = pq.poll();\n      if (facetCount != null) {\n        results.addToTopFacets(facetCount);\n      }\n    }\n    return results;\n  }\n\n  protected FacetScorer createScorer(AntiGamingFilter antiGamingFilter) {\n    if (searchRequestInfo.rankingOptions != null) {\n      return new DefaultFacetScorer(searchRequestInfo.getSearchQuery(),\n                                    searchRequestInfo.rankingOptions,\n                                    antiGamingFilter,\n                                    sortingMode);\n    } else {\n      return new FacetScorer() {\n        @Override\n        protected void startSegment(EarlybirdIndexSegmentAtomicReader reader) {\n        }\n\n        @Override\n        public void incrementCounts(Accumulator accumulator, int internalDocID) throws IOException {\n          accumulator.accessor.incrementData.accumulators = accumulator.accumulators;\n          accumulator.accessor.incrementData.weightedCountIncrement = 1;\n          accumulator.accessor.incrementData.penaltyIncrement = 0;\n          accumulator.accessor.incrementData.languageId = ThriftLanguage.UNKNOWN.getValue();\n          accumulator.accessor.collect(internalDocID);\n        }\n\n        @Override\n        public FacetAccumulator getFacetAccumulator(FacetLabelProvider labelProvider) {\n          return new HashingAndPruningFacetAccumulator(labelProvider, facetComparator);\n        }\n      };\n    }\n  }\n\n  protected Accumulator newPerSegmentAccumulator(EarlybirdIndexSegmentAtomicReader indexReader) {\n    final FacetIDMap facetIDMap = indexReader.getFacetIDMap();\n    final FacetCountIterator accessor =\n        indexReader.getFacetCountingArray().getIterator(\n            indexReader,\n            getSearchRequestInfo().getFacetCountState(),\n            TweetSearchFacetCountIteratorFactory.FACTORY);\n\n    final FacetAccumulator<ThriftFacetFieldResults>[] accumulators =\n        (FacetAccumulator<ThriftFacetFieldResults>[])\n            new FacetAccumulator[facetIDMap.getNumberOfFacetFields()];\n\n    Map<String, FacetLabelProvider> labelProviders = indexReader.getFacetLabelProviders();\n    for (FacetField f : facetIDMap.getFacetFields()) {\n      int id = f.getFacetId();\n      if (getSearchRequestInfo().getFacetCountState().isCountField(f.getFieldInfo())) {\n        accumulators[id] = (FacetAccumulator<ThriftFacetFieldResults>) facetScorer\n                .getFacetAccumulator(labelProviders.get(f.getFacetName()));\n      } else {\n        // Dummmy accumulator does nothing.\n        accumulators[id] = new DummyFacetAccumulator();\n      }\n    }\n\n    return new Accumulator(accumulators, accessor, facetIDMap);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/FacetScorer.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.io.IOException;\n\nimport com.twitter.search.core.earlybird.facets.FacetAccumulator;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.earlybird.search.facets.FacetResultsCollector.Accumulator;\n\npublic abstract class FacetScorer {\n  protected abstract void startSegment(EarlybirdIndexSegmentAtomicReader reader) throws IOException;\n\n  /**\n   * Increments facet counts for the given document.\n   */\n  public abstract void incrementCounts(Accumulator accumulator, int internalDocID)\n      throws IOException;\n\n  /**\n   * Returns a FacetAccumulator for counting facets. It will use the given FacetLabelProvider\n   * for facet result labeling.\n   */\n  public abstract FacetAccumulator<?> getFacetAccumulator(FacetLabelProvider labelProvider);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/FacetSearchRequestInfo.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport org.apache.lucene.search.Query;\n\nimport com.twitter.search.common.ranking.thriftjava.ThriftFacetRankingOptions;\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.core.earlybird.facets.FacetCountState;\nimport com.twitter.search.earlybird.search.SearchRequestInfo;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\n\npublic class FacetSearchRequestInfo extends SearchRequestInfo {\n  protected final FacetCountState facetCountState;\n  protected final ThriftFacetRankingOptions rankingOptions;\n\n  public FacetSearchRequestInfo(ThriftSearchQuery searchQuery,\n                                ThriftFacetRankingOptions rankingOptions,\n                                Query query,\n                                FacetCountState facetCountState,\n                                TerminationTracker terminationTracker) {\n    super(searchQuery, query, terminationTracker);\n    this.facetCountState = facetCountState;\n    this.rankingOptions = rankingOptions;\n  }\n\n  public final FacetCountState getFacetCountState() {\n    return this.facetCountState;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/HashingAndPruningFacetAccumulator.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.util.Arrays;\nimport java.util.Comparator;\nimport java.util.PriorityQueue;\n\nimport com.twitter.search.common.ranking.thriftjava.ThriftFacetEarlybirdSortingMode;\nimport com.twitter.search.core.earlybird.facets.FacetAccumulator;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider;\nimport com.twitter.search.core.earlybird.facets.FacetLabelProvider.FacetLabelAccessor;\nimport com.twitter.search.core.earlybird.facets.LanguageHistogram;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCount;\nimport com.twitter.search.earlybird.thrift.ThriftFacetCountMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftFacetFieldResults;\n\npublic class HashingAndPruningFacetAccumulator extends FacetAccumulator {\n  private static final int DEFAULT_HASH_SIZE = 4096;\n  /**\n   * 4 longs per entry accommodates long termIDs.\n   * Although entries could be encoded in 3 bytes, 4 ensures that no entry is split\n   * across cache lines.\n   */\n  protected static final int LONGS_PER_ENTRY = 4;\n  private static final double LOAD_FACTOR = 0.5;\n  private static final long BITSHIFT_MAX_TWEEPCRED = 32;\n  private static final long PENALTY_COUNT_MASK = (1L << BITSHIFT_MAX_TWEEPCRED) - 1;\n\n  protected static final long UNASSIGNED = -1;\n\n  protected LanguageHistogram languageHistogram = new LanguageHistogram();\n\n  protected static final class HashTable {\n    protected final long[] hash;\n    protected final int size;\n    protected final int maxLoad;\n    protected final int mask;\n\n    public HashTable(int size) {\n      hash = new long[LONGS_PER_ENTRY * size];\n      Arrays.fill(hash, UNASSIGNED);\n      this.size = size;\n      // Ensure alignment to LONGS_PER_ENTRY-byte boundaries\n      this.mask = LONGS_PER_ENTRY * (size - 1);\n      this.maxLoad = (int) (size * LOAD_FACTOR);\n    }\n\n    protected void reset() {\n      Arrays.fill(hash, UNASSIGNED);\n    }\n\n    private final Cursor cursor = new Cursor();\n\n    public int findHashPosition(long termID) {\n      int code = (new Long(termID)).hashCode();\n      int hashPos = code & mask;\n\n      if (cursor.readFromHash(hashPos) && (cursor.termID != termID)) {\n        final int inc = ((code >> 8) + code) | 1;\n        do {\n          code += inc;\n          hashPos = code & this.mask;\n        } while (cursor.readFromHash(hashPos) && (cursor.termID != termID));\n      }\n\n      return hashPos;\n    }\n\n    /**\n     * The cursor can be used to access the different fields of a hash entry.\n     * Callers should always position the cursor with readFromHash() before\n     * accessing the members.\n     */\n    private final class Cursor {\n      private int simpleCount;\n      private int weightedCount;\n      private int penaltyCount;\n      private int maxTweepcred;\n      private long termID;\n\n      public void writeToHash(int position) {\n        long payload = (((long) maxTweepcred) << BITSHIFT_MAX_TWEEPCRED)\n                       | ((long) penaltyCount);\n\n        assert itemPenaltyCount(payload) == penaltyCount : payload + \", \"\n                      + itemPenaltyCount(payload) + \" != \" + penaltyCount;\n        assert itemMaxTweepCred(payload) == maxTweepcred;\n\n        hash[position] = termID;\n        hash[position + 1] = simpleCount;\n        hash[position + 2] = weightedCount;\n        hash[position + 3] = payload;\n      }\n\n      /** Returns the item ID, or UNASSIGNED */\n      public boolean readFromHash(int position) {\n        long entry = hash[position];\n        if (entry == UNASSIGNED) {\n          termID = UNASSIGNED;\n          return false;\n        }\n\n        termID = entry;\n\n        simpleCount = (int) hash[position + 1];\n        weightedCount = (int) hash[position + 2];\n        long payload = hash[position + 3];\n\n        penaltyCount = itemPenaltyCount(payload);\n        maxTweepcred = itemMaxTweepCred(payload);\n\n        return true;\n      }\n    }\n  }\n\n  protected static int itemPenaltyCount(long payload) {\n    return (int) (payload & PENALTY_COUNT_MASK);\n  }\n\n  protected static int itemMaxTweepCred(long payload) {\n    return (int) (payload >>> BITSHIFT_MAX_TWEEPCRED);\n  }\n\n  protected int numItems;\n  protected final HashTable hashTable;\n  protected final long[] sortBuffer;\n  private FacetLabelProvider facetLabelProvider;\n\n  private int totalSimpleCount;\n  private int totalWeightedCount;\n  private int totalPenalty;\n\n  static final double DEFAULT_QUERY_INDEPENDENT_PENALTY_WEIGHT = 1.0;\n  private final double queryIndependentPenaltyWeight;\n\n  private final FacetComparator facetComparator;\n\n  public HashingAndPruningFacetAccumulator(FacetLabelProvider facetLabelProvider,\n          FacetComparator comparator) {\n    this(DEFAULT_HASH_SIZE, facetLabelProvider,\n            DEFAULT_QUERY_INDEPENDENT_PENALTY_WEIGHT, comparator);\n  }\n\n  public HashingAndPruningFacetAccumulator(FacetLabelProvider facetLabelProvider,\n          double queryIndependentPenaltyWeight, FacetComparator comparator) {\n    this(DEFAULT_HASH_SIZE, facetLabelProvider, queryIndependentPenaltyWeight, comparator);\n  }\n\n  /**\n   * Creates a new, empty HashingAndPruningFacetAccumulator with the given initial size.\n   * HashSize will be rounded up to the next power-of-2 value.\n   */\n  public HashingAndPruningFacetAccumulator(int hashSize, FacetLabelProvider facetLabelProvider,\n          double queryIndependentPenaltyWeight, FacetComparator comparator) {\n    int powerOfTwoSize = 2;\n    while (hashSize > powerOfTwoSize) {\n      powerOfTwoSize *= 2;\n    }\n\n    this.facetComparator  = comparator;\n    hashTable = new HashTable(powerOfTwoSize);\n    sortBuffer = new long[LONGS_PER_ENTRY * (int) Math.ceil(LOAD_FACTOR * powerOfTwoSize)];\n    this.facetLabelProvider = facetLabelProvider;\n    this.queryIndependentPenaltyWeight = queryIndependentPenaltyWeight;\n  }\n\n  @Override\n  public void reset(FacetLabelProvider facetLabelProviderToReset) {\n    this.facetLabelProvider = facetLabelProviderToReset;\n    this.numItems = 0;\n    this.hashTable.reset();\n    this.totalSimpleCount = 0;\n    this.totalPenalty = 0;\n    this.totalWeightedCount = 0;\n    languageHistogram.clear();\n  }\n\n\n  @Override\n  public int add(long termID, int weightedCounterIncrement, int penaltyIncrement, int tweepCred) {\n    int hashPos = hashTable.findHashPosition(termID);\n\n    totalPenalty += penaltyIncrement;\n    totalSimpleCount++;\n    totalWeightedCount += weightedCounterIncrement;\n\n    if (hashTable.cursor.termID == UNASSIGNED) {\n      hashTable.cursor.termID = termID;\n      hashTable.cursor.simpleCount = 1;\n      hashTable.cursor.weightedCount = weightedCounterIncrement;\n      hashTable.cursor.penaltyCount = penaltyIncrement;\n      hashTable.cursor.maxTweepcred = tweepCred;\n      hashTable.cursor.writeToHash(hashPos);\n\n      numItems++;\n      if (numItems >= hashTable.maxLoad) {\n        prune();\n      }\n      return 1;\n    } else {\n\n      hashTable.cursor.simpleCount++;\n      hashTable.cursor.weightedCount += weightedCounterIncrement;\n\n      if (tweepCred > hashTable.cursor.maxTweepcred) {\n        hashTable.cursor.maxTweepcred = tweepCred;\n      }\n\n      hashTable.cursor.penaltyCount += penaltyIncrement;\n      hashTable.cursor.writeToHash(hashPos);\n      return hashTable.cursor.simpleCount;\n    }\n  }\n\n  @Override\n  public void recordLanguage(int languageId) {\n    languageHistogram.increment(languageId);\n  }\n\n  @Override\n  public LanguageHistogram getLanguageHistogram() {\n    return languageHistogram;\n  }\n\n  private void prune() {\n    copyToSortBuffer();\n    hashTable.reset();\n\n    int targetNumItems = (int) (hashTable.maxLoad >> 1);\n\n    int minCount = 2;\n    int nextMinCount = Integer.MAX_VALUE;\n\n    final int n = LONGS_PER_ENTRY * numItems;\n\n    while (numItems > targetNumItems) {\n      for (int i = 0; i < n; i += LONGS_PER_ENTRY) {\n        long item = sortBuffer[i];\n        if (item != UNASSIGNED) {\n          int count = (int) sortBuffer[i + 1];\n          if (count < minCount) {\n            evict(i);\n          } else if (count < nextMinCount) {\n            nextMinCount = count;\n          }\n        }\n      }\n      if (minCount == nextMinCount) {\n        minCount++;\n      } else {\n        minCount = nextMinCount;\n      }\n      nextMinCount = Integer.MAX_VALUE;\n    }\n\n    // rehash\n    for (int i = 0; i < n; i += LONGS_PER_ENTRY) {\n      long item = sortBuffer[i];\n      if (item != UNASSIGNED) {\n        final long termID = item;\n        int hashPos = hashTable.findHashPosition(termID);\n        for (int j = 0; j < LONGS_PER_ENTRY; ++j) {\n          hashTable.hash[hashPos + j] = sortBuffer[i + j];\n        }\n      }\n    }\n  }\n\n  // overridable for unit test\n  protected void evict(int index) {\n    sortBuffer[index] = UNASSIGNED;\n    numItems--;\n  }\n\n  @Override\n  public ThriftFacetFieldResults getAllFacets() {\n    return getTopFacets(numItems);\n  }\n\n  @Override\n  public ThriftFacetFieldResults getTopFacets(final int numRequested) {\n    int n = numRequested > numItems ? numItems : numRequested;\n\n    if (n == 0) {\n      return null;\n    }\n\n    ThriftFacetFieldResults facetResults = new ThriftFacetFieldResults();\n    facetResults.setTotalCount(totalSimpleCount);\n    facetResults.setTotalScore(totalWeightedCount);\n    facetResults.setTotalPenalty(totalPenalty);\n\n    copyToSortBuffer();\n\n    // sort table using the facet comparator\n    PriorityQueue<Item> pq = new PriorityQueue<>(numItems, facetComparator.getComparator(true));\n\n    for (int i = 0; i < LONGS_PER_ENTRY * numItems; i += LONGS_PER_ENTRY) {\n      pq.add(new Item(sortBuffer, i));\n    }\n\n    FacetLabelAccessor accessor = facetLabelProvider.getLabelAccessor();\n\n    for (int i = 0; i < n; i++) {\n      Item item = pq.poll();\n      long id = item.getTermId();\n\n      int penalty = item.getPenaltyCount() + (int) (queryIndependentPenaltyWeight\n              * accessor.getOffensiveCount(id));\n      ThriftFacetCount result = new ThriftFacetCount().setFacetLabel(accessor.getTermText(id));\n      result.setPenaltyCount(penalty);\n      result.setSimpleCount(item.getSimpleCount());\n      result.setWeightedCount(item.getWeightedCount());\n      result.setMetadata(new ThriftFacetCountMetadata().setMaxTweepCred(item.getMaxTweetCred()));\n\n      result.setFacetCount(result.getWeightedCount());\n      facetResults.addToTopFacets(result);\n    }\n\n    return facetResults;\n  }\n\n  // Compacts the hashtable entries in place by removing empty hashes.  After\n  // this operation it's no longer a hash table but a array of entries.\n  private void copyToSortBuffer() {\n    int upto = 0;\n\n    for (int i = 0; i < hashTable.hash.length; i += LONGS_PER_ENTRY) {\n      if (hashTable.hash[i] != UNASSIGNED) {\n        for (int j = 0; j < LONGS_PER_ENTRY; ++j) {\n          sortBuffer[upto + j] = hashTable.hash[i + j];\n        }\n        upto += LONGS_PER_ENTRY;\n      }\n    }\n    assert upto == numItems * LONGS_PER_ENTRY;\n  }\n\n  /**\n   * Sorts facets in the following order:\n   * 1) ascending by weightedCount\n   * 2) if weightedCount equal: ascending by simpleCount\n   * 3) if weightedCount and simpleCount equal: descending by penaltyCount\n   */\n  public static int compareFacetCounts(int weightedCount1, int simpleCount1, int penaltyCount1,\n                                       int weightedCount2, int simpleCount2, int penaltyCount2,\n                                       boolean simpleCountPrecedence) {\n    if (simpleCountPrecedence) {\n      if (simpleCount1 < simpleCount2) {\n        return -1;\n      } else if (simpleCount1 > simpleCount2) {\n        return 1;\n      } else {\n        if (weightedCount1 < weightedCount2) {\n          return -1;\n        } else if (weightedCount1 > weightedCount2) {\n          return 1;\n        } else {\n          if (penaltyCount1 < penaltyCount2) {\n            // descending\n            return 1;\n          } else if (penaltyCount1 > penaltyCount2) {\n            return -1;\n          } else {\n            return 0;\n          }\n        }\n      }\n    } else {\n      if (weightedCount1 < weightedCount2) {\n        return -1;\n      } else if (weightedCount1 > weightedCount2) {\n        return 1;\n      } else {\n        if (simpleCount1 < simpleCount2) {\n          return -1;\n        } else if (simpleCount1 > simpleCount2) {\n          return 1;\n        } else {\n          if (penaltyCount1 < penaltyCount2) {\n            // descending\n            return 1;\n          } else if (penaltyCount1 > penaltyCount2) {\n            return -1;\n          } else {\n            return 0;\n          }\n        }\n      }\n    }\n  }\n\n  public static final class FacetComparator {\n    private final Comparator<ThriftFacetCount> thriftComparator;\n    private final Comparator<Item> comparator;\n\n    private FacetComparator(Comparator<ThriftFacetCount> thriftComparator,\n                            Comparator<Item> comparator) {\n      this.thriftComparator = thriftComparator;\n      this.comparator = comparator;\n    }\n\n    public Comparator<ThriftFacetCount> getThriftComparator() {\n      return getThriftComparator(false);\n    }\n\n    public Comparator<ThriftFacetCount> getThriftComparator(boolean reverse) {\n      return reverse ? getReverseComparator(thriftComparator) : thriftComparator;\n    }\n\n    private Comparator<Item> getComparator(boolean reverse) {\n      return reverse ? getReverseComparator(comparator) : comparator;\n    }\n  }\n\n  public static final FacetComparator SIMPLE_COUNT_COMPARATOR = new FacetComparator(\n      (facet1, facet2) -> compareFacetCounts(\n          facet1.weightedCount, facet1.simpleCount, facet1.penaltyCount,\n          facet2.weightedCount, facet2.simpleCount, facet2.penaltyCount,\n          true),\n      (facet1, facet2) -> compareFacetCounts(\n          facet1.getWeightedCount(), facet1.getSimpleCount(), facet1.getPenaltyCount(),\n          facet2.getWeightedCount(), facet2.getSimpleCount(), facet2.getPenaltyCount(),\n          true));\n\n\n  public static final FacetComparator WEIGHTED_COUNT_COMPARATOR = new FacetComparator(\n      (facet1, facet2) -> compareFacetCounts(\n          facet1.weightedCount, facet1.simpleCount, facet1.penaltyCount,\n          facet2.weightedCount, facet2.simpleCount, facet2.penaltyCount,\n          false),\n      (facet1, facet2) -> compareFacetCounts(\n          facet1.getWeightedCount(), facet1.getSimpleCount(), facet1.getPenaltyCount(),\n          facet2.getWeightedCount(), facet2.getSimpleCount(), facet2.getPenaltyCount(),\n          false));\n\n  /**\n   * Returns the appropriate FacetComparator for the specified sortingMode.\n   */\n  public static FacetComparator getComparator(ThriftFacetEarlybirdSortingMode sortingMode) {\n    switch (sortingMode) {\n      case SORT_BY_WEIGHTED_COUNT:\n        return WEIGHTED_COUNT_COMPARATOR;\n      case SORT_BY_SIMPLE_COUNT:\n      default:\n        return SIMPLE_COUNT_COMPARATOR;\n    }\n  }\n\n  private static <T> Comparator<T> getReverseComparator(final Comparator<T> comparator) {\n    return (t1, t2) -> -comparator.compare(t1, t2);\n  }\n\n  static final class Item {\n    private final long[] data;\n    private final int offset;\n\n    Item(long[] data, int offset) {\n      this.data = data;\n      this.offset = offset;\n    }\n\n    public long getTermId() {\n      return data[offset];\n    }\n\n    public int getSimpleCount() {\n      return (int) data[offset + 1];\n    }\n\n    public int getWeightedCount() {\n      return (int) data[offset + 2];\n    }\n\n    public int getPenaltyCount() {\n      return itemPenaltyCount(data[offset + 3]);\n    }\n\n    public int getMaxTweetCred() {\n      return itemMaxTweepCred(data[offset + 3]);\n    }\n\n    @Override public int hashCode() {\n      return (int) (31 * getTermId());\n    }\n\n    @Override public boolean equals(Object o) {\n      return getTermId() == ((Item) o).getTermId();\n    }\n\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/NamedEntityCollector.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Lists;\n\nimport org.apache.commons.lang.StringUtils;\n\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.earlybird.thrift.NamedEntitySource;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultNamedEntity;\n\npublic class NamedEntityCollector extends AbstractFacetTermCollector {\n  private static final Map<String, NamedEntitySource> NAMED_ENTITY_WITH_TYPE_FIELDS =\n      ImmutableMap.of(\n          EarlybirdFieldConstant.NAMED_ENTITY_WITH_TYPE_FROM_TEXT_FIELD.getFieldName(),\n          NamedEntitySource.TEXT,\n          EarlybirdFieldConstant.NAMED_ENTITY_WITH_TYPE_FROM_URL_FIELD.getFieldName(),\n          NamedEntitySource.URL);\n\n  private List<ThriftSearchResultNamedEntity> namedEntities = Lists.newArrayList();\n\n  @Override\n  public boolean collect(int docID, long termID, int fieldID) {\n\n    String term = getTermFromFacet(termID, fieldID, NAMED_ENTITY_WITH_TYPE_FIELDS.keySet());\n    if (StringUtils.isEmpty(term)) {\n      return false;\n    }\n\n    int index = term.lastIndexOf(\":\");\n    namedEntities.add(new ThriftSearchResultNamedEntity(\n        term.substring(0, index),\n        term.substring(index + 1),\n        NAMED_ENTITY_WITH_TYPE_FIELDS.get(findFacetName(fieldID))));\n\n    return true;\n  }\n\n  @Override\n  public void fillResultAndClear(ThriftSearchResult result) {\n    getExtraMetadata(result).setNamedEntities(ImmutableList.copyOf(namedEntities));\n    namedEntities.clear();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/RetweetFacetCountIterator.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.NumericDocValues;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.facets.CSFFacetCountIterator;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\n/**\n * And iterator for counting retweets. Reads from shared_status_id CSF but doesn't count\n * replies.\n */\npublic class RetweetFacetCountIterator extends CSFFacetCountIterator {\n  private final NumericDocValues featureReaderIsRetweetFlag;\n\n  public RetweetFacetCountIterator(\n      EarlybirdIndexSegmentAtomicReader reader,\n      Schema.FieldInfo facetFieldInfo) throws IOException {\n    super(reader, facetFieldInfo);\n    featureReaderIsRetweetFlag =\n        reader.getNumericDocValues(EarlybirdFieldConstant.IS_RETWEET_FLAG.getFieldName());\n  }\n\n  @Override\n  protected boolean shouldCollect(int internalDocID, long termID) throws IOException {\n    // termID == 0 means that we didn't set shared_status_csf, so don't collect\n    // (tweet IDs are all positive)\n    // Also only collect if this doc is a retweet, not a reply\n    return termID > 0\n        && featureReaderIsRetweetFlag.advanceExact(internalDocID)\n        && (featureReaderIsRetweetFlag.longValue() != 0);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/SimpleCountRankingModule.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.util.Iterator;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.facets.FacetCountState;\nimport com.twitter.search.core.earlybird.facets.FacetCountState.FacetFieldResults;\nimport com.twitter.search.earlybird.search.EarlybirdLuceneSearcher;\nimport com.twitter.search.earlybird.thrift.ThriftFacetFieldResults;\n\npublic class SimpleCountRankingModule extends FacetRankingModule {\n\n  @Override\n  public void prepareResults(\n      EarlybirdLuceneSearcher.FacetSearchResults hits,\n      FacetCountState<ThriftFacetFieldResults> facetCountState) {\n    Iterator<FacetFieldResults<ThriftFacetFieldResults>> fieldResultsIterator =\n            facetCountState.getFacetFieldResultsIterator();\n    while (fieldResultsIterator.hasNext()) {\n      FacetFieldResults<ThriftFacetFieldResults> state = fieldResultsIterator.next();\n      if (!state.isFinished()) {\n        Schema.FieldInfo facetField =\n                facetCountState.getSchema().getFacetFieldByFacetName(state.facetName);\n        state.results = hits.getFacetResults(\n                facetField.getFieldType().getFacetName(), state.numResultsRequested);\n        if (state.results != null) {\n          state.numResultsFound = state.results.getTopFacetsSize();\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/SpaceFacetCollector.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.util.ArrayList;\nimport java.util.List;\n\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.Sets;\n\nimport org.apache.commons.lang.StringUtils;\n\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.earlybird.partition.AudioSpaceTable;\nimport com.twitter.search.earlybird.thrift.AudioSpaceState;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultAudioSpace;\n\npublic class SpaceFacetCollector extends AbstractFacetTermCollector {\n  private final List<ThriftSearchResultAudioSpace> spaces = new ArrayList<>();\n\n  private final AudioSpaceTable audioSpaceTable;\n\n  public SpaceFacetCollector(AudioSpaceTable audioSpaceTable) {\n    this.audioSpaceTable = audioSpaceTable;\n  }\n\n  @Override\n  public boolean collect(int docID, long termID, int fieldID) {\n\n    String spaceId = getTermFromFacet(termID, fieldID,\n        Sets.newHashSet(EarlybirdFieldConstant.SPACES_FACET));\n    if (StringUtils.isEmpty(spaceId)) {\n      return false;\n    }\n\n    spaces.add(new ThriftSearchResultAudioSpace(spaceId,\n        audioSpaceTable.isRunning(spaceId) ? AudioSpaceState.RUNNING\n            : AudioSpaceState.ENDED));\n\n    return true;\n  }\n\n  @Override\n  public void fillResultAndClear(ThriftSearchResult result) {\n    getExtraMetadata(result).setSpaces(ImmutableList.copyOf(spaces));\n    spaces.clear();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/TermStatisticsCollector.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.apache.commons.lang.StringUtils;\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.search.DocIdSetIterator;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchResultsStats;\nimport com.twitter.search.common.schema.SchemaUtil;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.search.EarlyTerminationState;\nimport com.twitter.search.common.util.earlybird.TermStatisticsUtil;\nimport com.twitter.search.core.earlybird.index.TimeMapper;\nimport com.twitter.search.earlybird.index.EarlybirdSingleSegmentSearcher;\nimport com.twitter.search.earlybird.search.AbstractResultsCollector;\nimport com.twitter.search.earlybird.search.SearchResultsInfo;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.ThriftHistogramSettings;\nimport com.twitter.search.earlybird.thrift.ThriftTermRequest;\nimport com.twitter.search.earlybird.thrift.ThriftTermResults;\n\npublic class TermStatisticsCollector extends AbstractResultsCollector\n        <TermStatisticsRequestInfo, TermStatisticsCollector.TermStatisticsSearchResults> {\n  private static final EarlyTerminationState TERMINATED_TERM_STATS_COUNTING_DONE =\n      new EarlyTerminationState(\"terminated_term_stats_counting_done\", true);\n\n  // Stats for tracking histogram results.\n  private static final SearchResultsStats TERM_STATS_HISTOGRAM_REQUESTS_WITH_MOVED_BACK_BINS =\n      SearchResultsStats.export(\"term_statistics_collector_queries_with_moved_back_bins\");\n  private static final SearchCounter TERM_STATS_SKIPPED_LARGER_OUT_OF_BOUNDS_HITS =\n      SearchCounter.export(\"term_statistics_collector_skipped_larger_out_of_bounds_hits\");\n\n  @VisibleForTesting\n  static final class TermStatistics {\n    private final ThriftTermRequest termRequest;\n    private final Term term;  // could be null, for count across all fields\n    private int termDF = 0;\n    private int termCount = 0;\n    private final int[] histogramBins;\n\n    // Per-segment information.\n    private PostingsEnum segmentDocsEnum;  // could be null, for count across all fields\n    private boolean segmentDone;\n\n    @VisibleForTesting\n    TermStatistics(ThriftTermRequest termRequest, Term term, int numBins) {\n      this.termRequest = termRequest;\n      this.term = term;\n      this.histogramBins = new int[numBins];\n    }\n\n    /**\n     * Take the currently accumulated counts and \"move them back\" to make room for counts from more\n     * recent binIds.\n     *\n     * For example, if the oldFirstBinID was set to 10, and the histogramBins were {3, 4, 5, 6, 7},\n     * after this call with newFirstBinID set to 12, the histogramBins will be set\n     * to {5, 6, 7, 0, 0}.\n     *\n     * @param oldFirstBinID the binId of the firstBin that's been used up to now.\n     * @param newFirstBinID the new binId of the firstBin that will be used from now on.\n     *     The newFirstBinID is presumed to be larger than the oldFirstBinID, and is asserted.\n     */\n    @VisibleForTesting\n    void moveBackTermCounts(int oldFirstBinID, int newFirstBinID) {\n      Preconditions.checkState(oldFirstBinID < newFirstBinID);\n      // move counts back by this many bins\n      final int moveBackBy = newFirstBinID - oldFirstBinID;\n\n      this.termCount = 0;\n      for (int i = 0; i < histogramBins.length; i++) {\n        int oldCount = histogramBins[i];\n        histogramBins[i] = 0;\n        int newIndex = i - moveBackBy;\n        if (newIndex >= 0) {\n          histogramBins[newIndex] = oldCount;\n          this.termCount += oldCount;\n        }\n      }\n    }\n\n    @VisibleForTesting void countHit(int bin) {\n      termCount++;\n      histogramBins[bin]++;\n    }\n\n    @VisibleForTesting int getTermCount() {\n      return termCount;\n    }\n\n    @VisibleForTesting int[] getHistogramBins() {\n      return histogramBins;\n    }\n  }\n\n  private TermStatistics[] termStatistics;\n\n  // Histogram fields.\n  private int numBins;\n  private int binSize;\n\n  private int numTimesBinsWereMovedBack = 0;\n  private int numLargerOutOfBoundsBinsSkipped = 0;\n\n  private static final int SEEN_OUT_OF_RANGE_THRESHOLD = 10;\n\n  private int seenOutOfRange = 0;\n\n  // ID of the first bin - effectively time / binSize.  This is calculated\n  // relative to the first collected in-order hit.\n  private int firstBinID = -1;\n  // List of per-segment debug information specifically useful for termstat request debugging.\n  private List<String> termStatisticsDebugInfo = new ArrayList<>();\n\n  /**\n   * Creates a new term stats collector.\n   */\n  public TermStatisticsCollector(\n      ImmutableSchemaInterface schema,\n      TermStatisticsRequestInfo searchRequestInfo,\n      EarlybirdSearcherStats searcherStats,\n      Clock clock,\n      int requestDebugMode) {\n    super(schema, searchRequestInfo, clock, searcherStats, requestDebugMode);\n\n    // Set up the histogram bins.\n    if (searchRequestInfo.isReturnHistogram()) {\n      ThriftHistogramSettings histogramSettings = searchRequestInfo.getHistogramSettings();\n      this.numBins = histogramSettings.getNumBins();\n      binSize = TermStatisticsUtil.determineBinSize(histogramSettings);\n    } else {\n      this.numBins = 0;\n      this.binSize = 0;\n    }\n\n    // Set up the term statistics array.\n    List<ThriftTermRequest> termRequests = searchRequestInfo.getTermRequests();\n    if (termRequests == null) {\n      this.termStatistics = new TermStatistics[0];\n      return;\n    }\n\n    this.termStatistics = new TermStatistics[searchRequestInfo.getTermRequests().size()];\n    for (int i = 0; i < searchRequestInfo.getTermRequests().size(); i++) {\n      final ThriftTermRequest termRequest = searchRequestInfo.getTermRequests().get(i);\n\n      Term term = null;\n      String fieldName = termRequest.getFieldName();\n      if (!StringUtils.isBlank(fieldName)) {\n        // First check if it's a facet field.\n        Schema.FieldInfo facetField = schema.getFacetFieldByFacetName(termRequest.getFieldName());\n        if (facetField != null) {\n          term = new Term(facetField.getName(), termRequest.getTerm());\n        } else {\n          // EarlybirdSearcher.validateRequest() should've already checked that the field exists in\n          // the schema, and that the term can be converted to the type of this field. However, if\n          // that did not happen for some reason, an exception will be thrown here, which will be\n          // converted to a TRANSIENT_ERROR response code.\n          Schema.FieldInfo fieldInfo = schema.getFieldInfo(fieldName);\n          Preconditions.checkNotNull(\n              fieldInfo,\n              \"Found a ThriftTermRequest for a field that's not in the schema: \" + fieldName\n              + \". This should've been caught by EarlybirdSearcher.validateRequest()!\");\n          term = new Term(fieldName, SchemaUtil.toBytesRef(fieldInfo, termRequest.getTerm()));\n        }\n      } else {\n        // NOTE: if the fieldName is empty, this is a catch-all term request for the count across\n        // all fields. We'll just use a null term in the TermStatistics object.\n      }\n\n      termStatistics[i] = new TermStatistics(termRequest, term, numBins);\n    }\n  }\n\n  @Override\n  public void startSegment() throws IOException {\n    termStatisticsDebugInfo.add(\n        \"Starting segment in timestamp range: [\" + timeMapper.getFirstTime()\n        + \", \" + timeMapper.getLastTime() + \"]\");\n    for (TermStatistics termStats : termStatistics) {\n      termStats.segmentDone = true;  // until we know it's false later.\n      TermsEnum termsEnum = null;\n      if (termStats.term != null) {\n        Terms terms = currTwitterReader.terms(termStats.term.field());\n        if (terms != null) {\n          termsEnum = terms.iterator();\n          if (termsEnum != null && termsEnum.seekExact(termStats.term.bytes())) {\n            termStats.termDF += termsEnum.docFreq();  // Only meaningful for matchAll queries.\n            termStats.segmentDocsEnum =\n                termsEnum.postings(termStats.segmentDocsEnum, PostingsEnum.FREQS);\n            termStats.segmentDone = termStats.segmentDocsEnum == null\n                 || termStats.segmentDocsEnum.nextDoc() == DocIdSetIterator.NO_MORE_DOCS;\n          } else {\n            // this term doesn't exist in this segment.\n          }\n        }\n      } else {\n        // Catch-all case\n        termStats.termDF += currTwitterReader.numDocs();   // Only meaningful for matchAll queries.\n        termStats.segmentDocsEnum = null;\n        termStats.segmentDone = false;\n      }\n    }\n  }\n\n  private int calculateBin(final int tweetTime) {\n    if (tweetTime == TimeMapper.ILLEGAL_TIME) {\n      return -1;\n    }\n\n    final int binID = Math.abs(tweetTime) / binSize;\n    final int expectedFirstBinId = binID - numBins + 1;\n\n    if (firstBinID == -1) {\n      firstBinID = expectedFirstBinId;\n    } else if (expectedFirstBinId > firstBinID) {\n      numTimesBinsWereMovedBack++;\n      final int oldOutOfOrderFirstBinID = firstBinID;\n      firstBinID = expectedFirstBinId;\n      // We got a more recent out of order bin, move previous counts back.\n      for (TermStatistics ts : termStatistics) {\n        ts.moveBackTermCounts(oldOutOfOrderFirstBinID, firstBinID);\n      }\n    }\n\n    final int binIndex = binID - firstBinID;\n    if (binIndex >= numBins) {\n      // In-order times should be decreasing,\n      // and out of order times seen after an in-order tweet should also be smaller than the\n      // first in-order tweet's time. Will track these and export as a stat.\n      numLargerOutOfBoundsBinsSkipped++;\n      return -1;\n    } else if (binIndex < 0) {\n      // Early termination criteria.\n      seenOutOfRange++;\n    } else {\n      // Reset the counter, since we want to see consecutive tweets that are out of our bin range\n      // not single anomalies.\n      seenOutOfRange = 0;\n    }\n\n    return binIndex;\n  }\n\n  @Override\n  public void doCollect(long tweetID) throws IOException {\n    if (searchRequestInfo.isReturnHistogram()) {\n      final int tweetTime = timeMapper.getTime(curDocId);\n      final int binIndex = calculateBin(tweetTime);\n      if (binIndex >= 0) {\n        for (TermStatistics ts : termStatistics) {\n          if (!ts.segmentDone) {\n            countHist(ts, binIndex);\n          }\n        }\n      }\n    } else {\n      for (TermStatistics ts : termStatistics) {\n        if (!ts.segmentDone) {\n          countNoHist(ts);\n        }\n      }\n    }\n  }\n\n  @Override\n  public void skipSegment(EarlybirdSingleSegmentSearcher searcher) {\n    // Do nothing here.\n    // We don't do accounting that's done in AbstractResultsCollector for Term Stats\n    // requests because otherwise the bin ID calculation will be confused.\n  }\n\n  private boolean advance(TermStatistics ts) throws IOException {\n    PostingsEnum docsEnum = ts.segmentDocsEnum;\n    if (docsEnum.docID() < curDocId) {\n      if (docsEnum.advance(curDocId) == DocIdSetIterator.NO_MORE_DOCS) {\n        ts.segmentDone = true;\n        return false;\n      }\n    }\n    return docsEnum.docID() == curDocId;\n  }\n\n  private boolean countHist(TermStatistics ts, int bin) throws IOException {\n    if (ts.term != null && !advance(ts)) {\n      return false;\n    }\n    ts.countHit(bin);\n    return true;\n  }\n\n  private boolean countNoHist(TermStatistics ts) throws IOException {\n    if (ts.term != null && !advance(ts)) {\n      return false;\n    }\n    ts.termCount++;\n    return true;\n  }\n\n  @Override\n  public EarlyTerminationState innerShouldCollectMore() {\n    if (readyToTerminate()) {\n      return setEarlyTerminationState(TERMINATED_TERM_STATS_COUNTING_DONE);\n    }\n    return EarlyTerminationState.COLLECTING;\n  }\n\n  /**\n   * The termination logic is simple - we know what our earliest bin is and once we see a result\n   * that's before our earliest bin, we terminate.\n   *\n   * Our results come with increasing internal doc ids, which should correspond to decreasing\n   * timestamps. See SEARCH-27729, TWEETYPIE-7031.\n   *\n   * We early terminate after we have seen enough tweets that are outside of the bin\n   * range that we want to return. This way we're not terminating too early because of single tweets\n   * with wrong timestamps.\n   */\n  @VisibleForTesting\n  boolean readyToTerminate() {\n    return this.seenOutOfRange >= SEEN_OUT_OF_RANGE_THRESHOLD;\n  }\n\n  @Override\n  public TermStatisticsSearchResults doGetResults() {\n    return new TermStatisticsSearchResults();\n  }\n\n  public final class TermStatisticsSearchResults extends SearchResultsInfo {\n    public final List<Integer> binIds;\n    public final Map<ThriftTermRequest, ThriftTermResults> results;\n    public final int lastCompleteBinId;\n    public final List<String>  termStatisticsDebugInfo;\n\n    private TermStatisticsSearchResults() {\n      // Initialize term stat debug info\n      termStatisticsDebugInfo = TermStatisticsCollector.this.termStatisticsDebugInfo;\n\n      if (termStatistics.length > 0) {\n        results = new HashMap<>();\n\n        if (searchRequestInfo.isReturnHistogram()) {\n          binIds = new ArrayList<>(numBins);\n          int minSearchedTime = TermStatisticsCollector.this.getMinSearchedTime();\n\n          if (shouldCollectDetailedDebugInfo()) {\n            termStatisticsDebugInfo.add(\"minSearchedTime: \" + minSearchedTime);\n            int maxSearchedTime = TermStatisticsCollector.this.getMaxSearchedTime();\n            termStatisticsDebugInfo.add(\"maxSearchedTime: \" + maxSearchedTime);\n          }\n\n          int lastCompleteBin = -1;\n\n          computeFirstBinId(TermStatisticsCollector.this.isSetMinSearchedTime(), minSearchedTime);\n          trackHistogramResultStats();\n\n          // Example:\n          //  minSearchTime = 53s\n          //  binSize = 10\n          //  firstBinId = 5\n          //  numBins = 4\n          //  binId = 5, 6, 7, 8\n          //  binTimeStamp = 50s, 60s, 70s, 80s\n          for (int i = 0; i < numBins; i++) {\n            int binId = firstBinID + i;\n            int binTimeStamp = binId * binSize;\n            binIds.add(binId);\n            if (lastCompleteBin == -1 && binTimeStamp > minSearchedTime) {\n              lastCompleteBin = binId;\n            }\n          }\n\n          if (!getEarlyTerminationState().isTerminated()) {\n            // only if we didn't early terminate we can be sure to use the firstBinID as\n            // lastCompleteBinId\n            lastCompleteBinId = firstBinID;\n            if (shouldCollectDetailedDebugInfo()) {\n              termStatisticsDebugInfo.add(\"no early termination\");\n            }\n          } else {\n            lastCompleteBinId = lastCompleteBin;\n            if (shouldCollectDetailedDebugInfo()) {\n              termStatisticsDebugInfo.add(\n                  \"early terminated for reason: \" + getEarlyTerminationReason());\n            }\n          }\n          if (shouldCollectDetailedDebugInfo()) {\n            termStatisticsDebugInfo.add(\"lastCompleteBinId: \" + lastCompleteBinId);\n          }\n        } else {\n          binIds = null;\n          lastCompleteBinId = -1;\n        }\n\n        for (TermStatistics ts : termStatistics) {\n          ThriftTermResults termResults = new ThriftTermResults().setTotalCount(ts.termCount);\n\n          if (searchRequestInfo.isReturnHistogram()) {\n            List<Integer> list = new ArrayList<>();\n            for (int count : ts.histogramBins) {\n              list.add(count);\n            }\n            termResults.setHistogramBins(list);\n          }\n\n          results.put(ts.termRequest, termResults);\n        }\n      } else {\n        binIds = null;\n        results = null;\n        lastCompleteBinId = -1;\n      }\n    }\n\n    @Override\n    public String toString() {\n      StringBuilder res = new StringBuilder();\n      res.append(\"TermStatisticsSearchResults(\\n\");\n      if (binIds != null) {\n        res.append(\"  binIds=\").append(binIds).append(\"\\n\");\n      }\n      res.append(\"  lastCompleteBinId=\").append(lastCompleteBinId).append(\"\\n\");\n      if (results != null) {\n        res.append(\"  results=\").append(results).append(\"\\n\");\n      }\n      res.append(\")\");\n      return res.toString();\n    }\n\n    public List<String> getTermStatisticsDebugInfo() {\n      return termStatisticsDebugInfo;\n    }\n  }\n\n  /**\n   * Figure out what the actual firstBinId is for this query.\n   */\n  private void computeFirstBinId(boolean isSetMinSearchedTime, int minSearchedTime) {\n    if (firstBinID == -1) {\n      if (!isSetMinSearchedTime) {\n        // This would only happen if we don't search any segments, which for now we have\n        // only seen happening if since_time or until_time don't intersect at all with\n        // the range of the served segments.\n        firstBinID = 0;\n      } else {\n        // Example:\n        //    minSearchedTime = 54\n        //    binSize = 10\n        //    firstBinId = 5\n        firstBinID = minSearchedTime / binSize;\n      }\n\n      if (shouldCollectDetailedDebugInfo()) {\n        termStatisticsDebugInfo.add(\"firstBinId: \" + firstBinID);\n      }\n    }\n  }\n\n  @VisibleForTesting\n  int getSeenOutOfRange() {\n    return seenOutOfRange;\n  }\n\n  private void trackHistogramResultStats() {\n    if (numLargerOutOfBoundsBinsSkipped > 0) {\n      TERM_STATS_SKIPPED_LARGER_OUT_OF_BOUNDS_HITS.increment();\n    }\n\n    if (numTimesBinsWereMovedBack > 0) {\n      TERM_STATS_HISTOGRAM_REQUESTS_WITH_MOVED_BACK_BINS.recordResults(numTimesBinsWereMovedBack);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/TermStatisticsRequestInfo.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.util.LinkedList;\nimport java.util.List;\nimport java.util.Set;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableSet;\n\nimport org.apache.lucene.search.Query;\n\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.common.util.text.NormalizerHelper;\nimport com.twitter.search.common.util.url.URLUtils;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.search.SearchRequestInfo;\nimport com.twitter.search.earlybird.thrift.ThriftHistogramSettings;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftTermRequest;\nimport com.twitter.search.earlybird.thrift.ThriftTermStatisticsRequest;\n\npublic class TermStatisticsRequestInfo extends SearchRequestInfo {\n  private static final Set<String> FACET_URL_FIELDS_TO_NORMALIZE = new ImmutableSet.Builder()\n      .add(EarlybirdFieldConstant.IMAGES_FACET)\n      .add(EarlybirdFieldConstant.VIDEOS_FACET)\n      .add(EarlybirdFieldConstant.NEWS_FACET)\n      .build();\n\n  protected final List<ThriftTermRequest> termRequests;\n  protected final ThriftHistogramSettings histogramSettings;\n\n  /**\n   * Creates a new TermStatisticsRequestInfo instance using the provided query.\n   */\n  public TermStatisticsRequestInfo(ThriftSearchQuery searchQuery,\n                                   Query luceneQuery,\n                                   ThriftTermStatisticsRequest termStatsRequest,\n                                   TerminationTracker terminationTracker) {\n    super(searchQuery, luceneQuery, terminationTracker);\n    this.termRequests = termStatsRequest.isSetTermRequests()\n                        ? termStatsRequest.getTermRequests() : new LinkedList<>();\n    this.histogramSettings = termStatsRequest.getHistogramSettings();\n    if (termStatsRequest.isIncludeGlobalCounts()) {\n      // Add an empty request to indicate we need a global count across all fields.\n      termRequests.add(new ThriftTermRequest().setFieldName(\"\").setTerm(\"\"));\n    }\n\n    // We only normalize TEXT terms and urls. All other terms, e.g. topics (named entities) are\n    // not normalized. Here the assumption is that the caller passes the exact terms back that\n    // the facet API returned\n    for (ThriftTermRequest termReq : termRequests) {\n      if (termReq.getTerm().isEmpty()) {\n        continue;  // the special catch-all term.\n      }\n\n      if (!termReq.isSetFieldName()\n          || termReq.getFieldName().equals(EarlybirdFieldConstant.TEXT_FIELD.getFieldName())) {\n        // normalize the TEXT term as it's normalized during ingestion\n        termReq.setTerm(NormalizerHelper.normalizeWithUnknownLocale(\n                            termReq.getTerm(), EarlybirdConfig.getPenguinVersion()));\n      } else if (FACET_URL_FIELDS_TO_NORMALIZE.contains(termReq.getFieldName())) {\n        // remove the trailing slash from the URL path. This operation is idempotent,\n        // so either a spiderduck URL or a facet URL can be used here. The latter would just\n        // be normalized twice, which is fine.\n        termReq.setTerm(URLUtils.normalizePath(termReq.getTerm()));\n      }\n    }\n  }\n\n  @Override\n  protected int calculateMaxHitsToProcess(ThriftSearchQuery searchQuery) {\n    Preconditions.checkNotNull(searchQuery.getCollectorParams());\n    if (!searchQuery.getCollectorParams().isSetTerminationParams()\n        || !searchQuery.getCollectorParams().getTerminationParams().isSetMaxHitsToProcess()) {\n      // Override the default value to all hits.\n      return Integer.MAX_VALUE;\n    } else {\n      return super.calculateMaxHitsToProcess(searchQuery);\n    }\n  }\n\n  public final List<ThriftTermRequest> getTermRequests() {\n    return this.termRequests;\n  }\n\n  public final ThriftHistogramSettings getHistogramSettings() {\n    return this.histogramSettings;\n  }\n\n  public final boolean isReturnHistogram() {\n    return this.histogramSettings != null;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/facets/TweetSearchFacetCountIteratorFactory.java",
    "content": "package com.twitter.search.earlybird.search.facets;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.facets.CSFFacetCountIterator;\nimport com.twitter.search.core.earlybird.facets.FacetCountIterator;\nimport com.twitter.search.core.earlybird.facets.FacetCountIteratorFactory;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\n\n/**\n * Factory of {@link FacetCountIterator} instances for tweet search.\n * It provides a special iterator for the retweets facet.\n */\npublic final class TweetSearchFacetCountIteratorFactory extends FacetCountIteratorFactory {\n  public static final TweetSearchFacetCountIteratorFactory FACTORY =\n      new TweetSearchFacetCountIteratorFactory();\n\n  private TweetSearchFacetCountIteratorFactory() {\n  }\n\n  @Override\n  public FacetCountIterator getFacetCountIterator(\n      EarlybirdIndexSegmentAtomicReader reader,\n      Schema.FieldInfo fieldInfo) throws IOException {\n    Preconditions.checkNotNull(reader);\n    Preconditions.checkNotNull(fieldInfo);\n    Preconditions.checkArgument(fieldInfo.getFieldType().isUseCSFForFacetCounting());\n\n    String facetName = fieldInfo.getFieldType().getFacetName();\n\n    if (EarlybirdFieldConstant.RETWEETS_FACET.equals(facetName)) {\n      return new RetweetFacetCountIterator(reader, fieldInfo);\n    } else {\n      return new CSFFacetCountIterator(reader, fieldInfo);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/BadUserRepFilter.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.query.DefaultFilterWeight;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.util.AllDocsIterator;\nimport com.twitter.search.core.earlybird.index.util.RangeFilterDISI;\n\npublic final class BadUserRepFilter extends Query {\n  /**\n   * Creates a query that filters out results coming from users with bad reputation.\n   *\n   * @param minTweepCred The lowest acceptable user reputation.\n   * @return A query that filters out results from bad reputation users.\n   */\n  public static Query getBadUserRepFilter(int minTweepCred) {\n    if (minTweepCred <= 0) {\n      return null;\n    }\n\n    return new BooleanQuery.Builder()\n        .add(new BadUserRepFilter(minTweepCred), BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  private final int minTweepCred;\n\n  private BadUserRepFilter(int minTweepCred) {\n    this.minTweepCred = minTweepCred;\n  }\n\n  @Override\n  public int hashCode() {\n    return minTweepCred;\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof BadUserRepFilter)) {\n      return false;\n    }\n\n    return minTweepCred == BadUserRepFilter.class.cast(obj).minTweepCred;\n  }\n\n  @Override\n  public String toString(String field) {\n    return \"BadUserRepFilter:\" + minTweepCred;\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n    return new DefaultFilterWeight(this) {\n      @Override\n      protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context) throws IOException {\n        LeafReader reader = context.reader();\n        if (!(reader instanceof EarlybirdIndexSegmentAtomicReader)) {\n          return new AllDocsIterator(reader);\n        }\n\n        return new BadUserExcludeDocIdSetIterator(\n            (EarlybirdIndexSegmentAtomicReader) context.reader(), minTweepCred);\n      }\n    };\n  }\n\n  private static final class BadUserExcludeDocIdSetIterator extends RangeFilterDISI {\n    private final NumericDocValues userReputationDocValues;\n    private final int minTweepCred;\n\n    BadUserExcludeDocIdSetIterator(EarlybirdIndexSegmentAtomicReader indexReader,\n                                   int minTweepCred) throws IOException {\n      super(indexReader);\n      this.userReputationDocValues =\n          indexReader.getNumericDocValues(EarlybirdFieldConstant.USER_REPUTATION.getFieldName());\n      this.minTweepCred = minTweepCred;\n    }\n\n    @Override\n    public boolean shouldReturnDoc() throws IOException {\n      // We need this explicit casting to byte, because of how we encode and decode features in our\n      // encoded_tweet_features field. If a feature is an int (uses all 32 bits of the int), then\n      // encoding the feature and then decoding it preserves its original value. However, if the\n      // feature does not use the entire int (and especially if it uses bits somewhere in the middle\n      // of the int), then the feature value is assumed to be unsigned when it goes through this\n      // process of encoding and decoding. So a user rep of\n      // RelevanceSignalConstants.UNSET_REPUTATION_SENTINEL (-128) will be correctly encoded as the\n      // binary value 10000000, but will be treated as an unsigned value when decoded, and therefore\n      // the decoded value will be 128.\n      //\n      // In retrospect, this seems like a really poor design decision. It seems like it would be\n      // better if all feature values were considered to be signed, even if most features can never\n      // have negative values. Unfortunately, making this change is not easy, because some features\n      // store normalized values, so we would also need to change the range of allowed values\n      // produced by those normalizers, as well as all code that depends on those values.\n      //\n      // So for now, just cast this value to a byte, to get the proper negative value.\n      return userReputationDocValues.advanceExact(docID())\n          && ((byte) userReputationDocValues.longValue() >= minTweepCred);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/CSFDisjunctionFilter.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\nimport java.util.Objects;\nimport java.util.Set;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.query.DefaultFilterWeight;\nimport com.twitter.search.core.earlybird.index.util.RangeFilterDISI;\n\n/**\n * CSFDisjunctionFilter provides an efficient mechanism to query for documents that have a\n * long CSF equal to one of the provided values.\n */\npublic final class CSFDisjunctionFilter extends Query {\n  private final String csfField;\n  private final Set<Long> values;\n\n  public static Query getCSFDisjunctionFilter(String csfField, Set<Long> values) {\n    return new BooleanQuery.Builder()\n        .add(new CSFDisjunctionFilter(csfField, values), BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  private CSFDisjunctionFilter(String csfField, Set<Long> values) {\n    this.csfField = csfField;\n    this.values = values;\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n    return new DefaultFilterWeight(this) {\n      @Override\n      protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context) throws IOException {\n        return new CSFDisjunctionFilterDISI(context.reader(), csfField, values);\n      }\n    };\n  }\n\n  @Override\n  public int hashCode() {\n    return (csfField == null ? 0 : csfField.hashCode()) * 17\n        + (values == null ? 0 : values.hashCode());\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof CSFDisjunctionFilter)) {\n      return false;\n    }\n\n    CSFDisjunctionFilter filter = CSFDisjunctionFilter.class.cast(obj);\n    return Objects.equals(csfField, filter.csfField) && Objects.equals(values, filter.values);\n  }\n\n  @Override\n  public String toString(String field) {\n    return \"CSFDisjunctionFilter:\" + csfField + \",count:\" + values.size();\n  }\n\n  private static final class CSFDisjunctionFilterDISI extends RangeFilterDISI {\n    private final NumericDocValues docValues;\n    private final Set<Long> values;\n\n    private CSFDisjunctionFilterDISI(LeafReader reader, String csfField, Set<Long> values)\n        throws IOException {\n      super(reader);\n      this.values = values;\n      this.docValues = reader.getNumericDocValues(csfField);\n    }\n\n    @Override\n    protected boolean shouldReturnDoc() throws IOException {\n      return docValues.advanceExact(docID()) && values.contains(docValues.longValue());\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/DocValRangeFilter.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\nimport java.util.Objects;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.query.DefaultFilterWeight;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFType;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.util.AllDocsIterator;\nimport com.twitter.search.core.earlybird.index.util.RangeFilterDISI;\n\n/**\n * Filters tweets according to the specified CSF field value.\n * Note that min value is inclusive, and max value is exclusive.\n */\npublic final class DocValRangeFilter extends Query {\n  private final String csfField;\n  private final ThriftCSFType csfFieldType;\n  private final Number minValInclusive;\n  private final Number maxValExclusive;\n\n  /**\n   * Returns a query that filters hits based on the value of a CSF.\n   *\n   * @param csfField The CSF name.\n   * @param csfFieldType The CSF type.\n   * @param minVal The minimum acceptable value (inclusive).\n   * @param maxVal The maximum acceptable value (exclusive).\n   * @return A query that filters hits based on the value of a CSF.\n   */\n  public static Query getDocValRangeQuery(String csfField, ThriftCSFType csfFieldType,\n                                          double minVal, double maxVal) {\n    return new BooleanQuery.Builder()\n        .add(new DocValRangeFilter(csfField, csfFieldType, minVal, maxVal),\n             BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  /**\n   * Returns a query that filters hits based on the value of a CSF.\n   *\n   * @param csfField The CSF name.\n   * @param csfFieldType The CSF type.\n   * @param minVal The minimum acceptable value (inclusive).\n   * @param maxVal The maximum acceptable value (exclusive).\n   * @return A query that filters hits based on the value of a CSF.\n   */\n  public static Query getDocValRangeQuery(String csfField, ThriftCSFType csfFieldType,\n                                          long minVal, long maxVal) {\n    return new BooleanQuery.Builder()\n        .add(new DocValRangeFilter(csfField, csfFieldType, minVal, maxVal),\n             BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  private DocValRangeFilter(String csfField, ThriftCSFType csfFieldType,\n                            double minVal, double maxVal) {\n    this.csfField = csfField;\n    this.csfFieldType = csfFieldType;\n    this.minValInclusive = new Float(minVal);\n    this.maxValExclusive = new Float(maxVal);\n  }\n\n  private DocValRangeFilter(String csfField, ThriftCSFType csfFieldType,\n                            long minVal, long maxVal) {\n    this.csfField = csfField;\n    this.csfFieldType = csfFieldType;\n    this.minValInclusive = new Long(minVal);\n    this.maxValExclusive = new Long(maxVal);\n  }\n\n  @Override\n  public int hashCode() {\n    return (csfField == null ? 0 : csfField.hashCode()) * 29\n        + (csfFieldType == null ? 0 : csfFieldType.hashCode()) * 17\n        + minValInclusive.hashCode() * 7\n        + maxValExclusive.hashCode();\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof DocValRangeFilter)) {\n      return false;\n    }\n\n    DocValRangeFilter filter = DocValRangeFilter.class.cast(obj);\n    return Objects.equals(csfField, filter.csfField)\n        && (csfFieldType == filter.csfFieldType)\n        && minValInclusive.equals(filter.minValInclusive)\n        && maxValExclusive.equals(filter.maxValExclusive);\n  }\n\n  @Override\n  public String toString(String field) {\n    return \"DocValRangeFilter:\" + csfField\n        + \",type:\" + csfFieldType.toString()\n        + \",min:\" + this.minValInclusive.toString()\n        + \",max:\" + this.maxValExclusive.toString();\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n    return new DefaultFilterWeight(this) {\n      @Override\n      protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context) throws IOException {\n        LeafReader reader = context.reader();\n        if (csfFieldType == null) {\n          return new AllDocsIterator(reader);\n        }\n\n        int smallestDoc = (reader instanceof EarlybirdIndexSegmentAtomicReader)\n            ? ((EarlybirdIndexSegmentAtomicReader) reader).getSmallestDocID() : 0;\n        int largestDoc = reader.maxDoc() - 1;\n        return new CSFRangeDocIdSetIterator(reader, csfField, csfFieldType,\n                                            smallestDoc, largestDoc,\n                                            minValInclusive, maxValExclusive);\n      }\n    };\n  }\n\n  private static final class CSFRangeDocIdSetIterator extends RangeFilterDISI {\n    private final NumericDocValues numericDocValues;\n    private final ThriftCSFType csfType;\n    private final Number minValInclusive;\n    private final Number maxValExclusive;\n\n    public CSFRangeDocIdSetIterator(LeafReader reader,\n                                    String csfField,\n                                    ThriftCSFType csfType,\n                                    int smallestDocID,\n                                    int largestDocID,\n                                    Number minValInclusive,\n                                    Number maxValExclusive) throws IOException {\n      super(reader, smallestDocID, largestDocID);\n      this.numericDocValues = reader.getNumericDocValues(csfField);\n      this.csfType = csfType;\n      this.minValInclusive = minValInclusive;\n      this.maxValExclusive = maxValExclusive;\n    }\n\n    @Override\n    protected boolean shouldReturnDoc() throws IOException {\n      if (!numericDocValues.advanceExact(docID())) {\n        return false;\n      }\n\n      long val = numericDocValues.longValue();\n      switch (csfType) {\n        case DOUBLE:\n          double doubleVal = Double.longBitsToDouble(val);\n          return doubleVal >= minValInclusive.doubleValue()\n              && doubleVal < maxValExclusive.doubleValue();\n        case FLOAT:\n          float floatVal = Float.intBitsToFloat((int) val);\n          return floatVal >= minValInclusive.doubleValue()\n              && floatVal < maxValExclusive.doubleValue();\n        case LONG:\n          return val >= minValInclusive.longValue() && val < maxValExclusive.longValue();\n        case INT:\n          return val >= minValInclusive.longValue() && (int) val < maxValExclusive.longValue();\n        case BYTE:\n          return (byte) val >= minValInclusive.longValue()\n              && (byte) val < maxValExclusive.longValue();\n        default:\n          return false;\n      }\n    }\n  }\n\n  //////////////////////////\n  // for unit tests only\n  //////////////////////////\n  @VisibleForTesting\n  public Number getMinValForTest() {\n    return minValInclusive;\n  }\n\n  @VisibleForTesting\n  public Number getMaxValForTest() {\n    return maxValExclusive;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/FeatureValueInAcceptListOrUnsetFilter.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\nimport java.util.Set;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.query.DefaultFilterWeight;\nimport com.twitter.search.core.earlybird.index.util.RangeFilterDISI;\n\npublic final class FeatureValueInAcceptListOrUnsetFilter extends Query {\n\n  private final String featureName;\n  private final Set<Long> idsAcceptList;\n\n  /**\n   * Creates a query that filters for hits that have the given feature unset, or that have the\n   * given feature set to a value in the given list of IDs.\n   *\n   * @param featureName The feature.\n   * @param ids A list of id values this filter will accept for the given feature.\n   * @return A query that filters out all hits that have the given feature set.\n   */\n  public static Query getFeatureValueInAcceptListOrUnsetFilter(String featureName, Set<Long> ids) {\n    return new BooleanQuery.Builder()\n        .add(new FeatureValueInAcceptListOrUnsetFilter(featureName, ids),\n            BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  @Override\n  public String toString(String s) {\n    return String.format(\"FeatureValueInAcceptListOrUnsetFilter(%s, AcceptList = (%s))\",\n        featureName,\n        idsAcceptList);\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof FeatureValueInAcceptListOrUnsetFilter)) {\n      return false;\n    }\n\n    FeatureValueInAcceptListOrUnsetFilter filter =\n        FeatureValueInAcceptListOrUnsetFilter.class.cast(obj);\n    return featureName.equals(filter.featureName) && idsAcceptList.equals(filter.idsAcceptList);\n  }\n\n  @Override\n  public int hashCode() {\n    return featureName.hashCode() * 7 + idsAcceptList.hashCode();\n  }\n\n  private FeatureValueInAcceptListOrUnsetFilter(String featureName, Set<Long> ids) {\n    this.featureName = Preconditions.checkNotNull(featureName);\n    this.idsAcceptList = Preconditions.checkNotNull(ids);\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n    return new DefaultFilterWeight(this) {\n      @Override\n      protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context) throws IOException {\n        return new FeatureValueInAcceptListOrUnsetDocIdSetIterator(\n            context.reader(), featureName, idsAcceptList);\n      }\n    };\n  }\n\n  private static final class FeatureValueInAcceptListOrUnsetDocIdSetIterator\n      extends RangeFilterDISI {\n    private final NumericDocValues featureDocValues;\n    private final Set<Long> idsAcceptList;\n\n    FeatureValueInAcceptListOrUnsetDocIdSetIterator(\n        LeafReader indexReader, String featureName, Set<Long> ids) throws IOException {\n      super(indexReader);\n      this.featureDocValues = indexReader.getNumericDocValues(featureName);\n      this.idsAcceptList = ids;\n    }\n\n    @Override\n    public boolean shouldReturnDoc() throws IOException {\n      // If featureDocValues is null, that means there were no documents indexed with the given\n      // field in the current segment.\n      //\n      // The advanceExact() method returns false if it cannot find the given docId in the\n      // NumericDocValues instance. So if advanceExact() returns false then we know the feature is\n      // unset.\n      // However, for realtime Earlybirds we have a custom implementation of NumericDocValues,\n      // ColumnStrideFieldDocValues, which will contain an entry for every indexed docId and use a\n      // value of 0 to indicate that a feature is unset.\n      //\n      // So to check if a feature is unset for a given docId, we first need to check if we can find\n      // the docId, and then we additionally need to check if the feature value is 0.\n      return featureDocValues == null\n          || !featureDocValues.advanceExact(docID())\n          || featureDocValues.longValue() == 0\n          || idsAcceptList.contains(featureDocValues.longValue());\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/GeoTwoPhaseQuery.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\nimport java.util.Set;\n\nimport org.apache.lucene.index.IndexReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.ConstantScoreQuery;\nimport org.apache.lucene.search.ConstantScoreScorer;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.Explanation;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.TwoPhaseIterator;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\n\n\npublic class GeoTwoPhaseQuery extends Query {\n  private static final boolean ENABLE_GEO_EARLY_TERMINATION =\n          EarlybirdConfig.getBool(\"early_terminate_geo_searches\", true);\n\n  private static final int GEO_TIMEOUT_OVERRIDE =\n          EarlybirdConfig.getInt(\"early_terminate_geo_searches_timeout_override\", -1);\n\n  // How many geo searches are early terminated due to timeout.\n  private static final SearchCounter GEO_SEARCH_TIMEOUT_COUNT =\n      SearchCounter.export(\"geo_search_timeout_count\");\n\n  private final SecondPhaseDocAccepter accepter;\n  private final TerminationTracker terminationTracker;\n  private final ConstantScoreQuery query;\n\n  public GeoTwoPhaseQuery(\n      Query query, SecondPhaseDocAccepter accepter, TerminationTracker terminationTracker) {\n    this.accepter = accepter;\n    this.terminationTracker = terminationTracker;\n\n    this.query = new ConstantScoreQuery(query);\n  }\n\n  @Override\n  public Query rewrite(IndexReader reader) throws IOException {\n    Query rewritten = query.getQuery().rewrite(reader);\n    if (rewritten != query.getQuery()) {\n      return new GeoTwoPhaseQuery(rewritten, accepter, terminationTracker);\n    }\n\n    return this;\n  }\n\n  @Override\n  public int hashCode() {\n    return query.hashCode();\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof GeoTwoPhaseQuery)) {\n      return false;\n    }\n    GeoTwoPhaseQuery that = (GeoTwoPhaseQuery) obj;\n    return query.equals(that.query)\n        && accepter.equals(that.accepter)\n        && terminationTracker.equals(that.terminationTracker);\n  }\n\n  @Override\n  public String toString(String field) {\n    return new StringBuilder(\"GeoTwoPhaseQuery(\")\n      .append(\"Accepter(\")\n      .append(accepter.toString())\n      .append(\") Geohashes(\")\n      .append(query.getQuery().toString(field))\n      .append(\"))\")\n      .toString();\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost)\n      throws IOException {\n    Weight innerWeight = query.createWeight(searcher, scoreMode, boost);\n    return new GeoTwoPhaseWeight(this, innerWeight, accepter, terminationTracker);\n  }\n\n  private static final class GeoTwoPhaseWeight extends Weight {\n    private final Weight innerWeight;\n    private final SecondPhaseDocAccepter accepter;\n    private final TerminationTracker terminationTracker;\n\n    private GeoTwoPhaseWeight(\n        Query query,\n        Weight innerWeight,\n        SecondPhaseDocAccepter accepter,\n        TerminationTracker terminationTracker) {\n      super(query);\n      this.innerWeight = innerWeight;\n      this.accepter = accepter;\n      this.terminationTracker = terminationTracker;\n    }\n\n    @Override\n    public void extractTerms(Set<Term> terms) {\n      innerWeight.extractTerms(terms);\n    }\n\n    @Override\n    public Explanation explain(LeafReaderContext context, int doc) throws IOException {\n      return innerWeight.explain(context, doc);\n    }\n\n    @Override\n    public Scorer scorer(LeafReaderContext context) throws IOException {\n      Scorer innerScorer = innerWeight.scorer(context);\n      if (innerScorer == null) {\n        return null;\n      }\n      if (ENABLE_GEO_EARLY_TERMINATION\n          && (terminationTracker == null || !terminationTracker.useLastSearchedDocIdOnTimeout())) {\n        innerScorer = new ConstantScoreScorer(\n            this,\n            0.0f,\n            ScoreMode.COMPLETE_NO_SCORES,\n            new TimedDocIdSetIterator(innerScorer.iterator(),\n                                      terminationTracker,\n                                      GEO_TIMEOUT_OVERRIDE,\n                                      GEO_SEARCH_TIMEOUT_COUNT));\n      }\n\n      accepter.initialize(context);\n      return new GeoTwoPhaseScorer(this, innerScorer, accepter);\n    }\n\n    @Override\n    public boolean isCacheable(LeafReaderContext ctx) {\n      return innerWeight.isCacheable(ctx);\n    }\n  }\n\n  private static final class GeoTwoPhaseScorer extends Scorer {\n    private final Scorer innerScorer;\n    private final SecondPhaseDocAccepter accepter;\n\n    private GeoTwoPhaseScorer(Weight weight, Scorer innerScorer, SecondPhaseDocAccepter accepter) {\n      super(weight);\n      this.innerScorer = innerScorer;\n      this.accepter = accepter;\n    }\n\n    @Override\n    public TwoPhaseIterator twoPhaseIterator() {\n      return new TwoPhaseIterator(innerScorer.iterator()) {\n        @Override\n        public boolean matches() throws IOException {\n          return checkDocExpensive(innerScorer.docID());\n        }\n\n        @Override\n        public float matchCost() {\n          return 0.0f;\n        }\n      };\n    }\n\n    @Override\n    public int docID() {\n      return iterator().docID();\n    }\n\n    @Override\n    public float score() throws IOException {\n      return innerScorer.score();\n    }\n\n    @Override\n    public DocIdSetIterator iterator() {\n      return new DocIdSetIterator() {\n        private int doNext(int startingDocId) throws IOException {\n          int docId = startingDocId;\n          while ((docId != NO_MORE_DOCS) && !checkDocExpensive(docId)) {\n            docId = innerScorer.iterator().nextDoc();\n          }\n          return docId;\n        }\n\n        @Override\n        public int docID() {\n          return innerScorer.iterator().docID();\n        }\n\n        @Override\n        public int nextDoc() throws IOException {\n          return doNext(innerScorer.iterator().nextDoc());\n        }\n\n        @Override\n        public int advance(int target) throws IOException {\n          return doNext(innerScorer.iterator().advance(target));\n        }\n\n        @Override\n        public long cost() {\n          return 2 * innerScorer.iterator().cost();\n        }\n      };\n    }\n\n    @Override\n    public float getMaxScore(int upTo) throws IOException {\n      return innerScorer.getMaxScore(upTo);\n    }\n\n    private boolean checkDocExpensive(int doc) throws IOException {\n      return accepter.accept(doc);\n    }\n  }\n\n  public abstract static class SecondPhaseDocAccepter {\n    /**\n     * Initializes this accepter with the given reader context.\n     */\n    public abstract void initialize(LeafReaderContext context) throws IOException;\n\n    /**\n     * Determines if the given doc ID is accepted by this accepter.\n     */\n    public abstract boolean accept(int doc) throws IOException;\n\n    /**\n     * Returns a string description for this SecondPhaseDocAccepter instance.\n     */\n    public abstract String toString();\n  }\n\n  public static final SecondPhaseDocAccepter ALL_DOCS_ACCEPTER = new SecondPhaseDocAccepter() {\n    @Override\n    public void initialize(LeafReaderContext context) { }\n\n    @Override\n    public boolean accept(int doc) {\n      return true;\n    }\n\n    @Override\n    public String toString() {\n      return \"AllDocsAccepter\";\n    }\n  };\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/MatchAllDocIdSet.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.search.DocIdSet;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.util.Bits;\nimport org.apache.lucene.util.RamUsageEstimator;\n\nimport com.twitter.search.core.earlybird.index.util.AllDocsIterator;\n\npublic final class MatchAllDocIdSet extends DocIdSet {\n  private final LeafReader reader;\n\n  public MatchAllDocIdSet(LeafReader reader) {\n    this.reader = reader;\n  }\n\n  @Override\n  public DocIdSetIterator iterator() throws IOException {\n    return new AllDocsIterator(reader);\n  }\n\n  @Override\n  public Bits bits() throws IOException {\n    return new Bits() {\n      @Override\n      public boolean get(int index) {\n        return true;\n      }\n\n      @Override\n      public int length() {\n        return reader.maxDoc();\n      }\n    };\n  }\n\n  @Override\n  public long ramBytesUsed() {\n    return RamUsageEstimator.shallowSizeOf(this);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/MatchAllDocsQuery.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\nimport java.util.Set;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.ConstantScoreScorer;\nimport org.apache.lucene.search.Explanation;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.util.RangeFilterDISI;\nimport com.twitter.search.earlybird.index.EarlybirdSingleSegmentSearcher;\n\n/**\n * A MatchAllDocsQuery implementation that does not assume that doc IDs are assigned sequentially.\n * Instead, it wraps the EarlybirdIndexSegmentAtomicReader into a RangeFilterDISI, and uses\n * this iterator to traverse only the valid doc IDs in this segment.\n *\n * Note that org.apache.lucene.index.MatchAllDocsQuery is final, so we cannot extend it.\n */\npublic class MatchAllDocsQuery extends Query {\n  private static class MatchAllDocsWeight extends Weight {\n    private final Weight luceneWeight;\n\n    public MatchAllDocsWeight(Query query, Weight luceneWeight) {\n      super(query);\n      this.luceneWeight = luceneWeight;\n    }\n\n    @Override\n    public void extractTerms(Set<Term> terms) {\n      luceneWeight.extractTerms(terms);\n    }\n\n    @Override\n    public Explanation explain(LeafReaderContext context, int doc) throws IOException {\n      return luceneWeight.explain(context, doc);\n    }\n\n    @Override\n    public Scorer scorer(LeafReaderContext context) throws IOException {\n      Preconditions.checkState(context.reader() instanceof EarlybirdIndexSegmentAtomicReader,\n                               \"Expected an EarlybirdIndexSegmentAtomicReader, but got a \"\n                               + context.reader().getClass().getName() + \" instance.\");\n      EarlybirdIndexSegmentAtomicReader reader =\n          (EarlybirdIndexSegmentAtomicReader) context.reader();\n      return new ConstantScoreScorer(\n          this, 1.0f, ScoreMode.COMPLETE_NO_SCORES, new RangeFilterDISI(reader));\n    }\n\n    @Override\n    public boolean isCacheable(LeafReaderContext ctx) {\n      return luceneWeight.isCacheable(ctx);\n    }\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n    org.apache.lucene.search.MatchAllDocsQuery luceneMatchAllDocsQuery =\n        new org.apache.lucene.search.MatchAllDocsQuery();\n    Weight luceneWeight = luceneMatchAllDocsQuery.createWeight(searcher, scoreMode, boost);\n    if (!(searcher instanceof EarlybirdSingleSegmentSearcher)) {\n      return luceneWeight;\n    }\n    return new MatchAllDocsWeight(this, luceneWeight);\n  }\n\n  @Override\n  public int hashCode() {\n    return 0;\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    return obj instanceof MatchAllDocsQuery;\n  }\n\n  // Copied from org.apache.lucene.search.MatchAllDocsWeight\n  @Override\n  public String toString(String field) {\n    return \"*:*\";\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/RequiredStatusIDsFilter.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\nimport java.util.Arrays;\nimport java.util.Collection;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.query.DefaultFilterWeight;\nimport com.twitter.search.common.search.IntArrayDocIdSetIterator;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.util.AllDocsIterator;\nimport com.twitter.search.earlybird.index.TweetIDMapper;\n\npublic final class RequiredStatusIDsFilter extends Query {\n  private final Collection<Long> statusIDs;\n\n  public static Query getRequiredStatusIDsQuery(Collection<Long> statusIDs) {\n    return new BooleanQuery.Builder()\n        .add(new RequiredStatusIDsFilter(statusIDs), BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  private RequiredStatusIDsFilter(Collection<Long> statusIDs) {\n    this.statusIDs = Preconditions.checkNotNull(statusIDs);\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n    return new DefaultFilterWeight(this) {\n      @Override\n      protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context) throws IOException {\n        LeafReader leafReader = context.reader();\n        if (!(leafReader instanceof EarlybirdIndexSegmentAtomicReader)) {\n          return DocIdSetIterator.empty();\n        }\n\n        EarlybirdIndexSegmentAtomicReader reader = (EarlybirdIndexSegmentAtomicReader) leafReader;\n        TweetIDMapper idMapper = (TweetIDMapper) reader.getSegmentData().getDocIDToTweetIDMapper();\n\n        int docIdsSize = 0;\n        int[] docIds = new int[statusIDs.size()];\n        for (long statusID : statusIDs) {\n          int docId = idMapper.getDocID(statusID);\n          if (docId >= 0) {\n            docIds[docIdsSize++] = docId;\n          }\n        }\n\n        Arrays.sort(docIds, 0, docIdsSize);\n        DocIdSetIterator statusesDISI =\n            new IntArrayDocIdSetIterator(Arrays.copyOf(docIds, docIdsSize));\n        DocIdSetIterator allDocsDISI = new AllDocsIterator(reader);\n\n        // We only want to return IDs for fully indexed documents. So we need to make sure that\n        // every doc ID we return exists in allDocsDISI. However, allDocsDISI has all documents in\n        // this segment, so driving by allDocsDISI would be very slow. So we want to drive by\n        // statusesDISI, and use allDocsDISI as a post-filter. What this comes down to is that we do\n        // not want to call allDocsDISI.nextDoc(); we only want to call allDocsDISI.advance(), and\n        // only on the doc IDs returned by statusesDISI.\n        return new DocIdSetIterator() {\n          @Override\n          public int docID() {\n            return statusesDISI.docID();\n          }\n\n          @Override\n          public int nextDoc() throws IOException {\n            statusesDISI.nextDoc();\n            return advanceToNextFullyIndexedDoc();\n          }\n\n          @Override\n          public int advance(int target) throws IOException {\n            statusesDISI.advance(target);\n            return advanceToNextFullyIndexedDoc();\n          }\n\n          private int advanceToNextFullyIndexedDoc() throws IOException {\n            while (docID() != DocIdSetIterator.NO_MORE_DOCS) {\n              // Check if the current doc is fully indexed.\n              // If it is, then we can return it. If it's not, then we need to keep searching.\n              int allDocsDocId = allDocsDISI.advance(docID());\n              if (allDocsDocId == docID()) {\n                break;\n              }\n\n              statusesDISI.advance(allDocsDocId);\n            }\n            return docID();\n          }\n\n          @Override\n          public long cost() {\n            return statusesDISI.cost();\n          }\n        };\n      }\n    };\n  }\n\n  @Override\n  public int hashCode() {\n    return statusIDs.hashCode();\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof RequiredStatusIDsFilter)) {\n      return false;\n    }\n\n    RequiredStatusIDsFilter filter = RequiredStatusIDsFilter.class.cast(obj);\n    return statusIDs.equals(filter.statusIDs);\n  }\n\n  @Override\n  public final String toString(String field) {\n    return String.format(\"RequiredStatusIDs[%s]\", statusIDs);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/SimpleTermQuery.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.search.ConstantScoreScorer;\nimport org.apache.lucene.search.ConstantScoreWeight;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\n/**\n * A version of a term query that we can use when we already know the term id (in case where we\n * previously looked it up), and have a TermsEnum to get the actual postings.\n *\n * This is can be used for constant score queries, where only iterating on the postings is required.\n */\nclass SimpleTermQuery extends Query {\n  private final TermsEnum termsEnum;\n  private final long termId;\n\n  public SimpleTermQuery(TermsEnum termsEnum, long termId) {\n    this.termsEnum = termsEnum;\n    this.termId = termId;\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost)\n      throws IOException {\n    return new SimpleTermQueryWeight(scoreMode);\n  }\n\n  @Override\n  public int hashCode() {\n    return (termsEnum == null ? 0 : termsEnum.hashCode()) * 13 + (int) termId;\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof SimpleTermQuery)) {\n      return false;\n    }\n\n    SimpleTermQuery query = SimpleTermQuery.class.cast(obj);\n    return (termsEnum == null ? query.termsEnum == null : termsEnum.equals(query.termsEnum))\n        && (termId == query.termId);\n  }\n\n  @Override\n  public String toString(String field) {\n    return \"SimpleTermQuery(\" + field + \":\" + termId + \")\";\n  }\n\n  private class SimpleTermQueryWeight extends ConstantScoreWeight {\n    private final ScoreMode scoreMode;\n\n    public SimpleTermQueryWeight(ScoreMode scoreMode) {\n      super(SimpleTermQuery.this, 1.0f);\n      this.scoreMode = scoreMode;\n    }\n\n    @Override\n    public String toString() {\n      return \"weight(\" + SimpleTermQuery.this + \")\";\n    }\n\n    @Override\n    public Scorer scorer(LeafReaderContext context) throws IOException {\n      termsEnum.seekExact(termId);\n\n      PostingsEnum docs = termsEnum.postings(\n          null, scoreMode.needsScores() ? PostingsEnum.FREQS : PostingsEnum.NONE);\n      assert docs != null;\n      return new ConstantScoreScorer(this, 0, scoreMode, docs);\n    }\n\n    @Override\n    public boolean isCacheable(LeafReaderContext ctx) {\n      return true;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/SinceMaxIDFilter.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.query.DefaultFilterWeight;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.util.AllDocsIterator;\nimport com.twitter.search.core.earlybird.index.util.RangeFilterDISI;\nimport com.twitter.search.earlybird.index.TweetIDMapper;\n\n/**\n * Filters tweet ids according to since_id and max_id parameter.\n *\n * Note that since_id is exclusive and max_id is inclusive.\n */\npublic final class SinceMaxIDFilter extends Query {\n  public static final long NO_FILTER = -1;\n\n  private final long sinceIdExclusive;\n  private final long maxIdInclusive;\n\n  public static Query getSinceMaxIDQuery(long sinceIdExclusive, long maxIdInclusive) {\n    return new BooleanQuery.Builder()\n        .add(new SinceMaxIDFilter(sinceIdExclusive, maxIdInclusive), BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  public static Query getSinceIDQuery(long sinceIdExclusive) {\n    return new BooleanQuery.Builder()\n        .add(new SinceMaxIDFilter(sinceIdExclusive, NO_FILTER), BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  public static Query getMaxIDQuery(long maxIdInclusive) {\n    return new BooleanQuery.Builder()\n        .add(new SinceMaxIDFilter(NO_FILTER, maxIdInclusive), BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  private SinceMaxIDFilter(long sinceIdExclusive, long maxIdInclusive) {\n    this.sinceIdExclusive = sinceIdExclusive;\n    this.maxIdInclusive = maxIdInclusive;\n  }\n\n  @Override\n  public int hashCode() {\n    return (int) (sinceIdExclusive * 13 + maxIdInclusive);\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof SinceMaxIDFilter)) {\n      return false;\n    }\n\n    SinceMaxIDFilter filter = SinceMaxIDFilter.class.cast(obj);\n    return (sinceIdExclusive == filter.sinceIdExclusive)\n        && (maxIdInclusive == filter.maxIdInclusive);\n  }\n\n  @Override\n  public String toString(String field) {\n    if (sinceIdExclusive != NO_FILTER && maxIdInclusive != NO_FILTER) {\n      return \"SinceIdFilter:\" + sinceIdExclusive + \",MaxIdFilter:\" + maxIdInclusive;\n    } else if (maxIdInclusive != NO_FILTER) {\n      return \"MaxIdFilter:\" + maxIdInclusive;\n    } else {\n      return \"SinceIdFilter:\" + sinceIdExclusive;\n    }\n  }\n\n  /**\n   * Determines if this segment is at least partially covered by the given tweet ID range.\n   */\n  public static boolean sinceMaxIDsInRange(\n      TweetIDMapper tweetIdMapper, long sinceIdExclusive, long maxIdInclusive) {\n    // Check for since id out of range. Note that since this ID is exclusive,\n    // equality is out of range too.\n    if (sinceIdExclusive != NO_FILTER && sinceIdExclusive >= tweetIdMapper.getMaxTweetID()) {\n      return false;\n    }\n\n    // Check for max id in range.\n    return maxIdInclusive == NO_FILTER || maxIdInclusive >= tweetIdMapper.getMinTweetID();\n  }\n\n  // Returns true if this segment is completely covered by these id filters.\n  private static boolean sinceMaxIdsCoverRange(\n      TweetIDMapper tweetIdMapper, long sinceIdExclusive, long maxIdInclusive) {\n    // Check for since_id specified AND since_id newer than than first tweet.\n    if (sinceIdExclusive != NO_FILTER && sinceIdExclusive >= tweetIdMapper.getMinTweetID()) {\n      return false;\n    }\n\n    // Check for max id in range.\n    return maxIdInclusive == NO_FILTER || maxIdInclusive > tweetIdMapper.getMaxTweetID();\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost)\n      throws IOException {\n    return new DefaultFilterWeight(this) {\n      @Override\n      protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context) throws IOException {\n        LeafReader reader = context.reader();\n        if (!(reader instanceof EarlybirdIndexSegmentAtomicReader)) {\n          return new AllDocsIterator(reader);\n        }\n\n        EarlybirdIndexSegmentAtomicReader twitterInMemoryIndexReader =\n            (EarlybirdIndexSegmentAtomicReader) reader;\n        TweetIDMapper tweetIdMapper =\n            (TweetIDMapper) twitterInMemoryIndexReader.getSegmentData().getDocIDToTweetIDMapper();\n\n        // Important to return a null DocIdSetIterator here, so the Scorer will skip searching\n        // this segment completely.\n        if (!sinceMaxIDsInRange(tweetIdMapper, sinceIdExclusive, maxIdInclusive)) {\n          return null;\n        }\n\n        // Optimization: just return a match-all iterator when the whole segment is in range.\n        // This avoids having to do so many status id lookups.\n        if (sinceMaxIdsCoverRange(tweetIdMapper, sinceIdExclusive, maxIdInclusive)) {\n          return new AllDocsIterator(reader);\n        }\n\n        return new SinceMaxIDDocIdSetIterator(\n            twitterInMemoryIndexReader, sinceIdExclusive, maxIdInclusive);\n      }\n    };\n  }\n\n  @VisibleForTesting\n  static class SinceMaxIDDocIdSetIterator extends RangeFilterDISI {\n    private final DocIDToTweetIDMapper docIdToTweetIdMapper;\n    private final long sinceIdExclusive;\n    private final long maxIdInclusive;\n\n    public SinceMaxIDDocIdSetIterator(EarlybirdIndexSegmentAtomicReader reader,\n                                      long sinceIdExclusive,\n                                      long maxIdInclusive) throws IOException {\n      super(reader,\n            findMaxIdDocID(reader, maxIdInclusive),\n            findSinceIdDocID(reader, sinceIdExclusive));\n      this.docIdToTweetIdMapper = reader.getSegmentData().getDocIDToTweetIDMapper();\n      this.sinceIdExclusive = sinceIdExclusive;  // sinceStatusId == NO_FILTER is OK, it's exclusive\n      this.maxIdInclusive = maxIdInclusive != NO_FILTER ? maxIdInclusive : Long.MAX_VALUE;\n    }\n\n    /**\n     * This is a necessary check when we have out of order tweets in the archive.\n     * When tweets are out of order, this guarantees that no false positive results are returned.\n     * I.e. we can still miss some tweets in the specified range, but we never incorrectly return\n     * anything that's not in the range.\n     */\n    @Override\n    protected boolean shouldReturnDoc() {\n      final long statusID = docIdToTweetIdMapper.getTweetID(docID());\n      return statusID > sinceIdExclusive && statusID <= maxIdInclusive;\n    }\n\n    private static int findSinceIdDocID(\n        EarlybirdIndexSegmentAtomicReader reader, long sinceIdExclusive) throws IOException {\n      TweetIDMapper tweetIdMapper =\n          (TweetIDMapper) reader.getSegmentData().getDocIDToTweetIDMapper();\n      if (sinceIdExclusive != SinceMaxIDFilter.NO_FILTER) {\n        // We use this as an upper bound on the search, so we want to find the highest possible\n        // doc ID for this tweet ID.\n        boolean findMaxDocID = true;\n        return tweetIdMapper.findDocIdBound(\n            sinceIdExclusive,\n            findMaxDocID,\n            reader.getSmallestDocID(),\n            reader.maxDoc() - 1);\n      } else {\n        return DocIDToTweetIDMapper.ID_NOT_FOUND;\n      }\n    }\n\n    private static int findMaxIdDocID(\n        EarlybirdIndexSegmentAtomicReader reader, long maxIdInclusive) throws IOException {\n      TweetIDMapper tweetIdMapper =\n          (TweetIDMapper) reader.getSegmentData().getDocIDToTweetIDMapper();\n      if (maxIdInclusive != SinceMaxIDFilter.NO_FILTER) {\n        // We use this as a lower bound on the search, so we want to find the lowest possible\n        // doc ID for this tweet ID.\n        boolean findMaxDocID = false;\n        return tweetIdMapper.findDocIdBound(\n            maxIdInclusive,\n            findMaxDocID,\n            reader.getSmallestDocID(),\n            reader.maxDoc() - 1);\n      } else {\n        return DocIDToTweetIDMapper.ID_NOT_FOUND;\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/SinceUntilFilter.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.query.DefaultFilterWeight;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.TimeMapper;\nimport com.twitter.search.core.earlybird.index.util.AllDocsIterator;\nimport com.twitter.search.core.earlybird.index.util.RangeFilterDISI;\n\n// Filters tweets according to since time and until time (in seconds).\n// Note that since time is inclusive, and until time is exclusive.\npublic final class SinceUntilFilter extends Query {\n  public static final int NO_FILTER = -1;\n\n  // These are both in seconds since the epoch.\n  private final int minTimeInclusive;\n  private final int maxTimeExclusive;\n\n  public static Query getSinceQuery(int sinceTimeSeconds) {\n    return new BooleanQuery.Builder()\n        .add(new SinceUntilFilter(sinceTimeSeconds, NO_FILTER), BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  public static Query getUntilQuery(int untilTimeSeconds) {\n    return new BooleanQuery.Builder()\n        .add(new SinceUntilFilter(NO_FILTER, untilTimeSeconds), BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  public static Query getSinceUntilQuery(int sinceTimeSeconds, int untilTimeSeconds) {\n    return new BooleanQuery.Builder()\n        .add(new SinceUntilFilter(sinceTimeSeconds, untilTimeSeconds), BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  private SinceUntilFilter(int sinceTime, int untilTime) {\n    this.minTimeInclusive = sinceTime != NO_FILTER ? sinceTime : 0;\n    this.maxTimeExclusive = untilTime != NO_FILTER ? untilTime : Integer.MAX_VALUE;\n  }\n\n  @Override\n  public int hashCode() {\n    return (int) (minTimeInclusive * 17 + maxTimeExclusive);\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof SinceUntilFilter)) {\n      return false;\n    }\n\n    SinceUntilFilter filter = SinceUntilFilter.class.cast(obj);\n    return (minTimeInclusive == filter.minTimeInclusive)\n        && (maxTimeExclusive == filter.maxTimeExclusive);\n  }\n\n  @Override\n  public String toString(String field) {\n    if (minTimeInclusive > 0 && maxTimeExclusive != Integer.MAX_VALUE) {\n      return \"SinceFilter:\" + this.minTimeInclusive + \",UntilFilter:\" + maxTimeExclusive;\n    } else if (minTimeInclusive > 0) {\n      return \"SinceFilter:\" + this.minTimeInclusive;\n    } else {\n      return \"UntilFilter:\" + this.maxTimeExclusive;\n    }\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost)\n      throws IOException {\n    return new DefaultFilterWeight(this) {\n      @Override\n      protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context) throws IOException {\n        LeafReader indexReader = context.reader();\n        if (!(indexReader instanceof EarlybirdIndexSegmentAtomicReader)) {\n          return new AllDocsIterator(indexReader);\n        }\n\n        EarlybirdIndexSegmentAtomicReader reader = (EarlybirdIndexSegmentAtomicReader) indexReader;\n        TimeMapper timeMapper = reader.getSegmentData().getTimeMapper();\n        int smallestDocID = timeMapper.findFirstDocId(maxTimeExclusive, reader.getSmallestDocID());\n        int largestDoc = timeMapper.findFirstDocId(minTimeInclusive, reader.getSmallestDocID());\n        int smallestDoc = smallestDocID > 0 ? smallestDocID - 1 : 0;\n        return new SinceUntilDocIdSetIterator(\n            reader,\n            timeMapper,\n            smallestDoc,\n            largestDoc,\n            minTimeInclusive,\n            maxTimeExclusive);\n      }\n    };\n  }\n\n  // Returns true if this TimeMapper is at least partially covered by these time filters.\n  public static boolean sinceUntilTimesInRange(\n      TimeMapper timeMapper, int sinceTime, int untilTime) {\n    return (sinceTime == NO_FILTER || sinceTime <= timeMapper.getLastTime())\n        && (untilTime == NO_FILTER || untilTime >= timeMapper.getFirstTime());\n  }\n\n  private static final class SinceUntilDocIdSetIterator extends RangeFilterDISI {\n    private final TimeMapper timeMapper;\n    private final int minTimeInclusive;\n    private final int maxTimeExclusive;\n\n    public SinceUntilDocIdSetIterator(EarlybirdIndexSegmentAtomicReader reader,\n                                      TimeMapper timeMapper,\n                                      int smallestDocID,\n                                      int largestDocID,\n                                      int minTimeInclusive,\n                                      int maxExclusive) throws IOException {\n      super(reader, smallestDocID, largestDocID);\n      this.timeMapper = timeMapper;\n      this.minTimeInclusive = minTimeInclusive;\n      this.maxTimeExclusive = maxExclusive;\n    }\n\n    @Override\n    protected boolean shouldReturnDoc() {\n      final int docTime = timeMapper.getTime(docID());\n      return docTime >= minTimeInclusive && docTime < maxTimeExclusive;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/TermQueryWithSafeToString.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.TermQuery;\n\n/**\n * Work around an issue where IntTerms and LongTerms are not valid utf8,\n * so calling toString on any TermQuery containing an IntTerm or a LongTerm may cause exceptions.\n * This code should produce the same output as TermQuery.toString\n */\npublic final class TermQueryWithSafeToString extends TermQuery {\n  private final String termValueForToString;\n\n  public TermQueryWithSafeToString(Term term, String termValueForToString) {\n    super(term);\n    this.termValueForToString = termValueForToString;\n  }\n\n  @Override\n  public String toString(String field) {\n    StringBuilder buffer = new StringBuilder();\n    if (!getTerm().field().equals(field)) {\n      buffer.append(getTerm().field());\n      buffer.append(\":\");\n    }\n    buffer.append(termValueForToString);\n    return buffer.toString();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/TimedDocIdSetIterator.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.lucene.search.DocIdSetIterator;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.search.EarlyTerminationState;\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\n\n/**\n * DocIdSetIterator whose nextDoc() and advance() will early terminate by returning NO_MORE_DOCS\n * after the given deadline.\n */\npublic class TimedDocIdSetIterator extends DocIdSetIterator {\n  // check deadline every NEXT_CALL_TIMEOUT_CHECK_PERIOD calls to nextDoc()\n  @VisibleForTesting\n  protected static final int NEXT_CALL_TIMEOUT_CHECK_PERIOD =\n      EarlybirdConfig.getInt(\"timed_doc_id_set_next_doc_deadline_check_period\", 1000);\n\n\n  // check deadline every ADVANCE_CALL_TIMEOUT_CHECK_PERIOD calls to advance()\n  private static final int ADVANCE_CALL_TIMEOUT_CHECK_PERIOD =\n      EarlybirdConfig.getInt(\"timed_doc_id_set_advance_deadline_check_period\", 100);\n\n  private final Clock clock;\n  private final DocIdSetIterator innerIterator;\n  private final SearchCounter timeoutCountStat;\n\n  @Nullable\n  private final TerminationTracker terminationTracker;\n  private final long deadlineMillisFromEpoch;\n\n  private int docId = -1;\n  private int nextCounter = 0;\n  private int advanceCounter = 0;\n\n  public TimedDocIdSetIterator(DocIdSetIterator innerIterator,\n                               @Nullable TerminationTracker terminationTracker,\n                               final long timeoutOverride,\n                               @Nullable SearchCounter timeoutCountStat) {\n    this(innerIterator, terminationTracker, timeoutOverride, timeoutCountStat, Clock.SYSTEM_CLOCK);\n  }\n\n  protected TimedDocIdSetIterator(DocIdSetIterator innerIterator,\n                                  @Nullable TerminationTracker terminationTracker,\n                                  final long timeoutOverride,\n                                  @Nullable SearchCounter timeoutCountStat,\n                                  Clock clock) {\n    this.clock = clock;\n    this.innerIterator = innerIterator;\n    this.timeoutCountStat = timeoutCountStat;\n    this.terminationTracker = terminationTracker;\n\n    if (terminationTracker == null) {\n      deadlineMillisFromEpoch = -1;\n    } else {\n      if (timeoutOverride > 0) {\n        deadlineMillisFromEpoch = terminationTracker.getClientStartTimeMillis() + timeoutOverride;\n      } else {\n        deadlineMillisFromEpoch = terminationTracker.getTimeoutEndTimeWithReservation();\n      }\n    }\n  }\n\n  @VisibleForTesting\n  protected TimedDocIdSetIterator(DocIdSetIterator innerIterator,\n          final long deadline,\n          @Nullable SearchCounter timeoutCountStat,\n          Clock clock) {\n    this.clock = clock;\n    this.innerIterator = innerIterator;\n    this.timeoutCountStat = timeoutCountStat;\n    this.terminationTracker = null;\n\n    this.deadlineMillisFromEpoch = deadline;\n  }\n\n\n  @Override\n  public int docID() {\n    return docId;\n  }\n\n  @Override\n  public int nextDoc() throws IOException {\n    if (++nextCounter % NEXT_CALL_TIMEOUT_CHECK_PERIOD == 0\n        && clock.nowMillis() > deadlineMillisFromEpoch) {\n      if (timeoutCountStat != null) {\n        timeoutCountStat.increment();\n      }\n      if (terminationTracker != null) {\n        terminationTracker.setEarlyTerminationState(\n            EarlyTerminationState.TERMINATED_TIME_OUT_EXCEEDED);\n      }\n\n      return docId = NO_MORE_DOCS;\n    }\n    return docId = innerIterator.nextDoc();\n  }\n\n  @Override\n  public int advance(int target) throws IOException {\n    if (++advanceCounter % ADVANCE_CALL_TIMEOUT_CHECK_PERIOD == 0\n        && clock.nowMillis() > deadlineMillisFromEpoch) {\n      if (timeoutCountStat != null) {\n        timeoutCountStat.increment();\n      }\n      if (terminationTracker != null) {\n        terminationTracker.setEarlyTerminationState(\n            EarlyTerminationState.TERMINATED_TIME_OUT_EXCEEDED);\n      }\n      return docId = NO_MORE_DOCS;\n    }\n\n    return docId = innerIterator.advance(target);\n  }\n\n  @Override\n  public long cost() {\n    return innerIterator.cost();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/UserFlagsExcludeFilter.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.query.DefaultFilterWeight;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.index.util.AllDocsIterator;\nimport com.twitter.search.core.earlybird.index.util.RangeFilterDISI;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\n\npublic final class UserFlagsExcludeFilter extends Query {\n  /**\n   * Returns a query that filters hits based on their author flags.\n   *\n   * @param excludeAntisocial Determines if the filter should exclude hits from antisocial users.\n   * @param excludeOffensive Determines if the filter should exclude hits from offensive users.\n   * @param excludeProtected Determines if the filter should exclude hits from protected users\n   * @return A query that filters hits based on their author flags.\n   */\n  public static Query getUserFlagsExcludeFilter(UserTable userTable,\n                                                boolean excludeAntisocial,\n                                                boolean excludeOffensive,\n                                                boolean excludeProtected) {\n    return new BooleanQuery.Builder()\n        .add(new UserFlagsExcludeFilter(\n                userTable, excludeAntisocial, excludeOffensive, excludeProtected),\n            BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  private final UserTable userTable;\n  private final boolean excludeAntisocial;\n  private final boolean excludeOffensive;\n  private final boolean excludeProtected;\n\n  private UserFlagsExcludeFilter(\n      UserTable userTable,\n      boolean excludeAntisocial,\n      boolean excludeOffensive,\n      boolean excludeProtected) {\n    this.userTable = userTable;\n    this.excludeAntisocial = excludeAntisocial;\n    this.excludeOffensive = excludeOffensive;\n    this.excludeProtected = excludeProtected;\n  }\n\n  @Override\n  public int hashCode() {\n    return (excludeAntisocial ? 13 : 0) + (excludeOffensive ? 1 : 0) + (excludeProtected ? 2 : 0);\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof UserFlagsExcludeFilter)) {\n      return false;\n    }\n\n    UserFlagsExcludeFilter filter = UserFlagsExcludeFilter.class.cast(obj);\n    return (excludeAntisocial == filter.excludeAntisocial)\n        && (excludeOffensive == filter.excludeOffensive)\n        && (excludeProtected == filter.excludeProtected);\n  }\n\n  @Override\n  public String toString(String field) {\n    return \"UserFlagsExcludeFilter\";\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n    return new DefaultFilterWeight(this) {\n      @Override\n      protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context) throws IOException {\n        LeafReader reader = context.reader();\n        if (userTable == null) {\n          return new AllDocsIterator(reader);\n        }\n\n        final int bits =\n            (excludeAntisocial ? UserTable.ANTISOCIAL_BIT : 0)\n                | (excludeOffensive ? UserTable.OFFENSIVE_BIT | UserTable.NSFW_BIT : 0)\n                | (excludeProtected ? UserTable.IS_PROTECTED_BIT : 0);\n        if (bits != 0) {\n          return new UserFlagsExcludeDocIdSetIterator(reader, userTable) {\n            @Override\n            protected boolean checkUserFlags(UserTable table, long userID) {\n              return !table.isSet(userID, bits);\n            }\n          };\n        }\n\n        return new AllDocsIterator(reader);\n      }\n    };\n  }\n\n  private abstract static class UserFlagsExcludeDocIdSetIterator extends RangeFilterDISI {\n    private final UserTable userTable;\n    private final NumericDocValues fromUserID;\n\n    public UserFlagsExcludeDocIdSetIterator(\n        LeafReader indexReader, UserTable table) throws IOException {\n      super(indexReader);\n      userTable = table;\n      fromUserID =\n          indexReader.getNumericDocValues(EarlybirdFieldConstant.FROM_USER_ID_CSF.getFieldName());\n    }\n\n    @Override\n    protected boolean shouldReturnDoc() throws IOException {\n      return fromUserID.advanceExact(docID())\n          && checkUserFlags(userTable, fromUserID.longValue());\n    }\n\n    protected abstract boolean checkUserFlags(UserTable table, long userID);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/UserIdMultiSegmentQuery.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\nimport java.util.Arrays;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.concurrent.TimeUnit;\nimport java.util.stream.Collectors;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.BulkScorer;\nimport org.apache.lucene.search.ConstantScoreQuery;\nimport org.apache.lucene.search.ConstantScoreWeight;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.search.common.decider.DeciderUtil;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.query.HitAttributeHelper;\nimport com.twitter.search.common.query.IDDisjunctionQuery;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.IndexedNumericFieldSettings;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.search.termination.QueryTimeout;\nimport com.twitter.search.common.util.analysis.LongTermAttributeImpl;\nimport com.twitter.search.common.util.analysis.SortableLongTermAttributeImpl;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentData;\nimport com.twitter.search.core.earlybird.index.inverted.InvertedIndex;\nimport com.twitter.search.core.earlybird.index.inverted.MultiSegmentTermDictionary;\nimport com.twitter.search.earlybird.partition.MultiSegmentTermDictionaryManager;\nimport com.twitter.search.earlybird.queryparser.EarlybirdQueryHelper;\nimport com.twitter.search.queryparser.query.QueryParserException;\n\n/**\n * A variant of a multi-term ID disjunction query (similar to {@link UserIdMultiSegmentQuery}),\n * that also uses a {@link MultiSegmentTermDictionary} where available, for more efficient\n * term lookups for queries that span multiple segments.\n *\n * By default, a IDDisjunctionQuery (or Lucene's MultiTermQuery), does a term dictionary lookup\n * for all of the terms in its disjunction, and it does it once for each segment (or AtomicReader)\n * that the query is searching.\n * This means that when the term dictionary is large, and the term lookups are expensive, and when\n * we are searching multiple segments, the query needs to make num_terms * num_segments expensive\n * term dictionary lookups.\n *\n * With the help of a MultiSegmentTermDictionary, this multi-term disjunction query implementation\n * only does one lookup for all of the segments managed by the MultiSegmentTermDictionary.\n * If a segment is not supported by the MultiSegmentTermDictionary (e.g. if it's not optimized yet),\n * a regular lookup in that segment's term dictionary will be performed.\n *\n * Usually, we will make 'num_terms' lookups in the current, un-optimized segment, and then if\n * more segments need to be searched, we will make another 'num_terms' lookups, once for all of\n * the remaining segments.\n *\n * When performing lookups in the MultiSegmentTermDictionary, for each supported segment, we save\n * a list of termIds from that segment for all the searched terms that appear in that segment.\n *\n * For example, when querying for UserIdMultiSegmentQuery with user ids: {1L, 2L, 3L} and\n * segments: {1, 2}, where segment 1 has user ids {1L, 2L} indexed under termIds {100, 200},\n * and segment 2 has user ids {1L, 2L, 3L} indexed under termIds {200, 300, 400}, we will build\n * up the following map once:\n *   segment1 -> [100, 200]\n *   segment2 -> [200, 300, 400]\n */\npublic class UserIdMultiSegmentQuery extends Query {\n  @VisibleForTesting\n  public static final SearchTimerStats TERM_LOOKUP_STATS =\n      SearchTimerStats.export(\"multi_segment_query_term_lookup\", TimeUnit.NANOSECONDS, false);\n  public static final SearchTimerStats QUERY_FROM_PRECOMPUTED =\n      SearchTimerStats.export(\"multi_segment_query_from_precomputed\", TimeUnit.NANOSECONDS, false);\n  public static final SearchTimerStats QUERY_REGULAR =\n      SearchTimerStats.export(\"multi_segment_query_regular\", TimeUnit.NANOSECONDS, false);\n\n  @VisibleForTesting\n  public static final SearchCounter USED_MULTI_SEGMENT_TERM_DICTIONARY_COUNT = SearchCounter.export(\n      \"user_id_multi_segment_query_used_multi_segment_term_dictionary_count\");\n  @VisibleForTesting\n  public static final SearchCounter USED_ORIGINAL_TERM_DICTIONARY_COUNT = SearchCounter.export(\n      \"user_id_multi_segment_query_used_original_term_dictionary_count\");\n\n  private static final SearchCounter NEW_QUERY_COUNT =\n      SearchCounter.export(\"user_id_multi_segment_new_query_count\");\n  private static final SearchCounter OLD_QUERY_COUNT =\n      SearchCounter.export(\"user_id_multi_segment_old_query_count\");\n\n  private static final HashMap<String, SearchCounter> QUERY_COUNT_BY_QUERY_NAME = new HashMap<>();\n  private static final HashMap<String, SearchCounter> QUERY_COUNT_BY_FIELD_NAME = new HashMap<>();\n\n  private static final String DECIDER_KEY_PREFIX = \"use_multi_segment_id_disjunction_queries_in_\";\n\n  /**\n   * Returns a new user ID disjunction query.\n   *\n   * @param ids The user IDs.\n   * @param field The field storing the user IDs.\n   * @param schemaSnapshot A snapshot of earlybird's schema.\n   * @param multiSegmentTermDictionaryManager The manager for the term dictionaries that span\n   *                                          multiple segments.\n   * @param decider The decider.\n   * @param earlybirdCluster The earlybird cluster.\n   * @param ranks The hit attribution ranks to be assigned to every user ID.\n   * @param hitAttributeHelper The helper that tracks hit attributions.\n   * @param queryTimeout The timeout to be enforced on this query.\n   * @return A new user ID disjunction query.\n   */\n  public static Query createIdDisjunctionQuery(\n      String queryName,\n      List<Long> ids,\n      String field,\n      ImmutableSchemaInterface schemaSnapshot,\n      MultiSegmentTermDictionaryManager multiSegmentTermDictionaryManager,\n      Decider decider,\n      EarlybirdCluster earlybirdCluster,\n      List<Integer> ranks,\n      @Nullable HitAttributeHelper hitAttributeHelper,\n      @Nullable QueryTimeout queryTimeout) throws QueryParserException {\n    QUERY_COUNT_BY_QUERY_NAME.computeIfAbsent(queryName, name ->\n        SearchCounter.export(\"multi_segment_query_name_\" + name)).increment();\n    QUERY_COUNT_BY_FIELD_NAME.computeIfAbsent(field, name ->\n        SearchCounter.export(\"multi_segment_query_count_for_field_\" + name)).increment();\n\n    if (DeciderUtil.isAvailableForRandomRecipient(decider, getDeciderName(earlybirdCluster))) {\n      NEW_QUERY_COUNT.increment();\n      MultiSegmentTermDictionary multiSegmentTermDictionary =\n          multiSegmentTermDictionaryManager.getMultiSegmentTermDictionary(field);\n      return new UserIdMultiSegmentQuery(\n          ids,\n          field,\n          schemaSnapshot,\n          multiSegmentTermDictionary,\n          ranks,\n          hitAttributeHelper,\n          queryTimeout);\n    } else {\n      OLD_QUERY_COUNT.increment();\n      return new IDDisjunctionQuery(ids, field, schemaSnapshot);\n    }\n  }\n\n  @VisibleForTesting\n  public static String getDeciderName(EarlybirdCluster earlybirdCluster) {\n    return DECIDER_KEY_PREFIX + earlybirdCluster.name().toLowerCase();\n  }\n\n  private final boolean useOrderPreservingEncoding;\n  private final HitAttributeHelper hitAttributeHelper;\n  private final QueryTimeout queryTimeout;\n  private final MultiSegmentTermDictionary multiSegmentTermDictionary;\n  private final Schema.FieldInfo fieldInfo;\n  private final String field;\n  private final List<Long> ids;\n\n  private final List<Integer> ranks;\n  // For each segment where we have a multi-segment term dictionary, this map will contain the\n  // termIds of all the terms that actually appear in that segment's index.\n  @Nullable\n  private Map<InvertedIndex, List<TermRankPair>> termIdsPerSegment;\n\n  // A wrap class helps to associate termId with corresponding search operator rank if exist\n  private final class TermRankPair {\n    private final int termId;\n    private final int rank;\n\n    TermRankPair(int termId, int rank) {\n      this.termId = termId;\n      this.rank = rank;\n    }\n\n    public int getTermId() {\n      return termId;\n    }\n\n    public int getRank() {\n      return rank;\n    }\n  }\n\n  @VisibleForTesting\n  public UserIdMultiSegmentQuery(\n      List<Long> ids,\n      String field,\n      ImmutableSchemaInterface schemaSnapshot,\n      MultiSegmentTermDictionary termDictionary,\n      List<Integer> ranks,\n      @Nullable HitAttributeHelper hitAttributeHelper,\n      @Nullable QueryTimeout queryTimeout) {\n    this.field = field;\n    this.ids = ids;\n    this.multiSegmentTermDictionary = termDictionary;\n    this.ranks = ranks;\n    this.hitAttributeHelper = hitAttributeHelper;\n    this.queryTimeout = queryTimeout;\n\n    // check ids and ranks have same size\n    Preconditions.checkArgument(ranks.size() == 0 || ranks.size() == ids.size());\n    // hitAttributeHelper is not null iff ranks is not empty\n    if (ranks.size() > 0) {\n      Preconditions.checkNotNull(hitAttributeHelper);\n    } else {\n      Preconditions.checkArgument(hitAttributeHelper == null);\n    }\n\n    if (!schemaSnapshot.hasField(field)) {\n      throw new IllegalStateException(\"Tried to search a field which does not exist in schema\");\n    }\n    this.fieldInfo = Preconditions.checkNotNull(schemaSnapshot.getFieldInfo(field));\n\n    IndexedNumericFieldSettings numericFieldSettings =\n        fieldInfo.getFieldType().getNumericFieldSettings();\n    if (numericFieldSettings == null) {\n      throw new IllegalStateException(\"Id field is not numerical\");\n    }\n\n    this.useOrderPreservingEncoding = numericFieldSettings.isUseSortableEncoding();\n  }\n\n  /**\n   * If it hasn't been built yet, build up the map containing termIds of all the terms being\n   * searched, for all of the segments that are managed by the multi-segment term dictionary.\n   *\n   * We only do this once, when we have to search the first segment that's supported by our\n   * multi-segment term dictionary.\n   *\n   * Flow here is to:\n   * 1. go through all the ids being queried.\n   * 2. for each id, get the termIds for that term in all of the segments in the term dictionary\n   * 3. for all of the segments that have that term, add the termId to that segment's list of\n   * term ids (in the 'termIdsPerSegment' map).\n   */\n  private void createTermIdsPerSegment() {\n    if (termIdsPerSegment != null) {\n      // already created the map\n      return;\n    }\n\n    long start = System.nanoTime();\n\n    final BytesRef termRef = useOrderPreservingEncoding\n        ? SortableLongTermAttributeImpl.newBytesRef()\n        : LongTermAttributeImpl.newBytesRef();\n\n    termIdsPerSegment = Maps.newHashMap();\n    List<? extends InvertedIndex> segmentIndexes = multiSegmentTermDictionary.getSegmentIndexes();\n\n    for (int idx = 0; idx < ids.size(); ++idx) {\n      long longTerm = ids.get(idx);\n\n      if (useOrderPreservingEncoding) {\n        SortableLongTermAttributeImpl.copyLongToBytesRef(termRef, longTerm);\n      } else {\n        LongTermAttributeImpl.copyLongToBytesRef(termRef, longTerm);\n      }\n\n      int[] termIds = multiSegmentTermDictionary.lookupTermIds(termRef);\n      Preconditions.checkState(segmentIndexes.size() == termIds.length,\n          \"SegmentIndexes: %s, field: %s, termIds: %s\",\n          segmentIndexes.size(), field, termIds.length);\n\n      for (int indexId = 0; indexId < termIds.length; indexId++) {\n        int termId = termIds[indexId];\n        if (termId != EarlybirdIndexSegmentAtomicReader.TERM_NOT_FOUND) {\n          InvertedIndex fieldIndex = segmentIndexes.get(indexId);\n\n          List<TermRankPair> termIdsList = termIdsPerSegment.get(fieldIndex);\n          if (termIdsList == null) {\n            termIdsList = Lists.newArrayList();\n            termIdsPerSegment.put(fieldIndex, termIdsList);\n          }\n          termIdsList.add(new TermRankPair(\n              termId, ranks.size() > 0 ? ranks.get(idx) : -1));\n        }\n      }\n    }\n\n    long elapsed = System.nanoTime() - start;\n    TERM_LOOKUP_STATS.timerIncrement(elapsed);\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n    return new UserIdMultiSegmentQueryWeight(searcher, scoreMode, boost);\n  }\n\n  @Override\n  public int hashCode() {\n    return Arrays.hashCode(\n        new Object[] {useOrderPreservingEncoding, queryTimeout, field, ids, ranks});\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof UserIdMultiSegmentQuery)) {\n      return false;\n    }\n\n    UserIdMultiSegmentQuery query = UserIdMultiSegmentQuery.class.cast(obj);\n    return Arrays.equals(\n        new Object[] {useOrderPreservingEncoding, queryTimeout, field, ids, ranks},\n        new Object[] {query.useOrderPreservingEncoding,\n                      query.queryTimeout,\n                      query.field,\n                      query.ids,\n                      query.ranks});\n  }\n\n  @Override\n  public String toString(String fieldName) {\n    StringBuilder builder = new StringBuilder();\n    builder.append(getClass().getSimpleName()).append(\"[\").append(fieldName).append(\":\");\n    for (Long id : this.ids) {\n      builder.append(id);\n      builder.append(\",\");\n    }\n    builder.setLength(builder.length() - 1);\n    builder.append(\"]\");\n    return builder.toString();\n  }\n\n  private final class UserIdMultiSegmentQueryWeight extends ConstantScoreWeight {\n    private final IndexSearcher searcher;\n    private final ScoreMode scoreMode;\n\n    private UserIdMultiSegmentQueryWeight(\n        IndexSearcher searcher,\n        ScoreMode scoreMode,\n        float boost) {\n      super(UserIdMultiSegmentQuery.this, boost);\n      this.searcher = searcher;\n      this.scoreMode = scoreMode;\n    }\n\n    @Override\n    public Scorer scorer(LeafReaderContext context) throws IOException {\n      Weight weight = rewrite(context);\n      if (weight != null) {\n        return weight.scorer(context);\n      } else {\n        return null;\n      }\n    }\n\n    @Override\n    public BulkScorer bulkScorer(LeafReaderContext context) throws IOException {\n      Weight weight = rewrite(context);\n      if (weight != null) {\n        return weight.bulkScorer(context);\n      } else {\n        return null;\n      }\n    }\n\n    @Override\n    public void extractTerms(Set<Term> terms) {\n      terms.addAll(ids\n          .stream()\n          .map(id -> new Term(field, LongTermAttributeImpl.copyIntoNewBytesRef(id)))\n          .collect(Collectors.toSet()));\n    }\n\n    @Override\n    public boolean isCacheable(LeafReaderContext ctx) {\n      return true;\n    }\n\n    private Weight rewrite(LeafReaderContext context) throws IOException {\n      final Terms terms = context.reader().terms(field);\n      if (terms == null) {\n        // field does not exist\n        return null;\n      }\n      final TermsEnum termsEnum = terms.iterator();\n      Preconditions.checkNotNull(termsEnum, \"No termsEnum for field: %s\", field);\n\n      BooleanQuery bq;\n      // See if the segment is supported by the multi-segment term dictionary. If so, build up\n      // the query using the termIds from the multi-segment term dictionary.\n      // If not (for the current segment), do the term lookups directly in the queried segment.\n      InvertedIndex fieldIndex = getFieldIndexFromMultiTermDictionary(context);\n      if (fieldIndex != null) {\n        createTermIdsPerSegment();\n\n        USED_MULTI_SEGMENT_TERM_DICTIONARY_COUNT.increment();\n        SearchTimer timer = QUERY_FROM_PRECOMPUTED.startNewTimer();\n        bq = addPrecomputedTermQueries(fieldIndex, termsEnum);\n        QUERY_FROM_PRECOMPUTED.stopTimerAndIncrement(timer);\n      } else {\n        USED_ORIGINAL_TERM_DICTIONARY_COUNT.increment();\n        // This segment is not supported by the multi-segment term dictionary. Lookup terms\n        // directly.\n        SearchTimer timer = QUERY_REGULAR.startNewTimer();\n        bq = addTermQueries(termsEnum);\n        QUERY_REGULAR.stopTimerAndIncrement(timer);\n      }\n\n      return searcher.rewrite(new ConstantScoreQuery(bq)).createWeight(\n          searcher, scoreMode, score());\n    }\n\n    /**\n     * If the multi-segment term dictionary supports this segment/LeafReader, then return the\n     * InvertedIndex representing this segment.\n     *\n     * If the segment being queried right now is not in the multi-segment term dictionary (e.g.\n     * if it's not optimized yet), return null.\n     */\n    @Nullable\n    private InvertedIndex getFieldIndexFromMultiTermDictionary(LeafReaderContext context)\n        throws IOException {\n      if (multiSegmentTermDictionary == null) {\n        return null;\n      }\n\n      if (context.reader() instanceof EarlybirdIndexSegmentAtomicReader) {\n        EarlybirdIndexSegmentAtomicReader reader =\n            (EarlybirdIndexSegmentAtomicReader) context.reader();\n\n        EarlybirdIndexSegmentData segmentData = reader.getSegmentData();\n        InvertedIndex fieldIndex = segmentData.getFieldIndex(field);\n\n        if (multiSegmentTermDictionary.supportSegmentIndex(fieldIndex)) {\n          return fieldIndex;\n        }\n      }\n\n      return null;\n    }\n\n    private BooleanQuery addPrecomputedTermQueries(\n        InvertedIndex fieldIndex,\n        TermsEnum termsEnum) throws IOException {\n\n      BooleanQuery.Builder bqBuilder = new BooleanQuery.Builder();\n      int numClauses = 0;\n\n      List<TermRankPair> termRankPairs = termIdsPerSegment.get(fieldIndex);\n      if (termRankPairs != null) {\n        for (TermRankPair pair : termRankPairs) {\n          int termId = pair.getTermId();\n          if (numClauses >= BooleanQuery.getMaxClauseCount()) {\n            BooleanQuery saved = bqBuilder.build();\n            bqBuilder = new BooleanQuery.Builder();\n            bqBuilder.add(saved, BooleanClause.Occur.SHOULD);\n            numClauses = 1;\n          }\n\n          Query query;\n          if (pair.getRank() != -1) {\n            query = EarlybirdQueryHelper.maybeWrapWithHitAttributionCollector(\n                new SimpleTermQuery(termsEnum, termId),\n                pair.getRank(),\n                fieldInfo,\n                hitAttributeHelper);\n          } else {\n            query = new SimpleTermQuery(termsEnum, termId);\n          }\n          bqBuilder.add(EarlybirdQueryHelper.maybeWrapWithTimeout(query, queryTimeout),\n                        BooleanClause.Occur.SHOULD);\n          ++numClauses;\n        }\n      }\n      return bqBuilder.build();\n    }\n\n    private BooleanQuery addTermQueries(TermsEnum termsEnum) throws IOException {\n      final BytesRef termRef = useOrderPreservingEncoding\n          ? SortableLongTermAttributeImpl.newBytesRef()\n          : LongTermAttributeImpl.newBytesRef();\n\n      BooleanQuery.Builder bqBuilder = new BooleanQuery.Builder();\n      int numClauses = 0;\n\n      for (int idx = 0; idx < ids.size(); ++idx) {\n        long longTerm = ids.get(idx);\n        if (useOrderPreservingEncoding) {\n          SortableLongTermAttributeImpl.copyLongToBytesRef(termRef, longTerm);\n        } else {\n          LongTermAttributeImpl.copyLongToBytesRef(termRef, longTerm);\n        }\n\n        if (termsEnum.seekExact(termRef)) {\n          if (numClauses >= BooleanQuery.getMaxClauseCount()) {\n            BooleanQuery saved = bqBuilder.build();\n            bqBuilder = new BooleanQuery.Builder();\n            bqBuilder.add(saved, BooleanClause.Occur.SHOULD);\n            numClauses = 1;\n          }\n\n          if (ranks.size() > 0) {\n            bqBuilder.add(EarlybirdQueryHelper.maybeWrapWithHitAttributionCollector(\n                              new SimpleTermQuery(termsEnum, termsEnum.ord()),\n                              ranks.get(idx),\n                              fieldInfo,\n                              hitAttributeHelper),\n                          BooleanClause.Occur.SHOULD);\n          } else {\n            bqBuilder.add(new SimpleTermQuery(termsEnum, termsEnum.ord()),\n                          BooleanClause.Occur.SHOULD);\n          }\n          ++numClauses;\n        }\n      }\n\n      return bqBuilder.build();\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/queries/UserScrubGeoFilter.java",
    "content": "package com.twitter.search.earlybird.search.queries;\n\nimport java.io.IOException;\nimport java.util.Objects;\n\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.NumericDocValues;\n\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.query.FilteredQuery;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.earlybird.common.userupdates.UserScrubGeoMap;\nimport com.twitter.search.earlybird.index.TweetIDMapper;\n\n/**\n * Filter that can be used with searches over geo field postings lists in order to filter out tweets\n * that have been geo scrubbed. Determines if a tweet has been geo scrubbed by comparing the\n * tweet's id against the max scrubbed tweet id for that tweet's author, which is stored in the\n * UserScrubGeoMap.\n *\n * See: go/realtime-geo-filtering\n */\npublic class UserScrubGeoFilter implements FilteredQuery.DocIdFilterFactory {\n\n  private UserScrubGeoMap userScrubGeoMap;\n\n  private final SearchRateCounter totalRequestsUsingFilterCounter =\n      SearchRateCounter.export(\"user_scrub_geo_filter_total_requests\");\n\n  public static FilteredQuery.DocIdFilterFactory getDocIdFilterFactory(\n      UserScrubGeoMap userScrubGeoMap) {\n    return new UserScrubGeoFilter(userScrubGeoMap);\n  }\n\n  public UserScrubGeoFilter(UserScrubGeoMap userScrubGeoMap) {\n    this.userScrubGeoMap = userScrubGeoMap;\n    totalRequestsUsingFilterCounter.increment();\n  }\n\n  @Override\n  public FilteredQuery.DocIdFilter getDocIdFilter(LeafReaderContext context) throws IOException {\n    // To determine if a given doc has been geo scrubbed we need two pieces of information about the\n    // doc: the associated tweet id and the user id of the tweet's author. We can get the tweet id\n    // from the TweetIDMapper for the segment we are currently searching, and we can get the user id\n    // of the tweet's author by looking up the doc id in the NumericDocValues for the\n    // FROM_USER_ID_CSF.\n    //\n    // With this information we can check the UserScrubGeoMap to find out if the tweet has been\n    // geo scrubbed and filter it out accordingly.\n    final EarlybirdIndexSegmentAtomicReader currTwitterReader =\n        (EarlybirdIndexSegmentAtomicReader) context.reader();\n    final TweetIDMapper tweetIdMapper =\n        (TweetIDMapper) currTwitterReader.getSegmentData().getDocIDToTweetIDMapper();\n    final NumericDocValues fromUserIdDocValues = currTwitterReader.getNumericDocValues(\n        EarlybirdFieldConstant.FROM_USER_ID_CSF.getFieldName());\n    return (docId) -> fromUserIdDocValues.advanceExact(docId)\n        && !userScrubGeoMap.isTweetGeoScrubbed(\n            tweetIdMapper.getTweetID(docId), fromUserIdDocValues.longValue());\n  }\n\n  @Override\n  public String toString() {\n    return \"UserScrubGeoFilter\";\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof UserScrubGeoMap)) {\n      return false;\n    }\n\n    UserScrubGeoFilter filter = UserScrubGeoFilter.class.cast(obj);\n    // filters are considered equal as long as they are using the same UserScrubGeoMap\n    return Objects.equals(userScrubGeoMap, filter.userScrubGeoMap);\n  }\n\n  @Override\n  public int hashCode() {\n    return userScrubGeoMap == null ? 0 : userScrubGeoMap.hashCode();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/LinearScoringData.java",
    "content": "package com.twitter.search.earlybird.search.relevance;\n\nimport java.util.Arrays;\nimport java.util.List;\n\nimport com.google.common.collect.Lists;\n\nimport com.twitter.search.common.constants.SearchCardType;\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\n\npublic class LinearScoringData {\n  public static final float NO_BOOST_VALUE = 1.0f;\n\n  // A signal value so we can tell if something is unset, also used in explanation.\n  public static final int UNSET_SIGNAL_VALUE = -999;\n\n  //This is somewhat arbitrary, and is here so that we have some limit on\n  //how many offline experimental features we support per query\n  public static final int MAX_OFFLINE_EXPERIMENTAL_FIELDS = 5;\n\n  public enum SkipReason {\n    NOT_SKIPPED,\n    ANTIGAMING,\n    LOW_REPUTATION,\n    LOW_TEXT_SCORE,\n    LOW_RETWEET_COUNT,\n    LOW_FAV_COUNT,\n    SOCIAL_FILTER,\n    LOW_FINAL_SCORE\n  }\n\n  // When you add fields here, make sure you also update the clear() function.\n  public double luceneScore;\n  public double textScore;\n  //I am not sure why this has to be double...\n  public double tokenAt140DividedByNumTokensBucket;\n  public double userRep;\n  public double parusScore;\n  public final double[] offlineExpFeatureValues = new double[MAX_OFFLINE_EXPERIMENTAL_FIELDS];\n\n  // v1 engagement counters\n  public double retweetCountPostLog2;\n  public double favCountPostLog2;\n  public double replyCountPostLog2;\n  public double embedsImpressionCount;\n  public double embedsUrlCount;\n  public double videoViewCount;\n\n  // v2 engagement counters (that have a v1 counter part)\n  public double retweetCountV2;\n  public double favCountV2;\n  public double replyCountV2;\n  public double embedsImpressionCountV2;\n  public double embedsUrlCountV2;\n  public double videoViewCountV2;\n  // pure v2 engagement counters, they started v2 only\n  public double quotedCount;\n  public double weightedRetweetCount;\n  public double weightedReplyCount;\n  public double weightedFavCount;\n  public double weightedQuoteCount;\n\n  // card related properties\n  public boolean hasCard;\n  public byte cardType;\n\n  public boolean hasUrl;\n  public boolean isReply;\n  public boolean isRetweet;\n  public boolean isOffensive;\n  public boolean hasTrend;\n  public boolean isFromVerifiedAccount;\n  public boolean isFromBlueVerifiedAccount;\n  public boolean isUserSpam;\n  public boolean isUserNSFW;\n  public boolean isUserBot;\n  public boolean isUserAntiSocial;\n  public boolean hasVisibleLink;\n\n  public double luceneContrib;\n  public double reputationContrib;\n  public double textScoreContrib;\n  public double favContrib;\n  public double replyContrib;\n  public double multipleReplyContrib;\n  public double retweetContrib;\n  public double parusContrib;\n  public final double[] offlineExpFeatureContributions =\n      new double[MAX_OFFLINE_EXPERIMENTAL_FIELDS];\n  public double embedsImpressionContrib;\n  public double embedsUrlContrib;\n  public double videoViewContrib;\n  public double quotedContrib;\n\n  public double hasUrlContrib;\n  public double isReplyContrib;\n  public double isFollowRetweetContrib;\n  public double isTrustedRetweetContrib;\n\n  // Value passed in the request (ThriftRankingParams.querySpecificScoreAdjustments)\n  public double querySpecificScore;\n\n  // Value passed in the request (ThriftRankingParams.authorSpecificScoreAdjustments)\n  public double authorSpecificScore;\n\n  public double normalizedLuceneScore;\n\n  public int tweetLangId;\n  public double uiLangMult;\n  public double userLangMult;\n  public boolean hasDifferentLang;\n  public boolean hasEnglishTweetAndDifferentUILang;\n  public boolean hasEnglishUIAndDifferentTweetLang;\n\n  public int tweetAgeInSeconds;\n  public double ageDecayMult;\n\n  // Intermediate scores\n  public double scoreBeforeBoost;\n  public double scoreAfterBoost;\n  public double scoreFinal;\n  public double scoreReturned;\n\n  public SkipReason skipReason;\n\n  public boolean isTrusted;\n  public boolean isFollow;\n  public boolean spamUserDampApplied;\n  public boolean nsfwUserDampApplied;\n  public boolean botUserDampApplied;\n  public boolean trustedCircleBoostApplied;\n  public boolean directFollowBoostApplied;\n  public boolean outOfNetworkReplyPenaltyApplied;\n  public boolean hasMultipleHashtagsOrTrends;\n\n  public boolean tweetHasTrendsBoostApplied;\n  public boolean tweetFromVerifiedAccountBoostApplied;\n  public boolean tweetFromBlueVerifiedAccountBoostApplied;\n  public boolean hasCardBoostApplied;\n  public boolean cardDomainMatchBoostApplied;\n  public boolean cardAuthorMatchBoostApplied;\n  public boolean cardTitleMatchBoostApplied;\n  public boolean cardDescriptionMatchBoostApplied;\n\n  public List<String> hitFields;\n  public boolean hasNoTextHitDemotionApplied;\n  public boolean hasUrlOnlyHitDemotionApplied;\n  public boolean hasNameOnlyHitDemotionApplied;\n  public boolean hasSeparateTextAndNameHitDemotionApplied;\n  public boolean hasSeparateTextAndUrlHitDemotionApplied;\n\n  public long fromUserId;\n  // This is actually retweet status ID, or the ID of the original tweet being (natively) retweeted\n  public long sharedStatusId;\n  public long referenceAuthorId; // SEARCH-8564\n\n  public boolean isSelfTweet;\n  public boolean selfTweetBoostApplied;\n  public double selfTweetMult;\n\n  public boolean hasImageUrl;\n  public boolean hasVideoUrl;\n  public boolean hasMedialUrlBoostApplied;\n  public boolean hasNewsUrl;\n  public boolean hasNewsUrlBoostApplied;\n\n  public boolean hasConsumerVideo;\n  public boolean hasProVideo;\n  public boolean hasVine;\n  public boolean hasPeriscope;\n  public boolean hasNativeImage;\n  public boolean isNullcast;\n  public boolean hasQuote;\n\n  public boolean isSensitiveContent;\n  public boolean hasMultipleMediaFlag;\n  public boolean profileIsEggFlag;\n  public boolean isUserNewFlag;\n\n  public int numMentions;\n  public int numHashtags;\n  public int linkLanguage;\n  public int prevUserTweetEngagement;\n\n  public boolean isComposerSourceCamera;\n\n  // health model scores by HML\n  public double toxicityScore; // go/toxicity\n  public double pBlockScore; // go/pblock\n  public double pSpammyTweetScore; // go/pspammytweet\n  public double pReportedTweetScore; // go/preportedtweet\n  public double spammyTweetContentScore; // go/spammy-tweet-content\n  public double experimentalHealthModelScore1;\n  public double experimentalHealthModelScore2;\n  public double experimentalHealthModelScore3;\n  public double experimentalHealthModelScore4;\n\n  public LinearScoringData() {\n    hitFields = Lists.newArrayList();\n    clear();\n  }\n\n  // the following three counters were added later and they got denormalized in standard way,\n  // you can choose to apply scalding (for legacy LinearScoringFunction) or\n  // not apply (for returning in metadata and display in debug).\n  public double getEmbedsImpressionCount(boolean scaleForScoring) {\n    return scaleForScoring ? logWith0(embedsImpressionCount) : embedsImpressionCount;\n  }\n  public double getEmbedsUrlCount(boolean scaleForScoring) {\n    return scaleForScoring ? logWith0(embedsUrlCount) : embedsUrlCount;\n  }\n  public double getVideoViewCount(boolean scaleForScoring) {\n    return scaleForScoring ? logWith0(videoViewCount) : videoViewCount;\n  }\n  private static double logWith0(double value) {\n    return value > 0 ? Math.log(value) : 0.0;\n  }\n\n  /**\n   * Returns a string description of all data stored in this instance.\n   */\n  public String getPropertyExplanation() {\n    StringBuilder sb = new StringBuilder();\n    sb.append(hasCard ? \"CARD \" + SearchCardType.cardTypeFromByteValue(cardType) : \"\");\n    sb.append(hasUrl ? \"URL \" : \"\");\n    sb.append(isReply ? \"REPLY \" : \"\");\n    sb.append(isRetweet ? \"RETWEET \" : \"\");\n    sb.append(isOffensive ? \"OFFENSIVE \" : \"\");\n    sb.append(hasTrend ? \"TREND \" : \"\");\n    sb.append(hasMultipleHashtagsOrTrends ? \"HASHTAG/TREND+ \" : \"\");\n    sb.append(isFromVerifiedAccount ? \"VERIFIED \" : \"\");\n    sb.append(isFromBlueVerifiedAccount ? \"BLUE_VERIFIED \" : \"\");\n    sb.append(isUserSpam ? \"SPAM \" : \"\");\n    sb.append(isUserNSFW ? \"NSFW \" : \"\");\n    sb.append(isUserBot ? \"BOT \" : \"\");\n    sb.append(isUserAntiSocial ? \"ANTISOCIAL \" : \"\");\n    sb.append(isTrusted ? \"TRUSTED \" : \"\");\n    sb.append(isFollow ? \"FOLLOW \" : \"\");\n    sb.append(isSelfTweet ? \"SELF \" : \"\");\n    sb.append(hasImageUrl ? \"IMAGE \" : \"\");\n    sb.append(hasVideoUrl ? \"VIDEO \" : \"\");\n    sb.append(hasNewsUrl ? \"NEWS \" : \"\");\n    sb.append(isNullcast ? \"NULLCAST\" : \"\");\n    sb.append(hasQuote ? \"QUOTE\" : \"\");\n    sb.append(isComposerSourceCamera ? \"Composer Source: CAMERA\" : \"\");\n    sb.append(favCountPostLog2 > 0 ? \"Faves:\" + favCountPostLog2 + \" \" : \"\");\n    sb.append(retweetCountPostLog2 > 0 ? \"Retweets:\" + retweetCountPostLog2 + \" \" : \"\");\n    sb.append(replyCountPostLog2 > 0 ? \"Replies:\" + replyCountPostLog2 + \" \" : \"\");\n    sb.append(getEmbedsImpressionCount(false) > 0\n        ? \"Embedded Imps:\" + getEmbedsImpressionCount(false) + \" \" : \"\");\n    sb.append(getEmbedsUrlCount(false) > 0\n        ? \"Embedded Urls:\" + getEmbedsUrlCount(false) + \" \" : \"\");\n    sb.append(getVideoViewCount(false) > 0\n        ? \"Video views:\" + getVideoViewCount(false) + \" \" : \"\");\n    sb.append(weightedRetweetCount > 0 ? \"Weighted Retweets:\"\n        + ((int) weightedRetweetCount) + \" \" : \"\");\n    sb.append(weightedReplyCount > 0\n        ? \"Weighted Replies:\" + ((int) weightedReplyCount) + \" \" : \"\");\n    sb.append(weightedFavCount > 0\n        ? \"Weighted Faves:\" + ((int) weightedFavCount) + \" \" : \"\");\n    sb.append(weightedQuoteCount > 0\n        ? \"Weighted Quotes:\" + ((int) weightedQuoteCount) + \" \" : \"\");\n    return sb.toString();\n  }\n\n  /**\n   * Resets all data stored in this instance.\n   */\n  public void clear() {\n    luceneScore = UNSET_SIGNAL_VALUE;\n    textScore = UNSET_SIGNAL_VALUE;\n    tokenAt140DividedByNumTokensBucket = UNSET_SIGNAL_VALUE;\n    userRep = UNSET_SIGNAL_VALUE;\n    retweetCountPostLog2 = UNSET_SIGNAL_VALUE;\n    favCountPostLog2 = UNSET_SIGNAL_VALUE;\n    replyCountPostLog2 = UNSET_SIGNAL_VALUE;\n    parusScore = UNSET_SIGNAL_VALUE;\n    Arrays.fill(offlineExpFeatureValues, 0);\n    embedsImpressionCount = UNSET_SIGNAL_VALUE;\n    embedsUrlCount = UNSET_SIGNAL_VALUE;\n    videoViewCount = UNSET_SIGNAL_VALUE;\n    // v2 engagement, these each have a v1 counterpart\n    retweetCountV2 = UNSET_SIGNAL_VALUE;\n    favCountV2 = UNSET_SIGNAL_VALUE;\n    replyCountV2 = UNSET_SIGNAL_VALUE;\n    embedsImpressionCountV2 = UNSET_SIGNAL_VALUE;\n    embedsUrlCountV2 = UNSET_SIGNAL_VALUE;\n    videoViewCountV2 = UNSET_SIGNAL_VALUE;\n    // new engagement counters, they only have one version with the v2 normalizer\n    quotedCount = UNSET_SIGNAL_VALUE;\n    weightedRetweetCount = UNSET_SIGNAL_VALUE;\n    weightedReplyCount = UNSET_SIGNAL_VALUE;\n    weightedFavCount = UNSET_SIGNAL_VALUE;\n    weightedQuoteCount = UNSET_SIGNAL_VALUE;\n\n    hasUrl = false;\n    isReply = false;\n    isRetweet = false;\n    isOffensive = false;\n    hasTrend = false;\n    isFromVerifiedAccount = false;\n    isFromBlueVerifiedAccount = false;\n    isUserSpam = false;\n    isUserNSFW = false;\n    isUserBot = false;\n    isUserAntiSocial = false;\n    hasVisibleLink = false;\n    isNullcast = false;\n\n    luceneContrib = UNSET_SIGNAL_VALUE;\n    reputationContrib = UNSET_SIGNAL_VALUE;\n    textScoreContrib = UNSET_SIGNAL_VALUE;\n    replyContrib = UNSET_SIGNAL_VALUE;\n    multipleReplyContrib = UNSET_SIGNAL_VALUE;\n    retweetContrib = UNSET_SIGNAL_VALUE;\n    favContrib = UNSET_SIGNAL_VALUE;\n    parusContrib = UNSET_SIGNAL_VALUE;\n    Arrays.fill(offlineExpFeatureContributions, 0);\n    embedsImpressionContrib = UNSET_SIGNAL_VALUE;\n    embedsUrlContrib = UNSET_SIGNAL_VALUE;\n    videoViewContrib = UNSET_SIGNAL_VALUE;\n    hasUrlContrib = UNSET_SIGNAL_VALUE;\n    isReplyContrib = UNSET_SIGNAL_VALUE;\n\n    querySpecificScore = UNSET_SIGNAL_VALUE;\n    authorSpecificScore = UNSET_SIGNAL_VALUE;\n\n    normalizedLuceneScore = NO_BOOST_VALUE;\n\n    tweetLangId = ThriftLanguage.UNKNOWN.getValue();\n    uiLangMult = NO_BOOST_VALUE;\n    userLangMult = NO_BOOST_VALUE;\n    hasDifferentLang = false;\n    hasEnglishTweetAndDifferentUILang = false;\n    hasEnglishUIAndDifferentTweetLang = false;\n\n    tweetAgeInSeconds = 0;\n    ageDecayMult = NO_BOOST_VALUE;\n\n    // Intermediate scores\n    scoreBeforeBoost = UNSET_SIGNAL_VALUE;\n    scoreAfterBoost = UNSET_SIGNAL_VALUE;\n    scoreFinal = UNSET_SIGNAL_VALUE;\n    scoreReturned = UNSET_SIGNAL_VALUE;\n\n    skipReason = SkipReason.NOT_SKIPPED;\n\n    isTrusted = false;  // Set later\n    isFollow = false; // Set later\n    trustedCircleBoostApplied = false;\n    directFollowBoostApplied = false;\n    outOfNetworkReplyPenaltyApplied = false;\n    hasMultipleHashtagsOrTrends = false;\n    spamUserDampApplied = false;\n    nsfwUserDampApplied = false;\n    botUserDampApplied = false;\n\n    tweetHasTrendsBoostApplied = false;\n    tweetFromVerifiedAccountBoostApplied = false;\n    tweetFromBlueVerifiedAccountBoostApplied = false;\n\n    fromUserId = UNSET_SIGNAL_VALUE;\n    sharedStatusId = UNSET_SIGNAL_VALUE;\n    referenceAuthorId = UNSET_SIGNAL_VALUE;\n\n    isSelfTweet = false;\n    selfTweetBoostApplied = false;\n    selfTweetMult = NO_BOOST_VALUE;\n\n    trustedCircleBoostApplied = false;\n    directFollowBoostApplied = false;\n\n    hasImageUrl = false;\n    hasVideoUrl = false;\n    hasMedialUrlBoostApplied = false;\n    hasNewsUrl = false;\n    hasNewsUrlBoostApplied = false;\n\n    hasCard = false;\n    cardType = SearchCardType.UNKNOWN.getByteValue();\n    hasCardBoostApplied = false;\n    cardDomainMatchBoostApplied = false;\n    cardAuthorMatchBoostApplied = false;\n    cardTitleMatchBoostApplied = false;\n    cardDescriptionMatchBoostApplied = false;\n\n    hitFields.clear();\n    hasNoTextHitDemotionApplied = false;\n    hasUrlOnlyHitDemotionApplied = false;\n    hasNameOnlyHitDemotionApplied = false;\n    hasSeparateTextAndNameHitDemotionApplied = false;\n    hasSeparateTextAndUrlHitDemotionApplied = false;\n\n    hasConsumerVideo = false;\n    hasProVideo = false;\n    hasVine = false;\n    hasPeriscope = false;\n    hasNativeImage = false;\n\n    isSensitiveContent = false;\n    hasMultipleMediaFlag = false;\n    profileIsEggFlag = false;\n    numMentions = 0;\n    numHashtags = 0;\n    isUserNewFlag = false;\n    linkLanguage = 0;\n    prevUserTweetEngagement = 0;\n\n    isComposerSourceCamera = false;\n\n    // health model scores by HML\n    toxicityScore = UNSET_SIGNAL_VALUE;\n    pBlockScore = UNSET_SIGNAL_VALUE;\n    pSpammyTweetScore = UNSET_SIGNAL_VALUE;\n    pReportedTweetScore = UNSET_SIGNAL_VALUE;\n    spammyTweetContentScore = UNSET_SIGNAL_VALUE;\n    experimentalHealthModelScore1 = UNSET_SIGNAL_VALUE;\n    experimentalHealthModelScore2 = UNSET_SIGNAL_VALUE;\n    experimentalHealthModelScore3 = UNSET_SIGNAL_VALUE;\n    experimentalHealthModelScore4 = UNSET_SIGNAL_VALUE;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/LinearScoringParams.java",
    "content": "package com.twitter.search.earlybird.search.relevance;\n\nimport java.util.Arrays;\nimport java.util.Map;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport com.twitter.search.common.constants.SearchCardType;\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.ranking.thriftjava.ThriftAgeDecayRankingParams;\nimport com.twitter.search.common.ranking.thriftjava.ThriftCardRankingParams;\nimport com.twitter.search.common.ranking.thriftjava.ThriftRankingParams;\nimport com.twitter.search.common.util.lang.ThriftLanguageUtil;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSocialFilterType;\n\n/*\n * The class for all query specific parameters, including the parameters from the relevanceOptions and\n * values that are extracted from the request itself.\n */\npublic class LinearScoringParams {\n\n  public static final double DEFAULT_FEATURE_WEIGHT = 0;\n  public static final double DEFAULT_FEATURE_MIN_VAL = 0;\n  public static final double DEFAULT_NO_BOOST = 1.0;\n  @VisibleForTesting\n  static final SearchCounter NULL_USER_LANGS_KEY =\n      SearchCounter.export(\"linear_scoring_params_null_user_langs_key\");\n\n  public final double luceneWeight;\n  public final double textScoreWeight;\n  public final double textScoreMinVal;\n  public final double retweetWeight;\n  public final double retweetMinVal;\n  public final double favWeight;\n  public final double favMinVal;\n  public final double replyWeight;\n  public final double multipleReplyWeight;\n  public final double multipleReplyMinVal;\n  public final double isReplyWeight;\n  public final double parusWeight;\n  public final double embedsImpressionWeight;\n  public final double embedsUrlWeight;\n  public final double videoViewWeight;\n  public final double quotedCountWeight;\n\n  public final double[] rankingOfflineExpWeights =\n      new double[LinearScoringData.MAX_OFFLINE_EXPERIMENTAL_FIELDS];\n\n  public final boolean applyBoosts;\n\n  // Storing ranking params for cards, avoid using maps for faster lookup\n  public final double[] hasCardBoosts = new double[SearchCardType.values().length];\n  public final double[] cardDomainMatchBoosts = new double[SearchCardType.values().length];\n  public final double[] cardAuthorMatchBoosts = new double[SearchCardType.values().length];\n  public final double[] cardTitleMatchBoosts = new double[SearchCardType.values().length];\n  public final double[] cardDescriptionMatchBoosts = new double[SearchCardType.values().length];\n\n  public final double urlWeight;\n  public final double reputationWeight;\n  public final double reputationMinVal;\n  public final double followRetweetWeight;\n  public final double trustedRetweetWeight;\n\n  // Adjustments for specific tweets (tweetId -> score)\n  public final Map<Long, Double> querySpecificScoreAdjustments;\n\n  // Adjustments for tweets posted by specific authors (userId -> score)\n  public final Map<Long, Double> authorSpecificScoreAdjustments;\n\n  public final double offensiveDamping;\n  public final double spamUserDamping;\n  public final double nsfwUserDamping;\n  public final double botUserDamping;\n  public final double trustedCircleBoost;\n  public final double directFollowBoost;\n  public final double minScore;\n\n  public final boolean applyFiltersAlways;\n\n  public final boolean useLuceneScoreAsBoost;\n  public final double maxLuceneScoreBoost;\n\n  public final double langEnglishTweetDemote;\n  public final double langEnglishUIDemote;\n  public final double langDefaultDemote;\n  public final boolean useUserLanguageInfo;\n  public final double unknownLanguageBoost;\n\n  public final double outOfNetworkReplyPenalty;\n\n  public final boolean useAgeDecay;\n  public final double ageDecayHalflife;\n  public final double ageDecayBase;\n  public final double ageDecaySlope;\n\n  // hit attribute demotions\n  public final boolean enableHitDemotion;\n  public final double noTextHitDemotion;\n  public final double urlOnlyHitDemotion;\n  public final double nameOnlyHitDemotion;\n  public final double separateTextAndNameHitDemotion;\n  public final double separateTextAndUrlHitDemotion;\n\n  // trends related params\n  public final double tweetHasTrendBoost;\n  public final double multipleHashtagsOrTrendsDamping;\n\n  public final double tweetFromVerifiedAccountBoost;\n\n  public final double tweetFromBlueVerifiedAccountBoost;\n\n  public final ThriftSocialFilterType socialFilterType;\n  public final int uiLangId;\n  // Confidences of the understandability of different languages for this user.\n  public final double[] userLangs = new double[ThriftLanguage.values().length];\n\n  public final long searcherId;\n  public final double selfTweetBoost;\n\n  public final double tweetHasMediaUrlBoost;\n  public final double tweetHasNewsUrlBoost;\n\n  // whether we need meta-data for replies what the reply is to.\n  public final boolean getInReplyToStatusId;\n\n  // Initialize from a ranking parameter\n  public LinearScoringParams(ThriftSearchQuery searchQuery, ThriftRankingParams params) {\n    // weights\n    luceneWeight = params.isSetLuceneScoreParams()\n        ? params.getLuceneScoreParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    textScoreWeight = params.isSetTextScoreParams()\n        ? params.getTextScoreParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    retweetWeight = params.isSetRetweetCountParams()\n        ? params.getRetweetCountParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    favWeight = params.isSetFavCountParams()\n        ? params.getFavCountParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    replyWeight = params.isSetReplyCountParams()\n        ? params.getReplyCountParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    multipleReplyWeight = params.isSetMultipleReplyCountParams()\n        ? params.getMultipleReplyCountParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    parusWeight = params.isSetParusScoreParams()\n        ? params.getParusScoreParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    for (int i = 0; i < LinearScoringData.MAX_OFFLINE_EXPERIMENTAL_FIELDS; i++) {\n      Byte featureTypeByte = (byte) i;\n      // default weight is 0, thus contribution for unset feature value will be 0.\n      rankingOfflineExpWeights[i] = params.getOfflineExperimentalFeatureRankingParamsSize() > 0\n          && params.getOfflineExperimentalFeatureRankingParams().containsKey(featureTypeByte)\n              ? params.getOfflineExperimentalFeatureRankingParams().get(featureTypeByte).getWeight()\n              : DEFAULT_FEATURE_WEIGHT;\n    }\n    embedsImpressionWeight = params.isSetEmbedsImpressionCountParams()\n        ? params.getEmbedsImpressionCountParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    embedsUrlWeight = params.isSetEmbedsUrlCountParams()\n        ? params.getEmbedsUrlCountParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    videoViewWeight = params.isSetVideoViewCountParams()\n        ? params.getVideoViewCountParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    quotedCountWeight = params.isSetQuotedCountParams()\n        ? params.getQuotedCountParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n\n    applyBoosts = params.isApplyBoosts();\n\n    // configure card values\n    Arrays.fill(hasCardBoosts, DEFAULT_NO_BOOST);\n    Arrays.fill(cardAuthorMatchBoosts, DEFAULT_NO_BOOST);\n    Arrays.fill(cardDomainMatchBoosts, DEFAULT_NO_BOOST);\n    Arrays.fill(cardTitleMatchBoosts, DEFAULT_NO_BOOST);\n    Arrays.fill(cardDescriptionMatchBoosts, DEFAULT_NO_BOOST);\n    if (params.isSetCardRankingParams()) {\n      for (SearchCardType cardType : SearchCardType.values()) {\n        byte cardTypeIndex = cardType.getByteValue();\n        ThriftCardRankingParams rankingParams = params.getCardRankingParams().get(cardTypeIndex);\n        if (rankingParams != null) {\n          hasCardBoosts[cardTypeIndex] = rankingParams.getHasCardBoost();\n          cardAuthorMatchBoosts[cardTypeIndex] = rankingParams.getAuthorMatchBoost();\n          cardDomainMatchBoosts[cardTypeIndex] = rankingParams.getDomainMatchBoost();\n          cardTitleMatchBoosts[cardTypeIndex] = rankingParams.getTitleMatchBoost();\n          cardDescriptionMatchBoosts[cardTypeIndex] = rankingParams.getDescriptionMatchBoost();\n        }\n      }\n    }\n\n    urlWeight = params.isSetUrlParams()\n        ? params.getUrlParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    reputationWeight = params.isSetReputationParams()\n        ? params.getReputationParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    isReplyWeight = params.isSetIsReplyParams()\n        ? params.getIsReplyParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    followRetweetWeight = params.isSetDirectFollowRetweetCountParams()\n        ? params.getDirectFollowRetweetCountParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n    trustedRetweetWeight = params.isSetTrustedCircleRetweetCountParams()\n        ? params.getTrustedCircleRetweetCountParams().getWeight() : DEFAULT_FEATURE_WEIGHT;\n\n    querySpecificScoreAdjustments = params.getQuerySpecificScoreAdjustments();\n    authorSpecificScoreAdjustments = params.getAuthorSpecificScoreAdjustments();\n\n    // min/max filters\n    textScoreMinVal = params.isSetTextScoreParams()\n        ? params.getTextScoreParams().getMin() : DEFAULT_FEATURE_MIN_VAL;\n    reputationMinVal = params.isSetReputationParams()\n        ? params.getReputationParams().getMin() : DEFAULT_FEATURE_MIN_VAL;\n    multipleReplyMinVal = params.isSetMultipleReplyCountParams()\n        ? params.getMultipleReplyCountParams().getMin() : DEFAULT_FEATURE_MIN_VAL;\n    retweetMinVal = params.isSetRetweetCountParams() && params.getRetweetCountParams().isSetMin()\n        ? params.getRetweetCountParams().getMin() : DEFAULT_FEATURE_MIN_VAL;\n    favMinVal = params.isSetFavCountParams() && params.getFavCountParams().isSetMin()\n        ? params.getFavCountParams().getMin() : DEFAULT_FEATURE_MIN_VAL;\n\n    // boosts\n    spamUserDamping = params.isSetSpamUserBoost() ? params.getSpamUserBoost() : 1.0;\n    nsfwUserDamping = params.isSetNsfwUserBoost() ? params.getNsfwUserBoost() : 1.0;\n    botUserDamping = params.isSetBotUserBoost() ? params.getBotUserBoost() : 1.0;\n    offensiveDamping = params.getOffensiveBoost();\n    trustedCircleBoost = params.getInTrustedCircleBoost();\n    directFollowBoost = params.getInDirectFollowBoost();\n\n    // language boosts\n    langEnglishTweetDemote = params.getLangEnglishTweetBoost();\n    langEnglishUIDemote = params.getLangEnglishUIBoost();\n    langDefaultDemote = params.getLangDefaultBoost();\n    useUserLanguageInfo = params.isUseUserLanguageInfo();\n    unknownLanguageBoost = params.getUnknownLanguageBoost();\n\n    // hit demotions\n    enableHitDemotion = params.isEnableHitDemotion();\n    noTextHitDemotion = params.getNoTextHitDemotion();\n    urlOnlyHitDemotion = params.getUrlOnlyHitDemotion();\n    nameOnlyHitDemotion = params.getNameOnlyHitDemotion();\n    separateTextAndNameHitDemotion = params.getSeparateTextAndNameHitDemotion();\n    separateTextAndUrlHitDemotion = params.getSeparateTextAndUrlHitDemotion();\n\n    outOfNetworkReplyPenalty = params.getOutOfNetworkReplyPenalty();\n\n    if (params.isSetAgeDecayParams()) {\n      // new age decay settings\n      ThriftAgeDecayRankingParams ageDecayParams = params.getAgeDecayParams();\n      ageDecaySlope = ageDecayParams.getSlope();\n      ageDecayHalflife = ageDecayParams.getHalflife();\n      ageDecayBase = ageDecayParams.getBase();\n      useAgeDecay = true;\n    } else if (params.isSetDeprecatedAgeDecayBase()\n        && params.isSetDeprecatedAgeDecayHalflife()\n        && params.isSetDeprecatedAgeDecaySlope()) {\n      ageDecaySlope = params.getDeprecatedAgeDecaySlope();\n      ageDecayHalflife = params.getDeprecatedAgeDecayHalflife();\n      ageDecayBase = params.getDeprecatedAgeDecayBase();\n      useAgeDecay = true;\n    } else {\n      ageDecaySlope = 0.0;\n      ageDecayHalflife = 0.0;\n      ageDecayBase = 0.0;\n      useAgeDecay = false;\n    }\n\n    // trends\n    tweetHasTrendBoost = params.getTweetHasTrendBoost();\n    multipleHashtagsOrTrendsDamping = params.getMultipleHashtagsOrTrendsBoost();\n\n    // verified accounts\n    tweetFromVerifiedAccountBoost = params.getTweetFromVerifiedAccountBoost();\n    tweetFromBlueVerifiedAccountBoost = params.getTweetFromBlueVerifiedAccountBoost();\n\n    // score filter\n    minScore = params.getMinScore();\n\n    applyFiltersAlways = params.isApplyFiltersAlways();\n\n    useLuceneScoreAsBoost = params.isUseLuceneScoreAsBoost();\n    maxLuceneScoreBoost = params.getMaxLuceneScoreBoost();\n\n    searcherId = searchQuery.isSetSearcherId() ? searchQuery.getSearcherId() : -1;\n    selfTweetBoost = params.getSelfTweetBoost();\n\n    socialFilterType = searchQuery.getSocialFilterType();\n\n    // the UI language and the confidences of the languages user can understand.\n    if (!searchQuery.isSetUiLang() || searchQuery.getUiLang().isEmpty()) {\n      uiLangId = ThriftLanguage.UNKNOWN.getValue();\n    } else {\n      uiLangId = ThriftLanguageUtil.getThriftLanguageOf(searchQuery.getUiLang()).getValue();\n    }\n    if (searchQuery.getUserLangsSize() > 0) {\n      for (Map.Entry<ThriftLanguage, Double> lang : searchQuery.getUserLangs().entrySet()) {\n        ThriftLanguage thriftLanguage = lang.getKey();\n        // SEARCH-13441\n        if (thriftLanguage != null) {\n          userLangs[thriftLanguage.getValue()] = lang.getValue();\n        } else {\n          NULL_USER_LANGS_KEY.increment();\n        }\n      }\n    }\n\n    // For now, we will use the same boost for both image, and video.\n    tweetHasMediaUrlBoost = params.getTweetHasImageUrlBoost();\n    tweetHasNewsUrlBoost = params.getTweetHasNewsUrlBoost();\n\n    getInReplyToStatusId =\n        searchQuery.isSetResultMetadataOptions()\n            && searchQuery.getResultMetadataOptions().isSetGetInReplyToStatusId()\n            && searchQuery.getResultMetadataOptions().isGetInReplyToStatusId();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/MinFeatureValueFilter.java",
    "content": "package com.twitter.search.earlybird.search.relevance;\n\nimport java.io.IOException;\nimport java.util.Objects;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.encoding.features.ByteNormalizer;\nimport com.twitter.search.common.encoding.features.ClampByteNormalizer;\nimport com.twitter.search.common.encoding.features.SingleBytePositiveFloatNormalizer;\nimport com.twitter.search.common.query.DefaultFilterWeight;\nimport com.twitter.search.common.query.FilteredQuery;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.index.util.RangeFilterDISI;\n\npublic final class MinFeatureValueFilter extends Query implements FilteredQuery.DocIdFilterFactory {\n  private final String featureName;\n  private final ByteNormalizer normalizer;\n  private final double minValue;\n\n  /**\n   * Creates a query that filters out all hits that have a value smaller than the given threshold\n   * for the given feature.\n   *\n   * @param featureName The feature.\n   * @param minValue The threshold for the feature values.\n   * @return A query that filters out all hits that have a value smaller than the given threshold\n   *         for the given feature.\n   */\n  public static Query getMinFeatureValueFilter(String featureName, double minValue) {\n    return new BooleanQuery.Builder()\n        .add(new MinFeatureValueFilter(featureName, minValue), BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  public static FilteredQuery.DocIdFilterFactory getDocIdFilterFactory(\n      String featureName, double minValue) {\n    return new MinFeatureValueFilter(featureName, minValue);\n  }\n\n  /**\n   * Returns the normalizer that should be used to normalize the values for the given feature.\n   *\n   * @param featureName The feature.\n   * @return The normalizer that should be used to normalize the values for the given feature.\n   */\n  @VisibleForTesting\n  public static ByteNormalizer getMinFeatureValueNormalizer(String featureName) {\n    if (featureName.equals(EarlybirdFieldConstant.USER_REPUTATION.getFieldName())) {\n      return new ClampByteNormalizer(0, 100);\n    }\n\n    if (featureName.equals(EarlybirdFieldConstant.FAVORITE_COUNT.getFieldName())\n        || featureName.equals(EarlybirdFieldConstant.PARUS_SCORE.getFieldName())\n        || featureName.equals(EarlybirdFieldConstant.REPLY_COUNT.getFieldName())\n        || featureName.equals(EarlybirdFieldConstant.RETWEET_COUNT.getFieldName())) {\n      return new SingleBytePositiveFloatNormalizer();\n    }\n\n    throw new IllegalArgumentException(\"Unknown normalization method for field \" + featureName);\n  }\n\n  @Override\n  public int hashCode() {\n    // Probably doesn't make sense to include the schemaSnapshot and normalizer here.\n    return (int) ((featureName == null ? 0 : featureName.hashCode() * 7) + minValue);\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof MinFeatureValueFilter)) {\n      return false;\n    }\n\n    // Probably doesn't make sense to include the schemaSnapshot and normalizer here.\n    MinFeatureValueFilter filter = MinFeatureValueFilter.class.cast(obj);\n    return Objects.equals(featureName, filter.featureName) && (minValue == filter.minValue);\n  }\n\n  @Override\n  public String toString(String field) {\n    return String.format(\"MinFeatureValueFilter(%s, %f)\", featureName, minValue);\n  }\n\n  private MinFeatureValueFilter(String featureName, double minValue) {\n    this.featureName = featureName;\n    this.normalizer = getMinFeatureValueNormalizer(featureName);\n    this.minValue = normalizer.normalize(minValue);\n  }\n\n  @Override\n  public FilteredQuery.DocIdFilter getDocIdFilter(LeafReaderContext context) throws IOException {\n    final NumericDocValues featureDocValues = context.reader().getNumericDocValues(featureName);\n    return (docId) -> featureDocValues.advanceExact(docId)\n        && ((byte) featureDocValues.longValue() >= minValue);\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) {\n    return new DefaultFilterWeight(this) {\n      @Override\n      protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context) throws IOException {\n        return new MinFeatureValueDocIdSetIterator(\n            context.reader(), featureName, minValue);\n      }\n    };\n  }\n\n  private static final class MinFeatureValueDocIdSetIterator extends RangeFilterDISI {\n    private final NumericDocValues featureDocValues;\n    private final double minValue;\n\n    MinFeatureValueDocIdSetIterator(LeafReader indexReader,\n                                    String featureName,\n                                    double minValue) throws IOException {\n      super(indexReader);\n      this.featureDocValues = indexReader.getNumericDocValues(featureName);\n      this.minValue = minValue;\n    }\n\n    @Override\n    public boolean shouldReturnDoc() throws IOException {\n      // We need this explicit casting to byte, because of how we encode and decode features in our\n      // encoded_tweet_features field. If a feature is an int (uses all 32 bits of the int), then\n      // encoding the feature and then decoding it preserves its original value. However, if the\n      // feature does not use the entire int (and especially if it uses bits somewhere in the middle\n      // of the int), then the feature value is assumed to be unsigned when it goes through this\n      // process of encoding and decoding. So a user rep of\n      // RelevanceSignalConstants.UNSET_REPUTATION_SENTINEL (-128) will be correctly encoded as the\n      // binary value 10000000, but will be treated as an unsigned value when decoded, and therefore\n      // the decoded value will be 128.\n      //\n      // In retrospect, this seems like a really poor design decision. It seems like it would be\n      // better if all feature values were considered to be signed, even if most features can never\n      // have negative values. Unfortunately, making this change is not easy, because some features\n      // store normalized values, so we would also need to change the range of allowed values\n      // produced by those normalizers, as well as all code that depends on those values.\n      //\n      // So for now, just cast this value to a byte, to get the proper negative value.\n      return featureDocValues.advanceExact(docID())\n          && ((byte) featureDocValues.longValue() >= minValue);\n    }\n  }\n\n  public double getMinValue() {\n    return minValue;\n  }\n\n  public ByteNormalizer getNormalizer() {\n    return normalizer;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/RelevanceHit.java",
    "content": "package com.twitter.search.earlybird.search.relevance;\n\nimport java.util.Comparator;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common_internal.collections.RandomAccessPriorityQueue;\nimport com.twitter.search.common.relevance.features.TweetIntegerShingleSignature;\nimport com.twitter.search.earlybird.search.Hit;\nimport com.twitter.search.earlybird.search.relevance.scoring.ScoringFunction;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\n\npublic class RelevanceHit extends Hit\n    implements RandomAccessPriorityQueue.SignatureProvider<TweetIntegerShingleSignature> {\n  @Nullable\n  private TweetIntegerShingleSignature signature;\n\n  public RelevanceHit() {\n    super(Long.MAX_VALUE, Long.MAX_VALUE);\n  }\n\n  public RelevanceHit(long timeSliceID, long statusID,\n                      TweetIntegerShingleSignature signature,\n                      ThriftSearchResultMetadata metadata) {\n    super(timeSliceID, statusID);\n    update(timeSliceID, statusID, signature, metadata);\n  }\n\n  /**\n   * Updates the data for this relevance hit.\n   *\n   * @param timeSliceID The timeslice ID of the segment that the segment came from.\n   * @param statusID The hit's tweet ID.\n   * @param tweetSignature The tweet signature generated for this hit.\n   * @param metadata The metadata associated with this hit.\n   */\n  public void update(long timeSliceID, long statusID, TweetIntegerShingleSignature tweetSignature,\n      ThriftSearchResultMetadata metadata) {\n    this.statusID = statusID;\n    this.timeSliceID = timeSliceID;\n    this.metadata = Preconditions.checkNotNull(metadata);\n    this.signature = Preconditions.checkNotNull(tweetSignature);\n  }\n\n  /**\n   * Returns the computed score for this hit.\n   */\n  public float getScore() {\n    if (metadata != null) {\n      return (float) metadata.getScore();\n    } else {\n      return ScoringFunction.SKIP_HIT;\n    }\n  }\n\n  // We want the score as a double (and not cast to a float) for COMPARATOR_BY_SCORE and\n  // PQ_COMPARATOR_BY_SCORE so that the results returned from Earlybirds will be sorted based on the\n  // scores in the ThriftSearchResultMetadata objects (and will not lose precision by being cast to\n  // floats). Thus, the sorted order on Earlybirds and Earlybird Roots will be consistent.\n  private double getScoreDouble() {\n    if (metadata != null) {\n      return metadata.getScore();\n    } else {\n      return (double) ScoringFunction.SKIP_HIT;\n    }\n  }\n\n  @Override @Nullable\n  public TweetIntegerShingleSignature getSignature() {\n    return signature;\n  }\n\n  @Override\n  public String toString() {\n    return \"RelevanceHit[tweetID=\" + statusID + \",timeSliceID=\" + timeSliceID\n        + \",score=\" + (metadata == null ? \"null\" : metadata.getScore())\n        + \",signature=\" + (signature == null ? \"null\" : signature) + \"]\";\n  }\n\n  public static final Comparator<RelevanceHit> COMPARATOR_BY_SCORE =\n      (d1, d2) -> {\n        // if two docs have the same score, then the first one (most recent) wins\n        if (d1.getScore() == d2.getScore()) {\n          return Long.compare(d2.getStatusID(), d1.getStatusID());\n        }\n        return Double.compare(d2.getScoreDouble(), d1.getScoreDouble());\n      };\n\n  public static final Comparator<RelevanceHit> PQ_COMPARATOR_BY_SCORE =\n      (d1, d2) -> {\n        // Reverse the order\n        return COMPARATOR_BY_SCORE.compare(d2, d1);\n      };\n\n  @Override\n  public void clear() {\n    timeSliceID = Long.MAX_VALUE;\n    statusID = Long.MAX_VALUE;\n    metadata = null;\n    signature = null;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/RelevanceSearchRequestInfo.java",
    "content": "package com.twitter.search.earlybird.search.relevance;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.search.Query;\n\nimport com.twitter.search.common.search.TerminationTracker;\nimport com.twitter.search.earlybird.QualityFactor;\nimport com.twitter.search.earlybird.search.SearchRequestInfo;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchRelevanceOptions;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadataOptions;\n\npublic class RelevanceSearchRequestInfo extends SearchRequestInfo {\n  private final ThriftSearchRelevanceOptions relevanceOptions;\n\n  public RelevanceSearchRequestInfo(\n      ThriftSearchQuery searchQuery, Query query,\n      TerminationTracker terminationTracker, QualityFactor qualityFactor) {\n    super(addResultMetadataOptionsIfUnset(searchQuery), query, terminationTracker, qualityFactor);\n    this.relevanceOptions = searchQuery.getRelevanceOptions();\n  }\n\n  private static ThriftSearchQuery addResultMetadataOptionsIfUnset(ThriftSearchQuery searchQuery) {\n    if (!searchQuery.isSetResultMetadataOptions()) {\n      searchQuery.setResultMetadataOptions(new ThriftSearchResultMetadataOptions());\n    }\n    return searchQuery;\n  }\n\n  @Override\n  protected int calculateMaxHitsToProcess(ThriftSearchQuery thriftSearchQuery) {\n    ThriftSearchRelevanceOptions searchRelevanceOptions = thriftSearchQuery.getRelevanceOptions();\n\n    // Don't use the value from the ThriftSearchQuery object if one is provided in the\n    // relevance options\n    int requestedMaxHitsToProcess = searchRelevanceOptions.isSetMaxHitsToProcess()\n        ? searchRelevanceOptions.getMaxHitsToProcess()\n        : super.calculateMaxHitsToProcess(thriftSearchQuery);\n\n    return qualityFactorMaxHitsToProcess(getNumResultsRequested(), requestedMaxHitsToProcess);\n  }\n\n  public ThriftSearchRelevanceOptions getRelevanceOptions() {\n    return this.relevanceOptions;\n  }\n\n  /**\n   * Reduces maxHitsToProcess based on quality factor. Never reduces it beyond\n   * numResults.\n   * @param numResults\n   * @param maxHitsToProcess\n   * @return Reduced maxHitsToProcess.\n   */\n  public int qualityFactorMaxHitsToProcess(int numResults, int maxHitsToProcess) {\n    Preconditions.checkNotNull(qualityFactor);\n\n    // Do not quality factor if there is no lower bound on maxHitsToProcess.\n    if (numResults > maxHitsToProcess) {\n      return maxHitsToProcess;\n    }\n\n    double currentQualityFactor = qualityFactor.get();\n    return Math.max(numResults, (int) (currentQualityFactor * maxHitsToProcess));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/RelevanceSearchResults.java",
    "content": "package com.twitter.search.earlybird.search.relevance;\n\nimport com.twitter.search.earlybird.search.Hit;\nimport com.twitter.search.earlybird.search.SimpleSearchResults;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultsRelevanceStats;\n\npublic class RelevanceSearchResults extends SimpleSearchResults {\n  public final ThriftSearchResultMetadata[] resultMetadata;\n  private ThriftSearchResultsRelevanceStats relevanceStats = null;\n  private long scoringTimeNanos = 0;\n\n  public RelevanceSearchResults(int size) {\n    super(size);\n    this.resultMetadata = new ThriftSearchResultMetadata[size];\n  }\n\n  public void setHit(Hit hit, int hitIndex) {\n    hits[hitIndex] = hit;\n    resultMetadata[hitIndex] = hit.getMetadata();\n  }\n\n  public void setRelevanceStats(ThriftSearchResultsRelevanceStats relevanceStats) {\n    this.relevanceStats = relevanceStats;\n  }\n  public ThriftSearchResultsRelevanceStats getRelevanceStats() {\n    return relevanceStats;\n  }\n\n  public void setScoringTimeNanos(long scoringTimeNanos) {\n    this.scoringTimeNanos = scoringTimeNanos;\n  }\n\n  public long getScoringTimeNanos() {\n    return scoringTimeNanos;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/ScoreFilterQuery.java",
    "content": "package com.twitter.search.earlybird.search.relevance;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.index.LeafReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.search.BooleanClause;\nimport org.apache.lucene.search.BooleanQuery;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\n\nimport com.twitter.search.common.query.DefaultFilterWeight;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.util.RangeFilterDISI;\nimport com.twitter.search.earlybird.search.relevance.scoring.ScoringFunction;\nimport com.twitter.search.earlybird.search.relevance.scoring.ScoringFunctionProvider;\nimport com.twitter.search.earlybird.search.relevance.scoring.ScoringFunctionProvider.NamedScoringFunctionProvider;\n\n/**\n * This filter only accepts documents for which the provided\n * {@link com.twitter.search.earlybird.search.relevance.scoring.ScoringFunction}\n * returns a score that's greater or equal to the passed-in minScore and smaller or equal\n * to maxScore.\n */\npublic final class ScoreFilterQuery extends Query {\n  private static final float DEFAULT_LUCENE_SCORE = 1.0F;\n\n  private final float minScore;\n  private final float maxScore;\n  private final NamedScoringFunctionProvider scoringFunctionProvider;\n  private final ImmutableSchemaInterface schema;\n\n  /**\n   * Returns a score filter.\n   *\n   * @param schema The schema to use to extract the feature scores.\n   * @param scoringFunctionProvider The scoring function provider.\n   * @param minScore The minimum score threshold.\n   * @param maxScore The maximum score threshold.\n   * @return A score filter with the given configuration.\n   */\n  public static Query getScoreFilterQuery(\n      ImmutableSchemaInterface schema,\n      NamedScoringFunctionProvider scoringFunctionProvider,\n      float minScore,\n      float maxScore) {\n    return new BooleanQuery.Builder()\n        .add(new ScoreFilterQuery(schema, scoringFunctionProvider, minScore, maxScore),\n             BooleanClause.Occur.FILTER)\n        .build();\n  }\n\n  private ScoreFilterQuery(ImmutableSchemaInterface schema,\n                           NamedScoringFunctionProvider scoringFunctionProvider,\n                           float minScore,\n                           float maxScore) {\n    this.schema = schema;\n    this.scoringFunctionProvider = scoringFunctionProvider;\n    this.minScore = minScore;\n    this.maxScore = maxScore;\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost)\n      throws IOException {\n    return new DefaultFilterWeight(this) {\n      @Override\n      protected DocIdSetIterator getDocIdSetIterator(LeafReaderContext context) throws IOException {\n        ScoringFunction scoringFunction = scoringFunctionProvider.getScoringFunction();\n        scoringFunction.setNextReader((EarlybirdIndexSegmentAtomicReader) context.reader());\n        return new ScoreFilterDocIdSetIterator(\n            context.reader(), scoringFunction, minScore, maxScore);\n      }\n    };\n  }\n\n  private static final class ScoreFilterDocIdSetIterator extends RangeFilterDISI {\n    private final ScoringFunction scoringFunction;\n    private final float minScore;\n    private final float maxScore;\n\n    public ScoreFilterDocIdSetIterator(LeafReader indexReader, ScoringFunction scoringFunction,\n                                       float minScore, float maxScore) throws IOException {\n      super(indexReader);\n      this.scoringFunction = scoringFunction;\n      this.minScore = minScore;\n      this.maxScore = maxScore;\n    }\n\n    @Override\n    protected boolean shouldReturnDoc() throws IOException {\n      float score = scoringFunction.score(docID(), DEFAULT_LUCENE_SCORE);\n      return score >= minScore && score <= maxScore;\n    }\n  }\n\n  public float getMinScoreForTest() {\n    return minScore;\n  }\n\n  public float getMaxScoreForTest() {\n    return maxScore;\n  }\n\n  public ScoringFunctionProvider getScoringFunctionProviderForTest() {\n    return scoringFunctionProvider;\n  }\n\n  @Override\n  public int hashCode() {\n    return (int) (minScore * 29\n                  + maxScore * 17\n                  + (scoringFunctionProvider == null ? 0 : scoringFunctionProvider.hashCode()));\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof ScoreFilterQuery)) {\n      return false;\n    }\n\n    ScoreFilterQuery filter = ScoreFilterQuery.class.cast(obj);\n    return (minScore == filter.minScore)\n        && (maxScore == filter.maxScore)\n        && (scoringFunctionProvider == null\n            ? filter.scoringFunctionProvider == null\n            : scoringFunctionProvider.equals(filter.scoringFunctionProvider));\n  }\n\n  @Override\n  public String toString(String field) {\n    return \"SCORE_FILTER_QUERY[minScore=\" + minScore + \",maxScore=\" + maxScore + \"]\";\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/collectors/AbstractRelevanceCollector.java",
    "content": "package com.twitter.search.earlybird.search.relevance.collectors;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.facets.LanguageHistogram;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.search.AbstractResultsCollector;\nimport com.twitter.search.earlybird.search.relevance.RelevanceSearchRequestInfo;\nimport com.twitter.search.earlybird.search.relevance.RelevanceSearchResults;\nimport com.twitter.search.earlybird.search.relevance.scoring.ScoringFunction;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadataOptions;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultsRelevanceStats;\n\n/**\n * AbstractRelevanceCollector is a results collector that collects RelevanceHit results\n * which include more detailed information than a normal Hit.\n */\npublic abstract class AbstractRelevanceCollector\n    extends AbstractResultsCollector<RelevanceSearchRequestInfo, RelevanceSearchResults> {\n  protected final ScoringFunction scoringFunction;\n  private final ThriftSearchResultsRelevanceStats relevanceStats;\n  private final EarlybirdCluster cluster;\n  private final UserTable userTable;\n\n  // Per-language result counts.\n  private final LanguageHistogram languageHistogram = new LanguageHistogram();\n\n  // Accumulated time spend on relevance scoring across all collected hits, including batch scoring.\n  private long scoringTimeNanos = 0;\n\n  public AbstractRelevanceCollector(\n      ImmutableSchemaInterface schema,\n      RelevanceSearchRequestInfo searchRequestInfo,\n      ScoringFunction scoringFunction,\n      EarlybirdSearcherStats searcherStats,\n      EarlybirdCluster cluster,\n      UserTable userTable,\n      Clock clock,\n      int requestDebugMode) {\n    super(schema, searchRequestInfo, clock, searcherStats, requestDebugMode);\n    this.scoringFunction = scoringFunction;\n    this.relevanceStats = new ThriftSearchResultsRelevanceStats();\n    this.cluster = cluster;\n    this.userTable = userTable;\n  }\n\n  /**\n   * Subclasses must implement this method to actually collect a scored relevance hit.\n   */\n  protected abstract void doCollectWithScore(long tweetID, float score) throws IOException;\n\n  @Override\n  public final void startSegment() throws IOException {\n    scoringFunction.setNextReader(currTwitterReader);\n\n    ThriftSearchResultMetadataOptions options =\n        searchRequestInfo.getSearchQuery().getResultMetadataOptions();\n    featuresRequested = options != null && options.isReturnSearchResultFeatures();\n  }\n\n  @Override\n  protected final void doCollect(long tweetID) throws IOException {\n    final long scoringStartNanos = getClock().nowNanos();\n    float luceneSore = scorer.score();\n    final float score = scoringFunction.score(curDocId, luceneSore);\n    final long scoringEndNanos = getClock().nowNanos();\n    addToOverallScoringTimeNanos(scoringStartNanos, scoringEndNanos);\n\n    scoringFunction.updateRelevanceStats(relevanceStats);\n\n    updateHitCounts(tweetID);\n\n    doCollectWithScore(tweetID, score);\n  }\n\n  protected final void addToOverallScoringTimeNanos(long scoringStartNanos, long scoringEndNanos) {\n    scoringTimeNanos += scoringEndNanos - scoringStartNanos;\n  }\n\n  protected final ThriftSearchResultMetadata collectMetadata() throws IOException {\n    ThriftSearchResultMetadataOptions options =\n        searchRequestInfo.getSearchQuery().getResultMetadataOptions();\n    Preconditions.checkNotNull(options);\n    ThriftSearchResultMetadata metadata =\n        Preconditions.checkNotNull(scoringFunction.getResultMetadata(options));\n    if (metadata.isSetLanguage()) {\n      languageHistogram.increment(metadata.getLanguage().getValue());\n    }\n\n    // Some additional metadata which is not provided by the scoring function, but\n    // by accessing the reader directly.\n    if (currTwitterReader != null) {\n      fillResultGeoLocation(metadata);\n      if (searchRequestInfo.isCollectConversationId()) {\n        long conversationId =\n            documentFeatures.getFeatureValue(EarlybirdFieldConstant.CONVERSATION_ID_CSF);\n        if (conversationId != 0) {\n          ensureExtraMetadataIsSet(metadata);\n          metadata.getExtraMetadata().setConversationId(conversationId);\n        }\n      }\n    }\n\n    // Check and collect hit attribution data, if it's available.\n    fillHitAttributionMetadata(metadata);\n\n    long fromUserId = documentFeatures.getFeatureValue(EarlybirdFieldConstant.FROM_USER_ID_CSF);\n    if (searchRequestInfo.isGetFromUserId()) {\n      metadata.setFromUserId(fromUserId);\n    }\n\n    collectExclusiveConversationAuthorId(metadata);\n    collectFacets(metadata);\n    collectFeatures(metadata);\n    collectIsProtected(metadata, cluster, userTable);\n\n    return metadata;\n  }\n\n  protected final ThriftSearchResultsRelevanceStats getRelevanceStats() {\n    return relevanceStats;\n  }\n\n  public final LanguageHistogram getLanguageHistogram() {\n    return languageHistogram;\n  }\n\n  @Override\n  protected final RelevanceSearchResults doGetResults() throws IOException {\n    final RelevanceSearchResults results = doGetRelevanceResults();\n    results.setScoringTimeNanos(scoringTimeNanos);\n    return results;\n  }\n\n  /**\n   * For subclasses to process and aggregate collected hits.\n   */\n  protected abstract RelevanceSearchResults doGetRelevanceResults() throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/collectors/BatchRelevanceTopCollector.java",
    "content": "package com.twitter.search.earlybird.search.relevance.collectors;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.concurrent.TimeUnit;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.features.thrift.ThriftSearchResultFeatures;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.search.EarlyTerminationState;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.search.relevance.LinearScoringData;\nimport com.twitter.search.earlybird.search.relevance.RelevanceSearchRequestInfo;\nimport com.twitter.search.earlybird.search.relevance.RelevanceSearchResults;\nimport com.twitter.search.earlybird.search.relevance.scoring.BatchHit;\nimport com.twitter.search.earlybird.search.relevance.scoring.ScoringFunction;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.ThriftSearchRelevanceOptions;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultExtraMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\n\n/**\n * BatchRelevanceTopCollector is similar to the `RelevanceTopCollector` in what it outputs:\n * Collects the top numResults by score, filtering out duplicates\n * and results with scores equal to Flat.MIN_VALUE.\n * The way that it achieves that is different though: it will score documents through the batch score\n * function instead of scoring documents one by one.\n */\npublic class BatchRelevanceTopCollector extends RelevanceTopCollector {\n  protected final List<BatchHit> hits;\n\n  public BatchRelevanceTopCollector(\n      ImmutableSchemaInterface schema,\n      RelevanceSearchRequestInfo searchRequestInfo,\n      ScoringFunction scoringFunction,\n      EarlybirdSearcherStats searcherStats,\n      EarlybirdCluster cluster,\n      UserTable userTable,\n      Clock clock,\n      int requestDebugMode) {\n    super(schema, searchRequestInfo, scoringFunction, searcherStats, cluster, userTable, clock,\n        requestDebugMode);\n    this.hits = new ArrayList<>((int) getMaxHitsToProcess());\n  }\n\n  @Override\n  protected void doCollectWithScore(long tweetID, float score) throws IOException {\n    Pair<LinearScoringData, ThriftSearchResultFeatures> pair =\n        scoringFunction.collectFeatures(score);\n    ThriftSearchResultMetadata metadata = collectMetadata();\n    hits.add(new BatchHit(pair.getFirst(),\n        pair.getSecond(),\n        metadata,\n        tweetID,\n        currTimeSliceID));\n  }\n\n  @Override\n  public EarlyTerminationState innerShouldCollectMore() {\n    if (hits.size() >= getMaxHitsToProcess()) {\n      return setEarlyTerminationState(EarlyTerminationState.TERMINATED_MAX_HITS_EXCEEDED);\n    }\n    return EarlyTerminationState.COLLECTING;\n  }\n\n  @Override\n  protected RelevanceSearchResults doGetRelevanceResults() throws IOException {\n    final long scoringStartNanos = getClock().nowNanos();\n    float[] scores = scoringFunction.batchScore(hits);\n    final long scoringEndNanos = getClock().nowNanos();\n    addToOverallScoringTimeNanos(scoringStartNanos, scoringEndNanos);\n    exportBatchScoringTime(scoringEndNanos - scoringStartNanos);\n\n    for (int i = 0; i < hits.size(); i++) {\n      BatchHit hit = hits.get(i);\n      ThriftSearchResultMetadata metadata = hit.getMetadata();\n\n      if (!metadata.isSetExtraMetadata()) {\n        metadata.setExtraMetadata(new ThriftSearchResultExtraMetadata());\n      }\n      metadata.getExtraMetadata().setFeatures(hit.getFeatures());\n\n\n      // Populate the ThriftSearchResultMetadata post batch scoring with information from the\n      // LinearScoringData, which now includes a score.\n      scoringFunction.populateResultMetadataBasedOnScoringData(\n          searchRequestInfo.getSearchQuery().getResultMetadataOptions(),\n          metadata,\n          hit.getScoringData());\n\n      collectWithScoreInternal(\n          hit.getTweetID(),\n          hit.getTimeSliceID(),\n          scores[i],\n          metadata\n      );\n    }\n    return getRelevanceResultsInternal();\n  }\n\n  private void exportBatchScoringTime(long scoringTimeNanos) {\n    ThriftSearchRelevanceOptions relevanceOptions = searchRequestInfo.getRelevanceOptions();\n    if (relevanceOptions.isSetRankingParams()\n        && relevanceOptions.getRankingParams().isSetSelectedTensorflowModel()) {\n      String model = relevanceOptions.getRankingParams().getSelectedTensorflowModel();\n      SearchTimerStats batchScoringPerModelTimer = SearchTimerStats.export(\n          String.format(\"batch_scoring_time_for_model_%s\", model),\n          TimeUnit.NANOSECONDS,\n          false,\n          true);\n      batchScoringPerModelTimer.timerIncrement(scoringTimeNanos);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/collectors/RelevanceAllCollector.java",
    "content": "package com.twitter.search.earlybird.search.relevance.collectors;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport com.google.common.collect.Lists;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.relevance.features.TweetIntegerShingleSignature;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.search.relevance.RelevanceHit;\nimport com.twitter.search.earlybird.search.relevance.RelevanceSearchRequestInfo;\nimport com.twitter.search.earlybird.search.relevance.RelevanceSearchResults;\nimport com.twitter.search.earlybird.search.relevance.scoring.ScoringFunction;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\n\n/**\n * RelevanceAllCollector is a results collector that collects all results sorted by score,\n * including signature-duplicates and results skipped by the scoring function.\n */\npublic class RelevanceAllCollector extends AbstractRelevanceCollector {\n  // All results.\n  protected final List<RelevanceHit> results;\n\n  public RelevanceAllCollector(\n      ImmutableSchemaInterface schema,\n      RelevanceSearchRequestInfo searchRequestInfo,\n      ScoringFunction scoringFunction,\n      EarlybirdSearcherStats searcherStats,\n      EarlybirdCluster cluster,\n      UserTable userTable,\n      Clock clock,\n      int requestDebugMode) {\n    super(schema, searchRequestInfo, scoringFunction, searcherStats, cluster, userTable, clock,\n        requestDebugMode);\n    this.results = Lists.newArrayList();\n  }\n\n  @Override\n  protected void doCollectWithScore(long tweetID, float score) throws IOException {\n    ThriftSearchResultMetadata metadata = collectMetadata();\n    scoringFunction.populateResultMetadataBasedOnScoringData(\n        searchRequestInfo.getSearchQuery().getResultMetadataOptions(),\n        metadata,\n        scoringFunction.getScoringDataForCurrentDocument());\n    results.add(new RelevanceHit(\n        currTimeSliceID,\n        tweetID,\n        TweetIntegerShingleSignature.deserialize(metadata.getSignature()),\n        metadata));\n  }\n\n  @Override\n  protected RelevanceSearchResults doGetRelevanceResults() {\n    final int numResults = results.size();\n    RelevanceSearchResults searchResults = new RelevanceSearchResults(numResults);\n\n    // Insert hits in decreasing order by score.\n    results.sort(RelevanceHit.COMPARATOR_BY_SCORE);\n    for (int i = 0; i < numResults; i++) {\n      searchResults.setHit(results.get(i), i);\n    }\n    searchResults.setRelevanceStats(getRelevanceStats());\n    searchResults.setNumHits(numResults);\n    return searchResults;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/collectors/RelevanceTopCollector.java",
    "content": "package com.twitter.search.earlybird.search.relevance.collectors;\n\nimport java.io.IOException;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.common_internal.collections.RandomAccessPriorityQueue;\nimport com.twitter.search.common.relevance.features.TweetIntegerShingleSignature;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.search.EarlyTerminationState;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.search.relevance.RelevanceHit;\nimport com.twitter.search.earlybird.search.relevance.RelevanceSearchRequestInfo;\nimport com.twitter.search.earlybird.search.relevance.RelevanceSearchResults;\nimport com.twitter.search.earlybird.search.relevance.scoring.ScoringFunction;\nimport com.twitter.search.earlybird.stats.EarlybirdSearcherStats;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultsRelevanceStats;\n\n/**\n * RelevanceTopCollector is a results collector that collects the top numResults by\n * score, filtering out duplicates.\n */\npublic class RelevanceTopCollector extends AbstractRelevanceCollector {\n  // Search results are collected in a min-heap.\n  protected final RandomAccessPriorityQueue<RelevanceHit, TweetIntegerShingleSignature> minQueue;\n\n  // Number of hits actually added to the min queue after dupe filtering and skipping.\n  // Less than or equal to numHitsProcessed.\n  protected int numHitsCollected;\n\n  // The 'top' of the min heap, or, the lowest scored document in the heap.\n  private RelevanceHit pqTop;\n  private float lowestScore = ScoringFunction.SKIP_HIT;\n\n  private final boolean isFilterDupes;\n\n  public RelevanceTopCollector(\n      ImmutableSchemaInterface schema,\n      RelevanceSearchRequestInfo searchRequestInfo,\n      ScoringFunction scoringFunction,\n      EarlybirdSearcherStats searcherStats,\n      EarlybirdCluster cluster,\n      UserTable userTable,\n      Clock clock,\n      int requestDebugMode) {\n    super(schema, searchRequestInfo, scoringFunction, searcherStats, cluster, userTable, clock,\n        requestDebugMode);\n    this.minQueue = new RandomAccessPriorityQueue<RelevanceHit, TweetIntegerShingleSignature>(\n        searchRequestInfo.getNumResultsRequested(), RelevanceHit.PQ_COMPARATOR_BY_SCORE) {\n      @Override\n      protected RelevanceHit getSentinelObject() {\n        return new RelevanceHit(); // default relevance constructor would create a hit with the\n                                   // lowest score possible.\n      }\n    };\n    this.pqTop = minQueue.top();\n    this.isFilterDupes = getSearchRequestInfo().getRelevanceOptions().isFilterDups();\n  }\n\n  protected void collectWithScoreInternal(\n      long tweetID,\n      long timeSliceID,\n      float score,\n      ThriftSearchResultMetadata metadata) {\n    // This collector cannot handle these scores:\n    assert !Float.isNaN(score);\n\n    if (score <= lowestScore) {\n      // Since docs are returned in-order (i.e., increasing doc Id), a document\n      // with equal score to pqTop.score cannot compete since HitQueue favors\n      // documents with lower doc Ids. Therefore reject those docs too.\n      // IMPORTANT: docs skipped by the scoring function will have scores set\n      // to ScoringFunction.SKIP_HIT, meaning they will not be collected.\n      return;\n    }\n\n    boolean dupFound = false;\n    Preconditions.checkState(metadata.isSetSignature(),\n        \"The signature should be set at metadata collection time, but it is null. \"\n            + \"Tweet id = %s, metadata = %s\",\n        tweetID,\n        metadata);\n    int signatureInt = metadata.getSignature();\n    final TweetIntegerShingleSignature signature =\n        TweetIntegerShingleSignature.deserialize(signatureInt);\n\n    if (isFilterDupes) {\n      // update duplicate if any\n      if (signatureInt != TweetIntegerShingleSignature.DEFAULT_NO_SIGNATURE) {\n        dupFound = minQueue.incrementElement(\n            signature,\n            element -> {\n              if (score > element.getScore()) {\n                element.update(timeSliceID, tweetID, signature, metadata);\n              }\n            }\n        );\n      }\n    }\n\n    if (!dupFound) {\n      numHitsCollected++;\n\n      // if we didn't find a duplicate element to update then we add it now as a new element to the\n      // pq\n      pqTop = minQueue.updateTop(top -> top.update(timeSliceID, tweetID, signature, metadata));\n\n      lowestScore = pqTop.getScore();\n    }\n  }\n\n  @Override\n  protected void doCollectWithScore(final long tweetID, final float score) throws IOException {\n    ThriftSearchResultMetadata metadata = collectMetadata();\n    scoringFunction.populateResultMetadataBasedOnScoringData(\n        searchRequestInfo.getSearchQuery().getResultMetadataOptions(),\n        metadata,\n        scoringFunction.getScoringDataForCurrentDocument());\n    collectWithScoreInternal(tweetID, currTimeSliceID, score, metadata);\n  }\n\n  @Override\n  public EarlyTerminationState innerShouldCollectMore() {\n    // Note that numHitsCollected here might be less than num results collected in the\n    // TwitterEarlyTerminationCollector, if we hit dups or there are very low scores.\n    if (numHitsCollected >= getMaxHitsToProcess()) {\n      return setEarlyTerminationState(EarlyTerminationState.TERMINATED_MAX_HITS_EXCEEDED);\n    }\n    return EarlyTerminationState.COLLECTING;\n  }\n\n  @Override\n  protected RelevanceSearchResults doGetRelevanceResults() throws IOException {\n    return getRelevanceResultsInternal();\n  }\n\n  protected RelevanceSearchResults getRelevanceResultsInternal() {\n    return resultsFromQueue(minQueue, getSearchRequestInfo().getNumResultsRequested(),\n                            getRelevanceStats());\n  }\n\n  private static RelevanceSearchResults resultsFromQueue(\n      RandomAccessPriorityQueue<RelevanceHit, TweetIntegerShingleSignature> pq,\n      int desiredNumResults,\n      ThriftSearchResultsRelevanceStats relevanceStats) {\n    // trim first in case we didn't fill up the queue to not get any sentinel values here\n    int numResults = pq.trim();\n    if (numResults > desiredNumResults) {\n      for (int i = 0; i < numResults - desiredNumResults; i++) {\n        pq.pop();\n      }\n      numResults = desiredNumResults;\n    }\n    RelevanceSearchResults results = new RelevanceSearchResults(numResults);\n    // insert hits in decreasing order by score\n    for (int i = numResults - 1; i >= 0; i--) {\n      RelevanceHit hit = pq.pop();\n      results.setHit(hit, i);\n    }\n    results.setRelevanceStats(relevanceStats);\n    results.setNumHits(numResults);\n    return results;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/BatchHit.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport com.twitter.search.common.features.thrift.ThriftSearchResultFeatures;\nimport com.twitter.search.earlybird.search.relevance.LinearScoringData;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\n\npublic class BatchHit {\n  private final LinearScoringData scoringData;\n  private final ThriftSearchResultFeatures features;\n  private final ThriftSearchResultMetadata metadata;\n  private final long tweetID;\n  private final long timeSliceID;\n\n  public BatchHit(\n      LinearScoringData scoringData,\n      ThriftSearchResultFeatures features,\n      ThriftSearchResultMetadata metadata,\n      long tweetID,\n      long timeSliceID\n  ) {\n    this.scoringData = scoringData;\n    this.features = features;\n    this.metadata = metadata;\n    this.tweetID = tweetID;\n    this.timeSliceID = timeSliceID;\n  }\n\n  public LinearScoringData getScoringData() {\n    return scoringData;\n  }\n\n  public ThriftSearchResultFeatures getFeatures() {\n    return features;\n  }\n\n  public ThriftSearchResultMetadata getMetadata() {\n    return metadata;\n  }\n\n  public long getTweetID() {\n    return tweetID;\n  }\n\n  public long getTimeSliceID() {\n    return timeSliceID;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/DefaultScoringFunction.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport org.apache.lucene.search.Explanation;\n\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultsRelevanceStats;\n\n/*\n * A sample scorer, doesn't really do anything, returns the same score for every document.\n */\npublic class DefaultScoringFunction extends ScoringFunction {\n  private float score;\n\n  public DefaultScoringFunction(ImmutableSchemaInterface schema) {\n    super(schema);\n  }\n\n  @Override\n  protected float score(float luceneQueryScore) {\n    score = luceneQueryScore;\n    return luceneQueryScore;\n  }\n\n  @Override\n  protected Explanation doExplain(float luceneScore) {\n    // just an example - this scoring function will go away soon\n    return Explanation.match(luceneScore, \"luceneScore=\" + luceneScore);\n  }\n\n  @Override\n  public void updateRelevanceStats(ThriftSearchResultsRelevanceStats relevanceStats) {\n    relevanceStats.setNumScored(relevanceStats.getNumScored() + 1);\n    if (score == ScoringFunction.SKIP_HIT) {\n      relevanceStats.setNumSkipped(relevanceStats.getNumSkipped() + 1);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/FeatureBasedScoringFunction.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport java.io.IOException;\nimport java.util.EnumSet;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.concurrent.TimeUnit;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableSet;\nimport com.google.common.collect.Iterables;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\nimport com.google.common.primitives.Ints;\nimport com.google.common.primitives.Longs;\n\nimport org.apache.lucene.search.Explanation;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common_internal.bloomfilter.BloomFilter;\nimport com.twitter.search.common.constants.SearchCardType;\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.database.DatabaseConfig;\nimport com.twitter.search.common.features.ExternalTweetFeature;\nimport com.twitter.search.common.features.FeatureHandler;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchemaEntry;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureType;\nimport com.twitter.search.common.features.thrift.ThriftSearchResultFeatures;\nimport com.twitter.search.common.query.QueryCommonFieldHitsVisitor;\nimport com.twitter.search.common.ranking.thriftjava.ThriftRankingParams;\nimport com.twitter.search.common.relevance.features.AgeDecay;\nimport com.twitter.search.common.relevance.features.RelevanceSignalConstants;\nimport com.twitter.search.common.relevance.text.VisibleTokenRatioNormalizer;\nimport com.twitter.search.common.results.thriftjava.FieldHitList;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.util.LongIntConverter;\nimport com.twitter.search.common.util.lang.ThriftLanguageUtil;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.search.AntiGamingFilter;\nimport com.twitter.search.earlybird.search.relevance.LinearScoringData;\nimport com.twitter.search.earlybird.search.relevance.LinearScoringData.SkipReason;\nimport com.twitter.search.earlybird.search.relevance.LinearScoringParams;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultExtraMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadataOptions;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultType;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultsRelevanceStats;\nimport com.twitter.search.earlybird.thrift.ThriftSocialFilterType;\n\n/**\n * Base class for scoring functions that rely on the extracted features stored in LinearScoringData.\n *\n * Extensions of this class must implement 2 methods:\n *\n * - computeScore\n * - generateExplanationForScoring\n *\n * They are called for scoring and generating the debug information of the document that it's\n * currently being evaluated. The field 'data' holds the features of the document.\n */\npublic abstract class FeatureBasedScoringFunction extends ScoringFunction {\n  private static final Logger LOG = LoggerFactory.getLogger(FeatureBasedScoringFunction.class);\n\n  // A multiplier that's applied to all scores to avoid scores too low.\n  public static final float SCORE_ADJUSTER = 100.0f;\n\n  private static final VisibleTokenRatioNormalizer VISIBLE_TOKEN_RATIO_NORMALIZER =\n      VisibleTokenRatioNormalizer.createInstance();\n\n  // Allow default values only for numeric types.\n  private static final Set<ThriftSearchFeatureType> ALLOWED_TYPES_FOR_DEFAULT_FEATURE_VALUES =\n      EnumSet.of(ThriftSearchFeatureType.INT32_VALUE,\n                 ThriftSearchFeatureType.LONG_VALUE,\n                 ThriftSearchFeatureType.DOUBLE_VALUE);\n\n  private static final Set<Integer> NUMERIC_FEATURES_FOR_WHICH_DEFAULTS_SHOULD_NOT_BE_SET =\n      ImmutableSet.of(EarlybirdFieldConstant.TWEET_SIGNATURE.getFieldId(),\n                      EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_LEAST_SIGNIFICANT_INT.getFieldId(),\n                      EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_MOST_SIGNIFICANT_INT.getFieldId());\n\n  // Name of the scoring function. Used for generating explanations.\n  private final String functionName;\n\n  private final BloomFilter trustedFilter;\n  private final BloomFilter followFilter;\n\n  // Current timestamp in seconds. Overridable by unit test or by timestamp set in search query.\n  private int now;\n\n  private final AntiGamingFilter antiGamingFilter;\n\n  @Nullable\n  private final AgeDecay ageDecay;\n\n  protected final LinearScoringParams params;  // Parameters and query-dependent values.\n\n  // In order for the API calls to retrieve the correct `LinearScoringData`\n  // for the passed `docId`, we need to maintain a map of `docId` -> `LinearScoringData`\n  // NOTE: THIS CAN ONLY BE REFERENCED AT HIT COLLECTION TIME, SINCE DOC IDS ARE NOT UNIQUE\n  // ACROSS SEGMENTS. IT'S NOT USABLE DURING BATCH SCORING.\n  private final Map<Integer, LinearScoringData> docIdToScoringData;\n\n  private final ThriftSearchResultType searchResultType;\n\n  private final UserTable userTable;\n\n  @VisibleForTesting\n  void setNow(int fakeNow) {\n    now = fakeNow;\n  }\n\n  public FeatureBasedScoringFunction(\n      String functionName,\n      ImmutableSchemaInterface schema,\n      ThriftSearchQuery searchQuery,\n      AntiGamingFilter antiGamingFilter,\n      ThriftSearchResultType searchResultType,\n      UserTable userTable) throws IOException {\n    super(schema);\n\n    this.functionName = functionName;\n    this.searchResultType = searchResultType;\n    this.userTable = userTable;\n\n    Preconditions.checkNotNull(searchQuery.getRelevanceOptions());\n    ThriftRankingParams rankingParams = searchQuery.getRelevanceOptions().getRankingParams();\n    Preconditions.checkNotNull(rankingParams);\n\n    params = new LinearScoringParams(searchQuery, rankingParams);\n    docIdToScoringData = new HashMap<>();\n\n    long timestamp = searchQuery.isSetTimestampMsecs() && searchQuery.getTimestampMsecs() > 0\n        ? searchQuery.getTimestampMsecs() : System.currentTimeMillis();\n    now = Ints.checkedCast(TimeUnit.MILLISECONDS.toSeconds(timestamp));\n\n    this.antiGamingFilter = antiGamingFilter;\n\n    this.ageDecay = params.useAgeDecay\n        ? new AgeDecay(params.ageDecayBase, params.ageDecayHalflife, params.ageDecaySlope)\n        : null;\n\n    if (searchQuery.isSetTrustedFilter()) {\n      trustedFilter = new BloomFilter(searchQuery.getTrustedFilter());\n    } else {\n      trustedFilter = null;\n    }\n\n    if (searchQuery.isSetDirectFollowFilter()) {\n      followFilter = new BloomFilter(searchQuery.getDirectFollowFilter());\n    } else {\n      followFilter = null;\n    }\n  }\n\n  @VisibleForTesting\n  final LinearScoringParams getScoringParams() {\n    return params;\n  }\n\n  /**\n   * Returns the LinearScoringData instance associated with the current doc ID. If it doesn't exist,\n   * an empty LinearScoringData is created.\n   */\n  @Override\n  public LinearScoringData getScoringDataForCurrentDocument() {\n    LinearScoringData data = docIdToScoringData.get(getCurrentDocID());\n    if (data == null) {\n      data = new LinearScoringData();\n      docIdToScoringData.put(getCurrentDocID(), data);\n    }\n    return data;\n  }\n\n  @Override\n  public void setDebugMode(int debugMode) {\n    super.setDebugMode(debugMode);\n  }\n\n  /**\n   * Normal the lucene score, which was unbounded, to a range of [1.0, maxLuceneScoreBoost].\n   * The normalized value increases almost linearly in the lucene score range 2.0 ~ 7.0, where\n   * most queries fall in. For rare long tail queries, like some hashtags, they have high idf and\n   * thus high lucene score, the normalized value won't have much difference between tweets.\n   * The normalization function is:\n   *   ls = luceneScore\n   *   norm = min(max, 1 + (max - 1.0) / 2.4 * ln(1 + ls)\n   */\n  static float normalizeLuceneScore(float luceneScore, float maxBoost) {\n    return (float) Math.min(maxBoost, 1.0 + (maxBoost - 1.0) / 2.4 * Math.log1p(luceneScore));\n  }\n\n  @Override\n  protected float score(float luceneQueryScore) throws IOException {\n    return scoreInternal(luceneQueryScore, null);\n  }\n\n  protected LinearScoringData updateLinearScoringData(float luceneQueryScore) throws IOException {\n    // Reset the data for each tweet!!!\n    LinearScoringData data = new LinearScoringData();\n    docIdToScoringData.put(getCurrentDocID(), data);\n\n    // Set proper version for engagement counters for this request.\n    data.skipReason = SkipReason.NOT_SKIPPED;\n    data.luceneScore = luceneQueryScore;\n    data.userRep = (byte) documentFeatures.getFeatureValue(EarlybirdFieldConstant.USER_REPUTATION);\n\n    if (antiGamingFilter != null && !antiGamingFilter.accept(getCurrentDocID())) {\n      data.skipReason = SkipReason.ANTIGAMING;\n      return data;\n    }\n\n    data.textScore = (byte) documentFeatures.getFeatureValue(EarlybirdFieldConstant.TEXT_SCORE);\n    data.tokenAt140DividedByNumTokensBucket = VISIBLE_TOKEN_RATIO_NORMALIZER.denormalize(\n        (byte) documentFeatures.getFeatureValue(EarlybirdFieldConstant.VISIBLE_TOKEN_RATIO));\n    data.fromUserId = documentFeatures.getFeatureValue(EarlybirdFieldConstant.FROM_USER_ID_CSF);\n    data.isFollow = followFilter != null\n        && followFilter.contains(Longs.toByteArray(data.fromUserId));\n    data.isTrusted = trustedFilter != null\n        && trustedFilter.contains(Longs.toByteArray(data.fromUserId));\n    data.isFromVerifiedAccount = documentFeatures.isFlagSet(\n        EarlybirdFieldConstant.FROM_VERIFIED_ACCOUNT_FLAG);\n    data.isFromBlueVerifiedAccount = documentFeatures.isFlagSet(\n        EarlybirdFieldConstant.FROM_BLUE_VERIFIED_ACCOUNT_FLAG);\n    data.isSelfTweet = data.fromUserId == params.searcherId;\n    // v1 engagement counters, note that the first three values are post-log2 version\n    // of the original unnormalized values.\n    data.retweetCountPostLog2 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.RETWEET_COUNT);\n    data.replyCountPostLog2 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.REPLY_COUNT);\n    data.favCountPostLog2 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.FAVORITE_COUNT);\n    data.embedsImpressionCount = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.EMBEDS_IMPRESSION_COUNT);\n    data.embedsUrlCount = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.EMBEDS_URL_COUNT);\n    data.videoViewCount = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.VIDEO_VIEW_COUNT);\n    // v2 engagement counters\n    data.retweetCountV2 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.RETWEET_COUNT_V2);\n    data.replyCountV2 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.REPLY_COUNT_V2);\n    data.favCountV2 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.FAVORITE_COUNT_V2);\n    // other v2 engagement counters\n    data.embedsImpressionCountV2 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.EMBEDS_IMPRESSION_COUNT_V2);\n    data.embedsUrlCountV2 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.EMBEDS_URL_COUNT_V2);\n    data.videoViewCountV2 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.VIDEO_VIEW_COUNT_V2);\n    // pure v2 engagement counters without v1 counterpart\n    data.quotedCount = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.QUOTE_COUNT);\n    data.weightedRetweetCount = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.WEIGHTED_RETWEET_COUNT);\n    data.weightedReplyCount = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.WEIGHTED_REPLY_COUNT);\n    data.weightedFavCount = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.WEIGHTED_FAVORITE_COUNT);\n    data.weightedQuoteCount = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.WEIGHTED_QUOTE_COUNT);\n\n    Double querySpecificScoreAdjustment = params.querySpecificScoreAdjustments == null ? null\n        : params.querySpecificScoreAdjustments.get(tweetIDMapper.getTweetID(getCurrentDocID()));\n    data.querySpecificScore =\n        querySpecificScoreAdjustment == null ? 0.0 : querySpecificScoreAdjustment;\n\n    data.authorSpecificScore = params.authorSpecificScoreAdjustments == null\n        ? 0.0\n        : params.authorSpecificScoreAdjustments.getOrDefault(data.fromUserId, 0.0);\n\n    // respect social filter type\n    if (params.socialFilterType != null && !data.isSelfTweet) {\n      if ((params.socialFilterType == ThriftSocialFilterType.ALL\n              && !data.isFollow && !data.isTrusted)\n          || (params.socialFilterType == ThriftSocialFilterType.TRUSTED && !data.isTrusted)\n          || (params.socialFilterType == ThriftSocialFilterType.FOLLOWS && !data.isFollow)) {\n        // we can skip this hit as we only want social results in this mode.\n        data.skipReason = SkipReason.SOCIAL_FILTER;\n        return data;\n      }\n    }\n\n    // 1. first apply all the filters to only non-follow tweets and non-verified accounts,\n    //    but be tender to sentinel values\n    // unless you specifically asked to apply filters regardless\n    if (params.applyFiltersAlways\n            || (!data.isSelfTweet && !data.isFollow && !data.isFromVerifiedAccount\n                && !data.isFromBlueVerifiedAccount)) {\n      if (data.userRep < params.reputationMinVal\n          // don't filter unset userreps, we give them the benefit of doubt and let it\n          // continue to scoring. userrep is unset when either user just signed up or\n          // during ingestion time we had trouble getting userrep from reputation service.\n          && data.userRep != RelevanceSignalConstants.UNSET_REPUTATION_SENTINEL) {\n        data.skipReason = SkipReason.LOW_REPUTATION;\n        return data;\n      } else if (data.textScore < params.textScoreMinVal\n                 // don't filter unset text scores, use goodwill value\n                 && data.textScore != RelevanceSignalConstants.UNSET_TEXT_SCORE_SENTINEL) {\n        data.skipReason = SkipReason.LOW_TEXT_SCORE;\n        return data;\n      } else if (data.retweetCountPostLog2 != LinearScoringData.UNSET_SIGNAL_VALUE\n                 && data.retweetCountPostLog2 < params.retweetMinVal) {\n        data.skipReason = SkipReason.LOW_RETWEET_COUNT;\n        return data;\n      } else if (data.favCountPostLog2 != LinearScoringData.UNSET_SIGNAL_VALUE\n                 && data.favCountPostLog2 < params.favMinVal) {\n        data.skipReason = SkipReason.LOW_FAV_COUNT;\n        return data;\n      }\n    }\n\n    // if sentinel value is set, assume goodwill score and let scoring continue.\n    if (data.textScore == RelevanceSignalConstants.UNSET_TEXT_SCORE_SENTINEL) {\n      data.textScore = RelevanceSignalConstants.GOODWILL_TEXT_SCORE;\n    }\n    if (data.userRep == RelevanceSignalConstants.UNSET_REPUTATION_SENTINEL) {\n      data.userRep = RelevanceSignalConstants.GOODWILL_REPUTATION;\n    }\n\n    data.tweetAgeInSeconds = now - timeMapper.getTime(getCurrentDocID());\n    if (data.tweetAgeInSeconds < 0) {\n      data.tweetAgeInSeconds = 0; // Age cannot be negative\n    }\n\n    // The PARUS_SCORE feature should be read as is.\n    data.parusScore = documentFeatures.getFeatureValue(EarlybirdFieldConstant.PARUS_SCORE);\n\n    data.isNullcast = documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_NULLCAST_FLAG);\n    data.hasUrl =  documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_LINK_FLAG);\n    data.hasImageUrl = documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_IMAGE_URL_FLAG);\n    data.hasVideoUrl = documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_VIDEO_URL_FLAG);\n    data.hasNewsUrl = documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_NEWS_URL_FLAG);\n    data.isReply =  documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_REPLY_FLAG);\n    data.isRetweet = documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_RETWEET_FLAG);\n    data.isOffensive = documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_OFFENSIVE_FLAG);\n    data.hasTrend = documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_TREND_FLAG);\n    data.hasMultipleHashtagsOrTrends =\n        documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_MULTIPLE_HASHTAGS_OR_TRENDS_FLAG);\n    data.isUserSpam = documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_USER_SPAM_FLAG);\n    data.isUserNSFW = documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_USER_NSFW_FLAG)\n        || userTable.isSet(data.fromUserId, UserTable.NSFW_BIT);\n    data.isUserAntiSocial =\n        userTable.isSet(data.fromUserId, UserTable.ANTISOCIAL_BIT);\n    data.isUserBot = documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_USER_BOT_FLAG);\n    data.hasCard = documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_CARD_FLAG);\n    data.cardType = SearchCardType.UNKNOWN.getByteValue();\n    if (data.hasCard) {\n      data.cardType =\n          (byte) documentFeatures.getFeatureValue(EarlybirdFieldConstant.CARD_TYPE_CSF_FIELD);\n    }\n    data.hasVisibleLink = documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_VISIBLE_LINK_FLAG);\n\n    data.hasConsumerVideo =\n        documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_CONSUMER_VIDEO_FLAG);\n    data.hasProVideo = documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_PRO_VIDEO_FLAG);\n    data.hasVine = documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_VINE_FLAG);\n    data.hasPeriscope = documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_PERISCOPE_FLAG);\n    data.hasNativeImage = documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_NATIVE_IMAGE_FLAG);\n    data.hasQuote = documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_QUOTE_FLAG);\n    data.isComposerSourceCamera =\n        documentFeatures.isFlagSet(EarlybirdFieldConstant.COMPOSER_SOURCE_IS_CAMERA_FLAG);\n\n    // Only read the shared status if the isRetweet or isReply bit is true (minor optimization).\n    if (data.isRetweet || (params.getInReplyToStatusId && data.isReply)) {\n      data.sharedStatusId =\n          documentFeatures.getFeatureValue(EarlybirdFieldConstant.SHARED_STATUS_ID_CSF);\n    }\n\n    // Only read the reference tweet author ID if the isRetweet or isReply bit\n    // is true (minor optimization).\n    if (data.isRetweet || data.isReply) {\n      // the REFERENCE_AUTHOR_ID_CSF stores the source tweet author id for all retweets\n      long referenceAuthorId =\n          documentFeatures.getFeatureValue(EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_CSF);\n      if (referenceAuthorId > 0) {\n        data.referenceAuthorId = referenceAuthorId;\n      } else {\n        // we also store the reference author id for retweets, directed at tweets, and self threaded\n        // tweets separately on Realtime/Protected Earlybirds. This data will be moved to the\n        // REFERENCE_AUTHOR_ID_CSF and these fields will be deprecated in SEARCH-34958.\n        referenceAuthorId = LongIntConverter.convertTwoIntToOneLong(\n            (int) documentFeatures.getFeatureValue(\n                EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_MOST_SIGNIFICANT_INT),\n            (int) documentFeatures.getFeatureValue(\n                EarlybirdFieldConstant.REFERENCE_AUTHOR_ID_LEAST_SIGNIFICANT_INT));\n        if (referenceAuthorId > 0) {\n          data.referenceAuthorId = referenceAuthorId;\n        }\n      }\n    }\n\n    // Convert language to a thrift language and then back to an int in order to\n    // ensure a value compatible with our current ThriftLanguage definition.\n    ThriftLanguage tweetLang = ThriftLanguageUtil.safeFindByValue(\n        (int) documentFeatures.getFeatureValue(EarlybirdFieldConstant.LANGUAGE));\n    data.tweetLangId = tweetLang.getValue();\n    // Set the language-related features here so that they can be later used in promotion/demotion\n    // and also be transferred to ThriftSearchResultMetadata\n    data.userLangMult = computeUserLangMultiplier(data, params);\n    data.hasDifferentLang = params.uiLangId != ThriftLanguage.UNKNOWN.getValue()\n        && params.uiLangId != data.tweetLangId;\n    data.hasEnglishTweetAndDifferentUILang = data.hasDifferentLang\n        && data.tweetLangId == ThriftLanguage.ENGLISH.getValue();\n    data.hasEnglishUIAndDifferentTweetLang = data.hasDifferentLang\n        && params.uiLangId == ThriftLanguage.ENGLISH.getValue();\n\n    // Exposed all these features for the clients.\n    data.isSensitiveContent =\n        documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_SENSITIVE_CONTENT);\n    data.hasMultipleMediaFlag =\n        documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_MULTIPLE_MEDIA_FLAG);\n    data.profileIsEggFlag = documentFeatures.isFlagSet(EarlybirdFieldConstant.PROFILE_IS_EGG_FLAG);\n    data.isUserNewFlag = documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_USER_NEW_FLAG);\n    data.numMentions = (int) documentFeatures.getFeatureValue(EarlybirdFieldConstant.NUM_MENTIONS);\n    data.numHashtags = (int) documentFeatures.getFeatureValue(EarlybirdFieldConstant.NUM_HASHTAGS);\n    data.linkLanguage =\n        (int) documentFeatures.getFeatureValue(EarlybirdFieldConstant.LINK_LANGUAGE);\n    data.prevUserTweetEngagement =\n        (int) documentFeatures.getFeatureValue(EarlybirdFieldConstant.PREV_USER_TWEET_ENGAGEMENT);\n\n    // health model scores by HML\n    data.toxicityScore = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.TOXICITY_SCORE);\n    data.pBlockScore = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.PBLOCK_SCORE);\n    data.pSpammyTweetScore = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.P_SPAMMY_TWEET_SCORE);\n    data.pReportedTweetScore = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.P_REPORTED_TWEET_SCORE);\n    data.spammyTweetContentScore = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.SPAMMY_TWEET_CONTENT_SCORE\n    );\n    data.experimentalHealthModelScore1 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.EXPERIMENTAL_HEALTH_MODEL_SCORE_1);\n    data.experimentalHealthModelScore2 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.EXPERIMENTAL_HEALTH_MODEL_SCORE_2);\n    data.experimentalHealthModelScore3 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.EXPERIMENTAL_HEALTH_MODEL_SCORE_3);\n    data.experimentalHealthModelScore4 = documentFeatures.getUnnormalizedFeatureValue(\n        EarlybirdFieldConstant.EXPERIMENTAL_HEALTH_MODEL_SCORE_4);\n\n    return data;\n  }\n\n  protected float scoreInternal(\n      float luceneQueryScore, ExplanationWrapper explanation) throws IOException {\n    LinearScoringData data = updateLinearScoringData(luceneQueryScore);\n    if (data.skipReason != null && data.skipReason != SkipReason.NOT_SKIPPED) {\n      return finalizeScore(data, explanation, SKIP_HIT);\n    }\n\n    double score = computeScore(data, explanation != null);\n    return postScoreComputation(data, score, true, explanation);\n  }\n\n  protected float postScoreComputation(\n      LinearScoringData data,\n      double score,\n      boolean boostScoreWithHitAttribution,\n      ExplanationWrapper explanation) throws IOException {\n    double modifiedScore = score;\n    data.scoreBeforeBoost = modifiedScore;\n    if (params.applyBoosts) {\n      modifiedScore =\n          applyBoosts(data, modifiedScore, boostScoreWithHitAttribution, explanation != null);\n    }\n    // Final adjustment to avoid too-low scores.\n    modifiedScore *= SCORE_ADJUSTER;\n    data.scoreAfterBoost = modifiedScore;\n\n    // 3. final score filter\n    data.scoreFinal = modifiedScore;\n    if ((params.applyFiltersAlways || (!data.isSelfTweet && !data.isFollow))\n        && modifiedScore < params.minScore) {\n      data.skipReason = SkipReason.LOW_FINAL_SCORE;\n      modifiedScore = SKIP_HIT;\n    }\n\n    // clear field hits\n    this.fieldHitAttribution = null;\n    return finalizeScore(data, explanation, modifiedScore);\n  }\n\n  /**\n   * Applying promotion/demotion to the scores generated by feature-based scoring functions\n   *\n   * @param data Original LinearScoringData (to be modified with boosts here)\n   * @param score Score generated by the feature-based scoring function\n   * @param withHitAttribution Determines if hit attribution data should be included.\n   * @param forExplanation Indicates if the score will be computed for generating the explanation.\n   * @return Score after applying promotion/demotion\n   */\n  private double applyBoosts(\n      LinearScoringData data,\n      double score,\n      boolean withHitAttribution,\n      boolean forExplanation) {\n    double boostedScore = score;\n\n    if (params.useLuceneScoreAsBoost) {\n      data.normalizedLuceneScore = normalizeLuceneScore(\n          (float) data.luceneScore, (float) params.maxLuceneScoreBoost);\n      boostedScore *= data.normalizedLuceneScore;\n    }\n    if (data.isOffensive) {\n      boostedScore *= params.offensiveDamping;\n    }\n    if (data.isUserSpam && params.spamUserDamping != LinearScoringData.NO_BOOST_VALUE) {\n      data.spamUserDampApplied = true;\n      boostedScore *= params.spamUserDamping;\n    }\n    if (data.isUserNSFW && params.nsfwUserDamping != LinearScoringData.NO_BOOST_VALUE) {\n      data.nsfwUserDampApplied = true;\n      boostedScore *= params.nsfwUserDamping;\n    }\n    if (data.isUserBot && params.botUserDamping != LinearScoringData.NO_BOOST_VALUE) {\n      data.botUserDampApplied = true;\n      boostedScore *= params.botUserDamping;\n    }\n\n    // cards\n    if (data.hasCard && params.hasCardBoosts[data.cardType] != LinearScoringData.NO_BOOST_VALUE) {\n      boostedScore *= params.hasCardBoosts[data.cardType];\n      data.hasCardBoostApplied = true;\n    }\n\n    // trends\n    if (data.hasMultipleHashtagsOrTrends) {\n      boostedScore *= params.multipleHashtagsOrTrendsDamping;\n    } else if (data.hasTrend) {\n      data.tweetHasTrendsBoostApplied = true;\n      boostedScore *= params.tweetHasTrendBoost;\n    }\n\n    // Media/News url boosts.\n    if (data.hasImageUrl || data.hasVideoUrl) {\n      data.hasMedialUrlBoostApplied = true;\n      boostedScore *= params.tweetHasMediaUrlBoost;\n    }\n    if (data.hasNewsUrl) {\n      data.hasNewsUrlBoostApplied = true;\n      boostedScore *= params.tweetHasNewsUrlBoost;\n    }\n\n    if (data.isFromVerifiedAccount) {\n      data.tweetFromVerifiedAccountBoostApplied = true;\n      boostedScore *= params.tweetFromVerifiedAccountBoost;\n    }\n\n    if (data.isFromBlueVerifiedAccount) {\n      data.tweetFromBlueVerifiedAccountBoostApplied = true;\n      boostedScore *= params.tweetFromBlueVerifiedAccountBoost;\n    }\n\n    if (data.isFollow) {\n      // direct follow, so boost both replies and non-replies.\n      data.directFollowBoostApplied = true;\n      boostedScore *= params.directFollowBoost;\n    } else if (data.isTrusted) {\n      // trusted circle\n      if (!data.isReply) {\n        // non-at-reply, in trusted network\n        data.trustedCircleBoostApplied = true;\n        boostedScore *= params.trustedCircleBoost;\n      }\n    } else if (data.isReply) {\n      // at-reply out of my network\n      data.outOfNetworkReplyPenaltyApplied = true;\n      boostedScore -= params.outOfNetworkReplyPenalty;\n    }\n\n    if (data.isSelfTweet) {\n      data.selfTweetBoostApplied = true;\n      data.selfTweetMult = params.selfTweetBoost;\n      boostedScore *= params.selfTweetBoost;\n    }\n\n    // Language Demotion\n    // User language based demotion\n    // The data.userLangMult is set in scoreInternal(), and this setting step is always before\n    // the applying boosts step\n    if (params.useUserLanguageInfo) {\n      boostedScore *= data.userLangMult;\n    }\n    // UI language based demotion\n    if (params.uiLangId != ThriftLanguage.UNKNOWN.getValue()\n        && params.uiLangId != data.tweetLangId) {\n      if (data.tweetLangId == ThriftLanguage.ENGLISH.getValue()) {\n        data.uiLangMult = params.langEnglishTweetDemote;\n      } else if (params.uiLangId == ThriftLanguage.ENGLISH.getValue()) {\n        data.uiLangMult = params.langEnglishUIDemote;\n      } else {\n        data.uiLangMult = params.langDefaultDemote;\n      }\n    } else {\n      data.uiLangMult = LinearScoringData.NO_BOOST_VALUE;\n    }\n    boostedScore *= data.uiLangMult;\n\n    if (params.useAgeDecay) {\n      // shallow sigmoid with an inflection point at ageDecayHalflife\n      data.ageDecayMult = ageDecay.getAgeDecayMultiplier(data.tweetAgeInSeconds);\n      boostedScore *= data.ageDecayMult;\n    }\n\n    // Hit Attribute Demotion\n    // Scoring is currently based on tokenized user name, text, and url in the tweet\n    // If hit attribute collection is enabled, we demote score based on these fields\n    if (hitAttributeHelper != null && params.enableHitDemotion) {\n\n      Map<Integer, List<String>> hitMap;\n      if (forExplanation && fieldHitAttribution != null) {\n        // if this scoring call is for generating an explanation,\n        // we'll use the fieldHitAttribution found in the search result's metadata because\n        // collectors are not called during the debug workflow\n        hitMap = Maps.transformValues(fieldHitAttribution.getHitMap(), FieldHitList::getHitFields);\n      } else if (withHitAttribution) {\n        hitMap = hitAttributeHelper.getHitAttribution(getCurrentDocID());\n      } else {\n        hitMap = Maps.newHashMap();\n      }\n      Set<String> uniqueFieldHits = ImmutableSet.copyOf(Iterables.concat(hitMap.values()));\n\n      data.hitFields.addAll(uniqueFieldHits);\n      // there should always be fields that are hit\n      // if there aren't, we assume this is a call from 'explain' in debug mode\n      // do not override hit attribute data if in debug mode\n      if (!uniqueFieldHits.isEmpty()) {\n        // demotions based strictly on field hits\n        if (uniqueFieldHits.size() == 1) {\n          if (uniqueFieldHits.contains(\n                  EarlybirdFieldConstant.RESOLVED_LINKS_TEXT_FIELD.getFieldName())) {\n            // if url was the only field that was hit, demote\n            data.hasUrlOnlyHitDemotionApplied = true;\n            boostedScore *= params.urlOnlyHitDemotion;\n          } else if (uniqueFieldHits.contains(\n                         EarlybirdFieldConstant.TOKENIZED_FROM_USER_FIELD.getFieldName())) {\n            // if name was the only field that was hit, demote\n            data.hasNameOnlyHitDemotionApplied = true;\n            boostedScore *= params.nameOnlyHitDemotion;\n          }\n        } else if (!uniqueFieldHits.contains(EarlybirdFieldConstant.TEXT_FIELD.getFieldName())\n            && !uniqueFieldHits.contains(EarlybirdFieldConstant.MENTIONS_FIELD.getFieldName())\n            && !uniqueFieldHits.contains(EarlybirdFieldConstant.HASHTAGS_FIELD.getFieldName())\n            && !uniqueFieldHits.contains(EarlybirdFieldConstant.STOCKS_FIELD.getFieldName())) {\n          // if text or special text was never hit, demote\n          data.hasNoTextHitDemotionApplied = true;\n          boostedScore *= params.noTextHitDemotion;\n        } else if (uniqueFieldHits.size() == 2) {\n          // demotions based on field hit combinations\n          // want to demote if we only hit two of the fields (one being text)\n          // but with separate terms\n          Set<String> fieldIntersections = QueryCommonFieldHitsVisitor.findIntersection(\n              hitAttributeHelper.getNodeToRankMap(),\n              hitMap,\n              query);\n\n          if (fieldIntersections.isEmpty()) {\n            if (uniqueFieldHits.contains(\n                    EarlybirdFieldConstant.TOKENIZED_FROM_USER_FIELD.getFieldName())) {\n              // if name is hit but has no hits in common with text, demote\n              // want to demote cases where we hit part of the person's name\n              // and tweet text separately\n              data.hasSeparateTextAndNameHitDemotionApplied = true;\n              boostedScore *= params.separateTextAndNameHitDemotion;\n            } else if (uniqueFieldHits.contains(\n                           EarlybirdFieldConstant.RESOLVED_LINKS_TEXT_FIELD.getFieldName())) {\n              // if url is hit but has no hits in common with text, demote\n              // want to demote cases where we hit a potential domain keyword\n              // and tweet text separately\n              data.hasSeparateTextAndUrlHitDemotionApplied = true;\n              boostedScore *= params.separateTextAndUrlHitDemotion;\n            }\n          }\n        }\n      }\n    }\n\n    return boostedScore;\n  }\n\n  /**\n   * Compute the user language based demotion multiplier\n   */\n  private static double computeUserLangMultiplier(\n      LinearScoringData data, LinearScoringParams params) {\n    if (data.tweetLangId == params.uiLangId\n        && data.tweetLangId != ThriftLanguage.UNKNOWN.getValue()) {\n      // Effectively the uiLang is considered a language that user knows with 1.0 confidence.\n      return LinearScoringData.NO_BOOST_VALUE;\n    }\n\n    if (params.userLangs[data.tweetLangId] > 0.0) {\n      return params.userLangs[data.tweetLangId];\n    }\n\n    return params.unknownLanguageBoost;\n  }\n\n  /**\n   * Computes the score of the document that it's currently being evaluated.\n   *\n   * The extracted features from the document are available in the field 'data'.\n   *\n   * @param data The LinearScoringData instance that will store the document features.\n   * @param forExplanation Indicates if the score will be computed for generating the explanation.\n   */\n  protected abstract double computeScore(\n      LinearScoringData data, boolean forExplanation) throws IOException;\n\n  private float finalizeScore(\n      LinearScoringData scoringData,\n      ExplanationWrapper explanation,\n      double score) throws IOException {\n    scoringData.scoreReturned = score;\n    if (explanation != null) {\n      explanation.explanation = generateExplanation(scoringData);\n    }\n    return (float) score;\n  }\n\n  @Override\n  protected void initializeNextSegment(EarlybirdIndexSegmentAtomicReader reader)\n      throws IOException {\n    if (antiGamingFilter != null) {\n      antiGamingFilter.startSegment(reader);\n    }\n  }\n\n  /*\n   * Generate the scoring explanation for debug.\n   */\n  private Explanation generateExplanation(LinearScoringData scoringData) throws IOException {\n    final List<Explanation> details = Lists.newArrayList();\n\n    details.add(Explanation.match(0.0f, \"[PROPERTIES] \"\n        + scoringData.getPropertyExplanation()));\n\n    // 1. Filters\n    boolean isHit = scoringData.skipReason == SkipReason.NOT_SKIPPED;\n    if (scoringData.skipReason == SkipReason.ANTIGAMING) {\n      details.add(Explanation.noMatch(\"SKIPPED for antigaming\"));\n    }\n    if (scoringData.skipReason == SkipReason.LOW_REPUTATION) {\n      details.add(Explanation.noMatch(\n          String.format(\"SKIPPED for low reputation: %.3f < %.3f\",\n              scoringData.userRep, params.reputationMinVal)));\n    }\n    if (scoringData.skipReason == SkipReason.LOW_TEXT_SCORE) {\n      details.add(Explanation.noMatch(\n          String.format(\"SKIPPED for low text score: %.3f < %.3f\",\n              scoringData.textScore, params.textScoreMinVal)));\n    }\n    if (scoringData.skipReason == SkipReason.LOW_RETWEET_COUNT) {\n      details.add(Explanation.noMatch(\n          String.format(\"SKIPPED for low retweet count: %.3f < %.3f\",\n              scoringData.retweetCountPostLog2, params.retweetMinVal)));\n    }\n    if (scoringData.skipReason == SkipReason.LOW_FAV_COUNT) {\n      details.add(Explanation.noMatch(\n          String.format(\"SKIPPED for low fav count: %.3f < %.3f\",\n              scoringData.favCountPostLog2, params.favMinVal)));\n    }\n    if (scoringData.skipReason == SkipReason.SOCIAL_FILTER) {\n      details.add(Explanation.noMatch(\"SKIPPED for not in the right social circle\"));\n    }\n\n    // 2. Explanation depending on the scoring type\n    generateExplanationForScoring(scoringData, isHit, details);\n\n    // 3. Explanation depending on boosts\n    if (params.applyBoosts) {\n      generateExplanationForBoosts(scoringData, isHit, details);\n    }\n\n    // 4. Final score filter.\n    if (scoringData.skipReason == SkipReason.LOW_FINAL_SCORE) {\n      details.add(Explanation.noMatch(\"SKIPPED for low final score: \" + scoringData.scoreFinal));\n      isHit = false;\n    }\n\n    String hostAndSegment = String.format(\"%s host = %s  segment = %s\",\n        functionName, DatabaseConfig.getLocalHostname(), DatabaseConfig.getDatabase());\n    if (isHit) {\n      return Explanation.match((float) scoringData.scoreFinal, hostAndSegment, details);\n    } else {\n      return Explanation.noMatch(hostAndSegment, details);\n    }\n  }\n\n  /**\n   * Generates the explanation for the document that is currently being evaluated.\n   *\n   * Implementations of this method must use the 'details' parameter to collect its output.\n   *\n   * @param scoringData Scoring components for the document\n   * @param isHit Indicates whether the document is not skipped\n   * @param details Details of the explanation. Used to collect the output.\n   */\n  protected abstract void generateExplanationForScoring(\n      LinearScoringData scoringData, boolean isHit, List<Explanation> details) throws IOException;\n\n  /**\n   * Generates the boosts part of the explanation for the document that is currently\n   * being evaluated.\n   */\n  private void generateExplanationForBoosts(\n      LinearScoringData scoringData,\n      boolean isHit,\n      List<Explanation> details) {\n    List<Explanation> boostDetails = Lists.newArrayList();\n\n    boostDetails.add(Explanation.match((float) scoringData.scoreBeforeBoost, \"Score before boost\"));\n\n\n    // Lucene score boost\n    if (params.useLuceneScoreAsBoost) {\n      boostDetails.add(Explanation.match(\n          (float) scoringData.normalizedLuceneScore,\n          String.format(\"[x] Lucene score boost, luceneScore=%.3f\",\n              scoringData.luceneScore)));\n    }\n\n    // card boost\n    if (scoringData.hasCardBoostApplied) {\n      boostDetails.add(Explanation.match((float) params.hasCardBoosts[scoringData.cardType],\n          \"[x] card boost for type \" + SearchCardType.cardTypeFromByteValue(scoringData.cardType)));\n    }\n\n    // Offensive\n    if (scoringData.isOffensive) {\n      boostDetails.add(Explanation.match((float) params.offensiveDamping, \"[x] Offensive damping\"));\n    } else {\n      boostDetails.add(Explanation.match(LinearScoringData.NO_BOOST_VALUE,\n          String.format(\"Not Offensive, damping=%.3f\", params.offensiveDamping)));\n    }\n\n    // Spam\n    if (scoringData.spamUserDampApplied) {\n      boostDetails.add(Explanation.match((float) params.spamUserDamping, \"[x] Spam\"));\n    }\n    // NSFW\n    if (scoringData.nsfwUserDampApplied) {\n      boostDetails.add(Explanation.match((float) params.nsfwUserDamping, \"[X] NSFW\"));\n    }\n    // Bot\n    if (scoringData.botUserDampApplied) {\n      boostDetails.add(Explanation.match((float) params.botUserDamping, \"[X] Bot\"));\n    }\n\n    // Multiple hashtags or trends\n    if (scoringData.hasMultipleHashtagsOrTrends) {\n      boostDetails.add(Explanation.match((float) params.multipleHashtagsOrTrendsDamping,\n          \"[x] Multiple hashtags or trends boost\"));\n    } else {\n      boostDetails.add(Explanation.match(LinearScoringData.NO_BOOST_VALUE,\n          String.format(\"No multiple hashtags or trends, damping=%.3f\",\n              params.multipleHashtagsOrTrendsDamping)));\n    }\n\n    if (scoringData.tweetHasTrendsBoostApplied) {\n      boostDetails.add(Explanation.match(\n          (float) params.tweetHasTrendBoost, \"[x] Tweet has trend boost\"));\n    }\n\n    if (scoringData.hasMedialUrlBoostApplied) {\n      boostDetails.add(Explanation.match(\n          (float) params.tweetHasMediaUrlBoost, \"[x] Media url boost\"));\n    }\n\n    if (scoringData.hasNewsUrlBoostApplied) {\n      boostDetails.add(Explanation.match(\n          (float) params.tweetHasNewsUrlBoost, \"[x] News url boost\"));\n    }\n\n    boostDetails.add(Explanation.match(0.0f, \"[FIELDS HIT] \" + scoringData.hitFields));\n\n    if (scoringData.hasNoTextHitDemotionApplied) {\n      boostDetails.add(Explanation.match(\n          (float) params.noTextHitDemotion, \"[x] No text hit demotion\"));\n    }\n\n    if (scoringData.hasUrlOnlyHitDemotionApplied) {\n      boostDetails.add(Explanation.match(\n          (float) params.urlOnlyHitDemotion, \"[x] URL only hit demotion\"));\n    }\n\n    if (scoringData.hasNameOnlyHitDemotionApplied) {\n      boostDetails.add(Explanation.match(\n          (float) params.nameOnlyHitDemotion, \"[x] Name only hit demotion\"));\n    }\n\n    if (scoringData.hasSeparateTextAndNameHitDemotionApplied) {\n      boostDetails.add(Explanation.match((float) params.separateTextAndNameHitDemotion,\n          \"[x] Separate text/name demotion\"));\n    }\n\n    if (scoringData.hasSeparateTextAndUrlHitDemotionApplied) {\n      boostDetails.add(Explanation.match((float) params.separateTextAndUrlHitDemotion,\n          \"[x] Separate text/url demotion\"));\n    }\n\n    if (scoringData.tweetFromVerifiedAccountBoostApplied) {\n      boostDetails.add(Explanation.match((float) params.tweetFromVerifiedAccountBoost,\n          \"[x] Verified account boost\"));\n    }\n\n    if (scoringData.tweetFromBlueVerifiedAccountBoostApplied) {\n      boostDetails.add(Explanation.match((float) params.tweetFromBlueVerifiedAccountBoost,\n          \"[x] Blue-verified account boost\"));\n    }\n\n    if (scoringData.selfTweetBoostApplied) {\n      boostDetails.add(Explanation.match((float) params.selfTweetBoost,\n          \"[x] Self tweet boost\"));\n    }\n\n    if (scoringData.skipReason == LinearScoringData.SkipReason.SOCIAL_FILTER) {\n      boostDetails.add(Explanation.noMatch(\"SKIPPED for social filter\"));\n    } else {\n      if (scoringData.directFollowBoostApplied) {\n        boostDetails.add(Explanation.match((float) params.directFollowBoost,\n            \"[x] Direct follow boost\"));\n      }\n      if (scoringData.trustedCircleBoostApplied) {\n        boostDetails.add(Explanation.match((float) params.trustedCircleBoost,\n            \"[x] Trusted circle boost\"));\n      }\n      if (scoringData.outOfNetworkReplyPenaltyApplied) {\n        boostDetails.add(Explanation.match((float) params.outOfNetworkReplyPenalty,\n            \"[-] Out of network reply penalty\"));\n      }\n    }\n\n    // Language demotions\n    String langDetails = String.format(\n        \"tweetLang=[%s] uiLang=[%s]\",\n        ThriftLanguageUtil.getLocaleOf(\n            ThriftLanguage.findByValue(scoringData.tweetLangId)).getLanguage(),\n        ThriftLanguageUtil.getLocaleOf(ThriftLanguage.findByValue(params.uiLangId)).getLanguage());\n    if (scoringData.uiLangMult == 1.0) {\n      boostDetails.add(Explanation.match(\n          LinearScoringData.NO_BOOST_VALUE, \"No UI Language demotion: \" + langDetails));\n    } else {\n      boostDetails.add(Explanation.match(\n          (float) scoringData.uiLangMult, \"[x] UI LangMult: \" + langDetails));\n    }\n    StringBuilder userLangDetails = new StringBuilder();\n    userLangDetails.append(\"userLang=[\");\n    for (int i = 0; i < params.userLangs.length; i++) {\n      if (params.userLangs[i] > 0.0) {\n        String lang = ThriftLanguageUtil.getLocaleOf(ThriftLanguage.findByValue(i)).getLanguage();\n        userLangDetails.append(String.format(\"%s:%.3f,\", lang, params.userLangs[i]));\n      }\n    }\n    userLangDetails.append(\"]\");\n    if (!params.useUserLanguageInfo) {\n      boostDetails.add(Explanation.noMatch(\n          \"No User Language Demotion: \" + userLangDetails.toString()));\n    } else {\n      boostDetails.add(Explanation.match(\n          (float) scoringData.userLangMult,\n          \"[x] User LangMult: \" + userLangDetails.toString()));\n    }\n\n    // Age decay\n    String ageDecayDetails = String.format(\n        \"age=%d seconds, slope=%.3f, base=%.1f, half-life=%.0f\",\n        scoringData.tweetAgeInSeconds, params.ageDecaySlope,\n        params.ageDecayBase, params.ageDecayHalflife);\n    if (params.useAgeDecay) {\n      boostDetails.add(Explanation.match(\n          (float) scoringData.ageDecayMult, \"[x] AgeDecay: \" + ageDecayDetails));\n    } else {\n      boostDetails.add(Explanation.match(1.0f, \"Age decay disabled: \" + ageDecayDetails));\n    }\n\n    // Score adjuster\n    boostDetails.add(Explanation.match(SCORE_ADJUSTER, \"[x] score adjuster\"));\n\n    Explanation boostCombo = isHit\n        ? Explanation.match((float) scoringData.scoreAfterBoost,\n          \"(MATCH) After Boosts and Demotions:\", boostDetails)\n        : Explanation.noMatch(\"After Boosts and Demotions:\", boostDetails);\n\n    details.add(boostCombo);\n  }\n\n  @Override\n  protected Explanation doExplain(float luceneQueryScore) throws IOException {\n    // Run the scorer again and get the explanation.\n    ExplanationWrapper explanation = new ExplanationWrapper();\n    scoreInternal(luceneQueryScore, explanation);\n    return explanation.explanation;\n  }\n\n  @Override\n  public void populateResultMetadataBasedOnScoringData(\n      ThriftSearchResultMetadataOptions options,\n      ThriftSearchResultMetadata metadata,\n      LinearScoringData data) throws IOException {\n    metadata.setResultType(searchResultType);\n    metadata.setScore(data.scoreReturned);\n    metadata.setFromUserId(data.fromUserId);\n\n    if (data.isTrusted) {\n      metadata.setIsTrusted(true);\n    }\n    if (data.isFollow) {\n      metadata.setIsFollow(true);\n    }\n    if (data.skipReason != SkipReason.NOT_SKIPPED) {\n      metadata.setSkipped(true);\n    }\n    if ((data.isRetweet || (params.getInReplyToStatusId && data.isReply))\n        && data.sharedStatusId != LinearScoringData.UNSET_SIGNAL_VALUE) {\n      metadata.setSharedStatusId(data.sharedStatusId);\n    }\n    if (data.hasCard) {\n      metadata.setCardType(data.cardType);\n    }\n\n    // Optional features.  Note: other optional metadata is populated by\n    // AbstractRelevanceCollector, not the scoring function.\n\n    if (options.isGetLuceneScore()) {\n      metadata.setLuceneScore(data.luceneScore);\n    }\n    if (options.isGetReferencedTweetAuthorId()\n        && data.referenceAuthorId != LinearScoringData.UNSET_SIGNAL_VALUE) {\n      metadata.setReferencedTweetAuthorId(data.referenceAuthorId);\n    }\n\n    if (options.isGetMediaBits()) {\n      metadata.setHasConsumerVideo(data.hasConsumerVideo);\n      metadata.setHasProVideo(data.hasProVideo);\n      metadata.setHasVine(data.hasVine);\n      metadata.setHasPeriscope(data.hasPeriscope);\n      boolean hasNativeVideo =\n          data.hasConsumerVideo || data.hasProVideo || data.hasVine || data.hasPeriscope;\n      metadata.setHasNativeVideo(hasNativeVideo);\n      metadata.setHasNativeImage(data.hasNativeImage);\n    }\n\n    metadata\n        .setIsOffensive(data.isOffensive)\n        .setIsReply(data.isReply)\n        .setIsRetweet(data.isRetweet)\n        .setHasLink(data.hasUrl)\n        .setHasTrend(data.hasTrend)\n        .setHasMultipleHashtagsOrTrends(data.hasMultipleHashtagsOrTrends)\n        .setRetweetCount((int) data.retweetCountPostLog2)\n        .setFavCount((int) data.favCountPostLog2)\n        .setReplyCount((int) data.replyCountPostLog2)\n        .setEmbedsImpressionCount((int) data.embedsImpressionCount)\n        .setEmbedsUrlCount((int) data.embedsUrlCount)\n        .setVideoViewCount((int) data.videoViewCount)\n        .setResultType(searchResultType)\n        .setFromVerifiedAccount(data.isFromVerifiedAccount)\n        .setIsUserSpam(data.isUserSpam)\n        .setIsUserNSFW(data.isUserNSFW)\n        .setIsUserBot(data.isUserBot)\n        .setHasImage(data.hasImageUrl)\n        .setHasVideo(data.hasVideoUrl)\n        .setHasNews(data.hasNewsUrl)\n        .setHasCard(data.hasCard)\n        .setHasVisibleLink(data.hasVisibleLink)\n        .setParusScore(data.parusScore)\n        .setTextScore(data.textScore)\n        .setUserRep(data.userRep)\n        .setTokenAt140DividedByNumTokensBucket(data.tokenAt140DividedByNumTokensBucket);\n\n    if (!metadata.isSetExtraMetadata()) {\n      metadata.setExtraMetadata(new ThriftSearchResultExtraMetadata());\n    }\n    ThriftSearchResultExtraMetadata extraMetadata = metadata.getExtraMetadata();\n\n    // Promotion/Demotion features\n    extraMetadata.setUserLangScore(data.userLangMult)\n        .setHasDifferentLang(data.hasDifferentLang)\n        .setHasEnglishTweetAndDifferentUILang(data.hasEnglishTweetAndDifferentUILang)\n        .setHasEnglishUIAndDifferentTweetLang(data.hasEnglishUIAndDifferentTweetLang)\n        .setHasQuote(data.hasQuote)\n        .setQuotedCount((int) data.quotedCount)\n        .setWeightedRetweetCount((int) data.weightedRetweetCount)\n        .setWeightedReplyCount((int) data.weightedReplyCount)\n        .setWeightedFavCount((int) data.weightedFavCount)\n        .setWeightedQuoteCount((int) data.weightedQuoteCount)\n        .setQuerySpecificScore(data.querySpecificScore)\n        .setAuthorSpecificScore(data.authorSpecificScore)\n        .setRetweetCountV2((int) data.retweetCountV2)\n        .setFavCountV2((int) data.favCountV2)\n        .setReplyCountV2((int) data.replyCountV2)\n        .setIsComposerSourceCamera(data.isComposerSourceCamera)\n        .setFromBlueVerifiedAccount(data.isFromBlueVerifiedAccount);\n\n    // Health model scores features\n    extraMetadata\n        .setToxicityScore(data.toxicityScore)\n        .setPBlockScore(data.pBlockScore)\n        .setPSpammyTweetScore(data.pSpammyTweetScore)\n        .setPReportedTweetScore(data.pReportedTweetScore)\n        .setSpammyTweetContentScore(data.spammyTweetContentScore)\n        .setExperimentalHealthModelScore1(data.experimentalHealthModelScore1)\n        .setExperimentalHealthModelScore2(data.experimentalHealthModelScore2)\n        .setExperimentalHealthModelScore3(data.experimentalHealthModelScore3)\n        .setExperimentalHealthModelScore4(data.experimentalHealthModelScore4);\n\n    // Return all extra features for clients to consume.\n    if (options.isGetAllFeatures()) {\n      extraMetadata.setIsSensitiveContent(data.isSensitiveContent)\n          .setHasMultipleMediaFlag(data.hasMultipleMediaFlag)\n          .setProfileIsEggFlag(data.profileIsEggFlag)\n          .setIsUserNewFlag(data.isUserNewFlag)\n          .setNumMentions(data.numMentions)\n          .setNumHashtags(data.numHashtags)\n          .setLinkLanguage(data.linkLanguage)\n          .setPrevUserTweetEngagement(data.prevUserTweetEngagement);\n    }\n\n    // Set features in new Feature Access API format, in the future this will be the only part\n    // needed in this method, we don't need to set any other metadata fields any more.\n    if (options.isReturnSearchResultFeatures()) {\n      // If the features are unset, and they were requested, then we can retrieve them. If they are\n      // already set, then we don't need to re-read the document features, and the reader\n      // is probably positioned over the wrong document so it will return incorrect results.\n      if (!extraMetadata.isSetFeatures()) {\n        // We ignore all features with default values when returning them in the response,\n        // because it saves a lot of network bandwidth.\n        ThriftSearchResultFeatures features = createFeaturesForDocument(data, true).getFeatures();\n        extraMetadata.setFeatures(features);\n      }\n\n      // The raw score may have changed since we created the features, so we should update it.\n      extraMetadata.getFeatures().getDoubleValues()\n          .put(ExternalTweetFeature.RAW_EARLYBIRD_SCORE.getId(), data.scoreFinal);\n    }\n\n    metadata\n        .setIsSelfTweet(data.isSelfTweet)\n        .setIsUserAntiSocial(data.isUserAntiSocial);\n  }\n\n  /**\n   * Create earlybird basic features and dervied features for current document.\n   * @return a FeatureHandler object where you can keep adding extra feature values, or you can\n   * call .getFeatures() on it to get a Thrift object to return.\n   */\n  protected FeatureHandler createFeaturesForDocument(\n      LinearScoringData data, boolean ignoreDefaultValues) throws IOException {\n    ThriftSearchResultFeatures features = documentFeatures.getSearchResultFeatures(getSchema());\n    if (!ignoreDefaultValues) {\n      setDefaultFeatureValues(features);\n    }\n\n    // add derived features\n    return new FeatureHandler(features, ignoreDefaultValues)\n        .addDouble(ExternalTweetFeature.LUCENE_SCORE, data.luceneScore)\n        .addInt(ExternalTweetFeature.TWEET_AGE_IN_SECS, data.tweetAgeInSeconds)\n        .addBoolean(ExternalTweetFeature.IS_SELF_TWEET, data.isSelfTweet)\n        .addBoolean(ExternalTweetFeature.IS_FOLLOW_RETWEET, data.isFollow && data.isRetweet)\n        .addBoolean(ExternalTweetFeature.IS_TRUSTED_RETWEET, data.isTrusted && data.isRetweet)\n        .addBoolean(ExternalTweetFeature.AUTHOR_IS_FOLLOW, data.isFollow)\n        .addBoolean(ExternalTweetFeature.AUTHOR_IS_TRUSTED, data.isTrusted)\n        .addBoolean(ExternalTweetFeature.AUTHOR_IS_ANTISOCIAL, data.isUserAntiSocial)\n        .addBoolean(ExternalTweetFeature.HAS_DIFF_LANG, data.hasDifferentLang)\n        .addBoolean(ExternalTweetFeature.HAS_ENGLISH_TWEET_DIFF_UI_LANG,\n            data.hasEnglishTweetAndDifferentUILang)\n        .addBoolean(ExternalTweetFeature.HAS_ENGLISH_UI_DIFF_TWEET_LANG,\n            data.hasEnglishUIAndDifferentTweetLang)\n        .addDouble(ExternalTweetFeature.SEARCHER_LANG_SCORE, data.userLangMult)\n        .addDouble(ExternalTweetFeature.QUERY_SPECIFIC_SCORE, data.querySpecificScore)\n        .addDouble(ExternalTweetFeature.AUTHOR_SPECIFIC_SCORE, data.authorSpecificScore);\n  }\n\n  /**\n   * Adds default values for most numeric features that do not have a value set yet in the given\n   * ThriftSearchResultFeatures instance.\n   *\n   * This method is needed because some models do not work properly with missing features. Instead,\n   * they expect all features to be present even if they are unset (their values are 0).\n   */\n  protected void setDefaultFeatureValues(ThriftSearchResultFeatures features) {\n    for (Map.Entry<Integer, ThriftSearchFeatureSchemaEntry> entry\n             : getSchema().getSearchFeatureSchema().getEntries().entrySet()) {\n      int featureId = entry.getKey();\n      ThriftSearchFeatureSchemaEntry schemaEntry = entry.getValue();\n      if (shouldSetDefaultValueForFeature(schemaEntry.getFeatureType(), featureId)) {\n        switch (schemaEntry.getFeatureType()) {\n          case INT32_VALUE:\n            features.getIntValues().putIfAbsent(featureId, 0);\n            break;\n          case LONG_VALUE:\n            features.getLongValues().putIfAbsent(featureId, 0L);\n            break;\n          case DOUBLE_VALUE:\n            features.getDoubleValues().putIfAbsent(featureId, 0.0);\n            break;\n          default:\n            throw new IllegalArgumentException(\n                \"Should set default values only for integer, long or double features. Instead, \"\n                + \"found feature \" + featureId + \" of type \" + schemaEntry.getFeatureType());\n        }\n      }\n    }\n  }\n\n  protected void overrideFeatureValues(ThriftSearchResultFeatures features,\n                                       ThriftSearchResultFeatures overrideFeatures) {\n    LOG.info(\"Features before override {}\", features);\n    if (overrideFeatures.isSetIntValues()) {\n      overrideFeatures.getIntValues().forEach(features::putToIntValues);\n    }\n    if (overrideFeatures.isSetLongValues()) {\n      overrideFeatures.getLongValues().forEach(features::putToLongValues);\n    }\n    if (overrideFeatures.isSetDoubleValues()) {\n      overrideFeatures.getDoubleValues().forEach(features::putToDoubleValues);\n    }\n    if (overrideFeatures.isSetBoolValues()) {\n      overrideFeatures.getBoolValues().forEach(features::putToBoolValues);\n    }\n    if (overrideFeatures.isSetStringValues()) {\n      overrideFeatures.getStringValues().forEach(features::putToStringValues);\n    }\n    if (overrideFeatures.isSetBytesValues()) {\n      overrideFeatures.getBytesValues().forEach(features::putToBytesValues);\n    }\n    if (overrideFeatures.isSetFeatureStoreDiscreteValues()) {\n      overrideFeatures.getFeatureStoreDiscreteValues().forEach(\n          features::putToFeatureStoreDiscreteValues);\n    }\n    if (overrideFeatures.isSetSparseBinaryValues()) {\n      overrideFeatures.getSparseBinaryValues().forEach(features::putToSparseBinaryValues);\n    }\n    if (overrideFeatures.isSetSparseContinuousValues()) {\n      overrideFeatures.getSparseContinuousValues().forEach(features::putToSparseContinuousValues);\n    }\n    if (overrideFeatures.isSetGeneralTensorValues()) {\n      overrideFeatures.getGeneralTensorValues().forEach(features::putToGeneralTensorValues);\n    }\n    if (overrideFeatures.isSetStringTensorValues()) {\n      overrideFeatures.getStringTensorValues().forEach(features::putToStringTensorValues);\n    }\n    LOG.info(\"Features after override {}\", features);\n  }\n\n  /**\n   * Check if a feature is eligible to have its default value automatically set when absent.\n   * We have a similar logic for building data record.\n   */\n  private static boolean shouldSetDefaultValueForFeature(\n      ThriftSearchFeatureType type, int featureId) {\n    return ALLOWED_TYPES_FOR_DEFAULT_FEATURE_VALUES.contains(type)\n        && !NUMERIC_FEATURES_FOR_WHICH_DEFAULTS_SHOULD_NOT_BE_SET.contains(featureId)\n        && (ExternalTweetFeature.EARLYBIRD_INDEXED_FEATURE_IDS.contains(featureId)\n            || ExternalTweetFeature.EARLYBIRD_DERIVED_FEATURE_IDS.contains(featureId));\n  }\n\n  @Override\n  public void updateRelevanceStats(ThriftSearchResultsRelevanceStats relevanceStats) {\n    if (relevanceStats == null) {\n      return;\n    }\n\n    LinearScoringData data = getScoringDataForCurrentDocument();\n\n    if (data.tweetAgeInSeconds > relevanceStats.getOldestScoredTweetAgeInSeconds()) {\n      relevanceStats.setOldestScoredTweetAgeInSeconds(data.tweetAgeInSeconds);\n    }\n    relevanceStats.setNumScored(relevanceStats.getNumScored() + 1);\n    if (data.scoreReturned == SKIP_HIT) {\n      relevanceStats.setNumSkipped(relevanceStats.getNumSkipped() + 1);\n      switch(data.skipReason) {\n        case ANTIGAMING:\n          relevanceStats.setNumSkippedForAntiGaming(\n              relevanceStats.getNumSkippedForAntiGaming() + 1);\n          break;\n        case LOW_REPUTATION:\n          relevanceStats.setNumSkippedForLowReputation(\n              relevanceStats.getNumSkippedForLowReputation() + 1);\n          break;\n        case LOW_TEXT_SCORE:\n          relevanceStats.setNumSkippedForLowTextScore(\n              relevanceStats.getNumSkippedForLowTextScore() + 1);\n          break;\n        case SOCIAL_FILTER:\n          relevanceStats.setNumSkippedForSocialFilter(\n              relevanceStats.getNumSkippedForSocialFilter() + 1);\n          break;\n        case LOW_FINAL_SCORE:\n          relevanceStats.setNumSkippedForLowFinalScore(\n              relevanceStats.getNumSkippedForLowFinalScore() + 1);\n          break;\n        case LOW_RETWEET_COUNT:\n          break;\n        default:\n          LOG.warn(\"Unknown SkipReason: \" + data.skipReason);\n      }\n    }\n\n    if (data.isFollow) {\n      relevanceStats.setNumFromDirectFollows(relevanceStats.getNumFromDirectFollows() + 1);\n    }\n    if (data.isTrusted) {\n      relevanceStats.setNumFromTrustedCircle(relevanceStats.getNumFromTrustedCircle() + 1);\n    }\n    if (data.isReply) {\n      relevanceStats.setNumReplies(relevanceStats.getNumReplies() + 1);\n      if (data.isTrusted) {\n        relevanceStats.setNumRepliesTrusted(relevanceStats.getNumRepliesTrusted() + 1);\n      } else if (!data.isFollow) {\n        relevanceStats.setNumRepliesOutOfNetwork(relevanceStats.getNumRepliesOutOfNetwork() + 1);\n      }\n    }\n    if (data.isSelfTweet) {\n      relevanceStats.setNumSelfTweets(relevanceStats.getNumSelfTweets() + 1);\n    }\n    if (data.hasImageUrl || data.hasVideoUrl) {\n      relevanceStats.setNumWithMedia(relevanceStats.getNumWithMedia() + 1);\n    }\n    if (data.hasNewsUrl) {\n      relevanceStats.setNumWithNews(relevanceStats.getNumWithNews() + 1);\n    }\n    if (data.isUserSpam) {\n      relevanceStats.setNumSpamUser(relevanceStats.getNumSpamUser() + 1);\n    }\n    if (data.isUserNSFW) {\n      relevanceStats.setNumOffensive(relevanceStats.getNumOffensive() + 1);\n    }\n    if (data.isUserBot) {\n      relevanceStats.setNumBot(relevanceStats.getNumBot() + 1);\n    }\n  }\n\n  @VisibleForTesting\n  static final class ExplanationWrapper {\n    private Explanation explanation;\n\n    public Explanation getExplanation() {\n      return explanation;\n    }\n\n    @Override\n    public String toString() {\n      return explanation.toString();\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/LegacyScoreAccumulator.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport com.twitter.search.common.util.ml.prediction_engine.BaseLegacyScoreAccumulator;\nimport com.twitter.search.common.util.ml.prediction_engine.LightweightLinearModel;\nimport com.twitter.search.earlybird.search.relevance.LinearScoringData;\nimport com.twitter.search.modeling.tweet_ranking.TweetScoringFeatures;\n\n/**\n * Legacy score accumulator in Earlybird with specific features added.\n * This class is created to avoid adding LinearScoringData as a dependency to search's common ML\n * library.\n *\n * @deprecated This class is retired and we suggest to switch to SchemaBasedScoreAccumulator.\n */\n@Deprecated\npublic class LegacyScoreAccumulator extends BaseLegacyScoreAccumulator<LinearScoringData> {\n  /**\n   * Constructs with a model and LinearScoringData\n   */\n  LegacyScoreAccumulator(LightweightLinearModel model) {\n    super(model);\n  }\n\n  /**\n   * Update the accumulator score with features, after this function the score should already\n   * be computed.\n   *\n   * @deprecated This function is retired and we suggest to switch to updateScoresWithFeatures in\n   * SchemaBasedScoreAccumulator.\n   */\n  @Override\n  @Deprecated\n  protected void updateScoreWithFeatures(LinearScoringData data) {\n    addContinuousFeature(TweetScoringFeatures.LUCENE_SCORE, data.luceneScore);\n    addContinuousFeature(TweetScoringFeatures.TEXT_SCORE, data.textScore);\n    addContinuousFeature(TweetScoringFeatures.TWEET_AGE_IN_SECONDS, data.tweetAgeInSeconds);\n    addContinuousFeature(TweetScoringFeatures.REPLY_COUNT, data.replyCountPostLog2);\n    addContinuousFeature(TweetScoringFeatures.RETWEET_COUNT, data.retweetCountPostLog2);\n    addContinuousFeature(TweetScoringFeatures.FAV_COUNT, data.favCountPostLog2);\n    addContinuousFeature(TweetScoringFeatures.REPLY_COUNT_V2, data.replyCountV2);\n    addContinuousFeature(TweetScoringFeatures.RETWEET_COUNT_V2, data.retweetCountV2);\n    addContinuousFeature(TweetScoringFeatures.FAV_COUNT_V2, data.favCountV2);\n    addContinuousFeature(TweetScoringFeatures.EMBEDS_IMPRESSION_COUNT,\n        data.getEmbedsImpressionCount(false));\n    addContinuousFeature(TweetScoringFeatures.EMBEDS_URL_COUNT, data.getEmbedsUrlCount(false));\n    addContinuousFeature(TweetScoringFeatures.VIDEO_VIEW_COUNT, data.getVideoViewCount(false));\n    addContinuousFeature(TweetScoringFeatures.QUOTED_COUNT, data.quotedCount);\n    addContinuousFeature(TweetScoringFeatures.WEIGHTED_RETWEET_COUNT, data.weightedRetweetCount);\n    addContinuousFeature(TweetScoringFeatures.WEIGHTED_REPLY_COUNT, data.weightedReplyCount);\n    addContinuousFeature(TweetScoringFeatures.WEIGHTED_FAV_COUNT, data.weightedFavCount);\n    addContinuousFeature(TweetScoringFeatures.WEIGHTED_QUOTE_COUNT, data.weightedQuoteCount);\n    addBinaryFeature(TweetScoringFeatures.HAS_URL, data.hasUrl);\n    addBinaryFeature(TweetScoringFeatures.HAS_CARD, data.hasCard);\n    addBinaryFeature(TweetScoringFeatures.HAS_VINE, data.hasVine);\n    addBinaryFeature(TweetScoringFeatures.HAS_PERISCOPE, data.hasPeriscope);\n    addBinaryFeature(TweetScoringFeatures.HAS_NATIVE_IMAGE, data.hasNativeImage);\n    addBinaryFeature(TweetScoringFeatures.HAS_IMAGE_URL, data.hasImageUrl);\n    addBinaryFeature(TweetScoringFeatures.HAS_NEWS_URL, data.hasNewsUrl);\n    addBinaryFeature(TweetScoringFeatures.HAS_VIDEO_URL, data.hasVideoUrl);\n    addBinaryFeature(TweetScoringFeatures.HAS_CONSUMER_VIDEO, data.hasConsumerVideo);\n    addBinaryFeature(TweetScoringFeatures.HAS_PRO_VIDEO, data.hasProVideo);\n    addBinaryFeature(TweetScoringFeatures.HAS_QUOTE, data.hasQuote);\n    addBinaryFeature(TweetScoringFeatures.HAS_TREND, data.hasTrend);\n    addBinaryFeature(TweetScoringFeatures.HAS_MULTIPLE_HASHTAGS_OR_TRENDS,\n        data.hasMultipleHashtagsOrTrends);\n    addBinaryFeature(TweetScoringFeatures.IS_OFFENSIVE, data.isOffensive);\n    addBinaryFeature(TweetScoringFeatures.IS_REPLY, data.isReply);\n    addBinaryFeature(TweetScoringFeatures.IS_RETWEET, data.isRetweet);\n    addBinaryFeature(TweetScoringFeatures.IS_SELF_TWEET, data.isSelfTweet);\n    addBinaryFeature(TweetScoringFeatures.IS_FOLLOW_RETWEET, data.isRetweet & data.isFollow);\n    addBinaryFeature(TweetScoringFeatures.IS_TRUSTED_RETWEET, data.isRetweet & data.isTrusted);\n    addContinuousFeature(TweetScoringFeatures.QUERY_SPECIFIC_SCORE, data.querySpecificScore);\n    addContinuousFeature(TweetScoringFeatures.AUTHOR_SPECIFIC_SCORE, data.authorSpecificScore);\n    addBinaryFeature(TweetScoringFeatures.AUTHOR_IS_FOLLOW, data.isFollow);\n    addBinaryFeature(TweetScoringFeatures.AUTHOR_IS_TRUSTED, data.isTrusted);\n    addBinaryFeature(TweetScoringFeatures.AUTHOR_IS_VERIFIED, data.isFromVerifiedAccount);\n    addBinaryFeature(TweetScoringFeatures.AUTHOR_IS_NSFW, data.isUserNSFW);\n    addBinaryFeature(TweetScoringFeatures.AUTHOR_IS_SPAM, data.isUserSpam);\n    addBinaryFeature(TweetScoringFeatures.AUTHOR_IS_BOT, data.isUserBot);\n    addBinaryFeature(TweetScoringFeatures.AUTHOR_IS_ANTISOCIAL, data.isUserAntiSocial);\n    addContinuousFeature(TweetScoringFeatures.AUTHOR_REPUTATION, data.userRep);\n    addContinuousFeature(TweetScoringFeatures.SEARCHER_LANG_SCORE, data.userLangMult);\n    addBinaryFeature(TweetScoringFeatures.HAS_DIFFERENT_LANG, data.hasDifferentLang);\n    addBinaryFeature(TweetScoringFeatures.HAS_ENGLISH_TWEET_AND_DIFFERENT_UI_LANG,\n        data.hasEnglishTweetAndDifferentUILang);\n    addBinaryFeature(TweetScoringFeatures.HAS_ENGLISH_UI_AND_DIFFERENT_TWEET_LANG,\n        data.hasEnglishUIAndDifferentTweetLang);\n    addBinaryFeature(TweetScoringFeatures.IS_SENSITIVE_CONTENT, data.isSensitiveContent);\n    addBinaryFeature(TweetScoringFeatures.HAS_MULTIPLE_MEDIA, data.hasMultipleMediaFlag);\n    addBinaryFeature(TweetScoringFeatures.AUTHOR_IS_PROFILE_EGG, data.profileIsEggFlag);\n    addBinaryFeature(TweetScoringFeatures.AUTHOR_IS_NEW, data.isUserNewFlag);\n    addContinuousFeature(TweetScoringFeatures.MENTIONS_COUNT, data.numMentions);\n    addContinuousFeature(TweetScoringFeatures.HASHTAGS_COUNT, data.numHashtags);\n    addContinuousFeature(TweetScoringFeatures.LINK_LANGUAGE_ID, data.linkLanguage);\n    addContinuousFeature(TweetScoringFeatures.LANGUAGE_ID, data.tweetLangId);\n    addBinaryFeature(TweetScoringFeatures.HAS_VISIBLE_LINK, data.hasVisibleLink);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/LinearScoringFunction.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport com.google.common.collect.Lists;\n\nimport org.apache.lucene.search.Explanation;\n\nimport com.twitter.search.common.relevance.features.MutableFeatureNormalizers;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.search.AntiGamingFilter;\nimport com.twitter.search.earlybird.search.relevance.LinearScoringData;\nimport com.twitter.search.earlybird.search.relevance.LinearScoringParams;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultType;\n\n/**\n * Scoring function that uses the weights and boosts provided in the scoring parameters from the\n * request.\n */\npublic class LinearScoringFunction extends FeatureBasedScoringFunction {\n  private static final double BASE_SCORE = 0.0001;\n\n  public LinearScoringFunction(\n      ImmutableSchemaInterface schema,\n      ThriftSearchQuery searchQuery,\n      AntiGamingFilter antiGamingFilter,\n      ThriftSearchResultType searchResultType,\n      UserTable userTable) throws IOException {\n    super(\"LinearScoringFunction\", schema, searchQuery, antiGamingFilter, searchResultType,\n        userTable);\n  }\n\n  @Override\n  protected double computeScore(LinearScoringData data, boolean forExplanation) throws IOException {\n    double score = BASE_SCORE;\n\n    data.luceneContrib = params.useLuceneScoreAsBoost\n        ? 0.0 : params.luceneWeight * data.luceneScore;\n\n    data.reputationContrib = params.reputationWeight * data.userRep;\n    data.textScoreContrib = params.textScoreWeight * data.textScore;\n    data.parusContrib = params.parusWeight * data.parusScore;\n\n    // contributions from engagement counters. Note that we have \"true\" argument for all getters,\n    // which means all values will get scaled down for scoring, they were unbounded in raw form.\n    data.retweetContrib = params.retweetWeight * data.retweetCountPostLog2;\n    data.favContrib = params.favWeight * data.favCountPostLog2;\n    data.replyContrib = params.replyWeight * data.replyCountPostLog2;\n    data.embedsImpressionContrib =\n        params.embedsImpressionWeight * data.getEmbedsImpressionCount(true);\n    data.embedsUrlContrib =\n        params.embedsUrlWeight * data.getEmbedsUrlCount(true);\n    data.videoViewContrib =\n        params.videoViewWeight * data.getVideoViewCount(true);\n    data.quotedContrib =\n        params.quotedCountWeight * data.quotedCount;\n\n    for (int i = 0; i < LinearScoringData.MAX_OFFLINE_EXPERIMENTAL_FIELDS; i++) {\n      data.offlineExpFeatureContributions[i] =\n          params.rankingOfflineExpWeights[i] * data.offlineExpFeatureValues[i];\n    }\n\n    data.hasUrlContrib = params.urlWeight * (data.hasUrl ? 1.0 : 0.0);\n    data.isReplyContrib = params.isReplyWeight * (data.isReply ? 1.0 : 0.0);\n    data.isFollowRetweetContrib =\n        params.followRetweetWeight * (data.isRetweet && data.isFollow ? 1.0 : 0.0);\n    data.isTrustedRetweetContrib =\n        params.trustedRetweetWeight * (data.isRetweet && data.isTrusted ? 1.0 : 0.0);\n    double replyCountOriginal = getUnscaledReplyCountFeatureValue();\n    data.multipleReplyContrib = params.multipleReplyWeight\n        * (replyCountOriginal < params.multipleReplyMinVal ? 0.0 : replyCountOriginal);\n\n    // We directly the query specific score as the contribution below as it doesn't need a weight\n    // for contribution computation.\n    score += data.luceneContrib\n        + data.reputationContrib\n        + data.textScoreContrib\n        + data.replyContrib\n        + data.multipleReplyContrib\n        + data.retweetContrib\n        + data.favContrib\n        + data.parusContrib\n        + data.embedsImpressionContrib\n        + data.embedsUrlContrib\n        + data.videoViewContrib\n        + data.quotedContrib\n        + data.hasUrlContrib\n        + data.isReplyContrib\n        + data.isFollowRetweetContrib\n        + data.isTrustedRetweetContrib\n        + data.querySpecificScore\n        + data.authorSpecificScore;\n\n    for (int i = 0; i < LinearScoringData.MAX_OFFLINE_EXPERIMENTAL_FIELDS; i++) {\n      score += data.offlineExpFeatureContributions[i];\n    }\n\n    return score;\n  }\n\n  /**\n   * Generates the explanation for the linear score.\n   */\n  @Override\n  protected void generateExplanationForScoring(\n      LinearScoringData scoringData, boolean isHit, List<Explanation> details) throws IOException {\n    // 1. Linear components\n    final List<Explanation> linearDetails = Lists.newArrayList();\n    addLinearElementExplanation(\n        linearDetails, \"[LuceneQueryScore]\",\n        params.luceneWeight, scoringData.luceneScore, scoringData.luceneContrib);\n    if (scoringData.hasCard) {\n      if (scoringData.cardAuthorMatchBoostApplied) {\n        linearDetails.add(Explanation.match(\n            (float) params.cardAuthorMatchBoosts[scoringData.cardType],\n            \"[x] card author match boost\"));\n      }\n      if (scoringData.cardDescriptionMatchBoostApplied) {\n        linearDetails.add(Explanation.match(\n            (float) params.cardDescriptionMatchBoosts[scoringData.cardType],\n            \"[x] card description match boost\"));\n      }\n      if (scoringData.cardDomainMatchBoostApplied) {\n        linearDetails.add(Explanation.match(\n            (float) params.cardDomainMatchBoosts[scoringData.cardType],\n            \"[x] card domain match boost\"));\n      }\n      if (scoringData.cardTitleMatchBoostApplied) {\n        linearDetails.add(Explanation.match(\n            (float) params.cardTitleMatchBoosts[scoringData.cardType],\n            \"[x] card title match boost\"));\n      }\n    }\n    addLinearElementExplanation(\n        linearDetails, \"reputation\",\n        params.reputationWeight, scoringData.userRep, scoringData.reputationContrib);\n    addLinearElementExplanation(\n        linearDetails, \"text score\",\n        params.textScoreWeight, scoringData.textScore, scoringData.textScoreContrib);\n    addLinearElementExplanation(\n        linearDetails, \"reply count (log2)\",\n        params.replyWeight, scoringData.replyCountPostLog2, scoringData.replyContrib);\n    addLinearElementExplanation(\n        linearDetails, \"multi reply\",\n        params.multipleReplyWeight,\n        getUnscaledReplyCountFeatureValue() > params.multipleReplyMinVal ? 1 : 0,\n        scoringData.multipleReplyContrib);\n    addLinearElementExplanation(\n        linearDetails, \"retweet count (log2)\",\n        params.retweetWeight, scoringData.retweetCountPostLog2, scoringData.retweetContrib);\n    addLinearElementExplanation(\n        linearDetails, \"fav count (log2)\",\n        params.favWeight, scoringData.favCountPostLog2, scoringData.favContrib);\n    addLinearElementExplanation(\n        linearDetails, \"parus score\",\n        params.parusWeight, scoringData.parusScore, scoringData.parusContrib);\n    for (int i = 0; i < LinearScoringData.MAX_OFFLINE_EXPERIMENTAL_FIELDS; i++) {\n      if (params.rankingOfflineExpWeights[i] != LinearScoringParams.DEFAULT_FEATURE_WEIGHT) {\n        addLinearElementExplanation(linearDetails,\n            \"ranking exp score offline experimental #\" + i,\n            params.rankingOfflineExpWeights[i], scoringData.offlineExpFeatureValues[i],\n            scoringData.offlineExpFeatureContributions[i]);\n      }\n    }\n    addLinearElementExplanation(linearDetails,\n        \"embedded tweet impression count\",\n        params.embedsImpressionWeight, scoringData.getEmbedsImpressionCount(false),\n        scoringData.embedsImpressionContrib);\n    addLinearElementExplanation(linearDetails,\n        \"embedded tweet url count\",\n        params.embedsUrlWeight, scoringData.getEmbedsUrlCount(false),\n        scoringData.embedsUrlContrib);\n    addLinearElementExplanation(linearDetails,\n        \"video view count\",\n        params.videoViewWeight, scoringData.getVideoViewCount(false),\n        scoringData.videoViewContrib);\n    addLinearElementExplanation(linearDetails,\n        \"quoted count\",\n        params.quotedCountWeight, scoringData.quotedCount, scoringData.quotedContrib);\n\n    addLinearElementExplanation(\n        linearDetails, \"has url\", params.urlWeight, scoringData.hasUrl ? 1.0 : 0.0,\n        scoringData.hasUrlContrib);\n\n    addLinearElementExplanation(\n        linearDetails, \"is reply\", params.isReplyWeight,\n        scoringData.isReply ? 1.0 : 0.0, scoringData.isReplyContrib);\n    addLinearElementExplanation(\n        linearDetails, \"is follow retweet\", params.followRetweetWeight,\n        scoringData.isRetweet && scoringData.isFollow ? 1.0 : 0.0,\n        scoringData.isFollowRetweetContrib);\n    addLinearElementExplanation(\n        linearDetails, \"is trusted retweet\", params.trustedRetweetWeight,\n        scoringData.isRetweet && scoringData.isTrusted ? 1.0 : 0.0,\n        scoringData.isTrustedRetweetContrib);\n\n    if (scoringData.querySpecificScore != 0.0) {\n      linearDetails.add(Explanation.match((float) scoringData.querySpecificScore,\n          \"[+] query specific score adjustment\"));\n    }\n    if (scoringData.authorSpecificScore != 0.0) {\n      linearDetails.add(Explanation.match((float) scoringData.authorSpecificScore,\n          \"[+] author specific score adjustment\"));\n    }\n\n\n    Explanation linearCombo = isHit\n        ? Explanation.match((float) scoringData.scoreBeforeBoost,\n          \"(MATCH) Linear components, sum of:\", linearDetails)\n        : Explanation.noMatch(\"Linear components, sum of:\", linearDetails);\n\n\n    details.add(linearCombo);\n  }\n\n  private void addLinearElementExplanation(List<Explanation> explanation,\n                                           String name,\n                                           double weight,\n                                           double componentValue,\n                                           double contrib) {\n    if (contrib == 0.0) {\n      return;\n    }\n    explanation.add(\n        Explanation.match((float) contrib,\n            String.format(\"[+] %s=%.3f weight=%.3f\", name, componentValue, weight)));\n  }\n\n  private double getUnscaledReplyCountFeatureValue() throws IOException {\n    byte featureValue = (byte) documentFeatures.getFeatureValue(EarlybirdFieldConstant.REPLY_COUNT);\n    return MutableFeatureNormalizers.BYTE_NORMALIZER.unnormLowerBound(featureValue);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/ModelBasedScoringFunction.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport java.io.IOException;\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.base.Optional;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\n\nimport org.apache.lucene.search.Explanation;\n\nimport com.twitter.search.common.features.thrift.ThriftSearchResultFeatures;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.ranking.thriftjava.ThriftRankingParams;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.util.ml.prediction_engine.LightweightLinearModel;\nimport com.twitter.search.common.util.ml.prediction_engine.SchemaBasedScoreAccumulator;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.exception.ClientException;\nimport com.twitter.search.earlybird.ml.ScoringModelsManager;\nimport com.twitter.search.earlybird.search.AntiGamingFilter;\nimport com.twitter.search.earlybird.search.relevance.LinearScoringData;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultType;\n\n/**\n * Scoring function that uses the scoring models specified from the request.\n */\npublic class ModelBasedScoringFunction extends FeatureBasedScoringFunction {\n  private final SelectedModel[] selectedModels;\n  private final boolean useLogitScore;\n  private final boolean isSchemaBased;\n\n  private static final SearchCounter NUM_LEGACY_MODELS =\n      SearchCounter.export(\"scoring_function_num_legacy_models\");\n  private static final SearchCounter NUM_SCHEMA_BASED_MODELS =\n      SearchCounter.export(\"scoring_function_num_schema_based_models\");\n  private static final SearchCounter MIXED_MODEL_TYPES =\n      SearchCounter.export(\"scoring_function_mixed_model_types\");\n\n  public ModelBasedScoringFunction(\n      ImmutableSchemaInterface schema,\n      ThriftSearchQuery searchQuery,\n      AntiGamingFilter antiGamingFilter,\n      ThriftSearchResultType searchResultType,\n      UserTable userTable,\n      ScoringModelsManager scoringModelsManager\n  ) throws IOException, ClientException {\n\n    super(\"ModelBasedScoringFunction\", schema, searchQuery, antiGamingFilter, searchResultType,\n        userTable);\n\n    ThriftRankingParams rankingParams = searchQuery.getRelevanceOptions().getRankingParams();\n    Preconditions.checkNotNull(rankingParams);\n\n    if (rankingParams.getSelectedModelsSize() <= 0) {\n      throw new ClientException(\"Scoring type is MODEL_BASED but no models were selected\");\n    }\n\n    Map<String, Double> models = rankingParams.getSelectedModels();\n\n    selectedModels = new SelectedModel[models.size()];\n    int numSchemaBased = 0;\n    int i = 0;\n    for (Map.Entry<String, Double> nameAndWeight : models.entrySet()) {\n      Optional<LightweightLinearModel> model =\n          scoringModelsManager.getModel(nameAndWeight.getKey());\n      if (!model.isPresent()) {\n          throw new ClientException(String.format(\n              \"Scoring function is MODEL_BASED. Selected model '%s' not found\",\n              nameAndWeight.getKey()));\n      }\n      selectedModels[i] =\n          new SelectedModel(nameAndWeight.getKey(), nameAndWeight.getValue(), model.get());\n\n      if (selectedModels[i].model.isSchemaBased()) {\n        ++numSchemaBased;\n        NUM_SCHEMA_BASED_MODELS.increment();\n      } else {\n        NUM_LEGACY_MODELS.increment();\n      }\n      ++i;\n    }\n\n    // We should either see all models schema-based, or none of them so, if this is not the case,\n    // we log an error message and fall back to use just the first model, whatever it is.\n    if (numSchemaBased > 0 && numSchemaBased != selectedModels.length) {\n      MIXED_MODEL_TYPES.increment();\n      throw new ClientException(\n          \"You cannot mix schema-based and non-schema-based models in the same request, \"\n          + \"models are: \" + models.keySet());\n    }\n\n    isSchemaBased = selectedModels[0].model.isSchemaBased();\n    useLogitScore = rankingParams.isUseLogitScore();\n  }\n\n  @Override\n  protected double computeScore(LinearScoringData data, boolean forExplanation) throws IOException {\n    ThriftSearchResultFeatures features =\n        isSchemaBased ? createFeaturesForDocument(data, false).getFeatures() : null;\n\n    double score = 0;\n    for (SelectedModel selectedModel : selectedModels) {\n      double modelScore = isSchemaBased\n          ? new SchemaBasedScoreAccumulator(selectedModel.model).scoreWith(features, useLogitScore)\n          : new LegacyScoreAccumulator(selectedModel.model).scoreWith(data, useLogitScore);\n      score += selectedModel.weight * modelScore;\n    }\n\n    return score;\n  }\n\n  @Override\n  protected void generateExplanationForScoring(\n      LinearScoringData scoringData, boolean isHit, List<Explanation> details) throws IOException {\n    boolean schemaBased = selectedModels[0].model.isSchemaBased();\n    ThriftSearchResultFeatures features =\n        schemaBased ? createFeaturesForDocument(scoringData, false).getFeatures() : null;\n\n    // 1. Model-based score\n    final List<Explanation> modelExplanations = Lists.newArrayList();\n    float finalScore = 0;\n    for (SelectedModel selectedModel : selectedModels) {\n      double modelScore = schemaBased\n          ? new SchemaBasedScoreAccumulator(selectedModel.model).scoreWith(features, useLogitScore)\n          : new LegacyScoreAccumulator(selectedModel.model).scoreWith(scoringData, useLogitScore);\n      float weightedScore = (float) (selectedModel.weight * modelScore);\n      details.add(Explanation.match(\n          weightedScore, String.format(\"model=%s score=%.6f weight=%.3f useLogitScore=%s\",\n          selectedModel.name, modelScore, selectedModel.weight, useLogitScore)));\n      finalScore += weightedScore;\n    }\n\n    details.add(Explanation.match(\n        finalScore, String.format(\"Total model-based score (hit=%s)\", isHit), modelExplanations));\n  }\n\n  private static final class SelectedModel {\n    public final String name;\n    public final double weight;\n    public final LightweightLinearModel model;\n\n    private SelectedModel(String name, double weight, LightweightLinearModel model) {\n      this.name = name;\n      this.weight = weight;\n      this.model = model;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/RelevanceQuery.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport java.io.IOException;\nimport java.util.Objects;\nimport java.util.Set;\n\nimport javax.annotation.Nullable;\n\nimport org.apache.lucene.index.IndexReader;\nimport org.apache.lucene.index.LeafReaderContext;\nimport org.apache.lucene.index.Term;\nimport org.apache.lucene.search.Explanation;\nimport org.apache.lucene.search.IndexSearcher;\nimport org.apache.lucene.search.Query;\nimport org.apache.lucene.search.Scorer;\nimport org.apache.lucene.search.ScoreMode;\nimport org.apache.lucene.search.Weight;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.results.thriftjava.FieldHitAttribution;\n\n/**\n * A wrapper for a Lucene query which first computes Lucene's query score\n * and then delegates to a {@link ScoringFunction} for final score computation.\n */\npublic class RelevanceQuery extends Query {\n  private static final Logger LOG = LoggerFactory.getLogger(RelevanceQuery.class.getName());\n\n  protected final Query luceneQuery;\n  protected final ScoringFunction scoringFunction;\n\n  // True when the lucene query's score should be ignored for debug explanations.\n  protected final boolean ignoreLuceneQueryScoreExplanation;\n\n  public RelevanceQuery(Query luceneQuery, ScoringFunction scoringFunction) {\n    this(luceneQuery, scoringFunction, false);\n  }\n\n  public RelevanceQuery(Query luceneQuery,\n                        ScoringFunction scoringFunction,\n                        boolean ignoreLuceneQueryScoreExplanation) {\n    this.luceneQuery = luceneQuery;\n    this.scoringFunction = scoringFunction;\n    this.ignoreLuceneQueryScoreExplanation = ignoreLuceneQueryScoreExplanation;\n  }\n\n  public ScoringFunction getScoringFunction() {\n    return scoringFunction;\n  }\n\n  public Query getLuceneQuery() {\n    return luceneQuery;\n  }\n\n  @Override\n  public Query rewrite(IndexReader reader) throws IOException {\n    Query rewritten = luceneQuery.rewrite(reader);\n    if (rewritten == luceneQuery) {\n      return this;\n    }\n    return new RelevanceQuery(rewritten, scoringFunction, ignoreLuceneQueryScoreExplanation);\n  }\n\n  @Override\n  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost)\n      throws IOException {\n    Weight luceneWeight = luceneQuery.createWeight(searcher, scoreMode, boost);\n    if (luceneWeight == null) {\n      return null;\n    }\n    return new RelevanceWeight(searcher, luceneWeight);\n  }\n\n  public class RelevanceWeight extends Weight {\n    private final Weight luceneWeight;\n\n    public RelevanceWeight(IndexSearcher searcher, Weight luceneWeight) {\n      super(RelevanceQuery.this);\n      this.luceneWeight = luceneWeight;\n    }\n\n    @Override\n    public void extractTerms(Set<Term> terms) {\n      this.luceneWeight.extractTerms(terms);\n    }\n\n\n    @Override\n    public Explanation explain(LeafReaderContext context, int doc) throws IOException {\n      return explain(context, doc, null);\n    }\n\n    /**\n     * Returns an explanation of the scoring for the given document.\n     *\n     * @param context The context of the reader that returned this document.\n     * @param doc The document.\n     * @param fieldHitAttribution Per-hit field attribution information.\n     * @return An explanation of the scoring for the given document.\n     */\n    public Explanation explain(LeafReaderContext context, int doc,\n        @Nullable FieldHitAttribution fieldHitAttribution) throws IOException {\n\n      Explanation luceneExplanation = Explanation.noMatch(\"LuceneQuery explain skipped\");\n      if (!ignoreLuceneQueryScoreExplanation) {\n        // get Lucene score\n        try {\n          luceneExplanation = luceneWeight.explain(context, doc);\n        } catch (Exception e) {\n          // We sometimes see exceptions resulting from term queries that do not store\n          // utf8-text, which TermQuery.toString() assumes.  Catch here and allow at least\n          // scoring function explanations to be returned.\n          LOG.error(\"Exception in explain\", e);\n          luceneExplanation = Explanation.noMatch(\"LuceneQuery explain failed\");\n        }\n      }\n\n      Explanation scoringFunctionExplanation;\n      scoringFunction.setFieldHitAttribution(fieldHitAttribution);\n      scoringFunctionExplanation = scoringFunction.explain(\n          context.reader(), doc, luceneExplanation.getValue().floatValue());\n\n      // just add a wrapper for a better structure of the final explanation\n      Explanation luceneExplanationWrapper = Explanation.match(\n          luceneExplanation.getValue(), \"LuceneQuery\", luceneExplanation);\n\n      return Explanation.match(scoringFunctionExplanation.getValue(), \"RelevanceQuery\",\n              scoringFunctionExplanation, luceneExplanationWrapper);\n    }\n\n    @Override\n    public Scorer scorer(LeafReaderContext context) throws IOException {\n      return luceneWeight.scorer(context);\n    }\n\n    @Override\n    public boolean isCacheable(LeafReaderContext ctx) {\n      return luceneWeight.isCacheable(ctx);\n    }\n  }\n\n  @Override\n  public int hashCode() {\n    return (luceneQuery == null ? 0 : luceneQuery.hashCode())\n        + (scoringFunction == null ? 0 : scoringFunction.hashCode()) * 13;\n  }\n\n  @Override\n  public boolean equals(Object obj) {\n    if (!(obj instanceof RelevanceQuery)) {\n      return false;\n    }\n\n    RelevanceQuery query = RelevanceQuery.class.cast(obj);\n    return Objects.equals(luceneQuery, query.luceneQuery)\n        && Objects.equals(scoringFunction, query.scoringFunction);\n  }\n\n  @Override\n  public String toString(String field) {\n    return \"RelevanceQuery[q=\" + luceneQuery.toString(field) + \"]\";\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/RetweetBasedTopTweetsScoringFunction.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport java.io.IOException;\n\nimport org.apache.lucene.search.Explanation;\n\nimport com.twitter.search.common.relevance.features.MutableFeatureNormalizers;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadataOptions;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultType;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultsRelevanceStats;\n\n/**\n * A toptweets query cache index selection scoring function that is based purely on retweet counts.\n * The goal of this scoring functon is to deprecate itweet score in entirety.\n *\n * Once all legacy itweet scores are drained from existing earlybird index, new parus score replaces\n * existing itweet score position, then this class will be deprecated, a new scoring function\n * using parus score shall replace this.\n *\n * this scoring function is only used in Query Cache for marking top tweets\n * in the background. When searched, those tweets are still ranked with linear or model-based\n * scoring function.\n *\n */\npublic class RetweetBasedTopTweetsScoringFunction extends ScoringFunction {\n  private static final double DEFAULT_RECENCY_SCORE_FRACTION = 0.1;\n  private static final double DEFAULT_SIGMOID_APLHA = 0.008;\n  private static final int DEFAULT_RECENCY_CENTER_MINUTES = 1080;\n\n  // if you update the default cut off, make sure you update the query cache filter in\n  // querycache.yml\n  //\n  // we know currently each time slice, each partition has about 10K entries in toptweets query\n  // cache. These are unique tweets. Looking at retweet updates, each time slice, each partition has\n  // about 650K unique tweets that received retweet. To create roughly similar number of entries in\n  // query cache, we need top 2% of such tweets, and that sets to min retweet count to 4.\n  // In this linear scoring function, we will rescale retweet count to [0, 1] range,\n  // with an input range of [0, 20]. Given the realtime factor's weight of 0.1, that give our\n  // minimal retweet score threshold to: 4/20 * 0.9 = 0.18.\n  // Testing on prod showed much higher volume due to the generous setting of max value of 20,\n  // (highest we have seen is 14). Adjusted to 0.21 which gave us similar volume.\n  private static final double DEFAULT_CUT_OFF_SCORE = 0.21;\n\n  // Normalize retweet counts from [0, 20] range to [0, 1] range\n  private static final double MAX_RETWEET_COUNT = 20.0;\n  private static final double MIN_USER_REPUTATION = 40.0;  // matches itweet system threshold\n\n  /**\n   * The scores for the retweet based top tweets have to be in the [0, 1] interval. So we can't use\n   * SKIP_HIT as the lowest possible score, and instead have to use Float.MIN_VALUE.\n   *\n   * It's OK to use different values for these constants, because they do not interfere with each\n   * other. This constant is only used in RetweetBasedTopTweetsScoringFunction, which is only used\n   * to filter the hits for the [score_filter retweets minScore maxScore] operator. So the scores\n   * returned by RetweetBasedTopTweetsScoringFunction.score() do not have any impact on the final\n   * hit score.\n   *\n   * See EarlybirdLuceneQueryVisitor.visitScoredFilterOperator() and ScoreFilterQuery for more details.\n   */\n  private static final float RETWEET_BASED_TOP_TWEETS_LOWEST_SCORE = Float.MIN_VALUE;\n\n  private final double recencyScoreFraction;\n  private final double sigmoidAlpha;\n  private final double cutOffScore;\n  private final int recencyCenterMinutes;\n  private final double maxRecency;\n\n  private final int currentTimeSeconds;\n\n  private ThriftSearchResultMetadata metadata = null;\n  private double score;\n  private double retweetCount;\n\n  public RetweetBasedTopTweetsScoringFunction(ImmutableSchemaInterface schema) {\n    this(schema, DEFAULT_RECENCY_SCORE_FRACTION,\n         DEFAULT_SIGMOID_APLHA,\n         DEFAULT_CUT_OFF_SCORE,\n         DEFAULT_RECENCY_CENTER_MINUTES);\n  }\n\n  /**\n   * Creates a no decay scoring function (used by top archive).\n   * Otherwise same as default constructor.\n   * @param nodecay  If no decay is set to true. Alpha is set to 0.0.\n   */\n  public RetweetBasedTopTweetsScoringFunction(ImmutableSchemaInterface schema, boolean nodecay) {\n    this(schema, DEFAULT_RECENCY_SCORE_FRACTION,\n         nodecay ? 0.0 : DEFAULT_SIGMOID_APLHA,\n         DEFAULT_CUT_OFF_SCORE,\n         DEFAULT_RECENCY_CENTER_MINUTES);\n  }\n\n  public RetweetBasedTopTweetsScoringFunction(ImmutableSchemaInterface schema,\n                                              double recencyScoreFraction, double sigmoidAlpha,\n                                              double cutOffScore, int recencyCenterMinutes) {\n    super(schema);\n    this.recencyScoreFraction = recencyScoreFraction;\n    this.sigmoidAlpha = sigmoidAlpha;\n    this.cutOffScore = cutOffScore;\n    this.recencyCenterMinutes = recencyCenterMinutes;\n    this.maxRecency = computeSigmoid(0);\n    this.currentTimeSeconds = (int) (System.currentTimeMillis() / 1000);\n  }\n\n  @Override\n  protected float score(float luceneQueryScore) throws IOException {\n    // Reset the data for each tweet!!!\n    metadata = null;\n    if (documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_OFFENSIVE_FLAG)\n        || (documentFeatures.getFeatureValue(EarlybirdFieldConstant.USER_REPUTATION)\n            < MIN_USER_REPUTATION)) {\n      score = RETWEET_BASED_TOP_TWEETS_LOWEST_SCORE;\n    } else {\n      // Note that here we want the post log2 value, as the MAX_RETWEET_COUNT was actually\n      // set up for that.\n      retweetCount = MutableFeatureNormalizers.BYTE_NORMALIZER.unnormAndLog2(\n          (byte) documentFeatures.getFeatureValue(EarlybirdFieldConstant.RETWEET_COUNT));\n      final double recencyScore = computeTopTweetRecencyScore();\n\n      score = (retweetCount / MAX_RETWEET_COUNT) * (1 - recencyScoreFraction)\n          + recencyScoreFraction * recencyScore;\n\n      if (score < this.cutOffScore) {\n        score = RETWEET_BASED_TOP_TWEETS_LOWEST_SCORE;\n      }\n    }\n\n    return (float) score;\n  }\n\n  private double computeSigmoid(double x) {\n    return 1.0f / (1.0f + Math.exp(sigmoidAlpha * (x - recencyCenterMinutes)));\n  }\n\n  private double computeTopTweetRecencyScore() {\n    double diffMinutes =\n        Math.max(0, currentTimeSeconds - timeMapper.getTime(getCurrentDocID())) / 60.0;\n    return computeSigmoid(diffMinutes) / maxRecency;\n  }\n\n  @Override\n  protected Explanation doExplain(float luceneScore) {\n    return null;\n  }\n\n  @Override\n  public ThriftSearchResultMetadata getResultMetadata(ThriftSearchResultMetadataOptions options) {\n    if (metadata == null) {\n      metadata = new ThriftSearchResultMetadata()\n          .setResultType(ThriftSearchResultType.POPULAR)\n          .setPenguinVersion(EarlybirdConfig.getPenguinVersionByte());\n      metadata.setRetweetCount((int) retweetCount);\n      metadata.setScore(score);\n    }\n    return metadata;\n  }\n\n  @Override\n  public void updateRelevanceStats(ThriftSearchResultsRelevanceStats relevanceStats) {\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/ScoringFunction.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.lucene.index.IndexReader;\nimport org.apache.lucene.search.Explanation;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.features.thrift.ThriftSearchResultFeatures;\nimport com.twitter.search.common.query.HitAttributeHelper;\nimport com.twitter.search.common.relevance.features.EarlybirdDocumentFeatures;\nimport com.twitter.search.common.results.thriftjava.FieldHitAttribution;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.TimeMapper;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.search.relevance.LinearScoringData;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadataOptions;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultType;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultsRelevanceStats;\nimport com.twitter.search.queryparser.query.Query;\n\n/**\n * Defines a ranking function which computes the score of a document that matches a query.\n */\npublic abstract class ScoringFunction {\n  /**\n   * Returned by a {@link #score(int, float)} to indicate that a hit should be scored below all.\n   *\n   * We have some equality tests like:\n   *   \"if (score == ScoringFunction.SKIP_HIT) {...}\" (DefaultScoringFunction#updateRelevanceStats)\n   * We might also have double to float casts.\n   *\n   * Such castings seem to work with the equality test, but there might corner cases when casting\n   * this float value to a double (and back) might not work properly.\n   *\n   * If possible, we should choose a constant that is not in the valid score range. Then we can\n   * turn the float equality tests into Math.abs(...) < EPSILON tests.\n   */\n  public static final float SKIP_HIT = -Float.MAX_VALUE;\n\n  private final ImmutableSchemaInterface schema;\n\n  // The current doc ID and the reader for the current segment should be private, because we don't\n  // want sub-classes to incorrectly update them. The doc ID should only be updated by the score()\n  // and explain() methods, and the reader should only be updated by the setNextReader() method.\n  private int currentDocID = -1;\n\n  protected DocIDToTweetIDMapper tweetIDMapper = null;\n  protected TimeMapper timeMapper = null;\n  protected EarlybirdDocumentFeatures documentFeatures;\n\n  protected int debugMode = 0;\n  protected HitAttributeHelper hitAttributeHelper;\n  protected Query query;\n\n  protected FieldHitAttribution fieldHitAttribution;\n\n  public ScoringFunction(ImmutableSchemaInterface schema) {\n    this.schema = Preconditions.checkNotNull(schema);\n  }\n\n  protected ImmutableSchemaInterface getSchema() {\n    return schema;\n  }\n\n  /**\n   * Updates the reader that will be used to retrieve the tweet IDs and creation times associated\n   * with scored doc IDs, as well as the values for various CSFs. Should be called every time the\n   * searcher starts searching in a new segment.\n   */\n  public void setNextReader(EarlybirdIndexSegmentAtomicReader reader) throws IOException {\n    tweetIDMapper = reader.getSegmentData().getDocIDToTweetIDMapper();\n    timeMapper = reader.getSegmentData().getTimeMapper();\n    documentFeatures = new EarlybirdDocumentFeatures(reader);\n    initializeNextSegment(reader);\n  }\n\n  public void setHitAttributeHelperAndQuery(HitAttributeHelper newHitAttributeHelper,\n                                            Query parsedQuery) {\n    this.hitAttributeHelper = newHitAttributeHelper;\n    this.query = parsedQuery;\n  }\n\n  public void setFieldHitAttribution(FieldHitAttribution fieldHitAttribution) {\n    this.fieldHitAttribution = fieldHitAttribution;\n  }\n\n  public void setDebugMode(int debugMode) {\n    this.debugMode = debugMode;\n  }\n\n  /**\n   * Allow scoring functions to perform more per-segment-specific setup.\n   */\n  protected void initializeNextSegment(EarlybirdIndexSegmentAtomicReader reader)\n      throws IOException {\n    // Noop by default\n  }\n\n  // Updates the current document ID and advances all NumericDocValues to this doc ID.\n  private void setCurrentDocID(int currentDocID) throws IOException {\n    this.currentDocID = currentDocID;\n    documentFeatures.advance(currentDocID);\n  }\n\n  /**\n   * Returns the current doc ID stored in this scoring function.\n   */\n  public int getCurrentDocID() {\n    return currentDocID;\n  }\n\n  /**\n   * Compute the score for the current hit.  This is not expected to be thread safe.\n   *\n   * @param internalDocID    internal id of the matching hit\n   * @param luceneQueryScore the score that lucene's text query computed for this hit\n   */\n  public float score(int internalDocID, float luceneQueryScore) throws IOException {\n    setCurrentDocID(internalDocID);\n    return score(luceneQueryScore);\n  }\n\n  /**\n   * Compute the score for the current hit.  This is not expected to be thread safe.\n   *\n   * @param luceneQueryScore the score that lucene's text query computed for this hit\n   */\n  protected abstract float score(float luceneQueryScore) throws IOException;\n\n  /** Returns an explanation for the given hit. */\n  public final Explanation explain(IndexReader reader, int internalDocID, float luceneScore)\n      throws IOException {\n    setNextReader((EarlybirdIndexSegmentAtomicReader) reader);\n    setCurrentDocID(internalDocID);\n    return doExplain(luceneScore);\n  }\n\n  /** Returns an explanation for the current document. */\n  protected abstract Explanation doExplain(float luceneScore) throws IOException;\n\n  /**\n   * Returns the scoring metadata for the current doc ID.\n   */\n  public ThriftSearchResultMetadata getResultMetadata(ThriftSearchResultMetadataOptions options)\n      throws IOException {\n    ThriftSearchResultMetadata metadata = new ThriftSearchResultMetadata();\n    metadata.setResultType(ThriftSearchResultType.RELEVANCE);\n    metadata.setPenguinVersion(EarlybirdConfig.getPenguinVersionByte());\n    metadata.setLanguage(ThriftLanguage.findByValue(\n        (int) documentFeatures.getFeatureValue(EarlybirdFieldConstant.LANGUAGE)));\n    metadata.setSignature(\n        (int) documentFeatures.getFeatureValue(EarlybirdFieldConstant.TWEET_SIGNATURE));\n    metadata.setIsNullcast(documentFeatures.isFlagSet(EarlybirdFieldConstant.IS_NULLCAST_FLAG));\n    return metadata;\n  }\n\n  /**\n   * Updates the given ThriftSearchResultsRelevanceStats instance based on the scoring metadata for\n   * the current doc ID.\n   */\n  public abstract void updateRelevanceStats(ThriftSearchResultsRelevanceStats relevanceStats);\n\n  /**\n   * Score a list of hits. Not thread safe.\n   */\n  public float[] batchScore(List<BatchHit> hits) throws IOException {\n    throw new UnsupportedOperationException(\"This operation (batchScore) is not implemented!\");\n  }\n\n  /**\n   * Collect the features and CSFs for the current document. Used for scoring and generating the\n   * returned metadata.\n   */\n  public Pair<LinearScoringData, ThriftSearchResultFeatures> collectFeatures(\n      float luceneQueryScore) throws IOException {\n    throw new UnsupportedOperationException(\"This operation (collectFeatures) is not implemented!\");\n  }\n\n  /**\n   * Implement this function to populate the result metadata based on the given scoring data.\n   * Otherwise, this is a no-op.\n   *\n   * Scoring functions that implement this should also implement getScoringData().\n   */\n  public void populateResultMetadataBasedOnScoringData(\n      ThriftSearchResultMetadataOptions options,\n      ThriftSearchResultMetadata metadata,\n      LinearScoringData data) throws IOException {\n    // Make sure that the scoring data passed in is null because getScoringDataForCurrentDocument()\n    // returns null by default and if a subclass overrides one of these two methods, it should\n    // override both.\n    Preconditions.checkState(data == null, \"LinearScoringData should be null\");\n  }\n\n  /**\n   * This should only be called at hit collection time because it relies on the internal doc id.\n   *\n   * Scoring functions that implement this should also implement the function\n   * populateResultMetadataBasedOnScoringData().\n   */\n  public LinearScoringData getScoringDataForCurrentDocument() {\n    return null;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/ScoringFunctionProvider.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport java.io.IOException;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.query.HitAttributeHelper;\nimport com.twitter.search.common.ranking.thriftjava.ThriftRankingParams;\nimport com.twitter.search.common.ranking.thriftjava.ThriftScoringFunctionType;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.util.ml.tensorflow_engine.TensorflowModelsManager;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.exception.ClientException;\nimport com.twitter.search.earlybird.ml.ScoringModelsManager;\nimport com.twitter.search.earlybird.search.AntiGamingFilter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultType;\nimport com.twitter.search.queryparser.query.Query;\n\n/**\n * Returns a scoring function for a particular experiment ID.\n *\n * Can be used for a/b testing of different scoring formulas.\n */\npublic abstract class ScoringFunctionProvider {\n  private static final Logger LOG = LoggerFactory.getLogger(ScoringFunctionProvider.class);\n\n  /**\n   * Returns the scoring function.\n   */\n  public abstract ScoringFunction getScoringFunction() throws IOException, ClientException;\n\n  public static final String RETWEETS_SCORER_NAME = \"retweets\";\n  public static final String NO_SPAM_SCORER_NAME = \"no_spam\";\n  public static final String TEST_SCORER_NAME = \"test\";\n\n  // Whether to avoid time decay when scoring top tweets.\n  // Top archive does not need time decay.\n  private static final boolean TOP_TWEET_WITH_DECAY =\n          EarlybirdConfig.getBool(\"top_tweet_scoring_with_decay\", true);\n\n  /**\n   * Abstract class that can be used for ScoringFunctions that don't throw a ClientException.\n   *\n   * It does throw an IOException but it doesn't throw a ClientException so the name can be a bit\n   * misleading.\n   */\n  public abstract static class NamedScoringFunctionProvider extends ScoringFunctionProvider {\n    /**\n     * Returns the scoring function.\n     */\n    public abstract ScoringFunction getScoringFunction() throws IOException;\n  }\n\n  /**\n   * Returns the scoring function provider with the given name, or null if no such provider exists.\n   */\n  public static NamedScoringFunctionProvider getScoringFunctionProviderByName(\n      String name, final ImmutableSchemaInterface schema) {\n    if (name.equals(NO_SPAM_SCORER_NAME)) {\n      return new NamedScoringFunctionProvider() {\n        @Override\n        public ScoringFunction getScoringFunction() throws IOException {\n          return new SpamVectorScoringFunction(schema);\n        }\n      };\n    } else if (name.equals(RETWEETS_SCORER_NAME)) {\n      return new NamedScoringFunctionProvider() {\n        @Override\n        public ScoringFunction getScoringFunction() throws IOException {\n          // Production top tweet actually uses this.\n          if (TOP_TWEET_WITH_DECAY) {\n            return new RetweetBasedTopTweetsScoringFunction(schema);\n          } else {\n            return new RetweetBasedTopTweetsScoringFunction(schema, true);\n          }\n        }\n      };\n    } else if (name.equals(TEST_SCORER_NAME)) {\n      return new NamedScoringFunctionProvider() {\n        @Override\n        public ScoringFunction getScoringFunction() throws IOException {\n          return new TestScoringFunction(schema);\n        }\n      };\n    }\n    return null;\n  }\n\n  /**\n   * Returns default scoring functions for different scoring function type\n   * and provides fallback behavior if model-based scoring function fails\n   */\n  public static class DefaultScoringFunctionProvider extends ScoringFunctionProvider {\n    private final EarlybirdRequest request;\n    private final ImmutableSchemaInterface schema;\n    private final ThriftSearchQuery searchQuery;\n    private final AntiGamingFilter antiGamingFilter;\n    private final UserTable userTable;\n    private final HitAttributeHelper hitAttributeHelper;\n    private final Query parsedQuery;\n    private final ScoringModelsManager scoringModelsManager;\n    private final TensorflowModelsManager tensorflowModelsManager;\n\n    private static final SearchCounter MODEL_BASED_SCORING_FUNCTION_CREATED =\n        SearchCounter.export(\"model_based_scoring_function_created\");\n    private static final SearchCounter MODEL_BASED_FALLBACK_TO_LINEAR_SCORING_FUNCTION =\n        SearchCounter.export(\"model_based_fallback_to_linear_scoring_function\");\n\n    private static final SearchCounter TENSORFLOW_BASED_SCORING_FUNCTION_CREATED =\n        SearchCounter.export(\"tensorflow_based_scoring_function_created\");\n    private static final SearchCounter TENSORFLOW_BASED_FALLBACK_TO_LINEAR_SCORING_FUNCTION =\n        SearchCounter.export(\"tensorflow_fallback_to_linear_function_scoring_function\");\n\n    public DefaultScoringFunctionProvider(\n        final EarlybirdRequest request,\n        final ImmutableSchemaInterface schema,\n        final ThriftSearchQuery searchQuery,\n        final AntiGamingFilter antiGamingFilter,\n        final UserTable userTable,\n        final HitAttributeHelper hitAttributeHelper,\n        final Query parsedQuery,\n        final ScoringModelsManager scoringModelsManager,\n        final TensorflowModelsManager tensorflowModelsManager) {\n      this.request = request;\n      this.schema = schema;\n      this.searchQuery = searchQuery;\n      this.antiGamingFilter = antiGamingFilter;\n      this.userTable = userTable;\n      this.hitAttributeHelper = hitAttributeHelper;\n      this.parsedQuery = parsedQuery;\n      this.scoringModelsManager = scoringModelsManager;\n      this.tensorflowModelsManager = tensorflowModelsManager;\n    }\n\n    @Override\n    public ScoringFunction getScoringFunction() throws IOException, ClientException {\n      if (searchQuery.isSetRelevanceOptions()\n          && searchQuery.getRelevanceOptions().isSetRankingParams()) {\n        ThriftRankingParams params = searchQuery.getRelevanceOptions().getRankingParams();\n        ThriftScoringFunctionType type = params.isSetType()\n            ? params.getType() : ThriftScoringFunctionType.LINEAR;  // default type\n        switch (type) {\n          case LINEAR:\n            return createLinear();\n          case MODEL_BASED:\n            if (scoringModelsManager.isEnabled()) {\n              MODEL_BASED_SCORING_FUNCTION_CREATED.increment();\n              return createModelBased();\n            } else {\n              // From ScoringModelsManager.NO_OP_MANAGER. Fall back to LinearScoringFunction\n              MODEL_BASED_FALLBACK_TO_LINEAR_SCORING_FUNCTION.increment();\n              return createLinear();\n            }\n          case TENSORFLOW_BASED:\n            if (tensorflowModelsManager.isEnabled()) {\n              TENSORFLOW_BASED_SCORING_FUNCTION_CREATED.increment();\n              return createTensorflowBased();\n            } else {\n              // Fallback to linear scoring if tf manager is disabled\n              TENSORFLOW_BASED_FALLBACK_TO_LINEAR_SCORING_FUNCTION.increment();\n              return createLinear();\n            }\n          case TOPTWEETS:\n            return createTopTweets();\n          default:\n            throw new IllegalArgumentException(\"Unknown scoring type: in \" + searchQuery);\n        }\n      } else {\n        LOG.error(\"No relevance options provided query = \" + searchQuery);\n        return new DefaultScoringFunction(schema);\n      }\n    }\n\n    private ScoringFunction createLinear() throws IOException {\n      LinearScoringFunction scoringFunction = new LinearScoringFunction(\n          schema, searchQuery, antiGamingFilter, ThriftSearchResultType.RELEVANCE,\n          userTable);\n      scoringFunction.setHitAttributeHelperAndQuery(hitAttributeHelper, parsedQuery);\n\n      return scoringFunction;\n    }\n\n    /**\n     * For model based scoring function, ClientException will be throw if client selects an\n     * unknown model for scoring manager.\n     * {@link com.twitter.search.earlybird.search.relevance.scoring.ModelBasedScoringFunction}\n     */\n    private ScoringFunction createModelBased() throws IOException, ClientException {\n      ModelBasedScoringFunction scoringFunction = new ModelBasedScoringFunction(\n          schema, searchQuery, antiGamingFilter, ThriftSearchResultType.RELEVANCE, userTable,\n          scoringModelsManager);\n      scoringFunction.setHitAttributeHelperAndQuery(hitAttributeHelper, parsedQuery);\n\n      return scoringFunction;\n    }\n\n    private ScoringFunction createTopTweets() throws IOException {\n      return new LinearScoringFunction(\n          schema, searchQuery, antiGamingFilter, ThriftSearchResultType.POPULAR, userTable);\n    }\n\n    private TensorflowBasedScoringFunction createTensorflowBased()\n      throws IOException, ClientException {\n      TensorflowBasedScoringFunction tfScoringFunction = new TensorflowBasedScoringFunction(\n          request, schema, searchQuery, antiGamingFilter,\n          ThriftSearchResultType.RELEVANCE, userTable, tensorflowModelsManager);\n      tfScoringFunction.setHitAttributeHelperAndQuery(hitAttributeHelper, parsedQuery);\n      return tfScoringFunction;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/SpamVectorScoringFunction.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport java.io.IOException;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.apache.lucene.search.Explanation;\n\nimport com.twitter.search.common.relevance.features.RelevanceSignalConstants;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadataOptions;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultsRelevanceStats;\n\npublic class SpamVectorScoringFunction extends ScoringFunction {\n  private static final int MIN_TWEEPCRED_WITH_LINK =\n      EarlybirdConfig.getInt(\"min_tweepcred_with_non_whitelisted_link\", 25);\n\n  // The engagement threshold that prevents us from filtering users with low tweepcred.\n  private static final int ENGAGEMENTS_NO_FILTER = 1;\n\n  @VisibleForTesting\n  static final float NOT_SPAM_SCORE = 0.5f;\n  @VisibleForTesting\n  static final float SPAM_SCORE = -0.5f;\n\n  public SpamVectorScoringFunction(ImmutableSchemaInterface schema) {\n    super(schema);\n  }\n\n  @Override\n  protected float score(float luceneQueryScore) throws IOException {\n    if (documentFeatures.isFlagSet(EarlybirdFieldConstant.FROM_VERIFIED_ACCOUNT_FLAG)) {\n      return NOT_SPAM_SCORE;\n    }\n\n    int tweepCredThreshold = 0;\n    if (documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_LINK_FLAG)\n        && !documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_IMAGE_URL_FLAG)\n        && !documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_VIDEO_URL_FLAG)\n        && !documentFeatures.isFlagSet(EarlybirdFieldConstant.HAS_NEWS_URL_FLAG)) {\n      // Contains a non-media non-news link, definite spam vector.\n      tweepCredThreshold = MIN_TWEEPCRED_WITH_LINK;\n    }\n\n    int tweepcred = (int) documentFeatures.getFeatureValue(EarlybirdFieldConstant.USER_REPUTATION);\n\n    // For new user, tweepcred is set to a sentinel value of -128, specified at\n    // src/thrift/com/twitter/search/common/indexing/status.thrift\n    if (tweepcred >= tweepCredThreshold\n        || tweepcred == (int) RelevanceSignalConstants.UNSET_REPUTATION_SENTINEL) {\n      return NOT_SPAM_SCORE;\n    }\n\n    double retweetCount =\n        documentFeatures.getUnnormalizedFeatureValue(EarlybirdFieldConstant.RETWEET_COUNT);\n    double replyCount =\n        documentFeatures.getUnnormalizedFeatureValue(EarlybirdFieldConstant.REPLY_COUNT);\n    double favoriteCount =\n        documentFeatures.getUnnormalizedFeatureValue(EarlybirdFieldConstant.FAVORITE_COUNT);\n\n    // If the tweet has enough engagements, do not mark it as spam.\n    if (retweetCount + replyCount + favoriteCount >= ENGAGEMENTS_NO_FILTER) {\n      return NOT_SPAM_SCORE;\n    }\n\n    return SPAM_SCORE;\n  }\n\n  @Override\n  protected Explanation doExplain(float luceneScore) {\n    return null;\n  }\n\n  @Override\n  public ThriftSearchResultMetadata getResultMetadata(ThriftSearchResultMetadataOptions options) {\n    return null;\n  }\n\n  @Override\n  public void updateRelevanceStats(ThriftSearchResultsRelevanceStats relevanceStats) {\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/SparseTensor.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport java.nio.ByteBuffer;\nimport java.nio.ByteOrder;\n\n// Ideally, this part should live somewhere in the Cortex common\n// code. Today, it is not possible to create\n// a `SparseTensor` that relies only on ByteBuffer.\npublic class SparseTensor {\n\n  private ByteBuffer sparseIndices;\n  private ByteBuffer sparseValues;\n  private ByteBuffer sparseShape;\n\n  private int numDocs;\n  private final long[] sparseShapeShapeDimension = new long[] {2L};\n  private final long inputBitSize = 1 << 63;\n\n  private long numRecordsSeen = 0;\n  private final long numFeatures;\n  private int numValuesSeen;\n\n  public SparseTensor(int numDocs, int numFeatures) {\n    this.numDocs = numDocs;\n    this.numFeatures = (long) numFeatures;\n    this.sparseValues =\n      ByteBuffer\n      .allocate(numFeatures * numDocs * Float.BYTES)\n      .order(ByteOrder.LITTLE_ENDIAN);\n    this.sparseIndices =\n      ByteBuffer\n        .allocate(2 * numFeatures * numDocs * Long.BYTES)\n        .order(ByteOrder.LITTLE_ENDIAN);\n    this.sparseShape =\n      ByteBuffer\n      .allocate(2 * Long.BYTES)\n      .order(ByteOrder.LITTLE_ENDIAN);\n  }\n\n  public void incNumRecordsSeen() {\n    numRecordsSeen++;\n  }\n\n  /**\n   * Adds the given value to this tensor.\n   */\n  public void addValue(long featureId, float value) {\n    sparseValues.putFloat(value);\n    sparseIndices.putLong(numRecordsSeen);\n    sparseIndices.putLong(featureId);\n    numValuesSeen++;\n  }\n\n  public ByteBuffer getSparseValues() {\n    sparseValues.limit(numValuesSeen * Float.BYTES);\n    sparseValues.rewind();\n    return sparseValues;\n  }\n\n  public long[] getSparseValuesShape() {\n    return new long[] {numValuesSeen};\n  }\n\n  public long[] getSparseIndicesShape() {\n    return new long[] {numValuesSeen, 2L};\n  }\n\n  public long[] getSparseShapeShape() {\n    return sparseShapeShapeDimension;\n  }\n\n  public ByteBuffer getSparseIndices() {\n    sparseIndices.limit(2 * numValuesSeen * Long.BYTES);\n    sparseIndices.rewind();\n    return sparseIndices;\n  }\n\n  /**\n   * Returns the sparse shape for this tensor.\n   */\n  public ByteBuffer getSparseShape() {\n    sparseShape.putLong(numRecordsSeen);\n    sparseShape.putLong(inputBitSize);\n    sparseShape.rewind();\n    return sparseShape;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/TensorflowBasedScoringFunction.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport java.io.IOException;\nimport java.nio.FloatBuffer;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.ImmutableMap;\n\nimport org.apache.lucene.search.Explanation;\nimport org.tensorflow.Tensor;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.search.common.constants.thriftjava.ThriftQuerySource;\nimport com.twitter.search.common.features.EarlybirdRankingDerivedFeature;\nimport com.twitter.search.common.features.FeatureHandler;\nimport com.twitter.search.common.features.thrift.ThriftSearchResultFeatures;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.util.ml.tensorflow_engine.TensorflowModelsManager;\nimport com.twitter.search.earlybird.EarlybirdSearcher;\nimport com.twitter.search.earlybird.common.userupdates.UserTable;\nimport com.twitter.search.earlybird.exception.ClientException;\nimport com.twitter.search.earlybird.search.AntiGamingFilter;\nimport com.twitter.search.earlybird.search.relevance.LinearScoringData;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchRelevanceOptions;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultType;\nimport com.twitter.search.modeling.common.TweetFeaturesUtils;\nimport com.twitter.tfcompute_java.TFModelRunner;\n\n/**\n * TensorflowBasedScoringFunction relies on a TF model for scoring tweets\n * Only the `batchScore` part is implemented\n */\npublic class TensorflowBasedScoringFunction extends FeatureBasedScoringFunction {\n  private final TFModelRunner tfModelRunner;\n\n  // https://stackoverflow.com/questions/37849322/how-to-understand-the-term-tensor-in-tensorflow\n  // for more information on this notation - in short, a TF graph is made\n  // of TF operations and doesn't have a first order notion of tensors\n  // The notation <operation>:<index> will maps to the <index> output of the\n  // <operation> contained in the TF graph.\n  private static final String INPUT_VALUES = \"input_sparse_tensor_values:0\";\n  private static final String INPUT_INDICES = \"input_sparse_tensor_indices:0\";\n  private static final String INPUT_SHAPE = \"input_sparse_tensor_shape:0\";\n  private static final String OUTPUT_NODE = \"output_scores:0\";\n\n  private final Map<Integer, Long> featureSchemaIdToMlApiId;\n  private final Map<Long, Float> tweetIdToScoreMap = new HashMap<>();\n  private final EarlybirdRequest request;\n\n  public TensorflowBasedScoringFunction(\n      EarlybirdRequest request,\n      ImmutableSchemaInterface schema,\n      ThriftSearchQuery searchQuery,\n      AntiGamingFilter antiGamingFilter,\n      ThriftSearchResultType searchResultType,\n      UserTable userTable,\n      TensorflowModelsManager tensorflowModelsManager\n      ) throws IOException, ClientException {\n    super(\n      \"TensorflowBasedScoringFunction\",\n      schema,\n      searchQuery,\n      antiGamingFilter,\n      searchResultType,\n        userTable\n    );\n    this.request = request;\n    String modelName = searchQuery.getRelevanceOptions().getRankingParams().selectedTensorflowModel;\n    this.featureSchemaIdToMlApiId = tensorflowModelsManager.getFeatureSchemaIdToMlApiId();\n\n    if (modelName == null) {\n      throw new ClientException(\"Scoring type is TENSORFLOW_BASED but no model was selected\");\n    } else if (!tensorflowModelsManager.getModel(modelName).isPresent()) {\n      throw new ClientException(\n        \"Scoring type is TENSORFLOW_BASED. Model \"\n        + modelName\n        + \" is not present.\"\n      );\n    }\n\n    if (searchQuery.getRelevanceOptions().getRankingParams().isEnableHitDemotion()) {\n      throw new ClientException(\n          \"Hit attribute demotion is not supported with TENSORFLOW_BASED scoring type\");\n    }\n\n    tfModelRunner = tensorflowModelsManager.getModel(modelName).get();\n  }\n\n  /**\n   * Single item scoring just returns the lucene score to be used during the batching phase.\n   */\n  @Override\n  protected float score(float luceneQueryScore) {\n    return luceneQueryScore;\n  }\n\n  @Override\n  public Pair<LinearScoringData, ThriftSearchResultFeatures> collectFeatures(\n      float luceneQueryScore) throws IOException {\n    LinearScoringData linearScoringData = updateLinearScoringData(luceneQueryScore);\n    ThriftSearchResultFeatures features =\n        createFeaturesForDocument(linearScoringData, true).getFeatures();\n\n    return new Pair<>(linearScoringData, features);\n  }\n\n  @Override\n  protected FeatureHandler createFeaturesForDocument(\n      LinearScoringData linearScoringData,\n      boolean ignoreDefaultValues) throws IOException {\n    return super.createFeaturesForDocument(linearScoringData,\n            ignoreDefaultValues)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_TREND_CLICK,\n            request.querySource == ThriftQuerySource.TREND_CLICK)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_TYPED_QUERY,\n            request.querySource == ThriftQuerySource.TYPED_QUERY)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_TYPEAHEAD_CLICK,\n            request.querySource == ThriftQuerySource.TYPEAHEAD_CLICK)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_HASHTAG_CLICK,\n            request.querySource == ThriftQuerySource.RECENT_SEARCH_CLICK)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_RECENT_SEARCH_CLICK,\n            request.querySource == ThriftQuerySource.RECENT_SEARCH_CLICK)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_PROFILE_CLICK,\n            request.querySource == ThriftQuerySource.PROFILE_CLICK)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_API_CALL,\n            request.querySource == ThriftQuerySource.API_CALL)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_PROMOTED_TREND_CLICK,\n            request.querySource == ThriftQuerySource.PROMOTED_TREND_CLICK)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_SAVED_SEARCH_CLICK,\n            request.querySource == ThriftQuerySource.SAVED_SEARCH_CLICK)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_CASHTAG_CLICK,\n            request.querySource == ThriftQuerySource.CASHTAG_CLICK)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_SPELLING_EXPANSION_REVERT_CLICK,\n            request.querySource == ThriftQuerySource.SPELLING_EXPANSION_REVERT_CLICK)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_SPELLING_SUGGESTION_CLICK,\n            request.querySource == ThriftQuerySource.SPELLING_SUGGESTION_CLICK)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_LOGGED_OUT_HOME_TREND_CLICK,\n            request.querySource == ThriftQuerySource.LOGGED_OUT_HOME_TREND_CLICK)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_RELATED_QUERY_CLICK,\n            request.querySource == ThriftQuerySource.RELATED_QUERY_CLICK)\n        .addBoolean(EarlybirdRankingDerivedFeature.QUERY_SOURCE_AUTO_SPELL_CORRECT_REVERT_CLICK,\n            request.querySource == ThriftQuerySource.AUTO_SPELL_CORRECT_REVERT_CLICK);\n  }\n\n  /**\n   * Return scores computed in batchScore() if forExplanation is true.\n   */\n  @Override\n  protected double computeScore(LinearScoringData data, boolean forExplanation) {\n    Preconditions.checkState(forExplanation,\n        \"forExplanation is false. computeScore() should only be used for explanation creation\");\n    return tweetIdToScoreMap.get(tweetIDMapper.getTweetID(getCurrentDocID()));\n  }\n\n  @Override\n  protected void generateExplanationForScoring(\n      LinearScoringData scoringData, boolean isHit, List<Explanation> details) {\n  }\n\n  @VisibleForTesting\n  SparseTensor createInputTensor(ThriftSearchResultFeatures[] featuresForDocs) {\n    // Moving this across outside of the request path\n    // would reduce the allocation cost and make the `ByteBuffer`s\n    // long lived - would need one per thread.\n    SparseTensor sparseTensor =\n        new SparseTensor(featuresForDocs.length, featureSchemaIdToMlApiId.size());\n    for (ThriftSearchResultFeatures features : featuresForDocs) {\n      updateSparseTensor(sparseTensor, features);\n    }\n    return sparseTensor;\n  }\n\n  private void addSchemaBooleanFeatures(SparseTensor sparseTensor,\n                                        Map<Integer, Boolean> booleanMap) {\n    if (booleanMap == null || booleanMap.isEmpty()) {\n      return;\n    }\n    for (Map.Entry<Integer, Boolean> entry : booleanMap.entrySet()) {\n      Preconditions.checkState(featureSchemaIdToMlApiId.containsKey(entry.getKey()));\n      sparseTensor.addValue(\n          featureSchemaIdToMlApiId.get(entry.getKey()), entry.getValue() ? 1f : 0f);\n    }\n  }\n\n  private void addSchemaContinuousFeatures(SparseTensor sparseTensor,\n                                           Map<Integer, ? extends Number> valueMap) {\n    if (valueMap == null || valueMap.isEmpty()) {\n      return;\n    }\n    for (Map.Entry<Integer, ? extends Number> entry : valueMap.entrySet()) {\n      Integer id = entry.getKey();\n      // SEARCH-26795\n      if (!TweetFeaturesUtils.isFeatureDiscrete(id)) {\n        Preconditions.checkState(featureSchemaIdToMlApiId.containsKey(id));\n        sparseTensor.addValue(\n            featureSchemaIdToMlApiId.get(id), entry.getValue().floatValue());\n      }\n    }\n  }\n\n  private void updateSparseTensor(SparseTensor sparseTensor, ThriftSearchResultFeatures features) {\n    addSchemaBooleanFeatures(sparseTensor, features.getBoolValues());\n    addSchemaContinuousFeatures(sparseTensor, features.getIntValues());\n    addSchemaContinuousFeatures(sparseTensor, features.getLongValues());\n    addSchemaContinuousFeatures(sparseTensor, features.getDoubleValues());\n\n    sparseTensor.incNumRecordsSeen();\n  }\n\n  private float[] batchScoreInternal(ThriftSearchResultFeatures[] featuresForDocs) {\n    int nbDocs = featuresForDocs.length;\n    float[] backingArrayResults = new float[nbDocs];\n    SparseTensor sparseTensor = createInputTensor(featuresForDocs);\n    Tensor<?> sparseValues =\n      Tensor.create(\n        Float.class,\n        sparseTensor.getSparseValuesShape(),\n        sparseTensor.getSparseValues());\n    Tensor<?> sparseIndices =\n      Tensor.create(\n        Long.class,\n        sparseTensor.getSparseIndicesShape(),\n        sparseTensor.getSparseIndices());\n    Tensor<?> sparseShape =\n      Tensor.create(\n        Long.class,\n        sparseTensor.getSparseShapeShape(),\n        sparseTensor.getSparseShape());\n    Map<String, Tensor<?>> inputMap = ImmutableMap.of(\n      INPUT_VALUES, sparseValues,\n      INPUT_INDICES, sparseIndices,\n      INPUT_SHAPE, sparseShape\n      );\n    List<String> output = ImmutableList.of(OUTPUT_NODE);\n\n    Map<String, Tensor<?>> outputs = tfModelRunner.run(\n      inputMap,\n      output,\n      ImmutableList.of()\n    );\n    Tensor<?> outputTensor = outputs.get(OUTPUT_NODE);\n    try {\n      FloatBuffer finalResultBuffer =\n        FloatBuffer.wrap(backingArrayResults, 0, nbDocs);\n\n      outputTensor.writeTo(finalResultBuffer);\n    } finally {\n      // Close tensors to avoid memory leaks\n      sparseValues.close();\n      sparseIndices.close();\n      sparseShape.close();\n      if (outputTensor != null) {\n        outputTensor.close();\n      }\n    }\n    return backingArrayResults;\n  }\n\n  /**\n   * Compute the score for a list of hits. Not thread safe.\n   * @return Array of scores\n   */\n  @Override\n  public float[] batchScore(List<BatchHit> hits) throws IOException {\n    ThriftSearchResultFeatures[] featuresForDocs = new ThriftSearchResultFeatures[hits.size()];\n\n    for (int i = 0; i < hits.size(); i++) {\n      // This is a gigantic allocation, but the models are trained to depend on unset values having\n      // a default.\n      BatchHit hit = hits.get(i);\n      ThriftSearchResultFeatures features = hit.getFeatures().deepCopy();\n\n      // Adjust features of a hit based on overrides provided by relevance options. Should mostly\n      // be used for debugging purposes.\n      adjustHitScoringFeatures(hit, features);\n\n      setDefaultFeatureValues(features);\n      featuresForDocs[i] = features;\n    }\n\n    float[] scores = batchScoreInternal(featuresForDocs);\n    float[] finalScores = new float[hits.size()];\n\n    for (int i = 0; i < hits.size(); i++) {\n      LinearScoringData data = hits.get(i).getScoringData();\n      if (data.skipReason != null && data.skipReason != LinearScoringData.SkipReason.NOT_SKIPPED) {\n        // If the hit should be skipped, overwrite the score with SKIP_HIT\n        scores[i] = SKIP_HIT;\n      }\n\n      // If explanations enabled, Add scores to map. Will be used in computeScore()\n      if (EarlybirdSearcher.explanationsEnabled(debugMode)) {\n        tweetIdToScoreMap.put(hits.get(i).getTweetID(), scores[i]);\n      }\n\n      finalScores[i] = postScoreComputation(\n          data,\n          scores[i],\n          false,  // cannot get the hit attribution info for this hit at this point in time\n          null);\n    }\n    return finalScores;\n  }\n\n  private void adjustHitScoringFeatures(BatchHit hit, ThriftSearchResultFeatures features) {\n\n    if (request.isSetSearchQuery() && request.getSearchQuery().isSetRelevanceOptions()) {\n      ThriftSearchRelevanceOptions relevanceOptions =\n          request.getSearchQuery().getRelevanceOptions();\n\n      if (relevanceOptions.isSetPerTweetFeaturesOverride()\n          && relevanceOptions.getPerTweetFeaturesOverride().containsKey(hit.getTweetID())) {\n        overrideFeatureValues(\n            features,\n            relevanceOptions.getPerTweetFeaturesOverride().get(hit.getTweetID()));\n      }\n\n      if (relevanceOptions.isSetPerUserFeaturesOverride()\n          && relevanceOptions.getPerUserFeaturesOverride().containsKey(\n              hit.getScoringData().fromUserId)) {\n        overrideFeatureValues(\n            features,\n            relevanceOptions.getPerUserFeaturesOverride().get(hit.getScoringData().fromUserId));\n      }\n\n      if (relevanceOptions.isSetGlobalFeaturesOverride()) {\n        overrideFeatureValues(\n            features, relevanceOptions.getGlobalFeaturesOverride());\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/search/relevance/scoring/TestScoringFunction.java",
    "content": "package com.twitter.search.earlybird.search.relevance.scoring;\n\nimport org.apache.lucene.search.Explanation;\n\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadataOptions;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultType;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultsRelevanceStats;\n\n/**\n * A dummy scoring function for test, the score is always tweetId/10000.0\n * Since score_filter: operator requires all score to be between [0, 1], if you want to use this\n * with it, don't use any tweet id larger than 10000 in your test.\n */\npublic class TestScoringFunction extends ScoringFunction {\n  private ThriftSearchResultMetadata metadata = null;\n  private float score;\n\n  public TestScoringFunction(ImmutableSchemaInterface schema) {\n    super(schema);\n  }\n\n  @Override\n  protected float score(float luceneQueryScore) {\n    long tweetId = tweetIDMapper.getTweetID(getCurrentDocID());\n    this.score = (float) (tweetId / 10000.0);\n    System.out.println(String.format(\"score for tweet %10d is %6.3f\", tweetId, score));\n    return this.score;\n  }\n\n  @Override\n  protected Explanation doExplain(float luceneScore) {\n    return null;\n  }\n\n  @Override\n  public ThriftSearchResultMetadata getResultMetadata(ThriftSearchResultMetadataOptions options) {\n    if (metadata == null) {\n      metadata = new ThriftSearchResultMetadata()\n          .setResultType(ThriftSearchResultType.RELEVANCE)\n          .setPenguinVersion(EarlybirdConfig.getPenguinVersionByte());\n      metadata.setScore(score);\n    }\n    return metadata;\n  }\n\n  @Override\n  public void updateRelevanceStats(ThriftSearchResultsRelevanceStats relevanceStats) {\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/segment/DLSegmentDataProvider.java",
    "content": "package com.twitter.search.earlybird.segment;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Set;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.partitioning.base.Segment;\nimport com.twitter.search.common.util.io.dl.DLReaderWriterFactory;\nimport com.twitter.search.common.util.io.dl.SegmentDLUtil;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\n\n/**\n * An implementation of SegmentDataProvider using DistributedLog.\n */\npublic class DLSegmentDataProvider implements SegmentDataProvider {\n  private final int hashPartitionID;\n  private final DLReaderWriterFactory dlFactory;\n  private final SegmentDataReaderSet readerSet;\n\n  public DLSegmentDataProvider(\n      int hashPartitionID,\n      EarlybirdIndexConfig earlybirdIndexConfig,\n      DLReaderWriterFactory dlReaderWriterFactory) throws IOException {\n    this(hashPartitionID, earlybirdIndexConfig, dlReaderWriterFactory,\n        Clock.SYSTEM_CLOCK);\n  }\n\n  public DLSegmentDataProvider(\n    int hashPartitionID,\n    EarlybirdIndexConfig earlybirdIndexConfig,\n    DLReaderWriterFactory dlReaderWriterFactory,\n    Clock clock) throws IOException {\n    this.hashPartitionID = hashPartitionID;\n    this.dlFactory = dlReaderWriterFactory;\n    this.readerSet = new DLSegmentDataReaderSet(\n        dlFactory,\n        earlybirdIndexConfig,\n        clock);\n  }\n\n  @Override\n  public SegmentDataReaderSet getSegmentDataReaderSet() {\n    return readerSet;\n  }\n\n  @Override\n  public List<Segment> newSegmentList() throws IOException {\n    Set<String> segmentNames = SegmentDLUtil.getSegmentNames(dlFactory, null, hashPartitionID);\n    List<Segment> segmentList = new ArrayList<>(segmentNames.size());\n    for (String segmentName : segmentNames) {\n      Segment segment = Segment.fromSegmentName(segmentName, EarlybirdConfig.getMaxSegmentSize());\n      segmentList.add(segment);\n    }\n    // Sort the segments by ID.\n    Collections.sort(segmentList);\n    return segmentList;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/segment/DLSegmentDataReaderSet.java",
    "content": "package com.twitter.search.earlybird.segment;\n\nimport java.io.IOException;\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.Optional;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Function;\nimport com.google.common.base.Preconditions;\n\nimport org.apache.thrift.TException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchCustomGauge;\nimport com.twitter.search.common.metrics.SearchRequestStats;\nimport com.twitter.search.common.schema.earlybird.EarlybirdThriftDocumentUtil;\nimport com.twitter.search.common.schema.thriftjava.ThriftIndexingEvent;\nimport com.twitter.search.common.util.io.ReaderWithStatsFactory;\nimport com.twitter.search.common.util.io.TransformingRecordReader;\nimport com.twitter.search.common.util.io.dl.DLMultiStreamReader;\nimport com.twitter.search.common.util.io.dl.DLReaderWriterFactory;\nimport com.twitter.search.common.util.io.dl.DLTimestampedReaderFactory;\nimport com.twitter.search.common.util.io.dl.SegmentDLUtil;\nimport com.twitter.search.common.util.io.recordreader.RecordReader;\nimport com.twitter.search.common.util.io.recordreader.RecordReaderFactory;\nimport com.twitter.search.common.util.thrift.ThriftUtils;\nimport com.twitter.search.earlybird.EarlybirdIndexConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.document.DocumentFactory;\nimport com.twitter.search.earlybird.document.TweetDocument;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\n\npublic class DLSegmentDataReaderSet implements SegmentDataReaderSet {\n  private static final Logger LOG = LoggerFactory.getLogger(DLSegmentDataReaderSet.class);\n\n  public static final SearchRequestStats STATUS_DL_READ_STATS =\n      SearchRequestStats.export(\"status_dlreader\", TimeUnit.MICROSECONDS, false);\n  private static final SearchRequestStats UPDATE_EVENT_DL_READ_STATS =\n      SearchRequestStats.export(\"update_events_dlreader\", TimeUnit.MICROSECONDS, false);\n  // The number of tweets not indexed because they failed deserialization.\n  private static final SearchCounter STATUS_SKIPPED_DUE_TO_FAILED_DESERIALIZATION_COUNTER =\n      SearchCounter.export(\"statuses_skipped_due_to_failed_deserialization\");\n\n  @VisibleForTesting\n  public static final int FRESH_READ_THRESHOLD = (int) TimeUnit.MINUTES.toMillis(1);\n\n  private final int documentReadFreshnessThreshold =\n      EarlybirdConfig.getInt(\"documents_reader_freshness_threshold_millis\", 10000);\n  private final int updateReadFreshnessThreshold =\n      EarlybirdConfig.getInt(\"updates_freshness_threshold_millis\", FRESH_READ_THRESHOLD);\n  private final int dlReaderVersion = EarlybirdConfig.getInt(\"dl_reader_version\");\n\n  private final DLReaderWriterFactory dlFactory;\n  private final RecordReaderFactory<byte[]> dlUpdateEventsFactory;\n  private final EarlybirdIndexConfig indexConfig;\n  private final Clock clock;\n\n  private RecordReader<TweetDocument> documentReader;\n\n  // RecordReaders for update events that span all live segments.\n  private final RecordReader<ThriftVersionedEvents> updateEventsReader;\n  private final DLMultiStreamReader updateEventsMultiReader;\n  private final Map<Long, RecordReader<ThriftVersionedEvents>> updateEventReaders = new HashMap<>();\n\n  DLSegmentDataReaderSet(\n      DLReaderWriterFactory dlFactory,\n      final EarlybirdIndexConfig indexConfig,\n      Clock clock) throws IOException {\n    this.dlFactory = dlFactory;\n    this.indexConfig = indexConfig;\n    this.clock = clock;\n\n    this.dlUpdateEventsFactory = new ReaderWithStatsFactory(\n        new DLTimestampedReaderFactory(dlFactory, clock, updateReadFreshnessThreshold),\n        UPDATE_EVENT_DL_READ_STATS);\n    this.updateEventsMultiReader =\n        new DLMultiStreamReader(\"update_events\", dlUpdateEventsFactory, true, clock);\n    this.updateEventsReader =\n        new TransformingRecordReader<>(updateEventsMultiReader, record ->\n            (record != null) ? deserializeTVE(record.getBytes()) : null);\n\n    SearchCustomGauge.export(\"open_dl_update_events_streams\", updateEventReaders::size);\n  }\n\n  private ThriftVersionedEvents deserializeTVE(byte[] bytes) {\n    ThriftVersionedEvents event = new ThriftVersionedEvents();\n    try {\n      ThriftUtils.fromCompactBinaryFormat(bytes, event);\n      return event;\n    } catch (TException e) {\n      LOG.error(\"error deserializing TVE\", e);\n      return null;\n    }\n  }\n\n  @Override\n  public void attachDocumentReaders(SegmentInfo segmentInfo) throws IOException {\n    // Close any document reader left open before.\n    if (documentReader != null) {\n      LOG.warn(\"Previous documentReader not closed: {}\", documentReader);\n      completeSegmentDocs(segmentInfo);\n    }\n    documentReader = newDocumentReader(segmentInfo);\n  }\n\n  @Override\n  public void attachUpdateReaders(SegmentInfo segmentInfo) throws IOException {\n    if (updateEventsMultiReader == null) {\n      return;\n    }\n\n    String segmentName = segmentInfo.getSegmentName();\n    if (getUpdateEventsReaderForSegment(segmentInfo) != null) {\n      LOG.info(\"Update events reader for segment {} is already attached.\", segmentName);\n      return;\n    }\n\n    long updateEventStreamOffsetTimestamp = segmentInfo.getUpdatesStreamOffsetTimestamp();\n    LOG.info(\"Attaching update events reader for segment {} with timestamp: {}.\",\n             segmentName, updateEventStreamOffsetTimestamp);\n\n    String topic = SegmentDLUtil.getDLTopicForUpdateEvents(segmentName, dlReaderVersion);\n    RecordReader<byte[]> recordReader =\n        dlUpdateEventsFactory.newRecordReaderForTimestamp(topic, updateEventStreamOffsetTimestamp);\n    updateEventsMultiReader.addRecordReader(recordReader, topic);\n    updateEventReaders.put(segmentInfo.getTimeSliceID(),\n        new TransformingRecordReader<>(recordReader, this::deserializeTVE));\n  }\n\n  @Override\n  public void stopAll() {\n    if (documentReader != null) {\n      documentReader.close();\n    }\n    if (updateEventsReader != null) {\n      updateEventsReader.close();\n    }\n    try {\n      dlFactory.close();\n    } catch (IOException e) {\n      LOG.error(\"Exception while closing DL factory\", e);\n    }\n  }\n\n  @Override\n  public void completeSegmentDocs(SegmentInfo segmentInfo) {\n    if (documentReader != null) {\n      documentReader.close();\n      documentReader = null;\n    }\n  }\n\n  @Override\n  public void stopSegmentUpdates(SegmentInfo segmentInfo) {\n    if (updateEventsMultiReader != null) {\n      updateEventsMultiReader.removeStream(\n          SegmentDLUtil.getDLTopicForUpdateEvents(segmentInfo.getSegmentName(), dlReaderVersion));\n      updateEventReaders.remove(segmentInfo.getTimeSliceID());\n    }\n  }\n\n  @Override\n  public RecordReader<TweetDocument> newDocumentReader(SegmentInfo segmentInfo) throws IOException {\n    String topic = SegmentDLUtil.getDLTopicForTweets(segmentInfo.getSegmentName(),\n        EarlybirdConfig.getPenguinVersion(), dlReaderVersion);\n    final long timeSliceId = segmentInfo.getTimeSliceID();\n    final DocumentFactory<ThriftIndexingEvent> docFactory = indexConfig.createDocumentFactory();\n\n    // Create the underlying DLRecordReader wrapped with the tweet reader stats.\n    RecordReader<byte[]> dlReader = new ReaderWithStatsFactory(\n        new DLTimestampedReaderFactory(\n            dlFactory,\n            clock,\n            documentReadFreshnessThreshold),\n        STATUS_DL_READ_STATS)\n        .newRecordReader(topic);\n\n    // Create the wrapped reader which transforms serialized byte[] to TweetDocument.\n    return new TransformingRecordReader<>(\n        dlReader,\n        new Function<byte[], TweetDocument>() {\n          @Override\n          public TweetDocument apply(byte[] input) {\n            ThriftIndexingEvent event = new ThriftIndexingEvent();\n            try {\n              ThriftUtils.fromCompactBinaryFormat(input, event);\n            } catch (TException e) {\n              LOG.error(\"Could not deserialize status document\", e);\n              STATUS_SKIPPED_DUE_TO_FAILED_DESERIALIZATION_COUNTER.increment();\n              return null;\n            }\n\n            Preconditions.checkNotNull(event.getDocument());\n            return new TweetDocument(\n                docFactory.getStatusId(event),\n                timeSliceId,\n                EarlybirdThriftDocumentUtil.getCreatedAtMs(event.getDocument()),\n                docFactory.newDocument(event));\n          }\n        });\n  }\n\n  @Override\n  public RecordReader<TweetDocument> getDocumentReader() {\n    return documentReader;\n  }\n\n  @Override\n  public RecordReader<ThriftVersionedEvents> getUpdateEventsReader() {\n    return updateEventsReader;\n  }\n\n  @Override\n  public RecordReader<ThriftVersionedEvents> getUpdateEventsReaderForSegment(\n      SegmentInfo segmentInfo) {\n    return updateEventReaders.get(segmentInfo.getTimeSliceID());\n  }\n\n  @Override\n  public Optional<Long> getUpdateEventsStreamOffsetForSegment(SegmentInfo segmentInfo) {\n    String topic =\n        SegmentDLUtil.getDLTopicForUpdateEvents(segmentInfo.getSegmentName(), dlReaderVersion);\n    return updateEventsMultiReader.getUnderlyingOffsetForSegmentWithTopic(topic);\n  }\n\n  @Override\n  public boolean allCaughtUp() {\n    return ((getDocumentReader() == null) || getDocumentReader().isCaughtUp())\n        && ((getUpdateEventsReader() == null) || getUpdateEventsReader().isCaughtUp());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/segment/EmptySegmentDataReaderSet.java",
    "content": "package com.twitter.search.earlybird.segment;\n\nimport java.util.Optional;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.util.io.EmptyRecordReader;\nimport com.twitter.search.common.util.io.recordreader.RecordReader;\nimport com.twitter.search.earlybird.document.TweetDocument;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\n\n/**\n * A SegmentDataReaderSet that returns no data. Uses a DocumentReader that is\n * always caught up, but never gets exhausted.\n * Can be used for bringing up an earlybird against a static set of segments,\n * and will not incorporate any new updates.\n */\npublic class EmptySegmentDataReaderSet implements SegmentDataReaderSet {\n  public static final EmptySegmentDataReaderSet INSTANCE = new EmptySegmentDataReaderSet();\n\n  @Override\n  public void attachDocumentReaders(SegmentInfo segmentInfo) {\n  }\n\n  @Override\n  public void attachUpdateReaders(SegmentInfo segmentInfo) {\n  }\n\n  @Override\n  public void completeSegmentDocs(SegmentInfo segmentInfo) {\n  }\n\n  @Override\n  public void stopSegmentUpdates(SegmentInfo segmentInfo) {\n  }\n\n  @Override\n  public void stopAll() {\n  }\n\n  @Override\n  public boolean allCaughtUp() {\n    // ALWAYS CAUGHT UP\n    return true;\n  }\n\n  @Override\n  public RecordReader<TweetDocument> newDocumentReader(SegmentInfo segmentInfo)\n      throws Exception {\n    return null;\n  }\n\n  @Override\n  public RecordReader<TweetDocument> getDocumentReader() {\n    return new EmptyRecordReader<>();\n  }\n\n  @Override\n  public RecordReader<ThriftVersionedEvents> getUpdateEventsReader() {\n    return null;\n  }\n\n  @Override\n  public RecordReader<ThriftVersionedEvents> getUpdateEventsReaderForSegment(\n      SegmentInfo segmentInfo) {\n    return null;\n  }\n\n  @Override\n  public Optional<Long> getUpdateEventsStreamOffsetForSegment(SegmentInfo segmentInfo) {\n    return Optional.of(0L);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/segment/SegmentDataProvider.java",
    "content": "package com.twitter.search.earlybird.segment;\n\n/**\n * SegmentDataProvider provides information about available segments for indexing. This interface\n * abstracts away the actual source of the segment data. It might be a MySQL database, a mock\n * object, or a directory of flat files. It also provides access to the segmentInfoMap itself, which\n * contains information about the indexing state of Segments.\n */\npublic interface SegmentDataProvider extends SegmentProvider {\n  /**\n   * Returns the set of segment data record readers.\n   */\n  SegmentDataReaderSet getSegmentDataReaderSet();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/segment/SegmentDataReaderSet.java",
    "content": "package com.twitter.search.earlybird.segment;\n\nimport java.io.IOException;\nimport java.util.Optional;\n\nimport com.twitter.search.common.indexing.thriftjava.ThriftVersionedEvents;\nimport com.twitter.search.common.util.io.recordreader.RecordReader;\nimport com.twitter.search.earlybird.document.TweetDocument;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\n\n/**\n * SegmentDataReaderSet provides an interface to create and manage the various\n * RecordReaders used to index Earlybird segments.\n */\npublic interface SegmentDataReaderSet {\n  /**\n   * Instruct the document RecordReaders (i.e. document, geo, ... as appropriate) to read from this\n   * segment.\n   */\n  void attachDocumentReaders(SegmentInfo segmentInfo) throws IOException;\n\n  /**\n   * Instruct the reader set to add segment to non-document RecordReaders (deletes, features, etc.)\n   */\n  void attachUpdateReaders(SegmentInfo segmentInfo) throws IOException;\n\n  /**\n   * Mark a segment as \"complete\", denoting that we are done reading document records from it.\n   *\n   * This instructs the reader set to stop reading documents from the segment (if it hasn't\n   * already), although for now geo-document  records can still be read.  Updates RecordReaders\n   * (deletes, etc.) may continue to read entries for the segment.\n   */\n  void completeSegmentDocs(SegmentInfo segmentInfo);\n\n  /**\n   * This instructs the reader set to stop reading updates for the Segment.  It\n   * should remove the segment from all non-document RecordReaders (deletes, etc.)\n   */\n  void stopSegmentUpdates(SegmentInfo segmentInfo);\n\n  /**\n   * Stops all RecordReaders and closes all resources.\n   */\n  void stopAll();\n\n  /**\n   * Returns true if all RecordReaders are 'caught up' with the data sources they\n   * are reading from.  This might mean that the end of a file has been reached,\n   * or that we are waiting/polling for new records from an append-only database.\n   */\n  boolean allCaughtUp();\n\n  /**\n   * Create a new DocumentReader for the given segment that is not managed by this set.\n   */\n  RecordReader<TweetDocument> newDocumentReader(SegmentInfo segmentInfo) throws Exception;\n\n  /**\n   * Returns the document reader for the current segment.\n   */\n  RecordReader<TweetDocument> getDocumentReader();\n\n  /**\n   * Returns a combined update events reader for all segments.\n   */\n  RecordReader<ThriftVersionedEvents> getUpdateEventsReader();\n\n  /**\n   * Returns the update events reader for the given segment.\n   */\n  RecordReader<ThriftVersionedEvents> getUpdateEventsReaderForSegment(SegmentInfo segmentInfo);\n\n  /**\n   * Returns the offset in the update events stream for the given segment that this earlybird should\n   * start indexing from.\n   */\n  Optional<Long> getUpdateEventsStreamOffsetForSegment(SegmentInfo segmentInfo);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/segment/SegmentProvider.java",
    "content": "package com.twitter.search.earlybird.segment;\n\nimport java.io.IOException;\nimport java.util.List;\n\nimport com.twitter.search.common.partitioning.base.Segment;\n\npublic interface SegmentProvider {\n  /**\n   * Returns a *new* sorted list of all available segments on disk / db / hdfs / etc.\n   */\n  List<Segment> newSegmentList() throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/stats/EarlybirdRPCStats.java",
    "content": "package com.twitter.search.earlybird.stats;\n\nimport java.util.concurrent.TimeUnit;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchRequestStats;\n\n/**\n * SearchRequestStats with earlybird-specific additional stats.\n */\npublic final class EarlybirdRPCStats {\n  private final SearchRequestStats requestStats;\n  // Number of queries that were terminated early.\n  private final SearchCounter earlyTerminatedRequests;\n\n  // We do not count client error in the response error rate, but track it separately.\n  private final SearchRateCounter responseClientErrors;\n\n  public EarlybirdRPCStats(String name) {\n    requestStats = SearchRequestStats.export(name, TimeUnit.MICROSECONDS, true, true);\n    earlyTerminatedRequests = SearchCounter.export(name + \"_early_terminated\");\n    responseClientErrors = SearchRateCounter.export(name + \"_client_error\");\n  }\n\n  public long getRequestRate() {\n    return (long) (double) requestStats.getRequestRate().read();\n  }\n\n  public long getAverageLatency() {\n    return (long) (double) requestStats.getTimerStats().read();\n  }\n\n  /**\n   * Records a completed earlybird request.\n   * @param latencyUs how long the request took to complete, in microseconds.\n   * @param resultsCount how many results were returned.\n   * @param success whether the request was successful or not.\n   * @param earlyTerminated whether the request terminated early or not.\n   * @param clientError whether the request failure is caused by client errors\n   */\n  public void requestComplete(long latencyUs, long resultsCount, boolean success,\n                              boolean earlyTerminated, boolean clientError) {\n    // We treat client errors as successes for top-line metrics to prevent bad client requests (like\n    // malformed queries) from dropping our success rate and generating alerts.\n    requestStats.requestComplete(latencyUs, resultsCount, success || clientError);\n\n    if (earlyTerminated) {\n      earlyTerminatedRequests.increment();\n    }\n    if (clientError) {\n      responseClientErrors.increment();\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/stats/EarlybirdSearcherStats.java",
    "content": "package com.twitter.search.earlybird.stats;\n\nimport java.util.EnumMap;\nimport java.util.Map;\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchMetricTimerOptions;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.ranking.thriftjava.ThriftRankingParams;\nimport com.twitter.search.common.ranking.thriftjava.ThriftScoringFunctionType;\nimport com.twitter.search.earlybird.EarlybirdSearcher;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.ThriftSearchRelevanceOptions;\n\n/**\n * Manages counter and timer stats for EarlybirdSearcher.\n */\npublic class EarlybirdSearcherStats {\n  private static final TimeUnit TIME_UNIT = TimeUnit.MICROSECONDS;\n\n  private final SearchStatsReceiver earlybirdServerStatsReceiver;\n\n  public final SearchCounter thriftQueryWithSerializedQuery;\n  public final SearchCounter thriftQueryWithLuceneQuery;\n  public final SearchCounter thriftQueryWithoutTextQuery;\n  public final SearchCounter addedFilterBadUserRep;\n  public final SearchCounter addedFilterFromUserIds;\n  public final SearchCounter addedFilterTweetIds;\n  public final SearchCounter unsetFiltersForSocialFilterTypeQuery;\n  public final SearchCounter querySpecificSignalMapTotalSize;\n  public final SearchCounter querySpecificSignalQueriesUsed;\n  public final SearchCounter querySpecificSignalQueriesErased;\n  public final SearchCounter authorSpecificSignalMapTotalSize;\n  public final SearchCounter authorSpecificSignalQueriesUsed;\n  public final SearchCounter authorSpecificSignalQueriesErased;\n  public final SearchCounter nullcastTweetsForceExcluded;\n  public final SearchCounter nullcastUnexpectedResults;\n  public final SearchCounter nullcastUnexpectedQueries;\n  public final SearchCounter relevanceAntiGamingFilterUsed;\n  public final SearchCounter relevanceAntiGamingFilterNotRequested;\n  public final SearchCounter relevanceAntiGamingFilterSpecifiedTweetsAndFromUserIds;\n  public final SearchCounter relevanceAntiGamingFilterSpecifiedTweets;\n  public final SearchCounter relevanceAntiGamingFilterSpecifiedFromUserIds;\n  public final SearchCounter numCollectorAdjustedMinSearchedStatusID;\n\n  public final Map<EarlybirdSearcher.QueryMode, SearchCounter> numRequestsWithBlankQuery;\n  private final Map<ThriftScoringFunctionType, SearchTimerStats> latencyByScoringFunctionType;\n  private final Map<ThriftScoringFunctionType,\n      Map<String, SearchTimerStats>> latencyByScoringFunctionTypeAndClient;\n  private final Map<String, SearchTimerStats> latencyByTensorflowModel;\n\n  public EarlybirdSearcherStats(SearchStatsReceiver earlybirdServerStatsReceiver) {\n    this.earlybirdServerStatsReceiver = earlybirdServerStatsReceiver;\n\n    this.thriftQueryWithLuceneQuery =\n        earlybirdServerStatsReceiver.getCounter(\"thrift_query_with_lucene_query\");\n    this.thriftQueryWithSerializedQuery =\n        earlybirdServerStatsReceiver.getCounter(\"thrift_query_with_serialized_query\");\n    this.thriftQueryWithoutTextQuery =\n        earlybirdServerStatsReceiver.getCounter(\"thrift_query_without_text_query\");\n\n    this.addedFilterBadUserRep =\n        earlybirdServerStatsReceiver.getCounter(\"added_filter_bad_user_rep\");\n    this.addedFilterFromUserIds =\n        earlybirdServerStatsReceiver.getCounter(\"added_filter_from_user_ids\");\n    this.addedFilterTweetIds =\n        earlybirdServerStatsReceiver.getCounter(\"added_filter_tweet_ids\");\n\n    this.unsetFiltersForSocialFilterTypeQuery =\n        earlybirdServerStatsReceiver.getCounter(\"unset_filters_for_social_filter_type_query\");\n    this.querySpecificSignalMapTotalSize =\n        earlybirdServerStatsReceiver.getCounter(\"query_specific_signal_map_total_size\");\n    this.querySpecificSignalQueriesUsed =\n        earlybirdServerStatsReceiver.getCounter(\"query_specific_signal_queries_used\");\n    this.querySpecificSignalQueriesErased =\n        earlybirdServerStatsReceiver.getCounter(\"query_specific_signal_queries_erased\");\n    this.authorSpecificSignalMapTotalSize =\n        earlybirdServerStatsReceiver.getCounter(\"author_specific_signal_map_total_size\");\n    this.authorSpecificSignalQueriesUsed =\n        earlybirdServerStatsReceiver.getCounter(\"author_specific_signal_queries_used\");\n    this.authorSpecificSignalQueriesErased =\n        earlybirdServerStatsReceiver.getCounter(\"author_specific_signal_queries_erased\");\n    this.nullcastTweetsForceExcluded =\n        earlybirdServerStatsReceiver.getCounter(\"force_excluded_nullcast_result_count\");\n    this.nullcastUnexpectedResults =\n        earlybirdServerStatsReceiver.getCounter(\"unexpected_nullcast_result_count\");\n    this.nullcastUnexpectedQueries =\n        earlybirdServerStatsReceiver.getCounter(\"queries_with_unexpected_nullcast_results\");\n    this.numCollectorAdjustedMinSearchedStatusID =\n        earlybirdServerStatsReceiver.getCounter(\"collector_adjusted_min_searched_status_id\");\n\n    this.relevanceAntiGamingFilterUsed = earlybirdServerStatsReceiver\n        .getCounter(\"relevance_anti_gaming_filter_used\");\n    this.relevanceAntiGamingFilterNotRequested = earlybirdServerStatsReceiver\n        .getCounter(\"relevance_anti_gaming_filter_not_requested\");\n    this.relevanceAntiGamingFilterSpecifiedTweetsAndFromUserIds = earlybirdServerStatsReceiver\n        .getCounter(\"relevance_anti_gaming_filter_specified_tweets_and_from_user_ids\");\n    this.relevanceAntiGamingFilterSpecifiedTweets = earlybirdServerStatsReceiver\n        .getCounter(\"relevance_anti_gaming_filter_specified_tweets\");\n    this.relevanceAntiGamingFilterSpecifiedFromUserIds = earlybirdServerStatsReceiver\n        .getCounter(\"relevance_anti_gaming_filter_specified_from_user_ids\");\n\n    this.latencyByScoringFunctionType = new EnumMap<>(ThriftScoringFunctionType.class);\n    this.latencyByScoringFunctionTypeAndClient = new EnumMap<>(ThriftScoringFunctionType.class);\n    this.latencyByTensorflowModel = new ConcurrentHashMap<>();\n\n    for (ThriftScoringFunctionType type : ThriftScoringFunctionType.values()) {\n      this.latencyByScoringFunctionType.put(type, getTimerStatsByName(getStatsNameByType(type)));\n      this.latencyByScoringFunctionTypeAndClient.put(type, new ConcurrentHashMap<>());\n    }\n\n    this.numRequestsWithBlankQuery = new EnumMap<>(EarlybirdSearcher.QueryMode.class);\n\n    for (EarlybirdSearcher.QueryMode queryMode : EarlybirdSearcher.QueryMode.values()) {\n      String counterName =\n          String.format(\"num_requests_with_blank_query_%s\", queryMode.name().toLowerCase());\n\n      this.numRequestsWithBlankQuery.put(\n          queryMode, earlybirdServerStatsReceiver.getCounter(counterName));\n    }\n  }\n\n  /**\n   * Records the latency for a request for the applicable stats.\n   * @param timer A stopped timer that timed the request.\n   * @param request The request that was timed.\n   */\n  public void recordRelevanceStats(SearchTimer timer, EarlybirdRequest request) {\n    Preconditions.checkNotNull(timer);\n    Preconditions.checkNotNull(request);\n    Preconditions.checkArgument(!timer.isRunning());\n\n    ThriftSearchRelevanceOptions relevanceOptions = request.getSearchQuery().getRelevanceOptions();\n\n    // Only record ranking searches with a set type.\n    if (!relevanceOptions.isSetRankingParams()\n        || !relevanceOptions.getRankingParams().isSetType()) {\n      return;\n    }\n\n    ThriftRankingParams rankingParams = relevanceOptions.getRankingParams();\n    ThriftScoringFunctionType scoringFunctionType = rankingParams.getType();\n\n    latencyByScoringFunctionType.get(scoringFunctionType).stoppedTimerIncrement(timer);\n\n    if (request.getClientId() != null) {\n      getTimerStatsByClient(scoringFunctionType, request.getClientId())\n          .stoppedTimerIncrement(timer);\n    }\n\n    if (scoringFunctionType != ThriftScoringFunctionType.TENSORFLOW_BASED) {\n      return;\n    }\n\n    String modelName = rankingParams.getSelectedTensorflowModel();\n\n    if (modelName != null) {\n      getTimerStatsByTensorflowModel(modelName).stoppedTimerIncrement(timer);\n    }\n  }\n\n  /**\n   * Creates a search timer with options specified by TweetsEarlybirdSearcherStats.\n   * @return A new SearchTimer.\n   */\n  public SearchTimer createTimer() {\n    return new SearchTimer(new SearchMetricTimerOptions.Builder()\n        .withTimeUnit(TIME_UNIT)\n        .build());\n  }\n\n  private SearchTimerStats getTimerStatsByClient(\n      ThriftScoringFunctionType type,\n      String clientId) {\n    Map<String, SearchTimerStats> latencyByClient = latencyByScoringFunctionTypeAndClient.get(type);\n\n    return latencyByClient.computeIfAbsent(clientId,\n        cid -> getTimerStatsByName(getStatsNameByClientAndType(type, cid)));\n  }\n\n  private SearchTimerStats getTimerStatsByTensorflowModel(String modelName) {\n    return latencyByTensorflowModel.computeIfAbsent(modelName,\n        mn -> getTimerStatsByName(getStatsNameByTensorflowModel(mn)));\n  }\n\n  private SearchTimerStats getTimerStatsByName(String name) {\n    return earlybirdServerStatsReceiver.getTimerStats(\n        name, TIME_UNIT, false, true, false);\n  }\n\n  public static String getStatsNameByType(ThriftScoringFunctionType type) {\n    return String.format(\n        \"search_relevance_scoring_function_%s_requests\", type.name().toLowerCase());\n  }\n\n  public static String getStatsNameByClientAndType(\n      ThriftScoringFunctionType type,\n      String clientId) {\n    return String.format(\"%s_%s\", ClientIdUtil.formatClientId(clientId), getStatsNameByType(type));\n  }\n\n  public static String getStatsNameByTensorflowModel(String modelName) {\n    return String.format(\n        \"model_%s_%s\", modelName, getStatsNameByType(ThriftScoringFunctionType.TENSORFLOW_BASED));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/stats/SegmentSyncStats.java",
    "content": "package com.twitter.search.earlybird.stats;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.Timer;\n\npublic class SegmentSyncStats {\n  private static final String CPU_TOTAL = \"_cpu_total_\";\n  private static final String CPU_USER  = \"_cpu_user_mode_\";\n  private static final String CPU_SYS   = \"_cpu_system_mode_\";\n\n  private final SearchCounter segmentSyncLatency;\n  private final SearchCounter segmentSyncLatencyCpuTotal;\n  private final SearchCounter segmentSyncLatencyCpuUserMode;\n  private final SearchCounter segmentSyncLatencyCpuSystemMode;\n  private final SearchCounter segmentSyncCount;\n  private final SearchCounter segmentErrorCount;\n\n  private SegmentSyncStats(SearchCounter segmentSyncLatency,\n                           SearchCounter segmentSyncLatencyCpuTotal,\n                           SearchCounter segmentSyncLatencyCpuUserMode,\n                           SearchCounter segmentSyncLatencyCpuSystemMode,\n                           SearchCounter segmentSyncCount,\n                           SearchCounter segmentErrorCount) {\n    this.segmentSyncLatency = segmentSyncLatency;\n    this.segmentSyncLatencyCpuTotal = segmentSyncLatencyCpuTotal;\n    this.segmentSyncLatencyCpuUserMode = segmentSyncLatencyCpuUserMode;\n    this.segmentSyncLatencyCpuSystemMode = segmentSyncLatencyCpuSystemMode;\n    this.segmentSyncCount = segmentSyncCount;\n    this.segmentErrorCount = segmentErrorCount;\n  }\n\n  /**\n   * Creates a new set of stats for the given segment sync action.\n   * @param action the name to be used for the sync stats.\n   */\n  public SegmentSyncStats(String action) {\n    this(SearchCounter.export(\"segment_\" + action + \"_latency_ms\"),\n         SearchCounter.export(\"segment_\" + action + \"_latency\" + CPU_TOTAL + \"ms\"),\n         SearchCounter.export(\"segment_\" + action + \"_latency\" + CPU_USER + \"ms\"),\n         SearchCounter.export(\"segment_\" + action + \"_latency\" + CPU_SYS + \"ms\"),\n         SearchCounter.export(\"segment_\" + action + \"_count\"),\n         SearchCounter.export(\"segment_\" + action + \"_error_count\"));\n  }\n\n  /**\n   * Records a completed action using the specified timer.\n   */\n  public void actionComplete(Timer timer) {\n    segmentSyncCount.increment();\n    segmentSyncLatency.add(timer.getElapsed());\n    segmentSyncLatencyCpuTotal.add(timer.getElapsedCpuTotal());\n    segmentSyncLatencyCpuUserMode.add(timer.getElapsedCpuUserMode());\n    segmentSyncLatencyCpuSystemMode.add(timer.getElapsedCpuSystemMode());\n  }\n\n  public void recordError() {\n    segmentErrorCount.increment();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/tools/EarlybirdThriftRequestDeserializerUtil.java",
    "content": "package com.twitter.search.earlybird.tools;\n\nimport java.io.BufferedReader;\nimport java.io.IOException;\nimport java.nio.charset.Charset;\nimport java.nio.file.FileSystems;\nimport java.nio.file.Files;\nimport java.nio.file.Path;\n\nimport com.google.common.base.Preconditions;\n\nimport org.apache.commons.codec.binary.Base64;\nimport org.apache.thrift.TDeserializer;\nimport org.apache.thrift.TException;\n\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\n\n/**\n *\n * This tool deserializes the collected thrift requests into human readable format.\n *\n * Takes zero or one parameter: path to the thrift request log file.\n *\n * To run: Launch main from IntelliJ / Eclipse.\n */\npublic final class EarlybirdThriftRequestDeserializerUtil {\n  private static final String DEFAULT_LOG_FILE_LOCATION = \"/tmp/eb_req.B64\";\n  // Not threadsafe. Single thread main().\n  private static final Base64 B64 = new Base64(0);\n  private static final TDeserializer DESERIALIZER = new TDeserializer();\n\n  private EarlybirdThriftRequestDeserializerUtil() {\n  }\n\n  /**\n   * Runs the EarlybirdThriftRequestDeserializerUtil tool with the given command-line arguments.\n   */\n  public static void main(String[] args) throws IOException {\n    Path logFile = null;\n    if (args.length == 1) {\n      logFile = FileSystems.getDefault().getPath(args[0]);\n    } else if (args.length == 0) {\n      logFile = FileSystems.getDefault().getPath(DEFAULT_LOG_FILE_LOCATION);\n    } else {\n      System.err.println(\"Usage: takes zero or one parameter (log file path). \"\n          + \"If no log file is specified, \" + DEFAULT_LOG_FILE_LOCATION + \" is used.\");\n      //CHECKSTYLE:OFF RegexpSinglelineJava\n      System.exit(-1);\n      //CHECKSTYLE:ON RegexpSinglelineJava\n    }\n    Preconditions.checkState(logFile.toFile().exists());\n\n    BufferedReader reader = Files.newBufferedReader(logFile, Charset.defaultCharset());\n    try {\n      String line;\n      while ((line = reader.readLine()) != null) {\n        EarlybirdRequest ebRequest = deserializeEBRequest(line);\n        if (ebRequest != null) {\n          System.out.println(ebRequest);\n        }\n      }\n    } finally {\n      reader.close();\n    }\n  }\n\n  private static EarlybirdRequest deserializeEBRequest(String line) {\n    EarlybirdRequest ebRequest = new EarlybirdRequest();\n    byte[] bytes = B64.decode(line);\n    try {\n      DESERIALIZER.deserialize(ebRequest, bytes);\n    } catch (TException e) {\n      System.err.println(\"Error deserializing thrift.\");\n    }\n    return ebRequest;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/ActionLogger.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.util.concurrent.Callable;\n\nimport com.google.common.base.Stopwatch;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic final class ActionLogger {\n  private static final Logger LOG = LoggerFactory.getLogger(ActionLogger.class);\n\n  private ActionLogger() {\n  }\n\n  /**\n   * Run a function, logging a message at the start and end, and the time it took.\n   */\n  public static <T> T call(String message, Callable<T> fn) throws Exception {\n    LOG.info(\"Action starting: '{}'.\", message);\n    Stopwatch stopwatch = Stopwatch.createStarted();\n    try {\n      return fn.call();\n    } catch (Throwable e) {\n      LOG.error(\"Action failed: '{}'.\", message, e);\n      throw e;\n    } finally {\n      LOG.info(\"Action finished in {} '{}'.\", stopwatch, message);\n    }\n  }\n\n  /**\n   * Run a function, logging a message at the start and end, and the time it took.\n   */\n  public static void run(String message, CheckedRunnable fn) throws Exception {\n    call(message, () -> {\n      fn.run();\n      return null;\n    });\n  }\n\n  @FunctionalInterface\n  public interface CheckedRunnable {\n    /**\n     * A nullary function that throws checked exceptions.\n     */\n    void run() throws Exception;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/CoordinatedEarlybirdAction.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.util.Optional;\nimport java.util.Random;\nimport java.util.concurrent.atomic.AtomicBoolean;\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.base.Stopwatch;\n\nimport org.apache.zookeeper.KeeperException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.base.ExceptionalFunction;\nimport com.twitter.common.quantity.Amount;\nimport com.twitter.common.quantity.Time;\nimport com.twitter.common.zookeeper.ServerSet;\nimport com.twitter.common.zookeeper.ZooKeeperClient;\nimport com.twitter.search.common.config.Config;\nimport com.twitter.search.common.database.DatabaseConfig;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchCustomGauge;\nimport com.twitter.search.common.util.zktrylock.TryLock;\nimport com.twitter.search.common.util.zktrylock.ZooKeeperTryLockFactory;\nimport com.twitter.search.earlybird.ServerSetMember;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\nimport com.twitter.search.earlybird.exception.AlreadyInServerSetUpdateException;\nimport com.twitter.search.earlybird.exception.EarlybirdException;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.exception.NotInServerSetUpdateException;\nimport com.twitter.search.earlybird.partition.DynamicPartitionConfig;\nimport com.twitter.search.earlybird.partition.PartitionConfig;\nimport com.twitter.search.earlybird.partition.SegmentSyncConfig;\n\n/**\n * Utility class for executing tasks on Earlybirds that need to be coordinated across replicas\n * on the same hash partition.\n * Can be used for things like coordinating optimization on the same timeslice.\n * When enabled, a try-lock will be taken out in zookeeper while the task is performed.\n * The action will attempt to leave the partition's server set. If the attempt fails, the action\n * is aborted.\n */\npublic class CoordinatedEarlybirdAction implements CoordinatedEarlybirdActionInterface {\n  private static final Logger LOG = LoggerFactory.getLogger(CoordinatedEarlybirdAction.class);\n\n  private static final Boolean COORDINATED_ACTION_FLAG = Boolean.TRUE;\n  private static final Boolean NOT_COORDINATED_ACTION_FLAG = Boolean.FALSE;\n\n  private final String actionName;\n  private final DynamicPartitionConfig dynamicPartitionConfig;\n  @Nullable private final ServerSetMember serverSetMember;\n  private final ZooKeeperTryLockFactory zooKeeperTryLockFactory;\n\n  // Whether this action should be coordinated through zookeeper in the first place (could be\n  // config'ed off).\n  // If the action is coordinated, this earlybird will leave its server set when performing the\n  // coordinated action.\n  private final AtomicBoolean shouldSynchronize;\n  // Whether this action should ensure that there are enough replicas in the serverset (defined by\n  // maxAllowedReplicasNotInServerSet) before leaving the serverset.\n  private final boolean checkNumReplicasInServerSet;\n  // If this many (or more) servers have left the partition, we cannot perform a coordinated action\n  private final int maxAllowedReplicasNotInServerSet;\n  // How long to lock out all other replicas in this hash partition for.\n  // Should be some small multiple of how long the action is expected to take, to allow for longer\n  // running cases.\n  private final long zkLockExpirationTimeMinutes;\n  // Prefix for the zookeeper lock used when coordinating daily updates.\n  // Full name should include the hash partition number.\n  private final String zkLockNamePrefix;\n  // If we're unable to re-join this earlybird's server set during coordinated updates,\n  // how many times to retry.\n  private final int joinServerSetRetries;\n  // How long to sleep between retries if unable to job back into server set.\n  private final int joinServerSetRetrySleepMillis;\n  // How long to sleep between leaving the serverset and executing the action\n  private final int sleepAfterLeaveServerSetMillis;\n\n  // How many times a this action was called within a lock block.\n  private final SearchCounter numCoordinatedFunctionCalls;\n  private final SearchCounter numCoordinatedLeaveServersetCalls;\n\n  private final CriticalExceptionHandler criticalExceptionHandler;\n  private final SegmentSyncConfig segmentSyncConfig;\n\n  /**\n   * Create a CoordinatedEarlybirdAction.\n   *\n   * @param actionName the name to be used for logging and the prefix for config options.\n   * @param dynamicPartitionConfig maintains the current partitioning configuration for this\n   * earlybird. Used mainly to determine the hash partition of this earlybird.\n   * @param serverSetMember the server that this action is running on. To be used to leaving and\n   * rejoining the server's server set.\n   */\n  public CoordinatedEarlybirdAction(\n      ZooKeeperTryLockFactory zooKeeperTryLockFactory,\n      String actionName,\n      DynamicPartitionConfig dynamicPartitionConfig,\n      @Nullable ServerSetMember serverSetMember,\n      CriticalExceptionHandler criticalExceptionHandler,\n      SegmentSyncConfig segmentSyncConfig) {\n    this.actionName = actionName;\n    this.dynamicPartitionConfig = dynamicPartitionConfig;\n    this.serverSetMember = serverSetMember;\n    this.criticalExceptionHandler = criticalExceptionHandler;\n    this.segmentSyncConfig = segmentSyncConfig;\n    this.zooKeeperTryLockFactory = zooKeeperTryLockFactory;\n    if (serverSetMember == null) {\n      Preconditions.checkState(Config.environmentIsTest(),\n          \"Should only have a null server in tests\");\n    }\n\n    this.shouldSynchronize = new AtomicBoolean(\n            EarlybirdConfig.getBool(actionName + \"_should_synchronize\", false));\n\n    // Export whether or not synchronization is enabled as a stat\n    SearchCustomGauge.export(\n        actionName + \"_should_synchronize\", () -> shouldSynchronize.get() ? 1 : 0);\n\n    this.checkNumReplicasInServerSet = EarlybirdProperty.CHECK_NUM_REPLICAS_IN_SERVER_SET.get();\n\n    int numReplicas =\n        dynamicPartitionConfig.getCurrentPartitionConfig().getNumReplicasInHashPartition();\n    this.maxAllowedReplicasNotInServerSet =\n        EarlybirdProperty.MAX_ALLOWED_REPLICAS_NOT_IN_SERVER_SET.get(numReplicas);\n\n    this.zkLockExpirationTimeMinutes =\n        EarlybirdConfig.getLong(actionName + \"_lock_expiration_time_minutes\", 60L);\n    this.zkLockNamePrefix = actionName + \"_for_hash_partition_\";\n    this.joinServerSetRetries =\n        EarlybirdConfig.getInt(actionName + \"_join_server_set_retries\", 20);\n    this.joinServerSetRetrySleepMillis =\n        EarlybirdConfig.getInt(actionName + \"_join_server_retry_sleep_millis\", 2000);\n    this.sleepAfterLeaveServerSetMillis =\n        EarlybirdConfig.getInt(\"coordinated_action_sleep_after_leave_server_set_millis\", 30000);\n\n    this.numCoordinatedFunctionCalls = SearchCounter.export(actionName + \"_num_coordinated_calls\");\n    this.numCoordinatedLeaveServersetCalls =\n        SearchCounter.export(actionName + \"_num_coordinated_leave_serverset_calls\");\n\n    if (this.checkNumReplicasInServerSet) {\n      LOG.info(\n          \"Coordinate action config ({}): allowedNotIn: {}, current number of replicas: {}, \"\n              + \"synchronization enabled: {}, checkNumReplicasInServerSet enabled: {}\",\n          actionName,\n          maxAllowedReplicasNotInServerSet,\n          dynamicPartitionConfig.getCurrentPartitionConfig().getNumReplicasInHashPartition(),\n          shouldSynchronize,\n          this.checkNumReplicasInServerSet);\n    } else {\n      LOG.info(\n          \"Coordinate action config ({}): synchronization enabled: {}, \"\n              + \"checkNumReplicasInServerSet enabled: {}\",\n          actionName,\n          shouldSynchronize,\n          this.checkNumReplicasInServerSet);\n    }\n  }\n\n\n  @Override\n  public <E extends Exception> boolean execute(\n      String description,\n      ExceptionalFunction<Boolean, Boolean, E> function)\n          throws E, CoordinatedEarlybirdActionLockFailed {\n    if (this.shouldSynchronize.get()) {\n      return executeWithCoordination(description, function);\n    } else {\n      return function.apply(NOT_COORDINATED_ACTION_FLAG);\n    }\n  }\n\n  enum LeaveServerSetResult {\n    SUCCESS,\n    FAILURE,\n    NOT_IN_SERVER_SET,\n    NO_SERVER_SET_MEMBER\n  }\n\n  private LeaveServerSetResult leaveServerSet() {\n    LOG.info(\"Leaving serving server set for \" + actionName);\n    try {\n      serverSetMember.leaveServerSet(\"CoordinatedAction: \" + actionName);\n      return LeaveServerSetResult.SUCCESS;\n    } catch (ServerSet.UpdateException ex) {\n      if (ex instanceof NotInServerSetUpdateException) {\n        LOG.info(\"No need to leave; already out of server set during: \"\n            + actionName, ex);\n        return LeaveServerSetResult.NOT_IN_SERVER_SET;\n      } else {\n        LOG.warn(\"Unable to leave server set during: \" + actionName, ex);\n        return LeaveServerSetResult.FAILURE;\n      }\n    }\n  }\n\n  private LeaveServerSetResult maybeLeaveServerSet() {\n    if (serverSetMember != null) {\n      if (serverSetMember.isInServerSet()) {\n\n        if (!checkNumReplicasInServerSet) {\n          return leaveServerSet();\n        } else {\n          PartitionConfig curPartitionConfig = dynamicPartitionConfig.getCurrentPartitionConfig();\n          final int minNumServers =\n              curPartitionConfig.getNumReplicasInHashPartition() - maxAllowedReplicasNotInServerSet;\n          Optional<Integer> numServerSetMembers = getNumberOfServerSetMembers();\n          LOG.info(\"Checking number of replicas before leaving server set for \" + actionName\n              + \". Number of members is: \" + numServerSetMembers + \" minMembers: \" + minNumServers);\n          if (numServerSetMembers.isPresent() && numServerSetMembers.get() > minNumServers) {\n            return leaveServerSet();\n          } else {\n            LOG.warn(\"Not leaving server set during: \" + actionName);\n            return LeaveServerSetResult.FAILURE;\n          }\n        }\n      } else {\n        LOG.info(\"Not in server set, no need to leave it.\");\n        return LeaveServerSetResult.NOT_IN_SERVER_SET;\n      }\n    }\n\n    return LeaveServerSetResult.NO_SERVER_SET_MEMBER;\n  }\n\n  private <E extends Exception> boolean executeWithCoordination(\n      final String description,\n      final ExceptionalFunction<Boolean, Boolean, E> function)\n      throws E, CoordinatedEarlybirdActionLockFailed {\n    PartitionConfig curPartitionConfig = dynamicPartitionConfig.getCurrentPartitionConfig();\n    TryLock lock = zooKeeperTryLockFactory.createTryLock(\n        DatabaseConfig.getLocalHostname(),\n        segmentSyncConfig.getZooKeeperSyncFullPath(),\n        zkLockNamePrefix\n        + curPartitionConfig.getIndexingHashPartitionID(),\n        Amount.of(zkLockExpirationTimeMinutes, Time.MINUTES)\n    );\n\n    final AtomicBoolean success = new AtomicBoolean(false);\n\n    boolean gotLock = lock.tryWithLock(() -> {\n      Stopwatch actionTiming = Stopwatch.createStarted();\n\n      LeaveServerSetResult leftServerSet = maybeLeaveServerSet();\n      if (leftServerSet == LeaveServerSetResult.FAILURE) {\n        LOG.info(\"Failed to leave the server set, will not execute action.\");\n        return;\n      }\n\n      LOG.info(\"maybeLeaveServerSet returned: {}\", leftServerSet);\n\n      // Sleep for a short time to give the server some time to finish requests that it is currently\n      // executing and allow roots some time to register that this host has left the server set.\n      // If we didn't do this and the coordinated action included a full GC, then latency and error\n      // rate at the root layer would spike higher at the time of the GC. SEARCH-35456\n      try {\n        Thread.sleep(sleepAfterLeaveServerSetMillis);\n      } catch (InterruptedException ex) {\n        Thread.currentThread().interrupt();\n      }\n\n      LOG.info(actionName + \" synchronization action for \" + description);\n\n      try {\n        numCoordinatedFunctionCalls.increment();\n        numCoordinatedLeaveServersetCalls.increment();\n\n        Boolean successValue = function.apply(COORDINATED_ACTION_FLAG);\n        success.set(successValue);\n      } finally {\n        if (leftServerSet == LeaveServerSetResult.SUCCESS) {\n          joinServerSet();\n        }\n        LOG.info(\"{} synchronization action for {} completed after {}, success: {}\",\n            actionName,\n            description,\n            actionTiming,\n            success.get());\n      }\n    });\n\n    if (!gotLock) {\n      String errorMsg = actionName + \": Failed to get zk indexing lock for \" + description;\n      LOG.info(errorMsg);\n      throw new CoordinatedEarlybirdActionLockFailed(errorMsg);\n    }\n    return success.get();\n  }\n\n  @Override\n  public void retryActionUntilRan(String description, Runnable action) {\n    Random random = new Random(System.currentTimeMillis());\n\n    boolean actionExecuted = false;\n    int attempts = 0;\n    while (!actionExecuted) {\n      try {\n        attempts++;\n        actionExecuted = this.execute(description, isCoordinated -> {\n          action.run();\n          return true;\n        });\n      } catch (CoordinatedEarlybirdActionLockFailed ex) {\n      }\n\n      if (!actionExecuted) {\n        // Variable sleep amount. The reason for the random sleeps\n        // is so that across multiple earlybirds this doesn't get\n        // executed in some sequence that depends on something else\n        // like maybe deploy times. It might be easier to catch possible\n        // problems if implicit orderings like this are not introduced.\n        long msToSleep = (10 + random.nextInt(5)) * 1000L;\n        try {\n          Thread.sleep(msToSleep);\n        } catch (InterruptedException ex) {\n          LOG.info(\"Interrupted while trying to execute\");\n          Thread.currentThread().interrupt();\n        }\n      } else {\n        LOG.info(\"Executed {} after {} attempts\", actionName, attempts);\n      }\n    }\n  }\n\n  /**\n   * Gets the current number of servers in this server's server set.\n   * @return absent Optional if we encountered an exception getting the number of hosts.\n   */\n  private Optional<Integer> getNumberOfServerSetMembers() {\n    try {\n      return serverSetMember != null ? Optional.of(serverSetMember.getNumberOfServerSetMembers())\n          : Optional.empty();\n    } catch (InterruptedException ex) {\n      LOG.warn(\"Action \" + actionName + \" was interrupted.\", ex);\n      Thread.currentThread().interrupt();\n      return Optional.empty();\n    } catch (ZooKeeperClient.ZooKeeperConnectionException | KeeperException ex) {\n      LOG.warn(\"Exception during \" + actionName, ex);\n      return Optional.empty();\n    }\n  }\n\n  /**\n   * After a coordinated action, join back this earlybird's server set with retries\n   * and sleeps in between.\n   */\n  private void joinServerSet() {\n    Preconditions.checkNotNull(serverSetMember);\n\n    boolean joined = false;\n    for (int i = 0; i < joinServerSetRetries; i++) {\n      try {\n        serverSetMember.joinServerSet(\"CoordinatedAction: \" + actionName);\n        joined = true;\n        break;\n      } catch (AlreadyInServerSetUpdateException ex) {\n        // Most likely leaving the server set failed\n        joined = true;\n        break;\n      } catch (ServerSet.UpdateException ex) {\n        LOG.warn(\"Unable to join server set after \" + actionName + \" on attempt \"\n                + i, ex);\n        if (i < (joinServerSetRetries - 1)) {\n          try {\n            Thread.sleep(joinServerSetRetrySleepMillis);\n          } catch (InterruptedException e) {\n            LOG.warn(\"Interrupted while waiting to join back server set for: \" + actionName);\n            // Preserve interrupt status.\n            Thread.currentThread().interrupt();\n            break;\n          }\n        }\n      }\n    }\n    if (!joined) {\n      String message = String.format(\n          \"Unable to join server set after %s, setting fatal flag.\",\n          actionName);\n      EarlybirdException exception = new EarlybirdException(message);\n\n      LOG.error(message, exception);\n      criticalExceptionHandler.handle(this, exception);\n    }\n  }\n\n\n  @Override\n  public boolean setShouldSynchronize(boolean shouldSynchronizeParam) {\n    boolean oldValue = this.shouldSynchronize.getAndSet(shouldSynchronizeParam);\n    LOG.info(\"Updated shouldSynchronize for: \" + actionName + \" from \" + oldValue\n            + \" to \" + shouldSynchronizeParam);\n    return oldValue;\n  }\n\n  @Override\n  @VisibleForTesting\n  public long getNumCoordinatedFunctionCalls() {\n    return this.numCoordinatedFunctionCalls.get();\n  }\n\n  @Override\n  @VisibleForTesting\n  public long getNumCoordinatedLeaveServersetCalls() {\n    return this.numCoordinatedLeaveServersetCalls.get();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/CoordinatedEarlybirdActionInterface.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport com.twitter.common.base.ExceptionalFunction;\n\npublic interface CoordinatedEarlybirdActionInterface {\n    /**\n     * Executes the provided Function associated with the given segment.\n     * @param description a name for the action to be exected.\n     * @param function the function to call in a coordinated manner.\n     *        As input, the function will receive a flag indicating whether or not it is being\n     *        called in a coordinated fashion. true if it is, and false otherwise.\n     * @return true iff the function was executed, and function.apply() returned true;\n     * throws CoordinatedEarlybirdActionLockFailed if function is not executed (because lock\n     * aquisition failed).\n     */\n    <E extends Exception> boolean execute(\n        String description,\n        ExceptionalFunction<Boolean, Boolean, E> function)\n          throws E, CoordinatedEarlybirdActionLockFailed;\n\n    /**\n     * Set whether this action should be synchronized.\n     * If not, the action is directly applied. If yes, Earlybirds will coordinate executing the\n     * action via ZooKeeperTryLocks.\n     */\n    boolean setShouldSynchronize(boolean shouldSynchronizeParam);\n\n    /**\n     * Number of times this coordinated actions has been executed.\n     * @return\n     */\n    @VisibleForTesting\n    long getNumCoordinatedFunctionCalls();\n\n    /**\n     * Number of times we have left the serverset.\n     * @return\n     */\n    @VisibleForTesting\n    long getNumCoordinatedLeaveServersetCalls();\n\n    /**\n     * Retry until we can run an action on a single instance in the serverset.\n     * @param description Text description of the action.\n     * @param action A runnable to be ran.\n     */\n    void retryActionUntilRan(String description, Runnable action);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/CoordinatedEarlybirdActionLockFailed.java",
    "content": "package com.twitter.search.earlybird.util;\n\n/**\n * This class represents that coordindated earlybird action can not acquire the lock so that it\n * throws this exception.\n */\npublic class CoordinatedEarlybirdActionLockFailed extends Exception {\n  public CoordinatedEarlybirdActionLockFailed(String message) {\n    super(message);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/EarlybirdDecider.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport scala.Some;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.decider.Decider;\nimport com.twitter.decider.Decider$;\nimport com.twitter.decider.RandomRecipient$;\nimport com.twitter.decider.Recipient;\nimport com.twitter.decider.decisionmaker.MutableDecisionMaker;\nimport com.twitter.search.common.decider.DeciderUtil;\nimport com.twitter.search.common.decider.SearchDeciderFactory;\nimport com.twitter.search.earlybird.common.config.EarlybirdProperty;\n\n/**\n * A Singleton to let any code in Earlybird have the ability to be guarded by a decider key.\n *\n * EarlybirdDecider is a thin wrapper around the Twitter Decider library to provide global access to a single\n * decider configuration. This way any code anywhere can easily be guarded by a Decider key. The initializer requires\n * EarlybirdConfig to be initialized already. Defaults to a NullDecider, which causes all requests for keys to return\n * false.\n */\npublic final class EarlybirdDecider {\n  public static final org.slf4j.Logger LOG =\n      org.slf4j.LoggerFactory.getLogger(EarlybirdDecider.class);\n  public static final String DECIDER_CONFIG = \"./config/earlybird-decider.yml\";\n\n  private static volatile Decider earlybirdDecider = Decider$.MODULE$.NullDecider();\n  private static volatile MutableDecisionMaker mutableDecisionMaker;\n\n  private EarlybirdDecider() { }\n\n  /**\n   * Initializes the global decider accessor. Requires EarlybirdConfig to be initialized.\n   *\n   * @return the new decider interface.\n   */\n  public static Decider initialize() {\n    return initialize(DECIDER_CONFIG);\n  }\n\n  /**\n   * Initializes the global decider accessor. Requires EarlybirdConfig to be initialized.\n   *\n   * @param configPath path to the base decider config file.\n   * @return the new decider interface.\n   */\n  @VisibleForTesting public static Decider initialize(String configPath) {\n    synchronized (EarlybirdDecider.class) {\n      Preconditions.checkState(earlybirdDecider == Decider$.MODULE$.NullDecider(),\n                               \"EarlybirdDecider can be initialized only once.\");\n\n      mutableDecisionMaker = new MutableDecisionMaker();\n\n      if (EarlybirdProperty.USE_DECIDER_OVERLAY.get(false)) {\n        String category = EarlybirdProperty.DECIDER_OVERLAY_CONFIG.get();\n        earlybirdDecider =\n            SearchDeciderFactory.createDeciderWithoutRefreshBaseWithOverlay(\n                configPath, category, mutableDecisionMaker);\n        LOG.info(\"EarlybirdDecider set to use the decider overlay \" + category);\n      } else {\n        earlybirdDecider =\n            SearchDeciderFactory.createDeciderWithRefreshBaseWithoutOverlay(\n                configPath, mutableDecisionMaker);\n        LOG.info(\"EarlybirdDecider set to only use the base config\");\n      }\n      return earlybirdDecider;\n    }\n  }\n\n  /**\n   * Check if feature is available based on randomness\n   *\n   * @param feature the feature name to test\n   * @return true if the feature is available, false otherwise\n   */\n  public static boolean isFeatureAvailable(String feature) {\n    return isFeatureAvailable(feature, RandomRecipient$.MODULE$);\n  }\n\n  /**\n   * Check if the feature is available based on the user\n   *\n   * The recipient'd id is hashed and used as the value to compare with the decider percentage. Therefore, the same user\n   * will always get the same result for a given percentage, and higher percentages should always be a superset of the\n   * lower percentage users.\n   *\n   * RandomRecipient can be used to get a random value for every call.\n   *\n   * @param feature the feature name to test\n   * @param recipient the recipient to base a decision on\n   * @return true if the feature is available, false otherwise\n   */\n  public static boolean isFeatureAvailable(String feature, Recipient recipient) {\n    if (earlybirdDecider == Decider$.MODULE$.NullDecider()) {\n      LOG.warn(\"EarlybirdDecider is uninitialized but requested feature \" + feature);\n    }\n\n    return earlybirdDecider.isAvailable(feature, Some.apply(recipient));\n  }\n\n  /**\n   * Get the raw decider value for a given feature.\n   *\n   * @param feature the feature name\n   * @return the integer value of the decider\n   */\n  public static int getAvailability(String feature) {\n    return DeciderUtil.getAvailability(earlybirdDecider, feature);\n  }\n\n  public static Decider getDecider() {\n    checkInitialized();\n    return earlybirdDecider;\n  }\n\n  public static MutableDecisionMaker getMutableDecisionMaker() {\n    checkInitialized();\n    return mutableDecisionMaker;\n  }\n\n  private static void checkInitialized() {\n    Preconditions.checkState(earlybirdDecider != Decider$.MODULE$.NullDecider(),\n        \"EarlybirdDecider is not initialized.\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/EarlybirdSearchResultUtil.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.collect.ImmutableMap;\n\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.database.DatabaseConfig;\nimport com.twitter.search.common.query.thriftjava.EarlyTerminationInfo;\nimport com.twitter.search.common.util.earlybird.ResultsUtil;\nimport com.twitter.search.common.util.earlybird.ThriftSearchResultUtil;\nimport com.twitter.search.common.util.earlybird.ThriftSearchResultsRelevanceStatsUtil;\nimport com.twitter.search.core.earlybird.facets.LanguageHistogram;\nimport com.twitter.search.earlybird.partition.PartitionConfig;\nimport com.twitter.search.earlybird.search.Hit;\nimport com.twitter.search.earlybird.search.SearchResultsInfo;\nimport com.twitter.search.earlybird.search.SimpleSearchResults;\nimport com.twitter.search.earlybird.search.relevance.RelevanceSearchResults;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultDebugInfo;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultsRelevanceStats;\n\n// EarlybirdSearchResultUtil contains some simple static methods for constructing\n// ThriftSearchResult objects.\npublic final class EarlybirdSearchResultUtil {\n  public static final double MIN_LANGUAGE_RATIO_TO_KEEP = 0.002;\n\n  private EarlybirdSearchResultUtil() { }\n\n  /**\n   * Update result stats on the ThriftSearchResult.\n   */\n  public static void setResultStatistics(ThriftSearchResults results, SearchResultsInfo info) {\n    results.setNumHitsProcessed(info.getNumHitsProcessed());\n    results.setNumPartitionsEarlyTerminated(info.isEarlyTerminated() ? 1 : 0);\n    if (info.isSetSearchedStatusIDs()) {\n      results.setMaxSearchedStatusID(info.getMaxSearchedStatusID());\n      results.setMinSearchedStatusID(info.getMinSearchedStatusID());\n    }\n\n    if (info.isSetSearchedTimes()) {\n      results.setMaxSearchedTimeSinceEpoch(info.getMaxSearchedTime());\n      results.setMinSearchedTimeSinceEpoch(info.getMinSearchedTime());\n    }\n  }\n\n  /**\n   * Create an EarlyTerminationInfo based on information inside a SearchResultsInfo.\n   */\n  public static EarlyTerminationInfo prepareEarlyTerminationInfo(SearchResultsInfo info) {\n    EarlyTerminationInfo earlyTerminationInfo = new EarlyTerminationInfo(info.isEarlyTerminated());\n    if (info.isEarlyTerminated()) {\n      earlyTerminationInfo.setEarlyTerminationReason(info.getEarlyTerminationReason());\n    }\n    return earlyTerminationInfo;\n  }\n\n  /**\n   * Populate language histogram inside ThriftSerachResults.\n   */\n  public static void setLanguageHistogram(ThriftSearchResults results,\n                                          LanguageHistogram languageHistogram) {\n    int sum = 0;\n    for (int value : languageHistogram.getLanguageHistogram()) {\n      sum += value;\n    }\n    if (sum == 0) {\n      return;\n    }\n    ImmutableMap.Builder<ThriftLanguage, Integer> builder = ImmutableMap.builder();\n    int threshold = (int) (sum * MIN_LANGUAGE_RATIO_TO_KEEP);\n    for (Map.Entry<ThriftLanguage, Integer> entry : languageHistogram.getLanguageHistogramAsMap()\n                                                                     .entrySet()) {\n      if (entry.getValue() > threshold) {\n        builder.put(entry.getKey(), entry.getValue());\n      }\n    }\n    Map<ThriftLanguage, Integer> langCounts = builder.build();\n    if (langCounts.size() > 0) {\n      results.setLanguageHistogram(langCounts);\n    }\n  }\n\n  private static void addDebugInfoToResults(List<ThriftSearchResult> resultArray,\n                                            @Nullable PartitionConfig partitionConfig) {\n    if (partitionConfig == null) {\n      return;\n    }\n    ThriftSearchResultDebugInfo debugInfo = new ThriftSearchResultDebugInfo();\n    debugInfo.setHostname(DatabaseConfig.getLocalHostname());\n    // These info can also come from EarlybirdServer.get().getPartitionConfig() if we add such a\n    // getter for partitionConfig().\n    debugInfo.setPartitionId(partitionConfig.getIndexingHashPartitionID());\n    debugInfo.setTiername(partitionConfig.getTierName());\n    debugInfo.setClusterName(partitionConfig.getClusterName());\n\n    for (ThriftSearchResult result : resultArray) {\n      result.setDebugInfo(debugInfo);\n    }\n  }\n\n  /**\n   * Write results into the result array.\n   * @param resultArray the result array to write into.\n   * @param hits the hits from the search.\n   * @param partitionConfig partition config used to fill in debug info. Pass in null if no debug\n   * info should be written into results.\n   */\n  public static void prepareResultsArray(List<ThriftSearchResult> resultArray,\n                                         SimpleSearchResults hits,\n                                         @Nullable PartitionConfig partitionConfig) {\n    for (int i = 0; i < hits.numHits(); i++) {\n      final Hit hit = hits.getHit(i);\n      final long id = hit.getStatusID();\n      final ThriftSearchResult result = new ThriftSearchResult(id);\n      final ThriftSearchResultMetadata resultMetadata = hit.getMetadata();\n      result.setMetadata(resultMetadata);\n      resultArray.add(result);\n    }\n    addDebugInfoToResults(resultArray, partitionConfig);\n  }\n\n  /**\n   * Write results into the result array.\n   * @param resultArray the result array to write into.\n   * @param hits the hits from the search.\n   * @param userIDWhitelist Used to set flag ThriftSearchResultMetadata.dontFilterUser.\n   * @param partitionConfig partition config used to fill in debug info. Pass in null if no debug\n   * info should be written into results.\n   */\n  public static void prepareRelevanceResultsArray(List<ThriftSearchResult> resultArray,\n                                                  RelevanceSearchResults hits,\n                                                  Set<Long> userIDWhitelist,\n                                                  @Nullable PartitionConfig partitionConfig) {\n    for (int i = 0; i < hits.numHits(); i++) {\n      final long id = hits.getHit(i).getStatusID();\n      final ThriftSearchResult result = new ThriftSearchResult(id);\n      final ThriftSearchResultMetadata resultMetadata = hits.resultMetadata[i];\n      result.setMetadata(resultMetadata);\n      if (userIDWhitelist != null) {\n        resultMetadata.setDontFilterUser(userIDWhitelist.contains(resultMetadata.getFromUserId()));\n      }\n\n      resultArray.add(result);\n    }\n    addDebugInfoToResults(resultArray, partitionConfig);\n  }\n\n  /**\n   * Merge a List of ThriftSearchResults into a single ThriftSearchResults object.\n   */\n  public static ThriftSearchResults mergeSearchResults(List<ThriftSearchResults> allSearchResults) {\n    ThriftSearchResults mergedResults = new ThriftSearchResults();\n    mergedResults.setRelevanceStats(new ThriftSearchResultsRelevanceStats());\n\n    mergedResults.setHitCounts(ResultsUtil.aggregateCountMap(allSearchResults,\n        ThriftSearchResultUtil.HIT_COUNTS_MAP_GETTER));\n\n    mergedResults.setLanguageHistogram(ResultsUtil.aggregateCountMap(allSearchResults,\n        ThriftSearchResultUtil.LANG_MAP_GETTER));\n\n    for (ThriftSearchResults searchResults : allSearchResults) {\n      // Add results\n      mergedResults.getResults().addAll(searchResults.getResults());\n      // Update counts\n      ThriftSearchResultUtil.incrementCounts(mergedResults, searchResults);\n      // Update relevance stats\n      if (searchResults.getRelevanceStats() != null) {\n        ThriftSearchResultsRelevanceStatsUtil.addRelevanceStats(mergedResults.getRelevanceStats(),\n            searchResults.getRelevanceStats());\n      }\n    }\n\n    return mergedResults;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/FieldTermCounter.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.util.Calendar;\nimport java.util.Collections;\nimport java.util.Map;\nimport java.util.TimeZone;\nimport java.util.concurrent.atomic.AtomicInteger;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Maps;\n\nimport org.apache.commons.lang.mutable.MutableInt;\nimport org.apache.commons.lang.mutable.MutableLong;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchLongGauge;\n\n/**\n * This class is used to count how many times a field happens in hourly and daily stats.\n * It is used by TermCountMonitor for iterating all fields in the index.\n *\n * There is one exception that this class is also used to count the number of tweets in the index.\n * Under the situation, the passed in fieldName would be empty string (as TWEET_COUNT_KEY).\n */\npublic class FieldTermCounter {\n  private static final Logger LOG = LoggerFactory.getLogger(FieldTermCounter.class);\n\n  static final TimeZone TIME_ZONE = TimeZone.getTimeZone(\"GMT\");\n  static final String TWEET_COUNT_KEY = \"\";\n\n  private final String fieldName;\n  private final int instanceCounter;\n\n  // The first date in format \"YYYYMMDDHH\" that we want to check counts for.\n  private final int startCheckHour;\n  // The last date in format \"YYYYMMDDHH\" that we want to check counts for.\n  private final int endCheckHour;\n  // Smallest number of docs we expect to have for each hour.\n  private final int hourlyMinCount;\n  //Smallest number of docs we expect to have for each day.\n  private final int dailyMinCount;\n\n  // Count of tweets for each day, keyed of by the hour in the format \"YYYYMMDD\".\n  private final Map<Integer, AtomicInteger> exportedHourlyCounts;\n\n  // Count of tweets for each day, keyed of by the day in the format \"YYYYMMDD\".\n  private final Map<Integer, MutableLong> dailyCounts;\n\n  // Only export hourly stats that are below minimum threshold.\n  private final Map<String, SearchLongGauge> exportedStats;\n\n  private final SearchLongGauge hoursWithNoTweetsStat;\n  private final SearchLongGauge daysWithNoTweetsStat;\n\n  public FieldTermCounter(\n      String fieldName,\n      int instanceCounter,\n      int startCheckHour,\n      int endCheckHour,\n      int hourlyMinCount,\n      int dailyMinCount) {\n    this.fieldName = fieldName;\n    this.instanceCounter = instanceCounter;\n    this.startCheckHour = startCheckHour;\n    this.endCheckHour = endCheckHour;\n    this.hourlyMinCount = hourlyMinCount;\n    this.dailyMinCount = dailyMinCount;\n    this.exportedHourlyCounts = Maps.newHashMap();\n    this.dailyCounts = Maps.newHashMap();\n    this.exportedStats = Maps.newHashMap();\n\n    this.hoursWithNoTweetsStat = SearchLongGauge.export(getAggregatedNoTweetStatName(true));\n    this.daysWithNoTweetsStat = SearchLongGauge.export(getAggregatedNoTweetStatName(false));\n  }\n\n  /**\n   * Updates the stats exported by this class based on the new counts provided in the given map.\n   */\n  public void runWithNewCounts(Map<Integer, MutableInt> newCounts) {\n    dailyCounts.clear();\n\n    // See go/rb/813442/#comment2566569\n    // 1. Update all existing hours\n    updateExistingHourlyCounts(newCounts);\n\n    // 2. Add and export all new hours\n    addAndExportNewHourlyCounts(newCounts);\n\n    // 3. fill in all the missing hours between know min and max days.\n    fillMissingHourlyCounts();\n\n    // 4. Export as a stat, how many hours don't have any tweets (i.e. <= 0)\n    exportMissingTweetStats();\n  }\n\n  // Input:\n  // . the new hourly count map in the current iteration\n  // . the existing hourly count map before the current iteration\n  // If the hourly key matches from the new hourly map to the existing hourly count map, update\n  // the value of the existing hourly count map to the value from the new hourly count map.\n  private void updateExistingHourlyCounts(Map<Integer, MutableInt> newCounts) {\n    for (Map.Entry<Integer, AtomicInteger> exportedCount : exportedHourlyCounts.entrySet()) {\n      Integer date = exportedCount.getKey();\n      AtomicInteger exportedCountValue = exportedCount.getValue();\n\n      MutableInt newCount = newCounts.get(date);\n      if (newCount == null) {\n        exportedCountValue.set(0);\n      } else {\n        exportedCountValue.set(newCount.intValue());\n        // clean up so that we don't check this date again when we look for new hours\n        newCounts.remove(date);\n      }\n    }\n  }\n\n  // Input:\n  // . the new hourly count map in the current iteration\n  // . the existing hourly count map before the current iteration\n  // This function is called after the above function of updateExistingHourlyCounts() so that all\n  // matching key value pairs have been removed from the new hourly count map.\n  // Move all remaining valid values from the new hourly count map to the existing hourly count\n  // map.\n  private void addAndExportNewHourlyCounts(Map<Integer, MutableInt> newCounts) {\n    for (Map.Entry<Integer, MutableInt> newCount : newCounts.entrySet()) {\n      Integer hour = newCount.getKey();\n      MutableInt newCountValue = newCount.getValue();\n      Preconditions.checkState(!exportedHourlyCounts.containsKey(hour),\n          \"Should have already processed and removed existing hours: \" + hour);\n\n      AtomicInteger newStat = new AtomicInteger(newCountValue.intValue());\n      exportedHourlyCounts.put(hour, newStat);\n    }\n  }\n\n  // Find whether the existing hourly count map has hourly holes.  If such holes exist, fill 0\n  // values so that they can be exported.\n  private void fillMissingHourlyCounts() {\n    // Figure out the time range for which we should have tweets in the index. At the very least,\n    // this range should cover [startCheckHour, endCheckHour) if endCheckHour is set, or\n    // [startCheckHour, latestHourInTheIndexWithTweets] if endCheckHour is not set (latest tier or\n    // realtime cluster).\n    int startHour = startCheckHour;\n    int endHour = endCheckHour < getHourValue(Calendar.getInstance(TIME_ZONE)) ? endCheckHour : -1;\n    for (int next : exportedHourlyCounts.keySet()) {\n      if (next < startHour) {\n        startHour = next;\n      }\n      if (next > endHour) {\n        endHour = next;\n      }\n    }\n\n    Calendar endHourCal = getCalendarValue(endHour);\n    Calendar hour = getCalendarValue(startHour);\n    for (; hour.before(endHourCal); hour.add(Calendar.HOUR_OF_DAY, 1)) {\n      int hourValue = getHourValue(hour);\n      if (!exportedHourlyCounts.containsKey(hourValue)) {\n        exportedHourlyCounts.put(hourValue, new AtomicInteger(0));\n      }\n    }\n  }\n\n  private void exportMissingTweetStats() {\n    int hoursWithNoTweets = 0;\n    int daysWithNoTweets = 0;\n\n    for (Map.Entry<Integer, AtomicInteger> hourlyCount : exportedHourlyCounts.entrySet()) {\n      int hour = hourlyCount.getKey();\n      if ((hour < startCheckHour) || (hour >= endCheckHour)) {\n        continue;\n      }\n\n      // roll up the days\n      int day = hour / 100;\n      MutableLong dayCount = dailyCounts.get(day);\n      if (dayCount == null) {\n        dailyCounts.put(day, new MutableLong(hourlyCount.getValue().get()));\n      } else {\n        dayCount.setValue(dayCount.longValue() + hourlyCount.getValue().get());\n      }\n      AtomicInteger exportedCountValue = hourlyCount.getValue();\n      if (exportedCountValue.get() <= hourlyMinCount) {\n        // We do not export hourly too few tweets for index fields as it can 10x the existing\n        // exported stats.\n        // We might consider whitelisting some high frequency fields later.\n        if (isFieldForTweet()) {\n          String statsName = getStatName(hourlyCount.getKey());\n          SearchLongGauge stat = SearchLongGauge.export(statsName);\n          stat.set(exportedCountValue.longValue());\n          exportedStats.put(statsName, stat);\n        }\n        LOG.warn(\"Found an hour with too few tweets. Field: <{}> Hour: {} count: {}\",\n            fieldName, hour, exportedCountValue);\n        hoursWithNoTweets++;\n      }\n    }\n\n    for (Map.Entry<Integer, MutableLong> dailyCount : dailyCounts.entrySet()) {\n      if (dailyCount.getValue().longValue() <= dailyMinCount) {\n        LOG.warn(\"Found a day with too few tweets. Field: <{}> Day: {} count: {}\",\n            fieldName, dailyCount.getKey(), dailyCount.getValue());\n        daysWithNoTweets++;\n      }\n    }\n\n    hoursWithNoTweetsStat.set(hoursWithNoTweets);\n    daysWithNoTweetsStat.set(daysWithNoTweets);\n  }\n\n  // When the fieldName is empty string (as TWEET_COUNT_KEY), it means that we are counting the\n  // number of tweets for the index, not for some specific fields.\n  private boolean isFieldForTweet() {\n    return TWEET_COUNT_KEY.equals(fieldName);\n  }\n\n  private String getAggregatedNoTweetStatName(boolean hourly) {\n    if (isFieldForTweet()) {\n      if (hourly) {\n        return \"hours_with_no_indexed_tweets_v_\" + instanceCounter;\n      } else {\n        return \"days_with_no_indexed_tweets_v_\" + instanceCounter;\n      }\n    } else {\n      if (hourly) {\n        return \"hours_with_no_indexed_fields_v_\" + fieldName + \"_\" + instanceCounter;\n      } else {\n        return \"days_with_no_indexed_fields_v_\" + fieldName + \"_\" + instanceCounter;\n      }\n    }\n  }\n\n  @VisibleForTesting\n  String getStatName(Integer date) {\n    return getStatName(fieldName, instanceCounter, date);\n  }\n\n  @VisibleForTesting\n  static String getStatName(String field, int instance, Integer date) {\n    if (TWEET_COUNT_KEY.equals(field)) {\n      return \"tweets_indexed_on_hour_v_\" + instance + \"_\" + date;\n    } else {\n      return \"tweets_indexed_on_hour_v_\" + instance + \"_\" + field + \"_\" + date;\n    }\n  }\n\n  @VisibleForTesting\n  Map<Integer, AtomicInteger> getExportedCounts() {\n    return Collections.unmodifiableMap(exportedHourlyCounts);\n  }\n\n  @VisibleForTesting\n  Map<Integer, MutableLong> getDailyCounts() {\n    return Collections.unmodifiableMap(dailyCounts);\n  }\n\n  @VisibleForTesting\n  long getHoursWithNoTweets() {\n    return hoursWithNoTweetsStat.get();\n  }\n\n  @VisibleForTesting\n  long getDaysWithNoTweets() {\n    return daysWithNoTweetsStat.get();\n  }\n\n  @VisibleForTesting\n  Map<String, SearchLongGauge> getExportedHourlyCountStats() {\n    return exportedStats;\n  }\n\n  /**\n   * Given a unit time in seconds since epoch UTC, will return the day in format \"YYYYMMDDHH\"\n   * as an int.\n   */\n  @VisibleForTesting\n  static int getHourValue(Calendar cal, int timeSecs) {\n    cal.setTimeInMillis(timeSecs * 1000L);\n    return getHourValue(cal);\n  }\n\n  static int getHourValue(Calendar cal) {\n    int year = cal.get(Calendar.YEAR) * 1000000;\n    int month = (cal.get(Calendar.MONTH) + 1) * 10000; // month is 0-based\n    int day = cal.get(Calendar.DAY_OF_MONTH) * 100;\n    int hour = cal.get(Calendar.HOUR_OF_DAY);\n    return year + month + day + hour;\n  }\n\n  @VisibleForTesting\n  static Calendar getCalendarValue(int hour) {\n    Calendar cal = Calendar.getInstance(TIME_ZONE);\n\n    int year = hour / 1000000;\n    int month = ((hour / 10000) % 100) - 1; // 0-based\n    int day = (hour / 100) % 100;\n    int hr = hour % 100;\n    cal.setTimeInMillis(0);  // reset all time fields\n    cal.set(year, month, day, hr, 0);\n    return cal;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/Histogram.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.List;\n\nimport com.google.common.base.Preconditions;\n\n/**\n * A histogram of int values with arbitrary buckets.\n * Keeps a count for each bucket, and a sum of values for each bucket.\n * The histogram view is returned as a list of {@link Histogram.Entry}s.\n * <p/>\n * Bucket boundaries are inclusive on the upper boundaries. Given buckets of [0, 10, 100],\n * items will be places in 4 bins, { X <= 0, 0 < X <= 10, 10 < X <= 100, X > 100 }.\n * <p/>\n * This class is not thread safe.\n *\n */\npublic class Histogram {\n  private final double[] buckets;\n  private final int[] itemsCount;\n  private final long[] itemsSum;\n  private int totalCount;\n  private long totalSum;\n\n  public static class Entry {\n    private final String bucketName;\n    private final int count;\n    private final double countPercent;\n    private final double countCumulative;\n    private final long sum;\n    private final double sumPercent;\n    private final double sumCumulative;\n\n    Entry(String bucketName,\n          int count, double countPercent, double countCumulative,\n          long sum, double sumPercent, double sumCumulative) {\n      this.bucketName = bucketName;\n      this.count = count;\n      this.countPercent = countPercent;\n      this.countCumulative = countCumulative;\n      this.sum = sum;\n      this.sumPercent = sumPercent;\n      this.sumCumulative = sumCumulative;\n    }\n\n    public String getBucketName() {\n      return bucketName;\n    }\n\n    public int getCount() {\n      return count;\n    }\n\n    public double getCountPercent() {\n      return countPercent;\n    }\n\n    public double getCountCumulative() {\n      return countCumulative;\n    }\n\n    public long getSum() {\n      return sum;\n    }\n\n    public double getSumPercent() {\n      return sumPercent;\n    }\n\n    public double getSumCumulative() {\n      return sumCumulative;\n    }\n  }\n\n  /**\n   * No buckets will put all items into a single bin.\n   * @param buckets the buckets to use for binnning data.\n   *       An item will be put in bin i if item <= buckets[i] and > buckets[i-1]\n   *       The bucket values must be strictly increasing.\n   */\n  public Histogram(double... buckets) {\n    Preconditions.checkNotNull(buckets);\n    this.buckets = new double[buckets.length];\n    for (int i = 0; i < buckets.length; i++) {\n      this.buckets[i] = buckets[i];\n      if (i > 0) {\n        Preconditions.checkState(this.buckets[i - 1] < this.buckets[i],\n               \"Histogram buckets must me strictly increasing: \" + Arrays.toString(buckets));\n      }\n    }\n    this.itemsCount = new int[buckets.length + 1];\n    this.itemsSum = new long[buckets.length + 1];\n    this.totalCount = 0;\n    this.totalSum = 0;\n  }\n\n  /**\n   * Add the given item to the appropriate bucket.\n   */\n  public void addItem(double item) {\n    int i = 0;\n    for (; i < this.buckets.length; i++) {\n      if (item <= buckets[i]) {\n        break;\n      }\n    }\n    this.itemsCount[i]++;\n    this.totalCount++;\n    this.itemsSum[i] += item;\n    this.totalSum += item;\n  }\n\n  /**\n   * returns the current view of all the bins.\n   */\n  public List<Entry> entries() {\n    List<Entry> entries = new ArrayList<>(this.itemsCount.length);\n    double countCumulative = 0;\n    double sumCumulative = 0;\n    for (int i = 0; i < this.itemsCount.length; i++) {\n      String bucketName;\n      if (i < this.buckets.length) {\n        bucketName = \"<= \" + this.buckets[i];\n      } else if (this.buckets.length > 0) {\n        bucketName = \" > \" + this.buckets[this.buckets.length - 1];\n      } else {\n        bucketName = \" * \";\n      }\n\n      int count = this.itemsCount[i];\n      double countPercent = this.totalCount == 0 ? 0 : ((double) this.itemsCount[i]) / totalCount;\n      countCumulative += countPercent;\n\n      long sum = this.itemsSum[i];\n      double sumPercent = this.totalSum == 0 ? 0 : ((double) this.itemsSum[i]) / totalSum;\n      sumCumulative += sumPercent;\n\n      Entry e = new Entry(bucketName, count, countPercent, countCumulative,\n                          sum, sumPercent, sumCumulative);\n      entries.add(e);\n    }\n    return entries;\n  }\n\n  /**\n   * Returns total number of items seen.\n   */\n  public int getTotalCount() {\n    return totalCount;\n  }\n\n  /**\n   * Returns sum of all the items seen.\n   */\n  public long getTotalSum() {\n    return totalSum;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/IndexViewer.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.io.IOException;\nimport java.io.PrintWriter;\nimport java.io.UnsupportedEncodingException;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.Comparator;\nimport java.util.List;\nimport java.util.Locale;\nimport java.util.Set;\nimport java.util.TreeSet;\n\nimport com.google.common.collect.ImmutableSet;\nimport com.google.common.collect.Lists;\n\nimport org.apache.lucene.index.IndexOptions;\nimport org.apache.lucene.index.NumericDocValues;\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.apache.lucene.util.BytesRef;\n\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.common.schema.earlybird.EarlybirdFieldConstants.EarlybirdFieldConstant;\nimport com.twitter.search.common.schema.thriftjava.ThriftCSFType;\nimport com.twitter.search.common.util.analysis.IntTermAttributeImpl;\nimport com.twitter.search.common.util.analysis.LongTermAttributeImpl;\nimport com.twitter.search.common.util.analysis.SortableLongTermAttributeImpl;\nimport com.twitter.search.common.util.spatial.GeoUtil;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.inverted.MPHTermDictionary;\nimport com.twitter.search.core.earlybird.index.inverted.RealtimeIndexTerms;\nimport com.twitter.search.earlybird.index.EarlybirdSingleSegmentSearcher;\n\nimport geo.google.datamodel.GeoCoordinate;\n\npublic class IndexViewer {\n  /**\n   * Fields whose terms are indexed using\n   * {@link com.twitter.search.common.util.analysis.IntTermAttribute}\n   */\n  private static final Set<String> INT_TERM_ATTRIBUTE_FIELDS = ImmutableSet.of(\n      EarlybirdFieldConstant.CREATED_AT_FIELD.getFieldName(),\n      EarlybirdFieldConstant.LINK_CATEGORY_FIELD.getFieldName(),\n      EarlybirdFieldConstant\n          .NORMALIZED_FAVORITE_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD.getFieldName(),\n      EarlybirdFieldConstant\n          .NORMALIZED_REPLY_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD.getFieldName(),\n      EarlybirdFieldConstant\n          .NORMALIZED_RETWEET_COUNT_GREATER_THAN_OR_EQUAL_TO_FIELD.getFieldName(),\n      EarlybirdFieldConstant.COMPOSER_SOURCE.getFieldName());\n\n  /**\n   * Fields whose terms are indexed using\n   * {@link com.twitter.search.common.util.analysis.LongTermAttribute}\n   */\n  private static final Set<String> LONG_TERM_ATTRIBUTE_FIELDS = ImmutableSet.of(\n      EarlybirdFieldConstant.CONVERSATION_ID_FIELD.getFieldName(),\n      EarlybirdFieldConstant.LIKED_BY_USER_ID_FIELD.getFieldName(),\n      EarlybirdFieldConstant.QUOTED_TWEET_ID_FIELD.getFieldName(),\n      EarlybirdFieldConstant.QUOTED_USER_ID_FIELD.getFieldName(),\n      EarlybirdFieldConstant.REPLIED_TO_BY_USER_ID.getFieldName(),\n      EarlybirdFieldConstant.RETWEETED_BY_USER_ID.getFieldName(),\n      EarlybirdFieldConstant.DIRECTED_AT_USER_ID_FIELD.getFieldName(),\n      EarlybirdFieldConstant.FROM_USER_ID_FIELD.getFieldName(),\n      EarlybirdFieldConstant.IN_REPLY_TO_TWEET_ID_FIELD.getFieldName(),\n      EarlybirdFieldConstant.IN_REPLY_TO_USER_ID_FIELD.getFieldName(),\n      EarlybirdFieldConstant.RETWEET_SOURCE_TWEET_ID_FIELD.getFieldName(),\n      EarlybirdFieldConstant.RETWEET_SOURCE_USER_ID_FIELD.getFieldName());\n\n  /**\n   * Fields whose terms index using SORTED\n   * {@link com.twitter.search.common.util.analysis.LongTermAttribute}\n   */\n  private static final Set<String> SORTED_LONG_TERM_ATTRIBUTE_FIELDS =\n      ImmutableSet.of(EarlybirdFieldConstant.ID_FIELD.getFieldName());\n\n  private final EarlybirdSingleSegmentSearcher searcher;\n  private final EarlybirdIndexSegmentAtomicReader twitterReader;\n\n  public long getTimeSliceId() {\n    return searcher.getTimeSliceID();\n  }\n\n  public static class Options {\n    private boolean dumpHexTerms = false;\n    private String charset;\n    private double[] histogramBuckets;\n    private boolean termLengthHistogram;\n\n    public Options setDumpHexTerms(boolean dumpHexTermsParam) {\n      this.dumpHexTerms = dumpHexTermsParam;\n      return this;\n    }\n\n    public Options setCharset(String charsetParam) {\n      this.charset = charsetParam;\n      return this;\n    }\n\n    public Options setHistogramBuckets(double[] histogramBucketsParam) {\n      this.histogramBuckets = histogramBucketsParam;\n      return this;\n    }\n\n    public Options setTermLengthHistogram(boolean termLengthHistogramParam) {\n      this.termLengthHistogram = termLengthHistogramParam;\n      return this;\n    }\n  }\n\n  /**\n   * Data Transfer Object for Terms, encapsulates the \"json\" serialization\n   * while maintaining streaming mode\n   */\n  private static class TermDto {\n\n    private final String field;\n    private final String term;\n    private final String docFreq;\n    private final String percent;\n    private final PostingsEnum docsEnum;\n    private final TermsEnum termsEnum;\n    private final Integer maxDocs;\n\n    public TermDto(String field, String term, String docFreq, String percent,\n                   PostingsEnum docsEnum, TermsEnum termsEnum, Integer maxDocs) {\n      this.field = field;\n      this.term = term;\n      this.docFreq = docFreq;\n      this.percent = percent;\n      this.docsEnum = docsEnum;\n      this.termsEnum = termsEnum;\n      this.maxDocs = maxDocs;\n    }\n\n    public void write(ViewerWriter writer,\n                      EarlybirdIndexSegmentAtomicReader twitterReader) throws IOException {\n      writer.beginObject();\n      writer.name(\"field\").value(field);\n      writer.name(\"term\").value(term);\n      writer.name(\"docFreq\").value(docFreq);\n      writer.name(\"percent\").value(percent);\n      if (docsEnum != null) {\n        appendFrequencyAndPositions(writer, field, docsEnum, twitterReader);\n      }\n      if (maxDocs != null) {\n        appendDocs(writer, termsEnum, maxDocs, twitterReader);\n      }\n      writer.endObject();\n    }\n  }\n\n  /**\n   * Data Transfer Object for Terms, encapsulates the \"json\" serialization\n   * while maintaining streaming mode\n   */\n  private static class StatsDto {\n\n    private final String field;\n    private final String numTerms;\n    private final String terms;\n\n\n    public StatsDto(String field, String numTerms, String terms) {\n      this.field = field;\n      this.numTerms = numTerms;\n      this.terms = terms;\n    }\n\n    public void write(ViewerWriter writer) throws IOException {\n      writer.beginObject();\n\n      writer.name(\"field\").value(field);\n      writer.name(\"numTerms\").value(numTerms);\n      writer.name(\"terms\").value(terms);\n\n      writer.endObject();\n    }\n  }\n\n  public IndexViewer(EarlybirdSingleSegmentSearcher searcher) {\n    this.searcher = searcher;\n    this.twitterReader = searcher.getTwitterIndexReader();\n  }\n\n  private boolean shouldSeekExact(Terms terms, TermsEnum termsEnum) {\n    return terms instanceof RealtimeIndexTerms\n           || termsEnum instanceof MPHTermDictionary.MPHTermsEnum;\n  }\n\n  /**\n   * Dumps all terms for a given tweet id.\n   * @param writer writer being used\n   * @param tweetId the tweet id to use\n   */\n  public void dumpTweetDataByTweetId(ViewerWriter writer, long tweetId, Options options)\n      throws IOException {\n    int docId = twitterReader.getSegmentData().getDocIDToTweetIDMapper().getDocID(tweetId);\n    dumpTweetDataByDocId(writer, docId, options);\n  }\n\n  /**\n   * Dumps all terms for a given doc id.\n   * @param writer writer being used\n   * @param docId the document id to use.\n   */\n  public void dumpTweetDataByDocId(ViewerWriter writer, int docId, Options options)\n      throws IOException {\n    writer.beginObject();\n\n    printHeader(writer);\n    long tweetID = twitterReader.getSegmentData().getDocIDToTweetIDMapper().getTweetID(docId);\n    if (docId < twitterReader.maxDoc() && tweetID >= 0) {\n      writer.name(\"docId\").value(Integer.toString(docId));\n      writer.name(\"tweetId\").value(Long.toString(tweetID));\n      dumpIndexedFields(writer, docId, options);\n      dumpCsfFields(writer, docId);\n    }\n    writer.endObject();\n  }\n\n  /**\n   * Dumps all tweet IDs in the current segment to the given file.\n   */\n  public void dumpTweetIds(ViewerWriter writer, String logFile, PrintWriter logWriter)\n      throws IOException {\n    writeTweetIdsToLogFile(logWriter);\n\n    writer.beginObject();\n    writer.name(Long.toString(searcher.getTimeSliceID())).value(logFile);\n    writer.endObject();\n  }\n\n  private void writeTweetIdsToLogFile(PrintWriter logWriter) {\n    DocIDToTweetIDMapper mapper = twitterReader.getSegmentData().getDocIDToTweetIDMapper();\n    int docId = Integer.MIN_VALUE;\n    while ((docId = mapper.getNextDocID(docId)) != DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      long tweetId = mapper.getTweetID(docId);\n\n      // Ensure tweet ID is valid and non-deleted\n      if ((tweetId > 0) && !twitterReader.getDeletesView().isDeleted(docId)) {\n        logWriter.println(tweetId);\n      }\n    }\n  }\n\n  private void dumpIndexedFields(ViewerWriter writer, int docId,\n                                 Options options) throws IOException {\n    writer.name(\"indexedFields\");\n    writer.beginArray();\n    writer.newline();\n    for (String field : sortedFields()) {\n      dumpTweetData(writer, field, docId, options);\n    }\n    writer.endArray();\n    writer.newline();\n  }\n\n  private void dumpCsfFields(ViewerWriter writer, int docId) throws IOException {\n    writer.name(\"csfFields\");\n    writer.beginArray();\n    writer.newline();\n    dumpCSFData(writer, docId);\n\n    writer.endArray();\n  }\n\n  /**\n   * Dumps all CSF values for a given doc id.\n   * @param writer writer being used\n   * @param docId the document id to use.\n   */\n  private void dumpCSFData(ViewerWriter writer, int docId) throws IOException {\n    Schema tweetSchema = twitterReader.getSchema();\n\n    // Sort the FieldInfo objects to generate fixed order to make testing easier\n    List<Schema.FieldInfo> sortedFieldInfos = new ArrayList<>(tweetSchema.getFieldInfos());\n    sortedFieldInfos.sort(Comparator.comparing(Schema.FieldInfo::getFieldId));\n\n    for (Schema.FieldInfo fieldInfo: sortedFieldInfos) {\n      String csfFieldInfoName = fieldInfo.getName();\n      ThriftCSFType csfType = tweetSchema.getCSFFieldType(csfFieldInfoName);\n      NumericDocValues csfDocValues = twitterReader.getNumericDocValues(csfFieldInfoName);\n      // If twitterReader.getNumericDocValues(value.getName()) == null,\n      // means no NumericDocValue was indexed for the field so ignore\n      if (csfType != null && csfDocValues != null && csfDocValues.advanceExact(docId)) {\n        long csfValue = csfDocValues.longValue();\n        writer.beginObject();\n        writer.name(\"field\").value(formatField(csfFieldInfoName));\n        writer.name(\"value\");\n        if (csfFieldInfoName.equals(EarlybirdFieldConstant.LAT_LON_CSF_FIELD.getFieldName())) {\n          writer.value(latlongDecode(csfValue));\n        } else if (csfFieldInfoName.equals(EarlybirdFieldConstant.LANGUAGE.getFieldName())) {\n          writer.value(languageDecode(csfValue));\n        } else if (csfFieldInfoName.equals(EarlybirdFieldConstant.CARD_LANG_CSF.getFieldName())) {\n          writer.value(languageDecode(csfValue));\n        } else {\n          writer.value(Long.toString(csfValue));\n        }\n        writer.endObject();\n        writer.newline();\n      }\n    }\n  }\n\n  /**\n   * Decipher long value gotten, put into format (lat, lon)\n   * Decode the stored long value by creating a geocode\n   */\n  private String latlongDecode(long csfValue) {\n    StringBuilder sb = new StringBuilder();\n    GeoCoordinate geoCoordinate = new GeoCoordinate();\n    if (GeoUtil.decodeLatLonFromInt64(csfValue, geoCoordinate)) {\n      sb.append(geoCoordinate.getLatitude()).append(\", \").append(geoCoordinate.getLongitude());\n    } else {\n      sb.append(csfValue).append(\" (Value Unset or Invalid Coordinate)\");\n    }\n    return sb.toString();\n  }\n\n  /**\n   * Decipher long value gotten into string of tweet's language\n   */\n  private String languageDecode(long csfValue) {\n    StringBuilder sb = new StringBuilder();\n    ThriftLanguage languageType = ThriftLanguage.findByValue((int) csfValue);\n    sb.append(csfValue).append(\" (\").append(languageType).append(\")\");\n    return sb.toString();\n  }\n\n  private void dumpTweetData(ViewerWriter writer,\n                             String field,\n                             int docId,\n                             Options options) throws IOException {\n\n    Terms terms = twitterReader.terms(field);\n    if (terms != null) {\n      TermsEnum termsEnum = terms.iterator();\n      if (shouldSeekExact(terms, termsEnum)) {\n        long numTerms = terms.size();\n        for (int i = 0; i < numTerms; i++) {\n          termsEnum.seekExact(i);\n          dumpTweetDataTerm(writer, field, termsEnum, docId, options);\n        }\n      } else {\n        while (termsEnum.next() != null) {\n          dumpTweetDataTerm(writer, field, termsEnum, docId, options);\n        }\n      }\n    }\n  }\n\n  private void dumpTweetDataTerm(ViewerWriter writer, String field, TermsEnum termsEnum,\n                                 int docId, Options options) throws IOException {\n    PostingsEnum docsAndPositionsEnum = termsEnum.postings(null, PostingsEnum.ALL);\n    if (docsAndPositionsEnum != null && docsAndPositionsEnum.advance(docId) == docId) {\n      printTerm(writer, field, termsEnum, docsAndPositionsEnum, null, options);\n    }\n  }\n\n  /**\n   * Prints the histogram for the currently viewed index.\n   * @param writer current viewerWriter\n   * @param field if null, will use all fields\n   * @param options options for dumping out text\n   */\n  public void dumpHistogram(ViewerWriter writer, String field, Options options) throws IOException {\n    writer.beginObject();\n    printHeader(writer);\n    writer.name(\"histogram\");\n    writer.beginArray();\n    writer.newline();\n    if (field == null) {\n      for (String field2 : sortedFields()) {\n        dumpFieldHistogram(writer, field2, options);\n      }\n    } else {\n      dumpFieldHistogram(writer, field, options);\n    }\n    writer.endArray();\n    writer.endObject();\n  }\n\n  private void dumpFieldHistogram(ViewerWriter writer, String field, Options options)\n      throws IOException {\n    Histogram histo = new Histogram(options.histogramBuckets);\n\n    Terms terms = twitterReader.terms(field);\n    if (terms != null) {\n      TermsEnum termsEnum = terms.iterator();\n      if (shouldSeekExact(terms, termsEnum)) {\n        long numTerms = terms.size();\n        for (int i = 0; i < numTerms; i++) {\n          termsEnum.seekExact(i);\n          countHistogram(options, histo, termsEnum);\n        }\n      } else {\n        while (termsEnum.next() != null) {\n          countHistogram(options, histo, termsEnum);\n        }\n      }\n      printHistogram(writer, field, options, histo);\n    }\n  }\n\n  private void printHistogram(ViewerWriter writer, String field, Options options,\n                              Histogram histo) throws IOException {\n\n    String bucket = options.termLengthHistogram ? \"termLength\" : \"df\";\n    for (Histogram.Entry histEntry : histo.entries()) {\n      String format =\n          String.format(Locale.US,\n              \"field: %s %sBucket: %11s count: %10d \"\n                  + \"percent: %6.2f%% cumulative: %6.2f%% totalCount: %10d\"\n                  + \" sum: %15d percent: %6.2f%% cumulative: %6.2f%% totalSum: %15d\",\n              formatField(field),\n              bucket,\n              histEntry.getBucketName(),\n              histEntry.getCount(),\n              histEntry.getCountPercent() * 100.0,\n              histEntry.getCountCumulative() * 100.0,\n              histo.getTotalCount(),\n              histEntry.getSum(),\n              histEntry.getSumPercent() * 100.0,\n              histEntry.getSumCumulative() * 100.0,\n              histo.getTotalSum()\n          );\n      writer.value(format);\n      writer.newline();\n    }\n  }\n\n  private void countHistogram(Options options, Histogram histo, TermsEnum termsEnum)\n          throws IOException {\n    if (options.termLengthHistogram) {\n      final BytesRef bytesRef = termsEnum.term();\n      histo.addItem(bytesRef.length);\n    } else {\n      histo.addItem(termsEnum.docFreq());\n    }\n  }\n\n\n  /**\n   * Prints terms and optionally documents for the currently viewed index.\n   * @param writer writer being used\n   * @param field if null, will use all fields\n   * @param term if null will use all terms\n   * @param maxTerms will print at most this many terms per field. If null will print 0 terms.\n   * @param maxDocs will print at most this many documents, If null, will not print docs.\n   * @param options options for dumping out text\n   */\n  public void dumpData(ViewerWriter writer, String field, String term, Integer maxTerms,\n        Integer maxDocs, Options options, boolean shouldSeekToTerm) throws IOException {\n\n    writer.beginObject();\n    printHeader(writer);\n\n    writer.name(\"terms\");\n    writer.beginArray();\n    writer.newline();\n    dumpDataInternal(writer, field, term, maxTerms, maxDocs, options, shouldSeekToTerm);\n    writer.endArray();\n    writer.endObject();\n  }\n\n  private void dumpDataInternal(ViewerWriter writer, String field, String term, Integer maxTerms,\n      Integer maxDocs, Options options, boolean shouldSeekToTerm) throws IOException {\n\n    if (field == null) {\n      dumpDataForAllFields(writer, term, maxTerms, maxDocs, options);\n      return;\n    }\n    if (term == null) {\n      dumpDataForAllTerms(writer, field, maxTerms, maxDocs, options);\n      return;\n    }\n    Terms terms = twitterReader.terms(field);\n    if (terms != null) {\n      TermsEnum termsEnum = terms.iterator();\n      TermsEnum.SeekStatus status = termsEnum.seekCeil(new BytesRef(term));\n      if (status == TermsEnum.SeekStatus.FOUND) {\n        printTerm(writer, field, termsEnum, null, maxDocs, options);\n      }\n      if (shouldSeekToTerm) {\n        dumpTermsAfterSeek(writer, field, terms, maxTerms, maxDocs, options, termsEnum, status);\n      }\n    }\n  }\n\n  /**\n   * if term (cursor) is found for an indexed segment - dump the next termsLeft words\n   * starting from the current position in the enum.  For an indexed segment,\n   * seekCeil will place the enum at the word or the next \"ceiling\" term.  For\n   * a realtime index, if the word is not found we do not paginate anything\n   * We also only paginate if the TermsEnum is not at the end.\n   */\n  private void dumpTermsAfterSeek(ViewerWriter writer, String field, Terms terms, Integer maxTerms,\n      Integer maxDocs, Options options, TermsEnum termsEnum, TermsEnum.SeekStatus status)\n      throws IOException {\n    if (status != TermsEnum.SeekStatus.END) {\n      // for realtime, to not repeat the found word\n      if (shouldSeekExact(terms, termsEnum)) {\n        termsEnum.next();\n      }\n      if (status != TermsEnum.SeekStatus.FOUND) {\n        // if not found, print out curr term before calling next()\n        printTerm(writer, field, termsEnum, null, maxDocs, options);\n      }\n      for (int termsLeft = maxTerms - 1; termsLeft > 0 && termsEnum.next() != null; termsLeft--) {\n        printTerm(writer, field, termsEnum, null, maxDocs, options);\n      }\n    }\n  }\n\n  private void dumpDataForAllFields(ViewerWriter writer, String term, Integer maxTerms,\n                                    Integer maxDocs, Options options) throws IOException {\n    for (String field : sortedFields()) {\n      dumpDataInternal(writer, field, term, maxTerms, maxDocs, options, false);\n    }\n  }\n\n  private List<String> sortedFields() {\n    // Tweet facets are added to a special $facets field, which is not part of the schema.\n    // We include it here, because seeing the facets for a tweet is generally useful.\n    List<String> fields = Lists.newArrayList(\"$facets\");\n    for (Schema.FieldInfo fieldInfo : twitterReader.getSchema().getFieldInfos()) {\n      if (fieldInfo.getFieldType().indexOptions() != IndexOptions.NONE) {\n        fields.add(fieldInfo.getName());\n      }\n    }\n    Collections.sort(fields);\n    return fields;\n  }\n\n  private void dumpDataForAllTerms(ViewerWriter writer,\n                                   String field,\n                                   Integer maxTerms,\n                                   Integer maxDocs,\n                                   Options options) throws IOException {\n    Terms terms = twitterReader.terms(field);\n    if (terms != null) {\n      TermsEnum termsEnum = terms.iterator();\n      if (shouldSeekExact(terms, termsEnum)) {\n        long numTerms = terms.size();\n        long termToDump = maxTerms == null ? 0 : Math.min(numTerms, maxTerms);\n        for (int i = 0; i < termToDump; i++) {\n          termsEnum.seekExact(i);\n          printTerm(writer, field, termsEnum, null, maxDocs, options);\n        }\n      } else {\n        int max = maxTerms == null ? 0 : maxTerms;\n        while (max > 0 && termsEnum.next() != null) {\n          printTerm(writer, field, termsEnum, null, maxDocs, options);\n          max--;\n        }\n      }\n    }\n  }\n\n  private String termToString(String field, BytesRef bytesTerm, Options options)\n      throws UnsupportedEncodingException {\n    if (INT_TERM_ATTRIBUTE_FIELDS.contains(field)) {\n      return Integer.toString(IntTermAttributeImpl.copyBytesRefToInt(bytesTerm));\n    } else if (LONG_TERM_ATTRIBUTE_FIELDS.contains(field)) {\n      return Long.toString(LongTermAttributeImpl.copyBytesRefToLong(bytesTerm));\n    } else if (SORTED_LONG_TERM_ATTRIBUTE_FIELDS.contains(field)) {\n      return Long.toString(SortableLongTermAttributeImpl.copyBytesRefToLong(bytesTerm));\n    } else {\n      if (options != null && options.charset != null && !options.charset.isEmpty()) {\n        return new String(bytesTerm.bytes, bytesTerm.offset, bytesTerm.length, options.charset);\n      } else {\n        return bytesTerm.utf8ToString();\n      }\n    }\n  }\n\n  private void printTerm(ViewerWriter writer, String field, TermsEnum termsEnum,\n                         PostingsEnum docsEnum, Integer maxDocs, Options options)\n      throws IOException {\n    final BytesRef bytesRef = termsEnum.term();\n    StringBuilder termToString = new StringBuilder();\n    termToString.append(termToString(field, bytesRef, options));\n    if (options != null && options.dumpHexTerms) {\n      termToString.append(\" \").append(bytesRef.toString());\n    }\n    final int df = termsEnum.docFreq();\n    double dfPercent = ((double) df / this.twitterReader.numDocs()) * 100.0;\n    TermDto termDto = new TermDto(field, termToString.toString(), Integer.toString(df),\n                                   String.format(Locale.US, \"%.2f%%\", dfPercent),\n                                   docsEnum, termsEnum, maxDocs);\n    termDto.write(writer, twitterReader);\n    writer.newline();\n  }\n\n  private static void appendFrequencyAndPositions(ViewerWriter writer, String field,\n      PostingsEnum docsEnum, EarlybirdIndexSegmentAtomicReader twitterReader) throws IOException {\n    final int frequency = docsEnum.freq();\n    writer.name(\"freq\").value(Integer.toString(frequency));\n\n    Schema schema = twitterReader.getSchema();\n    Schema.FieldInfo fieldInfo = schema.getFieldInfo(field);\n\n    if (fieldInfo != null\n            && (fieldInfo.getFieldType().indexOptions() == IndexOptions.DOCS_AND_FREQS_AND_POSITIONS\n            || fieldInfo.getFieldType().indexOptions()\n                == IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS)) {\n      appendPositions(writer, docsEnum);\n    }\n  }\n\n  private static void appendPositions(ViewerWriter writer, PostingsEnum docsAndPositionsEnum)\n      throws IOException {\n    writer.name(\"positions\");\n\n    writer.beginArray();\n    final int frequency = docsAndPositionsEnum.freq();\n    for (int i = 0; i < frequency; i++) {\n      int position = docsAndPositionsEnum.nextPosition();\n      writer.value(Integer.toString(position));\n    }\n    writer.endArray();\n  }\n\n  private static void appendDocs(ViewerWriter writer, TermsEnum termsEnum, int maxDocs,\n                                 EarlybirdIndexSegmentAtomicReader twitterReader)\n      throws IOException {\n    writer.name(\"docIds\");\n\n    writer.beginArray();\n\n    PostingsEnum docs = termsEnum.postings(null, 0);\n    int docsReturned = 0;\n    int docId;\n    boolean endedEarly = false;\n    DocIDToTweetIDMapper mapper = twitterReader.getSegmentData().getDocIDToTweetIDMapper();\n    while ((docId = docs.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {\n      if (docsReturned < maxDocs) {\n        docsReturned++;\n        long tweetID = mapper.getTweetID(docId);\n\n        writer.beginObject();\n        writer.name(\"docId\").value(Long.toString(docId));\n        writer.name(\"tweetId\").value(Long.toString(tweetID));\n        writer.endObject();\n      } else {\n        endedEarly = true;\n        break;\n      }\n    }\n    if (endedEarly) {\n      writer.beginObject();\n      writer.name(\"status\").value(\"ended early\");\n      writer.endObject();\n    }\n    writer.endArray();\n  }\n\n  /**\n   * Prints generic stats for all fields in the currently viewed index.\n   */\n  public void dumpStats(ViewerWriter writer) throws IOException {\n    writer.beginObject();\n\n    printHeader(writer);\n    // stats section\n    writer.name(\"stats\");\n    writer.beginArray();\n    writer.newline();\n    for (String field : sortedFields()) {\n      Terms terms = twitterReader.terms(field);\n      if (terms != null) {\n        printStats(writer, field, terms);\n      }\n    }\n    writer.endArray();\n    writer.endObject();\n  }\n\n  private void printStats(ViewerWriter writer, String field, Terms terms) throws IOException {\n    StatsDto statsDto = new StatsDto(\n        field, String.valueOf(terms.size()), terms.getClass().getCanonicalName());\n    statsDto.write(writer);\n    writer.newline();\n  }\n\n  private void printHeader(ViewerWriter writer) throws IOException {\n    writer.name(\"timeSliceId\").value(Long.toString(this.searcher.getTimeSliceID()));\n    writer.name(\"maxDocNumber\").value(Integer.toString(this.twitterReader.maxDoc()));\n    writer.newline();\n  }\n\n  private static String formatField(String field) {\n    return String.format(\"%20s\", field);\n  }\n\n  /**\n   * Dumps out the schema of the current segment.\n   * @param writer to be used for printing\n   */\n  public void dumpSchema(ViewerWriter writer) throws IOException {\n    writer.beginObject();\n    printHeader(writer);\n    writer.name(\"schemaFields\");\n    writer.beginArray();\n    writer.newline();\n    Schema schema = this.twitterReader.getSchema();\n    // The fields in the schema are not sorted. Sort them so that the output is deterministic\n    Set<String> fieldNameSet = new TreeSet<>();\n    for (Schema.FieldInfo fieldInfo: schema.getFieldInfos()) {\n      fieldNameSet.add(fieldInfo.getName());\n    }\n    for (String fieldName : fieldNameSet) {\n      writer.value(fieldName);\n      writer.newline();\n    }\n    writer.endArray();\n    writer.endObject();\n  }\n\n  /**\n   * Dumps out the indexed fields inside the current segment.\n   * Mainly used to help the front end populate the fields.\n   * @param writer writer to be used for printing\n   */\n  public void dumpFields(ViewerWriter writer) throws IOException {\n    writer.beginObject();\n    printHeader(writer);\n    writer.name(\"fields\");\n    writer.beginArray();\n    writer.newline();\n    for (String field : sortedFields()) {\n      writer.value(field);\n      writer.newline();\n    }\n    writer.endArray();\n    writer.endObject();\n  }\n\n  /**\n   * Dumps out the mapping of the tweet/tweetId to\n   * a docId as well as segment/timeslide pair.\n   * @param writer writer to be used for writing\n   * @param tweetId tweetId that is input by user\n   */\n  public void dumpTweetIdToDocIdMapping(ViewerWriter writer, long tweetId) throws IOException {\n    writer.beginObject();\n    printHeader(writer);\n    writer.name(\"tweetId\").value(Long.toString(tweetId));\n    int docId = twitterReader.getSegmentData().getDocIDToTweetIDMapper().getDocID(tweetId);\n\n    writer.name(\"docId\").value(Integer.toString(docId));\n    writer.endObject();\n    writer.newline();\n  }\n\n  /**\n   * Dumps out the mapping of the docId to\n   * tweetId and timeslice/segmentId pairs.\n   * @param writer writer to be used for writing\n   * @param docid docId that is input by user\n   */\n  public void dumpDocIdToTweetIdMapping(ViewerWriter writer, int docid) throws IOException {\n    writer.beginObject();\n    printHeader(writer);\n    long tweetId = twitterReader.getSegmentData().getDocIDToTweetIDMapper().getTweetID(docid);\n\n    writer.name(\"tweetId\");\n    if (tweetId >= 0) {\n      writer.value(Long.toString(tweetId));\n    } else {\n      writer.value(\"Does not exist in segment\");\n    }\n    writer.name(\"docid\").value(Integer.toString(docid));\n    writer.endObject();\n  }\n\n  /**\n   * Print a response indicating that the given tweet id is not found in the index.\n   *\n   * Note that this method does not actually need the underlying index, and hence is setup as\n   * a util function.\n   */\n  public static void writeTweetDoesNotExistResponse(ViewerWriter writer, long tweetId)\n      throws IOException {\n    writer.beginObject();\n    writer.name(\"tweetId\");\n    writer.value(Long.toString(tweetId));\n    writer.name(\"docId\");\n    writer.value(\"does not exist on this earlybird.\");\n    writer.endObject();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/JsonViewerWriter.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.io.IOException;\nimport java.io.Writer;\n\nimport com.google.gson.stream.JsonWriter;\n\n/**\n * Wrapper class for JsonWriter that implements the\n * ViewerWriter interface.\n */\npublic class JsonViewerWriter implements ViewerWriter {\n\n  private final JsonWriter writer;\n  private final Writer out;\n\n  public JsonViewerWriter(Writer out) {\n    this.out = out;\n    this.writer = new JsonWriter(out);\n  }\n\n\n  @Override\n  public ViewerWriter beginArray() throws IOException {\n    writer.beginArray();\n    return this;\n  }\n\n  @Override\n  public ViewerWriter beginObject() throws IOException {\n    writer.beginObject();\n    return this;\n  }\n\n  @Override\n  public ViewerWriter endArray() throws IOException {\n    writer.endArray();\n    return this;\n  }\n\n  @Override\n  public ViewerWriter endObject() throws IOException {\n    writer.endObject();\n    return this;\n  }\n\n  @Override\n  public ViewerWriter name(String field) throws IOException {\n    writer.name(field);\n    return this;\n  }\n\n  @Override\n  public ViewerWriter value(String s) throws IOException {\n    writer.value(s);\n    return this;\n  }\n\n  @Override\n  public ViewerWriter newline() throws IOException {\n    out.append('\\n');\n    return this;\n  }\n\n  public void flush() throws IOException {\n    out.flush();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/OneTaskScheduledExecutorManager.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.io.Closeable;\nimport java.io.IOException;\nimport java.util.concurrent.ScheduledExecutorService;\nimport java.util.concurrent.ScheduledFuture;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.concurrent.ScheduledExecutorServiceFactory;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\n\n/**\n * Executes a single periodic task.\n */\npublic abstract class OneTaskScheduledExecutorManager\n    extends ScheduledExecutorManager implements Closeable {\n  private final ScheduledExecutorTask scheduledTask;\n  private final PeriodicActionParams periodicActionParams;\n\n  public OneTaskScheduledExecutorManager(\n      ScheduledExecutorServiceFactory executorServiceFactory,\n      String threadNameFormat,\n      boolean isDaemon,\n      PeriodicActionParams periodicActionParams,\n      ShutdownWaitTimeParams shutdownTiming,\n      SearchStatsReceiver searchStatsReceiver,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    this(executorServiceFactory.build(threadNameFormat, isDaemon), periodicActionParams,\n        shutdownTiming, searchStatsReceiver, criticalExceptionHandler);\n  }\n\n  public OneTaskScheduledExecutorManager(\n      ScheduledExecutorService executor,\n      PeriodicActionParams periodicActionParams,\n      ShutdownWaitTimeParams shutdownTiming,\n      SearchStatsReceiver searchStatsReceiver,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    this(executor, periodicActionParams, shutdownTiming, searchStatsReceiver, null,\n        criticalExceptionHandler, Clock.SYSTEM_CLOCK);\n  }\n\n  public OneTaskScheduledExecutorManager(\n      ScheduledExecutorService executor,\n      PeriodicActionParams periodicActionParams,\n      ShutdownWaitTimeParams shutdownWaitTimeParams,\n      SearchStatsReceiver searchStatsReceiver,\n      SearchCounter iterationCounter,\n      CriticalExceptionHandler criticalExceptionHandler,\n      Clock clock) {\n    super(executor, shutdownWaitTimeParams, searchStatsReceiver, iterationCounter,\n        criticalExceptionHandler, clock);\n\n    this.periodicActionParams = periodicActionParams;\n    this.scheduledTask = new ScheduledExecutorTask(getIterationCounter(), clock) {\n      @Override\n      protected void runOneIteration() {\n        OneTaskScheduledExecutorManager.this.runOneIteration();\n      }\n    };\n  }\n\n  /**\n   * Schedule the single internally specified task returned by getScheduledTask.\n   */\n  public ScheduledFuture schedule() {\n    return this.scheduleNewTask(\n        this.getScheduledTask(),\n        this.periodicActionParams\n    );\n  }\n\n  /**\n   * The code that the task executes.\n   */\n  protected abstract void runOneIteration();\n\n  public ScheduledExecutorTask getScheduledTask() {\n    return scheduledTask;\n  }\n\n  @Override\n  public void close() throws IOException {\n    try {\n      shutdown();\n    } catch (InterruptedException e) {\n      throw new IOException(e);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/ParallelUtil.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.util.List;\nimport java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\nimport java.util.concurrent.ThreadFactory;\nimport java.util.stream.Collectors;\n\nimport com.google.common.util.concurrent.ThreadFactoryBuilder;\n\nimport com.twitter.util.Await;\nimport com.twitter.util.Future;\nimport com.twitter.util.Future$;\nimport com.twitter.util.FuturePool;\nimport com.twitter.util.FuturePool$;\n\npublic final class ParallelUtil {\n  private ParallelUtil() {\n  }\n\n  public static <T, R> List<R> parmap(String threadName, CheckedFunction<T, R> fn, List<T> input)\n      throws Exception {\n    return parmap(threadName, input.size(), fn, input);\n  }\n\n  /**\n   * Runs a function in parallel across the elements of the list, and throws an exception if any\n   * of the functions throws, or returns the results.\n   *\n   * Uses as many threads as there are elements in the input, so only use this for tasks that\n   * require significant CPU for each element, and have less elements than the number of cores.\n   */\n  public static <T, R> List<R> parmap(\n      String threadName, int threadPoolSize, CheckedFunction<T, R> fn, List<T> input)\n      throws Exception {\n    ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize,\n        buildThreadFactory(threadName));\n    FuturePool futurePool = FuturePool$.MODULE$.apply(executor);\n\n    List<Future<R>> futures = input\n        .stream()\n        .map(in -> futurePool.apply(() -> {\n          try {\n            return fn.apply(in);\n          } catch (Exception e) {\n            throw new RuntimeException(e);\n          }\n        })).collect(Collectors.toList());\n\n    try {\n      return Await.result(Future$.MODULE$.collect(futures));\n    } finally {\n      executor.shutdownNow();\n    }\n  }\n\n  private static ThreadFactory buildThreadFactory(String threadNameFormat) {\n    return new ThreadFactoryBuilder()\n        .setNameFormat(threadNameFormat)\n        .setDaemon(false)\n        .build();\n  }\n\n  @FunctionalInterface\n  public interface CheckedFunction<T, R> {\n    /**\n     * A function from T to R that throws checked Exceptions.\n     */\n    R apply(T t) throws Exception;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/PeriodicActionParams.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.util.concurrent.TimeUnit;\n\n/**\n * Specifies timing and type of period actions that we schedule.\n *\n * See:\n *  https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ScheduledExecutorService.html\n */\npublic final class PeriodicActionParams {\n  private enum DelayType {\n    FIXED_DELAY,\n    FIXED_RATE\n  }\n\n  private long initialDelayDuration;\n  private long intervalDuration;\n  private TimeUnit intervalUnit;\n  private DelayType delayType;\n\n  public long getInitialDelayDuration() {\n    return initialDelayDuration;\n  }\n\n  public long getIntervalDuration() {\n    return intervalDuration;\n  }\n\n  public TimeUnit getIntervalUnit() {\n    return intervalUnit;\n  }\n\n  public DelayType getDelayType() {\n    return delayType;\n  }\n\n  private PeriodicActionParams(\n      DelayType delayType,\n      long initialDelayDuration,\n      long intervalDuration,\n      TimeUnit intervalUnit) {\n    this.delayType = delayType;\n    this.intervalDuration = intervalDuration;\n    this.initialDelayDuration = initialDelayDuration;\n    this.intervalUnit = intervalUnit;\n  }\n\n  // Runs start at times start, start+X, start+2*X etc., so they can possibly overlap.\n  public static PeriodicActionParams atFixedRate(\n      long intervalDuration,\n      TimeUnit intervalUnit) {\n    return new PeriodicActionParams(DelayType.FIXED_RATE, 0,\n        intervalDuration, intervalUnit);\n  }\n\n  // Delay between every run.\n  // The order of what happens is:\n  //   initial delay, run task, wait X time, run task, wait X time, etc.\n  // Runs can't overlap.\n  public static PeriodicActionParams withIntialWaitAndFixedDelay(\n      long initialDelayDuration,\n      long intervalDuration,\n      TimeUnit intervalUnit) {\n    return new PeriodicActionParams(DelayType.FIXED_DELAY, initialDelayDuration,\n        intervalDuration, intervalUnit);\n  }\n\n  // Delay between every run.\n  public static PeriodicActionParams withFixedDelay(\n      long intervalDuration,\n      TimeUnit intervalUnit) {\n    return withIntialWaitAndFixedDelay(0, intervalDuration, intervalUnit);\n  }\n\n  boolean isFixedDelay() {\n    return this.delayType == DelayType.FIXED_DELAY;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/ScheduledExecutorManager.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.util.concurrent.ScheduledExecutorService;\nimport java.util.concurrent.ScheduledFuture;\nimport java.util.concurrent.TimeUnit;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\n\n/**\n * Base class for classes that run periodic tasks.\n */\npublic abstract class ScheduledExecutorManager {\n  private static final Logger LOG = LoggerFactory.getLogger(ScheduledExecutorManager.class);\n  private static final long SHUTDOWN_WAIT_INTERVAL_SEC = 30;\n\n  public static final String SCHEDULED_EXECUTOR_TASK_PREFIX = \"scheduled_executor_task_\";\n\n  private final String name;\n  private final ScheduledExecutorService executor;\n\n  private final ShutdownWaitTimeParams shutdownWaitTimeParams;\n\n  private final SearchCounter iterationCounter;\n  private final SearchStatsReceiver searchStatsReceiver;\n\n  protected final CriticalExceptionHandler criticalExceptionHandler;\n  private final Clock clock;\n\n  protected boolean shouldLog = true;\n\n  public ScheduledExecutorManager(\n      ScheduledExecutorService executor,\n      ShutdownWaitTimeParams shutdownWaitTimeParams,\n      SearchStatsReceiver searchStatsReceiver,\n      CriticalExceptionHandler criticalExceptionHandler,\n      Clock clock) {\n    this(executor, shutdownWaitTimeParams, searchStatsReceiver, null,\n        criticalExceptionHandler, clock);\n  }\n\n  ScheduledExecutorManager(\n      ScheduledExecutorService executor,\n      ShutdownWaitTimeParams shutdownWaitTimeParams,\n      SearchStatsReceiver searchStatsReceiver,\n      SearchCounter iterationCounter,\n      CriticalExceptionHandler criticalExceptionHandler,\n      Clock clock) {\n    this.name = getClass().getSimpleName();\n    this.executor = executor;\n    this.criticalExceptionHandler = criticalExceptionHandler;\n    this.shutdownWaitTimeParams = shutdownWaitTimeParams;\n\n    if (iterationCounter != null) {\n      this.iterationCounter = iterationCounter;\n    } else {\n      this.iterationCounter = searchStatsReceiver.getCounter(SCHEDULED_EXECUTOR_TASK_PREFIX + name);\n    }\n\n    this.searchStatsReceiver = searchStatsReceiver;\n    this.clock = clock;\n  }\n\n  /**\n   * Schedule a task.\n   */\n  protected final ScheduledFuture scheduleNewTask(\n      ScheduledExecutorTask task,\n      PeriodicActionParams periodicActionParams) {\n    long interval = periodicActionParams.getIntervalDuration();\n    TimeUnit timeUnit = periodicActionParams.getIntervalUnit();\n    long initialDelay = periodicActionParams.getInitialDelayDuration();\n\n    if (interval <= 0) {\n      String message = String.format(\n          \"Not scheduling manager %s for wrong interval %d %s\", name, interval, timeUnit);\n      LOG.error(message);\n      throw new UnsupportedOperationException(message);\n    }\n\n    if (shouldLog) {\n      LOG.info(\"Scheduling to run {} every {} {} with {}\", name, interval, timeUnit,\n              periodicActionParams.getDelayType());\n    }\n    final ScheduledFuture scheduledFuture;\n    if (periodicActionParams.isFixedDelay()) {\n      scheduledFuture = executor.scheduleWithFixedDelay(task, initialDelay, interval, timeUnit);\n    } else {\n      scheduledFuture = executor.scheduleAtFixedRate(task, initialDelay, interval, timeUnit);\n    }\n    return scheduledFuture;\n  }\n\n  /**\n   * Shutdown everything that's running with the executor.\n   */\n  public boolean shutdown() throws InterruptedException {\n    LOG.info(\"Start shutting down {}.\", name);\n    executor.shutdownNow();\n\n    boolean terminated = false;\n    long waitSeconds = shutdownWaitTimeParams.getWaitUnit().toSeconds(\n        shutdownWaitTimeParams.getWaitDuration()\n    );\n\n    if (waitSeconds == 0) {\n      LOG.info(\"Not waiting at all for {}, wait time is set to zero.\", name);\n    } else {\n      while (!terminated && waitSeconds > 0) {\n        long waitTime = Math.min(waitSeconds, SHUTDOWN_WAIT_INTERVAL_SEC);\n        terminated = executor.awaitTermination(waitTime, TimeUnit.SECONDS);\n        waitSeconds -= waitTime;\n\n        if (!terminated) {\n          LOG.info(\"Still shutting down {} ...\", name);\n        }\n      }\n    }\n\n    LOG.info(\"Done shutting down {}, terminated: {}\", name, terminated);\n\n    shutdownComponent();\n    return terminated;\n  }\n\n  protected ScheduledExecutorService getExecutor() {\n    return executor;\n  }\n\n  public final String getName() {\n    return name;\n  }\n\n  public SearchCounter getIterationCounter() {\n    return iterationCounter;\n  }\n\n  protected final SearchStatsReceiver getSearchStatsReceiver() {\n    return searchStatsReceiver;\n  }\n\n  // Override if you need to shutdown additional services.\n  protected void shutdownComponent() {\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/ScheduledExecutorTask.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.metrics.SearchCounter;\n\npublic abstract class ScheduledExecutorTask implements Runnable {\n  private final SearchCounter counter;\n  protected final Clock clock;\n\n  public ScheduledExecutorTask(SearchCounter counter, Clock clock) {\n    Preconditions.checkNotNull(counter);\n    this.counter = counter;\n    this.clock = clock;\n  }\n\n  @Override\n  public final void run() {\n    counter.increment();\n    runOneIteration();\n  }\n\n  @VisibleForTesting\n  protected abstract void runOneIteration();\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/ScrubGenUtil.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.text.ParseException;\nimport java.util.Date;\n\nimport org.apache.commons.lang3.time.FastDateFormat;\n\npublic final class ScrubGenUtil {\n  public static final FastDateFormat SCRUB_GEN_DATE_FORMAT = FastDateFormat.getInstance(\"yyyyMMdd\");\n\n  private ScrubGenUtil() { }\n\n  /**\n   * Helper method to parse a scrub gen from String to date\n   *\n   * @param scrubGen\n   * @return scrubGen in Date type\n   */\n  public static Date parseScrubGenToDate(String scrubGen) {\n    try {\n      return SCRUB_GEN_DATE_FORMAT.parse(scrubGen);\n    } catch (ParseException e) {\n      String msg = \"Malformed scrub gen date: \" + scrubGen;\n      // If we are running a scrub gen and the date is bad we should quit and not continue.\n      throw new RuntimeException(msg, e);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/ShutdownWaitTimeParams.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.util.concurrent.TimeUnit;\n\n/**\n * Specifies how much time do we wait when shutting down a task.\n */\npublic class ShutdownWaitTimeParams {\n  private long waitDuration;\n  private TimeUnit waitUnit;\n\n  public ShutdownWaitTimeParams(long waitDuration, TimeUnit waitUnit) {\n    this.waitDuration = waitDuration;\n    this.waitUnit = waitUnit;\n  }\n\n  public long getWaitDuration() {\n    return waitDuration;\n  }\n\n  public TimeUnit getWaitUnit() {\n    return waitUnit;\n  }\n\n  /**\n   * Returns a ShutdownWaitTimeParams instance that instructs the caller to wait indefinitely for\n   * the task to shut down.\n   */\n  public static ShutdownWaitTimeParams indefinitely() {\n    return new ShutdownWaitTimeParams(Long.MAX_VALUE, TimeUnit.DAYS);\n  }\n\n  /**\n   * Returns a ShutdownWaitTimeParams instance that instructs the caller to shut down the task\n   * immediately.\n   */\n  public static ShutdownWaitTimeParams immediately() {\n    return new ShutdownWaitTimeParams(0, TimeUnit.MILLISECONDS);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/TermCountMonitor.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.atomic.AtomicLong;\nimport java.util.function.Function;\nimport java.util.stream.Collectors;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.apache.commons.lang.mutable.MutableLong;\nimport org.apache.lucene.index.IndexOptions;\nimport org.apache.lucene.index.Terms;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.concurrent.ScheduledExecutorServiceFactory;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.index.EarlybirdSingleSegmentSearcher;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentManager;\n\n/**\n * A background task that periodically gets and exports the number of terms per field that are\n * indexed on this earlybird, averaged over all segments.\n * Specifically used for making sure that we are not missing terms for any fields in the search\n * archives.\n * The task loops though all the segments that are indexed by this earlybird, and for each segment\n * looks at the term counts for all fields in that segment.\n *\n * Also keeps track of the number of fields that do not have any term counts (or below the specified\n * threshold) in the data that is indexed on this earlybird.\n */\npublic class TermCountMonitor extends OneTaskScheduledExecutorManager {\n  private static final Logger LOG = LoggerFactory.getLogger(TermCountMonitor.class);\n\n  private static final String THREAD_NAME_FORMAT = \"TermCountMonitor-%d\";\n  private static final boolean THREAD_IS_DAEMON = true;\n\n  public static final String RUN_INTERVAL_MINUTES_CONFIG_NAME =\n      \"term_count_monitor_run_interval_minutes\";\n\n  private static Function<String, String> termStatNameFunc =\n      field -> \"term_count_on_field_\" + field;\n  private static Function<String, String> tokenStatNameFunc =\n      field -> \"token_count_on_field_\" + field;\n  private static Function<String, String> missingFieldStatNameFunc =\n      field -> \"term_count_monitor_missing_field_\" + field;\n\n  private static class RawFieldCounter {\n    private MutableLong numTerms = new MutableLong(0L);\n    private MutableLong numTokens = new MutableLong(0L);\n  }\n\n  @VisibleForTesting\n  static class ExportedFieldCounter {\n    private final AtomicLong numTerms;\n    private final AtomicLong numTokens;\n\n    ExportedFieldCounter(RawFieldCounter rawCounter) {\n      this.numTerms = new AtomicLong(rawCounter.numTerms.longValue());\n      this.numTokens = new AtomicLong(rawCounter.numTokens.longValue());\n    }\n\n    ExportedFieldCounter(long numInitialTerms, long numInitialTokens) {\n      this.numTerms = new AtomicLong(numInitialTerms);\n      this.numTokens = new AtomicLong(numInitialTokens);\n    }\n\n    @VisibleForTesting\n    long getNumTerms() {\n      return numTerms.longValue();\n    }\n\n    @VisibleForTesting\n    long getNumTokens() {\n      return numTokens.longValue();\n    }\n  }\n\n  private final int fieldMinTermCount =\n      EarlybirdConfig.getInt(\"term_count_monitor_min_count\", 0);\n\n  private final SegmentManager segmentManager;\n  private final Map<String, SearchLongGauge> missingFields;\n  private final Map<String, SearchLongGauge> termStats;\n  private final Map<String, SearchLongGauge> tokenStats;\n  private final Map<String, ExportedFieldCounter> exportedCounts;\n  private final SearchLongGauge termCountOnAllFields;\n  private final SearchLongGauge tokenCountOnAllFields;\n  private final SearchLongGauge fieldsWithNoTermCountStat;\n  private final SearchLongGauge isRunningStat;\n  private final SearchTimerStats checkTimeStat;\n\n  @Override\n  protected void runOneIteration() {\n    LOG.info(\"Starting to get per-field term counts\");\n    isRunningStat.set(1);\n    final SearchTimer timer = checkTimeStat.startNewTimer();\n    try {\n      updateFieldTermCounts();\n    } catch (Exception ex) {\n      LOG.error(\"Unexpected exception while getting per-field term counts\", ex);\n    } finally {\n      LOG.info(\n          \"Done getting per-field term counts. Fields with low term counts: {}\",\n          getFieldsWithLowTermCount());\n      isRunningStat.set(0);\n      checkTimeStat.stopTimerAndIncrement(timer);\n    }\n  }\n\n  /**\n   * Create a term count monitor which monitors the number of terms in segments\n   * managed by the given segment manager.\n   */\n  public TermCountMonitor(\n      SegmentManager segmentManager,\n      ScheduledExecutorServiceFactory executorServiceFactory,\n      long shutdownWaitDuration,\n      TimeUnit shutdownWaitUnit,\n      SearchStatsReceiver searchStatsReceiver,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    super(\n      executorServiceFactory,\n      THREAD_NAME_FORMAT,\n      THREAD_IS_DAEMON,\n      PeriodicActionParams.atFixedRate(\n        EarlybirdConfig.getInt(RUN_INTERVAL_MINUTES_CONFIG_NAME, -1),\n        TimeUnit.MINUTES),\n      new ShutdownWaitTimeParams(\n        shutdownWaitDuration,\n        shutdownWaitUnit\n      ),\n      searchStatsReceiver,\n        criticalExceptionHandler);\n    this.segmentManager = segmentManager;\n    this.missingFields = new HashMap<>();\n    this.termStats = new HashMap<>();\n    this.tokenStats = new HashMap<>();\n    this.exportedCounts = new HashMap<>();\n    this.termCountOnAllFields = getSearchStatsReceiver().getLongGauge(\"term_count_on_all_fields\");\n    this.tokenCountOnAllFields = getSearchStatsReceiver().getLongGauge(\"token_count_on_all_fields\");\n    this.fieldsWithNoTermCountStat =\n        getSearchStatsReceiver().getLongGauge(\"fields_with_low_term_counts\");\n    this.isRunningStat =\n        getSearchStatsReceiver().getLongGauge(\"term_count_monitor_is_running\");\n    this.checkTimeStat =\n        getSearchStatsReceiver().getTimerStats(\n            \"term_count_monitor_check_time\", TimeUnit.MILLISECONDS, true, true, false);\n  }\n\n  private SearchLongGauge getOrCreateLongGauge(\n      Map<String, SearchLongGauge> gauges, String field, Function<String, String> nameSupplier) {\n    SearchLongGauge stat = gauges.get(field);\n\n    if (stat == null) {\n      stat = getSearchStatsReceiver().getLongGauge(nameSupplier.apply(field));\n      gauges.put(field, stat);\n    }\n\n    return stat;\n  }\n\n  private void updateFieldTermCounts() {\n    // 0. Get the current per-field term counts\n    Map<String, RawFieldCounter> newCounts = getFieldStats();\n    LOG.info(\"Computed field stats for all segments\");\n\n    // 1. Update all existing keys\n    for (Map.Entry<String, ExportedFieldCounter> exportedCount : exportedCounts.entrySet()) {\n      String field = exportedCount.getKey();\n      ExportedFieldCounter exportedCountValue = exportedCount.getValue();\n\n      RawFieldCounter newCount = newCounts.get(field);\n      if (newCount == null) {\n        exportedCountValue.numTerms.set(0L);\n        exportedCountValue.numTokens.set(0L);\n      } else {\n        exportedCountValue.numTerms.set(newCount.numTerms.longValue());\n        exportedCountValue.numTokens.set(newCount.numTokens.longValue());\n\n        // clean up so that we don't check this field again when we look for new field\n        newCounts.remove(field);\n      }\n    }\n\n    // 2. Add and export all new fields' term counts\n    for (Map.Entry<String, RawFieldCounter> newCount: newCounts.entrySet()) {\n      String field = newCount.getKey();\n      Preconditions.checkState(!exportedCounts.containsKey(field),\n          \"Should have already processed and removed existing fields: \" + field);\n\n      ExportedFieldCounter newStat = new ExportedFieldCounter(newCount.getValue());\n      exportedCounts.put(field, newStat);\n    }\n\n    // 3. Export as a stat the term counts for all the known fields.\n    for (Map.Entry<String, ExportedFieldCounter> exportedCount : exportedCounts.entrySet()) {\n      String field = exportedCount.getKey();\n      ExportedFieldCounter counter = exportedCount.getValue();\n\n      getOrCreateLongGauge(termStats, field, termStatNameFunc).set(counter.numTerms.get());\n      getOrCreateLongGauge(tokenStats, field, tokenStatNameFunc).set(counter.numTokens.get());\n    }\n\n    // 4. Export as a stat, number of fields not having enough term counts (i.e. <= 0)\n    int fieldsWithNoTermCounts = 0;\n    for (Map.Entry<String, ExportedFieldCounter> fieldTermCount : exportedCounts.entrySet()) {\n      String field = fieldTermCount.getKey();\n      AtomicLong exportedCountValue = fieldTermCount.getValue().numTerms;\n      if (exportedCountValue.get() <= fieldMinTermCount) {\n        LOG.warn(\n            \"Found a field with too few term counts. Field: {} count: {}\",\n            field, exportedCountValue);\n        fieldsWithNoTermCounts++;\n      }\n    }\n    this.fieldsWithNoTermCountStat.set(fieldsWithNoTermCounts);\n  }\n\n  /**\n   * Loops through all segments, and for each field gets the average term/token count.\n   * Based on that, returns a map from each field to its term/token count (average per segment).\n   */\n  private Map<String, RawFieldCounter> getFieldStats() {\n    Iterable<SegmentInfo> segmentInfos = segmentManager.getSegmentInfos(\n        SegmentManager.Filter.Enabled, SegmentManager.Order.NEW_TO_OLD);\n    Map<String, RawFieldCounter> rawCounts = new HashMap<>();\n\n    ImmutableSchemaInterface schemaSnapshot =\n        segmentManager.getEarlybirdIndexConfig().getSchema().getSchemaSnapshot();\n    Set<String> missingFieldsCandidates = schemaSnapshot\n        .getFieldInfos()\n        .stream()\n        .filter(fieldInfo -> fieldInfo.getFieldType().indexOptions() != IndexOptions.NONE)\n        .map(Schema.FieldInfo::getName)\n        .collect(Collectors.toSet());\n    int segmentCount = 0;\n    for (SegmentInfo segmentInfo : segmentInfos) {\n      segmentCount++;\n      try {\n        EarlybirdSingleSegmentSearcher searcher = segmentManager.getSearcher(\n            segmentInfo.getTimeSliceID(), schemaSnapshot);\n        if (searcher != null) {\n          EarlybirdIndexSegmentAtomicReader reader = searcher.getTwitterIndexReader();\n          for (Schema.FieldInfo fieldInfo : schemaSnapshot.getFieldInfos()) {\n            if (fieldInfo.getFieldType().indexOptions() == IndexOptions.NONE) {\n              continue;\n            }\n\n            String fieldName = fieldInfo.getName();\n            RawFieldCounter count = rawCounts.get(fieldName);\n            if (count == null) {\n              count = new RawFieldCounter();\n              rawCounts.put(fieldName, count);\n            }\n            Terms terms = reader.terms(fieldName);\n            if (terms != null) {\n              missingFieldsCandidates.remove(fieldName);\n              count.numTerms.add(terms.size());\n              long sumTotalTermFreq = terms.getSumTotalTermFreq();\n              if (sumTotalTermFreq != -1) {\n                count.numTokens.add(sumTotalTermFreq);\n              }\n            }\n          }\n        }\n      } catch (Exception e) {\n        LOG.error(\"Exception getting average term count per field: \" + segmentInfo, e);\n      }\n    }\n\n    // Update missing fields stats.\n    missingFieldsCandidates.forEach(\n        field -> getOrCreateLongGauge(missingFields, field, missingFieldStatNameFunc).set(1));\n    missingFields.keySet().stream()\n        .filter(\n            field -> !missingFieldsCandidates.contains(field))\n        .forEach(\n            field -> getOrCreateLongGauge(missingFields, field, missingFieldStatNameFunc).set(0));\n\n    long totalTermCount = 0;\n    long totalTokenCount = 0;\n    if (segmentCount == 0) {\n      LOG.error(\"No segments are found to calculate per-field term counts.\");\n    } else {\n      LOG.debug(\"TermCountMonitor.getPerFieldTermCount.segmentCount = {}\", segmentCount);\n      LOG.debug(\"  field: term count (average per segment)\");\n      for (Map.Entry<String, RawFieldCounter> entry : rawCounts.entrySet()) {\n        String field = entry.getKey();\n        final long averageTermCount = entry.getValue().numTerms.longValue() / segmentCount;\n        final long averageTokenCount = entry.getValue().numTokens.longValue() / segmentCount;\n        totalTermCount += entry.getValue().numTerms.longValue();\n        totalTokenCount += entry.getValue().numTokens.longValue();\n\n        LOG.debug(\"  '{} term': {}\", field, averageTermCount);\n        LOG.debug(\"  '{} token': {}\", field, averageTokenCount);\n\n        entry.getValue().numTerms.setValue(averageTermCount);\n        entry.getValue().numTokens.setValue(averageTokenCount);\n      }\n    }\n    LOG.info(\"Total term count: {}\", totalTermCount);\n    LOG.info(\"Total token count: {}\", totalTokenCount);\n    this.termCountOnAllFields.set(totalTermCount);\n    this.tokenCountOnAllFields.set(totalTokenCount);\n\n    return rawCounts;\n  }\n\n  @VisibleForTesting\n  Map<String, ExportedFieldCounter> getExportedCounts() {\n    return Collections.unmodifiableMap(this.exportedCounts);\n  }\n\n  @VisibleForTesting\n  long getFieldsWithLowTermCount() {\n    return fieldsWithNoTermCountStat.get();\n  }\n\n  @VisibleForTesting\n  Map<String, SearchLongGauge> getMissingFields() {\n    return missingFields;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/TweetCountMonitor.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Calendar;\nimport java.util.Date;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.atomic.AtomicInteger;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.Maps;\n\nimport org.apache.commons.lang.mutable.MutableInt;\nimport org.apache.commons.lang.mutable.MutableLong;\nimport org.apache.lucene.index.IndexOptions;\nimport org.apache.lucene.index.PostingsEnum;\nimport org.apache.lucene.index.Terms;\nimport org.apache.lucene.index.TermsEnum;\nimport org.apache.lucene.search.DocIdSetIterator;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.search.common.concurrent.ScheduledExecutorServiceFactory;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchStatsReceiver;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.common.partitioning.base.Segment;\nimport com.twitter.search.common.schema.base.ImmutableSchemaInterface;\nimport com.twitter.search.common.schema.base.Schema;\nimport com.twitter.search.core.earlybird.index.DocIDToTweetIDMapper;\nimport com.twitter.search.core.earlybird.index.EarlybirdIndexSegmentAtomicReader;\nimport com.twitter.search.core.earlybird.index.TimeMapper;\nimport com.twitter.search.earlybird.common.config.EarlybirdConfig;\nimport com.twitter.search.earlybird.exception.CriticalExceptionHandler;\nimport com.twitter.search.earlybird.index.EarlybirdSingleSegmentSearcher;\nimport com.twitter.search.earlybird.partition.SegmentInfo;\nimport com.twitter.search.earlybird.partition.SegmentManager;\n\n/**\n * A background task that periodically gets and exports the number of tweets per hour that are\n * indexed on this earlybird.\n * Specifically used for making sure that we are not missing data for any hours in the search\n * archives.\n * The task loops though all the segments that are indexed by this earlybird, and for each segment\n * looks at all the createdAt dates for all of the documents in that segment.\n *\n * Also keeps track off an exposes as a stat the number of hours that do not have any tweets in the\n * min/max range of data that IS indexed on this earlybird. i.e if we only have data for\n * 2006/01/01:02 and 2006/01/01:04, it will consider 2006/01/01:03 as a missing hour.\n * Hours before 2006/01/01:02 or after 2006/01/01:04 will not be considered as missing.\n */\npublic class TweetCountMonitor extends OneTaskScheduledExecutorManager {\n  private static final Logger LOG = LoggerFactory.getLogger(TweetCountMonitor.class);\n\n  private static final String THREAD_NAME_FORMAT = \"TweetCountMonitor-%d\";\n  private static final boolean THREAD_IS_DAEMON = true;\n\n  public static final String RUN_INTERVAL_MINUTES_CONFIG_NAME =\n      \"tweet_count_monitor_run_interval_minutes\";\n  public static final String START_CHECK_HOUR_CONFIG_NAME =\n      \"tweet_count_monitor_start_check_hour\";\n  public static final String HOURLY_MIN_COUNT_CONFIG_NAME =\n      \"tweet_count_monitor_hourly_min_count\";\n  public static final String DAILY_MIN_COUNT_CONFIG_NAME =\n      \"tweet_count_monitor_daily_min_count\";\n\n  @VisibleForTesting\n  public static final AtomicInteger INSTANCE_COUNTER = new AtomicInteger(0);\n\n  private static final long MILLIS_IN_A_DAY = TimeUnit.DAYS.toMillis(1);\n\n  private final SegmentManager segmentManager;\n\n  private final SearchStatsReceiver searchStatsReceiver;\n  private final int instanceCounter;\n\n  // The first date in format \"YYYYMMDDHH\" that we want to check counts for.\n  private final int startCheckHour;\n  // The last date in format \"YYYYMMDDHH\" that we want to check counts for.\n  private final int endCheckHour;\n  //Smallest number of docs we expect to have for each day.\n  private final int dailyMinCount;\n  // Smallest number of docs we expect to have for each hour.\n  private final int hourlyMinCount;\n  // Binary stat, set to 0 when the monitor is running\n  private final SearchLongGauge isRunningStat;\n  // How long each iteration takes\n  private final SearchTimerStats checkTimeStat;\n\n  private final Map<String, FieldTermCounter> fieldTermCounters;\n  private final Map<String, SearchTimerStats> fieldCheckTimeStats;\n\n  /**\n   * Create a TweetCountMonitor to monitor all segments in the given segmentManager\n   */\n  public TweetCountMonitor(\n      SegmentManager segmentManager,\n      ScheduledExecutorServiceFactory executorServiceFactory,\n      long shutdownWaitDuration,\n      TimeUnit shutdownWaitUnit,\n      SearchStatsReceiver searchStatsReceiver,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    this(segmentManager,\n        EarlybirdConfig.getInt(START_CHECK_HOUR_CONFIG_NAME, 0),\n        EarlybirdConfig.getInt(RUN_INTERVAL_MINUTES_CONFIG_NAME, -1),\n        EarlybirdConfig.getInt(HOURLY_MIN_COUNT_CONFIG_NAME, 0),\n        EarlybirdConfig.getInt(DAILY_MIN_COUNT_CONFIG_NAME, 0),\n        executorServiceFactory,\n        shutdownWaitDuration,\n        shutdownWaitUnit,\n        searchStatsReceiver,\n        criticalExceptionHandler);\n  }\n\n  @VisibleForTesting\n  TweetCountMonitor(\n      SegmentManager segmentManager,\n      int startCheckHourFromConfig,\n      int schedulePeriodMinutes,\n      int hourlyMinCount,\n      int dailyMinCount,\n      ScheduledExecutorServiceFactory executorServiceFactory,\n      long shutdownWaitDuration,\n      TimeUnit shutdownWaitUnit,\n      SearchStatsReceiver searchStatsReceiver,\n      CriticalExceptionHandler criticalExceptionHandler) {\n    super(\n      executorServiceFactory,\n      THREAD_NAME_FORMAT,\n      THREAD_IS_DAEMON,\n      PeriodicActionParams.atFixedRate(\n        schedulePeriodMinutes,\n        TimeUnit.MINUTES\n      ),\n      new ShutdownWaitTimeParams(\n        shutdownWaitDuration,\n        shutdownWaitUnit\n      ),\n      searchStatsReceiver,\n        criticalExceptionHandler);\n    this.segmentManager = segmentManager;\n    this.searchStatsReceiver = searchStatsReceiver;\n    this.instanceCounter = INSTANCE_COUNTER.incrementAndGet();\n    this.hourlyMinCount = hourlyMinCount;\n    this.dailyMinCount = dailyMinCount;\n\n    String isRunningStatName = \"tweet_count_monitor_is_running_v_\" + this.instanceCounter;\n    this.isRunningStat = SearchLongGauge.export(isRunningStatName);\n    String checkTimeStatName = \"tweet_count_monitor_check_time_v_\" + this.instanceCounter;\n    this.checkTimeStat = SearchTimerStats.export(checkTimeStatName, TimeUnit.MILLISECONDS, true);\n\n    this.startCheckHour = Math.max(\n        startCheckHourFromConfig,\n        dateToHourValue(segmentManager.getPartitionConfig().getTierStartDate()));\n    this.endCheckHour = dateToHourValue(segmentManager.getPartitionConfig().getTierEndDate());\n\n    this.fieldTermCounters = Maps.newHashMap();\n    this.fieldTermCounters.put(\n        FieldTermCounter.TWEET_COUNT_KEY,\n        new FieldTermCounter(\n            FieldTermCounter.TWEET_COUNT_KEY,\n            instanceCounter,\n            startCheckHour,\n            endCheckHour,\n            hourlyMinCount,\n            dailyMinCount));\n    this.fieldCheckTimeStats = Maps.newHashMap();\n  }\n\n  private int dateToHourValue(Date date) {\n    Calendar cal = Calendar.getInstance(FieldTermCounter.TIME_ZONE);\n    cal.setTime(date);\n    return FieldTermCounter.getHourValue(cal);\n  }\n\n  private void updateHourlyCounts() {\n    // Iterate the current index to count all tweets anf field hits.\n    Map<String, Map<Integer, MutableInt>> newCountMap = getNewTweetCountMap();\n\n    for (Map.Entry<String, Map<Integer, MutableInt>> newCounts : newCountMap.entrySet()) {\n      final String fieldName = newCounts.getKey();\n      FieldTermCounter termCounter = fieldTermCounters.get(fieldName);\n      if (termCounter == null) {\n        termCounter = new FieldTermCounter(\n            fieldName,\n            instanceCounter,\n            startCheckHour,\n            endCheckHour,\n            hourlyMinCount,\n            dailyMinCount);\n        fieldTermCounters.put(fieldName, termCounter);\n      }\n      termCounter.runWithNewCounts(newCounts.getValue());\n    }\n  }\n\n  /**\n   * Loops through all segments, and all documents in each segment, and for each document\n   * gets the createdAt timestamp (in seconds) from the TimeMapper.\n   * Based on that, returns a map with the count of:\n   * . the number of tweets for each hour\n   * . the number of tweets corresponding to each field for each hour\n   */\n  private Map<String, Map<Integer, MutableInt>> getNewTweetCountMap() {\n    Iterable<SegmentInfo> segmentInfos = segmentManager.getSegmentInfos(\n        SegmentManager.Filter.Enabled, SegmentManager.Order.NEW_TO_OLD);\n    Map<String, Map<Integer, MutableInt>> newCountMap = Maps.newHashMap();\n\n    Map<Integer, MutableInt> newCounts = Maps.newHashMap();\n    newCountMap.put(FieldTermCounter.TWEET_COUNT_KEY, newCounts);\n\n    ImmutableSchemaInterface schemaSnapshot =\n        segmentManager.getEarlybirdIndexConfig().getSchema().getSchemaSnapshot();\n    Calendar cal = Calendar.getInstance(FieldTermCounter.TIME_ZONE);\n    for (SegmentInfo segmentInfo : segmentInfos) {\n      try {\n        EarlybirdSingleSegmentSearcher searcher = segmentManager.getSearcher(\n            segmentInfo.getTimeSliceID(), schemaSnapshot);\n        if (searcher != null) {\n          EarlybirdIndexSegmentAtomicReader reader = searcher.getTwitterIndexReader();\n          TimeMapper timeMapper = reader.getSegmentData().getTimeMapper();\n          List<Pair<String, Integer>> outsideEndDateRangeDocList = new ArrayList<>();\n\n          // Get the number of tweets for each hour.\n          int docsOutsideEndDateRange = getNewTweetCountsForSegment(\n              segmentInfo, reader, timeMapper, cal, newCounts);\n          if (docsOutsideEndDateRange > 0) {\n            outsideEndDateRangeDocList.add(new Pair<>(\n                FieldTermCounter.TWEET_COUNT_KEY, docsOutsideEndDateRange));\n          }\n\n          // Get the number of tweets with corresponding field for each hour.\n          for (Schema.FieldInfo fieldInfo : schemaSnapshot.getFieldInfos()) {\n            if (fieldInfo.getFieldType().indexOptions() == IndexOptions.NONE) {\n              continue;\n            }\n\n            String fieldName = fieldInfo.getName();\n            docsOutsideEndDateRange = getNewFieldTweetCountsForSegment(\n                segmentInfo, reader, timeMapper, cal, fieldName, newCountMap);\n            if (docsOutsideEndDateRange > 0) {\n              outsideEndDateRangeDocList.add(new Pair<>(fieldName, docsOutsideEndDateRange));\n            }\n          }\n\n          LOG.info(\"Inspected segment: \" + segmentInfo + \" found \"\n              + outsideEndDateRangeDocList.size()\n              + \" fields with documents outside of segment end date.\");\n          for (Pair<String, Integer> outsideEndRange : outsideEndDateRangeDocList) {\n            LOG.info(\"  outside end date range - segment: \" + segmentInfo.getSegmentName()\n                + \" field: \" + outsideEndRange.toString());\n          }\n        }\n      } catch (IOException e) {\n        LOG.error(\"Exception getting daily tweet counts for timeslice: \" + segmentInfo, e);\n      }\n    }\n    return newCountMap;\n  }\n\n  private void incrementNumDocsWithIllegalTimeCounter(String segmentName, String fieldSuffix) {\n    String statName = String.format(\n        \"num_docs_with_illegal_time_for_segment_%s%s_counter\", segmentName, fieldSuffix);\n    SearchCounter counter = SearchCounter.export(statName);\n    counter.increment();\n  }\n\n  private int getNewTweetCountsForSegment(\n      SegmentInfo segmentInfo,\n      EarlybirdIndexSegmentAtomicReader reader,\n      TimeMapper timeMapper,\n      Calendar cal,\n      Map<Integer, MutableInt> newTweetCounts) {\n    DocIDToTweetIDMapper tweetIdMapper = reader.getSegmentData().getDocIDToTweetIDMapper();\n    long dataEndTimeExclusiveMillis = getDataEndTimeExclusiveMillis(segmentInfo);\n    int docsOutsideEndDateRange = 0;\n    int docId = Integer.MIN_VALUE;\n    while ((docId = tweetIdMapper.getNextDocID(docId)) != DocIDToTweetIDMapper.ID_NOT_FOUND) {\n      UpdateCountType updateCountType =\n          updateTweetCount(timeMapper, docId, dataEndTimeExclusiveMillis, cal, newTweetCounts);\n      if (updateCountType == UpdateCountType.ILLEGAL_TIME) {\n        incrementNumDocsWithIllegalTimeCounter(segmentInfo.getSegmentName(), \"\");\n      } else if (updateCountType == UpdateCountType.OUT_OF_RANGE_TIME) {\n        docsOutsideEndDateRange++;\n      }\n    }\n    return docsOutsideEndDateRange;\n  }\n\n  private int getNewFieldTweetCountsForSegment(\n      SegmentInfo segmentInfo,\n      EarlybirdIndexSegmentAtomicReader reader,\n      TimeMapper timeMapper,\n      Calendar cal,\n      String field,\n      Map<String, Map<Integer, MutableInt>> newCountMap) throws IOException {\n    int docsOutsideEndDateRange = 0;\n    Map<Integer, MutableInt> fieldTweetCounts =\n        newCountMap.computeIfAbsent(field, k -> Maps.newHashMap());\n\n    Terms terms = reader.terms(field);\n    if (terms == null) {\n      LOG.warn(\"Field <\" + field + \"> is missing terms in segment: \"\n          + segmentInfo.getSegmentName());\n      return 0;\n    }\n    long startTimeMillis = System.currentTimeMillis();\n\n    long dataEndTimeExclusiveMillis = getDataEndTimeExclusiveMillis(segmentInfo);\n    for (TermsEnum termsEnum = terms.iterator(); termsEnum.next() != null;) {\n      DocIdSetIterator docsIterator = termsEnum.postings(null, PostingsEnum.NONE);\n      for (int docId = docsIterator.nextDoc();\n           docId != DocIdSetIterator.NO_MORE_DOCS; docId = docsIterator.nextDoc()) {\n        UpdateCountType updateCountType = updateTweetCount(\n            timeMapper, docId, dataEndTimeExclusiveMillis, cal, fieldTweetCounts);\n        if (updateCountType == UpdateCountType.ILLEGAL_TIME) {\n          incrementNumDocsWithIllegalTimeCounter(\n              segmentInfo.getSegmentName(), \"_and_field_\" + field);\n        } else if (updateCountType == UpdateCountType.OUT_OF_RANGE_TIME) {\n          docsOutsideEndDateRange++;\n        }\n      }\n    }\n    updateFieldRunTimeStats(field, System.currentTimeMillis() - startTimeMillis);\n\n    return docsOutsideEndDateRange;\n  }\n\n  private enum UpdateCountType {\n    OK_TIME,\n    ILLEGAL_TIME,\n    OUT_OF_RANGE_TIME,\n  }\n\n  private static UpdateCountType updateTweetCount(\n      TimeMapper timeMapper,\n      int docId,\n      long dataEndTimeExclusiveMillis,\n      Calendar cal,\n      Map<Integer, MutableInt> newTweetCounts) {\n    int timeSecs = timeMapper.getTime(docId);\n    if (timeSecs == TimeMapper.ILLEGAL_TIME) {\n      return UpdateCountType.ILLEGAL_TIME;\n    }\n    if (dataEndTimeExclusiveMillis == Segment.NO_DATA_END_TIME\n        || timeSecs * 1000L < dataEndTimeExclusiveMillis) {\n      Integer hourlyValue = FieldTermCounter.getHourValue(cal, timeSecs);\n      MutableInt count = newTweetCounts.get(hourlyValue);\n      if (count == null) {\n        count = new MutableInt(0);\n        newTweetCounts.put(hourlyValue, count);\n      }\n      count.increment();\n      return UpdateCountType.OK_TIME;\n    } else {\n      return UpdateCountType.OUT_OF_RANGE_TIME;\n    }\n  }\n\n  /**\n   * If a segment has an end date, return the last timestamp (exclusive, and in millis) for which\n   * we expect it to have data.\n   * @return Segment.NO_DATA_END_TIME if the segment does not have an end date.\n   */\n  private long getDataEndTimeExclusiveMillis(SegmentInfo segmentInfo) {\n    long dataEndDate = segmentInfo.getSegment().getDataEndDateInclusiveMillis();\n    if (dataEndDate == Segment.NO_DATA_END_TIME) {\n      return Segment.NO_DATA_END_TIME;\n    } else {\n      return dataEndDate + MILLIS_IN_A_DAY;\n    }\n  }\n\n  private void updateFieldRunTimeStats(String fieldName, long runTimeMs) {\n    SearchTimerStats timerStats = fieldCheckTimeStats.get(fieldName);\n    if (timerStats == null) {\n      final String statName = \"tweet_count_monitor_check_time_field_\" + fieldName;\n      timerStats = searchStatsReceiver.getTimerStats(\n          statName, TimeUnit.MILLISECONDS, false, false, false);\n      fieldCheckTimeStats.put(fieldName, timerStats);\n    }\n    timerStats.timerIncrement(runTimeMs);\n  }\n\n  @VisibleForTesting\n  String getStatName(String fieldName, Integer date) {\n    return FieldTermCounter.getStatName(fieldName, instanceCounter, date);\n  }\n\n  @VisibleForTesting\n  Map<Integer, AtomicInteger> getExportedCounts(String fieldName) {\n    if (fieldTermCounters.get(fieldName) == null) {\n      return null;\n    } else {\n      return fieldTermCounters.get(fieldName).getExportedCounts();\n    }\n  }\n\n  @VisibleForTesting\n  Map<Integer, MutableLong> getDailyCounts(String fieldName) {\n    if (fieldTermCounters.get(fieldName) == null) {\n      return null;\n    } else {\n      return fieldTermCounters.get(fieldName).getDailyCounts();\n    }\n  }\n\n  @VisibleForTesting\n  long getHoursWithNoTweets(String fieldName) {\n    return fieldTermCounters.get(fieldName).getHoursWithNoTweets();\n  }\n\n  @VisibleForTesting\n  long getDaysWithNoTweets(String fieldName) {\n    return fieldTermCounters.get(fieldName).getDaysWithNoTweets();\n  }\n\n  @VisibleForTesting\n  Map<String, SearchLongGauge> getExportedHourlyCountStats(String fieldName) {\n    return fieldTermCounters.get(fieldName).getExportedHourlyCountStats();\n  }\n\n  @Override\n  protected void runOneIteration() {\n    LOG.info(\"Starting to get hourly tweet counts\");\n    final long startTimeMillis = System.currentTimeMillis();\n\n    isRunningStat.set(1);\n    try {\n      updateHourlyCounts();\n    } catch (Exception ex) {\n      LOG.error(\"Unexpected exception while getting hourly tweet counts\", ex);\n    } finally {\n      isRunningStat.set(0);\n\n      long elapsedTimeMillis = System.currentTimeMillis() - startTimeMillis;\n      checkTimeStat.timerIncrement(elapsedTimeMillis);\n      LOG.info(\"Done getting daily tweet counts. Hours without tweets: \"\n          + getHoursWithNoTweets(FieldTermCounter.TWEET_COUNT_KEY));\n      LOG.info(\"Updating tweet count takes \" + (elapsedTimeMillis / 1000) + \" secs.\");\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird/util/ViewerWriter.java",
    "content": "package com.twitter.search.earlybird.util;\n\nimport java.io.IOException;\n\n/**\n * Interface class for writer.  Writer should be passed in\n * and have these methods.  Currently keeps the hierarchy for\n * completed and valid json, methods mirror the ones found in\n * JsonWriter\n * http://google-gson.googlecode.com/svn/trunk/gson/docs/javadocs/com/google/gson/stream/JsonWriter.html\n */\npublic interface ViewerWriter {\n  /**\n   * Writes a mark for the beginning of an array.\n   */\n  ViewerWriter beginArray() throws IOException;\n\n  /**\n   * Writes a mark for the beginning of an object.\n   */\n  ViewerWriter beginObject() throws IOException;\n\n  /**\n   * Writes a mark for the end of an array.\n   */\n  ViewerWriter endArray() throws IOException;\n\n  /**\n   * Writes a mark for the end of an object.\n   */\n  ViewerWriter endObject() throws IOException;\n\n  /**\n   * Writes the name (key) of a property.\n   */\n  ViewerWriter name(String field) throws IOException;\n\n  /**\n   * Writes the value of a property.\n   */\n  ViewerWriter value(String s) throws IOException;\n\n  /**\n   * Writes a new line.\n   */\n  ViewerWriter newline() throws IOException;\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/BUILD",
    "content": "java_library(\n    name = \"earlybird_root-lib\",\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/commons-lang\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"decider/src/main/scala\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authentication\",\n        \"finagle-internal/mtls/src/main/scala/com/twitter/finagle/mtls/authorization/server\",\n        \"finagle/finagle-memcached/src/main/java\",\n        \"finagle/finagle-mux/src/main/scala\",\n        \"finagle/finagle-thrift/src/main/java\",\n        \"finagle/finagle-thrift/src/main/scala\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"finatra/inject/inject-server/src/main/scala/com/twitter/inject/server\",\n        \"src/java/com/google/common/util/concurrent\",\n        \"src/java/com/twitter/common/collections\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/search/common/caching\",\n        \"src/java/com/twitter/search/common/clientstats\",\n        \"src/java/com/twitter/search/common/config\",\n        \"src/java/com/twitter/search/common/dark\",\n        \"src/java/com/twitter/search/common/decider\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/partitioning/base\",\n        \"src/java/com/twitter/search/common/partitioning/zookeeper\",\n        \"src/java/com/twitter/search/common/relevance:ranking\",\n        \"src/java/com/twitter/search/common/root\",\n        \"src/java/com/twitter/search/common/runtime\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/common/search\",\n        \"src/java/com/twitter/search/common/util/earlybird\",\n        \"src/java/com/twitter/search/common/util/io/periodic\",\n        \"src/java/com/twitter/search/common/util/zookeeper\",\n        \"src/java/com/twitter/search/earlybird/common\",\n        \"src/java/com/twitter/search/earlybird/config\",\n        \"src/java/com/twitter/search/earlybird_root/caching\",\n        \"src/java/com/twitter/search/earlybird_root/common\",\n        \"src/java/com/twitter/search/earlybird_root/filters\",\n        \"src/java/com/twitter/search/earlybird_root/mergers\",\n        \"src/java/com/twitter/search/earlybird_root/quota\",\n        \"src/java/com/twitter/search/earlybird_root/routers\",\n        \"src/java/com/twitter/search/earlybird_root/visitors\",\n        \"src/java/com/twitter/search/queryparser\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/java/com/twitter/search/queryparser/query/search:search-query-nodes\",\n        \"src/thrift/com/twitter/search:benchmark_query-java\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n        \"stitch/stitch-core\",\n        \"strato/src/main/scala/com/twitter/strato/catalog\",\n        \"strato/src/main/scala/com/twitter/strato/client\",\n        \"thrift-web-forms\",\n        \"thrift-web-forms/src/main/scala/com/twitter/thriftwebforms/model\",\n    ],\n)\n\njvm_binary(\n    name = \"earlybird_root-binary\",\n    basename = \"earlybird_root\",\n    # The main class is reset in the aurora files (it's a required param).\n    # We need to set it to something here, because hadoop_binary requires it.\n    main = \"com.twitter.search.earlybird_root.RealtimeRootAppMain\",\n    runtime_platform = \"java11\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/org/slf4j:slf4j-log4j12\",\n        \":earlybird_root-lib\",\n        \"src/java/com/twitter/search/common/logging:search-log4j\",\n        # For /admin/logging.\n        \"twitter-server/slf4j-log4j12/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/ClientBackupFilter.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.Map;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.finagle.client.BackupRequestFilter;\nimport com.twitter.finagle.service.ResponseClassifier;\nimport com.twitter.finagle.service.RetryBudgets;\nimport com.twitter.finagle.stats.StatsReceiver;\nimport com.twitter.finagle.util.DefaultTimer;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchCustomGauge;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.util.Future;\nimport com.twitter.util.tunable.Tunable;\n\npublic class ClientBackupFilter extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n  private static final Logger LOG = LoggerFactory.getLogger(ClientBackupFilter.class);\n\n  private final Map<String, BackupRequestFilter<EarlybirdRequest, EarlybirdResponse>>\n      clientBackupFilters = new ConcurrentHashMap<>();\n  private final boolean sendInterupts = false;\n  private final String statPrefix;\n  private final Tunable.Mutable<Object> maxExtraLoad;\n  private final StatsReceiver statsReceiver;\n  private final SearchDecider decider;\n  private final String backupRequestPrecentExtraLoadDecider;\n  private final int minSendBackupAfterMs = 1;\n\n  public ClientBackupFilter(String serviceName,\n                            String statPrefix,\n                            StatsReceiver statsReceiver,\n                            SearchDecider decider) {\n    this.statPrefix = statPrefix;\n    this.backupRequestPrecentExtraLoadDecider = serviceName + \"_backup_request_percent_extra_load\";\n    this.decider = decider;\n    this.maxExtraLoad = Tunable.mutable(\"backup_tunable\", getMaxExtraLoadFromDecider());\n    this.statsReceiver = statsReceiver;\n    SearchCustomGauge.export(serviceName + \"_backup_request_factor\",\n        () -> (maxExtraLoad.apply().isDefined()) ? (double) maxExtraLoad.apply().get() : -1);\n  }\n\n  private double getMaxExtraLoadFromDecider() {\n    return ((double) decider.getAvailability(backupRequestPrecentExtraLoadDecider)) / 100 / 100;\n  }\n\n  private BackupRequestFilter<EarlybirdRequest, EarlybirdResponse> backupFilter(String client) {\n    return new BackupRequestFilter<EarlybirdRequest, EarlybirdResponse>(\n        maxExtraLoad,\n        sendInterupts,\n        minSendBackupAfterMs,\n        ResponseClassifier.Default(),\n        RetryBudgets.newRetryBudget(),\n        statsReceiver.scope(statPrefix, client, \"backup_filter\"),\n        DefaultTimer.getInstance(),\n        client);\n  }\n\n  private void updateMaxExtraLoadIfNecessary() {\n    double maxExtraLoadDeciderValue = getMaxExtraLoadFromDecider();\n    if (maxExtraLoad.apply().isDefined()\n        && !maxExtraLoad.apply().get().equals(maxExtraLoadDeciderValue)) {\n      LOG.info(\"Updating maxExtraLoad from {} to {}\",\n          maxExtraLoad.apply().get(),\n          maxExtraLoadDeciderValue);\n      maxExtraLoad.set(maxExtraLoadDeciderValue);\n    }\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(EarlybirdRequest request,\n                                         Service<EarlybirdRequest, EarlybirdResponse> service) {\n    updateMaxExtraLoadIfNecessary();\n\n    String clientID = ClientIdUtil.getClientIdFromRequest(request);\n    BackupRequestFilter<EarlybirdRequest, EarlybirdResponse> filter =\n        clientBackupFilters.computeIfAbsent(clientID, this::backupFilter);\n\n    return filter\n        .andThen(service)\n        .apply(request);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/ClientLatencyFilter.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.clientstats.RequestCounters;\nimport com.twitter.search.common.clientstats.RequestCountersEventListener;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.filters.EarlybirdSuccessfulResponseHandler;\nimport com.twitter.util.Future;\n\npublic class ClientLatencyFilter extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n  // _client_latency_stats_for_ is intended to measure the latency of requests to services that this\n  // root depends on. This can be used to measure how long a request takes in transit between when\n  // it leaves a root and when a root receives the response, in case this latency is significantly\n  // different than Earlybird measured latency. We break it down by client, so that we can tell\n  // which customers are being hit by this latency.\n  private static final String STAT_FORMAT = \"%s_client_latency_stats_for_%s\";\n\n  private final ConcurrentHashMap<String, RequestCounters> requestCounterForClient =\n      new ConcurrentHashMap<>();\n  private final String prefix;\n\n  public ClientLatencyFilter(String prefix) {\n    this.prefix = prefix;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(EarlybirdRequest request,\n                                         Service<EarlybirdRequest, EarlybirdResponse> service) {\n\n    RequestCounters requestCounters = requestCounterForClient.computeIfAbsent(\n        ClientIdUtil.getClientIdFromRequest(request), client ->\n            new RequestCounters(String.format(STAT_FORMAT, prefix, client)));\n\n    RequestCountersEventListener<EarlybirdResponse> requestCountersEventListener =\n        new RequestCountersEventListener<>(requestCounters, Clock.SYSTEM_CLOCK,\n            EarlybirdSuccessfulResponseHandler.INSTANCE);\n    return service.apply(request).addEventListener(requestCountersEventListener);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdCacheCommonModule.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport com.google.inject.Provides;\n\nimport com.twitter.finagle.memcached.JavaClient;\nimport com.twitter.finagle.mtls.authentication.ServiceIdentifier;\nimport com.twitter.finagle.stats.StatsReceiver;\nimport com.twitter.inject.TwitterModule;\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.EarlybirdCacheSerializer;\nimport com.twitter.search.common.caching.SearchCacheBuilder;\nimport com.twitter.search.common.caching.SearchMemcacheClientConfig;\nimport com.twitter.search.common.caching.SearchMemcacheClientFactory;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.caching.CacheCommonUtil;\nimport com.twitter.search.earlybird_root.caching.CacheStats;\nimport com.twitter.search.earlybird_root.caching.DefaultForcedCacheMissDecider;\nimport com.twitter.search.earlybird_root.filters.PostCacheRequestTypeCountFilter;\nimport com.twitter.util.Duration;\n\n/**\n * Provides common bindings for cache related modules.\n */\npublic class EarlybirdCacheCommonModule extends TwitterModule {\n  private static final String CACHE_VERSION = \"1\";\n\n  @Override\n  public void configure() {\n    bind(PostCacheRequestTypeCountFilter.class).in(Singleton.class);\n    bind(DefaultForcedCacheMissDecider.class).in(Singleton.class);\n  }\n\n  @Provides\n  @Singleton\n  @Named(CacheCommonUtil.NAMED_MAX_CACHE_RESULTS)\n  Integer provideMaxCacheResults() {\n    return 100;\n  }\n\n  @Provides\n  @Singleton\n  JavaClient provideMemCacheClient(\n      StatsReceiver statsReceiver, ServiceIdentifier serviceIdentifier) {\n    SearchMemcacheClientConfig config = new SearchMemcacheClientConfig();\n    config.connectTimeoutMs = Duration.fromMilliseconds(100);\n    config.requestTimeoutMs = Duration.fromMilliseconds(100);\n    config.failureAccrualFailuresNumber = 150;\n    config.failureAccrualFailuresDurationMillis = 30000;\n    config.failureAccrualDuration = Duration.fromMilliseconds(60000);\n\n    return SearchMemcacheClientFactory.createMtlsClient(\n        \"\",\n        \"earlybird_root\",\n        statsReceiver,\n        config,\n        serviceIdentifier\n    );\n  }\n\n  /**\n   * Create a new Earlybird cache.\n   *\n   * @param client the memcache client to use.\n   * @param decider the decider to use for the cache.\n   * @param cachePrefix the common cache prefix for the cache type.\n   * @param serializedKeyPrefix the common cache prefix for the cluster.\n   * @param cacheExpiryMillis cache entry ttl in milliseconds.\n   */\n  static Cache<EarlybirdRequest, EarlybirdResponse> createCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      String cachePrefix,\n      String serializedKeyPrefix,\n      long cacheExpiryMillis,\n      int cacheKeyMaxBytes,\n      int cacheValueMaxBytes) {\n    return new SearchCacheBuilder<EarlybirdRequest, EarlybirdResponse>(\n        CACHE_VERSION,\n        client,\n        cachePrefix,\n        serializedKeyPrefix,\n        cacheExpiryMillis)\n        .withMaxKeyBytes(cacheKeyMaxBytes)\n        .withMaxValueBytes(cacheValueMaxBytes)\n        .withRequestTimeoutCounter(CacheStats.REQUEST_TIMEOUT_COUNTER)\n        .withRequestFailedCounter(CacheStats.REQUEST_FAILED_COUNTER)\n        .withCacheSerializer(new EarlybirdCacheSerializer())\n        .withForceCacheMissDecider(decider)\n        .withInProcessCache()\n        .build();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdChainedScatterGatherService.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.List;\n\nimport javax.inject.Inject;\n\nimport com.google.common.collect.Lists;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.search.common.root.PartitionLoggingSupport;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.util.Future;\n\n/**\n * A chain of scatter gather services.\n * Regular roots use ScatterGatherService directly. This class is only used by multi-tier roots.\n */\npublic class EarlybirdChainedScatterGatherService extends\n    Service<EarlybirdRequestContext, List<Future<EarlybirdResponse>>> {\n\n  private static final Logger LOG =\n    LoggerFactory.getLogger(EarlybirdChainedScatterGatherService.class);\n\n  private final List<Service<EarlybirdRequestContext, EarlybirdResponse>> serviceChain;\n\n  /**\n   * Construct a ScatterGatherServiceChain, by loading configurations from earlybird-tiers.yml.\n   */\n  @Inject\n  public EarlybirdChainedScatterGatherService(\n      EarlybirdServiceChainBuilder serviceChainBuilder,\n      EarlybirdServiceScatterGatherSupport scatterGatherSupport,\n      PartitionLoggingSupport<EarlybirdRequestContext> partitionLoggingSupport) {\n\n    serviceChain =\n        serviceChainBuilder.buildServiceChain(scatterGatherSupport, partitionLoggingSupport);\n\n    if (serviceChain.isEmpty()) {\n      LOG.error(\"At least one tier has to be enabled.\");\n      throw new RuntimeException(\"Root does not work with all tiers disabled.\");\n    }\n  }\n\n  @Override\n  public Future<List<Future<EarlybirdResponse>>> apply(EarlybirdRequestContext requestContext) {\n    // Hit all tiers in parallel.\n    List<Future<EarlybirdResponse>> resultList =\n        Lists.newArrayListWithCapacity(serviceChain.size());\n    for (final Service<EarlybirdRequestContext, EarlybirdResponse> service : serviceChain) {\n      resultList.add(service.apply(requestContext));\n    }\n    return Future.value(resultList);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdCommonModule.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.annotation.Nullable;\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport scala.PartialFunction;\n\nimport com.google.inject.Provides;\n\nimport org.apache.thrift.protocol.TProtocolFactory;\n\nimport com.twitter.app.Flag;\nimport com.twitter.app.Flaggable;\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.mtls.authorization.server.MtlsServerSessionTrackerFilter;\nimport com.twitter.finagle.service.ReqRep;\nimport com.twitter.finagle.service.ResponseClass;\nimport com.twitter.finagle.stats.StatsReceiver;\nimport com.twitter.finagle.thrift.RichServerParam;\nimport com.twitter.finagle.thrift.ThriftClientRequest;\nimport com.twitter.inject.TwitterModule;\nimport com.twitter.search.common.dark.DarkProxy;\nimport com.twitter.search.common.dark.ResolverProxy;\nimport com.twitter.search.common.partitioning.zookeeper.SearchZkClient;\nimport com.twitter.search.common.root.PartitionConfig;\nimport com.twitter.search.common.root.RemoteClientBuilder;\nimport com.twitter.search.common.root.RootClientServiceBuilder;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.common.root.ServerSetsConfig;\nimport com.twitter.search.common.util.zookeeper.ZooKeeperProxy;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird_root.common.EarlybirdFeatureSchemaMerger;\nimport com.twitter.search.earlybird_root.filters.PreCacheRequestTypeCountFilter;\nimport com.twitter.search.earlybird_root.filters.QueryLangStatFilter;\n\n/**\n * Provides common bindings.\n */\npublic class EarlybirdCommonModule extends TwitterModule {\n  static final String NAMED_ALT_CLIENT = \"alt_client\";\n  static final String NAMED_EXP_CLUSTER_CLIENT = \"exp_cluster_client\";\n\n  private final Flag<String> altZkRoleFlag = createFlag(\n      \"alt_zk_role\",\n      \"\",\n      \"The alternative ZooKeeper role\",\n      Flaggable.ofString());\n  private final Flag<String> altZkClientEnvFlag = createFlag(\n      \"alt_zk_client_env\",\n      \"\",\n      \"The alternative zk client environment\",\n      Flaggable.ofString());\n  private final Flag<String> altPartitionZkPathFlag = createFlag(\n      \"alt_partition_zk_path\",\n      \"\",\n      \"The alternative client partition zk path\",\n      Flaggable.ofString());\n\n  @Override\n  public void configure() {\n    bind(InitializeFilter.class).in(Singleton.class);\n    bind(PreCacheRequestTypeCountFilter.class).in(Singleton.class);\n\n    bind(Clock.class).toInstance(Clock.SYSTEM_CLOCK);\n    bind(QueryLangStatFilter.Config.class).toInstance(new QueryLangStatFilter.Config(100));\n  }\n\n  // Used in SearchRootModule.\n  @Provides\n  @Singleton\n  PartialFunction<ReqRep, ResponseClass> provideResponseClassifier() {\n    return new RootResponseClassifier();\n  }\n\n  @Provides\n  @Singleton\n  Service<byte[], byte[]> providesByteService(\n      EarlybirdService.ServiceIface svc,\n      DarkProxy<ThriftClientRequest, byte[]> darkProxy,\n      TProtocolFactory protocolFactory) {\n    return darkProxy.toFilter().andThen(\n        new EarlybirdService.Service(\n            svc, new RichServerParam(protocolFactory, SearchRootModule.SCROOGE_BUFFER_SIZE)));\n  }\n\n  @Provides\n  @Singleton\n  @Named(SearchRootModule.NAMED_SERVICE_INTERFACE)\n  Class providesServiceInterface() {\n    return EarlybirdService.ServiceIface.class;\n  }\n\n  @Provides\n  @Singleton\n  ZooKeeperProxy provideZookeeperClient() {\n    return SearchZkClient.getSZooKeeperClient();\n  }\n\n  @Provides\n  @Singleton\n  EarlybirdFeatureSchemaMerger provideFeatureSchemaMerger() {\n    return new EarlybirdFeatureSchemaMerger();\n  }\n\n  @Provides\n  @Singleton\n  @Nullable\n  @Named(NAMED_ALT_CLIENT)\n  ServerSetsConfig provideAltServerSetsConfig() {\n    if (!altZkRoleFlag.isDefined() || !altZkClientEnvFlag.isDefined()) {\n      return null;\n    }\n\n    return new ServerSetsConfig(altZkRoleFlag.apply(), altZkClientEnvFlag.apply());\n  }\n\n  @Provides\n  @Singleton\n  @Nullable\n  @Named(NAMED_ALT_CLIENT)\n  PartitionConfig provideAltPartitionConfig(PartitionConfig defaultPartitionConfig) {\n    if (!altPartitionZkPathFlag.isDefined()) {\n      return null;\n    }\n\n    return new PartitionConfig(\n        defaultPartitionConfig.getNumPartitions(), altPartitionZkPathFlag.apply());\n  }\n\n  @Provides\n  @Singleton\n  @Nullable\n  @Named(NAMED_ALT_CLIENT)\n  RootClientServiceBuilder<EarlybirdService.ServiceIface> provideAltRootClientServiceBuilder(\n      @Named(NAMED_ALT_CLIENT) @Nullable ServerSetsConfig serverSetsConfig,\n      @Named(SearchRootModule.NAMED_SERVICE_INTERFACE) Class serviceIface,\n      ResolverProxy resolverProxy,\n      RemoteClientBuilder<EarlybirdService.ServiceIface> remoteClientBuilder) {\n    if (serverSetsConfig == null) {\n      return null;\n    }\n\n    return new RootClientServiceBuilder<>(\n        serverSetsConfig, serviceIface, resolverProxy, remoteClientBuilder);\n  }\n\n  @Provides\n  @Singleton\n  @Named(NAMED_EXP_CLUSTER_CLIENT)\n  RootClientServiceBuilder<EarlybirdService.ServiceIface> provideExpClusterRootClientServiceBuilder(\n      @Named(SearchRootModule.NAMED_EXP_CLUSTER_SERVER_SETS_CONFIG)\n          ServerSetsConfig serverSetsConfig,\n      @Named(SearchRootModule.NAMED_SERVICE_INTERFACE) Class serviceIface,\n      ResolverProxy resolverProxy,\n      RemoteClientBuilder<EarlybirdService.ServiceIface> remoteClientBuilder) {\n    return new RootClientServiceBuilder<>(\n        serverSetsConfig, serviceIface, resolverProxy, remoteClientBuilder);\n  }\n\n  @Provides\n  @Singleton\n  MtlsServerSessionTrackerFilter<EarlybirdRequest, EarlybirdResponse>\n  provideMtlsServerSessionTrackerFilter(StatsReceiver statsReceiver) {\n    return new MtlsServerSessionTrackerFilter<>(statsReceiver);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdFullArchiveScatterGatherSupport.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\n\nimport com.twitter.search.common.partitioning.base.PartitionMappingManager;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird_root.common.EarlybirdFeatureSchemaMerger;\n\n/**\n * The EarlybirdServiceScatterGatherSupport implementation used to fan out requests to the earlybird\n * partitions in the full archive tiers.\n */\npublic class EarlybirdFullArchiveScatterGatherSupport extends EarlybirdServiceScatterGatherSupport {\n  /** Creates a new EarlybirdFullArchiveScatterGatherSupport instance. */\n  @Inject\n  EarlybirdFullArchiveScatterGatherSupport(\n      PartitionMappingManager partitionMappingManager,\n      EarlybirdFeatureSchemaMerger featureSchemaMerger) {\n    super(partitionMappingManager, EarlybirdCluster.FULL_ARCHIVE, featureSchemaMerger);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdProtectedScatterGatherSupport.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\n\nimport com.twitter.search.common.partitioning.base.PartitionMappingManager;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird_root.common.EarlybirdFeatureSchemaMerger;\n\n/**\n * The EarlybirdServiceScatterGatherSupport implementation used to fan out requests to the earlybird\n * partitions in the protected cluster.\n */\npublic class EarlybirdProtectedScatterGatherSupport extends EarlybirdServiceScatterGatherSupport {\n  /**\n   * Construct a EarlybirdProtectedScatterGatherSupport to do minUserFanOut,\n   * used only by protected. The main difference from the base class is that\n   * if the from user ID is not set, exception is thrown.\n   */\n  @Inject\n  EarlybirdProtectedScatterGatherSupport(\n      PartitionMappingManager partitionMappingManager,\n      EarlybirdFeatureSchemaMerger featureSchemaMerger) {\n    super(partitionMappingManager, EarlybirdCluster.PROTECTED, featureSchemaMerger);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdProtectedValidationBehavior.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\n\npublic class EarlybirdProtectedValidationBehavior extends EarlybirdServiceValidationBehavior {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(EarlybirdProtectedValidationBehavior.class);\n\n  @Override\n  public EarlybirdResponse getResponseIfInvalidRequest(EarlybirdRequest request) {\n    if (!request.isSetSearchQuery() || request.getSearchQuery() == null) {\n      String errorMsg = \"Invalid EarlybirdRequest, no ThriftSearchQuery specified. \" + request;\n      LOG.warn(errorMsg);\n      return createErrorResponse(errorMsg);\n    }\n    ThriftSearchQuery searchQuery = request.getSearchQuery();\n\n    // Make sure this request is valid for the protected tweets cluster.\n    if (!searchQuery.isSetFromUserIDFilter64() || searchQuery.getFromUserIDFilter64().isEmpty()) {\n      String errorMsg = \"ThriftSearchQuery.fromUserIDFilter64 not set. \" + request;\n      LOG.warn(errorMsg);\n      return createErrorResponse(errorMsg);\n    }\n\n    if (!searchQuery.isSetSearcherId()) {\n      String errorMsg = \"ThriftSearchQuery.searcherId not set. \" + request;\n      LOG.warn(errorMsg);\n      return createErrorResponse(errorMsg);\n    }\n\n    if (searchQuery.getSearcherId() < 0) {\n      String errorMsg = \"Invalid ThriftSearchQuery.searcherId: \" + searchQuery.getSearcherId()\n          + \". \" + request;\n      LOG.warn(errorMsg);\n      return createErrorResponse(errorMsg);\n    }\n\n    return super.getResponseIfInvalidRequest(request);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdProtectedWarmup.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.common.root.WarmupConfig;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\n\npublic class EarlybirdProtectedWarmup extends EarlybirdWarmup {\n\n  public EarlybirdProtectedWarmup(Clock clock, WarmupConfig config) {\n    super(clock, config);\n  }\n\n  /**\n   * The protected cluster requires all queries to specify a fromUserIdFilter and a searcherId.\n   */\n  @Override\n  protected EarlybirdRequest createRequest(int requestId) {\n    EarlybirdRequest request = super.createRequest(requestId);\n\n    Preconditions.checkState(request.isSetSearchQuery());\n    request.getSearchQuery().addToFromUserIDFilter64(requestId);\n    request.getSearchQuery().setSearcherId(0L);\n\n    return request;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdQueryRewriteFilter.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.List;\nimport java.util.Map;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Predicate;\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.Term;\nimport com.twitter.search.queryparser.query.annotation.Annotation;\nimport com.twitter.search.queryparser.rewriter.PredicateQueryNodeDropper;\nimport com.twitter.search.queryparser.visitors.TermExtractorVisitor;\nimport com.twitter.util.Future;\n\n/**\n * Filter that rewrites the serialized query on EarlybirdRequest.\n * As of now, this filter performs the following rewrites:\n *   - Drop \":v annotated variants based on decider, if the query has enough term nodes.\n */\npublic class EarlybirdQueryRewriteFilter extends\n    SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n\n  private static final Logger LOG =\n      LoggerFactory.getLogger(EarlybirdQueryRewriteFilter.class);\n\n  private static final String DROP_PHRASE_VARIANT_FROM_QUERY_DECIDER_KEY_PATTERN =\n      \"drop_variants_from_%s_%s_queries\";\n\n  // only drop variants from queries with more than this number of terms.\n  private static final String MIN_TERM_COUNT_FOR_VARIANT_DROPPING_DECIDER_KEY_PATTERN =\n      \"drop_variants_from_%s_%s_queries_term_count_threshold\";\n\n  private static final SearchCounter QUERY_PARSER_FAILURE_COUNT =\n      SearchCounter.export(\"query_rewrite_filter_query_parser_failure_count\");\n\n  // We currently add variants only to RECENCY and RELEVANCE requests, but it doesn't hurt to export\n  // stats for all request types.\n  @VisibleForTesting\n  static final Map<EarlybirdRequestType, SearchCounter> DROP_VARIANTS_QUERY_COUNTS =\n    Maps.newEnumMap(EarlybirdRequestType.class);\n  static {\n    for (EarlybirdRequestType requestType : EarlybirdRequestType.values()) {\n      DROP_VARIANTS_QUERY_COUNTS.put(\n          requestType,\n          SearchCounter.export(String.format(\"drop_%s_variants_query_count\",\n                                             requestType.getNormalizedName())));\n    }\n  }\n\n  private static final Predicate<Query> DROP_VARIANTS_PREDICATE =\n      q -> q.hasAnnotationType(Annotation.Type.VARIANT);\n\n  private static final PredicateQueryNodeDropper DROP_VARIANTS_VISITOR =\n    new PredicateQueryNodeDropper(DROP_VARIANTS_PREDICATE);\n\n  private final SearchDecider decider;\n  private final String normalizedSearchRootName;\n\n  @Inject\n  public EarlybirdQueryRewriteFilter(\n      SearchDecider decider,\n      @Named(SearchRootModule.NAMED_NORMALIZED_SEARCH_ROOT_NAME) String normalizedSearchRootName) {\n    this.decider = decider;\n    this.normalizedSearchRootName = normalizedSearchRootName;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n\n    Query query = requestContext.getParsedQuery();\n    // If there's no serialized query, no rewrite is necessary.\n    if (query == null) {\n      return service.apply(requestContext);\n    } else {\n      try {\n        Query variantsRemoved = maybeRemoveVariants(requestContext, query);\n\n        if (query == variantsRemoved) {\n          return service.apply(requestContext);\n        } else {\n          EarlybirdRequestContext clonedRequestContext =\n            EarlybirdRequestContext.copyRequestContext(requestContext, variantsRemoved);\n\n          return service.apply(clonedRequestContext);\n        }\n      } catch (QueryParserException e) {\n        // It is not clear here that the QueryParserException is the client's fault, or our fault.\n        // At this point it is most likely not the client's since we have a legitimate parsed Query\n        // from the client's request, and it's the rewriting that failed.\n        // In this case we choose to send the query as is (without the rewrite), instead of\n        // failing the entire request.\n        QUERY_PARSER_FAILURE_COUNT.increment();\n        LOG.warn(\"Failed to rewrite serialized query: \" + query.serialize(), e);\n        return service.apply(requestContext);\n      }\n    }\n  }\n\n  private Query maybeRemoveVariants(EarlybirdRequestContext requestContext, Query query)\n      throws QueryParserException {\n\n    if (shouldDropVariants(requestContext, query)) {\n      Query rewrittenQuery = DROP_VARIANTS_VISITOR.apply(query);\n      if (!query.equals(rewrittenQuery)) {\n        DROP_VARIANTS_QUERY_COUNTS.get(requestContext.getEarlybirdRequestType()).increment();\n        return rewrittenQuery;\n      }\n    }\n    return query;\n  }\n\n  private boolean shouldDropVariants(EarlybirdRequestContext requestContext, Query query)\n      throws QueryParserException {\n    TermExtractorVisitor termExtractorVisitor = new TermExtractorVisitor(false);\n    List<Term> terms = query.accept(termExtractorVisitor);\n\n    EarlybirdRequestType requestType = requestContext.getEarlybirdRequestType();\n\n    boolean shouldDropVariants = decider.isAvailable(getDropPhaseVariantDeciderKey(requestType));\n\n    return terms != null\n        && terms.size() >= decider.getAvailability(\n            getMinTermCountForVariantDroppingDeciderKey(requestType))\n        && shouldDropVariants;\n  }\n\n  private String getDropPhaseVariantDeciderKey(EarlybirdRequestType requestType) {\n    return String.format(DROP_PHRASE_VARIANT_FROM_QUERY_DECIDER_KEY_PATTERN,\n                         normalizedSearchRootName,\n                         requestType.getNormalizedName());\n  }\n\n  private String getMinTermCountForVariantDroppingDeciderKey(EarlybirdRequestType requestType) {\n    return String.format(MIN_TERM_COUNT_FOR_VARIANT_DROPPING_DECIDER_KEY_PATTERN,\n                         normalizedSearchRootName,\n                         requestType.getNormalizedName());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdRealtimeCgScatterGatherSupport.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\n\nimport com.twitter.search.common.partitioning.base.PartitionMappingManager;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird_root.common.EarlybirdFeatureSchemaMerger;\n\n/**\n * The EarlybirdServiceScatterGatherSupport implementation used to fan out requests to the earlybird\n * partitions in the realtime_cg cluster.\n */\npublic class EarlybirdRealtimeCgScatterGatherSupport extends EarlybirdServiceScatterGatherSupport {\n  /** Creates a new EarlybirdRealtimeCgScatterGatherSupport instance. */\n  @Inject\n  EarlybirdRealtimeCgScatterGatherSupport(\n      PartitionMappingManager partitionMappingManager,\n      EarlybirdFeatureSchemaMerger featureSchemaMerger) {\n    super(partitionMappingManager, EarlybirdCluster.REALTIME_CG, featureSchemaMerger);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdRealtimeScatterGatherSupport.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\n\nimport com.twitter.search.common.partitioning.base.PartitionMappingManager;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird_root.common.EarlybirdFeatureSchemaMerger;\n\n/**\n * The EarlybirdServiceScatterGatherSupport implementation used to fan out requests to the earlybird\n * partitions in the realtime cluster.\n */\npublic class EarlybirdRealtimeScatterGatherSupport extends EarlybirdServiceScatterGatherSupport {\n  /** Creates a new EarlybirdRealtimeScatterGatherSupport instance. */\n  @Inject\n  EarlybirdRealtimeScatterGatherSupport(\n      PartitionMappingManager partitionMappingManager,\n      EarlybirdFeatureSchemaMerger featureSchemaMerger) {\n    super(partitionMappingManager, EarlybirdCluster.REALTIME, featureSchemaMerger);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdRootQueryUtils.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.Map;\n\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.partitioning.base.PartitionMappingManager;\nimport com.twitter.search.earlybird_root.visitors.MultiTermDisjunctionPerPartitionVisitor;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\n\npublic final class EarlybirdRootQueryUtils {\n\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdRootQueryUtils.class);\n\n  private EarlybirdRootQueryUtils() {\n  }\n\n  /**\n   * Rewrite 'multi_term_disjunction from_user_id' or 'multi_term_disjunction id' based on partition\n   * for USER_ID/TWEET_ID partitioned cluster\n   * @return a map with partition id as key and rewritten query as value.\n   * If there is no 'multi_term_disjunction from_user_id/id' in query, the map will be empty; if all\n   * ids are truncated for a partition, it will add a NO_MATCH_CONJUNCTION here.\n   */\n  public static Map<Integer, Query> rewriteMultiTermDisjunctionPerPartitionFilter(\n      Query query, PartitionMappingManager partitionMappingManager, int numPartitions) {\n    Map<Integer, Query> m = Maps.newHashMap();\n    // If there is no parsed query, just return\n    if (query == null) {\n      return m;\n    }\n    for (int i = 0; i < numPartitions; ++i) {\n      MultiTermDisjunctionPerPartitionVisitor visitor =\n          new MultiTermDisjunctionPerPartitionVisitor(partitionMappingManager, i);\n      try {\n        Query q = query.accept(visitor);\n        if (q != null && q != query) {\n          m.put(i, q);\n        }\n      } catch (QueryParserException e) {\n        // Should not happen, put and log error here just in case\n        m.put(i, query);\n        LOG.error(\n            \"MultiTermDisjuctionPerPartitionVisitor cannot process query: \" + query.serialize());\n      }\n    }\n    return m;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdServiceChainBuilder.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.SortedSet;\nimport java.util.TreeSet;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.finagle.stats.StatsReceiver;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.root.PartitionConfig;\nimport com.twitter.search.common.root.PartitionLoggingSupport;\nimport com.twitter.search.common.root.RequestSuccessStats;\nimport com.twitter.search.common.root.RootClientServiceBuilder;\nimport com.twitter.search.common.root.ScatterGatherService;\nimport com.twitter.search.common.root.ScatterGatherSupport;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.config.TierConfig;\nimport com.twitter.search.earlybird.config.TierInfo;\nimport com.twitter.search.earlybird.config.TierInfoSource;\nimport com.twitter.search.earlybird.config.TierInfoUtil;\nimport com.twitter.search.earlybird.config.TierInfoWrapper;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.EarlybirdService.ServiceIface;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.filters.EarlybirdTimeRangeFilter;\nimport com.twitter.search.earlybird_root.filters.RequestContextToEarlybirdRequestFilter;\nimport com.twitter.util.Function;\nimport com.twitter.util.Future;\n\n@Singleton\npublic class EarlybirdServiceChainBuilder {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdServiceChainBuilder.class);\n\n  private static final String SEARCH_METHOD_NAME = \"search\";\n\n  private static final EarlybirdResponse TIER_SKIPPED_RESPONSE =\n      new EarlybirdResponse(EarlybirdResponseCode.TIER_SKIPPED, 0)\n          .setSearchResults(new ThriftSearchResults())\n          .setDebugString(\"Request to cluster dropped by decider, or sent as dark read.\");\n\n  private final EarlybirdTierThrottleDeciders tierThrottleDeciders;\n\n  private final RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter;\n\n  private final SearchDecider decider;\n  private final String normalizedSearchRootName;\n  private final RootClientServiceBuilder<ServiceIface> clientServiceBuilder;\n  private final String partitionPath;\n  private final int numPartitions;\n  private final SortedSet<TierInfo> tierInfos;\n  private final PartitionAccessController partitionAccessController;\n  private final StatsReceiver statsReceiver;\n\n  /**\n   * Construct a ScatterGatherServiceChain, by loading configurations from earlybird-tiers.yml.\n   */\n  @Inject\n  public EarlybirdServiceChainBuilder(\n      PartitionConfig partitionConfig,\n      RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter,\n      EarlybirdTierThrottleDeciders tierThrottleDeciders,\n      @Named(SearchRootModule.NAMED_NORMALIZED_SEARCH_ROOT_NAME) String normalizedSearchRootName,\n      SearchDecider decider,\n      TierInfoSource tierConfig,\n      RootClientServiceBuilder<ServiceIface> clientServiceBuilder,\n      PartitionAccessController partitionAccessController,\n      StatsReceiver statsReceiver) {\n    this.partitionAccessController = partitionAccessController;\n    this.tierThrottleDeciders = Preconditions.checkNotNull(tierThrottleDeciders);\n    this.requestContextToEarlybirdRequestFilter = requestContextToEarlybirdRequestFilter;\n    this.normalizedSearchRootName = normalizedSearchRootName;\n    this.decider = decider;\n    this.statsReceiver = statsReceiver;\n\n    List<TierInfo> tierInformation = tierConfig.getTierInformation();\n    if (tierInformation == null || tierInformation.isEmpty()) {\n      LOG.error(\n          \"No tier found in config file {} Did you set SEARCH_ENV correctly?\",\n          tierConfig.getConfigFileType());\n      throw new RuntimeException(\"No tier found in tier config file.\");\n    }\n\n    // Get the tier info from the tier config yml file\n    TreeSet<TierInfo> infos = new TreeSet<>(TierInfoUtil.TIER_COMPARATOR);\n    infos.addAll(tierInformation);\n    this.tierInfos = Collections.unmodifiableSortedSet(infos);\n    this.clientServiceBuilder = clientServiceBuilder;\n    this.partitionPath = partitionConfig.getPartitionPath();\n    this.numPartitions = partitionConfig.getNumPartitions();\n\n    LOG.info(\"Found the following tiers from config: {}\", tierInfos);\n  }\n\n  /** Builds the chain of services that should be queried on each request. */\n  public List<Service<EarlybirdRequestContext, EarlybirdResponse>> buildServiceChain(\n      ScatterGatherSupport<EarlybirdRequestContext, EarlybirdResponse> support,\n      PartitionLoggingSupport<EarlybirdRequestContext> partitionLoggingSupport) {\n    // Make sure the tier serving ranges do not overlap and do not have gaps.\n    TierInfoUtil.checkTierServingRanges(tierInfos);\n\n    List<Service<EarlybirdRequestContext, EarlybirdResponse>> chain = Lists.newArrayList();\n\n    for (TierInfo tierInfo : tierInfos) {\n      String tierName = tierInfo.getTierName();\n      if (tierInfo.isEnabled()) {\n        String rewrittenPartitionPath = partitionPath;\n        // This rewriting rule must match the rewriting rule inside\n        // EarlybirdServer#joinServerSet().\n        if (!TierConfig.DEFAULT_TIER_NAME.equals(tierName)) {\n          rewrittenPartitionPath = partitionPath + \"/\" + tierName;\n        }\n\n        clientServiceBuilder.initializeWithPathSuffix(\n            tierInfo.getTierName(),\n            numPartitions,\n            rewrittenPartitionPath);\n\n        try {\n          chain.add(createTierService(\n                        support, tierInfo, clientServiceBuilder, partitionLoggingSupport));\n        } catch (Exception e) {\n          LOG.error(\"Failed to build clients for tier: {}\", tierInfo.getTierName());\n          throw new RuntimeException(e);\n        }\n\n      } else {\n        LOG.info(\"Skipped disabled tier: {}\", tierName);\n      }\n    }\n\n    return chain;\n  }\n\n  private Service<EarlybirdRequestContext, EarlybirdResponse> createTierService(\n      ScatterGatherSupport<EarlybirdRequestContext, EarlybirdResponse> support,\n      final TierInfo tierInfo,\n      RootClientServiceBuilder<ServiceIface> builder,\n      PartitionLoggingSupport<EarlybirdRequestContext> partitionLoggingSupport) {\n\n    final String tierName = tierInfo.getTierName();\n    RequestSuccessStats stats = new RequestSuccessStats(tierName);\n\n    List<Service<EarlybirdRequest, EarlybirdResponse>> services =\n        builder.safeBuildServiceList(SEARCH_METHOD_NAME);\n\n    // Get the client list for this tier, and apply the degradationTrackerFilter to each response.\n    //\n    // We currently do this only for the EarlybirdSearchMultiTierAdaptor (the full archive cluster).\n    // If we want to do this for all clusters (or if we want to apply any other filter to all\n    // earlybird responses, for other clusters), we should change ScatterGatherService's constructor\n    // to take in a filter, and apply it there.\n    ClientBackupFilter backupFilter = new ClientBackupFilter(\n        \"root_\" + EarlybirdCluster.FULL_ARCHIVE.getNameForStats(),\n        tierName,\n        statsReceiver,\n        decider);\n    List<Service<EarlybirdRequestContext, EarlybirdResponse>> clients = Lists.newArrayList();\n    ClientLatencyFilter latencyFilter = new ClientLatencyFilter(tierName);\n    for (Service<EarlybirdRequest, EarlybirdResponse> client : services) {\n        clients.add(requestContextToEarlybirdRequestFilter\n            .andThen(backupFilter)\n            .andThen(latencyFilter)\n            .andThen(client));\n    }\n\n    clients = SkipPartitionFilter.wrapServices(tierName, clients, partitionAccessController);\n\n    // Build the scatter gather service for this tier.\n    // Each tier has their own stats.\n    ScatterGatherService<EarlybirdRequestContext, EarlybirdResponse> scatterGatherService =\n        new ScatterGatherService<>(\n            support, clients, stats, partitionLoggingSupport);\n\n    SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> tierThrottleFilter =\n        getTierThrottleFilter(tierInfo, tierName);\n\n    EarlybirdTimeRangeFilter timeRangeFilter =\n        EarlybirdTimeRangeFilter.newTimeRangeFilterWithQueryRewriter(\n            (requestContext, userOverride) -> new TierInfoWrapper(tierInfo, userOverride),\n            decider);\n\n    return tierThrottleFilter\n        .andThen(timeRangeFilter)\n        .andThen(scatterGatherService);\n  }\n\n  private SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> getTierThrottleFilter(\n      final TierInfo tierInfo,\n      final String tierName) {\n\n    // A filter that throttles request rate.\n    final String tierThrottleDeciderKey = tierThrottleDeciders.getTierThrottleDeciderKey(\n        normalizedSearchRootName, tierName);\n\n    SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> tierThrottleFilter =\n        new SimpleFilter<EarlybirdRequestContext, EarlybirdResponse>() {\n          private final Map<TierInfo.RequestReadType, SearchCounter> readCounts =\n              getReadCountsMap();\n\n          private Map<TierInfo.RequestReadType, SearchCounter> getReadCountsMap() {\n            Map<TierInfo.RequestReadType, SearchCounter> readCountsMap =\n                Maps.newEnumMap(TierInfo.RequestReadType.class);\n            for (TierInfo.RequestReadType readType : TierInfo.RequestReadType.values()) {\n              readCountsMap.put(readType,\n                  SearchCounter.export(\"earlybird_tier_\" + tierName + \"_\"\n                      + readType.name().toLowerCase() + \"_read_count\"));\n            }\n            return Collections.unmodifiableMap(readCountsMap);\n          }\n\n          private final SearchCounter tierRequestDroppedByDeciderCount =\n              SearchCounter.export(\"earlybird_tier_\" + tierName\n                  + \"_request_dropped_by_decider_count\");\n\n          @Override\n          public Future<EarlybirdResponse> apply(\n              EarlybirdRequestContext requestContext,\n              Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n\n            // a blank response is returned when a request is dropped by decider, or\n            // a request is sent as a dark read.\n            final Future<EarlybirdResponse> blankTierResponse = Future.value(TIER_SKIPPED_RESPONSE);\n            if (tierThrottleDeciders.shouldSendRequestToTier(tierThrottleDeciderKey)) {\n              TierInfoWrapper tierInfoWrapper =\n                  new TierInfoWrapper(tierInfo, requestContext.useOverrideTierConfig());\n\n              TierInfo.RequestReadType readType = tierInfoWrapper.getReadType();\n              readCounts.get(readType).increment();\n              switch (readType) {\n                case DARK:\n                  // dark read: call backend but do not wait for results\n                  service.apply(requestContext);\n                  return blankTierResponse;\n                case GREY:\n                  // grey read: call backend, wait for results, but discard results.\n                  return service.apply(requestContext).flatMap(\n                      new Function<EarlybirdResponse, Future<EarlybirdResponse>>() {\n                        @Override\n                        public Future<EarlybirdResponse> apply(EarlybirdResponse v1) {\n                          // No matter what's returned, always return blankTierResponse.\n                          return blankTierResponse;\n                        }\n                      });\n                case LIGHT:\n                  // light read: return the future from the backend service.\n                  return service.apply(requestContext);\n                default:\n                  throw new RuntimeException(\"Unknown read type: \" + readType);\n              }\n            } else {\n              // Request is dropped by throttle decider\n              tierRequestDroppedByDeciderCount.increment();\n              return blankTierResponse;\n            }\n          }\n        };\n    return tierThrottleFilter;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdServiceLoggingSupport.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.concurrent.TimeUnit;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.Timer;\nimport com.twitter.search.common.root.LoggingSupport;\nimport com.twitter.search.earlybird.common.EarlybirdRequestPostLogger;\nimport com.twitter.search.earlybird.common.EarlybirdRequestPreLogger;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\n\npublic class EarlybirdServiceLoggingSupport extends\n    LoggingSupport.DefaultLoggingSupport<EarlybirdRequest, EarlybirdResponse> {\n  private static final int LATENCY_WARN_THRESHOLD_MS = 100;\n\n  private static final Timer DUMMY_TIMER;\n\n  private final EarlybirdRequestPreLogger requestPreLogger;\n  private final EarlybirdRequestPostLogger requestPostLogger;\n\n\n  static {\n    DUMMY_TIMER = new Timer(TimeUnit.MILLISECONDS);\n    DUMMY_TIMER.stop();\n  }\n\n  public EarlybirdServiceLoggingSupport(SearchDecider decider) {\n    requestPreLogger = EarlybirdRequestPreLogger.buildForRoot(decider.getDecider());\n    requestPostLogger = EarlybirdRequestPostLogger.buildForRoot(LATENCY_WARN_THRESHOLD_MS,\n                                                                decider.getDecider());\n  }\n\n  @Override\n  public void prelogRequest(EarlybirdRequest req) {\n    requestPreLogger.logRequest(req);\n  }\n\n  @Override\n  public void postLogRequest(\n      EarlybirdRequest request,\n      EarlybirdResponse response,\n      long latencyNanos) {\n\n    Preconditions.checkNotNull(request);\n    Preconditions.checkNotNull(response);\n\n    response.setResponseTimeMicros(TimeUnit.NANOSECONDS.toMicros(latencyNanos));\n    response.setResponseTime(TimeUnit.NANOSECONDS.toMillis(latencyNanos));\n\n    requestPostLogger.logRequest(request, response, DUMMY_TIMER);\n  }\n\n  @Override\n  public void logExceptions(EarlybirdRequest req, Throwable t) {\n    ExceptionHandler.logException(req, t);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdServicePartitionLoggingSupport.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.Map;\nimport java.util.Random;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.root.PartitionLoggingSupport;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class EarlybirdServicePartitionLoggingSupport\n    extends PartitionLoggingSupport.DefaultPartitionLoggingSupport<EarlybirdRequestContext> {\n  private static final Logger PARTITION_LOG = LoggerFactory.getLogger(\"partitionLogger\");\n\n  private static final long LATENCY_LOG_PARTITIONS_THRESHOLD_MS = 500;\n  private static final double FRACTION_OF_REQUESTS_TO_LOG = 1.0 / 500.0;\n\n  private final Random random = new Random();\n\n  @Override\n  public void logPartitionLatencies(EarlybirdRequestContext requestContext,\n                                    String tierName,\n                                    Map<Integer, Long> partitionLatenciesMicros,\n                                    long latencyMs) {\n    String logReason = null;\n\n    if (random.nextFloat() <= FRACTION_OF_REQUESTS_TO_LOG) {\n      logReason = \"randomSample\";\n    } else if (latencyMs > LATENCY_LOG_PARTITIONS_THRESHOLD_MS) {\n      logReason = \"slow\";\n    }\n\n    EarlybirdRequest request = requestContext.getRequest();\n    if (logReason != null && request.isSetSearchQuery()) {\n      PARTITION_LOG.info(\"{};{};{};{};{};{}\", tierName, logReason, latencyMs,\n          partitionLatenciesMicros, request.getClientRequestID(),\n          request.getSearchQuery().getSerializedQuery());\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdServiceScatterGatherSupport.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Map;\nimport javax.inject.Inject;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\nimport com.google.common.collect.Sets;\n\nimport com.twitter.search.common.partitioning.base.PartitionDataType;\nimport com.twitter.search.common.partitioning.base.PartitionMappingManager;\nimport com.twitter.search.common.root.ScatterGatherSupport;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.util.earlybird.EarlybirdResponseUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird_root.common.EarlybirdFeatureSchemaMerger;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.mergers.EarlybirdResponseMerger;\nimport com.twitter.search.earlybird_root.mergers.PartitionResponseAccumulator;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.util.Future;\n\nimport static com.twitter.search.earlybird_root.visitors.MultiTermDisjunctionPerPartitionVisitor.NO_MATCH_CONJUNCTION;\n\npublic class EarlybirdServiceScatterGatherSupport\n    implements ScatterGatherSupport<EarlybirdRequestContext, EarlybirdResponse> {\n\n  private static final EarlybirdResponse EMPTY_RESPONSE = newEmptyResponse();\n\n  private final PartitionMappingManager partitionMappingManager;\n  private final EarlybirdCluster cluster;\n  private final EarlybirdFeatureSchemaMerger featureSchemaMerger;\n\n  @Inject\n  protected EarlybirdServiceScatterGatherSupport(PartitionMappingManager partitionMappingManager,\n                                                 EarlybirdCluster cluster,\n                                                 EarlybirdFeatureSchemaMerger featureSchemaMerger) {\n    this.partitionMappingManager = partitionMappingManager;\n    this.cluster = cluster;\n    this.featureSchemaMerger = featureSchemaMerger;\n  }\n\n  /**\n   * Fans out the original request to all partitions.\n   */\n  private List<EarlybirdRequestContext> fanoutToAllPartitions(\n      EarlybirdRequestContext requestContext, int numPartitions) {\n    // We don't need to create a deep copy of the original requestContext for every partition,\n    // because requests are not rewritten once they get to this level: our roots have filters\n    // that rewrite the requests at the top-level, but we do not rewrite requests per-partition.\n    List<EarlybirdRequestContext> requestContexts = new ArrayList<>(numPartitions);\n    for (int i = 0; i < numPartitions; ++i) {\n      requestContexts.add(requestContext);\n    }\n    return requestContexts;\n  }\n\n  private Map<Integer, List<Long>> populateIdsForPartition(EarlybirdRequestContext requestContext) {\n    Map<Integer, List<Long>> perPartitionIds = Maps.newHashMap();\n    // Based on partition type, populate map for every partition if needed.\n    if (partitionMappingManager.getPartitionDataType() == PartitionDataType.USER_ID\n        && requestContext.getRequest().getSearchQuery().getFromUserIDFilter64Size() > 0) {\n      for (long userId : requestContext.getRequest().getSearchQuery().getFromUserIDFilter64()) {\n        int userPartition = partitionMappingManager.getPartitionIdForUserId(userId);\n        if (!perPartitionIds.containsKey(userPartition)) {\n          perPartitionIds.put(userPartition, Lists.newArrayList());\n        }\n        perPartitionIds.get(userPartition).add(userId);\n      }\n    } else if (partitionMappingManager.getPartitionDataType() == PartitionDataType.TWEET_ID\n        && requestContext.getRequest().getSearchQuery().getSearchStatusIdsSize() > 0) {\n      for (long id : requestContext.getRequest().getSearchQuery().getSearchStatusIds()) {\n        int tweetPartition = partitionMappingManager.getPartitionIdForTweetId(id);\n        if (!perPartitionIds.containsKey(tweetPartition)) {\n          perPartitionIds.put(tweetPartition, Lists.newArrayList());\n        }\n        perPartitionIds.get(tweetPartition).add(id);\n      }\n    }\n    return perPartitionIds;\n  }\n\n  private void setPerPartitionIds(EarlybirdRequest request, List<Long> ids) {\n    if (partitionMappingManager.getPartitionDataType() == PartitionDataType.USER_ID) {\n      request.getSearchQuery().setFromUserIDFilter64(ids);\n    } else {\n      request.getSearchQuery().setSearchStatusIds(Sets.newHashSet(ids));\n    }\n  }\n\n  @Override\n  public EarlybirdResponse emptyResponse() {\n    return EMPTY_RESPONSE;\n  }\n\n  public static final EarlybirdResponse newEmptyResponse() {\n    return new EarlybirdResponse(EarlybirdResponseCode.PARTITION_SKIPPED, 0)\n        .setSearchResults(new ThriftSearchResults());\n  }\n\n  @Override\n  public List<EarlybirdRequestContext> rewriteRequest(\n      EarlybirdRequestContext requestContext, int rootNumPartitions) {\n    int numPartitions = partitionMappingManager.getNumPartitions();\n    Preconditions.checkState(rootNumPartitions == numPartitions,\n        \"Root's configured numPartitions is different from that configured in database.yml.\");\n    // Rewrite query based on \"multi_term_disjunction id/from_user_id\" and partition id if needed.\n    Map<Integer, Query> perPartitionQueryMap =\n        requestContext.getRequest().getSearchQuery().getSearchStatusIdsSize() == 0\n            ? EarlybirdRootQueryUtils.rewriteMultiTermDisjunctionPerPartitionFilter(\n            requestContext.getParsedQuery(),\n            partitionMappingManager,\n            numPartitions)\n            : Maps.newHashMap();\n\n    // Key: partition Id; Value: valid ids list for this partition\n    Map<Integer, List<Long>> perPartitionIds = populateIdsForPartition(requestContext);\n\n    if (perPartitionQueryMap.isEmpty() && perPartitionIds.isEmpty()) {\n      return fanoutToAllPartitions(requestContext, numPartitions);\n    }\n\n    List<EarlybirdRequestContext> requestContexts = new ArrayList<>(numPartitions);\n    for (int i = 0; i < numPartitions; ++i) {\n      requestContexts.add(null);\n    }\n\n    // Rewrite per partition queries if exist.\n    for (int i = 0; i < numPartitions; ++i) {\n      if (perPartitionIds.containsKey(i)) {\n        if (!perPartitionQueryMap.containsKey(i)) {\n          // Query does not need to be rewritten for the partition\n          // But we still need to create a copy, because we're gonna\n          // set fromUserIDFilter64/searchStatusIds\n          requestContexts.set(i, requestContext.deepCopy());\n          setPerPartitionIds(requestContexts.get(i).getRequest(), perPartitionIds.get(i));\n        } else if (perPartitionQueryMap.get(i) != NO_MATCH_CONJUNCTION) {\n          requestContexts.set(i, EarlybirdRequestContext.copyRequestContext(\n              requestContext, perPartitionQueryMap.get(i)));\n          setPerPartitionIds(requestContexts.get(i).getRequest(), perPartitionIds.get(i));\n        }\n      } else if (perPartitionIds.isEmpty()) {\n        // The fromUserIDFilter64/searchStatusIds field is not set on the original request,\n        // perPartitionQueryMap should decide if we send a request to this partition or not\n        if (!perPartitionQueryMap.containsKey(i)) {\n          // Query does not need to be rewritten for the partition\n          // Don't need to create a copy, because request context won't be changed afterwards\n          requestContexts.set(i, requestContext);\n        } else if (perPartitionQueryMap.get(i) != NO_MATCH_CONJUNCTION) {\n          requestContexts.set(i, EarlybirdRequestContext.copyRequestContext(\n              requestContext, perPartitionQueryMap.get(i)));\n        }\n      }\n    }\n    return requestContexts;\n  }\n\n  /**\n   * Merges all the sub-results indexed by the partition id. Sub-results with value null\n   * indicate an error with that partition such as timeout etc.\n   */\n  @Override\n  public Future<EarlybirdResponse> merge(EarlybirdRequestContext requestContext,\n                                         List<Future<EarlybirdResponse>> responses) {\n    EarlybirdResponseMerger merger = EarlybirdResponseMerger.getResponseMerger(\n        requestContext,\n        responses,\n        new PartitionResponseAccumulator(),\n        cluster,\n        featureSchemaMerger,\n        partitionMappingManager.getNumPartitions());\n    return merger.merge();\n  }\n\n  @Override\n  public boolean isSuccess(EarlybirdResponse earlybirdResponse) {\n    return EarlybirdResponseUtil.isSuccessfulResponse(earlybirdResponse);\n  }\n\n  @Override\n  public boolean isTimeout(EarlybirdResponse earlybirdResponse) {\n    return earlybirdResponse.getResponseCode() == EarlybirdResponseCode.SERVER_TIMEOUT_ERROR;\n  }\n\n  @Override\n  public boolean isClientCancel(EarlybirdResponse earlybirdResponse) {\n    return earlybirdResponse.getResponseCode() == EarlybirdResponseCode.CLIENT_CANCEL_ERROR;\n  }\n\n  @Override\n  public EarlybirdResponse errorResponse(String debugString) {\n    return new EarlybirdResponse()\n        .setResponseCode(EarlybirdResponseCode.TRANSIENT_ERROR)\n        .setDebugString(debugString);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdServiceValidationBehavior.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport org.apache.thrift.TException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.root.ValidationBehavior;\nimport com.twitter.search.earlybird.common.EarlybirdRequestUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdDebugInfo;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\n\npublic class EarlybirdServiceValidationBehavior\n    extends ValidationBehavior.DefaultValidationBehavior<EarlybirdRequest, EarlybirdResponse> {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(EarlybirdServiceValidationBehavior.class);\n\n  private static final EarlybirdDebugInfo EARLYBIRD_DEBUG_INFO =\n          new EarlybirdDebugInfo().setHost(\"earlybird_root\");\n\n  private static final SearchCounter INVALID_SUCCESS_RESPONSE_THRESHOLD_TOO_LOW =\n      SearchCounter.export(\"invalid_success_response_threshold_too_low\");\n  private static final SearchCounter INVALID_SUCCESS_RESPONSE_THRESHOLD_TOO_HIGH =\n      SearchCounter.export(\"invalid_success_response_threshold_too_high\");\n\n  protected EarlybirdResponse createErrorResponse(String errorMsg) {\n    EarlybirdResponse response = new EarlybirdResponse(EarlybirdResponseCode.CLIENT_ERROR, 0);\n\n    // We're changing some ERROR logs to WARN on our side, so we want to ensure\n    // that the response contains the debug information the client needs to\n    // resolve the problem.\n    response.setDebugInfo(EARLYBIRD_DEBUG_INFO);\n    response.setDebugString(errorMsg);\n\n    return response;\n  }\n\n  @Override\n  public EarlybirdResponse getResponseIfInvalidRequest(EarlybirdRequest request) {\n    // First, fix up the query.\n    EarlybirdRequestUtil.checkAndSetCollectorParams(request);\n    EarlybirdRequestUtil.logAndFixExcessiveValues(request);\n\n    try {\n      request.validate();\n    } catch (TException e) {\n      String errorMsg = \"Invalid EarlybirdRequest. \" + request;\n      LOG.warn(errorMsg);\n      return createErrorResponse(errorMsg);\n    }\n\n    if (request.isSetSearchSegmentId() && request.getSearchSegmentId() <= 0) {\n      String errorMsg = \"Bad time slice ID: \" + request.getSearchSegmentId();\n      LOG.warn(errorMsg);\n      return createErrorResponse(errorMsg);\n    }\n\n    if (request.isSetTermStatisticsRequest()\n        && request.getTermStatisticsRequest().isSetHistogramSettings()\n        && request.getTermStatisticsRequest().getHistogramSettings().getNumBins() == 0) {\n\n      String errorMsg = \"numBins for term statistics histograms request cannot be zero: \" + request;\n      LOG.warn(errorMsg);\n      return createErrorResponse(errorMsg);\n    }\n\n    if (!request.isSetSearchQuery()\n        || request.getSearchQuery() == null) {\n      String errorMsg = \"Invalid EarlybirdRequest, no ThriftSearchQuery specified. \" + request;\n      LOG.warn(errorMsg);\n      return createErrorResponse(errorMsg);\n    }\n\n    ThriftSearchQuery searchQuery = request.getSearchQuery();\n\n    if (!searchQuery.getCollectorParams().isSetNumResultsToReturn()) {\n      String errorMsg = \"ThriftSearchQuery.numResultsToReturn not set. \" + request;\n      LOG.warn(errorMsg);\n      return createErrorResponse(errorMsg);\n    }\n\n    if (searchQuery.getCollectorParams().getNumResultsToReturn() < 0) {\n      String errorMsg = \"Invalid ThriftSearchQuery.collectorParams.numResultsToReturn: \"\n          + searchQuery.getCollectorParams().getNumResultsToReturn() + \". \" + request;\n      LOG.warn(errorMsg);\n      return createErrorResponse(errorMsg);\n    }\n\n    if (request.isSetSuccessfulResponseThreshold()) {\n      double successfulResponseThreshold = request.getSuccessfulResponseThreshold();\n      if (successfulResponseThreshold <= 0) {\n        String errorMsg = \"Success response threshold is below or equal to 0: \"\n            + successfulResponseThreshold + \" request: \" + request;\n        LOG.warn(errorMsg);\n        INVALID_SUCCESS_RESPONSE_THRESHOLD_TOO_LOW.increment();\n        return createErrorResponse(errorMsg);\n      } else if (successfulResponseThreshold > 1) {\n        String errorMsg = \"Success response threshold is above 1: \" + successfulResponseThreshold\n            + \" request: \" + request;\n        LOG.warn(errorMsg);\n        INVALID_SUCCESS_RESPONSE_THRESHOLD_TOO_HIGH.increment();\n        return createErrorResponse(errorMsg);\n      }\n    }\n\n    return null;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdTierThrottleDeciders.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\nimport javax.inject.Singleton;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.decider.SearchDecider;\n\n/**\n * Controls fractions of requests that are sent out to each tier.\n */\n@Singleton\npublic class EarlybirdTierThrottleDeciders {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(EarlybirdTierThrottleDeciders.class);\n  private static final String TIER_THROTTLE_DECIDER_KEY_FORMAT =\n      \"percentage_to_hit_cluster_%s_tier_%s\";\n  private final SearchDecider decider;\n\n  /**\n   * Construct a decider using the singleton decider object injected by Guice for the\n   * specified tier.\n   * See {@link com.twitter.search.common.root.SearchRootModule#provideDecider()}\n   */\n  @Inject\n  public EarlybirdTierThrottleDeciders(SearchDecider decider) {\n    this.decider = decider;\n  }\n\n  /**\n   * Return the throttle decider key for the specified tier.\n   */\n  public String getTierThrottleDeciderKey(String clusterName, String tierName) {\n    String deciderKey = String.format(TIER_THROTTLE_DECIDER_KEY_FORMAT, clusterName, tierName);\n    if (!decider.getDecider().feature(deciderKey).exists()) {\n      LOG.warn(\"Decider key {} not found. Will always return unavailable.\", deciderKey);\n    }\n    return deciderKey;\n  }\n\n  /**\n   * Check whether a request should be sent to the specified tier.\n   */\n  public Boolean shouldSendRequestToTier(final String tierDarkReadDeciderKey) {\n    return decider.isAvailable(tierDarkReadDeciderKey);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/EarlybirdWarmup.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport scala.runtime.AbstractFunction0;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.thrift.ClientId;\nimport com.twitter.search.common.caching.thriftjava.CachingParams;\nimport com.twitter.search.common.query.thriftjava.CollectorParams;\nimport com.twitter.search.common.ranking.thriftjava.ThriftRankingParams;\nimport com.twitter.search.common.ranking.thriftjava.ThriftScoringFunctionType;\nimport com.twitter.search.common.root.SearchRootWarmup;\nimport com.twitter.search.common.root.WarmupConfig;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchRankingMode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchRelevanceOptions;\nimport com.twitter.util.Future;\n\n/**\n * Warm-up logic for Earlybird Roots.\n * Sends 60 rounds of requests with a 1 second timeout between each round.\n * The actual number of requests sent by each round can be configured.\n */\npublic class EarlybirdWarmup extends\n    SearchRootWarmup<EarlybirdService.ServiceIface, EarlybirdRequest, EarlybirdResponse> {\n\n  private static final int WARMUP_NUM_RESULTS = 20;\n\n  private static final String CLIENT_ID = \"earlybird_root_warmup\";\n\n  public EarlybirdWarmup(Clock clock, WarmupConfig config) {\n    super(clock, config);\n  }\n\n  @Override\n  protected EarlybirdRequest createRequest(int requestId) {\n    String query = \"(* \" + \"warmup\" + requestId + \")\";\n\n    return new EarlybirdRequest()\n        .setSearchQuery(\n            new ThriftSearchQuery()\n                .setNumResults(WARMUP_NUM_RESULTS)\n                .setCollectorParams(\n                    new CollectorParams().setNumResultsToReturn(WARMUP_NUM_RESULTS))\n                .setRankingMode(ThriftSearchRankingMode.RELEVANCE)\n                .setRelevanceOptions(new ThriftSearchRelevanceOptions()\n                    .setRankingParams(new ThriftRankingParams()\n                        .setType(ThriftScoringFunctionType.LINEAR)))\n                .setSerializedQuery(query))\n        .setCachingParams(new CachingParams().setCache(false))\n        .setClientId(CLIENT_ID);\n  }\n\n  @Override\n  protected Future<EarlybirdResponse> callService(\n      final EarlybirdService.ServiceIface service,\n      final EarlybirdRequest request) {\n\n    return ClientId.apply(CLIENT_ID).asCurrent(\n        new AbstractFunction0<Future<EarlybirdResponse>>() {\n          @Override\n          public Future<EarlybirdResponse> apply() {\n            return service.search(request);\n          }\n        });\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/ExceptionHandler.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\n\npublic final class ExceptionHandler {\n  private static final Logger LOG = LoggerFactory.getLogger(ExceptionHandler.class);\n\n  private ExceptionHandler() {\n  }\n\n  public static void logException(EarlybirdRequest request, Throwable e) {\n    LOG.error(\"Exception while handling request: {}\", request, e);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/FullArchiveRootAppMain.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.Arrays;\nimport java.util.Collection;\n\nimport com.google.inject.Module;\n\nimport com.twitter.search.common.root.SearchRootAppMain;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\n\npublic class FullArchiveRootAppMain extends SearchRootAppMain<FullArchiveRootServer> {\n  /**\n   * Boilerplate for the Java-friendly AbstractTwitterServer\n   */\n  public static class Main {\n    public static void main(String[] args) {\n      new FullArchiveRootAppMain().main(args);\n    }\n  }\n\n  @Override\n  protected Collection<? extends Module> getAdditionalModules() {\n    return Arrays.asList(\n        new EarlybirdCommonModule(),\n        new EarlybirdCacheCommonModule(),\n        new FullArchiveRootModule(),\n        new QuotaModule()\n    );\n  }\n\n  @Override\n  protected Class<FullArchiveRootServer> getSearchRootServerClass() {\n    return FullArchiveRootServer.class;\n  }\n\n  @Override\n  protected Class<?> getServiceIfaceClass() {\n    return EarlybirdService.ServiceIface.class;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/FullArchiveRootModule.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.List;\nimport java.util.concurrent.TimeUnit;\n\nimport javax.annotation.Nullable;\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport com.google.inject.Key;\nimport com.google.inject.Provides;\n\nimport com.twitter.app.Flag;\nimport com.twitter.app.Flaggable;\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.memcached.JavaClient;\nimport com.twitter.finagle.stats.StatsReceiver;\nimport com.twitter.inject.TwitterModule;\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.LoggingSupport;\nimport com.twitter.search.common.root.PartitionConfig;\nimport com.twitter.search.common.root.PartitionLoggingSupport;\nimport com.twitter.search.common.root.RootClientServiceBuilder;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.common.root.SearchRootWarmup;\nimport com.twitter.search.common.root.SplitterService;\nimport com.twitter.search.common.root.ValidationBehavior;\nimport com.twitter.search.common.root.WarmupConfig;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.config.TierInfoSource;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird_root.caching.DefaultForcedCacheMissDecider;\nimport com.twitter.search.earlybird_root.caching.RecencyCache;\nimport com.twitter.search.earlybird_root.caching.RelevanceCache;\nimport com.twitter.search.earlybird_root.caching.StrictRecencyCache;\nimport com.twitter.search.earlybird_root.caching.TermStatsCache;\nimport com.twitter.search.earlybird_root.caching.TopTweetsCache;\nimport com.twitter.search.earlybird_root.caching.TopTweetsServicePostProcessor;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.filters.RequestContextToEarlybirdRequestFilter;\nimport com.twitter.util.Future;\n\nimport static com.twitter.search.earlybird_root.EarlybirdCommonModule.NAMED_ALT_CLIENT;\n\npublic class FullArchiveRootModule extends TwitterModule {\n  private static final String CLUSTER = \"archive_full\";\n  private static final String ALT_TRAFFIC_PERCENTAGE_DECIDER_KEY =\n      \"full_archive_alt_client_traffic_percentage\";\n\n  private final Flag<Boolean> forceAltClientFlag = createFlag(\n      \"force_alt_client\",\n      false,\n      \"Always sends traffic to the alt client\",\n      Flaggable.ofJavaBoolean());\n\n  @Override\n  public void configure() {\n    bind(Key.get(EarlybirdCluster.class)).toInstance(EarlybirdCluster.FULL_ARCHIVE);\n\n    bind(EarlybirdServiceScatterGatherSupport.class)\n      .to(EarlybirdFullArchiveScatterGatherSupport.class);\n\n    bind(EarlybirdService.ServiceIface.class).to(FullArchiveRootService.class);\n  }\n\n  @Provides\n  LoggingSupport<EarlybirdRequest, EarlybirdResponse> provideLoggingSupport(\n      SearchDecider decider) {\n    return new EarlybirdServiceLoggingSupport(decider);\n  }\n\n  @Provides\n  PartitionLoggingSupport<EarlybirdRequestContext> providePartitionLoggingSupport() {\n    return new EarlybirdServicePartitionLoggingSupport();\n  }\n\n  @Provides\n  ValidationBehavior<EarlybirdRequest, EarlybirdResponse> provideValidationBehavior() {\n    return new EarlybirdServiceValidationBehavior();\n  }\n\n  @Provides\n  @Singleton\n  @Nullable\n  @Named(NAMED_ALT_CLIENT)\n  EarlybirdServiceChainBuilder provideAltEarlybirdServiceChainBuilder(\n      @Named(NAMED_ALT_CLIENT) @Nullable PartitionConfig altPartitionConfig,\n      RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter,\n      EarlybirdTierThrottleDeciders tierThrottleDeciders,\n      @Named(SearchRootModule.NAMED_NORMALIZED_SEARCH_ROOT_NAME) String normalizedSearchRootName,\n      SearchDecider decider,\n      TierInfoSource tierConfig,\n      @Named(NAMED_ALT_CLIENT) @Nullable\n          RootClientServiceBuilder<EarlybirdService.ServiceIface> altRootClientServiceBuilder,\n      PartitionAccessController partitionAccessController,\n      StatsReceiver statsReceiver\n  ) {\n    if (altPartitionConfig == null || altRootClientServiceBuilder == null) {\n      return null;\n    }\n\n    return new EarlybirdServiceChainBuilder(\n        altPartitionConfig,\n        requestContextToEarlybirdRequestFilter,\n        tierThrottleDeciders,\n        normalizedSearchRootName,\n        decider,\n        tierConfig,\n        altRootClientServiceBuilder,\n        partitionAccessController,\n        statsReceiver\n    );\n  }\n\n  @Provides\n  @Singleton\n  @Nullable\n  @Named(NAMED_ALT_CLIENT)\n  EarlybirdChainedScatterGatherService provideAltEarlybirdChainedScatterGatherService(\n      @Named(NAMED_ALT_CLIENT) @Nullable EarlybirdServiceChainBuilder altServiceChainBuilder,\n      EarlybirdServiceScatterGatherSupport scatterGatherSupport,\n      PartitionLoggingSupport<EarlybirdRequestContext> partitionLoggingSupport\n  ) {\n    if (altServiceChainBuilder == null) {\n      return null;\n    }\n\n    return new EarlybirdChainedScatterGatherService(\n        altServiceChainBuilder,\n        scatterGatherSupport,\n        partitionLoggingSupport\n    );\n  }\n\n  @Provides\n  @Singleton\n  Service<EarlybirdRequestContext, List<Future<EarlybirdResponse>>>\n  provideEarlybirdChainedScatterGatherService(\n      EarlybirdChainedScatterGatherService chainedScatterGatherService,\n      @Named(NAMED_ALT_CLIENT) @Nullable\n          EarlybirdChainedScatterGatherService altChainedScatterGatherService,\n      SearchDecider decider\n  ) {\n    if (forceAltClientFlag.apply()) {\n      if (altChainedScatterGatherService == null) {\n        throw new RuntimeException(\n            \"alt client cannot be null when 'force_alt_client' is set to true\");\n      } else {\n        return altChainedScatterGatherService;\n      }\n    }\n\n    if (altChainedScatterGatherService == null) {\n      return chainedScatterGatherService;\n    }\n\n    return new SplitterService<>(\n        chainedScatterGatherService,\n        altChainedScatterGatherService,\n        decider,\n        ALT_TRAFFIC_PERCENTAGE_DECIDER_KEY\n    );\n  }\n\n  @Provides\n  @Singleton\n  @RecencyCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideRecencyCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, CLUSTER + \"_recency_root\",\n        serializedKeyPrefix, TimeUnit.HOURS.toMillis(2), cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @RelevanceCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideRelevanceCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, CLUSTER + \"_relevance_root\",\n        serializedKeyPrefix, TimeUnit.HOURS.toMillis(2), cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @StrictRecencyCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideStrictRecencyCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, CLUSTER + \"_strict_recency_root\",\n        serializedKeyPrefix, TimeUnit.HOURS.toMillis(2), cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @TermStatsCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideTermStatsCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, CLUSTER + \"_termstats_root\",\n        serializedKeyPrefix, TimeUnit.MINUTES.toMillis(5), cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @TopTweetsCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideTopTweetsCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, CLUSTER + \"_toptweets_root\",\n        serializedKeyPrefix, TopTweetsServicePostProcessor.CACHE_AGE_IN_MS,\n        cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  SearchRootWarmup<EarlybirdService.ServiceIface, ?, ?> providesSearchRootWarmup(\n      Clock clock,\n      WarmupConfig config) {\n    return new EarlybirdWarmup(clock, config);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/FullArchiveRootServer.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.search.common.root.SearchRootServer;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\n\npublic class FullArchiveRootServer extends SearchRootServer<EarlybirdService.ServiceIface> {\n\n  @Inject\n  public FullArchiveRootServer(FullArchiveRootService svc, Service<byte[], byte[]> byteSvc) {\n    super(svc, byteSvc);\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/FullArchiveRootService.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.List;\nimport javax.inject.Inject;\nimport javax.inject.Singleton;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.mtls.authorization.server.MtlsServerSessionTrackerFilter;\nimport com.twitter.search.common.clientstats.FinagleClientStatsFilter;\nimport com.twitter.search.common.root.LoggingFilter;\nimport com.twitter.search.common.root.RequestValidationFilter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusResponse;\nimport com.twitter.search.earlybird_root.caching.RecencyCacheFilter;\nimport com.twitter.search.earlybird_root.caching.RelevanceCacheFilter;\nimport com.twitter.search.earlybird_root.caching.RelevanceZeroResultsCacheFilter;\nimport com.twitter.search.earlybird_root.caching.StrictRecencyCacheFilter;\nimport com.twitter.search.earlybird_root.caching.TermStatsCacheFilter;\nimport com.twitter.search.earlybird_root.caching.TopTweetsCacheFilter;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.filters.ClientIdQueryOperatorStatsFilter;\nimport com.twitter.search.earlybird_root.filters.ClientIdQuotaFilter;\nimport com.twitter.search.earlybird_root.filters.ClientIdTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.ClientRequestTimeFilter;\nimport com.twitter.search.earlybird_root.filters.DeadlineTimeoutStatsFilter;\nimport com.twitter.search.earlybird_root.filters.EarlybirdFeatureSchemaAnnotateFilter;\nimport com.twitter.search.earlybird_root.filters.FullArchiveProtectedOperatorFilter;\nimport com.twitter.search.earlybird_root.filters.InitializeRequestContextFilter;\nimport com.twitter.search.earlybird_root.filters.IsUserProtectedMetadataTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.MetadataTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.NullcastTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.PostCacheRequestTypeCountFilter;\nimport com.twitter.search.earlybird_root.filters.PreCacheRequestTypeCountFilter;\nimport com.twitter.search.earlybird_root.filters.QueryLangStatFilter;\nimport com.twitter.search.earlybird_root.filters.QueryOperatorStatFilter;\nimport com.twitter.search.earlybird_root.filters.RequestResultStatsFilter;\nimport com.twitter.search.earlybird_root.filters.RequestSuccessStatsFilter;\nimport com.twitter.search.earlybird_root.filters.ResponseCodeStatFilter;\nimport com.twitter.search.earlybird_root.filters.ResultTierCountFilter;\nimport com.twitter.search.earlybird_root.filters.SearchPayloadSizeLocalContextFilter;\nimport com.twitter.search.earlybird_root.filters.RejectRequestsByQuerySourceFilter;\nimport com.twitter.search.earlybird_root.filters.StratoAttributionClientIdFilter;\nimport com.twitter.search.earlybird_root.filters.TopLevelExceptionHandlingFilter;\nimport com.twitter.util.Future;\n\n@Singleton\npublic class FullArchiveRootService implements EarlybirdService.ServiceIface {\n\n  private final Service<EarlybirdRequest, EarlybirdResponse> allFiltersAndService;\n\n  @Inject\n  public FullArchiveRootService(\n      TopLevelExceptionHandlingFilter topLevelExceptionHandlingFilter,\n      ResponseCodeStatFilter responseCodeStatFilter,\n      LoggingFilter<EarlybirdRequest, EarlybirdResponse> loggingFilter,\n      RequestValidationFilter<EarlybirdRequest, EarlybirdResponse> validationFilter,\n      MtlsServerSessionTrackerFilter<EarlybirdRequest, EarlybirdResponse> mtlsFilter,\n      FinagleClientStatsFilter<EarlybirdRequest, EarlybirdResponse> finagleStatsFilter,\n      InitializeFilter initializeFilter,\n      InitializeRequestContextFilter initializeRequestContextFilter,\n      QueryLangStatFilter queryLangStatFilter,\n      FullArchiveProtectedOperatorFilter protectedOperatorFilter,\n      QueryOperatorStatFilter queryOperatorStatFilter,\n      ClientIdQueryOperatorStatsFilter clientIdQueryOperatorStatsFilter,\n      IsUserProtectedMetadataTrackingFilter isUserProtectedMetadataTrackingFilter,\n      RequestResultStatsFilter requestResultStatsFilter,\n      PreCacheRequestTypeCountFilter preCacheCountFilter,\n      RecencyCacheFilter recencyCacheFilter,\n      RelevanceCacheFilter relevanceCacheFilter,\n      RelevanceZeroResultsCacheFilter relevanceZeroResultsCacheFilter,\n      StrictRecencyCacheFilter strictRecencyCacheFilter,\n      TermStatsCacheFilter termStatsCacheFilter,\n      TopTweetsCacheFilter topTweetsCacheFilter,\n      PostCacheRequestTypeCountFilter postCacheCountFilter,\n      ClientIdTrackingFilter clientIdTrackingFilter,\n      ClientIdQuotaFilter quotaFilter,\n      RejectRequestsByQuerySourceFilter rejectRequestsByQuerySourceFilter,\n      MetadataTrackingFilter metadataTrackingFilter,\n      MultiTierResultsMergeFilter multiTierResultsMergeFilter,\n      RequestSuccessStatsFilter requestSuccessStatsFilter,\n      NullcastTrackingFilter nullcastTrackingFilter,\n      ClientRequestTimeFilter clientRequestTimeFilter,\n      DeadlineTimeoutStatsFilter deadlineTimeoutStatsFilter,\n      EarlybirdFeatureSchemaAnnotateFilter featureSchemaAnnotateFilter,\n      SearchPayloadSizeLocalContextFilter searchPayloadSizeLocalContextFilter,\n      EarlybirdQueryRewriteFilter queryRewriteFilter,\n      ResultTierCountFilter resultTierCountFilter,\n      StratoAttributionClientIdFilter stratoAttributionClientIdFilter,\n      Service<EarlybirdRequestContext, List<Future<EarlybirdResponse>>> chainedScatterGatherService\n      ) {\n\n    this.allFiltersAndService =\n        loggingFilter\n            .andThen(topLevelExceptionHandlingFilter)\n            .andThen(stratoAttributionClientIdFilter)\n            .andThen(clientRequestTimeFilter)\n            .andThen(searchPayloadSizeLocalContextFilter)\n            .andThen(requestSuccessStatsFilter)\n            .andThen(requestResultStatsFilter)\n            .andThen(responseCodeStatFilter)\n            .andThen(validationFilter)\n            .andThen(mtlsFilter)\n            .andThen(finagleStatsFilter)\n            .andThen(clientIdTrackingFilter)\n            .andThen(quotaFilter)\n            .andThen(rejectRequestsByQuerySourceFilter)\n            .andThen(metadataTrackingFilter)\n            .andThen(initializeFilter)\n            .andThen(initializeRequestContextFilter)\n            .andThen(deadlineTimeoutStatsFilter)\n            .andThen(queryLangStatFilter)\n            .andThen(protectedOperatorFilter)\n            .andThen(queryOperatorStatFilter)\n            .andThen(clientIdQueryOperatorStatsFilter)\n            .andThen(isUserProtectedMetadataTrackingFilter)\n            .andThen(preCacheCountFilter)\n            .andThen(nullcastTrackingFilter)\n            .andThen(recencyCacheFilter)\n            .andThen(relevanceCacheFilter)\n            .andThen(relevanceZeroResultsCacheFilter)\n            .andThen(strictRecencyCacheFilter)\n            .andThen(termStatsCacheFilter)\n            .andThen(topTweetsCacheFilter)\n            .andThen(postCacheCountFilter)\n            .andThen(queryRewriteFilter)\n            .andThen(featureSchemaAnnotateFilter)\n            .andThen(resultTierCountFilter)\n            .andThen(multiTierResultsMergeFilter)\n            .andThen(chainedScatterGatherService);\n  }\n\n  @Override\n  public Future<String> getName() {\n    return Future.value(\"fullarchive\");\n  }\n\n  @Override\n  public Future<EarlybirdStatusResponse> getStatus() {\n    throw new UnsupportedOperationException(\"not supported\");\n  }\n\n  @Override\n  public Future<EarlybirdResponse> search(EarlybirdRequest request) {\n    return allFiltersAndService.apply(request);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/InitializeFilter.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.relevance.ranking.ActionChainManager;\nimport com.twitter.search.common.runtime.ActionChainDebugManager;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.util.Future;\nimport com.twitter.util.FutureEventListener;\n\n/**\n * Initialize request-scope state and clean them at the end.\n */\npublic class InitializeFilter extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n  @Override\n  public Future<EarlybirdResponse> apply(EarlybirdRequest request,\n                                         Service<EarlybirdRequest, EarlybirdResponse> service) {\n    ActionChainDebugManager.update(new ActionChainManager(request.getDebugMode()), \"EarlybirdRoot\");\n    return service.apply(request).addEventListener(new FutureEventListener<EarlybirdResponse>() {\n      @Override\n      public void onSuccess(EarlybirdResponse response) {\n        cleanup();\n      }\n\n      @Override\n      public void onFailure(Throwable cause) {\n        cleanup();\n      }\n    });\n  }\n\n  private void cleanup() {\n    ActionChainDebugManager.clearLocals();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/MultiTierResultsMergeFilter.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.List;\n\nimport javax.inject.Inject;\n\nimport com.twitter.finagle.Filter;\nimport com.twitter.finagle.Service;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdFeatureSchemaMerger;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.mergers.EarlybirdResponseMerger;\nimport com.twitter.search.earlybird_root.mergers.TierResponseAccumulator;\nimport com.twitter.util.Function;\nimport com.twitter.util.Future;\n\n/**\n * Filter used to merge results from multiple tiers\n */\npublic class MultiTierResultsMergeFilter extends\n    Filter<EarlybirdRequestContext, EarlybirdResponse,\n        EarlybirdRequestContext, List<Future<EarlybirdResponse>>> {\n\n  private final EarlybirdFeatureSchemaMerger featureSchemaMerger;\n\n  @Inject\n  public MultiTierResultsMergeFilter(EarlybirdFeatureSchemaMerger featureSchemaMerger) {\n    this.featureSchemaMerger = featureSchemaMerger;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      final EarlybirdRequestContext request,\n      Service<EarlybirdRequestContext, List<Future<EarlybirdResponse>>> service) {\n    return service.apply(request).flatMap(Function.func(responses -> merge(request, responses)));\n  }\n\n  private Future<EarlybirdResponse> merge(\n      EarlybirdRequestContext requestContext,\n      List<Future<EarlybirdResponse>> responses) {\n\n    // For multi-tier response merging, the number of partitions do not have meaning because\n    // the response is not uniformly partitioned anymore.  We pass Integer.MAX_VALUE for stats\n    // counting purpose.\n    EarlybirdResponseMerger merger = EarlybirdResponseMerger.getResponseMerger(\n        requestContext,\n        responses,\n        new TierResponseAccumulator(),\n        EarlybirdCluster.FULL_ARCHIVE,\n        featureSchemaMerger,\n        Integer.MAX_VALUE);\n    return merger.merge();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/PartitionAccessController.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\n\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\n/**\n * Determines if a root should send requests to certain partitions based on if they have been turned\n * off by decider.\n */\npublic class PartitionAccessController {\n  private final String clusterName;\n  private final SearchDecider decider;\n\n  @Inject\n  public PartitionAccessController(\n      @Named(SearchRootModule.NAMED_SEARCH_ROOT_NAME) String clusterName,\n      @Named(SearchRootModule.NAMED_PARTITION_DECIDER) SearchDecider partitionDecider) {\n    this.clusterName = clusterName;\n    this.decider = partitionDecider;\n  }\n\n  /**\n   * Should root send requests to a given partition\n   * Designed to be used to quickly stop hitting a partition of there are problems with it.\n   */\n  public boolean canAccessPartition(\n      String tierName, int partitionNum, String clientId, EarlybirdRequestType requestType) {\n\n    String partitionDeciderName =\n        String.format(\"cluster_%s_skip_tier_%s_partition_%s\", clusterName, tierName, partitionNum);\n    if (decider.isAvailable(partitionDeciderName)) {\n      SearchCounter.export(partitionDeciderName).increment();\n      return false;\n    }\n\n    String clientDeciderName = String.format(\"cluster_%s_skip_tier_%s_partition_%s_client_id_%s\",\n        clusterName, tierName, partitionNum, clientId);\n    if (decider.isAvailable(clientDeciderName)) {\n      SearchCounter.export(clientDeciderName).increment();\n      return false;\n    }\n\n    String requestTypeDeciderName = String.format(\n        \"cluster_%s_skip_tier_%s_partition_%s_request_type_%s\",\n        clusterName, tierName, partitionNum, requestType.getNormalizedName());\n    if (decider.isAvailable(requestTypeDeciderName)) {\n      SearchCounter.export(requestTypeDeciderName).increment();\n      return false;\n    }\n\n    String clientRequestTypeDeciderName = String.format(\n        \"cluster_%s_skip_tier_%s_partition_%s_client_id_%s_request_type_%s\",\n        clusterName, tierName, partitionNum, clientId, requestType.getNormalizedName());\n    if (decider.isAvailable(clientRequestTypeDeciderName)) {\n      SearchCounter.export(clientRequestTypeDeciderName).increment();\n      return false;\n    }\n\n    return true;\n  }\n\n  public String getClusterName() {\n    return clusterName;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/ProtectedRootAppMain.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.Arrays;\nimport java.util.Collection;\n\nimport com.google.inject.Module;\n\nimport com.twitter.search.common.root.SearchRootAppMain;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\n\npublic class ProtectedRootAppMain extends SearchRootAppMain<ProtectedRootServer> {\n  /**\n   * Boilerplate for the Java-friendly AbstractTwitterServer\n   */\n  public static class Main {\n    public static void main(String[] args) {\n      new ProtectedRootAppMain().main(args);\n    }\n  }\n\n  @Override\n  protected Collection<? extends Module> getAdditionalModules() {\n    return Arrays.asList(\n        new EarlybirdCommonModule(),\n        new EarlybirdCacheCommonModule(),\n        new ProtectedRootAppModule(),\n        new ProtectedScatterGatherModule());\n  }\n\n  @Override\n  protected Class<ProtectedRootServer> getSearchRootServerClass() {\n    return ProtectedRootServer.class;\n  }\n\n  @Override\n  protected Class<?> getServiceIfaceClass() {\n    return EarlybirdService.ServiceIface.class;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/ProtectedRootAppModule.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport com.google.inject.Key;\nimport com.google.inject.Provides;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.memcached.JavaClient;\nimport com.twitter.inject.TwitterModule;\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.LoggingSupport;\nimport com.twitter.search.common.root.PartitionLoggingSupport;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.common.root.SearchRootWarmup;\nimport com.twitter.search.common.root.ValidationBehavior;\nimport com.twitter.search.common.root.WarmupConfig;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird_root.caching.DefaultForcedCacheMissDecider;\nimport com.twitter.search.earlybird_root.caching.RecencyCache;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class ProtectedRootAppModule extends TwitterModule {\n  @Override\n  public void configure() {\n    bind(Key.get(EarlybirdCluster.class)).toInstance(EarlybirdCluster.PROTECTED);\n\n    bind(EarlybirdServiceScatterGatherSupport.class)\n        .to(EarlybirdProtectedScatterGatherSupport.class);\n\n    bind(EarlybirdService.ServiceIface.class).to(ProtectedRootService.class);\n  }\n\n  @Provides\n  @Singleton\n  LoggingSupport<EarlybirdRequest, EarlybirdResponse> provideLoggingSupport(\n      SearchDecider decider) {\n    return new EarlybirdServiceLoggingSupport(decider);\n  }\n\n  @Provides\n  @Singleton\n  PartitionLoggingSupport<EarlybirdRequestContext> providePartitionLoggingSupport() {\n    return new EarlybirdServicePartitionLoggingSupport();\n  }\n\n  @Provides\n  @Singleton\n  ValidationBehavior<EarlybirdRequest, EarlybirdResponse> providesValidation() {\n    return new EarlybirdProtectedValidationBehavior();\n  }\n\n  @Provides\n  @Singleton\n  @RecencyCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideRecencyCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule\n        .createCache(client, decider, \"realtime_protected_recency_root\", serializedKeyPrefix,\n            20000L, cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  SearchRootWarmup<EarlybirdService.ServiceIface, ?, ?> providesSearchRootWarmup(\n      Clock clock,\n      WarmupConfig config) {\n    return new EarlybirdProtectedWarmup(clock, config);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/ProtectedRootServer.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.search.common.root.SearchRootServer;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\n\npublic class ProtectedRootServer extends SearchRootServer<EarlybirdService.ServiceIface> {\n\n  @Inject\n  public ProtectedRootServer(ProtectedRootService svc, Service<byte[], byte[]> byteSvc) {\n    super(svc, byteSvc);\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/ProtectedRootService.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.mtls.authorization.server.MtlsServerSessionTrackerFilter;\nimport com.twitter.search.common.clientstats.FinagleClientStatsFilter;\nimport com.twitter.search.common.root.LoggingFilter;\nimport com.twitter.search.common.root.RequestValidationFilter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusResponse;\nimport com.twitter.search.earlybird_root.caching.RecencyCacheFilter;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.filters.ClientIdTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.ClientRequestTimeFilter;\nimport com.twitter.search.earlybird_root.filters.DeadlineTimeoutStatsFilter;\nimport com.twitter.search.earlybird_root.filters.DropAllProtectedOperatorFilter;\nimport com.twitter.search.earlybird_root.filters.EarlybirdFeatureSchemaAnnotateFilter;\nimport com.twitter.search.earlybird_root.filters.InitializeRequestContextFilter;\nimport com.twitter.search.earlybird_root.filters.MetadataTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.NullcastTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.PostCacheRequestTypeCountFilter;\nimport com.twitter.search.earlybird_root.filters.PreCacheRequestTypeCountFilter;\nimport com.twitter.search.earlybird_root.filters.QueryLangStatFilter;\nimport com.twitter.search.earlybird_root.filters.QueryOperatorStatFilter;\nimport com.twitter.search.earlybird_root.filters.RequestResultStatsFilter;\nimport com.twitter.search.earlybird_root.filters.ResponseCodeStatFilter;\nimport com.twitter.search.earlybird_root.filters.SearchPayloadSizeLocalContextFilter;\nimport com.twitter.search.earlybird_root.filters.StratoAttributionClientIdFilter;\nimport com.twitter.search.earlybird_root.filters.TopLevelExceptionHandlingFilter;\nimport com.twitter.util.Future;\n\n@Singleton\npublic class ProtectedRootService implements EarlybirdService.ServiceIface {\n\n  private final Service<EarlybirdRequest, EarlybirdResponse> allFiltersAndService;\n\n  @Inject\n  public ProtectedRootService(\n      LoggingFilter<EarlybirdRequest, EarlybirdResponse> loggingFilter,\n      RequestValidationFilter<EarlybirdRequest, EarlybirdResponse> validationFilter,\n      MtlsServerSessionTrackerFilter<EarlybirdRequest, EarlybirdResponse> mtlsFilter,\n      FinagleClientStatsFilter<EarlybirdRequest, EarlybirdResponse> finagleStatsFilter,\n      TopLevelExceptionHandlingFilter topLevelExceptionHandlingFilter,\n      ResponseCodeStatFilter responseCodeStatFilter,\n      InitializeFilter initializeFilter,\n      InitializeRequestContextFilter initializeRequestContextFilter,\n      QueryLangStatFilter queryLangStatFilter,\n      DropAllProtectedOperatorFilter dropAllProtectedOperatorFilter,\n      QueryOperatorStatFilter queryOperatorStatFilter,\n      RequestResultStatsFilter requestResultStatsFilter,\n      PreCacheRequestTypeCountFilter preCacheCountFilter,\n      RecencyCacheFilter recencyCacheFilter,\n      PostCacheRequestTypeCountFilter postCacheCountFilter,\n      ClientIdTrackingFilter clientIdTrackingFilter,\n      MetadataTrackingFilter metadataTrackingFilter,\n      NullcastTrackingFilter nullcastTrackingFilter,\n      ClientRequestTimeFilter clientRequestTimeFilter,\n      DeadlineTimeoutStatsFilter deadlineTimeoutStatsFilter,\n      EarlybirdFeatureSchemaAnnotateFilter featureSchemaAnnotateFilter,\n      SearchPayloadSizeLocalContextFilter searchPayloadSizeLocalContextFilter,\n      @Named(ProtectedScatterGatherModule.NAMED_SCATTER_GATHER_SERVICE)\n          Service<EarlybirdRequestContext, EarlybirdResponse> scatterGatherService,\n      StratoAttributionClientIdFilter stratoAttributionClientIdFilter) {\n    allFiltersAndService = loggingFilter\n        .andThen(topLevelExceptionHandlingFilter)\n        .andThen(stratoAttributionClientIdFilter)\n        .andThen(clientRequestTimeFilter)\n        .andThen(searchPayloadSizeLocalContextFilter)\n        .andThen(responseCodeStatFilter)\n        .andThen(requestResultStatsFilter)\n        .andThen(validationFilter)\n        .andThen(mtlsFilter)\n        .andThen(finagleStatsFilter)\n        .andThen(clientIdTrackingFilter)\n        .andThen(metadataTrackingFilter)\n        .andThen(initializeFilter)\n        .andThen(initializeRequestContextFilter)\n        .andThen(deadlineTimeoutStatsFilter)\n        .andThen(queryLangStatFilter)\n        .andThen(nullcastTrackingFilter)\n        .andThen(dropAllProtectedOperatorFilter)\n        .andThen(queryOperatorStatFilter)\n        .andThen(preCacheCountFilter)\n        .andThen(recencyCacheFilter)\n        .andThen(postCacheCountFilter)\n        .andThen(featureSchemaAnnotateFilter)\n        .andThen(scatterGatherService);\n  }\n\n\n  @Override\n  public Future<String> getName() {\n    return Future.value(\"protectedroot\");\n  }\n\n  @Override\n  public Future<EarlybirdStatusResponse> getStatus() {\n    throw new UnsupportedOperationException(\"not supported\");\n  }\n\n  @Override\n  public Future<EarlybirdResponse> search(EarlybirdRequest request) {\n    return allFiltersAndService.apply(request);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/ProtectedScatterGatherModule.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.annotation.Nullable;\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport com.google.inject.Provides;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.stats.StatsReceiver;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.PartitionConfig;\nimport com.twitter.search.common.root.PartitionLoggingSupport;\nimport com.twitter.search.common.root.RequestSuccessStats;\nimport com.twitter.search.common.root.RootClientServiceBuilder;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.filters.RequestContextToEarlybirdRequestFilter;\n\npublic class ProtectedScatterGatherModule extends ScatterGatherModule {\n  /**\n   * Provides the scatterGatherService for the protected cluster.\n   */\n  @Provides\n  @Singleton\n  @Named(NAMED_SCATTER_GATHER_SERVICE)\n  @Override\n  public Service<EarlybirdRequestContext, EarlybirdResponse> provideScatterGatherService(\n      EarlybirdServiceScatterGatherSupport scatterGatherSupport,\n      RequestSuccessStats requestSuccessStats,\n      PartitionLoggingSupport<EarlybirdRequestContext> partitionLoggingSupport,\n      RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter,\n      PartitionAccessController partitionAccessController,\n      PartitionConfig partitionConfig,\n      RootClientServiceBuilder<EarlybirdService.ServiceIface> rootClientServiceBuilder,\n      @Named(EarlybirdCommonModule.NAMED_EXP_CLUSTER_CLIENT)\n          RootClientServiceBuilder<EarlybirdService.ServiceIface>\n          expClusterRootClientServiceBuilder, // unused in protected roots\n      @Named(EarlybirdCommonModule.NAMED_ALT_CLIENT) @Nullable PartitionConfig altPartitionConfig,\n      @Named(EarlybirdCommonModule.NAMED_ALT_CLIENT) @Nullable\n          RootClientServiceBuilder<EarlybirdService.ServiceIface> altRootClientServiceBuilder,\n      StatsReceiver statsReceiver,\n      EarlybirdCluster cluster,\n      SearchDecider decider) {\n    return buildScatterOrSplitterService(\n        scatterGatherSupport,\n        requestSuccessStats,\n        partitionLoggingSupport,\n        requestContextToEarlybirdRequestFilter,\n        partitionAccessController,\n        partitionConfig,\n        rootClientServiceBuilder,\n        altPartitionConfig,\n        altRootClientServiceBuilder,\n        statsReceiver,\n        cluster,\n        decider\n    );\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/QuotaModule.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.concurrent.Executors;\nimport java.util.concurrent.ScheduledExecutorService;\nimport javax.annotation.Nullable;\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.util.concurrent.ThreadFactoryBuilder;\nimport com.google.common.util.concurrent.TwitterRateLimiterProxyFactory;\nimport com.google.inject.Provides;\n\nimport com.twitter.app.Flag;\nimport com.twitter.app.Flaggable;\nimport com.twitter.common.util.Clock;\nimport com.twitter.inject.TwitterModule;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.earlybird_root.filters.ClientIdArchiveAccessFilter;\nimport com.twitter.search.earlybird_root.filters.ClientIdQuotaFilter;\nimport com.twitter.search.earlybird_root.filters.DisableClientByTierFilter;\nimport com.twitter.search.earlybird_root.quota.ConfigBasedQuotaConfig;\nimport com.twitter.search.earlybird_root.quota.ConfigRepoBasedQuotaManager;\n\npublic class QuotaModule extends TwitterModule {\n  @VisibleForTesting\n  public static final String NAMED_QUOTA_CONFIG_PATH = \"quotaConfigPath\";\n  public static final String NAMED_CLIENT_QUOTA_KEY = \"clientQuotaKey\";\n  private static final String NAMED_REQUIRE_QUOTA_CONFIG_FOR_CLIENTS\n      = \"requireQuotaConfigForClients\";\n\n  private final Flag<String> quotaConfigPathFlag = createMandatoryFlag(\n      \"quota_config_path\",\n      \"\",\n      \"Path to the quota config file\",\n      Flaggable.ofString());\n\n  private final Flag<String> clientQuotaKeyFlag = createFlag(\n      \"client_quota_key\",\n      \"quota\",\n      \"The key that will be used to extract client quotas\",\n      Flaggable.ofString());\n\n  private final Flag<Boolean> requireQuotaConfigForClientsFlag = createFlag(\n      \"require_quota_config_for_clients\",\n      true,\n      \"If true, require a quota value under <client_quota_key> for each client in the config\",\n      Flaggable.ofJavaBoolean());\n\n  @Provides\n  @Singleton\n  @Named(NAMED_QUOTA_CONFIG_PATH)\n  String provideQuotaConfigPath() {\n    return quotaConfigPathFlag.apply();\n  }\n\n  @Provides\n  @Singleton\n  @Named(NAMED_CLIENT_QUOTA_KEY)\n  String provideClientQuotaKey() {\n    return clientQuotaKeyFlag.apply();\n  }\n\n  @Provides\n  @Singleton\n  @Named(NAMED_REQUIRE_QUOTA_CONFIG_FOR_CLIENTS)\n  boolean provideRequireQuotaConfigForClients() {\n    return requireQuotaConfigForClientsFlag.apply();\n  }\n\n  @Provides\n  @Singleton\n  ClientIdQuotaFilter provideConfigRepoBasedClientIdQuotaFilter(\n      ConfigRepoBasedQuotaManager configRepoBasedQuotaManager,\n      TwitterRateLimiterProxyFactory rateLimiterProxyFactory) throws Exception {\n    return new ClientIdQuotaFilter(configRepoBasedQuotaManager, rateLimiterProxyFactory);\n  }\n\n  @Provides\n  @Singleton\n  ConfigBasedQuotaConfig providesConfigBasedQuotaConfig(\n      @Nullable @Named(NAMED_QUOTA_CONFIG_PATH) String quotaConfigPath,\n      @Nullable @Named(NAMED_CLIENT_QUOTA_KEY) String clientQuotaKey,\n      @Nullable @Named(NAMED_REQUIRE_QUOTA_CONFIG_FOR_CLIENTS) boolean requireQuotaConfigForClients,\n      Clock clock\n  ) throws Exception {\n    ScheduledExecutorService executorService = Executors.newSingleThreadScheduledExecutor(\n        new ThreadFactoryBuilder()\n            .setNameFormat(\"quota-config-reloader\")\n            .setDaemon(true)\n            .build());\n    return ConfigBasedQuotaConfig.newConfigBasedQuotaConfig(\n        quotaConfigPath, clientQuotaKey, requireQuotaConfigForClients, executorService, clock);\n  }\n\n  @Provides\n  @Singleton\n  DisableClientByTierFilter provideDisableClientByTierFilter(\n      ConfigRepoBasedQuotaManager configRepoBasedQuotaManager,\n      SearchDecider searchDecider) {\n    return new DisableClientByTierFilter(configRepoBasedQuotaManager, searchDecider);\n  }\n\n  @Provides\n  @Singleton\n  ClientIdArchiveAccessFilter clientIdArchiveAccessFilter(\n      ConfigRepoBasedQuotaManager configRepoBasedQuotaManager) {\n    return new ClientIdArchiveAccessFilter(configRepoBasedQuotaManager);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/README.md",
    "content": "# Search Index (Earlybird) Root\nEarlybird Roots are fanout services that fan out requests to different Earlybird clusters or partitions. \n\n## Architecture\n![in-network](img/serving.png)\n\nSuperroot serves as the entry point to Earlybird (Search Index) service. Request coming to superroot are first fanned out to realtime (public) and protected roots in parallel and may be fanned out to the archive root if realtime and protected clusters don't return enough results.\nThe realtime, protected and archive roots fanout requests to the earlybird partitions where the index is stored and served.\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/RealtimeCgRootAppMain.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.Arrays;\nimport java.util.Collection;\n\nimport com.google.inject.Module;\n\nimport com.twitter.search.common.root.SearchRootAppMain;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\n\npublic class RealtimeCgRootAppMain extends SearchRootAppMain<RealtimeCgRootServer> {\n  /**\n   * Boilerplate for the Java-friendly AbstractTwitterServer\n   */\n  public static class Main {\n    public static void main(String[] args) {\n      new RealtimeCgRootAppMain().main(args);\n    }\n  }\n\n  @Override\n  protected Collection<? extends Module> getAdditionalModules() {\n    return Arrays.asList(\n        new EarlybirdCommonModule(),\n        new EarlybirdCacheCommonModule(),\n        new RealtimeCgRootAppModule(),\n        new RealtimeCgScatterGatherModule(),\n        new QuotaModule());\n  }\n\n  @Override\n  protected Class<RealtimeCgRootServer> getSearchRootServerClass() {\n    return RealtimeCgRootServer.class;\n  }\n\n  @Override\n  protected Class<?> getServiceIfaceClass() {\n    return EarlybirdService.ServiceIface.class;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/RealtimeCgRootAppModule.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport com.google.inject.Key;\nimport com.google.inject.Provides;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.memcached.JavaClient;\nimport com.twitter.inject.TwitterModule;\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.LoggingSupport;\nimport com.twitter.search.common.root.PartitionLoggingSupport;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.common.root.SearchRootWarmup;\nimport com.twitter.search.common.root.ValidationBehavior;\nimport com.twitter.search.common.root.WarmupConfig;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird_root.caching.DefaultForcedCacheMissDecider;\nimport com.twitter.search.earlybird_root.caching.FacetsCache;\nimport com.twitter.search.earlybird_root.caching.RecencyCache;\nimport com.twitter.search.earlybird_root.caching.RelevanceCache;\nimport com.twitter.search.earlybird_root.caching.StrictRecencyCache;\nimport com.twitter.search.earlybird_root.caching.TermStatsCache;\nimport com.twitter.search.earlybird_root.caching.TopTweetsCache;\nimport com.twitter.search.earlybird_root.caching.TopTweetsServicePostProcessor;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class RealtimeCgRootAppModule extends TwitterModule {\n  private static final long RECENCY_CACHE_TTL_MILLIS = 20000L;\n  private static final long RELEVANCE_CACHE_TTL_MILLIS = 20000L;\n  private static final long FACETS_CACHE_TTL_MILLIS = 300000L;\n  private static final long TERMSTATS_CACHE_TTL_MILLIS = 300000L;\n\n  @Override\n  public void configure() {\n    bind(Key.get(EarlybirdCluster.class)).toInstance(EarlybirdCluster.REALTIME_CG);\n\n    bind(EarlybirdServiceScatterGatherSupport.class)\n      .to(EarlybirdRealtimeCgScatterGatherSupport.class);\n\n    bind(EarlybirdService.ServiceIface.class).to(RealtimeCgRootService.class);\n  }\n\n  @Provides\n  @Singleton\n  @RecencyCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideRecencyCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, \"realtime_cg_recency_root\",\n        serializedKeyPrefix, RECENCY_CACHE_TTL_MILLIS, cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @RelevanceCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideRelevanceCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, \"realtime_cg_relevance_root\",\n        serializedKeyPrefix, RELEVANCE_CACHE_TTL_MILLIS, cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @StrictRecencyCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideStrictRecencyCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(\n        client, decider, \"realtime_cg_strict_recency_root\", serializedKeyPrefix,\n        RECENCY_CACHE_TTL_MILLIS, cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @FacetsCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideFacetsCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, \"realtime_cg_facets_root\",\n        serializedKeyPrefix, FACETS_CACHE_TTL_MILLIS, cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @TermStatsCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideTermStatsCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, \"realtime_cg_termstats_root\",\n        serializedKeyPrefix, TERMSTATS_CACHE_TTL_MILLIS, cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @TopTweetsCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideTopTweetsCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, \"realtime_cg_toptweets_root\",\n        serializedKeyPrefix, TopTweetsServicePostProcessor.CACHE_AGE_IN_MS,\n        cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  SearchRootWarmup<EarlybirdService.ServiceIface, ?, ?> providesSearchRootWarmup(\n      Clock clock,\n      WarmupConfig config) {\n    return new EarlybirdWarmup(clock, config);\n  }\n\n  @Provides\n  public LoggingSupport<EarlybirdRequest, EarlybirdResponse> provideLoggingSupport(\n      SearchDecider decider) {\n    return new EarlybirdServiceLoggingSupport(decider);\n  }\n\n  @Provides\n  public PartitionLoggingSupport<EarlybirdRequestContext> providePartitionLoggingSupport() {\n    return new EarlybirdServicePartitionLoggingSupport();\n  }\n\n  @Provides\n  public ValidationBehavior<EarlybirdRequest, EarlybirdResponse> provideValidationBehavior() {\n    return new EarlybirdServiceValidationBehavior();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/RealtimeCgRootServer.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\nimport javax.inject.Singleton;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.search.common.root.SearchRootServer;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\n\n@Singleton\npublic class RealtimeCgRootServer extends SearchRootServer<EarlybirdService.ServiceIface> {\n\n  @Inject\n  public RealtimeCgRootServer(RealtimeCgRootService svc, Service<byte[], byte[]> byteSvc) {\n    super(svc, byteSvc);\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/RealtimeCgRootService.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.mtls.authorization.server.MtlsServerSessionTrackerFilter;\nimport com.twitter.search.common.clientstats.FinagleClientStatsFilter;\nimport com.twitter.search.common.root.LoggingFilter;\nimport com.twitter.search.common.root.RequestValidationFilter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusResponse;\nimport com.twitter.search.earlybird_root.caching.FacetsCacheFilter;\nimport com.twitter.search.earlybird_root.caching.RecencyCacheFilter;\nimport com.twitter.search.earlybird_root.caching.RelevanceCacheFilter;\nimport com.twitter.search.earlybird_root.caching.RelevanceZeroResultsCacheFilter;\nimport com.twitter.search.earlybird_root.caching.StrictRecencyCacheFilter;\nimport com.twitter.search.earlybird_root.caching.TermStatsCacheFilter;\nimport com.twitter.search.earlybird_root.caching.TopTweetsCacheFilter;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.filters.ClientIdQuotaFilter;\nimport com.twitter.search.earlybird_root.filters.ClientIdTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.ClientRequestTimeFilter;\nimport com.twitter.search.earlybird_root.filters.DeadlineTimeoutStatsFilter;\nimport com.twitter.search.earlybird_root.filters.DropAllProtectedOperatorFilter;\nimport com.twitter.search.earlybird_root.filters.EarlybirdFeatureSchemaAnnotateFilter;\nimport com.twitter.search.earlybird_root.filters.InitializeRequestContextFilter;\nimport com.twitter.search.earlybird_root.filters.MetadataTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.NullcastTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.PostCacheRequestTypeCountFilter;\nimport com.twitter.search.earlybird_root.filters.PreCacheRequestTypeCountFilter;\nimport com.twitter.search.earlybird_root.filters.QueryLangStatFilter;\nimport com.twitter.search.earlybird_root.filters.QueryOperatorStatFilter;\nimport com.twitter.search.earlybird_root.filters.RequestResultStatsFilter;\nimport com.twitter.search.earlybird_root.filters.ResponseCodeStatFilter;\nimport com.twitter.search.earlybird_root.filters.SearchPayloadSizeLocalContextFilter;\nimport com.twitter.search.earlybird_root.filters.StratoAttributionClientIdFilter;\nimport com.twitter.search.earlybird_root.filters.TopLevelExceptionHandlingFilter;\nimport com.twitter.util.Future;\n\n@Singleton\npublic class RealtimeCgRootService implements EarlybirdService.ServiceIface {\n\n  private final Service<EarlybirdRequest, EarlybirdResponse> allFiltersAndService;\n\n  @Inject\n  public RealtimeCgRootService(\n      TopLevelExceptionHandlingFilter topLevelExceptionHandlingFilter,\n      ResponseCodeStatFilter responseCodeStatFilter,\n      LoggingFilter<EarlybirdRequest, EarlybirdResponse> loggingFilter,\n      RequestValidationFilter<EarlybirdRequest, EarlybirdResponse> validationFilter,\n      MtlsServerSessionTrackerFilter<EarlybirdRequest, EarlybirdResponse> mtlsFilter,\n      FinagleClientStatsFilter<EarlybirdRequest, EarlybirdResponse> finagleStatsFilter,\n      InitializeFilter initializeFilter,\n      InitializeRequestContextFilter initializeRequestContextFilter,\n      QueryLangStatFilter queryLangStatFilter,\n      DropAllProtectedOperatorFilter dropAllProtectedOperatorFilter,\n      QueryOperatorStatFilter queryOperatorStatFilter,\n      RequestResultStatsFilter requestResultStatsFilter,\n      PreCacheRequestTypeCountFilter preCacheCountFilter,\n      RecencyCacheFilter recencyCacheFilter,\n      RelevanceCacheFilter relevanceCacheFilter,\n      RelevanceZeroResultsCacheFilter relevanceZeroResultsCacheFilter,\n      StrictRecencyCacheFilter strictRecencyCacheFilter,\n      FacetsCacheFilter facetsCacheFilter,\n      TermStatsCacheFilter termStatsCacheFilter,\n      TopTweetsCacheFilter topTweetsCacheFilter,\n      PostCacheRequestTypeCountFilter postCacheCountFilter,\n      ClientIdTrackingFilter clientIdTrackingFilter,\n      ClientIdQuotaFilter quotaFilter,\n      MetadataTrackingFilter metadataTrackingFilter,\n      NullcastTrackingFilter nullcastTrackingFilter,\n      ClientRequestTimeFilter clientRequestTimeFilter,\n      DeadlineTimeoutStatsFilter deadlineTimeoutStatsFilter,\n      EarlybirdFeatureSchemaAnnotateFilter featureSchemaAnnotateFilter,\n      SearchPayloadSizeLocalContextFilter searchPayloadSizeLocalContextFilter,\n      @Named(ProtectedScatterGatherModule.NAMED_SCATTER_GATHER_SERVICE)\n          Service<EarlybirdRequestContext, EarlybirdResponse> scatterGatherService,\n      StratoAttributionClientIdFilter stratoAttributionClientIdFilter) {\n    this.allFiltersAndService =\n        loggingFilter\n            .andThen(topLevelExceptionHandlingFilter)\n            .andThen(stratoAttributionClientIdFilter)\n            .andThen(clientRequestTimeFilter)\n            .andThen(searchPayloadSizeLocalContextFilter)\n            .andThen(responseCodeStatFilter)\n            .andThen(requestResultStatsFilter)\n            .andThen(validationFilter)\n            .andThen(mtlsFilter)\n            .andThen(finagleStatsFilter)\n            .andThen(clientIdTrackingFilter)\n            .andThen(quotaFilter)\n            .andThen(metadataTrackingFilter)\n            .andThen(initializeFilter)\n            .andThen(initializeRequestContextFilter)\n            .andThen(deadlineTimeoutStatsFilter)\n            .andThen(queryLangStatFilter)\n            .andThen(nullcastTrackingFilter)\n            .andThen(dropAllProtectedOperatorFilter)\n            .andThen(queryOperatorStatFilter)\n            .andThen(preCacheCountFilter)\n            .andThen(recencyCacheFilter)\n            .andThen(relevanceCacheFilter)\n            .andThen(relevanceZeroResultsCacheFilter)\n            .andThen(strictRecencyCacheFilter)\n            .andThen(facetsCacheFilter)\n            .andThen(termStatsCacheFilter)\n            .andThen(topTweetsCacheFilter)\n            .andThen(postCacheCountFilter)\n            .andThen(featureSchemaAnnotateFilter)\n            .andThen(scatterGatherService);\n  }\n\n  @Override\n  public Future<String> getName() {\n    return Future.value(\"realtime_cg root\");\n  }\n\n  @Override\n  public Future<EarlybirdStatusResponse> getStatus() {\n    throw new UnsupportedOperationException(\"not supported\");\n  }\n\n  @Override\n  public Future<EarlybirdResponse> search(EarlybirdRequest request) {\n    return allFiltersAndService.apply(request);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/RealtimeCgScatterGatherModule.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.annotation.Nullable;\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport com.google.inject.Provides;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.stats.StatsReceiver;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.PartitionConfig;\nimport com.twitter.search.common.root.PartitionLoggingSupport;\nimport com.twitter.search.common.root.RequestSuccessStats;\nimport com.twitter.search.common.root.RootClientServiceBuilder;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.filters.RequestContextToEarlybirdRequestFilter;\n\npublic class RealtimeCgScatterGatherModule extends ScatterGatherModule {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(RealtimeCgScatterGatherModule.class);\n\n  /**\n   * Provides a scatter gather service for the realtime_cg cluster.\n   */\n  @Provides\n  @Singleton\n  @Named(NAMED_SCATTER_GATHER_SERVICE)\n  @Override\n  public Service<EarlybirdRequestContext, EarlybirdResponse> provideScatterGatherService(\n      EarlybirdServiceScatterGatherSupport scatterGatherSupport,\n      RequestSuccessStats requestSuccessStats,\n      PartitionLoggingSupport<EarlybirdRequestContext> partitionLoggingSupport,\n      RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter,\n      PartitionAccessController partitionAccessController,\n      PartitionConfig partitionConfig,\n      RootClientServiceBuilder<EarlybirdService.ServiceIface> rootClientServiceBuilder,\n      @Named(EarlybirdCommonModule.NAMED_EXP_CLUSTER_CLIENT)\n          RootClientServiceBuilder<EarlybirdService.ServiceIface>\n          expClusterRootClientServiceBuilder,\n      @Named(EarlybirdCommonModule.NAMED_ALT_CLIENT) @Nullable PartitionConfig altPartitionConfig,\n      @Named(EarlybirdCommonModule.NAMED_ALT_CLIENT) @Nullable\n          RootClientServiceBuilder<EarlybirdService.ServiceIface> altRootClientServiceBuilder,\n      StatsReceiver statsReceiver,\n      EarlybirdCluster cluster,\n      SearchDecider decider) {\n\n\n    return\n        buildScatterOrSplitterService(\n            scatterGatherSupport,\n            requestSuccessStats,\n            partitionLoggingSupport,\n            requestContextToEarlybirdRequestFilter,\n            partitionAccessController,\n            partitionConfig,\n            rootClientServiceBuilder,\n            altPartitionConfig,\n            altRootClientServiceBuilder,\n            statsReceiver,\n            cluster,\n            decider\n        );\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/RealtimeRootAppMain.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.Arrays;\nimport java.util.Collection;\n\nimport com.google.inject.Module;\n\nimport com.twitter.search.common.root.SearchRootAppMain;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\n\npublic class RealtimeRootAppMain extends SearchRootAppMain<RealtimeRootServer> {\n  /**\n   * Boilerplate for the Java-friendly AbstractTwitterServer\n   */\n  public static class Main {\n    public static void main(String[] args) {\n      new RealtimeRootAppMain().main(args);\n    }\n  }\n\n  @Override\n  protected Collection<? extends Module> getAdditionalModules() {\n    return Arrays.asList(\n        new EarlybirdCommonModule(),\n        new EarlybirdCacheCommonModule(),\n        new RealtimeRootAppModule(),\n        new RealtimeScatterGatherModule());\n  }\n\n  @Override\n  protected Class<RealtimeRootServer> getSearchRootServerClass() {\n    return RealtimeRootServer.class;\n  }\n\n  @Override\n  protected Class<?> getServiceIfaceClass() {\n    return EarlybirdService.ServiceIface.class;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/RealtimeRootAppModule.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport com.google.inject.Key;\nimport com.google.inject.Provides;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.memcached.JavaClient;\nimport com.twitter.inject.TwitterModule;\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.LoggingSupport;\nimport com.twitter.search.common.root.PartitionLoggingSupport;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.common.root.SearchRootWarmup;\nimport com.twitter.search.common.root.ValidationBehavior;\nimport com.twitter.search.common.root.WarmupConfig;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird_root.caching.DefaultForcedCacheMissDecider;\nimport com.twitter.search.earlybird_root.caching.FacetsCache;\nimport com.twitter.search.earlybird_root.caching.RecencyCache;\nimport com.twitter.search.earlybird_root.caching.RelevanceCache;\nimport com.twitter.search.earlybird_root.caching.StrictRecencyCache;\nimport com.twitter.search.earlybird_root.caching.TermStatsCache;\nimport com.twitter.search.earlybird_root.caching.TopTweetsCache;\nimport com.twitter.search.earlybird_root.caching.TopTweetsServicePostProcessor;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class RealtimeRootAppModule extends TwitterModule {\n  private static final long RECENCY_CACHE_TTL_MILLIS = 20000L;\n  private static final long RELEVANCE_CACHE_TTL_MILLIS = 20000L;\n  private static final long FACETS_CACHE_TTL_MILLIS = 300000L;\n  private static final long TERMSTATS_CACHE_TTL_MILLIS = 300000L;\n\n  @Override\n  public void configure() {\n    bind(Key.get(EarlybirdCluster.class)).toInstance(EarlybirdCluster.REALTIME);\n\n    bind(EarlybirdServiceScatterGatherSupport.class)\n      .to(EarlybirdRealtimeScatterGatherSupport.class);\n\n    bind(EarlybirdService.ServiceIface.class).to(RealtimeRootService.class);\n  }\n\n  @Provides\n  @Singleton\n  @RecencyCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideRecencyCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, \"realtime_recency_root\",\n        serializedKeyPrefix, RECENCY_CACHE_TTL_MILLIS, cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @RelevanceCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideRelevanceCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, \"realtime_relevance_root\",\n        serializedKeyPrefix, RELEVANCE_CACHE_TTL_MILLIS, cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @StrictRecencyCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideStrictRecencyCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, \"realtime_strict_recency_root\",\n        serializedKeyPrefix, RECENCY_CACHE_TTL_MILLIS, cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @FacetsCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideFacetsCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, \"realtime_facets_root\",\n        serializedKeyPrefix, FACETS_CACHE_TTL_MILLIS, cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @TermStatsCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideTermStatsCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, \"realtime_termstats_root\",\n        serializedKeyPrefix, TERMSTATS_CACHE_TTL_MILLIS, cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  @Singleton\n  @TopTweetsCache\n  Cache<EarlybirdRequest, EarlybirdResponse> provideTopTweetsCache(\n      JavaClient client,\n      DefaultForcedCacheMissDecider decider,\n      @Named(SearchRootModule.NAMED_SERIALIZED_KEY_PREFIX) String serializedKeyPrefix,\n      @Named(SearchRootModule.NAMED_CACHE_KEY_MAX_BYTES) int cacheKeyMaxBytes,\n      @Named(SearchRootModule.NAMED_CACHE_VALUE_MAX_BYTES) int cacheValueMaxBytes) {\n    return EarlybirdCacheCommonModule.createCache(client, decider, \"realtime_toptweets_root\",\n        serializedKeyPrefix, TopTweetsServicePostProcessor.CACHE_AGE_IN_MS,\n        cacheKeyMaxBytes, cacheValueMaxBytes);\n  }\n\n  @Provides\n  SearchRootWarmup<EarlybirdService.ServiceIface, ?, ?> providesSearchRootWarmup(\n      Clock clock,\n      WarmupConfig config) {\n    return new EarlybirdWarmup(clock, config);\n  }\n\n  @Provides\n  public LoggingSupport<EarlybirdRequest, EarlybirdResponse> provideLoggingSupport(\n      SearchDecider decider) {\n    return new EarlybirdServiceLoggingSupport(decider);\n  }\n\n  @Provides\n  public PartitionLoggingSupport<EarlybirdRequestContext> providePartitionLoggingSupport() {\n    return new EarlybirdServicePartitionLoggingSupport();\n  }\n\n  @Provides\n  public ValidationBehavior<EarlybirdRequest, EarlybirdResponse> provideValidationBehavior() {\n    return new EarlybirdServiceValidationBehavior();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/RealtimeRootServer.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\nimport javax.inject.Singleton;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.search.common.root.SearchRootServer;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\n\n@Singleton\npublic class RealtimeRootServer extends SearchRootServer<EarlybirdService.ServiceIface> {\n\n  @Inject\n  public RealtimeRootServer(RealtimeRootService svc, Service<byte[], byte[]> byteSvc) {\n    super(svc, byteSvc);\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/RealtimeRootService.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.mtls.authorization.server.MtlsServerSessionTrackerFilter;\nimport com.twitter.search.common.clientstats.FinagleClientStatsFilter;\nimport com.twitter.search.common.root.LoggingFilter;\nimport com.twitter.search.common.root.RequestValidationFilter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusResponse;\nimport com.twitter.search.earlybird_root.caching.FacetsCacheFilter;\nimport com.twitter.search.earlybird_root.caching.RecencyCacheFilter;\nimport com.twitter.search.earlybird_root.caching.RelevanceCacheFilter;\nimport com.twitter.search.earlybird_root.caching.RelevanceZeroResultsCacheFilter;\nimport com.twitter.search.earlybird_root.caching.StrictRecencyCacheFilter;\nimport com.twitter.search.earlybird_root.caching.TermStatsCacheFilter;\nimport com.twitter.search.earlybird_root.caching.TopTweetsCacheFilter;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.filters.ClientIdTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.ClientRequestTimeFilter;\nimport com.twitter.search.earlybird_root.filters.DeadlineTimeoutStatsFilter;\nimport com.twitter.search.earlybird_root.filters.DropAllProtectedOperatorFilter;\nimport com.twitter.search.earlybird_root.filters.EarlybirdFeatureSchemaAnnotateFilter;\nimport com.twitter.search.earlybird_root.filters.InitializeRequestContextFilter;\nimport com.twitter.search.earlybird_root.filters.MetadataTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.NullcastTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.PostCacheRequestTypeCountFilter;\nimport com.twitter.search.earlybird_root.filters.PreCacheRequestTypeCountFilter;\nimport com.twitter.search.earlybird_root.filters.QueryLangStatFilter;\nimport com.twitter.search.earlybird_root.filters.QueryOperatorStatFilter;\nimport com.twitter.search.earlybird_root.filters.RequestResultStatsFilter;\nimport com.twitter.search.earlybird_root.filters.ResponseCodeStatFilter;\nimport com.twitter.search.earlybird_root.filters.SearchPayloadSizeLocalContextFilter;\nimport com.twitter.search.earlybird_root.filters.StratoAttributionClientIdFilter;\nimport com.twitter.search.earlybird_root.filters.TopLevelExceptionHandlingFilter;\nimport com.twitter.util.Future;\n\n@Singleton\npublic class RealtimeRootService implements EarlybirdService.ServiceIface {\n\n  private final Service<EarlybirdRequest, EarlybirdResponse> allFiltersAndService;\n\n  @Inject\n  public RealtimeRootService(\n      TopLevelExceptionHandlingFilter topLevelExceptionHandlingFilter,\n      ResponseCodeStatFilter responseCodeStatFilter,\n      LoggingFilter<EarlybirdRequest, EarlybirdResponse> loggingFilter,\n      RequestValidationFilter<EarlybirdRequest, EarlybirdResponse> validationFilter,\n      MtlsServerSessionTrackerFilter<EarlybirdRequest, EarlybirdResponse> mtlsFilter,\n      FinagleClientStatsFilter<EarlybirdRequest, EarlybirdResponse> finagleStatsFilter,\n      InitializeFilter initializeFilter,\n      InitializeRequestContextFilter initializeRequestContextFilter,\n      QueryLangStatFilter queryLangStatFilter,\n      DropAllProtectedOperatorFilter dropAllProtectedOperatorFilter,\n      QueryOperatorStatFilter queryOperatorStatFilter,\n      RequestResultStatsFilter requestResultStatsFilter,\n      PreCacheRequestTypeCountFilter preCacheCountFilter,\n      RecencyCacheFilter recencyCacheFilter,\n      RelevanceCacheFilter relevanceCacheFilter,\n      RelevanceZeroResultsCacheFilter relevanceZeroResultsCacheFilter,\n      StrictRecencyCacheFilter strictRecencyCacheFilter,\n      FacetsCacheFilter facetsCacheFilter,\n      TermStatsCacheFilter termStatsCacheFilter,\n      TopTweetsCacheFilter topTweetsCacheFilter,\n      PostCacheRequestTypeCountFilter postCacheCountFilter,\n      ClientIdTrackingFilter clientIdTrackingFilter,\n      MetadataTrackingFilter metadataTrackingFilter,\n      NullcastTrackingFilter nullcastTrackingFilter,\n      ClientRequestTimeFilter clientRequestTimeFilter,\n      DeadlineTimeoutStatsFilter deadlineTimeoutStatsFilter,\n      EarlybirdFeatureSchemaAnnotateFilter featureSchemaAnnotateFilter,\n      SearchPayloadSizeLocalContextFilter searchPayloadSizeLocalContextFilter,\n      @Named(ProtectedScatterGatherModule.NAMED_SCATTER_GATHER_SERVICE)\n          Service<EarlybirdRequestContext, EarlybirdResponse> scatterGatherService,\n      StratoAttributionClientIdFilter stratoAttributionClientIdFilter) {\n    this.allFiltersAndService =\n        loggingFilter\n            .andThen(topLevelExceptionHandlingFilter)\n            .andThen(stratoAttributionClientIdFilter)\n            .andThen(clientRequestTimeFilter)\n            .andThen(searchPayloadSizeLocalContextFilter)\n            .andThen(responseCodeStatFilter)\n            .andThen(requestResultStatsFilter)\n            .andThen(validationFilter)\n            .andThen(mtlsFilter)\n            .andThen(finagleStatsFilter)\n            .andThen(clientIdTrackingFilter)\n            .andThen(metadataTrackingFilter)\n            .andThen(initializeFilter)\n            .andThen(initializeRequestContextFilter)\n            .andThen(deadlineTimeoutStatsFilter)\n            .andThen(queryLangStatFilter)\n            .andThen(nullcastTrackingFilter)\n            .andThen(dropAllProtectedOperatorFilter)\n            .andThen(queryOperatorStatFilter)\n            .andThen(preCacheCountFilter)\n            .andThen(recencyCacheFilter)\n            .andThen(relevanceCacheFilter)\n            .andThen(relevanceZeroResultsCacheFilter)\n            .andThen(strictRecencyCacheFilter)\n            .andThen(facetsCacheFilter)\n            .andThen(termStatsCacheFilter)\n            .andThen(topTweetsCacheFilter)\n            .andThen(postCacheCountFilter)\n            .andThen(featureSchemaAnnotateFilter)\n            .andThen(scatterGatherService);\n  }\n\n  @Override\n  public Future<String> getName() {\n    return Future.value(\"realtime root\");\n  }\n\n  @Override\n  public Future<EarlybirdStatusResponse> getStatus() {\n    throw new UnsupportedOperationException(\"not supported\");\n  }\n\n  @Override\n  public Future<EarlybirdResponse> search(EarlybirdRequest request) {\n    return allFiltersAndService.apply(request);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/RealtimeScatterGatherModule.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.Arrays;\nimport java.util.HashMap;\nimport java.util.Map;\nimport javax.annotation.Nullable;\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport com.google.inject.Provides;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.stats.StatsReceiver;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.PartitionConfig;\nimport com.twitter.search.common.root.PartitionLoggingSupport;\nimport com.twitter.search.common.root.RequestSuccessStats;\nimport com.twitter.search.common.root.RootClientServiceBuilder;\nimport com.twitter.search.common.root.ScatterGatherService;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird.thrift.ExperimentCluster;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.filters.RequestContextToEarlybirdRequestFilter;\nimport com.twitter.search.earlybird_root.filters.ScatterGatherWithExperimentRedirectsService;\n\npublic class RealtimeScatterGatherModule extends ScatterGatherModule {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(RealtimeScatterGatherModule.class);\n\n  /**\n   * Provides a scatter gather service for the realtime cluster that redirects to experimental\n   * clusters when the experiment cluster parameter is set on the EarlybirdRequest.\n   *\n   * Note: if an alternate client is specified via altPartitionConfig or\n   * altRootClientServiceBuilder, it will be built and used for the \"control\" cluster, but the\n   * experiment cluster takes precedence (if the experiment cluster is set on the request, the\n   * alternate client will never be used.\n   */\n  @Provides\n  @Singleton\n  @Named(NAMED_SCATTER_GATHER_SERVICE)\n  @Override\n  public Service<EarlybirdRequestContext, EarlybirdResponse> provideScatterGatherService(\n      EarlybirdServiceScatterGatherSupport scatterGatherSupport,\n      RequestSuccessStats requestSuccessStats,\n      PartitionLoggingSupport<EarlybirdRequestContext> partitionLoggingSupport,\n      RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter,\n      PartitionAccessController partitionAccessController,\n      PartitionConfig partitionConfig,\n      RootClientServiceBuilder<EarlybirdService.ServiceIface> rootClientServiceBuilder,\n      @Named(EarlybirdCommonModule.NAMED_EXP_CLUSTER_CLIENT)\n          RootClientServiceBuilder<EarlybirdService.ServiceIface>\n          expClusterRootClientServiceBuilder,\n      @Named(EarlybirdCommonModule.NAMED_ALT_CLIENT) @Nullable PartitionConfig altPartitionConfig,\n      @Named(EarlybirdCommonModule.NAMED_ALT_CLIENT) @Nullable\n          RootClientServiceBuilder<EarlybirdService.ServiceIface> altRootClientServiceBuilder,\n      StatsReceiver statsReceiver,\n      EarlybirdCluster cluster,\n      SearchDecider decider) {\n\n\n    Service<EarlybirdRequestContext, EarlybirdResponse> controlService =\n        buildScatterOrSplitterService(\n          scatterGatherSupport,\n          requestSuccessStats,\n          partitionLoggingSupport,\n          requestContextToEarlybirdRequestFilter,\n          partitionAccessController,\n          partitionConfig,\n          rootClientServiceBuilder,\n          altPartitionConfig,\n          altRootClientServiceBuilder,\n          statsReceiver,\n          cluster,\n          decider\n    );\n\n    Map<ExperimentCluster, ScatterGatherService<EarlybirdRequestContext, EarlybirdResponse>>\n        experimentScatterGatherServices = new HashMap<>();\n\n    LOG.info(\"Using ScatterGatherWithExperimentRedirectsService\");\n    LOG.info(\"Control Partition Path: {}\", partitionConfig.getPartitionPath());\n\n    Arrays.stream(ExperimentCluster.values())\n        .filter(v -> v.name().toLowerCase().startsWith(\"exp\"))\n        .forEach(experimentCluster -> {\n          String expPartitionPath = partitionConfig.getPartitionPath()\n              + \"-\" + experimentCluster.name().toLowerCase();\n\n          LOG.info(\"Experiment Partition Path: {}\", expPartitionPath);\n\n          experimentScatterGatherServices.put(experimentCluster,\n              createScatterGatherService(\n                  \"\",\n                  scatterGatherSupport,\n                  requestSuccessStats,\n                  partitionLoggingSupport,\n                  requestContextToEarlybirdRequestFilter,\n                  partitionAccessController,\n                  partitionConfig.getNumPartitions(),\n                  expPartitionPath,\n                  expClusterRootClientServiceBuilder,\n                  statsReceiver,\n                  cluster,\n                  decider,\n                  experimentCluster.name().toLowerCase()));\n        });\n\n    return new ScatterGatherWithExperimentRedirectsService(\n        controlService,\n        experimentScatterGatherServices);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/RootResponseClassifier.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport scala.PartialFunction;\nimport scala.runtime.AbstractPartialFunction;\n\nimport com.twitter.finagle.service.ReqRep;\nimport com.twitter.finagle.service.ResponseClass;\nimport com.twitter.finagle.service.ResponseClasses;\nimport com.twitter.finagle.service.ResponseClassifier;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.util.Try;\n\npublic class RootResponseClassifier extends AbstractPartialFunction<ReqRep, ResponseClass> {\n  private static final PartialFunction<ReqRep, ResponseClass> DEFAULT_CLASSIFIER =\n      ResponseClassifier.Default();\n\n  private static final SearchRateCounter NOT_EARLYBIRD_REQUEST_COUNTER =\n      SearchRateCounter.export(\"response_classifier_not_earlybird_request\");\n  private static final SearchRateCounter NOT_EARLYBIRD_RESPONSE_COUNTER =\n      SearchRateCounter.export(\"response_classifier_not_earlybird_response\");\n  private static final SearchRateCounter NON_RETRYABLE_FAILURE_COUNTER =\n      SearchRateCounter.export(\"response_classifier_non_retryable_failure\");\n  private static final SearchRateCounter RETRYABLE_FAILURE_COUNTER =\n      SearchRateCounter.export(\"response_classifier_retryable_failure\");\n  private static final SearchRateCounter SUCCESS_COUNTER =\n      SearchRateCounter.export(\"response_classifier_success\");\n\n  @Override\n  public boolean isDefinedAt(ReqRep reqRep) {\n    if (!(reqRep.request() instanceof EarlybirdService.search_args)) {\n      NOT_EARLYBIRD_REQUEST_COUNTER.increment();\n      return false;\n    }\n\n    if (!reqRep.response().isThrow() && (!(reqRep.response().get() instanceof EarlybirdResponse))) {\n      NOT_EARLYBIRD_RESPONSE_COUNTER.increment();\n      return false;\n    }\n\n    return true;\n  }\n\n  @Override\n  public ResponseClass apply(ReqRep reqRep) {\n    Try<?> responseTry = reqRep.response();\n    if (responseTry.isThrow()) {\n      return DEFAULT_CLASSIFIER.apply(reqRep);\n    }\n\n    // isDefinedAt() guarantees that the response is an EarlybirdResponse instance.\n    EarlybirdResponseCode responseCode = ((EarlybirdResponse) responseTry.get()).getResponseCode();\n    switch (responseCode) {\n      case PARTITION_NOT_FOUND:\n      case PARTITION_DISABLED:\n      case PERSISTENT_ERROR:\n        NON_RETRYABLE_FAILURE_COUNTER.increment();\n        return ResponseClasses.NON_RETRYABLE_FAILURE;\n      case TRANSIENT_ERROR:\n        RETRYABLE_FAILURE_COUNTER.increment();\n        return ResponseClasses.RETRYABLE_FAILURE;\n      default:\n        SUCCESS_COUNTER.increment();\n        return ResponseClasses.SUCCESS;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/ScatterGatherModule.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.ArrayList;\nimport java.util.List;\n\nimport javax.annotation.Nullable;\nimport javax.inject.Named;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.stats.StatsReceiver;\nimport com.twitter.inject.TwitterModule;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.PartitionConfig;\nimport com.twitter.search.common.root.PartitionLoggingSupport;\nimport com.twitter.search.common.root.RequestSuccessStats;\nimport com.twitter.search.common.root.RootClientServiceBuilder;\nimport com.twitter.search.common.root.ScatterGatherService;\nimport com.twitter.search.common.root.SplitterService;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.config.TierConfig;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.filters.RequestContextToEarlybirdRequestFilter;\n\npublic abstract class ScatterGatherModule extends TwitterModule {\n  private static final Logger LOG = LoggerFactory.getLogger(ScatterGatherModule.class);\n\n  private static final String SEARCH_METHOD_NAME = \"search\";\n  protected static final String ALT_TRAFFIC_PERCENTAGE_DECIDER_KEY_PREFIX =\n      \"alt_client_traffic_percentage_\";\n  static final String NAMED_SCATTER_GATHER_SERVICE = \"scatter_gather_service\";\n\n  /**\n   * Provides the scatterGatherService for single tier Earlybird clusters (Protected and Realtime).\n   */\n  public abstract Service<EarlybirdRequestContext, EarlybirdResponse> provideScatterGatherService(\n      EarlybirdServiceScatterGatherSupport scatterGatherSupport,\n      RequestSuccessStats requestSuccessStats,\n      PartitionLoggingSupport<EarlybirdRequestContext> partitionLoggingSupport,\n      RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter,\n      PartitionAccessController partitionAccessController,\n      PartitionConfig partitionConfig,\n      RootClientServiceBuilder<EarlybirdService.ServiceIface> rootClientServiceBuilder,\n      @Named(EarlybirdCommonModule.NAMED_EXP_CLUSTER_CLIENT)\n          RootClientServiceBuilder<EarlybirdService.ServiceIface>\n          expClusterRootClientServiceBuilder,\n      @Named(EarlybirdCommonModule.NAMED_ALT_CLIENT) @Nullable PartitionConfig altPartitionConfig,\n      @Named(EarlybirdCommonModule.NAMED_ALT_CLIENT) @Nullable\n          RootClientServiceBuilder<EarlybirdService.ServiceIface> altRootClientServiceBuilder,\n      StatsReceiver statsReceiver,\n      EarlybirdCluster cluster,\n      SearchDecider decider);\n\n  protected final Service<EarlybirdRequestContext, EarlybirdResponse> buildScatterOrSplitterService(\n      EarlybirdServiceScatterGatherSupport scatterGatherSupport,\n      RequestSuccessStats requestSuccessStats,\n      PartitionLoggingSupport<EarlybirdRequestContext> partitionLoggingSupport,\n      RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter,\n      PartitionAccessController partitionAccessController,\n      PartitionConfig partitionConfig,\n      RootClientServiceBuilder<EarlybirdService.ServiceIface> rootClientServiceBuilder,\n      @Named(EarlybirdCommonModule.NAMED_ALT_CLIENT) @Nullable PartitionConfig altPartitionConfig,\n      @Named(EarlybirdCommonModule.NAMED_ALT_CLIENT) @Nullable\n          RootClientServiceBuilder<EarlybirdService.ServiceIface> altRootClientServiceBuilder,\n      StatsReceiver statsReceiver,\n      EarlybirdCluster cluster,\n      SearchDecider decider\n  ) {\n    ScatterGatherService<EarlybirdRequestContext, EarlybirdResponse> scatterGatherService =\n        createScatterGatherService(\n            \"\",\n            scatterGatherSupport,\n            requestSuccessStats,\n            partitionLoggingSupport,\n            requestContextToEarlybirdRequestFilter,\n            partitionAccessController,\n            partitionConfig.getNumPartitions(),\n            partitionConfig.getPartitionPath(),\n            rootClientServiceBuilder,\n            statsReceiver,\n            cluster,\n            decider,\n            TierConfig.DEFAULT_TIER_NAME);\n\n    if (altPartitionConfig == null || altRootClientServiceBuilder == null) {\n      LOG.info(\"altPartitionConfig or altRootClientServiceBuilder is not available, \"\n          + \"not using SplitterService\");\n      return scatterGatherService;\n    }\n\n    LOG.info(\"alt client config available, using SplitterService\");\n\n    ScatterGatherService<EarlybirdRequestContext, EarlybirdResponse> altScatterGatherService =\n        createScatterGatherService(\n            \"_alt\",\n            scatterGatherSupport,\n            requestSuccessStats,\n            partitionLoggingSupport,\n            requestContextToEarlybirdRequestFilter,\n            partitionAccessController,\n            altPartitionConfig.getNumPartitions(),\n            altPartitionConfig.getPartitionPath(),\n            altRootClientServiceBuilder,\n            statsReceiver,\n            cluster,\n            decider,\n            TierConfig.DEFAULT_TIER_NAME);\n\n    return new SplitterService<>(\n        scatterGatherService,\n        altScatterGatherService,\n        decider,\n        ALT_TRAFFIC_PERCENTAGE_DECIDER_KEY_PREFIX + cluster.getNameForStats());\n  }\n\n  protected ScatterGatherService<EarlybirdRequestContext, EarlybirdResponse>\n      createScatterGatherService(\n          String nameSuffix,\n          EarlybirdServiceScatterGatherSupport scatterGatherSupport,\n          RequestSuccessStats requestSuccessStats,\n          PartitionLoggingSupport<EarlybirdRequestContext> partitionLoggingSupport,\n          RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter,\n          PartitionAccessController partitionAccessController,\n          int numPartitions,\n          String partitionPath,\n          RootClientServiceBuilder<EarlybirdService.ServiceIface> rootClientServiceBuilder,\n          StatsReceiver statsReceiver,\n          EarlybirdCluster cluster,\n          SearchDecider decider,\n          String clientName) {\n    rootClientServiceBuilder.initializeWithPathSuffix(clientName + nameSuffix,\n        numPartitions,\n        partitionPath);\n\n    ClientBackupFilter backupFilter =\n        new ClientBackupFilter(\n            \"root_\" + cluster.getNameForStats(),\n            \"earlybird\" + nameSuffix,\n            statsReceiver,\n            decider);\n\n    ClientLatencyFilter clientLatencyFilter = new ClientLatencyFilter(\"all\" + nameSuffix);\n\n    List<Service<EarlybirdRequestContext, EarlybirdResponse>> services = new ArrayList<>();\n    for (Service<EarlybirdRequest, EarlybirdResponse> service\n        : rootClientServiceBuilder\n        .<EarlybirdRequest, EarlybirdResponse>safeBuildServiceList(SEARCH_METHOD_NAME)) {\n      services.add(requestContextToEarlybirdRequestFilter\n          .andThen(backupFilter)\n          .andThen(clientLatencyFilter)\n          .andThen(service));\n    }\n    services = SkipPartitionFilter.wrapServices(TierConfig.DEFAULT_TIER_NAME, services,\n        partitionAccessController);\n\n    return new ScatterGatherService<>(\n        scatterGatherSupport,\n        services,\n        requestSuccessStats,\n        partitionLoggingSupport);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/SkipPartitionFilter.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.ArrayList;\nimport java.util.List;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\nimport com.twitter.util.Future;\n\n/**\n * Filter that returns a PARTITION_SKIPPED response instead of sending the request to a partition\n * if the partition PartitionAccessController says its disabled for a request.\n */\npublic final class SkipPartitionFilter extends\n    SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n\n  private static final Logger LOG = LoggerFactory.getLogger(SkipPartitionFilter.class);\n\n  private final String tierName;\n  private final int partitionNum;\n  private final PartitionAccessController controller;\n\n  private SkipPartitionFilter(String tierName, int partitionNum,\n                             PartitionAccessController controller) {\n    this.tierName = tierName;\n    this.partitionNum = partitionNum;\n    this.controller = controller;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n\n    EarlybirdRequest request = requestContext.getRequest();\n    if (!controller.canAccessPartition(tierName, partitionNum, request.getClientId(),\n        EarlybirdRequestType.of(request))) {\n      return Future.value(EarlybirdServiceScatterGatherSupport.newEmptyResponse());\n    }\n\n    return service.apply(requestContext);\n  }\n\n  /**\n   * Wrap the services with a SkipPartitionFilter\n   */\n  public static List<Service<EarlybirdRequestContext, EarlybirdResponse>> wrapServices(\n      String tierName,\n      List<Service<EarlybirdRequestContext, EarlybirdResponse>> clients,\n      PartitionAccessController controller) {\n\n    LOG.info(\"Creating SkipPartitionFilters for cluster: {}, tier: {}, partitions 0-{}\",\n        controller.getClusterName(), tierName, clients.size() - 1);\n\n    List<Service<EarlybirdRequestContext, EarlybirdResponse>> wrappedServices = new ArrayList<>();\n    for (int partitionNum = 0; partitionNum < clients.size(); partitionNum++) {\n      SkipPartitionFilter filter = new SkipPartitionFilter(tierName, partitionNum, controller);\n      wrappedServices.add(filter.andThen(clients.get(partitionNum)));\n    }\n\n    return wrappedServices;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/SuperRootAppMain.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.Arrays;\nimport java.util.Collection;\n\nimport com.google.inject.Module;\n\nimport com.twitter.search.common.root.SearchRootAppMain;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird_root.routers.FacetsRequestRouterModule;\nimport com.twitter.search.earlybird_root.routers.RecencyRequestRouterModule;\nimport com.twitter.search.earlybird_root.routers.RelevanceRequestRouterModule;\nimport com.twitter.search.earlybird_root.routers.TermStatsRequestRouterModule;\nimport com.twitter.search.earlybird_root.routers.TopTweetsRequestRouterModule;\n\npublic class SuperRootAppMain extends SearchRootAppMain<SuperRootServer> {\n  /**\n   * Boilerplate for the Java-friendly AbstractTwitterServer\n   */\n  public static class Main {\n    public static void main(String[] args) {\n      new SuperRootAppMain().main(args);\n    }\n  }\n\n  @Override\n  protected Collection<? extends Module> getAdditionalModules() {\n    return Arrays.asList(\n        new EarlybirdCommonModule(),\n        new SuperRootAppModule(),\n        new TermStatsRequestRouterModule(),\n        new RecencyRequestRouterModule(),\n        new RelevanceRequestRouterModule(),\n        new TopTweetsRequestRouterModule(),\n        new FacetsRequestRouterModule(),\n        new QuotaModule());\n  }\n\n  @Override\n  protected Class<SuperRootServer> getSearchRootServerClass() {\n    return SuperRootServer.class;\n  }\n\n  @Override\n  protected Class<?> getServiceIfaceClass() {\n    return EarlybirdService.ServiceIface.class;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/SuperRootAppModule.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Named;\nimport javax.inject.Singleton;\n\nimport com.google.inject.Key;\nimport com.google.inject.Provides;\nimport com.google.inject.util.Providers;\n\nimport com.twitter.app.Flag;\nimport com.twitter.app.Flaggable;\nimport com.twitter.common.util.Clock;\nimport com.twitter.common_internal.text.version.PenguinVersionConfig;\nimport com.twitter.finagle.Name;\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.stats.StatsReceiver;\nimport com.twitter.inject.TwitterModule;\nimport com.twitter.search.common.config.SearchPenguinVersionsConfig;\nimport com.twitter.search.common.dark.ResolverProxy;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.LoggingSupport;\nimport com.twitter.search.common.root.RemoteClientBuilder;\nimport com.twitter.search.common.root.SearchRootWarmup;\nimport com.twitter.search.common.root.ValidationBehavior;\nimport com.twitter.search.common.root.WarmupConfig;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird.thrift.ThriftTweetSource;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.InjectionNames;\nimport com.twitter.search.earlybird_root.filters.EarlybirdClusterAvailableFilter;\nimport com.twitter.search.earlybird_root.filters.MarkTweetSourceFilter;\nimport com.twitter.search.earlybird_root.filters.RequestContextToEarlybirdRequestFilter;\nimport com.twitter.search.earlybird_root.filters.RequestTypeCountFilter;\nimport com.twitter.search.earlybird_root.filters.ServiceExceptionHandlingFilter;\nimport com.twitter.search.earlybird_root.filters.ServiceResponseValidationFilter;\nimport com.twitter.search.earlybird_root.filters.UnsetSuperRootFieldsFilter;\nimport com.twitter.util.Future;\n\npublic class SuperRootAppModule extends TwitterModule {\n  private final Flag<String> rootRealtimeFlag = createFlag(\n      \"root-realtime\",\n      \"\",\n      \"Override the path to root-realtime\",\n      Flaggable.ofString());\n  private final Flag<String> rootProtectedFlag = createFlag(\n      \"root-protected\",\n      \"\",\n      \"Override the path to root-protected\",\n      Flaggable.ofString());\n  private final Flag<String> rootArchiveFullFlag = createFlag(\n      \"root-archive-full\",\n      \"\",\n      \"Override the path to root-archive-full\",\n      Flaggable.ofString());\n  private final Flag<String> penguinVersionsFlag = createMandatoryFlag(\n      \"penguin_versions\",\n      \"Penguin versions to be tokenized\",\n      \"\",\n      Flaggable.ofString());\n\n  @Override\n  public void configure() {\n    // SuperRoot uses all clusters, not just one. We bind EarlybirdCluster to null to indicate that\n    // there is not one specific cluster to use.\n    bind(Key.get(EarlybirdCluster.class)).toProvider(Providers.<EarlybirdCluster>of(null));\n\n    bind(EarlybirdService.ServiceIface.class).to(SuperRootService.class);\n  }\n\n  @Provides\n  SearchRootWarmup<EarlybirdService.ServiceIface, ?, ?> providesSearchRootWarmup(\n      Clock clock,\n      WarmupConfig config) {\n    return new EarlybirdWarmup(clock, config);\n  }\n\n  @Provides\n  @Singleton\n  @Named(InjectionNames.REALTIME)\n  private EarlybirdService.ServiceIface providesRealtimeIface(\n      RemoteClientBuilder<EarlybirdService.ServiceIface> builder,\n      ResolverProxy proxy) throws Exception {\n    Name name = proxy.resolve(rootRealtimeFlag.apply());\n    return builder.createRemoteClient(name, \"realtime\", \"realtime_\");\n  }\n\n  @Provides\n  @Singleton\n  @Named(InjectionNames.REALTIME)\n  private Service<EarlybirdRequestContext, EarlybirdResponse> providesRealtimeService(\n      @Named(InjectionNames.REALTIME)\n      EarlybirdService.ServiceIface realtimeServiceIface,\n      RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter,\n      StatsReceiver statsReceiver,\n      SearchDecider decider) {\n    return buildClientService(\n        realtimeServiceIface,\n        new EarlybirdClusterAvailableFilter(decider, EarlybirdCluster.REALTIME),\n        new MarkTweetSourceFilter(ThriftTweetSource.REALTIME_CLUSTER),\n        new ServiceExceptionHandlingFilter(EarlybirdCluster.REALTIME),\n        new ServiceResponseValidationFilter(EarlybirdCluster.REALTIME),\n        new RequestTypeCountFilter(EarlybirdCluster.REALTIME.getNameForStats()),\n        requestContextToEarlybirdRequestFilter,\n        new UnsetSuperRootFieldsFilter(),\n        new ClientLatencyFilter(EarlybirdCluster.REALTIME.getNameForStats()));\n  }\n\n  @Provides\n  @Singleton\n  @Named(InjectionNames.FULL_ARCHIVE)\n  private EarlybirdService.ServiceIface providesFullArchiveIface(\n      RemoteClientBuilder<EarlybirdService.ServiceIface> builder,\n      ResolverProxy proxy) throws Exception {\n    Name name = proxy.resolve(rootArchiveFullFlag.apply());\n    return builder.createRemoteClient(name, \"fullarchive\", \"full_archive_\");\n  }\n\n  @Provides\n  @Singleton\n  @Named(InjectionNames.FULL_ARCHIVE)\n  private Service<EarlybirdRequestContext, EarlybirdResponse> providesFullArchiveService(\n      @Named(InjectionNames.FULL_ARCHIVE)\n      EarlybirdService.ServiceIface fullArchiveServiceIface,\n      RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter,\n      StatsReceiver statsReceiver,\n      SearchDecider decider) {\n    return buildClientService(\n        fullArchiveServiceIface,\n        new EarlybirdClusterAvailableFilter(decider, EarlybirdCluster.FULL_ARCHIVE),\n        new MarkTweetSourceFilter(ThriftTweetSource.FULL_ARCHIVE_CLUSTER),\n        new ServiceExceptionHandlingFilter(EarlybirdCluster.FULL_ARCHIVE),\n        new ServiceResponseValidationFilter(EarlybirdCluster.FULL_ARCHIVE),\n        new RequestTypeCountFilter(EarlybirdCluster.FULL_ARCHIVE.getNameForStats()),\n        requestContextToEarlybirdRequestFilter,\n        // Disable unset followedUserIds for archive since archive earlybirds rely on this field\n        // to rewrite query to include protected Tweets\n        new UnsetSuperRootFieldsFilter(false),\n        new ClientLatencyFilter(EarlybirdCluster.FULL_ARCHIVE.getNameForStats()));\n  }\n\n  @Provides\n  @Singleton\n  @Named(InjectionNames.PROTECTED)\n  private EarlybirdService.ServiceIface providesProtectedIface(\n      RemoteClientBuilder<EarlybirdService.ServiceIface> builder,\n      ResolverProxy proxy) throws Exception {\n    Name name = proxy.resolve(rootProtectedFlag.apply());\n    return builder.createRemoteClient(name, \"protected\", \"protected_\");\n  }\n\n  @Provides\n  @Singleton\n  @Named(InjectionNames.PROTECTED)\n  private Service<EarlybirdRequestContext, EarlybirdResponse> providesProtectedService(\n      @Named(InjectionNames.PROTECTED)\n      EarlybirdService.ServiceIface protectedServiceIface,\n      RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter,\n      StatsReceiver statsReceiver,\n      SearchDecider decider) {\n    return buildClientService(\n        protectedServiceIface,\n        new EarlybirdClusterAvailableFilter(decider, EarlybirdCluster.PROTECTED),\n        new MarkTweetSourceFilter(ThriftTweetSource.REALTIME_PROTECTED_CLUSTER),\n        new ServiceExceptionHandlingFilter(EarlybirdCluster.PROTECTED),\n        new ServiceResponseValidationFilter(EarlybirdCluster.PROTECTED),\n        new RequestTypeCountFilter(EarlybirdCluster.PROTECTED.getNameForStats()),\n        requestContextToEarlybirdRequestFilter,\n        new UnsetSuperRootFieldsFilter(),\n        new ClientLatencyFilter(EarlybirdCluster.PROTECTED.getNameForStats()));\n  }\n\n  /**\n   * Builds a Finagle Service based on a EarlybirdService.ServiceIface.\n   */\n  private Service<EarlybirdRequestContext, EarlybirdResponse> buildClientService(\n      final EarlybirdService.ServiceIface serviceIface,\n      EarlybirdClusterAvailableFilter earlybirdClusterAvailableFilter,\n      MarkTweetSourceFilter markTweetSourceFilter,\n      ServiceExceptionHandlingFilter serviceExceptionHandlingFilter,\n      ServiceResponseValidationFilter serviceResponseValidationFilter,\n      RequestTypeCountFilter requestTypeCountFilter,\n      RequestContextToEarlybirdRequestFilter requestContextToEarlybirdRequestFilter,\n      UnsetSuperRootFieldsFilter unsetSuperRootFieldsFilter,\n      ClientLatencyFilter latencyFilter) {\n    Service<EarlybirdRequest, EarlybirdResponse> service =\n        new Service<EarlybirdRequest, EarlybirdResponse>() {\n\n          @Override\n          public Future<EarlybirdResponse> apply(EarlybirdRequest requestContext) {\n            return serviceIface.search(requestContext);\n          }\n        };\n\n    // We should apply ServiceResponseValidationFilter first, to validate the response.\n    // Then, if the response is valid, we should tag all results with the appropriate tweet source.\n    // ServiceExceptionHandlingFilter should come last, to catch all possible exceptions (that were\n    // thrown by the service, or by ServiceResponseValidationFilter and MarkTweetSourceFilter).\n    //\n    // But before we do all of this, we should apply the EarlybirdClusterAvailableFilter to see if\n    // we even need to send the request to this cluster.\n    return earlybirdClusterAvailableFilter\n        .andThen(serviceExceptionHandlingFilter)\n        .andThen(markTweetSourceFilter)\n        .andThen(serviceResponseValidationFilter)\n        .andThen(requestTypeCountFilter)\n        .andThen(requestContextToEarlybirdRequestFilter)\n        .andThen(latencyFilter)\n        .andThen(unsetSuperRootFieldsFilter)\n        .andThen(service);\n  }\n\n  @Provides\n  public LoggingSupport<EarlybirdRequest, EarlybirdResponse> provideLoggingSupport(\n      SearchDecider decider) {\n    return new EarlybirdServiceLoggingSupport(decider);\n  }\n\n  @Provides\n  public ValidationBehavior<EarlybirdRequest, EarlybirdResponse> provideValidationBehavior() {\n    return new EarlybirdServiceValidationBehavior();\n  }\n\n  /**\n   * Provides the penguin versions that we should use to retokenize the query if requested.\n   */\n  @Provides\n  @Singleton\n  public PenguinVersionConfig providePenguinVersions() {\n    return SearchPenguinVersionsConfig.deserialize(penguinVersionsFlag.apply());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/SuperRootRequestTypeRouter.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport java.util.Map;\n\nimport javax.inject.Inject;\nimport javax.inject.Singleton;\n\nimport com.google.common.collect.ImmutableMap;\nimport com.google.common.collect.Maps;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.ClientErrorException;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\nimport com.twitter.search.earlybird_root.routers.FacetsRequestRouter;\nimport com.twitter.search.earlybird_root.routers.RecencyRequestRouter;\nimport com.twitter.search.earlybird_root.routers.RelevanceRequestRouter;\nimport com.twitter.search.earlybird_root.routers.RequestRouter;\nimport com.twitter.search.earlybird_root.routers.TermStatsRequestRouter;\nimport com.twitter.search.earlybird_root.routers.TopTweetsRequestRouter;\nimport com.twitter.util.Future;\n\n@Singleton\npublic class SuperRootRequestTypeRouter\n    extends Service<EarlybirdRequestContext, EarlybirdResponse>  {\n\n  private final Map<EarlybirdRequestType, RequestRouter> routingMap;\n\n  /**\n   * constructor\n   */\n  @Inject\n  public SuperRootRequestTypeRouter(\n      FacetsRequestRouter facetsRequestRouter,\n      TermStatsRequestRouter termStatsRequestRouter,\n      TopTweetsRequestRouter topTweetsRequestRouter,\n      RecencyRequestRouter recencyRequestRouter,\n      RelevanceRequestRouter relevanceRequestRouter\n  ) {\n    routingMap = Maps.immutableEnumMap(\n        ImmutableMap.<EarlybirdRequestType, RequestRouter>builder()\n            .put(EarlybirdRequestType.FACETS, facetsRequestRouter)\n            .put(EarlybirdRequestType.TERM_STATS, termStatsRequestRouter)\n            .put(EarlybirdRequestType.TOP_TWEETS, topTweetsRequestRouter)\n            .put(EarlybirdRequestType.RECENCY, recencyRequestRouter)\n            .put(EarlybirdRequestType.STRICT_RECENCY, recencyRequestRouter)\n            .put(EarlybirdRequestType.RELEVANCE, relevanceRequestRouter)\n            .build());\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(EarlybirdRequestContext requestContext) {\n    EarlybirdRequest request = requestContext.getRequest();\n    if (request.getSearchQuery() == null) {\n      return Future.exception(new ClientErrorException(\n          \"Client must fill in search Query object in request\"));\n    }\n\n    EarlybirdRequestType requestType = requestContext.getEarlybirdRequestType();\n\n    if (routingMap.containsKey(requestType)) {\n      RequestRouter router = routingMap.get(requestType);\n      return router.route(requestContext);\n    } else {\n      return Future.exception(\n          new ClientErrorException(\n            \"Request type \" + requestType + \" is unsupported.  \"\n                  + \"Sorry this api is a bit hard to use.\\n\"\n                  + \"for facets, call earlybirdRequest.setFacetsRequest\\n\"\n                  + \"for termstats, call earluybirdRequest.setTermStatisticsRequest\\n\"\n                  + \"for recency, strict recency, relevance or toptweets,\\n\"\n                  + \"   call req.setSearchQuery() and req.getSearchQuery().setRankingMode()\\n\"\n                  + \"   with the correct ranking mode and for strict recency call\\n\"\n                  + \"   earlybirdRequest.setQuerySource(ThriftQuerySource.GNIP)\\n\"));\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/SuperRootServer.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\nimport javax.inject.Singleton;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.search.common.root.SearchRootServer;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird_root.filters.QueryTokenizerFilter;\nimport com.twitter.search.queryparser.query.QueryParserException;\n\n@Singleton\npublic class SuperRootServer extends SearchRootServer<EarlybirdService.ServiceIface> {\n  private final QueryTokenizerFilter queryTokenizerFilter;\n\n  @Inject\n  public SuperRootServer(\n      SuperRootService svc,\n      Service<byte[], byte[]> byteSvc,\n      QueryTokenizerFilter queryTokenizerFilter) {\n    super(svc, byteSvc);\n\n    this.queryTokenizerFilter = queryTokenizerFilter;\n  }\n\n  @Override\n  public void warmup() {\n    super.warmup();\n\n    try {\n      queryTokenizerFilter.performExpensiveInitialization();\n    } catch (QueryParserException e) {\n      throw new RuntimeException(e);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/SuperRootService.java",
    "content": "package com.twitter.search.earlybird_root;\n\nimport javax.inject.Inject;\nimport javax.inject.Singleton;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.mtls.authorization.server.MtlsServerSessionTrackerFilter;\nimport com.twitter.search.common.clientstats.FinagleClientStatsFilter;\nimport com.twitter.search.common.root.LoggingFilter;\nimport com.twitter.search.common.root.RequestValidationFilter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdService;\nimport com.twitter.search.earlybird.thrift.EarlybirdStatusResponse;\nimport com.twitter.search.earlybird_root.filters.ClientIdArchiveAccessFilter;\nimport com.twitter.search.earlybird_root.filters.ClientIdQuotaFilter;\nimport com.twitter.search.earlybird_root.filters.ClientIdTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.ClientRequestTimeFilter;\nimport com.twitter.search.earlybird_root.filters.DeadlineTimeoutStatsFilter;\nimport com.twitter.search.earlybird_root.filters.DisableClientByTierFilter;\nimport com.twitter.search.earlybird_root.filters.EarlybirdFeatureSchemaAnnotateFilter;\nimport com.twitter.search.earlybird_root.filters.InitializeRequestContextFilter;\nimport com.twitter.search.earlybird_root.filters.MetadataTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.NamedMultiTermDisjunctionStatsFilter;\nimport com.twitter.search.earlybird_root.filters.NullcastTrackingFilter;\nimport com.twitter.search.earlybird_root.filters.PreCacheRequestTypeCountFilter;\nimport com.twitter.search.earlybird_root.filters.QueryLangStatFilter;\nimport com.twitter.search.earlybird_root.filters.QueryOperatorStatFilter;\nimport com.twitter.search.earlybird_root.filters.QueryTokenizerFilter;\nimport com.twitter.search.earlybird_root.filters.RequestResultStatsFilter;\nimport com.twitter.search.earlybird_root.filters.RequestSuccessStatsFilter;\nimport com.twitter.search.earlybird_root.filters.ResponseCodeStatFilter;\nimport com.twitter.search.earlybird_root.filters.SearchPayloadSizeLocalContextFilter;\nimport com.twitter.search.earlybird_root.filters.RejectRequestsByQuerySourceFilter;\nimport com.twitter.search.earlybird_root.filters.StratoAttributionClientIdFilter;\nimport com.twitter.search.earlybird_root.filters.TopLevelExceptionHandlingFilter;\nimport com.twitter.search.earlybird_root.filters.VeryRecentTweetsFilter;\nimport com.twitter.util.Future;\n\n@Singleton\nclass SuperRootService implements EarlybirdService.ServiceIface {\n  private final Service<EarlybirdRequest, EarlybirdResponse> fullSearchMethod;\n\n  @Inject\n  public SuperRootService(\n      TopLevelExceptionHandlingFilter topLevelExceptionHandlingFilter,\n      ResponseCodeStatFilter responseCodeStatFilter,\n      LoggingFilter<EarlybirdRequest, EarlybirdResponse> loggingFilter,\n      NamedMultiTermDisjunctionStatsFilter namedMultiTermDisjunctionStatsFilter,\n      RequestValidationFilter<EarlybirdRequest, EarlybirdResponse> validationFilter,\n      MtlsServerSessionTrackerFilter<EarlybirdRequest, EarlybirdResponse> mtlsFilter,\n      FinagleClientStatsFilter<EarlybirdRequest, EarlybirdResponse> finagleStatsFilter,\n      InitializeFilter initializeFilter,\n      InitializeRequestContextFilter initializeRequestContextFilter,\n      QueryLangStatFilter queryLangStatFilter,\n      QueryOperatorStatFilter queryOperatorStatFilter,\n      RequestResultStatsFilter requestResultStatsFilter,\n      PreCacheRequestTypeCountFilter preCacheRequestTypeCountFilter,\n      ClientIdArchiveAccessFilter clientIdArchiveAccessFilter,\n      DisableClientByTierFilter disableClientByTierFilter,\n      ClientIdTrackingFilter clientIdTrackingFilter,\n      ClientIdQuotaFilter quotaFilter,\n      RejectRequestsByQuerySourceFilter rejectRequestsByQuerySourceFilter,\n      MetadataTrackingFilter metadataTrackingFilter,\n      VeryRecentTweetsFilter veryRecentTweetsFilter,\n      RequestSuccessStatsFilter requestSuccessStatsFilter,\n      NullcastTrackingFilter nullcastTrackingFilter,\n      QueryTokenizerFilter queryTokenizerFilter,\n      ClientRequestTimeFilter clientRequestTimeFilter,\n      DeadlineTimeoutStatsFilter deadlineTimeoutStatsFilter,\n      SuperRootRequestTypeRouter superRootSearchService,\n      EarlybirdFeatureSchemaAnnotateFilter featureSchemaAnnotateFilter,\n      SearchPayloadSizeLocalContextFilter searchPayloadSizeLocalContextFilter,\n      StratoAttributionClientIdFilter stratoAttributionClientIdFilter) {\n    this.fullSearchMethod =\n        loggingFilter\n            .andThen(topLevelExceptionHandlingFilter)\n            .andThen(stratoAttributionClientIdFilter)\n            .andThen(clientRequestTimeFilter)\n            .andThen(searchPayloadSizeLocalContextFilter)\n            .andThen(requestSuccessStatsFilter)\n            .andThen(requestResultStatsFilter)\n            .andThen(responseCodeStatFilter)\n            .andThen(validationFilter)\n            .andThen(mtlsFilter)\n            .andThen(finagleStatsFilter)\n            .andThen(disableClientByTierFilter)\n            .andThen(clientIdTrackingFilter)\n            .andThen(quotaFilter)\n            .andThen(clientIdArchiveAccessFilter)\n            .andThen(rejectRequestsByQuerySourceFilter)\n            .andThen(namedMultiTermDisjunctionStatsFilter)\n            .andThen(metadataTrackingFilter)\n            .andThen(veryRecentTweetsFilter)\n            .andThen(initializeFilter)\n            .andThen(initializeRequestContextFilter)\n            .andThen(deadlineTimeoutStatsFilter)\n            .andThen(queryLangStatFilter)\n            .andThen(nullcastTrackingFilter)\n            .andThen(queryOperatorStatFilter)\n            .andThen(preCacheRequestTypeCountFilter)\n            .andThen(queryTokenizerFilter)\n            .andThen(featureSchemaAnnotateFilter)\n            .andThen(superRootSearchService);\n  }\n\n  @Override\n  public Future<String> getName() {\n    return Future.value(\"superroot\");\n  }\n\n  @Override\n  public Future<EarlybirdStatusResponse> getStatus() {\n    throw new UnsupportedOperationException(\"not supported\");\n  }\n\n  @Override\n  public Future<EarlybirdResponse> search(EarlybirdRequest request) {\n    return fullSearchMethod.apply(request);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"finatra/inject/inject-core/src/main/scala\",\n        \"src/java/com/twitter/search/common/caching\",\n        \"src/java/com/twitter/search/common/decider\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/root\",\n        \"src/java/com/twitter/search/earlybird/common\",\n        \"src/java/com/twitter/search/earlybird_root/common\",\n        \"src/java/com/twitter/search/queryparser\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/CacheCommonUtil.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\n\npublic final class CacheCommonUtil {\n  public static final String NAMED_MAX_CACHE_RESULTS = \"maxCacheResults\";\n\n  private CacheCommonUtil() {\n  }\n\n  public static boolean hasResults(EarlybirdResponse response) {\n    return response.isSetSearchResults()\n      && (response.getSearchResults().getResults() != null)\n      && !response.getSearchResults().getResults().isEmpty();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/CacheStats.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.common.metrics.SearchRateCounter;\n\npublic final class CacheStats {\n  public static final SearchRateCounter REQUEST_FAILED_COUNTER =\n      SearchRateCounter.export(\"memcache_request_failed\");\n  public static final SearchRateCounter REQUEST_TIMEOUT_COUNTER =\n      SearchRateCounter.export(\"memcache_request_timeout\");\n\n  private CacheStats() {\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/DefaultForcedCacheMissDecider.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport javax.inject.Inject;\n\nimport com.twitter.common.base.Supplier;\nimport com.twitter.search.common.decider.SearchDecider;\n\n/**\n * A cache miss decider backed by a decider key.\n */\npublic class DefaultForcedCacheMissDecider implements Supplier<Boolean> {\n  private static final String DECIDER_KEY = \"default_forced_cache_miss_rate\";\n  private final SearchDecider decider;\n\n  @Inject\n  public DefaultForcedCacheMissDecider(SearchDecider decider) {\n    this.decider = decider;\n  }\n\n  @Override\n  public Boolean get() {\n    return decider.isAvailable(DECIDER_KEY);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/EarlybirdCachePostProcessor.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.google.common.base.Optional;\n\nimport com.twitter.search.common.caching.filter.CachePostProcessor;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class EarlybirdCachePostProcessor\n    extends CachePostProcessor<EarlybirdRequestContext, EarlybirdResponse> {\n\n  @Override\n  public final void recordCacheHit(EarlybirdResponse response) {\n    response.setCacheHit(true);\n  }\n\n  @Override\n  public Optional<EarlybirdResponse> processCacheResponse(EarlybirdRequestContext originalRequest,\n                                                          EarlybirdResponse cacheResponse) {\n    return Optional.of(cacheResponse);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/EarlybirdRequestPerClientCacheStats.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport java.util.Map;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport com.twitter.search.common.caching.filter.PerClientCacheStats;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.earlybird.common.EarlybirdRequestUtil;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class EarlybirdRequestPerClientCacheStats\n    extends PerClientCacheStats<EarlybirdRequestContext> {\n\n  private String cacheOffByClientStatFormat;\n  private final Map<String, SearchRateCounter> cacheTurnedOffByClient;\n\n  private String cacheHitsByClientStatFormat;\n  private final Map<String, SearchRateCounter> cacheHitsByClient;\n\n  public EarlybirdRequestPerClientCacheStats(String cacheRequestType) {\n    this.cacheOffByClientStatFormat =\n        cacheRequestType + \"_client_id_%s_cache_turned_off_in_request\";\n    this.cacheTurnedOffByClient = new ConcurrentHashMap<>();\n\n    this.cacheHitsByClientStatFormat = cacheRequestType + \"_client_id_%s_cache_hit_total\";\n    this.cacheHitsByClient = new ConcurrentHashMap<>();\n  }\n\n  @Override\n  public void recordRequest(EarlybirdRequestContext requestContext) {\n    if (!EarlybirdRequestUtil.isCachingAllowed(requestContext.getRequest())) {\n      String client = requestContext.getRequest().getClientId();\n      SearchRateCounter counter = cacheTurnedOffByClient.computeIfAbsent(client,\n          cl -> SearchRateCounter.export(String.format(cacheOffByClientStatFormat, cl)));\n      counter.increment();\n    }\n  }\n\n  @Override\n  public void recordCacheHit(EarlybirdRequestContext requestContext) {\n    String client = requestContext.getRequest().getClientId();\n    SearchRateCounter counter = cacheHitsByClient.computeIfAbsent(client,\n        cl -> SearchRateCounter.export(String.format(cacheHitsByClientStatFormat, cl)));\n    counter.increment();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/FacetsCache.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport java.lang.annotation.ElementType;\nimport java.lang.annotation.Retention;\nimport java.lang.annotation.Target;\n\nimport com.google.inject.BindingAnnotation;\n\nimport static java.lang.annotation.RetentionPolicy.RUNTIME;\n\n@Retention(RUNTIME)\n@Target({ ElementType.FIELD, ElementType.PARAMETER, ElementType.METHOD })\n@BindingAnnotation\npublic @interface FacetsCache {\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/FacetsCacheFilter.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.filter.CacheFilter;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class FacetsCacheFilter extends\n    CacheFilter<EarlybirdRequestContext, EarlybirdRequest, EarlybirdResponse> {\n  /**\n   * Constructs a new cache filter for facet requests.\n   */\n  @Inject\n  public FacetsCacheFilter(\n      @FacetsCache Cache<EarlybirdRequest, EarlybirdResponse> cache,\n      SearchDecider decider,\n      @Named(SearchRootModule.NAMED_NORMALIZED_SEARCH_ROOT_NAME) String normalizedSearchRootName) {\n    super(cache,\n          new FacetsQueryCachePredicate(decider, normalizedSearchRootName),\n          new FacetsCacheRequestNormalizer(),\n          new EarlybirdCachePostProcessor(),\n          new FacetsServicePostProcessor(cache),\n          new EarlybirdRequestPerClientCacheStats(EarlybirdRequestType.FACETS.getNormalizedName()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/FacetsCacheRequestNormalizer.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.google.common.base.Optional;\n\nimport com.twitter.search.common.caching.FacetsCacheUtil;\nimport com.twitter.search.common.caching.filter.CacheRequestNormalizer;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class FacetsCacheRequestNormalizer extends\n    CacheRequestNormalizer<EarlybirdRequestContext, EarlybirdRequest> {\n\n  @Override\n  public Optional<EarlybirdRequest> normalizeRequest(EarlybirdRequestContext requestContext) {\n    return Optional.fromNullable(FacetsCacheUtil.normalizeRequestForCache(\n        requestContext.getRequest()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/FacetsQueryCachePredicate.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.common.caching.filter.QueryCachePredicate;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.earlybird.common.EarlybirdRequestUtil;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class FacetsQueryCachePredicate extends QueryCachePredicate<EarlybirdRequestContext> {\n  private final SearchDecider decider;\n  private final String facetsCacheEnabledDeciderKey;\n\n  public FacetsQueryCachePredicate(SearchDecider decider, String normalizedSearchRootName) {\n    this.decider = decider;\n    this.facetsCacheEnabledDeciderKey = \"facets_cache_enabled_\" + normalizedSearchRootName;\n  }\n\n  @Override\n  public Boolean shouldQueryCache(EarlybirdRequestContext requestContext) {\n    return EarlybirdRequestType.FACETS == requestContext.getEarlybirdRequestType()\n        && EarlybirdRequestUtil.isCachingAllowed(requestContext.getRequest())\n        && decider.isAvailable(facetsCacheEnabledDeciderKey);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/FacetsServicePostProcessor.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.FacetsCacheUtil;\nimport com.twitter.search.common.caching.filter.ServicePostProcessor;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class FacetsServicePostProcessor\n    extends ServicePostProcessor<EarlybirdRequestContext, EarlybirdResponse> {\n\n  private final Cache<EarlybirdRequest, EarlybirdResponse> cache;\n\n  public FacetsServicePostProcessor(Cache<EarlybirdRequest, EarlybirdResponse> cache) {\n    this.cache = cache;\n  }\n\n  @Override\n  public void processServiceResponse(EarlybirdRequestContext requestContext,\n                                     EarlybirdResponse serviceResponse) {\n    FacetsCacheUtil.cacheResults(requestContext.getRequest(), serviceResponse, cache);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RecencyAndRelevanceCachePostProcessor.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.google.common.base.Optional;\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.caching.CacheUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.util.IdTimeRanges;\n\npublic class RecencyAndRelevanceCachePostProcessor extends EarlybirdCachePostProcessor {\n\n  private static final Logger LOG =\n      LoggerFactory.getLogger(RecencyAndRelevanceCachePostProcessor.class);\n\n  protected Optional<EarlybirdResponse> postProcessCacheResponse(\n      EarlybirdRequest earlybirdRequest,\n      EarlybirdResponse earlybirdResponse, long sinceID, long maxID) {\n    return CacheUtil.postProcessCacheResult(\n        earlybirdRequest, earlybirdResponse, sinceID, maxID);\n  }\n\n  @Override\n  public final Optional<EarlybirdResponse> processCacheResponse(\n      EarlybirdRequestContext requestContext,\n      EarlybirdResponse cacheResponse) {\n    EarlybirdRequest originalRequest = requestContext.getRequest();\n    Preconditions.checkArgument(originalRequest.isSetSearchQuery());\n\n    IdTimeRanges ranges;\n    Query query = requestContext.getParsedQuery();\n    if (query != null) {\n      try {\n        ranges = IdTimeRanges.fromQuery(query);\n      } catch (QueryParserException e) {\n        LOG.error(\n            \"Exception when parsing since and max IDs. Request: {} Response: {}\",\n            originalRequest,\n            cacheResponse,\n            e);\n        return Optional.absent();\n      }\n    } else {\n      ranges = null;\n    }\n\n    Optional<Long> sinceID;\n    Optional<Long> maxID;\n    if (ranges != null) {\n      sinceID = ranges.getSinceIDExclusive();\n      maxID = ranges.getMaxIDInclusive();\n    } else {\n      sinceID = Optional.absent();\n      maxID = Optional.absent();\n    }\n\n    return postProcessCacheResponse(\n        originalRequest, cacheResponse, sinceID.or(0L), maxID.or(Long.MAX_VALUE));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RecencyCache.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport java.lang.annotation.ElementType;\nimport java.lang.annotation.Retention;\nimport java.lang.annotation.Target;\n\nimport com.google.inject.BindingAnnotation;\n\nimport static java.lang.annotation.RetentionPolicy.RUNTIME;\n\n@Retention(RUNTIME)\n@Target({ ElementType.FIELD, ElementType.PARAMETER, ElementType.METHOD })\n@BindingAnnotation\npublic @interface RecencyCache {\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RecencyCacheFilter.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.filter.CacheFilter;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class RecencyCacheFilter extends\n    CacheFilter<EarlybirdRequestContext, EarlybirdRequest, EarlybirdResponse> {\n  /**\n   * Creates a cache filter for earlybird recency requests.\n   */\n  @Inject\n  public RecencyCacheFilter(\n      @RecencyCache Cache<EarlybirdRequest, EarlybirdResponse> cache,\n      SearchDecider decider,\n      @Named(SearchRootModule.NAMED_NORMALIZED_SEARCH_ROOT_NAME) String normalizedSearchRootName,\n      @Named(CacheCommonUtil.NAMED_MAX_CACHE_RESULTS) int maxCacheResults) {\n    super(cache,\n          new RecencyQueryCachePredicate(decider, normalizedSearchRootName),\n          new RecencyCacheRequestNormalizer(),\n          new RecencyAndRelevanceCachePostProcessor(),\n          new RecencyServicePostProcessor(cache, maxCacheResults),\n          new EarlybirdRequestPerClientCacheStats(\n              EarlybirdRequestType.RECENCY.getNormalizedName()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RecencyCacheRequestNormalizer.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.google.common.base.Optional;\n\nimport com.twitter.search.common.caching.CacheUtil;\nimport com.twitter.search.common.caching.filter.CacheRequestNormalizer;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class RecencyCacheRequestNormalizer extends\n    CacheRequestNormalizer<EarlybirdRequestContext, EarlybirdRequest> {\n  @Override\n  public Optional<EarlybirdRequest> normalizeRequest(EarlybirdRequestContext requestContext) {\n    return Optional.fromNullable(CacheUtil.normalizeRequestForCache(requestContext.getRequest()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RecencyQueryCachePredicate.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.common.caching.filter.QueryCachePredicate;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.earlybird.common.EarlybirdRequestUtil;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class RecencyQueryCachePredicate extends QueryCachePredicate<EarlybirdRequestContext> {\n  private final SearchDecider decider;\n  private final String recencyCacheEnabledDeciderKey;\n\n  public RecencyQueryCachePredicate(SearchDecider decider, String normalizedSearchRootName) {\n    this.decider = decider;\n    this.recencyCacheEnabledDeciderKey = \"recency_cache_enabled_\" + normalizedSearchRootName;\n  }\n\n  @Override\n  public Boolean shouldQueryCache(EarlybirdRequestContext request) {\n    return EarlybirdRequestType.RECENCY == request.getEarlybirdRequestType()\n        && EarlybirdRequestUtil.isCachingAllowed(request.getRequest())\n        && decider.isAvailable(recencyCacheEnabledDeciderKey);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RecencyServicePostProcessor.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.CacheUtil;\nimport com.twitter.search.common.caching.filter.ServicePostProcessor;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class RecencyServicePostProcessor\n    extends ServicePostProcessor<EarlybirdRequestContext, EarlybirdResponse> {\n  private final Cache<EarlybirdRequest, EarlybirdResponse> cache;\n  private final int maxCacheResults;\n\n  public RecencyServicePostProcessor(\n      Cache<EarlybirdRequest, EarlybirdResponse> cache,\n      int maxCacheResults) {\n    this.cache = cache;\n    this.maxCacheResults = maxCacheResults;\n  }\n\n  @Override\n  public void processServiceResponse(EarlybirdRequestContext requestContext,\n                                     EarlybirdResponse serviceResponse) {\n    CacheUtil.cacheResults(cache, requestContext.getRequest(), serviceResponse, maxCacheResults);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RelevanceCache.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport java.lang.annotation.ElementType;\nimport java.lang.annotation.Retention;\nimport java.lang.annotation.Target;\n\nimport com.google.inject.BindingAnnotation;\n\nimport static java.lang.annotation.RetentionPolicy.RUNTIME;\n\n@Retention(RUNTIME)\n@Target({ ElementType.FIELD, ElementType.PARAMETER, ElementType.METHOD })\n@BindingAnnotation\npublic @interface RelevanceCache {\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RelevanceCacheFilter.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.filter.CacheFilter;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class RelevanceCacheFilter extends\n    CacheFilter<EarlybirdRequestContext, EarlybirdRequest, EarlybirdResponse> {\n  /**\n   * Creates a cache filter for earlybird relevance requests\n   */\n  @Inject\n  public RelevanceCacheFilter(\n      @RelevanceCache Cache<EarlybirdRequest, EarlybirdResponse> cache,\n      SearchDecider decider,\n      @Named(SearchRootModule.NAMED_NORMALIZED_SEARCH_ROOT_NAME) String normalizedSearchRootName) {\n    super(cache,\n          new RelevanceQueryCachePredicate(decider, normalizedSearchRootName),\n          new RelevanceCacheRequestNormalizer(decider, normalizedSearchRootName),\n          new RecencyAndRelevanceCachePostProcessor(),\n          new RelevanceServicePostProcessor(cache),\n          new EarlybirdRequestPerClientCacheStats(\n              EarlybirdRequestType.RELEVANCE.getNormalizedName()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RelevanceCacheRequestNormalizer.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.google.common.base.Optional;\n\nimport com.twitter.search.common.caching.CacheUtil;\nimport com.twitter.search.common.caching.filter.CacheRequestNormalizer;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class RelevanceCacheRequestNormalizer extends\n    CacheRequestNormalizer<EarlybirdRequestContext, EarlybirdRequest> {\n  private static final SearchCounter RELEVANCE_FORCE_CACHED_LOGGED_IN_REQUEST =\n      SearchCounter.export(\"relevance_force_cached_logged_in_request\");\n\n  private final SearchDecider decider;\n  private final String relevanceStripPersonalizationFieldsDeciderKey;\n\n  public RelevanceCacheRequestNormalizer(\n      SearchDecider decider,\n      String normalizedSearchRootName) {\n    this.decider = decider;\n    this.relevanceStripPersonalizationFieldsDeciderKey =\n        String.format(\"relevance_%s_force_cache_logged_in_requests\", normalizedSearchRootName);\n  }\n\n  @Override\n  public Optional<EarlybirdRequest> normalizeRequest(EarlybirdRequestContext requestContext) {\n    boolean cacheLoggedInRequest =\n        decider.isAvailable(relevanceStripPersonalizationFieldsDeciderKey);\n\n    if (cacheLoggedInRequest) {\n      RELEVANCE_FORCE_CACHED_LOGGED_IN_REQUEST.increment();\n    }\n\n    return Optional.fromNullable(CacheUtil.normalizeRequestForCache(\n                                     requestContext.getRequest(), cacheLoggedInRequest));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RelevanceQueryCachePredicate.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.common.caching.filter.QueryCachePredicate;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.earlybird.common.EarlybirdRequestUtil;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class RelevanceQueryCachePredicate extends QueryCachePredicate<EarlybirdRequestContext> {\n  private final SearchDecider decider;\n  private final String relevanceCacheEnabledDeciderKey;\n\n  public RelevanceQueryCachePredicate(SearchDecider decider, String normalizedSearchRootName) {\n    this.decider = decider;\n    this.relevanceCacheEnabledDeciderKey = \"relevance_cache_enabled_\" + normalizedSearchRootName;\n  }\n\n  @Override\n  public Boolean shouldQueryCache(EarlybirdRequestContext requestContext) {\n    return EarlybirdRequestType.RELEVANCE == requestContext.getEarlybirdRequestType()\n        && EarlybirdRequestUtil.isCachingAllowed(requestContext.getRequest())\n        && decider.isAvailable(relevanceCacheEnabledDeciderKey);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RelevanceServicePostProcessor.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.CacheUtil;\nimport com.twitter.search.common.caching.filter.ServicePostProcessor;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class RelevanceServicePostProcessor\n    extends ServicePostProcessor<EarlybirdRequestContext, EarlybirdResponse> {\n  private final Cache<EarlybirdRequest, EarlybirdResponse> cache;\n\n  public RelevanceServicePostProcessor(\n      Cache<EarlybirdRequest, EarlybirdResponse> cache) {\n    this.cache = cache;\n  }\n\n  @Override\n  public void processServiceResponse(EarlybirdRequestContext requestContext,\n                                     EarlybirdResponse serviceResponse) {\n    CacheUtil.cacheResults(cache, requestContext.getRequest(), serviceResponse, Integer.MAX_VALUE);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RelevanceZeroResultsCacheFilter.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.filter.CacheFilter;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\n/**\n * A filter that:\n *  - Strips the request of all personalization fields, normalizes it and looks it up in the cache.\n *    If it finds a response with 0 results in the cache, it returns it.\n *  - Caches the response for a personalized query, whenever the response has 0 results. The cache\n *    key is the normalized request with all personalization fields stripped.\n *\n * If a query (from a logged in or logged out user) returns 0 results, then the same query will\n * always return 0 results, for all users. So we can cache that result.\n */\npublic class RelevanceZeroResultsCacheFilter\n  extends CacheFilter<EarlybirdRequestContext, EarlybirdRequest, EarlybirdResponse> {\n\n  /** Creates a filter that caches relevance requests with 0 results. */\n  @Inject\n  public RelevanceZeroResultsCacheFilter(\n      @RelevanceCache Cache<EarlybirdRequest, EarlybirdResponse> cache,\n      SearchDecider decider,\n      @Named(SearchRootModule.NAMED_NORMALIZED_SEARCH_ROOT_NAME) String normalizedSearchRootName) {\n    super(cache,\n          new RelevanceZeroResultsQueryCachePredicate(decider, normalizedSearchRootName),\n          new RelevanceZeroResultsCacheRequestNormalizer(),\n          new RelevanceZeroResultsCachePostProcessor(),\n          new RelevanceZeroResultsServicePostProcessor(cache),\n          new EarlybirdRequestPerClientCacheStats(\"relevance_zero_results\"));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RelevanceZeroResultsCachePostProcessor.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.google.common.base.Optional;\n\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\n\npublic class RelevanceZeroResultsCachePostProcessor extends RecencyAndRelevanceCachePostProcessor {\n  @Override\n  protected Optional<EarlybirdResponse> postProcessCacheResponse(\n      EarlybirdRequest request, EarlybirdResponse response, long sinceId, long maxId) {\n    // If a query (from a logged in or logged out user) returns 0 results, then the same query will\n    // always return 0 results, for all users. So we can cache that result.\n    if (CacheCommonUtil.hasResults(response)) {\n      return Optional.absent();\n    }\n\n    return Optional.of(response);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RelevanceZeroResultsCacheRequestNormalizer.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.google.common.base.Optional;\n\nimport com.twitter.search.common.caching.CacheUtil;\nimport com.twitter.search.common.caching.SearchQueryNormalizer;\nimport com.twitter.search.common.caching.filter.CacheRequestNormalizer;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class RelevanceZeroResultsCacheRequestNormalizer\n    extends CacheRequestNormalizer<EarlybirdRequestContext, EarlybirdRequest> {\n  @Override\n  public Optional<EarlybirdRequest> normalizeRequest(EarlybirdRequestContext requestContext) {\n    // If the query is not personalized, it means that:\n    //   - RelevanceCacheRequestNormalizer has already normalized it into a cacheable query.\n    //   - RelevanceCacheFilter could not find a response for this query in the cache.\n    //\n    // So if we try to normalize it here again, we will succeed, but then\n    // RelevanceZeroResultsCacheFilter will do the same look up in the cache, which will again\n    // result in a cache miss. There is no need to do this look up twice, so if the query is not\n    // personalized, return Optional.absent().\n    //\n    // If the query is personalized, strip all personalization fields and normalize the request.\n    if (!SearchQueryNormalizer.queryIsPersonalized(requestContext.getRequest().getSearchQuery())) {\n      return Optional.absent();\n    }\n    return Optional.fromNullable(\n        CacheUtil.normalizeRequestForCache(requestContext.getRequest(), true));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RelevanceZeroResultsQueryCachePredicate.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.common.caching.filter.QueryCachePredicate;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.earlybird.common.EarlybirdRequestUtil;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class RelevanceZeroResultsQueryCachePredicate\n    extends QueryCachePredicate<EarlybirdRequestContext> {\n  private final SearchDecider decider;\n  private final String relevanceCacheEnabledDeciderKey;\n  private final String relevanceZeroResultsCacheEnabledDeciderKey;\n\n  public RelevanceZeroResultsQueryCachePredicate(\n      SearchDecider decider, String normalizedSearchRootName) {\n    this.decider = decider;\n    this.relevanceCacheEnabledDeciderKey =\n        \"relevance_cache_enabled_\" + normalizedSearchRootName;\n    this.relevanceZeroResultsCacheEnabledDeciderKey =\n        \"relevance_zero_results_cache_enabled_\" + normalizedSearchRootName;\n  }\n\n  @Override\n  public Boolean shouldQueryCache(EarlybirdRequestContext requestContext) {\n    return EarlybirdRequestType.RELEVANCE == requestContext.getEarlybirdRequestType()\n        && EarlybirdRequestUtil.isCachingAllowed(requestContext.getRequest())\n        && decider.isAvailable(relevanceCacheEnabledDeciderKey)\n        && decider.isAvailable(relevanceZeroResultsCacheEnabledDeciderKey);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/RelevanceZeroResultsServicePostProcessor.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.CacheUtil;\nimport com.twitter.search.common.caching.filter.ServicePostProcessor;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class RelevanceZeroResultsServicePostProcessor\n    extends ServicePostProcessor<EarlybirdRequestContext, EarlybirdResponse> {\n\n  private static final SearchCounter RELEVANCE_RESPONSES_WITH_ZERO_RESULTS_COUNTER =\n    SearchCounter.export(\"relevance_responses_with_zero_results\");\n\n  private final Cache<EarlybirdRequest, EarlybirdResponse> cache;\n\n  public RelevanceZeroResultsServicePostProcessor(\n      Cache<EarlybirdRequest, EarlybirdResponse> cache) {\n    this.cache = cache;\n  }\n\n  @Override\n  public void processServiceResponse(EarlybirdRequestContext requestContext,\n                                     EarlybirdResponse serviceResponse) {\n    // serviceResponse is the response to a personalized query. If it has zero results, then we can\n    // cache it and reuse it for other requests with the same query. Otherwise, it makes no sense to\n    // cache this response.\n    if (!CacheCommonUtil.hasResults(serviceResponse)) {\n      RELEVANCE_RESPONSES_WITH_ZERO_RESULTS_COUNTER.increment();\n      CacheUtil.cacheResults(\n          cache, requestContext.getRequest(), serviceResponse, Integer.MAX_VALUE);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/StrictRecencyCache.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport java.lang.annotation.ElementType;\nimport java.lang.annotation.Retention;\nimport java.lang.annotation.Target;\n\nimport com.google.inject.BindingAnnotation;\n\nimport static java.lang.annotation.RetentionPolicy.RUNTIME;\n\n@Retention(RUNTIME)\n@Target({ ElementType.FIELD, ElementType.PARAMETER, ElementType.METHOD })\n@BindingAnnotation\npublic @interface StrictRecencyCache {\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/StrictRecencyCacheFilter.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.filter.CacheFilter;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class StrictRecencyCacheFilter extends\n    CacheFilter<EarlybirdRequestContext, EarlybirdRequest, EarlybirdResponse> {\n  /**\n   * Creates a cache filter for earlybird strict recency requests.\n   */\n  @Inject\n  public StrictRecencyCacheFilter(\n      @StrictRecencyCache Cache<EarlybirdRequest, EarlybirdResponse> cache,\n      SearchDecider decider,\n      @Named(SearchRootModule.NAMED_NORMALIZED_SEARCH_ROOT_NAME) String normalizedSearchRootName,\n      @Named(CacheCommonUtil.NAMED_MAX_CACHE_RESULTS) int maxCacheResults) {\n    super(cache,\n          new StrictRecencyQueryCachePredicate(decider, normalizedSearchRootName),\n          new RecencyCacheRequestNormalizer(),\n          new RecencyAndRelevanceCachePostProcessor(),\n          new RecencyServicePostProcessor(cache, maxCacheResults),\n          new EarlybirdRequestPerClientCacheStats(\n              EarlybirdRequestType.STRICT_RECENCY.getNormalizedName()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/StrictRecencyQueryCachePredicate.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.common.caching.filter.QueryCachePredicate;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.earlybird.common.EarlybirdRequestUtil;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class StrictRecencyQueryCachePredicate extends QueryCachePredicate<EarlybirdRequestContext> {\n  private final SearchDecider decider;\n  private final String strictRecencyCacheEnabledDeciderKey;\n\n  public StrictRecencyQueryCachePredicate(SearchDecider decider, String normalizedSearchRootName) {\n    this.decider = decider;\n    this.strictRecencyCacheEnabledDeciderKey =\n        \"strict_recency_cache_enabled_\" + normalizedSearchRootName;\n  }\n\n  @Override\n  public Boolean shouldQueryCache(EarlybirdRequestContext requestContext) {\n    return EarlybirdRequestType.STRICT_RECENCY == requestContext.getEarlybirdRequestType()\n        && EarlybirdRequestUtil.isCachingAllowed(requestContext.getRequest())\n        && decider.isAvailable(strictRecencyCacheEnabledDeciderKey);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/TermStatsCache.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport java.lang.annotation.ElementType;\nimport java.lang.annotation.Retention;\nimport java.lang.annotation.Target;\n\nimport com.google.inject.BindingAnnotation;\n\nimport static java.lang.annotation.RetentionPolicy.RUNTIME;\n\n@Retention(RUNTIME)\n@Target({ ElementType.FIELD, ElementType.PARAMETER, ElementType.METHOD })\n@BindingAnnotation\npublic @interface TermStatsCache {\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/TermStatsCacheFilter.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.filter.CacheFilter;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class TermStatsCacheFilter extends\n    CacheFilter<EarlybirdRequestContext, EarlybirdRequest, EarlybirdResponse> {\n  /**\n   * Constructs a new cache filter for term stats requests.\n   */\n  @Inject\n  public TermStatsCacheFilter(\n      @TermStatsCache Cache<EarlybirdRequest, EarlybirdResponse> cache,\n      SearchDecider decider,\n      @Named(SearchRootModule.NAMED_NORMALIZED_SEARCH_ROOT_NAME) String normalizedSearchRootName) {\n    super(cache,\n          new TermStatsQueryCachePredicate(decider, normalizedSearchRootName),\n          new TermStatsCacheRequestNormalizer(),\n          new EarlybirdCachePostProcessor(),\n          new TermStatsServicePostProcessor(cache),\n          new EarlybirdRequestPerClientCacheStats(\n              EarlybirdRequestType.TERM_STATS.getNormalizedName()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/TermStatsCacheRequestNormalizer.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.google.common.base.Optional;\n\nimport com.twitter.search.common.caching.TermStatsCacheUtil;\nimport com.twitter.search.common.caching.filter.CacheRequestNormalizer;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class TermStatsCacheRequestNormalizer extends\n    CacheRequestNormalizer<EarlybirdRequestContext, EarlybirdRequest> {\n\n  @Override\n  public Optional<EarlybirdRequest> normalizeRequest(EarlybirdRequestContext requestContext) {\n    return Optional.fromNullable(TermStatsCacheUtil.normalizeForCache(requestContext.getRequest()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/TermStatsQueryCachePredicate.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.common.caching.filter.QueryCachePredicate;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.earlybird.common.EarlybirdRequestUtil;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class TermStatsQueryCachePredicate extends QueryCachePredicate<EarlybirdRequestContext> {\n  private final SearchDecider decider;\n  private final String termstatsCacheEnabledDeciderKey;\n\n  public TermStatsQueryCachePredicate(SearchDecider decider, String normalizedSearchRootName) {\n    this.decider = decider;\n    this.termstatsCacheEnabledDeciderKey = \"termstats_cache_enabled_\" + normalizedSearchRootName;\n  }\n\n  @Override\n  public Boolean shouldQueryCache(EarlybirdRequestContext requestContext) {\n    return EarlybirdRequestType.TERM_STATS == requestContext.getEarlybirdRequestType()\n        && EarlybirdRequestUtil.isCachingAllowed(requestContext.getRequest())\n        && decider.isAvailable(termstatsCacheEnabledDeciderKey);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/TermStatsServicePostProcessor.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.TermStatsCacheUtil;\nimport com.twitter.search.common.caching.filter.ServicePostProcessor;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class TermStatsServicePostProcessor\n    extends ServicePostProcessor<EarlybirdRequestContext, EarlybirdResponse> {\n  private final Cache<EarlybirdRequest, EarlybirdResponse> cache;\n\n  public TermStatsServicePostProcessor(Cache<EarlybirdRequest, EarlybirdResponse> cache) {\n    this.cache = Preconditions.checkNotNull(cache);\n  }\n\n  @Override\n  public void processServiceResponse(EarlybirdRequestContext requestContext,\n                                     EarlybirdResponse serviceResponse) {\n    TermStatsCacheUtil.cacheResults(cache, requestContext.getRequest(), serviceResponse);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/TopTweetsCache.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport java.lang.annotation.ElementType;\nimport java.lang.annotation.Retention;\nimport java.lang.annotation.Target;\n\nimport com.google.inject.BindingAnnotation;\n\nimport static java.lang.annotation.RetentionPolicy.RUNTIME;\n\n@Retention(RUNTIME)\n@Target({ ElementType.FIELD, ElementType.PARAMETER, ElementType.METHOD })\n@BindingAnnotation\npublic @interface TopTweetsCache {\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/TopTweetsCacheFilter.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport javax.inject.Inject;\nimport javax.inject.Named;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.filter.CacheFilter;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.root.SearchRootModule;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class TopTweetsCacheFilter extends\n    CacheFilter<EarlybirdRequestContext, EarlybirdRequest, EarlybirdResponse> {\n  /**\n   * Constructs a new cache filter for top tweets requests.\n   */\n  @Inject\n  public TopTweetsCacheFilter(\n      @TopTweetsCache Cache<EarlybirdRequest, EarlybirdResponse> cache,\n      SearchDecider decider,\n      @Named(SearchRootModule.NAMED_NORMALIZED_SEARCH_ROOT_NAME) String normalizedSearchRootName) {\n    super(cache,\n          new TopTweetsQueryCachePredicate(decider, normalizedSearchRootName),\n          new TopTweetsCacheRequestNormalizer(),\n          new EarlybirdCachePostProcessor(),\n          new TopTweetsServicePostProcessor(cache),\n          new EarlybirdRequestPerClientCacheStats(\n              EarlybirdRequestType.TOP_TWEETS.getNormalizedName()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/TopTweetsCacheRequestNormalizer.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.google.common.base.Optional;\n\nimport com.twitter.search.common.caching.TopTweetsCacheUtil;\nimport com.twitter.search.common.caching.filter.CacheRequestNormalizer;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\npublic class TopTweetsCacheRequestNormalizer extends\n    CacheRequestNormalizer<EarlybirdRequestContext, EarlybirdRequest> {\n\n  @Override\n  public Optional<EarlybirdRequest> normalizeRequest(EarlybirdRequestContext requestContext) {\n    return Optional.fromNullable(\n        TopTweetsCacheUtil.normalizeTopTweetsRequestForCache(requestContext.getRequest()));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/TopTweetsQueryCachePredicate.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport com.twitter.search.common.caching.filter.QueryCachePredicate;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.earlybird.common.EarlybirdRequestUtil;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\n\npublic class TopTweetsQueryCachePredicate extends QueryCachePredicate<EarlybirdRequestContext> {\n  private final SearchDecider decider;\n  private final String toptweetsCacheEnabledDeciderKey;\n\n  public TopTweetsQueryCachePredicate(SearchDecider decider, String normalizedSearchRootName) {\n    this.decider = decider;\n    this.toptweetsCacheEnabledDeciderKey = \"toptweets_cache_enabled_\" + normalizedSearchRootName;\n  }\n\n  @Override\n  public Boolean shouldQueryCache(EarlybirdRequestContext requestContext) {\n    return EarlybirdRequestType.TOP_TWEETS == requestContext.getEarlybirdRequestType()\n        && EarlybirdRequestUtil.isCachingAllowed(requestContext.getRequest())\n        && decider.isAvailable(toptweetsCacheEnabledDeciderKey);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/caching/TopTweetsServicePostProcessor.java",
    "content": "package com.twitter.search.earlybird_root.caching;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.caching.Cache;\nimport com.twitter.search.common.caching.TopTweetsCacheUtil;\nimport com.twitter.search.common.caching.filter.ServicePostProcessor;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\n\nimport static com.google.common.base.Preconditions.checkNotNull;\n\npublic class TopTweetsServicePostProcessor\n    extends ServicePostProcessor<EarlybirdRequestContext, EarlybirdResponse> {\n  private static final Logger LOG = LoggerFactory.getLogger(TopTweetsServicePostProcessor.class);\n\n  public static final int CACHE_AGE_IN_MS = 600000;\n  public static final int NO_RESULT_CACHE_AGE_IN_MS = 300000;\n\n  private final Cache<EarlybirdRequest, EarlybirdResponse> cache;\n\n  public TopTweetsServicePostProcessor(Cache<EarlybirdRequest, EarlybirdResponse> cache) {\n    this.cache = checkNotNull(cache);\n  }\n\n  @Override\n  public void processServiceResponse(EarlybirdRequestContext requestContext,\n                                     EarlybirdResponse serviceResponse) {\n\n    EarlybirdRequest originalRequest = requestContext.getRequest();\n    LOG.debug(\"Writing to top tweets cache. Request: {}, Response: {}\",\n        originalRequest, serviceResponse);\n    TopTweetsCacheUtil.cacheResults(originalRequest,\n        serviceResponse,\n        cache,\n        CACHE_AGE_IN_MS,\n        NO_RESULT_CACHE_AGE_IN_MS);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/collectors/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/log4j\",\n        \"src/java/com/twitter/search/common/relevance:utils\",\n        \"src/java/com/twitter/search/common/util/earlybird\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/collectors/MultiwayMergeCollector.java",
    "content": "package com.twitter.search.earlybird_root.collectors;\n\nimport java.util.Collections;\nimport java.util.Comparator;\nimport java.util.List;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\n\n/**\n * Generic MultiwayMergeCollector class for doing k-way merge of earlybird responses\n * that takes a comparator and returns a list of results sorted by the comparator.\n */\npublic abstract class MultiwayMergeCollector<T> {\n  protected static final Logger LOG = LoggerFactory.getLogger(MultiwayMergeCollector.class);\n\n  private final Comparator<T> resultComparator;\n  private final int numResponsesToMerge;\n  private final List<T> results = Lists.newArrayList();\n  private int numResponsesAdded = 0;\n\n  /**\n   * Constructor that does multi way merge and takes in a custom predicate search result filter.\n   */\n  public MultiwayMergeCollector(int numResponses,\n                                Comparator<T> comparator) {\n    Preconditions.checkNotNull(comparator);\n    this.resultComparator = comparator;\n    this.numResponsesToMerge = numResponses;\n  }\n\n  /**\n   * Add a single response from one partition, updates stats.\n   *\n   * @param response response from one partition\n   */\n  public final void addResponse(EarlybirdResponse response) {\n    // On prod, does it ever happen we receive more responses than numPartitions ?\n    Preconditions.checkArgument(numResponsesAdded++ < numResponsesToMerge,\n        String.format(\"Attempting to merge more than %d responses\", numResponsesToMerge));\n    if (!isResponseValid(response)) {\n      return;\n    }\n    collectStats(response);\n    List<T> resultsFromResponse = collectResults(response);\n    if (resultsFromResponse != null && resultsFromResponse.size() > 0) {\n      results.addAll(resultsFromResponse);\n    }\n  }\n\n  /**\n   * Parse the EarlybirdResponse and retrieve list of results to be appended.\n   *\n   * @param response earlybird response from where results are extracted\n   * @return  resultsList to be appended\n   */\n  protected abstract List<T> collectResults(EarlybirdResponse response);\n\n  /**\n   * It is recommended that sub-class overrides this function to add custom logic to\n   * collect more stat and call this base function.\n   */\n  protected void collectStats(EarlybirdResponse response) {\n  }\n\n  /**\n   * Get full list of results, after addResponse calls have been invoked.\n   *\n   * @return list of results extracted from all EarlybirdResponses that have been collected so far\n   */\n  protected final List<T> getResultsList() {\n    Collections.sort(results, resultComparator);\n    return results;\n  }\n\n  protected abstract boolean isResponseValid(EarlybirdResponse response);\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/collectors/RecencyMergeCollector.java",
    "content": "package com.twitter.search.earlybird_root.collectors;\n\nimport java.util.Comparator;\nimport java.util.List;\n\nimport com.twitter.search.common.relevance.utils.ResultComparators;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\n\n/**\n * {@link RecencyMergeCollector} inherits {@link MultiwayMergeCollector} for the type\n * {@link com.twitter.search.earlybird.thrift.ThriftSearchResult} as the result type.\n * <p/>\n * It also implements two public methods to retrieve the top-k or all results.\n */\npublic class RecencyMergeCollector extends MultiwayMergeCollector<ThriftSearchResult> {\n\n  // Container for the final results array and also stats like numHitsProcessed etc...\n  protected final ThriftSearchResults finalResults = new ThriftSearchResults();\n\n  public RecencyMergeCollector(int numResponses) {\n    this(numResponses, ResultComparators.ID_COMPARATOR);\n  }\n\n  protected RecencyMergeCollector(int numResponses, Comparator<ThriftSearchResult> comparator) {\n    super(numResponses, comparator);\n  }\n\n  @Override\n  protected void collectStats(EarlybirdResponse response) {\n    super.collectStats(response);\n\n    ThriftSearchResults searchResults = response.getSearchResults();\n    if (searchResults.isSetNumHitsProcessed()) {\n      finalResults.setNumHitsProcessed(\n          finalResults.getNumHitsProcessed() + searchResults.getNumHitsProcessed());\n    }\n    if (searchResults.isSetNumPartitionsEarlyTerminated()) {\n      finalResults.setNumPartitionsEarlyTerminated(\n              finalResults.getNumPartitionsEarlyTerminated()\n                      + searchResults.getNumPartitionsEarlyTerminated());\n    }\n  }\n\n  @Override\n  protected final List<ThriftSearchResult> collectResults(EarlybirdResponse response) {\n    if (response != null\n        && response.isSetSearchResults()\n        && response.getSearchResults().getResultsSize() > 0) {\n      return response.getSearchResults().getResults();\n    } else {\n      return null;\n    }\n  }\n\n  /**\n   * Gets all the results that has been collected.\n   *\n   * @return {@link ThriftSearchResults} containing a list of results sorted by provided\n   *         comparator in descending order.\n   */\n  public final ThriftSearchResults getAllSearchResults() {\n    return finalResults.setResults(getResultsList());\n  }\n\n  @Override\n  protected final boolean isResponseValid(EarlybirdResponse response) {\n    if (response == null || !response.isSetSearchResults()) {\n      LOG.warn(\"searchResults was null: \" + response);\n      return false;\n    }\n    return true;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/collectors/RelevanceMergeCollector.java",
    "content": "package com.twitter.search.earlybird_root.collectors;\n\nimport com.twitter.search.common.relevance.utils.ResultComparators;\nimport com.twitter.search.common.util.earlybird.ThriftSearchResultsRelevanceStatsUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultsRelevanceStats;\n\n/**\n * RelevanceMergeCollector class extends (@link RecencyMergeCollector} to do k-way merge of\n * earlybird responses, but sorted by relevance score.\n *\n * Note that this is a superset of functionality found in\n * {@link com.twitter.search.blender.services.earlybird.relevance.RelevanceCollector}\n * If you make changes here, evaluate if they should be made in RelevanceCollector as well.\n */\npublic class RelevanceMergeCollector extends RecencyMergeCollector {\n\n  public RelevanceMergeCollector(int numResponses) {\n    super(numResponses, ResultComparators.SCORE_COMPARATOR);\n  }\n\n  @Override\n  protected void collectStats(EarlybirdResponse response) {\n    super.collectStats(response);\n\n    if (!response.getSearchResults().isSetRelevanceStats()) {\n      return;\n    }\n\n    if (!finalResults.isSetRelevanceStats()) {\n      finalResults.setRelevanceStats(new ThriftSearchResultsRelevanceStats());\n    }\n\n    ThriftSearchResultsRelevanceStats base = finalResults.getRelevanceStats();\n    ThriftSearchResultsRelevanceStats delta = response.getSearchResults().getRelevanceStats();\n\n    ThriftSearchResultsRelevanceStatsUtil.addRelevanceStats(base, delta);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/common/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/commons-lang\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"src/java/com/twitter/common/util:system-mocks\",\n        \"src/java/com/twitter/search/common/decider\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/partitioning/snowflakeparser\",\n        \"src/java/com/twitter/search/queryparser\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/thrift/com/twitter/context:twitter-context-scala\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n        \"src/thrift/com/twitter/search/common:constants-java\",\n        \"src/thrift/com/twitter/search/common:features-java\",\n        \"twitter-context/src/main/scala\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/common/ClientErrorException.java",
    "content": "package com.twitter.search.earlybird_root.common;\n\npublic class ClientErrorException extends RuntimeException {\n\n  public ClientErrorException() {\n  }\n\n  public ClientErrorException(String message) {\n    super(message);\n  }\n\n  public ClientErrorException(String message, Throwable cause) {\n    super(message, cause);\n  }\n\n  public ClientErrorException(Throwable cause) {\n    super(cause);\n  }\n\n  public ClientErrorException(String message, Throwable cause,\n                              boolean enableSuppression, boolean writableStackTrace) {\n    super(message, cause, enableSuppression, writableStackTrace);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/common/EarlybirdFeatureSchemaMerger.java",
    "content": "package com.twitter.search.earlybird_root.common;\n\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.TreeSet;\nimport java.util.concurrent.ConcurrentHashMap;\n\nimport javax.annotation.concurrent.ThreadSafe;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableList;\nimport com.google.common.collect.Maps;\n\nimport org.apache.commons.lang.mutable.MutableInt;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchema;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchemaSpecifier;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.ThriftSearchRankingMode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\n\n@ThreadSafe\npublic class EarlybirdFeatureSchemaMerger {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdFeatureSchemaMerger.class);\n\n  private static final SearchLongGauge NUM_FEATURE_SCHEMAS_MAP = SearchLongGauge.export(\n      \"earlybird_feature_schema_cached_cnt\");\n\n  private class Stats {\n    public final SearchCounter fieldFormatResponses;\n    public final SearchCounter mapFormatResponses;\n    public final SearchCounter mapFormatSavedSchemaResponses;\n    public final SearchCounter mapFormatAllDownstreamMissingSchema;\n    public final SearchCounter mapFormatOneDownstreamMissingSchema;\n    public final SearchCounter mapFormatSchemaCachedMismatch;\n    public final SearchCounter numInvalidRankingModeRequests;\n    public final SearchCounter numEmptyResponses;\n\n    public Stats(String prefix) {\n      this.fieldFormatResponses =\n          SearchCounter.export(\n              \"earlybird_feature_schema_\" + prefix + \"_field_format_feature_responses\");\n      this.mapFormatResponses =\n          SearchCounter.export(\n              \"earlybird_feature_schema_\" + prefix + \"_map_format_feature_responses\");\n      this.mapFormatSavedSchemaResponses =\n          SearchCounter.export(\n              \"earlybird_feature_schema_\" + prefix + \"_map_format_feature_saved_schema_responses\");\n      this.mapFormatAllDownstreamMissingSchema =\n          SearchCounter.export(\n              \"earlybird_feature_schema_\" + prefix\n                  + \"_map_format_feature_all_downstream_missing_schema_error\");\n      this.mapFormatOneDownstreamMissingSchema =\n          SearchCounter.export(\n              \"earlybird_feature_schema_\" + prefix\n                  + \"_map_format_feature_one_downstream_missing_schema_error\");\n      this.mapFormatSchemaCachedMismatch =\n          SearchCounter.export(\n              \"earlybird_feature_schema_\" + prefix\n                  + \"_map_format_feature_schema_cached_mismatch_error\");\n      this.numInvalidRankingModeRequests =\n          SearchCounter.export(\n              \"earlybird_feature_schema_\" + prefix + \"_num_invalid_ranking_mode_requests\");\n      this.numEmptyResponses =\n          SearchCounter.export(\n              \"earlybird_feature_schema_\" + prefix\n                  + \"_num_empty_response_without_schema\");\n    }\n  }\n\n  private final ConcurrentHashMap<ThriftSearchFeatureSchemaSpecifier, ThriftSearchFeatureSchema>\n      featureSchemas = new ConcurrentHashMap<>();\n  private final ConcurrentHashMap<String, Stats> mergeStats = new ConcurrentHashMap<>();\n\n  /**\n   * Get all available cache schema list indicated by the schema specifier.\n   * @return identifiers for all the cached schema\n   */\n  public List<ThriftSearchFeatureSchemaSpecifier> getAvailableSchemaList() {\n    return ImmutableList.copyOf(featureSchemas.keySet());\n  }\n\n  /**\n   * Iterate all the responses and collect and cache feature schemas from response.\n   * Set the feature schema for the response in searchResults if needed.\n   * (This is done inside earlybird roots)\n   *\n   * @param searchResults the response\n   * @param requestContext the request, which should record the client cached feature schemas\n   * @param statPrefix the stats prefix string\n   * @param successfulResponses all successfull responses from downstream\n   */\n  public void collectAndSetFeatureSchemaInResponse(\n      ThriftSearchResults searchResults,\n      EarlybirdRequestContext requestContext,\n      String statPrefix,\n      List<EarlybirdResponse> successfulResponses) {\n    Stats stats = getOrCreateMergeStat(statPrefix);\n    EarlybirdRequest request = requestContext.getRequest();\n    if (!request.isSetSearchQuery()\n          || !request.getSearchQuery().isSetResultMetadataOptions()\n          || !request.getSearchQuery().getResultMetadataOptions().isReturnSearchResultFeatures()) {\n      // If the client does not want to get all features in map format, do not do anything.\n      stats.fieldFormatResponses.increment();\n      return;\n    }\n\n    // Find the most occurred schema from per-merge responses and return it in the post-merge\n    // response.\n    ThriftSearchFeatureSchemaSpecifier schemaMostOccurred = findMostOccurredSchema(\n        stats, request, successfulResponses);\n    if (schemaMostOccurred == null) {\n      return;\n    }\n\n    Set<ThriftSearchFeatureSchemaSpecifier> availableSchemasInClient =\n        requestContext.getFeatureSchemasAvailableInClient();\n    if (availableSchemasInClient != null && availableSchemasInClient.contains(schemaMostOccurred)) {\n      // The client already knows the schema that we used for this response, so we don't need to\n      // send it the full schema, just the ThriftSearchFeatureSchemaSpecifier.\n      ThriftSearchFeatureSchema schema = new ThriftSearchFeatureSchema();\n      schema.setSchemaSpecifier(schemaMostOccurred);\n      searchResults.setFeatureSchema(schema);\n      stats.mapFormatResponses.increment();\n      stats.mapFormatSavedSchemaResponses.increment();\n    } else {\n      ThriftSearchFeatureSchema schema = featureSchemas.get(schemaMostOccurred);\n      if (schema != null) {\n        Preconditions.checkState(schema.isSetEntries());\n        Preconditions.checkState(schema.isSetSchemaSpecifier());\n        searchResults.setFeatureSchema(schema);\n        stats.mapFormatResponses.increment();\n      } else {\n        stats.mapFormatSchemaCachedMismatch.increment();\n        LOG.error(\"The feature schema cache misses the schema entry {} it should cache for {}\",\n            schemaMostOccurred, request);\n      }\n    }\n  }\n\n  /**\n   * Merge the feature schema from each cluster's response and return it to the client.\n   * (This is done inside superroot)\n   * @param requestContext the search request context\n   * @param mergedResponse the merged result inside the superroot\n   * @param realtimeResponse the realtime tier resposne\n   * @param protectedResponse the protected tier response\n   * @param fullArchiveResponse the full archive tier response\n   * @param statsPrefix\n   */\n  public void mergeFeatureSchemaAcrossClusters(\n      EarlybirdRequestContext requestContext,\n      EarlybirdResponse mergedResponse,\n      String statsPrefix,\n      EarlybirdResponse realtimeResponse,\n      EarlybirdResponse protectedResponse,\n      EarlybirdResponse fullArchiveResponse) {\n    Stats superrootStats = getOrCreateMergeStat(statsPrefix);\n\n    // Only try to merge feature schema if there are search results.\n    ThriftSearchResults mergedResults = Preconditions.checkNotNull(\n        mergedResponse.getSearchResults());\n    if (mergedResults.getResults().isEmpty()) {\n      mergedResults.unsetFeatureSchema();\n      superrootStats.numEmptyResponses.increment();\n      return;\n    }\n\n    EarlybirdRequest request = requestContext.getRequest();\n    if (!request.isSetSearchQuery()\n        || !request.getSearchQuery().isSetResultMetadataOptions()\n        || !request.getSearchQuery().getResultMetadataOptions().isReturnSearchResultFeatures()) {\n      mergedResults.unsetFeatureSchema();\n\n      // If the client does not want to get all features in map format, do not do anything.\n      superrootStats.fieldFormatResponses.increment();\n      return;\n    }\n    if (request.getSearchQuery().getRankingMode() != ThriftSearchRankingMode.RELEVANCE\n        && request.getSearchQuery().getRankingMode() != ThriftSearchRankingMode.TOPTWEETS\n        && request.getSearchQuery().getRankingMode() != ThriftSearchRankingMode.RECENCY) {\n      mergedResults.unsetFeatureSchema();\n\n      // Only RELEVANCE, TOPTWEETS and RECENCY requests might need a feature schema in the response.\n      superrootStats.numInvalidRankingModeRequests.increment();\n      LOG.warn(\"Request asked for feature schema, but has incorrect ranking mode: {}\", request);\n      return;\n    }\n    superrootStats.mapFormatResponses.increment();\n\n    ThriftSearchFeatureSchema schema = updateReturnSchemaForClusterResponse(\n        null, realtimeResponse, request, superrootStats);\n    schema = updateReturnSchemaForClusterResponse(\n        schema, protectedResponse, request, superrootStats);\n    schema = updateReturnSchemaForClusterResponse(\n        schema, fullArchiveResponse, request, superrootStats);\n\n    if (schema != null) {\n      if (requestContext.getFeatureSchemasAvailableInClient() != null\n          && requestContext.getFeatureSchemasAvailableInClient().contains(\n          schema.getSchemaSpecifier())) {\n        mergedResults.setFeatureSchema(\n            new ThriftSearchFeatureSchema().setSchemaSpecifier(schema.getSchemaSpecifier()));\n      } else {\n        mergedResults.setFeatureSchema(schema);\n      }\n    } else {\n      superrootStats.mapFormatAllDownstreamMissingSchema.increment();\n      LOG.error(\"The response for request {} is missing feature schema from all clusters\", request);\n    }\n  }\n\n  /**\n   * Add the schema to both the schema map and and the schema list if it is not there yet.\n   *\n   * @param schema the feature schema for search results\n   */\n  private void addNewSchema(ThriftSearchFeatureSchema schema) {\n    if (!schema.isSetEntries()\n        || !schema.isSetSchemaSpecifier()\n        || featureSchemas.containsKey(schema.getSchemaSpecifier())) {\n      return;\n    }\n\n    synchronized (this) {\n      String oldExportedSchemaName = null;\n      if (!featureSchemas.isEmpty()) {\n        oldExportedSchemaName = getExportSchemasName();\n      }\n\n      if (featureSchemas.putIfAbsent(schema.getSchemaSpecifier(), schema) == null) {\n        LOG.info(\"Add new feature schema {} into the list\", schema);\n        NUM_FEATURE_SCHEMAS_MAP.set(featureSchemas.size());\n\n        if (oldExportedSchemaName != null) {\n          SearchLongGauge.export(oldExportedSchemaName).reset();\n        }\n        SearchLongGauge.export(getExportSchemasName()).set(1);\n        LOG.info(\"Expanded feature schema: {}\", ImmutableList.copyOf(featureSchemas.keySet()));\n      }\n    }\n  }\n\n  private String getExportSchemasName() {\n    StringBuilder builder = new StringBuilder(\"earlybird_feature_schema_cached\");\n    TreeSet<String> exportedVersions = new TreeSet<>();\n\n    // We do not need checksum for exported vars as all cached schemas are from the majority of the\n    // responses.\n    featureSchemas.keySet().stream().forEach(key -> exportedVersions.add(key.getVersion()));\n    exportedVersions.stream().forEach(version -> {\n      builder.append('_');\n      builder.append(version);\n    });\n    return builder.toString();\n  }\n\n  // Get the updated the feature schema based on the earlybird response from the search cluster.\n  // . If the existingSchema is not null, the function would return the existing schema.  Under the\n  //   situation, we would still check whether the feature in earlybird response is valid.\n  // . Otherwise, the function would extract the feature schema from the earlybird response.\n  private ThriftSearchFeatureSchema updateReturnSchemaForClusterResponse(\n      ThriftSearchFeatureSchema existingSchema,\n      EarlybirdResponse clusterResponse,\n      EarlybirdRequest request,\n      Stats stats) {\n    // If there is no response or search result for this cluster, do not update returned schema.\n    if ((clusterResponse == null) || !clusterResponse.isSetSearchResults()) {\n      return existingSchema;\n    }\n    ThriftSearchResults results = clusterResponse.getSearchResults();\n    if (results.getResults().isEmpty()) {\n      return existingSchema;\n    }\n\n    if (!results.isSetFeatureSchema() || !results.getFeatureSchema().isSetSchemaSpecifier()) {\n      stats.mapFormatOneDownstreamMissingSchema.increment();\n      LOG.error(\"The downstream response {} is missing feature schema for request {}\",\n          clusterResponse, request);\n      return existingSchema;\n    }\n\n    ThriftSearchFeatureSchema schema = results.getFeatureSchema();\n\n    // Even if existingSchema is already set, we would still try to cache the returned schema.\n    // In this way, the next time earlybird roots don't have to send the full schema back again.\n    if (schema.isSetEntries()) {\n      addNewSchema(schema);\n    } else if (featureSchemas.containsKey(schema.getSchemaSpecifier())) {\n      stats.mapFormatSavedSchemaResponses.increment();\n    } else {\n      stats.mapFormatSchemaCachedMismatch.increment();\n      LOG.error(\n          \"The feature schema cache misses the schema entry {}, it should cache {} in {}\",\n          schema.getSchemaSpecifier(), request, clusterResponse);\n    }\n\n    ThriftSearchFeatureSchema updatedSchema = existingSchema;\n    if (updatedSchema == null) {\n      updatedSchema = featureSchemas.get(schema.getSchemaSpecifier());\n      if (updatedSchema != null) {\n        Preconditions.checkState(updatedSchema.isSetEntries());\n        Preconditions.checkState(updatedSchema.isSetSchemaSpecifier());\n      }\n    }\n    return updatedSchema;\n  }\n\n  private ThriftSearchFeatureSchemaSpecifier findMostOccurredSchema(\n      Stats stats,\n      EarlybirdRequest request,\n      List<EarlybirdResponse> successfulResponses) {\n    boolean hasResults = false;\n    Map<ThriftSearchFeatureSchemaSpecifier, MutableInt> schemaCount =\n        Maps.newHashMapWithExpectedSize(successfulResponses.size());\n    for (EarlybirdResponse response : successfulResponses) {\n      if (!response.isSetSearchResults()\n          || response.getSearchResults().getResultsSize() == 0) {\n        continue;\n      }\n\n      hasResults = true;\n      if (response.getSearchResults().isSetFeatureSchema()) {\n        ThriftSearchFeatureSchema schema = response.getSearchResults().getFeatureSchema();\n        if (schema.isSetSchemaSpecifier()) {\n          MutableInt cnt = schemaCount.get(schema.getSchemaSpecifier());\n          if (cnt != null) {\n            cnt.increment();\n          } else {\n            schemaCount.put(schema.getSchemaSpecifier(), new MutableInt(1));\n          }\n\n          if (schema.isSetEntries()) {\n            addNewSchema(schema);\n          }\n        }\n      } else {\n        stats.mapFormatOneDownstreamMissingSchema.increment();\n        LOG.error(\"The downstream response {} is missing feature schema for request {}\",\n            response, request);\n      }\n    }\n\n    int numMostOccurred = 0;\n    ThriftSearchFeatureSchemaSpecifier schemaMostOccurred = null;\n    for (Map.Entry<ThriftSearchFeatureSchemaSpecifier, MutableInt> entry : schemaCount.entrySet()) {\n      if (entry.getValue().toInteger() > numMostOccurred) {\n        numMostOccurred = entry.getValue().toInteger();\n        schemaMostOccurred = entry.getKey();\n      }\n    }\n\n    if (schemaMostOccurred == null && hasResults) {\n      stats.mapFormatAllDownstreamMissingSchema.increment();\n      LOG.error(\"None of the downstream host returned feature schema for {}\", request);\n    }\n    return schemaMostOccurred;\n  }\n\n  private Stats getOrCreateMergeStat(String statPrefix) {\n    Stats stats = mergeStats.get(statPrefix);\n    if (stats == null) {\n      Stats newStats = new Stats(statPrefix);\n      stats = mergeStats.putIfAbsent(statPrefix, newStats);\n      if (stats == null) {\n        stats = newStats;\n      }\n    }\n    return stats;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/common/EarlybirdRequestContext.java",
    "content": "package com.twitter.search.earlybird_root.common;\n\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Set;\n\nimport javax.annotation.Nullable;\n\nimport scala.Option;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.ImmutableSet;\nimport com.google.common.collect.Sets;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.context.thriftscala.Viewer;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchemaSpecifier;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\n\n/**\n * A class that wraps a request and additional per-request data that should be passed to services.\n *\n * This class should be immutable. At the very least, it must be thread-safe. In practice, since\n * EarlybirdRequest is a mutable Thrift structure, the users of this class need to make sure that\n * once a request is used to create a RequestContext instance, it is not modified.\n *\n * If the request needs to be modified, a new RequestContext with the modified EarlybirdRequest\n * should be created.\n */\npublic final class EarlybirdRequestContext {\n\n  private static final String OVERRIDE_TIER_CONFIGS_DECIDER_KEY = \"use_override_tier_configs\";\n\n  /**\n   * Creates a new context with the provided earlybird request, and using the given decider.\n   */\n  public static EarlybirdRequestContext newContext(\n      EarlybirdRequest request,\n      SearchDecider decider,\n      Option<Viewer> twitterContextViewer,\n      Clock clock) throws QueryParserException {\n\n    // Try to capture created time as early as possible. For example, we want to account for query\n    // parsing time.\n    long createdTimeMillis = clock.nowMillis();\n\n    boolean useOverrideTierConfig = decider.isAvailable(OVERRIDE_TIER_CONFIGS_DECIDER_KEY);\n\n    Query parsedQuery = QueryParsingUtils.getParsedQuery(request);\n\n    return new EarlybirdRequestContext(\n        request,\n        parsedQuery,\n        useOverrideTierConfig,\n        createdTimeMillis,\n        twitterContextViewer);\n  }\n\n  /**\n   * Intersection of the userID and the flock response, which is set in the followedUserIds field.\n   * This is used for protected cluster.\n   */\n  public static EarlybirdRequestContext newContextWithRestrictFromUserIdFilter64(\n      EarlybirdRequestContext requestContext) {\n    Preconditions.checkArgument(requestContext.getRequest().isSetFollowedUserIds());\n\n    EarlybirdRequest request = requestContext.getRequest().deepCopy();\n    List<Long> toIntersect = request.getFollowedUserIds();\n    ThriftSearchQuery searchQuery = request.getSearchQuery();\n    if (!searchQuery.isSetFromUserIDFilter64()) {\n      searchQuery.setFromUserIDFilter64(new ArrayList<>(toIntersect));\n    } else {\n      Set<Long> intersection = Sets.intersection(\n          Sets.newHashSet(searchQuery.getFromUserIDFilter64()),\n          Sets.newHashSet(toIntersect));\n      searchQuery.setFromUserIDFilter64(new ArrayList<>(intersection));\n    }\n\n    return new EarlybirdRequestContext(requestContext, request, requestContext.getParsedQuery());\n  }\n\n  /**\n   * Makes an exact copy of the provided request context, by cloning the underlying earlybird\n   * request.\n   */\n  public static EarlybirdRequestContext copyRequestContext(\n      EarlybirdRequestContext requestContext,\n      Query parsedQuery) {\n    return new EarlybirdRequestContext(requestContext, parsedQuery);\n  }\n\n  /**\n   * Creates a new context with the provided request, context and reset both the feature schemas\n   * cached in client and the feature schemas cached in the local cache.\n   */\n  public static EarlybirdRequestContext newContext(\n      EarlybirdRequest oldRequest,\n      EarlybirdRequestContext oldRequestContext,\n      List<ThriftSearchFeatureSchemaSpecifier> featureSchemasAvailableInCache,\n      List<ThriftSearchFeatureSchemaSpecifier> featureSchemasAvailableInClient) {\n    EarlybirdRequest request = oldRequest.deepCopy();\n    request.getSearchQuery().getResultMetadataOptions()\n        .setFeatureSchemasAvailableInClient(featureSchemasAvailableInCache);\n\n    ImmutableSet<ThriftSearchFeatureSchemaSpecifier> featureSchemaSetAvailableInClient = null;\n    if (featureSchemasAvailableInClient != null) {\n      featureSchemaSetAvailableInClient = ImmutableSet.copyOf(featureSchemasAvailableInClient);\n    }\n\n    return new EarlybirdRequestContext(\n        request,\n        EarlybirdRequestType.of(request),\n        oldRequestContext.getParsedQuery(),\n        oldRequestContext.useOverrideTierConfig(),\n        oldRequestContext.getCreatedTimeMillis(),\n        oldRequestContext.getTwitterContextViewer(),\n        featureSchemaSetAvailableInClient);\n  }\n\n  public EarlybirdRequestContext deepCopy() {\n    return new EarlybirdRequestContext(request.deepCopy(), parsedQuery, useOverrideTierConfig,\n        createdTimeMillis, twitterContextViewer);\n  }\n\n  private final EarlybirdRequest request;\n  // EarlybirdRequestType should not change for a given request. Computing it once here so that we\n  // don't need to compute it from the request every time we want to use it.\n  private final EarlybirdRequestType earlybirdRequestType;\n  // The parsed query matching the serialized query in the request. May be null if the request does\n  // not contain a serialized query.\n  // If a request's serialized query needs to be rewritten for any reason, a new\n  // EarlybirdRequestContext should be created, with a new EarlybirdRequest (with a new serialized\n  // query), and a new parsed query (matching the new serialized query).\n  @Nullable\n  private final Query parsedQuery;\n  private final boolean useOverrideTierConfig;\n  private final long createdTimeMillis;\n  private final Option<Viewer> twitterContextViewer;\n\n  @Nullable\n  private final ImmutableSet<ThriftSearchFeatureSchemaSpecifier> featureSchemasAvailableInClient;\n\n  private EarlybirdRequestContext(\n      EarlybirdRequest request,\n      Query parsedQuery,\n      boolean useOverrideTierConfig,\n      long createdTimeMillis,\n      Option<Viewer> twitterContextViewer) {\n    this(request,\n        EarlybirdRequestType.of(request),\n        parsedQuery,\n        useOverrideTierConfig,\n        createdTimeMillis,\n        twitterContextViewer,\n        null);\n  }\n\n  private EarlybirdRequestContext(\n      EarlybirdRequest request,\n      EarlybirdRequestType earlybirdRequestType,\n      Query parsedQuery,\n      boolean useOverrideTierConfig,\n      long createdTimeMillis,\n      Option<Viewer> twitterContextViewer,\n      @Nullable ImmutableSet<ThriftSearchFeatureSchemaSpecifier> featureSchemasAvailableInClient) {\n    this.request = Preconditions.checkNotNull(request);\n    this.earlybirdRequestType = earlybirdRequestType;\n    this.parsedQuery = parsedQuery;\n    this.useOverrideTierConfig = useOverrideTierConfig;\n    this.createdTimeMillis = createdTimeMillis;\n    this.twitterContextViewer = twitterContextViewer;\n    this.featureSchemasAvailableInClient = featureSchemasAvailableInClient;\n  }\n\n  private EarlybirdRequestContext(EarlybirdRequestContext otherContext, Query otherParsedQuery) {\n    this(otherContext, otherContext.getRequest().deepCopy(), otherParsedQuery);\n  }\n\n  private EarlybirdRequestContext(EarlybirdRequestContext otherContext,\n                                  EarlybirdRequest otherRequest,\n                                  Query otherParsedQuery) {\n    this(otherRequest,\n        otherContext.earlybirdRequestType,\n        otherParsedQuery,\n        otherContext.useOverrideTierConfig,\n        otherContext.createdTimeMillis,\n        otherContext.twitterContextViewer,\n        null);\n\n    Preconditions.checkState(request.isSetSearchQuery());\n    this.request.getSearchQuery().setSerializedQuery(otherParsedQuery.serialize());\n  }\n\n  public EarlybirdRequest getRequest() {\n    return request;\n  }\n\n  public boolean useOverrideTierConfig() {\n    return useOverrideTierConfig;\n  }\n\n  public EarlybirdRequestType getEarlybirdRequestType() {\n    return earlybirdRequestType;\n  }\n\n  @Nullable\n  public Query getParsedQuery() {\n    return parsedQuery;\n  }\n\n  public long getCreatedTimeMillis() {\n    return createdTimeMillis;\n  }\n\n  public Option<Viewer> getTwitterContextViewer() {\n    return twitterContextViewer;\n  }\n\n  @Nullable\n  public Set<ThriftSearchFeatureSchemaSpecifier> getFeatureSchemasAvailableInClient() {\n    return featureSchemasAvailableInClient;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/common/EarlybirdRequestType.java",
    "content": "package com.twitter.search.earlybird_root.common;\n\nimport javax.annotation.Nonnull;\n\nimport com.twitter.search.common.constants.thriftjava.ThriftQuerySource;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.ThriftSearchRankingMode;\n\n/**\n * Earlybird roots distinguish these types of requests and treat them differently.\n */\npublic enum EarlybirdRequestType {\n  FACETS,\n  RECENCY,\n  RELEVANCE,\n  STRICT_RECENCY,\n  TERM_STATS,\n  TOP_TWEETS;\n\n  /**\n   * Returns the type of the given requests.\n   */\n  @Nonnull\n  public static EarlybirdRequestType of(EarlybirdRequest request) {\n    if (request.isSetFacetRequest()) {\n      return FACETS;\n    } else if (request.isSetTermStatisticsRequest()) {\n      return TERM_STATS;\n    } else if (request.isSetSearchQuery() && request.getSearchQuery().isSetRankingMode()) {\n      ThriftSearchRankingMode rankingMode = request.getSearchQuery().getRankingMode();\n      switch (rankingMode) {\n        case RECENCY:\n          if (shouldUseStrictRecency(request)) {\n            return STRICT_RECENCY;\n          } else {\n            return RECENCY;\n          }\n        case RELEVANCE:\n          return RELEVANCE;\n        case TOPTWEETS:\n          return TOP_TWEETS;\n        default:\n          throw new IllegalArgumentException();\n      }\n    } else {\n      throw new UnsupportedOperationException();\n    }\n  }\n\n  private static boolean shouldUseStrictRecency(EarlybirdRequest request) {\n    // For now, we decide to do strict merging solely based on the QuerySource, and only for GNIP.\n    return request.isSetQuerySource() && request.getQuerySource() == ThriftQuerySource.GNIP;\n  }\n\n  private final String normalizedName;\n\n  EarlybirdRequestType() {\n    this.normalizedName = name().toLowerCase();\n  }\n\n  /**\n   * Returns the \"normalized\" name of this request type, that can be used for stat and decider\n   * names.\n   */\n  public String getNormalizedName() {\n    return normalizedName;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/common/EarlybirdRequestUtil.java",
    "content": "package com.twitter.search.earlybird_root.common;\n\nimport com.google.common.base.Optional;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.partitioning.snowflakeparser.SnowflakeIdParser;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.util.IdTimeRanges;\n\npublic final class EarlybirdRequestUtil {\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdRequestUtil.class);\n\n  private EarlybirdRequestUtil() {\n  }\n\n  /**\n   * Returns the max ID specified in the query. The max ID is determined based on the max_id\n   * operator, and the returned value is an inclusive max ID (that is, the returned response is\n   * allowed to have a tweet with this ID).\n   *\n   * If the query is null, could not be parsed or does not have a max_id operator, Optional.absent()\n   * is returned.\n   *\n   * @param query The query.\n   * @return The max ID specified in the given query (based on the max_id operator).\n   */\n  public static Optional<Long> getRequestMaxId(Query query) {\n    if (query == null) {\n      return Optional.absent();\n    }\n\n    IdTimeRanges idTimeRanges = null;\n    try {\n      idTimeRanges = IdTimeRanges.fromQuery(query);\n    } catch (QueryParserException e) {\n      LOG.warn(\"Exception while getting max_id/until_time from query: \" + query, e);\n    }\n\n    if (idTimeRanges == null) {\n      // An exception was thrown or the query doesn't accept the boundary operators.\n      return Optional.absent();\n    }\n\n    return idTimeRanges.getMaxIDInclusive();\n  }\n\n  /**\n   * Returns the max ID specified in the query, based on the until_time operator. The returned ID\n   * is inclusive (that is, the returned response is allowed to have a tweet with this ID).\n   *\n   * If the query is null, could not be parsed or does not have an until_time operator,\n   * Optional.absent() is returned.\n   *\n   * @param query The query.\n   * @return The max ID specified in the given query (based on the until_time operator).\n   */\n  public static Optional<Long> getRequestMaxIdFromUntilTime(Query query) {\n    if (query == null) {\n      return Optional.absent();\n    }\n\n    IdTimeRanges idTimeRanges = null;\n    try {\n      idTimeRanges = IdTimeRanges.fromQuery(query);\n    } catch (QueryParserException e) {\n      LOG.warn(\"Exception while getting max_id/until_time from query: \" + query, e);\n    }\n\n    if (idTimeRanges == null) {\n      // An exception was thrown or the query doesn't accept the boundary operators.\n      return Optional.absent();\n    }\n\n    Optional<Integer> queryUntilTimeExclusive = idTimeRanges.getUntilTimeExclusive();\n    Optional<Long> maxId = Optional.absent();\n    if (queryUntilTimeExclusive.isPresent()) {\n      long timestampMillis = queryUntilTimeExclusive.get() * 1000L;\n      if (SnowflakeIdParser.isUsableSnowflakeTimestamp(timestampMillis)) {\n        // Convert timestampMillis to an ID, and subtract 1, because the until_time operator is\n        // exclusive, and we need to return an inclusive max ID.\n        maxId = Optional.of(SnowflakeIdParser.generateValidStatusId(timestampMillis, 0) - 1);\n      }\n    }\n    return maxId;\n  }\n\n  /**\n   * Creates a copy of the given EarlybirdRequest and unsets all fields that are used\n   * only by the SuperRoot.\n   */\n  public static EarlybirdRequest unsetSuperRootFields(\n      EarlybirdRequest request, boolean unsetFollowedUserIds) {\n    EarlybirdRequest newRequest = request.deepCopy();\n    newRequest.unsetGetOlderResults();\n    newRequest.unsetGetProtectedTweetsOnly();\n    if (unsetFollowedUserIds) {\n      newRequest.unsetFollowedUserIds();\n    }\n    newRequest.unsetAdjustedProtectedRequestParams();\n    newRequest.unsetAdjustedFullArchiveRequestParams();\n    return newRequest;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/common/EarlybirdServiceResponse.java",
    "content": "package com.twitter.search.earlybird_root.common;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\n\n/**\n * A class that wraps an EarlybirdResponse and a flag that determines if a request was sent to a\n * service.\n */\npublic final class EarlybirdServiceResponse {\n  public static enum ServiceState {\n    // The service was called (or will be called).\n    SERVICE_CALLED(true),\n\n    // The service is not available (turned off by a decider, for example).\n    SERVICE_NOT_AVAILABLE(false),\n\n    // The client did not request results from this service.\n    SERVICE_NOT_REQUESTED(false),\n\n    // The service is available and the client wants results from this service, but the service\n    // was not called (because we got enough results from other services, for example).\n    SERVICE_NOT_CALLED(false);\n\n    private final boolean serviceWasCalled;\n\n    private ServiceState(boolean serviceWasCalled) {\n      this.serviceWasCalled = serviceWasCalled;\n    }\n\n    public boolean serviceWasCalled() {\n      return serviceWasCalled;\n    }\n\n    public boolean serviceWasRequested() {\n      return this != SERVICE_NOT_REQUESTED;\n    }\n\n  }\n\n  private final EarlybirdResponse earlybirdResponse;\n  private final ServiceState serviceState;\n\n  private EarlybirdServiceResponse(@Nullable EarlybirdResponse earlybirdResponse,\n                                   ServiceState serviceState) {\n    this.earlybirdResponse = earlybirdResponse;\n    this.serviceState = serviceState;\n    if (!serviceState.serviceWasCalled()) {\n      Preconditions.checkArgument(earlybirdResponse == null);\n    }\n  }\n\n  /**\n   * Creates a new EarlybirdServiceResponse instance, indicating that the service was not called.\n   *\n   * @param serviceState The state of the service.\n   * @return a new EarlybirdServiceResponse instance, indicating that the service was not called.\n   */\n  public static EarlybirdServiceResponse serviceNotCalled(ServiceState serviceState) {\n    Preconditions.checkArgument(!serviceState.serviceWasCalled());\n    return new EarlybirdServiceResponse(null, serviceState);\n  }\n\n  /**\n   * Creates a new EarlybirdServiceResponse instance that wraps the given earlybird response.\n   *\n   * @param earlybirdResponse The EarlybirdResponse instance returned by the service.\n   * @return a new EarlybirdServiceResponse instance that wraps the given earlybird response.\n   */\n  public static EarlybirdServiceResponse serviceCalled(EarlybirdResponse earlybirdResponse) {\n    return new EarlybirdServiceResponse(earlybirdResponse, ServiceState.SERVICE_CALLED);\n  }\n\n  /** Returns the wrapped earlybird response. */\n  @Nullable\n  public EarlybirdResponse getResponse() {\n    return earlybirdResponse;\n  }\n\n  /** Returns the state of the service. */\n  public ServiceState getServiceState() {\n    return serviceState;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/common/InjectionNames.java",
    "content": "package com.twitter.search.earlybird_root.common;\n\npublic final class InjectionNames {\n\n  public static final String FULL_ARCHIVE = \"full_archive\";\n  public static final String REALTIME = \"realtime\";\n  public static final String PROTECTED = \"protected\";\n\n  private InjectionNames() { }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/common/QueryParsingUtils.java",
    "content": "package com.twitter.search.earlybird_root.common;\n\nimport java.util.concurrent.TimeUnit;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.queryparser.parser.SerializedQueryParser;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.util.Future;\n\n/**\n * Common utils for parsing serialized queries, and handling query parser exceptions.\n */\npublic final class QueryParsingUtils {\n\n  private static final Logger LOG = LoggerFactory.getLogger(QueryParsingUtils.class);\n\n  @VisibleForTesting\n  public static final SearchCounter QUERYPARSE_COUNT =\n      SearchCounter.export(\"root_queryparse_count\");\n  private static final SearchTimerStats QUERYPARSE_TIMER =\n      SearchTimerStats.export(\"root_queryparse_time\", TimeUnit.NANOSECONDS, false, true);\n  private static final SearchCounter NO_PARSED_QUERY_COUNT =\n      SearchCounter.export(\"root_no_parsed_query_count\");\n\n  private QueryParsingUtils() { }\n\n  /**\n   * Takes an earlybird request, and parses its serialized query (if it is set).\n   * Expects the required ThriftSearchQuery to be set on the passed in EarlybirdRequest.\n   *\n   * @param request the earlybird request to parse.\n   * @return null if the request does not specify a serialized query.\n   * @throws QueryParserException if querry parsing fails.\n   */\n  @Nullable\n  static Query getParsedQuery(EarlybirdRequest request) throws QueryParserException {\n    // searchQuery is required on EarlybirdRequest.\n    Preconditions.checkState(request.isSetSearchQuery());\n    Query parsedQuery;\n    if (request.getSearchQuery().isSetSerializedQuery()) {\n      long startTime = System.nanoTime();\n      try {\n        String serializedQuery = request.getSearchQuery().getSerializedQuery();\n\n        parsedQuery = new SerializedQueryParser().parse(serializedQuery);\n      } finally {\n        QUERYPARSE_COUNT.increment();\n        QUERYPARSE_TIMER.timerIncrement(System.nanoTime() - startTime);\n      }\n    } else {\n      NO_PARSED_QUERY_COUNT.increment();\n      parsedQuery = null;\n    }\n    return parsedQuery;\n  }\n\n  /**\n   * Creates a new EarlybirdResponse with a CLIENT_ERROR response code, to be used as a response\n   * to a request where we failed to parse a user passed in serialized query.\n   */\n  public static Future<EarlybirdResponse> newClientErrorResponse(\n      EarlybirdRequest request,\n      QueryParserException e) {\n\n    String msg = \"Failed to parse query\";\n    LOG.warn(msg, e);\n\n    EarlybirdResponse errorResponse =\n        new EarlybirdResponse(EarlybirdResponseCode.CLIENT_ERROR, 0);\n    errorResponse.setDebugString(msg + \": \" + e.getMessage());\n    return Future.value(errorResponse);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/common/TwitterContextProvider.java",
    "content": "package com.twitter.search.earlybird_root.common;\n\nimport javax.inject.Singleton;\n\nimport scala.Option;\n\nimport com.twitter.context.TwitterContext;\nimport com.twitter.context.thriftscala.Viewer;\nimport com.twitter.search.TwitterContextPermit;\n\n/**\n * This class is needed to provide an easy way for unit tests to \"inject\"\n * a TwitterContext Viewer\n */\n@Singleton\npublic class TwitterContextProvider {\n  public Option<Viewer> get() {\n    return TwitterContext.acquire(TwitterContextPermit.get()).apply();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/config/BUILD.bazel",
    "content": "java_library(\n    sources = [\"*.java\"],\n    dependencies = [\n        \"src/java/com/twitter/common/util:system-mocks\",\n        \"src/java/com/twitter/search/earlybird/config\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/config/RootClusterBoundaryInfo.java",
    "content": "package com.twitter.search.earlybird_root.config;\n\nimport java.util.Date;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.search.earlybird.config.ServingRange;\nimport com.twitter.search.earlybird.config.TierServingBoundaryEndPoint;\n\n/**\n * Time boundary information for a root cluster.\n * Used by EarlybirdTimeRangeFilter.\n */\npublic class RootClusterBoundaryInfo implements ServingRange {\n\n  private final TierServingBoundaryEndPoint servingRangeSince;\n  private final TierServingBoundaryEndPoint servingRangeMax;\n\n  /**\n   * Build a time boundary information\n   */\n  public RootClusterBoundaryInfo(\n      Date startDate,\n      Date clusterEndDate,\n      String sinceIdBoundaryString,\n      String maxIdBoundaryString,\n      Clock clock) {\n    this.servingRangeSince = TierServingBoundaryEndPoint\n        .newTierServingBoundaryEndPoint(sinceIdBoundaryString, startDate, clock);\n    this.servingRangeMax = TierServingBoundaryEndPoint\n        .newTierServingBoundaryEndPoint(maxIdBoundaryString, clusterEndDate, clock);\n  }\n\n  public long getServingRangeSinceId() {\n    return servingRangeSince.getBoundaryTweetId();\n  }\n\n  public long getServingRangeMaxId() {\n    return servingRangeMax.getBoundaryTweetId();\n  }\n\n  public long getServingRangeSinceTimeSecondsFromEpoch() {\n    return servingRangeSince.getBoundaryTimeSecondsFromEpoch();\n  }\n\n  public long getServingRangeUntilTimeSecondsFromEpoch() {\n    return servingRangeMax.getBoundaryTimeSecondsFromEpoch();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/BUILD",
    "content": "java_library(\n    sources = [\"*.java\"],\n    platform = \"java8\",\n    tags = [\"bazel-compatible\"],\n    dependencies = [\n        \"3rdparty/jvm/com/google/guava\",\n        \"3rdparty/jvm/com/google/inject:guice\",\n        \"3rdparty/jvm/commons-io\",\n        \"3rdparty/jvm/org/slf4j:slf4j-api\",\n        \"snowflake/src/main/scala/com/twitter/snowflake/id\",\n        \"src/antlr/com/twitter/search/queryparser/antlr:queryparser-antlr\",\n        \"src/java/com/google/common/util/concurrent\",\n        \"src/java/com/twitter/common/collections\",\n        \"src/java/com/twitter/common/text/language:locale-util\",\n        \"src/java/com/twitter/common/util:system-mocks\",\n        \"src/java/com/twitter/common_internal/text/version\",\n        \"src/java/com/twitter/search/common/clientstats\",\n        \"src/java/com/twitter/search/common/decider\",\n        \"src/java/com/twitter/search/common/metrics\",\n        \"src/java/com/twitter/search/common/partitioning/snowflakeparser\",\n        \"src/java/com/twitter/search/common/root\",\n        \"src/java/com/twitter/search/common/schema/earlybird\",\n        \"src/java/com/twitter/search/common/util:finagleutil\",\n        \"src/java/com/twitter/search/common/util/date\",\n        \"src/java/com/twitter/search/common/util/earlybird\",\n        \"src/java/com/twitter/search/common/util/io/periodic\",\n        \"src/java/com/twitter/search/common/util/lang\",\n        \"src/java/com/twitter/search/common/util/thrift:text-protocol\",\n        \"src/java/com/twitter/search/earlybird/common\",\n        \"src/java/com/twitter/search/earlybird/config\",\n        \"src/java/com/twitter/search/earlybird_root/common\",\n        \"src/java/com/twitter/search/earlybird_root/quota\",\n        \"src/java/com/twitter/search/earlybird_root/validators\",\n        \"src/java/com/twitter/search/queryparser\",\n        \"src/java/com/twitter/search/queryparser/query:core-query-nodes\",\n        \"src/java/com/twitter/search/queryparser/query/search:search-query-nodes\",\n        \"src/thrift/com/twitter/context:twitter-context-scala\",\n        \"src/thrift/com/twitter/search:earlybird-java\",\n    ],\n)\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/ClientIdArchiveAccessFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.Optional;\n\nimport javax.inject.Inject;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird_root.quota.ClientIdQuotaManager;\nimport com.twitter.search.earlybird_root.quota.QuotaInfo;\nimport com.twitter.util.Future;\n\npublic class ClientIdArchiveAccessFilter extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n  private static final String UNAUTHORIZED_ARCHIVE_ACCESS_COUNTER_PATTERN =\n      \"unauthorized_access_to_full_archive_by_client_%s\";\n\n  private final ClientIdQuotaManager quotaManager;\n\n  /**\n   * Construct the filter by using ClientIdQuotaManager\n   */\n  @Inject\n  public ClientIdArchiveAccessFilter(ClientIdQuotaManager quotaManager) {\n    this.quotaManager = Preconditions.checkNotNull(quotaManager);\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(EarlybirdRequest request,\n                                         Service<EarlybirdRequest, EarlybirdResponse> service) {\n    String clientId = ClientIdUtil.getClientIdFromRequest(request);\n\n    Optional<QuotaInfo> quotaInfoOptional = quotaManager.getQuotaForClient(clientId);\n    QuotaInfo quotaInfo = quotaInfoOptional.orElseGet(quotaManager::getCommonPoolQuota);\n    if (!quotaInfo.hasArchiveAccess() && request.isGetOlderResults()) {\n      SearchCounter unauthorizedArchiveAccessCounter = SearchCounter.export(\n          String.format(UNAUTHORIZED_ARCHIVE_ACCESS_COUNTER_PATTERN, clientId));\n      unauthorizedArchiveAccessCounter.increment();\n\n      String message = String.format(\n          \"Client %s is not whitelisted for archive access. Request access at go/searchquota.\",\n          clientId);\n      EarlybirdResponse response = new EarlybirdResponse(\n          EarlybirdResponseCode.QUOTA_EXCEEDED_ERROR, 0)\n          .setDebugString(message);\n      return Future.value(response);\n    }\n    return service.apply(request);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/ClientIdQueryOperatorStatsFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.Arrays;\nimport java.util.EnumSet;\nimport java.util.HashSet;\nimport java.util.Set;\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.ConcurrentMap;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.clientstats.RequestCounters;\nimport com.twitter.search.common.clientstats.RequestCountersEventListener;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\nimport com.twitter.search.queryparser.visitors.DetectVisitor;\nimport com.twitter.util.Future;\n\n/**\n* This filter exports RequestCounters stats for each unique combination of client_id and\n* query_operator. RequestCounters produce 19 stats for each prefix, and we have numerous\n* clients and operators, so this filter can produce a large number of stats. To keep the\n* number of exported stats reasonable we use an allow list of operators. The list currently\n* includes the geo operators while we monitor the impacts of realtime geo filtering. See\n* SEARCH-33699 for project details.\n*\n* To find the stats look for query_client_operator_* exported by archive roots.\n*\n **/\n\npublic class ClientIdQueryOperatorStatsFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n\n  private static final Logger LOG = LoggerFactory.getLogger(ClientIdQueryOperatorStatsFilter.class);\n\n  public static final String COUNTER_PREFIX_PATTERN = \"query_client_operator_%s_%s\";\n  private final Clock clock;\n  private final ConcurrentMap<String, RequestCounters> requestCountersByClientIdAndOperator =\n      new ConcurrentHashMap<>();\n  private final Set<SearchOperator.Type> operatorsToRecordStatsFor = new HashSet<>(Arrays.asList(\n      SearchOperator.Type.GEO_BOUNDING_BOX,\n      SearchOperator.Type.GEOCODE,\n      SearchOperator.Type.GEOLOCATION_TYPE,\n      SearchOperator.Type.NEAR,\n      SearchOperator.Type.PLACE,\n      SearchOperator.Type.WITHIN));\n\n  public ClientIdQueryOperatorStatsFilter() {\n    this.clock = Clock.SYSTEM_CLOCK;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n    EarlybirdRequest req = requestContext.getRequest();\n    Query parsedQuery = requestContext.getParsedQuery();\n\n    if (parsedQuery == null) {\n      return service.apply(requestContext);\n    }\n\n    Set<SearchOperator.Type> operators = getOperators(parsedQuery);\n    Future<EarlybirdResponse> response = service.apply(requestContext);\n    for (SearchOperator.Type operator : operators) {\n\n      RequestCounters clientOperatorCounters = getClientOperatorCounters(req.clientId, operator);\n      RequestCountersEventListener<EarlybirdResponse> clientOperatorCountersEventListener =\n          new RequestCountersEventListener<>(\n              clientOperatorCounters, clock, EarlybirdSuccessfulResponseHandler.INSTANCE);\n\n      response = response.addEventListener(clientOperatorCountersEventListener);\n    }\n    return response;\n  }\n\n  /**\n   * Gets or creates RequestCounters for the given clientId and operatorType\n   */\n  private RequestCounters getClientOperatorCounters(String clientId,\n                                                    SearchOperator.Type operatorType) {\n    String counterPrefix = String.format(COUNTER_PREFIX_PATTERN, clientId, operatorType.toString());\n    RequestCounters clientCounters = requestCountersByClientIdAndOperator.get(counterPrefix);\n    if (clientCounters == null) {\n      clientCounters = new RequestCounters(counterPrefix);\n      RequestCounters existingCounters =\n          requestCountersByClientIdAndOperator.putIfAbsent(counterPrefix, clientCounters);\n      if (existingCounters != null) {\n        clientCounters = existingCounters;\n      }\n    }\n    return clientCounters;\n  }\n\n  /**\n   * Returns a set of the SearchOperator types that are:\n   * 1) used by the query\n   * 2) included in the allow list: operatorsToRecordStatsFor\n   */\n  private Set<SearchOperator.Type> getOperators(Query parsedQuery) {\n    final DetectVisitor detectVisitor = new DetectVisitor(false, SearchOperator.Type.values());\n    Set<SearchOperator.Type> detectedOperatorTypes = EnumSet.noneOf(SearchOperator.Type.class);\n\n    try {\n      parsedQuery.accept(detectVisitor);\n    } catch (QueryParserException e) {\n      LOG.error(\"Failed to detect SearchOperators in query: \" + parsedQuery.toString());\n      return detectedOperatorTypes;\n    }\n\n    for (Query query : detectVisitor.getDetectedQueries()) {\n      // This detectVisitor only matches on SearchOperators.\n      SearchOperator operator = (SearchOperator) query;\n      SearchOperator.Type operatorType = operator.getOperatorType();\n      if (operatorsToRecordStatsFor.contains(operatorType)) {\n        detectedOperatorTypes.add(operatorType);\n      }\n    }\n    return detectedOperatorTypes;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/ClientIdQuotaFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.Optional;\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.ConcurrentMap;\n\nimport javax.inject.Inject;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.cache.CacheBuilder;\nimport com.google.common.cache.CacheLoader;\nimport com.google.common.cache.LoadingCache;\nimport com.google.common.util.concurrent.RateLimiterProxy;\nimport com.google.common.util.concurrent.TwitterRateLimiterProxyFactory;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.SearchCustomGauge;\nimport com.twitter.search.common.metrics.SearchLongGauge;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.util.FinagleUtil;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird_root.quota.ClientIdQuotaManager;\nimport com.twitter.search.earlybird_root.quota.QuotaInfo;\nimport com.twitter.util.Future;\n\n/**\n * A filter that tracks and limits the per-client request rate. The ID of the client is determined\n * by looking at the Finagle client ID and the EarlybirdRequest.clientId field.\n *\n * The configuration currently has one config based implementation: see ConfigRepoBasedQuotaManager.\n *\n * If a client has a quota set, this filter will rate limit the requests from that client based on\n * that quota. Otherwise, the client is assumed to use a \"common request pool\", which has its own\n * quota. A quota for the common pool must always exist (even if it's set to 0).\n *\n * All rate limiters used in this class are tolerant to bursts. See TwitterRateLimiterFactory for\n * more details.\n *\n * If a client sends us more requests than its allowed quota, we keep track of the excess traffic\n * and export that number in a counter. However, we rate limit the requests from that client only if\n * the QuotaInfo returned from ClientIdQuotaManager has the shouldEnforceQuota property set to true.\n *\n * If a request is rate limited, the filter will return an EarlybirdResponse with a\n * QUOTA_EXCEEDED_ERROR response code.\n */\npublic class ClientIdQuotaFilter extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n  private static final class ClientQuota {\n    private final QuotaInfo quotaInfo;\n    private final boolean shouldAllowRequest;\n    private final ClientIdRequestCounters requestCounters;\n\n    private ClientQuota(\n        QuotaInfo quotaInfo,\n        boolean shouldAllowRequest,\n        ClientIdRequestCounters requestCounters) {\n\n      this.quotaInfo = quotaInfo;\n      this.shouldAllowRequest = shouldAllowRequest;\n      this.requestCounters = requestCounters;\n    }\n  }\n\n  private static final class ClientIdRequestCounters {\n    private static final String REQUESTS_RECEIVED_COUNTER_NAME_PATTERN =\n        \"quota_requests_received_for_client_id_%s\";\n\n    private static final String THROTTLED_REQUESTS_COUNTER_NAME_PATTERN =\n        \"quota_requests_throttled_for_client_id_%s\";\n\n    private static final String REQUESTS_ABOVE_QUOTA_COUNTER_NAME_PATTERN =\n        \"quota_requests_above_quota_for_client_id_%s\";\n\n    private static final String REQUESTS_WITHIN_QUOTA_COUNTER_NAME_PATTERN =\n        \"quota_requests_within_quota_for_client_id_%s\";\n\n    private static final String PER_CLIENT_QUOTA_GAUGE_NAME_PATTERN =\n        \"quota_for_client_id_%s\";\n\n    private final SearchRateCounter throttledRequestsCounter;\n    private final SearchRateCounter requestsReceivedCounter;\n    private final SearchRateCounter requestsAboveQuotaCounter;\n    private final SearchRateCounter requestsWithinQuotaCounter;\n    private final SearchLongGauge quotaClientGauge;\n\n    private ClientIdRequestCounters(String clientId) {\n      this.throttledRequestsCounter = SearchRateCounter.export(\n          String.format(THROTTLED_REQUESTS_COUNTER_NAME_PATTERN, clientId));\n\n      this.requestsReceivedCounter = SearchRateCounter.export(\n          String.format(REQUESTS_RECEIVED_COUNTER_NAME_PATTERN, clientId), true);\n\n      this.quotaClientGauge = SearchLongGauge.export(\n          String.format(PER_CLIENT_QUOTA_GAUGE_NAME_PATTERN, clientId));\n\n      this.requestsAboveQuotaCounter = SearchRateCounter.export(\n            String.format(REQUESTS_ABOVE_QUOTA_COUNTER_NAME_PATTERN, clientId));\n\n      this.requestsWithinQuotaCounter = SearchRateCounter.export(\n            String.format(REQUESTS_WITHIN_QUOTA_COUNTER_NAME_PATTERN, clientId));\n    }\n  }\n\n  private static final String REQUESTS_RECEIVED_FOR_EMAIL_COUNTER_NAME_PATTERN =\n      \"quota_requests_received_for_email_%s\";\n\n  // We have this aggregate stat only because doing sumany(...) on the\n  // per-client statistic is too expensive for an alert.\n  @VisibleForTesting\n  static final SearchRateCounter TOTAL_REQUESTS_RECEIVED_COUNTER =\n      SearchRateCounter.export(\"total_quota_requests_received\", true);\n\n  private static final int DEFAULT_BURST_FACTOR_SECONDS = 60;\n  private static final String QUOTA_STAT_CACHE_SIZE = \"quota_stat_cache_size\";\n  private static final String MISSING_QUOTA_FOR_CLIENT_ID_COUNTER_NAME_PATTERN =\n      \"quota_requests_with_missing_quota_for_client_id_%s\";\n\n  private static final Logger LOG = LoggerFactory.getLogger(ClientIdQuotaFilter.class);\n\n  private final ConcurrentMap<String, RateLimiterProxy> rateLimiterProxiesByClientId =\n      new ConcurrentHashMap<>();\n\n  private final ClientIdQuotaManager quotaManager;\n  private final TwitterRateLimiterProxyFactory rateLimiterProxyFactory;\n  private final LoadingCache<String, ClientIdRequestCounters> clientRequestCounters;\n  private final LoadingCache<String, SearchRateCounter> emailRequestCounters;\n\n  /** Creates a new ClientIdQuotaFilter instance. */\n  @Inject\n  public ClientIdQuotaFilter(ClientIdQuotaManager quotaManager,\n                             TwitterRateLimiterProxyFactory rateLimiterProxyFactory) {\n    this.quotaManager = quotaManager;\n    this.rateLimiterProxyFactory = rateLimiterProxyFactory;\n\n    this.clientRequestCounters = CacheBuilder.newBuilder()\n        .build(new CacheLoader<String, ClientIdRequestCounters>() {\n          @Override\n          public ClientIdRequestCounters load(String clientId) {\n            return new ClientIdRequestCounters(clientId);\n          }\n        });\n    this.emailRequestCounters = CacheBuilder.newBuilder()\n        .build(new CacheLoader<String, SearchRateCounter>() {\n          @Override\n          public SearchRateCounter load(String email) {\n            return SearchRateCounter.export(\n                String.format(REQUESTS_RECEIVED_FOR_EMAIL_COUNTER_NAME_PATTERN, email));\n          }\n        });\n\n    SearchCustomGauge.export(QUOTA_STAT_CACHE_SIZE, () -> clientRequestCounters.size());\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(EarlybirdRequest request,\n                                         Service<EarlybirdRequest, EarlybirdResponse> service) {\n    String finagleClientId = FinagleUtil.getFinagleClientName();\n    String requestClientId = ClientIdUtil.getClientIdFromRequest(request);\n    LOG.debug(String.format(\"Client id from request or attribution: %s\", requestClientId));\n\n    // Multiple client ids may be grouped into a single quota client id, all the\n    // unknown or unset client ids for example.\n    String quotaClientId = ClientIdUtil.getQuotaClientId(requestClientId);\n    LOG.debug(String.format(\"Client id used for checking quota: %s\", quotaClientId));\n\n    ClientQuota clientQuota = getClientQuota(quotaClientId);\n    if (!clientQuota.shouldAllowRequest && clientQuota.quotaInfo.shouldEnforceQuota()) {\n      clientQuota.requestCounters.throttledRequestsCounter.increment();\n\n      return Future.value(getQuotaExceededResponse(\n          finagleClientId,\n          clientQuota.quotaInfo.getQuotaClientId(),\n          clientQuota.quotaInfo.getQuota()));\n    }\n\n    return service.apply(request);\n  }\n\n  private ClientQuota getClientQuota(String clientId) {\n    Optional<QuotaInfo> quotaInfoOptional = quotaManager.getQuotaForClient(clientId);\n    if (!quotaInfoOptional.isPresent()) {\n      SearchRateCounter noQuotaFoundForClientCounter = SearchRateCounter.export(\n          String.format(MISSING_QUOTA_FOR_CLIENT_ID_COUNTER_NAME_PATTERN, clientId));\n      noQuotaFoundForClientCounter.increment();\n    }\n\n    // If a quota was set for this client, use it. Otherwise, use the common pool's quota.\n    // A quota for the common pool must always exist.\n    QuotaInfo quotaInfo = quotaInfoOptional.orElseGet(quotaManager::getCommonPoolQuota);\n\n    ClientIdRequestCounters requestCounters = clientRequestCounters\n        .getUnchecked(quotaInfo.getQuotaClientId());\n    emailRequestCounters.getUnchecked(quotaInfo.getQuotaEmail()).increment();\n\n    // Increment a stat for each request the filter receives.\n    requestCounters.requestsReceivedCounter.increment();\n\n    // Also increment the total stat\n    TOTAL_REQUESTS_RECEIVED_COUNTER.increment();\n\n    // If shouldEnforceQuota is false, we already know that the request will be allowed.\n    // However, we still want to update the rate limiter and the stats.\n    final boolean requestAllowed;\n    if (quotaInfo.getQuota() == 0) {\n      // If the quota for this client is set to 0, then the request should not be allowed.\n      //\n      // Do not update the rate limiter's rate: RateLimiter only accepts positive rates, and in any\n      // case, we already know that the request should not be allowed.\n      requestAllowed = false;\n    } else {\n      // The quota is not 0: update the rate limiter with the new quota, and see if the request\n      // should be allowed.\n      RateLimiterProxy rateLimiterProxy = getClientRateLimiterProxy(quotaInfo.getQuotaClientId(),\n          quotaInfo.getQuota());\n      requestAllowed = rateLimiterProxy.tryAcquire();\n    }\n\n    // Report the current quota for each client\n    requestCounters.quotaClientGauge.set(quotaInfo.getQuota());\n\n    // Update the corresponding counter, if the request should not be allowed.\n    if (!requestAllowed) {\n      requestCounters.requestsAboveQuotaCounter.increment();\n    } else {\n      requestCounters.requestsWithinQuotaCounter.increment();\n    }\n\n    // Throttle the request only if the quota for this service should be enforced.\n    return new ClientQuota(quotaInfo, requestAllowed, requestCounters);\n  }\n\n  private RateLimiterProxy getClientRateLimiterProxy(String clientId, int rate) {\n    // If a RateLimiter for this client doesn't exist, create one,\n    // unless another thread beat us to it.\n    RateLimiterProxy clientRateLimiterProxy = rateLimiterProxiesByClientId.get(clientId);\n    if (clientRateLimiterProxy == null) {\n      clientRateLimiterProxy =\n          rateLimiterProxyFactory.createRateLimiterProxy(rate, DEFAULT_BURST_FACTOR_SECONDS);\n      RateLimiterProxy existingClientRateLimiterProxy =\n        rateLimiterProxiesByClientId.putIfAbsent(clientId, clientRateLimiterProxy);\n      if (existingClientRateLimiterProxy != null) {\n        clientRateLimiterProxy = existingClientRateLimiterProxy;\n      }\n      LOG.info(\"Using rate limiter with rate {} for clientId {}.\",\n               clientRateLimiterProxy.getRate(), clientId);\n    }\n\n    // Update the quota, if needed.\n    if (clientRateLimiterProxy.getRate() != rate) {\n      LOG.info(\"Updating the rate from {} to {} for clientId {}.\",\n               clientRateLimiterProxy.getRate(), rate, clientId);\n      clientRateLimiterProxy.setRate(rate);\n    }\n\n    return clientRateLimiterProxy;\n  }\n\n  private static EarlybirdResponse getQuotaExceededResponse(\n      String finagleClientId, String quotaClientId, int quota) {\n    return new EarlybirdResponse(EarlybirdResponseCode.QUOTA_EXCEEDED_ERROR, 0)\n      .setSearchResults(new ThriftSearchResults())\n      .setDebugString(String.format(\n          \"Client %s (finagle client ID %s) has exceeded its request quota of %d. \"\n          + \"Please request more quota at go/searchquota.\",\n          quotaClientId, finagleClientId, quota));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/ClientIdTrackingFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.ConcurrentMap;\n\nimport javax.inject.Inject;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport com.twitter.common.collections.Pair;\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.clientstats.RequestCounters;\nimport com.twitter.search.common.clientstats.RequestCountersEventListener;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.util.FinagleUtil;\nimport com.twitter.search.common.util.earlybird.ThriftSearchQueryUtil;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.util.Future;\n\n/** Tracks the number of queries we get from each client. */\npublic class ClientIdTrackingFilter extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n  // Be careful when changing the names of these stats or adding new ones: make sure that they have\n  // prefixes/suffixes that will allow us to group them in Viz, without pulling in other stats.\n  // For example, we'll probably have a Viz graph for client_id_tracker_qps_for_client_id_*_all.\n  // So if you add a new stat named client_id_tracker_qps_for_client_id_%s_and_new_field_%s_all,\n  // then the graph will be grouping up the values from both stats, instead of grouping up the\n  // values only for client_id_tracker_qps_for_client_id_%s_all.\n  @VisibleForTesting\n  static final String QPS_ALL_STAT_PATTERN = \"client_id_tracker_qps_for_%s_all\";\n\n  @VisibleForTesting\n  static final String QPS_LOGGED_IN_STAT_PATTERN = \"client_id_tracker_qps_for_%s_logged_in\";\n\n  @VisibleForTesting\n  static final String QPS_LOGGED_OUT_STAT_PATTERN = \"client_id_tracker_qps_for_%s_logged_out\";\n\n  static final String SUPERROOT_REJECT_REQUESTS_WITH_UNKNOWN_FINAGLE_ID =\n      \"superroot_reject_requests_with_unknown_finagle_id\";\n\n  static final String UNKNOWN_FINAGLE_ID_DEBUG_STRING = \"Please specify a finagle client id.\";\n\n  private final ConcurrentMap<String, RequestCounters> requestCountersByClientId =\n    new ConcurrentHashMap<>();\n  private final ConcurrentMap<Pair<String, String>, RequestCounters>\n      requestCountersByFinagleIdAndClientId = new ConcurrentHashMap<>();\n  private final Clock clock;\n  private final SearchDecider decider;\n\n  @Inject\n  public ClientIdTrackingFilter(SearchDecider decider) {\n    this(decider, Clock.SYSTEM_CLOCK);\n  }\n\n  @VisibleForTesting\n  ClientIdTrackingFilter(SearchDecider decider, Clock clock) {\n    this.decider = decider;\n    this.clock = clock;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(EarlybirdRequest request,\n                                         Service<EarlybirdRequest, EarlybirdResponse> service) {\n    String clientId = ClientIdUtil.getClientIdFromRequest(request);\n    String finagleId = FinagleUtil.getFinagleClientName();\n    boolean isLoggedIn = ThriftSearchQueryUtil.requestInitiatedByLoggedInUser(request);\n    incrementCounters(clientId, finagleId, isLoggedIn);\n\n    if (decider.isAvailable(SUPERROOT_REJECT_REQUESTS_WITH_UNKNOWN_FINAGLE_ID)\n        && finagleId.equals(FinagleUtil.UNKNOWN_CLIENT_NAME)) {\n      EarlybirdResponse response = new EarlybirdResponse(\n          EarlybirdResponseCode.QUOTA_EXCEEDED_ERROR, 0)\n          .setDebugString(UNKNOWN_FINAGLE_ID_DEBUG_STRING);\n      return Future.value(response);\n    }\n\n    RequestCounters clientCounters = getClientCounters(clientId);\n    RequestCountersEventListener<EarlybirdResponse> clientCountersEventListener =\n        new RequestCountersEventListener<>(\n            clientCounters, clock, EarlybirdSuccessfulResponseHandler.INSTANCE);\n    RequestCounters finagleIdAndClientCounters = getFinagleIdClientCounters(clientId, finagleId);\n    RequestCountersEventListener<EarlybirdResponse> finagleIdAndClientCountersEventListener =\n        new RequestCountersEventListener<>(\n            finagleIdAndClientCounters, clock, EarlybirdSuccessfulResponseHandler.INSTANCE);\n\n    return service.apply(request)\n        .addEventListener(clientCountersEventListener)\n        .addEventListener(finagleIdAndClientCountersEventListener);\n  }\n\n  // Returns the RequestCounters instance tracking the requests from the given client ID.\n  private RequestCounters getClientCounters(String clientId) {\n    RequestCounters clientCounters = requestCountersByClientId.get(clientId);\n    if (clientCounters == null) {\n      clientCounters = new RequestCounters(ClientIdUtil.formatClientId(clientId));\n      RequestCounters existingCounters =\n        requestCountersByClientId.putIfAbsent(clientId, clientCounters);\n      if (existingCounters != null) {\n        clientCounters = existingCounters;\n      }\n    }\n    return clientCounters;\n  }\n\n  // Returns the RequestCounters instance tracking the requests from the given client ID.\n  private RequestCounters getFinagleIdClientCounters(String clientId, String finagleId) {\n    Pair<String, String> clientKey = Pair.of(clientId, finagleId);\n    RequestCounters counters = requestCountersByFinagleIdAndClientId.get(clientKey);\n    if (counters == null) {\n      counters = new RequestCounters(ClientIdUtil.formatFinagleClientIdAndClientId(\n          finagleId, clientId));\n      RequestCounters existingCounters = requestCountersByFinagleIdAndClientId.putIfAbsent(\n          clientKey, counters);\n      if (existingCounters != null) {\n        counters = existingCounters;\n      }\n    }\n    return counters;\n  }\n\n  // Increments the correct counters, based on the given clientId, finagleId, and whether or not the\n  // request came from a logged in user.\n  private static void incrementCounters(String clientId, String finagleId, boolean isLoggedIn) {\n    String clientIdForStats = ClientIdUtil.formatClientId(clientId);\n    String finagleClientIdAndClientIdForStats =\n      ClientIdUtil.formatFinagleClientIdAndClientId(finagleId, clientId);\n    SearchCounter.export(String.format(QPS_ALL_STAT_PATTERN, clientIdForStats)).increment();\n    SearchCounter.export(String.format(QPS_ALL_STAT_PATTERN, finagleClientIdAndClientIdForStats))\n      .increment();\n    if (isLoggedIn) {\n      SearchCounter.export(String.format(QPS_LOGGED_IN_STAT_PATTERN, clientIdForStats)).increment();\n      SearchCounter.export(\n          String.format(QPS_LOGGED_IN_STAT_PATTERN, finagleClientIdAndClientIdForStats))\n        .increment();\n    } else {\n      SearchCounter.export(String.format(QPS_LOGGED_OUT_STAT_PATTERN, clientIdForStats))\n        .increment();\n      SearchCounter.export(\n          String.format(QPS_LOGGED_OUT_STAT_PATTERN, finagleClientIdAndClientIdForStats))\n        .increment();\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/ClientRequestTimeFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport javax.inject.Inject;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.util.Future;\n\n/** A filter that sets the EarlybirdRequest.clientRequestTimeMs field if it's not already set. */\npublic class ClientRequestTimeFilter extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n  private static final SearchCounter CLIENT_REQUEST_TIME_MS_UNSET_COUNTER =\n      SearchCounter.export(\"client_request_time_ms_unset\");\n\n  private final Clock clock;\n\n  @Inject\n  public ClientRequestTimeFilter(Clock clock) {\n    this.clock = clock;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(EarlybirdRequest request,\n                                         Service<EarlybirdRequest, EarlybirdResponse> service) {\n    if (!request.isSetClientRequestTimeMs()) {\n      CLIENT_REQUEST_TIME_MS_UNSET_COUNTER.increment();\n      request.setClientRequestTimeMs(clock.nowMillis());\n    }\n    return service.apply(request);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/DeadlineTimeoutStatsFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.concurrent.TimeUnit;\nimport javax.inject.Inject;\n\nimport scala.Option;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.cache.CacheBuilder;\nimport com.google.common.cache.CacheLoader;\nimport com.google.common.cache.LoadingCache;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.finagle.context.Contexts$;\nimport com.twitter.finagle.context.Deadline;\nimport com.twitter.finagle.context.Deadline$;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.util.Future;\n\n/**\n * A filter for comparing the request deadline (set in the finagle request context) with the request\n * timeout, as set in the EarlybirdRequest.\n *\n * Tracks stats per client, for (1) requests where the request deadline is set to expire before the\n * EarlybirdRequest timeout, and also (2) requests where the deadline allows enough time for the\n * EarlybirdRequest timeout to kick in.\n */\npublic class DeadlineTimeoutStatsFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n\n  // All stats maps below are per client id, keyed by the client id.\n  private final LoadingCache<String, SearchCounter> requestTimeoutNotSetStats;\n  private final LoadingCache<String, SearchCounter> finagleDeadlineNotSetStats;\n  private final LoadingCache<String, SearchCounter> finagleDeadlineAndRequestTimeoutNotSetStats;\n  private final LoadingCache<String, SearchTimerStats> requestTimeoutStats;\n  private final LoadingCache<String, SearchTimerStats> finagleDeadlineStats;\n  private final LoadingCache<String, SearchTimerStats> deadlineLargerStats;\n  private final LoadingCache<String, SearchTimerStats> deadlineSmallerStats;\n\n  @Inject\n  public DeadlineTimeoutStatsFilter(Clock clock) {\n    this.requestTimeoutNotSetStats = CacheBuilder.newBuilder().build(\n        new CacheLoader<String, SearchCounter>() {\n          public SearchCounter load(String clientId) {\n            return SearchCounter.export(\n                \"deadline_for_client_id_\" + clientId + \"_request_timeout_not_set\");\n          }\n        });\n    this.finagleDeadlineNotSetStats = CacheBuilder.newBuilder().build(\n        new CacheLoader<String, SearchCounter>() {\n          public SearchCounter load(String clientId) {\n            return SearchCounter.export(\n                \"deadline_for_client_id_\" + clientId + \"_finagle_deadline_not_set\");\n          }\n        });\n    this.finagleDeadlineAndRequestTimeoutNotSetStats = CacheBuilder.newBuilder().build(\n        new CacheLoader<String, SearchCounter>() {\n          public SearchCounter load(String clientId) {\n            return SearchCounter.export(\n                \"deadline_for_client_id_\" + clientId\n                    + \"_finagle_deadline_and_request_timeout_not_set\");\n          }\n        });\n    this.requestTimeoutStats = CacheBuilder.newBuilder().build(\n        new CacheLoader<String, SearchTimerStats>() {\n          public SearchTimerStats load(String clientId) {\n            return SearchTimerStats.export(\n                \"deadline_for_client_id_\" + clientId + \"_request_timeout\",\n                TimeUnit.MILLISECONDS,\n                false,\n                true,\n                clock);\n          }\n        });\n    this.finagleDeadlineStats = CacheBuilder.newBuilder().build(\n        new CacheLoader<String, SearchTimerStats>() {\n          public SearchTimerStats load(String clientId) {\n            return SearchTimerStats.export(\n                \"deadline_for_client_id_\" + clientId + \"_finagle_deadline\",\n                TimeUnit.MILLISECONDS,\n                false,\n                true,\n                clock);\n          }\n        });\n    this.deadlineLargerStats = CacheBuilder.newBuilder().build(\n        new CacheLoader<String, SearchTimerStats>() {\n          public SearchTimerStats load(String clientId) {\n            return SearchTimerStats.export(\n                \"deadline_for_client_id_\" + clientId\n                    + \"_finagle_deadline_larger_than_request_timeout\",\n                TimeUnit.MILLISECONDS,\n                false,\n                true,\n                clock\n            );\n          }\n        });\n    this.deadlineSmallerStats = CacheBuilder.newBuilder().build(\n        new CacheLoader<String, SearchTimerStats>() {\n          public SearchTimerStats load(String clientId) {\n            return SearchTimerStats.export(\n                \"deadline_for_client_id_\" + clientId\n                    + \"_finagle_deadline_smaller_than_request_timeout\",\n                TimeUnit.MILLISECONDS,\n                false,\n                true,\n                clock\n            );\n          }\n        });\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n\n    EarlybirdRequest request = requestContext.getRequest();\n    String clientId = ClientIdUtil.getClientIdFromRequest(request);\n    long requestTimeoutMillis = getRequestTimeout(request);\n    Option<Deadline> deadline = Contexts$.MODULE$.broadcast().get(Deadline$.MODULE$);\n\n    // Tracking per-client timeouts specified in the EarlybirdRequest.\n    if (requestTimeoutMillis > 0) {\n      requestTimeoutStats.getUnchecked(clientId).timerIncrement(requestTimeoutMillis);\n    } else {\n      requestTimeoutNotSetStats.getUnchecked(clientId).increment();\n    }\n\n    // How much time does this request have, from its deadline start, to the effective deadline.\n    if (deadline.isDefined()) {\n      long deadlineEndTimeMillis = deadline.get().deadline().inMillis();\n      long deadlineStartTimeMillis = deadline.get().timestamp().inMillis();\n      long finagleDeadlineTimeMillis = deadlineEndTimeMillis - deadlineStartTimeMillis;\n      finagleDeadlineStats.getUnchecked(clientId).timerIncrement(finagleDeadlineTimeMillis);\n    } else {\n      finagleDeadlineNotSetStats.getUnchecked(clientId).increment();\n    }\n\n    // Explicitly track when both are not set.\n    if (requestTimeoutMillis <= 0 && deadline.isEmpty()) {\n      finagleDeadlineAndRequestTimeoutNotSetStats.getUnchecked(clientId).increment();\n    }\n\n    // If both timeout and the deadline are set, track how much over / under we are, when\n    // comparing the deadline, and the EarlybirdRequest timeout.\n    if (requestTimeoutMillis > 0 && deadline.isDefined()) {\n      long deadlineEndTimeMillis = deadline.get().deadline().inMillis();\n      Preconditions.checkState(request.isSetClientRequestTimeMs(),\n          \"Expect ClientRequestTimeFilter to always set the clientRequestTimeMs field. Request: %s\",\n          request);\n      long requestStartTimeMillis = request.getClientRequestTimeMs();\n      long requestEndTimeMillis = requestStartTimeMillis + requestTimeoutMillis;\n\n      long deadlineDiffMillis = deadlineEndTimeMillis - requestEndTimeMillis;\n      if (deadlineDiffMillis >= 0) {\n        deadlineLargerStats.getUnchecked(clientId).timerIncrement(deadlineDiffMillis);\n      } else {\n        // Track \"deadline is smaller\" as positive values.\n        deadlineSmallerStats.getUnchecked(clientId).timerIncrement(-deadlineDiffMillis);\n      }\n    }\n\n    return service.apply(requestContext);\n  }\n\n  private long getRequestTimeout(EarlybirdRequest request) {\n    if (request.isSetSearchQuery()\n        && request.getSearchQuery().isSetCollectorParams()\n        && request.getSearchQuery().getCollectorParams().isSetTerminationParams()\n        && request.getSearchQuery().getCollectorParams().getTerminationParams().isSetTimeoutMs()) {\n\n      return request.getSearchQuery().getCollectorParams().getTerminationParams().getTimeoutMs();\n    } else if (request.isSetTimeoutMs()) {\n      return request.getTimeoutMs();\n    } else {\n      return -1;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/DisableClientByTierFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.Optional;\n\nimport javax.inject.Inject;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Lists;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird_root.quota.ClientIdQuotaManager;\nimport com.twitter.search.earlybird_root.quota.QuotaInfo;\nimport com.twitter.util.Future;\n\npublic class DisableClientByTierFilter extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n  private static final String CLIENT_BLOCKED_RESPONSE_PATTERN =\n      \"Requests of client %s are blocked due to %s disable\";\n\n  private final SearchDecider decider;\n  private final ClientIdQuotaManager quotaManager;\n\n  /**\n   * Construct the filter by using ClientIdQuotaManager\n   */\n  @Inject\n  public DisableClientByTierFilter(ClientIdQuotaManager quotaManager, SearchDecider decider) {\n    this.quotaManager = Preconditions.checkNotNull(quotaManager);\n    this.decider = decider;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(EarlybirdRequest request,\n                                         Service<EarlybirdRequest, EarlybirdResponse> service) {\n    String clientId = ClientIdUtil.getClientIdFromRequest(request);\n    Optional<QuotaInfo> quotaInfoOptional = quotaManager.getQuotaForClient(clientId);\n    QuotaInfo quotaInfo = quotaInfoOptional.orElseGet(quotaManager::getCommonPoolQuota);\n    // Tier value should exist: if client's tier value not in config file, it will be\n    // set to \"no_tier\" by default in ConfigBasedQuotaConfig\n    String tier = quotaInfo.getClientTier();\n\n    Preconditions.checkNotNull(tier);\n\n    if (decider.isAvailable(\"superroot_unavailable_for_\" + tier + \"_clients\")) {\n      return Future.value(getClientBlockedResponse(clientId, tier));\n    } else {\n      return service.apply(request);\n    }\n  }\n\n  private static EarlybirdResponse getClientBlockedResponse(String clientId, String tier) {\n    return new EarlybirdResponse(EarlybirdResponseCode.CLIENT_BLOCKED_BY_TIER_ERROR, 0)\n        .setSearchResults(new ThriftSearchResults()\n            .setResults(Lists.<ThriftSearchResult>newArrayList()))\n        .setDebugString(String.format(CLIENT_BLOCKED_RESPONSE_PATTERN, clientId, tier));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/DropAllProtectedOperatorFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport javax.inject.Inject;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.visitors.DropAllProtectedOperatorVisitor;\nimport com.twitter.util.Future;\n\npublic class DropAllProtectedOperatorFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(DropAllProtectedOperatorFilter.class);\n  private static final SearchCounter QUERY_PARSER_FAILURE_COUNTER =\n      SearchCounter.export(\"protected_operator_filter_query_parser_failure_count\");\n  @VisibleForTesting\n  static final SearchCounter TOTAL_REQUESTS_COUNTER =\n      SearchCounter.export(\"drop_all_protected_operator_filter_total\");\n  @VisibleForTesting\n  static final SearchCounter OPERATOR_DROPPED_REQUESTS_COUNTER =\n      SearchCounter.export(\"drop_all_protected_operator_filter_operator_dropped\");\n\n  private final DropAllProtectedOperatorVisitor dropProtectedOperatorVisitor;\n\n  @Inject\n  public DropAllProtectedOperatorFilter(\n      DropAllProtectedOperatorVisitor dropProtectedOperatorVisitor\n  ) {\n    this.dropProtectedOperatorVisitor = dropProtectedOperatorVisitor;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n    TOTAL_REQUESTS_COUNTER.increment();\n    Query query = requestContext.getParsedQuery();\n    if (query == null) {\n      return service.apply(requestContext);\n    }\n\n    Query processedQuery = query;\n    try {\n      processedQuery = query.accept(dropProtectedOperatorVisitor);\n    } catch (QueryParserException e) {\n      // this should not happen since we already have a parsed query\n      QUERY_PARSER_FAILURE_COUNTER.increment();\n      LOG.warn(\n          \"Failed to drop protected operator for serialized query: \" + query.serialize(), e);\n    }\n\n    if (processedQuery == query) {\n      return service.apply(requestContext);\n    } else {\n      OPERATOR_DROPPED_REQUESTS_COUNTER.increment();\n      EarlybirdRequestContext clonedRequestContext =\n          EarlybirdRequestContext.copyRequestContext(requestContext, processedQuery);\n      return service.apply(clonedRequestContext);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/EarlybirdClusterAvailableFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.Collections;\nimport java.util.Map;\n\nimport javax.inject.Inject;\n\nimport com.google.common.collect.Maps;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\nimport com.twitter.util.Future;\n\n/**\n * A Finagle filter that determines if a certain cluster is available to the SuperRoot.\n *\n * Normally, all clusters should be available. However, if there's a problem with our systems, and\n * our search clusters are causing issues for other services (time outs, for example), then we might\n * want to be disable them, and return errors to our clients.\n */\npublic class EarlybirdClusterAvailableFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n  private final SearchDecider decider;\n  private final EarlybirdCluster cluster;\n  private final String allRequestsDeciderKey;\n  private final Map<EarlybirdRequestType, String> requestTypeDeciderKeys;\n  private final Map<EarlybirdRequestType, SearchCounter> disabledRequests;\n\n  /**\n   * Creates a new EarlybirdClusterAvailableFilter instance.\n   *\n   * @param decider The decider to use to determine if this cluster is available.\n   * @param cluster The cluster.\n   */\n  @Inject\n  public EarlybirdClusterAvailableFilter(SearchDecider decider, EarlybirdCluster cluster) {\n    this.decider = decider;\n    this.cluster = cluster;\n\n    String clusterName = cluster.getNameForStats();\n    this.allRequestsDeciderKey = \"superroot_\" + clusterName + \"_cluster_available_for_all_requests\";\n\n    Map<EarlybirdRequestType, String> tempDeciderKeys = Maps.newEnumMap(EarlybirdRequestType.class);\n    Map<EarlybirdRequestType, SearchCounter> tempCounters =\n      Maps.newEnumMap(EarlybirdRequestType.class);\n    for (EarlybirdRequestType requestType : EarlybirdRequestType.values()) {\n      String requestTypeName = requestType.getNormalizedName();\n      tempDeciderKeys.put(requestType, \"superroot_\" + clusterName + \"_cluster_available_for_\"\n                          + requestTypeName + \"_requests\");\n      tempCounters.put(requestType, SearchCounter.export(\n                           \"cluster_available_filter_\" + clusterName + \"_\"\n                           + requestTypeName + \"_disabled_requests\"));\n    }\n    requestTypeDeciderKeys = Collections.unmodifiableMap(tempDeciderKeys);\n    disabledRequests = Collections.unmodifiableMap(tempCounters);\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n    EarlybirdRequestType requestType = requestContext.getEarlybirdRequestType();\n    if (!decider.isAvailable(allRequestsDeciderKey)\n        || !decider.isAvailable(requestTypeDeciderKeys.get(requestType))) {\n      disabledRequests.get(requestType).increment();\n      return Future.value(\n          errorResponse(\"The \" + cluster.getNameForStats() + \" cluster is not available for \"\n                        + requestType.getNormalizedName() + \" requests.\"));\n    }\n\n    return service.apply(requestContext);\n  }\n\n  private EarlybirdResponse errorResponse(String debugMessage) {\n    return new EarlybirdResponse(EarlybirdResponseCode.PERSISTENT_ERROR, 0)\n      .setDebugString(debugMessage);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/EarlybirdFeatureSchemaAnnotateFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.List;\nimport javax.inject.Inject;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.features.thrift.ThriftSearchFeatureSchemaSpecifier;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdFeatureSchemaMerger;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.util.Future;\n\npublic class EarlybirdFeatureSchemaAnnotateFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n\n  private final EarlybirdFeatureSchemaMerger schemaMerger;\n\n  @Inject\n  public EarlybirdFeatureSchemaAnnotateFilter(EarlybirdFeatureSchemaMerger merger) {\n    this.schemaMerger = merger;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n    return service.apply(annotateRequestContext(requestContext));\n  }\n\n  /**\n   * Annotate the request to indicate the available features schemas before sending to earlybird.\n   *\n   * @param requestContext the earlybird request context\n   */\n  private EarlybirdRequestContext annotateRequestContext(EarlybirdRequestContext requestContext) {\n    EarlybirdRequest request = requestContext.getRequest();\n    if (request.isSetSearchQuery()\n        && request.getSearchQuery().isSetResultMetadataOptions()\n        && request.getSearchQuery().getResultMetadataOptions().isReturnSearchResultFeatures()) {\n      // Remember the available client side cached features schema in the context and prepare to\n      // reset it something new.\n      List<ThriftSearchFeatureSchemaSpecifier> featureSchemasAvailableInClient =\n          request.getSearchQuery().getResultMetadataOptions().getFeatureSchemasAvailableInClient();\n\n      return EarlybirdRequestContext.newContext(\n          request,\n          requestContext,\n          schemaMerger.getAvailableSchemaList(),  // Set the available feature schemas based on\n                                                  // what is cached in the current root.\n          featureSchemasAvailableInClient);\n    } else {\n      return requestContext;\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/EarlybirdResponseExceptionHandler.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.HashMap;\nimport java.util.Map;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.util.FinagleUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird_root.common.ClientErrorException;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\nimport com.twitter.util.Function;\nimport com.twitter.util.Future;\n\n/** Converts exceptions into EarlybirdResponses with error codes. */\npublic class EarlybirdResponseExceptionHandler {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(EarlybirdResponseExceptionHandler.class);\n\n  private final Map<EarlybirdRequestType, SearchCounter> requestTypeToCancelledExceptions\n    = new HashMap<>();\n  private final Map<EarlybirdRequestType, SearchCounter> requestTypeToTimeoutExceptions\n    = new HashMap<>();\n  private final Map<EarlybirdRequestType, SearchCounter> requestTypeToPersistentErrors\n    = new HashMap<>();\n  private final SearchCounter cancelledExceptions;\n  private final SearchCounter timeoutExceptions;\n  private final SearchCounter persistentErrors;\n\n  /**\n   * Creates a new top level filter for handling exceptions.\n   */\n  public EarlybirdResponseExceptionHandler(String statPrefix) {\n    this.cancelledExceptions = SearchCounter.export(\n        statPrefix + \"_exception_handler_cancelled_exceptions\");\n    this.timeoutExceptions = SearchCounter.export(\n        statPrefix + \"_exception_handler_timeout_exceptions\");\n    this.persistentErrors = SearchCounter.export(\n        statPrefix + \"_exception_handler_persistent_errors\");\n\n    for (EarlybirdRequestType requestType : EarlybirdRequestType.values()) {\n      String requestTypeNormalized = requestType.getNormalizedName();\n      requestTypeToCancelledExceptions.put(requestType,\n          SearchCounter.export(\n              statPrefix + \"_exception_handler_cancelled_exceptions_\"\n              + requestTypeNormalized));\n      requestTypeToTimeoutExceptions.put(requestType,\n          SearchCounter.export(\n              statPrefix + \"_exception_handler_timeout_exceptions_\"\n              + requestTypeNormalized));\n      requestTypeToPersistentErrors.put(requestType,\n          SearchCounter.export(\n              statPrefix + \"_exception_handler_persistent_errors_\"\n              + requestTypeNormalized));\n    }\n  }\n\n  /**\n   * If {@code responseFuture} is wraps an exception, converts it to an EarlybirdResponse instance\n   * with an appropriate error code.\n   *\n   * @param request The earlybird request.\n   * @param responseFuture The response future.\n   */\n  public Future<EarlybirdResponse> handleException(final EarlybirdRequest request,\n                                                   Future<EarlybirdResponse> responseFuture) {\n    return responseFuture.handle(\n        new Function<Throwable, EarlybirdResponse>() {\n          @Override\n          public EarlybirdResponse apply(Throwable t) {\n            if (t instanceof ClientErrorException) {\n              ClientErrorException clientExc = (ClientErrorException) t;\n              return new EarlybirdResponse()\n                  .setResponseCode(EarlybirdResponseCode.CLIENT_ERROR)\n                  .setDebugString(clientExc.getMessage());\n            } else if (FinagleUtil.isCancelException(t)) {\n              requestTypeToCancelledExceptions.get(EarlybirdRequestType.of(request))\n                  .increment();\n              cancelledExceptions.increment();\n              return new EarlybirdResponse()\n                  .setResponseCode(EarlybirdResponseCode.CLIENT_CANCEL_ERROR)\n                  .setDebugString(t.getMessage());\n            } else if (FinagleUtil.isTimeoutException(t)) {\n              requestTypeToTimeoutExceptions.get(EarlybirdRequestType.of(request))\n                  .increment();\n              timeoutExceptions.increment();\n              return new EarlybirdResponse()\n                  .setResponseCode(EarlybirdResponseCode.SERVER_TIMEOUT_ERROR)\n                  .setDebugString(t.getMessage());\n            } else {\n              // Unexpected exception: log it.\n              LOG.error(\"Caught unexpected exception.\", t);\n\n              requestTypeToPersistentErrors.get(EarlybirdRequestType.of(request))\n                  .increment();\n              persistentErrors.increment();\n              return new EarlybirdResponse()\n                  .setResponseCode(EarlybirdResponseCode.PERSISTENT_ERROR)\n                  .setDebugString(t.getMessage());\n            }\n          }\n        });\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/EarlybirdSuccessfulResponseHandler.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport com.twitter.search.common.clientstats.RequestCounters;\nimport com.twitter.search.common.clientstats.RequestCountersEventListener;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird.thrift.ThriftTermStatisticsResults;\n\nimport static com.twitter.search.common.util.earlybird.EarlybirdResponseUtil\n    .responseConsideredFailed;\n\n\n/**\n * Checks EarlybirdResponse's response to update stats.\n */\npublic final class EarlybirdSuccessfulResponseHandler\n    implements RequestCountersEventListener.SuccessfulResponseHandler<EarlybirdResponse> {\n\n  public static final EarlybirdSuccessfulResponseHandler INSTANCE =\n      new EarlybirdSuccessfulResponseHandler();\n\n  private EarlybirdSuccessfulResponseHandler() { }\n\n  @Override\n  public void handleSuccessfulResponse(\n      EarlybirdResponse response,\n      RequestCounters requestCounters) {\n\n    if (response == null) {\n      requestCounters.incrementRequestFailedCounter();\n      return;\n    }\n\n    if (response.getResponseCode() == EarlybirdResponseCode.CLIENT_CANCEL_ERROR) {\n      requestCounters.incrementRequestCancelCounter();\n    } else if (response.getResponseCode() == EarlybirdResponseCode.SERVER_TIMEOUT_ERROR) {\n      requestCounters.incrementRequestTimedOutCounter();\n    } else if (responseConsideredFailed(response.getResponseCode())) {\n      requestCounters.incrementRequestFailedCounter();\n    }\n\n    ThriftSearchResults results = response.getSearchResults();\n    if (results != null) {\n      requestCounters.incrementResultCounter(results.getResultsSize());\n    }\n\n    ThriftTermStatisticsResults termStats = response.getTermStatisticsResults();\n    if (termStats != null) {\n      requestCounters.incrementResultCounter(termStats.getTermResultsSize());\n    }\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/EarlybirdTimeFilterQueryRewriter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Map;\n\nimport javax.annotation.Nullable;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.Lists;\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.config.ServingRange;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\nimport com.twitter.search.queryparser.query.Conjunction;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\n\n/**\n * Adds query filters that filter out tweets outside a tier's serving range. Two tiers might load\n * the same timeslice, so if the filtering is not done, the two tiers might return duplicates. The\n * mergers should know how to handle the duplicates, but this might decrease the number or the\n * quality of the returned results.\n */\npublic class EarlybirdTimeFilterQueryRewriter {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(EarlybirdTimeFilterQueryRewriter.class);\n\n  private static final Map<EarlybirdRequestType, SearchCounter> NO_QUERY_COUNTS;\n  static {\n    final Map<EarlybirdRequestType, SearchCounter> tempMap =\n      Maps.newEnumMap(EarlybirdRequestType.class);\n    for (EarlybirdRequestType requestType : EarlybirdRequestType.values()) {\n      tempMap.put(requestType, SearchCounter.export(\n          \"time_filter_query_rewriter_\" + requestType.getNormalizedName() + \"_no_query_count\"));\n    }\n    NO_QUERY_COUNTS = Collections.unmodifiableMap(tempMap);\n  }\n\n  @VisibleForTesting\n  static final Map<EarlybirdRequestType, String> ADD_SINCE_ID_MAX_ID_DECIDER_KEY_MAP;\n  static {\n    final String ADD_SINCE_ID_MAX_ID_DECIDER_KEY_TEMPLATE =\n      \"add_since_id_max_id_operators_to_%s_query\";\n    final Map<EarlybirdRequestType, String> tempMap = Maps.newEnumMap(EarlybirdRequestType.class);\n    for (EarlybirdRequestType requestType : EarlybirdRequestType.values()) {\n      tempMap.put(\n          requestType,\n          String.format(ADD_SINCE_ID_MAX_ID_DECIDER_KEY_TEMPLATE, requestType.getNormalizedName()));\n    }\n    ADD_SINCE_ID_MAX_ID_DECIDER_KEY_MAP = Collections.unmodifiableMap(tempMap);\n  }\n\n  @VisibleForTesting\n  static final String ADD_SINCE_ID_MAX_ID_TO_NULL_SERIALIZED_QUERIES_DECIDER_KEY =\n      \"add_since_id_max_id_operators_to_null_serialized_queries\";\n\n  private final SearchDecider decider;\n  private final ServingRangeProvider servingRangeProvider;\n\n  EarlybirdTimeFilterQueryRewriter(\n      ServingRangeProvider servingRangeProvider,\n      SearchDecider decider) {\n\n    this.servingRangeProvider = servingRangeProvider;\n    this.decider = decider;\n  }\n\n  /**\n   * Add maxId and sinceId fields to the serialized query.\n   *\n   * This must be done after calculating the IdTimeRanges to prevent interfering with calculating\n   * IdTimeRanges\n   */\n  public EarlybirdRequestContext rewriteRequest(EarlybirdRequestContext requestContext)\n      throws QueryParserException {\n    Query q = requestContext.getParsedQuery();\n    if (q == null) {\n      if (requestContext.getEarlybirdRequestType() != EarlybirdRequestType.TERM_STATS) {\n        LOG.warn(\"Received request without a parsed query: \" + requestContext.getRequest());\n        NO_QUERY_COUNTS.get(requestContext.getEarlybirdRequestType()).increment();\n      }\n\n      if (!decider.isAvailable(ADD_SINCE_ID_MAX_ID_TO_NULL_SERIALIZED_QUERIES_DECIDER_KEY)) {\n        return requestContext;\n      }\n    }\n\n    return addOperators(requestContext, q);\n  }\n\n  private EarlybirdRequestContext addOperators(\n      EarlybirdRequestContext requestContext,\n      @Nullable Query query) throws QueryParserException {\n\n    // Add the SINCE_ID and MAX_ID operators only if the decider is enabled.\n    if (!decider.isAvailable(\n        ADD_SINCE_ID_MAX_ID_DECIDER_KEY_MAP.get(requestContext.getEarlybirdRequestType()))) {\n      return requestContext;\n    }\n\n    // Note: can't recompute the search operators because the serving range changes in real time\n    // for the most recent tier.\n    ServingRange servingRange = servingRangeProvider.getServingRange(\n        requestContext, requestContext.useOverrideTierConfig());\n\n    long tierSinceId = servingRange.getServingRangeSinceId();\n    SearchOperator sinceId = new SearchOperator(SearchOperator.Type.SINCE_ID,\n                                                Long.toString(tierSinceId));\n\n    long tierMaxId = servingRange.getServingRangeMaxId();\n    SearchOperator maxId = new SearchOperator(SearchOperator.Type.MAX_ID,\n                                              Long.toString(tierMaxId));\n\n    List<Query> conjunctionChildren = (query == null)\n        ? Lists.<Query>newArrayList(sinceId, maxId)\n        : Lists.newArrayList(query, sinceId, maxId);\n\n    Query restrictedQuery = new Conjunction(conjunctionChildren).simplify();\n\n    EarlybirdRequestContext copiedRequestContext =\n        EarlybirdRequestContext.copyRequestContext(requestContext, restrictedQuery);\n\n    return copiedRequestContext;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/EarlybirdTimeRangeFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.Collections;\nimport java.util.Map;\nimport java.util.Optional;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.Maps;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.util.earlybird.EarlybirdResponseUtil;\nimport com.twitter.search.earlybird.config.ServingRange;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.util.IdTimeRanges;\nimport com.twitter.util.Future;\n\n/**\n * A Finagle filter used to filter requests to tiers.\n * Parses serialized query on Earlybird request, and extracts since / until / since_id / max_id\n * operators. This filter then tests whether the request overlaps with the given tier. If there\n * is no overlap, an empty response is returned without actually forwarding the requests to the\n * underlying service.\n */\npublic class EarlybirdTimeRangeFilter extends\n    SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n\n  private static final Logger LOG = LoggerFactory.getLogger(EarlybirdTimeRangeFilter.class);\n\n  private static final EarlybirdResponse ERROR_RESPONSE =\n      new EarlybirdResponse(EarlybirdResponseCode.PERSISTENT_ERROR, 0)\n          .setSearchResults(new ThriftSearchResults());\n\n  private final ServingRangeProvider servingRangeProvider;\n  private final Optional<EarlybirdTimeFilterQueryRewriter> queryRewriter;\n\n  private static final Map<EarlybirdRequestType, SearchCounter> FAILED_REQUESTS;\n  static {\n    final Map<EarlybirdRequestType, SearchCounter> tempMap =\n      Maps.newEnumMap(EarlybirdRequestType.class);\n    for (EarlybirdRequestType requestType : EarlybirdRequestType.values()) {\n      tempMap.put(requestType, SearchCounter.export(\n          \"time_range_filter_\" + requestType.getNormalizedName() + \"_failed_requests\"));\n    }\n    FAILED_REQUESTS = Collections.unmodifiableMap(tempMap);\n  }\n\n  public static EarlybirdTimeRangeFilter newTimeRangeFilterWithQueryRewriter(\n      ServingRangeProvider servingRangeProvider,\n      SearchDecider decider) {\n\n    return new EarlybirdTimeRangeFilter(servingRangeProvider,\n        Optional.of(new EarlybirdTimeFilterQueryRewriter(servingRangeProvider, decider)));\n  }\n\n  public static EarlybirdTimeRangeFilter newTimeRangeFilterWithoutQueryRewriter(\n      ServingRangeProvider servingRangeProvider) {\n\n    return new EarlybirdTimeRangeFilter(servingRangeProvider, Optional.empty());\n  }\n\n  /**\n   * Construct a filter that avoids forwarding requests to unrelated tiers\n   * based on requests' since / until / since_id / max_id.\n   * @param provider Holds the boundary information.\n   */\n  EarlybirdTimeRangeFilter(\n      ServingRangeProvider provider,\n      Optional<EarlybirdTimeFilterQueryRewriter> rewriter) {\n\n    this.servingRangeProvider = provider;\n    this.queryRewriter = rewriter;\n  }\n\n  public ServingRangeProvider getServingRangeProvider() {\n    return servingRangeProvider;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n\n    Query parsedQuery = requestContext.getParsedQuery();\n    if (parsedQuery != null) {\n      // Only perform filtering if serialized query is set.\n      try {\n        IdTimeRanges queryRanges = IdTimeRanges.fromQuery(parsedQuery);\n        if (queryRanges == null) {\n          // No time ranges in query.\n          return issueServiceRequest(service, requestContext);\n        }\n\n        ServingRange servingRange =\n            servingRangeProvider.getServingRange(\n                requestContext, requestContext.useOverrideTierConfig());\n\n        if (queryDoesNotOverlapWithServingRange(queryRanges, servingRange)) {\n          return Future.value(tierSkippedResponse(requestContext.getEarlybirdRequestType(),\n                                                  servingRange));\n        } else {\n          return issueServiceRequest(service, requestContext);\n        }\n      } catch (QueryParserException e) {\n        LOG.warn(\"Unable to get IdTimeRanges from query: \" + parsedQuery.serialize());\n        // The failure here is not due to a miss-formed query from the client, since we already\n        // were able to successfully get a parsed Query from the request.\n        // If we can't determine the time ranges, pass the query along to the tier, and just\n        // restrict it to the timeranges of the tier.\n        return issueServiceRequest(service, requestContext);\n      }\n    } else {\n      // There's no serialized query. Just pass through like an identity filter.\n      return issueServiceRequest(service, requestContext);\n    }\n  }\n\n  private boolean queryDoesNotOverlapWithServingRange(IdTimeRanges queryRanges,\n        ServingRange servingRange) {\n    // As long as a query overlaps with the tier serving range on either side,\n    // the request is not filtered. I.e. we want to be conservative when doing this filtering,\n    // because it is just an optimization. We ignore the inclusiveness / exclusiveness of the\n    // boundaries. If the tier boundary and the query boundry happen to be the same, we do not\n    // filter the request.\n    return queryRanges.getSinceIDExclusive().or(0L)\n          > servingRange.getServingRangeMaxId()\n      || queryRanges.getMaxIDInclusive().or(Long.MAX_VALUE)\n          < servingRange.getServingRangeSinceId()\n      || queryRanges.getSinceTimeInclusive().or(0)\n          > servingRange.getServingRangeUntilTimeSecondsFromEpoch()\n      || queryRanges.getUntilTimeExclusive().or(Integer.MAX_VALUE)\n          < servingRange.getServingRangeSinceTimeSecondsFromEpoch();\n  }\n\n  private Future<EarlybirdResponse> issueServiceRequest(\n      Service<EarlybirdRequestContext, EarlybirdResponse> service,\n      EarlybirdRequestContext requestContext) {\n\n    try {\n      EarlybirdRequestContext request = requestContext;\n      if (queryRewriter.isPresent()) {\n        request = queryRewriter.get().rewriteRequest(requestContext);\n      }\n      return service.apply(request);\n    } catch (QueryParserException e) {\n      FAILED_REQUESTS.get(requestContext.getEarlybirdRequestType()).increment();\n      String msg = \"Failed to add time filter operators\";\n      LOG.error(msg, e);\n\n      // Note that in this case it is not clear whether the error is the client's fault or our\n      // fault, so we don't necessarily return a CLIENT_ERROR here.\n      // Currently this actually returns a PERSISTENT_ERROR.\n      if (requestContext.getRequest().getDebugMode() > 0) {\n        return Future.value(\n            ERROR_RESPONSE.deepCopy().setDebugString(msg + \": \" + e.getMessage()));\n      } else {\n        return Future.value(ERROR_RESPONSE);\n      }\n    }\n  }\n\n  /**\n   * Creates a tier skipped response, based on the given request type.\n   *\n   * For recency, relevance, facets and top tweets requests, this method returns a SUCCESS response\n   * with no search results and the minSearchedStatusID and maxSearchedStatusID appropriately set.\n   * For term stats response, it returns a TIER_SKIPPED response, but we need to revisit this.\n   *\n   * @param requestType The type of the request.\n   * @param servingRange The serving range of the tier that we're skipping.\n   */\n  @VisibleForTesting\n  public static EarlybirdResponse tierSkippedResponse(\n      EarlybirdRequestType requestType,\n      ServingRange servingRange) {\n    String debugMessage =\n      \"Tier skipped because it does not intersect with query time boundaries.\";\n    if (requestType == EarlybirdRequestType.TERM_STATS) {\n      // If it's a term stats request, return a TIER_SKIPPED response for now.\n      // But we need to figure out the right thing to do here.\n      return new EarlybirdResponse(EarlybirdResponseCode.TIER_SKIPPED, 0)\n        .setDebugString(debugMessage);\n    } else {\n      // minIds in ServingRange instances are set to tierLowerBoundary - 1, because the\n      // since_id operator is exclusive. The max_id operator on the other hand is inclusive,\n      // so maxIds in ServingRange instances are also set to tierUpperBoundary - 1.\n      // Here we want both of them to be inclusive, so we need to increment the minId by 1.\n      return EarlybirdResponseUtil.tierSkippedRootResponse(\n          servingRange.getServingRangeSinceId() + 1,\n          servingRange.getServingRangeMaxId(),\n          debugMessage);\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/FullArchiveProtectedOperatorFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.List;\n\nimport javax.inject.Inject;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.thrift.EarlybirdDebugInfo;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryNodeUtils;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\nimport com.twitter.search.queryparser.query.search.SearchOperatorConstants;\nimport com.twitter.search.queryparser.visitors.DropAllProtectedOperatorVisitor;\nimport com.twitter.search.queryparser.visitors.QueryTreeIndex;\nimport com.twitter.util.Future;\n\n/**\n * Full archive service filter validates requests with a protected operator, appends the\n * '[exclude protected]' operator by default, and appends '[filter protected]' operator instead if\n * 'getProtectedTweetsOnly' request param is set. A client error response is returned if any of the\n * following rules is violated.\n *   1. There is at most one 'protected' operator in the query.\n *   2. If there is a 'protected' operator, it must be in the query root node.\n *   3. The parent node of the 'protected' operator must not be negated and must be a conjunction.\n *   4. If there is a positive 'protected' operator, 'followedUserIds' and 'searcherId' request\n *   params must be set.\n */\npublic class FullArchiveProtectedOperatorFilter extends\n    SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n  private static final Logger LOG =\n      LoggerFactory.getLogger(FullArchiveProtectedOperatorFilter.class);\n  private static final SearchOperator EXCLUDE_PROTECTED_OPERATOR =\n      new SearchOperator(SearchOperator.Type.EXCLUDE, SearchOperatorConstants.PROTECTED);\n  private static final SearchOperator FILTER_PROTECTED_OPERATOR =\n      new SearchOperator(SearchOperator.Type.FILTER, SearchOperatorConstants.PROTECTED);\n  private static final SearchCounter QUERY_PARSER_FAILURE_COUNT =\n      SearchCounter.export(\"protected_operator_filter_query_parser_failure_count\");\n\n  private final DropAllProtectedOperatorVisitor dropProtectedOperatorVisitor;\n  private final SearchDecider decider;\n\n  @Inject\n  public FullArchiveProtectedOperatorFilter(\n      DropAllProtectedOperatorVisitor dropProtectedOperatorVisitor,\n      SearchDecider decider) {\n    this.dropProtectedOperatorVisitor = dropProtectedOperatorVisitor;\n    this.decider = decider;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n    Query query = requestContext.getParsedQuery();\n    if (query == null) {\n      return service.apply(requestContext);\n    }\n\n    QueryTreeIndex queryTreeIndex = QueryTreeIndex.buildFor(query);\n    List<Query> nodeList = queryTreeIndex.getNodeList();\n    // try to find a protected operator, returns error response if more than one protected\n    // operator is detected\n    SearchOperator protectedOperator = null;\n    for (Query node : nodeList) {\n      if (node instanceof SearchOperator) {\n        SearchOperator searchOp = (SearchOperator) node;\n        if (SearchOperatorConstants.PROTECTED.equals(searchOp.getOperand())) {\n          if (protectedOperator == null) {\n            protectedOperator = searchOp;\n          } else {\n            return createErrorResponse(\"Only one 'protected' operator is expected.\");\n          }\n        }\n      }\n    }\n\n    Query processedQuery;\n    if (protectedOperator == null) {\n      // no protected operator is detected, append '[exclude protected]' by default\n      processedQuery = QueryNodeUtils.appendAsConjunction(query, EXCLUDE_PROTECTED_OPERATOR);\n    } else {\n      // protected operator must be in the query root node\n      if (queryTreeIndex.getParentOf(protectedOperator) != query) {\n        return createErrorResponse(\"'protected' operator must be in the query root node\");\n      }\n      // the query node that contains protected operator must not be negated\n      if (query.mustNotOccur()) {\n        return createErrorResponse(\"The query node that contains a 'protected' operator must not\"\n            + \" be negated.\");\n      }\n      // the query node that contains protected operator must be a conjunction\n      if (!query.isTypeOf(Query.QueryType.CONJUNCTION)) {\n        return createErrorResponse(\"The query node that contains a 'protected' operator must\"\n            + \" be a conjunction.\");\n      }\n      // check the existence of 'followedUserIds' and 'searcherId' if it is a positive operator\n      if (isPositive(protectedOperator)) {\n        if (!validateRequestParam(requestContext.getRequest())) {\n          return createErrorResponse(\"'followedUserIds' and 'searcherId' are required \"\n              + \"by positive 'protected' operator.\");\n        }\n      }\n      processedQuery = query;\n    }\n    // update processedQuery if 'getProtectedTweetsOnly' is set to true, it takes precedence over\n    // the existing protected operators\n    if (requestContext.getRequest().isGetProtectedTweetsOnly()) {\n      if (!validateRequestParam(requestContext.getRequest())) {\n        return createErrorResponse(\"'followedUserIds' and 'searcherId' are required \"\n            + \"when 'getProtectedTweetsOnly' is set to true.\");\n      }\n      try {\n        processedQuery = processedQuery.accept(dropProtectedOperatorVisitor);\n      } catch (QueryParserException e) {\n        // this should not happen since we already have a parsed query\n        QUERY_PARSER_FAILURE_COUNT.increment();\n        LOG.warn(\n            \"Failed to drop protected operator for serialized query: \" + query.serialize(), e);\n      }\n      processedQuery =\n          QueryNodeUtils.appendAsConjunction(processedQuery, FILTER_PROTECTED_OPERATOR);\n    }\n\n    if (processedQuery == query) {\n      return service.apply(requestContext);\n    } else {\n      EarlybirdRequestContext clonedRequestContext =\n          EarlybirdRequestContext.copyRequestContext(requestContext, processedQuery);\n      return service.apply(clonedRequestContext);\n    }\n  }\n\n  private boolean validateRequestParam(EarlybirdRequest request) {\n    List<Long> followedUserIds = request.followedUserIds;\n    Long searcherId = (request.searchQuery != null && request.searchQuery.isSetSearcherId())\n        ? request.searchQuery.getSearcherId() : null;\n    return followedUserIds != null && !followedUserIds.isEmpty() && searcherId != null;\n  }\n\n  private boolean isPositive(SearchOperator searchOp) {\n    boolean isNegateExclude = searchOp.mustNotOccur()\n        && searchOp.getOperatorType() == SearchOperator.Type.EXCLUDE;\n    boolean isPositive = !searchOp.mustNotOccur()\n        && (searchOp.getOperatorType() == SearchOperator.Type.INCLUDE\n        || searchOp.getOperatorType() == SearchOperator.Type.FILTER);\n    return isNegateExclude || isPositive;\n  }\n\n  private Future<EarlybirdResponse> createErrorResponse(String errorMsg) {\n    EarlybirdResponse response = new EarlybirdResponse(EarlybirdResponseCode.CLIENT_ERROR, 0);\n    response.setDebugInfo(new EarlybirdDebugInfo().setHost(\"full_archive_root\"));\n    response.setDebugString(errorMsg);\n    return Future.value(response);\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/InitializeRequestContextFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport javax.inject.Inject;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.Filter;\nimport com.twitter.finagle.Service;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.common.EarlybirdRequestUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.QueryParsingUtils;\nimport com.twitter.search.earlybird_root.common.TwitterContextProvider;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.util.Future;\n\n/**\n * Creates a new RequestContext from an EarlybirdRequest, and passes the RequestContext down to\n * the rest of the filter/service chain.\n */\npublic class InitializeRequestContextFilter extends\n    Filter<EarlybirdRequest, EarlybirdResponse, EarlybirdRequestContext, EarlybirdResponse> {\n\n  @VisibleForTesting\n  static final SearchCounter FAILED_QUERY_PARSING =\n      SearchCounter.export(\"initialize_request_context_filter_query_parsing_failure\");\n\n  private final SearchDecider decider;\n  private final TwitterContextProvider twitterContextProvider;\n  private final Clock clock;\n\n  /**\n   * The constructor of the filter.\n   */\n  @Inject\n  public InitializeRequestContextFilter(SearchDecider decider,\n                                        TwitterContextProvider twitterContextProvider,\n                                        Clock clock) {\n    this.decider = decider;\n    this.twitterContextProvider = twitterContextProvider;\n    this.clock = clock;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequest request,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n\n    EarlybirdRequestUtil.recordClientClockDiff(request);\n\n    EarlybirdRequestContext requestContext;\n    try {\n      requestContext = EarlybirdRequestContext.newContext(\n          request, decider, twitterContextProvider.get(), clock);\n    } catch (QueryParserException e) {\n      FAILED_QUERY_PARSING.increment();\n      return QueryParsingUtils.newClientErrorResponse(request, e);\n    }\n\n    return service.apply(requestContext);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/IsUserProtectedMetadataTrackingFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.EnumMap;\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultExtraMetadata;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\nimport com.twitter.util.Future;\nimport com.twitter.util.FutureEventListener;\n\n/**\n * Filter tracks the isUserProtected metadata stats returned from Earlybirds.\n */\npublic class IsUserProtectedMetadataTrackingFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n  private static final String COUNTER_PREFIX = \"is_user_protected_metadata_count_filter_\";\n  @VisibleForTesting\n  final Map<EarlybirdRequestType, SearchCounter> totalCounterByRequestTypeMap;\n  @VisibleForTesting\n  final Map<EarlybirdRequestType, SearchCounter> isProtectedCounterByRequestTypeMap;\n\n  public IsUserProtectedMetadataTrackingFilter() {\n    this.totalCounterByRequestTypeMap = new EnumMap<>(EarlybirdRequestType.class);\n    this.isProtectedCounterByRequestTypeMap = new EnumMap<>(EarlybirdRequestType.class);\n    for (EarlybirdRequestType requestType : EarlybirdRequestType.values()) {\n      this.totalCounterByRequestTypeMap.put(requestType,\n          SearchCounter.export(COUNTER_PREFIX + requestType.getNormalizedName() + \"_total\"));\n      this.isProtectedCounterByRequestTypeMap.put(requestType,\n          SearchCounter.export(COUNTER_PREFIX + requestType.getNormalizedName() + \"_is_protected\"));\n    }\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext request,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n    Future<EarlybirdResponse> response = service.apply(request);\n\n    EarlybirdRequestType requestType = request.getEarlybirdRequestType();\n    response.addEventListener(new FutureEventListener<EarlybirdResponse>() {\n      @Override\n      public void onSuccess(EarlybirdResponse response) {\n        if (!response.isSetSearchResults() || response.getSearchResults().getResults().isEmpty()) {\n          return;\n        }\n        List<ThriftSearchResult> searchResults = response.getSearchResults().getResults();\n        int totalCount = searchResults.size();\n        int isUserProtectedCount = 0;\n        for (ThriftSearchResult searchResult : searchResults) {\n          if (searchResult.isSetMetadata() && searchResult.getMetadata().isSetExtraMetadata()) {\n            ThriftSearchResultExtraMetadata extraMetadata =\n                searchResult.getMetadata().getExtraMetadata();\n            if (extraMetadata.isIsUserProtected()) {\n              isUserProtectedCount++;\n            }\n          }\n        }\n        IsUserProtectedMetadataTrackingFilter.this\n            .totalCounterByRequestTypeMap.get(requestType).add(totalCount);\n        IsUserProtectedMetadataTrackingFilter.this\n            .isProtectedCounterByRequestTypeMap.get(requestType).add(isUserProtectedCount);\n      }\n\n      @Override\n      public void onFailure(Throwable cause) { }\n    });\n\n    return response;\n  }\n\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/MarkTweetSourceFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftTweetSource;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\nimport com.twitter.util.Function;\nimport com.twitter.util.Future;\n\npublic class MarkTweetSourceFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n  private final SearchCounter searchResultsNotSet;\n\n  private final ThriftTweetSource tweetSource;\n\n  public MarkTweetSourceFilter(ThriftTweetSource tweetSource) {\n    this.tweetSource = tweetSource;\n    searchResultsNotSet = SearchCounter.export(\n        tweetSource.name().toLowerCase() + \"_mark_tweet_source_filter_search_results_not_set\");\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      final EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n    return service.apply(requestContext).map(new Function<EarlybirdResponse, EarlybirdResponse>() {\n        @Override\n        public EarlybirdResponse apply(EarlybirdResponse response) {\n          if (response.getResponseCode() == EarlybirdResponseCode.SUCCESS\n              && requestContext.getEarlybirdRequestType() != EarlybirdRequestType.TERM_STATS) {\n            if (!response.isSetSearchResults()) {\n              searchResultsNotSet.increment();\n            } else {\n              for (ThriftSearchResult searchResult : response.getSearchResults().getResults()) {\n                searchResult.setTweetSource(tweetSource);\n              }\n            }\n          }\n          return response;\n        }\n      }\n    );\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/MetadataTrackingFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.List;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.cache.CacheBuilder;\nimport com.google.common.cache.CacheLoader;\nimport com.google.common.cache.LoadingCache;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.SearchMovingAverage;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResultMetadata;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\nimport com.twitter.util.Future;\nimport com.twitter.util.FutureEventListener;\n\n/**\n * Filter that is tracking the engagement stats returned from Earlybirds.\n */\npublic class MetadataTrackingFilter extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n\n  private static final String SCORING_SIGNAL_STAT_PREFIX = \"scoring_signal_\";\n  private static final String SCORE_STAT_PATTERN = \"client_id_score_tracker_for_%s_x100\";\n\n  @VisibleForTesting\n  static final SearchMovingAverage SCORING_SIGNAL_FAV_COUNT =\n      SearchMovingAverage.export(SCORING_SIGNAL_STAT_PREFIX + \"fav_count\");\n\n  @VisibleForTesting\n  static final SearchMovingAverage SCORING_SIGNAL_REPLY_COUNT =\n      SearchMovingAverage.export(SCORING_SIGNAL_STAT_PREFIX + \"reply_count\");\n\n  @VisibleForTesting\n  static final SearchMovingAverage SCORING_SIGNAL_RETWEET_COUNT =\n      SearchMovingAverage.export(SCORING_SIGNAL_STAT_PREFIX + \"retweet_count\");\n\n  @VisibleForTesting\n  static final LoadingCache<String, SearchMovingAverage> CLIENT_SCORE_METRICS_LOADING_CACHE =\n      CacheBuilder.newBuilder().build(new CacheLoader<String, SearchMovingAverage>() {\n        public SearchMovingAverage load(String clientId) {\n          return SearchMovingAverage.export(String.format(SCORE_STAT_PATTERN, clientId));\n        }\n      });\n\n  @Override\n  public Future<EarlybirdResponse> apply(final EarlybirdRequest request,\n                                         Service<EarlybirdRequest, EarlybirdResponse> service) {\n\n    Future<EarlybirdResponse> response = service.apply(request);\n\n    response.addEventListener(new FutureEventListener<EarlybirdResponse>() {\n      @Override\n      public void onSuccess(EarlybirdResponse earlybirdResponse) {\n        EarlybirdRequestType type = EarlybirdRequestType.of(request);\n\n        if (earlybirdResponse.responseCode == EarlybirdResponseCode.SUCCESS\n            && type == EarlybirdRequestType.RELEVANCE\n            && earlybirdResponse.isSetSearchResults()\n            && earlybirdResponse.getSearchResults().isSetResults()) {\n\n          List<ThriftSearchResult> searchResults = earlybirdResponse.getSearchResults()\n              .getResults();\n\n          long totalFavoriteAmount = 0;\n          long totalReplyAmount = 0;\n          long totalRetweetAmount = 0;\n          double totalScoreX100 = 0;\n\n          for (ThriftSearchResult result : searchResults) {\n            if (!result.isSetMetadata()) {\n              continue;\n            }\n\n            ThriftSearchResultMetadata metadata = result.getMetadata();\n\n            if (metadata.isSetFavCount()) {\n              totalFavoriteAmount += metadata.getFavCount();\n            }\n\n            if (metadata.isSetReplyCount()) {\n              totalReplyAmount += metadata.getReplyCount();\n            }\n\n            if (metadata.isSetRetweetCount()) {\n              totalRetweetAmount += metadata.getRetweetCount();\n            }\n\n            if (metadata.isSetScore()) {\n              // Scale up the score by 100 so that scores are at least 1 and visible on viz graph\n              totalScoreX100 += metadata.getScore() * 100;\n            }\n          }\n\n          // We only count present engagement counts but report the full size of the search results.\n          // This means that we consider the missing counts as being 0.\n          SCORING_SIGNAL_FAV_COUNT.addSamples(totalFavoriteAmount, searchResults.size());\n          SCORING_SIGNAL_REPLY_COUNT.addSamples(totalReplyAmount, searchResults.size());\n          SCORING_SIGNAL_RETWEET_COUNT.addSamples(totalRetweetAmount, searchResults.size());\n          // Export per client id average scores.\n          String requestClientId = ClientIdUtil.getClientIdFromRequest(request);\n          String quotaClientId = ClientIdUtil.getQuotaClientId(requestClientId);\n          CLIENT_SCORE_METRICS_LOADING_CACHE.getUnchecked(quotaClientId)\n              .addSamples((long) totalScoreX100, searchResults.size());\n        }\n      }\n\n      @Override\n      public void onFailure(Throwable cause) { }\n    });\n\n    return response;\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/NamedMultiTermDisjunctionStatsFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.List;\nimport java.util.Map;\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.ConcurrentMap;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.Percentile;\nimport com.twitter.search.common.metrics.PercentileUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.util.Future;\n\npublic class NamedMultiTermDisjunctionStatsFilter extends\n    SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n\n    private static final String STAT_FORMAT = \"named_disjunction_size_client_%s_key_%s\";\n    // ClientID -> disjunction name -> operand count\n    private static final ConcurrentMap<String, ConcurrentMap<String, Percentile<Integer>>>\n        NAMED_MULTI_TERM_DISJUNCTION_IDS_COUNT = new ConcurrentHashMap<>();\n\n    @Override\n    public Future<EarlybirdResponse> apply(EarlybirdRequest request,\n        Service<EarlybirdRequest, EarlybirdResponse> service) {\n\n        if (request.getSearchQuery().isSetNamedDisjunctionMap()) {\n            for (Map.Entry<String, List<Long>> entry\n                : request.getSearchQuery().getNamedDisjunctionMap().entrySet()) {\n\n                Map<String, Percentile<Integer>> statsForClient =\n                    NAMED_MULTI_TERM_DISJUNCTION_IDS_COUNT.computeIfAbsent(\n                        request.getClientId(), clientId -> new ConcurrentHashMap<>());\n                Percentile<Integer> stats = statsForClient.computeIfAbsent(entry.getKey(),\n                    keyName -> PercentileUtil.createPercentile(\n                        String.format(STAT_FORMAT, request.getClientId(), keyName)));\n\n                stats.record(entry.getValue().size());\n            }\n        }\n\n        return service.apply(request);\n    }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/NullcastTrackingFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.HashSet;\nimport java.util.Set;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.ImmutableSet;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.util.earlybird.EarlybirdResponseUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.queryparser.query.search.SearchOperatorConstants;\nimport com.twitter.search.queryparser.visitors.DetectPositiveOperatorVisitor;\n\n/**\n * Filter that is tracking the unexpected nullcast results from Earlybirds.\n */\npublic class NullcastTrackingFilter extends SensitiveResultsTrackingFilter {\n  public NullcastTrackingFilter() {\n    super(\"unexpected nullcast tweets\", true);\n  }\n\n  private static final Logger LOG = LoggerFactory.getLogger(NullcastTrackingFilter.class);\n\n  @VisibleForTesting\n  static final SearchCounter BAD_NULLCAST_QUERY_COUNT =\n      SearchCounter.export(\"unexpected_nullcast_query_count\");\n\n  @VisibleForTesting\n  static final SearchCounter BAD_NULLCAST_RESULT_COUNT =\n      SearchCounter.export(\"unexpected_nullcast_result_count\");\n\n  @Override\n  protected Logger getLogger() {\n    return LOG;\n  }\n\n  @Override\n  protected SearchCounter getSensitiveQueryCounter() {\n    return BAD_NULLCAST_QUERY_COUNT;\n  }\n\n  @Override\n  protected SearchCounter getSensitiveResultsCounter() {\n    return BAD_NULLCAST_RESULT_COUNT;\n  }\n\n  @Override\n  protected Set<Long> getSensitiveResults(EarlybirdRequestContext requestContext,\n                                          EarlybirdResponse earlybirdResponse) throws Exception {\n    if (!requestContext.getParsedQuery().accept(\n        new DetectPositiveOperatorVisitor(SearchOperatorConstants.NULLCAST))) {\n      return EarlybirdResponseUtil.findUnexpectedNullcastStatusIds(\n          earlybirdResponse.getSearchResults(), requestContext.getRequest());\n    } else {\n      return new HashSet<>();\n    }\n  }\n\n  /**\n   * Some Earlybird requests are not searches, instead, they are scoring requests.\n   * These requests supply a list of IDs to be scored.\n   * It is OK to return nullcast tweet result if the ID is supplied in the request.\n   * This extracts the scoring request tweet IDs.\n   */\n  @Override\n  protected Set<Long> getExceptedResults(EarlybirdRequestContext requestContext) {\n    EarlybirdRequest request = requestContext.getRequest();\n    if (request == null\n        || !request.isSetSearchQuery()\n        || request.getSearchQuery().getSearchStatusIdsSize() == 0) {\n      return ImmutableSet.of();\n    }\n    return request.getSearchQuery().getSearchStatusIds();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/PostCacheRequestTypeCountFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport javax.inject.Inject;\n\npublic class PostCacheRequestTypeCountFilter extends RequestTypeCountFilter {\n  @Inject\n  public PostCacheRequestTypeCountFilter() {\n    super(\"post_cache\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/PreCacheRequestTypeCountFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport javax.inject.Inject;\n\npublic class PreCacheRequestTypeCountFilter extends RequestTypeCountFilter {\n  @Inject\n  public PreCacheRequestTypeCountFilter() {\n    super(\"pre_cache\");\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/QueryLangStatFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.Map;\nimport java.util.concurrent.ConcurrentHashMap;\nimport javax.inject.Inject;\nimport javax.inject.Singleton;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.common.text.language.LocaleUtil;\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.constants.thriftjava.ThriftLanguage;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.util.lang.ThriftLanguageUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.util.Future;\n\n/**\n * Export stats for query languages.\n */\n@Singleton\npublic class QueryLangStatFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n\n  public static class Config {\n    // We put a limit here in case an error in the client are sending us random lang codes.\n    private int maxNumberOfLangs;\n\n    public Config(int maxNumberOfLangs) {\n      this.maxNumberOfLangs = maxNumberOfLangs;\n    }\n\n    public int getMaxNumberOfLangs() {\n      return maxNumberOfLangs;\n    }\n  }\n\n  @VisibleForTesting\n  protected static final String LANG_STATS_PREFIX = \"num_queries_in_lang_\";\n\n  private final Config config;\n  private final SearchCounter allCountsForLangsOverMaxNumLang =\n      SearchCounter.export(LANG_STATS_PREFIX + \"overflow\");\n\n  private final ConcurrentHashMap<String, SearchCounter> langCounters =\n      new ConcurrentHashMap<>();\n\n  @Inject\n  public QueryLangStatFilter(Config config) {\n    this.config = config;\n  }\n\n  private SearchCounter getCounter(String lang) {\n    Preconditions.checkNotNull(lang);\n\n    SearchCounter counter = langCounters.get(lang);\n    if (counter == null) {\n      if (langCounters.size() >= config.getMaxNumberOfLangs()) {\n        return allCountsForLangsOverMaxNumLang;\n      }\n      synchronized (langCounters) { // This double-checked locking is safe,\n                                    // since we're using a ConcurrentHashMap\n        counter = langCounters.get(lang);\n        if (counter == null) {\n          counter = SearchCounter.export(LANG_STATS_PREFIX + lang);\n          langCounters.put(lang, counter);\n        }\n      }\n    }\n\n    return counter;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n\n    String lang = null;\n\n    ThriftSearchQuery searchQuery = requestContext.getRequest().getSearchQuery();\n\n    lang = searchQuery.getQueryLang();\n\n    if (lang == null) {\n      // fallback to ui lang\n      lang = searchQuery.getUiLang();\n    }\n\n    if (lang == null && searchQuery.isSetUserLangs()) {\n      // fallback to the user lang with the highest confidence\n      double maxConfidence = Double.MIN_VALUE;\n\n      for (Map.Entry<ThriftLanguage, Double> entry : searchQuery.getUserLangs().entrySet()) {\n        if (entry.getValue() > maxConfidence) {\n          lang = ThriftLanguageUtil.getLanguageCodeOf(entry.getKey());\n          maxConfidence = entry.getValue();\n        }\n      }\n    }\n\n    if (lang == null) {\n      lang = LocaleUtil.UNDETERMINED_LANGUAGE;\n    }\n\n    getCounter(lang).increment();\n\n    return service.apply(requestContext);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/QueryOperatorStatFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.EnumSet;\nimport java.util.Set;\nimport java.util.concurrent.TimeUnit;\n\nimport scala.runtime.BoxedUnit;\n\nimport com.google.common.collect.ImmutableMap;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.search.queryparser.query.annotation.Annotation;\nimport com.twitter.search.queryparser.query.search.SearchOperator;\nimport com.twitter.search.queryparser.query.search.SearchOperatorConstants;\nimport com.twitter.search.queryparser.visitors.DetectAnnotationVisitor;\nimport com.twitter.search.queryparser.visitors.DetectVisitor;\nimport com.twitter.util.Future;\n\n/**\n * For a given query, increments counters if that query has a number of search operators or\n * annotations applied to it. Used to detect unusual traffic patterns.\n */\npublic class QueryOperatorStatFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n  private static final Logger LOG = LoggerFactory.getLogger(QueryOperatorStatFilter.class);\n\n  private final SearchCounter numQueryOperatorDetectionErrors =\n      SearchCounter.export(\"query_operator_detection_errors\");\n\n  private final SearchCounter numQueryOperatorConsideredRequests =\n      SearchCounter.export(\"query_operator_requests_considered\");\n\n  private final ImmutableMap<String, SearchTimerStats> filterOperatorStats;\n\n  // Keeps track of the number of queries with a filter applied, whose type we don't care about.\n  private final SearchCounter numUnknownFilterOperatorRequests =\n      SearchCounter.export(\"query_operator_filter_unknown_requests\");\n\n  private final ImmutableMap<String, SearchTimerStats> includeOperatorStats;\n\n  // Keeps track of the number of queries with an include operator applied, whose type we don't\n  // know about.\n  private final SearchCounter numUnknownIncludeOperatorRequests =\n      SearchCounter.export(\"query_operator_include_unknown_requests\");\n\n  private final ImmutableMap<SearchOperator.Type, SearchTimerStats> operatorTypeStats;\n\n  private final SearchCounter numVariantRequests =\n      SearchCounter.export(\"query_operator_variant_requests\");\n\n  /**\n   * Construct this QueryOperatorStatFilter by getting the complete set of possible filters a query\n   * might have and associating each with a counter.\n   */\n  public QueryOperatorStatFilter() {\n\n    ImmutableMap.Builder<String, SearchTimerStats> filterBuilder = new ImmutableMap.Builder<>();\n    for (String operand : SearchOperatorConstants.VALID_FILTER_OPERANDS) {\n      filterBuilder.put(\n          operand,\n          SearchTimerStats.export(\n              \"query_operator_filter_\" + operand + \"_requests\",\n              TimeUnit.MILLISECONDS,\n              false,\n              true));\n    }\n    filterOperatorStats = filterBuilder.build();\n\n    ImmutableMap.Builder<String, SearchTimerStats> includeBuilder = new ImmutableMap.Builder<>();\n    for (String operand : SearchOperatorConstants.VALID_INCLUDE_OPERANDS) {\n      includeBuilder.put(\n          operand,\n          SearchTimerStats.export(\n              \"query_operator_include_\" + operand + \"_requests\",\n              TimeUnit.MILLISECONDS,\n              false,\n              true));\n    }\n    includeOperatorStats = includeBuilder.build();\n\n    ImmutableMap.Builder<SearchOperator.Type, SearchTimerStats> operatorBuilder =\n        new ImmutableMap.Builder<>();\n    for (SearchOperator.Type operatorType : SearchOperator.Type.values()) {\n      operatorBuilder.put(\n          operatorType,\n          SearchTimerStats.export(\n              \"query_operator_\" + operatorType.name().toLowerCase() + \"_requests\",\n              TimeUnit.MILLISECONDS,\n              false,\n              true\n          ));\n    }\n    operatorTypeStats = operatorBuilder.build();\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n    numQueryOperatorConsideredRequests.increment();\n    Query parsedQuery = requestContext.getParsedQuery();\n\n    if (parsedQuery == null) {\n      return service.apply(requestContext);\n    }\n\n    SearchTimer timer = new SearchTimer();\n    timer.start();\n\n    return service.apply(requestContext).ensure(() -> {\n      timer.stop();\n\n      try {\n        updateTimersForOperatorsAndOperands(parsedQuery, timer);\n        updateCountersIfVariantAnnotation(parsedQuery);\n      } catch (QueryParserException e) {\n        LOG.warn(\"Unable to test if query has operators defined\", e);\n        numQueryOperatorDetectionErrors.increment();\n      }\n      return BoxedUnit.UNIT;\n    });\n  }\n\n  /**\n   * Tracks request stats for operators and operands.\n   *\n   * @param parsedQuery the query to check.\n   */\n  private void updateTimersForOperatorsAndOperands(Query parsedQuery, SearchTimer timer)\n      throws QueryParserException {\n    final DetectVisitor detectVisitor = new DetectVisitor(false, SearchOperator.Type.values());\n    parsedQuery.accept(detectVisitor);\n\n    Set<SearchOperator.Type> detectedOperatorTypes = EnumSet.noneOf(SearchOperator.Type.class);\n    for (Query query : detectVisitor.getDetectedQueries()) {\n      // This detectVisitor only matches on SearchOperators.\n      SearchOperator operator = (SearchOperator) query;\n      SearchOperator.Type operatorType = operator.getOperatorType();\n      detectedOperatorTypes.add(operatorType);\n\n      if (operatorType == SearchOperator.Type.INCLUDE) {\n        updateOperandStats(\n            operator,\n            includeOperatorStats,\n            timer,\n            numUnknownIncludeOperatorRequests);\n      }\n      if (operatorType == SearchOperator.Type.FILTER) {\n        updateOperandStats(\n            operator,\n            filterOperatorStats,\n            timer,\n            numUnknownFilterOperatorRequests);\n      }\n    }\n\n    for (SearchOperator.Type type : detectedOperatorTypes) {\n      operatorTypeStats.get(type).stoppedTimerIncrement(timer);\n    }\n  }\n\n  private void updateOperandStats(\n      SearchOperator operator,\n      ImmutableMap<String, SearchTimerStats> operandRequestStats,\n      SearchTimer timer,\n      SearchCounter unknownOperandStat) {\n    String operand = operator.getOperand();\n    SearchTimerStats stats = operandRequestStats.get(operand);\n\n    if (stats != null) {\n      stats.stoppedTimerIncrement(timer);\n    } else {\n      unknownOperandStat.increment();\n    }\n  }\n\n  private void updateCountersIfVariantAnnotation(Query parsedQuery) throws QueryParserException {\n    DetectAnnotationVisitor visitor = new DetectAnnotationVisitor(Annotation.Type.VARIANT);\n    if (parsedQuery.accept(visitor)) {\n      numVariantRequests.increment();\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/QueryTokenizerFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.concurrent.TimeUnit;\nimport javax.inject.Inject;\n\nimport com.twitter.common_internal.text.version.PenguinVersion;\nimport com.twitter.common_internal.text.version.PenguinVersionConfig;\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.finagle.tracing.Trace;\nimport com.twitter.finagle.tracing.Tracing;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.metrics.SearchTimer;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.QueryParsingUtils;\nimport com.twitter.search.queryparser.parser.SerializedQueryParser;\nimport com.twitter.search.queryparser.parser.SerializedQueryParser.TokenizationOption;\nimport com.twitter.search.queryparser.query.Query;\nimport com.twitter.search.queryparser.query.QueryParserException;\nimport com.twitter.util.Duration;\nimport com.twitter.util.Future;\n\npublic class QueryTokenizerFilter extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n  private static final String PREFIX = \"query_tokenizer_\";\n  private static final SearchRateCounter SUCCESS_COUNTER =\n      SearchRateCounter.export(PREFIX + \"success\");\n  private static final SearchRateCounter FAILURE_COUNTER =\n      SearchRateCounter.export(PREFIX + \"error\");\n  private static final SearchRateCounter SKIPPED_COUNTER =\n      SearchRateCounter.export(PREFIX + \"skipped\");\n  private static final SearchTimerStats QUERY_TOKENIZER_TIME =\n      SearchTimerStats.export(PREFIX + \"time\", TimeUnit.MILLISECONDS, false);\n\n  private final TokenizationOption tokenizationOption;\n\n  @Inject\n  public QueryTokenizerFilter(PenguinVersionConfig penguinversions) {\n    PenguinVersion[] supportedVersions = penguinversions\n        .getSupportedVersions().toArray(new PenguinVersion[0]);\n    tokenizationOption = new TokenizationOption(true, supportedVersions);\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n\n    if (!requestContext.getRequest().isRetokenizeSerializedQuery()\n        || !requestContext.getRequest().isSetSearchQuery()\n        || !requestContext.getRequest().getSearchQuery().isSetSerializedQuery()) {\n      SKIPPED_COUNTER.increment();\n      return service.apply(requestContext);\n    }\n\n    SearchTimer timer = QUERY_TOKENIZER_TIME.startNewTimer();\n    try {\n      String serializedQuery = requestContext.getRequest().getSearchQuery().getSerializedQuery();\n      Query parsedQuery = reparseQuery(serializedQuery);\n      SUCCESS_COUNTER.increment();\n      return service.apply(EarlybirdRequestContext.copyRequestContext(requestContext, parsedQuery));\n    } catch (QueryParserException e) {\n      FAILURE_COUNTER.increment();\n      return QueryParsingUtils.newClientErrorResponse(requestContext.getRequest(), e);\n    } finally {\n      long elapsed = timer.stop();\n      QUERY_TOKENIZER_TIME.timerIncrement(elapsed);\n      Tracing trace = Trace.apply();\n      if (trace.isActivelyTracing()) {\n        trace.record(PREFIX + \"time\", Duration.fromMilliseconds(elapsed));\n      }\n    }\n  }\n\n  public Query reparseQuery(String serializedQuery) throws QueryParserException {\n    SerializedQueryParser parser = new SerializedQueryParser(tokenizationOption);\n    return parser.parse(serializedQuery);\n  }\n\n  /**\n   * Initializing the query parser can take many seconds. We initialize it at warmup so that\n   * requests don't time out after we join the serverset. SEARCH-28801\n   */\n  public void performExpensiveInitialization() throws QueryParserException {\n    SerializedQueryParser queryParser = new SerializedQueryParser(tokenizationOption);\n\n    // The Korean query parser takes a few seconds on it's own to initialize.\n    String koreanQuery = \"스포츠\";\n    queryParser.parse(koreanQuery);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/RejectRequestsByQuerySourceFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.HashMap;\nimport java.util.Map;\nimport javax.annotation.Nullable;\nimport javax.inject.Inject;\n\nimport com.google.common.annotations.VisibleForTesting;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.constants.thriftjava.ThriftQuerySource;\nimport com.twitter.search.common.decider.SearchDecider;\nimport com.twitter.search.common.metrics.SearchRateCounter;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.util.Future;\n\n/**\n * Rejects requests based on the query source of the request. Intended to be used at super-root\n * or archive-root. If used to reject client request at super-root, the client will get a response\n * with empty results and a REQUEST_BLOCKED_ERROR status code. If used at archive-root the client\n * will get a response which might contain some results from realtime and protected and the status\n * code of the response will depend on how super-root combines responses from the three downstream\n * roots.\n */\npublic class RejectRequestsByQuerySourceFilter extends\n    SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n\n  @VisibleForTesting\n  protected static final String NUM_REJECTED_REQUESTS_STAT_NAME_PATTERN =\n      \"num_root_%s_rejected_requests_with_query_source_%s\";\n  @VisibleForTesting\n  protected static final String REJECT_REQUESTS_DECIDER_KEY_PATTERN =\n      \"root_%s_reject_requests_with_query_source_%s\";\n  private final Map<ThriftQuerySource, SearchRateCounter> rejectedRequestsCounterPerQuerySource =\n      new HashMap<>();\n  private final Map<ThriftQuerySource, String> rejectRequestsDeciderKeyPerQuerySource =\n      new HashMap<>();\n  private final SearchDecider searchDecider;\n\n\n  @Inject\n  public RejectRequestsByQuerySourceFilter(\n      @Nullable EarlybirdCluster cluster,\n      SearchDecider searchDecider) {\n\n    this.searchDecider = searchDecider;\n\n    String clusterName = cluster != null\n        ? cluster.getNameForStats()\n        : EarlybirdCluster.SUPERROOT.getNameForStats();\n\n    for (ThriftQuerySource querySource : ThriftQuerySource.values()) {\n      String querySourceName = querySource.name().toLowerCase();\n\n      rejectedRequestsCounterPerQuerySource.put(querySource,\n          SearchRateCounter.export(\n              String.format(\n                  NUM_REJECTED_REQUESTS_STAT_NAME_PATTERN, clusterName, querySourceName)));\n\n      rejectRequestsDeciderKeyPerQuerySource.put(querySource,\n          String.format(\n              REJECT_REQUESTS_DECIDER_KEY_PATTERN, clusterName, querySourceName));\n    }\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(EarlybirdRequest request,\n                                         Service<EarlybirdRequest, EarlybirdResponse> service) {\n\n    ThriftQuerySource querySource = request.isSetQuerySource()\n        ? request.getQuerySource()\n        : ThriftQuerySource.UNKNOWN;\n\n    String deciderKey = rejectRequestsDeciderKeyPerQuerySource.get(querySource);\n    if (searchDecider.isAvailable(deciderKey)) {\n      rejectedRequestsCounterPerQuerySource.get(querySource).increment();\n      return Future.value(getRejectedRequestResponse(querySource, deciderKey));\n    }\n    return service.apply(request);\n  }\n\n  private static EarlybirdResponse getRejectedRequestResponse(\n      ThriftQuerySource querySource, String deciderKey) {\n    return new EarlybirdResponse(EarlybirdResponseCode.REQUEST_BLOCKED_ERROR, 0)\n        .setSearchResults(new ThriftSearchResults())\n        .setDebugString(String.format(\n            \"Request with query source %s is blocked by decider %s\", querySource, deciderKey));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/RequestContextToEarlybirdRequestFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.concurrent.TimeUnit;\n\nimport com.twitter.finagle.Filter;\nimport com.twitter.finagle.Service;\nimport com.twitter.search.common.metrics.SearchTimerStats;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.util.Future;\n\n/**\n * A filter for transforming a RequestContext to an EarlybirdRequest.\n */\npublic class RequestContextToEarlybirdRequestFilter extends\n    Filter<EarlybirdRequestContext, EarlybirdResponse, EarlybirdRequest, EarlybirdResponse> {\n\n  private static final SearchTimerStats REQUEST_CONTEXT_TRIP_TIME =\n      SearchTimerStats.export(\"request_context_trip_time\", TimeUnit.MILLISECONDS, false,\n          true);\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequest, EarlybirdResponse> service) {\n\n    long tripTime = System.currentTimeMillis() - requestContext.getCreatedTimeMillis();\n    REQUEST_CONTEXT_TRIP_TIME.timerIncrement(tripTime);\n\n    return service.apply(requestContext.getRequest());\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/RequestResultStatsFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.List;\nimport java.util.Map;\nimport java.util.concurrent.ConcurrentHashMap;\nimport javax.inject.Inject;\n\nimport scala.runtime.BoxedUnit;\n\nimport com.twitter.common.util.Clock;\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.Percentile;\nimport com.twitter.search.common.metrics.PercentileUtil;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.query.thriftjava.CollectorParams;\nimport com.twitter.search.common.query.thriftjava.CollectorTerminationParams;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.ThriftSearchQuery;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResults;\nimport com.twitter.snowflake.id.SnowflakeId;\nimport com.twitter.util.Function;\nimport com.twitter.util.Future;\n\npublic class RequestResultStatsFilter\n    extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n  private final Clock clock;\n  private final RequestResultStats stats;\n\n  static class RequestResultStats {\n    private static final String PREFIX = \"request_result_properties_\";\n\n    private final SearchCounter resultsRequestedCount;\n    private final SearchCounter resultsReturnedCount;\n    private final SearchCounter maxHitsToProcessCount;\n    private final SearchCounter hitsProcessedCount;\n    private final SearchCounter docsProcessedCount;\n    private final SearchCounter timeoutMsCount;\n    private Map<String, Percentile<Integer>> requestedNumResultsPercentileByClientId;\n    private Map<String, Percentile<Integer>> returnedNumResultsPercentileByClientId;\n    private Map<String, Percentile<Long>> oldestResultPercentileByClientId;\n\n    RequestResultStats() {\n      // Request properties\n      resultsRequestedCount = SearchCounter.export(PREFIX + \"results_requested_cnt\");\n      maxHitsToProcessCount = SearchCounter.export(PREFIX + \"max_hits_to_process_cnt\");\n      timeoutMsCount = SearchCounter.export(PREFIX + \"timeout_ms_cnt\");\n      requestedNumResultsPercentileByClientId = new ConcurrentHashMap<>();\n\n      // Result properties\n      resultsReturnedCount = SearchCounter.export(PREFIX + \"results_returned_cnt\");\n      hitsProcessedCount = SearchCounter.export(PREFIX + \"hits_processed_cnt\");\n      docsProcessedCount = SearchCounter.export(PREFIX + \"docs_processed_cnt\");\n      returnedNumResultsPercentileByClientId = new ConcurrentHashMap<>();\n      oldestResultPercentileByClientId = new ConcurrentHashMap<>();\n    }\n\n    SearchCounter getResultsRequestedCount() {\n      return resultsRequestedCount;\n    }\n\n    SearchCounter getResultsReturnedCount() {\n      return resultsReturnedCount;\n    }\n\n    SearchCounter getMaxHitsToProcessCount() {\n      return maxHitsToProcessCount;\n    }\n\n    SearchCounter getHitsProcessedCount() {\n      return hitsProcessedCount;\n    }\n\n    SearchCounter getDocsProcessedCount() {\n      return docsProcessedCount;\n    }\n\n    SearchCounter getTimeoutMsCount() {\n      return timeoutMsCount;\n    }\n\n    Percentile<Long> getOldestResultPercentile(String clientId) {\n      return oldestResultPercentileByClientId.computeIfAbsent(clientId,\n          key -> PercentileUtil.createPercentile(statName(clientId, \"oldest_result_age_seconds\")));\n    }\n\n    Percentile<Integer> getRequestedNumResultsPercentile(String clientId) {\n      return requestedNumResultsPercentileByClientId.computeIfAbsent(clientId,\n          key -> PercentileUtil.createPercentile(statName(clientId, \"requested_num_results\")));\n    }\n\n    Percentile<Integer> getReturnedNumResultsPercentile(String clientId) {\n      return returnedNumResultsPercentileByClientId.computeIfAbsent(clientId,\n          key -> PercentileUtil.createPercentile(statName(clientId, \"returned_num_results\")));\n    }\n\n    private String statName(String clientId, String suffix) {\n      return String.format(\"%s%s_%s\", PREFIX, ClientIdUtil.formatClientId(clientId), suffix);\n    }\n  }\n\n  @Inject\n  RequestResultStatsFilter(Clock clock, RequestResultStats stats) {\n    this.clock = clock;\n    this.stats = stats;\n  }\n\n  private void updateRequestStats(EarlybirdRequest request) {\n    ThriftSearchQuery searchQuery = request.getSearchQuery();\n    CollectorParams collectorParams = searchQuery.getCollectorParams();\n\n    if (collectorParams != null) {\n      stats.getResultsRequestedCount().add(collectorParams.numResultsToReturn);\n      if (request.isSetClientId()) {\n        stats.getRequestedNumResultsPercentile(request.getClientId())\n            .record(collectorParams.numResultsToReturn);\n      }\n      CollectorTerminationParams terminationParams = collectorParams.getTerminationParams();\n      if (terminationParams != null) {\n        if (terminationParams.isSetMaxHitsToProcess()) {\n          stats.getMaxHitsToProcessCount().add(terminationParams.maxHitsToProcess);\n        }\n        if (terminationParams.isSetTimeoutMs()) {\n          stats.getTimeoutMsCount().add(terminationParams.timeoutMs);\n        }\n      }\n    } else {\n      if (searchQuery.isSetNumResults()) {\n        stats.getResultsRequestedCount().add(searchQuery.numResults);\n        if (request.isSetClientId()) {\n          stats.getRequestedNumResultsPercentile(request.getClientId())\n              .record(searchQuery.numResults);\n        }\n      }\n      if (searchQuery.isSetMaxHitsToProcess()) {\n        stats.getMaxHitsToProcessCount().add(searchQuery.maxHitsToProcess);\n      }\n      if (request.isSetTimeoutMs()) {\n        stats.getTimeoutMsCount().add(request.timeoutMs);\n      }\n    }\n  }\n\n  private void updateResultsStats(String clientId, ThriftSearchResults results) {\n    stats.getResultsReturnedCount().add(results.getResultsSize());\n    if (results.isSetNumHitsProcessed()) {\n      stats.getHitsProcessedCount().add(results.numHitsProcessed);\n    }\n\n    if (clientId != null) {\n      if (results.getResultsSize() > 0) {\n        List<ThriftSearchResult> resultsList = results.getResults();\n\n        long lastId = resultsList.get(resultsList.size() - 1).getId();\n        long tweetTime = SnowflakeId.timeFromId(lastId).inLongSeconds();\n        long tweetAge = (clock.nowMillis() / 1000) - tweetTime;\n        stats.getOldestResultPercentile(clientId).record(tweetAge);\n      }\n\n      stats.getReturnedNumResultsPercentile(clientId).record(results.getResultsSize());\n    }\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequest request,\n      Service<EarlybirdRequest, EarlybirdResponse> service) {\n\n    updateRequestStats(request);\n\n    return service.apply(request).onSuccess(\n        new Function<EarlybirdResponse, BoxedUnit>() {\n          @Override\n          public BoxedUnit apply(EarlybirdResponse response) {\n            if (response.isSetSearchResults()) {\n              updateResultsStats(request.getClientId(), response.searchResults);\n            }\n            return BoxedUnit.UNIT;\n          }\n        });\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/RequestSuccessStatsFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.concurrent.TimeUnit;\nimport javax.inject.Inject;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.root.RequestSuccessStats;\nimport com.twitter.search.common.util.FinagleUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.util.Future;\nimport com.twitter.util.FutureEventListener;\n\nimport static com.twitter.search.common.util.earlybird.EarlybirdResponseUtil.responseConsideredFailed;\n\n\n/**\n * Records cancellations, timeouts, and failures for requests that do not go through\n * ScatterGatherService (which also updates these stats, but for different requests).\n */\npublic class RequestSuccessStatsFilter\n    extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n\n  private final RequestSuccessStats stats;\n\n  @Inject\n  RequestSuccessStatsFilter(RequestSuccessStats stats) {\n    this.stats = stats;\n  }\n\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequest request,\n      Service<EarlybirdRequest, EarlybirdResponse> service) {\n\n    final long startTime = System.nanoTime();\n\n    return service.apply(request).addEventListener(\n        new FutureEventListener<EarlybirdResponse>() {\n          @Override\n          public void onSuccess(EarlybirdResponse response) {\n            boolean success = true;\n\n            if (response.getResponseCode() == EarlybirdResponseCode.CLIENT_CANCEL_ERROR) {\n              success = false;\n              stats.getCancelledRequestCount().increment();\n            } else if (response.getResponseCode() == EarlybirdResponseCode.SERVER_TIMEOUT_ERROR) {\n              success = false;\n              stats.getTimedoutRequestCount().increment();\n            } else if (responseConsideredFailed(response.getResponseCode())) {\n              success = false;\n              stats.getErroredRequestCount().increment();\n            }\n\n            long latencyNanos = System.nanoTime() - startTime;\n            stats.getRequestLatencyStats().requestComplete(\n                TimeUnit.NANOSECONDS.toMillis(latencyNanos), 0, success);\n          }\n\n          @Override\n          public void onFailure(Throwable cause) {\n            long latencyNanos = System.nanoTime() - startTime;\n            stats.getRequestLatencyStats().requestComplete(\n                TimeUnit.NANOSECONDS.toMillis(latencyNanos), 0, false);\n\n            if (FinagleUtil.isCancelException(cause)) {\n              stats.getCancelledRequestCount().increment();\n            } else if (FinagleUtil.isTimeoutException(cause)) {\n              stats.getTimedoutRequestCount().increment();\n            } else {\n              stats.getErroredRequestCount().increment();\n            }\n          }\n        });\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/ResponseCodeStatFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.Map;\n\nimport com.google.common.collect.Maps;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.util.Future;\nimport com.twitter.util.FutureEventListener;\n\npublic class ResponseCodeStatFilter\n    extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n\n  private final Map<EarlybirdResponseCode, SearchCounter> responseCodeCounters;\n\n  /**\n   * Create ResponseCodeStatFilter\n   */\n  public ResponseCodeStatFilter() {\n    responseCodeCounters = Maps.newEnumMap(EarlybirdResponseCode.class);\n    for (EarlybirdResponseCode code : EarlybirdResponseCode.values()) {\n      SearchCounter stat = SearchCounter.export(\"response_code_\" + code.name().toLowerCase());\n      responseCodeCounters.put(code, stat);\n    }\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      final EarlybirdRequest request,\n      final Service<EarlybirdRequest, EarlybirdResponse> service) {\n\n    return service.apply(request).addEventListener(\n        new FutureEventListener<EarlybirdResponse>() {\n\n          @Override\n          public void onSuccess(final EarlybirdResponse response) {\n            responseCodeCounters.get(response.getResponseCode()).increment();\n          }\n\n          @Override\n          public void onFailure(final Throwable cause) { }\n        });\n\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/ResultTierCountFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.Comparator;\nimport java.util.List;\nimport java.util.NavigableMap;\n\nimport javax.inject.Inject;\nimport javax.inject.Singleton;\n\nimport com.google.common.annotations.VisibleForTesting;\nimport com.google.common.collect.ImmutableSortedMap;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.metrics.SearchCustomGauge;\nimport com.twitter.search.earlybird.config.TierInfo;\nimport com.twitter.search.earlybird.config.TierInfoSource;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.ThriftSearchResult;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.snowflake.id.SnowflakeId;\nimport com.twitter.util.Future;\nimport com.twitter.util.FutureEventListener;\n\n/**\n * A filter to count the tier to which the oldest tweet in the results belong.\n */\n@Singleton\npublic class ResultTierCountFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n\n  private static final String COUNTER_PREFIX = \"result_tier_count\";\n  private final long firstTweetTimeSinceEpochSec;\n  private final NavigableMap<Long, SearchCounter> tierBuckets;\n  private final SearchCounter allCounter = SearchCounter.export(COUNTER_PREFIX + \"_all\");\n  private final SearchCounter noResultsCounter =\n      SearchCounter.export(COUNTER_PREFIX + \"_no_results\");\n\n  @Inject\n  @SuppressWarnings(\"unused\")\n  ResultTierCountFilter(TierInfoSource tierInfoSource) {\n    List<TierInfo> tierInfos = tierInfoSource.getTierInformation();\n    tierInfos.sort(Comparator.comparing(TierInfo::getDataStartDate));\n\n    firstTweetTimeSinceEpochSec = tierInfos.get(0).getServingRangeSinceTimeSecondsFromEpoch();\n\n    ImmutableSortedMap.Builder<Long, SearchCounter> builder = ImmutableSortedMap.naturalOrder();\n    Collections.reverse(tierInfos);\n\n    for (TierInfo tierInfo : tierInfos) {\n      SearchCounter searchCounter = SearchCounter.export(\n          String.format(\"%s_%s\", COUNTER_PREFIX, tierInfo.getTierName()));\n      builder.put(tierInfo.getServingRangeSinceTimeSecondsFromEpoch(), searchCounter);\n\n      // export cumulative metrics to sum from the latest to a lower tier\n      Collection<SearchCounter> counters = builder.build().values();\n      SearchCustomGauge.export(\n          String.format(\"%s_down_to_%s\", COUNTER_PREFIX, tierInfo.getTierName()),\n          () -> counters.stream()\n              .mapToLong(SearchCounter::get)\n              .sum());\n    }\n\n    tierBuckets = builder.build();\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext context,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n    return service.apply(context).addEventListener(\n        new FutureEventListener<EarlybirdResponse>() {\n          @Override\n          public void onFailure(Throwable cause) {\n            // do nothing\n          }\n\n          @Override\n          public void onSuccess(EarlybirdResponse response) {\n            record(response);\n          }\n        });\n  }\n\n  @VisibleForTesting\n  void record(EarlybirdResponse response) {\n    if (response.isSetSearchResults()) {\n      long minResultsStatusId = response.getSearchResults().getResults().stream()\n          .mapToLong(ThriftSearchResult::getId)\n          .min()\n          .orElse(-1);\n      getBucket(minResultsStatusId).increment();\n    }\n    allCounter.increment();\n  }\n\n  private SearchCounter getBucket(long statusId) {\n    if (statusId < 0) {\n      return noResultsCounter;\n    }\n\n    // If non-negative statusId is not a SnowflakeId, the tweet must have been created before\n    // Twepoch (2010-11-04T01:42:54Z) and thus belongs to full1.\n    long timeSinceEpochSec = firstTweetTimeSinceEpochSec;\n    if (SnowflakeId.isSnowflakeId(statusId)) {\n      timeSinceEpochSec = SnowflakeId.timeFromId(statusId).inSeconds();\n    }\n\n    return tierBuckets.floorEntry(timeSinceEpochSec).getValue();\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/ScatterGatherWithExperimentRedirectsService.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.Map;\n\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.search.common.root.ScatterGatherService;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.ExperimentCluster;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.util.Future;\n\npublic class ScatterGatherWithExperimentRedirectsService\n    extends Service<EarlybirdRequestContext, EarlybirdResponse> {\n  private final Service<EarlybirdRequestContext, EarlybirdResponse>\n      controlScatterGatherService;\n\n  private final Map<ExperimentCluster,\n      ScatterGatherService<EarlybirdRequestContext, EarlybirdResponse>>\n      experimentScatterGatherServices;\n\n  private static final Logger LOG =\n      LoggerFactory.getLogger(ScatterGatherWithExperimentRedirectsService.class);\n\n  public ScatterGatherWithExperimentRedirectsService(\n      Service<EarlybirdRequestContext, EarlybirdResponse> controlScatterGatherService,\n      Map<ExperimentCluster,\n          ScatterGatherService<EarlybirdRequestContext, EarlybirdResponse>>\n          experimentScatterGatherServices\n  ) {\n    this.controlScatterGatherService = controlScatterGatherService;\n    this.experimentScatterGatherServices = experimentScatterGatherServices;\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(EarlybirdRequestContext request) {\n    if (request.getRequest().isSetExperimentClusterToUse()) {\n      ExperimentCluster cluster = request.getRequest().getExperimentClusterToUse();\n\n      if (!experimentScatterGatherServices.containsKey(cluster)) {\n        String error = String.format(\n            \"Received invalid experiment cluster: %s\", cluster.name());\n\n        LOG.error(\"{} Request: {}\", error, request.getRequest());\n\n        return Future.value(new EarlybirdResponse()\n            .setResponseCode(EarlybirdResponseCode.CLIENT_ERROR)\n            .setDebugString(error));\n      }\n\n      return experimentScatterGatherServices.get(cluster).apply(request);\n    }\n\n    return controlScatterGatherService.apply(request);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/SearchPayloadSizeLocalContextFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.concurrent.atomic.AtomicReference;\n\nimport scala.Option;\n\nimport com.google.common.base.Preconditions;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.finagle.context.Contexts;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.root.SearchPayloadSizeFilter;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.util.Future;\n\n/**\n * A filter that sets the clientId in the local context, to be usd later by SearchPayloadSizeFilter.\n */\npublic class SearchPayloadSizeLocalContextFilter\n    extends SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n  private static final SearchCounter CLIENT_ID_CONTEXT_KEY_NOT_SET_COUNTER = SearchCounter.export(\n      \"search_payload_size_local_context_filter_client_id_context_key_not_set\");\n\n  @Override\n  public Future<EarlybirdResponse> apply(EarlybirdRequest request,\n                                         Service<EarlybirdRequest, EarlybirdResponse> service) {\n    // In production, the SearchPayloadSizeFilter.CLIENT_ID_CONTEXT_KEY should always be set\n    // (by ThriftServer). However, it's not set in tests, because tests do not start a ThriftServer.\n    Option<AtomicReference<String>> clientIdOption =\n        Contexts.local().get(SearchPayloadSizeFilter.CLIENT_ID_CONTEXT_KEY);\n    if (clientIdOption.isDefined()) {\n      AtomicReference<String> clientIdReference = clientIdOption.get();\n      Preconditions.checkArgument(clientIdReference.get() == null);\n      clientIdReference.set(request.getClientId());\n    } else {\n      CLIENT_ID_CONTEXT_KEY_NOT_SET_COUNTER.increment();\n    }\n\n    return service.apply(request);\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/SensitiveResultsTrackingFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.Set;\n\nimport com.google.common.base.Joiner;\n\nimport org.apache.thrift.TException;\nimport org.slf4j.Logger;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.metrics.SearchCounter;\nimport com.twitter.search.common.util.thrift.ThriftUtils;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.util.Future;\nimport com.twitter.util.FutureEventListener;\n\n/**\n * The general framework for earlybird root to track sensitive results.\n */\npublic abstract class SensitiveResultsTrackingFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n\n  /**\n   * The type name is used to distinguish different kinds of sensitive results in log.\n   */\n  private final String typeName;\n\n  /**\n   * The mark is to control whether to log expensive information.\n   */\n  private final boolean logDetails;\n\n  /**\n   * Constructor helps distinguish different sensitive content trackers.\n   * @param typeName The sensitive content's name (e.g. nullcast)\n   * @param logDetails Whether to log details such as serialized requests and responses\n   */\n  public SensitiveResultsTrackingFilter(final String typeName, boolean logDetails) {\n    super();\n    this.typeName = typeName;\n    this.logDetails = logDetails;\n  }\n\n  /**\n   * Get the LOG that the sensitive results can write to.\n   */\n  protected abstract Logger getLogger();\n\n  /**\n   * The counter which counts the number of queries with sensitive results.\n   */\n  protected abstract SearchCounter getSensitiveQueryCounter();\n\n  /**\n   * The counter which counts the number of sensitive results.\n   */\n  protected abstract SearchCounter getSensitiveResultsCounter();\n\n  /**\n   * The method defines how the sensitive results are identified.\n   */\n  protected abstract Set<Long> getSensitiveResults(\n      EarlybirdRequestContext requestContext,\n      EarlybirdResponse earlybirdResponse) throws Exception;\n\n  /**\n   * Get a set of tweets which should be exclude from the sensitive results set.\n   */\n  protected abstract Set<Long> getExceptedResults(EarlybirdRequestContext requestContext);\n\n  @Override\n  public final Future<EarlybirdResponse> apply(\n      final EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n    Future<EarlybirdResponse> response = service.apply(requestContext);\n\n    response.addEventListener(new FutureEventListener<EarlybirdResponse>() {\n      @Override\n      public void onSuccess(EarlybirdResponse earlybirdResponse) {\n        try {\n          if (earlybirdResponse.responseCode == EarlybirdResponseCode.SUCCESS\n              && earlybirdResponse.isSetSearchResults()\n              && requestContext.getParsedQuery() != null) {\n            Set<Long> statusIds = getSensitiveResults(requestContext, earlybirdResponse);\n            Set<Long> exceptedIds = getExceptedResults(requestContext);\n            statusIds.removeAll(exceptedIds);\n\n            if (statusIds.size() > 0) {\n              getSensitiveQueryCounter().increment();\n              getSensitiveResultsCounter().add(statusIds.size());\n              logContent(requestContext, earlybirdResponse, statusIds);\n            }\n          }\n        } catch (Exception e) {\n          getLogger().error(\"Caught exception while trying to log sensitive results for query: {}\",\n                            requestContext.getParsedQuery().serialize(), e);\n        }\n      }\n\n      @Override\n      public void onFailure(Throwable cause) {\n      }\n    });\n\n    return response;\n  }\n\n  private void logContent(\n      final EarlybirdRequestContext requestContext,\n      final EarlybirdResponse earlybirdResponse,\n      final Set<Long> statusIds) {\n\n    if (logDetails) {\n      String base64Request;\n      try {\n        base64Request = ThriftUtils.toBase64EncodedString(requestContext.getRequest());\n      } catch (TException e) {\n        base64Request = \"Failed to parse base 64 request\";\n      }\n      getLogger().error(\"Found \" + typeName\n              + \": {} | \"\n              + \"parsedQuery: {} | \"\n              + \"request: {} | \"\n              + \"base 64 request: {} | \"\n              + \"response: {}\",\n          Joiner.on(\",\").join(statusIds),\n          requestContext.getParsedQuery().serialize(),\n          requestContext.getRequest(),\n          base64Request,\n          earlybirdResponse);\n    } else {\n      getLogger().error(\"Found \" + typeName + \": {} for parsedQuery {}\",\n          Joiner.on(\",\").join(statusIds),\n          requestContext.getParsedQuery().serialize());\n    }\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/ServiceExceptionHandlingFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.util.Future;\n\n/** A per-service filter for handling exceptions. */\npublic class ServiceExceptionHandlingFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n  private final EarlybirdResponseExceptionHandler exceptionHandler;\n\n  /** Creates a new ServiceExceptionHandlingFilter instance. */\n  public ServiceExceptionHandlingFilter(EarlybirdCluster cluster) {\n    this.exceptionHandler = new EarlybirdResponseExceptionHandler(cluster.getNameForStats());\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n    return exceptionHandler.handleException(\n        requestContext.getRequest(), service.apply(requestContext));\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/ServiceResponseValidationFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport java.util.HashMap;\nimport java.util.Map;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.common.schema.earlybird.EarlybirdCluster;\nimport com.twitter.search.common.util.earlybird.EarlybirdResponseMergeUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestContext;\nimport com.twitter.search.earlybird_root.common.EarlybirdRequestType;\nimport com.twitter.search.earlybird_root.validators.FacetsResponseValidator;\nimport com.twitter.search.earlybird_root.validators.PassThroughResponseValidator;\nimport com.twitter.search.earlybird_root.validators.ServiceResponseValidator;\nimport com.twitter.search.earlybird_root.validators.TermStatsResultsValidator;\nimport com.twitter.search.earlybird_root.validators.TopTweetsResultsValidator;\nimport com.twitter.util.Function;\nimport com.twitter.util.Future;\n\n/**\n * Filter responsible for handling invalid response returned by downstream services, and\n * translating them into EarlybirdResponseExceptions.\n */\npublic class ServiceResponseValidationFilter\n    extends SimpleFilter<EarlybirdRequestContext, EarlybirdResponse> {\n\n  private final Map<EarlybirdRequestType, ServiceResponseValidator<EarlybirdResponse>>\n      requestTypeToResponseValidators = new HashMap<>();\n  private final EarlybirdCluster cluster;\n\n  /**\n   * Creates a new filter for handling invalid response\n   */\n  public ServiceResponseValidationFilter(EarlybirdCluster cluster) {\n    this.cluster = cluster;\n\n    ServiceResponseValidator<EarlybirdResponse> passThroughValidator =\n        new PassThroughResponseValidator();\n\n    requestTypeToResponseValidators\n        .put(EarlybirdRequestType.FACETS, new FacetsResponseValidator(cluster));\n    requestTypeToResponseValidators\n        .put(EarlybirdRequestType.RECENCY, passThroughValidator);\n    requestTypeToResponseValidators\n        .put(EarlybirdRequestType.RELEVANCE, passThroughValidator);\n    requestTypeToResponseValidators\n        .put(EarlybirdRequestType.STRICT_RECENCY, passThroughValidator);\n    requestTypeToResponseValidators\n        .put(EarlybirdRequestType.TERM_STATS, new TermStatsResultsValidator(cluster));\n    requestTypeToResponseValidators\n        .put(EarlybirdRequestType.TOP_TWEETS, new TopTweetsResultsValidator(cluster));\n  }\n\n  @Override\n  public Future<EarlybirdResponse> apply(\n      final EarlybirdRequestContext requestContext,\n      Service<EarlybirdRequestContext, EarlybirdResponse> service) {\n    return service.apply(requestContext).flatMap(\n        new Function<EarlybirdResponse, Future<EarlybirdResponse>>() {\n          @Override\n          public Future<EarlybirdResponse> apply(EarlybirdResponse response) {\n            if (response == null) {\n              return Future.exception(new IllegalStateException(\n                                          cluster + \" returned null response\"));\n            }\n\n            if (response.getResponseCode() == EarlybirdResponseCode.SUCCESS) {\n              return requestTypeToResponseValidators\n                .get(requestContext.getEarlybirdRequestType())\n                .validate(response);\n            }\n\n            return Future.value(EarlybirdResponseMergeUtil.transformInvalidResponse(\n                response,\n                String.format(\"Failure from %s (%s)\", cluster, response.getResponseCode())));\n          }\n        });\n  }\n}\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/filters/StratoAttributionClientIdFilter.java",
    "content": "package com.twitter.search.earlybird_root.filters;\n\nimport com.twitter.finagle.Service;\nimport com.twitter.finagle.SimpleFilter;\nimport com.twitter.search.earlybird.common.ClientIdUtil;\nimport com.twitter.search.earlybird.thrift.EarlybirdRequest;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.util.Future;\n\n/**\n * A filter that will set the clientId of the request to the strato HttpEndpoint Attribution.\n * <p>\n * If the clientId is already set to something non-null then that value is used.\n * If the clientId is null but Attribution.httpEndpoint() contains a value it will be set as\n * the clientId.\n */\npublic class StratoAttributionClientIdFilter extends\n    SimpleFilter<EarlybirdRequest, EarlybirdResponse> {\n  @Override\n  public Future<EarlybirdResponse> apply(\n      EarlybirdRequest request, Service<EarlybirdRequest, EarlybirdResponse> service\n  ) {\n    if (request.getClientId() == null) {\n      ClientIdUtil.getClientIdFromHttpEndpointAttribution().ifPresent(request::setClientId);\n    }\n\n    return service.apply(request);\n  }\n}\n\n"
  },
  {
    "path": "src/java/com/twitter/search/earlybird_root/mergers/AccumulatedResponses.java",
    "content": "package com.twitter.search.earlybird_root.mergers;\n\nimport java.util.List;\nimport java.util.Map;\n\nimport com.google.common.base.Preconditions;\nimport com.google.common.collect.Maps;\n\nimport com.twitter.search.common.query.thriftjava.EarlyTerminationInfo;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponse;\nimport com.twitter.search.earlybird.thrift.EarlybirdResponseCode;\nimport com.twitter.search.earlybird.thrift.TierResponse;\n\n/**\n * Collection of EarlybirdResponses and associated stats to be merged.\n */\npublic class AccumulatedResponses {\n  // The list of the successful responses from all earlybird futures. This does not include empty\n  // responses resulted from null requests.\n  private final List<EarlybirdResponse> successResponses;\n  // The list of the unsuccessful responses from all earlybird futures.\n  private final List<EarlybirdResponse> errorResponses;\n  // the list of max statusIds seen in each earlybird.\n  private final List<Long> maxIds;\n  // the list of min statusIds seen in each earlybird.\n  private final List<Long> minIds;\n\n  private final EarlyTerminationInfo mergedEarlyTerminationInfo;\n  private final boolean isMergingAcrossTiers;\n  private final PartitionCounts partitionCounts;\n  private final int numSearchedSegments;\n\n  public static final class PartitionCounts {\n    private final int numPartitions;\n    private final int numSuccessfulPartitions;\n    private final List<TierResponse> perTierResponse;\n\n    public PartitionCounts(int numPartitions, int numSuccessfulPartitions, List<TierResponse>\n        perTierResponse) {\n      this.numPartitions = numPartitions;\n      this.numSuccessfulPartitions = numSuccessfulPartitions;\n      this.perTierResponse = perTierResponse;\n    }\n\n    public int getNumPartitions() {\n      return numPartitions;\n    }\n\n    public int getNumSuccessfulPartitions() {\n      return numSuccessfulPartitions;\n    }\n\n    public List<TierResponse> getPerTierResponse() {\n      return perTierResponse;\n    }\n  }\n\n  /**\n   * Create AccumulatedResponses\n   */\n  public AccumulatedResponses(List<EarlybirdResponse> successResponses,\n                              List<EarlybirdResponse> errorResponses,\n                              List<Long> maxIds,\n                              List<Long> minIds,\n                              EarlyTerminationInfo mergedEarlyTerminationInfo,\n                              boolean isMergingAcrossTiers,\n                              PartitionCounts partitionCounts,\n                              int numSearchedSegments) {\n    this.successResponses = successResponses;\n    this.errorResponses = errorResponses;\n    this.maxIds = maxIds;\n    this.minIds = minIds;\n    this.mergedEarlyTerminationInfo = mergedEarlyTerminationInfo;\n    this.isMergingAcrossTiers = isMergingAcrossTiers;\n    this.partitionCounts = partitionCounts;\n    this.numSearchedSegments = numSearchedSegments;\n  }\n\n  public List<EarlybirdResponse> getSuccessResponses() {\n    return successResponses;\n  }\n\n  public List<EarlybirdResponse> getErrorResponses() {\n    return errorResponses;\n  }\n\n  public List<Long> getMaxIds() {\n    return maxIds;\n  }\n\n  public List<Long> getMinIds() {\n    return minIds;\n  }\n\n  public EarlyTerminationInfo getMergedEarlyTerminationInfo() {\n    return mergedEarlyTerminationInfo;\n  }\n\n  public boolean foundError() {\n    return !errorResponses.isEmpty();\n  }\n\n  /**\n   * Tries to return a merged EarlybirdResponse that propagates as much information from the error\n   * responses as possible.\n   *\n   * If all error responses have the same error response code, the merged response will have the\n   * same error response code, and the debugString/debugInfo on the merged response will be set to\n   * the debugString/debugInfo of one of the merged responses.\n   *\n   * If the error responses have at least 2 different response codes, TRANSIENT_ERROR will be set\n   * on the merged response. Also, we will look for the most common error response code, and will\n   * propagate the debugString/debugInfo from an error response with that response code.\n   */\n  public EarlybirdResponse getMergedErrorResponse() {\n    Preconditions.checkState(!errorResponses.isEmpty());\n\n    // Find a response that has the most common error response code.\n    int maxCount = 0;\n    EarlybirdResponse errorResponseWithMostCommonErrorResponseCode = null;\n    Map<EarlybirdResponseCode, Integer> responseCodeCounts = Maps.newHashMap();\n    for (EarlybirdResponse errorResponse : errorResponses) {\n      EarlybirdResponseCode responseCode = errorResponse.getResponseCode();\n      Integer responseCodeCount = responseCodeCounts.get(responseCode);\n      if (responseCodeCount == null) {\n        responseCodeCount = 0;\n      }\n      ++responseCodeCount;\n      responseCodeCounts.put(responseCode, responseCodeCount);\n      if (responseCodeCount > maxCount) {\n        errorResponseWithMostCommonErrorResponseCode = errorResponse;\n      }\n    }\n\n    // If all error responses have the same response code, set it on the merged response.\n    // Otherwise, set TRANSIENT_ERROR on the merged response.\n    EarlybirdResponseCode mergedResponseCode = EarlybirdResponseCode.TRANSIENT_ERROR;\n    if (responseCodeCounts.size() == 1) {\n      mergedResponseCode = responseCodeCounts.keySet().iterator().next();\n    }\n\n    EarlybirdResponse mergedResponse = new EarlybirdResponse()\n        .setResponseCode(mergedResponseCode);\n\n    // Propagate the debugString/debugInfo of the selected error response to the merged response.\n    Preconditions.checkNotNull(errorResponseWithMostCommonErrorResponseCode);\n    if (errorResponseWithMostCommonErrorResponseCode.isSetDebugString()) {\n      mergedResponse.setDebugString(errorResponseWithMostCommonErrorResponseCode.getDebugString());\n    }\n    if (errorResponseWithMostCommonErrorResponseCode.isSetDebugInfo()) {\n      mergedResponse.setDebugInfo(errorResponseWithMostCommonErrorResponseCode.getDebugInfo());\n    }\n\n    // Set the numPartitions and numPartitionsSucceeded on the mergedResponse\n    mergedResponse.setNumPartitions(partitionCounts.getNumPartitions());\n    mergedResponse.setNumSuccessfulPartitions(partitionCounts.getNumSuccessfulPartitions());\n\n    return mergedResponse;\n  }\n\n  public boolean isMergingAcrossTiers() {\n    return isMergingAcrossTiers;\n  }\n\n  public boolean isMergingPartitionsWithinATier() {\n    return !isMergingAcrossTiers;\n  }\n\n  public PartitionCounts getPartitionCounts() {\n    return partitionCounts;\n  }\n\n  public int getNumSearchedSegments() {\n    return numSearchedSegments;\n  }\n}\n"
  },
  {
    "path": "src/python/twitter/deepbird/projects/timelines/scripts/models/earlybird/__init__.py",
    "content": ""
  },
  {
    "path": "src/python/twitter/deepbird/projects/timelines/scripts/models/earlybird/lolly/__init__.py",
    "content": ""
  },
  {
    "path": "src/python/twitter/deepbird/projects/timelines/scripts/models/earlybird/tf_model/__init__.py",
    "content": ""
  },
  {
    "path": "trust_and_safety_models/toxicity/__init__.py",
    "content": ""
  },
  {
    "path": "trust_and_safety_models/toxicity/data/__init__.py",
    "content": ""
  },
  {
    "path": "trust_and_safety_models/toxicity/optim/__init__.py",
    "content": ""
  },
  {
    "path": "trust_and_safety_models/toxicity/settings/__init__.py",
    "content": ""
  },
  {
    "path": "trust_and_safety_models/toxicity/utils/__init__.py",
    "content": ""
  },
  {
    "path": "tweetypie/server/config/decider_staging.yml",
    "content": ""
  },
  {
    "path": "twml/twml/contrib/feature_importances/__init__.py",
    "content": ""
  },
  {
    "path": "twml/twml/saved_model_cli/__init__.py",
    "content": ""
  },
  {
    "path": "twml/twml/tensorboard/__init__.py",
    "content": ""
  },
  {
    "path": "twml/twml_common/__init__.py",
    "content": ""
  }
]